From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC946451982; Mon, 14 Sep 2026 11:58:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789387382; cv=none; b=rhyG3x6tB3Dl76yssfnhfBs3irpncn/UJ3NMBb+pe2k7/anwIHITdJwImM4pfwwstsbxN+dqtI47TKP6OXGXvZm0xZxraS560gOOHcYbUBbGjLOB+L01FdAZDGHNfj3qex9gvBk2/AyCkrFNDlYngwdqbUFJao+klHijmMiOQAU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789387382; c=relaxed/simple; bh=VbbpzX4i/9S1wp8kSFJZP6SQPmU4t62axMyG6q2Pk3A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=sXa2iopgcpytFxButiOW3q5kdAffbX0WJVmGuC1UqDEtCD3x7Bd4OElgjBXrbGXu7Uo3Y4XgJ6U2ucKCXPf+KtWZH6XZXuH4IJ0q/6YiWlgjpghMw95k/ESdu4x1I1siyaFDohp8RXjoSISHg6DBU1vyWOf44jMIElqD50A85IM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b=LCI8O6dX; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b="LCI8O6dX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF5AC1F0089A; Mon, 14 Sep 2026 11:58:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1789387124; bh=Q5eQDR0ANt3OeKeFLVHtLHV31v3qSmfK1BhLZpANsP0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LCI8O6dXfDOTOvffOVieNQMVB9zsEhtP0SxFLx9Nu4DB6O4PeYPDCuUt8EMALlszl ZMam6GtvX7qeQziJFIEltJ5EE9NpY/Df6wLXTfeUORY/b7G+AMEos5f1SDoL0YvDxV VlydHZiweaHJx25DQvWsqUtQRV/WBdclICd8AOcc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, torvalds@linux-foundation.org, stable@vger.kernel.org Cc: lwn@lwn.net, jslaby@suse.cz, Greg Kroah-Hartman Subject: Re: Linux 6.18.52 Date: Mon, 14 Sep 2026 13:58:21 +0200 Message-ID: <2026091421-aqueduct-skiing-4a70@gregkh> X-Mailer: git-send-email 2.55.0 In-Reply-To: <2026091420-unjustly-faceplate-5812@gregkh> References: <2026091420-unjustly-faceplate-5812@gregkh> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit diff --git a/Documentation/ABI/testing/configfs-usb-gadget-uac1_legacy b/Documentation/ABI/testing/configfs-usb-gadget-uac1_legacy index b2eaefd9bc49..6a681d219f43 100644 --- a/Documentation/ABI/testing/configfs-usb-gadget-uac1_legacy +++ b/Documentation/ABI/testing/configfs-usb-gadget-uac1_legacy @@ -5,8 +5,5 @@ Description: The attributes: audio_buf_size - audio buffer size - fn_cap - capture pcm device file name - fn_cntl - control device file name - fn_play - playback pcm device file name req_buf_size - ISO OUT endpoint request buffer size req_count - ISO OUT endpoint request count diff --git a/Documentation/admin-guide/device-mapper/dm-ima.rst b/Documentation/admin-guide/device-mapper/dm-ima.rst index a4aa50a828e0..2a3b50ffbee4 100644 --- a/Documentation/admin-guide/device-mapper/dm-ima.rst +++ b/Documentation/admin-guide/device-mapper/dm-ima.rst @@ -424,7 +424,8 @@ section above) has the following data format for 'integrity' target. target_attributes := "," "," "," "," "," [ ","] [ ","] "," - "," "," "," "," + "," "," "," "," + "," "," "," ";" target_name := "target_name=integrity" @@ -438,6 +439,7 @@ section above) has the following data format for 'integrity' target. block_size := "block_size=" recalculate := "recalculate=" allow_discards := "allow_discards=" + allow_discards_keyed := "allow_discards_keyed=" fix_padding := "fix_padding=" fix_hmac := "fix_hmac=" legacy_recalculate := "legacy_recalculate=" @@ -455,7 +457,8 @@ section above) has the following data format for 'integrity' target. dm_version=4.45.0; name=integrity1,uuid=,major=253,minor=1,minor_count=1,num_targets=1; target_index=0,target_begin=0,target_len=7856,target_name=integrity,target_version=1.10.0, - dev_name=253:0,start=0,tag_size=32,mode=J,recalculate=n,allow_discards=n,fix_padding=n, + dev_name=253:0,start=0,tag_size=32,mode=J,recalculate=n,allow_discards=n, + allow_discards_keyed=n,fix_padding=n, fix_hmac=n,legacy_recalculate=n,journal_sectors=88,interleave_sectors=32768,buffer_sectors=128; diff --git a/Documentation/admin-guide/device-mapper/dm-integrity.rst b/Documentation/admin-guide/device-mapper/dm-integrity.rst index c2e18ecc065c..9c21301423c9 100644 --- a/Documentation/admin-guide/device-mapper/dm-integrity.rst +++ b/Documentation/admin-guide/device-mapper/dm-integrity.rst @@ -190,6 +190,19 @@ allow_discards Allow block discard requests (a.k.a. TRIM) for the integrity device. Discards are only allowed to devices using internal hash. + A discarded block is marked with a constant filler tag that anyone + with raw write access to the backing device can forge without the + key. Use allow_discards_keyed instead on new volumes. + +allow_discards_keyed + Like allow_discards, but marks a discarded block with a keyed + checksum of the sector number, HMAC_key(salt || sector), instead of + the constant filler tag, so it can't be forged without the + integrity key. + + Not compatible with volumes that already have discarded blocks + marked the old way; only use on a freshly formatted volume. + fix_padding Use a smaller padding of the tag area that is more space-efficient. If this option is not present, large padding is diff --git a/Documentation/admin-guide/pm/amd-pstate.rst b/Documentation/admin-guide/pm/amd-pstate.rst index e1771f2225d5..8384b3b9d47b 100644 --- a/Documentation/admin-guide/pm/amd-pstate.rst +++ b/Documentation/admin-guide/pm/amd-pstate.rst @@ -280,16 +280,22 @@ A list of all the supported EPP preferences that could be used for These profiles represent different hints that are provided to the low-level firmware about the user's desired energy vs efficiency tradeoff. ``default`` represents the epp value is set by platform -firmware. This attribute is read-only. +firmware. ``custom`` designates that integer values 0-255 may be written +as well. This attribute is read-only. ``energy_performance_preference`` The current energy performance preference can be read from this attribute. and user can change current preference according to energy or performance needs -Please get all support profiles list from -``energy_performance_available_preferences`` attribute, all the profiles are -integer values defined between 0 to 255 when EPP feature is enabled by platform -firmware, if EPP feature is disabled, driver will ignore the written value +Coarse named profiles are available in the attribute +``energy_performance_available_preferences``. +Users can also write individual integer values between 0 to 255. +When dynamic EPP is enabled, writes to energy_performance_preference are blocked +even when EPP feature is enabled by platform firmware. Lower epp values shift the bias +towards improved performance while a higher epp value shifts the bias towards +power-savings. The exact impact can change from one platform to the other. +If a valid integer was last written, then a number will be returned on future reads. +If a valid string was last written then a string will be returned on future reads. This attribute is read-write. ``boost`` @@ -311,6 +317,24 @@ boost or `1` to enable it, for the respective CPU using the sysfs path Other performance and frequency values can be read back from ``/sys/devices/system/cpu/cpuX/acpi_cppc/``, see :ref:`cppc_sysfs`. +Dynamic energy performance profile +================================== +The amd-pstate driver supports dynamically selecting the energy performance +profile based on whether the machine is running on AC or DC power. + +Whether this behavior is enabled by default depends on the kernel +config option `CONFIG_X86_AMD_PSTATE_DYNAMIC_EPP`. This behavior can also be overridden +at runtime by the sysfs file ``/sys/devices/system/cpu/cpufreq/policyX/dynamic_epp``. + +When set to enabled, the driver will select a different energy performance +profile when the machine is running on battery or AC power. The driver will +also register with the platform profile handler to receive notifications of +user desired power state and react to those. +When set to disabled, the driver will not change the energy performance profile +based on the power source and will not react to user desired power state. + +Attempting to manually write to the ``energy_performance_preference`` sysfs +file will fail when ``dynamic_epp`` is enabled. ``amd-pstate`` vs ``acpi-cpufreq`` ====================================== diff --git a/Documentation/devicetree/bindings/media/nxp,imx8-isi.yaml b/Documentation/devicetree/bindings/media/nxp,imx8-isi.yaml index f43b91984f01..d5e631f339a0 100644 --- a/Documentation/devicetree/bindings/media/nxp,imx8-isi.yaml +++ b/Documentation/devicetree/bindings/media/nxp,imx8-isi.yaml @@ -66,7 +66,6 @@ required: - interrupts - clocks - clock-names - - fsl,blk-ctrl - ports allOf: @@ -109,6 +108,16 @@ allOf: - port@0 - port@1 + - if: + properties: + compatible: + not: + contains: + const: fsl,imx8ulp-isi + then: + required: + - fsl,blk-ctrl + additionalProperties: false examples: diff --git a/Documentation/hwmon/cros_ec_hwmon.rst b/Documentation/hwmon/cros_ec_hwmon.rst index 6db812708325..47a85cd40937 100644 --- a/Documentation/hwmon/cros_ec_hwmon.rst +++ b/Documentation/hwmon/cros_ec_hwmon.rst @@ -23,9 +23,23 @@ ChromeOS embedded controller used in Chromebooks and other devices. The channel labels exposed via hwmon are retrieved from the EC itself. -Fan and temperature readings are supported. PWM fan control is also supported if -the EC also supports setting fan PWM values and fan mode. Note that EC will -switch fan control mode back to auto when suspended. This driver will restore -the fan state to what they were before suspended when resumed. -If a fan is controllable, this driver will register that fan as a cooling device -in the thermal framework as well. +Supported features +------------------ + +Fan readings + Always supported. + +Temperature readings + Always supported. + +Temperature thresholds + If supported by the EC. + +PWM fan control + If the EC also supports setting fan PWM values and fan mode. + + Note that EC will switch fan control mode back to auto when suspended. + This driver will restore the fan state to what they were before suspended when resumed. + + If a fan is controllable, this driver will register that fan as a cooling device + in the thermal framework as well. diff --git a/Documentation/hwmon/emc1403.rst b/Documentation/hwmon/emc1403.rst index 57f833b1a800..ebf2435a76a6 100644 --- a/Documentation/hwmon/emc1403.rst +++ b/Documentation/hwmon/emc1403.rst @@ -71,10 +71,10 @@ and EMC14x8 support eight sensors (one internal, seven external). The chips implement three limits for each sensor: low (tempX_min), high (tempX_max) and critical (tempX_crit.) The chips also implement an -hysteresis mechanism which applies to all limits. The relative difference -is stored in a single register on the chip, which means that the relative -difference between the limit and its hysteresis is always the same for -all three limits. +hysteresis mechanism which applies to high and critical limits. The relative +difference is stored in a single register on the chip, which means that the +relative difference between the limit and its hysteresis is always the same +for high and critical limits. This implementation detail implies the following: diff --git a/Documentation/hwmon/hwmon-kernel-api.rst b/Documentation/hwmon/hwmon-kernel-api.rst index 1d7f1397a827..9fcde32a140d 100644 --- a/Documentation/hwmon/hwmon-kernel-api.rst +++ b/Documentation/hwmon/hwmon-kernel-api.rst @@ -85,9 +85,10 @@ removal. When using ``[devm_]hwmon_device_register_with_info()`` to register the hardware monitoring device, accesses using the associated access functions are serialised by the hardware monitoring core. If a driver needs locking -for other functions such as interrupt handlers or for attributes which are -fully implemented in the driver, hwmon_lock() and hwmon_unlock() can be used -to ensure that calls to those functions are serialized. +for other functions such as interrupt handlers, attributes which are fully +implemented in the driver, or debugfs functions, hwmon_lock() and hwmon_unlock() +can be used to ensure that calls to those functions are serialized. Those +functions also support guard() and scoped_guard() variants. Using devm_hwmon_device_register_with_info() -------------------------------------------- diff --git a/Documentation/leds/leds-st1202.rst b/Documentation/leds/leds-st1202.rst index 1a09fbfcedcf..a2353549469e 100644 --- a/Documentation/leds/leds-st1202.rst +++ b/Documentation/leds/leds-st1202.rst @@ -17,7 +17,7 @@ To be compatible with the hardware pattern format, maximum 8 tuples of brightness (PWM) and duration must be written to hw_pattern. - Min pattern duration: 22 ms -- Max pattern duration: 5660 ms +- Max pattern duration: 5610 ms The format of the hardware pattern values should be: "brightness duration brightness duration ..." diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml index 3db59c965869..426d5aa7d955 100644 --- a/Documentation/netlink/specs/devlink.yaml +++ b/Documentation/netlink/specs/devlink.yaml @@ -99,6 +99,8 @@ definitions: name: legacy - name: switchdev + - + name: switchdev-inactive - type: enum name: eswitch-inline-mode diff --git a/Documentation/networking/devlink/devlink-eswitch-attr.rst b/Documentation/networking/devlink/devlink-eswitch-attr.rst index 08bb39ab1528..eafe09abc40c 100644 --- a/Documentation/networking/devlink/devlink-eswitch-attr.rst +++ b/Documentation/networking/devlink/devlink-eswitch-attr.rst @@ -39,6 +39,10 @@ The following is a list of E-Switch attributes. rules. * ``switchdev`` allows for more advanced offloading capabilities of the E-Switch to hardware. + * ``switchdev_inactive`` switchdev mode but starts inactive, doesn't allow traffic + until explicitly activated. This mode is useful for orchestrators that + want to prepare the device in switchdev mode but only activate it when + all configurations are done. * - ``inline-mode`` - enum - Some HWs need the VF driver to put part of the packet @@ -74,3 +78,12 @@ Example Usage # enable encap-mode with legacy mode $ devlink dev eswitch set pci/0000:08:00.0 mode legacy inline-mode none encap-mode basic + + # start switchdev mode in inactive state + $ devlink dev eswitch set pci/0000:08:00.0 mode switchdev_inactive + + # setup switchdev configurations, representors, FDB entries, etc.. + ... + + # activate switchdev mode to allow traffic + $ devlink dev eswitch set pci/0000:08:00.0 mode switchdev diff --git a/Documentation/usb/gadget-testing.rst b/Documentation/usb/gadget-testing.rst index 5f90af1fb573..edc13740ff6b 100644 --- a/Documentation/usb/gadget-testing.rst +++ b/Documentation/usb/gadget-testing.rst @@ -712,9 +712,6 @@ The uac1 function provides these attributes in its function directory: =============== ==================================== audio_buf_size audio buffer size - fn_cap capture pcm device file name - fn_cntl control device file name - fn_play playback pcm device file name req_buf_size ISO OUT endpoint request buffer size req_count ISO OUT endpoint request count =============== ==================================== diff --git a/MAINTAINERS b/MAINTAINERS index 8262a885a699..022ad429aa9f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -10533,6 +10533,7 @@ S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/vdso F: include/asm-generic/vdso/vsyscall.h F: include/vdso/ +F: kernel/time/namespace_vdso.c F: kernel/time/vsyscall.c F: lib/vdso/ F: tools/testing/selftests/vDSO/ @@ -12388,6 +12389,7 @@ R: Eric Snowberg L: linux-integrity@vger.kernel.org S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/zohar/linux-integrity.git +F: include/linux/secure_boot.h F: security/integrity/ F: security/integrity/ima/ @@ -20502,6 +20504,7 @@ F: include/trace/events/timer* F: kernel/time/itimer.c F: kernel/time/posix-* F: kernel/time/namespace.c +F: kernel/time/namespace_vdso.c POWER MANAGEMENT CORE M: "Rafael J. Wysocki" diff --git a/Makefile b/Makefile index 392e92c29d42..a009a1618fd7 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 VERSION = 6 PATCHLEVEL = 18 -SUBLEVEL = 51 +SUBLEVEL = 52 EXTRAVERSION = NAME = Baby Opossum Posse @@ -1252,7 +1252,7 @@ PHONY += vmlinux_o vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS) $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_o -vmlinux.o modules.builtin.modinfo modules.builtin: vmlinux_o +vmlinux.o: vmlinux_o @: PHONY += vmlinux diff --git a/arch/Kconfig b/arch/Kconfig index 61130b88964b..ee9fd0150f48 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1841,4 +1841,7 @@ config ARCH_WANTS_PRE_LINK_VMLINUX config ARCH_HAS_CPU_ATTACK_VECTORS bool +config HAVE_ARCH_GET_SECUREBOOT + def_bool EFI + endmenu diff --git a/arch/arm/boot/dts/allwinner/sun4i-a10.dtsi b/arch/arm/boot/dts/allwinner/sun4i-a10.dtsi index 51a6464aab9a..cabf619c2e21 100644 --- a/arch/arm/boot/dts/allwinner/sun4i-a10.dtsi +++ b/arch/arm/boot/dts/allwinner/sun4i-a10.dtsi @@ -185,7 +185,7 @@ de: display-engine { pmu { compatible = "arm,cortex-a8-pmu"; - interrupts = <3>; + interrupts = <66>; }; reserved-memory { diff --git a/arch/arm/boot/dts/marvell/armada-388-helios4.dts b/arch/arm/boot/dts/marvell/armada-388-helios4.dts index ec134e22bae3..003b0708e965 100644 --- a/arch/arm/boot/dts/marvell/armada-388-helios4.dts +++ b/arch/arm/boot/dts/marvell/armada-388-helios4.dts @@ -169,6 +169,7 @@ expander0: gpio-expander@20 { gpio-controller; #gpio-cells = <2>; reg = <0x20>; + vcc-supply = <®_3p3v>; pinctrl-names = "default"; pinctrl-0 = <&pca0_pins>; interrupt-parent = <&gpio0>; @@ -201,6 +202,10 @@ temp_sensor: temp@4c { reg = <0x4c>; vcc-supply = <®_3p3v>; }; + + eeprom@53 { + vcc-supply = <®_3p3v>; + }; }; i2c@11100 { @@ -217,13 +222,17 @@ sata@a8000 { status = "okay"; #address-cells = <1>; #size-cells = <0>; + ahci-supply = <®_3p3v>; + phy-supply = <®_3p3v>; sata0: sata-port@0 { reg = <0>; + target-supply = <®_5p0v_hdd>; }; sata1: sata-port@1 { reg = <1>; + target-supply = <®_5p0v_hdd>; }; }; @@ -231,13 +240,17 @@ sata@e0000 { status = "okay"; #address-cells = <1>; #size-cells = <0>; + ahci-supply = <®_3p3v>; + phy-supply = <®_3p3v>; sata2: sata-port@0 { reg = <0>; + target-supply = <®_5p0v_hdd>; }; sata3: sata-port@1 { reg = <1>; + target-supply = <®_5p0v_hdd>; }; }; diff --git a/arch/arm/kernel/hw_breakpoint.c b/arch/arm/kernel/hw_breakpoint.c index cd4b34c96e35..38feb30dfb5f 100644 --- a/arch/arm/kernel/hw_breakpoint.c +++ b/arch/arm/kernel/hw_breakpoint.c @@ -929,10 +929,6 @@ static void hw_breakpoint_cfi_handler(struct pt_regs *regs) break; } } -#else -static void hw_breakpoint_cfi_handler(struct pt_regs *regs) -{ -} #endif /* @@ -964,9 +960,14 @@ static int hw_breakpoint_pending(unsigned long addr, unsigned int fsr, case ARM_ENTRY_SYNC_WATCHPOINT: watchpoint_handler(addr, fsr, regs); break; +#ifdef CONFIG_CFI case ARM_ENTRY_CFI_BREAKPOINT: - hw_breakpoint_cfi_handler(regs); + if (user_mode(regs)) + ret = 1; /* Don't handle userspace BKPT */ + else + hw_breakpoint_cfi_handler(regs); break; +#endif default: ret = 1; /* Unhandled fault. */ } diff --git a/arch/arm/mach-exynos/mcpm-exynos.c b/arch/arm/mach-exynos/mcpm-exynos.c index fd0dbeb93357..cb7d8a7b14e0 100644 --- a/arch/arm/mach-exynos/mcpm-exynos.c +++ b/arch/arm/mach-exynos/mcpm-exynos.c @@ -215,7 +215,7 @@ static const struct of_device_id exynos_dt_mcpm_match[] = { {}, }; -static void exynos_mcpm_setup_entry_point(void) +static void exynos_mcpm_setup_entry_point(void *data) { /* * U-Boot SPL is hardcoded to jump to the start of ns_sram_base_addr @@ -228,10 +228,14 @@ static void exynos_mcpm_setup_entry_point(void) __raw_writel(__pa_symbol(mcpm_entry_point), ns_sram_base_addr + 8); } -static struct syscore_ops exynos_mcpm_syscore_ops = { +static const struct syscore_ops exynos_mcpm_syscore_ops = { .resume = exynos_mcpm_setup_entry_point, }; +static struct syscore exynos_mcpm_syscore = { + .ops = &exynos_mcpm_syscore_ops, +}; + static int __init exynos_mcpm_init(void) { struct device_node *node; @@ -300,9 +304,9 @@ static int __init exynos_mcpm_init(void) pmu_raw_writel(value, EXYNOS_COMMON_OPTION(i)); } - exynos_mcpm_setup_entry_point(); + exynos_mcpm_setup_entry_point(NULL); - register_syscore_ops(&exynos_mcpm_syscore_ops); + register_syscore(&exynos_mcpm_syscore); return ret; } diff --git a/arch/arm/mach-exynos/suspend.c b/arch/arm/mach-exynos/suspend.c index 150a1e56dcae..22d723553f62 100644 --- a/arch/arm/mach-exynos/suspend.c +++ b/arch/arm/mach-exynos/suspend.c @@ -53,9 +53,9 @@ struct exynos_pm_data { void (*pm_prepare)(void); void (*pm_resume_prepare)(void); - void (*pm_resume)(void); - int (*pm_suspend)(void); int (*cpu_suspend)(unsigned long); + + const struct syscore_ops *syscore_ops; }; /* Used only on Exynos542x/5800 */ @@ -376,7 +376,7 @@ static void exynos5420_pm_prepare(void) } -static int exynos_pm_suspend(void) +static int exynos_pm_suspend(void *data) { exynos_pm_central_suspend(); @@ -390,7 +390,7 @@ static int exynos_pm_suspend(void) return 0; } -static int exynos5420_pm_suspend(void) +static int exynos5420_pm_suspend(void *data) { u32 this_cluster; @@ -408,7 +408,7 @@ static int exynos5420_pm_suspend(void) return 0; } -static void exynos_pm_resume(void) +static void exynos_pm_resume(void *data) { u32 cpuid = read_cpuid_part(); @@ -429,7 +429,7 @@ static void exynos_pm_resume(void) exynos_set_delayed_reset_assertion(true); } -static void exynos3250_pm_resume(void) +static void exynos3250_pm_resume(void *data) { u32 cpuid = read_cpuid_part(); @@ -473,7 +473,7 @@ static void exynos5420_prepare_pm_resume(void) } } -static void exynos5420_pm_resume(void) +static void exynos5420_pm_resume(void *data) { unsigned long tmp; @@ -596,41 +596,52 @@ static const struct platform_suspend_ops exynos_suspend_ops = { .valid = suspend_valid_only_mem, }; +static const struct syscore_ops exynos3250_syscore_ops = { + .suspend = exynos_pm_suspend, + .resume = exynos3250_pm_resume, +}; + static const struct exynos_pm_data exynos3250_pm_data = { .wkup_irq = exynos3250_wkup_irq, .wake_disable_mask = ((0xFF << 8) | (0x1F << 1)), - .pm_suspend = exynos_pm_suspend, - .pm_resume = exynos3250_pm_resume, .pm_prepare = exynos3250_pm_prepare, .cpu_suspend = exynos3250_cpu_suspend, + .syscore_ops = &exynos3250_syscore_ops, +}; + +static const struct syscore_ops exynos_syscore_ops = { + .suspend = exynos_pm_suspend, + .resume = exynos_pm_resume, }; static const struct exynos_pm_data exynos4_pm_data = { .wkup_irq = exynos4_wkup_irq, .wake_disable_mask = ((0xFF << 8) | (0x1F << 1)), - .pm_suspend = exynos_pm_suspend, - .pm_resume = exynos_pm_resume, .pm_prepare = exynos_pm_prepare, .cpu_suspend = exynos_cpu_suspend, + .syscore_ops = &exynos_syscore_ops, }; static const struct exynos_pm_data exynos5250_pm_data = { .wkup_irq = exynos5250_wkup_irq, .wake_disable_mask = ((0xFF << 8) | (0x1F << 1)), - .pm_suspend = exynos_pm_suspend, - .pm_resume = exynos_pm_resume, .pm_prepare = exynos_pm_prepare, .cpu_suspend = exynos_cpu_suspend, + .syscore_ops = &exynos_syscore_ops, +}; + +static const struct syscore_ops exynos5420_syscore_ops = { + .resume = exynos5420_pm_resume, + .suspend = exynos5420_pm_suspend, }; static const struct exynos_pm_data exynos5420_pm_data = { .wkup_irq = exynos5250_wkup_irq, .wake_disable_mask = (0x7F << 7) | (0x1F << 1), .pm_resume_prepare = exynos5420_prepare_pm_resume, - .pm_resume = exynos5420_pm_resume, - .pm_suspend = exynos5420_pm_suspend, .pm_prepare = exynos5420_pm_prepare, .cpu_suspend = exynos5420_cpu_suspend, + .syscore_ops = &exynos5420_syscore_ops, }; static const struct of_device_id exynos_pmu_of_device_ids[] __initconst = { @@ -656,7 +667,7 @@ static const struct of_device_id exynos_pmu_of_device_ids[] __initconst = { { /*sentinel*/ }, }; -static struct syscore_ops exynos_pm_syscore_ops; +static struct syscore exynos_pm_syscore; void __init exynos_pm_init(void) { @@ -684,10 +695,9 @@ void __init exynos_pm_init(void) tmp |= pm_data->wake_disable_mask; pmu_raw_writel(tmp, S5P_WAKEUP_MASK); - exynos_pm_syscore_ops.suspend = pm_data->pm_suspend; - exynos_pm_syscore_ops.resume = pm_data->pm_resume; + exynos_pm_syscore.ops = pm_data->syscore_ops; - register_syscore_ops(&exynos_pm_syscore_ops); + register_syscore(&exynos_pm_syscore); suspend_set_ops(&exynos_suspend_ops); /* diff --git a/arch/arm/mach-imx/src.c b/arch/arm/mach-imx/src.c index 59a8e8cc4469..c3c80b4c3d53 100644 --- a/arch/arm/mach-imx/src.c +++ b/arch/arm/mach-imx/src.c @@ -171,6 +171,7 @@ void __init imx_src_init(void) if (!np) return; src_base = of_iomap(np, 0); + of_node_put(np); WARN_ON(!src_base); /* @@ -195,6 +196,7 @@ void __init imx7_src_init(void) return; src_base = of_iomap(np, 0); + of_node_put(np); if (!src_base) return; @@ -203,6 +205,7 @@ void __init imx7_src_init(void) return; gpc_base = of_iomap(np, 0); + of_node_put(np); if (!gpc_base) return; } diff --git a/arch/arm/mach-lpc32xx/common.c b/arch/arm/mach-lpc32xx/common.c index 304ea61a0716..35ed3569c5a3 100644 --- a/arch/arm/mach-lpc32xx/common.c +++ b/arch/arm/mach-lpc32xx/common.c @@ -106,7 +106,7 @@ void __init lpc32xx_map_io(void) iotable_init(lpc32xx_io_desc, ARRAY_SIZE(lpc32xx_io_desc)); } -static int __init lpc32xx_check_uid(void) +void __init lpc32xx_check_uid(void) { u32 uid[4]; @@ -119,7 +119,4 @@ static int __init lpc32xx_check_uid(void) system_serial_low = uid[0]; system_serial_high = uid[1]; } - - return 1; } -arch_initcall(lpc32xx_check_uid); diff --git a/arch/arm/mach-lpc32xx/common.h b/arch/arm/mach-lpc32xx/common.h index 32f0ad217807..06b20bea324e 100644 --- a/arch/arm/mach-lpc32xx/common.h +++ b/arch/arm/mach-lpc32xx/common.h @@ -16,6 +16,8 @@ * Other arch specific structures and functions */ extern void __init lpc32xx_map_io(void); +extern void __init lpc32xx_check_uid(void); +extern void __init lpc32xx_pm_init(void); extern void __init lpc32xx_serial_init(void); /* diff --git a/arch/arm/mach-lpc32xx/phy3250.c b/arch/arm/mach-lpc32xx/phy3250.c index 66701bf43248..ddc6333ca55d 100644 --- a/arch/arm/mach-lpc32xx/phy3250.c +++ b/arch/arm/mach-lpc32xx/phy3250.c @@ -71,6 +71,8 @@ static const struct of_dev_auxdata lpc32xx_auxdata_lookup[] __initconst = { static void __init lpc3250_machine_init(void) { + lpc32xx_check_uid(); + lpc32xx_pm_init(); lpc32xx_serial_init(); of_platform_default_populate(NULL, lpc32xx_auxdata_lookup, NULL); diff --git a/arch/arm/mach-lpc32xx/pm.c b/arch/arm/mach-lpc32xx/pm.c index 2572bd89a5e8..9b5c5e1462ed 100644 --- a/arch/arm/mach-lpc32xx/pm.c +++ b/arch/arm/mach-lpc32xx/pm.c @@ -120,7 +120,7 @@ static const struct platform_suspend_ops lpc32xx_pm_ops = { #define EMC_DYN_MEM_CTRL_OFS 0x20 #define EMC_SRMMC (1 << 3) #define EMC_CTRL_REG io_p2v(LPC32XX_EMC_BASE + EMC_DYN_MEM_CTRL_OFS) -static int __init lpc32xx_pm_init(void) +void __init lpc32xx_pm_init(void) { /* * Setup SDRAM self-refresh clock to automatically disable o @@ -129,7 +129,4 @@ static int __init lpc32xx_pm_init(void) __raw_writel(__raw_readl(EMC_CTRL_REG) | EMC_SRMMC, EMC_CTRL_REG); suspend_set_ops(&lpc32xx_pm_ops); - - return 0; } -arch_initcall(lpc32xx_pm_init); diff --git a/arch/arm/mach-pxa/generic.h b/arch/arm/mach-pxa/generic.h index c9c2c46ecead..caad4fca8de3 100644 --- a/arch/arm/mach-pxa/generic.h +++ b/arch/arm/mach-pxa/generic.h @@ -34,9 +34,9 @@ extern void __init pxa27x_map_io(void); extern void __init pxa3xx_init_irq(void); extern void __init pxa3xx_map_io(void); -extern struct syscore_ops pxa_irq_syscore_ops; -extern struct syscore_ops pxa2xx_mfp_syscore_ops; -extern struct syscore_ops pxa3xx_mfp_syscore_ops; +extern struct syscore pxa_irq_syscore; +extern struct syscore pxa2xx_mfp_syscore; +extern struct syscore pxa3xx_mfp_syscore; void __init pxa_set_ffuart_info(void *info); void __init pxa_set_btuart_info(void *info); diff --git a/arch/arm/mach-pxa/irq.c b/arch/arm/mach-pxa/irq.c index 5bfce8aa4102..99acebbbf065 100644 --- a/arch/arm/mach-pxa/irq.c +++ b/arch/arm/mach-pxa/irq.c @@ -178,7 +178,7 @@ void __init pxa_init_irq(int irq_nr, int (*fn)(struct irq_data *, unsigned int)) static unsigned long saved_icmr[MAX_INTERNAL_IRQS/32]; static unsigned long saved_ipr[MAX_INTERNAL_IRQS]; -static int pxa_irq_suspend(void) +static int pxa_irq_suspend(void *data) { int i; @@ -197,7 +197,7 @@ static int pxa_irq_suspend(void) return 0; } -static void pxa_irq_resume(void) +static void pxa_irq_resume(void *data) { int i; @@ -219,11 +219,15 @@ static void pxa_irq_resume(void) #define pxa_irq_resume NULL #endif -struct syscore_ops pxa_irq_syscore_ops = { +static const struct syscore_ops pxa_irq_syscore_ops = { .suspend = pxa_irq_suspend, .resume = pxa_irq_resume, }; +struct syscore pxa_irq_syscore = { + .ops = &pxa_irq_syscore_ops, +}; + #ifdef CONFIG_OF static const struct of_device_id intc_ids[] __initconst = { { .compatible = "marvell,pxa-intc", }, diff --git a/arch/arm/mach-pxa/mfp-pxa2xx.c b/arch/arm/mach-pxa/mfp-pxa2xx.c index f5a3d890f682..d1347055fbe4 100644 --- a/arch/arm/mach-pxa/mfp-pxa2xx.c +++ b/arch/arm/mach-pxa/mfp-pxa2xx.c @@ -346,7 +346,7 @@ static unsigned long saved_gpdr[4]; static unsigned long saved_gplr[4]; static unsigned long saved_pgsr[4]; -static int pxa2xx_mfp_suspend(void) +static int pxa2xx_mfp_suspend(void *data) { int i; @@ -385,7 +385,7 @@ static int pxa2xx_mfp_suspend(void) return 0; } -static void pxa2xx_mfp_resume(void) +static void pxa2xx_mfp_resume(void *data) { int i; @@ -404,11 +404,15 @@ static void pxa2xx_mfp_resume(void) #define pxa2xx_mfp_resume NULL #endif -struct syscore_ops pxa2xx_mfp_syscore_ops = { +static const struct syscore_ops pxa2xx_mfp_syscore_ops = { .suspend = pxa2xx_mfp_suspend, .resume = pxa2xx_mfp_resume, }; +struct syscore pxa2xx_mfp_syscore = { + .ops = &pxa2xx_mfp_syscore_ops, +}; + static int __init pxa2xx_mfp_init(void) { int i; diff --git a/arch/arm/mach-pxa/mfp-pxa3xx.c b/arch/arm/mach-pxa/mfp-pxa3xx.c index d16ab7451efe..fe7498fbb62b 100644 --- a/arch/arm/mach-pxa/mfp-pxa3xx.c +++ b/arch/arm/mach-pxa/mfp-pxa3xx.c @@ -27,13 +27,13 @@ * a pull-down mode if they're an active low chip select, and we're * just entering standby. */ -static int pxa3xx_mfp_suspend(void) +static int pxa3xx_mfp_suspend(void *data) { mfp_config_lpm(); return 0; } -static void pxa3xx_mfp_resume(void) +static void pxa3xx_mfp_resume(void *data) { mfp_config_run(); @@ -49,7 +49,11 @@ static void pxa3xx_mfp_resume(void) #define pxa3xx_mfp_resume NULL #endif -struct syscore_ops pxa3xx_mfp_syscore_ops = { +static const struct syscore_ops pxa3xx_mfp_syscore_ops = { .suspend = pxa3xx_mfp_suspend, .resume = pxa3xx_mfp_resume, }; + +struct syscore pxa3xx_mfp_syscore = { + .ops = &pxa3xx_mfp_syscore_ops, +}; diff --git a/arch/arm/mach-pxa/pxa25x.c b/arch/arm/mach-pxa/pxa25x.c index 03e34841fc00..70509a599814 100644 --- a/arch/arm/mach-pxa/pxa25x.c +++ b/arch/arm/mach-pxa/pxa25x.c @@ -235,8 +235,8 @@ static int __init pxa25x_init(void) pxa25x_init_pm(); - register_syscore_ops(&pxa_irq_syscore_ops); - register_syscore_ops(&pxa2xx_mfp_syscore_ops); + register_syscore(&pxa_irq_syscore); + register_syscore(&pxa2xx_mfp_syscore); if (!of_have_populated_dt()) { software_node_register(&pxa2xx_gpiochip_node); diff --git a/arch/arm/mach-pxa/pxa27x.c b/arch/arm/mach-pxa/pxa27x.c index f8382477d629..ff6361979038 100644 --- a/arch/arm/mach-pxa/pxa27x.c +++ b/arch/arm/mach-pxa/pxa27x.c @@ -337,8 +337,8 @@ static int __init pxa27x_init(void) pxa27x_init_pm(); - register_syscore_ops(&pxa_irq_syscore_ops); - register_syscore_ops(&pxa2xx_mfp_syscore_ops); + register_syscore(&pxa_irq_syscore); + register_syscore(&pxa2xx_mfp_syscore); if (!of_have_populated_dt()) { software_node_register(&pxa2xx_gpiochip_node); diff --git a/arch/arm/mach-pxa/pxa3xx.c b/arch/arm/mach-pxa/pxa3xx.c index 1d1e5713464d..06c578ea658e 100644 --- a/arch/arm/mach-pxa/pxa3xx.c +++ b/arch/arm/mach-pxa/pxa3xx.c @@ -424,8 +424,8 @@ static int __init pxa3xx_init(void) if (cpu_is_pxa320()) enable_irq_wake(IRQ_WAKEUP1); - register_syscore_ops(&pxa_irq_syscore_ops); - register_syscore_ops(&pxa3xx_mfp_syscore_ops); + register_syscore(&pxa_irq_syscore); + register_syscore(&pxa3xx_mfp_syscore); } return ret; diff --git a/arch/arm/mach-pxa/smemc.c b/arch/arm/mach-pxa/smemc.c index 2d2a321d82f8..fb93a8f28356 100644 --- a/arch/arm/mach-pxa/smemc.c +++ b/arch/arm/mach-pxa/smemc.c @@ -18,7 +18,7 @@ static unsigned long msc[2]; static unsigned long sxcnfg, memclkcfg; static unsigned long csadrcfg[4]; -static int pxa3xx_smemc_suspend(void) +static int pxa3xx_smemc_suspend(void *data) { msc[0] = __raw_readl(MSC0); msc[1] = __raw_readl(MSC1); @@ -32,7 +32,7 @@ static int pxa3xx_smemc_suspend(void) return 0; } -static void pxa3xx_smemc_resume(void) +static void pxa3xx_smemc_resume(void *data) { __raw_writel(msc[0], MSC0); __raw_writel(msc[1], MSC1); @@ -46,11 +46,15 @@ static void pxa3xx_smemc_resume(void) __raw_writel(0x2, CSMSADRCFG); } -static struct syscore_ops smemc_syscore_ops = { +static const struct syscore_ops smemc_syscore_ops = { .suspend = pxa3xx_smemc_suspend, .resume = pxa3xx_smemc_resume, }; +static struct syscore smemc_syscore = { + .ops = &smemc_syscore_ops, +}; + static int __init smemc_init(void) { if (cpu_is_pxa3xx()) { @@ -64,7 +68,7 @@ static int __init smemc_init(void) */ __raw_writel(0x2, CSMSADRCFG); - register_syscore_ops(&smemc_syscore_ops); + register_syscore(&smemc_syscore); } return 0; diff --git a/arch/arm/mach-s3c/irq-pm-s3c64xx.c b/arch/arm/mach-s3c/irq-pm-s3c64xx.c index 4a1e935bada1..ab726c595001 100644 --- a/arch/arm/mach-s3c/irq-pm-s3c64xx.c +++ b/arch/arm/mach-s3c/irq-pm-s3c64xx.c @@ -58,7 +58,7 @@ static struct irq_grp_save { static u32 irq_uart_mask[SERIAL_SAMSUNG_UARTS]; -static int s3c64xx_irq_pm_suspend(void) +static int s3c64xx_irq_pm_suspend(void *data) { struct irq_grp_save *grp = eint_grp_save; int i; @@ -79,7 +79,7 @@ static int s3c64xx_irq_pm_suspend(void) return 0; } -static void s3c64xx_irq_pm_resume(void) +static void s3c64xx_irq_pm_resume(void *data) { struct irq_grp_save *grp = eint_grp_save; int i; @@ -100,18 +100,22 @@ static void s3c64xx_irq_pm_resume(void) S3C_PMDBG("%s: IRQ configuration restored\n", __func__); } -static struct syscore_ops s3c64xx_irq_syscore_ops = { +static const struct syscore_ops s3c64xx_irq_syscore_ops = { .suspend = s3c64xx_irq_pm_suspend, .resume = s3c64xx_irq_pm_resume, }; +static struct syscore s3c64xx_irq_syscore = { + .ops = &s3c64xx_irq_syscore_ops, +}; + static __init int s3c64xx_syscore_init(void) { /* Appropriate drivers (pinctrl, uart) handle this when using DT. */ if (of_have_populated_dt() || !soc_is_s3c64xx()) return 0; - register_syscore_ops(&s3c64xx_irq_syscore_ops); + register_syscore(&s3c64xx_irq_syscore); return 0; } diff --git a/arch/arm/mach-s5pv210/pm.c b/arch/arm/mach-s5pv210/pm.c index 6fa70f787df4..fa270750364c 100644 --- a/arch/arm/mach-s5pv210/pm.c +++ b/arch/arm/mach-s5pv210/pm.c @@ -195,20 +195,24 @@ static const struct platform_suspend_ops s5pv210_suspend_ops = { /* * Syscore operations used to delay restore of certain registers. */ -static void s5pv210_pm_resume(void) +static void s5pv210_pm_resume(void *data) { s3c_pm_do_restore_core(s5pv210_core_save, ARRAY_SIZE(s5pv210_core_save)); } -static struct syscore_ops s5pv210_pm_syscore_ops = { +static const struct syscore_ops s5pv210_pm_syscore_ops = { .resume = s5pv210_pm_resume, }; +static struct syscore s5pv210_pm_syscore = { + .ops = &s5pv210_pm_syscore_ops, +}; + /* * Initialization entry point. */ void __init s5pv210_pm_init(void) { - register_syscore_ops(&s5pv210_pm_syscore_ops); + register_syscore(&s5pv210_pm_syscore); suspend_set_ops(&s5pv210_suspend_ops); } diff --git a/arch/arm/mach-versatile/integrator_ap.c b/arch/arm/mach-versatile/integrator_ap.c index 4bd6712e9f52..ee90d6619d0d 100644 --- a/arch/arm/mach-versatile/integrator_ap.c +++ b/arch/arm/mach-versatile/integrator_ap.c @@ -63,13 +63,13 @@ static void __init ap_map_io(void) #ifdef CONFIG_PM static unsigned long ic_irq_enable; -static int irq_suspend(void) +static int irq_suspend(void *data) { ic_irq_enable = readl(VA_IC_BASE + IRQ_ENABLE); return 0; } -static void irq_resume(void) +static void irq_resume(void *data) { /* disable all irq sources */ cm_clear_irqs(); @@ -83,14 +83,18 @@ static void irq_resume(void) #define irq_resume NULL #endif -static struct syscore_ops irq_syscore_ops = { +static const struct syscore_ops irq_syscore_ops = { .suspend = irq_suspend, .resume = irq_resume, }; +static struct syscore irq_syscore = { + .ops = &irq_syscore_ops, +}; + static int __init irq_syscore_init(void) { - register_syscore_ops(&irq_syscore_ops); + register_syscore(&irq_syscore); return 0; } diff --git a/arch/arm/mm/cache-b15-rac.c b/arch/arm/mm/cache-b15-rac.c index 6f63b90f9e1a..e7807356dfab 100644 --- a/arch/arm/mm/cache-b15-rac.c +++ b/arch/arm/mm/cache-b15-rac.c @@ -256,7 +256,7 @@ static int b15_rac_dead_cpu(unsigned int cpu) return 0; } -static int b15_rac_suspend(void) +static int b15_rac_suspend(void *data) { /* Suspend the read-ahead cache oeprations, forcing our cache * implementation to fallback to the regular ARMv7 calls. @@ -271,7 +271,7 @@ static int b15_rac_suspend(void) return 0; } -static void b15_rac_resume(void) +static void b15_rac_resume(void *data) { /* Coming out of a S3 suspend/resume cycle, the read-ahead cache * register RAC_CONFIG0_REG will be restored to its default value, make @@ -282,11 +282,15 @@ static void b15_rac_resume(void) clear_bit(RAC_SUSPENDED, &b15_rac_flags); } -static struct syscore_ops b15_rac_syscore_ops = { +static const struct syscore_ops b15_rac_syscore_ops = { .suspend = b15_rac_suspend, .resume = b15_rac_resume, }; +static struct syscore b15_rac_syscore = { + .ops = &b15_rac_syscore_ops, +}; + static int __init b15_rac_init(void) { struct device_node *dn, *cpu_dn; @@ -347,7 +351,7 @@ static int __init b15_rac_init(void) } if (IS_ENABLED(CONFIG_PM_SLEEP)) - register_syscore_ops(&b15_rac_syscore_ops); + register_syscore(&b15_rac_syscore); spin_lock(&rac_lock); reg = __raw_readl(b15_rac_base + RAC_CONFIG0_REG); diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index ed4330cc3f4e..65b652b4dec3 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -204,7 +204,11 @@ __do_user_fault(unsigned long addr, unsigned int fsr, unsigned int sig, pr_err("8<--- cut here ---\n"); pr_err("%s: unhandled page fault (%d) at 0x%08lx, code 0x%03x\n", tsk->comm, sig, addr, fsr); - show_pte(KERN_ERR, tsk->mm, addr); + if (likely(addr < TASK_SIZE)) { + mmap_write_lock(tsk->mm); + show_pte(KERN_ERR, tsk->mm, addr); + mmap_write_unlock(tsk->mm); + } show_regs(regs); } #endif @@ -648,7 +652,15 @@ do_DataAbort(unsigned long addr, unsigned int fsr, struct pt_regs *regs) pr_alert("8<--- cut here ---\n"); pr_alert("Unhandled fault: %s (0x%03x) at 0x%08lx\n", inf->name, fsr, addr); - show_pte(KERN_ALERT, current->mm, addr); + if (likely(user_mode(regs))) { + if (addr < TASK_SIZE) { + mmap_write_lock(current->mm); + show_pte(KERN_ALERT, current->mm, addr); + mmap_write_unlock(current->mm); + } + } else { + show_pte(KERN_ALERT, current->mm, addr); + } arm_notify_die("", regs, inf->sig, inf->code, (void __user *)addr, fsr, 0); diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9e834f0f8dd0..f7c4cb4fbaf1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2112,6 +2112,8 @@ config ARM64_BTI_KERNEL depends on !CC_IS_GCC || GCC_VERSION >= 100100 # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106671 depends on !CC_IS_GCC + # https://github.com/llvm/llvm-project/issues/215547 + depends on !CC_IS_CLANG || CLANG_VERSION < 210000 depends on (!FUNCTION_GRAPH_TRACER || DYNAMIC_FTRACE_WITH_ARGS) help Build the kernel with Branch Target Identification annotations diff --git a/arch/arm64/boot/dts/allwinner/sun50i-a64-pinephone.dtsi b/arch/arm64/boot/dts/allwinner/sun50i-a64-pinephone.dtsi index 4bc6c1ef2cde..f958bdbb0d33 100644 --- a/arch/arm64/boot/dts/allwinner/sun50i-a64-pinephone.dtsi +++ b/arch/arm64/boot/dts/allwinner/sun50i-a64-pinephone.dtsi @@ -230,8 +230,8 @@ accelerometer@68 { interrupts = <7 5 IRQ_TYPE_EDGE_RISING>; /* PH5 */ vdd-supply = <®_dldo1>; vddio-supply = <®_dldo1>; - mount-matrix = "0", "1", "0", - "-1", "0", "0", + mount-matrix = "0", "-1", "0", + "1", "0", "0", "0", "0", "1"; }; }; diff --git a/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts b/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts index 9611775b81ee..7c8a8529dbc5 100644 --- a/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts +++ b/arch/arm64/boot/dts/amlogic/meson-axg-s400.dts @@ -432,6 +432,10 @@ gpio_speaker: gpio-controller@1f { }; }; +&mipi_pcie_analog_dphy { + status = "okay"; +}; + &pdm { pinctrl-0 = <&pdm_dclk_a14_pins>, <&pdm_din0_pins>, <&pdm_din1_pins>, <&pdm_din2_pins>, <&pdm_din3_pins>; diff --git a/arch/arm64/boot/dts/amlogic/meson-axg.dtsi b/arch/arm64/boot/dts/amlogic/meson-axg.dtsi index 3058b60338db..e9c1aae95fe7 100644 --- a/arch/arm64/boot/dts/amlogic/meson-axg.dtsi +++ b/arch/arm64/boot/dts/amlogic/meson-axg.dtsi @@ -481,7 +481,8 @@ mux { "nand_ale", "nand_cle", "nand_wen_clk", - "nand_ren_wr"; + "nand_ren_wr", + "nand_rb0"; function = "nand"; input-enable; bias-pull-up; diff --git a/arch/arm64/boot/dts/freescale/imx8-ss-audio.dtsi b/arch/arm64/boot/dts/freescale/imx8-ss-audio.dtsi index c32a6947ae9c..5ddb90be5460 100644 --- a/arch/arm64/boot/dts/freescale/imx8-ss-audio.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8-ss-audio.dtsi @@ -124,10 +124,10 @@ asrc0: asrc@59000000 { compatible = "fsl,imx8qm-asrc"; reg = <0x59000000 0x10000>; interrupts = ; - clocks = <&asrc0_lpcg IMX_LPCG_CLK_0>, - <&asrc0_lpcg IMX_LPCG_CLK_0>, - <&aud_pll_div0_lpcg IMX_LPCG_CLK_4>, - <&aud_pll_div1_lpcg IMX_LPCG_CLK_4>, + clocks = <&asrc0_lpcg IMX_LPCG_CLK_4>, + <&asrc0_lpcg IMX_LPCG_CLK_4>, + <&aud_pll_div0_lpcg IMX_LPCG_CLK_0>, + <&aud_pll_div1_lpcg IMX_LPCG_CLK_0>, <&acm IMX_ADMA_ACM_AUD_CLK0_SEL>, <&acm IMX_ADMA_ACM_AUD_CLK1_SEL>, <&clk_dummy>, diff --git a/arch/arm64/boot/dts/freescale/imx93-kontron-osm-s.dtsi b/arch/arm64/boot/dts/freescale/imx93-kontron-osm-s.dtsi index c79b1df339db..f881912cde46 100644 --- a/arch/arm64/boot/dts/freescale/imx93-kontron-osm-s.dtsi +++ b/arch/arm64/boot/dts/freescale/imx93-kontron-osm-s.dtsi @@ -15,9 +15,9 @@ aliases { rtc1 = &bbnsm_rtc; }; - memory@40000000 { + memory@80000000 { device_type = "memory"; - reg = <0x0 0x40000000 0 0x80000000>; + reg = <0x0 0x80000000 0 0x40000000>; }; chosen { diff --git a/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts b/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts index f4d73c8b1a6d..bd00e7df87c6 100644 --- a/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts +++ b/arch/arm64/boot/dts/marvell/armada-3720-turris-mox.dts @@ -290,7 +290,8 @@ connector { &usb3 { status = "okay"; - phys = <&comphy2 0>; + phys = <&usb2_utmi_otg_phy>, <&comphy2 0>; + phy-names = "usb2-phy", "usb3-phy"; }; &mdio { diff --git a/arch/arm64/boot/dts/qcom/hamoa.dtsi b/arch/arm64/boot/dts/qcom/hamoa.dtsi index adb18c10dbd9..6a456b3694fc 100644 --- a/arch/arm64/boot/dts/qcom/hamoa.dtsi +++ b/arch/arm64/boot/dts/qcom/hamoa.dtsi @@ -2826,7 +2826,7 @@ usb_1_ss0_hsphy: phy@fd3000 { reg = <0 0x00fd3000 0 0x154>; #phy-cells = <0>; - clocks = <&tcsr TCSR_USB2_1_CLKREF_EN>; + clocks = <&rpmhcc RPMH_CXO_CLK>; clock-names = "ref"; resets = <&gcc GCC_QUSB2PHY_PRIM_BCR>; @@ -2968,7 +2968,7 @@ usb_1_ss2_hsphy: phy@fde000 { reg = <0 0x00fde000 0 0x154>; #phy-cells = <0>; - clocks = <&tcsr TCSR_USB2_1_CLKREF_EN>; + clocks = <&tcsr TCSR_USB2_2_CLKREF_EN>; clock-names = "ref"; resets = <&gcc GCC_QUSB2PHY_TERT_BCR>; diff --git a/arch/arm64/boot/dts/qcom/ipq5018.dtsi b/arch/arm64/boot/dts/qcom/ipq5018.dtsi index f024b3cba33f..40b5de02d5c5 100644 --- a/arch/arm64/boot/dts/qcom/ipq5018.dtsi +++ b/arch/arm64/boot/dts/qcom/ipq5018.dtsi @@ -229,6 +229,9 @@ ge_phy: ethernet-phy@7 { compatible = "ethernet-phy-id004d.d0c0"; reg = <7>; + clocks = <&gcc GCC_GEPHY_RX_CLK>, + <&gcc GCC_GEPHY_TX_CLK>; + clock-names = "rx", "tx"; resets = <&gcc GCC_GEPHY_MISC_ARES>; }; }; @@ -256,7 +259,7 @@ cmn_pll: clock-controller@9b000 { "sys"; #clock-cells = <1>; assigned-clocks = <&cmn_pll IPQ5018_CMN_PLL_CLK>; - assigned-clock-rates-u64 = /bits/ 64 <9600000000>; + assigned-clock-rates-u64 = /bits/ 64 <4800000000>; }; qfprom: qfprom@a0000 { diff --git a/arch/arm64/boot/dts/qcom/lemans.dtsi b/arch/arm64/boot/dts/qcom/lemans.dtsi index 496e319d7b11..806b6b5d4b20 100644 --- a/arch/arm64/boot/dts/qcom/lemans.dtsi +++ b/arch/arm64/boot/dts/qcom/lemans.dtsi @@ -518,90 +518,18 @@ scm { }; }; - aggre1_noc: interconnect-aggre1-noc { - compatible = "qcom,sa8775p-aggre1-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - aggre2_noc: interconnect-aggre2-noc { - compatible = "qcom,sa8775p-aggre2-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - clk_virt: interconnect-clk-virt { compatible = "qcom,sa8775p-clk-virt"; #interconnect-cells = <2>; qcom,bcm-voters = <&apps_bcm_voter>; }; - config_noc: interconnect-config-noc { - compatible = "qcom,sa8775p-config-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - dc_noc: interconnect-dc-noc { - compatible = "qcom,sa8775p-dc-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - gem_noc: interconnect-gem-noc { - compatible = "qcom,sa8775p-gem-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - gpdsp_anoc: interconnect-gpdsp-anoc { - compatible = "qcom,sa8775p-gpdsp-anoc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - lpass_ag_noc: interconnect-lpass-ag-noc { - compatible = "qcom,sa8775p-lpass-ag-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - mc_virt: interconnect-mc-virt { compatible = "qcom,sa8775p-mc-virt"; #interconnect-cells = <2>; qcom,bcm-voters = <&apps_bcm_voter>; }; - mmss_noc: interconnect-mmss-noc { - compatible = "qcom,sa8775p-mmss-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - nspa_noc: interconnect-nspa-noc { - compatible = "qcom,sa8775p-nspa-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - nspb_noc: interconnect-nspb-noc { - compatible = "qcom,sa8775p-nspb-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - pcie_anoc: interconnect-pcie-anoc { - compatible = "qcom,sa8775p-pcie-anoc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - - system_noc: interconnect-system-noc { - compatible = "qcom,sa8775p-system-noc"; - #interconnect-cells = <2>; - qcom,bcm-voters = <&apps_bcm_voter>; - }; - /* Will be updated by the bootloader. */ memory@80000000 { device_type = "memory"; @@ -2689,6 +2617,408 @@ rng: rng@10d2000 { reg = <0 0x010d2000 0 0x1000>; }; + config_noc: interconnect@14c0000 { + compatible = "qcom,sa8775p-config-noc"; + reg = <0x0 0x014c0000 0x0 0x13080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + + system_noc: interconnect@1680000 { + compatible = "qcom,sa8775p-system-noc"; + reg = <0x0 0x01680000 0x0 0x15080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + + aggre1_noc: interconnect@16c0000 { + compatible = "qcom,sa8775p-aggre1-noc"; + reg = <0x0 0x016c0000 0x0 0x18080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + clocks = <&gcc GCC_AGGRE_UFS_PHY_AXI_CLK>, + <&gcc GCC_AGGRE_NOC_QUPV3_AXI_CLK>, + <&gcc GCC_AGGRE_USB2_PRIM_AXI_CLK>, + <&gcc GCC_AGGRE_USB3_PRIM_AXI_CLK>, + <&gcc GCC_AGGRE_USB3_SEC_AXI_CLK>; + }; + + aggre2_noc: interconnect@1700000 { + compatible = "qcom,sa8775p-aggre2-noc"; + reg = <0x0 0x01700000 0x0 0x1b080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + clocks = <&gcc GCC_AGGRE_UFS_CARD_AXI_CLK>, + <&rpmhcc RPMH_IPA_CLK>; + }; + + pcie_anoc: interconnect@1760000 { + compatible = "qcom,sa8775p-pcie-anoc"; + reg = <0x0 0x01760000 0x0 0xc080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + + gpdsp_anoc: interconnect@1780000 { + compatible = "qcom,sa8775p-gpdsp-anoc"; + reg = <0x0 0x01780000 0x0 0xe080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + + mmss_noc: interconnect@17a0000 { + compatible = "qcom,sa8775p-mmss-noc"; + reg = <0x0 0x017a0000 0x0 0x40000>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + + pcie0: pcie@1c00000 { + compatible = "qcom,pcie-sa8775p"; + reg = <0x0 0x01c00000 0x0 0x3000>, + <0x0 0x40000000 0x0 0xf20>, + <0x0 0x40000f20 0x0 0xa8>, + <0x0 0x40001000 0x0 0x4000>, + <0x0 0x40100000 0x0 0x100000>, + <0x0 0x01c03000 0x0 0x1000>; + reg-names = "parf", "dbi", "elbi", "atu", "config", "mhi"; + device_type = "pci"; + + #address-cells = <3>; + #size-cells = <2>; + ranges = <0x01000000 0x0 0x00000000 0x0 0x40200000 0x0 0x100000>, + <0x02000000 0x0 0x40300000 0x0 0x40300000 0x0 0x1fd00000>; + bus-range = <0x00 0xff>; + + dma-coherent; + + linux,pci-domain = <0>; + num-lanes = <2>; + + interrupts = , + , + , + , + , + , + , + , + ; + interrupt-names = "msi0", + "msi1", + "msi2", + "msi3", + "msi4", + "msi5", + "msi6", + "msi7", + "global"; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0x7>; + interrupt-map = <0 0 0 1 &intc GIC_SPI 434 IRQ_TYPE_LEVEL_HIGH>, + <0 0 0 2 &intc GIC_SPI 435 IRQ_TYPE_LEVEL_HIGH>, + <0 0 0 3 &intc GIC_SPI 438 IRQ_TYPE_LEVEL_HIGH>, + <0 0 0 4 &intc GIC_SPI 439 IRQ_TYPE_LEVEL_HIGH>; + + clocks = <&gcc GCC_PCIE_0_AUX_CLK>, + <&gcc GCC_PCIE_0_CFG_AHB_CLK>, + <&gcc GCC_PCIE_0_MSTR_AXI_CLK>, + <&gcc GCC_PCIE_0_SLV_AXI_CLK>, + <&gcc GCC_PCIE_0_SLV_Q2A_AXI_CLK>; + + clock-names = "aux", + "cfg", + "bus_master", + "bus_slave", + "slave_q2a"; + + assigned-clocks = <&gcc GCC_PCIE_0_AUX_CLK>; + assigned-clock-rates = <19200000>; + + interconnects = <&pcie_anoc MASTER_PCIE_0 0 &mc_virt SLAVE_EBI1 0>, + <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_0 0>; + interconnect-names = "pcie-mem", "cpu-pcie"; + + iommu-map = <0x0 &pcie_smmu 0x0000 0x0 0x1>, + <0x100 &pcie_smmu 0x0001 0x0 0x1>; + + resets = <&gcc GCC_PCIE_0_BCR>, + <&gcc GCC_PCIE_0_LINK_DOWN_BCR>; + reset-names = "pci", + "link_down"; + + power-domains = <&gcc PCIE_0_GDSC>; + + phys = <&pcie0_phy>; + phy-names = "pciephy"; + + eq-presets-8gts = /bits/ 16 <0x5555 0x5555>; + eq-presets-16gts = /bits/ 8 <0x55 0x55>; + + status = "disabled"; + + pcieport0: pcie@0 { + device_type = "pci"; + reg = <0x0 0x0 0x0 0x0 0x0>; + bus-range = <0x01 0xff>; + + #address-cells = <3>; + #size-cells = <2>; + ranges; + }; + }; + + pcie0_ep: pcie-ep@1c00000 { + compatible = "qcom,sa8775p-pcie-ep"; + reg = <0x0 0x01c00000 0x0 0x3000>, + <0x0 0x40000000 0x0 0xf20>, + <0x0 0x40000f20 0x0 0xa8>, + <0x0 0x40001000 0x0 0x4000>, + <0x0 0x40200000 0x0 0x1fe00000>, + <0x0 0x01c03000 0x0 0x1000>, + <0x0 0x40005000 0x0 0x2000>; + reg-names = "parf", "dbi", "elbi", "atu", "addr_space", + "mmio", "dma"; + + clocks = <&gcc GCC_PCIE_0_AUX_CLK>, + <&gcc GCC_PCIE_0_CFG_AHB_CLK>, + <&gcc GCC_PCIE_0_MSTR_AXI_CLK>, + <&gcc GCC_PCIE_0_SLV_AXI_CLK>, + <&gcc GCC_PCIE_0_SLV_Q2A_AXI_CLK>; + + clock-names = "aux", + "cfg", + "bus_master", + "bus_slave", + "slave_q2a"; + + interrupts = , + , + ; + + interrupt-names = "global", "doorbell", "dma"; + + interconnects = <&pcie_anoc MASTER_PCIE_0 0 &mc_virt SLAVE_EBI1 0>, + <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_0 0>; + interconnect-names = "pcie-mem", "cpu-pcie"; + + dma-coherent; + iommus = <&pcie_smmu 0x0000 0x7f>; + resets = <&gcc GCC_PCIE_0_BCR>; + reset-names = "core"; + power-domains = <&gcc PCIE_0_GDSC>; + phys = <&pcie0_phy>; + phy-names = "pciephy"; + num-lanes = <2>; + linux,pci-domain = <0>; + + status = "disabled"; + }; + + pcie0_phy: phy@1c04000 { + compatible = "qcom,sa8775p-qmp-gen4x2-pcie-phy"; + reg = <0x0 0x1c04000 0x0 0x2000>; + + clocks = <&gcc GCC_PCIE_0_PHY_AUX_CLK>, + <&gcc GCC_PCIE_0_CFG_AHB_CLK>, + <&gcc GCC_PCIE_CLKREF_EN>, + <&gcc GCC_PCIE_0_PHY_RCHNG_CLK>, + <&gcc GCC_PCIE_0_PIPE_CLK>, + <&gcc GCC_PCIE_0_PIPEDIV2_CLK>; + clock-names = "aux", + "cfg_ahb", + "ref", + "rchng", + "pipe", + "pipediv2"; + + assigned-clocks = <&gcc GCC_PCIE_0_PHY_RCHNG_CLK>; + assigned-clock-rates = <100000000>; + + resets = <&gcc GCC_PCIE_0_PHY_BCR>; + reset-names = "phy"; + + #clock-cells = <0>; + clock-output-names = "pcie_0_pipe_clk"; + + #phy-cells = <0>; + + status = "disabled"; + }; + + pcie1: pcie@1c10000 { + compatible = "qcom,pcie-sa8775p"; + reg = <0x0 0x01c10000 0x0 0x3000>, + <0x0 0x60000000 0x0 0xf20>, + <0x0 0x60000f20 0x0 0xa8>, + <0x0 0x60001000 0x0 0x4000>, + <0x0 0x60100000 0x0 0x100000>, + <0x0 0x01c13000 0x0 0x1000>; + reg-names = "parf", "dbi", "elbi", "atu", "config", "mhi"; + device_type = "pci"; + + #address-cells = <3>; + #size-cells = <2>; + ranges = <0x01000000 0x0 0x00000000 0x0 0x60200000 0x0 0x100000>, + <0x02000000 0x0 0x60300000 0x0 0x60300000 0x0 0x1fd00000>; + bus-range = <0x00 0xff>; + + dma-coherent; + + linux,pci-domain = <1>; + num-lanes = <4>; + + interrupts = , + , + , + , + , + , + , + , + ; + interrupt-names = "msi0", + "msi1", + "msi2", + "msi3", + "msi4", + "msi5", + "msi6", + "msi7", + "global"; + #interrupt-cells = <1>; + interrupt-map-mask = <0 0 0 0x7>; + interrupt-map = <0 0 0 1 &intc GIC_SPI 148 IRQ_TYPE_LEVEL_HIGH>, + <0 0 0 2 &intc GIC_SPI 149 IRQ_TYPE_LEVEL_HIGH>, + <0 0 0 3 &intc GIC_SPI 150 IRQ_TYPE_LEVEL_HIGH>, + <0 0 0 4 &intc GIC_SPI 151 IRQ_TYPE_LEVEL_HIGH>; + + clocks = <&gcc GCC_PCIE_1_AUX_CLK>, + <&gcc GCC_PCIE_1_CFG_AHB_CLK>, + <&gcc GCC_PCIE_1_MSTR_AXI_CLK>, + <&gcc GCC_PCIE_1_SLV_AXI_CLK>, + <&gcc GCC_PCIE_1_SLV_Q2A_AXI_CLK>; + + clock-names = "aux", + "cfg", + "bus_master", + "bus_slave", + "slave_q2a"; + + assigned-clocks = <&gcc GCC_PCIE_1_AUX_CLK>; + assigned-clock-rates = <19200000>; + + interconnects = <&pcie_anoc MASTER_PCIE_1 0 &mc_virt SLAVE_EBI1 0>, + <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_1 0>; + interconnect-names = "pcie-mem", "cpu-pcie"; + + iommu-map = <0x0 &pcie_smmu 0x0080 0x0 0x1>, + <0x100 &pcie_smmu 0x0081 0x0 0x1>; + + resets = <&gcc GCC_PCIE_1_BCR>, + <&gcc GCC_PCIE_1_LINK_DOWN_BCR>; + reset-names = "pci", + "link_down"; + + power-domains = <&gcc PCIE_1_GDSC>; + + phys = <&pcie1_phy>; + phy-names = "pciephy"; + + eq-presets-8gts = /bits/ 16 <0x5555 0x5555 0x5555 0x5555>; + eq-presets-16gts = /bits/ 8 <0x55 0x55 0x55 0x55>; + + status = "disabled"; + + pcie@0 { + device_type = "pci"; + reg = <0x0 0x0 0x0 0x0 0x0>; + bus-range = <0x01 0xff>; + + #address-cells = <3>; + #size-cells = <2>; + ranges; + }; + }; + + pcie1_ep: pcie-ep@1c10000 { + compatible = "qcom,sa8775p-pcie-ep"; + reg = <0x0 0x01c10000 0x0 0x3000>, + <0x0 0x60000000 0x0 0xf20>, + <0x0 0x60000f20 0x0 0xa8>, + <0x0 0x60001000 0x0 0x4000>, + <0x0 0x60200000 0x0 0x1fe00000>, + <0x0 0x01c13000 0x0 0x1000>, + <0x0 0x60005000 0x0 0x2000>; + reg-names = "parf", "dbi", "elbi", "atu", "addr_space", + "mmio", "dma"; + + clocks = <&gcc GCC_PCIE_1_AUX_CLK>, + <&gcc GCC_PCIE_1_CFG_AHB_CLK>, + <&gcc GCC_PCIE_1_MSTR_AXI_CLK>, + <&gcc GCC_PCIE_1_SLV_AXI_CLK>, + <&gcc GCC_PCIE_1_SLV_Q2A_AXI_CLK>; + + clock-names = "aux", + "cfg", + "bus_master", + "bus_slave", + "slave_q2a"; + + interrupts = , + , + ; + + interrupt-names = "global", "doorbell", "dma"; + + interconnects = <&pcie_anoc MASTER_PCIE_1 0 &mc_virt SLAVE_EBI1 0>, + <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_1 0>; + interconnect-names = "pcie-mem", "cpu-pcie"; + + dma-coherent; + iommus = <&pcie_smmu 0x80 0x7f>; + resets = <&gcc GCC_PCIE_1_BCR>; + reset-names = "core"; + power-domains = <&gcc PCIE_1_GDSC>; + phys = <&pcie1_phy>; + phy-names = "pciephy"; + num-lanes = <4>; + linux,pci-domain = <1>; + + status = "disabled"; + }; + + pcie1_phy: phy@1c14000 { + compatible = "qcom,sa8775p-qmp-gen4x4-pcie-phy"; + reg = <0x0 0x1c14000 0x0 0x4000>; + + clocks = <&gcc GCC_PCIE_1_PHY_AUX_CLK>, + <&gcc GCC_PCIE_1_CFG_AHB_CLK>, + <&gcc GCC_PCIE_CLKREF_EN>, + <&gcc GCC_PCIE_1_PHY_RCHNG_CLK>, + <&gcc GCC_PCIE_1_PIPE_CLK>, + <&gcc GCC_PCIE_1_PIPEDIV2_CLK>; + clock-names = "aux", + "cfg_ahb", + "ref", + "rchng", + "pipe", + "pipediv2"; + + assigned-clocks = <&gcc GCC_PCIE_1_PHY_RCHNG_CLK>; + assigned-clock-rates = <100000000>; + + resets = <&gcc GCC_PCIE_1_PHY_BCR>; + reset-names = "phy"; + + #clock-cells = <0>; + clock-output-names = "pcie_1_pipe_clk"; + + #phy-cells = <0>; + + status = "disabled"; + }; + ufs_mem_hc: ufshc@1d84000 { compatible = "qcom,sa8775p-ufshc", "qcom,ufshc", "jedec,ufs-2.0"; reg = <0x0 0x01d84000 0x0 0x3000>; @@ -2773,6 +3103,25 @@ cryptobam: dma-controller@1dc4000 { <&apps_smmu 0x481 0x00>; }; + crypto: crypto@1dfa000 { + compatible = "qcom,sa8775p-qce", "qcom,sm8150-qce", "qcom,qce"; + reg = <0x0 0x01dfa000 0x0 0x6000>; + dmas = <&cryptobam 4>, <&cryptobam 5>; + dma-names = "rx", "tx"; + iommus = <&apps_smmu 0x480 0x0>, + <&apps_smmu 0x481 0x0>; + interconnects = <&aggre2_noc MASTER_CRYPTO_CORE0 QCOM_ICC_TAG_ALWAYS + &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>; + interconnect-names = "memory"; + }; + + lpass_ag_noc: interconnect@3c40000 { + compatible = "qcom,sa8775p-lpass-ag-noc"; + reg = <0x0 0x03c40000 0x0 0x17200>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + ctcu@4001000 { compatible = "qcom,sa8775p-ctcu"; reg = <0x0 0x04001000 0x0 0x1000>; @@ -3905,6 +4254,32 @@ usb_0_hsphy: phy@88e4000 { status = "disabled"; }; + usb_1_hsphy: phy@88e6000 { + compatible = "qcom,sa8775p-usb-hs-phy", + "qcom,usb-snps-hs-5nm-phy"; + reg = <0 0x088e6000 0 0x120>; + clocks = <&gcc GCC_USB_CLKREF_EN>; + clock-names = "ref"; + resets = <&gcc GCC_USB2_PHY_SEC_BCR>; + + #phy-cells = <0>; + + status = "disabled"; + }; + + usb_2_hsphy: phy@88e7000 { + compatible = "qcom,sa8775p-usb-hs-phy", + "qcom,usb-snps-hs-5nm-phy"; + reg = <0 0x088e7000 0 0x120>; + clocks = <&gcc GCC_USB_CLKREF_EN>; + clock-names = "ref"; + resets = <&gcc GCC_USB3_PHY_TERT_BCR>; + + #phy-cells = <0>; + + status = "disabled"; + }; + usb_0_qmpphy: phy@88e8000 { compatible = "qcom,sa8775p-qmp-usb3-uni-phy"; reg = <0 0x088e8000 0 0x2000>; @@ -3929,6 +4304,50 @@ usb_0_qmpphy: phy@88e8000 { status = "disabled"; }; + usb_1_qmpphy: phy@88ea000 { + compatible = "qcom,sa8775p-qmp-usb3-uni-phy"; + reg = <0 0x088ea000 0 0x2000>; + + clocks = <&gcc GCC_USB3_SEC_PHY_AUX_CLK>, + <&gcc GCC_USB_CLKREF_EN>, + <&gcc GCC_USB3_SEC_PHY_COM_AUX_CLK>, + <&gcc GCC_USB3_SEC_PHY_PIPE_CLK>; + clock-names = "aux", "ref", "com_aux", "pipe"; + + resets = <&gcc GCC_USB3_PHY_SEC_BCR>, + <&gcc GCC_USB3PHY_PHY_SEC_BCR>; + reset-names = "phy", "phy_phy"; + + power-domains = <&gcc USB30_SEC_GDSC>; + + #clock-cells = <0>; + clock-output-names = "usb3_sec_phy_pipe_clk_src"; + + #phy-cells = <0>; + + status = "disabled"; + }; + + refgen: regulator@891c000 { + compatible = "qcom,sa8775p-refgen-regulator", + "qcom,sm8250-refgen-regulator"; + reg = <0x0 0x0891c000 0x0 0x84>; + }; + + dc_noc: interconnect@90e0000 { + compatible = "qcom,sa8775p-dc-noc"; + reg = <0x0 0x090e0000 0x0 0x5080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + + gem_noc: interconnect@9100000 { + compatible = "qcom,sa8775p-gem-noc"; + reg = <0x0 0x09100000 0x0 0xf6080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + usb_0: usb@a600000 { compatible = "qcom,sa8775p-dwc3", "qcom,snps-dwc3"; reg = <0 0x0a600000 0 0xfc100>; @@ -3977,43 +4396,6 @@ usb_0: usb@a600000 { status = "disabled"; }; - usb_1_hsphy: phy@88e6000 { - compatible = "qcom,sa8775p-usb-hs-phy", - "qcom,usb-snps-hs-5nm-phy"; - reg = <0 0x088e6000 0 0x120>; - clocks = <&gcc GCC_USB_CLKREF_EN>; - clock-names = "ref"; - resets = <&gcc GCC_USB2_PHY_SEC_BCR>; - - #phy-cells = <0>; - - status = "disabled"; - }; - - usb_1_qmpphy: phy@88ea000 { - compatible = "qcom,sa8775p-qmp-usb3-uni-phy"; - reg = <0 0x088ea000 0 0x2000>; - - clocks = <&gcc GCC_USB3_SEC_PHY_AUX_CLK>, - <&gcc GCC_USB_CLKREF_EN>, - <&gcc GCC_USB3_SEC_PHY_COM_AUX_CLK>, - <&gcc GCC_USB3_SEC_PHY_PIPE_CLK>; - clock-names = "aux", "ref", "com_aux", "pipe"; - - resets = <&gcc GCC_USB3_PHY_SEC_BCR>, - <&gcc GCC_USB3PHY_PHY_SEC_BCR>; - reset-names = "phy", "phy_phy"; - - power-domains = <&gcc USB30_SEC_GDSC>; - - #clock-cells = <0>; - clock-output-names = "usb3_sec_phy_pipe_clk_src"; - - #phy-cells = <0>; - - status = "disabled"; - }; - usb_1: usb@a800000 { compatible = "qcom,sa8775p-dwc3", "qcom,snps-dwc3"; reg = <0 0x0a800000 0 0xfc100>; @@ -4062,19 +4444,6 @@ usb_1: usb@a800000 { status = "disabled"; }; - usb_2_hsphy: phy@88e7000 { - compatible = "qcom,sa8775p-usb-hs-phy", - "qcom,usb-snps-hs-5nm-phy"; - reg = <0 0x088e7000 0 0x120>; - clocks = <&gcc GCC_USB_CLKREF_EN>; - clock-names = "ref"; - resets = <&gcc GCC_USB3_PHY_TERT_BCR>; - - #phy-cells = <0>; - - status = "disabled"; - }; - usb_2: usb@a400000 { compatible = "qcom,sa8775p-dwc3", "qcom,snps-dwc3"; reg = <0 0x0a400000 0 0xfc100>; @@ -4904,6 +5273,8 @@ mdss0_dsi0: dsi@ae94000 { operating-points-v2 = <&mdss_dsi_opp_table>; power-domains = <&rpmhpd SA8775P_MMCX>; + refgen-supply = <&refgen>; + #address-cells = <1>; #size-cells = <0>; @@ -4986,6 +5357,8 @@ mdss0_dsi1: dsi@ae96000 { operating-points-v2 = <&mdss_dsi_opp_table>; power-domains = <&rpmhpd SA8775P_MMCX>; + refgen-supply = <&refgen>; + #address-cells = <1>; #size-cells = <0>; @@ -6884,6 +7257,13 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, status = "disabled"; }; + nspa_noc: interconnect@260c0000 { + compatible = "qcom,sa8775p-nspa-noc"; + reg = <0x0 0x260c0000 0x0 0x16080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + remoteproc_cdsp0: remoteproc@26300000 { compatible = "qcom,sa8775p-cdsp0-pas"; reg = <0x0 0x26300000 0x0 0x10000>; @@ -7016,6 +7396,13 @@ compute-cb@11 { }; }; + nspb_noc: interconnect@2a0c0000 { + compatible = "qcom,sa8775p-nspb-noc"; + reg = <0x0 0x2a0c0000 0x0 0x16080>; + #interconnect-cells = <2>; + qcom,bcm-voters = <&apps_bcm_voter>; + }; + remoteproc_cdsp1: remoteproc@2a300000 { compatible = "qcom,sa8775p-cdsp1-pas"; reg = <0x0 0x2A300000 0x0 0x10000>; @@ -8264,350 +8651,4 @@ arch_timer: timer { , ; }; - - pcie0: pcie@1c00000 { - compatible = "qcom,pcie-sa8775p"; - reg = <0x0 0x01c00000 0x0 0x3000>, - <0x0 0x40000000 0x0 0xf20>, - <0x0 0x40000f20 0x0 0xa8>, - <0x0 0x40001000 0x0 0x4000>, - <0x0 0x40100000 0x0 0x100000>, - <0x0 0x01c03000 0x0 0x1000>; - reg-names = "parf", "dbi", "elbi", "atu", "config", "mhi"; - device_type = "pci"; - - #address-cells = <3>; - #size-cells = <2>; - ranges = <0x01000000 0x0 0x00000000 0x0 0x40200000 0x0 0x100000>, - <0x02000000 0x0 0x40300000 0x0 0x40300000 0x0 0x1fd00000>; - bus-range = <0x00 0xff>; - - dma-coherent; - - linux,pci-domain = <0>; - num-lanes = <2>; - - interrupts = , - , - , - , - , - , - , - , - ; - interrupt-names = "msi0", - "msi1", - "msi2", - "msi3", - "msi4", - "msi5", - "msi6", - "msi7", - "global"; - #interrupt-cells = <1>; - interrupt-map-mask = <0 0 0 0x7>; - interrupt-map = <0 0 0 1 &intc GIC_SPI 434 IRQ_TYPE_LEVEL_HIGH>, - <0 0 0 2 &intc GIC_SPI 435 IRQ_TYPE_LEVEL_HIGH>, - <0 0 0 3 &intc GIC_SPI 438 IRQ_TYPE_LEVEL_HIGH>, - <0 0 0 4 &intc GIC_SPI 439 IRQ_TYPE_LEVEL_HIGH>; - - clocks = <&gcc GCC_PCIE_0_AUX_CLK>, - <&gcc GCC_PCIE_0_CFG_AHB_CLK>, - <&gcc GCC_PCIE_0_MSTR_AXI_CLK>, - <&gcc GCC_PCIE_0_SLV_AXI_CLK>, - <&gcc GCC_PCIE_0_SLV_Q2A_AXI_CLK>; - - clock-names = "aux", - "cfg", - "bus_master", - "bus_slave", - "slave_q2a"; - - assigned-clocks = <&gcc GCC_PCIE_0_AUX_CLK>; - assigned-clock-rates = <19200000>; - - interconnects = <&pcie_anoc MASTER_PCIE_0 0 &mc_virt SLAVE_EBI1 0>, - <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_0 0>; - interconnect-names = "pcie-mem", "cpu-pcie"; - - iommu-map = <0x0 &pcie_smmu 0x0000 0x1>, - <0x100 &pcie_smmu 0x0001 0x1>; - - resets = <&gcc GCC_PCIE_0_BCR>, - <&gcc GCC_PCIE_0_LINK_DOWN_BCR>; - reset-names = "pci", - "link_down"; - - power-domains = <&gcc PCIE_0_GDSC>; - - phys = <&pcie0_phy>; - phy-names = "pciephy"; - - eq-presets-8gts = /bits/ 16 <0x5555 0x5555>; - eq-presets-16gts = /bits/ 8 <0x55 0x55>; - - status = "disabled"; - - pcieport0: pcie@0 { - device_type = "pci"; - reg = <0x0 0x0 0x0 0x0 0x0>; - bus-range = <0x01 0xff>; - - #address-cells = <3>; - #size-cells = <2>; - ranges; - }; - }; - - pcie0_ep: pcie-ep@1c00000 { - compatible = "qcom,sa8775p-pcie-ep"; - reg = <0x0 0x01c00000 0x0 0x3000>, - <0x0 0x40000000 0x0 0xf20>, - <0x0 0x40000f20 0x0 0xa8>, - <0x0 0x40001000 0x0 0x4000>, - <0x0 0x40200000 0x0 0x1fe00000>, - <0x0 0x01c03000 0x0 0x1000>, - <0x0 0x40005000 0x0 0x2000>; - reg-names = "parf", "dbi", "elbi", "atu", "addr_space", - "mmio", "dma"; - - clocks = <&gcc GCC_PCIE_0_AUX_CLK>, - <&gcc GCC_PCIE_0_CFG_AHB_CLK>, - <&gcc GCC_PCIE_0_MSTR_AXI_CLK>, - <&gcc GCC_PCIE_0_SLV_AXI_CLK>, - <&gcc GCC_PCIE_0_SLV_Q2A_AXI_CLK>; - - clock-names = "aux", - "cfg", - "bus_master", - "bus_slave", - "slave_q2a"; - - interrupts = , - , - ; - - interrupt-names = "global", "doorbell", "dma"; - - interconnects = <&pcie_anoc MASTER_PCIE_0 0 &mc_virt SLAVE_EBI1 0>, - <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_0 0>; - interconnect-names = "pcie-mem", "cpu-pcie"; - - dma-coherent; - iommus = <&pcie_smmu 0x0000 0x7f>; - resets = <&gcc GCC_PCIE_0_BCR>; - reset-names = "core"; - power-domains = <&gcc PCIE_0_GDSC>; - phys = <&pcie0_phy>; - phy-names = "pciephy"; - num-lanes = <2>; - linux,pci-domain = <0>; - - status = "disabled"; - }; - - pcie0_phy: phy@1c04000 { - compatible = "qcom,sa8775p-qmp-gen4x2-pcie-phy"; - reg = <0x0 0x1c04000 0x0 0x2000>; - - clocks = <&gcc GCC_PCIE_0_PHY_AUX_CLK>, - <&gcc GCC_PCIE_0_CFG_AHB_CLK>, - <&gcc GCC_PCIE_CLKREF_EN>, - <&gcc GCC_PCIE_0_PHY_RCHNG_CLK>, - <&gcc GCC_PCIE_0_PIPE_CLK>, - <&gcc GCC_PCIE_0_PIPEDIV2_CLK>; - clock-names = "aux", - "cfg_ahb", - "ref", - "rchng", - "pipe", - "pipediv2"; - - assigned-clocks = <&gcc GCC_PCIE_0_PHY_RCHNG_CLK>; - assigned-clock-rates = <100000000>; - - resets = <&gcc GCC_PCIE_0_PHY_BCR>; - reset-names = "phy"; - - #clock-cells = <0>; - clock-output-names = "pcie_0_pipe_clk"; - - #phy-cells = <0>; - - status = "disabled"; - }; - - pcie1: pcie@1c10000 { - compatible = "qcom,pcie-sa8775p"; - reg = <0x0 0x01c10000 0x0 0x3000>, - <0x0 0x60000000 0x0 0xf20>, - <0x0 0x60000f20 0x0 0xa8>, - <0x0 0x60001000 0x0 0x4000>, - <0x0 0x60100000 0x0 0x100000>, - <0x0 0x01c13000 0x0 0x1000>; - reg-names = "parf", "dbi", "elbi", "atu", "config", "mhi"; - device_type = "pci"; - - #address-cells = <3>; - #size-cells = <2>; - ranges = <0x01000000 0x0 0x00000000 0x0 0x60200000 0x0 0x100000>, - <0x02000000 0x0 0x60300000 0x0 0x60300000 0x0 0x1fd00000>; - bus-range = <0x00 0xff>; - - dma-coherent; - - linux,pci-domain = <1>; - num-lanes = <4>; - - interrupts = , - , - , - , - , - , - , - , - ; - interrupt-names = "msi0", - "msi1", - "msi2", - "msi3", - "msi4", - "msi5", - "msi6", - "msi7", - "global"; - #interrupt-cells = <1>; - interrupt-map-mask = <0 0 0 0x7>; - interrupt-map = <0 0 0 1 &intc GIC_SPI 148 IRQ_TYPE_LEVEL_HIGH>, - <0 0 0 2 &intc GIC_SPI 149 IRQ_TYPE_LEVEL_HIGH>, - <0 0 0 3 &intc GIC_SPI 150 IRQ_TYPE_LEVEL_HIGH>, - <0 0 0 4 &intc GIC_SPI 151 IRQ_TYPE_LEVEL_HIGH>; - - clocks = <&gcc GCC_PCIE_1_AUX_CLK>, - <&gcc GCC_PCIE_1_CFG_AHB_CLK>, - <&gcc GCC_PCIE_1_MSTR_AXI_CLK>, - <&gcc GCC_PCIE_1_SLV_AXI_CLK>, - <&gcc GCC_PCIE_1_SLV_Q2A_AXI_CLK>; - - clock-names = "aux", - "cfg", - "bus_master", - "bus_slave", - "slave_q2a"; - - assigned-clocks = <&gcc GCC_PCIE_1_AUX_CLK>; - assigned-clock-rates = <19200000>; - - interconnects = <&pcie_anoc MASTER_PCIE_1 0 &mc_virt SLAVE_EBI1 0>, - <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_1 0>; - interconnect-names = "pcie-mem", "cpu-pcie"; - - iommu-map = <0x0 &pcie_smmu 0x0080 0x1>, - <0x100 &pcie_smmu 0x0081 0x1>; - - resets = <&gcc GCC_PCIE_1_BCR>, - <&gcc GCC_PCIE_1_LINK_DOWN_BCR>; - reset-names = "pci", - "link_down"; - - power-domains = <&gcc PCIE_1_GDSC>; - - phys = <&pcie1_phy>; - phy-names = "pciephy"; - - eq-presets-8gts = /bits/ 16 <0x5555 0x5555 0x5555 0x5555>; - eq-presets-16gts = /bits/ 8 <0x55 0x55 0x55 0x55>; - - status = "disabled"; - - pcie@0 { - device_type = "pci"; - reg = <0x0 0x0 0x0 0x0 0x0>; - bus-range = <0x01 0xff>; - - #address-cells = <3>; - #size-cells = <2>; - ranges; - }; - }; - - pcie1_ep: pcie-ep@1c10000 { - compatible = "qcom,sa8775p-pcie-ep"; - reg = <0x0 0x01c10000 0x0 0x3000>, - <0x0 0x60000000 0x0 0xf20>, - <0x0 0x60000f20 0x0 0xa8>, - <0x0 0x60001000 0x0 0x4000>, - <0x0 0x60200000 0x0 0x1fe00000>, - <0x0 0x01c13000 0x0 0x1000>, - <0x0 0x60005000 0x0 0x2000>; - reg-names = "parf", "dbi", "elbi", "atu", "addr_space", - "mmio", "dma"; - - clocks = <&gcc GCC_PCIE_1_AUX_CLK>, - <&gcc GCC_PCIE_1_CFG_AHB_CLK>, - <&gcc GCC_PCIE_1_MSTR_AXI_CLK>, - <&gcc GCC_PCIE_1_SLV_AXI_CLK>, - <&gcc GCC_PCIE_1_SLV_Q2A_AXI_CLK>; - - clock-names = "aux", - "cfg", - "bus_master", - "bus_slave", - "slave_q2a"; - - interrupts = , - , - ; - - interrupt-names = "global", "doorbell", "dma"; - - interconnects = <&pcie_anoc MASTER_PCIE_1 0 &mc_virt SLAVE_EBI1 0>, - <&gem_noc MASTER_APPSS_PROC 0 &config_noc SLAVE_PCIE_1 0>; - interconnect-names = "pcie-mem", "cpu-pcie"; - - dma-coherent; - iommus = <&pcie_smmu 0x80 0x7f>; - resets = <&gcc GCC_PCIE_1_BCR>; - reset-names = "core"; - power-domains = <&gcc PCIE_1_GDSC>; - phys = <&pcie1_phy>; - phy-names = "pciephy"; - num-lanes = <4>; - linux,pci-domain = <1>; - - status = "disabled"; - }; - - pcie1_phy: phy@1c14000 { - compatible = "qcom,sa8775p-qmp-gen4x4-pcie-phy"; - reg = <0x0 0x1c14000 0x0 0x4000>; - - clocks = <&gcc GCC_PCIE_1_PHY_AUX_CLK>, - <&gcc GCC_PCIE_1_CFG_AHB_CLK>, - <&gcc GCC_PCIE_CLKREF_EN>, - <&gcc GCC_PCIE_1_PHY_RCHNG_CLK>, - <&gcc GCC_PCIE_1_PIPE_CLK>, - <&gcc GCC_PCIE_1_PIPEDIV2_CLK>; - clock-names = "aux", - "cfg_ahb", - "ref", - "rchng", - "pipe", - "pipediv2"; - - assigned-clocks = <&gcc GCC_PCIE_1_PHY_RCHNG_CLK>; - assigned-clock-rates = <100000000>; - - resets = <&gcc GCC_PCIE_1_PHY_BCR>; - reset-names = "phy"; - - #clock-cells = <0>; - clock-output-names = "pcie_1_pipe_clk"; - - #phy-cells = <0>; - - status = "disabled"; - }; }; diff --git a/arch/arm64/boot/dts/qcom/msm8976-longcheer-l9360.dts b/arch/arm64/boot/dts/qcom/msm8976-longcheer-l9360.dts index 18832a3b9a1c..57f549f06f73 100644 --- a/arch/arm64/boot/dts/qcom/msm8976-longcheer-l9360.dts +++ b/arch/arm64/boot/dts/qcom/msm8976-longcheer-l9360.dts @@ -455,14 +455,14 @@ sdc2_cd_sleep: sdc2-cd-sleep-state { bias-disable; }; - ts_int_default: ts-int-state { + ts_int_default: ts-int-default-state { pins = "gpio65"; function = "gpio"; drive-strength = <2>; bias-pull-down; }; - ts_int_sleep: ts-int-state { + ts_int_sleep: ts-int-sleep-state { pins = "gpio65"; function = "gpio"; drive-strength = <2>; diff --git a/arch/arm64/boot/dts/qcom/msm8996-xiaomi-gemini.dts b/arch/arm64/boot/dts/qcom/msm8996-xiaomi-gemini.dts index bd3f39e1b98f..9a9c67450120 100644 --- a/arch/arm64/boot/dts/qcom/msm8996-xiaomi-gemini.dts +++ b/arch/arm64/boot/dts/qcom/msm8996-xiaomi-gemini.dts @@ -39,7 +39,7 @@ &blsp2_i2c3 { haptics: drv2604@5a { compatible = "ti,drv2604"; reg = <0x5a>; - enable-gpio = <&tlmm 93 0x00>; + enable-gpios = <&tlmm 93 GPIO_ACTIVE_HIGH>; mode = ; library-sel = ; pinctrl-names = "default","sleep"; diff --git a/arch/arm64/boot/dts/qcom/msm8998-sony-xperia-yoshino.dtsi b/arch/arm64/boot/dts/qcom/msm8998-sony-xperia-yoshino.dtsi index 3650f2501886..04d4741cdb5f 100644 --- a/arch/arm64/boot/dts/qcom/msm8998-sony-xperia-yoshino.dtsi +++ b/arch/arm64/boot/dts/qcom/msm8998-sony-xperia-yoshino.dtsi @@ -229,10 +229,6 @@ rmi4-f11@11 { }; }; -&blsp1_i2c5_sleep { - bias-disable; -}; - &blsp1_uart3 { status = "okay"; diff --git a/arch/arm64/boot/dts/qcom/msm8998-xiaomi-sagit.dts b/arch/arm64/boot/dts/qcom/msm8998-xiaomi-sagit.dts index 0cac06f25a77..2ceeec827f8e 100644 --- a/arch/arm64/boot/dts/qcom/msm8998-xiaomi-sagit.dts +++ b/arch/arm64/boot/dts/qcom/msm8998-xiaomi-sagit.dts @@ -217,11 +217,6 @@ rmi4-f1a@1a { }; }; -&blsp1_i2c5_sleep { - /delete-property/ bias-pull-up; - bias-disable; -}; - &blsp1_uart3 { status = "okay"; diff --git a/arch/arm64/boot/dts/qcom/msm8998.dtsi b/arch/arm64/boot/dts/qcom/msm8998.dtsi index 5c75fba16ce2..515aa844a3e3 100644 --- a/arch/arm64/boot/dts/qcom/msm8998.dtsi +++ b/arch/arm64/boot/dts/qcom/msm8998.dtsi @@ -1183,7 +1183,7 @@ blsp1_i2c1_sleep: blsp1-i2c1-sleep-state-state { pins = "gpio2", "gpio3"; function = "blsp_i2c1"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp1_i2c2_default: blsp1-i2c2-default-state { @@ -1197,7 +1197,7 @@ blsp1_i2c2_sleep: blsp1-i2c2-sleep-state-state { pins = "gpio32", "gpio33"; function = "blsp_i2c2"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp1_i2c3_default: blsp1-i2c3-default-state { @@ -1211,7 +1211,7 @@ blsp1_i2c3_sleep: blsp1-i2c3-sleep-state { pins = "gpio47", "gpio48"; function = "blsp_i2c3"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp1_i2c4_default: blsp1-i2c4-default-state { @@ -1225,7 +1225,7 @@ blsp1_i2c4_sleep: blsp1-i2c4-sleep-state { pins = "gpio10", "gpio11"; function = "blsp_i2c4"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp1_i2c5_default: blsp1-i2c5-default-state { @@ -1239,7 +1239,7 @@ blsp1_i2c5_sleep: blsp1-i2c5-sleep-state { pins = "gpio87", "gpio88"; function = "blsp_i2c5"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp1_i2c6_default: blsp1-i2c6-default-state { @@ -1253,7 +1253,7 @@ blsp1_i2c6_sleep: blsp1-i2c6-sleep-state { pins = "gpio43", "gpio44"; function = "blsp_i2c6"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp1_spi_b_default: blsp1-spi-b-default-state { @@ -1318,7 +1318,7 @@ blsp2_i2c1_sleep: blsp2-i2c1-sleep-state { pins = "gpio55", "gpio56"; function = "blsp_i2c7"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp2_i2c2_default: blsp2-i2c2-default-state { @@ -1332,7 +1332,7 @@ blsp2_i2c2_sleep: blsp2-i2c2-sleep-state { pins = "gpio6", "gpio7"; function = "blsp_i2c8"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp2_i2c3_default: blsp2-i2c3-default-state { @@ -1346,7 +1346,7 @@ blsp2_i2c3_sleep: blsp2-i2c3-sleep-state { pins = "gpio51", "gpio52"; function = "blsp_i2c9"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp2_i2c4_default: blsp2-i2c4-default-state { @@ -1360,7 +1360,7 @@ blsp2_i2c4_sleep: blsp2-i2c4-sleep-state { pins = "gpio67", "gpio68"; function = "blsp_i2c10"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp2_i2c5_default: blsp2-i2c5-default-state { @@ -1374,7 +1374,7 @@ blsp2_i2c5_sleep: blsp2-i2c5-sleep-state { pins = "gpio60", "gpio61"; function = "blsp_i2c11"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp2_i2c6_default: blsp2-i2c6-default-state { @@ -1388,7 +1388,7 @@ blsp2_i2c6_sleep: blsp2-i2c6-sleep-state { pins = "gpio83", "gpio84"; function = "blsp_i2c12"; drive-strength = <2>; - bias-pull-up; + bias-disable; }; blsp2_spi1_default: blsp2-spi1-default-state { diff --git a/arch/arm64/boot/dts/qcom/qcm2290.dtsi b/arch/arm64/boot/dts/qcom/qcm2290.dtsi index e0e400fdd249..5a817824f924 100644 --- a/arch/arm64/boot/dts/qcom/qcm2290.dtsi +++ b/arch/arm64/boot/dts/qcom/qcm2290.dtsi @@ -2006,6 +2006,7 @@ dispcc: clock-controller@5f00000 { "gcc_disp_gpll0_div_clk_src", "dsi0_phy_pll_out_byteclk", "dsi0_phy_pll_out_dsiclk"; + power-domains = <&rpmpd QCM2290_VDDCX>; #power-domain-cells = <1>; #clock-cells = <1>; #reset-cells = <1>; diff --git a/arch/arm64/boot/dts/qcom/qcs404.dtsi b/arch/arm64/boot/dts/qcom/qcs404.dtsi index 4328c1dda898..736accfc34f9 100644 --- a/arch/arm64/boot/dts/qcom/qcs404.dtsi +++ b/arch/arm64/boot/dts/qcom/qcs404.dtsi @@ -677,11 +677,11 @@ usb3: usb@7678800 { assigned-clock-rates = <19200000>, <200000000>; interrupts = , - , - ; + , + ; interrupt-names = "pwr_event", - "hs_phy_irq", - "qusb2_phy"; + "qusb2_phy", + "hs_phy_irq"; status = "disabled"; @@ -716,11 +716,11 @@ usb2: usb@79b8800 { assigned-clock-rates = <19200000>, <133333333>; interrupts = , - , - ; + , + ; interrupt-names = "pwr_event", - "hs_phy_irq", - "qusb2_phy"; + "qusb2_phy", + "hs_phy_irq"; status = "disabled"; diff --git a/arch/arm64/boot/dts/qcom/qcs6490-rb3gen2.dts b/arch/arm64/boot/dts/qcom/qcs6490-rb3gen2.dts index 18cea8812001..6be927c18482 100644 --- a/arch/arm64/boot/dts/qcom/qcs6490-rb3gen2.dts +++ b/arch/arm64/boot/dts/qcom/qcs6490-rb3gen2.dts @@ -823,15 +823,15 @@ &pcie1 { pinctrl-0 = <&pcie1_reset_n>, <&pcie1_wake_n>, <&pcie1_clkreq_n>; pinctrl-names = "default"; - iommu-map = <0x0 &apps_smmu 0x1c80 0x1>, - <0x100 &apps_smmu 0x1c81 0x1>, - <0x208 &apps_smmu 0x1c84 0x1>, - <0x210 &apps_smmu 0x1c85 0x1>, - <0x218 &apps_smmu 0x1c86 0x1>, - <0x300 &apps_smmu 0x1c87 0x1>, - <0x400 &apps_smmu 0x1c88 0x1>, - <0x500 &apps_smmu 0x1c89 0x1>, - <0x501 &apps_smmu 0x1c90 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c80 0x0 0x1>, + <0x100 &apps_smmu 0x1c81 0x0 0x1>, + <0x208 &apps_smmu 0x1c84 0x0 0x1>, + <0x210 &apps_smmu 0x1c85 0x0 0x1>, + <0x218 &apps_smmu 0x1c86 0x0 0x1>, + <0x300 &apps_smmu 0x1c87 0x0 0x1>, + <0x400 &apps_smmu 0x1c88 0x0 0x1>, + <0x500 &apps_smmu 0x1c89 0x0 0x1>, + <0x501 &apps_smmu 0x1c90 0x0 0x1>; status = "okay"; }; diff --git a/arch/arm64/boot/dts/qcom/qcs8550-aim300.dtsi b/arch/arm64/boot/dts/qcom/qcs8550-aim300.dtsi index e6ac529e6b72..3ead078246f4 100644 --- a/arch/arm64/boot/dts/qcom/qcs8550-aim300.dtsi +++ b/arch/arm64/boot/dts/qcom/qcs8550-aim300.dtsi @@ -396,8 +396,8 @@ &usb_1_hsphy { }; &usb_dp_qmpphy { - vdda-phy-supply = <&vreg_l3e_1p2>; - vdda-pll-supply = <&vreg_l3f_0p88>; + vdda-phy-supply = <&vreg_l3f_0p88>; + vdda-pll-supply = <&vreg_l3e_1p2>; }; &xo_board { diff --git a/arch/arm64/boot/dts/qcom/sar2130p-qar2130p.dts b/arch/arm64/boot/dts/qcom/sar2130p-qar2130p.dts index 74778a5b19ba..80428c0b3b4e 100644 --- a/arch/arm64/boot/dts/qcom/sar2130p-qar2130p.dts +++ b/arch/arm64/boot/dts/qcom/sar2130p-qar2130p.dts @@ -551,8 +551,8 @@ &usb_1_hsphy { }; &usb_dp_qmpphy { - vdda-phy-supply = <&vreg_l3a_1p2>; - vdda-pll-supply = <&vreg_l1a_0p91>; + vdda-phy-supply = <&vreg_l1a_0p91>; + vdda-pll-supply = <&vreg_l3a_1p2>; status = "okay"; }; diff --git a/arch/arm64/boot/dts/qcom/sar2130p.dtsi b/arch/arm64/boot/dts/qcom/sar2130p.dtsi index d65ad0df6865..d7a80d832f08 100644 --- a/arch/arm64/boot/dts/qcom/sar2130p.dtsi +++ b/arch/arm64/boot/dts/qcom/sar2130p.dtsi @@ -1329,8 +1329,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, &config_noc SLAVE_PCIE_0 QCOM_ICC_TAG_ALWAYS>; interconnect-names = "pcie-mem", "cpu-pcie"; - iommu-map = <0x0 &apps_smmu 0x1c00 0x1>, - <0x100 &apps_smmu 0x1c01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c00 0x0 0x1>, + <0x100 &apps_smmu 0x1c01 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -1455,8 +1455,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, &config_noc SLAVE_PCIE_1 QCOM_ICC_TAG_ALWAYS>; interconnect-names = "pcie-mem", "cpu-pcie"; - iommu-map = <0x0 &apps_smmu 0x1e00 0x1>, - <0x100 &apps_smmu 0x1e01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1e00 0x0 0x1>, + <0x100 &apps_smmu 0x1e01 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>, <&gcc GCC_PCIE_1_LINK_DOWN_BCR>; diff --git a/arch/arm64/boot/dts/qcom/sc7180-trogdor.dtsi b/arch/arm64/boot/dts/qcom/sc7180-trogdor.dtsi index 74ab321d3333..d60fad977a28 100644 --- a/arch/arm64/boot/dts/qcom/sc7180-trogdor.dtsi +++ b/arch/arm64/boot/dts/qcom/sc7180-trogdor.dtsi @@ -98,6 +98,11 @@ chosen { stdout-path = "serial0:115200n8"; }; + firmware { + #address-cells = <2>; + #size-cells = <2>; + }; + /* FIXED REGULATORS - parents above children */ /* This is the top level supply and variable voltage */ diff --git a/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi b/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi index 5c5e4f1dd221..58ea0532c0fb 100644 --- a/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi +++ b/arch/arm64/boot/dts/qcom/sc7280-herobrine.dtsi @@ -25,6 +25,11 @@ chosen { stdout-path = "serial0:115200n8"; }; + firmware { + #address-cells = <2>; + #size-cells = <2>; + }; + /* * FIXED REGULATORS * diff --git a/arch/arm64/boot/dts/qcom/sc7280.dtsi b/arch/arm64/boot/dts/qcom/sc7280.dtsi index 0f566dd75c96..7d53e06e97b0 100644 --- a/arch/arm64/boot/dts/qcom/sc7280.dtsi +++ b/arch/arm64/boot/dts/qcom/sc7280.dtsi @@ -2273,8 +2273,8 @@ pcie0: pcie@1c00000 { "aggre0", "aggre1"; - iommu-map = <0x0 &apps_smmu 0x1c00 0x1>, - <0x100 &apps_smmu 0x1c01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c00 0x0 0x1>, + <0x100 &apps_smmu 0x1c01 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -2419,8 +2419,8 @@ pcie1: pcie@1c08000 { dma-coherent; - iommu-map = <0x0 &apps_smmu 0x1c80 0x1>, - <0x100 &apps_smmu 0x1c81 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c80 0x0 0x1>, + <0x100 &apps_smmu 0x1c81 0x0 0x1>; status = "disabled"; diff --git a/arch/arm64/boot/dts/qcom/sc8180x-lenovo-flex-5g.dts b/arch/arm64/boot/dts/qcom/sc8180x-lenovo-flex-5g.dts index 08d0784d0cbb..8f8f66a2297e 100644 --- a/arch/arm64/boot/dts/qcom/sc8180x-lenovo-flex-5g.dts +++ b/arch/arm64/boot/dts/qcom/sc8180x-lenovo-flex-5g.dts @@ -26,6 +26,7 @@ backlight: backlight { compatible = "pwm-backlight"; pwms = <&pmc8180c_lpg 4 1000000>; enable-gpios = <&pmc8180c_gpios 8 GPIO_ACTIVE_HIGH>; + power-supply = <&vled_bl_pw>; pinctrl-0 = <&bl_pwm_default>; pinctrl-names = "default"; @@ -162,14 +163,39 @@ cdsp_mem: cdsp-region@98900000 { }; }; - vph_pwr: vph-pwr-regulator { + vled_bl_pw: regulator-vled-bl-pw { compatible = "regulator-fixed"; - regulator-name = "vph_pwr"; - regulator-min-microvolt = <3700000>; - regulator-max-microvolt = <3700000>; + + regulator-name = "VLED_BL_PW"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + + gpio = <&pmc8180_2_gpios 1 GPIO_ACTIVE_HIGH>; + enable-active-high; + + pinctrl-0 = <&bl_pwr_en>; + pinctrl-names = "default"; + + regulator-boot-on; }; - vreg_s4a_1p8: pm8150-s4-regulator { + vreg_lcm_3v3: regulator-edp-3p3 { + compatible = "regulator-fixed"; + + regulator-name = "VREG_LCM_3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + + gpio = <&tlmm 130 GPIO_ACTIVE_HIGH>; + enable-active-high; + + pinctrl-0 = <&lcm_3v3_en>; + pinctrl-names = "default"; + + regulator-boot-on; + }; + + vreg_s4a_1p8: regulator-pm8150-s4 { compatible = "regulator-fixed"; regulator-name = "vreg_s4a_1p8"; @@ -182,6 +208,13 @@ vreg_s4a_1p8: pm8150-s4-regulator { vin-supply = <&vph_pwr>; }; + vph_pwr: regulator-vph-pwr { + compatible = "regulator-fixed"; + regulator-name = "vph_pwr"; + regulator-min-microvolt = <3700000>; + regulator-max-microvolt = <3700000>; + }; + usbprim-sbu-mux { compatible = "pericom,pi3usb102", "gpio-sbu-mux"; @@ -444,6 +477,7 @@ &mdss_edp { aux-bus { panel { compatible = "edp-panel"; + power-supply = <&vreg_lcm_3v3>; no-hpd; backlight = <&backlight>; @@ -478,6 +512,13 @@ &pcie3_phy { status = "okay"; }; +&pmc8180_2_gpios { + bl_pwr_en: bl-pwr-en-state { + pins = "gpio1"; + function = "normal"; + }; +}; + &pmc8180_pwrkey { status = "okay"; }; @@ -771,6 +812,12 @@ ts_int_default: ts-int-default-state { drive-strength = <2>; }; + lcm_3v3_en: lcm-3v3-en-state { + pins = "gpio130"; + function = "gpio"; + bias-disable; + }; + usbprim_sbu_default: usbprim-sbu-state { oe-n-pins { pins = "gpio152"; diff --git a/arch/arm64/boot/dts/qcom/sc8180x-primus.dts b/arch/arm64/boot/dts/qcom/sc8180x-primus.dts index 93de9fe918eb..76679e311ecf 100644 --- a/arch/arm64/boot/dts/qcom/sc8180x-primus.dts +++ b/arch/arm64/boot/dts/qcom/sc8180x-primus.dts @@ -27,9 +27,10 @@ backlight: backlight { compatible = "pwm-backlight"; pwms = <&pmc8180c_lpg 4 1000000>; enable-gpios = <&pmc8180c_gpios 8 GPIO_ACTIVE_HIGH>; + power-supply = <&vled_bl_pw>; - pinctrl-names = "default"; pinctrl-0 = <&bl_pwm_default>; + pinctrl-names = "default"; }; chosen { @@ -165,7 +166,39 @@ reserved-region@9a500000 { }; }; - vreg_nvme_0p9: nvme-0p9-regulator { + vled_bl_pw: regulator-vled-bl-pw { + compatible = "regulator-fixed"; + + regulator-name = "VLED_BL_PW"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + + gpio = <&pmc8180_2_gpios 1 GPIO_ACTIVE_HIGH>; + enable-active-high; + + pinctrl-0 = <&bl_pwr_en>; + pinctrl-names = "default"; + + regulator-boot-on; + }; + + vreg_lcm_3v3: regulator-edp-3p3 { + compatible = "regulator-fixed"; + + regulator-name = "VREG_LCM_3V3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + + gpio = <&tlmm 130 GPIO_ACTIVE_HIGH>; + enable-active-high; + + pinctrl-0 = <&lcm_3v3_en>; + pinctrl-names = "default"; + + regulator-boot-on; + }; + + vreg_nvme_0p9: regulator-nvme-0p9 { compatible = "regulator-fixed"; regulator-name = "vreg_nvme_0p9"; @@ -175,7 +208,7 @@ vreg_nvme_0p9: nvme-0p9-regulator { regulator-always-on; }; - vreg_nvme_3p3: nvme-3p3-regulator { + vreg_nvme_3p3: regulator-nvme-3p3 { compatible = "regulator-fixed"; regulator-name = "vreg_nvme_3p3"; @@ -188,7 +221,7 @@ vreg_nvme_3p3: nvme-3p3-regulator { regulator-always-on; }; - vdd_kb_tp_3v3: vdd-kb-tp-3v3-regulator { + vdd_kb_tp_3v3: regulator-vdd-kb-tp-3v3 { compatible = "regulator-fixed"; regulator-name = "vdd_kb_tp_3v3"; regulator-min-microvolt = <3300000>; @@ -203,7 +236,7 @@ vdd_kb_tp_3v3: vdd-kb-tp-3v3-regulator { pinctrl-0 = <&kb_tp_3v3_en_active_state>; }; - vph_pwr: vph-pwr-regulator { + vph_pwr: regulator-vph-pwr { compatible = "regulator-fixed"; regulator-name = "vph_pwr"; regulator-min-microvolt = <3700000>; @@ -539,6 +572,7 @@ &mdss_edp { aux-bus { panel { compatible = "edp-panel"; + power-supply = <&vreg_lcm_3v3>; backlight = <&backlight>; @@ -768,6 +802,12 @@ &wifi { }; /* PINCTRL */ +&pmc8180_2_gpios { + bl_pwr_en: bl-pwr-en-state { + pins = "gpio1"; + function = "normal"; + }; +}; &pmc8180c_gpios { bl_pwm_default: bl-pwm-default-state { @@ -949,4 +989,10 @@ rx-pins { bias-pull-up; }; }; + + lcm_3v3_en: lcm-3v3-en-state { + pins = "gpio130"; + function = "gpio"; + bias-disable; + }; }; diff --git a/arch/arm64/boot/dts/qcom/sc8180x.dtsi b/arch/arm64/boot/dts/qcom/sc8180x.dtsi index c6d96022c666..faf12c6e679c 100644 --- a/arch/arm64/boot/dts/qcom/sc8180x.dtsi +++ b/arch/arm64/boot/dts/qcom/sc8180x.dtsi @@ -1762,8 +1762,8 @@ pcie0: pcie@1c00000 { assigned-clocks = <&gcc GCC_PCIE_0_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1d80 0x1>, - <0x100 &apps_smmu 0x1d81 0x1>; + iommu-map = <0x0 &apps_smmu 0x1d80 0x0 0x1>, + <0x100 &apps_smmu 0x1d81 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -1881,8 +1881,8 @@ pcie3: pcie@1c08000 { assigned-clocks = <&gcc GCC_PCIE_3_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1e00 0x1>, - <0x100 &apps_smmu 0x1e01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1e00 0x0 0x1>, + <0x100 &apps_smmu 0x1e01 0x0 0x1>; resets = <&gcc GCC_PCIE_3_BCR>; reset-names = "pci"; @@ -2001,8 +2001,8 @@ pcie1: pcie@1c10000 { assigned-clocks = <&gcc GCC_PCIE_1_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1c80 0x1>, - <0x100 &apps_smmu 0x1c81 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c80 0x0 0x1>, + <0x100 &apps_smmu 0x1c81 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>; reset-names = "pci"; @@ -2121,8 +2121,8 @@ pcie2: pcie@1c18000 { assigned-clocks = <&gcc GCC_PCIE_2_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1d00 0x1>, - <0x100 &apps_smmu 0x1d01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1d00 0x0 0x1>, + <0x100 &apps_smmu 0x1d01 0x0 0x1>; resets = <&gcc GCC_PCIE_2_BCR>; reset-names = "pci"; diff --git a/arch/arm64/boot/dts/qcom/sc8280xp-crd.dts b/arch/arm64/boot/dts/qcom/sc8280xp-crd.dts index 490e970c54a2..5b2692bb1564 100644 --- a/arch/arm64/boot/dts/qcom/sc8280xp-crd.dts +++ b/arch/arm64/boot/dts/qcom/sc8280xp-crd.dts @@ -182,7 +182,7 @@ vreg_misc_3p3: regulator-misc-3p3 { regulator-min-microvolt = <3300000>; regulator-max-microvolt = <3300000>; - gpio = <&pmc8280_1_gpios 2 GPIO_ACTIVE_HIGH>; + gpio = <&pmc8280_1_gpios 1 GPIO_ACTIVE_HIGH>; enable-active-high; pinctrl-names = "default"; @@ -921,7 +921,7 @@ kypd_vol_up_n: kypd-vol-up-n-state { }; misc_3p3_reg_en: misc-3p3-reg-en-state { - pins = "gpio2"; + pins = "gpio1"; function = "normal"; }; }; diff --git a/arch/arm64/boot/dts/qcom/sc8280xp-lenovo-thinkpad-x13s.dts b/arch/arm64/boot/dts/qcom/sc8280xp-lenovo-thinkpad-x13s.dts index 637430719e6d..a7559eb41101 100644 --- a/arch/arm64/boot/dts/qcom/sc8280xp-lenovo-thinkpad-x13s.dts +++ b/arch/arm64/boot/dts/qcom/sc8280xp-lenovo-thinkpad-x13s.dts @@ -1561,7 +1561,7 @@ cam_rgb_default: cam-rgb-default-state { mclk-pins { pins = "gpio17"; function = "cam_mclk"; - drive-strength = <16>; + drive-strength = <6>; bias-disable; }; diff --git a/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-arcata.dts b/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-arcata.dts index aeed3ef152eb..cdc2b0dcca27 100644 --- a/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-arcata.dts +++ b/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-arcata.dts @@ -248,7 +248,7 @@ map1 { usb0-sbu-mux { compatible = "pericom,pi3usb102", "gpio-sbu-mux"; - enable-gpios = <&tlmm 101 GPIO_ACTIVE_LOW>; + enable-gpios = <&tlmm 100 GPIO_ACTIVE_LOW>; select-gpios = <&tlmm 164 GPIO_ACTIVE_HIGH>; pinctrl-0 = <&usb0_sbu_default>; @@ -1002,7 +1002,7 @@ tx-pins { usb0_sbu_default: usb0-sbu-state { oe-n-pins { - pins = "gpio101"; + pins = "gpio100"; function = "gpio"; bias-disable; drive-strength = <16>; diff --git a/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-blackrock.dts b/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-blackrock.dts index a40dccd70dfd..454defd25956 100644 --- a/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-blackrock.dts +++ b/arch/arm64/boot/dts/qcom/sc8280xp-microsoft-blackrock.dts @@ -769,20 +769,11 @@ &pmk8280_pon_resin { }; &pmk8280_rtc { - nvmem-cells = <&rtc_offset>; - nvmem-cell-names = "offset"; + qcom,uefi-rtc-info; status = "okay"; }; -&pmk8280_sdam_6 { - status = "okay"; - - rtc_offset: rtc-offset@bc { - reg = <0xbc 0x4>; - }; -}; - &pmk8280_vadc { channel@144 { reg = ; @@ -1009,8 +1000,8 @@ &usb_0_hsphy { }; &usb_0_qmpphy { - vdda-phy-supply = <&vreg_l4d>; - vdda-pll-supply = <&vreg_l9d>; + vdda-phy-supply = <&vreg_l9d>; + vdda-pll-supply = <&vreg_l4d>; orientation-switch; @@ -1046,8 +1037,8 @@ &usb_1_hsphy { }; &usb_1_qmpphy { - vdda-phy-supply = <&vreg_l3b>; - vdda-pll-supply = <&vreg_l4b>; + vdda-phy-supply = <&vreg_l4b>; + vdda-pll-supply = <&vreg_l3b>; orientation-switch; diff --git a/arch/arm64/boot/dts/qcom/sdm632-motorola-ocean.dts b/arch/arm64/boot/dts/qcom/sdm632-motorola-ocean.dts index 2f55db0c8ce3..9ea3e5e76bf9 100644 --- a/arch/arm64/boot/dts/qcom/sdm632-motorola-ocean.dts +++ b/arch/arm64/boot/dts/qcom/sdm632-motorola-ocean.dts @@ -130,7 +130,7 @@ led-controller@36 { led: led@1 { reg = <1>; - default-trigger = "backlight"; + linux,default-trigger = "backlight"; function = LED_FUNCTION_BACKLIGHT; led-sources = <0 1 2>; }; diff --git a/arch/arm64/boot/dts/qcom/sdm845.dtsi b/arch/arm64/boot/dts/qcom/sdm845.dtsi index 13c9515260ef..c49ef738d094 100644 --- a/arch/arm64/boot/dts/qcom/sdm845.dtsi +++ b/arch/arm64/boot/dts/qcom/sdm845.dtsi @@ -2367,22 +2367,22 @@ pcie0: pcie@1c00000 { "slave_q2a", "tbu"; - iommu-map = <0x0 &apps_smmu 0x1c10 0x1>, - <0x100 &apps_smmu 0x1c11 0x1>, - <0x200 &apps_smmu 0x1c12 0x1>, - <0x300 &apps_smmu 0x1c13 0x1>, - <0x400 &apps_smmu 0x1c14 0x1>, - <0x500 &apps_smmu 0x1c15 0x1>, - <0x600 &apps_smmu 0x1c16 0x1>, - <0x700 &apps_smmu 0x1c17 0x1>, - <0x800 &apps_smmu 0x1c18 0x1>, - <0x900 &apps_smmu 0x1c19 0x1>, - <0xa00 &apps_smmu 0x1c1a 0x1>, - <0xb00 &apps_smmu 0x1c1b 0x1>, - <0xc00 &apps_smmu 0x1c1c 0x1>, - <0xd00 &apps_smmu 0x1c1d 0x1>, - <0xe00 &apps_smmu 0x1c1e 0x1>, - <0xf00 &apps_smmu 0x1c1f 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c10 0x0 0x1>, + <0x100 &apps_smmu 0x1c11 0x0 0x1>, + <0x200 &apps_smmu 0x1c12 0x0 0x1>, + <0x300 &apps_smmu 0x1c13 0x0 0x1>, + <0x400 &apps_smmu 0x1c14 0x0 0x1>, + <0x500 &apps_smmu 0x1c15 0x0 0x1>, + <0x600 &apps_smmu 0x1c16 0x0 0x1>, + <0x700 &apps_smmu 0x1c17 0x0 0x1>, + <0x800 &apps_smmu 0x1c18 0x0 0x1>, + <0x900 &apps_smmu 0x1c19 0x0 0x1>, + <0xa00 &apps_smmu 0x1c1a 0x0 0x1>, + <0xb00 &apps_smmu 0x1c1b 0x0 0x1>, + <0xc00 &apps_smmu 0x1c1c 0x0 0x1>, + <0xd00 &apps_smmu 0x1c1d 0x0 0x1>, + <0xe00 &apps_smmu 0x1c1e 0x0 0x1>, + <0xf00 &apps_smmu 0x1c1f 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -2497,22 +2497,22 @@ pcie1: pcie@1c08000 { assigned-clocks = <&gcc GCC_PCIE_1_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1c00 0x1>, - <0x100 &apps_smmu 0x1c01 0x1>, - <0x200 &apps_smmu 0x1c02 0x1>, - <0x300 &apps_smmu 0x1c03 0x1>, - <0x400 &apps_smmu 0x1c04 0x1>, - <0x500 &apps_smmu 0x1c05 0x1>, - <0x600 &apps_smmu 0x1c06 0x1>, - <0x700 &apps_smmu 0x1c07 0x1>, - <0x800 &apps_smmu 0x1c08 0x1>, - <0x900 &apps_smmu 0x1c09 0x1>, - <0xa00 &apps_smmu 0x1c0a 0x1>, - <0xb00 &apps_smmu 0x1c0b 0x1>, - <0xc00 &apps_smmu 0x1c0c 0x1>, - <0xd00 &apps_smmu 0x1c0d 0x1>, - <0xe00 &apps_smmu 0x1c0e 0x1>, - <0xf00 &apps_smmu 0x1c0f 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c00 0x0 0x1>, + <0x100 &apps_smmu 0x1c01 0x0 0x1>, + <0x200 &apps_smmu 0x1c02 0x0 0x1>, + <0x300 &apps_smmu 0x1c03 0x0 0x1>, + <0x400 &apps_smmu 0x1c04 0x0 0x1>, + <0x500 &apps_smmu 0x1c05 0x0 0x1>, + <0x600 &apps_smmu 0x1c06 0x0 0x1>, + <0x700 &apps_smmu 0x1c07 0x0 0x1>, + <0x800 &apps_smmu 0x1c08 0x0 0x1>, + <0x900 &apps_smmu 0x1c09 0x0 0x1>, + <0xa00 &apps_smmu 0x1c0a 0x0 0x1>, + <0xb00 &apps_smmu 0x1c0b 0x0 0x1>, + <0xc00 &apps_smmu 0x1c0c 0x0 0x1>, + <0xd00 &apps_smmu 0x1c0d 0x0 0x1>, + <0xe00 &apps_smmu 0x1c0e 0x0 0x1>, + <0xf00 &apps_smmu 0x1c0f 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>; reset-names = "pci"; diff --git a/arch/arm64/boot/dts/qcom/sm6150.dtsi b/arch/arm64/boot/dts/qcom/sm6150.dtsi index a066ad5ffde5..ff3ae5688f03 100644 --- a/arch/arm64/boot/dts/qcom/sm6150.dtsi +++ b/arch/arm64/boot/dts/qcom/sm6150.dtsi @@ -1175,8 +1175,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, &config_noc SLAVE_PCIE_0 QCOM_ICC_TAG_ACTIVE_ONLY>; interconnect-names = "pcie-mem", "cpu-pcie"; - iommu-map = <0x0 &apps_smmu 0x400 0x1>, - <0x100 &apps_smmu 0x401 0x1>; + iommu-map = <0x0 &apps_smmu 0x400 0x0 0x1>, + <0x100 &apps_smmu 0x401 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; diff --git a/arch/arm64/boot/dts/qcom/sm7225-fairphone-fp4.dts b/arch/arm64/boot/dts/qcom/sm7225-fairphone-fp4.dts index 8cbe068645ea..b6d2a5d0c1c8 100644 --- a/arch/arm64/boot/dts/qcom/sm7225-fairphone-fp4.dts +++ b/arch/arm64/boot/dts/qcom/sm7225-fairphone-fp4.dts @@ -48,7 +48,7 @@ chosen { stdout-path = "serial0:115200n8"; - framebuffer0: framebuffer@a000000 { + framebuffer0: framebuffer@a0000000 { compatible = "simple-framebuffer"; reg = <0 0xa0000000 0 (2340 * 1080 * 4)>; width = <1080>; @@ -1179,8 +1179,8 @@ &usb_1_hsphy { }; &usb_1_qmpphy { - vdda-phy-supply = <&vreg_l22a>; - vdda-pll-supply = <&vreg_l16a>; + vdda-phy-supply = <&vreg_l16a>; + vdda-pll-supply = <&vreg_l22a>; status = "okay"; }; diff --git a/arch/arm64/boot/dts/qcom/sm8150.dtsi b/arch/arm64/boot/dts/qcom/sm8150.dtsi index e1f480e3ed0b..35981d6026b3 100644 --- a/arch/arm64/boot/dts/qcom/sm8150.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8150.dtsi @@ -1884,8 +1884,8 @@ pcie0: pcie@1c00000 { "bus_slave", "slave_q2a"; - iommu-map = <0x0 &apps_smmu 0x1d80 0x1>, - <0x100 &apps_smmu 0x1d81 0x1>; + iommu-map = <0x0 &apps_smmu 0x1d80 0x0 0x1>, + <0x100 &apps_smmu 0x1d81 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -2002,8 +2002,8 @@ pcie1: pcie@1c08000 { assigned-clocks = <&gcc GCC_PCIE_1_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1e00 0x1>, - <0x100 &apps_smmu 0x1e01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1e00 0x0 0x1>, + <0x100 &apps_smmu 0x1e01 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>; reset-names = "pci"; diff --git a/arch/arm64/boot/dts/qcom/sm8250-xiaomi-elish-common.dtsi b/arch/arm64/boot/dts/qcom/sm8250-xiaomi-elish-common.dtsi index 465fd6e954a3..a2ced6611c40 100644 --- a/arch/arm64/boot/dts/qcom/sm8250-xiaomi-elish-common.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8250-xiaomi-elish-common.dtsi @@ -28,7 +28,7 @@ / { /* required for bootloader to select correct board */ qcom,msm-id = ; /* SM8250 v2.1 */ - qcom,board-id = <0x10008 0>; + qcom,board-id = <0x2f 0>; aliases { serial0 = &uart6; diff --git a/arch/arm64/boot/dts/qcom/sm8250.dtsi b/arch/arm64/boot/dts/qcom/sm8250.dtsi index 79ba2c0c96b7..26ae05ab663c 100644 --- a/arch/arm64/boot/dts/qcom/sm8250.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8250.dtsi @@ -2190,8 +2190,8 @@ pcie0: pcie@1c00000 { "tbu", "ddrss_sf_tbu"; - iommu-map = <0x0 &apps_smmu 0x1c00 0x1>, - <0x100 &apps_smmu 0x1c01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c00 0x0 0x1>, + <0x100 &apps_smmu 0x1c01 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -2317,8 +2317,8 @@ pcie1: pcie@1c08000 { assigned-clocks = <&gcc GCC_PCIE_1_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1c80 0x1>, - <0x100 &apps_smmu 0x1c81 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c80 0x0 0x1>, + <0x100 &apps_smmu 0x1c81 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>; reset-names = "pci"; @@ -2444,8 +2444,8 @@ pcie2: pcie@1c10000 { assigned-clocks = <&gcc GCC_PCIE_2_AUX_CLK>; assigned-clock-rates = <19200000>; - iommu-map = <0x0 &apps_smmu 0x1d00 0x1>, - <0x100 &apps_smmu 0x1d01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1d00 0x0 0x1>, + <0x100 &apps_smmu 0x1d01 0x0 0x1>; resets = <&gcc GCC_PCIE_2_BCR>; reset-names = "pci"; @@ -4321,8 +4321,12 @@ venus: video-codec@aa00000 { interrupts = ; power-domains = <&videocc MVS0C_GDSC>, <&videocc MVS0_GDSC>, - <&rpmhpd RPMHPD_MX>; - power-domain-names = "venus", "vcodec0", "mx"; + <&rpmhpd RPMHPD_MX>, + <&rpmhpd RPMHPD_MMCX>; + power-domain-names = "venus", + "vcodec0", + "mx", + "mmcx"; operating-points-v2 = <&venus_opp_table>; clocks = <&gcc GCC_VIDEO_AXI0_CLK>, @@ -4346,24 +4350,28 @@ venus: video-codec@aa00000 { venus_opp_table: opp-table { compatible = "operating-points-v2"; - opp-720000000 { - opp-hz = /bits/ 64 <720000000>; - required-opps = <&rpmhpd_opp_low_svs>; + opp-240000000 { + opp-hz = /bits/ 64 <240000000>; + required-opps = <&rpmhpd_opp_svs>, + <&rpmhpd_opp_low_svs>; }; - opp-1014000000 { - opp-hz = /bits/ 64 <1014000000>; - required-opps = <&rpmhpd_opp_svs>; + opp-338000000 { + opp-hz = /bits/ 64 <338000000>; + required-opps = <&rpmhpd_opp_svs>, + <&rpmhpd_opp_svs>; }; - opp-1098000000 { - opp-hz = /bits/ 64 <1098000000>; - required-opps = <&rpmhpd_opp_svs_l1>; + opp-366000000 { + opp-hz = /bits/ 64 <366000000>; + required-opps = <&rpmhpd_opp_svs_l1>, + <&rpmhpd_opp_svs_l1>; }; - opp-1332000000 { - opp-hz = /bits/ 64 <1332000000>; - required-opps = <&rpmhpd_opp_nom>; + opp-444000000 { + opp-hz = /bits/ 64 <444000000>; + required-opps = <&rpmhpd_opp_svs_l1>, + <&rpmhpd_opp_nom>; }; }; }; diff --git a/arch/arm64/boot/dts/qcom/sm8350.dtsi b/arch/arm64/boot/dts/qcom/sm8350.dtsi index af892b76322f..2bbc1f28be88 100644 --- a/arch/arm64/boot/dts/qcom/sm8350.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8350.dtsi @@ -1575,8 +1575,8 @@ pcie0: pcie@1c00000 { "aggre1", "aggre0"; - iommu-map = <0x0 &apps_smmu 0x1c00 0x1>, - <0x100 &apps_smmu 0x1c01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c00 0x0 0x1>, + <0x100 &apps_smmu 0x1c01 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -1684,8 +1684,8 @@ pcie1: pcie@1c08000 { "ddrss_sf_tbu", "aggre1"; - iommu-map = <0x0 &apps_smmu 0x1c80 0x1>, - <0x100 &apps_smmu 0x1c81 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c80 0x0 0x1>, + <0x100 &apps_smmu 0x1c81 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>; reset-names = "pci"; diff --git a/arch/arm64/boot/dts/qcom/sm8450.dtsi b/arch/arm64/boot/dts/qcom/sm8450.dtsi index 991f3cbde1c8..cabd90b5024a 100644 --- a/arch/arm64/boot/dts/qcom/sm8450.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8450.dtsi @@ -2023,8 +2023,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, "aggre0", "aggre1"; - iommu-map = <0x0 &apps_smmu 0x1c00 0x1>, - <0x100 &apps_smmu 0x1c01 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c00 0x0 0x1>, + <0x100 &apps_smmu 0x1c01 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -2185,8 +2185,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, "ddrss_sf_tbu", "aggre1"; - iommu-map = <0x0 &apps_smmu 0x1c80 0x1>, - <0x100 &apps_smmu 0x1c81 0x1>; + iommu-map = <0x0 &apps_smmu 0x1c80 0x0 0x1>, + <0x100 &apps_smmu 0x1c81 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>; reset-names = "pci"; diff --git a/arch/arm64/boot/dts/qcom/sm8550.dtsi b/arch/arm64/boot/dts/qcom/sm8550.dtsi index 479c94f5066f..e38d3ea19e7d 100644 --- a/arch/arm64/boot/dts/qcom/sm8550.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8550.dtsi @@ -2009,8 +2009,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, msi-map = <0x0 &gic_its 0x1400 0x1>, <0x100 &gic_its 0x1401 0x1>; - iommu-map = <0x0 &apps_smmu 0x1400 0x1>, - <0x100 &apps_smmu 0x1401 0x1>; + iommu-map = <0x0 &apps_smmu 0x1400 0x0 0x1>, + <0x100 &apps_smmu 0x1401 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -2175,8 +2175,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, msi-map = <0x0 &gic_its 0x1480 0x1>, <0x100 &gic_its 0x1481 0x1>; - iommu-map = <0x0 &apps_smmu 0x1480 0x1>, - <0x100 &apps_smmu 0x1481 0x1>; + iommu-map = <0x0 &apps_smmu 0x1480 0x0 0x1>, + <0x100 &apps_smmu 0x1481 0x0 0x1>; resets = <&gcc GCC_PCIE_1_BCR>, <&gcc GCC_PCIE_1_LINK_DOWN_BCR>; diff --git a/arch/arm64/boot/dts/qcom/sm8650.dtsi b/arch/arm64/boot/dts/qcom/sm8650.dtsi index 6c775ef20cb0..349ce22c6b68 100644 --- a/arch/arm64/boot/dts/qcom/sm8650.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8650.dtsi @@ -3626,8 +3626,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, operating-points-v2 = <&pcie0_opp_table>; - iommu-map = <0 &apps_smmu 0x1400 0x1>, - <0x100 &apps_smmu 0x1401 0x1>; + iommu-map = <0 &apps_smmu 0x1400 0x0 0x1>, + <0x100 &apps_smmu 0x1401 0x0 0x1>; interrupt-map = <0 0 0 1 &intc 0 0 GIC_SPI 149 IRQ_TYPE_LEVEL_HIGH 0>, <0 0 0 2 &intc 0 0 GIC_SPI 150 IRQ_TYPE_LEVEL_HIGH 0>, @@ -3806,8 +3806,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, operating-points-v2 = <&pcie1_opp_table>; - iommu-map = <0 &apps_smmu 0x1480 0x1>, - <0x100 &apps_smmu 0x1481 0x1>; + iommu-map = <0 &apps_smmu 0x1480 0x0 0x1>, + <0x100 &apps_smmu 0x1481 0x0 0x1>; interrupt-map = <0 0 0 1 &intc 0 0 GIC_SPI 434 IRQ_TYPE_LEVEL_HIGH 0>, <0 0 0 2 &intc 0 0 GIC_SPI 435 IRQ_TYPE_LEVEL_HIGH 0>, diff --git a/arch/arm64/boot/dts/qcom/sm8750.dtsi b/arch/arm64/boot/dts/qcom/sm8750.dtsi index 2760c4f7e6f3..c6f4a43ff133 100644 --- a/arch/arm64/boot/dts/qcom/sm8750.dtsi +++ b/arch/arm64/boot/dts/qcom/sm8750.dtsi @@ -3393,8 +3393,8 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, interconnect-names = "pcie-mem", "cpu-pcie"; - iommu-map = <0x0 &apps_smmu 0x1400 0x1>, - <0x100 &apps_smmu 0x1401 0x1>; + iommu-map = <0x0 &apps_smmu 0x1400 0x0 0x1>, + <0x100 &apps_smmu 0x1401 0x0 0x1>; resets = <&gcc GCC_PCIE_0_BCR>; reset-names = "pci"; @@ -3558,6 +3558,7 @@ &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, phy-names = "ufsphy"; #reset-cells = <1>; + qcom,ice = <&ice>; status = "disabled"; diff --git a/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi b/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi index 9d07353df52c..b44668c902b7 100644 --- a/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3399-gru-chromebook.dtsi @@ -503,8 +503,7 @@ &pci_rootport { mvl_wifi: wifi@0,0 { compatible = "pci1b4b,2b42"; reg = <0x0000 0x0 0x0 0x0 0x0>; - interrupt-parent = <&gpio0>; - interrupts = <8 IRQ_TYPE_LEVEL_LOW>; + interrupts-extended = <&gpio0 8 IRQ_TYPE_LEVEL_LOW>; pinctrl-names = "default"; pinctrl-0 = <&wlan_host_wake_l>; wakeup-source; diff --git a/arch/arm64/boot/dts/rockchip/rk3566-bigtreetech-cb2.dtsi b/arch/arm64/boot/dts/rockchip/rk3566-bigtreetech-cb2.dtsi index b6cf03a7ba66..04cf285e6c2a 100644 --- a/arch/arm64/boot/dts/rockchip/rk3566-bigtreetech-cb2.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3566-bigtreetech-cb2.dtsi @@ -569,7 +569,7 @@ tft_tp: touchscreen@48 { reg = <0x48>; status = "okay"; ti,x-plate-ohms = <660>; - ti,rt-thr = <3000>; + ti,max-rt = <3000>; ti,fuzzx = <32>; ti,fuzzy = <16>; }; diff --git a/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi b/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi index 7e74e04057cf..a031f42c558e 100644 --- a/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3588-base.dtsi @@ -1648,8 +1648,8 @@ hdmi0_out: port@1 { edp0: edp@fdec0000 { compatible = "rockchip,rk3588-edp"; reg = <0x0 0xfdec0000 0x0 0x1000>; - clocks = <&cru CLK_EDP0_24M>, <&cru PCLK_EDP0>; - clock-names = "dp", "pclk"; + clocks = <&cru CLK_EDP0_24M>, <&cru PCLK_EDP0>, <&cru HCLK_VO1>; + clock-names = "dp", "pclk", "hclk"; interrupts = ; phys = <&hdptxphy0>; phy-names = "dp"; diff --git a/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi b/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi index a2640014ee04..b251bb129cdb 100644 --- a/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi @@ -285,8 +285,8 @@ hdmi1_out: port@1 { edp1: edp@fded0000 { compatible = "rockchip,rk3588-edp"; reg = <0x0 0xfded0000 0x0 0x1000>; - clocks = <&cru CLK_EDP1_24M>, <&cru PCLK_EDP1>; - clock-names = "dp", "pclk"; + clocks = <&cru CLK_EDP1_24M>, <&cru PCLK_EDP1>, <&cru HCLK_VO1>; + clock-names = "dp", "pclk", "hclk"; interrupts = ; phys = <&hdptxphy1>; phy-names = "dp"; diff --git a/arch/arm64/boot/dts/ti/k3-am64-main.dtsi b/arch/arm64/boot/dts/ti/k3-am64-main.dtsi index d872cc671094..b7e5de3669ad 100644 --- a/arch/arm64/boot/dts/ti/k3-am64-main.dtsi +++ b/arch/arm64/boot/dts/ti/k3-am64-main.dtsi @@ -1401,7 +1401,7 @@ tx_pru0_1: txpru@c000 { icssg0_mdio: mdio@32400 { compatible = "ti,davinci_mdio"; reg = <0x32400 0x100>; - clocks = <&k3_clks 62 3>; + clocks = <&k3_clks 81 0>; clock-names = "fck"; #address-cells = <1>; #size-cells = <0>; diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7bee8a357ceb..feff0d13f052 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -397,9 +397,6 @@ struct kvm_arch { /* Masks for VNCR-backed and general EL2 sysregs */ struct kvm_sysreg_masks *sysreg_masks; - /* Count the number of VNCR_EL2 currently mapped */ - atomic_t vncr_map_count; - /* * For an untrusted host VM, 'pkvm.handle' is used to lookup * the associated pKVM instance in the hypervisor. diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 61d62bfd5a7b..ba0517d80fa5 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -256,7 +256,7 @@ static inline void task_set_sve_vl_onexec(struct task_struct *task, #define SCTLR_USER_MASK \ (SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | SCTLR_ELx_ENDA | SCTLR_ELx_ENDB | \ - SCTLR_EL1_TCF0_MASK) + SCTLR_EL1_TCF0_MASK | SCTLR_EL1_TCSO0_MASK) static inline void arch_thread_struct_whitelist(unsigned long *offset, unsigned long *size) diff --git a/arch/arm64/include/asm/ptdump.h b/arch/arm64/include/asm/ptdump.h index fded5358641f..f70e8743fca2 100644 --- a/arch/arm64/include/asm/ptdump.h +++ b/arch/arm64/include/asm/ptdump.h @@ -50,6 +50,8 @@ struct ptdump_pg_state { const struct addr_marker *marker; const struct mm_struct *mm; unsigned long start_address; + /* exclusive end, ULONG_MAX represents an end at 1 << 64 */ + unsigned long end_address; int level; ptdesc_t current_prot; bool check_wx; diff --git a/arch/arm64/include/asm/rsi_cmds.h b/arch/arm64/include/asm/rsi_cmds.h index 2c8763876dfb..c1fab41f671e 100644 --- a/arch/arm64/include/asm/rsi_cmds.h +++ b/arch/arm64/include/asm/rsi_cmds.h @@ -88,6 +88,14 @@ static inline long rsi_set_addr_range_state(phys_addr_t start, return res.a0; } +#define RSI_ATTEST_CHALLENGE_MIN_SIZE 32 +#define RSI_ATTEST_CHALLENGE_MAX_SIZE 64 + +struct rsi_attestation_token_init_args { + unsigned long fid; + u8 challenge[RSI_ATTEST_CHALLENGE_MAX_SIZE]; +}; + /** * rsi_attestation_token_init - Initialise the operation to retrieve an * attestation token. @@ -109,18 +117,21 @@ static inline long rsi_set_addr_range_state(phys_addr_t start, static inline long rsi_attestation_token_init(const u8 *challenge, unsigned long size) { - struct arm_smccc_1_2_regs regs = { 0 }; + union { + struct arm_smccc_1_2_regs regs; + struct rsi_attestation_token_init_args init; + } args = { 0 }; - /* The challenge must be at least 32bytes and at most 64bytes */ - if (!challenge || size < 32 || size > 64) + if (!challenge || size < RSI_ATTEST_CHALLENGE_MIN_SIZE || + size > RSI_ATTEST_CHALLENGE_MAX_SIZE) return -EINVAL; - regs.a0 = SMC_RSI_ATTESTATION_TOKEN_INIT; - memcpy(®s.a1, challenge, size); - arm_smccc_1_2_smc(®s, ®s); + args.init.fid = SMC_RSI_ATTESTATION_TOKEN_INIT; + memcpy(args.init.challenge, challenge, size); + arm_smccc_1_2_smc(&args.regs, &args.regs); - if (regs.a0 == RSI_SUCCESS) - return regs.a1; + if (args.regs.a0 == RSI_SUCCESS) + return args.regs.a1; return -EINVAL; } diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 9717568518ba..7bf117427777 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -348,8 +348,10 @@ int swsusp_arch_suspend(void) crash_prepare_suspend(); ret = swsusp_mte_save_tags(); - if (ret) + if (ret) { + local_daif_restore(flags); return ret; + } sleep_cpu = smp_processor_id(); ret = swsusp_save(); @@ -465,9 +467,21 @@ int __nocfi swsusp_arch_resume(void) if (el2_reset_needed()) __hyp_set_vectors(el2_vectors); + /* + * It is necessary to mask all DAIF exceptions here as: + * + * - The copy of swsusp_arch_suspend_exit() in the hibernation + * text cannot handle taking any exceptions. + * + * - The suspended kernel masked all DAIF exceptions in + * swsusp_arch_resume(), and expects to be re-entered in the + * same state : with all DAIF exceptions masked. + */ + local_daif_save(); hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1, resume_hdr.reenter_kernel, restore_pblist, resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page)); + unreachable(); return 0; } diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index 2acc2c3dd033..ceb047e7d49f 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -801,7 +801,7 @@ static void sve_init_header_from_task(struct user_sve_header *header, if (active) header->size = SVE_PT_SIZE(vq, header->flags); else - header->size = sizeof(header); + header->size = sizeof(*header); header->max_size = SVE_PT_SIZE(sve_vq_from_vl(header->max_vl), SVE_PT_REGS_SVE); } @@ -837,7 +837,7 @@ static int sve_get_common(struct task_struct *target, * from the other mode to userspace. */ if (header.size == sizeof(header)) - return 0; + return to.left; switch ((header.flags & SVE_PT_REGS_MASK)) { case SVE_PT_REGS_FPSIMD: diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 92b57fc87ee9..6af97292e369 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -1081,7 +1081,7 @@ static void ipi_teardown(int cpu) disable_percpu_irq(ipi_irq_base + i); } } else { - disable_irq(irq_desc_get_irq(get_ipi_desc(cpu, i))); + disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, i))); } } } diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 9984c492305a..552f39d913be 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -427,8 +427,7 @@ static bool kvm_hyp_handle_tlbi_el2(struct kvm_vcpu *vcpu, u64 *exit_code) * If we have to check for any VNCR mapping being invalidated, * go back to the slow path for further processing. */ - if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu) && - atomic_read(&vcpu->kvm->arch.vncr_map_count)) + if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) return false; __kvm_skip_instr(vcpu); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index b69df61f145a..869209e8bd1c 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -27,7 +27,7 @@ struct vncr_tlb { bool hpa_writable; /* -1 when not mapped on a CPU */ - int cpu; + atomic_t cpu; /* * true if the TLB is valid. Can only be changed with the @@ -48,7 +48,6 @@ void kvm_init_nested(struct kvm *kvm) { kvm->arch.nested_mmus = NULL; kvm->arch.nested_mmus_size = 0; - atomic_set(&kvm->arch.vncr_map_count, 0); } static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) @@ -771,18 +770,47 @@ void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) } } +/* + * Unmapping an L1 VNCR can happen concurrently without the mmu lock being + * effective (vcpu_put() vs TLBI handling). The atomic_xchg below ensures + * that only one CPU sets it to -1 while getting a valid CPU number back. + */ +static int unmap_l1_vncr(struct vncr_tlb *vt) +{ + int cpu = atomic_xchg_relaxed(&vt->cpu, -1); + + if (cpu != -1) + clear_fixmap(vncr_fixmap(cpu)); + + return cpu; +} + +static void this_cpu_reset_vncr_fixmap(struct kvm_vcpu *vcpu) +{ + if (!host_data_test_flag(L1_VNCR_MAPPED)) + return; + + BUG_ON(is_hyp_ctxt(vcpu)); + + /* + * Unconditionally unmap the local VNCR if we have lost the race + * against a concurrent TLBI. Otherwise we could end-up running + * another vcpu with VNCR still mapped if the TLBI thread is + * preempted between the exchange and the clear_fixmap(). + * + * Note that we do not care about the TLBI nuking the fixmap behind + * the back of an running vcpu. This will only generate a fault and + * possibly a retranslation. + */ + if (unmap_l1_vncr(vcpu->arch.vncr_tlb) == -1) + clear_fixmap(vncr_fixmap(smp_processor_id())); + host_data_clear_flag(L1_VNCR_MAPPED); +} + void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) { /* Unconditionally drop the VNCR mapping if we have one */ - if (host_data_test_flag(L1_VNCR_MAPPED)) { - BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id()); - BUG_ON(is_hyp_ctxt(vcpu)); - - clear_fixmap(vncr_fixmap(vcpu->arch.vncr_tlb->cpu)); - vcpu->arch.vncr_tlb->cpu = -1; - host_data_clear_flag(L1_VNCR_MAPPED); - atomic_dec(&vcpu->kvm->arch.vncr_map_count); - } + this_cpu_reset_vncr_fixmap(vcpu); /* * Keep a reference on the associated stage-2 MMU if the vCPU is @@ -864,8 +892,7 @@ u16 get_asid_by_regime(struct kvm_vcpu *vcpu, enum trans_regime regime) static void invalidate_vncr(struct vncr_tlb *vt) { vt->valid = false; - if (vt->cpu != -1) - clear_fixmap(vncr_fixmap(vt->cpu)); + unmap_l1_vncr(vt); } static bool vncr_tlb_intersects(struct vncr_tlb *vt, u64 addr, @@ -1244,7 +1271,8 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) * We also prepare the next walk wilst we're at it. */ scoped_guard(write_lock, &vcpu->kvm->mmu_lock) { - invalidate_vncr(vt); + this_cpu_reset_vncr_fixmap(vcpu); + vt->valid = false; vt->wi = (struct s1_walk_info) { .regime = TR_EL20, @@ -1314,7 +1342,7 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) vt->hpa = pfn << PAGE_SHIFT; vt->hpa_writable = writable; vt->valid = true; - vt->cpu = -1; + atomic_set(&vt->cpu, -1); kvm_make_request(KVM_REQ_MAP_L1_VNCR_EL2, vcpu); kvm_release_faultin_page(vcpu->kvm, page, false, vt->wr.pw && vt->hpa_writable); @@ -1445,8 +1473,6 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu) if (vt->wr.nG && get_asid_by_regime(vcpu, TR_EL20) != vt->wr.asid) return; - vt->cpu = smp_processor_id(); - if (vt->hpa_writable && vt->wr.pw && vt->wr.pr) prot = PAGE_KERNEL; else if (vt->wr.pr) @@ -1461,9 +1487,9 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu) * FIXME: WO doesn't work at all, need POE support in the kernel. */ if (pgprot_val(prot) != pgprot_val(PAGE_NONE)) { - __set_fixmap(vncr_fixmap(vt->cpu), vt->hpa, prot); + atomic_set(&vt->cpu, smp_processor_id()); + __set_fixmap(vncr_fixmap(atomic_read(&vt->cpu)), vt->hpa, prot); host_data_set_flag(L1_VNCR_MAPPED); - atomic_inc(&vcpu->kvm->arch.vncr_map_count); } } diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile index 633e5223d944..a74507503a4a 100644 --- a/arch/arm64/lib/Makefile +++ b/arch/arm64/lib/Makefile @@ -1,4 +1,8 @@ # SPDX-License-Identifier: GPL-2.0 + +# KCSAN uses udelay for introducing watchpoint delay; avoid recursion. +KCSAN_SANITIZE_delay.o := n + lib-y := clear_user.o delay.o copy_from_user.o \ copy_to_user.o copy_page.o \ clear_page.o csum.o insn.o memchr.o memcpy.o \ diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 421a5de806c6..a1ad028bdb30 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -278,9 +278,19 @@ void note_page_pgd(struct ptdump_state *pt_st, unsigned long addr, pgd_t pgd) void note_page_flush(struct ptdump_state *pt_st) { + struct ptdump_pg_state *st = container_of(pt_st, struct ptdump_pg_state, ptdump); + unsigned long end = st->end_address; pte_t pte_zero = {0}; - note_page(pt_st, 0, -1, pte_val(pte_zero)); + /* + * Address spaces that end at 1 << 64 have end_address == ULONG_MAX, + * but note_page() expects the exclusive end. In this case adjust end + * to the wraparound value 0. + */ + if (end == ULONG_MAX) + end = 0; + + note_page(pt_st, end, -1, pte_val(pte_zero)); } void ptdump_walk(struct seq_file *s, struct ptdump_info *info) @@ -296,6 +306,7 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info *info) .marker = info->markers, .mm = info->mm, .pg_level = &kernel_pg_levels[0], + .end_address = end, .level = -1, .ptdump = { .note_page_pte = note_page_pte, @@ -337,6 +348,7 @@ bool ptdump_check_wx(void) { -1, NULL}, }, .pg_level = &kernel_pg_levels[0], + .end_address = ~0UL, .level = -1, .check_wx = true, .ptdump = { diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c563bae8fce1..7be20cb32a27 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1168,7 +1168,12 @@ static int add_exception_handler(const struct bpf_insn *insn, ex->insn = ins_offset; - if (BPF_CLASS(insn->code) != BPF_LDX) + /* + * A load-acquire is of BPF_STX class, but reads from src_reg into + * dst_reg like a BPF_LDX does, hence it must not be treated as a store + * here. + */ + if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn)) dst_reg = DONT_CLEAR; ex->fixup = FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg); @@ -1183,7 +1188,7 @@ static int add_exception_handler(const struct bpf_insn *insn, * memory access. Pass the reg holding the unmodified 32-bit address to * ex_handler_bpf. */ - if (BPF_CLASS(insn->code) == BPF_LDX) + if (BPF_CLASS(insn->code) == BPF_LDX || bpf_atomic_is_load_acq(insn)) arena_reg = bpf2a64[insn->src_reg]; else arena_reg = bpf2a64[insn->dst_reg]; @@ -2428,9 +2433,8 @@ static void clear_garbage(struct jit_ctx *ctx, int reg, int effective_bytes) } static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, - const struct btf_func_model *m, - const struct arg_aux *a, - bool for_call_origin) + const struct btf_func_model *m, const struct arg_aux *a, + bool for_call_origin, bool is_struct_ops) { int i; int reg; @@ -2450,7 +2454,15 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, bargs_off += 8; } - soff = 32; /* on stack arguments start from FP + 32 */ + /* + * On-stack arguments start above the frame(s) pushed by the trampoline + * prologue. Entered through the fentry call from a traced function, the + * prologue saves both the parent (FP/x9) and the traced function + * (FP/LR) frames, so the arguments start at FP + 32. A struct_ops + * callback is called indirectly and only the FP/LR frame is saved, so + * they start at FP + 16. + */ + soff = is_struct_ops ? 16 : 32; doff = (for_call_origin ? oargs_off : bargs_off); /* save on stack arguments */ @@ -2628,7 +2640,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, emit(A64_STR64I(A64_R(10), A64_SP, nfuncargs_off), ctx); /* save args for bpf */ - save_args(ctx, bargs_off, oargs_off, m, a, false); + save_args(ctx, bargs_off, oargs_off, m, a, false, is_struct_ops); /* save callee saved registers */ emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx); @@ -2660,7 +2672,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_CALL_ORIG) { /* save args for original func */ - save_args(ctx, bargs_off, oargs_off, m, a, true); + save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops); /* call original func */ emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx); emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx); diff --git a/arch/csky/kernel/entry.S b/arch/csky/kernel/entry.S index c68cdcc76d60..3261f46f2244 100644 --- a/arch/csky/kernel/entry.S +++ b/arch/csky/kernel/entry.S @@ -93,11 +93,11 @@ csky_syscall_trace: ldw a2, (sp, LSAVE_A2) ldw a3, (sp, LSAVE_A3) #if defined(__CSKYABIV2__) - subi sp, 8 ldw r9, (sp, LSAVE_A4) + ldw r10, (sp, LSAVE_A5) + subi sp, 8 stw r9, (sp, 0x0) - ldw r9, (sp, LSAVE_A5) - stw r9, (sp, 0x4) + stw r10, (sp, 0x4) jsr syscallid /* Do system call */ addi sp, 8 #else diff --git a/arch/loongarch/kernel/smp.c b/arch/loongarch/kernel/smp.c index fc3817bd1919..4e3d79c38c46 100644 --- a/arch/loongarch/kernel/smp.c +++ b/arch/loongarch/kernel/smp.c @@ -535,28 +535,32 @@ int hibernate_resume_nonboot_cpu_disable(void) */ #ifdef CONFIG_PM -static int loongson_ipi_suspend(void) +static int loongson_ipi_suspend(void *data) { return 0; } -static void loongson_ipi_resume(void) +static void loongson_ipi_resume(void *data) { iocsr_write32(0xffffffff, LOONGARCH_IOCSR_IPI_EN); } -static struct syscore_ops loongson_ipi_syscore_ops = { +static const struct syscore_ops loongson_ipi_syscore_ops = { .resume = loongson_ipi_resume, .suspend = loongson_ipi_suspend, }; +static struct syscore loongson_ipi_syscore = { + .ops = &loongson_ipi_syscore_ops, +}; + /* * Enable boot cpu ipi before enabling nonboot cpus * during syscore_resume. */ static int __init ipi_pm_init(void) { - register_syscore_ops(&loongson_ipi_syscore_ops); + register_syscore(&loongson_ipi_syscore); return 0; } diff --git a/arch/m68k/emu/nfcon.c b/arch/m68k/emu/nfcon.c index d41260672e24..f4657d9f4f15 100644 --- a/arch/m68k/emu/nfcon.c +++ b/arch/m68k/emu/nfcon.c @@ -49,7 +49,7 @@ static void nfcon_write(struct console *con, const char *str, static struct tty_driver *nfcon_device(struct console *con, int *index) { *index = 0; - return console_is_registered(con) ? nfcon_tty_driver : NULL; + return nfcon_tty_driver; } static struct console nf_console = { diff --git a/arch/m68k/kernel/traps.c b/arch/m68k/kernel/traps.c index d2f25e8895e5..60759ba3cbd4 100644 --- a/arch/m68k/kernel/traps.c +++ b/arch/m68k/kernel/traps.c @@ -936,8 +936,8 @@ void show_stack(struct task_struct *task, unsigned long *stack, int i; if (!stack) { - if (task) - stack = (unsigned long *)task->thread.esp0; + if (task && task != current) + stack = (unsigned long *)task->thread.ksp; else stack = (unsigned long *)&stack; } diff --git a/arch/mips/alchemy/common/dbdma.c b/arch/mips/alchemy/common/dbdma.c index 6a3c890f7bbf..6c2c2010bbae 100644 --- a/arch/mips/alchemy/common/dbdma.c +++ b/arch/mips/alchemy/common/dbdma.c @@ -982,7 +982,7 @@ u32 au1xxx_dbdma_put_dscr(u32 chanid, au1x_ddma_desc_t *dscr) static unsigned long alchemy_dbdma_pm_data[NUM_DBDMA_CHANS + 1][6]; -static int alchemy_dbdma_suspend(void) +static int alchemy_dbdma_suspend(void *data) { int i; void __iomem *addr; @@ -1019,7 +1019,7 @@ static int alchemy_dbdma_suspend(void) return 0; } -static void alchemy_dbdma_resume(void) +static void alchemy_dbdma_resume(void *data) { int i; void __iomem *addr; @@ -1044,11 +1044,15 @@ static void alchemy_dbdma_resume(void) } } -static struct syscore_ops alchemy_dbdma_syscore_ops = { +static const struct syscore_ops alchemy_dbdma_syscore_ops = { .suspend = alchemy_dbdma_suspend, .resume = alchemy_dbdma_resume, }; +static struct syscore alchemy_dbdma_syscore = { + .ops = &alchemy_dbdma_syscore_ops, +}; + static int __init dbdma_setup(unsigned int irq, dbdev_tab_t *idtable) { int ret; @@ -1071,7 +1075,7 @@ static int __init dbdma_setup(unsigned int irq, dbdev_tab_t *idtable) printk(KERN_ERR "Cannot grab DBDMA interrupt!\n"); else { dbdma_initialized = 1; - register_syscore_ops(&alchemy_dbdma_syscore_ops); + register_syscore(&alchemy_dbdma_syscore); } return ret; diff --git a/arch/mips/alchemy/common/irq.c b/arch/mips/alchemy/common/irq.c index da9f9220048f..2403afcd2fb9 100644 --- a/arch/mips/alchemy/common/irq.c +++ b/arch/mips/alchemy/common/irq.c @@ -758,7 +758,7 @@ static inline void alchemy_ic_resume_one(void __iomem *base, unsigned long *d) wmb(); } -static int alchemy_ic_suspend(void) +static int alchemy_ic_suspend(void *data) { alchemy_ic_suspend_one((void __iomem *)KSEG1ADDR(AU1000_IC0_PHYS_ADDR), alchemy_gpic_pmdata); @@ -767,7 +767,7 @@ static int alchemy_ic_suspend(void) return 0; } -static void alchemy_ic_resume(void) +static void alchemy_ic_resume(void *data) { alchemy_ic_resume_one((void __iomem *)KSEG1ADDR(AU1000_IC1_PHYS_ADDR), &alchemy_gpic_pmdata[7]); @@ -775,7 +775,7 @@ static void alchemy_ic_resume(void) alchemy_gpic_pmdata); } -static int alchemy_gpic_suspend(void) +static int alchemy_gpic_suspend(void *data) { void __iomem *base = (void __iomem *)KSEG1ADDR(AU1300_GPIC_PHYS_ADDR); int i; @@ -806,7 +806,7 @@ static int alchemy_gpic_suspend(void) return 0; } -static void alchemy_gpic_resume(void) +static void alchemy_gpic_resume(void *data) { void __iomem *base = (void __iomem *)KSEG1ADDR(AU1300_GPIC_PHYS_ADDR); int i; @@ -837,16 +837,24 @@ static void alchemy_gpic_resume(void) wmb(); } -static struct syscore_ops alchemy_ic_pmops = { +static const struct syscore_ops alchemy_ic_pmops = { .suspend = alchemy_ic_suspend, .resume = alchemy_ic_resume, }; -static struct syscore_ops alchemy_gpic_pmops = { +static struct syscore alchemy_ic_pm = { + .ops = &alchemy_ic_pmops, +}; + +static const struct syscore_ops alchemy_gpic_pmops = { .suspend = alchemy_gpic_suspend, .resume = alchemy_gpic_resume, }; +static struct syscore alchemy_gpic_pm = { + .ops = &alchemy_gpic_pmops, +}; + /******************************************************************************/ /* create chained handlers for the 4 IC requests to the MIPS IRQ ctrl */ @@ -880,7 +888,7 @@ static void __init au1000_init_irq(struct alchemy_irqmap *map) ic_init((void __iomem *)KSEG1ADDR(AU1000_IC0_PHYS_ADDR)); ic_init((void __iomem *)KSEG1ADDR(AU1000_IC1_PHYS_ADDR)); - register_syscore_ops(&alchemy_ic_pmops); + register_syscore(&alchemy_ic_pm); mips_cpu_irq_init(); /* register all 64 possible IC0+IC1 irq sources as type "none". @@ -925,7 +933,7 @@ static void __init alchemy_gpic_init_irq(const struct alchemy_irqmap *dints) int i; void __iomem *bank_base; - register_syscore_ops(&alchemy_gpic_pmops); + register_syscore(&alchemy_gpic_pm); mips_cpu_irq_init(); /* disable & ack all possible interrupt sources */ diff --git a/arch/mips/alchemy/common/usb.c b/arch/mips/alchemy/common/usb.c index 5d618547ebf0..a55f32bf517c 100644 --- a/arch/mips/alchemy/common/usb.c +++ b/arch/mips/alchemy/common/usb.c @@ -580,22 +580,26 @@ static void alchemy_usb_pm(int susp) } } -static int alchemy_usb_suspend(void) +static int alchemy_usb_suspend(void *data) { alchemy_usb_pm(1); return 0; } -static void alchemy_usb_resume(void) +static void alchemy_usb_resume(void *data) { alchemy_usb_pm(0); } -static struct syscore_ops alchemy_usb_pm_ops = { +static const struct syscore_ops alchemy_usb_pm_syscore_ops = { .suspend = alchemy_usb_suspend, .resume = alchemy_usb_resume, }; +static struct syscore alchemy_usb_pm_syscore = { + .ops = &alchemy_usb_pm_syscore_ops, +}; + static int __init alchemy_usb_init(void) { int ret = 0; @@ -620,7 +624,7 @@ static int __init alchemy_usb_init(void) } if (!ret) - register_syscore_ops(&alchemy_usb_pm_ops); + register_syscore(&alchemy_usb_pm_syscore); return ret; } diff --git a/arch/mips/kernel/ptrace.c b/arch/mips/kernel/ptrace.c index 3f4c94c88124..87102a03b6ea 100644 --- a/arch/mips/kernel/ptrace.c +++ b/arch/mips/kernel/ptrace.c @@ -1321,8 +1321,12 @@ long arch_ptrace(struct task_struct *child, long request, */ asmlinkage long syscall_trace_enter(struct pt_regs *regs) { + long syscall; + user_exit(); + syscall = current_thread_info()->syscall; + if (test_thread_flag(TIF_SYSCALL_TRACE)) { if (ptrace_report_syscall_entry(regs)) return -1; @@ -1342,7 +1346,7 @@ asmlinkage long syscall_trace_enter(struct pt_regs *regs) * Negative syscall numbers are mistaken for rejected syscalls, but * won't have had the return value set appropriately, so we do so now. */ - if (current_thread_info()->syscall < 0) + if (syscall < 0) syscall_set_return_value(current, regs, -ENOSYS, 0); return current_thread_info()->syscall; } diff --git a/arch/mips/pci/pci-alchemy.c b/arch/mips/pci/pci-alchemy.c index 58625d1b6465..6bfee0f71803 100644 --- a/arch/mips/pci/pci-alchemy.c +++ b/arch/mips/pci/pci-alchemy.c @@ -304,7 +304,7 @@ static int alchemy_pci_def_idsel(unsigned int devsel, int assert) } /* save PCI controller register contents. */ -static int alchemy_pci_suspend(void) +static int alchemy_pci_suspend(void *data) { struct alchemy_pci_context *ctx = __alchemy_pci_ctx; if (!ctx) @@ -326,7 +326,7 @@ static int alchemy_pci_suspend(void) return 0; } -static void alchemy_pci_resume(void) +static void alchemy_pci_resume(void *data) { struct alchemy_pci_context *ctx = __alchemy_pci_ctx; if (!ctx) @@ -354,9 +354,13 @@ static void alchemy_pci_resume(void) alchemy_pci_wired_entry(ctx); /* install it */ } -static struct syscore_ops alchemy_pci_pmops = { - .suspend = alchemy_pci_suspend, - .resume = alchemy_pci_resume, +static const struct syscore_ops alchemy_pci_syscore_ops = { + .suspend = alchemy_pci_suspend, + .resume = alchemy_pci_resume, +}; + +static struct syscore alchemy_pci_syscore = { + .ops = &alchemy_pci_syscore_ops, }; static int alchemy_pci_probe(struct platform_device *pdev) @@ -478,7 +482,7 @@ static int alchemy_pci_probe(struct platform_device *pdev) __alchemy_pci_ctx = ctx; platform_set_drvdata(pdev, ctx); - register_syscore_ops(&alchemy_pci_pmops); + register_syscore(&alchemy_pci_syscore); register_pci_controller(&ctx->alchemy_pci_ctrl); dev_info(&pdev->dev, "PCI controller at %ld MHz\n", diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 9537a61ebae0..7b02fe19d26a 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -1059,6 +1059,7 @@ config PPC_SECURE_BOOT depends on IMA_ARCH_POLICY imply IMA_SECURE_AND_OR_TRUSTED_BOOT select PSERIES_PLPKS if PPC_PSERIES + select HAVE_ARCH_GET_SECUREBOOT help Systems with firmware secure boot enabled need to define security policies to extend secure boot to the OS. This config allows a user diff --git a/arch/powerpc/configs/85xx-hw.config b/arch/powerpc/configs/85xx-hw.config index 8aff83217397..b655c36e56da 100644 --- a/arch/powerpc/configs/85xx-hw.config +++ b/arch/powerpc/configs/85xx-hw.config @@ -90,6 +90,7 @@ CONFIG_PPC_EPAPR_HV_BYTECHAN=y CONFIG_QE_GPIO=y CONFIG_QUICC_ENGINE=y CONFIG_RAPIDIO=y +CONFIG_RAS=y CONFIG_RTC_CLASS=y CONFIG_RTC_DRV_CMOS=y CONFIG_RTC_DRV_DS1307=y diff --git a/arch/powerpc/kernel/ima_arch.c b/arch/powerpc/kernel/ima_arch.c index b7029beed847..0d8892a03526 100644 --- a/arch/powerpc/kernel/ima_arch.c +++ b/arch/powerpc/kernel/ima_arch.c @@ -7,11 +7,6 @@ #include #include -bool arch_ima_get_secureboot(void) -{ - return is_ppc_secureboot_enabled(); -} - /* * The "secure_rules" are enabled only on "secureboot" enabled systems. * These rules verify the file signatures against known good values. diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c index a0e8b998c9b5..cb010830c1c4 100644 --- a/arch/powerpc/kernel/irq.c +++ b/arch/powerpc/kernel/irq.c @@ -217,8 +217,12 @@ static __always_inline void call_do_softirq(const void *sp) [sp] "b" (sp), [offset] "i" (THREAD_SIZE - STACK_FRAME_MIN_SIZE), [callee] "i" (__do_softirq) : // Clobbers - "lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6", - "cr7", "r0", "r3", "r4", "r5", "r6", "r7", "r8", "r9", "r10", + "lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6", "cr7", "r0", + /* r2 may be clobbered by the callee when using PCREL mode in the ELFv2 ABI. */ +#ifdef CONFIG_PPC_KERNEL_PCREL + "r2", +#endif + "r3", "r4", "r5", "r6", "r7", "r8", "r9", "r10", "r11", "r12" ); } @@ -275,8 +279,12 @@ static __always_inline void call_do_irq(struct pt_regs *regs, void *sp) [sp] "b" (sp), [offset] "i" (THREAD_SIZE - STACK_FRAME_MIN_SIZE), [callee] "i" (__do_irq) : // Clobbers - "lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6", - "cr7", "r0", "r4", "r5", "r6", "r7", "r8", "r9", "r10", + "lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6", "cr7", "r0", + /* r2 may be clobbered by the callee when using PCREL mode in the ELFv2 ABI. */ +#ifdef CONFIG_PPC_KERNEL_PCREL + "r2", +#endif + "r4", "r5", "r6", "r7", "r8", "r9", "r10", "r11", "r12" ); } diff --git a/arch/powerpc/kernel/secure_boot.c b/arch/powerpc/kernel/secure_boot.c index 3a28795b4ed8..28436c1599e0 100644 --- a/arch/powerpc/kernel/secure_boot.c +++ b/arch/powerpc/kernel/secure_boot.c @@ -5,6 +5,7 @@ */ #include #include +#include #include #include @@ -44,6 +45,11 @@ bool is_ppc_secureboot_enabled(void) return enabled; } +bool arch_get_secureboot(void) +{ + return is_ppc_secureboot_enabled(); +} + bool is_ppc_trustedboot_enabled(void) { struct device_node *node; diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 0cd9c0c21af3..b5518fba3b92 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -289,6 +289,9 @@ void smp_muxed_ipi_set_message(int cpu, int msg) void smp_muxed_ipi_message_pass(int cpu, int msg) { + if (!smp_ops->cause_ipi) + return; + smp_muxed_ipi_set_message(cpu, msg); /* diff --git a/arch/powerpc/kexec/crash.c b/arch/powerpc/kexec/crash.c index e6539f213b3d..a520f851c3a6 100644 --- a/arch/powerpc/kexec/crash.c +++ b/arch/powerpc/kexec/crash.c @@ -502,7 +502,7 @@ static void update_crash_elfcorehdr(struct kimage *image, struct memory_notify * ret = get_crash_memory_ranges(&cmem); if (ret) { pr_err("Failed to get crash mem range\n"); - return; + goto out; } /* diff --git a/arch/powerpc/platforms/44x/gpio.c b/arch/powerpc/platforms/44x/gpio.c index aea0d913b59d..4413a94cf7a6 100644 --- a/arch/powerpc/platforms/44x/gpio.c +++ b/arch/powerpc/platforms/44x/gpio.c @@ -169,6 +169,7 @@ static int ppc4xx_gpio_probe(struct platform_device *ofdev) gc = &chip->gc; + gc->parent = dev; gc->base = -1; gc->ngpio = 32; gc->direction_input = ppc4xx_gpio_dir_in; diff --git a/arch/powerpc/platforms/cell/spu_base.c b/arch/powerpc/platforms/cell/spu_base.c index 2c07387201d0..2ddb93df4817 100644 --- a/arch/powerpc/platforms/cell/spu_base.c +++ b/arch/powerpc/platforms/cell/spu_base.c @@ -726,7 +726,7 @@ static inline void crash_register_spus(struct list_head *list) } #endif -static void spu_shutdown(void) +static void spu_shutdown(void *data) { struct spu *spu; @@ -738,10 +738,14 @@ static void spu_shutdown(void) mutex_unlock(&spu_full_list_mutex); } -static struct syscore_ops spu_syscore_ops = { +static const struct syscore_ops spu_syscore_ops = { .shutdown = spu_shutdown, }; +static struct syscore spu_syscore = { + .ops = &spu_syscore_ops, +}; + static int __init init_spu_base(void) { int i, ret = 0; @@ -774,7 +778,7 @@ static int __init init_spu_base(void) crash_register_spus(&spu_full_list); mutex_unlock(&spu_full_list_mutex); spu_add_dev_attr(&dev_attr_stat); - register_syscore_ops(&spu_syscore_ops); + register_syscore(&spu_syscore); spu_init_affinity(); diff --git a/arch/powerpc/platforms/powermac/pic.c b/arch/powerpc/platforms/powermac/pic.c index c37783a03d25..1959cc13438f 100644 --- a/arch/powerpc/platforms/powermac/pic.c +++ b/arch/powerpc/platforms/powermac/pic.c @@ -600,7 +600,7 @@ static int pmacpic_find_viaint(void) return viaint; } -static int pmacpic_suspend(void) +static int pmacpic_suspend(void *data) { int viaint = pmacpic_find_viaint(); @@ -621,7 +621,7 @@ static int pmacpic_suspend(void) return 0; } -static void pmacpic_resume(void) +static void pmacpic_resume(void *data) { int i; @@ -634,15 +634,19 @@ static void pmacpic_resume(void) pmac_unmask_irq(irq_get_irq_data(i)); } -static struct syscore_ops pmacpic_syscore_ops = { +static const struct syscore_ops pmacpic_syscore_ops = { .suspend = pmacpic_suspend, .resume = pmacpic_resume, }; +static struct syscore pmacpic_syscore = { + .ops = &pmacpic_syscore_ops, +}; + static int __init init_pmacpic_syscore(void) { if (pmac_irq_hw[0]) - register_syscore_ops(&pmacpic_syscore_ops); + register_syscore(&pmacpic_syscore); return 0; } diff --git a/arch/powerpc/sysdev/cpm_common.c b/arch/powerpc/sysdev/cpm_common.c index 07ea605ab0e6..b5d200e3ad68 100644 --- a/arch/powerpc/sysdev/cpm_common.c +++ b/arch/powerpc/sysdev/cpm_common.c @@ -181,6 +181,18 @@ static int cpm2_gpio32_dir_in(struct gpio_chip *gc, unsigned int gpio) return 0; } +static int cpm2_gpio32_get_direction(struct gpio_chip *gc, unsigned int gpio) +{ + struct cpm2_gpio32_chip *cpm2_gc = gpiochip_get_data(gc); + struct cpm2_ioports __iomem *iop = cpm2_gc->regs; + u32 pin_mask = 1 << (31 - gpio); + + if (in_be32(&iop->dir) & pin_mask) + return GPIO_LINE_DIRECTION_OUT; + + return GPIO_LINE_DIRECTION_IN; +} + int cpm2_gpiochip_add32(struct device *dev) { struct device_node *np = dev->of_node; @@ -199,6 +211,7 @@ int cpm2_gpiochip_add32(struct device *dev) gc->ngpio = 32; gc->direction_input = cpm2_gpio32_dir_in; gc->direction_output = cpm2_gpio32_dir_out; + gc->get_direction = cpm2_gpio32_get_direction; gc->get = cpm2_gpio32_get; gc->set = cpm2_gpio32_set; gc->parent = dev; diff --git a/arch/powerpc/sysdev/fsl_lbc.c b/arch/powerpc/sysdev/fsl_lbc.c index 217cea150987..7ed07232a69a 100644 --- a/arch/powerpc/sysdev/fsl_lbc.c +++ b/arch/powerpc/sysdev/fsl_lbc.c @@ -350,7 +350,7 @@ static int fsl_lbc_ctrl_probe(struct platform_device *dev) #ifdef CONFIG_SUSPEND /* save lbc registers */ -static int fsl_lbc_syscore_suspend(void) +static int fsl_lbc_syscore_suspend(void *data) { struct fsl_lbc_ctrl *ctrl; struct fsl_lbc_regs __iomem *lbc; @@ -374,7 +374,7 @@ static int fsl_lbc_syscore_suspend(void) } /* restore lbc registers */ -static void fsl_lbc_syscore_resume(void) +static void fsl_lbc_syscore_resume(void *data) { struct fsl_lbc_ctrl *ctrl; struct fsl_lbc_regs __iomem *lbc; @@ -408,10 +408,14 @@ static const struct of_device_id fsl_lbc_match[] = { }; #ifdef CONFIG_SUSPEND -static struct syscore_ops lbc_syscore_pm_ops = { +static const struct syscore_ops lbc_syscore_pm_ops = { .suspend = fsl_lbc_syscore_suspend, .resume = fsl_lbc_syscore_resume, }; + +static struct syscore lbc_syscore_pm = { + .ops = &lbc_syscore_pm_ops, +}; #endif static struct platform_driver fsl_lbc_ctrl_driver = { @@ -425,7 +429,7 @@ static struct platform_driver fsl_lbc_ctrl_driver = { static int __init fsl_lbc_init(void) { #ifdef CONFIG_SUSPEND - register_syscore_ops(&lbc_syscore_pm_ops); + register_syscore(&lbc_syscore_pm); #endif return platform_driver_register(&fsl_lbc_ctrl_driver); } diff --git a/arch/powerpc/sysdev/fsl_pci.c b/arch/powerpc/sysdev/fsl_pci.c index ef7707ea0db7..4e501654cb41 100644 --- a/arch/powerpc/sysdev/fsl_pci.c +++ b/arch/powerpc/sysdev/fsl_pci.c @@ -1258,7 +1258,7 @@ static void fsl_pci_syscore_do_suspend(struct pci_controller *hose) send_pme_turnoff_message(hose); } -static int fsl_pci_syscore_suspend(void) +static int fsl_pci_syscore_suspend(void *data) { struct pci_controller *hose, *tmp; @@ -1291,7 +1291,7 @@ static void fsl_pci_syscore_do_resume(struct pci_controller *hose) setup_pci_atmu(hose); } -static void fsl_pci_syscore_resume(void) +static void fsl_pci_syscore_resume(void *data) { struct pci_controller *hose, *tmp; @@ -1299,10 +1299,14 @@ static void fsl_pci_syscore_resume(void) fsl_pci_syscore_do_resume(hose); } -static struct syscore_ops pci_syscore_pm_ops = { +static const struct syscore_ops pci_syscore_pm_ops = { .suspend = fsl_pci_syscore_suspend, .resume = fsl_pci_syscore_resume, }; + +static struct syscore pci_syscore_pm = { + .ops = &pci_syscore_pm_ops, +}; #endif void fsl_pcibios_fixup_phb(struct pci_controller *phb) @@ -1359,7 +1363,7 @@ static struct platform_driver fsl_pci_driver = { static int __init fsl_pci_init(void) { #ifdef CONFIG_PM_SLEEP - register_syscore_ops(&pci_syscore_pm_ops); + register_syscore(&pci_syscore_pm); #endif return platform_driver_register(&fsl_pci_driver); } diff --git a/arch/powerpc/sysdev/ipic.c b/arch/powerpc/sysdev/ipic.c index 70be2105865d..290ba8427239 100644 --- a/arch/powerpc/sysdev/ipic.c +++ b/arch/powerpc/sysdev/ipic.c @@ -817,7 +817,7 @@ static struct { u32 sercr; } ipic_saved_state; -static int ipic_suspend(void) +static int ipic_suspend(void *data) { struct ipic *ipic = primary_ipic; @@ -848,7 +848,7 @@ static int ipic_suspend(void) return 0; } -static void ipic_resume(void) +static void ipic_resume(void *data) { struct ipic *ipic = primary_ipic; @@ -870,18 +870,22 @@ static void ipic_resume(void) #define ipic_resume NULL #endif -static struct syscore_ops ipic_syscore_ops = { +static const struct syscore_ops ipic_syscore_ops = { .suspend = ipic_suspend, .resume = ipic_resume, }; +static struct syscore ipic_syscore = { + .ops = &ipic_syscore_ops, +}; + static int __init init_ipic_syscore(void) { if (!primary_ipic || !primary_ipic->regs) return -ENODEV; printk(KERN_DEBUG "Registering ipic system core operations\n"); - register_syscore_ops(&ipic_syscore_ops); + register_syscore(&ipic_syscore); return 0; } diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c index ad7310bba00b..67e51998d1ae 100644 --- a/arch/powerpc/sysdev/mpic.c +++ b/arch/powerpc/sysdev/mpic.c @@ -1944,7 +1944,7 @@ static void mpic_suspend_one(struct mpic *mpic) } } -static int mpic_suspend(void) +static int mpic_suspend(void *data) { struct mpic *mpic = mpics; @@ -1986,7 +1986,7 @@ static void mpic_resume_one(struct mpic *mpic) } /* end for loop */ } -static void mpic_resume(void) +static void mpic_resume(void *data) { struct mpic *mpic = mpics; @@ -1996,19 +1996,23 @@ static void mpic_resume(void) } } -static struct syscore_ops mpic_syscore_ops = { +static const struct syscore_ops mpic_syscore_ops = { .resume = mpic_resume, .suspend = mpic_suspend, }; +static struct syscore mpic_syscore = { + .ops = &mpic_syscore_ops, +}; + static int mpic_init_sys(void) { int rc; - register_syscore_ops(&mpic_syscore_ops); + register_syscore(&mpic_syscore); rc = subsys_system_register(&mpic_subsys, NULL); if (rc) { - unregister_syscore_ops(&mpic_syscore_ops); + unregister_syscore(&mpic_syscore); pr_err("mpic: Failed to register subsystem!\n"); return rc; } diff --git a/arch/powerpc/sysdev/mpic_timer.c b/arch/powerpc/sysdev/mpic_timer.c index 7166e2e0baaf..60f5b3934b51 100644 --- a/arch/powerpc/sysdev/mpic_timer.c +++ b/arch/powerpc/sysdev/mpic_timer.c @@ -519,7 +519,7 @@ static void __init timer_group_init(struct device_node *np) kfree(priv); } -static void mpic_timer_resume(void) +static void mpic_timer_resume(void *data) { struct timer_group_priv *priv; @@ -535,10 +535,14 @@ static const struct of_device_id mpic_timer_ids[] = { {}, }; -static struct syscore_ops mpic_timer_syscore_ops = { +static const struct syscore_ops mpic_timer_syscore_ops = { .resume = mpic_timer_resume, }; +static struct syscore mpic_timer_syscore = { + .ops = &mpic_timer_syscore_ops, +}; + static int __init mpic_timer_init(void) { struct device_node *np = NULL; @@ -546,7 +550,7 @@ static int __init mpic_timer_init(void) for_each_matching_node(np, mpic_timer_ids) timer_group_init(np); - register_syscore_ops(&mpic_timer_syscore_ops); + register_syscore(&mpic_timer_syscore); if (list_empty(&timer_group_list)) return -ENODEV; diff --git a/arch/riscv/boot/dts/sophgo/cv180x.dtsi b/arch/riscv/boot/dts/sophgo/cv180x.dtsi index ccdb45498653..b2453ae45216 100644 --- a/arch/riscv/boot/dts/sophgo/cv180x.dtsi +++ b/arch/riscv/boot/dts/sophgo/cv180x.dtsi @@ -391,7 +391,7 @@ sdhci1: mmc@4320000 { }; dmac: dma-controller@4330000 { - compatible = "snps,axi-dma-1.01a"; + compatible = "sophgo,cv1800b-axi-dma", "snps,axi-dma-1.01a"; reg = <0x04330000 0x1000>; interrupts = ; clocks = <&clk CLK_SDMA_AXI>, <&clk CLK_SDMA_AXI>; diff --git a/arch/riscv/boot/dts/spacemit/Makefile b/arch/riscv/boot/dts/spacemit/Makefile index 152832644870..95889e7269d1 100644 --- a/arch/riscv/boot/dts/spacemit/Makefile +++ b/arch/riscv/boot/dts/spacemit/Makefile @@ -1,4 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 dtb-$(CONFIG_ARCH_SPACEMIT) += k1-bananapi-f3.dtb dtb-$(CONFIG_ARCH_SPACEMIT) += k1-milkv-jupiter.dtb +dtb-$(CONFIG_ARCH_SPACEMIT) += k1-musepi-pro.dtb +dtb-$(CONFIG_ARCH_SPACEMIT) += k1-orangepi-r2s.dtb dtb-$(CONFIG_ARCH_SPACEMIT) += k1-orangepi-rv2.dtb diff --git a/arch/riscv/boot/dts/spacemit/k1-bananapi-f3.dts b/arch/riscv/boot/dts/spacemit/k1-bananapi-f3.dts index a58192ba67b2..b2c40bae6094 100644 --- a/arch/riscv/boot/dts/spacemit/k1-bananapi-f3.dts +++ b/arch/riscv/boot/dts/spacemit/k1-bananapi-f3.dts @@ -46,7 +46,7 @@ ð0 { phy-handle = <&rgmii0>; phy-mode = "rgmii-id"; pinctrl-names = "default"; - pinctrl-0 = <&gmac0_cfg>; + pinctrl-0 = <&gmac0_cfg>, <&gmac0_clk_ref_cfg>; rx-internal-delay-ps = <0>; tx-internal-delay-ps = <0>; status = "okay"; @@ -69,7 +69,7 @@ ð1 { phy-handle = <&rgmii1>; phy-mode = "rgmii-id"; pinctrl-names = "default"; - pinctrl-0 = <&gmac1_cfg>; + pinctrl-0 = <&gmac1_cfg>, <&gmac1_clk_ref_cfg>; rx-internal-delay-ps = <0>; tx-internal-delay-ps = <250>; status = "okay"; diff --git a/arch/riscv/boot/dts/spacemit/k1-milkv-jupiter.dts b/arch/riscv/boot/dts/spacemit/k1-milkv-jupiter.dts index a01f69f202e3..695af4919b94 100644 --- a/arch/riscv/boot/dts/spacemit/k1-milkv-jupiter.dts +++ b/arch/riscv/boot/dts/spacemit/k1-milkv-jupiter.dts @@ -26,7 +26,7 @@ ð0 { phy-handle = <&rgmii0>; phy-mode = "rgmii-id"; pinctrl-names = "default"; - pinctrl-0 = <&gmac0_cfg>; + pinctrl-0 = <&gmac0_cfg>, <&gmac0_clk_ref_cfg>; rx-internal-delay-ps = <0>; tx-internal-delay-ps = <0>; status = "okay"; @@ -49,7 +49,7 @@ ð1 { phy-handle = <&rgmii1>; phy-mode = "rgmii-id"; pinctrl-names = "default"; - pinctrl-0 = <&gmac1_cfg>; + pinctrl-0 = <&gmac1_cfg>, <&gmac1_clk_ref_cfg>; rx-internal-delay-ps = <0>; tx-internal-delay-ps = <250>; status = "okay"; diff --git a/arch/riscv/boot/dts/spacemit/k1-musepi-pro.dts b/arch/riscv/boot/dts/spacemit/k1-musepi-pro.dts new file mode 100644 index 000000000000..42efc8c1a581 --- /dev/null +++ b/arch/riscv/boot/dts/spacemit/k1-musepi-pro.dts @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) +/* + * Copyright (C) 2024 Yangyu Chen + * Copyright (C) 2025 SpacemiT, Inc + * Copyright (C) 2025 Troy Mitchell + */ + +/dts-v1/; + +#include "k1.dtsi" +#include "k1-pinctrl.dtsi" + +/ { + model = "SpacemiT MusePi Pro"; + compatible = "spacemit,musepi-pro", "spacemit,k1"; + + aliases { + ethernet0 = ð0; + serial0 = &uart0; + }; + + chosen { + stdout-path = "serial0"; + }; + + leds { + compatible = "gpio-leds"; + + led1 { + label = "sys-led"; + gpios = <&gpio K1_GPIO(96) GPIO_ACTIVE_HIGH>; + linux,default-trigger = "heartbeat"; + default-state = "on"; + }; + }; +}; + +&emmc { + bus-width = <8>; + mmc-hs400-1_8v; + mmc-hs400-enhanced-strobe; + non-removable; + no-sd; + no-sdio; + status = "okay"; +}; + +ð0 { + phy-handle = <&rgmii0>; + phy-mode = "rgmii-id"; + pinctrl-0 = <&gmac0_cfg>, <&gmac0_clk_ref_cfg>; + pinctrl-names = "default"; + rx-internal-delay-ps = <0>; + tx-internal-delay-ps = <0>; + status = "okay"; + + mdio-bus { + #address-cells = <0x1>; + #size-cells = <0x0>; + + reset-gpios = <&gpio K1_GPIO(110) GPIO_ACTIVE_LOW>; + reset-delay-us = <10000>; + reset-post-delay-us = <100000>; + + rgmii0: phy@1 { + reg = <0x1>; + }; + }; +}; + +&pdma { + status = "okay"; +}; + +&uart0 { + pinctrl-0 = <&uart0_2_cfg>; + pinctrl-names = "default"; + status = "okay"; +}; diff --git a/arch/riscv/boot/dts/spacemit/k1-orangepi-r2s.dts b/arch/riscv/boot/dts/spacemit/k1-orangepi-r2s.dts new file mode 100644 index 000000000000..312d5bc86072 --- /dev/null +++ b/arch/riscv/boot/dts/spacemit/k1-orangepi-r2s.dts @@ -0,0 +1,90 @@ +// SPDX-License-Identifier: (GPL-2.0 OR MIT) +/* + * Copyright (C) 2025 Michael Opdenacker + */ + +/dts-v1/; + +#include "k1.dtsi" +#include "k1-pinctrl.dtsi" + +/ { + model = "OrangePi R2S"; + compatible = "xunlong,orangepi-r2s", "spacemit,k1"; + + aliases { + serial0 = &uart0; + ethernet0 = ð0; + ethernet1 = ð1; + }; + + chosen { + stdout-path = "serial0"; + }; +}; + +&emmc { + bus-width = <8>; + mmc-hs400-1_8v; + mmc-hs400-enhanced-strobe; + non-removable; + no-sd; + no-sdio; + status = "okay"; +}; + +ð0 { + phy-handle = <&rgmii0>; + phy-mode = "rgmii-id"; + pinctrl-names = "default"; + pinctrl-0 = <&gmac0_cfg>, <&gmac0_clk_ref_cfg>; + rx-internal-delay-ps = <0>; + tx-internal-delay-ps = <0>; + status = "okay"; + + mdio-bus { + #address-cells = <0x1>; + #size-cells = <0x0>; + + reset-gpios = <&gpio K1_GPIO(110) GPIO_ACTIVE_LOW>; + reset-delay-us = <10000>; + reset-post-delay-us = <100000>; + + rgmii0: phy@1 { + reg = <0x1>; + }; + }; +}; + +ð1 { + phy-handle = <&rgmii1>; + phy-mode = "rgmii-id"; + pinctrl-names = "default"; + pinctrl-0 = <&gmac1_cfg>, <&gmac1_clk_ref_cfg>; + rx-internal-delay-ps = <0>; + tx-internal-delay-ps = <250>; + status = "okay"; + + mdio-bus { + #address-cells = <0x1>; + #size-cells = <0x0>; + + reset-gpios = <&gpio K1_GPIO(115) GPIO_ACTIVE_LOW>; + reset-delay-us = <10000>; + reset-post-delay-us = <100000>; + + rgmii1: phy@1 { + reg = <0x1>; + }; + }; +}; + +&pdma { + status = "okay"; +}; + +&uart0 { + pinctrl-names = "default"; + pinctrl-0 = <&uart0_2_cfg>; + status = "okay"; +}; diff --git a/arch/riscv/boot/dts/spacemit/k1-pinctrl.dtsi b/arch/riscv/boot/dts/spacemit/k1-pinctrl.dtsi index aff19c86d5ff..48c4b4131164 100644 --- a/arch/riscv/boot/dts/spacemit/k1-pinctrl.dtsi +++ b/arch/riscv/boot/dts/spacemit/k1-pinctrl.dtsi @@ -27,8 +27,16 @@ gmac0-pins { , /* gmac0_tx_en */ , /* gmac0_mdc */ , /* gmac0_mdio */ - , /* gmac0_int_n */ - ; /* gmac0_clk_ref */ + ; /* gmac0_int_n */ + + bias-pull-up = <0>; + drive-strength = <21>; + }; + }; + + gmac0_clk_ref_cfg: gmac0-clk-ref-cfg { + gmac0-clk-ref-pins { + pinmux = ; /* gmac0_clk_ref */ bias-pull-up = <0>; drive-strength = <21>; @@ -51,8 +59,16 @@ gmac1-pins { , /* gmac1_tx_en */ , /* gmac1_mdc */ , /* gmac1_mdio */ - , /* gmac1_int_n */ - ; /* gmac1_clk_ref */ + ; /* gmac1_int_n */ + + bias-pull-up = <0>; + drive-strength = <21>; + }; + }; + + gmac1_clk_ref_cfg: gmac1-clk-ref-cfg { + gmac1-clk-ref-pins { + pinmux = ; /* gmac1_clk_ref */ bias-pull-up = <0>; drive-strength = <21>; diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 2367e9755524..17d21a708fb7 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -409,7 +409,8 @@ static const unsigned int riscv_c_exts[] = { /* * The canonical order of ISA extension names in the ISA string is defined in - * chapter 27 of the unprivileged specification. + * Chapter 27 of the RISC-V Instruction Set Manual Volume I Unprivileged ISA + * (Document Version 20191213). * * Ordinarily, for in-kernel data structures, this order is unimportant but * isa_ext_arr defines the order of the ISA string in /proc/cpuinfo. diff --git a/arch/riscv/kernel/machine_kexec_file.c b/arch/riscv/kernel/machine_kexec_file.c index dd9d92a96517..6ef5bd34d130 100644 --- a/arch/riscv/kernel/machine_kexec_file.c +++ b/arch/riscv/kernel/machine_kexec_file.c @@ -61,7 +61,7 @@ static int prepare_elf_headers(void **addr, unsigned long *sz) unsigned int nr_ranges; int ret; - nr_ranges = 1; /* For exclusion of crashkernel region */ + nr_ranges = 2; /* For exclusion of crashkernel region */ walk_system_ram_res(0, -1, &nr_ranges, get_nr_ram_ranges_callback); cmem = kmalloc(struct_size(cmem, ranges, nr_ranges), GFP_KERNEL); @@ -76,8 +76,16 @@ static int prepare_elf_headers(void **addr, unsigned long *sz) /* Exclude crashkernel region */ ret = crash_exclude_mem_range(cmem, crashk_res.start, crashk_res.end); - if (!ret) - ret = crash_prepare_elf64_headers(cmem, true, addr, sz); + if (ret) + goto out; + + if (crashk_low_res.end) { + ret = crash_exclude_mem_range(cmem, crashk_low_res.start, crashk_low_res.end); + if (ret) + goto out; + } + + ret = crash_prepare_elf64_headers(cmem, true, addr, sz); out: kfree(cmem); diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 9e9e6dcfc482..3e4eea261ab9 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -18,7 +18,6 @@ #define RV_MAX_REG_ARGS 8 #define RV_FENTRY_NINSNS 2 #define RV_FENTRY_NBYTES (RV_FENTRY_NINSNS * 4) -#define RV_KCFI_NINSNS (IS_ENABLED(CONFIG_CFI) ? 1 : 0) /* imm that allows emit_imm to emit max count insns */ #define RV_MAX_COUNT_IMM 0x7FFF7FF7FF7FF7FF @@ -272,8 +271,8 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx) if (!is_tail_call) emit_addiw(RV_REG_A0, RV_REG_A5, 0, ctx); emit_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA, - /* kcfi, fentry and TCC init insns will be skipped on tailcall */ - is_tail_call ? (RV_KCFI_NINSNS + RV_FENTRY_NINSNS + 1) * 4 : 0, + /* fentry and TCC init insns will be skipped on tailcall */ + is_tail_call ? (RV_FENTRY_NINSNS + 1) * 4 : 0, ctx); } @@ -1771,9 +1770,10 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, for (idx = 0; idx < fm->nr_args; idx++) { u8 reg = bpf_to_rv_reg(BPF_REG_1 + idx, ctx); + bool sign = fm->arg_flags[idx] & BTF_FMODEL_SIGNED_ARG; - if (fm->arg_size[idx] == sizeof(int)) - emit_sextw(reg, reg, ctx); + if (sign_extend(reg, reg, fm->arg_size[idx], sign, ctx)) + return -EINVAL; } } @@ -1926,7 +1926,12 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, else ret = emit_atomic_rmw(rd, rs, insn, ctx); - ret = ret ?: add_exception_handler(insn, REG_DONT_CLEAR_MARKER, ctx); + /* ret can be 1 (skip-zext); extable entry still needs to be added */ + if (ret >= 0) + ret = add_exception_handler(insn, + bpf_atomic_is_load_acq(insn) ? rd : REG_DONT_CLEAR_MARKER, + ctx) ?: ret; + if (ret) return ret; break; @@ -1973,6 +1978,8 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) /* emit kcfi type preamble immediately before the first insn */ emit_kcfi(is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash, ctx); + /* bpf prog starts here as kcfi skipped during prog->bpf_func setup */ + /* nops reserved for auipc+jalr pair */ for (i = 0; i < RV_FENTRY_NINSNS; i++) emit(rv_nop(), ctx); diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index e4ab5bb9c9f6..09bd4eaac63e 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -251,6 +251,7 @@ void bpf_jit_free(struct bpf_prog *prog) */ if (jit_data) { bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header); + kfree(jit_data->ctx.offset); kfree(jit_data); } hdr = bpf_jit_binary_pack_hdr(prog); diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index f43a6570fd6f..4a663929ebc3 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -74,7 +74,6 @@ config S390 # # Note: keep this list sorted alphabetically # - imply IMA_SECURE_AND_OR_TRUSTED_BOOT select ALTERNATE_USER_ADDRESS_SPACE select ARCH_32BIT_USTAT_F_TINODE select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE @@ -175,6 +174,7 @@ config S390 select GENERIC_IOREMAP if PCI select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL + select HAVE_ARCH_GET_SECUREBOOT select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN diff --git a/arch/s390/include/asm/irqflags.h b/arch/s390/include/asm/irqflags.h index bcab456dfb80..6eb6de6a99f2 100644 --- a/arch/s390/include/asm/irqflags.h +++ b/arch/s390/include/asm/irqflags.h @@ -37,18 +37,24 @@ static __always_inline void __arch_local_irq_ssm(unsigned long flags) asm volatile("ssm %0" : : "Q" (flags) : "memory"); } -#ifdef CONFIG_KMSAN -#define arch_local_irq_attributes noinline notrace __no_sanitize_memory __maybe_unused +#if defined(CONFIG_KMSAN) && !defined(__DECOMPRESSOR) +unsigned long arch_local_save_flags(void); +unsigned long arch_local_irq_save(void); +void arch_local_irq_enable_external(void); +void arch_local_irq_enable(void); #else -#define arch_local_irq_attributes __always_inline +#define arch_local_save_flags __arch_local_save_flags +#define arch_local_irq_save __arch_local_irq_save +#define arch_local_irq_enable_external __arch_local_irq_enable_external +#define arch_local_irq_enable __arch_local_irq_enable #endif -static arch_local_irq_attributes unsigned long arch_local_save_flags(void) +static __always_inline unsigned long __arch_local_save_flags(void) { return __arch_local_irq_stnsm(0xff); } -static arch_local_irq_attributes unsigned long arch_local_irq_save(void) +static __always_inline unsigned long __arch_local_irq_save(void) { return __arch_local_irq_stnsm(0xfc); } @@ -58,12 +64,12 @@ static __always_inline void arch_local_irq_disable(void) arch_local_irq_save(); } -static arch_local_irq_attributes void arch_local_irq_enable_external(void) +static __always_inline void __arch_local_irq_enable_external(void) { __arch_local_irq_stosm(0x01); } -static arch_local_irq_attributes void arch_local_irq_enable(void) +static __always_inline void __arch_local_irq_enable(void) { __arch_local_irq_stosm(0x03); } diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile index eb06ff888314..9ad2f87aae28 100644 --- a/arch/s390/kernel/Makefile +++ b/arch/s390/kernel/Makefile @@ -74,7 +74,7 @@ obj-$(CONFIG_JUMP_LABEL) += jump_label.o obj-$(CONFIG_KEXEC_FILE) += machine_kexec_file.o kexec_image.o obj-$(CONFIG_KEXEC_FILE) += kexec_elf.o obj-$(CONFIG_CERT_STORE) += cert_store.o -obj-$(CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT) += ima_arch.o +obj-$(CONFIG_KMSAN) += irqflags.o obj-$(CONFIG_PERF_EVENTS) += perf_event.o obj-$(CONFIG_PERF_EVENTS) += perf_cpum_cf.o perf_cpum_sf.o diff --git a/arch/s390/kernel/debug.c b/arch/s390/kernel/debug.c index 112cf7b40dac..f5359b010051 100644 --- a/arch/s390/kernel/debug.c +++ b/arch/s390/kernel/debug.c @@ -906,8 +906,8 @@ void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas) mutex_unlock(&debug_mutex); } -/* Remove debugfs entries and remove from internal list. */ -static void _debug_unregister(debug_info_t *id) +/* Remove debugfs entries. */ +static void _debug_unregister_debugfs(debug_info_t *id) { int i; @@ -917,6 +917,11 @@ static void _debug_unregister(debug_info_t *id) debugfs_remove(id->debugfs_entries[i]); } debugfs_remove(id->debugfs_root_entry); +} + +/* Remove from internal list. */ +static void _debug_unregister(debug_info_t *id) +{ if (id == debug_area_first) debug_area_first = id->next; if (id == debug_area_last) @@ -942,6 +947,7 @@ void debug_unregister(debug_info_t *id) mutex_lock(&debug_mutex); _debug_unregister(id); mutex_unlock(&debug_mutex); + _debug_unregister_debugfs(id); debug_info_put(id); } diff --git a/arch/s390/kernel/ima_arch.c b/arch/s390/kernel/ima_arch.c deleted file mode 100644 index f3c3e6e1c5d3..000000000000 --- a/arch/s390/kernel/ima_arch.c +++ /dev/null @@ -1,14 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 - -#include -#include - -bool arch_ima_get_secureboot(void) -{ - return ipl_secure_flag; -} - -const char * const *arch_get_ima_policy(void) -{ - return NULL; -} diff --git a/arch/s390/kernel/ipl.c b/arch/s390/kernel/ipl.c index dcdc7e274848..3c346b02ceb9 100644 --- a/arch/s390/kernel/ipl.c +++ b/arch/s390/kernel/ipl.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -2377,7 +2378,7 @@ void __init setup_ipl(void) atomic_notifier_chain_register(&panic_notifier_list, &on_panic_nb); } -void s390_reset_system(void) +void __no_stack_protector s390_reset_system(void) { /* Disable prefixing */ set_prefix(0); @@ -2387,6 +2388,11 @@ void s390_reset_system(void) diag_amode31_ops.diag308_reset(); } +bool arch_get_secureboot(void) +{ + return ipl_secure_flag; +} + #ifdef CONFIG_KEXEC_FILE int ipl_report_add_component(struct ipl_report *report, struct kexec_buf *kbuf, diff --git a/arch/s390/kernel/irqflags.c b/arch/s390/kernel/irqflags.c new file mode 100644 index 000000000000..e192f59f8918 --- /dev/null +++ b/arch/s390/kernel/irqflags.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include + +noinstr unsigned long arch_local_save_flags(void) +{ + return __arch_local_save_flags(); +} +EXPORT_SYMBOL(arch_local_save_flags); + +noinstr unsigned long arch_local_irq_save(void) +{ + return __arch_local_irq_save(); +} +EXPORT_SYMBOL(arch_local_irq_save); + +noinstr void arch_local_irq_enable_external(void) +{ + __arch_local_irq_enable_external(); +} +EXPORT_SYMBOL(arch_local_irq_enable_external); + +noinstr void arch_local_irq_enable(void) +{ + __arch_local_irq_enable(); +} +EXPORT_SYMBOL(arch_local_irq_enable); diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 8b31d482712f..ed08e2f3a8ec 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -4705,8 +4705,8 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, unsigned long token) { - struct kvm_s390_interrupt inti; - struct kvm_s390_irq irq; + struct kvm_s390_interrupt inti = {}; + struct kvm_s390_irq irq = {}; if (start_token) { irq.u.ext.ext_params2 = token; diff --git a/arch/sh/mm/pmb.c b/arch/sh/mm/pmb.c index 68eb7cc6e564..482eec50f404 100644 --- a/arch/sh/mm/pmb.c +++ b/arch/sh/mm/pmb.c @@ -857,7 +857,7 @@ static int __init pmb_debugfs_init(void) subsys_initcall(pmb_debugfs_init); #ifdef CONFIG_PM -static void pmb_syscore_resume(void) +static void pmb_syscore_resume(void *data) { struct pmb_entry *pmbe; int i; @@ -874,13 +874,17 @@ static void pmb_syscore_resume(void) read_unlock(&pmb_rwlock); } -static struct syscore_ops pmb_syscore_ops = { +static const struct syscore_ops pmb_syscore_ops = { .resume = pmb_syscore_resume, }; +static struct syscore pmb_syscore = { + .ops = &pmb_syscore_ops, +}; + static int __init pmb_sysdev_init(void) { - register_syscore_ops(&pmb_syscore_ops); + register_syscore(&pmb_syscore); return 0; } subsys_initcall(pmb_sysdev_init); diff --git a/arch/x86/entry/entry_64_fred.S b/arch/x86/entry/entry_64_fred.S index fafbd3e68cb8..3b4051960207 100644 --- a/arch/x86/entry/entry_64_fred.S +++ b/arch/x86/entry/entry_64_fred.S @@ -6,6 +6,7 @@ #include #include +#include #include #include @@ -18,6 +19,7 @@ UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR PUSH_AND_CLEAR_REGS + ENCODE_FRAME_POINTER movq %rsp, %rdi /* %rdi -> pt_regs */ .endm diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c index 56918cd91115..23d834e7b565 100644 --- a/arch/x86/events/amd/ibs.c +++ b/arch/x86/events/amd/ibs.c @@ -1719,26 +1719,30 @@ static int x86_pmu_amd_ibs_starting_cpu(unsigned int cpu) #ifdef CONFIG_PM -static int perf_ibs_suspend(void) +static int perf_ibs_suspend(void *data) { clear_APIC_ibs(); return 0; } -static void perf_ibs_resume(void) +static void perf_ibs_resume(void *data) { ibs_eilvt_setup(); setup_APIC_ibs(); } -static struct syscore_ops perf_ibs_syscore_ops = { +static const struct syscore_ops perf_ibs_syscore_ops = { .resume = perf_ibs_resume, .suspend = perf_ibs_suspend, }; +static struct syscore perf_ibs_syscore = { + .ops = &perf_ibs_syscore_ops, +}; + static void perf_ibs_pm_init(void) { - register_syscore_ops(&perf_ibs_syscore_ops); + register_syscore(&perf_ibs_syscore); } #else diff --git a/arch/x86/events/amd/uncore.c b/arch/x86/events/amd/uncore.c index 9a13a9f21d2f..c96048122272 100644 --- a/arch/x86/events/amd/uncore.c +++ b/arch/x86/events/amd/uncore.c @@ -264,6 +264,29 @@ static void amd_uncore_del(struct perf_event *event, int flags) hwc->idx = -1; } +static bool amd_uncore_group_valid(struct perf_event *event) +{ + struct amd_uncore_pmu *pmu = event_to_amd_uncore_pmu(event); + struct perf_event *leader = event->group_leader; + struct perf_event *sibling; + int counters = 0; + + if (leader->pmu == event->pmu) + counters++; + + for_each_sibling_event(sibling, leader) { + if (sibling->pmu == event->pmu && + sibling->state > PERF_EVENT_STATE_OFF) + counters++; + } + + /* + * When pmu->event_init() is called, the event is yet to be linked to + * its leader's sibling list, so it is counted separately + */ + return (counters + 1) <= pmu->num_counters; +} + static int amd_uncore_event_init(struct perf_event *event) { struct amd_uncore_pmu *pmu; @@ -281,6 +304,14 @@ static int amd_uncore_event_init(struct perf_event *event) if (!ctx) return -ENODEV; + /* + * Ensure that all events in a group can be scheduled together so that + * a failure can be reported at perf_event_open() time rather than + * silently at pmu->add() time when no free counter is found + */ + if (event->group_leader != event && !amd_uncore_group_valid(event)) + return -EINVAL; + /* * NB and Last level cache counters (MSRs) are shared across all cores * that share the same NB / Last level cache. On family 16h and below, diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index 7d9f6a58f3f2..024b5ac58489 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -6975,12 +6975,6 @@ __init int intel_pmu_init(void) x86_add_quirk(intel_arch_events_quirk); /* Install first, so it runs last */ - if (version >= 5) { - x86_pmu.intel_cap.anythread_deprecated = edx.split.anythread_deprecated; - if (x86_pmu.intel_cap.anythread_deprecated) - pr_cont(" AnyThread deprecated, "); - } - /* * Many features on and after V6 require dynamic constraint, * e.g., Arch PEBS, ACR. @@ -7778,8 +7772,10 @@ __init int intel_pmu_init(void) &x86_pmu.intel_ctrl); /* AnyThread may be deprecated on arch perfmon v5 or later */ - if (x86_pmu.intel_cap.anythread_deprecated) + if (version >= 5 && edx.split.anythread_deprecated) { x86_pmu.format_attrs = intel_arch_formats_attr; + pr_cont("AnyThread deprecated, "); + } intel_pmu_check_event_constraints(x86_pmu.event_constraints, x86_pmu.cntr_mask64, diff --git a/arch/x86/events/intel/pt.c b/arch/x86/events/intel/pt.c index e8cf29d2b10c..a2937f82ccd4 100644 --- a/arch/x86/events/intel/pt.c +++ b/arch/x86/events/intel/pt.c @@ -501,6 +501,29 @@ static u64 pt_config_filters(struct perf_event *event) return rtit_ctl; } +static void pt_config_enable(struct perf_event *event) +{ + struct pt *pt = this_cpu_ptr(&pt_ctx); + + /* + * Allow resume before starting so as not to overwrite a value set by a + * PMI. + */ + barrier(); + WRITE_ONCE(pt->resume_allowed, 1); + /* Configuration is complete, it is now OK to handle an NMI */ + barrier(); + WRITE_ONCE(pt->handle_nmi, 1); + barrier(); + pt_config_start(event); + barrier(); + /* + * Allow pause after starting so its pt_config_stop() doesn't race with + * pt_config_start(). + */ + WRITE_ONCE(pt->pause_allowed, 1); +} + static void pt_config(struct perf_event *event) { struct pt *pt = this_cpu_ptr(&pt_ctx); @@ -540,23 +563,7 @@ static void pt_config(struct perf_event *event) event->hw.aux_config = reg; - /* - * Allow resume before starting so as not to overwrite a value set by a - * PMI. - */ - barrier(); - WRITE_ONCE(pt->resume_allowed, 1); - /* Configuration is complete, it is now OK to handle an NMI */ - barrier(); - WRITE_ONCE(pt->handle_nmi, 1); - barrier(); - pt_config_start(event); - barrier(); - /* - * Allow pause after starting so its pt_config_stop() doesn't race with - * pt_config_start(). - */ - WRITE_ONCE(pt->pause_allowed, 1); + pt_config_enable(event); } static void pt_config_stop(struct perf_event *event) @@ -1532,12 +1539,14 @@ void intel_pt_interrupt(void) perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0)); - if (!event->hw.state) { + event->hw.state |= PERF_HES_UPTODATE; + + if (!(event->hw.state & PERF_HES_STOPPED)) { int ret; buf = perf_aux_output_begin(&pt->handle, event); if (!buf) { - event->hw.state = PERF_HES_STOPPED; + event->hw.state |= PERF_HES_STOPPED; WRITE_ONCE(pt->resume_allowed, 0); return; } @@ -1553,6 +1562,8 @@ void intel_pt_interrupt(void) pt_config_buffer(buf); pt_config_start(event); + + event->hw.state &= ~PERF_HES_UPTODATE; } } @@ -1622,6 +1633,18 @@ static void pt_event_start(struct perf_event *event, int mode) return; } + /* + * Re-start subsequent to a call to pt_event_stop() without the + * PERF_EF_UPDATE flag. Absence of PERF_HES_UPTODATE indicates that + * perf_aux_output_begin() has already been called. This path can + * come about only in snapshot/overwrite mode - see pt_event_stop(). + */ + if (!(hwc->state & PERF_HES_UPTODATE)) { + hwc->state &= ~PERF_HES_STOPPED; + pt_config_enable(event); + return; + } + buf = perf_aux_output_begin(&pt->handle, event); if (!buf) goto fail_stop; @@ -1632,7 +1655,7 @@ static void pt_event_start(struct perf_event *event, int mode) goto fail_end_stop; } - hwc->state = 0; + hwc->state &= ~(PERF_HES_STOPPED | PERF_HES_UPTODATE); pt_config_buffer(buf); pt_config(event); @@ -1642,12 +1665,13 @@ static void pt_event_start(struct perf_event *event, int mode) fail_end_stop: perf_aux_output_end(&pt->handle, 0); fail_stop: - hwc->state = PERF_HES_STOPPED; + hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE; } static void pt_event_stop(struct perf_event *event, int mode) { struct pt *pt = this_cpu_ptr(&pt_ctx); + struct pt_buffer *buf; if (mode & PERF_EF_PAUSE) { if (READ_ONCE(pt->pause_allowed)) @@ -1673,17 +1697,24 @@ static void pt_event_stop(struct perf_event *event, int mode) pt_config_stop(event); - if (event->hw.state == PERF_HES_STOPPED) - return; - - event->hw.state = PERF_HES_STOPPED; + event->hw.state |= PERF_HES_STOPPED; - if (mode & PERF_EF_UPDATE) { - struct pt_buffer *buf = perf_get_aux(&pt->handle); + if (event->hw.state & PERF_HES_UPTODATE) + return; - if (!buf) - return; + buf = perf_get_aux(&pt->handle); + if (!buf) + return; + /* + * When not in snapshot/overwrite mode, there is a possibility that the + * buffer has run out of space. The accounting for that is handled by + * the update, so always update in that case. Snapshot/overwrite mode is + * treated differently to allow for pt_event_snapshot_aux() which can + * still get called if the AUX-sampling event is not stopped until after + * PT is stopped. + */ + if ((mode & PERF_EF_UPDATE) || !buf->snapshot) { if (WARN_ON_ONCE(pt->handle.event != event)) return; @@ -1698,6 +1729,7 @@ static void pt_event_stop(struct perf_event *event, int mode) local_xchg(&buf->data_size, buf->nr_pages << PAGE_SHIFT); perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0)); + event->hw.state |= PERF_HES_UPTODATE; } } @@ -1768,13 +1800,15 @@ static int pt_event_add(struct perf_event *event, int mode) if (pt->handle.event) goto fail; + event->hw.state |= PERF_HES_UPTODATE; + if (mode & PERF_EF_START) { pt_event_start(event, 0); ret = -EINVAL; - if (hwc->state == PERF_HES_STOPPED) + if (hwc->state & PERF_HES_STOPPED) goto fail; } else { - hwc->state = PERF_HES_STOPPED; + hwc->state |= PERF_HES_STOPPED; } ret = 0; diff --git a/arch/x86/events/intel/uncore.c b/arch/x86/events/intel/uncore.c index 8301a589d9a6..a5793847ed5a 100644 --- a/arch/x86/events/intel/uncore.c +++ b/arch/x86/events/intel/uncore.c @@ -1613,8 +1613,6 @@ static int uncore_event_cpu_online(unsigned int cpu) die = topology_logical_die_id(cpu); msr_ret = uncore_box_ref(uncore_msr_uncores, die, cpu); mmio_ret = uncore_box_ref(uncore_mmio_uncores, die, cpu); - if (msr_ret && mmio_ret) - return -ENOMEM; /* * Check if there is an online cpu in the package diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h index 2bafb8f0f907..da6f5cad665c 100644 --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -656,7 +656,7 @@ union perf_capabilities { u64 perf_metrics:1; u64 pebs_output_pt_available:1; u64 pebs_timing_info:1; - u64 anythread_deprecated:1; + u64 __reserved:1; u64 rdpmc_metrics_clear:1; }; u64 capabilities; diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c index e890fd37e9c2..085ef4f2e73a 100644 --- a/arch/x86/hyperv/hv_init.c +++ b/arch/x86/hyperv/hv_init.c @@ -351,7 +351,7 @@ static int __init hv_pci_init(void) return 1; } -static int hv_suspend(void) +static int hv_suspend(void *data) { union hv_x64_msr_hypercall_contents hypercall_msr; int ret; @@ -378,7 +378,7 @@ static int hv_suspend(void) return ret; } -static void hv_resume(void) +static void hv_resume(void *data) { union hv_x64_msr_hypercall_contents hypercall_msr; int ret; @@ -405,11 +405,15 @@ static void hv_resume(void) } /* Note: when the ops are called, only CPU0 is online and IRQs are disabled. */ -static struct syscore_ops hv_syscore_ops = { +static const struct syscore_ops hv_syscore_ops = { .suspend = hv_suspend, .resume = hv_resume, }; +static struct syscore hv_syscore = { + .ops = &hv_syscore_ops, +}; + static void (* __initdata old_setup_percpu_clockev)(void); static void __init hv_stimer_setup_percpu_clockev(void) @@ -569,7 +573,7 @@ void __init hyperv_init(void) x86_init.pci.arch_init = hv_pci_init; - register_syscore_ops(&hv_syscore_ops); + register_syscore(&hv_syscore); if (ms_hyperv.priv_high & HV_ACCESS_PARTITION_ID) hv_get_partition_id(); diff --git a/arch/x86/include/asm/efi.h b/arch/x86/include/asm/efi.h index f5932705f4b0..be58b7f5c806 100644 --- a/arch/x86/include/asm/efi.h +++ b/arch/x86/include/asm/efi.h @@ -402,9 +402,9 @@ extern int __init efi_memmap_split_count(efi_memory_desc_t *md, extern void __init efi_memmap_insert(struct efi_memory_map *old_memmap, void *buf, struct efi_mem_range *mem); -extern enum efi_secureboot_mode __x86_ima_efi_boot_mode(void); +enum efi_secureboot_mode __x86_efi_boot_mode(void); -#define arch_ima_efi_boot_mode __x86_ima_efi_boot_mode() +#define arch_efi_boot_mode __x86_efi_boot_mode() #ifdef CONFIG_EFI_RUNTIME_MAP int efi_get_runtime_map_size(void); diff --git a/arch/x86/include/asm/linkage.h b/arch/x86/include/asm/linkage.h index a7294656ad90..c9769a7b6e66 100644 --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -103,7 +103,7 @@ .byte 0xb8 ASM_NL \ .long __kcfi_typeid_##name ASM_NL \ CFI_POST_PADDING \ - SYM_FUNC_END(__cfi_##name) + SYM_END(__cfi_##name, SYM_T_FUNC) /* UML needs to be able to override memcpy() and friends for KASAN. */ #ifdef CONFIG_UML diff --git a/arch/x86/include/asm/pkeys.h b/arch/x86/include/asm/pkeys.h index 2e6c04d8a45b..32d327609cc4 100644 --- a/arch/x86/include/asm/pkeys.h +++ b/arch/x86/include/asm/pkeys.h @@ -89,6 +89,9 @@ int mm_pkey_alloc(struct mm_struct *mm) u16 all_pkeys_mask = ((1U << arch_max_pkey()) - 1); int ret; + if (!arch_pkeys_enabled()) + return -1; + /* * Are we out of pkeys? We must handle this specially * because ffz() behavior is undefined if there are no diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 3485d419c2f5..e6e68a31634c 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -591,7 +591,7 @@ static void gart_fixup_northbridges(void) } } -static void gart_resume(void) +static void gart_resume(void *data) { pr_info("PCI-DMA: Resuming GART IOMMU\n"); @@ -600,11 +600,15 @@ static void gart_resume(void) enable_gart_translations(); } -static struct syscore_ops gart_syscore_ops = { +static const struct syscore_ops gart_syscore_ops = { .resume = gart_resume, }; +static struct syscore gart_syscore = { + .ops = &gart_syscore_ops, +}; + /* * Private Northbridge GATT initialization in case we cannot use the * AGP driver for some reason. @@ -650,7 +654,7 @@ static __init int init_amd_gatt(struct agp_kern_info *info) agp_gatt_table = gatt; - register_syscore_ops(&gart_syscore_ops); + register_syscore(&gart_syscore); flush_gart(); diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index aa1b0ef5e931..a4431f0d2580 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -2382,7 +2382,7 @@ static struct { unsigned int apic_cmci; } apic_pm_state; -static int lapic_suspend(void) +static int lapic_suspend(void *data) { unsigned long flags; int maxlvt; @@ -2430,7 +2430,7 @@ static int lapic_suspend(void) return 0; } -static void lapic_resume(void) +static void lapic_resume(void *data) { unsigned int l, h; unsigned long flags; @@ -2510,11 +2510,15 @@ static void lapic_resume(void) * are needed on every CPU up until machine_halt/restart/poweroff. */ -static struct syscore_ops lapic_syscore_ops = { +static const struct syscore_ops lapic_syscore_ops = { .resume = lapic_resume, .suspend = lapic_suspend, }; +static struct syscore lapic_syscore = { + .ops = &lapic_syscore_ops, +}; + static void apic_pm_activate(void) { apic_pm_state.active = 1; @@ -2524,7 +2528,7 @@ static int __init init_lapic_sysfs(void) { /* XXX: remove suspend/resume procs if !apic_pm_state.active? */ if (boot_cpu_has(X86_FEATURE_APIC)) - register_syscore_ops(&lapic_syscore_ops); + register_syscore(&lapic_syscore); return 0; } diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c index 5ba2feb2c04c..84e200662ce6 100644 --- a/arch/x86/kernel/apic/io_apic.c +++ b/arch/x86/kernel/apic/io_apic.c @@ -2308,7 +2308,12 @@ static void resume_ioapic_id(int ioapic_idx) } } -static void ioapic_resume(void) +static int ioapic_suspend(void *data) +{ + return save_ioapic_entries(); +} + +static void ioapic_resume(void *data) { int ioapic_idx; @@ -2318,14 +2323,18 @@ static void ioapic_resume(void) restore_ioapic_entries(); } -static struct syscore_ops ioapic_syscore_ops = { - .suspend = save_ioapic_entries, +static const struct syscore_ops ioapic_syscore_ops = { + .suspend = ioapic_suspend, .resume = ioapic_resume, }; +static struct syscore ioapic_syscore = { + .ops = &ioapic_syscore_ops, +}; + static int __init ioapic_init_ops(void) { - register_syscore_ops(&ioapic_syscore_ops); + register_syscore(&ioapic_syscore); return 0; } diff --git a/arch/x86/kernel/cpu/aperfmperf.c b/arch/x86/kernel/cpu/aperfmperf.c index a315b0627dfb..7ffc78d5ebf2 100644 --- a/arch/x86/kernel/cpu/aperfmperf.c +++ b/arch/x86/kernel/cpu/aperfmperf.c @@ -37,7 +37,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = { .seq = SEQCNT_ZERO(cpu_samples.seq) }; -static void init_counter_refs(void) +static void init_counter_refs(void *data) { u64 aperf, mperf; @@ -289,16 +289,20 @@ static bool __init intel_set_max_freq_ratio(void) } #ifdef CONFIG_PM_SLEEP -static struct syscore_ops freq_invariance_syscore_ops = { +static const struct syscore_ops freq_invariance_syscore_ops = { .resume = init_counter_refs, }; -static void register_freq_invariance_syscore_ops(void) +static struct syscore freq_invariance_syscore = { + .ops = &freq_invariance_syscore_ops, +}; + +static void register_freq_invariance_syscore(void) { - register_syscore_ops(&freq_invariance_syscore_ops); + register_syscore(&freq_invariance_syscore); } #else -static inline void register_freq_invariance_syscore_ops(void) {} +static inline void register_freq_invariance_syscore(void) {} #endif static void freq_invariance_enable(void) @@ -308,7 +312,7 @@ static void freq_invariance_enable(void) return; } static_branch_enable_cpuslocked(&arch_scale_freq_key); - register_freq_invariance_syscore_ops(); + register_freq_invariance_syscore(); pr_info("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n", arch_max_freq_ratio); } @@ -535,7 +539,7 @@ static int __init bp_init_aperfmperf(void) if (!cpu_feature_enabled(X86_FEATURE_APERFMPERF)) return 0; - init_counter_refs(); + init_counter_refs(NULL); bp_init_freq_invariance(); return 0; } @@ -544,5 +548,5 @@ early_initcall(bp_init_aperfmperf); void ap_init_aperfmperf(void) { if (cpu_feature_enabled(X86_FEATURE_APERFMPERF)) - init_counter_refs(); + init_counter_refs(NULL); } diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 6cf7c2d0dc59..3ffeadad0244 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1210,9 +1210,6 @@ static const __initconst struct x86_cpu_id cpu_vuln_whitelist[] = { #define VULNBL_INTEL_STEPS(vfm, max_stepping, issues) \ X86_MATCH_VFM_STEPS(vfm, X86_STEP_MIN, max_stepping, issues) -#define VULNBL_INTEL_TYPE(vfm, cpu_type, issues) \ - X86_MATCH_VFM_CPU_TYPE(vfm, INTEL_CPU_TYPE_##cpu_type, issues) - #define VULNBL_AMD(family, blacklist) \ VULNBL(AMD, family, X86_MODEL_ANY, blacklist) @@ -1275,11 +1272,9 @@ static const struct x86_cpu_id cpu_vuln_blacklist[] __initconst = { VULNBL_INTEL_STEPS(INTEL_TIGERLAKE, X86_STEP_MAX, GDS | ITS | ITS_NATIVE_ONLY), VULNBL_INTEL_STEPS(INTEL_LAKEFIELD, X86_STEP_MAX, MMIO | MMIO_SBDS | RETBLEED), VULNBL_INTEL_STEPS(INTEL_ROCKETLAKE, X86_STEP_MAX, MMIO | RETBLEED | GDS | ITS | ITS_NATIVE_ONLY), - VULNBL_INTEL_TYPE(INTEL_ALDERLAKE, ATOM, RFDS | VMSCAPE), - VULNBL_INTEL_STEPS(INTEL_ALDERLAKE, X86_STEP_MAX, VMSCAPE), + VULNBL_INTEL_STEPS(INTEL_ALDERLAKE, X86_STEP_MAX, RFDS | VMSCAPE), VULNBL_INTEL_STEPS(INTEL_ALDERLAKE_L, X86_STEP_MAX, RFDS | VMSCAPE), - VULNBL_INTEL_TYPE(INTEL_RAPTORLAKE, ATOM, RFDS | VMSCAPE), - VULNBL_INTEL_STEPS(INTEL_RAPTORLAKE, X86_STEP_MAX, VMSCAPE), + VULNBL_INTEL_STEPS(INTEL_RAPTORLAKE, X86_STEP_MAX, RFDS | VMSCAPE), VULNBL_INTEL_STEPS(INTEL_RAPTORLAKE_P, X86_STEP_MAX, RFDS | VMSCAPE), VULNBL_INTEL_STEPS(INTEL_RAPTORLAKE_S, X86_STEP_MAX, RFDS | VMSCAPE), VULNBL_INTEL_STEPS(INTEL_METEORLAKE_L, X86_STEP_MAX, VMSCAPE), @@ -1347,7 +1342,21 @@ static bool __init vulnerable_to_rfds(u64 x86_arch_cap_msr) return true; /* Only consult the blacklist when there is no enumeration: */ - return cpu_matches(cpu_vuln_blacklist, RFDS); + if (!cpu_matches(cpu_vuln_blacklist, RFDS)) + return false; + + /* + * ADL and RPL are affected only if they have Atom CPUs. Hybrids have + * both Core and Atom CPUs. Mark unaffected when Atom CPUs are not + * present. + */ + if ((boot_cpu_data.x86_model == 0x97 || + boot_cpu_data.x86_model == 0xB7) && + boot_cpu_data.topo.intel_type != INTEL_CPU_TYPE_ATOM && + !boot_cpu_has(X86_FEATURE_HYBRID_CPU)) + return false; + + return true; } static bool __init vulnerable_to_its(u64 x86_arch_cap_msr) diff --git a/arch/x86/kernel/cpu/cpu.h b/arch/x86/kernel/cpu/cpu.h index 92032422a829..dca2d5845e42 100644 --- a/arch/x86/kernel/cpu/cpu.h +++ b/arch/x86/kernel/cpu/cpu.h @@ -42,15 +42,6 @@ extern const struct cpu_dev *const __x86_cpu_dev_start[], *const __x86_cpu_dev_end[]; #ifdef CONFIG_CPU_SUP_INTEL -enum tsx_ctrl_states { - TSX_CTRL_ENABLE, - TSX_CTRL_DISABLE, - TSX_CTRL_RTM_ALWAYS_ABORT, - TSX_CTRL_NOT_SUPPORTED, -}; - -extern __ro_after_init enum tsx_ctrl_states tsx_ctrl_state; - extern void __init tsx_init(void); void tsx_ap_init(void); void intel_unlock_cpuid_leafs(struct cpuinfo_x86 *c); diff --git a/arch/x86/kernel/cpu/intel_epb.c b/arch/x86/kernel/cpu/intel_epb.c index bc7671f920a7..2c56f8730f59 100644 --- a/arch/x86/kernel/cpu/intel_epb.c +++ b/arch/x86/kernel/cpu/intel_epb.c @@ -75,7 +75,7 @@ static u8 energ_perf_values[] = { [EPB_INDEX_POWERSAVE] = ENERGY_PERF_BIAS_POWERSAVE, }; -static int intel_epb_save(void) +static int intel_epb_save(void *data) { u64 epb; @@ -89,7 +89,7 @@ static int intel_epb_save(void) return 0; } -static void intel_epb_restore(void) +static void intel_epb_restore(void *data) { u64 val = this_cpu_read(saved_epb); u64 epb; @@ -114,11 +114,15 @@ static void intel_epb_restore(void) wrmsrq(MSR_IA32_ENERGY_PERF_BIAS, (epb & ~EPB_MASK) | val); } -static struct syscore_ops intel_epb_syscore_ops = { +static const struct syscore_ops intel_epb_syscore_ops = { .suspend = intel_epb_save, .resume = intel_epb_restore, }; +static struct syscore intel_epb_syscore = { + .ops = &intel_epb_syscore_ops, +}; + static const char * const energy_perf_strings[] = { [EPB_INDEX_PERFORMANCE] = "performance", [EPB_INDEX_BALANCE_PERFORMANCE] = "balance-performance", @@ -185,7 +189,7 @@ static int intel_epb_online(unsigned int cpu) { struct device *cpu_dev = get_cpu_device(cpu); - intel_epb_restore(); + intel_epb_restore(NULL); if (!cpuhp_tasks_frozen) sysfs_merge_group(&cpu_dev->kobj, &intel_epb_attr_group); @@ -199,7 +203,7 @@ static int intel_epb_offline(unsigned int cpu) if (!cpuhp_tasks_frozen) sysfs_unmerge_group(&cpu_dev->kobj, &intel_epb_attr_group); - intel_epb_save(); + intel_epb_save(NULL); return 0; } @@ -230,7 +234,7 @@ static __init int intel_epb_init(void) if (ret < 0) goto err_out_online; - register_syscore_ops(&intel_epb_syscore_ops); + register_syscore(&intel_epb_syscore); return 0; err_out_online: diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index f83a8bffdb54..877dd7d3bd75 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -2387,13 +2387,13 @@ static void vendor_disable_error_reporting(void) mce_disable_error_reporting(); } -static int mce_syscore_suspend(void) +static int mce_syscore_suspend(void *data) { vendor_disable_error_reporting(); return 0; } -static void mce_syscore_shutdown(void) +static void mce_syscore_shutdown(void *data) { vendor_disable_error_reporting(); } @@ -2403,7 +2403,7 @@ static void mce_syscore_shutdown(void) * Only one CPU is active at this time, the others get re-added later using * CPU hotplug: */ -static void mce_syscore_resume(void) +static void mce_syscore_resume(void *data) { __mcheck_cpu_init_generic(); __mcheck_cpu_init_vendor(raw_cpu_ptr(&cpu_info)); @@ -2411,12 +2411,16 @@ static void mce_syscore_resume(void) cr4_set_bits(X86_CR4_MCE); } -static struct syscore_ops mce_syscore_ops = { +static const struct syscore_ops mce_syscore_ops = { .suspend = mce_syscore_suspend, .shutdown = mce_syscore_shutdown, .resume = mce_syscore_resume, }; +static struct syscore mce_syscore = { + .ops = &mce_syscore_ops, +}; + /* * mce_device: Sysfs support */ @@ -2817,7 +2821,7 @@ static __init int mcheck_init_device(void) if (err < 0) goto err_out_online; - register_syscore_ops(&mce_syscore_ops); + register_syscore(&mce_syscore); return 0; diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c index 539edd6d6dc8..81aa079fad26 100644 --- a/arch/x86/kernel/cpu/microcode/core.c +++ b/arch/x86/kernel/cpu/microcode/core.c @@ -812,8 +812,17 @@ void microcode_bsp_resume(void) reload_early_microcode(cpu); } -static struct syscore_ops mc_syscore_ops = { - .resume = microcode_bsp_resume, +static void microcode_bsp_syscore_resume(void *data) +{ + microcode_bsp_resume(); +} + +static const struct syscore_ops mc_syscore_ops = { + .resume = microcode_bsp_syscore_resume, +}; + +static struct syscore mc_syscore = { + .ops = &mc_syscore_ops, }; static int mc_cpu_online(unsigned int cpu) @@ -892,7 +901,7 @@ static int __init microcode_init(void) } } - register_syscore_ops(&mc_syscore_ops); + register_syscore(&mc_syscore); cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "x86/microcode:online", mc_cpu_online, mc_cpu_down_prep); diff --git a/arch/x86/kernel/cpu/mtrr/legacy.c b/arch/x86/kernel/cpu/mtrr/legacy.c index d25882fcf181..2415ffaaf02c 100644 --- a/arch/x86/kernel/cpu/mtrr/legacy.c +++ b/arch/x86/kernel/cpu/mtrr/legacy.c @@ -41,7 +41,7 @@ struct mtrr_value { static struct mtrr_value *mtrr_value; -static int mtrr_save(void) +static int mtrr_save(void *data) { int i; @@ -56,7 +56,7 @@ static int mtrr_save(void) return 0; } -static void mtrr_restore(void) +static void mtrr_restore(void *data) { int i; @@ -69,11 +69,15 @@ static void mtrr_restore(void) } } -static struct syscore_ops mtrr_syscore_ops = { +static const struct syscore_ops mtrr_syscore_ops = { .suspend = mtrr_save, .resume = mtrr_restore, }; +static struct syscore mtrr_syscore = { + .ops = &mtrr_syscore_ops, +}; + void mtrr_register_syscore(void) { mtrr_value = kcalloc(num_var_ranges, sizeof(*mtrr_value), GFP_KERNEL); @@ -86,5 +90,5 @@ void mtrr_register_syscore(void) * TBD: is there any system with such CPU which supports * suspend/resume? If no, we should remove the code. */ - register_syscore_ops(&mtrr_syscore_ops); + register_syscore(&mtrr_syscore); } diff --git a/arch/x86/kernel/cpu/tsx.c b/arch/x86/kernel/cpu/tsx.c index 49782724a943..8be08ece2214 100644 --- a/arch/x86/kernel/cpu/tsx.c +++ b/arch/x86/kernel/cpu/tsx.c @@ -19,7 +19,14 @@ #undef pr_fmt #define pr_fmt(fmt) "tsx: " fmt -enum tsx_ctrl_states tsx_ctrl_state __ro_after_init = TSX_CTRL_NOT_SUPPORTED; +enum tsx_ctrl_states { + TSX_CTRL_ENABLE, + TSX_CTRL_DISABLE, + TSX_CTRL_RTM_ALWAYS_ABORT, + TSX_CTRL_NOT_SUPPORTED, +}; + +static enum tsx_ctrl_states tsx_ctrl_state __ro_after_init = TSX_CTRL_NOT_SUPPORTED; static void tsx_disable(void) { diff --git a/arch/x86/kernel/cpu/umwait.c b/arch/x86/kernel/cpu/umwait.c index 933fcd7ff250..e4a31c536642 100644 --- a/arch/x86/kernel/cpu/umwait.c +++ b/arch/x86/kernel/cpu/umwait.c @@ -86,15 +86,19 @@ static int umwait_cpu_offline(unsigned int cpu) * trust the firmware nor does it matter if the same value is written * again. */ -static void umwait_syscore_resume(void) +static void umwait_syscore_resume(void *data) { umwait_update_control_msr(NULL); } -static struct syscore_ops umwait_syscore_ops = { +static const struct syscore_ops umwait_syscore_ops = { .resume = umwait_syscore_resume, }; +static struct syscore umwait_syscore = { + .ops = &umwait_syscore_ops, +}; + /* sysfs interface */ /* @@ -226,7 +230,7 @@ static int __init umwait_init(void) return ret; } - register_syscore_ops(&umwait_syscore_ops); + register_syscore(&umwait_syscore); /* * Add umwait control interface. Ignore failure, so at least the diff --git a/arch/x86/kernel/i8237.c b/arch/x86/kernel/i8237.c index 2cd124ad9380..896d46b44284 100644 --- a/arch/x86/kernel/i8237.c +++ b/arch/x86/kernel/i8237.c @@ -19,7 +19,7 @@ * in asm/dma.h. */ -static void i8237A_resume(void) +static void i8237A_resume(void *data) { unsigned long flags; int i; @@ -41,10 +41,14 @@ static void i8237A_resume(void) release_dma_lock(flags); } -static struct syscore_ops i8237_syscore_ops = { +static const struct syscore_ops i8237_syscore_ops = { .resume = i8237A_resume, }; +static struct syscore i8237_syscore = { + .ops = &i8237_syscore_ops, +}; + static int __init i8237A_init_ops(void) { /* @@ -70,7 +74,7 @@ static int __init i8237A_init_ops(void) if (x86_pnpbios_disabled() && dmi_get_bios_year() >= 2017) return -ENODEV; - register_syscore_ops(&i8237_syscore_ops); + register_syscore(&i8237_syscore); return 0; } device_initcall(i8237A_init_ops); diff --git a/arch/x86/kernel/i8259.c b/arch/x86/kernel/i8259.c index 2bade73f49e3..f67063df6723 100644 --- a/arch/x86/kernel/i8259.c +++ b/arch/x86/kernel/i8259.c @@ -247,19 +247,19 @@ static void save_ELCR(char *trigger) trigger[1] = inb(PIC_ELCR2) & 0xDE; } -static void i8259A_resume(void) +static void i8259A_resume(void *data) { init_8259A(i8259A_auto_eoi); restore_ELCR(irq_trigger); } -static int i8259A_suspend(void) +static int i8259A_suspend(void *data) { save_ELCR(irq_trigger); return 0; } -static void i8259A_shutdown(void) +static void i8259A_shutdown(void *data) { /* Put the i8259A into a quiescent state that * the kernel initialization code can get it @@ -269,12 +269,16 @@ static void i8259A_shutdown(void) outb(0xff, PIC_SLAVE_IMR); /* mask all of 8259A-2 */ } -static struct syscore_ops i8259_syscore_ops = { +static const struct syscore_ops i8259_syscore_ops = { .suspend = i8259A_suspend, .resume = i8259A_resume, .shutdown = i8259A_shutdown, }; +static struct syscore i8259_syscore = { + .ops = &i8259_syscore_ops, +}; + static void mask_8259A(void) { unsigned long flags; @@ -444,7 +448,7 @@ EXPORT_SYMBOL(legacy_pic); static int __init i8259A_init_ops(void) { if (legacy_pic == &default_legacy_pic) - register_syscore_ops(&i8259_syscore_ops); + register_syscore(&i8259_syscore); return 0; } diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index b67d7c59dca0..1500852ba03c 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -720,7 +720,7 @@ static int kvm_cpu_down_prepare(unsigned int cpu) #endif -static int kvm_suspend(void) +static int kvm_suspend(void *data) { u64 val = 0; @@ -734,7 +734,7 @@ static int kvm_suspend(void) return 0; } -static void kvm_resume(void) +static void kvm_resume(void *data) { kvm_cpu_online(raw_smp_processor_id()); @@ -744,11 +744,15 @@ static void kvm_resume(void) #endif } -static struct syscore_ops kvm_syscore_ops = { +static const struct syscore_ops kvm_syscore_ops = { .suspend = kvm_suspend, .resume = kvm_resume, }; +static struct syscore kvm_syscore = { + .ops = &kvm_syscore_ops, +}; + static void kvm_pv_guest_cpu_reboot(void *unused) { kvm_guest_cpu_offline(true); @@ -858,7 +862,7 @@ static void __init kvm_guest_init(void) machine_ops.crash_shutdown = kvm_crash_shutdown; #endif - register_syscore_ops(&kvm_syscore_ops); + register_syscore(&kvm_syscore); /* * Hard lockup detection is enabled by default. Disable it, as guests diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index c4b8950c7abe..1f823bddbf95 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -5,8 +5,8 @@ ccflags-$(CONFIG_KVM_WERROR) += -Werror include $(srctree)/virt/kvm/Makefile.kvm -kvm-y += x86.o emulate.o irq.o lapic.o cpuid.o pmu.o mtrr.o \ - debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o +kvm-y += x86.o emulate.o irq.o lapic.o cpuid.o pmu.o regs.o \ + mtrr.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o kvm-$(CONFIG_KVM_IOAPIC) += i8259.o i8254.o ioapic.o diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c new file mode 100644 index 000000000000..7b997c4e6329 --- /dev/null +++ b/arch/x86/kvm/regs.c @@ -0,0 +1,787 @@ +// SPDX-License-Identifier: GPL-2.0-only +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include + +#include "lapic.h" +#include "mmu.h" +#include "x86.h" + +static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + if (vcpu->arch.emulate_regs_need_sync_to_vcpu) { + /* + * We are here if userspace calls get_regs() in the middle of + * instruction emulation. Registers state needs to be copied + * back from emulation context to vcpu. Userspace shouldn't do + * that usually, but some bad designed PV devices (vmware + * backdoor interface) need this to work + */ + emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); + vcpu->arch.emulate_regs_need_sync_to_vcpu = false; + } + regs->rax = kvm_rax_read(vcpu); + regs->rbx = kvm_rbx_read(vcpu); + regs->rcx = kvm_rcx_read(vcpu); + regs->rdx = kvm_rdx_read(vcpu); + regs->rsi = kvm_rsi_read(vcpu); + regs->rdi = kvm_rdi_read(vcpu); + regs->rsp = kvm_rsp_read(vcpu); + regs->rbp = kvm_rbp_read(vcpu); +#ifdef CONFIG_X86_64 + regs->r8 = kvm_r8_read(vcpu); + regs->r9 = kvm_r9_read(vcpu); + regs->r10 = kvm_r10_read(vcpu); + regs->r11 = kvm_r11_read(vcpu); + regs->r12 = kvm_r12_read(vcpu); + regs->r13 = kvm_r13_read(vcpu); + regs->r14 = kvm_r14_read(vcpu); + regs->r15 = kvm_r15_read(vcpu); +#endif + + regs->rip = kvm_rip_read(vcpu); + regs->rflags = kvm_get_rflags(vcpu); +} + +int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __get_regs(vcpu, regs); + vcpu_put(vcpu); + return 0; +} + +static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + vcpu->arch.emulate_regs_need_sync_from_vcpu = true; + vcpu->arch.emulate_regs_need_sync_to_vcpu = false; + + kvm_rax_write(vcpu, regs->rax); + kvm_rbx_write(vcpu, regs->rbx); + kvm_rcx_write(vcpu, regs->rcx); + kvm_rdx_write(vcpu, regs->rdx); + kvm_rsi_write(vcpu, regs->rsi); + kvm_rdi_write(vcpu, regs->rdi); + kvm_rsp_write(vcpu, regs->rsp); + kvm_rbp_write(vcpu, regs->rbp); +#ifdef CONFIG_X86_64 + kvm_r8_write(vcpu, regs->r8); + kvm_r9_write(vcpu, regs->r9); + kvm_r10_write(vcpu, regs->r10); + kvm_r11_write(vcpu, regs->r11); + kvm_r12_write(vcpu, regs->r12); + kvm_r13_write(vcpu, regs->r13); + kvm_r14_write(vcpu, regs->r14); + kvm_r15_write(vcpu, regs->r15); +#endif + + kvm_rip_write(vcpu, regs->rip); + kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED); + + vcpu->arch.exception.pending = false; + vcpu->arch.exception_vmexit.pending = false; + + kvm_make_request(KVM_REQ_EVENT, vcpu); +} + +int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __set_regs(vcpu, regs); + vcpu_put(vcpu); + return 0; +} + +static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); +} + +/* + * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise. + */ +int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) +{ + struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT; + gpa_t real_gpa; + int i; + int ret; + u64 pdpte[ARRAY_SIZE(mmu->pdptrs)]; + + /* + * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated + * to an L1 GPA. + */ + real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn), + PFERR_USER_MASK | PFERR_WRITE_MASK, NULL); + if (real_gpa == INVALID_GPA) + return 0; + + /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */ + ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte, + cr3 & GENMASK(11, 5), sizeof(pdpte)); + if (ret < 0) + return 0; + + for (i = 0; i < ARRAY_SIZE(pdpte); ++i) { + if ((pdpte[i] & PT_PRESENT_MASK) && + (pdpte[i] & pdptr_rsvd_bits(vcpu))) { + return 0; + } + } + + /* + * Marking VCPU_EXREG_PDPTR dirty doesn't work for !tdp_enabled. + * Shadow page roots need to be reconstructed instead. + */ + if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) + kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); + + memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)); + kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); + kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); + vcpu->arch.pdptrs_from_userspace = false; + + return 1; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); + +static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) +{ +#ifdef CONFIG_X86_64 + if (cr0 & 0xffffffff00000000UL) + return false; +#endif + + if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD)) + return false; + + if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE)) + return false; + + return kvm_x86_call(is_valid_cr0)(vcpu, cr0); +} + +int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) +{ + unsigned long old_cr0 = kvm_read_cr0(vcpu); + + if (!kvm_is_valid_cr0(vcpu, cr0)) + return 1; + + cr0 |= X86_CR0_ET; + + /* Write to CR0 reserved bits are ignored, even on Intel. */ + cr0 &= ~CR0_RESERVED_BITS; + +#ifdef CONFIG_X86_64 + if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) && + (cr0 & X86_CR0_PG)) { + int cs_db, cs_l; + + if (!is_pae(vcpu)) + return 1; + kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); + if (cs_l) + return 1; + } +#endif + if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) && + is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) && + !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) + return 1; + + if (!(cr0 & X86_CR0_PG) && + (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))) + return 1; + + if (!(cr0 & X86_CR0_WP) && kvm_is_cr4_bit_set(vcpu, X86_CR4_CET)) + return 1; + + kvm_x86_call(set_cr0)(vcpu, cr0); + + kvm_post_set_cr0(vcpu, old_cr0, cr0); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr0); + +void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) +{ + (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw); + +int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) +{ + bool skip_tlb_flush = false; + unsigned long pcid = 0; +#ifdef CONFIG_X86_64 + if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) { + skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH; + cr3 &= ~X86_CR3_PCID_NOFLUSH; + pcid = cr3 & X86_CR3_PCID_MASK; + } +#endif + + /* PDPTRs are always reloaded for PAE paging. */ + if (cr3 == kvm_read_cr3(vcpu) && !is_pae_paging(vcpu)) + goto handle_tlb_flush; + + /* + * Do not condition the GPA check on long mode, this helper is used to + * stuff CR3, e.g. for RSM emulation, and there is no guarantee that + * the current vCPU mode is accurate. + */ + if (!kvm_vcpu_is_legal_cr3(vcpu, cr3)) + return 1; + + if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) + return 1; + + if (cr3 != kvm_read_cr3(vcpu)) + kvm_mmu_new_pgd(vcpu, cr3); + + vcpu->arch.cr3 = cr3; + kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + /* Do not call post_set_cr3, we do not get here for confidential guests. */ + +handle_tlb_flush: + /* + * A load of CR3 that flushes the TLB flushes only the current PCID, + * even if PCID is disabled, in which case PCID=0 is flushed. It's a + * moot point in the end because _disabling_ PCID will flush all PCIDs, + * and it's impossible to use a non-zero PCID when PCID is disabled, + * i.e. only PCID=0 can be relevant. + */ + if (!skip_tlb_flush) + kvm_invalidate_pcid(vcpu, pcid); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr3); + +static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + return __kvm_is_valid_cr4(vcpu, cr4) && + kvm_x86_call(is_valid_cr4)(vcpu, cr4); +} + +void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4) +{ + if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) + kvm_mmu_reset_context(vcpu); + + /* + * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB + * according to the SDM; however, stale prev_roots could be reused + * incorrectly in the future after a MOV to CR3 with NOFLUSH=1, so we + * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST + * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed, + * so fall through. + */ + if (!tdp_enabled && + (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) + kvm_mmu_unload(vcpu); + + /* + * The TLB has to be flushed for all PCIDs if any of the following + * (architecturally required) changes happen: + * - CR4.PCIDE is changed from 1 to 0 + * - CR4.PGE is toggled + * + * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT. + */ + if (((cr4 ^ old_cr4) & X86_CR4_PGE) || + (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE))) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + + /* + * The TLB has to be flushed for the current PCID if any of the + * following (architecturally required) changes happen: + * - CR4.SMEP is changed from 0 to 1 + * - CR4.PAE is toggled + */ + else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || + ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) + kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); + +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr4); + +int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + unsigned long old_cr4 = kvm_read_cr4(vcpu); + + if (!kvm_is_valid_cr4(vcpu, cr4)) + return 1; + + if (is_long_mode(vcpu)) { + if (!(cr4 & X86_CR4_PAE)) + return 1; + if ((cr4 ^ old_cr4) & X86_CR4_LA57) + return 1; + } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE) + && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS) + && !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) + return 1; + + if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) { + /* PCID can not be enabled when cr3[11:0]!=000H or EFER.LMA=0 */ + if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu)) + return 1; + } + + if ((cr4 & X86_CR4_CET) && !kvm_is_cr0_bit_set(vcpu, X86_CR0_WP)) + return 1; + + kvm_x86_call(set_cr4)(vcpu, cr4); + + kvm_post_set_cr4(vcpu, old_cr4, cr4); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr4); + +int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8) +{ + if (cr8 & CR8_RESERVED_BITS) + return 1; + if (lapic_in_kernel(vcpu)) + kvm_lapic_set_tpr(vcpu, cr8); + else + vcpu->arch.cr8 = cr8; + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr8); + +unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu) +{ + if (lapic_in_kernel(vcpu)) + return kvm_lapic_get_cr8(vcpu); + else + return vcpu->arch.cr8; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_cr8); + +static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + struct desc_ptr dt; + + if (vcpu->arch.guest_state_protected) + goto skip_protected_regs; + + kvm_handle_exception_payload_quirk(vcpu); + + kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS); + kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS); + kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES); + kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS); + kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS); + kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS); + + kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR); + kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); + + kvm_x86_call(get_idt)(vcpu, &dt); + sregs->idt.limit = dt.size; + sregs->idt.base = dt.address; + kvm_x86_call(get_gdt)(vcpu, &dt); + sregs->gdt.limit = dt.size; + sregs->gdt.base = dt.address; + + sregs->cr2 = vcpu->arch.cr2; + sregs->cr3 = kvm_read_cr3(vcpu); + +skip_protected_regs: + sregs->cr0 = kvm_read_cr0(vcpu); + sregs->cr4 = kvm_read_cr4(vcpu); + sregs->cr8 = kvm_get_cr8(vcpu); + sregs->efer = vcpu->arch.efer; + sregs->apic_base = vcpu->arch.apic_base; +} + +static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + __get_sregs_common(vcpu, sregs); + + if (vcpu->arch.guest_state_protected) + return; + + if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft) + set_bit(vcpu->arch.interrupt.nr, + (unsigned long *)sregs->interrupt_bitmap); +} + +int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, + struct kvm_sregs *sregs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __get_sregs(vcpu, sregs); + vcpu_put(vcpu); + return 0; +} + +void kvm_vcpu_ioctl_x86_get_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2) +{ + int i; + + __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2); + + if (vcpu->arch.guest_state_protected) + return; + + if (is_pae_paging(vcpu)) { + kvm_vcpu_srcu_read_lock(vcpu); + for (i = 0 ; i < 4 ; i++) + sregs2->pdptrs[i] = kvm_pdptr_read(vcpu, i); + sregs2->flags |= KVM_SREGS2_FLAGS_PDPTRS_VALID; + kvm_vcpu_srcu_read_unlock(vcpu); + } +} + +static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) { + /* + * When EFER.LME and CR0.PG are set, the processor is in + * 64-bit mode (though maybe in a 32-bit code segment). + * CR4.PAE and EFER.LMA must be set. + */ + if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA)) + return false; + if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3)) + return false; + } else { + /* + * Not in 64-bit mode: EFER.LMA is clear and the code + * segment cannot be 64-bit. + */ + if (sregs->efer & EFER_LMA || sregs->cs.l) + return false; + } + + return kvm_is_valid_cr4(vcpu, sregs->cr4) && + kvm_is_valid_cr0(vcpu, sregs->cr0) && + kvm_valid_efer(vcpu, sregs->efer); +} + +static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs, + int *mmu_reset_needed, bool update_pdptrs) +{ + int idx; + struct desc_ptr dt; + + if (!kvm_is_valid_sregs(vcpu, sregs)) + return -EINVAL; + + if (kvm_apic_set_base(vcpu, sregs->apic_base, true)) + return -EINVAL; + + if (vcpu->arch.guest_state_protected) + return 0; + + dt.size = sregs->idt.limit; + dt.address = sregs->idt.base; + kvm_x86_call(set_idt)(vcpu, &dt); + dt.size = sregs->gdt.limit; + dt.address = sregs->gdt.base; + kvm_x86_call(set_gdt)(vcpu, &dt); + + vcpu->arch.cr2 = sregs->cr2; + *mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3; + vcpu->arch.cr3 = sregs->cr3; + kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); + kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); + + *mmu_reset_needed |= vcpu->arch.efer != sregs->efer; + kvm_x86_call(set_efer)(vcpu, sregs->efer); + + *mmu_reset_needed |= kvm_read_cr0(vcpu) != sregs->cr0; + kvm_x86_call(set_cr0)(vcpu, sregs->cr0); + + *mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4; + kvm_x86_call(set_cr4)(vcpu, sregs->cr4); + + if (update_pdptrs) { + idx = srcu_read_lock(&vcpu->kvm->srcu); + if (is_pae_paging(vcpu)) { + load_pdptrs(vcpu, kvm_read_cr3(vcpu)); + *mmu_reset_needed = 1; + } + srcu_read_unlock(&vcpu->kvm->srcu, idx); + } + + kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS); + kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS); + kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES); + kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS); + kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS); + kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS); + + kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); + kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); + + kvm_set_cr8(vcpu, sregs->cr8); + + /* Older userspace won't unhalt the vcpu on reset. */ + if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 && + sregs->cs.selector == 0xf000 && sregs->cs.base == 0xffff0000 && + !is_protmode(vcpu)) + kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); + + return 0; +} + +static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + int pending_vec, max_bits; + int mmu_reset_needed = 0; + int ret = __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true); + + if (ret) + return ret; + + if (mmu_reset_needed) { + kvm_mmu_reset_context(vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } + + max_bits = KVM_NR_INTERRUPTS; + pending_vec = find_first_bit( + (const unsigned long *)sregs->interrupt_bitmap, max_bits); + + if (pending_vec < max_bits) { + kvm_queue_interrupt(vcpu, pending_vec, false); + pr_debug("Set back pending irq %d\n", pending_vec); + kvm_make_request(KVM_REQ_EVENT, vcpu); + } + return 0; +} + +int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, + struct kvm_sregs *sregs) +{ + int ret; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + ret = __set_sregs(vcpu, sregs); + vcpu_put(vcpu); + return ret; +} + +int kvm_vcpu_ioctl_x86_set_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2) +{ + int mmu_reset_needed = 0; + bool valid_pdptrs = sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID; + bool pae = (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) && + !(sregs2->efer & EFER_LMA); + int i, ret; + + if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID) + return -EINVAL; + + if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected)) + return -EINVAL; + + ret = __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, + &mmu_reset_needed, !valid_pdptrs); + if (ret) + return ret; + + if (valid_pdptrs) { + for (i = 0; i < 4 ; i++) + kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); + + kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); + mmu_reset_needed = 1; + vcpu->arch.pdptrs_from_userspace = true; + } + if (mmu_reset_needed) { + kvm_mmu_reset_context(vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } + return 0; +} + +void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu) +{ + BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES); + + if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS) + __get_regs(vcpu, &vcpu->run->s.regs.regs); + + if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS) + __get_sregs(vcpu, &vcpu->run->s.regs.sregs); +} + +int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu) +{ + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) { + __set_regs(vcpu, &vcpu->run->s.regs.regs); + vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_REGS; + } + + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) { + struct kvm_sregs sregs = vcpu->run->s.regs.sregs; + + if (__set_sregs(vcpu, &sregs)) + return -EINVAL; + + vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_SREGS; + } + + return 0; +} + +void kvm_update_dr0123(struct kvm_vcpu *vcpu) +{ + int i; + + if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) { + for (i = 0; i < KVM_NR_DB_REGS; i++) + vcpu->arch.eff_db[i] = vcpu->arch.db[i]; + } +} + +void kvm_update_dr7(struct kvm_vcpu *vcpu) +{ + unsigned long dr7; + + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) + dr7 = vcpu->arch.guest_debug_dr7; + else + dr7 = vcpu->arch.dr7; + kvm_x86_call(set_dr7)(vcpu, dr7); + vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_BP_ENABLED; + if (dr7 & DR7_BP_EN_MASK) + vcpu->arch.switch_db_regs |= KVM_DEBUGREG_BP_ENABLED; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_update_dr7); + +static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu) +{ + u64 fixed = DR6_FIXED_1; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_RTM)) + fixed |= DR6_RTM; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT)) + fixed |= DR6_BUS_LOCK; + return fixed; +} + +int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val) +{ + size_t size = ARRAY_SIZE(vcpu->arch.db); + + switch (dr) { + case 0 ... 3: + vcpu->arch.db[array_index_nospec(dr, size)] = val; + if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) + vcpu->arch.eff_db[dr] = val; + break; + case 4: + case 6: + if (!kvm_dr6_valid(val)) + return 1; /* #GP */ + vcpu->arch.dr6 = (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu); + break; + case 5: + default: /* 7 */ + if (!kvm_dr7_valid(val)) + return 1; /* #GP */ + vcpu->arch.dr7 = (val & DR7_VOLATILE) | DR7_FIXED_1; + kvm_update_dr7(vcpu); + break; + } + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_dr); + +unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr) +{ + size_t size = ARRAY_SIZE(vcpu->arch.db); + + switch (dr) { + case 0 ... 3: + return vcpu->arch.db[array_index_nospec(dr, size)]; + case 4: + case 6: + return vcpu->arch.dr6; + case 5: + default: /* 7 */ + return vcpu->arch.dr7; + } +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr); + +int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs) +{ + unsigned int i; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + kvm_handle_exception_payload_quirk(vcpu); + + memset(dbgregs, 0, sizeof(*dbgregs)); + + BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) != ARRAY_SIZE(dbgregs->db)); + for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) + dbgregs->db[i] = vcpu->arch.db[i]; + + dbgregs->dr6 = vcpu->arch.dr6; + dbgregs->dr7 = vcpu->arch.dr7; + return 0; +} + +int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs) +{ + unsigned int i; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + if (dbgregs->flags) + return -EINVAL; + + if (!kvm_dr6_valid(dbgregs->dr6)) + return -EINVAL; + if (!kvm_dr7_valid(dbgregs->dr7)) + return -EINVAL; + + for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) + vcpu->arch.db[i] = dbgregs->db[i]; + + kvm_update_dr0123(vcpu); + vcpu->arch.dr6 = dbgregs->dr6; + vcpu->arch.dr7 = dbgregs->dr7; + kvm_update_dr7(vcpu); + + return 0; +} diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index a5a77dd0c77b..f08a34d3ff3d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -106,16 +106,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_host); #define emul_to_vcpu(ctxt) \ ((struct kvm_vcpu *)(ctxt)->vcpu) -/* EFER defaults: - * - enable syscall per default because its emulated by KVM - * - enable LME and LMA per default on 64 bit KVM - */ -#ifdef CONFIG_X86_64 -static -u64 __read_mostly efer_reserved_bits = ~((u64)(EFER_SCE | EFER_LME | EFER_LMA)); -#else +/* Enable syscall by default because its emulated by KVM */ static u64 __read_mostly efer_reserved_bits = ~((u64)EFER_SCE); -#endif #define KVM_EXIT_HYPERCALL_VALID_MASK (1 << KVM_HC_MAP_GPA_RANGE) @@ -132,9 +124,6 @@ static void store_regs(struct kvm_vcpu *vcpu); static int sync_regs(struct kvm_vcpu *vcpu); static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu); -static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); -static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); - static DEFINE_MUTEX(vendor_module_lock); static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu); static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu); @@ -1044,77 +1033,6 @@ static bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu) return (vcpu->arch.apf.msr_en_val & mask) == mask; } -static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); -} - -/* - * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise. - */ -int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) -{ - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; - gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT; - gpa_t real_gpa; - int i; - int ret; - u64 pdpte[ARRAY_SIZE(mmu->pdptrs)]; - - /* - * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated - * to an L1 GPA. - */ - real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn), - PFERR_USER_MASK | PFERR_WRITE_MASK, NULL); - if (real_gpa == INVALID_GPA) - return 0; - - /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */ - ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte, - cr3 & GENMASK(11, 5), sizeof(pdpte)); - if (ret < 0) - return 0; - - for (i = 0; i < ARRAY_SIZE(pdpte); ++i) { - if ((pdpte[i] & PT_PRESENT_MASK) && - (pdpte[i] & pdptr_rsvd_bits(vcpu))) { - return 0; - } - } - - /* - * Marking VCPU_EXREG_PDPTR dirty doesn't work for !tdp_enabled. - * Shadow page roots need to be reconstructed instead. - */ - if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) - kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); - - memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)); - kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); - kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); - vcpu->arch.pdptrs_from_userspace = false; - - return 1; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); - -static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) -{ -#ifdef CONFIG_X86_64 - if (cr0 & 0xffffffff00000000UL) - return false; -#endif - - if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD)) - return false; - - if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE)) - return false; - - return kvm_x86_call(is_valid_cr0)(vcpu, cr0); -} - void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0) { /* @@ -1157,56 +1075,6 @@ void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned lon } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr0); -int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) -{ - unsigned long old_cr0 = kvm_read_cr0(vcpu); - - if (!kvm_is_valid_cr0(vcpu, cr0)) - return 1; - - cr0 |= X86_CR0_ET; - - /* Write to CR0 reserved bits are ignored, even on Intel. */ - cr0 &= ~CR0_RESERVED_BITS; - -#ifdef CONFIG_X86_64 - if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) && - (cr0 & X86_CR0_PG)) { - int cs_db, cs_l; - - if (!is_pae(vcpu)) - return 1; - kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); - if (cs_l) - return 1; - } -#endif - if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) && - is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) && - !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) - return 1; - - if (!(cr0 & X86_CR0_PG) && - (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))) - return 1; - - if (!(cr0 & X86_CR0_WP) && kvm_is_cr4_bit_set(vcpu, X86_CR4_CET)) - return 1; - - kvm_x86_call(set_cr0)(vcpu, cr0); - - kvm_post_set_cr0(vcpu, old_cr0, cr0); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr0); - -void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) -{ - (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw); - void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu) { if (vcpu->arch.guest_state_protected) @@ -1322,89 +1190,7 @@ int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_xsetbv); -static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) -{ - return __kvm_is_valid_cr4(vcpu, cr4) && - kvm_x86_call(is_valid_cr4)(vcpu, cr4); -} - -void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4) -{ - if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) - kvm_mmu_reset_context(vcpu); - - /* - * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB - * according to the SDM; however, stale prev_roots could be reused - * incorrectly in the future after a MOV to CR3 with NOFLUSH=1, so we - * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST - * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed, - * so fall through. - */ - if (!tdp_enabled && - (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) - kvm_mmu_unload(vcpu); - - /* - * The TLB has to be flushed for all PCIDs if any of the following - * (architecturally required) changes happen: - * - CR4.PCIDE is changed from 1 to 0 - * - CR4.PGE is toggled - * - * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT. - */ - if (((cr4 ^ old_cr4) & X86_CR4_PGE) || - (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE))) - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - - /* - * The TLB has to be flushed for the current PCID if any of the - * following (architecturally required) changes happen: - * - CR4.SMEP is changed from 0 to 1 - * - CR4.PAE is toggled - */ - else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || - ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) - kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); - -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr4); - -int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) -{ - unsigned long old_cr4 = kvm_read_cr4(vcpu); - - if (!kvm_is_valid_cr4(vcpu, cr4)) - return 1; - - if (is_long_mode(vcpu)) { - if (!(cr4 & X86_CR4_PAE)) - return 1; - if ((cr4 ^ old_cr4) & X86_CR4_LA57) - return 1; - } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE) - && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS) - && !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) - return 1; - - if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) { - /* PCID can not be enabled when cr3[11:0]!=000H or EFER.LMA=0 */ - if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu)) - return 1; - } - - if ((cr4 & X86_CR4_CET) && !kvm_is_cr0_bit_set(vcpu, X86_CR0_WP)) - return 1; - - kvm_x86_call(set_cr4)(vcpu, cr4); - - kvm_post_set_cr4(vcpu, old_cr4, cr4); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr4); - -static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) +void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) { struct kvm_mmu *mmu = vcpu->arch.mmu; unsigned long roots_to_free = 0; @@ -1447,159 +1233,6 @@ static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free); } -int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) -{ - bool skip_tlb_flush = false; - unsigned long pcid = 0; -#ifdef CONFIG_X86_64 - if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) { - skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH; - cr3 &= ~X86_CR3_PCID_NOFLUSH; - pcid = cr3 & X86_CR3_PCID_MASK; - } -#endif - - /* PDPTRs are always reloaded for PAE paging. */ - if (cr3 == kvm_read_cr3(vcpu) && !is_pae_paging(vcpu)) - goto handle_tlb_flush; - - /* - * Do not condition the GPA check on long mode, this helper is used to - * stuff CR3, e.g. for RSM emulation, and there is no guarantee that - * the current vCPU mode is accurate. - */ - if (!kvm_vcpu_is_legal_cr3(vcpu, cr3)) - return 1; - - if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) - return 1; - - if (cr3 != kvm_read_cr3(vcpu)) - kvm_mmu_new_pgd(vcpu, cr3); - - vcpu->arch.cr3 = cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); - /* Do not call post_set_cr3, we do not get here for confidential guests. */ - -handle_tlb_flush: - /* - * A load of CR3 that flushes the TLB flushes only the current PCID, - * even if PCID is disabled, in which case PCID=0 is flushed. It's a - * moot point in the end because _disabling_ PCID will flush all PCIDs, - * and it's impossible to use a non-zero PCID when PCID is disabled, - * i.e. only PCID=0 can be relevant. - */ - if (!skip_tlb_flush) - kvm_invalidate_pcid(vcpu, pcid); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr3); - -int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8) -{ - if (cr8 & CR8_RESERVED_BITS) - return 1; - if (lapic_in_kernel(vcpu)) - kvm_lapic_set_tpr(vcpu, cr8); - else - vcpu->arch.cr8 = cr8; - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr8); - -unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu) -{ - if (lapic_in_kernel(vcpu)) - return kvm_lapic_get_cr8(vcpu); - else - return vcpu->arch.cr8; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_cr8); - -static void kvm_update_dr0123(struct kvm_vcpu *vcpu) -{ - int i; - - if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) { - for (i = 0; i < KVM_NR_DB_REGS; i++) - vcpu->arch.eff_db[i] = vcpu->arch.db[i]; - } -} - -void kvm_update_dr7(struct kvm_vcpu *vcpu) -{ - unsigned long dr7; - - if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) - dr7 = vcpu->arch.guest_debug_dr7; - else - dr7 = vcpu->arch.dr7; - kvm_x86_call(set_dr7)(vcpu, dr7); - vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_BP_ENABLED; - if (dr7 & DR7_BP_EN_MASK) - vcpu->arch.switch_db_regs |= KVM_DEBUGREG_BP_ENABLED; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_update_dr7); - -static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu) -{ - u64 fixed = DR6_FIXED_1; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_RTM)) - fixed |= DR6_RTM; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT)) - fixed |= DR6_BUS_LOCK; - return fixed; -} - -int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val) -{ - size_t size = ARRAY_SIZE(vcpu->arch.db); - - switch (dr) { - case 0 ... 3: - vcpu->arch.db[array_index_nospec(dr, size)] = val; - if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) - vcpu->arch.eff_db[dr] = val; - break; - case 4: - case 6: - if (!kvm_dr6_valid(val)) - return 1; /* #GP */ - vcpu->arch.dr6 = (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu); - break; - case 5: - default: /* 7 */ - if (!kvm_dr7_valid(val)) - return 1; /* #GP */ - vcpu->arch.dr7 = (val & DR7_VOLATILE) | DR7_FIXED_1; - kvm_update_dr7(vcpu); - break; - } - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_dr); - -unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr) -{ - size_t size = ARRAY_SIZE(vcpu->arch.db); - - switch (dr) { - case 0 ... 3: - return vcpu->arch.db[array_index_nospec(dr, size)]; - case 4: - case 6: - return vcpu->arch.dr6; - case 5: - default: /* 7 */ - return vcpu->arch.dr7; - } -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr); - int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) { u32 pmc = kvm_rcx_read(vcpu); @@ -5570,7 +5203,7 @@ static struct kvm_queued_exception *kvm_get_exception_to_save(struct kvm_vcpu *v return &vcpu->arch.exception; } -static void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu) +void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu) { struct kvm_queued_exception *ex = kvm_get_exception_to_save(vcpu); @@ -5774,56 +5407,6 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu, return 0; } -static int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, - struct kvm_debugregs *dbgregs) -{ - unsigned int i; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - kvm_handle_exception_payload_quirk(vcpu); - - memset(dbgregs, 0, sizeof(*dbgregs)); - - BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) != ARRAY_SIZE(dbgregs->db)); - for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) - dbgregs->db[i] = vcpu->arch.db[i]; - - dbgregs->dr6 = vcpu->arch.dr6; - dbgregs->dr7 = vcpu->arch.dr7; - return 0; -} - -static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, - struct kvm_debugregs *dbgregs) -{ - unsigned int i; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - if (dbgregs->flags) - return -EINVAL; - - if (!kvm_dr6_valid(dbgregs->dr6)) - return -EINVAL; - if (!kvm_dr7_valid(dbgregs->dr7)) - return -EINVAL; - - for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) - vcpu->arch.db[i] = dbgregs->db[i]; - - kvm_update_dr0123(vcpu); - vcpu->arch.dr6 = dbgregs->dr6; - vcpu->arch.dr7 = dbgregs->dr7; - kvm_update_dr7(vcpu); - - return 0; -} - static int kvm_vcpu_ioctl_x86_get_xsave2(struct kvm_vcpu *vcpu, u8 *state, unsigned int size) @@ -6661,7 +6244,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, r = -ENOMEM; if (!u.sregs2) goto out; - __get_sregs2(vcpu, u.sregs2); + kvm_vcpu_ioctl_x86_get_sregs2(vcpu, u.sregs2); r = -EFAULT; if (copy_to_user(argp, u.sregs2, sizeof(struct kvm_sregs2))) goto out; @@ -6680,7 +6263,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, u.sregs2 = NULL; goto out; } - r = __set_sregs2(vcpu, u.sregs2); + r = kvm_vcpu_ioctl_x86_set_sregs2(vcpu, u.sregs2); break; } case KVM_HAS_DEVICE_ATTR: @@ -10157,6 +9740,9 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) if (r != 0) goto out_mmu_exit; + if (kvm_cpu_cap_has(X86_FEATURE_LM)) + kvm_enable_efer_bits(EFER_LME | EFER_LMA); + enable_device_posted_irqs &= enable_apicv && irq_remapping_cap(IRQ_POSTING_CAP); @@ -12020,179 +11606,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) return r; } -static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - if (vcpu->arch.emulate_regs_need_sync_to_vcpu) { - /* - * We are here if userspace calls get_regs() in the middle of - * instruction emulation. Registers state needs to be copied - * back from emulation context to vcpu. Userspace shouldn't do - * that usually, but some bad designed PV devices (vmware - * backdoor interface) need this to work - */ - emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); - vcpu->arch.emulate_regs_need_sync_to_vcpu = false; - } - regs->rax = kvm_rax_read(vcpu); - regs->rbx = kvm_rbx_read(vcpu); - regs->rcx = kvm_rcx_read(vcpu); - regs->rdx = kvm_rdx_read(vcpu); - regs->rsi = kvm_rsi_read(vcpu); - regs->rdi = kvm_rdi_read(vcpu); - regs->rsp = kvm_rsp_read(vcpu); - regs->rbp = kvm_rbp_read(vcpu); -#ifdef CONFIG_X86_64 - regs->r8 = kvm_r8_read(vcpu); - regs->r9 = kvm_r9_read(vcpu); - regs->r10 = kvm_r10_read(vcpu); - regs->r11 = kvm_r11_read(vcpu); - regs->r12 = kvm_r12_read(vcpu); - regs->r13 = kvm_r13_read(vcpu); - regs->r14 = kvm_r14_read(vcpu); - regs->r15 = kvm_r15_read(vcpu); -#endif - - regs->rip = kvm_rip_read(vcpu); - regs->rflags = kvm_get_rflags(vcpu); -} - -int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __get_regs(vcpu, regs); - vcpu_put(vcpu); - return 0; -} - -static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - vcpu->arch.emulate_regs_need_sync_from_vcpu = true; - vcpu->arch.emulate_regs_need_sync_to_vcpu = false; - - kvm_rax_write(vcpu, regs->rax); - kvm_rbx_write(vcpu, regs->rbx); - kvm_rcx_write(vcpu, regs->rcx); - kvm_rdx_write(vcpu, regs->rdx); - kvm_rsi_write(vcpu, regs->rsi); - kvm_rdi_write(vcpu, regs->rdi); - kvm_rsp_write(vcpu, regs->rsp); - kvm_rbp_write(vcpu, regs->rbp); -#ifdef CONFIG_X86_64 - kvm_r8_write(vcpu, regs->r8); - kvm_r9_write(vcpu, regs->r9); - kvm_r10_write(vcpu, regs->r10); - kvm_r11_write(vcpu, regs->r11); - kvm_r12_write(vcpu, regs->r12); - kvm_r13_write(vcpu, regs->r13); - kvm_r14_write(vcpu, regs->r14); - kvm_r15_write(vcpu, regs->r15); -#endif - - kvm_rip_write(vcpu, regs->rip); - kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED); - - vcpu->arch.exception.pending = false; - vcpu->arch.exception_vmexit.pending = false; - - kvm_make_request(KVM_REQ_EVENT, vcpu); -} - -int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __set_regs(vcpu, regs); - vcpu_put(vcpu); - return 0; -} - -static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - struct desc_ptr dt; - - if (vcpu->arch.guest_state_protected) - goto skip_protected_regs; - - kvm_handle_exception_payload_quirk(vcpu); - - kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS); - kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS); - kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES); - kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS); - kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS); - kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS); - - kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR); - kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); - - kvm_x86_call(get_idt)(vcpu, &dt); - sregs->idt.limit = dt.size; - sregs->idt.base = dt.address; - kvm_x86_call(get_gdt)(vcpu, &dt); - sregs->gdt.limit = dt.size; - sregs->gdt.base = dt.address; - - sregs->cr2 = vcpu->arch.cr2; - sregs->cr3 = kvm_read_cr3(vcpu); - -skip_protected_regs: - sregs->cr0 = kvm_read_cr0(vcpu); - sregs->cr4 = kvm_read_cr4(vcpu); - sregs->cr8 = kvm_get_cr8(vcpu); - sregs->efer = vcpu->arch.efer; - sregs->apic_base = vcpu->arch.apic_base; -} - -static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - __get_sregs_common(vcpu, sregs); - - if (vcpu->arch.guest_state_protected) - return; - - if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft) - set_bit(vcpu->arch.interrupt.nr, - (unsigned long *)sregs->interrupt_bitmap); -} - -static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) -{ - int i; - - __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2); - - if (vcpu->arch.guest_state_protected) - return; - - if (is_pae_paging(vcpu)) { - kvm_vcpu_srcu_read_lock(vcpu); - for (i = 0 ; i < 4 ; i++) - sregs2->pdptrs[i] = kvm_pdptr_read(vcpu, i); - sregs2->flags |= KVM_SREGS2_FLAGS_PDPTRS_VALID; - kvm_vcpu_srcu_read_unlock(vcpu); - } -} - -int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, - struct kvm_sregs *sregs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __get_sregs(vcpu, sregs); - vcpu_put(vcpu); - return 0; -} - int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu, struct kvm_mp_state *mp_state) { @@ -12312,173 +11725,6 @@ int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_task_switch); -static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) { - /* - * When EFER.LME and CR0.PG are set, the processor is in - * 64-bit mode (though maybe in a 32-bit code segment). - * CR4.PAE and EFER.LMA must be set. - */ - if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA)) - return false; - if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3)) - return false; - } else { - /* - * Not in 64-bit mode: EFER.LMA is clear and the code - * segment cannot be 64-bit. - */ - if (sregs->efer & EFER_LMA || sregs->cs.l) - return false; - } - - return kvm_is_valid_cr4(vcpu, sregs->cr4) && - kvm_is_valid_cr0(vcpu, sregs->cr0); -} - -static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs, - int *mmu_reset_needed, bool update_pdptrs) -{ - int idx; - struct desc_ptr dt; - - if (!kvm_is_valid_sregs(vcpu, sregs)) - return -EINVAL; - - if (kvm_apic_set_base(vcpu, sregs->apic_base, true)) - return -EINVAL; - - if (vcpu->arch.guest_state_protected) - return 0; - - dt.size = sregs->idt.limit; - dt.address = sregs->idt.base; - kvm_x86_call(set_idt)(vcpu, &dt); - dt.size = sregs->gdt.limit; - dt.address = sregs->gdt.base; - kvm_x86_call(set_gdt)(vcpu, &dt); - - vcpu->arch.cr2 = sregs->cr2; - *mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3; - vcpu->arch.cr3 = sregs->cr3; - kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3); - kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); - - *mmu_reset_needed |= vcpu->arch.efer != sregs->efer; - kvm_x86_call(set_efer)(vcpu, sregs->efer); - - *mmu_reset_needed |= kvm_read_cr0(vcpu) != sregs->cr0; - kvm_x86_call(set_cr0)(vcpu, sregs->cr0); - - *mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4; - kvm_x86_call(set_cr4)(vcpu, sregs->cr4); - - if (update_pdptrs) { - idx = srcu_read_lock(&vcpu->kvm->srcu); - if (is_pae_paging(vcpu)) { - load_pdptrs(vcpu, kvm_read_cr3(vcpu)); - *mmu_reset_needed = 1; - } - srcu_read_unlock(&vcpu->kvm->srcu, idx); - } - - kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS); - kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS); - kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES); - kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS); - kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS); - kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS); - - kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); - kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); - - kvm_set_cr8(vcpu, sregs->cr8); - - /* Older userspace won't unhalt the vcpu on reset. */ - if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 && - sregs->cs.selector == 0xf000 && sregs->cs.base == 0xffff0000 && - !is_protmode(vcpu)) - kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); - - return 0; -} - -static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - int pending_vec, max_bits; - int mmu_reset_needed = 0; - int ret = __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true); - - if (ret) - return ret; - - if (mmu_reset_needed) { - kvm_mmu_reset_context(vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - } - - max_bits = KVM_NR_INTERRUPTS; - pending_vec = find_first_bit( - (const unsigned long *)sregs->interrupt_bitmap, max_bits); - - if (pending_vec < max_bits) { - kvm_queue_interrupt(vcpu, pending_vec, false); - pr_debug("Set back pending irq %d\n", pending_vec); - kvm_make_request(KVM_REQ_EVENT, vcpu); - } - return 0; -} - -static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) -{ - int mmu_reset_needed = 0; - bool valid_pdptrs = sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID; - bool pae = (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) && - !(sregs2->efer & EFER_LMA); - int i, ret; - - if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID) - return -EINVAL; - - if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected)) - return -EINVAL; - - ret = __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, - &mmu_reset_needed, !valid_pdptrs); - if (ret) - return ret; - - if (valid_pdptrs) { - for (i = 0; i < 4 ; i++) - kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); - - kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR); - mmu_reset_needed = 1; - vcpu->arch.pdptrs_from_userspace = true; - } - if (mmu_reset_needed) { - kvm_mmu_reset_context(vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - } - return 0; -} - -int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, - struct kvm_sregs *sregs) -{ - int ret; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - ret = __set_sregs(vcpu, sregs); - vcpu_put(vcpu); - return ret; -} - static void kvm_arch_vcpu_guestdbg_update_apicv_inhibit(struct kvm *kvm) { bool set = false; @@ -12634,13 +11880,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) static void store_regs(struct kvm_vcpu *vcpu) { - BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES); - - if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS) - __get_regs(vcpu, &vcpu->run->s.regs.regs); - - if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS) - __get_sregs(vcpu, &vcpu->run->s.regs.sregs); + kvm_run_sync_regs_to_user(vcpu); if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_EVENTS) kvm_vcpu_ioctl_x86_get_vcpu_events( @@ -12649,19 +11889,8 @@ static void store_regs(struct kvm_vcpu *vcpu) static int sync_regs(struct kvm_vcpu *vcpu) { - if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) { - __set_regs(vcpu, &vcpu->run->s.regs.regs); - vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_REGS; - } - - if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) { - struct kvm_sregs sregs = vcpu->run->s.regs.sregs; - - if (__set_sregs(vcpu, &sregs)) - return -EINVAL; - - vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_SREGS; - } + if (kvm_run_sync_regs_from_user(vcpu)) + return -EINVAL; if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_EVENTS) { struct kvm_vcpu_events events = vcpu->run->s.regs.events; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index f9f05ee3d132..f4cd73a07490 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -54,6 +54,20 @@ struct kvm_host_values { u64 arch_capabilities; }; +void kvm_vcpu_ioctl_x86_get_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2); +int kvm_vcpu_ioctl_x86_set_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2); + +void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu); +int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu); + +void kvm_update_dr0123(struct kvm_vcpu *vcpu); +int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs); +int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs); + void kvm_spurious_fault(void); #define SIZE_OF_MEMSLOTS_HASHTABLE \ @@ -439,6 +453,7 @@ int handle_ud(struct kvm_vcpu *vcpu); void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, struct kvm_queued_exception *ex); +void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu); int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data); int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata); @@ -629,6 +644,7 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu); int kvm_spec_ctrl_test_value(u64 value); int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r, struct x86_exception *e); +void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid); int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva); bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index fffb6ef1997d..de74be2e69bf 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -409,6 +409,8 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa) int collapsed = 0; int i; + spin_lock(&cpa_lock); + if (cpa->flags & (CPA_PAGES_ARRAY | CPA_ARRAY)) { for (i = 0; i < cpa->numpages; i++) collapsed += collapse_large_pages(__cpa_addr(cpa, i), @@ -422,8 +424,10 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa) collapsed += collapse_large_pages(addr, &pgtables); } - if (!collapsed) + if (!collapsed) { + spin_unlock(&cpa_lock); return; + } flush_tlb_all(); @@ -431,6 +435,8 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa) list_del(&ptdesc->pt_list); pagetable_free(ptdesc); } + + spin_unlock(&cpa_lock); } static void cpa_flush(struct cpa_data *cpa, int cache) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 88a1bbfa918f..39cf6974e11e 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2175,8 +2175,13 @@ st: if (is_imm8(insn->off)) * BPF_PROBE_ATOMIC) before being used for the memory access. Pass * the reg holding the unmodified 32-bit address to * ex_handler_bpf(). + * + * A load-acquire is of BPF_STX class, but reads from src_reg + * into dst_reg like a BPF_LDX does, hence it must not be + * treated as a store here. */ - if (BPF_CLASS(insn->code) == BPF_LDX) { + if (BPF_CLASS(insn->code) == BPF_LDX || + bpf_atomic_is_load_acq(insn)) { arena_reg = reg2pt_regs[src_reg]; fixup_reg = reg2pt_regs[dst_reg]; } else { @@ -3187,11 +3192,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) && (flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET))); - /* extra registers for struct arguments */ - for (i = 0; i < m->nr_args; i++) { - if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) - nr_regs += (m->arg_size[i] + 7) / 8 - 1; - } + for (i = 0; i < m->nr_args; i++) + nr_regs += (m->arg_size[i] + 7) / 8 - 1; /* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 * are passed through regs, the remains are through stack. diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c index 791c52c8393f..5b00b13f72d4 100644 --- a/arch/x86/platform/efi/efi.c +++ b/arch/x86/platform/efi/efi.c @@ -921,7 +921,7 @@ umode_t efi_attr_is_visible(struct kobject *kobj, struct attribute *attr, int n) return attr->mode; } -enum efi_secureboot_mode __x86_ima_efi_boot_mode(void) +enum efi_secureboot_mode __x86_efi_boot_mode(void) { return boot_params.secure_boot; } diff --git a/block/bio.c b/block/bio.c index b184f0a3fcb2..63994b8cd8e1 100644 --- a/block/bio.c +++ b/block/bio.c @@ -253,6 +253,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, bio->bi_write_hint = 0; bio->bi_write_stream = 0; bio->bi_status = 0; + bio->bi_bvec_gap_bit = 0; bio->bi_iter.bi_sector = 0; bio->bi_iter.bi_size = 0; bio->bi_iter.bi_idx = 0; @@ -833,6 +834,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; + bio->bi_bvec_gap_bit = bio_src->bi_bvec_gap_bit; bio->bi_iter = bio_src->bi_iter; if (bio->bi_bdev) { @@ -1716,6 +1718,14 @@ struct bio *bio_split(struct bio *bio, int sectors, bio_advance(bio, split->bi_iter.bi_size); + /* + * The gap bit is set when splitting to limits and only applies to the + * front bio that was split off. The remaining bio will calcualte its + * gap value when it is subsequently split to limits, so it is safe to + * re-initialize the value back to 0. + */ + bio->bi_bvec_gap_bit = 0; + if (bio_flagged(bio, BIO_TRACE_COMPLETION)) bio_set_flag(split, BIO_TRACE_COMPLETION); diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index a2347b579587..f605df3991c3 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1612,6 +1612,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol) if (queue_is_mq(q)) memflags = blk_mq_freeze_queue(q); + + mutex_lock(&q->blkcg_mutex); retry: spin_lock_irq(&q->queue_lock); @@ -1621,6 +1623,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol) if (blkg->pd[pol->plid]) continue; + if (hlist_unhashed(&blkg->blkcg_node)) + continue; /* If prealloc matches, use it; otherwise try GFP_NOWAIT */ if (blkg == pinned_blkg) { @@ -1674,6 +1678,7 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol) spin_unlock_irq(&q->queue_lock); out: + mutex_unlock(&q->blkcg_mutex); if (queue_is_mq(q)) blk_mq_unfreeze_queue(q, memflags); if (pinned_blkg) diff --git a/block/blk-core.c b/block/blk-core.c index 14ae73eebe0d..c2300b201d4f 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -628,7 +628,7 @@ static void __submit_bio(struct bio *bio) /* If plug is not used, add new plug here to cache nsecs time. */ struct blk_plug plug; - if (unlikely(!blk_crypto_bio_prep(&bio))) + if (unlikely(!blk_crypto_bio_prep(bio))) return; blk_start_plug(&plug); @@ -639,12 +639,10 @@ static void __submit_bio(struct bio *bio) struct gendisk *disk = bio->bi_bdev->bd_disk; if ((bio->bi_opf & REQ_POLLED) && - !(disk->queue->limits.features & BLK_FEAT_POLL)) { - bio->bi_status = BLK_STS_NOTSUPP; - bio_endio(bio); - } else { + !(disk->queue->limits.features & BLK_FEAT_POLL)) + bio_endio_status(bio, BLK_STS_NOTSUPP); + else disk->fops->submit_bio(bio); - } blk_queue_exit(disk->queue); } @@ -882,8 +880,7 @@ void submit_bio_noacct(struct bio *bio) not_supported: status = BLK_STS_NOTSUPP; end_io: - bio->bi_status = status; - bio_endio(bio); + bio_endio_status(bio, status); } EXPORT_SYMBOL(submit_bio_noacct); diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c index 86b27f96051a..f523b0cf143a 100644 --- a/block/blk-crypto-fallback.c +++ b/block/blk-crypto-fallback.c @@ -75,13 +75,13 @@ static bool tfms_inited[BLK_ENCRYPTION_MODE_MAX]; static struct blk_crypto_fallback_keyslot { enum blk_crypto_mode_num crypto_mode; - struct crypto_skcipher *tfms[BLK_ENCRYPTION_MODE_MAX]; + struct crypto_sync_skcipher *tfms[BLK_ENCRYPTION_MODE_MAX]; } *blk_crypto_keyslots; static struct blk_crypto_profile *blk_crypto_fallback_profile; static struct workqueue_struct *blk_crypto_wq; static mempool_t *blk_crypto_bounce_page_pool; -static struct bio_set crypto_bio_split; +static struct bio_set enc_bio_set; /* * This is the key we set when evicting a keyslot. This *should* be the all 0's @@ -98,7 +98,7 @@ static void blk_crypto_fallback_evict_keyslot(unsigned int slot) WARN_ON(slotp->crypto_mode == BLK_ENCRYPTION_MODE_INVALID); /* Clear the key in the skcipher */ - err = crypto_skcipher_setkey(slotp->tfms[crypto_mode], blank_key, + err = crypto_sync_skcipher_setkey(slotp->tfms[crypto_mode], blank_key, blk_crypto_modes[crypto_mode].keysize); WARN_ON(err); slotp->crypto_mode = BLK_ENCRYPTION_MODE_INVALID; @@ -119,7 +119,7 @@ blk_crypto_fallback_keyslot_program(struct blk_crypto_profile *profile, blk_crypto_fallback_evict_keyslot(slot); slotp->crypto_mode = crypto_mode; - err = crypto_skcipher_setkey(slotp->tfms[crypto_mode], key->bytes, + err = crypto_sync_skcipher_setkey(slotp->tfms[crypto_mode], key->bytes, key->size); if (err) { blk_crypto_fallback_evict_keyslot(slot); @@ -150,88 +150,39 @@ static void blk_crypto_fallback_encrypt_endio(struct bio *enc_bio) mempool_free(enc_bio->bi_io_vec[i].bv_page, blk_crypto_bounce_page_pool); - src_bio->bi_status = enc_bio->bi_status; + if (enc_bio->bi_status) + cmpxchg(&src_bio->bi_status, 0, enc_bio->bi_status); - bio_uninit(enc_bio); - kfree(enc_bio); + bio_put(enc_bio); bio_endio(src_bio); } -static struct bio *blk_crypto_fallback_clone_bio(struct bio *bio_src) +static struct bio *blk_crypto_alloc_enc_bio(struct bio *bio_src, + unsigned int nr_segs) { - unsigned int nr_segs = bio_segments(bio_src); - struct bvec_iter iter; - struct bio_vec bv; struct bio *bio; - bio = bio_kmalloc(nr_segs, GFP_NOIO); - if (!bio) - return NULL; - bio_init_inline(bio, bio_src->bi_bdev, nr_segs, bio_src->bi_opf); + bio = bio_alloc_bioset(bio_src->bi_bdev, nr_segs, bio_src->bi_opf, + GFP_NOIO, &enc_bio_set); if (bio_flagged(bio_src, BIO_REMAPPED)) bio_set_flag(bio, BIO_REMAPPED); + bio->bi_private = bio_src; + bio->bi_end_io = blk_crypto_fallback_encrypt_endio; bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_write_hint = bio_src->bi_write_hint; bio->bi_write_stream = bio_src->bi_write_stream; bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; - bio->bi_iter.bi_size = bio_src->bi_iter.bi_size; - - bio_for_each_segment(bv, bio_src, iter) - bio->bi_io_vec[bio->bi_vcnt++] = bv; - bio_clone_blkg_association(bio, bio_src); - return bio; } -static bool -blk_crypto_fallback_alloc_cipher_req(struct blk_crypto_keyslot *slot, - struct skcipher_request **ciph_req_ret, - struct crypto_wait *wait) +static struct crypto_sync_skcipher * +blk_crypto_fallback_tfm(struct blk_crypto_keyslot *slot) { - struct skcipher_request *ciph_req; - const struct blk_crypto_fallback_keyslot *slotp; - int keyslot_idx = blk_crypto_keyslot_index(slot); - - slotp = &blk_crypto_keyslots[keyslot_idx]; - ciph_req = skcipher_request_alloc(slotp->tfms[slotp->crypto_mode], - GFP_NOIO); - if (!ciph_req) - return false; - - skcipher_request_set_callback(ciph_req, - CRYPTO_TFM_REQ_MAY_BACKLOG | - CRYPTO_TFM_REQ_MAY_SLEEP, - crypto_req_done, wait); - *ciph_req_ret = ciph_req; + const struct blk_crypto_fallback_keyslot *slotp = + &blk_crypto_keyslots[blk_crypto_keyslot_index(slot)]; - return true; -} - -static bool blk_crypto_fallback_split_bio_if_needed(struct bio **bio_ptr) -{ - struct bio *bio = *bio_ptr; - unsigned int i = 0; - unsigned int num_sectors = 0; - struct bio_vec bv; - struct bvec_iter iter; - - bio_for_each_segment(bv, bio, iter) { - num_sectors += bv.bv_len >> SECTOR_SHIFT; - if (++i == BIO_MAX_VECS) - break; - } - - if (num_sectors < bio_sectors(bio)) { - bio = bio_submit_split_bioset(bio, num_sectors, - &crypto_bio_split); - if (!bio) - return false; - - *bio_ptr = bio; - } - - return true; + return slotp->tfms[slotp->crypto_mode]; } union blk_crypto_iv { @@ -248,59 +199,22 @@ static void blk_crypto_dun_to_iv(const u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE], iv->dun[i] = cpu_to_le64(dun[i]); } -/* - * The crypto API fallback's encryption routine. - * Allocate a bounce bio for encryption, encrypt the input bio using crypto API, - * and replace *bio_ptr with the bounce bio. May split input bio if it's too - * large. Returns true on success. Returns false and sets bio->bi_status on - * error. - */ -static bool blk_crypto_fallback_encrypt_bio(struct bio **bio_ptr) +static void __blk_crypto_fallback_encrypt_bio(struct bio *src_bio, + struct crypto_sync_skcipher *tfm) { - struct bio *src_bio, *enc_bio; - struct bio_crypt_ctx *bc; - struct blk_crypto_keyslot *slot; - int data_unit_size; - struct skcipher_request *ciph_req = NULL; - DECLARE_CRYPTO_WAIT(wait); + struct bio_crypt_ctx *bc = src_bio->bi_crypt_context; + int data_unit_size = bc->bc_key->crypto_cfg.data_unit_size; + SYNC_SKCIPHER_REQUEST_ON_STACK(ciph_req, tfm); u64 curr_dun[BLK_CRYPTO_DUN_ARRAY_SIZE]; struct scatterlist src, dst; union blk_crypto_iv iv; - unsigned int i, j; - bool ret = false; - blk_status_t blk_st; - - /* Split the bio if it's too big for single page bvec */ - if (!blk_crypto_fallback_split_bio_if_needed(bio_ptr)) - return false; - - src_bio = *bio_ptr; - bc = src_bio->bi_crypt_context; - data_unit_size = bc->bc_key->crypto_cfg.data_unit_size; - - /* Allocate bounce bio for encryption */ - enc_bio = blk_crypto_fallback_clone_bio(src_bio); - if (!enc_bio) { - src_bio->bi_status = BLK_STS_RESOURCE; - return false; - } - - /* - * Get a blk-crypto-fallback keyslot that contains a crypto_skcipher for - * this bio's algorithm and key. - */ - blk_st = blk_crypto_get_keyslot(blk_crypto_fallback_profile, - bc->bc_key, &slot); - if (blk_st != BLK_STS_OK) { - src_bio->bi_status = blk_st; - goto out_put_enc_bio; - } + unsigned int nr_enc_pages, enc_idx; + struct bio *enc_bio; + unsigned int i; - /* and then allocate an skcipher_request for it */ - if (!blk_crypto_fallback_alloc_cipher_req(slot, &ciph_req, &wait)) { - src_bio->bi_status = BLK_STS_RESOURCE; - goto out_release_keyslot; - } + skcipher_request_set_callback(ciph_req, + CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP, + NULL, NULL); memcpy(curr_dun, bc->bc_dun, sizeof(curr_dun)); sg_init_table(&src, 1); @@ -309,101 +223,111 @@ static bool blk_crypto_fallback_encrypt_bio(struct bio **bio_ptr) skcipher_request_set_crypt(ciph_req, &src, &dst, data_unit_size, iv.bytes); - /* Encrypt each page in the bounce bio */ - for (i = 0; i < enc_bio->bi_vcnt; i++) { - struct bio_vec *enc_bvec = &enc_bio->bi_io_vec[i]; - struct page *plaintext_page = enc_bvec->bv_page; - struct page *ciphertext_page = - mempool_alloc(blk_crypto_bounce_page_pool, GFP_NOIO); - - enc_bvec->bv_page = ciphertext_page; - - if (!ciphertext_page) { - src_bio->bi_status = BLK_STS_RESOURCE; - goto out_free_bounce_pages; - } - - sg_set_page(&src, plaintext_page, data_unit_size, - enc_bvec->bv_offset); - sg_set_page(&dst, ciphertext_page, data_unit_size, - enc_bvec->bv_offset); - - /* Encrypt each data unit in this page */ - for (j = 0; j < enc_bvec->bv_len; j += data_unit_size) { + /* + * Encrypt each page in the source bio. Because the source bio could + * have bio_vecs that span more than a single page, but the encrypted + * bios are limited to a single page per bio_vec, this can generate + * more than a single encrypted bio per source bio. + */ +new_bio: + nr_enc_pages = min(bio_segments(src_bio), BIO_MAX_VECS); + enc_bio = blk_crypto_alloc_enc_bio(src_bio, nr_enc_pages); + enc_idx = 0; + for (;;) { + struct bio_vec src_bv = + bio_iter_iovec(src_bio, src_bio->bi_iter); + struct page *enc_page; + + enc_page = mempool_alloc(blk_crypto_bounce_page_pool, + GFP_NOIO); + __bio_add_page(enc_bio, enc_page, src_bv.bv_len, + src_bv.bv_offset); + + sg_set_page(&src, src_bv.bv_page, data_unit_size, + src_bv.bv_offset); + sg_set_page(&dst, enc_page, data_unit_size, src_bv.bv_offset); + + /* + * Increment the index now that the encrypted page is added to + * the bio. This is important for the error unwind path. + */ + enc_idx++; + + /* + * Encrypt each data unit in this page. + */ + for (i = 0; i < src_bv.bv_len; i += data_unit_size) { blk_crypto_dun_to_iv(curr_dun, &iv); - if (crypto_wait_req(crypto_skcipher_encrypt(ciph_req), - &wait)) { - i++; - src_bio->bi_status = BLK_STS_IOERR; - goto out_free_bounce_pages; + if (crypto_skcipher_encrypt(ciph_req)) { + bio_io_error(enc_bio); + return; } bio_crypt_dun_increment(curr_dun, 1); src.offset += data_unit_size; dst.offset += data_unit_size; } - } - enc_bio->bi_private = src_bio; - enc_bio->bi_end_io = blk_crypto_fallback_encrypt_endio; - *bio_ptr = enc_bio; - ret = true; + bio_advance_iter_single(src_bio, &src_bio->bi_iter, + src_bv.bv_len); + if (!src_bio->bi_iter.bi_size) + break; - enc_bio = NULL; - goto out_free_ciph_req; + if (enc_idx == nr_enc_pages) { + /* + * For each additional encrypted bio submitted, + * increment the source bio's remaining count. Each + * encrypted bio's completion handler calls bio_endio on + * the source bio, so this keeps the source bio from + * completing until the last encrypted bio does. + */ + bio_inc_remaining(src_bio); + submit_bio(enc_bio); + goto new_bio; + } + } -out_free_bounce_pages: - while (i > 0) - mempool_free(enc_bio->bi_io_vec[--i].bv_page, - blk_crypto_bounce_page_pool); -out_free_ciph_req: - skcipher_request_free(ciph_req); -out_release_keyslot: - blk_crypto_put_keyslot(slot); -out_put_enc_bio: - if (enc_bio) - bio_uninit(enc_bio); - kfree(enc_bio); - return ret; + submit_bio(enc_bio); } /* - * The crypto API fallback's main decryption routine. - * Decrypts input bio in place, and calls bio_endio on the bio. + * The crypto API fallback's encryption routine. + * + * Allocate one or more bios for encryption, encrypt the input bio using the + * crypto API, and submit the encrypted bios. Sets bio->bi_status and + * completes the source bio on error */ -static void blk_crypto_fallback_decrypt_bio(struct work_struct *work) +static void blk_crypto_fallback_encrypt_bio(struct bio *src_bio) { - struct bio_fallback_crypt_ctx *f_ctx = - container_of(work, struct bio_fallback_crypt_ctx, work); - struct bio *bio = f_ctx->bio; - struct bio_crypt_ctx *bc = &f_ctx->crypt_ctx; + struct bio_crypt_ctx *bc = src_bio->bi_crypt_context; struct blk_crypto_keyslot *slot; - struct skcipher_request *ciph_req = NULL; - DECLARE_CRYPTO_WAIT(wait); + blk_status_t status; + + status = blk_crypto_get_keyslot(blk_crypto_fallback_profile, + bc->bc_key, &slot); + if (status != BLK_STS_OK) { + bio_endio_status(src_bio, status); + return; + } + __blk_crypto_fallback_encrypt_bio(src_bio, + blk_crypto_fallback_tfm(slot)); + blk_crypto_put_keyslot(slot); +} + +static blk_status_t __blk_crypto_fallback_decrypt_bio(struct bio *bio, + struct bio_crypt_ctx *bc, struct bvec_iter iter, + struct crypto_sync_skcipher *tfm) +{ + SYNC_SKCIPHER_REQUEST_ON_STACK(ciph_req, tfm); u64 curr_dun[BLK_CRYPTO_DUN_ARRAY_SIZE]; union blk_crypto_iv iv; struct scatterlist sg; struct bio_vec bv; - struct bvec_iter iter; const int data_unit_size = bc->bc_key->crypto_cfg.data_unit_size; unsigned int i; - blk_status_t blk_st; - - /* - * Get a blk-crypto-fallback keyslot that contains a crypto_skcipher for - * this bio's algorithm and key. - */ - blk_st = blk_crypto_get_keyslot(blk_crypto_fallback_profile, - bc->bc_key, &slot); - if (blk_st != BLK_STS_OK) { - bio->bi_status = blk_st; - goto out_no_keyslot; - } - /* and then allocate an skcipher_request for it */ - if (!blk_crypto_fallback_alloc_cipher_req(slot, &ciph_req, &wait)) { - bio->bi_status = BLK_STS_RESOURCE; - goto out; - } + skcipher_request_set_callback(ciph_req, + CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP, + NULL, NULL); memcpy(curr_dun, bc->bc_dun, sizeof(curr_dun)); sg_init_table(&sg, 1); @@ -411,7 +335,7 @@ static void blk_crypto_fallback_decrypt_bio(struct work_struct *work) iv.bytes); /* Decrypt each segment in the bio */ - __bio_for_each_segment(bv, bio, iter, f_ctx->crypt_iter) { + __bio_for_each_segment(bv, bio, iter, iter) { struct page *page = bv.bv_page; sg_set_page(&sg, page, data_unit_size, bv.bv_offset); @@ -419,22 +343,41 @@ static void blk_crypto_fallback_decrypt_bio(struct work_struct *work) /* Decrypt each data unit in the segment */ for (i = 0; i < bv.bv_len; i += data_unit_size) { blk_crypto_dun_to_iv(curr_dun, &iv); - if (crypto_wait_req(crypto_skcipher_decrypt(ciph_req), - &wait)) { - bio->bi_status = BLK_STS_IOERR; - goto out; - } + if (crypto_skcipher_decrypt(ciph_req)) + return BLK_STS_IOERR; bio_crypt_dun_increment(curr_dun, 1); sg.offset += data_unit_size; } } -out: - skcipher_request_free(ciph_req); - blk_crypto_put_keyslot(slot); -out_no_keyslot: + return BLK_STS_OK; +} + +/* + * The crypto API fallback's main decryption routine. + * + * Decrypts input bio in place, and calls bio_endio on the bio. + */ +static void blk_crypto_fallback_decrypt_bio(struct work_struct *work) +{ + struct bio_fallback_crypt_ctx *f_ctx = + container_of(work, struct bio_fallback_crypt_ctx, work); + struct bio *bio = f_ctx->bio; + struct bio_crypt_ctx *bc = &f_ctx->crypt_ctx; + struct blk_crypto_keyslot *slot; + blk_status_t status; + + status = blk_crypto_get_keyslot(blk_crypto_fallback_profile, + bc->bc_key, &slot); + if (status == BLK_STS_OK) { + status = __blk_crypto_fallback_decrypt_bio(bio, bc, + f_ctx->crypt_iter, + blk_crypto_fallback_tfm(slot)); + blk_crypto_put_keyslot(slot); + } mempool_free(f_ctx, bio_fallback_crypt_ctx_pool); - bio_endio(bio); + + bio_endio_status(bio, status); } /** @@ -466,44 +409,43 @@ static void blk_crypto_fallback_decrypt_endio(struct bio *bio) /** * blk_crypto_fallback_bio_prep - Prepare a bio to use fallback en/decryption + * @bio: bio to prepare * - * @bio_ptr: pointer to the bio to prepare + * If bio is doing a WRITE operation, allocate one or more bios to contain the + * encrypted payload and submit them. * - * If bio is doing a WRITE operation, this splits the bio into two parts if it's - * too big (see blk_crypto_fallback_split_bio_if_needed()). It then allocates a - * bounce bio for the first part, encrypts it, and updates bio_ptr to point to - * the bounce bio. - * - * For a READ operation, we mark the bio for decryption by using bi_private and + * For a READ operation, mark the bio for decryption by using bi_private and * bi_end_io. * - * In either case, this function will make the bio look like a regular bio (i.e. - * as if no encryption context was ever specified) for the purposes of the rest - * of the stack except for blk-integrity (blk-integrity and blk-crypto are not - * currently supported together). + * In either case, this function will make the submitted bio(s) look like + * regular bios (i.e. as if no encryption context was ever specified) for the + * purposes of the rest of the stack except for blk-integrity (blk-integrity and + * blk-crypto are not currently supported together). * - * Return: true on success. Sets bio->bi_status and returns false on error. + * Return: true if @bio should be submitted to the driver by the caller, else + * false. Sets bio->bi_status, calls bio_endio and returns false on error. */ -bool blk_crypto_fallback_bio_prep(struct bio **bio_ptr) +bool blk_crypto_fallback_bio_prep(struct bio *bio) { - struct bio *bio = *bio_ptr; struct bio_crypt_ctx *bc = bio->bi_crypt_context; struct bio_fallback_crypt_ctx *f_ctx; if (WARN_ON_ONCE(!tfms_inited[bc->bc_key->crypto_cfg.crypto_mode])) { /* User didn't call blk_crypto_start_using_key() first */ - bio->bi_status = BLK_STS_IOERR; + bio_io_error(bio); return false; } if (!__blk_crypto_cfg_supported(blk_crypto_fallback_profile, &bc->bc_key->crypto_cfg)) { - bio->bi_status = BLK_STS_NOTSUPP; + bio_endio_status(bio, BLK_STS_NOTSUPP); return false; } - if (bio_data_dir(bio) == WRITE) - return blk_crypto_fallback_encrypt_bio(bio_ptr); + if (bio_data_dir(bio) == WRITE) { + blk_crypto_fallback_encrypt_bio(bio); + return false; + } /* * bio READ case: Set up a f_ctx in the bio's bi_private and set the @@ -537,7 +479,7 @@ static int blk_crypto_fallback_init(void) get_random_bytes(blank_key, sizeof(blank_key)); - err = bioset_init(&crypto_bio_split, 64, 0, 0); + err = bioset_init(&enc_bio_set, 64, 0, BIOSET_NEED_BVECS); if (err) goto out; @@ -607,7 +549,7 @@ static int blk_crypto_fallback_init(void) fail_free_profile: kfree(blk_crypto_fallback_profile); fail_free_bioset: - bioset_exit(&crypto_bio_split); + bioset_exit(&enc_bio_set); out: return err; } @@ -641,7 +583,8 @@ int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num) for (i = 0; i < blk_crypto_num_keyslots; i++) { slotp = &blk_crypto_keyslots[i]; - slotp->tfms[mode_num] = crypto_alloc_skcipher(cipher_str, 0, 0); + slotp->tfms[mode_num] = crypto_alloc_sync_skcipher(cipher_str, + 0, 0); if (IS_ERR(slotp->tfms[mode_num])) { err = PTR_ERR(slotp->tfms[mode_num]); if (err == -ENOENT) { @@ -653,7 +596,7 @@ int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num) goto out_free_tfms; } - crypto_skcipher_set_flags(slotp->tfms[mode_num], + crypto_sync_skcipher_set_flags(slotp->tfms[mode_num], CRYPTO_TFM_REQ_FORBID_WEAK_KEYS); } @@ -667,7 +610,7 @@ int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num) out_free_tfms: for (i = 0; i < blk_crypto_num_keyslots; i++) { slotp = &blk_crypto_keyslots[i]; - crypto_free_skcipher(slotp->tfms[mode_num]); + crypto_free_sync_skcipher(slotp->tfms[mode_num]); slotp->tfms[mode_num] = NULL; } out: diff --git a/block/blk-crypto-internal.h b/block/blk-crypto-internal.h index ccf6dff6ff6b..d65023120341 100644 --- a/block/blk-crypto-internal.h +++ b/block/blk-crypto-internal.h @@ -165,11 +165,11 @@ static inline void bio_crypt_do_front_merge(struct request *rq, #endif } -bool __blk_crypto_bio_prep(struct bio **bio_ptr); -static inline bool blk_crypto_bio_prep(struct bio **bio_ptr) +bool __blk_crypto_bio_prep(struct bio *bio); +static inline bool blk_crypto_bio_prep(struct bio *bio) { - if (bio_has_crypt_ctx(*bio_ptr)) - return __blk_crypto_bio_prep(bio_ptr); + if (bio_has_crypt_ctx(bio)) + return __blk_crypto_bio_prep(bio); return true; } @@ -215,12 +215,12 @@ static inline int blk_crypto_rq_bio_prep(struct request *rq, struct bio *bio, return 0; } +bool blk_crypto_fallback_bio_prep(struct bio *bio); + #ifdef CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK int blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num); -bool blk_crypto_fallback_bio_prep(struct bio **bio_ptr); - int blk_crypto_fallback_evict_key(const struct blk_crypto_key *key); #else /* CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK */ @@ -232,13 +232,6 @@ blk_crypto_fallback_start_using_mode(enum blk_crypto_mode_num mode_num) return -ENOPKG; } -static inline bool blk_crypto_fallback_bio_prep(struct bio **bio_ptr) -{ - pr_warn_once("crypto API fallback disabled; failing request.\n"); - (*bio_ptr)->bi_status = BLK_STS_NOTSUPP; - return false; -} - static inline int blk_crypto_fallback_evict_key(const struct blk_crypto_key *key) { diff --git a/block/blk-crypto.c b/block/blk-crypto.c index 3e7bf1974cbd..9792e7273110 100644 --- a/block/blk-crypto.c +++ b/block/blk-crypto.c @@ -260,54 +260,54 @@ void __blk_crypto_free_request(struct request *rq) /** * __blk_crypto_bio_prep - Prepare bio for inline encryption - * - * @bio_ptr: pointer to original bio pointer + * @bio: bio to prepare * * If the bio crypt context provided for the bio is supported by the underlying * device's inline encryption hardware, do nothing. * * Otherwise, try to perform en/decryption for this bio by falling back to the - * kernel crypto API. When the crypto API fallback is used for encryption, - * blk-crypto may choose to split the bio into 2 - the first one that will - * continue to be processed and the second one that will be resubmitted via - * submit_bio_noacct. A bounce bio will be allocated to encrypt the contents - * of the aforementioned "first one", and *bio_ptr will be updated to this - * bounce bio. + * kernel crypto API. For encryption this means submitting newly allocated + * bios for the encrypted payload while keeping back the source bio until they + * complete, while for reads the decryption happens in-place by a hooked in + * completion handler. * * Caller must ensure bio has bio_crypt_ctx. * - * Return: true on success; false on error (and bio->bi_status will be set - * appropriately, and bio_endio() will have been called so bio - * submission should abort). + * Return: true if @bio should be submitted to the driver by the caller, else + * false. Sets bio->bi_status, calls bio_endio and returns false on error. */ -bool __blk_crypto_bio_prep(struct bio **bio_ptr) +bool __blk_crypto_bio_prep(struct bio *bio) { - struct bio *bio = *bio_ptr; const struct blk_crypto_key *bc_key = bio->bi_crypt_context->bc_key; + struct block_device *bdev = bio->bi_bdev; /* Error if bio has no data. */ if (WARN_ON_ONCE(!bio_has_data(bio))) { - bio->bi_status = BLK_STS_IOERR; - goto fail; + bio_io_error(bio); + return false; } if (!bio_crypt_check_alignment(bio)) { bio->bi_status = BLK_STS_INVAL; - goto fail; + bio_endio(bio); + return false; } /* - * Success if device supports the encryption context, or if we succeeded - * in falling back to the crypto API. + * If the device does not natively support the encryption context, try to use + * the fallback if available. */ - if (blk_crypto_config_supported_natively(bio->bi_bdev, - &bc_key->crypto_cfg)) - return true; - if (blk_crypto_fallback_bio_prep(bio_ptr)) - return true; -fail: - bio_endio(*bio_ptr); - return false; + if (!blk_crypto_config_supported_natively(bdev, &bc_key->crypto_cfg)) { + if (!IS_ENABLED(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK)) { + pr_warn_once("%pg: crypto API fallback disabled; failing request.\n", + bdev); + bio_endio_status(bio, BLK_STS_NOTSUPP); + return false; + } + return blk_crypto_fallback_bio_prep(bio); + } + + return true; } int __blk_crypto_rq_bio_prep(struct request *rq, struct bio *bio, diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 5bfd70311359..ecf496912505 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -1603,7 +1603,7 @@ static void ioc_lat_stat(struct ioc *ioc, u32 *missed_ppm_ar, u32 *rq_wait_pct_p u64 rq_wait_ns = 0; int cpu, rw; - for_each_online_cpu(cpu) { + for_each_possible_cpu(cpu) { struct ioc_pcpu_stat *stat = per_cpu_ptr(ioc->pcpu_stat, cpu); u64 this_rq_wait_ns; diff --git a/block/blk-map.c b/block/blk-map.c index 60faf036fb6e..17a1dc288678 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -459,6 +459,8 @@ int blk_rq_append_bio(struct request *rq, struct bio *bio) if (rq->bio) { if (!ll_back_merge_fn(rq, bio, nr_segs)) return -EINVAL; + rq->phys_gap_bit = bio_seg_gap(rq->q, rq->biotail, bio, + rq->phys_gap_bit); rq->biotail->bi_next = bio; rq->biotail = bio; rq->__data_len += bio->bi_iter.bi_size; @@ -469,6 +471,7 @@ int blk_rq_append_bio(struct request *rq, struct bio *bio) rq->nr_phys_segments = nr_segs; rq->bio = rq->biotail = bio; rq->__data_len = bio->bi_iter.bi_size; + rq->phys_gap_bit = bio->bi_bvec_gap_bit; return 0; } EXPORT_SYMBOL(blk_rq_append_bio); diff --git a/block/blk-merge.c b/block/blk-merge.c index 03b61923cf10..5d895c789898 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -122,8 +122,7 @@ struct bio *bio_submit_split_bioset(struct bio *bio, unsigned int split_sectors, struct bio *split = bio_split(bio, split_sectors, GFP_NOIO, bs); if (IS_ERR(split)) { - bio->bi_status = errno_to_blk_status(PTR_ERR(split)); - bio_endio(bio); + bio_endio_status(bio, errno_to_blk_status(PTR_ERR(split))); return NULL; } @@ -143,8 +142,7 @@ EXPORT_SYMBOL_GPL(bio_submit_split_bioset); static struct bio *bio_submit_split(struct bio *bio, int split_sectors) { if (unlikely(split_sectors < 0)) { - bio->bi_status = errno_to_blk_status(split_sectors); - bio_endio(bio); + bio_endio_status(bio, errno_to_blk_status(split_sectors)); return NULL; } @@ -315,6 +313,12 @@ static unsigned int bio_split_alignment(struct bio *bio, return lim->logical_block_size; } +static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv, + struct bio_vec *bv) +{ + return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len); +} + /** * bio_split_io_at - check if and where to split a bio * @bio: [in] bio to be split @@ -332,8 +336,8 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim, unsigned *segs, unsigned max_bytes, unsigned len_align_mask) { struct bio_vec bv, bvprv, *bvprvp = NULL; + unsigned nsegs = 0, bytes = 0, gaps = 0; struct bvec_iter iter; - unsigned nsegs = 0, bytes = 0; bio_for_each_bvec(bv, bio, iter) { if (bv.bv_offset & lim->dma_alignment || @@ -344,8 +348,11 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim, * If the queue doesn't support SG gaps and adding this * offset would create a gap, disallow it. */ - if (bvprvp && bvec_gap_to_prev(lim, bvprvp, bv.bv_offset)) - goto split; + if (bvprvp) { + if (bvec_gap_to_prev(lim, bvprvp, bv.bv_offset)) + goto split; + gaps |= bvec_seg_gap(bvprvp, &bv); + } if (nsegs < lim->max_segments && bytes + bv.bv_len <= max_bytes && @@ -363,6 +370,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim, } *segs = nsegs; + bio->bi_bvec_gap_bit = ffs(gaps); return 0; split: if (bio->bi_opf & REQ_ATOMIC) @@ -398,6 +406,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim, * big IO can be trival, disable iopoll when split needed. */ bio_clear_polled(bio); + bio->bi_bvec_gap_bit = ffs(gaps); return bytes >> SECTOR_SHIFT; } EXPORT_SYMBOL_GPL(bio_split_io_at); @@ -734,6 +743,24 @@ static bool blk_atomic_write_mergeable_rqs(struct request *rq, return (rq->cmd_flags & REQ_ATOMIC) == (next->cmd_flags & REQ_ATOMIC); } +u8 bio_seg_gap(struct request_queue *q, struct bio *prev, struct bio *next, + u8 gaps_bit) +{ + struct bio_vec pb, nb; + + if (!bio_has_data(prev)) + return 0; + + gaps_bit = min_not_zero(gaps_bit, prev->bi_bvec_gap_bit); + gaps_bit = min_not_zero(gaps_bit, next->bi_bvec_gap_bit); + + bio_get_last_bvec(prev, &pb); + bio_get_first_bvec(next, &nb); + if (!biovec_phys_mergeable(q, &pb, &nb)) + gaps_bit = min_not_zero(gaps_bit, ffs(bvec_seg_gap(&pb, &nb))); + return gaps_bit; +} + /* * For non-mq, this has to be called with the request spinlock acquired. * For mq with scheduling, the appropriate queue wide lock should be held. @@ -798,6 +825,9 @@ static struct request *attempt_merge(struct request_queue *q, if (next->start_time_ns < req->start_time_ns) req->start_time_ns = next->start_time_ns; + req->phys_gap_bit = bio_seg_gap(req->q, req->biotail, next->bio, + min_not_zero(next->phys_gap_bit, + req->phys_gap_bit)); req->biotail->bi_next = next->bio; req->biotail = next->biotail; @@ -921,6 +951,8 @@ enum bio_merge_status bio_attempt_back_merge(struct request *req, if (req->rq_flags & RQF_ZONE_WRITE_PLUGGING) blk_zone_write_plug_bio_merged(bio); + req->phys_gap_bit = bio_seg_gap(req->q, req->biotail, bio, + req->phys_gap_bit); req->biotail->bi_next = bio; req->biotail = bio; req->__data_len += bio->bi_iter.bi_size; @@ -955,6 +987,8 @@ static enum bio_merge_status bio_attempt_front_merge(struct request *req, blk_update_mixed_merge(req, bio, true); + req->phys_gap_bit = bio_seg_gap(req->q, bio, req->bio, + req->phys_gap_bit); bio->bi_next = req->bio; req->bio = bio; diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c index 449950029872..b7c26599f6b0 100644 --- a/block/blk-mq-dma.c +++ b/block/blk-mq-dma.c @@ -79,8 +79,7 @@ static bool blk_map_iter_next(struct request *req, struct blk_map_iter *iter, static inline bool blk_can_dma_map_iova(struct request *req, struct device *dma_dev) { - return !((queue_virt_boundary(req->q) + 1) & - dma_get_merge_boundary(dma_dev)); + return !(req_phys_gap_mask(req) & dma_get_merge_boundary(dma_dev)); } static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec) @@ -194,6 +193,7 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev, if (blk_can_dma_map_iova(req, dma_dev) && dma_iova_try_alloc(dma_dev, state, vec.paddr, total_len)) return blk_rq_dma_map_iova(req, dma_dev, state, iter, &vec); + memset(state, 0, sizeof(*state)); return blk_dma_map_direct(req, dma_dev, iter, &vec); } diff --git a/block/blk-mq.c b/block/blk-mq.c index 56158b70d0c7..e050ea631cda 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -377,6 +377,7 @@ void blk_rq_init(struct request_queue *q, struct request *rq) INIT_LIST_HEAD(&rq->queuelist); rq->q = q; rq->__sector = (sector_t) -1; + rq->phys_gap_bit = 0; INIT_HLIST_NODE(&rq->hash); RB_CLEAR_NODE(&rq->rb_node); rq->tag = BLK_MQ_NO_TAG; @@ -669,6 +670,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf, goto out_queue_exit; } rq->__data_len = 0; + rq->phys_gap_bit = 0; rq->__sector = (sector_t) -1; rq->bio = rq->biotail = NULL; return rq; @@ -749,6 +751,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, rq = blk_mq_rq_ctx_init(&data, blk_mq_tags_from_data(&data), tag); blk_mq_rq_time_init(rq, alloc_time_ns); rq->__data_len = 0; + rq->phys_gap_bit = 0; rq->__sector = (sector_t) -1; rq->bio = rq->biotail = NULL; return rq; @@ -2675,6 +2678,8 @@ static void blk_mq_bio_to_request(struct request *rq, struct bio *bio, rq->bio = rq->biotail = bio; rq->__sector = bio->bi_iter.bi_sector; rq->__data_len = bio->bi_iter.bi_size; + rq->phys_gap_bit = bio->bi_bvec_gap_bit; + rq->nr_phys_segments = nr_segs; if (bio_integrity(bio)) rq->nr_integrity_segments = blk_rq_count_integrity_sg(rq->q, @@ -3149,8 +3154,7 @@ void blk_mq_submit_bio(struct bio *bio) } if ((bio->bi_opf & REQ_POLLED) && !blk_mq_can_poll(q)) { - bio->bi_status = BLK_STS_NOTSUPP; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_NOTSUPP); goto queue_exit; } @@ -3193,8 +3197,7 @@ void blk_mq_submit_bio(struct bio *bio) ret = blk_crypto_rq_get_keyslot(rq); if (ret != BLK_STS_OK) { - bio->bi_status = ret; - bio_endio(bio); + bio_endio_status(bio, ret); blk_mq_free_request(rq); return; } @@ -3384,6 +3387,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, } rq->nr_phys_segments = rq_src->nr_phys_segments; rq->nr_integrity_segments = rq_src->nr_integrity_segments; + rq->phys_gap_bit = rq_src->phys_gap_bit; if (rq->bio && blk_crypto_rq_bio_prep(rq, rq->bio, gfp_mask) < 0) goto free_and_out; diff --git a/block/blk-stat.c b/block/blk-stat.c index 682a8ddb1173..5c5289b08550 100644 --- a/block/blk-stat.c +++ b/block/blk-stat.c @@ -83,7 +83,7 @@ static void blk_stat_timer_fn(struct timer_list *t) for (bucket = 0; bucket < cb->buckets; bucket++) blk_rq_stat_init(&cb->stat[bucket]); - for_each_online_cpu(cpu) { + for_each_possible_cpu(cpu) { struct blk_rq_stat *cpu_stat; cpu_stat = per_cpu_ptr(cb->cpu_stat, cpu); diff --git a/block/fops.c b/block/fops.c index 5e3db9fead77..bbc89773e07f 100644 --- a/block/fops.c +++ b/block/fops.c @@ -221,8 +221,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, ret = blkdev_iov_iter_get_pages(bio, iter, bdev); if (unlikely(ret)) { - bio->bi_status = BLK_STS_IOERR; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_IOERR); break; } if (iocb->ki_flags & IOCB_NOWAIT) { @@ -242,8 +241,10 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, } if (iocb->ki_flags & IOCB_HAS_METADATA) { ret = bio_integrity_map_iter(bio, iocb->private); - if (unlikely(ret)) - goto fail; + if (unlikely(ret)) { + bio_endio_status(bio, errno_to_blk_status(ret)); + break; + } } if (is_read) { diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index e3eaeea62e24..927df06e6b94 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -279,7 +279,7 @@ static void kyber_timer_fn(struct timer_list *t) bool bad = false; /* Sum all of the per-cpu latency histograms. */ - for_each_online_cpu(cpu) { + for_each_possible_cpu(cpu) { struct kyber_cpu_latency *cpu_latency; cpu_latency = per_cpu_ptr(kqd->cpu_latency, cpu); diff --git a/crypto/acompress.c b/crypto/acompress.c index 25af7697d6bb..aa427b6235a5 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -567,12 +567,22 @@ EXPORT_SYMBOL_GPL(acomp_walk_virt); struct acomp_req *acomp_request_clone(struct acomp_req *req, size_t total, gfp_t gfp) { + struct crypto_tfm *tfm = req->base.tfm; struct acomp_req *nreq; + size_t len; - nreq = container_of(crypto_request_clone(&req->base, total, gfp), - struct acomp_req, base); - if (nreq == req) + len = sizeof(*req) + + crypto_acomp_reqsize(crypto_acomp_reqtfm(req)); + len = ALIGN(len, CRYPTO_MINALIGN); + + nreq = kzalloc(len, gfp); + if (!nreq) { + req->base.tfm = tfm->fb; return req; + } + + memcpy(nreq, req, sizeof(*req)); + nreq->base.flags &= ~CRYPTO_TFM_REQ_ON_STACK; if (req->src == &req->chain.ssg) nreq->src = &nreq->chain.ssg; diff --git a/crypto/lskcipher.c b/crypto/lskcipher.c index c2e2c38b5aa8..091761d6ca0b 100644 --- a/crypto/lskcipher.c +++ b/crypto/lskcipher.c @@ -95,7 +95,6 @@ static int crypto_lskcipher_crypt_unaligned( while (len >= bs) { unsigned chunk = min((unsigned)PAGE_SIZE, len); - int err; if (chunk > cs) chunk &= ~(cs - 1); diff --git a/drivers/accel/amdxdna/amdxdna_gem.c b/drivers/accel/amdxdna/amdxdna_gem.c index a719dc6aab33..ca747457cec7 100644 --- a/drivers/accel/amdxdna/amdxdna_gem.c +++ b/drivers/accel/amdxdna/amdxdna_gem.c @@ -949,6 +949,9 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, goto put_obj; } + if (!args->size) + goto unpin; + if (is_import_bo(abo)) drm_clflush_sg(abo->base.sgt); else if (abo->mem.kva) @@ -958,6 +961,7 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, else drm_WARN(&xdna->ddev, 1, "Can not get flush memory"); +unpin: amdxdna_gem_unpin(abo); XDNA_DBG(xdna, "Sync bo %d offset 0x%llx, size 0x%llx\n", diff --git a/drivers/accessibility/speakup/keyhelp.c b/drivers/accessibility/speakup/keyhelp.c index 822ceac83068..e632c53d6246 100644 --- a/drivers/accessibility/speakup/keyhelp.c +++ b/drivers/accessibility/speakup/keyhelp.c @@ -8,6 +8,7 @@ */ #include +#include #include "spk_priv.h" #include "speakup.h" @@ -111,7 +112,7 @@ static void say_key(int key) spk_msg_get(MSG_KEYNAMES_START + (key - 1))); } -static int help_init(void) +static void help_init(void) { char start = SPACE; int i; @@ -120,13 +121,19 @@ static int help_init(void) state_tbl = spk_our_keys[0] + SHIFT_TBL_SIZE + 2; for (i = 0; i < num_funcs; i++) { char *cur_funcname = spk_msg_get(MSG_FUNCNAMES_START + i); + char first_letter; - if (start == *cur_funcname) + first_letter = tolower(*cur_funcname); + + /* Accept only 'a'..'z' to index letter_offsets[] safely */ + if (first_letter < 'a' || first_letter > 'z') + continue; + + if (start == first_letter) continue; - start = *cur_funcname; + start = first_letter; letter_offsets[(start & 31) - 1] = i; } - return 0; } int spk_handle_help(struct vc_data *vc, u_char type, u_char ch, u_short key) @@ -144,7 +151,7 @@ int spk_handle_help(struct vc_data *vc, u_char type, u_char ch, u_short key) synth_printf("%s\n", spk_msg_get(MSG_LEAVING_HELP)); return 1; } - ch |= 32; /* lower case */ + ch = tolower(ch); if (ch < 'a' || ch > 'z') return -1; if (letter_offsets[ch - 'a'] == -1) { diff --git a/drivers/accessibility/speakup/kobjects.c b/drivers/accessibility/speakup/kobjects.c index 0dfdb6608e02..943ef71b1329 100644 --- a/drivers/accessibility/speakup/kobjects.c +++ b/drivers/accessibility/speakup/kobjects.c @@ -92,9 +92,9 @@ static void report_char_chartab_status(int reset, int received, int used, if (reset) { pr_info("%s reset to defaults\n", object_type[do_characters]); } else if (received) { - len = snprintf(buf, sizeof(buf), - " updated %d of %d %s\n", - used, received, object_type[do_characters]); + len = scnprintf(buf, sizeof(buf), + " updated %d of %d %s\n", + used, received, object_type[do_characters]); if (rejected) snprintf(buf + (len - 1), sizeof(buf) - (len - 1), " with %d reject%s\n", diff --git a/drivers/acpi/battery.c b/drivers/acpi/battery.c index 8196c17b5a97..296c2981b1a7 100644 --- a/drivers/acpi/battery.c +++ b/drivers/acpi/battery.c @@ -151,27 +151,28 @@ static int acpi_battery_technology(struct acpi_battery *battery) static int acpi_battery_get_state(struct acpi_battery *battery); -static int acpi_battery_is_charged(struct acpi_battery *battery) +static bool acpi_battery_is_full(struct acpi_battery *battery) { - /* charging, discharging, critical low or charge limited */ - if (battery->state != 0) - return 0; - /* battery not reporting charge */ if (battery->capacity_now == ACPI_BATTERY_VALUE_UNKNOWN || battery->capacity_now == 0) - return 0; + return false; /* good batteries update full_charge as the batteries degrade */ if (battery->full_charge_capacity == battery->capacity_now) - return 1; + return true; /* fallback to using design values for broken batteries */ - if (battery->design_capacity <= battery->capacity_now) - return 1; + return battery->design_capacity <= battery->capacity_now; +} - /* we don't do any sort of metric based on percentages */ - return 0; +static int acpi_battery_is_charged(struct acpi_battery *battery) +{ + /* charging, discharging, critical low or charge limited */ + if (battery->state != 0) + return 0; + + return acpi_battery_is_full(battery); } static bool acpi_battery_is_degraded(struct acpi_battery *battery) @@ -212,13 +213,14 @@ static int acpi_battery_get_property(struct power_supply *psy, if (battery->state & ACPI_BATTERY_STATE_DISCHARGING) val->intval = acpi_battery_handle_discharging(battery); else if (battery->state & ACPI_BATTERY_STATE_CHARGING) - /* Validate the status by checking the current. */ - if (battery->rate_now != ACPI_BATTERY_VALUE_UNKNOWN && - battery->rate_now == 0) { - /* On charge but no current (0W/0mA). */ - val->intval = POWER_SUPPLY_STATUS_NOT_CHARGING; - } else { + /* Check the rate and capacity to validate the status. */ + if (!acpi_battery_is_full(battery) || + (battery->rate_now != ACPI_BATTERY_VALUE_UNKNOWN && + battery->rate_now > 0)) { val->intval = POWER_SUPPLY_STATUS_CHARGING; + } else { + /* Full and zero rate. */ + val->intval = POWER_SUPPLY_STATUS_NOT_CHARGING; } else if (battery->state & ACPI_BATTERY_STATE_CHARGE_LIMITING) val->intval = POWER_SUPPLY_STATUS_NOT_CHARGING; diff --git a/drivers/acpi/ec.c b/drivers/acpi/ec.c index 1f4fc78a124f..16213dfb3aab 100644 --- a/drivers/acpi/ec.c +++ b/drivers/acpi/ec.c @@ -1512,6 +1512,24 @@ static bool install_gpio_irq_event_handler(struct acpi_ec *ec) IRQF_SHARED | IRQF_ONESHOT, "ACPI EC", ec) >= 0; } +static int ec_prepare_gpio_irq(struct acpi_ec *ec, struct acpi_device *device) +{ + int irq; + + if (!device || ec->gpe >= 0 || ec->irq >= 0) + return 0; + + /* ACPI reduced hardware platforms use a GpioInt from _CRS. */ + irq = acpi_dev_gpio_irq_get(device, 0); + if (irq == -EPROBE_DEFER) + return irq; + + if (irq >= 0) + ec->irq = irq; + + return 0; +} + /** * ec_install_handlers - Install service callbacks and register query methods. * @ec: Target EC. @@ -1526,7 +1544,6 @@ static bool install_gpio_irq_event_handler(struct acpi_ec *ec) * Return: * -ENODEV if the address space handler cannot be installed, which means * "unable to handle transactions", - * -EPROBE_DEFER if GPIO IRQ acquisition needs to be deferred, * or 0 (success) otherwise. */ static int ec_install_handlers(struct acpi_ec *ec, struct acpi_device *device, @@ -1559,19 +1576,6 @@ static int ec_install_handlers(struct acpi_ec *ec, struct acpi_device *device, if (!device) return 0; - if (ec->gpe < 0) { - /* ACPI reduced hardware platforms use a GpioInt from _CRS. */ - int irq = acpi_dev_gpio_irq_get(device, 0); - /* - * Bail out right away for deferred probing or complete the - * initialization regardless of any other errors. - */ - if (irq == -EPROBE_DEFER) - return -EPROBE_DEFER; - else if (irq >= 0) - ec->irq = irq; - } - if (!test_bit(EC_FLAGS_QUERY_METHODS_INSTALLED, &ec->flags)) { /* Find and register all query methods */ acpi_walk_namespace(ACPI_TYPE_METHOD, ec->handle, 1, @@ -1649,6 +1653,14 @@ static int acpi_ec_setup(struct acpi_ec *ec, struct acpi_device *device, bool ca { int ret; + /* + * GPIO IRQ lookup can defer. Do it before publishing the EC + * OpRegion to AML to avoid a spurious _REG(disconnect). + */ + ret = ec_prepare_gpio_irq(ec, device); + if (ret) + return ret; + /* First EC capable of handling transactions */ if (!first_ec) first_ec = ec; diff --git a/drivers/acpi/pci_link.c b/drivers/acpi/pci_link.c index e4560b33b8ad..bed7dc85612e 100644 --- a/drivers/acpi/pci_link.c +++ b/drivers/acpi/pci_link.c @@ -761,7 +761,7 @@ static int acpi_pci_link_resume(struct acpi_pci_link *link) return 0; } -static void irqrouter_resume(void) +static void irqrouter_resume(void *data) { struct acpi_pci_link *link; @@ -888,10 +888,14 @@ static int __init acpi_irq_balance_set(char *str) __setup("acpi_irq_balance", acpi_irq_balance_set); -static struct syscore_ops irqrouter_syscore_ops = { +static const struct syscore_ops irqrouter_syscore_ops = { .resume = irqrouter_resume, }; +static struct syscore irqrouter_syscore = { + .ops = &irqrouter_syscore_ops, +}; + void __init acpi_pci_link_init(void) { if (acpi_noirq) @@ -904,6 +908,6 @@ void __init acpi_pci_link_init(void) else acpi_irq_balance = 0; } - register_syscore_ops(&irqrouter_syscore_ops); + register_syscore(&irqrouter_syscore); acpi_scan_add_handler(&pci_link_handler); } diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c index 74ade4160314..32b868bd8487 100644 --- a/drivers/acpi/pci_root.c +++ b/drivers/acpi/pci_root.c @@ -730,7 +730,6 @@ static int acpi_pci_root_add(struct acpi_device *device, dev_err(&device->dev, "Bus %04x:%02x not present in PCI namespace\n", root->segment, (unsigned int)root->secondary.start); - device->driver_data = NULL; result = -ENODEV; goto remove_dmar; } @@ -766,6 +765,7 @@ static int acpi_pci_root_add(struct acpi_device *device, if (hotadd) dmar_device_remove(handle); end: + device->driver_data = NULL; kfree(root); return result; } @@ -789,6 +789,7 @@ static void acpi_pci_root_remove(struct acpi_device *device) pci_unlock_rescan_remove(); + device->driver_data = NULL; kfree(root); } diff --git a/drivers/acpi/processor_core.c b/drivers/acpi/processor_core.c index a4498357bd16..3bf076c150fa 100644 --- a/drivers/acpi/processor_core.c +++ b/drivers/acpi/processor_core.c @@ -336,11 +336,26 @@ int acpi_get_cpuid(acpi_handle handle, int type, u32 acpi_id) EXPORT_SYMBOL_GPL(acpi_get_cpuid); #ifdef CONFIG_ACPI_HOTPLUG_IOAPIC -static int get_ioapic_id(struct acpi_subtable_header *entry, u32 gsi_base, +static bool madt_entry_is_valid(struct acpi_subtable_header *entry, + unsigned long end) +{ + unsigned long start = (unsigned long)entry; + + if (start >= end || end - start < sizeof(*entry)) + return false; + + return entry->length >= sizeof(*entry) && entry->length <= end - start; +} + +static int get_ioapic_id(struct acpi_subtable_header *entry, + const unsigned long end, u32 gsi_base, u64 *phys_addr, int *ioapic_id) { struct acpi_madt_io_apic *ioapic = (struct acpi_madt_io_apic *)entry; + if (!madt_entry_is_valid(entry, end) || BAD_MADT_ENTRY(ioapic, end)) + return 0; + if (ioapic->global_irq_base != gsi_base) return 0; @@ -361,17 +376,19 @@ static int parse_madt_ioapic_entry(u32 gsi_base, u64 *phys_addr) return apic_id; entry = (unsigned long)madt; + if (madt->header.length < sizeof(*madt)) + return apic_id; madt_end = entry + madt->header.length; /* Parse all entries looking for a match. */ entry += sizeof(struct acpi_table_madt); - while (entry + sizeof(struct acpi_subtable_header) < madt_end) { + while (madt_entry_is_valid((struct acpi_subtable_header *)entry, + madt_end)) { hdr = (struct acpi_subtable_header *)entry; if (hdr->type == ACPI_MADT_TYPE_IO_APIC && - get_ioapic_id(hdr, gsi_base, phys_addr, &apic_id)) + get_ioapic_id(hdr, madt_end, gsi_base, phys_addr, &apic_id)) break; - else - entry += hdr->length; + entry += hdr->length; } return apic_id; @@ -398,7 +415,9 @@ static int parse_mat_ioapic_entry(acpi_handle handle, u32 gsi_base, header = (struct acpi_subtable_header *)obj->buffer.pointer; if (header->type == ACPI_MADT_TYPE_IO_APIC) - get_ioapic_id(header, gsi_base, phys_addr, &apic_id); + get_ioapic_id(header, + (unsigned long)header + obj->buffer.length, + gsi_base, phys_addr, &apic_id); exit: kfree(buffer.pointer); diff --git a/drivers/acpi/processor_driver.c b/drivers/acpi/processor_driver.c index 65e779be64ff..06589bf488f7 100644 --- a/drivers/acpi/processor_driver.c +++ b/drivers/acpi/processor_driver.c @@ -259,9 +259,11 @@ static int __init acpi_processor_driver_init(void) acpi_processor_ignore_ppc_init(); } + acpi_processor_register_idle_driver(); + result = driver_register(&acpi_processor_driver); if (result < 0) - return result; + goto unregister_idle_drv; result = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "acpi/cpu-drv:online", @@ -283,8 +285,19 @@ static int __init acpi_processor_driver_init(void) acpi_idle_rescan_dead_smt_siblings(); return 0; + err: driver_unregister(&acpi_processor_driver); + +unregister_idle_drv: + acpi_processor_unregister_idle_driver(); + + if (acpi_processor_cpufreq_init) { + cpufreq_unregister_notifier(&acpi_processor_notifier_block, + CPUFREQ_POLICY_NOTIFIER); + acpi_processor_cpufreq_init = false; + } + return result; } @@ -302,6 +315,7 @@ static void __exit acpi_processor_driver_exit(void) cpuhp_remove_state_nocalls(hp_online); cpuhp_remove_state_nocalls(CPUHP_ACPI_CPUDRV_DEAD); driver_unregister(&acpi_processor_driver); + acpi_processor_unregister_idle_driver(); } module_init(acpi_processor_driver_init); diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c index 2468c1d971e7..5c6f736ba3b5 100644 --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -831,19 +831,13 @@ static int acpi_processor_setup_cstates(struct acpi_processor *pr) return 0; } -static inline void acpi_processor_cstate_first_run_checks(void) +static inline void acpi_processor_update_max_cstate(void) { - static int first_run; - - if (first_run) - return; dmi_check_system(processor_power_dmi_table); max_cstate = acpi_processor_cstate_check(max_cstate); if (max_cstate < ACPI_C_STATES_MAX) pr_notice("processor limited to max C-state %d\n", max_cstate); - first_run++; - if (nocst) return; @@ -852,7 +846,7 @@ static inline void acpi_processor_cstate_first_run_checks(void) #else static inline int disabled_by_idle_boot_param(void) { return 0; } -static inline void acpi_processor_cstate_first_run_checks(void) { } +static inline void acpi_processor_update_max_cstate(void) { } static int acpi_processor_get_cstate_info(struct acpi_processor *pr) { return -ENODEV; @@ -945,6 +939,13 @@ static int acpi_processor_evaluate_lpi(acpi_handle handle, if (obj->type == ACPI_TYPE_BUFFER) { struct acpi_power_register *reg; + if (obj->buffer.length < sizeof(*reg)) { + acpi_handle_debug(handle, + "Invalid register data for _LPI state %d\n", + state_idx); + continue; + } + reg = (struct acpi_power_register *)obj->buffer.pointer; if (reg->space_id != ACPI_ADR_SPACE_SYSTEM_IO && reg->space_id != ACPI_ADR_SPACE_FIXED_HARDWARE) @@ -961,13 +962,6 @@ static int acpi_processor_evaluate_lpi(acpi_handle handle, continue; } - /* elements[7,8] skipped for now i.e. Residency/Usage counter*/ - - obj = pkg_elem + 9; - if (obj->type == ACPI_TYPE_STRING) - strscpy(lpi_state->desc, obj->string.pointer, - ACPI_CX_DESC_LEN); - lpi_state->index = state_idx; if (obj_get_integer(pkg_elem + 0, &lpi_state->min_residency)) { pr_debug("No min. residency found, assuming 10 us\n"); @@ -990,6 +984,20 @@ static int acpi_processor_evaluate_lpi(acpi_handle handle, if (obj_get_integer(pkg_elem + 5, &lpi_state->enable_parent_state)) lpi_state->enable_parent_state = 0; + + /* Skip elements [7-8] i.e. Residency/Usage counters. */ + + /* + * Avoid out-of-bounds access if the size of the package is less + * than expected. + */ + if (element->package.count < 10) + continue; + + obj = pkg_elem + 9; + if (obj->type == ACPI_TYPE_STRING) + strscpy(lpi_state->desc, obj->string.pointer, + ACPI_CX_DESC_LEN); } acpi_handle_debug(handle, "Found %d power states\n", state_idx); @@ -1357,7 +1365,59 @@ int acpi_processor_power_state_has_changed(struct acpi_processor *pr) return 0; } -static int acpi_processor_registered; +void acpi_processor_register_idle_driver(void) +{ + struct acpi_processor *pr; + int ret = -ENODEV; + int cpu; + + /* + * If a cpuidle driver is already registered, there is no need to + * evaluate _CST or attempt to register the ACPI idle driver. + */ + if (cpuidle_get_driver()) { + pr_debug("cpuidle driver %pS already registered.\n", cpuidle_get_driver()); + return; + } + + acpi_processor_update_max_cstate(); + + /* + * ACPI idle driver is used by all possible CPUs. + * Use the processor power info of one in them to set up idle states. + * Note that the existing idle handler will be used on platforms that + * only support C1. + */ + for_each_possible_cpu(cpu) { + pr = per_cpu(processors, cpu); + if (!pr) + continue; + + ret = acpi_processor_get_power_info(pr); + if (!ret) { + pr->flags.power_setup_done = 1; + acpi_processor_setup_cpuidle_states(pr); + break; + } + } + + if (ret) { + pr_debug("No ACPI power information from any CPUs.\n"); + return; + } + + ret = cpuidle_register_driver(&acpi_idle_driver); + if (ret) { + pr_debug("register %s failed.\n", acpi_idle_driver.name); + return; + } + pr_debug("%s registered with cpuidle.\n", acpi_idle_driver.name); +} + +void acpi_processor_unregister_idle_driver(void) +{ + cpuidle_unregister_driver(&acpi_idle_driver); +} int acpi_processor_power_init(struct acpi_processor *pr) { @@ -1367,27 +1427,10 @@ int acpi_processor_power_init(struct acpi_processor *pr) if (disabled_by_idle_boot_param()) return 0; - acpi_processor_cstate_first_run_checks(); - if (!acpi_processor_get_power_info(pr)) pr->flags.power_setup_done = 1; - /* - * Install the idle handler if processor power management is supported. - * Note that we use previously set idle handler will be used on - * platforms that only support C1. - */ if (pr->flags.power) { - /* Register acpi_idle_driver if not already registered */ - if (!acpi_processor_registered) { - acpi_processor_setup_cpuidle_states(pr); - retval = cpuidle_register_driver(&acpi_idle_driver); - if (retval) - return retval; - pr_debug("%s registered with cpuidle\n", - acpi_idle_driver.name); - } - dev = kzalloc(sizeof(*dev), GFP_KERNEL); if (!dev) return -ENOMEM; @@ -1400,14 +1443,11 @@ int acpi_processor_power_init(struct acpi_processor *pr) */ retval = cpuidle_register_device(dev); if (retval) { - if (acpi_processor_registered == 0) - cpuidle_unregister_driver(&acpi_idle_driver); per_cpu(acpi_cpuidle_device, pr->id) = NULL; kfree(dev); return retval; } - acpi_processor_registered++; } return 0; } @@ -1421,10 +1461,6 @@ int acpi_processor_power_exit(struct acpi_processor *pr) if (pr->flags.power) { cpuidle_unregister_device(dev); - acpi_processor_registered--; - if (acpi_processor_registered == 0) - cpuidle_unregister_driver(&acpi_idle_driver); - kfree(dev); } diff --git a/drivers/acpi/riscv/irq.c b/drivers/acpi/riscv/irq.c index d9a2154d6c6a..f41ab6704037 100644 --- a/drivers/acpi/riscv/irq.c +++ b/drivers/acpi/riscv/irq.c @@ -299,6 +299,7 @@ static acpi_status riscv_acpi_irq_get_parent(struct acpi_resource *ares, void *c return AE_OK; ctx->handle = riscv_acpi_get_gsi_handle(eirq->interrupts[ctx->index]); + ctx->rc = 0; return AE_CTRL_TERMINATE; } @@ -314,10 +315,22 @@ static int riscv_acpi_irq_get_dep(acpi_handle handle, unsigned int index, acpi_h acpi_walk_resources(handle, METHOD_NAME__CRS, riscv_acpi_irq_get_parent, &ctx); *gsi_handle = ctx.handle; - if (*gsi_handle) - return 1; - return 0; + return ctx.rc; +} + +static bool acpi_prt_entry_valid(void *prt_entry) +{ + struct acpi_pci_routing_table *entry = prt_entry; + + return entry && entry->length > 0; +} + +static void *acpi_prt_next_entry(void *prt_entry) +{ + struct acpi_pci_routing_table *entry = prt_entry; + + return prt_entry + entry->length; } static u32 riscv_acpi_add_prt_dep(acpi_handle handle) @@ -338,9 +351,11 @@ static u32 riscv_acpi_add_prt_dep(acpi_handle handle) } entry = buffer.pointer; - while (entry && (entry->length > 0)) { + for (; acpi_prt_entry_valid(entry); entry = acpi_prt_next_entry(entry)) { if (entry->source[0]) { - acpi_get_handle(handle, entry->source, &link_handle); + status = acpi_get_handle(handle, entry->source, &link_handle); + if (ACPI_FAILURE(status)) + continue; dep_devices.count = 1; dep_devices.handles = kcalloc(1, sizeof(*dep_devices.handles), GFP_KERNEL); if (!dep_devices.handles) { @@ -362,9 +377,6 @@ static u32 riscv_acpi_add_prt_dep(acpi_handle handle) dep_devices.handles[0] = gsi_handle; count += acpi_scan_add_dep(handle, &dep_devices); } - - entry = (struct acpi_pci_routing_table *) - ((unsigned long)entry + entry->length); } kfree(buffer.pointer); @@ -379,8 +391,11 @@ static u32 riscv_acpi_add_irq_dep(acpi_handle handle) int i; for (i = 0; - riscv_acpi_irq_get_dep(handle, i, &gsi_handle); + !riscv_acpi_irq_get_dep(handle, i, &gsi_handle); i++) { + if (!gsi_handle) + continue; + dep_devices.count = 1; dep_devices.handles = kcalloc(1, sizeof(*dep_devices.handles), GFP_KERNEL); if (!dep_devices.handles) { diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c index 16704c2a730c..92928389ea0f 100644 --- a/drivers/acpi/scan.c +++ b/drivers/acpi/scan.c @@ -522,12 +522,10 @@ static void acpi_device_release(struct device *dev) kfree(acpi_dev); } -static void acpi_device_del(struct acpi_device *device) +static void acpi_device_cleanup(struct acpi_device *device) { struct acpi_device_bus_id *acpi_device_bus_id; - mutex_lock(&acpi_device_lock); - list_for_each_entry(acpi_device_bus_id, &acpi_bus_id_list, node) if (!strcmp(acpi_device_bus_id->bus_id, acpi_device_hid(device))) { @@ -542,6 +540,13 @@ static void acpi_device_del(struct acpi_device *device) } list_del(&device->wakeup_list); +} + +static void acpi_device_del(struct acpi_device *device) +{ + mutex_lock(&acpi_device_lock); + + acpi_device_cleanup(device); mutex_unlock(&acpi_device_lock); @@ -802,7 +807,7 @@ int acpi_device_add(struct acpi_device *device) err: mutex_lock(&acpi_device_lock); - list_del(&device->wakeup_list); + acpi_device_cleanup(device); err_unlock: mutex_unlock(&acpi_device_lock); diff --git a/drivers/acpi/sleep.c b/drivers/acpi/sleep.c index 0b7fa4a8c379..1b9579bb97c2 100644 --- a/drivers/acpi/sleep.c +++ b/drivers/acpi/sleep.c @@ -892,13 +892,13 @@ bool acpi_s2idle_wakeup(void) #ifdef CONFIG_PM_SLEEP static u32 saved_bm_rld; -static int acpi_save_bm_rld(void) +static int acpi_save_bm_rld(void *data) { acpi_read_bit_register(ACPI_BITREG_BUS_MASTER_RLD, &saved_bm_rld); return 0; } -static void acpi_restore_bm_rld(void) +static void acpi_restore_bm_rld(void *data) { u32 resumed_bm_rld = 0; @@ -909,14 +909,18 @@ static void acpi_restore_bm_rld(void) acpi_write_bit_register(ACPI_BITREG_BUS_MASTER_RLD, saved_bm_rld); } -static struct syscore_ops acpi_sleep_syscore_ops = { +static const struct syscore_ops acpi_sleep_syscore_ops = { .suspend = acpi_save_bm_rld, .resume = acpi_restore_bm_rld, }; +static struct syscore acpi_sleep_syscore = { + .ops = &acpi_sleep_syscore_ops, +}; + static void acpi_sleep_syscore_init(void) { - register_syscore_ops(&acpi_sleep_syscore_ops); + register_syscore(&acpi_sleep_syscore); } #else static inline void acpi_sleep_syscore_init(void) {} diff --git a/drivers/acpi/video_detect.c b/drivers/acpi/video_detect.c index 2c120ade8f51..2a562cb42508 100644 --- a/drivers/acpi/video_detect.c +++ b/drivers/acpi/video_detect.c @@ -137,8 +137,10 @@ static int video_detect_portege_r100(const struct dmi_system_id *d) struct pci_dev *dev; /* Search for Trident CyberBlade XP4m32 to confirm Portégé R100 */ dev = pci_get_device(PCI_VENDOR_ID_TRIDENT, 0x2100, NULL); - if (dev) + if (dev) { acpi_backlight_dmi = acpi_backlight_vendor; + pci_dev_put(dev); + } return 0; } diff --git a/drivers/base/cacheinfo.c b/drivers/base/cacheinfo.c index 613410705a47..338804742dce 100644 --- a/drivers/base/cacheinfo.c +++ b/drivers/base/cacheinfo.c @@ -382,9 +382,14 @@ static int cache_setup_properties(unsigned int cpu) else if (!acpi_disabled) ret = cache_setup_acpi(cpu); - // Assume there is no cache information available in DT/ACPI from now. - if (ret && use_arch_cache_info()) + /* + * No DT/ACPI cache nodes; fall back to arch-derived topology (e.g. + * arm64 CLIDR_EL1) and clear the error to avoid a spurious warning. + */ + if (ret && use_arch_cache_info()) { use_arch_info = true; + ret = 0; + } return ret; } diff --git a/drivers/base/firmware_loader/fallback.c b/drivers/base/firmware_loader/fallback.c index 3ef0b312ae71..00d6b6e59461 100644 --- a/drivers/base/firmware_loader/fallback.c +++ b/drivers/base/firmware_loader/fallback.c @@ -95,6 +95,16 @@ static int fw_load_sysfs_fallback(struct fw_sysfs *fw_sysfs, long timeout) retval = -EINTR; goto out; } + + /* + * device_add() exposes the loading interface before pending_list is + * linked into pending_fw_head, so fw_state_done() may run first. + */ + if (fw_state_is_done(fw_priv)) { + mutex_unlock(&fw_lock); + goto out; + } + list_add(&fw_priv->pending_list, &pending_fw_head); mutex_unlock(&fw_lock); diff --git a/drivers/base/firmware_loader/main.c b/drivers/base/firmware_loader/main.c index 0f3e1779d1e8..214cdd909666 100644 --- a/drivers/base/firmware_loader/main.c +++ b/drivers/base/firmware_loader/main.c @@ -1646,16 +1646,20 @@ static int fw_pm_notify(struct notifier_block *notify_block, } /* stop caching firmware once syscore_suspend is reached */ -static int fw_suspend(void) +static int fw_suspend(void *data) { fw_cache.state = FW_LOADER_NO_CACHE; return 0; } -static struct syscore_ops fw_syscore_ops = { +static const struct syscore_ops fw_syscore_ops = { .suspend = fw_suspend, }; +static struct syscore fw_syscore = { + .ops = &fw_syscore_ops, +}; + static int __init register_fw_pm_ops(void) { int ret; @@ -1671,14 +1675,14 @@ static int __init register_fw_pm_ops(void) if (ret) return ret; - register_syscore_ops(&fw_syscore_ops); + register_syscore(&fw_syscore); return ret; } static inline void unregister_fw_pm_ops(void) { - unregister_syscore_ops(&fw_syscore_ops); + unregister_syscore(&fw_syscore); unregister_pm_notifier(&fw_cache.pm_notify); } #else diff --git a/drivers/base/memory.c b/drivers/base/memory.c index fdbec49f5f5b..8adee0e8c753 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -198,7 +198,7 @@ static ssize_t state_show(struct device *dev, struct device_attribute *attr, break; default: WARN_ON(1); - return sysfs_emit(buf, "ERROR-UNKNOWN-%ld\n", mem->state); + return sysfs_emit(buf, "ERROR-UNKNOWN-%d\n", mem->state); } return sysfs_emit(buf, "%s\n", output); @@ -473,7 +473,7 @@ static ssize_t phys_device_show(struct device *dev, static int print_allowed_zone(char *buf, int len, int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages, - int online_type, struct zone *default_zone) + enum mmop online_type, struct zone *default_zone) { struct zone *zone; diff --git a/drivers/base/soc.c b/drivers/base/soc.c index 282c38aece0d..c43940940aa3 100644 --- a/drivers/base/soc.c +++ b/drivers/base/soc.c @@ -194,6 +194,7 @@ EXPORT_SYMBOL_GPL(soc_device_unregister); static int __init soc_bus_register(void) { + struct soc_device *soc_dev; int ret; ret = bus_register(&soc_bus_type); @@ -201,10 +202,20 @@ static int __init soc_bus_register(void) return ret; soc_bus_registered = true; - if (early_soc_dev_attr) - return PTR_ERR(soc_device_register(early_soc_dev_attr)); + if (early_soc_dev_attr) { + soc_dev = soc_device_register(early_soc_dev_attr); + if (IS_ERR(soc_dev)) { + ret = PTR_ERR(soc_dev); + goto err_unregister_bus; + } + } return 0; + +err_unregister_bus: + soc_bus_registered = false; + bus_unregister(&soc_bus_type); + return ret; } core_initcall(soc_bus_register); diff --git a/drivers/base/swnode.c b/drivers/base/swnode.c index be1e9e61a7bf..1e08492935ee 100644 --- a/drivers/base/swnode.c +++ b/drivers/base/swnode.c @@ -529,7 +529,7 @@ software_node_get_reference_args(const struct fwnode_handle *fwnode, if (prop->is_inline) return -EINVAL; - if ((index + 1) * sizeof(*ref) > prop->length) + if (index >= prop->length / sizeof(*ref)) return -ENOENT; ref_array = prop->pointer; diff --git a/drivers/base/syscore.c b/drivers/base/syscore.c index 13db1f78d2ce..483adb796654 100644 --- a/drivers/base/syscore.c +++ b/drivers/base/syscore.c @@ -11,32 +11,32 @@ #include #include -static LIST_HEAD(syscore_ops_list); -static DEFINE_MUTEX(syscore_ops_lock); +static LIST_HEAD(syscore_list); +static DEFINE_MUTEX(syscore_lock); /** - * register_syscore_ops - Register a set of system core operations. - * @ops: System core operations to register. + * register_syscore - Register a set of system core operations. + * @syscore: System core operations to register. */ -void register_syscore_ops(struct syscore_ops *ops) +void register_syscore(struct syscore *syscore) { - mutex_lock(&syscore_ops_lock); - list_add_tail(&ops->node, &syscore_ops_list); - mutex_unlock(&syscore_ops_lock); + mutex_lock(&syscore_lock); + list_add_tail(&syscore->node, &syscore_list); + mutex_unlock(&syscore_lock); } -EXPORT_SYMBOL_GPL(register_syscore_ops); +EXPORT_SYMBOL_GPL(register_syscore); /** - * unregister_syscore_ops - Unregister a set of system core operations. - * @ops: System core operations to unregister. + * unregister_syscore - Unregister a set of system core operations. + * @syscore: System core operations to unregister. */ -void unregister_syscore_ops(struct syscore_ops *ops) +void unregister_syscore(struct syscore *syscore) { - mutex_lock(&syscore_ops_lock); - list_del(&ops->node); - mutex_unlock(&syscore_ops_lock); + mutex_lock(&syscore_lock); + list_del(&syscore->node); + mutex_unlock(&syscore_lock); } -EXPORT_SYMBOL_GPL(unregister_syscore_ops); +EXPORT_SYMBOL_GPL(unregister_syscore); #ifdef CONFIG_PM_SLEEP /** @@ -46,7 +46,7 @@ EXPORT_SYMBOL_GPL(unregister_syscore_ops); */ int syscore_suspend(void) { - struct syscore_ops *ops; + struct syscore *syscore; int ret = 0; trace_suspend_resume(TPS("syscore_suspend"), 0, true); @@ -59,25 +59,27 @@ int syscore_suspend(void) WARN_ONCE(!irqs_disabled(), "Interrupts enabled before system core suspend.\n"); - list_for_each_entry_reverse(ops, &syscore_ops_list, node) - if (ops->suspend) { - pm_pr_dbg("Calling %pS\n", ops->suspend); - ret = ops->suspend(); + list_for_each_entry_reverse(syscore, &syscore_list, node) + if (syscore->ops->suspend) { + pm_pr_dbg("Calling %pS\n", syscore->ops->suspend); + ret = syscore->ops->suspend(syscore->data); if (ret) goto err_out; WARN_ONCE(!irqs_disabled(), - "Interrupts enabled after %pS\n", ops->suspend); + "Interrupts enabled after %pS\n", + syscore->ops->suspend); } trace_suspend_resume(TPS("syscore_suspend"), 0, false); return 0; err_out: - pr_err("PM: System core suspend callback %pS failed.\n", ops->suspend); + pr_err("PM: System core suspend callback %pS failed.\n", + syscore->ops->suspend); - list_for_each_entry_continue(ops, &syscore_ops_list, node) - if (ops->resume) - ops->resume(); + list_for_each_entry_continue(syscore, &syscore_list, node) + if (syscore->ops->resume) + syscore->ops->resume(syscore->data); return ret; } @@ -90,18 +92,19 @@ EXPORT_SYMBOL_GPL(syscore_suspend); */ void syscore_resume(void) { - struct syscore_ops *ops; + struct syscore *syscore; trace_suspend_resume(TPS("syscore_resume"), 0, true); WARN_ONCE(!irqs_disabled(), "Interrupts enabled before system core resume.\n"); - list_for_each_entry(ops, &syscore_ops_list, node) - if (ops->resume) { - pm_pr_dbg("Calling %pS\n", ops->resume); - ops->resume(); + list_for_each_entry(syscore, &syscore_list, node) + if (syscore->ops->resume) { + pm_pr_dbg("Calling %pS\n", syscore->ops->resume); + syscore->ops->resume(syscore->data); WARN_ONCE(!irqs_disabled(), - "Interrupts enabled after %pS\n", ops->resume); + "Interrupts enabled after %pS\n", + syscore->ops->resume); } trace_suspend_resume(TPS("syscore_resume"), 0, false); } @@ -113,16 +116,17 @@ EXPORT_SYMBOL_GPL(syscore_resume); */ void syscore_shutdown(void) { - struct syscore_ops *ops; + struct syscore *syscore; - mutex_lock(&syscore_ops_lock); + mutex_lock(&syscore_lock); - list_for_each_entry_reverse(ops, &syscore_ops_list, node) - if (ops->shutdown) { + list_for_each_entry_reverse(syscore, &syscore_list, node) + if (syscore->ops->shutdown) { if (initcall_debug) - pr_info("PM: Calling %pS\n", ops->shutdown); - ops->shutdown(); + pr_info("PM: Calling %pS\n", + syscore->ops->shutdown); + syscore->ops->shutdown(syscore->data); } - mutex_unlock(&syscore_ops_lock); + mutex_unlock(&syscore_lock); } diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 567192e371a8..a7bf990ce9c3 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -3048,6 +3048,8 @@ static int mtip_block_ioctl(struct block_device *dev, if (!dd) return -ENOTTY; + guard(mutex)(&dd->ioctl_mutex); + if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag))) return -ENOTTY; @@ -3086,6 +3088,8 @@ static int mtip_block_compat_ioctl(struct block_device *dev, if (!dd) return -ENOTTY; + guard(mutex)(&dd->ioctl_mutex); + if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag))) return -ENOTTY; @@ -3711,6 +3715,7 @@ static int mtip_pci_probe(struct pci_dev *pdev, dd = kzalloc_node(sizeof(struct driver_data), GFP_KERNEL, my_node); if (!dd) return -ENOMEM; + mutex_init(&dd->ioctl_mutex); /* Attach the private data to this PCI device. */ pci_set_drvdata(pdev, dd); @@ -3874,6 +3879,7 @@ static void mtip_pci_remove(struct pci_dev *pdev) } set_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag); + mutex_lock(&dd->ioctl_mutex); if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag)) del_gendisk(dd->disk); @@ -3902,6 +3908,7 @@ static void mtip_pci_remove(struct pci_dev *pdev) /* De-initialize the protocol layer. */ mtip_hw_exit(dd); + mutex_unlock(&dd->ioctl_mutex); if (dd->isr_workq) { destroy_workqueue(dd->isr_workq); diff --git a/drivers/block/mtip32xx/mtip32xx.h b/drivers/block/mtip32xx/mtip32xx.h index f7328f19ac5c..0963c07b5845 100644 --- a/drivers/block/mtip32xx/mtip32xx.h +++ b/drivers/block/mtip32xx/mtip32xx.h @@ -12,6 +12,7 @@ #define __MTIP32XX_H__ #include +#include #include #include #include @@ -432,6 +433,7 @@ struct driver_data { struct request_queue *queue; /* Our request queue. */ struct blk_mq_tag_set tags; /* blk_mq tags */ + struct mutex ioctl_mutex; struct mtip_port *port; /* Pointer to the port data structure. */ diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 50938881219b..35e3a4eb21d8 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -66,7 +66,7 @@ struct nullb_page { #define NULLB_PAGE_FREE (MAP_SZ - 2) static LIST_HEAD(nullb_list); -static struct mutex lock; +static DEFINE_MUTEX(lock); static int null_major; static DEFINE_IDA(nullb_indexes); static struct blk_mq_tag_set tag_set; @@ -340,7 +340,15 @@ static ssize_t nullb_device_bool_attr_store(bool *val, const char *page, return count; } -/* The following macro should only be used with TYPE = {uint, ulong, bool}. */ +/* + * The following macro should only be used with TYPE = {uint, ulong, bool}. + * + * The device configuration is modified under the global lock to serialize + * attribute changes against null_add_dev() and null_del_dev(): without this, + * an attribute could be changed while null_add_dev() is running, that is, + * before NULLB_DEV_FL_CONFIGURED is set, which would let null_add_dev() + * observe inconsistent values for the device configuration. + */ #define NULLB_DEVICE_ATTR(NAME, TYPE, APPLY) \ static ssize_t \ nullb_device_##NAME##_show(struct config_item *item, char *page) \ @@ -360,6 +368,7 @@ nullb_device_##NAME##_store(struct config_item *item, const char *page, \ ret = nullb_device_##TYPE##_attr_store(&new_value, page, count);\ if (ret < 0) \ return ret; \ + guard(mutex)(&lock); \ if (apply_fn) \ ret = apply_fn(dev, new_value); \ else if (test_bit(NULLB_DEV_FL_CONFIGURED, &dev->flags)) \ @@ -379,9 +388,20 @@ static int nullb_update_nr_hw_queues(struct nullb_device *dev, struct blk_mq_tag_set *set; int ret, nr_hw_queues; + lockdep_assert_held(&lock); + if (!dev->nullb) return 0; + /* + * A shared tag_set is mapped via the module-wide queue counts, so a + * per-device resize is meaningless. On shrink it would also leave + * mq_map[] pointing at NULLed hctx slots, causing a NULL deref in + * blk_mq_map_swqueue(). Reject it. + */ + if (dev->shared_tags) + return -EINVAL; + /* * Make sure at least one submit queue exists. */ @@ -421,25 +441,13 @@ static int nullb_update_nr_hw_queues(struct nullb_device *dev, static int nullb_apply_submit_queues(struct nullb_device *dev, unsigned int submit_queues) { - int ret; - - mutex_lock(&lock); - ret = nullb_update_nr_hw_queues(dev, submit_queues, dev->poll_queues); - mutex_unlock(&lock); - - return ret; + return nullb_update_nr_hw_queues(dev, submit_queues, dev->poll_queues); } static int nullb_apply_poll_queues(struct nullb_device *dev, unsigned int poll_queues) { - int ret; - - mutex_lock(&lock); - ret = nullb_update_nr_hw_queues(dev, dev->submit_queues, poll_queues); - mutex_unlock(&lock); - - return ret; + return nullb_update_nr_hw_queues(dev, dev->submit_queues, poll_queues); } NULLB_DEVICE_ATTR(size, ulong, NULL); @@ -579,8 +587,13 @@ static ssize_t nullb_device_zone_readonly_store(struct config_item *item, const char *page, size_t count) { struct nullb_device *dev = to_nullb_device(item); + ssize_t ret; - return zone_cond_store(dev, page, count, BLK_ZONE_COND_READONLY); + mutex_lock(&lock); + ret = zone_cond_store(dev, page, count, BLK_ZONE_COND_READONLY); + mutex_unlock(&lock); + + return ret; } CONFIGFS_ATTR_WO(nullb_device_, zone_readonly); @@ -588,8 +601,13 @@ static ssize_t nullb_device_zone_offline_store(struct config_item *item, const char *page, size_t count) { struct nullb_device *dev = to_nullb_device(item); + ssize_t ret; + + mutex_lock(&lock); + ret = zone_cond_store(dev, page, count, BLK_ZONE_COND_OFFLINE); + mutex_unlock(&lock); - return zone_cond_store(dev, page, count, BLK_ZONE_COND_OFFLINE); + return ret; } CONFIGFS_ATTR_WO(nullb_device_, zone_offline); @@ -836,7 +854,6 @@ static void null_free_dev(struct nullb_device *dev) if (!dev) return; - null_free_zoned_dev(dev); badblocks_exit(&dev->badblocks); kfree(dev); } @@ -1777,6 +1794,7 @@ static void null_del_dev(struct nullb *nullb) } put_disk(nullb->disk); + null_free_zoned_dev(dev); if (nullb->tag_set == &nullb->__tag_set) blk_mq_free_tag_set(nullb->tag_set); kfree(nullb->queues); @@ -2163,17 +2181,9 @@ static int __init null_init(void) config_group_init(&nullb_subsys.su_group); mutex_init(&nullb_subsys.su_mutex); - ret = configfs_register_subsystem(&nullb_subsys); - if (ret) - return ret; - - mutex_init(&lock); - null_major = register_blkdev(0, "nullb"); - if (null_major < 0) { - ret = null_major; - goto err_conf; - } + if (null_major < 0) + return null_major; for (i = 0; i < nr_devices; i++) { ret = null_create_dev(); @@ -2181,6 +2191,10 @@ static int __init null_init(void) goto err_dev; } + ret = configfs_register_subsystem(&nullb_subsys); + if (ret) + goto err_dev; + pr_info("module loaded\n"); return 0; @@ -2190,8 +2204,8 @@ static int __init null_init(void) null_destroy_dev(nullb); } unregister_blkdev(null_major, "nullb"); -err_conf: - configfs_unregister_subsystem(&nullb_subsys); + if (tag_set.ops) + blk_mq_free_tag_set(&tag_set); return ret; } @@ -2212,8 +2226,6 @@ static void __exit null_exit(void) if (tag_set.ops) blk_mq_free_tag_set(&tag_set); - - mutex_destroy(&lock); } module_init(null_init); diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index e7a77f27555e..a11805b29b63 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -603,7 +603,7 @@ static int ublk_validate_params(const struct ublk_device *ub) if (p->max_sectors < PAGE_SECTORS) return -EINVAL; - if (ublk_dev_is_zoned(ub) && !p->chunk_sectors) + if (ublk_dev_is_zoned(ub) && !is_power_of_2(p->chunk_sectors)) return -EINVAL; } else return -EINVAL; @@ -1050,7 +1050,10 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req, struct iov_iter iter; const int dir = ITER_DEST; - import_ubuf(dir, u64_to_user_ptr(io->buf.addr), rq_bytes, &iter); + if (import_ubuf(dir, u64_to_user_ptr(io->buf.addr), rq_bytes, + &iter) < 0) + return 0; + return ublk_copy_user_pages(req, 0, &iter, dir); } return rq_bytes; @@ -1071,7 +1074,10 @@ static int ublk_unmap_io(bool need_map, WARN_ON_ONCE(io->res > rq_bytes); - import_ubuf(dir, u64_to_user_ptr(io->buf.addr), io->res, &iter); + if (import_ubuf(dir, u64_to_user_ptr(io->buf.addr), io->res, + &iter) < 0) + return 0; + return ublk_copy_user_pages(req, 0, &iter, dir); } return rq_bytes; @@ -1190,8 +1196,14 @@ static inline void __ublk_complete_rq(struct request *req, struct ublk_io *io, * * Re-read simply for this unlikely case. */ - if (unlikely(unmapped_bytes < io->res)) + if (unlikely(unmapped_bytes < io->res)) { + if (unlikely(!unmapped_bytes)) { + res = BLK_STS_IOERR; + goto exit; + } + io->res = unmapped_bytes; + } /* * Run bio->bi_end_io() with softirqs disabled. If the final fput diff --git a/drivers/bluetooth/btintel.c b/drivers/bluetooth/btintel.c index 4d6b6bbb4267..f665682c91b6 100644 --- a/drivers/bluetooth/btintel.c +++ b/drivers/bluetooth/btintel.c @@ -3757,7 +3757,7 @@ int btintel_recv_event(struct hci_dev *hdev, struct sk_buff *skb) /* Handle all diagnostics events separately. May still call * hci_recv_frame. */ - if (len >= sizeof(diagnostics_hdr) && + if (len + 1 >= sizeof(diagnostics_hdr) && memcmp(&skb->data[2], diagnostics_hdr, sizeof(diagnostics_hdr)) == 0) { return btintel_diagnostics(hdev, skb); diff --git a/drivers/bluetooth/btmtk.c b/drivers/bluetooth/btmtk.c index df0b59e6f968..fb097f78ecd8 100644 --- a/drivers/bluetooth/btmtk.c +++ b/drivers/bluetooth/btmtk.c @@ -112,7 +112,11 @@ static void btmtk_coredump_notify(struct hci_dev *hdev, int state) void btmtk_fw_get_filename(char *buf, size_t size, u32 dev_id, u32 fw_ver, u32 fw_flavor) { - if (dev_id == 0x7925) + if (dev_id == 0x6639) + snprintf(buf, size, + "mediatek/mt7927/BT_RAM_CODE_MT%04x_2_%x_hdr.bin", + dev_id & 0xffff, (fw_ver & 0xff) + 1); + else if (dev_id == 0x7925) snprintf(buf, size, "mediatek/mt%04x/BT_RAM_CODE_MT%04x_1_%x_hdr.bin", dev_id & 0xffff, dev_id & 0xffff, (fw_ver & 0xff) + 1); @@ -128,7 +132,8 @@ void btmtk_fw_get_filename(char *buf, size_t size, u32 dev_id, u32 fw_ver, EXPORT_SYMBOL_GPL(btmtk_fw_get_filename); int btmtk_setup_firmware_79xx(struct hci_dev *hdev, const char *fwname, - wmt_cmd_sync_func_t wmt_cmd_sync) + wmt_cmd_sync_func_t wmt_cmd_sync, + u32 dev_id) { struct btmtk_hci_wmt_params wmt_params; struct btmtk_patch_header *hdr; @@ -166,6 +171,14 @@ int btmtk_setup_firmware_79xx(struct hci_dev *hdev, const char *fwname, section_offset = le32_to_cpu(sectionmap->secoffset); dl_size = le32_to_cpu(sectionmap->bin_info_spec.dlsize); + /* MT6639: only download sections where dlmode byte0 == 0x01, + * matching the Windows driver behavior which skips WiFi/other + * sections that would cause the chip to hang. + */ + if (dev_id == 0x6639 && dl_size > 0 && + (le32_to_cpu(sectionmap->bin_info_spec.dlmodecrctype) & 0xff) != 0x01) + continue; + if (dl_size > 0) { retry = 20; while (retry > 0) { @@ -442,6 +455,22 @@ int btmtk_process_coredump(struct hci_dev *hdev, struct sk_buff *skb) EXPORT_SYMBOL_GPL(btmtk_process_coredump); #if IS_ENABLED(CONFIG_BT_HCIBTUSB_MTK) +/* Known MT6639 (MT7927) Bluetooth USB devices. + * Used to scope the zero-CHIPID workaround to real MT6639 hardware, + * since some boards return 0x0000 from the MMIO chip ID register. + */ +static const struct { + u16 vendor; + u16 product; +} btmtk_mt6639_devs[] = { + { 0x0489, 0xe13a }, /* ASUS ROG Crosshair X870E Hero */ + { 0x0489, 0xe0fa }, /* Lenovo Legion Pro 7 16ARX9 */ + { 0x0489, 0xe10f }, /* Gigabyte Z790 AORUS MASTER X */ + { 0x0489, 0xe110 }, /* MSI X870E Ace Max */ + { 0x0489, 0xe116 }, /* TP-Link Archer TBE550E */ + { 0x13d3, 0x3588 }, /* ASUS ROG STRIX X870E-E */ +}; + static void btmtk_usb_wmt_recv(struct urb *urb) { struct hci_dev *hdev = urb->context; @@ -786,27 +815,21 @@ static int btmtk_usb_uhw_reg_read(struct hci_dev *hdev, u32 reg, u32 *val) static int btmtk_usb_reg_read(struct hci_dev *hdev, u32 reg, u32 *val) { struct btmtk_data *data = hci_get_priv(hdev); - int pipe, err, size = sizeof(u32); - void *buf; - - buf = kzalloc(size, GFP_KERNEL); - if (!buf) - return -ENOMEM; + u8 buf[sizeof(u32)]; + int err; - pipe = usb_rcvctrlpipe(data->udev, 0); - err = usb_control_msg(data->udev, pipe, 0x63, - USB_TYPE_VENDOR | USB_DIR_IN, - reg >> 16, reg & 0xffff, - buf, size, USB_CTRL_GET_TIMEOUT); + *val = 0; + err = usb_control_msg_recv(data->udev, 0, 0x63, + USB_TYPE_VENDOR | USB_DIR_IN, + reg >> 16, reg & 0xffff, + buf, sizeof(buf), USB_CTRL_GET_TIMEOUT, + GFP_KERNEL); if (err < 0) - goto err_free_buf; + return err; *val = get_unaligned_le32(buf); -err_free_buf: - kfree(buf); - - return err; + return 0; } static int btmtk_usb_id_get(struct hci_dev *hdev, u32 reg, u32 *id) @@ -825,6 +848,7 @@ static u32 btmtk_usb_reset_done(struct hci_dev *hdev) int btmtk_usb_subsys_reset(struct hci_dev *hdev, u32 dev_id) { + int reset_err = 0; u32 val; int err; @@ -847,7 +871,7 @@ int btmtk_usb_subsys_reset(struct hci_dev *hdev, u32 dev_id) if (err < 0) return err; msleep(100); - } else if (dev_id == 0x7925) { + } else if (dev_id == 0x7925 || dev_id == 0x6639) { err = btmtk_usb_uhw_reg_read(hdev, MTK_BT_RESET_REG_CONNV3, &val); if (err) return err; @@ -923,8 +947,10 @@ int btmtk_usb_subsys_reset(struct hci_dev *hdev, u32 dev_id) err = readx_poll_timeout(btmtk_usb_reset_done, hdev, val, val & MTK_BT_RST_DONE, 20000, 1000000); - if (err < 0) + if (err < 0) { bt_dev_err(hdev, "Reset timeout"); + reset_err = err; + } if (dev_id == 0x7922) { err = btmtk_usb_uhw_reg_write(hdev, MTK_UDMA_INT_STA_BT, 0x000000FF); @@ -933,10 +959,12 @@ int btmtk_usb_subsys_reset(struct hci_dev *hdev, u32 dev_id) } err = btmtk_usb_id_get(hdev, 0x70010200, &val); - if (err < 0 || !val) + if (err || (!val && dev_id != 0x6639)) { bt_dev_err(hdev, "Can't get device id, subsys reset fail."); + return err ? err : -ENODEV; + } - return err; + return reset_err; } EXPORT_SYMBOL_GPL(btmtk_usb_subsys_reset); @@ -1295,30 +1323,48 @@ int btmtk_usb_setup(struct hci_dev *hdev) calltime = ktime_get(); err = btmtk_usb_id_get(hdev, 0x80000008, &dev_id); - if (err < 0) { + if (err) { bt_dev_err(hdev, "Failed to get device id (%d)", err); return err; } if (!dev_id || dev_id != 0x7663) { err = btmtk_usb_id_get(hdev, 0x70010200, &dev_id); - if (err < 0) { + if (err) { bt_dev_err(hdev, "Failed to get device id (%d)", err); return err; } err = btmtk_usb_id_get(hdev, 0x80021004, &fw_version); - if (err < 0) { + if (err) { bt_dev_err(hdev, "Failed to get fw version (%d)", err); return err; } err = btmtk_usb_id_get(hdev, 0x70010020, &fw_flavor); - if (err < 0) { + if (err) { bt_dev_err(hdev, "Failed to get fw flavor (%d)", err); return err; } fw_flavor = (fw_flavor & 0x00000080) >> 7; } + if (!dev_id) { + u16 vid = le16_to_cpu(btmtk_data->udev->descriptor.idVendor); + u16 pid = le16_to_cpu(btmtk_data->udev->descriptor.idProduct); + int i; + + for (i = 0; i < ARRAY_SIZE(btmtk_mt6639_devs); i++) { + if (vid == btmtk_mt6639_devs[i].vendor && + pid == btmtk_mt6639_devs[i].product) { + dev_id = 0x6639; + break; + } + } + + if (dev_id) + bt_dev_info(hdev, "MT6639: CHIPID=0x0000 with VID=%04x PID=%04x, using 0x6639", + vid, pid); + } + btmtk_data->dev_id = dev_id; err = btmtk_register_coredump(hdev, btmtk_data->drv_name, fw_version); @@ -1335,11 +1381,14 @@ int btmtk_usb_setup(struct hci_dev *hdev) case 0x7922: case 0x7925: case 0x7961: + case 0x7902: + case 0x6639: btmtk_fw_get_filename(fw_bin_name, sizeof(fw_bin_name), dev_id, fw_version, fw_flavor); err = btmtk_setup_firmware_79xx(hdev, fw_bin_name, - btmtk_usb_hci_wmt_sync); + btmtk_usb_hci_wmt_sync, + dev_id); if (err < 0) { bt_dev_err(hdev, "Failed to set up firmware (%d)", err); return err; @@ -1506,3 +1555,4 @@ MODULE_FIRMWARE(FIRMWARE_MT7668); MODULE_FIRMWARE(FIRMWARE_MT7922); MODULE_FIRMWARE(FIRMWARE_MT7961); MODULE_FIRMWARE(FIRMWARE_MT7925); +MODULE_FIRMWARE(FIRMWARE_MT7927); diff --git a/drivers/bluetooth/btmtk.h b/drivers/bluetooth/btmtk.h index 5df7c3296624..8f40180ca92e 100644 --- a/drivers/bluetooth/btmtk.h +++ b/drivers/bluetooth/btmtk.h @@ -5,8 +5,10 @@ #define FIRMWARE_MT7663 "mediatek/mt7663pr2h.bin" #define FIRMWARE_MT7668 "mediatek/mt7668pr2h.bin" #define FIRMWARE_MT7922 "mediatek/BT_RAM_CODE_MT7922_1_1_hdr.bin" +#define FIRMWARE_MT7902 "mediatek/BT_RAM_CODE_MT7902_1_1_hdr.bin" #define FIRMWARE_MT7961 "mediatek/BT_RAM_CODE_MT7961_1_2_hdr.bin" #define FIRMWARE_MT7925 "mediatek/mt7925/BT_RAM_CODE_MT7925_1_1_hdr.bin" +#define FIRMWARE_MT7927 "mediatek/mt7927/BT_RAM_CODE_MT6639_2_1_hdr.bin" #define HCI_EV_WMT 0xe4 #define HCI_WMT_MAX_EVENT_SIZE 64 @@ -187,7 +189,8 @@ typedef int (*wmt_cmd_sync_func_t)(struct hci_dev *, int btmtk_set_bdaddr(struct hci_dev *hdev, const bdaddr_t *bdaddr); int btmtk_setup_firmware_79xx(struct hci_dev *hdev, const char *fwname, - wmt_cmd_sync_func_t wmt_cmd_sync); + wmt_cmd_sync_func_t wmt_cmd_sync, + u32 dev_id); int btmtk_setup_firmware(struct hci_dev *hdev, const char *fwname, wmt_cmd_sync_func_t wmt_cmd_sync); @@ -226,7 +229,8 @@ static inline int btmtk_set_bdaddr(struct hci_dev *hdev, static inline int btmtk_setup_firmware_79xx(struct hci_dev *hdev, const char *fwname, - wmt_cmd_sync_func_t wmt_cmd_sync) + wmt_cmd_sync_func_t wmt_cmd_sync, + u32 dev_id) { return -EOPNOTSUPP; } diff --git a/drivers/bluetooth/btmtksdio.c b/drivers/bluetooth/btmtksdio.c index d2583bc74263..f7dcabc11b85 100644 --- a/drivers/bluetooth/btmtksdio.c +++ b/drivers/bluetooth/btmtksdio.c @@ -42,24 +42,35 @@ struct btmtksdio_data { const char *fwname; u16 chipid; bool lp_mbox_supported; + bool pm_runtime_supported; }; static const struct btmtksdio_data mt7663_data = { .fwname = FIRMWARE_MT7663, .chipid = 0x7663, .lp_mbox_supported = false, + .pm_runtime_supported = true, }; static const struct btmtksdio_data mt7668_data = { .fwname = FIRMWARE_MT7668, .chipid = 0x7668, .lp_mbox_supported = false, + .pm_runtime_supported = true, }; static const struct btmtksdio_data mt7921_data = { .fwname = FIRMWARE_MT7961, .chipid = 0x7921, .lp_mbox_supported = true, + .pm_runtime_supported = true, +}; + +static const struct btmtksdio_data mt7902_data = { + .fwname = FIRMWARE_MT7902, + .chipid = 0x7902, + .lp_mbox_supported = false, + .pm_runtime_supported = false, }; static const struct sdio_device_id btmtksdio_table[] = { @@ -69,6 +80,8 @@ static const struct sdio_device_id btmtksdio_table[] = { .driver_data = (kernel_ulong_t)&mt7668_data }, {SDIO_DEVICE(SDIO_VENDOR_ID_MEDIATEK, SDIO_DEVICE_ID_MEDIATEK_MT7961), .driver_data = (kernel_ulong_t)&mt7921_data }, + {SDIO_DEVICE(SDIO_VENDOR_ID_MEDIATEK, SDIO_DEVICE_ID_MEDIATEK_MT7902), + .driver_data = (kernel_ulong_t)&mt7902_data }, { } /* Terminating entry */ }; MODULE_DEVICE_TABLE(sdio, btmtksdio_table); @@ -259,12 +272,24 @@ static int btmtksdio_tx_packet(struct btmtksdio_dev *bdev, struct sk_buff *skb) { struct mtkbtsdio_hdr *sdio_hdr; + unsigned int len, pad_len; int err; - /* Make sure that there are enough rooms for SDIO header */ - if (unlikely(skb_headroom(skb) < sizeof(*sdio_hdr))) { - err = pskb_expand_head(skb, sizeof(*sdio_hdr), 0, - GFP_ATOMIC); + /* Make sure that the data buffer is not shared with anyone else and + * that there is enough room for the SDIO header + */ + err = skb_cow_head(skb, sizeof(*sdio_hdr)); + if (err < 0) + return err; + + /* The transfer is rounded up to the SDIO block size, so the buffer + * has to provide tailroom for the padding as well + */ + len = skb->len + sizeof(*sdio_hdr); + pad_len = round_up(len, MTK_SDIO_BLOCK_SIZE) - len; + + if (unlikely(skb_tailroom(skb) < pad_len)) { + err = pskb_expand_head(skb, 0, pad_len, GFP_ATOMIC); if (err < 0) return err; } @@ -277,19 +302,22 @@ static int btmtksdio_tx_packet(struct btmtksdio_dev *bdev, sdio_hdr->reserved = cpu_to_le16(0); sdio_hdr->bt_type = hci_skb_pkt_type(skb); + /* Zero the padding so that no uninitialised memory is sent out */ + skb_put_zero(skb, pad_len); + clear_bit(BTMTKSDIO_HW_TX_READY, &bdev->tx_state); - err = sdio_writesb(bdev->func, MTK_REG_CTDR, skb->data, - round_up(skb->len, MTK_SDIO_BLOCK_SIZE)); + err = sdio_writesb(bdev->func, MTK_REG_CTDR, skb->data, skb->len); if (err < 0) - goto err_skb_pull; + goto err_skb_restore; - bdev->hdev->stat.byte_tx += skb->len; + bdev->hdev->stat.byte_tx += len; kfree_skb(skb); return 0; -err_skb_pull: +err_skb_restore: + skb_trim(skb, len); skb_pull(skb, sizeof(*sdio_hdr)); return err; @@ -871,7 +899,7 @@ static int mt79xx_setup(struct hci_dev *hdev, const char *fwname) u8 param = 0x1; int err; - err = btmtk_setup_firmware_79xx(hdev, fwname, mtk_hci_wmt_sync); + err = btmtk_setup_firmware_79xx(hdev, fwname, mtk_hci_wmt_sync, 0); if (err < 0) { bt_dev_err(hdev, "Failed to setup 79xx firmware (%d)", err); return err; @@ -1091,6 +1119,7 @@ static int btmtksdio_setup(struct hci_dev *hdev) set_bit(BTMTKSDIO_HW_TX_READY, &bdev->tx_state); switch (bdev->data->chipid) { + case 0x7902: case 0x7921: if (test_bit(BTMTKSDIO_HW_RESET_ACTIVE, &bdev->tx_state)) { err = btmtksdio_mtk_reg_read(hdev, MT7921_DLSTATUS, @@ -1168,22 +1197,24 @@ static int btmtksdio_setup(struct hci_dev *hdev) delta = ktime_sub(rettime, calltime); duration = (unsigned long long)ktime_to_ns(delta) >> 10; - pm_runtime_set_autosuspend_delay(bdev->dev, - MTKBTSDIO_AUTOSUSPEND_DELAY); - pm_runtime_use_autosuspend(bdev->dev); + if (bdev->data->pm_runtime_supported) { + pm_runtime_set_autosuspend_delay(bdev->dev, + MTKBTSDIO_AUTOSUSPEND_DELAY); + pm_runtime_use_autosuspend(bdev->dev); - err = pm_runtime_set_active(bdev->dev); - if (err < 0) - return err; + err = pm_runtime_set_active(bdev->dev); + if (err < 0) + return err; - /* Default forbid runtime auto suspend, that can be allowed by - * enable_autosuspend flag or the PM runtime entry under sysfs. - */ - pm_runtime_forbid(bdev->dev); - pm_runtime_enable(bdev->dev); + /* Default forbid runtime auto suspend, that can be allowed by + * enable_autosuspend flag or the PM runtime entry under sysfs. + */ + pm_runtime_forbid(bdev->dev); + pm_runtime_enable(bdev->dev); - if (enable_autosuspend) - pm_runtime_allow(bdev->dev); + if (enable_autosuspend) + pm_runtime_allow(bdev->dev); + } bt_dev_info(hdev, "Device setup in %llu usecs", duration); @@ -1465,6 +1496,9 @@ static void btmtksdio_remove(struct sdio_func *func) if (test_bit(BTMTKSDIO_FUNC_ENABLED, &bdev->tx_state)) btmtksdio_close(hdev); + if (bdev->data->pm_runtime_supported) + pm_runtime_dont_use_autosuspend(bdev->dev); + /* Be consistent the state in btmtksdio_probe */ pm_runtime_get_noresume(bdev->dev); diff --git a/drivers/bluetooth/btnxpuart.c b/drivers/bluetooth/btnxpuart.c index 970da4f84e60..509d837e353e 100644 --- a/drivers/bluetooth/btnxpuart.c +++ b/drivers/bluetooth/btnxpuart.c @@ -1368,12 +1368,21 @@ static int nxp_process_fw_dump(struct hci_dev *hdev, struct sk_buff *skb) { struct hci_acl_hdr *acl_hdr = (struct hci_acl_hdr *)skb_pull_data(skb, sizeof(*acl_hdr)); - struct nxp_fw_dump_hdr *fw_dump_hdr = (struct nxp_fw_dump_hdr *)skb->data; + struct nxp_fw_dump_hdr *fw_dump_hdr; struct btnxpuart_dev *nxpdev = hci_get_drvdata(hdev); - __u16 seq_num = __le16_to_cpu(fw_dump_hdr->seq_num); - __u16 buf_len = __le16_to_cpu(fw_dump_hdr->buf_len); + __u16 seq_num; + __u16 buf_len; int err; + fw_dump_hdr = skb_pull_data(skb, sizeof(*fw_dump_hdr)); + if (!fw_dump_hdr) { + bt_dev_warn(hdev, "FW dump: invalid or corrupt fw dump chunk"); + goto free_skb; + } + + seq_num = __le16_to_cpu(fw_dump_hdr->seq_num); + buf_len = __le16_to_cpu(fw_dump_hdr->buf_len); + if (seq_num == 0x0001) { if (test_and_set_bit(BTNXPUART_FW_DUMP_IN_PROGRESS, &nxpdev->tx_state)) { bt_dev_err(hdev, "FW dump already in progress"); diff --git a/drivers/bluetooth/btqca.c b/drivers/bluetooth/btqca.c index afab479ac894..f404eefbfc85 100644 --- a/drivers/bluetooth/btqca.c +++ b/drivers/bluetooth/btqca.c @@ -1014,8 +1014,7 @@ int qca_set_bdaddr(struct hci_dev *hdev, const bdaddr_t *bdaddr) baswap(&bdaddr_swapped, bdaddr); skb = __hci_cmd_sync_ev(hdev, EDL_WRITE_BD_ADDR_OPCODE, 6, - &bdaddr_swapped, HCI_EV_VENDOR, - HCI_INIT_TIMEOUT); + &bdaddr_swapped, 0, HCI_INIT_TIMEOUT); if (IS_ERR(skb)) { err = PTR_ERR(skb); bt_dev_err(hdev, "QCA Change address cmd failed (%d)", err); diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c index a869fd15bc89..68054ae289c2 100644 --- a/drivers/bluetooth/btusb.c +++ b/drivers/bluetooth/btusb.c @@ -976,6 +976,7 @@ struct btusb_data { bool usb_alt6_packet_flow; int isoc_altsetting; int suspend_count; + const struct usb_device_id *match_id; int (*recv_event)(struct hci_dev *hdev, struct sk_buff *skb); int (*recv_acl)(struct hci_dev *hdev, struct sk_buff *skb); @@ -3006,14 +3007,15 @@ static int btusb_set_bdaddr_ath3012(struct hci_dev *hdev, static int btusb_set_bdaddr_wcn6855(struct hci_dev *hdev, const bdaddr_t *bdaddr) { + bdaddr_t bdaddr_swapped; struct sk_buff *skb; - u8 buf[6]; long ret; - memcpy(buf, bdaddr, sizeof(bdaddr_t)); + baswap(&bdaddr_swapped, bdaddr); - skb = __hci_cmd_sync_ev(hdev, 0xfc14, sizeof(buf), buf, - HCI_EV_CMD_COMPLETE, HCI_INIT_TIMEOUT); + skb = __hci_cmd_sync_ev(hdev, 0xfc14, sizeof(bdaddr_swapped), + &bdaddr_swapped, HCI_EV_CMD_COMPLETE, + HCI_INIT_TIMEOUT); if (IS_ERR(skb)) { ret = PTR_ERR(skb); bt_dev_err(hdev, "Change address command failed (%ld)", ret); @@ -3629,8 +3631,10 @@ static int btusb_setup_qca(struct hci_dev *hdev) if (err) return err; - btdata->qca_dump.fw_version = le32_to_cpu(ver.patch_version); - btdata->qca_dump.controller_id = le32_to_cpu(ver.rom_version); + if (btdata->match_id->driver_info & BTUSB_QCA_WCN6855) { + btdata->qca_dump.fw_version = le32_to_cpu(ver.patch_version); + btdata->qca_dump.controller_id = le32_to_cpu(ver.rom_version); + } if (!(status & QCA_SYSCFG_UPDATED)) { err = btusb_setup_qca_load_nvm(hdev, &ver, info); @@ -3657,31 +3661,14 @@ static inline int __set_diag_interface(struct hci_dev *hdev) { struct btusb_data *data = hci_get_drvdata(hdev); struct usb_interface *intf = data->diag; - int i; + int ret; if (!data->diag) return -ENODEV; - data->diag_tx_ep = NULL; - data->diag_rx_ep = NULL; - - for (i = 0; i < intf->cur_altsetting->desc.bNumEndpoints; i++) { - struct usb_endpoint_descriptor *ep_desc; - - ep_desc = &intf->cur_altsetting->endpoint[i].desc; - - if (!data->diag_tx_ep && usb_endpoint_is_bulk_out(ep_desc)) { - data->diag_tx_ep = ep_desc; - continue; - } - - if (!data->diag_rx_ep && usb_endpoint_is_bulk_in(ep_desc)) { - data->diag_rx_ep = ep_desc; - continue; - } - } - - if (!data->diag_tx_ep || !data->diag_rx_ep) { + ret = usb_find_common_endpoints(intf->cur_altsetting, &data->diag_rx_ep, + &data->diag_tx_ep, NULL, NULL); + if (ret) { bt_dev_err(hdev, "invalid diagnostic descriptors"); return -ENODEV; } @@ -4012,12 +3999,11 @@ static struct hci_drv btusb_hci_drv = { static int btusb_probe(struct usb_interface *intf, const struct usb_device_id *id) { - struct usb_endpoint_descriptor *ep_desc; struct gpio_desc *reset_gpio; struct btusb_data *data; struct hci_dev *hdev; unsigned ifnum_base; - int i, err, priv_size; + int err, priv_size; BT_DBG("intf %p id %p", intf, id); @@ -4054,26 +4040,10 @@ static int btusb_probe(struct usb_interface *intf, if (!data) return -ENOMEM; - for (i = 0; i < intf->cur_altsetting->desc.bNumEndpoints; i++) { - ep_desc = &intf->cur_altsetting->endpoint[i].desc; - - if (!data->intr_ep && usb_endpoint_is_int_in(ep_desc)) { - data->intr_ep = ep_desc; - continue; - } - - if (!data->bulk_tx_ep && usb_endpoint_is_bulk_out(ep_desc)) { - data->bulk_tx_ep = ep_desc; - continue; - } - - if (!data->bulk_rx_ep && usb_endpoint_is_bulk_in(ep_desc)) { - data->bulk_rx_ep = ep_desc; - continue; - } - } - - if (!data->intr_ep || !data->bulk_tx_ep || !data->bulk_rx_ep) { + data->match_id = id; + err = usb_find_common_endpoints(intf->cur_altsetting, &data->bulk_rx_ep, + &data->bulk_tx_ep, &data->intr_ep, NULL); + if (err) { kfree(data); return -ENODEV; } diff --git a/drivers/bluetooth/virtio_bt.c b/drivers/bluetooth/virtio_bt.c index 9341c5eed5b5..152432cc2260 100644 --- a/drivers/bluetooth/virtio_bt.c +++ b/drivers/bluetooth/virtio_bt.c @@ -120,9 +120,13 @@ static int virtbt_setup_zephyr(struct hci_dev *hdev) if (IS_ERR(skb)) return PTR_ERR(skb); - bt_dev_info(hdev, "%s", (char *)(skb->data + 1)); + /* Bounded print: the backend controls skb->len. */ + if (skb->len > 1) { + int len = skb->len - 1; - hci_set_fw_info(hdev, "%s", skb->data + 1); + bt_dev_info(hdev, "%.*s", len, (char *)(skb->data + 1)); + hci_set_fw_info(hdev, "%.*s", len, skb->data + 1); + } kfree_skb(skb); return 0; diff --git a/drivers/bus/mhi/ep/main.c b/drivers/bus/mhi/ep/main.c index 2f58ad0f14b6..fd1d7c84fb1c 100644 --- a/drivers/bus/mhi/ep/main.c +++ b/drivers/bus/mhi/ep/main.c @@ -1340,14 +1340,19 @@ static int mhi_ep_create_device(struct mhi_ep_cntrl *mhi_cntrl, u32 ch_id) ret = dev_set_name(&mhi_dev->dev, "%s_%s", dev_name(&mhi_cntrl->mhi_dev->dev), mhi_dev->name); - if (ret) { - put_device(&mhi_dev->dev); - return ret; - } + if (ret) + goto err_put_channels; ret = device_add(&mhi_dev->dev); if (ret) - put_device(&mhi_dev->dev); + goto err_put_channels; + + return 0; + +err_put_channels: + put_device(&mhi_dev->dev); /* DL channel reference */ + put_device(&mhi_dev->dev); /* UL channel reference */ + put_device(&mhi_dev->dev); /* device_initialize() reference */ return ret; } diff --git a/drivers/bus/mhi/host/init.c b/drivers/bus/mhi/host/init.c index 099be8dd1900..504a5d3f7830 100644 --- a/drivers/bus/mhi/host/init.c +++ b/drivers/bus/mhi/host/init.c @@ -1042,7 +1042,7 @@ int mhi_register_controller(struct mhi_controller *mhi_cntrl, if (mhi_cntrl->edl_trigger) { ret = sysfs_create_file(&mhi_dev->dev.kobj, &dev_attr_trigger_edl.attr); if (ret) - goto err_release_dev; + goto err_del_dev; } mhi_cntrl->mhi_dev = mhi_dev; @@ -1051,6 +1051,8 @@ int mhi_register_controller(struct mhi_controller *mhi_cntrl, return 0; +err_del_dev: + device_del(&mhi_dev->dev); err_release_dev: put_device(&mhi_dev->dev); error_setup_irq: diff --git a/drivers/bus/mhi/host/main.c b/drivers/bus/mhi/host/main.c index 861551274319..eeb277dee748 100644 --- a/drivers/bus/mhi/host/main.c +++ b/drivers/bus/mhi/host/main.c @@ -170,6 +170,9 @@ EXPORT_SYMBOL_GPL(mhi_get_mhi_state); void mhi_soc_reset(struct mhi_controller *mhi_cntrl) { + int __maybe_unused ret; + u32 tmp; + if (mhi_cntrl->reset) { mhi_cntrl->reset(mhi_cntrl); return; @@ -178,6 +181,9 @@ void mhi_soc_reset(struct mhi_controller *mhi_cntrl) /* Generic MHI SoC reset */ mhi_write_reg(mhi_cntrl, mhi_cntrl->regs, MHI_SOC_RESET_REQ_OFFSET, MHI_SOC_RESET_REQ); + /* Flush the posted write to the device (ignore return value) */ + ret = mhi_read_reg(mhi_cntrl, mhi_cntrl->regs, MHI_SOC_RESET_REQ_OFFSET, + &tmp); } EXPORT_SYMBOL_GPL(mhi_soc_reset); diff --git a/drivers/bus/mvebu-mbus.c b/drivers/bus/mvebu-mbus.c index 00cb792bda18..dd94145c9b22 100644 --- a/drivers/bus/mvebu-mbus.c +++ b/drivers/bus/mvebu-mbus.c @@ -1006,7 +1006,7 @@ static __init int mvebu_mbus_debugfs_init(void) } fs_initcall(mvebu_mbus_debugfs_init); -static int mvebu_mbus_suspend(void) +static int mvebu_mbus_suspend(void *data) { struct mvebu_mbus_state *s = &mbus_state; int win; @@ -1040,7 +1040,7 @@ static int mvebu_mbus_suspend(void) return 0; } -static void mvebu_mbus_resume(void) +static void mvebu_mbus_resume(void *data) { struct mvebu_mbus_state *s = &mbus_state; int win; @@ -1069,9 +1069,13 @@ static void mvebu_mbus_resume(void) } } -static struct syscore_ops mvebu_mbus_syscore_ops = { - .suspend = mvebu_mbus_suspend, - .resume = mvebu_mbus_resume, +static const struct syscore_ops mvebu_mbus_syscore_ops = { + .suspend = mvebu_mbus_suspend, + .resume = mvebu_mbus_resume, +}; + +static struct syscore mvebu_mbus_syscore = { + .ops = &mvebu_mbus_syscore_ops, }; static int __init mvebu_mbus_common_init(struct mvebu_mbus_state *mbus, @@ -1118,7 +1122,7 @@ static int __init mvebu_mbus_common_init(struct mvebu_mbus_state *mbus, writel(UNIT_SYNC_BARRIER_ALL, mbus->mbuswins_base + UNIT_SYNC_BARRIER_OFF); - register_syscore_ops(&mvebu_mbus_syscore_ops); + register_syscore(&mvebu_mbus_syscore); return 0; } diff --git a/drivers/bus/qcom-ebi2.c b/drivers/bus/qcom-ebi2.c index c1fef1b4bd89..8d2eb955dc92 100644 --- a/drivers/bus/qcom-ebi2.c +++ b/drivers/bus/qcom-ebi2.c @@ -292,7 +292,6 @@ static void qcom_ebi2_setup_chipselect(struct device_node *np, static int qcom_ebi2_probe(struct platform_device *pdev) { struct device_node *np = pdev->dev.of_node; - struct device_node *child; struct device *dev = &pdev->dev; struct resource *res; void __iomem *ebi2_base; @@ -303,41 +302,23 @@ static int qcom_ebi2_probe(struct platform_device *pdev) u32 val; int ret; - ebi2xclk = devm_clk_get(dev, "ebi2x"); + ebi2xclk = devm_clk_get_enabled(dev, "ebi2x"); if (IS_ERR(ebi2xclk)) return PTR_ERR(ebi2xclk); - ret = clk_prepare_enable(ebi2xclk); - if (ret) { - dev_err(dev, "could not enable EBI2X clk (%d)\n", ret); - return ret; - } - - ebi2clk = devm_clk_get(dev, "ebi2"); - if (IS_ERR(ebi2clk)) { - ret = PTR_ERR(ebi2clk); - goto err_disable_2x_clk; - } - - ret = clk_prepare_enable(ebi2clk); - if (ret) { - dev_err(dev, "could not enable EBI2 clk\n"); - goto err_disable_2x_clk; - } + ebi2clk = devm_clk_get_enabled(dev, "ebi2"); + if (IS_ERR(ebi2clk)) + return PTR_ERR(ebi2clk); res = platform_get_resource(pdev, IORESOURCE_MEM, 0); ebi2_base = devm_ioremap_resource(dev, res); - if (IS_ERR(ebi2_base)) { - ret = PTR_ERR(ebi2_base); - goto err_disable_clk; - } + if (IS_ERR(ebi2_base)) + return PTR_ERR(ebi2_base); res = platform_get_resource(pdev, IORESOURCE_MEM, 1); ebi2_xmem = devm_ioremap_resource(dev, res); - if (IS_ERR(ebi2_xmem)) { - ret = PTR_ERR(ebi2_xmem); - goto err_disable_clk; - } + if (IS_ERR(ebi2_xmem)) + return PTR_ERR(ebi2_xmem); /* Allegedly this turns the power save mode off */ writel(0UL, ebi2_xmem + EBI2_XMEM_CFG); @@ -348,15 +329,13 @@ static int qcom_ebi2_probe(struct platform_device *pdev) writel(val, ebi2_base); /* Walk over the child nodes and see what chipselects we use */ - for_each_available_child_of_node(np, child) { + for_each_available_child_of_node_scoped(np, child) { u32 csindex; /* Figure out the chipselect */ ret = of_property_read_u32(child, "reg", &csindex); - if (ret) { - of_node_put(child); + if (ret) return ret; - } if (csindex > 5) { dev_err(dev, @@ -376,15 +355,9 @@ static int qcom_ebi2_probe(struct platform_device *pdev) } if (have_children) - return of_platform_default_populate(np, NULL, dev); - return 0; - -err_disable_clk: - clk_disable_unprepare(ebi2clk); -err_disable_2x_clk: - clk_disable_unprepare(ebi2xclk); + return devm_of_platform_populate(dev); - return ret; + return 0; } static const struct of_device_id qcom_ebi2_of_match[] = { diff --git a/drivers/bus/ti-sysc.c b/drivers/bus/ti-sysc.c index 610354ce7f8f..5c096eb3f427 100644 --- a/drivers/bus/ti-sysc.c +++ b/drivers/bus/ti-sysc.c @@ -682,6 +682,7 @@ static struct device_node *stdout_path; static void sysc_init_stdout_path(struct sysc *ddata) { + struct device_node *chosen; struct device_node *np = NULL; const char *uart; @@ -691,15 +692,18 @@ static void sysc_init_stdout_path(struct sysc *ddata) if (stdout_path) return; - np = of_find_node_by_path("/chosen"); - if (!np) + chosen = of_find_node_by_path("/chosen"); + if (!chosen) goto err; - uart = of_get_property(np, "stdout-path", NULL); - if (!uart) + uart = of_get_property(chosen, "stdout-path", NULL); + if (!uart) { + of_node_put(chosen); goto err; + } np = of_find_node_by_path(uart); + of_node_put(chosen); if (!np) goto err; diff --git a/drivers/char/hw_random/core.c b/drivers/char/hw_random/core.c index 036de7294bbd..054ffbc9e422 100644 --- a/drivers/char/hw_random/core.c +++ b/drivers/char/hw_random/core.c @@ -604,11 +604,13 @@ int hwrng_register(struct hwrng *rng) */ err = set_current_rng(rng); if (err) - goto out_unlock; + goto out_list_del; } } mutex_unlock(&rng_mutex); return 0; +out_list_del: + list_del_init(&rng->list); out_unlock: mutex_unlock(&rng_mutex); out: diff --git a/drivers/char/hw_random/imx-rngc.c b/drivers/char/hw_random/imx-rngc.c index 241664a9b5d9..447424f7ded1 100644 --- a/drivers/char/hw_random/imx-rngc.c +++ b/drivers/char/hw_random/imx-rngc.c @@ -314,8 +314,10 @@ static int __init imx_rngc_probe(struct platform_device *pdev) devm_pm_runtime_enable(&pdev->dev); ret = devm_hwrng_register(&pdev->dev, &rngc->rng); - if (ret) + if (ret) { + clk_disable_unprepare(rngc->clk); return dev_err_probe(&pdev->dev, ret, "hwrng registration failed\n"); + } dev_info(&pdev->dev, "Freescale RNG%c registered (HW revision %d.%02d)\n", diff --git a/drivers/char/hw_random/ks-sa-rng.c b/drivers/char/hw_random/ks-sa-rng.c index 9e408144a10c..4494f1e4ab4d 100644 --- a/drivers/char/hw_random/ks-sa-rng.c +++ b/drivers/char/hw_random/ks-sa-rng.c @@ -242,7 +242,14 @@ static int ks_sa_rng_probe(struct platform_device *pdev) return dev_err_probe(dev, ret, "Failed to enable SA power-domain\n"); } - return devm_hwrng_register(&pdev->dev, &ks_sa_rng->rng); + ret = devm_hwrng_register(dev, &ks_sa_rng->rng); + if (ret) { + pm_runtime_put_sync(dev); + pm_runtime_disable(dev); + return ret; + } + + return 0; } static void ks_sa_rng_remove(struct platform_device *pdev) diff --git a/drivers/char/ppdev.c b/drivers/char/ppdev.c index d1dfbd8d4d42..3dec6516a5eb 100644 --- a/drivers/char/ppdev.c +++ b/drivers/char/ppdev.c @@ -340,15 +340,17 @@ static enum ieee1284_phase init_phase(int mode) return IEEE1284_PH_FWD_IDLE; } -static int pp_set_timeout(struct pardevice *pdev, long tv_sec, int tv_usec) +static int pp_set_timeout(struct pardevice *pdev, s64 tv_sec, s64 tv_usec) { + struct timespec64 ts; long to_jiffies; - if ((tv_sec < 0) || (tv_usec < 0)) + if (tv_sec < 0 || tv_usec < 0 || tv_usec >= USEC_PER_SEC) return -EINVAL; - to_jiffies = usecs_to_jiffies(tv_usec); - to_jiffies += tv_sec * HZ; + ts.tv_sec = tv_sec; + ts.tv_nsec = tv_usec * NSEC_PER_USEC; + to_jiffies = timespec64_to_jiffies(&ts); if (to_jiffies <= 0) return -EINVAL; diff --git a/drivers/char/tpm/st33zp24/st33zp24.c b/drivers/char/tpm/st33zp24/st33zp24.c index e2b7451ea7cc..0e2deff94c36 100644 --- a/drivers/char/tpm/st33zp24/st33zp24.c +++ b/drivers/char/tpm/st33zp24/st33zp24.c @@ -93,7 +93,9 @@ static u8 st33zp24_status(struct tpm_chip *chip) struct st33zp24_dev *tpm_dev = dev_get_drvdata(&chip->dev); u8 data; - tpm_dev->ops->recv(tpm_dev->phy_id, TPM_STS, &data, 1); + if (tpm_dev->ops->recv(tpm_dev->phy_id, TPM_STS, &data, 1) != 1) + return 0; + return data; } @@ -104,10 +106,10 @@ static bool check_locality(struct tpm_chip *chip) { struct st33zp24_dev *tpm_dev = dev_get_drvdata(&chip->dev); u8 data; - u8 status; + int status; status = tpm_dev->ops->recv(tpm_dev->phy_id, TPM_ACCESS, &data, 1); - if (status && (data & + if (status == 1 && (data & (TPM_ACCESS_ACTIVE_LOCALITY | TPM_ACCESS_VALID)) == (TPM_ACCESS_ACTIVE_LOCALITY | TPM_ACCESS_VALID)) return true; diff --git a/drivers/char/xilinx_hwicap/xilinx_hwicap.c b/drivers/char/xilinx_hwicap/xilinx_hwicap.c index 34a345dc5e72..9bb5fa642fd8 100644 --- a/drivers/char/xilinx_hwicap/xilinx_hwicap.c +++ b/drivers/char/xilinx_hwicap/xilinx_hwicap.c @@ -760,7 +760,7 @@ static int __init hwicap_module_init(void) HWICAP_DEVICES, DRIVER_NAME); if (retval < 0) - return retval; + goto failed_class; retval = platform_driver_register(&hwicap_platform_driver); if (retval) @@ -771,6 +771,9 @@ static int __init hwicap_module_init(void) failed: unregister_chrdev_region(devt, HWICAP_DEVICES); + failed_class: + class_unregister(&icap_class); + return retval; } diff --git a/drivers/clk/at91/pmc.c b/drivers/clk/at91/pmc.c index 4ef4f7b9d0bd..073b2d506f78 100644 --- a/drivers/clk/at91/pmc.c +++ b/drivers/clk/at91/pmc.c @@ -115,7 +115,7 @@ struct pmc_data *pmc_data_allocate(unsigned int ncore, unsigned int nsystem, /* Address in SECURAM that say if we suspend to backup mode. */ static void __iomem *at91_pmc_backup_suspend; -static int at91_pmc_suspend(void) +static int at91_pmc_suspend(void *data) { unsigned int backup; @@ -129,7 +129,7 @@ static int at91_pmc_suspend(void) return clk_save_context(); } -static void at91_pmc_resume(void) +static void at91_pmc_resume(void *data) { unsigned int backup; @@ -143,11 +143,15 @@ static void at91_pmc_resume(void) clk_restore_context(); } -static struct syscore_ops pmc_syscore_ops = { +static const struct syscore_ops pmc_syscore_ops = { .suspend = at91_pmc_suspend, .resume = at91_pmc_resume, }; +static struct syscore pmc_syscore = { + .ops = &pmc_syscore_ops, +}; + static const struct of_device_id pmc_dt_ids[] = { { .compatible = "atmel,sama5d2-pmc" }, { .compatible = "microchip,sama7g5-pmc", }, @@ -185,7 +189,7 @@ static int __init pmc_register_ops(void) return -ENOMEM; } - register_syscore_ops(&pmc_syscore_ops); + register_syscore(&pmc_syscore); return 0; } diff --git a/drivers/clk/clk-devres.c b/drivers/clk/clk-devres.c index 5368d92d9b39..09fd5e8fb72c 100644 --- a/drivers/clk/clk-devres.c +++ b/drivers/clk/clk-devres.c @@ -99,6 +99,11 @@ struct clk *devm_clk_get_optional_enabled(struct device *dev, const char *id) } EXPORT_SYMBOL_GPL(devm_clk_get_optional_enabled); +static void devm_clk_disable_unprepare(void *data) +{ + clk_disable_unprepare(data); +} + struct clk *devm_clk_get_optional_enabled_with_rate(struct device *dev, const char *id, unsigned long rate) @@ -106,8 +111,7 @@ struct clk *devm_clk_get_optional_enabled_with_rate(struct device *dev, struct clk *clk; int ret; - clk = __devm_clk_get(dev, id, clk_get_optional, NULL, - clk_disable_unprepare); + clk = devm_clk_get_optional(dev, id); if (IS_ERR(clk)) return ERR_CAST(clk); @@ -119,6 +123,10 @@ struct clk *devm_clk_get_optional_enabled_with_rate(struct device *dev, if (ret) goto out_put_clk; + ret = devm_add_action_or_reset(dev, devm_clk_disable_unprepare, clk); + if (ret) + goto out_put_clk; + return clk; out_put_clk: diff --git a/drivers/clk/clk-moxart.c b/drivers/clk/clk-moxart.c index 3786a0153ad1..40663ef3ef0a 100644 --- a/drivers/clk/clk-moxart.c +++ b/drivers/clk/clk-moxart.c @@ -17,7 +17,6 @@ static void __init moxart_of_pll_clk_init(struct device_node *node) { void __iomem *base; struct clk_hw *hw; - struct clk *ref_clk; unsigned int mul; const char *name = node->name; const char *parent_name; @@ -34,12 +33,6 @@ static void __init moxart_of_pll_clk_init(struct device_node *node) mul = readl(base + 0x30) >> 3 & 0x3f; iounmap(base); - ref_clk = of_clk_get(node, 0); - if (IS_ERR(ref_clk)) { - pr_err("%pOF: of_clk_get failed\n", node); - return; - } - hw = clk_hw_register_fixed_factor(NULL, name, parent_name, 0, mul, 1); if (IS_ERR(hw)) { pr_err("%pOF: failed to register clock\n", node); @@ -56,7 +49,6 @@ static void __init moxart_of_apb_clk_init(struct device_node *node) { void __iomem *base; struct clk_hw *hw; - struct clk *pll_clk; unsigned int div, val; unsigned int div_idx[] = { 2, 3, 4, 6, 8}; const char *name = node->name; @@ -78,12 +70,6 @@ static void __init moxart_of_apb_clk_init(struct device_node *node) val = 0; div = div_idx[val] * 2; - pll_clk = of_clk_get(node, 0); - if (IS_ERR(pll_clk)) { - pr_err("%pOF: of_clk_get failed\n", node); - return; - } - hw = clk_hw_register_fixed_factor(NULL, name, parent_name, 0, 1, div); if (IS_ERR(hw)) { pr_err("%pOF: failed to register clock\n", node); diff --git a/drivers/clk/clk-palmas.c b/drivers/clk/clk-palmas.c index 39049f62dbbb..86a51edac827 100644 --- a/drivers/clk/clk-palmas.c +++ b/drivers/clk/clk-palmas.c @@ -194,6 +194,13 @@ static void palmas_clks_get_clk_data(struct platform_device *pdev, cinfo->ext_control_pin = prop; } +static void palmas_clks_unprepare_ext_control(void *data) +{ + struct palmas_clock_info *cinfo = data; + + clk_unprepare(cinfo->hw.clk); +} + static int palmas_clks_init_configure(struct palmas_clock_info *cinfo) { int ret; @@ -214,13 +221,18 @@ static int palmas_clks_init_configure(struct palmas_clock_info *cinfo) return ret; } + ret = devm_add_action_or_reset(cinfo->dev, + palmas_clks_unprepare_ext_control, + cinfo); + if (ret) + return ret; + ret = palmas_ext_control_req_config(cinfo->palmas, cinfo->clk_desc->sleep_reqstr_id, cinfo->ext_control_pin, true); if (ret < 0) { dev_err(cinfo->dev, "Ext config for %s failed, %d\n", cinfo->clk_desc->clk_name, ret); - clk_unprepare(cinfo->hw.clk); return ret; } } diff --git a/drivers/clk/clk-versaclock7.c b/drivers/clk/clk-versaclock7.c index adcc603e3259..e3a36dcd98b8 100644 --- a/drivers/clk/clk-versaclock7.c +++ b/drivers/clk/clk-versaclock7.c @@ -1197,7 +1197,7 @@ static int vc7_probe(struct i2c_client *client) if (ret) { dev_err_probe(&client->dev, ret, "unable to register output %d\n", i); - return ret; + goto err_clk; } switch (bank_src_map.type) { diff --git a/drivers/clk/hisilicon/reset.c b/drivers/clk/hisilicon/reset.c index 93cee17db8b1..c3b7daac9313 100644 --- a/drivers/clk/hisilicon/reset.c +++ b/drivers/clk/hisilicon/reset.c @@ -91,7 +91,7 @@ struct hisi_reset_controller *hisi_reset_init(struct platform_device *pdev) { struct hisi_reset_controller *rstc; - rstc = devm_kmalloc(&pdev->dev, sizeof(*rstc), GFP_KERNEL); + rstc = devm_kzalloc(&pdev->dev, sizeof(*rstc), GFP_KERNEL); if (!rstc) return NULL; diff --git a/drivers/clk/imx/clk-scu.c b/drivers/clk/imx/clk-scu.c index c03f7821824d..b2b0e5f05b23 100644 --- a/drivers/clk/imx/clk-scu.c +++ b/drivers/clk/imx/clk-scu.c @@ -465,7 +465,6 @@ struct clk_hw *__imx_clk_scu(struct device *dev, const char *name, clk->clk_type = clk_type; init.name = name; - init.ops = &clk_scu_ops; if (rsrc_id == IMX_SC_R_A35 || rsrc_id == IMX_SC_R_A53 || rsrc_id == IMX_SC_R_A72) init.ops = &clk_scu_cpu_ops; else if (rsrc_id == IMX_SC_R_PI_0_PLL) diff --git a/drivers/clk/imx/clk-vf610.c b/drivers/clk/imx/clk-vf610.c index 9e11f1c7c397..41eb38552a9c 100644 --- a/drivers/clk/imx/clk-vf610.c +++ b/drivers/clk/imx/clk-vf610.c @@ -139,7 +139,7 @@ static struct clk * __init vf610_get_fixed_clock( return clk; }; -static int vf610_clk_suspend(void) +static int vf610_clk_suspend(void *data) { int i; @@ -156,7 +156,7 @@ static int vf610_clk_suspend(void) return 0; } -static void vf610_clk_resume(void) +static void vf610_clk_resume(void *data) { int i; @@ -171,11 +171,15 @@ static void vf610_clk_resume(void) writel_relaxed(ccgr[i], CCM_CCGRx(i)); } -static struct syscore_ops vf610_clk_syscore_ops = { +static const struct syscore_ops vf610_clk_syscore_ops = { .suspend = vf610_clk_suspend, .resume = vf610_clk_resume, }; +static struct syscore vf610_clk_syscore = { + .ops = &vf610_clk_syscore_ops, +}; + static void __init vf610_clocks_init(struct device_node *ccm_node) { struct device_node *np; @@ -462,7 +466,7 @@ static void __init vf610_clocks_init(struct device_node *ccm_node) for (i = 0; i < ARRAY_SIZE(clks_init_on); i++) clk_prepare_enable(clk[clks_init_on[i]]); - register_syscore_ops(&vf610_clk_syscore_ops); + register_syscore(&vf610_clk_syscore); /* Add the clocks to provider list */ clk_data.clks = clk; diff --git a/drivers/clk/ingenic/jz4725b-cgu.c b/drivers/clk/ingenic/jz4725b-cgu.c index 590e9c85cb25..94cee44c854f 100644 --- a/drivers/clk/ingenic/jz4725b-cgu.c +++ b/drivers/clk/ingenic/jz4725b-cgu.c @@ -268,6 +268,6 @@ static void __init jz4725b_cgu_init(struct device_node *np) if (retval) pr_err("%s: failed to register CGU Clocks\n", __func__); - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } CLK_OF_DECLARE_DRIVER(jz4725b_cgu, "ingenic,jz4725b-cgu", jz4725b_cgu_init); diff --git a/drivers/clk/ingenic/jz4740-cgu.c b/drivers/clk/ingenic/jz4740-cgu.c index 3e0a30574ebb..2def3aedc8dd 100644 --- a/drivers/clk/ingenic/jz4740-cgu.c +++ b/drivers/clk/ingenic/jz4740-cgu.c @@ -266,6 +266,6 @@ static void __init jz4740_cgu_init(struct device_node *np) if (retval) pr_err("%s: failed to register CGU Clocks\n", __func__); - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } CLK_OF_DECLARE_DRIVER(jz4740_cgu, "ingenic,jz4740-cgu", jz4740_cgu_init); diff --git a/drivers/clk/ingenic/jz4755-cgu.c b/drivers/clk/ingenic/jz4755-cgu.c index f2c2d848dab7..17cf5dcaece9 100644 --- a/drivers/clk/ingenic/jz4755-cgu.c +++ b/drivers/clk/ingenic/jz4755-cgu.c @@ -337,7 +337,7 @@ static void __init jz4755_cgu_init(struct device_node *np) if (retval) pr_err("%s: failed to register CGU Clocks\n", __func__); - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } /* * CGU has some children devices, this is useful for probing children devices diff --git a/drivers/clk/ingenic/jz4760-cgu.c b/drivers/clk/ingenic/jz4760-cgu.c index e407f00bd594..372fe4b07992 100644 --- a/drivers/clk/ingenic/jz4760-cgu.c +++ b/drivers/clk/ingenic/jz4760-cgu.c @@ -436,7 +436,7 @@ static void __init jz4760_cgu_init(struct device_node *np) if (retval) pr_err("%s: failed to register CGU Clocks\n", __func__); - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } /* We only probe via devicetree, no need for a platform driver */ diff --git a/drivers/clk/ingenic/jz4770-cgu.c b/drivers/clk/ingenic/jz4770-cgu.c index 6ae1740367f9..58f1d3bad677 100644 --- a/drivers/clk/ingenic/jz4770-cgu.c +++ b/drivers/clk/ingenic/jz4770-cgu.c @@ -456,7 +456,7 @@ static void __init jz4770_cgu_init(struct device_node *np) if (retval) pr_err("%s: failed to register CGU Clocks\n", __func__); - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } /* We only probe via devicetree, no need for a platform driver */ diff --git a/drivers/clk/ingenic/jz4780-cgu.c b/drivers/clk/ingenic/jz4780-cgu.c index 07e2f3c5c454..1e88aef7ac0f 100644 --- a/drivers/clk/ingenic/jz4780-cgu.c +++ b/drivers/clk/ingenic/jz4780-cgu.c @@ -803,6 +803,6 @@ static void __init jz4780_cgu_init(struct device_node *np) return; } - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } CLK_OF_DECLARE_DRIVER(jz4780_cgu, "ingenic,jz4780-cgu", jz4780_cgu_init); diff --git a/drivers/clk/ingenic/pm.c b/drivers/clk/ingenic/pm.c index 341752b640d2..206d5cf2872f 100644 --- a/drivers/clk/ingenic/pm.c +++ b/drivers/clk/ingenic/pm.c @@ -15,7 +15,7 @@ static void __iomem * __maybe_unused ingenic_cgu_base; -static int __maybe_unused ingenic_cgu_pm_suspend(void) +static int __maybe_unused ingenic_cgu_pm_suspend(void *data) { u32 val = readl(ingenic_cgu_base + CGU_REG_LCR); @@ -24,22 +24,26 @@ static int __maybe_unused ingenic_cgu_pm_suspend(void) return 0; } -static void __maybe_unused ingenic_cgu_pm_resume(void) +static void __maybe_unused ingenic_cgu_pm_resume(void *data) { u32 val = readl(ingenic_cgu_base + CGU_REG_LCR); writel(val & ~LCR_LOW_POWER_MODE, ingenic_cgu_base + CGU_REG_LCR); } -static struct syscore_ops __maybe_unused ingenic_cgu_pm_ops = { +static const struct syscore_ops __maybe_unused ingenic_cgu_pm_ops = { .suspend = ingenic_cgu_pm_suspend, .resume = ingenic_cgu_pm_resume, }; -void ingenic_cgu_register_syscore_ops(struct ingenic_cgu *cgu) +static struct syscore __maybe_unused ingenic_cgu_pm = { + .ops = &ingenic_cgu_pm_ops, +}; + +void ingenic_cgu_register_syscore(struct ingenic_cgu *cgu) { if (IS_ENABLED(CONFIG_PM_SLEEP)) { ingenic_cgu_base = cgu->base; - register_syscore_ops(&ingenic_cgu_pm_ops); + register_syscore(&ingenic_cgu_pm); } } diff --git a/drivers/clk/ingenic/pm.h b/drivers/clk/ingenic/pm.h index fa7540407b6b..0dcb57dc64cb 100644 --- a/drivers/clk/ingenic/pm.h +++ b/drivers/clk/ingenic/pm.h @@ -7,6 +7,6 @@ struct ingenic_cgu; -void ingenic_cgu_register_syscore_ops(struct ingenic_cgu *cgu); +void ingenic_cgu_register_syscore(struct ingenic_cgu *cgu); #endif /* DRIVERS_CLK_INGENIC_PM_H */ diff --git a/drivers/clk/ingenic/tcu.c b/drivers/clk/ingenic/tcu.c index 7d04ef40b7cf..bc6a51da2072 100644 --- a/drivers/clk/ingenic/tcu.c +++ b/drivers/clk/ingenic/tcu.c @@ -455,7 +455,7 @@ static int __init ingenic_tcu_probe(struct device_node *np) return ret; } -static int __maybe_unused tcu_pm_suspend(void) +static int __maybe_unused tcu_pm_suspend(void *data) { struct ingenic_tcu *tcu = ingenic_tcu; @@ -465,7 +465,7 @@ static int __maybe_unused tcu_pm_suspend(void) return 0; } -static void __maybe_unused tcu_pm_resume(void) +static void __maybe_unused tcu_pm_resume(void *data) { struct ingenic_tcu *tcu = ingenic_tcu; @@ -473,11 +473,15 @@ static void __maybe_unused tcu_pm_resume(void) clk_enable(tcu->clk); } -static struct syscore_ops __maybe_unused tcu_pm_ops = { +static const struct syscore_ops __maybe_unused tcu_pm_ops = { .suspend = tcu_pm_suspend, .resume = tcu_pm_resume, }; +static struct syscore __maybe_unused tcu_pm = { + .ops = &tcu_pm_ops, +}; + static void __init ingenic_tcu_init(struct device_node *np) { int ret = ingenic_tcu_probe(np); @@ -486,7 +490,7 @@ static void __init ingenic_tcu_init(struct device_node *np) pr_crit("Failed to initialize TCU clocks: %d\n", ret); if (IS_ENABLED(CONFIG_PM_SLEEP)) - register_syscore_ops(&tcu_pm_ops); + register_syscore(&tcu_pm); } CLK_OF_DECLARE_DRIVER(jz4740_cgu, "ingenic,jz4740-tcu", ingenic_tcu_init); diff --git a/drivers/clk/ingenic/x1000-cgu.c b/drivers/clk/ingenic/x1000-cgu.c index d80886caf393..d89bdfb7c219 100644 --- a/drivers/clk/ingenic/x1000-cgu.c +++ b/drivers/clk/ingenic/x1000-cgu.c @@ -556,7 +556,7 @@ static void __init x1000_cgu_init(struct device_node *np) return; } - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } /* * CGU has some children devices, this is useful for probing children devices diff --git a/drivers/clk/ingenic/x1830-cgu.c b/drivers/clk/ingenic/x1830-cgu.c index 0fd46e50a513..acf856e5009e 100644 --- a/drivers/clk/ingenic/x1830-cgu.c +++ b/drivers/clk/ingenic/x1830-cgu.c @@ -463,7 +463,7 @@ static void __init x1830_cgu_init(struct device_node *np) return; } - ingenic_cgu_register_syscore_ops(cgu); + ingenic_cgu_register_syscore(cgu); } /* * CGU has some children devices, this is useful for probing children devices diff --git a/drivers/clk/mediatek/clk-mt2701.c b/drivers/clk/mediatek/clk-mt2701.c index 1e88ad8b93f4..d9f40fda73d1 100644 --- a/drivers/clk/mediatek/clk-mt2701.c +++ b/drivers/clk/mediatek/clk-mt2701.c @@ -978,7 +978,7 @@ static int mtk_apmixedsys_init(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - mtk_clk_register_plls(node, apmixed_plls, ARRAY_SIZE(apmixed_plls), + mtk_clk_register_plls(&pdev->dev, apmixed_plls, ARRAY_SIZE(apmixed_plls), clk_data); mtk_clk_register_factors(apmixed_fixed_divs, ARRAY_SIZE(apmixed_fixed_divs), clk_data); diff --git a/drivers/clk/mediatek/clk-mt2712-apmixedsys.c b/drivers/clk/mediatek/clk-mt2712-apmixedsys.c index a60622d251ff..54b18e9f83f8 100644 --- a/drivers/clk/mediatek/clk-mt2712-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt2712-apmixedsys.c @@ -119,7 +119,7 @@ static int clk_mt2712_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - r = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + r = mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); if (r) goto free_clk_data; diff --git a/drivers/clk/mediatek/clk-mt6735-apmixedsys.c b/drivers/clk/mediatek/clk-mt6735-apmixedsys.c index e0949911e8f7..b6eb6a581c31 100644 --- a/drivers/clk/mediatek/clk-mt6735-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt6735-apmixedsys.c @@ -93,8 +93,8 @@ static int clk_mt6735_apmixed_probe(struct platform_device *pdev) return -ENOMEM; platform_set_drvdata(pdev, clk_data); - ret = mtk_clk_register_plls(pdev->dev.of_node, apmixedsys_plls, - ARRAY_SIZE(apmixedsys_plls), clk_data); + ret = mtk_clk_register_plls(&pdev->dev, apmixedsys_plls, + ARRAY_SIZE(apmixedsys_plls), clk_data); if (ret) { dev_err(&pdev->dev, "Failed to register PLLs: %d\n", ret); return ret; @@ -102,9 +102,12 @@ static int clk_mt6735_apmixed_probe(struct platform_device *pdev) ret = devm_of_clk_add_hw_provider(&pdev->dev, of_clk_hw_onecell_get, clk_data); - if (ret) + if (ret) { dev_err(&pdev->dev, "Failed to register clock provider: %d\n", ret); + mtk_clk_unregister_plls(apmixedsys_plls, ARRAY_SIZE(apmixedsys_plls), + clk_data); + } return ret; } diff --git a/drivers/clk/mediatek/clk-mt6765.c b/drivers/clk/mediatek/clk-mt6765.c index d53731e7933f..60f6f9fa7dcf 100644 --- a/drivers/clk/mediatek/clk-mt6765.c +++ b/drivers/clk/mediatek/clk-mt6765.c @@ -740,7 +740,7 @@ static int clk_mt6765_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); mtk_clk_register_gates(&pdev->dev, node, apmixed_clks, ARRAY_SIZE(apmixed_clks), clk_data); diff --git a/drivers/clk/mediatek/clk-mt6779.c b/drivers/clk/mediatek/clk-mt6779.c index 86732f5acf93..4b9dcb910b03 100644 --- a/drivers/clk/mediatek/clk-mt6779.c +++ b/drivers/clk/mediatek/clk-mt6779.c @@ -1220,7 +1220,7 @@ static int clk_mt6779_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); mtk_clk_register_gates(&pdev->dev, node, apmixed_clks, ARRAY_SIZE(apmixed_clks), clk_data); diff --git a/drivers/clk/mediatek/clk-mt6795-apmixedsys.c b/drivers/clk/mediatek/clk-mt6795-apmixedsys.c index 91665d7f125e..123d5d7fea85 100644 --- a/drivers/clk/mediatek/clk-mt6795-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt6795-apmixedsys.c @@ -152,7 +152,7 @@ static int clk_mt6795_apmixed_probe(struct platform_device *pdev) return -ENOMEM; fhctl_parse_dt(fhctl_node, pllfhs, ARRAY_SIZE(pllfhs)); - ret = mtk_clk_register_pllfhs(node, plls, ARRAY_SIZE(plls), + ret = mtk_clk_register_pllfhs(dev, plls, ARRAY_SIZE(plls), pllfhs, ARRAY_SIZE(pllfhs), clk_data); if (ret) goto free_clk_data; diff --git a/drivers/clk/mediatek/clk-mt6797.c b/drivers/clk/mediatek/clk-mt6797.c index fb59e71af58e..ebf850ac57f5 100644 --- a/drivers/clk/mediatek/clk-mt6797.c +++ b/drivers/clk/mediatek/clk-mt6797.c @@ -655,7 +655,7 @@ static int mtk_apmixedsys_init(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); return of_clk_add_hw_provider(node, of_clk_hw_onecell_get, clk_data); } diff --git a/drivers/clk/mediatek/clk-mt7622-apmixedsys.c b/drivers/clk/mediatek/clk-mt7622-apmixedsys.c index 2350592d9a93..8a29eaab0cfc 100644 --- a/drivers/clk/mediatek/clk-mt7622-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt7622-apmixedsys.c @@ -96,7 +96,7 @@ static int clk_mt7622_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - ret = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + ret = mtk_clk_register_plls(dev, plls, ARRAY_SIZE(plls), clk_data); if (ret) return ret; diff --git a/drivers/clk/mediatek/clk-mt7629.c b/drivers/clk/mediatek/clk-mt7629.c index baf94e7bea37..e154771b1b8b 100644 --- a/drivers/clk/mediatek/clk-mt7629.c +++ b/drivers/clk/mediatek/clk-mt7629.c @@ -634,7 +634,7 @@ static int mtk_apmixedsys_init(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), + mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); mtk_clk_register_gates(&pdev->dev, node, apmixed_clks, diff --git a/drivers/clk/mediatek/clk-mt7981-apmixed.c b/drivers/clk/mediatek/clk-mt7981-apmixed.c index e8211eb4e09e..6606b54fb376 100644 --- a/drivers/clk/mediatek/clk-mt7981-apmixed.c +++ b/drivers/clk/mediatek/clk-mt7981-apmixed.c @@ -76,7 +76,7 @@ static int clk_mt7981_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); r = of_clk_add_hw_provider(node, of_clk_hw_onecell_get, clk_data); if (r) { diff --git a/drivers/clk/mediatek/clk-mt7986-apmixed.c b/drivers/clk/mediatek/clk-mt7986-apmixed.c index 93751abe6be8..1c79418d08a7 100644 --- a/drivers/clk/mediatek/clk-mt7986-apmixed.c +++ b/drivers/clk/mediatek/clk-mt7986-apmixed.c @@ -74,7 +74,7 @@ static int clk_mt7986_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); r = of_clk_add_hw_provider(node, of_clk_hw_onecell_get, clk_data); if (r) { diff --git a/drivers/clk/mediatek/clk-mt7988-apmixed.c b/drivers/clk/mediatek/clk-mt7988-apmixed.c index 63d33a78cb48..416a4b88d100 100644 --- a/drivers/clk/mediatek/clk-mt7988-apmixed.c +++ b/drivers/clk/mediatek/clk-mt7988-apmixed.c @@ -86,7 +86,7 @@ static int clk_mt7988_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - r = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + r = mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); if (r) goto free_apmixed_data; diff --git a/drivers/clk/mediatek/clk-mt8135-apmixedsys.c b/drivers/clk/mediatek/clk-mt8135-apmixedsys.c index bdadc35c64cb..19e4ee489ec3 100644 --- a/drivers/clk/mediatek/clk-mt8135-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8135-apmixedsys.c @@ -57,7 +57,8 @@ static int clk_mt8135_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - ret = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + ret = mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), + clk_data); if (ret) goto free_clk_data; diff --git a/drivers/clk/mediatek/clk-mt8135.c b/drivers/clk/mediatek/clk-mt8135.c index 084e48a554c2..1d20e15608f7 100644 --- a/drivers/clk/mediatek/clk-mt8135.c +++ b/drivers/clk/mediatek/clk-mt8135.c @@ -409,6 +409,9 @@ static const struct mtk_gate_regs infra_cg_regs = { GATE_MTK_FLAGS(_id, _name, _parent, &infra_cg_regs, _shift, \ &mtk_clk_gate_ops_setclr, CLK_IS_CRITICAL) +#define GATE_ICG_INV(_id, _name, _parent, _shift) \ + GATE_MTK(_id, _name, _parent, &infra_cg_regs, _shift, &mtk_clk_gate_ops_setclr_inv) + static const struct mtk_gate infra_clks[] = { GATE_DUMMY(CLK_DUMMY, "infra_dummy"), GATE_ICG(CLK_INFRA_PMIC_WRAP, "pmic_wrap_ck", "axi_sel", 23), @@ -419,7 +422,7 @@ static const struct mtk_gate infra_clks[] = { GATE_ICG(CLK_INFRA_CPUM, "cpum_ck", "cpum_tck_in", 15), GATE_ICG_AO(CLK_INFRA_M4U, "m4u_ck", "mem_sel", 8), GATE_ICG(CLK_INFRA_MFGAXI, "mfgaxi_ck", "axi_sel", 7), - GATE_ICG(CLK_INFRA_DEVAPC, "devapc_ck", "axi_sel", 6), + GATE_ICG_INV(CLK_INFRA_DEVAPC, "devapc_ck", "axi_sel", 6), GATE_ICG(CLK_INFRA_AUDIO, "audio_ck", "aud_intbus_sel", 5), GATE_ICG(CLK_INFRA_MFG_BUS, "mfg_bus_ck", "axi_sel", 2), GATE_ICG(CLK_INFRA_SMI, "smi_ck", "smi_sel", 1), diff --git a/drivers/clk/mediatek/clk-mt8167-apmixedsys.c b/drivers/clk/mediatek/clk-mt8167-apmixedsys.c index adf576786696..fb6c21bbeef8 100644 --- a/drivers/clk/mediatek/clk-mt8167-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8167-apmixedsys.c @@ -105,7 +105,7 @@ static int clk_mt8167_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - ret = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + ret = mtk_clk_register_plls(dev, plls, ARRAY_SIZE(plls), clk_data); if (ret) return ret; diff --git a/drivers/clk/mediatek/clk-mt8173-apmixedsys.c b/drivers/clk/mediatek/clk-mt8173-apmixedsys.c index 95385bb67d55..d7d416172ab3 100644 --- a/drivers/clk/mediatek/clk-mt8173-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8173-apmixedsys.c @@ -140,13 +140,13 @@ MODULE_DEVICE_TABLE(of, of_match_clk_mt8173_apmixed); static int clk_mt8173_apmixed_probe(struct platform_device *pdev) { const u8 *fhctl_node = "mediatek,mt8173-fhctl"; - struct device_node *node = pdev->dev.of_node; struct clk_hw_onecell_data *clk_data; + struct device *dev = &pdev->dev; void __iomem *base; struct clk_hw *hw; int r; - base = of_iomap(node, 0); + base = of_iomap(dev->of_node, 0); if (!base) return -ENOMEM; @@ -157,25 +157,25 @@ static int clk_mt8173_apmixed_probe(struct platform_device *pdev) } fhctl_parse_dt(fhctl_node, pllfhs, ARRAY_SIZE(pllfhs)); - r = mtk_clk_register_pllfhs(node, plls, ARRAY_SIZE(plls), - pllfhs, ARRAY_SIZE(pllfhs), clk_data); + r = mtk_clk_register_pllfhs(dev, plls, ARRAY_SIZE(plls), pllfhs, + ARRAY_SIZE(pllfhs), clk_data); if (r) goto free_clk_data; hw = mtk_clk_register_ref2usb_tx("ref2usb_tx", "clk26m", base + REGOFF_REF2USB); if (IS_ERR(hw)) { r = PTR_ERR(hw); - dev_err(&pdev->dev, "Failed to register ref2usb_tx: %d\n", r); + dev_err(dev, "Failed to register ref2usb_tx: %d\n", r); goto unregister_plls; } clk_data->hws[CLK_APMIXED_REF2USB_TX] = hw; - hw = devm_clk_hw_register_divider(&pdev->dev, "hdmi_ref", "tvdpll_594m", 0, + hw = devm_clk_hw_register_divider(dev, "hdmi_ref", "tvdpll_594m", 0, base + REGOFF_HDMI_REF, 16, 3, CLK_DIVIDER_POWER_OF_TWO, NULL); clk_data->hws[CLK_APMIXED_HDMI_REF] = hw; - r = of_clk_add_hw_provider(node, of_clk_hw_onecell_get, clk_data); + r = of_clk_add_hw_provider(dev->of_node, of_clk_hw_onecell_get, clk_data); if (r) goto unregister_ref2usb; diff --git a/drivers/clk/mediatek/clk-mt8183-apmixedsys.c b/drivers/clk/mediatek/clk-mt8183-apmixedsys.c index 551adbfd7ac9..6242d4f5376e 100644 --- a/drivers/clk/mediatek/clk-mt8183-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8183-apmixedsys.c @@ -155,7 +155,7 @@ static int clk_mt8183_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - ret = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + ret = mtk_clk_register_plls(dev, plls, ARRAY_SIZE(plls), clk_data); if (ret) return ret; diff --git a/drivers/clk/mediatek/clk-mt8186-apmixedsys.c b/drivers/clk/mediatek/clk-mt8186-apmixedsys.c index 4b2b16578232..d35dd2632e43 100644 --- a/drivers/clk/mediatek/clk-mt8186-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8186-apmixedsys.c @@ -151,7 +151,7 @@ static int clk_mt8186_apmixed_probe(struct platform_device *pdev) fhctl_parse_dt(fhctl_node, pllfhs, ARRAY_SIZE(pllfhs)); - r = mtk_clk_register_pllfhs(node, plls, ARRAY_SIZE(plls), + r = mtk_clk_register_pllfhs(&pdev->dev, plls, ARRAY_SIZE(plls), pllfhs, ARRAY_SIZE(pllfhs), clk_data); if (r) goto free_apmixed_data; diff --git a/drivers/clk/mediatek/clk-mt8188-apmixedsys.c b/drivers/clk/mediatek/clk-mt8188-apmixedsys.c index 21d7a9a2ab1a..a1de596bff99 100644 --- a/drivers/clk/mediatek/clk-mt8188-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8188-apmixedsys.c @@ -106,7 +106,7 @@ static int clk_mt8188_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - r = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + r = mtk_clk_register_plls(&pdev->dev, plls, ARRAY_SIZE(plls), clk_data); if (r) goto free_apmixed_data; diff --git a/drivers/clk/mediatek/clk-mt8192-apmixedsys.c b/drivers/clk/mediatek/clk-mt8192-apmixedsys.c index 0b66a27e4d5a..b0563a285bd6 100644 --- a/drivers/clk/mediatek/clk-mt8192-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8192-apmixedsys.c @@ -162,7 +162,7 @@ static int clk_mt8192_apmixed_probe(struct platform_device *pdev) fhctl_parse_dt(fhctl_node, pllfhs, ARRAY_SIZE(pllfhs)); - r = mtk_clk_register_pllfhs(node, plls, ARRAY_SIZE(plls), + r = mtk_clk_register_pllfhs(&pdev->dev, plls, ARRAY_SIZE(plls), pllfhs, ARRAY_SIZE(pllfhs), clk_data); if (r) goto free_clk_data; diff --git a/drivers/clk/mediatek/clk-mt8195-apmixedsys.c b/drivers/clk/mediatek/clk-mt8195-apmixedsys.c index 282a3137dc89..44917ab034c5 100644 --- a/drivers/clk/mediatek/clk-mt8195-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8195-apmixedsys.c @@ -181,7 +181,7 @@ static int clk_mt8195_apmixed_probe(struct platform_device *pdev) fhctl_parse_dt(fhctl_node, pllfhs, ARRAY_SIZE(pllfhs)); - r = mtk_clk_register_pllfhs(node, plls, ARRAY_SIZE(plls), + r = mtk_clk_register_pllfhs(&pdev->dev, plls, ARRAY_SIZE(plls), pllfhs, ARRAY_SIZE(pllfhs), clk_data); if (r) goto free_apmixed_data; diff --git a/drivers/clk/mediatek/clk-mt8195-apusys_pll.c b/drivers/clk/mediatek/clk-mt8195-apusys_pll.c index 8b45a3fad02f..a2d98ed58e34 100644 --- a/drivers/clk/mediatek/clk-mt8195-apusys_pll.c +++ b/drivers/clk/mediatek/clk-mt8195-apusys_pll.c @@ -66,7 +66,8 @@ static int clk_mt8195_apusys_pll_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - r = mtk_clk_register_plls(node, apusys_plls, ARRAY_SIZE(apusys_plls), clk_data); + r = mtk_clk_register_plls(&pdev->dev, apusys_plls, + ARRAY_SIZE(apusys_plls), clk_data); if (r) goto free_apusys_pll_data; diff --git a/drivers/clk/mediatek/clk-mt8196-apmixedsys.c b/drivers/clk/mediatek/clk-mt8196-apmixedsys.c index 617f5449b88b..c4ebb0170b82 100644 --- a/drivers/clk/mediatek/clk-mt8196-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8196-apmixedsys.c @@ -152,7 +152,8 @@ static int clk_mt8196_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - r = mtk_clk_register_plls(node, mcd->clks, mcd->num_clks, clk_data); + r = mtk_clk_register_plls(&pdev->dev, mcd->clks, mcd->num_clks, + clk_data); if (r) goto free_apmixed_data; diff --git a/drivers/clk/mediatek/clk-mt8196-mcu.c b/drivers/clk/mediatek/clk-mt8196-mcu.c index 5cbcc411ae73..13642fc673c2 100644 --- a/drivers/clk/mediatek/clk-mt8196-mcu.c +++ b/drivers/clk/mediatek/clk-mt8196-mcu.c @@ -122,7 +122,7 @@ static int clk_mt8196_mcu_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - r = mtk_clk_register_plls(node, plls, num_plls, clk_data); + r = mtk_clk_register_plls(&pdev->dev, plls, num_plls, clk_data); if (r) goto free_clk_data; diff --git a/drivers/clk/mediatek/clk-mt8196-mfg.c b/drivers/clk/mediatek/clk-mt8196-mfg.c index f40795b47ff1..a317183f1681 100644 --- a/drivers/clk/mediatek/clk-mt8196-mfg.c +++ b/drivers/clk/mediatek/clk-mt8196-mfg.c @@ -106,7 +106,7 @@ static int clk_mt8196_mfg_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - r = mtk_clk_register_plls(node, plls, num_plls, clk_data); + r = mtk_clk_register_plls(&pdev->dev, plls, num_plls, clk_data); if (r) goto free_clk_data; diff --git a/drivers/clk/mediatek/clk-mt8196-vlpckgen.c b/drivers/clk/mediatek/clk-mt8196-vlpckgen.c index d59a8a9d9855..7dcc164627c5 100644 --- a/drivers/clk/mediatek/clk-mt8196-vlpckgen.c +++ b/drivers/clk/mediatek/clk-mt8196-vlpckgen.c @@ -664,7 +664,7 @@ static int clk_mt8196_vlp_probe(struct platform_device *pdev) if (r) goto unregister_factors; - r = mtk_clk_register_plls(node, vlp_plls, ARRAY_SIZE(vlp_plls), + r = mtk_clk_register_plls(dev, vlp_plls, ARRAY_SIZE(vlp_plls), clk_data); if (r) goto unregister_muxes; diff --git a/drivers/clk/mediatek/clk-mt8365-apmixedsys.c b/drivers/clk/mediatek/clk-mt8365-apmixedsys.c index f41b991a0178..e331aa28a4bd 100644 --- a/drivers/clk/mediatek/clk-mt8365-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8365-apmixedsys.c @@ -133,7 +133,7 @@ static int clk_mt8365_apmixed_probe(struct platform_device *pdev) return PTR_ERR(hw); clk_data->hws[CLK_APMIXED_USB20_EN] = hw; - ret = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + ret = mtk_clk_register_plls(dev, plls, ARRAY_SIZE(plls), clk_data); if (ret) return ret; diff --git a/drivers/clk/mediatek/clk-mt8516-apmixedsys.c b/drivers/clk/mediatek/clk-mt8516-apmixedsys.c index edd9174d2f2f..2a6206cae2f0 100644 --- a/drivers/clk/mediatek/clk-mt8516-apmixedsys.c +++ b/drivers/clk/mediatek/clk-mt8516-apmixedsys.c @@ -87,7 +87,7 @@ static int clk_mt8516_apmixed_probe(struct platform_device *pdev) if (!clk_data) return -ENOMEM; - ret = mtk_clk_register_plls(node, plls, ARRAY_SIZE(plls), clk_data); + ret = mtk_clk_register_plls(dev, plls, ARRAY_SIZE(plls), clk_data); if (ret) return ret; diff --git a/drivers/clk/mediatek/clk-pll.c b/drivers/clk/mediatek/clk-pll.c index de3eb0267055..0f3759fcd9d0 100644 --- a/drivers/clk/mediatek/clk-pll.c +++ b/drivers/clk/mediatek/clk-pll.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include "clk-pll.h" @@ -368,7 +369,7 @@ struct clk_hw *mtk_clk_register_pll_ops(struct mtk_clk_pll *pll, init.parent_names = &parent_name; init.num_parents = 1; - ret = clk_hw_register(NULL, &pll->hw); + ret = clk_hw_register(pll->dev, &pll->hw); if (ret) return ERR_PTR(ret); @@ -376,7 +377,8 @@ struct clk_hw *mtk_clk_register_pll_ops(struct mtk_clk_pll *pll, return &pll->hw; } -struct clk_hw *mtk_clk_register_pll(const struct mtk_pll_data *data, +struct clk_hw *mtk_clk_register_pll(struct device *dev, + const struct mtk_pll_data *data, void __iomem *base) { struct mtk_clk_pll *pll; @@ -387,6 +389,8 @@ struct clk_hw *mtk_clk_register_pll(const struct mtk_pll_data *data, if (!pll) return ERR_PTR(-ENOMEM); + pll->dev = dev; + hw = mtk_clk_register_pll_ops(pll, data, base, pll_ops); if (IS_ERR(hw)) kfree(pll); @@ -407,7 +411,7 @@ void mtk_clk_unregister_pll(struct clk_hw *hw) kfree(pll); } -int mtk_clk_register_plls(struct device_node *node, +int mtk_clk_register_plls(struct device *dev, const struct mtk_pll_data *plls, int num_plls, struct clk_hw_onecell_data *clk_data) { @@ -415,7 +419,7 @@ int mtk_clk_register_plls(struct device_node *node, int i; struct clk_hw *hw; - base = of_iomap(node, 0); + base = of_iomap(dev->of_node, 0); if (!base) { pr_err("%s(): ioremap failed\n", __func__); return -EINVAL; @@ -426,11 +430,11 @@ int mtk_clk_register_plls(struct device_node *node, if (!IS_ERR_OR_NULL(clk_data->hws[pll->id])) { pr_warn("%pOF: Trying to register duplicate clock ID: %d\n", - node, pll->id); + dev->of_node, pll->id); continue; } - hw = mtk_clk_register_pll(pll, base); + hw = mtk_clk_register_pll(dev, pll, base); if (IS_ERR(hw)) { pr_err("Failed to register clk %s: %pe\n", pll->name, diff --git a/drivers/clk/mediatek/clk-pll.h b/drivers/clk/mediatek/clk-pll.h index de5a8fb7cbcf..f49dc2732ffe 100644 --- a/drivers/clk/mediatek/clk-pll.h +++ b/drivers/clk/mediatek/clk-pll.h @@ -10,9 +10,7 @@ #include #include -struct clk_ops; -struct clk_hw_onecell_data; -struct device_node; +struct device; struct mtk_pll_div_table { u32 div; @@ -64,6 +62,7 @@ struct mtk_pll_data { */ struct mtk_clk_pll { + struct device *dev; struct clk_hw hw; void __iomem *base_addr; void __iomem *pd_addr; @@ -79,9 +78,9 @@ struct mtk_clk_pll { const struct mtk_pll_data *data; }; -int mtk_clk_register_plls(struct device_node *node, - const struct mtk_pll_data *plls, int num_plls, - struct clk_hw_onecell_data *clk_data); +int mtk_clk_register_plls(struct device *dev, const struct mtk_pll_data *plls, + int num_plls, struct clk_hw_onecell_data *clk_data); + void mtk_clk_unregister_plls(const struct mtk_pll_data *plls, int num_plls, struct clk_hw_onecell_data *clk_data); @@ -111,7 +110,8 @@ struct clk_hw *mtk_clk_register_pll_ops(struct mtk_clk_pll *pll, const struct mtk_pll_data *data, void __iomem *base, const struct clk_ops *pll_ops); -struct clk_hw *mtk_clk_register_pll(const struct mtk_pll_data *data, +struct clk_hw *mtk_clk_register_pll(struct device *dev, + const struct mtk_pll_data *data, void __iomem *base); void mtk_clk_unregister_pll(struct clk_hw *hw); diff --git a/drivers/clk/mediatek/clk-pllfh.c b/drivers/clk/mediatek/clk-pllfh.c index 83630ee07ee9..d8b7eb6d18e1 100644 --- a/drivers/clk/mediatek/clk-pllfh.c +++ b/drivers/clk/mediatek/clk-pllfh.c @@ -10,6 +10,7 @@ #include #include #include +#include #include "clk-mtk.h" #include "clk-pllfh.h" @@ -149,7 +150,7 @@ static bool fhctl_is_supported_and_enabled(const struct mtk_pllfh_data *pllfh) } static struct clk_hw * -mtk_clk_register_pllfh(const struct mtk_pll_data *pll_data, +mtk_clk_register_pllfh(struct device *dev, const struct mtk_pll_data *pll_data, struct mtk_pllfh_data *pllfh_data, void __iomem *base) { struct clk_hw *hw; @@ -166,6 +167,8 @@ mtk_clk_register_pllfh(const struct mtk_pll_data *pll_data, goto out; } + fh->clk_pll.dev = dev; + hw = mtk_clk_register_pll_ops(&fh->clk_pll, pll_data, base, &mtk_pllfh_ops); @@ -194,16 +197,60 @@ static void mtk_clk_unregister_pllfh(struct clk_hw *hw) kfree(fh); } -int mtk_clk_register_pllfhs(struct device_node *node, +static void mtk_clk_cleanup_pllfhs(void __iomem *iomem_base, + const struct mtk_pll_data *plls, int num_plls, + void __iomem *iomem_fhctl_base, + struct mtk_pllfh_data *pllfhs, int num_fhs, + struct clk_hw_onecell_data *clk_data) +{ + void __iomem *base = iomem_base; + void __iomem *fhctl_base = iomem_fhctl_base; + int i; + + for (i = num_plls - 1; i >= 0; i--) { + const struct mtk_pll_data *pll = &plls[i]; + struct mtk_pllfh_data *pllfh; + bool use_fhctl; + + if (IS_ERR_OR_NULL(clk_data->hws[pll->id])) + continue; + + pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id); + use_fhctl = fhctl_is_supported_and_enabled(pllfh); + + if (!base) + base = mtk_clk_pll_get_base(clk_data->hws[pll->id], + pll); + + if (use_fhctl) { + if (!fhctl_base) + fhctl_base = pllfh->state.base; + mtk_clk_unregister_pllfh(clk_data->hws[pll->id]); + } else { + mtk_clk_unregister_pll(clk_data->hws[pll->id]); + } + + clk_data->hws[pll->id] = ERR_PTR(-ENOENT); + } + + if (fhctl_base) + iounmap(fhctl_base); + + if (base) + iounmap(base); +} + + +int mtk_clk_register_pllfhs(struct device *dev, const struct mtk_pll_data *plls, int num_plls, struct mtk_pllfh_data *pllfhs, int num_fhs, struct clk_hw_onecell_data *clk_data) { - void __iomem *base; + void __iomem *base, *fhctl_base = NULL; int i; struct clk_hw *hw; - base = of_iomap(node, 0); + base = of_iomap(dev->of_node, 0); if (!base) { pr_err("%s(): ioremap failed\n", __func__); return -EINVAL; @@ -218,9 +265,9 @@ int mtk_clk_register_pllfhs(struct device_node *node, use_fhctl = fhctl_is_supported_and_enabled(pllfh); if (use_fhctl) - hw = mtk_clk_register_pllfh(pll, pllfh, base); + hw = mtk_clk_register_pllfh(dev, pll, pllfh, base); else - hw = mtk_clk_register_pll(pll, base); + hw = mtk_clk_register_pll(dev, pll, base); if (IS_ERR(hw)) { pr_err("Failed to register %s clk %s: %ld\n", @@ -235,24 +282,8 @@ int mtk_clk_register_pllfhs(struct device_node *node, return 0; err: - while (--i >= 0) { - const struct mtk_pll_data *pll = &plls[i]; - struct mtk_pllfh_data *pllfh; - bool use_fhctl; - - pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id); - use_fhctl = fhctl_is_supported_and_enabled(pllfh); - - if (use_fhctl) - mtk_clk_unregister_pllfh(clk_data->hws[pll->id]); - else - mtk_clk_unregister_pll(clk_data->hws[pll->id]); - - clk_data->hws[pll->id] = ERR_PTR(-ENOENT); - } - - iounmap(base); - + mtk_clk_cleanup_pllfhs(base, plls, i, fhctl_base, pllfhs, num_fhs, + clk_data); return PTR_ERR(hw); } EXPORT_SYMBOL_GPL(mtk_clk_register_pllfhs); @@ -261,38 +292,10 @@ void mtk_clk_unregister_pllfhs(const struct mtk_pll_data *plls, int num_plls, struct mtk_pllfh_data *pllfhs, int num_fhs, struct clk_hw_onecell_data *clk_data) { - void __iomem *base = NULL, *fhctl_base = NULL; - int i; - if (!clk_data) return; - for (i = num_plls; i > 0; i--) { - const struct mtk_pll_data *pll = &plls[i - 1]; - struct mtk_pllfh_data *pllfh; - bool use_fhctl; - - if (IS_ERR_OR_NULL(clk_data->hws[pll->id])) - continue; - - pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id); - use_fhctl = fhctl_is_supported_and_enabled(pllfh); - - if (use_fhctl) { - fhctl_base = pllfh->state.base; - mtk_clk_unregister_pllfh(clk_data->hws[pll->id]); - } else { - base = mtk_clk_pll_get_base(clk_data->hws[pll->id], - pll); - mtk_clk_unregister_pll(clk_data->hws[pll->id]); - } - - clk_data->hws[pll->id] = ERR_PTR(-ENOENT); - } - - if (fhctl_base) - iounmap(fhctl_base); - - iounmap(base); + mtk_clk_cleanup_pllfhs(NULL, plls, num_plls, NULL, pllfhs, + num_fhs, clk_data); } EXPORT_SYMBOL_GPL(mtk_clk_unregister_pllfhs); diff --git a/drivers/clk/mediatek/clk-pllfh.h b/drivers/clk/mediatek/clk-pllfh.h index 5f419c2ec01f..a4f337acad71 100644 --- a/drivers/clk/mediatek/clk-pllfh.h +++ b/drivers/clk/mediatek/clk-pllfh.h @@ -68,7 +68,7 @@ struct fh_operation { int (*ssc_enable)(struct mtk_fh *fh, u32 rate); }; -int mtk_clk_register_pllfhs(struct device_node *node, +int mtk_clk_register_pllfhs(struct device *dev, const struct mtk_pll_data *plls, int num_plls, struct mtk_pllfh_data *pllfhs, int num_pllfhs, struct clk_hw_onecell_data *clk_data); diff --git a/drivers/clk/mvebu/common.c b/drivers/clk/mvebu/common.c index 785dbede4835..5adbbd91a6db 100644 --- a/drivers/clk/mvebu/common.c +++ b/drivers/clk/mvebu/common.c @@ -215,22 +215,26 @@ static struct clk *clk_gating_get_src( return ERR_PTR(-ENODEV); } -static int mvebu_clk_gating_suspend(void) +static int mvebu_clk_gating_suspend(void *data) { ctrl->saved_reg = readl(ctrl->base); return 0; } -static void mvebu_clk_gating_resume(void) +static void mvebu_clk_gating_resume(void *data) { writel(ctrl->saved_reg, ctrl->base); } -static struct syscore_ops clk_gate_syscore_ops = { +static const struct syscore_ops clk_gate_syscore_ops = { .suspend = mvebu_clk_gating_suspend, .resume = mvebu_clk_gating_resume, }; +static struct syscore clk_gate_syscore = { + .ops = &clk_gate_syscore_ops, +}; + void __init mvebu_clk_gating_setup(struct device_node *np, const struct clk_gating_soc_desc *desc) { @@ -284,7 +288,7 @@ void __init mvebu_clk_gating_setup(struct device_node *np, of_clk_add_provider(np, clk_gating_get_src, ctrl); - register_syscore_ops(&clk_gate_syscore_ops); + register_syscore(&clk_gate_syscore); return; gates_out: diff --git a/drivers/clk/nuvoton/clk-ma35d1-pll.c b/drivers/clk/nuvoton/clk-ma35d1-pll.c index 4620acfe47e8..c7c0dc91a012 100644 --- a/drivers/clk/nuvoton/clk-ma35d1-pll.c +++ b/drivers/clk/nuvoton/clk-ma35d1-pll.c @@ -48,7 +48,7 @@ #define PLL_CTL1_PD BIT(0) #define PLL_CTL1_BP BIT(1) #define PLL_CTL1_OUTDIV GENMASK(6, 4) -#define PLL_CTL1_FRAC GENMASK(31, 24) +#define PLL_CTL1_FRAC GENMASK(31, 8) #define PLL_CTL2_SLOPE GENMASK(23, 0) #define INDIV_MIN 1 @@ -92,7 +92,7 @@ static unsigned long ma35d1_calc_smic_pll_freq(u32 pll0_ctl0, p = FIELD_GET(SPLL0_CTL0_OUTDIV, pll0_ctl0); outdiv = 1 << p; pll_freq = (u64)parent_rate * n; - div_u64(pll_freq, m * outdiv); + pll_freq = div_u64(pll_freq, m * outdiv); return pll_freq; } @@ -110,12 +110,12 @@ static unsigned long ma35d1_calc_pll_freq(u8 mode, u32 *reg_ctl, unsigned long p if (mode == PLL_MODE_INT) { pll_freq = (u64)parent_rate * n; - div_u64(pll_freq, m * p); + pll_freq = div_u64(pll_freq, m * p); } else { x = FIELD_GET(PLL_CTL1_FRAC, reg_ctl[1]); - /* 2 decimal places floating to integer (ex. 1.23 to 123) */ - n = n * 100 + ((x * 100) / FIELD_MAX(PLL_CTL1_FRAC)); - pll_freq = div_u64(parent_rate * n, 100 * m * p); + /* convert 24-bit fraction to 3 decimal digits, rounding to closest */ + n = n * 1000 + DIV_ROUND_CLOSEST_ULL((u64)x * 1000, 1ULL << 24); + pll_freq = div_u64((u64)parent_rate * n, 1000 * m * p); } return pll_freq; } @@ -255,32 +255,32 @@ static int ma35d1_clk_pll_determine_rate(struct clk_hw *hw, if (req->best_parent_rate < PLL_FREF_MIN_FREQ || req->best_parent_rate > PLL_FREF_MAX_FREQ) return -EINVAL; - ret = ma35d1_pll_find_closest(pll, req->rate, req->best_parent_rate, - reg_ctl, &pll_freq); - if (ret < 0) - return ret; - switch (pll->id) { case CAPLL: + case DDRPLL: + /* Read-only PLLs: return current rate */ reg_ctl[0] = readl_relaxed(pll->ctl0_base); - pll_freq = ma35d1_calc_smic_pll_freq(reg_ctl[0], req->best_parent_rate); + if (pll->id == CAPLL) { + pll_freq = ma35d1_calc_smic_pll_freq(reg_ctl[0], req->best_parent_rate); + } else { + reg_ctl[1] = readl_relaxed(pll->ctl1_base); + pll_freq = ma35d1_calc_pll_freq(pll->mode, reg_ctl, req->best_parent_rate); + } req->rate = pll_freq; - return 0; - case DDRPLL: case APLL: case EPLL: case VPLL: - reg_ctl[0] = readl_relaxed(pll->ctl0_base); - reg_ctl[1] = readl_relaxed(pll->ctl1_base); - pll_freq = ma35d1_calc_pll_freq(pll->mode, reg_ctl, req->best_parent_rate); + /* Configurable PLLs: find closest achievable rate */ + ret = ma35d1_pll_find_closest(pll, req->rate, req->best_parent_rate, + reg_ctl, &pll_freq); + if (ret < 0) + return ret; req->rate = pll_freq; - return 0; } req->rate = 0; - return 0; } diff --git a/drivers/clk/qcom/camcc-sc8280xp.c b/drivers/clk/qcom/camcc-sc8280xp.c index 18f5a3eb313e..1de238a85ab9 100644 --- a/drivers/clk/qcom/camcc-sc8280xp.c +++ b/drivers/clk/qcom/camcc-sc8280xp.c @@ -1754,24 +1754,6 @@ static struct clk_branch camcc_csiphy3_clk = { }, }; -static struct clk_branch camcc_gdsc_clk = { - .halt_reg = 0xc1e4, - .halt_check = BRANCH_HALT, - .clkr = { - .enable_reg = 0xc1e4, - .enable_mask = BIT(0), - .hw.init = &(struct clk_init_data){ - .name = "camcc_gdsc_clk", - .parent_hws = (const struct clk_hw*[]){ - &camcc_xo_clk_src.clkr.hw, - }, - .num_parents = 1, - .flags = CLK_SET_RATE_PARENT, - .ops = &clk_branch2_ops, - }, - }, -}; - static struct clk_branch camcc_icp_ahb_clk = { .halt_reg = 0xc0d8, .halt_check = BRANCH_HALT, @@ -2840,7 +2822,6 @@ static struct clk_regmap *camcc_sc8280xp_clocks[] = { [CAMCC_CSIPHY2_CLK] = &camcc_csiphy2_clk.clkr, [CAMCC_CSIPHY3_CLK] = &camcc_csiphy3_clk.clkr, [CAMCC_FAST_AHB_CLK_SRC] = &camcc_fast_ahb_clk_src.clkr, - [CAMCC_GDSC_CLK] = &camcc_gdsc_clk.clkr, [CAMCC_ICP_AHB_CLK] = &camcc_icp_ahb_clk.clkr, [CAMCC_ICP_CLK] = &camcc_icp_clk.clkr, [CAMCC_ICP_CLK_SRC] = &camcc_icp_clk_src.clkr, diff --git a/drivers/clk/qcom/common.c b/drivers/clk/qcom/common.c index eec369d2173b..0e8f380873af 100644 --- a/drivers/clk/qcom/common.c +++ b/drivers/clk/qcom/common.c @@ -169,7 +169,7 @@ static int _qcom_cc_register_board_clk(struct device *dev, const char *path, if (!node) { fixed = devm_kzalloc(dev, sizeof(*fixed), GFP_KERNEL); if (!fixed) - return -EINVAL; + return -ENOMEM; fixed->fixed_rate = rate; fixed->hw.init = &init_data; @@ -186,7 +186,7 @@ static int _qcom_cc_register_board_clk(struct device *dev, const char *path, if (add_factor) { factor = devm_kzalloc(dev, sizeof(*factor), GFP_KERNEL); if (!factor) - return -EINVAL; + return -ENOMEM; factor->mult = factor->div = 1; factor->hw.init = &init_data; diff --git a/drivers/clk/qcom/gcc-glymur.c b/drivers/clk/qcom/gcc-glymur.c index eff3248d483a..7adbe59915f0 100644 --- a/drivers/clk/qcom/gcc-glymur.c +++ b/drivers/clk/qcom/gcc-glymur.c @@ -3671,21 +3671,6 @@ static struct clk_branch gcc_disp_hf_axi_clk = { }, }; -static struct clk_branch gcc_eva_ahb_clk = { - .halt_reg = 0x9b004, - .halt_check = BRANCH_HALT_VOTED, - .hwcg_reg = 0x9b004, - .hwcg_bit = 1, - .clkr = { - .enable_reg = 0x9b004, - .enable_mask = BIT(0), - .hw.init = &(const struct clk_init_data) { - .name = "gcc_eva_ahb_clk", - .ops = &clk_branch2_ops, - }, - }, -}; - static struct clk_branch gcc_eva_axi0_clk = { .halt_reg = 0x9b008, .halt_check = BRANCH_HALT_SKIP, @@ -3716,19 +3701,6 @@ static struct clk_branch gcc_eva_axi0c_clk = { }, }; -static struct clk_branch gcc_eva_xo_clk = { - .halt_reg = 0x9b024, - .halt_check = BRANCH_HALT, - .clkr = { - .enable_reg = 0x9b024, - .enable_mask = BIT(0), - .hw.init = &(const struct clk_init_data) { - .name = "gcc_eva_xo_clk", - .ops = &clk_branch2_ops, - }, - }, -}; - static struct clk_branch gcc_gp1_clk = { .halt_reg = 0x64000, .halt_check = BRANCH_HALT, @@ -7995,10 +7967,8 @@ static struct clk_regmap *gcc_glymur_clocks[] = { [GCC_CFG_NOC_USB_ANOC_AHB_CLK] = &gcc_cfg_noc_usb_anoc_ahb_clk.clkr, [GCC_CFG_NOC_USB_ANOC_SOUTH_AHB_CLK] = &gcc_cfg_noc_usb_anoc_south_ahb_clk.clkr, [GCC_DISP_HF_AXI_CLK] = &gcc_disp_hf_axi_clk.clkr, - [GCC_EVA_AHB_CLK] = &gcc_eva_ahb_clk.clkr, [GCC_EVA_AXI0_CLK] = &gcc_eva_axi0_clk.clkr, [GCC_EVA_AXI0C_CLK] = &gcc_eva_axi0c_clk.clkr, - [GCC_EVA_XO_CLK] = &gcc_eva_xo_clk.clkr, [GCC_GP1_CLK] = &gcc_gp1_clk.clkr, [GCC_GP1_CLK_SRC] = &gcc_gp1_clk_src.clkr, [GCC_GP2_CLK] = &gcc_gp2_clk.clkr, @@ -8547,6 +8517,8 @@ static u32 gcc_glymur_critical_cbcrs[] = { 0x71004, /* GCC_GPU_CFG_AHB_CLK */ 0x32004, /* GCC_VIDEO_AHB_CLK */ 0x32058, /* GCC_VIDEO_XO_CLK */ + 0x9b004, /* GCC_EVA_AHB_CLK */ + 0x9b024, /* GCC_EVA_XO_CLK */ }; static const struct regmap_config gcc_glymur_regmap_config = { @@ -8579,6 +8551,7 @@ static const struct qcom_cc_desc gcc_glymur_desc = { .num_resets = ARRAY_SIZE(gcc_glymur_resets), .gdscs = gcc_glymur_gdscs, .num_gdscs = ARRAY_SIZE(gcc_glymur_gdscs), + .use_rpm = true, .driver_data = &gcc_glymur_driver_data, }; diff --git a/drivers/clk/qcom/gcc-qcm2290.c b/drivers/clk/qcom/gcc-qcm2290.c index 6684cab63ae1..77cff0e9af93 100644 --- a/drivers/clk/qcom/gcc-qcm2290.c +++ b/drivers/clk/qcom/gcc-qcm2290.c @@ -116,7 +116,7 @@ static const struct alpha_pll_config gpll10_config = { .vco_mask = GENMASK(21, 20), .main_output_mask = BIT(0), .config_ctl_val = 0x4001055B, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, }; static struct clk_alpha_pll gpll10 = { @@ -148,7 +148,7 @@ static const struct alpha_pll_config gpll11_config = { .vco_mask = GENMASK(21, 20), .main_output_mask = BIT(0), .config_ctl_val = 0x4001055B, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, }; static struct clk_alpha_pll gpll11 = { @@ -309,7 +309,7 @@ static const struct alpha_pll_config gpll8_config = { .post_div_val = 0x1 << 8, .post_div_mask = GENMASK(11, 8), .config_ctl_val = 0x4001055B, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, }; static struct clk_alpha_pll gpll8 = { @@ -1082,7 +1082,7 @@ static struct clk_init_data gcc_qupv3_wrap0_s0_clk_src_init = { .name = "gcc_qupv3_wrap0_s0_clk_src", .parent_data = gcc_parents_1, .num_parents = ARRAY_SIZE(gcc_parents_1), - .ops = &clk_rcg2_shared_ops, + .ops = &clk_rcg2_shared_no_init_park_ops, }; static struct clk_rcg2 gcc_qupv3_wrap0_s0_clk_src = { @@ -1098,7 +1098,7 @@ static struct clk_init_data gcc_qupv3_wrap0_s1_clk_src_init = { .name = "gcc_qupv3_wrap0_s1_clk_src", .parent_data = gcc_parents_1, .num_parents = ARRAY_SIZE(gcc_parents_1), - .ops = &clk_rcg2_shared_ops, + .ops = &clk_rcg2_shared_no_init_park_ops, }; static struct clk_rcg2 gcc_qupv3_wrap0_s1_clk_src = { @@ -1114,7 +1114,7 @@ static struct clk_init_data gcc_qupv3_wrap0_s2_clk_src_init = { .name = "gcc_qupv3_wrap0_s2_clk_src", .parent_data = gcc_parents_1, .num_parents = ARRAY_SIZE(gcc_parents_1), - .ops = &clk_rcg2_shared_ops, + .ops = &clk_rcg2_shared_no_init_park_ops, }; static struct clk_rcg2 gcc_qupv3_wrap0_s2_clk_src = { @@ -1130,7 +1130,7 @@ static struct clk_init_data gcc_qupv3_wrap0_s3_clk_src_init = { .name = "gcc_qupv3_wrap0_s3_clk_src", .parent_data = gcc_parents_1, .num_parents = ARRAY_SIZE(gcc_parents_1), - .ops = &clk_rcg2_shared_ops, + .ops = &clk_rcg2_shared_no_init_park_ops, }; static struct clk_rcg2 gcc_qupv3_wrap0_s3_clk_src = { @@ -1146,7 +1146,7 @@ static struct clk_init_data gcc_qupv3_wrap0_s4_clk_src_init = { .name = "gcc_qupv3_wrap0_s4_clk_src", .parent_data = gcc_parents_1, .num_parents = ARRAY_SIZE(gcc_parents_1), - .ops = &clk_rcg2_shared_ops, + .ops = &clk_rcg2_shared_no_init_park_ops, }; static struct clk_rcg2 gcc_qupv3_wrap0_s4_clk_src = { @@ -1162,7 +1162,7 @@ static struct clk_init_data gcc_qupv3_wrap0_s5_clk_src_init = { .name = "gcc_qupv3_wrap0_s5_clk_src", .parent_data = gcc_parents_1, .num_parents = ARRAY_SIZE(gcc_parents_1), - .ops = &clk_rcg2_shared_ops, + .ops = &clk_rcg2_shared_no_init_park_ops, }; static struct clk_rcg2 gcc_qupv3_wrap0_s5_clk_src = { diff --git a/drivers/clk/qcom/gcc-qcs8300.c b/drivers/clk/qcom/gcc-qcs8300.c index 80831c7dea3b..fe0632687f9f 100644 --- a/drivers/clk/qcom/gcc-qcs8300.c +++ b/drivers/clk/qcom/gcc-qcs8300.c @@ -3268,7 +3268,7 @@ static struct gdsc gcc_pcie_0_gdsc = { .pd = { .name = "gcc_pcie_0_gdsc", }, - .pwrsts = PWRSTS_OFF_ON, + .pwrsts = PWRSTS_RET_ON, .flags = VOTABLE | RETAIN_FF_ENABLE | POLL_CFG_GDSCR, }; @@ -3282,7 +3282,7 @@ static struct gdsc gcc_pcie_1_gdsc = { .pd = { .name = "gcc_pcie_1_gdsc", }, - .pwrsts = PWRSTS_OFF_ON, + .pwrsts = PWRSTS_RET_ON, .flags = VOTABLE | RETAIN_FF_ENABLE | POLL_CFG_GDSCR, }; @@ -3306,7 +3306,7 @@ static struct gdsc gcc_usb20_prim_gdsc = { .pd = { .name = "gcc_usb20_prim_gdsc", }, - .pwrsts = PWRSTS_OFF_ON, + .pwrsts = PWRSTS_RET_ON, .flags = RETAIN_FF_ENABLE | POLL_CFG_GDSCR, }; @@ -3318,7 +3318,7 @@ static struct gdsc gcc_usb30_prim_gdsc = { .pd = { .name = "gcc_usb30_prim_gdsc", }, - .pwrsts = PWRSTS_OFF_ON, + .pwrsts = PWRSTS_RET_ON, .flags = RETAIN_FF_ENABLE | POLL_CFG_GDSCR, }; diff --git a/drivers/clk/qcom/gcc-sm6115.c b/drivers/clk/qcom/gcc-sm6115.c index 4c3804701e24..4bd2ebc15894 100644 --- a/drivers/clk/qcom/gcc-sm6115.c +++ b/drivers/clk/qcom/gcc-sm6115.c @@ -120,7 +120,7 @@ static const struct alpha_pll_config gpll10_config = { .vco_mask = GENMASK(21, 20), .main_output_mask = BIT(0), .config_ctl_val = 0x4001055b, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, .test_ctl_hi_mask = 0x1, }; @@ -173,7 +173,7 @@ static const struct alpha_pll_config gpll11_config = { .vco_val = 0x2 << 20, .vco_mask = GENMASK(21, 20), .config_ctl_val = 0x4001055b, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, .test_ctl_hi_mask = 0x1, }; @@ -367,7 +367,7 @@ static const struct alpha_pll_config gpll8_config = { .post_div_val = 0x1 << 8, .post_div_mask = GENMASK(11, 8), .config_ctl_val = 0x4001055b, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, .test_ctl_hi_mask = 0x1, }; diff --git a/drivers/clk/qcom/gdsc.c b/drivers/clk/qcom/gdsc.c index 95aa07120245..71826ccbd9bd 100644 --- a/drivers/clk/qcom/gdsc.c +++ b/drivers/clk/qcom/gdsc.c @@ -103,14 +103,21 @@ static int gdsc_hwctrl(struct gdsc *sc, bool en) static int gdsc_poll_status(struct gdsc *sc, enum gdsc_status status) { ktime_t start; + int ret; start = ktime_get(); do { - if (gdsc_check_status(sc, status)) + ret = gdsc_check_status(sc, status); + if (ret < 0) + return ret; + if (ret) return 0; } while (ktime_us_delta(ktime_get(), start) < STATUS_POLL_TIMEOUT_US); - if (gdsc_check_status(sc, status)) + ret = gdsc_check_status(sc, status); + if (ret < 0) + return ret; + if (ret) return 0; return -ETIMEDOUT; @@ -474,7 +481,9 @@ static int gdsc_init(struct gdsc *sc) } else if (sc->flags & ALWAYS_ON) { /* If ALWAYS_ON GDSCs are not ON, turn them ON */ - gdsc_enable(&sc->pd); + ret = gdsc_enable(&sc->pd); + if (ret) + return ret; on = true; } @@ -636,10 +645,18 @@ int gdsc_register(struct gdsc_desc *desc, void gdsc_unregister(struct gdsc_desc *desc) { struct device *dev = desc->dev; + struct gdsc **scs = desc->scs; size_t num = desc->num; + int i; - gdsc_pm_subdomain_remove(desc, num); of_genpd_del_provider(dev->of_node); + gdsc_pm_subdomain_remove(desc, num); + + for (i = 0; i < num; i++) { + if (!scs[i]) + continue; + pm_genpd_remove(&scs[i]->pd); + } } /* diff --git a/drivers/clk/qcom/gpucc-qcm2290.c b/drivers/clk/qcom/gpucc-qcm2290.c index dc369dff882e..3ccab8f3b4e0 100644 --- a/drivers/clk/qcom/gpucc-qcm2290.c +++ b/drivers/clk/qcom/gpucc-qcm2290.c @@ -144,7 +144,7 @@ static struct clk_rcg2 gpu_cc_gx_gfx3d_clk_src = { .parent_data = gpu_cc_parent_data_1, .num_parents = ARRAY_SIZE(gpu_cc_parent_data_1), .flags = CLK_SET_RATE_PARENT, - .ops = &clk_rcg2_ops, + .ops = &clk_rcg2_shared_ops, }, }; diff --git a/drivers/clk/rockchip/clk-pll.c b/drivers/clk/rockchip/clk-pll.c index 86dba3826a77..0dada00b375f 100644 --- a/drivers/clk/rockchip/clk-pll.c +++ b/drivers/clk/rockchip/clk-pll.c @@ -900,6 +900,13 @@ static void rockchip_rk3588_pll_get_params(struct rockchip_clk_pll *pll, rate->k = ((pllcon >> RK3588_PLLCON2_K_SHIFT) & RK3588_PLLCON2_K_MASK); } +/* + * 2250 MHz <= Fvco <= 4500 MHz + * For Fvco > 3 GHz: period jitter +-1% frac PLL, +-0.75% int PLL + * For Fvco < 3 GHz: period jitter +-2% frac PLL, +-1.50% int PLL + * Fvco = ((m + k / 65536) * Fin) / p + * Fout = ((m + k / 65536) * Fin) / (p * 2^s) + */ static unsigned long rockchip_rk3588_pll_recalc_rate(struct clk_hw *hw, unsigned long prate) { struct rockchip_clk_pll *pll = to_rockchip_clk_pll(hw); @@ -915,7 +922,7 @@ static unsigned long rockchip_rk3588_pll_recalc_rate(struct clk_hw *hw, unsigned /* fractional mode */ u64 frac_rate64 = prate * cur.k; - postdiv = cur.p * 65535; + postdiv = cur.p * 65536; do_div(frac_rate64, postdiv); rate64 += frac_rate64; } diff --git a/drivers/clk/rockchip/clk-rk3288.c b/drivers/clk/rockchip/clk-rk3288.c index 0a1e017df7c6..9cf3e1e43b78 100644 --- a/drivers/clk/rockchip/clk-rk3288.c +++ b/drivers/clk/rockchip/clk-rk3288.c @@ -871,7 +871,7 @@ static const int rk3288_saved_cru_reg_ids[] = { static u32 rk3288_saved_cru_regs[ARRAY_SIZE(rk3288_saved_cru_reg_ids)]; -static int rk3288_clk_suspend(void) +static int rk3288_clk_suspend(void *data) { int i, reg_id; @@ -906,7 +906,7 @@ static int rk3288_clk_suspend(void) return 0; } -static void rk3288_clk_resume(void) +static void rk3288_clk_resume(void *data) { int i, reg_id; @@ -923,11 +923,15 @@ static void rk3288_clk_shutdown(void) writel_relaxed(0xf3030000, rk3288_cru_base + RK3288_MODE_CON); } -static struct syscore_ops rk3288_clk_syscore_ops = { +static const struct syscore_ops rk3288_clk_syscore_ops = { .suspend = rk3288_clk_suspend, .resume = rk3288_clk_resume, }; +static struct syscore rk3288_clk_syscore = { + .ops = &rk3288_clk_syscore_ops, +}; + static void __init rk3288_common_init(struct device_node *np, enum rk3288_variant soc) { @@ -976,7 +980,7 @@ static void __init rk3288_common_init(struct device_node *np, rockchip_register_restart_notifier(ctx, RK3288_GLB_SRST_FST, rk3288_clk_shutdown); - register_syscore_ops(&rk3288_clk_syscore_ops); + register_syscore(&rk3288_clk_syscore); rockchip_clk_of_add_provider(np, ctx); } diff --git a/drivers/clk/rockchip/clk-rk3576.c b/drivers/clk/rockchip/clk-rk3576.c index 9bc0ef51ef68..95693e438143 100644 --- a/drivers/clk/rockchip/clk-rk3576.c +++ b/drivers/clk/rockchip/clk-rk3576.c @@ -315,6 +315,7 @@ PNAME(mux_100m_24m_lclk0_p) = { "clk_cpll_div10", "xin24m", "lclk_asrc_src_0" } PNAME(mux_100m_24m_lclk1_p) = { "clk_cpll_div10", "xin24m", "lclk_asrc_src_1" }; PNAME(mux_150m_100m_50m_24m_p) = { "clk_gpll_div8", "clk_cpll_div10", "clk_cpll_div20", "xin24m" }; PNAME(mux_200m_100m_50m_24m_p) = { "clk_gpll_div6", "clk_cpll_div10", "clk_cpll_div20", "xin24m" }; +PNAME(mux_200m_150m_100m_24m_p) = { "clk_gpll_div6", "clk_gpll_div8", "clk_cpll_div10", "xin24m" }; PNAME(mux_400m_200m_100m_24m_p) = { "clk_gpll_div3", "clk_gpll_div6", "clk_cpll_div10", "xin24m" }; PNAME(mux_500m_250m_100m_24m_p) = { "clk_cpll_div2", "clk_cpll_div4", "clk_cpll_div10", "xin24m" }; PNAME(mux_600m_400m_300m_24m_p) = { "clk_gpll_div2", "clk_gpll_div3", "clk_gpll_div4", "xin24m" }; @@ -706,19 +707,19 @@ static struct rockchip_clk_branch rk3576_clk_branches[] __initdata = { RK3576_CLKGATE_CON(16), 0, GFLAGS), GATE(PCLK_SPI4, "pclk_spi4", "pclk_bus_root", 0, RK3576_CLKGATE_CON(16), 1, GFLAGS), - COMPOSITE_NODIV(CLK_SPI0, "clk_spi0", mux_200m_100m_50m_24m_p, 0, + COMPOSITE_NODIV(CLK_SPI0, "clk_spi0", mux_200m_150m_100m_24m_p, 0, RK3576_CLKSEL_CON(70), 13, 2, MFLAGS, RK3576_CLKGATE_CON(16), 2, GFLAGS), - COMPOSITE_NODIV(CLK_SPI1, "clk_spi1", mux_200m_100m_50m_24m_p, 0, + COMPOSITE_NODIV(CLK_SPI1, "clk_spi1", mux_200m_150m_100m_24m_p, 0, RK3576_CLKSEL_CON(71), 0, 2, MFLAGS, RK3576_CLKGATE_CON(16), 3, GFLAGS), - COMPOSITE_NODIV(CLK_SPI2, "clk_spi2", mux_200m_100m_50m_24m_p, 0, + COMPOSITE_NODIV(CLK_SPI2, "clk_spi2", mux_200m_150m_100m_24m_p, 0, RK3576_CLKSEL_CON(71), 2, 2, MFLAGS, RK3576_CLKGATE_CON(16), 4, GFLAGS), - COMPOSITE_NODIV(CLK_SPI3, "clk_spi3", mux_200m_100m_50m_24m_p, 0, + COMPOSITE_NODIV(CLK_SPI3, "clk_spi3", mux_200m_150m_100m_24m_p, 0, RK3576_CLKSEL_CON(71), 4, 2, MFLAGS, RK3576_CLKGATE_CON(16), 5, GFLAGS), - COMPOSITE_NODIV(CLK_SPI4, "clk_spi4", mux_200m_100m_50m_24m_p, 0, + COMPOSITE_NODIV(CLK_SPI4, "clk_spi4", mux_200m_150m_100m_24m_p, 0, RK3576_CLKSEL_CON(71), 6, 2, MFLAGS, RK3576_CLKGATE_CON(16), 6, GFLAGS), GATE(PCLK_WDT0, "pclk_wdt0", "pclk_bus_root", 0, diff --git a/drivers/clk/samsung/clk-s5pv210-audss.c b/drivers/clk/samsung/clk-s5pv210-audss.c index b1fd8fac3a4c..c9fcb23de183 100644 --- a/drivers/clk/samsung/clk-s5pv210-audss.c +++ b/drivers/clk/samsung/clk-s5pv210-audss.c @@ -36,7 +36,7 @@ static unsigned long reg_save[][2] = { {ASS_CLK_GATE, 0}, }; -static int s5pv210_audss_clk_suspend(void) +static int s5pv210_audss_clk_suspend(void *data) { int i; @@ -46,7 +46,7 @@ static int s5pv210_audss_clk_suspend(void) return 0; } -static void s5pv210_audss_clk_resume(void) +static void s5pv210_audss_clk_resume(void *data) { int i; @@ -54,10 +54,14 @@ static void s5pv210_audss_clk_resume(void) writel(reg_save[i][1], reg_base + reg_save[i][0]); } -static struct syscore_ops s5pv210_audss_clk_syscore_ops = { +static const struct syscore_ops s5pv210_audss_clk_syscore_ops = { .suspend = s5pv210_audss_clk_suspend, .resume = s5pv210_audss_clk_resume, }; + +static struct syscore s5pv210_audss_clk_syscore = { + .ops = &s5pv210_audss_clk_syscore_ops, +}; #endif /* CONFIG_PM_SLEEP */ /* register s5pv210_audss clocks */ @@ -175,7 +179,7 @@ static int s5pv210_audss_clk_probe(struct platform_device *pdev) } #ifdef CONFIG_PM_SLEEP - register_syscore_ops(&s5pv210_audss_clk_syscore_ops); + register_syscore(&s5pv210_audss_clk_syscore); #endif return 0; diff --git a/drivers/clk/samsung/clk.c b/drivers/clk/samsung/clk.c index dbc9925ca8f4..c149ca6c2217 100644 --- a/drivers/clk/samsung/clk.c +++ b/drivers/clk/samsung/clk.c @@ -271,7 +271,7 @@ void __init samsung_clk_of_register_fixed_ext(struct samsung_clk_provider *ctx, } #ifdef CONFIG_PM_SLEEP -static int samsung_clk_suspend(void) +static int samsung_clk_suspend(void *data) { struct samsung_clock_reg_cache *reg_cache; @@ -284,7 +284,7 @@ static int samsung_clk_suspend(void) return 0; } -static void samsung_clk_resume(void) +static void samsung_clk_resume(void *data) { struct samsung_clock_reg_cache *reg_cache; @@ -293,11 +293,15 @@ static void samsung_clk_resume(void) reg_cache->rd_num); } -static struct syscore_ops samsung_clk_syscore_ops = { +static const struct syscore_ops samsung_clk_syscore_ops = { .suspend = samsung_clk_suspend, .resume = samsung_clk_resume, }; +static struct syscore samsung_clk_syscore = { + .ops = &samsung_clk_syscore_ops, +}; + void samsung_clk_extended_sleep_init(void __iomem *reg_base, const unsigned long *rdump, unsigned long nr_rdump, @@ -316,7 +320,7 @@ void samsung_clk_extended_sleep_init(void __iomem *reg_base, panic("could not allocate register dump storage.\n"); if (list_empty(&clock_reg_cache_list)) - register_syscore_ops(&samsung_clk_syscore_ops); + register_syscore(&samsung_clk_syscore); reg_cache->reg_base = reg_base; reg_cache->rd_num = nr_rdump; diff --git a/drivers/clk/stm32/clk-stm32mp21.c b/drivers/clk/stm32/clk-stm32mp21.c index c8a37b716bd5..bdb17419908c 100644 --- a/drivers/clk/stm32/clk-stm32mp21.c +++ b/drivers/clk/stm32/clk-stm32mp21.c @@ -4,6 +4,7 @@ * Author: Gabriel Fernandez for STMicroelectronics. */ +#include #include #include #include diff --git a/drivers/clk/stm32/clk-stm32mp25.c b/drivers/clk/stm32/clk-stm32mp25.c index 52f0e8a12926..eb0bc918ecee 100644 --- a/drivers/clk/stm32/clk-stm32mp25.c +++ b/drivers/clk/stm32/clk-stm32mp25.c @@ -4,6 +4,7 @@ * Author: Gabriel Fernandez for STMicroelectronics. */ +#include #include #include #include diff --git a/drivers/clk/tegra/clk-tegra124-emc.c b/drivers/clk/tegra/clk-tegra124-emc.c index 5f1af6dfe715..674aef278539 100644 --- a/drivers/clk/tegra/clk-tegra124-emc.c +++ b/drivers/clk/tegra/clk-tegra124-emc.c @@ -539,6 +539,7 @@ struct clk *tegra124_clk_register_emc(void __iomem *base, struct device_node *np clk = clk_register(NULL, &tegra->hw); if (IS_ERR(clk)) { + of_node_put(tegra->emc_node); kfree(tegra); return clk; } diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c index 412902f573b5..504d0ea997a5 100644 --- a/drivers/clk/tegra/clk-tegra210.c +++ b/drivers/clk/tegra/clk-tegra210.c @@ -3444,7 +3444,7 @@ static void tegra210_disable_cpu_clock(u32 cpu) static u32 spare_reg_ctx, misc_clk_enb_ctx, clk_msk_arm_ctx; static u32 cpu_softrst_ctx[3]; -static int tegra210_clk_suspend(void) +static int tegra210_clk_suspend(void *data) { unsigned int i; @@ -3465,7 +3465,7 @@ static int tegra210_clk_suspend(void) return 0; } -static void tegra210_clk_resume(void) +static void tegra210_clk_resume(void *data) { unsigned int i; @@ -3523,13 +3523,17 @@ static void tegra210_cpu_clock_resume(void) } #endif -static struct syscore_ops tegra_clk_syscore_ops = { +static const struct syscore_ops tegra_clk_syscore_ops = { #ifdef CONFIG_PM_SLEEP .suspend = tegra210_clk_suspend, .resume = tegra210_clk_resume, #endif }; +static struct syscore tegra_clk_syscore = { + .ops = &tegra_clk_syscore_ops, +}; + static struct tegra_cpu_car_ops tegra210_cpu_car_ops = { .wait_for_reset = tegra210_wait_cpu_in_reset, .disable_clock = tegra210_disable_cpu_clock, @@ -3813,6 +3817,6 @@ static void __init tegra210_clock_init(struct device_node *np) tegra_cpu_car_ops = &tegra210_cpu_car_ops; - register_syscore_ops(&tegra_clk_syscore_ops); + register_syscore(&tegra_clk_syscore); } CLK_OF_DECLARE(tegra210, "nvidia,tegra210-car", tegra210_clock_init); diff --git a/drivers/clk/ti/adpll.c b/drivers/clk/ti/adpll.c index e305fcbac647..8885d28face5 100644 --- a/drivers/clk/ti/adpll.c +++ b/drivers/clk/ti/adpll.c @@ -483,7 +483,7 @@ static const struct clk_ops ti_adpll_ops = { static int ti_adpll_init_dco(struct ti_adpll_data *d) { - struct clk_init_data init; + struct clk_init_data init = {}; struct clk *clock; const char *postfix; int width, err; @@ -576,7 +576,7 @@ static int ti_adpll_init_clkout(struct ti_adpll_data *d, struct clk *clk1) { struct ti_adpll_clkout_data *co; - struct clk_init_data init; + struct clk_init_data init = {}; struct clk_ops *ops; const char *parent_names[2]; const char *child_name; diff --git a/drivers/clk/ti/divider.c b/drivers/clk/ti/divider.c index 6f58a0f2e74a..ea744c343eac 100644 --- a/drivers/clk/ti/divider.c +++ b/drivers/clk/ti/divider.c @@ -311,7 +311,7 @@ static struct clk *_register_divider(struct device_node *node, u32 flags, struct clk_omap_divider *div) { - struct clk_init_data init; + struct clk_init_data init = {}; const char *parent_name; const char *name; diff --git a/drivers/clk/ti/mux.c b/drivers/clk/ti/mux.c index d6d247ff2be5..ef2a952a0afe 100644 --- a/drivers/clk/ti/mux.c +++ b/drivers/clk/ti/mux.c @@ -119,14 +119,14 @@ const struct clk_ops ti_clk_mux_ops = { }; static struct clk *_register_mux(struct device_node *node, const char *name, - const char * const *parent_names, + const struct clk_parent_data *parent_data, u8 num_parents, unsigned long flags, struct clk_omap_reg *reg, u8 shift, u32 mask, s8 latch, u8 clk_mux_flags, u32 *table) { + struct clk_init_data init = {}; struct clk_omap_mux *mux; struct clk *clk; - struct clk_init_data init; /* allocate the mux */ mux = kzalloc(sizeof(*mux), GFP_KERNEL); @@ -136,7 +136,7 @@ static struct clk *_register_mux(struct device_node *node, const char *name, init.name = name; init.ops = &ti_clk_mux_ops; init.flags = flags; - init.parent_names = parent_names; + init.parent_data = parent_data; init.num_parents = num_parents; /* struct clk_mux assignments */ @@ -167,24 +167,26 @@ static void of_mux_clk_setup(struct device_node *node) struct clk *clk; struct clk_omap_reg reg; unsigned int num_parents; - const char **parent_names; + struct clk_parent_data *parent_data; const char *name; u8 clk_mux_flags = 0; u32 mask = 0; u32 shift = 0; s32 latch = -EINVAL; u32 flags = CLK_SET_RATE_NO_REPARENT; + int i; num_parents = of_clk_get_parent_count(node); if (num_parents < 2) { pr_err("mux-clock %pOFn must have parents\n", node); return; } - parent_names = kcalloc(num_parents, sizeof(char *), GFP_KERNEL); - if (!parent_names) - goto cleanup; + parent_data = kcalloc(num_parents, sizeof(*parent_data), GFP_KERNEL); + if (!parent_data) + return; - of_clk_parent_fill(node, parent_names, num_parents); + for (i = 0; i < num_parents; i++) + parent_data[i].index = i; if (ti_clk_get_reg_addr(node, 0, ®)) goto cleanup; @@ -207,7 +209,7 @@ static void of_mux_clk_setup(struct device_node *node) mask = (1 << fls(mask)) - 1; name = ti_dt_clk_name(node); - clk = _register_mux(node, name, parent_names, num_parents, + clk = _register_mux(node, name, parent_data, num_parents, flags, ®, shift, mask, latch, clk_mux_flags, NULL); @@ -215,7 +217,7 @@ static void of_mux_clk_setup(struct device_node *node) of_clk_add_provider(node, of_clk_src_simple_get, clk); cleanup: - kfree(parent_names); + kfree(parent_data); } CLK_OF_DECLARE(mux_clk, "ti,mux-clock", of_mux_clk_setup); diff --git a/drivers/clk/visconti/clkc.c b/drivers/clk/visconti/clkc.c index d0b193b5d0b3..4018d1298880 100644 --- a/drivers/clk/visconti/clkc.c +++ b/drivers/clk/visconti/clkc.c @@ -81,9 +81,9 @@ static struct clk_hw *visconti_clk_register_gate(struct device *dev, u8 rs_idx, spinlock_t *lock) { + struct clk_init_data init = {}; struct visconti_clk_gate *gate; struct clk_parent_data *pdata; - struct clk_init_data init; struct clk_hw *hw; int ret; diff --git a/drivers/clk/x86/clk-pmc-atom.c b/drivers/clk/x86/clk-pmc-atom.c index 99291ba65da7..08c83e0abc41 100644 --- a/drivers/clk/x86/clk-pmc-atom.c +++ b/drivers/clk/x86/clk-pmc-atom.c @@ -160,6 +160,9 @@ static struct clk_plt *plt_clk_register(struct platform_device *pdev, int id, return ERR_PTR(-ENOMEM); init.name = kasprintf(GFP_KERNEL, "%s_%d", PLT_CLK_NAME_BASE, id); + if (!init.name) + return ERR_PTR(-ENOMEM); + init.ops = &plt_clk_ops; init.flags = 0; init.parent_names = parent_names; diff --git a/drivers/clocksource/clps711x-timer.c b/drivers/clocksource/clps711x-timer.c index bbceb0289d45..95f8a50ba1eb 100644 --- a/drivers/clocksource/clps711x-timer.c +++ b/drivers/clocksource/clps711x-timer.c @@ -94,7 +94,7 @@ static int __init clps711x_timer_init(struct device_node *np) switch (of_alias_get_id(np, "timer")) { case CLPS711X_CLKSRC_CLOCKSOURCE: clps711x_clksrc_init(clock, base); - break; + return 0; case CLPS711X_CLKSRC_CLOCKEVENT: ret = _clps711x_clkevt_init(clock, base, irq); break; diff --git a/drivers/clocksource/timer-armada-370-xp.c b/drivers/clocksource/timer-armada-370-xp.c index 54284c1c0651..fd441ca0b0ad 100644 --- a/drivers/clocksource/timer-armada-370-xp.c +++ b/drivers/clocksource/timer-armada-370-xp.c @@ -207,14 +207,14 @@ static int armada_370_xp_timer_dying_cpu(unsigned int cpu) static u32 timer0_ctrl_reg, timer0_local_ctrl_reg; -static int armada_370_xp_timer_suspend(void) +static int armada_370_xp_timer_suspend(void *data) { timer0_ctrl_reg = readl(timer_base + TIMER_CTRL_OFF); timer0_local_ctrl_reg = readl(local_base + TIMER_CTRL_OFF); return 0; } -static void armada_370_xp_timer_resume(void) +static void armada_370_xp_timer_resume(void *data) { writel(0xffffffff, timer_base + TIMER0_VAL_OFF); writel(0xffffffff, timer_base + TIMER0_RELOAD_OFF); @@ -222,11 +222,15 @@ static void armada_370_xp_timer_resume(void) writel(timer0_local_ctrl_reg, local_base + TIMER_CTRL_OFF); } -static struct syscore_ops armada_370_xp_timer_syscore_ops = { +static const struct syscore_ops armada_370_xp_timer_syscore_ops = { .suspend = armada_370_xp_timer_suspend, .resume = armada_370_xp_timer_resume, }; +static struct syscore armada_370_xp_timer_syscore = { + .ops = &armada_370_xp_timer_syscore_ops, +}; + static unsigned long armada_370_delay_timer_read(void) { return ~readl(timer_base + TIMER0_VAL_OFF); @@ -324,7 +328,7 @@ static int __init armada_370_xp_timer_common_init(struct device_node *np) return res; } - register_syscore_ops(&armada_370_xp_timer_syscore_ops); + register_syscore(&armada_370_xp_timer_syscore); return 0; } @@ -345,7 +349,11 @@ static int __init armada_xp_timer_init(struct device_node *np) timer_clk = clk_get_rate(clk); - return armada_370_xp_timer_common_init(np); + ret = armada_370_xp_timer_common_init(np); + if (ret) + clk_disable_unprepare(clk); + + return ret; } TIMER_OF_DECLARE(armada_xp, "marvell,armada-xp-timer", armada_xp_timer_init); @@ -383,7 +391,11 @@ static int __init armada_375_timer_init(struct device_node *np) timer25Mhz = false; } - return armada_370_xp_timer_common_init(np); + ret = armada_370_xp_timer_common_init(np); + if (ret) + clk_disable_unprepare(clk); + + return ret; } TIMER_OF_DECLARE(armada_375, "marvell,armada-375-timer", armada_375_timer_init); @@ -406,7 +418,11 @@ static int __init armada_370_timer_init(struct device_node *np) timer_clk = clk_get_rate(clk) / TIMER_DIVIDER; timer25Mhz = false; - return armada_370_xp_timer_common_init(np); + ret = armada_370_xp_timer_common_init(np); + if (ret) + clk_disable_unprepare(clk); + + return ret; } TIMER_OF_DECLARE(armada_370, "marvell,armada-370-timer", armada_370_timer_init); diff --git a/drivers/cpufreq/Kconfig.x86 b/drivers/cpufreq/Kconfig.x86 index 2c5c228408bf..027e6ea2e038 100644 --- a/drivers/cpufreq/Kconfig.x86 +++ b/drivers/cpufreq/Kconfig.x86 @@ -40,6 +40,8 @@ config X86_AMD_PSTATE select ACPI_PROCESSOR select ACPI_CPPC_LIB if X86_64 select CPU_FREQ_GOV_SCHEDUTIL if SMP + select ACPI_PLATFORM_PROFILE + select POWER_SUPPLY help This driver adds a CPUFreq driver which utilizes a fine grain processor performance frequency control range instead of legacy @@ -68,6 +70,18 @@ config X86_AMD_PSTATE_DEFAULT_MODE For details, take a look at: . +config X86_AMD_PSTATE_DYNAMIC_EPP + bool "AMD Processor P-State dynamic EPP support" + depends on X86_AMD_PSTATE + default n + help + Allow the kernel to dynamically change the energy performance + value from events like ACPI platform profile and AC adapter plug + events. + + This feature can also be changed at runtime, this configuration + option only sets the kernel default value behavior. + config X86_AMD_PSTATE_UT tristate "selftest for AMD Processor P-State driver" depends on X86 && ACPI_PROCESSOR diff --git a/drivers/cpufreq/amd-pstate-ut.c b/drivers/cpufreq/amd-pstate-ut.c index 447b9aa5ce40..93b80bf992a6 100644 --- a/drivers/cpufreq/amd-pstate-ut.c +++ b/drivers/cpufreq/amd-pstate-ut.c @@ -273,6 +273,11 @@ static int amd_pstate_ut_check_driver(u32 index) static int __init amd_pstate_ut_init(void) { u32 i = 0, arr_size = ARRAY_SIZE(amd_pstate_ut_cases); + enum amd_pstate_mode mode = amd_pstate_get_status(); + + /* don't test if no running amd-pstate driver */ + if (mode == AMD_PSTATE_UNDEFINED || mode == AMD_PSTATE_DISABLE) + return -EOPNOTSUPP; for (i = 0; i < arr_size; i++) { int ret = amd_pstate_ut_cases[i].func(i); diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index ce6d6b3ff58a..b1a200531628 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include @@ -86,6 +87,11 @@ static struct cpufreq_driver amd_pstate_driver; static struct cpufreq_driver amd_pstate_epp_driver; static int cppc_state = AMD_PSTATE_UNDEFINED; static bool amd_pstate_prefcore = true; +#ifdef CONFIG_X86_AMD_PSTATE_DYNAMIC_EPP +static bool dynamic_epp = CONFIG_X86_AMD_PSTATE_DYNAMIC_EPP; +#else +static bool dynamic_epp; +#endif static struct quirk_entry *quirks; /* @@ -103,6 +109,7 @@ static struct quirk_entry *quirks; * 2 balance_performance * 3 balance_power * 4 power + * 5 custom (for raw EPP values) */ enum energy_perf_value_index { EPP_INDEX_DEFAULT = 0, @@ -110,6 +117,8 @@ enum energy_perf_value_index { EPP_INDEX_BALANCE_PERFORMANCE, EPP_INDEX_BALANCE_POWERSAVE, EPP_INDEX_POWERSAVE, + EPP_INDEX_CUSTOM, + EPP_INDEX_MAX, }; static const char * const energy_perf_strings[] = { @@ -118,8 +127,9 @@ static const char * const energy_perf_strings[] = { [EPP_INDEX_BALANCE_PERFORMANCE] = "balance_performance", [EPP_INDEX_BALANCE_POWERSAVE] = "balance_power", [EPP_INDEX_POWERSAVE] = "power", - NULL + [EPP_INDEX_CUSTOM] = "custom", }; +static_assert(ARRAY_SIZE(energy_perf_strings) == EPP_INDEX_MAX); static unsigned int epp_values[] = { [EPP_INDEX_DEFAULT] = 0, @@ -127,7 +137,8 @@ static unsigned int epp_values[] = { [EPP_INDEX_BALANCE_PERFORMANCE] = AMD_CPPC_EPP_BALANCE_PERFORMANCE, [EPP_INDEX_BALANCE_POWERSAVE] = AMD_CPPC_EPP_BALANCE_POWERSAVE, [EPP_INDEX_POWERSAVE] = AMD_CPPC_EPP_POWERSAVE, - }; +}; +static_assert(ARRAY_SIZE(epp_values) == EPP_INDEX_MAX - 1); typedef int (*cppc_mode_transition_fn)(int); @@ -183,7 +194,7 @@ static inline int get_mode_idx_from_str(const char *str, size_t size) { int i; - for (i=0; i < AMD_PSTATE_MAX; i++) { + for (i = 0; i < AMD_PSTATE_MAX; i++) { if (!strncmp(str, amd_pstate_mode_string[i], size)) return i; } @@ -192,7 +203,7 @@ static inline int get_mode_idx_from_str(const char *str, size_t size) static DEFINE_MUTEX(amd_pstate_driver_lock); -static u8 msr_get_epp(struct amd_cpudata *cpudata) +static int msr_get_epp(struct amd_cpudata *cpudata) { u64 value; int ret; @@ -208,12 +219,12 @@ static u8 msr_get_epp(struct amd_cpudata *cpudata) DEFINE_STATIC_CALL(amd_pstate_get_epp, msr_get_epp); -static inline s16 amd_pstate_get_epp(struct amd_cpudata *cpudata) +static inline int amd_pstate_get_epp(struct amd_cpudata *cpudata) { return static_call(amd_pstate_get_epp)(cpudata); } -static u8 shmem_get_epp(struct amd_cpudata *cpudata) +static int shmem_get_epp(struct amd_cpudata *cpudata) { u64 epp; int ret; @@ -452,9 +463,6 @@ static int shmem_init_perf(struct amd_cpudata *cpudata) WRITE_ONCE(cpudata->perf, perf); WRITE_ONCE(cpudata->prefcore_ranking, cppc_perf.highest_perf); - if (cppc_state == AMD_PSTATE_ACTIVE) - return 0; - ret = cppc_get_auto_sel(cpudata->cpu, &auto_sel); if (ret) { pr_warn("failed to get auto_sel, ret: %d\n", ret); @@ -633,8 +641,31 @@ static void amd_pstate_update_min_max_limit(struct cpufreq_policy *policy) WRITE_ONCE(cpudata->max_limit_freq, policy->max); if (cpudata->policy == CPUFREQ_POLICY_PERFORMANCE) { - perf.min_limit_perf = min(perf.nominal_perf, perf.max_limit_perf); - WRITE_ONCE(cpudata->min_limit_freq, min(cpudata->nominal_freq, cpudata->max_limit_freq)); + u8 min_limit_perf = perf.bios_min_perf ?: perf.nominal_perf; + u32 min_limit_freq; + + /* + * For performance policy, set MinPerf to nominal_perf / bios_min_perf + * rather than highest_perf or lowest_nonlinear_perf. + * + * Per commit 0c411b39e4f4c, using highest_perf was observed + * to cause frequency throttling on power-limited platforms, leading to + * performance regressions. Using lowest_nonlinear_perf would limit + * performance too much for HPC workloads requiring high frequency + * operation and minimal wakeup latency from idle states. + * + * nominal_perf therefore provides a balanced default by avoiding + * throttling while still maintaining enough performance for HPC + * workloads when bios_min_perf is not available. + * + * When bios_min_perf is available, users have profiled their workloads + * to understand the best idling frequency. Use that instead. + */ + min_limit_perf = min(min_limit_perf, perf.max_limit_perf); + min_limit_freq = perf_to_freq(perf, cpudata->nominal_freq, min_limit_perf); + perf.min_limit_perf = min_limit_perf; + + WRITE_ONCE(cpudata->min_limit_freq, min(min_limit_freq, cpudata->max_limit_freq)); } else { perf.min_limit_perf = freq_to_perf(perf, cpudata->nominal_freq, policy->min); WRITE_ONCE(cpudata->min_limit_freq, policy->min); @@ -1063,6 +1094,174 @@ static void amd_pstate_cpu_exit(struct cpufreq_policy *policy) kfree(cpudata); } +static int amd_pstate_get_balanced_epp(struct cpufreq_policy *policy) +{ + struct amd_cpudata *cpudata = policy->driver_data; + + if (power_supply_is_system_supplied()) + return cpudata->epp_default_ac; + else + return cpudata->epp_default_dc; +} + +static int amd_pstate_power_supply_notifier(struct notifier_block *nb, + unsigned long event, void *data) +{ + struct amd_cpudata *cpudata = container_of(nb, struct amd_cpudata, power_nb); + struct cpufreq_policy *policy __free(put_cpufreq_policy) = cpufreq_cpu_get(cpudata->cpu); + u8 epp; + int ret; + + if (event != PSY_EVENT_PROP_CHANGED) + return NOTIFY_OK; + + /* dynamic actions are only applied while platform profile is in balanced */ + if (cpudata->current_profile != PLATFORM_PROFILE_BALANCED) + return 0; + + if (!policy) + return NOTIFY_OK; + + epp = amd_pstate_get_balanced_epp(policy); + + ret = amd_pstate_set_epp(policy, epp); + if (ret) + pr_warn("Failed to set CPU %d EPP %u: %d\n", cpudata->cpu, epp, ret); + + return NOTIFY_OK; +} + +static int amd_pstate_profile_probe(void *drvdata, unsigned long *choices) +{ + set_bit(PLATFORM_PROFILE_LOW_POWER, choices); + set_bit(PLATFORM_PROFILE_BALANCED, choices); + set_bit(PLATFORM_PROFILE_PERFORMANCE, choices); + + return 0; +} + +static int amd_pstate_profile_get(struct device *dev, + enum platform_profile_option *profile) +{ + struct amd_cpudata *cpudata = dev_get_drvdata(dev); + + *profile = cpudata->current_profile; + + return 0; +} + +static int amd_pstate_profile_set(struct device *dev, + enum platform_profile_option profile) +{ + struct amd_cpudata *cpudata = dev_get_drvdata(dev); + struct cpufreq_policy *policy __free(put_cpufreq_policy) = cpufreq_cpu_get(cpudata->cpu); + int ret; + + if (!policy) + return -ENODEV; + + switch (profile) { + case PLATFORM_PROFILE_LOW_POWER: + ret = amd_pstate_set_epp(policy, AMD_CPPC_EPP_POWERSAVE); + if (ret) + return ret; + break; + case PLATFORM_PROFILE_BALANCED: + ret = amd_pstate_set_epp(policy, + amd_pstate_get_balanced_epp(policy)); + if (ret) + return ret; + break; + case PLATFORM_PROFILE_PERFORMANCE: + ret = amd_pstate_set_epp(policy, AMD_CPPC_EPP_PERFORMANCE); + if (ret) + return ret; + break; + default: + pr_err("Unknown Platform Profile %d\n", profile); + return -EOPNOTSUPP; + } + + cpudata->current_profile = profile; + + return 0; +} + +static const struct platform_profile_ops amd_pstate_profile_ops = { + .probe = amd_pstate_profile_probe, + .profile_set = amd_pstate_profile_set, + .profile_get = amd_pstate_profile_get, +}; + +static void amd_pstate_clear_dynamic_epp(struct cpufreq_policy *policy) +{ + struct amd_cpudata *cpudata = policy->driver_data; + + if (cpudata->power_nb.notifier_call) + power_supply_unreg_notifier(&cpudata->power_nb); + if (cpudata->ppdev) { + platform_profile_remove(cpudata->ppdev); + cpudata->ppdev = NULL; + } + kfree(cpudata->profile_name); + cpudata->dynamic_epp = false; +} + +static int amd_pstate_set_dynamic_epp(struct cpufreq_policy *policy) +{ + struct amd_cpudata *cpudata = policy->driver_data; + int ret; + u8 epp; + + switch (cpudata->current_profile) { + case PLATFORM_PROFILE_PERFORMANCE: + epp = AMD_CPPC_EPP_PERFORMANCE; + break; + case PLATFORM_PROFILE_LOW_POWER: + epp = AMD_CPPC_EPP_POWERSAVE; + break; + case PLATFORM_PROFILE_BALANCED: + epp = amd_pstate_get_balanced_epp(policy); + break; + default: + pr_err("Unknown Platform Profile %d\n", cpudata->current_profile); + return -EOPNOTSUPP; + } + ret = amd_pstate_set_epp(policy, epp); + if (ret) + return ret; + + cpudata->profile_name = kasprintf(GFP_KERNEL, "amd-pstate-epp-cpu%d", cpudata->cpu); + if (!cpudata->profile_name) + return -ENOMEM; + + cpudata->ppdev = platform_profile_register(get_cpu_device(policy->cpu), + cpudata->profile_name, + policy->driver_data, + &amd_pstate_profile_ops); + if (IS_ERR(cpudata->ppdev)) { + ret = PTR_ERR(cpudata->ppdev); + goto cleanup; + } + + /* only enable notifier if things will actually change */ + if (cpudata->epp_default_ac != cpudata->epp_default_dc) { + cpudata->power_nb.notifier_call = amd_pstate_power_supply_notifier; + ret = power_supply_reg_notifier(&cpudata->power_nb); + if (ret) + goto cleanup; + } + + cpudata->dynamic_epp = true; + + return 0; + +cleanup: + amd_pstate_clear_dynamic_epp(policy); + + return ret; +} + /* Sysfs attributes */ /* @@ -1135,61 +1334,74 @@ static ssize_t show_amd_pstate_hw_prefcore(struct cpufreq_policy *policy, static ssize_t show_energy_performance_available_preferences( struct cpufreq_policy *policy, char *buf) { - int i = 0; - int offset = 0; + int offset = 0, i; struct amd_cpudata *cpudata = policy->driver_data; if (cpudata->policy == CPUFREQ_POLICY_PERFORMANCE) return sysfs_emit_at(buf, offset, "%s\n", energy_perf_strings[EPP_INDEX_PERFORMANCE]); - while (energy_perf_strings[i] != NULL) - offset += sysfs_emit_at(buf, offset, "%s ", energy_perf_strings[i++]); + for (i = 0; i < ARRAY_SIZE(energy_perf_strings); i++) + offset += sysfs_emit_at(buf, offset, "%s ", energy_perf_strings[i]); offset += sysfs_emit_at(buf, offset, "\n"); return offset; } -static ssize_t store_energy_performance_preference( - struct cpufreq_policy *policy, const char *buf, size_t count) +static ssize_t store_energy_performance_preference(struct cpufreq_policy *policy, + const char *buf, size_t count) { struct amd_cpudata *cpudata = policy->driver_data; - char str_preference[21]; ssize_t ret; + bool raw_epp = false; u8 epp; - ret = sscanf(buf, "%20s", str_preference); - if (ret != 1) - return -EINVAL; - - ret = match_string(energy_perf_strings, -1, str_preference); - if (ret < 0) - return -EINVAL; + if (cpudata->dynamic_epp) { + pr_debug("EPP cannot be set when dynamic EPP is enabled\n"); + return -EBUSY; + } - if (!ret) - epp = cpudata->epp_default; - else - epp = epp_values[ret]; + /* + * if the value matches a number, use that, otherwise see if + * matches an index in the energy_perf_strings array + */ + ret = kstrtou8(buf, 0, &epp); + raw_epp = !ret; + if (ret) { + ret = sysfs_match_string(energy_perf_strings, buf); + if (ret < 0 || ret == EPP_INDEX_CUSTOM) + return -EINVAL; + if (ret) + epp = epp_values[ret]; + else + epp = cpudata->epp_default_dc; + } - if (epp > 0 && policy->policy == CPUFREQ_POLICY_PERFORMANCE) { + if (epp > 0 && cpudata->policy == CPUFREQ_POLICY_PERFORMANCE) { pr_debug("EPP cannot be set under performance policy\n"); return -EBUSY; } ret = amd_pstate_set_epp(policy, epp); + if (ret) + return ret; - return ret ? ret : count; + cpudata->raw_epp = raw_epp; + + return count; } -static ssize_t show_energy_performance_preference( - struct cpufreq_policy *policy, char *buf) +static ssize_t show_energy_performance_preference(struct cpufreq_policy *policy, char *buf) { struct amd_cpudata *cpudata = policy->driver_data; u8 preference, epp; epp = FIELD_GET(AMD_CPPC_EPP_PERF_MASK, cpudata->cppc_req_cached); + if (cpudata->raw_epp) + return sysfs_emit(buf, "%u\n", epp); + switch (epp) { case AMD_CPPC_EPP_PERFORMANCE: preference = EPP_INDEX_PERFORMANCE; @@ -1210,12 +1422,87 @@ static ssize_t show_energy_performance_preference( return sysfs_emit(buf, "%s\n", energy_perf_strings[preference]); } +cpufreq_freq_attr_ro(amd_pstate_max_freq); +cpufreq_freq_attr_ro(amd_pstate_lowest_nonlinear_freq); + +cpufreq_freq_attr_ro(amd_pstate_highest_perf); +cpufreq_freq_attr_ro(amd_pstate_prefcore_ranking); +cpufreq_freq_attr_ro(amd_pstate_hw_prefcore); +cpufreq_freq_attr_rw(energy_performance_preference); +cpufreq_freq_attr_ro(energy_performance_available_preferences); + +struct freq_attr_visibility { + struct freq_attr *attr; + bool (*visibility_fn)(void); +}; + +/* For attributes which are always visible */ +static bool always_visible(void) +{ + return true; +} + +/* Determines whether prefcore related attributes should be visible */ +static bool prefcore_visibility(void) +{ + return amd_pstate_prefcore; +} + +/* Determines whether energy performance preference should be visible */ +static bool epp_visibility(void) +{ + return cppc_state == AMD_PSTATE_ACTIVE; +} + +static struct freq_attr_visibility amd_pstate_attr_visibility[] = { + {&amd_pstate_max_freq, always_visible}, + {&amd_pstate_lowest_nonlinear_freq, always_visible}, + {&amd_pstate_highest_perf, always_visible}, + {&amd_pstate_prefcore_ranking, prefcore_visibility}, + {&amd_pstate_hw_prefcore, prefcore_visibility}, + {&energy_performance_preference, epp_visibility}, + {&energy_performance_available_preferences, epp_visibility}, +}; + +static struct freq_attr **get_freq_attrs(void) +{ + bool attr_visible[ARRAY_SIZE(amd_pstate_attr_visibility)]; + struct freq_attr **attrs; + int i, j, count; + + for (i = 0, count = 0; i < ARRAY_SIZE(amd_pstate_attr_visibility); i++) { + struct freq_attr_visibility *v = &amd_pstate_attr_visibility[i]; + + attr_visible[i] = v->visibility_fn(); + if (attr_visible[i]) + count++; + } + + /* amd_pstate_{max_freq, lowest_nonlinear_freq, highest_perf} should always be visible */ + BUG_ON(!count); + + attrs = kcalloc(count + 1, sizeof(struct freq_attr *), GFP_KERNEL); + if (!attrs) + return ERR_PTR(-ENOMEM); + + for (i = 0, j = 0; i < ARRAY_SIZE(amd_pstate_attr_visibility); i++) { + if (!attr_visible[i]) + continue; + + attrs[j++] = amd_pstate_attr_visibility[i].attr; + } + + return attrs; +} + static void amd_pstate_driver_cleanup(void) { if (amd_pstate_prefcore) sched_clear_itmt_support(); cppc_state = AMD_PSTATE_DISABLE; + kfree(current_pstate_driver->attr); + current_pstate_driver->attr = NULL; current_pstate_driver = NULL; } @@ -1240,6 +1527,7 @@ static int amd_pstate_set_driver(int mode_idx) static int amd_pstate_register_driver(int mode) { + struct freq_attr **attr = NULL; int ret; ret = amd_pstate_set_driver(mode); @@ -1248,6 +1536,22 @@ static int amd_pstate_register_driver(int mode) cppc_state = mode; + /* + * Note: It is important to compute the attrs _after_ + * re-initializing the cppc_state. Some attributes become + * visible only when cppc_state is AMD_PSTATE_ACTIVE. + */ + attr = get_freq_attrs(); + if (IS_ERR(attr)) { + ret = (int) PTR_ERR(attr); + pr_err("Couldn't compute freq_attrs for current mode %s [%d]\n", + amd_pstate_get_mode_string(cppc_state), ret); + amd_pstate_driver_cleanup(); + return ret; + } + + current_pstate_driver->attr = attr; + /* at least one CPU supports CPB */ current_pstate_driver->boost_enabled = cpu_feature_enabled(X86_FEATURE_CPB); @@ -1291,6 +1595,8 @@ static int amd_pstate_change_driver_mode(int mode) { int ret; + lockdep_assert_held(&amd_pstate_driver_lock); + ret = amd_pstate_unregister_driver(0); if (ret) return ret; @@ -1390,40 +1696,50 @@ static ssize_t prefcore_show(struct device *dev, return sysfs_emit(buf, "%s\n", str_enabled_disabled(amd_pstate_prefcore)); } -cpufreq_freq_attr_ro(amd_pstate_max_freq); -cpufreq_freq_attr_ro(amd_pstate_lowest_nonlinear_freq); +static ssize_t dynamic_epp_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%s\n", str_enabled_disabled(dynamic_epp)); +} -cpufreq_freq_attr_ro(amd_pstate_highest_perf); -cpufreq_freq_attr_ro(amd_pstate_prefcore_ranking); -cpufreq_freq_attr_ro(amd_pstate_hw_prefcore); -cpufreq_freq_attr_rw(energy_performance_preference); -cpufreq_freq_attr_ro(energy_performance_available_preferences); -static DEVICE_ATTR_RW(status); -static DEVICE_ATTR_RO(prefcore); +static ssize_t dynamic_epp_store(struct device *a, struct device_attribute *b, + const char *buf, size_t count) +{ + bool enabled; + int ret; -static struct freq_attr *amd_pstate_attr[] = { - &amd_pstate_max_freq, - &amd_pstate_lowest_nonlinear_freq, - &amd_pstate_highest_perf, - &amd_pstate_prefcore_ranking, - &amd_pstate_hw_prefcore, - NULL, -}; + ret = kstrtobool(buf, &enabled); + if (ret) + return ret; -static struct freq_attr *amd_pstate_epp_attr[] = { - &amd_pstate_max_freq, - &amd_pstate_lowest_nonlinear_freq, - &amd_pstate_highest_perf, - &amd_pstate_prefcore_ranking, - &amd_pstate_hw_prefcore, - &energy_performance_preference, - &energy_performance_available_preferences, - NULL, -}; + guard(mutex)(&amd_pstate_driver_lock); + + if (cppc_state != AMD_PSTATE_ACTIVE) { + pr_debug("dynamic_epp can only be toggled in active mode\n"); + return -EINVAL; + } + + /* Nothing to do */ + if (dynamic_epp == enabled) + return count; + + /* reinitialize with desired dynamic EPP value */ + dynamic_epp = enabled; + ret = amd_pstate_change_driver_mode(cppc_state); + if (ret) + dynamic_epp = false; + + return ret ? ret : count; +} + +static DEVICE_ATTR_RW(status); +static DEVICE_ATTR_RO(prefcore); +static DEVICE_ATTR_RW(dynamic_epp); static struct attribute *pstate_global_attributes[] = { &dev_attr_status.attr, &dev_attr_prefcore.attr, + &dev_attr_dynamic_epp.attr, NULL }; @@ -1457,6 +1773,7 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) struct amd_cpudata *cpudata; union perf_cached perf; struct device *dev; + int default_epp; int ret; /* @@ -1508,6 +1825,13 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) policy->boost_supported = READ_ONCE(cpudata->boost_supported); + /* Fetch the firmware programmed default EPP value */ + default_epp = amd_pstate_get_epp(cpudata); + if (default_epp < 0) { + ret = default_epp; + goto free_cpudata1; + } + /* * Set the policy to provide a valid fallback value in case * the default cpufreq governor is neither powersave nor performance. @@ -1515,13 +1839,19 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) if (amd_pstate_acpi_pm_profile_server() || amd_pstate_acpi_pm_profile_undefined()) { policy->policy = CPUFREQ_POLICY_PERFORMANCE; - cpudata->epp_default = amd_pstate_get_epp(cpudata); + cpudata->epp_default_ac = cpudata->epp_default_dc = default_epp; + cpudata->current_profile = PLATFORM_PROFILE_PERFORMANCE; } else { policy->policy = CPUFREQ_POLICY_POWERSAVE; - cpudata->epp_default = AMD_CPPC_EPP_BALANCE_PERFORMANCE; + cpudata->epp_default_ac = AMD_CPPC_EPP_PERFORMANCE; + cpudata->epp_default_dc = AMD_CPPC_EPP_BALANCE_PERFORMANCE; + cpudata->current_profile = PLATFORM_PROFILE_BALANCED; } - ret = amd_pstate_set_epp(policy, cpudata->epp_default); + if (dynamic_epp) + ret = amd_pstate_set_dynamic_epp(policy); + else + ret = amd_pstate_set_epp(policy, cpudata->epp_default_dc); if (ret) goto free_cpudata1; @@ -1542,6 +1872,9 @@ static void amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy) if (cpudata) { union perf_cached perf = READ_ONCE(cpudata->perf); + if (cpudata->dynamic_epp) + amd_pstate_clear_dynamic_epp(policy); + /* Reset CPPC_REQ MSR to the BIOS value */ amd_pstate_update_perf(policy, perf.bios_min_perf, 0U, 0U, 0U, false); @@ -1687,7 +2020,6 @@ static struct cpufreq_driver amd_pstate_driver = { .set_boost = amd_pstate_set_boost, .update_limits = amd_pstate_update_limits, .name = "amd-pstate", - .attr = amd_pstate_attr, }; static struct cpufreq_driver amd_pstate_epp_driver = { @@ -1703,7 +2035,6 @@ static struct cpufreq_driver amd_pstate_epp_driver = { .update_limits = amd_pstate_update_limits, .set_boost = amd_pstate_set_boost, .name = "amd-pstate-epp", - .attr = amd_pstate_epp_attr, }; /* @@ -1849,7 +2180,7 @@ static int __init amd_pstate_init(void) return ret; global_attr_free: - cpufreq_unregister_driver(current_pstate_driver); + amd_pstate_unregister_driver(0); return ret; } device_initcall(amd_pstate_init); diff --git a/drivers/cpufreq/amd-pstate.h b/drivers/cpufreq/amd-pstate.h index 75136d2250c1..325d935718f6 100644 --- a/drivers/cpufreq/amd-pstate.h +++ b/drivers/cpufreq/amd-pstate.h @@ -9,6 +9,7 @@ #define _LINUX_AMD_PSTATE_H #include +#include /********************************************************************* * AMD P-state INTERFACE * @@ -77,6 +78,11 @@ struct amd_aperf_mperf { * Only when hw_prefcore and early prefcore param are true, * AMD P-State driver supports preferred core featue. * @policy: Cpufreq policy value + * @suspended: If CPU core if offlined + * @epp_default_ac: Default EPP value for AC power source + * @epp_default_dc: Default EPP value for DC power source + * @dynamic_epp: Whether dynamic EPP is enabled + * @power_nb: Notifier block for power events * * The amd_cpudata is key private data for each CPU thread in AMD P-State, and * represents all the attributes and goals that AMD P-State requests at runtime. @@ -105,7 +111,16 @@ struct amd_cpudata { /* EPP feature related attributes*/ u32 policy; bool suspended; - u8 epp_default; + u8 epp_default_ac; + u8 epp_default_dc; + bool dynamic_epp; + bool raw_epp; + struct notifier_block power_nb; + + /* platform profile */ + enum platform_profile_option current_profile; + struct device *ppdev; + char *profile_name; }; /* diff --git a/drivers/cpufreq/imx6q-cpufreq.c b/drivers/cpufreq/imx6q-cpufreq.c index e93697d3edfd..731f5721ff1e 100644 --- a/drivers/cpufreq/imx6q-cpufreq.c +++ b/drivers/cpufreq/imx6q-cpufreq.c @@ -55,7 +55,6 @@ static unsigned int max_freq; static unsigned int transition_latency; static u32 *imx6_soc_volt; -static u32 soc_opp_count; static int imx6q_set_target(struct cpufreq_policy *policy, unsigned int index) { @@ -330,6 +329,7 @@ static int imx6q_cpufreq_probe(struct platform_device *pdev) const struct property *prop; const __be32 *val; u32 nr, i, j; + u32 soc_opp_count = 0; cpu_dev = get_cpu_device(0); if (!cpu_dev) { @@ -400,7 +400,7 @@ static int imx6q_cpufreq_probe(struct platform_device *pdev) } /* Make imx6_soc_volt array's size same as arm opp number */ - imx6_soc_volt = devm_kcalloc(cpu_dev, num, sizeof(*imx6_soc_volt), + imx6_soc_volt = devm_kcalloc(&pdev->dev, num, sizeof(*imx6_soc_volt), GFP_KERNEL); if (imx6_soc_volt == NULL) { ret = -ENOMEM; @@ -485,6 +485,7 @@ static int imx6q_cpufreq_probe(struct platform_device *pdev) return 0; free_freq_table: + imx6_soc_volt = NULL; dev_pm_opp_free_cpufreq_table(cpu_dev, &freq_table); out_free_opp: dev_pm_opp_of_remove_table(cpu_dev); @@ -506,6 +507,7 @@ static int imx6q_cpufreq_probe(struct platform_device *pdev) static void imx6q_cpufreq_remove(struct platform_device *pdev) { cpufreq_unregister_driver(&imx6q_cpufreq_driver); + imx6_soc_volt = NULL; dev_pm_opp_free_cpufreq_table(cpu_dev, &freq_table); dev_pm_opp_of_remove_table(cpu_dev); regulator_put(arm_reg); diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index e6d6a30cf6c9..1de12358c0b9 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -2389,8 +2389,6 @@ static void intel_pstate_get_cpu_pstates(struct cpudata *cpu) if (pstate_funcs.get_vid) pstate_funcs.get_vid(cpu); - - intel_pstate_set_min_pstate(cpu); } /* @@ -3098,6 +3096,7 @@ static int __intel_pstate_cpu_init(struct cpufreq_policy *policy) static int intel_pstate_cpu_init(struct cpufreq_policy *policy) { int ret = __intel_pstate_cpu_init(policy); + struct cpudata *cpu; if (ret) return ret; @@ -3108,11 +3107,11 @@ static int intel_pstate_cpu_init(struct cpufreq_policy *policy) */ policy->policy = CPUFREQ_POLICY_POWERSAVE; - if (hwp_active) { - struct cpudata *cpu = all_cpu_data[policy->cpu]; - + cpu = all_cpu_data[policy->cpu]; + if (hwp_active) cpu->epp_cached = intel_pstate_get_epp(cpu, 0); - } + else + intel_pstate_set_min_pstate(cpu); return 0; } @@ -3336,8 +3335,6 @@ static int intel_cpufreq_cpu_init(struct cpufreq_policy *policy) return ret; policy->cpuinfo.transition_latency = INTEL_CPUFREQ_TRANSITION_LATENCY; - /* This reflects the intel_pstate_get_cpu_pstates() setting. */ - policy->cur = policy->cpuinfo.min_freq; req = kcalloc(2, sizeof(*req), GFP_KERNEL); if (!req) { @@ -3358,9 +3355,15 @@ static int intel_cpufreq_cpu_init(struct cpufreq_policy *policy) WRITE_ONCE(cpu->hwp_req_cached, value); cpu->epp_cached = intel_pstate_get_epp(cpu, value); + + intel_cpufreq_hwp_update(cpu, cpu->pstate.min_pstate, + cpu->pstate.max_pstate, + cpu->pstate.min_pstate, false); } else { policy->transition_delay_us = INTEL_CPUFREQ_TRANSITION_DELAY; + intel_pstate_set_min_pstate(cpu); } + policy->cur = policy->cpuinfo.min_freq; freq = DIV_ROUND_UP(cpu->pstate.turbo_freq * global.min_perf_pct, 100); diff --git a/drivers/cpufreq/spear-cpufreq.c b/drivers/cpufreq/spear-cpufreq.c index 2a1550e1aa21..6f8cd59baa36 100644 --- a/drivers/cpufreq/spear-cpufreq.c +++ b/drivers/cpufreq/spear-cpufreq.c @@ -79,9 +79,9 @@ static int spear1340_set_cpu_rate(struct clk *sys_pclk, unsigned long newfreq) int ret = 0; sys_clk = clk_get_parent(spear_cpufreq.clk); - if (IS_ERR(sys_clk)) { + if (!sys_clk) { pr_err("failed to get cpu's parent (sys) clock\n"); - return PTR_ERR(sys_clk); + return -EINVAL; } /* Set the rate of the source clock before changing the parent */ diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c index 72cdd8b1f451..49156e67db27 100644 --- a/drivers/cpuidle/cpuidle-psci.c +++ b/drivers/cpuidle/cpuidle-psci.c @@ -177,26 +177,30 @@ static void psci_idle_syscore_switch(bool suspend) } } -static int psci_idle_syscore_suspend(void) +static int psci_idle_syscore_suspend(void *data) { psci_idle_syscore_switch(true); return 0; } -static void psci_idle_syscore_resume(void) +static void psci_idle_syscore_resume(void *data) { psci_idle_syscore_switch(false); } -static struct syscore_ops psci_idle_syscore_ops = { +static const struct syscore_ops psci_idle_syscore_ops = { .suspend = psci_idle_syscore_suspend, .resume = psci_idle_syscore_resume, }; +static struct syscore psci_idle_syscore = { + .ops = &psci_idle_syscore_ops, +}; + static void psci_idle_init_syscore(void) { if (psci_cpuidle_use_syscore) - register_syscore_ops(&psci_idle_syscore_ops); + register_syscore(&psci_idle_syscore); } static void psci_idle_init_cpuhp(void) diff --git a/drivers/crypto/atmel-ecc.c b/drivers/crypto/atmel-ecc.c index 91edb42aba4a..93e3513541c7 100644 --- a/drivers/crypto/atmel-ecc.c +++ b/drivers/crypto/atmel-ecc.c @@ -165,6 +165,9 @@ static int atmel_ecdh_compute_shared_secret(struct kpp_request *req) return crypto_kpp_compute_shared_secret(req); } + if (!ctx->public_key) + return -EINVAL; + /* A P-256 public key must contain two 32-byte coordinates */ if (req->src_len != ATMEL_ECC_PUBKEY_SIZE) return -EINVAL; diff --git a/drivers/crypto/atmel-sha204a.c b/drivers/crypto/atmel-sha204a.c index 8a3520c9a0ff..d2031abbd8ef 100644 --- a/drivers/crypto/atmel-sha204a.c +++ b/drivers/crypto/atmel-sha204a.c @@ -31,10 +31,14 @@ static void atmel_sha204a_rng_done(struct atmel_i2c_work_data *work_data, struct atmel_i2c_client_priv *i2c_priv = work_data->ctx; struct hwrng *rng = areq; - if (status) + if (status) { dev_warn_ratelimited(&i2c_priv->client->dev, "i2c transaction failed (%d)\n", status); + kfree(work_data); + atomic_dec(&i2c_priv->tfm_count); + return; + } rng->priv = (unsigned long)work_data; atomic_dec(&i2c_priv->tfm_count); diff --git a/drivers/crypto/ccp/sev-dev.c b/drivers/crypto/ccp/sev-dev.c index 4eaad21fd848..5aa0f158c43b 100644 --- a/drivers/crypto/ccp/sev-dev.c +++ b/drivers/crypto/ccp/sev-dev.c @@ -1514,7 +1514,7 @@ static int __sev_platform_init_handle_init_ex_path(struct sev_device *sev) if (sev_init_ex_buffer) return 0; - page = alloc_pages(GFP_KERNEL, get_order(NV_LENGTH)); + page = alloc_pages(GFP_KERNEL | __GFP_ZERO, get_order(NV_LENGTH)); if (!page) { dev_err(sev->dev, "SEV: INIT_EX NV memory allocation failed\n"); return -ENOMEM; @@ -1524,7 +1524,7 @@ static int __sev_platform_init_handle_init_ex_path(struct sev_device *sev) rc = sev_read_init_ex_file(); if (rc) - return rc; + goto err_free; /* If SEV-SNP is initialized, transition to firmware page. */ if (sev->snp_initialized) { @@ -1533,11 +1533,22 @@ static int __sev_platform_init_handle_init_ex_path(struct sev_device *sev) npages = 1UL << get_order(NV_LENGTH); if (rmp_mark_pages_firmware(__pa(sev_init_ex_buffer), npages, false)) { dev_err(sev->dev, "SEV: INIT_EX NV memory page state change failed.\n"); - return -ENOMEM; + rc = -ENOMEM; + /* + * Pages can be in an inconsistent state, don't release them back to the + * system. + */ + goto err_reset; } } return 0; + +err_free: + __free_pages(page, get_order(NV_LENGTH)); +err_reset: + sev_init_ex_buffer = NULL; + return rc; } static int __sev_platform_init_locked(int *error) diff --git a/drivers/crypto/gemini/sl3516-ce-cipher.c b/drivers/crypto/gemini/sl3516-ce-cipher.c index 583010b2d007..02ec4282333b 100644 --- a/drivers/crypto/gemini/sl3516-ce-cipher.c +++ b/drivers/crypto/gemini/sl3516-ce-cipher.c @@ -56,10 +56,6 @@ static bool sl3516_ce_need_fallback(struct skcipher_request *areq) ce->fallback_mod16++; return true; } - if ((sg_dma_len(sg) % 16) != 0) { - ce->fallback_mod16++; - return true; - } if (!IS_ALIGNED(sg->offset, 16)) { ce->fallback_align16++; return true; @@ -72,10 +68,6 @@ static bool sl3516_ce_need_fallback(struct skcipher_request *areq) ce->fallback_mod16++; return true; } - if ((sg_dma_len(sg) % 16) != 0) { - ce->fallback_mod16++; - return true; - } if (!IS_ALIGNED(sg->offset, 16)) { ce->fallback_align16++; return true; diff --git a/drivers/crypto/hisilicon/sec2/sec_crypto.c b/drivers/crypto/hisilicon/sec2/sec_crypto.c index 2d0b248c59eb..66d76134f1df 100644 --- a/drivers/crypto/hisilicon/sec2/sec_crypto.c +++ b/drivers/crypto/hisilicon/sec2/sec_crypto.c @@ -96,7 +96,6 @@ #define IV_FLAGS_OFFSET 0x6 #define IV_CM_OFFSET 0x3 #define IV_LAST_BYTE1 1 -#define IV_LAST_BYTE2 2 #define IV_LAST_BYTE_MASK 0xFF #define IV_CTR_INIT 0x1 #define IV_BYTE_OFFSET 0x8 @@ -1697,7 +1696,7 @@ static void set_aead_auth_iv(struct sec_ctx *ctx, struct sec_req *req) struct sec_cipher_req *c_req = &req->c_req; u32 data_size = aead_req->cryptlen; u8 flage = 0; - u8 cm, cl; + u8 cm, cl, i; /* the specification has been checked in aead_iv_demension_check() */ cl = c_req->c_ivin[0] + 1; @@ -1721,15 +1720,16 @@ static void set_aead_auth_iv(struct sec_ctx *ctx, struct sec_req *req) * the last 32bit is counter's initial number, * but the nonce uses the first 16bit * the tail 16bit fill with the cipher length + * When CL is 3, the tail 24bit fill with the cipher length. */ if (!c_req->encrypt) data_size = aead_req->cryptlen - authsize; - a_req->a_ivin[ctx->c_ctx.ivsize - IV_LAST_BYTE1] = - data_size & IV_LAST_BYTE_MASK; - data_size >>= IV_BYTE_OFFSET; - a_req->a_ivin[ctx->c_ctx.ivsize - IV_LAST_BYTE2] = + for (i = 1; i <= cl; i++) { + a_req->a_ivin[ctx->c_ctx.ivsize - i] = data_size & IV_LAST_BYTE_MASK; + data_size >>= IV_BYTE_OFFSET; + } } static void sec_aead_set_iv(struct sec_ctx *ctx, struct sec_req *req) diff --git a/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c b/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c index 8a8f6c81e010..419f88af1031 100644 --- a/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c +++ b/drivers/crypto/intel/keembay/keembay-ocs-aes-core.c @@ -1541,7 +1541,7 @@ static int register_aes_algs(struct ocs_aes_dev *aes_dev) /* * If any algorithm fails to register, all preceding algorithms that - * were successfully registered will be automatically unregistered. + * were registered in the same call are automatically unregistered. */ ret = crypto_engine_register_aeads(algs_aead, ARRAY_SIZE(algs_aead)); if (ret) @@ -1549,7 +1549,7 @@ static int register_aes_algs(struct ocs_aes_dev *aes_dev) ret = crypto_engine_register_skciphers(algs, ARRAY_SIZE(algs)); if (ret) - crypto_engine_unregister_aeads(algs_aead, ARRAY_SIZE(algs)); + crypto_engine_unregister_aeads(algs_aead, ARRAY_SIZE(algs_aead)); return ret; } @@ -1561,6 +1561,7 @@ static const struct of_device_id kmb_ocs_aes_of_match[] = { }, {} }; +MODULE_DEVICE_TABLE(of, kmb_ocs_aes_of_match); static void kmb_ocs_aes_remove(struct platform_device *pdev) { @@ -1602,6 +1603,8 @@ static int kmb_ocs_aes_probe(struct platform_device *pdev) if (IS_ERR(aes_dev->base_reg)) return PTR_ERR(aes_dev->base_reg); + init_completion(&aes_dev->irq_completion); + /* Get and request IRQ */ aes_dev->irq = platform_get_irq(pdev, 0); if (aes_dev->irq < 0) @@ -1619,8 +1622,6 @@ static int kmb_ocs_aes_probe(struct platform_device *pdev) list_add_tail(&aes_dev->list, &ocs_aes.dev_list); spin_unlock(&ocs_aes.lock); - init_completion(&aes_dev->irq_completion); - /* Initialize crypto engine */ aes_dev->engine = crypto_engine_alloc_init(dev, true); if (!aes_dev->engine) { diff --git a/drivers/crypto/intel/qat/qat_common/adf_aer.c b/drivers/crypto/intel/qat/qat_common/adf_aer.c index f26cadc19d6b..ea826333c45f 100644 --- a/drivers/crypto/intel/qat/qat_common/adf_aer.c +++ b/drivers/crypto/intel/qat/qat_common/adf_aer.c @@ -190,6 +190,8 @@ static void adf_device_reset_worker(struct work_struct *work) queue_work(device_sriov_wq, &sriov_data.sriov_work); if (wait_for_completion_timeout(&sriov_data.compl, wait_jiffies)) adf_pf2vf_notify_restarted(accel_dev); + else + cancel_work_sync(&sriov_data.sriov_work); adf_dev_restarted_notify(accel_dev); clear_bit(ADF_STATUS_RESTARTING, &accel_dev->status); diff --git a/drivers/crypto/intel/qat/qat_common/adf_cfg.c b/drivers/crypto/intel/qat/qat_common/adf_cfg.c index 97273da40b43..7180dd2b504a 100644 --- a/drivers/crypto/intel/qat/qat_common/adf_cfg.c +++ b/drivers/crypto/intel/qat/qat_common/adf_cfg.c @@ -2,6 +2,7 @@ /* Copyright(c) 2014 - 2020 Intel Corporation */ #include #include +#include #include #include #include "adf_accel_devices.h" @@ -284,16 +285,13 @@ int adf_cfg_add_key_value_param(struct adf_accel_dev *accel_dev, return -ENOMEM; INIT_LIST_HEAD(&key_val->list); - strscpy(key_val->key, key, sizeof(key_val->key)); + strscpy(key_val->key, key); if (type == ADF_DEC) { snprintf(key_val->val, ADF_CFG_MAX_VAL_LEN_IN_BYTES, "%ld", (*((long *)val))); } else if (type == ADF_STR) { - strscpy(key_val->val, (char *)val, sizeof(key_val->val)); - } else if (type == ADF_HEX) { - snprintf(key_val->val, ADF_CFG_MAX_VAL_LEN_IN_BYTES, - "0x%lx", (unsigned long)val); + strscpy(key_val->val, (char *)val); } else { dev_err(&GET_DEV(accel_dev), "Unknown type given.\n"); kfree(key_val); @@ -350,7 +348,7 @@ int adf_cfg_section_add(struct adf_accel_dev *accel_dev, const char *name) if (!sec) return -ENOMEM; - strscpy(sec->name, name, sizeof(sec->name)); + strscpy(sec->name, name); INIT_LIST_HEAD(&sec->param_head); down_write(&cfg->lock); list_add_tail(&sec->list, &cfg->sec_list); diff --git a/drivers/crypto/intel/qat/qat_common/adf_cfg_common.h b/drivers/crypto/intel/qat/qat_common/adf_cfg_common.h index d63f4dcccbb5..5922d018f5b9 100644 --- a/drivers/crypto/intel/qat/qat_common/adf_cfg_common.h +++ b/drivers/crypto/intel/qat/qat_common/adf_cfg_common.h @@ -28,7 +28,6 @@ enum adf_cfg_service_type { enum adf_cfg_val_type { ADF_DEC, - ADF_HEX, ADF_STR }; diff --git a/drivers/crypto/intel/qat/qat_common/adf_cfg_services.c b/drivers/crypto/intel/qat/qat_common/adf_cfg_services.c index 7d00bcb41ce7..11cba347d12d 100644 --- a/drivers/crypto/intel/qat/qat_common/adf_cfg_services.c +++ b/drivers/crypto/intel/qat/qat_common/adf_cfg_services.c @@ -60,7 +60,7 @@ static int adf_service_string_to_mask(struct adf_accel_dev *accel_dev, const cha if (len > ADF_CFG_MAX_VAL_LEN_IN_BYTES - 1) return -EINVAL; - strscpy(services, buf, ADF_CFG_MAX_VAL_LEN_IN_BYTES); + strscpy(services, buf); substr = services; while ((token = strsep(&substr, ADF_SERVICES_DELIMITER))) { diff --git a/drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c b/drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c index 41cc763a74aa..df4600ab72d6 100644 --- a/drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c +++ b/drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c @@ -2,6 +2,7 @@ /* Copyright(c) 2024 Intel Corporation */ #include +#include #include #include "adf_mstate_mgr.h" @@ -158,7 +159,7 @@ static struct adf_mstate_sect_h *adf_mstate_sect_add_header(struct adf_mstate_mg return NULL; } - strscpy(sect->id, id, sizeof(sect->id)); + strscpy(sect->id, id); sect->size = 0; sect->sub_sects = 0; mgr->state += sizeof(*sect); diff --git a/drivers/crypto/intel/qat/qat_common/adf_transport_debug.c b/drivers/crypto/intel/qat/qat_common/adf_transport_debug.c index 6c22bc9b28e4..ad45ca7e8eb8 100644 --- a/drivers/crypto/intel/qat/qat_common/adf_transport_debug.c +++ b/drivers/crypto/intel/qat/qat_common/adf_transport_debug.c @@ -2,6 +2,7 @@ /* Copyright(c) 2014 - 2020 Intel Corporation */ #include #include +#include #include #include "adf_accel_devices.h" #include "adf_transport_internal.h" @@ -103,7 +104,7 @@ int adf_ring_debugfs_add(struct adf_etr_ring_data *ring, const char *name) if (!ring_debug) return -ENOMEM; - strscpy(ring_debug->ring_name, name, sizeof(ring_debug->ring_name)); + strscpy(ring_debug->ring_name, name); snprintf(entry_name, sizeof(entry_name), "ring_%02d", ring->ring_number); diff --git a/drivers/crypto/intel/qat/qat_common/qat_algs.c b/drivers/crypto/intel/qat/qat_common/qat_algs.c index 7f638a62e3ad..91663805d9e6 100644 --- a/drivers/crypto/intel/qat/qat_common/qat_algs.c +++ b/drivers/crypto/intel/qat/qat_common/qat_algs.c @@ -405,6 +405,7 @@ static void qat_alg_xts_reverse_key(const u8 *key_forward, unsigned int keylen, memcpy(key_reverse + AES_BLOCK_SIZE, key - AES_BLOCK_SIZE, AES_BLOCK_SIZE); } + memzero_explicit(&aes_expanded, sizeof(aes_expanded)); } static void qat_alg_skcipher_init_dec(struct qat_alg_skcipher_ctx *ctx, diff --git a/drivers/crypto/intel/qat/qat_common/qat_compression.c b/drivers/crypto/intel/qat/qat_common/qat_compression.c index 53a4db5507ec..69db4178a419 100644 --- a/drivers/crypto/intel/qat/qat_common/qat_compression.c +++ b/drivers/crypto/intel/qat/qat_common/qat_compression.c @@ -2,6 +2,7 @@ /* Copyright(c) 2022 Intel Corporation */ #include #include +#include #include "adf_accel_devices.h" #include "adf_common_drv.h" #include "adf_transport.h" @@ -123,7 +124,7 @@ static int qat_compression_create_instances(struct adf_accel_dev *accel_dev) int i; INIT_LIST_HEAD(&accel_dev->compression_list); - strscpy(key, ADF_NUM_DC, sizeof(key)); + strscpy(key, ADF_NUM_DC); ret = adf_cfg_get_param_value(accel_dev, SEC, key, val); if (ret) return ret; diff --git a/drivers/crypto/rockchip/rk3288_crypto_ahash.c b/drivers/crypto/rockchip/rk3288_crypto_ahash.c index b9f5a8b42e66..d3482619aa2f 100644 --- a/drivers/crypto/rockchip/rk3288_crypto_ahash.c +++ b/drivers/crypto/rockchip/rk3288_crypto_ahash.c @@ -324,7 +324,12 @@ static int rk_hash_run(struct crypto_engine *engine, void *breq) * efficiency, and make it response quickly when dma * complete. */ - readl_poll_timeout(rkc->reg + RK_CRYPTO_HASH_STS, v, v == 0, 10, 1000); + err = readl_poll_timeout(rkc->reg + RK_CRYPTO_HASH_STS, v, + v == 0, 10, 1000); + if (err) { + dev_err(rkc->dev, "HASH idle timeout\n"); + goto theend; + } for (i = 0; i < crypto_ahash_digestsize(tfm) / 4; i++) { v = readl(rkc->reg + RK_CRYPTO_HASH_DOUT_0 + i * 4); diff --git a/drivers/crypto/sa2ul.c b/drivers/crypto/sa2ul.c index 52fe4baeff93..73034aeb6386 100644 --- a/drivers/crypto/sa2ul.c +++ b/drivers/crypto/sa2ul.c @@ -2395,7 +2395,10 @@ static int sa_ul_probe(struct platform_device *pdev) return ret; } - sa_init_mem(dev_data); + ret = sa_init_mem(dev_data); + if (ret) + goto disable_pm; + ret = sa_dma_init(dev_data); if (ret) goto destroy_dma_pool; @@ -2430,6 +2433,7 @@ static int sa_ul_probe(struct platform_device *pdev) destroy_dma_pool: dma_pool_destroy(dev_data->sc_pool); +disable_pm: pm_runtime_put_sync(dev); pm_runtime_disable(dev); diff --git a/drivers/crypto/xilinx/xilinx-trng.c b/drivers/crypto/xilinx/xilinx-trng.c index 3b193471cc94..7f9762ea042b 100644 --- a/drivers/crypto/xilinx/xilinx-trng.c +++ b/drivers/crypto/xilinx/xilinx-trng.c @@ -91,8 +91,8 @@ static void xtrng_softreset(struct xilinx_rng *rng) xtrng_readwrite32(rng->rng_base + TRNG_CTRL_OFFSET, TRNG_CTRL_PRNGSRST_MASK, 0); } -/* Return no. of bytes read */ -static size_t xtrng_readblock32(void __iomem *rng_base, __be32 *buf, int blocks32, bool wait) +/* Return no. of bytes read or a negative error before any data is read. */ +static int xtrng_readblock32(void __iomem *rng_base, __be32 *buf, int blocks32, bool wait) { int read = 0, ret; int timeout = 1; @@ -107,8 +107,11 @@ static size_t xtrng_readblock32(void __iomem *rng_base, __be32 *buf, int blocks3 ret = readl_poll_timeout(rng_base + TRNG_STATUS_OFFSET, val, (val & TRNG_STATUS_QCNT_MASK) == TRNG_STATUS_QCNT_16_BYTES, !!wait, timeout); - if (ret) + if (ret) { + if (!read) + return ret; break; + } for (idx = 0; idx < TRNG_READ_4_WORD; idx++) { *(buf + read) = cpu_to_be32(ioread32(rng_base + TRNG_CORE_OUTPUT_OFFSET)); @@ -123,27 +126,40 @@ static int xtrng_collect_random_data(struct xilinx_rng *rng, u8 *rand_gen_buf, { u8 randbuf[TRNG_SEC_STRENGTH_BYTES]; int byteleft, blocks, count = 0; + int full_blocks_bytes; int ret; byteleft = no_of_random_bytes & (TRNG_SEC_STRENGTH_BYTES - 1); blocks = no_of_random_bytes >> TRNG_SEC_STRENGTH_SHIFT; + full_blocks_bytes = blocks * TRNG_SEC_STRENGTH_BYTES; xtrng_readwrite32(rng->rng_base + TRNG_CTRL_OFFSET, TRNG_CTRL_PRNGSTART_MASK, TRNG_CTRL_PRNGSTART_MASK); if (blocks) { ret = xtrng_readblock32(rng->rng_base, (__be32 *)rand_gen_buf, blocks, wait); - if (!ret) - return 0; + if (ret <= 0) { + count = ret; + goto out_stop; + } count += ret; + if (ret < full_blocks_bytes) + goto out_stop; } if (byteleft) { ret = xtrng_readblock32(rng->rng_base, (__be32 *)randbuf, 1, wait); + if (ret < 0) { + if (!count) + count = ret; + goto out_stop; + } if (!ret) - return count; - memcpy(rand_gen_buf + (blocks * TRNG_SEC_STRENGTH_BYTES), randbuf, byteleft); - count += byteleft; + goto out_stop; + ret = min(ret, no_of_random_bytes - count); + memcpy(rand_gen_buf + count, randbuf, ret); + count += ret; } +out_stop: xtrng_readwrite32(rng->rng_base + TRNG_CTRL_OFFSET, TRNG_CTRL_PRNGMODE_MASK | TRNG_CTRL_PRNGSTART_MASK, 0U); diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c index 7b0eeb0788d3..6e25223d3409 100644 --- a/drivers/cxl/core/features.c +++ b/drivers/cxl/core/features.c @@ -225,7 +225,7 @@ size_t cxl_get_feature(struct cxl_mailbox *cxl_mbox, const uuid_t *feat_uuid, void *feat_out, size_t feat_out_size, u16 offset, u16 *return_code) { - size_t data_to_rd_size, size_out; + size_t data_to_rd_size; struct cxl_mbox_get_feat_in pi; struct cxl_mbox_cmd mbox_cmd; size_t data_rcvd_size = 0; @@ -237,7 +237,6 @@ size_t cxl_get_feature(struct cxl_mailbox *cxl_mbox, const uuid_t *feat_uuid, if (!feat_out || !feat_out_size) return 0; - size_out = min(feat_out_size, cxl_mbox->payload_size); uuid_copy(&pi.uuid, feat_uuid); pi.selection = selection; do { @@ -250,7 +249,7 @@ size_t cxl_get_feature(struct cxl_mailbox *cxl_mbox, const uuid_t *feat_uuid, .opcode = CXL_MBOX_OP_GET_FEATURE, .size_in = sizeof(pi), .payload_in = &pi, - .size_out = size_out, + .size_out = data_to_rd_size, .payload_out = feat_out + data_rcvd_size, .min_out = data_to_rd_size, }; @@ -471,6 +470,10 @@ static void *cxlctl_get_feature(struct cxl_features_state *cxlfs, if (!count) return ERR_PTR(-EINVAL); + if (out_size < offsetof(struct fwctl_rpc_cxl_out, payload) || + count > out_size - offsetof(struct fwctl_rpc_cxl_out, payload)) + return ERR_PTR(-EINVAL); + struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) = kvzalloc(out_size, GFP_KERNEL); if (!rpc_out) @@ -516,6 +519,9 @@ static void *cxlctl_set_feature(struct cxl_features_state *cxlfs, flags = le32_to_cpu(feat_in->flags); out_size = *out_len; + if (out_size < offsetof(struct fwctl_rpc_cxl_out, payload)) + return ERR_PTR(-EINVAL); + struct fwctl_rpc_cxl_out *rpc_out __free(kvfree) = kvzalloc(out_size, GFP_KERNEL); if (!rpc_out) diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index daee364619f9..e1c23aae2a2e 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -379,11 +379,7 @@ static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox_cmd, } } - /* Prepare to handle a full payload for variable sized output */ - if (out_size == CXL_VARIABLE_PAYLOAD) - mbox_cmd->size_out = cxl_mbox->payload_size; - else - mbox_cmd->size_out = out_size; + mbox_cmd->size_out = min_t(size_t, out_size, cxl_mbox->payload_size); if (mbox_cmd->size_out) { mbox_cmd->payload_out = kvzalloc(mbox_cmd->size_out, GFP_KERNEL); @@ -1453,6 +1449,11 @@ int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len, if (rc) break; + if (!le16_to_cpu(po->count)) { + dev_dbg(&cxlmd->dev, "Poison empty payload!\n"); + break; + } + for (int i = 0; i < le16_to_cpu(po->count); i++) trace_cxl_poison(cxlmd, cxlr, &po->record[i], po->flags, po->overflow_ts, diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 4dff7f44d908..3bb5835aa548 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -870,7 +870,7 @@ static enum fw_upload_err cxl_fw_prepare(struct fw_upload *fwl, const u8 *data, if (!size) return FW_UPLOAD_ERR_INVALID_SIZE; - mds->fw.oneshot = struct_size(transfer, data, size) < + mds->fw.oneshot = struct_size(transfer, data, size) <= cxl_mbox->payload_size; if (cxl_mem_get_fw_info(mds)) diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index c53a13d4f166..2b9e166ea4e0 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -1732,8 +1732,8 @@ static int add_port_attach_ep(struct cxl_memdev *cxlmd, parent_dport, uport_dev, dport_dev); if (IS_ERR(dport)) { - /* Port already exists, restart iteration */ - if (PTR_ERR(dport) == -EAGAIN) + /* Port or dport already exists, restart iteration */ + if (PTR_ERR(dport) == -EAGAIN || PTR_ERR(dport) == -EBUSY) return 0; return PTR_ERR(dport); } diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index a1141471b458..d518d463b28c 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -1815,14 +1815,13 @@ static int find_pos_and_ways(struct cxl_port *port, struct range *range, break; } } - put_device(dev); - if (rc) dev_err(port->uport_dev, "failed to find %s:%s in target list of %s\n", dev_name(&port->dev), - dev_name(port->parent_dport->dport_dev), - dev_name(&cxlsd->cxld.dev)); + dev_name(port->parent_dport->dport_dev), dev_name(dev)); + + put_device(dev); return rc; } diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index 2c42ab75e56a..59c6d503437b 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -807,12 +807,6 @@ static int cxl_pci_type3_init_mailbox(struct cxl_dev_state *cxlds) { int rc; - /* - * Fail the init if there's no mailbox. For a type3 this is out of spec. - */ - if (!cxlds->reg_map.device_map.mbox.valid) - return -ENODEV; - rc = cxl_mailbox_init(&cxlds->cxl_mbox, cxlds->dev); if (rc) return rc; @@ -952,10 +946,13 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) */ rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT, &cxlds->reg_map); - if (rc) + if (rc) { + if (rc == -EPROBE_DEFER) + return rc; dev_warn(&pdev->dev, "No component registers (%d)\n", rc); - else if (!cxlds->reg_map.component_map.ras.valid) + } else if (!cxlds->reg_map.component_map.ras.valid) { dev_dbg(&pdev->dev, "RAS registers not found\n"); + } rc = cxl_map_component_regs(&cxlds->reg_map, &cxlds->regs.component, BIT(CXL_CM_CAP_CAP_ID_RAS)); diff --git a/drivers/dax/super.c b/drivers/dax/super.c index d7714d8afb0f..efa5f6186019 100644 --- a/drivers/dax/super.c +++ b/drivers/dax/super.c @@ -232,6 +232,7 @@ EXPORT_SYMBOL_GPL(dax_recovery_write); int dax_holder_notify_failure(struct dax_device *dax_dev, u64 off, u64 len, int mf_flags) { + const struct dax_holder_operations *ops; int rc, id; id = dax_read_lock(); @@ -240,12 +241,19 @@ int dax_holder_notify_failure(struct dax_device *dax_dev, u64 off, goto out; } - if (!dax_dev->holder_ops) { + /* + * Read holder_ops once: a concurrent fs_put_dax() can clear it without + * synchronizing against readers. Without the single fetch the compiler + * could reload between the NULL check and the call and dereference a + * NULL ops. + */ + ops = READ_ONCE(dax_dev->holder_ops); + if (!ops) { rc = -EOPNOTSUPP; goto out; } - rc = dax_dev->holder_ops->notify_failure(dax_dev, off, len, mf_flags); + rc = ops->notify_failure(dax_dev, off, len, mf_flags); out: dax_read_unlock(id); return rc; diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c index 8ab49924f8b7..0c628e4146e2 100644 --- a/drivers/dma-buf/dma-heap.c +++ b/drivers/dma-buf/dma-heap.c @@ -49,33 +49,6 @@ static dev_t dma_heap_devt; static struct class *dma_heap_class; static DEFINE_XARRAY_ALLOC(dma_heap_minors); -static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len, - u32 fd_flags, - u64 heap_flags) -{ - struct dma_buf *dmabuf; - int fd; - - /* - * Allocations from all heaps have to begin - * and end on page boundaries. - */ - len = PAGE_ALIGN(len); - if (!len) - return -EINVAL; - - dmabuf = heap->ops->allocate(heap, len, fd_flags, heap_flags); - if (IS_ERR(dmabuf)) - return PTR_ERR(dmabuf); - - fd = dma_buf_fd(dmabuf, fd_flags); - if (fd < 0) { - dma_buf_put(dmabuf); - /* just return, as put will call release and that will free */ - } - return fd; -} - static int dma_heap_open(struct inode *inode, struct file *file) { struct dma_heap *heap; @@ -93,30 +66,42 @@ static int dma_heap_open(struct inode *inode, struct file *file) return 0; } -static long dma_heap_ioctl_allocate(struct file *file, void *data) +static struct dma_buf *dma_heap_ioctl_allocate(struct file *file, void *data) { struct dma_heap_allocation_data *heap_allocation = data; struct dma_heap *heap = file->private_data; + struct dma_buf *dmabuf; int fd; + size_t len; if (heap_allocation->fd) - return -EINVAL; + return ERR_PTR(-EINVAL); if (heap_allocation->fd_flags & ~DMA_HEAP_VALID_FD_FLAGS) - return -EINVAL; + return ERR_PTR(-EINVAL); if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS) - return -EINVAL; + return ERR_PTR(-EINVAL); + + len = PAGE_ALIGN(heap_allocation->len); + if (!len) + return ERR_PTR(-EINVAL); + + dmabuf = heap->ops->allocate(heap, len, heap_allocation->fd_flags, + heap_allocation->heap_flags); - fd = dma_heap_buffer_alloc(heap, heap_allocation->len, - heap_allocation->fd_flags, - heap_allocation->heap_flags); - if (fd < 0) - return fd; + if (IS_ERR(dmabuf)) + return dmabuf; + + fd = get_unused_fd_flags(heap_allocation->fd_flags); + if (fd < 0) { + dma_buf_put(dmabuf); + return ERR_PTR(fd); + } heap_allocation->fd = fd; - return 0; + return dmabuf; } static unsigned int dma_heap_ioctl_cmds[] = { @@ -132,6 +117,8 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd, unsigned int in_size, out_size, drv_size, ksize; int nr = _IOC_NR(ucmd); int ret = 0; + int fd; + struct dma_buf *dmabuf; if (nr >= ARRAY_SIZE(dma_heap_ioctl_cmds)) return -EINVAL; @@ -168,15 +155,28 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd, switch (kcmd) { case DMA_HEAP_IOCTL_ALLOC: - ret = dma_heap_ioctl_allocate(file, kdata); + dmabuf = dma_heap_ioctl_allocate(file, kdata); + + if (IS_ERR(dmabuf)) { + ret = PTR_ERR(dmabuf); + break; + } + + fd = ((struct dma_heap_allocation_data *)kdata)->fd; + if (copy_to_user((void __user *)arg, kdata, out_size) != 0) { + put_unused_fd(fd); + dma_buf_put(dmabuf); + ret = -EFAULT; + } else { + dma_buf_fd_install(dmabuf, fd); + } + break; default: ret = -ENOTTY; goto err; } - if (copy_to_user((void __user *)arg, kdata, out_size) != 0) - ret = -EFAULT; err: if (kdata != stack_kdata) kfree(kdata); diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c index 5b35faf567bb..cb34385d605e 100644 --- a/drivers/dma/dw-edma/dw-edma-core.c +++ b/drivers/dma/dw-edma/dw-edma-core.c @@ -7,6 +7,7 @@ */ #include +#include #include #include #include @@ -201,6 +202,35 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan) return 1; } +static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd) +{ + list_del(&vd->node); + dma_cookie_complete(&vd->tx); + vchan_terminate_vdesc(vd); +} + +static void dw_edma_terminate_vdesc_list(struct list_head *head) +{ + struct virt_dma_desc *vd, *_vd; + + list_for_each_entry_safe(vd, _vd, head, node) + dw_edma_terminate_vdesc(vd); +} + +/* Must be called with vc.lock held. */ +static void dw_edma_terminate_all_descs(struct dw_edma_chan *chan) +{ + /* + * This order must not be reversed. Cookies are assigned when + * descriptors are submitted, so desc_issued contains older cookies + * than desc_submitted. Completing desc_submitted first could move + * chan->vc.chan.completed_cookie backwards when desc_issued is + * terminated afterwards. + */ + dw_edma_terminate_vdesc_list(&chan->vc.desc_issued); + dw_edma_terminate_vdesc_list(&chan->vc.desc_submitted); +} + static void dw_edma_device_caps(struct dma_chan *dchan, struct dma_slave_caps *caps) { @@ -235,6 +265,8 @@ static int dw_edma_device_pause(struct dma_chan *dchan) struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan); int err = 0; + guard(spinlock_irqsave)(&chan->vc.lock); + if (!chan->configured) err = -EPERM; else if (chan->status != EDMA_ST_BUSY) @@ -252,6 +284,8 @@ static int dw_edma_device_resume(struct dma_chan *dchan) struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan); int err = 0; + guard(spinlock_irqsave)(&chan->vc.lock); + if (!chan->configured) { err = -EPERM; } else if (chan->status != EDMA_ST_PAUSE) { @@ -272,25 +306,29 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan) struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan); int err = 0; + guard(spinlock_irqsave)(&chan->vc.lock); + if (!chan->configured) { - /* Do nothing */ + dw_edma_terminate_all_descs(chan); } else if (chan->status == EDMA_ST_PAUSE) { + dw_edma_terminate_all_descs(chan); chan->status = EDMA_ST_IDLE; - chan->configured = false; } else if (chan->status == EDMA_ST_IDLE) { - chan->configured = false; + dw_edma_terminate_all_descs(chan); } else if (dw_edma_core_ch_status(chan) == DMA_COMPLETE) { /* * The channel is in a false BUSY state, probably didn't * receive or lost an interrupt */ + dw_edma_terminate_all_descs(chan); chan->status = EDMA_ST_IDLE; - chan->configured = false; } else if (chan->request > EDMA_REQ_PAUSE) { err = -EPERM; } else { chan->request = EDMA_REQ_STOP; } + if (chan->status == EDMA_ST_IDLE) + chan->request = EDMA_REQ_NONE; return err; } @@ -300,11 +338,9 @@ static void dw_edma_device_issue_pending(struct dma_chan *dchan) struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan); unsigned long flags; - if (!chan->configured) - return; - spin_lock_irqsave(&chan->vc.lock, flags); - if (vchan_issue_pending(&chan->vc) && chan->request == EDMA_REQ_NONE && + if (chan->configured && vchan_issue_pending(&chan->vc) && + chan->request == EDMA_REQ_NONE && chan->status == EDMA_ST_IDLE) { chan->status = EDMA_ST_BUSY; dw_edma_start_transfer(chan); @@ -641,8 +677,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan) break; case EDMA_REQ_STOP: - list_del(&vd->node); - vchan_cookie_complete(vd); + dw_edma_terminate_all_descs(chan); chan->request = EDMA_REQ_NONE; chan->status = EDMA_ST_IDLE; break; @@ -661,14 +696,16 @@ static void dw_edma_abort_interrupt(struct dw_edma_chan *chan) spin_lock_irqsave(&chan->vc.lock, flags); vd = vchan_next_desc(&chan->vc); - if (vd) { + if (vd && chan->request == EDMA_REQ_STOP) { + dw_edma_terminate_all_descs(chan); + } else if (vd) { dw_hdma_set_callback_result(vd, DMA_TRANS_ABORTED); list_del(&vd->node); vchan_cookie_complete(vd); } - spin_unlock_irqrestore(&chan->vc.lock, flags); chan->request = EDMA_REQ_NONE; chan->status = EDMA_ST_IDLE; + spin_unlock_irqrestore(&chan->vc.lock, flags); } static inline irqreturn_t dw_edma_interrupt_write(int irq, void *data) @@ -709,21 +746,49 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan) return 0; } -static void dw_edma_free_chan_resources(struct dma_chan *dchan) +static void dw_edma_wait_termination(struct dma_chan *dchan) { + struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan); unsigned long timeout = jiffies + msecs_to_jiffies(5000); - int ret; + bool stopping; + /* + * A STOP may be deferred to a later interrupt while the channel is still + * running. Wait until that handler completes the termination. + */ while (time_before(jiffies, timeout)) { - ret = dw_edma_device_terminate_all(dchan); - if (!ret) - break; + scoped_guard(spinlock_irqsave, &chan->vc.lock) + stopping = chan->request == EDMA_REQ_STOP; - if (time_after_eq(jiffies, timeout)) + if (!stopping) return; - cpu_relax(); + fsleep(1000); } + + dev_warn(chan->dw->chip->dev, + "timeout waiting for channel termination\n"); +} + +static void dw_edma_device_synchronize(struct dma_chan *dchan) +{ + struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan); + + dw_edma_wait_termination(dchan); + vchan_synchronize(&chan->vc); +} + +static void dw_edma_free_chan_resources(struct dma_chan *dchan) +{ + struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan); + + dw_edma_device_terminate_all(dchan); + dw_edma_device_synchronize(dchan); + + scoped_guard(spinlock_irqsave, &chan->vc.lock) + chan->configured = false; + + vchan_free_chan_resources(&chan->vc); } static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc) @@ -820,6 +885,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc) dma->device_pause = dw_edma_device_pause; dma->device_resume = dw_edma_device_resume; dma->device_terminate_all = dw_edma_device_terminate_all; + dma->device_synchronize = dw_edma_device_synchronize; dma->device_issue_pending = dw_edma_device_issue_pending; dma->device_tx_status = dw_edma_device_tx_status; dma->device_prep_slave_sg = dw_edma_device_prep_slave_sg; diff --git a/drivers/dma/hisi_dma.c b/drivers/dma/hisi_dma.c index 25a4134be36b..ede094e0d090 100644 --- a/drivers/dma/hisi_dma.c +++ b/drivers/dma/hisi_dma.c @@ -983,7 +983,7 @@ static int hisi_dma_probe(struct pci_dev *pdev, const struct pci_device_id *id) hdma_dev = devm_kzalloc(dev, struct_size(hdma_dev, chan, chan_num), GFP_KERNEL); if (!hdma_dev) - return -EINVAL; + return -ENOMEM; hdma_dev->base = pcim_iomap_table(pdev)[PCI_BAR_2]; hdma_dev->pdev = pdev; diff --git a/drivers/dma/mediatek/mtk-uart-apdma.c b/drivers/dma/mediatek/mtk-uart-apdma.c index 96c18c815f1d..cdba08163797 100644 --- a/drivers/dma/mediatek/mtk-uart-apdma.c +++ b/drivers/dma/mediatek/mtk-uart-apdma.c @@ -530,7 +530,7 @@ static int mtk_uart_apdma_probe(struct platform_device *pdev) for (i = 0; i < mtkd->dma_requests; i++) { c = devm_kzalloc(mtkd->ddev.dev, sizeof(*c), GFP_KERNEL); if (!c) { - rc = -ENODEV; + rc = -ENOMEM; goto err_no_dma; } diff --git a/drivers/dma/qcom/bam_dma.c b/drivers/dma/qcom/bam_dma.c index 2cf060174795..781cea16196f 100644 --- a/drivers/dma/qcom/bam_dma.c +++ b/drivers/dma/qcom/bam_dma.c @@ -1391,6 +1391,7 @@ static void bam_dma_remove(struct platform_device *pdev) struct bam_device *bdev = platform_get_drvdata(pdev); u32 i; + pm_runtime_dont_use_autosuspend(&pdev->dev); pm_runtime_force_suspend(&pdev->dev); of_dma_controller_free(pdev->dev.of_node); diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c index 7dec5e6babe1..30e3db94ddf0 100644 --- a/drivers/dma/xilinx/xilinx_dma.c +++ b/drivers/dma/xilinx/xilinx_dma.c @@ -1872,8 +1872,10 @@ static irqreturn_t xilinx_mcdma_irq_handler(int irq, void *data) if (status & XILINX_MCDMA_IRQ_IOC_MASK) { spin_lock(&chan->lock); xilinx_dma_complete_descriptor(chan); - chan->idle = true; - chan->start_transfer(chan); + if (list_empty(&chan->active_list)) { + chan->idle = true; + chan->start_transfer(chan); + } spin_unlock(&chan->lock); } @@ -1929,8 +1931,10 @@ static irqreturn_t xilinx_dma_irq_handler(int irq, void *data) XILINX_DMA_DMASR_DLY_CNT_IRQ)) { spin_lock(&chan->lock); xilinx_dma_complete_descriptor(chan); - chan->idle = true; - chan->start_transfer(chan); + if (list_empty(&chan->active_list)) { + chan->idle = true; + chan->start_transfer(chan); + } spin_unlock(&chan->lock); } diff --git a/drivers/dma/xilinx/zynqmp_dma.c b/drivers/dma/xilinx/zynqmp_dma.c index f7e584de4335..23e07d72a522 100644 --- a/drivers/dma/xilinx/zynqmp_dma.c +++ b/drivers/dma/xilinx/zynqmp_dma.c @@ -1173,9 +1173,9 @@ static void zynqmp_dma_remove(struct platform_device *pdev) dma_async_device_unregister(&zdev->common); zynqmp_dma_chan_remove(zdev->chan); - if (pm_runtime_active(zdev->dev)) - zynqmp_dma_runtime_suspend(zdev->dev); pm_runtime_disable(zdev->dev); + if (!pm_runtime_status_suspended(zdev->dev)) + zynqmp_dma_runtime_suspend(zdev->dev); } static const struct of_device_id zynqmp_dma_of_match[] = { diff --git a/drivers/firewire/.kunitconfig b/drivers/firewire/.kunitconfig index 21b7e9eef63d..7406acb00478 100644 --- a/drivers/firewire/.kunitconfig +++ b/drivers/firewire/.kunitconfig @@ -6,3 +6,4 @@ CONFIG_FIREWIRE_KUNIT_DEVICE_ATTRIBUTE_TEST=y CONFIG_FIREWIRE_KUNIT_PACKET_SERDES_TEST=y CONFIG_FIREWIRE_KUNIT_SELF_ID_SEQUENCE_HELPER_TEST=y CONFIG_FIREWIRE_KUNIT_OHCI_SERDES_TEST=y +CONFIG_FIREWIRE_KUNIT_NODE_TREE_TEST=y diff --git a/drivers/firewire/Kconfig b/drivers/firewire/Kconfig index a5f5e250223a..b5abe00accc9 100644 --- a/drivers/firewire/Kconfig +++ b/drivers/firewire/Kconfig @@ -81,6 +81,21 @@ config FIREWIRE_KUNIT_SELF_ID_SEQUENCE_HELPER_TEST For more information on KUnit and unit tests in general, refer to the KUnit documentation in Documentation/dev-tools/kunit/. +config FIREWIRE_KUNIT_NODE_TREE_TEST + tristate "KUnit tests for node tree" if !KUNIT_ALL_TESTS + depends on FIREWIRE && KUNIT + default KUNIT_ALL_TESTS + help + This builds the KUnit tests for node tree. + + KUnit tests run during boot and output the results to the debug + log in TAP format (https://testanything.org/). Only useful for + kernel devs running KUnit test harness and are not for inclusion + into a production build. + + For more information on KUnit and unit tests in general, refer + to the KUnit documentation in Documentation/dev-tools/kunit/. + config FIREWIRE_OHCI tristate "OHCI-1394 controllers" depends on PCI && FIREWIRE diff --git a/drivers/firewire/core-topology.c b/drivers/firewire/core-topology.c index ed3ae8cdb0cd..14eb8e3f84c8 100644 --- a/drivers/firewire/core-topology.c +++ b/drivers/firewire/core-topology.c @@ -88,6 +88,17 @@ static inline struct fw_node *fw_node(struct list_head *l) return list_entry(l, struct fw_node, link); } +typedef void (*fw_node_callback_t)(struct fw_card *card, struct fw_node *node, + struct fw_node *parent); + +static void for_each_fw_node(struct fw_card *card, struct fw_node *root, + fw_node_callback_t callback); + +static void free_fw_node(struct fw_card *card, struct fw_node *node, struct fw_node *parent) +{ + kfree(node); +} + /* * This function builds the tree representation of the topology given * by the self IDs from the latest bus reset. During the construction @@ -119,8 +130,8 @@ static struct fw_node *build_tree(struct fw_card *card, const u32 *sid, int self while (enumerator.quadlet_count > 0) { unsigned int child_port_count = 0; + unsigned int parent_port_count = 0; unsigned int total_port_count = 0; - unsigned int parent_count = 0; unsigned int quadlet_count; const u32 *self_id_sequence; unsigned int port_capacity; @@ -134,7 +145,7 @@ static struct fw_node *build_tree(struct fw_card *card, const u32 *sid, int self if (PTR_ERR(self_id_sequence) != -ENODATA) { fw_err(card, "inconsistent extended self IDs: %ld\n", PTR_ERR(self_id_sequence)); - return NULL; + goto error; } break; } @@ -148,26 +159,38 @@ static struct fw_node *build_tree(struct fw_card *card, const u32 *sid, int self switch (port_status) { case PHY_PACKET_SELF_ID_PORT_STATUS_CHILD: ++child_port_count; - fallthrough; + break; case PHY_PACKET_SELF_ID_PORT_STATUS_PARENT: + ++parent_port_count; + break; case PHY_PACKET_SELF_ID_PORT_STATUS_NCONN: ++total_port_count; - fallthrough; + break; case PHY_PACKET_SELF_ID_PORT_STATUS_NONE: default: break; } } + total_port_count += child_port_count + parent_port_count; + + // Check that the node reports exactly one parent port, except for the root, which + // of course should have no parents. + if ((enumerator.quadlet_count == 0 && parent_port_count != 0) || + (enumerator.quadlet_count > 0 && parent_port_count != 1)) { + fw_err(card, "parent port inconsistency for node %d: parent_count=%d\n", + phy_id, parent_port_count); + goto error; + } if (phy_id != phy_packet_self_id_get_phy_id(self_id_sequence[0])) { fw_err(card, "PHY ID mismatch in self ID: %d != %d\n", phy_id, phy_packet_self_id_get_phy_id(self_id_sequence[0])); - return NULL; + goto error; } if (child_port_count > stack_depth) { fw_err(card, "topology stack underflow\n"); - return NULL; + goto error; } /* @@ -185,7 +208,7 @@ static struct fw_node *build_tree(struct fw_card *card, const u32 *sid, int self node = fw_node_create(self_id_sequence[0], total_port_count, card->color); if (node == NULL) { fw_err(card, "out of memory while building topology\n"); - return NULL; + goto error; } if (phy_id == (card->node_id & 0x3f)) @@ -203,7 +226,6 @@ static struct fw_node *build_tree(struct fw_card *card, const u32 *sid, int self // we temporarily abuse node->color for remembering the entry in // the node->ports array where the parent node should be. Later, // when we handle the parent node, we fix up the reference. - ++parent_count; node->color = port_index; break; @@ -221,21 +243,12 @@ static struct fw_node *build_tree(struct fw_card *card, const u32 *sid, int self } } - // Check that the node reports exactly one parent port, except for the root, which - // of course should have no parents. - if ((enumerator.quadlet_count == 0 && parent_count != 0) || - (enumerator.quadlet_count > 0 && parent_count != 1)) { - fw_err(card, "parent port inconsistency for node %d: " - "parent_count=%d\n", phy_id, parent_count); - return NULL; - } - /* Pop the child nodes off the stack and push the new node. */ __list_del(h->prev, &stack); list_add_tail(&node->link, &stack); stack_depth += 1 - child_port_count; - if (node->phy_speed == SCODE_BETA && parent_count + child_port_count > 1) + if (node->phy_speed == SCODE_BETA && parent_port_count + child_port_count > 1) beta_repeaters_present = true; // If PHYs report different gap counts, set an invalid count which will force a gap @@ -254,12 +267,13 @@ static struct fw_node *build_tree(struct fw_card *card, const u32 *sid, int self card->beta_repeaters_present = beta_repeaters_present; return local_node; +error: + ++card->color; + list_for_each_entry_safe(node, child, &stack, link) + for_each_fw_node(card, node, free_fw_node); + return NULL; } -typedef void (*fw_node_callback_t)(struct fw_card * card, - struct fw_node * node, - struct fw_node * parent); - static void for_each_fw_node(struct fw_card *card, struct fw_node *root, fw_node_callback_t callback) { @@ -505,3 +519,7 @@ void fw_core_handle_bus_reset(struct fw_card *card, int node_id, int generation, } } EXPORT_SYMBOL(fw_core_handle_bus_reset); + +#ifdef CONFIG_FIREWIRE_KUNIT_NODE_TREE_TEST +#include "node-tree-test.c" +#endif diff --git a/drivers/firewire/node-tree-test.c b/drivers/firewire/node-tree-test.c new file mode 100644 index 000000000000..5e6bb58537e9 --- /dev/null +++ b/drivers/firewire/node-tree-test.c @@ -0,0 +1,607 @@ +// SPDX-License-Identifier: GPL-2.0-only +// +// node-tree-test.c - An application of Kunit to test node tree. +// +// Copyright (c) 2026 Takashi Sakamoto +// +// This file can not be built independently since it is intentionally included in core-topology.c. + +#include +#include +#include + +struct private_data { + struct fw_card *card; + unsigned int release_count; +}; + +static int node_tree_test_init(struct kunit *test) +{ + struct private_data *data; + + data = kunit_kzalloc(test, sizeof(*data), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, data); + + data->card = kunit_kzalloc(test, sizeof(struct fw_card), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, data->card); + + data->card->device = kunit_device_register(test, "dummy-device"); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, data->card->device); + + test->priv = data; + + return 0; +} + +static void node_tree_test_exit(struct kunit *test) +{ + struct private_data *data = test->priv; + + kunit_device_unregister(test, data->card->device); + kunit_kfree(test, data->card); + kunit_kfree(test, data); +} + +static void release_fw_node(struct fw_card *card, struct fw_node *node, struct fw_node *parent) +{ + struct private_data *data = kunit_get_current_test()->priv; + + fw_node_put(node); + ++data->release_count; +} + +static void node_tree_test_two_nodes(struct kunit *test) +{ + // root + // ++============++ + // || phy 1 || + // || P0 P1 P2 || + // ++===|==|==|==++ + // | + // +-----+ + // | + // ++===|==x==x==++ + // || P0 P1 P2 || + // || phy 0 || + // ++============++ + // + static const u32 self_id_sequence[] = { + 0x80000080, + 0x8100005e, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x01; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + + struct fw_node *parent = node; + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 1); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 2); +} + +static void node_tree_test_two_nodes_1394a(struct kunit *test) +{ + // root + // ++===============++ + // || phy 0 || + // || P0 P1 P2 P3 || + // ++===|==|==|==|==++ + // | + // +--+ + // | + // ++===|==|==|==|==|==++ + // || P0 P1 P2 P3 P4 || + // || phy 1 || + // ++==================++ + // + // NOTE: Just for Self-ID Packets Zero and One. + static const u32 self_id_sequence[] = { + 0x80000065, 0x80814000, + 0x8100005d, 0x81810000, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x01; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 4); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + KUNIT_EXPECT_NULL(test, node->ports[3]); + + struct fw_node *parent = node; + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 5); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_PTR_EQ(test, node->ports[1], parent); + KUNIT_EXPECT_NULL(test, node->ports[2]); + KUNIT_EXPECT_NULL(test, node->ports[3]); + KUNIT_EXPECT_NULL(test, node->ports[4]); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 2); +} + +static void node_tree_test_three_nodes_case0(struct kunit *test) +{ + // root + // ++============++ + // || phy 2 || + // || P0 P1 P2 || + // ++===|==|==|==++ + // | | + // +--+ +-----------------+ + // | | + // ++===|==|==x==++ ++===|==|==|==++ + // || P0 P1 P2 || || P0 P1 P2 || + // || phy 0 || || phy 1 || + // ++============++ ++============++ + // + static const u32 self_id_sequence[] = { + 0x80000060, + 0x81000058, + 0x820000dc, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x02; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x02); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NOT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + + struct fw_node *parent = node; + node = parent->ports[0]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 2); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_PTR_EQ(test, node->ports[1], parent); + + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_PTR_EQ(test, node->ports[2], parent); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 3); +} + +static void node_tree_test_three_nodes_case1(struct kunit *test) +{ + // root + // ++============++ + // || phy 2 || + // || P0 P1 P2 || + // ++===|==|==x==++ + // | + // | +-----------+ + // | | | + // ++===|==|==|==++ ++===|==x==x==++ + // || P0 P1 P2 || || P0 P1 P2 || + // || phy 1 || || phy 0 || + // ++============++ ++============++ + // + static const u32 self_id_sequence[] = { + 0x80000080, + 0x8100006c, + 0x82000070, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x02; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x02); + KUNIT_EXPECT_EQ(test, node->port_count, 2); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[1]); + + struct fw_node *parent = node; + node = parent->ports[1]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_PTR_EQ(test, node->ports[1], parent); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + + parent = node; + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 1); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 3); +} + +static void node_tree_test_four_nodes_case0(struct kunit *test) +{ + // root + // ++============++ + // || phy 3 || + // || P0 P1 P2 || + // ++===|==|==|==++ + // | + // | +-----------+ +--------------+ + // | | | | | + // ++===|==|==|==++ ++===|==|==x==++ ++===|==x==x==++ + // || P0 P1 P2 || || P0 P1 P2 || || P0 P1 P2 || + // || phy 2 || || phy 1 || || phy 0 || + // ++============++ ++============++ ++============++ + // + static const u32 self_id_sequence[] = { + 0x80000080, + 0x810000b0, + 0x8200006c, + 0x83000074, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x03); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NULL(test, node->ports[2]); + + struct fw_node *parent = node; + node = parent->ports[1]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x02); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_PTR_EQ(test, node->ports[1], parent); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + + parent = node; + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 2); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + KUNIT_EXPECT_NOT_NULL(test, node->ports[1]); + + parent = node; + node = parent->ports[1]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 1); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 4); +} + +static void node_tree_test_four_nodes_case1(struct kunit *test) +{ + // root + // ++============++ + // || phy 3 || + // || P0 P1 P2 || + // ++===|==|==x==++ + // | + // | +--------------------------------+ + // | | +-----------+ | + // ++===|==|==|==++ ++===|==x==x==++ ++===|==|==|==++ + // || P0 P1 P2 || || P0 P1 P2 || || P0 P1 P2 || + // || phy 2 || || phy 1 || || phy 0 || + // ++============++ ++============++ ++============++ + // + static const u32 self_id_sequence[] = { + 0x80000094, + 0x81000080, + 0x820000bc, + 0x830000d0, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x03); + KUNIT_EXPECT_EQ(test, node->port_count, 2); + KUNIT_EXPECT_NOT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NULL(test, node->ports[1]); + + struct fw_node *parent = node; + node = parent->ports[0]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x02); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + KUNIT_EXPECT_NOT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + + parent = node; + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 1); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + + node = parent->ports[1]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NULL(test, node->ports[2]); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 4); +} + +static void node_tree_test_four_nodes_case2(struct kunit *test) +{ + // root + // ++============++ + // || phy 3 || + // || P0 P1 P2 || + // ++===|==|==|==++ + // | | + // | +-----------------------------+ + // | +--------------+ | + // ++===|==|==x==++ ++===|==|==|==++ ++===|==x==x==++ + // || P0 P1 P2 || || P0 P1 P2 || || P0 P1 P2 || + // || phy 1 || || phy 0 || || phy 2 || + // ++============++ ++============++ ++============++ + // + static const u32 self_id_sequence[] = { + 0x80000094, + 0x810000b0, + 0x82000080, + 0x830000dc, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x03); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NOT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + + struct fw_node *parent = node; + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x02); + KUNIT_EXPECT_EQ(test, node->port_count, 1); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + + node = parent->ports[0]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 2); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + KUNIT_EXPECT_NOT_NULL(test, node->ports[1]); + + parent = node; + node = parent->ports[1]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NULL(test, node->ports[2]); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 4); +} + +static void node_tree_test_four_nodes_case3(struct kunit *test) +{ + // root + // ++============++ + // || phy 3 || + // || P0 P1 P2 || + // ++===|==|==|==++ + // | | +--------------------------------+ + // | +--------------------+ | + // | | | + // ++===|==|==x==++ ++===|==|==|==++ ++===|==|==x==++ + // || P0 P1 P2 || || P0 P1 P2 || || P0 P1 P2 || + // || phy 0 || || phy 1 || || phy 2 || + // ++============++ ++============++ ++============++ + // + static const u32 self_id_sequence[] = { + 0x80000090, + 0x81000058, + 0x82000060, + 0x830000fc, + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NOT_NULL(test, card->local_node); + KUNIT_EXPECT_PTR_EQ(test, card->local_node, card->root_node); + + struct fw_node *node = card->root_node; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x03); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NOT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[1]); + KUNIT_EXPECT_NOT_NULL(test, node->ports[2]); + + struct fw_node *parent = node; + node = parent->ports[2]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x02); + KUNIT_EXPECT_EQ(test, node->port_count, 2); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_PTR_EQ(test, node->ports[1], parent); + + node = parent->ports[1]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x01); + KUNIT_EXPECT_EQ(test, node->port_count, 3); + KUNIT_EXPECT_NULL(test, node->ports[0]); + KUNIT_EXPECT_NULL(test, node->ports[1]); + KUNIT_EXPECT_PTR_EQ(test, node->ports[2], parent); + + node = parent->ports[0]; + KUNIT_EXPECT_EQ(test, node->node_id, LOCAL_BUS | 0x00); + KUNIT_EXPECT_EQ(test, node->port_count, 2); + KUNIT_EXPECT_PTR_EQ(test, node->ports[0], parent); + KUNIT_EXPECT_NULL(test, node->ports[1]); + + ++card->color; + for_each_fw_node(card, card->root_node, release_fw_node); + KUNIT_EXPECT_EQ(test, data->release_count, 4); +} + +static void node_tree_test_invalid_extended_self_id_sequence(struct kunit *test) +{ + // Use the same node tree as node_tree_test_four_nodes_case1, except for the invalid + // content of self ID packet for the phy 3. + static const u32 self_id_sequence[] = { + 0x80000094, + 0x81000080, + 0x820000bc, + 0x830000d1, // Invalid. + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NULL(test, card->local_node); +} + +static void node_tree_test_invalid_phy_id(struct kunit *test) +{ + // Use the same node tree as node_tree_test_four_nodes_case1, except for the invalid + // phy ID for phy 3. + static const u32 self_id_sequence[] = { + 0x80000094, + 0x81000080, + 0x820000bc, + 0x8f0000d0, // Invalid. + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NULL(test, card->local_node); +} + +static void node_tree_test_invalid_child_port_count(struct kunit *test) +{ + // Use the same node tree as node_tree_test_four_nodes_case1, except for the invalid + // count of child ports for phy 3. + static const u32 self_id_sequence[] = { + 0x80000094, + 0x81000080, + 0x820000bc, + 0x830000fc, // Invalid. + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NULL(test, card->local_node); +} + +static void node_tree_test_invalid_parent_port_count(struct kunit *test) +{ + // Use the same node tree as node_tree_test_four_nodes_case1, except for the invalid + // count of parent ports for phy 3. + static const u32 self_id_sequence[] = { + 0x80000094, + 0x81000080, + 0x820000bc, + 0x830000e8, // Invalid. + }; + struct private_data *data = test->priv; + struct fw_card *card = data->card; + + card->node_id = LOCAL_BUS | 0x03; + + card->local_node = build_tree(card, self_id_sequence, ARRAY_SIZE(self_id_sequence), 123); + KUNIT_EXPECT_NULL(test, card->local_node); +} + +static struct kunit_case node_tree_test_cases[] = { + KUNIT_CASE(node_tree_test_two_nodes), + KUNIT_CASE(node_tree_test_two_nodes_1394a), + KUNIT_CASE(node_tree_test_three_nodes_case0), + KUNIT_CASE(node_tree_test_three_nodes_case1), + KUNIT_CASE(node_tree_test_four_nodes_case0), + KUNIT_CASE(node_tree_test_four_nodes_case1), + KUNIT_CASE(node_tree_test_four_nodes_case2), + KUNIT_CASE(node_tree_test_four_nodes_case3), + KUNIT_CASE(node_tree_test_invalid_extended_self_id_sequence), + KUNIT_CASE(node_tree_test_invalid_phy_id), + KUNIT_CASE(node_tree_test_invalid_child_port_count), + KUNIT_CASE(node_tree_test_invalid_parent_port_count), + {} +}; + +static struct kunit_suite node_tree_test_suite = { + .name = "firewire-node-tree", + .init = node_tree_test_init, + .exit = node_tree_test_exit, + .test_cases = node_tree_test_cases, +}; +kunit_test_suite(node_tree_test_suite); diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c index c7698cfaa4e8..290047b46d51 100644 --- a/drivers/firmware/arm_scmi/bus.c +++ b/drivers/firmware/arm_scmi/bus.c @@ -136,17 +136,6 @@ static int scmi_protocol_device_request(const struct scmi_device_id *id_table) return ret; } -static int scmi_protocol_table_register(const struct scmi_device_id *id_table) -{ - int ret = 0; - const struct scmi_device_id *entry; - - for (entry = id_table; entry->name && ret == 0; entry++) - ret = scmi_protocol_device_request(entry); - - return ret; -} - /** * scmi_protocol_device_unrequest - Helper to unrequest a device * @@ -159,6 +148,7 @@ static int scmi_protocol_table_register(const struct scmi_device_id *id_table) */ static void scmi_protocol_device_unrequest(const struct scmi_device_id *id_table) { + struct scmi_requested_dev *rdev, *victim = NULL; struct list_head *phead; pr_debug("Unrequesting SCMI device (%s) for protocol %x\n", @@ -167,29 +157,48 @@ static void scmi_protocol_device_unrequest(const struct scmi_device_id *id_table mutex_lock(&scmi_requested_devices_mtx); phead = idr_find(&scmi_requested_devices, id_table->protocol_id); if (phead) { - struct scmi_requested_dev *victim, *tmp; - - list_for_each_entry_safe(victim, tmp, phead, node) { - if (!strcmp(victim->id_table->name, id_table->name)) { - list_del(&victim->node); - - mutex_unlock(&scmi_requested_devices_mtx); - blocking_notifier_call_chain(&scmi_requested_devices_nh, - SCMI_BUS_NOTIFY_DEVICE_UNREQUEST, - (void *)victim->id_table); - kfree(victim); - mutex_lock(&scmi_requested_devices_mtx); + list_for_each_entry(rdev, phead, node) { + if (!strcmp(rdev->id_table->name, id_table->name)) { + victim = rdev; + list_del(&rdev->node); break; } } - if (list_empty(phead)) { + if (victim && list_empty(phead)) { idr_remove(&scmi_requested_devices, id_table->protocol_id); kfree(phead); } } mutex_unlock(&scmi_requested_devices_mtx); + + if (victim) { + blocking_notifier_call_chain(&scmi_requested_devices_nh, + SCMI_BUS_NOTIFY_DEVICE_UNREQUEST, + (void *)victim->id_table); + kfree(victim); + } +} + +static int scmi_protocol_table_register(const struct scmi_device_id *id_table) +{ + const struct scmi_device_id *entry; + int ret; + + for (entry = id_table; entry->name; entry++) { + ret = scmi_protocol_device_request(entry); + if (ret) + goto err_unrequest; + } + + return 0; + +err_unrequest: + while (entry != id_table) + scmi_protocol_device_unrequest(--entry); + + return ret; } static void @@ -201,21 +210,33 @@ scmi_protocol_table_unregister(const struct scmi_device_id *id_table) scmi_protocol_device_unrequest(entry); } -static int scmi_dev_match_by_id_table(struct scmi_device *scmi_dev, - const struct scmi_device_id *id_table) +static bool scmi_device_is_transport(const struct scmi_device *scmi_dev) +{ + return !strncmp(scmi_dev->name, SCMI_TRANSPORT_DEVNAME_PREFIX, + strlen(SCMI_TRANSPORT_DEVNAME_PREFIX)); +} + +static int __scmi_dev_match_by_id_table(struct scmi_device *scmi_dev, + const struct scmi_device_id *id_table, + bool skip_transport) { if (!id_table || !id_table->name) return 0; - /* Always skip transport devices from matching */ for (; id_table->protocol_id && id_table->name; id_table++) if (id_table->protocol_id == scmi_dev->protocol_id && - strncmp(scmi_dev->name, "__scmi_transport_device", 23) && + !(skip_transport && scmi_device_is_transport(scmi_dev)) && !strcmp(id_table->name, scmi_dev->name)) return 1; return 0; } +static int scmi_dev_match_by_id_table(struct scmi_device *scmi_dev, + const struct scmi_device_id *id_table) +{ + return __scmi_dev_match_by_id_table(scmi_dev, id_table, true); +} + static int scmi_dev_match_id(struct scmi_device *scmi_dev, const struct scmi_driver *scmi_drv) { @@ -235,7 +256,7 @@ static int scmi_match_by_id_table(struct device *dev, const void *data) struct scmi_device *scmi_dev = to_scmi_dev(dev); const struct scmi_device_id *id_table = data; - return scmi_dev_match_by_id_table(scmi_dev, id_table); + return __scmi_dev_match_by_id_table(scmi_dev, id_table, false); } static struct scmi_device *scmi_child_dev_find(struct device *parent, @@ -377,10 +398,14 @@ int scmi_driver_register(struct scmi_driver *driver, struct module *owner, driver->driver.mod_name = mod_name; retval = driver_register(&driver->driver); - if (!retval) - pr_debug("Registered new scmi driver %s\n", driver->name); + if (retval) { + scmi_protocol_table_unregister(driver->id_table); + return retval; + } - return retval; + pr_debug("Registered new scmi driver %s\n", driver->name); + + return 0; } EXPORT_SYMBOL_GPL(scmi_driver_register); diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h index 7c35c95fddba..ef803e1d25e1 100644 --- a/drivers/firmware/arm_scmi/common.h +++ b/drivers/firmware/arm_scmi/common.h @@ -33,6 +33,8 @@ #define SCMI_SHMEM_MAX_PAYLOAD_SIZE 104 +#define SCMI_TRANSPORT_DEVNAME_PREFIX "__scmi_transport_device" + enum scmi_error_codes { SCMI_SUCCESS = 0, /* Success */ SCMI_ERR_SUPPORT = -1, /* Not supported */ diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c index 5caa9191a8d1..5a9d6df50679 100644 --- a/drivers/firmware/arm_scmi/driver.c +++ b/drivers/firmware/arm_scmi/driver.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -2540,21 +2541,31 @@ static int scmi_handle_put(const struct scmi_handle *handle) return 0; } -static void scmi_device_link_add(struct device *consumer, +static bool scmi_device_link_add(struct device *consumer, struct device *supplier) { struct device_link *link; link = device_link_add(consumer, supplier, DL_FLAG_AUTOREMOVE_CONSUMER); - WARN_ON(!link); + return !WARN_ON(!link); +} + +static void scmi_clear_handle(struct scmi_device *scmi_dev) +{ + if (!scmi_dev->handle) + return; + + scmi_handle_put(scmi_dev->handle); + scmi_dev->handle = NULL; } static void scmi_set_handle(struct scmi_device *scmi_dev) { scmi_dev->handle = scmi_handle_get(&scmi_dev->dev); - if (scmi_dev->handle) - scmi_device_link_add(&scmi_dev->dev, scmi_dev->handle->dev); + if (scmi_dev->handle && + !scmi_device_link_add(&scmi_dev->dev, scmi_dev->handle->dev)) + scmi_clear_handle(scmi_dev); } static int __scmi_xfer_info_init(struct scmi_info *sinfo, @@ -2663,6 +2674,9 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node, idx = tx ? 0 : 1; idr = tx ? &info->tx_idr : &info->rx_idr; + if (idr_find(idr, prot_id)) + return -EEXIST; + if (!info->desc->ops->chan_available(of_node, idx)) { cinfo = idr_find(idr, SCMI_PROTOCOL_BASE); if (unlikely(!cinfo)) /* Possible only if platform has no Rx */ @@ -2679,7 +2693,7 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node, cinfo->max_msg_size = info->desc->max_msg_size; /* Create a unique name for this transport device */ - snprintf(name, 32, "__scmi_transport_device_%s_%02X", + snprintf(name, sizeof(name), SCMI_TRANSPORT_DEVNAME_PREFIX "_%s_%02X", idx ? "rx" : "tx", prot_id); /* Create a uniquely named, dedicated transport device for this chan */ tdev = scmi_device_create(of_node, info->dev, prot_id, name); @@ -2693,6 +2707,7 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node, cinfo->id = prot_id; cinfo->dev = &tdev->dev; + cinfo->handle = &info->handle; ret = info->desc->ops->chan_setup(cinfo, info->dev, tx); if (ret) { of_node_put(of_node); @@ -2718,6 +2733,7 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node, "unable to allocate SCMI idr slot err %d\n", ret); /* Destroy channel and device only if created by this call. */ if (tdev) { + info->desc->ops->chan_free(prot_id, cinfo, idr); of_node_put(of_node); scmi_device_destroy(info->dev, prot_id, name); devm_kfree(info->dev, cinfo); @@ -2725,7 +2741,6 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node, return ret; } - cinfo->handle = &info->handle; return 0; } @@ -2783,9 +2798,11 @@ static int scmi_channels_setup(struct scmi_info *info) if (of_property_read_u32(child, "reg", &prot_id)) continue; - if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id)) + if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id)) { dev_err(info->dev, "Out of range protocol %d\n", prot_id); + continue; + } ret = scmi_txrx_setup(info, child, prot_id); if (ret) @@ -2795,7 +2812,7 @@ static int scmi_channels_setup(struct scmi_info *info) return 0; } -static int scmi_chan_destroy(int id, void *p, void *idr) +static int scmi_chan_destroy(int id, void *p, void *data) { struct scmi_chan_info *cinfo = p; @@ -2804,12 +2821,10 @@ static int scmi_chan_destroy(int id, void *p, void *idr) struct scmi_device *sdev = to_scmi_dev(cinfo->dev); of_node_put(cinfo->dev->of_node); - scmi_device_destroy(info->dev, id, sdev->name); + scmi_device_destroy(info->dev, cinfo->id, sdev->name); cinfo->dev = NULL; } - idr_remove(idr, id); - return 0; } @@ -2836,6 +2851,7 @@ static int scmi_bus_notifier(struct notifier_block *nb, { struct scmi_info *info = bus_nb_to_scmi_info(nb); struct scmi_device *sdev = to_scmi_dev(data); + const char *status; /* Skip devices of different SCMI instances */ if (sdev->dev.parent != info->dev) @@ -2845,18 +2861,22 @@ static int scmi_bus_notifier(struct notifier_block *nb, case BUS_NOTIFY_BIND_DRIVER: /* setup handle now as the transport is ready */ scmi_set_handle(sdev); + status = "about to be BOUND."; + break; + case BUS_NOTIFY_DRIVER_NOT_BOUND: + scmi_clear_handle(sdev); + status = "NOT BOUND."; break; case BUS_NOTIFY_UNBOUND_DRIVER: - scmi_handle_put(sdev->handle); - sdev->handle = NULL; + scmi_clear_handle(sdev); + status = "UNBOUND."; break; default: return NOTIFY_DONE; } dev_dbg(info->dev, "Device %s (%s) is now %s\n", dev_name(&sdev->dev), - sdev->name, action == BUS_NOTIFY_BIND_DRIVER ? - "about to be BOUND." : "UNBOUND."); + sdev->name, status); return NOTIFY_OK; } @@ -2868,7 +2888,9 @@ static int scmi_device_request_notifier(struct notifier_block *nb, struct scmi_device_id *id_table = data; struct scmi_info *info = req_nb_to_scmi_info(nb); + rcu_read_lock(); np = idr_find(&info->active_protocols, id_table->protocol_id); + rcu_read_unlock(); if (!np) return NOTIFY_DONE; @@ -3171,7 +3193,7 @@ static int scmi_probe(struct platform_device *pdev) ret = scmi_channels_setup(info); if (ret) { err_str = "failed to setup channels\n"; - goto clear_ida; + goto clear_txrx_setup; } ret = bus_register_notifier(&scmi_bus_type, &info->bus_nb); @@ -3233,7 +3255,7 @@ static int scmi_probe(struct platform_device *pdev) dev_err(dev, "%s", err_str); return 0; } - goto notification_exit; + goto raw_mode_cleanup; } mutex_lock(&scmi_list_mutex); @@ -3248,8 +3270,10 @@ static int scmi_probe(struct platform_device *pdev) if (of_property_read_u32(child, "reg", &prot_id)) continue; - if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id)) + if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id)) { dev_err(dev, "Out of range protocol %d\n", prot_id); + continue; + } if (!scmi_is_protocol_implemented(handle, prot_id)) { dev_err(dev, "SCMI protocol %d not implemented\n", @@ -3275,18 +3299,18 @@ static int scmi_probe(struct platform_device *pdev) return 0; -notification_exit: +raw_mode_cleanup: if (IS_ENABLED(CONFIG_ARM_SCMI_RAW_MODE_SUPPORT)) scmi_raw_mode_cleanup(info->raw); - scmi_notification_exit(&info->handle); clear_dev_req_notifier: blocking_notifier_chain_unregister(&scmi_requested_devices_nh, &info->dev_req_nb); clear_bus_notifier: bus_unregister_notifier(&scmi_bus_type, &info->bus_nb); clear_txrx_setup: + scmi_notification_quiesce(&info->handle); scmi_cleanup_txrx_channels(info); -clear_ida: + scmi_notification_exit(&info->handle); ida_free(&scmi_id, info->id); out_err: @@ -3309,6 +3333,12 @@ static void scmi_remove(struct platform_device *pdev) list_del(&info->node); mutex_unlock(&scmi_list_mutex); + blocking_notifier_chain_unregister(&scmi_requested_devices_nh, + &info->dev_req_nb); + + /* Stop transport callbacks before tearing down notifications. */ + scmi_notification_quiesce(&info->handle); + scmi_cleanup_txrx_channels(info); scmi_notification_exit(&info->handle); mutex_lock(&info->protocols_mtx); @@ -3319,13 +3349,8 @@ static void scmi_remove(struct platform_device *pdev) of_node_put(child); idr_destroy(&info->active_protocols); - blocking_notifier_chain_unregister(&scmi_requested_devices_nh, - &info->dev_req_nb); bus_unregister_notifier(&scmi_bus_type, &info->bus_nb); - /* Safe to free channels since no more users */ - scmi_cleanup_txrx_channels(info); - ida_free(&scmi_id, info->id); } diff --git a/drivers/firmware/arm_scmi/notify.c b/drivers/firmware/arm_scmi/notify.c index 9bf7f43ab868..672c91197a68 100644 --- a/drivers/firmware/arm_scmi/notify.c +++ b/drivers/firmware/arm_scmi/notify.c @@ -1706,6 +1706,25 @@ int scmi_notification_init(struct scmi_handle *handle) return -ENOMEM; } +/** + * scmi_notification_quiesce() - Stop notification late initialization + * @handle: The handle identifying the platform instance to quiesce + * + * Prevent new late-init work from being queued and wait for any already queued + * or running late-init work to complete before transport channels are torn + * down. + */ +void scmi_notification_quiesce(struct scmi_handle *handle) +{ + struct scmi_notify_instance *ni; + + ni = scmi_notification_instance_data_get(handle); + if (!ni) + return; + + disable_work_sync(&ni->init_work); +} + /** * scmi_notification_exit() - Shutdown and clean Notification core * @handle: The handle identifying the platform instance to shutdown @@ -1717,6 +1736,8 @@ void scmi_notification_exit(struct scmi_handle *handle) ni = scmi_notification_instance_data_get(handle); if (!ni) return; + + scmi_notification_quiesce(handle); scmi_notification_instance_data_set(handle, NULL); /* Destroy while letting pending work complete */ diff --git a/drivers/firmware/arm_scmi/notify.h b/drivers/firmware/arm_scmi/notify.h index 76758a736cf4..f18f98c5ab3b 100644 --- a/drivers/firmware/arm_scmi/notify.h +++ b/drivers/firmware/arm_scmi/notify.h @@ -82,6 +82,7 @@ struct scmi_protocol_events { }; int scmi_notification_init(struct scmi_handle *handle); +void scmi_notification_quiesce(struct scmi_handle *handle); void scmi_notification_exit(struct scmi_handle *handle); int scmi_register_protocol_events(const struct scmi_handle *handle, u8 proto_id, const struct scmi_protocol_handle *ph, diff --git a/drivers/firmware/arm_scmi/transports/mailbox.c b/drivers/firmware/arm_scmi/transports/mailbox.c index ae0f67e6cc45..308736c3ead9 100644 --- a/drivers/firmware/arm_scmi/transports/mailbox.c +++ b/drivers/firmware/arm_scmi/transports/mailbox.c @@ -211,13 +211,18 @@ static int mailbox_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, cl->tx_block = false; cl->knows_txdone = tx; + cinfo->transport_info = smbox; + smbox->cinfo = cinfo; + mutex_init(&smbox->chan_lock); + smbox->chan = mbox_request_channel(cl, tx ? 0 : p2a_chan); if (IS_ERR(smbox->chan)) { ret = PTR_ERR(smbox->chan); + smbox->chan = NULL; if (ret != -EPROBE_DEFER) dev_err(cdev, "failed to request SCMI %s mailbox\n", desc); - return ret; + goto err_clear_cinfo; } /* Additional unidirectional channel for TX if needed */ @@ -225,9 +230,10 @@ static int mailbox_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, smbox->chan_receiver = mbox_request_channel(cl, a2p_rx_chan); if (IS_ERR(smbox->chan_receiver)) { ret = PTR_ERR(smbox->chan_receiver); + smbox->chan_receiver = NULL; if (ret != -EPROBE_DEFER) dev_err(cdev, "failed to request SCMI Tx Receiver mailbox\n"); - return ret; + goto err_free_chan; } } @@ -235,17 +241,23 @@ static int mailbox_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, smbox->chan_platform_receiver = mbox_request_channel(cl, p2a_rx_chan); if (IS_ERR(smbox->chan_platform_receiver)) { ret = PTR_ERR(smbox->chan_platform_receiver); + smbox->chan_platform_receiver = NULL; if (ret != -EPROBE_DEFER) dev_err(cdev, "failed to request SCMI P2A Receiver mailbox\n"); - return ret; + goto err_free_chan; } } - cinfo->transport_info = smbox; - smbox->cinfo = cinfo; - mutex_init(&smbox->chan_lock); - return 0; + +err_free_chan: + mbox_free_channel(smbox->chan); +err_clear_cinfo: + cinfo->transport_info = NULL; + smbox->cinfo = NULL; + devm_iounmap(dev, smbox->shmem); + devm_kfree(dev, smbox); + return ret; } static int mailbox_chan_free(int id, void *p, void *data) diff --git a/drivers/firmware/arm_scmi/transports/smc.c b/drivers/firmware/arm_scmi/transports/smc.c index 21abb571e4f2..1fce3ccdeb7f 100644 --- a/drivers/firmware/arm_scmi/transports/smc.c +++ b/drivers/firmware/arm_scmi/transports/smc.c @@ -172,6 +172,13 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, scmi_info->param_page = SHMEM_PAGE(res.start); scmi_info->param_offset = SHMEM_OFFSET(res.start); } + + scmi_info->func_id = func_id; + scmi_info->cap_id = cap_id; + scmi_info->cinfo = cinfo; + smc_channel_lock_init(scmi_info); + cinfo->transport_info = scmi_info; + /* * If there is an interrupt named "a2p", then the service and * completion of a message is signaled by an interrupt rather than by @@ -183,18 +190,14 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, IRQF_NO_SUSPEND, dev_name(dev), scmi_info); if (ret) { dev_err(dev, "failed to setup SCMI smc irq\n"); + cinfo->transport_info = NULL; + scmi_info->cinfo = NULL; return ret; } } else { cinfo->no_completion_irq = true; } - scmi_info->func_id = func_id; - scmi_info->cap_id = cap_id; - scmi_info->cinfo = cinfo; - smc_channel_lock_init(scmi_info); - cinfo->transport_info = scmi_info; - return 0; } diff --git a/drivers/firmware/google/coreboot_table.c b/drivers/firmware/google/coreboot_table.c index 882db32e51be..2f1122635098 100644 --- a/drivers/firmware/google/coreboot_table.c +++ b/drivers/firmware/google/coreboot_table.c @@ -101,16 +101,20 @@ void coreboot_driver_unregister(struct coreboot_driver *driver) } EXPORT_SYMBOL(coreboot_driver_unregister); -static int coreboot_table_populate(struct device *dev, void *ptr) +static int coreboot_table_populate(struct device *dev, void *ptr, resource_size_t len) { int i, ret; void *ptr_entry; struct coreboot_device *device; struct coreboot_table_entry *entry; struct coreboot_table_header *header = ptr; + void *ptr_end; + ptr_end = ptr + len; ptr_entry = ptr + header->header_bytes; for (i = 0; i < header->table_entries; i++) { + if (ptr_entry + sizeof(*entry) > ptr_end) + return -EINVAL; entry = ptr_entry; if (entry->size < sizeof(*entry)) { @@ -118,6 +122,9 @@ static int coreboot_table_populate(struct device *dev, void *ptr) return -EINVAL; } + if (ptr_entry + entry->size > ptr_end) + return -EINVAL; + device = kzalloc(sizeof(device->dev) + entry->size, GFP_KERNEL); if (!device) return -ENOMEM; @@ -152,6 +159,7 @@ static int coreboot_table_populate(struct device *dev, void *ptr) static int coreboot_table_probe(struct platform_device *pdev) { resource_size_t len; + resource_size_t table_span; struct coreboot_table_header *header; struct resource *res; struct device *dev = &pdev->dev; @@ -163,7 +171,7 @@ static int coreboot_table_probe(struct platform_device *pdev) return -EINVAL; len = resource_size(res); - if (!res->start || !len) + if (!res->start || len < sizeof(*header)) return -EINVAL; /* Check just the header first to make sure things are sane */ @@ -171,19 +179,27 @@ static int coreboot_table_probe(struct platform_device *pdev) if (!header) return -ENOMEM; - len = header->header_bytes + header->table_bytes; ret = strncmp(header->signature, "LBIO", sizeof(header->signature)); + + if (!ret && + (header->header_bytes < sizeof(*header) || + check_add_overflow((resource_size_t)header->header_bytes, + (resource_size_t)header->table_bytes, + &table_span) || + table_span > len)) + ret = -EINVAL; + memunmap(header); if (ret) { dev_warn(dev, "coreboot table missing or corrupt!\n"); return -ENODEV; } - ptr = memremap(res->start, len, MEMREMAP_WB); + ptr = memremap(res->start, table_span, MEMREMAP_WB); if (!ptr) return -ENOMEM; - ret = coreboot_table_populate(dev, ptr); + ret = coreboot_table_populate(dev, ptr, table_span); memunmap(ptr); diff --git a/drivers/firmware/qcom/Makefile b/drivers/firmware/qcom/Makefile index 0be40a1abc13..b679d3fc2c26 100644 --- a/drivers/firmware/qcom/Makefile +++ b/drivers/firmware/qcom/Makefile @@ -5,6 +5,7 @@ obj-$(CONFIG_QCOM_SCM) += qcom-scm.o qcom-scm-objs += qcom_scm.o qcom_scm-smc.o qcom_scm-legacy.o +CFLAGS_qcom_scm-smc.o := -I$(src) obj-$(CONFIG_QCOM_TZMEM) += qcom_tzmem.o obj-$(CONFIG_QCOM_QSEECOM) += qcom_qseecom.o obj-$(CONFIG_QCOM_QSEECOM_UEFISECAPP) += qcom_qseecom_uefisecapp.o diff --git a/drivers/firmware/qcom/qcom_scm-smc.c b/drivers/firmware/qcom/qcom_scm-smc.c index 574930729ddd..127365ab11fc 100644 --- a/drivers/firmware/qcom/qcom_scm-smc.c +++ b/drivers/firmware/qcom/qcom_scm-smc.c @@ -24,6 +24,9 @@ struct arm_smccc_args { unsigned long args[8]; }; +#define CREATE_TRACE_POINTS +#include "qcom_scm_trace.h" + static DEFINE_MUTEX(qcom_scm_lock); #define QCOM_SCM_EBUSY_WAIT_MS 30 @@ -44,6 +47,7 @@ static void __scm_smc_do_quirk(const struct arm_smccc_args *smc, quirk.state.a6 = 0; do { + trace_scm_smc_request(a0, smc); arm_smccc_smc_quirk(a0, smc->args[1], smc->args[2], smc->args[3], smc->args[4], smc->args[5], quirk.state.a6, smc->args[7], res, &quirk); @@ -83,6 +87,7 @@ int scm_get_wq_ctx(u32 *wq_ctx, u32 *flags, u32 *more_pending) if (ret) return ret; + trace_scm_waitq_get_wq_ctx(get_wq_res.a1, get_wq_res.a2, get_wq_res.a3); *wq_ctx = get_wq_res.a1; *flags = get_wq_res.a2; *more_pending = get_wq_res.a3; @@ -105,10 +110,12 @@ static int __scm_smc_do_quirk_handle_waitq(struct device *dev, struct arm_smccc_ wq_ctx = res->a1; smc_call_ctx = res->a2; - ret = qcom_scm_wait_for_wq_completion(wq_ctx); + trace_scm_waitq_sleep(wq_ctx, smc_call_ctx); + ret = qcom_scm_wait_for_wq_completion(dev, wq_ctx); if (ret) return ret; + trace_scm_waitq_resume(smc_call_ctx); fill_wq_resume_args(&resume, smc_call_ctx); smc = &resume; } @@ -201,6 +208,9 @@ int __scm_smc_call(struct device *dev, const struct qcom_scm_desc *desc, } ret = __scm_smc_do(dev, &smc, &smc_res, atomic); + + trace_scm_smc_done(ret, smc.args[0], &smc_res); + if (ret) return ret; diff --git a/drivers/firmware/qcom/qcom_scm.c b/drivers/firmware/qcom/qcom_scm.c index 3379607eaf94..226265b311fe 100644 --- a/drivers/firmware/qcom/qcom_scm.c +++ b/drivers/firmware/qcom/qcom_scm.c @@ -30,18 +30,25 @@ #include #include +#include + #include "qcom_scm.h" #include "qcom_tzmem.h" static u32 download_mode; +#define GIC_SPI_BASE 32 +#define GIC_MAX_SPI 1019 // SPIs in GICv3 spec range from 32..1019 +#define GIC_ESPI_BASE 4096 +#define GIC_MAX_ESPI 5119 // ESPIs in GICv3 spec range from 4096..5119 + struct qcom_scm { struct device *dev; struct clk *core_clk; struct clk *iface_clk; struct clk *bus_clk; struct icc_path *path; - struct completion waitq_comp; + struct completion *waitq_comps; struct reset_controller_dev reset; /* control access to the interconnect path */ @@ -51,6 +58,7 @@ struct qcom_scm { u64 dload_mode_addr; struct qcom_tzmem_pool *mempool; + unsigned int wq_cnt; }; struct qcom_scm_current_perm_info { @@ -130,6 +138,8 @@ static const u8 qcom_scm_cpu_warm_bits[QCOM_SCM_BOOT_MAX_CPUS] = { #define QCOM_DLOAD_MINIDUMP 2 #define QCOM_DLOAD_BOTHDUMP 3 +#define QCOM_SCM_DEFAULT_WAITQ_COUNT 1 + static const char * const qcom_scm_convention_names[] = { [SMC_CONVENTION_UNKNOWN] = "unknown", [SMC_CONVENTION_ARM_32] = "smc arm 32", @@ -558,6 +568,40 @@ static void qcom_scm_set_download_mode(u32 dload_mode) dev_err(__scm->dev, "failed to set download mode: %d\n", ret); } +/** + * devm_qcom_scm_pas_context_alloc() - Allocate peripheral authentication service + * context for a given peripheral + * + * PAS context is device-resource managed, so the caller does not need + * to worry about freeing the context memory. + * + * @dev: PAS firmware device + * @pas_id: peripheral authentication service id + * @mem_phys: Subsystem reserve memory start address + * @mem_size: Subsystem reserve memory size + * + * Returns: The new PAS context, or ERR_PTR() on failure. + */ +struct qcom_scm_pas_context *devm_qcom_scm_pas_context_alloc(struct device *dev, + u32 pas_id, + phys_addr_t mem_phys, + size_t mem_size) +{ + struct qcom_scm_pas_context *ctx; + + ctx = devm_kzalloc(dev, sizeof(*ctx), GFP_KERNEL); + if (!ctx) + return ERR_PTR(-ENOMEM); + + ctx->dev = dev; + ctx->pas_id = pas_id; + ctx->mem_phys = mem_phys; + ctx->mem_size = mem_size; + + return ctx; +} +EXPORT_SYMBOL_GPL(devm_qcom_scm_pas_context_alloc); + /** * qcom_scm_pas_init_image() - Initialize peripheral authentication service * state machine for a given peripheral, using the @@ -2223,42 +2267,104 @@ bool qcom_scm_is_available(void) } EXPORT_SYMBOL_GPL(qcom_scm_is_available); -static int qcom_scm_assert_valid_wq_ctx(u32 wq_ctx) +static int qcom_scm_fill_irq_fwspec_params(struct irq_fwspec *fwspec, u32 hwirq) { - /* FW currently only supports a single wq_ctx (zero). - * TODO: Update this logic to include dynamic allocation and lookup of - * completion structs when FW supports more wq_ctx values. - */ - if (wq_ctx != 0) { - dev_err(__scm->dev, "Firmware unexpectedly passed non-zero wq_ctx\n"); - return -EINVAL; + if (hwirq >= GIC_SPI_BASE && hwirq <= GIC_MAX_SPI) { + fwspec->param[0] = GIC_SPI; + fwspec->param[1] = hwirq - GIC_SPI_BASE; + } else if (hwirq >= GIC_ESPI_BASE && hwirq <= GIC_MAX_ESPI) { + fwspec->param[0] = GIC_ESPI; + fwspec->param[1] = hwirq - GIC_ESPI_BASE; + } else { + WARN(1, "Unexpected hwirq: %d\n", hwirq); + return -ENXIO; } + fwspec->param[2] = IRQ_TYPE_EDGE_RISING; + fwspec->param_count = 3; + return 0; } -int qcom_scm_wait_for_wq_completion(u32 wq_ctx) +static int qcom_scm_query_waitq_count(struct qcom_scm *scm) { + struct qcom_scm_desc desc = { + .svc = QCOM_SCM_SVC_WAITQ, + .cmd = QCOM_SCM_WAITQ_GET_INFO, + .owner = ARM_SMCCC_OWNER_SIP + }; + struct qcom_scm_res res; int ret; - ret = qcom_scm_assert_valid_wq_ctx(wq_ctx); + ret = qcom_scm_call_atomic(scm->dev, &desc, &res); if (ret) return ret; - wait_for_completion(&__scm->waitq_comp); - - return 0; + return res.result[0] & GENMASK(7, 0); } -static int qcom_scm_waitq_wakeup(unsigned int wq_ctx) +static int qcom_scm_get_waitq_irq(struct qcom_scm *scm) { + struct qcom_scm_desc desc = { + .svc = QCOM_SCM_SVC_WAITQ, + .cmd = QCOM_SCM_WAITQ_GET_INFO, + .owner = ARM_SMCCC_OWNER_SIP + }; + struct device_node *parent_irq_node; + struct irq_fwspec fwspec; + struct qcom_scm_res res; + u32 hwirq; int ret; - ret = qcom_scm_assert_valid_wq_ctx(wq_ctx); + ret = qcom_scm_call_atomic(scm->dev, &desc, &res); + if (ret) + return ret; + + hwirq = res.result[1] & GENMASK(15, 0); + ret = qcom_scm_fill_irq_fwspec_params(&fwspec, hwirq); if (ret) return ret; - complete(&__scm->waitq_comp); + parent_irq_node = of_irq_find_parent(scm->dev->of_node); + if (!parent_irq_node) + return -ENODEV; + + fwspec.fwnode = of_fwnode_handle(parent_irq_node); + + return irq_create_fwspec_mapping(&fwspec); +} + +static struct completion *qcom_scm_get_completion(struct qcom_scm *scm, u32 wq_ctx) +{ + if (WARN_ON_ONCE(wq_ctx >= scm->wq_cnt)) + return ERR_PTR(-EINVAL); + + return &scm->waitq_comps[wq_ctx]; +} + +int qcom_scm_wait_for_wq_completion(struct device *dev, u32 wq_ctx) +{ + struct qcom_scm *scm = dev_get_drvdata(dev); + struct completion *wq; + + wq = qcom_scm_get_completion(scm, wq_ctx); + if (IS_ERR(wq)) + return PTR_ERR(wq); + + wait_for_completion(wq); + + return 0; +} + +static int qcom_scm_waitq_wakeup(struct qcom_scm *scm, unsigned int wq_ctx) +{ + struct completion *wq; + + wq = qcom_scm_get_completion(scm, wq_ctx); + if (IS_ERR(wq)) + return PTR_ERR(wq); + + complete(wq); return 0; } @@ -2281,7 +2387,7 @@ static irqreturn_t qcom_scm_irq_handler(int irq, void *data) goto out; } - ret = qcom_scm_waitq_wakeup(wq_ctx); + ret = qcom_scm_waitq_wakeup(scm, wq_ctx); if (ret) goto out; } while (more_pending); @@ -2334,17 +2440,18 @@ static int qcom_scm_probe(struct platform_device *pdev) struct qcom_tzmem_pool_config pool_config; struct qcom_scm *scm; int irq, ret; + int i; scm = devm_kzalloc(&pdev->dev, sizeof(*scm), GFP_KERNEL); if (!scm) return -ENOMEM; scm->dev = &pdev->dev; + platform_set_drvdata(pdev, scm); ret = qcom_scm_find_dload_address(&pdev->dev, &scm->dload_mode_addr); if (ret < 0) return ret; - init_completion(&scm->waitq_comp); mutex_init(&scm->scm_bw_lock); scm->path = devm_of_icc_get(&pdev->dev, NULL); @@ -2382,9 +2489,11 @@ static int qcom_scm_probe(struct platform_device *pdev) "Failed to setup the reserved memory region for TZ mem\n"); ret = qcom_tzmem_enable(scm->dev); - if (ret) - return dev_err_probe(scm->dev, ret, - "Failed to enable the TrustZone memory allocator\n"); + if (ret) { + ret = dev_err_probe(scm->dev, ret, + "Failed to enable the TrustZone memory allocator\n"); + goto err_rmem; + } memset(&pool_config, 0, sizeof(pool_config)); pool_config.initial_size = 0; @@ -2392,11 +2501,26 @@ static int qcom_scm_probe(struct platform_device *pdev) pool_config.max_size = SZ_256K; scm->mempool = devm_qcom_tzmem_pool_new(scm->dev, &pool_config); - if (IS_ERR(scm->mempool)) - return dev_err_probe(scm->dev, PTR_ERR(scm->mempool), - "Failed to create the SCM memory pool\n"); + if (IS_ERR(scm->mempool)) { + ret = dev_err_probe(scm->dev, PTR_ERR(scm->mempool), + "Failed to create the SCM memory pool\n"); + goto err_rmem; + } + + ret = qcom_scm_query_waitq_count(scm); + scm->wq_cnt = ret < 0 ? QCOM_SCM_DEFAULT_WAITQ_COUNT : ret; + scm->waitq_comps = devm_kcalloc(&pdev->dev, scm->wq_cnt, sizeof(*scm->waitq_comps), + GFP_KERNEL); + if (!scm->waitq_comps) + return -ENOMEM; + + for (i = 0; i < scm->wq_cnt; i++) + init_completion(&scm->waitq_comps[i]); + + irq = qcom_scm_get_waitq_irq(scm); + if (irq < 0) + irq = platform_get_irq_optional(pdev, 0); - irq = platform_get_irq_optional(pdev, 0); if (irq < 0) { if (irq != -ENXIO) return irq; @@ -2449,6 +2573,10 @@ static int qcom_scm_probe(struct platform_device *pdev) qcom_scm_qtee_init(scm); return 0; + +err_rmem: + of_reserved_mem_device_release(scm->dev); + return ret; } static void qcom_scm_shutdown(struct platform_device *pdev) diff --git a/drivers/firmware/qcom/qcom_scm.h b/drivers/firmware/qcom/qcom_scm.h index a56c8212cc0c..2111188f9fad 100644 --- a/drivers/firmware/qcom/qcom_scm.h +++ b/drivers/firmware/qcom/qcom_scm.h @@ -66,7 +66,7 @@ struct qcom_scm_res { u64 result[MAX_QCOM_SCM_RETS]; }; -int qcom_scm_wait_for_wq_completion(u32 wq_ctx); +int qcom_scm_wait_for_wq_completion(struct device *dev, u32 wq_ctx); int scm_get_wq_ctx(u32 *wq_ctx, u32 *flags, u32 *more_pending); #define SCM_SMC_FNID(s, c) ((((s) & 0xFF) << 8) | ((c) & 0xFF)) @@ -152,6 +152,7 @@ int qcom_scm_shm_bridge_enable(struct device *scm_dev); #define QCOM_SCM_SVC_WAITQ 0x24 #define QCOM_SCM_WAITQ_RESUME 0x02 #define QCOM_SCM_WAITQ_GET_WQ_CTX 0x03 +#define QCOM_SCM_WAITQ_GET_INFO 0x04 #define QCOM_SCM_SVC_GPU 0x28 #define QCOM_SCM_SVC_GPU_INIT_REGS 0x01 diff --git a/drivers/firmware/qcom/qcom_scm_trace.h b/drivers/firmware/qcom/qcom_scm_trace.h new file mode 100644 index 000000000000..6c911124fc56 --- /dev/null +++ b/drivers/firmware/qcom/qcom_scm_trace.h @@ -0,0 +1,143 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + */ + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM qcom_scm + +#if !defined(_TRACE_SCM_SMC_INTERFACE_H) || defined(TRACE_HEADER_MULTI_READ) + +#define _TRACE_SCM_SMC_INTERFACE_H + +#include + +TRACE_EVENT(scm_smc_request, + + TP_PROTO(unsigned long a0, const struct arm_smccc_args *smc), + + TP_ARGS(a0, smc), + + TP_STRUCT__entry( + __field(u64, smc_id) + __field(u8, svc_id) + __field(u8, cmd_id) + __field(u8, args_cnt) + __dynamic_array(unsigned long, args, + min_t(u8, (smc->args[1] & 0xF), (u8)6)) + ), + + TP_fast_assign( + __entry->smc_id = a0; + __entry->svc_id = (smc->args[0] >> 8) & 0xFF; + __entry->cmd_id = smc->args[0] & 0xFF; + u8 n = min_t(u8, (smc->args[1] & 0xF), (u8)6); + + __entry->args_cnt = n; + + unsigned long *dst = __get_dynamic_array(args); + + for (int i = 0; i < n; i++) + dst[i] = smc->args[2 + i]; + ), + + TP_printk("smc_id:0x%08llx svc_id:0x%02x cmd_id:0x%02x args_cnt:%u args:%s", + __entry->smc_id, __entry->svc_id, __entry->cmd_id, __entry->args_cnt, + __print_dynamic_array(args, sizeof(unsigned long))) +); + +TRACE_EVENT(scm_waitq_sleep, + + TP_PROTO(u32 wq_ctx, u32 smc_ctx), + + TP_ARGS(wq_ctx, smc_ctx), + + TP_STRUCT__entry( + __field(u32, wq_ctx) + __field(u32, smc_call_ctx) + ), + + TP_fast_assign( + __entry->wq_ctx = wq_ctx; + __entry->smc_call_ctx = smc_ctx; + ), + + TP_printk("wq_ctx:%u, smc_call_ctx:%u", __entry->wq_ctx, __entry->smc_call_ctx) +); + +TRACE_EVENT(scm_waitq_resume, + + TP_PROTO(u32 smc_ctx), + + TP_ARGS(smc_ctx), + + TP_STRUCT__entry( + __field(u32, smc_call_ctx) + ), + + TP_fast_assign( + __entry->smc_call_ctx = smc_ctx; + ), + + TP_printk("smc_call_ctx:%u", __entry->smc_call_ctx) +); + +TRACE_EVENT(scm_waitq_get_wq_ctx, + + TP_PROTO(u32 wq_ctx, u32 flags, u32 pending), + + TP_ARGS(wq_ctx, flags, pending), + + TP_STRUCT__entry( + __field(u32, wq_ctx) + __field(u32, flags) + __field(u32, more_pending) + ), + + TP_fast_assign( + __entry->wq_ctx = wq_ctx; + __entry->flags = flags; + __entry->more_pending = pending; + ), + + TP_printk("wq_ctx:%u, flags:%u, more_pending:%u", + __entry->wq_ctx, __entry->flags, __entry->more_pending) +); + +TRACE_EVENT(scm_smc_done, + + TP_PROTO(int ret, u64 smc_id, struct arm_smccc_res *smc_res), + + TP_ARGS(ret, smc_id, smc_res), + + TP_STRUCT__entry( + __field(int, ret) + __field(u64, smc_id) + __field(unsigned long, res) + __field(unsigned long, res0) + __field(unsigned long, res1) + __field(unsigned long, res2) + ), + + TP_fast_assign( + __entry->ret = ret; + __entry->smc_id = smc_id; + __entry->res = smc_res->a0; + __entry->res0 = smc_res->a1; + __entry->res1 = smc_res->a2; + __entry->res2 = smc_res->a3; + ), + + TP_printk("smc_id:0x%08llx, ret:%d res_to_callee:0x%lx res0:0x%lx res1:0x%lx res2:0x%lx", + __entry->smc_id, __entry->ret, __entry->res, + __entry->res0, __entry->res1, __entry->res2) +); + +#endif /* _TRACE_SCM_SMC_INTERFACE_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#define TRACE_INCLUDE_FILE qcom_scm_trace + +#include + diff --git a/drivers/firmware/qcom/qcom_tzmem.c b/drivers/firmware/qcom/qcom_tzmem.c index 9f232e53115e..b867b9d8122a 100644 --- a/drivers/firmware/qcom/qcom_tzmem.c +++ b/drivers/firmware/qcom/qcom_tzmem.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include #include @@ -508,14 +509,18 @@ phys_addr_t qcom_tzmem_to_phys(void *vaddr) } EXPORT_SYMBOL_GPL(qcom_tzmem_to_phys); +static void qcom_tzmem_do_init(int *result) +{ + *result = qcom_tzmem_init(); +} + int qcom_tzmem_enable(struct device *dev) { - if (qcom_tzmem_dev) - return -EBUSY; + static int result; qcom_tzmem_dev = dev; - - return qcom_tzmem_init(); + DO_ONCE(qcom_tzmem_do_init, &result); + return result; } EXPORT_SYMBOL_GPL(qcom_tzmem_enable); diff --git a/drivers/gpio/gpio-mxc.c b/drivers/gpio/gpio-mxc.c index dbdf0f41b6bb..12f11a6c9665 100644 --- a/drivers/gpio/gpio-mxc.c +++ b/drivers/gpio/gpio-mxc.c @@ -675,7 +675,7 @@ static const struct dev_pm_ops mxc_gpio_dev_pm_ops = { RUNTIME_PM_OPS(mxc_gpio_runtime_suspend, mxc_gpio_runtime_resume, NULL) }; -static int mxc_gpio_syscore_suspend(void) +static int mxc_gpio_syscore_suspend(void *data) { struct mxc_gpio_port *port; int ret; @@ -692,7 +692,7 @@ static int mxc_gpio_syscore_suspend(void) return 0; } -static void mxc_gpio_syscore_resume(void) +static void mxc_gpio_syscore_resume(void *data) { struct mxc_gpio_port *port; int ret; @@ -709,11 +709,15 @@ static void mxc_gpio_syscore_resume(void) } } -static struct syscore_ops mxc_gpio_syscore_ops = { +static const struct syscore_ops mxc_gpio_syscore_ops = { .suspend = mxc_gpio_syscore_suspend, .resume = mxc_gpio_syscore_resume, }; +static struct syscore mxc_gpio_syscore = { + .ops = &mxc_gpio_syscore_ops, +}; + static struct platform_driver mxc_gpio_driver = { .driver = { .name = "gpio-mxc", @@ -726,7 +730,7 @@ static struct platform_driver mxc_gpio_driver = { static int __init gpio_mxc_init(void) { - register_syscore_ops(&mxc_gpio_syscore_ops); + register_syscore(&mxc_gpio_syscore); return platform_driver_register(&mxc_gpio_driver); } diff --git a/drivers/gpio/gpio-pxa.c b/drivers/gpio/gpio-pxa.c index fa22f3faa163..664cf1eef494 100644 --- a/drivers/gpio/gpio-pxa.c +++ b/drivers/gpio/gpio-pxa.c @@ -747,7 +747,7 @@ static int __init pxa_gpio_dt_init(void) device_initcall(pxa_gpio_dt_init); #ifdef CONFIG_PM -static int pxa_gpio_suspend(void) +static int pxa_gpio_suspend(void *data) { struct pxa_gpio_chip *pchip = pxa_gpio_chip; struct pxa_gpio_bank *c; @@ -768,7 +768,7 @@ static int pxa_gpio_suspend(void) return 0; } -static void pxa_gpio_resume(void) +static void pxa_gpio_resume(void *data) { struct pxa_gpio_chip *pchip = pxa_gpio_chip; struct pxa_gpio_bank *c; @@ -792,14 +792,18 @@ static void pxa_gpio_resume(void) #define pxa_gpio_resume NULL #endif -static struct syscore_ops pxa_gpio_syscore_ops = { +static const struct syscore_ops pxa_gpio_syscore_ops = { .suspend = pxa_gpio_suspend, .resume = pxa_gpio_resume, }; +static struct syscore pxa_gpio_syscore = { + .ops = &pxa_gpio_syscore_ops, +}; + static int __init pxa_gpio_sysinit(void) { - register_syscore_ops(&pxa_gpio_syscore_ops); + register_syscore(&pxa_gpio_syscore); return 0; } postcore_initcall(pxa_gpio_sysinit); diff --git a/drivers/gpio/gpio-sa1100.c b/drivers/gpio/gpio-sa1100.c index 7f6a62f5d1ee..1938ffa2f4f3 100644 --- a/drivers/gpio/gpio-sa1100.c +++ b/drivers/gpio/gpio-sa1100.c @@ -256,7 +256,7 @@ static void sa1100_gpio_handler(struct irq_desc *desc) } while (mask); } -static int sa1100_gpio_suspend(void) +static int sa1100_gpio_suspend(void *data) { struct sa1100_gpio_chip *sgc = &sa1100_gpio_chip; @@ -275,19 +275,23 @@ static int sa1100_gpio_suspend(void) return 0; } -static void sa1100_gpio_resume(void) +static void sa1100_gpio_resume(void *data) { sa1100_update_edge_regs(&sa1100_gpio_chip); } -static struct syscore_ops sa1100_gpio_syscore_ops = { +static const struct syscore_ops sa1100_gpio_syscore_ops = { .suspend = sa1100_gpio_suspend, .resume = sa1100_gpio_resume, }; +static struct syscore sa1100_gpio_syscore = { + .ops = &sa1100_gpio_syscore_ops, +}; + static int __init sa1100_gpio_init_devicefs(void) { - register_syscore_ops(&sa1100_gpio_syscore_ops); + register_syscore(&sa1100_gpio_syscore); return 0; } diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c index 066cdf6863e1..ef399fce3ef3 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c @@ -1881,11 +1881,13 @@ static int gfx_v6_0_ring_test_ring(struct amdgpu_ring *ring) return r; } -static void gfx_v6_0_ring_emit_vgt_flush(struct amdgpu_ring *ring) +static void gfx_v6_0_ring_emit_event_write(struct amdgpu_ring *ring, + uint32_t event_type, + uint32_t event_index) { amdgpu_ring_write(ring, PACKET3(PACKET3_EVENT_WRITE, 0)); - amdgpu_ring_write(ring, EVENT_TYPE(VGT_FLUSH) | - EVENT_INDEX(0)); + amdgpu_ring_write(ring, EVENT_TYPE(event_type) | + EVENT_INDEX(event_index)); } static void gfx_v6_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, @@ -1924,12 +1926,6 @@ static void gfx_v6_0_ring_emit_ib(struct amdgpu_ring *ring, unsigned vmid = AMDGPU_JOB_GET_VMID(job); u32 header, control = 0; - /* insert SWITCH_BUFFER packet before first IB in the ring frame */ - if (flags & AMDGPU_HAVE_CTX_SWITCH) { - amdgpu_ring_write(ring, PACKET3(PACKET3_SWITCH_BUFFER, 0)); - amdgpu_ring_write(ring, 0); - } - if (ib->flags & AMDGPU_IB_FLAG_CE) header = PACKET3(PACKET3_INDIRECT_BUFFER_CONST, 2); else @@ -2351,7 +2347,7 @@ static int gfx_v6_0_cp_resume(struct amdgpu_device *adev) static void gfx_v6_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) { - int usepfp = (ring->funcs->type == AMDGPU_RING_TYPE_GFX); + int usepfp = 1; uint32_t seq = ring->fence_drv.sync_seq; uint64_t addr = ring->fence_drv.gpu_addr; @@ -2364,20 +2360,12 @@ static void gfx_v6_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) amdgpu_ring_write(ring, seq); amdgpu_ring_write(ring, 0xffffffff); amdgpu_ring_write(ring, 4); /* poll interval */ - - if (usepfp) { - /* synce CE with ME to prevent CE fetch CEIB before context switch done */ - amdgpu_ring_write(ring, PACKET3(PACKET3_SWITCH_BUFFER, 0)); - amdgpu_ring_write(ring, 0); - amdgpu_ring_write(ring, PACKET3(PACKET3_SWITCH_BUFFER, 0)); - amdgpu_ring_write(ring, 0); - } } static void gfx_v6_0_ring_emit_vm_flush(struct amdgpu_ring *ring, unsigned vmid, uint64_t pd_addr) { - int usepfp = (ring->funcs->type == AMDGPU_RING_TYPE_GFX); + int usepfp = 1; amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr); @@ -2395,19 +2383,13 @@ static void gfx_v6_0_ring_emit_vm_flush(struct amdgpu_ring *ring, /* sync PFP to ME, otherwise we might get invalid PFP reads */ amdgpu_ring_write(ring, PACKET3(PACKET3_PFP_SYNC_ME, 0)); amdgpu_ring_write(ring, 0x0); - - /* synce CE with ME to prevent CE fetch CEIB before context switch done */ - amdgpu_ring_write(ring, PACKET3(PACKET3_SWITCH_BUFFER, 0)); - amdgpu_ring_write(ring, 0); - amdgpu_ring_write(ring, PACKET3(PACKET3_SWITCH_BUFFER, 0)); - amdgpu_ring_write(ring, 0); } } static void gfx_v6_0_ring_emit_wreg(struct amdgpu_ring *ring, uint32_t reg, uint32_t val) { - int usepfp = (ring->funcs->type == AMDGPU_RING_TYPE_GFX); + int usepfp = 1; amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, 3)); amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(usepfp) | @@ -2996,12 +2978,30 @@ static uint64_t gfx_v6_0_get_gpu_clock_counter(struct amdgpu_device *adev) return clock; } +static void gfx_v6_0_ring_emit_sb(struct amdgpu_ring *ring) +{ + amdgpu_ring_write(ring, PACKET3(PACKET3_SWITCH_BUFFER, 0)); + amdgpu_ring_write(ring, 0); +} + static void gfx_v6_ring_emit_cntxcntl(struct amdgpu_ring *ring, uint32_t flags) { - if (flags & AMDGPU_HAVE_CTX_SWITCH) - gfx_v6_0_ring_emit_vgt_flush(ring); + u32 dw2 = 0x80000000; /* set load_enable otherwise this package is just NOPs */ + + if (flags & AMDGPU_HAVE_CTX_SWITCH) { + gfx_v6_0_ring_emit_event_write(ring, VS_PARTIAL_FLUSH, 4); + gfx_v6_0_ring_emit_event_write(ring, VGT_FLUSH, 0); + + /* set load_global_config (load_global_uconfig doesn't exist on GFX6) */ + dw2 |= 0x1; + /* set load_cs_sh_regs */ + dw2 |= 0x01000000; + /* set load_per_context_state & load_gfx_sh_regs */ + dw2 |= 0x10002; + } + amdgpu_ring_write(ring, PACKET3(PACKET3_CONTEXT_CONTROL, 1)); - amdgpu_ring_write(ring, 0x80000000); + amdgpu_ring_write(ring, dw2); amdgpu_ring_write(ring, 0); } @@ -3527,11 +3527,12 @@ static const struct amdgpu_ring_funcs gfx_v6_0_ring_funcs_gfx = { .emit_frame_size = 5 + 5 + /* hdp flush / invalidate */ 14 + 14 + 14 + /* gfx_v6_0_ring_emit_fence x3 for user fence, vm fence */ - 7 + 4 + /* gfx_v6_0_ring_emit_pipeline_sync */ - SI_FLUSH_GPU_TLB_NUM_WREG * 5 + 7 + 6 + /* gfx_v6_0_ring_emit_vm_flush */ - 3 + 2 + /* gfx_v6_ring_emit_cntxcntl including vgt flush */ + 7 + /* gfx_v6_0_ring_emit_pipeline_sync */ + SI_FLUSH_GPU_TLB_NUM_WREG * 5 + 7 + 2 + /* gfx_v6_0_ring_emit_vm_flush */ + 3 * 2 + /* gfx_v6_0_ring_emit_sb x3 (from amdgpu_vm_flush, amdgpu_ib_schedule) */ + 3 + 2 + 2 + /* gfx_v6_ring_emit_cntxcntl including VGT flush */ 5, /* SURFACE_SYNC */ - .emit_ib_size = 6, /* gfx_v6_0_ring_emit_ib */ + .emit_ib_size = 4, /* gfx_v6_0_ring_emit_ib */ .emit_ib = gfx_v6_0_ring_emit_ib, .emit_fence = gfx_v6_0_ring_emit_fence, .emit_pipeline_sync = gfx_v6_0_ring_emit_pipeline_sync, @@ -3539,6 +3540,7 @@ static const struct amdgpu_ring_funcs gfx_v6_0_ring_funcs_gfx = { .test_ring = gfx_v6_0_ring_test_ring, .test_ib = gfx_v6_0_ring_test_ib, .insert_nop = amdgpu_ring_insert_nop, + .emit_switch_buffer = gfx_v6_0_ring_emit_sb, .emit_cntxcntl = gfx_v6_ring_emit_cntxcntl, .emit_wreg = gfx_v6_0_ring_emit_wreg, .emit_mem_sync = gfx_v6_0_emit_mem_sync, @@ -3554,10 +3556,11 @@ static const struct amdgpu_ring_funcs gfx_v6_0_ring_funcs_compute = { .emit_frame_size = 5 + 5 + /* hdp flush / invalidate */ 7 + /* gfx_v6_0_ring_emit_pipeline_sync */ - SI_FLUSH_GPU_TLB_NUM_WREG * 5 + 7 + /* gfx_v6_0_ring_emit_vm_flush */ + SI_FLUSH_GPU_TLB_NUM_WREG * 5 + 7 + 2 + /* gfx_v6_0_ring_emit_vm_flush */ 14 + 14 + 14 + /* gfx_v6_0_ring_emit_fence x3 for user fence, vm fence */ + 3 * 2 + /* gfx_v6_0_ring_emit_sb x3 (from amdgpu_vm_flush, amdgpu_ib_schedule) */ 5, /* SURFACE_SYNC */ - .emit_ib_size = 6, /* gfx_v6_0_ring_emit_ib */ + .emit_ib_size = 4, /* gfx_v6_0_ring_emit_ib */ .emit_ib = gfx_v6_0_ring_emit_ib, .emit_fence = gfx_v6_0_ring_emit_fence, .emit_pipeline_sync = gfx_v6_0_ring_emit_pipeline_sync, @@ -3565,6 +3568,7 @@ static const struct amdgpu_ring_funcs gfx_v6_0_ring_funcs_compute = { .test_ring = gfx_v6_0_ring_test_ring, .test_ib = gfx_v6_0_ring_test_ib, .insert_nop = amdgpu_ring_insert_nop, + .emit_switch_buffer = gfx_v6_0_ring_emit_sb, .emit_wreg = gfx_v6_0_ring_emit_wreg, .emit_mem_sync = gfx_v6_0_emit_mem_sync, }; diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c index 6eb2514c7c30..06eb499f1363 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c @@ -3040,17 +3040,26 @@ static int dm_hw_init(struct amdgpu_ip_block *ip_block) struct amdgpu_device *adev = ip_block->adev; int r; + adev->dm.i2c_devres_group = devres_open_group(adev->dev, NULL, GFP_KERNEL); + if (!adev->dm.i2c_devres_group) + return -ENOMEM; + /* Create DAL display manager */ r = amdgpu_dm_init(adev); if (r) - return r; + goto err_release_i2c; amdgpu_dm_hpd_init(adev); r = dm_oem_i2c_hw_init(adev); if (r) drm_info(adev_to_drm(adev), "Failed to add OEM i2c bus\n"); + devres_close_group(adev->dev, adev->dm.i2c_devres_group); return 0; + +err_release_i2c: + devres_release_group(adev->dev, adev->dm.i2c_devres_group); + return r; } /** @@ -3065,6 +3074,9 @@ static int dm_hw_fini(struct amdgpu_ip_block *ip_block) { struct amdgpu_device *adev = ip_block->adev; + if (adev->dm.i2c_devres_group) + devres_release_group(adev->dev, adev->dm.i2c_devres_group); + amdgpu_dm_hpd_fini(adev); amdgpu_dm_irq_fini(adev); @@ -4958,10 +4970,10 @@ static int get_brightness_range(const struct amdgpu_dm_backlight_caps *caps, return 1; } -/* Rescale from [min..max] to [0..AMDGPU_MAX_BL_LEVEL] */ -static inline u32 scale_input_to_fw(int min, int max, u64 input) +/* Rescale userspace [0..max] to the firmware curve's [0..255]. */ +static inline u32 scale_input_to_fw(int max, u64 input) { - return DIV_ROUND_CLOSEST_ULL(input * AMDGPU_MAX_BL_LEVEL, max - min); + return DIV_ROUND_CLOSEST_ULL(input * AMDGPU_MAX_BL_LEVEL, max); } /* Rescale from [0..AMDGPU_MAX_BL_LEVEL] to [min..max] */ @@ -4974,7 +4986,7 @@ static void convert_custom_brightness(const struct amdgpu_dm_backlight_caps *cap unsigned int min, unsigned int max, uint32_t *user_brightness) { - u32 brightness = scale_input_to_fw(min, max, *user_brightness); + u32 brightness = scale_input_to_fw(max, *user_brightness); u8 lower_signal, upper_signal, upper_lum, lower_lum, lum; int left, right; @@ -5228,11 +5240,11 @@ amdgpu_dm_register_backlight_device(struct amdgpu_dm_connector *aconnector) caps = &dm->backlight_caps[aconnector->bl_idx]; if (get_brightness_range(caps, &min, &max)) { if (power_supply_is_system_supplied() > 0) - props.brightness = DIV_ROUND_CLOSEST((max - min) * caps->ac_level, 100); + props.brightness = DIV_ROUND_CLOSEST(max * caps->ac_level, 100); else - props.brightness = DIV_ROUND_CLOSEST((max - min) * caps->dc_level, 100); + props.brightness = DIV_ROUND_CLOSEST(max * caps->dc_level, 100); /* min is zero, so max needs to be adjusted */ - props.max_brightness = max - min; + props.max_brightness = max; drm_dbg(drm, "Backlight caps: min: %d, max: %d, ac %d, dc %d\n", min, max, caps->ac_level, caps->dc_level); } else diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h index cd362d22a277..caf6f7387a20 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h @@ -664,6 +664,13 @@ struct amdgpu_display_manager { */ void *bb_from_dmub; + /** + * @i2c_devres_group: + * + * Devres group for DM i2c adapter lifetime management. + */ + void *i2c_devres_group; + /** * @oem_i2c: * diff --git a/drivers/gpu/drm/amd/display/dc/hubp/dcn10/dcn10_hubp.c b/drivers/gpu/drm/amd/display/dc/hubp/dcn10/dcn10_hubp.c index 9b026600b90e..99625cba3df6 100644 --- a/drivers/gpu/drm/amd/display/dc/hubp/dcn10/dcn10_hubp.c +++ b/drivers/gpu/drm/amd/display/dc/hubp/dcn10/dcn10_hubp.c @@ -769,8 +769,7 @@ bool hubp1_is_flip_pending(struct hubp *hubp) if (flip_pending) return true; - if (hubp && - earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part) + if (earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part) return true; return false; diff --git a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/smu7_hwmgr.c b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/smu7_hwmgr.c index 6529a91a613b..ba953654a020 100644 --- a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/smu7_hwmgr.c +++ b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/smu7_hwmgr.c @@ -2216,12 +2216,24 @@ static int smu7_patch_voltage_dependency_tables_with_lookup_table( if (data->vdd_gfx_control == SMU7_VOLTAGE_CONTROL_BY_SVID2) { for (entry_id = 0; entry_id < sclk_table->count; ++entry_id) { voltage_id = sclk_table->entries[entry_id].vddInd; + if (voltage_id >= table_info->vddgfx_lookup_table->count) { + pr_err("amdgpu: sclk[%u] vddgfx index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddgfx_lookup_table->count); + return -EINVAL; + } sclk_table->entries[entry_id].vddgfx = table_info->vddgfx_lookup_table->entries[voltage_id].us_vdd; } } else { for (entry_id = 0; entry_id < sclk_table->count; ++entry_id) { voltage_id = sclk_table->entries[entry_id].vddInd; + if (voltage_id >= table_info->vddc_lookup_table->count) { + pr_err("amdgpu: sclk[%u] vddc index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddc_lookup_table->count); + return -EINVAL; + } sclk_table->entries[entry_id].vddc = table_info->vddc_lookup_table->entries[voltage_id].us_vdd; } @@ -2229,12 +2241,24 @@ static int smu7_patch_voltage_dependency_tables_with_lookup_table( for (entry_id = 0; entry_id < mclk_table->count; ++entry_id) { voltage_id = mclk_table->entries[entry_id].vddInd; + if (voltage_id >= table_info->vddc_lookup_table->count) { + pr_err("amdgpu: mclk[%u] vddc index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddc_lookup_table->count); + return -EINVAL; + } mclk_table->entries[entry_id].vddc = table_info->vddc_lookup_table->entries[voltage_id].us_vdd; } for (entry_id = 0; entry_id < mm_table->count; ++entry_id) { voltage_id = mm_table->entries[entry_id].vddcInd; + if (voltage_id >= table_info->vddc_lookup_table->count) { + pr_err("amdgpu: mm[%u] vddc index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddc_lookup_table->count); + return -EINVAL; + } mm_table->entries[entry_id].vddc = table_info->vddc_lookup_table->entries[voltage_id].us_vdd; } diff --git a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_hwmgr.c b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_hwmgr.c index 9ace863792d4..b3fed2a478aa 100644 --- a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_hwmgr.c +++ b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_hwmgr.c @@ -685,10 +685,18 @@ static int vega10_patch_voltage_dependency_tables_with_lookup_table( case 3: vdt = table_info->vdd_dep_on_pixclk; break; case 4: vdt = table_info->vdd_dep_on_dispclk; break; case 5: vdt = table_info->vdd_dep_on_phyclk; break; + default: + continue; } for (entry_id = 0; entry_id < vdt->count; entry_id++) { voltage_id = vdt->entries[entry_id].vddInd; + if (voltage_id >= table_info->vddc_lookup_table->count) { + pr_err("amdgpu: clk_dep[%u][%u] vddc index %u out of bounds (%u)\n", + i, entry_id, voltage_id, + table_info->vddc_lookup_table->count); + return -EINVAL; + } vdt->entries[entry_id].vddc = table_info->vddc_lookup_table->entries[voltage_id].us_vdd; } @@ -696,23 +704,48 @@ static int vega10_patch_voltage_dependency_tables_with_lookup_table( for (entry_id = 0; entry_id < mm_table->count; ++entry_id) { voltage_id = mm_table->entries[entry_id].vddcInd; + if (voltage_id >= table_info->vddc_lookup_table->count) { + pr_err("amdgpu: mm[%u] vddc index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddc_lookup_table->count); + return -EINVAL; + } mm_table->entries[entry_id].vddc = table_info->vddc_lookup_table->entries[voltage_id].us_vdd; } for (entry_id = 0; entry_id < mclk_table->count; ++entry_id) { voltage_id = mclk_table->entries[entry_id].vddInd; + if (voltage_id >= table_info->vddc_lookup_table->count) { + pr_err("amdgpu: mclk[%u] vddc index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddc_lookup_table->count); + return -EINVAL; + } mclk_table->entries[entry_id].vddc = table_info->vddc_lookup_table->entries[voltage_id].us_vdd; + voltage_id = mclk_table->entries[entry_id].vddciInd; + if (voltage_id >= table_info->vddci_lookup_table->count) { + pr_err("amdgpu: mclk[%u] vddci index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddci_lookup_table->count); + return -EINVAL; + } mclk_table->entries[entry_id].vddci = table_info->vddci_lookup_table->entries[voltage_id].us_vdd; + voltage_id = mclk_table->entries[entry_id].mvddInd; + if (voltage_id >= table_info->vddmem_lookup_table->count) { + pr_err("amdgpu: mclk[%u] vddmem index %u out of bounds (%u)\n", + entry_id, voltage_id, + table_info->vddmem_lookup_table->count); + return -EINVAL; + } mclk_table->entries[entry_id].mvdd = table_info->vddmem_lookup_table->entries[voltage_id].us_vdd; } - return 0; } diff --git a/drivers/gpu/drm/bridge/cadence/cdns-dsi-core.c b/drivers/gpu/drm/bridge/cadence/cdns-dsi-core.c index de2a32112913..16b7e5614200 100644 --- a/drivers/gpu/drm/bridge/cadence/cdns-dsi-core.c +++ b/drivers/gpu/drm/bridge/cadence/cdns-dsi-core.c @@ -1015,7 +1015,7 @@ cdns_dsi_bridge_atomic_reset(struct drm_bridge *bridge) dsi_state = kzalloc(sizeof(*dsi_state), GFP_KERNEL); if (!dsi_state) - return NULL; + return ERR_PTR(-ENOMEM); memset(dsi_state, 0, sizeof(*dsi_state)); dsi_state->base.bridge = bridge; diff --git a/drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c b/drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c index 7ee19b7cc92f..f81db7da711d 100644 --- a/drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c +++ b/drivers/gpu/drm/bridge/cadence/cdns-mhdp8546-core.c @@ -2088,7 +2088,7 @@ cdns_mhdp_bridge_atomic_reset(struct drm_bridge *bridge) cdns_mhdp_state = kzalloc(sizeof(*cdns_mhdp_state), GFP_KERNEL); if (!cdns_mhdp_state) - return NULL; + return ERR_PTR(-ENOMEM); __drm_atomic_helper_bridge_reset(bridge, &cdns_mhdp_state->base); diff --git a/drivers/gpu/drm/bridge/display-connector.c b/drivers/gpu/drm/bridge/display-connector.c index 9a64fefbc903..ba1d41816f3e 100644 --- a/drivers/gpu/drm/bridge/display-connector.c +++ b/drivers/gpu/drm/bridge/display-connector.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -25,6 +26,8 @@ struct display_connector { struct regulator *supply; struct gpio_desc *ddc_en; + + struct work_struct hpd_work; }; static inline struct display_connector * @@ -94,6 +97,34 @@ display_connector_bridge_detect(struct drm_bridge *bridge, struct drm_connector return display_connector_detect(bridge); } +static void display_connector_hpd_enable(struct drm_bridge *bridge) +{ + struct display_connector *conn = to_display_connector(bridge); + + enable_irq(conn->hpd_irq); + + if (conn->bridge.type == DRM_MODE_CONNECTOR_DisplayPort) + schedule_work(&conn->hpd_work); +} + +static void display_connector_hpd_disable(struct drm_bridge *bridge) +{ + struct display_connector *conn = to_display_connector(bridge); + + if (conn->bridge.type == DRM_MODE_CONNECTOR_DisplayPort) + cancel_work_sync(&conn->hpd_work); + + disable_irq(conn->hpd_irq); +} + +static void display_connector_hpd_work(struct work_struct *work) +{ + struct display_connector *conn = container_of(work, struct display_connector, hpd_work); + struct drm_bridge *bridge = &conn->bridge; + + drm_bridge_hpd_notify(bridge, display_connector_detect(bridge)); +} + static const struct drm_edid *display_connector_edid_read(struct drm_bridge *bridge, struct drm_connector *connector) { @@ -186,6 +217,8 @@ static const struct drm_bridge_funcs display_connector_bridge_funcs = { .attach = display_connector_attach, .destroy = display_connector_destroy, .detect = display_connector_bridge_detect, + .hpd_enable = display_connector_hpd_enable, + .hpd_disable = display_connector_hpd_disable, .edid_read = display_connector_edid_read, .atomic_get_output_bus_fmts = display_connector_get_output_bus_fmts, .atomic_get_input_bus_fmts = display_connector_get_input_bus_fmts, @@ -315,6 +348,7 @@ static int display_connector_probe(struct platform_device *pdev) NULL, display_connector_hpd_irq, IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING | + IRQF_NO_AUTOEN | IRQF_ONESHOT, "HPD", conn); if (ret) { @@ -386,6 +420,8 @@ static int display_connector_probe(struct platform_device *pdev) conn->bridge.ops |= DRM_BRIDGE_OP_DETECT; if (conn->hpd_irq >= 0) conn->bridge.ops |= DRM_BRIDGE_OP_HPD; + if (conn->hpd_irq >= 0 && type == DRM_MODE_CONNECTOR_DisplayPort) + INIT_WORK(&conn->hpd_work, display_connector_hpd_work); dev_dbg(&pdev->dev, "Found %s display connector '%s' %s DDC bus and %s HPD GPIO (ops 0x%x)\n", diff --git a/drivers/gpu/drm/bridge/synopsys/dw-dp.c b/drivers/gpu/drm/bridge/synopsys/dw-dp.c index e82960163018..901bdec4a76c 100644 --- a/drivers/gpu/drm/bridge/synopsys/dw-dp.c +++ b/drivers/gpu/drm/bridge/synopsys/dw-dp.c @@ -352,12 +352,6 @@ enum { DW_DP_YCBCR420_16BIT, }; -enum { - DW_DP_MP_SINGLE_PIXEL, - DW_DP_MP_DUAL_PIXEL, - DW_DP_MP_QUAD_PIXEL, -}; - enum { DW_DP_SDP_VERTICAL_INTERVAL = BIT(0), DW_DP_SDP_HORIZONTAL_INTERVAL = BIT(1), @@ -1984,7 +1978,7 @@ struct dw_dp *dw_dp_bind(struct device *dev, struct drm_encoder *encoder, return ERR_CAST(dp); dp->dev = dev; - dp->pixel_mode = DW_DP_MP_QUAD_PIXEL; + dp->pixel_mode = plat_data->pixel_mode; dp->plat_data.max_link_rate = plat_data->max_link_rate; bridge = &dp->bridge; @@ -2098,6 +2092,12 @@ struct dw_dp *dw_dp_bind(struct device *dev, struct drm_encoder *encoder, } EXPORT_SYMBOL_GPL(dw_dp_bind); +void dw_dp_unbind(struct dw_dp *dp) +{ + drm_dp_aux_unregister(&dp->aux); +} +EXPORT_SYMBOL_GPL(dw_dp_unbind); + MODULE_AUTHOR("Andy Yan "); MODULE_DESCRIPTION("DW DP Core Library"); MODULE_LICENSE("GPL"); diff --git a/drivers/gpu/drm/bridge/tc358767.c b/drivers/gpu/drm/bridge/tc358767.c index 4097fef4b86b..26ba5a6c1b74 100644 --- a/drivers/gpu/drm/bridge/tc358767.c +++ b/drivers/gpu/drm/bridge/tc358767.c @@ -527,7 +527,7 @@ static ssize_t tc_aux_transfer(struct drm_dp_aux *aux, * address-only transfer */ if (size) - size = FIELD_GET(AUX_BYTES, auxstatus); + size = min_t(size_t, size, FIELD_GET(AUX_BYTES, auxstatus)); msg->reply = FIELD_GET(AUX_STATUS, auxstatus); switch (request) { diff --git a/drivers/gpu/drm/drm_dumb_buffers.c b/drivers/gpu/drm/drm_dumb_buffers.c index 70032bba1c97..7b4ba9fab299 100644 --- a/drivers/gpu/drm/drm_dumb_buffers.c +++ b/drivers/gpu/drm/drm_dumb_buffers.c @@ -29,7 +29,6 @@ #include #include "drm_crtc_internal.h" -#include "drm_internal.h" /** * DOC: overview diff --git a/drivers/gpu/drm/lima/lima_device.c b/drivers/gpu/drm/lima/lima_device.c index 0bf7105c8748..7c873e62c16d 100644 --- a/drivers/gpu/drm/lima/lima_device.c +++ b/drivers/gpu/drm/lima/lima_device.c @@ -368,12 +368,6 @@ int lima_device_init(struct lima_device *ldev) if (err) goto err_out0; - ldev->empty_vm = lima_vm_create(ldev); - if (!ldev->empty_vm) { - err = -ENOMEM; - goto err_out1; - } - ldev->va_start = 0; if (ldev->id == lima_gpu_mali450) { ldev->va_end = LIMA_VA_RESERVE_START; @@ -387,6 +381,12 @@ int lima_device_init(struct lima_device *ldev) } else ldev->va_end = LIMA_VA_RESERVE_END; + ldev->empty_vm = lima_vm_create(ldev); + if (!ldev->empty_vm) { + err = -ENOMEM; + goto err_out1; + } + ldev->iomem = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(ldev->iomem)) { dev_err(ldev->dev, "fail to ioremap iomem\n"); diff --git a/drivers/gpu/drm/msm/adreno/a6xx_catalog.c b/drivers/gpu/drm/msm/adreno/a6xx_catalog.c index 31974a4d7e14..b43850966a5f 100644 --- a/drivers/gpu/drm/msm/adreno/a6xx_catalog.c +++ b/drivers/gpu/drm/msm/adreno/a6xx_catalog.c @@ -1167,7 +1167,7 @@ static const struct adreno_reglist a730_hwcg[] = { { REG_A6XX_RBBM_CLOCK_DELAY_SP0, 0x00000080 }, { REG_A6XX_RBBM_CLOCK_CNTL_TP0, 0x22222220 }, { REG_A6XX_RBBM_CLOCK_CNTL2_TP0, 0x22222222 }, - { REG_A6XX_RBBM_CLOCK_CNTL3_TP0, 0x22222222 }, + { REG_A6XX_RBBM_CLOCK_CNTL3_TP0, 0x22220222 }, { REG_A6XX_RBBM_CLOCK_CNTL4_TP0, 0x00222222 }, { REG_A6XX_RBBM_CLOCK_HYST_TP0, 0x77777777 }, { REG_A6XX_RBBM_CLOCK_HYST2_TP0, 0x77777777 }, diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gmu.c b/drivers/gpu/drm/msm/adreno/a6xx_gmu.c index 21a3f9b0ab4c..3f17cfaca8b4 100644 --- a/drivers/gpu/drm/msm/adreno/a6xx_gmu.c +++ b/drivers/gpu/drm/msm/adreno/a6xx_gmu.c @@ -564,7 +564,7 @@ static void a6xx_rpmh_stop(struct a6xx_gmu *gmu) int ret; u32 val; - if (test_and_clear_bit(GMU_STATUS_FW_START, &gmu->status)) + if (!test_and_clear_bit(GMU_STATUS_FW_START, &gmu->status)) return; gmu_write(gmu, REG_A6XX_GMU_RSCC_CONTROL_REQ, 1); @@ -584,22 +584,19 @@ static inline void pdc_write(void __iomem *ptr, u32 offset, u32 value) writel(value, ptr + (offset << 2)); } -static void __iomem *a6xx_gmu_get_mmio(struct platform_device *pdev, - const char *name); - static void a6xx_gmu_rpmh_init(struct a6xx_gmu *gmu) { struct a6xx_gpu *a6xx_gpu = container_of(gmu, struct a6xx_gpu, gmu); struct adreno_gpu *adreno_gpu = &a6xx_gpu->base; struct platform_device *pdev = to_platform_device(gmu->dev); - void __iomem *pdcptr = a6xx_gmu_get_mmio(pdev, "gmu_pdc"); + void __iomem *pdcptr = devm_platform_ioremap_resource_byname(pdev, "gmu_pdc"); u32 seqmem0_drv0_reg = REG_A6XX_RSCC_SEQ_MEM_0_DRV0; void __iomem *seqptr = NULL; uint32_t pdc_address_offset; bool pdc_in_aop = false; if (IS_ERR(pdcptr)) - goto err; + return; if (adreno_is_a650_family(adreno_gpu) || adreno_is_a7xx(adreno_gpu)) @@ -612,9 +609,9 @@ static void a6xx_gmu_rpmh_init(struct a6xx_gmu *gmu) pdc_address_offset = 0x30080; if (!pdc_in_aop) { - seqptr = a6xx_gmu_get_mmio(pdev, "gmu_pdc_seq"); + seqptr = devm_platform_ioremap_resource_byname(pdev, "gmu_pdc_seq"); if (IS_ERR(seqptr)) - goto err; + return; } /* Disable SDE clock gating */ @@ -704,12 +701,6 @@ static void a6xx_gmu_rpmh_init(struct a6xx_gmu *gmu) /* ensure no writes happen before the uCode is fully written */ wmb(); - -err: - if (!IS_ERR_OR_NULL(pdcptr)) - iounmap(pdcptr); - if (!IS_ERR_OR_NULL(seqptr)) - iounmap(seqptr); } /* @@ -1256,6 +1247,9 @@ static void a6xx_gmu_shutdown(struct a6xx_gmu *gmu) /* Stop the interrupts and mask the hardware */ a6xx_gmu_irq_disable(gmu); + /* Halt the gmu cm3 core */ + gmu_write(gmu, REG_A6XX_GMU_CM3_SYSRESET, 1); + /* Tell RPMh to power off the GPU */ a6xx_rpmh_stop(gmu); @@ -1799,27 +1793,6 @@ static int a6xx_gmu_clocks_probe(struct a6xx_gmu *gmu) return 0; } -static void __iomem *a6xx_gmu_get_mmio(struct platform_device *pdev, - const char *name) -{ - void __iomem *ret; - struct resource *res = platform_get_resource_byname(pdev, - IORESOURCE_MEM, name); - - if (!res) { - DRM_DEV_ERROR(&pdev->dev, "Unable to find the %s registers\n", name); - return ERR_PTR(-EINVAL); - } - - ret = ioremap(res->start, resource_size(res)); - if (!ret) { - DRM_DEV_ERROR(&pdev->dev, "Unable to map the %s registers\n", name); - return ERR_PTR(-EINVAL); - } - - return ret; -} - static int a6xx_gmu_get_irq(struct a6xx_gmu *gmu, struct platform_device *pdev, const char *name, irq_handler_t handler) { @@ -1870,7 +1843,6 @@ void a6xx_gmu_remove(struct a6xx_gpu *a6xx_gpu) { struct adreno_gpu *adreno_gpu = &a6xx_gpu->base; struct a6xx_gmu *gmu = &a6xx_gpu->gmu; - struct platform_device *pdev = to_platform_device(gmu->dev); mutex_lock(&gmu->lock); if (!gmu->initialized) { @@ -1899,8 +1871,6 @@ void a6xx_gmu_remove(struct a6xx_gpu *a6xx_gpu) qmp_put(gmu->qmp); iounmap(gmu->mmio); - if (platform_get_resource_byname(pdev, IORESOURCE_MEM, "rscc")) - iounmap(gmu->rscc); gmu->mmio = NULL; gmu->rscc = NULL; @@ -1926,10 +1896,38 @@ static int cxpd_notifier_cb(struct notifier_block *nb, return 0; } +static void __iomem *a6xx_gmu_get_mmio(struct platform_device *pdev, + const char *name, resource_size_t *start) +{ + void __iomem *ret; + struct resource *res = platform_get_resource_byname(pdev, + IORESOURCE_MEM, name); + + if (!res) { + DRM_DEV_ERROR(&pdev->dev, "Unable to find the %s registers\n", name); + return ERR_PTR(-EINVAL); + } + + ret = ioremap(res->start, resource_size(res)); + if (!ret) { + DRM_DEV_ERROR(&pdev->dev, "Unable to map the %s registers\n", name); + return ERR_PTR(-EINVAL); + } + + if (start) + *start = res->start; + + return ret; +} + int a6xx_gmu_wrapper_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node) { struct platform_device *pdev = of_find_device_by_node(node); + struct adreno_gpu *adreno_gpu = &a6xx_gpu->base; + struct msm_gpu *gpu = &adreno_gpu->base; struct a6xx_gmu *gmu = &a6xx_gpu->gmu; + resource_size_t start; + struct resource *res; int ret; if (!pdev) @@ -1947,12 +1945,21 @@ int a6xx_gmu_wrapper_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node) gmu->legacy = true; /* Map the GMU registers */ - gmu->mmio = a6xx_gmu_get_mmio(pdev, "gmu"); + gmu->mmio = a6xx_gmu_get_mmio(pdev, "gmu", &start); if (IS_ERR(gmu->mmio)) { ret = PTR_ERR(gmu->mmio); goto err_mmio; } + res = platform_get_resource_byname(gpu->pdev, IORESOURCE_MEM, "kgsl_3d0_reg_memory"); + if (!res) { + ret = -EINVAL; + goto err_mmio; + } + + /* Identify gmu base offset from gpu base address */ + gmu->mmio_offset = (u32)(start - res->start); + gmu->cxpd = dev_pm_domain_attach_by_name(gmu->dev, "cx"); if (IS_ERR(gmu->cxpd)) { ret = PTR_ERR(gmu->cxpd); @@ -1993,10 +2000,13 @@ int a6xx_gmu_wrapper_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node) int a6xx_gmu_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node) { + struct platform_device *pdev = of_find_device_by_node(node); struct adreno_gpu *adreno_gpu = &a6xx_gpu->base; + struct msm_gpu *gpu = &adreno_gpu->base; struct a6xx_gmu *gmu = &a6xx_gpu->gmu; - struct platform_device *pdev = of_find_device_by_node(node); struct device_link *link; + resource_size_t start; + struct resource *res; int ret; if (!pdev) @@ -2091,15 +2101,24 @@ int a6xx_gmu_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node) goto err_memory; /* Map the GMU registers */ - gmu->mmio = a6xx_gmu_get_mmio(pdev, "gmu"); + gmu->mmio = a6xx_gmu_get_mmio(pdev, "gmu", &start); if (IS_ERR(gmu->mmio)) { ret = PTR_ERR(gmu->mmio); goto err_memory; } + res = platform_get_resource_byname(gpu->pdev, IORESOURCE_MEM, "kgsl_3d0_reg_memory"); + if (!res) { + ret = -EINVAL; + goto err_mmio; + } + + /* Identify gmu base offset from gpu base address */ + gmu->mmio_offset = (u32)(start - res->start); + if (adreno_is_a650_family(adreno_gpu) || adreno_is_a7xx(adreno_gpu)) { - gmu->rscc = a6xx_gmu_get_mmio(pdev, "rscc"); + gmu->rscc = devm_platform_ioremap_resource_byname(pdev, "rscc"); if (IS_ERR(gmu->rscc)) { ret = -ENODEV; goto err_mmio; @@ -2177,8 +2196,6 @@ int a6xx_gmu_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node) err_mmio: iounmap(gmu->mmio); - if (platform_get_resource_byname(pdev, IORESOURCE_MEM, "rscc")) - iounmap(gmu->rscc); free_irq(gmu->gmu_irq, gmu); free_irq(gmu->hfi_irq, gmu); diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gmu.h b/drivers/gpu/drm/msm/adreno/a6xx_gmu.h index 06cfc294016f..55b1c78daa8b 100644 --- a/drivers/gpu/drm/msm/adreno/a6xx_gmu.h +++ b/drivers/gpu/drm/msm/adreno/a6xx_gmu.h @@ -68,6 +68,7 @@ struct a6xx_gmu { struct drm_gpuvm *vm; void __iomem *mmio; + u32 mmio_offset; void __iomem *rscc; int hfi_irq; @@ -130,20 +131,23 @@ struct a6xx_gmu { unsigned long status; }; +#define GMU_BYTE_OFFSET(gmu, offset) (((offset) << 2) - (gmu)->mmio_offset) + static inline u32 gmu_read(struct a6xx_gmu *gmu, u32 offset) { - return readl(gmu->mmio + (offset << 2)); + /* The 'offset' is based on GPU's start address. Adjust it */ + return readl(gmu->mmio + GMU_BYTE_OFFSET(gmu, offset)); } static inline void gmu_write(struct a6xx_gmu *gmu, u32 offset, u32 value) { - writel(value, gmu->mmio + (offset << 2)); + writel(value, gmu->mmio + GMU_BYTE_OFFSET(gmu, offset)); } static inline void gmu_write_bulk(struct a6xx_gmu *gmu, u32 offset, const u32 *data, u32 size) { - memcpy_toio(gmu->mmio + (offset << 2), data, size); + memcpy_toio(gmu->mmio + GMU_BYTE_OFFSET(gmu, offset), data, size); wmb(); } @@ -160,17 +164,17 @@ static inline u64 gmu_read64(struct a6xx_gmu *gmu, u32 lo, u32 hi) { u64 val; - val = (u64) readl(gmu->mmio + (lo << 2)); - val |= ((u64) readl(gmu->mmio + (hi << 2)) << 32); + val = gmu_read(gmu, lo); + val |= ((u64) gmu_read(gmu, hi) << 32); return val; } #define gmu_poll_timeout(gmu, addr, val, cond, interval, timeout) \ - readl_poll_timeout((gmu)->mmio + ((addr) << 2), val, cond, \ - interval, timeout) + readl_poll_timeout((gmu)->mmio + (GMU_BYTE_OFFSET(gmu, addr)), val, \ + cond, interval, timeout) #define gmu_poll_timeout_atomic(gmu, addr, val, cond, interval, timeout) \ - readl_poll_timeout_atomic((gmu)->mmio + ((addr) << 2), val, cond, \ + readl_poll_timeout_atomic((gmu)->mmio + (GMU_BYTE_OFFSET(gmu, addr)), val, cond, \ interval, timeout) static inline u32 gmu_read_rscc(struct a6xx_gmu *gmu, u32 offset) diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c b/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c index 918d2e504ade..cdd1a27ab810 100644 --- a/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c +++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c @@ -1244,9 +1244,12 @@ static void a6xx_get_gmu_registers(struct msm_gpu *gpu, _a6xx_get_gmu_registers(gpu, a6xx_state, &a6xx_gmu_reglist[1], &a6xx_state->gmu_registers[1], true); - if (adreno_is_a621(adreno_gpu) || adreno_is_a623(adreno_gpu)) + if (adreno_is_a621(adreno_gpu)) _a6xx_get_gmu_registers(gpu, a6xx_state, &a621_gpucc_reg, &a6xx_state->gmu_registers[2], false); + else if (adreno_is_a623(adreno_gpu) || adreno_is_a663(adreno_gpu)) + _a6xx_get_gmu_registers(gpu, a6xx_state, &a623_gpucc_reg, + &a6xx_state->gmu_registers[2], false); else _a6xx_get_gmu_registers(gpu, a6xx_state, &a6xx_gpucc_reg, &a6xx_state->gmu_registers[2], false); diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.h b/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.h index 1c18499b60bb..e6fcae8d4bd3 100644 --- a/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.h +++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu_state.h @@ -343,48 +343,59 @@ static const struct a6xx_registers a6xx_gbif_reglist = static const u32 a6xx_gmu_gx_registers[] = { /* GMU GX */ - 0x0000, 0x0000, 0x0010, 0x0013, 0x0016, 0x0016, 0x0018, 0x001b, - 0x001e, 0x001e, 0x0020, 0x0023, 0x0026, 0x0026, 0x0028, 0x002b, - 0x002e, 0x002e, 0x0030, 0x0033, 0x0036, 0x0036, 0x0038, 0x003b, - 0x003e, 0x003e, 0x0040, 0x0043, 0x0046, 0x0046, 0x0080, 0x0084, - 0x0100, 0x012b, 0x0140, 0x0140, + 0x1a800, 0x1a800, 0x1a810, 0x1a813, 0x1a816, 0x1a816, 0x1a818, 0x1a81b, + 0x1a81e, 0x1a81e, 0x1a820, 0x1a823, 0x1a826, 0x1a826, 0x1a828, 0x1a82b, + 0x1a82e, 0x1a82e, 0x1a830, 0x1a833, 0x1a836, 0x1a836, 0x1a838, 0x1a83b, + 0x1a83e, 0x1a83e, 0x1a840, 0x1a843, 0x1a846, 0x1a846, 0x1a880, 0x1a884, + 0x1a900, 0x1a92b, 0x1a940, 0x1a940, }; static const u32 a6xx_gmu_cx_registers[] = { /* GMU CX */ - 0x4c00, 0x4c07, 0x4c10, 0x4c12, 0x4d00, 0x4d00, 0x4d07, 0x4d0a, - 0x5000, 0x5004, 0x5007, 0x5008, 0x500b, 0x500c, 0x500f, 0x501c, - 0x5024, 0x502a, 0x502d, 0x5030, 0x5040, 0x5053, 0x5087, 0x5089, - 0x50a0, 0x50a2, 0x50a4, 0x50af, 0x50c0, 0x50c3, 0x50d0, 0x50d0, - 0x50e4, 0x50e4, 0x50e8, 0x50ec, 0x5100, 0x5103, 0x5140, 0x5140, - 0x5142, 0x5144, 0x514c, 0x514d, 0x514f, 0x5151, 0x5154, 0x5154, - 0x5157, 0x5158, 0x515d, 0x515d, 0x5162, 0x5162, 0x5164, 0x5165, - 0x5180, 0x5186, 0x5190, 0x519e, 0x51c0, 0x51c0, 0x51c5, 0x51cc, - 0x51e0, 0x51e2, 0x51f0, 0x51f0, 0x5200, 0x5201, + 0x1f400, 0x1f407, 0x1f410, 0x1f412, 0x1f500, 0x1f500, 0x1f507, 0x1f50a, + 0x1f800, 0x1f804, 0x1f807, 0x1f808, 0x1f80b, 0x1f80c, 0x1f80f, 0x1f81c, + 0x1f824, 0x1f82a, 0x1f82d, 0x1f830, 0x1f840, 0x1f853, 0x1f887, 0x1f889, + 0x1f8a0, 0x1f8a2, 0x1f8a4, 0x1f8af, 0x1f8c0, 0x1f8c3, 0x1f8d0, 0x1f8d0, + 0x1f8e4, 0x1f8e4, 0x1f8e8, 0x1f8ec, 0x1f900, 0x1f903, 0x1f940, 0x1f940, + 0x1f942, 0x1f944, 0x1f94c, 0x1f94d, 0x1f94f, 0x1f951, 0x1f954, 0x1f954, + 0x1f957, 0x1f958, 0x1f95d, 0x1f95d, 0x1f962, 0x1f962, 0x1f964, 0x1f965, + 0x1f980, 0x1f986, 0x1f990, 0x1f99e, 0x1f9c0, 0x1f9c0, 0x1f9c5, 0x1f9cc, + 0x1f9e0, 0x1f9e2, 0x1f9f0, 0x1f9f0, 0x1fa00, 0x1fa01, /* GMU AO */ - 0x9300, 0x9316, 0x9400, 0x9400, + 0x23b00, 0x23b16, 0x23c00, 0x23c00, }; static const u32 a6xx_gmu_gpucc_registers[] = { /* GPU CC */ - 0x9800, 0x9812, 0x9840, 0x9852, 0x9c00, 0x9c04, 0x9c07, 0x9c0b, - 0x9c15, 0x9c1c, 0x9c1e, 0x9c2d, 0x9c3c, 0x9c3d, 0x9c3f, 0x9c40, - 0x9c42, 0x9c49, 0x9c58, 0x9c5a, 0x9d40, 0x9d5e, 0xa000, 0xa002, - 0xa400, 0xa402, 0xac00, 0xac02, 0xb000, 0xb002, 0xb400, 0xb402, - 0xb800, 0xb802, + 0x24000, 0x24012, 0x24040, 0x24052, 0x24400, 0x24404, 0x24407, 0x2440b, + 0x24415, 0x2441c, 0x2441e, 0x2442d, 0x2443c, 0x2443d, 0x2443f, 0x24440, + 0x24442, 0x24449, 0x24458, 0x2445a, 0x24540, 0x2455e, 0x24800, 0x24802, + 0x24c00, 0x24c02, 0x25400, 0x25402, 0x25800, 0x25802, 0x25c00, 0x25c02, + 0x26000, 0x26002, /* GPU CC ACD */ - 0xbc00, 0xbc16, 0xbc20, 0xbc27, + 0x26400, 0x26416, 0x26420, 0x26427, }; static const u32 a621_gmu_gpucc_registers[] = { /* GPU CC */ - 0x9800, 0x980e, 0x9c00, 0x9c0e, 0xb000, 0xb004, 0xb400, 0xb404, - 0xb800, 0xb804, 0xbc00, 0xbc05, 0xbc14, 0xbc1d, 0xbc2a, 0xbc30, - 0xbc32, 0xbc32, 0xbc41, 0xbc55, 0xbc66, 0xbc68, 0xbc78, 0xbc7a, - 0xbc89, 0xbc8a, 0xbc9c, 0xbc9e, 0xbca0, 0xbca3, 0xbcb3, 0xbcb5, - 0xbcc5, 0xbcc7, 0xbcd6, 0xbcd8, 0xbce8, 0xbce9, 0xbcf9, 0xbcfc, - 0xbd0b, 0xbd0c, 0xbd1c, 0xbd1e, 0xbd40, 0xbd70, 0xbe00, 0xbe16, - 0xbe20, 0xbe2d, + 0x24000, 0x2400e, 0x24400, 0x2440e, 0x24800, 0x24805, 0x24c00, 0x24cff, + 0x25800, 0x25804, 0x25c00, 0x25c04, 0x26000, 0x26004, 0x26400, 0x26405, + 0x26414, 0x2641d, 0x2642a, 0x26430, 0x26432, 0x26432, 0x26441, 0x26455, + 0x26466, 0x26468, 0x26478, 0x2647a, 0x26489, 0x2648a, 0x2649c, 0x2649e, + 0x264a0, 0x264a3, 0x264b3, 0x264b5, 0x264c5, 0x264c7, 0x264d6, 0x264d8, + 0x264e8, 0x264e9, 0x264f9, 0x264fc, 0x2650b, 0x2650c, 0x2651c, 0x2651e, + 0x26540, 0x26570, 0x26600, 0x26616, 0x26620, 0x2662d, +}; + +static const u32 a623_gmu_gpucc_registers[] = { + /* GPU CC */ + 0x24000, 0x2400e, 0x24400, 0x2440e, 0x25800, 0x25804, 0x25c00, 0x25c04, + 0x26000, 0x26004, 0x26400, 0x26405, 0x26414, 0x2641d, 0x2642a, 0x26430, + 0x26432, 0x26432, 0x26441, 0x26455, 0x26466, 0x26468, 0x26478, 0x2647a, + 0x26489, 0x2648a, 0x2649c, 0x2649e, 0x264a0, 0x264a3, 0x264b3, 0x264b5, + 0x264c5, 0x264c7, 0x264d6, 0x264d8, 0x264e8, 0x264e9, 0x264f9, 0x264fc, + 0x2650b, 0x2650c, 0x2651c, 0x2651e, 0x26540, 0x26570, 0x26600, 0x26616, + 0x26620, 0x2662d, }; static const u32 a6xx_gmu_cx_rscc_registers[] = { @@ -402,6 +413,7 @@ static const struct a6xx_registers a6xx_gmu_reglist[] = { static const struct a6xx_registers a6xx_gpucc_reg = REGS(a6xx_gmu_gpucc_registers, 0, 0); static const struct a6xx_registers a621_gpucc_reg = REGS(a621_gmu_gpucc_registers, 0, 0); +static const struct a6xx_registers a623_gpucc_reg = REGS(a623_gmu_gpucc_registers, 0, 0); static u32 a6xx_get_cp_roq_size(struct msm_gpu *gpu); static u32 a7xx_get_cp_roq_size(struct msm_gpu *gpu); diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_core_perf.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_core_perf.c index 13cc658065c5..6524531bd8bd 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_core_perf.c +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_core_perf.c @@ -394,6 +394,10 @@ int dpu_core_perf_crtc_update(struct drm_crtc *crtc, trace_dpu_core_perf_update_clk(kms->dev, !crtc->enabled, clk_rate); + /* If we're going offline, PM callbacks will disable the clocks instead */ + if (!clk_rate) + return 0; + clk_rate = min(clk_rate, kms->perf.max_core_clk_rate); ret = dev_pm_opp_set_rate(&kms->pdev->dev, clk_rate); if (ret) { diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c index 777eab5ad844..e21aa5cdbaf2 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c @@ -709,8 +709,7 @@ void dpu_encoder_update_topology(struct drm_encoder *drm_enc, if (fb && MSM_FORMAT_IS_YUV(msm_framebuffer_format(fb))) topology->num_cdm++; } else if (disp_info->intf_type == INTF_DP) { - if (msm_dp_is_yuv_420_enabled(priv->kms->dp[disp_info->h_tile_instance[0]], - adj_mode)) + if (drm_mode_is_420_only(&connector->display_info, adj_mode)) topology->num_cdm++; } } diff --git a/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c b/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c index 809ca191e9de..c5cd19d1a486 100644 --- a/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c +++ b/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c @@ -405,7 +405,7 @@ static int mdp4_kms_init(struct drm_device *dev) ret = mdp_kms_init(&mdp4_kms->base, &kms_funcs); if (ret) { DRM_DEV_ERROR(dev->dev, "failed to init kms\n"); - goto fail; + return ret; } kms = priv->kms; @@ -416,7 +416,7 @@ static int mdp4_kms_init(struct drm_device *dev) ret = regulator_enable(mdp4_kms->vdd); if (ret) { DRM_DEV_ERROR(dev->dev, "failed to enable regulator vdd: %d\n", ret); - goto fail; + return ret; } } @@ -428,7 +428,7 @@ static int mdp4_kms_init(struct drm_device *dev) DRM_DEV_ERROR(dev->dev, "unexpected MDP version: v%d.%d\n", major, minor); ret = -ENXIO; - goto fail; + return ret; } mdp4_kms->rev = minor; @@ -437,7 +437,7 @@ static int mdp4_kms_init(struct drm_device *dev) if (!mdp4_kms->lut_clk) { DRM_DEV_ERROR(dev->dev, "failed to get lut_clk\n"); ret = -ENODEV; - goto fail; + return ret; } clk_set_rate(mdp4_kms->lut_clk, max_clk); } @@ -459,7 +459,7 @@ static int mdp4_kms_init(struct drm_device *dev) vm = msm_kms_init_vm(mdp4_kms->dev, NULL); if (IS_ERR(vm)) { ret = PTR_ERR(vm); - goto fail; + return ret; } kms->vm = vm; @@ -467,7 +467,7 @@ static int mdp4_kms_init(struct drm_device *dev) ret = modeset_init(mdp4_kms); if (ret) { DRM_DEV_ERROR(dev->dev, "modeset_init failed: %d\n", ret); - goto fail; + return ret; } mdp4_kms->blank_cursor_bo = msm_gem_new(dev, SZ_16K, MSM_BO_WC | MSM_BO_SCANOUT); @@ -475,14 +475,14 @@ static int mdp4_kms_init(struct drm_device *dev) ret = PTR_ERR(mdp4_kms->blank_cursor_bo); DRM_DEV_ERROR(dev->dev, "could not allocate blank-cursor bo: %d\n", ret); mdp4_kms->blank_cursor_bo = NULL; - goto fail; + return ret; } ret = msm_gem_get_and_pin_iova(mdp4_kms->blank_cursor_bo, kms->vm, &mdp4_kms->blank_cursor_iova); if (ret) { DRM_DEV_ERROR(dev->dev, "could not pin blank-cursor bo: %d\n", ret); - goto fail; + return ret; } dev->mode_config.min_width = 0; @@ -491,12 +491,6 @@ static int mdp4_kms_init(struct drm_device *dev) dev->mode_config.max_height = 2048; return 0; - -fail: - if (kms) - mdp4_destroy(kms); - - return ret; } static const struct dev_pm_ops mdp4_pm_ops = { diff --git a/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c b/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c index 61edf6864092..0b6b97be3ce9 100644 --- a/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c +++ b/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c @@ -514,7 +514,7 @@ static int mdp5_kms_init(struct drm_device *dev) ret = mdp_kms_init(&mdp5_kms->base, &kms_funcs); if (ret) { DRM_DEV_ERROR(&pdev->dev, "failed to init kms\n"); - goto fail; + return ret; } config = mdp5_cfg_get_config(mdp5_kms->cfg); @@ -537,7 +537,7 @@ static int mdp5_kms_init(struct drm_device *dev) vm = msm_kms_init_vm(mdp5_kms->dev, pdev->dev.parent); if (IS_ERR(vm)) { ret = PTR_ERR(vm); - goto fail; + return ret; } kms->vm = vm; @@ -547,7 +547,7 @@ static int mdp5_kms_init(struct drm_device *dev) ret = modeset_init(mdp5_kms); if (ret) { DRM_DEV_ERROR(&pdev->dev, "modeset_init failed: %d\n", ret); - goto fail; + return ret; } dev->mode_config.min_width = 0; @@ -559,11 +559,6 @@ static int mdp5_kms_init(struct drm_device *dev) dev->vblank_disable_immediate = true; return 0; -fail: - if (kms) - mdp5_kms_destroy(kms); - - return ret; } static void mdp5_destroy(struct mdp5_kms *mdp5_kms) diff --git a/drivers/gpu/drm/msm/dp/dp_ctrl.c b/drivers/gpu/drm/msm/dp/dp_ctrl.c index 38ed4de8313e..2ff7db77a82b 100644 --- a/drivers/gpu/drm/msm/dp/dp_ctrl.c +++ b/drivers/gpu/drm/msm/dp/dp_ctrl.c @@ -1941,13 +1941,12 @@ static int msm_dp_ctrl_reinitialize_mainlink(struct msm_dp_ctrl_private *ctrl) msm_dp_ctrl_mainlink_disable(ctrl); ctrl->phy_opts.dp.lanes = ctrl->link->link_params.num_lanes; phy_configure(phy, &ctrl->phy_opts); + /* * Disable and re-enable the mainlink clock since the * link clock might have been adjusted as part of the * link maintenance. */ - dev_pm_opp_set_rate(ctrl->dev, 0); - msm_dp_ctrl_link_clk_disable(&ctrl->msm_dp_ctrl); phy_power_off(phy); @@ -1973,7 +1972,6 @@ static int msm_dp_ctrl_deinitialize_mainlink(struct msm_dp_ctrl_private *ctrl) msm_dp_ctrl_reset(&ctrl->msm_dp_ctrl); - dev_pm_opp_set_rate(ctrl->dev, 0); msm_dp_ctrl_link_clk_disable(&ctrl->msm_dp_ctrl); phy_power_off(phy); @@ -2620,7 +2618,6 @@ void msm_dp_ctrl_off(struct msm_dp_ctrl *msm_dp_ctrl) ctrl->stream_clks_on = false; } - dev_pm_opp_set_rate(ctrl->dev, 0); msm_dp_ctrl_link_clk_disable(&ctrl->msm_dp_ctrl); phy_power_off(phy); diff --git a/drivers/gpu/drm/msm/dp/dp_display.c b/drivers/gpu/drm/msm/dp/dp_display.c index c6f5422b60dd..68043dbe72bf 100644 --- a/drivers/gpu/drm/msm/dp/dp_display.c +++ b/drivers/gpu/drm/msm/dp/dp_display.c @@ -930,6 +930,7 @@ enum drm_mode_status msm_dp_bridge_mode_valid(struct drm_bridge *bridge, u32 mode_rate_khz = 0, supported_rate_khz = 0, mode_bpp = 0; struct msm_dp *dp; int mode_pclk_khz = mode->clock; + bool is_yuv_420; dp = to_dp_bridge(bridge)->msm_dp_display; @@ -941,9 +942,16 @@ enum drm_mode_status msm_dp_bridge_mode_valid(struct drm_bridge *bridge, msm_dp_display = container_of(dp, struct msm_dp_display_private, msm_dp_display); link_info = &msm_dp_display->panel->link_info; - if ((drm_mode_is_420_only(&dp->connector->display_info, mode) && - msm_dp_display->panel->vsc_sdp_supported) || - msm_dp_wide_bus_available(dp)) + is_yuv_420 = drm_mode_is_420_only(&dp->connector->display_info, mode); + + /* + * YUV 420 is carried over DP by signalling the colorimetry through a + * VSC SDP, so a 420-only mode cannot be driven without VSC SDP support. + */ + if (is_yuv_420 && !msm_dp_display->panel->vsc_sdp_supported) + return MODE_NO_420; + + if (is_yuv_420 || msm_dp_display->wide_bus_supported) mode_pclk_khz /= 2; if (mode_pclk_khz > DP_MAX_PIXEL_CLK_KHZ) @@ -1506,22 +1514,10 @@ void __exit msm_dp_unregister(void) platform_driver_unregister(&msm_dp_display_driver); } -bool msm_dp_is_yuv_420_enabled(const struct msm_dp *msm_dp_display, - const struct drm_display_mode *mode) -{ - struct msm_dp_display_private *dp; - const struct drm_display_info *info; - - dp = container_of(msm_dp_display, struct msm_dp_display_private, msm_dp_display); - info = &msm_dp_display->connector->display_info; - - return dp->panel->vsc_sdp_supported && drm_mode_is_420_only(info, mode); -} - bool msm_dp_needs_periph_flush(const struct msm_dp *msm_dp_display, const struct drm_display_mode *mode) { - return msm_dp_is_yuv_420_enabled(msm_dp_display, mode); + return drm_mode_is_420_only(&msm_dp_display->connector->display_info, mode); } bool msm_dp_wide_bus_available(const struct msm_dp *msm_dp_display) diff --git a/drivers/gpu/drm/msm/dp/dp_panel.c b/drivers/gpu/drm/msm/dp/dp_panel.c index 15b7f6c7146e..9191f0fc3d81 100644 --- a/drivers/gpu/drm/msm/dp/dp_panel.c +++ b/drivers/gpu/drm/msm/dp/dp_panel.c @@ -254,8 +254,10 @@ int msm_dp_panel_get_modes(struct msm_dp_panel *msm_dp_panel, return -EINVAL; } - if (msm_dp_panel->drm_edid) + if (msm_dp_panel->drm_edid) { + drm_edid_connector_update(connector, msm_dp_panel->drm_edid); return drm_edid_connector_add_modes(connector); + } return 0; } diff --git a/drivers/gpu/drm/msm/dsi/dsi_host.c b/drivers/gpu/drm/msm/dsi/dsi_host.c index 2350934f270a..5110fe05ca84 100644 --- a/drivers/gpu/drm/msm/dsi/dsi_host.c +++ b/drivers/gpu/drm/msm/dsi/dsi_host.c @@ -549,8 +549,6 @@ int dsi_link_clk_enable_v2(struct msm_dsi_host *msm_host) void dsi_link_clk_disable_6g(struct msm_dsi_host *msm_host) { - /* Drop the performance state vote */ - dev_pm_opp_set_rate(&msm_host->pdev->dev, 0); clk_disable_unprepare(msm_host->esc_clk); clk_disable_unprepare(msm_host->pixel_clk); clk_disable_unprepare(msm_host->byte_intf_clk); diff --git a/drivers/gpu/drm/msm/dsi/phy/dsi_phy.h b/drivers/gpu/drm/msm/dsi/phy/dsi_phy.h index 3cbf08231492..e391505fdaf0 100644 --- a/drivers/gpu/drm/msm/dsi/phy/dsi_phy.h +++ b/drivers/gpu/drm/msm/dsi/phy/dsi_phy.h @@ -109,6 +109,7 @@ struct msm_dsi_phy { struct msm_dsi_dphy_timing timing; const struct msm_dsi_phy_cfg *cfg; void *tuning_cfg; + void *pll_data; enum msm_dsi_phy_usecase usecase; bool regulator_ldo_mode; diff --git a/drivers/gpu/drm/msm/dsi/phy/dsi_phy_7nm.c b/drivers/gpu/drm/msm/dsi/phy/dsi_phy_7nm.c index c5e1d2016bcc..32f06edd21a9 100644 --- a/drivers/gpu/drm/msm/dsi/phy/dsi_phy_7nm.c +++ b/drivers/gpu/drm/msm/dsi/phy/dsi_phy_7nm.c @@ -426,8 +426,11 @@ static void dsi_pll_enable_pll_bias(struct dsi_pll_7nm *pll) u32 data; spin_lock_irqsave(&pll->pll_enable_lock, flags); - pll->pll_enable_cnt++; - WARN_ON(pll->pll_enable_cnt == INT_MAX); + if (pll->pll_enable_cnt++) { + spin_unlock_irqrestore(&pll->pll_enable_lock, flags); + WARN_ON(pll->pll_enable_cnt == INT_MAX); + return; + } data = readl(pll->phy->base + REG_DSI_7nm_PHY_CMN_CTRL_0); data |= DSI_7nm_PHY_CMN_CTRL_0_PLL_SHUTDOWNB; @@ -873,6 +876,7 @@ static int dsi_pll_7nm_init(struct msm_dsi_phy *phy) spin_lock_init(&pll_7nm->pll_enable_lock); pll_7nm->phy = phy; + phy->pll_data = pll_7nm; ret = pll_7nm_register(pll_7nm, phy->provided_clocks->hws); if (ret) { @@ -961,8 +965,10 @@ static int dsi_7nm_phy_enable(struct msm_dsi_phy *phy, u32 const delay_us = 5; u32 const timeout_us = 1000; struct msm_dsi_dphy_timing *timing = &phy->timing; + struct dsi_pll_7nm *pll = phy->pll_data; void __iomem *base = phy->base; bool less_than_1500_mhz; + unsigned long flags; u32 vreg_ctrl_0, vreg_ctrl_1, lane_ctrl0; u32 glbl_pemph_ctrl_0; u32 glbl_str_swi_cal_sel_ctrl, glbl_hstx_str_ctrl_0; @@ -1084,10 +1090,13 @@ static int dsi_7nm_phy_enable(struct msm_dsi_phy *phy, glbl_rescode_bot_ctrl = 0x3c; } + spin_lock_irqsave(&pll->pll_enable_lock, flags); + pll->pll_enable_cnt = 1; /* de-assert digital and pll power down */ data = DSI_7nm_PHY_CMN_CTRL_0_DIGTOP_PWRDN_B | DSI_7nm_PHY_CMN_CTRL_0_PLL_SHUTDOWNB; writel(data, base + REG_DSI_7nm_PHY_CMN_CTRL_0); + spin_unlock_irqrestore(&pll->pll_enable_lock, flags); /* Assert PLL core reset */ writel(0x00, base + REG_DSI_7nm_PHY_CMN_PLL_CNTRL); @@ -1200,7 +1209,9 @@ static bool dsi_7nm_set_continuous_clock(struct msm_dsi_phy *phy, bool enable) static void dsi_7nm_phy_disable(struct msm_dsi_phy *phy) { + struct dsi_pll_7nm *pll = phy->pll_data; void __iomem *base = phy->base; + unsigned long flags; u32 data; DBG(""); @@ -1227,8 +1238,11 @@ static void dsi_7nm_phy_disable(struct msm_dsi_phy *phy) writel(data, base + REG_DSI_7nm_PHY_CMN_CTRL_0); writel(0, base + REG_DSI_7nm_PHY_CMN_LANE_CTRL0); + spin_lock_irqsave(&pll->pll_enable_lock, flags); + pll->pll_enable_cnt = 0; /* Turn off all PHY blocks */ writel(0x00, base + REG_DSI_7nm_PHY_CMN_CTRL_0); + spin_unlock_irqrestore(&pll->pll_enable_lock, flags); /* make sure phy is turned off */ wmb(); diff --git a/drivers/gpu/drm/msm/msm_drv.h b/drivers/gpu/drm/msm/msm_drv.h index 6d847d593f1a..d40793845541 100644 --- a/drivers/gpu/drm/msm/msm_drv.h +++ b/drivers/gpu/drm/msm/msm_drv.h @@ -357,8 +357,6 @@ void __exit msm_dp_unregister(void); int msm_dp_modeset_init(struct msm_dp *dp_display, struct drm_device *dev, struct drm_encoder *encoder, bool yuv_supported); void msm_dp_snapshot(struct msm_disp_state *disp_state, struct msm_dp *dp_display); -bool msm_dp_is_yuv_420_enabled(const struct msm_dp *dp_display, - const struct drm_display_mode *mode); bool msm_dp_needs_periph_flush(const struct msm_dp *dp_display, const struct drm_display_mode *mode); bool msm_dp_wide_bus_available(const struct msm_dp *dp_display); @@ -383,12 +381,6 @@ static inline void msm_dp_snapshot(struct msm_disp_state *disp_state, struct msm { } -static inline bool msm_dp_is_yuv_420_enabled(const struct msm_dp *dp_display, - const struct drm_display_mode *mode) -{ - return false; -} - static inline bool msm_dp_needs_periph_flush(const struct msm_dp *dp_display, const struct drm_display_mode *mode) { diff --git a/drivers/gpu/drm/msm/msm_gem_vma.c b/drivers/gpu/drm/msm/msm_gem_vma.c index 9016ef978be5..701fe4a0b73c 100644 --- a/drivers/gpu/drm/msm/msm_gem_vma.c +++ b/drivers/gpu/drm/msm/msm_gem_vma.c @@ -452,6 +452,8 @@ msm_gem_vm_bo_validate(struct drm_gpuvm_bo *vm_bo, struct drm_exec *exec) return ret; } + drm_gpuvm_bo_evict(vm_bo, false); + return 0; } diff --git a/drivers/gpu/drm/msm/msm_gpu.c b/drivers/gpu/drm/msm/msm_gpu.c index a4377ee84c72..569a1ba02e60 100644 --- a/drivers/gpu/drm/msm/msm_gpu.c +++ b/drivers/gpu/drm/msm/msm_gpu.c @@ -505,6 +505,8 @@ static void recover_worker(struct kthread_work *work) */ if (!vm->managed) msm_gem_vm_unusable(submit->vm); + + put_task_struct(task); } get_comm_cmdline(submit, &comm, &cmd); @@ -548,11 +550,11 @@ static void recover_worker(struct kthread_work *work) msm_update_fence(ring->fctx, fence); } + gpu->funcs->recover(gpu); + /* retire completed submits, plus the one that hung: */ retire_submits(gpu); - gpu->funcs->recover(gpu); - /* * Replay all remaining submits starting with highest priority * ring diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index b2f612e5dc79..e09ec7f398d5 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -109,9 +109,9 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, ring->memptrs_iova = memptrs_iova; ret = drm_sched_init(&ring->sched, &args); - if (ret) { + if (ret) goto fail; - } + ring->sched_initialized = true; INIT_LIST_HEAD(&ring->submits); spin_lock_init(&ring->submit_lock); @@ -133,7 +133,8 @@ void msm_ringbuffer_destroy(struct msm_ringbuffer *ring) if (IS_ERR_OR_NULL(ring)) return; - drm_sched_fini(&ring->sched); + if (ring->sched_initialized) + drm_sched_fini(&ring->sched); msm_fence_context_free(ring->fctx); diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.h b/drivers/gpu/drm/msm/msm_ringbuffer.h index d1e49f701c81..bace58a3dffb 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.h +++ b/drivers/gpu/drm/msm/msm_ringbuffer.h @@ -54,6 +54,7 @@ struct msm_ringbuffer { * The job scheduler for this ring. */ struct drm_gpu_scheduler sched; + bool sched_initialized; /* * List of in-flight submits on this ring. Protected by submit_lock. diff --git a/drivers/gpu/drm/msm/registers/adreno/a6xx_gmu.xml b/drivers/gpu/drm/msm/registers/adreno/a6xx_gmu.xml index b15a242d974d..09b8a0b9c0de 100644 --- a/drivers/gpu/drm/msm/registers/adreno/a6xx_gmu.xml +++ b/drivers/gpu/drm/msm/registers/adreno/a6xx_gmu.xml @@ -40,56 +40,56 @@ xsi:schemaLocation="https://gitlab.freedesktop.org/freedreno/ rules-fd.xsd"> - - - - - - - - - - - - - + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + @@ -99,15 +99,15 @@ xsi:schemaLocation="https://gitlab.freedesktop.org/freedreno/ rules-fd.xsd"> - + - + - + @@ -119,71 +119,71 @@ xsi:schemaLocation="https://gitlab.freedesktop.org/freedreno/ rules-fd.xsd"> - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -191,27 +191,27 @@ xsi:schemaLocation="https://gitlab.freedesktop.org/freedreno/ rules-fd.xsd"> - - - - - + + + + + - - - - - - - - - - - - - - + + + + + + + + + + + + + + diff --git a/drivers/gpu/drm/mxsfb/lcdif_kms.c b/drivers/gpu/drm/mxsfb/lcdif_kms.c index 1c3b33be6c40..d9499d8c7128 100644 --- a/drivers/gpu/drm/mxsfb/lcdif_kms.c +++ b/drivers/gpu/drm/mxsfb/lcdif_kms.c @@ -373,14 +373,23 @@ static void lcdif_disable_controller(struct lcdif_drm_private *lcdif) int ret; reg = readl(lcdif->base + LCDC_V8_CTRLDESCL0_5); + /* Disable the layer for DMA. */ reg &= ~CTRLDESCL0_5_EN; + /* + * It is necessary to wait for the full frame to finish streaming + * through the DMA engine before we can safely disable it by removing + * the DISP_PARA_DISP_ON bit. Disabling it in-flight can leave the + * hardware confused and unable to resume streaming for the next frame. + */ + reg |= CTRLDESCL0_5_SHADOW_LOAD_EN; writel(reg, lcdif->base + LCDC_V8_CTRLDESCL0_5); + /* Wait for the frame to finish or timeout after 50 ms. */ ret = readl_poll_timeout(lcdif->base + LCDC_V8_CTRLDESCL0_5, - reg, !(reg & CTRLDESCL0_5_EN), - 0, 36000); /* Wait ~2 frame times max */ + reg, !(reg & CTRLDESCL0_5_SHADOW_LOAD_EN), + 200, 50000); if (ret) - drm_err(lcdif->drm, "Failed to disable controller!\n"); + drm_err(lcdif->drm, "Timed out waiting for final vblank!\n"); reg = readl(lcdif->base + LCDC_V8_DISP_PARA); reg &= ~DISP_PARA_DISP_ON; diff --git a/drivers/gpu/drm/omapdrm/dss/dsi.c b/drivers/gpu/drm/omapdrm/dss/dsi.c index b129e5a8d791..7bfd7eb32eba 100644 --- a/drivers/gpu/drm/omapdrm/dss/dsi.c +++ b/drivers/gpu/drm/omapdrm/dss/dsi.c @@ -455,15 +455,10 @@ static irqreturn_t omap_dsi_irq_handler(int irq, void *arg) timer_delete(&dsi->te_timer); #endif - /* make a copy and unlock, so that isrs can unregister - * themselves */ - memcpy(&dsi->isr_tables_copy, &dsi->isr_tables, - sizeof(dsi->isr_tables)); + dsi_handle_isrs(&dsi->isr_tables, irqstatus, vcstatus, ciostatus); spin_unlock(&dsi->irq_lock); - dsi_handle_isrs(&dsi->isr_tables_copy, irqstatus, vcstatus, ciostatus); - dsi_handle_irq_errors(dsi, irqstatus, vcstatus, ciostatus); dsi_collect_irq_stats(dsi, irqstatus, vcstatus, ciostatus); diff --git a/drivers/gpu/drm/omapdrm/dss/dsi.h b/drivers/gpu/drm/omapdrm/dss/dsi.h index 601707c0ecc4..2b25247ea893 100644 --- a/drivers/gpu/drm/omapdrm/dss/dsi.h +++ b/drivers/gpu/drm/omapdrm/dss/dsi.h @@ -379,8 +379,6 @@ struct dsi_data { spinlock_t irq_lock; struct dsi_isr_tables isr_tables; - /* space for a copy used by the interrupt handler */ - struct dsi_isr_tables isr_tables_copy; int update_vc; #ifdef DSI_PERF_MEASURE diff --git a/drivers/gpu/drm/panel/panel-samsung-s6d16d0.c b/drivers/gpu/drm/panel/panel-samsung-s6d16d0.c index ba1a02000bb9..04e19b5f22f8 100644 --- a/drivers/gpu/drm/panel/panel-samsung-s6d16d0.c +++ b/drivers/gpu/drm/panel/panel-samsung-s6d16d0.c @@ -89,16 +89,22 @@ static int s6d16d0_prepare(struct drm_panel *panel) MIPI_DSI_DCS_TEAR_MODE_VBLANK); if (ret) { dev_err(s6->dev, "failed to enable vblank TE (%d)\n", ret); - return ret; + goto err_power_off; } /* Exit sleep mode and power on */ ret = mipi_dsi_dcs_exit_sleep_mode(dsi); if (ret) { dev_err(s6->dev, "failed to exit sleep mode (%d)\n", ret); - return ret; + goto err_power_off; } return 0; + +err_power_off: + gpiod_set_value_cansleep(s6->reset_gpio, 1); + regulator_disable(s6->supply); + + return ret; } static int s6d16d0_enable(struct drm_panel *panel) diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c index 4c202fc5ce05..687fa282b093 100644 --- a/drivers/gpu/drm/panthor/panthor_drv.c +++ b/drivers/gpu/drm/panthor/panthor_drv.c @@ -1630,7 +1630,7 @@ static int panthor_probe(struct platform_device *pdev) ptdev = devm_drm_dev_alloc(&pdev->dev, &panthor_drm_driver, struct panthor_device, base); if (IS_ERR(ptdev)) - return -ENOMEM; + return PTR_ERR(ptdev); platform_set_drvdata(pdev, ptdev); diff --git a/drivers/gpu/drm/rockchip/analogix_dp-rockchip.c b/drivers/gpu/drm/rockchip/analogix_dp-rockchip.c index b905e28afa07..7100fcb1650a 100644 --- a/drivers/gpu/drm/rockchip/analogix_dp-rockchip.c +++ b/drivers/gpu/drm/rockchip/analogix_dp-rockchip.c @@ -8,6 +8,7 @@ * Jeff Chen */ +#include #include #include #include @@ -223,7 +224,6 @@ static void rockchip_dp_drm_encoder_enable(struct drm_encoder *encoder, struct drm_crtc *crtc; struct drm_crtc_state *old_crtc_state; struct of_endpoint endpoint; - struct device_node *remote_port, *remote_port_parent; char name[32]; u32 port_id; int ret; @@ -247,18 +247,22 @@ static void rockchip_dp_drm_encoder_enable(struct drm_encoder *encoder, if (ret < 0) return; - remote_port_parent = of_graph_get_remote_port_parent(endpoint.local_node); + struct device_node *remote_port_parent __free(device_node) = + of_graph_get_remote_port_parent(endpoint.local_node); if (remote_port_parent) { - if (of_get_child_by_name(remote_port_parent, "ports")) { - remote_port = of_graph_get_remote_port(endpoint.local_node); + struct device_node *ports __free(device_node) = + of_get_child_by_name(remote_port_parent, "ports"); + + if (ports) { + struct device_node *remote_port __free(device_node) = + of_graph_get_remote_port(endpoint.local_node); + of_property_read_u32(remote_port, "reg", &port_id); - of_node_put(remote_port); sprintf(name, "%s vp%d", remote_port_parent->full_name, port_id); } else { sprintf(name, "%s %s", remote_port_parent->full_name, endpoint.id ? "vopl" : "vopb"); } - of_node_put(remote_port_parent); DRM_DEV_DEBUG(dp->dev, "vop %s output to dp\n", (ret) ? "LIT" : "BIG"); } @@ -328,6 +332,7 @@ static int rockchip_dp_of_probe(struct rockchip_dp_device *dp) { struct device *dev = dp->dev; struct device_node *np = dev->of_node; + struct clk *clk; dp->grf = syscon_regmap_lookup_by_phandle(np, "rockchip,grf"); if (IS_ERR(dp->grf)) { @@ -351,6 +356,11 @@ static int rockchip_dp_of_probe(struct rockchip_dp_device *dp) return PTR_ERR(dp->pclk); } + clk = devm_clk_get_optional_enabled(dev, "hclk"); + if (IS_ERR(clk)) + return dev_err_probe(dev, PTR_ERR(clk), + "failed to get hclk property\n"); + dp->rst = devm_reset_control_get(dev, "dp"); if (IS_ERR(dp->rst)) { DRM_DEV_ERROR(dev, "failed to get dp reset control\n"); diff --git a/drivers/gpu/drm/rockchip/dw_dp-rockchip.c b/drivers/gpu/drm/rockchip/dw_dp-rockchip.c index 6d57e1c74627..410e5f457330 100644 --- a/drivers/gpu/drm/rockchip/dw_dp-rockchip.c +++ b/drivers/gpu/drm/rockchip/dw_dp-rockchip.c @@ -76,7 +76,7 @@ static const struct drm_encoder_helper_funcs dw_dp_encoder_helper_funcs = { static int dw_dp_rockchip_bind(struct device *dev, struct device *master, void *data) { struct platform_device *pdev = to_platform_device(dev); - struct dw_dp_plat_data plat_data; + const struct dw_dp_plat_data *plat_data; struct drm_device *drm_dev = data; struct rockchip_dw_dp *dp; struct drm_encoder *encoder; @@ -90,7 +90,10 @@ static int dw_dp_rockchip_bind(struct device *dev, struct device *master, void * dp->dev = dev; platform_set_drvdata(pdev, dp); - plat_data.max_link_rate = 810000; + plat_data = of_device_get_match_data(dev); + if (!plat_data) + return -ENODEV; + encoder = &dp->encoder.encoder; encoder->possible_crtcs = drm_of_find_possible_crtcs(drm_dev, dev->of_node); rockchip_drm_encoder_set_crtc_endpoint_id(&dp->encoder, dev->of_node, 0, 0); @@ -100,32 +103,36 @@ static int dw_dp_rockchip_bind(struct device *dev, struct device *master, void * return ret; drm_encoder_helper_add(encoder, &dw_dp_encoder_helper_funcs); - dp->base = dw_dp_bind(dev, encoder, &plat_data); - if (IS_ERR(dp->base)) { - ret = PTR_ERR(dp->base); - return ret; - } + dp->base = dw_dp_bind(dev, encoder, plat_data); + if (IS_ERR(dp->base)) + return PTR_ERR(dp->base); connector = drm_bridge_connector_init(drm_dev, encoder); if (IS_ERR(connector)) { - ret = PTR_ERR(connector); - return dev_err_probe(dev, ret, "Failed to init bridge connector"); + dw_dp_unbind(dp->base); + return dev_err_probe(dev, PTR_ERR(connector), + "Failed to init bridge connector\n"); } - drm_connector_attach_encoder(connector, encoder); + return drm_connector_attach_encoder(connector, encoder); +} - return 0; +static void dw_dp_rockchip_unbind(struct device *dev, struct device *master, + void *data) +{ + struct rockchip_dw_dp *dp = dev_get_drvdata(dev); + + dw_dp_unbind(dp->base); } static const struct component_ops dw_dp_rockchip_component_ops = { .bind = dw_dp_rockchip_bind, + .unbind = dw_dp_rockchip_unbind, }; static int dw_dp_probe(struct platform_device *pdev) { - struct device *dev = &pdev->dev; - - return component_add(dev, &dw_dp_rockchip_component_ops); + return component_add(&pdev->dev, &dw_dp_rockchip_component_ops); } static void dw_dp_remove(struct platform_device *pdev) @@ -133,8 +140,16 @@ static void dw_dp_remove(struct platform_device *pdev) component_del(&pdev->dev, &dw_dp_rockchip_component_ops); } +static const struct dw_dp_plat_data rk3588_dp_plat_data = { + .max_link_rate = 810000, + .pixel_mode = DW_DP_MP_QUAD_PIXEL, +}; + static const struct of_device_id dw_dp_of_match[] = { - { .compatible = "rockchip,rk3588-dp", }, + { + .compatible = "rockchip,rk3588-dp", + .data = &rk3588_dp_plat_data, + }, {} }; MODULE_DEVICE_TABLE(of, dw_dp_of_match); diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c index 7ec7bea5e38e..3c48984d6244 100644 --- a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c +++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c @@ -336,7 +336,8 @@ static bool vop2_output_uv_swap(u32 bus_format, u32 output_mode) static bool vop2_output_rg_swap(struct vop2 *vop2, u32 bus_format) { - if (vop2->version == VOP_VERSION_RK3588) { + if (vop2->version == VOP_VERSION_RK3588 || + vop2->version == VOP_VERSION_RK3576) { if (bus_format == MEDIA_BUS_FMT_YUV8_1X24 || bus_format == MEDIA_BUS_FMT_YUV10_1X30) return true; @@ -350,6 +351,8 @@ static bool is_yuv_output(u32 bus_format) switch (bus_format) { case MEDIA_BUS_FMT_YUV8_1X24: case MEDIA_BUS_FMT_YUV10_1X30: + case MEDIA_BUS_FMT_YUYV10_1X20: + case MEDIA_BUS_FMT_UYVY10_1X20: case MEDIA_BUS_FMT_UYYVYY8_0_5X24: case MEDIA_BUS_FMT_UYYVYY10_0_5X30: case MEDIA_BUS_FMT_YUYV8_2X8: diff --git a/drivers/gpu/drm/rockchip/rockchip_vop2_reg.c b/drivers/gpu/drm/rockchip/rockchip_vop2_reg.c index f3950e8476a7..7cac271704c1 100644 --- a/drivers/gpu/drm/rockchip/rockchip_vop2_reg.c +++ b/drivers/gpu/drm/rockchip/rockchip_vop2_reg.c @@ -2289,15 +2289,6 @@ static void rk3568_vop2_setup_layer_mixer(struct vop2_video_port *vp) * lead to the configuration of the previous VP being take effect along with the VSYNC * of the new VP. */ - if (layer_sel != old_layer_sel || port_sel != old_port_sel) - ovl_ctrl |= FIELD_PREP(RK3568_OVL_CTRL__LAYERSEL_REGDONE_SEL, vp->id); - vop2_writel(vop2, RK3568_OVL_CTRL, ovl_ctrl); - - if (port_sel != old_port_sel) { - vop2_writel(vop2, RK3568_OVL_PORT_SEL, port_sel); - vop2_cfg_done(vp); - rk3568_vop2_wait_for_port_mux_done(vop2); - } if (layer_sel != old_layer_sel && atv_layer_sel != old_layer_sel) { cfg_done = vop2_readl(vop2, RK3568_REG_CFG_DONE); @@ -2307,7 +2298,17 @@ static void rk3568_vop2_setup_layer_mixer(struct vop2_video_port *vp) * Changes of other VPs' overlays have not taken effect */ if (cfg_done) - rk3568_vop2_wait_for_layer_cfg_done(vop2, vop2->old_layer_sel); + rk3568_vop2_wait_for_layer_cfg_done(vop2, old_layer_sel); + } + + if (layer_sel != old_layer_sel || port_sel != old_port_sel) + ovl_ctrl |= FIELD_PREP(RK3568_OVL_CTRL__LAYERSEL_REGDONE_SEL, vp->id); + vop2_writel(vop2, RK3568_OVL_CTRL, ovl_ctrl); + + if (port_sel != old_port_sel) { + vop2_writel(vop2, RK3568_OVL_PORT_SEL, port_sel); + vop2_cfg_done(vp); + rk3568_vop2_wait_for_port_mux_done(vop2); } vop2_writel(vop2, RK3568_OVL_LAYER_SEL, layer_sel); diff --git a/drivers/gpu/drm/sun4i/sun4i_crtc.c b/drivers/gpu/drm/sun4i/sun4i_crtc.c index 18e74047b0f5..3e90aeb68c48 100644 --- a/drivers/gpu/drm/sun4i/sun4i_crtc.c +++ b/drivers/gpu/drm/sun4i/sun4i_crtc.c @@ -208,7 +208,7 @@ struct sun4i_crtc *sun4i_crtc_init(struct drm_device *drm, planes = sunxi_engine_layers_init(drm, engine); if (IS_ERR(planes)) { dev_err(drm->dev, "Couldn't create the planes\n"); - return NULL; + return ERR_CAST(planes); } /* find primary and cursor planes for drm_crtc_init_with_planes */ diff --git a/drivers/gpu/drm/sun4i/sun4i_hdmi_enc.c b/drivers/gpu/drm/sun4i/sun4i_hdmi_enc.c index ab0938ba61f7..c7b539b3497f 100644 --- a/drivers/gpu/drm/sun4i/sun4i_hdmi_enc.c +++ b/drivers/gpu/drm/sun4i/sun4i_hdmi_enc.c @@ -146,7 +146,7 @@ static void sun4i_hdmi_enable(struct drm_encoder *encoder, drm_atomic_helper_connector_hdmi_update_infoframes(connector, state); - val |= SUN4I_HDMI_PKT_CTRL_TYPE(0, SUN4I_HDMI_PKT_AVI); + val = SUN4I_HDMI_PKT_CTRL_TYPE(0, SUN4I_HDMI_PKT_AVI); val |= SUN4I_HDMI_PKT_CTRL_TYPE(1, SUN4I_HDMI_PKT_END); writel(val, hdmi->base + SUN4I_HDMI_PKT_CTRL_REG(0)); diff --git a/drivers/gpu/drm/sun4i/sun4i_tcon.c b/drivers/gpu/drm/sun4i/sun4i_tcon.c index 960e83c8291d..67047e3b657f 100644 --- a/drivers/gpu/drm/sun4i/sun4i_tcon.c +++ b/drivers/gpu/drm/sun4i/sun4i_tcon.c @@ -717,9 +717,11 @@ void sun4i_tcon_mode_set(struct sun4i_tcon *tcon, case DRM_MODE_ENCODER_DSI: /* DSI is tied to special case of CPU interface */ sun4i_tcon0_mode_set_cpu(tcon, encoder, mode); + sun4i_tcon_set_mux(tcon, 0, encoder); break; case DRM_MODE_ENCODER_LVDS: sun4i_tcon0_mode_set_lvds(tcon, encoder, mode); + sun4i_tcon_set_mux(tcon, 0, encoder); break; case DRM_MODE_ENCODER_NONE: sun4i_tcon0_mode_set_rgb(tcon, encoder, mode); @@ -970,6 +972,7 @@ static int sun4i_tcon_of_get_id_from_port(struct device_node *port) continue; ret = of_property_read_u32(remote, "reg", ®); + of_node_put(remote); if (ret) continue; @@ -1407,7 +1410,7 @@ static int sun8i_r40_tcon_tv_set_mux(struct sun4i_tcon *tcon, { struct device_node *port, *remote; struct platform_device *pdev; - int id, ret; + int id, ret = 0; /* find TCON TOP platform device and TCON id */ @@ -1430,21 +1433,20 @@ static int sun8i_r40_tcon_tv_set_mux(struct sun4i_tcon *tcon, if (IS_ENABLED(CONFIG_DRM_SUN8I_TCON_TOP) && encoder->encoder_type == DRM_MODE_ENCODER_TMDS) { ret = sun8i_tcon_top_set_hdmi_src(&pdev->dev, id); - if (ret) { - put_device(&pdev->dev); - return ret; - } + if (ret) + goto out_put_device; } if (IS_ENABLED(CONFIG_DRM_SUN8I_TCON_TOP)) { ret = sun8i_tcon_top_de_config(&pdev->dev, tcon->id, id); - if (ret) { - put_device(&pdev->dev); - return ret; - } + if (ret) + goto out_put_device; } - return 0; +out_put_device: + put_device(&pdev->dev); + + return ret; } static const struct sun4i_tcon_quirks sun4i_a10_quirks = { diff --git a/drivers/gpu/drm/sun4i/sun8i_dw_hdmi.c b/drivers/gpu/drm/sun4i/sun8i_dw_hdmi.c index 96532709c2a7..51cdfa1ae3bf 100644 --- a/drivers/gpu/drm/sun4i/sun8i_dw_hdmi.c +++ b/drivers/gpu/drm/sun4i/sun8i_dw_hdmi.c @@ -83,6 +83,8 @@ static u32 sun8i_dw_hdmi_find_possible_crtcs(struct drm_device *drm, of_node_put(remote_port); } } + + of_node_put(port); } else { crtcs = drm_of_find_possible_crtcs(drm, node); } diff --git a/drivers/gpu/drm/sun4i/sun8i_hdmi_phy.c b/drivers/gpu/drm/sun4i/sun8i_hdmi_phy.c index 4fa69c463dc4..489ea94693ff 100644 --- a/drivers/gpu/drm/sun4i/sun8i_hdmi_phy.c +++ b/drivers/gpu/drm/sun4i/sun8i_hdmi_phy.c @@ -91,7 +91,7 @@ static const struct dw_hdmi_mpll_config sun50i_h6_mpll_cfg[] = { }, }, { 594000000, { - { 0x1a40, 0x0003 }, + { 0x1a7c, 0x0003 }, { 0x3b4c, 0x0003 }, { 0x5a64, 0x0003 }, }, diff --git a/drivers/gpu/drm/sun4i/sun8i_vi_scaler.c b/drivers/gpu/drm/sun4i/sun8i_vi_scaler.c index 82df6244af88..695491b7739b 100644 --- a/drivers/gpu/drm/sun4i/sun8i_vi_scaler.c +++ b/drivers/gpu/drm/sun4i/sun8i_vi_scaler.c @@ -893,20 +893,26 @@ static void sun8i_vi_scaler_set_coeff(struct regmap *map, u32 base, lan3coefftab32_left[offset + i]); regmap_write(map, SUN8I_SCALER_VSU_YHCOEFF1(base, i), lan3coefftab32_right[offset + i]); + } + offset = sun8i_vi_scaler_coef_index(vstep) * + SUN8I_VI_SCALER_COEFF_COUNT; + for (i = 0; i < SUN8I_VI_SCALER_COEFF_COUNT; i++) + regmap_write(map, SUN8I_SCALER_VSU_YVCOEFF(base, i), + lan2coefftab32[offset + i]); + + offset = sun8i_vi_scaler_coef_index(hstep / format->hsub) * + SUN8I_VI_SCALER_COEFF_COUNT; + for (i = 0; i < SUN8I_VI_SCALER_COEFF_COUNT; i++) { regmap_write(map, SUN8I_SCALER_VSU_CHCOEFF0(base, i), ch_left[offset + i]); regmap_write(map, SUN8I_SCALER_VSU_CHCOEFF1(base, i), ch_right[offset + i]); } - - offset = sun8i_vi_scaler_coef_index(hstep) * + offset = sun8i_vi_scaler_coef_index(vstep / format->vsub) * SUN8I_VI_SCALER_COEFF_COUNT; - for (i = 0; i < SUN8I_VI_SCALER_COEFF_COUNT; i++) { - regmap_write(map, SUN8I_SCALER_VSU_YVCOEFF(base, i), - lan2coefftab32[offset + i]); + for (i = 0; i < SUN8I_VI_SCALER_COEFF_COUNT; i++) regmap_write(map, SUN8I_SCALER_VSU_CVCOEFF(base, i), cy[offset + i]); - } } void sun8i_vi_scaler_enable(struct sun8i_mixer *mixer, int layer, bool enable) diff --git a/drivers/gpu/drm/tve200/tve200_drv.c b/drivers/gpu/drm/tve200/tve200_drv.c index a048e37f1c2c..7d0329292ef6 100644 --- a/drivers/gpu/drm/tve200/tve200_drv.c +++ b/drivers/gpu/drm/tve200/tve200_drv.c @@ -259,6 +259,7 @@ static const struct of_device_id tve200_of_match[] = { }, {}, }; +MODULE_DEVICE_TABLE(of, tve200_of_match); static struct platform_driver tve200_driver = { .driver = { diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 5bbf1b3e60ec..9b9838696579 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -214,15 +214,11 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job) struct v3d_dev *v3d = job->base.v3d; struct v3d_queue_state *queue = &v3d->queue[V3D_BIN]; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; unsigned long irqflags; - if (unlikely(job->base.base.s_fence->finished.error)) { - spin_lock_irqsave(&queue->queue_lock, irqflags); - queue->active_job = NULL; - spin_unlock_irqrestore(&queue->queue_lock, irqflags); - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; /* Lock required around bin_job update vs * v3d_overflow_mem_work(). @@ -239,7 +235,7 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job) fence = v3d_fence_create(v3d, V3D_BIN); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -267,6 +263,12 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job) V3D_CORE_WRITE(0, V3D_CLE_CT0QEA, job->end); return fence; + +out_clean_job: + spin_lock_irqsave(&queue->queue_lock, irqflags); + queue->active_job = NULL; + spin_unlock_irqrestore(&queue->queue_lock, irqflags); + return fence; } static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) @@ -274,12 +276,10 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) struct v3d_render_job *job = to_render_job(sched_job); struct v3d_dev *v3d = job->base.v3d; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; - if (unlikely(job->base.base.s_fence->finished.error)) { - v3d->queue[V3D_RENDER].active_job = NULL; - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; v3d->queue[V3D_RENDER].active_job = &job->base; @@ -293,7 +293,7 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) fence = v3d_fence_create(v3d, V3D_RENDER); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -314,6 +314,10 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) V3D_CORE_WRITE(0, V3D_CLE_CT1QEA, job->end); return fence; + +out_clean_job: + v3d->queue[V3D_RENDER].active_job = NULL; + return fence; } static struct dma_fence * @@ -322,18 +326,16 @@ v3d_tfu_job_run(struct drm_sched_job *sched_job) struct v3d_tfu_job *job = to_tfu_job(sched_job); struct v3d_dev *v3d = job->base.v3d; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; - if (unlikely(job->base.base.s_fence->finished.error)) { - v3d->queue[V3D_TFU].active_job = NULL; - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; v3d->queue[V3D_TFU].active_job = &job->base; fence = v3d_fence_create(v3d, V3D_TFU); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -361,6 +363,10 @@ v3d_tfu_job_run(struct drm_sched_job *sched_job) V3D_WRITE(V3D_TFU_ICFG(v3d->ver), job->args.icfg | V3D_TFU_ICFG_IOC); return fence; + +out_clean_job: + v3d->queue[V3D_TFU].active_job = NULL; + return fence; } static struct dma_fence * @@ -369,13 +375,11 @@ v3d_csd_job_run(struct drm_sched_job *sched_job) struct v3d_csd_job *job = to_csd_job(sched_job); struct v3d_dev *v3d = job->base.v3d; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; int i, csd_cfg0_reg; - if (unlikely(job->base.base.s_fence->finished.error)) { - v3d->queue[V3D_CSD].active_job = NULL; - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; /* The HW interprets a workgroup size of 0 as 65536; however, the * user-space driver exposes a maximum of 65535. Therefore, a 0 in @@ -393,7 +397,7 @@ v3d_csd_job_run(struct drm_sched_job *sched_job) fence = v3d_fence_create(v3d, V3D_CSD); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -420,6 +424,10 @@ v3d_csd_job_run(struct drm_sched_job *sched_job) V3D_CORE_WRITE(0, csd_cfg0_reg, job->args.cfg[0]); return fence; + +out_clean_job: + v3d->queue[V3D_CSD].active_job = NULL; + return fence; } static void diff --git a/drivers/gpu/drm/xe/tests/xe_migrate.c b/drivers/gpu/drm/xe/tests/xe_migrate.c index 5904d658d1f2..e056fac1a86b 100644 --- a/drivers/gpu/drm/xe/tests/xe_migrate.c +++ b/drivers/gpu/drm/xe/tests/xe_migrate.c @@ -197,8 +197,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test, err = xe_bo_vmap(bo); if (err) { - KUNIT_FAIL(test, "Failed to vmap our pagetables: %li\n", - PTR_ERR(bo)); + KUNIT_FAIL(test, "Failed to vmap our pagetables: %d\n", err); return; } diff --git a/drivers/gpu/drm/xe/xe_gt_idle.c b/drivers/gpu/drm/xe/xe_gt_idle.c index 3e3d1d52f630..81e3ec7ff4fd 100644 --- a/drivers/gpu/drm/xe/xe_gt_idle.c +++ b/drivers/gpu/drm/xe/xe_gt_idle.c @@ -236,7 +236,8 @@ int xe_gt_idle_pg_print(struct xe_gt *gt, struct drm_printer *p) xe_force_wake_put(gt_to_fw(gt), fw_ref); } - if (gt->info.engine_mask & XE_HW_ENGINE_RCS_MASK) { + if (gt->info.engine_mask & + (XE_HW_ENGINE_RCS_MASK | XE_HW_ENGINE_CCS_MASK)) { drm_printf(p, "Render Power Gating Enabled: %s\n", str_yes_no(pg_enabled & RENDER_POWERGATE_ENABLE)); diff --git a/drivers/gpu/host1x/debug.c b/drivers/gpu/host1x/debug.c index 6433c00d5d7e..b828f773fc06 100644 --- a/drivers/gpu/host1x/debug.c +++ b/drivers/gpu/host1x/debug.c @@ -31,7 +31,7 @@ void host1x_debug_output(struct output *o, const char *fmt, ...) int len; va_start(args, fmt); - len = vsnprintf(o->buf, sizeof(o->buf), fmt, args); + len = vscnprintf(o->buf, sizeof(o->buf), fmt, args); va_end(args); o->fn(o->ctx, o->buf, len, false); @@ -43,7 +43,7 @@ void host1x_debug_cont(struct output *o, const char *fmt, ...) int len; va_start(args, fmt); - len = vsnprintf(o->buf, sizeof(o->buf), fmt, args); + len = vscnprintf(o->buf, sizeof(o->buf), fmt, args); va_end(args); o->fn(o->ctx, o->buf, len, true); diff --git a/drivers/gpu/host1x/hw/channel_hw.c b/drivers/gpu/host1x/hw/channel_hw.c index d44b8de890be..46b6494768b4 100644 --- a/drivers/gpu/host1x/hw/channel_hw.c +++ b/drivers/gpu/host1x/hw/channel_hw.c @@ -36,10 +36,9 @@ static void trace_write_gather(struct host1x_cdma *cdma, struct host1x_bo *bo, for (i = 0; i < words; i += TRACE_MAX_LENGTH) { u32 num_words = min(words - i, TRACE_MAX_LENGTH); - offset += i * sizeof(u32); - trace_host1x_cdma_push_gather(dev_name(dev), bo, - num_words, offset, + num_words, + offset + i * sizeof(u32), mem); } diff --git a/drivers/hid/hid-core.c b/drivers/hid/hid-core.c index 7f442a2798d8..78adb5a48a61 100644 --- a/drivers/hid/hid-core.c +++ b/drivers/hid/hid-core.c @@ -2450,9 +2450,16 @@ EXPORT_SYMBOL_GPL(hid_hw_start); * * This is usually called from remove function or from probe when something * failed and hid_hw_start was called already. + * + * If the caller enabled HID input via hid_device_io_start() and is unwinding + * without an explicit hid_device_io_stop(), quiesce input first so that + * in-flight reports cannot reach handlers (e.g. hidraw_report_event) whose + * backing objects hid_disconnect() is about to free. */ void hid_hw_stop(struct hid_device *hdev) { + if (hdev->io_started) + hid_device_io_stop(hdev); hid_disconnect(hdev); hdev->ll_driver->stop(hdev); } @@ -2845,6 +2852,8 @@ static int __hid_device_probe(struct hid_device *hdev, struct hid_driver *hdrv) */ if (ret) { + if (hdev->io_started) + hid_device_io_stop(hdev); devres_release_group(&hdev->dev, hdev->devres_group_id); hid_close_report(hdev); hdev->driver = NULL; diff --git a/drivers/hid/hid-haptic.c b/drivers/hid/hid-haptic.c index fc8a9997f815..575692580ab9 100644 --- a/drivers/hid/hid-haptic.c +++ b/drivers/hid/hid-haptic.c @@ -187,7 +187,7 @@ static void fill_effect_buf(struct hid_haptic_device *haptic, value = waveform_ordinal; break; default: - break; + continue; } field->value[j] = value; diff --git a/drivers/hid/hid-lg4ff.c b/drivers/hid/hid-lg4ff.c index 32b711723f2a..9ddd669d88f1 100644 --- a/drivers/hid/hid-lg4ff.c +++ b/drivers/hid/hid-lg4ff.c @@ -336,6 +336,8 @@ int lg4ff_raw_event(struct hid_device *hdev, struct hid_report *report, if (entry->wdata.combine) { switch (entry->wdata.product_id) { case USB_DEVICE_ID_LOGITECH_WHEEL: + if (size < 7) + return 0; rd[5] = rd[3]; rd[6] = 0x7F; return 1; @@ -343,10 +345,14 @@ int lg4ff_raw_event(struct hid_device *hdev, struct hid_report *report, case USB_DEVICE_ID_LOGITECH_WINGMAN_FFG: case USB_DEVICE_ID_LOGITECH_MOMO_WHEEL: case USB_DEVICE_ID_LOGITECH_MOMO_WHEEL2: + if (size < 6) + return 0; rd[4] = rd[3]; rd[5] = 0x7F; return 1; case USB_DEVICE_ID_LOGITECH_DFP_WHEEL: + if (size < 7) + return 0; rd[5] = rd[4]; rd[6] = 0x7F; return 1; @@ -366,6 +372,8 @@ int lg4ff_raw_event(struct hid_device *hdev, struct hid_report *report, } /* Compute a combined axis when wheel does not supply it */ + if (size <= offset + 1) + return 0; rd[offset] = (0xFF + rd[offset] - rd[offset+1]) >> 1; rd[offset+1] = 0x7F; return 1; diff --git a/drivers/hid/hid-multitouch.c b/drivers/hid/hid-multitouch.c index 1959481dc782..fd1b45ece454 100644 --- a/drivers/hid/hid-multitouch.c +++ b/drivers/hid/hid-multitouch.c @@ -2567,7 +2567,7 @@ static const struct hid_device_id mt_devices[] = { HID_ANY_ID) }, /* Hantick */ - { .driver_data = MT_CLS_NSMU, + { .driver_data = MT_CLS_WIN_8_FORCE_MULTI_INPUT_NSMU, HID_DEVICE(BUS_I2C, HID_GROUP_MULTITOUCH_WIN_8, I2C_VENDOR_ID_HANTICK, I2C_PRODUCT_ID_HANTICK_5288) }, diff --git a/drivers/hid/hid-nintendo.c b/drivers/hid/hid-nintendo.c index d5f049424f08..34642352b439 100644 --- a/drivers/hid/hid-nintendo.c +++ b/drivers/hid/hid-nintendo.c @@ -1453,7 +1453,6 @@ static void joycon_parse_imu_report(struct joycon_ctlr *ctlr, dropped_threshold = ctlr->imu_avg_delta_ms * 3 / 2; dropped_pkts = (delta - min(delta, dropped_threshold)) / ctlr->imu_avg_delta_ms; - ctlr->imu_timestamp_us += 1000 * ctlr->imu_avg_delta_ms; if (dropped_pkts > JC_IMU_DROPPED_PKT_WARNING) { hid_warn_ratelimited(ctlr->hdev, "compensating for %u dropped IMU reports\n", diff --git a/drivers/hid/hid-roccat-kone.c b/drivers/hid/hid-roccat-kone.c index fabc08efcfd8..2510001fead9 100644 --- a/drivers/hid/hid-roccat-kone.c +++ b/drivers/hid/hid-roccat-kone.c @@ -36,6 +36,8 @@ static uint profile_numbers[5] = {0, 1, 2, 3, 4}; static void kone_profile_activated(struct kone_device *kone, uint new_profile) { + if (new_profile < 1 || new_profile > ARRAY_SIZE(kone->profiles)) + new_profile = 1; kone->actual_profile = new_profile; kone->actual_dpi = kone->profiles[new_profile - 1].startup_dpi; } @@ -793,8 +795,10 @@ static void kone_keep_values_up_to_date(struct kone_device *kone, { switch (event->event) { case kone_mouse_event_switch_profile: - kone->actual_dpi = kone->profiles[event->value - 1]. - startup_dpi; + if (event->value >= 1 && + event->value <= ARRAY_SIZE(kone->profiles)) + kone->actual_dpi = + kone->profiles[event->value - 1].startup_dpi; fallthrough; case kone_mouse_event_osd_profile: kone->actual_profile = event->value; diff --git a/drivers/hid/hid-steam.c b/drivers/hid/hid-steam.c index 197126d6e081..80d2eabc930e 100644 --- a/drivers/hid/hid-steam.c +++ b/drivers/hid/hid-steam.c @@ -43,6 +43,7 @@ #include #include #include +#include #include "hid-ids.h" MODULE_DESCRIPTION("HID driver for Valve Steam Controller"); @@ -68,13 +69,14 @@ static LIST_HEAD(steam_devices); /* Joystick runs are about 5 mm and 32768 units */ #define STEAM_DECK_JOYSTICK_RESOLUTION 6553 /* Accelerometer has 16 bit resolution and a range of +/- 2g */ -#define STEAM_DECK_ACCEL_RES_PER_G 16384 -#define STEAM_DECK_ACCEL_RANGE 32768 +#define STEAM_ACCEL_RES_PER_G 16384 +#define STEAM_ACCEL_RANGE 32768 +#define STEAM_ACCEL_FUZZ 128 #define STEAM_DECK_ACCEL_FUZZ 32 /* Gyroscope has 16 bit resolution and a range of +/- 2000 dps */ -#define STEAM_DECK_GYRO_RES_PER_DPS 16 -#define STEAM_DECK_GYRO_RANGE 32768 -#define STEAM_DECK_GYRO_FUZZ 1 +#define STEAM_GYRO_RES_PER_DPS 16 +#define STEAM_GYRO_RANGE 32768 +#define STEAM_GYRO_FUZZ 0 #define STEAM_PAD_FUZZ 256 @@ -149,7 +151,7 @@ enum { SETTING_USB_DEBUG_MODE, SETTING_LEFT_TRACKPAD_MODE, SETTING_RIGHT_TRACKPAD_MODE, - SETTING_MOUSE_POINTER_ENABLED, + SETTING_LIZARD_MODE, /* 10 */ SETTING_DPAD_DEADZONE, @@ -243,14 +245,39 @@ enum { /* Input report identifiers */ enum { - ID_CONTROLLER_STATE = 1, - ID_CONTROLLER_DEBUG = 2, - ID_CONTROLLER_WIRELESS = 3, - ID_CONTROLLER_STATUS = 4, - ID_CONTROLLER_DEBUG2 = 5, - ID_CONTROLLER_SECONDARY_STATE = 6, - ID_CONTROLLER_BLE_STATE = 7, - ID_CONTROLLER_DECK_STATE = 9 + ID_CONTROLLER_STATE = 1, + ID_CONTROLLER_DEBUG = 2, + ID_CONTROLLER_WIRELESS = 3, + ID_CONTROLLER_STATUS = 4, + ID_CONTROLLER_DEBUG2 = 5, + ID_CONTROLLER_SECONDARY_STATE = 6, + ID_CONTROLLER_BLE_STATE = 7, + ID_CONTROLLER_DECK_STATE = 9, +}; + +/* Read-only attributes */ +enum { + ATTRIB_UNIQUE_ID, // deprecated + ATTRIB_PRODUCT_ID, + ATTRIB_PRODUCT_REVISON, // deprecated + ATTRIB_CAPABILITIES = ATTRIB_PRODUCT_REVISON, // intentional aliasing + ATTRIB_FIRMWARE_VERSION, // deprecated + ATTRIB_FIRMWARE_BUILD_TIME, + ATTRIB_RADIO_FIRMWARE_BUILD_TIME, + ATTRIB_RADIO_DEVICE_ID0, + ATTRIB_RADIO_DEVICE_ID1, + ATTRIB_DONGLE_FIRMWARE_BUILD_TIME, + ATTRIB_HW_ID, // AKA BOARD_REVISION, + ATTRIB_BOOTLOADER_BUILD_TIME, + ATTRIB_CONNECTION_INTERVAL_IN_US, + ATTRIB_SECONDARY_FIRMWARE_BUILD_TIME, + ATTRIB_SECONDARY_BOOTLOADER_BUILD_TIME, + ATTRIB_SECONDARY_HW_ID, // AKA BOARD_REVISION, + ATTRIB_STREAMING, + ATTRIB_TRACKPAD_ID, + ATTRIB_SECONDARY_TRACKPAD_ID, + + ATTRIB_COUNT }; /* String attribute identifiers */ @@ -259,14 +286,14 @@ enum { ATTRIB_STR_UNIT_SERIAL, }; -/* Values for GYRO_MODE (bitmask) */ +/* Values for IMU_MODE (bitmask) */ enum { - SETTING_GYRO_MODE_OFF = 0, - SETTING_GYRO_MODE_STEERING = BIT(0), - SETTING_GYRO_MODE_TILT = BIT(1), - SETTING_GYRO_MODE_SEND_ORIENTATION = BIT(2), - SETTING_GYRO_MODE_SEND_RAW_ACCEL = BIT(3), - SETTING_GYRO_MODE_SEND_RAW_GYRO = BIT(4), + SETTING_IMU_MODE_OFF = 0, + SETTING_IMU_MODE_STEERING = BIT(0), + SETTING_IMU_MODE_TILT = BIT(1), + SETTING_IMU_MODE_SEND_ORIENTATION = BIT(2), + SETTING_IMU_MODE_SEND_RAW_ACCEL = BIT(3), + SETTING_IMU_MODE_SEND_RAW_GYRO = BIT(4), }; /* Trackpad modes */ @@ -282,6 +309,11 @@ enum { TRACKPAD_GESTURE_KEYBOARD, }; +struct steam_controller_attribute { + unsigned char tag; + __le32 value; +} __packed; + /* Pad identifiers for the deck */ #define STEAM_PAD_LEFT 0 #define STEAM_PAD_RIGHT 1 @@ -313,6 +345,7 @@ struct steam_device { u16 rumble_left; u16 rumble_right; unsigned int sensor_timestamp_us; + unsigned int sensor_update_rate_us; struct work_struct unregister_work; }; @@ -323,6 +356,13 @@ static int steam_recv_report(struct steam_device *steam, u8 *buf; int ret; + /* + * All reports start with a two byte header. + * We must read at least two bytes to get a sensible output. + */ + if (size < 2) + return -EINVAL; + r = steam->hdev->report_enum[HID_FEATURE_REPORT].report_id_hash[0]; if (!r) { hid_err(steam->hdev, "No HID_FEATURE_REPORT submitted - nothing to read\n"); @@ -345,10 +385,31 @@ static int steam_recv_report(struct steam_device *steam, ret = hid_hw_raw_request(steam->hdev, 0x00, buf, hid_report_len(r) + 1, HID_FEATURE_REPORT, HID_REQ_GET_REPORT); - if (ret > 0) - memcpy(data, buf + 1, min(size, ret - 1)); + if (ret > 0) { + /* Remove the report ID from the return buffer */ + ret--; + size = min(size, ret); + memcpy(data, buf + 1, size); + } kfree(buf); - return ret; + + if (ret < 0) + hid_err(steam->hdev, "%s: error %d\n", __func__, ret); + else + hid_dbg(steam->hdev, "Received report %*ph\n", size, data); + if (ret < 0) + return ret; + + if (ret < 2) { + hid_err(steam->hdev, "%s: reply too short\n", __func__); + return -EPROTO; + } + if (ret < data[1] + 2) { + hid_err(steam->hdev, "%s: expected %u bytes, read %i\n", + __func__, data[1] + 2, ret); + return -EPROTO; + } + return size; } static int steam_send_report(struct steam_device *steam, @@ -375,6 +436,8 @@ static int steam_send_report(struct steam_device *steam, /* The report ID is always 0 */ memcpy(buf + 1, cmd, size); + hid_dbg(steam->hdev, "Sending report %*ph\n", size, cmd); + /* * Sometimes the wireless controller fails with EPIPE * when sending a feature report. @@ -447,25 +510,59 @@ static int steam_get_serial(struct steam_device *steam) u8 cmd[] = {ID_GET_STRING_ATTRIBUTE, sizeof(steam->serial_no), ATTRIB_STR_UNIT_SERIAL}; u8 reply[3 + STEAM_SERIAL_LEN + 1]; - mutex_lock(&steam->report_mutex); + guard(mutex)(&steam->report_mutex); ret = steam_send_report(steam, cmd, sizeof(cmd)); if (ret < 0) - goto out; + return ret; ret = steam_recv_report(steam, reply, sizeof(reply)); if (ret < 0) - goto out; + return ret; if (reply[0] != ID_GET_STRING_ATTRIBUTE || reply[1] < 1 || reply[1] > sizeof(steam->serial_no) || reply[2] != ATTRIB_STR_UNIT_SERIAL) { - ret = -EIO; - goto out; + hid_err(steam->hdev, "%s: invalid reply (%*ph)\n", __func__, + (int)sizeof(reply), reply); + return -EIO; } reply[3 + STEAM_SERIAL_LEN] = 0; strscpy(steam->serial_no, reply + 3, reply[1]); -out: - mutex_unlock(&steam->report_mutex); return ret; } +static int steam_get_attributes(struct steam_device *steam) +{ + int ret = 0; + u8 cmd[] = {ID_GET_ATTRIBUTES_VALUES, 0}; + u8 reply[64] = {}; + u8 size; + int i; + struct steam_controller_attribute *attr; + + guard(mutex)(&steam->report_mutex); + ret = steam_send_report(steam, cmd, sizeof(cmd)); + if (ret < 0) + return ret; + ret = steam_recv_report(steam, reply, sizeof(reply)); + if (ret < 0) + return ret; + if (reply[0] != ID_GET_ATTRIBUTES_VALUES || reply[1] < 2) { + hid_err(steam->hdev, "%s: invalid reply (%*ph)\n", __func__, + (int)sizeof(reply), reply); + return -EIO; + } + + size = min(reply[1], sizeof(reply) - 2); + for (i = 0; i + sizeof(*attr) <= size; i += sizeof(*attr)) { + attr = (struct steam_controller_attribute *)&reply[i + 2]; + if (attr->tag == ATTRIB_CONNECTION_INTERVAL_IN_US) { + steam->sensor_update_rate_us = get_unaligned_le32(&attr->value); + hid_dbg(steam->hdev, "Sensor update rate: %uus\n", + steam->sensor_update_rate_us); + } + } + + return 0; +} + /* * This command requests the wireless adaptor to post an event * with the connection status. Useful if this driver is loaded when @@ -473,11 +570,8 @@ static int steam_get_serial(struct steam_device *steam) */ static inline int steam_request_conn_status(struct steam_device *steam) { - int ret; - mutex_lock(&steam->report_mutex); - ret = steam_send_report_byte(steam, ID_DONGLE_GET_WIRELESS_STATE); - mutex_unlock(&steam->report_mutex); - return ret; + guard(mutex)(&steam->report_mutex); + return steam_send_report_byte(steam, ID_DONGLE_GET_WIRELESS_STATE); } /* @@ -624,6 +718,42 @@ static void steam_input_close(struct input_dev *dev) } } +static int steam_sensor_open(struct input_dev *dev) +{ + struct steam_device *steam = input_get_drvdata(dev); + unsigned long flags; + bool client_opened; + + spin_lock_irqsave(&steam->lock, flags); + client_opened = steam->client_opened; + spin_unlock_irqrestore(&steam->lock, flags); + if (client_opened) + return 0; + + guard(mutex)(&steam->report_mutex); + steam_write_settings(steam, SETTING_IMU_MODE, + SETTING_IMU_MODE_SEND_RAW_ACCEL | SETTING_IMU_MODE_SEND_RAW_GYRO, + 0); + + return 0; +} + +static void steam_sensor_close(struct input_dev *dev) +{ + struct steam_device *steam = input_get_drvdata(dev); + unsigned long flags; + bool client_opened; + + spin_lock_irqsave(&steam->lock, flags); + client_opened = steam->client_opened; + spin_unlock_irqrestore(&steam->lock, flags); + if (client_opened) + return; + + guard(mutex)(&steam->report_mutex); + steam_write_settings(steam, SETTING_IMU_MODE, 0, 0); +} + static enum power_supply_property steam_battery_props[] = { POWER_SUPPLY_PROP_PRESENT, POWER_SUPPLY_PROP_SCOPE, @@ -837,9 +967,6 @@ static int steam_sensors_register(struct steam_device *steam) struct input_dev *sensors; int ret; - if (!(steam->quirks & STEAM_QUIRK_DECK)) - return 0; - rcu_read_lock(); sensors = rcu_dereference(steam->sensors); rcu_read_unlock(); @@ -854,8 +981,14 @@ static int steam_sensors_register(struct steam_device *steam) input_set_drvdata(sensors, steam); sensors->dev.parent = &hdev->dev; + if (!(steam->quirks & STEAM_QUIRK_DECK)) { + sensors->open = steam_sensor_open; + sensors->close = steam_sensor_close; + } - sensors->name = "Steam Deck Motion Sensors"; + sensors->name = steam->quirks & STEAM_QUIRK_DECK ? + "Steam Deck Motion Sensors" : + "Steam Controller Motion Sensors"; sensors->phys = hdev->phys; sensors->uniq = steam->serial_no; sensors->id.bustype = hdev->bus; @@ -867,25 +1000,34 @@ static int steam_sensors_register(struct steam_device *steam) __set_bit(EV_MSC, sensors->evbit); __set_bit(MSC_TIMESTAMP, sensors->mscbit); - input_set_abs_params(sensors, ABS_X, -STEAM_DECK_ACCEL_RANGE, - STEAM_DECK_ACCEL_RANGE, STEAM_DECK_ACCEL_FUZZ, 0); - input_set_abs_params(sensors, ABS_Y, -STEAM_DECK_ACCEL_RANGE, - STEAM_DECK_ACCEL_RANGE, STEAM_DECK_ACCEL_FUZZ, 0); - input_set_abs_params(sensors, ABS_Z, -STEAM_DECK_ACCEL_RANGE, - STEAM_DECK_ACCEL_RANGE, STEAM_DECK_ACCEL_FUZZ, 0); - input_abs_set_res(sensors, ABS_X, STEAM_DECK_ACCEL_RES_PER_G); - input_abs_set_res(sensors, ABS_Y, STEAM_DECK_ACCEL_RES_PER_G); - input_abs_set_res(sensors, ABS_Z, STEAM_DECK_ACCEL_RES_PER_G); - - input_set_abs_params(sensors, ABS_RX, -STEAM_DECK_GYRO_RANGE, - STEAM_DECK_GYRO_RANGE, STEAM_DECK_GYRO_FUZZ, 0); - input_set_abs_params(sensors, ABS_RY, -STEAM_DECK_GYRO_RANGE, - STEAM_DECK_GYRO_RANGE, STEAM_DECK_GYRO_FUZZ, 0); - input_set_abs_params(sensors, ABS_RZ, -STEAM_DECK_GYRO_RANGE, - STEAM_DECK_GYRO_RANGE, STEAM_DECK_GYRO_FUZZ, 0); - input_abs_set_res(sensors, ABS_RX, STEAM_DECK_GYRO_RES_PER_DPS); - input_abs_set_res(sensors, ABS_RY, STEAM_DECK_GYRO_RES_PER_DPS); - input_abs_set_res(sensors, ABS_RZ, STEAM_DECK_GYRO_RES_PER_DPS); + if (steam->quirks & STEAM_QUIRK_DECK) { + input_set_abs_params(sensors, ABS_X, -STEAM_ACCEL_RANGE, + STEAM_ACCEL_RANGE, STEAM_DECK_ACCEL_FUZZ, 0); + input_set_abs_params(sensors, ABS_Y, -STEAM_ACCEL_RANGE, + STEAM_ACCEL_RANGE, STEAM_DECK_ACCEL_FUZZ, 0); + input_set_abs_params(sensors, ABS_Z, -STEAM_ACCEL_RANGE, + STEAM_ACCEL_RANGE, STEAM_DECK_ACCEL_FUZZ, 0); + } else { + input_set_abs_params(sensors, ABS_X, -STEAM_ACCEL_RANGE, + STEAM_ACCEL_RANGE, STEAM_ACCEL_FUZZ, 0); + input_set_abs_params(sensors, ABS_Y, -STEAM_ACCEL_RANGE, + STEAM_ACCEL_RANGE, STEAM_ACCEL_FUZZ, 0); + input_set_abs_params(sensors, ABS_Z, -STEAM_ACCEL_RANGE, + STEAM_ACCEL_RANGE, STEAM_ACCEL_FUZZ, 0); + } + input_abs_set_res(sensors, ABS_X, STEAM_ACCEL_RES_PER_G); + input_abs_set_res(sensors, ABS_Y, STEAM_ACCEL_RES_PER_G); + input_abs_set_res(sensors, ABS_Z, STEAM_ACCEL_RES_PER_G); + + input_set_abs_params(sensors, ABS_RX, -STEAM_GYRO_RANGE, + STEAM_GYRO_RANGE, STEAM_GYRO_FUZZ, 0); + input_set_abs_params(sensors, ABS_RY, -STEAM_GYRO_RANGE, + STEAM_GYRO_RANGE, STEAM_GYRO_FUZZ, 0); + input_set_abs_params(sensors, ABS_RZ, -STEAM_GYRO_RANGE, + STEAM_GYRO_RANGE, STEAM_GYRO_FUZZ, 0); + input_abs_set_res(sensors, ABS_RX, STEAM_GYRO_RES_PER_DPS); + input_abs_set_res(sensors, ABS_RY, STEAM_GYRO_RES_PER_DPS); + input_abs_set_res(sensors, ABS_RZ, STEAM_GYRO_RES_PER_DPS); ret = input_register_device(sensors); if (ret) @@ -916,9 +1058,6 @@ static void steam_sensors_unregister(struct steam_device *steam) { struct input_dev *sensors; - if (!(steam->quirks & STEAM_QUIRK_DECK)) - return; - rcu_read_lock(); sensors = rcu_dereference(steam->sensors); rcu_read_unlock(); @@ -966,6 +1105,12 @@ static int steam_register(struct steam_device *steam) strscpy(steam->serial_no, "XXXXXXXXXX", sizeof(steam->serial_no)); + ret = steam_get_attributes(steam); + if (ret < 0) + hid_err(steam->hdev, + "%s:steam_get_attributes failed with error %d\n", + __func__, ret); + hid_info(steam->hdev, "Steam Controller '%s' connected", steam->serial_no); @@ -1049,6 +1194,7 @@ static void steam_mode_switch_cb(struct work_struct *work) return; steam->gamepad_mode = !steam->gamepad_mode; + hid_dbg(steam->hdev, "%s: switching gamepad mode to %i\n", __func__, steam->gamepad_mode); if (steam->gamepad_mode) steam_set_lizard_mode(steam, false); else { @@ -1244,6 +1390,10 @@ static int steam_probe(struct hid_device *hdev, INIT_LIST_HEAD(&steam->list); INIT_WORK(&steam->rumble_work, steam_haptic_rumble_cb); steam->sensor_timestamp_us = 0; + if (steam->quirks & STEAM_QUIRK_DECK) + steam->sensor_update_rate_us = 4000; + else + steam->sensor_update_rate_us = 9000; INIT_WORK(&steam->unregister_work, steam_work_unregister_cb); /* @@ -1354,13 +1504,45 @@ static void steam_do_connect_event(struct steam_device *steam, bool connected) * Clamp the values to 32767..-32767 so that the range is * symmetrical and can be negated safely. */ -static inline s16 steam_le16(u8 *data) +static inline s16 steam_le16(const u8 *data) { - s16 x = (s16) le16_to_cpup((__le16 *)data); + s16 x = (s16) get_unaligned_le16((const __le16 *)data); return x == -32768 ? -32767 : x; } +struct steam_button_mapping { + int code; + u8 byte; + u8 bit; +}; + +struct steam_axis_mapping { + int code; + s8 sign; + u8 byte; +}; + +static void steam_map_buttons(struct input_dev *input, + const struct steam_button_mapping *mappings, const u8 *data) +{ + const struct steam_button_mapping *mapping; + + for (mapping = mappings; mapping->code; mapping++) + input_report_key(input, mapping->code, + data[mapping->byte] & BIT(mapping->bit)); +} + +static void steam_map_axes(struct input_dev *input, + const struct steam_axis_mapping *mappings, const u8 *data) +{ + const struct steam_axis_mapping *mapping; + + for (mapping = mappings; mapping->sign; mapping++) + input_report_abs(input, mapping->code, + mapping->sign * steam_le16(&data[mapping->byte])); +} + /* * The size for this message payload is 60. * The known values are: @@ -1427,18 +1609,52 @@ static inline s16 steam_le16(u8 *data) * 10.7 | -- | lpad_and_joy */ +static const struct steam_button_mapping steam_controller_button_mappings[] = { + { BTN_TR2, 8, 0 }, + { BTN_TL2, 8, 1 }, + { BTN_TR, 8, 2 }, + { BTN_TL, 8, 3 }, + { BTN_Y, 8, 4 }, + { BTN_B, 8, 5 }, + { BTN_X, 8, 6 }, + { BTN_A, 8, 7 }, + { BTN_SELECT, 9, 4 }, + { BTN_MODE, 9, 5 }, + { BTN_START, 9, 6 }, + { BTN_GRIPL, 9, 7 }, + { BTN_GRIPR, 10, 0 }, + { BTN_THUMBR, 10, 2 }, + { BTN_THUMBL, 10, 6 }, + { BTN_THUMB2, 10, 4 }, + { BTN_DPAD_UP, 9, 0 }, + { BTN_DPAD_RIGHT, 9, 1 }, + { BTN_DPAD_LEFT, 9, 2 }, + { BTN_DPAD_DOWN, 9, 3 }, + { /* sentinel */ }, +}; + +static const struct steam_axis_mapping steam_controller_axis_mappings[] = { + { ABS_RX, 1, 20 }, + { ABS_RY, -1, 22 }, + { /* sentinel */ }, +}; + +static const struct steam_axis_mapping steam_controller_imu_mappings[] = { + { ABS_X, 1, 28 }, + { ABS_Z, -1, 30 }, + { ABS_Y, 1, 32 }, + { ABS_RX, 1, 34 }, + { ABS_RZ, 1, 36 }, + { ABS_RY, 1, 38 }, + { /* sentinel */ }, +}; + static void steam_do_input_event(struct steam_device *steam, struct input_dev *input, u8 *data) { - /* 24 bits of buttons */ - u8 b8, b9, b10; s16 x, y; bool lpad_touched, lpad_and_joy; - b8 = data[8]; - b9 = data[9]; - b10 = data[10]; - input_report_abs(input, ABS_HAT2Y, data[11]); input_report_abs(input, ABS_HAT2X, data[12]); @@ -1450,8 +1666,8 @@ static void steam_do_input_event(struct steam_device *steam, * joystick values. * (lpad_touched || lpad_and_joy) tells if the lpad is really touched. */ - lpad_touched = b10 & BIT(3); - lpad_and_joy = b10 & BIT(7); + lpad_touched = data[10] & BIT(3); + lpad_and_joy = data[10] & BIT(7); x = steam_le16(data + 16); y = -steam_le16(data + 18); @@ -1467,35 +1683,25 @@ static void steam_do_input_event(struct steam_device *steam, input_report_abs(input, ABS_HAT0X, 0); input_report_abs(input, ABS_HAT0Y, 0); } + input_report_key(input, BTN_THUMB, lpad_touched || lpad_and_joy); - input_report_abs(input, ABS_RX, steam_le16(data + 20)); - input_report_abs(input, ABS_RY, -steam_le16(data + 22)); - - input_event(input, EV_KEY, BTN_TR2, !!(b8 & BIT(0))); - input_event(input, EV_KEY, BTN_TL2, !!(b8 & BIT(1))); - input_event(input, EV_KEY, BTN_TR, !!(b8 & BIT(2))); - input_event(input, EV_KEY, BTN_TL, !!(b8 & BIT(3))); - input_event(input, EV_KEY, BTN_Y, !!(b8 & BIT(4))); - input_event(input, EV_KEY, BTN_B, !!(b8 & BIT(5))); - input_event(input, EV_KEY, BTN_X, !!(b8 & BIT(6))); - input_event(input, EV_KEY, BTN_A, !!(b8 & BIT(7))); - input_event(input, EV_KEY, BTN_SELECT, !!(b9 & BIT(4))); - input_event(input, EV_KEY, BTN_MODE, !!(b9 & BIT(5))); - input_event(input, EV_KEY, BTN_START, !!(b9 & BIT(6))); - input_event(input, EV_KEY, BTN_GRIPL, !!(b9 & BIT(7))); - input_event(input, EV_KEY, BTN_GRIPR, !!(b10 & BIT(0))); - input_event(input, EV_KEY, BTN_THUMBR, !!(b10 & BIT(2))); - input_event(input, EV_KEY, BTN_THUMBL, !!(b10 & BIT(6))); - input_event(input, EV_KEY, BTN_THUMB, lpad_touched || lpad_and_joy); - input_event(input, EV_KEY, BTN_THUMB2, !!(b10 & BIT(4))); - input_event(input, EV_KEY, BTN_DPAD_UP, !!(b9 & BIT(0))); - input_event(input, EV_KEY, BTN_DPAD_RIGHT, !!(b9 & BIT(1))); - input_event(input, EV_KEY, BTN_DPAD_LEFT, !!(b9 & BIT(2))); - input_event(input, EV_KEY, BTN_DPAD_DOWN, !!(b9 & BIT(3))); + steam_map_buttons(input, steam_controller_button_mappings, data); + steam_map_axes(input, steam_controller_axis_mappings, data); input_sync(input); } +static void steam_do_sensors_event(struct steam_device *steam, + struct input_dev *sensors, u8 *data) +{ + steam->sensor_timestamp_us += steam->sensor_update_rate_us; + + input_event(sensors, EV_MSC, MSC_TIMESTAMP, steam->sensor_timestamp_us); + steam_map_axes(sensors, steam_controller_imu_mappings, data); + + input_sync(sensors); +} + /* * The size for this message payload is 56. * The known values are: @@ -1594,23 +1800,68 @@ static void steam_do_input_event(struct steam_device *steam, * 15.6 | -- | unknown * 15.7 | -- | unknown */ + +static const struct steam_button_mapping steam_deck_button_mappings[] = { + { BTN_TR2, 8, 0 }, + { BTN_TL2, 8, 1 }, + { BTN_TR, 8, 2 }, + { BTN_TL, 8, 3 }, + { BTN_Y, 8, 4 }, + { BTN_B, 8, 5 }, + { BTN_X, 8, 6 }, + { BTN_A, 8, 7 }, + { BTN_SELECT, 9, 4 }, + { BTN_MODE, 9, 5 }, + { BTN_START, 9, 6 }, + { BTN_GRIPL2, 9, 7 }, + { BTN_GRIPR2, 10, 0 }, + { BTN_THUMBL, 10, 6 }, + { BTN_THUMBR, 11, 2 }, + { BTN_DPAD_UP, 9, 0 }, + { BTN_DPAD_RIGHT, 9, 1 }, + { BTN_DPAD_LEFT, 9, 2 }, + { BTN_DPAD_DOWN, 9, 3 }, + { BTN_THUMB, 10, 1 }, + { BTN_THUMB2, 10, 2 }, + { BTN_GRIPL, 13, 1 }, + { BTN_GRIPR, 13, 2 }, + { BTN_BASE, 14, 2 }, + { /* sentinel */ }, +}; + +static const struct steam_axis_mapping steam_deck_axis_mappings[] = { + { ABS_X, 1, 48 }, + { ABS_Y, -1, 50 }, + { ABS_RX, 1, 52 }, + { ABS_RY, -1, 54 }, + { ABS_HAT2Y, 1, 44 }, + { ABS_HAT2X, 1, 46 }, + { /* sentinel */ }, +}; + +static const struct steam_axis_mapping steam_deck_imu_mappings[] = { + { ABS_X, 1, 24 }, + { ABS_Z, -1, 26 }, + { ABS_Y, 1, 28 }, + { ABS_RX, 1, 30 }, + { ABS_RZ, -1, 32 }, + { ABS_RY, 1, 34 }, + { /* sentinel */ }, +}; + static void steam_do_deck_input_event(struct steam_device *steam, struct input_dev *input, u8 *data) { - u8 b8, b9, b10, b11, b13, b14; + bool start_pressed; bool lpad_touched, rpad_touched; - b8 = data[8]; - b9 = data[9]; - b10 = data[10]; - b11 = data[11]; - b13 = data[13]; - b14 = data[14]; + start_pressed = data[9] & BIT(6); - if (!(b9 & BIT(6)) && steam->did_mode_switch) { + if (!start_pressed && steam->did_mode_switch) { steam->did_mode_switch = false; cancel_delayed_work(&steam->mode_switch); - } else if (!steam->client_opened && (b9 & BIT(6)) && !steam->did_mode_switch) { + } else if (!steam->client_opened && start_pressed && !steam->did_mode_switch) { + hid_dbg(steam->hdev, "%s: doing mode switch\n", __func__); steam->did_mode_switch = true; schedule_delayed_work(&steam->mode_switch, 45 * HZ / 100); } @@ -1618,8 +1869,8 @@ static void steam_do_deck_input_event(struct steam_device *steam, if (!steam->gamepad_mode && lizard_mode) return; - lpad_touched = b10 & BIT(3); - rpad_touched = b10 & BIT(4); + lpad_touched = data[10] & BIT(3); + rpad_touched = data[10] & BIT(4); if (lpad_touched) { input_report_abs(input, ABS_HAT0X, steam_le16(data + 16)); @@ -1637,38 +1888,8 @@ static void steam_do_deck_input_event(struct steam_device *steam, input_report_abs(input, ABS_HAT1Y, 0); } - input_report_abs(input, ABS_X, steam_le16(data + 48)); - input_report_abs(input, ABS_Y, -steam_le16(data + 50)); - input_report_abs(input, ABS_RX, steam_le16(data + 52)); - input_report_abs(input, ABS_RY, -steam_le16(data + 54)); - - input_report_abs(input, ABS_HAT2Y, steam_le16(data + 44)); - input_report_abs(input, ABS_HAT2X, steam_le16(data + 46)); - - input_event(input, EV_KEY, BTN_TR2, !!(b8 & BIT(0))); - input_event(input, EV_KEY, BTN_TL2, !!(b8 & BIT(1))); - input_event(input, EV_KEY, BTN_TR, !!(b8 & BIT(2))); - input_event(input, EV_KEY, BTN_TL, !!(b8 & BIT(3))); - input_event(input, EV_KEY, BTN_Y, !!(b8 & BIT(4))); - input_event(input, EV_KEY, BTN_B, !!(b8 & BIT(5))); - input_event(input, EV_KEY, BTN_X, !!(b8 & BIT(6))); - input_event(input, EV_KEY, BTN_A, !!(b8 & BIT(7))); - input_event(input, EV_KEY, BTN_SELECT, !!(b9 & BIT(4))); - input_event(input, EV_KEY, BTN_MODE, !!(b9 & BIT(5))); - input_event(input, EV_KEY, BTN_START, !!(b9 & BIT(6))); - input_event(input, EV_KEY, BTN_GRIPL2, !!(b9 & BIT(7))); - input_event(input, EV_KEY, BTN_GRIPR2, !!(b10 & BIT(0))); - input_event(input, EV_KEY, BTN_THUMBL, !!(b10 & BIT(6))); - input_event(input, EV_KEY, BTN_THUMBR, !!(b11 & BIT(2))); - input_event(input, EV_KEY, BTN_DPAD_UP, !!(b9 & BIT(0))); - input_event(input, EV_KEY, BTN_DPAD_RIGHT, !!(b9 & BIT(1))); - input_event(input, EV_KEY, BTN_DPAD_LEFT, !!(b9 & BIT(2))); - input_event(input, EV_KEY, BTN_DPAD_DOWN, !!(b9 & BIT(3))); - input_event(input, EV_KEY, BTN_THUMB, !!(b10 & BIT(1))); - input_event(input, EV_KEY, BTN_THUMB2, !!(b10 & BIT(2))); - input_event(input, EV_KEY, BTN_GRIPL, !!(b13 & BIT(1))); - input_event(input, EV_KEY, BTN_GRIPR, !!(b13 & BIT(2))); - input_event(input, EV_KEY, BTN_BASE, !!(b14 & BIT(2))); + steam_map_buttons(input, steam_deck_button_mappings, data); + steam_map_axes(input, steam_deck_axis_mappings, data); input_sync(input); } @@ -1676,25 +1897,13 @@ static void steam_do_deck_input_event(struct steam_device *steam, static void steam_do_deck_sensors_event(struct steam_device *steam, struct input_dev *sensors, u8 *data) { - /* - * The deck input report is received every 4 ms on average, - * with a jitter of +/- 4 ms even though the USB descriptor claims - * that it uses 1 kHz. - * Since the HID report does not include a sensor timestamp, - * use a fixed increment here. - */ - steam->sensor_timestamp_us += 4000; + steam->sensor_timestamp_us += steam->sensor_update_rate_us; if (!steam->gamepad_mode && lizard_mode) return; input_event(sensors, EV_MSC, MSC_TIMESTAMP, steam->sensor_timestamp_us); - input_report_abs(sensors, ABS_X, steam_le16(data + 24)); - input_report_abs(sensors, ABS_Z, -steam_le16(data + 26)); - input_report_abs(sensors, ABS_Y, steam_le16(data + 28)); - input_report_abs(sensors, ABS_RX, steam_le16(data + 30)); - input_report_abs(sensors, ABS_RZ, -steam_le16(data + 32)); - input_report_abs(sensors, ABS_RY, steam_le16(data + 34)); + steam_map_axes(sensors, steam_deck_imu_mappings, data); input_sync(sensors); } @@ -1773,6 +1982,9 @@ static int steam_raw_event(struct hid_device *hdev, input = rcu_dereference(steam->input); if (likely(input)) steam_do_input_event(steam, input, data); + sensors = rcu_dereference(steam->sensors); + if (likely(sensors)) + steam_do_sensors_event(steam, sensors, data); rcu_read_unlock(); break; case ID_CONTROLLER_DECK_STATE: diff --git a/drivers/hid/i2c-hid/Makefile b/drivers/hid/i2c-hid/Makefile index 55bd5e0f35af..38d5d827f3ce 100644 --- a/drivers/hid/i2c-hid/Makefile +++ b/drivers/hid/i2c-hid/Makefile @@ -8,7 +8,7 @@ obj-$(CONFIG_I2C_HID_CORE) += i2c-hid.o i2c-hid-objs = i2c-hid-core.o i2c-hid-$(CONFIG_DMI) += i2c-hid-dmi-quirks.o -obj-$(CONFIG_I2C_HID_ACPI) += i2c-hid-acpi.o +obj-$(CONFIG_I2C_HID_ACPI) += i2c-hid-acpi.o i2c-hid-acpi-prp0001.o obj-$(CONFIG_I2C_HID_OF) += i2c-hid-of.o obj-$(CONFIG_I2C_HID_OF_ELAN) += i2c-hid-of-elan.o obj-$(CONFIG_I2C_HID_OF_GOODIX) += i2c-hid-of-goodix.o diff --git a/drivers/hid/i2c-hid/i2c-hid-acpi-prp0001.c b/drivers/hid/i2c-hid/i2c-hid-acpi-prp0001.c new file mode 100644 index 000000000000..d2cf4714ae7f --- /dev/null +++ b/drivers/hid/i2c-hid/i2c-hid-acpi-prp0001.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * HID over I2C driver for PRP0001 devices missing hid-descr-addr + * + * Some devices, for example the Lenovo KaiTian N60d and Inspur CP300L3, use + * _HID "PRP0001" with _DSD compatible "hid-over-i2c" but lack "hid-descr-addr" + * from the _DSD. The HID descriptor address is provided only through an ACPI + * _DSM. The TPD0 node in the DSDT shows _DSM Function 1 returning 0x20. + * + * Copyright (C) 2026 谢致邦 (XIE Zhibang) + */ + +#include +#include +#include +#include +#include + +#include "i2c-hid.h" +#include "i2c-hid-acpi.h" + +static int i2c_hid_acpi_prp0001_power_up(struct i2chid_ops *ops) +{ + /* give the device time to power up */ + msleep(750); + return 0; +} + +static struct i2chid_ops i2c_hid_acpi_prp0001_ops = { + .power_up = i2c_hid_acpi_prp0001_power_up, + /* + * No .restore_sequence needed: the _DSM on these devices returns a + * constant (0x20) with no side effects, unlike some PNP0C50 _DSM + * implementations that switch the hardware between PS/2 and I2C modes. + */ +}; + +static int i2c_hid_acpi_prp0001_probe(struct i2c_client *client) +{ + struct device *dev = &client->dev; + struct acpi_device *adev; + u16 hid_descriptor_address; + int ret; + + /* If hid-descr-addr is present, let i2c-hid-of handle it */ + if (device_property_present(dev, "hid-descr-addr")) + return -ENODEV; + + adev = ACPI_COMPANION(dev); + if (!adev) + return -ENODEV; + + ret = i2c_hid_acpi_get_descriptor(adev); + if (ret < 0) + return ret; + dev_warn(dev, + "hid-descr-addr device property NOT found, using ACPI _DSM fallback. Contact vendor for firmware update!\n"); + hid_descriptor_address = ret; + + /* + * No acpi_device_fix_up_power() needed: TPD0 has no _PS0, _PS3, _PSC + * or _PRx methods and follows I2C bus power. + */ + return i2c_hid_core_probe(client, &i2c_hid_acpi_prp0001_ops, + hid_descriptor_address, 0); +} + +static const struct of_device_id i2c_hid_acpi_prp0001_of_match[] = { + { .compatible = "hid-over-i2c" }, + {}, +}; +MODULE_DEVICE_TABLE(of, i2c_hid_acpi_prp0001_of_match); + +static const struct i2c_device_id i2c_hid_acpi_prp0001_id[] = { + { .name = "hid-over-i2c" }, + { } +}; +MODULE_DEVICE_TABLE(i2c, i2c_hid_acpi_prp0001_id); + +static struct i2c_driver i2c_hid_acpi_prp0001_driver = { + .driver = { + .name = "i2c_hid_acpi_prp0001", + .pm = &i2c_hid_core_pm, + .probe_type = PROBE_PREFER_ASYNCHRONOUS, + /* + * of_match_ptr() makes this NULL when CONFIG_OF=n, but that's + * fine: the I2C id_table with "hid-over-i2c" handles matching + * via client->name (set by acpi_set_modalias() from the _DSD + * compatible property). + */ + .of_match_table = of_match_ptr(i2c_hid_acpi_prp0001_of_match), + }, + + .probe = i2c_hid_acpi_prp0001_probe, + .remove = i2c_hid_core_remove, + .shutdown = i2c_hid_core_shutdown, + .id_table = i2c_hid_acpi_prp0001_id, +}; + +module_i2c_driver(i2c_hid_acpi_prp0001_driver); + +MODULE_DESCRIPTION("HID over I2C driver for PRP0001 devices missing hid-descr-addr"); +MODULE_AUTHOR("谢致邦 (XIE Zhibang) "); +MODULE_LICENSE("GPL"); diff --git a/drivers/hid/i2c-hid/i2c-hid-acpi.c b/drivers/hid/i2c-hid/i2c-hid-acpi.c index abd700a101f4..13f977d6aab6 100644 --- a/drivers/hid/i2c-hid/i2c-hid-acpi.c +++ b/drivers/hid/i2c-hid/i2c-hid-acpi.c @@ -25,9 +25,9 @@ #include #include #include -#include #include "i2c-hid.h" +#include "i2c-hid-acpi.h" struct i2c_hid_acpi { struct i2chid_ops ops; @@ -48,39 +48,11 @@ static const struct acpi_device_id i2c_hid_acpi_blacklist[] = { { } }; -/* HID I²C Device: 3cdff6f7-4267-4555-ad05-b30a3d8938de */ -static guid_t i2c_hid_guid = - GUID_INIT(0x3CDFF6F7, 0x4267, 0x4555, - 0xAD, 0x05, 0xB3, 0x0A, 0x3D, 0x89, 0x38, 0xDE); - -static int i2c_hid_acpi_get_descriptor(struct i2c_hid_acpi *ihid_acpi) -{ - struct acpi_device *adev = ihid_acpi->adev; - acpi_handle handle = acpi_device_handle(adev); - union acpi_object *obj; - u16 hid_descriptor_address; - - if (acpi_match_device_ids(adev, i2c_hid_acpi_blacklist) == 0) - return -ENODEV; - - obj = acpi_evaluate_dsm_typed(handle, &i2c_hid_guid, 1, 1, NULL, - ACPI_TYPE_INTEGER); - if (!obj) { - acpi_handle_err(handle, "Error _DSM call to get HID descriptor address failed\n"); - return -ENODEV; - } - - hid_descriptor_address = obj->integer.value; - ACPI_FREE(obj); - - return hid_descriptor_address; -} - static void i2c_hid_acpi_restore_sequence(struct i2chid_ops *ops) { struct i2c_hid_acpi *ihid_acpi = container_of(ops, struct i2c_hid_acpi, ops); - i2c_hid_acpi_get_descriptor(ihid_acpi); + i2c_hid_acpi_get_descriptor(ihid_acpi->adev); } static void i2c_hid_acpi_shutdown_tail(struct i2chid_ops *ops) @@ -93,24 +65,28 @@ static void i2c_hid_acpi_shutdown_tail(struct i2chid_ops *ops) static int i2c_hid_acpi_probe(struct i2c_client *client) { struct device *dev = &client->dev; + struct acpi_device *adev = ACPI_COMPANION(dev); struct i2c_hid_acpi *ihid_acpi; u16 hid_descriptor_address; int ret; - ihid_acpi = devm_kzalloc(&client->dev, sizeof(*ihid_acpi), GFP_KERNEL); + if (acpi_match_device_ids(adev, i2c_hid_acpi_blacklist) == 0) + return -ENODEV; + + ret = i2c_hid_acpi_get_descriptor(adev); + if (ret < 0) + return ret; + hid_descriptor_address = ret; + + ihid_acpi = devm_kzalloc(dev, sizeof(*ihid_acpi), GFP_KERNEL); if (!ihid_acpi) return -ENOMEM; - ihid_acpi->adev = ACPI_COMPANION(dev); + ihid_acpi->adev = adev; ihid_acpi->ops.shutdown_tail = i2c_hid_acpi_shutdown_tail; ihid_acpi->ops.restore_sequence = i2c_hid_acpi_restore_sequence; - ret = i2c_hid_acpi_get_descriptor(ihid_acpi); - if (ret < 0) - return ret; - hid_descriptor_address = ret; - - acpi_device_fix_up_power(ihid_acpi->adev); + acpi_device_fix_up_power(adev); return i2c_hid_core_probe(client, &ihid_acpi->ops, hid_descriptor_address, 0); diff --git a/drivers/hid/i2c-hid/i2c-hid-acpi.h b/drivers/hid/i2c-hid/i2c-hid-acpi.h new file mode 100644 index 000000000000..0bbed1853313 --- /dev/null +++ b/drivers/hid/i2c-hid/i2c-hid-acpi.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef _I2C_HID_ACPI_H +#define _I2C_HID_ACPI_H + +#include +#include + +static inline int i2c_hid_acpi_get_descriptor(struct acpi_device *adev) +{ + /* HID I²C Device: 3cdff6f7-4267-4555-ad05-b30a3d8938de */ + static const guid_t i2c_hid_guid = + GUID_INIT(0x3CDFF6F7, 0x4267, 0x4555, + 0xAD, 0x05, 0xB3, 0x0A, 0x3D, 0x89, 0x38, 0xDE); + + acpi_handle handle = acpi_device_handle(adev); + union acpi_object *obj; + u16 addr; + + obj = acpi_evaluate_dsm_typed(handle, &i2c_hid_guid, + 1, 1, NULL, ACPI_TYPE_INTEGER); + if (!obj) { + acpi_handle_err(handle, + "Error _DSM call to get HID descriptor address failed\n"); + return -ENODEV; + } + + addr = obj->integer.value; + ACPI_FREE(obj); + return addr; +} + +#endif diff --git a/drivers/hid/i2c-hid/i2c-hid-core.c b/drivers/hid/i2c-hid/i2c-hid-core.c index e0a302544cef..6d407cbed188 100644 --- a/drivers/hid/i2c-hid/i2c-hid-core.c +++ b/drivers/hid/i2c-hid/i2c-hid-core.c @@ -790,7 +790,7 @@ static int i2c_hid_parse(struct hid_device *hid) ihid->hdesc.wReportDescRegister, rdesc, rsize); if (ret) { - hid_err(hid, "reading report descriptor failed\n"); + dev_err(&client->dev, "reading report descriptor failed\n"); goto out; } } diff --git a/drivers/hid/i2c-hid/i2c-hid-of-goodix.c b/drivers/hid/i2c-hid/i2c-hid-of-goodix.c index f1597ad67e7c..f4dbcd1d1d47 100644 --- a/drivers/hid/i2c-hid/i2c-hid-of-goodix.c +++ b/drivers/hid/i2c-hid/i2c-hid-of-goodix.c @@ -51,8 +51,10 @@ static int goodix_i2c_hid_power_up(struct i2chid_ops *ops) return ret; ret = regulator_enable(ihid_goodix->vddio); - if (ret) + if (ret) { + regulator_disable(ihid_goodix->vdd); return ret; + } if (ihid_goodix->timings->post_power_delay_ms) msleep(ihid_goodix->timings->post_power_delay_ms); diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index 4f1f5f294a73..b1552a26b8d3 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -2895,7 +2895,7 @@ static void hv_crash_handler(struct pt_regs *regs) hv_synic_disable_regs(cpu); }; -static int hv_synic_suspend(void) +static int hv_synic_suspend(void *data) { /* * When we reach here, all the non-boot CPUs have been offlined. @@ -2922,7 +2922,7 @@ static int hv_synic_suspend(void) return 0; } -static void hv_synic_resume(void) +static void hv_synic_resume(void *data) { hv_synic_enable_regs(0); @@ -2934,11 +2934,15 @@ static void hv_synic_resume(void) } /* The callbacks run only on CPU0, with irqs_disabled. */ -static struct syscore_ops hv_synic_syscore_ops = { +static const struct syscore_ops hv_synic_syscore_ops = { .suspend = hv_synic_suspend, .resume = hv_synic_resume, }; +static struct syscore hv_synic_syscore = { + .ops = &hv_synic_syscore_ops, +}; + static int __init hv_acpi_init(void) { int ret; @@ -2947,6 +2951,13 @@ static int __init hv_acpi_init(void) return -ENODEV; if (hv_root_partition() && !hv_nested) + /* + * A non-nested root partition does not need VMBus client + * functionality. However, the mshv_root module may have + * a dependency on the VMBus module as described in + * commit 840b740a35bf. Return success so the module + * loads even though no VMBus initialization is done. + */ return 0; /* @@ -2981,7 +2992,7 @@ static int __init hv_acpi_init(void) hv_setup_kexec_handler(hv_kexec_handler); hv_setup_crash_handler(hv_crash_handler); - register_syscore_ops(&hv_synic_syscore_ops); + register_syscore(&hv_synic_syscore); return 0; @@ -2995,7 +3006,15 @@ static void __exit vmbus_exit(void) { int cpu; - unregister_syscore_ops(&hv_synic_syscore_ops); + if (hv_root_partition() && !hv_nested) + /* + * If a non-nested root partition loaded the VMBus module, + * hv_acpi_init() did not do any VMBus initialization. + * There's nothing to clean up, so just return. + */ + return; + + unregister_syscore(&hv_synic_syscore); hv_remove_kexec_handler(); hv_remove_crash_handler(); diff --git a/drivers/hwmon/coretemp.c b/drivers/hwmon/coretemp.c index ad79db5a183e..a84a32ada8bd 100644 --- a/drivers/hwmon/coretemp.c +++ b/drivers/hwmon/coretemp.c @@ -650,6 +650,7 @@ static void coretemp_device_remove(int zoneid) struct platform_data *pdata = platform_get_drvdata(pdev); ida_destroy(&pdata->ida); + kfree(pdata->core_data); kfree(pdata); platform_device_unregister(pdev); } diff --git a/drivers/hwmon/cros_ec_hwmon.c b/drivers/hwmon/cros_ec_hwmon.c index 48331703f2f5..f39401606e75 100644 --- a/drivers/hwmon/cros_ec_hwmon.c +++ b/drivers/hwmon/cros_ec_hwmon.c @@ -5,6 +5,7 @@ * Copyright (C) 2024 Thomas Weißschuh */ +#include #include #include #include @@ -25,9 +26,11 @@ struct cros_ec_hwmon_priv { struct cros_ec_device *cros_ec; + struct device *hwmon_dev; const char *temp_sensor_names[EC_TEMP_SENSOR_ENTRIES + EC_TEMP_SENSOR_B_ENTRIES]; u8 usable_fans; bool fan_control_supported; + bool temp_threshold_supported; u8 manual_fans; /* bits to indicate whether the fan is set to manual */ u8 manual_fan_pwm[EC_FAN_SPEED_ENTRIES]; }; @@ -102,6 +105,23 @@ static int cros_ec_hwmon_read_temp(struct cros_ec_device *cros_ec, u8 index, u8 return 0; } +static int cros_ec_hwmon_read_temp_threshold(struct cros_ec_device *cros_ec, u8 index, + enum ec_temp_thresholds threshold, u32 *temp) +{ + struct ec_params_thermal_get_threshold_v1 req = {}; + struct ec_thermal_config resp; + int ret; + + req.sensor_num = index; + ret = cros_ec_cmd(cros_ec, 1, EC_CMD_THERMAL_GET_THRESHOLD, + &req, sizeof(req), &resp, sizeof(resp)); + if (ret < 0) + return ret; + + *temp = resp.temp_host[threshold]; + return 0; +} + static bool cros_ec_hwmon_is_error_fan(u16 speed) { return speed == EC_FAN_SPEED_NOT_PRESENT || speed == EC_FAN_SPEED_STALLED; @@ -120,12 +140,29 @@ static long cros_ec_hwmon_temp_to_millicelsius(u8 temp) return kelvin_to_millicelsius((((long)temp) + EC_TEMP_SENSOR_OFFSET)); } +static bool cros_ec_hwmon_attr_is_temp_threshold(u32 attr) +{ + return attr == hwmon_temp_max || + attr == hwmon_temp_crit || + attr == hwmon_temp_emergency; +} + +static enum ec_temp_thresholds cros_ec_hwmon_attr_to_thres(u32 attr) +{ + if (attr == hwmon_temp_max) + return EC_TEMP_THRESH_WARN; + else if (attr == hwmon_temp_crit) + return EC_TEMP_THRESH_HIGH; + return EC_TEMP_THRESH_HALT; /* attr == hwmon_temp_emergency */ +} + static int cros_ec_hwmon_read(struct device *dev, enum hwmon_sensor_types type, u32 attr, int channel, long *val) { struct cros_ec_hwmon_priv *priv = dev_get_drvdata(dev); int ret = -EOPNOTSUPP; u8 control_method; + u32 threshold; u8 pwm_value; u16 speed; u8 temp; @@ -168,6 +205,13 @@ static int cros_ec_hwmon_read(struct device *dev, enum hwmon_sensor_types type, ret = cros_ec_hwmon_read_temp(priv->cros_ec, channel, &temp); if (ret == 0) *val = cros_ec_hwmon_is_error_temp(temp); + + } else if (cros_ec_hwmon_attr_is_temp_threshold(attr)) { + ret = cros_ec_hwmon_read_temp_threshold(priv->cros_ec, channel, + cros_ec_hwmon_attr_to_thres(attr), + &threshold); + if (ret == 0) + *val = kelvin_to_millicelsius(threshold); } } @@ -267,8 +311,14 @@ static umode_t cros_ec_hwmon_is_visible(const void *data, enum hwmon_sensor_type if (priv->fan_control_supported && priv->usable_fans & BIT(channel)) return 0644; } else if (type == hwmon_temp) { - if (priv->temp_sensor_names[channel]) - return 0444; + if (priv->temp_sensor_names[channel]) { + if (cros_ec_hwmon_attr_is_temp_threshold(attr)) { + if (priv->temp_threshold_supported) + return 0444; + } else { + return 0444; + } + } } return 0; @@ -286,31 +336,33 @@ static const struct hwmon_channel_info * const cros_ec_hwmon_info[] = { HWMON_PWM_INPUT | HWMON_PWM_ENABLE, HWMON_PWM_INPUT | HWMON_PWM_ENABLE, HWMON_PWM_INPUT | HWMON_PWM_ENABLE), +#define CROS_EC_HWMON_TEMP_PARAMS (HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL | \ + HWMON_T_MAX | HWMON_T_CRIT | HWMON_T_EMERGENCY) HWMON_CHANNEL_INFO(temp, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL, - HWMON_T_INPUT | HWMON_T_FAULT | HWMON_T_LABEL), + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS, + CROS_EC_HWMON_TEMP_PARAMS), NULL }; @@ -328,6 +380,8 @@ static int cros_ec_hwmon_cooling_get_cur_state(struct thermal_cooling_device *cd u8 read_val; int ret; + guard(hwmon_lock)(priv->hwmon_priv->hwmon_dev); + ret = cros_ec_hwmon_read_pwm_value(priv->hwmon_priv->cros_ec, priv->index, &read_val); if (ret) return ret; @@ -341,6 +395,8 @@ static int cros_ec_hwmon_cooling_set_cur_state(struct thermal_cooling_device *cd { const struct cros_ec_hwmon_cooling_priv *priv = cdev->devdata; + guard(hwmon_lock)(priv->hwmon_priv->hwmon_dev); + return cros_ec_hwmon_write_pwm_input(priv->hwmon_priv->cros_ec, priv->index, val); } @@ -474,7 +530,6 @@ static int cros_ec_hwmon_probe(struct platform_device *pdev) struct cros_ec_dev *ec_dev = dev_get_drvdata(dev->parent); struct cros_ec_device *cros_ec = ec_dev->ec_dev; struct cros_ec_hwmon_priv *priv; - struct device *hwmon_dev; u8 thermal_version; int ret; @@ -495,13 +550,19 @@ static int cros_ec_hwmon_probe(struct platform_device *pdev) cros_ec_hwmon_probe_temp_sensors(dev, priv, thermal_version); cros_ec_hwmon_probe_fans(priv); priv->fan_control_supported = cros_ec_hwmon_probe_fan_control_supported(priv->cros_ec); + priv->temp_threshold_supported = is_cros_ec_cmd_available(priv->cros_ec, + EC_CMD_THERMAL_GET_THRESHOLD, 1); + + priv->hwmon_dev = devm_hwmon_device_register_with_info(dev, "cros_ec", priv, + &cros_ec_hwmon_chip_info, NULL); + if (IS_ERR(priv->hwmon_dev)) + return PTR_ERR(priv->hwmon_dev); + cros_ec_hwmon_register_fan_cooling_devices(dev, priv); - hwmon_dev = devm_hwmon_device_register_with_info(dev, "cros_ec", priv, - &cros_ec_hwmon_chip_info, NULL); platform_set_drvdata(pdev, priv); - return PTR_ERR_OR_ZERO(hwmon_dev); + return 0; } static int cros_ec_hwmon_suspend(struct platform_device *pdev, pm_message_t state) diff --git a/drivers/hwmon/emc1403.c b/drivers/hwmon/emc1403.c index eca33220d34a..39f69adb88c2 100644 --- a/drivers/hwmon/emc1403.c +++ b/drivers/hwmon/emc1403.c @@ -17,7 +17,6 @@ #include #include #include -#include #include #include @@ -30,7 +29,6 @@ enum emc1403_chip { emc1402, emc1403, emc1404, emc1428 }; struct thermal_data { enum emc1403_chip chip; struct regmap *regmap; - struct mutex mutex; }; static ssize_t power_state_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -268,8 +266,8 @@ static s8 emc1403_temp_regs_low[][4] = { }, }; -static int __emc1403_get_temp(struct thermal_data *data, int channel, - enum emc1403_reg_map map, long *val) +static int emc1403_get_temp(struct thermal_data *data, int channel, + enum emc1403_reg_map map, long *val) { unsigned int regvalh; unsigned int regvall = 0; @@ -295,38 +293,22 @@ static int __emc1403_get_temp(struct thermal_data *data, int channel, return 0; } -static int emc1403_get_temp(struct thermal_data *data, int channel, - enum emc1403_reg_map map, long *val) -{ - int ret; - - mutex_lock(&data->mutex); - ret = __emc1403_get_temp(data, channel, map, val); - mutex_unlock(&data->mutex); - - return ret; -} - static int emc1403_get_hyst(struct thermal_data *data, int channel, enum emc1403_reg_map map, long *val) { int hyst, ret; long limit; - mutex_lock(&data->mutex); - ret = __emc1403_get_temp(data, channel, map, &limit); + ret = emc1403_get_temp(data, channel, map, &limit); if (ret < 0) - goto unlock; + return ret; ret = regmap_read(data->regmap, 0x21, &hyst); if (ret < 0) - goto unlock; - if (map == temp_min) - *val = limit + hyst * 1000; - else - *val = limit - hyst * 1000; -unlock: - mutex_unlock(&data->mutex); - return ret; + return ret; + + *val = limit - hyst * 1000; + + return 0; } static int emc1403_temp_read(struct thermal_data *data, u32 attr, int channel, long *val) @@ -341,9 +323,6 @@ static int emc1403_temp_read(struct thermal_data *data, u32 attr, int channel, l case hwmon_temp_input: ret = emc1403_get_temp(data, channel, ema1403_temp_map[attr], val); break; - case hwmon_temp_min_hyst: - ret = emc1403_get_hyst(data, channel, temp_min, val); - break; case hwmon_temp_max_hyst: ret = emc1403_get_hyst(data, channel, temp_max, val); break; @@ -451,20 +430,16 @@ static int emc1403_set_hyst(struct thermal_data *data, long val) else val = clamp_val(val, 0, 255000); - mutex_lock(&data->mutex); - ret = __emc1403_get_temp(data, 0, temp_crit, &limit); + ret = emc1403_get_temp(data, 0, temp_crit, &limit); if (ret < 0) - goto unlock; + return ret; hyst = limit - val; if (data->chip == emc1428) hyst = clamp_val(DIV_ROUND_CLOSEST(hyst, 1000), 0, 127); else hyst = clamp_val(DIV_ROUND_CLOSEST(hyst, 1000), 0, 255); - ret = regmap_write(data->regmap, 0x21, hyst); -unlock: - mutex_unlock(&data->mutex); - return ret; + return regmap_write(data->regmap, 0x21, hyst); } static int emc1403_set_temp(struct thermal_data *data, int channel, @@ -478,7 +453,6 @@ static int emc1403_set_temp(struct thermal_data *data, int channel, regh = emc1403_temp_regs[channel][map]; regl = emc1403_temp_regs_low[channel][map]; - mutex_lock(&data->mutex); if (regl >= 0) { if (data->chip == emc1428) val = clamp_val(val, -128000, 127875); @@ -487,7 +461,7 @@ static int emc1403_set_temp(struct thermal_data *data, int channel, regval = DIV_ROUND_CLOSEST(val, 125); ret = regmap_write(data->regmap, regh, (regval >> 3) & 0xff); if (ret < 0) - goto unlock; + return ret; ret = regmap_write(data->regmap, regl, (regval & 0x07) << 5); } else { if (data->chip == emc1428) @@ -497,8 +471,6 @@ static int emc1403_set_temp(struct thermal_data *data, int channel, regval = DIV_ROUND_CLOSEST(val, 1000); ret = regmap_write(data->regmap, regh, regval); } -unlock: - mutex_unlock(&data->mutex); return ret; } @@ -572,7 +544,6 @@ static umode_t emc1403_temp_is_visible(const void *_data, u32 attr, int channel) case hwmon_temp_max_alarm: case hwmon_temp_crit_alarm: case hwmon_temp_fault: - case hwmon_temp_min_hyst: case hwmon_temp_max_hyst: return 0444; case hwmon_temp_min: @@ -615,35 +586,35 @@ static const struct hwmon_channel_info * const emc1403_info[] = { HWMON_CHANNEL_INFO(chip, HWMON_C_UPDATE_INTERVAL), HWMON_CHANNEL_INFO(temp, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT, HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX | - HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST | + HWMON_T_CRIT | HWMON_T_MAX_HYST | HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM | HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT ), @@ -695,8 +666,6 @@ static int emc1403_probe(struct i2c_client *client) if (IS_ERR(data->regmap)) return PTR_ERR(data->regmap); - mutex_init(&data->mutex); - hwmon_dev = devm_hwmon_device_register_with_info(&client->dev, client->name, data, &emc1403_chip_info, diff --git a/drivers/hwspinlock/hwspinlock_core.c b/drivers/hwspinlock/hwspinlock_core.c index cc8e952a6772..a509b73da190 100644 --- a/drivers/hwspinlock/hwspinlock_core.c +++ b/drivers/hwspinlock/hwspinlock_core.c @@ -472,7 +472,7 @@ static int hwspin_lock_register_single(struct hwspinlock *hwlock, int id) out: mutex_unlock(&hwspinlock_tree_lock); - return 0; + return ret; } static struct hwspinlock *hwspin_lock_unregister_single(unsigned int id) diff --git a/drivers/hwtracing/coresight/coresight-etm4x-cfg.c b/drivers/hwtracing/coresight/coresight-etm4x-cfg.c index c302072b293a..e1a59b434505 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-cfg.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-cfg.c @@ -76,7 +76,7 @@ static int etm4_cfg_map_reg_offset(struct etmv4_drvdata *drvdata, } else if ((offset & GENMASK(11, 4)) == TRCSEQEVRn(0)) { /* sequencer state control registers */ idx = (offset & GENMASK(3, 0)) / 4; - if (idx < ETM_MAX_SEQ_STATES) { + if (idx < ETM_MAX_SEQ_TRANSITIONS) { reg_csdev->driver_regval = &drvcfg->seq_ctrl[idx]; err = 0; } diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index 2271f5ea8121..522e12817446 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -92,7 +92,7 @@ static int etm4_probe_cpu(unsigned int cpu); static bool etm4x_sspcicrn_present(struct etmv4_drvdata *drvdata, int n) { return (n < drvdata->nr_ss_cmp) && - drvdata->nr_pe && + drvdata->nr_pe_cmp && (drvdata->config.ss_status[n] & TRCSSCSRn_PC); } @@ -541,7 +541,8 @@ static int etm4_enable_hw(struct etmv4_drvdata *drvdata) etm4x_relaxed_write32(csa, config->vissctlr, TRCVISSCTLR); if (drvdata->nr_pe_cmp) etm4x_relaxed_write32(csa, config->vipcssctlr, TRCVIPCSSCTLR); - for (i = 0; i < drvdata->nrseqstate - 1; i++) + + for (i = 0; i < drvdata->nr_seq_ctrls; i++) etm4x_relaxed_write32(csa, config->seq_ctrl[i], TRCSEQEVRn(i)); if (drvdata->nrseqstate) { etm4x_relaxed_write32(csa, config->seq_rst, TRCSEQRSTEVR); @@ -640,18 +641,33 @@ static void etm4_enable_sysfs_smp_call(void *info) * TRCRSCTLR1 (always true) used to get the counter to decrement. From * there a resource selector is configured with the counter and the * timestamp control register to use the resource selector to trigger the - * event that will insert a timestamp packet in the stream. + * event that will insert a timestamp packet in the stream: + * + * +--------------+ + * | Resource 1 | fixed "always-true" resource + * +--------------+ + * | + * +------v-------+ + * | Counter x | (reload to 1 on underflow) + * +--------------+ + * | + * +------v--------------+ + * | Resource Selector y | (trigger on counter x == 0) + * +---------------------+ + * | + * +------v---------------+ + * | Timestamp Generator | (timestamp on resource y) + * +----------------------+ */ static int etm4_config_timestamp_event(struct etmv4_drvdata *drvdata) { - int ctridx, ret = -EINVAL; - int counter, rselector; - u32 val = 0; + int ctridx; + int rselector; struct etmv4_config *config = &drvdata->config; /* No point in trying if we don't have at least one counter */ if (!drvdata->nr_cntr) - goto out; + return -EINVAL; /* Find a counter that hasn't been initialised */ for (ctridx = 0; ctridx < drvdata->nr_cntr; ctridx++) @@ -661,15 +677,19 @@ static int etm4_config_timestamp_event(struct etmv4_drvdata *drvdata) /* All the counters have been configured already, bail out */ if (ctridx == drvdata->nr_cntr) { pr_debug("%s: no available counter found\n", __func__); - ret = -ENOSPC; - goto out; + return -ENOSPC; } /* - * Searching for an available resource selector to use, starting at - * '2' since every implementation has at least 2 resource selector. - * ETMIDR4 gives the number of resource selector _pairs_, - * hence multiply by 2. + * Searching for an available resource selector to use, starting at '2' + * since resource 0 is the fixed 'always returns false' resource and 1 + * is the fixed 'always returns true' resource. See IHI0064H_b '7.3.64 + * TRCRSCTLRn, Resource Selection Control Registers, n=2-31'. If there + * are no resources, there would also be no counters so wouldn't get + * here. + * + * ETMIDR4 gives the number of resource selector _pairs_, hence multiply + * by 2. */ for (rselector = 2; rselector < drvdata->nr_resource * 2; rselector++) if (!config->res_ctrl[rselector]) @@ -678,13 +698,9 @@ static int etm4_config_timestamp_event(struct etmv4_drvdata *drvdata) if (rselector == drvdata->nr_resource * 2) { pr_debug("%s: no available resource selector found\n", __func__); - ret = -ENOSPC; - goto out; + return -ENOSPC; } - /* Remember what counter we used */ - counter = 1 << ctridx; - /* * Initialise original and reload counter value to the smallest * possible value in order to get as much precision as we can. @@ -692,26 +708,41 @@ static int etm4_config_timestamp_event(struct etmv4_drvdata *drvdata) config->cntr_val[ctridx] = 1; config->cntrldvr[ctridx] = 1; - /* Set the trace counter control register */ - val = 0x1 << 16 | /* Bit 16, reload counter automatically */ - 0x0 << 7 | /* Select single resource selector */ - 0x1; /* Resource selector 1, i.e always true */ - - config->cntr_ctrl[ctridx] = val; - - val = 0x2 << 16 | /* Group 0b0010 - Counter and sequencers */ - counter << 0; /* Counter to use */ - - config->res_ctrl[rselector] = val; + /* + * Trace Counter Control Register TRCCNTCTLRn + * + * CNTCHAIN = 0, don't reload on the previous counter + * RLDSELF = true, reload counter automatically on underflow + * RLDEVENT = RES_SEL_FALSE (0), reload on single false resource (never reload) + * CNTEVENT = RES_SEL_TRUE (1), count single fixed 'always true' resource (always decrement) + */ + config->cntr_ctrl[ctridx] = TRCCNTCTLRn_RLDSELF | + FIELD_PREP(TRCCNTCTLRn_RLDEVENT_MASK, + etm4_res_sel_single(ETM4_RES_SEL_FALSE)) | + FIELD_PREP(TRCCNTCTLRn_CNTEVENT_MASK, + etm4_res_sel_single(ETM4_RES_SEL_TRUE)); - val = 0x0 << 7 | /* Select single resource selector */ - rselector; /* Resource selector */ + /* + * Resource Selection Control Register TRCRSCTLRn + * + * PAIRINV = 0, INV = 0, don't invert + * GROUP = 2, SELECT = ctridx, trigger when counter 'ctridx' reaches 0 + * + * Multiple counters can be selected, and each bit signifies a counter, + * so set bit 'ctridx' to select our counter. + */ + config->res_ctrl[rselector] = FIELD_PREP(TRCRSCTLRn_GROUP_MASK, 2) | + FIELD_PREP(TRCRSCTLRn_SELECT_MASK, 1 << ctridx); - config->ts_ctrl = val; + /* + * Global Timestamp Control Register TRCTSCTLR + * + * EVENT = generate timestamp on single resource 'rselector' + */ + config->ts_ctrl = FIELD_PREP(TRCTSCTLR_EVENT_MASK, + etm4_res_sel_single(rselector)); - ret = 0; -out: - return ret; + return 0; } static int etm4_parse_event_config(struct coresight_device *csdev, @@ -879,8 +910,10 @@ static int etm4_enable_sysfs(struct coresight_device *csdev, struct coresight_pa cscfg_config_sysfs_get_active_cfg(&cfg_hash, &preset); if (cfg_hash) { ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset); - if (ret) + if (ret) { + etm4_release_trace_id(drvdata); return ret; + } } raw_spin_lock(&drvdata->spinlock); @@ -1476,6 +1509,8 @@ static void etm4_init_arch_data(void *info) drvdata->lpoverride = (etmidr5 & TRCIDR5_LPOVERRIDE) && (!drvdata->skip_power_up); /* NUMSEQSTATE, bits[27:25] number of sequencer states implemented */ drvdata->nrseqstate = FIELD_GET(TRCIDR5_NUMSEQSTATE_MASK, etmidr5); + if (drvdata->nrseqstate) + drvdata->nr_seq_ctrls = ETM_MAX_SEQ_TRANSITIONS; /* NUMCNTR, bits[30:28] number of counters available for tracing */ drvdata->nr_cntr = FIELD_GET(TRCIDR5_NUMCNTR_MASK, etmidr5); @@ -1889,7 +1924,7 @@ static int __etm4_cpu_save(struct etmv4_drvdata *drvdata) if (drvdata->nr_pe_cmp) state->trcvipcssctlr = etm4x_read32(csa, TRCVIPCSSCTLR); - for (i = 0; i < drvdata->nrseqstate - 1; i++) + for (i = 0; i < drvdata->nr_seq_ctrls; i++) state->trcseqevr[i] = etm4x_read32(csa, TRCSEQEVRn(i)); if (drvdata->nrseqstate) { @@ -2020,7 +2055,7 @@ static void __etm4_cpu_restore(struct etmv4_drvdata *drvdata) if (drvdata->nr_pe_cmp) etm4x_relaxed_write32(csa, state->trcvipcssctlr, TRCVIPCSSCTLR); - for (i = 0; i < drvdata->nrseqstate - 1; i++) + for (i = 0; i < drvdata->nr_seq_ctrls; i++) etm4x_relaxed_write32(csa, state->trcseqevr[i], TRCSEQEVRn(i)); if (drvdata->nrseqstate) { diff --git a/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c b/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c index e9eeea6240d5..cc6cdd3ae29d 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-sysfs.c @@ -223,7 +223,7 @@ static ssize_t reset_store(struct device *dev, config->vipcssctlr = 0x0; /* Disable seq events */ - for (i = 0; i < drvdata->nrseqstate-1; i++) + for (i = 0; i < drvdata->nr_seq_ctrls; i++) config->seq_ctrl[i] = 0x0; config->seq_rst = 0x0; config->seq_state = 0x0; @@ -1395,9 +1395,11 @@ static ssize_t seq_idx_store(struct device *dev, struct etmv4_drvdata *drvdata = dev_get_drvdata(dev->parent); struct etmv4_config *config = &drvdata->config; + if (!drvdata->nr_seq_ctrls) + return -ENOTSUPP; if (kstrtoul(buf, 16, &val)) return -EINVAL; - if (val >= drvdata->nrseqstate - 1) + if (val >= drvdata->nr_seq_ctrls) return -EINVAL; /* diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h index 012c52fd1933..0489496babb3 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x.h +++ b/drivers/hwtracing/coresight/coresight-etm4x.h @@ -225,6 +225,50 @@ #define TRCRSCTLRn_GROUP_MASK GENMASK(19, 16) #define TRCRSCTLRn_SELECT_MASK GENMASK(15, 0) +#define TRCCNTCTLRn_CNTCHAIN BIT(17) +#define TRCCNTCTLRn_RLDSELF BIT(16) +#define TRCCNTCTLRn_RLDEVENT_MASK GENMASK(15, 8) +#define TRCCNTCTLRn_CNTEVENT_MASK GENMASK(7, 0) + +#define TRCTSCTLR_EVENT_MASK GENMASK(7, 0) + +#define ETM4_RES_SEL_FALSE 0 /* Fixed function 'always false' resource selector */ +#define ETM4_RES_SEL_TRUE 1 /* Fixed function 'always true' resource selector */ + +#define ETM4_RES_SEL_SINGLE_MASK GENMASK(4, 0) +#define ETM4_RES_SEL_PAIR_MASK GENMASK(3, 0) +#define ETM4_RES_SEL_TYPE_PAIR BIT(7) + +/* + * Utilities for programming EVENT resource selectors, e.g. TRCCNTCTLRn_RLDEVENT. + * + * Resource selectors have a common format across registers: + * + * 7 6 5 4 0 + * +------+------+-------+ + * | TYPE | RES0 | SEL | + * +------+------+-------+ + * + * Where TYPE indicates whether the selector is for a single event or a pair. + * When TYPE is pair, SEL is 4 bits wide and using pair 0 is UNPREDICTABLE. + * Otherwise for single it's 5 bits wide. + */ +static inline u32 etm4_res_sel_single(u8 res_sel_idx) +{ + WARN_ON_ONCE(!FIELD_FIT(ETM4_RES_SEL_SINGLE_MASK, res_sel_idx)); + return FIELD_PREP(ETM4_RES_SEL_SINGLE_MASK, res_sel_idx); +} + +static inline u32 etm4_res_sel_pair(u8 res_sel_idx) +{ + if (__builtin_constant_p(res_sel_idx)) + BUILD_BUG_ON(res_sel_idx == 0); + WARN_ON_ONCE(!FIELD_FIT(ETM4_RES_SEL_PAIR_MASK, res_sel_idx) || + (res_sel_idx == 0)); + return FIELD_PREP(ETM4_RES_SEL_PAIR_MASK, res_sel_idx) | + ETM4_RES_SEL_TYPE_PAIR; +} + /* * System instructions to access ETM registers. * See ETMv4.4 spec ARM IHI0064F section 4.3.6 System instructions @@ -570,6 +614,7 @@ #define ETM_MAX_NR_PE 8 #define ETMv4_MAX_CNTR 4 #define ETM_MAX_SEQ_STATES 4 +#define ETM_MAX_SEQ_TRANSITIONS 3 #define ETM_MAX_EXT_INP_SEL 4 #define ETM_MAX_EXT_INP 256 #define ETM_MAX_EXT_OUT 4 @@ -824,8 +869,7 @@ struct etmv4_config { u32 eventctrl0; u32 eventctrl1; u32 stall_ctrl; - u32 ts_ctrl; - u32 syncfreq; + u32 ts_ctrl; /* TRCTSCTLR */ u32 ccctlr; u32 bb_ctrl; u32 vinst_ctrl; @@ -833,15 +877,16 @@ struct etmv4_config { u32 vissctlr; u32 vipcssctlr; u8 seq_idx; - u32 seq_ctrl[ETM_MAX_SEQ_STATES]; + u8 syncfreq; + u32 seq_ctrl[ETM_MAX_SEQ_TRANSITIONS]; u32 seq_rst; u32 seq_state; u8 cntr_idx; - u32 cntrldvr[ETMv4_MAX_CNTR]; - u32 cntr_ctrl[ETMv4_MAX_CNTR]; - u32 cntr_val[ETMv4_MAX_CNTR]; + u32 cntrldvr[ETMv4_MAX_CNTR]; /* TRCCNTRLDVRn */ + u32 cntr_ctrl[ETMv4_MAX_CNTR]; /* TRCCNTCTLRn */ + u32 cntr_val[ETMv4_MAX_CNTR]; /* TRCCNTVRn */ u8 res_idx; - u32 res_ctrl[ETM_MAX_RES_SEL]; + u32 res_ctrl[ETM_MAX_RES_SEL]; /* TRCRSCTLRn */ u8 ss_idx; u32 ss_ctrl[ETM_MAX_SS_CMP]; u32 ss_status[ETM_MAX_SS_CMP]; @@ -884,7 +929,7 @@ struct etmv4_save_state { u32 trcvissctlr; u32 trcvipcssctlr; - u32 trcseqevr[ETM_MAX_SEQ_STATES]; + u32 trcseqevr[ETM_MAX_SEQ_TRANSITIONS]; u32 trcseqrstevr; u32 trcseqstr; u32 trcextinselr; @@ -937,6 +982,7 @@ struct etmv4_save_state { * @numcidc: Number of contextID comparators. * @numvmidc: Number of VMID comparators. * @nrseqstate: The number of sequencer states that are implemented. + * @nr_seq_ctrls: The number of sequence state transition control registers. * @nr_event: Indicates how many events the trace unit support. * @nr_resource:The number of resource selection pairs available for tracing. * @nr_ss_cmp: Number of single-shot comparator controls that are available. @@ -1002,6 +1048,7 @@ struct etmv4_drvdata { u8 numextinsel; u8 numvmidc; u8 nrseqstate; + u8 nr_seq_ctrls; u8 nr_event; u8 nr_resource; u8 nr_ss_cmp; diff --git a/drivers/i2c/busses/i2c-ocores.c b/drivers/i2c/busses/i2c-ocores.c index 0f67e57cdeff..9534e218382c 100644 --- a/drivers/i2c/busses/i2c-ocores.c +++ b/drivers/i2c/busses/i2c-ocores.c @@ -761,7 +761,11 @@ static int ocores_i2c_resume(struct device *dev) rate = clk_get_rate(i2c->clk) / 1000; if (rate) i2c->ip_clock_khz = rate; - return ocores_init(dev, i2c); + ret = ocores_init(dev, i2c); + if (ret) + clk_disable_unprepare(i2c->clk); + + return ret; } static DEFINE_NOIRQ_DEV_PM_OPS(ocores_i2c_pm, diff --git a/drivers/i2c/busses/i2c-qcom-cci.c b/drivers/i2c/busses/i2c-qcom-cci.c index 791e1a9fd1ba..776c3e36e915 100644 --- a/drivers/i2c/busses/i2c-qcom-cci.c +++ b/drivers/i2c/busses/i2c-qcom-cci.c @@ -246,7 +246,7 @@ static int cci_reset(struct cci *cci) return 0; } -static int cci_init(struct cci *cci) +static void cci_init(struct cci *cci) { u32 val = CCI_IRQ_MASK_0_I2C_M0_RD_DONE | CCI_IRQ_MASK_0_I2C_M0_Q0_REPORT | @@ -287,8 +287,6 @@ static int cci_init(struct cci *cci) val = hw->scl_stretch_en << 8 | hw->trdhld << 4 | hw->tsp; writel(val, cci->base + CCI_I2C_Mm_MISC_CTL(i)); } - - return 0; } static int cci_run_queue(struct cci *cci, u8 master, u8 queue) @@ -596,16 +594,16 @@ static int cci_probe(struct platform_device *pdev) ret = cci_reset(cci); if (ret < 0) - goto error; + goto disable_clocks; - ret = cci_init(cci); - if (ret < 0) - goto error; + cci_init(cci); pm_runtime_set_autosuspend_delay(dev, MSEC_PER_SEC); + ret = devm_pm_runtime_set_active_enabled(dev); + if (ret) + goto disable_clocks; + pm_runtime_use_autosuspend(dev); - pm_runtime_set_active(dev); - pm_runtime_enable(dev); for (i = 0; i < cci->data->num_masters; i++) { if (!cci->master[i].cci) @@ -621,8 +619,6 @@ static int cci_probe(struct platform_device *pdev) return 0; error_i2c: - pm_runtime_disable(dev); - pm_runtime_dont_use_autosuspend(dev); for (--i ; i >= 0; i--) { if (cci->master[i].cci) { @@ -630,8 +626,6 @@ static int cci_probe(struct platform_device *pdev) of_node_put(cci->master[i].adap.dev.of_node); } } -error: - disable_irq(cci->irq); disable_clocks: cci_disable_clocks(cci); @@ -650,10 +644,6 @@ static void cci_remove(struct platform_device *pdev) cci_halt(cci, i); } } - - disable_irq(cci->irq); - pm_runtime_disable(&pdev->dev); - pm_runtime_set_suspended(&pdev->dev); } static const struct cci_data cci_v1_data = { diff --git a/drivers/i3c/master.c b/drivers/i3c/master.c index 41452a27262d..c8ee43b70d5e 100644 --- a/drivers/i3c/master.c +++ b/drivers/i3c/master.c @@ -799,6 +799,11 @@ static struct attribute *i3c_masterdev_attrs[] = { }; ATTRIBUTE_GROUPS(i3c_masterdev); +static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev) +{ + kfree(dev); +} + static void i3c_masterdev_release(struct device *dev) { struct i3c_master_controller *master = dev_to_i3cmaster(dev); @@ -811,6 +816,8 @@ static void i3c_masterdev_release(struct device *dev) i3c_bus_cleanup(bus); of_node_put(dev->of_node); + + i3c_master_free_i3c_dev(master->this); } static const struct device_type i3c_masterdev_type = { @@ -1023,11 +1030,6 @@ static void i3c_device_release(struct device *dev) kfree(i3cdev); } -static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev) -{ - kfree(dev); -} - static struct i3c_dev_desc * i3c_master_alloc_i3c_dev(struct i3c_master_controller *master, const struct i3c_device_info *info) @@ -1787,12 +1789,21 @@ static int i3c_master_early_i3c_dev_add(struct i3c_master_controller *master, static void i3c_master_register_new_i3c_devs(struct i3c_master_controller *master) { + struct i3c_device *i3cdev, *tmp; struct i3c_dev_desc *desc; + LIST_HEAD(i3c_unreg_devs); int ret; if (!master->init_done) return; + i3c_bus_maintenance_lock(&master->bus); + + if (master->shutting_down) { + i3c_bus_maintenance_unlock(&master->bus); + return; + } + i3c_bus_for_each_i3cdev(&master->bus, desc) { if (desc->dev || !desc->info.dyn_addr || desc == master->this) continue; @@ -1813,23 +1824,37 @@ i3c_master_register_new_i3c_devs(struct i3c_master_controller *master) if (desc->boardinfo) desc->dev->dev.of_node = desc->boardinfo->of_node; - ret = device_register(&desc->dev->dev); - if (ret) { - dev_err(&master->dev, - "Failed to add I3C device (err = %d)\n", ret); - put_device(&desc->dev->dev); - } + list_add_tail(&desc->dev->node, &i3c_unreg_devs); } + + i3c_bus_maintenance_unlock(&master->bus); + + list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) { + ret = device_register(&i3cdev->dev); + if (ret) + dev_err(&master->dev, "Failed to add I3C device (err = %d)\n", ret); + else + list_del_init(&i3cdev->node); + } + + i3c_bus_maintenance_lock(&master->bus); + + list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) { + list_del(&i3cdev->node); + desc = i3cdev->desc; + i3cdev->desc = NULL; + put_device(&i3cdev->dev); + desc->dev = NULL; + } + + i3c_bus_maintenance_unlock(&master->bus); } static void i3c_master_reg_work_fn(struct work_struct *work) { struct i3c_master_controller *master = container_of(work, typeof(*master), reg_work); - i3c_bus_normaluse_lock(&master->bus); - if (!master->shutting_down) - i3c_master_register_new_i3c_devs(master); - i3c_bus_normaluse_unlock(&master->bus); + i3c_master_register_new_i3c_devs(master); } /** @@ -2023,6 +2048,8 @@ int i3c_master_set_info(struct i3c_master_controller *master, return 0; err_free_dev: + master->bus.cur_master = NULL; + master->this = NULL; i3c_master_free_i3c_dev(i3cdev); return ret; @@ -2043,7 +2070,8 @@ static void i3c_master_detach_free_devs(struct i3c_master_controller *master) i3cdev->boardinfo->init_dyn_addr, I3C_ADDR_SLOT_FREE); - i3c_master_free_i3c_dev(i3cdev); + if (i3cdev != master->this) + i3c_master_free_i3c_dev(i3cdev); } list_for_each_entry_safe(i2cdev, i2ctmp, &master->bus.devs.i2c, @@ -2272,7 +2300,8 @@ i3c_master_search_i3c_dev_duplicate(struct i3c_dev_desc *refdev) struct i3c_dev_desc *i3cdev; i3c_bus_for_each_i3cdev(&master->bus, i3cdev) { - if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid) + if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid && + i3cdev != master->this) return i3cdev; } diff --git a/drivers/i3c/master/adi-i3c-master.c b/drivers/i3c/master/adi-i3c-master.c index f0c2ddb2f4b2..178029769bab 100644 --- a/drivers/i3c/master/adi-i3c-master.c +++ b/drivers/i3c/master/adi-i3c-master.c @@ -932,6 +932,7 @@ static const struct of_device_id adi_i3c_master_of_match[] = { { .compatible = "adi,i3c-master-v1" }, {} }; +MODULE_DEVICE_TABLE(of, adi_i3c_master_of_match); static int adi_i3c_master_probe(struct platform_device *pdev) { diff --git a/drivers/i3c/master/dw-i3c-master.c b/drivers/i3c/master/dw-i3c-master.c index 675c257ebe2c..4a4fae96be37 100644 --- a/drivers/i3c/master/dw-i3c-master.c +++ b/drivers/i3c/master/dw-i3c-master.c @@ -883,7 +883,15 @@ static int dw_i3c_master_daa(struct i3c_master_controller *m) if (!wait_for_completion_timeout(&xfer->comp, XFER_TIMEOUT)) dw_i3c_master_dequeue_xfer(master, xfer); - newdevs = GENMASK(master->maxdevs - cmd->rx_len - 1, 0); + /* + * cmd->rx_len holds the number of addresses ENTDAA left unassigned. + * On an empty bus rx_len == maxdevs, so avoid GENMASK(-1, 0). + */ + if (cmd->rx_len >= master->maxdevs) + newdevs = 0; + else + newdevs = GENMASK(master->maxdevs - cmd->rx_len - 1, 0); + newdevs &= ~olddevs; for (pos = 0; pos < master->maxdevs; pos++) { diff --git a/drivers/i3c/master/renesas-i3c.c b/drivers/i3c/master/renesas-i3c.c index b6c4dfc51fe5..91c04f3a4c55 100644 --- a/drivers/i3c/master/renesas-i3c.c +++ b/drivers/i3c/master/renesas-i3c.c @@ -647,6 +647,10 @@ static int renesas_i3c_daa(struct i3c_master_controller *m) if (!xfer) return -ENOMEM; + init_completion(&xfer->comp); + cmd = xfer->cmds; + cmd->rx_count = i3c->maxdevs; + /* Enable I3C bus. */ renesas_i3c_bus_enable(m, true); @@ -668,10 +672,6 @@ static int renesas_i3c_daa(struct i3c_master_controller *m) renesas_writel(i3c->regs, DATBAS(pos), datbas_dvdyad_with_parity(ret)); } - init_completion(&xfer->comp); - cmd = xfer->cmds; - cmd->rx_count = 0; - ret = renesas_i3c_get_free_pos(i3c); if (ret < 0) return ret; @@ -760,13 +760,13 @@ static int renesas_i3c_send_ccc_cmd(struct i3c_master_controller *m, if (!xfer) return -ENOMEM; - renesas_i3c_bus_enable(m, true); - init_completion(&xfer->comp); cmd = xfer->cmds; cmd->rnw = ccc->rnw; cmd->cmd0 = 0; + renesas_i3c_bus_enable(m, true); + /* Calculate the command descriptor. */ switch (ccc->id) { case I3C_CCC_SETDASA: @@ -818,15 +818,15 @@ static int renesas_i3c_priv_xfers(struct i3c_dev_desc *dev, struct i3c_priv_xfer struct renesas_i3c_i2c_dev_data *data = i3c_dev_get_master_data(dev); int i; - /* Enable I3C bus. */ - renesas_i3c_bus_enable(m, true); - struct renesas_i3c_xfer *xfer __free(kfree) = renesas_i3c_alloc_xfer(i3c, 1); if (!xfer) return -ENOMEM; init_completion(&xfer->comp); + /* Enable I3C bus. */ + renesas_i3c_bus_enable(m, true); + for (i = 0; i < i3c_nxfers; i++) { struct renesas_i3c_cmd *cmd = xfer->cmds; @@ -940,19 +940,19 @@ static int renesas_i3c_i2c_xfers(struct i2c_dev_desc *dev, u8 start_bit = CNDCTL_STCND; int i; - struct renesas_i3c_xfer *xfer __free(kfree) = renesas_i3c_alloc_xfer(i3c, 1); - if (!xfer) - return -ENOMEM; - if (!i2c_nxfers) return 0; - renesas_i3c_bus_enable(m, false); + struct renesas_i3c_xfer *xfer __free(kfree) = renesas_i3c_alloc_xfer(i3c, 1); + if (!xfer) + return -ENOMEM; init_completion(&xfer->comp); xfer->is_i2c_xfer = true; cmd = xfer->cmds; + renesas_i3c_bus_enable(m, false); + if (!(renesas_readl(i3c->regs, BCST) & BCST_BFREF)) { cmd->err = -EBUSY; return cmd->err; diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c index 9ba83954c255..922ed32f73ed 100644 --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -76,6 +77,10 @@ static unsigned int preferred_states_mask __read_mostly; static bool force_irq_on __read_mostly; static bool ibrs_off __read_mostly; +/* The maximum allowed length for the 'table' module parameter */ +#define MAX_CMDLINE_TABLE_LEN 256 +static char cmdline_table_str[MAX_CMDLINE_TABLE_LEN] __read_mostly; + static struct cpuidle_device __percpu *intel_idle_cpuidle_devices; static unsigned long auto_demotion_disable_flags; @@ -107,6 +112,9 @@ static struct device *sysfs_root __initdata; static const struct idle_cpu *icpu __initdata; static struct cpuidle_state *cpuidle_state_table __initdata; +/* C-states data from the 'intel_idle.table' cmdline parameter */ +static struct cpuidle_state cmdline_states[CPUIDLE_STATE_MAX] __initdata; + static unsigned int mwait_substates __initdata; /* @@ -2420,6 +2428,152 @@ static void __init intel_idle_sysfs_uninit(void) put_device(sysfs_root); } + /** + * get_cmdline_field - Get the current field from a cmdline string. + * @args: The cmdline string to get the current field from. + * @field: Pointer to the current field upon return. + * @sep: The fields separator character. + * + * Examples: + * Input: args="C1:1:1,C1E:2:10", sep=':' + * Output: field="C1", return "1:1,C1E:2:10" + * Input: args="C1:1:1,C1E:2:10", sep=',' + * Output: field="C1:1:1", return "C1E:2:10" + * Ipnut: args="::", sep=':' + * Output: field="", return ":" + * + * Return: The continuation of the cmdline string after the field or NULL. + */ +static char *get_cmdline_field(char *args, char **field, char sep) +{ + unsigned int i; + + for (i = 0; args[i] && !isspace(args[i]); i++) { + if (args[i] == sep) + break; + } + + *field = args; + + if (args[i] != sep) + return NULL; + + args[i] = '\0'; + return args + i + 1; +} + +/** + * cmdline_table_adjust - Adjust the C-states table with data from cmdline. + * @drv: cpuidle driver (assumed to point to intel_idle_driver). + * + * Adjust the C-states table with data from the 'intel_idle.table' module + * parameter (if specified). + */ +static void __init cmdline_table_adjust(struct cpuidle_driver *drv) +{ + char *args = cmdline_table_str; + struct cpuidle_state *state; + int i; + + if (args[0] == '\0') + /* The 'intel_idle.table' module parameter was not specified */ + return; + + /* Create a copy of the C-states table */ + for (i = 0; i < drv->state_count; i++) + cmdline_states[i] = drv->states[i]; + + /* + * Adjust the C-states table copy with data from the 'intel_idle.table' + * module parameter. + */ + while (args) { + char *fields, *name, *val; + + /* + * Get the next C-state definition, which is expected to be + * '::'. Treat "empty" + * fields as unchanged. For example, + * '::' leaves the latency unchanged. + */ + args = get_cmdline_field(args, &fields, ','); + + /* name */ + fields = get_cmdline_field(fields, &name, ':'); + if (!fields) + goto error; + + if (!strcmp(name, "POLL")) { + pr_err("Cannot adjust POLL\n"); + continue; + } + + /* Find the C-state by its name */ + state = NULL; + for (i = 0; i < drv->state_count; i++) { + if (!strcmp(name, drv->states[i].name)) { + state = &cmdline_states[i]; + break; + } + } + + if (!state) { + pr_err("C-state '%s' was not found\n", name); + continue; + } + + /* Latency */ + fields = get_cmdline_field(fields, &val, ':'); + if (!fields) + goto error; + + if (*val) { + if (kstrtouint(val, 0, &state->exit_latency)) + goto error; + } + + /* Target residency */ + fields = get_cmdline_field(fields, &val, ':'); + + if (*val) { + if (kstrtouint(val, 0, &state->target_residency)) + goto error; + } + + /* + * Allow for 3 more fields, but ignore them. Helps to make + * possible future extensions of the cmdline format backward + * compatible. + */ + for (i = 0; fields && i < 3; i++) { + fields = get_cmdline_field(fields, &val, ':'); + if (!fields) + break; + } + + if (fields) { + pr_err("Too many fields for C-state '%s'\n", state->name); + goto error; + } + + pr_info("C-state from cmdline: name=%s, latency=%u, residency=%u\n", + state->name, state->exit_latency, state->target_residency); + } + + /* Copy the adjusted C-states table back */ + for (i = 1; i < drv->state_count; i++) + drv->states[i] = cmdline_states[i]; + + pr_info("Adjusted C-states with data from 'intel_idle.table'\n"); + return; + +error: + pr_info("Failed to adjust C-states with data from 'intel_idle.table'\n"); +} + +#define INTEL_IDLE_INIT_QOS 20 +static struct pm_qos_request qos_req __initdata; + static int __init intel_idle_init(void) { const struct x86_cpu_id *id; @@ -2485,11 +2639,19 @@ static int __init intel_idle_init(void) if (!intel_idle_cpuidle_devices) return -ENOMEM; + intel_idle_cpuidle_driver_init(&intel_idle_driver); + cmdline_table_adjust(&intel_idle_driver); + retval = intel_idle_sysfs_init(); if (retval) pr_warn("failed to initialized sysfs"); - intel_idle_cpuidle_driver_init(&intel_idle_driver); + /* + * Some platforms, in particular the Intel S1200BTL motherboard, have a + * problem with using package idle states too early, so prevent that + * from taking place until the device_initcall() phase is over. + */ + cpu_latency_qos_add_request(&qos_req, INTEL_IDLE_INIT_QOS); retval = cpuidle_register_driver(&intel_idle_driver); if (retval) { @@ -2515,6 +2677,9 @@ static int __init intel_idle_init(void) intel_idle_cpuidle_devices_uninit(); cpuidle_unregister_driver(&intel_idle_driver); init_driver_fail: + if (cpu_latency_qos_request_active((&qos_req))) + cpu_latency_qos_remove_request(&qos_req); + intel_idle_sysfs_uninit(); free_percpu(intel_idle_cpuidle_devices); return retval; @@ -2522,6 +2687,15 @@ static int __init intel_idle_init(void) } subsys_initcall_sync(intel_idle_init); +static int __init intel_idle_init_complete(void) +{ + if (cpu_latency_qos_request_active((&qos_req))) + cpu_latency_qos_remove_request(&qos_req); + + return 0; +} +device_initcall_sync(intel_idle_init_complete); + /* * We are not really modular, but we used to support that. Meaning we also * support "intel_idle.max_cstate=..." at boot and also a read-only export of @@ -2560,3 +2734,21 @@ module_param(force_irq_on, bool, 0444); */ module_param(ibrs_off, bool, 0444); MODULE_PARM_DESC(ibrs_off, "Disable IBRS when idle"); + +/* + * Define the C-states table from a user input string. Expected format is + * 'name:latency:residency', where: + * - name: The C-state name. + * - latency: The C-state exit latency in us. + * - residency: The C-state target residency in us. + * + * Multiple C-states can be defined by separating them with commas: + * 'name1:latency1:residency1,name2:latency2:residency2' + * + * Example: intel_idle.table=C1:1:1,C1E:5:10,C6:100:600 + * + * To leave latency or residency unchanged, use an empty field, for example: + * 'C1:1:1,C1E::10' - leaves C1E latency unchanged. + */ +module_param_string(table, cmdline_table_str, MAX_CMDLINE_TABLE_LEN, 0444); +MODULE_PARM_DESC(table, "Build the C-states table from a user input string"); diff --git a/drivers/iio/accel/dmard09.c b/drivers/iio/accel/dmard09.c index d9290e3b9c46..5297ab2ee0bb 100644 --- a/drivers/iio/accel/dmard09.c +++ b/drivers/iio/accel/dmard09.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #define DMARD09_DRV_NAME "dmard09" @@ -79,6 +80,12 @@ static int dmard09_read_raw(struct iio_dev *indio_dev, *val = accel; return IIO_VAL_INT; + case IIO_CHAN_INFO_SCALE: + *val = 0; + /* 1 g / 32 LSB, in m/s^2 */ + *val2 = IIO_G_TO_M_S_2(NANO / 32); + + return IIO_VAL_INT_PLUS_NANO; default: return -EINVAL; } diff --git a/drivers/iio/light/apds9306.c b/drivers/iio/light/apds9306.c index 389125675caa..5ce01037ae55 100644 --- a/drivers/iio/light/apds9306.c +++ b/drivers/iio/light/apds9306.c @@ -471,9 +471,9 @@ static int apds9306_read_data(struct apds9306_data *data, int *val, int reg) int status = 0; u8 buff[3]; - ret = pm_runtime_resume_and_get(data->dev); - if (ret) - return ret; + PM_RUNTIME_ACQUIRE_AUTOSUSPEND(data->dev, pm); + if (PM_RUNTIME_ACQUIRE_ERR(&pm)) + return PM_RUNTIME_ACQUIRE_ERR(&pm); ret = regmap_field_read(rf->intg_time, &intg_time_idx); if (ret) @@ -537,8 +537,6 @@ static int apds9306_read_data(struct apds9306_data *data, int *val, int reg) *val = get_unaligned_le24(&buff); - pm_runtime_put_autosuspend(data->dev); - return 0; } diff --git a/drivers/iio/light/gp2ap002.c b/drivers/iio/light/gp2ap002.c index b62611099037..bb17ef2a829b 100644 --- a/drivers/iio/light/gp2ap002.c +++ b/drivers/iio/light/gp2ap002.c @@ -342,6 +342,10 @@ static int gp2ap002_write_event_config(struct iio_dev *indio_dev, bool state) { struct gp2ap002 *gp2ap002 = iio_priv(indio_dev); + int ret; + + if (state == gp2ap002->enabled) + return 0; if (state) { /* @@ -349,13 +353,16 @@ static int gp2ap002_write_event_config(struct iio_dev *indio_dev, * already) and reintialize the sensor by using runtime_pm * callbacks. */ - pm_runtime_get_sync(gp2ap002->dev); - gp2ap002->enabled = true; + ret = pm_runtime_resume_and_get(gp2ap002->dev); + if (ret) + return ret; + } else { pm_runtime_put_autosuspend(gp2ap002->dev); - gp2ap002->enabled = false; } + gp2ap002->enabled = state; + return 0; } @@ -642,6 +649,7 @@ static int gp2ap002_runtime_suspend(struct device *dev) /* Disable chip and IRQ, everything off */ ret = regmap_write(gp2ap002->map, GP2AP002_OPMOD, 0x00); if (ret) { + enable_irq(gp2ap002->irq); dev_err(gp2ap002->dev, "error setting up operation mode\n"); return ret; } diff --git a/drivers/iio/light/isl29028.c b/drivers/iio/light/isl29028.c index 374bccad9119..f89c5d9ee524 100644 --- a/drivers/iio/light/isl29028.c +++ b/drivers/iio/light/isl29028.c @@ -409,7 +409,7 @@ static int isl29028_write_raw(struct iio_dev *indio_dev, if (ret < 0) return ret; - return ret; + return 0; } static int isl29028_read_raw(struct iio_dev *indio_dev, diff --git a/drivers/iio/light/opt4060.c b/drivers/iio/light/opt4060.c index 0f8716d72cf6..9be752535d75 100644 --- a/drivers/iio/light/opt4060.c +++ b/drivers/iio/light/opt4060.c @@ -731,6 +731,7 @@ static ssize_t opt4060_read_ev_period(struct opt4060_chip *chip, int *val, { int ret, pers, fault_count, int_time; u64 uval; + u32 rem; int_time = opt4060_int_time_reg[chip->int_time][0]; @@ -756,7 +757,8 @@ static ssize_t opt4060_read_ev_period(struct opt4060_chip *chip, int *val, } uval = mul_u32_u32(int_time, pers); - *val = div_u64_rem(uval, MICRO, val2); + *val = div_u64_rem(uval, MICRO, &rem); + *val2 = rem; return IIO_VAL_INT_PLUS_MICRO; } diff --git a/drivers/iio/light/tsl2583.c b/drivers/iio/light/tsl2583.c index 8801a491de77..15693bc3e8e8 100644 --- a/drivers/iio/light/tsl2583.c +++ b/drivers/iio/light/tsl2583.c @@ -794,7 +794,7 @@ static int tsl2583_write_raw(struct iio_dev *indio_dev, if (ret < 0) return ret; - return ret; + return 0; } static const struct iio_info tsl2583_info = { diff --git a/drivers/iio/light/tsl2772.c b/drivers/iio/light/tsl2772.c index 0b171106441a..ffbde1540114 100644 --- a/drivers/iio/light/tsl2772.c +++ b/drivers/iio/light/tsl2772.c @@ -1263,7 +1263,7 @@ static int tsl2772_read_raw(struct iio_dev *indio_dev, } break; case IIO_CHAN_INFO_CALIBSCALE: - if (chan->type == IIO_LIGHT) + if (chan->type == IIO_INTENSITY) *val = tsl2772_als_gain[chip->settings.als_gain]; else *val = tsl2772_prox_gain[chip->settings.prox_gain]; diff --git a/drivers/iio/orientation/hid-sensor-rotation.c b/drivers/iio/orientation/hid-sensor-rotation.c index 83a0b0283605..061934f0cb99 100644 --- a/drivers/iio/orientation/hid-sensor-rotation.c +++ b/drivers/iio/orientation/hid-sensor-rotation.c @@ -353,12 +353,6 @@ static int hid_dev_rot_probe(struct platform_device *pdev) return ret; } - ret = iio_device_register(indio_dev); - if (ret) { - dev_err(&pdev->dev, "device register failed\n"); - goto error_remove_trigger; - } - rot_state->callbacks.send_event = dev_rot_proc_event; rot_state->callbacks.capture_sample = dev_rot_capture_sample; rot_state->callbacks.pdev = pdev; @@ -366,13 +360,19 @@ static int hid_dev_rot_probe(struct platform_device *pdev) &rot_state->callbacks); if (ret) { dev_err(&pdev->dev, "callback reg failed\n"); - goto error_iio_unreg; + goto error_remove_trigger; + } + + ret = iio_device_register(indio_dev); + if (ret) { + dev_err(&pdev->dev, "device register failed\n"); + goto error_remove_callback; } return 0; -error_iio_unreg: - iio_device_unregister(indio_dev); +error_remove_callback: + sensor_hub_remove_callback(hsdev, hsdev->usage); error_remove_trigger: hid_sensor_remove_trigger(indio_dev, &rot_state->common_attributes); return ret; @@ -385,8 +385,8 @@ static void hid_dev_rot_remove(struct platform_device *pdev) struct iio_dev *indio_dev = platform_get_drvdata(pdev); struct dev_rot_state *rot_state = iio_priv(indio_dev); - sensor_hub_remove_callback(hsdev, hsdev->usage); iio_device_unregister(indio_dev); + sensor_hub_remove_callback(hsdev, hsdev->usage); hid_sensor_remove_trigger(indio_dev, &rot_state->common_attributes); } diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 648acc2222cf..a73a879706c4 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -3523,10 +3523,12 @@ static void addr_handler(int status, struct sockaddr *src_addr, memcpy(addr, src_addr, rdma_addr_size(src_addr)); if (!status && !id_priv->cma_dev) { status = cma_acquire_dev_by_src_ip(id_priv); - if (status) + if (status) { pr_debug_ratelimited("RDMA CM: ADDR_ERROR: failed to acquire device. status %d\n", status); - rdma_restrack_add(&id_priv->res); + } else { + rdma_restrack_add(&id_priv->res); + } } else if (status) { pr_debug_ratelimited("RDMA CM: ADDR_ERROR: failed to resolve IP. status %d\n", status); } diff --git a/drivers/infiniband/core/counters.c b/drivers/infiniband/core/counters.c index 5dad5d77ce27..5dfd64b0449d 100644 --- a/drivers/infiniband/core/counters.c +++ b/drivers/infiniband/core/counters.c @@ -226,7 +226,6 @@ static void rdma_counter_free(struct rdma_counter *counter) mutex_unlock(&port_counter->lock); - rdma_restrack_del(&counter->res); rdma_free_hw_stats_struct(counter->stats); kfree(counter); } @@ -321,6 +320,7 @@ static void counter_release(struct kref *kref) counter = container_of(kref, struct rdma_counter, kref); counter_history_stat_update(counter); + rdma_restrack_del(&counter->res); counter->device->ops.counter_dealloc(counter); rdma_counter_free(counter); } @@ -482,7 +482,8 @@ static struct rdma_counter *rdma_get_counter_by_id(struct ib_device *dev, return NULL; counter = container_of(res, struct rdma_counter, res); - kref_get(&counter->kref); + if (!kref_get_unless_zero(&counter->kref)) + counter = NULL; rdma_restrack_put(res); return counter; diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c index 584537c71545..4ea0dfcff95e 100644 --- a/drivers/infiniband/core/cq.c +++ b/drivers/infiniband/core/cq.c @@ -324,6 +324,7 @@ void ib_free_cq(struct ib_cq *cq) if (WARN_ON_ONCE(cq->cqe_used)) return; + rdma_restrack_del(&cq->res); if (cq->device->ops.pre_destroy_cq) { ret = cq->device->ops.pre_destroy_cq(cq); WARN_ONCE(ret, "Disable of kernel CQ shouldn't fail"); @@ -350,7 +351,6 @@ void ib_free_cq(struct ib_cq *cq) else ret = cq->device->ops.destroy_cq(cq, NULL); WARN_ONCE(ret, "Destroy of kernel CQ shouldn't fail"); - rdma_restrack_del(&cq->res); kfree(cq->wc); kfree(cq); } diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index ac9aaef1e5e6..c8b7942739a0 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -2848,6 +2848,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops) SET_DEVICE_OP(dev_ops, query_gid); SET_DEVICE_OP(dev_ops, query_pkey); SET_DEVICE_OP(dev_ops, query_port); + SET_DEVICE_OP(dev_ops, query_port_speed); SET_DEVICE_OP(dev_ops, query_qp); SET_DEVICE_OP(dev_ops, query_srq); SET_DEVICE_OP(dev_ops, query_ucontext); @@ -3169,6 +3170,7 @@ static void __exit ib_core_cleanup(void) /* Make sure that any pending umem accounting work is done. */ destroy_workqueue(ib_wq); destroy_workqueue(ib_unreg_wq); + rcu_barrier(); WARN_ON(!xa_empty(&clients)); WARN_ON(!xa_empty(&devices)); } diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c index 2220a2dfab24..c92c24e681f1 100644 --- a/drivers/infiniband/core/nldev.c +++ b/drivers/infiniband/core/nldev.c @@ -679,7 +679,7 @@ static int fill_res_mr_entry(struct sk_buff *msg, bool has_cap_net_admin, struct rdma_restrack_entry *res, uint32_t port) { struct ib_mr *mr = container_of(res, struct ib_mr, res); - struct ib_device *dev = mr->pd->device; + struct ib_device *dev = mr->device; if (has_cap_net_admin) { if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_RKEY, mr->rkey)) @@ -695,9 +695,12 @@ static int fill_res_mr_entry(struct sk_buff *msg, bool has_cap_net_admin, if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_MRN, res->id)) return -EMSGSIZE; - if (!rdma_is_kernel_res(res) && - nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_PDN, mr->pd->res.id)) - return -EMSGSIZE; + if (!rdma_is_kernel_res(res)) { + struct ib_pd *pd = READ_ONCE(mr->pd); + + if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_PDN, pd->res.id)) + return -EMSGSIZE; + } if (fill_res_name_pid(msg, res)) return -EMSGSIZE; @@ -711,7 +714,7 @@ static int fill_res_mr_raw_entry(struct sk_buff *msg, bool has_cap_net_admin, struct rdma_restrack_entry *res, uint32_t port) { struct ib_mr *mr = container_of(res, struct ib_mr, res); - struct ib_device *dev = mr->pd->device; + struct ib_device *dev = mr->device; if (!dev->ops.fill_res_mr_entry_raw) return -EINVAL; @@ -1001,7 +1004,7 @@ static int fill_stat_mr_entry(struct sk_buff *msg, bool has_cap_net_admin, struct rdma_restrack_entry *res, uint32_t port) { struct ib_mr *mr = container_of(res, struct ib_mr, res); - struct ib_device *dev = mr->pd->device; + struct ib_device *dev = mr->device; if (nla_put_u32(msg, RDMA_NLDEV_ATTR_RES_MRN, res->id)) goto err; @@ -2100,6 +2103,11 @@ static int nldev_stat_set_counter_dynamic_doit(struct nlattr *tb[], nla_for_each_nested(entry_attr, tb[RDMA_NLDEV_ATTR_STAT_HWCOUNTERS], rem) { + if (nla_len(entry_attr) != sizeof(u32)) { + ret = -EINVAL; + goto out; + } + index = nla_get_u32(entry_attr); if ((index >= stats->num_counters) || !(stats->descs[index].flags & IB_STAT_FLAG_OPTIONAL)) { diff --git a/drivers/infiniband/core/restrack.c b/drivers/infiniband/core/restrack.c index a7de6f403fca..56aa1def56fa 100644 --- a/drivers/infiniband/core/restrack.c +++ b/drivers/infiniband/core/restrack.c @@ -71,6 +71,8 @@ int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type, xa_lock(&rt->xa); xas_for_each(&xas, e, U32_MAX) { + if (xa_is_zero(e)) + continue; if (xa_get_mark(&rt->xa, e->id, RESTRACK_DD) && !show_details) continue; cnt++; @@ -127,6 +129,46 @@ static void rdma_restrack_attach_task(struct rdma_restrack_entry *res, res->user = true; } +static struct rdma_restrack_root *res_to_rt(struct rdma_restrack_entry *res) +{ + struct ib_device *dev = res_to_dev(res); + + if (WARN_ON(!dev)) + return NULL; + + return &dev->res[res->type]; +} + +static void restrack_drain_res(struct rdma_restrack_root *rt, + struct rdma_restrack_entry *res) +{ + if (rt) { + struct rdma_restrack_entry *old; + + old = xa_cmpxchg(&rt->xa, res->id, res, XA_ZERO_ENTRY, + GFP_KERNEL); + WARN_ON(old != res); + } + + rdma_restrack_put(res); + wait_for_completion(&res->comp); +} + +static void restrack_restore_res(struct rdma_restrack_root *rt, + struct rdma_restrack_entry *res) +{ + reinit_completion(&res->comp); + kref_init(&res->kref); + + if (rt) { + struct rdma_restrack_entry *old; + + old = xa_cmpxchg(&rt->xa, res->id, XA_ZERO_ENTRY, res, + GFP_KERNEL); + WARN_ON(old); + } +} + /** * rdma_restrack_set_name() - set the task for this resource * @res: resource entry @@ -175,22 +217,23 @@ void rdma_restrack_new(struct rdma_restrack_entry *res, EXPORT_SYMBOL(rdma_restrack_new); /** - * rdma_restrack_add() - add object to the reource tracking database + * rdma_restrack_add() - add object to the resource tracking database. + * If this resource reuses an ID of a resource that was already destroyed + * after calling rdma_restrack_begin() but didn't yet call + * rdma_restrack_commit_del() it can result in an untracked QP. * @res: resource entry */ void rdma_restrack_add(struct rdma_restrack_entry *res) { - struct ib_device *dev = res_to_dev(res); struct rdma_restrack_root *rt; int ret = 0; - if (!dev) - return; - if (res->no_track) goto out; - rt = &dev->res[res->type]; + rt = res_to_rt(res); + if (!rt) + return; if (res->type == RDMA_RESTRACK_QP) { /* Special case to ensure that LQPN points to right QP */ @@ -227,6 +270,28 @@ void rdma_restrack_add(struct rdma_restrack_entry *res) } EXPORT_SYMBOL(rdma_restrack_add); +/** + * rdma_restrack_abort_del() - re-add object to the resource tracking database + * it can only be used after rdma_restrack_begin_del(). + * @res: resource entry + */ +void rdma_restrack_abort_del(struct rdma_restrack_entry *res) +{ + struct rdma_restrack_root *rt = NULL; + + if (!res->valid) + return; + + if (!res->no_track) { + rt = res_to_rt(res); + if (!rt) + return; + } + + restrack_restore_res(rt, res); +} +EXPORT_SYMBOL(rdma_restrack_abort_del); + int __must_check rdma_restrack_get(struct rdma_restrack_entry *res) { return kref_get_unless_zero(&res->kref); @@ -263,7 +328,7 @@ static void restrack_release(struct kref *kref) struct rdma_restrack_entry *res; res = container_of(kref, struct rdma_restrack_entry, kref); - if (res->task) { + if (res->task && !res->valid) { put_task_struct(res->task); res->task = NULL; } @@ -277,14 +342,43 @@ int rdma_restrack_put(struct rdma_restrack_entry *res) EXPORT_SYMBOL(rdma_restrack_put); /** - * rdma_restrack_del() - delete object from the reource tracking database + * rdma_restrack_sync() - Fence concurrent netlink dumps on an entry + * @res: resource entry + * + * After this returns any concurrent netlink dump threads will see the current + * value of the object. This is useful if the object has to be changed and there + * is not locking to protect the nl side. Eg for mr->pd. This effectively + * destroys the object from a kref/xarray perspective and then immediately + * restores it. The kref is acting like a lock to barrier concurrent nl threads. + * Callers must ensure rdma_restrack_del() is not concurrently called. + */ +void rdma_restrack_sync(struct rdma_restrack_entry *res) +{ + struct rdma_restrack_root *rt; + + if (!res->valid || res->no_track) + return; + + rt = res_to_rt(res); + if (!rt) + return; + + if (WARN_ON(xa_get_mark(&rt->xa, res->id, RESTRACK_DD))) + return; + + restrack_drain_res(rt, res); + restrack_restore_res(rt, res); +} +EXPORT_SYMBOL(rdma_restrack_sync); + +/** + * rdma_restrack_del() - delete object from the resource tracking database * @res: resource entry */ void rdma_restrack_del(struct rdma_restrack_entry *res) { struct rdma_restrack_entry *old; struct rdma_restrack_root *rt; - struct ib_device *dev; if (!res->valid) { if (res->task) { @@ -297,12 +391,10 @@ void rdma_restrack_del(struct rdma_restrack_entry *res) if (res->no_track) goto out; - dev = res_to_dev(res); - if (WARN_ON(!dev)) + rt = res_to_rt(res); + if (!rt) return; - rt = &dev->res[res->type]; - old = xa_erase(&rt->xa, res->id); WARN_ON(old != res); @@ -310,5 +402,61 @@ void rdma_restrack_del(struct rdma_restrack_entry *res) res->valid = false; rdma_restrack_put(res); wait_for_completion(&res->comp); + if (res->task) { + put_task_struct(res->task); + res->task = NULL; + } } EXPORT_SYMBOL(rdma_restrack_del); + +/** + * rdma_restrack_begin_del() - invalidate the object from the resource tracking + * database but preserve its index in the array. + * Since this preserves the index in the array until rdma_restrack_commit_del() + * is called, if rdma_restrack_add() is called in between with an old QP ID it + * can result in an untracked QP. + * @res: resource entry + */ +void rdma_restrack_begin_del(struct rdma_restrack_entry *res) +{ + struct rdma_restrack_root *rt = NULL; + + if (!res->valid) + return; + + if (!res->no_track) { + rt = res_to_rt(res); + if (!rt) + return; + } + + restrack_drain_res(rt, res); +} +EXPORT_SYMBOL(rdma_restrack_begin_del); + +/** + * rdma_restrack_commit_del() - delete object from the resource tracking + * database and free the task. + * @res: resource entry + */ +void rdma_restrack_commit_del(struct rdma_restrack_entry *res) +{ + struct rdma_restrack_root *rt; + + if (!res->valid || res->no_track) + goto out; + + rt = res_to_rt(res); + if (!rt) + return; + + xa_erase(&rt->xa, res->id); + +out: + res->valid = false; + if (res->task) { + put_task_struct(res->task); + res->task = NULL; + } +} +EXPORT_SYMBOL(rdma_restrack_commit_del); diff --git a/drivers/infiniband/core/restrack.h b/drivers/infiniband/core/restrack.h index 6a04fc41f738..2df78e084e10 100644 --- a/drivers/infiniband/core/restrack.h +++ b/drivers/infiniband/core/restrack.h @@ -26,7 +26,11 @@ struct rdma_restrack_root { int rdma_restrack_init(struct ib_device *dev); void rdma_restrack_clean(struct ib_device *dev); void rdma_restrack_add(struct rdma_restrack_entry *res); +void rdma_restrack_abort_del(struct rdma_restrack_entry *res); void rdma_restrack_del(struct rdma_restrack_entry *res); +void rdma_restrack_sync(struct rdma_restrack_entry *res); +void rdma_restrack_begin_del(struct rdma_restrack_entry *res); +void rdma_restrack_commit_del(struct rdma_restrack_entry *res); void rdma_restrack_new(struct rdma_restrack_entry *res, enum rdma_restrack_type type); void rdma_restrack_set_name(struct rdma_restrack_entry *res, diff --git a/drivers/infiniband/core/ucma.c b/drivers/infiniband/core/ucma.c index 5d76879218ad..797efd4a4728 100644 --- a/drivers/infiniband/core/ucma.c +++ b/drivers/infiniband/core/ucma.c @@ -951,7 +951,7 @@ static ssize_t ucma_query_path(struct ucma_context *ctx, resp->num_paths = ctx->cm_id->route.num_pri_alt_paths; for (i = 0, out_len -= sizeof(*resp); - i < resp->num_paths && out_len > sizeof(struct ib_path_rec_data); + i < resp->num_paths && out_len >= sizeof(struct ib_path_rec_data); i++, out_len -= sizeof(struct ib_path_rec_data)) { struct sa_path_rec *rec = &ctx->cm_id->route.path_rec[i]; diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c index f4616deeca54..95c1974e0266 100644 --- a/drivers/infiniband/core/uverbs_cmd.c +++ b/drivers/infiniband/core/uverbs_cmd.c @@ -47,6 +47,7 @@ #include "uverbs.h" #include "core_priv.h" +#include "restrack.h" /* * Copy a response to userspace. If the provided 'resp' is larger than the @@ -830,6 +831,10 @@ static int ib_uverbs_rereg_mr(struct uverbs_attr_bundle *attrs) ret = PTR_ERR(new_pd); goto put_uobjs; } + if (new_pd == orig_pd) { + uobj_put_obj_read(new_pd); + cmd.flags &= ~IB_MR_REREG_PD; + } } else { new_pd = mr->pd; } @@ -875,9 +880,10 @@ static int ib_uverbs_rereg_mr(struct uverbs_attr_bundle *attrs) mr = new_mr; } else { if (cmd.flags & IB_MR_REREG_PD) { - atomic_dec(&orig_pd->usecnt); - mr->pd = new_pd; atomic_inc(&new_pd->usecnt); + WRITE_ONCE(mr->pd, new_pd); + rdma_restrack_sync(&mr->res); + atomic_dec(&orig_pd->usecnt); } if (cmd.flags & IB_MR_REREG_TRANS) { mr->iova = cmd.hca_va; diff --git a/drivers/infiniband/core/uverbs_std_types_device.c b/drivers/infiniband/core/uverbs_std_types_device.c index c0fd283d9d6c..a28f9f21bed8 100644 --- a/drivers/infiniband/core/uverbs_std_types_device.c +++ b/drivers/infiniband/core/uverbs_std_types_device.c @@ -209,6 +209,39 @@ static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_PORT)( &resp, sizeof(resp)); } +static int UVERBS_HANDLER(UVERBS_METHOD_QUERY_PORT_SPEED)( + struct uverbs_attr_bundle *attrs) +{ + struct ib_ucontext *ucontext; + struct ib_device *ib_dev; + u32 port_num; + u64 speed; + int ret; + + ucontext = ib_uverbs_get_ucontext(attrs); + if (IS_ERR(ucontext)) + return PTR_ERR(ucontext); + ib_dev = ucontext->device; + + if (!ib_dev->ops.query_port_speed) + return -EOPNOTSUPP; + + ret = uverbs_get_const(&port_num, attrs, + UVERBS_ATTR_QUERY_PORT_SPEED_PORT_NUM); + if (ret) + return ret; + + if (!rdma_is_port_valid(ib_dev, port_num)) + return -EINVAL; + + ret = ib_dev->ops.query_port_speed(ib_dev, port_num, &speed); + if (ret) + return ret; + + return uverbs_copy_to(attrs, UVERBS_ATTR_QUERY_PORT_SPEED_RESP, + &speed, sizeof(speed)); +} + static int UVERBS_HANDLER(UVERBS_METHOD_GET_CONTEXT)( struct uverbs_attr_bundle *attrs) { @@ -469,6 +502,14 @@ DECLARE_UVERBS_NAMED_METHOD( active_speed_ex), UA_MANDATORY)); +DECLARE_UVERBS_NAMED_METHOD( + UVERBS_METHOD_QUERY_PORT_SPEED, + UVERBS_ATTR_CONST_IN(UVERBS_ATTR_QUERY_PORT_SPEED_PORT_NUM, u32, + UA_MANDATORY), + UVERBS_ATTR_PTR_OUT(UVERBS_ATTR_QUERY_PORT_SPEED_RESP, + UVERBS_ATTR_TYPE(u64), + UA_MANDATORY)); + DECLARE_UVERBS_NAMED_METHOD( UVERBS_METHOD_QUERY_GID_TABLE, UVERBS_ATTR_CONST_IN(UVERBS_ATTR_QUERY_GID_TABLE_ENTRY_SIZE, u64, @@ -498,6 +539,7 @@ DECLARE_UVERBS_GLOBAL_METHODS(UVERBS_OBJECT_DEVICE, &UVERBS_METHOD(UVERBS_METHOD_INVOKE_WRITE), &UVERBS_METHOD(UVERBS_METHOD_INFO_HANDLES), &UVERBS_METHOD(UVERBS_METHOD_QUERY_PORT), + &UVERBS_METHOD(UVERBS_METHOD_QUERY_PORT_SPEED), &UVERBS_METHOD(UVERBS_METHOD_QUERY_CONTEXT), &UVERBS_METHOD(UVERBS_METHOD_QUERY_GID_TABLE), &UVERBS_METHOD(UVERBS_METHOD_QUERY_GID_ENTRY)); diff --git a/drivers/infiniband/core/uverbs_std_types_dmah.c b/drivers/infiniband/core/uverbs_std_types_dmah.c index 97101e093826..9873ab49a601 100644 --- a/drivers/infiniband/core/uverbs_std_types_dmah.c +++ b/drivers/infiniband/core/uverbs_std_types_dmah.c @@ -18,11 +18,14 @@ static int uverbs_free_dmah(struct ib_uobject *uobject, if (atomic_read(&dmah->usecnt)) return -EBUSY; + rdma_restrack_begin_del(&dmah->res); ret = dmah->device->ops.dealloc_dmah(dmah, attrs); - if (ret) + if (ret) { + rdma_restrack_abort_del(&dmah->res); return ret; + } - rdma_restrack_del(&dmah->res); + rdma_restrack_commit_del(&dmah->res); kfree(dmah); return 0; } diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index bc1878da55cd..4357164a9976 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -213,6 +213,57 @@ __attribute_const__ int ib_rate_to_mbps(enum ib_rate rate) } EXPORT_SYMBOL(ib_rate_to_mbps); +struct ib_speed_attr { + const char *str; + int speed; +}; + +#define IB_SPEED_ATTR(speed_type, _str, _speed) \ + [speed_type] = {.str = _str, .speed = _speed} + +static const struct ib_speed_attr ib_speed_attrs[] = { + IB_SPEED_ATTR(IB_SPEED_SDR, " SDR", 25), + IB_SPEED_ATTR(IB_SPEED_DDR, " DDR", 50), + IB_SPEED_ATTR(IB_SPEED_QDR, " QDR", 100), + IB_SPEED_ATTR(IB_SPEED_FDR10, " FDR10", 100), + IB_SPEED_ATTR(IB_SPEED_FDR, " FDR", 140), + IB_SPEED_ATTR(IB_SPEED_EDR, " EDR", 250), + IB_SPEED_ATTR(IB_SPEED_HDR, " HDR", 500), + IB_SPEED_ATTR(IB_SPEED_NDR, " NDR", 1000), + IB_SPEED_ATTR(IB_SPEED_XDR, " XDR", 2000), +}; + +int ib_port_attr_to_speed_info(struct ib_port_attr *attr, + struct ib_port_speed_info *speed_info) +{ + int speed_idx = attr->active_speed; + + switch (attr->active_speed) { + case IB_SPEED_DDR: + case IB_SPEED_QDR: + case IB_SPEED_FDR10: + case IB_SPEED_FDR: + case IB_SPEED_EDR: + case IB_SPEED_HDR: + case IB_SPEED_NDR: + case IB_SPEED_XDR: + case IB_SPEED_SDR: + break; + default: + speed_idx = IB_SPEED_SDR; /* Default to SDR for invalid rates */ + break; + } + + speed_info->str = ib_speed_attrs[speed_idx].str; + speed_info->rate = ib_speed_attrs[speed_idx].speed; + speed_info->rate *= ib_width_enum_to_int(attr->active_width); + if (speed_info->rate < 0) + return -EINVAL; + + return 0; +} +EXPORT_SYMBOL(ib_port_attr_to_speed_info); + __attribute_const__ enum rdma_transport_type rdma_node_get_transport(unsigned int node_type) { @@ -335,6 +386,7 @@ int ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata) { int ret; + rdma_restrack_begin_del(&pd->res); if (pd->__internal_mr) { ret = pd->device->ops.dereg_mr(pd->__internal_mr, NULL); WARN_ON(ret); @@ -342,10 +394,12 @@ int ib_dealloc_pd_user(struct ib_pd *pd, struct ib_udata *udata) } ret = pd->device->ops.dealloc_pd(pd, udata); - if (ret) + if (ret) { + rdma_restrack_abort_del(&pd->res); return ret; + } - rdma_restrack_del(&pd->res); + rdma_restrack_commit_del(&pd->res); kfree(pd); return ret; } @@ -1083,16 +1137,20 @@ int ib_destroy_srq_user(struct ib_srq *srq, struct ib_udata *udata) if (atomic_read(&srq->usecnt)) return -EBUSY; + rdma_restrack_begin_del(&srq->res); + ret = srq->device->ops.destroy_srq(srq, udata); - if (ret) + if (ret) { + rdma_restrack_abort_del(&srq->res); return ret; + } atomic_dec(&srq->pd->usecnt); if (srq->srq_type == IB_SRQT_XRC && srq->ext.xrc.xrcd) atomic_dec(&srq->ext.xrc.xrcd->usecnt); if (ib_srq_has_cq(srq->srq_type)) atomic_dec(&srq->ext.cq->usecnt); - rdma_restrack_del(&srq->res); + rdma_restrack_commit_del(&srq->res); kfree(srq); return ret; @@ -2098,6 +2156,8 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata) if (qp->real_qp != qp) return __ib_destroy_shared_qp(qp); + rdma_restrack_begin_del(&qp->res); + sec = qp->qp_sec; if (sec) ib_destroy_qp_security_begin(sec); @@ -2110,6 +2170,7 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata) if (ret) { if (sec) ib_destroy_qp_security_abort(sec); + rdma_restrack_abort_del(&qp->res); return ret; } @@ -2122,7 +2183,7 @@ int ib_destroy_qp_user(struct ib_qp *qp, struct ib_udata *udata) if (sec) ib_destroy_qp_security_end(sec); - rdma_restrack_del(&qp->res); + rdma_restrack_commit_del(&qp->res); kfree(qp); return ret; } @@ -2187,11 +2248,15 @@ int ib_destroy_cq_user(struct ib_cq *cq, struct ib_udata *udata) if (atomic_read(&cq->usecnt)) return -EBUSY; + rdma_restrack_begin_del(&cq->res); + ret = cq->device->ops.destroy_cq(cq, udata); - if (ret) + if (ret) { + rdma_restrack_abort_del(&cq->res); return ret; + } - rdma_restrack_del(&cq->res); + rdma_restrack_commit_del(&cq->res); kfree(cq); return ret; } diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 25d88c2c5735..64b6e572e704 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -4570,11 +4570,13 @@ int bnxt_re_mmap(struct ib_ucontext *ib_uctx, struct vm_area_struct *vma) case BNXT_RE_MMAP_DBR_PAGE: case BNXT_RE_MMAP_TOGGLE_PAGE: /* Driver doesn't expect write access for user space */ - if (vma->vm_flags & VM_WRITE) + if (vma->vm_flags & VM_WRITE) { ret = -EFAULT; - else + } else { + vm_flags_clear(vma, VM_MAYWRITE); ret = vm_insert_page(vma, vma->vm_start, virt_to_page((void *)bnxt_entry->mem_offset)); + } break; default: ret = -EINVAL; diff --git a/drivers/infiniband/hw/cxgb4/device.c b/drivers/infiniband/hw/cxgb4/device.c index d03936d007b0..0d023c361ef8 100644 --- a/drivers/infiniband/hw/cxgb4/device.c +++ b/drivers/infiniband/hw/cxgb4/device.c @@ -934,6 +934,7 @@ static void c4iw_rdev_close(struct c4iw_rdev *rdev) void c4iw_dealloc(struct uld_ctx *ctx) { + debugfs_remove_recursive(ctx->dev->debugfs_root); c4iw_rdev_close(&ctx->dev->rdev); WARN_ON(!xa_empty(&ctx->dev->cqs)); WARN_ON(!xa_empty(&ctx->dev->qps)); @@ -958,7 +959,6 @@ static void c4iw_remove(struct uld_ctx *ctx) if (!ctx->dev) return; - debugfs_remove_recursive(ctx->dev->debugfs_root); c4iw_unregister_device(ctx->dev); c4iw_dealloc(ctx); } diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c index 40dd6ac5f91a..ac349de9d2e4 100644 --- a/drivers/infiniband/hw/cxgb4/mem.c +++ b/drivers/infiniband/hw/cxgb4/mem.c @@ -277,6 +277,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry, int err; struct fw_ri_tpte *tpt; u32 stag_idx; + bool stag_idx_allocated = false; static atomic_t key; if (c4iw_fatal_error(rdev)) @@ -299,6 +300,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry, return -ENOMEM; } mutex_lock(&rdev->stats.lock); + stag_idx_allocated = true; rdev->stats.stag.cur += 32; if (rdev->stats.stag.cur > rdev->stats.stag.max) rdev->stats.stag.max = rdev->stats.stag.cur; @@ -333,7 +335,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry, (rdev->lldi.vr->stag.start >> 5), sizeof(*tpt), tpt, skb, wr_waitp); - if (reset_tpt_entry) { + if (reset_tpt_entry || (err && stag_idx_allocated)) { c4iw_put_resource(&rdev->resource.tpt_table, stag_idx); mutex_lock(&rdev->stats.lock); rdev->stats.stag.cur -= 32; diff --git a/drivers/infiniband/hw/efa/efa_verbs.c b/drivers/infiniband/hw/efa/efa_verbs.c index 0bb3389d761f..18db10741cee 100644 --- a/drivers/infiniband/hw/efa/efa_verbs.c +++ b/drivers/infiniband/hw/efa/efa_verbs.c @@ -1392,9 +1392,11 @@ static int pbl_chunk_list_create(struct efa_dev *dev, struct pbl_context *pbl) chunk_list->chunks[i].length = EFA_CHUNK_USED_SIZE; } - chunk_list->chunks[chunk_list_size - 1].length = - ((page_cnt % EFA_PTRS_PER_CHUNK) * EFA_CHUNK_PAYLOAD_PTR_SIZE) + - EFA_CHUNK_PTR_SIZE; + + if (page_cnt % EFA_PTRS_PER_CHUNK != 0) + chunk_list->chunks[chunk_list_size - 1].length = + ((page_cnt % EFA_PTRS_PER_CHUNK) * EFA_CHUNK_PAYLOAD_PTR_SIZE) + + EFA_CHUNK_PTR_SIZE; /* fill the dma addresses of sg list pages to chunks: */ chunk_idx = 0; @@ -1406,9 +1408,12 @@ static int pbl_chunk_list_create(struct efa_dev *dev, struct pbl_context *pbl) rdma_block_iter_dma_address(&biter); if (payload_idx == EFA_PTRS_PER_CHUNK) { + payload_idx = 0; chunk_idx++; + if (chunk_idx >= chunk_list_size) + break; + cur_chunk_buf = chunk_list->chunks[chunk_idx].buf; - payload_idx = 0; } } diff --git a/drivers/infiniband/hw/erdma/Kconfig b/drivers/infiniband/hw/erdma/Kconfig index 267fc1f3c42a..745e5551773f 100644 --- a/drivers/infiniband/hw/erdma/Kconfig +++ b/drivers/infiniband/hw/erdma/Kconfig @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only config INFINIBAND_ERDMA tristate "Alibaba Elastic RDMA Adapter (ERDMA) support" - depends on PCI_MSI && 64BIT + depends on PCI_MSI && 64BIT && !CPU_BIG_ENDIAN depends on INFINIBAND_ADDR_TRANS depends on INFINIBAND_USER_ACCESS help diff --git a/drivers/infiniband/hw/erdma/erdma_cm.c b/drivers/infiniband/hw/erdma/erdma_cm.c index e0acc185e719..f0c8ebf62a05 100644 --- a/drivers/infiniband/hw/erdma/erdma_cm.c +++ b/drivers/infiniband/hw/erdma/erdma_cm.c @@ -1021,10 +1021,9 @@ int erdma_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) if (laddr->sa_family != AF_INET || raddr->sa_family != AF_INET) return -EAFNOSUPPORT; - qp = find_qp_by_qpn(dev, params->qpn); + qp = erdma_qp_get_by_qpn(dev, params->qpn); if (!qp) return -ENOENT; - erdma_qp_get(qp); ret = sock_create(AF_INET, SOCK_STREAM, IPPROTO_TCP, &s); if (ret < 0) @@ -1154,10 +1153,9 @@ int erdma_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) return -ECONNRESET; } - qp = find_qp_by_qpn(dev, params->qpn); + qp = erdma_qp_get_by_qpn(dev, params->qpn); if (!qp) return -ENOENT; - erdma_qp_get(qp); down_write(&qp->state_lock); if (qp->attrs.iwarp.state > ERDMA_QPS_IWARP_RTR) { diff --git a/drivers/infiniband/hw/erdma/erdma_eq.c b/drivers/infiniband/hw/erdma/erdma_eq.c index 6486234a2360..e4161e85422c 100644 --- a/drivers/infiniband/hw/erdma/erdma_eq.c +++ b/drivers/infiniband/hw/erdma/erdma_eq.c @@ -52,7 +52,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev) if (FIELD_GET(ERDMA_AEQE_HDR_TYPE_MASK, le32_to_cpu(aeqe->hdr)) == ERDMA_AE_TYPE_CQ_ERR) { cqn = le32_to_cpu(aeqe->event_data0); - cq = find_cq_by_cqn(dev, cqn); + cq = erdma_cq_get_by_cqn(dev, cqn); if (!cq) continue; @@ -62,9 +62,10 @@ void erdma_aeq_event_handler(struct erdma_dev *dev) if (cq->ibcq.event_handler) cq->ibcq.event_handler(&event, cq->ibcq.cq_context); + erdma_cq_put(cq); } else { qpn = le32_to_cpu(aeqe->event_data0); - qp = find_qp_by_qpn(dev, qpn); + qp = erdma_qp_get_by_qpn(dev, qpn); if (!qp) continue; @@ -74,6 +75,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev) if (qp->ibqp.event_handler) qp->ibqp.event_handler(&event, qp->ibqp.qp_context); + erdma_qp_put(qp); } } @@ -157,7 +159,7 @@ void erdma_ceq_completion_handler(struct erdma_eq_cb *ceq_cb) poll_cnt++; cqn = FIELD_GET(ERDMA_CEQE_HDR_CQN_MASK, READ_ONCE(*ceqe)); - cq = find_cq_by_cqn(dev, cqn); + cq = erdma_cq_get_by_cqn(dev, cqn); if (!cq) continue; @@ -166,6 +168,7 @@ void erdma_ceq_completion_handler(struct erdma_eq_cb *ceq_cb) if (cq->ibcq.comp_handler) cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context); + erdma_cq_put(cq); } notify_eq(&ceq_cb->eq); @@ -219,6 +222,7 @@ static void erdma_free_ceq_irq(struct erdma_dev *dev, u16 ceqn) irq_set_affinity_hint(eqc->irq.msix_vector, NULL); free_irq(eqc->irq.msix_vector, eqc); + tasklet_kill(&eqc->tasklet); } static int create_eq_cmd(struct erdma_dev *dev, u32 eqn, struct erdma_eq *eq) diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c index 058edc42de58..fdd67756f3bc 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.c +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c @@ -1303,8 +1303,15 @@ int erdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL, true); + /* + * A timeout disables the command queue, so retry cannot succeed. Treat + * terminal command failures as diagnostic; propagating them can make + * forced uverbs cleanup discard the last software resource pointers. + */ if (ret) - return ret; + ibdev_warn_ratelimited(&dev->ibdev, + "failed to deregister MR 0x%x: %d\n", + ibmr->lkey, ret); erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_STAG_IDX], ibmr->lkey >> 8); @@ -1320,6 +1327,7 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) struct erdma_dev *dev = to_edev(ibcq->device); struct erdma_ucontext *ctx = rdma_udata_to_drv_context( udata, struct erdma_ucontext, ibucontext); + unsigned long flags; int err; struct erdma_cmdq_destroy_cq_req req; @@ -1330,7 +1338,16 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL, true); if (err) - return err; + ibdev_warn_ratelimited(&dev->ibdev, + "failed to destroy CQ %u: %d\n", + cq->cqn, err); + + xa_lock_irqsave(&dev->cq_xa, flags); + __xa_erase(&dev->cq_xa, cq->cqn); + xa_unlock_irqrestore(&dev->cq_xa, flags); + + erdma_cq_put(cq); + wait_for_completion(&cq->free); if (rdma_is_kernel_res(&cq->ibcq.res)) { dma_free_coherent(&dev->pdev->dev, cq->depth << CQE_SHIFT, @@ -1342,8 +1359,6 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) put_mtt_entries(dev, &cq->user_cq.qbuf_mem); } - xa_erase(&dev->cq_xa, cq->cqn); - return 0; } @@ -1355,6 +1370,7 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) udata, struct erdma_ucontext, ibucontext); struct erdma_cmdq_destroy_qp_req req; union erdma_mod_qp_params params; + unsigned long flags; int err; down_write(&qp->state_lock); @@ -1378,7 +1394,13 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL, true); if (err) - return err; + ibdev_warn_ratelimited(&dev->ibdev, + "failed to destroy QP %u: %d\n", + QP_ID(qp), err); + + xa_lock_irqsave(&dev->qp_xa, flags); + __xa_erase(&dev->qp_xa, QP_ID(qp)); + xa_unlock_irqrestore(&dev->qp_xa, flags); erdma_qp_put(qp); wait_for_completion(&qp->safe_free); @@ -1393,7 +1415,6 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) if (qp->cep) erdma_cep_put(qp->cep); - xa_erase(&dev->qp_xa, QP_ID(qp)); return 0; } @@ -1970,6 +1991,8 @@ int erdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, cq->ibcq.cqe = depth; cq->depth = depth; cq->assoc_eqn = attr->comp_vector + 1; + refcount_set(&cq->refcount, 1); + init_completion(&cq->free); ret = xa_alloc_cyclic(&dev->cq_xa, &cq->cqn, cq, XA_LIMIT(1, dev->attrs.max_cq - 1), @@ -2282,7 +2305,9 @@ int erdma_destroy_ah(struct ib_ah *ibah, u32 flags) ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL, flags & RDMA_DESTROY_AH_SLEEPABLE); if (ret) - return ret; + ibdev_warn_ratelimited(&dev->ibdev, + "failed to destroy AH %u: %d\n", + ah->ahn, ret); erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_AH], ah->ahn); diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h index 7d8d3fe501d5..c73cecf92f61 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.h +++ b/drivers/infiniband/hw/erdma/erdma_verbs.h @@ -7,6 +7,9 @@ #ifndef __ERDMA_VERBS_H__ #define __ERDMA_VERBS_H__ +#include +#include + #include "erdma.h" /* RDMA Capability. */ @@ -341,6 +344,8 @@ struct erdma_cq { u32 depth; u32 assoc_eqn; + refcount_t refcount; + struct completion free; union { struct erdma_kcq_info kern_cq; @@ -355,9 +360,40 @@ static inline struct erdma_qp *find_qp_by_qpn(struct erdma_dev *dev, int id) return (struct erdma_qp *)xa_load(&dev->qp_xa, id); } -static inline struct erdma_cq *find_cq_by_cqn(struct erdma_dev *dev, int id) +static inline struct erdma_qp *erdma_qp_get_by_qpn(struct erdma_dev *dev, + int id) +{ + struct erdma_qp *qp; + unsigned long flags; + + xa_lock_irqsave(&dev->qp_xa, flags); + qp = xa_load(&dev->qp_xa, id); + if (qp && !kref_get_unless_zero(&qp->ref)) + qp = NULL; + xa_unlock_irqrestore(&dev->qp_xa, flags); + + return qp; +} + +static inline struct erdma_cq *erdma_cq_get_by_cqn(struct erdma_dev *dev, + int id) +{ + struct erdma_cq *cq; + unsigned long flags; + + xa_lock_irqsave(&dev->cq_xa, flags); + cq = xa_load(&dev->cq_xa, id); + if (cq && !refcount_inc_not_zero(&cq->refcount)) + cq = NULL; + xa_unlock_irqrestore(&dev->cq_xa, flags); + + return cq; +} + +static inline void erdma_cq_put(struct erdma_cq *cq) { - return (struct erdma_cq *)xa_load(&dev->cq_xa, id); + if (refcount_dec_and_test(&cq->refcount)) + complete(&cq->free); } void erdma_qp_get(struct erdma_qp *qp); diff --git a/drivers/infiniband/hw/hfi1/chip.c b/drivers/infiniband/hw/hfi1/chip.c index 0781ab756d44..57c70384d210 100644 --- a/drivers/infiniband/hw/hfi1/chip.c +++ b/drivers/infiniband/hw/hfi1/chip.c @@ -15047,7 +15047,7 @@ int hfi1_init_dd(struct hfi1_devdata *dd) */ ret = hfi1_pcie_ddinit(dd, pdev); if (ret < 0) - goto bail_free; + goto bail; /* Save PCI space registers to rewrite after device reset */ ret = save_pci_variables(dd); @@ -15302,8 +15302,6 @@ int hfi1_init_dd(struct hfi1_devdata *dd) bail_cleanup: hfi1_free_rx(dd); hfi1_pcie_ddcleanup(dd); -bail_free: - hfi1_free_devdata(dd); bail: return ret; } diff --git a/drivers/infiniband/hw/hfi1/hfi.h b/drivers/infiniband/hw/hfi1/hfi.h index cb630551cf1a..20942a8ac249 100644 --- a/drivers/infiniband/hw/hfi1/hfi.h +++ b/drivers/infiniband/hw/hfi1/hfi.h @@ -2044,9 +2044,7 @@ struct cc_state *get_cc_state_protected(struct hfi1_pportdata *ppd) /* waiting for an urgent packet to arrive */ #define HFI1_CTXT_WAITING_URG 4 -/* free up any allocated data at closes */ int hfi1_init_dd(struct hfi1_devdata *dd); -void hfi1_free_devdata(struct hfi1_devdata *dd); /* LED beaconing functions */ void hfi1_start_led_override(struct hfi1_pportdata *ppd, unsigned int timeon, diff --git a/drivers/infiniband/hw/hfi1/init.c b/drivers/infiniband/hw/hfi1/init.c index 7e0298c62882..f0b3d5324368 100644 --- a/drivers/infiniband/hw/hfi1/init.c +++ b/drivers/infiniband/hw/hfi1/init.c @@ -630,8 +630,6 @@ void hfi1_init_pportdata(struct pci_dev *pdev, struct hfi1_pportdata *ppd, ppd->sm_trap_qp = 0x0; ppd->sa_qp = 0x1; - ppd->hfi1_wq = NULL; - spin_lock_init(&ppd->cca_timer_lock); for (i = 0; i < OPA_MAX_SLS; i++) { @@ -1163,7 +1161,7 @@ static void finalize_asic_data(struct hfi1_devdata *dd, * It cleans up and frees all data structures set up by * by hfi1_alloc_devdata(). */ -void hfi1_free_devdata(struct hfi1_devdata *dd) +static void hfi1_free_devdata(struct hfi1_devdata *dd) { struct hfi1_asic_data *ad; unsigned long flags; @@ -1227,8 +1225,9 @@ static struct hfi1_devdata *hfi1_alloc_devdata(struct pci_dev *pdev, GFP_KERNEL); if (ret < 0) { dev_err(&pdev->dev, - "Could not allocate unit ID: error %d\n", -ret); - goto bail; + "Could not allocate unit ID: error %pe\n", ERR_PTR(ret)); + rvt_dealloc_device(&dd->verbs_dev.rdi); + return ERR_PTR(ret); } /* @@ -1572,15 +1571,6 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) /* First, lock the non-writable module parameters */ HFI1_CAP_LOCK(); - /* Validate dev ids */ - if (!(ent->device == PCI_DEVICE_ID_INTEL0 || - ent->device == PCI_DEVICE_ID_INTEL1)) { - dev_err(&pdev->dev, "Failing on unknown Intel deviceid 0x%x\n", - ent->device); - ret = -ENODEV; - goto bail; - } - /* Allocate the dd so we can get to work */ dd = hfi1_alloc_devdata(pdev, NUM_IB_PORTS * sizeof(struct hfi1_pportdata)); @@ -1634,17 +1624,17 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) if (ret) goto bail; + ret = create_workqueues(dd); + if (ret) + goto free_devdata; + /* * Do device-specific initialization, function table setup, dd * allocation, etc. */ ret = hfi1_init_dd(dd); if (ret) - goto clean_bail; /* error already printed */ - - ret = create_workqueues(dd); - if (ret) - goto clean_bail; + goto destroy_workqueues; /* error already printed */ /* do the generic initialization */ initfail = hfi1_init(dd, 0); @@ -1657,11 +1647,8 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) * we still create devices, so diags, etc. can be used * to determine cause of problem. */ - if (!initfail && !ret) { + if (!initfail && !ret) dd->flags |= HFI1_INITTED; - /* create debufs files after init and ib register */ - hfi1_dbg_ibdev_init(&dd->verbs_dev); - } j = hfi1_device_create(dd); if (j) @@ -1670,7 +1657,6 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) if (initfail || ret) { msix_clean_up_interrupts(dd); stop_timers(dd); - flush_workqueue(ib_wq); for (pidx = 0; pidx < dd->num_pports; ++pidx) { hfi1_quiet_serdes(dd->pport + pidx); ppd = dd->pport + pidx; @@ -1687,6 +1673,7 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) hfi1_device_remove(dd); if (!ret) hfi1_unregister_ib_device(dd); + hfi1_free_rx(dd); postinit_cleanup(dd); if (initfail) ret = initfail; @@ -1694,10 +1681,14 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) } sdma_start(dd); + hfi1_dbg_ibdev_init(&dd->verbs_dev); return 0; -clean_bail: +destroy_workqueues: + destroy_workqueues(dd); +free_devdata: + hfi1_free_devdata(dd); hfi1_pcie_cleanup(pdev); bail: return ret; @@ -1743,9 +1734,6 @@ static void remove_one(struct pci_dev *pdev) stop_timers(dd); - /* wait until all of our (qsfp) queue_work() calls complete */ - flush_workqueue(ib_wq); - postinit_cleanup(dd); } diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c index 9b1aece1b080..ae8ff7d5a37c 100644 --- a/drivers/infiniband/hw/hfi1/user_sdma.c +++ b/drivers/infiniband/hw/hfi1/user_sdma.c @@ -1028,6 +1028,7 @@ static int set_txreq_header_ahg(struct user_sdma_request *req, struct user_sdma_txreq *tx, u32 datalen) { u32 ahg[AHG_KDETH_ARRAY_SIZE]; + int ret; int idx = 0; u8 omfactor; /* KDETH.OM */ struct hfi1_user_sdma_pkt_q *pq = req->pq; @@ -1132,11 +1133,13 @@ static int set_txreq_header_ahg(struct user_sdma_request *req, trace_hfi1_sdma_user_header_ahg(pq->dd, pq->ctxt, pq->subctxt, req->info.comp_idx, req->sde->this_idx, req->ahg_idx, ahg, idx, tidval); - sdma_txinit_ahg(&tx->txreq, - SDMA_TXREQ_F_USE_AHG, - datalen, req->ahg_idx, idx, - ahg, sizeof(req->hdr), - user_sdma_txreq_cb); + ret = sdma_txinit_ahg(&tx->txreq, + SDMA_TXREQ_F_USE_AHG, + datalen, req->ahg_idx, idx, + ahg, sizeof(req->hdr), + user_sdma_txreq_cb); + if (ret) + return ret; return idx; } diff --git a/drivers/infiniband/hw/irdma/utils.c b/drivers/infiniband/hw/irdma/utils.c index e5e226b34621..835c11d6cc85 100644 --- a/drivers/infiniband/hw/irdma/utils.c +++ b/drivers/infiniband/hw/irdma/utils.c @@ -1169,6 +1169,12 @@ void irdma_free_qp_rsrc(struct irdma_qp *iwqp) iwqp->kqp.dma_mem.va = NULL; kfree(iwqp->kqp.sq_wrid_mem); kfree(iwqp->kqp.rq_wrid_mem); + + if (iwqp->user_mode && iwqp->iwpbl) { + struct irdma_mr *iwmr = iwqp->iwpbl->iwmr; + + refcount_dec(&iwmr->user_ring_refs); + } } /** diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c index 4084168d0194..9ada0bc00bd0 100644 --- a/drivers/infiniband/hw/irdma/verbs.c +++ b/drivers/infiniband/hw/irdma/verbs.c @@ -462,6 +462,9 @@ static struct irdma_pbl *irdma_get_pbl(unsigned long va, list_for_each_entry (iwpbl, pbl_list, list) { if (iwpbl->user_base == va) { + struct irdma_mr *iwmr = iwpbl->iwmr; + + refcount_inc(&iwmr->user_ring_refs); list_del(&iwpbl->list); iwpbl->on_list = false; return iwpbl; @@ -1887,6 +1890,11 @@ static void irdma_srq_free_rsrc(struct irdma_pci_f *rf, struct irdma_srq *iwsrq) dma_free_coherent(rf->sc_dev.hw->device, iwsrq->kmem.size, iwsrq->kmem.va, iwsrq->kmem.pa); iwsrq->kmem.va = NULL; + } else { + /* Not called in any failure path, so iwpbl is valid. */ + struct irdma_mr *iwmr = iwsrq->iwpbl->iwmr; + + refcount_dec(&iwmr->user_ring_refs); } irdma_free_rsrc(rf, rf->allocated_srqs, srq->srq_uk.srq_id); @@ -1909,6 +1917,21 @@ static void irdma_cq_free_rsrc(struct irdma_pci_f *rf, struct irdma_cq *iwcq) iwcq->kmem_shadow.size, iwcq->kmem_shadow.va, iwcq->kmem_shadow.pa); iwcq->kmem_shadow.va = NULL; + } else { + struct irdma_mr *iwmr; + + /* May be called in a failure path before iwpbl is valid. */ + if (iwcq->iwpbl) { + iwmr = iwcq->iwpbl->iwmr; + + refcount_dec(&iwmr->user_ring_refs); + } + + if (iwcq->iwpbl_shadow) { + iwmr = iwcq->iwpbl_shadow->iwmr; + + refcount_dec(&iwmr->user_ring_refs); + } } irdma_free_rsrc(rf, rf->allocated_cqs, cq->cq_uk.cq_id); @@ -2024,7 +2047,7 @@ static int irdma_resize_cq(struct ib_cq *ibcq, int entries, struct irdma_modify_cq_info info = {}; struct irdma_dma_mem kmem_buf; struct irdma_cq_mr *cqmr_buf; - struct irdma_pbl *iwpbl_buf; + struct irdma_pbl *iwpbl_buf = NULL; struct irdma_device *iwdev; struct irdma_pci_f *rf; struct irdma_cq_buf *cq_buf = NULL; @@ -2130,6 +2153,19 @@ static int irdma_resize_cq(struct ib_cq *ibcq, int entries, goto error; spin_lock_irqsave(&iwcq->lock, flags); + if (udata) { + struct irdma_pbl *old_iwpbl = iwcq->iwpbl; + + /* Only update if the resize was successful. Otherwise, HW is + * still pointing to the old PBL. + */ + iwcq->iwpbl = iwpbl_buf; + if (old_iwpbl) { + struct irdma_mr *old_iwmr = old_iwpbl->iwmr; + + refcount_dec(&old_iwmr->user_ring_refs); + } + } if (cq_buf) { cq_buf->kmem_buf = iwcq->kmem; cq_buf->hw = dev->hw; @@ -2145,6 +2181,11 @@ static int irdma_resize_cq(struct ib_cq *ibcq, int entries, return 0; error: + if (iwpbl_buf) { + struct irdma_mr *iwmr = iwpbl_buf->iwmr; + + refcount_dec(&iwmr->user_ring_refs); + } if (!udata) { dma_free_coherent(dev->hw->device, kmem_buf.size, kmem_buf.va, kmem_buf.pa); @@ -2422,6 +2463,11 @@ static int irdma_create_srq(struct ib_srq *ibsrq, dma_free_coherent(rf->hw.device, iwsrq->kmem.size, iwsrq->kmem.va, iwsrq->kmem.pa); free_rsrc: + if (iwsrq->user_mode && iwsrq->iwpbl) { + struct irdma_mr *iwmr = iwsrq->iwpbl->iwmr; + + refcount_dec(&iwmr->user_ring_refs); + } irdma_free_rsrc(rf, rf->allocated_srqs, iwsrq->srq_num); return err_code; } @@ -2500,6 +2546,8 @@ static int irdma_create_cq(struct ib_cq *ibcq, INIT_LIST_HEAD(&iwcq->resize_list); INIT_LIST_HEAD(&iwcq->cmpl_generated); iwcq->cq_num = cq_num; + iwcq->iwpbl = NULL; + iwcq->iwpbl_shadow = NULL; info.dev = dev; ukinfo->cq_size = max(entries, 4); ukinfo->cq_id = cq_num; @@ -2518,8 +2566,6 @@ static int irdma_create_cq(struct ib_cq *ibcq, struct irdma_ucontext *ucontext; struct irdma_create_cq_req req = {}; struct irdma_cq_mr *cqmr; - struct irdma_pbl *iwpbl; - struct irdma_pbl *iwpbl_shadow; struct irdma_cq_mr *cqmr_shadow; iwcq->user_mode = true; @@ -2533,34 +2579,34 @@ static int irdma_create_cq(struct ib_cq *ibcq, } spin_lock_irqsave(&ucontext->cq_reg_mem_list_lock, flags); - iwpbl = irdma_get_pbl((unsigned long)req.user_cq_buf, - &ucontext->cq_reg_mem_list); + iwcq->iwpbl = irdma_get_pbl((unsigned long)req.user_cq_buf, + &ucontext->cq_reg_mem_list); spin_unlock_irqrestore(&ucontext->cq_reg_mem_list_lock, flags); - if (!iwpbl) { + if (!iwcq->iwpbl) { err_code = -EPROTO; goto cq_free_rsrc; } - cqmr = &iwpbl->cq_mr; + cqmr = &iwcq->iwpbl->cq_mr; if (rf->sc_dev.hw_attrs.uk_attrs.feature_flags & IRDMA_FEATURE_CQ_RESIZE) { spin_lock_irqsave(&ucontext->cq_reg_mem_list_lock, flags); - iwpbl_shadow = irdma_get_pbl( + iwcq->iwpbl_shadow = irdma_get_pbl( (unsigned long)req.user_shadow_area, &ucontext->cq_reg_mem_list); spin_unlock_irqrestore(&ucontext->cq_reg_mem_list_lock, flags); - if (!iwpbl_shadow) { + if (!iwcq->iwpbl_shadow) { err_code = -EPROTO; goto cq_free_rsrc; } - cqmr_shadow = &iwpbl_shadow->cq_mr; + cqmr_shadow = &iwcq->iwpbl_shadow->cq_mr; info.shadow_area_pa = cqmr_shadow->cq_pbl.addr; } else { info.shadow_area_pa = cqmr->shadow; } - if (iwpbl->pbl_allocated) { + if (iwcq->iwpbl->pbl_allocated) { info.virtual_map = true; info.pbl_chunk_size = 1; info.first_pm_pbl_idx = cqmr->cq_pbl.idx; @@ -3363,6 +3409,7 @@ static struct irdma_mr *irdma_alloc_iwmr(struct ib_umem *region, if (!iwmr) return ERR_PTR(-ENOMEM); + refcount_set(&iwmr->user_ring_refs, 1); iwpbl = &iwmr->iwpbl; iwpbl->iwmr = iwmr; iwmr->region = region; @@ -3862,41 +3909,41 @@ static struct ib_mr *irdma_get_dma_mr(struct ib_pd *pd, int acc) * irdma_del_memlist - Deleting pbl list entries for CQ/QP * @iwmr: iwmr for IB's user page addresses * @ucontext: ptr to user context + * + * Return: True if the MR is currently in-use by a QP/CQ/SRQ ring. */ -static void irdma_del_memlist(struct irdma_mr *iwmr, +static bool irdma_del_memlist(struct irdma_mr *iwmr, struct irdma_ucontext *ucontext) { struct irdma_pbl *iwpbl = &iwmr->iwpbl; unsigned long flags; + spinlock_t *lock; + bool in_use = false; switch (iwmr->type) { case IRDMA_MEMREG_TYPE_CQ: - spin_lock_irqsave(&ucontext->cq_reg_mem_list_lock, flags); - if (iwpbl->on_list) { - iwpbl->on_list = false; - list_del(&iwpbl->list); - } - spin_unlock_irqrestore(&ucontext->cq_reg_mem_list_lock, flags); + lock = &ucontext->cq_reg_mem_list_lock; break; case IRDMA_MEMREG_TYPE_QP: - spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags); - if (iwpbl->on_list) { - iwpbl->on_list = false; - list_del(&iwpbl->list); - } - spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags); + lock = &ucontext->qp_reg_mem_list_lock; break; case IRDMA_MEMREG_TYPE_SRQ: - spin_lock_irqsave(&ucontext->srq_reg_mem_list_lock, flags); - if (iwpbl->on_list) { - iwpbl->on_list = false; - list_del(&iwpbl->list); - } - spin_unlock_irqrestore(&ucontext->srq_reg_mem_list_lock, flags); + lock = &ucontext->srq_reg_mem_list_lock; break; default: - break; + return false; } + + spin_lock_irqsave(lock, flags); + if (!refcount_dec_if_one(&iwmr->user_ring_refs)) { + in_use = true; + } else if (iwpbl->on_list) { + iwpbl->on_list = false; + list_del(&iwpbl->list); + } + spin_unlock_irqrestore(lock, flags); + + return in_use; } /** @@ -3918,7 +3965,12 @@ static int irdma_dereg_mr(struct ib_mr *ib_mr, struct ib_udata *udata) ucontext = rdma_udata_to_drv_context(udata, struct irdma_ucontext, ibucontext); - irdma_del_memlist(iwmr, ucontext); + + /* Do not allow the MR to be unpinned if it is still + * backing a user ring. + */ + if (irdma_del_memlist(iwmr, ucontext)) + return -EBUSY; } goto done; } diff --git a/drivers/infiniband/hw/irdma/verbs.h b/drivers/infiniband/hw/irdma/verbs.h index 289ebc9b23ca..a1651641eb71 100644 --- a/drivers/infiniband/hw/irdma/verbs.h +++ b/drivers/infiniband/hw/irdma/verbs.h @@ -120,6 +120,7 @@ struct irdma_mr { u64 len; u64 pgaddrmem[IRDMA_MAX_SAVED_PHY_PGADDR]; struct irdma_pbl iwpbl; + refcount_t user_ring_refs; }; struct irdma_srq { @@ -152,6 +153,8 @@ struct irdma_cq { struct list_head resize_list; struct irdma_cq_poll_info cur_cqe; struct list_head cmpl_generated; + struct irdma_pbl *iwpbl; + struct irdma_pbl *iwpbl_shadow; }; struct irdma_cmpl_gen { diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c index f00bf3b015e7..285b3702198b 100644 --- a/drivers/infiniband/hw/mana/qp.c +++ b/drivers/infiniband/hw/mana/qp.c @@ -459,6 +459,12 @@ static void mana_table_remove_rc_qp(struct mana_ib_dev *mdev, struct mana_ib_qp xa_erase_irq(&mdev->qp_table_wq, qp->ibqp.qp_num); } +static void mana_table_drain_qp_ref(struct mana_ib_qp *qp) +{ + mana_put_qp_ref(qp); + wait_for_completion(&qp->free); +} + static int mana_table_store_ud_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *qp) { u32 qids = qp->ud_qp.queues[MANA_UD_SEND_QUEUE].id | MANA_SENDQ_MASK; @@ -477,6 +483,7 @@ static int mana_table_store_ud_qp(struct mana_ib_dev *mdev, struct mana_ib_qp *q remove_sq: xa_erase_irq(&mdev->qp_table_wq, qids); + mana_table_drain_qp_ref(qp); return err; } @@ -524,8 +531,7 @@ static void mana_table_remove_qp(struct mana_ib_dev *mdev, qp->ibqp.qp_type); return; } - mana_put_qp_ref(qp); - wait_for_completion(&qp->free); + mana_table_drain_qp_ref(qp); } static int mana_ib_create_rc_qp(struct ib_qp *ibqp, struct ib_pd *ibpd, diff --git a/drivers/infiniband/hw/mlx5/cong.c b/drivers/infiniband/hw/mlx5/cong.c index a78a067e3ce7..e3c4b1c92816 100644 --- a/drivers/infiniband/hw/mlx5/cong.c +++ b/drivers/infiniband/hw/mlx5/cong.c @@ -361,7 +361,7 @@ static int mlx5_ib_set_cc_params(struct mlx5_ib_dev *dev, u32 port_num, MLX5_SET(field_select_r_roce_rp, field, field_select_r_roce_rp, attr_mask); - err = mlx5_cmd_exec_in(dev->mdev, modify_cong_params, in); + err = mlx5_cmd_exec_in(mdev, modify_cong_params, in); kvfree(in); alloc_err: mlx5_ib_put_native_port_mdev(dev, port_num + 1); @@ -399,15 +399,13 @@ static ssize_t get_param(struct file *filp, char __user *buf, size_t count, int offset = param->offset; u32 var = 0; int ret; - char lbuf[11]; + char lbuf[12]; ret = mlx5_ib_get_cc_params(param->dev, param->port_num, offset, &var); if (ret) return ret; - ret = snprintf(lbuf, sizeof(lbuf), "%d\n", var); - if (ret < 0) - return ret; + ret = scnprintf(lbuf, sizeof(lbuf), "%u\n", var); return simple_read_from_buffer(buf, count, pos, lbuf, ret); } diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index 09709ae5bd47..7c82518e9f04 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -1585,6 +1585,130 @@ static int mlx5_ib_rep_query_pkey(struct ib_device *ibdev, u32 port, u16 index, return 0; } +static int mlx5_ib_query_port_speed_from_port(struct mlx5_ib_dev *dev, + u32 port_num, u64 *speed) +{ + struct ib_port_speed_info speed_info; + struct ib_port_attr attr = {}; + int err; + + err = mlx5_ib_query_port(&dev->ib_dev, port_num, &attr); + if (err) + return err; + + if (attr.state == IB_PORT_DOWN) { + *speed = 0; + return 0; + } + + err = ib_port_attr_to_speed_info(&attr, &speed_info); + if (err) + return err; + + *speed = speed_info.rate; + return 0; +} + +static int mlx5_ib_query_port_speed_from_vport(struct mlx5_core_dev *mdev, + u8 op_mod, u16 vport, + u8 other_vport, u64 *speed, + struct mlx5_ib_dev *dev, + u32 port_num) +{ + u32 max_tx_speed; + u8 vport_state; + int err; + + err = mlx5_query_vport_max_tx_speed(mdev, op_mod, vport, other_vport, + &max_tx_speed, &vport_state); + if (err) + return err; + + if (vport_state == VPORT_STATE_DOWN || max_tx_speed == 0) + /* Value 0 indicates field not supported, fallback */ + return mlx5_ib_query_port_speed_from_port(dev, port_num, + speed); + + *speed = max_tx_speed; + return 0; +} + +static int mlx5_ib_query_port_speed_from_bond(struct mlx5_ib_dev *dev, + u32 port_num, u64 *speed) +{ + struct mlx5_core_dev *mdev = dev->mdev; + u32 bond_speed; + int err; + + err = mlx5_lag_query_bond_speed(mdev, &bond_speed); + if (err) + return err; + + *speed = bond_speed / MLX5_MAX_TX_SPEED_UNIT; + + return 0; +} + +static int mlx5_ib_query_port_speed_non_rep(struct mlx5_ib_dev *dev, + u32 port_num, u64 *speed) +{ + u16 op_mod = MLX5_VPORT_STATE_OP_MOD_VNIC_VPORT; + + if (mlx5_lag_is_roce(dev->mdev)) + return mlx5_ib_query_port_speed_from_bond(dev, port_num, + speed); + + return mlx5_ib_query_port_speed_from_vport(dev->mdev, op_mod, 0, false, + speed, dev, port_num); +} + +static int mlx5_ib_query_port_speed_rep(struct mlx5_ib_dev *dev, u32 port_num, + u64 *speed) +{ + struct mlx5_eswitch_rep *rep; + struct mlx5_core_dev *mdev; + u16 op_mod; + + if (!dev->port[port_num - 1].rep) { + mlx5_ib_warn(dev, "Representor doesn't exist for port %u\n", + port_num); + return -EINVAL; + } + + rep = dev->port[port_num - 1].rep; + mdev = mlx5_eswitch_get_core_dev(rep->esw); + if (!mdev) + return -ENODEV; + + if (rep->vport == MLX5_VPORT_UPLINK) { + if (mlx5_lag_is_sriov(mdev)) + return mlx5_ib_query_port_speed_from_bond(dev, + port_num, + speed); + + return mlx5_ib_query_port_speed_from_port(dev, port_num, + speed); + } + + op_mod = MLX5_VPORT_STATE_OP_MOD_ESW_VPORT; + return mlx5_ib_query_port_speed_from_vport(dev->mdev, op_mod, + rep->vport, true, speed, dev, + port_num); +} + +int mlx5_ib_query_port_speed(struct ib_device *ibdev, u32 port_num, u64 *speed) +{ + struct mlx5_ib_dev *dev = to_mdev(ibdev); + + if (mlx5_ib_port_link_layer(ibdev, port_num) == + IB_LINK_LAYER_INFINIBAND || mlx5_core_mp_enabled(dev->mdev)) + return mlx5_ib_query_port_speed_from_port(dev, port_num, speed); + else if (!dev->is_rep) + return mlx5_ib_query_port_speed_non_rep(dev, port_num, speed); + else + return mlx5_ib_query_port_speed_rep(dev, port_num, speed); +} + static int mlx5_ib_query_gid(struct ib_device *ibdev, u32 port, int index, union ib_gid *gid) { @@ -4359,6 +4483,7 @@ static const struct ib_device_ops mlx5_ib_dev_ops = { .query_device = mlx5_ib_query_device, .query_gid = mlx5_ib_query_gid, .query_pkey = mlx5_ib_query_pkey, + .query_port_speed = mlx5_ib_query_port_speed, .query_qp = mlx5_ib_query_qp, .query_srq = mlx5_ib_query_srq, .query_ucontext = mlx5_ib_query_ucontext, @@ -5225,6 +5350,7 @@ static int __init mlx5_ib_init(void) dd_err: mlx5r_rep_cleanup(); rep_err: + rcu_barrier(); mlx5_ib_qp_event_cleanup(); qp_event_err: destroy_workqueue(mlx5_ib_event_wq); @@ -5238,6 +5364,7 @@ static void __exit mlx5_ib_cleanup(void) auxiliary_driver_unregister(&mlx5r_driver); auxiliary_driver_unregister(&mlx5r_mp_driver); mlx5r_rep_cleanup(); + rcu_barrier(); mlx5_ib_qp_event_cleanup(); destroy_workqueue(mlx5_ib_event_wq); diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index ba55e6db3bca..20591530a17b 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1446,6 +1446,8 @@ int mlx5_query_mad_ifc_port(struct ib_device *ibdev, u32 port, struct ib_port_attr *props); int mlx5_ib_query_port(struct ib_device *ibdev, u32 port, struct ib_port_attr *props); +int mlx5_ib_query_port_speed(struct ib_device *ibdev, u32 port_num, + u64 *speed); void mlx5_ib_populate_pas(struct ib_umem *umem, size_t page_size, __be64 *pas, u64 access_flags); int mlx5_ib_get_cqe_size(struct ib_cq *ibcq); diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index 02a0f4920cab..50b20264c4bd 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -637,6 +637,7 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev, struct ib_qp_init_attr *attr) { int desc_sz = 1 << qp->sq.wqe_shift; + int rq_buf_size, sq_buf_size; if (desc_sz > MLX5_CAP_GEN(dev->mdev, max_wqe_sz_sq)) { mlx5_ib_warn(dev, "desc_sz %d, max_sq_desc_sz %d\n", @@ -661,11 +662,21 @@ static int set_user_buf_size(struct mlx5_ib_dev *dev, if (attr->qp_type == IB_QPT_RAW_PACKET || qp->flags & IB_QP_CREATE_SOURCE_QPN) { - base->ubuffer.buf_size = qp->rq.wqe_cnt << qp->rq.wqe_shift; - qp->raw_packet_qp.sq.ubuffer.buf_size = qp->sq.wqe_cnt << 6; + if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift, + &base->ubuffer.buf_size)) + return -EINVAL; + if (check_shl_overflow(qp->sq.wqe_cnt, 6, + &qp->raw_packet_qp.sq.ubuffer.buf_size)) + return -EINVAL; } else { - base->ubuffer.buf_size = (qp->rq.wqe_cnt << qp->rq.wqe_shift) + - (qp->sq.wqe_cnt << 6); + if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift, + &rq_buf_size)) + return -EINVAL; + if (check_shl_overflow(qp->sq.wqe_cnt, 6, &sq_buf_size)) + return -EINVAL; + if (check_add_overflow(rq_buf_size, sq_buf_size, + &base->ubuffer.buf_size)) + return -EINVAL; } return 0; @@ -989,7 +1000,11 @@ static int _create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd, qp->rq.offset = 0; qp->sq.wqe_shift = ilog2(MLX5_SEND_WQE_BB); - qp->sq.offset = qp->rq.wqe_cnt << qp->rq.wqe_shift; + if (check_shl_overflow(qp->rq.wqe_cnt, qp->rq.wqe_shift, + &qp->sq.offset)) { + err = -EINVAL; + goto err_bfreg; + } err = set_user_buf_size(dev, qp, ucmd, base, attr); if (err) diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c index 4d4e3b324dd2..16b3fbdf2f86 100644 --- a/drivers/infiniband/sw/rxe/rxe_odp.c +++ b/drivers/infiniband/sw/rxe/rxe_odp.c @@ -115,6 +115,7 @@ int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, err = rxe_odp_init_pages(mr); if (err) { ib_umem_odp_release(umem_odp); + mr->umem = NULL; return err; } diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 12d03f390b09..24f5c044363f 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -701,6 +701,21 @@ int rxe_requester(struct rxe_qp *qp) if (unlikely(!wqe)) goto exit; + /* + * Don't trust user space data: a user QP's WQE comes from an mmap'd + * ring, so num_sge/cur_sge are attacker-controlled. Bound num_sge like + * get_srq_wqe(); bound cur_sge only when payload exists (dma.resid), + * since copy_data() skips dma->sge[] on a zero-length copy (all a + * max_sge == 0 QP can post). + */ + if (unlikely(wqe->dma.num_sge > qp->sq.max_sge || + (wqe->dma.resid && + wqe->dma.cur_sge >= qp->sq.max_sge))) { + rxe_dbg_qp(qp, "invalid num_sge/cur_sge in send wqe\n"); + wqe->status = IB_WC_LOC_QP_OP_ERR; + goto err; + } + if (rxe_wqe_is_fenced(qp, wqe)) { qp->req.wait_fence = 1; goto exit; diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 995805e16d78..6aabf1d07ea3 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -1203,7 +1203,14 @@ static enum resp_states do_complete(struct rxe_qp *qp, spin_lock_irqsave(&qp->state_lock, flags); if (unlikely(qp_state(qp) == IB_QPS_ERR)) { spin_unlock_irqrestore(&qp->state_lock, flags); - return RESPST_CHK_RESOURCE; + /* The packet was executed and completed before the QP + * moved to ERROR; it must be consumed exactly once. + * Re-entering the request chain with the stale packet + * would copy it into every remaining recv WQE as a new + * completion. Remaining WQEs are flushed by the drain + * path at rxe_receiver() entry. + */ + return pkt ? RESPST_CLEANUP : RESPST_CHK_RESOURCE; } spin_unlock_irqrestore(&qp->state_lock, flags); diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 49ff121f77fe..bb7d90963907 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -1701,7 +1701,7 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) cep->state = SIW_EPSTATE_CLOSED; siw_free_cm_id(cep); - if (qp->cep) { + if (qp->cep == cep) { siw_cep_put(cep); qp->cep = NULL; } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index 5b4d76e97437..b897c1c3e4c8 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -2755,6 +2755,7 @@ static int __init ipoib_init_module(void) err_sa: ib_sa_unregister_client(&ipoib_sa_client); destroy_workqueue(ipoib_workqueue); + rcu_barrier(); err_fs: ipoib_unregister_debugfs(); @@ -2772,6 +2773,7 @@ static void __exit ipoib_cleanup_module(void) ib_sa_unregister_client(&ipoib_sa_client); ipoib_unregister_debugfs(); destroy_workqueue(ipoib_workqueue); + rcu_barrier(); } module_init(ipoib_init_module); diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 640634f96d72..6483a55170cd 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -59,6 +59,8 @@ static void isert_recv_done(struct ib_cq *cq, struct ib_wc *wc); static void isert_send_done(struct ib_cq *cq, struct ib_wc *wc); static void isert_login_recv_done(struct ib_cq *cq, struct ib_wc *wc); static void isert_login_send_done(struct ib_cq *cq, struct ib_wc *wc); +static void isert_unmap_tx_desc(struct iser_tx_desc *tx_desc, + struct ib_device *ib_dev); static int isert_sg_tablesize_set(const char *val, const struct kernel_param *kp) { @@ -498,6 +500,8 @@ isert_connect_release(struct isert_conn *isert_conn) if (isert_conn->qp) isert_destroy_qp(isert_conn); + isert_unmap_tx_desc(&isert_conn->login_tx_desc, device->ib_device); + if (isert_conn->login_desc) isert_free_login_buf(isert_conn); @@ -945,27 +949,16 @@ isert_put_login_tx(struct iscsit_conn *conn, struct iscsi_login *login, } if (!login->login_failed) { if (login->login_complete) { - ret = isert_alloc_rx_descriptors(isert_conn); - if (ret) - return ret; - - ret = isert_post_recvm(isert_conn, - ISERT_QP_MAX_RECV_DTOS); - if (ret) - return ret; - - /* Now we are in FULL_FEATURE phase */ - mutex_lock(&isert_conn->mutex); - isert_conn->state = ISER_CONN_FULL_FEATURE; - mutex_unlock(&isert_conn->mutex); - goto post_send; + /* Posted and sent from isert_get_rx_pdu(). */ + isert_conn->login_rsp_pending = true; + return 0; } ret = isert_login_post_recv(isert_conn); if (ret) return ret; } -post_send: + ret = isert_login_post_send(isert_conn, tx_desc); if (ret) return ret; @@ -973,6 +966,21 @@ isert_put_login_tx(struct iscsit_conn *conn, struct iscsi_login *login, return 0; } +static int +isert_check_login_req(struct isert_conn *isert_conn) +{ + struct iscsi_hdr *hdr = isert_get_iscsi_hdr(isert_conn->login_desc); + u32 dlength = ntoh24(hdr->dlength); + + if (unlikely(dlength > (u32)isert_conn->login_req_len)) { + isert_dbg("login PDU declares %u data bytes but only %d were received\n", + dlength, isert_conn->login_req_len); + return -EINVAL; + } + + return 0; +} + static void isert_rx_login_req(struct isert_conn *isert_conn) { @@ -1335,6 +1343,21 @@ isert_recv_done(struct ib_cq *cq, struct ib_wc *wc) ib_dma_sync_single_for_cpu(ib_dev, rx_desc->dma_addr, ISER_RX_SIZE, DMA_FROM_DEVICE); + /* + * The data segment length declared in the BHS is attacker controlled + * and is used further down to read that many bytes out of the fixed + * size receive descriptor, so it has to be checked against the number + * of bytes that were actually received. Comparing without subtracting + * also rejects PDUs shorter than the iSER and iSCSI headers, which + * would otherwise be parsed out of stale descriptor contents. + */ + if (unlikely(wc->byte_len < ISER_HEADERS_LEN + ntoh24(hdr->dlength))) { + isert_err("PDU declares %u data bytes but only %u bytes were received\n", + ntoh24(hdr->dlength), wc->byte_len); + iscsit_cause_connection_reinstatement(isert_conn->conn, 0); + return; + } + isert_dbg("DMA: 0x%llx, iSCSI opcode: 0x%02x, ITT: 0x%08x, flags: 0x%02x dlen: %d\n", rx_desc->dma_addr, hdr->opcode, hdr->itt, hdr->flags, (int)(wc->byte_len - ISER_HEADERS_LEN)); @@ -1396,8 +1419,12 @@ isert_login_recv_done(struct ib_cq *cq, struct ib_wc *wc) if (isert_conn->conn) { struct iscsi_login *login = isert_conn->conn->conn_login; - if (login && !login->first_request) + if (login && !login->first_request) { + if (isert_check_login_req(isert_conn)) + return; + isert_rx_login_req(isert_conn); + } } mutex_lock(&isert_conn->mutex); @@ -2362,6 +2389,10 @@ isert_get_login_rx(struct iscsit_conn *conn, struct iscsi_login *login) if (!login->first_request) return 0; + ret = isert_check_login_req(isert_conn); + if (ret) + return ret; + isert_rx_login_req(isert_conn); isert_info("before login_comp conn: %p\n", conn); @@ -2587,8 +2618,30 @@ static void isert_free_conn(struct iscsit_conn *conn) static void isert_get_rx_pdu(struct iscsit_conn *conn) { + struct isert_conn *isert_conn = conn->context; struct completion comp; + /* The login timeout timer can fail the login after isert_put_login_tx(). */ + if (!isert_conn->login_rsp_pending) + return; + + isert_conn->login_rsp_pending = false; + + /* The session is registered by now; see isert_put_login_tx(). */ + if (isert_alloc_rx_descriptors(isert_conn)) + return; + + if (isert_post_recvm(isert_conn, ISERT_QP_MAX_RECV_DTOS)) + return; + + /* Now we are in FULL_FEATURE phase */ + mutex_lock(&isert_conn->mutex); + isert_conn->state = ISER_CONN_FULL_FEATURE; + mutex_unlock(&isert_conn->mutex); + + if (isert_login_post_send(isert_conn, &isert_conn->login_tx_desc)) + return; + init_completion(&comp); wait_for_completion_interruptible(&comp); diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h index 0b2dfd6e7e27..0bac5aa66c80 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.h +++ b/drivers/infiniband/ulp/isert/ib_isert.h @@ -178,6 +178,7 @@ struct isert_conn { struct completion login_comp; struct completion login_req_comp; struct iser_tx_desc login_tx_desc; + bool login_rsp_pending; struct rdma_cm_id *cm_id; struct ib_qp *qp; struct ib_cq *cq; diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 7b696a07e603..99131492abf7 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -1943,7 +1943,8 @@ static void srp_process_rsp(struct srp_rdma_ch *ch, struct srp_rsp *rsp, ch->req_lim += be32_to_cpu(rsp->req_lim_delta); if (rsp->tag == ch->tsk_mgmt_tag) { ch->tsk_mgmt_status = -1; - if (be32_to_cpu(rsp->resp_data_len) >= 4) + if (be32_to_cpu(rsp->resp_data_len) >= 4 && + byte_len >= sizeof(*rsp) + 4) ch->tsk_mgmt_status = rsp->data[3]; complete(&ch->tsk_mgmt_done); } else { @@ -2043,13 +2044,20 @@ static int srp_response_common(struct srp_rdma_ch *ch, s32 req_delta, } static void srp_process_cred_req(struct srp_rdma_ch *ch, - struct srp_cred_req *req) + struct srp_cred_req *req, u32 byte_len) { - struct srp_cred_rsp rsp = { - .opcode = SRP_CRED_RSP, - .tag = req->tag, - }; - s32 delta = be32_to_cpu(req->req_lim_delta); + struct srp_cred_rsp rsp = { .opcode = SRP_CRED_RSP }; + s32 delta; + + if (byte_len < sizeof(*req)) { + shost_printk(KERN_ERR, ch->target->scsi_host, PFX + "dropping truncated SRP_CRED_REQ (%u bytes received, %zu expected)\n", + byte_len, sizeof(*req)); + return; + } + + rsp.tag = req->tag; + delta = be32_to_cpu(req->req_lim_delta); if (srp_response_common(ch, delta, &rsp, sizeof(rsp))) shost_printk(KERN_ERR, ch->target->scsi_host, PFX @@ -2057,14 +2065,21 @@ static void srp_process_cred_req(struct srp_rdma_ch *ch, } static void srp_process_aer_req(struct srp_rdma_ch *ch, - struct srp_aer_req *req) + struct srp_aer_req *req, u32 byte_len) { struct srp_target_port *target = ch->target; - struct srp_aer_rsp rsp = { - .opcode = SRP_AER_RSP, - .tag = req->tag, - }; - s32 delta = be32_to_cpu(req->req_lim_delta); + struct srp_aer_rsp rsp = { .opcode = SRP_AER_RSP }; + s32 delta; + + if (byte_len < sizeof(*req)) { + shost_printk(KERN_ERR, target->scsi_host, PFX + "dropping truncated SRP_AER_REQ (%u bytes received, %zu expected)\n", + byte_len, sizeof(*req)); + return; + } + + rsp.tag = req->tag; + delta = be32_to_cpu(req->req_lim_delta); shost_printk(KERN_ERR, target->scsi_host, PFX "ignoring AER for LUN %llu\n", scsilun_to_int(&req->lun)); @@ -2106,11 +2121,11 @@ static void srp_recv_done(struct ib_cq *cq, struct ib_wc *wc) break; case SRP_CRED_REQ: - srp_process_cred_req(ch, iu->buf); + srp_process_cred_req(ch, iu->buf, wc->byte_len); break; case SRP_AER_REQ: - srp_process_aer_req(ch, iu->buf); + srp_process_aer_req(ch, iu->buf, wc->byte_len); break; case SRP_T_LOGOUT: diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index ba70c64a0aa9..212d4c7662c8 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -959,6 +959,7 @@ static int srpt_alloc_rw_ctxs(struct srpt_send_ioctx *ioctx, struct srpt_rdma_ch *ch = ioctx->ch; struct scatterlist *prev = NULL; unsigned prev_nents; + u8 n_rdma, n_rw_ctx; int ret, i; if (nbufs == 1) { @@ -970,6 +971,9 @@ static int srpt_alloc_rw_ctxs(struct srpt_send_ioctx *ioctx, return -ENOMEM; } + n_rw_ctx = ioctx->n_rw_ctx; + n_rdma = ioctx->n_rdma; + for (i = ioctx->n_rw_ctx; i < nbufs; i++, db++) { struct srpt_rw_ctx *ctx = &ioctx->rw_ctxs[i]; u64 remote_addr = be64_to_cpu(db->va); @@ -1016,6 +1020,9 @@ static int srpt_alloc_rw_ctxs(struct srpt_send_ioctx *ioctx, } if (ioctx->rw_ctxs != &ioctx->s_rw_ctx) kfree(ioctx->rw_ctxs); + ioctx->rw_ctxs = NULL; + ioctx->n_rw_ctx = n_rw_ctx; + ioctx->n_rdma = n_rdma; return ret; } @@ -1596,7 +1603,7 @@ static void srpt_handle_cmd(struct srpt_rdma_ch *ch, rc = target_init_cmd(cmd, ch->sess, &send_ioctx->sense_data[0], scsilun_to_int(&srp_cmd->lun), data_len, - TCM_SIMPLE_TAG, dir, TARGET_SCF_ACK_KREF); + cmd->sam_task_attr, dir, TARGET_SCF_ACK_KREF); if (rc != 0) { pr_debug("target_submit_cmd() returned %d for tag %#llx\n", rc, srp_cmd->tag); diff --git a/drivers/iommu/amd/debugfs.c b/drivers/iommu/amd/debugfs.c index 3909a1fb218e..f7bb551d285f 100644 --- a/drivers/iommu/amd/debugfs.c +++ b/drivers/iommu/amd/debugfs.c @@ -176,19 +176,13 @@ static ssize_t devid_write(struct file *filp, const char __user *ubuf, kfree(srcid_ptr); return -ENODEV; } - break; - } - - if (pci_seg->id != seg) { + sbdf = PCI_SEG_DEVID_TO_SBDF(seg, devid); kfree(srcid_ptr); - return -EINVAL; + return cnt; } - sbdf = PCI_SEG_DEVID_TO_SBDF(seg, devid); - kfree(srcid_ptr); - - return cnt; + return -EINVAL; } static int devid_show(struct seq_file *m, void *unused) diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 69170146d442..6bb9d3661bae 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -3059,7 +3059,7 @@ static void disable_iommus(void) * disable suspend until real resume implemented */ -static void amd_iommu_resume(void) +static void amd_iommu_resume(void *data) { struct amd_iommu *iommu; @@ -3074,7 +3074,7 @@ static void amd_iommu_resume(void) amd_iommu_enable_interrupts(); } -static int amd_iommu_suspend(void) +static int amd_iommu_suspend(void *data) { /* disable IOMMUs to go out of the way for BIOS */ disable_iommus(); @@ -3082,11 +3082,15 @@ static int amd_iommu_suspend(void) return 0; } -static struct syscore_ops amd_iommu_syscore_ops = { +static const struct syscore_ops amd_iommu_syscore_ops = { .suspend = amd_iommu_suspend, .resume = amd_iommu_resume, }; +static struct syscore amd_iommu_syscore = { + .ops = &amd_iommu_syscore_ops, +}; + static void __init free_iommu_resources(void) { free_iommu_all(); @@ -3096,14 +3100,31 @@ static void __init free_iommu_resources(void) /* SB IOAPIC is always on this device in AMD systems */ #define IOAPIC_SB_DEVID ((0x00 << 8) | PCI_DEVFN(0x14, 0)) +/* SB IOAPIC for Hygon family 18h model 4h is on the device 0xb */ +#define IOAPIC_SB_DEVID_FAM18H_M4H ((0x00 << 8) | PCI_DEVFN(0xb, 0)) + +/* + * The Southbridge IOAPIC is assigned a GSI Base of 0 (handling interrupts + * 0 through 23). + */ +static int __init get_sb_ioapic_id(void) +{ + int idx = mp_find_ioapic(0); + + if (idx < 0) + return -ENODEV; + + return mpc_ioapic_id(idx); +} + static bool __init check_ioapic_information(void) { const char *fw_bug = FW_BUG; bool ret, has_sb_ioapic; - int idx; + int idx, sb_apicid; has_sb_ioapic = false; - ret = false; + ret = true; /* * If we have map overrides on the kernel command line the @@ -3113,6 +3134,16 @@ static bool __init check_ioapic_information(void) if (cmdline_maps) fw_bug = ""; + sb_apicid = get_sb_ioapic_id(); + if (sb_apicid < 0) { + /* + * Lack of SB IOAPIC registration is not a firmware bug, + * e.g. kernel booted with noapic or noacpi. + */ + fw_bug = ""; + goto out; + } + for (idx = 0; idx < nr_ioapics; idx++) { int devid, id = mpc_ioapic_id(idx); @@ -3121,12 +3152,16 @@ static bool __init check_ioapic_information(void) pr_err("%s: IOAPIC[%d] not in IVRS table\n", fw_bug, id); ret = false; - } else if (devid == IOAPIC_SB_DEVID) { + } else if (id == sb_apicid && (devid == IOAPIC_SB_DEVID || + (boot_cpu_data.x86_vendor == X86_VENDOR_HYGON && + boot_cpu_data.x86 == 0x18 && + boot_cpu_data.x86_model >= 0x4 && + boot_cpu_data.x86_model <= 0xf && + devid == IOAPIC_SB_DEVID_FAM18H_M4H))) { has_sb_ioapic = true; - ret = true; } } - +out: if (!has_sb_ioapic) { /* * We expect the SB IOAPIC to be listed in the IVRS @@ -3137,6 +3172,7 @@ static bool __init check_ioapic_information(void) * device id for the IOAPIC in the system. */ pr_err("%s: No southbridge IOAPIC found\n", fw_bug); + ret = false; } if (!ret) @@ -3458,7 +3494,7 @@ static int __init state_next(void) init_state = IOMMU_ENABLED; break; case IOMMU_ENABLED: - register_syscore_ops(&amd_iommu_syscore_ops); + register_syscore(&amd_iommu_syscore); iommu_snp_enable(); amd_iommu_apply_erratum_snp(); @@ -3576,12 +3612,12 @@ int __init amd_iommu_enable(void) void amd_iommu_disable(void) { - amd_iommu_suspend(); + amd_iommu_suspend(NULL); } int amd_iommu_reenable(int mode) { - amd_iommu_resume(); + amd_iommu_resume(NULL); return 0; } diff --git a/drivers/iommu/amd/ppr.c b/drivers/iommu/amd/ppr.c index 80369ca1e316..1dde19a79e80 100644 --- a/drivers/iommu/amd/ppr.c +++ b/drivers/iommu/amd/ppr.c @@ -140,7 +140,7 @@ static void iommu_call_iopf_notifier(struct amd_iommu *iommu, u64 *raw) if (event.fault.prm.pasid == 0 || event.fault.prm.pasid >= dev_data->max_pasids) { pr_info_ratelimited("Invalid PASID : 0x%x, device : 0x%x\n", - event.fault.prm.pasid, pdev->dev.id); + event.fault.prm.pasid, dev_data->devid); goto out; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 01f448acd8dc..b677390e95bb 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -1097,9 +1097,9 @@ VISIBLE_IF_KUNIT void arm_smmu_get_ste_update_safe(const __le64 *cur, const __le64 *target, __le64 *safe_bits) { - const __le64 eats_s1chk = + const u64 eats_s1chk = FIELD_PREP(STRTAB_STE_1_EATS, STRTAB_STE_1_EATS_S1CHK); - const __le64 eats_trans = + const u64 eats_trans = FIELD_PREP(STRTAB_STE_1_EATS, STRTAB_STE_1_EATS_TRANS); /* @@ -3992,8 +3992,9 @@ static int arm_smmu_write_reg_sync(struct arm_smmu_device *smmu, u32 val, u32 reg; writel_relaxed(val, smmu->base + reg_off); - return readl_relaxed_poll_timeout(smmu->base + ack_off, reg, reg == val, - 1, ARM_SMMU_POLL_TIMEOUT_US); + return readl_relaxed_poll_timeout_atomic(smmu->base + ack_off, reg, + reg == val, 1, + ARM_SMMU_POLL_TIMEOUT_US); } /* GBPA is "special" */ diff --git a/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c b/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c index cee489855311..2b55a221c35a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c +++ b/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c @@ -322,16 +322,30 @@ static void tegra241_vintf0_handle_error(struct tegra241_vintf *vintf) while (map) { unsigned long lidx = __ffs64(map); - struct tegra241_vcmdq *vcmdq = vintf->lvcmdqs[lidx]; - u32 gerror = readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, GERROR)); + struct tegra241_vcmdq *vcmdq; + u32 gerror; + map &= ~BIT_ULL(lidx); + + /* Pairs with smp_store_release() publishing it */ + vcmdq = smp_load_acquire(&vintf->lvcmdqs[lidx]); + if (!vcmdq) + continue; + + gerror = readl_relaxed(REG_VCMDQ_PAGE0(vcmdq, GERROR)); __arm_smmu_cmdq_skip_err(&vintf->cmdqv->smmu, &vcmdq->cmdq); writel(gerror, REG_VCMDQ_PAGE0(vcmdq, GERRORN)); - map &= ~BIT_ULL(lidx); } } } +/* + * The CMDQV error interrupt is edge-triggered, so a pending VINTF error fires + * this ISR once and does not re-assert. An unacked guest therefore cannot + * storm the host. The HW latches and forwards each new error event on its + * own, so an already-set ERR_MAP bit does not suppress the interrupt for a + * new error. + */ static irqreturn_t tegra241_cmdqv_isr(int irq, void *devid) { struct tegra241_cmdqv *cmdqv = (struct tegra241_cmdqv *)devid; @@ -354,16 +368,27 @@ static irqreturn_t tegra241_cmdqv_isr(int irq, void *devid) /* Handle VINTF0 and its LVCMDQs */ if (vintf_map & BIT_ULL(0)) { - tegra241_vintf0_handle_error(cmdqv->vintfs[0]); + struct tegra241_vintf *vintf0; + vintf_map &= ~BIT_ULL(0); + + /* NULL until tegra241_cmdqv_init_structures() publishes it */ + vintf0 = smp_load_acquire(&cmdqv->vintfs[0]); + if (vintf0) + tegra241_vintf0_handle_error(vintf0); } /* Handle other user VINTFs and their LVCMDQs */ while (vintf_map) { unsigned long idx = __ffs64(vintf_map); + struct tegra241_vintf *vintf; - tegra241_vintf_user_handle_error(cmdqv->vintfs[idx]); vintf_map &= ~BIT_ULL(idx); + + /* The slot may be published or torn down (NULL'd) concurrently */ + vintf = smp_load_acquire(&cmdqv->vintfs[idx]); + if (vintf) + tegra241_vintf_user_handle_error(vintf); } return IRQ_HANDLED; @@ -670,7 +695,6 @@ static int tegra241_vintf_init_lvcmdq(struct tegra241_vintf *vintf, u16 lidx, vcmdq->page0 = cmdqv->base + TEGRA241_VINTFi_LVCMDQ_PAGE0(idx, lidx); vcmdq->page1 = cmdqv->base + TEGRA241_VINTFi_LVCMDQ_PAGE1(idx, lidx); - vintf->lvcmdqs[lidx] = vcmdq; return 0; } @@ -686,7 +710,7 @@ static void tegra241_vintf_free_lvcmdq(struct tegra241_vintf *vintf, u16 lidx) dev_dbg(vintf->cmdqv->dev, "%sdeallocated\n", lvcmdq_error_header(vcmdq, header, 64)); /* Guest-owned VCMDQ is free-ed with hw_queue by iommufd core */ - if (vcmdq->vintf->hyp_own) + if (!vcmdq->vintf->idx) kfree(vcmdq); } @@ -709,14 +733,15 @@ tegra241_vintf_alloc_lvcmdq(struct tegra241_vintf *vintf, u16 lidx) /* Build an arm_smmu_cmdq for each LVCMDQ */ ret = tegra241_vcmdq_alloc_smmu_cmdq(vcmdq); if (ret) - goto deinit_lvcmdq; + goto free_vcmdq; + + /* Pairs with the smp_load_acquire() in the error ISR */ + smp_store_release(&vintf->lvcmdqs[lidx], vcmdq); dev_dbg(cmdqv->dev, "%sallocated\n", lvcmdq_error_header(vcmdq, header, 64)); return vcmdq; -deinit_lvcmdq: - tegra241_vintf_deinit_lvcmdq(vintf, lidx); free_vcmdq: kfree(vcmdq); return ERR_PTR(ret); @@ -727,8 +752,18 @@ tegra241_vintf_alloc_lvcmdq(struct tegra241_vintf *vintf, u16 lidx) static void tegra241_cmdqv_deinit_vintf(struct tegra241_cmdqv *cmdqv, u16 idx) { kfree(cmdqv->vintfs[idx]->lvcmdqs); + /* + * Clear the slot and drain any in-flight ISR before returning idx to + * the IDA, so a concurrent create that reuses idx cannot have its + * freshly published VINTF erased here. A plain WRITE_ONCE() suffices + * since clearing the slot publishes no data. This also covers the + * init-failure unwind, which reaches deinit_vintf() without the + * destroy callback. + */ + WRITE_ONCE(cmdqv->vintfs[idx], NULL); + if (cmdqv->irq > 0) + synchronize_irq(cmdqv->irq); ida_free(&cmdqv->vintf_ids, idx); - cmdqv->vintfs[idx] = NULL; } static int tegra241_cmdqv_init_vintf(struct tegra241_cmdqv *cmdqv, u16 max_idx, @@ -754,7 +789,8 @@ static int tegra241_cmdqv_init_vintf(struct tegra241_cmdqv *cmdqv, u16 max_idx, return -ENOMEM; } - cmdqv->vintfs[idx] = vintf; + /* Pairs with the smp_load_acquire() in tegra241_cmdqv_isr() */ + smp_store_release(&cmdqv->vintfs[idx], vintf); return ret; } @@ -772,7 +808,7 @@ static void tegra241_cmdqv_remove_vintf(struct tegra241_cmdqv *cmdqv, u16 idx) dev_dbg(cmdqv->dev, "VINTF%u: deallocated\n", vintf->idx); tegra241_cmdqv_deinit_vintf(cmdqv, idx); - if (!vintf->hyp_own) { + if (vintf->idx) { mutex_destroy(&vintf->lvcmdq_mutex); ida_destroy(&vintf->sids); /* Guest-owned VINTF is free-ed with viommu by iommufd core */ @@ -798,6 +834,14 @@ static void tegra241_cmdqv_remove(struct arm_smmu_device *smmu) container_of(smmu, struct tegra241_cmdqv, smmu); u16 idx; + /* + * Free the IRQ before tearing down the VINTFs. free_irq() waits for any + * in-flight tegra241_cmdqv_isr() to finish and blocks new ones, so the + * ISR cannot dereference a VINTF that is freed by the loop below. + */ + if (cmdqv->irq > 0) + free_irq(cmdqv->irq, cmdqv); + /* Remove VINTF resources */ for (idx = 0; idx < cmdqv->num_vintfs; idx++) { if (cmdqv->vintfs[idx]) { @@ -810,8 +854,6 @@ static void tegra241_cmdqv_remove(struct arm_smmu_device *smmu) /* Remove cmdqv resources */ ida_destroy(&cmdqv->vintf_ids); - if (cmdqv->irq > 0) - free_irq(cmdqv->irq, cmdqv); iounmap(cmdqv->base); kfree(cmdqv->vintfs); put_device(cmdqv->dev); /* smmu->impl_dev */ @@ -946,6 +988,12 @@ static int tegra241_cmdqv_init_structures(struct arm_smmu_device *smmu) ret = tegra241_cmdqv_init_vintf(cmdqv, 0, vintf); if (ret) { dev_err(cmdqv->dev, "failed to init vintf0: %d\n", ret); + /* + * tegra241_cmdqv_init_vintf() failed to publish the vintf0 to + * cmdqv->vintfs[], so the probe unwind path that goes through + * cmdqv->vintfs[] would miss it. Free it here. + */ + kfree(vintf); return ret; } @@ -1006,17 +1054,6 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res, cmdqv->dev = smmu->impl_dev; cmdqv->base_phys = res->start; - if (cmdqv->irq > 0) { - ret = request_threaded_irq(irq, NULL, tegra241_cmdqv_isr, - IRQF_ONESHOT, "tegra241-cmdqv", - cmdqv); - if (ret) { - dev_err(cmdqv->dev, "failed to request irq (%d): %d\n", - cmdqv->irq, ret); - goto iounmap; - } - } - regval = readl_relaxed(REG_CMDQV(cmdqv, PARAM)); cmdqv->num_vintfs = 1 << FIELD_GET(CMDQV_NUM_VINTF_LOG2, regval); cmdqv->num_vcmdqs = 1 << FIELD_GET(CMDQV_NUM_VCMDQ_LOG2, regval); @@ -1027,10 +1064,25 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res, cmdqv->vintfs = kcalloc(cmdqv->num_vintfs, sizeof(*cmdqv->vintfs), GFP_KERNEL); if (!cmdqv->vintfs) - goto free_irq; + goto iounmap; ida_init(&cmdqv->vintf_ids); + /* + * Request the IRQ only after cmdqv->vintfs is allocated and zeroed, so + * the ISR would not walk an uninitialized array. + */ + if (cmdqv->irq > 0) { + ret = request_threaded_irq(irq, NULL, tegra241_cmdqv_isr, + IRQF_ONESHOT, "tegra241-cmdqv", + cmdqv); + if (ret) { + dev_err(cmdqv->dev, "failed to request irq (%d): %d\n", + cmdqv->irq, ret); + goto free_vintfs; + } + } + #ifdef CONFIG_IOMMU_DEBUGFS if (!cmdqv_debugfs_dir) { cmdqv_debugfs_dir = @@ -1045,9 +1097,9 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res, return new_smmu; -free_irq: - if (cmdqv->irq > 0) - free_irq(cmdqv->irq, cmdqv); +free_vintfs: + ida_destroy(&cmdqv->vintf_ids); + kfree(cmdqv->vintfs); iounmap: iounmap(base); return NULL; @@ -1202,13 +1254,15 @@ static int tegra241_vintf_alloc_lvcmdq_user(struct iommufd_hw_queue *hw_queue, if (ret) goto unmap_lvcmdq; + /* No lockless reader of a user VINTF's lvcmdqs[]; mutex-serialized */ + vintf->lvcmdqs[lidx] = vcmdq; + hw_queue->destroy = &tegra241_vintf_destroy_lvcmdq_user; mutex_unlock(&vintf->lvcmdq_mutex); return 0; unmap_lvcmdq: tegra241_vcmdq_unmap_lvcmdq(vcmdq); - tegra241_vintf_deinit_lvcmdq(vintf, lidx); undepend_vcmdq: if (vcmdq->prev) iommufd_hw_queue_undepend(vcmdq, vcmdq->prev, core); @@ -1254,7 +1308,8 @@ static int tegra241_vintf_init_vsid(struct iommufd_vdevice *vdev) if (virt_sid > FIELD_MAX(VINTF_SID_MATCH_VIRT_SID)) return -EINVAL; - WARN_ON_ONCE(master->num_streams != 1); + if (master->num_streams != 1) + return -EOPNOTSUPP; /* Find an empty pair of SID_REPLACE and SID_MATCH */ sidx = ida_alloc_max(&vintf->sids, vintf->cmdqv->num_sids_per_vintf - 1, diff --git a/drivers/iommu/arm/arm-smmu/qcom_iommu.c b/drivers/iommu/arm/arm-smmu/qcom_iommu.c index 9c1166a3af6c..82c3ffe0c783 100644 --- a/drivers/iommu/arm/arm-smmu/qcom_iommu.c +++ b/drivers/iommu/arm/arm-smmu/qcom_iommu.c @@ -200,7 +200,7 @@ static irqreturn_t qcom_iommu_fault(int irq, void *dev) fsynr = iommu_readl(ctx, ARM_SMMU_CB_FSYNR0); iova = iommu_readq(ctx, ARM_SMMU_CB_FAR); - if (!report_iommu_fault(ctx->domain, ctx->dev, iova, 0)) { + if (report_iommu_fault(ctx->domain, ctx->dev, iova, 0)) { dev_err_ratelimited(ctx->dev, "Unhandled context fault: fsr=0x%x, " "iova=0x%016llx, fsynr=0x%x, cb=%d\n", @@ -858,7 +858,7 @@ static int qcom_iommu_device_probe(struct platform_device *pdev) ret = iommu_device_register(&qcom_iommu->iommu, &qcom_iommu_ops, dev); if (ret) { dev_err(dev, "Failed to register iommu\n"); - goto err_pm_disable; + goto err_sysfs_remove; } if (qcom_iommu->local_base) { @@ -869,6 +869,8 @@ static int qcom_iommu_device_probe(struct platform_device *pdev) return 0; +err_sysfs_remove: + iommu_device_sysfs_remove(&qcom_iommu->iommu); err_pm_disable: pm_runtime_disable(dev); return ret; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index b0dca7e7429a..039e84a452d7 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1647,13 +1647,16 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle, } if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && - !gfpflags_allow_blocking(gfp) && !coherent) + !gfpflags_allow_blocking(gfp) && !coherent) { page = dma_alloc_from_pool(dev, PAGE_ALIGN(size), &cpu_addr, - gfp, NULL); - else + gfp, NULL); + if (!page) + return NULL; + } else { cpu_addr = iommu_dma_alloc_pages(dev, size, &page, gfp, attrs); - if (!cpu_addr) - return NULL; + if (!cpu_addr) + return NULL; + } *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, dev->coherent_dma_mask); @@ -2160,6 +2163,19 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev, dma_addr_t iova; int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO; size_t size = cookie_msi_granule(domain); + static DEFINE_MUTEX(msi_prepare_lock); + + /* + * Normally a device's default domain is only ever attached to that + * device's own group, and the group mutex held by + * iommu_group_mutex_assert()'s callers is enough on its own. A VFIO + * type1 container is the one case that breaks that assumption: it + * can merge devices from different groups onto one domain, so two + * devices' group mutexes don't serialize each other here. A static + * lock is sufficient due to the expectation that this is a corner + * case that will never be contended in practice. + */ + guard(mutex)(&msi_prepare_lock); msi_addr &= ~(phys_addr_t)(size - 1); list_for_each_entry(msi_page, msi_page_list, list) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index aec1a05d48d0..77e51712040b 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1421,7 +1421,8 @@ static void copied_context_tear_down(struct intel_iommu *iommu, assert_spin_locked(&iommu->lock); did_old = context_domain_id(context); - context_clear_entry(context); + context_clear_present(context); + __iommu_flush_cache(iommu, context, sizeof(*context)); if (did_old < cap_ndoms(iommu->cap)) { iommu->flush.flush_context(iommu, did_old, @@ -1432,6 +1433,9 @@ static void copied_context_tear_down(struct intel_iommu *iommu, DMA_TLB_DSI_FLUSH); } + context_clear_entry(context); + __iommu_flush_cache(iommu, context, sizeof(*context)); + clear_context_copied(iommu, bus, devfn); } @@ -1728,7 +1732,7 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8 context_clear_present(context); __iommu_flush_cache(iommu, context, sizeof(*context)); spin_unlock(&iommu->lock); - intel_context_flush_no_pasid(info, context, did); + intel_context_flush_no_pasid(info, context, did, PCI_DEVID(bus, devfn)); context_clear_entry(context); __iommu_flush_cache(iommu, context, sizeof(*context)); } @@ -1924,7 +1928,7 @@ static int copy_context_table(struct intel_iommu *iommu, struct context_entry **tbl, int bus, bool ext) { - int tbl_idx, pos = 0, idx, devfn, ret = 0, did; + int tbl_idx, tbl_slot = 0, idx, devfn, ret = 0, did; struct context_entry *new_ce = NULL, ce; struct context_entry *old_ce = NULL; struct root_entry re; @@ -1940,10 +1944,9 @@ static int copy_context_table(struct intel_iommu *iommu, if (idx == 0) { /* First save what we may have and clean up */ if (new_ce) { - tbl[tbl_idx] = new_ce; + tbl[tbl_idx + tbl_slot] = new_ce; __iommu_flush_cache(iommu, new_ce, VTD_PAGE_SIZE); - pos = 1; } if (old_ce) @@ -1965,6 +1968,9 @@ static int copy_context_table(struct intel_iommu *iommu, } } + /* Track if saving UCTP or LCTP entries in scalable mode */ + tbl_slot = ext && devfn >= 0x80 ? 1 : 0; + ret = -ENOMEM; old_ce = memremap(old_ce_phys, PAGE_SIZE, MEMREMAP_WB); @@ -1993,7 +1999,7 @@ static int copy_context_table(struct intel_iommu *iommu, new_ce[idx] = ce; } - tbl[tbl_idx + pos] = new_ce; + tbl[tbl_idx + tbl_slot] = new_ce; __iommu_flush_cache(iommu, new_ce, VTD_PAGE_SIZE); @@ -2303,7 +2309,7 @@ static void iommu_flush_all(void) } } -static int iommu_suspend(void) +static int iommu_suspend(void *data) { struct dmar_drhd_unit *drhd; struct intel_iommu *iommu = NULL; @@ -2330,7 +2336,7 @@ static int iommu_suspend(void) return 0; } -static void iommu_resume(void) +static void iommu_resume(void *data) { struct dmar_drhd_unit *drhd; struct intel_iommu *iommu = NULL; @@ -2361,14 +2367,18 @@ static void iommu_resume(void) } } -static struct syscore_ops iommu_syscore_ops = { +static const struct syscore_ops iommu_syscore_ops = { .resume = iommu_resume, .suspend = iommu_suspend, }; +static struct syscore iommu_syscore = { + .ops = &iommu_syscore_ops, +}; + static void __init init_iommu_pm_ops(void) { - register_syscore_ops(&iommu_syscore_ops); + register_syscore(&iommu_syscore); } #else @@ -3858,6 +3868,7 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev) return &iommu->iommu; free_table: + intel_pasid_teardown_sm_context(dev); intel_pasid_free_table(dev); clear_rbtree: device_rbtree_remove(info); diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 8e2df00e881c..e696e019e969 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -1320,7 +1320,7 @@ void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long start, unsigned long end); void intel_context_flush_no_pasid(struct device_domain_info *info, - struct context_entry *context, u16 did); + struct context_entry *context, u16 did, u16 sid); int intel_iommu_enable_prq(struct intel_iommu *iommu); int intel_iommu_finish_prq(struct intel_iommu *iommu); diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c index 85c3116e351c..48e21743f207 100644 --- a/drivers/iommu/intel/pasid.c +++ b/drivers/iommu/intel/pasid.c @@ -749,7 +749,7 @@ static void device_pasid_table_teardown(struct device *dev, u8 bus, u8 devfn) context_clear_present(context); __iommu_flush_cache(iommu, context, sizeof(*context)); spin_unlock(&iommu->lock); - intel_context_flush_no_pasid(info, context, did); + intel_context_flush_no_pasid(info, context, did, PCI_DEVID(bus, devfn)); context_clear_entry(context); __iommu_flush_cache(iommu, context, sizeof(*context)); } @@ -953,9 +953,12 @@ static void __context_flush_dev_iotlb(struct device_domain_info *info) * This helper can only be used when IOMMU is working in the legacy mode or * IOMMU is in scalable mode but all PASID table entries of the device are * non-present. + * + * @sid identifies the context entry that was modified, which may be a DMA + * alias of @info->dev rather than its own requester ID. */ void intel_context_flush_no_pasid(struct device_domain_info *info, - struct context_entry *context, u16 did) + struct context_entry *context, u16 did, u16 sid) { struct intel_iommu *iommu = info->iommu; @@ -965,7 +968,7 @@ void intel_context_flush_no_pasid(struct device_domain_info *info, * when operating in scalable mode. Therefore the @did value doesn't * matter in scalable mode. */ - iommu->flush.flush_context(iommu, did, PCI_DEVID(info->bus, info->devfn), + iommu->flush.flush_context(iommu, did, sid, DMA_CCMD_MASK_NOBIT, DMA_CCMD_DEVICE_INVL); /* diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index c40515bf5017..3216deab98e3 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -148,29 +148,22 @@ static void iommufd_device_remove_vdev(struct iommufd_device *idev) if (!idev->vdev) goto out_unlock; - vdev = iommufd_get_vdevice(idev->ictx, idev->vdev->obj.id); + vdev = idev->vdev; + /* * An ongoing vdev destroy ioctl has removed the vdev from the object * xarray, but has not finished iommufd_vdevice_destroy() yet as it * needs the same mutex. We exit the locking then wait on wait_cnt * reference for the vdev destruction. */ - if (IS_ERR(vdev)) - goto out_unlock; - - /* Should never happen */ - if (WARN_ON(vdev != idev->vdev)) { - iommufd_put_object(idev->ictx, &vdev->obj); + if (iommufd_try_inc_users(idev->ictx, &vdev->obj)) goto out_unlock; - } /* * vdev is still alive. Hold a users refcount to prevent racing with * userspace destruction, then use iommufd_object_tombstone_user() to * destroy it and leave a tombstone. */ - refcount_inc(&vdev->obj.users); - iommufd_put_object(idev->ictx, &vdev->obj); mutex_unlock(&idev->igroup->lock); iommufd_object_tombstone_user(idev->ictx, &vdev->obj); return; diff --git a/drivers/iommu/iommufd/iommufd_private.h b/drivers/iommu/iommufd/iommufd_private.h index 53d37e77c4b9..c208563eb367 100644 --- a/drivers/iommu/iommufd/iommufd_private.h +++ b/drivers/iommu/iommufd/iommufd_private.h @@ -181,6 +181,7 @@ static inline bool iommufd_lock_obj(struct iommufd_object *obj) return true; } +int iommufd_try_inc_users(struct iommufd_ctx *ictx, struct iommufd_object *obj); struct iommufd_object *iommufd_get_object(struct iommufd_ctx *ictx, u32 id, enum iommufd_object_type type); static inline void iommufd_put_object(struct iommufd_ctx *ictx, @@ -695,14 +696,6 @@ void iommufd_vdevice_abort(struct iommufd_object *obj); int iommufd_hw_queue_alloc_ioctl(struct iommufd_ucmd *ucmd); void iommufd_hw_queue_destroy(struct iommufd_object *obj); -static inline struct iommufd_vdevice * -iommufd_get_vdevice(struct iommufd_ctx *ictx, u32 id) -{ - return container_of(iommufd_get_object(ictx, id, - IOMMUFD_OBJ_VDEVICE), - struct iommufd_vdevice, obj); -} - #ifdef CONFIG_IOMMUFD_TEST int iommufd_test(struct iommufd_ucmd *ucmd); void iommufd_selftest_destroy(struct iommufd_object *obj); diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c index ce775fbbae94..02d06a02d5e6 100644 --- a/drivers/iommu/iommufd/main.c +++ b/drivers/iommu/iommufd/main.c @@ -180,6 +180,26 @@ struct iommufd_object *iommufd_get_object(struct iommufd_ctx *ictx, u32 id, return obj; } +/* + * Increment the users count of an object outside the context of an ioctl that + * has already locked it. The users refcount cannot be increased on an already + * created object unless the object is installed in the xarray, otherwise things + * are racing with a parallel destruction. + */ +int iommufd_try_inc_users(struct iommufd_ctx *ictx, struct iommufd_object *obj) +{ + struct iommufd_object *cur; + + xa_lock(&ictx->objects); + cur = xa_load(&ictx->objects, obj->id); + if (cur == obj) + refcount_inc(&obj->users); + xa_unlock(&ictx->objects); + if (cur != obj) + return -EBUSY; + return 0; +} + static int iommufd_object_dec_wait(struct iommufd_ctx *ictx, struct iommufd_object *to_destroy) { diff --git a/drivers/iommu/iommufd/selftest.c b/drivers/iommu/iommufd/selftest.c index 35c42ff4355a..cd3d1380cf6c 100644 --- a/drivers/iommu/iommufd/selftest.c +++ b/drivers/iommu/iommufd/selftest.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -1806,6 +1807,9 @@ static int iommufd_test_dirty(struct iommufd_ucmd *ucmd, unsigned int mockpt_id, if (!page_size || !length || iova % page_size || length % page_size || !uptr) return -EINVAL; + max = length / page_size; + if (max > SZ_16M * BITS_PER_BYTE) + return -EOVERFLOW; hwpt = get_md_pagetable(ucmd, mockpt_id, &mock); if (IS_ERR(hwpt)) @@ -1816,7 +1820,6 @@ static int iommufd_test_dirty(struct iommufd_ucmd *ucmd, unsigned int mockpt_id, goto out_put; } - max = length / page_size; tmp = kvzalloc(DIV_ROUND_UP(max, BITS_PER_LONG) * sizeof(unsigned long), GFP_KERNEL_ACCOUNT); if (!tmp) { diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c index d5e2fc9b0111..1dfa6f69c28f 100644 --- a/drivers/iommu/msm_iommu.c +++ b/drivers/iommu/msm_iommu.c @@ -719,7 +719,7 @@ static int msm_iommu_probe(struct platform_device *pdev) iommu = devm_kzalloc(&pdev->dev, sizeof(*iommu), GFP_KERNEL); if (!iommu) - return -ENODEV; + return -ENOMEM; iommu->dev = &pdev->dev; INIT_LIST_HEAD(&iommu->ctx_list); diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c index 44b965a2db92..ad2dbd29fef8 100644 --- a/drivers/iommu/mtk_iommu_v1.c +++ b/drivers/iommu/mtk_iommu_v1.c @@ -88,7 +88,7 @@ struct dma_iommu_mapping { /* MTK generation one iommu HW only support 4K size mapping */ #define MT2701_IOMMU_PAGE_SHIFT 12 #define MT2701_IOMMU_PAGE_SIZE (1UL << MT2701_IOMMU_PAGE_SHIFT) -#define MT2701_LARB_NR_MAX 3 +#define MT2701_LARB_NR_MAX 4 /* * MTK m4u support 4GB iova address space, and only support 4K page diff --git a/drivers/ipack/devices/ipoctal.c b/drivers/ipack/devices/ipoctal.c index ba2e9e52d72b..474b58858dca 100644 --- a/drivers/ipack/devices/ipoctal.c +++ b/drivers/ipack/devices/ipoctal.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -25,6 +26,8 @@ static const struct tty_operations ipoctal_fops; +static void ipoctal_release(struct kref *kref); + struct ipoctal_channel { struct ipoctal_stats stats; unsigned int nb_bytes; @@ -49,6 +52,9 @@ struct ipoctal { struct tty_driver *tty_drv; u8 __iomem *mem8_space; u8 __iomem *int_space; + struct kref kref; + struct module *carrier_owner; + bool removed; }; static inline struct ipoctal *chan_to_ipoctal(struct ipoctal_channel *chan, @@ -70,8 +76,14 @@ static void ipoctal_reset_channel(struct ipoctal_channel *channel) static int ipoctal_port_activate(struct tty_port *port, struct tty_struct *tty) { struct ipoctal_channel *channel; + struct ipoctal *ipoctal; channel = dev_get_drvdata(tty->dev); + ipoctal = chan_to_ipoctal(channel, tty->index); + + + if (ipoctal->removed) + return -ENODEV; /* * Enable RX. TX will be enabled when @@ -95,6 +107,7 @@ static int ipoctal_install(struct tty_driver *driver, struct tty_struct *tty) if (res) goto err_put_carrier; + kref_get(&ipoctal->kref); tty->driver_data = channel; return 0; @@ -460,8 +473,13 @@ static ssize_t ipoctal_write_tty(struct tty_struct *tty, const u8 *buf, size_t count) { struct ipoctal_channel *channel = tty->driver_data; + struct ipoctal *ipoctal = chan_to_ipoctal(channel, tty->index); size_t char_copied; + + if (ipoctal->removed || !channel->tty_port.xmit_buf) + return 0; + char_copied = ipoctal_copy_write_buffer(channel, buf, count); /* As the IP-OCTAL 485 only supports half duplex, do it manually */ @@ -501,8 +519,13 @@ static void ipoctal_set_termios(struct tty_struct *tty, unsigned char mr2 = 0; unsigned char csr = 0; struct ipoctal_channel *channel = tty->driver_data; + struct ipoctal *ipoctal = chan_to_ipoctal(channel, tty->index); speed_t baud; + + if (ipoctal->removed) + return; + cflag = tty->termios.c_cflag; /* Disable and reset everything before change the setup */ @@ -631,10 +654,16 @@ static void ipoctal_hangup(struct tty_struct *tty) { unsigned long flags; struct ipoctal_channel *channel = tty->driver_data; + struct ipoctal *ipoctal; if (channel == NULL) return; + ipoctal = chan_to_ipoctal(channel, tty->index); + + if (ipoctal->removed) + return; + spin_lock_irqsave(&channel->lock, flags); channel->nb_bytes = 0; channel->pointer_read = 0; @@ -651,10 +680,16 @@ static void ipoctal_hangup(struct tty_struct *tty) static void ipoctal_shutdown(struct tty_struct *tty) { struct ipoctal_channel *channel = tty->driver_data; + struct ipoctal *ipoctal; if (channel == NULL) return; + ipoctal = chan_to_ipoctal(channel, tty->index); + + if (ipoctal->removed) + return; + ipoctal_reset_channel(channel); tty_port_set_initialized(&channel->tty_port, false); } @@ -664,8 +699,9 @@ static void ipoctal_cleanup(struct tty_struct *tty) struct ipoctal_channel *channel = tty->driver_data; struct ipoctal *ipoctal = chan_to_ipoctal(channel, tty->index); - /* release the carrier driver */ - ipack_put_carrier(ipoctal->dev); + /* release the carrier driver via cached owner */ + module_put(ipoctal->carrier_owner); + kref_put(&ipoctal->kref, ipoctal_release); } static const struct tty_operations ipoctal_fops = { @@ -683,6 +719,13 @@ static const struct tty_operations ipoctal_fops = { .cleanup = ipoctal_cleanup, }; +static void ipoctal_release(struct kref *kref) +{ + struct ipoctal *ipoctal = container_of(kref, struct ipoctal, kref); + + kfree(ipoctal); +} + static int ipoctal_probe(struct ipack_device *dev) { int res; @@ -692,7 +735,10 @@ static int ipoctal_probe(struct ipack_device *dev) if (ipoctal == NULL) return -ENOMEM; + kref_init(&ipoctal->kref); + ipoctal->dev = dev; + ipoctal->carrier_owner = dev->bus->owner; res = ipoctal_inst_slot(ipoctal, dev->bus->bus_nr, dev->slot); if (res) goto out_uninst; @@ -701,7 +747,7 @@ static int ipoctal_probe(struct ipack_device *dev) return 0; out_uninst: - kfree(ipoctal); + kref_put(&ipoctal->kref, ipoctal_release); return res; } @@ -709,6 +755,8 @@ static void __ipoctal_remove(struct ipoctal *ipoctal) { int i; + ipoctal->removed = true; + ipoctal->dev->bus->ops->free_irq(ipoctal->dev); for (i = 0; i < NR_CHANNELS; i++) { @@ -725,7 +773,7 @@ static void __ipoctal_remove(struct ipoctal *ipoctal) tty_unregister_driver(ipoctal->tty_drv); kfree(ipoctal->tty_drv->name); tty_driver_kref_put(ipoctal->tty_drv); - kfree(ipoctal); + kref_put(&ipoctal->kref, ipoctal_release); } static void ipoctal_remove(struct ipack_device *idev) diff --git a/drivers/irqchip/exynos-combiner.c b/drivers/irqchip/exynos-combiner.c index e7dfcf0cda43..495848442b35 100644 --- a/drivers/irqchip/exynos-combiner.c +++ b/drivers/irqchip/exynos-combiner.c @@ -200,12 +200,13 @@ static void __init combiner_init(void __iomem *combiner_base, /** * combiner_suspend - save interrupt combiner state before suspend + * @data: syscore context * * Save the interrupt enable set register for all combiner groups since * the state is lost when the system enters into a sleep state. * */ -static int combiner_suspend(void) +static int combiner_suspend(void *data) { int i; @@ -218,12 +219,13 @@ static int combiner_suspend(void) /** * combiner_resume - restore interrupt combiner state after resume + * @data: syscore context * * Restore the interrupt enable set register for all combiner groups since * the state is lost when the system enters into a sleep state on suspend. * */ -static void combiner_resume(void) +static void combiner_resume(void *data) { int i; @@ -240,11 +242,15 @@ static void combiner_resume(void) #define combiner_resume NULL #endif -static struct syscore_ops combiner_syscore_ops = { +static const struct syscore_ops combiner_syscore_ops = { .suspend = combiner_suspend, .resume = combiner_resume, }; +static struct syscore combiner_syscore = { + .ops = &combiner_syscore_ops, +}; + static int __init combiner_of_init(struct device_node *np, struct device_node *parent) { @@ -264,7 +270,7 @@ static int __init combiner_of_init(struct device_node *np, combiner_init(combiner_base, np); - register_syscore_ops(&combiner_syscore_ops); + register_syscore(&combiner_syscore); return 0; } diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c index a44c49e985b7..a4d03a2d1569 100644 --- a/drivers/irqchip/irq-armada-370-xp.c +++ b/drivers/irqchip/irq-armada-370-xp.c @@ -726,7 +726,7 @@ static void __exception_irq_entry mpic_handle_irq(struct pt_regs *regs) } while (1); } -static int mpic_suspend(void) +static int mpic_suspend(void *data) { struct mpic *mpic = mpic_data; @@ -735,7 +735,7 @@ static int mpic_suspend(void) return 0; } -static void mpic_resume(void) +static void mpic_resume(void *data) { struct mpic *mpic = mpic_data; bool src0, src1; @@ -788,11 +788,15 @@ static void mpic_resume(void) mpic_ipi_resume(mpic); } -static struct syscore_ops mpic_syscore_ops = { +static const struct syscore_ops mpic_syscore_ops = { .suspend = mpic_suspend, .resume = mpic_resume, }; +static struct syscore mpic_syscore = { + .ops = &mpic_syscore_ops, +}; + static int __init mpic_map_region(struct device_node *np, int index, void __iomem **base, phys_addr_t *phys_base) { @@ -905,7 +909,7 @@ static int __init mpic_of_init(struct device_node *node, struct device_node *par mpic_handle_cascade_irq, mpic); } - register_syscore_ops(&mpic_syscore_ops); + register_syscore(&mpic_syscore); return 0; } diff --git a/drivers/irqchip/irq-bcm7038-l1.c b/drivers/irqchip/irq-bcm7038-l1.c index 821b288587ca..674138668f1c 100644 --- a/drivers/irqchip/irq-bcm7038-l1.c +++ b/drivers/irqchip/irq-bcm7038-l1.c @@ -291,7 +291,7 @@ static int bcm7038_l1_init_one(struct device_node *dn, unsigned int idx, static LIST_HEAD(bcm7038_l1_intcs_list); static DEFINE_RAW_SPINLOCK(bcm7038_l1_intcs_lock); -static int bcm7038_l1_suspend(void) +static int bcm7038_l1_suspend(void *data) { struct bcm7038_l1_chip *intc; int boot_cpu, word; @@ -317,7 +317,7 @@ static int bcm7038_l1_suspend(void) return 0; } -static void bcm7038_l1_resume(void) +static void bcm7038_l1_resume(void *data) { struct bcm7038_l1_chip *intc; int boot_cpu, word; @@ -338,11 +338,15 @@ static void bcm7038_l1_resume(void) } } -static struct syscore_ops bcm7038_l1_syscore_ops = { +static const struct syscore_ops bcm7038_l1_syscore_ops = { .suspend = bcm7038_l1_suspend, .resume = bcm7038_l1_resume, }; +static struct syscore bcm7038_l1_syscore = { + .ops = &bcm7038_l1_syscore_ops, +}; + static int bcm7038_l1_set_wake(struct irq_data *d, unsigned int on) { struct bcm7038_l1_chip *intc = irq_data_get_irq_chip_data(d); @@ -430,7 +434,7 @@ static int bcm7038_l1_probe(struct platform_device *pdev, struct device_node *pa raw_spin_unlock(&bcm7038_l1_intcs_lock); if (list_is_singular(&bcm7038_l1_intcs_list)) - register_syscore_ops(&bcm7038_l1_syscore_ops); + register_syscore(&bcm7038_l1_syscore); #endif pr_info("registered BCM7038 L1 intc (%pOF, IRQs: %d)\n", diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c index a1661657391d..cc03ab698987 100644 --- a/drivers/irqchip/irq-gic-v3-its.c +++ b/drivers/irqchip/irq-gic-v3-its.c @@ -4593,6 +4593,13 @@ static int its_vpe_init(struct its_vpe *vpe) static void its_vpe_teardown(struct its_vpe *vpe) { + /* + * If vpt_page is NULL, then its_vpe_init() has failed, and + * there is nothing to do as no resource has been allocated. + */ + if (vpe->vpt_page == NULL) + return; + its_vpe_db_proxy_unmap(vpe); its_vpe_id_free(vpe->vpe_id); its_free_pending_table(vpe->vpt_page); @@ -4673,8 +4680,10 @@ static int its_vpe_irq_domain_alloc(struct irq_domain *domain, unsigned int virq irqd_set_resend_when_in_progress(irq_get_irq_data(virq + i)); } - if (err) + if (err) { + its_vpe_teardown(vm->vpes[i]); its_vpe_irq_domain_free(domain, virq, i); + } return err; } @@ -4994,7 +5003,7 @@ static void its_enable_quirks(struct its_node *its) its_quirks, its); } -static int its_save_disable(void) +static int its_save_disable(void *data) { struct its_node *its; int err = 0; @@ -5030,7 +5039,7 @@ static int its_save_disable(void) return err; } -static void its_restore_enable(void) +static void its_restore_enable(void *data) { struct its_node *its; int ret; @@ -5090,11 +5099,15 @@ static void its_restore_enable(void) raw_spin_unlock(&its_lock); } -static struct syscore_ops its_syscore_ops = { +static const struct syscore_ops its_syscore_ops = { .suspend = its_save_disable, .resume = its_restore_enable, }; +static struct syscore its_syscore = { + .ops = &its_syscore_ops, +}; + static void __init __iomem *its_map_one(struct resource *res, int *err) { void __iomem *its_base; @@ -5322,7 +5335,7 @@ static int __init its_probe_one(struct its_node *its) err = its_init_domain(its); if (err) - goto out_free_tables; + goto out_free_collection; raw_spin_lock(&its_lock); list_add(&its->entry, &its_nodes); @@ -5330,6 +5343,8 @@ static int __init its_probe_one(struct its_node *its) return 0; +out_free_collection: + kfree(its->collections); out_free_tables: its_free_tables(its); out_free_cmd: @@ -5742,9 +5757,13 @@ static int __init gic_acpi_parse_madt_its(union acpi_subtable_headers *header, its->flags |= ITS_FLAGS_FORCE_NON_SHAREABLE; err = its_probe_one(its); - if (!err) - return 0; + if (err) + goto probe_err; + + return 0; +probe_err: + its_node_destroy(its); node_err: iort_deregister_domain_token(its_entry->translation_id); dom_err: @@ -5866,7 +5885,7 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists, } } - register_syscore_ops(&its_syscore_ops); + register_syscore(&its_syscore); return 0; } diff --git a/drivers/irqchip/irq-gic-v5-irs.c b/drivers/irqchip/irq-gic-v5-irs.c index ce2732d649a3..a650eccdb37a 100644 --- a/drivers/irqchip/irq-gic-v5-irs.c +++ b/drivers/irqchip/irq-gic-v5-irs.c @@ -198,7 +198,7 @@ int gicv5_irs_iste_alloc(const u32 lpi) if (!gicv5_global_data.ist.l2) return 0; - irs_data = per_cpu(per_cpu_irs_data, smp_processor_id()); + irs_data = per_cpu(per_cpu_irs_data, 0); if (!irs_data) return -ENOENT; diff --git a/drivers/irqchip/irq-gic-v5.c b/drivers/irqchip/irq-gic-v5.c index fa86dd03e307..ebeb26b25dac 100644 --- a/drivers/irqchip/irq-gic-v5.c +++ b/drivers/irqchip/irq-gic-v5.c @@ -811,6 +811,9 @@ void __init gicv5_init_lpi_domain(void) void __init gicv5_free_lpi_domain(void) { + if (!gicv5_global_data.lpi_domain) + return; + irq_domain_remove(gicv5_global_data.lpi_domain); gicv5_global_data.lpi_domain = NULL; } @@ -925,6 +928,7 @@ static void gicv5_cpu_disable_interrupts(void) cr0 = FIELD_PREP(ICC_CR0_EL1_EN, 0); write_sysreg_s(cr0, SYS_ICC_CR0_EL1); + isb(); } static void gicv5_cpu_enable_interrupts(void) @@ -1106,7 +1110,7 @@ static int __init gicv5_of_init(struct device_node *node, struct device_node *pa ret = gicv5_starting_cpu(smp_processor_id()); if (ret) - goto out_dom; + goto out_int; ret = set_handle_irq(gicv5_handle_irq); if (ret) @@ -1114,7 +1118,7 @@ static int __init gicv5_of_init(struct device_node *node, struct device_node *pa ret = gicv5_irs_enable(); if (ret) - goto out_int; + goto out_handle; gicv5_smp_init(); @@ -1124,9 +1128,10 @@ static int __init gicv5_of_init(struct device_node *node, struct device_node *pa return 0; +out_handle: + set_handle_irq(NULL); out_int: gicv5_cpu_disable_interrupts(); -out_dom: gicv5_free_domains(); out_irs: gicv5_irs_remove(); diff --git a/drivers/irqchip/irq-i8259.c b/drivers/irqchip/irq-i8259.c index 91b2f587119c..cca77f9948a3 100644 --- a/drivers/irqchip/irq-i8259.c +++ b/drivers/irqchip/irq-i8259.c @@ -202,13 +202,13 @@ static void mask_and_ack_8259A(struct irq_data *d) } } -static void i8259A_resume(void) +static void i8259A_resume(void *data) { if (i8259A_auto_eoi >= 0) init_8259A(i8259A_auto_eoi); } -static void i8259A_shutdown(void) +static void i8259A_shutdown(void *data) { /* Put the i8259A into a quiescent state that * the kernel initialization code can get it @@ -220,11 +220,15 @@ static void i8259A_shutdown(void) } } -static struct syscore_ops i8259_syscore_ops = { +static const struct syscore_ops i8259_syscore_ops = { .resume = i8259A_resume, .shutdown = i8259A_shutdown, }; +static struct syscore i8259_syscore = { + .ops = &i8259_syscore_ops, +}; + static void init_8259A(int auto_eoi) { unsigned long flags; @@ -320,7 +324,7 @@ struct irq_domain * __init __init_i8259_irqs(struct device_node *node) if (request_irq(irq, no_action, IRQF_NO_THREAD, "cascade", NULL)) pr_err("Failed to register cascade interrupt\n"); - register_syscore_ops(&i8259_syscore_ops); + register_syscore(&i8259_syscore); return domain; } diff --git a/drivers/irqchip/irq-imx-gpcv2.c b/drivers/irqchip/irq-imx-gpcv2.c index b91f5c14b405..04f7ba0657be 100644 --- a/drivers/irqchip/irq-imx-gpcv2.c +++ b/drivers/irqchip/irq-imx-gpcv2.c @@ -33,7 +33,7 @@ static void __iomem *gpcv2_idx_to_reg(struct gpcv2_irqchip_data *cd, int i) return cd->gpc_base + cd->cpu2wakeup + i * 4; } -static int gpcv2_wakeup_source_save(void) +static int gpcv2_wakeup_source_save(void *data) { struct gpcv2_irqchip_data *cd; void __iomem *reg; @@ -52,7 +52,7 @@ static int gpcv2_wakeup_source_save(void) return 0; } -static void gpcv2_wakeup_source_restore(void) +static void gpcv2_wakeup_source_restore(void *data) { struct gpcv2_irqchip_data *cd; int i; @@ -65,9 +65,13 @@ static void gpcv2_wakeup_source_restore(void) writel_relaxed(cd->saved_irq_mask[i], gpcv2_idx_to_reg(cd, i)); } -static struct syscore_ops imx_gpcv2_syscore_ops = { - .suspend = gpcv2_wakeup_source_save, - .resume = gpcv2_wakeup_source_restore, +static const struct syscore_ops gpcv2_syscore_ops = { + .suspend = gpcv2_wakeup_source_save, + .resume = gpcv2_wakeup_source_restore, +}; + +static struct syscore gpcv2_syscore = { + .ops = &gpcv2_syscore_ops, }; static int imx_gpcv2_irq_set_wake(struct irq_data *d, unsigned int on) @@ -276,7 +280,7 @@ static int __init imx_gpcv2_irqchip_init(struct device_node *node, writel_relaxed(~0x1, cd->gpc_base + cd->cpu2wakeup); imx_gpcv2_instance = cd; - register_syscore_ops(&imx_gpcv2_syscore_ops); + register_syscore(&gpcv2_syscore); /* * Clear the OF_POPULATED flag set in of_irq_init so that diff --git a/drivers/irqchip/irq-loongson-eiointc.c b/drivers/irqchip/irq-loongson-eiointc.c index 39e5a72ccd3c..ad2105685b48 100644 --- a/drivers/irqchip/irq-loongson-eiointc.c +++ b/drivers/irqchip/irq-loongson-eiointc.c @@ -407,21 +407,25 @@ static struct irq_domain *acpi_get_vec_parent(int node, struct acpi_vector_group return NULL; } -static int eiointc_suspend(void) +static int eiointc_suspend(void *data) { return 0; } -static void eiointc_resume(void) +static void eiointc_resume(void *data) { eiointc_router_init(0); } -static struct syscore_ops eiointc_syscore_ops = { +static const struct syscore_ops eiointc_syscore_ops = { .suspend = eiointc_suspend, .resume = eiointc_resume, }; +static struct syscore eiointc_syscore = { + .ops = &eiointc_syscore_ops, +}; + static int __init pch_pic_parse_madt(union acpi_subtable_headers *header, const unsigned long end) { @@ -540,7 +544,7 @@ static int __init eiointc_init(struct eiointc_priv *priv, int parent_irq, eiointc_router_init(0); if (nr_pics == 1) { - register_syscore_ops(&eiointc_syscore_ops); + register_syscore(&eiointc_syscore); cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_EIOINTC_STARTING, "irqchip/loongarch/eiointc:starting", eiointc_router_init, NULL); diff --git a/drivers/irqchip/irq-loongson-htpic.c b/drivers/irqchip/irq-loongson-htpic.c index f4abdf156de7..1c691c4be989 100644 --- a/drivers/irqchip/irq-loongson-htpic.c +++ b/drivers/irqchip/irq-loongson-htpic.c @@ -71,15 +71,19 @@ static void htpic_reg_init(void) writel(0xffff, htpic->base + HTINT_EN_OFF); } -static void htpic_resume(void) +static void htpic_resume(void *data) { htpic_reg_init(); } -struct syscore_ops htpic_syscore_ops = { +static const struct syscore_ops htpic_syscore_ops = { .resume = htpic_resume, }; +static struct syscore htpic_syscore = { + .ops = &htpic_syscore_ops, +}; + static int __init htpic_of_init(struct device_node *node, struct device_node *parent) { unsigned int parent_irq[4]; @@ -130,7 +134,7 @@ static int __init htpic_of_init(struct device_node *node, struct device_node *pa htpic_irq_dispatch, htpic); } - register_syscore_ops(&htpic_syscore_ops); + register_syscore(&htpic_syscore); return 0; diff --git a/drivers/irqchip/irq-loongson-htvec.c b/drivers/irqchip/irq-loongson-htvec.c index d8558eb35044..d2be8e954e92 100644 --- a/drivers/irqchip/irq-loongson-htvec.c +++ b/drivers/irqchip/irq-loongson-htvec.c @@ -159,7 +159,7 @@ static void htvec_reset(struct htvec *priv) } } -static int htvec_suspend(void) +static int htvec_suspend(void *data) { int i; @@ -169,7 +169,7 @@ static int htvec_suspend(void) return 0; } -static void htvec_resume(void) +static void htvec_resume(void *data) { int i; @@ -177,11 +177,15 @@ static void htvec_resume(void) writel(htvec_priv->saved_vec_en[i], htvec_priv->base + HTVEC_EN_OFF + 4 * i); } -static struct syscore_ops htvec_syscore_ops = { +static const struct syscore_ops htvec_syscore_ops = { .suspend = htvec_suspend, .resume = htvec_resume, }; +static struct syscore htvec_syscore = { + .ops = &htvec_syscore_ops, +}; + static int htvec_init(phys_addr_t addr, unsigned long size, int num_parents, int parent_irq[], struct fwnode_handle *domain_handle) { @@ -214,7 +218,7 @@ static int htvec_init(phys_addr_t addr, unsigned long size, htvec_priv = priv; - register_syscore_ops(&htvec_syscore_ops); + register_syscore(&htvec_syscore); return 0; diff --git a/drivers/irqchip/irq-loongson-pch-lpc.c b/drivers/irqchip/irq-loongson-pch-lpc.c index 912bf50a5c7c..3a125f3e4287 100644 --- a/drivers/irqchip/irq-loongson-pch-lpc.c +++ b/drivers/irqchip/irq-loongson-pch-lpc.c @@ -151,7 +151,7 @@ static int pch_lpc_disabled(struct pch_lpc *priv) (readl(priv->base + LPC_INT_STS) == 0xffffffff); } -static int pch_lpc_suspend(void) +static int pch_lpc_suspend(void *data) { pch_lpc_priv->saved_reg_ctl = readl(pch_lpc_priv->base + LPC_INT_CTL); pch_lpc_priv->saved_reg_ena = readl(pch_lpc_priv->base + LPC_INT_ENA); @@ -159,18 +159,22 @@ static int pch_lpc_suspend(void) return 0; } -static void pch_lpc_resume(void) +static void pch_lpc_resume(void *data) { writel(pch_lpc_priv->saved_reg_ctl, pch_lpc_priv->base + LPC_INT_CTL); writel(pch_lpc_priv->saved_reg_ena, pch_lpc_priv->base + LPC_INT_ENA); writel(pch_lpc_priv->saved_reg_pol, pch_lpc_priv->base + LPC_INT_POL); } -static struct syscore_ops pch_lpc_syscore_ops = { +static const struct syscore_ops pch_lpc_syscore_ops = { .suspend = pch_lpc_suspend, .resume = pch_lpc_resume, }; +static struct syscore pch_lpc_syscore = { + .ops = &pch_lpc_syscore_ops, +}; + int __init pch_lpc_acpi_init(struct irq_domain *parent, struct acpi_madt_lpc_pic *acpi_pchlpc) { @@ -222,7 +226,7 @@ int __init pch_lpc_acpi_init(struct irq_domain *parent, pch_lpc_priv = priv; pch_lpc_handle = irq_handle; - register_syscore_ops(&pch_lpc_syscore_ops); + register_syscore(&pch_lpc_syscore); return 0; diff --git a/drivers/irqchip/irq-loongson-pch-pic.c b/drivers/irqchip/irq-loongson-pch-pic.c index 62e6bf3a0611..c6b369a974a7 100644 --- a/drivers/irqchip/irq-loongson-pch-pic.c +++ b/drivers/irqchip/irq-loongson-pch-pic.c @@ -278,7 +278,7 @@ static void pch_pic_reset(struct pch_pic *priv) } } -static int pch_pic_suspend(void) +static int pch_pic_suspend(void *data) { int i, j; @@ -296,7 +296,7 @@ static int pch_pic_suspend(void) return 0; } -static void pch_pic_resume(void) +static void pch_pic_resume(void *data) { int i, j; @@ -313,11 +313,15 @@ static void pch_pic_resume(void) } } -static struct syscore_ops pch_pic_syscore_ops = { +static const struct syscore_ops pch_pic_syscore_ops = { .suspend = pch_pic_suspend, .resume = pch_pic_resume, }; +static struct syscore pch_pic_syscore = { + .ops = &pch_pic_syscore_ops, +}; + static int pch_pic_init(phys_addr_t addr, unsigned long size, int vec_base, struct irq_domain *parent_domain, struct fwnode_handle *domain_handle, u32 gsi_base) @@ -356,7 +360,7 @@ static int pch_pic_init(phys_addr_t addr, unsigned long size, int vec_base, pch_pic_priv[nr_pics++] = priv; if (nr_pics == 1) - register_syscore_ops(&pch_pic_syscore_ops); + register_syscore(&pch_pic_syscore); return 0; diff --git a/drivers/irqchip/irq-mchp-eic.c b/drivers/irqchip/irq-mchp-eic.c index 979bb86929f8..31093a8ab67c 100644 --- a/drivers/irqchip/irq-mchp-eic.c +++ b/drivers/irqchip/irq-mchp-eic.c @@ -109,7 +109,7 @@ static int mchp_eic_irq_set_wake(struct irq_data *d, unsigned int on) return 0; } -static int mchp_eic_irq_suspend(void) +static int mchp_eic_irq_suspend(void *data) { unsigned int hwirq; @@ -123,7 +123,7 @@ static int mchp_eic_irq_suspend(void) return 0; } -static void mchp_eic_irq_resume(void) +static void mchp_eic_irq_resume(void *data) { unsigned int hwirq; @@ -135,11 +135,15 @@ static void mchp_eic_irq_resume(void) MCHP_EIC_SCFG(hwirq)); } -static struct syscore_ops mchp_eic_syscore_ops = { +static const struct syscore_ops mchp_eic_syscore_ops = { .suspend = mchp_eic_irq_suspend, .resume = mchp_eic_irq_resume, }; +static struct syscore mchp_eic_syscore = { + .ops = &mchp_eic_syscore_ops, +}; + static struct irq_chip mchp_eic_chip = { .name = "eic", .flags = IRQCHIP_MASK_ON_SUSPEND | IRQCHIP_SET_TYPE_MASKED, @@ -258,7 +262,7 @@ static int mchp_eic_probe(struct platform_device *pdev, struct device_node *pare goto clk_unprepare; } - register_syscore_ops(&mchp_eic_syscore_ops); + register_syscore(&mchp_eic_syscore); pr_info("%pOF: EIC registered, nr_irqs %u\n", node, MCHP_EIC_NIRQ); diff --git a/drivers/irqchip/irq-mst-intc.c b/drivers/irqchip/irq-mst-intc.c index 9643cc3a77d7..7f760f555a76 100644 --- a/drivers/irqchip/irq-mst-intc.c +++ b/drivers/irqchip/irq-mst-intc.c @@ -143,7 +143,7 @@ static void mst_intc_polarity_restore(struct mst_intc_chip_data *cd) writew_relaxed(cd->saved_polarity_conf[i], addr + i * 4); } -static void mst_irq_resume(void) +static void mst_irq_resume(void *data) { struct mst_intc_chip_data *cd; @@ -151,7 +151,7 @@ static void mst_irq_resume(void) mst_intc_polarity_restore(cd); } -static int mst_irq_suspend(void) +static int mst_irq_suspend(void *data) { struct mst_intc_chip_data *cd; @@ -160,14 +160,18 @@ static int mst_irq_suspend(void) return 0; } -static struct syscore_ops mst_irq_syscore_ops = { +static const struct syscore_ops mst_irq_syscore_ops = { .suspend = mst_irq_suspend, .resume = mst_irq_resume, }; +static struct syscore mst_irq_syscore = { + .ops = &mst_irq_syscore_ops, +}; + static int __init mst_irq_pm_init(void) { - register_syscore_ops(&mst_irq_syscore_ops); + register_syscore(&mst_irq_syscore); return 0; } late_initcall(mst_irq_pm_init); diff --git a/drivers/irqchip/irq-mtk-cirq.c b/drivers/irqchip/irq-mtk-cirq.c index de481ba340f8..9571f622774e 100644 --- a/drivers/irqchip/irq-mtk-cirq.c +++ b/drivers/irqchip/irq-mtk-cirq.c @@ -199,7 +199,7 @@ static const struct irq_domain_ops cirq_domain_ops = { }; #ifdef CONFIG_PM_SLEEP -static int mtk_cirq_suspend(void) +static int mtk_cirq_suspend(void *data) { void __iomem *reg; u32 value, mask; @@ -257,7 +257,7 @@ static int mtk_cirq_suspend(void) return 0; } -static void mtk_cirq_resume(void) +static void mtk_cirq_resume(void *data) { void __iomem *reg = mtk_cirq_reg(cirq_data, CIRQ_CONTROL); u32 value; @@ -272,14 +272,18 @@ static void mtk_cirq_resume(void) writel_relaxed(value, reg); } -static struct syscore_ops mtk_cirq_syscore_ops = { +static const struct syscore_ops mtk_cirq_syscore_ops = { .suspend = mtk_cirq_suspend, .resume = mtk_cirq_resume, }; +static struct syscore mtk_cirq_syscore = { + .ops = &mtk_cirq_syscore_ops, +}; + static void mtk_cirq_syscore_init(void) { - register_syscore_ops(&mtk_cirq_syscore_ops); + register_syscore(&mtk_cirq_syscore); } #else static inline void mtk_cirq_syscore_init(void) {} diff --git a/drivers/irqchip/irq-realtek-rtl.c b/drivers/irqchip/irq-realtek-rtl.c index 942c1f8c363d..90fb1a14c1f1 100644 --- a/drivers/irqchip/irq-realtek-rtl.c +++ b/drivers/irqchip/irq-realtek-rtl.c @@ -1,5 +1,10 @@ // SPDX-License-Identifier: GPL-2.0-only /* + * Realtek Interrupt controller. + * + * The Realtek Interrupt controller is a big endian device found in the + * Realtek MIPS SoCs. + * * Copyright (C) 2020 Birger Koblitz * Copyright (C) 2020 Bert Vermeulen * Copyright (C) 2020 John Crispin @@ -23,10 +28,15 @@ #define RTL_ICTL_NUM_INPUTS 32 -#define REG(x) (realtek_ictl_base + x) +#define REG(cpu, x) (realtek_ictl_base[cpu] + x) + +struct realtek_ictl_output { + struct irq_domain *domain; + u32 mask; +}; static DEFINE_RAW_SPINLOCK(irq_lock); -static void __iomem *realtek_ictl_base; +static void __iomem *realtek_ictl_base[NR_CPUS]; /* * IRR0-IRR3 store 4 bits per interrupt, but Realtek uses inverted numbering, @@ -37,60 +47,85 @@ static void __iomem *realtek_ictl_base; #define IRR_OFFSET(idx) (4 * (3 - (idx * 4) / 32)) #define IRR_SHIFT(idx) ((idx * 4) % 32) -static void write_irr(void __iomem *irr0, int idx, u32 value) +static inline void enable_gimr(unsigned int cpu, unsigned int hw_irq) +{ + u32 gimr; + + gimr = readl_be(REG(cpu, RTL_ICTL_GIMR)); + gimr |= BIT(hw_irq); + writel_be(gimr, REG(cpu, RTL_ICTL_GIMR)); +} + +static inline void disable_gimr(unsigned int cpu, unsigned int hw_irq) +{ + u32 gimr; + + gimr = readl_be(REG(cpu, RTL_ICTL_GIMR)); + gimr &= ~BIT(hw_irq); + writel_be(gimr, REG(cpu, RTL_ICTL_GIMR)); +} + +static void write_irr(unsigned int cpu, int hw_irq, u32 value) { - unsigned int offset = IRR_OFFSET(idx); - unsigned int shift = IRR_SHIFT(idx); + void __iomem *irr0 = REG(cpu, RTL_ICTL_IRR0); + unsigned int offset = IRR_OFFSET(hw_irq); + unsigned int shift = IRR_SHIFT(hw_irq); u32 irr; - irr = readl(irr0 + offset) & ~(0xf << shift); + irr = readl_be(irr0 + offset) & ~(0xf << shift); irr |= (value & 0xf) << shift; - writel(irr, irr0 + offset); + writel_be(irr, irr0 + offset); } static void realtek_ictl_unmask_irq(struct irq_data *i) { - unsigned long flags; - u32 value; + unsigned int cpu; - raw_spin_lock_irqsave(&irq_lock, flags); + guard(raw_spinlock)(&irq_lock); + for_each_cpu(cpu, irq_data_get_effective_affinity_mask(i)) + enable_gimr(cpu, i->hwirq); +} - value = readl(REG(RTL_ICTL_GIMR)); - value |= BIT(i->hwirq); - writel(value, REG(RTL_ICTL_GIMR)); +static void realtek_ictl_mask_irq(struct irq_data *i) +{ + unsigned int cpu; - raw_spin_unlock_irqrestore(&irq_lock, flags); + guard(raw_spinlock)(&irq_lock); + for_each_cpu(cpu, irq_data_get_effective_affinity_mask(i)) + disable_gimr(cpu, i->hwirq); } -static void realtek_ictl_mask_irq(struct irq_data *i) +static int realtek_ictl_irq_affinity(struct irq_data *i, const struct cpumask *dest, bool force) { - unsigned long flags; - u32 value; + if (!irqd_irq_masked(i)) + realtek_ictl_mask_irq(i); - raw_spin_lock_irqsave(&irq_lock, flags); + irq_data_update_effective_affinity(i, dest); - value = readl(REG(RTL_ICTL_GIMR)); - value &= ~BIT(i->hwirq); - writel(value, REG(RTL_ICTL_GIMR)); + if (!irqd_irq_masked(i)) + realtek_ictl_unmask_irq(i); - raw_spin_unlock_irqrestore(&irq_lock, flags); + return IRQ_SET_MASK_OK; } static struct irq_chip realtek_ictl_irq = { - .name = "realtek-rtl-intc", - .irq_mask = realtek_ictl_mask_irq, - .irq_unmask = realtek_ictl_unmask_irq, + .name = "realtek-rtl-intc", + .irq_mask = realtek_ictl_mask_irq, + .irq_unmask = realtek_ictl_unmask_irq, + .irq_set_affinity = realtek_ictl_irq_affinity, }; -static int intc_map(struct irq_domain *d, unsigned int irq, irq_hw_number_t hw) +static int intc_map(struct irq_domain *d, unsigned int irq, irq_hw_number_t hw_irq) { - unsigned long flags; + struct realtek_ictl_output *output = d->host_data; + unsigned int cpu; irq_set_chip_and_handler(irq, &realtek_ictl_irq, handle_level_irq); - raw_spin_lock_irqsave(&irq_lock, flags); - write_irr(REG(RTL_ICTL_IRR0), hw, 1); - raw_spin_unlock_irqrestore(&irq_lock, flags); + guard(raw_spinlock_irqsave)(&irq_lock); + output->mask |= BIT(hw_irq); + for_each_present_cpu(cpu) + write_irr(cpu, hw_irq, 1); return 0; } @@ -102,73 +137,99 @@ static const struct irq_domain_ops irq_domain_ops = { static void realtek_irq_dispatch(struct irq_desc *desc) { + struct realtek_ictl_output *output = irq_desc_get_handler_data(desc); struct irq_chip *chip = irq_desc_get_chip(desc); - struct irq_domain *domain; + unsigned int cpu = smp_processor_id(); unsigned long pending; - unsigned int soc_int; + unsigned int hw_irq; chained_irq_enter(chip, desc); - pending = readl(REG(RTL_ICTL_GIMR)) & readl(REG(RTL_ICTL_GISR)); + pending = readl_be(REG(cpu, RTL_ICTL_GIMR)) & + readl_be(REG(cpu, RTL_ICTL_GISR)) & output->mask; if (unlikely(!pending)) { spurious_interrupt(); goto out; } - domain = irq_desc_get_handler_data(desc); - for_each_set_bit(soc_int, &pending, 32) - generic_handle_domain_irq(domain, soc_int); + for_each_set_bit(hw_irq, &pending, RTL_ICTL_NUM_INPUTS) + generic_handle_domain_irq(output->domain, hw_irq); out: chained_irq_exit(chip, desc); } -static int __init realtek_rtl_of_init(struct device_node *node, struct device_node *parent) +static int __init realtek_setup_parents(struct device_node *node) { + int err, parent_irq, num_parents = of_irq_count(node); + struct realtek_ictl_output *output; struct of_phandle_args oirq; struct irq_domain *domain; - unsigned int soc_irq; - int parent_irq; - realtek_ictl_base = of_iomap(node, 0); - if (!realtek_ictl_base) - return -ENXIO; - - /* Disable all cascaded interrupts and clear routing */ - writel(0, REG(RTL_ICTL_GIMR)); - for (soc_irq = 0; soc_irq < RTL_ICTL_NUM_INPUTS; soc_irq++) - write_irr(REG(RTL_ICTL_IRR0), soc_irq, 0); + output = kcalloc(1, sizeof(*output), GFP_KERNEL); + if (!output) + return -ENOMEM; - if (WARN_ON(!of_irq_count(node))) { + if (WARN_ON(!num_parents)) { /* - * If DT contains no parent interrupts, assume MIPS CPU IRQ 2 - * (HW0) is connected to the first output. This is the case for - * all known hardware anyway. "interrupt-map" is deprecated, so - * don't bother trying to parse that. + * If DT contains no parent interrupts, assume MIPS IRQ 2 (HW0) is + * connected to the first output. This is the case for all known hardware. */ - oirq.np = of_find_compatible_node(NULL, NULL, "mti,cpu-interrupt-controller"); + oirq.np = of_find_compatible_node(NULL, NULL, + "mti,cpu-interrupt-controller"); + if (!oirq.np) { + err = -EINVAL; + goto err_out; + } + oirq.args_count = 1; oirq.args[0] = 2; - parent_irq = irq_create_of_mapping(&oirq); - of_node_put(oirq.np); } else { parent_irq = of_irq_get(node, 0); } - if (parent_irq < 0) - return parent_irq; - else if (!parent_irq) - return -ENODEV; + if (parent_irq <= 0) { + err = parent_irq ? parent_irq : -ENODEV; + goto err_out; + } - domain = irq_domain_create_linear(of_fwnode_handle(node), RTL_ICTL_NUM_INPUTS, &irq_domain_ops, NULL); - if (!domain) - return -ENOMEM; + domain = irq_domain_create_linear(of_fwnode_handle(node), RTL_ICTL_NUM_INPUTS, + &irq_domain_ops, output); + if (!domain) { + err = -ENOMEM; + goto err_out; + } - irq_set_chained_handler_and_data(parent_irq, realtek_irq_dispatch, domain); + output->domain = domain; + irq_set_chained_handler_and_data(parent_irq, realtek_irq_dispatch, output); return 0; + +err_out: + kfree(output); + + return err; +} + +static int __init realtek_rtl_of_init(struct device_node *node, struct device_node *parent) +{ + unsigned int cpu; + + for_each_present_cpu(cpu) { + realtek_ictl_base[cpu] = of_iomap(node, cpu); + if (!realtek_ictl_base[cpu]) + return -ENXIO; + + /* Disable all cascaded interrupts and clear routing */ + for (unsigned int hw_irq = 0; hw_irq < RTL_ICTL_NUM_INPUTS; hw_irq++) { + disable_gimr(cpu, hw_irq); + write_irr(cpu, hw_irq, 0); + } + } + + return realtek_setup_parents(node); } IRQCHIP_DECLARE(realtek_rtl_intc, "realtek,rtl-intc", realtek_rtl_of_init); diff --git a/drivers/irqchip/irq-renesas-irqc.c b/drivers/irqchip/irq-renesas-irqc.c index a20a6471b0e4..1ff3535a4617 100644 --- a/drivers/irqchip/irq-renesas-irqc.c +++ b/drivers/irqchip/irq-renesas-irqc.c @@ -176,6 +176,7 @@ static int irqc_probe(struct platform_device *pdev) goto err_runtime_pm_disable; } + p->irq_domain->flags |= IRQ_DOMAIN_FLAG_DESTROY_GC; ret = irq_alloc_domain_generic_chips(p->irq_domain, p->number_of_irqs, 1, "irqc", handle_level_irq, 0, 0, IRQ_GC_INIT_NESTED_LOCK); diff --git a/drivers/irqchip/irq-renesas-rzg2l.c b/drivers/irqchip/irq-renesas-rzg2l.c index c938ab159289..709a84247918 100644 --- a/drivers/irqchip/irq-renesas-rzg2l.c +++ b/drivers/irqchip/irq-renesas-rzg2l.c @@ -103,7 +103,7 @@ static void rzg2l_clear_irq_int(struct rzg2l_irqc_priv *priv, unsigned int hwirq * falling/rising-edge. */ if ((iscr & bit) && (iitsr & IITSR_IITSEL_MASK(hw_irq))) { - writel_relaxed(iscr & ~bit, priv->base + ISCR); + writel_relaxed(~bit, priv->base + ISCR); /* * Enforce that the posted write is flushed to prevent that the * just handled interrupt is raised again. @@ -119,7 +119,7 @@ static void rzg2l_clear_tint_int(struct rzg2l_irqc_priv *priv, unsigned int hwir reg = readl_relaxed(priv->base + TSCR); if (reg & bit) { - writel_relaxed(reg & ~bit, priv->base + TSCR); + writel_relaxed(~bit, priv->base + TSCR); /* * Enforce that the posted write is flushed to prevent that the * just handled interrupt is raised again. @@ -398,7 +398,7 @@ static int rzg2l_irqc_set_type(struct irq_data *d, unsigned int type) return irq_chip_set_type_parent(d, IRQ_TYPE_LEVEL_HIGH); } -static int rzg2l_irqc_irq_suspend(void) +static int rzg2l_irqc_irq_suspend(void *data) { struct rzg2l_irqc_reg_cache *cache = &rzg2l_irqc_data->cache; void __iomem *base = rzg2l_irqc_data->base; @@ -410,7 +410,7 @@ static int rzg2l_irqc_irq_suspend(void) return 0; } -static void rzg2l_irqc_irq_resume(void) +static void rzg2l_irqc_irq_resume(void *data) { struct rzg2l_irqc_reg_cache *cache = &rzg2l_irqc_data->cache; void __iomem *base = rzg2l_irqc_data->base; @@ -425,11 +425,15 @@ static void rzg2l_irqc_irq_resume(void) writel_relaxed(cache->iitsr, base + IITSR); } -static struct syscore_ops rzg2l_irqc_syscore_ops = { +static const struct syscore_ops rzg2l_irqc_syscore_ops = { .suspend = rzg2l_irqc_irq_suspend, .resume = rzg2l_irqc_irq_resume, }; +static struct syscore rzg2l_irqc_syscore = { + .ops = &rzg2l_irqc_syscore_ops, +}; + static const struct irq_chip rzg2l_irqc_chip = { .name = "rzg2l-irqc", .irq_eoi = rzg2l_irqc_eoi, @@ -577,7 +581,7 @@ static int rzg2l_irqc_common_probe(struct platform_device *pdev, struct device_n return -ENOMEM; } - register_syscore_ops(&rzg2l_irqc_syscore_ops); + register_syscore(&rzg2l_irqc_syscore); return 0; } diff --git a/drivers/irqchip/irq-sa11x0.c b/drivers/irqchip/irq-sa11x0.c index d8d4dff16276..e5f24c5f3f41 100644 --- a/drivers/irqchip/irq-sa11x0.c +++ b/drivers/irqchip/irq-sa11x0.c @@ -85,7 +85,7 @@ static struct sa1100irq_state { unsigned int iccr; } sa1100irq_state; -static int sa1100irq_suspend(void) +static int sa1100irq_suspend(void *data) { struct sa1100irq_state *st = &sa1100irq_state; @@ -102,7 +102,7 @@ static int sa1100irq_suspend(void) return 0; } -static void sa1100irq_resume(void) +static void sa1100irq_resume(void *data) { struct sa1100irq_state *st = &sa1100irq_state; @@ -114,14 +114,18 @@ static void sa1100irq_resume(void) } } -static struct syscore_ops sa1100irq_syscore_ops = { +static const struct syscore_ops sa1100irq_syscore_ops = { .suspend = sa1100irq_suspend, .resume = sa1100irq_resume, }; +static struct syscore sa1100irq_syscore = { + .ops = &sa1100irq_syscore_ops, +}; + static int __init sa1100irq_init_devicefs(void) { - register_syscore_ops(&sa1100irq_syscore_ops); + register_syscore(&sa1100irq_syscore); return 0; } diff --git a/drivers/irqchip/irq-sifive-plic.c b/drivers/irqchip/irq-sifive-plic.c index 0799c15c745d..ca70641982ed 100644 --- a/drivers/irqchip/irq-sifive-plic.c +++ b/drivers/irqchip/irq-sifive-plic.c @@ -250,7 +250,7 @@ static int plic_irq_set_type(struct irq_data *d, unsigned int type) return IRQ_SET_MASK_OK; } -static int plic_irq_suspend(void) +static int plic_irq_suspend(void *data) { unsigned int i, cpu; unsigned long flags; @@ -282,7 +282,7 @@ static int plic_irq_suspend(void) return 0; } -static void plic_irq_resume(void) +static void plic_irq_resume(void *data) { unsigned int i, index, cpu; unsigned long flags; @@ -313,11 +313,15 @@ static void plic_irq_resume(void) } } -static struct syscore_ops plic_irq_syscore_ops = { +static const struct syscore_ops plic_irq_syscore_ops = { .suspend = plic_irq_suspend, .resume = plic_irq_resume, }; +static struct syscore plic_irq_syscore = { + .ops = &plic_irq_syscore_ops, +}; + static int plic_irqdomain_map(struct irq_domain *d, unsigned int irq, irq_hw_number_t hwirq) { @@ -683,7 +687,7 @@ static int plic_probe(struct fwnode_handle *fwnode) cpuhp_setup_state(CPUHP_AP_IRQ_SIFIVE_PLIC_STARTING, "irqchip/sifive/plic:starting", plic_starting_cpu, plic_dying_cpu); - register_syscore_ops(&plic_irq_syscore_ops); + register_syscore(&plic_irq_syscore); plic_global_setup_done = true; } } diff --git a/drivers/irqchip/irq-sun6i-r.c b/drivers/irqchip/irq-sun6i-r.c index 37d4b29763bc..23251831c06e 100644 --- a/drivers/irqchip/irq-sun6i-r.c +++ b/drivers/irqchip/irq-sun6i-r.c @@ -268,7 +268,7 @@ static const struct irq_domain_ops sun6i_r_intc_domain_ops = { .free = irq_domain_free_irqs_common, }; -static int sun6i_r_intc_suspend(void) +static int sun6i_r_intc_suspend(void *data) { u32 buf[BITS_TO_U32(MAX(SUN6I_NR_TOP_LEVEL_IRQS, SUN6I_NR_MUX_BITS))]; int i; @@ -284,7 +284,7 @@ static int sun6i_r_intc_suspend(void) return 0; } -static void sun6i_r_intc_resume(void) +static void sun6i_r_intc_resume(void *data) { int i; @@ -294,17 +294,21 @@ static void sun6i_r_intc_resume(void) writel_relaxed(0, base + SUN6I_IRQ_ENABLE(i)); } -static void sun6i_r_intc_shutdown(void) +static void sun6i_r_intc_shutdown(void *data) { - sun6i_r_intc_suspend(); + sun6i_r_intc_suspend(data); } -static struct syscore_ops sun6i_r_intc_syscore_ops = { +static const struct syscore_ops sun6i_r_intc_syscore_ops = { .suspend = sun6i_r_intc_suspend, .resume = sun6i_r_intc_resume, .shutdown = sun6i_r_intc_shutdown, }; +static struct syscore sun6i_r_intc_syscore = { + .ops = &sun6i_r_intc_syscore_ops, +}; + static int __init sun6i_r_intc_init(struct device_node *node, struct device_node *parent, const struct sun6i_r_intc_variant *v) @@ -346,10 +350,10 @@ static int __init sun6i_r_intc_init(struct device_node *node, return -ENOMEM; } - register_syscore_ops(&sun6i_r_intc_syscore_ops); + register_syscore(&sun6i_r_intc_syscore); sun6i_r_intc_ack_nmi(); - sun6i_r_intc_resume(); + sun6i_r_intc_resume(NULL); return 0; } diff --git a/drivers/irqchip/irq-tegra.c b/drivers/irqchip/irq-tegra.c index 66cbb9f77ff3..b6382cf6359a 100644 --- a/drivers/irqchip/irq-tegra.c +++ b/drivers/irqchip/irq-tegra.c @@ -132,7 +132,7 @@ static int tegra_set_wake(struct irq_data *d, unsigned int enable) return 0; } -static int tegra_ictlr_suspend(void) +static int tegra_ictlr_suspend(void *data) { unsigned long flags; unsigned int i; @@ -161,7 +161,7 @@ static int tegra_ictlr_suspend(void) return 0; } -static void tegra_ictlr_resume(void) +static void tegra_ictlr_resume(void *data) { unsigned long flags; unsigned int i; @@ -184,14 +184,18 @@ static void tegra_ictlr_resume(void) local_irq_restore(flags); } -static struct syscore_ops tegra_ictlr_syscore_ops = { +static const struct syscore_ops tegra_ictlr_syscore_ops = { .suspend = tegra_ictlr_suspend, .resume = tegra_ictlr_resume, }; +static struct syscore tegra_ictlr_syscore = { + .ops = &tegra_ictlr_syscore_ops, +}; + static void tegra_ictlr_syscore_init(void) { - register_syscore_ops(&tegra_ictlr_syscore_ops); + register_syscore(&tegra_ictlr_syscore); } #else #define tegra_set_wake NULL diff --git a/drivers/irqchip/irq-vic.c b/drivers/irqchip/irq-vic.c index 2bcdf216a000..e38104c5064e 100644 --- a/drivers/irqchip/irq-vic.c +++ b/drivers/irqchip/irq-vic.c @@ -120,7 +120,7 @@ static void resume_one_vic(struct vic_device *vic) writel(~vic->soft_int, base + VIC_INT_SOFT_CLEAR); } -static void vic_resume(void) +static void vic_resume(void *data) { int id; @@ -146,7 +146,7 @@ static void suspend_one_vic(struct vic_device *vic) writel(~vic->resume_irqs, base + VIC_INT_ENABLE_CLEAR); } -static int vic_suspend(void) +static int vic_suspend(void *data) { int id; @@ -156,11 +156,15 @@ static int vic_suspend(void) return 0; } -static struct syscore_ops vic_syscore_ops = { +static const struct syscore_ops vic_syscore_ops = { .suspend = vic_suspend, .resume = vic_resume, }; +static struct syscore vic_syscore = { + .ops = &vic_syscore_ops, +}; + /** * vic_pm_init - initcall to register VIC pm * @@ -171,7 +175,7 @@ static struct syscore_ops vic_syscore_ops = { static int __init vic_pm_init(void) { if (vic_id > 0) - register_syscore_ops(&vic_syscore_ops); + register_syscore(&vic_syscore); return 0; } diff --git a/drivers/leds/leds-gpio.c b/drivers/leds/leds-gpio.c index a3428b22de3a..49a918731618 100644 --- a/drivers/leds/leds-gpio.c +++ b/drivers/leds/leds-gpio.c @@ -278,6 +278,7 @@ static int gpio_led_probe(struct platform_device *pdev) if (IS_ERR(led_dat->gpiod)) { dev_info(dev, "Skipping unavailable LED gpio %d (%s)\n", template->gpio, template->name); + led_dat->gpiod = NULL; continue; } diff --git a/drivers/leds/leds-pca9532.c b/drivers/leds/leds-pca9532.c index 0344189bb991..af141dec4b92 100644 --- a/drivers/leds/leds-pca9532.c +++ b/drivers/leds/leds-pca9532.c @@ -325,9 +325,9 @@ static int pca9532_gpio_set_value(struct gpio_chip *gc, unsigned int offset, struct pca9532_led *led = &data->leds[offset]; if (val) - led->state = PCA9532_ON; - else led->state = PCA9532_OFF; + else + led->state = PCA9532_ON; pca9532_setled(led); @@ -347,7 +347,7 @@ static int pca9532_gpio_get_value(struct gpio_chip *gc, unsigned offset) static int pca9532_gpio_direction_input(struct gpio_chip *gc, unsigned offset) { /* To use as input ensure pin is not driven */ - pca9532_gpio_set_value(gc, offset, 0); + pca9532_gpio_set_value(gc, offset, 1); return 0; } @@ -395,10 +395,14 @@ static int pca9532_configure(struct i2c_client *client, for (i = 0; i < 2; i++) { data->pwm[i] = pdata->pwm[i]; data->psc[i] = pdata->psc[i]; - i2c_smbus_write_byte_data(client, PCA9532_REG_PWM(maxleds, i), - data->pwm[i]); - i2c_smbus_write_byte_data(client, PCA9532_REG_PSC(maxleds, i), - data->psc[i]); + err = i2c_smbus_write_byte_data(client, PCA9532_REG_PWM(maxleds, i), + data->pwm[i]); + if (err < 0) + return err; + err = i2c_smbus_write_byte_data(client, PCA9532_REG_PSC(maxleds, i), + data->psc[i]); + if (err < 0) + return err; } data->hw_blink = true; diff --git a/drivers/leds/leds-st1202.c b/drivers/leds/leds-st1202.c index 4e5dd76d714d..f5b53a9ed59c 100644 --- a/drivers/leds/leds-st1202.c +++ b/drivers/leds/leds-st1202.c @@ -32,10 +32,11 @@ #define ST1202_ILED_REG0 0x09 #define ST1202_MAX_LEDS 12 #define ST1202_MAX_PATTERNS 8 -#define ST1202_MILLIS_PATTERN_DUR_MAX 5660 +#define ST1202_MILLIS_PATTERN_DUR_MAX (ST1202_MILLIS_PATTERN_DUR_MIN * U8_MAX) #define ST1202_MILLIS_PATTERN_DUR_MIN 22 #define ST1202_PATTERN_DUR 0x16 #define ST1202_PATTERN_PWM 0x1E +#define ST1202_PATTERN_PWM_FULL 0x0FFF #define ST1202_PATTERN_REP 0x15 struct st1202_led { @@ -86,7 +87,7 @@ static int st1202_write_reg(struct st1202_chip *chip, int reg, uint8_t val) static uint8_t st1202_prescalar_to_miliseconds(unsigned int value) { - return value / ST1202_MILLIS_PATTERN_DUR_MIN - 1; + return value / ST1202_MILLIS_PATTERN_DUR_MIN; } static int st1202_pwm_pattern_write(struct st1202_chip *chip, int led_num, @@ -128,37 +129,11 @@ static int st1202_duration_pattern_write(struct st1202_chip *chip, int pattern, st1202_prescalar_to_miliseconds(value)); } -static void st1202_brightness_set(struct led_classdev *led_cdev, - enum led_brightness value) -{ - struct st1202_led *led = cdev_to_st1202_led(led_cdev); - struct st1202_chip *chip = led->chip; - - guard(mutex)(&chip->lock); - - st1202_write_reg(chip, ST1202_ILED_REG0 + led->led_num, value); -} - -static enum led_brightness st1202_brightness_get(struct led_classdev *led_cdev) -{ - struct st1202_led *led = cdev_to_st1202_led(led_cdev); - struct st1202_chip *chip = led->chip; - u8 value = 0; - - guard(mutex)(&chip->lock); - - st1202_read_reg(chip, ST1202_ILED_REG0 + led->led_num, &value); - - return value; -} - -static int st1202_channel_set(struct st1202_chip *chip, int led_num, bool active) +static int __st1202_channel_set(struct st1202_chip *chip, int led_num, bool active) { u8 chan_low, chan_high; int ret; - guard(mutex)(&chip->lock); - if (led_num <= 7) { ret = st1202_read_reg(chip, ST1202_CHAN_ENABLE_LOW, &chan_low); if (ret < 0) @@ -186,6 +161,40 @@ static int st1202_channel_set(struct st1202_chip *chip, int led_num, bool active return 0; } +static int st1202_channel_set(struct st1202_chip *chip, int led_num, bool active) +{ + guard(mutex)(&chip->lock); + + return __st1202_channel_set(chip, led_num, active); +} + +static void st1202_brightness_set(struct led_classdev *led_cdev, + enum led_brightness value) +{ + struct st1202_led *led = cdev_to_st1202_led(led_cdev); + struct st1202_chip *chip = led->chip; + + guard(mutex)(&chip->lock); + + for (int patt = 0; patt < ST1202_MAX_PATTERNS; patt++) + st1202_pwm_pattern_write(chip, led->led_num, patt, ST1202_PATTERN_PWM_FULL); + st1202_write_reg(chip, ST1202_ILED_REG0 + led->led_num, value); + __st1202_channel_set(chip, led->led_num, !!value); +} + +static enum led_brightness st1202_brightness_get(struct led_classdev *led_cdev) +{ + struct st1202_led *led = cdev_to_st1202_led(led_cdev); + struct st1202_chip *chip = led->chip; + u8 value = 0; + + guard(mutex)(&chip->lock); + + st1202_read_reg(chip, ST1202_ILED_REG0 + led->led_num, &value); + + return value; +} + static int st1202_led_set(struct led_classdev *ldev, enum led_brightness value) { struct st1202_led *led = cdev_to_st1202_led(ldev); @@ -201,12 +210,16 @@ static int st1202_led_pattern_clear(struct led_classdev *ldev) guard(mutex)(&chip->lock); + ret = st1202_write_reg(chip, ST1202_CONFIG_REG, ST1202_CONFIG_REG_SHFT); + if (ret != 0) + return ret; + for (int patt = 0; patt < ST1202_MAX_PATTERNS; patt++) { - ret = st1202_pwm_pattern_write(chip, led->led_num, patt, LED_OFF); + ret = st1202_pwm_pattern_write(chip, led->led_num, patt, ST1202_PATTERN_PWM_FULL); if (ret != 0) return ret; - ret = st1202_duration_pattern_write(chip, patt, ST1202_MILLIS_PATTERN_DUR_MIN); + ret = st1202_write_reg(chip, ST1202_PATTERN_DUR + patt, 0); if (ret != 0) return ret; } @@ -227,6 +240,10 @@ static int st1202_led_pattern_set(struct led_classdev *ldev, guard(mutex)(&chip->lock); + ret = st1202_write_reg(chip, ST1202_CONFIG_REG, ST1202_CONFIG_REG_SHFT); + if (ret != 0) + return ret; + for (int patt = 0; patt < len; patt++) { if (pattern[patt].delta_t < ST1202_MILLIS_PATTERN_DUR_MIN || pattern[patt].delta_t > ST1202_MILLIS_PATTERN_DUR_MAX) @@ -245,6 +262,10 @@ static int st1202_led_pattern_set(struct led_classdev *ldev, if (ret != 0) return ret; + ret = __st1202_channel_set(chip, led->led_num, true); + if (ret != 0) + return ret; + ret = st1202_write_reg(chip, ST1202_CONFIG_REG, (ST1202_CONFIG_REG_PATSR | ST1202_CONFIG_REG_PATS | ST1202_CONFIG_REG_SHFT)); if (ret != 0) @@ -257,13 +278,19 @@ static int st1202_dt_init(struct st1202_chip *chip) { struct device *dev = &chip->client->dev; struct st1202_led *led; - int err, reg; + int err; + u32 reg; for_each_available_child_of_node_scoped(dev_of_node(dev), child) { err = of_property_read_u32(child, "reg", ®); if (err) return dev_err_probe(dev, err, "Invalid register\n"); + if (reg >= ST1202_MAX_LEDS) + return dev_err_probe(dev, -EINVAL, + "LED reg %u out of range [0, %d]\n", + reg, ST1202_MAX_LEDS - 1); + led = &chip->leds[reg]; led->is_active = true; led->fwnode = of_fwnode_handle(child); @@ -323,11 +350,6 @@ static int st1202_setup(struct st1202_chip *chip) if (ret < 0) return ret; - ret = st1202_write_reg(chip, ST1202_CONFIG_REG, - ST1202_CONFIG_REG_PATS | ST1202_CONFIG_REG_PATSR); - if (ret < 0) - return ret; - return 0; } diff --git a/drivers/leds/trigger/ledtrig-cpu.c b/drivers/leds/trigger/ledtrig-cpu.c index 05848a2fecff..679323c2ccda 100644 --- a/drivers/leds/trigger/ledtrig-cpu.c +++ b/drivers/leds/trigger/ledtrig-cpu.c @@ -94,28 +94,32 @@ void ledtrig_cpu(enum cpu_led_event ledevt) } EXPORT_SYMBOL(ledtrig_cpu); -static int ledtrig_cpu_syscore_suspend(void) +static int ledtrig_cpu_syscore_suspend(void *data) { ledtrig_cpu(CPU_LED_STOP); return 0; } -static void ledtrig_cpu_syscore_resume(void) +static void ledtrig_cpu_syscore_resume(void *data) { ledtrig_cpu(CPU_LED_START); } -static void ledtrig_cpu_syscore_shutdown(void) +static void ledtrig_cpu_syscore_shutdown(void *data) { ledtrig_cpu(CPU_LED_HALTED); } -static struct syscore_ops ledtrig_cpu_syscore_ops = { +static const struct syscore_ops ledtrig_cpu_syscore_ops = { .shutdown = ledtrig_cpu_syscore_shutdown, .suspend = ledtrig_cpu_syscore_suspend, .resume = ledtrig_cpu_syscore_resume, }; +static struct syscore ledtrig_cpu_syscore = { + .ops = &ledtrig_cpu_syscore_ops, +}; + static int ledtrig_online_cpu(unsigned int cpu) { ledtrig_cpu(CPU_LED_START); @@ -157,7 +161,7 @@ static int __init ledtrig_cpu_init(void) led_trigger_register_simple(trig->name, &trig->_trig); } - register_syscore_ops(&ledtrig_cpu_syscore_ops); + register_syscore(&ledtrig_cpu_syscore); ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "leds/trigger:starting", ledtrig_online_cpu, ledtrig_prepare_down_cpu); diff --git a/drivers/macintosh/via-pmu.c b/drivers/macintosh/via-pmu.c index b0f09c70f1ff..5fe47e784d43 100644 --- a/drivers/macintosh/via-pmu.c +++ b/drivers/macintosh/via-pmu.c @@ -2600,7 +2600,7 @@ void pmu_blink(int n) #if defined(CONFIG_SUSPEND) && defined(CONFIG_PPC32) int pmu_sys_suspended; -static int pmu_syscore_suspend(void) +static int pmu_syscore_suspend(void *data) { /* Suspend PMU event interrupts */ pmu_suspend(); @@ -2614,7 +2614,7 @@ static int pmu_syscore_suspend(void) return 0; } -static void pmu_syscore_resume(void) +static void pmu_syscore_resume(void *data) { struct adb_request req; @@ -2634,14 +2634,18 @@ static void pmu_syscore_resume(void) pmu_sys_suspended = 0; } -static struct syscore_ops pmu_syscore_ops = { +static const struct syscore_ops pmu_syscore_ops = { .suspend = pmu_syscore_suspend, .resume = pmu_syscore_resume, }; +static struct syscore pmu_syscore = { + .ops = &pmu_syscore_ops, +}; + static int pmu_syscore_register(void) { - register_syscore_ops(&pmu_syscore_ops); + register_syscore(&pmu_syscore); return 0; } diff --git a/drivers/mailbox/pcc.c b/drivers/mailbox/pcc.c index 713022aed2e2..5189791c2790 100644 --- a/drivers/mailbox/pcc.c +++ b/drivers/mailbox/pcc.c @@ -93,12 +93,11 @@ struct pcc_chan_reg { * @plat_irq: platform interrupt * @type: PCC subspace type * @plat_irq_flags: platform interrupt flags - * @chan_in_use: this flag is used just to check if the interrupt needs - * handling when it is shared. Since only one transfer can occur - * at a time and mailbox takes care of locking, this flag can be - * accessed without a lock. Note: the type only support the - * communication from OSPM to Platform, like type3, use it, and - * other types completely ignore it. + * @chan_in_use: lockless flag used by type 3 initiator subspaces to filter + * platform interrupts. Only one transfer can occur at a time, but + * the interrupt handler may sample the flag on another CPU, so all + * accesses must use READ_ONCE() or WRITE_ONCE(). Other subspace + * types do not test it. */ struct pcc_chan_info { struct pcc_mbox_chan chan; @@ -322,8 +321,13 @@ static irqreturn_t pcc_mbox_irq(int irq, void *p) if (pcc_chan_reg_read_modify_write(&pchan->plat_irq_ack)) return IRQ_NONE; + /* + * Initiator subspaces use this flag to filter shared interrupts. Use + * READ_ONCE() to sample the lockless flag written by pcc_send_data() + * on another CPU. + */ if (pchan->type == ACPI_PCCT_TYPE_EXT_PCC_MASTER_SUBSPACE && - !pchan->chan_in_use) + !READ_ONCE(pchan->chan_in_use)) return IRQ_NONE; if (!pcc_mbox_cmd_complete_check(pchan)) @@ -333,12 +337,12 @@ static irqreturn_t pcc_mbox_irq(int irq, void *p) return IRQ_NONE; /* - * Clear this flag after updating interrupt ack register and just - * before mbox_chan_received_data() which might call pcc_send_data() - * where the flag is set again to start new transfer. This is - * required to avoid any possible race in updatation of this flag. + * Clear this flag after updating the interrupt ack register and before + * notifying the client and mailbox core. mbox_chan_txdone() may submit + * the next queued transfer and set the flag again. Use WRITE_ONCE() for + * the lockless update observed by the send and interrupt paths. */ - pchan->chan_in_use = false; + WRITE_ONCE(pchan->chan_in_use, false); mbox_chan_received_data(chan, NULL); pcc_chan_acknowledge(pchan); @@ -437,9 +441,18 @@ static int pcc_send_data(struct mbox_chan *chan, void *data) if (ret) return ret; + /* + * Set chan_in_use before ringing the doorbell so a fast completion + * interrupt is not mistaken for a shared interrupt from another + * subspace. Use WRITE_ONCE() for the lockless flag update. The + * ordered I/O accessor used to ring the doorbell orders this store + * before the platform is notified. + */ + if (pchan->plat_irq > 0) + WRITE_ONCE(pchan->chan_in_use, true); ret = pcc_chan_reg_read_modify_write(&pchan->db); - if (!ret && pchan->plat_irq > 0) - pchan->chan_in_use = true; + if (ret && pchan->plat_irq > 0) + WRITE_ONCE(pchan->chan_in_use, false); return ret; } diff --git a/drivers/mailbox/qcom-cpucp-mbox.c b/drivers/mailbox/qcom-cpucp-mbox.c index 44f4ed15f818..57f731418faf 100644 --- a/drivers/mailbox/qcom-cpucp-mbox.c +++ b/drivers/mailbox/qcom-cpucp-mbox.c @@ -56,14 +56,25 @@ static irqreturn_t qcom_cpucp_mbox_irq_fn(int irq, void *data) for_each_set_bit(i, (unsigned long *)&status, APSS_CPUCP_IPC_CHAN_SUPPORTED) { u32 val = readl(cpucp->rx_base + APSS_CPUCP_RX_MBOX_CMD(i) + APSS_CPUCP_MBOX_CMD_OFF); struct mbox_chan *chan = &cpucp->chans[i]; + struct mbox_client *cl; unsigned long flags; - /* Provide mutual exclusion with changes to chan->cl */ + /* + * Provide mutual exclusion with changes to chan->cl. + * Save cl locally and clear the HW interrupt inside the lock, + * then invoke mbox_chan_received_data() outside the lock to + * avoid a PREEMPT_RT self-deadlock: mbox_chan_received_data() + * can call back into mbox_send_message() via scmi_rx_callback() + * -> mailbox_clear_channel(), which re-acquires chan->lock + * (converted to an rt_spinlock under PREEMPT_RT). + */ spin_lock_irqsave(&chan->lock, flags); - if (chan->cl) - mbox_chan_received_data(chan, &val); + cl = chan->cl; writeq(BIT(i), cpucp->rx_base + APSS_CPUCP_RX_MBOX_CLEAR); spin_unlock_irqrestore(&chan->lock, flags); + + if (cl) + mbox_chan_received_data(chan, &val); } return IRQ_HANDLED; @@ -99,6 +110,14 @@ static int qcom_cpucp_mbox_send_data(struct mbox_chan *chan, void *data) unsigned long chan_id = channel_number(chan); u32 *val = data; + /* + * mailbox_clear_channel() calls mbox_send_message() with NULL data to + * signal the remote side that the channel has been cleared. Nothing + * needs to be written to the TX register in that case, so just return. + */ + if (!val) + return 0; + writel(*val, cpucp->tx_base + APSS_CPUCP_TX_MBOX_CMD(chan_id) + APSS_CPUCP_MBOX_CMD_OFF); return 0; diff --git a/drivers/mailbox/riscv-sbi-mpxy-mbox.c b/drivers/mailbox/riscv-sbi-mpxy-mbox.c index 7c9c006b7244..714f7fb97a2f 100644 --- a/drivers/mailbox/riscv-sbi-mpxy-mbox.c +++ b/drivers/mailbox/riscv-sbi-mpxy-mbox.c @@ -314,8 +314,13 @@ static int mpxy_get_notifications(u32 channel_id, channel_id, 0, 0, 0, 0, 0); if (sret.error) goto err_put_cpu; + if (sret.value < 0 || mpxy_shmem_size < sizeof(*notif_data) || + sret.value > mpxy_shmem_size - sizeof(*notif_data)) { + put_cpu(); + return -EOVERFLOW; + } - memcpy(notif_data, mpxy->shmem, sret.value + 16); + memcpy(notif_data, mpxy->shmem, sret.value + sizeof(*notif_data)); *events_data_len = sret.value; err_put_cpu: @@ -480,11 +485,14 @@ static void mpxy_mbox_peek_rpmi_data(struct mbox_chan *chan, struct rpmi_mbox_message msg; unsigned long pos = 0; - while (pos < events_data_len && (events_data_len - pos) <= sizeof(*event)) { + while (events_data_len - pos >= sizeof(*event)) { event = (struct rpmi_notification_event *)(notif->events_data + pos); msg.type = RPMI_MBOX_MSG_TYPE_NOTIFICATION_EVENT; msg.notif.event_datalen = le16_to_cpu(event->event_datalen); + if (msg.notif.event_datalen > + events_data_len - pos - sizeof(*event)) + break; msg.notif.event_id = event->event_id; msg.notif.event_data = event->event_data; msg.error = 0; diff --git a/drivers/mailbox/rockchip-mailbox.c b/drivers/mailbox/rockchip-mailbox.c index 4d966cb2ed03..5775362de508 100644 --- a/drivers/mailbox/rockchip-mailbox.c +++ b/drivers/mailbox/rockchip-mailbox.c @@ -201,19 +201,10 @@ static int rockchip_mbox_probe(struct platform_device *pdev) /* Each channel has two buffers for A2B and B2A */ mb->buf_size = (size_t)resource_size(res) / (drv_data->num_chans * 2); - mb->pclk = devm_clk_get(&pdev->dev, "pclk_mailbox"); - if (IS_ERR(mb->pclk)) { - ret = PTR_ERR(mb->pclk); - dev_err(&pdev->dev, "failed to get pclk_mailbox clock: %d\n", - ret); - return ret; - } - - ret = clk_prepare_enable(mb->pclk); - if (ret) { - dev_err(&pdev->dev, "failed to enable pclk: %d\n", ret); - return ret; - } + mb->pclk = devm_clk_get_enabled(&pdev->dev, "pclk_mailbox"); + if (IS_ERR(mb->pclk)) + return dev_err_probe(&pdev->dev, PTR_ERR(mb->pclk), + "failed to get and enable pclk_mailbox clock\n"); for (i = 0; i < mb->mbox.num_chans; i++) { irq = platform_get_irq(pdev, i); diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c index 46a815f20b09..6536143f8912 100644 --- a/drivers/md/dm-integrity.c +++ b/drivers/md/dm-integrity.c @@ -65,6 +65,7 @@ #define SB_VERSION_4 4 #define SB_VERSION_5 5 #define SB_VERSION_6 6 +#define SB_VERSION_7 7 #define SB_SECTORS 8 #define MAX_SECTORS_PER_BLOCK 8 @@ -90,6 +91,7 @@ struct superblock { #define SB_FLAG_FIXED_PADDING 0x8 #define SB_FLAG_FIXED_HMAC 0x10 #define SB_FLAG_INLINE 0x20 +#define SB_FLAG_DISCARD_KEYED 0x40 #define JOURNAL_ENTRY_ROUNDUP 8 @@ -276,6 +278,7 @@ struct dm_integrity_c { bool recalculate_flag; bool reset_recalculate_flag; bool discard; + bool discard_keyed; bool fix_padding; bool fix_hmac; bool legacy_recalculate; @@ -482,7 +485,9 @@ static void wraparound_section(struct dm_integrity_c *ic, unsigned int *sec_ptr) static void sb_set_version(struct dm_integrity_c *ic) { - if (ic->sb->flags & cpu_to_le32(SB_FLAG_INLINE)) + if (ic->sb->flags & cpu_to_le32(SB_FLAG_DISCARD_KEYED)) + ic->sb->version = SB_VERSION_7; + else if (ic->sb->flags & cpu_to_le32(SB_FLAG_INLINE)) ic->sb->version = SB_VERSION_6; else if (ic->sb->flags & cpu_to_le32(SB_FLAG_FIXED_HMAC)) ic->sb->version = SB_VERSION_5; @@ -1415,7 +1420,7 @@ static int dm_integrity_rw_tag(struct dm_integrity_c *ic, unsigned char *tag, se { unsigned int hash_offset = 0; unsigned char mismatch_hash = 0; - unsigned char mismatch_filler = !ic->discard; + unsigned char mismatch_filler = !ic->discard || ic->discard_keyed; do { unsigned char *data, *dp; @@ -1467,7 +1472,7 @@ static int dm_integrity_rw_tag(struct dm_integrity_c *ic, unsigned char *tag, se } hash_offset = 0; mismatch_hash = 0; - mismatch_filler = !ic->discard; + mismatch_filler = !ic->discard || ic->discard_keyed; } } } @@ -1645,7 +1650,8 @@ static void integrity_end_io(struct bio *bio) } static void integrity_sector_checksum_shash(struct dm_integrity_c *ic, sector_t sector, - const char *data, unsigned offset, char *result) + const char *data, unsigned offset, + unsigned int len, char *result) { __le64 sector_le = cpu_to_le64(sector); SHASH_DESC_ON_STACK(req, ic->internal_shash); @@ -1674,10 +1680,12 @@ static void integrity_sector_checksum_shash(struct dm_integrity_c *ic, sector_t goto failed; } - r = crypto_shash_update(req, data + offset, ic->sectors_per_block << SECTOR_SHIFT); - if (unlikely(r < 0)) { - dm_integrity_io_error(ic, "crypto_shash_update", r); - goto failed; + if (likely(len)) { + r = crypto_shash_update(req, data + offset, len); + if (unlikely(r < 0)) { + dm_integrity_io_error(ic, "crypto_shash_update", r); + goto failed; + } } r = crypto_shash_final(req, result); @@ -1698,7 +1706,8 @@ static void integrity_sector_checksum_shash(struct dm_integrity_c *ic, sector_t } static void integrity_sector_checksum_ahash(struct dm_integrity_c *ic, struct ahash_request **ahash_req, - sector_t sector, struct page *page, unsigned offset, char *result) + sector_t sector, struct page *page, unsigned offset, + unsigned int len, char *result) { __le64 sector_le = cpu_to_le64(sector); struct ahash_request *req; @@ -1707,6 +1716,7 @@ static void integrity_sector_checksum_ahash(struct dm_integrity_c *ic, struct ah int r; unsigned int digest_size; unsigned int nbytes = 0; + unsigned int nents = 1 + (len ? 1 : 0); might_sleep(); @@ -1720,12 +1730,12 @@ static void integrity_sector_checksum_ahash(struct dm_integrity_c *ic, struct ah ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP, crypto_req_done, &wait); if (ic->sb->flags & cpu_to_le32(SB_FLAG_FIXED_HMAC)) { - sg_init_table(sg, 3); + sg_init_table(sg, nents + 1); sg_set_buf(s, (const __u8 *)&ic->sb->salt, SALT_SIZE); nbytes += SALT_SIZE; s++; } else { - sg_init_table(sg, 2); + sg_init_table(sg, nents); } if (likely(!is_vmalloc_addr(§or_le))) { @@ -1738,8 +1748,10 @@ static void integrity_sector_checksum_ahash(struct dm_integrity_c *ic, struct ah nbytes += sizeof(sector_le); s++; - sg_set_page(s, page, ic->sectors_per_block << SECTOR_SHIFT, offset); - nbytes += ic->sectors_per_block << SECTOR_SHIFT; + if (likely(len)) { + sg_set_page(s, page, len, offset); + nbytes += len; + } ahash_request_set_crypt(req, sg, result, nbytes); @@ -1762,11 +1774,41 @@ static void integrity_sector_checksum_ahash(struct dm_integrity_c *ic, struct ah static void integrity_sector_checksum(struct dm_integrity_c *ic, struct ahash_request **ahash_req, sector_t sector, const char *data, unsigned offset, char *result) +{ + unsigned int len = ic->sectors_per_block << SECTOR_SHIFT; + + if (likely(ic->internal_shash != NULL)) + integrity_sector_checksum_shash(ic, sector, data, offset, len, result); + else + integrity_sector_checksum_ahash(ic, ahash_req, sector, (struct page *)data, + offset, len, result); +} + +/* + * Authenticated marker for a discarded block: HMAC_key(salt || sector), with + * no data payload. Because a real data tag's input always covers a full + * block, its length differs from this marker's, so the two can never + * collide structurally, regardless of block content. + */ +static void integrity_discard_checksum(struct dm_integrity_c *ic, struct ahash_request **ahash_req, + sector_t sector, char *result) { if (likely(ic->internal_shash != NULL)) - integrity_sector_checksum_shash(ic, sector, data, offset, result); + integrity_sector_checksum_shash(ic, sector, NULL, 0, 0, result); else - integrity_sector_checksum_ahash(ic, ahash_req, sector, (struct page *)data, offset, result); + integrity_sector_checksum_ahash(ic, ahash_req, sector, NULL, 0, 0, result); +} + +static void integrity_discard_fill_tags(struct dm_integrity_c *ic, struct ahash_request **ahash_req, + unsigned char *checksums, sector_t *sector, + unsigned int blocks) +{ + unsigned int i; + + for (i = 0; i < blocks; i++) { + integrity_discard_checksum(ic, ahash_req, *sector, checksums + i * ic->tag_size); + *sector += ic->sectors_per_block; + } } static void *integrity_kmap(struct dm_integrity_c *ic, struct page *p) @@ -1795,6 +1837,29 @@ static void *integrity_identity(struct dm_integrity_c *ic, void *data) return virt_to_page(data); } +static int integrity_recheck_verify_tag(struct dm_integrity_io *dio, char *checksum, + char *on_disk_tag, sector_t logical_sector) +{ + struct dm_integrity_c *ic = dio->ic; + int r; + + if (!ic->discard_keyed) + return dm_integrity_rw_tag(ic, checksum, &dio->metadata_block, + &dio->metadata_offset, ic->tag_size, TAG_CMP); + + r = dm_integrity_rw_tag(ic, on_disk_tag, &dio->metadata_block, + &dio->metadata_offset, ic->tag_size, TAG_READ); + if (unlikely(r)) + return r; + + r = crypto_memneq(on_disk_tag, checksum, ic->tag_size); + if (unlikely(r)) { + integrity_discard_checksum(ic, &dio->ahash_req, logical_sector, checksum); + r = crypto_memneq(on_disk_tag, checksum, ic->tag_size); + } + return r; +} + static noinline void integrity_recheck(struct dm_integrity_io *dio, char *checksum) { struct bio *bio = dm_bio_from_per_bio_data(dio, sizeof(struct dm_integrity_io)); @@ -1820,6 +1885,7 @@ static noinline void integrity_recheck(struct dm_integrity_io *dio, char *checks char *mem; char *buffer = page_to_virt(page); unsigned int buffer_offset; + char on_disk_tag[MAX_T(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)]; int r; struct dm_io_request io_req; struct dm_io_region io_loc; @@ -1847,8 +1913,8 @@ static noinline void integrity_recheck(struct dm_integrity_io *dio, char *checks } integrity_sector_checksum(ic, &dio->ahash_req, logical_sector, integrity_identity(ic, buffer), buffer_offset, checksum); - r = dm_integrity_rw_tag(ic, checksum, &dio->metadata_block, - &dio->metadata_offset, ic->tag_size, TAG_CMP); + r = integrity_recheck_verify_tag(dio, checksum, on_disk_tag, + logical_sector); if (r) { if (r > 0) { DMERR_LIMIT("%pg: Checksum failed at sector 0x%llx", @@ -1913,14 +1979,19 @@ static void integrity_metadata(struct work_struct *w) if (unlikely(dio->op == REQ_OP_DISCARD)) { unsigned int bi_size = dio->bio_details.bi_iter.bi_size; unsigned int max_size = likely(checksums != checksums_onstack) ? PAGE_SIZE : HASH_MAX_DIGESTSIZE; - unsigned int max_blocks = max_size / ic->tag_size; + unsigned int max_blocks = (max_size - extra_space) / ic->tag_size; + sector_t sector = dio->range.logical_sector; - memset(checksums, DISCARD_FILLER, max_size); + if (!ic->discard_keyed) + memset(checksums, DISCARD_FILLER, max_size); while (bi_size) { unsigned int this_step_blocks = bi_size >> (SECTOR_SHIFT + ic->sb->log2_sectors_per_block); this_step_blocks = min(this_step_blocks, max_blocks); + if (ic->discard_keyed) + integrity_discard_fill_tags(ic, &dio->ahash_req, checksums, + §or, this_step_blocks); r = dm_integrity_rw_tag(ic, checksums, &dio->metadata_block, &dio->metadata_offset, this_step_blocks * ic->tag_size, TAG_WRITE); if (unlikely(r)) { @@ -3797,6 +3868,8 @@ static void dm_integrity_resume(struct dm_target *ti) ic->wrote_to_journal = false; flags = ic->sb->flags & cpu_to_le32(SB_FLAG_RECALCULATING); + if (ic->discard_keyed) + flags |= cpu_to_le32(SB_FLAG_DISCARD_KEYED); r = sync_rw_sb(ic, REQ_OP_READ); if (r) dm_integrity_io_error(ic, "reading superblock", r); @@ -3944,7 +4017,8 @@ static void dm_integrity_status(struct dm_target *ti, status_type_t type, arg_count += ic->sectors_per_block != 1; arg_count += !!(ic->sb->flags & cpu_to_le32(SB_FLAG_RECALCULATING)); arg_count += ic->reset_recalculate_flag; - arg_count += ic->discard; + arg_count += ic->discard && !ic->discard_keyed; + arg_count += ic->discard_keyed; arg_count += ic->mode != 'I'; /* interleave_sectors */ arg_count += ic->mode == 'J'; /* journal_sectors */ arg_count += ic->mode == 'J'; /* journal_watermark */ @@ -3967,8 +4041,10 @@ static void dm_integrity_status(struct dm_target *ti, status_type_t type, DMEMIT(" recalculate"); if (ic->reset_recalculate_flag) DMEMIT(" reset_recalculate"); - if (ic->discard) + if (ic->discard && !ic->discard_keyed) DMEMIT(" allow_discards"); + if (ic->discard_keyed) + DMEMIT(" allow_discards_keyed"); if (ic->mode != 'I') DMEMIT(" interleave_sectors:%u", 1U << ic->sb->log2_interleave_sectors); DMEMIT(" buffer_sectors:%u", 1U << ic->log2_buffer_sectors); @@ -4018,6 +4094,7 @@ static void dm_integrity_status(struct dm_target *ti, status_type_t type, DMEMIT(",recalculate=%c", (ic->sb->flags & cpu_to_le32(SB_FLAG_RECALCULATING)) ? 'y' : 'n'); DMEMIT(",allow_discards=%c", ic->discard ? 'y' : 'n'); + DMEMIT(",allow_discards_keyed=%c", ic->discard_keyed ? 'y' : 'n'); DMEMIT(",fix_padding=%c", ((ic->sb->flags & cpu_to_le32(SB_FLAG_FIXED_PADDING)) != 0) ? 'y' : 'n'); DMEMIT(",fix_hmac=%c", @@ -4179,6 +4256,9 @@ static int initialize_superblock(struct dm_integrity_c *ic, get_random_bytes(ic->sb->salt, SALT_SIZE); } + if (ic->discard_keyed) + ic->sb->flags |= cpu_to_le32(SB_FLAG_DISCARD_KEYED); + if (!ic->meta_dev) { if (ic->fix_padding) ic->sb->flags |= cpu_to_le32(SB_FLAG_FIXED_PADDING); @@ -4838,6 +4918,9 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv ic->reset_recalculate_flag = true; } else if (!strcmp(opt_string, "allow_discards")) { ic->discard = true; + } else if (!strcmp(opt_string, "allow_discards_keyed")) { + ic->discard = true; + ic->discard_keyed = true; } else if (!strcmp(opt_string, "fix_padding")) { ic->fix_padding = true; } else if (!strcmp(opt_string, "fix_hmac")) { @@ -4966,6 +5049,11 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv ti->error = "Discard can be only used with internal hash"; goto bad; } + if (ic->discard_keyed && !ic->internal_hash_alg.key) { + r = -EINVAL; + ti->error = "Keyed discard can only be used with keyed internal hash"; + goto bad; + } ic->autocommit_jiffies = msecs_to_jiffies(sync_msec); ic->autocommit_msec = sync_msec; @@ -5084,7 +5172,7 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv should_write_sb = true; } - if (!ic->sb->version || ic->sb->version > SB_VERSION_6) { + if (!ic->sb->version || ic->sb->version > SB_VERSION_7) { r = -EINVAL; ti->error = "Unknown version"; goto bad; @@ -5132,6 +5220,11 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv goto bad; } } + if (!ic->discard_keyed && (ic->sb->flags & cpu_to_le32(SB_FLAG_DISCARD_KEYED))) { + r = -EINVAL; + ti->error = "Keyed discard cannot be disabled once enabled"; + goto bad; + } if (!!(ic->sb->flags & cpu_to_le32(SB_FLAG_HAVE_JOURNAL_MAC)) != !!ic->journal_mac_alg.alg_string) { r = -EINVAL; ti->error = "Journal mac mismatch"; @@ -5432,7 +5525,7 @@ static void dm_integrity_dtr(struct dm_target *ti) static struct target_type integrity_target = { .name = "integrity", - .version = {1, 14, 0}, + .version = {1, 15, 0}, .module = THIS_MODULE, .features = DM_TARGET_SINGLETON | DM_TARGET_INTEGRITY, .ctr = dm_integrity_ctr, diff --git a/drivers/md/dm-pcache/cache.c b/drivers/md/dm-pcache/cache.c index 887157fde68d..8635fbf780aa 100644 --- a/drivers/md/dm-pcache/cache.c +++ b/drivers/md/dm-pcache/cache.c @@ -197,6 +197,7 @@ static int cache_tail_init(struct pcache_cache *cache) { struct dm_pcache *pcache = CACHE_TO_PCACHE(cache); bool new_cache = !(cache->cache_info.flags & PCACHE_CACHE_FLAGS_INIT_DONE); + int ret; if (new_cache) { __set_bit(0, cache->seg_map); @@ -213,6 +214,12 @@ static int cache_tail_init(struct pcache_cache *cache) pcache_dev_err(pcache, "Corrupted key tail or dirty tail.\n"); return -EIO; } + + ret = cache_verify_dirty_tail(cache); + if (ret) { + pcache_dev_err(pcache, "dirty tail chain does not terminate (crafted cache image?)\n"); + return ret; + } } return 0; diff --git a/drivers/md/dm-pcache/cache.h b/drivers/md/dm-pcache/cache.h index afc112b79496..bf61a4a487bc 100644 --- a/drivers/md/dm-pcache/cache.h +++ b/drivers/md/dm-pcache/cache.h @@ -180,6 +180,7 @@ struct pcache_cache { u32 advance; int ret; } writeback_ctx; + atomic_t writeback_errors; char gc_kset_onmedia_buf[PCACHE_KSET_ONMEDIA_SIZE_MAX]; struct delayed_work gc_work; @@ -651,6 +652,8 @@ static inline int cache_decode_dirty_tail(struct pcache_cache *cache) &cache->dirty_tail_index); } +int cache_verify_dirty_tail(struct pcache_cache *cache); + int pcache_cache_init(void); void pcache_cache_exit(void); #endif /* _PCACHE_CACHE_H */ diff --git a/drivers/md/dm-pcache/cache_gc.c b/drivers/md/dm-pcache/cache_gc.c index 3088cfef951b..99999a92c4e6 100644 --- a/drivers/md/dm-pcache/cache_gc.c +++ b/drivers/md/dm-pcache/cache_gc.c @@ -134,6 +134,11 @@ void pcache_cache_gc_fn(struct work_struct *work) continue; } + if (get_kset_onmedia_size(kset_onmedia) > cache_seg_remain(&key_tail)) { + atomic_inc(&cache->gc_errors); + return; + } + for (i = 0; i < kset_onmedia->key_num; i++) { struct pcache_cache_key key_tmp = { 0 }; diff --git a/drivers/md/dm-pcache/cache_key.c b/drivers/md/dm-pcache/cache_key.c index a3bb1f95afb9..c5a8ac02b216 100644 --- a/drivers/md/dm-pcache/cache_key.c +++ b/drivers/md/dm-pcache/cache_key.c @@ -90,10 +90,19 @@ int cache_key_decode(struct pcache_cache *cache, struct pcache_cache_key *key) { struct dm_pcache *pcache = CACHE_TO_PCACHE(cache); + u64 dev_bytes = (u64)cache->dev_size << SECTOR_SHIFT; key->off = key_onmedia->off; key->len = key_onmedia->len; + if (key_onmedia->len == 0 || + key_onmedia->len > dev_bytes || + key_onmedia->off > dev_bytes - key_onmedia->len) { + pcache_dev_err(pcache, "key off %llu + len %u exceeds device size\n", + key_onmedia->off, key_onmedia->len); + return -EIO; + } + key->cache_pos.cache_seg = &cache->segments[key_onmedia->cache_seg_id]; key->cache_pos.seg_off = key_onmedia->cache_seg_off; @@ -811,6 +820,11 @@ int cache_replay(struct pcache_cache *cache) } /* Replay the kset and check for errors. */ + if (get_kset_onmedia_size(kset_onmedia) > cache_seg_remain(pos)) { + ret = -EIO; + goto out; + } + ret = kset_replay(cache, kset_onmedia); if (ret) goto out; @@ -832,6 +846,75 @@ int cache_replay(struct pcache_cache *cache) return ret; } +/* + * cache_verify_dirty_tail - reject a persisted dirty_tail whose last-kset + * chain does not terminate. + * + * dirty_tail is decoded independently of the key_tail chain cache_replay() + * walks, so replay's hop cap does not cover it. A crafted chain that loops + * back on itself makes the writeback worker re-arm forever; walk it once here + * with the same cap and fail the load if it does not end within n_segs hops. + */ +int cache_verify_dirty_tail(struct pcache_cache *cache) +{ + struct pcache_cache_pos pos; + struct pcache_cache_kset_onmedia *kset_onmedia; + u32 to_copy, last_hops = 0, count = 0; + int ret = 0; + + kset_onmedia = kzalloc(PCACHE_KSET_ONMEDIA_SIZE_MAX, GFP_KERNEL); + if (!kset_onmedia) + return -ENOMEM; + + cache_pos_copy(&pos, &cache->dirty_tail); + + while (true) { + to_copy = min(PCACHE_KSET_ONMEDIA_SIZE_MAX, cache_seg_remain(&pos)); + ret = copy_mc_to_kernel(kset_onmedia, cache_pos_addr(&pos), to_copy); + if (ret) { + ret = -EIO; + goto out; + } + + /* A missing, short or corrupt kset is the normal end of the chain. */ + if (!kset_onmedia_valid(kset_onmedia) || + kset_onmedia->crc != cache_kset_crc(kset_onmedia)) { + ret = 0; + goto out; + } + + if (kset_onmedia->flags & PCACHE_KSET_FLAGS_LAST) { + if (kset_onmedia->next_cache_seg_id >= cache->cache_info.n_segs) { + ret = -EIO; + goto out; + } + + if (++last_hops > cache->n_segs) { + ret = -EIO; + goto out; + } + + pos.cache_seg = &cache->segments[kset_onmedia->next_cache_seg_id]; + pos.seg_off = 0; + continue; + } + + if (get_kset_onmedia_size(kset_onmedia) > cache_seg_remain(&pos)) { + ret = -EIO; + goto out; + } + + cache_pos_advance(&pos, get_kset_onmedia_size(kset_onmedia)); + if (++count > 512) { + cond_resched(); + count = 0; + } + } +out: + kfree(kset_onmedia); + return ret; +} + int cache_tree_init(struct pcache_cache *cache, struct pcache_cache_tree *cache_tree, u32 n_subtrees) { int ret; diff --git a/drivers/md/dm-pcache/cache_writeback.c b/drivers/md/dm-pcache/cache_writeback.c index 3df8687e2382..656e4d048912 100644 --- a/drivers/md/dm-pcache/cache_writeback.c +++ b/drivers/md/dm-pcache/cache_writeback.c @@ -229,6 +229,9 @@ void cache_writeback_fn(struct work_struct *work) if (pcache_is_stopping(pcache)) goto unlock; + if (atomic_read(&cache->writeback_errors)) + goto unlock; + kset_onmedia = (struct pcache_cache_kset_onmedia *)cache->wb_kset_onmedia_buf; mutex_lock(&cache->dirty_tail_lock); @@ -246,6 +249,11 @@ void cache_writeback_fn(struct work_struct *work) goto queue_work; } + if (get_kset_onmedia_size(kset_onmedia) > cache_seg_remain(&dirty_tail)) { + atomic_inc(&cache->writeback_errors); + goto unlock; + } + ret = cache_kset_insert_tree(cache, kset_onmedia); if (ret) { delay = PCACHE_CACHE_WRITEBACK_INTERVAL; diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c index 2a9584078292..781cbfb2d348 100644 --- a/drivers/md/md-bitmap.c +++ b/drivers/md/md-bitmap.c @@ -2629,10 +2629,12 @@ static ssize_t location_store(struct mddev *mddev, const char *buf, size_t len) { int rv; + unsigned int noio_flags; rv = mddev_suspend_and_lock(mddev); if (rv) return rv; + noio_flags = memalloc_noio_save(); if (mddev->pers) { if (mddev->recovery || mddev->sync_thread) { @@ -2719,6 +2721,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len) } rv = 0; out: + memalloc_noio_restore(noio_flags); mddev_unlock_and_resume(mddev); if (rv) return rv; @@ -2862,7 +2865,7 @@ backlog_store(struct mddev *mddev, const char *buf, size_t len) if (!has_write_mostly) { pr_warn_ratelimited("%s: can't set backlog, no write mostly device available\n", mdname(mddev)); - mddev_unlock(mddev); + mddev_unlock_and_resume(mddev); return -EINVAL; } diff --git a/drivers/md/md-bitmap.h b/drivers/md/md-bitmap.h index f46674bdfeb9..7535742982fc 100644 --- a/drivers/md/md-bitmap.h +++ b/drivers/md/md-bitmap.h @@ -29,6 +29,7 @@ enum bitmap_state { BITMAP_FIRST_USE = 3, /* llbitmap is just created */ BITMAP_CLEAN = 4, /* llbitmap is created with assume_clean */ BITMAP_DAEMON_BUSY = 5, /* llbitmap daemon is not finished after daemon_sleep */ + BITMAP_SHUTDOWN = 6, /* llbitmap is being destroyed */ BITMAP_HOSTENDIAN =15, }; diff --git a/drivers/md/md-llbitmap.c b/drivers/md/md-llbitmap.c index dc9b72494a81..5c60b26ad246 100644 --- a/drivers/md/md-llbitmap.c +++ b/drivers/md/md-llbitmap.c @@ -662,6 +662,7 @@ static enum llbitmap_state llbitmap_state_machine(struct llbitmap *llbitmap, if (state == BitNeedSync) need_resync = !mddev->degraded; else if (state == BitDirty && + !test_bit(BITMAP_SHUTDOWN, &llbitmap->flags) && !timer_pending(&llbitmap->pending_timer)) mod_timer(&llbitmap->pending_timer, jiffies + mddev->bitmap_info.daemon_sleep * HZ); @@ -854,7 +855,7 @@ static int llbitmap_read_sb(struct llbitmap *llbitmap) else mddev->bitmap_info.space = mddev->bitmap_info.default_space; } - llbitmap->flags = le32_to_cpu(sb->state); + llbitmap->flags = le32_to_cpu(sb->state) & ~BIT(BITMAP_SHUTDOWN); if (test_and_clear_bit(BITMAP_FIRST_USE, &llbitmap->flags)) { ret = llbitmap_init(llbitmap); goto out_put_page; @@ -910,6 +911,9 @@ static void llbitmap_pending_timer_fn(struct timer_list *pending_timer) struct llbitmap *llbitmap = container_of(pending_timer, struct llbitmap, pending_timer); + if (test_bit(BITMAP_SHUTDOWN, &llbitmap->flags)) + return; + if (work_busy(&llbitmap->daemon_work)) { pr_warn("md/llbitmap: %s daemon_work not finished in %lu seconds\n", mdname(llbitmap->mddev), @@ -930,6 +934,9 @@ static void md_llbitmap_daemon_fn(struct work_struct *work) bool restart; int idx; + if (test_bit(BITMAP_SHUTDOWN, &llbitmap->flags)) + return; + if (llbitmap->mddev->degraded) return; retry: @@ -969,7 +976,7 @@ static void md_llbitmap_daemon_fn(struct work_struct *work) goto retry; /* If some page is dirty but not expired, setup timer again */ - if (restart) + if (restart && !test_bit(BITMAP_SHUTDOWN, &llbitmap->flags)) mod_timer(&llbitmap->pending_timer, jiffies + llbitmap->mddev->bitmap_info.daemon_sleep * HZ); } @@ -999,10 +1006,11 @@ static int llbitmap_create(struct mddev *mddev) mutex_lock(&mddev->bitmap_info.mutex); mddev->bitmap = llbitmap; ret = llbitmap_read_sb(llbitmap); + if (ret) + mddev->bitmap = NULL; mutex_unlock(&mddev->bitmap_info.mutex); if (ret) { kfree(llbitmap); - mddev->bitmap = NULL; } return ret; @@ -1051,7 +1059,9 @@ static void llbitmap_destroy(struct mddev *mddev) mutex_lock(&mddev->bitmap_info.mutex); - timer_delete_sync(&llbitmap->pending_timer); + set_bit(BITMAP_SHUTDOWN, &llbitmap->flags); + timer_shutdown_sync(&llbitmap->pending_timer); + cancel_work_sync(&llbitmap->daemon_work); flush_workqueue(md_llbitmap_io_wq); flush_workqueue(md_llbitmap_unplug_wq); @@ -1375,7 +1385,7 @@ static void llbitmap_update_sb(void *data) sb = kmap_local_page(sb_page); sb->events = cpu_to_le64(mddev->events); - sb->state = cpu_to_le32(llbitmap->flags); + sb->state = cpu_to_le32(llbitmap->flags & ~BIT(BITMAP_SHUTDOWN)); sb->chunksize = cpu_to_le32(llbitmap->chunksize); sb->sync_size = cpu_to_le64(mddev->resync_max_sectors); sb->events_cleared = cpu_to_le64(llbitmap->events_cleared); diff --git a/drivers/md/md.c b/drivers/md/md.c index 5304e920dbcc..d03d0fdf763d 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -97,7 +97,7 @@ static struct workqueue_struct *md_misc_wq; static int remove_and_add_spares(struct mddev *mddev, struct md_rdev *this); static void mddev_detach(struct mddev *mddev); -static void export_rdev(struct md_rdev *rdev, struct mddev *mddev); +static void export_rdev(struct md_rdev *rdev); static void md_wakeup_thread_directly(struct md_thread __rcu **thread); /* @@ -234,23 +234,22 @@ static int rdev_need_serial(struct md_rdev *rdev) void mddev_create_serial_pool(struct mddev *mddev, struct md_rdev *rdev) { int ret = 0; + unsigned int noio_flags; - if (rdev && !rdev_need_serial(rdev) && + if (!test_bit(MD_SERIALIZE_POLICY, &mddev->flags) && + rdev && !rdev_need_serial(rdev) && !test_bit(CollisionCheck, &rdev->flags)) return; + noio_flags = memalloc_noio_save(); if (!rdev) ret = rdevs_init_serial(mddev); else ret = rdev_init_serial(rdev); if (ret) - return; + goto out; if (mddev->serial_info_pool == NULL) { - /* - * already in memalloc noio context by - * mddev_suspend() - */ mddev->serial_info_pool = mempool_create_kmalloc_pool(NR_SERIAL_INFOS, sizeof(struct serial_info)); @@ -259,6 +258,8 @@ void mddev_create_serial_pool(struct mddev *mddev, struct md_rdev *rdev) pr_err("can't alloc memory pool for serialization\n"); } } +out: + memalloc_noio_restore(noio_flags); } /* @@ -517,9 +518,6 @@ int mddev_suspend(struct mddev *mddev, bool interruptible) */ WRITE_ONCE(mddev->suspended, mddev->suspended + 1); - /* restrict memory reclaim I/O during raid array is suspend */ - mddev->noio_flag = memalloc_noio_save(); - mutex_unlock(&mddev->suspend_mutex); return 0; } @@ -536,9 +534,6 @@ static void __mddev_resume(struct mddev *mddev, bool recovery_needed) return; } - /* entred the memalloc scope from mddev_suspend() */ - memalloc_noio_restore(mddev->noio_flag); - percpu_ref_resurrect(&mddev->active_io); wake_up(&mddev->sb_wait); @@ -972,7 +967,7 @@ void mddev_unlock(struct mddev *mddev) list_for_each_entry_safe(rdev, tmp, &delete, same_set) { list_del_init(&rdev->same_set); kobject_del(&rdev->kobj); - export_rdev(rdev, mddev); + export_rdev(rdev); } if (!legacy_async_del_gendisk) { @@ -2647,7 +2642,7 @@ void md_autodetect_dev(dev_t dev); /* just for claiming the bdev */ static struct md_rdev claim_rdev; -static void export_rdev(struct md_rdev *rdev, struct mddev *mddev) +static void export_rdev(struct md_rdev *rdev) { pr_debug("md: export_rdev(%pg)\n", rdev->bdev); md_rdev_clear(rdev); @@ -4047,6 +4042,7 @@ level_store(struct mddev *mddev, const char *buf, size_t len) char clevel[16]; ssize_t rv; size_t slen = len; + unsigned int noio_flags; struct md_personality *pers, *oldpers; long level; void *priv, *oldpriv; @@ -4058,6 +4054,7 @@ level_store(struct mddev *mddev, const char *buf, size_t len) rv = mddev_suspend_and_lock(mddev); if (rv) return rv; + noio_flags = memalloc_noio_save(); if (mddev->pers == NULL) { memcpy(mddev->clevel, buf, slen); @@ -4233,6 +4230,7 @@ level_store(struct mddev *mddev, const char *buf, size_t len) md_new_event(); rv = len; out_unlock: + memalloc_noio_restore(noio_flags); mddev_unlock_and_resume(mddev); return rv; } @@ -4412,6 +4410,7 @@ static ssize_t raid_disks_store(struct mddev *mddev, const char *buf, size_t len) { unsigned int n; + unsigned int noio_flags; int err; err = kstrtouint(buf, 10, &n); @@ -4421,9 +4420,11 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len) err = mddev_suspend_and_lock(mddev); if (err) return err; - if (mddev->pers) - err = update_raid_disks(mddev, n); - else if (mddev->reshape_position != MaxSector) { + noio_flags = memalloc_noio_save(); + if (mddev->pers) { + if (n != mddev->raid_disks) + err = update_raid_disks(mddev, n); + } else if (mddev->reshape_position != MaxSector) { struct md_rdev *rdev; int olddisks = mddev->raid_disks - mddev->delta_disks; @@ -4443,6 +4444,7 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len) } else mddev->raid_disks = n; out_unlock: + memalloc_noio_restore(noio_flags); mddev_unlock_and_resume(mddev); return err ? err : len; } @@ -4823,6 +4825,7 @@ new_dev_store(struct mddev *mddev, const char *buf, size_t len) int minor; dev_t dev; struct md_rdev *rdev; + unsigned int noio_flags; int err; if (!*buf || *e != ':' || !e[1] || e[1] == '\n') @@ -4838,6 +4841,7 @@ new_dev_store(struct mddev *mddev, const char *buf, size_t len) err = mddev_suspend_and_lock(mddev); if (err) return err; + noio_flags = memalloc_noio_save(); if (mddev->persistent) { rdev = md_import_device(dev, mddev->major_version, mddev->minor_version); @@ -4856,13 +4860,15 @@ new_dev_store(struct mddev *mddev, const char *buf, size_t len) rdev = md_import_device(dev, -1, -1); if (IS_ERR(rdev)) { + memalloc_noio_restore(noio_flags); mddev_unlock_and_resume(mddev); return PTR_ERR(rdev); } err = bind_rdev_to_array(rdev, mddev); out: if (err) - export_rdev(rdev, mddev); + export_rdev(rdev); + memalloc_noio_restore(noio_flags); mddev_unlock_and_resume(mddev); if (!err) md_new_event(); @@ -6951,8 +6957,8 @@ static void __md_stop(struct mddev *mddev) { struct md_personality *pers = mddev->pers; - md_bitmap_destroy(mddev); mddev_detach(mddev); + md_bitmap_destroy(mddev); spin_lock(&mddev->lock); mddev->pers = NULL; spin_unlock(&mddev->lock); @@ -7188,7 +7194,7 @@ static void autorun_devices(int part) rdev_for_each_list(rdev, tmp, &candidates) { list_del_init(&rdev->same_set); if (bind_rdev_to_array(rdev, mddev)) - export_rdev(rdev, mddev); + export_rdev(rdev); } autorun_array(mddev); mddev_unlock_and_resume(mddev); @@ -7198,7 +7204,7 @@ static void autorun_devices(int part) */ rdev_for_each_list(rdev, tmp, &candidates) { list_del_init(&rdev->same_set); - export_rdev(rdev, mddev); + export_rdev(rdev); } mddev_put(mddev); } @@ -7386,13 +7392,13 @@ int md_add_new_disk(struct mddev *mddev, struct mdu_disk_info_s *info) pr_warn("md: %pg has different UUID to %pg\n", rdev->bdev, rdev0->bdev); - export_rdev(rdev, mddev); + export_rdev(rdev); return -EINVAL; } } err = bind_rdev_to_array(rdev, mddev); if (err) - export_rdev(rdev, mddev); + export_rdev(rdev); return err; } @@ -7435,7 +7441,7 @@ int md_add_new_disk(struct mddev *mddev, struct mdu_disk_info_s *info) /* This was a hot-add request, but events doesn't * match, so reject it. */ - export_rdev(rdev, mddev); + export_rdev(rdev); return -EINVAL; } @@ -7461,7 +7467,7 @@ int md_add_new_disk(struct mddev *mddev, struct mdu_disk_info_s *info) } } if (has_journal || mddev->bitmap) { - export_rdev(rdev, mddev); + export_rdev(rdev); return -EBUSY; } set_bit(Journal, &rdev->flags); @@ -7476,7 +7482,7 @@ int md_add_new_disk(struct mddev *mddev, struct mdu_disk_info_s *info) /* --add initiated by this node */ err = mddev->cluster_ops->add_new_disk(mddev, rdev); if (err) { - export_rdev(rdev, mddev); + export_rdev(rdev); return err; } } @@ -7486,7 +7492,7 @@ int md_add_new_disk(struct mddev *mddev, struct mdu_disk_info_s *info) err = bind_rdev_to_array(rdev, mddev); if (err) - export_rdev(rdev, mddev); + export_rdev(rdev); if (mddev_is_clustered(mddev)) { if (info->state & (1 << MD_DISK_CANDIDATE)) { @@ -7549,7 +7555,7 @@ int md_add_new_disk(struct mddev *mddev, struct mdu_disk_info_s *info) err = bind_rdev_to_array(rdev, mddev); if (err) { - export_rdev(rdev, mddev); + export_rdev(rdev); return err; } } @@ -7661,7 +7667,7 @@ static int hot_add_disk(struct mddev *mddev, dev_t dev) return 0; abort_export: - export_rdev(rdev, mddev); + export_rdev(rdev); return err; } @@ -8209,8 +8215,10 @@ static int md_ioctl(struct block_device *bdev, blk_mode_t mode, unsigned int cmd, unsigned long arg) { int err = 0; + unsigned int noio_flags = 0; void __user *argp = (void __user *)arg; struct mddev *mddev = NULL; + bool suspend; err = md_ioctl_valid(cmd); if (err) @@ -8260,13 +8268,15 @@ static int md_ioctl(struct block_device *bdev, blk_mode_t mode, if (!md_is_rdwr(mddev)) flush_work(&mddev->sync_work); - err = md_ioctl_need_suspend(cmd) ? mddev_suspend_and_lock(mddev) : - mddev_lock(mddev); + suspend = md_ioctl_need_suspend(cmd); + err = suspend ? mddev_suspend_and_lock(mddev) : mddev_lock(mddev); if (err) { pr_debug("md: ioctl lock interrupted, reason %d, cmd %d\n", err, cmd); goto out; } + if (suspend) + noio_flags = memalloc_noio_save(); if (cmd == SET_ARRAY_INFO) { err = __md_set_array_info(mddev, argp); @@ -8391,8 +8401,12 @@ static int md_ioctl(struct block_device *bdev, blk_mode_t mode, err != -EINVAL) mddev->hold_active = 0; - md_ioctl_need_suspend(cmd) ? mddev_unlock_and_resume(mddev) : - mddev_unlock(mddev); + if (suspend) { + memalloc_noio_restore(noio_flags); + mddev_unlock_and_resume(mddev); + } else { + mddev_unlock(mddev); + } out: if (cmd == STOP_ARRAY_RO || (err && cmd == STOP_ARRAY)) @@ -9299,6 +9313,8 @@ static void md_clone_bio(struct mddev *mddev, struct bio **bio) md_io_clone->mddev = mddev; if (blk_queue_io_stat(bdev->bd_disk->queue)) md_io_clone->start_time = bio_start_io_acct(*bio); + else + md_io_clone->start_time = 0; if (bio_data_dir(*bio) == WRITE && md_bitmap_enabled(mddev, false)) { md_io_clone->offset = (*bio)->bi_iter.bi_sector; @@ -10071,19 +10087,34 @@ static void md_start_sync(struct work_struct *ws) struct mddev *mddev = container_of(ws, struct mddev, sync_work); int spares = 0; bool suspend = false; + unsigned int noio_flags = 0; char *name; /* * If reshape is still in progress, spares won't be added or removed * from conf until reshape is done. */ - if (mddev->reshape_position == MaxSector && + if ((mddev->reshape_position == MaxSector || !md_is_rdwr(mddev)) && md_spares_need_change(mddev)) { suspend = true; mddev_suspend(mddev, false); + noio_flags = memalloc_noio_save(); } mddev_lock_nointr(mddev); + + /* + * The spare configuration can change before reconfig_mutex is acquired. + * Recheck while holding the lock and suspend if needed. + */ + if (!suspend && (mddev->reshape_position == MaxSector || !md_is_rdwr(mddev)) && + md_spares_need_change(mddev)) { + mddev_unlock(mddev); + mddev_suspend_and_lock_nointr(mddev); + suspend = true; + noio_flags = memalloc_noio_save(); + } + if (!md_is_rdwr(mddev)) { /* * On a read-only array we can: @@ -10127,8 +10158,10 @@ static void md_start_sync(struct work_struct *ws) * https://bugzilla.kernel.org/show_bug.cgi?id=218200 * Therefore, use __mddev_resume(mddev, false). */ - if (suspend) + if (suspend) { + memalloc_noio_restore(noio_flags); __mddev_resume(mddev, false); + } md_wakeup_thread(mddev->sync_thread); sysfs_notify_dirent_safe(mddev->sysfs_action); md_new_event(); @@ -10147,8 +10180,10 @@ static void md_start_sync(struct work_struct *ws) * https://bugzilla.kernel.org/show_bug.cgi?id=218200 * Therefore, use __mddev_resume(mddev, false). */ - if (suspend) + if (suspend) { + memalloc_noio_restore(noio_flags); __mddev_resume(mddev, false); + } wake_up(&resync_wait); if (test_and_clear_bit(MD_RECOVERY_RECOVER, &mddev->recovery) && diff --git a/drivers/md/md.h b/drivers/md/md.h index 2960a9874760..7400f7739d2e 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -617,7 +617,6 @@ struct mddev { struct md_cluster_info *cluster_info; struct md_cluster_operations *cluster_ops; unsigned int good_device_nr; /* good device num within cluster raid */ - unsigned int noio_flag; /* for memalloc scope API */ /* * Temporarily store rdev that will be finally removed when diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 16625b79788b..1370be1e5cc7 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -3480,8 +3480,6 @@ static void *raid1_takeover(struct mddev *mddev) mddev->new_chunk_sectors = 0; conf = setup_conf(mddev); if (!IS_ERR(conf)) { - /* Array must appear to be quiesced */ - conf->array_frozen = 1; mddev_clear_unsupported_flags(mddev, UNSUPPORTED_MDDEV_FLAGS); } diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 1ae9a587a959..c4c01619bf7e 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -1348,6 +1348,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, int i, k; sector_t sectors; int max_sectors; + bool atomic = bio->bi_opf & REQ_ATOMIC; if ((mddev_is_clustered(mddev) && mddev->cluster_ops->area_resyncing(mddev, WRITE, @@ -1454,16 +1455,6 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, if (is_bad) { int good_sectors; - /* - * We cannot atomically write this, so just - * error in that case. It could be possible to - * atomically write other mirrors, but the - * complexity of supporting that is not worth - * the benefit. - */ - if (bio->bi_opf & REQ_ATOMIC) - goto err_handle; - good_sectors = first_bad - dev_sector; if (good_sectors < max_sectors) max_sectors = good_sectors; @@ -1483,6 +1474,9 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, r10_bio->sectors = max_sectors; if (r10_bio->sectors < bio_sectors(bio)) { + if (atomic) + goto err_handle; + allow_barrier(conf); bio = bio_submit_split_bioset(bio, r10_bio->sectors, &conf->bio_split); diff --git a/drivers/md/raid5-ppl.c b/drivers/md/raid5-ppl.c index 56b234683ee6..2678e82bfa52 100644 --- a/drivers/md/raid5-ppl.c +++ b/drivers/md/raid5-ppl.c @@ -643,8 +643,10 @@ static void ppl_do_flush(struct ppl_io_unit *io) log->disk_flush_bitmap = 0; for (i = flushed_disks ; i < raid_disks; i++) { - if (atomic_dec_and_test(&io->pending_flushes)) + if (atomic_dec_and_test(&io->pending_flushes)) { ppl_io_unit_finished(io); + break; + } } } diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index b19d870e4171..8ff8cce3da7b 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -2456,11 +2456,6 @@ static int scribble_alloc(struct raid5_percpu *percpu, sizeof(unsigned int) * (num + 2); void *scribble; - /* - * If here is in raid array suspend context, it is in memalloc noio - * context as well, there is no potential recursive memory reclaim - * I/Os with the GFP_KERNEL flag. - */ scribble = kvmalloc_array(cnt, obj_size, GFP_KERNEL); if (!scribble) return -ENOMEM; @@ -2475,6 +2470,7 @@ static int scribble_alloc(struct raid5_percpu *percpu, static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors) { unsigned long cpu; + unsigned int noio_flags; int err = 0; /* Never shrink. */ @@ -2483,6 +2479,7 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors) return 0; raid5_quiesce(conf->mddev, true); + noio_flags = memalloc_noio_save(); cpus_read_lock(); for_each_present_cpu(cpu) { @@ -2496,6 +2493,7 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors) } cpus_read_unlock(); + memalloc_noio_restore(noio_flags); raid5_quiesce(conf->mddev, false); if (!err) { @@ -3541,7 +3539,7 @@ static void __add_stripe_bio(struct stripe_head *sh, struct bio *bi, sh->dev[dd_idx].sector); if (conf->mddev->bitmap && firstwrite && !sh->batch_head) { - sh->bm_seq = conf->seq_flush+1; + sh->bm_seq = READ_ONCE(conf->seq_flush) + 1; set_bit(STRIPE_BIT_DELAY, &sh->state); } } @@ -3740,11 +3738,10 @@ static int want_replace(struct stripe_head *sh, int disk_idx) int rv = 0; rdev = sh->raid_conf->disks[disk_idx].replacement; - if (rdev - && !test_bit(Faulty, &rdev->flags) - && !test_bit(In_sync, &rdev->flags) - && (rdev->recovery_offset <= sh->sector - || rdev->mddev->resync_offset <= sh->sector)) + if (rdev && !test_bit(Faulty, &rdev->flags) && + !test_bit(In_sync, &rdev->flags) && + (READ_ONCE(rdev->recovery_offset) <= sh->sector || + rdev->mddev->resync_offset <= sh->sector)) rv = 1; return rv; } @@ -4673,7 +4670,8 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s) */ rdev = conf->disks[i].replacement; if (rdev && !test_bit(Faulty, &rdev->flags) && - rdev->recovery_offset >= sh->sector + RAID5_STRIPE_SECTORS(conf) && + READ_ONCE(rdev->recovery_offset) >= + sh->sector + RAID5_STRIPE_SECTORS(conf) && !rdev_has_badblock(rdev, sh->sector, RAID5_STRIPE_SECTORS(conf))) set_bit(R5_ReadRepl, &dev->flags); @@ -4715,7 +4713,7 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s) } else if (test_bit(In_sync, &rdev->flags)) set_bit(R5_Insync, &dev->flags); else if (sh->sector + RAID5_STRIPE_SECTORS(conf) <= - rdev->recovery_offset) { + READ_ONCE(rdev->recovery_offset)) { /* * in sync if: * - normal IO, or @@ -5448,13 +5446,13 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio) rdev = conf->disks[dd_idx].replacement; if (!rdev || test_bit(Faulty, &rdev->flags) || - rdev->recovery_offset < end_sector) { + READ_ONCE(rdev->recovery_offset) < end_sector) { rdev = conf->disks[dd_idx].rdev; if (!rdev) return 0; if (test_bit(Faulty, &rdev->flags) || !(test_bit(In_sync, &rdev->flags) || - rdev->recovery_offset >= end_sector)) + READ_ONCE(rdev->recovery_offset) >= end_sector)) return 0; } @@ -5767,7 +5765,7 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi) } spin_unlock_irq(&sh->stripe_lock); if (conf->mddev->bitmap) { - sh->bm_seq = conf->seq_flush + 1; + sh->bm_seq = READ_ONCE(conf->seq_flush) + 1; set_bit(STRIPE_BIT_DELAY, &sh->state); } @@ -5924,8 +5922,11 @@ static void raid5_bitmap_sector(struct mddev *mddev, sector_t *offset, sectors_per_chunk = conf->chunk_sectors * (conf->raid_disks - conf->max_degraded); - start = round_down(start, sectors_per_chunk); - end = round_up(end, sectors_per_chunk); + sector_div(start, sectors_per_chunk); + start *= sectors_per_chunk; + if (sector_div(end, sectors_per_chunk)) + end++; + end *= sectors_per_chunk; start = raid5_compute_sector(conf, start, 0, &dd_idx, NULL); end = raid5_compute_sector(conf, end, 0, &dd_idx, NULL); @@ -5943,8 +5944,10 @@ static void raid5_bitmap_sector(struct mddev *mddev, sector_t *offset, sectors_per_chunk = conf->prev_chunk_sectors * (conf->previous_raid_disks - conf->max_degraded); - prev_start = round_down(prev_start, sectors_per_chunk); - prev_end = round_down(prev_end, sectors_per_chunk); + sector_div(prev_start, sectors_per_chunk); + prev_start *= sectors_per_chunk; + sector_div(prev_end, sectors_per_chunk); + prev_end *= sectors_per_chunk; prev_start = raid5_compute_sector(conf, prev_start, 1, &dd_idx, NULL); prev_end = raid5_compute_sector(conf, prev_end, 1, &dd_idx, NULL); @@ -6380,8 +6383,8 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr, int *sk if (rdev->raid_disk >= 0 && !test_bit(Journal, &rdev->flags) && !test_bit(In_sync, &rdev->flags) && - rdev->recovery_offset < sector_nr) - rdev->recovery_offset = sector_nr; + READ_ONCE(rdev->recovery_offset) < sector_nr) + WRITE_ONCE(rdev->recovery_offset, sector_nr); conf->reshape_checkpoint = jiffies; set_bit(MD_SB_CHANGE_DEVS, &mddev->sb_flags); @@ -6489,8 +6492,8 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr, int *sk if (rdev->raid_disk >= 0 && !test_bit(Journal, &rdev->flags) && !test_bit(In_sync, &rdev->flags) && - rdev->recovery_offset < sector_nr) - rdev->recovery_offset = sector_nr; + READ_ONCE(rdev->recovery_offset) < sector_nr) + WRITE_ONCE(rdev->recovery_offset, sector_nr); conf->reshape_checkpoint = jiffies; set_bit(MD_SB_CHANGE_DEVS, &mddev->sb_flags); md_wakeup_thread(mddev->thread); @@ -6802,12 +6805,14 @@ static void raid5d(struct md_thread *thread) if ( !list_empty(&conf->bitmap_list)) { /* Now is a good time to flush some bitmap updates */ - conf->seq_flush++; + int seq = conf->seq_flush + 1; + + WRITE_ONCE(conf->seq_flush, seq); spin_unlock_irq(&conf->device_lock); if (md_bitmap_enabled(mddev, true)) mddev->bitmap_ops->unplug(mddev, true); spin_lock_irq(&conf->device_lock); - conf->seq_write = conf->seq_flush; + conf->seq_write = seq; activate_bit_delay(conf, conf->temp_inactive_list); } raid5_activate_delayed(conf); @@ -6992,6 +6997,7 @@ raid5_store_stripe_size(struct mddev *mddev, const char *page, size_t len) { struct r5conf *conf; unsigned long new; + unsigned int noio_flags = 0; int err; int size; @@ -7032,6 +7038,7 @@ raid5_store_stripe_size(struct mddev *mddev, const char *page, size_t len) goto out_unlock; } + noio_flags = memalloc_noio_save(); mutex_lock(&conf->cache_size_mutex); size = conf->max_nr_stripes; @@ -7048,6 +7055,7 @@ raid5_store_stripe_size(struct mddev *mddev, const char *page, size_t len) mutex_unlock(&conf->cache_size_mutex); out_unlock: + memalloc_noio_restore(noio_flags); mddev_unlock_and_resume(mddev); return err ?: len; } @@ -7999,9 +8007,9 @@ static int raid5_run(struct mddev *mddev) /* Hack because v0.91 doesn't store recovery_offset properly. */ if (mddev->major_version == 0 && mddev->minor_version > 90) - rdev->recovery_offset = reshape_offset; + WRITE_ONCE(rdev->recovery_offset, reshape_offset); - if (rdev->recovery_offset < reshape_offset) { + if (READ_ONCE(rdev->recovery_offset) < reshape_offset) { /* We need to check old and new layout */ if (!only_parity(rdev->raid_disk, conf->algorithm, @@ -8152,10 +8160,10 @@ static int raid5_spare_active(struct mddev *mddev) for (i = 0; i < conf->raid_disks; i++) { rdev = conf->disks[i].rdev; replacement = conf->disks[i].replacement; - if (replacement - && replacement->recovery_offset == MaxSector - && !test_bit(Faulty, &replacement->flags) - && !test_and_set_bit(In_sync, &replacement->flags)) { + if (replacement && + READ_ONCE(replacement->recovery_offset) == MaxSector && + !test_bit(Faulty, &replacement->flags) && + !test_and_set_bit(In_sync, &replacement->flags)) { /* Replacement has just become active. */ if (!rdev || !test_and_clear_bit(In_sync, &rdev->flags)) @@ -8170,10 +8178,10 @@ static int raid5_spare_active(struct mddev *mddev) rdev->sysfs_state); } sysfs_notify_dirent_safe(replacement->sysfs_state); - } else if (rdev - && rdev->recovery_offset == MaxSector - && !test_bit(Faulty, &rdev->flags) - && !test_and_set_bit(In_sync, &rdev->flags)) { + } else if (rdev && + READ_ONCE(rdev->recovery_offset) == MaxSector && + !test_bit(Faulty, &rdev->flags) && + !test_and_set_bit(In_sync, &rdev->flags)) { count++; sysfs_notify_dirent_safe(rdev->sysfs_state); } @@ -8545,7 +8553,7 @@ static int raid5_start_reshape(struct mddev *mddev) >= conf->previous_raid_disks) set_bit(In_sync, &rdev->flags); else - rdev->recovery_offset = 0; + WRITE_ONCE(rdev->recovery_offset, 0); /* Failure here is OK */ sysfs_link_rdev(mddev, rdev); @@ -8597,7 +8605,7 @@ static void end_reshape(struct r5conf *conf) if (rdev->raid_disk >= 0 && !test_bit(Journal, &rdev->flags) && !test_bit(In_sync, &rdev->flags)) - rdev->recovery_offset = MaxSector; + WRITE_ONCE(rdev->recovery_offset, MaxSector); spin_unlock_irq(&conf->device_lock); wake_up(&conf->wait_for_reshape); @@ -8917,6 +8925,7 @@ static void *raid6_takeover(struct mddev *mddev) static int raid5_change_consistency_policy(struct mddev *mddev, const char *buf) { struct r5conf *conf; + unsigned int noio_flags; int err; err = mddev_suspend_and_lock(mddev); @@ -8928,6 +8937,7 @@ static int raid5_change_consistency_policy(struct mddev *mddev, const char *buf) return -ENODEV; } + noio_flags = memalloc_noio_save(); if (strncmp(buf, "ppl", 3) == 0) { /* ppl only works with RAID 5 */ if (!raid5_has_ppl(conf) && conf->level == 5) { @@ -8967,6 +8977,7 @@ static int raid5_change_consistency_policy(struct mddev *mddev, const char *buf) if (!err) md_update_sb(mddev, 1); + memalloc_noio_restore(noio_flags); mddev_unlock_and_resume(mddev); return err; diff --git a/drivers/media/cec/core/cec-pin.c b/drivers/media/cec/core/cec-pin.c index 4d7155281daa..754a4bfcc87c 100644 --- a/drivers/media/cec/core/cec-pin.c +++ b/drivers/media/cec/core/cec-pin.c @@ -115,7 +115,7 @@ static void cec_pin_update(struct cec_pin *pin, bool v, bool force) return; pin->adap->cec_pin_is_high = v; - if (atomic_read(&pin->work_pin_num_events) < CEC_NUM_PIN_EVENTS) { + if (atomic_read_acquire(&pin->work_pin_num_events) < CEC_NUM_PIN_EVENTS) { u8 ev = v; if (pin->work_pin_events_dropped) { @@ -126,7 +126,7 @@ static void cec_pin_update(struct cec_pin *pin, bool v, bool force) pin->work_pin_ts[pin->work_pin_events_wr] = ktime_get(); pin->work_pin_events_wr = (pin->work_pin_events_wr + 1) % CEC_NUM_PIN_EVENTS; - atomic_inc(&pin->work_pin_num_events); + atomic_inc_return_release(&pin->work_pin_num_events); } else { pin->work_pin_events_dropped = true; pin->work_pin_events_dropped_cnt++; @@ -1101,7 +1101,7 @@ static int cec_pin_thread_func(void *_adap) pin->work_tx_ts); } - while (atomic_read(&pin->work_pin_num_events)) { + while (atomic_read_acquire(&pin->work_pin_num_events)) { unsigned int idx = pin->work_pin_events_rd; u8 v = pin->work_pin_events[idx]; @@ -1110,7 +1110,7 @@ static int cec_pin_thread_func(void *_adap) v & CEC_PIN_EVENT_FL_DROPPED, pin->work_pin_ts[idx]); pin->work_pin_events_rd = (idx + 1) % CEC_NUM_PIN_EVENTS; - atomic_dec(&pin->work_pin_num_events); + atomic_dec_return_release(&pin->work_pin_num_events); } switch (atomic_xchg(&pin->work_irq_change, diff --git a/drivers/media/i2c/rdacm21.c b/drivers/media/i2c/rdacm21.c index bcab462708c7..ece8a410e7ce 100644 --- a/drivers/media/i2c/rdacm21.c +++ b/drivers/media/i2c/rdacm21.c @@ -588,10 +588,12 @@ static int rdacm21_probe(struct i2c_client *client) ret = v4l2_async_register_subdev(&dev->sd); if (ret) - goto error_free_ctrls; + goto error_entity_cleanup; return 0; +error_entity_cleanup: + media_entity_cleanup(&dev->sd.entity); error_free_ctrls: v4l2_ctrl_handler_free(&dev->ctrls); error: @@ -606,6 +608,7 @@ static void rdacm21_remove(struct i2c_client *client) v4l2_async_unregister_subdev(&dev->sd); v4l2_ctrl_handler_free(&dev->ctrls); + media_entity_cleanup(&dev->sd.entity); i2c_unregister_device(dev->isp); } diff --git a/drivers/media/pci/intel/ipu6/ipu6.c b/drivers/media/pci/intel/ipu6/ipu6.c index fab7783c664b..f900de14ba60 100644 --- a/drivers/media/pci/intel/ipu6/ipu6.c +++ b/drivers/media/pci/intel/ipu6/ipu6.c @@ -400,7 +400,6 @@ ipu6_isys_init(struct pci_dev *pdev, struct device *parent, &ipdata->hw_variant); if (IS_ERR(isys_adev->mmu)) { put_device(&isys_adev->auxdev.dev); - kfree(pdata); return dev_err_cast_probe(dev, isys_adev->mmu, "ipu6_mmu_init(isys_adev->mmu) failed\n"); } @@ -408,10 +407,8 @@ ipu6_isys_init(struct pci_dev *pdev, struct device *parent, isys_adev->mmu->dev = &isys_adev->auxdev.dev; ret = ipu6_bus_add_device(isys_adev); - if (ret) { - kfree(pdata); + if (ret) return ERR_PTR(ret); - } return isys_adev; } @@ -444,7 +441,6 @@ ipu6_psys_init(struct pci_dev *pdev, struct device *parent, &ipdata->hw_variant); if (IS_ERR(psys_adev->mmu)) { put_device(&psys_adev->auxdev.dev); - kfree(pdata); return dev_err_cast_probe(&pdev->dev, psys_adev->mmu, "ipu6_mmu_init(psys_adev->mmu) failed\n"); } @@ -452,10 +448,8 @@ ipu6_psys_init(struct pci_dev *pdev, struct device *parent, psys_adev->mmu->dev = &psys_adev->auxdev.dev; ret = ipu6_bus_add_device(psys_adev); - if (ret) { - kfree(pdata); + if (ret) return ERR_PTR(ret); - } return psys_adev; } diff --git a/drivers/media/platform/broadcom/bcm2835-unicam.c b/drivers/media/platform/broadcom/bcm2835-unicam.c index f10064107d54..fd862300e06f 100644 --- a/drivers/media/platform/broadcom/bcm2835-unicam.c +++ b/drivers/media/platform/broadcom/bcm2835-unicam.c @@ -2594,6 +2594,7 @@ static int unicam_async_nf_init(struct unicam_device *unicam) return 0; error: + v4l2_async_nf_cleanup(&unicam->notifier); fwnode_handle_put(ep_handle); return ret; } @@ -2726,6 +2727,7 @@ static void unicam_remove(struct platform_device *pdev) v4l2_device_unregister(&unicam->v4l2_dev); media_device_unregister(&unicam->mdev); v4l2_async_nf_unregister(&unicam->notifier); + v4l2_async_nf_cleanup(&unicam->notifier); unicam_subdev_cleanup(unicam); diff --git a/drivers/media/platform/qcom/iris/iris_core.c b/drivers/media/platform/qcom/iris/iris_core.c index 8406c48d635b..e337f8b7e6f0 100644 --- a/drivers/media/platform/qcom/iris/iris_core.c +++ b/drivers/media/platform/qcom/iris/iris_core.c @@ -12,18 +12,24 @@ void iris_core_deinit(struct iris_core *core) { - pm_runtime_resume_and_get(core->dev); + int ret; + + ret = pm_runtime_resume_and_get(core->dev); mutex_lock(&core->lock); if (core->state != IRIS_CORE_DEINIT) { iris_fw_unload(core); - iris_vpu_power_off(core); + + if (!ret) + iris_vpu_power_off(core); + iris_hfi_queues_deinit(core); core->state = IRIS_CORE_DEINIT; } mutex_unlock(&core->lock); - pm_runtime_put_sync(core->dev); + if (!ret) + pm_runtime_put_sync(core->dev); } static int iris_wait_for_system_response(struct iris_core *core) diff --git a/drivers/media/platform/qcom/iris/iris_state.c b/drivers/media/platform/qcom/iris/iris_state.c index e991f34916ec..5552725c614e 100644 --- a/drivers/media/platform/qcom/iris/iris_state.c +++ b/drivers/media/platform/qcom/iris/iris_state.c @@ -269,7 +269,7 @@ bool iris_allow_cmd(struct iris_inst *inst, u32 cmd) return true; } else if (cmd == V4L2_DEC_CMD_STOP || cmd == V4L2_ENC_CMD_STOP) { if (vb2_is_streaming(src_q)) - if (inst->sub_state != IRIS_INST_SUB_DRAIN) + if (!(inst->sub_state & IRIS_INST_SUB_DRAIN)) return true; } diff --git a/drivers/media/v4l2-core/v4l2-async.c b/drivers/media/v4l2-core/v4l2-async.c index 70284f50e1f9..5f6685e34d90 100644 --- a/drivers/media/v4l2-core/v4l2-async.c +++ b/drivers/media/v4l2-core/v4l2-async.c @@ -897,9 +897,18 @@ void v4l2_async_unregister_subdev(struct v4l2_subdev *sd) sd->subdev_notifier = NULL; if (sd->asc_list.next) { - list_for_each_entry_safe(asc, asc_tmp, &sd->asc_list, - asc_subdev_entry) { - v4l2_async_unbind_subdev_one(asc->notifier, asc); + if (list_empty(&sd->asc_list)) { + /* + * If the sub-device was registered through other means + * than v4l2-async, there are no async connections but + * the sub-device may still well be registered. + * Unregister it now. + */ + v4l2_device_unregister_subdev(sd); + } else { + list_for_each_entry_safe(asc, asc_tmp, &sd->asc_list, + asc_subdev_entry) + v4l2_async_unbind_subdev_one(asc->notifier, asc); } } diff --git a/drivers/mfd/iqs62x.c b/drivers/mfd/iqs62x.c index ee017617d1d1..412ae7777f72 100644 --- a/drivers/mfd/iqs62x.c +++ b/drivers/mfd/iqs62x.c @@ -237,7 +237,7 @@ static int iqs62x_firmware_parse(struct iqs62x_core *iqs62x, fw_rec = (struct iqs62x_fw_rec *)(fw->data + pos); pos += sizeof(*fw_rec); - if (pos + fw_rec->len - 1 > fw->size) { + if (!fw_rec->len || fw_rec->len - 1 > fw->size - pos) { ret = -EINVAL; break; } diff --git a/drivers/mfd/macsmc.c b/drivers/mfd/macsmc.c index 3228e79c86eb..90c434c9215b 100644 --- a/drivers/mfd/macsmc.c +++ b/drivers/mfd/macsmc.c @@ -406,7 +406,7 @@ static int apple_smc_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct apple_smc *smc; - u32 count; + __be32 count; int ret; smc = devm_kzalloc(dev, sizeof(*smc), GFP_KERNEL); @@ -457,8 +457,10 @@ static int apple_smc_probe(struct platform_device *pdev) dev_set_drvdata(&pdev->dev, smc); BLOCKING_INIT_NOTIFIER_HEAD(&smc->event_handlers); - ret = apple_smc_read_u32(smc, SMC_KEY(#KEY), &count); - if (ret) + ret = apple_smc_read(smc, SMC_KEY(#KEY), &count, sizeof(count)); + if (ret >= 0 && ret != sizeof(count)) + ret = -EINVAL; + if (ret < 0) return dev_err_probe(smc->dev, ret, "Failed to get key count"); smc->key_count = be32_to_cpu(count); diff --git a/drivers/mfd/rave-sp.c b/drivers/mfd/rave-sp.c index c1b78d127a26..05d26d92df03 100644 --- a/drivers/mfd/rave-sp.c +++ b/drivers/mfd/rave-sp.c @@ -63,6 +63,12 @@ #define RAVE_SP_TX_BUFFER_SIZE \ (RAVE_SP_STX_ETX_SIZE + 2 * RAVE_SP_RX_BUFFER_SIZE) +enum rave_sp_frame_offset { + RAVE_SP_FRAME_CODE_OFFSET, + RAVE_SP_FRAME_ACK_ID_OFFSET, + RAVE_SP_FRAME_DATA_OFFSET, +}; + /** * enum rave_sp_deframer_state - Possible state for de-framer * @@ -352,7 +358,7 @@ int rave_sp_exec(struct rave_sp *sp, int command, ret = 0; u8 ackid; - command = sp->variant->cmd.translate(data[0]); + command = sp->variant->cmd.translate(data[RAVE_SP_FRAME_CODE_OFFSET]); if (command < 0) return command; @@ -366,8 +372,8 @@ int rave_sp_exec(struct rave_sp *sp, sp->reply = &reply; mutex_unlock(&sp->reply_lock); - data[0] = command; - data[1] = ackid; + data[RAVE_SP_FRAME_CODE_OFFSET] = command; + data[RAVE_SP_FRAME_ACK_ID_OFFSET] = ackid; rave_sp_write(sp, data, data_size); @@ -388,16 +394,23 @@ EXPORT_SYMBOL_GPL(rave_sp_exec); static void rave_sp_receive_event(struct rave_sp *sp, const unsigned char *data, size_t length) { - u8 cmd[] = { - [0] = rave_sp_reply_code(data[0]), - [1] = data[1], - }; + unsigned long action; + u8 cmd[RAVE_SP_FRAME_DATA_OFFSET]; + + if (length < RAVE_SP_FRAME_DATA_OFFSET + 1) { + dev_warn(&sp->serdev->dev, "Dropping short event frame\n"); + return; + } + + cmd[RAVE_SP_FRAME_CODE_OFFSET] = + rave_sp_reply_code(data[RAVE_SP_FRAME_CODE_OFFSET]); + cmd[RAVE_SP_FRAME_ACK_ID_OFFSET] = data[RAVE_SP_FRAME_ACK_ID_OFFSET]; rave_sp_write(sp, cmd, sizeof(cmd)); - blocking_notifier_call_chain(&sp->event_notifier_list, - rave_sp_action_pack(data[0], data[2]), - NULL); + action = rave_sp_action_pack(data[RAVE_SP_FRAME_CODE_OFFSET], + data[RAVE_SP_FRAME_DATA_OFFSET]); + blocking_notifier_call_chain(&sp->event_notifier_list, action, NULL); } static void rave_sp_receive_reply(struct rave_sp *sp, @@ -405,27 +418,35 @@ static void rave_sp_receive_reply(struct rave_sp *sp, { struct device *dev = &sp->serdev->dev; struct rave_sp_reply *reply; - const size_t payload_length = length - 2; + size_t payload_length; + + if (length < RAVE_SP_FRAME_DATA_OFFSET) { + dev_warn(dev, "Dropping short reply frame\n"); + return; + } + payload_length = length - RAVE_SP_FRAME_DATA_OFFSET; mutex_lock(&sp->reply_lock); reply = sp->reply; if (reply) { - if (reply->code == data[0] && reply->ackid == data[1] && + if (reply->code == data[RAVE_SP_FRAME_CODE_OFFSET] && + reply->ackid == data[RAVE_SP_FRAME_ACK_ID_OFFSET] && payload_length >= reply->length) { /* * We are relying on memcpy(dst, src, 0) to be a no-op * when handling commands that have a no-payload reply */ - memcpy(reply->data, &data[2], reply->length); + memcpy(reply->data, &data[RAVE_SP_FRAME_DATA_OFFSET], + reply->length); complete(&reply->received); sp->reply = NULL; } else { dev_err(dev, "Ignoring incorrect reply\n"); dev_dbg(dev, "Code: expected = 0x%08x received = 0x%08x\n", - reply->code, data[0]); + reply->code, data[RAVE_SP_FRAME_CODE_OFFSET]); dev_dbg(dev, "ACK ID: expected = 0x%08x received = 0x%08x\n", - reply->ackid, data[1]); + reply->ackid, data[RAVE_SP_FRAME_ACK_ID_OFFSET]); dev_dbg(dev, "Length: expected = %zu received = %zu\n", reply->length, payload_length); } @@ -439,10 +460,10 @@ static void rave_sp_receive_frame(struct rave_sp *sp, size_t length) { const size_t checksum_length = sp->variant->checksum->length; - const size_t payload_length = length - checksum_length; - const u8 *crc_reported = &data[payload_length]; struct device *dev = &sp->serdev->dev; u8 crc_calculated[RAVE_SP_CHECKSUM_SIZE]; + const u8 *crc_reported; + size_t payload_length; if (unlikely(checksum_length > sizeof(crc_calculated))) { dev_warn(dev, "Checksum too long, dropping\n"); @@ -457,6 +478,9 @@ static void rave_sp_receive_frame(struct rave_sp *sp, return; } + payload_length = length - checksum_length; + crc_reported = &data[payload_length]; + sp->variant->checksum->subroutine(data, payload_length, crc_calculated); @@ -465,10 +489,10 @@ static void rave_sp_receive_frame(struct rave_sp *sp, return; } - if (rave_sp_id_is_event(data[0])) - rave_sp_receive_event(sp, data, length); + if (rave_sp_id_is_event(data[RAVE_SP_FRAME_CODE_OFFSET])) + rave_sp_receive_event(sp, data, payload_length); else - rave_sp_receive_reply(sp, data, length); + rave_sp_receive_reply(sp, data, payload_length); } static size_t rave_sp_receive_buf(struct serdev_device *serdev, diff --git a/drivers/misc/ad525x_dpot-i2c.c b/drivers/misc/ad525x_dpot-i2c.c index 469478f7a1d3..896ad61bb9e1 100644 --- a/drivers/misc/ad525x_dpot-i2c.c +++ b/drivers/misc/ad525x_dpot-i2c.c @@ -105,6 +105,7 @@ MODULE_DEVICE_TABLE(i2c, ad_dpot_id); static struct i2c_driver ad_dpot_i2c_driver = { .driver = { .name = "ad_dpot", + .dev_groups = ad_dpot_groups, }, .probe = ad_dpot_i2c_probe, .remove = ad_dpot_i2c_remove, diff --git a/drivers/misc/ad525x_dpot-spi.c b/drivers/misc/ad525x_dpot-spi.c index 263055bda48b..1ebe629715a8 100644 --- a/drivers/misc/ad525x_dpot-spi.c +++ b/drivers/misc/ad525x_dpot-spi.c @@ -131,6 +131,7 @@ MODULE_DEVICE_TABLE(spi, ad_dpot_spi_id); static struct spi_driver ad_dpot_spi_driver = { .driver = { .name = "ad_dpot", + .dev_groups = ad_dpot_groups, }, .probe = ad_dpot_spi_probe, .remove = ad_dpot_spi_remove, diff --git a/drivers/misc/ad525x_dpot.c b/drivers/misc/ad525x_dpot.c index 04683b981e54..3b55b3f6c7f4 100644 --- a/drivers/misc/ad525x_dpot.c +++ b/drivers/misc/ad525x_dpot.c @@ -630,66 +630,132 @@ static struct attribute *ad525x_attributes_commands[] = { NULL }; -static const struct attribute_group ad525x_group_commands = { - .attrs = ad525x_attributes_commands, +static struct attribute *ad525x_attributes[] = { + &dev_attr_rdac0.attr, + &dev_attr_rdac1.attr, + &dev_attr_rdac2.attr, + &dev_attr_rdac3.attr, + &dev_attr_rdac4.attr, + &dev_attr_rdac5.attr, + &dev_attr_eeprom0.attr, + &dev_attr_eeprom1.attr, + &dev_attr_eeprom2.attr, + &dev_attr_eeprom3.attr, + &dev_attr_eeprom4.attr, + &dev_attr_eeprom5.attr, + &dev_attr_tolerance0.attr, + &dev_attr_tolerance1.attr, + &dev_attr_tolerance2.attr, + &dev_attr_tolerance3.attr, + &dev_attr_tolerance4.attr, + &dev_attr_tolerance5.attr, + &dev_attr_otp0.attr, + &dev_attr_otp1.attr, + &dev_attr_otp2.attr, + &dev_attr_otp3.attr, + &dev_attr_otp4.attr, + &dev_attr_otp5.attr, + &dev_attr_otp0en.attr, + &dev_attr_otp1en.attr, + &dev_attr_otp2en.attr, + &dev_attr_otp3en.attr, + &dev_attr_otp4en.attr, + &dev_attr_otp5en.attr, + &dev_attr_inc_all.attr, + &dev_attr_dec_all.attr, + &dev_attr_inc_all_6db.attr, + &dev_attr_dec_all_6db.attr, + NULL }; -static int ad_dpot_add_files(struct device *dev, - unsigned int features, unsigned int rdac) +static int ad525x_attr_index(struct attribute *attr, + const struct attribute * const *attrs) { - int err = sysfs_create_file(&dev->kobj, - dpot_attrib_wipers[rdac]); - if (features & F_CMD_EEP) - err |= sysfs_create_file(&dev->kobj, - dpot_attrib_eeprom[rdac]); - if (features & F_CMD_TOL) - err |= sysfs_create_file(&dev->kobj, - dpot_attrib_tolerance[rdac]); - if (features & F_CMD_OTP) { - err |= sysfs_create_file(&dev->kobj, - dpot_attrib_otp_en[rdac]); - err |= sysfs_create_file(&dev->kobj, - dpot_attrib_otp[rdac]); - } + int i; - if (err) - dev_err(dev, "failed to register sysfs hooks for RDAC%d\n", - rdac); + for (i = 0; attrs[i]; i++) + if (attr == attrs[i]) + return i; - return err; + return -ENOENT; } -static inline void ad_dpot_remove_files(struct device *dev, - unsigned int features, unsigned int rdac) +static bool ad525x_is_command_attr(struct attribute *attr) { - sysfs_remove_file(&dev->kobj, - dpot_attrib_wipers[rdac]); - if (features & F_CMD_EEP) - sysfs_remove_file(&dev->kobj, - dpot_attrib_eeprom[rdac]); - if (features & F_CMD_TOL) - sysfs_remove_file(&dev->kobj, - dpot_attrib_tolerance[rdac]); - if (features & F_CMD_OTP) { - sysfs_remove_file(&dev->kobj, - dpot_attrib_otp_en[rdac]); - sysfs_remove_file(&dev->kobj, - dpot_attrib_otp[rdac]); + int i; + + for (i = 0; ad525x_attributes_commands[i]; i++) { + if (attr == ad525x_attributes_commands[i]) + return true; } + + return false; +} + +static umode_t ad525x_is_visible(struct kobject *kobj, struct attribute *attr, + int n) +{ + struct device *dev = kobj_to_dev(kobj); + struct dpot_data *data = dev_get_drvdata(dev); + int rdac; + + if (!data) + return 0; + + rdac = ad525x_attr_index(attr, dpot_attrib_wipers); + if (rdac >= 0) + return data->wipers & BIT(rdac) ? attr->mode : 0; + + rdac = ad525x_attr_index(attr, dpot_attrib_eeprom); + if (rdac >= 0) + return (data->wipers & BIT(rdac)) && (data->feat & F_CMD_EEP) ? + attr->mode : 0; + + rdac = ad525x_attr_index(attr, dpot_attrib_tolerance); + if (rdac >= 0) + return (data->wipers & BIT(rdac)) && (data->feat & F_CMD_TOL) ? + attr->mode : 0; + + rdac = ad525x_attr_index(attr, dpot_attrib_otp); + if (rdac >= 0) + return (data->wipers & BIT(rdac)) && (data->feat & F_CMD_OTP) ? + attr->mode : 0; + + rdac = ad525x_attr_index(attr, dpot_attrib_otp_en); + if (rdac >= 0) + return (data->wipers & BIT(rdac)) && (data->feat & F_CMD_OTP) ? + attr->mode : 0; + + if (ad525x_is_command_attr(attr)) + return data->feat & F_CMD_INC ? attr->mode : 0; + + return attr->mode; } +static const struct attribute_group ad525x_group = { + .attrs = ad525x_attributes, + .is_visible = ad525x_is_visible, +}; + +const struct attribute_group *ad_dpot_groups[] = { + &ad525x_group, + NULL +}; +EXPORT_SYMBOL(ad_dpot_groups); + int ad_dpot_probe(struct device *dev, struct ad_dpot_bus_data *bdata, unsigned long devid, const char *name) { struct dpot_data *data; - int i, err = 0; + int i; data = kzalloc(sizeof(struct dpot_data), GFP_KERNEL); if (!data) { - err = -ENOMEM; - goto exit; + dev_err(dev, "failed to create client for %s ID 0x%lX\n", + name, devid); + return -ENOMEM; } dev_set_drvdata(dev, data); @@ -705,51 +771,22 @@ int ad_dpot_probe(struct device *dev, data->wipers = DPOT_WIPERS(devid); for (i = DPOT_RDAC0; i < MAX_RDACS; i++) - if (data->wipers & (1 << i)) { - err = ad_dpot_add_files(dev, data->feat, i); - if (err) - goto exit_remove_files; + if (data->wipers & BIT(i)) { /* power-up midscale */ if (data->feat & F_RDACS_WONLY) data->rdac_cache[i] = data->max_pos / 2; } - if (data->feat & F_CMD_INC) - err = sysfs_create_group(&dev->kobj, &ad525x_group_commands); - - if (err) { - dev_err(dev, "failed to register sysfs hooks\n"); - goto exit_free; - } - dev_info(dev, "%s %d-Position Digital Potentiometer registered\n", name, data->max_pos); return 0; - -exit_remove_files: - for (i = DPOT_RDAC0; i < MAX_RDACS; i++) - if (data->wipers & (1 << i)) - ad_dpot_remove_files(dev, data->feat, i); - -exit_free: - kfree(data); - dev_set_drvdata(dev, NULL); -exit: - dev_err(dev, "failed to create client for %s ID 0x%lX\n", - name, devid); - return err; } EXPORT_SYMBOL(ad_dpot_probe); void ad_dpot_remove(struct device *dev) { struct dpot_data *data = dev_get_drvdata(dev); - int i; - - for (i = DPOT_RDAC0; i < MAX_RDACS; i++) - if (data->wipers & (1 << i)) - ad_dpot_remove_files(dev, data->feat, i); kfree(data); } diff --git a/drivers/misc/ad525x_dpot.h b/drivers/misc/ad525x_dpot.h index 72a9d6801937..2e877c89523b 100644 --- a/drivers/misc/ad525x_dpot.h +++ b/drivers/misc/ad525x_dpot.h @@ -10,6 +10,8 @@ #include +struct attribute_group; + #define DPOT_CONF(features, wipers, max_pos, uid) \ (((features) << 18) | (((wipers) & 0xFF) << 10) | \ ((max_pos & 0xF) << 6) | (uid & 0x3F)) @@ -210,5 +212,6 @@ struct ad_dpot_bus_data { int ad_dpot_probe(struct device *dev, struct ad_dpot_bus_data *bdata, unsigned long devid, const char *name); void ad_dpot_remove(struct device *dev); +extern const struct attribute_group *ad_dpot_groups[]; #endif diff --git a/drivers/misc/bcm-vk/bcm_vk_msg.c b/drivers/misc/bcm-vk/bcm_vk_msg.c index 665a3888708a..a36a6c7d384a 100644 --- a/drivers/misc/bcm-vk/bcm_vk_msg.c +++ b/drivers/misc/bcm-vk/bcm_vk_msg.c @@ -108,7 +108,8 @@ u32 msgq_avail_space(const struct bcm_vk_msgq __iomem *msgq, bool bcm_vk_drv_access_ok(struct bcm_vk *vk) { - return (!!atomic_read(&vk->msgq_inited)); + /* Pair with the release store after message queue initialization. */ + return !!atomic_read_acquire(&vk->msgq_inited); } void bcm_vk_set_host_alert(struct bcm_vk *vk, u32 bit_mask) @@ -501,7 +502,8 @@ int bcm_vk_sync_msgq(struct bcm_vk *vk, bool force_sync) msgq++; } } - atomic_set(&vk->msgq_inited, 1); + /* Publish message queue info before allowing driver access. */ + atomic_set_release(&vk->msgq_inited, 1); return ret; } diff --git a/drivers/misc/cardreader/rtsx_pcr.c b/drivers/misc/cardreader/rtsx_pcr.c index f9952d76d6ed..84a49156a247 100644 --- a/drivers/misc/cardreader/rtsx_pcr.c +++ b/drivers/misc/cardreader/rtsx_pcr.c @@ -1196,6 +1196,8 @@ static int rtsx_pci_init_hw(struct rtsx_pcr *pcr) /* Gating real mcu clock */ err = rtsx_pci_write_register(pcr, RTS5261_FW_CFG1, RTS5261_MCU_CLOCK_GATING, 0); + if (err < 0) + return err; err = rtsx_pci_write_register(pcr, RTS5261_REG_FPDCTL, SSC_POWER_DOWN, 0); } else { diff --git a/drivers/misc/cardreader/rtsx_usb.c b/drivers/misc/cardreader/rtsx_usb.c index 1830e9ed2521..a127744918f4 100644 --- a/drivers/misc/cardreader/rtsx_usb.c +++ b/drivers/misc/cardreader/rtsx_usb.c @@ -312,6 +312,9 @@ int rtsx_usb_get_card_status(struct rtsx_ucr *ucr, u16 *status) if (ret < 0) return ret; + ucr->card_status_cache = *status; + ucr->card_status_valid = true; + return 0; } EXPORT_SYMBOL_GPL(rtsx_usb_get_card_status); @@ -623,6 +626,7 @@ static int rtsx_usb_probe(struct usb_interface *intf, { struct usb_device *usb_dev = interface_to_usbdev(intf); struct rtsx_ucr *ucr; + u16 status; int ret; dev_dbg(&intf->dev, @@ -659,6 +663,9 @@ static int rtsx_usb_probe(struct usb_interface *intf, if (ret) goto out_init_fail; + /* Prime cached status for runtime autosuspend decisions. */ + rtsx_usb_get_card_status(ucr, &status); + /* initialize USB SG transfer timer */ timer_setup(&ucr->sg_timer, rtsx_usb_sg_timed_out, 0); @@ -713,22 +720,29 @@ static int rtsx_usb_suspend(struct usb_interface *intf, pm_message_t message) struct rtsx_ucr *ucr = (struct rtsx_ucr *)usb_get_intfdata(intf); u16 val = 0; + bool valid = false; dev_dbg(&intf->dev, "%s called with pm message 0x%04x\n", __func__, message.event); if (PMSG_IS_AUTO(message)) { if (mutex_trylock(&ucr->dev_mutex)) { - rtsx_usb_get_card_status(ucr, &val); + valid = ucr->card_status_valid; + if (valid) + val = ucr->card_status_cache; mutex_unlock(&ucr->dev_mutex); - /* Defer the autosuspend if card exists */ - if (val & (SD_CD | MS_CD)) { + /* + * Do not issue USB commands from runtime autosuspend. + * Raw SD_CD is not authoritative on tray-based readers, + * while a real SD card is protected by the SD/MMC child + * runtime-PM reference once the card is powered. Keep + * the historical Memory Stick autosuspend deferral when + * the cached status says MS media is present. + */ + if (valid && (val & MS_CD)) { device_for_each_child(&intf->dev, NULL, rtsx_usb_resume_child); return -EAGAIN; - } else { - /* if the card does not exists, clear OCP status */ - rtsx_usb_write_register(ucr, OCPCTL, MS_OCP_CLEAR, MS_OCP_CLEAR); } } else { /* There is an ongoing operation*/ diff --git a/drivers/misc/fastrpc.c b/drivers/misc/fastrpc.c index e240b9464480..29f9003999a0 100644 --- a/drivers/misc/fastrpc.c +++ b/drivers/misc/fastrpc.c @@ -1681,24 +1681,20 @@ static int fastrpc_dmabuf_alloc(struct fastrpc_user *fl, char __user *argp) return err; } - bp.fd = dma_buf_fd(buf->dmabuf, O_ACCMODE); + bp.fd = get_unused_fd_flags(O_ACCMODE); if (bp.fd < 0) { dma_buf_put(buf->dmabuf); - return -EINVAL; + return bp.fd; } if (copy_to_user(argp, &bp, sizeof(bp))) { - /* - * The usercopy failed, but we can't do much about it, as - * dma_buf_fd() already called fd_install() and made the - * file descriptor accessible for the current process. It - * might already be closed and dmabuf no longer valid when - * we reach this point. Therefore "leak" the fd and rely on - * the process exit path to do any required cleanup. - */ + put_unused_fd(bp.fd); + dma_buf_put(buf->dmabuf); return -EFAULT; } + dma_buf_fd_install(buf->dmabuf, bp.fd); + return 0; } diff --git a/drivers/misc/lan966x_pci.c b/drivers/misc/lan966x_pci.c index 9c79b58137e5..f6e358389af7 100644 --- a/drivers/misc/lan966x_pci.c +++ b/drivers/misc/lan966x_pci.c @@ -183,6 +183,7 @@ static int lan966x_pci_probe(struct pci_dev *pdev, const struct pci_device_id *i return 0; err_unload_overlay: + of_platform_depopulate(dev); lan966x_pci_unload_overlay(data); return ret; } diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c index 1c0fd185114f..fdce31f9fa43 100644 --- a/drivers/misc/pci_endpoint_test.c +++ b/drivers/misc/pci_endpoint_test.c @@ -859,6 +859,7 @@ static int pci_endpoint_test_doorbell(struct pci_endpoint_test *test) struct pci_dev *pdev = test->pdev; struct device *dev = &pdev->dev; int irq_type = test->irq_type; + int ret = 0; enum pci_barno bar; u32 data, status; u32 addr; @@ -900,8 +901,11 @@ static int pci_endpoint_test_doorbell(struct pci_endpoint_test *test) status = pci_endpoint_test_readl(test, PCI_ENDPOINT_TEST_STATUS); - if (!left || !(status & STATUS_DOORBELL_SUCCESS)) + if (!left || !(status & STATUS_DOORBELL_SUCCESS)) { dev_err(dev, "Failed to trigger doorbell in endpoint\n"); + /* Store error code, but continue to disable doorbell. */ + ret = -EINVAL; + } pci_endpoint_test_writel(test, PCI_ENDPOINT_TEST_COMMAND, COMMAND_DISABLE_DOORBELL); @@ -915,10 +919,7 @@ static int pci_endpoint_test_doorbell(struct pci_endpoint_test *test) return -EINVAL; } - if (!(status & STATUS_DOORBELL_SUCCESS)) - return -EINVAL; - - return 0; + return ret; } static long pci_endpoint_test_ioctl(struct file *file, unsigned int cmd, diff --git a/drivers/misc/sgi-gru/grufault.c b/drivers/misc/sgi-gru/grufault.c index 3557d78ee47a..5a87c12f444a 100644 --- a/drivers/misc/sgi-gru/grufault.c +++ b/drivers/misc/sgi-gru/grufault.c @@ -166,13 +166,8 @@ static void get_clear_fault_map(struct gru_state *gru, } /* - * Atomic (interrupt context) & non-atomic (user context) functions to - * convert a vaddr into a physical address. The size of the page - * is returned in pageshift. - * returns: - * 0 - successful - * < 0 - error code - * 1 - (atomic only) try again in non-atomic context + * Convert a user virtual address to a physical address in process context. + * The size of the page is returned in pageshift. */ static int non_atomic_pte_lookup(struct vm_area_struct *vma, unsigned long vaddr, int write, @@ -192,87 +187,25 @@ static int non_atomic_pte_lookup(struct vm_area_struct *vma, return 0; } -/* - * atomic_pte_lookup - * - * Convert a user virtual address to a physical address - * Only supports Intel large pages (2MB only) on x86_64. - * ZZZ - hugepage support is incomplete - * - * NOTE: mmap_lock is already held on entry to this function. This - * guarantees existence of the page tables. - */ -static int atomic_pte_lookup(struct vm_area_struct *vma, unsigned long vaddr, - int write, unsigned long *paddr, int *pageshift) -{ - pgd_t *pgdp; - p4d_t *p4dp; - pud_t *pudp; - pmd_t *pmdp; - pte_t pte; - - pgdp = pgd_offset(vma->vm_mm, vaddr); - if (unlikely(pgd_none(*pgdp))) - goto err; - - p4dp = p4d_offset(pgdp, vaddr); - if (unlikely(p4d_none(*p4dp))) - goto err; - - pudp = pud_offset(p4dp, vaddr); - if (unlikely(pud_none(*pudp))) - goto err; - - pmdp = pmd_offset(pudp, vaddr); - if (unlikely(pmd_none(*pmdp))) - goto err; -#ifdef CONFIG_X86_64 - if (unlikely(pmd_leaf(*pmdp))) - pte = ptep_get((pte_t *)pmdp); - else -#endif - pte = *pte_offset_kernel(pmdp, vaddr); - - if (unlikely(!pte_present(pte) || - (write && (!pte_write(pte) || !pte_dirty(pte))))) - return 1; - - *paddr = pte_pfn(pte) << PAGE_SHIFT; -#ifdef CONFIG_HUGETLB_PAGE - *pageshift = is_vm_hugetlb_page(vma) ? HPAGE_SHIFT : PAGE_SHIFT; -#else - *pageshift = PAGE_SHIFT; -#endif - return 0; - -err: - return 1; -} - static int gru_vtop(struct gru_thread_state *gts, unsigned long vaddr, int write, int atomic, unsigned long *gpa, int *pageshift) { struct mm_struct *mm = gts->ts_mm; struct vm_area_struct *vma; unsigned long paddr; - int ret, ps; + int ps; vma = find_vma(mm, vaddr); if (!vma) goto inval; - /* - * Atomic lookup is faster & usually works even if called in non-atomic - * context. - */ - rmb(); /* Must/check ms_range_active before loading PTEs */ - ret = atomic_pte_lookup(vma, vaddr, write, &paddr, &ps); - if (ret) { - if (atomic) - goto upm; - if (non_atomic_pte_lookup(vma, vaddr, write, &paddr, &ps)) - goto inval; - } + if (atomic) + goto upm; + + /* Order the caller's ms_range_active check before loading PTEs. */ + rmb(); + if (non_atomic_pte_lookup(vma, vaddr, write, &paddr, &ps)) + goto inval; if (is_gru_paddr(paddr)) goto inval; paddr = paddr & ~((1UL << ps) - 1); @@ -569,19 +502,9 @@ static irqreturn_t gru_intr(int chiplet, int blade) continue; } - /* - * This is running in interrupt context. Trylock the mmap_lock. - * If it fails, retry the fault in user context. - */ + /* Address translation may sleep, so retry the fault in user context. */ gts->ustats.fmm_tlbmiss++; - if (!gts->ts_force_cch_reload && - mmap_read_trylock(gts->ts_mm)) { - gru_try_dropin(gru, gts, tfh, NULL); - mmap_read_unlock(gts->ts_mm); - } else { - tfh_user_polling_mode(tfh); - STAT(intr_mm_lock_failed); - } + tfh_user_polling_mode(tfh); } return IRQ_HANDLED; } diff --git a/drivers/misc/sgi-gru/gruprocfs.c b/drivers/misc/sgi-gru/gruprocfs.c index 97b8b38ab47d..b8139c27bc7f 100644 --- a/drivers/misc/sgi-gru/gruprocfs.c +++ b/drivers/misc/sgi-gru/gruprocfs.c @@ -54,7 +54,6 @@ static int statistics_show(struct seq_file *s, void *p) printstat(s, intr_cbr); printstat(s, intr_tfh); printstat(s, intr_spurious); - printstat(s, intr_mm_lock_failed); printstat(s, call_os); printstat(s, call_os_wait_queue); printstat(s, user_flush_tlb); diff --git a/drivers/misc/sgi-gru/grutables.h b/drivers/misc/sgi-gru/grutables.h index 640daf1994df..3348552925c6 100644 --- a/drivers/misc/sgi-gru/grutables.h +++ b/drivers/misc/sgi-gru/grutables.h @@ -182,7 +182,6 @@ struct gru_stats_s { atomic_long_t intr_cbr; atomic_long_t intr_tfh; atomic_long_t intr_spurious; - atomic_long_t intr_mm_lock_failed; atomic_long_t call_os; atomic_long_t call_os_wait_queue; atomic_long_t user_flush_tlb; diff --git a/drivers/misc/vmw_vmci/vmci_event.c b/drivers/misc/vmw_vmci/vmci_event.c index 9a41ab65378d..89ab9f05b88c 100644 --- a/drivers/misc/vmw_vmci/vmci_event.c +++ b/drivers/misc/vmw_vmci/vmci_event.c @@ -179,16 +179,16 @@ int vmci_event_subscribe(u32 event, } } + *new_subscription_id = sub->id; if (have_new_id) { list_add_rcu(&sub->node, &subscriber_array[event]); retval = VMCI_SUCCESS; } else { + kfree(sub); retval = VMCI_ERROR_NO_RESOURCES; } mutex_unlock(&subscriber_mutex); - - *new_subscription_id = sub->id; return retval; } EXPORT_SYMBOL_GPL(vmci_event_subscribe); diff --git a/drivers/mtd/devices/mtd_intel_dg.c b/drivers/mtd/devices/mtd_intel_dg.c index 114e69135b8d..c9855d6f1f95 100644 --- a/drivers/mtd/devices/mtd_intel_dg.c +++ b/drivers/mtd/devices/mtd_intel_dg.c @@ -15,14 +15,18 @@ #include #include #include +#include #include #include #include #include +#define INTEL_DG_NVM_RPM_TIMEOUT_MS 500 + struct intel_dg_nvm { struct kref refcnt; struct mtd_info mtd; + struct device *dev; struct mutex lock; /* region access lock */ void __iomem *base; void __iomem *base2; @@ -421,6 +425,8 @@ static int intel_dg_nvm_init(struct intel_dg_nvm *nvm, struct device *device, unsigned int i, n; int ret; + nvm->dev = device; + /* clean error register, previous errors are ignored */ idg_nvm_error(nvm); @@ -498,6 +504,7 @@ static int intel_dg_mtd_erase(struct mtd_info *mtd, struct erase_info *info) size_t len; u8 region; u64 addr; + int ret; if (WARN_ON(!nvm)) return -EINVAL; @@ -512,20 +519,29 @@ static int intel_dg_mtd_erase(struct mtd_info *mtd, struct erase_info *info) total_len = info->len; addr = info->addr; + ret = pm_runtime_resume_and_get(nvm->dev); + if (ret < 0) { + dev_err(&mtd->dev, "rpm: get failed %d\n", ret); + return ret; + } + + ret = 0; guard(mutex)(&nvm->lock); while (total_len > 0) { if (!IS_ALIGNED(addr, SZ_4K) || !IS_ALIGNED(total_len, SZ_4K)) { dev_err(&mtd->dev, "unaligned erase %llx %zx\n", addr, total_len); info->fail_addr = addr; - return -ERANGE; + ret = -ERANGE; + break; } idx = idg_nvm_get_region(nvm, addr); if (idx >= nvm->nregions) { dev_err(&mtd->dev, "out of range"); info->fail_addr = MTD_FAIL_ADDR_UNKNOWN; - return -ERANGE; + ret = -ERANGE; + break; } from = addr - nvm->regions[idx].offset; @@ -541,14 +557,16 @@ static int intel_dg_mtd_erase(struct mtd_info *mtd, struct erase_info *info) if (bytes < 0) { dev_dbg(&mtd->dev, "erase failed with %zd\n", bytes); info->fail_addr += nvm->regions[idx].offset; - return bytes; + ret = bytes; + break; } addr += len; total_len -= len; } - return 0; + pm_runtime_put_autosuspend(nvm->dev); + return ret; } static int intel_dg_mtd_read(struct mtd_info *mtd, loff_t from, size_t len, @@ -577,17 +595,24 @@ static int intel_dg_mtd_read(struct mtd_info *mtd, loff_t from, size_t len, if (len > nvm->regions[idx].size - from) len = nvm->regions[idx].size - from; + ret = pm_runtime_resume_and_get(nvm->dev); + if (ret < 0) { + dev_err(&mtd->dev, "rpm: get failed %zd\n", ret); + return ret; + } + guard(mutex)(&nvm->lock); ret = idg_read(nvm, region, from, len, buf); if (ret < 0) { dev_dbg(&mtd->dev, "read failed with %zd\n", ret); - return ret; + } else { + *retlen = ret; + ret = 0; } - *retlen = ret; - - return 0; + pm_runtime_put_autosuspend(nvm->dev); + return ret; } static int intel_dg_mtd_write(struct mtd_info *mtd, loff_t to, size_t len, @@ -616,17 +641,24 @@ static int intel_dg_mtd_write(struct mtd_info *mtd, loff_t to, size_t len, if (len > nvm->regions[idx].size - to) len = nvm->regions[idx].size - to; + ret = pm_runtime_resume_and_get(nvm->dev); + if (ret < 0) { + dev_err(&mtd->dev, "rpm: get failed %zd\n", ret); + return ret; + } + guard(mutex)(&nvm->lock); ret = idg_write(nvm, region, to, len, buf); if (ret < 0) { dev_dbg(&mtd->dev, "write failed with %zd\n", ret); - return ret; + } else { + *retlen = ret; + ret = 0; } - *retlen = ret; - - return 0; + pm_runtime_put_autosuspend(nvm->dev); + return ret; } static void intel_dg_nvm_release(struct kref *kref) @@ -748,7 +780,7 @@ static int intel_dg_mtd_probe(struct auxiliary_device *aux_dev, dev_name(&aux_dev->dev), invm->regions[i].name); if (!name) { ret = -ENOMEM; - goto err; + goto err_norpm; } nvm->regions[n].name = name; @@ -756,6 +788,21 @@ static int intel_dg_mtd_probe(struct auxiliary_device *aux_dev, n++; } + ret = devm_pm_runtime_enable(device); + if (ret < 0) { + dev_err(device, "rpm: enable failed %d\n", ret); + goto err_norpm; + } + + pm_runtime_set_autosuspend_delay(device, INTEL_DG_NVM_RPM_TIMEOUT_MS); + pm_runtime_use_autosuspend(device); + + ret = pm_runtime_resume_and_get(device); + if (ret < 0) { + dev_err(device, "rpm: get failed %d\n", ret); + goto err_norpm; + } + nvm->base = devm_ioremap_resource(device, &invm->bar); if (IS_ERR(nvm->base)) { ret = PTR_ERR(nvm->base); @@ -784,9 +831,12 @@ static int intel_dg_mtd_probe(struct auxiliary_device *aux_dev, dev_set_drvdata(&aux_dev->dev, nvm); + pm_runtime_put(device); return 0; err: + pm_runtime_put(device); +err_norpm: kref_put(&nvm->refcnt, intel_dg_nvm_release); return ret; } diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c index 2876501a7814..fb92b7a0ed5f 100644 --- a/drivers/mtd/mtdpart.c +++ b/drivers/mtd/mtdpart.c @@ -254,7 +254,8 @@ int mtd_add_partition(struct mtd_info *parent, const char *name, /* the direct offset is expected */ if (offset == MTDPART_OFS_APPEND || - offset == MTDPART_OFS_NXTBLK) + offset == MTDPART_OFS_NXTBLK || + offset == MTDPART_OFS_RETAIN) return -EINVAL; if (length == MTDPART_SIZ_FULL) diff --git a/drivers/mtd/mtdswap.c b/drivers/mtd/mtdswap.c index 42ff7deace80..d38528210350 100644 --- a/drivers/mtd/mtdswap.c +++ b/drivers/mtd/mtdswap.c @@ -1452,6 +1452,7 @@ static void mtdswap_add_mtd(struct mtd_blktrans_ops *tr, struct mtd_info *mtd) debugfs_failed: del_mtd_blktrans_dev(mbd_dev); + mbd_dev = NULL; cleanup: mtdswap_cleanup(d); diff --git a/drivers/mtd/nand/raw/pl35x-nand-controller.c b/drivers/mtd/nand/raw/pl35x-nand-controller.c index 81dde993ee9b..12b9e0936c8f 100644 --- a/drivers/mtd/nand/raw/pl35x-nand-controller.c +++ b/drivers/mtd/nand/raw/pl35x-nand-controller.c @@ -914,7 +914,6 @@ static int pl35x_nand_init_hw_ecc_controller(struct pl35x_nandc *nfc, chip->ecc.steps = mtd->writesize / chip->ecc.size; chip->ecc.read_page = pl35x_nand_read_page_hwecc; chip->ecc.write_page = pl35x_nand_write_page_hwecc; - chip->ecc.write_page_raw = nand_monolithic_write_page_raw; pl35x_smc_set_ecc_pg_size(nfc, chip, mtd->writesize); nfc->ecc_buf = devm_kmalloc(nfc->dev, chip->ecc.bytes * chip->ecc.steps, @@ -973,15 +972,19 @@ static int pl35x_nand_attach_chip(struct nand_chip *chip) switch (chip->ecc.engine_type) { case NAND_ECC_ENGINE_TYPE_ON_DIE: + dev_dbg(nfc->dev, "Using on-die hardware ECC\n"); /* Keep these legacy BBT descriptors for ON_DIE situations */ chip->bbt_td = &bbt_main_descr; chip->bbt_md = &bbt_mirror_descr; fallthrough; case NAND_ECC_ENGINE_TYPE_NONE: + dev_dbg(nfc->dev, "Using no ECC engine\n"); + break; case NAND_ECC_ENGINE_TYPE_SOFT: - chip->ecc.write_page_raw = nand_monolithic_write_page_raw; + dev_dbg(nfc->dev, "Using software ECC\n"); break; case NAND_ECC_ENGINE_TYPE_ON_HOST: + dev_dbg(nfc->dev, "Using on-host hardware ECC\n"); ret = pl35x_nand_init_hw_ecc_controller(nfc, chip); if (ret) return ret; @@ -992,6 +995,9 @@ static int pl35x_nand_attach_chip(struct nand_chip *chip) return -EINVAL; } + chip->ecc.read_page_raw = nand_monolithic_read_page_raw; + chip->ecc.write_page_raw = nand_monolithic_write_page_raw; + return 0; } diff --git a/drivers/mtd/ubi/attach.c b/drivers/mtd/ubi/attach.c index 884171871d0e..2b1629fdb41b 100644 --- a/drivers/mtd/ubi/attach.c +++ b/drivers/mtd/ubi/attach.c @@ -771,7 +771,7 @@ void ubi_remove_av(struct ubi_attach_info *ai, struct ubi_ainf_volume *av) static int early_erase_peb(struct ubi_device *ubi, const struct ubi_attach_info *ai, int pnum, int ec) { - int err; + int err, torture = 0; struct ubi_ec_hdr *ec_hdr; if ((long long)ec >= UBI_MAX_ERASECOUNTER) { @@ -790,7 +790,7 @@ static int early_erase_peb(struct ubi_device *ubi, ec_hdr->ec = cpu_to_be64(ec); - err = ubi_io_sync_erase(ubi, pnum, 0); + err = ubi_io_sync_erase(ubi, pnum, &torture); if (err < 0) goto out_free; diff --git a/drivers/mtd/ubi/build.c b/drivers/mtd/ubi/build.c index ef6a22f372f9..fb121b6d0848 100644 --- a/drivers/mtd/ubi/build.c +++ b/drivers/mtd/ubi/build.c @@ -1105,6 +1105,7 @@ int ubi_detach_mtd_dev(int ubi_num, int anyway) ubi->ref_count -= 1; if (ubi->ref_count) { if (!anyway) { + put_device(&ubi->dev); spin_unlock(&ubi_devices_lock); return -EBUSY; } @@ -1317,10 +1318,10 @@ static int __init ubi_init_attach(void) return 0; out_detach: - for (k = 0; k < i; k++) + for (k = 0; k < UBI_MAX_DEVICES; k++) if (ubi_devices[k]) { mutex_lock(&ubi_devices_mutex); - ubi_detach_mtd_dev(ubi_devices[k]->ubi_num, 1); + ubi_detach_mtd_dev(k, 1); mutex_unlock(&ubi_devices_mutex); } return err; @@ -1574,7 +1575,7 @@ static int ubi_mtd_param_parse(const char *val, const struct kernel_param *kp) } module_param_call(mtd, ubi_mtd_param_parse, NULL, NULL, 0400); -MODULE_PARM_DESC(mtd, "MTD devices to attach. Parameter format: mtd=[,[,max_beb_per1024[,ubi_num]]].\n" +MODULE_PARM_DESC(mtd, "MTD devices to attach. Parameter format: mtd=[,[,max_beb_per1024[,ubi_num[,enable_fm[,need_resv_pool]]]]].\n" "Multiple \"mtd\" parameters may be specified.\n" "MTD devices may be specified by their number, name, or path to the MTD character device node.\n" "Optional \"vid_hdr_offs\" parameter specifies UBI VID header position to be used by UBI. (default value if 0)\n" @@ -1587,9 +1588,9 @@ MODULE_PARM_DESC(mtd, "MTD devices to attach. Parameter format: mtd=e[i]) { - ret = ubi_sync_erase(ubi, old_fm->e[i], 0); + ret = ubi_sync_erase(ubi, old_fm->e[i], + &old_fm->to_be_tortured[i]); if (ret < 0) { ubi_err(ubi, "could not erase old fastmap PEB"); @@ -1578,7 +1579,8 @@ int ubi_update_fastmap(struct ubi_device *ubi) if (old_fm) { /* no fresh anchor PEB was found, reuse the old one */ if (!tmp_e) { - ret = ubi_sync_erase(ubi, old_fm->e[0], 0); + ret = ubi_sync_erase(ubi, old_fm->e[0], + &old_fm->to_be_tortured[0]); if (ret < 0) { ubi_err(ubi, "could not erase old anchor PEB"); diff --git a/drivers/mtd/ubi/io.c b/drivers/mtd/ubi/io.c index 915eb64cb001..d70d1313ad4e 100644 --- a/drivers/mtd/ubi/io.c +++ b/drivers/mtd/ubi/io.c @@ -519,8 +519,10 @@ static int nor_erase_prepare(struct ubi_device *ubi, int pnum) /** * ubi_io_sync_erase - synchronously erase a physical eraseblock. * @ubi: UBI device description object - * @pnum: physical eraseblock number to erase - * @torture: if this physical eraseblock has to be tortured + * @pnum: the physical eraseblock number to erase + * @torture: if this physical eraseblock has to be tortured; cleared to zero + * once the torture test has completed successfully so that a retry + * of the erase does not torture the physical eraseblock again * * This function synchronously erases physical eraseblock @pnum. If @torture * flag is not zero, the physical eraseblock is checked by means of writing @@ -532,7 +534,7 @@ static int nor_erase_prepare(struct ubi_device *ubi, int pnum) * codes in case of other errors. Note, %-EIO means that the physical * eraseblock is bad. */ -int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int torture) +int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int *torture) { int err, ret = 0; @@ -560,10 +562,11 @@ int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int torture) return err; } - if (torture) { + if (*torture) { ret = torture_peb(ubi, pnum); if (ret < 0) return ret; + *torture = 0; } err = do_sync_erase(ubi, pnum); diff --git a/drivers/mtd/ubi/ubi.h b/drivers/mtd/ubi/ubi.h index 44803d3329f4..edc68c1603e5 100644 --- a/drivers/mtd/ubi/ubi.h +++ b/drivers/mtd/ubi/ubi.h @@ -901,7 +901,7 @@ int self_check_eba(struct ubi_device *ubi, struct ubi_attach_info *ai_fastmap, struct ubi_attach_info *ai_scan); /* wl.c */ -int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int torture); +int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int *torture); int ubi_wl_get_peb(struct ubi_device *ubi); int ubi_wl_put_peb(struct ubi_device *ubi, int vol_id, int lnum, int pnum, int torture); @@ -923,7 +923,7 @@ int ubi_io_read(const struct ubi_device *ubi, void *buf, int pnum, int offset, int len); int ubi_io_write(struct ubi_device *ubi, const void *buf, int pnum, int offset, int len); -int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int torture); +int ubi_io_sync_erase(struct ubi_device *ubi, int pnum, int *torture); int ubi_io_is_bad(const struct ubi_device *ubi, int pnum); int ubi_io_mark_bad(const struct ubi_device *ubi, int pnum); int ubi_io_read_ec_hdr(struct ubi_device *ubi, int pnum, diff --git a/drivers/mtd/ubi/wl.c b/drivers/mtd/ubi/wl.c index fbd399cf6503..fefcbaf25002 100644 --- a/drivers/mtd/ubi/wl.c +++ b/drivers/mtd/ubi/wl.c @@ -443,12 +443,14 @@ static int prot_queue_del(struct ubi_device *ubi, int pnum) * ubi_sync_erase - synchronously erase a physical eraseblock. * @ubi: UBI device description object * @e: the physical eraseblock to erase - * @torture: if the physical eraseblock has to be tortured + * @torture: if the physical eraseblock has to be tortured; cleared to zero + * once the torture test has completed successfully so that a retry + * of the erase does not torture the physical eraseblock again * * This function returns zero in case of success and a negative error code in * case of failure. */ -int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int torture) +int ubi_sync_erase(struct ubi_device *ubi, struct ubi_wl_entry *e, int *torture) { int err; struct ubi_ec_hdr *ec_hdr; @@ -1113,7 +1115,7 @@ static int __erase_worker(struct ubi_device *ubi, struct ubi_work *wl_wrk) dbg_wl("erase PEB %d EC %d LEB %d:%d", pnum, e->ec, wl_wrk->vol_id, wl_wrk->lnum); - err = ubi_sync_erase(ubi, e, wl_wrk->torture); + err = ubi_sync_erase(ubi, e, &wl_wrk->torture); if (!err) { spin_lock(&ubi->wl_lock); @@ -1150,7 +1152,8 @@ static int __erase_worker(struct ubi_device *ubi, struct ubi_work *wl_wrk) int err1; /* Re-schedule the LEB for erasure */ - err1 = schedule_erase(ubi, e, vol_id, lnum, 0, true); + err1 = schedule_erase(ubi, e, vol_id, lnum, wl_wrk->torture, + true); if (err1) { spin_lock(&ubi->wl_lock); wl_entry_destroy(ubi, e); @@ -1757,7 +1760,7 @@ static void shutdown_work(struct ubi_device *ubi) static int erase_aeb(struct ubi_device *ubi, struct ubi_ainf_peb *aeb, bool sync) { struct ubi_wl_entry *e; - int err; + int err, torture = 0; e = kmem_cache_alloc(ubi_wl_entry_slab, GFP_KERNEL); if (!e) @@ -1768,7 +1771,7 @@ static int erase_aeb(struct ubi_device *ubi, struct ubi_ainf_peb *aeb, bool sync ubi->lookuptbl[e->pnum] = e; if (sync) { - err = ubi_sync_erase(ubi, e, false); + err = ubi_sync_erase(ubi, e, &torture); if (err) goto out_free; diff --git a/drivers/net/amt.c b/drivers/net/amt.c index 2bc8781f07c2..01d2223fce09 100644 --- a/drivers/net/amt.c +++ b/drivers/net/amt.c @@ -3221,6 +3221,9 @@ static int amt_newlink(struct net_device *dev, struct nlattr **tb = params->tb; int err = -EINVAL; + if (!net_eq(link_net, dev_net(dev))) + return err; + amt->net = link_net; amt->mode = nla_get_u32(data[IFLA_AMT_MODE]); diff --git a/drivers/net/bonding/bond_netlink.c b/drivers/net/bonding/bond_netlink.c index 08d4d0814f67..4367cac1f4e8 100644 --- a/drivers/net/bonding/bond_netlink.c +++ b/drivers/net/bonding/bond_netlink.c @@ -220,7 +220,7 @@ static int bond_changelink(struct net_device *bond_dev, struct nlattr *tb[], struct bonding *bond = netdev_priv(bond_dev); struct bond_opt_value newval; int miimon = 0; - int err; + int err = 0; if (!data) return 0; diff --git a/drivers/net/dsa/b53/b53_common.c b/drivers/net/dsa/b53/b53_common.c index 7d6ec2eb7c75..508b46237011 100644 --- a/drivers/net/dsa/b53/b53_common.c +++ b/drivers/net/dsa/b53/b53_common.c @@ -2204,7 +2204,7 @@ int b53_fdb_dump(struct dsa_switch *ds, int port, mutex_unlock(&priv->arl_mutex); - return 0; + return ret; } EXPORT_SYMBOL(b53_fdb_dump); diff --git a/drivers/net/dsa/mv88e6xxx/pcs-6352.c b/drivers/net/dsa/mv88e6xxx/pcs-6352.c index 36993400837e..e09019ed19fb 100644 --- a/drivers/net/dsa/mv88e6xxx/pcs-6352.c +++ b/drivers/net/dsa/mv88e6xxx/pcs-6352.c @@ -305,13 +305,16 @@ static bool mv88e6352_pcs_link_check(struct marvell_c22_pcs *mpcs) struct mv88e6xxx_port *port = mpcs->port; struct mv88e6xxx_chip *chip = port->chip; u8 cmode; + int err; /* Port 4 can be in auto-media mode. Check that the port is * associated with the mpcs. */ mv88e6xxx_reg_lock(chip); - chip->info->ops->port_get_cmode(chip, port->port, &cmode); + err = chip->info->ops->port_get_cmode(chip, port->port, &cmode); mv88e6xxx_reg_unlock(chip); + if (err) + return false; return cmode == MV88E6XXX_PORT_STS_CMODE_100BASEX || cmode == MV88E6XXX_PORT_STS_CMODE_1000BASEX || diff --git a/drivers/net/ethernet/airoha/airoha_npu.c b/drivers/net/ethernet/airoha/airoha_npu.c index d6547834d3c7..ddc5a6a7681b 100644 --- a/drivers/net/ethernet/airoha/airoha_npu.c +++ b/drivers/net/ethernet/airoha/airoha_npu.c @@ -686,7 +686,7 @@ static int airoha_npu_probe(struct platform_device *pdev) npu->irqs[i] = irq; } - err = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)); + err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); if (err) return err; diff --git a/drivers/net/ethernet/allwinner/sun4i-emac.c b/drivers/net/ethernet/allwinner/sun4i-emac.c index 2f516b950f4e..a573c3c61f76 100644 --- a/drivers/net/ethernet/allwinner/sun4i-emac.c +++ b/drivers/net/ethernet/allwinner/sun4i-emac.c @@ -996,9 +996,9 @@ static int emac_probe(struct platform_device *pdev) /* fill in parameters for net-dev structure */ ndev->base_addr = (unsigned long)db->membase; ndev->irq = irq_of_parse_and_map(np, 0); - if (ndev->irq == -ENXIO) { + if (!ndev->irq) { netdev_err(ndev, "No irq resource\n"); - ret = ndev->irq; + ret = -ENXIO; goto out_iounmap; } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 186fc98a555d..e2bceec96932 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -11772,9 +11772,11 @@ static int bnxt_request_irq(struct bnxt *bp) #endif /* Enable TPH support as part of IRQ request */ - rc = pcie_enable_tph(bp->pdev, PCI_TPH_ST_IV_MODE); - if (!rc) - bp->tph_mode = PCI_TPH_ST_IV_MODE; + if (BNXT_SUPPORTS_QUEUE_API(bp)) { + rc = pcie_enable_tph(bp->pdev, PCI_TPH_ST_IV_MODE); + if (!rc) + bp->tph_mode = PCI_TPH_ST_IV_MODE; + } for (i = 0, j = 0; i < bp->cp_nr_rings; i++) { int map_idx = bnxt_cp_num_to_irq_num(bp, i); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_hwmon.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_hwmon.c index de3427c6c6aa..faed0cf726e1 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_hwmon.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_hwmon.c @@ -40,7 +40,7 @@ void bnxt_hwmon_notify_event(struct bnxt *bp) return; } - hwmon_notify_event(&bp->pdev->dev, hwmon_temp, attr, 0); + hwmon_notify_event(bp->hwmon_dev, hwmon_temp, attr, 0); } static int bnxt_hwrm_temp_query(struct bnxt *bp, u8 *temp) diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c index 6603ed696296..dec047c8128d 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.c +++ b/drivers/net/ethernet/freescale/enetc/enetc.c @@ -72,6 +72,33 @@ void enetc_reset_mac_addr_filter(struct enetc_mac_filter *filter) } EXPORT_SYMBOL_GPL(enetc_reset_mac_addr_filter); +void enetc_set_congestion_mode(struct enetc_ndev_priv *priv, bool enable) +{ + struct enetc_si *si = priv->si; + struct enetc_hw *hw = &si->hw; + + spin_lock(&si->gen_lock); + + if (enable) + set_bit(ENETC_RXBDR_CM, &priv->flags); + else + clear_bit(ENETC_RXBDR_CM, &priv->flags); + + for (int i = 0; i < priv->num_rx_rings; i++) { + u32 old_rbmr = enetc_rxbdr_rd(hw, i, ENETC_RBMR); + u32 rbmr; + + rbmr = u32_replace_bits(old_rbmr, enable, ENETC_RBMR_CM); + if (rbmr == old_rbmr) + continue; + + enetc_rxbdr_wr(hw, i, ENETC_RBMR, rbmr); + } + + spin_unlock(&si->gen_lock); +} +EXPORT_SYMBOL_GPL(enetc_set_congestion_mode); + static int enetc_num_stack_tx_queues(struct enetc_ndev_priv *priv) { int num_tx_rings = priv->num_tx_rings; @@ -2624,7 +2651,6 @@ static void enetc_setup_rxbdr(struct enetc_hw *hw, struct enetc_bdr *rx_ring, bool extended) { int idx = rx_ring->index; - u32 rbmr = 0; enetc_rxbdr_wr(hw, idx, ENETC_RBBAR0, lower_32_bits(rx_ring->bd_dma_base)); @@ -2652,12 +2678,6 @@ static void enetc_setup_rxbdr(struct enetc_hw *hw, struct enetc_bdr *rx_ring, enetc_rxbdr_wr(hw, idx, ENETC_RBICR0, ENETC_RBICR0_ICEN | 0x1); rx_ring->ext_en = extended; - if (rx_ring->ext_en) - rbmr |= ENETC_RBMR_BDS; - - if (rx_ring->ndev->features & NETIF_F_HW_VLAN_CTAG_RX) - rbmr |= ENETC_RBMR_VTE; - rx_ring->rcir = hw->reg + ENETC_BDR(RX, idx, ENETC_RBCIR); rx_ring->idr = hw->reg + ENETC_SIRXIDR; @@ -2668,8 +2688,6 @@ static void enetc_setup_rxbdr(struct enetc_hw *hw, struct enetc_bdr *rx_ring, enetc_lock_mdio(); enetc_refill_rx_ring(rx_ring, enetc_bd_unused(rx_ring)); enetc_unlock_mdio(); - - enetc_rxbdr_wr(hw, idx, ENETC_RBMR, rbmr); } static void enetc_setup_bdrs(struct enetc_ndev_priv *priv, bool extended) @@ -2696,21 +2714,34 @@ static void enetc_enable_txbdr(struct enetc_hw *hw, struct enetc_bdr *tx_ring) static void enetc_enable_rxbdr(struct enetc_hw *hw, struct enetc_bdr *rx_ring) { + struct enetc_ndev_priv *priv = netdev_priv(rx_ring->ndev); int idx = rx_ring->index; - u32 rbmr; + u32 rbmr = ENETC_RBMR_EN; + + if (rx_ring->ext_en) + rbmr |= ENETC_RBMR_BDS; + + if (rx_ring->ndev->features & NETIF_F_HW_VLAN_CTAG_RX) + rbmr |= ENETC_RBMR_VTE; + + if (test_bit(ENETC_RXBDR_CM, &priv->flags)) + rbmr |= ENETC_RBMR_CM; - rbmr = enetc_rxbdr_rd(hw, idx, ENETC_RBMR); - rbmr |= ENETC_RBMR_EN; enetc_rxbdr_wr(hw, idx, ENETC_RBMR, rbmr); } static void enetc_enable_rx_bdrs(struct enetc_ndev_priv *priv) { - struct enetc_hw *hw = &priv->si->hw; + struct enetc_si *si = priv->si; + struct enetc_hw *hw = &si->hw; int i; + spin_lock(&si->gen_lock); + for (i = 0; i < priv->num_rx_rings; i++) enetc_enable_rxbdr(hw, priv->rx_ring[i]); + + spin_unlock(&si->gen_lock); } static void enetc_enable_tx_bdrs(struct enetc_ndev_priv *priv) @@ -2740,11 +2771,16 @@ static void enetc_disable_txbdr(struct enetc_hw *hw, struct enetc_bdr *rx_ring) static void enetc_disable_rx_bdrs(struct enetc_ndev_priv *priv) { - struct enetc_hw *hw = &priv->si->hw; + struct enetc_si *si = priv->si; + struct enetc_hw *hw = &si->hw; int i; + spin_lock(&si->gen_lock); + for (i = 0; i < priv->num_rx_rings; i++) enetc_disable_rxbdr(hw, priv->rx_ring[i]); + + spin_unlock(&si->gen_lock); } static void enetc_disable_tx_bdrs(struct enetc_ndev_priv *priv) @@ -3336,11 +3372,16 @@ EXPORT_SYMBOL_GPL(enetc_get_stats); static void enetc_enable_rxvlan(struct net_device *ndev, bool en) { struct enetc_ndev_priv *priv = netdev_priv(ndev); - struct enetc_hw *hw = &priv->si->hw; + struct enetc_si *si = priv->si; + struct enetc_hw *hw = &si->hw; int i; + spin_lock(&si->gen_lock); + for (i = 0; i < priv->num_rx_rings; i++) enetc_bdr_enable_rxvlan(hw, i, en); + + spin_unlock(&si->gen_lock); } static void enetc_enable_txvlan(struct net_device *ndev, bool en) @@ -3670,6 +3711,7 @@ int enetc_pci_probe(struct pci_dev *pdev, const char *name, int sizeof_priv) si = PTR_ALIGN(p, ENETC_SI_ALIGN); si->pad = (char *)si - (char *)p; + spin_lock_init(&si->gen_lock); pci_set_drvdata(pdev, si); si->pdev = pdev; diff --git a/drivers/net/ethernet/freescale/enetc/enetc.h b/drivers/net/ethernet/freescale/enetc/enetc.h index 60c7205ea9ff..0e2effa947b2 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc.h +++ b/drivers/net/ethernet/freescale/enetc/enetc.h @@ -309,6 +309,13 @@ struct enetc_si { struct net_device *ndev; /* back ref. */ + /* General-purpose lock serializing updates that must not race, + * e.g. read-modify-write of shared hardware registers and of + * selected priv->flags bits between the phylink link callbacks + * and the ring (re)configuration path. + */ + spinlock_t gen_lock; + union { struct enetc_cbdr cbd_ring; /* Only ENETC 1.0 */ struct ntmp_user ntmp_user; /* ENETC 4.1 and later */ @@ -413,6 +420,7 @@ enum enetc_active_offloads { enum enetc_flags_bit { ENETC_TX_ONESTEP_TSTAMP_IN_PROGRESS = 0, ENETC_TX_DOWN, + ENETC_RXBDR_CM, }; /* interrupt coalescing modes */ @@ -509,6 +517,7 @@ int enetc_get_driver_data(struct enetc_si *si); void enetc_add_mac_addr_ht_filter(struct enetc_mac_filter *filter, const unsigned char *addr); void enetc_reset_mac_addr_filter(struct enetc_mac_filter *filter); +void enetc_set_congestion_mode(struct enetc_ndev_priv *priv, bool enable); int enetc_open(struct net_device *ndev); int enetc_close(struct net_device *ndev); diff --git a/drivers/net/ethernet/freescale/enetc/enetc_pf.c b/drivers/net/ethernet/freescale/enetc/enetc_pf.c index de0fb272c847..ba630f098210 100644 --- a/drivers/net/ethernet/freescale/enetc/enetc_pf.c +++ b/drivers/net/ethernet/freescale/enetc/enetc_pf.c @@ -691,8 +691,7 @@ static void enetc_pl_mac_link_up(struct phylink_config *config, struct enetc_hw *hw = &pf->si->hw; struct enetc_si *si = pf->si; struct enetc_ndev_priv *priv; - u32 rbmr, cmd_cfg; - int idx; + u32 cmd_cfg; priv = netdev_priv(pf->si->ndev); @@ -704,16 +703,7 @@ static void enetc_pl_mac_link_up(struct phylink_config *config, enetc_force_rgmii_mac(si, speed, duplex); /* Flow control */ - for (idx = 0; idx < priv->num_rx_rings; idx++) { - rbmr = enetc_rxbdr_rd(hw, idx, ENETC_RBMR); - - if (tx_pause) - rbmr |= ENETC_RBMR_CM; - else - rbmr &= ~ENETC_RBMR_CM; - - enetc_rxbdr_wr(hw, idx, ENETC_RBMR, rbmr); - } + enetc_set_congestion_mode(priv, tx_pause); if (tx_pause) { /* When the port first enters congestion, send a PAUSE request diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c index 7699a970cb25..b37ab4f826bd 100644 --- a/drivers/net/ethernet/freescale/fec_main.c +++ b/drivers/net/ethernet/freescale/fec_main.c @@ -4642,7 +4642,8 @@ fec_probe(struct platform_device *pdev) failed_irq: fec_enet_deinit(ndev); failed_init: - fec_ptp_stop(pdev); + if (fep->bufdesc_ex) + fec_ptp_stop(pdev); failed_reset: pm_runtime_put_noidle(&pdev->dev); pm_runtime_disable(&pdev->dev); @@ -4684,7 +4685,8 @@ fec_drv_remove(struct platform_device *pdev) ERR_PTR(ret)); cancel_work_sync(&fep->tx_timeout_work); - fec_ptp_stop(pdev); + if (fep->bufdesc_ex) + fec_ptp_stop(pdev); unregister_netdev(ndev); fec_enet_mii_remove(fep); if (fep->reg_phy) diff --git a/drivers/net/ethernet/huawei/hinic3/hinic3_tx.c b/drivers/net/ethernet/huawei/hinic3/hinic3_tx.c index 92c43c05e3f2..6e29c892a97d 100644 --- a/drivers/net/ethernet/huawei/hinic3/hinic3_tx.c +++ b/drivers/net/ethernet/huawei/hinic3/hinic3_tx.c @@ -240,8 +240,7 @@ static int hinic3_tx_csum(struct hinic3_txq *txq, struct hinic3_sq_task *task, ((struct udphdr *)skb_transport_header(skb))->dest != VXLAN_OFFLOAD_PORT_LE) { /* Unsupported tunnel packet, disable csum offload */ - skb_checksum_help(skb); - return 0; + return skb_checksum_help(skb); } } @@ -391,6 +390,10 @@ static u32 hinic3_tx_offload(struct sk_buff *skb, struct hinic3_sq_task *task, offload |= HINIC3_TX_OFFLOAD_TSO; } else { tso_cs_en = hinic3_tx_csum(txq, task, skb); + if (tso_cs_en < 0) { + offload = HINIC3_TX_OFFLOAD_INVALID; + return offload; + } if (tso_cs_en) offload |= HINIC3_TX_OFFLOAD_CSUM; } @@ -524,6 +527,7 @@ static netdev_tx_t hinic3_send_one_skb(struct sk_buff *skb, skb->len = MIN_SKB_LEN; } + offload = hinic3_tx_offload(skb, &task, &queue_info, txq); num_sge = skb_shinfo(skb)->nr_frags + 1; /* assume normal wqe format + 1 wqebb for task info */ wqebb_cnt = num_sge + 1; @@ -539,7 +543,6 @@ static netdev_tx_t hinic3_send_one_skb(struct sk_buff *skb, return NETDEV_TX_BUSY; } - offload = hinic3_tx_offload(skb, &task, &queue_info, txq); if (unlikely(offload == HINIC3_TX_OFFLOAD_INVALID)) { goto err_drop_pkt; } else if (!offload) { diff --git a/drivers/net/ethernet/intel/ice/devlink/devlink.c b/drivers/net/ethernet/intel/ice/devlink/devlink.c index 839b7bfa1935..4bd377a66842 100644 --- a/drivers/net/ethernet/intel/ice/devlink/devlink.c +++ b/drivers/net/ethernet/intel/ice/devlink/devlink.c @@ -1884,27 +1884,18 @@ static int ice_devlink_nvm_snapshot(struct devlink *devlink, */ for (i = 0; i < num_blks; i++) { u32 read_sz = min_t(u32, ICE_DEVLINK_READ_BLK_SIZE, left); - - status = ice_acquire_nvm(hw, ICE_RES_READ); - if (status) { - dev_dbg(dev, "ice_acquire_nvm failed, err %d aq_err %d\n", - status, hw->adminq.sq_last_status); - NL_SET_ERR_MSG_MOD(extack, "Failed to acquire NVM semaphore"); - vfree(nvm_data); - return -EIO; - } + enum libie_aq_err read_aq_err = LIBIE_AQ_RC_OK; status = ice_read_flat_nvm(hw, i * ICE_DEVLINK_READ_BLK_SIZE, - &read_sz, tmp, read_shadow_ram); + &read_sz, tmp, read_shadow_ram, + &read_aq_err); if (status) { dev_dbg(dev, "ice_read_flat_nvm failed after reading %u bytes, err %d aq_err %d\n", - read_sz, status, hw->adminq.sq_last_status); + read_sz, status, read_aq_err); NL_SET_ERR_MSG_MOD(extack, "Failed to read NVM contents"); - ice_release_nvm(hw); vfree(nvm_data); return -EIO; } - ice_release_nvm(hw); tmp += read_sz; left -= read_sz; @@ -1937,6 +1928,7 @@ static int ice_devlink_nvm_read(struct devlink *devlink, struct netlink_ext_ack *extack, u64 offset, u32 size, u8 *data) { + enum libie_aq_err read_aq_err = LIBIE_AQ_RC_OK; struct ice_pf *pf = devlink_priv(devlink); struct device *dev = ice_pf_to_dev(pf); struct ice_hw *hw = &pf->hw; @@ -1960,24 +1952,14 @@ static int ice_devlink_nvm_read(struct devlink *devlink, return -ERANGE; } - status = ice_acquire_nvm(hw, ICE_RES_READ); - if (status) { - dev_dbg(dev, "ice_acquire_nvm failed, err %d aq_err %d\n", - status, hw->adminq.sq_last_status); - NL_SET_ERR_MSG_MOD(extack, "Failed to acquire NVM semaphore"); - return -EIO; - } - status = ice_read_flat_nvm(hw, (u32)offset, &size, data, - read_shadow_ram); + read_shadow_ram, &read_aq_err); if (status) { dev_dbg(dev, "ice_read_flat_nvm failed after reading %u bytes, err %d aq_err %d\n", - size, status, hw->adminq.sq_last_status); + size, status, read_aq_err); NL_SET_ERR_MSG_MOD(extack, "Failed to read NVM contents"); - ice_release_nvm(hw); return -EIO; } - ice_release_nvm(hw); return 0; } diff --git a/drivers/net/ethernet/intel/ice/ice_ethtool.c b/drivers/net/ethernet/intel/ice/ice_ethtool.c index 5944ab6efd44..c10d0b92e63e 100644 --- a/drivers/net/ethernet/intel/ice/ice_ethtool.c +++ b/drivers/net/ethernet/intel/ice/ice_ethtool.c @@ -794,8 +794,7 @@ static int ice_get_extended_regs(struct net_device *netdev, void *p) static void ice_get_regs(struct net_device *netdev, struct ethtool_regs *regs, void *p) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); struct ice_hw *hw = &pf->hw; u32 *regs_buf = (u32 *)p; unsigned int i; @@ -810,8 +809,7 @@ ice_get_regs(struct net_device *netdev, struct ethtool_regs *regs, void *p) static u32 ice_get_msglevel(struct net_device *netdev) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); #ifndef CONFIG_DYNAMIC_DEBUG if (pf->hw.debug_mask) @@ -824,8 +822,7 @@ static u32 ice_get_msglevel(struct net_device *netdev) static void ice_set_msglevel(struct net_device *netdev, u32 data) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); #ifndef CONFIG_DYNAMIC_DEBUG if (ICE_DBG_USER & data) @@ -840,16 +837,14 @@ static void ice_set_msglevel(struct net_device *netdev, u32 data) static void ice_get_link_ext_stats(struct net_device *netdev, struct ethtool_link_ext_stats *stats) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); stats->link_down_events = pf->link_down_events; } static int ice_get_eeprom_len(struct net_device *netdev) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); return (int)pf->hw.flash.flash_size; } @@ -858,9 +853,8 @@ static int ice_get_eeprom(struct net_device *netdev, struct ethtool_eeprom *eeprom, u8 *bytes) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_vsi *vsi = np->vsi; - struct ice_pf *pf = vsi->back; + enum libie_aq_err read_aq_err = LIBIE_AQ_RC_OK; + struct ice_pf *pf = ice_netdev_to_pf(netdev); struct ice_hw *hw = &pf->hw; struct device *dev; int ret; @@ -876,24 +870,15 @@ ice_get_eeprom(struct net_device *netdev, struct ethtool_eeprom *eeprom, if (!buf) return -ENOMEM; - ret = ice_acquire_nvm(hw, ICE_RES_READ); - if (ret) { - dev_err(dev, "ice_acquire_nvm failed, err %d aq_err %s\n", - ret, libie_aq_str(hw->adminq.sq_last_status)); - goto out; - } - ret = ice_read_flat_nvm(hw, eeprom->offset, &eeprom->len, buf, - false); + false, &read_aq_err); if (ret) { dev_err(dev, "ice_read_flat_nvm failed, err %d aq_err %s\n", - ret, libie_aq_str(hw->adminq.sq_last_status)); - goto release; + ret, libie_aq_str(read_aq_err)); + goto out; } memcpy(bytes, buf, eeprom->len); -release: - ice_release_nvm(hw); out: kfree(buf); return ret; @@ -959,8 +944,7 @@ static u64 ice_link_test(struct net_device *netdev) */ static u64 ice_eeprom_test(struct net_device *netdev) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); netdev_info(netdev, "EEPROM test\n"); return !!(ice_nvm_validate_checksum(&pf->hw)); @@ -1274,9 +1258,8 @@ static int ice_lbtest_receive_frames(struct ice_rx_ring *rx_ring) */ static u64 ice_loopback_test(struct net_device *netdev) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_vsi *orig_vsi = np->vsi, *test_vsi; - struct ice_pf *pf = orig_vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); + struct ice_vsi *test_vsi; u8 *tx_frame __free(kfree) = NULL; u8 broadcast[ETH_ALEN], ret = 0; int num_frames, valid_frames; @@ -1365,8 +1348,7 @@ static u64 ice_loopback_test(struct net_device *netdev) */ static u64 ice_intr_test(struct net_device *netdev) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); u16 swic_old = pf->sw_int_count; netdev_info(netdev, "interrupt test\n"); @@ -1394,9 +1376,8 @@ static void ice_self_test(struct net_device *netdev, struct ethtool_test *eth_test, u64 *data) { - struct ice_netdev_priv *np = netdev_priv(netdev); + struct ice_pf *pf = ice_netdev_to_pf(netdev); bool if_running = netif_running(netdev); - struct ice_pf *pf = np->vsi->back; struct device *dev; dev = ice_pf_to_dev(pf); @@ -1720,9 +1701,7 @@ static int ice_nway_reset(struct net_device *netdev) */ static u32 ice_get_priv_flags(struct net_device *netdev) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_vsi *vsi = np->vsi; - struct ice_pf *pf = vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); u32 i, ret_flags = 0; for (i = 0; i < ICE_PRIV_FLAG_ARRAY_SIZE; i++) { @@ -4432,9 +4411,7 @@ static int ice_get_module_info(struct net_device *netdev, struct ethtool_modinfo *modinfo) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_vsi *vsi = np->vsi; - struct ice_pf *pf = vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); struct ice_hw *hw = &pf->hw; u8 sff8472_comp = 0; u8 sff8472_swap = 0; @@ -4506,12 +4483,10 @@ static int ice_get_module_eeprom(struct net_device *netdev, struct ethtool_eeprom *ee, u8 *data) { - struct ice_netdev_priv *np = netdev_priv(netdev); + struct ice_pf *pf = ice_netdev_to_pf(netdev); #define SFF_READ_BLOCK_SIZE 8 u8 value[SFF_READ_BLOCK_SIZE] = { 0 }; u8 addr = ICE_I2C_EEPROM_DEV_ADDR; - struct ice_vsi *vsi = np->vsi; - struct ice_pf *pf = vsi->back; struct ice_hw *hw = &pf->hw; bool is_sfp = false; unsigned int i; @@ -4690,8 +4665,7 @@ static void ice_get_fec_stats(struct net_device *netdev, */ static int ice_ethtool_reset(struct net_device *dev, u32 *flags) { - struct ice_netdev_priv *np = netdev_priv(dev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(dev); enum ice_reset_req reset; switch (*flags) { diff --git a/drivers/net/ethernet/intel/ice/ice_flex_pipe.c b/drivers/net/ethernet/intel/ice/ice_flex_pipe.c index 013c93b6605e..c8cb492fddf4 100644 --- a/drivers/net/ethernet/intel/ice/ice_flex_pipe.c +++ b/drivers/net/ethernet/intel/ice/ice_flex_pipe.c @@ -574,9 +574,7 @@ ice_destroy_tunnel(struct ice_hw *hw, u16 index, enum ice_tunnel_type type, int ice_udp_tunnel_set_port(struct net_device *netdev, unsigned int table, unsigned int idx, struct udp_tunnel_info *ti) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_vsi *vsi = np->vsi; - struct ice_pf *pf = vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); enum ice_tunnel_type tnl_type; int status; u16 index; @@ -598,9 +596,7 @@ int ice_udp_tunnel_set_port(struct net_device *netdev, unsigned int table, int ice_udp_tunnel_unset_port(struct net_device *netdev, unsigned int table, unsigned int idx, struct udp_tunnel_info *ti) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_vsi *vsi = np->vsi; - struct ice_pf *pf = vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); enum ice_tunnel_type tnl_type; int status; diff --git a/drivers/net/ethernet/intel/ice/ice_lag.c b/drivers/net/ethernet/intel/ice/ice_lag.c index e8ab36d0f11d..b2ea0532f07b 100644 --- a/drivers/net/ethernet/intel/ice/ice_lag.c +++ b/drivers/net/ethernet/intel/ice/ice_lag.c @@ -2177,8 +2177,7 @@ static void ice_lag_chk_disabled_bond(struct ice_lag *lag, void *ptr) */ static void ice_lag_disable_sriov_bond(struct ice_lag *lag) { - struct ice_netdev_priv *np = netdev_priv(lag->netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(lag->netdev); ice_clear_feature_support(pf, ICE_F_SRIOV_LAG); ice_clear_feature_support(pf, ICE_F_SRIOV_AA_LAG); diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c index 3d14932871c5..851fab181251 100644 --- a/drivers/net/ethernet/intel/ice/ice_lib.c +++ b/drivers/net/ethernet/intel/ice/ice_lib.c @@ -2865,6 +2865,9 @@ int ice_vsi_release(struct ice_vsi *vsi) return -ENODEV; pf = vsi->back; + if (ice_is_vsi_dflt_vsi(vsi)) + ice_clear_dflt_vsi(vsi); + if (test_bit(ICE_FLAG_RSS_ENA, pf->flags)) ice_rss_clean(vsi); diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c index 2f1dd9eea2b2..f85051072ba2 100644 --- a/drivers/net/ethernet/intel/ice/ice_main.c +++ b/drivers/net/ethernet/intel/ice/ice_main.c @@ -8116,9 +8116,7 @@ static int ice_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq, struct net_device *dev, u32 filter_mask, int nlflags) { - struct ice_netdev_priv *np = netdev_priv(dev); - struct ice_vsi *vsi = np->vsi; - struct ice_pf *pf = vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(dev); u16 bmode; bmode = pf->first_sw->bridge_mode; @@ -8188,8 +8186,7 @@ ice_bridge_setlink(struct net_device *dev, struct nlmsghdr *nlh, u16 __always_unused flags, struct netlink_ext_ack __always_unused *extack) { - struct ice_netdev_priv *np = netdev_priv(dev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(dev); struct nlattr *attr, *br_spec; struct ice_hw *hw = &pf->hw; struct ice_sw *pf_sw; @@ -9623,8 +9620,7 @@ ice_indr_setup_tc_cb(struct net_device *netdev, struct Qdisc *sch, */ int ice_open(struct net_device *netdev) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); if (ice_is_reset_in_progress(pf->state)) { netdev_err(netdev, "can't open net device while reset is in progress"); diff --git a/drivers/net/ethernet/intel/ice/ice_nvm.c b/drivers/net/ethernet/intel/ice/ice_nvm.c index 7e187a804dfa..21f3b615dbbf 100644 --- a/drivers/net/ethernet/intel/ice/ice_nvm.c +++ b/drivers/net/ethernet/intel/ice/ice_nvm.c @@ -53,17 +53,27 @@ int ice_aq_read_nvm(struct ice_hw *hw, u16 module_typeid, u32 offset, * @length: (in) number of bytes to read; (out) number of bytes actually read * @data: buffer to return data in (sized to fit the specified length) * @read_shadow_ram: if true, read from shadow RAM instead of NVM + * @read_aq_err: if non-NULL, receives the AQ error status of the failing read * * Reads a portion of the NVM, as a flat memory space. This function correctly * breaks read requests across Shadow RAM sectors and ensures that no single * read request exceeds the maximum 4KB read for a single AdminQ command. * + * FW caps the read lock at a maximum of 3000ms, so a read spanning multiple + * 4KB sectors cannot be done under a single lock without FW reclaiming it + * mid-read. The NVM lock is therefore acquired and released around each AQ + * read, so this function must be called without the lock held. + * + * Since ice_release_nvm() issues an AQ command that overwrites + * hw->adminq.sq_last_status, callers that need the failing read's AQ error + * must use @read_aq_err rather than inspecting sq_last_status afterwards. + * * Returns a status code on failure. Note that the data pointer may be * partially updated if some reads succeed before a failure. */ int ice_read_flat_nvm(struct ice_hw *hw, u32 offset, u32 *length, u8 *data, - bool read_shadow_ram) + bool read_shadow_ram, enum libie_aq_err *read_aq_err) { u32 inlen = *length; u32 bytes_read = 0; @@ -92,12 +102,30 @@ ice_read_flat_nvm(struct ice_hw *hw, u32 offset, u32 *length, u8 *data, last_cmd = !(bytes_read + read_size < inlen); + status = ice_acquire_nvm(hw, ICE_RES_READ); + if (status) { + ice_debug(hw, ICE_DBG_NVM, "Failed to acquire NVM lock, err %d aq_err %s\n", + status, libie_aq_str(hw->adminq.sq_last_status)); + break; + } + status = ice_aq_read_nvm(hw, ICE_AQC_NVM_START_POINT, offset, read_size, data + bytes_read, last_cmd, read_shadow_ram, NULL); - if (status) + if (status) { + /* Capture the read's AQ error before ice_release_nvm() + * issues its own AQ command and overwrites + * sq_last_status. + */ + if (read_aq_err) + *read_aq_err = hw->adminq.sq_last_status; + + ice_release_nvm(hw); break; + } + + ice_release_nvm(hw); bytes_read += read_size; offset += read_size; @@ -177,14 +205,19 @@ int ice_aq_erase_nvm(struct ice_hw *hw, u16 module_typeid, struct ice_sq_cd *cd) } /** - * ice_read_sr_word_aq - Reads Shadow RAM via AQ + * ice_read_sr_word - Reads Shadow RAM word * @hw: pointer to the HW structure * @offset: offset of the Shadow RAM word to read (0x000000 - 0x001FFF) * @data: word read from the Shadow RAM * * Reads one 16 bit word from the Shadow RAM using ice_read_flat_nvm. + * + * The NVM lock is acquired and released internally by ice_read_flat_nvm() + * around the FW read, so this function must be called without the lock held. + * + * Return: zero on success, or a negative error code on failure. */ -static int ice_read_sr_word_aq(struct ice_hw *hw, u16 offset, u16 *data) +int ice_read_sr_word(struct ice_hw *hw, u16 offset, u16 *data) { u32 bytes = sizeof(u16); __le16 data_local; @@ -194,7 +227,7 @@ static int ice_read_sr_word_aq(struct ice_hw *hw, u16 offset, u16 *data) * Shadow RAM sector restrictions necessary when reading from the NVM. */ status = ice_read_flat_nvm(hw, offset * sizeof(u16), &bytes, - (__force u8 *)&data_local, true); + (__force u8 *)&data_local, true, NULL); if (status) return status; @@ -330,13 +363,8 @@ ice_read_flash_module(struct ice_hw *hw, enum ice_bank_select bank, u16 module, return -EINVAL; } - status = ice_acquire_nvm(hw, ICE_RES_READ); - if (status) - return status; - - status = ice_read_flat_nvm(hw, start + offset, &length, data, false); - - ice_release_nvm(hw); + status = ice_read_flat_nvm(hw, start + offset, &length, data, false, + NULL); return status; } @@ -418,27 +446,6 @@ ice_read_netlist_module(struct ice_hw *hw, enum ice_bank_select bank, u32 offset return status; } -/** - * ice_read_sr_word - Reads Shadow RAM word and acquire NVM if necessary - * @hw: pointer to the HW structure - * @offset: offset of the Shadow RAM word to read (0x000000 - 0x001FFF) - * @data: word read from the Shadow RAM - * - * Reads one 16 bit word from the Shadow RAM using the ice_read_sr_word_aq. - */ -int ice_read_sr_word(struct ice_hw *hw, u16 offset, u16 *data) -{ - int status; - - status = ice_acquire_nvm(hw, ICE_RES_READ); - if (!status) { - status = ice_read_sr_word_aq(hw, offset, data); - ice_release_nvm(hw); - } - - return status; -} - /** * ice_get_pfa_module_tlv - Reads sub module TLV from NVM PFA * @hw: pointer to hardware structure @@ -856,20 +863,18 @@ int ice_get_inactive_netlist_ver(struct ice_hw *hw, struct ice_netlist_info *net static int ice_discover_flash_size(struct ice_hw *hw) { u32 min_size = 0, max_size = ICE_AQC_NVM_MAX_OFFSET + 1; - int status; - - status = ice_acquire_nvm(hw, ICE_RES_READ); - if (status) - return status; + int status = 0; while ((max_size - min_size) > 1) { + enum libie_aq_err read_aq_err = LIBIE_AQ_RC_OK; u32 offset = (max_size + min_size) / 2; u32 len = 1; u8 data; - status = ice_read_flat_nvm(hw, offset, &len, &data, false); + status = ice_read_flat_nvm(hw, offset, &len, &data, false, + &read_aq_err); if (status == -EIO && - hw->adminq.sq_last_status == LIBIE_AQ_RC_EINVAL) { + read_aq_err == LIBIE_AQ_RC_EINVAL) { ice_debug(hw, ICE_DBG_NVM, "%s: New upper bound of %u bytes\n", __func__, offset); status = 0; @@ -880,7 +885,7 @@ static int ice_discover_flash_size(struct ice_hw *hw) min_size = offset; } else { /* an unexpected error occurred */ - goto err_read_flat_nvm; + return status; } } @@ -888,9 +893,6 @@ static int ice_discover_flash_size(struct ice_hw *hw) hw->flash.flash_size = max_size; -err_read_flat_nvm: - ice_release_nvm(hw); - return status; } diff --git a/drivers/net/ethernet/intel/ice/ice_nvm.h b/drivers/net/ethernet/intel/ice/ice_nvm.h index 63cdc6bdac58..e1d1a11f5ca4 100644 --- a/drivers/net/ethernet/intel/ice/ice_nvm.h +++ b/drivers/net/ethernet/intel/ice/ice_nvm.h @@ -19,7 +19,7 @@ int ice_aq_read_nvm(struct ice_hw *hw, u16 module_typeid, u32 offset, bool read_shadow_ram, struct ice_sq_cd *cd); int ice_read_flat_nvm(struct ice_hw *hw, u32 offset, u32 *length, u8 *data, - bool read_shadow_ram); + bool read_shadow_ram, enum libie_aq_err *read_aq_err); int ice_get_pfa_module_tlv(struct ice_hw *hw, u16 *module_tlv, u16 *module_tlv_len, u16 module_type); diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.c b/drivers/net/ethernet/intel/ice/ice_ptp.c index 30a34ed2ae60..7808d583b9ee 100644 --- a/drivers/net/ethernet/intel/ice/ice_ptp.c +++ b/drivers/net/ethernet/intel/ice/ice_ptp.c @@ -2164,8 +2164,7 @@ static int ice_ptp_getcrosststamp(struct ptp_clock_info *info, int ice_ptp_hwtstamp_get(struct net_device *netdev, struct kernel_hwtstamp_config *config) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); if (pf->ptp.state != ICE_PTP_READY) return -EIO; @@ -2236,8 +2235,7 @@ int ice_ptp_hwtstamp_set(struct net_device *netdev, struct kernel_hwtstamp_config *config, struct netlink_ext_ack *extack) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); int err; if (pf->ptp.state != ICE_PTP_READY) diff --git a/drivers/net/ethernet/intel/ice/ice_ptp_hw.c b/drivers/net/ethernet/intel/ice/ice_ptp_hw.c index 99bf38cf352a..f67e53c5dc87 100644 --- a/drivers/net/ethernet/intel/ice/ice_ptp_hw.c +++ b/drivers/net/ethernet/intel/ice/ice_ptp_hw.c @@ -4771,15 +4771,12 @@ static int ice_ptp_prep_phy_adj_ll_e810(struct ice_hw *hw, s32 adj) !FIELD_GET(REG_LL_PROXY_H_EXEC, val), 10, REG_LL_PROXY_H_TIMEOUT_US, false, hw, REG_LL_PROXY_H); - if (err) { - ice_debug(hw, ICE_DBG_PTP, "Failed to prepare PHY timer adjustment using low latency interface\n"); - spin_unlock_irq(¶ms->atqbal_wq.lock); - return err; - } - spin_unlock_irq(¶ms->atqbal_wq.lock); - return 0; + if (err) + ice_debug(hw, ICE_DBG_PTP, "Failed to prepare PHY timer adjustment using low latency interface\n"); + + return err; } /** @@ -4800,8 +4797,12 @@ static int ice_ptp_prep_phy_adj_e810(struct ice_hw *hw, s32 adj) u8 tmr_idx; int err; - if (hw->dev_caps.ts_dev_info.ll_phy_tmr_update) - return ice_ptp_prep_phy_adj_ll_e810(hw, adj); + if (hw->dev_caps.ts_dev_info.ll_phy_tmr_update) { + err = ice_ptp_prep_phy_adj_ll_e810(hw, adj); + if (err != -ETIMEDOUT) + return err; + ice_debug(hw, ICE_DBG_PTP, "LL adj timed out, falling back to SBQ\n"); + } tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned; @@ -4864,15 +4865,12 @@ static int ice_ptp_prep_phy_incval_ll_e810(struct ice_hw *hw, u64 incval) !FIELD_GET(REG_LL_PROXY_H_EXEC, val), 10, REG_LL_PROXY_H_TIMEOUT_US, false, hw, REG_LL_PROXY_H); - if (err) { - ice_debug(hw, ICE_DBG_PTP, "Failed to prepare PHY timer increment using low latency interface\n"); - spin_unlock_irq(¶ms->atqbal_wq.lock); - return err; - } - spin_unlock_irq(¶ms->atqbal_wq.lock); - return 0; + if (err) + ice_debug(hw, ICE_DBG_PTP, "Failed to prepare PHY timer increment using low latency interface\n"); + + return err; } /** @@ -4890,8 +4888,12 @@ static int ice_ptp_prep_phy_incval_e810(struct ice_hw *hw, u64 incval) u8 tmr_idx; int err; - if (hw->dev_caps.ts_dev_info.ll_phy_tmr_update) - return ice_ptp_prep_phy_incval_ll_e810(hw, incval); + if (hw->dev_caps.ts_dev_info.ll_phy_tmr_update) { + err = ice_ptp_prep_phy_incval_ll_e810(hw, incval); + if (err != -ETIMEDOUT) + return err; + ice_debug(hw, ICE_DBG_PTP, "LL incval timed out, falling back to SBQ\n"); + } tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned; low = lower_32_bits(incval); diff --git a/drivers/net/ethernet/intel/ice/ice_sriov.c b/drivers/net/ethernet/intel/ice/ice_sriov.c index 6a0b724e46f9..38f835647013 100644 --- a/drivers/net/ethernet/intel/ice/ice_sriov.c +++ b/drivers/net/ethernet/intel/ice/ice_sriov.c @@ -1192,8 +1192,7 @@ ice_vf_lan_overflow_event(struct ice_pf *pf, struct ice_rq_event_info *event) */ int ice_set_vf_spoofchk(struct net_device *netdev, int vf_id, bool ena) { - struct ice_netdev_priv *np = netdev_priv(netdev); - struct ice_pf *pf = np->vsi->back; + struct ice_pf *pf = ice_netdev_to_pf(netdev); struct ice_vsi *vf_vsi; struct device *dev; struct ice_vf *vf; diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethernet/intel/idpf/idpf_txrx.c index e6563ad31f5c..4a4e00c70d93 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c @@ -2392,7 +2392,7 @@ void idpf_tx_splitq_build_flow_desc(union idpf_tx_flex_desc *desc, struct idpf_tx_splitq_params *params, u16 td_cmd, u16 size) { - *(u32 *)&desc->flow.qw1.cmd_dtype = (u8)(params->dtype | td_cmd); + *(__le32 *)&desc->flow.qw1.cmd_dtype = cpu_to_le32((u8)(params->dtype | td_cmd)); desc->flow.qw1.rxr_bufsize = cpu_to_le16((u16)size); desc->flow.qw1.compl_tag = cpu_to_le16(params->compl_tag); } diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index 664c762622dc..a8e7f4895e29 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -3074,7 +3074,8 @@ static void igc_xdp_xmit_zc(struct igc_ring *ring) olinfo_status = xdp_desc.len << IGC_ADVTXD_PAYLEN_SHIFT; dma = xsk_buff_raw_get_dma(pool, xdp_desc.addr); - meta = xsk_buff_get_metadata(pool, xdp_desc.addr); + meta = xsk_buff_get_metadata(pool, xdp_desc.addr, + xdp_desc.options); xsk_buff_raw_dma_sync_for_device(pool, dma, xdp_desc.len); bi = &ring->tx_buffer_info[ntu]; diff --git a/drivers/net/ethernet/marvell/octeontx2/af/mcs_rvu_if.c b/drivers/net/ethernet/marvell/octeontx2/af/mcs_rvu_if.c index a80c8e7c94f2..d6c69ea7f10d 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/mcs_rvu_if.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/mcs_rvu_if.c @@ -856,7 +856,7 @@ int rvu_mbox_handler_mcs_ctrl_pkt_rule_write(struct rvu *rvu, static void rvu_mcs_set_lmac_bmap(struct rvu *rvu) { struct mcs *mcs = mcs_get_pdata(0); - unsigned long lmac_bmap; + unsigned long lmac_bmap = 0; int cgx, lmac, port; for (port = 0; port < mcs->hw->lmac_cnt; port++) { diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_debugfs.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_debugfs.c index beb514ebda02..91a9ad0d37c0 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_debugfs.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_debugfs.c @@ -1689,6 +1689,12 @@ static int rvu_dbg_nix_tm_tree_display(struct seq_file *m, void *unused) return -EINVAL; pfvf = rvu_get_pfvf(rvu, pcifunc); + + if (!pfvf->sq_ctx) { + seq_printf(m, "SQ context is not initialized for pcifunc 0x%x\n", pcifunc); + return -EINVAL; + } + max_id = pfvf->sq_ctx->qsize; memset(&aq_req, 0, sizeof(struct nix_aq_enq_req)); diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c index 7de06eb8284a..adedca065210 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c @@ -2436,8 +2436,8 @@ static int nix_smq_flush(struct rvu *rvu, int blkaddr, int pf = rvu_get_pf(rvu->pdev, pcifunc); u8 cgx_id = 0, lmac_id = 0; u16 tl2_tl3_link_schq; - u8 link, link_level; u64 cfg, bmap = 0; + u8 link_level; if (!is_rvu_otx2(rvu)) { /* Skip SMQ flush if pkt count is zero */ @@ -2469,7 +2469,6 @@ static int nix_smq_flush(struct rvu *rvu, int blkaddr, link_level = rvu_read64(rvu, blkaddr, NIX_AF_PSE_CHANNEL_LEVEL) & 0x01 ? NIX_TXSCH_LVL_TL3 : NIX_TXSCH_LVL_TL2; tl2_tl3_link_schq = smq_flush_ctx->smq_tree_ctx[link_level].schq; - link = smq_flush_ctx->smq_tree_ctx[NIX_TXSCH_LVL_TL1].schq; /* SMQ set enqueue xoff */ cfg = rvu_read64(rvu, blkaddr, NIX_AF_SMQX_CFG(smq)); @@ -2479,13 +2478,13 @@ static int nix_smq_flush(struct rvu *rvu, int blkaddr, /* Clear all NIX_AF_TL3_TL2_LINK_CFG[ENA] for the TL3/TL2 queue */ for (i = 0; i < (rvu->hw->cgx_links + rvu->hw->lbk_links); i++) { cfg = rvu_read64(rvu, blkaddr, - NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, link)); + NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, i)); if (!(cfg & BIT_ULL(12))) continue; bmap |= BIT_ULL(i); cfg &= ~BIT_ULL(12); rvu_write64(rvu, blkaddr, - NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, link), cfg); + NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, i), cfg); } /* Do SMQ flush and set enqueue xoff */ @@ -2506,10 +2505,10 @@ static int nix_smq_flush(struct rvu *rvu, int blkaddr, if (!(bmap & BIT_ULL(i))) continue; cfg = rvu_read64(rvu, blkaddr, - NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, link)); + NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, i)); cfg |= BIT_ULL(12); rvu_write64(rvu, blkaddr, - NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, link), cfg); + NIX_AF_TL3_TL2X_LINKX_CFG(tl2_tl3_link_schq, i), cfg); } /* clear XOFF on TL2s */ diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c index 062bc899d955..83a51de89154 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c @@ -333,7 +333,8 @@ int otx2_set_rss_table(struct otx2_nic *pfvf, int ctx_id, const u32 *ind_tbl) /* Get memory to put this msg */ for (idx = 0; idx < rss->rss_size; idx++) { /* Ignore the queue if AF_XDP zero copy is enabled */ - if (test_bit(ind_tbl[idx], pfvf->af_xdp_zc_qidx)) + if (pfvf->af_xdp_zc_qidx && + test_bit(ind_tbl[idx], pfvf->af_xdp_zc_qidx)) continue; aq = otx2_mbox_alloc_msg_nix_aq_enq(mbox); @@ -1496,7 +1497,8 @@ int otx2_pool_init(struct otx2_nic *pfvf, u16 pool_id, if (type != AURA_NIX_RQ) return 0; - if (!test_bit(pool_id, pfvf->af_xdp_zc_qidx)) { + if (!pfvf->af_xdp_zc_qidx || + !test_bit(pool_id, pfvf->af_xdp_zc_qidx)) { pp_params.order = get_order(buf_size); pp_params.flags = PP_FLAG_DMA_MAP; pp_params.pool_size = min(OTX2_PAGE_POOL_SZ, numptrs); diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_ethtool.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_ethtool.c index b6449f0a9e7d..95081cfb5338 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_ethtool.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_ethtool.c @@ -934,7 +934,8 @@ static int otx2_get_rxfh(struct net_device *dev, for (idx = 0; idx < rss->rss_size; idx++) { /* Ignore if the rx queue is AF_XDP zero copy enabled */ - if (test_bit(rss->ind_tbl[idx], pfvf->af_xdp_zc_qidx)) + if (pfvf->af_xdp_zc_qidx && + test_bit(rss->ind_tbl[idx], pfvf->af_xdp_zc_qidx)) continue; indir[idx] = rss->ind_tbl[idx]; } diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c index 5534c2c8db0f..67f01bb6073b 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c @@ -736,16 +736,16 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (err) goto err_ptp_destroy; + err = otx2_vf_wq_init(vf); + if (err) + goto err_ipsec_clean; + err = register_netdev(netdev); if (err) { dev_err(dev, "Failed to register netdevice\n"); - goto err_ipsec_clean; + goto err_wq_destroy; } - err = otx2_vf_wq_init(vf); - if (err) - goto err_unreg_netdev; - otx2vf_set_ethtool_ops(netdev); err = otx2vf_mcam_flow_init(vf); @@ -788,6 +788,10 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id) otx2_shutdown_tc(vf); err_unreg_netdev: unregister_netdev(netdev); +err_wq_destroy: + cancel_work_sync(&vf->reset_task); + cancel_work_sync(&vf->rx_mode_work); + destroy_workqueue(vf->otx2_wq); err_ipsec_clean: cn10k_ipsec_clean(vf); err_ptp_destroy: @@ -835,11 +839,13 @@ static void otx2vf_remove(struct pci_dev *pdev) } #endif - cancel_work_sync(&vf->reset_task); otx2_unregister_dl(vf); unregister_netdev(netdev); - if (vf->otx2_wq) + if (vf->otx2_wq) { + cancel_work_sync(&vf->reset_task); + cancel_work_sync(&vf->rx_mode_work); destroy_workqueue(vf->otx2_wq); + } cn10k_ipsec_clean(vf); otx2_ptp_destroy(vf); otx2_mcam_flow_del(vf); diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_xsk.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_xsk.c index 7d67b4cbaf71..0e8a6a6486c4 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_xsk.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_xsk.c @@ -193,7 +193,8 @@ int otx2_xsk_wakeup(struct net_device *dev, u32 queue_id, u32 flags) void otx2_attach_xsk_buff(struct otx2_nic *pfvf, struct otx2_snd_queue *sq, int qidx) { - if (test_bit(qidx, pfvf->af_xdp_zc_qidx)) + if (pfvf->af_xdp_zc_qidx && + test_bit(qidx, pfvf->af_xdp_zc_qidx)) sq->xsk_pool = xsk_get_pool_from_qid(pfvf->netdev, qidx); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c index 827538d92ac2..a7e3fe275d14 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c @@ -97,7 +97,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) xdptxd.dma_addr = xsk_buff_raw_get_dma(pool, desc.addr); xdptxd.data = xsk_buff_raw_get_data(pool, desc.addr); xdptxd.len = desc.len; - meta = xsk_buff_get_metadata(pool, desc.addr); + meta = xsk_buff_get_metadata(pool, desc.addr, desc.options); xsk_buff_raw_dma_sync_for_device(pool, xdptxd.dma_addr, xdptxd.len); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c index 8b6544124624..b716f7cfe8af 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c @@ -687,11 +687,13 @@ static int mlx5e_rep_open(struct net_device *dev) if (err) goto unlock; + mutex_lock(&rep->esw->state_lock); if (!mlx5_modify_vport_admin_state(priv->mdev, MLX5_VPORT_STATE_OP_MOD_ESW_VPORT, rep->vport, 1, MLX5_VPORT_ADMIN_STATE_UP)) netif_carrier_on(dev); + mutex_unlock(&rep->esw->state_lock); unlock: mutex_unlock(&priv->state_lock); @@ -706,10 +708,12 @@ static int mlx5e_rep_close(struct net_device *dev) int ret; mutex_lock(&priv->state_lock); + mutex_lock(&rep->esw->state_lock); mlx5_modify_vport_admin_state(priv->mdev, MLX5_VPORT_STATE_OP_MOD_ESW_VPORT, rep->vport, 1, MLX5_VPORT_ADMIN_STATE_DOWN); + mutex_unlock(&rep->esw->state_lock); ret = mlx5e_close_locked(dev); mutex_unlock(&priv->state_lock); return ret; @@ -781,22 +785,25 @@ static int mlx5e_rep_change_carrier(struct net_device *dev, bool new_carrier) struct mlx5e_priv *priv = netdev_priv(dev); struct mlx5e_rep_priv *rpriv = priv->ppriv; struct mlx5_eswitch_rep *rep = rpriv->rep; - int err; + int err = 0; + mutex_lock(&rep->esw->state_lock); if (new_carrier) { err = mlx5_modify_vport_admin_state(priv->mdev, MLX5_VPORT_STATE_OP_MOD_ESW_VPORT, rep->vport, 1, MLX5_VPORT_ADMIN_STATE_UP); if (err) - return err; + goto unlock; netif_carrier_on(dev); } else { err = mlx5_modify_vport_admin_state(priv->mdev, MLX5_VPORT_STATE_OP_MOD_ESW_VPORT, rep->vport, 1, MLX5_VPORT_ADMIN_STATE_DOWN); if (err) - return err; + goto unlock; netif_carrier_off(dev); } - return 0; +unlock: + mutex_unlock(&rep->esw->state_lock); + return err; } static const struct net_device_ops mlx5e_netdev_ops_rep = { @@ -1335,9 +1342,12 @@ static void mlx5e_uplink_rep_enable(struct mlx5e_priv *priv) mlx5e_rep_tc_enable(priv); - if (MLX5_CAP_GEN(mdev, uplink_follow)) + if (MLX5_CAP_GEN(mdev, uplink_follow)) { + mutex_lock(&mdev->priv.eswitch->state_lock); mlx5_modify_vport_admin_state(mdev, MLX5_VPORT_STATE_OP_MOD_UPLINK, 0, 0, MLX5_VPORT_ADMIN_STATE_AUTO); + mutex_unlock(&mdev->priv.eswitch->state_lock); + } mlx5_lag_add_netdev(mdev, netdev); priv->events_nb.notifier_call = uplink_rep_async_event; mlx5_notifier_register(mdev, &priv->events_nb); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 2ffa4e6b8c37..17b74a97138b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -2014,11 +2014,14 @@ mlx5e_shampo_fill_skb_data(struct sk_buff *skb, struct mlx5e_rq *rq, struct mlx5e_frag_page *frag_page, u32 data_bcnt, u32 data_offset) { + u32 page_size = BIT(rq->mpwqe.page_shift); + net_prefetchw(skb->data); do { /* Non-linear mode, hence non-XSK, which always uses PAGE_SIZE. */ - u32 pg_consumed_bytes = min_t(u32, PAGE_SIZE - data_offset, data_bcnt); + u32 pg_consumed_bytes = min_t(u32, page_size - data_offset, + data_bcnt); unsigned int truesize = pg_consumed_bytes; mlx5e_add_skb_frag(rq, skb, frag_page, data_offset, @@ -2039,6 +2042,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w u16 headlen = min_t(u16, MLX5E_RX_MAX_HEAD, cqe_bcnt); struct mlx5e_frag_page *head_page = frag_page; struct mlx5e_xdp_buff *mxbuf = &rq->mxbuf; + u32 page_size = BIT(rq->mpwqe.page_shift); u32 frag_offset = head_offset; u32 byte_cnt = cqe_bcnt; struct skb_shared_info *sinfo; @@ -2084,9 +2088,9 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w linear_hr = skb_headroom(skb); linear_data_len = headlen; linear_frame_sz = MLX5_SKB_FRAG_SZ(skb_end_offset(skb)); - if (unlikely(frag_offset >= PAGE_SIZE)) { + if (unlikely(frag_offset >= page_size)) { frag_page++; - frag_offset -= PAGE_SIZE; + frag_offset -= page_size; } } @@ -2098,7 +2102,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w while (byte_cnt) { /* Non-linear mode, hence non-XSK, which always uses PAGE_SIZE. */ pg_consumed_bytes = - min_t(u32, PAGE_SIZE - frag_offset, byte_cnt); + min_t(u32, page_size - frag_offset, byte_cnt); if (test_bit(MLX5E_RQ_STATE_SHAMPO, &rq->state)) truesize += pg_consumed_bytes; @@ -2135,7 +2139,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w new_nr_frags = sinfo->nr_frags; nr_frags_free = old_nr_frags - new_nr_frags; if (unlikely(nr_frags_free)) - truesize -= (nr_frags_free - 1) * PAGE_SIZE + + truesize -= (nr_frags_free - 1) * page_size + ALIGN(pg_consumed_bytes, BIT(rq->mpwqe.log_stride_sz)); @@ -2349,15 +2353,16 @@ mlx5e_shampo_flush_skb(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, bool match) rq->hw_gro_data->skb = NULL; } -static bool -mlx5e_hw_gro_skb_has_enough_space(struct sk_buff *skb, u16 data_bcnt) +static bool mlx5e_hw_gro_skb_has_enough_space(struct sk_buff *skb, + u16 data_bcnt, + u32 page_size) { int nr_frags = skb_shinfo(skb)->nr_frags; - if (PAGE_SIZE >= GRO_LEGACY_MAX_SIZE) + if (page_size >= GRO_LEGACY_MAX_SIZE) return skb->len + data_bcnt <= GRO_LEGACY_MAX_SIZE; else - return PAGE_SIZE * nr_frags + data_bcnt <= GRO_LEGACY_MAX_SIZE; + return page_size * nr_frags + data_bcnt <= GRO_LEGACY_MAX_SIZE; } static void mlx5e_handle_rx_cqe_mpwrq_shampo(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe) @@ -2366,18 +2371,19 @@ static void mlx5e_handle_rx_cqe_mpwrq_shampo(struct mlx5e_rq *rq, struct mlx5_cq u16 header_index = mlx5e_shampo_get_cqe_header_index(rq, cqe); u32 wqe_offset = be32_to_cpu(cqe->shampo.data_offset); u16 cstrides = mpwrq_get_cqe_consumed_strides(cqe); - u32 data_offset = wqe_offset & (PAGE_SIZE - 1); u32 cqe_bcnt = mpwrq_get_cqe_byte_cnt(cqe); u16 wqe_id = be16_to_cpu(cqe->wqe_id); - u32 page_idx = wqe_offset >> PAGE_SHIFT; u16 head_size = cqe->shampo.header_size; struct sk_buff **skb = &rq->hw_gro_data->skb; bool flush = cqe->shampo.flush; bool match = cqe->shampo.match; + u32 page_size = BIT(rq->mpwqe.page_shift); struct mlx5e_rq_stats *stats = rq->stats; struct mlx5e_rx_wqe_ll *wqe; struct mlx5e_mpw_info *wi; struct mlx5_wq_ll *wq; + u32 data_offset; + u32 page_idx; wi = mlx5e_get_mpw_info(rq, wqe_id); wi->consumed_strides += cstrides; @@ -2393,7 +2399,16 @@ static void mlx5e_handle_rx_cqe_mpwrq_shampo(struct mlx5e_rq *rq, struct mlx5_cq goto mpwrq_cqe_out; } - if (*skb && (!match || !(mlx5e_hw_gro_skb_has_enough_space(*skb, data_bcnt)))) { + data_offset = wqe_offset & (page_size - 1); + page_idx = wqe_offset >> rq->mpwqe.page_shift; + if (unlikely(cqe_bcnt <= ETH_ZLEN + 2 * VLAN_HLEN)) { + match = false; + flush = true; + } + + if (*skb && + !(match && mlx5e_hw_gro_skb_has_enough_space(*skb, data_bcnt, + page_size))) { match = false; mlx5e_shampo_flush_skb(rq, cqe, match); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c index 76108299ea57..65758e721174 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c @@ -247,7 +247,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget) } if (unlikely(aff_change && busy_xsk)) { - mlx5e_trigger_irq(&c->icosq); + mlx5e_trigger_napi_icosq(c); ch_stats->force_irq++; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c index 0091ba697bae..8eaef73f3d77 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c @@ -4,16 +4,33 @@ #include "fs_core.h" #include "eswitch.h" -enum { - MLX5_ADJ_VPORT_DISCONNECT = 0x0, - MLX5_ADJ_VPORT_CONNECT = 0x1, -}; - -static int mlx5_esw_adj_vport_modify(struct mlx5_core_dev *dev, - u16 vport, bool connect) +int mlx5_esw_adj_vport_modify(struct mlx5_core_dev *dev, u16 vport, + bool connect) { u32 in[MLX5_ST_SZ_DW(modify_vport_state_in)] = {}; + lockdep_assert_held(&dev->priv.eswitch->state_lock); + + if (MLX5_CAP_ESW(dev, esw_vport_state_max_tx_speed)) { + u8 op_mod = MLX5_VPORT_STATE_OP_MOD_ESW_VPORT; + struct mlx5_vport *esw_vport; + u32 speed = 0; + int err; + + err = mlx5_query_vport_max_tx_speed(dev, op_mod, vport, + true, &speed, NULL); + if (err) { + esw_vport = mlx5_eswitch_get_vport(dev->priv.eswitch, + vport); + speed = IS_ERR(esw_vport) ? 0 : + esw_vport->agg_max_tx_speed; + mlx5_core_dbg(dev, + "Failed to query vport %d max tx speed, err=%d, using cached %u\n", + vport, err, speed); + } + MLX5_SET(modify_vport_state_in, in, max_tx_speed, speed); + } + MLX5_SET(modify_vport_state_in, in, opcode, MLX5_CMD_OP_MODIFY_VPORT_STATE); MLX5_SET(modify_vport_state_in, in, op_mod, @@ -24,7 +41,7 @@ static int mlx5_esw_adj_vport_modify(struct mlx5_core_dev *dev, MLX5_SET(modify_vport_state_in, in, egress_connect_valid, 1); MLX5_SET(modify_vport_state_in, in, ingress_connect, connect); MLX5_SET(modify_vport_state_in, in, egress_connect, connect); - + MLX5_SET(modify_vport_state_in, in, admin_state, connect); return mlx5_cmd_exec_in(dev, modify_vport_state, in); } @@ -96,7 +113,6 @@ static int mlx5_esw_adj_vport_create(struct mlx5_eswitch *esw, u16 vhca_id, if (err) goto acl_ns_remove; - mlx5_esw_adj_vport_modify(esw->dev, vport_num, MLX5_ADJ_VPORT_CONNECT); return 0; acl_ns_remove: @@ -117,8 +133,7 @@ static void mlx5_esw_adj_vport_destroy(struct mlx5_eswitch *esw, esw_debug(esw->dev, "Destroying adjacent vport %d for vhca_id 0x%x\n", vport_num, vport->vhca_id); - mlx5_esw_adj_vport_modify(esw->dev, vport_num, - MLX5_ADJ_VPORT_DISCONNECT); + mlx5_esw_offloads_rep_remove(esw, vport); mlx5_fs_vport_egress_acl_ns_remove(esw->dev->priv.steering, vport->index); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c index c38deabcb7b9..bc7ceb4c76e5 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c @@ -904,6 +904,24 @@ static void esw_vport_cleanup(struct mlx5_eswitch *esw, struct mlx5_vport *vport esw_vport_cleanup_acl(esw, vport); } +static void mlx5_esw_vport_set_max_tx_speed(struct mlx5_eswitch *esw, + struct mlx5_vport *vport) +{ + int ret; + + if (!MLX5_CAP_ESW(esw->dev, esw_vport_state_max_tx_speed)) + return; + + ret = mlx5_modify_vport_max_tx_speed(esw->dev, + MLX5_VPORT_STATE_OP_MOD_ESW_VPORT, + vport->vport, true, + vport->agg_max_tx_speed); + if (ret) + mlx5_core_dbg(esw->dev, + "Failed to set vport %d speed %d, err=%d\n", + vport->vport, vport->agg_max_tx_speed, ret); +} + int mlx5_esw_vport_enable(struct mlx5_eswitch *esw, struct mlx5_vport *vport, enum mlx5_eswitch_vport_event enabled_events) { @@ -950,6 +968,9 @@ int mlx5_esw_vport_enable(struct mlx5_eswitch *esw, struct mlx5_vport *vport, esw->enabled_vports++; esw_debug(esw->dev, "Enabled VPORT(%d)\n", vport_num); + + if (vport->agg_max_tx_speed) + mlx5_esw_vport_set_max_tx_speed(esw, vport); done: mutex_unlock(&esw->state_lock); return ret; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h index 558055d214e1..eca24c1929eb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h @@ -246,6 +246,7 @@ struct mlx5_vport { enum mlx5_eswitch_vport_event enabled_events; int index; struct mlx5_devlink_port *dl_port; + u32 agg_max_tx_speed; }; struct mlx5_esw_indir_table; @@ -264,6 +265,9 @@ struct mlx5_eswitch_fdb { struct offloads_fdb { struct mlx5_flow_namespace *ns; + struct mlx5_flow_table *drop_root; + struct mlx5_flow_handle *drop_root_rule; + struct mlx5_fc *drop_root_fc; struct mlx5_flow_table *tc_miss_table; struct mlx5_flow_table *slow_fdb; struct mlx5_flow_group *send_to_vport_grp; @@ -394,6 +398,7 @@ struct mlx5_eswitch { struct mlx5_esw_offload offloads; u32 last_vport_idx; int mode; + bool offloads_inactive; u16 manager_vport; u16 first_host_vport; u8 num_peers; @@ -636,6 +641,8 @@ const u32 *mlx5_esw_query_functions(struct mlx5_core_dev *dev); void mlx5_esw_adjacent_vhcas_setup(struct mlx5_eswitch *esw); void mlx5_esw_adjacent_vhcas_cleanup(struct mlx5_eswitch *esw); +int mlx5_esw_adj_vport_modify(struct mlx5_core_dev *dev, u16 vport, + bool connect); #define MLX5_DEBUG_ESWITCH_MASK BIT(3) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c index 8be0961cb6c7..d5fc07bab190 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c @@ -1575,6 +1575,7 @@ esw_chains_create(struct mlx5_eswitch *esw, struct mlx5_flow_table *miss_fdb) attr.max_grp_num = esw->params.large_group_num; attr.default_ft = miss_fdb; attr.mapping = esw->offloads.reg_c0_obj_pool; + attr.fs_base_prio = FDB_BYPASS_PATH; chains = mlx5_chains_create(dev, &attr); if (IS_ERR(chains)) { @@ -2353,6 +2354,135 @@ static void esw_mode_change(struct mlx5_eswitch *esw, u16 mode) mlx5_devcom_comp_unlock(esw->dev->priv.hca_devcom_comp); } +static void mlx5_esw_fdb_drop_destroy(struct mlx5_eswitch *esw) +{ + if (!esw->fdb_table.offloads.drop_root) + return; + + esw_debug(esw->dev, "Destroying FDB drop root table %#x fc %#x\n", + esw->fdb_table.offloads.drop_root->id, + esw->fdb_table.offloads.drop_root_fc->id); + mlx5_del_flow_rules(esw->fdb_table.offloads.drop_root_rule); + /* Don't free flow counter here, can be reused on a later activation */ + mlx5_destroy_flow_table(esw->fdb_table.offloads.drop_root); + esw->fdb_table.offloads.drop_root_rule = NULL; + esw->fdb_table.offloads.drop_root = NULL; +} + +static int mlx5_esw_fdb_drop_create(struct mlx5_eswitch *esw) +{ + struct mlx5_flow_destination drop_fc_dst = {}; + struct mlx5_flow_table_attr ft_attr = {}; + struct mlx5_flow_destination *dst = NULL; + struct mlx5_core_dev *dev = esw->dev; + struct mlx5_flow_namespace *root_ns; + struct mlx5_flow_act flow_act = {}; + struct mlx5_flow_handle *flow_rule; + struct mlx5_flow_table *table; + int err = 0, dst_num = 0; + + if (esw->fdb_table.offloads.drop_root) + return 0; + + root_ns = esw->fdb_table.offloads.ns; + + ft_attr.prio = FDB_DROP_ROOT; + ft_attr.max_fte = 1; + ft_attr.autogroup.max_num_groups = 1; + table = mlx5_create_auto_grouped_flow_table(root_ns, &ft_attr); + if (IS_ERR(table)) { + esw_warn(dev, "Failed to create fdb drop root table, err %pe\n", + table); + return PTR_ERR(table); + } + + /* Drop FC reusable, create once on first deactivation of FDB */ + if (!esw->fdb_table.offloads.drop_root_fc) { + struct mlx5_fc *counter = mlx5_fc_create(dev, 0); + + err = PTR_ERR_OR_ZERO(counter); + if (err) + esw_warn(esw->dev, "create fdb drop fc err %d\n", err); + else + esw->fdb_table.offloads.drop_root_fc = counter; + } + + flow_act.action = MLX5_FLOW_CONTEXT_ACTION_DROP; + + if (esw->fdb_table.offloads.drop_root_fc) { + flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_COUNT; + drop_fc_dst.type = MLX5_FLOW_DESTINATION_TYPE_COUNTER; + drop_fc_dst.counter = esw->fdb_table.offloads.drop_root_fc; + dst = &drop_fc_dst; + dst_num++; + } + + flow_rule = mlx5_add_flow_rules(table, NULL, &flow_act, dst, dst_num); + err = PTR_ERR_OR_ZERO(flow_rule); + if (err) { + esw_warn(esw->dev, + "fs offloads: Failed to add vport rx drop rule err %d\n", + err); + goto err_flow_rule; + } + + esw->fdb_table.offloads.drop_root = table; + esw->fdb_table.offloads.drop_root_rule = flow_rule; + esw_debug(esw->dev, "Created FDB drop root table %#x fc %#x\n", + table->id, dst ? dst->counter->id : 0); + return 0; + +err_flow_rule: + /* no need to free drop fc, esw_offloads_steering_cleanup will do it */ + mlx5_destroy_flow_table(table); + return err; +} + +static void mlx5_esw_fdb_active(struct mlx5_eswitch *esw) +{ + struct mlx5_vport *vport; + unsigned long i; + + mlx5_esw_fdb_drop_destroy(esw); + mlx5_mpfs_enable(esw->dev); + + mutex_lock(&esw->state_lock); + mlx5_esw_for_each_vf_vport(esw, i, vport, U16_MAX) { + if (!vport->adjacent) + continue; + esw_debug(esw->dev, "Connecting vport %d to eswitch\n", + vport->vport); + mlx5_esw_adj_vport_modify(esw->dev, vport->vport, true); + } + mutex_unlock(&esw->state_lock); + + esw->offloads_inactive = false; + esw_warn(esw->dev, "MPFS/FDB active\n"); +} + +static void mlx5_esw_fdb_inactive(struct mlx5_eswitch *esw) +{ + struct mlx5_vport *vport; + unsigned long i; + + mlx5_mpfs_disable(esw->dev); + mlx5_esw_fdb_drop_create(esw); + + mutex_lock(&esw->state_lock); + mlx5_esw_for_each_vf_vport(esw, i, vport, U16_MAX) { + if (!vport->adjacent) + continue; + esw_debug(esw->dev, "Disconnecting vport %u from eswitch\n", + vport->vport); + + mlx5_esw_adj_vport_modify(esw->dev, vport->vport, false); + } + mutex_unlock(&esw->state_lock); + + esw->offloads_inactive = true; + esw_warn(esw->dev, "MPFS/FDB inactive\n"); +} + static int esw_offloads_start(struct mlx5_eswitch *esw, struct netlink_ext_ack *extack) { @@ -3436,6 +3566,10 @@ static int esw_offloads_steering_init(struct mlx5_eswitch *esw) static void esw_offloads_steering_cleanup(struct mlx5_eswitch *esw) { + mlx5_esw_fdb_drop_destroy(esw); + if (esw->fdb_table.offloads.drop_root_fc) + mlx5_fc_destroy(esw->dev, esw->fdb_table.offloads.drop_root_fc); + esw->fdb_table.offloads.drop_root_fc = NULL; esw_destroy_vport_rx_drop_rule(esw); esw_destroy_vport_rx_drop_group(esw); esw_destroy_vport_rx_group(esw); @@ -3604,6 +3738,11 @@ int esw_offloads_enable(struct mlx5_eswitch *esw) if (err) goto err_steering_init; + if (esw->offloads_inactive) + mlx5_esw_fdb_inactive(esw); + else + mlx5_esw_fdb_active(esw); + /* Representor will control the vport link state */ mlx5_esw_for_each_vf_vport(esw, i, vport, esw->esw_funcs.num_vfs) vport->info.link_state = MLX5_VPORT_ADMIN_STATE_DOWN; @@ -3672,6 +3811,9 @@ void esw_offloads_disable(struct mlx5_eswitch *esw) esw_offloads_metadata_uninit(esw); mlx5_rdma_disable_roce(esw->dev); mlx5_esw_adjacent_vhcas_cleanup(esw); + /* must be done after vhcas cleanup to avoid adjacent vports connect */ + if (esw->offloads_inactive) + mlx5_esw_fdb_active(esw); /* legacy mode always active */ mutex_destroy(&esw->offloads.termtbl_mutex); } @@ -3682,6 +3824,7 @@ static int esw_mode_from_devlink(u16 mode, u16 *mlx5_mode) *mlx5_mode = MLX5_ESWITCH_LEGACY; break; case DEVLINK_ESWITCH_MODE_SWITCHDEV: + case DEVLINK_ESWITCH_MODE_SWITCHDEV_INACTIVE: *mlx5_mode = MLX5_ESWITCH_OFFLOADS; break; default: @@ -3691,14 +3834,17 @@ static int esw_mode_from_devlink(u16 mode, u16 *mlx5_mode) return 0; } -static int esw_mode_to_devlink(u16 mlx5_mode, u16 *mode) +static int esw_mode_to_devlink(struct mlx5_eswitch *esw, u16 *mode) { - switch (mlx5_mode) { + switch (esw->mode) { case MLX5_ESWITCH_LEGACY: *mode = DEVLINK_ESWITCH_MODE_LEGACY; break; case MLX5_ESWITCH_OFFLOADS: - *mode = DEVLINK_ESWITCH_MODE_SWITCHDEV; + if (esw->offloads_inactive) + *mode = DEVLINK_ESWITCH_MODE_SWITCHDEV_INACTIVE; + else + *mode = DEVLINK_ESWITCH_MODE_SWITCHDEV; break; default: return -EINVAL; @@ -3804,6 +3950,45 @@ static bool mlx5_devlink_netdev_netns_immutable_set(struct devlink *devlink, return ret; } +/* Returns true when only changing between active and inactive switchdev mode */ +static bool mlx5_devlink_switchdev_active_mode_change(struct mlx5_eswitch *esw, + u16 devlink_mode) +{ + /* current mode is not switchdev */ + if (esw->mode != MLX5_ESWITCH_OFFLOADS) + return false; + + /* new mode is not switchdev */ + if (devlink_mode != DEVLINK_ESWITCH_MODE_SWITCHDEV && + devlink_mode != DEVLINK_ESWITCH_MODE_SWITCHDEV_INACTIVE) + return false; + + /* already inactive: no change in current state */ + if (devlink_mode == DEVLINK_ESWITCH_MODE_SWITCHDEV_INACTIVE && + esw->offloads_inactive) + return false; + + /* already active: no change in current state */ + if (devlink_mode == DEVLINK_ESWITCH_MODE_SWITCHDEV && + !esw->offloads_inactive) + return false; + + down_write(&esw->mode_lock); + esw->offloads_inactive = !esw->offloads_inactive; + esw->eswitch_operation_in_progress = true; + up_write(&esw->mode_lock); + + if (esw->offloads_inactive) + mlx5_esw_fdb_inactive(esw); + else + mlx5_esw_fdb_active(esw); + + down_write(&esw->mode_lock); + esw->eswitch_operation_in_progress = false; + up_write(&esw->mode_lock); + return true; +} + int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode, struct netlink_ext_ack *extack) { @@ -3823,12 +4008,16 @@ int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode, if (esw_mode_from_devlink(mode, &mlx5_mode)) return -EINVAL; - if (mode == DEVLINK_ESWITCH_MODE_SWITCHDEV && mlx5_get_sd(esw->dev)) { + if (mlx5_mode == MLX5_ESWITCH_OFFLOADS && mlx5_get_sd(esw->dev)) { NL_SET_ERR_MSG_MOD(extack, "Can't change E-Switch mode to switchdev when multi-PF netdev (Socket Direct) is configured."); return -EPERM; } + /* Avoid try_lock, active/inactive mode change is not restricted */ + if (mlx5_devlink_switchdev_active_mode_change(esw, mode)) + return 0; + mlx5_lag_disable_change(esw->dev); err = mlx5_esw_try_lock(esw); if (err < 0) { @@ -3851,7 +4040,7 @@ int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode, esw->eswitch_operation_in_progress = true; up_write(&esw->mode_lock); - if (mode == DEVLINK_ESWITCH_MODE_SWITCHDEV && + if (mlx5_mode == MLX5_ESWITCH_OFFLOADS && !mlx5_devlink_netdev_netns_immutable_set(devlink, true)) { NL_SET_ERR_MSG_MOD(extack, "Can't change E-Switch mode to switchdev when netdev net namespace has diverged from the devlink's."); @@ -3859,27 +4048,29 @@ int mlx5_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode, goto skip; } - if (mode == DEVLINK_ESWITCH_MODE_LEGACY) + if (mlx5_mode == MLX5_ESWITCH_LEGACY) esw->dev->priv.flags |= MLX5_PRIV_FLAGS_SWITCH_LEGACY; if (mlx5_mode == MLX5_ESWITCH_OFFLOADS) esw->dev->priv.flags &= ~MLX5_PRIV_FLAGS_SWITCH_LEGACY; mlx5_eswitch_disable_locked(esw); - if (mode == DEVLINK_ESWITCH_MODE_SWITCHDEV) { + if (mlx5_mode == MLX5_ESWITCH_OFFLOADS) { if (mlx5_devlink_trap_get_num_active(esw->dev)) { NL_SET_ERR_MSG_MOD(extack, "Can't change mode while devlink traps are active"); err = -EOPNOTSUPP; goto skip; } + esw->offloads_inactive = + (mode == DEVLINK_ESWITCH_MODE_SWITCHDEV_INACTIVE); err = esw_offloads_start(esw, extack); - } else if (mode == DEVLINK_ESWITCH_MODE_LEGACY) { + } else if (mlx5_mode == MLX5_ESWITCH_LEGACY) { err = esw_offloads_stop(esw, extack); } else { err = -EINVAL; } skip: - if (mode == DEVLINK_ESWITCH_MODE_SWITCHDEV && err) + if (mlx5_mode == MLX5_ESWITCH_OFFLOADS && err) mlx5_devlink_netdev_netns_immutable_set(devlink, false); down_write(&esw->mode_lock); esw->eswitch_operation_in_progress = false; @@ -3898,7 +4089,7 @@ int mlx5_devlink_eswitch_mode_get(struct devlink *devlink, u16 *mode) if (IS_ERR(esw)) return PTR_ERR(esw); - return esw_mode_to_devlink(esw->mode, mode); + return esw_mode_to_devlink(esw, mode); } static int mlx5_esw_vports_inline_set(struct mlx5_eswitch *esw, u8 mlx5_mode, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c index 2db3ffb0a2b2..2ca3bddbdf05 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c @@ -3520,6 +3520,11 @@ static int init_fdb_root_ns(struct mlx5_flow_steering *steering) if (!steering->fdb_root_ns) return -ENOMEM; + maj_prio = fs_create_prio(&steering->fdb_root_ns->ns, FDB_DROP_ROOT, 1); + err = PTR_ERR_OR_ZERO(maj_prio); + if (err) + goto out_err; + err = create_fdb_bypass(steering); if (err) goto out_err; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c index 2d6019a32669..e9ad59f74e46 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c @@ -233,14 +233,25 @@ static void mlx5_ldev_free(struct kref *ref) { struct mlx5_lag *ldev = container_of(ref, struct mlx5_lag, ref); struct net *net; + int i; if (ldev->nb.notifier_call) { net = read_pnet(&ldev->net); unregister_netdevice_notifier_net(net, &ldev->nb); } + mlx5_ldev_for_each(i, 0, ldev) { + if (ldev->pf[i].dev && + ldev->pf[i].port_change_nb.nb.notifier_call) { + struct mlx5_nb *nb = &ldev->pf[i].port_change_nb; + + mlx5_eq_notifier_unregister(ldev->pf[i].dev, nb); + } + } + mlx5_lag_mp_cleanup(ldev); cancel_delayed_work_sync(&ldev->bond_work); + cancel_work_sync(&ldev->speed_update_work); destroy_workqueue(ldev->wq); mutex_destroy(&ldev->lock); kfree(ldev); @@ -274,6 +285,7 @@ static struct mlx5_lag *mlx5_lag_dev_alloc(struct mlx5_core_dev *dev) kref_init(&ldev->ref); mutex_init(&ldev->lock); INIT_DELAYED_WORK(&ldev->bond_work, mlx5_do_bond_work); + INIT_WORK(&ldev->speed_update_work, mlx5_mpesw_speed_update_work); ldev->nb.notifier_call = mlx5_lag_netdev_event; write_pnet(&ldev->net, mlx5_core_net(dev)); @@ -996,6 +1008,144 @@ static bool mlx5_lag_should_disable_lag(struct mlx5_lag *ldev, bool do_bond) ldev->mode != MLX5_LAG_MODE_MPESW; } +#ifdef CONFIG_MLX5_ESWITCH +static int +mlx5_lag_sum_devices_speed(struct mlx5_lag *ldev, u32 *sum_speed, + int (*get_speed)(struct mlx5_core_dev *, u32 *)) +{ + struct mlx5_core_dev *pf_mdev; + int pf_idx; + u32 speed; + int ret; + + *sum_speed = 0; + mlx5_ldev_for_each(pf_idx, 0, ldev) { + pf_mdev = ldev->pf[pf_idx].dev; + if (!pf_mdev) + continue; + + ret = get_speed(pf_mdev, &speed); + if (ret) { + mlx5_core_dbg(pf_mdev, + "Failed to get device speed using %ps. Device %s speed is not available (err=%d)\n", + get_speed, dev_name(pf_mdev->device), + ret); + return ret; + } + + *sum_speed += speed; + } + + return 0; +} + +static int mlx5_lag_sum_devices_max_speed(struct mlx5_lag *ldev, u32 *max_speed) +{ + return mlx5_lag_sum_devices_speed(ldev, max_speed, + mlx5_port_max_linkspeed); +} + +static int mlx5_lag_sum_devices_oper_speed(struct mlx5_lag *ldev, + u32 *oper_speed) +{ + return mlx5_lag_sum_devices_speed(ldev, oper_speed, + mlx5_port_oper_linkspeed); +} + +static void mlx5_lag_modify_device_vports_speed(struct mlx5_core_dev *mdev, + u32 speed) +{ + u16 op_mod = MLX5_VPORT_STATE_OP_MOD_ESW_VPORT; + struct mlx5_eswitch *esw = mdev->priv.eswitch; + struct mlx5_vport *vport; + unsigned long i; + int ret; + + if (!esw) + return; + + if (!MLX5_CAP_ESW(mdev, esw_vport_state_max_tx_speed)) + return; + + mutex_lock(&esw->state_lock); + mlx5_esw_for_each_vport(esw, i, vport) { + if (!vport) + continue; + + if (vport->vport == MLX5_VPORT_UPLINK) + continue; + + vport->agg_max_tx_speed = speed; + + if (!vport->enabled) + continue; + + ret = mlx5_modify_vport_max_tx_speed(mdev, op_mod, + vport->vport, true, speed); + if (ret) + mlx5_core_dbg(mdev, + "Failed to set vport %d speed %d, err=%d\n", + vport->vport, speed, ret); + } + mutex_unlock(&esw->state_lock); +} + +void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev) +{ + struct mlx5_core_dev *mdev; + u32 speed; + int pf_idx; + + if (ldev->mode == MLX5_LAG_MODE_MPESW) { + if (mlx5_lag_sum_devices_oper_speed(ldev, &speed)) + return; + } else { + speed = ldev->tracker.bond_speed_mbps; + if (speed == SPEED_UNKNOWN) + return; + } + + /* If speed is not set, use the sum of max speeds of all PFs */ + if (!speed && mlx5_lag_sum_devices_max_speed(ldev, &speed)) + return; + + speed = speed / MLX5_MAX_TX_SPEED_UNIT; + + mlx5_ldev_for_each(pf_idx, 0, ldev) { + mdev = ldev->pf[pf_idx].dev; + if (!mdev) + continue; + + mlx5_lag_modify_device_vports_speed(mdev, speed); + } +} + +void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev) +{ + struct mlx5_core_dev *mdev; + u32 speed; + int pf_idx; + int ret; + + mlx5_ldev_for_each(pf_idx, 0, ldev) { + mdev = ldev->pf[pf_idx].dev; + if (!mdev) + continue; + + ret = mlx5_port_oper_linkspeed(mdev, &speed); + if (ret) { + mlx5_core_dbg(mdev, + "Failed to reset vports speed for device %s. Oper speed is not available (err=%d)\n", + dev_name(mdev->device), ret); + continue; + } + + speed = speed / MLX5_MAX_TX_SPEED_UNIT; + mlx5_lag_modify_device_vports_speed(mdev, speed); + } +} +#endif + static void mlx5_do_bond(struct mlx5_lag *ldev) { int idx = mlx5_lag_get_dev_index_by_seq(ldev, MLX5_LAG_P1); @@ -1083,9 +1233,12 @@ static void mlx5_do_bond(struct mlx5_lag *ldev) ndev); dev_put(ndev); } + mlx5_lag_set_vports_agg_speed(ldev); } else if (mlx5_lag_should_modify_lag(ldev, do_bond)) { mlx5_modify_lag(ldev, &tracker); + mlx5_lag_set_vports_agg_speed(ldev); } else if (mlx5_lag_should_disable_lag(ldev, do_bond)) { + mlx5_lag_reset_vports_speed(ldev); mlx5_disable_lag(ldev); } } @@ -1286,6 +1439,65 @@ static int mlx5_handle_changeinfodata_event(struct mlx5_lag *ldev, return 1; } +static void mlx5_lag_update_tracker_speed(struct lag_tracker *tracker, + struct net_device *ndev) +{ + struct ethtool_link_ksettings lksettings; + struct net_device *bond_dev; + int err; + + if (netif_is_lag_master(ndev)) + bond_dev = ndev; + else + bond_dev = netdev_master_upper_dev_get(ndev); + + if (!bond_dev) { + tracker->bond_speed_mbps = SPEED_UNKNOWN; + return; + } + + err = __ethtool_get_link_ksettings(bond_dev, &lksettings); + if (err) { + netdev_dbg(bond_dev, + "Failed to get speed for bond dev %s, err=%d\n", + bond_dev->name, err); + tracker->bond_speed_mbps = SPEED_UNKNOWN; + return; + } + + if (lksettings.base.speed == SPEED_UNKNOWN) + tracker->bond_speed_mbps = 0; + else + tracker->bond_speed_mbps = lksettings.base.speed; +} + +/* Returns speed in Mbps. */ +int mlx5_lag_query_bond_speed(struct mlx5_core_dev *mdev, u32 *speed) +{ + struct mlx5_lag *ldev; + unsigned long flags; + int ret = 0; + + spin_lock_irqsave(&lag_lock, flags); + ldev = mlx5_lag_dev(mdev); + if (!ldev) { + ret = -ENODEV; + goto unlock; + } + + *speed = ldev->tracker.bond_speed_mbps; + + if (*speed == SPEED_UNKNOWN) { + mlx5_core_dbg(mdev, "Bond speed is unknown\n"); + ret = -EINVAL; + } + +unlock: + spin_unlock_irqrestore(&lag_lock, flags); + return ret; +} +EXPORT_SYMBOL_GPL(mlx5_lag_query_bond_speed); + /* this handler is always registered to netdev events */ static int mlx5_lag_netdev_event(struct notifier_block *this, unsigned long event, void *ptr) @@ -1317,6 +1529,9 @@ static int mlx5_lag_netdev_event(struct notifier_block *this, break; } + if (changed) + mlx5_lag_update_tracker_speed(&tracker, ndev); + ldev->tracker = tracker; if (changed) @@ -1362,6 +1577,10 @@ static void mlx5_ldev_add_mdev(struct mlx5_lag *ldev, ldev->pf[fn].dev = dev; dev->priv.lag = ldev; + + MLX5_NB_INIT(&ldev->pf[fn].port_change_nb, + mlx5_lag_mpesw_port_change_event, PORT_CHANGE); + mlx5_eq_notifier_register(dev, &ldev->pf[fn].port_change_nb); } static void mlx5_ldev_remove_mdev(struct mlx5_lag *ldev, @@ -1373,6 +1592,9 @@ static void mlx5_ldev_remove_mdev(struct mlx5_lag *ldev, if (ldev->pf[fn].dev != dev) return; + if (ldev->pf[fn].port_change_nb.nb.notifier_call) + mlx5_eq_notifier_unregister(dev, &ldev->pf[fn].port_change_nb); + ldev->pf[fn].dev = NULL; dev->priv.lag = NULL; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h index 4918eee2b3da..be1afece5fdc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.h @@ -39,6 +39,7 @@ struct lag_func { struct mlx5_core_dev *dev; struct net_device *netdev; bool has_drop; + struct mlx5_nb port_change_nb; }; /* Used for collection of netdev event info. */ @@ -48,6 +49,7 @@ struct lag_tracker { unsigned int is_bonded:1; unsigned int has_inactive:1; enum netdev_lag_hash hash_type; + u32 bond_speed_mbps; }; /* LAG data of a ConnectX card. @@ -66,6 +68,7 @@ struct mlx5_lag { struct lag_tracker tracker; struct workqueue_struct *wq; struct delayed_work bond_work; + struct work_struct speed_update_work; struct notifier_block nb; possible_net_t net; struct lag_mp lag_mp; @@ -116,6 +119,14 @@ int mlx5_deactivate_lag(struct mlx5_lag *ldev); void mlx5_lag_add_devices(struct mlx5_lag *ldev); struct mlx5_devcom_comp_dev *mlx5_lag_get_devcom_comp(struct mlx5_lag *ldev); +#ifdef CONFIG_MLX5_ESWITCH +void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev); +void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev); +#else +static inline void mlx5_lag_set_vports_agg_speed(struct mlx5_lag *ldev) {} +static inline void mlx5_lag_reset_vports_speed(struct mlx5_lag *ldev) {} +#endif + static inline bool mlx5_lag_is_supported(struct mlx5_core_dev *dev) { if (!MLX5_CAP_GEN(dev, vport_group_manager) || diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.c index fc914ee25986..567e51f0e5da 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.c @@ -110,6 +110,8 @@ static int mlx5_lag_enable_mpesw(struct mlx5_lag *ldev) goto err_rescan_drivers; } + mlx5_lag_set_vports_agg_speed(ldev); + return 0; err_rescan_drivers: @@ -226,3 +228,40 @@ bool mlx5_lag_is_mpesw(struct mlx5_core_dev *dev) return ldev && ldev->mode == MLX5_LAG_MODE_MPESW; } EXPORT_SYMBOL(mlx5_lag_is_mpesw); + +void mlx5_mpesw_speed_update_work(struct work_struct *work) +{ + struct mlx5_lag *ldev = container_of(work, struct mlx5_lag, + speed_update_work); + + mutex_lock(&ldev->lock); + if (ldev->mode == MLX5_LAG_MODE_MPESW) { + if (ldev->mode_changes_in_progress) + queue_work(ldev->wq, &ldev->speed_update_work); + else + mlx5_lag_set_vports_agg_speed(ldev); + } + + mutex_unlock(&ldev->lock); +} + +int mlx5_lag_mpesw_port_change_event(struct notifier_block *nb, + unsigned long event, void *data) +{ + struct mlx5_nb *mlx5_nb = container_of(nb, struct mlx5_nb, nb); + struct lag_func *lag_func = container_of(mlx5_nb, + struct lag_func, + port_change_nb); + struct mlx5_core_dev *dev = lag_func->dev; + struct mlx5_lag *ldev = dev->priv.lag; + struct mlx5_eqe *eqe = data; + + if (!ldev) + return NOTIFY_DONE; + + if (eqe->sub_type == MLX5_PORT_CHANGE_SUBTYPE_DOWN || + eqe->sub_type == MLX5_PORT_CHANGE_SUBTYPE_ACTIVE) + queue_work(ldev->wq, &ldev->speed_update_work); + + return NOTIFY_OK; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.h b/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.h index 46de93ed790d..b767dbb4f457 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/mpesw.h @@ -37,4 +37,18 @@ void mlx5_lag_disable_mpesw(struct mlx5_lag *ldev); static inline void mlx5_lag_disable_mpesw(struct mlx5_lag *ldev) {} #endif /* CONFIG_MLX5_ESWITCH */ +#ifdef CONFIG_MLX5_ESWITCH +void mlx5_mpesw_speed_update_work(struct work_struct *work); +int mlx5_lag_mpesw_port_change_event(struct notifier_block *nb, + unsigned long event, void *data); +#else +static inline void mlx5_mpesw_speed_update_work(struct work_struct *work) {} +static inline int mlx5_lag_mpesw_port_change_event(struct notifier_block *nb, + unsigned long event, + void *data) +{ + return NOTIFY_DONE; +} +#endif /* CONFIG_MLX5_ESWITCH */ + #endif /* __MLX5_LAG_MPESW_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.c index 4450091e181a..4a88a42ae4f7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.c @@ -65,13 +65,14 @@ static int del_l2table_entry_cmd(struct mlx5_core_dev *dev, u32 index) /* UC L2 table hash node */ struct l2table_node { struct l2addr_node node; - u32 index; /* index in HW l2 table */ + int index; /* index in HW l2 table */ int ref_count; }; struct mlx5_mpfs { struct hlist_head hash[MLX5_L2_ADDR_HASH_SIZE]; struct mutex lock; /* Synchronize l2 table access */ + bool enabled; u32 size; unsigned long *bitmap; }; @@ -114,6 +115,8 @@ int mlx5_mpfs_init(struct mlx5_core_dev *dev) return -ENOMEM; } + mpfs->enabled = true; + dev->priv.mpfs = mpfs; return 0; } @@ -135,7 +138,7 @@ int mlx5_mpfs_add_mac(struct mlx5_core_dev *dev, u8 *mac) struct mlx5_mpfs *mpfs = dev->priv.mpfs; struct l2table_node *l2addr; int err = 0; - u32 index; + int index; if (!mpfs) return 0; @@ -148,30 +151,34 @@ int mlx5_mpfs_add_mac(struct mlx5_core_dev *dev, u8 *mac) goto out; } - err = alloc_l2table_index(mpfs, &index); - if (err) - goto out; - l2addr = l2addr_hash_add(mpfs->hash, mac, struct l2table_node, GFP_KERNEL); if (!l2addr) { err = -ENOMEM; - goto hash_add_err; + goto out; } - err = set_l2table_entry_cmd(dev, index, mac); - if (err) - goto set_table_entry_err; + index = -1; + + if (mpfs->enabled) { + err = alloc_l2table_index(mpfs, &index); + if (err) + goto hash_del; + err = set_l2table_entry_cmd(dev, index, mac); + if (err) + goto free_l2table_index; + mlx5_core_dbg(dev, "MPFS entry %pM, set @index (%d)\n", + l2addr->node.addr, index); + } l2addr->index = index; l2addr->ref_count = 1; mlx5_core_dbg(dev, "MPFS mac added %pM, index (%d)\n", mac, index); goto out; - -set_table_entry_err: - l2addr_hash_del(l2addr); -hash_add_err: +free_l2table_index: free_l2table_index(mpfs, index); +hash_del: + l2addr_hash_del(l2addr); out: mutex_unlock(&mpfs->lock); return err; @@ -183,7 +190,7 @@ int mlx5_mpfs_del_mac(struct mlx5_core_dev *dev, u8 *mac) struct mlx5_mpfs *mpfs = dev->priv.mpfs; struct l2table_node *l2addr; int err = 0; - u32 index; + int index; if (!mpfs) return 0; @@ -200,12 +207,87 @@ int mlx5_mpfs_del_mac(struct mlx5_core_dev *dev, u8 *mac) goto unlock; index = l2addr->index; - del_l2table_entry_cmd(dev, index); + if (index >= 0) { + del_l2table_entry_cmd(dev, index); + free_l2table_index(mpfs, index); + mlx5_core_dbg(dev, "MPFS entry %pM, deleted @index (%d)\n", + mac, index); + } l2addr_hash_del(l2addr); - free_l2table_index(mpfs, index); mlx5_core_dbg(dev, "MPFS mac deleted %pM, index (%d)\n", mac, index); unlock: mutex_unlock(&mpfs->lock); return err; } EXPORT_SYMBOL(mlx5_mpfs_del_mac); + +int mlx5_mpfs_enable(struct mlx5_core_dev *dev) +{ + struct mlx5_mpfs *mpfs = dev->priv.mpfs; + struct l2table_node *l2addr; + struct hlist_node *n; + int err = 0, i; + + if (!mpfs) + return -ENODEV; + + mutex_lock(&mpfs->lock); + if (mpfs->enabled) + goto out; + mpfs->enabled = true; + mlx5_core_dbg(dev, "MPFS enabling mpfs\n"); + + mlx5_mpfs_foreach(l2addr, n, mpfs, i) { + u32 index; + + err = alloc_l2table_index(mpfs, &index); + if (err) { + mlx5_core_err(dev, "Failed to allocated MPFS index for %pM, err(%d)\n", + l2addr->node.addr, err); + goto out; + } + + err = set_l2table_entry_cmd(dev, index, l2addr->node.addr); + if (err) { + mlx5_core_err(dev, "Failed to set MPFS l2table entry for %pM index=%d, err(%d)\n", + l2addr->node.addr, index, err); + free_l2table_index(mpfs, index); + goto out; + } + + l2addr->index = index; + mlx5_core_dbg(dev, "MPFS entry %pM, set @index (%d)\n", + l2addr->node.addr, l2addr->index); + } +out: + mutex_unlock(&mpfs->lock); + return err; +} + +void mlx5_mpfs_disable(struct mlx5_core_dev *dev) +{ + struct mlx5_mpfs *mpfs = dev->priv.mpfs; + struct l2table_node *l2addr; + struct hlist_node *n; + int i; + + if (!mpfs) + return; + + mutex_lock(&mpfs->lock); + if (!mpfs->enabled) + goto unlock; + mlx5_mpfs_foreach(l2addr, n, mpfs, i) { + if (l2addr->index < 0) + continue; + del_l2table_entry_cmd(dev, l2addr->index); + free_l2table_index(mpfs, l2addr->index); + mlx5_core_dbg(dev, "MPFS entry %pM, deleted @index (%d)\n", + l2addr->node.addr, l2addr->index); + l2addr->index = -1; + } + mpfs->enabled = false; + mlx5_core_dbg(dev, "MPFS disabled\n"); +unlock: + mutex_unlock(&mpfs->lock); +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.h index 4a293542a7aa..9c63838ce1f3 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/mpfs.h @@ -45,6 +45,10 @@ struct l2addr_node { u8 addr[ETH_ALEN]; }; +#define mlx5_mpfs_foreach(hs, tmp, mpfs, i) \ + for (i = 0; i < MLX5_L2_ADDR_HASH_SIZE; i++) \ + hlist_for_each_entry_safe(hs, tmp, &(mpfs)->hash[i], node.hlist) + #define for_each_l2hash_node(hn, tmp, hash, i) \ for (i = 0; i < MLX5_L2_ADDR_HASH_SIZE; i++) \ hlist_for_each_entry_safe(hn, tmp, &(hash)[i], hlist) @@ -82,11 +86,16 @@ struct l2addr_node { }) #ifdef CONFIG_MLX5_MPFS +struct mlx5_core_dev; int mlx5_mpfs_init(struct mlx5_core_dev *dev); void mlx5_mpfs_cleanup(struct mlx5_core_dev *dev); +int mlx5_mpfs_enable(struct mlx5_core_dev *dev); +void mlx5_mpfs_disable(struct mlx5_core_dev *dev); #else /* #ifndef CONFIG_MLX5_MPFS */ static inline int mlx5_mpfs_init(struct mlx5_core_dev *dev) { return 0; } static inline void mlx5_mpfs_cleanup(struct mlx5_core_dev *dev) {} +static inline int mlx5_mpfs_enable(struct mlx5_core_dev *dev) { return 0; } +static inline void mlx5_mpfs_disable(struct mlx5_core_dev *dev) {} #endif #endif diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index e08903d002a1..f9195f0345df 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -382,6 +382,7 @@ const struct mlx5_link_info *mlx5_port_ptys2info(struct mlx5_core_dev *mdev, u32 mlx5_port_info2linkmodes(struct mlx5_core_dev *mdev, struct mlx5_link_info *info, bool force_legacy); +int mlx5_port_oper_linkspeed(struct mlx5_core_dev *mdev, u32 *speed); int mlx5_port_max_linkspeed(struct mlx5_core_dev *mdev, u32 *speed); #define MLX5_PPS_CAP(mdev) (MLX5_CAP_GEN((mdev), pps) && \ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/port.c b/drivers/net/ethernet/mellanox/mlx5/core/port.c index 23796fb353e4..296a011df02f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/port.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/port.c @@ -1202,6 +1202,30 @@ u32 mlx5_port_info2linkmodes(struct mlx5_core_dev *mdev, return link_modes; } +int mlx5_port_oper_linkspeed(struct mlx5_core_dev *mdev, u32 *speed) +{ + const struct mlx5_link_info *table; + struct mlx5_port_eth_proto eproto; + u32 oper_speed = 0; + u32 max_size; + bool ext; + int err; + int i; + + ext = mlx5_ptys_ext_supported(mdev); + err = mlx5_port_query_eth_proto(mdev, 1, ext, &eproto); + if (err) + return err; + + mlx5e_port_get_link_mode_info_arr(mdev, &table, &max_size, false); + for (i = 0; i < max_size; ++i) + if (eproto.oper & MLX5E_PROT_MASK(i)) + oper_speed = max(oper_speed, table[i].speed); + + *speed = oper_speed; + return 0; +} + int mlx5_port_max_linkspeed(struct mlx5_core_dev *mdev, u32 *speed) { const struct mlx5_link_info *table; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/vport.c b/drivers/net/ethernet/mellanox/mlx5/core/vport.c index a44214c660b0..6b90f6c4f2eb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/vport.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/vport.c @@ -62,11 +62,61 @@ u8 mlx5_query_vport_state(struct mlx5_core_dev *mdev, u8 opmod, u16 vport) return MLX5_GET(query_vport_state_out, out, state); } +static int mlx5_query_vport_admin_state(struct mlx5_core_dev *mdev, u8 opmod, + u16 vport, u8 other_vport, + u8 *admin_state) +{ + u32 out[MLX5_ST_SZ_DW(query_vport_state_out)] = {}; + u32 in[MLX5_ST_SZ_DW(query_vport_state_in)] = {}; + int err; + + MLX5_SET(query_vport_state_in, in, opcode, + MLX5_CMD_OP_QUERY_VPORT_STATE); + MLX5_SET(query_vport_state_in, in, op_mod, opmod); + MLX5_SET(query_vport_state_in, in, vport_number, vport); + MLX5_SET(query_vport_state_in, in, other_vport, other_vport); + + err = mlx5_cmd_exec_inout(mdev, query_vport_state, in, out); + if (err) + return err; + + *admin_state = MLX5_GET(query_vport_state_out, out, admin_state); + return 0; +} + int mlx5_modify_vport_admin_state(struct mlx5_core_dev *mdev, u8 opmod, u16 vport, u8 other_vport, u8 state) { u32 in[MLX5_ST_SZ_DW(modify_vport_state_in)] = {}; +#ifdef CONFIG_MLX5_ESWITCH + lockdep_assert_held(&mdev->priv.eswitch->state_lock); +#endif + + if (MLX5_CAP_ESW(mdev, esw_vport_state_max_tx_speed) && + opmod == MLX5_VPORT_STATE_OP_MOD_ESW_VPORT && + vport != MLX5_VPORT_UPLINK) { + u32 speed = 0; + int err; + + err = mlx5_query_vport_max_tx_speed(mdev, opmod, vport, + other_vport, &speed, NULL); + if (err) { +#ifdef CONFIG_MLX5_ESWITCH + struct mlx5_vport *esw_vport; + + esw_vport = mlx5_eswitch_get_vport(mdev->priv.eswitch, + vport); + speed = IS_ERR(esw_vport) ? 0 : + esw_vport->agg_max_tx_speed; +#endif + mlx5_core_dbg(mdev, + "Failed to query vport %d max tx speed, err=%d, using cached %u\n", + vport, err, speed); + } + MLX5_SET(modify_vport_state_in, in, max_tx_speed, speed); + } + MLX5_SET(modify_vport_state_in, in, opcode, MLX5_CMD_OP_MODIFY_VPORT_STATE); MLX5_SET(modify_vport_state_in, in, op_mod, opmod); @@ -77,6 +127,58 @@ int mlx5_modify_vport_admin_state(struct mlx5_core_dev *mdev, u8 opmod, return mlx5_cmd_exec_in(mdev, modify_vport_state, in); } +int mlx5_modify_vport_max_tx_speed(struct mlx5_core_dev *mdev, u8 opmod, + u16 vport, u8 other_vport, u16 max_tx_speed) +{ + u32 in[MLX5_ST_SZ_DW(modify_vport_state_in)] = {}; + u8 admin_state; + int err; + +#ifdef CONFIG_MLX5_ESWITCH + lockdep_assert_held(&mdev->priv.eswitch->state_lock); +#endif + + err = mlx5_query_vport_admin_state(mdev, opmod, vport, other_vport, + &admin_state); + if (err) + return err; + + MLX5_SET(modify_vport_state_in, in, opcode, + MLX5_CMD_OP_MODIFY_VPORT_STATE); + MLX5_SET(modify_vport_state_in, in, op_mod, opmod); + MLX5_SET(modify_vport_state_in, in, vport_number, vport); + MLX5_SET(modify_vport_state_in, in, other_vport, other_vport); + MLX5_SET(modify_vport_state_in, in, admin_state, admin_state); + MLX5_SET(modify_vport_state_in, in, max_tx_speed, max_tx_speed); + + return mlx5_cmd_exec_in(mdev, modify_vport_state, in); +} + +int mlx5_query_vport_max_tx_speed(struct mlx5_core_dev *mdev, u8 op_mod, + u16 vport, u8 other_vport, + u32 *max_tx_speed, u8 *state) +{ + u32 out[MLX5_ST_SZ_DW(query_vport_state_out)] = {}; + u32 in[MLX5_ST_SZ_DW(query_vport_state_in)] = {}; + int err; + + MLX5_SET(query_vport_state_in, in, opcode, + MLX5_CMD_OP_QUERY_VPORT_STATE); + MLX5_SET(query_vport_state_in, in, op_mod, op_mod); + MLX5_SET(query_vport_state_in, in, vport_number, vport); + MLX5_SET(query_vport_state_in, in, other_vport, other_vport); + + err = mlx5_cmd_exec_inout(mdev, query_vport_state, in, out); + if (err) + return err; + + *max_tx_speed = MLX5_GET(query_vport_state_out, out, max_tx_speed); + if (state) + *state = MLX5_GET(query_vport_state_out, out, state); + return 0; +} +EXPORT_SYMBOL_GPL(mlx5_query_vport_max_tx_speed); + static int mlx5_query_nic_vport_context(struct mlx5_core_dev *mdev, u16 vport, u32 *out) { diff --git a/drivers/net/ethernet/nvidia/forcedeth.c b/drivers/net/ethernet/nvidia/forcedeth.c index 01eb1c75f2da..c16baa28ad75 100644 --- a/drivers/net/ethernet/nvidia/forcedeth.c +++ b/drivers/net/ethernet/nvidia/forcedeth.c @@ -2740,7 +2740,7 @@ static void nv_tx_timeout(struct net_device *dev, unsigned int txqueue) netdev_info(dev, "Ring at %lx\n", (unsigned long)np->ring_addr); netdev_info(dev, "Dumping tx registers\n"); - for (i = 0; i <= np->register_size; i += 32) { + for (i = 0; i + 32 <= np->register_size; i += 32) { netdev_info(dev, "%3x: %08x %08x %08x %08x " "%08x %08x %08x %08x\n", diff --git a/drivers/net/ethernet/pensando/ionic/ionic_txrx.c b/drivers/net/ethernet/pensando/ionic/ionic_txrx.c index 73998d61593a..61d48c172587 100644 --- a/drivers/net/ethernet/pensando/ionic/ionic_txrx.c +++ b/drivers/net/ethernet/pensando/ionic/ionic_txrx.c @@ -706,11 +706,7 @@ static void ionic_rx_clean(struct ionic_queue *q, __le64 *cq_desc_hwstamp; u64 hwstamp; - cq_desc_hwstamp = - (void *)comp + - qcq->cq.desc_size - - sizeof(struct ionic_rxq_comp) - - IONIC_HWSTAMP_CQ_NEGOFFSET; + cq_desc_hwstamp = (void *)comp - IONIC_HWSTAMP_CQ_NEGOFFSET; hwstamp = le64_to_cpu(*cq_desc_hwstamp); @@ -734,7 +730,12 @@ static bool __ionic_rx_service(struct ionic_cq *cq, struct bpf_prog *xdp_prog) struct ionic_queue *q = cq->bound_q; struct ionic_rxq_comp *comp; - comp = &((struct ionic_rxq_comp *)cq->base)[cq->tail_idx]; + if (likely(cq->desc_size == sizeof(*comp))) + comp = &((struct ionic_rxq_comp *)cq->base)[cq->tail_idx]; + else + comp = cq->base + + cq->desc_size * cq->tail_idx + + cq->desc_size - sizeof(*comp); if (!color_match(comp->pkt_type_color, cq->done_color)) return false; @@ -1185,7 +1186,6 @@ static void ionic_tx_clean(struct ionic_queue *q, bool in_napi) { struct ionic_tx_stats *stats = q_to_tx_stats(q); - struct ionic_qcq *qcq = q_to_qcq(q); struct sk_buff *skb; if (desc_info->xdpf) { @@ -1210,11 +1210,7 @@ static void ionic_tx_clean(struct ionic_queue *q, __le64 *cq_desc_hwstamp; u64 hwstamp; - cq_desc_hwstamp = - (void *)comp + - qcq->cq.desc_size - - sizeof(struct ionic_txq_comp) - - IONIC_HWSTAMP_CQ_NEGOFFSET; + cq_desc_hwstamp = (void *)comp - IONIC_HWSTAMP_CQ_NEGOFFSET; hwstamp = le64_to_cpu(*cq_desc_hwstamp); @@ -1249,7 +1245,12 @@ static bool ionic_tx_service(struct ionic_cq *cq, unsigned int pkts = 0; u16 index; - comp = &((struct ionic_txq_comp *)cq->base)[cq->tail_idx]; + if (likely(cq->desc_size == sizeof(*comp))) + comp = &((struct ionic_txq_comp *)cq->base)[cq->tail_idx]; + else + comp = cq->base + + cq->desc_size * cq->tail_idx + + cq->desc_size - sizeof(*comp); if (!color_match(comp->color, cq->done_color)) return false; diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c index 09f20c794754..d26c26c21cbe 100644 --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c @@ -6,6 +6,7 @@ #include "qlcnic.h" #include "qlcnic_hw.h" +#include struct crb_addr_pair { u32 addr; @@ -740,159 +741,212 @@ qlcnic_has_mn(struct qlcnic_adapter *adapter) return 0; } -static -struct uni_table_desc *qlcnic_get_table_desc(const u8 *unirom, int section) -{ - u32 i, entries; - struct uni_table_desc *directory = (struct uni_table_desc *) &unirom[0]; - entries = le32_to_cpu(directory->num_entries); - - for (i = 0; i < entries; i++) { +#define FILEHEADER_SIZE (14 * 4) +#define QLCNIC_UNI_DIR_TYPE_OFF (8 * sizeof(__le32)) +#define QLCNIC_UNI_DIR_ENTRY_MIN_SIZE (9 * sizeof(__le32)) +#define QLCNIC_UNI_PRODUCT_ENTRY_MIN_SIZE \ + ((QLCNIC_UNI_FIRMWARE_IDX_OFF + 1) * sizeof(__le32)) +#define QLCNIC_UNI_VERSION_TAIL_SIZE 17 +#define QLCNIC_UNI_BOOTLD_SIZE \ + (QLCNIC_IMAGE_START - QLCNIC_BOOTLD_START) + +struct qlcnic_uni_data { + u32 offset; + u32 size; +}; - u32 offs = le32_to_cpu(directory->findex) + - i * le32_to_cpu(directory->entry_size); - u32 tab_type = le32_to_cpu(*((__le32 *)&unirom[offs] + 8)); +static bool qlcnic_rom_range_valid(size_t size, size_t offset, size_t len) +{ + return offset <= size && len <= size - offset; +} - if (tab_type == section) - return (struct uni_table_desc *) &unirom[offs]; - } +static bool qlcnic_rom_table_valid(size_t size, u32 offset, u32 entries, + u32 entry_size, u32 min_entry_size) +{ + if (entry_size < min_entry_size || offset > size) + return false; - return NULL; + return entries <= (size - offset) / entry_size; } -#define FILEHEADER_SIZE (14 * 4) - -static int -qlcnic_validate_header(struct qlcnic_adapter *adapter) +static int qlcnic_get_directory(struct qlcnic_adapter *adapter, + size_t *offset, u32 *entries, u32 *entry_size) { - const u8 *unirom = adapter->fw->data; - struct uni_table_desc *directory = (struct uni_table_desc *) &unirom[0]; - u32 entries, entry_size, tab_size, fw_file_size; - - fw_file_size = adapter->fw->size; + const struct firmware *fw = adapter->fw; + const u8 *directory = fw->data; - if (fw_file_size < FILEHEADER_SIZE) + if (fw->size < FILEHEADER_SIZE) return -EINVAL; - entries = le32_to_cpu(directory->num_entries); - entry_size = le32_to_cpu(directory->entry_size); - tab_size = le32_to_cpu(directory->findex) + (entries * entry_size); + *offset = get_unaligned_le32(directory + + offsetof(struct uni_table_desc, findex)); + *entries = get_unaligned_le32(directory + + offsetof(struct uni_table_desc, num_entries)); + *entry_size = get_unaligned_le32(directory + + offsetof(struct uni_table_desc, entry_size)); - if (fw_file_size < tab_size) + if (!qlcnic_rom_table_valid(fw->size, *offset, *entries, *entry_size, + QLCNIC_UNI_DIR_ENTRY_MIN_SIZE)) return -EINVAL; return 0; } -static int -qlcnic_validate_bootld(struct qlcnic_adapter *adapter) +static int qlcnic_get_table_desc(struct qlcnic_adapter *adapter, int section, + size_t *desc_offset) { - struct uni_table_desc *tab_desc; - struct uni_data_desc *descr; - u32 offs, tab_size, data_size, idx; const u8 *unirom = adapter->fw->data; - __le32 temp; - - temp = *((__le32 *)&unirom[adapter->file_prd_off] + - QLCNIC_UNI_BOOTLD_IDX_OFF); - idx = le32_to_cpu(temp); - tab_desc = qlcnic_get_table_desc(unirom, QLCNIC_UNI_DIR_SECT_BOOTLD); - - if (!tab_desc) - return -EINVAL; - - tab_size = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * (idx + 1); - - if (adapter->fw->size < tab_size) - return -EINVAL; + size_t directory_offset; + u32 entries, entry_size; + size_t i; + int ret; - offs = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * idx; - descr = (struct uni_data_desc *)&unirom[offs]; + ret = qlcnic_get_directory(adapter, &directory_offset, &entries, + &entry_size); + if (ret) + return ret; - data_size = le32_to_cpu(descr->findex) + le32_to_cpu(descr->size); + for (i = 0; i < entries; i++) { + size_t offset = directory_offset + i * entry_size; + u32 table_type; - if (adapter->fw->size < data_size) - return -EINVAL; + table_type = get_unaligned_le32(unirom + offset + + QLCNIC_UNI_DIR_TYPE_OFF); + if (table_type == section) { + *desc_offset = offset; + return 0; + } + } - return 0; + return -ENOENT; } static int -qlcnic_validate_fw(struct qlcnic_adapter *adapter) +qlcnic_validate_header(struct qlcnic_adapter *adapter) { - struct uni_table_desc *tab_desc; - struct uni_data_desc *descr; - const u8 *unirom = adapter->fw->data; - u32 offs, tab_size, data_size, idx; - __le32 temp; + u32 entries, entry_size; + size_t offset; + + return qlcnic_get_directory(adapter, &offset, &entries, &entry_size); +} - temp = *((__le32 *)&unirom[adapter->file_prd_off] + - QLCNIC_UNI_FIRMWARE_IDX_OFF); - idx = le32_to_cpu(temp); - tab_desc = qlcnic_get_table_desc(unirom, QLCNIC_UNI_DIR_SECT_FW); +static int qlcnic_get_data_desc(struct qlcnic_adapter *adapter, u32 section, + u32 index_offset, struct qlcnic_uni_data *data) +{ + size_t table_desc_offset, table_offset, desc_offset; + const struct firmware *fw = adapter->fw; + const u8 *unirom = fw->data; + size_t product_index_offset; + u32 entries, entry_size, idx; + int ret; - if (!tab_desc) + product_index_offset = adapter->file_prd_off + + (size_t)index_offset * sizeof(__le32); + if (!qlcnic_rom_range_valid(fw->size, product_index_offset, + sizeof(__le32))) return -EINVAL; - tab_size = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * (idx + 1); + idx = get_unaligned_le32(unirom + product_index_offset); + ret = qlcnic_get_table_desc(adapter, section, &table_desc_offset); + if (ret) + return ret; - if (adapter->fw->size < tab_size) + table_offset = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, findex)); + entries = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, num_entries)); + entry_size = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, entry_size)); + if (!qlcnic_rom_table_valid(fw->size, table_offset, entries, + entry_size, sizeof(struct uni_data_desc)) || + idx >= entries) return -EINVAL; - offs = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * idx; - descr = (struct uni_data_desc *)&unirom[offs]; - data_size = le32_to_cpu(descr->findex) + le32_to_cpu(descr->size); + desc_offset = table_offset + (size_t)idx * entry_size; + data->offset = get_unaligned_le32(unirom + desc_offset + + offsetof(struct uni_data_desc, findex)); + data->size = get_unaligned_le32(unirom + desc_offset + + offsetof(struct uni_data_desc, size)); - if (adapter->fw->size < data_size) + if (!qlcnic_rom_range_valid(fw->size, data->offset, data->size)) return -EINVAL; return 0; } +static int qlcnic_validate_bootld(struct qlcnic_adapter *adapter) +{ + struct qlcnic_uni_data data; + int ret; + + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD, + QLCNIC_UNI_BOOTLD_IDX_OFF, &data); + if (ret) + return ret; + + return data.size < QLCNIC_UNI_BOOTLD_SIZE ? -EINVAL : 0; +} + +static int qlcnic_validate_fw(struct qlcnic_adapter *adapter) +{ + struct qlcnic_uni_data data; + int ret; + + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, + QLCNIC_UNI_FIRMWARE_IDX_OFF, &data); + if (ret) + return ret; + + return data.size < QLCNIC_UNI_VERSION_TAIL_SIZE ? -EINVAL : 0; +} + static int qlcnic_validate_product_offs(struct qlcnic_adapter *adapter) { - struct uni_table_desc *ptab_descr; + size_t table_desc_offset, table_offset; const u8 *unirom = adapter->fw->data; int mn_present = qlcnic_has_mn(adapter); - u32 entries, entry_size, tab_size, i; - __le32 temp; - - ptab_descr = qlcnic_get_table_desc(unirom, - QLCNIC_UNI_DIR_SECT_PRODUCT_TBL); - if (!ptab_descr) - return -EINVAL; + u32 entries, entry_size; + size_t i; + int ret; - entries = le32_to_cpu(ptab_descr->num_entries); - entry_size = le32_to_cpu(ptab_descr->entry_size); - tab_size = le32_to_cpu(ptab_descr->findex) + (entries * entry_size); + ret = qlcnic_get_table_desc(adapter, QLCNIC_UNI_DIR_SECT_PRODUCT_TBL, + &table_desc_offset); + if (ret) + return ret; - if (adapter->fw->size < tab_size) + table_offset = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, findex)); + entries = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, num_entries)); + entry_size = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, entry_size)); + if (!qlcnic_rom_table_valid(adapter->fw->size, table_offset, entries, + entry_size, + QLCNIC_UNI_PRODUCT_ENTRY_MIN_SIZE)) return -EINVAL; nomn: for (i = 0; i < entries; i++) { - - u32 flags, file_chiprev, offs; + size_t offset = table_offset + i * entry_size; u8 chiprev = adapter->ahw->revision_id; + u32 flags, file_chiprev; u32 flagbit; - offs = le32_to_cpu(ptab_descr->findex) + - i * le32_to_cpu(ptab_descr->entry_size); - temp = *((__le32 *)&unirom[offs] + QLCNIC_UNI_FLAGS_OFF); - flags = le32_to_cpu(temp); - temp = *((__le32 *)&unirom[offs] + QLCNIC_UNI_CHIP_REV_OFF); - file_chiprev = le32_to_cpu(temp); + flags = get_unaligned_le32(unirom + offset + + QLCNIC_UNI_FLAGS_OFF * sizeof(__le32)); + file_chiprev = get_unaligned_le32(unirom + offset + + QLCNIC_UNI_CHIP_REV_OFF * + sizeof(__le32)); flagbit = mn_present ? 1 : 2; if ((chiprev == file_chiprev) && ((1ULL << flagbit) & flags)) { - adapter->file_prd_off = offs; + if (offset > U32_MAX) + return -EINVAL; + + adapter->file_prd_off = offset; return 0; } } @@ -933,93 +987,81 @@ qlcnic_validate_unified_romimage(struct qlcnic_adapter *adapter) return 0; } -static -struct uni_data_desc *qlcnic_get_data_desc(struct qlcnic_adapter *adapter, - u32 section, u32 idx_offset) -{ - const u8 *unirom = adapter->fw->data; - struct uni_table_desc *tab_desc; - u32 offs, idx; - __le32 temp; - - temp = *((__le32 *)&unirom[adapter->file_prd_off] + idx_offset); - idx = le32_to_cpu(temp); - - tab_desc = qlcnic_get_table_desc(unirom, section); - - if (tab_desc == NULL) - return NULL; - - offs = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * idx; - - return (struct uni_data_desc *)&unirom[offs]; -} - -static u8 * -qlcnic_get_bootld_offs(struct qlcnic_adapter *adapter) +static int qlcnic_get_bootld_data(struct qlcnic_adapter *adapter, + const u8 **bootld) { u32 offs = QLCNIC_BOOTLD_START; - struct uni_data_desc *data_desc; - - data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD, - QLCNIC_UNI_BOOTLD_IDX_OFF); + struct qlcnic_uni_data data; + int ret; - if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) - offs = le32_to_cpu(data_desc->findex); + if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) { + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD, + QLCNIC_UNI_BOOTLD_IDX_OFF, &data); + if (ret || data.size < QLCNIC_UNI_BOOTLD_SIZE) + return -EINVAL; + offs = data.offset; + } else if (!qlcnic_rom_range_valid(adapter->fw->size, offs, + QLCNIC_UNI_BOOTLD_SIZE)) { + return -EINVAL; + } - return (u8 *)&adapter->fw->data[offs]; + *bootld = adapter->fw->data + offs; + return 0; } -static u8 * -qlcnic_get_fw_offs(struct qlcnic_adapter *adapter) +static int qlcnic_get_fw_data(struct qlcnic_adapter *adapter, + const u8 **image, u32 *image_size) { u32 offs = QLCNIC_IMAGE_START; - struct uni_data_desc *data_desc; - - data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, - QLCNIC_UNI_FIRMWARE_IDX_OFF); - if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) - offs = le32_to_cpu(data_desc->findex); - - return (u8 *)&adapter->fw->data[offs]; -} + struct qlcnic_uni_data data; + int ret; -static u32 qlcnic_get_fw_size(struct qlcnic_adapter *adapter) -{ - struct uni_data_desc *data_desc; - const u8 *unirom = adapter->fw->data; + if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) { + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, + QLCNIC_UNI_FIRMWARE_IDX_OFF, &data); + if (ret) + return ret; + offs = data.offset; + *image_size = data.size; + } else { + if (!qlcnic_rom_range_valid(adapter->fw->size, + QLCNIC_FW_SIZE_OFFSET, + sizeof(__le32))) + return -EINVAL; + *image_size = get_unaligned_le32(adapter->fw->data + + QLCNIC_FW_SIZE_OFFSET); + } - data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, - QLCNIC_UNI_FIRMWARE_IDX_OFF); + if (!qlcnic_rom_range_valid(adapter->fw->size, offs, *image_size)) + return -EINVAL; - if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) - return le32_to_cpu(data_desc->size); - else - return le32_to_cpu(*(__le32 *)&unirom[QLCNIC_FW_SIZE_OFFSET]); + *image = adapter->fw->data + offs; + return 0; } static u32 qlcnic_get_fw_version(struct qlcnic_adapter *adapter) { - struct uni_data_desc *fw_data_desc; + char ver_str[QLCNIC_UNI_VERSION_TAIL_SIZE + 1]; const struct firmware *fw = adapter->fw; + struct qlcnic_uni_data data; u32 major, minor, sub; - __le32 version_offset; - const u8 *ver_str; int i, ret; - if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) { - version_offset = *(__le32 *)&fw->data[QLCNIC_FW_VERSION_OFFSET]; - return le32_to_cpu(version_offset); - } + if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) + return get_unaligned_le32(fw->data + QLCNIC_FW_VERSION_OFFSET); + + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, + QLCNIC_UNI_FIRMWARE_IDX_OFF, &data); + if (ret || data.size < QLCNIC_UNI_VERSION_TAIL_SIZE) + return 0; - fw_data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, - QLCNIC_UNI_FIRMWARE_IDX_OFF); - ver_str = fw->data + le32_to_cpu(fw_data_desc->findex) + - le32_to_cpu(fw_data_desc->size) - 17; + memcpy(ver_str, fw->data + data.offset + data.size - + QLCNIC_UNI_VERSION_TAIL_SIZE, + QLCNIC_UNI_VERSION_TAIL_SIZE); + ver_str[QLCNIC_UNI_VERSION_TAIL_SIZE] = '\0'; for (i = 0; i < 12; i++) { - if (!strncmp(&ver_str[i], "REV=", 4)) { + if (!strncmp(ver_str + i, "REV=", 4)) { ret = sscanf(&ver_str[i+4], "%u.%u.%u ", &major, &minor, &sub); if (ret != 3) @@ -1034,18 +1076,15 @@ static u32 qlcnic_get_fw_version(struct qlcnic_adapter *adapter) static u32 qlcnic_get_bios_version(struct qlcnic_adapter *adapter) { - const struct firmware *fw = adapter->fw; u32 bios_ver, prd_off = adapter->file_prd_off; - u8 *version_offset; - __le32 temp; + const struct firmware *fw = adapter->fw; - if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) { - version_offset = (u8 *)&fw->data[QLCNIC_BIOS_VERSION_OFFSET]; - return le32_to_cpu(*(__le32 *)version_offset); - } + if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) + return get_unaligned_le32(fw->data + QLCNIC_BIOS_VERSION_OFFSET); - temp = *((__le32 *)(&fw->data[prd_off]) + QLCNIC_UNI_BIOS_VERSION_OFF); - bios_ver = le32_to_cpu(temp); + bios_ver = get_unaligned_le32(fw->data + prd_off + + QLCNIC_UNI_BIOS_VERSION_OFF * + sizeof(__le32)); return (bios_ver << 16) + ((bios_ver >> 8) & 0xff00) + (bios_ver >> 24); } @@ -1106,24 +1145,26 @@ static const char *fw_name[] = { int qlcnic_load_firmware(struct qlcnic_adapter *adapter) { - __le64 *ptr64; - u32 i, flashaddr, size; const struct firmware *fw = adapter->fw; struct pci_dev *pdev = adapter->pdev; + const u8 *bootld, *image; + u32 i, flashaddr, image_size; + int ret; dev_info(&pdev->dev, "loading firmware from %s\n", fw_name[adapter->ahw->fw_type]); if (fw) { + u32 words, remainder; u64 data; - size = (QLCNIC_IMAGE_START - QLCNIC_BOOTLD_START) / 8; - - ptr64 = (__le64 *)qlcnic_get_bootld_offs(adapter); + ret = qlcnic_get_bootld_data(adapter, &bootld); + if (ret) + return ret; flashaddr = QLCNIC_BOOTLD_START; - for (i = 0; i < size; i++) { - data = le64_to_cpu(ptr64[i]); + for (i = 0; i < QLCNIC_UNI_BOOTLD_SIZE / sizeof(u64); i++) { + data = get_unaligned_le64(bootld + i * sizeof(u64)); if (qlcnic_pci_mem_write_2M(adapter, flashaddr, data)) return -EIO; @@ -1131,13 +1172,15 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) flashaddr += 8; } - size = qlcnic_get_fw_size(adapter) / 8; - - ptr64 = (__le64 *)qlcnic_get_fw_offs(adapter); + ret = qlcnic_get_fw_data(adapter, &image, &image_size); + if (ret) + return ret; + words = image_size / sizeof(u64); + remainder = image_size % sizeof(u64); flashaddr = QLCNIC_IMAGE_START; - for (i = 0; i < size; i++) { - data = le64_to_cpu(ptr64[i]); + for (i = 0; i < words; i++) { + data = get_unaligned_le64(image + i * sizeof(u64)); if (qlcnic_pci_mem_write_2M(adapter, flashaddr, data)) @@ -1146,9 +1189,11 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) flashaddr += 8; } - size = qlcnic_get_fw_size(adapter) % 8; - if (size) { - data = le64_to_cpu(ptr64[i]); + if (remainder) { + __le64 tail = 0; + + memcpy(&tail, image + words * sizeof(u64), remainder); + data = le64_to_cpu(tail); if (qlcnic_pci_mem_write_2M(adapter, flashaddr, data)) @@ -1156,10 +1201,9 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) } } else { - u64 data; - u32 hi, lo; - int ret; struct qlcnic_flt_entry bootld_entry; + u32 hi, lo, size; + u64 data; ret = qlcnic_get_flt_entry(adapter, QLCNIC_BOOTLD_REGION, &bootld_entry); @@ -1200,26 +1244,33 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) static int qlcnic_validate_firmware(struct qlcnic_adapter *adapter) { - u32 val; - u32 ver, bios, min_size; - struct pci_dev *pdev = adapter->pdev; const struct firmware *fw = adapter->fw; + struct pci_dev *pdev = adapter->pdev; u8 fw_type = adapter->ahw->fw_type; + u32 ver, bios, min_size; + const u8 *data; + u32 data_size; + u32 val; + + if (fw_type == QLCNIC_UNIFIED_ROMIMAGE) + min_size = QLCNIC_UNI_FW_MIN_SIZE; + else + min_size = QLCNIC_FW_MIN_SIZE; + + if (fw->size < min_size) + return -EINVAL; if (fw_type == QLCNIC_UNIFIED_ROMIMAGE) { if (qlcnic_validate_unified_romimage(adapter)) return -EINVAL; - - min_size = QLCNIC_UNI_FW_MIN_SIZE; } else { - val = le32_to_cpu(*(__le32 *)&fw->data[QLCNIC_FW_MAGIC_OFFSET]); + val = get_unaligned_le32(fw->data + QLCNIC_FW_MAGIC_OFFSET); if (val != QLCNIC_BDINFO_MAGIC) return -EINVAL; - - min_size = QLCNIC_FW_MIN_SIZE; } - if (fw->size < min_size) + if (qlcnic_get_bootld_data(adapter, &data) || + qlcnic_get_fw_data(adapter, &data, &data_size)) return -EINVAL; val = qlcnic_get_fw_version(adapter); diff --git a/drivers/net/ethernet/qualcomm/rmnet/rmnet_map_data.c b/drivers/net/ethernet/qualcomm/rmnet/rmnet_map_data.c index 305ae15ae8f3..e6f48dc9fe64 100644 --- a/drivers/net/ethernet/qualcomm/rmnet/rmnet_map_data.c +++ b/drivers/net/ethernet/qualcomm/rmnet/rmnet_map_data.c @@ -394,6 +394,7 @@ struct sk_buff *rmnet_map_deaggregate(struct sk_buff *skb, if (!skbn) return NULL; + skbn->dev = skb->dev; skb_reserve(skbn, RMNET_MAP_DEAGGR_HEADROOM); skb_put(skbn, packet_len); memcpy(skbn->data, skb->data, packet_len); diff --git a/drivers/net/ethernet/realtek/8139cp.c b/drivers/net/ethernet/realtek/8139cp.c index 5652da8a178c..9016527e229a 100644 --- a/drivers/net/ethernet/realtek/8139cp.c +++ b/drivers/net/ethernet/realtek/8139cp.c @@ -2066,7 +2066,7 @@ static int __maybe_unused cp_suspend(struct device *device) /* Disable Rx and Tx */ cpw16 (IntrMask, 0); - cpw8 (Cmd, cpr8 (Cmd) & (~RxOn | ~TxOn)); + cpw8 (Cmd, cpr8 (Cmd) & ~(RxOn | TxOn)); spin_unlock_irqrestore (&cp->lock, flags); diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c b/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c index 5bf717188c92..8a786bececda 100644 --- a/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c +++ b/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c @@ -187,7 +187,7 @@ static void dwmac1000_set_filter(struct mac_device_info *hw, dwmac1000_set_mchash(ioaddr, mc_filter, mcbitslog2); /* Handle multiple unicast addresses (perfect filtering) */ - if (netdev_uc_count(dev) > perfect_addr_number) + if (netdev_uc_count(dev) + 1 > perfect_addr_number) /* Switch to promiscuous mode if more than unicast * addresses are requested than supported by hardware. */ diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c b/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c index 1f61177ece45..666aed0906b5 100644 --- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c +++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c @@ -515,7 +515,7 @@ static void dwmac4_set_filter(struct mac_device_info *hw, value |= GMAC_PACKET_FILTER_HPF; /* Handle multiple unicast addresses */ - if (netdev_uc_count(dev) > hw->unicast_filter_entries) { + if (netdev_uc_count(dev) + 1 > hw->unicast_filter_entries) { /* Switch to promiscuous mode if more than 128 addrs * are required */ diff --git a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_core.c b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_core.c index ee41c77426f1..42cd0a01a9cd 100644 --- a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_core.c +++ b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_core.c @@ -541,7 +541,7 @@ static void dwxgmac2_set_filter(struct mac_device_info *hw, dwxgmac2_set_mchash(ioaddr, mc_filter, mcbitslog2); /* Handle multiple unicast addresses */ - if (netdev_uc_count(dev) > hw->unicast_filter_entries) { + if (netdev_uc_count(dev) + 1 > hw->unicast_filter_entries) { value |= XGMAC_FILTER_PR; } else { struct netdev_hw_addr *ha; diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index bf2fa56b861b..a3eeb68c8d3f 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -1126,6 +1126,10 @@ static int stmmac_init_phy(struct net_device *dev) struct phy_device *phydev; if (addr < 0) { + /* If a custom PCS is in use, no PHY is needed */ + if (priv->hw->phylink_pcs) + return 0; + netdev_err(priv->dev, "no phy found\n"); return -ENODEV; } @@ -1322,9 +1326,9 @@ static void stmmac_display_rings(struct stmmac_priv *priv, static unsigned int stmmac_rx_offset(struct stmmac_priv *priv) { if (stmmac_xdp_is_enabled(priv)) - return XDP_PACKET_HEADROOM; + return XDP_PACKET_HEADROOM + NET_IP_ALIGN; - return NET_SKB_PAD; + return NET_SKB_PAD + NET_IP_ALIGN; } static int stmmac_set_bfsize(int mtu, int bufsize) @@ -2541,7 +2545,8 @@ static bool stmmac_xdp_xmit_zc(struct stmmac_priv *priv, u32 queue, u32 budget) tx_desc = tx_q->dma_tx + entry; dma_addr = xsk_buff_raw_get_dma(pool, xdp_desc.addr); - meta = xsk_buff_get_metadata(pool, xdp_desc.addr); + meta = xsk_buff_get_metadata(pool, xdp_desc.addr, + xdp_desc.options); xsk_buff_raw_dma_sync_for_device(pool, dma_addr, xdp_desc.len); tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XSK_TX; diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c index a01bc394d1ac..3f215ef3c89b 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c @@ -415,11 +415,17 @@ static int stmmac_test_mmc(struct stmmac_priv *priv) stmmac_mmc_read(priv, priv->mmcaddr, &final); /* - * The number of MMC counters available depends on HW configuration - * so we just use this one to validate the feature. I hope there is - * not a version without this counter. + * The number of MMC counters available depends on HW configuration, + * and there doesn't seem to be a way to enumerate the implemented + * counters. + * + * Let's check a hand-picked set of counters, knowing that : + * - Starfive JH7110 doesn't implement mmc_tx_framecount_g + * - Amlogic SM1 doesn't implement any mmc_tx_* + * */ - if (final.mmc_tx_framecount_g <= initial.mmc_tx_framecount_g) + if (final.mmc_tx_framecount_g <= initial.mmc_tx_framecount_g && + final.mmc_rx_framecount_gb <= initial.mmc_rx_framecount_gb) return -EINVAL; return 0; @@ -493,6 +499,21 @@ static int stmmac_filter_check(struct stmmac_priv *priv) return -EOPNOTSUPP; } +static int stmmac_uc_filter_check(struct stmmac_priv *priv) +{ + /* For tests involving the UC filter, we need at least one empty + * slot in the UC filter. The UC filters contains netdev_uc_count() + 1 + * entries: The dev->uc list + one entry for the HW address. + * + * Having an empty slot therefore means netdev_uc_count() + 2 entries + * can fit in the filter + */ + if (netdev_uc_count(priv->dev) + 2 > priv->hw->unicast_filter_entries) + return -EOPNOTSUPP; + + return 0; +} + static bool stmmac_hash_check(struct stmmac_priv *priv, unsigned char *addr) { int mc_offset = 32 - priv->hw->mcast_bits_log2; @@ -584,7 +605,7 @@ static int stmmac_test_pfilt(struct stmmac_priv *priv) if (stmmac_filter_check(priv)) return -EOPNOTSUPP; - if (netdev_uc_count(priv->dev) >= priv->hw->unicast_filter_entries) + if (stmmac_uc_filter_check(priv)) return -EOPNOTSUPP; while (--tries) { @@ -628,7 +649,7 @@ static int stmmac_test_mcfilt(struct stmmac_priv *priv) if (stmmac_filter_check(priv)) return -EOPNOTSUPP; - if (netdev_uc_count(priv->dev) >= priv->hw->unicast_filter_entries) + if (stmmac_uc_filter_check(priv)) return -EOPNOTSUPP; if (netdev_mc_count(priv->dev) >= priv->hw->multicast_filter_bins) return -EOPNOTSUPP; @@ -674,7 +695,7 @@ static int stmmac_test_ucfilt(struct stmmac_priv *priv) if (stmmac_filter_check(priv)) return -EOPNOTSUPP; - if (netdev_uc_count(priv->dev) >= priv->hw->unicast_filter_entries) + if (stmmac_uc_filter_check(priv)) return -EOPNOTSUPP; if (netdev_mc_count(priv->dev) >= priv->hw->multicast_filter_bins) return -EOPNOTSUPP; @@ -1452,11 +1473,11 @@ static int __stmmac_test_l4filt(struct stmmac_priv *priv, u32 dst, u32 src, struct { struct flow_dissector_key_basic bkey; struct flow_dissector_key_ports key; - } __aligned(BITS_PER_LONG / 8) keys; + } __aligned(BITS_PER_LONG / 8) keys = { }; struct { struct flow_dissector_key_basic bmask; struct flow_dissector_key_ports mask; - } __aligned(BITS_PER_LONG / 8) masks; + } __aligned(BITS_PER_LONG / 8) masks = { }; unsigned long dummy_cookie = 0xdeadbeef; struct stmmac_packet_attrs attr = { }; struct flow_dissector *dissector; @@ -1509,6 +1530,8 @@ static int __stmmac_test_l4filt(struct stmmac_priv *priv, u32 dst, u32 src, keys.bkey.ip_proto = udp ? IPPROTO_UDP : IPPROTO_TCP; keys.key.src = htons(src); keys.key.dst = htons(dst); + /* Match the full IP proto field */ + masks.bmask.ip_proto = 0xff; masks.mask.src = src_mask; masks.mask.dst = dst_mask; diff --git a/drivers/net/ethernet/wangxun/libwx/wx_ethtool.c b/drivers/net/ethernet/wangxun/libwx/wx_ethtool.c index 06f401bd975c..9aa3964187e1 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_ethtool.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_ethtool.c @@ -51,6 +51,11 @@ static const struct wx_stats wx_gstrings_fdir_stats[] = { WX_STAT("fdir_miss", stats.fdirmiss), }; +static const struct wx_stats wx_gstrings_rsc_stats[] = { + WX_STAT("rsc_aggregated", rsc_count), + WX_STAT("rsc_flushed", rsc_flush), +}; + /* drivers allocates num_tx_queues and num_rx_queues symmetrically so * we set the num_rx_queues to evaluate to num_tx_queues. This is * used because we do not have a good way to get the max number of @@ -64,16 +69,21 @@ static const struct wx_stats wx_gstrings_fdir_stats[] = { (sizeof(struct wx_queue_stats) / sizeof(u64))) #define WX_GLOBAL_STATS_LEN ARRAY_SIZE(wx_gstrings_stats) #define WX_FDIR_STATS_LEN ARRAY_SIZE(wx_gstrings_fdir_stats) +#define WX_RSC_STATS_LEN ARRAY_SIZE(wx_gstrings_rsc_stats) #define WX_STATS_LEN (WX_GLOBAL_STATS_LEN + WX_QUEUE_STATS_LEN) int wx_get_sset_count(struct net_device *netdev, int sset) { struct wx *wx = netdev_priv(netdev); + int len = WX_STATS_LEN; switch (sset) { case ETH_SS_STATS: - return (test_bit(WX_FLAG_FDIR_CAPABLE, wx->flags)) ? - WX_STATS_LEN + WX_FDIR_STATS_LEN : WX_STATS_LEN; + if (test_bit(WX_FLAG_FDIR_CAPABLE, wx->flags)) + len += WX_FDIR_STATS_LEN; + if (test_bit(WX_FLAG_RSC_CAPABLE, wx->flags)) + len += WX_RSC_STATS_LEN; + return len; default: return -EOPNOTSUPP; } @@ -94,6 +104,10 @@ void wx_get_strings(struct net_device *netdev, u32 stringset, u8 *data) for (i = 0; i < WX_FDIR_STATS_LEN; i++) ethtool_puts(&p, wx_gstrings_fdir_stats[i].stat_string); } + if (test_bit(WX_FLAG_RSC_CAPABLE, wx->flags)) { + for (i = 0; i < WX_RSC_STATS_LEN; i++) + ethtool_puts(&p, wx_gstrings_rsc_stats[i].stat_string); + } for (i = 0; i < netdev->num_tx_queues; i++) { ethtool_sprintf(&p, "tx_queue_%u_packets", i); ethtool_sprintf(&p, "tx_queue_%u_bytes", i); @@ -131,6 +145,13 @@ void wx_get_ethtool_stats(struct net_device *netdev, } } + if (test_bit(WX_FLAG_RSC_CAPABLE, wx->flags)) { + for (k = 0; k < WX_RSC_STATS_LEN; k++) { + p = (char *)wx + wx_gstrings_rsc_stats[k].stat_offset; + data[i++] = *(u64 *)p; + } + } + for (j = 0; j < netdev->num_tx_queues; j++) { ring = wx->tx_ring[j]; if (!ring) { @@ -322,6 +343,40 @@ int wx_get_coalesce(struct net_device *netdev, } EXPORT_SYMBOL(wx_get_coalesce); +static void wx_update_rsc(struct wx *wx) +{ + struct net_device *netdev = wx->netdev; + bool need_reset = false; + + /* nothing to do if LRO or RSC are not enabled */ + if (!test_bit(WX_FLAG_RSC_CAPABLE, wx->flags) || + !(netdev->features & NETIF_F_LRO)) + return; + + /* check the feature flag value and enable RSC if necessary */ + if (wx->rx_itr_setting == 1 || + wx->rx_itr_setting > WX_MIN_RSC_ITR) { + if (!test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) { + set_bit(WX_FLAG_RSC_ENABLED, wx->flags); + dev_info(&wx->pdev->dev, + "rx-usecs value high enough to re-enable RSC\n"); + + need_reset = true; + } + /* if interrupt rate is too high then disable RSC */ + } else if (test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) { + clear_bit(WX_FLAG_RSC_ENABLED, wx->flags); + dev_info(&wx->pdev->dev, + "rx-usecs set too low, disabling RSC\n"); + + need_reset = true; + } + + /* reset the device to apply the new RSC setting */ + if (need_reset && wx->do_reset) + wx->do_reset(netdev); +} + int wx_set_coalesce(struct net_device *netdev, struct ethtool_coalesce *ec, struct kernel_ethtool_coalesce *kernel_coal, @@ -414,6 +469,8 @@ int wx_set_coalesce(struct net_device *netdev, wx_write_eitr(q_vector); } + wx_update_rsc(wx); + return 0; } EXPORT_SYMBOL(wx_set_coalesce); diff --git a/drivers/net/ethernet/wangxun/libwx/wx_hw.c b/drivers/net/ethernet/wangxun/libwx/wx_hw.c index be59a1022890..dece90d3469e 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_hw.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_hw.c @@ -1779,7 +1779,9 @@ EXPORT_SYMBOL(wx_set_rx_mode); static void wx_set_rx_buffer_len(struct wx *wx) { struct net_device *netdev = wx->netdev; + struct wx_ring *rx_ring; u32 mhadd, max_frame; + int i; max_frame = netdev->mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN; /* adjust max frame to be at least the size of a standard frame */ @@ -1789,6 +1791,19 @@ static void wx_set_rx_buffer_len(struct wx *wx) mhadd = rd32(wx, WX_PSR_MAX_SZ); if (max_frame != mhadd) wr32(wx, WX_PSR_MAX_SZ, max_frame); + + /* + * Setup the HW Rx Head and Tail Descriptor Pointers and + * the Base and Length of the Rx Descriptor Ring + */ + for (i = 0; i < wx->num_rx_queues; i++) { + rx_ring = wx->rx_ring[i]; + rx_ring->rx_buf_len = WX_RXBUFFER_2K; +#if (PAGE_SIZE < 8192) + if (test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) + rx_ring->rx_buf_len = WX_RXBUFFER_3K; +#endif + } } /** @@ -1865,11 +1880,27 @@ static void wx_configure_srrctl(struct wx *wx, srrctl |= WX_RXBUFFER_256 << WX_PX_RR_CFG_BHDRSIZE_SHIFT; /* configure the packet buffer length */ - srrctl |= WX_RX_BUFSZ >> WX_PX_RR_CFG_BSIZEPKT_SHIFT; + srrctl |= rx_ring->rx_buf_len >> WX_PX_RR_CFG_BSIZEPKT_SHIFT; wr32(wx, WX_PX_RR_CFG(reg_idx), srrctl); } +static void wx_configure_rscctl(struct wx *wx, + struct wx_ring *ring) +{ + u8 reg_idx = ring->reg_idx; + u32 rscctrl; + + if (!test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) + return; + + rscctrl = rd32(wx, WX_PX_RR_CFG(reg_idx)); + rscctrl |= WX_PX_RR_CFG_RSC; + rscctrl |= WX_PX_RR_CFG_MAX_RSCBUF_16; + + wr32(wx, WX_PX_RR_CFG(reg_idx), rscctrl); +} + static void wx_configure_tx_ring(struct wx *wx, struct wx_ring *ring) { @@ -1935,6 +1966,10 @@ static void wx_configure_rx_ring(struct wx *wx, rxdctl |= (ring->count / 128) << WX_PX_RR_CFG_RR_SIZE_SHIFT; rxdctl |= 0x1 << WX_PX_RR_CFG_RR_THER_SHIFT; + + if (test_bit(WX_FLAG_RX_MERGE_ENABLED, wx->flags)) + rxdctl |= WX_PX_RR_CFG_DESC_MERGE; + wr32(wx, WX_PX_RR_CFG(reg_idx), rxdctl); /* reset head and tail pointers */ @@ -1943,6 +1978,7 @@ static void wx_configure_rx_ring(struct wx *wx, ring->tail = wx->hw_addr + WX_PX_RR_WP(reg_idx); wx_configure_srrctl(wx, ring); + wx_configure_rscctl(wx, ring); /* initialize rx_buffer_info */ memset(ring->rx_buffer_info, 0, @@ -2181,7 +2217,9 @@ void wx_configure_rx(struct wx *wx) /* RSC Setup */ psrctl = rd32(wx, WX_PSR_CTL); psrctl |= WX_PSR_CTL_RSC_ACK; /* Disable RSC for ACK packets */ - psrctl |= WX_PSR_CTL_RSC_DIS; + psrctl &= ~WX_PSR_CTL_RSC_DIS; + if (!test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) + psrctl |= WX_PSR_CTL_RSC_DIS; wr32(wx, WX_PSR_CTL, psrctl); } @@ -2190,6 +2228,12 @@ void wx_configure_rx(struct wx *wx) /* set_rx_buffer_len must be called before ring initialization */ wx_set_rx_buffer_len(wx); + if (test_bit(WX_FLAG_RX_MERGE_ENABLED, wx->flags)) { + wr32(wx, WX_RDM_DCACHE_CTL, WX_RDM_DCACHE_CTL_EN); + wr32m(wx, WX_RDM_RSC_CTL, + WX_RDM_RSC_CTL_FREE_CTL | WX_RDM_RSC_CTL_FREE_CNT_DIS, + WX_RDM_RSC_CTL_FREE_CTL); + } /* Setup the HW Rx Head and Tail Descriptor Pointers and * the Base and Length of the Rx Descriptor Ring */ @@ -2464,8 +2508,11 @@ int wx_sw_init(struct wx *wx) return -ENOMEM; } + spin_lock_init(&wx->hw_stats_lock); + mutex_init(&wx->reset_lock); bitmap_zero(wx->state, WX_STATE_NBITS); bitmap_zero(wx->flags, WX_PF_FLAGS_NBITS); + set_bit(WX_STATE_DOWN, wx->state); wx->misc_irq_domain = false; return 0; @@ -2795,6 +2842,12 @@ void wx_update_stats(struct wx *wx) u64 restart_queue = 0, tx_busy = 0; u32 i; + if (test_bit(WX_STATE_DOWN, wx->state) || + test_bit(WX_STATE_RESETTING, wx->state)) + return; + + spin_lock(&wx->hw_stats_lock); + /* gather some stats to the wx struct that are per queue */ for (i = 0; i < wx->num_rx_queues; i++) { struct wx_ring *rx_ring = wx->rx_ring[i]; @@ -2809,6 +2862,18 @@ void wx_update_stats(struct wx *wx) wx->hw_csum_rx_error = hw_csum_rx_error; wx->hw_csum_rx_good = hw_csum_rx_good; + if (test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) { + u64 rsc_count = 0; + u64 rsc_flush = 0; + + for (i = 0; i < wx->num_rx_queues; i++) { + rsc_count += wx->rx_ring[i]->rx_stats.rsc_count; + rsc_flush += wx->rx_ring[i]->rx_stats.rsc_flush; + } + wx->rsc_count = rsc_count; + wx->rsc_flush = rsc_flush; + } + for (i = 0; i < wx->num_tx_queues; i++) { struct wx_ring *tx_ring = wx->tx_ring[i]; @@ -2851,6 +2916,8 @@ void wx_update_stats(struct wx *wx) for (i = wx->num_vfs * wx->num_rx_queues_per_pool; i < wx->mac.max_rx_queues; i++) hwstats->qmprc += rd32(wx, WX_PX_MPRC(i)); + + spin_unlock(&wx->hw_stats_lock); } EXPORT_SYMBOL(wx_update_stats); diff --git a/drivers/net/ethernet/wangxun/libwx/wx_lib.c b/drivers/net/ethernet/wangxun/libwx/wx_lib.c index 380edc64e45e..4a6851612368 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_lib.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_lib.c @@ -235,7 +235,7 @@ static struct sk_buff *wx_build_skb(struct wx_ring *rx_ring, { unsigned int size = le16_to_cpu(rx_desc->wb.upper.length); #if (PAGE_SIZE < 8192) - unsigned int truesize = WX_RX_BUFSZ; + unsigned int truesize = wx_rx_pg_size(rx_ring) / 2; #else unsigned int truesize = ALIGN(size, L1_CACHE_BYTES); #endif @@ -341,7 +341,7 @@ void wx_alloc_rx_buffers(struct wx_ring *rx_ring, u16 cleaned_count) /* sync the buffer for use by the device */ dma_sync_single_range_for_device(rx_ring->dev, bi->dma, bi->page_offset, - WX_RX_BUFSZ, + rx_ring->rx_buf_len, DMA_FROM_DEVICE); rx_desc->read.pkt_addr = @@ -404,6 +404,7 @@ static bool wx_is_non_eop(struct wx_ring *rx_ring, union wx_rx_desc *rx_desc, struct sk_buff *skb) { + struct wx *wx = rx_ring->q_vector->wx; u32 ntc = rx_ring->next_to_clean + 1; /* fetch, update, and store next to clean */ @@ -412,6 +413,24 @@ static bool wx_is_non_eop(struct wx_ring *rx_ring, prefetch(WX_RX_DESC(rx_ring, ntc)); + /* update RSC append count if present */ + if (test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) { + __le32 rsc_enabled = rx_desc->wb.lower.lo_dword.data & + cpu_to_le32(WX_RXD_RSCCNT_MASK); + + if (unlikely(rsc_enabled)) { + u32 rsc_cnt = le32_to_cpu(rsc_enabled); + + rsc_cnt >>= WX_RXD_RSCCNT_SHIFT; + WX_CB(skb)->append_cnt += rsc_cnt - 1; + + /* update ntc based on RSC value */ + ntc = le32_to_cpu(rx_desc->wb.upper.status_error); + ntc &= WX_RXD_NEXTP_MASK; + ntc >>= WX_RXD_NEXTP_SHIFT; + } + } + /* if we are the last buffer then there is nothing else to do */ if (likely(wx_test_staterr(rx_desc, WX_RXD_STAT_EOP))) return false; @@ -582,6 +601,33 @@ static void wx_rx_vlan(struct wx_ring *ring, union wx_rx_desc *rx_desc, } } +static void wx_set_rsc_gso_size(struct wx_ring *ring, + struct sk_buff *skb) +{ + u16 hdr_len = skb_headlen(skb); + + /* set gso_size to avoid messing up TCP MSS */ + skb_shinfo(skb)->gso_size = DIV_ROUND_UP((skb->len - hdr_len), + WX_CB(skb)->append_cnt); + skb_shinfo(skb)->gso_type = SKB_GSO_TCPV4; +} + +static void wx_update_rsc_stats(struct wx_ring *rx_ring, + struct sk_buff *skb) +{ + /* if append_cnt is 0 then frame is not RSC */ + if (!WX_CB(skb)->append_cnt) + return; + + rx_ring->rx_stats.rsc_count += WX_CB(skb)->append_cnt; + rx_ring->rx_stats.rsc_flush++; + + wx_set_rsc_gso_size(rx_ring, skb); + + /* gso_size is computed using append_cnt so always clear it last */ + WX_CB(skb)->append_cnt = 0; +} + /** * wx_process_skb_fields - Populate skb header fields from Rx descriptor * @rx_ring: rx descriptor ring packet is being transacted on @@ -598,6 +644,9 @@ static void wx_process_skb_fields(struct wx_ring *rx_ring, { struct wx *wx = netdev_priv(rx_ring->netdev); + if (test_bit(WX_FLAG_RSC_CAPABLE, wx->flags)) + wx_update_rsc_stats(rx_ring, skb); + wx_rx_hash(rx_ring, rx_desc, skb); wx_rx_checksum(rx_ring, rx_desc, skb); @@ -823,7 +872,7 @@ static bool wx_clean_tx_irq(struct wx_q_vector *q_vector, if (__netif_subqueue_stopped(tx_ring->netdev, tx_ring->queue_index) && - netif_running(tx_ring->netdev)) { + !test_bit(WX_STATE_DOWN, wx->state)) { netif_wake_subqueue(tx_ring->netdev, tx_ring->queue_index); ++tx_ring->tx_stats.restart_queue; @@ -911,7 +960,7 @@ static int wx_poll(struct napi_struct *napi, int budget) if (likely(napi_complete_done(napi, work_done))) { if (wx->adaptive_itr) wx_update_dim_sample(q_vector); - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) wx_intr_enable(wx, WX_INTR_Q(q_vector->v_idx)); } @@ -2290,6 +2339,8 @@ int wx_init_interrupt_scheme(struct wx *wx) wx_cache_ring_rss(wx); + set_bit(WX_STATE_DOWN, wx->state); + return 0; } EXPORT_SYMBOL(wx_init_interrupt_scheme); @@ -2535,7 +2586,7 @@ static void wx_clean_rx_ring(struct wx_ring *rx_ring) dma_sync_single_range_for_cpu(rx_ring->dev, rx_buffer->dma, rx_buffer->page_offset, - WX_RX_BUFSZ, + rx_ring->rx_buf_len, DMA_FROM_DEVICE); /* free resources associated with mapping */ @@ -2734,13 +2785,14 @@ static int wx_alloc_page_pool(struct wx_ring *rx_ring) struct page_pool_params pp_params = { .flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV, - .order = 0, - .pool_size = rx_ring->count, + .order = wx_rx_pg_order(rx_ring), + .pool_size = rx_ring->count * rx_ring->rx_buf_len / + wx_rx_pg_size(rx_ring), .nid = dev_to_node(rx_ring->dev), .dev = rx_ring->dev, .dma_dir = DMA_FROM_DEVICE, .offset = 0, - .max_len = PAGE_SIZE, + .max_len = wx_rx_pg_size(rx_ring), }; rx_ring->page_pool = page_pool_create(&pp_params); @@ -3029,8 +3081,25 @@ int wx_set_features(struct net_device *netdev, netdev_features_t features) else if (changed & (NETIF_F_HW_VLAN_CTAG_RX | NETIF_F_HW_VLAN_CTAG_FILTER)) wx_set_rx_mode(netdev); + if (test_bit(WX_FLAG_RSC_CAPABLE, wx->flags)) { + if (!(features & NETIF_F_LRO)) { + if (test_bit(WX_FLAG_RSC_ENABLED, wx->flags)) + need_reset = true; + clear_bit(WX_FLAG_RSC_ENABLED, wx->flags); + } else if (!(test_bit(WX_FLAG_RSC_ENABLED, wx->flags))) { + if (wx->rx_itr_setting == 1 || + wx->rx_itr_setting > WX_MIN_RSC_ITR) { + set_bit(WX_FLAG_RSC_ENABLED, wx->flags); + need_reset = true; + } else if (changed & NETIF_F_LRO) { + dev_info(&wx->pdev->dev, + "rx-usecs set too low, disable RSC\n"); + } + } + } + if (!(test_bit(WX_FLAG_FDIR_CAPABLE, wx->flags))) - return 0; + goto out; /* Check if Flow Director n-tuple support was enabled or disabled. If * the state changed, we need to reset. @@ -3056,6 +3125,7 @@ int wx_set_features(struct net_device *netdev, netdev_features_t features) break; } +out: if (need_reset && wx->do_reset) wx->do_reset(netdev); @@ -3105,6 +3175,14 @@ netdev_features_t wx_fix_features(struct net_device *netdev, } } + /* If Rx checksum is disabled, then RSC/LRO should also be disabled */ + if (!(features & NETIF_F_RXCSUM)) + features &= ~NETIF_F_LRO; + + /* Turn off LRO if not RSC capable */ + if (!test_bit(WX_FLAG_RSC_CAPABLE, wx->flags)) + features &= ~NETIF_F_LRO; + return features; } EXPORT_SYMBOL(wx_fix_features); @@ -3204,7 +3282,8 @@ EXPORT_SYMBOL(wx_set_ring); void wx_service_event_schedule(struct wx *wx) { - if (!test_and_set_bit(WX_STATE_SERVICE_SCHED, wx->state)) + if (!test_bit(WX_STATE_DOWN, wx->state) && + !test_and_set_bit(WX_STATE_SERVICE_SCHED, wx->state)) queue_work(system_power_efficient_wq, &wx->service_task); } EXPORT_SYMBOL(wx_service_event_schedule); diff --git a/drivers/net/ethernet/wangxun/libwx/wx_ptp.c b/drivers/net/ethernet/wangxun/libwx/wx_ptp.c index 44f3e6505246..1165518d5522 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_ptp.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_ptp.c @@ -555,13 +555,11 @@ static int wx_ptp_set_timestamp_mode(struct wx *wx, { u32 tsync_tx_ctl = WX_TSC_1588_CTL_ENABLED; u32 tsync_rx_ctl = WX_PSR_1588_CTL_ENABLED; - DECLARE_BITMAP(flags, WX_PF_FLAGS_NBITS); u32 tsync_rx_mtrl = PTP_EV_PORT << 16; + bool rx_tstamp = false; bool is_l2 = false; u32 regval; - memcpy(flags, wx->flags, sizeof(wx->flags)); - switch (config->tx_type) { case HWTSTAMP_TX_OFF: tsync_tx_ctl = 0; @@ -576,20 +574,16 @@ static int wx_ptp_set_timestamp_mode(struct wx *wx, case HWTSTAMP_FILTER_NONE: tsync_rx_ctl = 0; tsync_rx_mtrl = 0; - clear_bit(WX_FLAG_RX_HWTSTAMP_ENABLED, flags); - clear_bit(WX_FLAG_RX_HWTSTAMP_IN_REGISTER, flags); break; case HWTSTAMP_FILTER_PTP_V1_L4_SYNC: tsync_rx_ctl |= WX_PSR_1588_CTL_TYPE_L4_V1; tsync_rx_mtrl |= WX_PSR_1588_MSG_V1_SYNC; - set_bit(WX_FLAG_RX_HWTSTAMP_ENABLED, flags); - set_bit(WX_FLAG_RX_HWTSTAMP_IN_REGISTER, flags); + rx_tstamp = true; break; case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ: tsync_rx_ctl |= WX_PSR_1588_CTL_TYPE_L4_V1; tsync_rx_mtrl |= WX_PSR_1588_MSG_V1_DELAY_REQ; - set_bit(WX_FLAG_RX_HWTSTAMP_ENABLED, flags); - set_bit(WX_FLAG_RX_HWTSTAMP_IN_REGISTER, flags); + rx_tstamp = true; break; case HWTSTAMP_FILTER_PTP_V2_EVENT: case HWTSTAMP_FILTER_PTP_V2_L2_EVENT: @@ -602,9 +596,8 @@ static int wx_ptp_set_timestamp_mode(struct wx *wx, case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ: tsync_rx_ctl |= WX_PSR_1588_CTL_TYPE_EVENT_V2; is_l2 = true; + rx_tstamp = true; config->rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT; - set_bit(WX_FLAG_RX_HWTSTAMP_ENABLED, flags); - set_bit(WX_FLAG_RX_HWTSTAMP_IN_REGISTER, flags); break; default: /* register PSR_1588_MSG must be set in order to do V1 packets, @@ -643,7 +636,8 @@ static int wx_ptp_set_timestamp_mode(struct wx *wx, WX_WRITE_FLUSH(wx); /* configure adapter flags only when HW is actually configured */ - memcpy(wx->flags, flags, sizeof(wx->flags)); + assign_bit(WX_FLAG_RX_HWTSTAMP_ENABLED, wx->flags, rx_tstamp); + assign_bit(WX_FLAG_RX_HWTSTAMP_IN_REGISTER, wx->flags, rx_tstamp); /* clear TX/RX timestamp state, just to be sure */ wx_ptp_clear_tx_timestamp(wx); diff --git a/drivers/net/ethernet/wangxun/libwx/wx_sriov.c b/drivers/net/ethernet/wangxun/libwx/wx_sriov.c index c6d158cd70da..6ce5d5936d58 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_sriov.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_sriov.c @@ -122,6 +122,10 @@ static int __wx_enable_sriov(struct wx *wx, u8 num_vfs) WX_CFG_PORT_CTL_NUM_VT_MASK, value); + /* Disable RSC when in SR-IOV mode */ + clear_bit(WX_FLAG_RSC_CAPABLE, wx->flags); + clear_bit(WX_FLAG_RSC_ENABLED, wx->flags); + return ret; } @@ -896,7 +900,7 @@ static void wx_set_vf_link_state(struct wx *wx, int vf, int state) wx->vfinfo[vf].link_state = state; switch (state) { case IFLA_VF_LINK_STATE_AUTO: - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) wx->vfinfo[vf].link_enable = true; else wx->vfinfo[vf].link_enable = false; diff --git a/drivers/net/ethernet/wangxun/libwx/wx_type.h b/drivers/net/ethernet/wangxun/libwx/wx_type.h index 12bb1e19d88d..d66ea92acead 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_type.h +++ b/drivers/net/ethernet/wangxun/libwx/wx_type.h @@ -83,8 +83,13 @@ /*********************** Receive DMA registers **************************/ #define WX_RDM_VF_RE(_i) (0x12004 + ((_i) * 4)) +#define WX_RDM_RSC_CTL 0x1200C +#define WX_RDM_RSC_CTL_FREE_CNT_DIS BIT(8) +#define WX_RDM_RSC_CTL_FREE_CTL BIT(7) #define WX_RDM_PF_QDE(_i) (0x12080 + ((_i) * 4)) #define WX_RDM_VFRE_CLR(_i) (0x120A0 + ((_i) * 4)) +#define WX_RDM_DCACHE_CTL 0x120A8 +#define WX_RDM_DCACHE_CTL_EN BIT(0) #define WX_RDM_DRP_PKT 0x12500 #define WX_RDM_PKT_CNT 0x12504 #define WX_RDM_BYTE_CNT_LSB 0x12508 @@ -421,6 +426,7 @@ enum WX_MSCA_CMD_value { #define WX_7K_ITR 595 #define WX_12K_ITR 336 #define WX_20K_ITR 200 +#define WX_MIN_RSC_ITR 24 #define WX_SP_MAX_EITR 0x00000FF8U #define WX_AML_MAX_EITR 0x00000FFFU #define WX_EM_MAX_EITR 0x00007FFCU @@ -448,7 +454,10 @@ enum WX_MSCA_CMD_value { /* PX_RR_CFG bit definitions */ #define WX_PX_RR_CFG_VLAN BIT(31) #define WX_PX_RR_CFG_DROP_EN BIT(30) +#define WX_PX_RR_CFG_RSC BIT(29) #define WX_PX_RR_CFG_SPLIT_MODE BIT(26) +#define WX_PX_RR_CFG_MAX_RSCBUF_16 FIELD_PREP(GENMASK(24, 23), 3) +#define WX_PX_RR_CFG_DESC_MERGE BIT(19) #define WX_PX_RR_CFG_RR_THER_SHIFT 16 #define WX_PX_RR_CFG_RR_HDR_SZ GENMASK(15, 12) #define WX_PX_RR_CFG_RR_BUF_SZ GENMASK(11, 8) @@ -545,14 +554,9 @@ enum WX_MSCA_CMD_value { /* Supported Rx Buffer Sizes */ #define WX_RXBUFFER_256 256 /* Used for skb receive header */ #define WX_RXBUFFER_2K 2048 +#define WX_RXBUFFER_3K 3072 #define WX_MAX_RXBUFFER 16384 /* largest size for single descriptor */ -#if MAX_SKB_FRAGS < 8 -#define WX_RX_BUFSZ ALIGN(WX_MAX_RXBUFFER / MAX_SKB_FRAGS, 1024) -#else -#define WX_RX_BUFSZ WX_RXBUFFER_2K -#endif - #define WX_RX_BUFFER_WRITE 16 /* Must be power of 2 */ #define WX_MAX_DATA_PER_TXD BIT(14) @@ -644,6 +648,12 @@ enum wx_l2_ptypes { #define WX_RXD_PKTTYPE(_rxd) \ ((le32_to_cpu((_rxd)->wb.lower.lo_dword.data) >> 9) & 0xFF) + +#define WX_RXD_RSCCNT_MASK GENMASK(20, 17) +#define WX_RXD_RSCCNT_SHIFT 17 +#define WX_RXD_NEXTP_MASK GENMASK(19, 4) +#define WX_RXD_NEXTP_SHIFT 4 + /*********************** Transmit Descriptor Config Masks ****************/ #define WX_TXD_STAT_DD BIT(0) /* Descriptor Done */ #define WX_TXD_DTYP_DATA 0 /* Adv Data Descriptor */ @@ -1030,6 +1040,8 @@ struct wx_rx_queue_stats { u64 csum_good_cnt; u64 csum_err; u64 alloc_rx_buff_failed; + u64 rsc_count; + u64 rsc_flush; }; /* iterator for handling rings in ring container */ @@ -1070,6 +1082,7 @@ struct wx_ring { */ u16 next_to_use; u16 next_to_clean; + u16 rx_buf_len; union { u16 next_to_alloc; struct { @@ -1165,6 +1178,7 @@ struct wx_hw_stats { }; enum wx_state { + WX_STATE_DOWN, WX_STATE_RESETTING, WX_STATE_SWFW_BUSY, WX_STATE_PTP_RUNNING, @@ -1226,6 +1240,7 @@ enum wx_pf_flags { WX_FLAG_FDIR_HASH, WX_FLAG_FDIR_PERFECT, WX_FLAG_RSC_CAPABLE, + WX_FLAG_RSC_ENABLED, WX_FLAG_RX_HWTSTAMP_ENABLED, WX_FLAG_RX_HWTSTAMP_IN_REGISTER, WX_FLAG_PTP_PPS_ENABLED, @@ -1233,6 +1248,7 @@ enum wx_pf_flags { WX_FLAG_NEED_MODULE_RESET, WX_FLAG_NEED_UPDATE_LINK, WX_FLAG_NEED_DO_RESET, + WX_FLAG_RX_MERGE_ENABLED, WX_PF_FLAGS_NBITS /* must be last */ }; @@ -1333,12 +1349,15 @@ struct wx { bool default_up; struct wx_hw_stats stats; + spinlock_t hw_stats_lock; /* spinlock for accessing to hw stats */ u64 tx_busy; u64 non_eop_descs; u64 restart_queue; u64 hw_csum_rx_good; u64 hw_csum_rx_error; u64 alloc_rx_buff_failed; + u64 rsc_count; + u64 rsc_flush; unsigned int num_vfs; struct vf_data_storage *vfinfo; struct vf_macvlans vf_mvs; @@ -1377,10 +1396,11 @@ struct wx { struct timer_list service_timer; struct work_struct service_task; + struct mutex reset_lock; /* mutex for reset */ }; #define WX_INTR_ALL (~0ULL) -#define WX_INTR_Q(i) BIT((i)) +#define WX_INTR_Q(i) BIT_ULL((i)) /* register operations */ #define wr32(a, reg, value) writel((value), ((a)->hw_addr + (reg))) @@ -1455,19 +1475,15 @@ static inline struct wx *phylink_to_wx(struct phylink_config *config) return container_of(config, struct wx, phylink_config); } -static inline int wx_set_state_reset(struct wx *wx) +static inline unsigned int wx_rx_pg_order(struct wx_ring *ring) { - u8 timeout = 50; - - while (test_and_set_bit(WX_STATE_RESETTING, wx->state)) { - timeout--; - if (!timeout) - return -EBUSY; - - usleep_range(1000, 2000); - } - +#if (PAGE_SIZE < 8192) + if (ring->rx_buf_len == WX_RXBUFFER_3K) + return 1; +#endif return 0; } +#define wx_rx_pg_size(_ring) (PAGE_SIZE << wx_rx_pg_order(_ring)) + #endif /* _WX_TYPE_H_ */ diff --git a/drivers/net/ethernet/wangxun/libwx/wx_vf.h b/drivers/net/ethernet/wangxun/libwx/wx_vf.h index 3f16de0fa427..ecb198592393 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_vf.h +++ b/drivers/net/ethernet/wangxun/libwx/wx_vf.h @@ -74,6 +74,7 @@ #define WX_VXRXDCTL_BUFSZ(f) FIELD_PREP(GENMASK(11, 8), f) #define WX_VXRXDCTL_HDRSZ_MASK GENMASK(15, 12) #define WX_VXRXDCTL_HDRSZ(f) FIELD_PREP(GENMASK(15, 12), f) +#define WX_VXRXDCTL_DESC_MERGE BIT(19) #define WX_VXRXDCTL_RSCMAX_MASK GENMASK(24, 23) #define WX_VXRXDCTL_RSCMAX(f) FIELD_PREP(GENMASK(24, 23), f) #define WX_VXRXDCTL_RSCEN BIT(29) diff --git a/drivers/net/ethernet/wangxun/libwx/wx_vf_common.c b/drivers/net/ethernet/wangxun/libwx/wx_vf_common.c index 5478f2fdfce8..11ee2436b9c9 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_vf_common.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_vf_common.c @@ -67,7 +67,7 @@ static irqreturn_t wx_msix_misc_vf(int __always_unused irq, void *data) set_bit(WX_FLAG_NEED_UPDATE_LINK, wx->flags); /* Clear the interrupt */ - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) wr32(wx, WX_VXIMC, wx->eims_other); return IRQ_HANDLED; @@ -277,6 +277,7 @@ static void wxvf_up_complete(struct wx *wx) wx_configure_msix_vf(wx); smp_mb__before_atomic(); + clear_bit(WX_STATE_DOWN, wx->state); wx_napi_enable_all(wx); /* clear any pending interrupts, may auto mask */ @@ -326,6 +327,9 @@ static void wxvf_down(struct wx *wx) { struct net_device *netdev = wx->netdev; + if (test_and_set_bit(WX_STATE_DOWN, wx->state)) + return; + timer_delete_sync(&wx->service_timer); netif_tx_stop_all_queues(netdev); netif_tx_disable(netdev); @@ -339,14 +343,16 @@ static void wxvf_down(struct wx *wx) static void wxvf_reinit_locked(struct wx *wx) { - while (test_and_set_bit(WX_STATE_RESETTING, wx->state)) - usleep_range(1000, 2000); + mutex_lock(&wx->reset_lock); + set_bit(WX_STATE_RESETTING, wx->state); + wxvf_down(wx); wx_free_irq(wx); wx_configure_vf(wx); wx_request_msix_irqs_vf(wx); wxvf_up_complete(wx); clear_bit(WX_STATE_RESETTING, wx->state); + mutex_unlock(&wx->reset_lock); } static void wxvf_reset_subtask(struct wx *wx) @@ -357,7 +363,7 @@ static void wxvf_reset_subtask(struct wx *wx) rtnl_lock(); if (test_bit(WX_STATE_RESETTING, wx->state) || - !(netif_running(wx->netdev))) { + test_bit(WX_STATE_DOWN, wx->state)) { rtnl_unlock(); return; } diff --git a/drivers/net/ethernet/wangxun/libwx/wx_vf_lib.c b/drivers/net/ethernet/wangxun/libwx/wx_vf_lib.c index a87887b9f8ee..f54107f3c6d7 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_vf_lib.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_vf_lib.c @@ -272,6 +272,9 @@ void wx_configure_rx_ring_vf(struct wx *wx, struct wx_ring *ring) rxdctl |= WX_VXRXDCTL_RSCMAX(0); rxdctl |= WX_VXRXDCTL_RSCEN; + if (test_bit(WX_FLAG_RX_MERGE_ENABLED, wx->flags)) + rxdctl |= WX_VXRXDCTL_DESC_MERGE; + wr32(wx, WX_VXRXDCTL(reg_idx), rxdctl); /* pf/vf reuse */ diff --git a/drivers/net/ethernet/wangxun/ngbe/ngbe_ethtool.c b/drivers/net/ethernet/wangxun/ngbe/ngbe_ethtool.c index 662f28bdde8a..646977521d39 100644 --- a/drivers/net/ethernet/wangxun/ngbe/ngbe_ethtool.c +++ b/drivers/net/ethernet/wangxun/ngbe/ngbe_ethtool.c @@ -64,9 +64,8 @@ static int ngbe_set_ringparam(struct net_device *netdev, new_rx_count == wx->rx_ring_count) return 0; - err = wx_set_state_reset(wx); - if (err) - return err; + mutex_lock(&wx->reset_lock); + set_bit(WX_STATE_RESETTING, wx->state); if (!netif_running(wx->netdev)) { for (i = 0; i < wx->num_tx_queues; i++) @@ -97,6 +96,7 @@ static int ngbe_set_ringparam(struct net_device *netdev, clear_reset: clear_bit(WX_STATE_RESETTING, wx->state); + mutex_unlock(&wx->reset_lock); return err; } diff --git a/drivers/net/ethernet/wangxun/ngbe/ngbe_main.c b/drivers/net/ethernet/wangxun/ngbe/ngbe_main.c index e80962d40f7e..35d7a0c76ca0 100644 --- a/drivers/net/ethernet/wangxun/ngbe/ngbe_main.c +++ b/drivers/net/ethernet/wangxun/ngbe/ngbe_main.c @@ -138,6 +138,26 @@ static int ngbe_sw_init(struct wx *wx) return 0; } +/** + * ngbe_service_task - manages and runs subtasks + * @work: pointer to work_struct containing our data + **/ +static void ngbe_service_task(struct work_struct *work) +{ + struct wx *wx = container_of(work, struct wx, service_task); + + wx_update_stats(wx); + + wx_service_event_complete(wx); +} + +static void ngbe_init_service(struct wx *wx) +{ + timer_setup(&wx->service_timer, wx_service_timer, 0); + INIT_WORK(&wx->service_task, ngbe_service_task); + clear_bit(WX_STATE_SERVICE_SCHED, wx->state); +} + /** * ngbe_irq_enable - Enable default interrupt generation settings * @wx: board private structure @@ -186,7 +206,7 @@ static irqreturn_t ngbe_intr(int __always_unused irq, void *data) /* shared interrupt alert! * the interrupt that we masked before the EICR read. */ - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) ngbe_irq_enable(wx, true); return IRQ_NONE; /* Not our interrupt */ } @@ -202,7 +222,7 @@ static irqreturn_t ngbe_intr(int __always_unused irq, void *data) /* would disable interrupts here but it is auto disabled */ napi_schedule_irqoff(&q_vector->napi); - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) ngbe_irq_enable(wx, false); return IRQ_HANDLED; @@ -217,7 +237,7 @@ static irqreturn_t __ngbe_msix_misc(struct wx *wx, u32 eicr) wx_ptp_check_pps_event(wx); /* re-enable the original interrupt state, no lsc, no queues */ - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) ngbe_irq_enable(wx, false); return IRQ_HANDLED; @@ -244,7 +264,7 @@ static irqreturn_t ngbe_misc_and_queue(int __always_unused irq, void *data) /* queue */ q_vector = wx->q_vector[0]; napi_schedule_irqoff(&q_vector->napi); - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) ngbe_irq_enable(wx, true); return IRQ_HANDLED; } @@ -345,6 +365,9 @@ static void ngbe_disable_device(struct wx *wx) struct net_device *netdev = wx->netdev; u32 i; + if (test_and_set_bit(WX_STATE_DOWN, wx->state)) + return; + if (wx->num_vfs) { /* Clear EITR Select mapping */ wr32(wx, WX_PX_ITRSEL, 0); @@ -370,6 +393,10 @@ static void ngbe_disable_device(struct wx *wx) wx_napi_disable_all(wx); netif_tx_stop_all_queues(netdev); netif_tx_disable(netdev); + + timer_delete_sync(&wx->service_timer); + cancel_work_sync(&wx->service_task); + if (wx->gpio_ctrl) ngbe_sfp_modules_txrx_powerctl(wx, false); wx_irq_disable(wx); @@ -406,9 +433,11 @@ void ngbe_up(struct wx *wx) /* make sure to complete pre-operations */ smp_mb__before_atomic(); + clear_bit(WX_STATE_DOWN, wx->state); wx_napi_enable_all(wx); /* enable transmits */ netif_tx_start_all_queues(wx->netdev); + mod_timer(&wx->service_timer, jiffies); /* clear any pending interrupts, may auto mask */ rd32(wx, WX_PX_IC(0)); @@ -772,9 +801,11 @@ static int ngbe_probe(struct pci_dev *pdev, eth_hw_addr_set(netdev, wx->mac.perm_addr); wx_mac_set_default_filter(wx, wx->mac.perm_addr); + ngbe_init_service(wx); + err = wx_init_interrupt_scheme(wx); if (err) - goto err_free_mac_table; + goto err_cancel_service; /* phy Interface Configuration */ err = ngbe_mdio_init(wx); @@ -794,6 +825,9 @@ static int ngbe_probe(struct pci_dev *pdev, wx_control_hw(wx, false); err_clear_interrupt_scheme: wx_clear_interrupt_scheme(wx); +err_cancel_service: + timer_delete_sync(&wx->service_timer); + cancel_work_sync(&wx->service_task); err_free_mac_table: kfree(wx->rss_key); kfree(wx->mac_table); @@ -822,6 +856,10 @@ static void ngbe_remove(struct pci_dev *pdev) netdev = wx->netdev; wx_disable_sriov(wx); unregister_netdev(netdev); + + timer_shutdown_sync(&wx->service_timer); + cancel_work_sync(&wx->service_task); + phylink_destroy(wx->phylink); pci_release_selected_regions(pdev, pci_select_bars(pdev, IORESOURCE_MEM)); diff --git a/drivers/net/ethernet/wangxun/txgbe/txgbe_ethtool.c b/drivers/net/ethernet/wangxun/txgbe/txgbe_ethtool.c index d7f905359458..659006b4b828 100644 --- a/drivers/net/ethernet/wangxun/txgbe/txgbe_ethtool.c +++ b/drivers/net/ethernet/wangxun/txgbe/txgbe_ethtool.c @@ -58,9 +58,8 @@ static int txgbe_set_ringparam(struct net_device *netdev, new_rx_count == wx->rx_ring_count) return 0; - err = wx_set_state_reset(wx); - if (err) - return err; + mutex_lock(&wx->reset_lock); + set_bit(WX_STATE_RESETTING, wx->state); if (!netif_running(wx->netdev)) { for (i = 0; i < wx->num_tx_queues; i++) @@ -90,6 +89,7 @@ static int txgbe_set_ringparam(struct net_device *netdev, clear_reset: clear_bit(WX_STATE_RESETTING, wx->state); + mutex_unlock(&wx->reset_lock); return err; } diff --git a/drivers/net/ethernet/wangxun/txgbe/txgbe_irq.c b/drivers/net/ethernet/wangxun/txgbe/txgbe_irq.c index 3885283681ec..5ceca13eaa5b 100644 --- a/drivers/net/ethernet/wangxun/txgbe/txgbe_irq.c +++ b/drivers/net/ethernet/wangxun/txgbe/txgbe_irq.c @@ -141,7 +141,7 @@ static irqreturn_t txgbe_misc_irq_handle(int irq, void *data) /* shared interrupt alert! * the interrupt that we masked before the ICR read. */ - if (netif_running(wx->netdev)) + if (!test_bit(WX_STATE_DOWN, wx->state)) txgbe_irq_enable(wx, true); return IRQ_NONE; /* Not our interrupt */ } @@ -164,6 +164,7 @@ static irqreturn_t txgbe_misc_irq_thread_fn(int irq, void *data) struct wx *wx = txgbe->wx; unsigned int nhandled = 0; unsigned int sub_irq; + u64 misc_mask; u32 eicr; eicr = txgbe->eicr; @@ -183,7 +184,9 @@ static irqreturn_t txgbe_misc_irq_thread_fn(int irq, void *data) nhandled++; } - wx_intr_enable(wx, TXGBE_INTR_MISC(wx)); + misc_mask = wx->pdev->msix_enabled ? TXGBE_INTR_MISC(wx) : BIT(0); + if (!test_bit(WX_STATE_DOWN, wx->state)) + wx_intr_enable(wx, misc_mask); return (nhandled > 0 ? IRQ_HANDLED : IRQ_NONE); } diff --git a/drivers/net/ethernet/wangxun/txgbe/txgbe_main.c b/drivers/net/ethernet/wangxun/txgbe/txgbe_main.c index 76d2940a49fa..9d32c4a7f1e9 100644 --- a/drivers/net/ethernet/wangxun/txgbe/txgbe_main.c +++ b/drivers/net/ethernet/wangxun/txgbe/txgbe_main.c @@ -130,6 +130,7 @@ static void txgbe_service_task(struct work_struct *work) txgbe_module_detection_subtask(wx); txgbe_link_config_subtask(wx); + wx_update_stats(wx); wx_service_event_complete(wx); } @@ -151,6 +152,7 @@ static void txgbe_up_complete(struct wx *wx) /* make sure to complete pre-operations */ smp_mb__before_atomic(); + clear_bit(WX_STATE_DOWN, wx->state); wx_napi_enable_all(wx); switch (wx->mac.type) { @@ -217,6 +219,9 @@ static void txgbe_disable_device(struct wx *wx) struct net_device *netdev = wx->netdev; u32 i; + if (test_and_set_bit(WX_STATE_DOWN, wx->state)) + return; + wx_disable_pcie_master(wx); /* disable receives */ wx_disable_rx(wx); @@ -398,6 +403,7 @@ static int txgbe_sw_init(struct wx *wx) wx->configure_fdir = txgbe_configure_fdir; set_bit(WX_FLAG_RSC_CAPABLE, wx->flags); + set_bit(WX_FLAG_RSC_ENABLED, wx->flags); set_bit(WX_FLAG_MULTI_64_FUNC, wx->flags); /* enable itr by default in dynamic mode */ @@ -423,6 +429,7 @@ static int txgbe_sw_init(struct wx *wx) break; case wx_mac_aml: case wx_mac_aml40: + set_bit(WX_FLAG_RX_MERGE_ENABLED, wx->flags); set_bit(WX_FLAG_SWFW_RING, wx->flags); wx->swfw_index = 0; break; @@ -600,20 +607,16 @@ int txgbe_setup_tc(struct net_device *dev, u8 tc) static void txgbe_reinit_locked(struct wx *wx) { - int err = 0; - netif_trans_update(wx->netdev); - err = wx_set_state_reset(wx); - if (err) { - wx_err(wx, "wait device reset timeout\n"); - return; - } + mutex_lock(&wx->reset_lock); + set_bit(WX_STATE_RESETTING, wx->state); txgbe_down(wx); txgbe_up(wx); clear_bit(WX_STATE_RESETTING, wx->state); + mutex_unlock(&wx->reset_lock); } void txgbe_do_reset(struct net_device *netdev) @@ -801,6 +804,8 @@ static int txgbe_probe(struct pci_dev *pdev, netdev->features |= NETIF_F_HIGHDMA; netdev->hw_features |= NETIF_F_GRO; netdev->features |= NETIF_F_GRO; + netdev->hw_features |= NETIF_F_LRO; + netdev->features |= NETIF_F_LRO; netdev->features |= NETIF_F_RX_UDP_TUNNEL_PORT; netdev->priv_flags |= IFF_UNICAST_FLT; diff --git a/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h b/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h index 4d77da720eba..70f89a42b0b3 100644 --- a/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h +++ b/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h @@ -302,7 +302,7 @@ struct txgbe_fdir_filter { #define TXGBE_DEFAULT_RX_WORK 128 #endif -#define TXGBE_INTR_MISC(A) BIT((A)->num_q_vectors) +#define TXGBE_INTR_MISC(A) BIT_ULL((A)->num_q_vectors) #define TXGBE_INTR_QALL(A) (TXGBE_INTR_MISC(A) - 1) #define TXGBE_MAX_EITR GENMASK(11, 3) diff --git a/drivers/net/ethernet/wangxun/txgbevf/txgbevf_main.c b/drivers/net/ethernet/wangxun/txgbevf/txgbevf_main.c index 72663e3c4205..52c1e223bbd7 100644 --- a/drivers/net/ethernet/wangxun/txgbevf/txgbevf_main.c +++ b/drivers/net/ethernet/wangxun/txgbevf/txgbevf_main.c @@ -157,6 +157,17 @@ static int txgbevf_sw_init(struct wx *wx) wx->set_num_queues = txgbevf_set_num_queues; + switch (wx->mac.type) { + case wx_mac_sp: + break; + case wx_mac_aml: + case wx_mac_aml40: + set_bit(WX_FLAG_RX_MERGE_ENABLED, wx->flags); + break; + default: + break; + } + return 0; err_reset_hw: kfree(wx->vfinfo); diff --git a/drivers/net/gtp.c b/drivers/net/gtp.c index af45118083ca..070f37cbddbf 100644 --- a/drivers/net/gtp.c +++ b/drivers/net/gtp.c @@ -1549,6 +1549,8 @@ static int gtp_newlink(struct net_device *dev, out_encap: gtp_encap_disable(gtp); out_hashtable: + /* Wait for RCU readers that may still reference this gtp_dev. */ + synchronize_net(); kfree(gtp->addr_hash); kfree(gtp->tid_hash); return err; diff --git a/drivers/net/ipa/ipa_main.c b/drivers/net/ipa/ipa_main.c index 30fe12e3582b..eadd38913457 100644 --- a/drivers/net/ipa/ipa_main.c +++ b/drivers/net/ipa/ipa_main.c @@ -966,12 +966,12 @@ static void ipa_remove(struct platform_device *pdev) } if (ret) { /* - * Not cleaning up here properly might also yield a - * crash later on. As the device is still unregistered - * in this case, this might even yield a crash later on. + * Continuing teardown after failing to stop the modem + * could crash, so leave the remaining resources allocated. */ dev_err(dev, "Failed to stop modem (%pe), leaking resources\n", ERR_PTR(ret)); + pm_runtime_put_noidle(dev); return; } diff --git a/drivers/net/netdevsim/netdev.c b/drivers/net/netdevsim/netdev.c index 7a8e17bf46bd..37260e488b9e 100644 --- a/drivers/net/netdevsim/netdev.c +++ b/drivers/net/netdevsim/netdev.c @@ -576,6 +576,7 @@ static void nsim_del_napi(struct netdevsim *ns) for (i = 0; i < dev->num_rx_queues; i++) { struct nsim_rq *rq = ns->rq[i]; + netif_queue_set_napi(dev, i, NETDEV_QUEUE_TYPE_RX, NULL); napi_disable_locked(&rq->napi); __netif_napi_del_locked(&rq->napi); } @@ -839,6 +840,7 @@ nsim_queue_start(struct net_device *dev, struct netdev_queue_config *qcfg, } ns->rq[idx] = qmem->rq; + netif_queue_set_napi(dev, idx, NETDEV_QUEUE_TYPE_RX, &ns->rq[idx]->napi); napi_enable_locked(&ns->rq[idx]->napi); return 0; diff --git a/drivers/net/ppp/pppox.c b/drivers/net/ppp/pppox.c index 08364f10a43f..61b3453469dc 100644 --- a/drivers/net/ppp/pppox.c +++ b/drivers/net/ppp/pppox.c @@ -74,7 +74,9 @@ int pppox_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) switch (cmd) { case PPPIOCGCHAN: { + struct sk_buff *skb; int index; + rc = -ENOTCONN; if (!(sk->sk_state & PPPOX_CONNECTED)) break; @@ -85,7 +87,22 @@ int pppox_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg) break; rc = 0; + /* PPPIOCGCHAN historically marks the userspace handoff to + * generic PPP; pppd then attaches the returned channel to + * /dev/ppp. + */ sk->sk_state |= PPPOX_BOUND; + /* Let lockless receive paths finish queueing against the old + * state. + */ + synchronize_net(); + /* Drain packets queued before the handoff because a bound + * socket is no longer readable. + */ + while ((skb = skb_dequeue(&sk->sk_receive_queue))) { + skb_orphan(skb); + ppp_input(&po->chan, skb); + } break; } default: diff --git a/drivers/net/slip/slip.c b/drivers/net/slip/slip.c index 59daf095cad6..d16897686d1b 100644 --- a/drivers/net/slip/slip.c +++ b/drivers/net/slip/slip.c @@ -886,8 +886,6 @@ static int slip_open(struct tty_struct *tty) * Close down a SLIP channel. * This means flushing out any pending queues, and then returning. This * call is serialized against other ldisc functions. - * - * We also use this method fo a hangup event */ static void slip_close(struct tty_struct *tty) @@ -915,11 +913,6 @@ static void slip_close(struct tty_struct *tty) unregister_netdev(sl->dev); /* sl_uninit() has dropped the slip_devs[] entry by now */ } - -static void slip_hangup(struct tty_struct *tty) -{ - slip_close(tty); -} /************************************************************************ * STANDARD SLIP ENCAPSULATION * ************************************************************************/ @@ -1280,7 +1273,6 @@ static struct tty_ldisc_ops sl_ldisc = { .name = "slip", .open = slip_open, .close = slip_close, - .hangup = slip_hangup, .ioctl = slip_ioctl, .receive_buf = slip_receive_buf, .write_wakeup = slip_write_wakeup, diff --git a/drivers/net/thunderbolt/main.c b/drivers/net/thunderbolt/main.c index 7bb9aa20fab9..096f3ff36a43 100644 --- a/drivers/net/thunderbolt/main.c +++ b/drivers/net/thunderbolt/main.c @@ -907,9 +907,11 @@ static int tbnet_poll(struct napi_struct *napi, int budget) le32_to_cpu(net->rx_hdr.frame_count) - 1; rx_packets++; - net->stats.rx_bytes += frame_size; if (last) { + /* Before eth_type_trans() pulls the Ethernet header. */ + net->stats.rx_packets++; + net->stats.rx_bytes += skb->len; skb->protocol = eth_type_trans(skb, net->dev); trace_tbnet_rx_skb(skb); napi_gro_receive(&net->napi, skb); @@ -917,8 +919,6 @@ static int tbnet_poll(struct napi_struct *napi, int budget) } } - net->stats.rx_packets += rx_packets; - if (cleaned_count) tbnet_alloc_rx_buffers(net, cleaned_count); diff --git a/drivers/net/vxlan/vxlan_mdb.c b/drivers/net/vxlan/vxlan_mdb.c index 2ddcbf102b89..70d2d180f6d2 100644 --- a/drivers/net/vxlan/vxlan_mdb.c +++ b/drivers/net/vxlan/vxlan_mdb.c @@ -1428,14 +1428,17 @@ static void vxlan_mdb_flush(struct vxlan_dev *vxlan, struct vxlan_mdb_entry *mdb_entry; struct hlist_node *tmp; - /* The removal of an entry cannot trigger the removal of another entry - * since entries are always added to the head of the list. - */ hlist_for_each_entry_safe(mdb_entry, tmp, &vxlan->mdb_list, mdb_node) { if (desc->src_vni && desc->src_vni != mdb_entry->key.vni) continue; vxlan_mdb_remotes_flush(vxlan, mdb_entry, desc); + /* The flush can remove the (S, G) entries created for the + * source list of this entry, including the one saved by + * hlist_for_each_entry_safe(), so re-read it while this entry + * is still linked. + */ + tmp = mdb_entry->mdb_node.next; /* Entry will only be removed if its remotes list is empty. */ vxlan_mdb_entry_put(vxlan, mdb_entry); } diff --git a/drivers/net/vxlan/vxlan_vnifilter.c b/drivers/net/vxlan/vxlan_vnifilter.c index 215e82876662..3f2d2396243a 100644 --- a/drivers/net/vxlan/vxlan_vnifilter.c +++ b/drivers/net/vxlan/vxlan_vnifilter.c @@ -462,10 +462,8 @@ static int vxlan_vnifilter_dump(struct sk_buff *skb, struct netlink_callback *cb static const struct nla_policy vni_filter_entry_policy[VXLAN_VNIFILTER_ENTRY_MAX + 1] = { [VXLAN_VNIFILTER_ENTRY_START] = { .type = NLA_U32 }, [VXLAN_VNIFILTER_ENTRY_END] = { .type = NLA_U32 }, - [VXLAN_VNIFILTER_ENTRY_GROUP] = { .type = NLA_BINARY, - .len = sizeof_field(struct iphdr, daddr) }, - [VXLAN_VNIFILTER_ENTRY_GROUP6] = { .type = NLA_BINARY, - .len = sizeof(struct in6_addr) }, + [VXLAN_VNIFILTER_ENTRY_GROUP] = NLA_POLICY_EXACT_LEN(sizeof_field(struct iphdr, daddr)), + [VXLAN_VNIFILTER_ENTRY_GROUP6] = NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)), }; static const struct nla_policy vni_filter_policy[VXLAN_VNIFILTER_MAX + 1] = { diff --git a/drivers/net/wireless/ath/ath10k/snoc.c b/drivers/net/wireless/ath/ath10k/snoc.c index b3f6424c17d3..0785ea2b8270 100644 --- a/drivers/net/wireless/ath/ath10k/snoc.c +++ b/drivers/net/wireless/ath/ath10k/snoc.c @@ -5,6 +5,7 @@ #include #include +#include #include #include #include @@ -1456,11 +1457,15 @@ static void ath10k_msa_dump_memory(struct ath10k *ar, hdr->length = cpu_to_le32(ar->msa.mem_size); if (current_region->len < ar->msa.mem_size) { - memcpy(buf, ar->msa.vaddr, current_region->len); + memcpy_fromio(buf, + (const void __iomem __force *)ar->msa.vaddr, + current_region->len); ath10k_warn(ar, "msa dump length is less than msa size %x, %x\n", current_region->len, ar->msa.mem_size); } else { - memcpy(buf, ar->msa.vaddr, ar->msa.mem_size); + memcpy_fromio(buf, + (const void __iomem __force *)ar->msa.vaddr, + ar->msa.mem_size); } } diff --git a/drivers/net/wireless/ath/ath11k/dp_rx.c b/drivers/net/wireless/ath/ath11k/dp_rx.c index 330446f279cd..05b11a1b6376 100644 --- a/drivers/net/wireless/ath/ath11k/dp_rx.c +++ b/drivers/net/wireless/ath/ath11k/dp_rx.c @@ -2332,10 +2332,10 @@ static void ath11k_dp_rx_h_rate(struct ath11k *ar, struct hal_rx_desc *rx_desc, case RX_MSDU_START_PKT_TYPE_11N: rx_status->encoding = RX_ENC_HT; if (rate_mcs > ATH11K_HT_MCS_MAX) { - ath11k_warn(ar->ab, - "Received with invalid mcs in HT mode %d\n", - rate_mcs); - break; + ath11k_dbg(ar->ab, ATH11K_DBG_DP_RX, + "Received HT frame with out-of-range mcs %d, capping to %d\n", + rate_mcs, ATH11K_HT_MCS_MAX); + rate_mcs = ATH11K_HT_MCS_MAX; } rx_status->rate_idx = rate_mcs + (8 * (nss - 1)); if (sgi) @@ -2344,13 +2344,13 @@ static void ath11k_dp_rx_h_rate(struct ath11k *ar, struct hal_rx_desc *rx_desc, break; case RX_MSDU_START_PKT_TYPE_11AC: rx_status->encoding = RX_ENC_VHT; - rx_status->rate_idx = rate_mcs; if (rate_mcs > ATH11K_VHT_MCS_MAX) { - ath11k_warn(ar->ab, - "Received with invalid mcs in VHT mode %d\n", - rate_mcs); - break; + ath11k_dbg(ar->ab, ATH11K_DBG_DP_RX, + "Received VHT frame with out-of-range mcs %d, capping to %d\n", + rate_mcs, ATH11K_VHT_MCS_MAX); + rate_mcs = ATH11K_VHT_MCS_MAX; } + rx_status->rate_idx = rate_mcs; rx_status->nss = nss; if (sgi) rx_status->enc_flags |= RX_ENC_FLAG_SHORT_GI; @@ -2360,14 +2360,14 @@ static void ath11k_dp_rx_h_rate(struct ath11k *ar, struct hal_rx_desc *rx_desc, rx_status->enc_flags |= RX_ENC_FLAG_LDPC; break; case RX_MSDU_START_PKT_TYPE_11AX: - rx_status->rate_idx = rate_mcs; + rx_status->encoding = RX_ENC_HE; if (rate_mcs > ATH11K_HE_MCS_MAX) { - ath11k_warn(ar->ab, - "Received with invalid mcs in HE mode %d\n", - rate_mcs); - break; + ath11k_dbg(ar->ab, ATH11K_DBG_DP_RX, + "Received HE frame with out-of-range mcs %d, capping to %d\n", + rate_mcs, ATH11K_HE_MCS_MAX); + rate_mcs = ATH11K_HE_MCS_MAX; } - rx_status->encoding = RX_ENC_HE; + rx_status->rate_idx = rate_mcs; rx_status->nss = nss; rx_status->he_gi = ath11k_mac_he_gi_to_nl80211_he_gi(sgi); rx_status->bw = ath11k_mac_bw_to_mac80211_bw(bw); diff --git a/drivers/net/wireless/ath/ath11k/wmi.c b/drivers/net/wireless/ath/ath11k/wmi.c index e1b00dc811e7..3305ec66ddbf 100644 --- a/drivers/net/wireless/ath/ath11k/wmi.c +++ b/drivers/net/wireless/ath/ath11k/wmi.c @@ -159,6 +159,8 @@ static const struct wmi_tlv_policy wmi_tlv_policies[] = { .min_len = sizeof(struct ath11k_wmi_p2p_noa_info) }, [WMI_TAG_P2P_NOA_EVENT] = { .min_len = sizeof(struct wmi_p2p_noa_event) }, + [WMI_TAG_PDEV_CSA_SWITCH_COUNT_STATUS_EVENT] = { + .min_len = sizeof(struct wmi_pdev_csa_switch_ev) }, }; #define PRIMAP(_hw_mode_) \ @@ -262,6 +264,13 @@ const void **ath11k_wmi_tlv_parse_alloc(struct ath11k_base *ab, return tb; } +static u32 ath11k_wmi_tlv_data_len(const void *data) +{ + const struct wmi_tlv *tlv = (const struct wmi_tlv *)data - 1; + + return FIELD_GET(WMI_TLV_LEN, tlv->header); +} + static int ath11k_wmi_cmd_send_nowait(struct ath11k_pdev_wmi *wmi, struct sk_buff *skb, u32 cmd_id) { @@ -2411,8 +2420,8 @@ int ath11k_wmi_send_scan_start_cmd(struct ath11k *ar, for (i = 0; i < params->num_hint_bssid; ++i) { hint_bssid->freq_flags = params->hint_bssid[i].freq_flags; - ether_addr_copy(¶ms->hint_bssid[i].bssid.addr[0], - &hint_bssid->bssid.addr[0]); + ether_addr_copy(&hint_bssid->bssid.addr[0], + ¶ms->hint_bssid[i].bssid.addr[0]); hint_bssid++; } } @@ -4747,14 +4756,16 @@ static int ath11k_wmi_tlv_mac_phy_caps_parse(struct ath11k_base *soc, if (svc_rdy_ext->n_mac_phy_caps >= svc_rdy_ext->tot_phy_id) return -ENOBUFS; - len = min_t(u16, len, sizeof(struct wmi_mac_phy_capabilities)); if (!svc_rdy_ext->n_mac_phy_caps) { - svc_rdy_ext->mac_phy_caps = kcalloc(svc_rdy_ext->tot_phy_id, - len, GFP_ATOMIC); + svc_rdy_ext->mac_phy_caps = + kzalloc_objs(*svc_rdy_ext->mac_phy_caps, + svc_rdy_ext->tot_phy_id, + GFP_ATOMIC); if (!svc_rdy_ext->mac_phy_caps) return -ENOMEM; } + len = min_t(u16, len, sizeof(struct wmi_mac_phy_capabilities)); memcpy(svc_rdy_ext->mac_phy_caps + svc_rdy_ext->n_mac_phy_caps, ptr, len); svc_rdy_ext->n_mac_phy_caps++; return 0; @@ -5066,6 +5077,7 @@ static int ath11k_service_ready_ext_event(struct ath11k_base *ab, return 0; err: + kfree(svc_rdy_ext.mac_phy_caps); ath11k_wmi_free_dbring_caps(ab); return ret; } @@ -8302,15 +8314,23 @@ ath11k_wmi_process_csa_switch_count_event(struct ath11k_base *ab, const struct wmi_pdev_csa_switch_ev *ev, const u32 *vdev_ids) { - int i; + u32 vdev_ids_len = ath11k_wmi_tlv_data_len(vdev_ids); + u32 num_vdevs = ev->num_vdevs; struct ath11k_vif *arvif; + int i; /* Finish CSA once the switch count becomes NULL */ if (ev->current_switch_count) return; + if (num_vdevs > vdev_ids_len / sizeof(*vdev_ids)) { + ath11k_warn(ab, "csa switch count num_vdevs %u exceeds tlv array length %u\n", + num_vdevs, vdev_ids_len); + return; + } + rcu_read_lock(); - for (i = 0; i < ev->num_vdevs; i++) { + for (i = 0; i < num_vdevs; i++) { arvif = ath11k_mac_get_arvif_by_vdev_id(ab, vdev_ids[i]); if (!arvif) { @@ -8757,13 +8777,15 @@ static void ath11k_wmi_tlv_op_rx(struct ath11k_base *ab, struct sk_buff *skb) struct wmi_cmd_hdr *cmd_hdr; enum wmi_tlv_event_id id; + if (skb->len < sizeof(*cmd_hdr)) + goto out; + cmd_hdr = (struct wmi_cmd_hdr *)skb->data; id = FIELD_GET(WMI_CMD_HDR_CMD_ID, (cmd_hdr->cmd_id)); trace_ath11k_wmi_event(ab, id, skb->data, skb->len); - if (skb_pull(skb, sizeof(struct wmi_cmd_hdr)) == NULL) - goto out; + skb_pull(skb, sizeof(*cmd_hdr)); switch (id) { /* Process all the WMI events here */ diff --git a/drivers/net/wireless/ath/ath12k/ahb.c b/drivers/net/wireless/ath/ath12k/ahb.c index b30527c402f6..af68d5cd6a9d 100644 --- a/drivers/net/wireless/ath/ath12k/ahb.c +++ b/drivers/net/wireless/ath/ath12k/ahb.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "ahb.h" #include "debug.h" #include "hif.h" @@ -345,24 +346,25 @@ static int ath12k_ahb_power_up(struct ath12k_base *ab) char fw2_name[ATH12K_USERPD_FW_NAME_LEN]; struct device *dev = ab->dev; const struct firmware *fw, *fw2; - struct reserved_mem *rmem = NULL; unsigned long time_left; phys_addr_t mem_phys; + struct resource res; void *mem_region; size_t mem_size; u32 pasid; int ret; - rmem = ath12k_core_get_reserved_mem(ab, 0); - if (!rmem) - return -ENODEV; + ret = of_reserved_mem_region_to_resource_byname(dev->of_node, "q6-region", + &res); + if (ret) + return ret; - mem_phys = rmem->base; - mem_size = rmem->size; + mem_phys = res.start; + mem_size = resource_size(&res); mem_region = devm_memremap(dev, mem_phys, mem_size, MEMREMAP_WC); if (IS_ERR(mem_region)) { - ath12k_err(ab, "unable to map memory region: %pa+%pa\n", - &rmem->base, &rmem->size); + ath12k_err(ab, "unable to map memory region: %pa+%zx\n", + &res.start, mem_size); return PTR_ERR(mem_region); } diff --git a/drivers/net/wireless/ath/ath12k/core.c b/drivers/net/wireless/ath/ath12k/core.c index cc352eef1939..a02db6fd2354 100644 --- a/drivers/net/wireless/ath/ath12k/core.c +++ b/drivers/net/wireless/ath/ath12k/core.c @@ -46,7 +46,7 @@ ath12k_mem_profile_based_param ath12k_mem_profile_based_param[] = { .dp_params = { .tx_comp_ring_size = 32768, .rxdma_monitor_buf_ring_size = 4096, - .rxdma_monitor_dst_ring_size = 8092, + .rxdma_monitor_dst_ring_size = 8192, .num_pool_tx_desc = 32768, .rx_desc_count = 12288, }, @@ -633,31 +633,6 @@ u32 ath12k_core_get_max_peers_per_radio(struct ath12k_base *ab) return ath12k_core_get_max_station_per_radio(ab) + TARGET_NUM_VDEVS(ab); } -struct reserved_mem *ath12k_core_get_reserved_mem(struct ath12k_base *ab, - int index) -{ - struct device *dev = ab->dev; - struct reserved_mem *rmem; - struct device_node *node; - - node = of_parse_phandle(dev->of_node, "memory-region", index); - if (!node) { - ath12k_dbg(ab, ATH12K_DBG_BOOT, - "failed to parse memory-region for index %d\n", index); - return NULL; - } - - rmem = of_reserved_mem_lookup(node); - of_node_put(node); - if (!rmem) { - ath12k_dbg(ab, ATH12K_DBG_BOOT, - "unable to get memory-region for index %d\n", index); - return NULL; - } - - return rmem; -} - static inline void ath12k_core_to_group_ref_get(struct ath12k_base *ab) { diff --git a/drivers/net/wireless/ath/ath12k/core.h b/drivers/net/wireless/ath/ath12k/core.h index d7688b383f62..93cdfcaf95bd 100644 --- a/drivers/net/wireless/ath/ath12k/core.h +++ b/drivers/net/wireless/ath/ath12k/core.h @@ -1369,8 +1369,6 @@ void ath12k_fw_stats_init(struct ath12k *ar); void ath12k_fw_stats_bcn_free(struct list_head *head); void ath12k_fw_stats_free(struct ath12k_fw_stats *stats); void ath12k_fw_stats_reset(struct ath12k *ar); -struct reserved_mem *ath12k_core_get_reserved_mem(struct ath12k_base *ab, - int index); enum ath12k_qmi_mem_mode ath12k_core_get_memory_mode(struct ath12k_base *ab); static inline const char *ath12k_scan_state_str(enum ath12k_scan_state state) diff --git a/drivers/net/wireless/ath/ath12k/qmi.c b/drivers/net/wireless/ath/ath12k/qmi.c index 8de9aee2498e..b66cff3c70e2 100644 --- a/drivers/net/wireless/ath/ath12k/qmi.c +++ b/drivers/net/wireless/ath/ath12k/qmi.c @@ -13,6 +13,7 @@ #include #include #include +#include #define SLEEP_CLOCK_SELECT_INTERNAL_BIT 0x02 #define HOST_CSTATE_BIT 0x04 @@ -2683,121 +2684,96 @@ static int ath12k_qmi_alloc_target_mem_chunk(struct ath12k_base *ab) return ret; } +static const char *ath12k_qmi_get_mem_reg_name(int mem_type) +{ + switch (mem_type) { + case HOST_DDR_REGION_TYPE: + case BDF_MEM_REGION_TYPE: + return "q6-region"; + case M3_DUMP_REGION_TYPE: + return "m3-dump"; + case CALDB_MEM_REGION_TYPE: + return "q6-caldb"; + case MLO_GLOBAL_MEM_REGION_TYPE: + return "mlo-global-mem"; + default: + return NULL; + } +} + static int ath12k_qmi_assign_target_mem_chunk(struct ath12k_base *ab) { - struct reserved_mem *rmem; - size_t avail_rmem_size; + struct device_node *np = ab->dev->of_node; + size_t avail_rmem_size, offset = 0; + struct target_mem_chunk *chunk; + struct resource res; + const char *rname; int i, idx, ret; for (i = 0, idx = 0; i < ab->qmi.mem_seg_count; i++) { - switch (ab->qmi.target_mem[i].type) { - case HOST_DDR_REGION_TYPE: - rmem = ath12k_core_get_reserved_mem(ab, 0); - if (!rmem) { - ret = -ENODEV; - goto out; - } - - avail_rmem_size = rmem->size; - if (avail_rmem_size < ab->qmi.target_mem[i].size) { - ath12k_dbg(ab, ATH12K_DBG_QMI, - "failed to assign mem type %u req size %u avail size %zu\n", - ab->qmi.target_mem[i].type, - ab->qmi.target_mem[i].size, - avail_rmem_size); - ret = -EINVAL; - goto out; - } - - ab->qmi.target_mem[idx].paddr = rmem->base; - ab->qmi.target_mem[idx].v.ioaddr = - ioremap(ab->qmi.target_mem[idx].paddr, - ab->qmi.target_mem[i].size); - if (!ab->qmi.target_mem[idx].v.ioaddr) { - ret = -EIO; - goto out; - } - ab->qmi.target_mem[idx].size = ab->qmi.target_mem[i].size; - ab->qmi.target_mem[idx].type = ab->qmi.target_mem[i].type; - idx++; - break; - case BDF_MEM_REGION_TYPE: - rmem = ath12k_core_get_reserved_mem(ab, 0); - if (!rmem) { - ret = -ENODEV; - goto out; - } - - avail_rmem_size = rmem->size - ab->hw_params->bdf_addr_offset; - if (avail_rmem_size < ab->qmi.target_mem[i].size) { - ath12k_dbg(ab, ATH12K_DBG_QMI, - "failed to assign mem type %u req size %u avail size %zu\n", - ab->qmi.target_mem[i].type, - ab->qmi.target_mem[i].size, - avail_rmem_size); - ret = -EINVAL; - goto out; - } - ab->qmi.target_mem[idx].paddr = - rmem->base + ab->hw_params->bdf_addr_offset; - ab->qmi.target_mem[idx].v.ioaddr = - ioremap(ab->qmi.target_mem[idx].paddr, - ab->qmi.target_mem[i].size); - if (!ab->qmi.target_mem[idx].v.ioaddr) { - ret = -EIO; - goto out; - } - ab->qmi.target_mem[idx].size = ab->qmi.target_mem[i].size; - ab->qmi.target_mem[idx].type = ab->qmi.target_mem[i].type; - idx++; - break; - case CALDB_MEM_REGION_TYPE: - /* Cold boot calibration is not enabled in Ath12k. Hence, + chunk = &ab->qmi.target_mem[i]; + if (chunk->type == CALDB_MEM_REGION_TYPE) { + /* + * Cold boot calibration is not enabled in Ath12k. Hence, * assign paddr = 0. * Once cold boot calibration is enabled add support to * assign reserved memory from DT. */ ab->qmi.target_mem[idx].paddr = 0; ab->qmi.target_mem[idx].v.ioaddr = NULL; - ab->qmi.target_mem[idx].size = ab->qmi.target_mem[i].size; - ab->qmi.target_mem[idx].type = ab->qmi.target_mem[i].type; + ab->qmi.target_mem[idx].size = chunk->size; + ab->qmi.target_mem[idx].type = chunk->type; idx++; - break; - case M3_DUMP_REGION_TYPE: - rmem = ath12k_core_get_reserved_mem(ab, 1); - if (!rmem) { - ret = -EINVAL; - goto out; - } + continue; + } - avail_rmem_size = rmem->size; - if (avail_rmem_size < ab->qmi.target_mem[i].size) { - ath12k_dbg(ab, ATH12K_DBG_QMI, - "failed to assign mem type %u req size %u avail size %zu\n", - ab->qmi.target_mem[i].type, - ab->qmi.target_mem[i].size, + rname = ath12k_qmi_get_mem_reg_name(chunk->type); + if (!rname) { + ath12k_warn(ab, "qmi ignore invalid mem req type %u\n", + chunk->type); + continue; + } + + ret = of_reserved_mem_region_to_resource_byname(np, rname, &res); + if (ret) + goto out; + + avail_rmem_size = resource_size(&res); + if (chunk->type == BDF_MEM_REGION_TYPE || + chunk->type == HOST_DDR_REGION_TYPE) { + if (ab->hw_params->bdf_addr_offset > avail_rmem_size || + offset > avail_rmem_size - ab->hw_params->bdf_addr_offset) { + ath12k_err(ab, "qmi mem offset overflow: bdf_offset=%u offset=%zu size=%zu\n", + ab->hw_params->bdf_addr_offset, offset, avail_rmem_size); ret = -EINVAL; goto out; } - ab->qmi.target_mem[idx].paddr = rmem->base; - ab->qmi.target_mem[idx].v.ioaddr = - ioremap(ab->qmi.target_mem[idx].paddr, - ab->qmi.target_mem[i].size); - if (!ab->qmi.target_mem[idx].v.ioaddr) { - ret = -EIO; - goto out; - } - ab->qmi.target_mem[idx].size = ab->qmi.target_mem[i].size; - ab->qmi.target_mem[idx].type = ab->qmi.target_mem[i].type; - idx++; - break; - default: - ath12k_warn(ab, "qmi ignore invalid mem req type %u\n", - ab->qmi.target_mem[i].type); - break; + avail_rmem_size -= ab->hw_params->bdf_addr_offset + offset; + res.start += ab->hw_params->bdf_addr_offset + offset; + offset += chunk->size; } + + if (avail_rmem_size < chunk->size) { + ath12k_dbg(ab, ATH12K_DBG_QMI, + "failed to assign mem type %u req size %u avail size %zu\n", + chunk->type, chunk->size, avail_rmem_size); + ret = -EINVAL; + goto out; + } + + ab->qmi.target_mem[idx].paddr = res.start; + ab->qmi.target_mem[idx].v.ioaddr = ioremap(ab->qmi.target_mem[idx].paddr, + chunk->size); + if (!ab->qmi.target_mem[idx].v.ioaddr) { + ret = -EIO; + goto out; + } + + ab->qmi.target_mem[idx].size = chunk->size; + ab->qmi.target_mem[idx].type = chunk->type; + idx++; } ab->qmi.mem_seg_count = idx; diff --git a/drivers/net/wireless/ath/ath12k/wmi.c b/drivers/net/wireless/ath/ath12k/wmi.c index f3474a13e32a..f4392462ae3a 100644 --- a/drivers/net/wireless/ath/ath12k/wmi.c +++ b/drivers/net/wireless/ath/ath12k/wmi.c @@ -2816,8 +2816,8 @@ int ath12k_wmi_send_scan_start_cmd(struct ath12k *ar, for (i = 0; i < arg->num_hint_bssid; ++i) { hint_bssid->freq_flags = arg->hint_bssid[i].freq_flags; - ether_addr_copy(&arg->hint_bssid[i].bssid.addr[0], - &hint_bssid->bssid.addr[0]); + ether_addr_copy(&hint_bssid->bssid.addr[0], + &arg->hint_bssid[i].bssid.addr[0]); hint_bssid++; } } @@ -4525,14 +4525,16 @@ static int ath12k_wmi_mac_phy_caps_parse(struct ath12k_base *soc, if (svc_rdy_ext->n_mac_phy_caps >= svc_rdy_ext->tot_phy_id) return -ENOBUFS; - len = min_t(u16, len, sizeof(struct ath12k_wmi_mac_phy_caps_params)); if (!svc_rdy_ext->n_mac_phy_caps) { - svc_rdy_ext->mac_phy_caps = kzalloc((svc_rdy_ext->tot_phy_id) * len, - GFP_ATOMIC); + svc_rdy_ext->mac_phy_caps = + kzalloc_objs(*svc_rdy_ext->mac_phy_caps, + svc_rdy_ext->tot_phy_id, + GFP_ATOMIC); if (!svc_rdy_ext->mac_phy_caps) return -ENOMEM; } + len = min_t(u16, len, sizeof(struct ath12k_wmi_mac_phy_caps_params)); memcpy(svc_rdy_ext->mac_phy_caps + svc_rdy_ext->n_mac_phy_caps, ptr, len); svc_rdy_ext->n_mac_phy_caps++; return 0; @@ -9410,6 +9412,7 @@ static void ath12k_wmi_process_tpc_stats(struct ath12k_base *ab, void *ptr = skb->data; struct ath12k *ar; u16 tlv_tag; + u16 tlv_len; u32 event_count; int ret; @@ -9425,6 +9428,7 @@ static void ath12k_wmi_process_tpc_stats(struct ath12k_base *ab, tlv = (struct wmi_tlv *)ptr; tlv_tag = le32_get_bits(tlv->header, WMI_TLV_TAG); + tlv_len = le32_get_bits(tlv->header, WMI_TLV_LEN); ptr += sizeof(*tlv); if (tlv_tag != WMI_TAG_HALPHY_CTRL_PATH_EVENT_FIXED_PARAM) { @@ -9432,6 +9436,12 @@ static void ath12k_wmi_process_tpc_stats(struct ath12k_base *ab, return; } + if (tlv_len < sizeof(*fixed_param)) { + ath12k_warn(ab, "TPC stats fixed param tlv len %u too short\n", + tlv_len); + return; + } + fixed_param = (struct ath12k_wmi_pdev_tpc_stats_event_fixed_params *)ptr; rcu_read_lock(); ar = ath12k_mac_get_ar_by_pdev_id(ab, le32_to_cpu(fixed_param->pdev_id) + 1); @@ -9739,12 +9749,12 @@ static void ath12k_wmi_op_rx(struct ath12k_base *ab, struct sk_buff *skb) struct wmi_cmd_hdr *cmd_hdr; enum wmi_tlv_event_id id; - cmd_hdr = (struct wmi_cmd_hdr *)skb->data; - id = le32_get_bits(cmd_hdr->cmd_id, WMI_CMD_HDR_CMD_ID); - - if (!skb_pull(skb, sizeof(struct wmi_cmd_hdr))) + cmd_hdr = skb_pull_data(skb, sizeof(*cmd_hdr)); + if (!cmd_hdr) goto out; + id = le32_get_bits(cmd_hdr->cmd_id, WMI_CMD_HDR_CMD_ID); + switch (id) { /* Process all the WMI events here */ case WMI_SERVICE_READY_EVENTID: diff --git a/drivers/net/wireless/ath/ath6kl/wmi.c b/drivers/net/wireless/ath/ath6kl/wmi.c index 0cdcbc3c7796..2353b6d18c6c 100644 --- a/drivers/net/wireless/ath/ath6kl/wmi.c +++ b/drivers/net/wireless/ath/ath6kl/wmi.c @@ -1296,6 +1296,9 @@ static int ath6kl_wmi_scan_complete_rx(struct wmi *wmi, u8 *datap, int len, { struct wmi_scan_complete_event *ev; + if (len < sizeof(*ev)) + return -EINVAL; + ev = (struct wmi_scan_complete_event *) datap; ath6kl_scan_complete_evt(vif, a_sle32_to_cpu(ev->status)); @@ -3372,7 +3375,12 @@ static int ath6kl_wmi_get_pmkid_list_event_rx(struct wmi *wmi, u8 *datap, static int ath6kl_wmi_addba_req_event_rx(struct wmi *wmi, u8 *datap, int len, struct ath6kl_vif *vif) { - struct wmi_addba_req_event *cmd = (struct wmi_addba_req_event *) datap; + struct wmi_addba_req_event *cmd; + + if (len < sizeof(*cmd)) + return -EINVAL; + + cmd = (struct wmi_addba_req_event *)datap; aggr_recv_addba_req_evt(vif, cmd->tid, le16_to_cpu(cmd->st_seq_no), cmd->win_sz); @@ -3383,7 +3391,12 @@ static int ath6kl_wmi_addba_req_event_rx(struct wmi *wmi, u8 *datap, int len, static int ath6kl_wmi_delba_req_event_rx(struct wmi *wmi, u8 *datap, int len, struct ath6kl_vif *vif) { - struct wmi_delba_event *cmd = (struct wmi_delba_event *) datap; + struct wmi_delba_event *cmd; + + if (len < sizeof(*cmd)) + return -EINVAL; + + cmd = (struct wmi_delba_event *)datap; aggr_recv_delba_req_evt(vif, cmd->tid); diff --git a/drivers/net/wireless/intel/iwlwifi/fw/dump.c b/drivers/net/wireless/intel/iwlwifi/fw/dump.c index ddd714cff2f4..87acbb482435 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/dump.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/dump.c @@ -436,7 +436,7 @@ static void iwl_fwrt_dump_fseq_regs(struct iwl_fw_runtime *fwrt) void iwl_fwrt_dump_error_logs(struct iwl_fw_runtime *fwrt) { struct iwl_pc_data *pc_data; - u8 count; + u32 count; if (!iwl_trans_device_enabled(fwrt->trans)) { IWL_ERR(fwrt, diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c index 5240dacf1360..fe37c8f75c71 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* - * Copyright (C) 2018-2025 Intel Corporation + * Copyright (C) 2018-2026 Intel Corporation */ #include #include "iwl-drv.h" @@ -602,6 +602,9 @@ static int iwl_dbg_tlv_alloc_fragments(struct iwl_fw_runtime *fwrt, cpu_to_le32(IWL_FW_INI_LOCATION_DRAM_PATH)) return 0; + if (!fw_mon_cfg->req_size) + return -EIO; + num_frags = le32_to_cpu(fw_mon_cfg->max_frags_num); if (fwrt->trans->mac_cfg->device_family < IWL_DEVICE_FAMILY_AX210) { if (alloc_id != IWL_FW_INI_ALLOCATION_ID_DBGC1) @@ -612,6 +615,9 @@ static int iwl_dbg_tlv_alloc_fragments(struct iwl_fw_runtime *fwrt, return -EIO; } + if (!num_frags) + return -EIO; + remain_pages = DIV_ROUND_UP(le32_to_cpu(fw_mon_cfg->req_size), PAGE_SIZE); num_frags = min_t(u32, num_frags, BUF_ALLOC_MAX_NUM_FRAGS); diff --git a/drivers/net/wireless/intel/iwlwifi/mei/main.c b/drivers/net/wireless/intel/iwlwifi/mei/main.c index dce0b7cf7b26..1486702d37e7 100644 --- a/drivers/net/wireless/intel/iwlwifi/mei/main.c +++ b/drivers/net/wireless/intel/iwlwifi/mei/main.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only /* * Copyright (C) 2021-2024 Intel Corporation + * Copyright (C) 2026 Intel Corporation */ #include @@ -457,7 +458,7 @@ static int iwl_mei_send_sap_msg_payload(struct mei_cl_device *cldev, notif_q = &dir->q_ctrl_blk[SAP_QUEUE_IDX_NOTIF]; q_head = mei->shared_mem.q_head[SAP_DIRECTION_HOST_TO_ME][SAP_QUEUE_IDX_NOTIF]; q_sz = mei->shared_mem.q_size[SAP_DIRECTION_HOST_TO_ME][SAP_QUEUE_IDX_NOTIF]; - ret = iwl_mei_write_cyclic_buf(q_head, notif_q, q_head, hdr, q_sz); + ret = iwl_mei_write_cyclic_buf(cldev, notif_q, q_head, hdr, q_sz); if (ret < 0) return ret; @@ -1147,6 +1148,11 @@ static void iwl_mei_handle_sap_rx_cmd(struct mei_cl_device *cldev, iwl_mei_read_from_q(q_head, q_sz, &rd, wr, hdr, sizeof(*hdr)); valid_rx_sz -= sizeof(*hdr); len = le16_to_cpu(hdr->len); + if (len + sizeof(*hdr) > PAGE_SIZE) { + dev_err(&cldev->dev, + "SAP message is too big: %u\n", len); + break; + } if (valid_rx_sz < len) break; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c index 8e6913c7712f..d6a13174b561 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c @@ -954,7 +954,7 @@ static void iwl_mvm_frob_txf_key_iter(struct ieee80211_hw *hw, } match++; if (match == keylen) { - memset(txf->buf + i - keylen, 0xAA, keylen); + memset(txf->buf + i + 1 - keylen, 0xAA, keylen); match = 0; } } diff --git a/drivers/net/wireless/mediatek/mt76/Kconfig b/drivers/net/wireless/mediatek/mt76/Kconfig index a86f800b8bf5..71e58689e12c 100644 --- a/drivers/net/wireless/mediatek/mt76/Kconfig +++ b/drivers/net/wireless/mediatek/mt76/Kconfig @@ -37,6 +37,10 @@ config MT792x_USB tristate select MT76_USB +config MT76_NPU + tristate + depends on NET_AIROHA_NPU=y || MT76_CORE=NET_AIROHA_NPU + source "drivers/net/wireless/mediatek/mt76/mt76x0/Kconfig" source "drivers/net/wireless/mediatek/mt76/mt76x2/Kconfig" source "drivers/net/wireless/mediatek/mt76/mt7603/Kconfig" diff --git a/drivers/net/wireless/mediatek/mt76/Makefile b/drivers/net/wireless/mediatek/mt76/Makefile index 87512d101a91..df2ac99c8872 100644 --- a/drivers/net/wireless/mediatek/mt76/Makefile +++ b/drivers/net/wireless/mediatek/mt76/Makefile @@ -12,6 +12,11 @@ mt76-y := \ mmio.o util.o trace.o dma.o mac80211.o debugfs.o eeprom.o \ tx.o agg-rx.o mcu.o wed.o scan.o channel.o +ifdef CONFIG_MT76_NPU +# CONFIG_MT76_NPU is tristate to simplify dependency tracking, +# but it behaves as a bool symbol here. +mt76-y += npu.o +endif mt76-$(CONFIG_PCI) += pci.o mt76-$(CONFIG_NL80211_TESTMODE) += testmode.o diff --git a/drivers/net/wireless/mediatek/mt76/dma.c b/drivers/net/wireless/mediatek/mt76/dma.c index 9ef073c27f30..a487887ef544 100644 --- a/drivers/net/wireless/mediatek/mt76/dma.c +++ b/drivers/net/wireless/mediatek/mt76/dma.c @@ -6,37 +6,7 @@ #include #include "mt76.h" #include "dma.h" - -#if IS_ENABLED(CONFIG_NET_MEDIATEK_SOC_WED) - -#define Q_READ(_q, _field) ({ \ - u32 _offset = offsetof(struct mt76_queue_regs, _field); \ - u32 _val; \ - if ((_q)->flags & MT_QFLAG_WED) \ - _val = mtk_wed_device_reg_read((_q)->wed, \ - ((_q)->wed_regs + \ - _offset)); \ - else \ - _val = readl(&(_q)->regs->_field); \ - _val; \ -}) - -#define Q_WRITE(_q, _field, _val) do { \ - u32 _offset = offsetof(struct mt76_queue_regs, _field); \ - if ((_q)->flags & MT_QFLAG_WED) \ - mtk_wed_device_reg_write((_q)->wed, \ - ((_q)->wed_regs + _offset), \ - _val); \ - else \ - writel(_val, &(_q)->regs->_field); \ -} while (0) - -#else - -#define Q_READ(_q, _field) readl(&(_q)->regs->_field) -#define Q_WRITE(_q, _field, _val) writel(_val, &(_q)->regs->_field) - -#endif +#include "mt76_connac.h" static struct mt76_txwi_cache * mt76_alloc_txwi(struct mt76_dev *dev) @@ -216,15 +186,35 @@ mt76_dma_queue_magic_cnt_init(struct mt76_dev *dev, struct mt76_queue *q) } } +/* A hung bus (e.g. after a PCIe AER error) reads 0xffffffff from every + * register, so clamp an out-of-range index to the fallback to keep it from + * corrupting q->head/q->tail. + */ +static int +mt76_dma_read_dma_idx(struct mt76_queue *q, int fallback) +{ + u32 idx = Q_READ(q, dma_idx); + + return idx < q->ndesc ? idx : fallback; +} + static void mt76_dma_sync_idx(struct mt76_dev *dev, struct mt76_queue *q) { - Q_WRITE(q, desc_base, q->desc_dma); - if (q->flags & MT_QFLAG_WED_RRO_EN) + if ((q->flags & MT_QFLAG_WED_RRO_EN) && + (!is_mt7992(dev) || !mt76_npu_device_active(dev))) Q_WRITE(q, ring_size, MT_DMA_RRO_EN | q->ndesc); else Q_WRITE(q, ring_size, q->ndesc); - q->head = Q_READ(q, dma_idx); + + if (mt76_queue_is_npu_tx(q)) { + writel(q->ndesc, &q->regs->ring_size); + writel(q->desc_dma, &q->regs->desc_base); + } + + Q_WRITE(q, desc_base, q->desc_dma); + + q->head = mt76_dma_read_dma_idx(q, 0); q->tail = q->head; } @@ -235,7 +225,7 @@ void mt76_dma_queue_reset(struct mt76_dev *dev, struct mt76_queue *q, return; if (!mt76_queue_is_wed_rro_ind(q) && - !mt76_queue_is_wed_rro_rxdmad_c(q)) { + !mt76_queue_is_wed_rro_rxdmad_c(q) && !mt76_queue_is_npu(q)) { int i; /* clear descriptors */ @@ -442,10 +432,11 @@ mt76_dma_tx_cleanup(struct mt76_dev *dev, struct mt76_queue *q, bool flush) if (flush) last = -1; else - last = Q_READ(q, dma_idx); + last = mt76_dma_read_dma_idx(q, -1); while (q->queued > 0 && q->tail != last) { mt76_dma_tx_cleanup_idx(dev, q, q->tail, &entry); + mt76_npu_txdesc_cleanup(q, q->tail); mt76_queue_tx_complete(dev, q, &entry); if (entry.txwi) { @@ -454,7 +445,7 @@ mt76_dma_tx_cleanup(struct mt76_dev *dev, struct mt76_queue *q, bool flush) } if (!flush && q->tail == last) - last = Q_READ(q, dma_idx); + last = mt76_dma_read_dma_idx(q, -1); } spin_unlock_bh(&q->cleanup_lock); @@ -556,8 +547,13 @@ mt76_dma_get_buf(struct mt76_dev *dev, struct mt76_queue *q, int idx, t->ptr = NULL; mt76_put_rxwi(dev, t); - if (drop) +#ifdef CONFIG_NET_MEDIATEK_SOC_WED + /* the WO MCU owns the RX path only on WED v2, on newer + * versions this buf1 bit carries no drop information + */ + if (drop && dev->mmio.wed.version == 2) *drop |= !!(buf1 & MT_DMA_CTL_WO_DROP); +#endif } else { dma_sync_single_for_cpu(dev->dma_dev, e->dma_addr[0], SKB_WITH_OVERHEAD(q->buf_size), @@ -617,6 +613,15 @@ mt76_dma_dequeue(struct mt76_dev *dev, struct mt76_queue *q, bool flush, q->desc[idx].ctrl |= cpu_to_le32(MT_DMA_CTL_DMA_DONE); else if (!(q->desc[idx].ctrl & cpu_to_le32(MT_DMA_CTL_DMA_DONE))) return NULL; +#ifdef CONFIG_NET_MEDIATEK_SOC_WED + /* on WED v3 the M_DONE bit signals that WED is done reading + * the txfree descriptor; WED v2 does not set it + */ + else if (dev->mmio.wed.version > 2 && + mt76_queue_is_wed_tx_free(q) && + !(q->desc[idx].ctrl & cpu_to_le32(MT_DMA_CTL_M_DONE))) + return NULL; +#endif } done: q->tail = (q->tail + 1) % q->ndesc; @@ -647,8 +652,8 @@ mt76_dma_tx_queue_skb_raw(struct mt76_dev *dev, struct mt76_queue *q, buf.len = skb->len; spin_lock_bh(&q->lock); - mt76_dma_add_buf(dev, q, &buf, 1, tx_info, skb, NULL); - mt76_dma_kick_queue(dev, q); + if (mt76_dma_add_buf(dev, q, &buf, 1, tx_info, skb, NULL) >= 0) + mt76_dma_kick_queue(dev, q); spin_unlock_bh(&q->lock); return 0; @@ -680,6 +685,10 @@ mt76_dma_tx_queue_skb(struct mt76_phy *phy, struct mt76_queue *q, if (test_bit(MT76_RESET, &phy->state)) goto free_skb; + /* TODO: Take into account unlinear skbs */ + if (mt76_npu_device_active(dev) && skb_linearize(skb)) + goto free_skb; + t = mt76_get_txwi(dev); if (!t) goto free_skb; @@ -727,6 +736,9 @@ mt76_dma_tx_queue_skb(struct mt76_phy *phy, struct mt76_queue *q, if (ret < 0) goto unmap; + if (mt76_npu_device_active(dev)) + return mt76_npu_dma_add_buf(phy, q, skb, &tx_info.buf[1], txwi); + return mt76_dma_add_buf(dev, q, tx_info.buf, tx_info.nbuf, tx_info.info, tx_info.skb, t); @@ -826,8 +838,15 @@ mt76_dma_alloc_queue(struct mt76_dev *dev, struct mt76_queue *q, q->buf_size = bufsize; q->hw_idx = idx; - size = mt76_queue_is_wed_rro_ind(q) ? sizeof(struct mt76_wed_rro_desc) - : sizeof(struct mt76_desc); + if (mt76_queue_is_wed_rro_ind(q)) + size = sizeof(struct mt76_wed_rro_desc); + else if (mt76_queue_is_npu_tx(q)) + size = sizeof(struct airoha_npu_tx_dma_desc); + else if (mt76_queue_is_npu_rx(q)) + size = sizeof(struct airoha_npu_rx_dma_desc); + else + size = sizeof(struct mt76_desc); + q->desc = dmam_alloc_coherent(dev->dma_dev, q->ndesc * size, &q->desc_dma, GFP_KERNEL); if (!q->desc) @@ -843,6 +862,7 @@ mt76_dma_alloc_queue(struct mt76_dev *dev, struct mt76_queue *q, if (ret) return ret; + mt76_npu_queue_setup(dev, q); ret = mt76_wed_dma_setup(dev, q, false); if (ret) return ret; @@ -870,6 +890,11 @@ mt76_dma_rx_cleanup(struct mt76_dev *dev, struct mt76_queue *q) if (!q->ndesc) return; + if (mt76_queue_is_npu(q)) { + mt76_npu_queue_cleanup(dev, q); + return; + } + do { spin_lock_bh(&q->lock); buf = mt76_dma_dequeue(dev, q, true, NULL, NULL, &more, NULL); @@ -905,7 +930,7 @@ mt76_dma_rx_reset(struct mt76_dev *dev, enum mt76_rxq_id qid) return; if (!mt76_queue_is_wed_rro_ind(q) && - !mt76_queue_is_wed_rro_rxdmad_c(q)) { + !mt76_queue_is_wed_rro_rxdmad_c(q) && !mt76_queue_is_npu(q)) { int i; for (i = 0; i < q->ndesc; i++) @@ -925,7 +950,10 @@ mt76_dma_rx_reset(struct mt76_dev *dev, enum mt76_rxq_id qid) return; mt76_dma_sync_idx(dev, q); - mt76_dma_rx_fill_buf(dev, q, false); + if (mt76_queue_is_npu(q)) + mt76_npu_fill_rx_queue(dev, q); + else + mt76_dma_rx_fill(dev, q, false); } static void @@ -962,23 +990,24 @@ mt76_dma_rx_process(struct mt76_dev *dev, struct mt76_queue *q, int budget) struct sk_buff *skb; unsigned char *data; bool check_ddone = false; - bool allow_direct = !mt76_queue_is_wed_rx(q); + bool allow_direct = !mt76_queue_is_wed_rx(q) && + !mt76_queue_is_wed_rro_rxdmad_c(q); bool more; if ((q->flags & MT_QFLAG_WED_RRO_EN) || (IS_ENABLED(CONFIG_NET_MEDIATEK_SOC_WED) && mt76_queue_is_wed_tx_free(q))) { - dma_idx = Q_READ(q, dma_idx); + dma_idx = mt76_dma_read_dma_idx(q, q->tail); check_ddone = true; } while (done < budget) { bool drop = false; - u32 info; + u32 info = 0; if (check_ddone) { if (q->tail == dma_idx) - dma_idx = Q_READ(q, dma_idx); + dma_idx = mt76_dma_read_dma_idx(q, q->tail); if (q->tail == dma_idx) break; diff --git a/drivers/net/wireless/mediatek/mt76/dma.h b/drivers/net/wireless/mediatek/mt76/dma.h index 17a80e1757fc..1c556c8c8634 100644 --- a/drivers/net/wireless/mediatek/mt76/dma.h +++ b/drivers/net/wireless/mediatek/mt76/dma.h @@ -11,7 +11,7 @@ #define MT_DMA_CTL_SD_LEN1 GENMASK(13, 0) #define MT_DMA_CTL_LAST_SEC1 BIT(14) -#define MT_DMA_CTL_BURST BIT(15) +#define MT_DMA_CTL_M_DONE BIT(15) #define MT_DMA_CTL_SD_LEN0 GENMASK(29, 16) #define MT_DMA_CTL_LAST_SEC0 BIT(30) #define MT_DMA_CTL_DMA_DONE BIT(31) @@ -46,6 +46,73 @@ #define MT_FCE_INFO_LEN 4 #define MT_RX_RXWI_LEN 32 +#if IS_ENABLED(CONFIG_NET_MEDIATEK_SOC_WED) + +#define Q_READ(_q, _field) ({ \ + u32 _offset = offsetof(struct mt76_queue_regs, _field); \ + u32 _val; \ + if ((_q)->flags & MT_QFLAG_WED) \ + _val = mtk_wed_device_reg_read((_q)->wed, \ + ((_q)->wed_regs + \ + _offset)); \ + else \ + _val = readl(&(_q)->regs->_field); \ + _val; \ +}) + +#define Q_WRITE(_q, _field, _val) do { \ + u32 _offset = offsetof(struct mt76_queue_regs, _field); \ + if ((_q)->flags & MT_QFLAG_WED) \ + mtk_wed_device_reg_write((_q)->wed, \ + ((_q)->wed_regs + _offset), \ + _val); \ + else \ + writel(_val, &(_q)->regs->_field); \ +} while (0) + +#elif IS_ENABLED(CONFIG_MT76_NPU) + +#define Q_READ(_q, _field) ({ \ + u32 _offset = offsetof(struct mt76_queue_regs, _field); \ + u32 _val = 0; \ + if ((_q)->flags & MT_QFLAG_NPU) { \ + struct airoha_npu *npu; \ + \ + rcu_read_lock(); \ + npu = rcu_dereference(q->dev->mmio.npu); \ + if (npu) \ + regmap_read(npu->regmap, \ + ((_q)->wed_regs + _offset), &_val); \ + rcu_read_unlock(); \ + } else { \ + _val = readl(&(_q)->regs->_field); \ + } \ + _val; \ +}) + +#define Q_WRITE(_q, _field, _val) do { \ + u32 _offset = offsetof(struct mt76_queue_regs, _field); \ + if ((_q)->flags & MT_QFLAG_NPU) { \ + struct airoha_npu *npu; \ + \ + rcu_read_lock(); \ + npu = rcu_dereference(q->dev->mmio.npu); \ + if (npu) \ + regmap_write(npu->regmap, \ + ((_q)->wed_regs + _offset), _val); \ + rcu_read_unlock(); \ + } else { \ + writel(_val, &(_q)->regs->_field); \ + } \ +} while (0) + +#else + +#define Q_READ(_q, _field) readl(&(_q)->regs->_field) +#define Q_WRITE(_q, _field, _val) writel(_val, &(_q)->regs->_field) + +#endif + struct mt76_desc { __le32 buf0; __le32 ctrl; diff --git a/drivers/net/wireless/mediatek/mt76/mac80211.c b/drivers/net/wireless/mediatek/mt76/mac80211.c index 7348ac5cf7e6..2d8cd627ef4c 100644 --- a/drivers/net/wireless/mediatek/mt76/mac80211.c +++ b/drivers/net/wireless/mediatek/mt76/mac80211.c @@ -632,6 +632,8 @@ int mt76_create_page_pool(struct mt76_dev *dev, struct mt76_queue *q) case MT_RXQ_MAIN: case MT_RXQ_BAND1: case MT_RXQ_BAND2: + case MT_RXQ_NPU0: + case MT_RXQ_NPU1: pp_params.pool_size = 256; break; default: @@ -817,6 +819,7 @@ void mt76_free_device(struct mt76_dev *dev) destroy_workqueue(dev->wq); dev->wq = NULL; } + mt76_npu_deinit(dev); ieee80211_free_hw(dev->hw); } EXPORT_SYMBOL_GPL(mt76_free_device); @@ -883,6 +886,7 @@ static void mt76_rx_release_amsdu(struct mt76_phy *phy, enum mt76_rxq_id q) struct sk_buff *skb = phy->rx_amsdu[q].head; struct mt76_rx_status *status = (struct mt76_rx_status *)skb->cb; struct mt76_dev *dev = phy->dev; + struct mt76_queue *rxq = &dev->q_rx[q]; phy->rx_amsdu[q].head = NULL; phy->rx_amsdu[q].tail = NULL; @@ -911,6 +915,13 @@ static void mt76_rx_release_amsdu(struct mt76_phy *phy, enum mt76_rxq_id q) return; } } + + /* RRO 3.0 data queue skbs are processed and completed in the context + * of the indicator queue NAPI, which only polls its own skb list + */ + if (mt76_queue_is_wed_rro_data(rxq) && dev->hwrro_mode == MT76_HWRRO_V3) + q = MT_RXQ_RRO_IND; + __skb_queue_tail(&dev->rx_skb[q], skb); } @@ -1554,7 +1565,8 @@ void mt76_rx_poll_complete(struct mt76_dev *dev, enum mt76_rxq_id q, while ((skb = __skb_dequeue(&dev->rx_skb[q])) != NULL) { mt76_check_sta(dev, skb); - if (mtk_wed_device_active(&dev->mmio.wed)) + if (mtk_wed_device_active(&dev->mmio.wed) || + mt76_npu_device_active(dev)) __skb_queue_tail(&frames, skb); else mt76_rx_aggr_reorder(skb, &frames); diff --git a/drivers/net/wireless/mediatek/mt76/mcu.c b/drivers/net/wireless/mediatek/mt76/mcu.c index 65d4c2adb538..951781dcf0ad 100644 --- a/drivers/net/wireless/mediatek/mt76/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mcu.c @@ -78,15 +78,19 @@ int mt76_mcu_skb_send_and_get_msg(struct mt76_dev *dev, struct sk_buff *skb, unsigned long expires; int ret, seq; - if (mt76_is_sdio(dev)) - if (test_bit(MT76_RESET, &dev->phy.state) && atomic_read(&dev->bus_hung)) - return -EIO; - if (ret_skb) *ret_skb = NULL; mutex_lock(&dev->mcu.mutex); + if ((mt76_is_mmio(dev) && atomic_read(&dev->bus_hung)) || + (mt76_is_sdio(dev) && test_bit(MT76_RESET, &dev->phy.state) && + atomic_read(&dev->bus_hung))) { + orig_skb = skb; + ret = -EIO; + goto out; + } + if (dev->mcu_ops->mcu_skb_prepare_msg) { orig_skb = skb; ret = dev->mcu_ops->mcu_skb_prepare_msg(dev, skb, cmd, &seq); diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h index 125ac1eb2d54..2a186b47659b 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76.h +++ b/drivers/net/wireless/mediatek/mt76/mt76.h @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,7 @@ #define MT_QFLAG_WED_RRO BIT(6) #define MT_QFLAG_WED_RRO_EN BIT(7) #define MT_QFLAG_EMI_EN BIT(8) +#define MT_QFLAG_NPU BIT(9) #define __MT_WED_Q(_type, _n) (MT_QFLAG_WED | \ FIELD_PREP(MT_QFLAG_WED_TYPE, _type) | \ @@ -48,6 +50,12 @@ #define MT_WED_RRO_Q_IND __MT_WED_RRO_Q(MT76_WED_RRO_Q_IND, 0) #define MT_WED_RRO_Q_RXDMAD_C __MT_WED_RRO_Q(MT76_WED_RRO_Q_RXDMAD_C, 0) +#define __MT_NPU_Q(_type, _n) (MT_QFLAG_NPU | \ + FIELD_PREP(MT_QFLAG_WED_TYPE, _type) | \ + FIELD_PREP(MT_QFLAG_WED_RING, _n)) +#define MT_NPU_Q_TX(_n) __MT_NPU_Q(MT76_WED_Q_TX, _n) +#define MT_NPU_Q_RX(_n) __MT_NPU_Q(MT76_WED_Q_RX, _n) + struct mt76_dev; struct mt76_phy; struct mt76_wcid; @@ -139,6 +147,8 @@ enum mt76_rxq_id { MT_RXQ_TXFREE_BAND2, MT_RXQ_RRO_IND, MT_RXQ_RRO_RXDMAD_C, + MT_RXQ_NPU0, + MT_RXQ_NPU1, __MT_RXQ_MAX }; @@ -706,6 +716,11 @@ struct mt76_mmio { struct mtk_wed_device wed_hif2; struct completion wed_reset; struct completion wed_reset_complete; + + struct airoha_ppe_dev __rcu *ppe_dev; + struct airoha_npu __rcu *npu; + phys_addr_t phy_addr; + int npu_type; }; struct mt76_rx_status { @@ -1611,6 +1626,109 @@ int mt76_testmode_dump(struct ieee80211_hw *hw, struct sk_buff *skb, int mt76_testmode_set_state(struct mt76_phy *phy, enum mt76_testmode_state state); int mt76_testmode_alloc_skb(struct mt76_phy *phy, u32 len); +#if IS_ENABLED(CONFIG_MT76_NPU) +void mt76_npu_check_ppe(struct mt76_dev *dev, struct sk_buff *skb, + u32 info); +int mt76_npu_dma_add_buf(struct mt76_phy *phy, struct mt76_queue *q, + struct sk_buff *skb, struct mt76_queue_buf *buf, + void *txwi_ptr); +int mt76_npu_rx_queue_init(struct mt76_dev *dev, struct mt76_queue *q); +int mt76_npu_fill_rx_queue(struct mt76_dev *dev, struct mt76_queue *q); +void mt76_npu_queue_cleanup(struct mt76_dev *dev, struct mt76_queue *q); +void mt76_npu_disable_irqs(struct mt76_dev *dev); +int mt76_npu_init(struct mt76_dev *dev, phys_addr_t phy_addr, int type); +void mt76_npu_deinit(struct mt76_dev *dev); +void mt76_npu_queue_setup(struct mt76_dev *dev, struct mt76_queue *q); +void mt76_npu_txdesc_cleanup(struct mt76_queue *q, int index); +int mt76_npu_net_setup_tc(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + struct net_device *dev, enum tc_setup_type type, + void *type_data); +#else +static inline void mt76_npu_check_ppe(struct mt76_dev *dev, + struct sk_buff *skb, u32 info) +{ +} + +static inline int mt76_npu_dma_add_buf(struct mt76_phy *phy, + struct mt76_queue *q, + struct sk_buff *skb, + struct mt76_queue_buf *buf, + void *txwi_ptr) +{ + return -EOPNOTSUPP; +} + +static inline int mt76_npu_fill_rx_queue(struct mt76_dev *dev, + struct mt76_queue *q) +{ + return 0; +} + +static inline void mt76_npu_queue_cleanup(struct mt76_dev *dev, + struct mt76_queue *q) +{ +} + +static inline void mt76_npu_disable_irqs(struct mt76_dev *dev) +{ +} + +static inline int mt76_npu_init(struct mt76_dev *dev, phys_addr_t phy_addr, + int type) +{ + return 0; +} + +static inline void mt76_npu_deinit(struct mt76_dev *dev) +{ +} + +static inline void mt76_npu_queue_setup(struct mt76_dev *dev, + struct mt76_queue *q) +{ +} + +static inline void mt76_npu_txdesc_cleanup(struct mt76_queue *q, + int index) +{ +} + +static inline int mt76_npu_net_setup_tc(struct ieee80211_hw *hw, + struct ieee80211_vif *vif, + struct net_device *dev, + enum tc_setup_type type, + void *type_data) +{ + return -EOPNOTSUPP; +} +#endif /* CONFIG_MT76_NPU */ + +static inline bool mt76_npu_device_active(struct mt76_dev *dev) +{ + return mt76_is_mmio(dev) && !!rcu_access_pointer(dev->mmio.npu); +} + +static inline bool mt76_ppe_device_active(struct mt76_dev *dev) +{ + return mt76_is_mmio(dev) && !!rcu_access_pointer(dev->mmio.ppe_dev); +} + +static inline int mt76_npu_send_msg(struct airoha_npu *npu, int ifindex, + enum airoha_npu_wlan_set_cmd cmd, + u32 val, gfp_t gfp) +{ + return airoha_npu_wlan_send_msg(npu, ifindex, cmd, &val, sizeof(val), + gfp); +} + +static inline int mt76_npu_get_msg(struct airoha_npu *npu, int ifindex, + enum airoha_npu_wlan_get_cmd cmd, + u32 *val, gfp_t gfp) +{ + return airoha_npu_wlan_get_msg(npu, ifindex, cmd, val, sizeof(*val), + gfp); +} + static inline void mt76_testmode_reset(struct mt76_phy *phy, bool disable) { #ifdef CONFIG_NL80211_TESTMODE @@ -1852,6 +1970,23 @@ static inline bool mt76_queue_is_emi(struct mt76_queue *q) return q->flags & MT_QFLAG_EMI_EN; } +static inline bool mt76_queue_is_npu(struct mt76_queue *q) +{ + return q->flags & MT_QFLAG_NPU; +} + +static inline bool mt76_queue_is_npu_tx(struct mt76_queue *q) +{ + return mt76_queue_is_npu(q) && + FIELD_GET(MT_QFLAG_WED_TYPE, q->flags) == MT76_WED_Q_TX; +} + +static inline bool mt76_queue_is_npu_rx(struct mt76_queue *q) +{ + return mt76_queue_is_npu(q) && + FIELD_GET(MT_QFLAG_WED_TYPE, q->flags) == MT76_WED_Q_RX; +} + struct mt76_txwi_cache * mt76_token_release(struct mt76_dev *dev, int token, bool *wake); int mt76_token_consume(struct mt76_dev *dev, struct mt76_txwi_cache **ptxwi); @@ -1936,6 +2071,9 @@ mt76_vif_link(struct mt76_dev *dev, struct ieee80211_vif *vif, int link_id) if (!link_id) return mlink; + if (link_id >= IEEE80211_MLD_MAX_NUM_LINKS) + return NULL; + return mt76_dereference(mvif->link[link_id], dev); } diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac.h b/drivers/net/wireless/mediatek/mt76/mt76_connac.h index 192dcc374a64..ba7eaf3ebe98 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76_connac.h +++ b/drivers/net/wireless/mediatek/mt76/mt76_connac.h @@ -48,6 +48,11 @@ enum rx_pkt_type { #define MT_TXD_LEN_MSDU_LAST BIT(14) #define MT_TXD_LEN_AMSDU_LAST BIT(15) +/* PCIE part */ +#define PCIE_AER_UNC_STATUS_OFFSET 0x204 +#define PCIE_AER_UNC_MASK_OFFSET 0x208 +#define PCIE_AER_CO_STATUS_OFFSET 0x210 + enum { CMD_CBW_20MHZ = IEEE80211_STA_RX_BW_20, CMD_CBW_40MHZ = IEEE80211_STA_RX_BW_40, @@ -177,6 +182,11 @@ static inline bool is_mt7925(struct mt76_dev *dev) return mt76_chip(dev) == 0x7925; } +static inline bool is_320mhz_supported(struct mt76_dev *dev) +{ + return mt76_chip(dev) == 0x7927; +} + static inline bool is_mt7920(struct mt76_dev *dev) { return mt76_chip(dev) == 0x7920; @@ -224,7 +234,8 @@ static inline bool is_mt798x(struct mt76_dev *dev) static inline bool is_mt7996(struct mt76_dev *dev) { - return mt76_chip(dev) == 0x7990; + u16 chip = mt76_chip(dev); + return chip == 0x7990 || chip == 0x7991; } static inline bool is_mt7992(struct mt76_dev *dev) diff --git a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c index 83488b2d6efb..06f17da2c010 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt76x02_mac.c @@ -848,7 +848,7 @@ int mt76x02_mac_process_rx(struct mt76x02_dev *dev, struct sk_buff *skb, } } - if (WARN_ON_ONCE(len > skb->len)) + if (len > skb->len) return -EINVAL; if (pskb_trim(skb, len)) diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c index 38dfd5de365c..14c0770229bc 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c @@ -257,7 +257,7 @@ void mt7915_eeprom_parse_hw_cap(struct mt7915_dev *dev, nss = min_t(u8, min_t(u8, nss_max, nss), path); mphy->chainmask = BIT(path) - 1; - if (band) + if (band && dev->dbdc_support) mphy->chainmask <<= dev->chainshift; mphy->antenna_mask = BIT(nss) - 1; dev->chainmask |= mphy->chainmask; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/init.c b/drivers/net/wireless/mediatek/mt76/mt7915/init.c index d2b163a5fce5..f24784652013 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/init.c @@ -1265,14 +1265,19 @@ int mt7915_register_device(struct mt7915_dev *dev) ret = mt7915_init_debugfs(&dev->phy); if (ret) - goto unreg_thermal; + goto unreg_ext_phy; ret = mt7915_coredump_register(dev); if (ret) - goto unreg_thermal; + goto unreg_ext_phy; return 0; +unreg_ext_phy: + if (phy2) { + mt7915_unregister_ext_phy(dev); + phy2 = NULL; + } unreg_thermal: mt7915_unregister_thermal(&dev->phy); unreg_dev: diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c index 50af0d9f240b..dda4e7d17953 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c @@ -437,7 +437,7 @@ mt7915_mac_fill_rx(struct mt7915_dev *dev, struct sk_buff *skb, if (v0 & MT_PRXV_HT_AD_CODE) status->enc_flags |= RX_ENC_FLAG_LDPC; - status->chains = mphy->antenna_mask; + status->chains = mt7915_band_chainmask(phy); status->chain_signal[0] = to_rssi(MT_PRXV_RCPI0, v1); status->chain_signal[1] = to_rssi(MT_PRXV_RCPI1, v1); status->chain_signal[2] = to_rssi(MT_PRXV_RCPI2, v1); @@ -2345,8 +2345,10 @@ void mt7915_mac_add_twt_setup(struct ieee80211_hw *hw, } flow->tsf = le64_to_cpu(twt_agrt->twt); - if (mt7915_mcu_twt_agrt_update(dev, msta->vif, flow, MCU_TWT_AGRT_ADD)) + if (mt7915_mcu_twt_agrt_update(dev, msta->vif, flow, MCU_TWT_AGRT_ADD)) { + list_del(&flow->list); goto unlock; + } setup_cmd = TWT_SETUP_CMD_ACCEPT; dev->twt.table_mask |= BIT(table_id); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/main.c b/drivers/net/wireless/mediatek/mt76/mt7915/main.c index 6f594677474b..9d546eb8022a 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/main.c @@ -249,7 +249,7 @@ static int mt7915_add_interface(struct ieee80211_hw *hw, idx = mt76_wcid_alloc(dev->mt76.wcid_mask, mt7915_wtbl_size(dev)); if (idx < 0) { ret = -ENOSPC; - goto out; + goto err; } INIT_LIST_HEAD(&mvif->sta.rc_list); @@ -277,7 +277,17 @@ static int mt7915_add_interface(struct ieee80211_hw *hw, mt7915_mcu_add_sta(dev, vif, NULL, CONN_STATE_PORT_SECURE, true); rcu_assign_pointer(dev->mt76.wcid[idx], &mvif->sta.wcid); + mutex_unlock(&dev->mt76.mutex); + + return 0; + +err: + dev->mt76.vif_mask &= ~BIT_ULL(mvif->mt76.idx); + phy->omac_mask &= ~BIT_ULL(mvif->mt76.omac_idx); + mt7915_mcu_add_dev_info(phy, vif, false); out: + if (phy->monitor_vif == vif) + phy->monitor_vif = NULL; mutex_unlock(&dev->mt76.mutex); return ret; @@ -294,7 +304,6 @@ static void mt7915_remove_interface(struct ieee80211_hw *hw, mt7915_mcu_add_bss_info(phy, vif, false); mt7915_mcu_add_sta(dev, vif, NULL, CONN_STATE_DISCONNECT, false); - mt76_wcid_mask_clear(dev->mt76.wcid_mask, mvif->sta.wcid.idx); mutex_lock(&dev->mt76.mutex); mt76_testmode_reset(phy->mt76, true); @@ -310,6 +319,7 @@ static void mt7915_remove_interface(struct ieee80211_hw *hw, mutex_lock(&dev->mt76.mutex); dev->mt76.vif_mask &= ~BIT_ULL(mvif->mt76.idx); phy->omac_mask &= ~BIT_ULL(mvif->mt76.omac_idx); + mt76_wcid_mask_clear(dev->mt76.wcid_mask, mvif->sta.wcid.idx); mutex_unlock(&dev->mt76.mutex); spin_lock_bh(&dev->mt76.sta_poll_lock); @@ -493,7 +503,7 @@ static int mt7915_config(struct ieee80211_hw *hw, int radio_idx, mt76_rmw_field(dev, MT_DMA_DCR0(band), MT_DMA_DCR0_RXD_G5_EN, enabled); - mt76_rmw_field(dev, MT_DMA_DCR0(band), MT_MDP_DCR0_RX_HDR_TRANS_EN, + mt76_rmw_field(dev, MT_MDP_DCR0, MT_MDP_DCR0_RX_HDR_TRANS_EN, !dev->monitor_mask); mt76_testmode_reset(phy->mt76, true); mt76_wr(dev, MT_WF_RFCR(band), rxfilter); @@ -1123,7 +1133,7 @@ mt7915_set_antenna(struct ieee80211_hw *hw, int radio_idx, u32 tx_ant, u32 rx_an struct mt7915_dev *dev = mt7915_hw_dev(hw); struct mt7915_phy *phy = mt7915_hw_phy(hw); int max_nss = hweight8(hw->wiphy->available_antennas_tx); - u8 chainshift = dev->chainshift; + u8 shift = mt7915_band_chainshift(phy); u8 band = phy->mt76->band_idx; if (!tx_ant || tx_ant != rx_ant || ffs(tx_ant) > max_nss) @@ -1136,9 +1146,9 @@ mt7915_set_antenna(struct ieee80211_hw *hw, int radio_idx, u32 tx_ant, u32 rx_an /* handle a variant of mt7916/mt7981 which has 3T3R but nss2 on 5 GHz band */ if ((is_mt7916(&dev->mt76) || is_mt7981(&dev->mt76)) && band && hweight8(tx_ant) == max_nss) - phy->mt76->chainmask = (dev->chainmask >> chainshift) << chainshift; + phy->mt76->chainmask = (dev->chainmask >> shift) << shift; else - phy->mt76->chainmask = tx_ant << (chainshift * band); + phy->mt76->chainmask = tx_ant << shift; mt76_set_stream_caps(phy->mt76, true); mt7915_set_stream_vht_txbf_caps(phy); @@ -1738,7 +1748,7 @@ mt7915_net_fill_forward_path(struct ieee80211_hw *hw, path->mtk_wdma.wdma_idx = wed->wdma_idx; path->mtk_wdma.bss = mvif->mt76.idx; path->mtk_wdma.wcid = is_mt7915(&dev->mt76) ? msta->wcid.idx : 0x3ff; - path->mtk_wdma.queue = phy != &dev->phy; + path->mtk_wdma.queue = phy->mt76->band_idx; ctx->dev = NULL; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c index 1646b9ba2980..29b3f0cda1ce 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c @@ -51,7 +51,7 @@ mt7915_mcu_get_sta_nss(u16 mcs_map) break; } - return nss - 1; + return nss ? nss - 1 : 0; } static void @@ -576,9 +576,9 @@ mt7915_mcu_bss_ra_tlv(struct sk_buff *skb, struct ieee80211_vif *vif, ra->rx_streams = max_nss; ra->algo = 4; ra->train_up_rule = 2; - ra->train_up_high_thres = 110; - ra->train_up_rule_rssi = -70; - ra->low_traffic_thres = 2; + ra->train_up_high_thres = cpu_to_le16(110); + ra->train_up_rule_rssi = cpu_to_le16(-70); + ra->low_traffic_thres = cpu_to_le16(2); ra->phy_cap = cpu_to_le32(0xfdf); ra->interval = cpu_to_le32(500); ra->fast_interval = cpu_to_le32(100); @@ -2804,7 +2804,7 @@ int mt7915_mcu_set_chan_info(struct mt7915_phy *phy, int cmd) .center_ch = ieee80211_frequency_to_channel(freq1), .bw = mt76_connac_chan_bw(chandef), .tx_path_num = hweight16(phy->mt76->chainmask), - .rx_path = phy->mt76->chainmask >> (dev->chainshift * band), + .rx_path = mt7915_band_chainmask(phy), .band_idx = band, .channel_band = ch_band[chandef->chan->band], }; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.h b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.h index 4049ed864003..4c25abb0297d 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.h +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.h @@ -318,9 +318,9 @@ struct bss_info_ra { u8 antenna_idx; u8 train_up_rule; u8 rsv[3]; - unsigned short train_up_high_thres; - short train_up_rule_rssi; - unsigned short low_traffic_thres; + __le16 train_up_high_thres; + __le16 train_up_rule_rssi; + __le16 low_traffic_thres; __le16 max_phyrate; __le32 phy_cap; __le32 interval; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h index f1194d147dc8..27857347f05f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h @@ -393,6 +393,24 @@ mt7915_ext_phy(struct mt7915_dev *dev) return phy->priv; } +/* without dbdc, the chainmask is stored unshifted, even if the phy is + * bound to band 1 + */ +static inline u8 mt7915_band_chainshift(struct mt7915_phy *phy) +{ + struct mt7915_dev *dev = phy->dev; + + if (!dev->dbdc_support) + return 0; + + return phy->mt76->band_idx * dev->chainshift; +} + +static inline u16 mt7915_band_chainmask(struct mt7915_phy *phy) +{ + return phy->mt76->chainmask >> mt7915_band_chainshift(phy); +} + static inline u32 mt7915_check_adie(struct mt7915_dev *dev, bool sku) { u32 mask = sku ? MT_CONNINFRA_SKU_MASK : MT_ADIE_TYPE_MASK; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/pci.c b/drivers/net/wireless/mediatek/mt76/mt7915/pci.c index 07b0a5766eab..c24f1b12f064 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/pci.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/pci.c @@ -135,7 +135,6 @@ static int mt7915_pci_probe(struct pci_dev *pdev, mdev = &dev->mt76; mt7915_wfsys_reset(dev); - hif2 = mt7915_pci_init_hif2(pdev); ret = mt7915_mmio_wed_init(dev, pdev, true, &irq); if (ret < 0) @@ -145,16 +144,20 @@ static int mt7915_pci_probe(struct pci_dev *pdev, hif2 = mt7915_pci_init_hif2(pdev); ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES); - if (ret < 0) + if (ret < 0) { + mt7915_put_hif2(hif2); goto free_device; + } irq = pdev->irq; } ret = devm_request_irq(mdev->dev, irq, mt7915_irq_handler, IRQF_SHARED, KBUILD_MODNAME, dev); - if (ret) + if (ret) { + mt7915_put_hif2(hif2); goto free_wed_or_irq_vector; + } /* master switch of PCIe tnterrupt enable */ mt76_wr(dev, MT_PCIE_MAC_INT_ENABLE, 0xff); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/soc.c b/drivers/net/wireless/mediatek/mt76/mt7915/soc.c index c823a7554a3a..df641c133480 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/soc.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/soc.c @@ -913,7 +913,7 @@ static void mt7986_wmac_clock_enable(struct mt7915_dev *dev, u32 adie_type) read_poll_timeout(mt76_rr, cur, !(cur & MT_SLP_CTRL_BSY_MASK), USEC_PER_MSEC, 50 * USEC_PER_MSEC, false, - dev, MT_ADIE_SLP_CTRL_CK0(0)); + dev, MT_ADIE_SLP_CTRL_CK0(1)); } mt76_wmac_spi_unlock(dev); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/testmode.c b/drivers/net/wireless/mediatek/mt76/mt7915/testmode.c index d534fff5c952..2a9e5d9ff101 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/testmode.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/testmode.c @@ -694,9 +694,7 @@ mt7915_tm_set_params(struct mt76_phy *mphy, struct nlattr **tb, { struct mt76_testmode_data *td = &mphy->test; struct mt7915_phy *phy = mphy->priv; - struct mt7915_dev *dev = phy->dev; - u32 chainmask = mphy->chainmask, changed = 0; - bool ext_phy = phy != &dev->phy; + u32 chainmask = mt7915_band_chainmask(phy), changed = 0; int i; BUILD_BUG_ON(NUM_TM_CHANGED >= 32); @@ -705,7 +703,6 @@ mt7915_tm_set_params(struct mt76_phy *mphy, struct nlattr **tb, td->state == MT76_TM_STATE_OFF) return 0; - chainmask = ext_phy ? chainmask >> dev->chainshift : chainmask; if (td->tx_antenna_mask > chainmask) return -EINVAL; diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mac.c b/drivers/net/wireless/mediatek/mt76/mt7921/mac.c index 251a9a1f664d..0d48dbb2354f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/mac.c @@ -673,6 +673,9 @@ void mt7921_mac_reset_work(struct work_struct *work) cancel_work_sync(&pm->wake_work); for (i = 0; i < 10; i++) { + if (atomic_read(&dev->mt76.bus_hung)) + return; + mutex_lock(&dev->mt76.mutex); ret = mt792x_dev_reset(dev); mutex_unlock(&dev->mt76.mutex); diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/pci.c b/drivers/net/wireless/mediatek/mt76/mt7921/pci.c index a0c9df3c2cc7..85caa2fbb78f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/pci.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/pci.c @@ -554,6 +554,108 @@ static int mt7921_pci_resume(struct device *device) return err; } +static u32 mt7921_aer_rr(struct mt76_dev *mdev, u32 offset) +{ + return 0; +} + +static void mt7921_aer_wr(struct mt76_dev *mdev, u32 offset, u32 val) +{ + ; +} + +static u32 mt791_aer_rmw(struct mt76_dev *mdev, u32 offset, u32 mask, u32 val) +{ + return 0; +} + +static const struct mt76_bus_ops mt7921_aer_bus_hung_ops = { + .rr = mt7921_aer_rr, + .wr = mt7921_aer_wr, + .rmw = mt791_aer_rmw, + .type = MT76_BUS_MMIO +}; + +static void mt7921_pci_set_aer_bus_hung_ops(struct mt792x_dev *dev) +{ + if (READ_ONCE(dev->mt76.bus) == &mt7921_aer_bus_hung_ops) + return; + + atomic_set(&dev->mt76.bus_hung, true); + WRITE_ONCE(dev->mt76.bus, &mt7921_aer_bus_hung_ops); +} + +static pci_ers_result_t mt7921_error_detected(struct pci_dev *pdev, + pci_channel_state_t state) +{ + struct mt76_dev *mdev = pci_get_drvdata(pdev); + struct mt792x_dev *dev = container_of(mdev, struct mt792x_dev, mt76); + u32 aer_unc_val = 0, aer_co_val = 0; + + dev_err(mdev->dev, "PCIE error detect state: %d\n", state); + + /* Clear SW IRQ tasklet first */ + tasklet_kill(&mdev->irq_tasklet); + + if (state == pci_channel_io_perm_failure) { + mt7921_pci_set_aer_bus_hung_ops(dev); + return PCI_ERS_RESULT_DISCONNECT; + } + + pci_read_config_dword(pdev, PCIE_AER_UNC_STATUS_OFFSET, &aer_unc_val); + pci_read_config_dword(pdev, PCIE_AER_CO_STATUS_OFFSET, &aer_co_val); + + dev_warn(mdev->dev, "PCIE_AER_UNC_STATUS_OFFSET: 0x%x\n", aer_unc_val); + dev_warn(mdev->dev, "PCIE_AER_CO_STATUS_OFFSET: 0x%x\n", aer_co_val); + + /** + * Due to this error is from link error and this AER is un-correctable, + * so can't covered by device + **/ + if (aer_unc_val != 0) { + mt7921_pci_set_aer_bus_hung_ops(dev); + return PCI_ERS_RESULT_DISCONNECT; + } + + /** + * Try to recover it when state is pci_channel_io_frozen or + * AER is correctable error + **/ + if (state == pci_channel_io_frozen || aer_co_val != 0) { + /* Disable PCIE activity first. */ + pci_disable_device(pdev); + return PCI_ERS_RESULT_NEED_RESET; + } + + return PCI_ERS_RESULT_NONE; +} + +static pci_ers_result_t mt7921_slot_reset(struct pci_dev *pdev) +{ + struct mt76_dev *mdev = pci_get_drvdata(pdev); + int ret = 0; + + ret = pci_enable_device_mem(pdev); + + if (ret) { + dev_err(mdev->dev, "pci_enable_device_mem failed: %d\n", ret); + return PCI_ERS_RESULT_DISCONNECT; + } + + pci_set_master(pdev); + pci_restore_state(pdev); + pci_save_state(pdev); + /* Also try do the vendor reset to let it more clear. */ + mt792x_reset(mdev); + + return PCI_ERS_RESULT_RECOVERED; +} + +static const struct pci_error_handlers mt7921_err_handler = { + .error_detected = mt7921_error_detected, + .slot_reset = mt7921_slot_reset, +}; + static void mt7921_pci_shutdown(struct pci_dev *pdev) { mt7921_pci_remove(pdev); @@ -568,6 +670,7 @@ static struct pci_driver mt7921_pci_driver = { .remove = mt7921_pci_remove, .shutdown = mt7921_pci_shutdown, .driver.pm = pm_sleep_ptr(&mt7921_pm_ops), + .err_handler = &mt7921_err_handler, }; module_pci_driver(mt7921_pci_driver); diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/main.c b/drivers/net/wireless/mediatek/mt76/mt7925/main.c index 8ef0d981adc9..1eeb33981c74 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7925/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7925/main.c @@ -178,20 +178,36 @@ mt7925_init_eht_caps(struct mt792x_phy *phy, enum nl80211_band band, IEEE80211_EHT_PHY_CAP0_SU_BEAMFORMER | IEEE80211_EHT_PHY_CAP0_SU_BEAMFORMEE; + if (band == NL80211_BAND_6GHZ && is_320mhz_supported(&phy->dev->mt76)) + eht_cap_elem->phy_cap_info[0] |= + IEEE80211_EHT_PHY_CAP0_320MHZ_IN_6GHZ; + + val = (sts > 3) ? sts - 1 : 3; + eht_cap_elem->phy_cap_info[0] |= - u8_encode_bits(u8_get_bits(sts - 1, BIT(0)), + u8_encode_bits(u8_get_bits(val, BIT(0)), IEEE80211_EHT_PHY_CAP0_BEAMFORMEE_SS_80MHZ_MASK); eht_cap_elem->phy_cap_info[1] = - u8_encode_bits(u8_get_bits(sts - 1, GENMASK(2, 1)), + u8_encode_bits(u8_get_bits(val, GENMASK(2, 1)), IEEE80211_EHT_PHY_CAP1_BEAMFORMEE_SS_80MHZ_MASK) | - u8_encode_bits(sts - 1, + u8_encode_bits(val, IEEE80211_EHT_PHY_CAP1_BEAMFORMEE_SS_160MHZ_MASK); + if (band == NL80211_BAND_6GHZ && is_320mhz_supported(&phy->dev->mt76)) + eht_cap_elem->phy_cap_info[1] |= + u8_encode_bits(val, + IEEE80211_EHT_PHY_CAP1_BEAMFORMEE_SS_320MHZ_MASK); + eht_cap_elem->phy_cap_info[2] = u8_encode_bits(sts - 1, IEEE80211_EHT_PHY_CAP2_SOUNDING_DIM_80MHZ_MASK) | u8_encode_bits(sts - 1, IEEE80211_EHT_PHY_CAP2_SOUNDING_DIM_160MHZ_MASK); + if (band == NL80211_BAND_6GHZ && is_320mhz_supported(&phy->dev->mt76)) + eht_cap_elem->phy_cap_info[2] |= + u8_encode_bits(sts - 1, + IEEE80211_EHT_PHY_CAP2_SOUNDING_DIM_320MHZ_MASK); + eht_cap_elem->phy_cap_info[3] = IEEE80211_EHT_PHY_CAP3_NG_16_SU_FEEDBACK | IEEE80211_EHT_PHY_CAP3_NG_16_MU_FEEDBACK | @@ -212,7 +228,8 @@ mt7925_init_eht_caps(struct mt792x_phy *phy, enum nl80211_band band, u8_encode_bits(u8_get_bits(0x11, GENMASK(1, 0)), IEEE80211_EHT_PHY_CAP5_MAX_NUM_SUPP_EHT_LTF_MASK); - val = width == NL80211_CHAN_WIDTH_160 ? 0x7 : + val = width == NL80211_CHAN_WIDTH_320 ? 0xf : + width == NL80211_CHAN_WIDTH_160 ? 0x7 : width == NL80211_CHAN_WIDTH_80 ? 0x3 : 0x1; eht_cap_elem->phy_cap_info[6] = u8_encode_bits(u8_get_bits(0x11, GENMASK(4, 2)), @@ -234,6 +251,11 @@ mt7925_init_eht_caps(struct mt792x_phy *phy, enum nl80211_band band, eht_nss->bw._160.rx_tx_mcs9_max_nss = val; eht_nss->bw._160.rx_tx_mcs11_max_nss = val; eht_nss->bw._160.rx_tx_mcs13_max_nss = val; + if (band == NL80211_BAND_6GHZ && is_320mhz_supported(&phy->dev->mt76)) { + eht_nss->bw._320.rx_tx_mcs9_max_nss = val; + eht_nss->bw._320.rx_tx_mcs11_max_nss = val; + eht_nss->bw._320.rx_tx_mcs13_max_nss = val; + } } int mt7925_init_mlo_caps(struct mt792x_phy *phy) @@ -1688,9 +1710,15 @@ static int mt7925_set_sar_specs(struct ieee80211_hw *hw, int err; mt792x_mutex_acquire(dev); + err = mt7925_mcu_set_clc(dev, dev->mt76.alpha2, + dev->country_ie_env); + if (err < 0) + goto out; + err = mt7925_set_tx_sar_pwr(hw, sar); - mt792x_mutex_release(dev); +out: + mt792x_mutex_release(dev); return err; } diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c index bc19e800b41c..d11c7ff14bc8 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c @@ -3733,7 +3733,7 @@ mt7925_mcu_rate_txpower_band(struct mt76_phy *phy, memcpy(tx_power_tlv->alpha2, dev->alpha2, sizeof(dev->alpha2)); tx_power_tlv->n_chan = num_ch; tx_power_tlv->tag = cpu_to_le16(0x1); - tx_power_tlv->len = cpu_to_le16(msg_len); + tx_power_tlv->len = cpu_to_le16(msg_len - 4); switch (band) { case NL80211_BAND_2GHZ: diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c b/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c index 0ab827f52fd7..1d8b8bec8e0e 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/debugfs.c @@ -645,7 +645,7 @@ mt7996_sta_hw_queue_read(void *data, struct ieee80211_sta *sta) if (!mlink) continue; - msta_link = rcu_dereference(msta->link[link_id]); + msta_link = mt7996_sta_link(msta, link_id); if (!msta_link) continue; @@ -953,16 +953,34 @@ bool mt7996_debugfs_rx_log(struct mt7996_dev *dev, const void *data, int len) #ifdef CONFIG_MAC80211_DEBUGFS /** per-station debugfs **/ -static ssize_t mt7996_sta_fixed_rate_set(struct file *file, - const char __user *user_buf, - size_t count, loff_t *ppos) +static int +mt7996_queues_show(struct seq_file *s, void *data) +{ + struct ieee80211_sta *sta = s->private; + + mt7996_sta_hw_queue_read(s, sta); + + return 0; +} + +DEFINE_SHOW_ATTRIBUTE(mt7996_queues); + +void mt7996_sta_add_debugfs(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + struct ieee80211_sta *sta, struct dentry *dir) +{ + debugfs_create_file("hw-queues", 0400, dir, sta, &mt7996_queues_fops); +} + +static ssize_t mt7996_link_sta_fixed_rate_set(struct file *file, + const char __user *user_buf, + size_t count, loff_t *ppos) { #define SHORT_PREAMBLE 0 #define LONG_PREAMBLE 1 - struct ieee80211_sta *sta = file->private_data; - struct mt7996_sta *msta = (struct mt7996_sta *)sta->drv_priv; + struct ieee80211_link_sta *link_sta = file->private_data; + struct mt7996_sta *msta = (struct mt7996_sta *)link_sta->sta->drv_priv; struct mt7996_dev *dev = msta->vif->deflink.phy->dev; - struct mt7996_sta_link *msta_link = &msta->deflink; + struct mt7996_sta_link *msta_link; struct ra_rate phy = {}; char buf[100]; int ret; @@ -981,12 +999,13 @@ static ssize_t mt7996_sta_fixed_rate_set(struct file *file, /* mode - cck: 0, ofdm: 1, ht: 2, gf: 3, vht: 4, he_su: 8, he_er: 9 EHT: 15 * bw - bw20: 0, bw40: 1, bw80: 2, bw160: 3, BW320: 4 - * nss - vht: 1~4, he: 1~4, eht: 1~4, others: ignore * mcs - cck: 0~4, ofdm: 0~7, ht: 0~32, vht: 0~9, he_su: 0~11, he_er: 0~2, eht: 0~13 + * nss - vht: 1~4, he: 1~4, eht: 1~4, others: ignore * gi - (ht/vht) lgi: 0, sgi: 1; (he) 0.8us: 0, 1.6us: 1, 3.2us: 2 * preamble - short: 1, long: 0 - * ldpc - off: 0, on: 1 * stbc - off: 0, on: 1 + * ldpc - off: 0, on: 1 + * spe - off: 0, on: 1 * ltf - 1xltf: 0, 2xltf: 1, 4xltf: 2 */ if (sscanf(buf, "%hhu %hhu %hhu %hhu %hu %hhu %hhu %hhu %hhu %hu", @@ -994,9 +1013,16 @@ static ssize_t mt7996_sta_fixed_rate_set(struct file *file, &phy.preamble, &phy.stbc, &phy.ldpc, &phy.spe, <f) != 10) { dev_warn(dev->mt76.dev, "format: Mode BW MCS NSS GI Preamble STBC LDPC SPE ltf\n"); - goto out; + return -EINVAL; } + mutex_lock(&dev->mt76.mutex); + + msta_link = mt7996_sta_link_protected(dev, msta, link_sta->link_id); + if (!msta_link) { + ret = -EINVAL; + goto out; + } phy.wlan_idx = cpu_to_le16(msta_link->wcid.idx); phy.gi = cpu_to_le16(gi); phy.ltf = cpu_to_le16(ltf); @@ -1005,36 +1031,26 @@ static ssize_t mt7996_sta_fixed_rate_set(struct file *file, ret = mt7996_mcu_set_fixed_rate_ctrl(dev, &phy, 0); if (ret) - return -EFAULT; + goto out; + ret = count; out: - return count; + mutex_unlock(&dev->mt76.mutex); + return ret; } static const struct file_operations fops_fixed_rate = { - .write = mt7996_sta_fixed_rate_set, + .write = mt7996_link_sta_fixed_rate_set, .open = simple_open, .owner = THIS_MODULE, .llseek = default_llseek, }; -static int -mt7996_queues_show(struct seq_file *s, void *data) -{ - struct ieee80211_sta *sta = s->private; - - mt7996_sta_hw_queue_read(s, sta); - - return 0; -} - -DEFINE_SHOW_ATTRIBUTE(mt7996_queues); - -void mt7996_sta_add_debugfs(struct ieee80211_hw *hw, struct ieee80211_vif *vif, - struct ieee80211_sta *sta, struct dentry *dir) +void mt7996_link_sta_add_debugfs(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + struct ieee80211_link_sta *link_sta, + struct dentry *dir) { - debugfs_create_file("fixed_rate", 0600, dir, sta, &fops_fixed_rate); - debugfs_create_file("hw-queues", 0400, dir, sta, &mt7996_queues_fops); + debugfs_create_file("fixed_rate", 0600, dir, link_sta, &fops_fixed_rate); } #endif diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/init.c b/drivers/net/wireless/mediatek/mt76/mt7996/init.c index efbd46d64901..d0a4530db1cf 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/init.c @@ -34,7 +34,6 @@ static const struct ieee80211_iface_combination if_comb_global = { BIT(NL80211_CHAN_WIDTH_40) | BIT(NL80211_CHAN_WIDTH_80) | BIT(NL80211_CHAN_WIDTH_160), - .beacon_int_min_gcd = 100, }; static const struct ieee80211_iface_combination if_comb_global_7992 = { @@ -47,7 +46,6 @@ static const struct ieee80211_iface_combination if_comb_global_7992 = { BIT(NL80211_CHAN_WIDTH_40) | BIT(NL80211_CHAN_WIDTH_80) | BIT(NL80211_CHAN_WIDTH_160), - .beacon_int_min_gcd = 100, }; static const struct ieee80211_iface_limit if_limits[] = { @@ -1502,7 +1500,6 @@ mt7996_init_eht_caps(struct mt7996_phy *phy, enum nl80211_band band, struct ieee80211_sta_eht_cap *eht_cap = &data->eht_cap; struct ieee80211_eht_cap_elem_fixed *eht_cap_elem = &eht_cap->eht_cap_elem; struct ieee80211_eht_mcs_nss_supp *eht_nss = &eht_cap->eht_mcs_nss_supp; - enum nl80211_chan_width width = phy->mt76->chandef.width; int nss = hweight8(phy->mt76->antenna_mask); int sts = hweight16(phy->mt76->chainmask); u8 val; @@ -1578,11 +1575,16 @@ mt7996_init_eht_caps(struct mt7996_phy *phy, enum nl80211_band band, u8_encode_bits(u8_get_bits(1, GENMASK(1, 0)), IEEE80211_EHT_PHY_CAP5_MAX_NUM_SUPP_EHT_LTF_MASK); - val = width == NL80211_CHAN_WIDTH_320 ? 0xf : - width == NL80211_CHAN_WIDTH_160 ? 0x7 : - width == NL80211_CHAN_WIDTH_80 ? 0x3 : 0x1; - eht_cap_elem->phy_cap_info[6] = - u8_encode_bits(val, IEEE80211_EHT_PHY_CAP6_MCS15_SUPP_MASK); + eht_cap_elem->phy_cap_info[6] = IEEE80211_EHT_PHY_CAP6_MCS15_SUPP_MASK; + if (band != NL80211_BAND_6GHZ) { + eht_cap_elem->phy_cap_info[6] &= + ~IEEE80211_EHT_PHY_CAP6_MCS15_SUPP_320MHZ; + + if (band != NL80211_BAND_5GHZ) + eht_cap_elem->phy_cap_info[6] &= + ~(IEEE80211_EHT_PHY_CAP6_MCS15_SUPP_160MHZ | + IEEE80211_EHT_PHY_CAP6_MCS15_SUPP_80MHZ); + } val = u8_encode_bits(nss, IEEE80211_EHT_MCS_NSS_RX) | u8_encode_bits(nss, IEEE80211_EHT_MCS_NSS_TX); diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c index e55bb232bdeb..dc1a3e91218a 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c @@ -87,7 +87,7 @@ static struct mt76_wcid *mt7996_rx_get_wcid(struct mt7996_dev *dev, if (mlink->band_idx != band_idx) continue; - msta_link = rcu_dereference(msta->link[i]); + msta_link = mt7996_sta_link(msta, i); break; } @@ -480,7 +480,13 @@ mt7996_mac_fill_rx(struct mt7996_dev *dev, enum mt76_rxq_id q, memset(status, 0, sizeof(*status)); band_idx = FIELD_GET(MT_RXD1_NORMAL_BAND_IDX, rxd1); + if (!mt7996_band_valid(dev, band_idx)) + return -EINVAL; + mphy = dev->mt76.phys[band_idx]; + if (!mphy) + return -EINVAL; + phy = mphy->priv; status->phy_idx = mphy->band_idx; @@ -893,6 +899,33 @@ mt7996_mac_write_txwi_80211(struct mt7996_dev *dev, __le32 *txwi, txwi[6] |= cpu_to_le32(MT_TXD6_DIS_MAT); } +/* The WLAN_IDX in the TXD and TXP must belong to the primary or secondary + * link of an MLD station; any other link id can make the firmware spin when + * that link is in powersave. Completion events carry the same index, so the + * wcid used for status tracking and accounting must match it + */ +struct mt76_wcid *mt7996_get_tx_wcid(struct mt76_wcid *wcid) +{ + struct mt7996_sta_link *msta_link; + struct mt7996_sta *msta; + + if (!wcid->sta) + return wcid; + + msta_link = container_of(wcid, struct mt7996_sta_link, wcid); + msta = msta_link->sta; + + if (!msta || wcid->link_id == msta->seclink_id || + wcid->link_id == msta->deflink_id) + return wcid; + + msta_link = mt7996_sta_link(msta, msta->deflink_id); + if (msta_link) + return &msta_link->wcid; + + return wcid; +} + void mt7996_mac_write_txwi(struct mt7996_dev *dev, __le32 *txwi, struct sk_buff *skb, struct mt76_wcid *wcid, struct ieee80211_key_conf *key, int pid, @@ -1069,10 +1102,15 @@ int mt7996_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, IEEE80211_TX_CTRL_MLO_LINK); } + /* non-MLD frames are LINK_UNSPECIFIED; use the wcid's own link */ + if (link_id == IEEE80211_LINK_UNSPECIFIED && + wcid != &dev->mt76.global_wcid) + link_id = wcid->link_id; + if (link_id != wcid->link_id && link_id != IEEE80211_LINK_UNSPECIFIED) { if (msta) { struct mt7996_sta_link *msta_link = - rcu_dereference(msta->link[link_id]); + mt7996_sta_link(msta, link_id); if (msta_link) wcid = &msta_link->wcid; @@ -1125,6 +1163,8 @@ int mt7996_tx_prepare_skb(struct mt76_dev *mdev, void *txwi_ptr, tx_info->buf[1].len, DMA_TO_DEVICE); } + wcid = mt7996_get_tx_wcid(wcid); + pid = mt76_tx_status_skb_add(mdev, wcid, tx_info->skb); memset(txwi_ptr, 0, MT_TXD_SIZE); /* Transmit non qos data by 802.11 header and need to fill txd by host*/ @@ -1408,7 +1448,7 @@ mt7996_mac_tx_free(struct mt7996_dev *dev, void *data, int len) IEEE80211_MLD_MAX_NUM_LINKS) { struct mt7996_sta_link *msta_link; - msta_link = rcu_dereference(msta->link[id]); + msta_link = mt7996_sta_link(msta, id); if (!msta_link) continue; @@ -2476,6 +2516,7 @@ mt7996_mac_full_reset(struct mt7996_dev *dev) dev->recovery.hw_full_reset = true; + set_bit(MT76_MCU_RESET, &dev->mphy.state); wake_up(&dev->mt76.mcu.wait); ieee80211_stop_queues(hw); @@ -2498,10 +2539,10 @@ mt7996_mac_full_reset(struct mt7996_dev *dev) phy->omac_mask = 0; ieee80211_iterate_stations_atomic(hw, mt7996_mac_reset_sta_iter, dev); + mt76_reset_device(&dev->mt76); ieee80211_iterate_active_interfaces_atomic(hw, IEEE80211_IFACE_SKIP_SDATA_NOT_IN_DRIVER, mt7996_mac_reset_vif_iter, dev); - mt76_reset_device(&dev->mt76); INIT_LIST_HEAD(&dev->sta_rc_list); INIT_LIST_HEAD(&dev->twt_list); @@ -2579,8 +2620,8 @@ void mt7996_mac_reset_work(struct work_struct *work) set_bit(MT76_RESET, &dev->mphy.state); set_bit(MT76_MCU_RESET, &dev->mphy.state); - mt76_abort_scan(&dev->mt76); wake_up(&dev->mt76.mcu.wait); + mt76_abort_scan(&dev->mt76); cancel_work_sync(&dev->wed_rro.work); mt7996_for_each_phy(dev, phy) { @@ -2589,6 +2630,8 @@ void mt7996_mac_reset_work(struct work_struct *work) cancel_delayed_work_sync(&phy->mt76->mac_work); } + mutex_lock(&dev->mt76.mutex); + mt76_worker_disable(&dev->mt76.tx_worker); mt76_for_each_q_rx(&dev->mt76, i) { if (mtk_wed_device_active(&dev->mt76.mmio.wed) && @@ -2599,8 +2642,6 @@ void mt7996_mac_reset_work(struct work_struct *work) } napi_disable(&dev->mt76.tx_napi); - mutex_lock(&dev->mt76.mutex); - mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_DMA_STOPPED); if (mt7996_wait_reset_state(dev, MT_MCU_CMD_RESET_DONE)) { diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/main.c b/drivers/net/wireless/mediatek/mt76/mt7996/main.c index 20da0c10669a..c6d7088b15ca 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/main.c @@ -207,8 +207,7 @@ mt7996_set_hw_key(struct ieee80211_hw *hw, enum set_key_cmd cmd, struct mt7996_sta *msta; msta = (struct mt7996_sta *)sta->drv_priv; - msta_link = mt76_dereference(msta->link[link_id], - &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, link_id); if (!msta_link) return 0; @@ -363,7 +362,8 @@ int mt7996_vif_link_add(struct mt76_phy *mphy, struct ieee80211_vif *vif, CONN_STATE_PORT_SECURE, true); rcu_assign_pointer(dev->mt76.wcid[idx], &msta_link->wcid); - ieee80211_iter_keys(mphy->hw, vif, mt7996_key_iter, &it); + if (!mlink->wcid->offchannel) + ieee80211_iter_keys(mphy->hw, vif, mt7996_key_iter, &it); if (vif->txq && !mlink->wcid->offchannel && mvif->mt76.deflink_id == IEEE80211_LINK_UNSPECIFIED) { @@ -950,12 +950,24 @@ mt7996_post_channel_switch(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct cfg80211_chan_def *chandef = &link_conf->chanreq.oper; struct mt7996_dev *dev = mt7996_hw_dev(hw); struct mt7996_phy *phy = mt7996_band_phy(dev, chandef->chan->band); - int ret; + struct mt7996_vif_link *link; + int ret = -EINVAL; mutex_lock(&dev->mt76.mutex); + link = mt7996_vif_conf_link(dev, vif, link_conf); + if (!link) + goto out; + + ret = mt7996_mcu_update_bss_rfch(phy, link); + if (ret) + goto out; + + ieee80211_iterate_stations_mtx(hw, mt7996_mcu_update_sta_rec_bw, link); + ret = mt7996_mcu_rdd_resume_tx(phy); +out: mutex_unlock(&dev->mt76.mutex); return ret; @@ -1224,7 +1236,7 @@ mt7996_mac_sta_event(struct mt7996_dev *dev, struct ieee80211_vif *vif, if (!link) continue; - msta_link = mt76_dereference(msta->link[link_id], &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, link_id); if (!msta_link) continue; @@ -1349,20 +1361,26 @@ static void mt7996_tx(struct ieee80211_hw *hw, struct ieee80211_vif *vif = info->control.vif; struct mt7996_vif *mvif = vif ? (void *)vif->drv_priv : NULL; struct mt76_wcid *wcid = &dev->mt76.global_wcid; + u8 deflink_id = IEEE80211_LINK_UNSPECIFIED; u8 link_id = u32_get_bits(info->control.flags, IEEE80211_TX_CTRL_MLO_LINK); rcu_read_lock(); + if (msta) + deflink_id = msta->deflink_id; + else if (mvif) + deflink_id = mvif->mt76.deflink_id; + + /* the primary link is unset until the first link has been added */ + if (deflink_id >= IEEE80211_MLD_MAX_NUM_LINKS) + deflink_id = 0; + /* Use primary link_id if the value from mac80211 is set to * IEEE80211_LINK_UNSPECIFIED. */ - if (link_id == IEEE80211_LINK_UNSPECIFIED) { - if (msta) - link_id = msta->deflink_id; - else if (mvif) - link_id = mvif->mt76.deflink_id; - } + if (link_id == IEEE80211_LINK_UNSPECIFIED) + link_id = deflink_id; if (vif && ieee80211_vif_is_mld(vif)) { struct ieee80211_bss_conf *link_conf; @@ -1372,7 +1390,7 @@ static void mt7996_tx(struct ieee80211_hw *hw, link_sta = rcu_dereference(sta->link[link_id]); if (!link_sta) - link_sta = rcu_dereference(sta->link[msta->deflink_id]); + link_sta = rcu_dereference(sta->link[deflink_id]); if (link_sta) { memcpy(hdr->addr1, link_sta->addr, ETH_ALEN); @@ -1414,11 +1432,11 @@ static void mt7996_tx(struct ieee80211_hw *hw, if (msta) { struct mt7996_sta_link *msta_link; - msta_link = rcu_dereference(msta->link[link_id]); + msta_link = mt7996_sta_link(msta, link_id); if (msta_link) wcid = &msta_link->wcid; } - mt76_tx(mphy, control->sta, wcid, skb); + mt76_tx(mphy, control->sta, mt7996_get_tx_wcid(wcid), skb); unlock: rcu_read_unlock(); } @@ -1722,8 +1740,6 @@ static void mt7996_sta_statistics(struct ieee80211_hw *hw, sinfo->txrate.flags = txrate->flags; sinfo->filled |= BIT_ULL(NL80211_STA_INFO_TX_BITRATE); } - sinfo->txrate.flags = txrate->flags; - sinfo->filled |= BIT_ULL(NL80211_STA_INFO_TX_BITRATE); sinfo->tx_failed = msta_link->wcid.stats.tx_failed; sinfo->filled |= BIT_ULL(NL80211_STA_INFO_TX_FAILED); @@ -1780,7 +1796,7 @@ static void mt7996_link_sta_rc_update(struct ieee80211_hw *hw, rcu_read_lock(); - msta_link = rcu_dereference(msta->link[link_sta->link_id]); + msta_link = mt7996_sta_link(msta, link_sta->link_id); if (msta_link) { struct mt7996_dev *dev = mt7996_hw_dev(hw); @@ -1801,7 +1817,7 @@ static void mt7996_sta_rate_ctrl_update(void *data, struct ieee80211_sta *sta) if (msta->vif != mvif) return; - msta_link = rcu_dereference(msta->link[msta->deflink_id]); + msta_link = mt7996_sta_link(msta, msta->deflink_id); if (msta_link) mt7996_link_rate_ctrl_update(&changed, msta_link); } @@ -1850,7 +1866,7 @@ static void mt7996_sta_set_4addr(struct ieee80211_hw *hw, if (!link) continue; - msta_link = mt76_dereference(msta->link[link_id], &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, link_id); if (!msta_link) continue; @@ -1888,7 +1904,7 @@ static void mt7996_sta_set_decap_offload(struct ieee80211_hw *hw, if (!link) continue; - msta_link = mt76_dereference(msta->link[link_id], &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, link_id); if (!msta_link) continue; @@ -2230,7 +2246,7 @@ mt7996_net_fill_forward_path(struct ieee80211_hw *hw, if (!mlink) return -EIO; - msta_link = rcu_dereference(msta->link[msta->deflink_id]); + msta_link = mt7996_sta_link(msta, msta->deflink_id); if (!msta_link) return -EIO; @@ -2288,6 +2304,7 @@ mt7996_change_vif_links(struct ieee80211_hw *hw, struct ieee80211_vif *vif, idx = get_free_idx(dev->mld_remap_idx_mask, 0, 15) - 1; if (idx < 0) { + dev->mld_idx_mask &= ~BIT_ULL(mvif->mld_group_idx); ret = -ENOSPC; goto out; } @@ -2371,6 +2388,7 @@ const struct ieee80211_ops mt7996_ops = { .twt_teardown_request = mt7996_twt_teardown_request, #ifdef CONFIG_MAC80211_DEBUGFS .sta_add_debugfs = mt7996_sta_add_debugfs, + .link_sta_add_debugfs = mt7996_link_sta_add_debugfs, #endif .set_radar_background = mt7996_set_radar_background, #ifdef CONFIG_NET_MEDIATEK_SOC_WED diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c index 72851399575f..92f8a28cd4a5 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c @@ -1063,7 +1063,7 @@ mt7996_mcu_bss_basic_tlv(struct sk_buff *skb, struct mt7996_sta_link *msta_link; int link_id = link_conf->link_id; - msta_link = rcu_dereference(msta->link[link_id]); + msta_link = mt7996_sta_link(msta, link_id); if (msta_link) sta_wlan_idx = msta_link->wcid.idx; } @@ -1177,6 +1177,22 @@ int mt7996_mcu_add_bss_info(struct mt7996_phy *phy, struct ieee80211_vif *vif, MCU_WMWA_UNI_CMD(BSS_INFO_UPDATE), true); } +int mt7996_mcu_update_bss_rfch(struct mt7996_phy *phy, struct mt7996_vif_link *link) +{ + struct mt7996_dev *dev = phy->dev; + struct sk_buff *skb; + + skb = __mt7996_mcu_alloc_bss_req(&dev->mt76, &link->mt76, + MT7996_BSS_UPDATE_MAX_SIZE); + if (IS_ERR(skb)) + return PTR_ERR(skb); + + mt7996_mcu_bss_rfch_tlv(skb, phy); + + return mt76_mcu_skb_send_msg(&dev->mt76, skb, + MCU_WMWA_UNI_CMD(BSS_INFO_UPDATE), true); +} + int mt7996_mcu_set_timing(struct mt7996_phy *phy, struct ieee80211_vif *vif, struct ieee80211_bss_conf *link_conf) { @@ -1239,7 +1255,7 @@ int mt7996_mcu_add_tx_ba(struct mt7996_dev *dev, struct mt7996_sta_link *msta_link; struct mt7996_vif_link *link; - msta_link = mt76_dereference(msta->link[link_id], &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, link_id); if (!msta_link) continue; @@ -1273,7 +1289,7 @@ int mt7996_mcu_add_rx_ba(struct mt7996_dev *dev, struct mt7996_sta_link *msta_link; struct mt7996_vif_link *link; - msta_link = mt76_dereference(msta->link[link_id], &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, link_id); if (!msta_link) continue; @@ -2007,7 +2023,7 @@ int mt7996_mcu_set_fixed_field(struct mt7996_dev *dev, struct mt7996_sta *msta, if (!mlink) goto error_unlock; - msta_link = rcu_dereference(msta->link[link_id]); + msta_link = mt7996_sta_link(msta, link_id); if (!msta_link) goto error_unlock; @@ -2096,7 +2112,7 @@ mt7996_mcu_add_rate_ctrl_fixed(struct mt7996_dev *dev, struct mt7996_sta *msta, if (!link) goto error_unlock; - msta_link = rcu_dereference(msta->link[link_id]); + msta_link = mt7996_sta_link(msta, link_id); if (!msta_link) goto error_unlock; @@ -2303,7 +2319,7 @@ int mt7996_mcu_add_rate_ctrl(struct mt7996_dev *dev, struct mt7996_sta *msta, if (!link) goto error_unlock; - msta_link = rcu_dereference(msta->link[link_id]); + msta_link = mt7996_sta_link(msta, link_id); if (!msta_link) goto error_unlock; @@ -2392,7 +2408,7 @@ mt7996_mcu_sta_mld_setup_tlv(struct mt7996_dev *dev, struct sk_buff *skb, unsigned int link_id; struct tlv *tlv; - msta_link = mt76_dereference(msta->link[msta->deflink_id], &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, msta->deflink_id); if (!msta_link) return; @@ -2406,8 +2422,8 @@ mt7996_mcu_sta_mld_setup_tlv(struct mt7996_dev *dev, struct sk_buff *skb, mld_setup->primary_id = cpu_to_le16(msta_link->wcid.idx); if (nlinks > 1) { - msta_link = mt76_dereference(msta->link[msta->seclink_id], - &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, + msta->seclink_id); if (!msta_link) return; } @@ -2418,7 +2434,7 @@ mt7996_mcu_sta_mld_setup_tlv(struct mt7996_dev *dev, struct sk_buff *skb, for_each_sta_active_link(vif, sta, link_sta, link_id) { struct mt7996_vif_link *link; - msta_link = mt76_dereference(msta->link[link_id], &dev->mt76); + msta_link = mt7996_sta_link_protected(dev, msta, link_id); if (!msta_link) continue; @@ -2541,6 +2557,49 @@ int mt7996_mcu_teardown_mld_sta(struct mt7996_dev *dev, MCU_WMWA_UNI_CMD(STA_REC_UPDATE), true); } +void mt7996_mcu_update_sta_rec_bw(void *data, struct ieee80211_sta *sta) +{ + struct mt7996_vif_link *link = (struct mt7996_vif_link *)data; + struct mt7996_sta *msta = (struct mt7996_sta *)sta->drv_priv; + struct mt7996_sta_link *msta_link; + struct mt7996_dev *dev; + struct ieee80211_bss_conf *link_conf; + struct ieee80211_link_sta *link_sta; + struct ieee80211_vif *vif; + struct sk_buff *skb; + int link_id; + + if (link->mt76.mvif != &msta->vif->mt76) + return; + + dev = link->phy->dev; + link_id = link->msta_link.wcid.link_id; + link_sta = link_sta_dereference_protected(sta, link_id); + if (!link_sta) + return; + + msta_link = mt7996_sta_link_protected(dev, msta, link_id); + if (!msta_link) + return; + + vif = container_of((void *)msta->vif, struct ieee80211_vif, drv_priv); + link_conf = link_conf_dereference_protected(vif, link_id); + if (!link_conf) + return; + + skb = __mt76_connac_mcu_alloc_sta_req(&dev->mt76, &link->mt76, + &msta_link->wcid, + MT7996_STA_UPDATE_MAX_SIZE); + if (IS_ERR(skb)) + return; + + mt7996_mcu_sta_bfer_tlv(dev, skb, link_conf, link_sta, link); + mt7996_mcu_sta_rate_ctrl_tlv(skb, dev, vif, link_conf, link_sta, link); + + mt76_mcu_skb_send_msg(&dev->mt76, skb, + MCU_WMWA_UNI_CMD(STA_REC_UPDATE), true); +} + static int mt7996_mcu_sta_key_tlv(struct mt76_dev *dev, struct mt76_wcid *wcid, struct sk_buff *skb, @@ -3676,6 +3735,9 @@ int mt7996_mcu_rdd_background_enable(struct mt7996_phy *phy, struct mt7996_dev *dev = phy->dev; int err, region, rdd_idx = mt7996_get_rdd_idx(phy, true); + if (rdd_idx < 0) + return -EINVAL; + if (!chandef) { /* disable offchain */ err = mt7996_mcu_rdd_cmd(dev, RDD_STOP, rdd_idx, 0); if (err) @@ -3866,8 +3928,14 @@ int mt7996_mcu_get_eeprom(struct mt7996_dev *dev, u32 offset, u8 *buf, u32 buf_l if (valid) { u32 addr = le32_to_cpu(*(__le32 *)(skb->data + 12)); - if (!buf) + if (!buf) { + if (addr > dev->mt76.eeprom.size - + MT7996_EEPROM_BLOCK_SIZE) { + dev_kfree_skb(skb); + return -EINVAL; + } buf = (u8 *)dev->mt76.eeprom.data + addr; + } if (!buf_len || buf_len > MT7996_EEPROM_BLOCK_SIZE) buf_len = MT7996_EEPROM_BLOCK_SIZE; @@ -3936,21 +4004,31 @@ int mt7996_mcu_get_chip_config(struct mt7996_dev *dev, u32 *cap) return ret; /* fixed field */ + if (skb->len < 4) { + dev_kfree_skb(skb); + return -EINVAL; + } skb_pull(skb, 4); buf = skb->data; - while (buf - skb->data < skb->len) { + while (buf - skb->data + sizeof(struct tlv) <= skb->len) { struct tlv *tlv = (struct tlv *)buf; + u16 tlv_len = le16_to_cpu(tlv->len); + + if (tlv_len < sizeof(*tlv) || + tlv_len > skb->len - (buf - skb->data)) + break; switch (le16_to_cpu(tlv->tag)) { case UNI_EVENT_CHIP_CONFIG_EFUSE_VERSION: - *cap = le32_to_cpu(*(__le32 *)(buf + sizeof(*tlv))); + if (tlv_len >= sizeof(*tlv) + sizeof(__le32)) + *cap = le32_to_cpu(*(__le32 *)(buf + sizeof(*tlv))); break; default: break; } - buf += le16_to_cpu(tlv->len); + buf += tlv_len; } dev_kfree_skb(skb); diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h index 7b51d7346bca..848c6c575115 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h @@ -828,7 +828,7 @@ enum { #define MT7996_BEACON_UPDATE_SIZE (sizeof(struct bss_req_hdr) + \ sizeof(struct bss_bcn_content_tlv) + \ 4 + MT_TXD_SIZE + \ - sizeof(struct bss_bcn_cntdwn_tlv) + \ + sizeof(struct bss_bcn_cntdwn_tlv) * 2 + \ sizeof(struct bss_bcn_mbss_tlv)) #define MT7996_MAX_BSS_OFFLOAD_SIZE 2048 #define MT7996_MAX_BEACON_SIZE (MT7996_MAX_BSS_OFFLOAD_SIZE - \ diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c b/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c index 80db102ed809..e37b95f5b00b 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c @@ -17,6 +17,8 @@ static bool wed_enable; module_param(wed_enable, bool, 0644); +#define INVALID_REG_ADDR 0xffffffff + static const struct __base mt7996_reg_base[] = { [WF_AGG_BASE] = { { 0x820e2000, 0x820f2000, 0x830e2000 } }, [WF_ARB_BASE] = { { 0x820e3000, 0x820f3000, 0x830e3000 } }, @@ -54,6 +56,14 @@ static const u32 mt7996_offs[] = { [MIB_BSCR7] = 0x9e8, [MIB_BSCR17] = 0xa10, [MIB_TRDR1] = 0xa28, + [MIB_TSCR0] = 0x6b0, + [MIB_TSCR1] = 0x6b4, + [MIB_TSCR2] = 0x6b8, + [MIB_TSCR3] = 0x6bc, + [MIB_TSCR4] = 0x6c0, + [MIB_TSCR5] = 0x6c4, + [MIB_TSCR6] = 0x6c8, + [MIB_TSCR7] = 0x6d0, [HIF_REMAP_L1] = 0x24, [HIF_REMAP_BASE_L1] = 0x130000, [HIF_REMAP_L2] = 0x1b4, @@ -91,6 +101,14 @@ static const u32 mt7992_offs[] = { [MIB_BSCR7] = 0xae4, [MIB_BSCR17] = 0xb0c, [MIB_TRDR1] = 0xb24, + [MIB_TSCR0] = 0x6b0, + [MIB_TSCR1] = 0x6b4, + [MIB_TSCR2] = 0x6b8, + [MIB_TSCR3] = 0x6bc, + [MIB_TSCR4] = 0x6c0, + [MIB_TSCR5] = 0x6c4, + [MIB_TSCR6] = 0x6c8, + [MIB_TSCR7] = 0x6d0, [HIF_REMAP_L1] = 0x8, [HIF_REMAP_BASE_L1] = 0x40000, [HIF_REMAP_L2] = 0x1b4, @@ -128,6 +146,14 @@ static const u32 mt7990_offs[] = { [MIB_BSCR7] = 0xbd4, [MIB_BSCR17] = 0xbfc, [MIB_TRDR1] = 0xc14, + [MIB_TSCR0] = 0x750, + [MIB_TSCR1] = 0x754, + [MIB_TSCR2] = 0x758, + [MIB_TSCR3] = 0x75c, + [MIB_TSCR4] = 0x760, + [MIB_TSCR5] = 0x764, + [MIB_TSCR6] = 0x768, + [MIB_TSCR7] = 0x770, [HIF_REMAP_L1] = 0x8, [HIF_REMAP_BASE_L1] = 0x40000, [HIF_REMAP_L2] = 0x1b8, @@ -334,7 +360,7 @@ static u32 __mt7996_reg_addr(struct mt7996_dev *dev, u32 addr) return dev->reg.map[i].mapped + ofs; } - return 0; + return INVALID_REG_ADDR; } static u32 __mt7996_reg_remap_addr(struct mt7996_dev *dev, u32 addr) @@ -366,7 +392,7 @@ void mt7996_memcpy_fromio(struct mt7996_dev *dev, void *buf, u32 offset, { u32 addr = __mt7996_reg_addr(dev, offset); - if (addr) { + if (addr != INVALID_REG_ADDR) { memcpy_fromio(buf, dev->mt76.mmio.regs + addr, len); return; } @@ -382,7 +408,7 @@ static u32 mt7996_rr(struct mt76_dev *mdev, u32 offset) struct mt7996_dev *dev = container_of(mdev, struct mt7996_dev, mt76); u32 addr = __mt7996_reg_addr(dev, offset), val; - if (addr) + if (addr != INVALID_REG_ADDR) return dev->bus_ops->rr(mdev, addr); spin_lock_bh(&dev->reg_lock); @@ -397,7 +423,7 @@ static void mt7996_wr(struct mt76_dev *mdev, u32 offset, u32 val) struct mt7996_dev *dev = container_of(mdev, struct mt7996_dev, mt76); u32 addr = __mt7996_reg_addr(dev, offset); - if (addr) { + if (addr != INVALID_REG_ADDR) { dev->bus_ops->wr(mdev, addr, val); return; } @@ -412,7 +438,7 @@ static u32 mt7996_rmw(struct mt76_dev *mdev, u32 offset, u32 mask, u32 val) struct mt7996_dev *dev = container_of(mdev, struct mt7996_dev, mt76); u32 addr = __mt7996_reg_addr(dev, offset); - if (addr) + if (addr != INVALID_REG_ADDR) return dev->bus_ops->rmw(mdev, addr, mask, val); spin_lock_bh(&dev->reg_lock); @@ -464,6 +490,9 @@ int mt7996_mmio_wed_init(struct mt7996_dev *dev, void *pdev_ptr, if (!wed_enable) return 0; + if (hif2 && !mtk_wed_device_active(&dev->mt76.mmio.wed)) + return 0; + dev->mt76.hwrro_mode = is_mt7996(&dev->mt76) ? MT76_HWRRO_V3 : MT76_HWRRO_V3_1; diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h b/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h index b8ffa42c5a1d..e371b2f174e5 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mt7996.h @@ -618,6 +618,25 @@ mt7996_vif_conf_link(struct mt7996_dev *dev, struct ieee80211_vif *vif, link_conf); } +static inline struct mt7996_sta_link * +mt7996_sta_link(struct mt7996_sta *msta, u8 link_id) +{ + if (link_id >= IEEE80211_MLD_MAX_NUM_LINKS) + return NULL; + + return rcu_dereference(msta->link[link_id]); +} + +static inline struct mt7996_sta_link * +mt7996_sta_link_protected(struct mt7996_dev *dev, struct mt7996_sta *msta, + u8 link_id) +{ + if (link_id >= IEEE80211_MLD_MAX_NUM_LINKS) + return NULL; + + return mt76_dereference(msta->link[link_id], &dev->mt76); +} + #define mt7996_for_each_phy(dev, phy) \ for (int __i = 0; __i < ARRAY_SIZE((dev)->radio_phy); __i++) \ if (((phy) = (dev)->radio_phy[__i]) != NULL) @@ -670,6 +689,8 @@ int mt7996_mcu_add_bss_info(struct mt7996_phy *phy, struct ieee80211_vif *vif, struct ieee80211_bss_conf *link_conf, struct mt76_vif_link *mlink, struct mt7996_sta_link *msta_link, int enable); +int mt7996_mcu_update_bss_rfch(struct mt7996_phy *phy, + struct mt7996_vif_link *link); int mt7996_mcu_add_sta(struct mt7996_dev *dev, struct ieee80211_bss_conf *link_conf, struct ieee80211_link_sta *link_sta, @@ -679,6 +700,7 @@ int mt7996_mcu_add_sta(struct mt7996_dev *dev, int mt7996_mcu_teardown_mld_sta(struct mt7996_dev *dev, struct mt7996_vif_link *link, struct mt7996_sta_link *msta_link); +void mt7996_mcu_update_sta_rec_bw(void *data, struct ieee80211_sta *sta); int mt7996_mcu_add_tx_ba(struct mt7996_dev *dev, struct ieee80211_ampdu_params *params, struct ieee80211_vif *vif, bool enable); @@ -810,6 +832,7 @@ bool mt7996_mac_wtbl_update(struct mt7996_dev *dev, int idx, u32 mask); void mt7996_mac_reset_counters(struct mt7996_phy *phy); void mt7996_mac_cca_stats_reset(struct mt7996_phy *phy); void mt7996_mac_enable_nf(struct mt7996_dev *dev, u8 band); +struct mt76_wcid *mt7996_get_tx_wcid(struct mt76_wcid *wcid); void mt7996_mac_write_txwi(struct mt7996_dev *dev, __le32 *txwi, struct sk_buff *skb, struct mt76_wcid *wcid, struct ieee80211_key_conf *key, int pid, @@ -867,6 +890,9 @@ int mt7996_mcu_cp_support(struct mt7996_dev *dev, u8 mode); #ifdef CONFIG_MAC80211_DEBUGFS void mt7996_sta_add_debugfs(struct ieee80211_hw *hw, struct ieee80211_vif *vif, struct ieee80211_sta *sta, struct dentry *dir); +void mt7996_link_sta_add_debugfs(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + struct ieee80211_link_sta *link_sta, + struct dentry *dir); #endif int mt7996_mmio_wed_init(struct mt7996_dev *dev, void *pdev_ptr, bool hif2, int *irq); diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/regs.h b/drivers/net/wireless/mediatek/mt76/mt7996/regs.h index 0fa325f87fcd..a46c790550c6 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/regs.h +++ b/drivers/net/wireless/mediatek/mt76/mt7996/regs.h @@ -64,6 +64,14 @@ enum offs_rev { MIB_BSCR7, MIB_BSCR17, MIB_TRDR1, + MIB_TSCR0, + MIB_TSCR1, + MIB_TSCR2, + MIB_TSCR3, + MIB_TSCR4, + MIB_TSCR5, + MIB_TSCR6, + MIB_TSCR7, HIF_REMAP_L1, HIF_REMAP_BASE_L1, HIF_REMAP_L2, @@ -247,9 +255,9 @@ enum offs_rev { #define MT_MIB_BSCR7(_band) MT_WF_MIB(_band, __OFFS(MIB_BSCR7)) #define MT_MIB_BSCR17(_band) MT_WF_MIB(_band, __OFFS(MIB_BSCR17)) -#define MT_MIB_TSCR5(_band) MT_WF_MIB(_band, 0x6c4) -#define MT_MIB_TSCR6(_band) MT_WF_MIB(_band, 0x6c8) -#define MT_MIB_TSCR7(_band) MT_WF_MIB(_band, 0x6d0) +#define MT_MIB_TSCR5(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR5)) +#define MT_MIB_TSCR6(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR6)) +#define MT_MIB_TSCR7(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR7)) #define MT_MIB_RSCR1(_band) MT_WF_MIB(_band, __OFFS(MIB_RSCR1)) /* rx mpdu counter, full 32 bits */ @@ -265,14 +273,14 @@ enum offs_rev { #define MT_MIB_RSCR36(_band) MT_WF_MIB(_band, __OFFS(MIB_RSCR36)) /* tx ampdu cnt, full 32 bits */ -#define MT_MIB_TSCR0(_band) MT_WF_MIB(_band, 0x6b0) -#define MT_MIB_TSCR2(_band) MT_WF_MIB(_band, 0x6b8) +#define MT_MIB_TSCR0(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR0)) +#define MT_MIB_TSCR2(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR2)) /* counts all mpdus in ampdu, regardless of success */ -#define MT_MIB_TSCR3(_band) MT_WF_MIB(_band, 0x6bc) +#define MT_MIB_TSCR3(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR3)) /* counts all successfully tx'd mpdus in ampdu */ -#define MT_MIB_TSCR4(_band) MT_WF_MIB(_band, 0x6c0) +#define MT_MIB_TSCR4(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR4)) /* rx ampdu count, 32-bit */ #define MT_MIB_RSCR27(_band) MT_WF_MIB(_band, __OFFS(MIB_RSCR27)) @@ -296,7 +304,7 @@ enum offs_rev { #define MT_MIB_RVSR1(_band) MT_WF_MIB(_band, __OFFS(MIB_RVSR1)) /* rx blockack count, 32 bits */ -#define MT_MIB_TSCR1(_band) MT_WF_MIB(_band, 0x6b4) +#define MT_MIB_TSCR1(_band) MT_WF_MIB(_band, __OFFS(MIB_TSCR1)) #define MT_MIB_BTSCR0(_band) MT_WF_MIB(_band, 0x5e0) #define MT_MIB_BTSCR5(_band) MT_WF_MIB(_band, __OFFS(MIB_BTSCR5)) diff --git a/drivers/net/wireless/mediatek/mt76/npu.c b/drivers/net/wireless/mediatek/mt76/npu.c new file mode 100644 index 000000000000..9679237f7398 --- /dev/null +++ b/drivers/net/wireless/mediatek/mt76/npu.c @@ -0,0 +1,502 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2025 AIROHA Inc + * Author: Lorenzo Bianconi + */ +#include +#include +#include + +#include "mt76.h" +#include "dma.h" +#include "mt76_connac.h" + +#define MT76_NPU_RX_BUF_SIZE (1800 + \ + SKB_DATA_ALIGN(sizeof(struct skb_shared_info))) + +int mt76_npu_fill_rx_queue(struct mt76_dev *dev, struct mt76_queue *q) +{ + int nframes = 0; + + while (q->queued < q->ndesc - 1) { + struct airoha_npu_rx_dma_desc *desc = (void *)q->desc; + struct mt76_queue_entry *e = &q->entry[q->head]; + struct page *page; + int offset; + + e->buf = mt76_get_page_pool_buf(q, &offset, q->buf_size); + if (!e->buf) + break; + + e->dma_len[0] = SKB_WITH_OVERHEAD(q->buf_size); + page = virt_to_head_page(e->buf); + e->dma_addr[0] = page_pool_get_dma_addr(page) + offset; + + memset(&desc[q->head], 0, sizeof(*desc)); + desc[q->head].addr = e->dma_addr[0]; + + q->head = (q->head + 1) % q->ndesc; + q->queued++; + nframes++; + } + + return nframes; +} + +void mt76_npu_queue_cleanup(struct mt76_dev *dev, struct mt76_queue *q) +{ + spin_lock_bh(&q->lock); + while (q->queued > 0) { + struct mt76_queue_entry *e = &q->entry[q->tail]; + + dma_sync_single_for_cpu(dev->dma_dev, e->dma_addr[0], + e->dma_len[0], + page_pool_get_dma_dir(q->page_pool)); + mt76_put_page_pool_buf(e->buf, false); + q->tail = (q->tail + 1) % q->ndesc; + q->queued--; + } + spin_unlock_bh(&q->lock); +} + +static struct sk_buff *mt76_npu_dequeue(struct mt76_dev *dev, + struct mt76_queue *q, + u32 *info) +{ + struct airoha_npu_rx_dma_desc *desc = (void *)q->desc; + int i, nframes, index = q->tail; + struct sk_buff *skb = NULL; + + nframes = FIELD_GET(NPU_RX_DMA_PKT_COUNT_MASK, desc[index].info); + nframes = max_t(int, nframes, 1); + + for (i = 0; i < nframes; i++) { + struct mt76_queue_entry *e = &q->entry[index]; + int len = FIELD_GET(NPU_RX_DMA_DESC_CUR_LEN_MASK, + desc[index].ctrl); + + if (!FIELD_GET(NPU_RX_DMA_DESC_DONE_MASK, desc[index].ctrl)) { + dev_kfree_skb(skb); + return NULL; + } + + dma_sync_single_for_cpu(dev->dma_dev, e->dma_addr[0], + e->dma_len[0], + page_pool_get_dma_dir(q->page_pool)); + + if (!skb) { + skb = napi_build_skb(e->buf, q->buf_size); + if (!skb) + return NULL; + + __skb_put(skb, len); + skb_reset_mac_header(skb); + skb_mark_for_recycle(skb); + } else { + struct skb_shared_info *shinfo = skb_shinfo(skb); + struct page *page = virt_to_head_page(e->buf); + int nr_frags = shinfo->nr_frags; + + if (nr_frags < ARRAY_SIZE(shinfo->frags)) + skb_add_rx_frag(skb, nr_frags, page, + e->buf - page_address(page), + len, q->buf_size); + } + + *info = desc[index].info; + index = (index + 1) % q->ndesc; + } + q->tail = index; + q->queued -= i; + Q_WRITE(q, dma_idx, q->tail); + + return skb; +} + +void mt76_npu_check_ppe(struct mt76_dev *dev, struct sk_buff *skb, + u32 info) +{ + struct airoha_ppe_dev *ppe_dev; + u16 reason, hash; + + if (!mt76_npu_device_active(dev)) + return; + + rcu_read_lock(); + + ppe_dev = rcu_dereference(dev->mmio.ppe_dev); + if (!ppe_dev) + goto out; + + hash = FIELD_GET(NPU_RX_DMA_FOE_ID_MASK, info); + skb_set_hash(skb, hash, PKT_HASH_TYPE_L4); + + reason = FIELD_GET(NPU_RX_DMA_CRSN_MASK, info); + if (reason == PPE_CPU_REASON_HIT_UNBIND_RATE_REACHED) { + skb_set_mac_header(skb, 0); + airoha_ppe_dev_check_skb(ppe_dev, skb, hash, true); + } +out: + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(mt76_npu_check_ppe); + +static int mt76_npu_rx_poll(struct napi_struct *napi, int budget) +{ + struct mt76_dev *dev = mt76_priv(napi->dev); + enum mt76_rxq_id qid = napi - dev->napi; + struct airoha_npu *npu; + int done = 0; + + rcu_read_lock(); + + npu = rcu_dereference(dev->mmio.npu); + if (!npu) + goto out; + + while (done < budget) { + struct sk_buff *skb; + u32 info = 0; + + skb = mt76_npu_dequeue(dev, &dev->q_rx[qid], &info); + if (!skb) + break; + + dev->drv->rx_skb(dev, qid, skb, &info); + mt76_rx_poll_complete(dev, qid, napi); + done++; + } + + mt76_npu_fill_rx_queue(dev, &dev->q_rx[qid]); +out: + if (done < budget && napi_complete(napi)) + dev->drv->rx_poll_complete(dev, qid); + + rcu_read_unlock(); + + return done; +} + +static irqreturn_t mt76_npu_irq_handler(int irq, void *q_instance) +{ + struct mt76_queue *q = q_instance; + struct mt76_dev *dev = q->dev; + int qid = q - &dev->q_rx[0]; + int index = qid - MT_RXQ_NPU0; + struct airoha_npu *npu; + u32 status; + + rcu_read_lock(); + + npu = rcu_dereference(dev->mmio.npu); + if (!npu) + goto out; + + status = airoha_npu_wlan_get_irq_status(npu, index); + airoha_npu_wlan_set_irq_status(npu, status); + + airoha_npu_wlan_disable_irq(npu, index); + napi_schedule(&dev->napi[qid]); +out: + rcu_read_unlock(); + + return IRQ_HANDLED; +} + +int mt76_npu_dma_add_buf(struct mt76_phy *phy, struct mt76_queue *q, + struct sk_buff *skb, struct mt76_queue_buf *buf, + void *txwi_ptr) +{ + u16 txwi_len = min_t(u16, phy->dev->drv->txwi_size, NPU_TXWI_LEN); + struct airoha_npu_tx_dma_desc *desc = (void *)q->desc; + int ret; + + /* TODO: Take into account unlinear skbs */ + memcpy(desc[q->head].txwi, txwi_ptr, txwi_len); + desc[q->head].addr = buf->addr; + desc[q->head].ctrl = FIELD_PREP(NPU_TX_DMA_DESC_VEND_LEN_MASK, txwi_len) | + FIELD_PREP(NPU_TX_DMA_DESC_LEN_MASK, skb->len) | + NPU_TX_DMA_DESC_DONE_MASK; + + ret = q->head; + q->entry[q->head].skip_buf0 = true; + q->entry[q->head].skip_buf1 = true; + q->entry[q->head].txwi = NULL; + q->entry[q->head].skb = NULL; + q->entry[q->head].wcid = 0xffff; + + q->head = (q->head + 1) % q->ndesc; + q->queued++; + + return ret; +} + +void mt76_npu_txdesc_cleanup(struct mt76_queue *q, int index) +{ + struct airoha_npu_tx_dma_desc *desc = (void *)q->desc; + + if (!mt76_queue_is_npu_tx(q)) + return; + + desc[index].ctrl &= ~NPU_TX_DMA_DESC_DONE_MASK; +} + +void mt76_npu_queue_setup(struct mt76_dev *dev, struct mt76_queue *q) +{ + int qid = FIELD_GET(MT_QFLAG_WED_RING, q->flags); + bool xmit = mt76_queue_is_npu_tx(q); + struct airoha_npu *npu; + + if (!mt76_queue_is_npu(q)) + return; + + npu = rcu_dereference_protected(dev->mmio.npu, &dev->mutex); + if (npu) + q->wed_regs = airoha_npu_wlan_get_queue_addr(npu, qid, xmit); +} + +int mt76_npu_rx_queue_init(struct mt76_dev *dev, struct mt76_queue *q) +{ + int err, irq, qid = q - &dev->q_rx[0]; + int size, index = qid - MT_RXQ_NPU0; + struct airoha_npu *npu; + const char *name; + + mutex_lock(&dev->mutex); + + npu = rcu_dereference_protected(dev->mmio.npu, &dev->mutex); + irq = npu && index < ARRAY_SIZE(npu->irqs) ? npu->irqs[index] + : -EINVAL; + if (irq < 0) { + err = irq; + goto out; + } + + q->flags = MT_NPU_Q_RX(index); + size = qid == MT_RXQ_NPU1 ? NPU_RX1_DESC_NUM : NPU_RX0_DESC_NUM; + err = dev->queue_ops->alloc(dev, q, 0, size, + MT76_NPU_RX_BUF_SIZE, 0); + if (err) + goto out; + + name = devm_kasprintf(dev->dev, GFP_KERNEL, "mt76-npu.%d", index); + if (!name) { + err = -ENOMEM; + goto out; + } + + err = devm_request_irq(dev->dev, irq, mt76_npu_irq_handler, + IRQF_SHARED, name, q); + if (err) + goto out; + + netif_napi_add(dev->napi_dev, &dev->napi[qid], mt76_npu_rx_poll); + mt76_npu_fill_rx_queue(dev, q); + napi_enable(&dev->napi[qid]); +out: + mutex_unlock(&dev->mutex); + + return err; +} +EXPORT_SYMBOL_GPL(mt76_npu_rx_queue_init); + +static int mt76_npu_setup_tc_block_cb(enum tc_setup_type type, + void *type_data, void *cb_priv) +{ + struct mt76_phy *phy = cb_priv; + struct mt76_dev *dev = phy->dev; + struct airoha_ppe_dev *ppe_dev; + int err = -EOPNOTSUPP; + + if (type != TC_SETUP_CLSFLOWER) + return -EOPNOTSUPP; + + mutex_lock(&dev->mutex); + + ppe_dev = rcu_dereference_protected(dev->mmio.ppe_dev, &dev->mutex); + if (ppe_dev) + err = airoha_ppe_dev_setup_tc_block_cb(ppe_dev, type_data); + + mutex_unlock(&dev->mutex); + + return err; +} + +static int mt76_npu_setup_tc_block(struct mt76_phy *phy, + struct net_device *dev, + struct flow_block_offload *f) +{ + flow_setup_cb_t *cb = mt76_npu_setup_tc_block_cb; + static LIST_HEAD(block_cb_list); + struct flow_block_cb *block_cb; + + if (f->binder_type != FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS) + return -EOPNOTSUPP; + + if (!tc_can_offload(dev)) + return -EOPNOTSUPP; + + f->driver_block_list = &block_cb_list; + switch (f->command) { + case FLOW_BLOCK_BIND: + block_cb = flow_block_cb_lookup(f->block, cb, dev); + if (block_cb) { + flow_block_cb_incref(block_cb); + return 0; + } + + block_cb = flow_block_cb_alloc(cb, dev, phy, NULL); + if (IS_ERR(block_cb)) + return PTR_ERR(block_cb); + + flow_block_cb_incref(block_cb); + flow_block_cb_add(block_cb, f); + list_add_tail(&block_cb->driver_list, &block_cb_list); + return 0; + case FLOW_BLOCK_UNBIND: + block_cb = flow_block_cb_lookup(f->block, cb, dev); + if (!block_cb) + return -ENOENT; + + if (!flow_block_cb_decref(block_cb)) { + flow_block_cb_remove(block_cb, f); + list_del(&block_cb->driver_list); + } + return 0; + default: + return -EOPNOTSUPP; + } +} + +int mt76_npu_net_setup_tc(struct ieee80211_hw *hw, struct ieee80211_vif *vif, + struct net_device *dev, enum tc_setup_type type, + void *type_data) +{ + struct mt76_phy *phy = hw->priv; + + if (!tc_can_offload(dev)) + return -EOPNOTSUPP; + + if (!mt76_npu_device_active(phy->dev)) + return -EOPNOTSUPP; + + switch (type) { + case TC_SETUP_BLOCK: + case TC_SETUP_FT: + return mt76_npu_setup_tc_block(phy, dev, type_data); + default: + return -EOPNOTSUPP; + } +} +EXPORT_SYMBOL_GPL(mt76_npu_net_setup_tc); + +void mt76_npu_disable_irqs(struct mt76_dev *dev) +{ + struct airoha_npu *npu; + int i; + + rcu_read_lock(); + + npu = rcu_dereference(dev->mmio.npu); + if (!npu) + goto unlock; + + for (i = MT_RXQ_NPU0; i <= MT_RXQ_NPU1; i++) { + int qid = i - MT_RXQ_NPU0; + u32 status; + + status = airoha_npu_wlan_get_irq_status(npu, qid); + airoha_npu_wlan_set_irq_status(npu, status); + airoha_npu_wlan_disable_irq(npu, qid); + } +unlock: + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(mt76_npu_disable_irqs); + +int mt76_npu_init(struct mt76_dev *dev, phys_addr_t phy_addr, int type) +{ + struct airoha_ppe_dev *ppe_dev; + struct airoha_npu *npu; + int err = 0; + + /* NPU offloading is only supported by MT7992 */ + if (!is_mt7992(dev)) + return 0; + + mutex_lock(&dev->mutex); + + npu = airoha_npu_get(dev->dev); + if (IS_ERR(npu)) { + request_module("airoha-npu"); + npu = airoha_npu_get(dev->dev); + } + + if (IS_ERR(npu)) { + err = PTR_ERR(npu); + goto error_unlock; + } + + ppe_dev = airoha_ppe_get_dev(dev->dev); + if (IS_ERR(ppe_dev)) { + request_module("airoha-eth"); + ppe_dev = airoha_ppe_get_dev(dev->dev); + } + + if (IS_ERR(ppe_dev)) { + err = PTR_ERR(ppe_dev); + goto error_npu_put; + } + + err = airoha_npu_wlan_init_reserved_memory(npu); + if (err) + goto error_ppe_put; + + dev->dma_dev = npu->dev; + dev->mmio.phy_addr = phy_addr; + dev->mmio.npu_type = type; + /* NPU offloading requires HW-RRO for RX packet reordering. */ + dev->hwrro_mode = MT76_HWRRO_V3_1; + dev->rx_token_size = 32768; + + rcu_assign_pointer(dev->mmio.npu, npu); + rcu_assign_pointer(dev->mmio.ppe_dev, ppe_dev); + synchronize_rcu(); + + mutex_unlock(&dev->mutex); + + return 0; + +error_ppe_put: + airoha_ppe_put_dev(ppe_dev); +error_npu_put: + airoha_npu_put(npu); +error_unlock: + mutex_unlock(&dev->mutex); + + return err; +} +EXPORT_SYMBOL_GPL(mt76_npu_init); + +void mt76_npu_deinit(struct mt76_dev *dev) +{ + struct airoha_ppe_dev *ppe_dev; + struct airoha_npu *npu; + + mutex_lock(&dev->mutex); + + npu = rcu_replace_pointer(dev->mmio.npu, NULL, + lockdep_is_held(&dev->mutex)); + if (npu) + airoha_npu_put(npu); + + ppe_dev = rcu_replace_pointer(dev->mmio.ppe_dev, NULL, + lockdep_is_held(&dev->mutex)); + if (ppe_dev) + airoha_ppe_put_dev(ppe_dev); + + mutex_unlock(&dev->mutex); + + mt76_npu_queue_cleanup(dev, &dev->q_rx[MT_RXQ_NPU0]); + mt76_npu_queue_cleanup(dev, &dev->q_rx[MT_RXQ_NPU1]); +} diff --git a/drivers/net/wireless/mediatek/mt76/scan.c b/drivers/net/wireless/mediatek/mt76/scan.c index 6cfca5108bc7..2faa84c516e7 100644 --- a/drivers/net/wireless/mediatek/mt76/scan.c +++ b/drivers/net/wireless/mediatek/mt76/scan.c @@ -46,6 +46,7 @@ mt76_scan_send_probe(struct mt76_dev *dev, struct cfg80211_ssid *ssid) struct mt76_phy *phy = dev->scan.phy; struct ieee80211_tx_info *info; struct sk_buff *skb; + u8 link_id; skb = ieee80211_probereq_get(phy->hw, vif->addr, ssid->ssid, ssid->ssid_len, req->ie_len); @@ -75,6 +76,10 @@ mt76_scan_send_probe(struct mt76_dev *dev, struct cfg80211_ssid *ssid) info->flags |= IEEE80211_TX_CTL_NO_CCK_RATE; info->control.flags |= IEEE80211_TX_CTRL_DONT_USE_RATE_MASK; + link_id = mvif->wcid ? mvif->wcid->link_id : IEEE80211_LINK_UNSPECIFIED; + info->control.flags &= ~IEEE80211_TX_CTRL_MLO_LINK; + info->control.flags |= u32_encode_bits(link_id, IEEE80211_TX_CTRL_MLO_LINK); + mt76_tx(phy, NULL, mvif->wcid, skb); out: diff --git a/drivers/net/wireless/mediatek/mt76/sdio.c b/drivers/net/wireless/mediatek/mt76/sdio.c index 8e9576747052..e4c3dcc34f68 100644 --- a/drivers/net/wireless/mediatek/mt76/sdio.c +++ b/drivers/net/wireless/mediatek/mt76/sdio.c @@ -519,6 +519,10 @@ mt76s_tx_queue_skb(struct mt76_phy *phy, struct mt76_queue *q, enum mt76_txq_id qid, struct sk_buff *skb, struct mt76_wcid *wcid, struct ieee80211_sta *sta) { + struct ieee80211_tx_status status = { + .sta = sta, + }; + struct mt76_tx_info tx_info = { .skb = skb, }; @@ -531,8 +535,13 @@ mt76s_tx_queue_skb(struct mt76_phy *phy, struct mt76_queue *q, skb->prev = skb->next = NULL; err = dev->drv->tx_prepare_skb(dev, NULL, qid, wcid, sta, &tx_info); - if (err < 0) + if (err < 0) { + status.skb = tx_info.skb; + spin_lock_bh(&dev->rx_lock); + ieee80211_tx_status_ext(dev->hw, &status); + spin_unlock_bh(&dev->rx_lock); return err; + } q->entry[q->head].skb = tx_info.skb; q->entry[q->head].buf_sz = len; diff --git a/drivers/net/wireless/mediatek/mt76/tx.c b/drivers/net/wireless/mediatek/mt76/tx.c index 9fb0cca5524a..28630d75bdb2 100644 --- a/drivers/net/wireless/mediatek/mt76/tx.c +++ b/drivers/net/wireless/mediatek/mt76/tx.c @@ -311,10 +311,6 @@ __mt76_tx_queue_skb(struct mt76_phy *phy, int qid, struct sk_buff *skb, if (idx < 0 || !sta) return idx; - wcid = (struct mt76_wcid *)sta->drv_priv; - if (!wcid->sta) - return idx; - q->entry[idx].wcid = wcid->idx; if (!non_aql) @@ -619,6 +615,7 @@ mt76_txq_schedule_pending_wcid(struct mt76_phy *phy, struct mt76_wcid *wcid, !ieee80211_is_data(hdr->frame_control) && (!ieee80211_is_bufferable_mmpdu(skb) || ieee80211_is_deauth(hdr->frame_control) || + ieee80211_is_disassoc(hdr->frame_control) || head == &wcid->tx_offchannel)) qid = MT_TXQ_PSD; @@ -670,8 +667,8 @@ static void mt76_txq_schedule_pending(struct mt76_phy *phy) ret = mt76_txq_schedule_pending_wcid(phy, wcid, &wcid->tx_pending); spin_lock(&phy->tx_lock); - if (!skb_queue_empty(&wcid->tx_pending) && - !skb_queue_empty(&wcid->tx_offchannel) && + if ((!skb_queue_empty(&wcid->tx_pending) || + !skb_queue_empty(&wcid->tx_offchannel)) && list_empty(&wcid->tx_list)) list_add_tail(&wcid->tx_list, &phy->tx_list); } diff --git a/drivers/net/wireless/realtek/rtlwifi/pci.c b/drivers/net/wireless/realtek/rtlwifi/pci.c index f0010336e78c..8fb9604c3c53 100644 --- a/drivers/net/wireless/realtek/rtlwifi/pci.c +++ b/drivers/net/wireless/realtek/rtlwifi/pci.c @@ -2226,13 +2226,17 @@ int rtl_pci_probe(struct pci_dev *pdev, rtl_dbg(rtlpriv, COMP_INIT, DBG_DMESG, "%s: failed to register IRQ handler\n", wiphy_name(hw->wiphy)); - goto fail3; + goto fail6; } rtlpci->irq_alloc = 1; set_bit(RTL_STATUS_INTERFACE_START, &rtlpriv->status); return 0; +fail6: + rtl_deinit_rfkill(hw); + rtl_debug_remove_one(hw); + ieee80211_unregister_hw(hw); fail5: rtl_pci_deinit(hw); fail4: diff --git a/drivers/net/wireless/realtek/rtw89/cam.c b/drivers/net/wireless/realtek/rtw89/cam.c index 385a238fe5cc..93a029439635 100644 --- a/drivers/net/wireless/realtek/rtw89/cam.c +++ b/drivers/net/wireless/realtek/rtw89/cam.c @@ -236,7 +236,8 @@ static int __rtw89_cam_detach_sec_cam(struct rtw89_dev *rtwdev, if (ret) rtw89_err(rtwdev, "failed to update dctl cam del key: %d\n", ret); - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL, + RTW89_ROLE_INFO_CHANGE); if (ret) rtw89_err(rtwdev, "failed to update cam del key: %d\n", ret); } @@ -276,7 +277,8 @@ static int __rtw89_cam_attach_sec_cam(struct rtw89_dev *rtwdev, ret); return ret; } - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL, + RTW89_ROLE_INFO_CHANGE); if (ret) { rtw89_err(rtwdev, "failed to update addr cam sec entry: %d\n", ret); @@ -760,7 +762,8 @@ int rtw89_cam_init(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link) int rtw89_cam_fill_bssid_cam_info(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link, - struct rtw89_sta_link *rtwsta_link, u8 *cmd) + struct rtw89_sta_link *rtwsta_link, + struct rtw89_h2c_addr_cam_v0 *h2c) { struct rtw89_bssid_cam_entry *bssid_cam = rtw89_get_bssid_cam_of(rtwvif_link, rtwsta_link); @@ -780,20 +783,19 @@ int rtw89_cam_fill_bssid_cam_info(struct rtw89_dev *rtwdev, rcu_read_unlock(); - FWCMD_SET_ADDR_BSSID_IDX(cmd, bssid_cam->bssid_cam_idx); - FWCMD_SET_ADDR_BSSID_OFFSET(cmd, bssid_cam->offset); - FWCMD_SET_ADDR_BSSID_LEN(cmd, bssid_cam->len); - FWCMD_SET_ADDR_BSSID_VALID(cmd, bssid_cam->valid); - FWCMD_SET_ADDR_BSSID_MASK(cmd, bss_mask); - FWCMD_SET_ADDR_BSSID_BB_SEL(cmd, bssid_cam->phy_idx); - FWCMD_SET_ADDR_BSSID_BSS_COLOR(cmd, bss_color); - - FWCMD_SET_ADDR_BSSID_BSSID0(cmd, bssid_cam->bssid[0]); - FWCMD_SET_ADDR_BSSID_BSSID1(cmd, bssid_cam->bssid[1]); - FWCMD_SET_ADDR_BSSID_BSSID2(cmd, bssid_cam->bssid[2]); - FWCMD_SET_ADDR_BSSID_BSSID3(cmd, bssid_cam->bssid[3]); - FWCMD_SET_ADDR_BSSID_BSSID4(cmd, bssid_cam->bssid[4]); - FWCMD_SET_ADDR_BSSID_BSSID5(cmd, bssid_cam->bssid[5]); + h2c->w12 = le32_encode_bits(bssid_cam->bssid_cam_idx, ADDR_CAM_W12_BSSID_IDX) | + le32_encode_bits(bssid_cam->offset, ADDR_CAM_W12_BSSID_OFFSET) | + le32_encode_bits(bssid_cam->len, ADDR_CAM_W12_BSSID_LEN); + h2c->w13 = le32_encode_bits(bssid_cam->valid, ADDR_CAM_W13_BSSID_VALID) | + le32_encode_bits(bss_mask, ADDR_CAM_W13_BSSID_MASK) | + le32_encode_bits(bssid_cam->phy_idx, ADDR_CAM_W13_BSSID_BB_SEL) | + le32_encode_bits(bss_color, ADDR_CAM_W13_BSSID_BSS_COLOR) | + le32_encode_bits(bssid_cam->bssid[0], ADDR_CAM_W13_BSSID_BSSID0) | + le32_encode_bits(bssid_cam->bssid[1], ADDR_CAM_W13_BSSID_BSSID1); + h2c->w14 = le32_encode_bits(bssid_cam->bssid[2], ADDR_CAM_W14_BSSID_BSSID2) | + le32_encode_bits(bssid_cam->bssid[3], ADDR_CAM_W14_BSSID_BSSID3) | + le32_encode_bits(bssid_cam->bssid[4], ADDR_CAM_W14_BSSID_BSSID4) | + le32_encode_bits(bssid_cam->bssid[5], ADDR_CAM_W14_BSSID_BSSID5); return 0; } @@ -813,7 +815,7 @@ void rtw89_cam_fill_addr_cam_info(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link, struct rtw89_sta_link *rtwsta_link, const u8 *scan_mac_addr, - u8 *cmd) + struct rtw89_h2c_addr_cam_v0 *h2c) { struct ieee80211_vif *vif = rtwvif_link_to_vif(rtwvif_link); struct rtw89_addr_cam_entry *addr_cam = @@ -845,69 +847,65 @@ void rtw89_cam_fill_addr_cam_info(struct rtw89_dev *rtwdev, sma_hash = rtw89_cam_addr_hash(sma_start, sma); tma_hash = rtw89_cam_addr_hash(tma_start, tma); - FWCMD_SET_ADDR_IDX(cmd, addr_cam->addr_cam_idx); - FWCMD_SET_ADDR_OFFSET(cmd, addr_cam->offset); - FWCMD_SET_ADDR_LEN(cmd, addr_cam->len); - - FWCMD_SET_ADDR_VALID(cmd, addr_cam->valid); - FWCMD_SET_ADDR_NET_TYPE(cmd, rtwvif_link->net_type); - FWCMD_SET_ADDR_BCN_HIT_COND(cmd, rtwvif_link->bcn_hit_cond); - FWCMD_SET_ADDR_HIT_RULE(cmd, rtwvif_link->hit_rule); - FWCMD_SET_ADDR_BB_SEL(cmd, rtwvif_link->phy_idx); - FWCMD_SET_ADDR_ADDR_MASK(cmd, addr_cam->addr_mask); - FWCMD_SET_ADDR_MASK_SEL(cmd, addr_cam->mask_sel); - FWCMD_SET_ADDR_SMA_HASH(cmd, sma_hash); - FWCMD_SET_ADDR_TMA_HASH(cmd, tma_hash); - - FWCMD_SET_ADDR_BSSID_CAM_IDX(cmd, addr_cam->bssid_cam_idx); - - FWCMD_SET_ADDR_SMA0(cmd, sma[0]); - FWCMD_SET_ADDR_SMA1(cmd, sma[1]); - FWCMD_SET_ADDR_SMA2(cmd, sma[2]); - FWCMD_SET_ADDR_SMA3(cmd, sma[3]); - FWCMD_SET_ADDR_SMA4(cmd, sma[4]); - FWCMD_SET_ADDR_SMA5(cmd, sma[5]); - - FWCMD_SET_ADDR_TMA0(cmd, tma[0]); - FWCMD_SET_ADDR_TMA1(cmd, tma[1]); - FWCMD_SET_ADDR_TMA2(cmd, tma[2]); - FWCMD_SET_ADDR_TMA3(cmd, tma[3]); - FWCMD_SET_ADDR_TMA4(cmd, tma[4]); - FWCMD_SET_ADDR_TMA5(cmd, tma[5]); - - FWCMD_SET_ADDR_PORT_INT(cmd, rtwvif_link->port); - FWCMD_SET_ADDR_TSF_SYNC(cmd, rtwvif_link->port); - FWCMD_SET_ADDR_TF_TRS(cmd, rtwvif_link->trigger); - FWCMD_SET_ADDR_LSIG_TXOP(cmd, rtwvif_link->lsig_txop); - FWCMD_SET_ADDR_TGT_IND(cmd, rtwvif_link->tgt_ind); - FWCMD_SET_ADDR_FRM_TGT_IND(cmd, rtwvif_link->frm_tgt_ind); - FWCMD_SET_ADDR_MACID(cmd, rtwsta_link ? rtwsta_link->mac_id : - rtwvif_link->mac_id); + h2c->w1 = le32_encode_bits(addr_cam->addr_cam_idx, ADDR_CAM_W1_IDX) | + le32_encode_bits(addr_cam->offset, ADDR_CAM_W1_OFFSET) | + le32_encode_bits(addr_cam->len, ADDR_CAM_W1_LEN); + h2c->w2 = le32_encode_bits(addr_cam->valid, ADDR_CAM_W2_VALID) | + le32_encode_bits(rtwvif_link->net_type, ADDR_CAM_W2_NET_TYPE) | + le32_encode_bits(rtwvif_link->bcn_hit_cond, ADDR_CAM_W2_BCN_HIT_COND) | + le32_encode_bits(rtwvif_link->hit_rule, ADDR_CAM_W2_HIT_RULE) | + le32_encode_bits(rtwvif_link->phy_idx, ADDR_CAM_W2_BB_SEL) | + le32_encode_bits(addr_cam->addr_mask, ADDR_CAM_W2_ADDR_MASK) | + le32_encode_bits(addr_cam->mask_sel, ADDR_CAM_W2_MASK_SEL) | + le32_encode_bits(sma_hash, ADDR_CAM_W2_SMA_HASH) | + le32_encode_bits(tma_hash, ADDR_CAM_W2_TMA_HASH); + h2c->w3 = le32_encode_bits(addr_cam->bssid_cam_idx, ADDR_CAM_W3_BSSID_CAM_IDX); + h2c->w4 = le32_encode_bits(sma[0], ADDR_CAM_W4_SMA0) | + le32_encode_bits(sma[1], ADDR_CAM_W4_SMA1) | + le32_encode_bits(sma[2], ADDR_CAM_W4_SMA2) | + le32_encode_bits(sma[3], ADDR_CAM_W4_SMA3); + h2c->w5 = le32_encode_bits(sma[4], ADDR_CAM_W5_SMA4) | + le32_encode_bits(sma[5], ADDR_CAM_W5_SMA5) | + le32_encode_bits(tma[0], ADDR_CAM_W5_TMA0) | + le32_encode_bits(tma[1], ADDR_CAM_W5_TMA1); + h2c->w6 = le32_encode_bits(tma[2], ADDR_CAM_W6_TMA2) | + le32_encode_bits(tma[3], ADDR_CAM_W6_TMA3) | + le32_encode_bits(tma[4], ADDR_CAM_W6_TMA4) | + le32_encode_bits(tma[5], ADDR_CAM_W6_TMA5); + h2c->w8 = le32_encode_bits(rtwvif_link->port, ADDR_CAM_W8_PORT_INT) | + le32_encode_bits(rtwvif_link->port, ADDR_CAM_W8_TSF_SYNC) | + le32_encode_bits(rtwvif_link->trigger, ADDR_CAM_W8_TF_TRS) | + le32_encode_bits(rtwvif_link->lsig_txop, ADDR_CAM_W8_LSIG_TXOP) | + le32_encode_bits(rtwvif_link->tgt_ind, ADDR_CAM_W8_TGT_IND) | + le32_encode_bits(rtwvif_link->frm_tgt_ind, ADDR_CAM_W8_FRM_TGT_IND) | + le32_encode_bits(rtwsta_link ? rtwsta_link->mac_id : + rtwvif_link->mac_id, ADDR_CAM_W8_MACID); + if (rtwvif_link->net_type == RTW89_NET_TYPE_INFRA) - FWCMD_SET_ADDR_AID12(cmd, vif->cfg.aid & 0xfff); + h2c->w9 = le32_encode_bits(vif->cfg.aid & 0xfff, ADDR_CAM_W9_AID12); else if (rtwvif_link->net_type == RTW89_NET_TYPE_AP_MODE) - FWCMD_SET_ADDR_AID12(cmd, sta ? sta->aid & 0xfff : 0); - FWCMD_SET_ADDR_WOL_PATTERN(cmd, rtwvif_link->wowlan_pattern); - FWCMD_SET_ADDR_WOL_UC(cmd, rtwvif_link->wowlan_uc); - FWCMD_SET_ADDR_WOL_MAGIC(cmd, rtwvif_link->wowlan_magic); - FWCMD_SET_ADDR_WAPI(cmd, addr_cam->wapi); - FWCMD_SET_ADDR_SEC_ENT_MODE(cmd, addr_cam->sec_ent_mode); - FWCMD_SET_ADDR_SEC_ENT0_KEYID(cmd, addr_cam->sec_ent_keyid[0]); - FWCMD_SET_ADDR_SEC_ENT1_KEYID(cmd, addr_cam->sec_ent_keyid[1]); - FWCMD_SET_ADDR_SEC_ENT2_KEYID(cmd, addr_cam->sec_ent_keyid[2]); - FWCMD_SET_ADDR_SEC_ENT3_KEYID(cmd, addr_cam->sec_ent_keyid[3]); - FWCMD_SET_ADDR_SEC_ENT4_KEYID(cmd, addr_cam->sec_ent_keyid[4]); - FWCMD_SET_ADDR_SEC_ENT5_KEYID(cmd, addr_cam->sec_ent_keyid[5]); - FWCMD_SET_ADDR_SEC_ENT6_KEYID(cmd, addr_cam->sec_ent_keyid[6]); - - FWCMD_SET_ADDR_SEC_ENT_VALID(cmd, addr_cam->sec_cam_map[0] & 0xff); - FWCMD_SET_ADDR_SEC_ENT0(cmd, addr_cam->sec_ent[0]); - FWCMD_SET_ADDR_SEC_ENT1(cmd, addr_cam->sec_ent[1]); - FWCMD_SET_ADDR_SEC_ENT2(cmd, addr_cam->sec_ent[2]); - FWCMD_SET_ADDR_SEC_ENT3(cmd, addr_cam->sec_ent[3]); - FWCMD_SET_ADDR_SEC_ENT4(cmd, addr_cam->sec_ent[4]); - FWCMD_SET_ADDR_SEC_ENT5(cmd, addr_cam->sec_ent[5]); - FWCMD_SET_ADDR_SEC_ENT6(cmd, addr_cam->sec_ent[6]); + h2c->w9 = le32_encode_bits(sta ? sta->aid & 0xfff : 0, ADDR_CAM_W9_AID12); + + h2c->w9 |= le32_encode_bits(rtwvif_link->wowlan_pattern, ADDR_CAM_W9_WOL_PATTERN) | + le32_encode_bits(rtwvif_link->wowlan_uc, ADDR_CAM_W9_WOL_UC) | + le32_encode_bits(rtwvif_link->wowlan_magic, ADDR_CAM_W9_WOL_MAGIC) | + le32_encode_bits(addr_cam->wapi, ADDR_CAM_W9_WAPI) | + le32_encode_bits(addr_cam->sec_ent_mode, ADDR_CAM_W9_SEC_ENT_MODE) | + le32_encode_bits(addr_cam->sec_ent_keyid[0], ADDR_CAM_W9_SEC_ENT0_KEYID) | + le32_encode_bits(addr_cam->sec_ent_keyid[1], ADDR_CAM_W9_SEC_ENT1_KEYID) | + le32_encode_bits(addr_cam->sec_ent_keyid[2], ADDR_CAM_W9_SEC_ENT2_KEYID) | + le32_encode_bits(addr_cam->sec_ent_keyid[3], ADDR_CAM_W9_SEC_ENT3_KEYID) | + le32_encode_bits(addr_cam->sec_ent_keyid[4], ADDR_CAM_W9_SEC_ENT4_KEYID) | + le32_encode_bits(addr_cam->sec_ent_keyid[5], ADDR_CAM_W9_SEC_ENT5_KEYID) | + le32_encode_bits(addr_cam->sec_ent_keyid[6], ADDR_CAM_W9_SEC_ENT6_KEYID); + h2c->w10 = le32_encode_bits(addr_cam->sec_cam_map[0] & 0xff, ADDR_CAM_W10_SEC_ENT_VALID) | + le32_encode_bits(addr_cam->sec_ent[0], ADDR_CAM_W10_SEC_ENT0) | + le32_encode_bits(addr_cam->sec_ent[1], ADDR_CAM_W10_SEC_ENT1) | + le32_encode_bits(addr_cam->sec_ent[2], ADDR_CAM_W10_SEC_ENT2); + h2c->w11 = le32_encode_bits(addr_cam->sec_ent[3], ADDR_CAM_W11_SEC_ENT3) | + le32_encode_bits(addr_cam->sec_ent[4], ADDR_CAM_W11_SEC_ENT4) | + le32_encode_bits(addr_cam->sec_ent[5], ADDR_CAM_W11_SEC_ENT5) | + le32_encode_bits(addr_cam->sec_ent[6], ADDR_CAM_W11_SEC_ENT6); rcu_read_unlock(); } diff --git a/drivers/net/wireless/realtek/rtw89/cam.h b/drivers/net/wireless/realtek/rtw89/cam.h index 8fd2d776408e..4436bacf0348 100644 --- a/drivers/net/wireless/realtek/rtw89/cam.h +++ b/drivers/net/wireless/realtek/rtw89/cam.h @@ -12,345 +12,98 @@ #define RTW89_BSSID_MATCH_ALL GENMASK(5, 0) #define RTW89_BSSID_MATCH_5_BYTES GENMASK(4, 0) -static inline void FWCMD_SET_ADDR_IDX(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 1, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_OFFSET(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 1, value, GENMASK(15, 8)); -} - -static inline void FWCMD_SET_ADDR_LEN(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 1, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_VALID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, BIT(0)); -} - -static inline void FWCMD_SET_ADDR_NET_TYPE(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, GENMASK(2, 1)); -} - -static inline void FWCMD_SET_ADDR_BCN_HIT_COND(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, GENMASK(4, 3)); -} - -static inline void FWCMD_SET_ADDR_HIT_RULE(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, GENMASK(6, 5)); -} - -static inline void FWCMD_SET_ADDR_BB_SEL(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, BIT(7)); -} - -static inline void FWCMD_SET_ADDR_ADDR_MASK(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, GENMASK(13, 8)); -} - -static inline void FWCMD_SET_ADDR_MASK_SEL(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, GENMASK(15, 14)); -} - -static inline void FWCMD_SET_ADDR_SMA_HASH(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_TMA_HASH(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 2, value, GENMASK(31, 24)); -} - -static inline void FWCMD_SET_ADDR_BSSID_CAM_IDX(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 3, value, GENMASK(5, 0)); -} - -static inline void FWCMD_SET_ADDR_SMA0(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 4, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_SMA1(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 4, value, GENMASK(15, 8)); -} - -static inline void FWCMD_SET_ADDR_SMA2(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 4, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_SMA3(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 4, value, GENMASK(31, 24)); -} - -static inline void FWCMD_SET_ADDR_SMA4(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 5, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_SMA5(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 5, value, GENMASK(15, 8)); -} - -static inline void FWCMD_SET_ADDR_TMA0(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 5, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_TMA1(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 5, value, GENMASK(31, 24)); -} - -static inline void FWCMD_SET_ADDR_TMA2(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 6, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_TMA3(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 6, value, GENMASK(15, 8)); -} - -static inline void FWCMD_SET_ADDR_TMA4(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 6, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_TMA5(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 6, value, GENMASK(31, 24)); -} - -static inline void FWCMD_SET_ADDR_MACID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 8, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_PORT_INT(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 8, value, GENMASK(10, 8)); -} - -static inline void FWCMD_SET_ADDR_TSF_SYNC(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 8, value, GENMASK(13, 11)); -} - -static inline void FWCMD_SET_ADDR_TF_TRS(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 8, value, BIT(14)); -} - -static inline void FWCMD_SET_ADDR_LSIG_TXOP(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 8, value, BIT(15)); -} - -static inline void FWCMD_SET_ADDR_TGT_IND(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 8, value, GENMASK(26, 24)); -} - -static inline void FWCMD_SET_ADDR_FRM_TGT_IND(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 8, value, GENMASK(29, 27)); -} - -static inline void FWCMD_SET_ADDR_AID12(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(11, 0)); -} - -static inline void FWCMD_SET_ADDR_AID12_0(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_AID12_1(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(11, 8)); -} - -static inline void FWCMD_SET_ADDR_WOL_PATTERN(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, BIT(12)); -} - -static inline void FWCMD_SET_ADDR_WOL_UC(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, BIT(13)); -} - -static inline void FWCMD_SET_ADDR_WOL_MAGIC(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, BIT(14)); -} - -static inline void FWCMD_SET_ADDR_WAPI(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, BIT(15)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT_MODE(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(17, 16)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT0_KEYID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(19, 18)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT1_KEYID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(21, 20)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT2_KEYID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(23, 22)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT3_KEYID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(25, 24)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT4_KEYID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(27, 26)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT5_KEYID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(29, 28)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT6_KEYID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 9, value, GENMASK(31, 30)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT_VALID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 10, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT0(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 10, value, GENMASK(15, 8)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT1(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 10, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT2(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 10, value, GENMASK(31, 24)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT3(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 11, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT4(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 11, value, GENMASK(15, 8)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT5(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 11, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_SEC_ENT6(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 11, value, GENMASK(31, 24)); -} - -static inline void FWCMD_SET_ADDR_BSSID_IDX(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 12, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_BSSID_OFFSET(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 12, value, GENMASK(15, 8)); -} - -static inline void FWCMD_SET_ADDR_BSSID_LEN(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 12, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_BSSID_VALID(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 13, value, BIT(0)); -} - -static inline void FWCMD_SET_ADDR_BSSID_BB_SEL(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 13, value, BIT(1)); -} - -static inline void FWCMD_SET_ADDR_BSSID_MASK(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 13, value, GENMASK(7, 2)); -} - -static inline void FWCMD_SET_ADDR_BSSID_BSS_COLOR(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 13, value, GENMASK(13, 8)); -} - -static inline void FWCMD_SET_ADDR_BSSID_BSSID0(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 13, value, GENMASK(23, 16)); -} - -static inline void FWCMD_SET_ADDR_BSSID_BSSID1(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 13, value, GENMASK(31, 24)); -} - -static inline void FWCMD_SET_ADDR_BSSID_BSSID2(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 14, value, GENMASK(7, 0)); -} - -static inline void FWCMD_SET_ADDR_BSSID_BSSID3(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 14, value, GENMASK(15, 8)); -} +struct rtw89_h2c_addr_cam_v0 { + __le32 w0; + __le32 w1; + __le32 w2; + __le32 w3; + __le32 w4; + __le32 w5; + __le32 w6; + __le32 w7; + __le32 w8; + __le32 w9; + __le32 w10; + __le32 w11; + __le32 w12; + __le32 w13; + __le32 w14; +} __packed; -static inline void FWCMD_SET_ADDR_BSSID_BSSID4(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 14, value, GENMASK(23, 16)); -} +struct rtw89_h2c_addr_cam { + struct rtw89_h2c_addr_cam_v0 v0; + __le32 w15; +} __packed; -static inline void FWCMD_SET_ADDR_BSSID_BSSID5(void *cmd, u32 value) -{ - le32p_replace_bits((__le32 *)(cmd) + 14, value, GENMASK(31, 24)); -} +#define ADDR_CAM_W1_IDX GENMASK(7, 0) +#define ADDR_CAM_W1_OFFSET GENMASK(15, 8) +#define ADDR_CAM_W1_LEN GENMASK(23, 16) +#define ADDR_CAM_W2_VALID BIT(0) +#define ADDR_CAM_W2_NET_TYPE GENMASK(2, 1) +#define ADDR_CAM_W2_BCN_HIT_COND GENMASK(4, 3) +#define ADDR_CAM_W2_HIT_RULE GENMASK(6, 5) +#define ADDR_CAM_W2_BB_SEL BIT(7) +#define ADDR_CAM_W2_ADDR_MASK GENMASK(13, 8) +#define ADDR_CAM_W2_MASK_SEL GENMASK(15, 14) +#define ADDR_CAM_W2_SMA_HASH GENMASK(23, 16) +#define ADDR_CAM_W2_TMA_HASH GENMASK(31, 24) +#define ADDR_CAM_W3_BSSID_CAM_IDX GENMASK(5, 0) +#define ADDR_CAM_W4_SMA0 GENMASK(7, 0) +#define ADDR_CAM_W4_SMA1 GENMASK(15, 8) +#define ADDR_CAM_W4_SMA2 GENMASK(23, 16) +#define ADDR_CAM_W4_SMA3 GENMASK(31, 24) +#define ADDR_CAM_W5_SMA4 GENMASK(7, 0) +#define ADDR_CAM_W5_SMA5 GENMASK(15, 8) +#define ADDR_CAM_W5_TMA0 GENMASK(23, 16) +#define ADDR_CAM_W5_TMA1 GENMASK(31, 24) +#define ADDR_CAM_W6_TMA2 GENMASK(7, 0) +#define ADDR_CAM_W6_TMA3 GENMASK(15, 8) +#define ADDR_CAM_W6_TMA4 GENMASK(23, 16) +#define ADDR_CAM_W6_TMA5 GENMASK(31, 24) +#define ADDR_CAM_W8_MACID GENMASK(7, 0) +#define ADDR_CAM_W8_PORT_INT GENMASK(10, 8) +#define ADDR_CAM_W8_TSF_SYNC GENMASK(13, 11) +#define ADDR_CAM_W8_TF_TRS BIT(14) +#define ADDR_CAM_W8_LSIG_TXOP BIT(15) +#define ADDR_CAM_W8_TGT_IND GENMASK(26, 24) +#define ADDR_CAM_W8_FRM_TGT_IND GENMASK(29, 27) +#define ADDR_CAM_W9_AID12 GENMASK(11, 0) +#define ADDR_CAM_W9_AID12_0 GENMASK(7, 0) +#define ADDR_CAM_W9_AID12_1 GENMASK(11, 8) +#define ADDR_CAM_W9_WOL_PATTERN BIT(12) +#define ADDR_CAM_W9_WOL_UC BIT(13) +#define ADDR_CAM_W9_WOL_MAGIC BIT(14) +#define ADDR_CAM_W9_WAPI BIT(15) +#define ADDR_CAM_W9_SEC_ENT_MODE GENMASK(17, 16) +#define ADDR_CAM_W9_SEC_ENT0_KEYID GENMASK(19, 18) +#define ADDR_CAM_W9_SEC_ENT1_KEYID GENMASK(21, 20) +#define ADDR_CAM_W9_SEC_ENT2_KEYID GENMASK(23, 22) +#define ADDR_CAM_W9_SEC_ENT3_KEYID GENMASK(25, 24) +#define ADDR_CAM_W9_SEC_ENT4_KEYID GENMASK(27, 26) +#define ADDR_CAM_W9_SEC_ENT5_KEYID GENMASK(29, 28) +#define ADDR_CAM_W9_SEC_ENT6_KEYID GENMASK(31, 30) +#define ADDR_CAM_W10_SEC_ENT_VALID GENMASK(7, 0) +#define ADDR_CAM_W10_SEC_ENT0 GENMASK(15, 8) +#define ADDR_CAM_W10_SEC_ENT1 GENMASK(23, 16) +#define ADDR_CAM_W10_SEC_ENT2 GENMASK(31, 24) +#define ADDR_CAM_W11_SEC_ENT3 GENMASK(7, 0) +#define ADDR_CAM_W11_SEC_ENT4 GENMASK(15, 8) +#define ADDR_CAM_W11_SEC_ENT5 GENMASK(23, 16) +#define ADDR_CAM_W11_SEC_ENT6 GENMASK(31, 24) +#define ADDR_CAM_W12_BSSID_IDX GENMASK(7, 0) +#define ADDR_CAM_W12_BSSID_OFFSET GENMASK(15, 8) +#define ADDR_CAM_W12_BSSID_LEN GENMASK(23, 16) +#define ADDR_CAM_W13_BSSID_VALID BIT(0) +#define ADDR_CAM_W13_BSSID_BB_SEL BIT(1) +#define ADDR_CAM_W13_BSSID_MASK GENMASK(7, 2) +#define ADDR_CAM_W13_BSSID_BSS_COLOR GENMASK(13, 8) +#define ADDR_CAM_W13_BSSID_BSSID0 GENMASK(23, 16) +#define ADDR_CAM_W13_BSSID_BSSID1 GENMASK(31, 24) +#define ADDR_CAM_W14_BSSID_BSSID2 GENMASK(7, 0) +#define ADDR_CAM_W14_BSSID_BSSID3 GENMASK(15, 8) +#define ADDR_CAM_W14_BSSID_BSSID4 GENMASK(23, 16) +#define ADDR_CAM_W14_BSSID_BSSID5 GENMASK(31, 24) +#define ADDR_CAM_W15_UPD_MODE GENMASK(2, 0) struct rtw89_h2c_dctlinfo_ud_v1 { __le32 c0; @@ -552,9 +305,10 @@ int rtw89_cam_init_bssid_cam(struct rtw89_dev *rtwdev, void rtw89_cam_deinit_bssid_cam(struct rtw89_dev *rtwdev, struct rtw89_bssid_cam_entry *bssid_cam); void rtw89_cam_fill_addr_cam_info(struct rtw89_dev *rtwdev, - struct rtw89_vif_link *vif, + struct rtw89_vif_link *rtwvif_link, struct rtw89_sta_link *rtwsta_link, - const u8 *scan_mac_addr, u8 *cmd); + const u8 *scan_mac_addr, + struct rtw89_h2c_addr_cam_v0 *h2c); void rtw89_cam_fill_dctl_sec_cam_info_v1(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link, struct rtw89_sta_link *rtwsta_link, @@ -565,7 +319,8 @@ void rtw89_cam_fill_dctl_sec_cam_info_v2(struct rtw89_dev *rtwdev, struct rtw89_h2c_dctlinfo_ud_v2 *h2c); int rtw89_cam_fill_bssid_cam_info(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link, - struct rtw89_sta_link *rtwsta_link, u8 *cmd); + struct rtw89_sta_link *rtwsta_link, + struct rtw89_h2c_addr_cam_v0 *h2c); int rtw89_cam_sec_key_add(struct rtw89_dev *rtwdev, struct ieee80211_vif *vif, struct ieee80211_sta *sta, diff --git a/drivers/net/wireless/realtek/rtw89/core.c b/drivers/net/wireless/realtek/rtw89/core.c index 018857d3569a..46c5de687dfc 100644 --- a/drivers/net/wireless/realtek/rtw89/core.c +++ b/drivers/net/wireless/realtek/rtw89/core.c @@ -2499,12 +2499,14 @@ static u16 rtw89_bcn_get_histogram_bound(struct rtw89_dev *rtwdev, u8 target) } static u16 rtw89_bcn_get_rx_time(struct rtw89_dev *rtwdev, - const struct rtw89_chan *chan) + struct rtw89_vif_link *rtwvif_link) { #define RTW89_SYMBOL_TIME_2GHZ 192 #define RTW89_SYMBOL_TIME_5GHZ 20 #define RTW89_SYMBOL_TIME_6GHZ 20 - struct rtw89_pkt_stat *pkt_stat = &rtwdev->phystat.cur_pkt_stat; + const struct rtw89_chan *chan = rtw89_chan_get(rtwdev, rtwvif_link->chanctx_idx); + struct rtw89_bb_ctx *bb = rtw89_get_bb_ctx(rtwdev, rtwvif_link->phy_idx); + struct rtw89_pkt_stat *pkt_stat = &bb->cur_pkt_stat; u16 bitrate, val; if (!rtw89_legacy_rate_to_bitrate(rtwdev, pkt_stat->beacon_rate, &bitrate)) @@ -2535,15 +2537,15 @@ static void rtw89_bcn_calc_timeout(struct rtw89_dev *rtwdev, #define RTW89_BCN_TRACK_EXTEND_TIMEOUT 5 #define RTW89_BCN_TRACK_COVERAGE_TH 0 /* unit: TU */ #define RTW89_BCN_TRACK_STRONG_RSSI 80 - const struct rtw89_chan *chan = rtw89_chan_get(rtwdev, rtwvif_link->chanctx_idx); - struct rtw89_pkt_stat *pkt_stat = &rtwdev->phystat.cur_pkt_stat; + struct rtw89_bb_ctx *bb = rtw89_get_bb_ctx(rtwdev, rtwvif_link->phy_idx); struct rtw89_beacon_stat *bcn_stat = &rtwdev->phystat.bcn_stat; struct rtw89_beacon_track_info *bcn_track = &rtwdev->bcn_track; + struct rtw89_pkt_stat *pkt_stat = &bb->cur_pkt_stat; struct rtw89_beacon_dist *bcn_dist = &bcn_stat->bcn_dist; u16 outlier_high_bcn_th = bcn_track->outlier_high_bcn_th; u16 outlier_low_bcn_th = bcn_track->outlier_low_bcn_th; - u8 rssi = ewma_rssi_read(&rtwdev->phystat.bcn_rssi); u16 target_bcn_th = bcn_track->target_bcn_th; + u8 rssi = ewma_rssi_read(&bb->bcn_rssi); u16 low_bcn_th = bcn_track->low_bcn_th; u16 med_bcn_th = bcn_track->med_bcn_th; u16 beacon_int = bcn_track->beacon_int; @@ -2589,7 +2591,7 @@ static void rtw89_bcn_calc_timeout(struct rtw89_dev *rtwdev, bcn_timeout = bcn_stat->drift[target_bcn_th]; out: - bcn_track->bcn_timeout = bcn_timeout + rtw89_bcn_get_rx_time(rtwdev, chan); + bcn_track->bcn_timeout = bcn_timeout + rtw89_bcn_get_rx_time(rtwdev, rtwvif_link); } static void rtw89_bcn_update_timeout(struct rtw89_dev *rtwdev, @@ -2733,7 +2735,6 @@ static void rtw89_vif_rx_stats_iter(void *data, u8 *mac, struct rtw89_vif_rx_stats_iter_data *iter_data = data; struct rtw89_dev *rtwdev = iter_data->rtwdev; struct rtw89_vif *rtwvif = vif_to_rtwvif(vif); - struct rtw89_pkt_stat *pkt_stat = &rtwdev->phystat.cur_pkt_stat; struct rtw89_rx_desc_info *desc_info = iter_data->desc_info; struct sk_buff *skb = iter_data->skb; struct ieee80211_rx_status *rx_status = IEEE80211_SKB_RXCB(skb); @@ -2743,6 +2744,8 @@ static void rtw89_vif_rx_stats_iter(void *data, u8 *mac, struct ieee80211_bss_conf *bss_conf; struct rtw89_vif_link *rtwvif_link; const u8 *bssid = iter_data->bssid; + struct rtw89_pkt_stat *pkt_stat; + struct rtw89_bb_ctx *bb; const u8 *target_bssid; if (rtwdev->scanning && @@ -2776,6 +2779,9 @@ static void rtw89_vif_rx_stats_iter(void *data, u8 *mac, rx_status->link_id = rtwvif_link->link_id; } + bb = rtw89_get_bb_ctx(rtwdev, rtwvif_link->phy_idx); + pkt_stat = &bb->cur_pkt_stat; + if (ieee80211_is_beacon(hdr->frame_control)) { if (vif->type == NL80211_IFTYPE_STATION && !test_bit(RTW89_FLAG_WOWLAN, rtwdev->flags)) { @@ -2784,7 +2790,7 @@ static void rtw89_vif_rx_stats_iter(void *data, u8 *mac, } if (phy_ppdu) { - ewma_rssi_add(&rtwdev->phystat.bcn_rssi, phy_ppdu->rssi_avg); + ewma_rssi_add(&bb->bcn_rssi, phy_ppdu->rssi_avg); if (!test_bit(RTW89_FLAG_LOW_POWER_MODE, rtwdev->flags)) rtwvif_link->bcn_bw_idx = phy_ppdu->bw_idx; } @@ -4270,14 +4276,22 @@ static void rtw89_core_mlsr_link_decision(struct rtw89_dev *rtwdev, { unsigned int sel_link_id = IEEE80211_MLD_MAX_NUM_LINKS; struct ieee80211_vif *vif = rtwvif_to_vif(rtwvif); + u8 decided_bands = BIT(RTW89_BAND_NUM) - 1; struct rtw89_vif_link *rtwvif_link; const struct rtw89_chan *chan; unsigned long usable_links; + struct rtw89_bb_ctx *bb; unsigned int link_id; - u8 decided_bands; u8 rssi; - rssi = ewma_rssi_read(&rtwdev->phystat.bcn_rssi); + usable_links = ieee80211_vif_usable_links(vif); + + rtwvif_link = rtw89_get_designated_link(rtwvif); + if (unlikely(!rtwvif_link)) + goto select; + + bb = rtw89_get_bb_ctx(rtwdev, rtwvif_link->phy_idx); + rssi = ewma_rssi_read(&bb->bcn_rssi); if (unlikely(!rssi)) return; @@ -4288,12 +4302,6 @@ static void rtw89_core_mlsr_link_decision(struct rtw89_dev *rtwdev, else return; - usable_links = ieee80211_vif_usable_links(vif); - - rtwvif_link = rtw89_get_designated_link(rtwvif); - if (unlikely(!rtwvif_link)) - goto select; - chan = rtw89_chan_get(rtwdev, rtwvif_link->chanctx_idx); if (decided_bands & BIT(chan->band_type)) return; @@ -4703,7 +4711,8 @@ int rtw89_core_sta_link_disconnect(struct rtw89_dev *rtwdev, } /* update cam aid mac_id net_type */ - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL, + RTW89_ROLE_CON_DISCONN); if (ret) { rtw89_warn(rtwdev, "failed to send h2c cam\n"); return ret; @@ -4777,7 +4786,8 @@ int rtw89_core_sta_link_assoc(struct rtw89_dev *rtwdev, } /* update cam aid mac_id net_type */ - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL, + RTW89_ROLE_CON_DISCONN); if (ret) { rtw89_warn(rtwdev, "failed to send h2c cam\n"); return ret; @@ -5895,7 +5905,8 @@ void rtw89_core_scan_start(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwv rtw89_phy_config_edcca(rtwdev, bb, true); rtw89_tas_scan(rtwdev, true); - rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, mac_addr); + rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, mac_addr, + RTW89_ROLE_INFO_CHANGE); } void rtw89_core_scan_complete(struct rtw89_dev *rtwdev, @@ -5915,7 +5926,8 @@ void rtw89_core_scan_complete(struct rtw89_dev *rtwdev, rcu_read_unlock(); - rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL); + rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL, + RTW89_ROLE_INFO_CHANGE); rtw89_chip_rfk_scan(rtwdev, rtwvif_link, false); rtw89_btc_ntfy_scan_finish(rtwdev, rtwvif_link->phy_idx); diff --git a/drivers/net/wireless/realtek/rtw89/core.h b/drivers/net/wireless/realtek/rtw89/core.h index e146fd082f21..4f2a9ac1ffd6 100644 --- a/drivers/net/wireless/realtek/rtw89/core.h +++ b/drivers/net/wireless/realtek/rtw89/core.h @@ -38,6 +38,8 @@ extern const struct ieee80211_ops rtw89_ops; #define RFREG_MASK 0xfffff #define INV_RF_DATA 0xffffffff #define BYPASS_CR_DATA 0xbabecafe +#define RTW89_R32_EA 0xEAEAEAEA +#define RTW89_R32_DEAD 0xDEADBEEF #define RTW89_TRACK_WORK_PERIOD round_jiffies_relative(HZ * 2) #define RTW89_TRACK_PS_WORK_PERIOD msecs_to_jiffies(100) @@ -3652,6 +3654,8 @@ struct rtw89_hci_ops { void (*write16)(struct rtw89_dev *rtwdev, u32 addr, u16 data); void (*write32)(struct rtw89_dev *rtwdev, u32 addr, u32 data); + u32 (*read32_pci_cfg)(struct rtw89_dev *rtwdev, u32 addr); + int (*mac_pre_init)(struct rtw89_dev *rtwdev); int (*mac_pre_deinit)(struct rtw89_dev *rtwdev); int (*mac_post_init)(struct rtw89_dev *rtwdev); @@ -4640,6 +4644,7 @@ enum rtw89_fw_feature { RTW89_FW_FEATURE_RFK_NTFY_MCC_V0, RTW89_FW_FEATURE_LPS_DACK_BY_C2H_REG, RTW89_FW_FEATURE_BEACON_TRACKING, + RTW89_FW_FEATURE_ADDR_CAM_V0, }; struct rtw89_fw_suit { @@ -4700,6 +4705,7 @@ struct rtw89_fw_elm_info { struct rtw89_phy_rfk_log_fmt *rfk_log_fmt; const struct rtw89_regd_data *regd; const struct rtw89_fw_element_hdr *afe; + const struct rtw89_fw_element_hdr *diag_mac; }; enum rtw89_fw_mss_dev_type { @@ -5042,6 +5048,7 @@ enum rtw89_flags { RTW89_FLAG_CHANGING_INTERFACE, RTW89_FLAG_HW_RFKILL_STATE, RTW89_FLAG_UNPLUGGED, + RTW89_FLAG_SHUTDOWN, NUM_OF_RTW89_FLAGS, }; @@ -5129,9 +5136,6 @@ DECLARE_EWMA(thermal, 4, 4); struct rtw89_phy_stat { struct ewma_thermal avg_thermal[RF_PATH_MAX]; u8 last_thermal_max; - struct ewma_rssi bcn_rssi; - struct rtw89_pkt_stat cur_pkt_stat; - struct rtw89_pkt_stat last_pkt_stat; struct rtw89_beacon_stat bcn_stat; }; @@ -6056,6 +6060,9 @@ struct rtw89_dev { struct rtw89_dig_info dig; struct rtw89_phy_ch_info ch_info; struct rtw89_edcca_bak edcca_bak; + struct ewma_rssi bcn_rssi; + struct rtw89_pkt_stat cur_pkt_stat; + struct rtw89_pkt_stat last_pkt_stat; } bbs[RTW89_PHY_NUM]; struct wiphy_delayed_work track_work; @@ -6624,6 +6631,15 @@ rtw89_write_rf(struct rtw89_dev *rtwdev, enum rtw89_rf_path rf_path, mutex_unlock(&rtwdev->rf_mutex); } +static inline u32 rtw89_read32_pci_cfg(struct rtw89_dev *rtwdev, u32 addr) +{ + if (rtwdev->hci.type != RTW89_HCI_TYPE_PCIE || + !rtwdev->hci.ops->read32_pci_cfg) + return RTW89_R32_EA; + + return rtwdev->hci.ops->read32_pci_cfg(rtwdev, addr); +} + static inline struct ieee80211_txq *rtw89_txq_to_txq(struct rtw89_txq *rtwtxq) { void *p = rtwtxq; diff --git a/drivers/net/wireless/realtek/rtw89/debug.c b/drivers/net/wireless/realtek/rtw89/debug.c index a82df3814069..531b8b90cb75 100644 --- a/drivers/net/wireless/realtek/rtw89/debug.c +++ b/drivers/net/wireless/realtek/rtw89/debug.c @@ -87,6 +87,7 @@ struct rtw89_debugfs { struct rtw89_debugfs_priv disable_dm; struct rtw89_debugfs_priv mlo_mode; struct rtw89_debugfs_priv beacon_info; + struct rtw89_debugfs_priv diag_mac; }; struct rtw89_debugfs_iter_data { @@ -3914,38 +3915,20 @@ static const struct rtw89_rx_rate_cnt_info { {FIRST_RATE_GEV1(EHT_NSS2_MCS0), 14, 0, "EHT 2SS:"}, }; -static ssize_t rtw89_debug_priv_phy_info_get(struct rtw89_dev *rtwdev, - struct rtw89_debugfs_priv *debugfs_priv, - char *buf, size_t bufsz) +static int rtw89_get_rx_pkt_stat(struct rtw89_dev *rtwdev, struct rtw89_bb_ctx *bb, + char *buf, size_t bufsz) { - struct rtw89_traffic_stats *stats = &rtwdev->stats; - struct rtw89_pkt_stat *pkt_stat = &rtwdev->phystat.last_pkt_stat; + struct rtw89_pkt_stat *pkt_stat = &bb->last_pkt_stat; const struct rtw89_chip_info *chip = rtwdev->chip; - struct rtw89_debugfs_iter_data iter_data; const struct rtw89_rx_rate_cnt_info *info; - struct rtw89_hal *hal = &rtwdev->hal; + u8 rssi = ewma_rssi_read(&bb->bcn_rssi); char *p = buf, *end = buf + bufsz; enum rtw89_hw_rate first_rate; - u8 rssi; int i; - rssi = ewma_rssi_read(&rtwdev->phystat.bcn_rssi); - - p += scnprintf(p, end - p, "TP TX: %u [%u] Mbps (lv: %d", - stats->tx_throughput, stats->tx_throughput_raw, - stats->tx_tfc_lv); - if (hal->thermal_prot_lv) - p += scnprintf(p, end - p, ", duty: %d%%", - 100 - hal->thermal_prot_lv * RTW89_THERMAL_PROT_STEP); - p += scnprintf(p, end - p, "), RX: %u [%u] Mbps (lv: %d)\n", - stats->rx_throughput, stats->rx_throughput_raw, - stats->rx_tfc_lv); - p += scnprintf(p, end - p, "Beacon: %u (%d dBm), TF: %u\n", + p += scnprintf(p, end - p, "Beacon: %u (%d dBm)\n", pkt_stat->beacon_nr, - RTW89_RSSI_RAW_TO_DBM(rssi), stats->rx_tf_periodic); - p += scnprintf(p, end - p, "Avg packet length: TX=%u, RX=%u\n", - stats->tx_avg_len, - stats->rx_avg_len); + RTW89_RSSI_RAW_TO_DBM(rssi)); p += scnprintf(p, end - p, "RX count:\n"); @@ -3966,6 +3949,39 @@ static ssize_t rtw89_debug_priv_phy_info_get(struct rtw89_dev *rtwdev, p += scnprintf(p, end - p, "]\n"); } + return p - buf; +} + +static ssize_t rtw89_debug_priv_phy_info_get(struct rtw89_dev *rtwdev, + struct rtw89_debugfs_priv *debugfs_priv, + char *buf, size_t bufsz) +{ + struct rtw89_traffic_stats *stats = &rtwdev->stats; + struct rtw89_debugfs_iter_data iter_data; + struct rtw89_hal *hal = &rtwdev->hal; + char *p = buf, *end = buf + bufsz; + struct rtw89_bb_ctx *bb; + + p += scnprintf(p, end - p, "TP TX: %u [%u] Mbps (lv: %d", + stats->tx_throughput, stats->tx_throughput_raw, + stats->tx_tfc_lv); + if (hal->thermal_prot_lv) + p += scnprintf(p, end - p, ", duty: %d%%", + 100 - hal->thermal_prot_lv * RTW89_THERMAL_PROT_STEP); + p += scnprintf(p, end - p, "), RX: %u [%u] Mbps (lv: %d)\n", + stats->rx_throughput, stats->rx_throughput_raw, + stats->rx_tfc_lv); + p += scnprintf(p, end - p, "Avg packet length: TX=%u, RX=%u\n", + stats->tx_avg_len, + stats->rx_avg_len); + p += scnprintf(p, end - p, "TF: %u\n", stats->rx_tf_periodic); + + rtw89_for_each_active_bb(rtwdev, bb) { + p += scnprintf(p, end - p, "\n[PHY %u]\n", bb->phy_idx); + p += rtw89_get_rx_pkt_stat(rtwdev, bb, p, end - p); + } + p += scnprintf(p, end - p, "\n"); + rtw89_debugfs_iter_data_setup(&iter_data, p, end - p); ieee80211_iterate_stations_atomic(rtwdev->hw, rtw89_sta_info_get_iter, &iter_data); p += iter_data.written_sz; @@ -4361,12 +4377,322 @@ rtw89_debug_priv_mlo_mode_set(struct rtw89_dev *rtwdev, return count; } +enum __diag_mac_cmd { + __CMD_EQUALV, + __CMD_EQUALO, + __CMD_NEQUALV, + __CMD_NEQUALO, + __CMD_SETEQUALV, + __CMD_SETEQUALO, + __CMD_CMPWCR, + __CMD_CMPWWD, + __CMD_NEQ_CMPWCR, + __CMD_NEQ_CMPWWD, + __CMD_INCREMENT, + __CMD_MESSAGE, +}; + +enum __diag_mac_io { + __IO_NORMAL, + __IO_NORMAL_PCIE, + __IO_NORMAL_USB, + __IO_NORMAL_SDIO, + __IO_PCIE_CFG, + __IO_SDIO_CCCR, +}; + +struct __diag_mac_rule_header { + u8 sheet; + u8 cmd; + u8 seq_major; + u8 seq_minor; + u8 io_band; + #define __DIAG_MAC_IO GENMASK(3, 0) + #define __DIAG_MAC_N_BAND BIT(4) + #define __DIAG_MAC_HAS_BAND BIT(5) + u8 len; /* include header. Unit: 4 bytes */ + u8 rsvd[2]; +} __packed; + +struct __diag_mac_rule_equal { + struct __diag_mac_rule_header header; + __le32 addr; + __le32 addr_name_offset; + __le32 mask; + __le32 val; + __le32 msg_offset; + u8 rsvd[4]; +} __packed; + +struct __diag_mac_rule_increment { + struct __diag_mac_rule_header header; + __le32 addr; + __le32 addr_name_offset; + __le32 mask; + __le16 sel; + __le16 delay; + __le32 msg_offset; + u8 rsvd[4]; +} __packed; + +struct __diag_mac_msg_buf { + __le16 len; + char string[]; +} __packed; + +static ssize_t rtw89_mac_diag_do_equalv(struct rtw89_dev *rtwdev, + char *buf, size_t bufsz, + const struct __diag_mac_rule_equal *r, + const void *msg_start, + u64 *positive_bmp) +{ + const struct __diag_mac_msg_buf *name = msg_start + + le32_to_cpu(r->addr_name_offset); + const struct __diag_mac_msg_buf *msg = msg_start + + le32_to_cpu(r->msg_offset); + bool want_eq = r->header.cmd == __CMD_EQUALV; + char *p = buf, *end = buf + bufsz; + bool equal = false; + u32 val; + + *positive_bmp <<= 1; + + if (u8_get_bits(r->header.io_band, __DIAG_MAC_IO) == __IO_PCIE_CFG) + val = rtw89_read32_pci_cfg(rtwdev, le32_to_cpu(r->addr)); + else + val = rtw89_read32(rtwdev, le32_to_cpu(r->addr)); + + if ((val & le32_to_cpu(r->mask)) == le32_to_cpu(r->val)) + equal = true; + + if (want_eq == equal) { + *positive_bmp |= BIT(0); + return p - buf; + } + + p += scnprintf(p, end - p, "sheet: %d, cmd: %d, Reg: %.*s => %x, %.*s\n", + r->header.sheet, r->header.cmd, le16_to_cpu(name->len), + name->string, val, le16_to_cpu(msg->len), msg->string); + + return p - buf; +} + +static ssize_t rtw89_mac_diag_do_increment(struct rtw89_dev *rtwdev, + char *buf, size_t bufsz, + const struct __diag_mac_rule_increment *r, + const void *msg_start, + u64 *positive_bmp) +{ + const struct __diag_mac_msg_buf *name = msg_start + + le32_to_cpu(r->addr_name_offset); + const struct __diag_mac_msg_buf *msg = msg_start + + le32_to_cpu(r->msg_offset); + char *p = buf, *end = buf + bufsz; + u32 addr = le32_to_cpu(r->addr); + u32 mask = le32_to_cpu(r->mask); + u16 sel = le16_to_cpu(r->sel); + u32 val1, val2; + + *positive_bmp <<= 1; + + rtw89_write32(rtwdev, addr, sel); + + if (u8_get_bits(r->header.io_band, __DIAG_MAC_IO) == __IO_PCIE_CFG) + val1 = rtw89_read32_pci_cfg(rtwdev, addr); + else + val1 = rtw89_read32(rtwdev, addr); + + mdelay(le16_to_cpu(r->delay)); + + if (u8_get_bits(r->header.io_band, __DIAG_MAC_IO) == __IO_PCIE_CFG) + val2 = rtw89_read32_pci_cfg(rtwdev, addr); + else + val2 = rtw89_read32(rtwdev, addr); + + if ((val2 & mask) > (val1 & mask)) { + *positive_bmp |= BIT(0); + return p - buf; + } + + p += scnprintf(p, end - p, "sheet: %d, cmd: %d, Reg: %.*s [%d]=> %x, %.*s\n", + r->header.sheet, r->header.cmd, le16_to_cpu(name->len), + name->string, le16_to_cpu(r->sel), val1, + le16_to_cpu(msg->len), msg->string); + + return p - buf; +} + +static bool rtw89_mac_diag_match_hci(struct rtw89_dev *rtwdev, + const struct __diag_mac_rule_header *rh) +{ + switch (u8_get_bits(rh->io_band, __DIAG_MAC_IO)) { + case __IO_NORMAL: + default: + return true; + case __IO_NORMAL_PCIE: + case __IO_PCIE_CFG: + if (rtwdev->hci.type == RTW89_HCI_TYPE_PCIE) + return true; + break; + case __IO_NORMAL_USB: + if (rtwdev->hci.type == RTW89_HCI_TYPE_USB) + return true; + break; + case __IO_NORMAL_SDIO: + case __IO_SDIO_CCCR: + if (rtwdev->hci.type == RTW89_HCI_TYPE_SDIO) + return true; + break; + } + + return false; +} + +static bool rtw89_mac_diag_match_band(struct rtw89_dev *rtwdev, + const struct __diag_mac_rule_header *rh) +{ + u8 active_bands; + bool has_band; + u8 band; + + has_band = u8_get_bits(rh->io_band, __DIAG_MAC_HAS_BAND); + if (!has_band) + return true; + + band = u8_get_bits(rh->io_band, __DIAG_MAC_N_BAND); + active_bands = rtw89_get_active_phy_bitmap(rtwdev); + + if (active_bands & BIT(band)) + return true; + + return false; +} + +static ssize_t rtw89_mac_diag_iter_all(struct rtw89_dev *rtwdev, + char *buf, size_t bufsz) +{ + const struct rtw89_fw_element_hdr *elm = rtwdev->fw.elm_info.diag_mac; + u32 n_plains = 0, n_rules = 0, n_positive = 0, n_ignore = 0; + char *p = buf, *end = buf + bufsz, *p_rewind; + const void *rule, *rule_end; + u32 elm_size, rule_size; + const void *msg_start; + u64 positive_bmp = 0; + u8 prev_sheet = 0; + u8 prev_seq = 0; + int limit; + + if (!elm) { + p += scnprintf(p, end - p, "No diag_mac entry\n"); + goto out; + } + + rule_size = le32_to_cpu(elm->u.diag_mac.rule_size); + elm_size = le32_to_cpu(elm->size); + + if (ALIGN(rule_size, 16) > elm_size) { + p += scnprintf(p, end - p, "rule size (%u) exceed elm_size (%u)\n", + ALIGN(rule_size, 16), elm_size); + goto out; + } + + rule = &elm->u.diag_mac.rules_and_msgs[0]; + rule_end = &elm->u.diag_mac.rules_and_msgs[rule_size]; + msg_start = &elm->u.diag_mac.rules_and_msgs[ALIGN(rule_size, 16)]; + + for (limit = 0; limit < 5000 && rule < rule_end; limit++) { + const struct __diag_mac_rule_header *rh = rule; + u8 sheet = rh->sheet; + u8 seq = rh->seq_major; + + if (!rtw89_mac_diag_match_hci(rtwdev, rh) || + !rtw89_mac_diag_match_band(rtwdev, rh)) { + n_ignore++; + goto next; + } + + if (!seq || prev_sheet != sheet || prev_seq != seq) { + if (positive_bmp) { + n_positive++; + /* + * discard output for negative results if one in + * a sequence set is positive. + */ + if (p_rewind) + p = p_rewind; + } + p_rewind = seq ? p : NULL; + positive_bmp = 0; + n_rules++; + } + + switch (rh->cmd) { + case __CMD_EQUALV: + case __CMD_NEQUALV: + p += rtw89_mac_diag_do_equalv(rtwdev, p, end - p, rule, + msg_start, &positive_bmp); + break; + case __CMD_INCREMENT: + p += rtw89_mac_diag_do_increment(rtwdev, p, end - p, rule, + msg_start, &positive_bmp); + break; + default: + p += scnprintf(p, end - p, "unknown rule cmd %u\n", rh->cmd); + break; + } + +next: + n_plains++; + rule += rh->len * 4; + prev_seq = seq; + prev_sheet = sheet; + } + + if (positive_bmp) { + n_positive++; + if (p_rewind) + p = p_rewind; + } + + p += scnprintf(p, end - p, "\nPlain(Ignore)/Rules/Positive: %u(%u)/%u/%u\n", + n_plains, n_ignore, n_rules, n_positive); + +out: + return p - buf; +} + +static ssize_t +rtw89_debug_priv_diag_mac_get(struct rtw89_dev *rtwdev, + struct rtw89_debugfs_priv *debugfs_priv, + char *buf, size_t bufsz) +{ + lockdep_assert_wiphy(rtwdev->hw->wiphy); + + rtw89_leave_lps(rtwdev); + + return rtw89_mac_diag_iter_all(rtwdev, buf, bufsz); +} + +static int rtw89_get_beacon_info(struct rtw89_dev *rtwdev, struct rtw89_bb_ctx *bb, + char *buf, size_t bufsz) +{ + struct rtw89_pkt_stat *pkt_stat = &bb->last_pkt_stat; + char *p = buf, *end = buf + bufsz; + + p += scnprintf(p, end - p, "[PHY %u]\n", bb->phy_idx); + p += scnprintf(p, end - p, "Beacon: %u\n", pkt_stat->beacon_nr); + p += scnprintf(p, end - p, "raw rssi: %lu\n", ewma_rssi_read(&bb->bcn_rssi)); + p += scnprintf(p, end - p, "hw rate: %u\n", pkt_stat->beacon_rate); + p += scnprintf(p, end - p, "length: %u\n\n", pkt_stat->beacon_len); + + return p - buf; +} + static ssize_t rtw89_debug_priv_beacon_info_get(struct rtw89_dev *rtwdev, struct rtw89_debugfs_priv *debugfs_priv, char *buf, size_t bufsz) { - struct rtw89_pkt_stat *pkt_stat = &rtwdev->phystat.last_pkt_stat; struct rtw89_beacon_track_info *bcn_track = &rtwdev->bcn_track; struct rtw89_beacon_stat *bcn_stat = &rtwdev->phystat.bcn_stat; struct rtw89_beacon_dist *bcn_dist = &bcn_stat->bcn_dist; @@ -4374,17 +4700,16 @@ rtw89_debug_priv_beacon_info_get(struct rtw89_dev *rtwdev, char *p = buf, *end = buf + bufsz; u16 *drift = bcn_stat->drift; u8 bcn_num = bcn_stat->num; + struct rtw89_bb_ctx *bb; u8 count; u8 i; + rtw89_for_each_active_bb(rtwdev, bb) + p += rtw89_get_beacon_info(rtwdev, bb, p, end - p); + p += scnprintf(p, end - p, "[Beacon info]\n"); - p += scnprintf(p, end - p, "count: %u\n", pkt_stat->beacon_nr); p += scnprintf(p, end - p, "interval: %u\n", bcn_track->beacon_int); p += scnprintf(p, end - p, "dtim: %u\n", bcn_track->dtim); - p += scnprintf(p, end - p, "raw rssi: %lu\n", - ewma_rssi_read(&rtwdev->phystat.bcn_rssi)); - p += scnprintf(p, end - p, "hw rate: %u\n", pkt_stat->beacon_rate); - p += scnprintf(p, end - p, "length: %u\n", pkt_stat->beacon_len); p += scnprintf(p, end - p, "\n[Distribution]\n"); p += scnprintf(p, end - p, "tbtt\n"); @@ -4478,6 +4803,7 @@ static const struct rtw89_debugfs rtw89_debugfs_templ = { .disable_dm = rtw89_debug_priv_set_and_get(disable_dm, RWLOCK), .mlo_mode = rtw89_debug_priv_set_and_get(mlo_mode, RWLOCK), .beacon_info = rtw89_debug_priv_get(beacon_info), + .diag_mac = rtw89_debug_priv_get(diag_mac, RSIZE_16K, RLOCK), }; #define rtw89_debugfs_add(name, mode, fopname, parent) \ @@ -4524,6 +4850,7 @@ void rtw89_debugfs_add_sec1(struct rtw89_dev *rtwdev, struct dentry *debugfs_top rtw89_debugfs_add_rw(disable_dm); rtw89_debugfs_add_rw(mlo_mode); rtw89_debugfs_add_r(beacon_info); + rtw89_debugfs_add_r(diag_mac); } void rtw89_debugfs_init(struct rtw89_dev *rtwdev) diff --git a/drivers/net/wireless/realtek/rtw89/fw.c b/drivers/net/wireless/realtek/rtw89/fw.c index 15ac357bbdba..c6533a085c7f 100644 --- a/drivers/net/wireless/realtek/rtw89/fw.c +++ b/drivers/net/wireless/realtek/rtw89/fw.c @@ -870,6 +870,7 @@ static const struct __fw_feat_cfg fw_feat_tbl[] = { __CFG_FW_FEAT(RTL8922A, ge, 0, 35, 76, 0, LPS_DACK_BY_C2H_REG), __CFG_FW_FEAT(RTL8922A, ge, 0, 35, 79, 0, CRASH_TRIGGER_TYPE_1), __CFG_FW_FEAT(RTL8922A, ge, 0, 35, 80, 0, BEACON_TRACKING), + __CFG_FW_FEAT(RTL8922A, lt, 0, 35, 84, 0, ADDR_CAM_V0), }; static void rtw89_fw_iterate_feature_cfg(struct rtw89_fw_info *fw, @@ -1298,6 +1299,18 @@ int rtw89_build_afe_pwr_seq_from_elm(struct rtw89_dev *rtwdev, return 0; } +static +int rtw89_recognize_diag_mac_from_elm(struct rtw89_dev *rtwdev, + const struct rtw89_fw_element_hdr *elm, + const union rtw89_fw_element_arg arg) +{ + struct rtw89_fw_elm_info *elm_info = &rtwdev->fw.elm_info; + + elm_info->diag_mac = elm; + + return 0; +} + static const struct rtw89_fw_element_handler __fw_element_handlers[] = { [RTW89_FW_ELEMENT_ID_BBMCU0] = {__rtw89_fw_recognize_from_elm, { .fw_type = RTW89_FW_BBMCU0 }, NULL}, @@ -1386,6 +1399,9 @@ static const struct rtw89_fw_element_handler __fw_element_handlers[] = { [RTW89_FW_ELEMENT_ID_AFE_PWR_SEQ] = { rtw89_build_afe_pwr_seq_from_elm, {}, "AFE", }, + [RTW89_FW_ELEMENT_ID_DIAG_MAC] = { + rtw89_recognize_diag_mac_from_elm, {}, NULL, + }, }; int rtw89_fw_recognize_elements(struct rtw89_dev *rtwdev) @@ -2110,28 +2126,48 @@ void rtw89_fw_log_dump(struct rtw89_dev *rtwdev, u8 *buf, u32 len) } -#define H2C_CAM_LEN 60 int rtw89_fw_h2c_cam(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link, - struct rtw89_sta_link *rtwsta_link, const u8 *scan_mac_addr) + struct rtw89_sta_link *rtwsta_link, const u8 *scan_mac_addr, + enum rtw89_upd_mode upd_mode) { + const struct rtw89_chip_info *chip = rtwdev->chip; + struct rtw89_h2c_addr_cam_v0 *h2c_v0; + struct rtw89_h2c_addr_cam *h2c; + u32 len = sizeof(*h2c); struct sk_buff *skb; + u8 ver = U8_MAX; int ret; - skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, H2C_CAM_LEN); + if (RTW89_CHK_FW_FEATURE(ADDR_CAM_V0, &rtwdev->fw) || + chip->chip_gen == RTW89_CHIP_AX) { + len = sizeof(*h2c_v0); + ver = 0; + } + + skb = rtw89_fw_h2c_alloc_skb_with_hdr(rtwdev, len); if (!skb) { rtw89_err(rtwdev, "failed to alloc skb for fw dl\n"); return -ENOMEM; } - skb_put(skb, H2C_CAM_LEN); - rtw89_cam_fill_addr_cam_info(rtwdev, rtwvif_link, rtwsta_link, scan_mac_addr, - skb->data); - rtw89_cam_fill_bssid_cam_info(rtwdev, rtwvif_link, rtwsta_link, skb->data); + skb_put(skb, len); + h2c_v0 = (struct rtw89_h2c_addr_cam_v0 *)skb->data; + + rtw89_cam_fill_addr_cam_info(rtwdev, rtwvif_link, rtwsta_link, + scan_mac_addr, h2c_v0); + rtw89_cam_fill_bssid_cam_info(rtwdev, rtwvif_link, rtwsta_link, h2c_v0); + if (ver == 0) + goto hdr; + + h2c = (struct rtw89_h2c_addr_cam *)skb->data; + h2c->w15 = le32_encode_bits(upd_mode, ADDR_CAM_W15_UPD_MODE); + +hdr: rtw89_h2c_pkt_set_hdr(rtwdev, skb, FWCMD_TYPE_H2C, H2C_CAT_MAC, H2C_CL_MAC_ADDR_CAM_UPDATE, H2C_FUNC_MAC_ADDR_CAM_UPD, 0, 1, - H2C_CAM_LEN); + len); ret = rtw89_h2c_tx(rtwdev, skb, false); if (ret) { @@ -2972,14 +3008,15 @@ int rtw89_fw_h2c_lps_ml_cmn_info(struct rtw89_dev *rtwdev, struct rtw89_vif *rtwvif) { const struct rtw89_phy_bb_gain_info_be *gain = &rtwdev->bb_gain.be; - struct rtw89_pkt_stat *pkt_stat = &rtwdev->phystat.cur_pkt_stat; static const u8 bcn_bw_ofst[] = {0, 0, 0, 3, 6, 9, 0, 12}; const struct rtw89_chip_info *chip = rtwdev->chip; struct rtw89_efuse *efuse = &rtwdev->efuse; struct rtw89_h2c_lps_ml_cmn_info *h2c; struct rtw89_vif_link *rtwvif_link; + struct rtw89_pkt_stat *pkt_stat; const struct rtw89_chan *chan; u8 bw_idx = RTW89_BB_BW_20_40; + struct rtw89_bb_ctx *bb; u32 len = sizeof(*h2c); unsigned int link_id; struct sk_buff *skb; @@ -3010,11 +3047,14 @@ int rtw89_fw_h2c_lps_ml_cmn_info(struct rtw89_dev *rtwdev, path = rtwvif_link->phy_idx == RTW89_PHY_1 ? RF_PATH_B : RF_PATH_A; chan = rtw89_chan_get(rtwdev, rtwvif_link->chanctx_idx); gain_band = rtw89_subband_to_gain_band_be(chan->subband_type); + bb = rtw89_get_bb_ctx(rtwdev, rtwvif_link->phy_idx); h2c->central_ch[rtwvif_link->phy_idx] = chan->channel; h2c->pri_ch[rtwvif_link->phy_idx] = chan->primary_channel; h2c->band[rtwvif_link->phy_idx] = chan->band_type; h2c->bw[rtwvif_link->phy_idx] = chan->band_width; + + pkt_stat = &bb->cur_pkt_stat; if (pkt_stat->beacon_rate < RTW89_HW_RATE_OFDM6) h2c->bcn_rate_type[rtwvif_link->phy_idx] = 0x1; else diff --git a/drivers/net/wireless/realtek/rtw89/fw.h b/drivers/net/wireless/realtek/rtw89/fw.h index 47e5cbec306d..98078a3ef89e 100644 --- a/drivers/net/wireless/realtek/rtw89/fw.h +++ b/drivers/net/wireless/realtek/rtw89/fw.h @@ -3989,6 +3989,7 @@ enum rtw89_fw_element_id { RTW89_FW_ELEMENT_ID_TXPWR_DA_LMT_RU_5GHZ = 25, RTW89_FW_ELEMENT_ID_TXPWR_DA_LMT_RU_6GHZ = 26, RTW89_FW_ELEMENT_ID_AFE_PWR_SEQ = 27, + RTW89_FW_ELEMENT_ID_DIAG_MAC = 28, RTW89_FW_ELEMENT_ID_NUM, }; @@ -4166,6 +4167,11 @@ struct rtw89_fw_element_hdr { __le32 val; } __packed infos[]; } __packed afe; + struct { + __le32 rule_size; + u8 rsvd[4]; + u8 rules_and_msgs[]; + } __packed diag_mac; struct __rtw89_fw_txpwr_element txpwr; struct __rtw89_fw_regd_element regd; } __packed u; @@ -4827,7 +4833,8 @@ int rtw89_fw_h2c_tbtt_tuning(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link, u32 offset); int rtw89_fw_h2c_pwr_lvl(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link); int rtw89_fw_h2c_cam(struct rtw89_dev *rtwdev, struct rtw89_vif_link *vif, - struct rtw89_sta_link *rtwsta_link, const u8 *scan_mac_addr); + struct rtw89_sta_link *rtwsta_link, const u8 *scan_mac_addr, + enum rtw89_upd_mode upd_mode); int rtw89_fw_h2c_dctl_sec_cam_v1(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_link, struct rtw89_sta_link *rtwsta_link); diff --git a/drivers/net/wireless/realtek/rtw89/mac.c b/drivers/net/wireless/realtek/rtw89/mac.c index 71194ea68bce..f0773f7c3749 100644 --- a/drivers/net/wireless/realtek/rtw89/mac.c +++ b/drivers/net/wireless/realtek/rtw89/mac.c @@ -4786,7 +4786,7 @@ int rtw89_mac_vif_init(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif_l if (ret) return ret; - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL, RTW89_ROLE_CREATE); if (ret) return ret; @@ -4811,7 +4811,7 @@ int rtw89_mac_vif_deinit(struct rtw89_dev *rtwdev, struct rtw89_vif_link *rtwvif rtw89_cam_deinit(rtwdev, rtwvif_link); - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL, RTW89_ROLE_REMOVE); if (ret) return ret; @@ -4885,7 +4885,7 @@ static void rtw89_mac_check_he_obss_narrow_bw_ru_iter(struct wiphy *wiphy, elem = cfg80211_find_elem(WLAN_EID_EXT_CAPABILITY, ies->data, ies->len); - if (!elem || elem->datalen < 10 || + if (!elem || elem->datalen < 11 || !(elem->data[10] & WLAN_EXT_CAPA10_OBSS_NARROW_BW_RU_TOLERANCE_SUPPORT)) *tolerated = false; rcu_read_unlock(); diff --git a/drivers/net/wireless/realtek/rtw89/mac.h b/drivers/net/wireless/realtek/rtw89/mac.h index 51e37c183a35..1ac1d0cd7959 100644 --- a/drivers/net/wireless/realtek/rtw89/mac.h +++ b/drivers/net/wireless/realtek/rtw89/mac.h @@ -574,8 +574,6 @@ enum rtw89_mac_bf_rrsc_rate { RTW89_MAC_BF_RRSC_MAX = 32 }; -#define RTW89_R32_EA 0xEAEAEAEA -#define RTW89_R32_DEAD 0xDEADBEEF #define MAC_REG_POOL_COUNT 10 #define ACCESS_CMAC(_addr) \ ({typeof(_addr) __addr = (_addr); \ diff --git a/drivers/net/wireless/realtek/rtw89/mac80211.c b/drivers/net/wireless/realtek/rtw89/mac80211.c index 474be7a5e49c..fd818d4b3531 100644 --- a/drivers/net/wireless/realtek/rtw89/mac80211.c +++ b/drivers/net/wireless/realtek/rtw89/mac80211.c @@ -745,7 +745,7 @@ static void rtw89_ops_link_info_changed(struct ieee80211_hw *hw, if (changed & BSS_CHANGED_BSSID) { ether_addr_copy(rtwvif_link->bssid, conf->bssid); rtw89_cam_bssid_changed(rtwdev, rtwvif_link); - rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL); + rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL, RTW89_ROLE_INFO_CHANGE); WRITE_ONCE(rtwvif_link->sync_bcn_tsf, 0); } @@ -800,11 +800,36 @@ static int rtw89_ops_start_ap(struct ieee80211_hw *hw, ether_addr_copy(rtwvif_link->bssid, link_conf->bssid); rtw89_cam_bssid_changed(rtwdev, rtwvif_link); - rtw89_mac_port_update(rtwdev, rtwvif_link); - rtw89_chip_h2c_assoc_cmac_tbl(rtwdev, rtwvif_link, NULL); - rtw89_fw_h2c_role_maintain(rtwdev, rtwvif_link, NULL, RTW89_ROLE_TYPE_CHANGE); - rtw89_fw_h2c_join_info(rtwdev, rtwvif_link, NULL, true); - rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL); + ret = rtw89_mac_port_update(rtwdev, rtwvif_link); + if (ret) { + rtw89_warn(rtwdev, "failed to update mac port\n"); + return ret; + } + + ret = rtw89_chip_h2c_assoc_cmac_tbl(rtwdev, rtwvif_link, NULL); + if (ret) { + rtw89_warn(rtwdev, "failed to send h2c cmac table\n"); + return ret; + } + + ret = rtw89_fw_h2c_role_maintain(rtwdev, rtwvif_link, NULL, RTW89_ROLE_TYPE_CHANGE); + if (ret) { + rtw89_warn(rtwdev, "failed to send h2c role info\n"); + return ret; + } + + ret = rtw89_fw_h2c_join_info(rtwdev, rtwvif_link, NULL, true); + if (ret) { + rtw89_warn(rtwdev, "failed to send h2c join info\n"); + return ret; + } + + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, NULL, NULL, RTW89_ROLE_TYPE_CHANGE); + if (ret) { + rtw89_warn(rtwdev, "failed to send h2c cam\n"); + return ret; + } + rtw89_chip_rfk_channel(rtwdev, rtwvif_link); if (RTW89_CHK_FW_FEATURE(NOTIFY_AP_INFO, &rtwdev->fw)) { @@ -1879,7 +1904,8 @@ static void rtw89_ops_rfkill_poll(struct ieee80211_hw *hw) lockdep_assert_wiphy(hw->wiphy); /* wl_disable GPIO get floating when entering LPS */ - if (test_bit(RTW89_FLAG_RUNNING, rtwdev->flags)) + if (test_bit(RTW89_FLAG_RUNNING, rtwdev->flags) || + test_bit(RTW89_FLAG_SHUTDOWN, rtwdev->flags)) return; rtw89_core_rfkill_poll(rtwdev, false); diff --git a/drivers/net/wireless/realtek/rtw89/pci.c b/drivers/net/wireless/realtek/rtw89/pci.c index 17a199ee1d4a..b24237c1e7df 100644 --- a/drivers/net/wireless/realtek/rtw89/pci.c +++ b/drivers/net/wireless/realtek/rtw89/pci.c @@ -2071,6 +2071,20 @@ static void rtw89_pci_ops_write32(struct rtw89_dev *rtwdev, u32 addr, u32 data) writel(data, rtwpci->mmap + addr); } +static u32 rtw89_pci_ops_read32_pci_cfg(struct rtw89_dev *rtwdev, u32 addr) +{ + struct rtw89_pci *rtwpci = (struct rtw89_pci *)rtwdev->priv; + struct pci_dev *pdev = rtwpci->pdev; + u32 value; + int ret; + + ret = pci_read_config_dword(pdev, addr, &value); + if (ret) + return RTW89_R32_EA; + + return value; +} + static void rtw89_pci_ctrl_dma_trx(struct rtw89_dev *rtwdev, bool enable) { const struct rtw89_pci_info *info = rtwdev->pci_info; @@ -4691,6 +4705,8 @@ static const struct rtw89_hci_ops rtw89_pci_ops = { .write16 = rtw89_pci_ops_write16, .write32 = rtw89_pci_ops_write32, + .read32_pci_cfg = rtw89_pci_ops_read32_pci_cfg, + .mac_pre_init = rtw89_pci_ops_mac_pre_init, .mac_pre_deinit = rtw89_pci_ops_mac_pre_deinit, .mac_post_init = rtw89_pci_ops_mac_post_init, @@ -4829,6 +4845,19 @@ void rtw89_pci_remove(struct pci_dev *pdev) } EXPORT_SYMBOL(rtw89_pci_remove); +void rtw89_pci_shutdown(struct pci_dev *pdev) +{ + struct ieee80211_hw *hw = pci_get_drvdata(pdev); + struct rtw89_dev *rtwdev; + + if (!hw) + return; + + rtwdev = hw->priv; + set_bit(RTW89_FLAG_SHUTDOWN, rtwdev->flags); +} +EXPORT_SYMBOL(rtw89_pci_shutdown); + MODULE_AUTHOR("Realtek Corporation"); MODULE_DESCRIPTION("Realtek PCI 802.11ax wireless driver"); MODULE_LICENSE("Dual BSD/GPL"); diff --git a/drivers/net/wireless/realtek/rtw89/pci.h b/drivers/net/wireless/realtek/rtw89/pci.h index cb05c83dfd56..09fca0e46cd1 100644 --- a/drivers/net/wireless/realtek/rtw89/pci.h +++ b/drivers/net/wireless/realtek/rtw89/pci.h @@ -1737,6 +1737,7 @@ struct pci_device_id; int rtw89_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id); void rtw89_pci_remove(struct pci_dev *pdev); +void rtw89_pci_shutdown(struct pci_dev *pdev); void rtw89_pci_basic_cfg(struct rtw89_dev *rtwdev, bool resume); void rtw89_pci_ops_reset(struct rtw89_dev *rtwdev); int rtw89_pci_ltr_set(struct rtw89_dev *rtwdev, bool en); diff --git a/drivers/net/wireless/realtek/rtw89/phy.c b/drivers/net/wireless/realtek/rtw89/phy.c index 36dee482bc34..4424dae1c7aa 100644 --- a/drivers/net/wireless/realtek/rtw89/phy.c +++ b/drivers/net/wireless/realtek/rtw89/phy.c @@ -5392,6 +5392,7 @@ static void rtw89_phy_stat_rssi_update(struct rtw89_dev *rtwdev) static void rtw89_phy_stat_init(struct rtw89_dev *rtwdev) { struct rtw89_phy_stat *phystat = &rtwdev->phystat; + struct rtw89_bb_ctx *bb; int i; for (i = 0; i < rtwdev->chip->rf_path_num; i++) @@ -5399,24 +5400,28 @@ static void rtw89_phy_stat_init(struct rtw89_dev *rtwdev) rtw89_phy_stat_thermal_update(rtwdev); - memset(&phystat->cur_pkt_stat, 0, sizeof(phystat->cur_pkt_stat)); - memset(&phystat->last_pkt_stat, 0, sizeof(phystat->last_pkt_stat)); + rtw89_for_each_capab_bb(rtwdev, bb) { + memset(&bb->cur_pkt_stat, 0, sizeof(bb->cur_pkt_stat)); + memset(&bb->last_pkt_stat, 0, sizeof(bb->last_pkt_stat)); - ewma_rssi_init(&phystat->bcn_rssi); + ewma_rssi_init(&bb->bcn_rssi); + } rtwdev->hal.thermal_prot_lv = 0; } void rtw89_phy_stat_track(struct rtw89_dev *rtwdev) { - struct rtw89_phy_stat *phystat = &rtwdev->phystat; + struct rtw89_bb_ctx *bb; rtw89_phy_stat_thermal_update(rtwdev); rtw89_phy_thermal_protect(rtwdev); rtw89_phy_stat_rssi_update(rtwdev); - phystat->last_pkt_stat = phystat->cur_pkt_stat; - memset(&phystat->cur_pkt_stat, 0, sizeof(phystat->cur_pkt_stat)); + rtw89_for_each_active_bb(rtwdev, bb) { + bb->last_pkt_stat = bb->cur_pkt_stat; + memset(&bb->cur_pkt_stat, 0, sizeof(bb->cur_pkt_stat)); + } } static u16 rtw89_phy_ccx_us_to_idx(struct rtw89_dev *rtwdev, diff --git a/drivers/net/wireless/realtek/rtw89/rtw8851be.c b/drivers/net/wireless/realtek/rtw89/rtw8851be.c index ce59ac9f56ba..79ea3ccb7957 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8851be.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8851be.c @@ -92,6 +92,7 @@ static struct pci_driver rtw89_8851be_driver = { .id_table = rtw89_8851be_id_table, .probe = rtw89_pci_probe, .remove = rtw89_pci_remove, + .shutdown = rtw89_pci_shutdown, .driver.pm = &rtw89_pm_ops, .err_handler = &rtw89_pci_err_handler, }; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852a.c b/drivers/net/wireless/realtek/rtw89/rtw8852a.c index 232f4c1bee1b..3defb38223e8 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852a.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852a.c @@ -2107,13 +2107,14 @@ static void rtw8852a_query_ppdu(struct rtw89_dev *rtwdev, struct rtw89_rx_phy_ppdu *phy_ppdu, struct ieee80211_rx_status *status) { - u8 path; + struct rtw89_bb_ctx *bb = rtw89_get_bb_ctx(rtwdev, phy_ppdu->phy_idx); u8 *rx_power = phy_ppdu->rssi; + u8 path; u8 raw; if (!status->signal) { - if (phy_ppdu->to_self) - raw = ewma_rssi_read(&rtwdev->phystat.bcn_rssi); + if (phy_ppdu->to_self && ewma_rssi_read(&bb->bcn_rssi)) + raw = ewma_rssi_read(&bb->bcn_rssi); else raw = max(rx_power[RF_PATH_A], rx_power[RF_PATH_B]); diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852ae.c b/drivers/net/wireless/realtek/rtw89/rtw8852ae.c index 9e05e831569d..4f1624740dd2 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852ae.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852ae.c @@ -94,6 +94,7 @@ static struct pci_driver rtw89_8852ae_driver = { .id_table = rtw89_8852ae_id_table, .probe = rtw89_pci_probe, .remove = rtw89_pci_remove, + .shutdown = rtw89_pci_shutdown, .driver.pm = &rtw89_pm_ops, .err_handler = &rtw89_pci_err_handler, }; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852be.c b/drivers/net/wireless/realtek/rtw89/rtw8852be.c index 12db0d0be547..72bbaf22893f 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852be.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852be.c @@ -96,6 +96,7 @@ static struct pci_driver rtw89_8852be_driver = { .id_table = rtw89_8852be_id_table, .probe = rtw89_pci_probe, .remove = rtw89_pci_remove, + .shutdown = rtw89_pci_shutdown, .driver.pm = &rtw89_pm_ops, .err_handler = &rtw89_pci_err_handler, }; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852bte.c b/drivers/net/wireless/realtek/rtw89/rtw8852bte.c index 8c995aa95325..b10587fa5f32 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852bte.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852bte.c @@ -98,6 +98,7 @@ static struct pci_driver rtw89_8852bte_driver = { .id_table = rtw89_8852bte_id_table, .probe = rtw89_pci_probe, .remove = rtw89_pci_remove, + .shutdown = rtw89_pci_shutdown, .driver.pm = &rtw89_pm_ops, .err_handler = &rtw89_pci_err_handler, }; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852ce.c b/drivers/net/wireless/realtek/rtw89/rtw8852ce.c index 150fed189414..281f46455de4 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8852ce.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8852ce.c @@ -121,6 +121,7 @@ static struct pci_driver rtw89_8852ce_driver = { .id_table = rtw89_8852ce_id_table, .probe = rtw89_pci_probe, .remove = rtw89_pci_remove, + .shutdown = rtw89_pci_shutdown, .driver.pm = &rtw89_pm_ops, .err_handler = &rtw89_pci_err_handler, }; diff --git a/drivers/net/wireless/realtek/rtw89/rtw8922ae.c b/drivers/net/wireless/realtek/rtw89/rtw8922ae.c index 90c62b757c57..1ad7d44d0352 100644 --- a/drivers/net/wireless/realtek/rtw89/rtw8922ae.c +++ b/drivers/net/wireless/realtek/rtw89/rtw8922ae.c @@ -109,6 +109,7 @@ static struct pci_driver rtw89_8922ae_driver = { .id_table = rtw89_8922ae_id_table, .probe = rtw89_pci_probe, .remove = rtw89_pci_remove, + .shutdown = rtw89_pci_shutdown, .driver.pm = &rtw89_pm_ops_be, .err_handler = &rtw89_pci_err_handler, }; diff --git a/drivers/net/wireless/realtek/rtw89/wow.c b/drivers/net/wireless/realtek/rtw89/wow.c index f34cd863d100..f6cdbc15dcc4 100644 --- a/drivers/net/wireless/realtek/rtw89/wow.c +++ b/drivers/net/wireless/realtek/rtw89/wow.c @@ -1225,7 +1225,8 @@ static int rtw89_wow_cfg_wake(struct rtw89_dev *rtwdev, bool wow) } } - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL, + RTW89_ROLE_INFO_CHANGE); if (ret) { rtw89_warn(rtwdev, "failed to send h2c cam\n"); return ret; @@ -1322,7 +1323,8 @@ static int rtw89_wow_swap_fw(struct rtw89_dev *rtwdev, bool wow) return ret; } - ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL); + ret = rtw89_fw_h2c_cam(rtwdev, rtwvif_link, rtwsta_link, NULL, + RTW89_ROLE_FW_RESTORE); if (ret) { rtw89_warn(rtwdev, "failed to send h2c cam\n"); return ret; diff --git a/drivers/net/wireless/virtual/mac80211_hwsim.c b/drivers/net/wireless/virtual/mac80211_hwsim.c index a0724e1c5307..3d66f4aa8393 100644 --- a/drivers/net/wireless/virtual/mac80211_hwsim.c +++ b/drivers/net/wireless/virtual/mac80211_hwsim.c @@ -2159,6 +2159,7 @@ static int mac80211_hwsim_start(struct ieee80211_hw *hw) static void mac80211_hwsim_stop(struct ieee80211_hw *hw, bool suspend) { struct mac80211_hwsim_data *data = hw->priv; + struct sk_buff *skb; int i; data->started = false; @@ -2166,8 +2167,8 @@ static void mac80211_hwsim_stop(struct ieee80211_hw *hw, bool suspend) for (i = 0; i < ARRAY_SIZE(data->link_data); i++) hrtimer_cancel(&data->link_data[i].beacon_timer); - while (!skb_queue_empty(&data->pending)) - ieee80211_free_txskb(hw, skb_dequeue(&data->pending)); + while ((skb = skb_dequeue(&data->pending))) + ieee80211_free_txskb(hw, skb); wiphy_dbg(hw->wiphy, "%s\n", __func__); } diff --git a/drivers/net/wireless/zydas/zd1211rw/zd_usb.c b/drivers/net/wireless/zydas/zd1211rw/zd_usb.c index 8ee15a15f4ca..ebf4e29c566c 100644 --- a/drivers/net/wireless/zydas/zd1211rw/zd_usb.c +++ b/drivers/net/wireless/zydas/zd1211rw/zd_usb.c @@ -1353,6 +1353,14 @@ static int probe(struct usb_interface *intf, const struct usb_device_id *id) struct zd_usb *usb; struct ieee80211_hw *hw = NULL; + /* + * ZD1211 devices are single-function. Reject secondary interfaces + * to prevent multiple instances from conflicting on hardcoded endpoints + * and triggering recursive locking warnings. + */ + if (intf->cur_altsetting->desc.bInterfaceNumber != 0) + return -ENODEV; + print_id(udev); if (id->driver_info & DEVICE_INSTALLER) diff --git a/drivers/nfc/pn533/pn533.c b/drivers/nfc/pn533/pn533.c index 663cf494fea9..9db9e559aeae 100644 --- a/drivers/nfc/pn533/pn533.c +++ b/drivers/nfc/pn533/pn533.c @@ -434,6 +434,18 @@ static int pn533_send_async_complete(struct pn533 *dev) return rc; } +static int pn533_send_cmd_frame(struct pn533 *dev, struct pn533_cmd *cmd) +{ + struct sk_buff *req = cmd->req; + int rc; + + skb_get(req); + dev->cmd = cmd; + rc = dev->phy_ops->send_frame(dev, req); + dev_kfree_skb(req); + return rc; +} + static int __pn533_send_async(struct pn533 *dev, u8 cmd_code, struct sk_buff *req, pn533_send_async_complete_t complete_cb, @@ -458,8 +470,7 @@ static int __pn533_send_async(struct pn533 *dev, u8 cmd_code, mutex_lock(&dev->cmd_lock); if (!dev->cmd_pending) { - dev->cmd = cmd; - rc = dev->phy_ops->send_frame(dev, req); + rc = pn533_send_cmd_frame(dev, cmd); if (rc) { dev->cmd = NULL; goto error; @@ -529,8 +540,7 @@ static int pn533_send_cmd_direct_async(struct pn533 *dev, u8 cmd_code, pn533_build_cmd_frame(dev, cmd_code, req); - dev->cmd = cmd; - rc = dev->phy_ops->send_frame(dev, req); + rc = pn533_send_cmd_frame(dev, cmd); if (rc < 0) { dev->cmd = NULL; kfree(cmd); @@ -569,8 +579,7 @@ static void pn533_wq_cmd(struct work_struct *work) mutex_unlock(&dev->cmd_lock); - dev->cmd = cmd; - rc = dev->phy_ops->send_frame(dev, cmd->req); + rc = pn533_send_cmd_frame(dev, cmd); if (rc < 0) { dev->cmd = NULL; dev_kfree_skb(cmd->req); diff --git a/drivers/nvdimm/label.c b/drivers/nvdimm/label.c index 584a9c27ce4b..93ff219d91fc 100644 --- a/drivers/nvdimm/label.c +++ b/drivers/nvdimm/label.c @@ -145,10 +145,21 @@ static int __nd_label_validate(struct nvdimm_drvdata *ndd) /* label sizes larger than 128 arrived with v1.2 */ version = __le16_to_cpu(nsindex[i]->major) * 100 + __le16_to_cpu(nsindex[i]->minor); - if (version >= 102) + if (version >= 102) { + /* + * labelsize feeds the shift below; only 0 (128-byte) + * and 1 (256-byte) are valid -- a larger value would + * overflow or exceed the width of int. + */ + if (nsindex[i]->labelsize > 1) { + dev_dbg(dev, "nsindex%d labelsize: %d invalid\n", + i, nsindex[i]->labelsize); + continue; + } labelsize = 1 << (7 + nsindex[i]->labelsize); - else + } else { labelsize = 128; + } if (labelsize != sizeof_namespace_label(ndd)) { dev_dbg(dev, "nsindex%d labelsize %d invalid\n", diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c index af82385be7c6..942ded0a27be 100644 --- a/drivers/nvdimm/nd_virtio.c +++ b/drivers/nvdimm/nd_virtio.c @@ -9,26 +9,48 @@ #include "virtio_pmem.h" #include "nd.h" +struct virtio_pmem_flush_work { + struct work_struct work; + struct nd_region *nd_region; + struct bio *bio; +}; + +static void virtio_pmem_req_release(struct kref *kref) +{ + struct virtio_pmem_request *req; + + req = container_of(kref, struct virtio_pmem_request, kref); + kfree(req); +} + +static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem) +{ + struct virtio_pmem_request *req_buf; + + if (list_empty(&vpmem->req_list)) + return; + + req_buf = list_first_entry(&vpmem->req_list, + struct virtio_pmem_request, list); + list_del_init(&req_buf->list); + WRITE_ONCE(req_buf->wq_buf_avail, true); + wake_up(&req_buf->wq_buf); +} + /* The interrupt handler */ void virtio_pmem_host_ack(struct virtqueue *vq) { struct virtio_pmem *vpmem = vq->vdev->priv; - struct virtio_pmem_request *req_data, *req_buf; + struct virtio_pmem_request *req_data; unsigned long flags; unsigned int len; spin_lock_irqsave(&vpmem->pmem_lock, flags); while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) { - req_data->done = true; + virtio_pmem_wake_one_waiter(vpmem); + WRITE_ONCE(req_data->done, true); wake_up(&req_data->host_acked); - - if (!list_empty(&vpmem->req_list)) { - req_buf = list_first_entry(&vpmem->req_list, - struct virtio_pmem_request, list); - req_buf->wq_buf_avail = true; - wake_up(&req_buf->wq_buf); - list_del(&req_buf->list); - } + kref_put(&req_data->kref, virtio_pmem_req_release); } spin_unlock_irqrestore(&vpmem->pmem_lock, flags); } @@ -59,7 +81,8 @@ static int virtio_pmem_flush(struct nd_region *nd_region) if (!req_data) return -ENOMEM; - req_data->done = false; + kref_init(&req_data->kref); + WRITE_ONCE(req_data->done, false); init_waitqueue_head(&req_data->host_acked); init_waitqueue_head(&req_data->wq_buf); INIT_LIST_HEAD(&req_data->list); @@ -76,18 +99,32 @@ static int virtio_pmem_flush(struct nd_region *nd_region) * to req_list and wait for host_ack to wake us up when free * slots are available. */ - while ((err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data, - GFP_ATOMIC)) == -ENOSPC) { + for (;;) { + err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data, + GFP_ATOMIC); + if (!err) { + /* + * Take the virtqueue reference while @pmem_lock is + * held so completion cannot run concurrently. + */ + kref_get(&req_data->kref); + break; + } - dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n"); - req_data->wq_buf_avail = false; + if (err != -ENOSPC) + break; + + dev_info_ratelimited(&vdev->dev, + "failed to send command to virtio pmem device, no free slots in the virtqueue\n"); + WRITE_ONCE(req_data->wq_buf_avail, false); list_add_tail(&req_data->list, &vpmem->req_list); spin_unlock_irqrestore(&vpmem->pmem_lock, flags); /* A host response results in "host_ack" getting called */ - wait_event(req_data->wq_buf, req_data->wq_buf_avail); + wait_event(req_data->wq_buf, READ_ONCE(req_data->wq_buf_avail)); spin_lock_irqsave(&vpmem->pmem_lock, flags); } + err1 = virtqueue_kick(vpmem->req_vq); spin_unlock_irqrestore(&vpmem->pmem_lock, flags); /* @@ -99,38 +136,54 @@ static int virtio_pmem_flush(struct nd_region *nd_region) err = -EIO; } else { /* A host response results in "host_ack" getting called */ - wait_event(req_data->host_acked, req_data->done); + wait_event(req_data->host_acked, READ_ONCE(req_data->done)); err = le32_to_cpu(req_data->resp.ret); } - kfree(req_data); + kref_put(&req_data->kref, virtio_pmem_req_release); return err; }; +static void virtio_pmem_flush_work(struct work_struct *work) +{ + struct virtio_pmem_flush_work *flush; + int err; + + flush = container_of(work, struct virtio_pmem_flush_work, work); + err = virtio_pmem_flush(flush->nd_region); + if (err > 0) + err = -EIO; + if (err) + flush->bio->bi_status = errno_to_blk_status(err); + bio_endio(flush->bio); + kfree(flush); +} + /* The asynchronous flush callback function */ int async_pmem_flush(struct nd_region *nd_region, struct bio *bio) { - /* - * Create child bio for asynchronous flush and chain with - * parent bio. Otherwise directly call nd_region flush. - */ - if (bio && bio->bi_iter.bi_sector != -1) { - struct bio *child = bio_alloc(bio->bi_bdev, 0, - REQ_OP_WRITE | REQ_PREFLUSH, - GFP_ATOMIC); + struct virtio_device *vdev = nd_region->provider_data; + struct virtio_pmem *vpmem = vdev->priv; + struct virtio_pmem_flush_work *flush; + int err; - if (!child) + if (bio && bio->bi_iter.bi_sector != -1) { + flush = kmalloc_obj(*flush, GFP_NOIO); + if (!flush) return -ENOMEM; - bio_clone_blkg_association(child, bio); - child->bi_iter.bi_sector = -1; - bio_chain(child, bio); - submit_bio(child); - return 0; + + INIT_WORK(&flush->work, virtio_pmem_flush_work); + flush->nd_region = nd_region; + flush->bio = bio; + queue_work(vpmem->flush_wq, &flush->work); + return NVDIMM_FLUSH_ASYNC; } - if (virtio_pmem_flush(nd_region)) + + err = virtio_pmem_flush(nd_region); + if (err > 0) return -EIO; - return 0; + return err; }; EXPORT_SYMBOL_GPL(async_pmem_flush); MODULE_DESCRIPTION("Virtio Persistent Memory Driver"); diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 05785ff21a8b..614701a539ab 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -208,8 +208,14 @@ static void pmem_submit_bio(struct bio *bio) struct pmem_device *pmem = bio->bi_bdev->bd_disk->private_data; struct nd_region *nd_region = to_region(pmem); - if (bio->bi_opf & REQ_PREFLUSH) - ret = nvdimm_flush(nd_region, bio); + if (bio->bi_opf & REQ_PREFLUSH) { + ret = nvdimm_flush(nd_region, NULL); + if (ret) { + bio->bi_status = errno_to_blk_status(ret); + bio_endio(bio); + return; + } + } do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue); if (do_acct) @@ -229,8 +235,11 @@ static void pmem_submit_bio(struct bio *bio) if (do_acct) bio_end_io_acct(bio, start); - if (bio->bi_opf & REQ_FUA) + if ((bio->bi_opf & REQ_FUA) && !bio->bi_status) { ret = nvdimm_flush(nd_region, bio); + if (ret == NVDIMM_FLUSH_ASYNC) + return; + } if (ret) bio->bi_status = errno_to_blk_status(ret); diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c index 5a1c68c701cf..bb0ee554097b 100644 --- a/drivers/nvdimm/region_devs.c +++ b/drivers/nvdimm/region_devs.c @@ -1094,7 +1094,10 @@ int nvdimm_flush(struct nd_region *nd_region, struct bio *bio) if (!nd_region->flush) rc = generic_nvdimm_flush(nd_region); else { - if (nd_region->flush(nd_region, bio)) + rc = nd_region->flush(nd_region, bio); + if (rc > 0) + return rc; + if (rc && rc != -ENOMEM) rc = -EIO; } diff --git a/drivers/nvdimm/virtio_pmem.c b/drivers/nvdimm/virtio_pmem.c index 77b196661905..9cf822a6c0c3 100644 --- a/drivers/nvdimm/virtio_pmem.c +++ b/drivers/nvdimm/virtio_pmem.c @@ -67,10 +67,17 @@ static int virtio_pmem_probe(struct virtio_device *vdev) mutex_init(&vpmem->flush_lock); vpmem->vdev = vdev; vdev->priv = vpmem; + vpmem->flush_wq = alloc_ordered_workqueue("virtio-pmem-flush", + WQ_MEM_RECLAIM); + if (!vpmem->flush_wq) { + err = -ENOMEM; + goto out_err; + } + err = init_vq(vpmem); if (err) { dev_err(&vdev->dev, "failed to initialize virtio pmem vq's\n"); - goto out_err; + goto out_wq; } if (virtio_has_feature(vdev, VIRTIO_PMEM_F_SHMEM_REGION)) { @@ -131,6 +138,8 @@ static int virtio_pmem_probe(struct virtio_device *vdev) nvdimm_bus_unregister(vpmem->nvdimm_bus); out_vq: vdev->config->del_vqs(vdev); +out_wq: + destroy_workqueue(vpmem->flush_wq); out_err: return err; } @@ -138,14 +147,20 @@ static int virtio_pmem_probe(struct virtio_device *vdev) static void virtio_pmem_remove(struct virtio_device *vdev) { struct nvdimm_bus *nvdimm_bus = dev_get_drvdata(&vdev->dev); + struct virtio_pmem *vpmem = vdev->priv; nvdimm_bus_unregister(nvdimm_bus); + drain_workqueue(vpmem->flush_wq); vdev->config->del_vqs(vdev); virtio_reset_device(vdev); + destroy_workqueue(vpmem->flush_wq); } static int virtio_pmem_freeze(struct virtio_device *vdev) { + struct virtio_pmem *vpmem = vdev->priv; + + drain_workqueue(vpmem->flush_wq); vdev->config->del_vqs(vdev); virtio_reset_device(vdev); diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h index f72cf17f9518..3af92588bd9d 100644 --- a/drivers/nvdimm/virtio_pmem.h +++ b/drivers/nvdimm/virtio_pmem.h @@ -12,11 +12,14 @@ #include #include +#include #include #include #include +#include struct virtio_pmem_request { + struct kref kref; struct virtio_pmem_req req; struct virtio_pmem_resp resp; @@ -39,6 +42,9 @@ struct virtio_pmem { /* Serialize flush requests to the device. */ struct mutex flush_lock; + /* Complete asynchronous FUA flushes outside the submit path. */ + struct workqueue_struct *flush_wq; + /* nvdimm bus registers virtio pmem device */ struct nvdimm_bus *nvdimm_bus; struct nvdimm_bus_descriptor nd_desc; diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index 9cc614227bc7..e3cd538d950a 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -47,9 +47,6 @@ #define APPLE_ANS_BOOT_STATUS 0x1300 #define APPLE_ANS_BOOT_STATUS_OK 0xde71ce55 -#define APPLE_ANS_UNKNOWN_CTRL 0x24008 -#define APPLE_ANS_PRP_NULL_CHECK BIT(11) - #define APPLE_ANS_LINEAR_SQ_CTRL 0x24908 #define APPLE_ANS_LINEAR_SQ_EN BIT(0) @@ -151,6 +148,23 @@ struct apple_nvme_queue { bool enabled; }; +static inline bool apple_nvme_queue_enabled(struct apple_nvme_queue *q) +{ + /* Pair with apple_nvme_enable_queue(). */ + return smp_load_acquire(&q->enabled); +} + +static inline void apple_nvme_enable_queue(struct apple_nvme_queue *q) +{ + /* Publish queue initialization before setting q->enabled. */ + smp_store_release(&q->enabled, true); +} + +static inline void apple_nvme_disable_queue(struct apple_nvme_queue *q) +{ + WRITE_ONCE(q->enabled, false); +} + /* * The apple_nvme_iod describes the data in an I/O. * @@ -318,13 +332,15 @@ static void apple_nvme_submit_cmd_t8103(struct apple_nvme_queue *q, u32 tag = nvme_tag_from_cid(cmd->common.command_id); struct apple_nvmmu_tcb *tcb = &q->tcbs[tag]; - tcb->opcode = cmd->common.opcode; + tcb->opcode = 0; tcb->prp1 = cmd->common.dptr.prp1; tcb->prp2 = cmd->common.dptr.prp2; tcb->length = cmd->rw.length; tcb->command_id = tag; - if (nvme_is_write(cmd)) + if (!cmd->common.dptr.prp1) + tcb->dma_flags = 0; + else if (nvme_is_write(cmd)) tcb->dma_flags = APPLE_ANS_TCB_DMA_TO_DEVICE; else tcb->dma_flags = APPLE_ANS_TCB_DMA_FROM_DEVICE; @@ -677,7 +693,7 @@ static bool apple_nvme_handle_cq(struct apple_nvme_queue *q, bool force) bool found; DEFINE_IO_COMP_BATCH(iob); - if (!READ_ONCE(q->enabled) && !force) + if (!apple_nvme_queue_enabled(q) && !force) return false; found = apple_nvme_poll_cq(q, &iob); @@ -780,7 +796,7 @@ static blk_status_t apple_nvme_queue_rq(struct blk_mq_hw_ctx *hctx, * We should not need to do this, but we're still using this to * ensure we can drain requests on a dying queue. */ - if (unlikely(!READ_ONCE(q->enabled))) + if (unlikely(!apple_nvme_queue_enabled(q))) return BLK_STS_IOERR; if (!nvme_check_ready(&anv->ctrl, req, true)) @@ -863,7 +879,7 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown) nvme_quiesce_io_queues(&anv->ctrl); if (!dead) { - if (READ_ONCE(anv->ioq.enabled)) { + if (apple_nvme_queue_enabled(&anv->ioq)) { apple_nvme_remove_sq(anv); apple_nvme_remove_cq(anv); } @@ -887,8 +903,8 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown) nvme_disable_ctrl(&anv->ctrl, false); } - WRITE_ONCE(anv->ioq.enabled, false); - WRITE_ONCE(anv->adminq.enabled, false); + apple_nvme_disable_queue(&anv->ioq); + apple_nvme_disable_queue(&anv->adminq); mb(); /* ensure that nvme_queue_rq() sees that enabled is cleared */ nvme_quiesce_admin_queue(&anv->ctrl); @@ -1016,8 +1032,7 @@ static void apple_nvme_init_queue(struct apple_nvme_queue *q) memset(q->tcbs, 0, anv->hw->max_queue_depth * sizeof(struct apple_nvmmu_tcb)); memset(q->cqes, 0, depth * sizeof(struct nvme_completion)); - WRITE_ONCE(q->enabled, true); - wmb(); /* ensure the first interrupt sees the initialization */ + apple_nvme_enable_queue(q); } static void apple_nvme_reset_work(struct work_struct *work) @@ -1125,17 +1140,6 @@ static void apple_nvme_reset_work(struct work_struct *work) /* Setup the NVMMU for the maximum admin and IO queue depth */ writel(anv->hw->max_queue_depth - 1, anv->mmio_nvme + APPLE_NVMMU_NUM_TCBS); - - /* - * This is probably a chicken bit: without it all commands - * where any PRP is set to zero (including those that don't use - * that field) fail and the co-processor complains about - * "completed with err BAD_CMD-" or a "NULL_PRP_PTR_ERR" in the - * syslog - */ - writel(readl(anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL) & - ~APPLE_ANS_PRP_NULL_CHECK, - anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL); } /* Setup the admin queue */ @@ -1277,6 +1281,7 @@ static const struct nvme_ctrl_ops nvme_ctrl_ops = { .reg_read64 = apple_nvme_reg_read64, .free_ctrl = apple_nvme_free_ctrl, .get_address = apple_nvme_get_address, + .get_virt_boundary = nvme_get_virt_boundary, }; static void apple_nvme_async_probe(void *data, async_cookie_t cookie) @@ -1580,7 +1585,8 @@ static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev) } ret = nvme_init_ctrl(&anv->ctrl, anv->dev, &nvme_ctrl_ops, - NVME_QUIRK_SKIP_CID_GEN | NVME_QUIRK_IDENTIFY_CNS); + NVME_QUIRK_SKIP_CID_GEN | NVME_QUIRK_IDENTIFY_CNS | + NVME_QUIRK_ADMIN_PAGE_ALIGN); if (ret) { dev_err_probe(dev, ret, "Failed to initialize nvme_ctrl"); goto put_dev; @@ -1635,6 +1641,15 @@ static void apple_nvme_remove(struct platform_device *pdev) nvme_stop_ctrl(&anv->ctrl); nvme_remove_namespaces(&anv->ctrl); apple_nvme_disable(anv, true); + if (anv->ctrl.admin_q && !blk_queue_dying(anv->ctrl.admin_q)) { + /* + * If the controller was reset during removal, it's possible + * user requests may be waiting on a stopped queue. Start the + * queue to flush these to completion. + */ + nvme_unquiesce_admin_queue(&anv->ctrl); + blk_mq_destroy_queue(anv->ctrl.admin_q); + } nvme_uninit_ctrl(&anv->ctrl); if (apple_rtkit_is_running(anv->rtk)) { diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 8f04f863a42d..dcf729aacb1f 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -1882,26 +1882,6 @@ static bool nvme_init_integrity(struct nvme_ns_head *head, return true; } -static void nvme_config_discard(struct nvme_ns *ns, struct queue_limits *lim) -{ - struct nvme_ctrl *ctrl = ns->ctrl; - - if (ctrl->dmrsl && ctrl->dmrsl <= nvme_sect_to_lba(ns->head, UINT_MAX)) - lim->max_hw_discard_sectors = - nvme_lba_to_sect(ns->head, ctrl->dmrsl); - else if (ctrl->oncs & NVME_CTRL_ONCS_DSM) - lim->max_hw_discard_sectors = UINT_MAX; - else - lim->max_hw_discard_sectors = 0; - - lim->discard_granularity = lim->logical_block_size; - - if (ctrl->dmrl) - lim->max_discard_segments = ctrl->dmrl; - else - lim->max_discard_segments = NVME_DSM_MAX_RANGES; -} - static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -2069,21 +2049,25 @@ static u32 nvme_max_drv_segments(struct nvme_ctrl *ctrl) } static void nvme_set_ctrl_limits(struct nvme_ctrl *ctrl, - struct queue_limits *lim) + struct queue_limits *lim, bool is_admin) { lim->max_hw_sectors = ctrl->max_hw_sectors; lim->max_segments = min_t(u32, USHRT_MAX, min_not_zero(nvme_max_drv_segments(ctrl), ctrl->max_segments)); lim->max_integrity_segments = ctrl->max_integrity_segments; - lim->virt_boundary_mask = NVME_CTRL_PAGE_SIZE - 1; + lim->virt_boundary_mask = ctrl->ops->get_virt_boundary(ctrl, is_admin); lim->max_segment_size = UINT_MAX; - lim->dma_alignment = 3; + if (is_admin && (ctrl->quirks & NVME_QUIRK_ADMIN_PAGE_ALIGN)) + lim->dma_alignment = NVME_CTRL_PAGE_SIZE - 1; + else + lim->dma_alignment = 3; } static bool nvme_update_disk_info(struct nvme_ns *ns, struct nvme_id_ns *id, struct queue_limits *lim) { struct nvme_ns_head *head = ns->head; + struct nvme_ctrl *ctrl = ns->ctrl; u32 bs = 1U << head->lba_shift; u32 atomic_bs, phys_bs, io_opt = 0; bool valid = true; @@ -2118,11 +2102,26 @@ static bool nvme_update_disk_info(struct nvme_ns *ns, struct nvme_id_ns *id, lim->physical_block_size = min(phys_bs, atomic_bs); lim->io_min = phys_bs; lim->io_opt = io_opt; - if ((ns->ctrl->quirks & NVME_QUIRK_DEALLOCATE_ZEROES) && - (ns->ctrl->oncs & NVME_CTRL_ONCS_DSM)) + if ((ctrl->quirks & NVME_QUIRK_DEALLOCATE_ZEROES) && + (ctrl->oncs & NVME_CTRL_ONCS_DSM)) lim->max_write_zeroes_sectors = UINT_MAX; else - lim->max_write_zeroes_sectors = ns->ctrl->max_zeroes_sectors; + lim->max_write_zeroes_sectors = ctrl->max_zeroes_sectors; + + if (ctrl->dmrsl && ctrl->dmrsl <= nvme_sect_to_lba(ns->head, UINT_MAX)) + lim->max_hw_discard_sectors = + nvme_lba_to_sect(ns->head, ctrl->dmrsl); + else if (ctrl->oncs & NVME_CTRL_ONCS_DSM) + lim->max_hw_discard_sectors = UINT_MAX; + else + lim->max_hw_discard_sectors = 0; + + lim->discard_granularity = lim->logical_block_size; + + if (ctrl->dmrl) + lim->max_discard_segments = ctrl->dmrl; + else + lim->max_discard_segments = NVME_DSM_MAX_RANGES; return valid; } @@ -2177,7 +2176,7 @@ static int nvme_update_ns_info_generic(struct nvme_ns *ns, int ret; lim = queue_limits_start_update(ns->disk->queue); - nvme_set_ctrl_limits(ns->ctrl, &lim); + nvme_set_ctrl_limits(ns->ctrl, &lim, false); memflags = blk_mq_freeze_queue(ns->disk->queue); ret = queue_limits_commit_update(ns->disk->queue, &lim); @@ -2381,16 +2380,32 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns, ns->head->lba_shift = id->lbaf[lbaf].ds; ns->head->nuse = le64_to_cpu(id->nuse); capacity = nvme_lba_to_sect(ns->head, le64_to_cpu(id->nsze)); - nvme_set_ctrl_limits(ns->ctrl, &lim); + nvme_set_ctrl_limits(ns->ctrl, &lim, false); nvme_configure_metadata(ns->ctrl, ns->head, id, nvm, info); nvme_set_chunk_sectors(ns, id, &lim); if (!nvme_update_disk_info(ns, id, &lim)) capacity = 0; - nvme_config_discard(ns, &lim); + /* + * A failed zone info query leaves zi zero-initialized, so skip the + * zoned limits update instead of configuring the queue from it. + * During a revalidation that keeps the zone geometry the queue was + * last validated with; on a first scan the namespace is registered + * without zoned limits, so that it is still available as a handle + * for admin commands. + */ if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) && - ns->head->ids.csi == NVME_CSI_ZNS) - nvme_update_zone_info(ns, &lim, &zi); + ns->head->ids.csi == NVME_CSI_ZNS) { + if (zi.zone_size) + nvme_update_zone_info(ns, &lim, &zi); + else + dev_warn(ns->ctrl->device, + "zone info query failed for nsid %u, %s\n", + ns->head->ns_id, + blk_queue_is_zoned(ns->disk->queue) ? + "keeping the previous zone limits" : + "not enabling zoned mode"); + } if ((ns->ctrl->vwc & NVME_CTRL_VWC_PRESENT) && !info->no_vwc) lim.features |= BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA; @@ -3589,7 +3604,7 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl) min_not_zero(ctrl->max_hw_sectors, max_hw_sectors); lim = queue_limits_start_update(ctrl->admin_q); - nvme_set_ctrl_limits(ctrl, &lim); + nvme_set_ctrl_limits(ctrl, &lim, true); ret = queue_limits_commit_update(ctrl->admin_q, &lim); if (ret) goto out_free; diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h index 1b58ee7d0dce..caf5503d0833 100644 --- a/drivers/nvme/host/fabrics.h +++ b/drivers/nvme/host/fabrics.h @@ -217,6 +217,12 @@ static inline unsigned int nvmf_nr_io_queues(struct nvmf_ctrl_options *opts) min(opts->nr_poll_queues, num_online_cpus()); } +static inline unsigned long nvmf_get_virt_boundary(struct nvme_ctrl *ctrl, + bool is_admin) +{ + return 0; +} + int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val); int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val); int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val); diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index bdfbd5701ba6..0c4df5bc7831 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2098,9 +2098,15 @@ __nvme_fc_init_request(struct nvme_fc_ctrl *ctrl, dev_err(ctrl->dev, "FCP Op failed - rspiu dma mapping failed.\n"); ret = -EFAULT; + goto out_unmap; } atomic_set(&op->state, FCPOP_STATE_IDLE); + return 0; + +out_unmap: + fc_dma_unmap_single(ctrl->lport->dev, op->fcp_req.cmddma, + sizeof(op->cmd_iu), DMA_TO_DEVICE); out_on_error: return ret; } @@ -3369,6 +3375,7 @@ static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = { .submit_async_event = nvme_fc_submit_async_event, .delete_ctrl = nvme_fc_delete_ctrl, .get_address = nvmf_get_address, + .get_virt_boundary = nvmf_get_virt_boundary, }; static void diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 5bbaf257fd6c..fb8533127116 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -14,45 +14,54 @@ enum { NVME_IOCTL_PARTITION = (1 << 1), }; -static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, - unsigned int flags, bool open_for_write) +static bool nvme_admin_cmd_allowed(struct nvme_ctrl *ctrl, + struct nvme_command *c) { - u32 effects; - - /* - * Do not allow unprivileged passthrough on partitions, as that allows an - * escape from the containment of the partition. - */ - if (flags & NVME_IOCTL_PARTITION) - goto admin; - - /* - * Do not allow unprivileged processes to send vendor specific or fabrics - * commands as we can't be sure about their effects. - */ - if (c->common.opcode >= nvme_cmd_vendor_start || - c->common.opcode == nvme_fabrics_command) - goto admin; - /* * Do not allow unprivileged passthrough of admin commands except * for a subset of identify commands that contain information required * to form proper I/O commands in userspace and do not expose any * potentially sensitive information. */ - if (!ns) { - if (c->common.opcode == nvme_admin_identify) { - switch (c->identify.cns) { - case NVME_ID_CNS_NS: - case NVME_ID_CNS_CS_NS: - case NVME_ID_CNS_NS_CS_INDEP: - case NVME_ID_CNS_CS_CTRL: - case NVME_ID_CNS_CTRL: - return true; - } + switch (c->common.opcode) { + case nvme_admin_identify: + switch (c->identify.cns) { + case NVME_ID_CNS_NS: + case NVME_ID_CNS_CS_NS: + case NVME_ID_CNS_NS_CS_INDEP: + case NVME_ID_CNS_CS_CTRL: + case NVME_ID_CNS_CTRL: + return true; + } + break; + case nvme_admin_set_features: + /* + * Reject Set Features that change controller state the driver + * manages itself; setting them behind the driver's back from + * userspace leaves it unable to react correctly. Keep Alive is + * only armed for fabrics - on other transports it has no + * reserved tag and harms idle power states. + */ + switch (le32_to_cpu(c->features.fid) & 0xff) { + case NVME_FEAT_KATO: + if (ctrl->ops->flags & NVME_F_FABRICS) + break; + fallthrough; + case NVME_FEAT_HOST_BEHAVIOR: + case NVME_FEAT_HOST_MEM_BUF: + case NVME_FEAT_NUM_QUEUES: + case NVME_FEAT_AUTO_PST: + return false; } - goto admin; + break; } + return capable(CAP_SYS_ADMIN); +} + +static bool nvme_ns_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, + bool open_for_write) +{ + u32 effects; /* * Check if the controller provides a Commands Supported and Effects log @@ -61,7 +70,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, */ effects = nvme_command_effects(ns->ctrl, ns, c->common.opcode); if (!(effects & NVME_CMD_EFFECTS_CSUPP)) - goto admin; + return capable(CAP_SYS_ADMIN); /* * Don't allow passthrough for command that have intrusive (or unknown) @@ -70,7 +79,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_UUID_SEL | NVME_CMD_EFFECTS_SCOPE_MASK)) - goto admin; + return capable(CAP_SYS_ADMIN); /* * Only allow I/O commands that transfer data to the controller or that @@ -79,11 +88,34 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c, */ if ((nvme_is_write(c) || (effects & NVME_CMD_EFFECTS_LBCC)) && !open_for_write) - goto admin; + return capable(CAP_SYS_ADMIN); return true; -admin: - return capable(CAP_SYS_ADMIN); +} + +static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns, + struct nvme_command *c, unsigned int flags, + bool open_for_write) +{ + /* + * Do not allow unprivileged passthrough on partitions, as that + * allows an escape from the containment of the partition. + */ + if (flags & NVME_IOCTL_PARTITION) + return capable(CAP_SYS_ADMIN); + + /* + * Do not allow unprivileged processes to send vendor specific or + * fabrics commands as we can't be sure about their effects. + */ + if (c->common.opcode >= nvme_cmd_vendor_start || + c->common.opcode == nvme_fabrics_command) + return capable(CAP_SYS_ADMIN); + + if (!ns) + return nvme_admin_cmd_allowed(ctrl, c); + + return nvme_ns_cmd_allowed(ns, c, open_for_write); } /* @@ -202,7 +234,8 @@ static int nvme_submit_user_cmd(struct request_queue *q, return ret; } -static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) +static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio, + unsigned int flags, bool open_for_write) { struct nvme_user_io io; struct nvme_command c; @@ -260,6 +293,9 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio) c.rw.lbat = cpu_to_le16(io.apptag); c.rw.lbatm = cpu_to_le16(io.appmask); + if (!nvme_cmd_allowed(ns->ctrl, ns, &c, flags, open_for_write)) + return -EACCES; + return nvme_submit_user_cmd(ns->queue, &c, io.addr, length, metadata, meta_len, NULL, 0, 0); } @@ -307,7 +343,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, c.common.cdw14 = cpu_to_le32(cmd.cdw14); c.common.cdw15 = cpu_to_le32(cmd.cdw15); - if (!nvme_cmd_allowed(ns, &c, 0, open_for_write)) + if (!nvme_cmd_allowed(ctrl, ns, &c, 0, open_for_write)) return -EACCES; if (cmd.timeout_ms) @@ -354,7 +390,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns, c.common.cdw14 = cpu_to_le32(cmd.cdw14); c.common.cdw15 = cpu_to_le32(cmd.cdw15); - if (!nvme_cmd_allowed(ns, &c, flags, open_for_write)) + if (!nvme_cmd_allowed(ctrl, ns, &c, flags, open_for_write)) return -EACCES; if (cmd.timeout_ms) @@ -440,6 +476,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); const struct nvme_uring_cmd *cmd = io_uring_sqe_cmd(ioucmd->sqe); struct request_queue *q = ns ? ns->queue : ctrl->admin_q; + bool open_for_write = ioucmd->file->f_mode & FMODE_WRITE; struct nvme_uring_data d; struct nvme_command c; struct iov_iter iter; @@ -470,7 +507,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, c.common.cdw14 = cpu_to_le32(READ_ONCE(cmd->cdw14)); c.common.cdw15 = cpu_to_le32(READ_ONCE(cmd->cdw15)); - if (!nvme_cmd_allowed(ns, &c, 0, ioucmd->file->f_mode & FMODE_WRITE)) + if (!nvme_cmd_allowed(ctrl, ns, &c, 0, open_for_write)) return -EACCES; d.metadata = READ_ONCE(cmd->metadata); @@ -586,7 +623,7 @@ static int nvme_ns_ioctl(struct nvme_ns *ns, unsigned int cmd, case NVME_IOCTL_SUBMIT_IO32: #endif case NVME_IOCTL_SUBMIT_IO: - return nvme_submit_io(ns, argp); + return nvme_submit_io(ns, argp, flags, open_for_write); case NVME_IOCTL_IO64_CMD_VEC: flags |= NVME_IOCTL_VEC; fallthrough; diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 102fae6a231c..ec03a07617af 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -178,8 +178,69 @@ enum nvme_quirks { * Align dma pool segment size to 512 bytes */ NVME_QUIRK_DMAPOOL_ALIGN_512 = (1 << 22), + + /* + * Admin queue DMA buffers must be page aligned + */ + NVME_QUIRK_ADMIN_PAGE_ALIGN = (1 << 23), }; +static inline char *nvme_quirk_name(enum nvme_quirks q) +{ + switch (q) { + case NVME_QUIRK_STRIPE_SIZE: + return "stripe_size"; + case NVME_QUIRK_IDENTIFY_CNS: + return "identify_cns"; + case NVME_QUIRK_DEALLOCATE_ZEROES: + return "deallocate_zeroes"; + case NVME_QUIRK_DELAY_BEFORE_CHK_RDY: + return "delay_before_chk_rdy"; + case NVME_QUIRK_NO_APST: + return "no_apst"; + case NVME_QUIRK_NO_DEEPEST_PS: + return "no_deepest_ps"; + case NVME_QUIRK_QDEPTH_ONE: + return "qdepth_one"; + case NVME_QUIRK_MEDIUM_PRIO_SQ: + return "medium_prio_sq"; + case NVME_QUIRK_IGNORE_DEV_SUBNQN: + return "ignore_dev_subnqn"; + case NVME_QUIRK_DISABLE_WRITE_ZEROES: + return "disable_write_zeroes"; + case NVME_QUIRK_SIMPLE_SUSPEND: + return "simple_suspend"; + case NVME_QUIRK_SINGLE_VECTOR: + return "single_vector"; + case NVME_QUIRK_128_BYTES_SQES: + return "128_bytes_sqes"; + case NVME_QUIRK_SHARED_TAGS: + return "shared_tags"; + case NVME_QUIRK_NO_TEMP_THRESH_CHANGE: + return "no_temp_thresh_change"; + case NVME_QUIRK_NO_NS_DESC_LIST: + return "no_ns_desc_list"; + case NVME_QUIRK_DMA_ADDRESS_BITS_48: + return "dma_address_bits_48"; + case NVME_QUIRK_SKIP_CID_GEN: + return "skip_cid_gen"; + case NVME_QUIRK_BOGUS_NID: + return "bogus_nid"; + case NVME_QUIRK_NO_SECONDARY_TEMP_THRESH: + return "no_secondary_temp_thresh"; + case NVME_QUIRK_FORCE_NO_SIMPLE_SUSPEND: + return "force_no_simple_suspend"; + case NVME_QUIRK_BROKEN_MSI: + return "broken_msi"; + case NVME_QUIRK_DMAPOOL_ALIGN_512: + return "dmapool_align_512"; + case NVME_QUIRK_ADMIN_PAGE_ALIGN: + return "admin_page_align"; + } + + return "unknown"; +} + /* * Common request structure for NVMe passthrough. All drivers must have * this structure as the first member of their request-private data. @@ -558,6 +619,12 @@ static inline bool nvme_ns_has_pi(struct nvme_ns_head *head) return head->pi_type && head->ms == head->pi_size; } +static inline unsigned long nvme_get_virt_boundary(struct nvme_ctrl *ctrl, + bool is_admin) +{ + return NVME_CTRL_PAGE_SIZE - 1; +} + struct nvme_ctrl_ops { const char *name; struct module *module; @@ -578,6 +645,7 @@ struct nvme_ctrl_ops { int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size); void (*print_device_info)(struct nvme_ctrl *ctrl); bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); + unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl, bool is_admin); }; /* diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 0a46c60583d7..e414ba56a6be 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -618,9 +618,22 @@ static inline enum nvme_use_sgl nvme_pci_use_sgls(struct nvme_dev *dev, struct nvme_queue *nvmeq = req->mq_hctx->driver_data; if (nvmeq->qid && nvme_ctrl_sgl_supported(&dev->ctrl)) { - if (nvme_req(req)->flags & NVME_REQ_USERCMD) - return SGL_FORCED; - if (req->nr_integrity_segments > 1) + /* + * When the controller is capable of using SGL, there are + * several conditions that we force to use it: + * + * 1. A request containing page gaps within the controller's + * mask can not use the PRP format. + * + * 2. User commands use SGL because that lets the device + * validate the requested transfer lengths. + * + * 3. Multiple integrity segments must use SGL as that's the + * only way to describe such a command in NVMe. + */ + if (req_phys_gap_mask(req) & (NVME_CTRL_PAGE_SIZE - 1) || + nvme_req(req)->flags & NVME_REQ_USERCMD || + req->nr_integrity_segments > 1) return SGL_FORCED; return SGL_SUPPORTED; } @@ -3286,6 +3299,14 @@ static bool nvme_pci_supports_pci_p2pdma(struct nvme_ctrl *ctrl) return dma_pci_p2pdma_supported(dev->dev); } +static unsigned long nvme_pci_get_virt_boundary(struct nvme_ctrl *ctrl, + bool is_admin) +{ + if (!nvme_ctrl_sgl_supported(ctrl) || is_admin) + return NVME_CTRL_PAGE_SIZE - 1; + return 0; +} + static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .name = "pcie", .module = THIS_MODULE, @@ -3300,6 +3321,7 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .get_address = nvme_pci_get_address, .print_device_info = nvme_pci_print_device_info, .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma, + .get_virt_boundary = nvme_pci_get_virt_boundary, }; static int nvme_dev_map(struct nvme_dev *dev) @@ -3557,6 +3579,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) nvme_dev_remove_admin(dev); nvme_dbbuf_dma_free(dev); nvme_free_queues(dev, 0); + nvme_release_descriptor_pools(dev); out_release_iod_mempool: mempool_destroy(dev->dmavec_mempool); out_dev_unmap: diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 190a4cfa8a5e..35c0822edb2d 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2202,6 +2202,7 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = { .delete_ctrl = nvme_rdma_delete_ctrl, .get_address = nvmf_get_address, .stop_ctrl = nvme_rdma_stop_ctrl, + .get_virt_boundary = nvme_get_virt_boundary, }; /* diff --git a/drivers/nvme/host/sysfs.c b/drivers/nvme/host/sysfs.c index 29430949ce2f..16c6fea4b2db 100644 --- a/drivers/nvme/host/sysfs.c +++ b/drivers/nvme/host/sysfs.c @@ -601,6 +601,28 @@ static ssize_t dctype_show(struct device *dev, } static DEVICE_ATTR_RO(dctype); +static ssize_t quirks_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + int count = 0, i; + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + unsigned long quirks = ctrl->quirks; + + if (!quirks) + return sysfs_emit(buf, "none\n"); + + for (i = 0; quirks; ++i) { + if (quirks & 1) { + count += sysfs_emit_at(buf, count, "%s\n", + nvme_quirk_name(BIT(i))); + } + quirks >>= 1; + } + + return count; +} +static DEVICE_ATTR_RO(quirks); + #ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -742,6 +764,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_kato.attr, &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, + &dev_attr_quirks.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr, diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index a5fe7c191b35..8aaec6096d6d 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2923,6 +2923,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ctrl_ops = { .delete_ctrl = nvme_tcp_delete_ctrl, .get_address = nvme_tcp_get_address, .stop_ctrl = nvme_tcp_stop_ctrl, + .get_virt_boundary = nvmf_get_virt_boundary, }; static bool diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c index 1cb07d279b44..1f145b88120b 100644 --- a/drivers/nvme/target/core.c +++ b/drivers/nvme/target/core.c @@ -601,7 +601,8 @@ int nvmet_ns_enable(struct nvmet_ns *ns) goto out_dev_put; } - if (percpu_ref_init(&ns->ref, nvmet_destroy_namespace, 0, GFP_KERNEL)) + ret = percpu_ref_init(&ns->ref, nvmet_destroy_namespace, 0, GFP_KERNEL); + if (ret) goto out_pr_exit; nvmet_ns_changed(subsys, ns->nsid); diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c index 27c15748fc7b..9642f1360165 100644 --- a/drivers/nvme/target/fabrics-cmd-auth.c +++ b/drivers/nvme/target/fabrics-cmd-auth.c @@ -31,12 +31,16 @@ void nvmet_auth_sq_init(struct nvmet_sq *sq) sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE; } -static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d) +static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl) { struct nvmet_ctrl *ctrl = req->sq->ctrl; struct nvmf_auth_dhchap_negotiate_data *data = d; int i, hash_id = 0, fallback_hash_id = 0, dhgid, fallback_dhgid; + if (tl < sizeof(*data) + + sizeof(struct nvmf_auth_dhchap_protocol_descriptor)) + return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD; + pr_debug("%s: ctrl %d qid %d: data sc_d %d napd %d authid %d halen %d dhlen %d\n", __func__, ctrl->cntlid, req->sq->qid, data->sc_c, data->napd, data->auth_protocol[0].dhchap.authid, @@ -72,6 +76,10 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d) NVME_AUTH_DHCHAP_AUTH_ID) return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD; + if (data->auth_protocol[0].dhchap.dhlen > NVME_AUTH_DHCHAP_MAX_DH_IDS || + data->auth_protocol[0].dhchap.halen > NVME_AUTH_DHCHAP_MAX_HASH_IDS) + return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD; + for (i = 0; i < data->auth_protocol[0].dhchap.halen; i++) { u8 host_hmac_id = data->auth_protocol[0].dhchap.idlist[i]; @@ -317,7 +325,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) } else if (data->auth_id != req->sq->dhchap_step) goto done_failure1; /* Validate negotiation parameters */ - dhchap_status = nvmet_auth_negotiate(req, d); + dhchap_status = nvmet_auth_negotiate(req, d, tl); if (dhchap_status == 0) req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE; diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f85a8441bcc6..fc8e7c9ad858 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -511,6 +511,7 @@ static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = { .submit_async_event = nvme_loop_submit_async_event, .delete_ctrl = nvme_loop_delete_ctrl_host, .get_address = nvmf_get_address, + .get_virt_boundary = nvme_get_virt_boundary, }; static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl) diff --git a/drivers/nvme/target/pr.c b/drivers/nvme/target/pr.c index cd22d8333314..783e6a5ec3e0 100644 --- a/drivers/nvme/target/pr.c +++ b/drivers/nvme/target/pr.c @@ -355,9 +355,15 @@ static u16 nvmet_pr_replace(struct nvmet_req *req, u16 status = NVME_SC_RESERVATION_CONFLICT | NVME_STATUS_DNR; struct nvmet_ctrl *ctrl = req->sq->ctrl; struct nvmet_pr *pr = &req->ns->pr; - struct nvmet_pr_registrant *reg; + struct nvmet_pr_registrant *reg, *new = NULL; u64 nrkey = le64_to_cpu(d->nrkey); + if (ignore_key && nrkey) { + new = kzalloc_obj(*new); + if (!new) + return NVME_SC_INTERNAL; + } + down(&pr->pr_sem); list_for_each_entry_rcu(reg, &pr->registrant_list, entry) { if (uuid_equal(®->hostid, &ctrl->hostid)) { @@ -365,9 +371,26 @@ static u16 nvmet_pr_replace(struct nvmet_req *req, status = nvmet_pr_update_reg_attr(pr, reg, nvmet_pr_update_reg_rkey, &nrkey); - break; + goto free_data; + } + } + + if (ignore_key) { + if (!nrkey) { + status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; + goto free_data; } + INIT_LIST_HEAD(&new->entry); + new->rkey = nrkey; + uuid_copy(&new->hostid, &ctrl->hostid); + list_add_tail_rcu(&new->entry, &pr->registrant_list); + status = NVME_SC_SUCCESS; + goto out; } + +free_data: + kfree(new); +out: up(&pr->pr_sem); return status; } diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index 4b493226d07c..97eb0c61baa1 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -658,18 +658,25 @@ static void nvmet_rdma_rw_ctx_destroy(struct nvmet_rdma_rsp *rsp) req->sg, req->sg_cnt, nvmet_data_dir(req)); } -static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp) +static void nvmet_rdma_free_rsp_resources(struct nvmet_rdma_rsp *rsp) { struct nvmet_rdma_queue *queue = rsp->queue; - atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail); - if (rsp->n_rdma) nvmet_rdma_rw_ctx_destroy(rsp); if (rsp->req.sg < rsp->cmd->inline_sg || rsp->req.sg >= rsp->cmd->inline_sg + queue->dev->inline_page_count) nvmet_req_free_sgls(&rsp->req); +} + +static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp) +{ + struct nvmet_rdma_queue *queue = rsp->queue; + + atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail); + + nvmet_rdma_free_rsp_resources(rsp); if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list))) nvmet_rdma_process_wr_wait_list(queue); @@ -1339,9 +1346,27 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue) goto out; } +static bool nvmet_rdma_reclaim_rsp(struct sbitmap *sb, unsigned int bitnr, + void *data) +{ + struct nvmet_rdma_queue *queue = data; + + nvmet_rdma_free_rsp_resources(&queue->rsps[bitnr]); + + return true; +} + static void nvmet_rdma_destroy_queue_ib(struct nvmet_rdma_queue *queue) { ib_drain_qp(queue->qp); + + /* + * Reclaim resources of a response that is still in-flight when the + * queue is being torn down. This happens when the connection was + * forcefully disconnected while an I/O is in flight. + */ + sbitmap_for_each_set(&queue->rsp_tags, nvmet_rdma_reclaim_rsp, queue); + if (queue->cm_id) rdma_destroy_id(queue->cm_id); ib_destroy_qp(queue->qp); diff --git a/drivers/nvme/target/zns.c b/drivers/nvme/target/zns.c index 15a579cf528c..95af4b67108c 100644 --- a/drivers/nvme/target/zns.c +++ b/drivers/nvme/target/zns.c @@ -120,7 +120,7 @@ void nvmet_execute_identify_ns_zns(struct nvmet_req *req) mutex_unlock(&req->ns->subsys->lock); } - if (!bdev_is_zoned(req->ns->bdev)) { + if (!req->ns->bdev || !bdev_is_zoned(req->ns->bdev)) { status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; req->error_loc = offsetof(struct nvme_identify, nsid); goto out; diff --git a/drivers/pci/controller/cadence/pci-j721e.c b/drivers/pci/controller/cadence/pci-j721e.c index 0413d163cfea..e2ec684cc54c 100644 --- a/drivers/pci/controller/cadence/pci-j721e.c +++ b/drivers/pci/controller/cadence/pci-j721e.c @@ -381,7 +381,7 @@ static const struct j721e_pcie_data j7200_pcie_rc_data = { .quirk_detect_quiet_flag = true, .linkdown_irq_regfield = J7200_LINK_DOWN, .byte_access_allowed = true, - .max_lanes = 2, + .max_lanes = 4, }; static const struct j721e_pcie_data j7200_pcie_ep_data = { @@ -389,7 +389,7 @@ static const struct j721e_pcie_data j7200_pcie_ep_data = { .quirk_detect_quiet_flag = true, .linkdown_irq_regfield = J7200_LINK_DOWN, .quirk_disable_flr = true, - .max_lanes = 2, + .max_lanes = 4, }; static const struct j721e_pcie_data am64_pcie_rc_data = { diff --git a/drivers/pci/controller/pci-xgene.c b/drivers/pci/controller/pci-xgene.c index b95afa35201d..83c9a2930eec 100644 --- a/drivers/pci/controller/pci-xgene.c +++ b/drivers/pci/controller/pci-xgene.c @@ -58,7 +58,6 @@ #define XGENE_PCIE_IP_VER_2 2 struct xgene_pcie { - struct device_node *node; struct device *dev; struct clk *clk; void __iomem *csr_base; @@ -526,7 +525,7 @@ static void xgene_pcie_setup_ib_reg(struct xgene_pcie *port, static int xgene_pcie_parse_map_dma_ranges(struct xgene_pcie *port) { - struct device_node *np = port->node; + struct device_node *np = port->dev->of_node; struct of_pci_range range; struct of_pci_range_parser parser; struct device *dev = port->dev; @@ -612,7 +611,6 @@ static bool xgene_check_pcie_msi_ready(void) static int xgene_pcie_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; - struct device_node *dn = dev->of_node; struct xgene_pcie *port; struct pci_host_bridge *bridge; int ret; @@ -627,7 +625,6 @@ static int xgene_pcie_probe(struct platform_device *pdev) port = pci_host_bridge_priv(bridge); - port->node = of_node_get(dn); port->dev = dev; port->version = XGENE_PCIE_IP_VER_1; diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c index 628f8c8d6747..fab44054a5de 100644 --- a/drivers/pci/controller/plda/pcie-starfive.c +++ b/drivers/pci/controller/plda/pcie-starfive.c @@ -419,7 +419,11 @@ static int starfive_pcie_probe(struct platform_device *pdev) return ret; pm_runtime_enable(&pdev->dev); - pm_runtime_get_sync(&pdev->dev); + ret = pm_runtime_resume_and_get(&pdev->dev); + if (ret < 0) { + pm_runtime_disable(&pdev->dev); + return dev_err_probe(dev, ret, "failed to resume device\n"); + } plda->host_ops = &sf_host_ops; plda->num_events = PLDA_MAX_EVENT_NUM; @@ -445,9 +449,9 @@ static void starfive_pcie_remove(struct platform_device *pdev) { struct starfive_jh7110_pcie *pcie = platform_get_drvdata(pdev); - pm_runtime_put(&pdev->dev); - pm_runtime_disable(&pdev->dev); plda_pcie_host_deinit(&pcie->plda); + pm_runtime_put_sync(&pdev->dev); + pm_runtime_disable(&pdev->dev); platform_set_drvdata(pdev, NULL); } diff --git a/drivers/pci/hotplug/pnv_php.c b/drivers/pci/hotplug/pnv_php.c index c5345bff9a55..e9c8f7fb2213 100644 --- a/drivers/pci/hotplug/pnv_php.c +++ b/drivers/pci/hotplug/pnv_php.c @@ -813,7 +813,7 @@ static struct pnv_php_slot *pnv_php_alloc_slot(struct device_node *dn) if (dn->child && PCI_DN(dn->child)) php_slot->slot_no = PCI_SLOT(PCI_DN(dn->child)->devfn); else - php_slot->slot_no = -1; /* Placeholder slot */ + php_slot->slot_no = PCI_SLOT_PLACEHOLDER; /* Placeholder slot */ kref_init(&php_slot->kref); php_slot->state = PNV_PHP_STATE_INITIALIZED; diff --git a/drivers/pci/hotplug/rpaphp_slot.c b/drivers/pci/hotplug/rpaphp_slot.c index 779eab12e981..ac8c512190df 100644 --- a/drivers/pci/hotplug/rpaphp_slot.c +++ b/drivers/pci/hotplug/rpaphp_slot.c @@ -85,7 +85,7 @@ int rpaphp_register_slot(struct slot *slot) struct device_node *child; u32 my_index; int retval; - int slotno = -1; + int slotno = PCI_SLOT_PLACEHOLDER; dbg("%s registering slot:path[%pOF] index[%x], name[%s] pdomain[%x] type[%d]\n", __func__, slot->dn, slot->index, slot->name, diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 439045ab6931..2259b9855f62 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4828,8 +4828,9 @@ static int pci_reset_hotplug_slot(struct hotplug_slot *hotplug, bool probe) static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe) { - if (dev->multifunction || dev->subordinate || !dev->slot || - dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET) + if (dev->subordinate || !dev->slot || + dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET || + (dev->multifunction && !dev->slot->per_func_slot)) return -ENOTTY; return pci_reset_hotplug_slot(dev->slot->hotplug, probe); diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index a1f275259f7d..dac81130edd3 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -821,6 +821,7 @@ static void pcie_aspm_override_default_link_state(struct pcie_link_state *link) static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) { struct pci_dev *child = link->downstream, *parent = link->pdev; + struct pci_dev *fn; u16 parent_lnkctl, child_lnkctl; struct pci_bus *linkbus = parent->subordinate; @@ -854,10 +855,11 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) /* Disable L0s/L1 before updating L1SS config */ if (FIELD_GET(PCI_EXP_LNKCTL_ASPMC, child_lnkctl) || FIELD_GET(PCI_EXP_LNKCTL_ASPMC, parent_lnkctl)) { - pcie_capability_write_word(child, PCI_EXP_LNKCTL, - child_lnkctl & ~PCI_EXP_LNKCTL_ASPMC); - pcie_capability_write_word(parent, PCI_EXP_LNKCTL, - parent_lnkctl & ~PCI_EXP_LNKCTL_ASPMC); + list_for_each_entry(fn, &linkbus->devices, bus_list) + pcie_capability_clear_and_set_word(fn, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, 0); + pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, 0); } /* @@ -887,8 +889,13 @@ static void pcie_aspm_cap_init(struct pcie_link_state *link, int blacklist) /* Restore L0s/L1 if they were enabled */ if (FIELD_GET(PCI_EXP_LNKCTL_ASPMC, child_lnkctl) || FIELD_GET(PCI_EXP_LNKCTL_ASPMC, parent_lnkctl)) { - pcie_capability_write_word(parent, PCI_EXP_LNKCTL, parent_lnkctl); - pcie_capability_write_word(child, PCI_EXP_LNKCTL, child_lnkctl); + pcie_capability_clear_and_set_word(parent, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, + parent_lnkctl & PCI_EXP_LNKCTL_ASPMC); + list_for_each_entry(fn, &linkbus->devices, bus_list) + pcie_capability_clear_and_set_word(fn, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, + child_lnkctl & PCI_EXP_LNKCTL_ASPMC); } /* Save default state */ diff --git a/drivers/pci/slot.c b/drivers/pci/slot.c index 50fb3eb595fe..179aa6861253 100644 --- a/drivers/pci/slot.c +++ b/drivers/pci/slot.c @@ -37,11 +37,20 @@ static const struct sysfs_ops pci_slot_sysfs_ops = { static ssize_t address_read_file(struct pci_slot *slot, char *buf) { - if (slot->number == 0xff) + if (slot->number == PCI_SLOT_PLACEHOLDER) return sysfs_emit(buf, "%04x:%02x\n", pci_domain_nr(slot->bus), slot->bus->number); + /* + * Preserve legacy ABI expectations that hotplug drivers that manage + * multiple devices per slot emit 0 for the device number. + */ + if (slot->number == PCI_SLOT_ALL_DEVICES) + return sysfs_emit(buf, "%04x:%02x:00\n", + pci_domain_nr(slot->bus), + slot->bus->number); + return sysfs_emit(buf, "%04x:%02x:%02x\n", pci_domain_nr(slot->bus), slot->bus->number, @@ -63,6 +72,23 @@ static ssize_t cur_speed_read_file(struct pci_slot *slot, char *buf) return bus_speed_read(slot->bus->cur_bus_speed, buf); } +static bool pci_dev_matches_slot(struct pci_dev *dev, struct pci_slot *slot) +{ + if (slot->per_func_slot) + return dev->devfn == slot->number; + + return slot->number == PCI_SLOT_ALL_DEVICES || + PCI_SLOT(dev->devfn) == slot->number; +} + +static bool pci_slot_enabled_per_func(void) +{ + if (IS_ENABLED(CONFIG_S390)) + return true; + + return false; +} + static void pci_slot_release(struct kobject *kobj) { struct pci_dev *dev; @@ -73,7 +99,7 @@ static void pci_slot_release(struct kobject *kobj) down_read(&pci_bus_sem); list_for_each_entry(dev, &slot->bus->devices, bus_list) - if (PCI_SLOT(dev->devfn) == slot->number) + if (pci_dev_matches_slot(dev, slot)) dev->slot = NULL; up_read(&pci_bus_sem); @@ -166,7 +192,7 @@ void pci_dev_assign_slot(struct pci_dev *dev) mutex_lock(&pci_slot_mutex); list_for_each_entry(slot, &dev->bus->slots, list) - if (PCI_SLOT(dev->devfn) == slot->number) + if (pci_dev_matches_slot(dev, slot)) dev->slot = slot; mutex_unlock(&pci_slot_mutex); } @@ -188,7 +214,8 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr) /** * pci_create_slot - create or increment refcount for physical PCI slot * @parent: struct pci_bus of parent bridge - * @slot_nr: PCI_SLOT(pci_dev->devfn) or -1 for placeholder + * @slot_nr: PCI_SLOT(pci_dev->devfn), PCI_SLOT_PLACEHOLDER for placeholder, or + * PCI_SLOT_ALL_DEVICES * @name: user visible string presented in /sys/bus/pci/slots/ * @hotplug: set if caller is hotplug driver, NULL otherwise * @@ -213,15 +240,26 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr) * In most cases, @pci_bus, @slot_nr will be sufficient to uniquely identify * a slot. There is one notable exception - pSeries (rpaphp), where the * @slot_nr cannot be determined until a device is actually inserted into - * the slot. In this scenario, the caller may pass -1 for @slot_nr. + * the slot. In this scenario, the caller may pass PCI_SLOT_PLACEHOLDER for @slot_nr. * * The following semantics are imposed when the caller passes @slot_nr == - * -1. First, we no longer check for an existing %struct pci_slot, as there - * may be many slots with @slot_nr of -1. The other change in semantics is - * user-visible, which is the 'address' parameter presented in sysfs will - * consist solely of a dddd:bb tuple, where dddd is the PCI domain of the - * %struct pci_bus and bb is the bus number. In other words, the devfn of - * the 'placeholder' slot will not be displayed. + * PCI_SLOT_PLACEHOLDER. First, we no longer check for an existing %struct + * pci_slot, as there may be many slots with @slot_nr of + * PCI_SLOT_PLACEHOLDER. The other change in semantics is user-visible, + * which is the 'address' parameter presented in sysfs will consist solely + * of a dddd:bb tuple, where dddd is the PCI domain of the %struct pci_bus + * and bb is the bus number. In other words, the devfn of the 'placeholder' + * slot will not be displayed. + * + * Bus-wide slots: + * For PCIe hotplug, the physical slot encompasses the entire secondary + * bus, not just a single device number. If the device supports ARI and ARI + * Forwarding is enabled in the upstream bridge, a multi-function device + * may include functions that appear to have several different device + * numbers, i.e., PCI_SLOT() values. Pass @slot_nr == PCI_SLOT_ALL_DEVICES + * to create a slot that matches all devices on the bus. Unlike placeholder + * slots, bus-wide slots go through normal slot lookup and reuse existing + * slots if present. */ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, const char *name, @@ -234,7 +272,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, mutex_lock(&pci_slot_mutex); - if (slot_nr == -1) + if (slot_nr == PCI_SLOT_PLACEHOLDER) goto placeholder; /* @@ -265,6 +303,9 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, slot->bus = pci_bus_get(parent); slot->number = slot_nr; + if (pci_slot_enabled_per_func()) + slot->per_func_slot = 1; + slot->kobj.kset = pci_slots_kset; slot_name = make_slot_name(name); @@ -285,7 +326,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, down_read(&pci_bus_sem); list_for_each_entry(dev, &parent->devices, bus_list) - if (PCI_SLOT(dev->devfn) == slot_nr) + if (pci_dev_matches_slot(dev, slot)) dev->slot = slot; up_read(&pci_bus_sem); diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index 69c5cc8f5606..265c10f2c8b0 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -1351,7 +1351,7 @@ static int branch_records_alloc(struct arm_pmu *armpmu) struct pmu_hw_events *events_cpu; events_cpu = per_cpu_ptr(armpmu->hw_events, cpu); - events_cpu->branch_stack = kmalloc(size, GFP_KERNEL); + events_cpu->branch_stack = kzalloc(size, GFP_KERNEL); if (!events_cpu->branch_stack) return -ENOMEM; } diff --git a/drivers/perf/arm_spe_pmu.c b/drivers/perf/arm_spe_pmu.c index e4e4e63c64c4..7bc24db50d1f 100644 --- a/drivers/perf/arm_spe_pmu.c +++ b/drivers/perf/arm_spe_pmu.c @@ -552,7 +552,7 @@ static u64 __arm_spe_pmu_next_off(struct perf_output_handle *handle) * the page boundary following it. Keep the tail boundary if * that's lower. */ - if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) + if ((handle->wakeup - handle->head) < handle->size && head <= wakeup) limit = min(limit, round_up(wakeup, PAGE_SIZE)); if (limit > head) diff --git a/drivers/perf/cxl_pmu.c b/drivers/perf/cxl_pmu.c index 68a54d97d2a8..39b46550a510 100644 --- a/drivers/perf/cxl_pmu.c +++ b/drivers/perf/cxl_pmu.c @@ -635,7 +635,7 @@ static void cxl_pmu_event_start(struct perf_event *event, int flags) cfg = cxl_pmu_config2_get_hdm_decoder(event); else cfg = GENMASK(31, 0); /* No filtering if 0xFFFF_FFFF */ - writeq(cfg, base + CXL_PMU_FILTER_CFG_REG(hwc->idx, 0)); + writel(cfg, base + CXL_PMU_FILTER_CFG_REG(hwc->idx, 0)); } cfg = readq(base + CXL_PMU_COUNTER_CFG_REG(hwc->idx)); diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-combo.c b/drivers/phy/qualcomm/phy-qcom-qmp-combo.c index 7b5af30f1d02..3ac29c6eaa68 100644 --- a/drivers/phy/qualcomm/phy-qcom-qmp-combo.c +++ b/drivers/phy/qualcomm/phy-qcom-qmp-combo.c @@ -29,7 +29,12 @@ #include "phy-qcom-qmp-common.h" #include "phy-qcom-qmp.h" +#include "phy-qcom-qmp-pcs-aon-v6.h" +#include "phy-qcom-qmp-pcs-aon-v8.h" #include "phy-qcom-qmp-pcs-misc-v3.h" +#include "phy-qcom-qmp-pcs-misc-v4.h" +#include "phy-qcom-qmp-pcs-misc-v5.h" +#include "phy-qcom-qmp-pcs-misc-v8.h" #include "phy-qcom-qmp-pcs-usb-v4.h" #include "phy-qcom-qmp-pcs-usb-v5.h" #include "phy-qcom-qmp-pcs-usb-v6.h" @@ -42,6 +47,9 @@ #include "phy-qcom-qmp-dp-phy-v4.h" #include "phy-qcom-qmp-dp-phy-v5.h" #include "phy-qcom-qmp-dp-phy-v6.h" +#include "phy-qcom-qmp-dp-phy-v8.h" + +#include "phy-qcom-qmp-usb43-pcs-v8.h" /* QPHY_V3_DP_COM_RESET_OVRD_CTRL register bits */ /* DP PHY soft reset */ @@ -78,6 +86,7 @@ enum qphy_reg_layout { QPHY_PCS_AUTONOMOUS_MODE_CTRL, QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR, QPHY_PCS_POWER_DOWN_CONTROL, + QPHY_PCS_CLAMP_ENABLE, QPHY_COM_RESETSM_CNTRL, QPHY_COM_C_READY_STATUS, @@ -93,6 +102,8 @@ enum qphy_reg_layout { QPHY_TX_HIGHZ_DRVR_EN, QPHY_TX_TRANSCEIVER_BIAS_EN, + QPHY_AON_TOGGLE_ENABLE, + QPHY_DP_AON_TOGGLE_ENABLE, /* Keep last to ensure regs_layout arrays are properly initialized */ QPHY_LAYOUT_SIZE }; @@ -105,6 +116,8 @@ static const unsigned int qmp_v3_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = { [QPHY_PCS_AUTONOMOUS_MODE_CTRL] = QPHY_V3_PCS_AUTONOMOUS_MODE_CTRL, [QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = QPHY_V3_PCS_LFPS_RXTERM_IRQ_CLEAR, + [QPHY_PCS_CLAMP_ENABLE] = QPHY_V3_PCS_MISC_CLAMP_ENABLE, + [QPHY_COM_RESETSM_CNTRL] = QSERDES_V3_COM_RESETSM_CNTRL, [QPHY_COM_C_READY_STATUS] = QSERDES_V3_COM_C_READY_STATUS, [QPHY_COM_CMN_STATUS] = QSERDES_V3_COM_CMN_STATUS, @@ -130,6 +143,8 @@ static const unsigned int qmp_v45_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = { [QPHY_PCS_AUTONOMOUS_MODE_CTRL] = QPHY_V4_PCS_USB3_AUTONOMOUS_MODE_CTRL, [QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = QPHY_V4_PCS_USB3_LFPS_RXTERM_IRQ_CLEAR, + [QPHY_PCS_CLAMP_ENABLE] = QPHY_V4_PCS_MISC_CLAMP_ENABLE, + [QPHY_COM_RESETSM_CNTRL] = QSERDES_V4_COM_RESETSM_CNTRL, [QPHY_COM_C_READY_STATUS] = QSERDES_V4_COM_C_READY_STATUS, [QPHY_COM_CMN_STATUS] = QSERDES_V4_COM_CMN_STATUS, @@ -155,6 +170,8 @@ static const unsigned int qmp_v5_5nm_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = { [QPHY_PCS_AUTONOMOUS_MODE_CTRL] = QPHY_V5_PCS_USB3_AUTONOMOUS_MODE_CTRL, [QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = QPHY_V5_PCS_USB3_LFPS_RXTERM_IRQ_CLEAR, + [QPHY_PCS_CLAMP_ENABLE] = QPHY_V5_PCS_MISC_CLAMP_ENABLE, + [QPHY_COM_RESETSM_CNTRL] = QSERDES_V5_COM_RESETSM_CNTRL, [QPHY_COM_C_READY_STATUS] = QSERDES_V5_COM_C_READY_STATUS, [QPHY_COM_CMN_STATUS] = QSERDES_V5_COM_CMN_STATUS, @@ -180,6 +197,8 @@ static const unsigned int qmp_v6_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = { [QPHY_PCS_AUTONOMOUS_MODE_CTRL] = QPHY_V6_PCS_USB3_AUTONOMOUS_MODE_CTRL, [QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = QPHY_V6_PCS_USB3_LFPS_RXTERM_IRQ_CLEAR, + [QPHY_PCS_CLAMP_ENABLE] = QPHY_V6_PCS_AON_CLAMP_ENABLE, + [QPHY_COM_RESETSM_CNTRL] = QSERDES_V6_COM_RESETSM_CNTRL, [QPHY_COM_C_READY_STATUS] = QSERDES_V6_COM_C_READY_STATUS, [QPHY_COM_CMN_STATUS] = QSERDES_V6_COM_CMN_STATUS, @@ -205,6 +224,8 @@ static const unsigned int qmp_v6_n4_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = { [QPHY_PCS_AUTONOMOUS_MODE_CTRL] = QPHY_V6_PCS_USB3_AUTONOMOUS_MODE_CTRL, [QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = QPHY_V6_PCS_USB3_LFPS_RXTERM_IRQ_CLEAR, + [QPHY_PCS_CLAMP_ENABLE] = QPHY_V6_PCS_AON_CLAMP_ENABLE, + [QPHY_COM_RESETSM_CNTRL] = QSERDES_V6_COM_RESETSM_CNTRL, [QPHY_COM_C_READY_STATUS] = QSERDES_V6_COM_C_READY_STATUS, [QPHY_COM_CMN_STATUS] = QSERDES_V6_COM_CMN_STATUS, @@ -245,6 +266,237 @@ static const unsigned int qmp_v8_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = { [QPHY_TX_TRANSCEIVER_BIAS_EN] = QSERDES_V8_TX_TRANSCEIVER_BIAS_EN, }; +static const unsigned int qmp_v8_n3_usb43dpphy_regs_layout[QPHY_LAYOUT_SIZE] = { + [QPHY_SW_RESET] = QPHY_V8_USB43_PCS_SW_RESET, + [QPHY_START_CTRL] = QPHY_V8_USB43_PCS_START_CONTROL, + [QPHY_PCS_STATUS] = QPHY_V8_USB43_PCS_PCS_STATUS1, + [QPHY_PCS_POWER_DOWN_CONTROL] = QPHY_V8_USB43_PCS_POWER_DOWN_CONTROL, + + /* In PCS_USB */ + [QPHY_PCS_AUTONOMOUS_MODE_CTRL] = QPHY_V8_PCS_USB_AUTONOMOUS_MODE_CTRL, + [QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = QPHY_V8_PCS_USB_LFPS_RXTERM_IRQ_CLEAR, + + [QPHY_PCS_CLAMP_ENABLE] = QPHY_V8_PCS_AON_USB3_AON_CLAMP_ENABLE, + [QPHY_AON_TOGGLE_ENABLE] = QPHY_V8_PCS_AON_USB3_AON_TOGGLE_ENABLE, + [QPHY_DP_AON_TOGGLE_ENABLE] = QPHY_V8_PCS_AON_DP_AON_TOGGLE_ENABLE, + + [QPHY_COM_RESETSM_CNTRL] = QSERDES_V8_COM_RESETSM_CNTRL, + [QPHY_COM_C_READY_STATUS] = QSERDES_V8_COM_C_READY_STATUS, + [QPHY_COM_CMN_STATUS] = QSERDES_V8_COM_CMN_STATUS, + [QPHY_COM_BIAS_EN_CLKBUFLR_EN] = QSERDES_V8_COM_BIAS_EN_CLKBUFLR_EN, + + [QPHY_DP_PHY_STATUS] = QSERDES_V8_DP_PHY_STATUS, + [QPHY_DP_PHY_VCO_DIV] = QSERDES_V8_DP_PHY_VCO_DIV, + + [QPHY_TX_TX_DRV_LVL] = QSERDES_V8_LALB_TX0_DRV_LVL, + [QPHY_TX_TX_EMP_POST1_LVL] = QSERDES_V8_LALB_TX0_EMP_POST1_LVL, + [QPHY_TX_HIGHZ_DRVR_EN] = QSERDES_V8_LALB_HIGHZ_DRVR_EN, + [QPHY_TX_TRANSCEIVER_BIAS_EN] = QSERDES_V8_LALB_TRANSMITTER_EN_CTRL, +}; + +static const struct qmp_phy_init_tbl glymur_usb43dp_serdes_tbl[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE1, 0xe1), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE2_MODE1, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CP_CTRL_MODE1, 0x06), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_RCTRL_MODE1, 0x16), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_CCTRL_MODE1, 0x36), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CORECLK_DIV_MODE1, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP1_MODE1, 0x1a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP2_MODE1, 0x41), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MODE1, 0x41), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MSB_MODE1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START1_MODE1, 0xab), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE1, 0xaa), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE1, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_HSCLK_SEL_1, 0x13), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_INTEGLOOP_GAIN0_MODE1, 0x3f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE1_MODE1, 0x4d), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE2_MODE1, 0x03), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE1, 0x95), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE1, 0x1e), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE0, 0x4b), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE0, 0x0f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE0, 0xe1), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE2_MODE0, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CP_CTRL_MODE0, 0x06), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_RCTRL_MODE0, 0x16), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_CCTRL_MODE0, 0x36), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CORECLK_DIV_MODE0, 0x05), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP1_MODE0, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP2_MODE0, 0x1a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MODE0, 0x41), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MSB_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START1_MODE0, 0xab), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE0, 0xaa), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE0, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_HSCLK_HS_SWITCH_SEL_1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_INTEGLOOP_GAIN0_MODE0, 0x3f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE1_MODE0, 0x4d), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE2_MODE0, 0x03), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BG_TIMER, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_EN_CENTER, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_PER1, 0x62), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_PER2, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYSCLK_BUF_ENABLE, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_IVCO, 0x0f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_IVCO_MODE1, 0x0f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYSCLK_EN_SEL, 0x1a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP_EN, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP_CFG, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE_CTRL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE_MAP, 0x14), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CORE_CLK_EN, 0xa0), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_CONFIG_1, 0x76), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SVS_MODE_CLK_SEL, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_HSCLK_SEL_1, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_SPARE_FOR_ECO, 0x40), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DCC_CAL_1, 0x40), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DCC_CAL_2, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DCC_CAL_3, 0x60), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PSM_CAL_EN, 0x05), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_FWD_CONFIG_1, 0x33), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_IP_CTRL_AND_DP_SEL, 0xaf), +}; + +static const struct qmp_phy_init_tbl glymur_usb43dp_pcs_misc_tbl[] = { + QMP_PHY_INIT_CFG(QPHY_V8_PCS_MISC_PCS_MISC_CONFIG1, 0x01), +}; + +static const struct qmp_phy_init_tbl glymur_usb43dp_pcs_tbl[] = { + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG1, 0xc4), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG2, 0x89), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG3, 0x20), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG6, 0x13), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_REFGEN_REQ_CONFIG1, 0x21), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_RX_SIGDET_LVL, 0x55), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_RCVR_DTCT_DLY_P1U2_L, 0xe7), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_RCVR_DTCT_DLY_P1U2_H, 0x03), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_TSYNC_RSYNC_TIME, 0xa4), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_RX_CONFIG, 0x0a), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_TSYNC_DLY_TIME, 0x04), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_ALIGN_DETECT_CONFIG1, 0xd4), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_ALIGN_DETECT_CONFIG2, 0x30), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_PCS_TX_RX_CONFIG, 0x0c), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_EQ_CONFIG1, 0x4b), + QMP_PHY_INIT_CFG(QPHY_V8_USB43_PCS_EQ_CONFIG5, 0x10), +}; + +static const struct qmp_phy_init_tbl glymur_usb43dp_pcs_usb_tbl[] = { + QMP_PHY_INIT_CFG(QPHY_V8_PCS_USB_LFPS_DET_HIGH_COUNT_VAL, 0xf8), + QMP_PHY_INIT_CFG(QPHY_V8_PCS_USB_RXEQTRAINING_DFE_TIME_S2, 0x07), +}; + +static const struct qmp_phy_init_tbl glymur_usb43dp_lalb_tbl[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CLKBUF_ENABLE, 0x81), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX_LVL_UPDATE_CTRL, 0x0d), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL3, 0x80), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL4, 0x8D), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TRANSMITTER_EN_CTRL, 0x13), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_LANE_MODE_1, 0x0c), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_LANE_MODE_2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_LANE_MODE_3, 0x11), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_LANE_MODE_4, 0x11), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_RESTRIM_CAL_CTRL, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX1_RESTRIM_CAL_CTRL, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_RESTRIM_POST_CAL_OFFSET, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_RESTRIM_VREF_SEL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX1_RESTRIM_VREF_SEL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_ANA_INTERFACE_SELECT2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_PCS_INTERFACE_SELECT1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B0, 0xa4), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B1, 0xa2), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B2, 0x6e), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B3, 0x51), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B4, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B5, 0x26), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B6, 0x12), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE_0_1_B7, 0x2a), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B0, 0x4c), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B1, 0xc4), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B2, 0x38), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B3, 0x64), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B4, 0x0c), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B5, 0x4b), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B6, 0x12), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_MODE_RATE2_B7, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX_DCC_ANA_CTRL2, 0x0c), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT1_RATE1, 0x26), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT2_RATE1, 0x26), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT1_RATE2, 0x26), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT2_RATE2, 0x26), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_INIT_RATE_0_1, 0x11), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_INIT_RATE_2_3, 0x11), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_CODE_OVRD_RATE1, 0x03), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_CODE_OVRD_RATE2, 0x03), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF1_RATE1, 0x15), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF2_RATE1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF1_RATE2, 0x22), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF2_RATE2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KP_CODE_OVRD_RATE_2_3, 0x22), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND1_RATE1, 0xff), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND2_RATE1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND1_RATE2, 0xff), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND2_RATE2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KP_CAL_LOWER_FREQ_DIFF_BND_RATE1, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_KP_CAL_LOWER_FREQ_DIFF_BND_RATE2, 0x09), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_SUMMER_CAL_SPD_MODE_RATE_0123, 0x2f), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_IVCM_CAL_CODE_OVERRIDE_RATE1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_IVCM_CAL_CODE_OVERRIDE_RATE2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_IVCM_CAL_CTRL2, 0x85), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_IVCM_CAL_CTRL3, 0x45), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_IVCM_POSTCAL_OFFSET_RATE1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_IVCM_POSTCAL_OFFSET_RATE2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_SIGDET_ENABLES, 0x0c), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_SIGDET_CNTRL, 0xa3), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_SIGDET_LVL, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_SIGDET_DEGLITCH_CNTRL, 0x0e), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_SIGDET_CAL_CTRL1, 0x14), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_SIGDET_CAL_CTRL2_AND_CDR_LOCK_EDGE, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_SIGDET_CAL_TRIM, 0x66), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_FREQ_LOCK_DET_DLY_RATE1, 0xff), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_FREQ_LOCK_DET_DLY_RATE2, 0x32), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_CP_CUR_FLL_RATE1, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_CP_CUR_FLL_RATE2, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_CP_CUR_PLL_RATE1, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_CP_CUR_PLL_RATE2, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_CCODE_RATE_01, 0x76), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_CCODE_RATE_23, 0x67), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_RCODE_FAST_RATE_0_1, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_RCODE_FAST_RATE_2_3, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_RCODE_FLL_RATE_0_1, 0x33), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_RCODE_FLL_RATE_2_3, 0x43), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_RCODE_PLL_RATE_0_1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_RCODE_PLL_RATE_2_3, 0x51), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_FLL_DIV_RATIO_RATE_0123, 0xe5), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_CAP_CODE_RATE_0123, 0xf5), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_TYPE_CONFIG, 0x1f), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_EN_LOWFREQ, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_LOOP_FUNC_CTRL, 0xd0), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_GM_CAL_EN, 0x1f), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_GM_CAL_RES_RATE0_1, 0x88), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_GM_CAL_RES_RATE2_3, 0x88), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_AUX_CLK_CTRL, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_EOM_CTRL1, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL2, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL3, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL4, 0xaa), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CTLE_POST_CAL_OFFSET_RATE_0_1_2, 0x77), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_VGA_CAL_CNTRL1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_VGA_CAL_MAN_VAL_RATE0_1, 0xdd), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_VGA_CAL_MAN_VAL_RATE2_3, 0xd8), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_DFE_TAP1_DAC_ENABLE, 0x1c), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_DFE_TAP2_DAC_ENABLE, 0x1c), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_DFE_TAP345_DAC_ENABLE, 0x18), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_DFE_TAP67_DAC_ENABLE, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_IQTUNE_CTRL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_IQTUNE_MAN_INDEX, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_IQTUNE_DIV2_CTRL_RATE0123, 0x1C), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CDR_VCO_CAP_CODE_OVRD_MUXES, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_DIG_BKUP_CTRL16, 0x37), +}; + static const struct qmp_phy_init_tbl qmp_v3_usb3_serdes_tbl[] = { QMP_PHY_INIT_CFG(QSERDES_V3_COM_PLL_IVCO, 0x07), QMP_PHY_INIT_CFG(QSERDES_V3_COM_SYSCLK_EN_SEL, 0x14), @@ -1131,6 +1383,38 @@ static const struct qmp_phy_init_tbl qmp_v6_n4_dp_serdes_tbl[] = { QMP_PHY_INIT_CFG(QSERDES_V6_COM_CORE_CLK_EN, 0x0f), }; +static const struct qmp_phy_init_tbl qmp_v8_dp_serdes_tbl[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE2_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CP_CTRL_MODE0, 0x06), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_RCTRL_MODE1, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_CCTRL_MODE1, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CORECLK_DIV_MODE0, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START1_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_INTEGLOOP_GAIN0_MODE0, 0x3f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_INTEGLOOP_GAIN1_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BG_TIMER, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_EN_CENTER, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_ADJ_PER1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_PER1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_PER2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_ENABLE1, 0x0c), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYS_CLK_CTRL, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYSCLK_BUF_ENABLE, 0x06), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_IVCO, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYSCLK_EN_SEL, 0x3b), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP_EN, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE_CTRL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE_MAP, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_SELECT, 0x30), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CORE_CLK_EN, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_CONFIG_1, 0x56), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SVS_MODE_CLK_SEL, 0x15), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_MODE_CONTD1, 0x24), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DCC_CAL_1, 0x40), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DCC_CAL_3, 0x60), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PSM_CAL_EN, 0x01), +}; + static const struct qmp_phy_init_tbl qmp_v6_dp_tx_tbl[] = { QMP_PHY_INIT_CFG(QSERDES_V6_TX_VMODE_CTRL1, 0x40), QMP_PHY_INIT_CFG(QSERDES_V6_TX_PRE_STALL_LDO_BOOST_EN, 0x30), @@ -1158,6 +1442,33 @@ static const struct qmp_phy_init_tbl qmp_v6_n4_dp_tx_tbl[] = { QMP_PHY_INIT_CFG(QSERDES_V6_N4_TX_TX_BAND, 0x1), }; +static const struct qmp_phy_init_tbl qmp_v8_n3p_dp_tx_tbl[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TRANSMITTER_EN_CTRL, 0x3f), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_VMODE_CTRL1, 0x40), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_ANA_INTERFACE_SELECT1, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_ANA_INTERFACE_SELECT2, 0x18), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_PCS_INTERFACE_SELECT1, 0x50), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_LANE_MODE_1, 0x0d), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_CLKBUF_ENABLE, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_RESET_TSYNC_EN_CTRL, 0x0a), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX_LVL_UPDATE_CTRL, 0x0f), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TRAN_DRVR_EMP_EN, 0x5f), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_EMP_POST1_LVL, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX1_EMP_POST1_LVL, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_PRE1_EMPH, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX1_PRE1_EMPH, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_DRV_LVL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX1_DRV_LVL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_HIGHZ_DRVR_EN, 0x30), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_LANE_MODE_2, 0x50), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_LANE_MODE_3, 0x51), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX_DCC_ANA_CTRL2, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_RESTRIM_CAL_CTRL, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX1_RESTRIM_CAL_CTRL, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX0_RESTRIM_POST_CAL_OFFSET, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_LALB_TX1_RESTRIM_POST_CAL_OFFSET, 0x10), +}; + static const struct qmp_phy_init_tbl qmp_v6_dp_serdes_tbl_rbr[] = { QMP_PHY_INIT_CFG(QSERDES_V6_COM_HSCLK_SEL_1, 0x05), QMP_PHY_INIT_CFG(QSERDES_V6_COM_DEC_START_MODE0, 0x34), @@ -1274,6 +1585,109 @@ static const struct qmp_phy_init_tbl qmp_v6_n4_dp_serdes_tbl_hbr3[] = { QMP_PHY_INIT_CFG(QSERDES_V6_COM_SSC_STEP_SIZE2_MODE0, 0x01), }; +static const struct qmp_phy_init_tbl qmp_v8_dp_serdes_tbl_rbr[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_HSCLK_SEL_1, 0x05), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE0, 0x7a), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE0, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE0, 0x83), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP1_MODE0, 0x37), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP2_MODE0, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MODE0, 0x54), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE0, 0x06), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE1_MODE0, 0xfe), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE2_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_HSCLK_SEL_1, 0x05), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_MODE_CONTD3, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_FWD_CONFIG_1, 0x30), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_IP_CTRL_AND_DP_SEL, 0xa4), +}; + +static const struct qmp_phy_init_tbl qmp_v8_dp_serdes_tbl_hbr[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_HSCLK_SEL_1, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE0, 0x21), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE0, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE0, 0x18), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP1_MODE0, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP2_MODE0, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MODE0, 0x46), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE0, 0x05), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE1_MODE0, 0xae), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE2_MODE0, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_HSCLK_SEL_1, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_MODE_CONTD3, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_FWD_CONFIG_1, 0x3f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_IP_CTRL_AND_DP_SEL, 0xa3), +}; + +static const struct qmp_phy_init_tbl qmp_v8_dp_serdes_tbl_hbr2[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_HSCLK_SEL_1, 0x03), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE0, 0xf6), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE0, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE0, 0x0), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_RCTRL_MODE0, 0x16), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_CCTRL_MODE0, 0x36), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP1_MODE0, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP2_MODE0, 0x0e), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MODE0, 0x46), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE0, 0x05), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE1_MODE0, 0xae), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE2_MODE0, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_HSCLK_SEL_1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_IP_CTRL_AND_DP_SEL, 0xbf), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIAS_EN_CLKBUFLR_EN, 0x1c), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_RESETSM_CNTRL, 0x20), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_MODE_CONTD3, 0x03), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_FWD_CONFIG_1, 0x3f), +}; + +static const struct qmp_phy_init_tbl qmp_v8_dp_serdes_tbl_hbr3[] = { + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_HSCLK_SEL_1, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE0, 0x63), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE0, 0x0c), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE0, 0x5b), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_STEP_SIZE2_MODE0, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CP_CTRL_MODE0, 0x06), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_RCTRL_MODE0, 0x16), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_CCTRL_MODE0, 0x36), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CORECLK_DIV_MODE0, 0x0a), + + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP1_MODE0, 0x17), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_LOCK_CMP2_MODE0, 0x15), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DEC_START_MODE0, 0x4f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START1_MODE0, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE0, 0xa0), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE0, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_INTEGLOOP_GAIN0_MODE0, 0x3f), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_INTEGLOOP_GAIN1_MODE0, 0x00), + + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE1_MODE0, 0xa0), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE2_MODE0, 0x01), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_ADJ_PER1, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_PER1, 0x6b), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SSC_PER2, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_ENABLE1, 0x0c), + + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYS_CLK_CTRL, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYSCLK_BUF_ENABLE, 0x06), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_PLL_IVCO, 0x07), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SYSCLK_EN_SEL, 0x04), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE_CTRL, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_VCO_TUNE_MAP, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_SELECT, 0x30), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CORE_CLK_EN, 0x00), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_CONFIG_1, 0x16), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_SVS_MODE_CLK_SEL, 0x15), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CLK_FWD_CONFIG_1, 0x30), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIAS_EN_CLKBUFLR_EN, 0x10), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_MODE_CONTD3, 0x05), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_CMN_MODE_CONTD1, 0x24), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_BIN_VCOCAL_HSCLK_SEL_1, 0x02), + QMP_PHY_INIT_CFG(QSERDES_V8_USB43_COM_IP_CTRL_AND_DP_SEL, 0x84), +}; + static const struct qmp_phy_init_tbl sc8280xp_usb43dp_serdes_tbl[] = { QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_EN_CENTER, 0x01), QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_PER1, 0x31), @@ -1643,14 +2057,15 @@ static const struct qmp_phy_init_tbl x1e80100_usb43dp_pcs_usb_tbl[] = { }; /* list of regulators */ -struct qmp_regulator_data { - const char *name; - unsigned int enable_load; +static struct regulator_bulk_data qmp_phy_vreg_l[] = { + { .supply = "vdda-phy", .init_load_uA = 21800, }, + { .supply = "vdda-pll", .init_load_uA = 36000, }, }; -static struct qmp_regulator_data qmp_phy_vreg_l[] = { - { .name = "vdda-phy", .enable_load = 21800 }, - { .name = "vdda-pll", .enable_load = 36000 }, +static struct regulator_bulk_data qmp_phy_vreg_refgen[] = { + { .supply = "vdda-phy", .init_load_uA = 21800 }, + { .supply = "vdda-pll", .init_load_uA = 36000 }, + { .supply = "refgen", .init_load_uA = 3270 }, }; static const u8 qmp_dp_v3_pre_emphasis_hbr3_hbr2[4][4] = { @@ -1691,7 +2106,7 @@ static const u8 qmp_dp_v4_pre_emphasis_hbr3_hbr2[4][4] = { static const u8 qmp_dp_v4_pre_emphasis_hbr_rbr[4][4] = { { 0x00, 0x0d, 0x14, 0x1a }, { 0x00, 0x0e, 0x15, 0xff }, - { 0x00, 0x0d, 0xff, 0xff }, + { 0x00, 0x0e, 0xff, 0xff }, { 0x03, 0xff, 0xff, 0xff } }; @@ -1755,6 +2170,7 @@ struct qmp_combo_offsets { u16 usb3_serdes; u16 usb3_pcs_misc; u16 usb3_pcs; + u16 usb3_pcs_aon; u16 usb3_pcs_usb; u16 dp_serdes; u16 dp_txa; @@ -1776,6 +2192,8 @@ struct qmp_phy_cfg { int pcs_tbl_num; const struct qmp_phy_init_tbl *pcs_usb_tbl; int pcs_usb_tbl_num; + const struct qmp_phy_init_tbl *pcs_misc_tbl; + int pcs_misc_tbl_num; const struct qmp_phy_init_tbl *dp_serdes_tbl; int dp_serdes_tbl_num; @@ -1799,6 +2217,7 @@ struct qmp_phy_cfg { const u8 (*pre_emphasis_hbr3_hbr2)[4][4]; /* DP PHY callbacks */ + int (*configure_dp_clocks)(struct qmp_combo *qmp); int (*configure_dp_phy)(struct qmp_combo *qmp); void (*configure_dp_tx)(struct qmp_combo *qmp); int (*calibrate_dp_phy)(struct qmp_combo *qmp); @@ -1808,7 +2227,7 @@ struct qmp_phy_cfg { const char * const *reset_list; int num_resets; /* regulators to be requested */ - const struct qmp_regulator_data *vreg_list; + const struct regulator_bulk_data *vreg_list; int num_vregs; /* array of registers with different offsets */ @@ -1836,6 +2255,7 @@ struct qmp_combo { void __iomem *tx2; void __iomem *rx2; void __iomem *pcs_misc; + void __iomem *pcs_aon; void __iomem *pcs_usb; void __iomem *dp_serdes; @@ -1875,13 +2295,17 @@ struct qmp_combo { static void qmp_v3_dp_aux_init(struct qmp_combo *qmp); static void qmp_v3_configure_dp_tx(struct qmp_combo *qmp); +static int qmp_v3_configure_dp_clocks(struct qmp_combo *qmp); static int qmp_v3_configure_dp_phy(struct qmp_combo *qmp); static int qmp_v3_calibrate_dp_phy(struct qmp_combo *qmp); static void qmp_v4_dp_aux_init(struct qmp_combo *qmp); static void qmp_v4_configure_dp_tx(struct qmp_combo *qmp); static int qmp_v4_configure_dp_phy(struct qmp_combo *qmp); -static int qmp_v4_calibrate_dp_phy(struct qmp_combo *qmp); + +static void qmp_v8_dp_aux_init(struct qmp_combo *qmp); +static int qmp_v8_configure_dp_clocks(struct qmp_combo *qmp); +static int qmp_v8_configure_dp_phy(struct qmp_combo *qmp); static inline void qphy_setbits(void __iomem *base, u32 offset, u32 val) { @@ -1960,6 +2384,7 @@ static const struct qmp_combo_offsets qmp_combo_offsets_v8 = { .usb3_serdes = 0x1000, .usb3_pcs_misc = 0x1c00, .usb3_pcs = 0x1e00, + .usb3_pcs_aon = 0x2000, .usb3_pcs_usb = 0x2100, .dp_serdes = 0x3000, .dp_txa = 0x3400, @@ -1967,6 +2392,19 @@ static const struct qmp_combo_offsets qmp_combo_offsets_v8 = { .dp_dp_phy = 0x3c00, }; +static const struct qmp_combo_offsets qmp_combo_usb43dp_offsets_v8 = { + .com = 0x0000, + .usb3_pcs_aon = 0x0100, + .usb3_serdes = 0x1000, + .usb3_pcs_misc = 0x1400, + .usb3_pcs = 0x1600, + .usb3_pcs_usb = 0x1900, + .dp_serdes = 0x2000, + .dp_dp_phy = 0x2400, + .txa = 0x4000, + .txb = 0x5000, +}; + static const struct qmp_phy_cfg sar2130p_usb3dpphy_cfg = { .offsets = &qmp_combo_offsets_v3, @@ -2002,8 +2440,8 @@ static const struct qmp_phy_cfg sar2130p_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .regs = qmp_v6_usb3phy_regs_layout, .reset_list = msm8996_usb3phy_reset_l, @@ -2137,8 +2575,8 @@ static const struct qmp_phy_cfg sc8180x_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .reset_list = msm8996_usb3phy_reset_l, .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l), @@ -2183,8 +2621,8 @@ static const struct qmp_phy_cfg sc8280xp_usb43dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .reset_list = msm8996_usb3phy_reset_l, .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l), @@ -2228,8 +2666,8 @@ static const struct qmp_phy_cfg x1e80100_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .reset_list = msm8996_usb3phy_reset_l, .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l), @@ -2316,8 +2754,8 @@ static const struct qmp_phy_cfg sm8250_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .reset_list = msm8996_usb3phy_reset_l, .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l), @@ -2364,8 +2802,8 @@ static const struct qmp_phy_cfg sm8350_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .reset_list = msm8996_usb3phy_reset_l, .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l), @@ -2411,8 +2849,8 @@ static const struct qmp_phy_cfg sm8550_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .regs = qmp_v6_usb3phy_regs_layout, .reset_list = msm8996_usb3phy_reset_l, @@ -2456,8 +2894,8 @@ static const struct qmp_phy_cfg sm8650_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .regs = qmp_v6_usb3phy_regs_layout, .reset_list = msm8996_usb3phy_reset_l, @@ -2501,8 +2939,8 @@ static const struct qmp_phy_cfg sm8750_usb3dpphy_cfg = { .dp_aux_init = qmp_v4_dp_aux_init, .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v3_configure_dp_clocks, .configure_dp_phy = qmp_v4_configure_dp_phy, - .calibrate_dp_phy = qmp_v4_calibrate_dp_phy, .regs = qmp_v8_usb3phy_regs_layout, .reset_list = msm8996_usb3phy_reset_l, @@ -2511,6 +2949,51 @@ static const struct qmp_phy_cfg sm8750_usb3dpphy_cfg = { .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l), }; +static const struct qmp_phy_cfg glymur_usb3dpphy_cfg = { + .offsets = &qmp_combo_usb43dp_offsets_v8, + + .serdes_tbl = glymur_usb43dp_serdes_tbl, + .serdes_tbl_num = ARRAY_SIZE(glymur_usb43dp_serdes_tbl), + .tx_tbl = glymur_usb43dp_lalb_tbl, + .tx_tbl_num = ARRAY_SIZE(glymur_usb43dp_lalb_tbl), + .pcs_tbl = glymur_usb43dp_pcs_tbl, + .pcs_tbl_num = ARRAY_SIZE(glymur_usb43dp_pcs_tbl), + .pcs_usb_tbl = glymur_usb43dp_pcs_usb_tbl, + .pcs_usb_tbl_num = ARRAY_SIZE(glymur_usb43dp_pcs_usb_tbl), + .pcs_misc_tbl = glymur_usb43dp_pcs_misc_tbl, + .pcs_misc_tbl_num = ARRAY_SIZE(glymur_usb43dp_pcs_misc_tbl), + + .dp_serdes_tbl = qmp_v8_dp_serdes_tbl, + .dp_serdes_tbl_num = ARRAY_SIZE(qmp_v8_dp_serdes_tbl), + .dp_tx_tbl = qmp_v8_n3p_dp_tx_tbl, + .dp_tx_tbl_num = ARRAY_SIZE(qmp_v8_n3p_dp_tx_tbl), + + .serdes_tbl_rbr = qmp_v8_dp_serdes_tbl_rbr, + .serdes_tbl_rbr_num = ARRAY_SIZE(qmp_v8_dp_serdes_tbl_rbr), + .serdes_tbl_hbr = qmp_v8_dp_serdes_tbl_hbr, + .serdes_tbl_hbr_num = ARRAY_SIZE(qmp_v8_dp_serdes_tbl_hbr), + .serdes_tbl_hbr2 = qmp_v8_dp_serdes_tbl_hbr2, + .serdes_tbl_hbr2_num = ARRAY_SIZE(qmp_v8_dp_serdes_tbl_hbr2), + .serdes_tbl_hbr3 = qmp_v8_dp_serdes_tbl_hbr3, + .serdes_tbl_hbr3_num = ARRAY_SIZE(qmp_v8_dp_serdes_tbl_hbr3), + + .swing_hbr_rbr = &qmp_dp_v6_voltage_swing_hbr_rbr, + .pre_emphasis_hbr_rbr = &qmp_dp_v6_pre_emphasis_hbr_rbr, + .swing_hbr3_hbr2 = &qmp_dp_v5_voltage_swing_hbr3_hbr2, + .pre_emphasis_hbr3_hbr2 = &qmp_dp_v5_pre_emphasis_hbr3_hbr2, + + .dp_aux_init = qmp_v8_dp_aux_init, + .configure_dp_tx = qmp_v4_configure_dp_tx, + .configure_dp_clocks = qmp_v8_configure_dp_clocks, + .configure_dp_phy = qmp_v8_configure_dp_phy, + + .regs = qmp_v8_n3_usb43dpphy_regs_layout, + .reset_list = msm8996_usb3phy_reset_l, + .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l), + .vreg_list = qmp_phy_vreg_refgen, + .num_vregs = ARRAY_SIZE(qmp_phy_vreg_refgen), +}; + static int qmp_combo_dp_serdes_init(struct qmp_combo *qmp) { const struct qmp_phy_cfg *cfg = qmp->cfg; @@ -2673,7 +3156,7 @@ static bool qmp_combo_configure_dp_mode(struct qmp_combo *qmp) return reverse; } -static int qmp_combo_configure_dp_clocks(struct qmp_combo *qmp) +static int qmp_v3_configure_dp_clocks(struct qmp_combo *qmp) { const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts; u32 phy_vco_div; @@ -2720,7 +3203,7 @@ static int qmp_v3_configure_dp_phy(struct qmp_combo *qmp) writel(0x05, qmp->dp_dp_phy + QSERDES_V3_DP_PHY_TX0_TX1_LANE_CTL); writel(0x05, qmp->dp_dp_phy + QSERDES_V3_DP_PHY_TX2_TX3_LANE_CTL); - ret = qmp_combo_configure_dp_clocks(qmp); + ret = qmp_v3_configure_dp_clocks(qmp); if (ret) return ret; @@ -2798,7 +3281,34 @@ static void qmp_v4_dp_aux_init(struct qmp_combo *qmp) writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG7); writel(0xb7, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG8); writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG9); - qmp->dp_aux_cfg = 0; + + writel(PHY_AUX_STOP_ERR_MASK | PHY_AUX_DEC_ERR_MASK | + PHY_AUX_SYNC_ERR_MASK | PHY_AUX_ALIGN_ERR_MASK | + PHY_AUX_REQ_ERR_MASK, + qmp->dp_dp_phy + QSERDES_V4_DP_PHY_AUX_INTERRUPT_MASK); +} + +static void qmp_v8_dp_aux_init(struct qmp_combo *qmp) +{ + const struct qmp_phy_cfg *cfg = qmp->cfg; + + writel(DP_PHY_PD_CTL_PWRDN | DP_PHY_PD_CTL_PSR_PWRDN | DP_PHY_PD_CTL_AUX_PWRDN | + DP_PHY_PD_CTL_PLL_PWRDN | DP_PHY_PD_CTL_DP_CLAMP_EN, + qmp->dp_dp_phy + QSERDES_DP_PHY_PD_CTL); + + /* Turn on BIAS current for PHY/PLL */ + writel(0x1c, qmp->dp_serdes + cfg->regs[QPHY_COM_BIAS_EN_CLKBUFLR_EN]); + + writel(0x00, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG0); + writel(0x13, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG1); + writel(0x06, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG2); + writel(0x00, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG3); + writel(0x0a, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG4); + writel(0x26, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG5); + writel(0x0a, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG6); + writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG7); + writel(0xb7, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG8); + writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG9); writel(PHY_AUX_STOP_ERR_MASK | PHY_AUX_DEC_ERR_MASK | PHY_AUX_SYNC_ERR_MASK | PHY_AUX_ALIGN_ERR_MASK | @@ -2820,6 +3330,58 @@ static void qmp_v4_configure_dp_tx(struct qmp_combo *qmp) qmp_combo_configure_dp_swing(qmp); } +static int qmp_v8_configure_dp_clocks(struct qmp_combo *qmp) +{ + const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts; + u32 phy_vco_div; + unsigned long pixel_freq; + const struct qmp_phy_cfg *cfg = qmp->cfg; + + switch (dp_opts->link_rate) { + case 1620: + phy_vco_div = 0x4; + pixel_freq = 1620000000UL / 2; + break; + case 2700: + phy_vco_div = 0x2; + pixel_freq = 2700000000UL / 2; + break; + case 5400: + phy_vco_div = 0x4; + pixel_freq = 5400000000UL / 4; + break; + case 8100: + phy_vco_div = 0x3; + pixel_freq = 8100000000UL / 6; + break; + default: + /* Other link rates aren't supported */ + return -EINVAL; + } + writel(phy_vco_div, qmp->dp_dp_phy + cfg->regs[QPHY_DP_PHY_VCO_DIV]); + + /* disable core reset tsync */ + writel(0x09, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG); + + writel(0x04, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_AUXLESS_SETUP_CYC); + writel(0x08, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_AUXLESS_SILENCE_CYC); + writel(0x08, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_LFPS_CYC); + writel(0x11, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_LFPS_PERIOD); + + writel(0x3e, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_TSYNC_OVRD); + writel(0x05, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_TX2_TX3_LANE_CTL); + writel(0x05, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_TX0_TX1_LANE_CTL); + writel(0x01, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_AUXLESS_CFG1); + writel(0x11, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_LFPS_PERIOD); + writel(0x1f, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_LN0_DRV_LVL); + writel(0x1f, qmp->dp_dp_phy + QSERDES_V8_DP_PHY_LN1_DRV_LVL); + + clk_set_rate(qmp->dp_link_hw.clk, dp_opts->link_rate * 100000); + clk_set_rate(qmp->dp_pixel_hw.clk, pixel_freq); + + return 0; +} + static int qmp_v456_configure_dp_phy(struct qmp_combo *qmp) { const struct qmp_phy_cfg *cfg = qmp->cfg; @@ -2836,7 +3398,7 @@ static int qmp_v456_configure_dp_phy(struct qmp_combo *qmp) writel(0x05, qmp->dp_dp_phy + QSERDES_V4_DP_PHY_TX0_TX1_LANE_CTL); writel(0x05, qmp->dp_dp_phy + QSERDES_V4_DP_PHY_TX2_TX3_LANE_CTL); - ret = qmp_combo_configure_dp_clocks(qmp); + ret = qmp->cfg->configure_dp_clocks(qmp); if (ret) return ret; @@ -2950,20 +3512,58 @@ static int qmp_v4_configure_dp_phy(struct qmp_combo *qmp) return 0; } -/* - * We need to calibrate the aux setting here as many times - * as the caller tries - */ -static int qmp_v4_calibrate_dp_phy(struct qmp_combo *qmp) +static int qmp_v8_configure_dp_phy(struct qmp_combo *qmp) { - static const u8 cfg1_settings[] = { 0x20, 0x13, 0x23, 0x1d }; - u8 val; + const struct qmp_phy_cfg *cfg = qmp->cfg; + bool reverse = (qmp->orientation == TYPEC_ORIENTATION_REVERSE); + const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts; + u32 bias0_en, drvr0_en, bias1_en, drvr1_en; + u32 status; + int ret; - qmp->dp_aux_cfg++; - qmp->dp_aux_cfg %= ARRAY_SIZE(cfg1_settings); - val = cfg1_settings[qmp->dp_aux_cfg]; + ret = qmp_v456_configure_dp_phy(qmp); + if (ret < 0) + return ret; - writel(val, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG1); + if (dp_opts->lanes == 1) { + bias0_en = reverse ? 0x3e : 0x15; + bias1_en = reverse ? 0x15 : 0x3e; + drvr0_en = reverse ? 0x13 : 0x10; + drvr1_en = reverse ? 0x10 : 0x13; + } else if (dp_opts->lanes == 2) { + bias0_en = reverse ? 0x3f : 0x15; + bias1_en = reverse ? 0x15 : 0x3f; + drvr0_en = 0x10; + drvr1_en = 0x10; + } else { + bias0_en = 0x3f; + bias1_en = 0x3f; + drvr0_en = 0x34; + drvr1_en = 0x34; + } + + writel(drvr0_en, qmp->dp_tx + cfg->regs[QPHY_TX_HIGHZ_DRVR_EN]); + writel(bias0_en, qmp->dp_tx + cfg->regs[QPHY_TX_TRANSCEIVER_BIAS_EN]); + writel(drvr1_en, qmp->dp_tx2 + cfg->regs[QPHY_TX_HIGHZ_DRVR_EN]); + writel(bias1_en, qmp->dp_tx2 + cfg->regs[QPHY_TX_TRANSCEIVER_BIAS_EN]); + + writel(0x08, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG); + udelay(100); + writel(0x09, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG); + udelay(500); + + if (readl_poll_timeout(qmp->dp_dp_phy + cfg->regs[QPHY_DP_PHY_STATUS], + status, + ((status & BIT(1)) > 0), + 500, + 10000)) + return -ETIMEDOUT; + + writel(0x00, qmp->dp_tx + cfg->regs[QPHY_TX_TX_DRV_LVL]); + writel(0x00, qmp->dp_tx2 + cfg->regs[QPHY_TX_TX_DRV_LVL]); + + writel(0x2b, qmp->dp_tx + cfg->regs[QPHY_TX_TX_EMP_POST1_LVL]); + writel(0x2b, qmp->dp_tx2 + cfg->regs[QPHY_TX_TX_EMP_POST1_LVL]); return 0; } @@ -3007,6 +3607,7 @@ static int qmp_combo_com_init(struct qmp_combo *qmp, bool force) { const struct qmp_phy_cfg *cfg = qmp->cfg; void __iomem *com = qmp->com; + void __iomem *pcs_aon = qmp->pcs_aon; int ret; u32 val; @@ -3042,6 +3643,12 @@ static int qmp_combo_com_init(struct qmp_combo *qmp, bool force) SW_DPPHY_RESET_MUX | SW_DPPHY_RESET | SW_USB3PHY_RESET_MUX | SW_USB3PHY_RESET); + /* override hardware control for reset of qmp phy */ + if (pcs_aon && cfg->regs[QPHY_AON_TOGGLE_ENABLE]) { + qphy_clrbits(pcs_aon, cfg->regs[QPHY_AON_TOGGLE_ENABLE], 0x1); + qphy_clrbits(pcs_aon, cfg->regs[QPHY_DP_AON_TOGGLE_ENABLE], 0x1); + } + /* Use software based port select and switch on typec orientation */ val = SW_PORTSELECT_MUX; if (qmp->orientation == TYPEC_ORIENTATION_REVERSE) @@ -3219,6 +3826,8 @@ static int qmp_combo_usb_power_on(struct phy *phy) qmp_configure_lane(qmp->dev, rx2, cfg->rx_tbl, cfg->rx_tbl_num, 2); qmp_configure(qmp->dev, pcs, cfg->pcs_tbl, cfg->pcs_tbl_num); + qmp_configure(qmp->dev, qmp->pcs_misc, cfg->pcs_misc_tbl, cfg->pcs_misc_tbl_num); + if (pcs_usb) qmp_configure(qmp->dev, pcs_usb, cfg->pcs_usb_tbl, @@ -3345,6 +3954,7 @@ static void qmp_combo_enable_autonomous_mode(struct qmp_combo *qmp) const struct qmp_phy_cfg *cfg = qmp->cfg; void __iomem *pcs_usb = qmp->pcs_usb ?: qmp->pcs; void __iomem *pcs_misc = qmp->pcs_misc; + void __iomem *pcs_aon = qmp->pcs_aon; u32 intr_mask; if (qmp->phy_mode == PHY_MODE_USB_HOST_SS || @@ -3364,9 +3974,14 @@ static void qmp_combo_enable_autonomous_mode(struct qmp_combo *qmp) /* Enable required PHY autonomous mode interrupts */ qphy_setbits(pcs_usb, cfg->regs[QPHY_PCS_AUTONOMOUS_MODE_CTRL], intr_mask); - /* Enable i/o clamp_n for autonomous mode */ - if (pcs_misc) - qphy_clrbits(pcs_misc, QPHY_V3_PCS_MISC_CLAMP_ENABLE, CLAMP_EN); + /* + * Enable i/o clamp_n for autonomous mode + * V6 and later versions use pcs aon clamp register + */ + if (pcs_aon) + qphy_clrbits(pcs_aon, cfg->regs[QPHY_PCS_CLAMP_ENABLE], CLAMP_EN); + else if (pcs_misc) + qphy_clrbits(pcs_misc, cfg->regs[QPHY_PCS_CLAMP_ENABLE], CLAMP_EN); } static void qmp_combo_disable_autonomous_mode(struct qmp_combo *qmp) @@ -3374,10 +3989,13 @@ static void qmp_combo_disable_autonomous_mode(struct qmp_combo *qmp) const struct qmp_phy_cfg *cfg = qmp->cfg; void __iomem *pcs_usb = qmp->pcs_usb ?: qmp->pcs; void __iomem *pcs_misc = qmp->pcs_misc; + void __iomem *pcs_aon = qmp->pcs_aon; /* Disable i/o clamp_n on resume for normal mode */ - if (pcs_misc) - qphy_setbits(pcs_misc, QPHY_V3_PCS_MISC_CLAMP_ENABLE, CLAMP_EN); + if (pcs_aon) + qphy_setbits(pcs_aon, cfg->regs[QPHY_PCS_CLAMP_ENABLE], CLAMP_EN); + else if (pcs_misc) + qphy_setbits(pcs_misc, cfg->regs[QPHY_PCS_CLAMP_ENABLE], CLAMP_EN); qphy_clrbits(pcs_usb, cfg->regs[QPHY_PCS_AUTONOMOUS_MODE_CTRL], ARCVR_DTCT_EN | ARCVR_DTCT_EVENT_SEL | ALFPS_DTCT_EN); @@ -3439,39 +4057,6 @@ static const struct dev_pm_ops qmp_combo_pm_ops = { qmp_combo_runtime_resume, NULL) }; -static int qmp_combo_vreg_init(struct qmp_combo *qmp) -{ - const struct qmp_phy_cfg *cfg = qmp->cfg; - struct device *dev = qmp->dev; - int num = cfg->num_vregs; - int ret, i; - - qmp->vregs = devm_kcalloc(dev, num, sizeof(*qmp->vregs), GFP_KERNEL); - if (!qmp->vregs) - return -ENOMEM; - - for (i = 0; i < num; i++) - qmp->vregs[i].supply = cfg->vreg_list[i].name; - - ret = devm_regulator_bulk_get(dev, num, qmp->vregs); - if (ret) { - dev_err(dev, "failed at devm_regulator_bulk_get\n"); - return ret; - } - - for (i = 0; i < num; i++) { - ret = regulator_set_load(qmp->vregs[i].consumer, - cfg->vreg_list[i].enable_load); - if (ret) { - dev_err(dev, "failed to set load at %s\n", - qmp->vregs[i].supply); - return ret; - } - } - - return 0; -} - static int qmp_combo_reset_init(struct qmp_combo *qmp) { const struct qmp_phy_cfg *cfg = qmp->cfg; @@ -4075,6 +4660,8 @@ static int qmp_combo_parse_dt(struct qmp_combo *qmp) qmp->serdes = base + offs->usb3_serdes; qmp->pcs_misc = base + offs->usb3_pcs_misc; qmp->pcs = base + offs->usb3_pcs; + if (offs->usb3_pcs_aon) + qmp->pcs_aon = base + offs->usb3_pcs_aon; qmp->pcs_usb = base + offs->usb3_pcs_usb; qmp->dp_serdes = base + offs->dp_serdes; @@ -4144,7 +4731,8 @@ static int qmp_combo_probe(struct platform_device *pdev) if (ret) return ret; - ret = qmp_combo_vreg_init(qmp); + ret = devm_regulator_bulk_get_const(dev, qmp->cfg->num_vregs, + qmp->cfg->vreg_list, &qmp->vregs); if (ret) return ret; @@ -4230,6 +4818,10 @@ static int qmp_combo_probe(struct platform_device *pdev) } static const struct of_device_id qmp_combo_of_match_table[] = { + { + .compatible = "qcom,glymur-qmp-usb3-dp-phy", + .data = &glymur_usb3dpphy_cfg, + }, { .compatible = "qcom,sar2130p-qmp-usb3-dp-phy", .data = &sar2130p_usb3dpphy_cfg, diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-dp-phy-v8.h b/drivers/phy/qualcomm/phy-qcom-qmp-dp-phy-v8.h new file mode 100644 index 000000000000..b6a8ab59c2ff --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-dp-phy-v8.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2017, The Linux Foundation. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_DP_PHY_V8_H_ +#define QCOM_PHY_QMP_DP_PHY_V8_H_ + +/* Only for QMP V8 PHY - DP PHY registers */ +#define QSERDES_V8_DP_PHY_VCO_DIV 0x070 +#define QSERDES_V8_DP_PHY_AUX_INTERRUPT_STATUS 0x0e0 +#define QSERDES_V8_DP_PHY_TSYNC_OVRD 0x074 +#define QSERDES_V8_DP_PHY_TX0_TX1_LANE_CTL 0x078 +#define QSERDES_V8_DP_PHY_TX2_TX3_LANE_CTL 0x0bc +#define QSERDES_V8_DP_PHY_AUXLESS_CFG1 0x0c8 +#define QSERDES_V8_DP_PHY_LFPS_PERIOD 0x0d0 +#define QSERDES_V8_DP_PHY_LFPS_CYC 0x0d4 +#define QSERDES_V8_DP_PHY_AUXLESS_SETUP_CYC 0x0d8 +#define QSERDES_V8_DP_PHY_AUXLESS_SILENCE_CYC 0x0d8 +#define QSERDES_V8_DP_PHY_LN0_DRV_LVL 0x0e0 +#define QSERDES_V8_DP_PHY_LN1_DRV_LVL 0x0e4 +#define QSERDES_V8_DP_PHY_STATUS 0x114 + + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-dp-qserdes-com-v8.h b/drivers/phy/qualcomm/phy-qcom-qmp-dp-qserdes-com-v8.h new file mode 100644 index 000000000000..2bef1eecdc56 --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-dp-qserdes-com-v8.h @@ -0,0 +1,52 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2025 Linaro Ltd. + */ + +#ifndef QCOM_PHY_QMP_DP_QSERDES_COM_V8_H_ +#define QCOM_PHY_QMP_DP_QSERDES_COM_V8_H_ + +/* Only for DP QMP V8 PHY - QSERDES COM registers */ +#define DP_QSERDES_V8_COM_HSCLK_SEL_1 0x03c +#define DP_QSERDES_V8_COM_BIN_VCOCAL_CMP_CODE1_MODE0 0x058 +#define DP_QSERDES_V8_COM_BIN_VCOCAL_CMP_CODE2_MODE0 0x05c +#define DP_QSERDES_V8_COM_SSC_STEP_SIZE1_MODE0 0x060 +#define DP_QSERDES_V8_COM_SSC_STEP_SIZE2_MODE0 0x064 +#define DP_QSERDES_V8_COM_CP_CTRL_MODE0 0x070 +#define DP_QSERDES_V8_COM_PLL_RCTRL_MODE0 0x074 +#define DP_QSERDES_V8_COM_PLL_CCTRL_MODE0 0x078 +#define DP_QSERDES_V8_COM_CORECLK_DIV_MODE0 0x07c +#define DP_QSERDES_V8_COM_LOCK_CMP1_MODE0 0x080 +#define DP_QSERDES_V8_COM_LOCK_CMP2_MODE0 0x084 +#define DP_QSERDES_V8_COM_DEC_START_MODE0 0x088 +#define DP_QSERDES_V8_COM_DIV_FRAC_START1_MODE0 0x090 +#define DP_QSERDES_V8_COM_DIV_FRAC_START2_MODE0 0x094 +#define DP_QSERDES_V8_COM_DIV_FRAC_START3_MODE0 0x098 +#define DP_QSERDES_V8_COM_INTEGLOOP_GAIN0_MODE0 0x0a0 +#define DP_QSERDES_V8_COM_VCO_TUNE1_MODE0 0x0a8 +#define DP_QSERDES_V8_COM_INTEGLOOP_GAIN1_MODE0 0x0a4 +#define DP_QSERDES_V8_COM_VCO_TUNE2_MODE0 0x0ac +#define DP_QSERDES_V8_COM_BG_TIMER 0x0bc +#define DP_QSERDES_V8_COM_SSC_EN_CENTER 0x0c0 +#define DP_QSERDES_V8_COM_SSC_ADJ_PER1 0x0c4 +#define DP_QSERDES_V8_COM_SSC_PER1 0x0cc +#define DP_QSERDES_V8_COM_SSC_PER2 0x0d0 +#define DP_QSERDES_V8_COM_BIAS_EN_CLKBUFLR_EN 0x0dc +#define DP_QSERDES_V8_COM_CLK_ENABLE1 0x0e0 +#define DP_QSERDES_V8_COM_SYS_CLK_CTRL 0x0e4 +#define DP_QSERDES_V8_COM_SYSCLK_BUF_ENABLE 0x0e8 +#define DP_QSERDES_V8_COM_PLL_IVCO 0x0f4 +#define DP_QSERDES_V8_COM_SYSCLK_EN_SEL 0x110 +#define DP_QSERDES_V8_COM_RESETSM_CNTRL 0x118 +#define DP_QSERDES_V8_COM_LOCK_CMP_EN 0x120 +#define DP_QSERDES_V8_COM_VCO_TUNE_CTRL 0x13c +#define DP_QSERDES_V8_COM_VCO_TUNE_MAP 0x140 +#define DP_QSERDES_V8_COM_CLK_SELECT 0x164 +#define DP_QSERDES_V8_COM_CORE_CLK_EN 0x170 +#define DP_QSERDES_V8_COM_CMN_CONFIG_1 0x174 +#define DP_QSERDES_V8_COM_SVS_MODE_CLK_SEL 0x180 +#define DP_QSERDES_V8_COM_CLK_FWD_CONFIG_1 0x2f4 +#define DP_QSERDES_V8_COM_CMN_STATUS 0x314 +#define DP_QSERDES_V8_COM_C_READY_STATUS 0x33c + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c b/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c index 62b1c845b627..456cdb6e7213 100644 --- a/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c +++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c @@ -3288,6 +3288,7 @@ static const struct qmp_pcie_offsets qmp_pcie_offsets_v5 = { .pcs_misc = 0x0600, .tx = 0x0e00, .rx = 0x1000, + .pcs_lane1 = 0x1400, .tx2 = 0x1600, .rx2 = 0x1800, }; diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-aon-v6.h b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-aon-v6.h new file mode 100644 index 000000000000..52db31a7cf22 --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-aon-v6.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025, Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_PCS_AON_V6_H_ +#define QCOM_PHY_QMP_PCS_AON_V6_H_ + +/* Only for QMP V6 PHY - PCS_AON registers */ +#define QPHY_V6_PCS_AON_CLAMP_ENABLE 0x00 + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-aon-v8.h b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-aon-v8.h new file mode 100644 index 000000000000..f6a275c0938f --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-aon-v8.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025, Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_PCS_AON_V8_H_ +#define QCOM_PHY_QMP_PCS_AON_V8_H_ + +/* Only for QMP V8 PHY - PCS_AON registers */ +#define QPHY_V8_PCS_AON_USB3_AON_CLAMP_ENABLE 0x00 +#define QPHY_V8_PCS_AON_USB4_AON_CLAMP_ENABLE 0x04 +#define QPHY_V8_PCS_AON_USB3_AON_TOGGLE_ENABLE 0x08 +#define QPHY_V8_PCS_AON_USB4_AON_TOGGLE_ENABLE 0x0c +#define QPHY_V8_PCS_AON_DP_AON_TOGGLE_ENABLE 0x10 +#define QPHY_V8_PCS_AON_DUMMY_STATUS 0x14 + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-misc-v5.h b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-misc-v5.h new file mode 100644 index 000000000000..77d04c6a1644 --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-misc-v5.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_PCS_MISC_V5_H_ +#define QCOM_PHY_QMP_PCS_MISC_V5_H_ + +/* Only for QMP V5 PHY - PCS_MISC registers */ +#define QPHY_V5_PCS_MISC_CLAMP_ENABLE 0x0c + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-misc-v8.h b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-misc-v8.h new file mode 100644 index 000000000000..a93ef2faa894 --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-misc-v8.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_PCS_MISC_V8_H_ +#define QCOM_PHY_QMP_PCS_MISC_V8_H_ + +/* Only for QMP V8 PHY - PCS_MISC registers */ +#define QPHY_V8_PCS_MISC_PCS_MISC_CONFIG1 0x08 + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-qserdes-lalb-v8.h b/drivers/phy/qualcomm/phy-qcom-qmp-qserdes-lalb-v8.h new file mode 100644 index 000000000000..60ba730620f8 --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-qserdes-lalb-v8.h @@ -0,0 +1,639 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_QSERDES_V8_LALBH_ +#define QCOM_PHY_QMP_QSERDES_V8_LALBH_ + +#define QSERDES_V8_LALB_BIST_MODE_LANENO 0x0 +#define QSERDES_V8_LALB_BIST_INVERT 0x4 +#define QSERDES_V8_LALB_PERL_LENGTH1 0x8 +#define QSERDES_V8_LALB_PERL_LENGTH2 0xc +#define QSERDES_V8_LALB_BIST_PATTERN1 0x10 +#define QSERDES_V8_LALB_BIST_PATTERN2 0x14 +#define QSERDES_V8_LALB_BIST_PATTERN3 0x18 +#define QSERDES_V8_LALB_BIST_PATTERN4 0x1c +#define QSERDES_V8_LALB_BIST_PATTERN5 0x20 +#define QSERDES_V8_LALB_BIST_PATTERN6 0x24 +#define QSERDES_V8_LALB_BIST_PATTERN7 0x28 +#define QSERDES_V8_LALB_BIST_PATTERN8 0x2c +#define QSERDES_V8_LALB_PRBS_SEED1 0x30 +#define QSERDES_V8_LALB_PRBS_SEED2 0x34 +#define QSERDES_V8_LALB_PRBS_SEED3 0x38 +#define QSERDES_V8_LALB_PRBS_SEED4 0x3c +#define QSERDES_V8_LALB_PRBS_SEED5 0x40 +#define QSERDES_V8_LALB_PRBS_SEED6 0x44 +#define QSERDES_V8_LALB_PRBS_SEED7 0x48 +#define QSERDES_V8_LALB_SW_RESET_PWRDNB 0x4c +#define QSERDES_V8_LALB_RESET_GEN 0x50 +#define QSERDES_V8_LALB_RESET_TSYNC_EN_CTRL 0x54 +#define QSERDES_V8_LALB_CDR_EN_RXEQ_RESET 0x58 +#define QSERDES_V8_LALB_CLKBUF_ENABLE 0x5c +#define QSERDES_V8_LALB_TX0_EMP_POST1_LVL 0x60 +#define QSERDES_V8_LALB_TX1_EMP_POST1_LVL 0x64 +#define QSERDES_V8_LALB_TX0_IDLE_CTRL 0x68 +#define QSERDES_V8_LALB_TX1_IDLE_CTRL 0x6c +#define QSERDES_V8_LALB_TX0_DRV_LVL 0x70 +#define QSERDES_V8_LALB_TX0_DRV_LVL_OFFSET 0x74 +#define QSERDES_V8_LALB_TX1_DRV_LVL 0x78 +#define QSERDES_V8_LALB_TX1_DRV_LVL_OFFSET 0x7c +#define QSERDES_V8_LALB_TRAN_DRVR_EMP_EN 0x80 +#define QSERDES_V8_LALB_TX_LVL_UPDATE_CTRL 0x84 +#define QSERDES_V8_LALB_TX0_PRE1_EMPH 0x88 +#define QSERDES_V8_LALB_TX1_PRE1_EMPH 0x8c +#define QSERDES_V8_LALB_TX0_PRE2_EMPH 0x90 +#define QSERDES_V8_LALB_TX1_PRE2_EMPH 0x94 +#define QSERDES_V8_LALB_STALL_LDO_BOOST_EN 0x98 +#define QSERDES_V8_LALB_PRE_EMPH_EN_CTRL 0x9c +#define QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL1 0xa0 +#define QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL2 0xa4 +#define QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL3 0xa8 +#define QSERDES_V8_LALB_PCIE5_TOP_LDO_CODE_CTRL4 0xac +#define QSERDES_V8_LALB_TRANSMITTER_EN_CTRL 0xb0 +#define QSERDES_V8_LALB_HIGHZ_DRVR_EN 0xb4 +#define QSERDES_V8_LALB_TX_MISC_CTRL1 0xb8 +#define QSERDES_V8_LALB_LPB_EN_CTRL1 0xbc +#define QSERDES_V8_LALB_LBP_EN_CTRL2 0xc0 +#define QSERDES_V8_LALB_TX0_SERDES_BYP_CTRL 0xc4 +#define QSERDES_V8_LALB_TX1_SERDES_BYP_CTRL 0xc8 +#define QSERDES_V8_LALB_LANE_MODE_1 0xcc +#define QSERDES_V8_LALB_LANE_MODE_2 0xd0 +#define QSERDES_V8_LALB_LANE_MODE_3 0xd4 +#define QSERDES_V8_LALB_LANE_MODE_4 0xd8 +#define QSERDES_V8_LALB_ATB_SEL1 0xdc +#define QSERDES_V8_LALB_ATB_SEL2 0xe0 +#define QSERDES_V8_LALB_TX0_RES_CODE_LANE 0xe4 +#define QSERDES_V8_LALB_TX0_RESTRIM_ICAL_OVRD 0xe8 +#define QSERDES_V8_LALB_TX0_RESTRIM_CAL_CTRL 0xec +#define QSERDES_V8_LALB_TX0_RESTRIM_INIT_CODE 0xf0 +#define QSERDES_V8_LALB_TX0_RESTRIM_POST_CAL_OFFSET 0xf4 +#define QSERDES_V8_LALB_TX1_RES_CODE_LANE 0xf8 +#define QSERDES_V8_LALB_TX1_RESTRIM_ICAL_OVRD 0xfc +#define QSERDES_V8_LALB_TX1_RESTRIM_CAL_CTRL 0x100 +#define QSERDES_V8_LALB_TX1_RESTRIM_INIT_CODE 0x104 +#define QSERDES_V8_LALB_TX1_RESTRIM_POST_CAL_OFFSET 0x108 +#define QSERDES_V8_LALB_TX0_RESTRIM_VREF_SEL 0x10c +#define QSERDES_V8_LALB_TX1_RESTRIM_VREF_SEL 0x110 +#define QSERDES_V8_LALB_VMODE_CTRL1 0x114 +#define QSERDES_V8_LALB_SLEW_CNTL_RATE01 0x118 +#define QSERDES_V8_LALB_SLEW_CNTL_RATE23 0x11c +#define QSERDES_V8_LALB_SLEW_CNTL_RATE4 0x120 +#define QSERDES_V8_LALB_ANA_INTERFACE_SELECT1 0x124 +#define QSERDES_V8_LALB_ANA_INTERFACE_SELECT2 0x128 +#define QSERDES_V8_LALB_ANA_INTERFACE_SELECT3 0x12c +#define QSERDES_V8_LALB_PCS_INTERFACE_SELECT1 0x130 +#define QSERDES_V8_LALB_PCS_INTERFACE_SELECT2 0x134 +#define QSERDES_V8_LALB_LDO_TIMER_CTRL 0x138 +#define QSERDES_V8_LALB_AC_JTAG_ENABLE 0x13c +#define QSERDES_V8_LALB_AC_JTAG_INITP 0x140 +#define QSERDES_V8_LALB_AC_JTAG_INITN 0x144 +#define QSERDES_V8_LALB_AC_JTAG_LVL 0x148 +#define QSERDES_V8_LALB_AC_JTAG_MODE 0x14c +#define QSERDES_V8_LALB_AC_JTAG_RESET 0x150 +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B0 0x154 +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B1 0x158 +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B2 0x15c +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B3 0x160 +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B4 0x164 +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B5 0x168 +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B6 0x16c +#define QSERDES_V8_LALB_RX_MODE_RATE_0_1_B7 0x170 +#define QSERDES_V8_LALB_RX_MODE_RATE2_B0 0x174 +#define QSERDES_V8_LALB_RX_MODE_RATE2_B1 0x178 +#define QSERDES_V8_LALB_RX_MODE_RATE2_B2 0x17c +#define QSERDES_V8_LALB_RX_MODE_RATE2_B3 0x180 +#define QSERDES_V8_LALB_RX_MODE_RATE2_B4 0x184 +#define QSERDES_V8_LALB_RX_MODE_RATE2_B5 0x188 +#define QSERDES_V8_LALB_RX_MODE_RATE2_B6 0x18c +#define QSERDES_V8_LALB_RX_MODE_RATE2_B7 0x190 +#define QSERDES_V8_LALB_RX_MODE_RATE3_B0 0x194 +#define QSERDES_V8_LALB_RX_MODE_RATE3_B1 0x198 +#define QSERDES_V8_LALB_RX_MODE_RATE3_B2 0x19c +#define QSERDES_V8_LALB_RX_MODE_RATE3_B3 0x1a0 +#define QSERDES_V8_LALB_RX_MODE_RATE3_B4 0x1a4 +#define QSERDES_V8_LALB_RX_MODE_RATE3_B5 0x1a8 +#define QSERDES_V8_LALB_RX_MODE_RATE3_B6 0x1ac +#define QSERDES_V8_LALB_RX_MODE_RATE3_B7 0x1b0 +#define QSERDES_V8_LALB_RX_MODE_RATE4_B0 0x1b4 +#define QSERDES_V8_LALB_RX_MODE_RATE4_B1 0x1b8 +#define QSERDES_V8_LALB_RX_MODE_RATE4_B2 0x1bc +#define QSERDES_V8_LALB_RX_MODE_RATE4_B3 0x1c0 +#define QSERDES_V8_LALB_RX_MODE_RATE4_B4 0x1c4 +#define QSERDES_V8_LALB_RX_MODE_RATE4_B5 0x1c8 +#define QSERDES_V8_LALB_RX_MODE_RATE4_B6 0x1cc +#define QSERDES_V8_LALB_RX_MODE_RATE4_B7 0x1d0 +#define QSERDES_V8_LALB_TX_DCC_ANA_CTRL1 0x1d4 +#define QSERDES_V8_LALB_TX_DCC_ANA_CTRL2 0x1d8 +#define QSERDES_V8_LALB_CMUX_DCC_CTRL1 0x1dc +#define QSERDES_V8_LALB_CMUX_DCC_POSTCAL_OFFSET 0x1e0 +#define QSERDES_V8_LALB_CMUX_DCC_OVRD 0x1e4 +#define QSERDES_V8_LALB_TX_DCC_CTRL 0x1e8 +#define QSERDES_V8_LALB_TX0_CTUNE_DCC_CONFIG 0x1ec +#define QSERDES_V8_LALB_TX0_CTUNE_DCC_POSTCAL_OFFSET 0x1f0 +#define QSERDES_V8_LALB_TX0_CTUNE_DCC_OVRD 0x1f4 +#define QSERDES_V8_LALB_TX0_FTUNE_MSB_DCC_CONFIG 0x1f8 +#define QSERDES_V8_LALB_TX0_FTUNE_MSB_DCC_OFFSET_AND_OVRD 0x1fc +#define QSERDES_V8_LALB_TX0_FTUNE_LSB_DCC_CONFIG 0x200 +#define QSERDES_V8_LALB_TX0_FTUNE_LSB_DCC_OFFSET_AND_OVRD 0x204 +#define QSERDES_V8_LALB_TX1_CTUNE_DCC_CONFIG 0x208 +#define QSERDES_V8_LALB_TX1_CTUNE_DCC_POSTCAL_OFFSET 0x20c +#define QSERDES_V8_LALB_TX1_CTUNE_DCC_OVRD 0x210 +#define QSERDES_V8_LALB_TX1_FTUNE_MSB_DCC_CONFIG 0x214 +#define QSERDES_V8_LALB_TX1_FTUNE_MSB_DCC_OFFSET_AND_OVRD 0x218 +#define QSERDES_V8_LALB_TX1_FTUNE_LSB_DCC_CONFIG 0x21c +#define QSERDES_V8_LALB_TX1_FTUNE_LSB_DCC_OFFSET_AND_OVRD 0x220 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_CTRL 0x224 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_CODE_OVRD_RATE0 0x228 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_CODE_OVRD_RATE1 0x22c +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_CODE_OVRD_RATE2 0x230 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_CODE_OVRD_RATE3 0x234 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_CODE_OVRD_RATE4 0x238 +#define QSERDES_V8_LALB_CDR_VCO_CAL_CTRL 0x23c +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT1_RATE0 0x240 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT2_RATE0 0x244 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT1_RATE1 0x248 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT2_RATE1 0x24c +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT1_RATE2 0x250 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT2_RATE2 0x254 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT1_RATE3 0x258 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT2_RATE3 0x25c +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT1_RATE4 0x260 +#define QSERDES_V8_LALB_CDR_VCO_CTUNE_MEAS_CNT2_RATE4 0x264 +#define QSERDES_V8_LALB_CDR_VCTRL_RATE_0_1 0x268 +#define QSERDES_V8_LALB_CDR_VCTRL_RATE_2_3 0x26c +#define QSERDES_V8_LALB_CDR_VCTRL_RATE_4 0x270 +#define QSERDES_V8_LALB_KVCO_INIT_RATE_0_1 0x274 +#define QSERDES_V8_LALB_KVCO_INIT_RATE_2_3 0x278 +#define QSERDES_V8_LALB_KVCO_INIT_RATE_4 0x27c +#define QSERDES_V8_LALB_KVCO_CODE_OVRD_RATE0 0x280 +#define QSERDES_V8_LALB_KVCO_CODE_OVRD_RATE1 0x284 +#define QSERDES_V8_LALB_KVCO_CODE_OVRD_RATE2 0x288 +#define QSERDES_V8_LALB_KVCO_CODE_OVRD_RATE3 0x28c +#define QSERDES_V8_LALB_KVCO_CODE_OVRD_RATE4 0x290 +#define QSERDES_V8_LALB_KVCO_CAL_VCTRL_HIGH_RATE_0_1 0x294 +#define QSERDES_V8_LALB_KVCO_CAL_VCTRL_HIGH_RATE_2_3 0x298 +#define QSERDES_V8_LALB_KVCO_CAL_VCTRL_HIGH_RATE_4 0x29c +#define QSERDES_V8_LALB_KVCO_CAL_VCTRL_LOW_RATE_0_1 0x2a0 +#define QSERDES_V8_LALB_KVCO_CAL_VCTRL_LOW_RATE_2_3 0x2a4 +#define QSERDES_V8_LALB_KVCO_CAL_VCTRL_LOW_RATE_4 0x2a8 +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF1_RATE0 0x2ac +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF2_RATE0 0x2b0 +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF1_RATE1 0x2b4 +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF2_RATE1 0x2b8 +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF1_RATE2 0x2bc +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF2_RATE2 0x2c0 +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF1_RATE3 0x2c4 +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF2_RATE3 0x2c8 +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF1_RATE4 0x2cc +#define QSERDES_V8_LALB_KVCO_IDEAL_FREQ_DIFF2_RATE4 0x2d0 +#define QSERDES_V8_LALB_KP_CDR_UP_DN 0x2d4 +#define QSERDES_V8_LALB_KP_CODE_OVRD_RATE_0_1 0x2d8 +#define QSERDES_V8_LALB_KP_CODE_OVRD_RATE_2_3 0x2dc +#define QSERDES_V8_LALB_KP_CODE_OVRD_RATE4 0x2e0 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND1_RATE0 0x2e4 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND2_RATE0 0x2e8 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND1_RATE1 0x2ec +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND2_RATE1 0x2f0 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND1_RATE2 0x2f4 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND2_RATE2 0x2f8 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND1_RATE3 0x2fc +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND2_RATE3 0x300 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND1_RATE4 0x304 +#define QSERDES_V8_LALB_KP_CAL_UPPER_FREQ_DIFF_BND2_RATE4 0x308 +#define QSERDES_V8_LALB_KP_CAL_LOWER_FREQ_DIFF_BND_RATE0 0x30c +#define QSERDES_V8_LALB_KP_CAL_LOWER_FREQ_DIFF_BND_RATE1 0x310 +#define QSERDES_V8_LALB_KP_CAL_LOWER_FREQ_DIFF_BND_RATE2 0x314 +#define QSERDES_V8_LALB_KP_CAL_LOWER_FREQ_DIFF_BND_RATE3 0x318 +#define QSERDES_V8_LALB_KP_CAL_LOWER_FREQ_DIFF_BND_RATE4 0x31c +#define QSERDES_V8_LALB_CDR_KVCO_KP_CAL_FREQ_MEAS_CTRL 0x320 +#define QSERDES_V8_LALB_PLLLOCK_CMP_DEBUG_CTRL 0x324 +#define QSERDES_V8_LALB_PLLLOCK_CMP_DEBUG_CNT1 0x328 +#define QSERDES_V8_LALB_PLLLOCK_CMP_DEBUG_CNT2 0x32c +#define QSERDES_V8_LALB_PLLLOCK_CMP_DEBUG_CNT3 0x330 +#define QSERDES_V8_LALB_RX_SUMMER_CAL_SPD_MODE_RATE_0123 0x334 +#define QSERDES_V8_LALB_RX_SUMMER_CAL_SPD_MODE_RATE_4 0x338 +#define QSERDES_V8_LALB_RX_IVCM_CAL_CODE_OVERRIDE_RATE0 0x33c +#define QSERDES_V8_LALB_RX_IVCM_CAL_CODE_OVERRIDE_RATE1 0x340 +#define QSERDES_V8_LALB_RX_IVCM_CAL_CODE_OVERRIDE_RATE2 0x344 +#define QSERDES_V8_LALB_RX_IVCM_CAL_CODE_OVERRIDE_RATE3 0x348 +#define QSERDES_V8_LALB_RX_IVCM_CAL_CODE_OVERRIDE_RATE4 0x34c +#define QSERDES_V8_LALB_RX_IVCM_CAL_CTRL1 0x350 +#define QSERDES_V8_LALB_RX_IVCM_CAL_CTRL2 0x354 +#define QSERDES_V8_LALB_RX_IVCM_CAL_CTRL3 0x358 +#define QSERDES_V8_LALB_RX_IVCM_CAL_CTRL4 0x35c +#define QSERDES_V8_LALB_RX_IVCM_POSTCAL_OFFSET_RATE0 0x360 +#define QSERDES_V8_LALB_RX_IVCM_POSTCAL_OFFSET_RATE1 0x364 +#define QSERDES_V8_LALB_RX_IVCM_POSTCAL_OFFSET_RATE2 0x368 +#define QSERDES_V8_LALB_RX_IVCM_POSTCAL_OFFSET_RATE3 0x36c +#define QSERDES_V8_LALB_RX_IVCM_POSTCAL_OFFSET_RATE4 0x370 +#define QSERDES_V8_LALB_RX_IDAC_I0_DC_OFFSETS 0x374 +#define QSERDES_V8_LALB_RX_IDAC_I0BAR_DC_OFFSETS 0x378 +#define QSERDES_V8_LALB_RX_IDAC_I1_DC_OFFSETS 0x37c +#define QSERDES_V8_LALB_RX_IDAC_I1BAR_DC_OFFSETS 0x380 +#define QSERDES_V8_LALB_RX_IDAC_Q_DC_OFFSETS 0x384 +#define QSERDES_V8_LALB_RX_IDAC_QBAR_DC_OFFSETS 0x388 +#define QSERDES_V8_LALB_RX_IDAC_A_DC_OFFSETS 0x38c +#define QSERDES_V8_LALB_RX_IDAC_ABAR_DC_OFFSETS 0x390 +#define QSERDES_V8_LALB_RX_IDAC_EN 0x394 +#define QSERDES_V8_LALB_DATA_SLICER_INIT_TIMER_CTRL 0x398 +#define QSERDES_V8_LALB_RX_IDAC_ENABLES 0x39c +#define QSERDES_V8_LALB_RX_IDAC_SIGN 0x3a0 +#define QSERDES_V8_LALB_RX_IDAC_TSETTLE 0x3a4 +#define QSERDES_V8_LALB_SIGDET_ENABLES 0x3a8 +#define QSERDES_V8_LALB_SIGDET_CNTRL 0x3ac +#define QSERDES_V8_LALB_SIGDET_LVL 0x3b0 +#define QSERDES_V8_LALB_SIGDET_DEGLITCH_CNTRL 0x3b4 +#define QSERDES_V8_LALB_SIGDET_CAL_CTRL1 0x3b8 +#define QSERDES_V8_LALB_SIGDET_CAL_CTRL2_AND_CDR_LOCK_EDGE 0x3bc +#define QSERDES_V8_LALB_SIGDET_CAL_TRIM 0x3c0 +#define QSERDES_V8_LALB_IA_OFFSET_CENTER_CAL_CTRL 0x3c4 +#define QSERDES_V8_LALB_FREQ_LOCK_DET_DLY_RATE0 0x3c8 +#define QSERDES_V8_LALB_FREQ_LOCK_DET_DLY_RATE1 0x3cc +#define QSERDES_V8_LALB_FREQ_LOCK_DET_DLY_RATE2 0x3d0 +#define QSERDES_V8_LALB_FREQ_LOCK_DET_DLY_RATE3 0x3d4 +#define QSERDES_V8_LALB_FREQ_LOCK_DET_DLY_RATE4 0x3d8 +#define QSERDES_V8_LALB_CDR_PHASE_LOCK_CNT_RATE0 0x3dc +#define QSERDES_V8_LALB_CDR_PHASE_LOCK_CNT_RATE1 0x3e0 +#define QSERDES_V8_LALB_CDR_PHASE_LOCK_CNT_RATE2 0x3e4 +#define QSERDES_V8_LALB_CDR_PHASE_LOCK_CNT_RATE3 0x3e8 +#define QSERDES_V8_LALB_CDR_PHASE_LOCK_CNT_RATE4 0x3ec +#define QSERDES_V8_LALB_CDR_LOCK_CTRL 0x3f0 +#define QSERDES_V8_LALB_CDR_CP_CUR_FLL_RATE0 0x3f4 +#define QSERDES_V8_LALB_CDR_CP_CUR_FLL_RATE1 0x3f8 +#define QSERDES_V8_LALB_CDR_CP_CUR_FLL_RATE2 0x3fc +#define QSERDES_V8_LALB_CDR_CP_CUR_FLL_RATE3 0x400 +#define QSERDES_V8_LALB_CDR_CP_CUR_FLL_RATE4 0x404 +#define QSERDES_V8_LALB_CDR_CP_CUR_PLL_RATE0 0x408 +#define QSERDES_V8_LALB_CDR_CP_CUR_PLL_RATE1 0x40c +#define QSERDES_V8_LALB_CDR_CP_CUR_PLL_RATE2 0x410 +#define QSERDES_V8_LALB_CDR_CP_CUR_PLL_RATE3 0x414 +#define QSERDES_V8_LALB_CDR_CP_CUR_PLL_RATE4 0x418 +#define QSERDES_V8_LALB_CDR_FLL_DIV_RATIO_RATE_0123 0x41c +#define QSERDES_V8_LALB_CDR_FLL_DIV_RATIO_RATE4 0x420 +#define QSERDES_V8_LALB_CDR_LOOP_CCODE_RATE_01 0x424 +#define QSERDES_V8_LALB_CDR_LOOP_CCODE_RATE_23 0x428 +#define QSERDES_V8_LALB_CDR_LOOP_CCODE_RATE4 0x42c +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_FAST_RATE_0_1 0x430 +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_FAST_RATE_2_3 0x434 +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_FAST_RATE4 0x438 +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_FLL_RATE_0_1 0x43c +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_FLL_RATE_2_3 0x440 +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_FLL_RATE4 0x444 +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_PLL_RATE_0_1 0x448 +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_PLL_RATE_2_3 0x44c +#define QSERDES_V8_LALB_CDR_LOOP_RCODE_PLL_RATE4 0x450 +#define QSERDES_V8_LALB_CDR_VCO_CAP_CODE_RATE_0123 0x454 +#define QSERDES_V8_LALB_CDR_VCO_CAP_CODE_RATE4 0x458 +#define QSERDES_V8_LALB_CDR_VCO_TYPE_CONFIG 0x45c +#define QSERDES_V8_LALB_CDR_VCO_EN_LOWFREQ 0x460 +#define QSERDES_V8_LALB_CDR_FAST_SLOW_VCO_OVRD 0x464 +#define QSERDES_V8_LALB_CDR_LOOP_FUNC_CTRL 0x468 +#define QSERDES_V8_LALB_CDR_FAST_LOCK_EN_CTRL 0x46c +#define QSERDES_V8_LALB_RX_RCVR_EN 0x470 +#define QSERDES_V8_LALB_LANE_RATE_CTRL 0x474 +#define QSERDES_V8_LALB_RX_TERM_RCVR_CTRL 0x478 +#define QSERDES_V8_LALB_REC_DETECT_CTRL 0x47c +#define QSERDES_V8_LALB_RCV_DETECT_LVL 0x480 +#define QSERDES_V8_LALB_GM_CAL_EN 0x484 +#define QSERDES_V8_LALB_GM_CAL_RES_RATE0_1 0x488 +#define QSERDES_V8_LALB_GM_CAL_RES_RATE2_3 0x48c +#define QSERDES_V8_LALB_GM_CAL_RES_RATE4 0x490 +#define QSERDES_V8_LALB_RX_TERM_BW_RATE_0123 0x494 +#define QSERDES_V8_LALB_RX_TERM_BW_RATE4 0x498 +#define QSERDES_V8_LALB_AUX_CLK_CTRL 0x49c +#define QSERDES_V8_LALB_AUX_OFFSET_CONTROL 0x4a0 +#define QSERDES_V8_LALB_AUXDATA_TB 0x4a4 +#define QSERDES_V8_LALB_EOM_CTRL1 0x4a8 +#define QSERDES_V8_LALB_EOM_CTRL2 0x4ac +#define QSERDES_V8_LALB_EOM_CTRL3 0x4b0 +#define QSERDES_V8_LALB_EOM_CTRL4 0x4b4 +#define QSERDES_V8_LALB_DFE_EN_TIMER 0x4b8 +#define QSERDES_V8_LALB_RX_EQ_OFFSET_LSB 0x4bc +#define QSERDES_V8_LALB_RX_EQ_OFFSET_MSB 0x4c0 +#define QSERDES_V8_LALB_RX_EQ_OFFSET_ADAPTOR_CNTRL1 0x4c4 +#define QSERDES_V8_LALB_RX_OFFSET_ADAPTOR_CNTRL2 0x4c8 +#define QSERDES_V8_LALB_RX_OFFSET_ADAPTOR_CNTRL3 0x4cc +#define QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL1 0x4d0 +#define QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL2 0x4d4 +#define QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL3 0x4d8 +#define QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL4 0x4dc +#define QSERDES_V8_LALB_RX_EQU_ADAPTOR_CNTRL5 0x4e0 +#define QSERDES_V8_LALB_RX_EQU_KEQ_UP_LSB 0x4e4 +#define QSERDES_V8_LALB_RX_EQU_KEQ_UP_MSB 0x4e8 +#define QSERDES_V8_LALB_RX_EQU_KEQ_DN_LSB 0x4ec +#define QSERDES_V8_LALB_RX_EQU_KEQ_DN_MSB 0x4f0 +#define QSERDES_V8_LALB_CTLE_ADP_RESET_INIT_CODE_RATE_0_1_2 0x4f4 +#define QSERDES_V8_LALB_CTLE_ADP_RESET_INIT_CODE_RATE_3_4 0x4f8 +#define QSERDES_V8_LALB_CTLE_POST_CAL_OFFSET_RATE_0_1_2 0x4fc +#define QSERDES_V8_LALB_CTLE_POST_CAL_OFFSET_RATE_3_4 0x500 +#define QSERDES_V8_LALB_RX_VGA_GAIN2_BLK1 0x504 +#define QSERDES_V8_LALB_RX_VGA_GAIN2_BLK2 0x508 +#define QSERDES_V8_LALB_VGA_CAL_CNTRL1 0x50c +#define QSERDES_V8_LALB_VGA_CAL_CNTRL2 0x510 +#define QSERDES_V8_LALB_VGA_CAL_MAN_VAL_RATE0_1 0x514 +#define QSERDES_V8_LALB_VGA_CAL_MAN_VAL_RATE2_3 0x518 +#define QSERDES_V8_LALB_VGA_CAL_MAN_VAL_RATE4 0x51c +#define QSERDES_V8_LALB_KVGA_CTRL1 0x520 +#define QSERDES_V8_LALB_KVGA_CTRL2 0x524 +#define QSERDES_V8_LALB_VTHRESH_CAL_CNTRL1 0x528 +#define QSERDES_V8_LALB_VTHRESH_CAL_CNTRL2 0x52c +#define QSERDES_V8_LALB_VTHRESH_CAL_MAN_VAL_RATE0 0x530 +#define QSERDES_V8_LALB_VTHRESH_CAL_MAN_VAL_RATE1 0x534 +#define QSERDES_V8_LALB_VTHRESH_CAL_MAN_VAL_RATE2 0x538 +#define QSERDES_V8_LALB_VTHRESH_CAL_MAN_VAL_RATE3 0x53c +#define QSERDES_V8_LALB_VTHRESH_CAL_MAN_VAL_RATE4 0x540 +#define QSERDES_V8_LALB_VTHRESH_CAL_MAN_CAL_PAM3 0x544 +#define QSERDES_V8_LALB_VTH_POST_CAL_OFFSET_RATE_0_1 0x548 +#define QSERDES_V8_LALB_VTH_POST_CAL_OFFSET_RATE_2_3 0x54c +#define QSERDES_V8_LALB_VTH_POST_CAL_OFFSET_RATE4 0x550 +#define QSERDES_V8_LALB_DFE_TAP1_CTRL 0x554 +#define QSERDES_V8_LALB_DFE_TAP1_MANVAL_KTAP 0x558 +#define QSERDES_V8_LALB_DFE_TAP1_POST_CAL_OFFSET_RATE_0_1_2 0x55c +#define QSERDES_V8_LALB_DFE_TAP1_POST_CAL_OFFSET_RATE_3_4 0x560 +#define QSERDES_V8_LALB_DFE_TAP2_CTRL 0x564 +#define QSERDES_V8_LALB_DFE_TAP2_MANVAL_KTAP 0x568 +#define QSERDES_V8_LALB_DFE_TAP3_CTRL 0x56c +#define QSERDES_V8_LALB_DFE_TAP3_MANVAL_KTAP 0x570 +#define QSERDES_V8_LALB_DFE_TAP4_CTRL 0x574 +#define QSERDES_V8_LALB_DFE_TAP4_MANVAL_KTAP 0x578 +#define QSERDES_V8_LALB_DFE_TAP5_CTRL 0x57c +#define QSERDES_V8_LALB_DFE_TAP5_MANVAL_KTAP 0x580 +#define QSERDES_V8_LALB_DFE_TAP6_CTRL 0x584 +#define QSERDES_V8_LALB_DFE_TAP6_MANVAL_KTAP 0x588 +#define QSERDES_V8_LALB_DFE_TAP7_CTRL 0x58c +#define QSERDES_V8_LALB_DFE_TAP7_MANVAL_KTAP 0x590 +#define QSERDES_V8_LALB_DFE_TAP1_DAC_ENABLE 0x594 +#define QSERDES_V8_LALB_DFE_TAP2_DAC_ENABLE 0x598 +#define QSERDES_V8_LALB_DFE_TAP345_DAC_ENABLE 0x59c +#define QSERDES_V8_LALB_DFE_TAP67_DAC_ENABLE 0x5a0 +#define QSERDES_V8_LALB_CDR_IQTUNE_CTRL 0x5a4 +#define QSERDES_V8_LALB_CDR_IQTUNE_GAIN 0x5a8 +#define QSERDES_V8_LALB_CDR_IQTUNE_MAN_INDEX 0x5ac +#define QSERDES_V8_LALB_CDR_IQTUNE_FILTER_CAL_CTRL1 0x5b0 +#define QSERDES_V8_LALB_CDR_IQTUNE_FILTER_CAL_CTRL2 0x5b4 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK0_CAL_CODE_RATE0 0x5b8 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK0_CAL_CODE_RATE1 0x5bc +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK0_CAL_CODE_RATE2 0x5c0 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK0_CAL_CODE_RATE3 0x5c4 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK0_CAL_CODE_RATE4 0x5c8 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK90_CAL_CODE_RATE0 0x5cc +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK90_CAL_CODE_RATE1 0x5d0 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK90_CAL_CODE_RATE2 0x5d4 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK90_CAL_CODE_RATE3 0x5d8 +#define QSERDES_V8_LALB_CDR_IQTUNE_CLK90_CAL_CODE_RATE4 0x5dc +#define QSERDES_V8_LALB_CDR_IQTUNE_ANA_CTRL 0x5e0 +#define QSERDES_V8_LALB_CDR_IQTUNE_VDCC_CTRL 0x5e4 +#define QSERDES_V8_LALB_CDR_IQTUNE_DIV2_CTRL_RATE0123 0x5e8 +#define QSERDES_V8_LALB_CDR_IQTUNE_DIV2_CTRL_RATE4 0x5ec +#define QSERDES_V8_LALB_BLW_CTRL 0x5f0 +#define QSERDES_V8_LALB_BLW_ANA_VER_CTRL 0x5f4 +#define QSERDES_V8_LALB_BLW_GAIN_CAL_CTRL 0x5f8 +#define QSERDES_V8_LALB_BLW_GAIN_FORCE_CODE 0x5fc +#define QSERDES_V8_LALB_BLW_MAN_VAL_RATE3 0x600 +#define QSERDES_V8_LALB_BLW_MAN_VAL_RATE4 0x604 +#define QSERDES_V8_LALB_IVTH_CAL_CTRL1 0x608 +#define QSERDES_V8_LALB_IVTH_CAL_CTRL2 0x60c +#define QSERDES_V8_LALB_IVTH_CAL_CTRL3 0x610 +#define QSERDES_V8_LALB_VTH_I_UP_CNTRL_VAL 0x614 +#define QSERDES_V8_LALB_VTH_I_DN_CNTRL_VAL 0x618 +#define QSERDES_V8_LALB_NRZ_EYE_HEIGHT_SEL_VAL 0x61c +#define QSERDES_V8_LALB_IVTH_CAL_VAL_OVRD_MUX 0x620 +#define QSERDES_V8_LALB_CDR_VCO_CAP_CODE_OVRD_MUXES 0x624 +#define QSERDES_V8_LALB_VCO_CTUNE_LOWER_BND_RATE0 0x628 +#define QSERDES_V8_LALB_VCO_CTUNE_LOWER_BND_RATE1 0x62c +#define QSERDES_V8_LALB_VCO_CTUNE_LOWER_BND_RATE2 0x630 +#define QSERDES_V8_LALB_VCO_CTUNE_LOWER_BND_RATE3 0x634 +#define QSERDES_V8_LALB_VCO_CTUNE_LOWER_BND_RATE4 0x638 +#define QSERDES_V8_LALB_VCO_CTUNE_UPPER_BND_RATE0 0x63c +#define QSERDES_V8_LALB_VCO_CTUNE_UPPER_BND_RATE1 0x640 +#define QSERDES_V8_LALB_VCO_CTUNE_UPPER_BND_RATE2 0x644 +#define QSERDES_V8_LALB_VCO_CTUNE_UPPER_BND_RATE3 0x648 +#define QSERDES_V8_LALB_VCO_CTUNE_UPPER_BND_RATE4 0x64c +#define QSERDES_V8_LALB_CDR_LOCK_KVCO_OFFSET_RATE0 0x650 +#define QSERDES_V8_LALB_CDR_LOCK_KVCO_OFFSET_RATE1 0x654 +#define QSERDES_V8_LALB_CDR_LOCK_KVCO_OFFSET_RATE2 0x658 +#define QSERDES_V8_LALB_CDR_LOCK_KVCO_OFFSET_RATE3 0x65c +#define QSERDES_V8_LALB_CDR_LOCK_KVCO_OFFSET_RATE4 0x660 +#define QSERDES_V8_LALB_CDR_LOCK_KP_OFFSET_RATE0 0x664 +#define QSERDES_V8_LALB_CDR_LOCK_KP_OFFSET_RATE1 0x668 +#define QSERDES_V8_LALB_CDR_LOCK_KP_OFFSET_RATE2 0x66c +#define QSERDES_V8_LALB_CDR_LOCK_KP_OFFSET_RATE3 0x670 +#define QSERDES_V8_LALB_CDR_LOCK_KP_OFFSET_RATE4 0x674 +#define QSERDES_V8_LALB_CDR_FASTLOCK_CP_CUR_PLL_RATE0 0x678 +#define QSERDES_V8_LALB_CDR_FASTLOCK_CP_CUR_PLL_RATE1 0x67c +#define QSERDES_V8_LALB_CDR_FASTLOCK_CP_CUR_PLL_RATE2 0x680 +#define QSERDES_V8_LALB_CDR_FASTLOCK_CP_CUR_PLL_RATE3 0x684 +#define QSERDES_V8_LALB_CDR_FASTLOCK_CP_CUR_PLL_RATE4 0x688 +#define QSERDES_V8_LALB_DEBUG_BUS_SEL 0x68c +#define QSERDES_V8_LALB_BIST_STATUS 0x690 +#define QSERDES_V8_LALB_BIST_ERROR_COUNT1 0x694 +#define QSERDES_V8_LALB_BIST_ERROR_COUNT2 0x698 +#define QSERDES_V8_LALB_AC_JTAG_OUTP 0x69c +#define QSERDES_V8_LALB_AC_JTAG_OUTN 0x6a0 +#define QSERDES_V8_LALB_DATA_SLICER_DEBUG_STATUS 0x6a4 +#define QSERDES_V8_LALB_DATA_SLICER_TIMER1_STATUS 0x6a8 +#define QSERDES_V8_LALB_DATA_SLICER_TIMER2_STATUS 0x6ac +#define QSERDES_V8_LALB_TX0_RESTRIM_CODE_STATUS 0x6b0 +#define QSERDES_V8_LALB_TX0_RESTRIM_ICAL_CODE_STATUS 0x6b4 +#define QSERDES_V8_LALB_TX0_RESTRIM_CAL_STATUS 0x6b8 +#define QSERDES_V8_LALB_TX1_RESTRIM_CODE_STATUS 0x6bc +#define QSERDES_V8_LALB_TX1_RESTRIM_ICAL_CODE_STATUS 0x6c0 +#define QSERDES_V8_LALB_TX1_RESTRIM_CAL_STATUS 0x6c4 +#define QSERDES_V8_LALB_CMUX_DCC_CAL_FSM_STATUS 0x6c8 +#define QSERDES_V8_LALB_CMUX_DCC_READCODE_STATUS 0x6cc +#define QSERDES_V8_LALB_TX_DCC_CAL_ANA_STATUS 0x6d0 +#define QSERDES_V8_LALB_TX0_CTUNE_DCC_FSM_DEBUG_STATUS 0x6d4 +#define QSERDES_V8_LALB_TX0_COARSE_DCC_READCODE_STATUS 0x6d8 +#define QSERDES_V8_LALB_TX0_FTUNE_MSB_DCC_FSM_DEBUG_STATUS 0x6dc +#define QSERDES_V8_LALB_TX0_FTUNE_LSB_DCC_FSM_DEBUG_STATUS 0x6e0 +#define QSERDES_V8_LALB_TX0_FINE_DCC_READCODE_STATUS 0x6e4 +#define QSERDES_V8_LALB_TX1_CTUNE_DCC_FSM_DEBUG_STATUS 0x6e8 +#define QSERDES_V8_LALB_TX1_COARSE_DCC_READCODE_STATUS 0x6ec +#define QSERDES_V8_LALB_TX1_FTUNE_MSB_DCC_FSM_DEBUG_STATUS 0x6f0 +#define QSERDES_V8_LALB_TX1_FTUNE_LSB_DCC_FSM_DEBUG_STATUS 0x6f4 +#define QSERDES_V8_LALB_TX1_FINE_DCC_READCODE_STATUS 0x6f8 +#define QSERDES_V8_LALB_CDR_VCO_CAL_STATUS 0x6fc +#define QSERDES_V8_LALB_CDR_VCTRL_STATUS 0x700 +#define QSERDES_V8_LALB_CDR_VCO_CAP_CODE_STATUS 0x704 +#define QSERDES_V8_LALB_KVCO_CAL_DEBUG1_STATUS 0x708 +#define QSERDES_V8_LALB_KVCO_CAL_DEBUG2_STATUS 0x70c +#define QSERDES_V8_LALB_KP_CAL_DEBUG1_STATUS 0x710 +#define QSERDES_V8_LALB_KP_CAL_DEBUG2_STATUS 0x714 +#define QSERDES_V8_LALB_CDR_VCO_FREQ_DEBUG1_STATUS 0x718 +#define QSERDES_V8_LALB_CDR_VCO_FREQ_DEBUG2_STATUS 0x71c +#define QSERDES_V8_LALB_CDR_VCO_FREQ_DEBUG3_STATUS 0x720 +#define QSERDES_V8_LALB_CDR_VCO_FREQ_DEBUG4_STATUS 0x724 +#define QSERDES_V8_LALB_IVCM_CAL_STATUS 0x728 +#define QSERDES_V8_LALB_IVCM_CAL_DEBUG_STATUS 0x72c +#define QSERDES_V8_LALB_IDAC_STATUS_I0 0x730 +#define QSERDES_V8_LALB_IDAC_STATUS_I0BAR 0x734 +#define QSERDES_V8_LALB_IDAC_STATUS_I1 0x738 +#define QSERDES_V8_LALB_IDAC_STATUS_I1BAR 0x73c +#define QSERDES_V8_LALB_IDAC_STATUS_Q 0x740 +#define QSERDES_V8_LALB_IDAC_STATUS_QBAR 0x744 +#define QSERDES_V8_LALB_IDAC_STATUS_A 0x748 +#define QSERDES_V8_LALB_IDAC_STATUS_ABAR 0x74c +#define QSERDES_V8_LALB_IDAC_STATUS_SM_ON 0x750 +#define QSERDES_V8_LALB_IDAC_STATUS_SIGNERROR 0x754 +#define QSERDES_V8_LALB_RX_SIGDET_STATUS 0x758 +#define QSERDES_V8_LALB_SIGDET_CAL_CODE_STATUS 0x75c +#define QSERDES_V8_LALB_SIGDET_CAL_FSM_DEBUG_STATUS 0x760 +#define QSERDES_V8_LALB_CDR_FREQ_LOCK_CNT_STATUS 0x764 +#define QSERDES_V8_LALB_CDR_PHASE_LOCK_CNT_STATUS 0x768 +#define QSERDES_V8_LALB_CDR_LOCK_DEBUG_STATUS 0x76c +#define QSERDES_V8_LALB_IDATA_HIGH_STATUS1 0x770 +#define QSERDES_V8_LALB_IDATA_HIGH_STATUS2 0x774 +#define QSERDES_V8_LALB_IDATA_HIGH_STATUS3 0x778 +#define QSERDES_V8_LALB_IDATA_HIGH_STATUS4 0x77c +#define QSERDES_V8_LALB_IDATA_LOW_STATUS1 0x780 +#define QSERDES_V8_LALB_IDATA_LOW_STATUS2 0x784 +#define QSERDES_V8_LALB_IDATA_LOW_STATUS3 0x788 +#define QSERDES_V8_LALB_IDATA_LOW_STATUS4 0x78c +#define QSERDES_V8_LALB_QDATA_STATUS1 0x790 +#define QSERDES_V8_LALB_QDATA_STATUS2 0x794 +#define QSERDES_V8_LALB_QDATA_STATUS3 0x798 +#define QSERDES_V8_LALB_QDATA_STATUS4 0x79c +#define QSERDES_V8_LALB_IA_ERROR_COUNTER_LOW 0x7a0 +#define QSERDES_V8_LALB_IA_ERROR_COUNTER_HIGH 0x7a4 +#define QSERDES_V8_LALB_EOM_ERR_CNT_LSB_STATUS 0x7a8 +#define QSERDES_V8_LALB_EOM_ERR_CNT_MSB_STATUS 0x7ac +#define QSERDES_V8_LALB_EOM_OP_STATUS 0x7b0 +#define QSERDES_V8_LALB_AUX_MIXER_INDEX_STATUS 0x7b4 +#define QSERDES_V8_LALB_AUX_OFFSET_STATUS 0x7b8 +#define QSERDES_V8_LALB_AUXDATA_TB_STATUS 0x7bc +#define QSERDES_V8_LALB_AUX_MIXER_CTRL_0_STATUS 0x7c0 +#define QSERDES_V8_LALB_AUX_MIXER_CTRL_90_STATUS 0x7c4 +#define QSERDES_V8_LALB_AUX_MIXER_CTRL_180_STATUS 0x7c8 +#define QSERDES_V8_LALB_IQ_MIXER_INDEX_STATUS 0x7cc +#define QSERDES_V8_LALB_IQTUNE_FLTR_INDEX_STATUS 0x7d0 +#define QSERDES_V8_LALB_IQ_MIXER_CTRL_0_STATUS 0x7d4 +#define QSERDES_V8_LALB_IQ_MIXER_CTRL_90_STATUS 0x7d8 +#define QSERDES_V8_LALB_IQ_MIXER_CTRL_180_STATUS 0x7dc +#define QSERDES_V8_LALB_READ_EQCODE 0x7e0 +#define QSERDES_V8_LALB_READ_OFFSETCODE 0x7e4 +#define QSERDES_V8_LALB_VGA_READ_CODE 0x7e8 +#define QSERDES_V8_LALB_VTHRESH_READ_CODE 0x7ec +#define QSERDES_V8_LALB_DFE_TAP1_READ_CODE 0x7f0 +#define QSERDES_V8_LALB_DFE_TAP2_READ_CODE 0x7f4 +#define QSERDES_V8_LALB_DFE_TAP3_READ_CODE 0x7f8 +#define QSERDES_V8_LALB_DFE_TAP4_READ_CODE 0x7fc +#define QSERDES_V8_LALB_DFE_TAP5_READ_CODE 0x800 +#define QSERDES_V8_LALB_DFE_TAP6_READ_CODE 0x804 +#define QSERDES_V8_LALB_DFE_TAP7_READ_CODE 0x808 +#define QSERDES_V8_LALB_CDR_IQTUNE_FILTER_BIN_CODE 0x80c +#define QSERDES_V8_LALB_CDR_IQTUNE_FILTER_CLK0_CODE 0x810 +#define QSERDES_V8_LALB_CDR_IQTUNE_FILTER_CLK90_CODE 0x814 +#define QSERDES_V8_LALB_BLW_READ_CODE 0x818 +#define QSERDES_V8_LALB_IA_OFFSET_CAL_DEBUG_STATUS 0x81c +#define QSERDES_V8_LALB_IA_OFFSET_CAL_STATUS 0x820 +#define QSERDES_V8_LALB_IVTH_CAL_STATUS 0x824 +#define QSERDES_V8_LALB_IVTH_NRZ_EYE_HEIGHT_STATUS 0x828 +#define QSERDES_V8_LALB_IVTH_UPPER_EYE_MAX_STATUS 0x82c +#define QSERDES_V8_LALB_IVTH_UPPER_EYE_MIN_STATUS 0x830 +#define QSERDES_V8_LALB_IVTH_LOWER_EYE_MAX_STATUS 0x834 +#define QSERDES_V8_LALB_IVTH_LOWER_EYE_MIN_STATUS 0x838 +#define QSERDES_V8_LALB_IVTH_UP_INIT_CTR_STATUS 0x83c +#define QSERDES_V8_LALB_VTH_I_UP_CNTRL_STATUS 0x840 +#define QSERDES_V8_LALB_VTH_I_DN_CNTRL_STATUS 0x844 +#define QSERDES_V8_LALB_NRZ_EYE_HEIGHT_SEL_STATUS 0x848 +#define QSERDES_V8_LALB_DEBUG_BUS0 0x84c +#define QSERDES_V8_LALB_DEBUG_BUS1 0x850 +#define QSERDES_V8_LALB_DEBUG_BUS2 0x854 +#define QSERDES_V8_LALB_DEBUG_BUS3 0x858 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL1 0x85c +#define QSERDES_V8_LALB_DIG_BKUP_CTRL2 0x860 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL3 0x864 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL4 0x868 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL5 0x86c +#define QSERDES_V8_LALB_DIG_BKUP_CTRL6 0x870 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL7 0x874 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL8 0x878 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL9 0x87c +#define QSERDES_V8_LALB_DIG_BKUP_CTRL10 0x880 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL11 0x884 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL12 0x888 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL13 0x88c +#define QSERDES_V8_LALB_DIG_BKUP_CTRL14 0x890 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL15 0x894 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL16 0x898 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL17 0x89c +#define QSERDES_V8_LALB_DIG_BKUP_CTRL18 0x8a0 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL19 0x8a4 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL20 0x8a8 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL21 0x8ac +#define QSERDES_V8_LALB_DIG_BKUP_CTRL22 0x8b0 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL23 0x8b4 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL24 0x8b8 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL25 0x8bc +#define QSERDES_V8_LALB_DIG_BKUP_CTRL26 0x8c0 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL27 0x8c4 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL28 0x8c8 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL29 0x8cc +#define QSERDES_V8_LALB_DIG_BKUP_CTRL30 0x8d0 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL31 0x8d4 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL32 0x8d8 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL_V2_1 0x8dc +#define QSERDES_V8_LALB_DIG_BKUP_CTRL_V2_2 0x8e0 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL_V2_3 0x8e4 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL_V2_4 0x8e8 +#define QSERDES_V8_LALB_DIG_BKUP_CTRL_V2_5 0x8ec +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS1 0x8f0 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS2 0x8f4 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS3 0x8f8 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS4 0x8fc +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS5 0x900 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS6 0x904 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS7 0x908 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS8 0x90c +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS9 0x910 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS10 0x914 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS11 0x918 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS12 0x91c +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS13 0x920 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS14 0x924 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS15 0x928 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS16 0x92c +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS17 0x930 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS18 0x934 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS19 0x938 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS20 0x93c +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS21 0x940 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS22 0x944 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS23 0x948 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS24 0x94c +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS25 0x950 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS26 0x954 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS27 0x958 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS28 0x95c +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS29 0x960 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS30 0x964 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS31 0x968 +#define QSERDES_V8_LALB_DIG_BKUP_RO_BUS32 0x96c +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS1 0x970 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS2 0x974 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS3 0x978 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS4 0x97c +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS5 0x980 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS6 0x984 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS7 0x988 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS8 0x98c +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS9 0x990 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS10 0x994 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS11 0x998 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS12 0x99c +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS13 0x9a0 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS14 0x9a4 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS15 0x9a8 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS16 0x9ac +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS17 0x9b0 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS18 0x9b4 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS19 0x9b8 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS20 0x9bc +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS21 0x9c0 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS22 0x9c4 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS23 0x9c8 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS24 0x9cc +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS25 0x9d0 +#define QSERDES_V8_LALB_DIG_BKUP_RO_V2_BUS26 0x9d4 +#endif /* QCOM_PHY_QMP_QSERDES_V8_LALBH_ */ diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-usb-legacy.c b/drivers/phy/qualcomm/phy-qcom-qmp-usb-legacy.c index 8bf951b0490c..fc490589c8e4 100644 --- a/drivers/phy/qualcomm/phy-qcom-qmp-usb-legacy.c +++ b/drivers/phy/qualcomm/phy-qcom-qmp-usb-legacy.c @@ -542,6 +542,8 @@ struct qmp_usb { enum phy_mode mode; + bool phy_initialized; + struct phy *phy; struct clk_fixed_rate pipe_clk_fixed; @@ -895,6 +897,7 @@ static int qmp_usb_legacy_power_off(struct phy *phy) static int qmp_usb_legacy_enable(struct phy *phy) { + struct qmp_usb *qmp = phy_get_drvdata(phy); int ret; ret = qmp_usb_legacy_init(phy); @@ -904,14 +907,19 @@ static int qmp_usb_legacy_enable(struct phy *phy) ret = qmp_usb_legacy_power_on(phy); if (ret) qmp_usb_legacy_exit(phy); + else + qmp->phy_initialized = true; return ret; } static int qmp_usb_legacy_disable(struct phy *phy) { + struct qmp_usb *qmp = phy_get_drvdata(phy); int ret; + qmp->phy_initialized = false; + ret = qmp_usb_legacy_power_off(phy); if (ret) return ret; @@ -988,7 +996,7 @@ static int __maybe_unused qmp_usb_legacy_runtime_suspend(struct device *dev) dev_vdbg(dev, "Suspending QMP phy, mode:%d\n", qmp->mode); - if (!qmp->phy->init_count) { + if (!qmp->phy_initialized) { dev_vdbg(dev, "PHY not initialized, bailing out\n"); return 0; } @@ -1009,7 +1017,7 @@ static int __maybe_unused qmp_usb_legacy_runtime_resume(struct device *dev) dev_vdbg(dev, "Resuming QMP phy, mode:%d\n", qmp->mode); - if (!qmp->phy->init_count) { + if (!qmp->phy_initialized) { dev_vdbg(dev, "PHY not initialized, bailing out\n"); return 0; } diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-usb.c b/drivers/phy/qualcomm/phy-qcom-qmp-usb.c index ed646a7e705b..e7dd140ad29e 100644 --- a/drivers/phy/qualcomm/phy-qcom-qmp-usb.c +++ b/drivers/phy/qualcomm/phy-qcom-qmp-usb.c @@ -1300,6 +1300,8 @@ struct qmp_usb { enum phy_mode mode; + bool phy_initialized; + struct phy *phy; struct clk_fixed_rate pipe_clk_fixed; @@ -1848,6 +1850,7 @@ static int qmp_usb_power_off(struct phy *phy) static int qmp_usb_enable(struct phy *phy) { + struct qmp_usb *qmp = phy_get_drvdata(phy); int ret; ret = qmp_usb_init(phy); @@ -1857,14 +1860,19 @@ static int qmp_usb_enable(struct phy *phy) ret = qmp_usb_power_on(phy); if (ret) qmp_usb_exit(phy); + else + qmp->phy_initialized = true; return ret; } static int qmp_usb_disable(struct phy *phy) { + struct qmp_usb *qmp = phy_get_drvdata(phy); int ret; + qmp->phy_initialized = false; + ret = qmp_usb_power_off(phy); if (ret) return ret; @@ -1940,7 +1948,7 @@ static int __maybe_unused qmp_usb_runtime_suspend(struct device *dev) dev_vdbg(dev, "Suspending QMP phy, mode:%d\n", qmp->mode); - if (!qmp->phy->init_count) { + if (!qmp->phy_initialized) { dev_vdbg(dev, "PHY not initialized, bailing out\n"); return 0; } @@ -1960,7 +1968,7 @@ static int __maybe_unused qmp_usb_runtime_resume(struct device *dev) dev_vdbg(dev, "Resuming QMP phy, mode:%d\n", qmp->mode); - if (!qmp->phy->init_count) { + if (!qmp->phy_initialized) { dev_vdbg(dev, "PHY not initialized, bailing out\n"); return 0; } diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-usb43-pcs-v8.h b/drivers/phy/qualcomm/phy-qcom-qmp-usb43-pcs-v8.h new file mode 100644 index 000000000000..4f387c8ed9e5 --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-usb43-pcs-v8.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_USB43_PCS_V8_H_ +#define QCOM_PHY_QMP_USB43_PCS_V8_H_ + +#define QPHY_V8_USB43_PCS_SW_RESET 0x000 +#define QPHY_V8_USB43_PCS_PCS_STATUS1 0x014 +#define QPHY_V8_USB43_PCS_POWER_DOWN_CONTROL 0x040 +#define QPHY_V8_USB43_PCS_START_CONTROL 0x044 +#define QPHY_V8_USB43_PCS_POWER_STATE_CONFIG1 0x090 +#define QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG1 0x0c4 +#define QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG2 0x0c8 +#define QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG3 0x0cc +#define QPHY_V8_USB43_PCS_LOCK_DETECT_CONFIG6 0x0d8 +#define QPHY_V8_USB43_PCS_REFGEN_REQ_CONFIG1 0x0dc +#define QPHY_V8_USB43_PCS_RX_SIGDET_LVL 0x188 +#define QPHY_V8_USB43_PCS_RCVR_DTCT_DLY_P1U2_L 0x190 +#define QPHY_V8_USB43_PCS_RCVR_DTCT_DLY_P1U2_H 0x194 +#define QPHY_V8_USB43_PCS_RATE_SLEW_CNTRL1 0x198 +#define QPHY_V8_USB43_PCS_TSYNC_RSYNC_TIME 0x1ac +#define QPHY_V8_USB43_PCS_RX_CONFIG 0x1b0 +#define QPHY_V8_USB43_PCS_TSYNC_DLY_TIME 0x1b4 +#define QPHY_V8_USB43_PCS_ALIGN_DETECT_CONFIG1 0x1c0 +#define QPHY_V8_USB43_PCS_ALIGN_DETECT_CONFIG2 0x1c4 +#define QPHY_V8_USB43_PCS_PCS_TX_RX_CONFIG 0x1d0 +#define QPHY_V8_USB43_PCS_EQ_CONFIG1 0x1dc +#define QPHY_V8_USB43_PCS_EQ_CONFIG2 0x1e0 +#define QPHY_V8_USB43_PCS_EQ_CONFIG5 0x1ec + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-usb43-qserdes-com-v8.h b/drivers/phy/qualcomm/phy-qcom-qmp-usb43-qserdes-com-v8.h new file mode 100644 index 000000000000..e9c743fce9d1 --- /dev/null +++ b/drivers/phy/qualcomm/phy-qcom-qmp-usb43-qserdes-com-v8.h @@ -0,0 +1,224 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2025 Qualcomm Innovation Center, Inc. All rights reserved. + */ + +#ifndef QCOM_PHY_QMP_USB43_QSERDES_COM_V8_H_ +#define QCOM_PHY_QMP_USB43_QSERDES_COM_V8_H_ + +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE1 0x000 +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE2_MODE1 0x004 +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE3_MODE1 0x008 +#define QSERDES_V8_USB43_COM_CLK_EP_DIV_MODE1 0x00c +#define QSERDES_V8_USB43_COM_CP_CTRL_MODE1 0x010 +#define QSERDES_V8_USB43_COM_PLL_RCTRL_MODE1 0x014 +#define QSERDES_V8_USB43_COM_PLL_CCTRL_MODE1 0x018 +#define QSERDES_V8_USB43_COM_CORECLK_DIV_MODE1 0x01c +#define QSERDES_V8_USB43_COM_LOCK_CMP1_MODE1 0x020 +#define QSERDES_V8_USB43_COM_LOCK_CMP2_MODE1 0x024 +#define QSERDES_V8_USB43_COM_DEC_START_MODE1 0x028 +#define QSERDES_V8_USB43_COM_DEC_START_MSB_MODE1 0x02c +#define QSERDES_V8_USB43_COM_DIV_FRAC_START1_MODE1 0x030 +#define QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE1 0x034 +#define QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE1 0x038 +#define QSERDES_V8_USB43_COM_HSCLK_SEL_1 0x03c +#define QSERDES_V8_USB43_COM_INTEGLOOP_GAIN0_MODE1 0x040 +#define QSERDES_V8_USB43_COM_INTEGLOOP_GAIN1_MODE1 0x044 +#define QSERDES_V8_USB43_COM_VCO_TUNE1_MODE1 0x048 +#define QSERDES_V8_USB43_COM_VCO_TUNE2_MODE1 0x04c +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE1 0x050 +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE1 0x054 +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE0 0x058 +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE0 0x05c +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE0 0x060 +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE2_MODE0 0x064 +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE3_MODE0 0x068 +#define QSERDES_V8_USB43_COM_CLK_EP_DIV_MODE0 0x06c +#define QSERDES_V8_USB43_COM_CP_CTRL_MODE0 0x070 +#define QSERDES_V8_USB43_COM_PLL_RCTRL_MODE0 0x074 +#define QSERDES_V8_USB43_COM_PLL_CCTRL_MODE0 0x078 +#define QSERDES_V8_USB43_COM_CORECLK_DIV_MODE0 0x07c +#define QSERDES_V8_USB43_COM_LOCK_CMP1_MODE0 0x080 +#define QSERDES_V8_USB43_COM_LOCK_CMP2_MODE0 0x084 +#define QSERDES_V8_USB43_COM_DEC_START_MODE0 0x088 +#define QSERDES_V8_USB43_COM_DEC_START_MSB_MODE0 0x08c +#define QSERDES_V8_USB43_COM_DIV_FRAC_START1_MODE0 0x090 +#define QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE0 0x094 +#define QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE0 0x098 +#define QSERDES_V8_USB43_COM_HSCLK_HS_SWITCH_SEL_1 0x09c +#define QSERDES_V8_USB43_COM_INTEGLOOP_GAIN0_MODE0 0x0a0 +#define QSERDES_V8_USB43_COM_INTEGLOOP_GAIN1_MODE0 0x0a4 +#define QSERDES_V8_USB43_COM_VCO_TUNE1_MODE0 0x0a8 +#define QSERDES_V8_USB43_COM_VCO_TUNE2_MODE0 0x0ac +#define QSERDES_V8_USB43_COM_ATB_SEL1 0x0b0 +#define QSERDES_V8_USB43_COM_ATB_SEL2 0x0b4 +#define QSERDES_V8_USB43_COM_FREQ_UPDATE 0x0b8 +#define QSERDES_V8_USB43_COM_BG_TIMER 0x0bc +#define QSERDES_V8_USB43_COM_SSC_EN_CENTER 0x0c0 +#define QSERDES_V8_USB43_COM_SSC_ADJ_PER1 0x0c4 +#define QSERDES_V8_USB43_COM_SSC_ADJ_PER2 0x0c8 +#define QSERDES_V8_USB43_COM_SSC_PER1 0x0cc +#define QSERDES_V8_USB43_COM_SSC_PER2 0x0d0 +#define QSERDES_V8_USB43_COM_POST_DIV 0x0d4 +#define QSERDES_V8_USB43_COM_POST_DIV_MUX 0x0d8 +#define QSERDES_V8_USB43_COM_BIAS_EN_CLKBUFLR_EN 0x0dc +#define QSERDES_V8_USB43_COM_CLK_ENABLE1 0x0e0 +#define QSERDES_V8_USB43_COM_SYS_CLK_CTRL 0x0e4 +#define QSERDES_V8_USB43_COM_SYSCLK_BUF_ENABLE 0x0e8 +#define QSERDES_V8_USB43_COM_PLL_EN 0x0ec +#define QSERDES_V8_USB43_COM_DEBUG_BUS_OVRD 0x0f0 +#define QSERDES_V8_USB43_COM_PLL_IVCO 0x0f4 +#define QSERDES_V8_USB43_COM_PLL_IVCO_MODE1 0x0f8 +#define QSERDES_V8_USB43_COM_CMN_IETRIM 0x0fc +#define QSERDES_V8_USB43_COM_CMN_IPTRIM 0x100 +#define QSERDES_V8_USB43_COM_EP_CLOCK_DETECT_CTRL 0x104 +#define QSERDES_V8_USB43_COM_PLL_CNTRL 0x108 +#define QSERDES_V8_USB43_COM_BIAS_EN_CTRL_BY_PSM 0x10c +#define QSERDES_V8_USB43_COM_SYSCLK_EN_SEL 0x110 +#define QSERDES_V8_USB43_COM_CML_SYSCLK_SEL 0x114 +#define QSERDES_V8_USB43_COM_RESETSM_CNTRL 0x118 +#define QSERDES_V8_USB43_COM_RESETSM_CNTRL2 0x11c +#define QSERDES_V8_USB43_COM_LOCK_CMP_EN 0x120 +#define QSERDES_V8_USB43_COM_LOCK_CMP_CFG 0x124 +#define QSERDES_V8_USB43_COM_INTEGLOOP_INITVAL 0x128 +#define QSERDES_V8_USB43_COM_INTEGLOOP_EN 0x12c +#define QSERDES_V8_USB43_COM_INTEGLOOP_P_PATH_GAIN0 0x130 +#define QSERDES_V8_USB43_COM_INTEGLOOP_P_PATH_GAIN1 0x134 +#define QSERDES_V8_USB43_COM_VCOCAL_DEADMAN_CTRL 0x138 +#define QSERDES_V8_USB43_COM_VCO_TUNE_CTRL 0x13c +#define QSERDES_V8_USB43_COM_VCO_TUNE_MAP 0x140 +#define QSERDES_V8_USB43_COM_VCO_TUNE_INITVAL1 0x144 +#define QSERDES_V8_USB43_COM_VCO_TUNE_INITVAL2 0x148 +#define QSERDES_V8_USB43_COM_VCO_TUNE_MINVAL1 0x14c +#define QSERDES_V8_USB43_COM_VCO_TUNE_MINVAL2 0x150 +#define QSERDES_V8_USB43_COM_VCO_TUNE_MAXVAL1 0x154 +#define QSERDES_V8_USB43_COM_VCO_TUNE_MAXVAL2 0x158 +#define QSERDES_V8_USB43_COM_VCO_TUNE_TIMER1 0x15c +#define QSERDES_V8_USB43_COM_VCO_TUNE_TIMER2 0x160 +#define QSERDES_V8_USB43_COM_CLK_SELECT 0x164 +#define QSERDES_V8_USB43_COM_PLL_ANALOG 0x168 +#define QSERDES_V8_USB43_COM_SW_RESET 0x16c +#define QSERDES_V8_USB43_COM_CORE_CLK_EN 0x170 +#define QSERDES_V8_USB43_COM_CMN_CONFIG_1 0x174 +#define QSERDES_V8_USB43_COM_CMN_CONFIG_3 0x178 +#define QSERDES_V8_USB43_COM_CMN_RATE_OVERRIDE 0x17c +#define QSERDES_V8_USB43_COM_SVS_MODE_CLK_SEL 0x180 +#define QSERDES_V8_USB43_COM_DEBUG_BUS_SEL 0x184 +#define QSERDES_V8_USB43_COM_CMN_MISC1 0x188 +#define QSERDES_V8_USB43_COM_CMN_MODE 0x18c +#define QSERDES_V8_USB43_COM_CMN_MODE_CONTD 0x190 +#define QSERDES_V8_USB43_COM_CMN_MODE_CONTD1 0x194 +#define QSERDES_V8_USB43_COM_CMN_MODE_CONTD2 0x198 +#define QSERDES_V8_USB43_COM_VCO_DC_LEVEL_CTRL 0x19c +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_HSCLK_SEL_1 0x1a0 +#define QSERDES_V8_USB43_COM_ADDITIONAL_CTRL_1 0x1a4 +#define QSERDES_V8_USB43_COM_AUTO_GAIN_ADJ_CTRL_1 0x1a8 +#define QSERDES_V8_USB43_COM_AUTO_GAIN_ADJ_CTRL_2 0x1ac +#define QSERDES_V8_USB43_COM_AUTO_GAIN_ADJ_CTRL_3 0x1b0 +#define QSERDES_V8_USB43_COM_AUTO_GAIN_ADJ_CTRL_4 0x1b4 +#define QSERDES_V8_USB43_COM_ADDITIONAL_MISC 0x1b8 +#define QSERDES_V8_USB43_COM_ADDITIONAL_MISC_2 0x1bc +#define QSERDES_V8_USB43_COM_ADDITIONAL_MISC_3 0x1c0 +#define QSERDES_V8_USB43_COM_ADDITIONAL_MISC_4 0x1c4 +#define QSERDES_V8_USB43_COM_ADDITIONAL_MISC_5 0x1c8 +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE1_MODE2 0x1cc +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE2_MODE2 0x1d0 +#define QSERDES_V8_USB43_COM_SSC_STEP_SIZE3_MODE2 0x1d4 +#define QSERDES_V8_USB43_COM_CLK_EP_DIV_MODE2 0x1d8 +#define QSERDES_V8_USB43_COM_CP_CTRL_MODE2 0x1dc +#define QSERDES_V8_USB43_COM_PLL_RCTRL_MODE2 0x1e0 +#define QSERDES_V8_USB43_COM_PLL_CCTRL_MODE2 0x1e4 +#define QSERDES_V8_USB43_COM_CORECLK_DIV_MODE2 0x1e8 +#define QSERDES_V8_USB43_COM_LOCK_CMP1_MODE2 0x1ec +#define QSERDES_V8_USB43_COM_LOCK_CMP2_MODE2 0x1f0 +#define QSERDES_V8_USB43_COM_DEC_START_MODE2 0x1f4 +#define QSERDES_V8_USB43_COM_DEC_START_MSB_MODE2 0x1f8 +#define QSERDES_V8_USB43_COM_DIV_FRAC_START1_MODE2 0x1fc +#define QSERDES_V8_USB43_COM_DIV_FRAC_START2_MODE2 0x200 +#define QSERDES_V8_USB43_COM_DIV_FRAC_START3_MODE2 0x204 +#define QSERDES_V8_USB43_COM_INTEGLOOP_GAIN0_MODE2 0x208 +#define QSERDES_V8_USB43_COM_INTEGLOOP_GAIN1_MODE2 0x20c +#define QSERDES_V8_USB43_COM_VCO_TUNE1_MODE2 0x210 +#define QSERDES_V8_USB43_COM_VCO_TUNE2_MODE2 0x214 +#define QSERDES_V8_USB43_COM_PLL_IVCO_MODE2 0x218 +#define QSERDES_V8_USB43_COM_HSCLK_SEL_2 0x21c +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE1_MODE2 0x220 +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_CMP_CODE2_MODE2 0x224 +#define QSERDES_V8_USB43_COM_HSCLK_HS_SWITCH_SEL_2 0x228 +#define QSERDES_V8_USB43_COM_CMN_CONFIG_2 0x22c +#define QSERDES_V8_USB43_COM_BIN_VCOCAL_HSCLK_SEL_2 0x230 +#define QSERDES_V8_USB43_COM_IVCOCAL_CONFIG_0 0x234 +#define QSERDES_V8_USB43_COM_IVCOCAL_CONFIG_1 0x238 +#define QSERDES_V8_USB43_COM_IVCOCAL_CONFIG_2 0x23c +#define QSERDES_V8_USB43_COM_IVCOCAL_CONFIG_3 0x240 +#define QSERDES_V8_USB43_COM_IVCOCAL_CONFIG_4 0x244 +#define QSERDES_V8_USB43_COM_IVCOCAL_CONFIG_5 0x248 +#define QSERDES_V8_USB43_COM_LOCK_CMP1_EARLY_MODE0 0x24c +#define QSERDES_V8_USB43_COM_LOCK_CMP2_EARLY_MODE0 0x250 +#define QSERDES_V8_USB43_COM_LOCK_CMP1_EARLY_MODE1 0x254 +#define QSERDES_V8_USB43_COM_LOCK_CMP2_EARLY_MODE1 0x258 +#define QSERDES_V8_USB43_COM_LOCK_CMP1_EARLY_MODE2 0x25c +#define QSERDES_V8_USB43_COM_LOCK_CMP2_EARLY_MODE2 0x260 +#define QSERDES_V8_USB43_COM_EARLY_LOCK_CONFIG_0 0x264 +#define QSERDES_V8_USB43_COM_EARLY_LOCK_CONFIG_1 0x268 +#define QSERDES_V8_USB43_COM_ADAPTIVE_ANALOG_CONFIG 0x26c +#define QSERDES_V8_USB43_COM_CP_CTRL_ADAPTIVE_MODE0 0x270 +#define QSERDES_V8_USB43_COM_PLL_RCCTRL_ADAPTIVE_MODE0 0x274 +#define QSERDES_V8_USB43_COM_PLL_CCTRL_ADAPTIVE_MODE0 0x278 +#define QSERDES_V8_USB43_COM_CP_CTRL_ADAPTIVE_MODE1 0x27c +#define QSERDES_V8_USB43_COM_PLL_RCCTRL_ADAPTIVE_MODE1 0x280 +#define QSERDES_V8_USB43_COM_PLL_CCTRL_ADAPTIVE_MODE1 0x284 +#define QSERDES_V8_USB43_COM_CP_CTRL_ADAPTIVE_MODE2 0x288 +#define QSERDES_V8_USB43_COM_PLL_RCCTRL_ADAPTIVE_MODE2 0x28c +#define QSERDES_V8_USB43_COM_PLL_CCTRL_ADAPTIVE_MODE2 0x290 +#define QSERDES_V8_USB43_COM_CMN_MODE_CONTD3 0x294 +#define QSERDES_V8_USB43_COM_CMN_MODE_CONTD4 0x298 +#define QSERDES_V8_USB43_COM_CMN_MODE_CONTD5 0x29c +#define QSERDES_V8_USB43_COM_CMN_MODE_CONTD6 0x2a0 +#define QSERDES_V8_USB43_COM_ADDITIONAL_MISC_6 0x2a4 +#define QSERDES_V8_USB43_COM_ADDITIONAL_MISC_7 0x2a8 +#define QSERDES_V8_USB43_COM_VCO_WAIT_CYCLES 0x2ac +#define QSERDES_V8_USB43_COM_BIAS_WAIT_CYCLES 0x2b0 +#define QSERDES_V8_USB43_COM_AUX_CLK_PSM_ENABLE 0x2b4 +#define QSERDES_V8_USB43_COM_PLL_SPARE_FOR_ECO 0x2b8 +#define QSERDES_V8_USB43_COM_PLL_SPARE_FOR_ECO_1 0x2bc +#define QSERDES_V8_USB43_COM_PLL_SPARE_FOR_ECO_2 0x2c0 +#define QSERDES_V8_USB43_COM_LDO_CAL_1 0x2c4 +#define QSERDES_V8_USB43_COM_LDO_CAL_2 0x2c8 +#define QSERDES_V8_USB43_COM_LDO_CAL_3 0x2cc +#define QSERDES_V8_USB43_COM_LDO_CAL_4 0x2d0 +#define QSERDES_V8_USB43_COM_LDO_CAL_5 0x2d4 +#define QSERDES_V8_USB43_COM_DCC_CAL_1 0x2d8 +#define QSERDES_V8_USB43_COM_DCC_CAL_2 0x2dc +#define QSERDES_V8_USB43_COM_DCC_CAL_3 0x2e0 +#define QSERDES_V8_USB43_COM_DCC_CAL_4 0x2e4 +#define QSERDES_V8_USB43_COM_DCC_CAL_5 0x2e8 +#define QSERDES_V8_USB43_COM_DCC_CAL_6 0x2ec +#define QSERDES_V8_USB43_COM_PSM_CAL_EN 0x2f0 +#define QSERDES_V8_USB43_COM_CLK_FWD_CONFIG_1 0x2f4 +#define QSERDES_V8_USB43_COM_CLK_FWD_CONFIG_2 0x2f8 +#define QSERDES_V8_USB43_COM_IP_CTRL_AND_DP_SEL 0x2fc +#define QSERDES_V8_USB43_COM_DCC_CAL_7 0x300 +#define QSERDES_V8_USB43_COM_DCC_CAL_8 0x304 +#define QSERDES_V8_USB43_COM_DCC_CAL_9 0x308 +#define QSERDES_V8_USB43_COM_MODE_OPERATION_STATUS 0x30c +#define QSERDES_V8_USB43_COM_SYSCLK_DET_COMP_STATUS 0x310 +#define QSERDES_V8_USB43_COM_CMN_STATUS 0x314 +#define QSERDES_V8_USB43_COM_RESET_SM_STATUS 0x318 +#define QSERDES_V8_USB43_COM_RESTRIM_CODE_STATUS 0x31c +#define QSERDES_V8_USB43_COM_PLLCAL_CODE1_STATUS 0x320 +#define QSERDES_V8_USB43_COM_PLLCAL_CODE2_STATUS 0x324 +#define QSERDES_V8_USB43_COM_INTEGLOOP_BINCODE_STATUS 0x328 +#define QSERDES_V8_USB43_COM_DEBUG_BUS0 0x32c +#define QSERDES_V8_USB43_COM_DEBUG_BUS1 0x330 +#define QSERDES_V8_USB43_COM_DEBUG_BUS2 0x334 +#define QSERDES_V8_USB43_COM_DEBUG_BUS3 0x338 +#define QSERDES_V8_USB43_COM_C_READY_STATUS 0x33c +#define QSERDES_V8_USB43_COM_READ_DUMMY_1 0x340 +#define QSERDES_V8_USB43_COM_READ_DUMMY_2 0x344 +#define QSERDES_V8_USB43_COM_READ_DUMMY_3 0x348 +#define QSERDES_V8_USB43_COM_IVCO_CAL_CODE_STATUS 0x34c +#define QSERDES_V8_USB43_COM_PLL_LDO_CAL_STATUS_2 0x350 +#define QSERDES_V8_USB43_COM_PLL_LDO_CAL_STATUS_3 0x354 + +#endif diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.h b/drivers/phy/qualcomm/phy-qcom-qmp.h index f58c82b2dd23..aa34b6ca6b56 100644 --- a/drivers/phy/qualcomm/phy-qcom-qmp.h +++ b/drivers/phy/qualcomm/phy-qcom-qmp.h @@ -32,7 +32,9 @@ #include "phy-qcom-qmp-qserdes-txrx-v7.h" #include "phy-qcom-qmp-qserdes-com-v8.h" +#include "phy-qcom-qmp-usb43-qserdes-com-v8.h" #include "phy-qcom-qmp-qserdes-txrx-v8.h" +#include "phy-qcom-qmp-qserdes-lalb-v8.h" #include "phy-qcom-qmp-qserdes-pll.h" diff --git a/drivers/phy/qualcomm/phy-qcom-sgmii-eth.c b/drivers/phy/qualcomm/phy-qcom-sgmii-eth.c index 5b1c82459c12..8de0fba25ecf 100644 --- a/drivers/phy/qualcomm/phy-qcom-sgmii-eth.c +++ b/drivers/phy/qualcomm/phy-qcom-sgmii-eth.c @@ -10,6 +10,7 @@ #include #include #include +#include #include "phy-qcom-qmp-pcs-sgmii.h" #include "phy-qcom-qmp-qserdes-com-v5.h" @@ -25,7 +26,15 @@ #define QSERDES_PCS_SGMIIPHY_READY BIT(7) #define QSERDES_COM_C_PLL_LOCKED BIT(1) +static const struct regulator_bulk_data qcom_dwmac_sgmii_phy_vregs[] = { + { .supply = "vdda-0p9", .init_load_uA = 46000 }, + { .supply = "vdda-1p2", .init_load_uA = 15000 }, +}; + +#define QCOM_SGMII_NUM_SUPPLIES ARRAY_SIZE(qcom_dwmac_sgmii_phy_vregs) + struct qcom_dwmac_sgmii_phy_data { + struct regulator_bulk_data *vregs; struct regmap *regmap; struct clk *refclk; int speed; @@ -267,8 +276,28 @@ static int qcom_dwmac_sgmii_phy_calibrate(struct phy *phy) static int qcom_dwmac_sgmii_phy_power_on(struct phy *phy) { struct qcom_dwmac_sgmii_phy_data *data = phy_get_drvdata(phy); + int ret; + + ret = regulator_bulk_enable(QCOM_SGMII_NUM_SUPPLIES, data->vregs); + if (ret) + return ret; - return clk_prepare_enable(data->refclk); + ret = clk_prepare_enable(data->refclk); + if (ret) + goto err_disable_regulators; + + ret = qcom_dwmac_sgmii_phy_calibrate(phy); + if (ret) + goto err_disable_clk; + + return 0; + +err_disable_clk: + clk_disable_unprepare(data->refclk); +err_disable_regulators: + regulator_bulk_disable(QCOM_SGMII_NUM_SUPPLIES, data->vregs); + + return ret; } static int qcom_dwmac_sgmii_phy_power_off(struct phy *phy) @@ -283,6 +312,8 @@ static int qcom_dwmac_sgmii_phy_power_off(struct phy *phy) clk_disable_unprepare(data->refclk); + regulator_bulk_disable(QCOM_SGMII_NUM_SUPPLIES, data->vregs); + return 0; } @@ -293,6 +324,9 @@ static int qcom_dwmac_sgmii_phy_set_speed(struct phy *phy, int speed) if (speed != data->speed) data->speed = speed; + if (phy->power_count == 0) + return 0; + return qcom_dwmac_sgmii_phy_calibrate(phy); } @@ -319,6 +353,7 @@ static int qcom_dwmac_sgmii_phy_probe(struct platform_device *pdev) struct phy_provider *provider; void __iomem *base; struct phy *phy; + int ret; data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL); if (!data) @@ -343,6 +378,12 @@ static int qcom_dwmac_sgmii_phy_probe(struct platform_device *pdev) if (IS_ERR(data->refclk)) return PTR_ERR(data->refclk); + ret = devm_regulator_bulk_get_const(dev, QCOM_SGMII_NUM_SUPPLIES, + qcom_dwmac_sgmii_phy_vregs, + &data->vregs); + if (ret) + return ret; + provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate); if (IS_ERR(provider)) return PTR_ERR(provider); diff --git a/drivers/phy/qualcomm/phy-qcom-snps-femto-v2.c b/drivers/phy/qualcomm/phy-qcom-snps-femto-v2.c index eb0b0f61d98e..980ad1fb1e2e 100644 --- a/drivers/phy/qualcomm/phy-qcom-snps-femto-v2.c +++ b/drivers/phy/qualcomm/phy-qcom-snps-femto-v2.c @@ -599,8 +599,18 @@ static int qcom_snps_hsphy_probe(struct platform_device *pdev) return dev_err_probe(dev, ret, "failed to get regulator supplies\n"); + /* + * Enable runtime PM before creating the PHY, phy_create() only enables + * it on the PHY device if already enabled on the parent. Hold a usage + * reference so callbacks cannot run before the PHY is ready. + */ + pm_runtime_get_noresume(dev); pm_runtime_set_active(dev); - pm_runtime_enable(dev); + ret = devm_pm_runtime_enable(dev); + if (ret) { + pm_runtime_put_noidle(dev); + return ret; + } /* * Prevent runtime pm from being ON by default. Users can enable * it using power/control in sysfs. @@ -611,6 +621,7 @@ static int qcom_snps_hsphy_probe(struct platform_device *pdev) if (IS_ERR(generic_phy)) { ret = PTR_ERR(generic_phy); dev_err(dev, "failed to create phy, %d\n", ret); + pm_runtime_put_noidle(dev); return ret; } hsphy->phy = generic_phy; @@ -620,12 +631,15 @@ static int qcom_snps_hsphy_probe(struct platform_device *pdev) qcom_snps_hsphy_read_override_param_seq(dev); phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate); - if (!IS_ERR(phy_provider)) - dev_dbg(dev, "Registered Qcom-SNPS HS phy\n"); - else - pm_runtime_disable(dev); + if (IS_ERR(phy_provider)) { + pm_runtime_put_noidle(dev); + return PTR_ERR(phy_provider); + } - return PTR_ERR_OR_ZERO(phy_provider); + dev_dbg(dev, "Registered Qcom-SNPS HS phy\n"); + pm_runtime_put(dev); + + return 0; } static struct platform_driver qcom_snps_hsphy_driver = { diff --git a/drivers/phy/renesas/Kconfig b/drivers/phy/renesas/Kconfig index e342eef0640b..e128b9c177a5 100644 --- a/drivers/phy/renesas/Kconfig +++ b/drivers/phy/renesas/Kconfig @@ -29,6 +29,7 @@ config PHY_RCAR_GEN3_USB2 depends on ARCH_RENESAS depends on EXTCON || !EXTCON # if EXTCON=m, this cannot be built-in depends on USB_SUPPORT + depends on REGULATOR select GENERIC_PHY select USB_COMMON help diff --git a/drivers/phy/renesas/phy-rcar-gen3-usb2.c b/drivers/phy/renesas/phy-rcar-gen3-usb2.c index a38ead7c8055..4739de1e99fb 100644 --- a/drivers/phy/renesas/phy-rcar-gen3-usb2.c +++ b/drivers/phy/renesas/phy-rcar-gen3-usb2.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -140,6 +141,7 @@ struct rcar_gen3_chan { bool extcon_host; bool is_otg_channel; bool uses_otg_pins; + bool otg_internal_reg; }; struct rcar_gen3_phy_drv_data { @@ -203,28 +205,43 @@ static void rcar_gen3_set_linectrl(struct rcar_gen3_chan *ch, int dp, int dm) writel(val, usb2_base + USB2_LINECTRL1); } -static void rcar_gen3_enable_vbus_ctrl(struct rcar_gen3_chan *ch, int vbus) +static void rcar_gen3_phy_usb2_set_vbus(struct rcar_gen3_chan *ch, + u32 vbus_ctrl_reg, + u32 vbus_ctrl_val, + bool enable) { void __iomem *usb2_base = ch->base; - u32 vbus_ctrl_reg = USB2_ADPCTRL; - u32 vbus_ctrl_val = USB2_ADPCTRL_DRVVBUS; u32 val; + val = readl(usb2_base + vbus_ctrl_reg); + if (enable) + val |= vbus_ctrl_val; + else + val &= ~vbus_ctrl_val; + writel(val, usb2_base + vbus_ctrl_reg); + + dev_vdbg(ch->dev, "%s: reg=0x%08x, val=%08x, enable=%d\n", + __func__, vbus_ctrl_reg, val, enable); +} + +static void rcar_gen3_enable_vbus_ctrl(struct rcar_gen3_chan *ch, int vbus) +{ + if (ch->otg_internal_reg) { + regulator_hardware_enable(ch->vbus, vbus); + return; + } + if (ch->phy_data->no_adp_ctrl || ch->phy_data->vblvl_ctrl) { if (ch->vbus) regulator_hardware_enable(ch->vbus, vbus); - vbus_ctrl_reg = USB2_VBCTRL; - vbus_ctrl_val = USB2_VBCTRL_VBOUT; + rcar_gen3_phy_usb2_set_vbus(ch, USB2_VBCTRL, + USB2_VBCTRL_VBOUT, vbus); + return; } - val = readl(usb2_base + vbus_ctrl_reg); - if (vbus) - val |= vbus_ctrl_val; - else - val &= ~vbus_ctrl_val; - dev_vdbg(ch->dev, "%s: %08x, %d\n", __func__, val, vbus); - writel(val, usb2_base + vbus_ctrl_reg); + rcar_gen3_phy_usb2_set_vbus(ch, USB2_ADPCTRL, + USB2_ADPCTRL_DRVVBUS, vbus); } static void rcar_gen3_control_otg_irq(struct rcar_gen3_chan *ch, int enable) @@ -582,7 +599,7 @@ static int rcar_gen3_phy_usb2_power_on(struct phy *p) u32 val; int ret = 0; - if (channel->vbus) { + if (channel->vbus && !channel->otg_internal_reg) { ret = regulator_enable(channel->vbus); if (ret) return ret; @@ -623,7 +640,7 @@ static int rcar_gen3_phy_usb2_power_off(struct phy *p) } } - if (channel->vbus) + if (channel->vbus && !channel->otg_internal_reg) ret = regulator_disable(channel->vbus); return ret; @@ -809,6 +826,139 @@ static int rcar_gen3_phy_usb2_init_bus(struct rcar_gen3_chan *channel) return ret; } +static int rcar_gen3_phy_usb2_regulator_endisable(struct regulator_dev *rdev, + bool enable) +{ + struct rcar_gen3_chan *channel = rdev_get_drvdata(rdev); + struct device *dev = channel->dev; + int ret; + + ret = pm_runtime_resume_and_get(dev); + if (ret < 0) { + dev_warn(dev, "pm_runtime_get failed: %i\n", ret); + return ret; + } + + rcar_gen3_phy_usb2_set_vbus(channel, USB2_VBCTRL, + USB2_VBCTRL_VBOUT, enable); + pm_runtime_put_noidle(dev); + + return ret; +} + +static int rcar_gen3_phy_usb2_regulator_enable(struct regulator_dev *rdev) +{ + return rcar_gen3_phy_usb2_regulator_endisable(rdev, true); +} + +static int rcar_gen3_phy_usb2_regulator_disable(struct regulator_dev *rdev) +{ + return rcar_gen3_phy_usb2_regulator_endisable(rdev, false); +} + +static int rcar_gen3_phy_usb2_regulator_is_enabled(struct regulator_dev *rdev) +{ + struct rcar_gen3_chan *channel = rdev_get_drvdata(rdev); + void __iomem *usb2_base = channel->base; + struct device *dev = channel->dev; + u32 vbus_ctrl_reg = USB2_VBCTRL; + u32 val; + int ret; + + ret = pm_runtime_resume_and_get(dev); + if (ret < 0) { + dev_warn(dev, "pm_runtime_get failed: %i\n", ret); + return ret; + } + + val = readl(usb2_base + vbus_ctrl_reg); + + pm_runtime_put_noidle(dev); + dev_dbg(channel->dev, "%s: %08x\n", __func__, val); + + return (val & USB2_VBCTRL_VBOUT) ? 1 : 0; +} + +static const struct regulator_ops rcar_gen3_phy_usb2_regulator_ops = { + .enable = rcar_gen3_phy_usb2_regulator_enable, + .disable = rcar_gen3_phy_usb2_regulator_disable, + .is_enabled = rcar_gen3_phy_usb2_regulator_is_enabled, +}; + +static const struct regulator_desc rcar_gen3_phy_usb2_regulator = { + .name = "otg-vbus-regulator", + .of_match = of_match_ptr("vbus-regulator"), + .ops = &rcar_gen3_phy_usb2_regulator_ops, + .type = REGULATOR_VOLTAGE, + .owner = THIS_MODULE, + .fixed_uV = 5000000, + .n_voltages = 1, +}; + +static void rcar_gen3_phy_usb2_vbus_disable_action(void *data) +{ + struct regulator *vbus = data; + + regulator_disable(vbus); +} + +static int rcar_gen3_phy_usb2_vbus_regulator_get_exclusive_enable(struct rcar_gen3_chan *channel, + bool enable) +{ + struct device *dev = channel->dev; + int ret; + + channel->vbus = devm_regulator_get_exclusive(dev, "vbus"); + if (IS_ERR(channel->vbus)) { + ret = PTR_ERR(channel->vbus); + /* If vbus-regulator node was present vbus regulator should be available */ + if (channel->otg_internal_reg) + return ret; + + if (ret == -EPROBE_DEFER) + return ret; + + return 0; + } + + if (enable) { + ret = regulator_enable(channel->vbus); + if (ret) + return ret; + } + + if (regulator_is_enabled(channel->vbus)) + return devm_add_action_or_reset(dev, rcar_gen3_phy_usb2_vbus_disable_action, + channel->vbus); + + return 0; +} + +static int rcar_gen3_phy_usb2_vbus_regulator_register(struct rcar_gen3_chan *channel) +{ + struct device *dev = channel->dev; + struct regulator_config rcfg = { .dev = dev, }; + struct regulator_dev *rdev; + bool enable = false; + + rcfg.of_node = of_get_available_child_by_name(dev->of_node, + "vbus-regulator"); + if (rcfg.of_node) { + rcfg.driver_data = channel; + rdev = devm_regulator_register(dev, &rcar_gen3_phy_usb2_regulator, + &rcfg); + of_node_put(rcfg.of_node); + if (IS_ERR(rdev)) + return dev_err_probe(dev, PTR_ERR(rdev), + "Failed to create vbus-regulator\n"); + + channel->otg_internal_reg = true; + enable = true; + } + + return rcar_gen3_phy_usb2_vbus_regulator_get_exclusive_enable(channel, enable); +} + static int rcar_gen3_phy_usb2_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; @@ -881,10 +1031,13 @@ static int rcar_gen3_phy_usb2_probe(struct platform_device *pdev) phy_set_drvdata(channel->rphys[i].phy, &channel->rphys[i]); } - if (channel->phy_data->no_adp_ctrl && channel->is_otg_channel) - channel->vbus = devm_regulator_get_exclusive(dev, "vbus"); - else + if (channel->phy_data->no_adp_ctrl && channel->is_otg_channel) { + ret = rcar_gen3_phy_usb2_vbus_regulator_register(channel); + if (ret) + return ret; + } else { channel->vbus = devm_regulator_get_optional(dev, "vbus"); + } if (IS_ERR(channel->vbus)) { if (PTR_ERR(channel->vbus) == -EPROBE_DEFER) { ret = PTR_ERR(channel->vbus); diff --git a/drivers/phy/rockchip/phy-rockchip-inno-csidphy.c b/drivers/phy/rockchip/phy-rockchip-inno-csidphy.c index c79fb53d8ee5..5281f8dea0ad 100644 --- a/drivers/phy/rockchip/phy-rockchip-inno-csidphy.c +++ b/drivers/phy/rockchip/phy-rockchip-inno-csidphy.c @@ -170,7 +170,7 @@ static const struct hsfreq_range rk1808_mipidphy_hsfreq_ranges[] = { { 299, 0x06}, { 399, 0x08}, { 499, 0x0b}, { 599, 0x0e}, { 699, 0x10}, { 799, 0x12}, { 999, 0x16}, {1199, 0x1e}, {1399, 0x23}, {1599, 0x2d}, {1799, 0x32}, {1999, 0x37}, - {2199, 0x3c}, {2399, 0x41}, {2499, 0x46} + {2199, 0x3c}, {2399, 0x41}, {2500, 0x46} }; static const struct hsfreq_range rk3326_mipidphy_hsfreq_ranges[] = { diff --git a/drivers/phy/starfive/phy-jh7110-dphy-rx.c b/drivers/phy/starfive/phy-jh7110-dphy-rx.c index 0b039e1f71c5..d06f21ad6332 100644 --- a/drivers/phy/starfive/phy-jh7110-dphy-rx.c +++ b/drivers/phy/starfive/phy-jh7110-dphy-rx.c @@ -150,6 +150,7 @@ static int stf_dphy_probe(struct platform_device *pdev) { struct phy_provider *phy_provider; struct stf_dphy *dphy; + int ret; dphy = devm_kzalloc(&pdev->dev, sizeof(*dphy), GFP_KERNEL); if (!dphy) @@ -190,7 +191,9 @@ static int stf_dphy_probe(struct platform_device *pdev) return PTR_ERR(dphy->phy); } - pm_runtime_enable(&pdev->dev); + ret = devm_pm_runtime_enable(&pdev->dev); + if (ret) + return ret; phy_set_drvdata(dphy->phy, dphy); phy_provider = devm_of_phy_provider_register(&pdev->dev, diff --git a/drivers/phy/starfive/phy-jh7110-dphy-tx.c b/drivers/phy/starfive/phy-jh7110-dphy-tx.c index c64d1c91b130..181491a93807 100644 --- a/drivers/phy/starfive/phy-jh7110-dphy-tx.c +++ b/drivers/phy/starfive/phy-jh7110-dphy-tx.c @@ -392,6 +392,7 @@ static int stf_dphy_probe(struct platform_device *pdev) { struct phy_provider *phy_provider; struct stf_dphy *dphy; + int ret; dphy = devm_kzalloc(&pdev->dev, sizeof(*dphy), GFP_KERNEL); if (!dphy) @@ -406,7 +407,9 @@ static int stf_dphy_probe(struct platform_device *pdev) if (IS_ERR(dphy->topsys)) return PTR_ERR(dphy->topsys); - pm_runtime_enable(&pdev->dev); + ret = devm_pm_runtime_enable(&pdev->dev); + if (ret) + return ret; dphy->txesc_clk = devm_clk_get(&pdev->dev, "txesc"); if (IS_ERR(dphy->txesc_clk)) diff --git a/drivers/phy/sunplus/phy-sunplus-usb2.c b/drivers/phy/sunplus/phy-sunplus-usb2.c index 637a5fbae6d9..0ad4c7160d17 100644 --- a/drivers/phy/sunplus/phy-sunplus-usb2.c +++ b/drivers/phy/sunplus/phy-sunplus-usb2.c @@ -116,11 +116,11 @@ static int sp_uphy_init(struct phy *phy) ret = clk_prepare_enable(usbphy->phy_clk); if (ret) - goto err_clk; + return ret; ret = reset_control_deassert(usbphy->rstc); if (ret) - goto err_reset; + goto err_clk; /* Default value modification */ writel(HIGH_MASK_BITS | 0x4002, usbphy->moon4_regs + UPHY_CONTROL0); @@ -129,7 +129,7 @@ static int sp_uphy_init(struct phy *phy) /* disconnect voltage */ ret = update_disc_vol(usbphy); if (ret < 0) - return ret; + goto err_reset; /* board uphy 0 internal register modification for tid certification */ val = readl(usbphy->phy_regs + CONFIG9); diff --git a/drivers/pinctrl/bcm/pinctrl-bcm2835.c b/drivers/pinctrl/bcm/pinctrl-bcm2835.c index c165674c5b4d..bda4273138ff 100644 --- a/drivers/pinctrl/bcm/pinctrl-bcm2835.c +++ b/drivers/pinctrl/bcm/pinctrl-bcm2835.c @@ -1351,7 +1351,6 @@ static int bcm2835_pinctrl_probe(struct platform_device *pdev) pc->pctl_desc = *pdata->pctl_desc; pc->pctl_dev = devm_pinctrl_register(dev, &pc->pctl_desc, pc); if (IS_ERR(pc->pctl_dev)) { - gpiochip_remove(&pc->gpio_chip); return PTR_ERR(pc->pctl_dev); } diff --git a/drivers/pinctrl/mediatek/mtk-eint.c b/drivers/pinctrl/mediatek/mtk-eint.c index 5f12af59a91b..df99401c0b7d 100644 --- a/drivers/pinctrl/mediatek/mtk-eint.c +++ b/drivers/pinctrl/mediatek/mtk-eint.c @@ -12,8 +12,10 @@ */ #include +#include #include #include +#include #include #include #include @@ -504,6 +506,27 @@ int mtk_eint_find_irq(struct mtk_eint *eint, unsigned long eint_n) } EXPORT_SYMBOL_GPL(mtk_eint_find_irq); +static void mtk_eint_teardown(void *data) +{ + struct mtk_eint *eint = data; + unsigned int i, virq; + + /* Detach the demux handler so it can no longer reference freed data. */ + irq_set_chained_handler_and_data(eint->irq, NULL, NULL); + + /* Wait for any in-flight handler to finish before tearing down. */ + synchronize_irq(eint->irq); + + /* Dispose of all child mappings before the domain is removed. */ + for (i = 0; i < eint->hw->ap_num; i++) { + virq = irq_find_mapping(eint->domain, i); + if (virq) + irq_dispose_mapping(virq); + } + + irq_domain_remove(eint->domain); +} + int mtk_eint_do_init(struct mtk_eint *eint, struct mtk_eint_pin *eint_pin) { unsigned int size, i, port, virq, inst = 0; @@ -596,7 +619,7 @@ int mtk_eint_do_init(struct mtk_eint *eint, struct mtk_eint_pin *eint_pin) irq_set_chained_handler_and_data(eint->irq, mtk_eint_irq_handler, eint); - return 0; + return devm_add_action_or_reset(eint->dev, mtk_eint_teardown, eint); err_eint: for (i = 0; i < eint->nbase; i++) { diff --git a/drivers/pinctrl/mediatek/pinctrl-airoha.c b/drivers/pinctrl/mediatek/pinctrl-airoha.c index 9f8e05e3671d..bdb5e7012d46 100644 --- a/drivers/pinctrl/mediatek/pinctrl-airoha.c +++ b/drivers/pinctrl/mediatek/pinctrl-airoha.c @@ -49,7 +49,7 @@ /* MUX */ #define REG_GPIO_2ND_I2C_MODE 0x0214 -#define GPIO_MDC_IO_MASTER_MODE_MODE BIT(14) +#define GPIO_MDC_IO_MASTER_MODE_MASK BIT(14) #define GPIO_I2C_MASTER_MODE_MODE BIT(13) #define GPIO_I2S_MODE_MASK BIT(12) #define GPIO_I2C_SLAVE_MODE_MODE BIT(11) @@ -66,10 +66,12 @@ #define GPIO_2ND_I2C_MODE_MASK BIT(0) #define REG_GPIO_SPI_CS1_MODE 0x0218 +#define AN7583_GPIO_MDC_IO_MASTER_MODE_MASK BIT(22) #define GPIO_PCM_SPI_CS4_MODE_MASK BIT(21) #define GPIO_PCM_SPI_CS3_MODE_MASK BIT(20) #define GPIO_PCM_SPI_CS2_MODE_P156_MASK BIT(19) #define GPIO_PCM_SPI_CS2_MODE_P128_MASK BIT(18) +#define AN7583_GPIO_PCM_SPI_CS2_MODE_MASK BIT(18) #define GPIO_PCM_SPI_CS1_MODE_MASK BIT(17) #define GPIO_PCM_SPI_MODE_MASK BIT(16) #define GPIO_PCM2_MODE_MASK BIT(13) @@ -83,6 +85,18 @@ #define GPIO_SPI_CS1_MODE_MASK BIT(0) #define REG_GPIO_PON_MODE 0x021c +#define AN7583_MDIO_0_GPIO_MODE_MASK BIT(26) +#define AN7583_MDC_0_GPIO_MODE_MASK BIT(25) +#define AN7583_UART_RXD_GPIO_MODE_MASK BIT(24) +#define AN7583_UART_TXD_GPIO_MODE_MASK BIT(23) +#define AN7583_SPI_MISO_GPIO_MODE_MASK BIT(22) +#define AN7583_SPI_MOSI_GPIO_MODE_MASK BIT(21) +#define AN7583_SPI_CS_GPIO_MODE_MASK BIT(20) +#define AN7583_SPI_CLK_GPIO_MODE_MASK BIT(19) +#define AN7583_I2C1_SDA_GPIO_MODE_MASK BIT(18) +#define AN7583_I2C1_SCL_GPIO_MODE_MASK BIT(17) +#define AN7583_I2C0_SDA_GPIO_MODE_MASK BIT(16) +#define AN7583_I2C0_SCL_GPIO_MODE_MASK BIT(15) #define GPIO_PARALLEL_NAND_MODE_MASK BIT(14) #define GPIO_SGMII_MDIO_MODE_MASK BIT(13) #define GPIO_PCIE_RESET2_MASK BIT(12) @@ -127,6 +141,8 @@ /* CONF */ #define REG_I2C_SDA_E2 0x001c +#define AN7583_I2C1_SCL_E2_MASK BIT(16) +#define AN7583_I2C1_SDA_E2_MASK BIT(15) #define SPI_MISO_E2_MASK BIT(14) #define SPI_MOSI_E2_MASK BIT(13) #define SPI_CLK_E2_MASK BIT(12) @@ -134,12 +150,16 @@ #define PCIE2_RESET_E2_MASK BIT(10) #define PCIE1_RESET_E2_MASK BIT(9) #define PCIE0_RESET_E2_MASK BIT(8) +#define AN7583_MDIO_0_E2_MASK BIT(5) +#define AN7583_MDC_0_E2_MASK BIT(4) #define UART1_RXD_E2_MASK BIT(3) #define UART1_TXD_E2_MASK BIT(2) #define I2C_SCL_E2_MASK BIT(1) #define I2C_SDA_E2_MASK BIT(0) #define REG_I2C_SDA_E4 0x0020 +#define AN7583_I2C1_SCL_E4_MASK BIT(16) +#define AN7583_I2C1_SDA_E4_MASK BIT(15) #define SPI_MISO_E4_MASK BIT(14) #define SPI_MOSI_E4_MASK BIT(13) #define SPI_CLK_E4_MASK BIT(12) @@ -147,6 +167,8 @@ #define PCIE2_RESET_E4_MASK BIT(10) #define PCIE1_RESET_E4_MASK BIT(9) #define PCIE0_RESET_E4_MASK BIT(8) +#define AN7583_MDIO_0_E4_MASK BIT(5) +#define AN7583_MDC_0_E4_MASK BIT(4) #define UART1_RXD_E4_MASK BIT(3) #define UART1_TXD_E4_MASK BIT(2) #define I2C_SCL_E4_MASK BIT(1) @@ -158,6 +180,8 @@ #define REG_GPIO_H_E4 0x0030 #define REG_I2C_SDA_PU 0x0044 +#define AN7583_I2C1_SCL_PU_MASK BIT(16) +#define AN7583_I2C1_SDA_PU_MASK BIT(15) #define SPI_MISO_PU_MASK BIT(14) #define SPI_MOSI_PU_MASK BIT(13) #define SPI_CLK_PU_MASK BIT(12) @@ -165,12 +189,16 @@ #define PCIE2_RESET_PU_MASK BIT(10) #define PCIE1_RESET_PU_MASK BIT(9) #define PCIE0_RESET_PU_MASK BIT(8) +#define AN7583_MDIO_0_PU_MASK BIT(5) +#define AN7583_MDC_0_PU_MASK BIT(4) #define UART1_RXD_PU_MASK BIT(3) #define UART1_TXD_PU_MASK BIT(2) #define I2C_SCL_PU_MASK BIT(1) #define I2C_SDA_PU_MASK BIT(0) #define REG_I2C_SDA_PD 0x0048 +#define AN7583_I2C1_SCL_PD_MASK BIT(16) +#define AN7583_I2C1_SDA_PD_MASK BIT(15) #define SPI_MISO_PD_MASK BIT(14) #define SPI_MOSI_PD_MASK BIT(13) #define SPI_CLK_PD_MASK BIT(12) @@ -178,6 +206,8 @@ #define PCIE2_RESET_PD_MASK BIT(10) #define PCIE1_RESET_PD_MASK BIT(9) #define PCIE0_RESET_PD_MASK BIT(8) +#define AN7583_MDIO_0_PD_MASK BIT(5) +#define AN7583_MDC_0_PD_MASK BIT(4) #define UART1_RXD_PD_MASK BIT(3) #define UART1_TXD_PD_MASK BIT(2) #define I2C_SCL_PD_MASK BIT(1) @@ -452,9 +482,9 @@ static struct pinctrl_pin_desc en7581_pinctrl_pins[] = { PINCTRL_PIN(57, "gpio44"), PINCTRL_PIN(58, "gpio45"), PINCTRL_PIN(59, "gpio46"), - PINCTRL_PIN(61, "pcie_reset0"), - PINCTRL_PIN(62, "pcie_reset1"), - PINCTRL_PIN(63, "pcie_reset2"), + PINCTRL_PIN(60, "pcie_reset0"), + PINCTRL_PIN(61, "pcie_reset1"), + PINCTRL_PIN(62, "pcie_reset2"), }; static const int en7581_pon_pins[] = { 49, 50, 51, 52, 53, 54 }; @@ -537,9 +567,12 @@ static const int en7581_gpio43_pins[] = { 56 }; static const int en7581_gpio44_pins[] = { 57 }; static const int en7581_gpio45_pins[] = { 58 }; static const int en7581_gpio46_pins[] = { 59 }; -static const int en7581_pcie_reset0_pins[] = { 61 }; -static const int en7581_pcie_reset1_pins[] = { 62 }; -static const int en7581_pcie_reset2_pins[] = { 63 }; +static const int en7581_gpio47_pins[] = { 60 }; +static const int en7581_gpio48_pins[] = { 61 }; +static const int en7581_gpio49_pins[] = { 62 }; +static const int en7581_pcie_reset0_pins[] = { 60 }; +static const int en7581_pcie_reset1_pins[] = { 61 }; +static const int en7581_pcie_reset2_pins[] = { 62 }; static const struct pingroup en7581_pinctrl_groups[] = { PINCTRL_PIN_GROUP("pon", en7581_pon), @@ -622,15 +655,243 @@ static const struct pingroup en7581_pinctrl_groups[] = { PINCTRL_PIN_GROUP("gpio44", en7581_gpio44), PINCTRL_PIN_GROUP("gpio45", en7581_gpio45), PINCTRL_PIN_GROUP("gpio46", en7581_gpio46), + PINCTRL_PIN_GROUP("gpio47", en7581_gpio47), + PINCTRL_PIN_GROUP("gpio48", en7581_gpio48), + PINCTRL_PIN_GROUP("gpio49", en7581_gpio49), PINCTRL_PIN_GROUP("pcie_reset0", en7581_pcie_reset0), PINCTRL_PIN_GROUP("pcie_reset1", en7581_pcie_reset1), PINCTRL_PIN_GROUP("pcie_reset2", en7581_pcie_reset2), }; +static struct pinctrl_pin_desc an7583_pinctrl_pins[] = { + PINCTRL_PIN(2, "gpio0"), + PINCTRL_PIN(3, "gpio1"), + PINCTRL_PIN(4, "gpio2"), + PINCTRL_PIN(5, "gpio3"), + PINCTRL_PIN(6, "gpio4"), + PINCTRL_PIN(7, "gpio5"), + PINCTRL_PIN(8, "gpio6"), + PINCTRL_PIN(9, "gpio7"), + PINCTRL_PIN(10, "gpio8"), + PINCTRL_PIN(11, "gpio9"), + PINCTRL_PIN(12, "gpio10"), + PINCTRL_PIN(13, "gpio11"), + PINCTRL_PIN(14, "gpio12"), + PINCTRL_PIN(15, "gpio13"), + PINCTRL_PIN(16, "gpio14"), + PINCTRL_PIN(17, "gpio15"), + PINCTRL_PIN(18, "gpio16"), + PINCTRL_PIN(19, "gpio17"), + PINCTRL_PIN(20, "gpio18"), + PINCTRL_PIN(21, "gpio19"), + PINCTRL_PIN(22, "gpio20"), + PINCTRL_PIN(23, "gpio21"), + PINCTRL_PIN(24, "gpio22"), + PINCTRL_PIN(25, "gpio23"), + PINCTRL_PIN(26, "gpio24"), + PINCTRL_PIN(27, "gpio25"), + PINCTRL_PIN(28, "gpio26"), + PINCTRL_PIN(29, "gpio27"), + PINCTRL_PIN(30, "gpio28"), + PINCTRL_PIN(31, "gpio29"), + PINCTRL_PIN(32, "gpio30"), + PINCTRL_PIN(33, "gpio31"), + PINCTRL_PIN(34, "gpio32"), + PINCTRL_PIN(35, "gpio33"), + PINCTRL_PIN(36, "gpio34"), + PINCTRL_PIN(37, "gpio35"), + PINCTRL_PIN(38, "gpio36"), + PINCTRL_PIN(39, "gpio37"), + PINCTRL_PIN(40, "gpio38"), + PINCTRL_PIN(41, "i2c0_scl"), + PINCTRL_PIN(42, "i2c0_sda"), + PINCTRL_PIN(43, "i2c1_scl"), + PINCTRL_PIN(44, "i2c1_sda"), + PINCTRL_PIN(45, "spi_clk"), + PINCTRL_PIN(46, "spi_cs"), + PINCTRL_PIN(47, "spi_mosi"), + PINCTRL_PIN(48, "spi_miso"), + PINCTRL_PIN(49, "uart_txd"), + PINCTRL_PIN(50, "uart_rxd"), + PINCTRL_PIN(51, "pcie_reset0"), + PINCTRL_PIN(52, "pcie_reset1"), + PINCTRL_PIN(53, "mdc_0"), + PINCTRL_PIN(54, "mdio_0"), +}; + +static const int an7583_pon_pins[] = { 15, 16, 17, 18, 19, 20 }; +static const int an7583_pon_tod_1pps_pins[] = { 32 }; +static const int an7583_gsw_tod_1pps_pins[] = { 32 }; +static const int an7583_sipo_pins[] = { 34, 35 }; +static const int an7583_sipo_rclk_pins[] = { 34, 35, 33 }; +static const int an7583_mdio_pins[] = { 43, 44 }; +static const int an7583_uart2_pins[] = { 34, 35 }; +static const int an7583_uart2_cts_rts_pins[] = { 32, 33 }; +static const int an7583_hsuart_pins[] = { 30, 31 }; +static const int an7583_hsuart_cts_rts_pins[] = { 28, 29 }; +static const int an7583_npu_uart_pins[] = { 7, 8 }; +static const int an7583_uart4_pins[] = { 7, 8 }; +static const int an7583_uart5_pins[] = { 23, 24 }; +static const int an7583_i2c0_pins[] = { 41, 42 }; +static const int an7583_i2c1_pins[] = { 43, 44 }; +static const int an7583_jtag_udi_pins[] = { 23, 24, 22, 25, 26 }; +static const int an7583_jtag_dfd_pins[] = { 23, 24, 22, 25, 26 }; +static const int an7583_pcm1_pins[] = { 10, 11, 12, 13, 14 }; +static const int an7583_pcm2_pins[] = { 28, 29, 30, 31, 24 }; +static const int an7583_spi_pins[] = { 45, 46, 47, 48 }; +static const int an7583_spi_quad_pins[] = { 25, 26 }; +static const int an7583_spi_cs1_pins[] = { 27 }; +static const int an7583_pcm_spi_pins[] = { 28, 29, 30, 31, 10, 11, 12, 13 }; +static const int an7583_pcm_spi_rst_pins[] = { 14 }; +static const int an7583_pcm_spi_cs1_pins[] = { 24 }; +static const int an7583_emmc_pins[] = { 7, 8, 9, 22, 23, 24, 25, 26, 45, 46, 47 }; +static const int an7583_pnand_pins[] = { 7, 8, 9, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 45, 46, 47, 48 }; +static const int an7583_gpio0_pins[] = { 2 }; +static const int an7583_gpio1_pins[] = { 3 }; +static const int an7583_gpio2_pins[] = { 4 }; +static const int an7583_gpio3_pins[] = { 5 }; +static const int an7583_gpio4_pins[] = { 6 }; +static const int an7583_gpio5_pins[] = { 7 }; +static const int an7583_gpio6_pins[] = { 8 }; +static const int an7583_gpio7_pins[] = { 9 }; +static const int an7583_gpio8_pins[] = { 10 }; +static const int an7583_gpio9_pins[] = { 11 }; +static const int an7583_gpio10_pins[] = { 12 }; +static const int an7583_gpio11_pins[] = { 13 }; +static const int an7583_gpio12_pins[] = { 14 }; +static const int an7583_gpio13_pins[] = { 15 }; +static const int an7583_gpio14_pins[] = { 16 }; +static const int an7583_gpio15_pins[] = { 17 }; +static const int an7583_gpio16_pins[] = { 18 }; +static const int an7583_gpio17_pins[] = { 19 }; +static const int an7583_gpio18_pins[] = { 20 }; +static const int an7583_gpio19_pins[] = { 21 }; +static const int an7583_gpio20_pins[] = { 22 }; +static const int an7583_gpio21_pins[] = { 23 }; +static const int an7583_gpio22_pins[] = { 24 }; +static const int an7583_gpio23_pins[] = { 25 }; +static const int an7583_gpio24_pins[] = { 26 }; +static const int an7583_gpio25_pins[] = { 27 }; +static const int an7583_gpio26_pins[] = { 28 }; +static const int an7583_gpio27_pins[] = { 29 }; +static const int an7583_gpio28_pins[] = { 30 }; +static const int an7583_gpio29_pins[] = { 31 }; +static const int an7583_gpio30_pins[] = { 32 }; +static const int an7583_gpio31_pins[] = { 33 }; +static const int an7583_gpio32_pins[] = { 34 }; +static const int an7583_gpio33_pins[] = { 35 }; +static const int an7583_gpio34_pins[] = { 36 }; +static const int an7583_gpio35_pins[] = { 37 }; +static const int an7583_gpio36_pins[] = { 38 }; +static const int an7583_gpio37_pins[] = { 39 }; +static const int an7583_gpio38_pins[] = { 40 }; +static const int an7583_gpio39_pins[] = { 41 }; +static const int an7583_gpio40_pins[] = { 42 }; +static const int an7583_gpio41_pins[] = { 43 }; +static const int an7583_gpio42_pins[] = { 44 }; +static const int an7583_gpio43_pins[] = { 45 }; +static const int an7583_gpio44_pins[] = { 46 }; +static const int an7583_gpio45_pins[] = { 47 }; +static const int an7583_gpio46_pins[] = { 48 }; +static const int an7583_gpio47_pins[] = { 49 }; +static const int an7583_gpio48_pins[] = { 50 }; +static const int an7583_gpio49_pins[] = { 51 }; +static const int an7583_gpio50_pins[] = { 52 }; +static const int an7583_gpio51_pins[] = { 53 }; +static const int an7583_gpio52_pins[] = { 54 }; +static const int an7583_pcie_reset0_pins[] = { 51 }; +static const int an7583_pcie_reset1_pins[] = { 52 }; + +static const struct pingroup an7583_pinctrl_groups[] = { + PINCTRL_PIN_GROUP("pon", an7583_pon), + PINCTRL_PIN_GROUP("pon_tod_1pps", an7583_pon_tod_1pps), + PINCTRL_PIN_GROUP("gsw_tod_1pps", an7583_gsw_tod_1pps), + PINCTRL_PIN_GROUP("sipo", an7583_sipo), + PINCTRL_PIN_GROUP("sipo_rclk", an7583_sipo_rclk), + PINCTRL_PIN_GROUP("mdio", an7583_mdio), + PINCTRL_PIN_GROUP("uart2", an7583_uart2), + PINCTRL_PIN_GROUP("uart2_cts_rts", an7583_uart2_cts_rts), + PINCTRL_PIN_GROUP("hsuart", an7583_hsuart), + PINCTRL_PIN_GROUP("hsuart_cts_rts", an7583_hsuart_cts_rts), + PINCTRL_PIN_GROUP("npu_uart", an7583_npu_uart), + PINCTRL_PIN_GROUP("uart4", an7583_uart4), + PINCTRL_PIN_GROUP("uart5", an7583_uart5), + PINCTRL_PIN_GROUP("i2c0", an7583_i2c0), + PINCTRL_PIN_GROUP("i2c1", an7583_i2c1), + PINCTRL_PIN_GROUP("jtag_udi", an7583_jtag_udi), + PINCTRL_PIN_GROUP("jtag_dfd", an7583_jtag_dfd), + PINCTRL_PIN_GROUP("pcm1", an7583_pcm1), + PINCTRL_PIN_GROUP("pcm2", an7583_pcm2), + PINCTRL_PIN_GROUP("spi", an7583_spi), + PINCTRL_PIN_GROUP("spi_quad", an7583_spi_quad), + PINCTRL_PIN_GROUP("spi_cs1", an7583_spi_cs1), + PINCTRL_PIN_GROUP("pcm_spi", an7583_pcm_spi), + PINCTRL_PIN_GROUP("pcm_spi_rst", an7583_pcm_spi_rst), + PINCTRL_PIN_GROUP("pcm_spi_cs1", an7583_pcm_spi_cs1), + PINCTRL_PIN_GROUP("emmc", an7583_emmc), + PINCTRL_PIN_GROUP("pnand", an7583_pnand), + PINCTRL_PIN_GROUP("gpio0", an7583_gpio0), + PINCTRL_PIN_GROUP("gpio1", an7583_gpio1), + PINCTRL_PIN_GROUP("gpio2", an7583_gpio2), + PINCTRL_PIN_GROUP("gpio3", an7583_gpio3), + PINCTRL_PIN_GROUP("gpio4", an7583_gpio4), + PINCTRL_PIN_GROUP("gpio5", an7583_gpio5), + PINCTRL_PIN_GROUP("gpio6", an7583_gpio6), + PINCTRL_PIN_GROUP("gpio7", an7583_gpio7), + PINCTRL_PIN_GROUP("gpio8", an7583_gpio8), + PINCTRL_PIN_GROUP("gpio9", an7583_gpio9), + PINCTRL_PIN_GROUP("gpio10", an7583_gpio10), + PINCTRL_PIN_GROUP("gpio11", an7583_gpio11), + PINCTRL_PIN_GROUP("gpio12", an7583_gpio12), + PINCTRL_PIN_GROUP("gpio13", an7583_gpio13), + PINCTRL_PIN_GROUP("gpio14", an7583_gpio14), + PINCTRL_PIN_GROUP("gpio15", an7583_gpio15), + PINCTRL_PIN_GROUP("gpio16", an7583_gpio16), + PINCTRL_PIN_GROUP("gpio17", an7583_gpio17), + PINCTRL_PIN_GROUP("gpio18", an7583_gpio18), + PINCTRL_PIN_GROUP("gpio19", an7583_gpio19), + PINCTRL_PIN_GROUP("gpio20", an7583_gpio20), + PINCTRL_PIN_GROUP("gpio21", an7583_gpio21), + PINCTRL_PIN_GROUP("gpio22", an7583_gpio22), + PINCTRL_PIN_GROUP("gpio23", an7583_gpio23), + PINCTRL_PIN_GROUP("gpio24", an7583_gpio24), + PINCTRL_PIN_GROUP("gpio25", an7583_gpio25), + PINCTRL_PIN_GROUP("gpio26", an7583_gpio26), + PINCTRL_PIN_GROUP("gpio27", an7583_gpio27), + PINCTRL_PIN_GROUP("gpio28", an7583_gpio28), + PINCTRL_PIN_GROUP("gpio29", an7583_gpio29), + PINCTRL_PIN_GROUP("gpio30", an7583_gpio30), + PINCTRL_PIN_GROUP("gpio31", an7583_gpio31), + PINCTRL_PIN_GROUP("gpio32", an7583_gpio32), + PINCTRL_PIN_GROUP("gpio33", an7583_gpio33), + PINCTRL_PIN_GROUP("gpio34", an7583_gpio34), + PINCTRL_PIN_GROUP("gpio35", an7583_gpio35), + PINCTRL_PIN_GROUP("gpio36", an7583_gpio36), + PINCTRL_PIN_GROUP("gpio37", an7583_gpio37), + PINCTRL_PIN_GROUP("gpio38", an7583_gpio38), + PINCTRL_PIN_GROUP("gpio39", an7583_gpio39), + PINCTRL_PIN_GROUP("gpio40", an7583_gpio40), + PINCTRL_PIN_GROUP("gpio41", an7583_gpio41), + PINCTRL_PIN_GROUP("gpio42", an7583_gpio42), + PINCTRL_PIN_GROUP("gpio43", an7583_gpio43), + PINCTRL_PIN_GROUP("gpio44", an7583_gpio44), + PINCTRL_PIN_GROUP("gpio45", an7583_gpio45), + PINCTRL_PIN_GROUP("gpio46", an7583_gpio46), + PINCTRL_PIN_GROUP("gpio47", an7583_gpio47), + PINCTRL_PIN_GROUP("gpio48", an7583_gpio48), + PINCTRL_PIN_GROUP("gpio49", an7583_gpio49), + PINCTRL_PIN_GROUP("gpio50", an7583_gpio50), + PINCTRL_PIN_GROUP("gpio51", an7583_gpio51), + PINCTRL_PIN_GROUP("gpio52", an7583_gpio52), + PINCTRL_PIN_GROUP("pcie_reset0", an7583_pcie_reset0), + PINCTRL_PIN_GROUP("pcie_reset1", an7583_pcie_reset1), +}; + static const char *const pon_groups[] = { "pon" }; static const char *const tod_1pps_groups[] = { "pon_tod_1pps", "gsw_tod_1pps" }; static const char *const sipo_groups[] = { "sipo", "sipo_rclk" }; static const char *const mdio_groups[] = { "mdio" }; +static const char *const an7583_mdio_groups[] = { "mdio" }; static const char *const uart_groups[] = { "uart2", "uart2_cts_rts", "hsuart", "hsuart_cts_rts", "uart4", "uart5" }; @@ -643,11 +904,20 @@ static const char *const pcm_spi_groups[] = { "pcm_spi", "pcm_spi_int", "pcm_spi_cs2_p156", "pcm_spi_cs2_p128", "pcm_spi_cs3", "pcm_spi_cs4" }; +static const char *const an7583_pcm_spi_groups[] = { "pcm_spi", + "pcm_spi_rst", "pcm_spi_cs1" }; static const char *const i2s_groups[] = { "i2s" }; static const char *const emmc_groups[] = { "emmc" }; static const char *const pnand_groups[] = { "pnand" }; +static const char *const gpio_groups[] = { "gpio47", "gpio48", "gpio49" }; static const char *const pcie_reset_groups[] = { "pcie_reset0", "pcie_reset1", "pcie_reset2" }; +static const char *const an7583_gpio_groups[] = { "gpio39", "gpio40", "gpio41", + "gpio42", "gpio43", "gpio44", + "gpio45", "gpio46", "gpio47", + "gpio48", "gpio49", "gpio50", + "gpio51", "gpio52" }; +static const char *const an7583_pcie_reset_groups[] = { "pcie_reset0", "pcie_reset1" }; static const char *const pwm_groups[] = { "gpio0", "gpio1", "gpio2", "gpio3", "gpio4", "gpio5", @@ -669,7 +939,32 @@ static const char *const pwm_groups[] = { "gpio0", "gpio1", "gpio40", "gpio41", "gpio42", "gpio43", "gpio44", "gpio45", - "gpio46", "gpio47" }; + "gpio46", "gpio47", + "gpio48", "gpio49" }; +static const char *const an7583_pwm_groups[] = { "gpio0", "gpio1", + "gpio2", "gpio3", + "gpio4", "gpio5", + "gpio6", "gpio7", + "gpio8", "gpio9", + "gpio10", "gpio11", + "gpio12", "gpio13", + "gpio14", "gpio15", + "gpio16", "gpio17", + "gpio18", "gpio19", + "gpio20", "gpio21", + "gpio22", "gpio23", + "gpio24", "gpio25", + "gpio26", "gpio27", + "gpio28", "gpio29", + "gpio30", "gpio31", + "gpio36", "gpio37", + "gpio38", "gpio39", + "gpio40", "gpio41", + "gpio42", "gpio43", + "gpio44", "gpio45", + "gpio46", "gpio47", + "gpio48", "gpio49", + "gpio50", "gpio51" }; static const char *const phy1_led0_groups[] = { "gpio33", "gpio34", "gpio35", "gpio42" }; static const char *const phy2_led0_groups[] = { "gpio33", "gpio34", @@ -686,6 +981,22 @@ static const char *const phy3_led1_groups[] = { "gpio43", "gpio44", "gpio45", "gpio46" }; static const char *const phy4_led1_groups[] = { "gpio43", "gpio44", "gpio45", "gpio46" }; +static const char *const an7583_phy1_led0_groups[] = { "gpio1", "gpio2", + "gpio3", "gpio4" }; +static const char *const an7583_phy2_led0_groups[] = { "gpio1", "gpio2", + "gpio3", "gpio4" }; +static const char *const an7583_phy3_led0_groups[] = { "gpio1", "gpio2", + "gpio3", "gpio4" }; +static const char *const an7583_phy4_led0_groups[] = { "gpio1", "gpio2", + "gpio3", "gpio4" }; +static const char *const an7583_phy1_led1_groups[] = { "gpio8", "gpio9", + "gpio10", "gpio11" }; +static const char *const an7583_phy2_led1_groups[] = { "gpio8", "gpio9", + "gpio10", "gpio11" }; +static const char *const an7583_phy3_led1_groups[] = { "gpio8", "gpio9", + "gpio10", "gpio11" }; +static const char *const an7583_phy4_led1_groups[] = { "gpio8", "gpio9", + "gpio10", "gpio11" }; static const struct airoha_pinctrl_func_group pon_func_group[] = { { @@ -750,8 +1061,8 @@ static const struct airoha_pinctrl_func_group mdio_func_group[] = { .regmap[0] = { AIROHA_FUNC_MUX, REG_GPIO_2ND_I2C_MODE, - GPIO_MDC_IO_MASTER_MODE_MODE, - GPIO_MDC_IO_MASTER_MODE_MODE + GPIO_MDC_IO_MASTER_MODE_MASK, + GPIO_MDC_IO_MASTER_MODE_MASK }, .regmap[1] = { AIROHA_FUNC_MUX, @@ -763,6 +1074,25 @@ static const struct airoha_pinctrl_func_group mdio_func_group[] = { }, }; +static const struct airoha_pinctrl_func_group an7583_mdio_func_group[] = { + { + .name = "mdio", + .regmap[0] = { + AIROHA_FUNC_MUX, + REG_GPIO_PON_MODE, + GPIO_SGMII_MDIO_MODE_MASK, + GPIO_SGMII_MDIO_MODE_MASK + }, + .regmap[1] = { + AIROHA_FUNC_MUX, + REG_GPIO_SPI_CS1_MODE, + AN7583_GPIO_MDC_IO_MASTER_MODE_MASK, + AN7583_GPIO_MDC_IO_MASTER_MODE_MASK + }, + .regmap_size = 2, + }, +}; + static const struct airoha_pinctrl_func_group uart_func_group[] = { { .name = "uart2", @@ -1004,989 +1334,605 @@ static const struct airoha_pinctrl_func_group pcm_spi_func_group[] = { }, }; -static const struct airoha_pinctrl_func_group i2s_func_group[] = { - { - .name = "i2s", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_I2S_MODE_MASK, - GPIO_I2S_MODE_MASK - }, - .regmap_size = 1, - }, -}; - -static const struct airoha_pinctrl_func_group emmc_func_group[] = { - { - .name = "emmc", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_PON_MODE, - GPIO_EMMC_MODE_MASK, - GPIO_EMMC_MODE_MASK - }, - .regmap_size = 1, - }, -}; - -static const struct airoha_pinctrl_func_group pnand_func_group[] = { - { - .name = "pnand", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_PON_MODE, - GPIO_PARALLEL_NAND_MODE_MASK, - GPIO_PARALLEL_NAND_MODE_MASK - }, - .regmap_size = 1, - }, -}; - -static const struct airoha_pinctrl_func_group pcie_reset_func_group[] = { +static const struct airoha_pinctrl_func_group an7583_pcm_spi_func_group[] = { { - .name = "pcie_reset0", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_PON_MODE, - GPIO_PCIE_RESET0_MASK, - GPIO_PCIE_RESET0_MASK - }, - .regmap_size = 1, - }, { - .name = "pcie_reset1", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_PON_MODE, - GPIO_PCIE_RESET1_MASK, - GPIO_PCIE_RESET1_MASK - }, - .regmap_size = 1, - }, { - .name = "pcie_reset2", + .name = "pcm_spi", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_PON_MODE, - GPIO_PCIE_RESET2_MASK, - GPIO_PCIE_RESET2_MASK - }, - .regmap_size = 1, - }, -}; - -/* PWM */ -static const struct airoha_pinctrl_func_group pwm_func_group[] = { - { - .name = "gpio0", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO0_FLASH_MODE_CFG, - GPIO0_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio1", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO1_FLASH_MODE_CFG, - GPIO1_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio2", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO2_FLASH_MODE_CFG, - GPIO2_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio3", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO3_FLASH_MODE_CFG, - GPIO3_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio4", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO4_FLASH_MODE_CFG, - GPIO4_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio5", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO5_FLASH_MODE_CFG, - GPIO5_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio6", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO6_FLASH_MODE_CFG, - GPIO6_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio7", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO7_FLASH_MODE_CFG, - GPIO7_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio8", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO8_FLASH_MODE_CFG, - GPIO8_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio9", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO9_FLASH_MODE_CFG, - GPIO9_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio10", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO10_FLASH_MODE_CFG, - GPIO10_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio11", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO11_FLASH_MODE_CFG, - GPIO11_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio12", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO12_FLASH_MODE_CFG, - GPIO12_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio13", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO13_FLASH_MODE_CFG, - GPIO13_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio14", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO14_FLASH_MODE_CFG, - GPIO14_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio15", - .regmap[0] = { - AIROHA_FUNC_PWM_MUX, - REG_GPIO_FLASH_MODE_CFG, - GPIO15_FLASH_MODE_CFG, - GPIO15_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio16", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO16_FLASH_MODE_CFG, - GPIO16_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio17", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO17_FLASH_MODE_CFG, - GPIO17_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio18", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO18_FLASH_MODE_CFG, - GPIO18_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio19", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO19_FLASH_MODE_CFG, - GPIO19_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio20", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO20_FLASH_MODE_CFG, - GPIO20_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio21", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO21_FLASH_MODE_CFG, - GPIO21_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio22", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO22_FLASH_MODE_CFG, - GPIO22_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio23", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO23_FLASH_MODE_CFG, - GPIO23_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio24", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO24_FLASH_MODE_CFG, - GPIO24_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio25", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO25_FLASH_MODE_CFG, - GPIO25_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio26", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO26_FLASH_MODE_CFG, - GPIO26_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio27", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO27_FLASH_MODE_CFG, - GPIO27_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio28", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO28_FLASH_MODE_CFG, - GPIO28_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio29", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO29_FLASH_MODE_CFG, - GPIO29_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio30", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO30_FLASH_MODE_CFG, - GPIO30_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio31", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO31_FLASH_MODE_CFG, - GPIO31_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio36", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO36_FLASH_MODE_CFG, - GPIO36_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio37", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO37_FLASH_MODE_CFG, - GPIO37_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio38", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO38_FLASH_MODE_CFG, - GPIO38_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio39", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO39_FLASH_MODE_CFG, - GPIO39_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio40", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO40_FLASH_MODE_CFG, - GPIO40_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio41", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO41_FLASH_MODE_CFG, - GPIO41_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio42", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO42_FLASH_MODE_CFG, - GPIO42_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio43", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO43_FLASH_MODE_CFG, - GPIO43_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio44", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO44_FLASH_MODE_CFG, - GPIO44_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio45", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO45_FLASH_MODE_CFG, - GPIO45_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio46", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO46_FLASH_MODE_CFG, - GPIO46_FLASH_MODE_CFG - }, - .regmap_size = 1, - }, { - .name = "gpio47", - .regmap[0] = { - AIROHA_FUNC_PWM_EXT_MUX, - REG_GPIO_FLASH_MODE_CFG_EXT, - GPIO47_FLASH_MODE_CFG, - GPIO47_FLASH_MODE_CFG + REG_GPIO_SPI_CS1_MODE, + GPIO_PCM_SPI_MODE_MASK, + GPIO_PCM_SPI_MODE_MASK }, .regmap_size = 1, - }, -}; - -static const struct airoha_pinctrl_func_group phy1_led0_func_group[] = { - { - .name = "gpio33", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED0_MODE_MASK, - GPIO_LAN0_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, { - .name = "gpio34", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED0_MODE_MASK, - GPIO_LAN1_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, { - .name = "gpio35", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED0_MODE_MASK, - GPIO_LAN2_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, { - .name = "gpio42", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED0_MODE_MASK, - GPIO_LAN3_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, -}; - -static const struct airoha_pinctrl_func_group phy2_led0_func_group[] = { - { - .name = "gpio33", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED0_MODE_MASK, - GPIO_LAN0_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(1) - }, - .regmap_size = 2, }, { - .name = "gpio34", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED0_MODE_MASK, - GPIO_LAN1_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(1) - }, - .regmap_size = 2, - }, { - .name = "gpio35", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED0_MODE_MASK, - GPIO_LAN2_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(1) - }, - .regmap_size = 2, - }, { - .name = "gpio42", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED0_MODE_MASK, - GPIO_LAN3_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(1) - }, - .regmap_size = 2, - }, -}; - -static const struct airoha_pinctrl_func_group phy3_led0_func_group[] = { - { - .name = "gpio33", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED0_MODE_MASK, - GPIO_LAN0_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(2) - }, - .regmap_size = 2, - }, { - .name = "gpio34", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED0_MODE_MASK, - GPIO_LAN1_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(2) - }, - .regmap_size = 2, - }, { - .name = "gpio35", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED0_MODE_MASK, - GPIO_LAN2_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(2) - }, - .regmap_size = 2, - }, { - .name = "gpio42", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED0_MODE_MASK, - GPIO_LAN3_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(2) - }, - .regmap_size = 2, - }, -}; - -static const struct airoha_pinctrl_func_group phy4_led0_func_group[] = { - { - .name = "gpio33", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED0_MODE_MASK, - GPIO_LAN0_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(3) - }, - .regmap_size = 2, - }, { - .name = "gpio34", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED0_MODE_MASK, - GPIO_LAN1_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(3) - }, - .regmap_size = 2, - }, { - .name = "gpio35", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED0_MODE_MASK, - GPIO_LAN2_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(3) - }, - .regmap_size = 2, - }, { - .name = "gpio42", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED0_MODE_MASK, - GPIO_LAN3_LED0_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED0_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(3) - }, - .regmap_size = 2, - }, -}; - -static const struct airoha_pinctrl_func_group phy1_led1_func_group[] = { - { - .name = "gpio43", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED1_MODE_MASK, - GPIO_LAN0_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, { - .name = "gpio44", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED1_MODE_MASK, - GPIO_LAN1_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, { - .name = "gpio45", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED1_MODE_MASK, - GPIO_LAN2_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, { - .name = "gpio46", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED1_MODE_MASK, - GPIO_LAN3_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(0) - }, - .regmap_size = 2, - }, -}; - -static const struct airoha_pinctrl_func_group phy2_led1_func_group[] = { - { - .name = "gpio43", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED1_MODE_MASK, - GPIO_LAN0_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(1) - }, - .regmap_size = 2, - }, { - .name = "gpio44", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED1_MODE_MASK, - GPIO_LAN1_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(1) - }, - .regmap_size = 2, - }, { - .name = "gpio45", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED1_MODE_MASK, - GPIO_LAN2_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(1) - }, - .regmap_size = 2, - }, { - .name = "gpio46", - .regmap[0] = { - AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED1_MODE_MASK, - GPIO_LAN3_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(1) - }, - .regmap_size = 2, - }, -}; - -static const struct airoha_pinctrl_func_group phy3_led1_func_group[] = { - { - .name = "gpio43", + .name = "pcm_spi_int", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED1_MODE_MASK, - GPIO_LAN0_LED1_MODE_MASK + REG_GPIO_SPI_CS1_MODE, + GPIO_PCM_INT_MODE_MASK, + GPIO_PCM_INT_MODE_MASK }, - .regmap[1] = { + .regmap_size = 1, + }, { + .name = "pcm_spi_rst", + .regmap[0] = { AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(2) + REG_GPIO_SPI_CS1_MODE, + GPIO_PCM_RESET_MODE_MASK, + GPIO_PCM_RESET_MODE_MASK }, - .regmap_size = 2, + .regmap_size = 1, }, { - .name = "gpio44", + .name = "pcm_spi_cs1", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED1_MODE_MASK, - GPIO_LAN1_LED1_MODE_MASK + REG_GPIO_SPI_CS1_MODE, + GPIO_PCM_SPI_CS1_MODE_MASK, + GPIO_PCM_SPI_CS1_MODE_MASK }, - .regmap[1] = { + .regmap_size = 1, + }, { + .name = "pcm_spi_cs2", + .regmap[0] = { AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(2) + REG_GPIO_SPI_CS1_MODE, + AN7583_GPIO_PCM_SPI_CS2_MODE_MASK, + AN7583_GPIO_PCM_SPI_CS2_MODE_MASK }, - .regmap_size = 2, + .regmap_size = 1, }, { - .name = "gpio45", + .name = "pcm_spi_cs3", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED1_MODE_MASK, - GPIO_LAN2_LED1_MODE_MASK + REG_GPIO_SPI_CS1_MODE, + GPIO_PCM_SPI_CS3_MODE_MASK, + GPIO_PCM_SPI_CS3_MODE_MASK }, - .regmap[1] = { + .regmap_size = 1, + }, { + .name = "pcm_spi_cs4", + .regmap[0] = { AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(2) + REG_GPIO_SPI_CS1_MODE, + GPIO_PCM_SPI_CS4_MODE_MASK, + GPIO_PCM_SPI_CS4_MODE_MASK }, - .regmap_size = 2, - }, { - .name = "gpio46", + .regmap_size = 1, + }, +}; + +static const struct airoha_pinctrl_func_group i2s_func_group[] = { + { + .name = "i2s", .regmap[0] = { AIROHA_FUNC_MUX, REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED1_MODE_MASK, - GPIO_LAN3_LED1_MODE_MASK + GPIO_I2S_MODE_MASK, + GPIO_I2S_MODE_MASK }, - .regmap[1] = { + .regmap_size = 1, + }, +}; + +static const struct airoha_pinctrl_func_group emmc_func_group[] = { + { + .name = "emmc", + .regmap[0] = { AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(2) + REG_GPIO_PON_MODE, + GPIO_EMMC_MODE_MASK, + GPIO_EMMC_MODE_MASK }, - .regmap_size = 2, + .regmap_size = 1, }, }; -static const struct airoha_pinctrl_func_group phy4_led1_func_group[] = { +static const struct airoha_pinctrl_func_group pnand_func_group[] = { { - .name = "gpio43", + .name = "pnand", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN0_LED1_MODE_MASK, - GPIO_LAN0_LED1_MODE_MASK + REG_GPIO_PON_MODE, + GPIO_PARALLEL_NAND_MODE_MASK, + GPIO_PARALLEL_NAND_MODE_MASK }, - .regmap[1] = { + .regmap_size = 1, + }, +}; + +#define AIROHA_PINCTRL_GPIO(gpio, mux_val) \ + { \ + .name = (gpio), \ + .regmap[0] = { \ + AIROHA_FUNC_MUX, \ + REG_GPIO_PON_MODE, \ + (mux_val), \ + (mux_val) \ + }, \ + .regmap_size = 1, \ + } + +#define AIROHA_PINCTRL_GPIO_EXT(gpio, mux_val, smux_val) \ + { \ + .name = (gpio), \ + .regmap[0] = { \ + AIROHA_FUNC_PWM_EXT_MUX, \ + REG_GPIO_FLASH_MODE_CFG_EXT, \ + (mux_val), \ + 0 \ + }, \ + .regmap[1] = { \ + AIROHA_FUNC_MUX, \ + REG_GPIO_PON_MODE, \ + (smux_val), \ + (smux_val) \ + }, \ + .regmap_size = 2, \ + } + +static const struct airoha_pinctrl_func_group gpio_func_group[] = { + AIROHA_PINCTRL_GPIO_EXT("gpio47", GPIO47_FLASH_MODE_CFG, + GPIO_PCIE_RESET0_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio48", GPIO48_FLASH_MODE_CFG, + GPIO_PCIE_RESET1_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio49", GPIO49_FLASH_MODE_CFG, + GPIO_PCIE_RESET2_MASK), +}; + +static const struct airoha_pinctrl_func_group pcie_reset_func_group[] = { + { + .name = "pcie_reset0", + .regmap[0] = { AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN0_LED_MAPPING_MASK, - LAN0_PHY_LED_MAP(3) + REG_GPIO_PON_MODE, + GPIO_PCIE_RESET0_MASK, + 0 }, - .regmap_size = 2, + .regmap_size = 1, }, { - .name = "gpio44", + .name = "pcie_reset1", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN1_LED1_MODE_MASK, - GPIO_LAN1_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN1_LED_MAPPING_MASK, - LAN1_PHY_LED_MAP(3) + REG_GPIO_PON_MODE, + GPIO_PCIE_RESET1_MASK, + 0 }, - .regmap_size = 2, + .regmap_size = 1, }, { - .name = "gpio45", + .name = "pcie_reset2", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN2_LED1_MODE_MASK, - GPIO_LAN2_LED1_MODE_MASK + REG_GPIO_PON_MODE, + GPIO_PCIE_RESET2_MASK, + 0 }, - .regmap[1] = { + .regmap_size = 1, + }, +}; + +static const struct airoha_pinctrl_func_group an7583_gpio_func_group[] = { + AIROHA_PINCTRL_GPIO_EXT("gpio39", GPIO39_FLASH_MODE_CFG, + AN7583_I2C0_SCL_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio40", GPIO40_FLASH_MODE_CFG, + AN7583_I2C0_SDA_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio41", GPIO41_FLASH_MODE_CFG, + AN7583_I2C1_SCL_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio42", GPIO42_FLASH_MODE_CFG, + AN7583_I2C1_SDA_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio43", GPIO43_FLASH_MODE_CFG, + AN7583_SPI_CLK_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio44", GPIO44_FLASH_MODE_CFG, + AN7583_SPI_CS_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio45", GPIO45_FLASH_MODE_CFG, + AN7583_SPI_MOSI_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio46", GPIO46_FLASH_MODE_CFG, + AN7583_SPI_MISO_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio47", GPIO47_FLASH_MODE_CFG, + AN7583_UART_TXD_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio48", GPIO48_FLASH_MODE_CFG, + AN7583_UART_RXD_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio49", GPIO49_FLASH_MODE_CFG, + GPIO_PCIE_RESET0_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio50", GPIO50_FLASH_MODE_CFG, + GPIO_PCIE_RESET1_MASK), + AIROHA_PINCTRL_GPIO_EXT("gpio51", GPIO51_FLASH_MODE_CFG, + AN7583_MDC_0_GPIO_MODE_MASK), + AIROHA_PINCTRL_GPIO("gpio52", AN7583_MDIO_0_GPIO_MODE_MASK), +}; + +static const struct airoha_pinctrl_func_group an7583_pcie_reset_func_group[] = { + { + .name = "pcie_reset0", + .regmap[0] = { AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN2_LED_MAPPING_MASK, - LAN2_PHY_LED_MAP(3) + REG_GPIO_PON_MODE, + GPIO_PCIE_RESET0_MASK, + 0 }, - .regmap_size = 2, + .regmap_size = 1, }, { - .name = "gpio46", + .name = "pcie_reset1", .regmap[0] = { AIROHA_FUNC_MUX, - REG_GPIO_2ND_I2C_MODE, - GPIO_LAN3_LED1_MODE_MASK, - GPIO_LAN3_LED1_MODE_MASK - }, - .regmap[1] = { - AIROHA_FUNC_MUX, - REG_LAN_LED1_MAPPING, - LAN3_LED_MAPPING_MASK, - LAN3_PHY_LED_MAP(3) + REG_GPIO_PON_MODE, + GPIO_PCIE_RESET1_MASK, + 0 }, - .regmap_size = 2, + .regmap_size = 1, }, }; +/* PWM */ +#define AIROHA_PINCTRL_PWM(gpio, mux_val) \ + { \ + .name = (gpio), \ + .regmap[0] = { \ + AIROHA_FUNC_PWM_MUX, \ + REG_GPIO_FLASH_MODE_CFG, \ + (mux_val), \ + (mux_val) \ + }, \ + .regmap_size = 1, \ + } \ + +#define AIROHA_PINCTRL_PWM_EXT(gpio, mux_val) \ + { \ + .name = (gpio), \ + .regmap[0] = { \ + AIROHA_FUNC_PWM_EXT_MUX, \ + REG_GPIO_FLASH_MODE_CFG_EXT, \ + (mux_val), \ + (mux_val) \ + }, \ + .regmap_size = 1, \ + } \ + +#define AIROHA_PINCTRL_PWM_EXT_SEC(gpio, mux_val, smux_val) \ + { \ + .name = (gpio), \ + .regmap[0] = { \ + AIROHA_FUNC_PWM_EXT_MUX, \ + REG_GPIO_FLASH_MODE_CFG_EXT, \ + (mux_val), \ + (mux_val) \ + }, \ + .regmap[1] = { \ + AIROHA_FUNC_MUX, \ + REG_GPIO_PON_MODE, \ + (smux_val), \ + (smux_val) \ + }, \ + .regmap_size = 2, \ + } + +static const struct airoha_pinctrl_func_group pwm_func_group[] = { + AIROHA_PINCTRL_PWM("gpio0", GPIO0_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio1", GPIO1_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio2", GPIO2_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio3", GPIO3_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio4", GPIO4_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio5", GPIO5_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio6", GPIO6_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio7", GPIO7_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio8", GPIO8_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio9", GPIO9_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio10", GPIO10_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio11", GPIO11_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio12", GPIO12_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio13", GPIO13_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio14", GPIO14_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio15", GPIO15_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio16", GPIO16_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio17", GPIO17_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio18", GPIO18_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio19", GPIO19_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio20", GPIO20_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio21", GPIO21_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio22", GPIO22_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio23", GPIO23_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio24", GPIO24_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio25", GPIO25_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio26", GPIO26_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio27", GPIO27_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio28", GPIO28_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio29", GPIO29_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio30", GPIO30_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio31", GPIO31_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio36", GPIO36_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio37", GPIO37_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio38", GPIO38_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio39", GPIO39_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio40", GPIO40_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio41", GPIO41_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio42", GPIO42_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio43", GPIO43_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio44", GPIO44_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio45", GPIO45_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio46", GPIO46_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio47", GPIO47_FLASH_MODE_CFG, + GPIO_PCIE_RESET0_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio48", GPIO48_FLASH_MODE_CFG, + GPIO_PCIE_RESET1_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio49", GPIO49_FLASH_MODE_CFG, + GPIO_PCIE_RESET2_MASK), +}; + +static const struct airoha_pinctrl_func_group an7583_pwm_func_group[] = { + AIROHA_PINCTRL_PWM("gpio0", GPIO0_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio1", GPIO1_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio2", GPIO2_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio3", GPIO3_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio4", GPIO4_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio5", GPIO5_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio6", GPIO6_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio7", GPIO7_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio8", GPIO8_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio9", GPIO9_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio10", GPIO10_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio11", GPIO11_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio12", GPIO12_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio13", GPIO13_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio14", GPIO14_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM("gpio15", GPIO15_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio16", GPIO16_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio17", GPIO17_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio18", GPIO18_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio19", GPIO19_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio20", GPIO20_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio21", GPIO21_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio22", GPIO22_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio23", GPIO23_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio24", GPIO24_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio25", GPIO25_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio26", GPIO26_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio27", GPIO27_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio28", GPIO28_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio29", GPIO29_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio30", GPIO30_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio31", GPIO31_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio36", GPIO36_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio37", GPIO37_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT("gpio38", GPIO38_FLASH_MODE_CFG), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio39", GPIO39_FLASH_MODE_CFG, + AN7583_I2C0_SCL_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio40", GPIO40_FLASH_MODE_CFG, + AN7583_I2C0_SDA_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio41", GPIO41_FLASH_MODE_CFG, + AN7583_I2C1_SCL_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio42", GPIO42_FLASH_MODE_CFG, + AN7583_I2C1_SDA_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio43", GPIO43_FLASH_MODE_CFG, + AN7583_SPI_CLK_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio44", GPIO44_FLASH_MODE_CFG, + AN7583_SPI_CS_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio45", GPIO45_FLASH_MODE_CFG, + AN7583_SPI_MOSI_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio46", GPIO46_FLASH_MODE_CFG, + AN7583_SPI_MISO_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio47", GPIO47_FLASH_MODE_CFG, + AN7583_UART_TXD_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio48", GPIO48_FLASH_MODE_CFG, + AN7583_UART_RXD_GPIO_MODE_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio49", GPIO49_FLASH_MODE_CFG, + GPIO_PCIE_RESET0_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio50", GPIO50_FLASH_MODE_CFG, + GPIO_PCIE_RESET1_MASK), + AIROHA_PINCTRL_PWM_EXT_SEC("gpio51", GPIO51_FLASH_MODE_CFG, + AN7583_MDC_0_GPIO_MODE_MASK), +}; + +#define AIROHA_PINCTRL_PHY_LED0(gpio, mux_val, map_mask, map_val) \ + { \ + .name = (gpio), \ + .regmap[0] = { \ + AIROHA_FUNC_MUX, \ + REG_GPIO_2ND_I2C_MODE, \ + (mux_val), \ + (mux_val), \ + }, \ + .regmap[1] = { \ + AIROHA_FUNC_MUX, \ + REG_LAN_LED0_MAPPING, \ + (map_mask), \ + (map_val), \ + }, \ + .regmap_size = 2, \ + } + +#define AIROHA_PINCTRL_PHY_LED1(gpio, mux_val, map_mask, map_val) \ + { \ + .name = (gpio), \ + .regmap[0] = { \ + AIROHA_FUNC_MUX, \ + REG_GPIO_2ND_I2C_MODE, \ + (mux_val), \ + (mux_val), \ + }, \ + .regmap[1] = { \ + AIROHA_FUNC_MUX, \ + REG_LAN_LED1_MAPPING, \ + (map_mask), \ + (map_val), \ + }, \ + .regmap_size = 2, \ + } + +static const struct airoha_pinctrl_func_group phy1_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio33", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED0("gpio34", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED0("gpio35", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED0("gpio42", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(0)), +}; + +static const struct airoha_pinctrl_func_group phy2_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio33", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED0("gpio34", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED0("gpio35", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED0("gpio42", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(1)), +}; + +static const struct airoha_pinctrl_func_group phy3_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio33", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED0("gpio34", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED0("gpio35", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED0("gpio42", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(2)), +}; + +static const struct airoha_pinctrl_func_group phy4_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio33", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED0("gpio34", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED0("gpio35", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED0("gpio42", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(3)), +}; + +static const struct airoha_pinctrl_func_group phy1_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio43", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED1("gpio44", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED1("gpio45", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED1("gpio46", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(0)), +}; + +static const struct airoha_pinctrl_func_group phy2_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio43", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED1("gpio44", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED1("gpio45", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED1("gpio46", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(1)), +}; + +static const struct airoha_pinctrl_func_group phy3_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio43", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED1("gpio44", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED1("gpio45", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED1("gpio46", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(2)), +}; + +static const struct airoha_pinctrl_func_group phy4_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio43", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED1("gpio44", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED1("gpio45", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED1("gpio46", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(3)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy1_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio1", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED0("gpio2", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED0("gpio3", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED0("gpio4", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(0)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy2_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio1", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED0("gpio2", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED0("gpio3", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED0("gpio4", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(1)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy3_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio1", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED0("gpio2", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED0("gpio3", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED0("gpio4", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(2)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy4_led0_func_group[] = { + AIROHA_PINCTRL_PHY_LED0("gpio1", GPIO_LAN0_LED0_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED0("gpio2", GPIO_LAN1_LED0_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED0("gpio3", GPIO_LAN2_LED0_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED0("gpio4", GPIO_LAN3_LED0_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(3)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy1_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio8", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED1("gpio9", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED1("gpio10", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(0)), + AIROHA_PINCTRL_PHY_LED1("gpio11", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(0)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy2_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio8", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED1("gpio9", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED1("gpio10", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(1)), + AIROHA_PINCTRL_PHY_LED1("gpio11", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(1)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy3_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio8", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED1("gpio9", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED1("gpio10", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(2)), + AIROHA_PINCTRL_PHY_LED1("gpio11", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(2)), +}; + +static const struct airoha_pinctrl_func_group an7583_phy4_led1_func_group[] = { + AIROHA_PINCTRL_PHY_LED1("gpio8", GPIO_LAN0_LED1_MODE_MASK, + LAN0_LED_MAPPING_MASK, LAN0_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED1("gpio9", GPIO_LAN1_LED1_MODE_MASK, + LAN1_LED_MAPPING_MASK, LAN1_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED1("gpio10", GPIO_LAN2_LED1_MODE_MASK, + LAN2_LED_MAPPING_MASK, LAN2_PHY_LED_MAP(3)), + AIROHA_PINCTRL_PHY_LED1("gpio11", GPIO_LAN3_LED1_MODE_MASK, + LAN3_LED_MAPPING_MASK, LAN3_PHY_LED_MAP(3)), +}; + static const struct airoha_pinctrl_func en7581_pinctrl_funcs[] = { PINCTRL_FUNC_DESC("pon", pon), PINCTRL_FUNC_DESC("tod_1pps", tod_1pps), @@ -2001,6 +1947,7 @@ static const struct airoha_pinctrl_func en7581_pinctrl_funcs[] = { PINCTRL_FUNC_DESC("i2s", i2s), PINCTRL_FUNC_DESC("emmc", emmc), PINCTRL_FUNC_DESC("pnand", pnand), + PINCTRL_FUNC_DESC("gpio", gpio), PINCTRL_FUNC_DESC("pcie_reset", pcie_reset), PINCTRL_FUNC_DESC("pwm", pwm), PINCTRL_FUNC_DESC("phy1_led0", phy1_led0), @@ -2013,11 +1960,37 @@ static const struct airoha_pinctrl_func en7581_pinctrl_funcs[] = { PINCTRL_FUNC_DESC("phy4_led1", phy4_led1), }; +static const struct airoha_pinctrl_func an7583_pinctrl_funcs[] = { + PINCTRL_FUNC_DESC("pon", pon), + PINCTRL_FUNC_DESC("tod_1pps", tod_1pps), + PINCTRL_FUNC_DESC("sipo", sipo), + PINCTRL_FUNC_DESC("mdio", an7583_mdio), + PINCTRL_FUNC_DESC("uart", uart), + PINCTRL_FUNC_DESC("i2c", i2c), + PINCTRL_FUNC_DESC("jtag", jtag), + PINCTRL_FUNC_DESC("pcm", pcm), + PINCTRL_FUNC_DESC("spi", spi), + PINCTRL_FUNC_DESC("pcm_spi", an7583_pcm_spi), + PINCTRL_FUNC_DESC("emmc", emmc), + PINCTRL_FUNC_DESC("pnand", pnand), + PINCTRL_FUNC_DESC("gpio", an7583_gpio), + PINCTRL_FUNC_DESC("pcie_reset", an7583_pcie_reset), + PINCTRL_FUNC_DESC("pwm", an7583_pwm), + PINCTRL_FUNC_DESC("phy1_led0", an7583_phy1_led0), + PINCTRL_FUNC_DESC("phy2_led0", an7583_phy2_led0), + PINCTRL_FUNC_DESC("phy3_led0", an7583_phy3_led0), + PINCTRL_FUNC_DESC("phy4_led0", an7583_phy4_led0), + PINCTRL_FUNC_DESC("phy1_led1", an7583_phy1_led1), + PINCTRL_FUNC_DESC("phy2_led1", an7583_phy2_led1), + PINCTRL_FUNC_DESC("phy3_led1", an7583_phy3_led1), + PINCTRL_FUNC_DESC("phy4_led1", an7583_phy4_led1), +}; + static const struct airoha_pinctrl_conf en7581_pinctrl_pullup_conf[] = { PINCTRL_CONF_DESC(0, REG_I2C_SDA_PU, UART1_TXD_PU_MASK), PINCTRL_CONF_DESC(1, REG_I2C_SDA_PU, UART1_RXD_PU_MASK), - PINCTRL_CONF_DESC(2, REG_I2C_SDA_PU, I2C_SDA_PU_MASK), - PINCTRL_CONF_DESC(3, REG_I2C_SDA_PU, I2C_SCL_PU_MASK), + PINCTRL_CONF_DESC(2, REG_I2C_SDA_PU, I2C_SCL_PU_MASK), + PINCTRL_CONF_DESC(3, REG_I2C_SDA_PU, I2C_SDA_PU_MASK), PINCTRL_CONF_DESC(4, REG_I2C_SDA_PU, SPI_CS0_PU_MASK), PINCTRL_CONF_DESC(5, REG_I2C_SDA_PU, SPI_CLK_PU_MASK), PINCTRL_CONF_DESC(6, REG_I2C_SDA_PU, SPI_MOSI_PU_MASK), @@ -2069,16 +2042,72 @@ static const struct airoha_pinctrl_conf en7581_pinctrl_pullup_conf[] = { PINCTRL_CONF_DESC(57, REG_GPIO_H_PU, BIT(12)), PINCTRL_CONF_DESC(58, REG_GPIO_H_PU, BIT(13)), PINCTRL_CONF_DESC(59, REG_GPIO_H_PU, BIT(14)), - PINCTRL_CONF_DESC(61, REG_I2C_SDA_PU, PCIE0_RESET_PU_MASK), - PINCTRL_CONF_DESC(62, REG_I2C_SDA_PU, PCIE1_RESET_PU_MASK), - PINCTRL_CONF_DESC(63, REG_I2C_SDA_PU, PCIE2_RESET_PU_MASK), + PINCTRL_CONF_DESC(60, REG_I2C_SDA_PU, PCIE0_RESET_PU_MASK), + PINCTRL_CONF_DESC(61, REG_I2C_SDA_PU, PCIE1_RESET_PU_MASK), + PINCTRL_CONF_DESC(62, REG_I2C_SDA_PU, PCIE2_RESET_PU_MASK), +}; + +static const struct airoha_pinctrl_conf an7583_pinctrl_pullup_conf[] = { + PINCTRL_CONF_DESC(2, REG_GPIO_L_PU, BIT(0)), + PINCTRL_CONF_DESC(3, REG_GPIO_L_PU, BIT(1)), + PINCTRL_CONF_DESC(4, REG_GPIO_L_PU, BIT(2)), + PINCTRL_CONF_DESC(5, REG_GPIO_L_PU, BIT(3)), + PINCTRL_CONF_DESC(6, REG_GPIO_L_PU, BIT(4)), + PINCTRL_CONF_DESC(7, REG_GPIO_L_PU, BIT(5)), + PINCTRL_CONF_DESC(8, REG_GPIO_L_PU, BIT(6)), + PINCTRL_CONF_DESC(9, REG_GPIO_L_PU, BIT(7)), + PINCTRL_CONF_DESC(10, REG_GPIO_L_PU, BIT(8)), + PINCTRL_CONF_DESC(11, REG_GPIO_L_PU, BIT(9)), + PINCTRL_CONF_DESC(12, REG_GPIO_L_PU, BIT(10)), + PINCTRL_CONF_DESC(13, REG_GPIO_L_PU, BIT(11)), + PINCTRL_CONF_DESC(14, REG_GPIO_L_PU, BIT(12)), + PINCTRL_CONF_DESC(15, REG_GPIO_L_PU, BIT(13)), + PINCTRL_CONF_DESC(16, REG_GPIO_L_PU, BIT(14)), + PINCTRL_CONF_DESC(17, REG_GPIO_L_PU, BIT(15)), + PINCTRL_CONF_DESC(18, REG_GPIO_L_PU, BIT(16)), + PINCTRL_CONF_DESC(19, REG_GPIO_L_PU, BIT(17)), + PINCTRL_CONF_DESC(20, REG_GPIO_L_PU, BIT(18)), + PINCTRL_CONF_DESC(21, REG_GPIO_L_PU, BIT(19)), + PINCTRL_CONF_DESC(22, REG_GPIO_L_PU, BIT(20)), + PINCTRL_CONF_DESC(23, REG_GPIO_L_PU, BIT(21)), + PINCTRL_CONF_DESC(24, REG_GPIO_L_PU, BIT(22)), + PINCTRL_CONF_DESC(25, REG_GPIO_L_PU, BIT(23)), + PINCTRL_CONF_DESC(26, REG_GPIO_L_PU, BIT(24)), + PINCTRL_CONF_DESC(27, REG_GPIO_L_PU, BIT(25)), + PINCTRL_CONF_DESC(28, REG_GPIO_L_PU, BIT(26)), + PINCTRL_CONF_DESC(29, REG_GPIO_L_PU, BIT(27)), + PINCTRL_CONF_DESC(30, REG_GPIO_L_PU, BIT(28)), + PINCTRL_CONF_DESC(31, REG_GPIO_L_PU, BIT(29)), + PINCTRL_CONF_DESC(32, REG_GPIO_L_PU, BIT(30)), + PINCTRL_CONF_DESC(33, REG_GPIO_L_PU, BIT(31)), + PINCTRL_CONF_DESC(34, REG_GPIO_H_PU, BIT(0)), + PINCTRL_CONF_DESC(35, REG_GPIO_H_PU, BIT(1)), + PINCTRL_CONF_DESC(36, REG_GPIO_H_PU, BIT(2)), + PINCTRL_CONF_DESC(37, REG_GPIO_H_PU, BIT(3)), + PINCTRL_CONF_DESC(38, REG_GPIO_H_PU, BIT(4)), + PINCTRL_CONF_DESC(39, REG_GPIO_H_PU, BIT(5)), + PINCTRL_CONF_DESC(40, REG_GPIO_H_PU, BIT(6)), + PINCTRL_CONF_DESC(41, REG_I2C_SDA_PU, I2C_SCL_PU_MASK), + PINCTRL_CONF_DESC(42, REG_I2C_SDA_PU, I2C_SDA_PU_MASK), + PINCTRL_CONF_DESC(43, REG_I2C_SDA_PU, AN7583_I2C1_SCL_PU_MASK), + PINCTRL_CONF_DESC(44, REG_I2C_SDA_PU, AN7583_I2C1_SDA_PU_MASK), + PINCTRL_CONF_DESC(45, REG_I2C_SDA_PU, SPI_CLK_PU_MASK), + PINCTRL_CONF_DESC(46, REG_I2C_SDA_PU, SPI_CS0_PU_MASK), + PINCTRL_CONF_DESC(47, REG_I2C_SDA_PU, SPI_MOSI_PU_MASK), + PINCTRL_CONF_DESC(48, REG_I2C_SDA_PU, SPI_MISO_PU_MASK), + PINCTRL_CONF_DESC(49, REG_I2C_SDA_PU, UART1_TXD_PU_MASK), + PINCTRL_CONF_DESC(50, REG_I2C_SDA_PU, UART1_RXD_PU_MASK), + PINCTRL_CONF_DESC(51, REG_I2C_SDA_PU, PCIE0_RESET_PU_MASK), + PINCTRL_CONF_DESC(52, REG_I2C_SDA_PU, PCIE1_RESET_PU_MASK), + PINCTRL_CONF_DESC(53, REG_I2C_SDA_PU, AN7583_MDC_0_PU_MASK), + PINCTRL_CONF_DESC(54, REG_I2C_SDA_PU, AN7583_MDIO_0_PU_MASK), }; static const struct airoha_pinctrl_conf en7581_pinctrl_pulldown_conf[] = { PINCTRL_CONF_DESC(0, REG_I2C_SDA_PD, UART1_TXD_PD_MASK), PINCTRL_CONF_DESC(1, REG_I2C_SDA_PD, UART1_RXD_PD_MASK), - PINCTRL_CONF_DESC(2, REG_I2C_SDA_PD, I2C_SDA_PD_MASK), - PINCTRL_CONF_DESC(3, REG_I2C_SDA_PD, I2C_SCL_PD_MASK), + PINCTRL_CONF_DESC(2, REG_I2C_SDA_PD, I2C_SCL_PD_MASK), + PINCTRL_CONF_DESC(3, REG_I2C_SDA_PD, I2C_SDA_PD_MASK), PINCTRL_CONF_DESC(4, REG_I2C_SDA_PD, SPI_CS0_PD_MASK), PINCTRL_CONF_DESC(5, REG_I2C_SDA_PD, SPI_CLK_PD_MASK), PINCTRL_CONF_DESC(6, REG_I2C_SDA_PD, SPI_MOSI_PD_MASK), @@ -2130,16 +2159,72 @@ static const struct airoha_pinctrl_conf en7581_pinctrl_pulldown_conf[] = { PINCTRL_CONF_DESC(57, REG_GPIO_H_PD, BIT(12)), PINCTRL_CONF_DESC(58, REG_GPIO_H_PD, BIT(13)), PINCTRL_CONF_DESC(59, REG_GPIO_H_PD, BIT(14)), - PINCTRL_CONF_DESC(61, REG_I2C_SDA_PD, PCIE0_RESET_PD_MASK), - PINCTRL_CONF_DESC(62, REG_I2C_SDA_PD, PCIE1_RESET_PD_MASK), - PINCTRL_CONF_DESC(63, REG_I2C_SDA_PD, PCIE2_RESET_PD_MASK), + PINCTRL_CONF_DESC(60, REG_I2C_SDA_PD, PCIE0_RESET_PD_MASK), + PINCTRL_CONF_DESC(61, REG_I2C_SDA_PD, PCIE1_RESET_PD_MASK), + PINCTRL_CONF_DESC(62, REG_I2C_SDA_PD, PCIE2_RESET_PD_MASK), +}; + +static const struct airoha_pinctrl_conf an7583_pinctrl_pulldown_conf[] = { + PINCTRL_CONF_DESC(2, REG_GPIO_L_PD, BIT(0)), + PINCTRL_CONF_DESC(3, REG_GPIO_L_PD, BIT(1)), + PINCTRL_CONF_DESC(4, REG_GPIO_L_PD, BIT(2)), + PINCTRL_CONF_DESC(5, REG_GPIO_L_PD, BIT(3)), + PINCTRL_CONF_DESC(6, REG_GPIO_L_PD, BIT(4)), + PINCTRL_CONF_DESC(7, REG_GPIO_L_PD, BIT(5)), + PINCTRL_CONF_DESC(8, REG_GPIO_L_PD, BIT(6)), + PINCTRL_CONF_DESC(9, REG_GPIO_L_PD, BIT(7)), + PINCTRL_CONF_DESC(10, REG_GPIO_L_PD, BIT(8)), + PINCTRL_CONF_DESC(11, REG_GPIO_L_PD, BIT(9)), + PINCTRL_CONF_DESC(12, REG_GPIO_L_PD, BIT(10)), + PINCTRL_CONF_DESC(13, REG_GPIO_L_PD, BIT(11)), + PINCTRL_CONF_DESC(14, REG_GPIO_L_PD, BIT(12)), + PINCTRL_CONF_DESC(15, REG_GPIO_L_PD, BIT(13)), + PINCTRL_CONF_DESC(16, REG_GPIO_L_PD, BIT(14)), + PINCTRL_CONF_DESC(17, REG_GPIO_L_PD, BIT(15)), + PINCTRL_CONF_DESC(18, REG_GPIO_L_PD, BIT(16)), + PINCTRL_CONF_DESC(19, REG_GPIO_L_PD, BIT(17)), + PINCTRL_CONF_DESC(20, REG_GPIO_L_PD, BIT(18)), + PINCTRL_CONF_DESC(21, REG_GPIO_L_PD, BIT(19)), + PINCTRL_CONF_DESC(22, REG_GPIO_L_PD, BIT(20)), + PINCTRL_CONF_DESC(23, REG_GPIO_L_PD, BIT(21)), + PINCTRL_CONF_DESC(24, REG_GPIO_L_PD, BIT(22)), + PINCTRL_CONF_DESC(25, REG_GPIO_L_PD, BIT(23)), + PINCTRL_CONF_DESC(26, REG_GPIO_L_PD, BIT(24)), + PINCTRL_CONF_DESC(27, REG_GPIO_L_PD, BIT(25)), + PINCTRL_CONF_DESC(28, REG_GPIO_L_PD, BIT(26)), + PINCTRL_CONF_DESC(29, REG_GPIO_L_PD, BIT(27)), + PINCTRL_CONF_DESC(30, REG_GPIO_L_PD, BIT(28)), + PINCTRL_CONF_DESC(31, REG_GPIO_L_PD, BIT(29)), + PINCTRL_CONF_DESC(32, REG_GPIO_L_PD, BIT(30)), + PINCTRL_CONF_DESC(33, REG_GPIO_L_PD, BIT(31)), + PINCTRL_CONF_DESC(34, REG_GPIO_H_PD, BIT(0)), + PINCTRL_CONF_DESC(35, REG_GPIO_H_PD, BIT(1)), + PINCTRL_CONF_DESC(36, REG_GPIO_H_PD, BIT(2)), + PINCTRL_CONF_DESC(37, REG_GPIO_H_PD, BIT(3)), + PINCTRL_CONF_DESC(38, REG_GPIO_H_PD, BIT(4)), + PINCTRL_CONF_DESC(39, REG_GPIO_H_PD, BIT(5)), + PINCTRL_CONF_DESC(40, REG_GPIO_H_PD, BIT(6)), + PINCTRL_CONF_DESC(41, REG_I2C_SDA_PD, I2C_SCL_PD_MASK), + PINCTRL_CONF_DESC(42, REG_I2C_SDA_PD, I2C_SDA_PD_MASK), + PINCTRL_CONF_DESC(43, REG_I2C_SDA_PD, AN7583_I2C1_SCL_PD_MASK), + PINCTRL_CONF_DESC(44, REG_I2C_SDA_PD, AN7583_I2C1_SDA_PD_MASK), + PINCTRL_CONF_DESC(45, REG_I2C_SDA_PD, SPI_CLK_PD_MASK), + PINCTRL_CONF_DESC(46, REG_I2C_SDA_PD, SPI_CS0_PD_MASK), + PINCTRL_CONF_DESC(47, REG_I2C_SDA_PD, SPI_MOSI_PD_MASK), + PINCTRL_CONF_DESC(48, REG_I2C_SDA_PD, SPI_MISO_PD_MASK), + PINCTRL_CONF_DESC(49, REG_I2C_SDA_PD, UART1_TXD_PD_MASK), + PINCTRL_CONF_DESC(50, REG_I2C_SDA_PD, UART1_RXD_PD_MASK), + PINCTRL_CONF_DESC(51, REG_I2C_SDA_PD, PCIE0_RESET_PD_MASK), + PINCTRL_CONF_DESC(52, REG_I2C_SDA_PD, PCIE1_RESET_PD_MASK), + PINCTRL_CONF_DESC(53, REG_I2C_SDA_PD, AN7583_MDC_0_PD_MASK), + PINCTRL_CONF_DESC(54, REG_I2C_SDA_PD, AN7583_MDIO_0_PD_MASK), }; static const struct airoha_pinctrl_conf en7581_pinctrl_drive_e2_conf[] = { PINCTRL_CONF_DESC(0, REG_I2C_SDA_E2, UART1_TXD_E2_MASK), PINCTRL_CONF_DESC(1, REG_I2C_SDA_E2, UART1_RXD_E2_MASK), - PINCTRL_CONF_DESC(2, REG_I2C_SDA_E2, I2C_SDA_E2_MASK), - PINCTRL_CONF_DESC(3, REG_I2C_SDA_E2, I2C_SCL_E2_MASK), + PINCTRL_CONF_DESC(2, REG_I2C_SDA_E2, I2C_SCL_E2_MASK), + PINCTRL_CONF_DESC(3, REG_I2C_SDA_E2, I2C_SDA_E2_MASK), PINCTRL_CONF_DESC(4, REG_I2C_SDA_E2, SPI_CS0_E2_MASK), PINCTRL_CONF_DESC(5, REG_I2C_SDA_E2, SPI_CLK_E2_MASK), PINCTRL_CONF_DESC(6, REG_I2C_SDA_E2, SPI_MOSI_E2_MASK), @@ -2191,16 +2276,72 @@ static const struct airoha_pinctrl_conf en7581_pinctrl_drive_e2_conf[] = { PINCTRL_CONF_DESC(57, REG_GPIO_H_E2, BIT(12)), PINCTRL_CONF_DESC(58, REG_GPIO_H_E2, BIT(13)), PINCTRL_CONF_DESC(59, REG_GPIO_H_E2, BIT(14)), - PINCTRL_CONF_DESC(61, REG_I2C_SDA_E2, PCIE0_RESET_E2_MASK), - PINCTRL_CONF_DESC(62, REG_I2C_SDA_E2, PCIE1_RESET_E2_MASK), - PINCTRL_CONF_DESC(63, REG_I2C_SDA_E2, PCIE2_RESET_E2_MASK), + PINCTRL_CONF_DESC(60, REG_I2C_SDA_E2, PCIE0_RESET_E2_MASK), + PINCTRL_CONF_DESC(61, REG_I2C_SDA_E2, PCIE1_RESET_E2_MASK), + PINCTRL_CONF_DESC(62, REG_I2C_SDA_E2, PCIE2_RESET_E2_MASK), +}; + +static const struct airoha_pinctrl_conf an7583_pinctrl_drive_e2_conf[] = { + PINCTRL_CONF_DESC(2, REG_GPIO_L_E2, BIT(0)), + PINCTRL_CONF_DESC(3, REG_GPIO_L_E2, BIT(1)), + PINCTRL_CONF_DESC(4, REG_GPIO_L_E2, BIT(2)), + PINCTRL_CONF_DESC(5, REG_GPIO_L_E2, BIT(3)), + PINCTRL_CONF_DESC(6, REG_GPIO_L_E2, BIT(4)), + PINCTRL_CONF_DESC(7, REG_GPIO_L_E2, BIT(5)), + PINCTRL_CONF_DESC(8, REG_GPIO_L_E2, BIT(6)), + PINCTRL_CONF_DESC(9, REG_GPIO_L_E2, BIT(7)), + PINCTRL_CONF_DESC(10, REG_GPIO_L_E2, BIT(8)), + PINCTRL_CONF_DESC(11, REG_GPIO_L_E2, BIT(9)), + PINCTRL_CONF_DESC(12, REG_GPIO_L_E2, BIT(10)), + PINCTRL_CONF_DESC(13, REG_GPIO_L_E2, BIT(11)), + PINCTRL_CONF_DESC(14, REG_GPIO_L_E2, BIT(12)), + PINCTRL_CONF_DESC(15, REG_GPIO_L_E2, BIT(13)), + PINCTRL_CONF_DESC(16, REG_GPIO_L_E2, BIT(14)), + PINCTRL_CONF_DESC(17, REG_GPIO_L_E2, BIT(15)), + PINCTRL_CONF_DESC(18, REG_GPIO_L_E2, BIT(16)), + PINCTRL_CONF_DESC(19, REG_GPIO_L_E2, BIT(17)), + PINCTRL_CONF_DESC(20, REG_GPIO_L_E2, BIT(18)), + PINCTRL_CONF_DESC(21, REG_GPIO_L_E2, BIT(19)), + PINCTRL_CONF_DESC(22, REG_GPIO_L_E2, BIT(20)), + PINCTRL_CONF_DESC(23, REG_GPIO_L_E2, BIT(21)), + PINCTRL_CONF_DESC(24, REG_GPIO_L_E2, BIT(22)), + PINCTRL_CONF_DESC(25, REG_GPIO_L_E2, BIT(23)), + PINCTRL_CONF_DESC(26, REG_GPIO_L_E2, BIT(24)), + PINCTRL_CONF_DESC(27, REG_GPIO_L_E2, BIT(25)), + PINCTRL_CONF_DESC(28, REG_GPIO_L_E2, BIT(26)), + PINCTRL_CONF_DESC(29, REG_GPIO_L_E2, BIT(27)), + PINCTRL_CONF_DESC(30, REG_GPIO_L_E2, BIT(28)), + PINCTRL_CONF_DESC(31, REG_GPIO_L_E2, BIT(29)), + PINCTRL_CONF_DESC(32, REG_GPIO_L_E2, BIT(30)), + PINCTRL_CONF_DESC(33, REG_GPIO_L_E2, BIT(31)), + PINCTRL_CONF_DESC(34, REG_GPIO_H_E2, BIT(0)), + PINCTRL_CONF_DESC(35, REG_GPIO_H_E2, BIT(1)), + PINCTRL_CONF_DESC(36, REG_GPIO_H_E2, BIT(2)), + PINCTRL_CONF_DESC(37, REG_GPIO_H_E2, BIT(3)), + PINCTRL_CONF_DESC(38, REG_GPIO_H_E2, BIT(4)), + PINCTRL_CONF_DESC(39, REG_GPIO_H_E2, BIT(5)), + PINCTRL_CONF_DESC(40, REG_GPIO_H_E2, BIT(6)), + PINCTRL_CONF_DESC(41, REG_I2C_SDA_E2, I2C_SCL_E2_MASK), + PINCTRL_CONF_DESC(42, REG_I2C_SDA_E2, I2C_SDA_E2_MASK), + PINCTRL_CONF_DESC(43, REG_I2C_SDA_E2, AN7583_I2C1_SCL_E2_MASK), + PINCTRL_CONF_DESC(44, REG_I2C_SDA_E2, AN7583_I2C1_SDA_E2_MASK), + PINCTRL_CONF_DESC(45, REG_I2C_SDA_E2, SPI_CLK_E2_MASK), + PINCTRL_CONF_DESC(46, REG_I2C_SDA_E2, SPI_CS0_E2_MASK), + PINCTRL_CONF_DESC(47, REG_I2C_SDA_E2, SPI_MOSI_E2_MASK), + PINCTRL_CONF_DESC(48, REG_I2C_SDA_E2, SPI_MISO_E2_MASK), + PINCTRL_CONF_DESC(49, REG_I2C_SDA_E2, UART1_TXD_E2_MASK), + PINCTRL_CONF_DESC(50, REG_I2C_SDA_E2, UART1_RXD_E2_MASK), + PINCTRL_CONF_DESC(51, REG_I2C_SDA_E2, PCIE0_RESET_E2_MASK), + PINCTRL_CONF_DESC(52, REG_I2C_SDA_E2, PCIE1_RESET_E2_MASK), + PINCTRL_CONF_DESC(53, REG_I2C_SDA_E2, AN7583_MDC_0_E2_MASK), + PINCTRL_CONF_DESC(54, REG_I2C_SDA_E2, AN7583_MDIO_0_E2_MASK), }; static const struct airoha_pinctrl_conf en7581_pinctrl_drive_e4_conf[] = { PINCTRL_CONF_DESC(0, REG_I2C_SDA_E4, UART1_TXD_E4_MASK), PINCTRL_CONF_DESC(1, REG_I2C_SDA_E4, UART1_RXD_E4_MASK), - PINCTRL_CONF_DESC(2, REG_I2C_SDA_E4, I2C_SDA_E4_MASK), - PINCTRL_CONF_DESC(3, REG_I2C_SDA_E4, I2C_SCL_E4_MASK), + PINCTRL_CONF_DESC(2, REG_I2C_SDA_E4, I2C_SCL_E4_MASK), + PINCTRL_CONF_DESC(3, REG_I2C_SDA_E4, I2C_SDA_E4_MASK), PINCTRL_CONF_DESC(4, REG_I2C_SDA_E4, SPI_CS0_E4_MASK), PINCTRL_CONF_DESC(5, REG_I2C_SDA_E4, SPI_CLK_E4_MASK), PINCTRL_CONF_DESC(6, REG_I2C_SDA_E4, SPI_MOSI_E4_MASK), @@ -2252,15 +2393,76 @@ static const struct airoha_pinctrl_conf en7581_pinctrl_drive_e4_conf[] = { PINCTRL_CONF_DESC(57, REG_GPIO_H_E4, BIT(12)), PINCTRL_CONF_DESC(58, REG_GPIO_H_E4, BIT(13)), PINCTRL_CONF_DESC(59, REG_GPIO_H_E4, BIT(14)), - PINCTRL_CONF_DESC(61, REG_I2C_SDA_E4, PCIE0_RESET_E4_MASK), - PINCTRL_CONF_DESC(62, REG_I2C_SDA_E4, PCIE1_RESET_E4_MASK), - PINCTRL_CONF_DESC(63, REG_I2C_SDA_E4, PCIE2_RESET_E4_MASK), + PINCTRL_CONF_DESC(60, REG_I2C_SDA_E4, PCIE0_RESET_E4_MASK), + PINCTRL_CONF_DESC(61, REG_I2C_SDA_E4, PCIE1_RESET_E4_MASK), + PINCTRL_CONF_DESC(62, REG_I2C_SDA_E4, PCIE2_RESET_E4_MASK), +}; + +static const struct airoha_pinctrl_conf an7583_pinctrl_drive_e4_conf[] = { + PINCTRL_CONF_DESC(2, REG_GPIO_L_E4, BIT(0)), + PINCTRL_CONF_DESC(3, REG_GPIO_L_E4, BIT(1)), + PINCTRL_CONF_DESC(4, REG_GPIO_L_E4, BIT(2)), + PINCTRL_CONF_DESC(5, REG_GPIO_L_E4, BIT(3)), + PINCTRL_CONF_DESC(6, REG_GPIO_L_E4, BIT(4)), + PINCTRL_CONF_DESC(7, REG_GPIO_L_E4, BIT(5)), + PINCTRL_CONF_DESC(8, REG_GPIO_L_E4, BIT(6)), + PINCTRL_CONF_DESC(9, REG_GPIO_L_E4, BIT(7)), + PINCTRL_CONF_DESC(10, REG_GPIO_L_E4, BIT(8)), + PINCTRL_CONF_DESC(11, REG_GPIO_L_E4, BIT(9)), + PINCTRL_CONF_DESC(12, REG_GPIO_L_E4, BIT(10)), + PINCTRL_CONF_DESC(13, REG_GPIO_L_E4, BIT(11)), + PINCTRL_CONF_DESC(14, REG_GPIO_L_E4, BIT(12)), + PINCTRL_CONF_DESC(15, REG_GPIO_L_E4, BIT(13)), + PINCTRL_CONF_DESC(16, REG_GPIO_L_E4, BIT(14)), + PINCTRL_CONF_DESC(17, REG_GPIO_L_E4, BIT(15)), + PINCTRL_CONF_DESC(18, REG_GPIO_L_E4, BIT(16)), + PINCTRL_CONF_DESC(19, REG_GPIO_L_E4, BIT(17)), + PINCTRL_CONF_DESC(20, REG_GPIO_L_E4, BIT(18)), + PINCTRL_CONF_DESC(21, REG_GPIO_L_E4, BIT(19)), + PINCTRL_CONF_DESC(22, REG_GPIO_L_E4, BIT(20)), + PINCTRL_CONF_DESC(23, REG_GPIO_L_E4, BIT(21)), + PINCTRL_CONF_DESC(24, REG_GPIO_L_E4, BIT(22)), + PINCTRL_CONF_DESC(25, REG_GPIO_L_E4, BIT(23)), + PINCTRL_CONF_DESC(26, REG_GPIO_L_E4, BIT(24)), + PINCTRL_CONF_DESC(27, REG_GPIO_L_E4, BIT(25)), + PINCTRL_CONF_DESC(28, REG_GPIO_L_E4, BIT(26)), + PINCTRL_CONF_DESC(29, REG_GPIO_L_E4, BIT(27)), + PINCTRL_CONF_DESC(30, REG_GPIO_L_E4, BIT(28)), + PINCTRL_CONF_DESC(31, REG_GPIO_L_E4, BIT(29)), + PINCTRL_CONF_DESC(32, REG_GPIO_L_E4, BIT(30)), + PINCTRL_CONF_DESC(33, REG_GPIO_L_E4, BIT(31)), + PINCTRL_CONF_DESC(34, REG_GPIO_H_E4, BIT(0)), + PINCTRL_CONF_DESC(35, REG_GPIO_H_E4, BIT(1)), + PINCTRL_CONF_DESC(36, REG_GPIO_H_E4, BIT(2)), + PINCTRL_CONF_DESC(37, REG_GPIO_H_E4, BIT(3)), + PINCTRL_CONF_DESC(38, REG_GPIO_H_E4, BIT(4)), + PINCTRL_CONF_DESC(39, REG_GPIO_H_E4, BIT(5)), + PINCTRL_CONF_DESC(40, REG_GPIO_H_E4, BIT(6)), + PINCTRL_CONF_DESC(41, REG_I2C_SDA_E4, I2C_SCL_E4_MASK), + PINCTRL_CONF_DESC(42, REG_I2C_SDA_E4, I2C_SDA_E4_MASK), + PINCTRL_CONF_DESC(43, REG_I2C_SDA_E4, AN7583_I2C1_SCL_E4_MASK), + PINCTRL_CONF_DESC(44, REG_I2C_SDA_E4, AN7583_I2C1_SDA_E4_MASK), + PINCTRL_CONF_DESC(45, REG_I2C_SDA_E4, SPI_CLK_E4_MASK), + PINCTRL_CONF_DESC(46, REG_I2C_SDA_E4, SPI_CS0_E4_MASK), + PINCTRL_CONF_DESC(47, REG_I2C_SDA_E4, SPI_MOSI_E4_MASK), + PINCTRL_CONF_DESC(48, REG_I2C_SDA_E4, SPI_MISO_E4_MASK), + PINCTRL_CONF_DESC(49, REG_I2C_SDA_E4, UART1_TXD_E4_MASK), + PINCTRL_CONF_DESC(50, REG_I2C_SDA_E4, UART1_RXD_E4_MASK), + PINCTRL_CONF_DESC(51, REG_I2C_SDA_E4, PCIE0_RESET_E4_MASK), + PINCTRL_CONF_DESC(52, REG_I2C_SDA_E4, PCIE1_RESET_E4_MASK), + PINCTRL_CONF_DESC(53, REG_I2C_SDA_E4, AN7583_MDC_0_E4_MASK), + PINCTRL_CONF_DESC(54, REG_I2C_SDA_E4, AN7583_MDIO_0_E4_MASK), }; static const struct airoha_pinctrl_conf en7581_pinctrl_pcie_rst_od_conf[] = { - PINCTRL_CONF_DESC(61, REG_PCIE_RESET_OD, PCIE0_RESET_OD_MASK), - PINCTRL_CONF_DESC(62, REG_PCIE_RESET_OD, PCIE1_RESET_OD_MASK), - PINCTRL_CONF_DESC(63, REG_PCIE_RESET_OD, PCIE2_RESET_OD_MASK), + PINCTRL_CONF_DESC(60, REG_PCIE_RESET_OD, PCIE0_RESET_OD_MASK), + PINCTRL_CONF_DESC(61, REG_PCIE_RESET_OD, PCIE1_RESET_OD_MASK), + PINCTRL_CONF_DESC(62, REG_PCIE_RESET_OD, PCIE2_RESET_OD_MASK), +}; + +static const struct airoha_pinctrl_conf an7583_pinctrl_pcie_rst_od_conf[] = { + PINCTRL_CONF_DESC(51, REG_PCIE_RESET_OD, PCIE0_RESET_OD_MASK), + PINCTRL_CONF_DESC(52, REG_PCIE_RESET_OD, PCIE1_RESET_OD_MASK), }; static int airoha_convert_pin_to_reg_offset(struct pinctrl_dev *pctrl_dev, @@ -2302,6 +2504,22 @@ static int airoha_gpio_get(struct gpio_chip *chip, unsigned int gpio) return err ? err : !!(val & BIT(pin)); } +static int airoha_gpio_get_direction(struct gpio_chip *chip, unsigned int gpio) +{ + struct airoha_pinctrl *pinctrl = gpiochip_get_data(chip); + u32 val, mask; + u8 index; + int err; + + index = gpio / AIROHA_REG_GPIOCTRL_NUM_PIN; + err = regmap_read(pinctrl->regmap, pinctrl->gpiochip.dir[index], &val); + if (err) + return err; + + mask = BIT(2 * (gpio % AIROHA_REG_GPIOCTRL_NUM_PIN)); + return val & mask ? GPIO_LINE_DIRECTION_OUT : GPIO_LINE_DIRECTION_IN; +} + static int airoha_gpio_direction_output(struct gpio_chip *chip, unsigned int gpio, int value) { @@ -2317,18 +2535,18 @@ static int airoha_gpio_direction_output(struct gpio_chip *chip, /* irq callbacks */ static void airoha_irq_unmask(struct irq_data *data) { + struct gpio_chip *gc = irq_data_get_irq_chip_data(data); + struct airoha_pinctrl *pinctrl = gpiochip_get_data(gc); + struct airoha_pinctrl_gpiochip *gpiochip = &pinctrl->gpiochip; u8 offset = data->hwirq % AIROHA_REG_GPIOCTRL_NUM_PIN; u8 index = data->hwirq / AIROHA_REG_GPIOCTRL_NUM_PIN; u32 mask = GENMASK(2 * offset + 1, 2 * offset); - struct airoha_pinctrl_gpiochip *gpiochip; - struct airoha_pinctrl *pinctrl; u32 val = BIT(2 * offset); - gpiochip = irq_data_get_irq_chip_data(data); if (WARN_ON_ONCE(data->hwirq >= ARRAY_SIZE(gpiochip->irq_type))) return; - pinctrl = container_of(gpiochip, struct airoha_pinctrl, gpiochip); + gpiochip_enable_irq(gc, irqd_to_hwirq(data)); switch (gpiochip->irq_type[data->hwirq]) { case IRQ_TYPE_LEVEL_LOW: val = val << 1; @@ -2354,34 +2572,64 @@ static void airoha_irq_unmask(struct irq_data *data) static void airoha_irq_mask(struct irq_data *data) { + struct gpio_chip *gc = irq_data_get_irq_chip_data(data); + struct airoha_pinctrl *pinctrl = gpiochip_get_data(gc); + struct airoha_pinctrl_gpiochip *gpiochip = &pinctrl->gpiochip; u8 offset = data->hwirq % AIROHA_REG_GPIOCTRL_NUM_PIN; u8 index = data->hwirq / AIROHA_REG_GPIOCTRL_NUM_PIN; u32 mask = GENMASK(2 * offset + 1, 2 * offset); - struct airoha_pinctrl_gpiochip *gpiochip; - struct airoha_pinctrl *pinctrl; - gpiochip = irq_data_get_irq_chip_data(data); - pinctrl = container_of(gpiochip, struct airoha_pinctrl, gpiochip); + if (data->hwirq >= ARRAY_SIZE(gpiochip->irq_type)) + return; regmap_clear_bits(pinctrl->regmap, gpiochip->level[index], mask); regmap_clear_bits(pinctrl->regmap, gpiochip->edge[index], mask); + gpiochip_disable_irq(gc, irqd_to_hwirq(data)); +} + +static void airoha_irq_ack(struct irq_data *data) +{ + struct gpio_chip *gc = irq_data_get_irq_chip_data(data); + struct airoha_pinctrl *pinctrl = gpiochip_get_data(gc); + struct airoha_pinctrl_gpiochip *gpiochip = &pinctrl->gpiochip; + u8 offset = data->hwirq % AIROHA_PIN_BANK_SIZE; + u8 index = data->hwirq / AIROHA_PIN_BANK_SIZE; + + if (data->hwirq >= ARRAY_SIZE(gpiochip->irq_type)) + return; + + regmap_write(pinctrl->regmap, gpiochip->status[index], BIT(offset)); } static int airoha_irq_type(struct irq_data *data, unsigned int type) { - struct airoha_pinctrl_gpiochip *gpiochip; + struct gpio_chip *gc = irq_data_get_irq_chip_data(data); + struct airoha_pinctrl *pinctrl = gpiochip_get_data(gc); + struct airoha_pinctrl_gpiochip *gpiochip = &pinctrl->gpiochip; - gpiochip = irq_data_get_irq_chip_data(data); if (data->hwirq >= ARRAY_SIZE(gpiochip->irq_type)) return -EINVAL; + if (type == IRQ_TYPE_NONE) { + gpiochip->irq_type[data->hwirq] = IRQ_TYPE_NONE; + irq_set_handler_locked(data, handle_bad_irq); + + return 0; + } + if (type == IRQ_TYPE_PROBE) { if (gpiochip->irq_type[data->hwirq]) return 0; type = IRQ_TYPE_EDGE_RISING | IRQ_TYPE_EDGE_FALLING; } + gpiochip->irq_type[data->hwirq] = type & IRQ_TYPE_SENSE_MASK; + if (type & IRQ_TYPE_EDGE_BOTH) + irq_set_handler_locked(data, handle_edge_irq); + else + irq_set_handler_locked(data, handle_level_irq); + return 0; } @@ -2406,8 +2654,7 @@ static irqreturn_t airoha_irq_handler(int irq, void *data) for_each_set_bit(irq, &status, AIROHA_PIN_BANK_SIZE) { u32 offset = irq + i * AIROHA_PIN_BANK_SIZE; - generic_handle_irq(irq_find_mapping(girq->domain, - offset)); + generic_handle_domain_irq(girq->domain, offset); regmap_write(pinctrl->regmap, pinctrl->gpiochip.status[i], BIT(irq)); } @@ -2421,13 +2668,14 @@ static const struct irq_chip airoha_gpio_irq_chip = { .name = "airoha-gpio-irq", .irq_unmask = airoha_irq_unmask, .irq_mask = airoha_irq_mask, - .irq_mask_ack = airoha_irq_mask, + .irq_ack = airoha_irq_ack, .irq_set_type = airoha_irq_type, .flags = IRQCHIP_SET_TYPE_MASKED | IRQCHIP_IMMUTABLE, + GPIOCHIP_IRQ_RESOURCE_HELPERS, }; static int airoha_pinctrl_add_gpiochip(struct airoha_pinctrl *pinctrl, - struct platform_device *pdev) + struct platform_device *pdev) { struct airoha_pinctrl_gpiochip *chip = &pinctrl->gpiochip; struct gpio_chip *gc = &chip->chip; @@ -2435,26 +2683,20 @@ static int airoha_pinctrl_add_gpiochip(struct airoha_pinctrl *pinctrl, struct device *dev = &pdev->dev; int irq, err; - chip->data = gpio_data_regs; - chip->dir = gpio_dir_regs; - chip->out = gpio_out_regs; - chip->status = irq_status_regs; - chip->level = irq_level_regs; - chip->edge = irq_edge_regs; - gc->parent = dev; gc->label = dev_name(dev); gc->request = gpiochip_generic_request; gc->free = gpiochip_generic_free; gc->direction_input = pinctrl_gpio_direction_input; gc->direction_output = airoha_gpio_direction_output; + gc->get_direction = airoha_gpio_get_direction; gc->set = airoha_gpio_set; gc->get = airoha_gpio_get; gc->base = -1; gc->ngpio = AIROHA_NUM_PINS; girq->default_type = IRQ_TYPE_NONE; - girq->handler = handle_simple_irq; + girq->handler = handle_bad_irq; gpio_irq_chip_set_chip(girq, &airoha_gpio_irq_chip); irq = platform_get_irq(pdev, 0); @@ -2462,7 +2704,7 @@ static int airoha_pinctrl_add_gpiochip(struct airoha_pinctrl *pinctrl, return irq; err = devm_request_irq(dev, irq, airoha_irq_handler, IRQF_SHARED, - dev_name(dev), pinctrl); + dev_name(dev), pinctrl); if (err) { dev_err(dev, "error requesting irq %d: %d\n", irq, err); return err; @@ -2526,8 +2768,8 @@ static int airoha_pinmux_set_mux(struct pinctrl_dev *pctrl_dev, } static int airoha_pinmux_set_direction(struct pinctrl_dev *pctrl_dev, - struct pinctrl_gpio_range *range, - unsigned int p, bool input) + struct pinctrl_gpio_range *range, + unsigned int p, bool input) { struct airoha_pinctrl *pinctrl = pinctrl_dev_get_drvdata(pctrl_dev); u32 mask, index; @@ -2617,7 +2859,7 @@ static int airoha_pinctrl_set_conf(struct airoha_pinctrl *pinctrl, if (regmap_update_bits(pinctrl->chip_scu, reg->offset, reg->mask, - val << __ffs(reg->mask))) + val << __ffs(reg->mask))) return -EINVAL; return 0; @@ -2657,21 +2899,18 @@ static int airoha_pinctrl_set_conf(struct airoha_pinctrl *pinctrl, static int airoha_pinconf_get_direction(struct pinctrl_dev *pctrl_dev, u32 p) { struct airoha_pinctrl *pinctrl = pinctrl_dev_get_drvdata(pctrl_dev); - u32 val, mask; - int err, pin; - u8 index; + int ret, pin; pin = airoha_convert_pin_to_reg_offset(pctrl_dev, NULL, p); if (pin < 0) return pin; - index = pin / AIROHA_REG_GPIOCTRL_NUM_PIN; - err = regmap_read(pinctrl->regmap, pinctrl->gpiochip.dir[index], &val); - if (err) - return err; + ret = airoha_gpio_get_direction(&pinctrl->gpiochip.chip, pin); + if (ret < 0) + return ret; - mask = BIT(2 * (pin % AIROHA_REG_GPIOCTRL_NUM_PIN)); - return val & mask ? PIN_CONFIG_OUTPUT_ENABLE : PIN_CONFIG_INPUT_ENABLE; + return ret == GPIO_LINE_DIRECTION_OUT ? + PIN_CONFIG_OUTPUT_ENABLE : PIN_CONFIG_INPUT_ENABLE; } static int airoha_pinconf_get(struct pinctrl_dev *pctrl_dev, @@ -2834,8 +3073,8 @@ static int airoha_pinconf_group_get(struct pinctrl_dev *pctrl_dev, for (i = 0; i < pinctrl->grps[group].npins; i++) { if (airoha_pinconf_get(pctrl_dev, - pinctrl->grps[group].pins[i], - config)) + pinctrl->grps[group].pins[i], + config)) return -ENOTSUPP; if (i && cur_config != *config) @@ -2917,6 +3156,17 @@ static int airoha_pinctrl_probe(struct platform_device *pdev) pinctrl->desc.pins = data->pins, pinctrl->desc.npins = data->num_pins, + /* + * some pinctrl operations (ex: get_direction) might use gpio registers + * before gpio chip abstraction will be completely initialized. + */ + pinctrl->gpiochip.data = gpio_data_regs; + pinctrl->gpiochip.dir = gpio_dir_regs; + pinctrl->gpiochip.out = gpio_out_regs; + pinctrl->gpiochip.status = irq_status_regs; + pinctrl->gpiochip.level = irq_level_regs; + pinctrl->gpiochip.edge = irq_edge_regs; + err = devm_pinctrl_register_and_init(dev, &pinctrl->desc, pinctrl, &pinctrl->ctrl); if (err) @@ -2994,8 +3244,40 @@ static const struct airoha_pinctrl_match_data en7581_pinctrl_match_data = { }, }; +static const struct airoha_pinctrl_match_data an7583_pinctrl_match_data = { + .pins = an7583_pinctrl_pins, + .num_pins = ARRAY_SIZE(an7583_pinctrl_pins), + .grps = an7583_pinctrl_groups, + .num_grps = ARRAY_SIZE(an7583_pinctrl_groups), + .funcs = an7583_pinctrl_funcs, + .num_funcs = ARRAY_SIZE(an7583_pinctrl_funcs), + .confs_info = { + [AIROHA_PINCTRL_CONFS_PULLUP] = { + .confs = an7583_pinctrl_pullup_conf, + .num_confs = ARRAY_SIZE(an7583_pinctrl_pullup_conf), + }, + [AIROHA_PINCTRL_CONFS_PULLDOWN] = { + .confs = an7583_pinctrl_pulldown_conf, + .num_confs = ARRAY_SIZE(an7583_pinctrl_pulldown_conf), + }, + [AIROHA_PINCTRL_CONFS_DRIVE_E2] = { + .confs = an7583_pinctrl_drive_e2_conf, + .num_confs = ARRAY_SIZE(an7583_pinctrl_drive_e2_conf), + }, + [AIROHA_PINCTRL_CONFS_DRIVE_E4] = { + .confs = an7583_pinctrl_drive_e4_conf, + .num_confs = ARRAY_SIZE(an7583_pinctrl_drive_e4_conf), + }, + [AIROHA_PINCTRL_CONFS_PCIE_RST_OD] = { + .confs = an7583_pinctrl_pcie_rst_od_conf, + .num_confs = ARRAY_SIZE(an7583_pinctrl_pcie_rst_od_conf), + }, + }, +}; + static const struct of_device_id airoha_pinctrl_of_match[] = { { .compatible = "airoha,en7581-pinctrl", .data = &en7581_pinctrl_match_data }, + { .compatible = "airoha,an7583-pinctrl", .data = &an7583_pinctrl_match_data }, { /* sentinel */ } }; MODULE_DEVICE_TABLE(of, airoha_pinctrl_of_match); diff --git a/drivers/pinctrl/pinctrl-eic7700.c b/drivers/pinctrl/pinctrl-eic7700.c index ffcd0ec5c2dc..1e72931feca5 100644 --- a/drivers/pinctrl/pinctrl-eic7700.c +++ b/drivers/pinctrl/pinctrl-eic7700.c @@ -423,6 +423,9 @@ static int eic7700_pin_config_set(struct pinctrl_dev *pctldev, unsigned int pin, else value &= ~EIC7700_ST; break; + /* All pins are persistent over suspend */ + case PIN_CONFIG_PERSIST_STATE: + return 0; default: return -EOPNOTSUPP; } diff --git a/drivers/pinctrl/pinctrl-rockchip.c b/drivers/pinctrl/pinctrl-rockchip.c index f1cba3d2367a..834229831de9 100644 --- a/drivers/pinctrl/pinctrl-rockchip.c +++ b/drivers/pinctrl/pinctrl-rockchip.c @@ -3675,6 +3675,16 @@ static struct rockchip_pin_ctrl *rockchip_pinctrl_get_soc_data( pmu_offs = ctrl->pmu_mux_offset; drv_pmu_offs = ctrl->pmu_drv_offset; drv_grf_offs = ctrl->grf_drv_offset; + + /* + * This function mutates the static per-SoC data. Most of it is + * idempotent: recalculated iomux and drv offsets anchor at the + * values calculated by a previous run. The pin count is not, so + * reset it here; otherwise it accumulates when the probe runs + * again after a probe deferral, shifting every bank's pin_base. + */ + ctrl->nr_pins = 0; + bank = ctrl->pin_banks; for (i = 0; i < ctrl->nr_banks; ++i, ++bank) { int bank_pins = 0; diff --git a/drivers/pinctrl/spacemit/pinctrl-k1.c b/drivers/pinctrl/spacemit/pinctrl-k1.c index 8d797bf24b77..813f86865dcd 100644 --- a/drivers/pinctrl/spacemit/pinctrl-k1.c +++ b/drivers/pinctrl/spacemit/pinctrl-k1.c @@ -475,13 +475,14 @@ static int spacemit_pinconf_get(struct pinctrl_dev *pctldev, unsigned int pin, unsigned long *config) { struct spacemit_pinctrl *pctrl = pinctrl_dev_get_drvdata(pctldev); + const struct spacemit_pin *spin = spacemit_get_pin(pctrl, pin); int param = pinconf_to_config_param(*config); u32 value, arg = 0; - if (!pin) + if (!spin) return -EINVAL; - value = readl(spacemit_pin_to_reg(pctrl, pin)); + value = readl(spacemit_pin_to_reg(pctrl, spin->pin)); switch (param) { case PIN_CONFIG_SLEW_RATE: @@ -654,6 +655,11 @@ static int spacemit_pinconf_group_set(struct pinctrl_dev *pctldev, if (spacemit_pinconf_generate_config(spin, configs, num_configs, &value)) return -EINVAL; + for (i = 0; i < group->grp.npins; i++) { + if (!spacemit_get_pin(pctrl, group->grp.pins[i])) + return -EINVAL; + } + for (i = 0; i < group->grp.npins; i++) spacemit_pin_set_config(pctrl, group->grp.pins[i], value); diff --git a/drivers/platform/chrome/cros_ec_debugfs.c b/drivers/platform/chrome/cros_ec_debugfs.c index d10f9561990c..e482b23b9f63 100644 --- a/drivers/platform/chrome/cros_ec_debugfs.c +++ b/drivers/platform/chrome/cros_ec_debugfs.c @@ -513,7 +513,7 @@ static int cros_ec_debugfs_probe(struct platform_device *pd) ret = blocking_notifier_chain_register(&ec->ec_dev->panic_notifier, &debug_info->notifier_panic); if (ret) - goto remove_debugfs; + goto cleanup_console_log; ec->debug_info = debug_info; @@ -521,6 +521,8 @@ static int cros_ec_debugfs_probe(struct platform_device *pd) return 0; +cleanup_console_log: + cros_ec_cleanup_console_log(debug_info); remove_debugfs: debugfs_remove_recursive(debug_info->dir); return ret; @@ -530,6 +532,8 @@ static void cros_ec_debugfs_remove(struct platform_device *pd) { struct cros_ec_dev *ec = dev_get_drvdata(pd->dev.parent); + blocking_notifier_chain_unregister(&ec->ec_dev->panic_notifier, + &ec->debug_info->notifier_panic); debugfs_remove_recursive(ec->debug_info->dir); cros_ec_cleanup_console_log(ec->debug_info); } diff --git a/drivers/platform/chrome/cros_ec_sensorhub_ring.c b/drivers/platform/chrome/cros_ec_sensorhub_ring.c index 302d037b90a1..b5970e8620bf 100644 --- a/drivers/platform/chrome/cros_ec_sensorhub_ring.c +++ b/drivers/platform/chrome/cros_ec_sensorhub_ring.c @@ -825,8 +825,15 @@ static void cros_ec_sensorhub_ring_handler(struct cros_ec_sensorhub *sensorhub) sensorhub->msg->outsize = 1; sensorhub->msg->insize = fifo_info_length; - if (cros_ec_cmd_xfer_status(ec->ec_dev, sensorhub->msg) < 0) + ret = cros_ec_cmd_xfer_status(ec->ec_dev, sensorhub->msg); + if (ret < 0) + goto error; + if (ret != fifo_info_length) { + dev_warn_ratelimited(sensorhub->dev, + "Mismatch read length: size %d - expected %d\n", + ret, fifo_info_length); goto error; + } memcpy(fifo_info, &sensorhub->resp->fifo_info, fifo_info_length); diff --git a/drivers/platform/chrome/cros_ec_typec.c b/drivers/platform/chrome/cros_ec_typec.c index b712bcff6fb2..03ca776a837f 100644 --- a/drivers/platform/chrome/cros_ec_typec.c +++ b/drivers/platform/chrome/cros_ec_typec.c @@ -1118,6 +1118,12 @@ static void cros_typec_register_partner_pdos(struct cros_typec_data *typec, if (!resp->source_cap_count && !resp->sink_cap_count) return; + if (resp->source_cap_count > PDO_MAX_OBJECTS || + resp->sink_cap_count > PDO_MAX_OBJECTS) { + dev_warn(typec->dev, "Invalid PDO count from EC, port: %d\n", port_num); + return; + } + port->partner_pd = typec_partner_usb_power_delivery_register(port->partner, &desc); if (IS_ERR(port->partner_pd)) { dev_warn(typec->dev, "Failed to register partner PD device, port: %d\n", port_num); diff --git a/drivers/platform/mellanox/mlxbf-bootctl.c b/drivers/platform/mellanox/mlxbf-bootctl.c index f67c7f56ab2b..9ddc7fa1a973 100644 --- a/drivers/platform/mellanox/mlxbf-bootctl.c +++ b/drivers/platform/mellanox/mlxbf-bootctl.c @@ -10,6 +10,7 @@ #include #include +#include #include #include #include diff --git a/drivers/platform/mellanox/mlxbf-pmc.c b/drivers/platform/mellanox/mlxbf-pmc.c index 5ec1ad471696..2ad9e2b0493c 100644 --- a/drivers/platform/mellanox/mlxbf-pmc.c +++ b/drivers/platform/mellanox/mlxbf-pmc.c @@ -2262,13 +2262,19 @@ static int mlxbf_pmc_map_counters(struct device *dev) static int mlxbf_pmc_probe(struct platform_device *pdev) { - struct acpi_device *acpi_dev = ACPI_COMPANION(&pdev->dev); - const char *hid = acpi_device_hid(acpi_dev); struct device *dev = &pdev->dev; + struct acpi_device *acpi_dev; struct arm_smccc_res res; + const char *hid; guid_t guid; int ret; + acpi_dev = ACPI_COMPANION(&pdev->dev); + if (!acpi_dev) + return -ENODEV; + + hid = acpi_device_hid(acpi_dev); + /* Ensure we have the UUID we expect for this service. */ arm_smccc_smc(MLXBF_PMC_SIP_SVC_UID, 0, 0, 0, 0, 0, 0, 0, &res); guid_parse(mlxbf_pmc_svc_uuid_str, &guid); diff --git a/drivers/platform/surface/surface_acpi_notify.c b/drivers/platform/surface/surface_acpi_notify.c index 3b30cfe3466b..3bad4e689f5c 100644 --- a/drivers/platform/surface/surface_acpi_notify.c +++ b/drivers/platform/surface/surface_acpi_notify.c @@ -777,12 +777,16 @@ static int san_consumer_links_setup(struct platform_device *pdev) static int san_probe(struct platform_device *pdev) { - struct acpi_device *san = ACPI_COMPANION(&pdev->dev); struct ssam_controller *ctrl; + struct acpi_device *san; struct san_data *data; acpi_status astatus; int status; + san = ACPI_COMPANION(&pdev->dev); + if (!san) + return -ENODEV; + ctrl = ssam_client_bind(&pdev->dev); if (IS_ERR(ctrl)) return PTR_ERR(ctrl) == -ENODEV ? -EPROBE_DEFER : PTR_ERR(ctrl); diff --git a/drivers/platform/x86/acer-wmi.c b/drivers/platform/x86/acer-wmi.c index d848afc91f87..55b198031354 100644 --- a/drivers/platform/x86/acer-wmi.c +++ b/drivers/platform/x86/acer-wmi.c @@ -1530,7 +1530,9 @@ static int WMI_gaming_execute_u32_u64(u32 method_id, u32 in, u64 *out) return -EIO; obj = result.pointer; - if (obj && out) { + if (!obj && out) { + ret = -ENOMSG; + } else if (obj && out) { switch (obj->type) { case ACPI_TYPE_INTEGER: *out = obj->integer.value; diff --git a/drivers/platform/x86/amd/hsmp/hwmon.c b/drivers/platform/x86/amd/hsmp/hwmon.c index 0cc9a742497f..c8314eee06f4 100644 --- a/drivers/platform/x86/amd/hsmp/hwmon.c +++ b/drivers/platform/x86/amd/hsmp/hwmon.c @@ -31,6 +31,9 @@ static int hsmp_hwmon_write(struct device *dev, enum hwmon_sensor_types type, if (attr != hwmon_power_cap) return -EOPNOTSUPP; + if (val < 0) + return -EINVAL; + msg.num_args = 1; msg.args[0] = val / MICROWATT_PER_MILLIWATT; msg.msg_id = HSMP_SET_SOCKET_POWER_LIMIT; diff --git a/drivers/platform/x86/amd/pmc/mp1_stb.c b/drivers/platform/x86/amd/pmc/mp1_stb.c index 1a06425f596d..c2f4e6caf58b 100644 --- a/drivers/platform/x86/amd/pmc/mp1_stb.c +++ b/drivers/platform/x86/amd/pmc/mp1_stb.c @@ -157,7 +157,7 @@ static int amd_stb_debugfs_open_v2(struct inode *inode, struct file *filp) struct amd_pmc_dev *dev = filp->f_inode->i_private; u32 fsize, num_samples, val, stb_rdptr_offset = 0; struct amd_stb_v2_data *stb_data_arr; - int ret; + int ret = 0; /* Write dummy postcode while reading the STB buffer */ ret = amd_stb_write(dev, AMD_PMC_STB_DUMMY_PC); @@ -176,22 +176,24 @@ static int amd_stb_debugfs_open_v2(struct inode *inode, struct file *filp) * the enhanced dram size. Note that we land here only for the * platforms that support enhanced dram size reporting. */ - if (dump_custom_stb) - return amd_stb_handle_efr(filp); + if (dump_custom_stb) { + ret = amd_stb_handle_efr(filp); + goto out; + } /* Get the num_samples to calculate the last push location */ ret = amd_pmc_send_cmd(dev, S2D_NUM_SAMPLES, &num_samples, dev->stb_arg.s2d_msg_id, true); - /* Clear msg_port for other SMU operation */ - dev->msg_port = MSG_PORT_PMC; if (ret) { dev_err(dev->dev, "error: S2D_NUM_SAMPLES not supported : %d\n", ret); - return ret; + goto out; } fsize = min(num_samples, S2D_TELEMETRY_BYTES_MAX); stb_data_arr = kmalloc(struct_size(stb_data_arr, data, fsize), GFP_KERNEL); - if (!stb_data_arr) - return -ENOMEM; + if (!stb_data_arr) { + ret = -ENOMEM; + goto out; + } stb_data_arr->size = fsize; @@ -214,7 +216,10 @@ static int amd_stb_debugfs_open_v2(struct inode *inode, struct file *filp) filp->private_data = stb_data_arr; - return 0; +out: + /* Restore the default message port for subsequent SMU operations */ + dev->msg_port = MSG_PORT_PMC; + return ret; } static ssize_t amd_stb_debugfs_read_v2(struct file *filp, char __user *buf, size_t size, diff --git a/drivers/platform/x86/dell/dell-wmi-base.c b/drivers/platform/x86/dell/dell-wmi-base.c index 907f1da01c8d..ba6c88e2a5ec 100644 --- a/drivers/platform/x86/dell/dell-wmi-base.c +++ b/drivers/platform/x86/dell/dell-wmi-base.c @@ -851,9 +851,22 @@ static int __init dell_wmi_init(void) err = dell_privacy_register_driver(); if (err) - return err; + goto out_smbios; - return wmi_driver_register(&dell_wmi_driver); + err = wmi_driver_register(&dell_wmi_driver); + if (err) + goto out_privacy; + + return 0; + +out_privacy: + dell_privacy_unregister_driver(); + +out_smbios: + if (wmi_requires_smbios_request) + dell_wmi_events_set_enabled(false); + + return err; } late_initcall(dell_wmi_init); diff --git a/drivers/platform/x86/dell/dell-wmi-privacy.c b/drivers/platform/x86/dell/dell-wmi-privacy.c index 4b65e1655d42..7c2b6fae96c5 100644 --- a/drivers/platform/x86/dell/dell-wmi-privacy.c +++ b/drivers/platform/x86/dell/dell-wmi-privacy.c @@ -69,11 +69,11 @@ bool dell_privacy_has_mic_mute(void) { struct privacy_wmi_data *priv; - mutex_lock(&list_mutex); + guard(mutex)(&list_mutex); + priv = list_first_entry_or_null(&wmi_list, struct privacy_wmi_data, list); - mutex_unlock(&list_mutex); return priv && (priv->features_present & BIT(DELL_PRIVACY_TYPE_AUDIO)); } diff --git a/drivers/platform/x86/dell/dell-wmi-sysman/dell-wmi-sysman.h b/drivers/platform/x86/dell/dell-wmi-sysman/dell-wmi-sysman.h index 817ee7ba07ca..b0810a02e6da 100644 --- a/drivers/platform/x86/dell/dell-wmi-sysman/dell-wmi-sysman.h +++ b/drivers/platform/x86/dell/dell-wmi-sysman/dell-wmi-sysman.h @@ -107,7 +107,7 @@ enum { static int get_##type##_instance_id(struct kobject *kobj) \ { \ int i; \ - for (i = 0; i <= wmi_priv.type##_instances_count; i++) { \ + for (i = 0; i < wmi_priv.type##_instances_count; i++) { \ if (!(strcmp(kobj->name, wmi_priv.type##_data[i].attribute_name)))\ return i; \ } \ diff --git a/drivers/platform/x86/hp/hp-bioscfg/passwdobj-attributes.c b/drivers/platform/x86/hp/hp-bioscfg/passwdobj-attributes.c index 3c7fbb1a87e2..6b94f90bea07 100644 --- a/drivers/platform/x86/hp/hp-bioscfg/passwdobj-attributes.c +++ b/drivers/platform/x86/hp/hp-bioscfg/passwdobj-attributes.c @@ -353,6 +353,11 @@ static int hp_populate_password_elements_from_package(union acpi_object *passwor case PSWD_ENCODINGS: size = min_t(u32, password_data->encodings_size, MAX_ENCODINGS_SIZE); for (pos_values = 0; pos_values < size; pos_values++) { + if (elem + pos_values >= password_obj_count) { + pr_err("Error elem-objects package is too small\n"); + return -EINVAL; + } + ret = hp_convert_hexstr_to_str(password_obj[elem + pos_values].string.pointer, password_obj[elem + pos_values].string.length, &str_value, &value_len); diff --git a/drivers/platform/x86/hp/hp-wmi.c b/drivers/platform/x86/hp/hp-wmi.c index 550d4b39a92a..5eb2aade44cc 100644 --- a/drivers/platform/x86/hp/hp-wmi.c +++ b/drivers/platform/x86/hp/hp-wmi.c @@ -194,6 +194,10 @@ static const struct dmi_system_id victus_s_thermal_profile_boards[] __initconst .matches = { DMI_MATCH(DMI_BOARD_NAME, "8D41") }, .driver_data = (void *)&victus_s_thermal_params, }, + { + .matches = { DMI_MATCH(DMI_BOARD_NAME, "8DD6") }, + .driver_data = (void *)&omen_v1_thermal_params, + }, {}, }; diff --git a/drivers/platform/x86/intel/int1092/intel_sar.c b/drivers/platform/x86/intel/int1092/intel_sar.c index e526841aff60..7263114f0b3d 100644 --- a/drivers/platform/x86/intel/int1092/intel_sar.c +++ b/drivers/platform/x86/intel/int1092/intel_sar.c @@ -91,8 +91,10 @@ static acpi_status parse_package(struct wwan_sar_context *context, union acpi_ob item->package.count <= data->total_dev_mode) return AE_ERROR; - data->device_mode_info = kmalloc_array(data->total_dev_mode, - sizeof(struct wwan_device_mode_info), GFP_KERNEL); + data->device_mode_info = devm_kmalloc_array(&context->sar_device->dev, + data->total_dev_mode, + sizeof(*data->device_mode_info), + GFP_KERNEL); if (!data->device_mode_info) return AE_ERROR; @@ -245,21 +247,26 @@ static void sar_get_data(int reg, struct wwan_sar_context *context) static int sar_probe(struct platform_device *device) { struct wwan_sar_context *context; + acpi_handle handle; int reg; int result; - context = kzalloc(sizeof(*context), GFP_KERNEL); + handle = ACPI_HANDLE(&device->dev); + if (!handle) + return -ENODEV; + + context = devm_kzalloc(&device->dev, sizeof(*context), GFP_KERNEL); if (!context) return -ENOMEM; context->sar_device = device; - context->handle = ACPI_HANDLE(&device->dev); + context->handle = handle; dev_set_drvdata(&device->dev, context); result = guid_parse(SAR_DSM_UUID, &context->guid); if (result) { dev_err(&device->dev, "SAR UUID parse error: %d\n", result); - goto r_free; + return result; } for (reg = 0; reg < MAX_REGULATORY; reg++) @@ -267,43 +274,29 @@ static int sar_probe(struct platform_device *device) if (sar_get_device_mode(device) != AE_OK) { dev_err(&device->dev, "Failed to get device mode\n"); - result = -EIO; - goto r_free; + return -EIO; } result = sysfs_create_group(&device->dev.kobj, &intcsar_group); if (result) { dev_err(&device->dev, "sysfs creation failed\n"); - goto r_free; + return result; } if (acpi_install_notify_handler(ACPI_HANDLE(&device->dev), ACPI_DEVICE_NOTIFY, sar_notify, (void *)device) != AE_OK) { dev_err(&device->dev, "Failed acpi_install_notify_handler\n"); - result = -EIO; - goto r_sys; + sysfs_remove_group(&device->dev.kobj, &intcsar_group); + return -EIO; } return 0; - -r_sys: - sysfs_remove_group(&device->dev.kobj, &intcsar_group); -r_free: - kfree(context); - return result; } static void sar_remove(struct platform_device *device) { - struct wwan_sar_context *context = dev_get_drvdata(&device->dev); - int reg; - acpi_remove_notify_handler(ACPI_HANDLE(&device->dev), ACPI_DEVICE_NOTIFY, sar_notify); sysfs_remove_group(&device->dev.kobj, &intcsar_group); - for (reg = 0; reg < MAX_REGULATORY; reg++) - kfree(context->config_data[reg].device_mode_info); - - kfree(context); } static struct platform_driver sar_driver = { diff --git a/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c b/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c index 2e2e168d301c..7c45b05190f3 100644 --- a/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c +++ b/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c @@ -625,7 +625,7 @@ static long isst_if_core_power_state(void __user *argp) return -EINVAL; if (core_power.get_set) { - if (power_domain_info->write_blocked) + if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN)) return -EPERM; if (core_power.enable > SST_CP_MAX_ENABLE || @@ -684,7 +684,7 @@ static long isst_if_clos_param(void __user *argp) return -EINVAL; if (clos_param.get_set) { - if (power_domain_info->write_blocked) + if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN)) return -EPERM; if (!in_range(clos_param.min_freq_mhz / SST_MUL_FACTOR_FREQ, 0, SST_MAX_FREQ + 1)) @@ -795,7 +795,8 @@ static long isst_if_clos_assoc(void __user *argp) power_domain_info = &sst_inst->power_domain_info[part][punit_id]; - if (assoc_cmds.get_set && power_domain_info->write_blocked) + if (assoc_cmds.get_set && (power_domain_info->write_blocked || + !capable(CAP_SYS_ADMIN))) return -EPERM; offset = SST_CLOS_ASSOC_0_OFFSET + @@ -973,7 +974,10 @@ static int isst_if_set_perf_level(void __user *argp) if (!power_domain_info) return -EINVAL; - if (power_domain_info->write_blocked) + if (perf_level.level > power_domain_info->max_level) + return -EINVAL; + + if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN)) return -EPERM; if (!(power_domain_info->pp_header.allowed_level_mask & BIT(perf_level.level))) @@ -1033,7 +1037,7 @@ static int isst_if_set_perf_feature(void __user *argp) if (!power_domain_info) return -EINVAL; - if (power_domain_info->write_blocked) + if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN)) return -EPERM; if (perf_feature.feature & ~SST_PP_FEATURE_STATE_VALID_MASK) diff --git a/drivers/platform/x86/lg-laptop.c b/drivers/platform/x86/lg-laptop.c index 6af6cf477c5b..f4e43e09cbe4 100644 --- a/drivers/platform/x86/lg-laptop.c +++ b/drivers/platform/x86/lg-laptop.c @@ -98,6 +98,7 @@ static u32 inited; #define INIT_SPARSE_KEYMAP 0x80 static int battery_limit_use_wmbb; +static bool kbd_backlight_available; static struct led_classdev kbd_backlight; static enum led_brightness get_kbd_backlight_level(struct device *dev); @@ -212,6 +213,7 @@ static union acpi_object *lg_wmbb(struct device *dev, u32 method_id, u32 arg1, u static void wmi_notify(union acpi_object *obj, void *context) { long data = (long)context; + unsigned int brightness; pr_debug("event guid %li\n", data); if (!obj) @@ -222,8 +224,11 @@ static void wmi_notify(union acpi_object *obj, void *context) struct key_entry *key; if (eventcode == 0x10000000) { - led_classdev_notify_brightness_hw_changed( - &kbd_backlight, get_kbd_backlight_level(kbd_backlight.dev->parent)); + if (kbd_backlight_available) { + brightness = get_kbd_backlight_level(kbd_backlight.dev->parent); + led_classdev_notify_brightness_hw_changed(&kbd_backlight, + brightness); + } } else { key = sparse_keymap_entry_from_scancode( wmi_input_dev, eventcode); @@ -269,11 +274,6 @@ static void wmi_input_setup(void) } } -static void acpi_notify(struct acpi_device *device, u32 event) -{ - acpi_handle_debug(device->handle, "notify: %d\n", event); -} - static ssize_t fan_mode_store(struct device *dev, struct device_attribute *attr, const char *buffer, size_t count) @@ -753,13 +753,13 @@ static void lg_laptop_remove_address_space_handler(void *data) &lg_laptop_address_space_handler); } -static int acpi_add(struct acpi_device *device) +static int acpi_probe(struct platform_device *pdev) { struct platform_device_info pdev_info = { - .fwnode = acpi_fwnode_handle(device), .name = PLATFORM_NAME, .id = PLATFORM_DEVID_NONE, }; + struct acpi_device *device; acpi_status status; int ret; const char *product; @@ -768,13 +768,19 @@ static int acpi_add(struct acpi_device *device) if (pf_device) return 0; + device = ACPI_COMPANION(&pdev->dev); + if (!device) + return -ENODEV; + + pdev_info.fwnode = acpi_fwnode_handle(device), + status = acpi_install_address_space_handler(device->handle, LG_ADDRESS_SPACE_ID, &lg_laptop_address_space_handler, - NULL, &device->dev); + NULL, &pdev->dev); if (ACPI_FAILURE(status)) return -ENODEV; - ret = devm_add_action_or_reset(&device->dev, lg_laptop_remove_address_space_handler, + ret = devm_add_action_or_reset(&pdev->dev, lg_laptop_remove_address_space_handler, device); if (ret < 0) return ret; @@ -844,8 +850,13 @@ static int acpi_add(struct acpi_device *device) goto out_platform_device; /* LEDs are optional */ - led_classdev_register(&pf_device->dev, &kbd_backlight); - led_classdev_register(&pf_device->dev, &tpad_led); + ret = devm_led_classdev_register(&pdev->dev, &kbd_backlight); + if (ret < 0) + kbd_backlight_available = false; + else + kbd_backlight_available = true; + + devm_led_classdev_register(&pdev->dev, &tpad_led); wmi_input_setup(); battery_hook_register(&battery_hook); @@ -859,13 +870,10 @@ static int acpi_add(struct acpi_device *device) return ret; } -static void acpi_remove(struct acpi_device *device) +static void acpi_remove(struct platform_device *pdev) { sysfs_remove_group(&pf_device->dev.kobj, &dev_attribute_group); - led_classdev_unregister(&tpad_led); - led_classdev_unregister(&kbd_backlight); - battery_hook_unregister(&battery_hook); wmi_input_destroy(); platform_device_unregister(pf_device); @@ -879,34 +887,13 @@ static const struct acpi_device_id device_ids[] = { }; MODULE_DEVICE_TABLE(acpi, device_ids); -static struct acpi_driver acpi_driver = { - .name = "LG Gram Laptop Support", - .class = "lg-laptop", - .ids = device_ids, - .ops = { - .add = acpi_add, - .remove = acpi_remove, - .notify = acpi_notify, - }, +static struct platform_driver acpi_driver = { + .probe = acpi_probe, + .remove = acpi_remove, + .driver = { + .name = "LG Gram Laptop Support", + .acpi_match_table = device_ids, + }, }; -static int __init acpi_init(void) -{ - int result; - - result = acpi_bus_register_driver(&acpi_driver); - if (result < 0) { - pr_debug("Error registering driver\n"); - return -ENODEV; - } - - return 0; -} - -static void __exit acpi_exit(void) -{ - acpi_bus_unregister_driver(&acpi_driver); -} - -module_init(acpi_init); -module_exit(acpi_exit); +module_platform_driver(acpi_driver); diff --git a/drivers/pmdomain/bcm/bcm2835-power.c b/drivers/pmdomain/bcm/bcm2835-power.c index f2472f1e1752..f3df1198ec7e 100644 --- a/drivers/pmdomain/bcm/bcm2835-power.c +++ b/drivers/pmdomain/bcm/bcm2835-power.c @@ -675,7 +675,12 @@ static int bcm2835_power_probe(struct platform_device *pdev) if (ret) goto fail; - of_genpd_add_provider_onecell(dev->parent->of_node, &power->pd_xlate); + ret = of_genpd_add_provider_onecell(dev->parent->of_node, + &power->pd_xlate); + if (ret) { + dev_err_probe(dev, ret, "failed to add genpd provider\n"); + goto fail; + } dev_info(dev, "Broadcom BCM2835 power domains driver"); return 0; diff --git a/drivers/power/reset/sc27xx-poweroff.c b/drivers/power/reset/sc27xx-poweroff.c index 90287c31992c..393bd1c33b73 100644 --- a/drivers/power/reset/sc27xx-poweroff.c +++ b/drivers/power/reset/sc27xx-poweroff.c @@ -28,7 +28,7 @@ static struct regmap *regmap; * taking cpus down to avoid racing regmap or spi mutex lock when poweroff * system through PMIC. */ -static void sc27xx_poweroff_shutdown(void) +static void sc27xx_poweroff_shutdown(void *data) { #ifdef CONFIG_HOTPLUG_CPU int cpu; @@ -40,10 +40,14 @@ static void sc27xx_poweroff_shutdown(void) #endif } -static struct syscore_ops poweroff_syscore_ops = { +static const struct syscore_ops poweroff_syscore_ops = { .shutdown = sc27xx_poweroff_shutdown, }; +static struct syscore poweroff_syscore = { + .ops = &poweroff_syscore_ops, +}; + static void sc27xx_poweroff_do_poweroff(void) { /* Disable the external subsys connection's power firstly */ @@ -62,7 +66,7 @@ static int sc27xx_poweroff_probe(struct platform_device *pdev) return -ENODEV; pm_power_off = sc27xx_poweroff_do_poweroff; - register_syscore_ops(&poweroff_syscore_ops); + register_syscore(&poweroff_syscore); return 0; } diff --git a/drivers/power/supply/bd99954-charger.h b/drivers/power/supply/bd99954-charger.h index f58897925383..d053ac901274 100644 --- a/drivers/power/supply/bd99954-charger.h +++ b/drivers/power/supply/bd99954-charger.h @@ -279,17 +279,7 @@ enum bd9995x_fields { F_VBUS_EXTID, F_VBUS_IDRDET, F_VBUS_INDO, - F_VBUS_UCDSWEN, - F_VBUS_RREF_EN, - F_VBUS_DPPU_EN, - F_VBUS_DPREF_EN, - F_VBUS_DMREF_EN, - F_VBUS_DPDET_EN, - F_VBUS_DMDET_EN, - F_VBUS_DPSINK_EN, - F_VBUS_DMSINK_EN, - F_VBUS_DP_BUFF_EN, - F_VBUS_DM_BUFF_EN, + F_VBUS_EXTCLKENBL, F_VBUS_PLSTESTEN, F_VBUS_UCDSWEN_TSTENB, @@ -626,17 +616,6 @@ static const struct reg_field bd9995x_reg_fields[] = { [F_VBUS_EXTID] = REG_FIELD(VBUS_IDD_STATUS, 5, 5), [F_VBUS_IDRDET] = REG_FIELD(VBUS_IDD_STATUS, 4, 4), [F_VBUS_INDO] = REG_FIELD(VBUS_IDD_STATUS, 0, 3), - [F_VBUS_UCDSWEN] = REG_FIELD(VCC_UCD_FCTRL_SET, 10, 10), - [F_VBUS_RREF_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 9, 9), - [F_VBUS_DPPU_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 8, 8), - [F_VBUS_DPREF_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 7, 7), - [F_VBUS_DMREF_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 6, 6), - [F_VBUS_DPDET_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 5, 5), - [F_VBUS_DMDET_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 4, 4), - [F_VBUS_DPSINK_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 3, 3), - [F_VBUS_DMSINK_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 2, 2), - [F_VBUS_DP_BUFF_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 1, 1), - [F_VBUS_DM_BUFF_EN] = REG_FIELD(VCC_UCD_FCTRL_SET, 0, 0), [F_VBUS_EXTCLKENBL] = REG_FIELD(VBUS_UCD_FCTRL_EN, 15, 15), [F_VBUS_PLSTESTEN] = REG_FIELD(VBUS_UCD_FCTRL_EN, 14, 14), diff --git a/drivers/power/supply/bq27xxx_battery.c b/drivers/power/supply/bq27xxx_battery.c index 45f0e39b8c2d..01c9ec1116fa 100644 --- a/drivers/power/supply/bq27xxx_battery.c +++ b/drivers/power/supply/bq27xxx_battery.c @@ -330,7 +330,7 @@ static u8 [BQ27XXX_REG_FLAGS] = 0x0a, [BQ27XXX_REG_TTE] = 0x16, [BQ27XXX_REG_TTF] = INVALID_REG_ADDR, - [BQ27XXX_REG_TTES] = 0x1c, + [BQ27XXX_REG_TTES] = 0x1a, [BQ27XXX_REG_TTECP] = INVALID_REG_ADDR, [BQ27XXX_REG_NAC] = 0x0c, [BQ27XXX_REG_RC] = 0x10, @@ -495,7 +495,7 @@ static u8 [BQ27XXX_REG_RC] = 0x10, [BQ27XXX_REG_FCC] = 0x12, [BQ27XXX_REG_CYCT] = 0x2a, - [BQ27XXX_REG_AE] = 0x22, + [BQ27XXX_REG_AE] = INVALID_REG_ADDR, [BQ27XXX_REG_SOC] = 0x2c, [BQ27XXX_REG_DCAP] = 0x3c, [BQ27XXX_REG_AP] = 0x22, @@ -516,7 +516,7 @@ static u8 [BQ27XXX_REG_RC] = 0x10, [BQ27XXX_REG_FCC] = 0x12, [BQ27XXX_REG_CYCT] = 0x2a, - [BQ27XXX_REG_AE] = 0x22, + [BQ27XXX_REG_AE] = INVALID_REG_ADDR, [BQ27XXX_REG_SOC] = 0x2c, [BQ27XXX_REG_DCAP] = 0x3c, [BQ27XXX_REG_AP] = 0x22, diff --git a/drivers/power/supply/isp1704_charger.c b/drivers/power/supply/isp1704_charger.c index 237912a92272..e329321d06db 100644 --- a/drivers/power/supply/isp1704_charger.c +++ b/drivers/power/supply/isp1704_charger.c @@ -482,6 +482,7 @@ static void isp1704_charger_remove(struct platform_device *pdev) struct isp1704_charger *isp = platform_get_drvdata(pdev); usb_unregister_notifier(isp->phy, &isp->nb); + cancel_work_sync(&isp->work); power_supply_unregister(isp->psy); isp1704_charger_set_power(isp, 0); } diff --git a/drivers/power/supply/sbs-battery.c b/drivers/power/supply/sbs-battery.c index 43c48196c167..3947429f6f81 100644 --- a/drivers/power/supply/sbs-battery.c +++ b/drivers/power/supply/sbs-battery.c @@ -217,6 +217,7 @@ struct sbs_info { u32 flags; int technology; char strings[NR_STRING_BUFFERS][I2C_SMBUS_BLOCK_MAX + 1]; + char serial[5]; }; static char *sbs_get_string_buf(struct sbs_info *chip, @@ -821,18 +822,18 @@ static int sbs_get_battery_capacity(struct i2c_client *client, return 0; } -static char sbs_serial[5]; static int sbs_get_battery_serial_number(struct i2c_client *client, union power_supply_propval *val) { + struct sbs_info *chip = i2c_get_clientdata(client); int ret; ret = sbs_read_word_data(client, sbs_data[REG_SERIAL_NUMBER].addr); if (ret < 0) return ret; - sprintf(sbs_serial, "%04x", ret); - val->strval = sbs_serial; + sprintf(chip->serial, "%04x", ret); + val->strval = chip->serial; return 0; } diff --git a/drivers/power/supply/sc2731_charger.c b/drivers/power/supply/sc2731_charger.c index 58b86fd78771..2b25e44da797 100644 --- a/drivers/power/supply/sc2731_charger.c +++ b/drivers/power/supply/sc2731_charger.c @@ -466,6 +466,7 @@ static int sc2731_charger_probe(struct platform_device *pdev) mutex_init(&info->lock); info->dev = &pdev->dev; INIT_WORK(&info->work, sc2731_charger_work); + platform_set_drvdata(pdev, info); info->regmap = dev_get_regmap(pdev->dev.parent, NULL); if (!info->regmap) { @@ -516,6 +517,7 @@ static void sc2731_charger_remove(struct platform_device *pdev) struct sc2731_charger_info *info = platform_get_drvdata(pdev); usb_unregister_notifier(info->usb_phy, &info->usb_notify); + cancel_work_sync(&info->work); } static const struct of_device_id sc2731_charger_of_match[] = { diff --git a/drivers/powercap/intel_rapl_tpmi.c b/drivers/powercap/intel_rapl_tpmi.c index 34c0bd1edd61..ee2a5e839787 100644 --- a/drivers/powercap/intel_rapl_tpmi.c +++ b/drivers/powercap/intel_rapl_tpmi.c @@ -314,7 +314,10 @@ static int intel_rapl_tpmi_probe(struct auxiliary_device *auxdev, goto err; } - rapl_package_add_pmu(trp->rp); + ret = rapl_package_add_pmu(trp->rp); + if (ret) + dev_info(&auxdev->dev, "Failed to add RAPL PMU for Package%d, %d\n", + info->package_id, ret); auxiliary_set_drvdata(auxdev, trp); diff --git a/drivers/pps/clients/pps-gpio.c b/drivers/pps/clients/pps-gpio.c index 935da68610c7..1eba112afeb8 100644 --- a/drivers/pps/clients/pps-gpio.c +++ b/drivers/pps/clients/pps-gpio.c @@ -32,38 +32,37 @@ struct pps_gpio_device_data { struct gpio_desc *echo_pin; struct timer_list echo_timer; /* timer to reset echo active state */ bool assert_falling_edge; - bool capture_clear; unsigned int echo_active_ms; /* PPS echo active duration */ unsigned long echo_timeout; /* timer timeout value in jiffies */ + struct pps_event_time ts; /* timestamp captured in hardirq */ }; /* * Report the PPS event */ -static irqreturn_t pps_gpio_irq_handler(int irq, void *data) +/* + * Primary hardirq handler -- runs in hardirq context even on PREEMPT_RT. + * Only captures the timestamp; all other work is deferred to the thread. + */ +static irqreturn_t pps_gpio_irq_hardirq(int irq, void *data) { - const struct pps_gpio_device_data *info; - struct pps_event_time ts; - int rising_edge; - - /* Get the time stamp first */ - pps_get_ts(&ts); - - info = data; - - /* Small trick to bypass the check on edge's direction when capture_clear is unset */ - rising_edge = info->capture_clear ? - gpiod_get_value(info->gpio_pin) : !info->assert_falling_edge; - if ((rising_edge && !info->assert_falling_edge) || - (!rising_edge && info->assert_falling_edge)) - pps_event(info->pps, &ts, PPS_CAPTUREASSERT, data); - else if (info->capture_clear && - ((rising_edge && info->assert_falling_edge) || - (!rising_edge && !info->assert_falling_edge))) - pps_event(info->pps, &ts, PPS_CAPTURECLEAR, data); - else - dev_warn_ratelimited(&info->pps->dev, "IRQ did not trigger any PPS event\n"); + struct pps_gpio_device_data *info = data; + + pps_get_ts(&info->ts); + + return IRQ_WAKE_THREAD; +} + +/* + * Threaded handler -- processes the PPS event using the timestamp + * captured in hardirq context above. + */ +static irqreturn_t pps_gpio_irq_thread(int irq, void *data) +{ + struct pps_gpio_device_data *info = data; + + pps_event(info->pps, &info->ts, PPS_CAPTUREASSERT, data); return IRQ_HANDLED; } @@ -79,11 +78,6 @@ static void pps_gpio_echo(struct pps_device *pps, int event, void *data) if (pps->params.mode & PPS_ECHOASSERT) gpiod_set_value(info->echo_pin, 1); break; - - case PPS_CAPTURECLEAR: - if (pps->params.mode & PPS_ECHOCLEAR) - gpiod_set_value(info->echo_pin, 1); - break; } /* fire the timer */ @@ -145,15 +139,8 @@ static int pps_gpio_setup(struct device *dev) static unsigned long get_irqf_trigger_flags(const struct pps_gpio_device_data *data) { - unsigned long flags = data->assert_falling_edge ? - IRQF_TRIGGER_FALLING : IRQF_TRIGGER_RISING; - - if (data->capture_clear) { - flags |= ((flags & IRQF_TRIGGER_RISING) ? - IRQF_TRIGGER_FALLING : IRQF_TRIGGER_RISING); - } - - return flags; + return data->assert_falling_edge ? IRQF_TRIGGER_FALLING : + IRQF_TRIGGER_RISING; } static int pps_gpio_probe(struct platform_device *pdev) @@ -186,9 +173,6 @@ static int pps_gpio_probe(struct platform_device *pdev) /* initialize PPS specific parts of the bookkeeping data structure. */ data->info.mode = PPS_CAPTUREASSERT | PPS_OFFSETASSERT | PPS_ECHOASSERT | PPS_CANWAIT | PPS_TSFMT_TSPEC; - if (data->capture_clear) - data->info.mode |= PPS_CAPTURECLEAR | PPS_OFFSETCLEAR | - PPS_ECHOCLEAR; data->info.owner = THIS_MODULE; snprintf(data->info.name, PPS_MAX_NAME_LEN - 1, "%s.%d", pdev->name, pdev->id); @@ -200,8 +184,6 @@ static int pps_gpio_probe(struct platform_device *pdev) /* register PPS source */ pps_default_params = PPS_CAPTUREASSERT | PPS_OFFSETASSERT; - if (data->capture_clear) - pps_default_params |= PPS_CAPTURECLEAR | PPS_OFFSETCLEAR; data->pps = pps_register_source(&data->info, pps_default_params); if (IS_ERR(data->pps)) { dev_err(dev, "failed to register IRQ %d as PPS source\n", @@ -210,8 +192,10 @@ static int pps_gpio_probe(struct platform_device *pdev) } /* register IRQ interrupt handler */ - ret = request_irq(data->irq, pps_gpio_irq_handler, - get_irqf_trigger_flags(data), data->info.name, data); + ret = request_threaded_irq(data->irq, + pps_gpio_irq_hardirq, pps_gpio_irq_thread, + get_irqf_trigger_flags(data) | IRQF_ONESHOT, + data->info.name, data); if (ret) { pps_unregister_source(data->pps); dev_err(dev, "failed to acquire IRQ %d\n", data->irq); diff --git a/drivers/pps/pps.c b/drivers/pps/pps.c index c6b8b6478276..2e04636f4380 100644 --- a/drivers/pps/pps.c +++ b/drivers/pps/pps.c @@ -63,13 +63,19 @@ static int pps_cdev_pps_fetch(struct pps_device *pps, struct pps_fdata *fdata) err = wait_event_interruptible(pps->queue, ev != pps->last_ev); else { + struct timespec64 ts; unsigned long ticks; dev_dbg(&pps->dev, "timeout %lld.%09d\n", (long long) fdata->timeout.sec, fdata->timeout.nsec); - ticks = fdata->timeout.sec * HZ; - ticks += fdata->timeout.nsec / (NSEC_PER_SEC / HZ); + + if (fdata->timeout.sec < 0) + return -ETIMEDOUT; + + ts.tv_sec = fdata->timeout.sec; + ts.tv_nsec = fdata->timeout.nsec; + ticks = timespec64_to_jiffies(&ts); if (ticks != 0) { err = wait_event_interruptible_timeout( diff --git a/drivers/ptp/ptp_netc.c b/drivers/ptp/ptp_netc.c index 1c20d7efab92..19ca99d80e95 100644 --- a/drivers/ptp/ptp_netc.c +++ b/drivers/ptp/ptp_netc.c @@ -440,7 +440,7 @@ static int net_timer_enable_perout(struct netc_timer *priv, } if (on) { - u64 period_ns, gclk_period, max_period, min_period; + u64 period_ns, gclk_period, min_period; struct timespec64 period, stime; u32 integral_period; int alarm_id; @@ -450,12 +450,12 @@ static int net_timer_enable_perout(struct netc_timer *priv, period_ns = timespec64_to_ns(&period); integral_period = netc_timer_get_integral_period(priv); - max_period = (u64)NETC_TMR_DEFAULT_FIPER + integral_period; gclk_period = netc_timer_get_gclk_period(priv); min_period = gclk_period * 4 + integral_period; - if (period_ns > max_period || period_ns < min_period) { - dev_err(dev, "The period range is %llu ~ %llu\n", - min_period, max_period); + if (period_ns > NETC_TMR_DEFAULT_FIPER || + period_ns < min_period) { + dev_err(dev, "The period range is %llu ~ %lu\n", + min_period, NETC_TMR_DEFAULT_FIPER); err = -EINVAL; goto unlock_spinlock; } @@ -482,6 +482,9 @@ static int net_timer_enable_perout(struct netc_timer *priv, netc_timer_enable_periodic_pulse(priv, channel); } else { + if (!pp->enabled) + goto unlock_spinlock; + netc_timer_disable_periodic_pulse(priv, channel); priv->fs_alarm_bitmap &= ~BIT(pp->alarm_id); memset(pp, 0, sizeof(*pp)); diff --git a/drivers/rapidio/rio-scan.c b/drivers/rapidio/rio-scan.c index dcd6619a4b02..3cc25d053451 100644 --- a/drivers/rapidio/rio-scan.c +++ b/drivers/rapidio/rio-scan.c @@ -874,6 +874,7 @@ static struct rio_net *rio_scan_alloc_net(struct rio_mport *mport, net->dev.release = rio_scan_release_dev; if (rio_add_net(net)) { put_device(&net->dev); + mport->net = NULL; net = NULL; } } diff --git a/drivers/regulator/adp5055-regulator.c b/drivers/regulator/adp5055-regulator.c index 4b004a6b2f84..5b83f36351a2 100644 --- a/drivers/regulator/adp5055-regulator.c +++ b/drivers/regulator/adp5055-regulator.c @@ -225,7 +225,7 @@ static int adp5055_of_parse_cb(struct device_node *np, adp5055->dvs_limit_upper[id] = pval; if (adp5055->dvs_limit_upper[id] > 192000 || adp5055->dvs_limit_upper[id] < 12000) - return dev_err_probe(config->dev, adp5055->dvs_limit_upper[id], + return dev_err_probe(config->dev, -EINVAL, "Out of range - dvs-limit-upper-microvolt value."); ret = of_property_read_u32(np, "adi,dvs-limit-lower-microvolt", &pval); diff --git a/drivers/regulator/core.c b/drivers/regulator/core.c index 019606bc36b9..97b4ee5d2990 100644 --- a/drivers/regulator/core.c +++ b/drivers/regulator/core.c @@ -27,6 +27,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -6604,8 +6605,9 @@ static int __init regulator_init_complete(void) * we'd only do this on systems that need it, and a kernel * command line option might be useful. */ - schedule_delayed_work(®ulator_init_complete_work, - msecs_to_jiffies(30000)); + queue_delayed_work(system_freezable_wq, + ®ulator_init_complete_work, + msecs_to_jiffies(30000)); return 0; } diff --git a/drivers/regulator/qcom-rpmh-regulator.c b/drivers/regulator/qcom-rpmh-regulator.c index 109f0aae09b1..933eb0c28fa6 100644 --- a/drivers/regulator/qcom-rpmh-regulator.c +++ b/drivers/regulator/qcom-rpmh-regulator.c @@ -77,6 +77,7 @@ enum rpmh_regulator_type { * @hpm_min_load_uA: Minimum load current in microamps that requires * high power mode (HPM) operation. This is used * for LDO hardware type regulators only. + * @pmic_bypass_mode: The PMIC bypass mode value. * @pmic_mode_map: Array indexed by regulator framework mode * containing PMIC hardware modes. Must be large * enough to index all framework modes supported @@ -91,6 +92,7 @@ struct rpmh_vreg_hw_data { int n_linear_ranges; int n_voltages; int hpm_min_load_uA; + int pmic_bypass_mode; const int *pmic_mode_map; unsigned int (*of_map_mode)(unsigned int mode); }; @@ -277,7 +279,7 @@ static int rpmh_regulator_vrm_set_mode_bypass(struct rpmh_vreg *vreg, return pmic_mode; if (bypassed) - cmd.data = PMIC4_BOB_MODE_PASS; + cmd.data = vreg->hw_data->pmic_bypass_mode; else cmd.data = pmic_mode; @@ -692,6 +694,7 @@ static const struct rpmh_vreg_hw_data pmic4_bob = { }, .n_linear_ranges = 1, .n_voltages = 84, + .pmic_bypass_mode = PMIC4_BOB_MODE_PASS, .pmic_mode_map = pmic_mode_map_pmic4_bob, .of_map_mode = rpmh_regulator_pmic4_bob_of_map_mode, }; @@ -900,6 +903,7 @@ static const struct rpmh_vreg_hw_data pmic5_bob = { }, .n_linear_ranges = 1, .n_voltages = 32, + .pmic_bypass_mode = PMIC5_BOB_MODE_PASS, .pmic_mode_map = pmic_mode_map_pmic5_bob, .of_map_mode = rpmh_regulator_pmic4_bob_of_map_mode, }; diff --git a/drivers/regulator/tps6594-regulator.c b/drivers/regulator/tps6594-regulator.c index 645e83462c64..31a5218d5510 100644 --- a/drivers/regulator/tps6594-regulator.c +++ b/drivers/regulator/tps6594-regulator.c @@ -669,13 +669,20 @@ static int tps6594_regulator_probe(struct platform_device *pdev) * buck_configured to avoid creating bucks for every buck in multiphase */ for (multi = 0; multi < desc->num_multi_phase_regs; multi++) { + struct device_node *np_parent; + multi_regs = &desc->multi_phase_regs[multi]; np = of_find_node_by_name(tps->dev->of_node, multi_regs->supply_name); - npname = of_node_full_name(np); - np_pmic_parent = of_get_parent(of_get_parent(np)); - if (of_node_cmp(of_node_full_name(np_pmic_parent), tps->dev->of_node->full_name)) + if (!np) continue; - if (strcmp(npname, multi_regs->supply_name) == 0) { + + npname = of_node_full_name(np); + np_parent = of_get_parent(np); + np_pmic_parent = of_get_parent(np_parent); + + if (np_pmic_parent && + !of_node_cmp(of_node_full_name(np_pmic_parent), tps->dev->of_node->full_name) && + strcmp(npname, multi_regs->supply_name) == 0) { switch (multi) { case MULTI_BUCK12: buck_multi[0] = true; @@ -706,6 +713,10 @@ static int tps6594_regulator_probe(struct platform_device *pdev) break; } } + + of_node_put(np_pmic_parent); + of_node_put(np_parent); + of_node_put(np); } reg_irq_nb = desc->num_irq_types * (desc->num_buck_regs + desc->num_ldo_regs); diff --git a/drivers/remoteproc/qcom_common.c b/drivers/remoteproc/qcom_common.c index 5bbbe36d0964..506678d1633e 100644 --- a/drivers/remoteproc/qcom_common.c +++ b/drivers/remoteproc/qcom_common.c @@ -242,8 +242,10 @@ void qcom_add_glink_subdev(struct rproc *rproc, struct qcom_rproc_glink *glink, return; glink->ssr_name = kstrdup_const(ssr_name, GFP_KERNEL); - if (!glink->ssr_name) + if (!glink->ssr_name) { + of_node_put(glink->node); return; + } glink->dev = dev; glink->subdev.start = glink_subdev_start; diff --git a/drivers/remoteproc/qcom_q6v5_adsp.c b/drivers/remoteproc/qcom_q6v5_adsp.c index e98b7e03162c..b7d472afd5a9 100644 --- a/drivers/remoteproc/qcom_q6v5_adsp.c +++ b/drivers/remoteproc/qcom_q6v5_adsp.c @@ -355,6 +355,7 @@ static int adsp_map_carveout(struct rproc *rproc) return ret; sid = args.args[0] & SID_MASK_DEFAULT; + of_node_put(args.np); /* Add SID configuration for ADSP Firmware to SMMU */ iova = adsp->mem_phys | (sid << 32); diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c index 825672100528..bbc781e41a1b 100644 --- a/drivers/remoteproc/remoteproc_core.c +++ b/drivers/remoteproc/remoteproc_core.c @@ -1016,60 +1016,55 @@ static rproc_handle_resource_t rproc_loading_handlers[RSC_LAST] = { [RSC_VDEV] = rproc_handle_vdev, }; -/* handle firmware resource entries before booting the remote processor */ -static int rproc_handle_resources(struct rproc *rproc, - rproc_handle_resource_t handlers[RSC_LAST]) +struct rproc_rsc_cb_data { + struct rproc *rproc; + rproc_handle_resource_t *handlers; +}; + +static int rproc_handle_rsc_entry(u32 type, void *rsc, int offset, + int avail, void *data) { + struct rproc_rsc_cb_data *d = data; + struct rproc *rproc = d->rproc; struct device *dev = &rproc->dev; rproc_handle_resource_t handler; - int ret = 0, i; - - if (!rproc->table_ptr) - return 0; - - for (i = 0; i < rproc->table_ptr->num; i++) { - int offset = rproc->table_ptr->offset[i]; - struct fw_rsc_hdr *hdr = (void *)rproc->table_ptr + offset; - int avail = rproc->table_sz - offset - sizeof(*hdr); - void *rsc = (void *)hdr + sizeof(*hdr); + int ret; - /* make sure table isn't truncated */ - if (avail < 0) { - dev_err(dev, "rsc table is truncated\n"); - return -EINVAL; - } + dev_dbg(dev, "rsc: type %d\n", type); - dev_dbg(dev, "rsc: type %d\n", hdr->type); + if (type >= RSC_VENDOR_START && type <= RSC_VENDOR_END) { + ret = rproc_handle_rsc(rproc, type, rsc, offset, avail); + if (ret == RSC_HANDLED) + return 0; + if (ret < 0) + return ret; + dev_warn(dev, "unsupported vendor resource %d\n", type); + return 0; + } - if (hdr->type >= RSC_VENDOR_START && - hdr->type <= RSC_VENDOR_END) { - ret = rproc_handle_rsc(rproc, hdr->type, rsc, - offset + sizeof(*hdr), avail); - if (ret == RSC_HANDLED) - continue; - else if (ret < 0) - break; + if (type >= RSC_LAST) { + dev_warn(dev, "unsupported resource %d\n", type); + return 0; + } - dev_warn(dev, "unsupported vendor resource %d\n", - hdr->type); - continue; - } + handler = d->handlers[type]; + if (!handler) + return 0; - if (hdr->type >= RSC_LAST) { - dev_warn(dev, "unsupported resource %d\n", hdr->type); - continue; - } + return handler(rproc, rsc, offset, avail); +} - handler = handlers[hdr->type]; - if (!handler) - continue; +/* handle firmware resource entries before booting the remote processor */ +static int rproc_handle_resources(struct rproc *rproc, + rproc_handle_resource_t handlers[RSC_LAST]) +{ + struct rproc_rsc_cb_data d = { .rproc = rproc, .handlers = handlers }; - ret = handler(rproc, rsc, offset + sizeof(*hdr), avail); - if (ret) - break; - } + if (!rproc->table_ptr) + return 0; - return ret; + return rsc_table_for_each_entry(rproc->table_ptr, rproc->table_sz, + &rproc->dev, rproc_handle_rsc_entry, &d); } static int rproc_prepare_subdevices(struct rproc *rproc) @@ -1678,18 +1673,21 @@ static void rproc_auto_boot_callback(const struct firmware *fw, void *context) release_firmware(fw); } +static void rproc_attach_work(struct work_struct *work) +{ + struct rproc *rproc = container_of(work, struct rproc, attach_work); + + rproc_boot(rproc); +} + static int rproc_trigger_auto_boot(struct rproc *rproc) { int ret; - /* - * Since the remote processor is in a detached state, it has already - * been booted by another entity. As such there is no point in waiting - * for a firmware image to be loaded, we can simply initiate the process - * of attaching to it immediately. - */ - if (rproc->state == RPROC_DETACHED) - return rproc_boot(rproc); + if (rproc->state == RPROC_DETACHED) { + schedule_work(&rproc->attach_work); + return 0; + } /* * We're initiating an asynchronous firmware loading, so we can @@ -1838,6 +1836,11 @@ int rproc_trigger_recovery(struct rproc *rproc) if (ret) return ret; + if (READ_ONCE(rproc->deleting)) { + ret = -ENODEV; + goto unlock_mutex; + } + /* State could have changed before we got the mutex */ if (rproc->state != RPROC_CRASHED) goto unlock_mutex; @@ -1870,6 +1873,11 @@ static void rproc_crash_handler_work(struct work_struct *work) mutex_lock(&rproc->lock); + if (READ_ONCE(rproc->deleting)) { + mutex_unlock(&rproc->lock); + goto out; + } + if (rproc->state == RPROC_CRASHED) { /* handle only the first crash detected */ mutex_unlock(&rproc->lock); @@ -1925,9 +1933,9 @@ int rproc_boot(struct rproc *rproc) return ret; } - if (rproc->state == RPROC_DELETED) { + if (READ_ONCE(rproc->deleting)) { ret = -ENODEV; - dev_err(dev, "can't boot deleted rproc %s\n", rproc->name); + dev_err(dev, "can't boot deleting rproc %s\n", rproc->name); goto unlock_mutex; } @@ -1989,7 +1997,8 @@ EXPORT_SYMBOL(rproc_boot); int rproc_shutdown(struct rproc *rproc) { struct device *dev = &rproc->dev; - int ret = 0; + bool crashed; + int ret; ret = mutex_lock_interruptible(&rproc->lock); if (ret) { @@ -1998,16 +2007,18 @@ int rproc_shutdown(struct rproc *rproc) } if (rproc->state != RPROC_RUNNING && - rproc->state != RPROC_ATTACHED) { + rproc->state != RPROC_ATTACHED && + rproc->state != RPROC_CRASHED) { ret = -EINVAL; goto out; } + crashed = rproc->state == RPROC_CRASHED; /* if the remote proc is still needed, bail out */ if (!atomic_dec_and_test(&rproc->power)) goto out; - ret = rproc_stop(rproc, false); + ret = rproc_stop(rproc, crashed); if (ret) { atomic_inc(&rproc->power); goto out; @@ -2516,7 +2527,9 @@ struct rproc *rproc_alloc(struct device *dev, const char *name, INIT_LIST_HEAD(&rproc->subdevs); INIT_LIST_HEAD(&rproc->dump_segments); + INIT_WORK(&rproc->attach_work, rproc_attach_work); INIT_WORK(&rproc->crash_handler, rproc_crash_handler_work); + spin_lock_init(&rproc->crash_handler_lock); rproc->state = RPROC_OFFLINE; @@ -2580,16 +2593,21 @@ EXPORT_SYMBOL(rproc_put); */ int rproc_del(struct rproc *rproc) { + unsigned long flags; + if (!rproc) return -EINVAL; + spin_lock_irqsave(&rproc->crash_handler_lock, flags); + WRITE_ONCE(rproc->deleting, true); + spin_unlock_irqrestore(&rproc->crash_handler_lock, flags); + + if (cancel_work_sync(&rproc->crash_handler)) + pm_relax(rproc->dev.parent); + /* TODO: make sure this works with rproc->power > 1 */ rproc_shutdown(rproc); - mutex_lock(&rproc->lock); - rproc->state = RPROC_DELETED; - mutex_unlock(&rproc->lock); - rproc_delete_debug_dir(rproc); /* the rproc is downref'ed as soon as it's removed from the klist */ @@ -2701,18 +2719,26 @@ EXPORT_SYMBOL(rproc_get_by_child); */ void rproc_report_crash(struct rproc *rproc, enum rproc_crash_type type) { + unsigned long flags; + if (!rproc) { pr_err("NULL rproc pointer\n"); return; } + spin_lock_irqsave(&rproc->crash_handler_lock, flags); + if (READ_ONCE(rproc->deleting)) { + spin_unlock_irqrestore(&rproc->crash_handler_lock, flags); + return; + } + /* Prevent suspend while the remoteproc is being recovered */ pm_stay_awake(rproc->dev.parent); + queue_work(rproc_recovery_wq, &rproc->crash_handler); + spin_unlock_irqrestore(&rproc->crash_handler_lock, flags); dev_err(&rproc->dev, "crash detected in %s: type %s\n", rproc->name, rproc_crash_to_string(type)); - - queue_work(rproc_recovery_wq, &rproc->crash_handler); } EXPORT_SYMBOL(rproc_report_crash); diff --git a/drivers/remoteproc/remoteproc_sysfs.c b/drivers/remoteproc/remoteproc_sysfs.c index 138e752c5e4e..925b0cdbe577 100644 --- a/drivers/remoteproc/remoteproc_sysfs.c +++ b/drivers/remoteproc/remoteproc_sysfs.c @@ -168,7 +168,6 @@ static const char * const rproc_state_string[] = { [RPROC_SUSPENDED] = "suspended", [RPROC_RUNNING] = "running", [RPROC_CRASHED] = "crashed", - [RPROC_DELETED] = "deleted", [RPROC_ATTACHED] = "attached", [RPROC_DETACHED] = "detached", [RPROC_LAST] = "invalid", diff --git a/drivers/rpmsg/qcom_glink_native.c b/drivers/rpmsg/qcom_glink_native.c index 833ff9cb8afe..7605fde9e678 100644 --- a/drivers/rpmsg/qcom_glink_native.c +++ b/drivers/rpmsg/qcom_glink_native.c @@ -1395,26 +1395,28 @@ static int qcom_glink_announce_create(struct rpmsg_device *rpdev) return 0; } -static void qcom_glink_destroy_ept(struct rpmsg_endpoint *ept) +static void qcom_glink_remove_rpmsg_device(struct qcom_glink *glink, struct glink_channel *channel) { - struct glink_channel *channel = to_glink_channel(ept); - struct qcom_glink *glink = channel->glink; struct rpmsg_channel_info chinfo; - unsigned long flags; - - spin_lock_irqsave(&channel->recv_lock, flags); - channel->ept.cb = NULL; - spin_unlock_irqrestore(&channel->recv_lock, flags); - /* Decouple the potential rpdev from the channel */ if (channel->rpdev) { strscpy_pad(chinfo.name, channel->name, sizeof(chinfo.name)); chinfo.src = RPMSG_ADDR_ANY; chinfo.dst = RPMSG_ADDR_ANY; - rpmsg_unregister_device(glink->dev, &chinfo); } channel->rpdev = NULL; +} + +static void qcom_glink_destroy_ept(struct rpmsg_endpoint *ept) +{ + struct glink_channel *channel = to_glink_channel(ept); + struct qcom_glink *glink = channel->glink; + unsigned long flags; + + spin_lock_irqsave(&channel->recv_lock, flags); + channel->ept.cb = NULL; + spin_unlock_irqrestore(&channel->recv_lock, flags); qcom_glink_send_close_req(glink, channel); } @@ -1704,7 +1706,6 @@ static int qcom_glink_rx_open(struct qcom_glink *glink, unsigned int rcid, static void qcom_glink_rx_close(struct qcom_glink *glink, unsigned int rcid) { - struct rpmsg_channel_info chinfo; struct glink_channel *channel; unsigned long flags; @@ -1720,14 +1721,7 @@ static void qcom_glink_rx_close(struct qcom_glink *glink, unsigned int rcid) /* cancel pending rx_done work */ cancel_work_sync(&channel->intent_work); - if (channel->rpdev) { - strscpy_pad(chinfo.name, channel->name, sizeof(chinfo.name)); - chinfo.src = RPMSG_ADDR_ANY; - chinfo.dst = RPMSG_ADDR_ANY; - - rpmsg_unregister_device(glink->dev, &chinfo); - } - channel->rpdev = NULL; + qcom_glink_remove_rpmsg_device(glink, channel); qcom_glink_send_close_ack(glink, channel); @@ -1741,7 +1735,6 @@ static void qcom_glink_rx_close(struct qcom_glink *glink, unsigned int rcid) static void qcom_glink_rx_close_ack(struct qcom_glink *glink, unsigned int lcid) { - struct rpmsg_channel_info chinfo; struct glink_channel *channel; unsigned long flags; @@ -1763,14 +1756,7 @@ static void qcom_glink_rx_close_ack(struct qcom_glink *glink, unsigned int lcid) spin_unlock_irqrestore(&glink->idr_lock, flags); /* Decouple the potential rpdev from the channel */ - if (channel->rpdev) { - strscpy(chinfo.name, channel->name, sizeof(chinfo.name)); - chinfo.src = RPMSG_ADDR_ANY; - chinfo.dst = RPMSG_ADDR_ANY; - - rpmsg_unregister_device(glink->dev, &chinfo); - } - channel->rpdev = NULL; + qcom_glink_remove_rpmsg_device(glink, channel); kref_put(&channel->refcount, qcom_glink_channel_release); } diff --git a/drivers/rtc/rtc-gamecube.c b/drivers/rtc/rtc-gamecube.c index 045d5d45ab4b..889028cecf4a 100644 --- a/drivers/rtc/rtc-gamecube.c +++ b/drivers/rtc/rtc-gamecube.c @@ -355,9 +355,7 @@ static int gamecube_rtc_probe(struct platform_device *pdev) rtc->range_max = U32_MAX; rtc->ops = &gamecube_rtc_ops; - devm_rtc_register_device(rtc); - - return 0; + return devm_rtc_register_device(rtc); } static const struct of_device_id gamecube_rtc_of_match[] = { diff --git a/drivers/rtc/rtc-pcf85363.c b/drivers/rtc/rtc-pcf85363.c index 540042b9eec8..ccc7834e5759 100644 --- a/drivers/rtc/rtc-pcf85363.c +++ b/drivers/rtc/rtc-pcf85363.c @@ -426,8 +426,8 @@ static int pcf85363_probe(struct i2c_client *client) err = pcf85363_load_capacitance(pcf85363, client->dev.of_node); if (err < 0) - dev_warn(&client->dev, "failed to set xtal load capacitance: %d", - err); + return dev_err_probe(&client->dev, err, + "failed to set xtal load capacitance\n"); pcf85363->rtc->ops = &rtc_ops; pcf85363->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000; @@ -436,9 +436,16 @@ static int pcf85363_probe(struct i2c_client *client) wakeup_source = device_property_read_bool(&client->dev, "wakeup-source"); if (client->irq > 0 || wakeup_source) { - regmap_write(pcf85363->regmap, CTRL_FLAGS, 0); - regmap_update_bits(pcf85363->regmap, CTRL_PIN_IO, - PIN_IO_INTAPM, PIN_IO_INTA_OUT); + err = regmap_write(pcf85363->regmap, CTRL_FLAGS, 0); + if (err) + return dev_err_probe(&client->dev, err, + "failed to clear flags\n"); + + err = regmap_update_bits(pcf85363->regmap, CTRL_PIN_IO, + PIN_IO_INTAPM, PIN_IO_INTA_OUT); + if (err) + return dev_err_probe(&client->dev, err, + "failed to set interrupt pin mode\n"); } if (client->irq > 0) { diff --git a/drivers/rtc/rtc-pcf8563.c b/drivers/rtc/rtc-pcf8563.c index b281e9489df1..7083d927820f 100644 --- a/drivers/rtc/rtc-pcf8563.c +++ b/drivers/rtc/rtc-pcf8563.c @@ -449,7 +449,9 @@ static struct clk *pcf8563_clkout_register_clk(struct pcf8563 *pcf8563) clk = devm_clk_register(&pcf8563->rtc->dev, &pcf8563->clkout_hw); if (!IS_ERR(clk)) - of_clk_add_provider(node, of_clk_src_simple_get, clk); + devm_of_clk_add_hw_provider(pcf8563->rtc->dev.parent, + of_clk_hw_simple_get, + &pcf8563->clkout_hw); return clk; } diff --git a/drivers/rtc/rtc-spacemit-p1.c b/drivers/rtc/rtc-spacemit-p1.c index 43ab62494bb4..1de7bd995d29 100644 --- a/drivers/rtc/rtc-spacemit-p1.c +++ b/drivers/rtc/rtc-spacemit-p1.c @@ -57,8 +57,9 @@ static int p1_rtc_read_time(struct device *dev, struct rtc_time *t) u8 time[6]; int ret; - if (!regmap_test_bits(regmap, RTC_CTRL, RTC_EN)) - return -EINVAL; /* RTC is disabled */ + ret = regmap_test_bits(regmap, RTC_CTRL, RTC_EN); + if (ret <= 0) + return ret ?: -EINVAL; /* RTC is disabled or error */ ret = regmap_bulk_read(regmap, RTC_TIME, time, sizeof(time)); if (ret) diff --git a/drivers/rtc/rtc-zynqmp.c b/drivers/rtc/rtc-zynqmp.c index 856bc1678e7d..bf2ecd0dfe23 100644 --- a/drivers/rtc/rtc-zynqmp.c +++ b/drivers/rtc/rtc-zynqmp.c @@ -335,10 +335,9 @@ static int xlnx_rtc_probe(struct platform_device *pdev) /* Getting the rtc info */ xrtcdev->rtc_clk = devm_clk_get_optional(&pdev->dev, "rtc"); - if (IS_ERR(xrtcdev->rtc_clk)) { - if (PTR_ERR(xrtcdev->rtc_clk) != -EPROBE_DEFER) - dev_warn(&pdev->dev, "Device clock not found.\n"); - } + if (IS_ERR(xrtcdev->rtc_clk)) + return dev_err_probe(&pdev->dev, PTR_ERR(xrtcdev->rtc_clk), + "Failed to get rtc clock\n"); xrtcdev->freq = clk_get_rate(xrtcdev->rtc_clk); if (!xrtcdev->freq) { ret = of_property_read_u32(pdev->dev.of_node, "calibration", diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 0d652db8fe24..55888ac6ed3a 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -5871,6 +5871,8 @@ _base_release_memory_pools(struct MPT3SAS_ADAPTER *ioc) if (ioc->pcie_sgl_dma_pool) { for (i = 0; i < ioc->scsiio_depth; i++) { + if (!ioc->pcie_sg_lookup[i].pcie_sgl) + continue; dma_pool_free(ioc->pcie_sgl_dma_pool, ioc->pcie_sg_lookup[i].pcie_sgl, ioc->pcie_sg_lookup[i].pcie_sgl_dma); diff --git a/drivers/scsi/qla2xxx/qla_attr.c b/drivers/scsi/qla2xxx/qla_attr.c index b103e3b1056a..0768b3aad50b 100644 --- a/drivers/scsi/qla2xxx/qla_attr.c +++ b/drivers/scsi/qla2xxx/qla_attr.c @@ -580,7 +580,6 @@ qla2x00_sysfs_read_vpd(struct file *filp, struct kobject *kobj, ha->isp_ops->read_optrom(vha, ha->vpd, faddr, ha->vpd_size); mutex_unlock(&ha->optrom_mutex); - ha->isp_ops->read_optrom(vha, ha->vpd, faddr, ha->vpd_size); skip: return memory_read_from_buffer(buf, count, &off, ha->vpd, ha->vpd_size); } diff --git a/drivers/scsi/qla2xxx/qla_nx2.c b/drivers/scsi/qla2xxx/qla_nx2.c index 41ff6fbdb933..04d7ab6ad037 100644 --- a/drivers/scsi/qla2xxx/qla_nx2.c +++ b/drivers/scsi/qla2xxx/qla_nx2.c @@ -3507,7 +3507,7 @@ qla8044_poll_flash_status_reg(struct scsi_qla_host *vha) msleep(QLA8044_FLASH_STATUS_REG_POLL_DELAY); } - if (!retries) + if (retries == -1) ret_val = QLA_FUNCTION_FAILED; return ret_val; diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index d9bae23cb929..41ad0e9101b8 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -2328,8 +2328,8 @@ static int sd_done(struct scsi_cmnd *SCpnt) } sdkp->medium_access_timed_out = 0; - if (!scsi_status_is_check_condition(result) && - (!sense_valid || sense_deferred)) + if (!scsi_status_is_check_condition(result) || + !sense_valid || sense_deferred) goto out; switch (sshdr.sense_key) { diff --git a/drivers/scsi/smartpqi/smartpqi_init.c b/drivers/scsi/smartpqi/smartpqi_init.c index 090e786470b3..88cffb1fb59c 100644 --- a/drivers/scsi/smartpqi/smartpqi_init.c +++ b/drivers/scsi/smartpqi/smartpqi_init.c @@ -66,6 +66,12 @@ static struct pqi_cmd_priv *pqi_cmd_priv(struct scsi_cmnd *cmd) return scsi_cmd_priv(cmd); } +static int pqi_init_cmd_priv(struct Scsi_Host *shost, struct scsi_cmnd *cmd) +{ + memset(pqi_cmd_priv(cmd), 0, sizeof(struct pqi_cmd_priv)); + return 0; +} + static void pqi_verify_structures(void); static void pqi_take_ctrl_offline(struct pqi_ctrl_info *ctrl_info, enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason); @@ -5943,6 +5949,17 @@ void pqi_prep_for_scsi_done(struct scsi_cmnd *scmd) struct pqi_scsi_dev *device; struct completion *wait; + /* + * Clear the AIO-retry marker on final completion so the tag + * starts clean on its next dispatch. On DID_IMM_RETRY leave + * it intact: pqi_aio_io_complete() sets DID_IMM_RETRY and + * bumps the marker to steer the requeue onto the RAID path, + * and pqi_process_raid_io_error() consumes the non-zero + * marker to offline a misbehaving drive. + */ + if (host_byte(scmd->result) != DID_IMM_RETRY) + pqi_cmd_priv(scmd)->this_residual = 0; + if (!scmd->device) { set_host_byte(scmd, DID_NO_CONNECT); return; @@ -7596,6 +7613,7 @@ static const struct scsi_host_template pqi_driver_template = { .sdev_groups = pqi_sdev_groups, .shost_groups = pqi_shost_groups, .cmd_size = sizeof(struct pqi_cmd_priv), + .init_cmd_priv = pqi_init_cmd_priv, }; static int pqi_register_scsi(struct pqi_ctrl_info *ctrl_info) diff --git a/drivers/sh/clk/core.c b/drivers/sh/clk/core.c index 7a73f5e4a1fc..f02e12dfa5f6 100644 --- a/drivers/sh/clk/core.c +++ b/drivers/sh/clk/core.c @@ -569,7 +569,7 @@ long clk_round_rate(struct clk *clk, unsigned long rate) EXPORT_SYMBOL_GPL(clk_round_rate); #ifdef CONFIG_PM -static void clks_core_resume(void) +static void clks_core_resume(void *data) { struct clk *clkp; @@ -588,13 +588,17 @@ static void clks_core_resume(void) } } -static struct syscore_ops clks_syscore_ops = { +static const struct syscore_ops clks_syscore_ops = { .resume = clks_core_resume, }; +static struct syscore clks_syscore = { + .ops = &clks_syscore_ops, +}; + static int __init clk_syscore_init(void) { - register_syscore_ops(&clks_syscore_ops); + register_syscore(&clks_syscore); return 0; } diff --git a/drivers/sh/intc/core.c b/drivers/sh/intc/core.c index ea571eeb3078..3dde703b7766 100644 --- a/drivers/sh/intc/core.c +++ b/drivers/sh/intc/core.c @@ -394,7 +394,7 @@ int __init register_intc_controller(struct intc_desc *desc) return -ENOMEM; } -static int intc_suspend(void) +static int intc_suspend(void *data) { struct intc_desc_int *d; @@ -420,7 +420,7 @@ static int intc_suspend(void) return 0; } -static void intc_resume(void) +static void intc_resume(void *data) { struct intc_desc_int *d; @@ -450,11 +450,15 @@ static void intc_resume(void) } } -struct syscore_ops intc_syscore_ops = { +static const struct syscore_ops intc_syscore_ops = { .suspend = intc_suspend, .resume = intc_resume, }; +static struct syscore intc_syscore = { + .ops = &intc_syscore_ops, +}; + const struct bus_type intc_subsys = { .name = "intc", .dev_name = "intc", @@ -477,7 +481,7 @@ static int __init register_intc_devs(void) struct intc_desc_int *d; int error; - register_syscore_ops(&intc_syscore_ops); + register_syscore(&intc_syscore); error = subsys_system_register(&intc_subsys, NULL); if (!error) { diff --git a/drivers/soc/bcm/brcmstb/biuctrl.c b/drivers/soc/bcm/brcmstb/biuctrl.c index 364ddbe365c2..bd830649b60d 100644 --- a/drivers/soc/bcm/brcmstb/biuctrl.c +++ b/drivers/soc/bcm/brcmstb/biuctrl.c @@ -298,7 +298,7 @@ static int __init setup_hifcpubiuctrl_regs(struct device_node *np) #ifdef CONFIG_PM_SLEEP static u32 cpubiuctrl_reg_save[NUM_CPU_BIUCTRL_REGS]; -static int brcmstb_cpu_credit_reg_suspend(void) +static int brcmstb_cpu_credit_reg_suspend(void *data) { unsigned int i; @@ -311,7 +311,7 @@ static int brcmstb_cpu_credit_reg_suspend(void) return 0; } -static void brcmstb_cpu_credit_reg_resume(void) +static void brcmstb_cpu_credit_reg_resume(void *data) { unsigned int i; @@ -322,10 +322,14 @@ static void brcmstb_cpu_credit_reg_resume(void) cbc_writel(cpubiuctrl_reg_save[i], i); } -static struct syscore_ops brcmstb_cpu_credit_syscore_ops = { +static const struct syscore_ops brcmstb_cpu_credit_syscore_ops = { .suspend = brcmstb_cpu_credit_reg_suspend, .resume = brcmstb_cpu_credit_reg_resume, }; + +static struct syscore brcmstb_cpu_credit_syscore = { + .ops = &brcmstb_cpu_credit_syscore_ops, +}; #endif @@ -354,7 +358,7 @@ static int __init brcmstb_biuctrl_init(void) a72_b53_rac_enable_all(np); mcp_a72_b53_set(); #ifdef CONFIG_PM_SLEEP - register_syscore_ops(&brcmstb_cpu_credit_syscore_ops); + register_syscore(&brcmstb_cpu_credit_syscore); #endif ret = 0; out_put: diff --git a/drivers/soc/fsl/qe/gpio.c b/drivers/soc/fsl/qe/gpio.c index c54154b404df..e6ef71372654 100644 --- a/drivers/soc/fsl/qe/gpio.c +++ b/drivers/soc/fsl/qe/gpio.c @@ -135,6 +135,30 @@ static int qe_gpio_dir_out(struct gpio_chip *gc, unsigned int gpio, int val) return 0; } +static int qe_gpio_get_direction(struct gpio_chip *gc, unsigned int gpio) +{ + struct qe_gpio_chip *qe_gc = gpiochip_get_data(gc); + struct qe_pio_regs __iomem *regs = qe_gc->regs; + unsigned long flags; + u32 val, mask; + + spin_lock_irqsave(&qe_gc->lock, flags); + + if (gpio < QE_PIO_PINS / 2) + val = ioread32be(®s->cpdir1); + else + val = ioread32be(®s->cpdir2); + + spin_unlock_irqrestore(&qe_gc->lock, flags); + + mask = (u32)QE_PIO_DIR_OUT << (QE_PIO_PINS - 2 - (gpio % (QE_PIO_PINS / 2)) * 2); + + if (val & mask) + return GPIO_LINE_DIRECTION_OUT; + else + return GPIO_LINE_DIRECTION_IN; +} + struct qe_pin { /* * The qe_gpio_chip name is unfortunate, we should change that to @@ -308,6 +332,7 @@ static int qe_gpio_probe(struct platform_device *ofdev) gc->ngpio = QE_PIO_PINS; gc->direction_input = qe_gpio_dir_in; gc->direction_output = qe_gpio_dir_out; + gc->get_direction = qe_gpio_get_direction; gc->get = qe_gpio_get; gc->set = qe_gpio_set; gc->set_multiple = qe_gpio_set_multiple; diff --git a/drivers/soc/fsl/qe/qe_ic.c b/drivers/soc/fsl/qe/qe_ic.c index 943911053af6..bccc7d612009 100644 --- a/drivers/soc/fsl/qe/qe_ic.c +++ b/drivers/soc/fsl/qe/qe_ic.c @@ -473,7 +473,6 @@ static struct platform_driver qe_ic_driver = static int __init qe_ic_of_init(void) { - platform_driver_register(&qe_ic_driver); - return 0; + return platform_driver_register(&qe_ic_driver); } subsys_initcall(qe_ic_of_init); diff --git a/drivers/soc/fsl/qe/qe_io.c b/drivers/soc/fsl/qe/qe_io.c index a5e2d0e5ab51..150913fce981 100644 --- a/drivers/soc/fsl/qe/qe_io.c +++ b/drivers/soc/fsl/qe/qe_io.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -184,3 +185,17 @@ int par_io_of_config(struct device_node *np) return 0; } EXPORT_SYMBOL(par_io_of_config); + +static int __init par_io_populate(void) +{ + struct device_node *np = of_find_node_by_type(NULL, "par_io"); + + if (!np) + return 0; + + of_platform_default_populate(np, NULL, NULL); + of_node_put(np); + + return 0; +} +arch_initcall(par_io_populate); diff --git a/drivers/soc/qcom/rpmh-rsc.c b/drivers/soc/qcom/rpmh-rsc.c index c6f7d5c9c493..66928ca40b9a 100644 --- a/drivers/soc/qcom/rpmh-rsc.c +++ b/drivers/soc/qcom/rpmh-rsc.c @@ -944,17 +944,30 @@ static int rpmh_rsc_pd_callback(struct notifier_block *nfb, return NOTIFY_OK; } +static void rpmh_rsc_pd_detach(void *data) +{ + dev_pm_genpd_remove_notifier(data); +} + static int rpmh_rsc_pd_attach(struct rsc_drv *drv, struct device *dev) { int ret; - pm_runtime_enable(dev); + ret = devm_pm_runtime_enable(dev); + if (ret) + return ret; + drv->genpd_nb.notifier_call = rpmh_rsc_pd_callback; ret = dev_pm_genpd_add_notifier(dev, &drv->genpd_nb); if (ret) - pm_runtime_disable(dev); + return ret; - return ret; + return devm_add_action_or_reset(dev, rpmh_rsc_pd_detach, dev); +} + +static void rpmh_rsc_cpu_pm_unregister(void *data) +{ + cpu_pm_unregister_notifier(data); } static int rpmh_probe_tcs_config(struct platform_device *pdev, struct rsc_drv *drv) @@ -1107,7 +1120,15 @@ static int rpmh_rsc_probe(struct platform_device *pdev) return ret; } else { drv->rsc_pm.notifier_call = rpmh_rsc_cpu_pm_callback; - cpu_pm_register_notifier(&drv->rsc_pm); + ret = cpu_pm_register_notifier(&drv->rsc_pm); + if (ret) + return ret; + + ret = devm_add_action_or_reset(&pdev->dev, + rpmh_rsc_cpu_pm_unregister, + &drv->rsc_pm); + if (ret) + return ret; } } @@ -1122,13 +1143,7 @@ static int rpmh_rsc_probe(struct platform_device *pdev) dev_set_drvdata(&pdev->dev, drv); drv->dev = &pdev->dev; - ret = devm_of_platform_populate(&pdev->dev); - if (ret && pdev->dev.pm_domain) { - dev_pm_genpd_remove_notifier(&pdev->dev); - pm_runtime_disable(&pdev->dev); - } - - return ret; + return devm_of_platform_populate(&pdev->dev); } static const struct of_device_id rpmh_drv_match[] = { diff --git a/drivers/soc/renesas/Kconfig b/drivers/soc/renesas/Kconfig index 340a1ff7e92b..f151b05fbb8f 100644 --- a/drivers/soc/renesas/Kconfig +++ b/drivers/soc/renesas/Kconfig @@ -355,7 +355,6 @@ config ARCH_R8A779H0 config ARCH_R8A78000 bool "ARM64 Platform support for R8A78000 (R-Car X5H)" default y if ARCH_RENESAS - default ARCH_RENESAS select ARCH_RCAR_GEN5 help This enables support for the Renesas R-Car X5H SoC. diff --git a/drivers/soc/samsung/exynos-pmu.c b/drivers/soc/samsung/exynos-pmu.c index f8fe1a5965ab..d91049973113 100644 --- a/drivers/soc/samsung/exynos-pmu.c +++ b/drivers/soc/samsung/exynos-pmu.c @@ -300,8 +300,8 @@ static const struct mfd_cell exynos_pmu_devs[] = { */ struct regmap *exynos_get_pmu_regmap(void) { - struct device_node *np = of_find_matching_node(NULL, - exynos_pmu_of_device_ids); + struct device_node *np __free(device_node) = + of_find_matching_node(NULL, exynos_pmu_of_device_ids); if (np) return exynos_get_pmu_regmap_by_phandle(np, NULL); return ERR_PTR(-ENODEV); diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c index 034a2a535a1e..93bbebd68001 100644 --- a/drivers/soc/tegra/pmc.c +++ b/drivers/soc/tegra/pmc.c @@ -466,7 +466,7 @@ struct tegra_pmc { unsigned long *wake_type_dual_edge_map; unsigned long *wake_sw_status_map; unsigned long *wake_cntrl_level_map; - struct syscore_ops syscore; + struct syscore syscore; }; static struct tegra_pmc *pmc = &(struct tegra_pmc) { @@ -3147,7 +3147,7 @@ static void tegra186_pmc_process_wake_events(struct tegra_pmc *pmc, unsigned int } } -static void tegra186_pmc_wake_syscore_resume(void) +static void tegra186_pmc_wake_syscore_resume(void *data) { u32 status, mask; unsigned int i; @@ -3160,7 +3160,7 @@ static void tegra186_pmc_wake_syscore_resume(void) } } -static int tegra186_pmc_wake_syscore_suspend(void) +static int tegra186_pmc_wake_syscore_suspend(void *data) { wke_read_sw_wake_status(pmc); @@ -3179,6 +3179,11 @@ static int tegra186_pmc_wake_syscore_suspend(void) return 0; } +static const struct syscore_ops tegra186_pmc_wake_syscore_ops = { + .suspend = tegra186_pmc_wake_syscore_suspend, + .resume = tegra186_pmc_wake_syscore_resume, +}; + #if defined(CONFIG_PM_SLEEP) && defined(CONFIG_ARM) static int tegra_pmc_suspend(struct device *dev) { @@ -3829,10 +3834,8 @@ static const struct tegra_pmc_regs tegra186_pmc_regs = { static void tegra186_pmc_init(struct tegra_pmc *pmc) { - pmc->syscore.suspend = tegra186_pmc_wake_syscore_suspend; - pmc->syscore.resume = tegra186_pmc_wake_syscore_resume; - - register_syscore_ops(&pmc->syscore); + pmc->syscore.ops = &tegra186_pmc_wake_syscore_ops; + register_syscore(&pmc->syscore); } static void tegra186_pmc_setup_irq_polarity(struct tegra_pmc *pmc, diff --git a/drivers/soundwire/bus.c b/drivers/soundwire/bus.c index 14e1351a3f8a..45d786f379d8 100644 --- a/drivers/soundwire/bus.c +++ b/drivers/soundwire/bus.c @@ -1372,6 +1372,37 @@ int sdw_slave_get_current_bank(struct sdw_slave *slave) } EXPORT_SYMBOL_GPL(sdw_slave_get_current_bank); +/** + * sdw_slave_wait_for_init - Wait for device initialisation + * @slave: Pointer to the SoundWire peripheral. + * @timeout_ms: Timeout in milliseconds. + * + * Wait for a peripheral device to enumerate and be initialised by the + * SoundWire core. + * + * Return: Zero on success, and a negative error code on failure. + */ +int sdw_slave_wait_for_init(struct sdw_slave *slave, int timeout_ms) +{ + unsigned long time; + + if (!slave->unattach_request) + return 0; + + time = wait_for_completion_timeout(&slave->initialization_complete, + msecs_to_jiffies(timeout_ms)); + if (!time) { + dev_err(&slave->dev, "Initialization not complete\n"); + sdw_show_ping_status(slave->bus, true); + return -ETIMEDOUT; + } + + slave->unattach_request = 0; + + return 0; +} +EXPORT_SYMBOL_GPL(sdw_slave_wait_for_init); + static int sdw_slave_set_frequency(struct sdw_slave *slave) { int scale_index; diff --git a/drivers/soundwire/qcom.c b/drivers/soundwire/qcom.c index 5b3078220189..a4f108fdffdd 100644 --- a/drivers/soundwire/qcom.c +++ b/drivers/soundwire/qcom.c @@ -1201,7 +1201,7 @@ static int qcom_swrm_stream_alloc_ports(struct qcom_swrm_ctrl *ctrl, else pn = find_first_zero_bit(port_mask, maxport); - if (pn > maxport) { + if (pn >= maxport) { dev_err(ctrl->dev, "All ports busy\n"); ret = -EBUSY; goto out; diff --git a/drivers/spi/spi-amlogic-spisg.c b/drivers/spi/spi-amlogic-spisg.c index 970fddbb2585..8ddb7a1a4056 100644 --- a/drivers/spi/spi-amlogic-spisg.c +++ b/drivers/spi/spi-amlogic-spisg.c @@ -640,7 +640,7 @@ static int aml_spisg_target_abort(struct spi_controller *ctlr) static int aml_spisg_clk_init(struct spisg_device *spisg, void __iomem *base) { struct device *dev = &spisg->pdev->dev; - struct clk_init_data init; + struct clk_init_data init = {}; struct clk_divider *div; struct clk_div_table *tbl; char name[32]; diff --git a/drivers/spi/spi-davinci.c b/drivers/spi/spi-davinci.c index a29934422356..c0963483e05c 100644 --- a/drivers/spi/spi-davinci.c +++ b/drivers/spi/spi-davinci.c @@ -868,7 +868,7 @@ static int davinci_spi_probe(struct platform_device *pdev) int ret = 0; u32 spipc0; - host = spi_alloc_host(&pdev->dev, sizeof(struct davinci_spi)); + host = devm_spi_alloc_host(&pdev->dev, sizeof(struct davinci_spi)); if (host == NULL) { ret = -ENOMEM; goto err; @@ -998,7 +998,6 @@ static int davinci_spi_probe(struct platform_device *pdev) dma_release_channel(dspi->dma_tx); } free_host: - spi_controller_put(host); err: return ret; } @@ -1022,6 +1021,8 @@ static void davinci_spi_remove(struct platform_device *pdev) spi_bitbang_stop(&dspi->bitbang); + devm_free_irq(&pdev->dev, dspi->irq, dspi); + /* This bit needs to be cleared to disable dpsi->clk */ clear_io_bits(dspi->base + SPIGCR1, SPIGCR1_POWERDOWN_MASK); @@ -1029,8 +1030,6 @@ static void davinci_spi_remove(struct platform_device *pdev) dma_release_channel(dspi->dma_rx); dma_release_channel(dspi->dma_tx); } - - spi_controller_put(host); } static struct platform_driver davinci_spi_driver = { diff --git a/drivers/spi/spi-geni-qcom.c b/drivers/spi/spi-geni-qcom.c index 736120107184..ec40fcb1067b 100644 --- a/drivers/spi/spi-geni-qcom.c +++ b/drivers/spi/spi-geni-qcom.c @@ -657,25 +657,30 @@ static int spi_geni_init(struct spi_geni_master *mas) u32 spi_tx_cfg, fifo_disable; int ret = -ENXIO; - pm_runtime_get_sync(mas->dev); + PM_RUNTIME_ACQUIRE_IF_ENABLED(mas->dev, pm); + ret = PM_RUNTIME_ACQUIRE_ERR(&pm); + if (ret < 0) { + dev_err(mas->dev, "Failed to resume and get %d\n", ret); + return ret; + } proto = geni_se_read_proto(se); if (spi->target) { if (proto != GENI_SE_SPI_SLAVE) { dev_err(mas->dev, "Invalid proto %d\n", proto); - goto out_pm; + return -EINVAL; } spi_slv_setup(mas); } else if (proto == GENI_SE_INVALID_PROTO) { ret = geni_load_se_firmware(se, GENI_SE_SPI); if (ret) { dev_err(mas->dev, "spi master firmware load failed ret: %d\n", ret); - goto out_pm; + return ret; } } else if (proto != GENI_SE_SPI) { dev_err(mas->dev, "Invalid proto %d\n", proto); - goto out_pm; + return -EINVAL; } mas->tx_fifo_depth = geni_se_get_tx_fifo_depth(se); @@ -708,7 +713,7 @@ static int spi_geni_init(struct spi_geni_master *mas) dev_dbg(mas->dev, "Using GPI DMA mode for SPI\n"); break; } else if (ret == -EPROBE_DEFER) { - goto out_pm; + return ret; } /* * in case of failure to get gpi dma channel, we can still do the @@ -737,8 +742,6 @@ static int spi_geni_init(struct spi_geni_master *mas) writel(spi_tx_cfg, se->base + SE_SPI_TRANS_CFG); } -out_pm: - pm_runtime_put(mas->dev); return ret; } diff --git a/drivers/spi/spi-img-spfi.c b/drivers/spi/spi-img-spfi.c index a6d15b9ca509..4f3683a07555 100644 --- a/drivers/spi/spi-img-spfi.c +++ b/drivers/spi/spi-img-spfi.c @@ -616,7 +616,7 @@ static int img_spfi_probe(struct platform_device *pdev) ret = PTR_ERR(spfi->tx_ch); spfi->tx_ch = NULL; if (ret == -EPROBE_DEFER) - goto disable_pm; + goto free_dma; } spfi->rx_ch = dma_request_chan(spfi->dev, "rx"); @@ -624,7 +624,7 @@ static int img_spfi_probe(struct platform_device *pdev) ret = PTR_ERR(spfi->rx_ch); spfi->rx_ch = NULL; if (ret == -EPROBE_DEFER) - goto disable_pm; + goto free_dma; } if (!spfi->tx_ch || !spfi->rx_ch) { @@ -652,6 +652,7 @@ static int img_spfi_probe(struct platform_device *pdev) disable_pm: pm_runtime_disable(spfi->dev); +free_dma: if (spfi->rx_ch) dma_release_channel(spfi->rx_ch); if (spfi->tx_ch) diff --git a/drivers/spi/spi-oc-tiny.c b/drivers/spi/spi-oc-tiny.c index cba229920357..cc5bceb1cc88 100644 --- a/drivers/spi/spi-oc-tiny.c +++ b/drivers/spi/spi-oc-tiny.c @@ -211,11 +211,11 @@ static int tiny_spi_probe(struct platform_device *pdev) struct tiny_spi_platform_data *platp = dev_get_platdata(&pdev->dev); struct tiny_spi *hw; struct spi_controller *host; - int err = -ENODEV; + int err; - host = spi_alloc_host(&pdev->dev, sizeof(struct tiny_spi)); + host = devm_spi_alloc_host(&pdev->dev, sizeof(struct tiny_spi)); if (!host) - return err; + return -ENOMEM; /* setup the host state. */ host->bus_num = pdev->id; @@ -233,10 +233,8 @@ static int tiny_spi_probe(struct platform_device *pdev) /* find and map our resources */ hw->base = devm_platform_ioremap_resource(pdev, 0); - if (IS_ERR(hw->base)) { - err = PTR_ERR(hw->base); - goto exit; - } + if (IS_ERR(hw->base)) + return PTR_ERR(hw->base); /* irq is optional */ hw->irq = platform_get_irq(pdev, 0); if (hw->irq >= 0) { @@ -244,7 +242,7 @@ static int tiny_spi_probe(struct platform_device *pdev) err = devm_request_irq(&pdev->dev, hw->irq, tiny_spi_irq, 0, pdev->name, hw); if (err) - goto exit; + return err; } /* find platform data */ if (platp) { @@ -253,29 +251,23 @@ static int tiny_spi_probe(struct platform_device *pdev) } else { err = tiny_spi_of_probe(pdev); if (err) - goto exit; + return err; } /* register our spi controller */ err = spi_bitbang_start(&hw->bitbang); if (err) - goto exit; + return err; dev_info(&pdev->dev, "base %p, irq %d\n", hw->base, hw->irq); return 0; - -exit: - spi_controller_put(host); - return err; } static void tiny_spi_remove(struct platform_device *pdev) { struct tiny_spi *hw = platform_get_drvdata(pdev); - struct spi_controller *host = hw->bitbang.ctlr; spi_bitbang_stop(&hw->bitbang); - spi_controller_put(host); } #ifdef CONFIG_OF diff --git a/drivers/spi/spi-sprd-adi.c b/drivers/spi/spi-sprd-adi.c index f25b34a91756..72f2f30e657e 100644 --- a/drivers/spi/spi-sprd-adi.c +++ b/drivers/spi/spi-sprd-adi.c @@ -553,6 +553,12 @@ static int sprd_adi_probe(struct platform_device *pdev) return -ENXIO; } else { switch (ret) { + case 0: + /* + * Only reachable with CONFIG_HWSPINLOCK=n, where the + * of_hwspin_lock_get_id() stub returns 0. + */ + fallthrough; case -ENOENT: dev_info(&pdev->dev, "no hardware spinlock supplied\n"); break; diff --git a/drivers/staging/fbtft/fbtft-sysfs.c b/drivers/staging/fbtft/fbtft-sysfs.c index e45c90a03a90..7bd6cbf2f1e5 100644 --- a/drivers/staging/fbtft/fbtft-sysfs.c +++ b/drivers/staging/fbtft/fbtft-sysfs.c @@ -98,7 +98,7 @@ sprintf_gamma(struct fbtft_par *par, u32 *curves, char *buf) mutex_lock(&par->gamma.lock); for (i = 0; i < par->gamma.num_curves; i++) { for (j = 0; j < par->gamma.num_values; j++) - len += scnprintf(&buf[len], PAGE_SIZE, + len += sysfs_emit_at(buf, len, "%04x ", curves[i * par->gamma.num_values + j]); buf[len - 1] = '\n'; } diff --git a/drivers/staging/gpib/common/gpib_os.c b/drivers/staging/gpib/common/gpib_os.c index a2bed6bd757a..b223bd23a842 100644 --- a/drivers/staging/gpib/common/gpib_os.c +++ b/drivers/staging/gpib/common/gpib_os.c @@ -289,18 +289,19 @@ int autopoll_all_devices(struct gpib_board *board) dev_dbg(board->gpib_dev, "autopoll has board lock\n"); retval = serial_poll_all(board, serial_timeout); - if (retval < 0) { - mutex_unlock(&board->big_gpib_mutex); - mutex_unlock(&board->user_mutex); - return retval; + if (retval >= 0) { + dev_dbg(board->gpib_dev, "complete\n"); + /* + * need to wake wait queue in case someone is + * waiting on RQS + */ + wake_up_interruptible(&board->wait); } - dev_dbg(board->gpib_dev, "complete\n"); - /* - * need to wake wait queue in case someone is - * waiting on RQS - */ - wake_up_interruptible(&board->wait); + if (retval <= 0) { + atomic_set(&board->stuck_srq, 1); + set_bit(SRQI_NUM, &board->status); + } mutex_unlock(&board->big_gpib_mutex); mutex_unlock(&board->user_mutex); diff --git a/drivers/staging/gpib/common/iblib.c b/drivers/staging/gpib/common/iblib.c index b672dd6aad25..511e1d61c1fb 100644 --- a/drivers/staging/gpib/common/iblib.c +++ b/drivers/staging/gpib/common/iblib.c @@ -193,9 +193,6 @@ static int autospoll_thread(void *board_void) } if (retval <= 0) { dev_err(board->gpib_dev, "stuck SRQ\n"); - - atomic_set(&board->stuck_srq, 1); // XXX could be better - set_bit(SRQI_NUM, &board->status); } } return retval; diff --git a/drivers/staging/greybus/audio_gb.c b/drivers/staging/greybus/audio_gb.c index 9d8994fdb41a..144591f1a512 100644 --- a/drivers/staging/greybus/audio_gb.c +++ b/drivers/staging/greybus/audio_gb.c @@ -37,6 +37,19 @@ int gb_audio_gb_get_topology(struct gb_connection *connection, return ret; } + /* + * The size_* fields are supplied by the module and are used by + * gbaudio_tplg_parse_data() to compute offsets into the blob; make + * sure the sections fit within the fetched topology, so walking it + * cannot read out of bounds. + */ + if ((u64)le32_to_cpu(topo->size_dais) + le32_to_cpu(topo->size_controls) + + le32_to_cpu(topo->size_widgets) + le32_to_cpu(topo->size_routes) > + size - sizeof(*topo)) { + kfree(topo); + return -EINVAL; + } + *topology = topo; return 0; diff --git a/drivers/staging/greybus/audio_manager_sysfs.c b/drivers/staging/greybus/audio_manager_sysfs.c index fcd518f9540c..ff323ca8154f 100644 --- a/drivers/staging/greybus/audio_manager_sysfs.c +++ b/drivers/staging/greybus/audio_manager_sysfs.c @@ -23,7 +23,7 @@ static ssize_t manager_sysfs_add_store(struct kobject *kobj, desc.name, &desc.vid, &desc.pid, &desc.intf_id, &desc.ip_devices, &desc.op_devices); - if (num != 7) + if (num != 6) return -EINVAL; num = gb_audio_manager_add(&desc); diff --git a/drivers/staging/media/ipu7/ipu7-isys.c b/drivers/staging/media/ipu7/ipu7-isys.c index bf262c01a2b8..601e5a79ef8e 100644 --- a/drivers/staging/media/ipu7/ipu7-isys.c +++ b/drivers/staging/media/ipu7/ipu7-isys.c @@ -233,6 +233,7 @@ static int isys_notifier_init(struct ipu7_isys *isys) err_parse: fwnode_handle_put(ep); + v4l2_async_nf_cleanup(&isys->notifier); return ret; } diff --git a/drivers/staging/media/ipu7/ipu7.c b/drivers/staging/media/ipu7/ipu7.c index 8ec571c6bd07..34fe95145c82 100644 --- a/drivers/staging/media/ipu7/ipu7.c +++ b/drivers/staging/media/ipu7/ipu7.c @@ -2343,7 +2343,7 @@ static int ipu7_init_fw_code_region_by_sys(struct ipu7_bus_device *sys, return ret; } - ret = pm_runtime_get_sync(dev); + ret = pm_runtime_resume_and_get(dev); if (ret < 0) { dev_err(dev, "Failed to get runtime PM\n"); return ret; @@ -2702,7 +2702,7 @@ static int ipu7_resume(struct device *dev) if (ret) dev_err(dev, "IPC reset protocol failed!\n"); - ret = pm_runtime_get_sync(&isp->psys->auxdev.dev); + ret = pm_runtime_resume_and_get(&isp->psys->auxdev.dev); if (ret < 0) { dev_err(dev, "Failed to get runtime PM\n"); return 0; diff --git a/drivers/staging/octeon/ethernet-mem.c b/drivers/staging/octeon/ethernet-mem.c index 532594957ebc..68c3ef984e56 100644 --- a/drivers/staging/octeon/ethernet-mem.c +++ b/drivers/staging/octeon/ethernet-mem.c @@ -5,6 +5,7 @@ * Copyright (c) 2003-2010 Cavium Networks */ +#include #include #include #include @@ -40,11 +41,13 @@ static int cvm_oct_fill_hw_skbuff(int pool, int size, int elements) /** * cvm_oct_free_hw_skbuff- free hardware pool skbuffs + * @pdev: Platform device for logging * @pool: Pool to allocate an skbuff for * @size: Size of the buffer needed for the pool * @elements: Number of buffers to allocate */ -static void cvm_oct_free_hw_skbuff(int pool, int size, int elements) +static void cvm_oct_free_hw_skbuff(struct platform_device *pdev, + int pool, int size, int elements) { char *memory; @@ -59,22 +62,24 @@ static void cvm_oct_free_hw_skbuff(int pool, int size, int elements) } while (memory); if (elements < 0) - pr_warn("Freeing of pool %u had too many skbuffs (%d)\n", - pool, elements); + dev_warn(&pdev->dev, "Freeing of pool %u had too many skbuffs (%d)\n", + pool, elements); else if (elements > 0) - pr_warn("Freeing of pool %u is missing %d skbuffs\n", - pool, elements); + dev_warn(&pdev->dev, "Freeing of pool %u is missing %d skbuffs\n", + pool, elements); } /** * cvm_oct_fill_hw_memory - fill a hardware pool with memory. + * @pdev: Platform device for logging * @pool: Pool to populate * @size: Size of each buffer in the pool * @elements: Number of buffers to allocate * * Returns the actual number of buffers allocated. */ -static int cvm_oct_fill_hw_memory(int pool, int size, int elements) +static int cvm_oct_fill_hw_memory(struct platform_device *pdev, int pool, int size, + int elements) { char *memory; char *fpa; @@ -93,8 +98,8 @@ static int cvm_oct_fill_hw_memory(int pool, int size, int elements) */ memory = kmalloc(size + 256, GFP_ATOMIC); if (unlikely(!memory)) { - pr_warn("Unable to allocate %u bytes for FPA pool %d\n", - elements * size, pool); + dev_warn(&pdev->dev, "Unable to allocate %u bytes for FPA pool %d\n", + elements * size, pool); break; } fpa = (char *)(((unsigned long)memory + 256) & ~0x7fUL); @@ -107,11 +112,13 @@ static int cvm_oct_fill_hw_memory(int pool, int size, int elements) /** * cvm_oct_free_hw_memory - Free memory allocated by cvm_oct_fill_hw_memory + * @pdev: Platform device for logging * @pool: FPA pool to free * @size: Size of each buffer in the pool * @elements: Number of buffers that should be in the pool */ -static void cvm_oct_free_hw_memory(int pool, int size, int elements) +static void cvm_oct_free_hw_memory(struct platform_device *pdev, + int pool, int size, int elements) { char *memory; char *fpa; @@ -127,28 +134,28 @@ static void cvm_oct_free_hw_memory(int pool, int size, int elements) } while (fpa); if (elements < 0) - pr_warn("Freeing of pool %u had too many buffers (%d)\n", - pool, elements); + dev_warn(&pdev->dev, "Freeing of pool %u had too many buffers (%d)\n", + pool, elements); else if (elements > 0) - pr_warn("Warning: Freeing of pool %u is missing %d buffers\n", - pool, elements); + dev_warn(&pdev->dev, "Freeing of pool %u is missing %d buffers\n", + pool, elements); } -int cvm_oct_mem_fill_fpa(int pool, int size, int elements) +int cvm_oct_mem_fill_fpa(struct platform_device *pdev, int pool, int size, int elements) { int freed; if (pool == CVMX_FPA_PACKET_POOL) freed = cvm_oct_fill_hw_skbuff(pool, size, elements); else - freed = cvm_oct_fill_hw_memory(pool, size, elements); + freed = cvm_oct_fill_hw_memory(pdev, pool, size, elements); return freed; } -void cvm_oct_mem_empty_fpa(int pool, int size, int elements) +void cvm_oct_mem_empty_fpa(struct platform_device *pdev, int pool, int size, int elements) { if (pool == CVMX_FPA_PACKET_POOL) - cvm_oct_free_hw_skbuff(pool, size, elements); + cvm_oct_free_hw_skbuff(pdev, pool, size, elements); else - cvm_oct_free_hw_memory(pool, size, elements); + cvm_oct_free_hw_memory(pdev, pool, size, elements); } diff --git a/drivers/staging/octeon/ethernet-mem.h b/drivers/staging/octeon/ethernet-mem.h index 692dcdb7154d..9279bb0de2db 100644 --- a/drivers/staging/octeon/ethernet-mem.h +++ b/drivers/staging/octeon/ethernet-mem.h @@ -5,5 +5,9 @@ * Copyright (c) 2003-2007 Cavium Networks */ -int cvm_oct_mem_fill_fpa(int pool, int size, int elements); -void cvm_oct_mem_empty_fpa(int pool, int size, int elements); +struct platform_device; + +int cvm_oct_mem_fill_fpa(struct platform_device *pdev, int pool, int size, + int elements); +void cvm_oct_mem_empty_fpa(struct platform_device *pdev, int pool, int size, + int elements); diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c index 965330eec80a..3e9d58d32156 100644 --- a/drivers/staging/octeon/ethernet-rx.c +++ b/drivers/staging/octeon/ethernet-rx.c @@ -5,6 +5,7 @@ * Copyright (c) 2003-2010 Cavium Networks */ +#include #include #include #include @@ -31,12 +32,6 @@ static atomic_t oct_rx_ready = ATOMIC_INIT(0); -static struct oct_rx_group { - int irq; - int group; - struct napi_struct napi; -} oct_rx_group[16]; - /** * cvm_oct_do_interrupt - interrupt handler. * @irq: Interrupt number. @@ -397,7 +392,7 @@ static int cvm_oct_poll(struct oct_rx_group *rx_group, int budget) /* Restore the scratch area */ cvmx_scratch_write64(CVMX_SCR_SCRATCH, old_scratch); } - cvm_oct_rx_refill_pool(0); + cvm_oct_rx_refill_pool(rx_group->pdev, 0); return rx_count; } @@ -434,24 +429,28 @@ static int cvm_oct_napi_poll(struct napi_struct *napi, int budget) */ void cvm_oct_poll_controller(struct net_device *dev) { + struct platform_device *pdev = to_platform_device(dev->dev.parent); + struct octeon_ethernet_platform *plat = platform_get_drvdata(pdev); int i; if (!atomic_read(&oct_rx_ready)) return; - for (i = 0; i < ARRAY_SIZE(oct_rx_group); i++) { + for (i = 0; i < ARRAY_SIZE(plat->rx_group); i++) { if (!(pow_receive_groups & BIT(i))) continue; - cvm_oct_poll(&oct_rx_group[i], 16); + cvm_oct_poll(&plat->rx_group[i], 16); } } #endif -void cvm_oct_rx_initialize(void) +void cvm_oct_rx_initialize(struct platform_device *pdev) { int i; struct net_device *dev_for_napi = NULL; + struct octeon_ethernet_platform *plat = platform_get_drvdata(pdev); + struct oct_rx_group *rx_group = plat->rx_group; for (i = 0; i < TOTAL_NUMBER_OF_PORTS; i++) { if (cvm_oct_device[i]) { @@ -463,27 +462,28 @@ void cvm_oct_rx_initialize(void) if (!dev_for_napi) panic("No net_devices were allocated."); - for (i = 0; i < ARRAY_SIZE(oct_rx_group); i++) { + for (i = 0; i < ARRAY_SIZE(plat->rx_group); i++) { int ret; if (!(pow_receive_groups & BIT(i))) continue; - netif_napi_add_weight(dev_for_napi, &oct_rx_group[i].napi, + netif_napi_add_weight(dev_for_napi, &rx_group[i].napi, cvm_oct_napi_poll, rx_napi_weight); - napi_enable(&oct_rx_group[i].napi); + napi_enable(&rx_group[i].napi); - oct_rx_group[i].irq = OCTEON_IRQ_WORKQ0 + i; - oct_rx_group[i].group = i; + rx_group[i].irq = OCTEON_IRQ_WORKQ0 + i; + rx_group[i].group = i; + rx_group[i].pdev = pdev; /* Register an IRQ handler to receive POW interrupts */ - ret = request_irq(oct_rx_group[i].irq, cvm_oct_do_interrupt, 0, - "Ethernet", &oct_rx_group[i].napi); + ret = request_irq(rx_group[i].irq, cvm_oct_do_interrupt, 0, + "Ethernet", &rx_group[i].napi); if (ret) panic("Could not acquire Ethernet IRQ %d\n", - oct_rx_group[i].irq); + rx_group[i].irq); - disable_irq_nosync(oct_rx_group[i].irq); + disable_irq_nosync(rx_group[i].irq); /* Enable POW interrupt when our port has at least one packet */ if (OCTEON_IS_MODEL(OCTEON_CN68XX)) { @@ -515,16 +515,17 @@ void cvm_oct_rx_initialize(void) /* Schedule NAPI now. This will indirectly enable the * interrupt. */ - napi_schedule(&oct_rx_group[i].napi); + napi_schedule(&rx_group[i].napi); } atomic_inc(&oct_rx_ready); } -void cvm_oct_rx_shutdown(void) +void cvm_oct_rx_shutdown(struct platform_device *pdev) { + struct octeon_ethernet_platform *plat = platform_get_drvdata(pdev); int i; - for (i = 0; i < ARRAY_SIZE(oct_rx_group); i++) { + for (i = 0; i < ARRAY_SIZE(plat->rx_group); i++) { if (!(pow_receive_groups & BIT(i))) continue; @@ -534,9 +535,11 @@ void cvm_oct_rx_shutdown(void) else cvmx_write_csr(CVMX_POW_WQ_INT_THRX(i), 0); + napi_disable(&plat->rx_group[i].napi); + /* Free the interrupt handler */ - free_irq(oct_rx_group[i].irq, cvm_oct_device); + free_irq(plat->rx_group[i].irq, &plat->rx_group[i].napi); - netif_napi_del(&oct_rx_group[i].napi); + netif_napi_del(&plat->rx_group[i].napi); } } diff --git a/drivers/staging/octeon/ethernet-rx.h b/drivers/staging/octeon/ethernet-rx.h index ff6482fa20d6..6093694326cb 100644 --- a/drivers/staging/octeon/ethernet-rx.h +++ b/drivers/staging/octeon/ethernet-rx.h @@ -5,11 +5,14 @@ * Copyright (c) 2003-2007 Cavium Networks */ +struct platform_device; + void cvm_oct_poll_controller(struct net_device *dev); -void cvm_oct_rx_initialize(void); -void cvm_oct_rx_shutdown(void); +void cvm_oct_rx_initialize(struct platform_device *pdev); +void cvm_oct_rx_shutdown(struct platform_device *pdev); -static inline void cvm_oct_rx_refill_pool(int fill_threshold) +static inline void cvm_oct_rx_refill_pool(struct platform_device *pdev, + int fill_threshold) { int number_to_free; int num_freed; @@ -20,7 +23,7 @@ static inline void cvm_oct_rx_refill_pool(int fill_threshold) if (number_to_free > fill_threshold) { cvmx_fau_atomic_add32(FAU_NUM_PACKET_BUFFERS_TO_FREE, -number_to_free); - num_freed = cvm_oct_mem_fill_fpa(CVMX_FPA_PACKET_POOL, + num_freed = cvm_oct_mem_fill_fpa(pdev, CVMX_FPA_PACKET_POOL, CVMX_FPA_PACKET_POOL_SIZE, number_to_free); if (num_freed != number_to_free) { diff --git a/drivers/staging/octeon/ethernet-tx.c b/drivers/staging/octeon/ethernet-tx.c index f5bbedac6a65..1b30fe1bcf43 100644 --- a/drivers/staging/octeon/ethernet-tx.c +++ b/drivers/staging/octeon/ethernet-tx.c @@ -670,4 +670,6 @@ void cvm_oct_tx_shutdown(void) { /* Free the interrupt handler */ free_irq(OCTEON_IRQ_TIMER1, cvm_oct_device); + + tasklet_kill(&cvm_oct_tx_cleanup_tasklet); } diff --git a/drivers/staging/octeon/ethernet.c b/drivers/staging/octeon/ethernet.c index eadb74fc14c8..0a24c62ae32b 100644 --- a/drivers/staging/octeon/ethernet.c +++ b/drivers/staging/octeon/ethernet.c @@ -104,11 +104,10 @@ struct net_device *cvm_oct_device[TOTAL_NUMBER_OF_PORTS]; u64 cvm_oct_tx_poll_interval; -static void cvm_oct_rx_refill_worker(struct work_struct *work); -static DECLARE_DELAYED_WORK(cvm_oct_rx_refill_work, cvm_oct_rx_refill_worker); - static void cvm_oct_rx_refill_worker(struct work_struct *work) { + struct octeon_ethernet_platform *plat = container_of(work, + struct octeon_ethernet_platform, rx_refill_work.work); /* * FPA 0 may have been drained, try to refill it if we need * more than num_packet_buffers / 2, otherwise normal receive @@ -116,10 +115,10 @@ static void cvm_oct_rx_refill_worker(struct work_struct *work) * could be received so cvm_oct_napi_poll would never be * invoked to do the refill. */ - cvm_oct_rx_refill_pool(num_packet_buffers / 2); + cvm_oct_rx_refill_pool(plat->pdev, num_packet_buffers / 2); if (!atomic_read(&cvm_oct_poll_queue_stopping)) - schedule_delayed_work(&cvm_oct_rx_refill_work, HZ); + schedule_delayed_work(&plat->rx_refill_work, HZ); } static void cvm_oct_periodic_worker(struct work_struct *work) @@ -138,16 +137,16 @@ static void cvm_oct_periodic_worker(struct work_struct *work) schedule_delayed_work(&priv->port_periodic_work, HZ); } -static void cvm_oct_configure_common_hw(void) +static void cvm_oct_configure_common_hw(struct platform_device *pdev) { /* Setup the FPA */ cvmx_fpa_enable(); - cvm_oct_mem_fill_fpa(CVMX_FPA_PACKET_POOL, CVMX_FPA_PACKET_POOL_SIZE, + cvm_oct_mem_fill_fpa(pdev, CVMX_FPA_PACKET_POOL, CVMX_FPA_PACKET_POOL_SIZE, num_packet_buffers); - cvm_oct_mem_fill_fpa(CVMX_FPA_WQE_POOL, CVMX_FPA_WQE_POOL_SIZE, + cvm_oct_mem_fill_fpa(pdev, CVMX_FPA_WQE_POOL, CVMX_FPA_WQE_POOL_SIZE, num_packet_buffers); if (CVMX_FPA_OUTPUT_BUFFER_POOL != CVMX_FPA_PACKET_POOL) - cvm_oct_mem_fill_fpa(CVMX_FPA_OUTPUT_BUFFER_POOL, + cvm_oct_mem_fill_fpa(pdev, CVMX_FPA_OUTPUT_BUFFER_POOL, CVMX_FPA_OUTPUT_BUFFER_POOL_SIZE, 1024); #ifdef __LITTLE_ENDIAN @@ -678,6 +677,15 @@ static int cvm_oct_probe(struct platform_device *pdev) int qos; struct device_node *pip; int mtu_overhead = ETH_HLEN + ETH_FCS_LEN; + struct octeon_ethernet_platform *plat; + + plat = devm_kzalloc(&pdev->dev, sizeof(*plat), GFP_KERNEL); + if (!plat) + return -ENOMEM; + + plat->pdev = pdev; + INIT_DELAYED_WORK(&plat->rx_refill_work, cvm_oct_rx_refill_worker); + platform_set_drvdata(pdev, plat); #if IS_ENABLED(CONFIG_VLAN_8021Q) mtu_overhead += VLAN_HLEN; @@ -689,7 +697,7 @@ static int cvm_oct_probe(struct platform_device *pdev) return -EINVAL; } - cvm_oct_configure_common_hw(); + cvm_oct_configure_common_hw(pdev); cvmx_helper_initialize_packet_io_global(); @@ -912,28 +920,29 @@ static int cvm_oct_probe(struct platform_device *pdev) } cvm_oct_tx_initialize(); - cvm_oct_rx_initialize(); + cvm_oct_rx_initialize(pdev); /* * 150 uS: about 10 1500-byte packets at 1GE. */ cvm_oct_tx_poll_interval = 150 * (octeon_get_clock_rate() / 1000000); - schedule_delayed_work(&cvm_oct_rx_refill_work, HZ); + schedule_delayed_work(&plat->rx_refill_work, HZ); return 0; } static void cvm_oct_remove(struct platform_device *pdev) { + struct octeon_ethernet_platform *plat = platform_get_drvdata(pdev); int port; cvmx_ipd_disable(); atomic_inc_return(&cvm_oct_poll_queue_stopping); - cancel_delayed_work_sync(&cvm_oct_rx_refill_work); + cancel_delayed_work_sync(&plat->rx_refill_work); - cvm_oct_rx_shutdown(); + cvm_oct_rx_shutdown(pdev); cvm_oct_tx_shutdown(); cvmx_pko_disable(); @@ -958,12 +967,12 @@ static void cvm_oct_remove(struct platform_device *pdev) cvmx_ipd_free_ptr(); /* Free the HW pools */ - cvm_oct_mem_empty_fpa(CVMX_FPA_PACKET_POOL, CVMX_FPA_PACKET_POOL_SIZE, + cvm_oct_mem_empty_fpa(pdev, CVMX_FPA_PACKET_POOL, CVMX_FPA_PACKET_POOL_SIZE, num_packet_buffers); - cvm_oct_mem_empty_fpa(CVMX_FPA_WQE_POOL, CVMX_FPA_WQE_POOL_SIZE, + cvm_oct_mem_empty_fpa(pdev, CVMX_FPA_WQE_POOL, CVMX_FPA_WQE_POOL_SIZE, num_packet_buffers); if (CVMX_FPA_OUTPUT_BUFFER_POOL != CVMX_FPA_PACKET_POOL) - cvm_oct_mem_empty_fpa(CVMX_FPA_OUTPUT_BUFFER_POOL, + cvm_oct_mem_empty_fpa(pdev, CVMX_FPA_OUTPUT_BUFFER_POOL, CVMX_FPA_OUTPUT_BUFFER_POOL_SIZE, 128); } diff --git a/drivers/staging/octeon/octeon-ethernet.h b/drivers/staging/octeon/octeon-ethernet.h index a6140705706f..0ac430db1e6e 100644 --- a/drivers/staging/octeon/octeon-ethernet.h +++ b/drivers/staging/octeon/octeon-ethernet.h @@ -11,6 +11,7 @@ #ifndef OCTEON_ETHERNET_H #define OCTEON_ETHERNET_H +#include #include #include @@ -74,6 +75,19 @@ struct octeon_ethernet { struct device_node *of_node; }; +struct oct_rx_group { + int irq; + int group; + struct napi_struct napi; + struct platform_device *pdev; +}; + +struct octeon_ethernet_platform { + struct platform_device *pdev; + struct delayed_work rx_refill_work; + struct oct_rx_group rx_group[16]; +}; + int cvm_oct_free_work(void *work_queue_entry); int cvm_oct_rgmii_open(struct net_device *dev); diff --git a/drivers/staging/rtl8723bs/core/rtw_ap.c b/drivers/staging/rtl8723bs/core/rtw_ap.c index 0908f2234f67..98405bedf5da 100644 --- a/drivers/staging/rtl8723bs/core/rtw_ap.c +++ b/drivers/staging/rtl8723bs/core/rtw_ap.c @@ -1262,13 +1262,13 @@ u8 rtw_ap_set_pairwise_key(struct adapter *padapter, struct sta_info *psta) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_KERNEL); if (!ph2c) { res = _FAIL; goto exit; } - psetstakey_para = rtw_zmalloc(sizeof(struct set_stakey_parm)); + psetstakey_para = kzalloc(sizeof(*psetstakey_para), GFP_KERNEL); if (!psetstakey_para) { kfree(ph2c); res = _FAIL; @@ -1304,12 +1304,12 @@ static int rtw_ap_set_key( struct cmd_priv *pcmdpriv = &padapter->cmdpriv; int res = _SUCCESS; - pcmd = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd = kzalloc(sizeof(*pcmd), GFP_KERNEL); if (!pcmd) { res = _FAIL; goto exit; } - psetkeyparm = rtw_zmalloc(sizeof(struct setkey_parm)); + psetkeyparm = kzalloc(sizeof(*psetkeyparm), GFP_KERNEL); if (!psetkeyparm) { kfree(pcmd); res = _FAIL; diff --git a/drivers/staging/rtl8723bs/core/rtw_cmd.c b/drivers/staging/rtl8723bs/core/rtw_cmd.c index ef2d92b5588a..2701a80ad07d 100644 --- a/drivers/staging/rtl8723bs/core/rtw_cmd.c +++ b/drivers/staging/rtl8723bs/core/rtw_cmd.c @@ -170,15 +170,13 @@ int rtw_init_cmd_priv(struct cmd_priv *pcmdpriv) pcmdpriv->cmd_seq = 1; - pcmdpriv->cmd_allocated_buf = rtw_zmalloc(MAX_CMDSZ + CMDBUFF_ALIGN_SZ); - + pcmdpriv->cmd_allocated_buf = kzalloc(MAX_CMDSZ + CMDBUFF_ALIGN_SZ, GFP_ATOMIC); if (!pcmdpriv->cmd_allocated_buf) return -ENOMEM; pcmdpriv->cmd_buf = pcmdpriv->cmd_allocated_buf + CMDBUFF_ALIGN_SZ - ((SIZE_PTR)(pcmdpriv->cmd_allocated_buf) & (CMDBUFF_ALIGN_SZ-1)); - pcmdpriv->rsp_allocated_buf = rtw_zmalloc(MAX_RSPSZ + 4); - + pcmdpriv->rsp_allocated_buf = kzalloc(MAX_RSPSZ + 4, GFP_ATOMIC); if (!pcmdpriv->rsp_allocated_buf) { kfree(pcmdpriv->cmd_allocated_buf); return -ENOMEM; @@ -534,11 +532,11 @@ u8 rtw_sitesurvey_cmd(struct adapter *padapter, struct ndis_802_11_ssid *ssid, if (check_fwstate(pmlmepriv, _FW_LINKED)) rtw_lps_ctrl_wk_cmd(padapter, LPS_CTRL_SCAN, 1); - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) return _FAIL; - psurveyPara = rtw_zmalloc(sizeof(struct sitesurvey_parm)); + psurveyPara = kzalloc(sizeof(*psurveyPara), GFP_ATOMIC); if (!psurveyPara) { kfree(ph2c); return _FAIL; @@ -602,7 +600,7 @@ u8 rtw_createbss_cmd(struct adapter *padapter) struct wlan_bssid_ex *pdev_network = &padapter->registrypriv.dev_network; u8 res = _SUCCESS; - pcmd = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd = kzalloc(sizeof(*pcmd), GFP_ATOMIC); if (!pcmd) { res = _FAIL; goto exit; @@ -635,7 +633,7 @@ int rtw_startbss_cmd(struct adapter *padapter, int flags) start_bss_network(padapter); } else { /* need enqueue, prepare cmd_obj and enqueue */ - pcmd = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd = kzalloc(sizeof(*pcmd), GFP_KERNEL); if (!pcmd) { res = _FAIL; goto exit; @@ -687,7 +685,7 @@ u8 rtw_joinbss_cmd(struct adapter *padapter, struct wlan_network *pnetwork) u32 tmp_len; u8 *ptmp = NULL; - pcmd = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd = kzalloc(sizeof(*pcmd), GFP_KERNEL); if (!pcmd) { res = _FAIL; goto exit; @@ -796,7 +794,7 @@ u8 rtw_disassoc_cmd(struct adapter *padapter, u32 deauth_timeout_ms, bool enqueu u8 res = _SUCCESS; /* prepare cmd parameter */ - param = rtw_zmalloc(sizeof(*param)); + param = kzalloc(sizeof(*param), GFP_KERNEL); if (!param) { res = _FAIL; goto exit; @@ -805,7 +803,7 @@ u8 rtw_disassoc_cmd(struct adapter *padapter, u32 deauth_timeout_ms, bool enqueu if (enqueue) { /* need enqueue, prepare cmd_obj and enqueue */ - cmdobj = rtw_zmalloc(sizeof(*cmdobj)); + cmdobj = kzalloc(sizeof(*cmdobj), GFP_KERNEL); if (!cmdobj) { res = _FAIL; kfree(param); @@ -832,8 +830,7 @@ u8 rtw_setopmode_cmd(struct adapter *padapter, enum ndis_802_11_network_infrast struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - psetop = rtw_zmalloc(sizeof(struct setopmode_parm)); - + psetop = kzalloc(sizeof(*psetop), GFP_KERNEL); if (!psetop) { res = _FAIL; goto exit; @@ -841,7 +838,7 @@ u8 rtw_setopmode_cmd(struct adapter *padapter, enum ndis_802_11_network_infrast psetop->mode = (u8)networktype; if (enqueue) { - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_KERNEL); if (!ph2c) { kfree(psetop); res = _FAIL; @@ -868,7 +865,7 @@ u8 rtw_setstakey_cmd(struct adapter *padapter, struct sta_info *sta, u8 unicast_ struct security_priv *psecuritypriv = &padapter->securitypriv; u8 res = _SUCCESS; - psetstakey_para = rtw_zmalloc(sizeof(struct set_stakey_parm)); + psetstakey_para = kzalloc(sizeof(*psetstakey_para), GFP_KERNEL); if (!psetstakey_para) { res = _FAIL; goto exit; @@ -890,17 +887,17 @@ u8 rtw_setstakey_cmd(struct adapter *padapter, struct sta_info *sta, u8 unicast_ padapter->securitypriv.busetkipkey = true; if (enqueue) { - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_KERNEL); if (!ph2c) { - kfree(psetstakey_para); + kfree_sensitive(psetstakey_para); res = _FAIL; goto exit; } - psetstakey_rsp = rtw_zmalloc(sizeof(struct set_stakey_rsp)); + psetstakey_rsp = kzalloc(sizeof(*psetstakey_rsp), GFP_KERNEL); if (!psetstakey_rsp) { kfree(ph2c); - kfree(psetstakey_para); + kfree_sensitive(psetstakey_para); res = _FAIL; goto exit; } @@ -911,7 +908,7 @@ u8 rtw_setstakey_cmd(struct adapter *padapter, struct sta_info *sta, u8 unicast_ res = rtw_enqueue_cmd(pcmdpriv, ph2c); } else { set_stakey_hdl(padapter, (u8 *)psetstakey_para); - kfree(psetstakey_para); + kfree_sensitive(psetstakey_para); } exit: return res; @@ -935,23 +932,23 @@ u8 rtw_clearstakey_cmd(struct adapter *padapter, struct sta_info *sta, u8 enqueu rtw_camid_free(padapter, cam_id); } } else { - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_KERNEL); if (!ph2c) { res = _FAIL; goto exit; } - psetstakey_para = rtw_zmalloc(sizeof(struct set_stakey_parm)); + psetstakey_para = kzalloc(sizeof(*psetstakey_para), GFP_KERNEL); if (!psetstakey_para) { kfree(ph2c); res = _FAIL; goto exit; } - psetstakey_rsp = rtw_zmalloc(sizeof(struct set_stakey_rsp)); + psetstakey_rsp = kzalloc(sizeof(*psetstakey_rsp), GFP_KERNEL); if (!psetstakey_rsp) { kfree(ph2c); - kfree(psetstakey_para); + kfree_sensitive(psetstakey_para); res = _FAIL; goto exit; } @@ -978,13 +975,13 @@ u8 rtw_addbareq_cmd(struct adapter *padapter, u8 tid, u8 *addr) u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - paddbareq_parm = rtw_zmalloc(sizeof(struct addBaReq_parm)); + paddbareq_parm = kzalloc(sizeof(*paddbareq_parm), GFP_ATOMIC); if (!paddbareq_parm) { kfree(ph2c); res = _FAIL; @@ -1010,13 +1007,13 @@ u8 rtw_reset_securitypriv_cmd(struct adapter *padapter) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1043,13 +1040,13 @@ u8 rtw_free_assoc_resources_cmd(struct adapter *padapter) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1077,13 +1074,13 @@ u8 rtw_dynamic_chk_wk_cmd(struct adapter *padapter) u8 res = _SUCCESS; /* only primary padapter does this cmd */ - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1318,13 +1315,13 @@ u8 rtw_lps_ctrl_wk_cmd(struct adapter *padapter, u8 lps_ctrl_type, u8 enqueue) u8 res = _SUCCESS; if (enqueue) { - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1359,13 +1356,13 @@ u8 rtw_dm_in_lps_wk_cmd(struct adapter *padapter) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1421,13 +1418,13 @@ u8 rtw_dm_ra_mask_wk_cmd(struct adapter *padapter, u8 *psta) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1456,13 +1453,13 @@ u8 rtw_ps_cmd(struct adapter *padapter) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ppscmd = rtw_zmalloc(sizeof(struct cmd_obj)); + ppscmd = kzalloc(sizeof(*ppscmd), GFP_ATOMIC); if (!ppscmd) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ppscmd); res = _FAIL; @@ -1528,13 +1525,13 @@ u8 rtw_chk_hi_queue_cmd(struct adapter *padapter) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1620,13 +1617,13 @@ u8 rtw_c2h_packet_wk_cmd(struct adapter *padapter, u8 *pbuf, u16 length) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_ATOMIC); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; @@ -1655,13 +1652,13 @@ u8 rtw_c2h_wk_cmd(struct adapter *padapter, u8 *c2h_evt) struct cmd_priv *pcmdpriv = &padapter->cmdpriv; u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_KERNEL); if (!ph2c) { res = _FAIL; goto exit; } - pdrvextra_cmd_parm = rtw_zmalloc(sizeof(struct drvextra_cmd_parm)); + pdrvextra_cmd_parm = kzalloc(sizeof(*pdrvextra_cmd_parm), GFP_KERNEL); if (!pdrvextra_cmd_parm) { kfree(ph2c); res = _FAIL; diff --git a/drivers/staging/rtl8723bs/core/rtw_efuse.c b/drivers/staging/rtl8723bs/core/rtw_efuse.c index d5c53b614f61..62a40d4672d3 100644 --- a/drivers/staging/rtl8723bs/core/rtw_efuse.c +++ b/drivers/staging/rtl8723bs/core/rtw_efuse.c @@ -253,7 +253,6 @@ void EFUSE_ShadowMapUpdate(struct adapter *padapter, u8 efuseType) /* void *)&pHalData->EfuseMap[EFUSE_INIT_MAP][0], mapLen); */ } /* EFUSE_ShadowMapUpdate */ - /*----------------------------------------------------------------------------- * Function: EFUSE_ShadowRead * diff --git a/drivers/staging/rtl8723bs/core/rtw_ioctl_set.c b/drivers/staging/rtl8723bs/core/rtw_ioctl_set.c index 587a87fbffeb..df664efb2126 100644 --- a/drivers/staging/rtl8723bs/core/rtw_ioctl_set.c +++ b/drivers/staging/rtl8723bs/core/rtw_ioctl_set.c @@ -345,7 +345,6 @@ u8 rtw_set_802_11_infrastructure_mode(struct adapter *padapter, return true; } - u8 rtw_set_802_11_disassociate(struct adapter *padapter) { struct mlme_priv *pmlmepriv = &padapter->mlmepriv; diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme.c b/drivers/staging/rtl8723bs/core/rtw_mlme.c index f2b5455591fd..039d47f221cf 100644 --- a/drivers/staging/rtl8723bs/core/rtw_mlme.c +++ b/drivers/staging/rtl8723bs/core/rtw_mlme.c @@ -612,6 +612,8 @@ int rtw_is_desired_network(struct adapter *adapter, struct wlan_network *pnetwor privacy = pnetwork->network.privacy; if (check_fwstate(pmlmepriv, WIFI_UNDER_WPS)) { + if (pnetwork->network.ie_length < _FIXED_IE_LENGTH_) + return false; if (rtw_get_wps_ie(pnetwork->network.ies+_FIXED_IE_LENGTH_, pnetwork->network.ie_length-_FIXED_IE_LENGTH_, NULL, &wps_ielen)) return true; else @@ -625,11 +627,15 @@ int rtw_is_desired_network(struct adapter *adapter, struct wlan_network *pnetwor bselected = false; if (psecuritypriv->ndisauthtype == Ndis802_11AuthModeWPA2PSK) { - p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_)); - if (p && ie_len > 0) - bselected = true; - else + if (pnetwork->network.ie_length < _BEACON_IE_OFFSET_) { bselected = false; + } else { + p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_)); + if (p && ie_len > 0) + bselected = true; + else + bselected = false; + } } } @@ -1901,13 +1907,13 @@ signed int rtw_set_auth(struct adapter *adapter, struct security_priv *psecurity struct cmd_priv *pcmdpriv = &adapter->cmdpriv; signed int res = _SUCCESS; - pcmd = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd = kzalloc(sizeof(*pcmd), GFP_KERNEL); if (!pcmd) { res = _FAIL; /* try again */ goto exit; } - psetauthparm = rtw_zmalloc(sizeof(struct setauth_parm)); + psetauthparm = kzalloc(sizeof(*psetauthparm), GFP_KERNEL); if (!psetauthparm) { kfree(pcmd); res = _FAIL; @@ -1938,7 +1944,7 @@ signed int rtw_set_key(struct adapter *adapter, struct security_priv *psecurityp struct cmd_priv *pcmdpriv = &adapter->cmdpriv; signed int res = _SUCCESS; - psetkeyparm = rtw_zmalloc(sizeof(struct setkey_parm)); + psetkeyparm = kzalloc(sizeof(*psetkeyparm), GFP_KERNEL); if (!psetkeyparm) { res = _FAIL; goto exit; @@ -1980,7 +1986,7 @@ signed int rtw_set_key(struct adapter *adapter, struct security_priv *psecurityp } if (enqueue) { - pcmd = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd = kzalloc(sizeof(*pcmd), GFP_KERNEL); if (!pcmd) { kfree(psetkeyparm); res = _FAIL; /* try again */ @@ -2029,7 +2035,10 @@ int rtw_restruct_wmm_ie(struct adapter *adapter, u8 *in_ie, u8 *out_ie, uint in_ break; } - i += (in_ie[i+1]+2); /* to the next IE element */ + if (i + 1 >= in_len) + break; + + i += (in_ie[i + 1] + 2); /* to the next IE element */ } return ielength; @@ -2090,7 +2099,7 @@ static void rtw_report_sec_ie(struct adapter *adapter, u8 authmode, u8 *sec_ie) buff = NULL; if (authmode == WLAN_EID_VENDOR_SPECIFIC) { - buff = rtw_zmalloc(IW_CUSTOM_MAX); + buff = kzalloc(IW_CUSTOM_MAX, GFP_ATOMIC); if (!buff) return; diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c index 9c991ef20d21..8d33631ab8f0 100644 --- a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c +++ b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c @@ -536,7 +536,6 @@ unsigned int OnProbeReq(struct adapter *padapter, union recv_frame *precv_frame) p = rtw_get_ie(pframe + WLAN_HDR_A3_LEN + _PROBEREQ_IE_OFFSET_, WLAN_EID_SSID, (int *)&ielen, len - WLAN_HDR_A3_LEN - _PROBEREQ_IE_OFFSET_); - /* check (wildcard) SSID */ if (p) { if (is_valid_p2p_probereq) @@ -795,7 +794,6 @@ unsigned int OnAuth(struct adapter *padapter, union recv_frame *precv_frame) if (pstat->auth_seq == 0) pstat->expire_to = pstapriv->auth_to; - if ((pstat->auth_seq + 1) != seq) { status = WLAN_STATUS_UNKNOWN_AUTH_TRANSACTION; goto auth_fail; @@ -844,7 +842,6 @@ unsigned int OnAuth(struct adapter *padapter, union recv_frame *precv_frame) } } - /* Now, we are going to issue_auth... */ pstat->auth_seq = seq + 1; @@ -853,7 +850,6 @@ unsigned int OnAuth(struct adapter *padapter, union recv_frame *precv_frame) if (pstat->state & WIFI_FW_AUTH_SUCCESS) pstat->auth_seq = 0; - return _SUCCESS; auth_fail: @@ -979,7 +975,6 @@ unsigned int OnAssocReq(struct adapter *padapter, union recv_frame *precv_frame) else /* WIFI_REASSOCREQ */ ie_offset = _REASOCREQ_IE_OFFSET_; - if (pkt_len < sizeof(struct ieee80211_hdr_3addr) + ie_offset) return _FAIL; @@ -1009,7 +1004,6 @@ unsigned int OnAssocReq(struct adapter *padapter, union recv_frame *precv_frame) pstat->state |= WIFI_FW_ASSOC_STATE; } - pstat->capability = capab_info; /* now parse all ieee802_11 ie to point to elems */ @@ -1146,7 +1140,6 @@ unsigned int OnAssocReq(struct adapter *padapter, union recv_frame *precv_frame) pstat->flags |= WLAN_STA_MAYBE_WPS; } - /* AP support WPA/RSN, and sta is going to do WPS, but AP is not ready */ /* that the selected registrar of AP is _FLASE */ if ((psecuritypriv->wpa_psk > 0) @@ -1181,13 +1174,11 @@ unsigned int OnAssocReq(struct adapter *padapter, union recv_frame *precv_frame) copy_len = ((wpa_ie_len+2) > sizeof(pstat->wpa_ie)) ? (sizeof(pstat->wpa_ie)):(wpa_ie_len+2); } - if (copy_len > 0) memcpy(pstat->wpa_ie, wpa_ie-2, copy_len); } - /* check if there is WMM IE & support WWM-PS */ pstat->flags &= ~WLAN_STA_WME; pstat->qos_option = 0; @@ -1260,13 +1251,11 @@ unsigned int OnAssocReq(struct adapter *padapter, union recv_frame *precv_frame) } else pstat->flags &= ~WLAN_STA_HT; - if ((pmlmepriv->htpriv.ht_option == false) && (pstat->flags&WLAN_STA_HT)) { status = WLAN_STATUS_CHALLENGE_FAIL; goto OnAssocReqFail; } - if ((pstat->flags & WLAN_STA_HT) && ((pstat->wpa2_pairwise_cipher&WPA_CIPHER_TKIP) || (pstat->wpa_pairwise_cipher&WPA_CIPHER_TKIP))) { @@ -1314,13 +1303,11 @@ unsigned int OnAssocReq(struct adapter *padapter, union recv_frame *precv_frame) goto OnAssocReqFail; - } else { pstapriv->sta_aid[pstat->aid - 1] = pstat; } } - pstat->state &= (~WIFI_FW_ASSOC_STATE); pstat->state |= WIFI_FW_ASSOC_SUCCESS; @@ -1519,7 +1506,6 @@ unsigned int OnDeAuth(struct adapter *padapter, union recv_frame *precv_frame) associated_clients_update(padapter, updated); } - return _SUCCESS; } @@ -2162,7 +2148,6 @@ void issue_beacon(struct adapter *padapter, int timeout_ms) pframe = (u8 *)(pmgntframe->buf_addr) + TXDESC_OFFSET; pwlanhdr = (struct ieee80211_hdr *)pframe; - fctrl = &(pwlanhdr->frame_control); *(fctrl) = 0; @@ -2251,12 +2236,10 @@ void issue_beacon(struct adapter *padapter, int timeout_ms) pframe = rtw_set_ie(pframe, WLAN_EID_ERP_INFO, 1, &erpinfo, &pattrib->pktlen); } - /* EXTERNDED SUPPORTED RATE */ if (rate_len > 8) pframe = rtw_set_ie(pframe, WLAN_EID_EXT_SUPP_RATES, (rate_len - 8), (cur_network->supported_rates + 8), &pattrib->pktlen); - /* todo:HT for adhoc */ _issue_bcn: @@ -2265,8 +2248,11 @@ void issue_beacon(struct adapter *padapter, int timeout_ms) spin_unlock_bh(&pmlmepriv->bcn_update_lock); - if ((pattrib->pktlen + TXDESC_SIZE) > 512) + if ((pattrib->pktlen + TXDESC_SIZE) > 512) { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } pattrib->last_txcmdsz = pattrib->pktlen; @@ -2328,9 +2314,11 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p pattrib->pktlen = pattrib->hdrlen; pframe += pattrib->hdrlen; - - if (cur_network->ie_length > MAX_IE_SZ) + if (cur_network->ie_length > MAX_IE_SZ) { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } if ((pmlmeinfo->state&0x03) == WIFI_FW_AP_STATE) { pwps_ie = rtw_get_wps_ie(cur_network->ies+_FIXED_IE_LENGTH_, cur_network->ie_length-_FIXED_IE_LENGTH_, NULL, &wps_ielen); @@ -2376,9 +2364,12 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p u8 *buf; u8 *ies = pmgntframe->buf_addr+TXDESC_OFFSET+sizeof(struct ieee80211_hdr_3addr); - buf = rtw_zmalloc(MAX_IE_SZ); - if (!buf) + buf = kzalloc(MAX_IE_SZ, GFP_ATOMIC); + if (!buf) { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } ssid_ie = rtw_get_ie(ies+_FIXED_IE_LENGTH_, WLAN_EID_SSID, &ssid_ielen, (pframe-ies)-_FIXED_IE_LENGTH_); @@ -2452,19 +2443,16 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p pframe = rtw_set_ie(pframe, WLAN_EID_ERP_INFO, 1, &erpinfo, &pattrib->pktlen); } - /* EXTERNDED SUPPORTED RATE */ if (rate_len > 8) pframe = rtw_set_ie(pframe, WLAN_EID_EXT_SUPP_RATES, (rate_len - 8), (cur_network->supported_rates + 8), &pattrib->pktlen); - /* todo:HT for adhoc */ } pattrib->last_txcmdsz = pattrib->pktlen; - dump_mgntframe(padapter, pmgntframe); return; @@ -2496,7 +2484,6 @@ static int _issue_probereq(struct adapter *padapter, pattrib = &pmgntframe->attrib; update_mgntframe_attrib(padapter, pattrib); - memset(pmgntframe->buf_addr, 0, WLANHDR_OFFSET + TXDESC_OFFSET); pframe = (u8 *)(pmgntframe->buf_addr) + TXDESC_OFFSET; @@ -2640,7 +2627,6 @@ void issue_auth(struct adapter *padapter, struct sta_info *psta, unsigned short pframe += sizeof(struct ieee80211_hdr_3addr); pattrib->pktlen = sizeof(struct ieee80211_hdr_3addr); - if (psta) { /* for AP mode */ memcpy(pwlanhdr->addr1, psta->hwaddr, ETH_ALEN); memcpy(pwlanhdr->addr2, myid(&(padapter->eeprompriv)), ETH_ALEN); @@ -2701,7 +2687,6 @@ void issue_auth(struct adapter *padapter, struct sta_info *psta, unsigned short le_tmp = cpu_to_le16(pmlmeinfo->auth_seq); pframe = rtw_set_fixed_ie(pframe, _AUTH_SEQ_NUM_, (unsigned char *)&le_tmp, &(pattrib->pktlen)); - /* setting status code... */ le_tmp = cpu_to_le16(status); pframe = rtw_set_fixed_ie(pframe, _STATUS_CODE_, (unsigned char *)&le_tmp, &(pattrib->pktlen)); @@ -2730,7 +2715,6 @@ void issue_auth(struct adapter *padapter, struct sta_info *psta, unsigned short dump_mgntframe(padapter, pmgntframe); } - void issue_asocrsp(struct adapter *padapter, unsigned short status, struct sta_info *pstat, int pkt_type) { struct xmit_frame *pmgntframe; @@ -2755,7 +2739,6 @@ void issue_asocrsp(struct adapter *padapter, unsigned short status, struct sta_i pattrib = &pmgntframe->attrib; update_mgntframe_attrib(padapter, pattrib); - memset(pmgntframe->buf_addr, 0, WLANHDR_OFFSET + TXDESC_OFFSET); pframe = (u8 *)(pmgntframe->buf_addr) + TXDESC_OFFSET; @@ -2768,13 +2751,15 @@ void issue_asocrsp(struct adapter *padapter, unsigned short status, struct sta_i memcpy((void *)GetAddr2Ptr(pwlanhdr), myid(&(padapter->eeprompriv)), ETH_ALEN); memcpy((void *)GetAddr3Ptr(pwlanhdr), get_my_bssid(&(pmlmeinfo->network)), ETH_ALEN); - SetSeqNum(pwlanhdr, pmlmeext->mgnt_seq); pmlmeext->mgnt_seq++; - if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP)) + if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP)) { SetFrameSubType(pwlanhdr, pkt_type); - else + } else { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } pattrib->hdrlen = sizeof(struct ieee80211_hdr_3addr); pattrib->pktlen += pattrib->hdrlen; @@ -2927,7 +2912,6 @@ void issue_assocreq(struct adapter *padapter) if (pmlmeext->cur_channel == 14) /* for JAPAN, channel 14 can only uses B Mode(CCK) */ sta_bssrate_len = 4; - /* for (i = 0; i < sta_bssrate_len; i++) { */ /* */ @@ -2936,12 +2920,10 @@ void issue_assocreq(struct adapter *padapter) break; } - for (i = 0; i < NDIS_802_11_LENGTH_RATES_EX; i++) { if (pmlmeinfo->network.supported_rates[i] == 0) break; - /* Check if the AP's supported rates are also supported by STA. */ for (j = 0; j < sta_bssrate_len; j++) { /* Avoid the proprietary data rate (22Mbps) of Handlink WSG-4000 AP */ @@ -2963,7 +2945,6 @@ void issue_assocreq(struct adapter *padapter) goto exit; /* don't connect to AP if no joint supported rate */ } - if (bssrate_len > 8) { pframe = rtw_set_ie(pframe, WLAN_EID_SUPP_RATES, 8, bssrate, &(pattrib->pktlen)); pframe = rtw_set_ie(pframe, WLAN_EID_EXT_SUPP_RATES, (bssrate_len - 8), (bssrate + 8), &(pattrib->pktlen)); @@ -3024,7 +3005,6 @@ void issue_assocreq(struct adapter *padapter) if (pmlmeinfo->assoc_AP_vendor == HT_IOT_PEER_REALTEK) pframe = rtw_set_ie(pframe, WLAN_EID_VENDOR_SPECIFIC, 6, REALTEK_96B_IE, &(pattrib->pktlen)); - pattrib->last_txcmdsz = pattrib->pktlen; dump_mgntframe(padapter, pmgntframe); @@ -3121,7 +3101,6 @@ int issue_nulldata(struct adapter *padapter, unsigned char *da, unsigned int pow struct mlme_ext_info *pmlmeinfo = &(pmlmeext->mlmext_info); struct sta_info *psta; - /* da == NULL, assume it's null data for sta to ap*/ if (!da) da = get_my_bssid(&(pmlmeinfo->network)); @@ -3171,7 +3150,6 @@ s32 issue_nulldata_in_interrupt(struct adapter *padapter, u8 *da) struct mlme_ext_priv *pmlmeext; struct mlme_ext_info *pmlmeinfo; - pmlmeext = &padapter->mlmeextpriv; pmlmeinfo = &pmlmeext->mlmext_info; @@ -3340,7 +3318,6 @@ static int _issue_deauth(struct adapter *padapter, unsigned char *da, pattrib->last_txcmdsz = pattrib->pktlen; - if (wait_ack) { ret = dump_mgntframe_and_wait_ack(padapter, pmgntframe); } else { @@ -3662,7 +3639,6 @@ static void issue_action_BSSCoexistPacket(struct adapter *padapter) pframe = rtw_set_fixed_ie(pframe, 1, &(category), &(pattrib->pktlen)); pframe = rtw_set_fixed_ie(pframe, 1, &(action), &(pattrib->pktlen)); - /* */ if (pmlmepriv->num_FortyMHzIntolerant > 0) { u8 iedata = 0; @@ -3673,7 +3649,6 @@ static void issue_action_BSSCoexistPacket(struct adapter *padapter) } - /* */ memset(ICS, 0, sizeof(ICS)); if (pmlmepriv->num_sta_no_ht > 0) { @@ -3714,7 +3689,6 @@ static void issue_action_BSSCoexistPacket(struct adapter *padapter) spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); - for (i = 0; i < 8; i++) { if (ICS[i][0] == 1) { int j, k = 0; @@ -3742,7 +3716,6 @@ static void issue_action_BSSCoexistPacket(struct adapter *padapter) } - pattrib->last_txcmdsz = pattrib->pktlen; dump_mgntframe(padapter, pmgntframe); @@ -4223,7 +4196,6 @@ void start_clnt_auth(struct adapter *padapter) pmlmeinfo->link_count = 0; pmlmeext->retry = 0; - netdev_dbg(padapter->pnetdev, "start auth\n"); issue_auth(padapter, NULL, 0); @@ -4231,7 +4203,6 @@ void start_clnt_auth(struct adapter *padapter) } - void start_clnt_assoc(struct adapter *padapter) { struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv; @@ -4278,7 +4249,6 @@ static void process_80211d(struct adapter *padapter, struct wlan_bssid_ex *bssid u8 channel; u8 i; - pregistrypriv = &padapter->registrypriv; pmlmeext = &padapter->mlmeextpriv; @@ -4435,12 +4405,12 @@ void report_survey_event(struct adapter *padapter, union recv_frame *precv_frame pmlmeext = &padapter->mlmeextpriv; pcmdpriv = &padapter->cmdpriv; - pcmd_obj = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd_obj = kzalloc(sizeof(*pcmd_obj), GFP_ATOMIC); if (!pcmd_obj) return; cmdsz = (sizeof(struct survey_event) + sizeof(struct C2HEvent_Header)); - pevtcmd = rtw_zmalloc(cmdsz); + pevtcmd = kzalloc(cmdsz, GFP_ATOMIC); if (!pevtcmd) { kfree(pcmd_obj); return; @@ -4488,12 +4458,12 @@ void report_surveydone_event(struct adapter *padapter) struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv; struct cmd_priv *pcmdpriv = &padapter->cmdpriv; - pcmd_obj = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd_obj = kzalloc(sizeof(*pcmd_obj), GFP_ATOMIC); if (!pcmd_obj) return; cmdsz = (sizeof(struct surveydone_event) + sizeof(struct C2HEvent_Header)); - pevtcmd = rtw_zmalloc(cmdsz); + pevtcmd = kzalloc(cmdsz, GFP_ATOMIC); if (!pevtcmd) { kfree(pcmd_obj); return; @@ -4533,12 +4503,12 @@ void report_join_res(struct adapter *padapter, int res) struct mlme_ext_info *pmlmeinfo = &(pmlmeext->mlmext_info); struct cmd_priv *pcmdpriv = &padapter->cmdpriv; - pcmd_obj = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd_obj = kzalloc(sizeof(*pcmd_obj), GFP_ATOMIC); if (!pcmd_obj) return; cmdsz = (sizeof(struct joinbss_event) + sizeof(struct C2HEvent_Header)); - pevtcmd = rtw_zmalloc(cmdsz); + pevtcmd = kzalloc(cmdsz, GFP_ATOMIC); if (!pevtcmd) { kfree(pcmd_obj); return; @@ -4562,10 +4532,8 @@ void report_join_res(struct adapter *padapter, int res) memcpy((unsigned char *)(&(pjoinbss_evt->network.network)), &(pmlmeinfo->network), sizeof(struct wlan_bssid_ex)); pjoinbss_evt->network.join_res = pjoinbss_evt->network.aid = res; - rtw_joinbss_event_prehandle(padapter, (u8 *)&pjoinbss_evt->network); - rtw_enqueue_cmd(pcmdpriv, pcmd_obj); return; @@ -4582,12 +4550,12 @@ void report_wmm_edca_update(struct adapter *padapter) struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv; struct cmd_priv *pcmdpriv = &padapter->cmdpriv; - pcmd_obj = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd_obj = kzalloc(sizeof(*pcmd_obj), GFP_ATOMIC); if (!pcmd_obj) return; cmdsz = (sizeof(struct wmm_event) + sizeof(struct C2HEvent_Header)); - pevtcmd = rtw_zmalloc(cmdsz); + pevtcmd = kzalloc(cmdsz, GFP_ATOMIC); if (!pevtcmd) { kfree(pcmd_obj); return; @@ -4628,12 +4596,12 @@ void report_del_sta_event(struct adapter *padapter, unsigned char *MacAddr, unsi struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv; struct cmd_priv *pcmdpriv = &padapter->cmdpriv; - pcmd_obj = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd_obj = kzalloc(sizeof(*pcmd_obj), GFP_ATOMIC); if (!pcmd_obj) return; cmdsz = (sizeof(struct stadel_event) + sizeof(struct C2HEvent_Header)); - pevtcmd = rtw_zmalloc(cmdsz); + pevtcmd = kzalloc(cmdsz, GFP_ATOMIC); if (!pevtcmd) { kfree(pcmd_obj); return; @@ -4657,7 +4625,6 @@ void report_del_sta_event(struct adapter *padapter, unsigned char *MacAddr, unsi memcpy((unsigned char *)(&(pdel_sta_evt->macaddr)), MacAddr, ETH_ALEN); memcpy((unsigned char *)(pdel_sta_evt->rsvd), (unsigned char *)(&reason), 2); - psta = rtw_get_stainfo(&padapter->stapriv, MacAddr); if (psta) mac_id = (int)psta->mac_id; @@ -4679,12 +4646,12 @@ void report_add_sta_event(struct adapter *padapter, unsigned char *MacAddr, int struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv; struct cmd_priv *pcmdpriv = &padapter->cmdpriv; - pcmd_obj = rtw_zmalloc(sizeof(struct cmd_obj)); + pcmd_obj = kzalloc(sizeof(*pcmd_obj), GFP_ATOMIC); if (!pcmd_obj) return; cmdsz = (sizeof(struct stassoc_event) + sizeof(struct C2HEvent_Header)); - pevtcmd = rtw_zmalloc(cmdsz); + pevtcmd = kzalloc(cmdsz, GFP_ATOMIC); if (!pevtcmd) { kfree(pcmd_obj); return; @@ -4846,7 +4813,6 @@ void mlmeext_joinbss_event_callback(struct adapter *padapter, int join_res) /* update bc/mc sta_info */ update_bmc_sta(padapter); - /* turn on dynamic functions */ Switch_DM_Func(padapter, DYNAMIC_ALL_FUNC_ENABLE, true); @@ -5130,11 +5096,11 @@ void survey_timer_hdl(struct timer_list *t) pmlmeext->scan_abort = false;/* reset */ } - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) return; - psurveyPara = rtw_zmalloc(sizeof(struct sitesurvey_parm)); + psurveyPara = kzalloc(sizeof(*psurveyPara), GFP_ATOMIC); if (!psurveyPara) { kfree(ph2c); return; @@ -5152,7 +5118,6 @@ void link_timer_hdl(struct timer_list *t) struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv; struct mlme_ext_info *pmlmeinfo = &(pmlmeext->mlmext_info); - if (pmlmeinfo->state & WIFI_FW_AUTH_NULL) { pmlmeinfo->state = WIFI_FW_NULL_STATE; report_join_res(padapter, -3); @@ -5339,7 +5304,6 @@ u8 join_cmd_hdl(struct adapter *padapter, u8 *pbuf) /* Set_MSR(padapter, _HW_STATE_NOLINK_); */ Set_MSR(padapter, _HW_STATE_STATION_); - rtw_hal_set_hwreg(padapter, HW_VAR_MLME_DISCONNECT, NULL); } @@ -5748,7 +5712,7 @@ u8 chk_bmc_sleepq_cmd(struct adapter *padapter) struct cmd_priv *pcmdpriv = &(padapter->cmdpriv); u8 res = _SUCCESS; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; @@ -5772,13 +5736,13 @@ u8 set_tx_beacon_cmd(struct adapter *padapter) u8 res = _SUCCESS; int len_diff = 0; - ph2c = rtw_zmalloc(sizeof(struct cmd_obj)); + ph2c = kzalloc(sizeof(*ph2c), GFP_ATOMIC); if (!ph2c) { res = _FAIL; goto exit; } - ptxBeacon_parm = rtw_zmalloc(sizeof(struct Tx_Beacon_param)); + ptxBeacon_parm = kzalloc(sizeof(*ptxBeacon_parm), GFP_ATOMIC); if (!ptxBeacon_parm) { kfree(ph2c); res = _FAIL; diff --git a/drivers/staging/rtl8723bs/core/rtw_recv.c b/drivers/staging/rtl8723bs/core/rtw_recv.c index e893cb6fa273..4727f94df435 100644 --- a/drivers/staging/rtl8723bs/core/rtw_recv.c +++ b/drivers/staging/rtl8723bs/core/rtw_recv.c @@ -60,7 +60,6 @@ signed int _rtw_init_recv_priv(struct recv_priv *precvpriv, struct adapter *pada precvframe = (union recv_frame *) precvpriv->precv_frame_buf; - for (i = 0; i < NR_RECVFRAME; i++) { INIT_LIST_HEAD(&(precvframe->u.list)); @@ -180,9 +179,6 @@ int rtw_free_recvframe(union recv_frame *precvframe, struct __queue *pfree_recv_ return _SUCCESS; } - - - signed int _rtw_enqueue_recvframe(union recv_frame *precvframe, struct __queue *queue) { @@ -192,7 +188,6 @@ signed int _rtw_enqueue_recvframe(union recv_frame *precvframe, struct __queue * /* INIT_LIST_HEAD(&(precvframe->u.hdr.list)); */ list_del_init(&(precvframe->u.hdr.list)); - list_add_tail(&(precvframe->u.hdr.list), get_list_head(queue)); if (padapter) @@ -257,7 +252,6 @@ u32 rtw_free_uc_swdec_pending_queue(struct adapter *adapter) return cnt; } - signed int rtw_enqueue_recvbuf_to_head(struct recv_buf *precvbuf, struct __queue *queue) { spin_lock_bh(&queue->lock); @@ -405,7 +399,6 @@ static signed int recvframe_chkmic(struct adapter *adapter, union recv_frame *p bmic_err = true; } - if (bmic_err == true) { /* double check key_index for some timing issue , */ /* cannot compare with psecuritypriv->dot118021XGrpKeyid also cause timing issue */ @@ -772,8 +765,6 @@ static signed int sta2sta_data_frame(struct adapter *adapter, union recv_frame * } else ret = _FAIL; - - if (bmcast) *psta = rtw_get_bcmc_stainfo(adapter); else @@ -817,7 +808,6 @@ static signed int ap2sta_data_frame(struct adapter *adapter, union recv_frame *p goto exit; } - /* check BSSID */ if (is_zero_ether_addr(pattrib->bssid) || is_zero_ether_addr(mybssid) || @@ -858,14 +848,12 @@ static signed int ap2sta_data_frame(struct adapter *adapter, union recv_frame *p /* */ memcpy(pattrib->bssid, mybssid, ETH_ALEN); - *psta = rtw_get_stainfo(pstapriv, pattrib->bssid); /* get sta_info */ if (!*psta) { ret = _FAIL; goto exit; } - } else if (check_fwstate(pmlmepriv, WIFI_AP_STATE) == true) { /* Special case */ ret = RTW_RX_HANDLED; @@ -1109,7 +1097,6 @@ static union recv_frame *recvframe_defrag(struct adapter *adapter, pnextrframe = (union recv_frame *)plist; pnfhdr = &pnextrframe->u.hdr; - /* check the fragment sequence (2nd ~n fragment frame) */ if (curfragnum != pnfhdr->attrib.frag_num) { @@ -1197,7 +1184,6 @@ static union recv_frame *recvframe_chk_defrag(struct adapter *padapter, union re /* free current defrag_q */ rtw_free_recvframe_queue(pdefrag_q, pfree_recv_queue); - /* Then enqueue the 0~(n-1) fragment into the defrag_q */ /* spin_lock(&pdefrag_q->lock); */ @@ -1236,7 +1222,6 @@ static union recv_frame *recvframe_chk_defrag(struct adapter *padapter, union re } - if ((prtnframe) && (prtnframe->u.hdr.attrib.privacy)) { /* after defrag we must check tkip mic code */ if (recvframe_chkmic(padapter, prtnframe) == _FAIL) { @@ -1345,7 +1330,6 @@ static signed int validate_recv_data_frame(struct adapter *adapter, union recv_f goto exit; } - if (!psta) { ret = _FAIL; goto exit; @@ -1355,7 +1339,6 @@ static signed int validate_recv_data_frame(struct adapter *adapter, union recv_f /* psta->signal_quality = prxcmd->sq; */ precv_frame->u.hdr.psta = psta; - pattrib->amsdu = 0; pattrib->ack_policy = 0; /* parsing QC field */ @@ -1373,7 +1356,6 @@ static signed int validate_recv_data_frame(struct adapter *adapter, union recv_f pattrib->hdrlen = pattrib->to_fr_ds == 3 ? 30 : 24; } - if (pattrib->order)/* HT-CTRL 11n */ pattrib->hdrlen += 4; @@ -1425,7 +1407,7 @@ static signed int validate_80211w_mgmt(struct adapter *adapter, union recv_frame memcpy(pattrib->ta, GetAddr2Ptr(ptr), ETH_ALEN); /* actual management data frame body */ data_len = pattrib->pkt_len - pattrib->hdrlen - pattrib->iv_len - pattrib->icv_len; - mgmt_DATA = rtw_zmalloc(data_len); + mgmt_DATA = kzalloc(data_len, GFP_ATOMIC); if (!mgmt_DATA) goto validate_80211w_fail; precv_frame = decryptor(adapter, precv_frame); @@ -1827,7 +1809,6 @@ static int enqueue_reorder_recvframe(struct recv_reorder_ctrl *preorder_ctrl, un /* spin_lock_irqsave(&ppending_recvframe_queue->lock, irql); */ /* spin_lock(&ppending_recvframe_queue->lock); */ - phead = get_list_head(ppending_recvframe_queue); plist = get_next(phead); @@ -1846,7 +1827,6 @@ static int enqueue_reorder_recvframe(struct recv_reorder_ctrl *preorder_ctrl, un } - /* spin_lock_irqsave(&ppending_recvframe_queue->lock, irql); */ /* spin_lock(&ppending_recvframe_queue->lock); */ @@ -1975,7 +1955,6 @@ static int recv_indicatepkts_in_order(struct adapter *padapter, struct recv_reor /* error condition; */ } - /* Update local variables. */ bPktInBuf = false; @@ -2058,7 +2037,6 @@ static int recv_indicatepkt_reorder(struct adapter *padapter, union recv_frame * goto _err_exit; } - /* s4. */ /* Indication process. */ /* After Packet dropping and Sliding Window shifting as above, we can now just indicate the packets */ @@ -2086,7 +2064,6 @@ static int recv_indicatepkt_reorder(struct adapter *padapter, union recv_frame * return _FAIL; } - void rtw_reordering_ctrl_timeout_handler(struct timer_list *t) { struct recv_reorder_ctrl *preorder_ctrl = @@ -2094,7 +2071,6 @@ void rtw_reordering_ctrl_timeout_handler(struct timer_list *t) struct adapter *padapter = preorder_ctrl->padapter; struct __queue *ppending_recvframe_queue = &preorder_ctrl->pending_recvframe_queue; - if (padapter->bDriverStopped || padapter->bSurpriseRemoved) return; @@ -2242,7 +2218,6 @@ static int recv_func(struct adapter *padapter, union recv_frame *rframe) return ret; } - s32 rtw_recv_entry(union recv_frame *precvframe) { struct adapter *padapter; diff --git a/drivers/staging/rtl8723bs/core/rtw_security.c b/drivers/staging/rtl8723bs/core/rtw_security.c index 5406f60dd31c..6b5c9831af02 100644 --- a/drivers/staging/rtl8723bs/core/rtw_security.c +++ b/drivers/staging/rtl8723bs/core/rtw_security.c @@ -62,14 +62,14 @@ void rtw_wep_encrypt(struct adapter *padapter, u8 *pxmitframe) keylength = psecuritypriv->dot11DefKeylen[psecuritypriv->dot11PrivacyKeyIndex]; for (curfragnum = 0; curfragnum < pattrib->nr_frags; curfragnum++) { - iv = pframe+pattrib->hdrlen; + iv = pframe + pattrib->hdrlen; memcpy(&wepkey[0], iv, 3); memcpy(&wepkey[3], &psecuritypriv->dot11DefKey[psecuritypriv->dot11PrivacyKeyIndex].skey[0], keylength); - payload = pframe+pattrib->iv_len+pattrib->hdrlen; + payload = pframe + pattrib->iv_len + pattrib->hdrlen; - if ((curfragnum+1) == pattrib->nr_frags) { /* the last fragment */ + if ((curfragnum + 1) == pattrib->nr_frags) { /* the last fragment */ - length = pattrib->last_txcmdsz-pattrib->hdrlen-pattrib->iv_len-pattrib->icv_len; + length = pattrib->last_txcmdsz - pattrib->hdrlen - pattrib->iv_len - pattrib->icv_len; crc.f0 = cpu_to_le32(~crc32_le(~0, payload, length)); @@ -78,7 +78,7 @@ void rtw_wep_encrypt(struct adapter *padapter, u8 *pxmitframe) arc4_crypt(ctx, payload + length, crc.f1, 4); } else { - length = pxmitpriv->frag_len-pattrib->hdrlen-pattrib->iv_len-pattrib->icv_len; + length = pxmitpriv->frag_len - pattrib->hdrlen - pattrib->iv_len - pattrib->icv_len; crc.f0 = cpu_to_le32(~crc32_le(~0, payload, length)); arc4_setkey(ctx, wepkey, 3 + keylength); arc4_crypt(ctx, payload, payload, length); @@ -107,16 +107,16 @@ void rtw_wep_decrypt(struct adapter *padapter, u8 *precvframe) /* start to decrypt recvframe */ if ((prxattrib->encrypt == _WEP40_) || (prxattrib->encrypt == _WEP104_)) { - iv = pframe+prxattrib->hdrlen; + iv = pframe + prxattrib->hdrlen; /* keyindex =(iv[3]&0x3); */ keyindex = prxattrib->key_index; keylength = psecuritypriv->dot11DefKeylen[keyindex]; memcpy(&wepkey[0], iv, 3); /* memcpy(&wepkey[3], &psecuritypriv->dot11DefKey[psecuritypriv->dot11PrivacyKeyIndex].skey[0], keylength); */ memcpy(&wepkey[3], &psecuritypriv->dot11DefKey[keyindex].skey[0], keylength); - length = ((union recv_frame *)precvframe)->u.hdr.len-prxattrib->hdrlen-prxattrib->iv_len; + length = ((union recv_frame *)precvframe)->u.hdr.len - prxattrib->hdrlen - prxattrib->iv_len; - payload = pframe+prxattrib->iv_len+prxattrib->hdrlen; + payload = pframe + prxattrib->iv_len + prxattrib->hdrlen; /* decrypt payload include icv */ arc4_setkey(ctx, wepkey, 3 + keylength); @@ -174,7 +174,7 @@ void rtw_secmicsetkey(struct mic_data *pmicdata, u8 *key) void rtw_secmicappendbyte(struct mic_data *pmicdata, u8 b) { /* Append the byte to our word-sized buffer */ - pmicdata->M |= ((unsigned long)b) << (8*pmicdata->nBytesInM); + pmicdata->M |= ((unsigned long)b) << (8 * pmicdata->nBytesInM); pmicdata->nBytesInM++; /* Process the word if it is full. */ if (pmicdata->nBytesInM >= 4) { @@ -261,7 +261,7 @@ void rtw_seccalctkipmic(u8 *key, u8 *header, u8 *data, u32 data_len, u8 *mic_cod #define Mk16(hi, lo) ((lo) ^ (((u16)(hi)) << 8)) /* select the Nth 16-bit word of the temporal key unsigned char array TK[] */ -#define TK16(N) Mk16(tk[2*(N)+1], tk[2*(N)]) +#define TK16(N) Mk16(tk[2 * (N) + 1], tk[2 * (N)]) /* S-box lookup: 16 bits --> 16 bits */ #define _S_(v16) (Sbox1[0][Lo8(v16)] ^ Sbox1[1][Hi8(v16)]) @@ -375,11 +375,11 @@ static void phase1(u16 *p1k, const u8 *tk, const u8 *ta, u32 iv32) /* size on the 80-bit block P1K[], using the 128-bit key TK[] */ for (i = 0; i < PHASE1_LOOP_CNT; i++) { /* Each add operation here is mod 2**16 */ - p1k[0] += _S_(p1k[4] ^ TK16((i&1)+0)); - p1k[1] += _S_(p1k[0] ^ TK16((i&1)+2)); - p1k[2] += _S_(p1k[1] ^ TK16((i&1)+4)); - p1k[3] += _S_(p1k[2] ^ TK16((i&1)+6)); - p1k[4] += _S_(p1k[3] ^ TK16((i&1)+0)); + p1k[0] += _S_(p1k[4] ^ TK16((i & 1) + 0)); + p1k[1] += _S_(p1k[0] ^ TK16((i & 1) + 2)); + p1k[2] += _S_(p1k[1] ^ TK16((i & 1) + 4)); + p1k[3] += _S_(p1k[2] ^ TK16((i & 1) + 6)); + p1k[4] += _S_(p1k[3] ^ TK16((i & 1) + 0)); p1k[4] += (unsigned short)i; /* avoid "slide attacks" */ } } @@ -417,7 +417,7 @@ static void phase2(u8 *rc4key, const u8 *tk, const u16 *p1k, u16 iv16) for (i = 0; i < 5; i++) PPK[i] = p1k[i]; /* first, copy P1K to PPK */ - PPK[5] = p1k[4]+iv16; /* next, add in IV16 */ + PPK[5] = p1k[4] + iv16; /* next, add in IV16 */ /* Bijective non-linear mixing of the 96 bits of PPK[0..5] */ PPK[0] += _S_(PPK[5] ^ TK16(0)); /* Mix key in each "round" */ @@ -448,8 +448,8 @@ static void phase2(u8 *rc4key, const u8 *tk, const u16 *p1k, u16 iv16) /* Copy 96 bits of PPK[0..5] to RC4KEY[4..15] (little-endian) */ for (i = 0; i < 6; i++) { - rc4key[4+2*i] = Lo8(PPK[i]); - rc4key[5+2*i] = Hi8(PPK[i]); + rc4key[4 + 2 * i] = Lo8(PPK[i]); + rc4key[5 + 2 * i] = Hi8(PPK[i]); } } @@ -492,20 +492,20 @@ u32 rtw_tkip_encrypt(struct adapter *padapter, u8 *pxmitframe) prwskey = pattrib->dot118021x_UncstKey.skey; for (curfragnum = 0; curfragnum < pattrib->nr_frags; curfragnum++) { - iv = pframe+pattrib->hdrlen; - payload = pframe+pattrib->iv_len+pattrib->hdrlen; + iv = pframe + pattrib->hdrlen; + payload = pframe + pattrib->iv_len + pattrib->hdrlen; GET_TKIP_PN(iv, dot11txpn); pnl = (u16)(dot11txpn.val); - pnh = (u32)(dot11txpn.val>>16); + pnh = (u32)(dot11txpn.val >> 16); phase1((u16 *)&ttkey[0], prwskey, &pattrib->ta[0], pnh); phase2(&rc4key[0], prwskey, (u16 *)&ttkey[0], pnl); - if ((curfragnum+1) == pattrib->nr_frags) { /* 4 the last fragment */ - length = pattrib->last_txcmdsz-pattrib->hdrlen-pattrib->iv_len-pattrib->icv_len; + if ((curfragnum + 1) == pattrib->nr_frags) { /* 4 the last fragment */ + length = pattrib->last_txcmdsz - pattrib->hdrlen - pattrib->iv_len - pattrib->icv_len; crc.f0 = cpu_to_le32(~crc32_le(~0, payload, length)); arc4_setkey(ctx, rc4key, 16); @@ -513,7 +513,7 @@ u32 rtw_tkip_encrypt(struct adapter *padapter, u8 *pxmitframe) arc4_crypt(ctx, payload + length, crc.f1, 4); } else { - length = pxmitpriv->frag_len-pattrib->hdrlen-pattrib->iv_len-pattrib->icv_len; + length = pxmitpriv->frag_len - pattrib->hdrlen - pattrib->iv_len - pattrib->icv_len; crc.f0 = cpu_to_le32(~crc32_le(~0, payload, length)); arc4_setkey(ctx, rc4key, 16); @@ -601,14 +601,14 @@ u32 rtw_tkip_decrypt(struct adapter *padapter, u8 *precvframe) prwskey = &stainfo->dot118021x_UncstKey.skey[0]; } - iv = pframe+prxattrib->hdrlen; - payload = pframe+prxattrib->iv_len+prxattrib->hdrlen; - length = ((union recv_frame *)precvframe)->u.hdr.len-prxattrib->hdrlen-prxattrib->iv_len; + iv = pframe + prxattrib->hdrlen; + payload = pframe + prxattrib->iv_len + prxattrib->hdrlen; + length = ((union recv_frame *)precvframe)->u.hdr.len - prxattrib->hdrlen - prxattrib->iv_len; GET_TKIP_PN(iv, dot11txpn); pnl = (u16)(dot11txpn.val); - pnh = (u32)(dot11txpn.val>>16); + pnh = (u32)(dot11txpn.val >> 16); phase1((u16 *)&ttkey[0], prwskey, &prxattrib->ta[0], pnh); phase2(&rc4key[0], prwskey, (unsigned short *)&ttkey[0], pnl); @@ -758,7 +758,7 @@ static void construct_mic_header2(u8 *mic_header2, if (!qc_exists && a4_exists) { for (i = 0; i < 6; i++) - mic_header2[8+i] = mpdu[24+i]; /* A4 */ + mic_header2[8 + i] = mpdu[24 + i]; /* A4 */ } if (qc_exists && !a4_exists) { @@ -768,7 +768,7 @@ static void construct_mic_header2(u8 *mic_header2, if (qc_exists && a4_exists) { for (i = 0; i < 6; i++) - mic_header2[8+i] = mpdu[24+i]; /* A4 */ + mic_header2[8 + i] = mpdu[24 + i]; /* A4 */ mic_header2[14] = mpdu[30] & 0x0f; mic_header2[15] = mpdu[31] & 0x00; @@ -839,16 +839,16 @@ static signed int aes_cipher(u8 *key, uint hdrlen, uint frtype = GetFrameType(pframe); uint frsubtype = GetFrameSubType(pframe); - frsubtype = frsubtype>>4; + frsubtype = frsubtype >> 4; if ((hdrlen == WLAN_HDR_A3_LEN) || (hdrlen == WLAN_HDR_A3_QOS_LEN)) a4_exists = 0; else a4_exists = 1; - if (((frtype|frsubtype) == WIFI_DATA_CFACK) || - ((frtype|frsubtype) == WIFI_DATA_CFPOLL) || - ((frtype|frsubtype) == WIFI_DATA_CFACKPOLL)) { + if (((frtype | frsubtype) == WIFI_DATA_CFACK) || + ((frtype | frsubtype) == WIFI_DATA_CFPOLL) || + ((frtype | frsubtype) == WIFI_DATA_CFACKPOLL)) { qc_exists = 1; if (hdrlen != WLAN_HDR_A3_QOS_LEN) hdrlen += 2; @@ -867,11 +867,11 @@ static signed int aes_cipher(u8 *key, uint hdrlen, } pn_vector[0] = pframe[hdrlen]; - pn_vector[1] = pframe[hdrlen+1]; - pn_vector[2] = pframe[hdrlen+4]; - pn_vector[3] = pframe[hdrlen+5]; - pn_vector[4] = pframe[hdrlen+6]; - pn_vector[5] = pframe[hdrlen+7]; + pn_vector[1] = pframe[hdrlen + 1]; + pn_vector[2] = pframe[hdrlen + 4]; + pn_vector[3] = pframe[hdrlen + 5]; + pn_vector[4] = pframe[hdrlen + 6]; + pn_vector[5] = pframe[hdrlen + 7]; construct_mic_iv(mic_iv, qc_exists, @@ -927,12 +927,12 @@ static signed int aes_cipher(u8 *key, uint hdrlen, /* Insert MIC into payload */ for (j = 0; j < 8; j++) - pframe[payload_index+j] = mic[j]; + pframe[payload_index + j] = mic[j]; payload_index = hdrlen + 8; for (i = 0; i < num_blocks; i++) { construct_ctr_preload(ctr_preload, a4_exists, qc_exists, pframe, /* message, */ - pn_vector, i+1, frtype); + pn_vector, i + 1, frtype); /* add for CONFIG_IEEE80211W, none 11w also can use */ aes128k128d(key, ctr_preload, aes_out); crypto_xor_cpy(chain_buffer, aes_out, &pframe[payload_index], 16); @@ -944,13 +944,13 @@ static signed int aes_cipher(u8 *key, uint hdrlen, /* If there is a short final block, then pad it,*/ /* encrypt it and copy the unpadded part back */ construct_ctr_preload(ctr_preload, a4_exists, qc_exists, pframe, /* message, */ - pn_vector, num_blocks+1, frtype); + pn_vector, num_blocks + 1, frtype); /* add for CONFIG_IEEE80211W, none 11w also can use */ for (j = 0; j < 16; j++) padded_buffer[j] = 0x00; for (j = 0; j < payload_remainder; j++) - padded_buffer[j] = pframe[payload_index+j]; + padded_buffer[j] = pframe[payload_index + j]; aes128k128d(key, ctr_preload, aes_out); crypto_xor_cpy(chain_buffer, aes_out, padded_buffer, 16); @@ -966,7 +966,7 @@ static signed int aes_cipher(u8 *key, uint hdrlen, for (j = 0; j < 16; j++) padded_buffer[j] = 0x00; for (j = 0; j < 8; j++) - padded_buffer[j] = pframe[j+hdrlen+8+plen]; + padded_buffer[j] = pframe[j + hdrlen + 8 + plen]; aes128k128d(key, ctr_preload, aes_out); crypto_xor_cpy(chain_buffer, aes_out, padded_buffer, 16); @@ -1006,12 +1006,12 @@ u32 rtw_aes_encrypt(struct adapter *padapter, u8 *pxmitframe) prwskey = pattrib->dot118021x_UncstKey.skey; for (curfragnum = 0; curfragnum < pattrib->nr_frags; curfragnum++) { - if ((curfragnum+1) == pattrib->nr_frags) { /* 4 the last fragment */ - length = pattrib->last_txcmdsz-pattrib->hdrlen-pattrib->iv_len-pattrib->icv_len; + if ((curfragnum + 1) == pattrib->nr_frags) { /* 4 the last fragment */ + length = pattrib->last_txcmdsz - pattrib->hdrlen - pattrib->iv_len - pattrib->icv_len; aes_cipher(prwskey, pattrib->hdrlen, pframe, length); } else { - length = pxmitpriv->frag_len-pattrib->hdrlen-pattrib->iv_len-pattrib->icv_len; + length = pxmitpriv->frag_len - pattrib->hdrlen - pattrib->iv_len - pattrib->icv_len; aes_cipher(prwskey, pattrib->hdrlen, pframe, length); pframe += pxmitpriv->frag_len; @@ -1044,13 +1044,13 @@ static signed int aes_decipher(u8 *key, uint hdrlen, uint frtype = GetFrameType(pframe); uint frsubtype = GetFrameSubType(pframe); - frsubtype = frsubtype>>4; + frsubtype = frsubtype >> 4; /* start to decrypt the payload */ - num_blocks = (plen-8) / 16; /* plen including LLC, payload_length and mic) */ + num_blocks = (plen - 8) / 16; /* plen including LLC, payload_length and mic) */ - payload_remainder = (plen-8) % 16; + payload_remainder = (plen - 8) % 16; pn_vector[0] = pframe[hdrlen]; pn_vector[1] = pframe[hdrlen + 1]; @@ -1064,9 +1064,9 @@ static signed int aes_decipher(u8 *key, uint hdrlen, else a4_exists = 1; - if (((frtype|frsubtype) == WIFI_DATA_CFACK) || - ((frtype|frsubtype) == WIFI_DATA_CFPOLL) || - ((frtype|frsubtype) == WIFI_DATA_CFACKPOLL)) { + if (((frtype | frsubtype) == WIFI_DATA_CFACK) || + ((frtype | frsubtype) == WIFI_DATA_CFPOLL) || + ((frtype | frsubtype) == WIFI_DATA_CFACKPOLL)) { qc_exists = 1; if (hdrlen != WLAN_HDR_A3_QOS_LEN) hdrlen += 2; @@ -1105,13 +1105,13 @@ static signed int aes_decipher(u8 *key, uint hdrlen, /* If there is a short final block, then pad it,*/ /* encrypt it and copy the unpadded part back */ construct_ctr_preload(ctr_preload, a4_exists, qc_exists, pframe, pn_vector, - num_blocks+1, frtype); + num_blocks + 1, frtype); /* add for CONFIG_IEEE80211W, none 11w also can use */ for (j = 0; j < 16; j++) padded_buffer[j] = 0x00; for (j = 0; j < payload_remainder; j++) - padded_buffer[j] = pframe[payload_index+j]; + padded_buffer[j] = pframe[payload_index + j]; aes128k128d(key, ctr_preload, aes_out); crypto_xor_cpy(chain_buffer, aes_out, padded_buffer, 16); @@ -1120,25 +1120,25 @@ static signed int aes_decipher(u8 *key, uint hdrlen, } /* start to calculate the mic */ - if ((hdrlen + plen+8) <= MAX_MSG_SIZE) - memcpy((void *)message, pframe, (hdrlen + plen+8)); /* 8 is for ext iv len */ + if ((hdrlen + plen + 8) <= MAX_MSG_SIZE) + memcpy((void *)message, pframe, (hdrlen + plen + 8)); /* 8 is for ext iv len */ pn_vector[0] = pframe[hdrlen]; - pn_vector[1] = pframe[hdrlen+1]; - pn_vector[2] = pframe[hdrlen+4]; - pn_vector[3] = pframe[hdrlen+5]; - pn_vector[4] = pframe[hdrlen+6]; - pn_vector[5] = pframe[hdrlen+7]; + pn_vector[1] = pframe[hdrlen + 1]; + pn_vector[2] = pframe[hdrlen + 4]; + pn_vector[3] = pframe[hdrlen + 5]; + pn_vector[4] = pframe[hdrlen + 6]; + pn_vector[5] = pframe[hdrlen + 7]; - construct_mic_iv(mic_iv, qc_exists, a4_exists, message, plen-8, pn_vector, frtype); + construct_mic_iv(mic_iv, qc_exists, a4_exists, message, plen - 8, pn_vector, frtype); /* add for CONFIG_IEEE80211W, none 11w also can use */ construct_mic_header1(mic_header1, hdrlen, message, frtype); /* add for CONFIG_IEEE80211W, none 11w also can use */ construct_mic_header2(mic_header2, message, a4_exists, qc_exists); - payload_remainder = (plen-8) % 16; - num_blocks = (plen-8) / 16; + payload_remainder = (plen - 8) % 16; + num_blocks = (plen - 8) / 16; /* Find start of payload */ payload_index = (hdrlen + 8); @@ -1173,11 +1173,11 @@ static signed int aes_decipher(u8 *key, uint hdrlen, /* Insert MIC into payload */ for (j = 0; j < 8; j++) - message[payload_index+j] = mic[j]; + message[payload_index + j] = mic[j]; payload_index = hdrlen + 8; for (i = 0; i < num_blocks; i++) { - construct_ctr_preload(ctr_preload, a4_exists, qc_exists, message, pn_vector, i+1, + construct_ctr_preload(ctr_preload, a4_exists, qc_exists, message, pn_vector, i + 1, frtype); /* add for CONFIG_IEEE80211W, none 11w also can use */ aes128k128d(key, ctr_preload, aes_out); @@ -1190,13 +1190,13 @@ static signed int aes_decipher(u8 *key, uint hdrlen, /* If there is a short final block, then pad it,*/ /* encrypt it and copy the unpadded part back */ construct_ctr_preload(ctr_preload, a4_exists, qc_exists, message, pn_vector, - num_blocks+1, frtype); + num_blocks + 1, frtype); /* add for CONFIG_IEEE80211W, none 11w also can use */ for (j = 0; j < 16; j++) padded_buffer[j] = 0x00; for (j = 0; j < payload_remainder; j++) - padded_buffer[j] = message[payload_index+j]; + padded_buffer[j] = message[payload_index + j]; aes128k128d(key, ctr_preload, aes_out); crypto_xor_cpy(chain_buffer, aes_out, padded_buffer, 16); @@ -1211,7 +1211,7 @@ static signed int aes_decipher(u8 *key, uint hdrlen, for (j = 0; j < 16; j++) padded_buffer[j] = 0x00; for (j = 0; j < 8; j++) - padded_buffer[j] = message[j+hdrlen+8+plen-8]; + padded_buffer[j] = message[j + hdrlen + 8 + plen - 8]; aes128k128d(key, ctr_preload, aes_out); crypto_xor_cpy(chain_buffer, aes_out, padded_buffer, 16); @@ -1298,7 +1298,7 @@ u32 rtw_aes_decrypt(struct adapter *padapter, u8 *precvframe) prwskey = &stainfo->dot118021x_UncstKey.skey[0]; } - length = ((union recv_frame *)precvframe)->u.hdr.len-prxattrib->hdrlen-prxattrib->iv_len; + length = ((union recv_frame *)precvframe)->u.hdr.len - prxattrib->hdrlen - prxattrib->iv_len; res = aes_decipher(prwskey, prxattrib->hdrlen, pframe, length); @@ -1323,9 +1323,8 @@ u32 rtw_BIP_verify(struct adapter *padapter, u8 *precvframe) __le16 le_tmp; __le64 le_tmp64 = 0; - ori_len = pattrib->pkt_len-WLAN_HDR_A3_LEN+BIP_AAD_SIZE; - BIP_AAD = rtw_zmalloc(ori_len); - + ori_len = pattrib->pkt_len - WLAN_HDR_A3_LEN + BIP_AAD_SIZE; + BIP_AAD = kzalloc(ori_len, GFP_KERNEL); if (!BIP_AAD) return _FAIL; @@ -1334,28 +1333,28 @@ u32 rtw_BIP_verify(struct adapter *padapter, u8 *precvframe) /* mapping to wlan header */ pwlanhdr = (struct ieee80211_hdr *)pframe; /* save the frame body + MME */ - memcpy(BIP_AAD+BIP_AAD_SIZE, pframe+WLAN_HDR_A3_LEN, pattrib->pkt_len-WLAN_HDR_A3_LEN); + memcpy(BIP_AAD + BIP_AAD_SIZE, pframe + WLAN_HDR_A3_LEN, pattrib->pkt_len - WLAN_HDR_A3_LEN); /* find MME IE pointer */ - p = rtw_get_ie(BIP_AAD+BIP_AAD_SIZE, WLAN_EID_MMIE, &len, pattrib->pkt_len-WLAN_HDR_A3_LEN); + p = rtw_get_ie(BIP_AAD + BIP_AAD_SIZE, WLAN_EID_MMIE, &len, pattrib->pkt_len - WLAN_HDR_A3_LEN); /* Baron */ if (p) { u16 keyid = 0; u64 temp_ipn = 0; /* save packet number */ - memcpy(&le_tmp64, p+4, 6); + memcpy(&le_tmp64, p + 4, 6); temp_ipn = le64_to_cpu(le_tmp64); /* BIP packet number should bigger than previous BIP packet */ if (temp_ipn <= pmlmeext->mgnt_80211w_IPN_rx) goto BIP_exit; /* copy key index */ - memcpy(&le_tmp, p+2, 2); + memcpy(&le_tmp, p + 2, 2); keyid = le16_to_cpu(le_tmp); if (keyid != padapter->securitypriv.dot11wBIPKeyid) goto BIP_exit; /* clear the MIC field of MME to zero */ - memset(p+2+len-8, 0, 8); + memset(p + 2 + len - 8, 0, 8); /* conscruct AAD, copy frame control field */ memcpy(BIP_AAD, &pwlanhdr->frame_control, 2); @@ -1515,7 +1514,7 @@ u8 rtw_handle_tkip_countermeasure(struct adapter *adapter, const char *caller) if (securitypriv->btkip_countermeasure) { unsigned long passing_ms = jiffies_to_msecs(jiffies - securitypriv->btkip_countermeasure_time); - if (passing_ms > 60*1000) { + if (passing_ms > 60 * 1000) { netdev_dbg(adapter->pnetdev, "%s(%s) countermeasure time:%lus > 60s\n", caller, ADPT_ARG(adapter), diff --git a/drivers/staging/rtl8723bs/core/rtw_wlan_util.c b/drivers/staging/rtl8723bs/core/rtw_wlan_util.c index 671b5a01aa72..ccfbef80c00c 100644 --- a/drivers/staging/rtl8723bs/core/rtw_wlan_util.c +++ b/drivers/staging/rtl8723bs/core/rtw_wlan_util.c @@ -341,7 +341,6 @@ void set_channel_bwmode(struct adapter *padapter, unsigned char channel, unsigne center_ch = rtw_get_center_ch(channel, bwmode, channel_offset); - /* set Channel */ if (mutex_lock_interruptible(&(adapter_to_dvobj(padapter)->setch_mutex))) return; @@ -1136,7 +1135,7 @@ int rtw_check_bcn_info(struct adapter *Adapter, u8 *pframe, u32 packet_len) if (memcmp(cur_network->network.mac_address, pbssid, 6)) return true; - bssid = rtw_zmalloc(sizeof(struct wlan_bssid_ex)); + bssid = kzalloc(sizeof(*bssid), GFP_KERNEL); if (!bssid) return true; diff --git a/drivers/staging/rtl8723bs/core/rtw_xmit.c b/drivers/staging/rtl8723bs/core/rtw_xmit.c index 21690857fd62..c9e0b91d75d3 100644 --- a/drivers/staging/rtl8723bs/core/rtw_xmit.c +++ b/drivers/staging/rtl8723bs/core/rtw_xmit.c @@ -78,7 +78,7 @@ s32 _rtw_init_xmit_priv(struct xmit_priv *pxmitpriv, struct adapter *padapter) } pxmitpriv->pxmit_frame_buf = (u8 *)N_BYTE_ALIGMENT((SIZE_PTR)(pxmitpriv->pallocated_frame_buf), 4); - pxframe = (struct xmit_frame *) pxmitpriv->pxmit_frame_buf; + pxframe = (struct xmit_frame *)pxmitpriv->pxmit_frame_buf; for (i = 0; i < NR_XMITFRAME; i++) { INIT_LIST_HEAD(&pxframe->list); @@ -238,7 +238,9 @@ s32 _rtw_init_xmit_priv(struct xmit_priv *pxmitpriv, struct adapter *padapter) pxmitbuf->padapter = padapter; pxmitbuf->buf_tag = XMITBUF_CMD; - res = rtw_os_xmit_resource_alloc(padapter, pxmitbuf, MAX_CMDBUF_SZ+XMITBUF_ALIGN_SZ, true); + res = rtw_os_xmit_resource_alloc(padapter, pxmitbuf, + MAX_CMDBUF_SZ + XMITBUF_ALIGN_SZ, + true); if (res == _FAIL) { res = _FAIL; goto exit; @@ -248,7 +250,7 @@ s32 _rtw_init_xmit_priv(struct xmit_priv *pxmitpriv, struct adapter *padapter) pxmitbuf->pend = pxmitbuf->pbuf + MAX_CMDBUF_SZ; pxmitbuf->len = 0; pxmitbuf->pdata = pxmitbuf->ptail = pxmitbuf->phead; - pxmitbuf->alloc_sz = MAX_CMDBUF_SZ+XMITBUF_ALIGN_SZ; + pxmitbuf->alloc_sz = MAX_CMDBUF_SZ + XMITBUF_ALIGN_SZ; } } @@ -274,7 +276,7 @@ void _rtw_free_xmit_priv(struct xmit_priv *pxmitpriv) { int i; struct adapter *padapter = pxmitpriv->adapter; - struct xmit_frame *pxmitframe = (struct xmit_frame *) pxmitpriv->pxmit_frame_buf; + struct xmit_frame *pxmitframe = (struct xmit_frame *)pxmitpriv->pxmit_frame_buf; struct xmit_buf *pxmitbuf = (struct xmit_buf *)pxmitpriv->pxmitbuf; rtw_hal_free_xmit_priv(padapter); @@ -321,7 +323,9 @@ void _rtw_free_xmit_priv(struct xmit_priv *pxmitpriv) for (i = 0; i < CMDBUF_MAX; i++) { pxmitbuf = &pxmitpriv->pcmd_xmitbuf[i]; if (pxmitbuf) - rtw_os_xmit_resource_free(padapter, pxmitbuf, MAX_CMDBUF_SZ+XMITBUF_ALIGN_SZ, true); + rtw_os_xmit_resource_free(padapter, pxmitbuf, + MAX_CMDBUF_SZ + XMITBUF_ALIGN_SZ, + true); } rtw_free_hwxmits(padapter); @@ -736,7 +740,7 @@ static s32 update_attrib(struct adapter *padapter, struct sk_buff *pkt, struct p pattrib->subtype = WIFI_DATA_TYPE; pattrib->priority = 0; - if (check_fwstate(pmlmepriv, WIFI_AP_STATE|WIFI_ADHOC_STATE|WIFI_ADHOC_MASTER_STATE)) { + if (check_fwstate(pmlmepriv, WIFI_AP_STATE | WIFI_ADHOC_STATE | WIFI_ADHOC_MASTER_STATE)) { if (pattrib->qos_en) set_qos(&pktfile, pattrib); } else { @@ -787,15 +791,15 @@ static s32 xmitframe_addmic(struct adapter *padapter, struct xmit_frame *pxmitfr rtw_secmicsetkey(&micdata, &pattrib->dot11tkiptxmickey.skey[0]); } - if (pframe[1]&1) { /* ToDS == 1 */ + if (pframe[1] & 1) { /* ToDS == 1 */ rtw_secmicappend(&micdata, &pframe[16], 6); /* DA */ - if (pframe[1]&2) /* From Ds == 1 */ + if (pframe[1] & 2) /* From Ds == 1 */ rtw_secmicappend(&micdata, &pframe[24], 6); else rtw_secmicappend(&micdata, &pframe[10], 6); } else { /* ToDS == 0 */ rtw_secmicappend(&micdata, &pframe[4], 6); /* DA */ - if (pframe[1]&2) /* From Ds == 1 */ + if (pframe[1] & 2) /* From Ds == 1 */ rtw_secmicappend(&micdata, &pframe[16], 6); else rtw_secmicappend(&micdata, &pframe[10], 6); @@ -810,16 +814,20 @@ static s32 xmitframe_addmic(struct adapter *padapter, struct xmit_frame *pxmitfr for (curfragnum = 0; curfragnum < pattrib->nr_frags; curfragnum++) { payload = (u8 *)round_up((SIZE_PTR)(payload), 4); - payload = payload+pattrib->hdrlen+pattrib->iv_len; + payload = payload + pattrib->hdrlen + pattrib->iv_len; - if ((curfragnum+1) == pattrib->nr_frags) { - length = pattrib->last_txcmdsz-pattrib->hdrlen-pattrib->iv_len-((pattrib->bswenc) ? pattrib->icv_len : 0); + if ((curfragnum + 1) == pattrib->nr_frags) { + length = pattrib->last_txcmdsz - pattrib->hdrlen - + pattrib->iv_len - + ((pattrib->bswenc) ? pattrib->icv_len : 0); rtw_secmicappend(&micdata, payload, length); - payload = payload+length; + payload = payload + length; } else { - length = pxmitpriv->frag_len-pattrib->hdrlen-pattrib->iv_len-((pattrib->bswenc) ? pattrib->icv_len : 0); + length = pxmitpriv->frag_len - pattrib->hdrlen - + pattrib->iv_len - + ((pattrib->bswenc) ? pattrib->icv_len : 0); rtw_secmicappend(&micdata, payload, length); - payload = payload+length+pattrib->icv_len; + payload = payload + length + pattrib->icv_len; } } rtw_secgetmic(&micdata, &mic[0]); @@ -1108,8 +1116,10 @@ s32 rtw_xmitframe_coalesce(struct adapter *padapter, struct sk_buff *pkt, struct if (bmcst || (rtw_endofpktfile(&pktfile) == true)) { pattrib->nr_frags = frg_inx; - pattrib->last_txcmdsz = pattrib->hdrlen + pattrib->iv_len + ((pattrib->nr_frags == 1) ? llc_sz:0) + - ((pattrib->bswenc) ? pattrib->icv_len : 0) + mem_sz; + pattrib->last_txcmdsz = pattrib->hdrlen + pattrib->iv_len + + ((pattrib->nr_frags == 1) ? llc_sz : 0) + + ((pattrib->bswenc) ? pattrib->icv_len : 0) + + mem_sz; ClearMFrag(mem_start); @@ -1158,8 +1168,8 @@ s32 rtw_mgmt_xmitframe_coalesce(struct adapter *padapter, struct sk_buff *pkt, s mem_start = pframe = (u8 *)(pxmitframe->buf_addr) + TXDESC_OFFSET; pwlanhdr = (struct ieee80211_hdr *)pframe; - ori_len = BIP_AAD_SIZE+pattrib->pktlen; - tmp_buf = BIP_AAD = rtw_zmalloc(ori_len); + ori_len = BIP_AAD_SIZE + pattrib->pktlen; + tmp_buf = BIP_AAD = kzalloc(ori_len, GFP_ATOMIC); subtype = GetFrameSubType(pframe); /* bit(7)~bit(2) */ if (!BIP_AAD) @@ -1211,14 +1221,14 @@ s32 rtw_mgmt_xmitframe_coalesce(struct adapter *padapter, struct sk_buff *pkt, s /* conscruct AAD, copy address 1 to address 3 */ memcpy(BIP_AAD + 2, &pwlanhdr->addrs, sizeof(pwlanhdr->addrs)); /* copy management fram body */ - memcpy(BIP_AAD+BIP_AAD_SIZE, MGMT_body, frame_body_len); + memcpy(BIP_AAD + BIP_AAD_SIZE, MGMT_body, frame_body_len); /* calculate mic */ if (omac1_aes_128(padapter->securitypriv.dot11wBIPKey[padapter->securitypriv.dot11wBIPKeyid].skey - , BIP_AAD, BIP_AAD_SIZE+frame_body_len, mic)) + , BIP_AAD, BIP_AAD_SIZE + frame_body_len, mic)) goto xmitframe_coalesce_fail; /* copy right BIP mic value, total is 128bits, we use the 0~63 bits */ - memcpy(pframe-8, mic, 8); + memcpy(pframe - 8, mic, 8); } else { /* unicast mgmt frame TX */ /* start to encrypt mgmt frame */ if (subtype == WIFI_DEAUTH || subtype == WIFI_DISASSOC || @@ -1267,9 +1277,10 @@ s32 rtw_mgmt_xmitframe_coalesce(struct adapter *padapter, struct sk_buff *pkt, s memcpy(pframe, pattrib->iv, pattrib->iv_len); pframe += pattrib->iv_len; /* copy mgmt data portion after CCMP header */ - memcpy(pframe, tmp_buf+pattrib->hdrlen, pattrib->pktlen-pattrib->hdrlen); + memcpy(pframe, tmp_buf + pattrib->hdrlen, + pattrib->pktlen - pattrib->hdrlen); /* move pframe to end of mgmt pkt */ - pframe += pattrib->pktlen-pattrib->hdrlen; + pframe += pattrib->pktlen - pattrib->hdrlen; /* add 8 bytes CCMP IV header to length */ pattrib->pktlen += pattrib->iv_len; if ((pattrib->icv_len > 0) && (pattrib->bswenc)) { @@ -1375,7 +1386,7 @@ void rtw_count_tx_stats(struct adapter *padapter, struct xmit_frame *pxmitframe, struct mlme_priv *pmlmepriv = &padapter->mlmepriv; u8 pkt_num = 1; - if ((pxmitframe->frame_tag&0x0f) == DATA_FRAMETAG) { + if ((pxmitframe->frame_tag & 0x0f) == DATA_FRAMETAG) { pkt_num = pxmitframe->agg_num; pmlmepriv->LinkDetectInfo.NumTxOkInPeriod += pkt_num; @@ -1662,8 +1673,7 @@ struct xmit_frame *rtw_alloc_xmitframe_once(struct xmit_priv *pxmitpriv) struct xmit_frame *pxframe = NULL; u8 *alloc_addr; - alloc_addr = rtw_zmalloc(sizeof(struct xmit_frame) + 4); - + alloc_addr = kzalloc(sizeof(*pxframe) + 4, GFP_ATOMIC); if (!alloc_addr) goto exit; @@ -1836,8 +1846,7 @@ s32 rtw_alloc_hwxmits(struct adapter *padapter) pxmitpriv->hwxmits = NULL; - pxmitpriv->hwxmits = rtw_zmalloc(sizeof(struct hw_xmit) * pxmitpriv->hwxmit_entry); - + pxmitpriv->hwxmits = kcalloc(pxmitpriv->hwxmit_entry, sizeof(*hwxmits), GFP_ATOMIC); if (!pxmitpriv->hwxmits) return _FAIL; @@ -2070,7 +2079,7 @@ signed int xmitframe_enqueue_for_sleeping_sta(struct adapter *padapter, struct x spin_lock_bh(&psta->sleep_q.lock); - if (psta->state&WIFI_SLEEP_STATE) { + if (psta->state & WIFI_SLEEP_STATE) { u8 wmmps_ac = 0; if (pstapriv->sta_dz_bitmap & BIT(psta->aid)) { @@ -2083,20 +2092,20 @@ signed int xmitframe_enqueue_for_sleeping_sta(struct adapter *padapter, struct x switch (pattrib->priority) { case 1: case 2: - wmmps_ac = psta->uapsd_bk&BIT(0); + wmmps_ac = psta->uapsd_bk & BIT(0); break; case 4: case 5: - wmmps_ac = psta->uapsd_vi&BIT(0); + wmmps_ac = psta->uapsd_vi & BIT(0); break; case 6: case 7: - wmmps_ac = psta->uapsd_vo&BIT(0); + wmmps_ac = psta->uapsd_vo & BIT(0); break; case 0: case 3: default: - wmmps_ac = psta->uapsd_be&BIT(0); + wmmps_ac = psta->uapsd_be & BIT(0); break; } @@ -2214,20 +2223,20 @@ void wakeup_sta_to_xmit(struct adapter *padapter, struct sta_info *psta) switch (pxmitframe->attrib.priority) { case 1: case 2: - wmmps_ac = psta->uapsd_bk&BIT(1); + wmmps_ac = psta->uapsd_bk & BIT(1); break; case 4: case 5: - wmmps_ac = psta->uapsd_vi&BIT(1); + wmmps_ac = psta->uapsd_vi & BIT(1); break; case 6: case 7: - wmmps_ac = psta->uapsd_vo&BIT(1); + wmmps_ac = psta->uapsd_vo & BIT(1); break; case 0: case 3: default: - wmmps_ac = psta->uapsd_be&BIT(1); + wmmps_ac = psta->uapsd_be & BIT(1); break; } @@ -2259,7 +2268,7 @@ void wakeup_sta_to_xmit(struct adapter *padapter, struct sta_info *psta) pstapriv->tim_bitmap &= ~BIT(psta->aid); - if (psta->state&WIFI_SLEEP_STATE) + if (psta->state & WIFI_SLEEP_STATE) psta->state ^= WIFI_SLEEP_STATE; if (psta->state & WIFI_STA_ALIVE_CHK_STATE) { @@ -2274,7 +2283,7 @@ void wakeup_sta_to_xmit(struct adapter *padapter, struct sta_info *psta) if (!psta_bmc) goto _exit; - if ((pstapriv->sta_dz_bitmap&0xfffe) == 0x0) { /* no any sta in ps mode */ + if ((pstapriv->sta_dz_bitmap & 0xfffe) == 0x0) { /* no any sta in ps mode */ xmitframe_phead = get_list_head(&psta_bmc->sleep_q); list_for_each_safe(xmitframe_plist, tmp, xmitframe_phead) { pxmitframe = list_entry(xmitframe_plist, @@ -2327,20 +2336,20 @@ void xmit_delivery_enabled_frames(struct adapter *padapter, struct sta_info *pst switch (pxmitframe->attrib.priority) { case 1: case 2: - wmmps_ac = psta->uapsd_bk&BIT(1); + wmmps_ac = psta->uapsd_bk & BIT(1); break; case 4: case 5: - wmmps_ac = psta->uapsd_vi&BIT(1); + wmmps_ac = psta->uapsd_vi & BIT(1); break; case 6: case 7: - wmmps_ac = psta->uapsd_vo&BIT(1); + wmmps_ac = psta->uapsd_vo & BIT(1); break; case 0: case 3: default: - wmmps_ac = psta->uapsd_be&BIT(1); + wmmps_ac = psta->uapsd_be & BIT(1); break; } diff --git a/drivers/staging/rtl8723bs/hal/rtl8723bs_recv.c b/drivers/staging/rtl8723bs/hal/rtl8723bs_recv.c index 399edfbf8ec6..ca3fa3e399c9 100644 --- a/drivers/staging/rtl8723bs/hal/rtl8723bs_recv.c +++ b/drivers/staging/rtl8723bs/hal/rtl8723bs_recv.c @@ -382,7 +382,7 @@ s32 rtl8723bs_init_recv_priv(struct adapter *padapter) spin_lock_init(&precvpriv->recv_buf_pending_queue.lock); n = NR_RECVBUFF * sizeof(struct recv_buf) + 4; - precvpriv->pallocated_recv_buf = rtw_zmalloc(n); + precvpriv->pallocated_recv_buf = kzalloc(n, GFP_KERNEL); if (!precvpriv->pallocated_recv_buf) { res = _FAIL; goto exit; diff --git a/drivers/staging/rtl8723bs/hal/sdio_ops.c b/drivers/staging/rtl8723bs/hal/sdio_ops.c index 8736c124f857..54228c41579e 100644 --- a/drivers/staging/rtl8723bs/hal/sdio_ops.c +++ b/drivers/staging/rtl8723bs/hal/sdio_ops.c @@ -895,7 +895,7 @@ void sd_int_dpc(struct adapter *adapter) if (hal->sdio_hisr & SDIO_HISR_C2HCMD) { struct c2h_evt_hdr_88xx *c2h_evt; - c2h_evt = rtw_zmalloc(16); + c2h_evt = kzalloc(16, GFP_ATOMIC); if (c2h_evt) { if (c2h_evt_read_88xx(adapter, (u8 *)c2h_evt) == _SUCCESS) { if (c2h_id_filter_ccx_8723b((u8 *)c2h_evt)) { diff --git a/drivers/staging/rtl8723bs/include/basic_types.h b/drivers/staging/rtl8723bs/include/basic_types.h index 1c2da18e6210..16b270fe0203 100644 --- a/drivers/staging/rtl8723bs/include/basic_types.h +++ b/drivers/staging/rtl8723bs/include/basic_types.h @@ -12,6 +12,7 @@ #define FAIL (-1) #include +#include #define FIELD_OFFSET(s, field) ((__kernel_ssize_t)&((s *)(0))->field) diff --git a/drivers/staging/rtl8723bs/os_dep/ioctl_cfg80211.c b/drivers/staging/rtl8723bs/os_dep/ioctl_cfg80211.c index 0eb40b7bd0ba..06bf5d176e7d 100644 --- a/drivers/staging/rtl8723bs/os_dep/ioctl_cfg80211.c +++ b/drivers/staging/rtl8723bs/os_dep/ioctl_cfg80211.c @@ -111,6 +111,7 @@ static struct ieee80211_supported_band *rtw_spt_band_alloc( { struct ieee80211_supported_band *spt_band = NULL; int n_channels, n_bitrates; + size_t alloc_sz; if (band == NL80211_BAND_2GHZ) { n_channels = RTW_2G_CHANNELS_NUM; @@ -119,9 +120,10 @@ static struct ieee80211_supported_band *rtw_spt_band_alloc( goto exit; } - spt_band = rtw_zmalloc(sizeof(struct ieee80211_supported_band) + - sizeof(struct ieee80211_channel) * n_channels + - sizeof(struct ieee80211_rate) * n_bitrates); + alloc_sz = sizeof(*spt_band); + alloc_sz = size_add(alloc_sz, array_size(n_channels, sizeof(struct ieee80211_channel))); + alloc_sz = size_add(alloc_sz, array_size(n_bitrates, sizeof(struct ieee80211_rate))); + spt_band = kzalloc(alloc_sz, GFP_KERNEL); if (!spt_band) goto exit; @@ -841,11 +843,9 @@ static int cfg80211_rtw_add_key(struct wiphy *wiphy, struct net_device *ndev, struct mlme_priv *pmlmepriv = &padapter->mlmepriv; param_len = sizeof(struct ieee_param) + params->key_len; - param = rtw_malloc(param_len); + param = kzalloc(param_len, GFP_KERNEL); if (!param) - return -1; - - memset(param, 0, param_len); + return -ENOMEM; param->cmd = IEEE_CMD_SET_ENCRYPTION; eth_broadcast_addr(param->sta_addr); @@ -1431,7 +1431,7 @@ static int rtw_cfg80211_set_wpa_ie(struct adapter *padapter, u8 *pie, size_t iel goto exit; } - buf = rtw_zmalloc(ielen); + buf = kzalloc(ielen, GFP_KERNEL); if (!buf) { ret = -ENOMEM; goto exit; @@ -1721,15 +1721,14 @@ static int cfg80211_rtw_connect(struct wiphy *wiphy, struct net_device *ndev, if (wep_key_len > 0) { wep_key_len = wep_key_len <= 5 ? 5 : 13; - wep_total_len = wep_key_len + FIELD_OFFSET(struct ndis_802_11_wep, key_material); - pwep = rtw_malloc(wep_total_len); + wep_total_len = wep_key_len + + offsetof(struct ndis_802_11_wep, key_material); + pwep = kzalloc(wep_total_len, GFP_KERNEL); if (!pwep) { ret = -ENOMEM; goto exit; } - memset(pwep, 0, wep_total_len); - pwep->key_length = wep_key_len; pwep->length = wep_total_len; @@ -2159,7 +2158,7 @@ static int rtw_cfg80211_add_monitor_if(struct adapter *padapter, char *name, str pnpi->sizeof_priv = sizeof(struct adapter); /* wdev */ - mon_wdev = rtw_zmalloc(sizeof(struct wireless_dev)); + mon_wdev = kzalloc(sizeof(*mon_wdev), GFP_KERNEL); if (!mon_wdev) { ret = -ENOMEM; goto out; @@ -2269,7 +2268,7 @@ static int rtw_add_beacon(struct adapter *adapter, const u8 *head, size_t head_l if (head_len < 24) return -EINVAL; - pbuf = rtw_zmalloc(head_len + tail_len); + pbuf = kzalloc(head_len + tail_len, GFP_KERNEL); if (!pbuf) return -ENOMEM; @@ -2740,7 +2739,7 @@ int rtw_wdev_alloc(struct adapter *padapter, struct device *dev) goto free_wiphy; /* wdev */ - wdev = rtw_zmalloc(sizeof(struct wireless_dev)); + wdev = kzalloc(sizeof(*wdev), GFP_KERNEL); if (!wdev) { ret = -ENOMEM; goto unregister_wiphy; diff --git a/drivers/staging/rtl8723bs/os_dep/os_intfs.c b/drivers/staging/rtl8723bs/os_dep/os_intfs.c index 6ca6dc548805..9f7f6b40093c 100644 --- a/drivers/staging/rtl8723bs/os_dep/os_intfs.c +++ b/drivers/staging/rtl8723bs/os_dep/os_intfs.c @@ -560,7 +560,7 @@ struct dvobj_priv *devobj_init(void) { struct dvobj_priv *pdvobj = NULL; - pdvobj = rtw_zmalloc(sizeof(*pdvobj)); + pdvobj = kzalloc(sizeof(*pdvobj), GFP_KERNEL); if (!pdvobj) return NULL; diff --git a/drivers/staging/rtl8723bs/os_dep/osdep_service.c b/drivers/staging/rtl8723bs/os_dep/osdep_service.c index a00f9f0c85c5..a9502ed7e000 100644 --- a/drivers/staging/rtl8723bs/os_dep/osdep_service.c +++ b/drivers/staging/rtl8723bs/os_dep/osdep_service.c @@ -223,12 +223,9 @@ struct rtw_cbuf *rtw_cbuf_alloc(u32 size) { struct rtw_cbuf *cbuf; - cbuf = rtw_malloc(struct_size(cbuf, bufs, size)); - - if (cbuf) { - cbuf->write = cbuf->read = 0; + cbuf = kzalloc(struct_size(cbuf, bufs, size), GFP_KERNEL); + if (cbuf) cbuf->size = size; - } return cbuf; } diff --git a/drivers/staging/rtl8723bs/os_dep/sdio_intf.c b/drivers/staging/rtl8723bs/os_dep/sdio_intf.c index 139ace51486d..ae379e99736a 100644 --- a/drivers/staging/rtl8723bs/os_dep/sdio_intf.c +++ b/drivers/staging/rtl8723bs/os_dep/sdio_intf.c @@ -307,8 +307,8 @@ static struct adapter *rtw_sdio_if1_init(struct dvobj_priv *dvobj, const struct status = _SUCCESS; free_hal_data: - if (status != _SUCCESS && padapter->HalData) - kfree(padapter->HalData); + if (status != _SUCCESS) + rtw_hal_data_deinit(padapter); if (status != _SUCCESS) { rtw_wdev_unregister(padapter->rtw_wdev); diff --git a/drivers/staging/rtl8723bs/os_dep/xmit_linux.c b/drivers/staging/rtl8723bs/os_dep/xmit_linux.c index 944b9c724b32..20d2926a4480 100644 --- a/drivers/staging/rtl8723bs/os_dep/xmit_linux.c +++ b/drivers/staging/rtl8723bs/os_dep/xmit_linux.c @@ -46,7 +46,7 @@ signed int rtw_endofpktfile(struct pkt_file *pfile) int rtw_os_xmit_resource_alloc(struct adapter *padapter, struct xmit_buf *pxmitbuf, u32 alloc_sz, u8 flag) { if (alloc_sz > 0) { - pxmitbuf->pallocated_buf = rtw_zmalloc(alloc_sz); + pxmitbuf->pallocated_buf = kzalloc(alloc_sz, GFP_KERNEL); if (!pxmitbuf->pallocated_buf) return _FAIL; diff --git a/drivers/staging/sm750fb/Kconfig b/drivers/staging/sm750fb/Kconfig index 08bcccdd0f1c..25fe422f55f2 100644 --- a/drivers/staging/sm750fb/Kconfig +++ b/drivers/staging/sm750fb/Kconfig @@ -6,6 +6,7 @@ config FB_SM750 select FB_CFB_FILLRECT select FB_CFB_COPYAREA select FB_CFB_IMAGEBLIT + select FB_IOMEM_FOPS help Frame buffer driver for the Silicon Motion SM750 chip with 2D acceleration and dual head support. diff --git a/drivers/staging/sm750fb/sm750.c b/drivers/staging/sm750fb/sm750.c index 24e927a1669d..676ef4457724 100644 --- a/drivers/staging/sm750fb/sm750.c +++ b/drivers/staging/sm750fb/sm750.c @@ -273,7 +273,7 @@ static void lynxfb_ops_imageblit(struct fb_info *info, spin_lock(&sm750_dev->slock); sm750_dev->accel.de_imageblit(&sm750_dev->accel, - image->data, image->width >> 3, 0, + image->data, 0, base, pitch, bpp, image->dx, image->dy, image->width, image->height, @@ -603,7 +603,7 @@ static int sm750fb_set_drv(struct lynxfb_par *par) crtc = &par->crtc; crtc->vidmem_size = sm750_dev->vidmem_size; - if (sm750_dev->fb_count > 1) + if (g_dualview) crtc->vidmem_size >>= 1; /* setup crtc and output member */ @@ -968,7 +968,7 @@ static void sm750fb_setup(struct sm750_dev *sm750_dev, char *src) NO_PARAM: if (sm750_dev->revid != SM750LE_REVISION_ID) { - if (sm750_dev->fb_count > 1) { + if (g_dualview) { if (swap) sm750_dev->dataflow = sm750_dual_swap; else diff --git a/drivers/staging/sm750fb/sm750.h b/drivers/staging/sm750fb/sm750.h index fcb7d586ebf0..2845b8ec48ac 100644 --- a/drivers/staging/sm750fb/sm750.h +++ b/drivers/staging/sm750fb/sm750.h @@ -73,7 +73,7 @@ struct lynx_accel { u32 rop2); int (*de_imageblit)(struct lynx_accel *accel, const char *p_srcbuf, - u32 src_delta, u32 start_bit, u32 d_base, u32 d_pitch, + u32 start_bit, u32 d_base, u32 d_pitch, u32 byte_per_pixel, u32 dx, u32 dy, u32 width, u32 height, u32 f_color, u32 b_color, u32 rop2); diff --git a/drivers/staging/sm750fb/sm750_accel.c b/drivers/staging/sm750fb/sm750_accel.c index b07c1aa68621..47230896a5b0 100644 --- a/drivers/staging/sm750fb/sm750_accel.c +++ b/drivers/staging/sm750fb/sm750_accel.c @@ -300,8 +300,6 @@ static unsigned int deGetTransparency(struct lynx_accel *accel) * sm750_hw_imageblit * @accel: Acceleration device data * @pSrcbuf: pointer to start of source buffer in system memory - * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down - * and -ive mean button up * @startBit: Mono data can start at any bit in a byte, this value should be * 0 to 7 * @dBase: Address of destination: offset in frame buffer @@ -316,7 +314,7 @@ static unsigned int deGetTransparency(struct lynx_accel *accel) * @rop2: ROP value */ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf, - u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch, + u32 startBit, u32 dBase, u32 dPitch, u32 bytePerPixel, u32 dx, u32 dy, u32 width, u32 height, u32 fColor, u32 bColor, u32 rop2) { @@ -405,7 +403,7 @@ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf, write_dpPort(accel, *(unsigned int *)ajRemain); } - pSrcbuf += srcDelta; + pSrcbuf += ulBytesPerScan; } return 0; diff --git a/drivers/staging/sm750fb/sm750_accel.h b/drivers/staging/sm750fb/sm750_accel.h index 2c79cb730a0a..df48ed57a054 100644 --- a/drivers/staging/sm750fb/sm750_accel.h +++ b/drivers/staging/sm750fb/sm750_accel.h @@ -220,8 +220,6 @@ int sm750_hw_copyarea(struct lynx_accel *accel, /** * sm750_hw_imageblit * @pSrcbuf: pointer to start of source buffer in system memory - * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down - *>----- and -ive mean button up * @startBit: Mono data can start at any bit in a byte, this value should be *>----- 0 to 7 * @dBase: Address of destination: offset in frame buffer @@ -236,7 +234,7 @@ int sm750_hw_copyarea(struct lynx_accel *accel, * @rop2: ROP value */ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf, - u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch, + u32 startBit, u32 dBase, u32 dPitch, u32 bytePerPixel, u32 dx, u32 dy, u32 width, u32 height, u32 fColor, u32 bColor, u32 rop2); diff --git a/drivers/thermal/airoha_thermal.c b/drivers/thermal/airoha_thermal.c index b9fd6bfc88e5..829a7327fc40 100644 --- a/drivers/thermal/airoha_thermal.c +++ b/drivers/thermal/airoha_thermal.c @@ -273,7 +273,7 @@ static int airoha_thermal_set_trips(struct thermal_zone_device *tz, int low, if (low != -INT_MAX) { /* Validate low and clamp it to a supported value */ - low = clamp_t(int, high, RAW_TO_TEMP(priv, 0), + low = clamp_t(int, low, RAW_TO_TEMP(priv, 0), RAW_TO_TEMP(priv, FIELD_MAX(EN7581_DOUT_TADC_MASK))); /* We offset the low temp of 1°C to trigger correct event */ @@ -403,7 +403,7 @@ static void airoha_thermal_setup_monitor(struct airoha_thermal_priv *priv) * sen interval is 379 * 52.715us = 19.97ms */ writel(FIELD_PREP(EN7581_FILT_INTERVAL, 1) | - FIELD_PREP(EN7581_FILT_INTERVAL, 379), + FIELD_PREP(EN7581_SEN_INTERVAL, 379), priv->base + EN7581_TEMPMONCTL2); /* AHB poll is set to 146 * 68.64 = 10.02us */ diff --git a/drivers/thermal/intel/int340x_thermal/int3400_thermal.c b/drivers/thermal/intel/int340x_thermal/int3400_thermal.c index 908cc1bf57f1..2aa03ceb570e 100644 --- a/drivers/thermal/intel/int340x_thermal/int3400_thermal.c +++ b/drivers/thermal/intel/int340x_thermal/int3400_thermal.c @@ -354,8 +354,10 @@ static void cleanup_odvp(struct int3400_thermal_priv *priv) kfree(priv->odvp_attrs[i].attr.attr.name); } kfree(priv->odvp_attrs); + priv->odvp_attrs = NULL; } kfree(priv->odvp); + priv->odvp = NULL; priv->odvp_count = 0; } @@ -635,7 +637,6 @@ static int int3400_thermal_probe(struct platform_device *pdev) acpi_remove_notify_handler(priv->adev->handle, ACPI_DEVICE_NOTIFY, int3400_notify); free_sysfs: - cleanup_odvp(priv); if (!ZERO_OR_NULL_PTR(priv->data_vault)) { device_remove_bin_file(&pdev->dev, &bin_attr_data_vault); kfree(priv->data_vault); @@ -649,6 +650,7 @@ static int int3400_thermal_probe(struct platform_device *pdev) acpi_thermal_rel_misc_device_remove(priv->adev->handle); thermal_zone_device_unregister(priv->thermal); free_art_trt: + cleanup_odvp(priv); kfree(priv->trts); kfree(priv->arts); free_priv: diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c index bd2fca7dc017..8a2f441cd2ec 100644 --- a/drivers/thermal/intel/intel_hfi.c +++ b/drivers/thermal/intel/intel_hfi.c @@ -592,7 +592,7 @@ static void hfi_disable_instance(void *ptr) hfi_disable(); } -static void hfi_syscore_resume(void) +static void hfi_syscore_resume(void *data) { /* This code runs only on the boot CPU. */ struct hfi_cpu_info *info = &per_cpu(hfi_cpu_info, 0); @@ -603,7 +603,7 @@ static void hfi_syscore_resume(void) hfi_enable_instance(hfi_instance); } -static int hfi_syscore_suspend(void) +static int hfi_syscore_suspend(void *data) { /* No locking needed. There is no concurrency with CPU offline. */ hfi_disable(); @@ -611,11 +611,15 @@ static int hfi_syscore_suspend(void) return 0; } -static struct syscore_ops hfi_pm_ops = { +static const struct syscore_ops hfi_pm_ops = { .resume = hfi_syscore_resume, .suspend = hfi_syscore_suspend, }; +static struct syscore hfi_pm = { + .ops = &hfi_pm_ops, +}; + static int hfi_thermal_notify(struct notifier_block *nb, unsigned long state, void *_notify) { @@ -710,7 +714,7 @@ void __init intel_hfi_init(void) if (thermal_genl_register_notifier(&hfi_thermal_nb)) goto err_nl_notif; - register_syscore_ops(&hfi_pm_ops); + register_syscore(&hfi_pm); return; diff --git a/drivers/thermal/qcom/qcom-spmi-adc-tm5.c b/drivers/thermal/qcom/qcom-spmi-adc-tm5.c index d7f2e6ca92c2..d1b086737bcd 100644 --- a/drivers/thermal/qcom/qcom-spmi-adc-tm5.c +++ b/drivers/thermal/qcom/qcom-spmi-adc-tm5.c @@ -369,9 +369,6 @@ static int adc_tm5_get_temp(struct thermal_zone_device *tz, int *temp) if (ret < 0) return ret; - if (ret != IIO_VAL_INT) - return -EINVAL; - return 0; } diff --git a/drivers/thermal/renesas/rcar_thermal.c b/drivers/thermal/renesas/rcar_thermal.c index fdd7afdc4ff6..27adc64757f5 100644 --- a/drivers/thermal/renesas/rcar_thermal.c +++ b/drivers/thermal/renesas/rcar_thermal.c @@ -492,12 +492,6 @@ static int rcar_thermal_probe(struct platform_device *pdev) "rcar_thermal", trips, ARRAY_SIZE(trips), priv, &rcar_thermal_zone_ops, NULL, 0, idle); - - ret = thermal_zone_device_enable(priv->zone); - if (ret) { - thermal_zone_device_unregister(priv->zone); - priv->zone = ERR_PTR(ret); - } } if (IS_ERR(priv->zone)) { dev_err(dev, "can't register thermal zone\n"); @@ -506,11 +500,12 @@ static int rcar_thermal_probe(struct platform_device *pdev) goto error_unregister; } - if (chip->use_of_thermal) { + if (chip->use_of_thermal) ret = thermal_add_hwmon_sysfs(priv->zone); - if (ret) - goto error_unregister; - } + else + ret = thermal_zone_device_enable(priv->zone); + if (ret) + goto error_unregister; rcar_thermal_irq_enable(priv); diff --git a/drivers/tty/hvc/Kconfig b/drivers/tty/hvc/Kconfig index c2a4e88b328f..5866195de26a 100644 --- a/drivers/tty/hvc/Kconfig +++ b/drivers/tty/hvc/Kconfig @@ -79,7 +79,7 @@ config HVC_UDBG config HVC_DCC bool "ARM JTAG DCC console" - depends on ARM || ARM64 + depends on (ARM && (CPU_V6 || CPU_V6K || CPU_V7)) || ARM64 select HVC_DRIVER select SERIAL_CORE_CONSOLE help diff --git a/drivers/tty/serial/8250/8250_core.c b/drivers/tty/serial/8250/8250_core.c index bfa421ab3253..ccd5a18f5335 100644 --- a/drivers/tty/serial/8250/8250_core.c +++ b/drivers/tty/serial/8250/8250_core.c @@ -689,6 +689,7 @@ static void serial_8250_overrun_backoff_work(struct work_struct *work) int serial8250_register_8250_port(const struct uart_8250_port *up) { struct uart_8250_port *uart; + bool cons_flow; int ret; if (up->port.uartclk == 0) @@ -712,6 +713,9 @@ int serial8250_register_8250_port(const struct uart_8250_port *up) if (uart->port.type == PORT_8250_CIR) return -ENODEV; + /* Preserve specified console flow control. */ + cons_flow = uart_cons_flow_enabled(&uart->port); + if (uart->port.dev) uart_remove_one_port(&serial8250_reg, &uart->port); @@ -742,6 +746,8 @@ int serial8250_register_8250_port(const struct uart_8250_port *up) uart->lsr_save_mask = up->lsr_save_mask; uart->dma = up->dma; + uart_set_cons_flow_enabled(&uart->port, uart_cons_flow_enabled(&up->port) | cons_flow); + /* Take tx_loadsz from fifosize if it wasn't set separately */ if (uart->port.fifosize && !uart->tx_loadsz) uart->tx_loadsz = uart->port.fifosize; diff --git a/drivers/tty/serial/8250/8250_port.c b/drivers/tty/serial/8250/8250_port.c index 0c633639f765..d3ffba16c0f9 100644 --- a/drivers/tty/serial/8250/8250_port.c +++ b/drivers/tty/serial/8250/8250_port.c @@ -1990,8 +1990,14 @@ static void wait_for_xmitr(struct uart_8250_port *up, int bits) wait_for_lsr(up, bits); - /* Wait up to 1s for flow control if necessary */ - if (up->port.flags & UPF_CONS_FLOW) { + /* + * Wait up to 1s for flow control if necessary. + * When 'no_console_suspend' is active (in the window between + * suspend() and resume()), flow control is temporarily ignored + * because the canary workaround is not reliable in all situations, + * leading to flow control timeouts for every character. + */ + if (uart_console_hwflow_active(&up->port) && !up->canary) { for (tmout = 1000000; tmout; tmout--) { unsigned int msr = serial_in(up, UART_MSR); up->msr_saved_flags |= msr & MSR_SAVE_FLAGS; @@ -2785,6 +2791,12 @@ serial8250_do_set_termios(struct uart_port *port, struct ktermios *termios, serial8250_set_efr(port, termios); serial8250_set_divisor(port, baud, quot, frac); serial8250_set_fcr(port, termios); + /* Consoles manually poll CTS for hardware flow control. */ + if (uart_console(port) && + !(port->rs485.flags & SER_RS485_ENABLED) + && termios->c_cflag & CRTSCTS) { + port->mctrl |= TIOCM_RTS; + } serial8250_set_mctrl(port, port->mctrl); } @@ -3354,7 +3366,7 @@ void serial8250_console_write(struct uart_8250_port *up, const char *s, * it regardless of the CTS state. Therefore, only use fifo * if we don't use control flow. */ - !(up->port.flags & UPF_CONS_FLOW); + !uart_console_hwflow_active(&up->port); if (likely(use_fifo)) serial8250_console_fifo_write(up, s, count); @@ -3424,6 +3436,9 @@ int serial8250_console_setup(struct uart_port *port, char *options, bool probe) if (ret) return ret; + /* Track user-specified console flow control. */ + uart_set_cons_flow_enabled(port, flow == 'r'); + if (port->dev) pm_runtime_get_sync(port->dev); diff --git a/drivers/tty/serial/amba-pl011.c b/drivers/tty/serial/amba-pl011.c index 9bfccb021383..4b213d0c6133 100644 --- a/drivers/tty/serial/amba-pl011.c +++ b/drivers/tty/serial/amba-pl011.c @@ -2446,7 +2446,7 @@ static int pl011_console_setup(struct console *co, char *options) /* Allow pins to be muxed in and configured */ pinctrl_pm_select_default_state(uap->port.dev); - ret = clk_prepare(uap->clk); + ret = clk_prepare_enable(uap->clk); if (ret) return ret; @@ -2475,6 +2475,15 @@ static int pl011_console_setup(struct console *co, char *options) return uart_set_options(&uap->port, co, baud, parity, bits, flow); } +static int pl011_console_exit(struct console *co) +{ + struct uart_amba_port *uap = amba_ports[co->index]; + + clk_disable_unprepare(uap->clk); + + return 0; +} + /** * pl011_console_match - non-standard console matching * @co: registering console @@ -2544,8 +2553,6 @@ pl011_console_write_atomic(struct console *co, struct nbcon_write_context *wctxt if (!nbcon_enter_unsafe(wctxt)) return; - clk_enable(uap->clk); - if (!uap->vendor->always_enabled) { old_cr = pl011_read(uap, REG_CR); pl011_write((old_cr & ~UART011_CR_CTSEN) | (UART01x_CR_UARTEN | UART011_CR_TXE), @@ -2562,8 +2569,6 @@ pl011_console_write_atomic(struct console *co, struct nbcon_write_context *wctxt if (!uap->vendor->always_enabled) pl011_write(old_cr, uap, REG_CR); - clk_disable(uap->clk); - nbcon_exit_unsafe(wctxt); } @@ -2576,8 +2581,6 @@ pl011_console_write_thread(struct console *co, struct nbcon_write_context *wctxt if (!nbcon_enter_unsafe(wctxt)) return; - clk_enable(uap->clk); - if (!uap->vendor->always_enabled) { old_cr = pl011_read(uap, REG_CR); pl011_write((old_cr & ~UART011_CR_CTSEN) | (UART01x_CR_UARTEN | UART011_CR_TXE), @@ -2606,8 +2609,6 @@ pl011_console_write_thread(struct console *co, struct nbcon_write_context *wctxt if (!uap->vendor->always_enabled) pl011_write(old_cr, uap, REG_CR); - clk_disable(uap->clk); - nbcon_exit_unsafe(wctxt); } @@ -2628,6 +2629,7 @@ static struct console amba_console = { .name = "ttyAMA", .device = uart_console_device, .setup = pl011_console_setup, + .exit = pl011_console_exit, .match = pl011_console_match, .write_atomic = pl011_console_write_atomic, .write_thread = pl011_console_write_thread, @@ -2966,21 +2968,45 @@ static void pl011_remove(struct amba_device *dev) static int pl011_suspend(struct device *dev) { struct uart_amba_port *uap = dev_get_drvdata(dev); + int ret; if (!uap) return -EINVAL; - return uart_suspend_port(&amba_reg, &uap->port); + ret = uart_suspend_port(&amba_reg, &uap->port); + if (ret) + return ret; + + if (console_suspend_enabled && uap->port.suspended && + uart_console_registered(&uap->port)) + clk_disable_unprepare(uap->clk); + + return 0; } static int pl011_resume(struct device *dev) { struct uart_amba_port *uap = dev_get_drvdata(dev); + bool resume_console; + int ret; if (!uap) return -EINVAL; - return uart_resume_port(&amba_reg, &uap->port); + resume_console = console_suspend_enabled && + uap->port.suspended && + uart_console_registered(&uap->port); + if (resume_console) { + ret = clk_prepare_enable(uap->clk); + if (ret) + return ret; + } + + ret = uart_resume_port(&amba_reg, &uap->port); + if (ret && resume_console) + clk_disable_unprepare(uap->clk); + + return ret; } #endif diff --git a/drivers/tty/serial/bcm63xx_uart.c b/drivers/tty/serial/bcm63xx_uart.c index 51df9d2d8bfc..544695cb184c 100644 --- a/drivers/tty/serial/bcm63xx_uart.c +++ b/drivers/tty/serial/bcm63xx_uart.c @@ -675,7 +675,7 @@ static void wait_for_xmitr(struct uart_port *port) } /* Wait up to 1s for flow control if necessary */ - if (port->flags & UPF_CONS_FLOW) { + if (uart_cons_flow_enabled(port)) { tmout = 1000000; while (--tmout) { unsigned int val; diff --git a/drivers/tty/serial/ma35d1_serial.c b/drivers/tty/serial/ma35d1_serial.c index 285b0fe41a86..920fe7ff5083 100644 --- a/drivers/tty/serial/ma35d1_serial.c +++ b/drivers/tty/serial/ma35d1_serial.c @@ -608,8 +608,14 @@ static int __init ma35d1serial_console_setup(struct console *co, char *options) if (!np || !p) return -ENODEV; - if (of_property_read_u32_array(np, "reg", val32, ARRAY_SIZE(val32)) != 0) + if (of_property_read_u32_array(np, "reg", val32, ARRAY_SIZE(val32)) != 0) { + of_node_put(np); + ma35d1serial_uart_nodes[co->index] = NULL; return -EINVAL; + } + + of_node_put(np); + ma35d1serial_uart_nodes[co->index] = NULL; p->port.iobase = val32[1]; p->port.membase = ioremap(p->port.iobase, MA35_UART_REG_SIZE); @@ -648,8 +654,10 @@ static void ma35d1serial_console_init_port(void) of_node_get(np); ma35d1serial_uart_nodes[i] = np; i++; - if (i == MA35_UART_NR) + if (i == MA35_UART_NR) { + of_node_put(np); break; + } } } } diff --git a/drivers/tty/serial/omap-serial.c b/drivers/tty/serial/omap-serial.c index 0b85f47ff19e..a689d190940c 100644 --- a/drivers/tty/serial/omap-serial.c +++ b/drivers/tty/serial/omap-serial.c @@ -1092,7 +1092,7 @@ static void __maybe_unused wait_for_xmitr(struct uart_omap_port *up) } while (!uart_lsr_tx_empty(status)); /* Wait up to 1s for flow control if necessary */ - if (up->port.flags & UPF_CONS_FLOW) { + if (uart_cons_flow_enabled(&up->port)) { for (tmout = 1000000; tmout; tmout--) { unsigned int msr = serial_in(up, UART_MSR); diff --git a/drivers/tty/serial/pch_uart.c b/drivers/tty/serial/pch_uart.c index 9992fa231e4e..c3f32e0590b7 100644 --- a/drivers/tty/serial/pch_uart.c +++ b/drivers/tty/serial/pch_uart.c @@ -1451,7 +1451,7 @@ static void wait_for_xmitr(struct eg20t_port *up, int bits) } /* Wait up to 1s for flow control if necessary */ - if (up->port.flags & UPF_CONS_FLOW) { + if (uart_cons_flow_enabled(&up->port)) { unsigned int tmout; for (tmout = 1000000; tmout; tmout--) { unsigned int msr = ioread8(up->membase + UART_MSR); diff --git a/drivers/tty/serial/pxa.c b/drivers/tty/serial/pxa.c index e395ff29c1a2..027d936b5023 100644 --- a/drivers/tty/serial/pxa.c +++ b/drivers/tty/serial/pxa.c @@ -573,7 +573,7 @@ static void wait_for_xmitr(struct uart_pxa_port *up) } while (!uart_lsr_tx_empty(status)); /* Wait up to 1s for flow control if necessary */ - if (up->port.flags & UPF_CONS_FLOW) { + if (uart_cons_flow_enabled(&up->port)) { tmout = 1000000; while (--tmout && ((serial_in(up, UART_MSR) & UART_MSR_CTS) == 0)) diff --git a/drivers/tty/serial/qcom_geni_serial.c b/drivers/tty/serial/qcom_geni_serial.c index 1d561a00cbbd..953b73735710 100644 --- a/drivers/tty/serial/qcom_geni_serial.c +++ b/drivers/tty/serial/qcom_geni_serial.c @@ -136,6 +136,7 @@ struct qcom_geni_serial_port { unsigned int tx_remaining; unsigned int tx_queued; + bool tx_dma_stale; int wakeup_irq; bool rx_tx_swap; bool cts_rts_swap; @@ -688,6 +689,7 @@ static void qcom_geni_serial_start_tx_dma(struct uart_port *uport) } port->tx_remaining = xmit_size; + port->tx_dma_stale = false; } static void qcom_geni_serial_start_tx_fifo(struct uart_port *uport) @@ -1020,6 +1022,7 @@ static void qcom_geni_serial_handle_tx_dma(struct uart_port *uport) struct qcom_geni_serial_port *port = to_dev_port(uport); struct tty_port *tport = &uport->state->port; unsigned int fifo_len = kfifo_len(&tport->xmit_fifo); + bool tx_dma_stale = port->tx_dma_stale; /* * Only advance the kfifo if it still contains the bytes that were @@ -1030,12 +1033,13 @@ static void qcom_geni_serial_handle_tx_dma(struct uart_port *uport) * kfifo->in, making kfifo_len() wrap to UART_XMIT_SIZE - tx_remaining * and triggering a spurious large DMA transfer of stale data. */ - if (fifo_len >= port->tx_remaining) + if (!tx_dma_stale && fifo_len >= port->tx_remaining) uart_xmit_advance(uport, port->tx_remaining); geni_se_tx_dma_unprep(&port->se, port->tx_dma_addr, port->tx_remaining); port->tx_dma_addr = 0; port->tx_remaining = 0; + port->tx_dma_stale = false; if (!kfifo_is_empty(&tport->xmit_fifo)) qcom_geni_serial_start_tx_dma(uport); @@ -1173,6 +1177,10 @@ static void qcom_geni_serial_shutdown(struct uart_port *uport) static void qcom_geni_serial_flush_buffer_fifo(struct uart_port *uport) { + struct qcom_geni_serial_port *port = to_dev_port(uport); + + if (port->tx_dma_addr) + port->tx_dma_stale = true; qcom_geni_serial_cancel_tx_cmd(uport); } diff --git a/drivers/tty/serial/samsung_tty.c b/drivers/tty/serial/samsung_tty.c index 7a0b89d856c2..59d58fb743a3 100644 --- a/drivers/tty/serial/samsung_tty.c +++ b/drivers/tty/serial/samsung_tty.c @@ -311,7 +311,7 @@ static void s3c24xx_serial_stop_tx(struct uart_port *port) ourport->tx_enabled = 0; ourport->tx_in_progress = 0; - if (port->flags & UPF_CONS_FLOW) + if (uart_cons_flow_enabled(port)) s3c24xx_serial_rx_enable(port); ourport->tx_mode = 0; @@ -485,7 +485,7 @@ static void s3c24xx_serial_start_tx(struct uart_port *port) struct tty_port *tport = &port->state->port; if (!ourport->tx_enabled) { - if (port->flags & UPF_CONS_FLOW) + if (uart_cons_flow_enabled(port)) s3c24xx_serial_rx_disable(port); ourport->tx_enabled = 1; @@ -773,7 +773,7 @@ static void s3c24xx_serial_rx_drain_fifo(struct s3c24xx_uart_port *ourport) uerstat = rd_regl(port, S3C2410_UERSTAT); ch = rd_reg(port, S3C2410_URXH); - if (port->flags & UPF_CONS_FLOW) { + if (uart_cons_flow_enabled(port)) { bool txe = s3c24xx_serial_txempty_nofifo(port); if (ourport->rx_enabled) { @@ -1828,7 +1828,7 @@ static int s3c24xx_serial_init_port(struct s3c24xx_uart_port *ourport, if (cfg->uart_flags & UPF_CONS_FLOW) { dev_dbg(port->dev, "enabling flow control\n"); - port->flags |= UPF_CONS_FLOW; + uart_set_cons_flow_enabled(port, true); } /* sort our the physical and virtual addresses for each UART */ diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c index ca72454b3eb0..96599dee967b 100644 --- a/drivers/tty/serial/serial_core.c +++ b/drivers/tty/serial/serial_core.c @@ -2263,6 +2263,18 @@ uart_set_options(struct uart_port *port, struct console *co, port->mctrl |= TIOCM_DTR; port->ops->set_termios(port, &termios, &dummy); + + /* + * If console hardware flow control was specified and is supported, + * the related policy UPSTAT_CTS_ENABLE must be set to allow console + * drivers to identify if CTS should be used for polling. + */ + if (flow == 'r' && (termios.c_cflag & CRTSCTS)) { + /* Synchronize @status RMW update against the console. */ + guard(uart_port_lock_irqsave)(port); + port->status |= UPSTAT_CTS_ENABLE; + } + /* * Allow the setting of the UART parameters with a NULL console * too: @@ -2569,7 +2581,14 @@ uart_configure_port(struct uart_driver *drv, struct uart_state *state, * We probably don't need a spinlock around this, but */ scoped_guard(uart_port_lock_irqsave, port) { - port->mctrl &= TIOCM_DTR; + unsigned int mask = TIOCM_DTR; + + /* Console hardware flow control polls CTS. */ + if (uart_console_hwflow_active(port)) + mask |= TIOCM_RTS; + + port->mctrl &= mask; + if (!(port->rs485.flags & SER_RS485_ENABLED)) port->ops->set_mctrl(port, port->mctrl); } diff --git a/drivers/tty/serial/serial_txx9.c b/drivers/tty/serial/serial_txx9.c index 436a559234df..4ae9a45c8e3a 100644 --- a/drivers/tty/serial/serial_txx9.c +++ b/drivers/tty/serial/serial_txx9.c @@ -422,7 +422,7 @@ static void wait_for_xmitr(struct uart_port *up) udelay(1); /* Wait up to 1s for flow control if necessary */ - if (up->flags & UPF_CONS_FLOW) { + if (uart_cons_flow_enabled(up)) { tmout = 1000000; while (--tmout && (sio_in(up, TXX9_SICISR) & TXX9_SICISR_CTSS)) @@ -857,7 +857,7 @@ serial_txx9_console_write(struct console *co, const char *s, unsigned int count) * Disable flow-control if enabled (and unnecessary) */ flcr = sio_in(up, TXX9_SIFLCR); - if (!(up->flags & UPF_CONS_FLOW) && (flcr & TXX9_SIFLCR_TES)) + if (!uart_cons_flow_enabled(up) && (flcr & TXX9_SIFLCR_TES)) sio_out(up, TXX9_SIFLCR, flcr & ~TXX9_SIFLCR_TES); uart_console_write(up, s, count, serial_txx9_console_putchar); diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c index 383141fe7ba0..f71937fbe8aa 100644 --- a/drivers/tty/serial/sunsu.c +++ b/drivers/tty/serial/sunsu.c @@ -1245,7 +1245,7 @@ static void wait_for_xmitr(struct uart_sunsu_port *up) } while (!uart_lsr_tx_empty(status)); /* Wait up to 1s for flow control if necessary */ - if (up->port.flags & UPF_CONS_FLOW) { + if (uart_cons_flow_enabled(&up->port)) { tmout = 1000000; while (--tmout && ((serial_in(up, UART_MSR) & UART_MSR_CTS) == 0)) diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c index e2d92cf70eb7..f779f8fb138e 100644 --- a/drivers/tty/tty_io.c +++ b/drivers/tty/tty_io.c @@ -3167,8 +3167,10 @@ static int tty_cdev_add(struct tty_driver *driver, dev_t dev, driver->cdevs[index]->ops = &tty_fops; driver->cdevs[index]->owner = driver->owner; err = cdev_add(driver->cdevs[index], dev, count); - if (err) + if (err) { kobject_put(&driver->cdevs[index]->kobj); + driver->cdevs[index] = NULL; + } return err; } diff --git a/drivers/ufs/core/ufs-debugfs.c b/drivers/ufs/core/ufs-debugfs.c index e3baed6c70bd..6dece24baad1 100644 --- a/drivers/ufs/core/ufs-debugfs.c +++ b/drivers/ufs/core/ufs-debugfs.c @@ -165,7 +165,7 @@ static ssize_t ufs_saved_err_write(struct file *file, const char __user *buf, char val_str[16] = { }; int val, ret; - if (count > sizeof(val_str)) + if (count >= sizeof(val_str)) return -EINVAL; if (copy_from_user(val_str, buf, count)) return -EFAULT; diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c index 504600f1e08c..87578e8824d2 100644 --- a/drivers/ufs/core/ufshcd.c +++ b/drivers/ufs/core/ufshcd.c @@ -1338,6 +1338,7 @@ static int ufshcd_wait_for_pending_cmds(struct ufs_hba *hba, break; } + __set_current_state(TASK_UNINTERRUPTIBLE); io_schedule_timeout(msecs_to_jiffies(20)); if (ktime_to_us(ktime_sub(ktime_get(), start)) > wait_timeout_us) { diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c index d93ed4e86a17..4029585bf22c 100644 --- a/drivers/uio/uio.c +++ b/drivers/uio/uio.c @@ -1051,6 +1051,11 @@ int __uio_register_device(struct module *owner, err_request_irq: uio_dev_del_attributes(idev); err_uio_dev_add_attributes: + mutex_lock(&idev->info_lock); + idev->info = NULL; + mutex_unlock(&idev->info_lock); + wake_up_interruptible(&idev->wait); + kill_fasync(&idev->async_queue, SIGIO, POLL_HUP); device_del(&idev->dev); err_device_create: uio_free_minor(idev->minor); diff --git a/drivers/usb/atm/usbatm.c b/drivers/usb/atm/usbatm.c index 5f3ad9a99d9e..780be13af364 100644 --- a/drivers/usb/atm/usbatm.c +++ b/drivers/usb/atm/usbatm.c @@ -917,8 +917,8 @@ static int usbatm_atm_init(struct usbatm_data *instance) instance->atm_dev = atm_dev; - atm_dev->ci_range.vpi_bits = ATM_CI_MAX; - atm_dev->ci_range.vci_bits = ATM_CI_MAX; + atm_dev->ci_range.vpi_bits = 8; + atm_dev->ci_range.vci_bits = 16; atm_dev->signal = ATM_PHY_SIG_UNKNOWN; /* temp init ATM device, set to 128kbit */ diff --git a/drivers/usb/cdns3/cdnsp-gadget.c b/drivers/usb/cdns3/cdnsp-gadget.c index dc9a25317990..1ffd3bb86707 100644 --- a/drivers/usb/cdns3/cdnsp-gadget.c +++ b/drivers/usb/cdns3/cdnsp-gadget.c @@ -1820,6 +1820,82 @@ static void cdnsp_get_rev_cap(struct cdnsp_device *pdev) readl(&pdev->rev_cap->tx_buff_size)); } +static void cdnsp_set_event_deq(struct cdnsp_device *pdev) +{ + dma_addr_t deq; + u64 temp; + + deq = cdnsp_trb_virt_to_dma(pdev->event_ring->deq_seg, + pdev->event_ring->dequeue); + + /* Update controller event ring dequeue pointer */ + temp = cdnsp_read_64(&pdev->ir_set->erst_dequeue); + temp &= ERST_PTR_MASK; + + /* + * Don't clear the EHB bit (which is RW1C) because + * there might be more events to service. + */ + temp &= ~ERST_EHB; + + cdnsp_write_64(((u64)deq & (u64)~ERST_PTR_MASK) | temp, + &pdev->ir_set->erst_dequeue); +} + +static void cdnsp_add_interrupter(struct cdnsp_device *pdev) +{ + u64 erst_base; + u32 erst_size; + + /* Set ERST count with the number of entries in the segment table. */ + erst_size = readl(&pdev->ir_set->erst_size); + erst_size &= ERST_SIZE_MASK; + erst_size |= ERST_NUM_SEGS; + writel(erst_size, &pdev->ir_set->erst_size); + + /* Set the segment table base address. */ + erst_base = cdnsp_read_64(&pdev->ir_set->erst_base); + erst_base &= ERST_PTR_MASK; + erst_base |= (pdev->erst.erst_dma_addr & (u64)~ERST_PTR_MASK); + cdnsp_write_64(erst_base, &pdev->ir_set->erst_base); + + /* Set the event ring dequeue address. */ + cdnsp_set_event_deq(pdev); +} + +/* Set up basic CDNSP registers */ +static void cdnsp_init(struct cdnsp_device *pdev) +{ + unsigned int val; + u64 val_64; + + val = readl(&pdev->op_regs->config_reg); + val |= ((val & ~MAX_DEVS) | CDNSP_DEV_MAX_SLOTS) | CONFIG_U3E; + writel(val, &pdev->op_regs->config_reg); + + /* Initialize the Command ring */ + cdnsp_ring_init(pdev, pdev->cmd_ring); + + /* Set the address in the Command Ring Control register */ + val_64 = cdnsp_read_64(&pdev->op_regs->cmd_ring); + val_64 = (val_64 & (u64)CMD_RING_RSVD_BITS) | + (pdev->cmd_ring->first_seg->dma & (u64)~CMD_RING_RSVD_BITS) | + pdev->cmd_ring->cycle_state; + cdnsp_write_64(val_64, &pdev->op_regs->cmd_ring); + + /* Set Device Context Base Address Array pointer */ + cdnsp_write_64(pdev->dcbaa->dma, &pdev->op_regs->dcbaa_ptr); + + /* Set Doorbell array pointer */ + val = readl(&pdev->cap_regs->db_off); + val &= DBOFF_MASK; + pdev->dba = (void __iomem *)pdev->cap_regs + val; + + /* Initialize the Primary interrupter */ + cdnsp_ring_init(pdev, pdev->event_ring); + cdnsp_add_interrupter(pdev); +} + static int cdnsp_gen_setup(struct cdnsp_device *pdev) { int ret; @@ -1885,6 +1961,8 @@ static int cdnsp_gen_setup(struct cdnsp_device *pdev) if (ret) return ret; + cdnsp_init(pdev); + /* * Software workaround for U1: after transition * to U1 the controller starts gating clock, and in some cases, @@ -2015,9 +2093,6 @@ static int cdnsp_gadget_suspend(struct cdns *cdns, bool do_wakeup) struct cdnsp_device *pdev = cdns->gadget_dev; unsigned long flags; - if (pdev->link_state == XDEV_U3) - return 0; - spin_lock_irqsave(&pdev->lock, flags); cdnsp_disconnect_gadget(pdev); cdnsp_stop(pdev); @@ -2031,12 +2106,38 @@ static int cdnsp_gadget_resume(struct cdns *cdns, bool lost_power) struct cdnsp_device *pdev = cdns->gadget_dev; enum usb_device_speed max_speed; unsigned long flags; + bool context_lost; + u32 val; int ret; if (!pdev->gadget_driver) return 0; spin_lock_irqsave(&pdev->lock, flags); + val = readl(&pdev->port3x_regs->mode_2); + context_lost = !!(val & CFG_3XPORT_U1_PIPE_CLK_GATE_EN) || lost_power; + + if (context_lost) { + cdnsp_halt(pdev); + cdnsp_set_apb_timeout_value(pdev); + + /* Reset the internal controller memory state and registers. */ + ret = cdnsp_reset(pdev); + if (ret) + goto unlock; + + val = readl(&pdev->port3x_regs->mode_2); + val &= ~CFG_3XPORT_U1_PIPE_CLK_GATE_EN; + writel(val, &pdev->port3x_regs->mode_2); + + cdnsp_clear_cmd_ring(pdev); + + memset(pdev->event_ring->first_seg->trbs, 0, + sizeof(union cdnsp_trb) * (TRBS_PER_SEGMENT)); + + cdnsp_init(pdev); + } + max_speed = pdev->gadget_driver->max_speed; /* Limit speed if necessary. */ @@ -2044,9 +2145,10 @@ static int cdnsp_gadget_resume(struct cdns *cdns, bool lost_power) ret = cdnsp_run(pdev, max_speed); - if (pdev->link_state == XDEV_U3) + if (!context_lost && pdev->link_state == XDEV_U3) __cdnsp_gadget_wakeup(pdev); +unlock: spin_unlock_irqrestore(&pdev->lock, flags); return ret; diff --git a/drivers/usb/cdns3/cdnsp-gadget.h b/drivers/usb/cdns3/cdnsp-gadget.h index a91cca509db0..c157023c9376 100644 --- a/drivers/usb/cdns3/cdnsp-gadget.h +++ b/drivers/usb/cdns3/cdnsp-gadget.h @@ -1509,6 +1509,7 @@ int cdnsp_endpoint_init(struct cdnsp_device *pdev, int cdnsp_ring_expansion(struct cdnsp_device *pdev, struct cdnsp_ring *ring, unsigned int num_trbs, gfp_t flags); +void cdnsp_ring_init(struct cdnsp_device *pdev, struct cdnsp_ring *ring); struct cdnsp_ring *cdnsp_dma_to_transfer_ring(struct cdnsp_ep *ep, u64 address); int cdnsp_alloc_stream_info(struct cdnsp_device *pdev, struct cdnsp_ep *pep, diff --git a/drivers/usb/cdns3/cdnsp-mem.c b/drivers/usb/cdns3/cdnsp-mem.c index 3f35c8af271e..4b66307baaeb 100644 --- a/drivers/usb/cdns3/cdnsp-mem.c +++ b/drivers/usb/cdns3/cdnsp-mem.c @@ -394,13 +394,6 @@ static struct cdnsp_ring *cdnsp_ring_alloc(struct cdnsp_device *pdev, if (ret) goto fail; - /* Only event ring does not use link TRB. */ - if (type != TYPE_EVENT) - ring->last_seg->trbs[TRBS_PER_SEGMENT - 1].link.control |= - cpu_to_le32(LINK_TOGGLE); - - cdnsp_initialize_ring_info(ring); - trace_cdnsp_ring_alloc(ring); return ring; fail: kfree(ring); @@ -604,6 +597,7 @@ int cdnsp_alloc_stream_info(struct cdnsp_device *pdev, if (!cur_ring) goto cleanup_rings; + cdnsp_ring_init(pdev, cur_ring); cur_ring->stream_id = cur_stream; cur_ring->trb_address_map = &stream_info->trb_address_map; @@ -699,6 +693,8 @@ static int cdnsp_alloc_priv_device(struct cdnsp_device *pdev) if (!pdev->eps[0].ring) goto fail; + cdnsp_ring_init(pdev, pdev->eps[0].ring); + /* Point to output device context in dcbaa. */ pdev->dcbaa->dev_context_ptrs[1] = cpu_to_le64(pdev->out_ctx.dma); pdev->cmd.in_ctx = &pdev->in_ctx; @@ -992,6 +988,8 @@ int cdnsp_endpoint_init(struct cdnsp_device *pdev, if (!pep->ring) return -ENOMEM; + cdnsp_ring_init(pdev, pep->ring); + pep->skip = false; /* Fill the endpoint context */ @@ -1099,28 +1097,6 @@ void cdnsp_mem_cleanup(struct cdnsp_device *pdev) pdev->active_port = NULL; } -static void cdnsp_set_event_deq(struct cdnsp_device *pdev) -{ - dma_addr_t deq; - u64 temp; - - deq = cdnsp_trb_virt_to_dma(pdev->event_ring->deq_seg, - pdev->event_ring->dequeue); - - /* Update controller event ring dequeue pointer */ - temp = cdnsp_read_64(&pdev->ir_set->erst_dequeue); - temp &= ERST_PTR_MASK; - - /* - * Don't clear the EHB bit (which is RW1C) because - * there might be more events to service. - */ - temp &= ~ERST_EHB; - - cdnsp_write_64(((u64)deq & (u64)~ERST_PTR_MASK) | temp, - &pdev->ir_set->erst_dequeue); -} - static void cdnsp_add_in_port(struct cdnsp_device *pdev, struct cdnsp_port *port, __le32 __iomem *addr) @@ -1200,6 +1176,36 @@ static int cdnsp_setup_port_arrays(struct cdnsp_device *pdev) return 0; } +static void cdnsp_initialize_ring_segments(struct cdnsp_device *pdev, struct cdnsp_ring *ring) +{ + struct cdnsp_segment *seg; + + /* Only event ring does not use link TRB. */ + if (ring->type == TYPE_EVENT) + return; + + seg = ring->first_seg; + + while (seg) { + struct cdnsp_segment *next = seg->next; + + cdnsp_link_segments(pdev, seg, next, ring->type); + if (next == ring->first_seg) + break; + + seg = next; + } + + ring->last_seg->trbs[TRBS_PER_SEGMENT - 1].link.control |= cpu_to_le32(LINK_TOGGLE); +} + +void cdnsp_ring_init(struct cdnsp_device *pdev, struct cdnsp_ring *ring) +{ + cdnsp_initialize_ring_segments(pdev, ring); + cdnsp_initialize_ring_info(ring); + trace_cdnsp_ring_alloc(ring); +} + /* * Initialize memory for CDNSP (one-time init). * @@ -1211,10 +1217,8 @@ int cdnsp_mem_init(struct cdnsp_device *pdev) { struct device *dev = pdev->dev; int ret = -ENOMEM; - unsigned int val; dma_addr_t dma; u32 page_size; - u64 val_64; /* * Use 4K pages, since that's common and the minimum the @@ -1222,10 +1226,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev) */ page_size = 1 << 12; - val = readl(&pdev->op_regs->config_reg); - val |= ((val & ~MAX_DEVS) | CDNSP_DEV_MAX_SLOTS) | CONFIG_U3E; - writel(val, &pdev->op_regs->config_reg); - /* * Doorbell array must be physically contiguous * and 64-byte (cache line) aligned. @@ -1237,8 +1237,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev) pdev->dcbaa->dma = dma; - cdnsp_write_64(dma, &pdev->op_regs->dcbaa_ptr); - /* * Initialize the ring segment pool. The ring must be a contiguous * structure comprised of TRBs. The TRBs must be 16 byte aligned, @@ -1264,17 +1262,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev) if (!pdev->cmd_ring) goto destroy_device_pool; - /* Set the address in the Command Ring Control register */ - val_64 = cdnsp_read_64(&pdev->op_regs->cmd_ring); - val_64 = (val_64 & (u64)CMD_RING_RSVD_BITS) | - (pdev->cmd_ring->first_seg->dma & (u64)~CMD_RING_RSVD_BITS) | - pdev->cmd_ring->cycle_state; - cdnsp_write_64(val_64, &pdev->op_regs->cmd_ring); - - val = readl(&pdev->cap_regs->db_off); - val &= DBOFF_MASK; - pdev->dba = (void __iomem *)pdev->cap_regs + val; - /* Set ir_set to interrupt register set 0 */ pdev->ir_set = &pdev->run_regs->ir_set[0]; @@ -1291,21 +1278,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev) if (ret) goto free_event_ring; - /* Set ERST count with the number of entries in the segment table. */ - val = readl(&pdev->ir_set->erst_size); - val &= ERST_SIZE_MASK; - val |= ERST_NUM_SEGS; - writel(val, &pdev->ir_set->erst_size); - - /* Set the segment table base address. */ - val_64 = cdnsp_read_64(&pdev->ir_set->erst_base); - val_64 &= ERST_PTR_MASK; - val_64 |= (pdev->erst.erst_dma_addr & (u64)~ERST_PTR_MASK); - cdnsp_write_64(val_64, &pdev->ir_set->erst_base); - - /* Set the event ring dequeue address. */ - cdnsp_set_event_deq(pdev); - ret = cdnsp_setup_port_arrays(pdev); if (ret) goto free_erst; diff --git a/drivers/usb/core/driver.c b/drivers/usb/core/driver.c index 74b8bdc27dbf..acd05f87b8e1 100644 --- a/drivers/usb/core/driver.c +++ b/drivers/usb/core/driver.c @@ -228,14 +228,16 @@ static void usb_free_dynids(struct usb_driver *usb_drv) } static const struct usb_device_id *usb_match_dynamic_id(struct usb_interface *intf, - const struct usb_driver *drv) + const struct usb_driver *drv, + struct usb_device_id *id_copy) { struct usb_dynid *dynid; guard(mutex)(&usb_dynids_lock); list_for_each_entry(dynid, &drv->dynids.list, node) { if (usb_match_one_id(intf, &dynid->id)) { - return &dynid->id; + *id_copy = dynid->id; + return id_copy; } } return NULL; @@ -321,6 +323,7 @@ static int usb_probe_interface(struct device *dev) struct usb_interface *intf = to_usb_interface(dev); struct usb_device *udev = interface_to_usbdev(intf); const struct usb_device_id *id; + struct usb_device_id id_copy; int error = -ENODEV; int lpm_disable_error = -ENODEV; @@ -340,7 +343,7 @@ static int usb_probe_interface(struct device *dev) return error; } - id = usb_match_dynamic_id(intf, driver); + id = usb_match_dynamic_id(intf, driver, &id_copy); if (!id) id = usb_match_id(intf, driver->id_table); if (!id) @@ -892,6 +895,7 @@ static int usb_device_match(struct device *dev, const struct device_driver *drv) struct usb_interface *intf; const struct usb_driver *usb_drv; const struct usb_device_id *id; + struct usb_device_id id_copy; /* device drivers never match interfaces */ if (is_usb_device_driver(drv)) @@ -904,7 +908,7 @@ static int usb_device_match(struct device *dev, const struct device_driver *drv) if (id) return 1; - id = usb_match_dynamic_id(intf, usb_drv); + id = usb_match_dynamic_id(intf, usb_drv, &id_copy); if (id) return 1; } diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c index 6bcac85c5550..aa0a25efb6aa 100644 --- a/drivers/usb/gadget/configfs.c +++ b/drivers/usb/gadget/configfs.c @@ -1177,7 +1177,7 @@ static ssize_t os_desc_qw_sign_show(struct config_item *item, char *page) struct gadget_info *gi = os_desc_item_to_gadget_info(item); int res; - res = utf16s_to_utf8s((wchar_t *) gi->qw_sign, OS_STRING_QW_SIGN_LEN, + res = utf16s_to_utf8s((wchar_t *) gi->qw_sign, OS_STRING_QW_SIGN_LEN / 2, UTF16_LITTLE_ENDIAN, page, PAGE_SIZE - 1); page[res++] = '\n'; @@ -1199,7 +1199,7 @@ static ssize_t os_desc_qw_sign_store(struct config_item *item, const char *page, mutex_lock(&gi->lock); res = utf8s_to_utf16s(page, l, UTF16_LITTLE_ENDIAN, (wchar_t *) gi->qw_sign, - OS_STRING_QW_SIGN_LEN); + OS_STRING_QW_SIGN_LEN / 2); if (res > 0) res = len; mutex_unlock(&gi->lock); diff --git a/drivers/usb/gadget/function/f_fs.c b/drivers/usb/gadget/function/f_fs.c index 09d6872ad38b..d73b082d5d83 100644 --- a/drivers/usb/gadget/function/f_fs.c +++ b/drivers/usb/gadget/function/f_fs.c @@ -1683,13 +1683,13 @@ static int ffs_dmabuf_transfer(struct file *file, /* In the meantime, endpoint got disabled or changed. */ if (epfile->ep != ep) { ret = -ESHUTDOWN; - goto err_fence_put; + goto err_fence_free; } usb_req = usb_ep_alloc_request(ep->ep, GFP_ATOMIC); if (!usb_req) { ret = -ENOMEM; - goto err_fence_put; + goto err_fence_free; } /* @@ -1738,9 +1738,9 @@ static int ffs_dmabuf_transfer(struct file *file, return ret; -err_fence_put: +err_fence_free: spin_unlock_irq(&epfile->ffs->eps_lock); - dma_fence_put(&fence->base); + kfree(fence); err_resv_unlock: dma_resv_unlock(dmabuf->resv); err_attachment_put: diff --git a/drivers/usb/gadget/function/f_mass_storage.c b/drivers/usb/gadget/function/f_mass_storage.c index 6debf6684f4a..6970d0fa806b 100644 --- a/drivers/usb/gadget/function/f_mass_storage.c +++ b/drivers/usb/gadget/function/f_mass_storage.c @@ -2748,6 +2748,9 @@ int fsg_common_set_num_buffers(struct fsg_common *common, unsigned int n) struct fsg_buffhd *bh, *buffhds; int i; + if (n < 2) + return -EINVAL; + buffhds = kcalloc(n, sizeof(*buffhds), GFP_KERNEL); if (!buffhds) return -ENOMEM; diff --git a/drivers/usb/gadget/function/f_uac1_legacy.c b/drivers/usb/gadget/function/f_uac1_legacy.c index 4981af8337ab..c234f9fea27c 100644 --- a/drivers/usb/gadget/function/f_uac1_legacy.c +++ b/drivers/usb/gadget/function/f_uac1_legacy.c @@ -888,60 +888,10 @@ UAC1_INT_ATTRIBUTE(req_buf_size); UAC1_INT_ATTRIBUTE(req_count); UAC1_INT_ATTRIBUTE(audio_buf_size); -#define UAC1_STR_ATTRIBUTE(name) \ -static ssize_t f_uac1_opts_##name##_show(struct config_item *item, \ - char *page) \ -{ \ - struct f_uac1_legacy_opts *opts = to_f_uac1_opts(item); \ - int result; \ - \ - mutex_lock(&opts->lock); \ - result = sprintf(page, "%s\n", opts->name); \ - mutex_unlock(&opts->lock); \ - \ - return result; \ -} \ - \ -static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \ - const char *page, size_t len) \ -{ \ - struct f_uac1_legacy_opts *opts = to_f_uac1_opts(item); \ - int ret = -EBUSY; \ - char *tmp; \ - \ - mutex_lock(&opts->lock); \ - if (opts->refcnt) \ - goto end; \ - \ - tmp = kstrndup(page, len, GFP_KERNEL); \ - if (tmp) { \ - ret = -ENOMEM; \ - goto end; \ - } \ - if (opts->name##_alloc) \ - kfree(opts->name); \ - opts->name##_alloc = true; \ - opts->name = tmp; \ - ret = len; \ - \ -end: \ - mutex_unlock(&opts->lock); \ - return ret; \ -} \ - \ -CONFIGFS_ATTR(f_uac1_opts_, name) - -UAC1_STR_ATTRIBUTE(fn_play); -UAC1_STR_ATTRIBUTE(fn_cap); -UAC1_STR_ATTRIBUTE(fn_cntl); - static struct configfs_attribute *f_uac1_attrs[] = { &f_uac1_opts_attr_req_buf_size, &f_uac1_opts_attr_req_count, &f_uac1_opts_attr_audio_buf_size, - &f_uac1_opts_attr_fn_play, - &f_uac1_opts_attr_fn_cap, - &f_uac1_opts_attr_fn_cntl, NULL, }; @@ -956,12 +906,6 @@ static void f_audio_free_inst(struct usb_function_instance *f) struct f_uac1_legacy_opts *opts; opts = container_of(f, struct f_uac1_legacy_opts, func_inst); - if (opts->fn_play_alloc) - kfree(opts->fn_play); - if (opts->fn_cap_alloc) - kfree(opts->fn_cap); - if (opts->fn_cntl_alloc) - kfree(opts->fn_cntl); kfree(opts); } diff --git a/drivers/usb/gadget/function/u_uac1_legacy.h b/drivers/usb/gadget/function/u_uac1_legacy.h index b5df9bcbbeba..b9ddae550ff3 100644 --- a/drivers/usb/gadget/function/u_uac1_legacy.h +++ b/drivers/usb/gadget/function/u_uac1_legacy.h @@ -62,9 +62,6 @@ struct f_uac1_legacy_opts { char *fn_cap; char *fn_cntl; unsigned bound:1; - unsigned fn_play_alloc:1; - unsigned fn_cap_alloc:1; - unsigned fn_cntl_alloc:1; struct mutex lock; int refcnt; }; diff --git a/drivers/usb/gadget/udc/aspeed_udc.c b/drivers/usb/gadget/udc/aspeed_udc.c index 353bfb1ff0a1..1757570f7b3a 100644 --- a/drivers/usb/gadget/udc/aspeed_udc.c +++ b/drivers/usb/gadget/udc/aspeed_udc.c @@ -1434,25 +1434,12 @@ static void ast_udc_init_hw(struct ast_udc_dev *udc) ast_udc_write(udc, 0, AST_UDC_EP0_CTRL); } -static void ast_udc_remove(struct platform_device *pdev) +static void ast_udc_cleanup(struct platform_device *pdev) { struct ast_udc_dev *udc = platform_get_drvdata(pdev); unsigned long flags; u32 ctrl; - usb_del_gadget_udc(&udc->gadget); - if (udc->driver) { - /* - * This is broken as only some cleanup is skipped, *udev is - * freed and the register mapping goes away. Any further usage - * probably crashes. Also the device is unbound, so the skipped - * cleanup is never catched up later. - */ - dev_alert(&pdev->dev, - "Driver is busy and still going away. Fasten your seat belts!\n"); - return; - } - spin_lock_irqsave(&udc->lock, flags); /* Disable upstream port connection */ @@ -1472,6 +1459,26 @@ static void ast_udc_remove(struct platform_device *pdev) udc->ep0_buf = NULL; } +static void ast_udc_remove(struct platform_device *pdev) +{ + struct ast_udc_dev *udc = platform_get_drvdata(pdev); + + usb_del_gadget_udc(&udc->gadget); + if (udc->driver) { + /* + * This is broken as only some cleanup is skipped, *udev is + * freed and the register mapping goes away. Any further usage + * probably crashes. Also the device is unbound, so the skipped + * cleanup is never catched up later. + */ + dev_alert(&pdev->dev, + "Driver is busy and still going away. Fasten your seat belts!\n"); + return; + } + + ast_udc_cleanup(pdev); +} + static int ast_udc_probe(struct platform_device *pdev) { enum usb_device_speed max_speed; @@ -1524,6 +1531,12 @@ static int ast_udc_probe(struct platform_device *pdev) AST_UDC_NUM_ENDPOINTS, &udc->ep0_buf_dma, GFP_KERNEL); + if (!udc->ep0_buf) { + clk_disable_unprepare(udc->clk); + rc = -ENOMEM; + goto err; + } + udc->gadget.speed = USB_SPEED_UNKNOWN; udc->gadget.max_speed = USB_SPEED_HIGH; udc->creq = udc->reg + AST_UDC_SETUP0; @@ -1553,20 +1566,20 @@ static int ast_udc_probe(struct platform_device *pdev) udc->irq = platform_get_irq(pdev, 0); if (udc->irq < 0) { rc = udc->irq; - goto err; + goto err_cleanup; } rc = devm_request_irq(&pdev->dev, udc->irq, ast_udc_isr, 0, KBUILD_MODNAME, udc); if (rc) { dev_err(&pdev->dev, "Failed to request interrupt\n"); - goto err; + goto err_cleanup; } rc = usb_add_gadget_udc(&pdev->dev, &udc->gadget); if (rc) { dev_err(&pdev->dev, "Failed to add gadget udc\n"); - goto err; + goto err_cleanup; } dev_info(&pdev->dev, "Initialized udc in USB%s mode\n", @@ -1574,9 +1587,10 @@ static int ast_udc_probe(struct platform_device *pdev) return 0; +err_cleanup: + ast_udc_cleanup(pdev); err: dev_err(&pdev->dev, "Failed to udc probe, rc:0x%x\n", rc); - ast_udc_remove(pdev); return rc; } diff --git a/drivers/usb/gadget/udc/r8a66597-udc.c b/drivers/usb/gadget/udc/r8a66597-udc.c index e5c2630e3711..29ae81c95d0a 100644 --- a/drivers/usb/gadget/udc/r8a66597-udc.c +++ b/drivers/usb/gadget/udc/r8a66597-udc.c @@ -1951,7 +1951,6 @@ static int r8a66597_probe(struct platform_device *pdev) return 0; err_add_udc: - r8a66597_free_request(&r8a66597->ep[0].ep, r8a66597->ep0_req); clean_up2: if (r8a66597->pdata->on_chip) clk_disable_unprepare(r8a66597->clk); diff --git a/drivers/usb/misc/usb-ljca.c b/drivers/usb/misc/usb-ljca.c index c562630d862c..b19cc99edfeb 100644 --- a/drivers/usb/misc/usb-ljca.c +++ b/drivers/usb/misc/usb-ljca.c @@ -585,6 +585,9 @@ static int ljca_enumerate_gpio(struct ljca_adapter *adap) if (gpio_num > LJCA_MAX_GPIO_NUM) return -EINVAL; + if (desc->bank_num > ARRAY_SIZE(valid_pin)) + return -EINVAL; + /* construct platform data */ gpio_info = kzalloc(sizeof *gpio_info, GFP_KERNEL); if (!gpio_info) diff --git a/drivers/usb/mtu3/mtu3_core.c b/drivers/usb/mtu3/mtu3_core.c index a3a6282893d0..ca8cb6c3a01b 100644 --- a/drivers/usb/mtu3/mtu3_core.c +++ b/drivers/usb/mtu3/mtu3_core.c @@ -1037,9 +1037,14 @@ int ssusb_gadget_suspend(struct ssusb_mtk *ssusb, pm_message_t msg) if (!mtu->gadget_driver) return 0; - if (mtu->connected) + /* Prevent runtime suspend when active connection exists */ + if (mtu->connected && PMSG_IS_AUTO(msg)) return -EBUSY; + /* Perform soft disconnect for system suspend */ + if (mtu->softconnect && !PMSG_IS_AUTO(msg)) + mtu3_dev_on_off(mtu, 0); + mtu3_dev_suspend(mtu); synchronize_irq(mtu->irq); @@ -1055,5 +1060,9 @@ int ssusb_gadget_resume(struct ssusb_mtk *ssusb, pm_message_t msg) mtu3_dev_resume(mtu); + /* Restore soft connect for system resume */ + if (mtu->softconnect && !PMSG_IS_AUTO(msg)) + mtu3_dev_on_off(mtu, 1); + return 0; } diff --git a/drivers/usb/renesas_usbhs/common.c b/drivers/usb/renesas_usbhs/common.c index 8a79548e1569..fddb57b8d847 100644 --- a/drivers/usb/renesas_usbhs/common.c +++ b/drivers/usb/renesas_usbhs/common.c @@ -813,9 +813,6 @@ static void usbhs_remove(struct platform_device *pdev) flush_delayed_work(&priv->notify_hotplug_work); - usbhs_platform_call(priv, hardware_exit, pdev); - reset_control_assert(priv->rsts); - /* * Explicitly free the IRQ to ensure the interrupt handler is * disabled and synchronized before freeing resources. @@ -832,6 +829,9 @@ static void usbhs_remove(struct platform_device *pdev) if (!usbhs_get_dparam(priv, runtime_pwctrl)) usbhsc_power_ctrl(priv, 0); + usbhs_platform_call(priv, hardware_exit, pdev); + reset_control_assert(priv->rsts); + usbhsc_clk_put(priv); pm_runtime_disable(&pdev->dev); } diff --git a/drivers/usb/typec/ucsi/debugfs.c b/drivers/usb/typec/ucsi/debugfs.c index f73f2b54554e..414ffe1d2378 100644 --- a/drivers/usb/typec/ucsi/debugfs.c +++ b/drivers/usb/typec/ucsi/debugfs.c @@ -129,6 +129,7 @@ void ucsi_debugfs_unregister(struct ucsi *ucsi) debugfs_remove_recursive(ucsi->debugfs->dentry); kfree(ucsi->debugfs); + ucsi->debugfs = NULL; } void ucsi_debugfs_init(void) diff --git a/drivers/usb/typec/ucsi/ucsi.c b/drivers/usb/typec/ucsi/ucsi.c index 85e2e7fcd02c..2acd48a0a6a2 100644 --- a/drivers/usb/typec/ucsi/ucsi.c +++ b/drivers/usb/typec/ucsi/ucsi.c @@ -2275,6 +2275,8 @@ void ucsi_unregister(struct ucsi *ucsi) cancel_delayed_work_sync(&ucsi->work); cancel_work_sync(&ucsi->resume_work); + ucsi_debugfs_unregister(ucsi); + /* Disable notifications */ ucsi->ops->async_control(ucsi, cmd); diff --git a/drivers/usb/typec/ucsi/ucsi_huawei_gaokun.c b/drivers/usb/typec/ucsi/ucsi_huawei_gaokun.c index c5965656baba..4d53f37ae119 100644 --- a/drivers/usb/typec/ucsi/ucsi_huawei_gaokun.c +++ b/drivers/usb/typec/ucsi/ucsi_huawei_gaokun.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include "ucsi.h" @@ -82,6 +83,10 @@ struct gaokun_ucsi_port { struct gaokun_ucsi *ucsi; struct auxiliary_device *bridge; + struct typec_mux *typec_mux; + struct typec_mux_state state; + struct typec_altmode dp_alt; + int idx; enum gaokun_ucsi_ccx ccx; enum gaokun_ucsi_mux mux; @@ -100,6 +105,7 @@ struct gaokun_ucsi { struct notifier_block nb; u16 version; u8 num_ports; + bool registered; }; /* -------------------------------------------------------------------------- */ @@ -226,19 +232,18 @@ static void gaokun_ucsi_port_update(struct gaokun_ucsi_port *port, port->hpd_state = FIELD_GET(GAOKUN_HPD_STATE_MASK, ddi); port->hpd_irq = FIELD_GET(GAOKUN_HPD_IRQ_MASK, ddi); - /* Mode and SVID are unused; keeping them to make things clearer */ switch (port->mode) { case USBC_DPAM_PAN_C: case USBC_DPAM_PAN_C_REVERSE: - port->mode = DP_PIN_ASSIGN_C; /* correct it for usb later */ + port->mode = TYPEC_DP_STATE_C; /* correct it for usb later */ break; case USBC_DPAM_PAN_D: case USBC_DPAM_PAN_D_REVERSE: - port->mode = DP_PIN_ASSIGN_D; + port->mode = TYPEC_DP_STATE_D; break; case USBC_DPAM_PAN_E: case USBC_DPAM_PAN_E_REVERSE: - port->mode = DP_PIN_ASSIGN_E; + port->mode = TYPEC_DP_STATE_E; break; case USBC_DPAM_PAN_NONE: port->mode = TYPEC_STATE_SAFE; @@ -287,18 +292,30 @@ static int gaokun_ucsi_refresh(struct gaokun_ucsi *uec) return idx; } -static void gaokun_ucsi_handle_altmode(struct gaokun_ucsi_port *port) +static void gaokun_ucsi_handle_usb_mode(struct gaokun_ucsi_port *port) { struct gaokun_ucsi *uec = port->ucsi; - int idx = port->idx; + int idx = port->idx, ret; + + /* + * For every typec port on this platform, the only mode-switch is + * controlled by its qmp combo phy which consumes svid and mode only. + */ + port->dp_alt.svid = port->svid; + port->state.mode = port->mode; + port->state.alt = &port->dp_alt; - if (idx >= uec->ucsi->cap.num_connectors) { + if (idx >= uec->num_ports) { dev_warn(uec->dev, "altmode port out of range: %d\n", idx); return; } + ret = typec_mux_set(port->typec_mux, &port->state); + if (ret) + dev_err(uec->dev, "failed to set mux %d\n", ret); + /* UCSI callback .connector_status() have set orientation */ - if (port->bridge) + if (port->bridge && port->svid == USB_TYPEC_DP_SID) drm_aux_hpd_bridge_notify(&port->bridge->dev, port->hpd_state ? connector_status_connected : @@ -307,7 +324,7 @@ static void gaokun_ucsi_handle_altmode(struct gaokun_ucsi_port *port) gaokun_ec_ucsi_pan_ack(uec->ec, port->idx); } -static void gaokun_ucsi_altmode_notify_ind(struct gaokun_ucsi *uec) +static void gaokun_ucsi_usb_notify_ind(struct gaokun_ucsi *uec) { int idx; @@ -320,7 +337,7 @@ static void gaokun_ucsi_altmode_notify_ind(struct gaokun_ucsi *uec) if (idx == GAOKUN_UCSI_NO_PORT_UPDATE) gaokun_ec_ucsi_pan_ack(uec->ec, idx); /* ack directly if no update */ else - gaokun_ucsi_handle_altmode(&uec->ports[idx]); + gaokun_ucsi_handle_usb_mode(&uec->ports[idx]); } /* @@ -352,7 +369,7 @@ static void gaokun_ucsi_handle_no_usb_event(struct gaokun_ucsi *uec, int idx) port = &uec->ports[idx]; if (!wait_for_completion_timeout(&port->usb_ack, 2 * HZ)) { dev_warn(uec->dev, "No USB EVENT, triggered by UCSI EVENT"); - gaokun_ucsi_altmode_notify_ind(uec); + gaokun_ucsi_usb_notify_ind(uec); } } @@ -366,7 +383,7 @@ static int gaokun_ucsi_notify(struct notifier_block *nb, switch (action) { case EC_EVENT_USB: gaokun_ucsi_complete_usb_ack(uec); - gaokun_ucsi_altmode_notify_ind(uec); + gaokun_ucsi_usb_notify_ind(uec); return NOTIFY_OK; case EC_EVENT_UCSI: @@ -429,8 +446,15 @@ static int gaokun_ucsi_ports_init(struct gaokun_ucsi *uec) fwnode_handle_put(fwnode); return PTR_ERR(ucsi_port->bridge); } - } + ucsi_port->typec_mux = fwnode_typec_mux_get(fwnode); + if (IS_ERR(ucsi_port->typec_mux)) { + fwnode_handle_put(fwnode); + return dev_err_probe(dev, PTR_ERR(ucsi_port->typec_mux), + "failed to acquire mode-switch for port: %d\n", + port); + } + } for (i = 0; i < num_ports; i++) { if (!uec->ports[i].bridge) continue; @@ -459,8 +483,13 @@ static void gaokun_ucsi_register_worker(struct work_struct *work) } ret = ucsi_register(ucsi); - if (ret) + if (ret) { dev_err_probe(ucsi->dev, ret, "ucsi register failed\n"); + gaokun_ec_unregister_notify(uec->ec, &uec->nb); + return; + } + + uec->registered = true; } static int gaokun_ucsi_probe(struct auxiliary_device *adev, @@ -502,10 +531,17 @@ static int gaokun_ucsi_probe(struct auxiliary_device *adev, static void gaokun_ucsi_remove(struct auxiliary_device *adev) { struct gaokun_ucsi *uec = auxiliary_get_drvdata(adev); + int i; disable_delayed_work_sync(&uec->work); - gaokun_ec_unregister_notify(uec->ec, &uec->nb); - ucsi_unregister(uec->ucsi); + if (uec->registered) { + gaokun_ec_unregister_notify(uec->ec, &uec->nb); + ucsi_unregister(uec->ucsi); + } + + for (i = 0; i < uec->num_ports; ++i) + typec_mux_put(uec->ports[i].typec_mux); + ucsi_destroy(uec->ucsi); } diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 2450440dae2e..5ec01d672481 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -482,7 +482,7 @@ static int add_direct_chain(struct mlx5_vdpa_dev *mvdev, return 0; err_alloc: - list_for_each_entry_safe(dmr, n, &mr->head, list) { + list_for_each_entry_safe(dmr, n, &tmp, list) { list_del_init(&dmr->list); unmap_direct_mr(mvdev, dmr); kfree(dmr); diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c index df9c7ddc5d78..445c45080d45 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c @@ -231,8 +231,11 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr, kthread_init_work(&vdpasim->work, vdpasim_work_fn); vdpasim->worker = kthread_run_worker(0, "vDPA sim worker: %s", dev_attr->name); - if (IS_ERR(vdpasim->worker)) + if (IS_ERR(vdpasim->worker)) { + ret = PTR_ERR(vdpasim->worker); + vdpasim->worker = NULL; goto err_iommu; + } mutex_init(&vdpasim->mutex); spin_lock_init(&vdpasim->iommu_lock); @@ -744,18 +747,24 @@ static void vdpasim_free(struct vdpa_device *vdpa) struct vdpasim *vdpasim = vdpa_to_sim(vdpa); int i; - kthread_cancel_work_sync(&vdpasim->work); - kthread_destroy_worker(vdpasim->worker); + if (vdpasim->worker) { + kthread_cancel_work_sync(&vdpasim->work); + kthread_destroy_worker(vdpasim->worker); + } - for (i = 0; i < vdpasim->dev_attr.nvqs; i++) { - vringh_kiov_cleanup(&vdpasim->vqs[i].out_iov); - vringh_kiov_cleanup(&vdpasim->vqs[i].in_iov); + if (vdpasim->vqs) { + for (i = 0; i < vdpasim->dev_attr.nvqs; i++) { + vringh_kiov_cleanup(&vdpasim->vqs[i].out_iov); + vringh_kiov_cleanup(&vdpasim->vqs[i].in_iov); + } } vdpasim->dev_attr.free(vdpasim); - for (i = 0; i < vdpasim->dev_attr.nas; i++) - vhost_iotlb_reset(&vdpasim->iommu[i]); + if (vdpasim->iommu) { + for (i = 0; i < vdpasim->dev_attr.nas; i++) + vhost_iotlb_reset(&vdpasim->iommu[i]); + } kfree(vdpasim->iommu); kfree(vdpasim->iommu_pt); kfree(vdpasim->vqs); diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c index 8f02f236b5b4..1050bd7fd7ce 100644 --- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -1257,6 +1257,7 @@ static int vfio_msi_cap_len(struct vfio_pci_core_device *vdev, u8 pos) ret = init_pci_cap_msi_perm(vdev->msi_perm, len, flags); if (ret) { kfree(vdev->msi_perm); + vdev->msi_perm = NULL; return ret; } diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 89d5438ba7af..72bd44f2f142 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -1778,7 +1778,8 @@ static long vhost_net_ioctl(struct file *f, unsigned int ioctl, return -EFAULT; /* Zero the trailing space provided by user-space, if any */ - if (clear_user(argp, size_mul(count - copied, sizeof(u64)))) + if (clear_user(argp + size_mul(copied, sizeof(u64)), + size_mul(count - copied, sizeof(u64)))) return -EFAULT; return 0; case VHOST_SET_FEATURES_ARRAY: diff --git a/drivers/video/fbdev/clps711x-fb.c b/drivers/video/fbdev/clps711x-fb.c index 5e61a349a4ab..c350340fb7b9 100644 --- a/drivers/video/fbdev/clps711x-fb.c +++ b/drivers/video/fbdev/clps711x-fb.c @@ -332,8 +332,6 @@ static int clps711x_fb_probe(struct platform_device *pdev) return 0; - unregister_framebuffer(info); - out_fb_dealloc_cmap: regmap_update_bits(cfb->syscon, SYSCON_OFFSET, SYSCON1_LCDEN, 0); fb_dealloc_cmap(&info->cmap); diff --git a/drivers/video/fbdev/kyro/fbdev.c b/drivers/video/fbdev/kyro/fbdev.c index c8b1dfa456a3..c44d6809859e 100644 --- a/drivers/video/fbdev/kyro/fbdev.c +++ b/drivers/video/fbdev/kyro/fbdev.c @@ -369,6 +369,9 @@ static int kyro_dev_overlay_create(u32 ulWidth, static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight) { + u32 right; + u32 bottom; + if (deviceInfo.ulOverlayOffset == 0) /* probably haven't called CreateOverlay yet */ return -EINVAL; @@ -378,11 +381,30 @@ static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight (x < 2 && ulWidth + 2 == 0)) return -EINVAL; + /* + * SetOverlayViewPort() adjusts X coordinates by +2 (left) and +1 + * (right) before packing them into 16-bit register fields. + */ + if (x > U16_MAX - 2 || y > U16_MAX) + return -EINVAL; + + right = x + ulWidth; + bottom = y + ulHeight; + + if (right < x || bottom < y) + return -EINVAL; + + right--; + bottom--; + + if (right > U16_MAX - 1 || bottom > U16_MAX) + return -EINVAL; + /* Stop Ramdac Output */ DisableRamdacOutput(deviceInfo.pSTGReg); SetOverlayViewPort(deviceInfo.pSTGReg, - x, y, x + ulWidth - 1, y + ulHeight - 1); + x, y, right, bottom); EnableOverlayPlane(deviceInfo.pSTGReg); /* Start Ramdac Output */ diff --git a/drivers/video/fbdev/tdfxfb.c b/drivers/video/fbdev/tdfxfb.c index c5095358209e..eb8f33ae94e7 100644 --- a/drivers/video/fbdev/tdfxfb.c +++ b/drivers/video/fbdev/tdfxfb.c @@ -1383,7 +1383,7 @@ static int tdfxfb_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (err) return err; - err = pci_enable_device(pdev); + err = pcim_enable_device(pdev); if (err) { printk(KERN_ERR "tdfxfb: Can't enable pdev: %d\n", err); return err; diff --git a/drivers/virt/coco/arm-cca-guest/arm-cca-guest.c b/drivers/virt/coco/arm-cca-guest/arm-cca-guest.c index 66d00b6ceb78..a38df08da6fa 100644 --- a/drivers/virt/coco/arm-cca-guest/arm-cca-guest.c +++ b/drivers/virt/coco/arm-cca-guest/arm-cca-guest.c @@ -16,54 +16,38 @@ /** * struct arm_cca_token_info - a descriptor for the token buffer. - * @challenge: Pointer to the challenge data - * @challenge_size: Size of the challenge data * @granule: PA of the granule to which the token will be written * @offset: Offset within granule to start of buffer in bytes - * @result: result of rsi_attestation_token_continue operation */ struct arm_cca_token_info { - void *challenge; - unsigned long challenge_size; phys_addr_t granule; unsigned long offset; - unsigned long result; }; -static void arm_cca_attestation_init(void *param) -{ - struct arm_cca_token_info *info; - - info = (struct arm_cca_token_info *)param; - - info->result = rsi_attestation_token_init(info->challenge, - info->challenge_size); -} - /** * arm_cca_attestation_continue - Retrieve the attestation token data. * - * @param: pointer to the arm_cca_token_info + * @info: pointer to the arm_cca_token_info * * Attestation token generation is a long running operation and therefore * the token data may not be retrieved in a single call. Moreover, the * token retrieval operation must be requested on the same CPU on which the * attestation token generation was initialised. - * This helper function is therefore scheduled on the same CPU multiple + * This helper function must therefore be executed on the same CPU multiple * times until the entire token data is retrieved. */ -static void arm_cca_attestation_continue(void *param) +static unsigned long +arm_cca_attestation_continue(struct arm_cca_token_info *info) { + unsigned long ret; unsigned long len; unsigned long size; - struct arm_cca_token_info *info; - - info = (struct arm_cca_token_info *)param; size = RSI_GRANULE_SIZE - info->offset; - info->result = rsi_attestation_token_continue(info->granule, - info->offset, size, &len); + ret = rsi_attestation_token_continue(info->granule, info->offset, size, + &len); info->offset += len; + return ret; } /** @@ -74,8 +58,8 @@ static void arm_cca_attestation_continue(void *param) * * Initialise the attestation token generation using the challenge data * passed in the TSM descriptor. Allocate memory for the attestation token - * and schedule calls to retrieve the attestation token on the same CPU - * on which the attestation token generation was initialised. + * and retrieve the attestation token on the same CPU on which the + * attestation token generation was initialised. * * The challenge data must be at least 32 bytes and no more than 64 bytes. If * less than 64 bytes are provided it will be zero padded to 64 bytes. @@ -85,12 +69,11 @@ static void arm_cca_attestation_continue(void *param) * * %-EINVAL - A parameter was not valid. * * %-ENOMEM - Out of memory. * * %-EFAULT - Failed to get IPA for memory page(s). - * * A negative status code as returned by smp_call_function_single(). */ static int arm_cca_report_new(struct tsm_report *report, void *data) { - int ret; - int cpu; + int ret = 0; + unsigned long rsi_result; long max_size; unsigned long token_size = 0; struct arm_cca_token_info info; @@ -103,37 +86,33 @@ static int arm_cca_report_new(struct tsm_report *report, void *data) /* * The attestation token 'init' and 'continue' calls must be - * performed on the same CPU. smp_call_function_single() is used - * instead of simply calling get_cpu() because of the need to - * allocate outblob based on the returned value from the 'init' - * call and that cannot be done in an atomic context. + * performed on the same CPU, so disable CPU migration around + * those operations. */ - cpu = smp_processor_id(); + migrate_disable(); - info.challenge = desc->inblob; - info.challenge_size = desc->inblob_len; - - ret = smp_call_function_single(cpu, arm_cca_attestation_init, - &info, true); - if (ret) - return ret; - max_size = info.result; - - if (max_size <= 0) - return -EINVAL; + max_size = rsi_attestation_token_init(desc->inblob, desc->inblob_len); + if (max_size <= 0) { + ret = -EINVAL; + goto exit_migrate_enable; + } /* Allocate outblob */ token = kvzalloc(max_size, GFP_KERNEL); - if (!token) - return -ENOMEM; + if (!token) { + ret = -ENOMEM; + goto exit_migrate_enable; + } /* * Since the outblob may not be physically contiguous, use a page * to bounce the buffer from RMM. */ buf = alloc_pages_exact(RSI_GRANULE_SIZE, GFP_KERNEL); - if (!buf) - return -ENOMEM; + if (!buf) { + ret = -ENOMEM; + goto exit_migrate_enable; + } /* Get the PA of the memory page(s) that were allocated */ info.granule = (unsigned long)virt_to_phys(buf); @@ -144,21 +123,15 @@ static int arm_cca_report_new(struct tsm_report *report, void *data) info.offset = 0; do { /* - * Schedule a call to retrieve a sub-granule chunk - * of data per loop iteration. + * Retrieve a sub-granule chunk of data per loop + * iteration. */ - ret = smp_call_function_single(cpu, - arm_cca_attestation_continue, - (void *)&info, true); - if (ret != 0) { - token_size = 0; - goto exit_free_granule_page; - } - } while (info.result == RSI_INCOMPLETE && + rsi_result = arm_cca_attestation_continue(&info); + } while (rsi_result == RSI_INCOMPLETE && info.offset < RSI_GRANULE_SIZE); /* Break out in case of failure */ - if (info.result != RSI_SUCCESS && info.result != RSI_INCOMPLETE) { + if (rsi_result != RSI_SUCCESS && rsi_result != RSI_INCOMPLETE) { ret = -ENXIO; token_size = 0; goto exit_free_granule_page; @@ -173,12 +146,14 @@ static int arm_cca_report_new(struct tsm_report *report, void *data) break; memcpy(&token[token_size], buf, info.offset); token_size += info.offset; - } while (info.result == RSI_INCOMPLETE); + } while (rsi_result == RSI_INCOMPLETE); report->outblob = no_free_ptr(token); exit_free_granule_page: report->outblob_len = token_size; free_pages_exact(buf, RSI_GRANULE_SIZE); +exit_migrate_enable: + migrate_enable(); return ret; } diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c index 5bdc6b82b30b..3b7bab73d895 100644 --- a/drivers/virtio/virtio.c +++ b/drivers/virtio/virtio.c @@ -401,6 +401,32 @@ static const struct cpumask *virtio_irq_get_affinity(struct device *_d, return dev->config->get_vq_affinity(dev, irq_vec); } +/** + * virtio_device_shutdown - break and reset a device on shutdown + * @dev: the device + * + * Drivers with their own .shutdown method should quiesce their activity and + * then call this to stop the device the way the generic shutdown path does. + */ +void virtio_device_shutdown(struct virtio_device *dev) +{ + /* + * Some devices get wedged if you kick them after they are + * reset. Mark all vqs as broken to make sure we don't. + */ + virtio_break_device(dev); + /* + * Guarantee that any callback will see vq->broken as true. + */ + virtio_synchronize_cbs(dev); + /* + * As IOMMUs are reset on shutdown, this will block device access to memory. + * Some devices get wedged if this happens, so reset to make sure it does not. + */ + dev->config->reset(dev); +} +EXPORT_SYMBOL_GPL(virtio_device_shutdown); + static void virtio_dev_shutdown(struct device *_d) { struct virtio_device *dev = dev_to_virtio(_d); @@ -419,20 +445,7 @@ static void virtio_dev_shutdown(struct device *_d) return; } - /* - * Some devices get wedged if you kick them after they are - * reset. Mark all vqs as broken to make sure we don't. - */ - virtio_break_device(dev); - /* - * Guarantee that any callback will see vq->broken as true. - */ - virtio_synchronize_cbs(dev); - /* - * As IOMMUs are reset on shutdown, this will block device access to memory. - * Some devices get wedged if this happens, so reset to make sure it does not. - */ - dev->config->reset(dev); + virtio_device_shutdown(dev); } static const struct bus_type virtio_bus = { diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c index 1b93d8c64361..a3995e77a652 100644 --- a/drivers/virtio/virtio_balloon.c +++ b/drivers/virtio/virtio_balloon.c @@ -7,6 +7,7 @@ */ #include +#include #include #include #include @@ -1116,31 +1117,50 @@ static void remove_common(struct virtio_balloon *vb) vb->vdev->config->del_vqs(vb->vdev); } -static void virtballoon_remove(struct virtio_device *vdev) +/* + * Stop all asynchronous balloon work. The device must still be alive so that + * in-flight requests can drain via the host before it is reset or freed. + */ +static void virtballoon_quiesce(struct virtio_balloon *vb) { - struct virtio_balloon *vb = vdev->priv; + struct virtio_device *vdev = vb->vdev; - if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_REPORTING)) + if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_REPORTING)) page_reporting_unregister(&vb->pr_dev_info); - if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) + if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) unregister_oom_notifier(&vb->oom_nb); - if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) + if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) virtio_balloon_unregister_shrinker(vb); + spin_lock_irq(&vb->stop_update_lock); vb->stop_update = true; spin_unlock_irq(&vb->stop_update_lock); cancel_work_sync(&vb->update_balloon_size_work); cancel_work_sync(&vb->update_balloon_stats_work); - if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) { + if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) cancel_work_sync(&vb->report_free_page_work); +} + +static void virtballoon_remove(struct virtio_device *vdev) +{ + struct virtio_balloon *vb = vdev->priv; + + virtballoon_quiesce(vb); + + if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) destroy_workqueue(vb->balloon_wq); - } remove_common(vb); kfree(vb); } +static void virtballoon_shutdown(struct virtio_device *vdev) +{ + virtballoon_quiesce(vdev->priv); + virtio_device_shutdown(vdev); +} + #ifdef CONFIG_PM_SLEEP static int virtballoon_freeze(struct virtio_device *vdev) { @@ -1185,6 +1205,11 @@ static int virtballoon_validate(struct virtio_device *vdev) else if (!virtio_has_feature(vdev, VIRTIO_BALLOON_F_PAGE_POISON)) __virtio_clear_bit(vdev, VIRTIO_BALLOON_F_REPORTING); + /* + * Disable indirect descriptors to avoid memory allocation in + * virtqueue_add during page reporting. + */ + __virtio_clear_bit(vdev, VIRTIO_RING_F_INDIRECT_DESC); __virtio_clear_bit(vdev, VIRTIO_F_ACCESS_PLATFORM); return 0; } @@ -1206,6 +1231,7 @@ static struct virtio_driver virtio_balloon_driver = { .validate = virtballoon_validate, .probe = virtballoon_probe, .remove = virtballoon_remove, + .shutdown = virtballoon_shutdown, .config_changed = virtballoon_changed, #ifdef CONFIG_PM_SLEEP .freeze = virtballoon_freeze, diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c index 30c6627a5ce4..ef87e7584756 100644 --- a/drivers/virtio/virtio_pci_common.c +++ b/drivers/virtio/virtio_pci_common.c @@ -502,7 +502,7 @@ static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs, if (!avq_num) return 0; sprintf(avq->name, "avq.%u", avq->vq_index); - vq = vp_setup_vq(vdev, queue_idx++, vp_modern_avq_done, avq->name, + vq = vp_setup_vq(vdev, avq->vq_index, vp_modern_avq_done, avq->name, false, VIRTIO_MSI_NO_VECTOR, &vp_dev->admin_vq.info); if (IS_ERR(vq)) { diff --git a/drivers/virtio/virtio_rtc_driver.c b/drivers/virtio/virtio_rtc_driver.c index 4419735b0f0d..74616ba5be11 100644 --- a/drivers/virtio/virtio_rtc_driver.c +++ b/drivers/virtio/virtio_rtc_driver.c @@ -574,8 +574,8 @@ static int viortc_msg_xfer(struct viortc_vq *vq, struct viortc_msg *msg, * read requests */ -/** timeout for clock readings, where timeouts are considered non-fatal */ -#define VIORTC_MSG_READ_TIMEOUT secs_to_jiffies(60) +/** timeout for runtime requests, where timeouts are considered non-fatal */ +#define VIORTC_MSG_TIMEOUT secs_to_jiffies(60) /** * viortc_read() - VIRTIO_RTC_REQ_READ wrapper @@ -600,7 +600,7 @@ int viortc_read(struct viortc_dev *viortc, u16 vio_clk_id, u64 *reading) VIORTC_MSG_WRITE(hdl, clock_id, &vio_clk_id); ret = viortc_msg_xfer(&viortc->vqs[VIORTC_REQUESTQ], VIORTC_MSG(hdl), - VIORTC_MSG_READ_TIMEOUT); + VIORTC_MSG_TIMEOUT); if (ret) { dev_dbg(&viortc->vdev->dev, "%s: xfer returned %d\n", __func__, ret); @@ -642,7 +642,7 @@ int viortc_read_cross(struct viortc_dev *viortc, u16 vio_clk_id, u8 hw_counter, VIORTC_MSG_WRITE(hdl, hw_counter, &hw_counter); ret = viortc_msg_xfer(&viortc->vqs[VIORTC_REQUESTQ], VIORTC_MSG(hdl), - VIORTC_MSG_READ_TIMEOUT); + VIORTC_MSG_TIMEOUT); if (ret) { dev_dbg(&viortc->vdev->dev, "%s: xfer returned %d\n", __func__, ret); @@ -809,7 +809,7 @@ int viortc_read_alarm(struct viortc_dev *viortc, u16 vio_clk_id, VIORTC_MSG_WRITE(hdl, clock_id, &vio_clk_id); ret = viortc_msg_xfer(&viortc->vqs[VIORTC_REQUESTQ], VIORTC_MSG(hdl), - 0); + VIORTC_MSG_TIMEOUT); if (ret) { dev_dbg(&viortc->vdev->dev, "%s: xfer returned %d\n", __func__, ret); @@ -858,7 +858,7 @@ int viortc_set_alarm(struct viortc_dev *viortc, u16 vio_clk_id, u64 alarm_time, VIORTC_MSG_WRITE(hdl, flags, &flags); ret = viortc_msg_xfer(&viortc->vqs[VIORTC_REQUESTQ], VIORTC_MSG(hdl), - 0); + VIORTC_MSG_TIMEOUT); if (ret) { dev_dbg(&viortc->vdev->dev, "%s: xfer returned %d\n", __func__, ret); @@ -900,7 +900,7 @@ int viortc_set_alarm_enabled(struct viortc_dev *viortc, u16 vio_clk_id, VIORTC_MSG_WRITE(hdl, flags, &flags); ret = viortc_msg_xfer(&viortc->vqs[VIORTC_REQUESTQ], VIORTC_MSG(hdl), - 0); + VIORTC_MSG_TIMEOUT); if (ret) { dev_dbg(&viortc->vdev->dev, "%s: xfer returned %d\n", __func__, ret); diff --git a/drivers/w1/masters/ds2482.c b/drivers/w1/masters/ds2482.c index e2a568c9a43a..0c93bf338a6e 100644 --- a/drivers/w1/masters/ds2482.c +++ b/drivers/w1/masters/ds2482.c @@ -310,6 +310,10 @@ static u8 ds2482_w1_triplet(void *data, u8 dbit) mutex_unlock(&pdev->access_lock); + /* On bus error, decode to 3 (no device responded) to abort the search */ + if (status < 0) + status = 3 << 5; + /* Decode the status */ return (status >> 5); } diff --git a/drivers/xen/xen-acpi-processor.c b/drivers/xen/xen-acpi-processor.c index 520756159d3d..8d1860bd5d57 100644 --- a/drivers/xen/xen-acpi-processor.c +++ b/drivers/xen/xen-acpi-processor.c @@ -492,7 +492,7 @@ static void xen_acpi_processor_resume_worker(struct work_struct *dummy) pr_info("ACPI data upload failed, error = %d\n", rc); } -static void xen_acpi_processor_resume(void) +static void xen_acpi_processor_resume(void *data) { static DECLARE_WORK(wq, xen_acpi_processor_resume_worker); @@ -506,10 +506,14 @@ static void xen_acpi_processor_resume(void) schedule_work(&wq); } -static struct syscore_ops xap_syscore_ops = { +static const struct syscore_ops xap_syscore_ops = { .resume = xen_acpi_processor_resume, }; +static struct syscore xap_syscore = { + .ops = &xap_syscore_ops, +}; + static int __init xen_acpi_processor_init(void) { int i; @@ -560,7 +564,7 @@ static int __init xen_acpi_processor_init(void) if (rc) goto err_unregister; - register_syscore_ops(&xap_syscore_ops); + register_syscore(&xap_syscore); return 0; err_unregister: @@ -577,7 +581,7 @@ static void __exit xen_acpi_processor_exit(void) { int i; - unregister_syscore_ops(&xap_syscore_ops); + unregister_syscore(&xap_syscore); bitmap_free(acpi_ids_done); bitmap_free(acpi_id_present); bitmap_free(acpi_id_cst_present); diff --git a/drivers/xen/xenbus/xenbus_xs.c b/drivers/xen/xenbus/xenbus_xs.c index 528682bf0c7f..f312b9a57d97 100644 --- a/drivers/xen/xenbus/xenbus_xs.c +++ b/drivers/xen/xenbus/xenbus_xs.c @@ -912,19 +912,27 @@ int xs_init(void) int err; struct task_struct *task; - register_reboot_notifier(&xs_reboot_nb); + err = register_reboot_notifier(&xs_reboot_nb); + if (err) + return err; /* Initialize the shared memory rings to talk to xenstored */ err = xb_init_comms(); if (err) - return err; + goto err_unregister_reboot_notifier; task = kthread_run(xenwatch_thread, NULL, "xenwatch"); - if (IS_ERR(task)) - return PTR_ERR(task); + if (IS_ERR(task)) { + err = PTR_ERR(task); + goto err_unregister_reboot_notifier; + } /* shutdown watches for kexec boot */ xs_reset_watches(); return 0; + +err_unregister_reboot_notifier: + unregister_reboot_notifier(&xs_reboot_nb); + return err; } diff --git a/fs/btrfs/bio.c b/fs/btrfs/bio.c index 7d8aaa019c8c..afe022a3b8fc 100644 --- a/fs/btrfs/bio.c +++ b/fs/btrfs/bio.c @@ -172,7 +172,21 @@ static void btrfs_end_repair_bio(struct btrfs_bio *repair_bbio, struct btrfs_inode *inode = repair_bbio->inode; struct btrfs_fs_info *fs_info = inode->root->fs_info; struct bio_vec *bv = bio_first_bvec_all(&repair_bbio->bio); + /* + * We can not move forward the saved_iter, as it will be later + * utilized by repair_bbio again. + */ + struct bvec_iter saved_iter = repair_bbio->saved_iter; + const u32 step = min(fs_info->sectorsize, PAGE_SIZE); + const u64 logical = repair_bbio->saved_iter.bi_sector << SECTOR_SHIFT; + const u32 nr_steps = repair_bbio->saved_iter.bi_size / step; int mirror = repair_bbio->mirror_num; + phys_addr_t paddrs[BTRFS_MAX_BLOCKSIZE / PAGE_SIZE]; + phys_addr_t paddr; + unsigned int slot = 0; + + /* Repair bbio should be eaxctly one block sized. */ + ASSERT(repair_bbio->saved_iter.bi_size == fs_info->sectorsize); if (repair_bbio->bio.bi_status || !btrfs_data_csum_ok(repair_bbio, dev, 0, bvec_phys(bv))) { @@ -190,12 +204,17 @@ static void btrfs_end_repair_bio(struct btrfs_bio *repair_bbio, return; } + btrfs_bio_for_each_block(paddr, &repair_bbio->bio, &saved_iter, step) { + ASSERT(slot < nr_steps); + paddrs[slot] = paddr; + slot++; + } + do { mirror = prev_repair_mirror(fbio, mirror); btrfs_repair_io_failure(fs_info, btrfs_ino(inode), repair_bbio->file_offset, fs_info->sectorsize, - repair_bbio->saved_iter.bi_sector << SECTOR_SHIFT, - bvec_phys(bv), mirror); + logical, paddrs, step, mirror); } while (mirror != fbio->bbio->mirror_num); done: @@ -865,18 +884,36 @@ void btrfs_submit_bbio(struct btrfs_bio *bbio, int mirror_num) * * The I/O is issued synchronously to block the repair read completion from * freeing the bio. + * + * @ino: Offending inode number + * @fileoff: File offset inside the inode + * @length: Length of the repair write + * @logical: Logical address of the range + * @paddrs: Physical address array of the content + * @step: Length of for each paddrs + * @mirror_num: Mirror number to write to. Must not be zero */ -int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start, - u64 length, u64 logical, phys_addr_t paddr, int mirror_num) +int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 fileoff, + u32 length, u64 logical, const phys_addr_t paddrs[], + unsigned int step, int mirror_num) { + const u32 nr_steps = DIV_ROUND_UP_POW2(length, step); struct btrfs_io_stripe smap = { 0 }; - struct bio_vec bvec; - struct bio bio; + struct bio *bio = NULL; int ret = 0; ASSERT(!(fs_info->sb->s_flags & SB_RDONLY)); BUG_ON(!mirror_num); + /* Basic alignment checks. */ + ASSERT(IS_ALIGNED(logical, fs_info->sectorsize)); + ASSERT(IS_ALIGNED(length, fs_info->sectorsize)); + ASSERT(IS_ALIGNED(fileoff, fs_info->sectorsize)); + /* Either it's a single data or metadata block. */ + ASSERT(length <= BTRFS_MAX_BLOCKSIZE); + ASSERT(step <= length); + ASSERT(is_power_of_2(step)); + if (btrfs_repair_one_zone(fs_info, logical)) return 0; @@ -896,24 +933,27 @@ int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start, goto out_counter_dec; } - bio_init(&bio, smap.dev->bdev, &bvec, 1, REQ_OP_WRITE | REQ_SYNC); - bio.bi_iter.bi_sector = smap.physical >> SECTOR_SHIFT; - __bio_add_page(&bio, phys_to_page(paddr), length, offset_in_page(paddr)); - ret = submit_bio_wait(&bio); + bio = bio_alloc(smap.dev->bdev, nr_steps, REQ_OP_WRITE | REQ_SYNC, GFP_NOFS); + bio->bi_iter.bi_sector = smap.physical >> SECTOR_SHIFT; + for (int i = 0; i < nr_steps; i++) { + ret = bio_add_page(bio, phys_to_page(paddrs[i]), step, offset_in_page(paddrs[i])); + /* We should have allocated enough slots to contain all the different pages. */ + ASSERT(ret == step); + } + ret = submit_bio_wait(bio); + bio_put(bio); if (ret) { /* try to remap that extent elsewhere? */ btrfs_dev_stat_inc_and_print(smap.dev, BTRFS_DEV_STAT_WRITE_ERRS); - goto out_bio_uninit; + goto out_counter_dec; } btrfs_info_rl(fs_info, "read error corrected: ino %llu off %llu (dev %s sector %llu)", - ino, start, btrfs_dev_name(smap.dev), + ino, fileoff, btrfs_dev_name(smap.dev), smap.physical >> SECTOR_SHIFT); ret = 0; -out_bio_uninit: - bio_uninit(&bio); out_counter_dec: btrfs_bio_counter_dec(fs_info); return ret; diff --git a/fs/btrfs/bio.h b/fs/btrfs/bio.h index 69fe54f564fc..fd09b6501c96 100644 --- a/fs/btrfs/bio.h +++ b/fs/btrfs/bio.h @@ -127,7 +127,8 @@ void btrfs_bio_end_io(struct btrfs_bio *bbio, blk_status_t status); void btrfs_submit_bbio(struct btrfs_bio *bbio, int mirror_num); void btrfs_submit_repair_write(struct btrfs_bio *bbio, int mirror_num, bool dev_replace); -int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start, - u64 length, u64 logical, phys_addr_t paddr, int mirror_num); +int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 fileoff, + u32 length, u64 logical, const phys_addr_t paddrs[], + unsigned int step, int mirror_num); #endif diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c index 93e0063bb1fa..e9ee10bee2ec 100644 --- a/fs/btrfs/block-group.c +++ b/fs/btrfs/block-group.c @@ -1057,8 +1057,7 @@ static int remove_block_group_item(struct btrfs_trans_handle *trans, if (ret < 0) return ret; - ret = btrfs_del_item(trans, root, path); - return ret; + return btrfs_del_item(trans, root, path); } int btrfs_remove_block_group(struct btrfs_trans_handle *trans, diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c index 0f2e9f33dd0c..c7c2e4405c47 100644 --- a/fs/btrfs/ctree.c +++ b/fs/btrfs/ctree.c @@ -3990,8 +3990,7 @@ int btrfs_split_item(struct btrfs_trans_handle *trans, if (ret) return ret; - ret = split_item(trans, path, new_key, split_offset); - return ret; + return split_item(trans, path, new_key, split_offset); } /* diff --git a/fs/btrfs/defrag.c b/fs/btrfs/defrag.c index 2e3c011d410a..c59548a0b9c5 100644 --- a/fs/btrfs/defrag.c +++ b/fs/btrfs/defrag.c @@ -1150,20 +1150,15 @@ static_assert(PAGE_ALIGNED(CLUSTER_SIZE)); * * - Extent bits are locked */ -static int defrag_one_locked_target(struct btrfs_inode *inode, - struct defrag_target_range *target, - struct folio **folios, int nr_pages, - struct extent_state **cached_state) +static void defrag_one_locked_target(struct btrfs_inode *inode, + struct defrag_target_range *target, + struct folio **folios, int nr_pages, + struct extent_state **cached_state) { struct btrfs_fs_info *fs_info = inode->root->fs_info; - struct extent_changeset *data_reserved = NULL; const u64 start = target->start; const u64 len = target->len; - int ret = 0; - ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len); - if (ret < 0) - return ret; btrfs_clear_extent_bit(&inode->io_tree, start, start + len - 1, EXTENT_DELALLOC | EXTENT_DO_ACCOUNTING | EXTENT_DEFRAG, cached_state); @@ -1184,10 +1179,6 @@ static int defrag_one_locked_target(struct btrfs_inode *inode, btrfs_folio_clamp_clear_checked(fs_info, folio, start, len); btrfs_folio_clamp_set_dirty(fs_info, folio, start, len); } - btrfs_delalloc_release_extents(inode, len); - extent_changeset_free(data_reserved); - - return ret; } static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, @@ -1203,6 +1194,8 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, u64 cur = start; const unsigned int nr_pages = ((start + len - 1) >> PAGE_SHIFT) - (start >> PAGE_SHIFT) + 1; + struct extent_changeset *data_reserved = NULL; + u64 last_defrag_end = start; int ret = 0; ASSERT(nr_pages <= CLUSTER_SIZE / PAGE_SIZE); @@ -1212,6 +1205,22 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, if (!folios) return -ENOMEM; + /* + * Reserve delalloc space before locking the range and before locking + * and dirtying any folios - otherwise we could deadlock, for example + * after defrag of one range we dirty folios and keep them locked when + * we move to the next range, so reserving delalloc space right before + * each range could trigger flushing of delalloc and deadlock on the + * extent lock or trigger a transaction commit with flushoncommit, which + * can either deadlock on the lock of a folio made dirty in the previous + * range or the extent lock. + */ + ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len); + if (ret < 0) { + kfree(folios); + return ret; + } + /* Prepare all pages */ for (int i = 0; cur < start + len && i < nr_pages; i++) { folios[i] = defrag_prepare_one_folio(inode, cur >> PAGE_SHIFT); @@ -1246,10 +1255,11 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, goto unlock_extent; list_for_each_entry(entry, &target_list, list) { - ret = defrag_one_locked_target(inode, entry, folios, nr_pages, - &cached_state); - if (ret < 0) - break; + defrag_one_locked_target(inode, entry, folios, nr_pages, &cached_state); + if (entry->start > last_defrag_end) + btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end, + entry->start - last_defrag_end, true); + last_defrag_end = entry->start + entry->len; } list_for_each_entry_safe(entry, tmp, &target_list, list) { @@ -1266,6 +1276,12 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, folio_put(folios[i]); } kfree(folios); + btrfs_delalloc_release_extents(inode, len); + if (last_defrag_end < start + len) + btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end, + start + len - last_defrag_end, true); + extent_changeset_free(data_reserved); + return ret; } diff --git a/fs/btrfs/delayed-inode.c b/fs/btrfs/delayed-inode.c index ea48706a3d81..e1d1f71153fc 100644 --- a/fs/btrfs/delayed-inode.c +++ b/fs/btrfs/delayed-inode.c @@ -1140,8 +1140,8 @@ __btrfs_commit_inode_delayed_items(struct btrfs_trans_handle *trans, ret = btrfs_record_root_in_trans(trans, node->root); if (ret) return ret; - ret = btrfs_update_delayed_inode(trans, node->root, path, node); - return ret; + + return btrfs_update_delayed_inode(trans, node->root, path, node); } /* diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 655eed981078..15b6a67d17dd 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -183,27 +183,32 @@ static int btrfs_repair_eb_io_failure(const struct extent_buffer *eb, int mirror_num) { struct btrfs_fs_info *fs_info = eb->fs_info; - int ret = 0; + const u32 step = min(fs_info->nodesize, PAGE_SIZE); + const u32 nr_steps = eb->len / step; + phys_addr_t paddrs[BTRFS_MAX_BLOCKSIZE / PAGE_SIZE]; if (sb_rdonly(fs_info->sb)) return -EROFS; - for (int i = 0; i < num_extent_folios(eb); i++) { + for (int i = 0; i < num_extent_pages(eb); i++) { struct folio *folio = eb->folios[i]; - u64 start = max_t(u64, eb->start, folio_pos(folio)); - u64 end = min_t(u64, eb->start + eb->len, - folio_pos(folio) + eb->folio_size); - u32 len = end - start; - phys_addr_t paddr = PFN_PHYS(folio_pfn(folio)) + - offset_in_folio(folio, start); - - ret = btrfs_repair_io_failure(fs_info, 0, start, len, start, - paddr, mirror_num); - if (ret) - break; + + /* No large folio support yet. */ + ASSERT(folio_order(folio) == 0); + ASSERT(i < nr_steps); + + /* + * For nodesize < page size, there is just one paddr, with some + * offset inside the page. + * + * For nodesize >= page size, it's one or more paddrs, and eb->start + * must be aligned to page boundary. + */ + paddrs[i] = page_to_phys(&folio->page) + offset_in_page(eb->start); } - return ret; + return btrfs_repair_io_failure(fs_info, 0, eb->start, eb->len, + eb->start, paddrs, step, mirror_num); } /* @@ -273,14 +278,15 @@ int btree_csum_one_bio(struct btrfs_bio *bbio) return -EIO; /* - * If an extent_buffer is marked as EXTENT_BUFFER_ZONED_ZEROOUT, don't - * checksum it but zero-out its content. This is done to preserve - * ordering of I/O without unnecessarily writing out data. + * An extent_buffer marked EXTENT_BUFFER_ZONED_ZEROOUT is written out as + * zeros to preserve ordering of I/O without persisting the now + * unnecessary block. The bio is fed from the shared zero page (see + * write_one_eb()), so there is nothing to checksum here. Crucially, the + * buffer's own content is left intact: it may still be referenced, e.g. + * btrfs_free_tree_block() reads its header to add a delayed reference. */ - if (test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags)) { - memzero_extent_buffer(eb, 0, eb->len); + if (test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags)) return 0; - } if (WARN_ON_ONCE(found_start != eb->start)) return -EIO; @@ -2183,11 +2189,10 @@ static int load_global_roots(struct btrfs_root *tree_root) return ret; if (!btrfs_fs_compat_ro(tree_root->fs_info, FREE_SPACE_TREE)) return ret; - ret = load_global_roots_objectid(tree_root, path, - BTRFS_FREE_SPACE_TREE_OBJECTID, - "free space"); - return ret; + return load_global_roots_objectid(tree_root, path, + BTRFS_FREE_SPACE_TREE_OBJECTID, + "free space"); } static int btrfs_read_roots(struct btrfs_fs_info *fs_info) diff --git a/fs/btrfs/extent-io-tree.c b/fs/btrfs/extent-io-tree.c index bb2ca1c9c7b0..2a2bce0f1f7c 100644 --- a/fs/btrfs/extent-io-tree.c +++ b/fs/btrfs/extent-io-tree.c @@ -187,18 +187,18 @@ static int add_extent_changeset(struct extent_state *state, u32 bits, struct extent_changeset *changeset, int set) { - int ret; - if (!changeset) return 0; if (set && (state->state & bits) == bits) return 0; if (!set && (state->state & bits) == 0) return 0; + changeset->bytes_changed += state->end - state->start + 1; - ret = ulist_add(&changeset->range_changed, state->start, state->end, - GFP_ATOMIC); - return ret; + if (!extent_changeset_tracks_ranges(changeset)) + return 0; + + return ulist_add(&changeset->range_changed, state->start, state->end, GFP_ATOMIC); } static inline struct extent_state *next_state(struct extent_state *state) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index d2d96f366dac..7c199d59a546 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2557,7 +2557,6 @@ static u64 get_alloc_profile_by_root(struct btrfs_root *root, int data) { struct btrfs_fs_info *fs_info = root->fs_info; u64 flags; - u64 ret; if (data) flags = BTRFS_BLOCK_GROUP_DATA; @@ -2566,8 +2565,7 @@ static u64 get_alloc_profile_by_root(struct btrfs_root *root, int data) else flags = BTRFS_BLOCK_GROUP_METADATA; - ret = btrfs_get_alloc_profile(fs_info, flags); - return ret; + return btrfs_get_alloc_profile(fs_info, flags); } static u64 first_logical_byte(struct btrfs_fs_info *fs_info) @@ -4192,10 +4190,8 @@ static int find_free_extent_update_loop(struct btrfs_fs_info *fs_info, else trans = btrfs_join_transaction(root); - if (IS_ERR(trans)) { - ret = PTR_ERR(trans); - return ret; - } + if (IS_ERR(trans)) + return PTR_ERR(trans); ret = btrfs_chunk_alloc(trans, space_info, ffe_ctl->flags, CHUNK_ALLOC_FORCE_FOR_EXTENT); diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 80fc2d59bfeb..90a6b161d7c1 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -2182,14 +2182,17 @@ static struct extent_buffer *find_extent_buffer_nolock( static void end_bbio_meta_write(struct btrfs_bio *bbio) { struct extent_buffer *eb = bbio->private; - struct folio_iter fi; if (bbio->bio.bi_status != BLK_STS_OK) set_btree_ioerr(eb); - bio_for_each_folio_all(fi, &bbio->bio) { - btrfs_meta_folio_clear_writeback(fi.folio, eb); - } + /* + * Clear writeback on the buffer's own folios. The bio may carry the + * shared zero page instead (EXTENT_BUFFER_ZONED_ZEROOUT), so iterate + * the extent buffer folios rather than the bio folios. + */ + for (int i = 0; i < num_extent_folios(eb); i++) + btrfs_meta_folio_clear_writeback(eb->folios[i], eb); buffer_tree_clear_mark(eb, PAGECACHE_TAG_WRITEBACK); clear_and_wake_up_bit(EXTENT_BUFFER_WRITEBACK, &eb->bflags); @@ -2230,7 +2233,8 @@ static noinline_for_stack void write_one_eb(struct extent_buffer *eb, struct btrfs_fs_info *fs_info = eb->fs_info; struct btrfs_bio *bbio; - prepare_eb_write(eb); + if (!test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags)) + prepare_eb_write(eb); bbio = btrfs_bio_alloc(INLINE_EXTENT_BUFFER_PAGES, REQ_OP_WRITE | REQ_META | wbc_to_write_flags(wbc), @@ -2250,8 +2254,21 @@ static noinline_for_stack void write_one_eb(struct extent_buffer *eb, btrfs_meta_folio_set_writeback(folio, eb); if (!folio_test_dirty(folio)) wbc->nr_to_write -= folio_nr_pages(folio); - bio_add_folio_nofail(&bbio->bio, folio, range_len, - offset_in_folio(folio, range_start)); + if (test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags)) { + u32 off = 0; + + while (off < range_len) { + u32 add = min_t(u32, PAGE_SIZE, range_len - off); + + bio_add_folio_nofail(&bbio->bio, + page_folio(ZERO_PAGE(0)), + add, 0); + off += add; + } + } else { + bio_add_folio_nofail(&bbio->bio, folio, range_len, + offset_in_folio(folio, range_start)); + } wbc_account_cgroup_owner(wbc, folio, range_len); folio_unlock(folio); } diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h index 73571d5d3d5a..65d1ed3fcd5b 100644 --- a/fs/btrfs/extent_io.h +++ b/fs/btrfs/extent_io.h @@ -196,6 +196,25 @@ static inline void extent_changeset_init(struct extent_changeset *changeset) ulist_init(&changeset->range_changed); } +/* + * Sentinel value for range_changed.prealloc indicating that the changeset + * only tracks bytes_changed and does not record individual ranges. This + * avoids GFP_ATOMIC allocations inside add_extent_changeset() when the + * caller doesn't need to iterate the changed ranges afterwards. + */ +#define EXTENT_CHANGESET_BYTES_ONLY ((struct ulist_node *)1) + +static inline void extent_changeset_init_bytes_only(struct extent_changeset *changeset) +{ + changeset->bytes_changed = 0; + changeset->range_changed.prealloc = EXTENT_CHANGESET_BYTES_ONLY; +} + +static inline bool extent_changeset_tracks_ranges(const struct extent_changeset *changeset) +{ + return changeset->range_changed.prealloc != EXTENT_CHANGESET_BYTES_ONLY; +} + static inline struct extent_changeset *extent_changeset_alloc(void) { struct extent_changeset *ret; @@ -210,6 +229,7 @@ static inline struct extent_changeset *extent_changeset_alloc(void) static inline void extent_changeset_prealloc(struct extent_changeset *changeset, gfp_t gfp_mask) { + ASSERT(extent_changeset_tracks_ranges(changeset)); ulist_prealloc(&changeset->range_changed, gfp_mask); } @@ -218,7 +238,8 @@ static inline void extent_changeset_release(struct extent_changeset *changeset) if (!changeset) return; changeset->bytes_changed = 0; - ulist_release(&changeset->range_changed); + if (extent_changeset_tracks_ranges(changeset)) + ulist_release(&changeset->range_changed); } static inline void extent_changeset_free(struct extent_changeset *changeset) diff --git a/fs/btrfs/fiemap.c b/fs/btrfs/fiemap.c index f2eaaef8422b..30a0f66c9c9e 100644 --- a/fs/btrfs/fiemap.c +++ b/fs/btrfs/fiemap.c @@ -661,7 +661,7 @@ static int extent_fiemap(struct btrfs_inode *inode, range_end = round_up(start + len, sectorsize); prev_extent_end = range_start; - btrfs_lock_extent(&inode->io_tree, range_start, range_end, &cached_state); + btrfs_lock_extent(&inode->io_tree, range_start, range_end - 1, &cached_state); ret = fiemap_find_last_extent_offset(inode, path, &last_extent_end); if (ret < 0) @@ -841,7 +841,7 @@ static int extent_fiemap(struct btrfs_inode *inode, } out_unlock: - btrfs_unlock_extent(&inode->io_tree, range_start, range_end, &cached_state); + btrfs_unlock_extent(&inode->io_tree, range_start, range_end - 1, &cached_state); if (ret == BTRFS_FIEMAP_FLUSH_CACHE) { btrfs_release_path(path); diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 66c76f425afb..a15062f976ba 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -881,62 +881,56 @@ static noinline int prepare_one_folio(struct inode *inode, struct folio **folio_ /* * Locks the extent and properly waits for data=ordered extents to finish - * before allowing the folios to be modified if need. + * before allowing the folios to be modified. * * Return: - * 1 - the extent is locked - * 0 - the extent is not locked, and everything is OK + * 0 - the extent is locked * -EAGAIN - need to prepare the folios again */ static noinline int -lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct folio *folio, - loff_t pos, size_t write_bytes, - u64 *lockstart, u64 *lockend, bool nowait, - struct extent_state **cached_state) +lock_and_cleanup_extent(struct btrfs_inode *inode, struct folio *folio, + loff_t pos, size_t write_bytes, + u64 *lockstart, u64 *lockend, bool nowait, + struct extent_state **cached_state) { struct btrfs_fs_info *fs_info = inode->root->fs_info; + struct btrfs_ordered_extent *ordered; u64 start_pos; u64 last_pos; - int ret = 0; start_pos = round_down(pos, fs_info->sectorsize); last_pos = round_up(pos + write_bytes, fs_info->sectorsize) - 1; - if (start_pos < inode->vfs_inode.i_size) { - struct btrfs_ordered_extent *ordered; - - if (nowait) { - if (!btrfs_try_lock_extent(&inode->io_tree, start_pos, - last_pos, cached_state)) { - folio_unlock(folio); - folio_put(folio); - return -EAGAIN; - } - } else { - btrfs_lock_extent(&inode->io_tree, start_pos, last_pos, - cached_state); - } - - ordered = btrfs_lookup_ordered_range(inode, start_pos, - last_pos - start_pos + 1); - if (ordered && - ordered->file_offset + ordered->num_bytes > start_pos && - ordered->file_offset <= last_pos) { - btrfs_unlock_extent(&inode->io_tree, start_pos, last_pos, - cached_state); + if (nowait) { + if (!btrfs_try_lock_extent(&inode->io_tree, start_pos, + last_pos, cached_state)) { folio_unlock(folio); folio_put(folio); - btrfs_start_ordered_extent(ordered); - btrfs_put_ordered_extent(ordered); return -EAGAIN; } - if (ordered) - btrfs_put_ordered_extent(ordered); + } else { + btrfs_lock_extent(&inode->io_tree, start_pos, last_pos, + cached_state); + } - *lockstart = start_pos; - *lockend = last_pos; - ret = 1; + ordered = btrfs_lookup_ordered_range(inode, start_pos, + last_pos - start_pos + 1); + if (ordered && + ordered->file_offset + ordered->num_bytes > start_pos && + ordered->file_offset <= last_pos) { + btrfs_unlock_extent(&inode->io_tree, start_pos, last_pos, + cached_state); + folio_unlock(folio); + folio_put(folio); + btrfs_start_ordered_extent(ordered); + btrfs_put_ordered_extent(ordered); + return -EAGAIN; } + if (ordered) + btrfs_put_ordered_extent(ordered); + + *lockstart = start_pos; + *lockend = last_pos; /* * We should be called after prepare_one_folio() which should have locked @@ -944,7 +938,7 @@ lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct folio *folio, */ WARN_ON(!folio_test_locked(folio)); - return ret; + return 0; } /* @@ -1201,7 +1195,6 @@ static int copy_one_range(struct btrfs_inode *inode, struct iov_iter *iter, const u64 reserved_start = round_down(start, fs_info->sectorsize); u64 reserved_len; struct folio *folio = NULL; - int extents_locked; u64 lockstart; u64 lockend; bool only_release_metadata = false; @@ -1256,18 +1249,15 @@ static int copy_one_range(struct btrfs_inode *inode, struct iov_iter *iter, reserved_len = last_block - reserved_start; } - extents_locked = lock_and_cleanup_extent_if_need(inode, folio, start, - write_bytes, &lockstart, - &lockend, nowait, - &cached_state); - if (extents_locked < 0) { - if (!nowait && extents_locked == -EAGAIN) + ret = lock_and_cleanup_extent(inode, folio, start, write_bytes, + &lockstart, &lockend, nowait, &cached_state); + if (ret < 0) { + if (!nowait) goto again; btrfs_delalloc_release_extents(inode, reserved_len); release_space(inode, *data_reserved, reserved_start, reserved_len, only_release_metadata); - ret = extents_locked; return ret; } @@ -1292,11 +1282,8 @@ static int copy_one_range(struct btrfs_inode *inode, struct iov_iter *iter, /* No copied bytes, unlock, release reserved space and exit. */ if (copied == 0) { - if (extents_locked) - btrfs_unlock_extent(&inode->io_tree, lockstart, lockend, - &cached_state); - else - btrfs_free_extent_state(cached_state); + btrfs_unlock_extent(&inode->io_tree, lockstart, lockend, + &cached_state); btrfs_delalloc_release_extents(inode, reserved_len); release_space(inode, *data_reserved, reserved_start, reserved_len, only_release_metadata); @@ -1315,17 +1302,7 @@ static int copy_one_range(struct btrfs_inode *inode, struct iov_iter *iter, ret = btrfs_dirty_folio(inode, folio, start, copied, &cached_state, only_release_metadata); - /* - * If we have not locked the extent range, because the range's start - * offset is >= i_size, we might still have a non-NULL cached extent - * state, acquired while marking the extent range as delalloc through - * btrfs_dirty_page(). Therefore free any possible cached extent state - * to avoid a memory leak. - */ - if (extents_locked) - btrfs_unlock_extent(&inode->io_tree, lockstart, lockend, &cached_state); - else - btrfs_free_extent_state(cached_state); + btrfs_unlock_extent(&inode->io_tree, lockstart, lockend, &cached_state); btrfs_delalloc_release_extents(inode, reserved_len); if (ret) { diff --git a/fs/btrfs/free-space-tree.c b/fs/btrfs/free-space-tree.c index f9ffb1c8988d..fa8fbb19ff65 100644 --- a/fs/btrfs/free-space-tree.c +++ b/fs/btrfs/free-space-tree.c @@ -1533,9 +1533,7 @@ int btrfs_remove_block_group_free_space(struct btrfs_trans_handle *trans, btrfs_release_path(path); } - ret = 0; - - return ret; + return 0; } static int load_free_space_bitmaps(struct btrfs_caching_control *caching_ctl, diff --git a/fs/btrfs/inode-item.c b/fs/btrfs/inode-item.c index b73e1dd97208..a864f8c99729 100644 --- a/fs/btrfs/inode-item.c +++ b/fs/btrfs/inode-item.c @@ -371,14 +371,13 @@ int btrfs_insert_empty_inode(struct btrfs_trans_handle *trans, struct btrfs_path *path, u64 objectid) { struct btrfs_key key; - int ret; + key.objectid = objectid; key.type = BTRFS_INODE_ITEM_KEY; key.offset = 0; - ret = btrfs_insert_empty_item(trans, root, path, &key, - sizeof(struct btrfs_inode_item)); - return ret; + return btrfs_insert_empty_item(trans, root, path, &key, + sizeof(struct btrfs_inode_item)); } int btrfs_lookup_inode(struct btrfs_trans_handle *trans, struct btrfs_root diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 7af5ea0cf55f..cca9840bee89 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -2348,7 +2348,6 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct folio *locked_fol u64 start, u64 end, struct writeback_control *wbc) { const bool zoned = btrfs_is_zoned(inode->root->fs_info); - int ret; /* * The range must cover part of the @locked_folio, or a return of 1 @@ -2356,10 +2355,8 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct folio *locked_fol */ ASSERT(!(end <= folio_pos(locked_folio) || start >= folio_end(locked_folio))); - if (should_nocow(inode, start, end)) { - ret = run_delalloc_nocow(inode, locked_folio, start, end); - return ret; - } + if (should_nocow(inode, start, end)) + return run_delalloc_nocow(inode, locked_folio, start, end); if (btrfs_inode_can_compress(inode) && inode_need_compress(inode, start, end) && @@ -2367,11 +2364,9 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct folio *locked_fol return 1; if (zoned) - ret = run_delalloc_cow(inode, locked_folio, start, end, wbc, - true); + return run_delalloc_cow(inode, locked_folio, start, end, wbc, true); else - ret = cow_file_range(inode, locked_folio, start, end, NULL, 0); - return ret; + return cow_file_range(inode, locked_folio, start, end, NULL, 0); } void btrfs_split_delalloc_extent(struct btrfs_inode *inode, diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index b4c6cbbcd168..f654639334fc 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -694,7 +694,6 @@ void btrfs_free_qgroup_config(struct btrfs_fs_info *fs_info) static int add_qgroup_relation_item(struct btrfs_trans_handle *trans, u64 src, u64 dst) { - int ret; struct btrfs_root *quota_root = trans->fs_info->quota_root; BTRFS_PATH_AUTO_FREE(path); struct btrfs_key key; @@ -707,8 +706,7 @@ static int add_qgroup_relation_item(struct btrfs_trans_handle *trans, u64 src, key.type = BTRFS_QGROUP_RELATION_KEY; key.offset = dst; - ret = btrfs_insert_empty_item(trans, quota_root, path, &key, 0); - return ret; + return btrfs_insert_empty_item(trans, quota_root, path, &key, 0); } static int del_qgroup_relation_item(struct btrfs_trans_handle *trans, u64 src, @@ -833,9 +831,7 @@ static int del_qgroup_item(struct btrfs_trans_handle *trans, u64 qgroupid) if (ret > 0) return -ENOENT; - ret = btrfs_del_item(trans, quota_root, path); - - return ret; + return btrfs_del_item(trans, quota_root, path); } static int update_qgroup_limit_item(struct btrfs_trans_handle *trans, @@ -2682,10 +2678,8 @@ int btrfs_qgroup_trace_subtree(struct btrfs_trans_handle *trans, return ret; } - if (root_level == 0) { - ret = btrfs_qgroup_trace_leaf_items(trans, root_eb); - return ret; - } + if (root_level == 0) + return btrfs_qgroup_trace_leaf_items(trans, root_eb); path = btrfs_alloc_path(); if (!path) @@ -4350,12 +4344,13 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode, struct ulist_node *unode; struct ulist_iterator uiter; struct extent_changeset changeset; + const u32 sectorsize = root->fs_info->sectorsize; + const u64 aligned_start = round_down(start, sectorsize); + const u64 aligned_len = round_up(start + len, sectorsize) - aligned_start; u64 freed = 0; int ret; - extent_changeset_init(&changeset); - len = round_up(start + len, root->fs_info->sectorsize); - start = round_down(start, root->fs_info->sectorsize); + extent_changeset_init_bytes_only(&changeset); ULIST_ITER_INIT(&uiter); while ((unode = ulist_next(&reserved->range_changed, &uiter))) { @@ -4367,12 +4362,15 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode, extent_changeset_release(&changeset); - /* Only free range in range [start, start + len) */ - if (range_start >= start + len || - range_start + range_len <= start) + /* + * Only free the range within + * [aligned_start, aligned_start + aligned_len). + */ + if (range_start >= aligned_start + aligned_len || + range_start + range_len <= aligned_start) continue; - free_start = max(range_start, start); - free_len = min(start + len, range_start + range_len) - + free_start = max(range_start, aligned_start); + free_len = min(aligned_start + aligned_len, range_start + range_len) - free_start; /* * TODO: To also modify reserved->ranges_reserved to reflect @@ -4418,7 +4416,7 @@ static int __btrfs_qgroup_release_data(struct btrfs_inode *inode, WARN_ON(!free && reserved); if (free && reserved) return qgroup_free_reserved_data(inode, reserved, start, len, released); - extent_changeset_init(&changeset); + extent_changeset_init_bytes_only(&changeset); ret = btrfs_clear_record_extent_bits(&inode->io_tree, start, start + len - 1, EXTENT_QGROUP_RESERVED, &changeset); if (ret < 0) @@ -4673,6 +4671,7 @@ void btrfs_qgroup_check_reserved_leak(struct btrfs_inode *inode) WARN_ON(ret < 0); if (WARN_ON(changeset.bytes_changed)) { + ASSERT(extent_changeset_tracks_ranges(&changeset)); ULIST_ITER_INIT(&iter); while ((unode = ulist_next(&changeset.range_changed, &iter))) { btrfs_warn(inode->root->fs_info, diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c index 37f20f401b0e..b76cd0b1c355 100644 --- a/fs/btrfs/relocation.c +++ b/fs/btrfs/relocation.c @@ -3266,7 +3266,6 @@ static int delete_v1_space_cache(struct extent_buffer *leaf, struct btrfs_key key; bool found = false; int i; - int ret; if (btrfs_header_owner(leaf) != BTRFS_ROOT_TREE_OBJECTID) return 0; @@ -3290,8 +3289,8 @@ static int delete_v1_space_cache(struct extent_buffer *leaf, } if (!found) return -ENOENT; - ret = delete_block_group_cache(block_group, NULL, space_cache_ino); - return ret; + + return delete_block_group_cache(block_group, NULL, space_cache_ino); } /* diff --git a/fs/btrfs/verity.c b/fs/btrfs/verity.c index 46bd8ca58670..7fc30a58d8d9 100644 --- a/fs/btrfs/verity.c +++ b/fs/btrfs/verity.c @@ -731,14 +731,18 @@ static struct page *btrfs_read_merkle_tree_page(struct inode *inode, goto out; folio_lock(folio); - /* If it's not uptodate after we have the lock, we got a read error. */ - if (!folio_test_uptodate(folio)) { + /* Folio was truncated from mapping. */ + if (!folio->mapping) { folio_unlock(folio); folio_put(folio); - return ERR_PTR(-EIO); + goto again; } - folio_unlock(folio); - goto out; + /* Another reader may have filled the folio while we waited. */ + if (folio_test_uptodate(folio)) { + folio_unlock(folio); + goto out; + } + goto read_folio; } folio = filemap_alloc_folio(mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS), @@ -755,6 +759,7 @@ static struct page *btrfs_read_merkle_tree_page(struct inode *inode, return ERR_PTR(ret); } +read_folio: /* * Merkle item keys are indexed from byte 0 in the merkle tree. * They have the form: @@ -764,6 +769,7 @@ static struct page *btrfs_read_merkle_tree_page(struct inode *inode, ret = read_key_bytes(BTRFS_I(inode), BTRFS_VERITY_MERKLE_ITEM_KEY, off, folio_address(folio), PAGE_SIZE, folio); if (ret < 0) { + folio_unlock(folio); folio_put(folio); return ERR_PTR(ret); } diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 863f69b9cf14..51c3eab54bde 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -2456,9 +2456,7 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, free_fs_devices(cur_devices); } - ret = btrfs_commit_transaction(trans); - - return ret; + return btrfs_commit_transaction(trans); error_undo: if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state)) { @@ -7347,7 +7345,6 @@ static int read_one_dev(struct extent_buffer *leaf, struct btrfs_fs_devices *fs_devices = fs_info->fs_devices; struct btrfs_device *device; u64 devid; - int ret; u8 fs_uuid[BTRFS_FSID_SIZE]; u8 dev_uuid[BTRFS_UUID_SIZE]; @@ -7446,8 +7443,8 @@ static int read_one_dev(struct extent_buffer *leaf, atomic64_add(device->total_bytes - device->bytes_used, &fs_info->free_chunk_space); } - ret = 0; - return ret; + + return 0; } int btrfs_read_sys_array(struct btrfs_fs_info *fs_info) diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index e4577a80da58..24abe4dbf774 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -1173,7 +1173,7 @@ static struct dentry *ceph_mkdir(struct mnt_idmap *idmap, struct inode *dir, !req->r_reply_info.head->is_target && !req->r_reply_info.head->is_dentry) err = ceph_handle_notrace_create(dir, dentry); - ret = ERR_PTR(err); + ret = err ? ERR_PTR(err) : NULL; out_req: if (!IS_ERR(ret) && req->r_dentry != dentry) /* Some other dentry was spliced in */ diff --git a/fs/ceph/file.c b/fs/ceph/file.c index f640aff749ed..fa582c8537c5 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -2412,6 +2412,54 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) if (err < 0) goto out; + /* + * For O_APPEND writes we may have waited for Fwx exclusive caps + * while the previous Fwx holder (another client) extended the + * file. i_size has been updated via the cap grant message from + * the MDS, but ki_pos is still the old EOF. Re-read i_size here + * (no extra MDS round-trip needed) and adjust ki_pos to the true + * EOF. Since we hold Fwx, no other client can change the file. + */ + if (iocb->ki_flags & IOCB_APPEND) { + loff_t cur_eof = i_size_read(inode); + + if (cur_eof != pos) { + doutc(cl, + "%p %llx.%llx O_APPEND: pos adjusted %lld -> %lld\n", + inode, ceph_vinop(inode), pos, cur_eof); + iocb->ki_pos = cur_eof; + pos = cur_eof; + if (pos >= limit) { + err = -EFBIG; + goto out_caps; + } + iov_iter_truncate(from, limit - pos); + count = iov_iter_count(from); + + /* + * ceph_get_caps() validated the old endoff + * against i_max_size; adjusting ki_pos forward + * may have shifted the write range beyond the + * granted max_size. Re-check and truncate if + * necessary. + */ + spin_lock(&ci->i_ceph_lock); + if (pos + count > (loff_t)ci->i_max_size) { + loff_t max_size = ci->i_max_size; + + spin_unlock(&ci->i_ceph_lock); + if (pos >= max_size) { + err = -EFBIG; + goto out_caps; + } + iov_iter_truncate(from, max_size - pos); + count = iov_iter_count(from); + } else { + spin_unlock(&ci->i_ceph_lock); + } + } + } + err = file_update_time(file); if (err) goto out_caps; diff --git a/fs/erofs/Kconfig b/fs/erofs/Kconfig index 0be8a38cfb11..473a22167c15 100644 --- a/fs/erofs/Kconfig +++ b/fs/erofs/Kconfig @@ -121,11 +121,13 @@ config EROFS_FS_ZIP_LZMA If unsure, say N. config EROFS_FS_ZIP_LZMA_DEFAULT_MAX_STREAMS - int "EROFS LZMA default maximum decompression streams" + int "EROFS LZMA default maximum decompression streams" if SMP depends on EROFS_FS_ZIP_LZMA range 1 NR_CPUS if SMP range 1 1 if !SMP - default 16 + default 16 if SMP && NR_CPUS >= 16 + default NR_CPUS if SMP + default 1 help By default EROFS allocates one LZMA decompression stream per CPU. Each stream can hold a dictionary of up to 8 MiB taken from the @@ -133,7 +135,7 @@ config EROFS_FS_ZIP_LZMA_DEFAULT_MAX_STREAMS of memory. This caps the default; the lzma_streams module parameter still overrides it. - If unsure, keep the default of 16. + If unsure, keep the suggested default (16 or lower). config EROFS_FS_ZIP_DEFLATE bool "EROFS DEFLATE compressed data support" diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c index e4e59a4e0d90..fbd59109f46c 100644 --- a/fs/erofs/decompressor.c +++ b/fs/erofs/decompressor.c @@ -142,7 +142,6 @@ static void *z_erofs_lz4_handle_overlap(const struct z_erofs_decompress_req *rq, oend = rq->pageofs_out + rq->outputsize; omargin = PAGE_ALIGN(oend) - oend; if (!rq->partial_decoding && may_inplace && - rq->outpages >= rq->inpages && omargin >= LZ4_DECOMPRESS_INPLACE_MARGIN(rq->inputsize)) { for (i = 0; i < rq->inpages; ++i) if (rq->out[rq->outpages - rq->inpages + i] != @@ -305,6 +304,7 @@ static int z_erofs_transform_plain(struct z_erofs_decompress_req *rq, return -EOPNOTSUPP; if (rq->alg == Z_EROFS_COMPRESSION_INTERLACED) { cur = bs - (rq->pageofs_out & (bs - 1)); + DBG_BUGON(rq->pageofs_in & (bs - 1)); pi = (rq->pageofs_in + rq->inputsize - cur) & ~PAGE_MASK; cur = min(cur, rq->outputsize); if (cur && rq->out[0]) { diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 87edbb4366d1..e1f0174d468d 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -286,7 +286,7 @@ struct erofs_inode { #ifdef CONFIG_EROFS_FS_ZIP struct { unsigned short z_advise; - unsigned char z_algorithmtype[2]; + unsigned char z_algofmt[2]; unsigned char z_lclusterbits; union { u64 z_tailextent_headlcn; @@ -338,20 +338,19 @@ static inline struct folio *erofs_grab_folio_nowait(struct address_space *as, readahead_gfp_mask(as) & ~__GFP_RECLAIM); } -/* Has a disk mapping */ -#define EROFS_MAP_MAPPED 0x0001 +/* Allocated on disk at @m_pa (e.g. NOT a fragment extent) */ +#define EROFS_MAP_MAPPED 0x0001 /* Located in metadata (could be copied from bd_inode) */ -#define EROFS_MAP_META 0x0002 -/* The extent is encoded */ -#define EROFS_MAP_ENCODED 0x0004 -/* The length of extent is full */ -#define EROFS_MAP_FULL_MAPPED 0x0008 +#define EROFS_MAP_META 0x0002 +/* @m_llen may be truncated by the runtime compared to the on-disk record */ +#define EROFS_MAP_PARTIAL_MAPPED 0x0004 +/* The on-disk @m_llen may cover only part of the encoded data */ +#define EROFS_MAP_PARTIAL_REF 0x0008 /* Located in the special packed inode */ -#define __EROFS_MAP_FRAGMENT 0x0010 -/* The extent refers to partial decompressed data */ -#define EROFS_MAP_PARTIAL_REF 0x0020 - -#define EROFS_MAP_FRAGMENT (EROFS_MAP_MAPPED | __EROFS_MAP_FRAGMENT) +#define EROFS_MAP_FRAGMENT 0x0010 +/* The encoded on-disk data will be fully handled (decompressed) */ +#define EROFS_MAP_FULL(f) (!((f) & (EROFS_MAP_PARTIAL_MAPPED | \ + EROFS_MAP_PARTIAL_REF))) struct erofs_map_blocks { struct erofs_buf buf; diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index a02ce7c06f9e..e07c0253893e 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -128,7 +128,17 @@ struct z_erofs_pcluster_slab { #define _PCLP(n) { .maxpages = n } static struct z_erofs_pcluster_slab pcluster_pool[] __read_mostly = { - _PCLP(1), _PCLP(4), _PCLP(16), _PCLP(64), _PCLP(128), + _PCLP(1), + _PCLP(4), +#if Z_EROFS_PCLUSTER_MAX_PAGES > 16 + _PCLP(16), +#endif +#if Z_EROFS_PCLUSTER_MAX_PAGES > 64 + _PCLP(64), +#endif +#if Z_EROFS_PCLUSTER_MAX_PAGES > 128 + _PCLP(128), +#endif _PCLP(Z_EROFS_PCLUSTER_MAX_PAGES + 1) }; @@ -520,7 +530,7 @@ static bool z_erofs_should_alloc_cache(struct z_erofs_frontend *fe) if (cachestrategy <= EROFS_ZIP_CACHE_DISABLED) return false; - if (!(fe->map.m_flags & EROFS_MAP_FULL_MAPPED)) + if (fe->map.m_flags & EROFS_MAP_PARTIAL_MAPPED) return true; if (cachestrategy >= EROFS_ZIP_CACHE_READAROUND && @@ -1034,10 +1044,7 @@ static int z_erofs_scan_folio(struct z_erofs_frontend *f, /* bump split parts first to avoid several separate cases */ ++split; - if (!(map->m_flags & EROFS_MAP_MAPPED)) { - folio_zero_segment(folio, cur, end); - tight = false; - } else if (map->m_flags & __EROFS_MAP_FRAGMENT) { + if (map->m_flags & EROFS_MAP_FRAGMENT) { erofs_off_t fpos = offset + cur - map->m_la; err = z_erofs_read_fragment(inode->i_sb, folio, cur, @@ -1046,6 +1053,9 @@ static int z_erofs_scan_folio(struct z_erofs_frontend *f, if (err) break; tight = false; + } else if (!(map->m_flags & EROFS_MAP_MAPPED)) { + folio_zero_segment(folio, cur, end); + tight = false; } else { if (!f->pcl) { err = z_erofs_pcluster_begin(f); @@ -1081,14 +1091,13 @@ static int z_erofs_scan_folio(struct z_erofs_frontend *f, f->pcl->length = offset + end - map->m_la; f->pcl->pageofs_out = map->m_la & ~PAGE_MASK; } - if ((map->m_flags & EROFS_MAP_FULL_MAPPED) && - !(map->m_flags & EROFS_MAP_PARTIAL_REF) && + if (EROFS_MAP_FULL(map->m_flags) && f->pcl->length == map->m_llen) f->pcl->partial = false; } /* shorten the remaining extent to update progress */ map->m_llen = offset + cur - map->m_la; - map->m_flags &= ~EROFS_MAP_FULL_MAPPED; + map->m_flags |= EROFS_MAP_PARTIAL_MAPPED; if (cur <= pgs) { split = cur < pgs; tight = (bs == PAGE_SIZE); @@ -1829,7 +1838,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_frontend *f, map->m_la = end; err = z_erofs_map_blocks_iter(inode, map, EROFS_GET_BLOCKS_READMORE); - if (err || !(map->m_flags & EROFS_MAP_ENCODED)) + if (err || !(map->m_flags & EROFS_MAP_MAPPED)) return; /* expand ra for the trailing edge if readahead */ @@ -1841,7 +1850,7 @@ static void z_erofs_pcluster_readmore(struct z_erofs_frontend *f, end = round_up(end, PAGE_SIZE); } else { end = round_up(map->m_la, PAGE_SIZE); - if (!(map->m_flags & EROFS_MAP_ENCODED) || !map->m_llen) + if (!(map->m_flags & EROFS_MAP_MAPPED) || !map->m_llen) return; } diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c index abf7ddc64c63..b5d888883213 100644 --- a/fs/erofs/zmap.c +++ b/fs/erofs/zmap.c @@ -418,7 +418,7 @@ static int z_erofs_map_blocks_fo(struct inode *inode, if ((flags & EROFS_GET_BLOCKS_FINDTAIL) && ztailpacking) vi->z_fragmentoff = m.nextpackoff; - map->m_flags = EROFS_MAP_MAPPED | EROFS_MAP_ENCODED; + map->m_flags = EROFS_MAP_MAPPED | EROFS_MAP_PARTIAL_MAPPED; end = (m.lcn + 1ULL) << lclusterbits; if (m.type != Z_EROFS_LCLUSTER_TYPE_NONHEAD && endoff >= m.clusterofs) { @@ -434,7 +434,7 @@ static int z_erofs_map_blocks_fo(struct inode *inode, } else { if (m.type != Z_EROFS_LCLUSTER_TYPE_NONHEAD) { end = (m.lcn << lclusterbits) | m.clusterofs; - map->m_flags |= EROFS_MAP_FULL_MAPPED; + map->m_flags &= ~EROFS_MAP_PARTIAL_MAPPED; m.delta[0] = 1; } /* get the corresponding first chunk */ @@ -472,19 +472,14 @@ static int z_erofs_map_blocks_fo(struct inode *inode, } if (m.headtype == Z_EROFS_LCLUSTER_TYPE_PLAIN) { - if (map->m_llen > map->m_plen) { - DBG_BUGON(1); - err = -EFSCORRUPTED; - goto unmap_out; - } - if (vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER) + if ((vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER) && + !(map->m_flags & EROFS_MAP_META)) map->m_algorithmformat = Z_EROFS_COMPRESSION_INTERLACED; else map->m_algorithmformat = Z_EROFS_COMPRESSION_SHIFTED; - } else if (m.headtype == Z_EROFS_LCLUSTER_TYPE_HEAD2) { - map->m_algorithmformat = vi->z_algorithmtype[1]; } else { - map->m_algorithmformat = vi->z_algorithmtype[0]; + map->m_algorithmformat = + vi->z_algofmt[m.headtype == Z_EROFS_LCLUSTER_TYPE_HEAD2]; } if ((flags & EROFS_GET_BLOCKS_FIEMAP) || @@ -495,7 +490,7 @@ static int z_erofs_map_blocks_fo(struct inode *inode, map->m_llen >= i_blocksize(inode))) { err = z_erofs_get_extent_decompressedlen(&m); if (!err) - map->m_flags |= EROFS_MAP_FULL_MAPPED; + map->m_flags &= ~EROFS_MAP_PARTIAL_MAPPED; } unmap_out: @@ -593,15 +588,19 @@ static int z_erofs_map_blocks_ext(struct inode *inode, if (recsz > offsetof(struct z_erofs_extent, pstart_lo)) vi->z_fragmentoff |= map->m_pa << 32; } else if (map->m_plen & Z_EROFS_EXTENT_PLEN_MASK) { - map->m_flags |= EROFS_MAP_MAPPED | - EROFS_MAP_FULL_MAPPED | EROFS_MAP_ENCODED; + map->m_flags |= EROFS_MAP_MAPPED; fmt = map->m_plen >> Z_EROFS_EXTENT_PLEN_FMT_BIT; if (map->m_plen & Z_EROFS_EXTENT_PLEN_PARTIAL) map->m_flags |= EROFS_MAP_PARTIAL_REF; map->m_plen &= Z_EROFS_EXTENT_PLEN_MASK; - if (fmt) - map->m_algorithmformat = fmt - 1; - else if (interlaced && !((map->m_pa | map->m_plen) & bmask)) + if (fmt) { + map->m_algorithmformat = --fmt; + if (fmt >= Z_EROFS_COMPRESSION_MAX) { + erofs_err(sb, "unknown algorithm %d @ pos %llu for nid %llu, please upgrade kernel", + fmt, map->m_la, vi->nid); + return -EOPNOTSUPP; + } + } else if (interlaced && !((map->m_pa | map->m_plen) & bmask)) map->m_algorithmformat = Z_EROFS_COMPRESSION_INTERLACED; else @@ -619,7 +618,7 @@ static int z_erofs_fill_inode(struct inode *inode, struct erofs_map_blocks *map) struct super_block *const sb = inode->i_sb; struct z_erofs_map_header *h; erofs_off_t pos; - int err = 0; + int err = 0, nr; if (test_bit(EROFS_I_Z_INITED_BIT, &vi->flags)) { /* @@ -662,12 +661,19 @@ static int z_erofs_fill_inode(struct inode *inode, struct erofs_map_blocks *map) goto done; } - vi->z_algorithmtype[0] = h->h_algorithmtype & 15; - vi->z_algorithmtype[1] = h->h_algorithmtype >> 4; if (vi->z_advise & Z_EROFS_ADVISE_FRAGMENT_PCLUSTER) vi->z_fragmentoff = le32_to_cpu(h->h_fragmentoff); else if (vi->z_advise & Z_EROFS_ADVISE_INLINE_PCLUSTER) vi->z_idata_size = le16_to_cpu(h->h_idata_size); + for (nr = 0; nr < 2; ++nr) { + vi->z_algofmt[nr] = (h->h_algorithmtype >> (4 * nr)) & 15; + if (vi->z_algofmt[nr] >= Z_EROFS_COMPRESSION_MAX) { + erofs_err(sb, "unknown HEAD%u format %u for nid %llu, please upgrade kernel", + nr + 1, vi->z_algofmt[nr], vi->nid); + err = -EOPNOTSUPP; + goto out_unlock; + } + } if (!erofs_sb_has_big_pcluster(EROFS_SB(sb)) && vi->z_advise & (Z_EROFS_ADVISE_BIG_PCLUSTER_1 | @@ -713,17 +719,25 @@ static int z_erofs_map_sanity_check(struct inode *inode, struct erofs_sb_info *sbi = EROFS_I_SB(inode); u64 pend; - if (!(map->m_flags & EROFS_MAP_ENCODED)) + if (!(map->m_flags & EROFS_MAP_MAPPED)) return 0; - if (unlikely(map->m_algorithmformat >= Z_EROFS_COMPRESSION_RUNTIME_MAX)) { - erofs_err(inode->i_sb, "unknown algorithm %d @ pos %llu for nid %llu, please upgrade kernel", - map->m_algorithmformat, map->m_la, EROFS_I(inode)->nid); - return -EOPNOTSUPP; - } - if (unlikely(map->m_algorithmformat < Z_EROFS_COMPRESSION_MAX && - !(sbi->available_compr_algs & (1 << map->m_algorithmformat)))) { - erofs_err(inode->i_sb, "inconsistent algorithmtype %u for nid %llu", - map->m_algorithmformat, EROFS_I(inode)->nid); + + DBG_BUGON(map->m_algorithmformat >= Z_EROFS_COMPRESSION_RUNTIME_MAX); + if (map->m_algorithmformat < Z_EROFS_COMPRESSION_MAX) { + if (!(sbi->available_compr_algs & BIT(map->m_algorithmformat))) { + erofs_err(inode->i_sb, "inconsistent algorithmtype %u for nid %llu", + map->m_algorithmformat, EROFS_I(inode)->nid); + return -EFSCORRUPTED; + } + if (EROFS_MAP_FULL(map->m_flags) && map->m_llen < map->m_plen && + map->m_la + map->m_llen < inode->i_size) { + erofs_err(inode->i_sb, "too much compressed data @ la %llu of nid %llu", + map->m_la, EROFS_I(inode)->nid); + return -EFSCORRUPTED; + } + } else if (map->m_llen > map->m_plen) { + erofs_err(inode->i_sb, "not enough plain data on disk @ la %llu of nid %llu", + map->m_la, EROFS_I(inode)->nid); return -EFSCORRUPTED; } if (unlikely(map->m_plen > Z_EROFS_PCLUSTER_MAX_SIZE || @@ -780,10 +794,12 @@ static int z_erofs_iomap_begin_report(struct inode *inode, loff_t offset, iomap->bdev = inode->i_sb->s_bdev; iomap->offset = map.m_la; iomap->length = map.m_llen; - if (map.m_flags & EROFS_MAP_MAPPED) { + if (map.m_flags & EROFS_MAP_FRAGMENT) { + iomap->type = IOMAP_MAPPED; + iomap->addr = IOMAP_NULL_ADDR; + } else if (map.m_flags & EROFS_MAP_MAPPED) { iomap->type = IOMAP_MAPPED; - iomap->addr = map.m_flags & __EROFS_MAP_FRAGMENT ? - IOMAP_NULL_ADDR : map.m_pa; + iomap->addr = map.m_pa; } else { iomap->type = IOMAP_HOLE; iomap->addr = IOMAP_NULL_ADDR; diff --git a/fs/ext4/dir.c b/fs/ext4/dir.c index d4164c507a90..290a9aeac15f 100644 --- a/fs/ext4/dir.c +++ b/fs/ext4/dir.c @@ -137,6 +137,7 @@ static int ext4_readdir(struct file *file, struct dir_context *ctx) struct buffer_head *bh = NULL; struct fscrypt_str fstr = FSTR_INIT(NULL, 0); struct dir_private_info *info = file->private_data; + bool has_csum = ext4_has_feature_metadata_csum(sb); err = fscrypt_prepare_readdir(inode); if (err) @@ -148,7 +149,7 @@ static int ext4_readdir(struct file *file, struct dir_context *ctx) return err; /* Can we just clear INDEX flag to ignore htree information? */ - if (!ext4_has_feature_metadata_csum(sb)) { + if (!has_csum) { /* * We don't set the inode dirty flag since it's not * critical that it gets flushed back to the disk. @@ -234,7 +235,10 @@ static int ext4_readdir(struct file *file, struct dir_context *ctx) * dirent right now. Scan from the start of the block * to make sure. */ if (!inode_eq_iversion(inode, info->cookie)) { - for (i = 0; i < sb->s_blocksize && i < offset; ) { + for (i = 0; + i <= sb->s_blocksize - + ext4_dir_rec_len(1, has_csum ? NULL : inode) && + i < offset;) { de = (struct ext4_dir_entry_2 *) (bh->b_data + i); /* It's too expensive to do a full @@ -256,6 +260,17 @@ static int ext4_readdir(struct file *file, struct dir_context *ctx) info->cookie = inode_query_iversion(inode); } + if (unlikely(offset < sb->s_blocksize && + offset > sb->s_blocksize - + ext4_dir_rec_len(1, has_csum ? NULL : inode))) { + EXT4_ERROR_FILE(file, bh->b_blocknr, + "bad entry in directory: %s - offset=%u, size=%lu", + "directory entry too close to block end", + offset, sb->s_blocksize); + ctx->pos = round_up(ctx->pos, sb->s_blocksize); + goto next_block; + } + while (ctx->pos < inode->i_size && offset < sb->s_blocksize) { de = (struct ext4_dir_entry_2 *) (bh->b_data + offset); @@ -311,6 +326,7 @@ static int ext4_readdir(struct file *file, struct dir_context *ctx) ctx->pos += ext4_rec_len_from_disk(de->rec_len, sb->s_blocksize); } +next_block: if ((ctx->pos < inode->i_size) && !dir_relax_shared(inode)) goto done; brelse(bh); diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 54be698b9a1c..996ffdc2dd13 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -3074,6 +3074,7 @@ int do_journal_get_write_access(handle_t *handle, struct inode *inode, void ext4_set_inode_mapping_order(struct inode *inode); #define FALL_BACK_TO_NONDELALLOC 1 #define CONVERT_INLINE_DATA 2 +#define EXT4_WRITE_DATA_INLINE 4 typedef enum { EXT4_IGET_NORMAL = 0, diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 7a8b30932189..149a7fe5a096 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -292,6 +292,13 @@ static ssize_t ext4_buffered_write_iter(struct kiocb *iocb, return -EOPNOTSUPP; inode_lock(inode); + + /* + * Prevent concurrent direct I/O and buffered I/O to the same file + * range. Wait for in-flight DIO to finish before dirtying pages. + */ + inode_dio_wait(inode); + ret = ext4_write_checks(iocb, from); if (ret <= 0) goto out; diff --git a/fs/ext4/inline.c b/fs/ext4/inline.c index 408677fa8196..672085e4ec24 100644 --- a/fs/ext4/inline.c +++ b/fs/ext4/inline.c @@ -812,7 +812,19 @@ int ext4_write_inline_data_end(struct inode *inode, loff_t pos, unsigned len, goto out; } ext4_write_lock_xattr(inode, &no_expand); - BUG_ON(!ext4_has_inline_data(inode)); + /* + * We could have raced with ext4_page_mkwrite() converting + * the inode and clearing the inline data flag, so we just + * release resources and retry the whole write. + */ + if (unlikely(!ext4_has_inline_data(inode))) { + ext4_write_unlock_xattr(inode, &no_expand); + brelse(iloc.bh); + folio_unlock(folio); + folio_put(folio); + ext4_journal_stop(handle); + return 0; + } /* * ei->i_inline_off may have changed since @@ -1454,6 +1466,8 @@ int ext4_read_inline_dir(struct file *file, /* for other entry, the real offset in * the buf has to be tuned accordingly. */ + if (i + ext4_dir_rec_len(1, NULL) > extra_size) + break; de = (struct ext4_dir_entry_2 *) (dir_buf + i - extra_offset); /* It's too expensive to do a full @@ -1488,10 +1502,17 @@ int ext4_read_inline_dir(struct file *file, continue; } + /* + * de lives at dir_buf + ctx->pos - extra_offset, within the + * kmalloc(inline_size) buffer. Make sure its header fits before + * ext4_check_dir_entry() dereferences de->rec_len. + */ + if (ctx->pos + ext4_dir_rec_len(1, NULL) > extra_size) + goto out; de = (struct ext4_dir_entry_2 *) (dir_buf + ctx->pos - extra_offset); if (ext4_check_dir_entry(inode, file, de, iloc.bh, dir_buf, - extra_size, ctx->pos)) + inline_size, ctx->pos)) goto out; if (le32_to_cpu(de->inode)) { if (!dir_emit(ctx, de->name, de->name_len, diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 2fd18dd19eef..3b2891f18540 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -1300,6 +1300,8 @@ static int ext4_write_begin(const struct kiocb *iocb, if (unlikely(ret)) return ret; + *fsdata = (void *)((unsigned long)*fsdata & ~EXT4_WRITE_DATA_INLINE); + trace_ext4_write_begin(inode, pos, len); /* * Reserve one block more for addition to orphan list in case @@ -1314,8 +1316,10 @@ static int ext4_write_begin(const struct kiocb *iocb, foliop); if (ret < 0) return ret; - if (ret == 1) + if (ret == 1) { + *fsdata = (void *)((unsigned long)*fsdata | EXT4_WRITE_DATA_INLINE); return 0; + } } /* @@ -1451,8 +1455,7 @@ static int ext4_write_end(const struct kiocb *iocb, trace_ext4_write_end(inode, pos, len, copied); - if (ext4_has_inline_data(inode) && - ext4_test_inode_state(inode, EXT4_STATE_MAY_INLINE_DATA)) + if ((unsigned long)fsdata & EXT4_WRITE_DATA_INLINE) return ext4_write_inline_data_end(inode, pos, len, copied, folio); @@ -1562,8 +1565,7 @@ static int ext4_journalled_write_end(const struct kiocb *iocb, BUG_ON(!ext4_handle_valid(handle)); - if (ext4_has_inline_data(inode) && - ext4_test_inode_state(inode, EXT4_STATE_MAY_INLINE_DATA)) + if ((unsigned long)fsdata & EXT4_WRITE_DATA_INLINE) return ext4_write_inline_data_end(inode, pos, len, copied, folio); @@ -2687,13 +2689,25 @@ static int mpage_prepare_extent_to_map(struct mpage_da_data *mpd) * page is already under writeback and we are not doing * a data integrity writeback, skip the page */ - if (!folio_test_dirty(folio) || - (folio_test_writeback(folio) && - (mpd->wbc->sync_mode == WB_SYNC_NONE)) || + if ((folio_test_writeback(folio) && + mpd->wbc->sync_mode == WB_SYNC_NONE) || unlikely(folio->mapping != mapping)) { folio_unlock(folio); continue; } + /* + * If the folio is clean, skip writing it back. + * Cycle the folio through the writeback state + * though, to clear stale xarray tags. + */ + if (!folio_test_dirty(folio)) { + if (!folio_test_writeback(folio)) { + __folio_start_writeback(folio, false); + folio_end_writeback(folio); + } + folio_unlock(folio); + continue; + } folio_wait_writeback(folio); BUG_ON(folio_test_writeback(folio)); @@ -3149,8 +3163,10 @@ static int ext4_da_write_begin(const struct kiocb *iocb, foliop, fsdata, true); if (ret < 0) return ret; - if (ret == 1) + if (ret == 1) { + *fsdata = (void *)((unsigned long)*fsdata | EXT4_WRITE_DATA_INLINE); return 0; + } } retry: @@ -3283,17 +3299,15 @@ static int ext4_da_write_end(const struct kiocb *iocb, struct folio *folio, void *fsdata) { struct inode *inode = mapping->host; - int write_mode = (int)(unsigned long)fsdata; + unsigned long write_mode = (unsigned long)fsdata; - if (write_mode == FALL_BACK_TO_NONDELALLOC) + if (write_mode & FALL_BACK_TO_NONDELALLOC) return ext4_write_end(iocb, mapping, pos, len, copied, folio, fsdata); trace_ext4_da_write_end(inode, pos, len, copied); - if (write_mode != CONVERT_INLINE_DATA && - ext4_test_inode_state(inode, EXT4_STATE_MAY_INLINE_DATA) && - ext4_has_inline_data(inode)) + if (write_mode & EXT4_WRITE_DATA_INLINE) return ext4_write_inline_data_end(inode, pos, len, copied, folio); @@ -6434,6 +6448,16 @@ static int ext4_try_to_expand_extra_isize(struct inode *inode, if (ext4_test_inode_state(inode, EXT4_STATE_NO_EXPAND)) return -EOVERFLOW; + /* + * Skip expansion during mount (!SB_ACTIVE). Expanding extra isize + * may move xattrs to external blocks and release ea_inodes via iput. + * When !SB_ACTIVE, iput triggers write_inode_now() which acquires + * s_writepages_rwsem, causing a deadlock with the caller's active + * jbd2 handle (lock order: s_writepages_rwsem -> jbd2_handle). + */ + if (unlikely(!(inode->i_sb->s_flags & SB_ACTIVE))) + return -EBUSY; + /* * In nojournal mode, we can immediately attempt to expand * the inode. When journaled, we first need to obtain extra diff --git a/fs/ext4/migrate.c b/fs/ext4/migrate.c index 1b0dfd963d3f..84e3b703ab46 100644 --- a/fs/ext4/migrate.c +++ b/fs/ext4/migrate.c @@ -458,6 +458,7 @@ int ext4_ext_migrate(struct inode *inode) if (IS_ERR(tmp_inode)) { retval = PTR_ERR(tmp_inode); ext4_journal_stop(handle); + tmp_inode = NULL; goto out_unlock; } /* @@ -585,9 +586,9 @@ int ext4_ext_migrate(struct inode *inode) ext4_journal_stop(handle); out_tmp_inode: unlock_new_inode(tmp_inode); - iput(tmp_inode); out_unlock: ext4_writepages_up_write(inode->i_sb, alloc_ctx); + iput(tmp_inode); return retval; } diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c index 2e35453a56f2..2c661155158f 100644 --- a/fs/ext4/namei.c +++ b/fs/ext4/namei.c @@ -1467,6 +1467,8 @@ int ext4_search_dir(struct buffer_head *bh, char *search_buf, int buf_size, /* this code is executed quadratically often */ /* do minimal checking `by hand' */ if (de->name + de->name_len <= dlimit && + (!ext4_hash_in_dirent(dir) || + (char *)de + ext4_dir_rec_len(de->name_len, dir) <= dlimit) && ext4_match(dir, fname, de)) { /* found a match - just to be sure, do * a full check */ diff --git a/fs/ext4/orphan.c b/fs/ext4/orphan.c index fb57bba0d19d..47289d17152b 100644 --- a/fs/ext4/orphan.c +++ b/fs/ext4/orphan.c @@ -388,7 +388,7 @@ void ext4_orphan_cleanup(struct super_block *sb, struct ext4_super_block *es) struct ext4_orphan_info *oi = &EXT4_SB(sb)->s_orphan_info; int inodes_per_ob = ext4_inodes_per_orphan_block(sb); - if (!es->s_last_orphan && !oi->of_blocks) { + if (!es->s_last_orphan && ext4_orphan_file_empty(sb)) { ext4_debug("no orphan inodes to clean up\n"); return; } @@ -572,6 +572,7 @@ int ext4_init_orphan_info(struct super_block *sb) int i, j; int ret; int free; + int loaded = 0; __le32 *bdata; int inodes_per_ob = ext4_inodes_per_orphan_block(sb); struct ext4_orphan_block_tail *ot; @@ -615,6 +616,7 @@ int ext4_init_orphan_info(struct super_block *sb) ret = -EIO; goto out_free; } + loaded++; ot = ext4_orphan_block_tail(sb, oi->of_binfo[i].ob_bh); if (le32_to_cpu(ot->ob_magic) != EXT4_ORPHAN_BLOCK_MAGIC) { ext4_error(sb, "orphan file block %d: bad magic", i); @@ -637,8 +639,10 @@ int ext4_init_orphan_info(struct super_block *sb) iput(inode); return 0; out_free: - for (i--; i >= 0; i--) - brelse(oi->of_binfo[i].ob_bh); + while (loaded > 0) { + loaded--; + brelse(oi->of_binfo[loaded].ob_bh); + } kvfree(oi->of_binfo); out_put: iput(inode); diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index f1d7a22293b3..bb3f41f088bd 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -4019,10 +4019,26 @@ static inline struct inode *fio_inode(struct f2fs_io_info *fio) #define MIN_FRAGMENT_SIZE 1 #define MAX_FRAGMENT_SIZE 512 -static inline bool f2fs_need_rand_seg(struct f2fs_sb_info *sbi) +static inline bool f2fs_need_rand_blk(struct f2fs_sb_info *sbi, + enum log_type type) { - return F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG || - F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK; + if (type == CURSEG_COLD_DATA_PINNED) + return false; + return F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK; +} + +static inline bool f2fs_need_rand_seg(struct f2fs_sb_info *sbi, + enum log_type type) +{ + if (type == CURSEG_COLD_DATA_PINNED) + return false; + return F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG; +} + +static inline bool f2fs_need_rand_seg_blk(struct f2fs_sb_info *sbi, + enum log_type type) +{ + return f2fs_need_rand_blk(sbi, type) || f2fs_need_rand_seg(sbi, type); } /* diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index 429fc5bd4ae0..4c7d07aa0838 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -312,10 +312,11 @@ static void select_policy(struct f2fs_sb_info *sbi, int gc_type, p->max_search = sbi->max_victim_search; /* let's select beginning hot/small space first. */ - if (f2fs_need_rand_seg(sbi)) + if (f2fs_need_rand_seg_blk(sbi, type)) { p->offset = get_random_u32_below(MAIN_SECS(sbi) * SEGS_PER_SEC(sbi)); - else if (type == CURSEG_HOT_DATA || IS_NODESEG(type)) + SIT_I(sbi)->last_victim[p->gc_mode] = p->offset; + } else if (type == CURSEG_HOT_DATA || IS_NODESEG(type)) p->offset = 0; else p->offset = SIT_I(sbi)->last_victim[p->gc_mode]; diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 5e2789cd96b5..8d2f5680ac7d 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -2964,7 +2964,7 @@ static unsigned int __get_next_segno(struct f2fs_sb_info *sbi, int type) sanity_check_seg_type(sbi, seg_type); if (__is_large_section(sbi)) { - if (f2fs_need_rand_seg(sbi)) { + if (f2fs_need_rand_seg_blk(sbi, type)) { unsigned int hint = GET_SEC_FROM_SEG(sbi, curseg->segno); if (GET_SEC_FROM_SEG(sbi, curseg->segno + 1) != hint) @@ -2973,7 +2973,7 @@ static unsigned int __get_next_segno(struct f2fs_sb_info *sbi, int type) GET_SEG_FROM_SEC(sbi, hint + 1) - 1); } return curseg->segno; - } else if (f2fs_need_rand_seg(sbi)) { + } else if (f2fs_need_rand_seg_blk(sbi, type)) { return get_random_u32_below(MAIN_SECS(sbi) * SEGS_PER_SEC(sbi)); } @@ -3029,7 +3029,7 @@ static int new_curseg(struct f2fs_sb_info *sbi, int type, bool new_sec) curseg->next_segno = segno; reset_curseg(sbi, type, 1); curseg->alloc_type = LFS; - if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK) + if (f2fs_need_rand_blk(sbi, type)) curseg->fragment_remained_chunk = get_random_u32_inclusive(1, sbi->max_fragment_chunk); return 0; @@ -3832,7 +3832,7 @@ int f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct folio *folio, curseg->next_blkoff = f2fs_find_next_ssr_block(sbi, curseg); } else { curseg->next_blkoff++; - if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK) + if (f2fs_need_rand_blk(sbi, type)) f2fs_randomize_chunk(sbi, curseg); } if (curseg->next_blkoff >= f2fs_usable_blks_in_seg(sbi, curseg->segno)) diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 6f787cbeefd0..c558cc4cf8fb 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2401,9 +2401,9 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root) seq_puts(seq, "adaptive"); else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_LFS) seq_puts(seq, "lfs"); - else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG) + else if (f2fs_need_rand_seg(sbi, NO_CHECK_TYPE)) seq_puts(seq, "fragment:segment"); - else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK) + else if (f2fs_need_rand_blk(sbi, NO_CHECK_TYPE)) seq_puts(seq, "fragment:block"); seq_printf(seq, ",active_logs=%u", F2FS_OPTION(sbi).active_logs); if (test_opt(sbi, RESERVE_ROOT) || test_opt(sbi, RESERVE_NODE)) diff --git a/fs/fat/nfs.c b/fs/fat/nfs.c index 509eea96a457..6e1b371711ed 100644 --- a/fs/fat/nfs.c +++ b/fs/fat/nfs.c @@ -250,8 +250,10 @@ struct inode *fat_rebuild_parent(struct super_block *sb, int parent_logstart) MSDOS_I(dummy_grand_parent)->i_pos = -1; } - if (!fat_scan_logstart(dummy_grand_parent, clus_to_match, &sinfo)) + if (!fat_scan_logstart(dummy_grand_parent, clus_to_match, &sinfo)) { parent = fat_build_inode(sb, sinfo.de, sinfo.i_pos); + brelse(sinfo.bh); + } brelse(parent_bh); iput(dummy_grand_parent); diff --git a/fs/fuse/cuse.c b/fs/fuse/cuse.c index e43d12f69d65..792620c2fa4b 100644 --- a/fs/fuse/cuse.c +++ b/fs/fuse/cuse.c @@ -650,6 +650,11 @@ static void __exit cuse_exit(void) { misc_deregister(&cuse_miscdev); class_destroy(cuse_class); + /* + * Wait for pending call_rcu() callbacks that call back into + * this module via fc->release (cuse_fc_release). + */ + rcu_barrier(); } module_init(cuse_init); diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index ae520c738821..a0a64dfffb75 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -1743,6 +1743,8 @@ static int fuse_fill_super_submount(struct super_block *sb, fuse_fill_attr_from_inode(&root_attr, parent_fi); root = fuse_iget(sb, parent_fi->nodeid, 0, &root_attr, 0, 0, fuse_get_evict_ctr(fm->fc)); + if (!root) + return -ENOMEM; /* * This inode is just a duplicate, so it is not looked up and * its nlookup should not be incremented. fuse_iget() does diff --git a/fs/hfsplus/catalog.c b/fs/hfsplus/catalog.c index 6c8380f7208d..32eb1283d7d0 100644 --- a/fs/hfsplus/catalog.c +++ b/fs/hfsplus/catalog.c @@ -204,7 +204,7 @@ int hfsplus_find_cat(struct super_block *sb, u32 cnid, return err; type = be16_to_cpu(tmp.type); - if (type != HFSPLUS_FOLDER_THREAD && type != HFSPLUS_FILE_THREAD) { + if (!is_hfs_thread_record_type(type)) { pr_err("found bad thread record in catalog\n"); return -EIO; } @@ -350,23 +350,22 @@ int hfsplus_delete_cat(u32 cnid, struct inode *dir, const struct qstr *str) goto out; if (!str) { - int len; + hfsplus_cat_entry entry = {0}; hfsplus_cat_build_key_with_cnid(sb, fd.search_key, cnid); - err = hfs_brec_find(&fd, hfs_find_rec_by_key); + err = hfsplus_brec_read_cat(&fd, &entry); if (err) goto out; - off = fd.entryoffset + - offsetof(struct hfsplus_cat_thread, nodeName); - fd.search_key->cat.parent = cpu_to_be32(dir->i_ino); - hfs_bnode_read(fd.bnode, - &fd.search_key->cat.name.length, off, 2); - len = be16_to_cpu(fd.search_key->cat.name.length) * 2; - hfs_bnode_read(fd.bnode, - &fd.search_key->cat.name.unicode, - off + 2, len); - fd.search_key->key_len = cpu_to_be16(6 + len); + type = be16_to_cpu(entry.type); + if (!is_hfs_thread_record_type(type)) { + pr_err("found bad thread record in catalog\n"); + err = -EIO; + goto out; + } + + hfsplus_cat_build_key_uni(fd.search_key, dir->i_ino, + &entry.thread.nodeName); } else { err = hfsplus_cat_build_key(sb, fd.search_key, dir->i_ino, str); if (unlikely(err)) diff --git a/fs/hfsplus/hfsplus_fs.h b/fs/hfsplus/hfsplus_fs.h index 2da2bd52d200..e66773cbb333 100644 --- a/fs/hfsplus/hfsplus_fs.h +++ b/fs/hfsplus/hfsplus_fs.h @@ -514,6 +514,12 @@ static inline u32 hfsplus_cat_thread_size(const struct hfsplus_cat_thread *threa be16_to_cpu(thread->nodeName.length) * sizeof(hfsplus_unichr); } +static inline +bool is_hfs_thread_record_type(u16 type) +{ + return type == HFSPLUS_FOLDER_THREAD || type == HFSPLUS_FILE_THREAD; +} + int hfsplus_brec_read_cat(struct hfs_find_data *fd, hfsplus_cat_entry *entry); /* diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index f42548ee9083..34ed9966f2ea 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -1433,7 +1433,8 @@ hugetlbfs_fill_super(struct super_block *sb, struct fs_context *fc) goto out_free; return 0; out_free: - kfree(sbinfo->spool); + if (sbinfo->spool) + hugepage_put_subpool(sbinfo->spool); kfree(sbinfo); return -ENOMEM; } diff --git a/fs/inode.c b/fs/inode.c index 2c55ec49b023..a931126398ce 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -2703,8 +2703,8 @@ struct timespec64 inode_set_ctime_to_ts(struct inode *inode, struct timespec64 t { trace_inode_set_ctime_to_ts(inode, &ts); set_normalized_timespec64(&ts, ts.tv_sec, ts.tv_nsec); - inode->i_ctime_sec = ts.tv_sec; - inode->i_ctime_nsec = ts.tv_nsec; + WRITE_ONCE(inode->i_ctime_sec, ts.tv_sec); + WRITE_ONCE(inode->i_ctime_nsec, ts.tv_nsec); return ts; } EXPORT_SYMBOL(inode_set_ctime_to_ts); @@ -2778,7 +2778,7 @@ struct timespec64 inode_set_ctime_current(struct inode *inode) */ cns = smp_load_acquire(&inode->i_ctime_nsec); if (cns & I_CTIME_QUERIED) { - struct timespec64 ctime = { .tv_sec = inode->i_ctime_sec, + struct timespec64 ctime = { .tv_sec = inode_get_ctime_sec(inode), .tv_nsec = cns & ~I_CTIME_QUERIED }; if (timespec64_compare(&now, &ctime) <= 0) { @@ -2790,7 +2790,7 @@ struct timespec64 inode_set_ctime_current(struct inode *inode) mgtime_counter_inc(mg_ctime_updates); /* No need to cmpxchg if it's exactly the same */ - if (cns == now.tv_nsec && inode->i_ctime_sec == now.tv_sec) { + if (cns == now.tv_nsec && inode_get_ctime_sec(inode) == now.tv_sec) { trace_ctime_xchg_skip(inode, &now); goto out; } @@ -2799,7 +2799,7 @@ struct timespec64 inode_set_ctime_current(struct inode *inode) /* Try to swap the nsec value into place. */ if (try_cmpxchg(&inode->i_ctime_nsec, &cur, now.tv_nsec)) { /* If swap occurred, then we're (mostly) done */ - inode->i_ctime_sec = now.tv_sec; + WRITE_ONCE(inode->i_ctime_sec, now.tv_sec); trace_ctime_ns_xchg(inode, cns, now.tv_nsec, cur); mgtime_counter_inc(mg_ctime_swaps); } else { @@ -2814,7 +2814,7 @@ struct timespec64 inode_set_ctime_current(struct inode *inode) goto retry; } /* Otherwise, keep the existing ctime */ - now.tv_sec = inode->i_ctime_sec; + now.tv_sec = inode_get_ctime_sec(inode); now.tv_nsec = cur & ~I_CTIME_QUERIED; } out: @@ -2847,7 +2847,7 @@ struct timespec64 inode_set_ctime_deleg(struct inode *inode, struct timespec64 u /* pairs with try_cmpxchg below */ cur = smp_load_acquire(&inode->i_ctime_nsec); cur_ts.tv_nsec = cur & ~I_CTIME_QUERIED; - cur_ts.tv_sec = inode->i_ctime_sec; + cur_ts.tv_sec = inode_get_ctime_sec(inode); /* If the update is older than the existing value, skip it. */ if (timespec64_compare(&update, &cur_ts) <= 0) @@ -2873,7 +2873,7 @@ struct timespec64 inode_set_ctime_deleg(struct inode *inode, struct timespec64 u retry: old = cur; if (try_cmpxchg(&inode->i_ctime_nsec, &cur, update.tv_nsec)) { - inode->i_ctime_sec = update.tv_sec; + WRITE_ONCE(inode->i_ctime_sec, update.tv_sec); mgtime_counter_inc(mg_ctime_swaps); return update; } @@ -2889,7 +2889,7 @@ struct timespec64 inode_set_ctime_deleg(struct inode *inode, struct timespec64 u goto retry; /* Otherwise, it was a new timestamp. */ - cur_ts.tv_sec = inode->i_ctime_sec; + cur_ts.tv_sec = inode_get_ctime_sec(inode); cur_ts.tv_nsec = cur & ~I_CTIME_QUERIED; return cur_ts; } diff --git a/fs/isofs/compress.c b/fs/isofs/compress.c index 5f3b6da0e022..4500b14f802a 100644 --- a/fs/isofs/compress.c +++ b/fs/isofs/compress.c @@ -65,12 +65,14 @@ static loff_t zisofs_uncompress_block(struct inode *inode, loff_t block_start, /* Empty block? */ if (block_size == 0) { for ( i = 0 ; i < pcount ; i++ ) { + unsigned int off = i ? 0 : poffset; + if (!pages[i]) continue; - memzero_page(pages[i], 0, PAGE_SIZE); + memzero_page(pages[i], off, PAGE_SIZE - off); SetPageUptodate(pages[i]); } - return ((loff_t)pcount) << PAGE_SHIFT; + return (((loff_t)pcount) << PAGE_SHIFT) - poffset; } /* Because zlib is not thread-safe, do all the I/O at the top. */ @@ -291,6 +293,7 @@ static int zisofs_fill_pages(struct inode *inode, int full_page, int pcount, memzero_page(*pages, poffset, PAGE_SIZE - poffset); SetPageUptodate(*pages); } + brelse(bh); return 0; } diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 134d7f760a33..4b0d1bb50dd6 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -85,15 +85,17 @@ bl_free_device(struct pnfs_block_dev *dev) { bl_unregister_dev(dev); - if (dev->nr_children) { + if (dev->children) { int i; for (i = 0; i < dev->nr_children; i++) bl_free_device(&dev->children[i]); kfree(dev->children); - } else { - if (dev->bdev_file) - fput(dev->bdev_file); + dev->children = NULL; + dev->nr_children = 0; + } else if (dev->bdev_file) { + fput(dev->bdev_file); + dev->bdev_file = NULL; } } @@ -434,6 +436,7 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, out_blkdev_put: fput(d->bdev_file); + d->bdev_file = NULL; return error; } @@ -469,8 +472,11 @@ bl_parse_concat(struct nfs_server *server, struct pnfs_block_dev *d, for (i = 0; i < v->concat.volumes_count; i++) { ret = bl_parse_deviceid(server, &d->children[i], volumes, v->concat.volumes[i], gfp_mask); - if (ret) + if (ret) { + bl_free_device(&d->children[i]); + bl_free_device(d); return ret; + } d->nr_children++; d->children[i].start += len; @@ -498,8 +504,11 @@ bl_parse_stripe(struct nfs_server *server, struct pnfs_block_dev *d, for (i = 0; i < v->stripe.volumes_count; i++) { ret = bl_parse_deviceid(server, &d->children[i], volumes, v->stripe.volumes[i], gfp_mask); - if (ret) + if (ret) { + bl_free_device(&d->children[i]); + bl_free_device(d); return ret; + } d->nr_children++; len += d->children[i].len; diff --git a/fs/nfs/client.c b/fs/nfs/client.c index 404e75b0444d..e3cd92b5aaa3 100644 --- a/fs/nfs/client.c +++ b/fs/nfs/client.c @@ -215,9 +215,21 @@ static void nfs_cb_idr_remove_locked(struct nfs_client *clp) { struct nfs_net *nn = net_generic(clp->cl_net, nfs_net_id); - if (clp->cl_cb_ident) + if (clp->cl_cb_ident) { idr_remove(&nn->cb_ident_idr, clp->cl_cb_ident); + clp->cl_cb_ident = 0; + } +} + +void nfs_cb_idr_remove(struct nfs_client *clp) +{ + struct nfs_net *nn = net_generic(clp->cl_net, nfs_net_id); + + spin_lock(&nn->nfs_client_lock); + nfs_cb_idr_remove_locked(clp); + spin_unlock(&nn->nfs_client_lock); } +EXPORT_SYMBOL_GPL(nfs_cb_idr_remove); static void pnfs_init_server(struct nfs_server *server) { diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c index 31ca8bbfb970..38b0834e3ebf 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -2275,6 +2275,13 @@ int nfs_atomic_open_v23(struct inode *dir, struct dentry *dentry, if (open_flags & O_CREAT) { error = nfs_do_create(dir, dentry, mode, open_flags); if (!error) { + /* With UNCHECKED mode, a server may return NFS3_OK for + * a pre-existing non-regular file (e.g. a symlink). + * Let the VFS handle it; calling finish_open() would + * hit no_open() and return -ENXIO. + */ + if (!d_is_reg(dentry)) + return finish_no_open(file, NULL); file->f_mode |= FMODE_CREATED; return finish_open(file, dentry, NULL); } else if (error != -EEXIST || open_flags & O_EXCL) diff --git a/fs/nfs/filelayout/filelayoutdev.c b/fs/nfs/filelayout/filelayoutdev.c index df79aeb68db4..e13d880de581 100644 --- a/fs/nfs/filelayout/filelayoutdev.c +++ b/fs/nfs/filelayout/filelayoutdev.c @@ -170,7 +170,8 @@ nfs4_fl_alloc_deviceid_node(struct nfs_server *server, struct pnfs_device *pdev, goto out_err_free_deviceid; } - dsaddr->ds_list[i] = nfs4_pnfs_ds_add(net, &dsaddrs, gfp_flags); + dsaddr->ds_list[i] = nfs4_pnfs_ds_add(net, &dsaddrs, 4, + gfp_flags); if (!dsaddr->ds_list[i]) goto out_err_drain_dsaddrs; trace_fl_getdevinfo(server, &pdev->dev_id, dsaddr->ds_list[i]->ds_remotestr); diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c index 4604f1088272..8099bfc4fe2f 100644 --- a/fs/nfs/flexfilelayout/flexfilelayout.c +++ b/fs/nfs/flexfilelayout/flexfilelayout.c @@ -1324,7 +1324,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task, struct pnfs_layout_hdr *lo = lseg->pls_layout; struct inode *inode = lo->plh_inode; struct nfs4_deviceid_node *devid = FF_LAYOUT_DEVID_NODE(lseg, idx, dss_id); - struct nfs4_slot_table *tbl = &clp->cl_session->fc_slot_table; + struct nfs4_slot_table *tbl = nfs4_has_session(clp) ? + &clp->cl_session->fc_slot_table : clp->cl_slot_tbl; switch (op_status) { case NFS4_OK: diff --git a/fs/nfs/flexfilelayout/flexfilelayoutdev.c b/fs/nfs/flexfilelayout/flexfilelayoutdev.c index c2d8a13a9dbd..cb88c51346d9 100644 --- a/fs/nfs/flexfilelayout/flexfilelayoutdev.c +++ b/fs/nfs/flexfilelayout/flexfilelayoutdev.c @@ -149,7 +149,8 @@ nfs4_ff_alloc_deviceid_node(struct nfs_server *server, struct pnfs_device *pdev, new_ds->ds_versions = ds_versions; new_ds->ds_versions_cnt = version_count; - new_ds->ds = nfs4_pnfs_ds_add(net, &dsaddrs, gfp_flags); + new_ds->ds = nfs4_pnfs_ds_add(net, &dsaddrs, ds_versions[0].version, + gfp_flags); if (!new_ds->ds) goto out_err_drain_dsaddrs; diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index 8c2bfcc323e0..970e67768d67 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -1546,9 +1546,7 @@ int nfs_clear_invalid_mapping(struct address_space *mapping) ret = nfs_invalidate_mapping(inode, mapping); trace_nfs_invalidate_mapping_exit(inode, ret); - clear_bit_unlock(NFS_INO_INVALIDATING, bitlock); - smp_mb__after_atomic(); - wake_up_bit(bitlock, NFS_INO_INVALIDATING); + clear_and_wake_up_bit(NFS_INO_INVALIDATING, bitlock); out: return ret; } diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h index ea239a7e5ef6..8d0d238b7f88 100644 --- a/fs/nfs/internal.h +++ b/fs/nfs/internal.h @@ -225,6 +225,7 @@ void nfs_server_copy_userdata(struct nfs_server *, struct nfs_server *); extern void nfs_put_client(struct nfs_client *); extern void nfs_free_client(struct nfs_client *); +void nfs_cb_idr_remove(struct nfs_client *clp); extern struct nfs_client *nfs4_find_client_ident(struct net *, int); extern struct nfs_client * nfs4_find_client_sessionid(struct net *, const struct sockaddr *, diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c index 8f7ed5c1549e..c0880a966b33 100644 --- a/fs/nfs/nfs4client.c +++ b/fs/nfs/nfs4client.c @@ -272,6 +272,7 @@ struct nfs_client *nfs4_alloc_client(const struct nfs_client_initdata *cl_init) return clp; error: + nfs_cb_idr_remove(clp); nfs_free_client(clp); return ERR_PTR(err); } diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 66b8da741a06..84ba899bf0a7 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -7862,6 +7862,7 @@ static int nfs4_add_lease(struct file *file, int arg, struct file_lease **lease, { struct inode *inode = file_inode(file); fmode_t type = arg == F_RDLCK ? FMODE_READ : FMODE_WRITE; + fl_owner_t owner = (*lease)->c.flc_owner; int ret; /* No delegation, no lease */ @@ -7871,7 +7872,8 @@ static int nfs4_add_lease(struct file *file, int arg, struct file_lease **lease, if (ret || nfs4_have_delegation(inode, type, 0)) return ret; /* We raced with a delegation return */ - nfs4_delete_lease(file, priv); + dprintk("%s: raced with a delegation return\n", __func__); + nfs4_delete_lease(file, &owner); return -EAGAIN; } diff --git a/fs/nfs/nfs4session.c b/fs/nfs/nfs4session.c index 5db460476bf2..c59e710eb7cc 100644 --- a/fs/nfs/nfs4session.c +++ b/fs/nfs/nfs4session.c @@ -634,16 +634,22 @@ int nfs4_init_ds_session(struct nfs_client *clp, unsigned long lease_time) int ret; spin_lock(&clp->cl_lock); - if (test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) { - /* - * Do not set NFS_CS_CHECK_LEASE_TIME instead set the - * DS lease to be equal to the MDS lease. - */ + /* + * Do not set NFS_CS_CHECK_LEASE_TIME instead set the + * DS lease to be equal to the MDS lease. + * + * A v4.0 DS has no session, so seed the lease every time. + */ + if (!session || + test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) { clp->cl_lease_time = lease_time; clp->cl_last_renewal = jiffies; } spin_unlock(&clp->cl_lock); + if (!session) + return 0; + ret = nfs41_check_session_ready(clp); if (ret) return ret; diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index 57067589786f..cf0dcf9e35d7 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -2088,15 +2088,6 @@ static bool pnfs_is_first_layoutget(struct pnfs_layout_hdr *lo) return test_bit(NFS_LAYOUT_FIRST_LAYOUTGET, &lo->plh_flags); } -static void pnfs_clear_first_layoutget(struct pnfs_layout_hdr *lo) -{ - unsigned long *bitlock = &lo->plh_flags; - - clear_bit_unlock(NFS_LAYOUT_FIRST_LAYOUTGET, bitlock); - smp_mb__after_atomic(); - wake_up_bit(bitlock, NFS_LAYOUT_FIRST_LAYOUTGET); -} - static void _add_to_server_list(struct pnfs_layout_hdr *lo, struct nfs_server *server) { @@ -2272,7 +2263,8 @@ pnfs_update_layout(struct inode *ino, iomode, lo, lseg, PNFS_UPDATE_LAYOUT_INVALID_OPEN); nfs4_schedule_stateid_recovery(server, ctx->state); - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, + &lo->plh_flags); pnfs_put_layout_hdr(lo); goto lookup_again; } @@ -2341,7 +2333,8 @@ pnfs_update_layout(struct inode *ino, if (!exception.retry) goto out_put_layout_hdr; if (first) - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, + &lo->plh_flags); trace_pnfs_update_layout(ino, pos, count, iomode, lo, lseg, PNFS_UPDATE_LAYOUT_RETRY); pnfs_put_layout_hdr(lo); @@ -2353,7 +2346,7 @@ pnfs_update_layout(struct inode *ino, out_put_layout_hdr: if (first) - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, &lo->plh_flags); trace_pnfs_update_layout(ino, pos, count, iomode, lo, lseg, PNFS_UPDATE_LAYOUT_EXIT); pnfs_put_layout_hdr(lo); @@ -2445,7 +2438,7 @@ static void _lgopen_prepare_attached(struct nfs4_opendata *data, lgp = pnfs_alloc_init_layoutget_args(ino, ctx, ¤t_stateid, &rng, nfs_io_gfp_mask()); if (!lgp) { - pnfs_clear_first_layoutget(lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, &lo->plh_flags); nfs_layoutget_end(lo); pnfs_put_layout_hdr(lo); return; @@ -2549,7 +2542,8 @@ void nfs4_lgopen_release(struct nfs4_layoutget *lgp) { if (lgp != NULL) { if (lgp->lo) { - pnfs_clear_first_layoutget(lgp->lo); + clear_and_wake_up_bit(NFS_LAYOUT_FIRST_LAYOUTGET, + &lgp->lo->plh_flags); nfs_layoutget_end(lgp->lo); } pnfs_layoutget_free(lgp); @@ -3262,15 +3256,6 @@ pnfs_generic_pg_readpages(struct nfs_pageio_descriptor *desc) } EXPORT_SYMBOL_GPL(pnfs_generic_pg_readpages); -static void pnfs_clear_layoutcommitting(struct inode *inode) -{ - unsigned long *bitlock = &NFS_I(inode)->flags; - - clear_bit_unlock(NFS_INO_LAYOUTCOMMITTING, bitlock); - smp_mb__after_atomic(); - wake_up_bit(bitlock, NFS_INO_LAYOUTCOMMITTING); -} - /* * There can be multiple RW segments. */ @@ -3295,7 +3280,7 @@ static void pnfs_list_write_lseg_done(struct inode *inode, struct list_head *lis pnfs_put_lseg(lseg); } - pnfs_clear_layoutcommitting(inode); + clear_and_wake_up_bit(NFS_INO_LAYOUTCOMMITTING, &NFS_I(inode)->flags); } void pnfs_set_lo_fail(struct pnfs_layout_segment *lseg) @@ -3435,7 +3420,7 @@ pnfs_layoutcommit_inode(struct inode *inode, bool sync) spin_unlock(&inode->i_lock); kfree(data); clear_layoutcommitting: - pnfs_clear_layoutcommitting(inode); + clear_and_wake_up_bit(NFS_INO_LAYOUTCOMMITTING, &NFS_I(inode)->flags); goto out; } EXPORT_SYMBOL_GPL(pnfs_layoutcommit_inode); diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index 3db8f13d8fe4..7b3ba2e64e6f 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -63,6 +63,7 @@ struct nfs4_pnfs_ds { const struct net *ds_net; struct nfs_client *ds_clp; refcount_t ds_count; + u32 ds_version; /* cache key, with ds_addrs */ unsigned long ds_state; #define NFS4DS_CONNECTING 0 /* ds is establishing connection */ }; @@ -417,7 +418,7 @@ void pnfs_generic_write_commit_done(struct rpc_task *task, void *data); void nfs4_pnfs_ds_put(struct nfs4_pnfs_ds *ds); struct nfs4_pnfs_ds *nfs4_pnfs_ds_add(const struct net *net, struct list_head *dsaddrs, - gfp_t gfp_flags); + u32 version, gfp_t gfp_flags); void nfs4_pnfs_v3_ds_connect_unload(void); int nfs4_pnfs_ds_connect(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds, struct nfs4_deviceid_node *devid, unsigned int timeo, diff --git a/fs/nfs/pnfs_nfs.c b/fs/nfs/pnfs_nfs.c index 1e291fd23d46..5e85ea0ef584 100644 --- a/fs/nfs/pnfs_nfs.c +++ b/fs/nfs/pnfs_nfs.c @@ -603,15 +603,17 @@ _same_data_server_addrs_locked(const struct list_head *dsaddrs1, } /* - * Lookup DS by addresses. nfs4_ds_cache_lock is held + * Lookup DS by addresses and NFS version. nfs4_ds_cache_lock is held */ static struct nfs4_pnfs_ds * -_data_server_lookup_locked(const struct nfs_net *nn, const struct list_head *dsaddrs) +_data_server_lookup_locked(const struct nfs_net *nn, + const struct list_head *dsaddrs, u32 version) { struct nfs4_pnfs_ds *ds; list_for_each_entry(ds, &nn->nfs4_data_server_cache, ds_node) - if (_same_data_server_addrs_locked(&ds->ds_addrs, dsaddrs)) + if (ds->ds_version == version && + _same_data_server_addrs_locked(&ds->ds_addrs, dsaddrs)) return ds; return NULL; } @@ -719,7 +721,8 @@ nfs4_pnfs_remotestr(struct list_head *dsaddrs, gfp_t gfp_flags) * uncached and return cached struct nfs4_pnfs_ds. */ struct nfs4_pnfs_ds * -nfs4_pnfs_ds_add(const struct net *net, struct list_head *dsaddrs, gfp_t gfp_flags) +nfs4_pnfs_ds_add(const struct net *net, struct list_head *dsaddrs, u32 version, + gfp_t gfp_flags) { struct nfs_net *nn = net_generic(net, nfs_net_id); struct nfs4_pnfs_ds *tmp_ds, *ds = NULL; @@ -738,7 +741,7 @@ nfs4_pnfs_ds_add(const struct net *net, struct list_head *dsaddrs, gfp_t gfp_fla remotestr = nfs4_pnfs_remotestr(dsaddrs, gfp_flags); spin_lock(&nn->nfs4_data_server_lock); - tmp_ds = _data_server_lookup_locked(nn, dsaddrs); + tmp_ds = _data_server_lookup_locked(nn, dsaddrs, version); if (tmp_ds == NULL) { INIT_LIST_HEAD(&ds->ds_addrs); list_splice_init(dsaddrs, &ds->ds_addrs); @@ -747,6 +750,7 @@ nfs4_pnfs_ds_add(const struct net *net, struct list_head *dsaddrs, gfp_t gfp_fla INIT_LIST_HEAD(&ds->ds_node); ds->ds_net = net; ds->ds_clp = NULL; + ds->ds_version = version; list_add(&ds->ds_node, &nn->nfs4_data_server_cache); dprintk("%s add new data server %s\n", __func__, ds->ds_remotestr); diff --git a/fs/nilfs2/page.c b/fs/nilfs2/page.c index 56c4da417b6a..89180e2f59fe 100644 --- a/fs/nilfs2/page.c +++ b/fs/nilfs2/page.c @@ -243,6 +243,7 @@ static void nilfs_copy_folio(struct folio *dst, struct folio *src, int nilfs_copy_dirty_pages(struct address_space *dmap, struct address_space *smap) { + struct inode *smap_inode = smap->host; struct folio_batch fbatch; unsigned int i; pgoff_t index = 0; @@ -258,8 +259,19 @@ int nilfs_copy_dirty_pages(struct address_space *dmap, struct folio *folio = fbatch.folios[i], *dfolio; folio_lock(folio); - if (unlikely(!folio_test_dirty(folio))) - NILFS_FOLIO_BUG(folio, "inconsistent dirty state"); + if (unlikely(!folio_test_dirty(folio))) { + if (WARN_ONCE(!sb_rdonly(smap_inode->i_sb), + "inconsistent dirty state\n")) + goto unlock_folio; + + /* + * If the filesystem has been forced to read-only + * due to metadata corruption. + */ + folio_unlock(folio); + err = -EROFS; + break; + } dfolio = filemap_grab_folio(dmap, folio->index); if (IS_ERR(dfolio)) { @@ -277,6 +289,7 @@ int nilfs_copy_dirty_pages(struct address_space *dmap, folio_unlock(dfolio); folio_put(dfolio); +unlock_folio: folio_unlock(folio); } folio_batch_release(&fbatch); diff --git a/fs/nilfs2/segment.c b/fs/nilfs2/segment.c index 0bc1f0f02f31..c20340f4e0dc 100644 --- a/fs/nilfs2/segment.c +++ b/fs/nilfs2/segment.c @@ -2561,6 +2561,10 @@ int nilfs_clean_segments(struct super_block *sb, struct nilfs_argv *argv, break; nilfs_warn(sb, "error %d cleaning segments", err); + + if (unlikely(err == -EROFS)) + goto out_unlock; + set_current_state(TASK_INTERRUPTIBLE); schedule_timeout(sci->sc_interval); } diff --git a/fs/nilfs2/the_nilfs.c b/fs/nilfs2/the_nilfs.c index d0bcf744c553..9911a484398a 100644 --- a/fs/nilfs2/the_nilfs.c +++ b/fs/nilfs2/the_nilfs.c @@ -461,6 +461,12 @@ static int nilfs_store_disk_layout(struct the_nilfs *nilfs, nilfs->ns_inode_size); return -EINVAL; } + if (NILFS_SR_BYTES(nilfs->ns_inode_size) > nilfs->ns_blocksize) { + nilfs_err(nilfs->ns_sb, + "too large inode size for super root: %d bytes", + nilfs->ns_inode_size); + return -EINVAL; + } nilfs->ns_first_ino = le32_to_cpu(sbp->s_first_ino); if (nilfs->ns_first_ino < NILFS_USER_INO) { diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c index cf57eabfed45..cdb69c23e2e6 100644 --- a/fs/notify/fanotify/fanotify.c +++ b/fs/notify/fanotify/fanotify.c @@ -601,6 +601,7 @@ static struct fanotify_event *fanotify_alloc_perm_event(const void *data, pevent->hdr.pad = 0; pevent->hdr.len = 0; pevent->state = FAN_EVENT_INIT; + pevent->watchdog_cnt = 0; pevent->path = *path; pevent->pos = range ? range->pos : FANOTIFY_NO_RANGE; pevent->count = range ? range->count : 0; diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c index eb808eda11f2..629407c3e873 100644 --- a/fs/notify/fanotify/fanotify_user.c +++ b/fs/notify/fanotify/fanotify_user.c @@ -111,7 +111,12 @@ static DECLARE_DELAYED_WORK(perm_group_work, perm_group_watchdog); static void perm_group_watchdog_schedule(void) { - schedule_delayed_work(&perm_group_work, secs_to_jiffies(perm_group_timeout)); + int timeout = READ_ONCE(perm_group_timeout); + + if (!timeout) + return; + + schedule_delayed_work(&perm_group_work, secs_to_jiffies(timeout)); } static void perm_group_watchdog(struct work_struct *work) @@ -1153,11 +1158,13 @@ static long fanotify_ioctl(struct file *file, unsigned int cmd, unsigned long ar { struct fsnotify_group *group; struct fsnotify_event *fsn_event; + unsigned int info_mode; void __user *p; int ret = -ENOTTY; size_t send_len = 0; group = file->private_data; + info_mode = FAN_GROUP_FLAG(group, FANOTIFY_INFO_MODES); p = (void __user *) arg; @@ -1165,7 +1172,8 @@ static long fanotify_ioctl(struct file *file, unsigned int cmd, unsigned long ar case FIONREAD: spin_lock(&group->notification_lock); list_for_each_entry(fsn_event, &group->notification_list, list) - send_len += FAN_EVENT_METADATA_LEN; + send_len += fanotify_event_len(info_mode, + FANOTIFY_E(fsn_event)); spin_unlock(&group->notification_lock); ret = put_user(send_len, (int __user *) p); break; diff --git a/fs/ntfs3/fslog.c b/fs/ntfs3/fslog.c index 5437dc78209e..79d5ae5863b9 100644 --- a/fs/ntfs3/fslog.c +++ b/fs/ntfs3/fslog.c @@ -875,6 +875,9 @@ static inline struct RESTART_TABLE *extend_rsttbl(struct RESTART_TABLE *tbl, u32 used = le16_to_cpu(tbl->used); struct RESTART_TABLE *rt; + if (used + add > U16_MAX) + return NULL; + rt = init_rsttbl(esize, used + add); if (!rt) return NULL; @@ -2296,7 +2299,15 @@ static int read_log_rec_buf(struct ntfs_log *log, */ for (;;) { bool usa_error; - u32 tail = log->page_size - off; + u32 tail; + + /* off comes from the on-disk restart area; bound it. */ + if (off > log->page_size) { + err = -EINVAL; + goto out; + } + + tail = log->page_size - off; if (tail >= data_len) tail = data_len; diff --git a/fs/ntfs3/fsntfs.c b/fs/ntfs3/fsntfs.c index 83df92df1ee0..5e3506815c27 100644 --- a/fs/ntfs3/fsntfs.c +++ b/fs/ntfs3/fsntfs.c @@ -2293,8 +2293,8 @@ int ntfs_reparse_init(struct ntfs_sb_info *sbi) goto out; } - root_r = resident_data(attr); - if (root_r->type != ATTR_ZERO || + root_r = resident_data_ex(attr, sizeof(struct INDEX_ROOT)); + if (!root_r || root_r->type != ATTR_ZERO || root_r->rule != NTFS_COLLATION_TYPE_UINTS) { err = -EINVAL; goto out; @@ -2331,8 +2331,8 @@ int ntfs_objid_init(struct ntfs_sb_info *sbi) goto out; } - root = resident_data(attr); - if (root->type != ATTR_ZERO || + root = resident_data_ex(attr, sizeof(struct INDEX_ROOT)); + if (!root || root->type != ATTR_ZERO || root->rule != NTFS_COLLATION_TYPE_UINTS) { err = -EINVAL; goto out; diff --git a/fs/ntfs3/record.c b/fs/ntfs3/record.c index 714c7ecedca8..0f8ba76094fe 100644 --- a/fs/ntfs3/record.c +++ b/fs/ntfs3/record.c @@ -202,7 +202,7 @@ struct ATTRIB *mi_enum_attr(struct ntfs_inode *ni, struct mft_inode *mi, u32 used = le32_to_cpu(rec->used); u32 t32, off, asize, prev_type; u16 t16; - u64 data_size, alloc_size, tot_size; + u64 svcn, evcn, data_size, alloc_size, tot_size; if (!attr) { u32 total = le32_to_cpu(rec->total); @@ -310,10 +310,38 @@ struct ATTRIB *mi_enum_attr(struct ntfs_inode *ni, struct mft_inode *mi, if (t32 && le16_to_cpu(attr->name_off) + t32 > t16) goto out; - /* Check start/end vcn. */ - if (le64_to_cpu(attr->nres.svcn) > le64_to_cpu(attr->nres.evcn) + 1) + /* + * Check start/end vcn. svcn == 0 with evcn == -1 (U64_MAX) is the + * sentinel for an empty non-resident attribute (no allocated + * clusters) and must be accepted: "svcn > evcn + 1" tolerates it, + * since "(u64)-1 + 1" is 0 and "0 > 0" is false. + * + * For a non-empty attribute evcn is a cluster index and must lie + * within the volume (sbi->used.bitmap.nbits, set up in + * ntfs_init_from_boot() before any caller of mi_enum_attr() runs). + * Bounding evcn also prevents a malformed value close to U64_MAX + * from slipping through the near-wrap "evcn + 1" upper bound. + */ + svcn = le64_to_cpu(attr->nres.svcn); + evcn = le64_to_cpu(attr->nres.evcn); + if (svcn > evcn + 1) goto out; + if (is_attr_ext(attr)) { + /* sparsed/compressed attribute. */ +#ifdef CONFIG_NTFS3_64BIT_CLUSTER + /* No limits. */ +#else + /* Check evcn fits into 32 bits. */ + if (evcn != U64_MAX && evcn >= (1ull << 32)) + goto out; +#endif + } else { + /* Check out of volume for normal attribute. */ + if (evcn != U64_MAX && evcn >= mi->sbi->used.bitmap.nbits) + goto out; + } + data_size = le64_to_cpu(attr->nres.data_size); if (le64_to_cpu(attr->nres.valid_size) > data_size) goto out; diff --git a/fs/ntfs3/super.c b/fs/ntfs3/super.c index 9a2e3d0efd99..98cfa3357162 100644 --- a/fs/ntfs3/super.c +++ b/fs/ntfs3/super.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include #include @@ -918,7 +919,7 @@ static int ntfs_init_from_boot(struct super_block *sb, u32 sector_size, struct ntfs_sb_info *sbi = sb->s_fs_info; int err; u32 mb, gb, boot_sector_size, sct_per_clst, record_size; - u64 sectors, clusters, mlcn, mlcn2, dev_size0; + u64 sectors, clusters, mlcn, mlcn2, mft_pos, mft2_pos, dev_size0; struct NTFS_BOOT *boot; struct buffer_head *bh; struct MFT_REC *rec; @@ -987,7 +988,15 @@ static int ntfs_init_from_boot(struct super_block *sb, u32 sector_size, mlcn2 = le64_to_cpu(boot->mft2_clst); sectors = le64_to_cpu(boot->sectors_per_volume); - if (mlcn * sct_per_clst >= sectors || mlcn2 * sct_per_clst >= sectors) { + /* + * Convert mlcn/mlcn2 to sector positions before comparing with + * 'sectors'. All three are u64 values that come from the boot + * sector, so use check_mul_overflow() to keep a wraparound from + * silently bypassing the comparison. + */ + if (check_mul_overflow(mlcn, (u64)sct_per_clst, &mft_pos) || + check_mul_overflow(mlcn2, (u64)sct_per_clst, &mft2_pos) || + mft_pos >= sectors || mft2_pos >= sectors) { ntfs_err( sb, "%s: start of MFT 0x%llx (0x%llx) is out of volume 0x%llx.", diff --git a/fs/ntfs3/xattr.c b/fs/ntfs3/xattr.c index d719e1073dbb..a664ac5398c6 100644 --- a/fs/ntfs3/xattr.c +++ b/fs/ntfs3/xattr.c @@ -146,26 +146,29 @@ static int ntfs_read_ea(struct ntfs_inode *ni, struct EA_FULL **ea, for (off = 0; off < size; off += ea_size) { const struct EA_FULL *ef = Add2Ptr(ea_p, off); u32 bytes = size - off; + size_t need; /* Check if we can use field ea->size. */ if (bytes < sizeof(ef->size)) goto out1; + /* Check if we can use fields ef->name_len and ef->elength. */ + if (bytes < offsetof(struct EA_FULL, name)) + goto out1; + + /* Size needed to hold this record's name and value. */ + need = struct_size(ef, name, + 1 + ef->name_len + le16_to_cpu(ef->elength)); + if (ef->size) { ea_size = le32_to_cpu(ef->size); - if (ea_size > bytes) + /* ef->size must fit the list and cover the record. */ + if (ea_size > bytes || ea_size < need) goto out1; continue; } - /* Check if we can use fields ef->name_len and ef->elength. */ - if (bytes < offsetof(struct EA_FULL, name)) - goto out1; - - ea_size = ALIGN(struct_size(ef, name, - 1 + ef->name_len + - le16_to_cpu(ef->elength)), - 4); + ea_size = ALIGN(need, 4); if (ea_size > bytes) goto out1; } diff --git a/fs/ocfs2/acl.c b/fs/ocfs2/acl.c index 62464d194da3..095ce96c2484 100644 --- a/fs/ocfs2/acl.c +++ b/fs/ocfs2/acl.c @@ -109,8 +109,7 @@ static void *ocfs2_acl_to_xattr(const struct posix_acl *acl, size_t *size) return ocfs2_acl; } -static struct posix_acl *ocfs2_get_acl_nolock(struct inode *inode, - int type, +static struct posix_acl *ocfs2_get_acl_nolock(struct inode *inode, int type, struct buffer_head *di_bh) { int name_index; @@ -348,63 +347,105 @@ int ocfs2_acl_chmod(struct inode *inode, struct buffer_head *bh) * Initialize the ACLs of a new inode. If parent directory has default ACL, * then clone to new inode. Called from ocfs2_mknod. */ -int ocfs2_init_acl(handle_t *handle, - struct inode *inode, - struct inode *dir, - struct buffer_head *di_bh, - struct buffer_head *dir_bh, - struct ocfs2_alloc_context *meta_ac, - struct ocfs2_alloc_context *data_ac) +void ocfs2_acl_init_release(struct ocfs2_acl_state *state) +{ + posix_acl_release(state->default_acl); + posix_acl_release(state->acl); + state->default_acl = NULL; + state->acl = NULL; +} + +int ocfs2_acl_init_prepare(struct inode *inode, struct inode *dir, + struct buffer_head *dir_bh, + struct ocfs2_acl_state *state) { struct ocfs2_super *osb = OCFS2_SB(inode->i_sb); - struct posix_acl *acl = NULL; - int ret = 0, ret2; - umode_t mode; - - if (!S_ISLNK(inode->i_mode)) { - if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) { - down_read(&OCFS2_I(dir)->ip_xattr_sem); - acl = ocfs2_get_acl_nolock(dir, ACL_TYPE_DEFAULT, - dir_bh); - up_read(&OCFS2_I(dir)->ip_xattr_sem); - if (IS_ERR(acl)) - return PTR_ERR(acl); + int ret = 0; + + state->default_acl = NULL; + state->acl = NULL; + state->mode = inode->i_mode; + + if (S_ISLNK(inode->i_mode)) + return 0; + + if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) { + down_read(&OCFS2_I(dir)->ip_xattr_sem); + state->default_acl = + ocfs2_get_acl_nolock(dir, ACL_TYPE_DEFAULT, dir_bh); + up_read(&OCFS2_I(dir)->ip_xattr_sem); + if (IS_ERR(state->default_acl)) { + ret = PTR_ERR(state->default_acl); + state->default_acl = NULL; + return ret; } - if (!acl) { - mode = inode->i_mode & ~current_umask(); - ret = ocfs2_acl_set_mode(inode, di_bh, handle, mode); - if (ret) { - mlog_errno(ret); + if (state->default_acl) { + state->acl = posix_acl_dup(state->default_acl); + if (!state->acl) { + ret = -ENOMEM; goto cleanup; } + ret = __posix_acl_create(&state->acl, GFP_NOFS, + &state->mode); + if (ret < 0) + goto cleanup; + if (ret == 0) { + posix_acl_release(state->acl); + state->acl = NULL; + } + if (!S_ISDIR(inode->i_mode)) { + posix_acl_release(state->default_acl); + state->default_acl = NULL; + } + } else { + state->mode &= ~current_umask(); } + } else { + state->mode &= ~current_umask(); } - if ((osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) && acl) { - if (S_ISDIR(inode->i_mode)) { + + return 0; +cleanup: + ocfs2_acl_init_release(state); + return ret; +} + +int ocfs2_init_acl(handle_t *handle, struct inode *inode, + struct buffer_head *di_bh, + struct ocfs2_alloc_context *meta_ac, + struct ocfs2_alloc_context *data_ac, + struct ocfs2_acl_state *state) +{ + struct ocfs2_super *osb = OCFS2_SB(inode->i_sb); + int ret = 0; + + if (S_ISLNK(inode->i_mode)) + return 0; + + if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) { + if (S_ISDIR(inode->i_mode) && state->default_acl) { ret = ocfs2_set_acl(handle, inode, di_bh, - ACL_TYPE_DEFAULT, acl, - meta_ac, data_ac); + ACL_TYPE_DEFAULT, + state->default_acl, meta_ac, + data_ac); if (ret) - goto cleanup; + return ret; } - mode = inode->i_mode; - ret = __posix_acl_create(&acl, GFP_NOFS, &mode); - if (ret < 0) - return ret; + } - ret2 = ocfs2_acl_set_mode(inode, di_bh, handle, mode); - if (ret2) { - mlog_errno(ret2); - ret = ret2; - goto cleanup; - } - if (ret > 0) { - ret = ocfs2_set_acl(handle, inode, - di_bh, ACL_TYPE_ACCESS, - acl, meta_ac, data_ac); + ret = ocfs2_acl_set_mode(inode, di_bh, handle, state->mode); + if (ret) { + mlog_errno(ret); + return ret; + } + + if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) { + if (state->acl) { + ret = ocfs2_set_acl(handle, inode, di_bh, + ACL_TYPE_ACCESS, state->acl, + meta_ac, data_ac); } } -cleanup: - posix_acl_release(acl); + return ret; } diff --git a/fs/ocfs2/acl.h b/fs/ocfs2/acl.h index 667c6f03fa60..a91f9ce278d6 100644 --- a/fs/ocfs2/acl.h +++ b/fs/ocfs2/acl.h @@ -20,9 +20,20 @@ struct posix_acl *ocfs2_iop_get_acl(struct inode *inode, int type, bool rcu); int ocfs2_iop_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, struct posix_acl *acl, int type); extern int ocfs2_acl_chmod(struct inode *, struct buffer_head *); -extern int ocfs2_init_acl(handle_t *, struct inode *, struct inode *, - struct buffer_head *, struct buffer_head *, - struct ocfs2_alloc_context *, - struct ocfs2_alloc_context *); +struct ocfs2_acl_state { + struct posix_acl *default_acl; + struct posix_acl *acl; + umode_t mode; +}; + +int ocfs2_acl_init_prepare(struct inode *inode, struct inode *dir, + struct buffer_head *dir_bh, + struct ocfs2_acl_state *state); +void ocfs2_acl_init_release(struct ocfs2_acl_state *state); +int ocfs2_init_acl(handle_t *handle, struct inode *inode, + struct buffer_head *di_bh, + struct ocfs2_alloc_context *meta_ac, + struct ocfs2_alloc_context *data_ac, + struct ocfs2_acl_state *state); #endif /* OCFS2_ACL_H */ diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index 798d76f479d0..38bf8fadb346 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -211,6 +211,7 @@ struct o2hb_region { /* protected by the hr_callback_sem */ struct task_struct *hr_task; + u8 hr_node_num; unsigned int hr_blocks; unsigned long long hr_start_block; @@ -358,12 +359,12 @@ static void o2hb_disarm_timeout(struct o2hb_region *reg) cancel_delayed_work_sync(®->hr_nego_timeout_work); } -static int o2hb_send_nego_msg(int key, int type, u8 target) +static int o2hb_send_nego_msg(int key, int type, u8 target, u8 node_num) { struct o2hb_nego_msg msg; int status, ret; - msg.node_num = o2nm_this_node(); + msg.node_num = node_num; again: ret = o2net_send_message(type, key, &msg, sizeof(msg), target, &status); @@ -381,8 +382,10 @@ static void o2hb_nego_timeout(struct work_struct *work) unsigned long live_node_bitmap[BITS_TO_LONGS(O2NM_MAX_NODES)]; int master_node, i, ret; struct o2hb_region *reg; + u8 node_num; reg = container_of(work, struct o2hb_region, hr_nego_timeout_work.work); + node_num = reg->hr_node_num; /* don't negotiate timeout if last hb failed since it is very * possible io failed. Should let write timeout fence self. */ @@ -393,10 +396,10 @@ static void o2hb_nego_timeout(struct work_struct *work) /* lowest node as master node to make negotiate decision. */ master_node = find_first_bit(live_node_bitmap, O2NM_MAX_NODES); - if (master_node == o2nm_this_node()) { + if (master_node == node_num) { if (!test_bit(master_node, reg->hr_nego_node_bitmap)) { printk(KERN_NOTICE "o2hb: node %d hb write hung for %ds on region %s (%pg).\n", - o2nm_this_node(), O2HB_NEGO_TIMEOUT_MS/1000, + node_num, O2HB_NEGO_TIMEOUT_MS / 1000, config_item_name(®->hr_item), reg_bdev(reg)); set_bit(master_node, reg->hr_nego_node_bitmap); } @@ -425,7 +428,7 @@ static void o2hb_nego_timeout(struct work_struct *work) mlog(ML_HEARTBEAT, "send NEGO_APPROVE msg to node %d\n", i); ret = o2hb_send_nego_msg(reg->hr_key, - O2HB_NEGO_APPROVE_MSG, i); + O2HB_NEGO_APPROVE_MSG, i, node_num); if (ret) mlog(ML_ERROR, "send NEGO_APPROVE msg to node %d fail %d\n", i, ret); @@ -433,10 +436,10 @@ static void o2hb_nego_timeout(struct work_struct *work) } else { /* negotiate timeout with master node. */ printk(KERN_NOTICE "o2hb: node %d hb write hung for %ds on region %s (%pg), negotiate timeout with node %d.\n", - o2nm_this_node(), O2HB_NEGO_TIMEOUT_MS/1000, config_item_name(®->hr_item), + node_num, O2HB_NEGO_TIMEOUT_MS / 1000, config_item_name(®->hr_item), reg_bdev(reg), master_node); ret = o2hb_send_nego_msg(reg->hr_key, O2HB_NEGO_TIMEOUT_MSG, - master_node); + master_node, node_num); if (ret) mlog(ML_ERROR, "send NEGO_TIMEOUT msg to node %d fail %d\n", master_node, ret); @@ -609,7 +612,9 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, o2hb_bio_wait_init(write_wc); - slot = o2nm_this_node(); + slot = reg->hr_node_num; + if (slot >= O2NM_MAX_NODES) + return -EINVAL; bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, REQ_OP_WRITE | REQ_SYNC); @@ -678,8 +683,12 @@ static int o2hb_check_own_slot(struct o2hb_region *reg) struct o2hb_disk_slot *slot; struct o2hb_disk_heartbeat_block *hb_block; char *errstr; + u8 node_num = reg->hr_node_num; + + if (node_num >= O2NM_MAX_NODES) + return 0; - slot = ®->hr_slots[o2nm_this_node()]; + slot = ®->hr_slots[node_num]; /* Don't check on our 1st timestamp */ if (!slot->ds_last_time) return 0; @@ -720,7 +729,10 @@ static inline void o2hb_prepare_block(struct o2hb_region *reg, struct o2hb_disk_slot *slot; struct o2hb_disk_heartbeat_block *hb_block; - node_num = o2nm_this_node(); + node_num = reg->hr_node_num; + if (node_num >= O2NM_MAX_NODES) + return; + slot = ®->hr_slots[node_num]; hb_block = (struct o2hb_disk_heartbeat_block *)slot->ds_raw_block; @@ -1214,7 +1226,7 @@ static int o2hb_thread(void *data) set_user_nice(current, MIN_NICE); /* Pin node */ - ret = o2nm_depend_this_node(); + ret = o2nm_depend_node(reg->hr_node_num); if (ret) { mlog(ML_ERROR, "Node has been deleted, ret = %d\n", ret); reg->hr_node_deleted = 1; @@ -1223,7 +1235,8 @@ static int o2hb_thread(void *data) } while (!kthread_should_stop() && - !reg->hr_unclean_stop && !reg->hr_aborted_start) { + !reg->hr_unclean_stop && !reg->hr_aborted_start && + o2nm_this_node() == reg->hr_node_num) { /* We track the time spent inside * o2hb_do_disk_heartbeat so that we avoid more than * hr_timeout_ms between disk writes. On busy systems @@ -1272,7 +1285,7 @@ static int o2hb_thread(void *data) } /* Unpin node */ - o2nm_undepend_this_node(); + o2nm_undepend_node(reg->hr_node_num); mlog(ML_HEARTBEAT|ML_KTHREAD, "o2hb thread exiting\n"); @@ -1464,13 +1477,38 @@ void o2hb_init(void) o2hb_debug_init(); } -/* if we're already in a callback then we're already serialized by the sem */ -static void o2hb_fill_node_map_from_callback(unsigned long *map, - unsigned int bits) +static void __o2hb_fill_node_map(unsigned long *map, unsigned int bits) { bitmap_copy(map, o2hb_live_node_bitmap, bits); } +void o2hb_callback_read_lock(void) +{ + down_read(&o2hb_callback_sem); +} + +void o2hb_callback_read_unlock(void) +{ + up_read(&o2hb_callback_sem); +} + +void o2hb_synchronize_callbacks(void) +{ + down_write(&o2hb_callback_sem); + up_write(&o2hb_callback_sem); +} + +/* + * Callers must already hold o2hb_callback_sem for read or write so the copy + * stays serialized with callback delivery. + */ +void o2hb_fill_node_map_locked(unsigned long *map, unsigned int bits) +{ + spin_lock(&o2hb_live_lock); + __o2hb_fill_node_map(map, bits); + spin_unlock(&o2hb_live_lock); +} + /* * get a map of all nodes that are heartbeating in any regions */ @@ -1478,11 +1516,9 @@ void o2hb_fill_node_map(unsigned long *map, unsigned int bits) { /* callers want to serialize this map and callbacks so that they * can trust that they don't miss nodes coming to the party */ - down_read(&o2hb_callback_sem); - spin_lock(&o2hb_live_lock); - o2hb_fill_node_map_from_callback(map, bits); - spin_unlock(&o2hb_live_lock); - up_read(&o2hb_callback_sem); + o2hb_callback_read_lock(); + o2hb_fill_node_map_locked(map, bits); + o2hb_callback_read_unlock(); } EXPORT_SYMBOL_GPL(o2hb_fill_node_map); @@ -1784,7 +1820,8 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, /* We can't heartbeat without having had our node number * configured yet. */ - if (o2nm_this_node() == O2NM_MAX_NODES) + reg->hr_node_num = o2nm_this_node(); + if (reg->hr_node_num == O2NM_MAX_NODES) return -EINVAL; ret = kstrtol(p, 0, &fd); @@ -2017,6 +2054,7 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g ret = -ENAMETOOLONG; goto free; } + reg->hr_node_num = O2NM_MAX_NODES; spin_lock(&o2hb_live_lock); reg->hr_region_num = 0; @@ -2552,7 +2590,7 @@ int o2hb_check_node_heartbeating_no_sem(u8 node_num) unsigned long testing_map[BITS_TO_LONGS(O2NM_MAX_NODES)]; spin_lock(&o2hb_live_lock); - o2hb_fill_node_map_from_callback(testing_map, O2NM_MAX_NODES); + __o2hb_fill_node_map(testing_map, O2NM_MAX_NODES); spin_unlock(&o2hb_live_lock); if (!test_bit(node_num, testing_map)) { mlog(ML_HEARTBEAT, @@ -2569,7 +2607,7 @@ int o2hb_check_node_heartbeating_from_callback(u8 node_num) { unsigned long testing_map[BITS_TO_LONGS(O2NM_MAX_NODES)]; - o2hb_fill_node_map_from_callback(testing_map, O2NM_MAX_NODES); + o2hb_fill_node_map_locked(testing_map, O2NM_MAX_NODES); if (!test_bit(node_num, testing_map)) { mlog(ML_HEARTBEAT, "node (%u) does not have heartbeating enabled.\n", diff --git a/fs/ocfs2/cluster/heartbeat.h b/fs/ocfs2/cluster/heartbeat.h index 8ef8c1b9eeb7..2ca2b657583c 100644 --- a/fs/ocfs2/cluster/heartbeat.h +++ b/fs/ocfs2/cluster/heartbeat.h @@ -58,6 +58,11 @@ int o2hb_register_callback(const char *region_uuid, struct o2hb_callback_func *hc); void o2hb_unregister_callback(const char *region_uuid, struct o2hb_callback_func *hc); +void o2hb_callback_read_lock(void); +void o2hb_callback_read_unlock(void); +void o2hb_synchronize_callbacks(void); +void o2hb_fill_node_map_locked(unsigned long *map, + unsigned int bits); void o2hb_fill_node_map(unsigned long *map, unsigned int bits); void o2hb_exit(void); diff --git a/fs/ocfs2/cluster/nodemanager.c b/fs/ocfs2/cluster/nodemanager.c index 5fffbed779da..b65fc6eeae6e 100644 --- a/fs/ocfs2/cluster/nodemanager.c +++ b/fs/ocfs2/cluster/nodemanager.c @@ -325,6 +325,7 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page, struct o2nm_node *node = to_o2nm_node(item); struct o2nm_cluster *cluster; unsigned long tmp; + bool starting = false; char *p = (char *)page; ssize_t ret; @@ -361,11 +362,13 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page, ret = o2net_start_listening(node); if (ret) goto out; + starting = true; } if (!tmp && cluster->cl_has_local && cluster->cl_local_node == node->nd_num) { o2net_stop_listening(node); + cluster->cl_has_local = 0; cluster->cl_local_node = O2NM_INVALID_NODE_NUM; } @@ -373,6 +376,8 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page, if (node->nd_local) { cluster->cl_has_local = tmp; cluster->cl_local_node = node->nd_num; + if (starting) + o2net_complete_start_listening(node); } ret = count; @@ -787,12 +792,12 @@ void o2nm_undepend_item(struct config_item *item) configfs_undepend_item(item); } -int o2nm_depend_this_node(void) +int o2nm_depend_node(u8 node_num) { int ret = 0; struct o2nm_node *local_node; - local_node = o2nm_get_node_by_num(o2nm_this_node()); + local_node = o2nm_get_node_by_num(node_num); if (!local_node) { ret = -EINVAL; goto out; @@ -805,17 +810,27 @@ int o2nm_depend_this_node(void) return ret; } -void o2nm_undepend_this_node(void) +void o2nm_undepend_node(u8 node_num) { struct o2nm_node *local_node; - local_node = o2nm_get_node_by_num(o2nm_this_node()); + local_node = o2nm_get_node_by_num(node_num); BUG_ON(!local_node); o2nm_undepend_item(&local_node->nd_item); o2nm_node_put(local_node); } +int o2nm_depend_this_node(void) +{ + return o2nm_depend_node(o2nm_this_node()); +} + +void o2nm_undepend_this_node(void) +{ + o2nm_undepend_node(o2nm_this_node()); +} + static void __exit exit_o2nm(void) { diff --git a/fs/ocfs2/cluster/nodemanager.h b/fs/ocfs2/cluster/nodemanager.h index 2f72f56996bd..ca3483fb5450 100644 --- a/fs/ocfs2/cluster/nodemanager.h +++ b/fs/ocfs2/cluster/nodemanager.h @@ -66,6 +66,8 @@ void o2nm_node_put(struct o2nm_node *node); int o2nm_depend_item(struct config_item *item); int o2nm_depend_item_unlocked(struct config_item *item); void o2nm_undepend_item(struct config_item *item); +int o2nm_depend_node(u8 node_num); +void o2nm_undepend_node(u8 node_num); int o2nm_depend_this_node(void); void o2nm_undepend_this_node(void); diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c index b05d4e9d13b2..822553530e74 100644 --- a/fs/ocfs2/cluster/tcp.c +++ b/fs/ocfs2/cluster/tcp.c @@ -105,6 +105,8 @@ static struct socket *o2net_listen_sock; * destroying the work queue. */ static struct workqueue_struct *o2net_wq; +/* Heartbeat callbacks stay registered across local-node off/on. */ +static bool o2net_listening; static struct work_struct o2net_listen_work; static struct o2hb_callback_func o2net_hb_up, o2net_hb_down; @@ -1692,6 +1694,19 @@ static void o2net_still_up(struct work_struct *work) /* ------------------------------------------------------------ */ +static void o2net_hb_node_up(struct o2net_node *nn) +{ + /* ensure an immediate connect attempt */ + nn->nn_last_connect_attempt = jiffies - + (msecs_to_jiffies(o2net_reconnect_delay()) + 1); + + spin_lock(&nn->nn_lock); + atomic_set(&nn->nn_timeout, 0); + if (nn->nn_persistent_error) + o2net_set_nn_state(nn, NULL, 0, 0); + spin_unlock(&nn->nn_lock); +} + void o2net_disconnect_node(struct o2nm_node *node) { struct o2net_node *nn = o2net_nn_from_num(node->nd_num); @@ -1713,41 +1728,39 @@ void o2net_disconnect_node(struct o2nm_node *node) static void o2net_hb_node_down_cb(struct o2nm_node *node, int node_num, void *data) { + u8 this_node; + o2quo_hb_down(node_num); if (!node) - return; + goto out; - if (node_num != o2nm_this_node()) + this_node = o2nm_this_node(); + if (!READ_ONCE(o2net_listening) || this_node == O2NM_MAX_NODES) + goto out; + + if (node_num != this_node) o2net_disconnect_node(node); +out: BUG_ON(atomic_read(&o2net_connected_peers) < 0); } static void o2net_hb_node_up_cb(struct o2nm_node *node, int node_num, void *data) { - struct o2net_node *nn = o2net_nn_from_num(node_num); + u8 this_node; o2quo_hb_up(node_num); BUG_ON(!node); - /* ensure an immediate connect attempt */ - nn->nn_last_connect_attempt = jiffies - - (msecs_to_jiffies(o2net_reconnect_delay()) + 1); + this_node = o2nm_this_node(); + if (!READ_ONCE(o2net_listening) || this_node == O2NM_MAX_NODES) + return; - if (node_num != o2nm_this_node()) { - /* believe it or not, accept and node heartbeating testing - * can succeed for this node before we got here.. so - * only use set_nn_state to clear the persistent error - * if that hasn't already happened */ - spin_lock(&nn->nn_lock); - atomic_set(&nn->nn_timeout, 0); - if (nn->nn_persistent_error) - o2net_set_nn_state(nn, NULL, 0, 0); - spin_unlock(&nn->nn_lock); - } + if (node_num != this_node) + o2net_hb_node_up(o2net_nn_from_num(node_num)); } void o2net_unregister_hb_callbacks(void) @@ -1756,6 +1769,37 @@ void o2net_unregister_hb_callbacks(void) o2hb_unregister_callback(NULL, &o2net_hb_down); } +/* + * Delay heartbeat-driven network work until the local node is fully published + * through o2nm_this_node(), then replay the nodes that are already live while + * callback delivery stays blocked. + */ +void o2net_complete_start_listening(struct o2nm_node *node) +{ + unsigned long live_nodes[BITS_TO_LONGS(O2NM_MAX_NODES)]; + unsigned long node_num; + u8 local_node; + + local_node = o2nm_this_node(); + if (WARN_ON_ONCE(local_node == O2NM_MAX_NODES)) + return; + if (WARN_ON_ONCE(local_node != node->nd_num)) + return; + if (WARN_ON_ONCE(!o2net_wq)) + return; + + o2hb_callback_read_lock(); + WRITE_ONCE(o2net_listening, true); + o2hb_fill_node_map_locked(live_nodes, O2NM_MAX_NODES); + for_each_set_bit(node_num, live_nodes, O2NM_MAX_NODES) { + if (node_num == local_node) + continue; + + o2net_hb_node_up(o2net_nn_from_num(node_num)); + } + o2hb_callback_read_unlock(); +} + int o2net_register_hb_callbacks(void) { int ret; @@ -2034,6 +2078,8 @@ int o2net_start_listening(struct o2nm_node *node) { int ret = 0; + if (WARN_ON_ONCE(READ_ONCE(o2net_listening))) + return -EBUSY; BUG_ON(o2net_wq != NULL); BUG_ON(o2net_listen_sock != NULL); @@ -2065,6 +2111,9 @@ void o2net_stop_listening(struct o2nm_node *node) BUG_ON(o2net_wq == NULL); BUG_ON(o2net_listen_sock == NULL); + WRITE_ONCE(o2net_listening, false); + o2hb_synchronize_callbacks(); + /* stop the listening socket from generating work */ write_lock_bh(&sock->sk->sk_callback_lock); sock->sk->sk_data_ready = sock->sk->sk_user_data; diff --git a/fs/ocfs2/cluster/tcp.h b/fs/ocfs2/cluster/tcp.h index a75b551d31c7..2e86d42b5faf 100644 --- a/fs/ocfs2/cluster/tcp.h +++ b/fs/ocfs2/cluster/tcp.h @@ -96,6 +96,7 @@ struct o2nm_node; int o2net_register_hb_callbacks(void); void o2net_unregister_hb_callbacks(void); int o2net_start_listening(struct o2nm_node *node); +void o2net_complete_start_listening(struct o2nm_node *node); void o2net_stop_listening(struct o2nm_node *node); void o2net_disconnect_node(struct o2nm_node *node); int o2net_num_connected_peers(void); diff --git a/fs/ocfs2/inode.c b/fs/ocfs2/inode.c index d247d69349e4..9ec3a7657b5b 100644 --- a/fs/ocfs2/inode.c +++ b/fs/ocfs2/inode.c @@ -1538,6 +1538,22 @@ int ocfs2_validate_inode_block(struct super_block *sb, goto bail; } + if ((le32_to_cpu(di->i_flags) & OCFS2_ORPHANED_FL) && + le16_to_cpu(di->i_orphaned_slot) >= OCFS2_SB(sb)->max_slots) { + rc = ocfs2_error(sb, "Invalid dinode %llu: orphaned slot %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(di->i_orphaned_slot)); + goto bail; + } + + if ((le32_to_cpu(di->i_flags) & OCFS2_DIO_ORPHANED_FL) && + le16_to_cpu(di->i_dio_orphaned_slot) >= OCFS2_SB(sb)->max_slots) { + rc = ocfs2_error(sb, "Invalid dinode %llu: DIO orphaned slot %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(di->i_dio_orphaned_slot)); + goto bail; + } + /* * Reject dinodes whose i_mode does not name one of the seven * canonical POSIX file types. ocfs2_populate_inode() copies diff --git a/fs/ocfs2/namei.c b/fs/ocfs2/namei.c index c90b254da75e..20276ad6e17e 100644 --- a/fs/ocfs2/namei.c +++ b/fs/ocfs2/namei.c @@ -255,6 +255,7 @@ static int ocfs2_mknod(struct mnt_idmap *idmap, sigset_t oldset; int did_block_signals = 0; struct ocfs2_dentry_lock *dl = NULL; + struct ocfs2_acl_state acl_state = { 0 }; trace_ocfs2_mknod(dir, dentry, dentry->d_name.len, dentry->d_name.name, (unsigned long long)OCFS2_I(dir)->ip_blkno, @@ -329,10 +330,14 @@ static int ocfs2_mknod(struct mnt_idmap *idmap, } } + status = ocfs2_acl_init_prepare(inode, dir, parent_fe_bh, &acl_state); + if (status < 0) + goto leave; + /* calculate meta data/clusters for setting security and acl xattr */ - status = ocfs2_calc_xattr_init(dir, parent_fe_bh, mode, - &si, &want_clusters, - &xattr_credits, &want_meta); + status = ocfs2_calc_xattr_init(dir, mode, &si, &want_clusters, + &xattr_credits, &want_meta, + &acl_state); if (status < 0) { mlog_errno(status); goto leave; @@ -410,8 +415,8 @@ static int ocfs2_mknod(struct mnt_idmap *idmap, inc_nlink(dir); } - status = ocfs2_init_acl(handle, inode, dir, new_fe_bh, parent_fe_bh, - meta_ac, data_ac); + status = ocfs2_init_acl(handle, inode, new_fe_bh, meta_ac, data_ac, + &acl_state); if (status < 0) { mlog_errno(status); @@ -476,6 +481,8 @@ static int ocfs2_mknod(struct mnt_idmap *idmap, brelse(parent_fe_bh); kfree(si.value); + ocfs2_acl_init_release(&acl_state); + ocfs2_free_dir_lookup_result(&lookup); if (inode_ac) diff --git a/fs/ocfs2/xattr.c b/fs/ocfs2/xattr.c index cd9c8d4dd9f7..3a796d9159d8 100644 --- a/fs/ocfs2/xattr.c +++ b/fs/ocfs2/xattr.c @@ -607,13 +607,10 @@ int ocfs2_calc_security_init(struct inode *dir, return ret; } -int ocfs2_calc_xattr_init(struct inode *dir, - struct buffer_head *dir_bh, - umode_t mode, +int ocfs2_calc_xattr_init(struct inode *dir, umode_t mode, struct ocfs2_security_xattr_info *si, - int *want_clusters, - int *xattr_credits, - int *want_meta) + int *want_clusters, int *xattr_credits, + int *want_meta, struct ocfs2_acl_state *acl_state) { int ret = 0; struct ocfs2_super *osb = OCFS2_SB(dir->i_sb); @@ -624,19 +621,15 @@ int ocfs2_calc_xattr_init(struct inode *dir, si->value_len); if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) { - down_read(&OCFS2_I(dir)->ip_xattr_sem); - acl_len = ocfs2_xattr_get_nolock(dir, dir_bh, - OCFS2_XATTR_INDEX_POSIX_ACL_DEFAULT, - "", NULL, 0); - up_read(&OCFS2_I(dir)->ip_xattr_sem); - if (acl_len > 0) { - a_size = ocfs2_xattr_entry_real_size(0, acl_len); - if (S_ISDIR(mode)) - a_size <<= 1; - } else if (acl_len != 0 && acl_len != -ENODATA) { - ret = acl_len; - mlog_errno(ret); - return ret; + if (acl_state->default_acl && S_ISDIR(mode)) { + acl_len = acl_state->default_acl->a_count * + sizeof(struct ocfs2_acl_entry); + a_size += ocfs2_xattr_entry_real_size(0, acl_len); + } + if (acl_state->acl) { + acl_len = acl_state->acl->a_count * + sizeof(struct ocfs2_acl_entry); + a_size += ocfs2_xattr_entry_real_size(0, acl_len); } } @@ -679,14 +672,33 @@ int ocfs2_calc_xattr_init(struct inode *dir, new_clusters); *want_clusters += new_clusters; } - if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL && - acl_len > OCFS2_XATTR_INLINE_SIZE) { - /* for directory, it has DEFAULT and ACCESS two types of acls */ - new_clusters = (S_ISDIR(mode) ? 2 : 1) * - ocfs2_clusters_for_bytes(dir->i_sb, acl_len); - *xattr_credits += ocfs2_clusters_to_blocks(dir->i_sb, - new_clusters); - *want_clusters += new_clusters; + if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) { + if (acl_state->default_acl && S_ISDIR(mode)) { + acl_len = acl_state->default_acl->a_count * + sizeof(struct ocfs2_acl_entry); + if (acl_len > OCFS2_XATTR_INLINE_SIZE) { + new_clusters = + ocfs2_clusters_for_bytes(dir->i_sb, + acl_len); + *xattr_credits += + ocfs2_clusters_to_blocks(dir->i_sb, + new_clusters); + *want_clusters += new_clusters; + } + } + if (acl_state->acl) { + acl_len = acl_state->acl->a_count * + sizeof(struct ocfs2_acl_entry); + if (acl_len > OCFS2_XATTR_INLINE_SIZE) { + new_clusters = + ocfs2_clusters_for_bytes(dir->i_sb, + acl_len); + *xattr_credits += + ocfs2_clusters_to_blocks(dir->i_sb, + new_clusters); + *want_clusters += new_clusters; + } + } } return ret; @@ -3445,9 +3457,10 @@ static int __ocfs2_xattr_set_handle(struct inode *inode, } /* - * This function only called duing creating inode - * for init security/acl xattrs of the new inode. - * All transanction credits have been reserved in mknod. + * This helper is only for setting initial ACL or security xattrs on an inode + * that is still unpublished, unhashed, and unattached to a dentry. + * Ordinary xattr updates must use ocfs2_xattr_set(). + * All transaction credits have been reserved in mknod or symlink callers. */ int ocfs2_xattr_set_handle(handle_t *handle, struct inode *inode, @@ -3504,8 +3517,6 @@ int ocfs2_xattr_set_handle(handle_t *handle, xis.inode_bh = xbs.inode_bh = di_bh; di = (struct ocfs2_dinode *)di_bh->b_data; - down_write(&OCFS2_I(inode)->ip_xattr_sem); - ret = ocfs2_xattr_ibody_find(inode, name_index, name, &xis); if (ret) goto cleanup; @@ -3518,7 +3529,6 @@ int ocfs2_xattr_set_handle(handle_t *handle, ret = __ocfs2_xattr_set_handle(inode, di, &xi, &xis, &xbs, &ctxt); cleanup: - up_write(&OCFS2_I(inode)->ip_xattr_sem); brelse(xbs.xattr_bh); ocfs2_xattr_bucket_free(xbs.bucket); @@ -7208,6 +7218,7 @@ int ocfs2_init_security_and_acl(struct inode *dir, { int ret = 0; struct buffer_head *dir_bh = NULL; + struct ocfs2_acl_state acl_state = { 0 }; ret = ocfs2_init_security_get(inode, dir, qstr, NULL); if (ret) { @@ -7220,10 +7231,17 @@ int ocfs2_init_security_and_acl(struct inode *dir, mlog_errno(ret); goto leave; } - ret = ocfs2_init_acl(NULL, inode, dir, NULL, dir_bh, NULL, NULL); + + ret = ocfs2_acl_init_prepare(inode, dir, dir_bh, &acl_state); + if (ret) + goto unlock; + + ret = ocfs2_init_acl(NULL, inode, NULL, NULL, NULL, &acl_state); if (ret) mlog_errno(ret); +unlock: + ocfs2_acl_init_release(&acl_state); ocfs2_inode_unlock(dir, 0); brelse(dir_bh); leave: diff --git a/fs/ocfs2/xattr.h b/fs/ocfs2/xattr.h index 65e9aa743919..5cdd6c6b4006 100644 --- a/fs/ocfs2/xattr.h +++ b/fs/ocfs2/xattr.h @@ -55,9 +55,12 @@ int ocfs2_init_security_set(handle_t *, struct inode *, int ocfs2_calc_security_init(struct inode *, struct ocfs2_security_xattr_info *, int *, int *, struct ocfs2_alloc_context **); -int ocfs2_calc_xattr_init(struct inode *, struct buffer_head *, - umode_t, struct ocfs2_security_xattr_info *, - int *, int *, int *); + +struct ocfs2_acl_state; +int ocfs2_calc_xattr_init(struct inode *dir, umode_t mode, + struct ocfs2_security_xattr_info *si, + int *want_clusters, int *xattr_credits, + int *want_meta, struct ocfs2_acl_state *acl_state); /* * xattrs can live inside an inode, as part of an external xattr block, diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c index 08e26c0c9fb9..1f81bd671155 100644 --- a/fs/resctrl/rdtgroup.c +++ b/fs/resctrl/rdtgroup.c @@ -2528,15 +2528,24 @@ static void rdtgroup_kn_get(struct rdtgroup *rdtgrp, struct kernfs_node *kn) static void rdtgroup_kn_put(struct rdtgroup *rdtgrp, struct kernfs_node *kn) { - if (atomic_dec_and_test(&rdtgrp->waitcount) && - (rdtgrp->flags & RDT_DELETED)) { + bool needs_free; + + if (!atomic_dec_and_mutex_lock(&rdtgrp->waitcount, &rdtgroup_mutex)) { + kernfs_unbreak_active_protection(kn); + return; + } + + needs_free = rdtgrp->flags & RDT_DELETED; + + mutex_unlock(&rdtgroup_mutex); + + kernfs_unbreak_active_protection(kn); + + if (needs_free) { if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP || rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) rdtgroup_pseudo_lock_remove(rdtgrp); - kernfs_unbreak_active_protection(kn); rdtgroup_remove(rdtgrp); - } else { - kernfs_unbreak_active_protection(kn); } } diff --git a/fs/smb/client/cifs_debug.c b/fs/smb/client/cifs_debug.c index 08538ad0f8dc..3045db555c3c 100644 --- a/fs/smb/client/cifs_debug.c +++ b/fs/smb/client/cifs_debug.c @@ -776,7 +776,7 @@ static ssize_t cifs_stats_proc_write(struct file *file, atomic_set(&server->smb2slowcmd[i], 0); server->time_per_cmd[i] = 0; server->slowest_cmd[i] = 0; - server->fastest_cmd[0] = 0; + server->fastest_cmd[i] = 0; } #endif /* CONFIG_CIFS_STATS2 */ list_for_each_entry(ses, &server->smb_ses_list, smb_ses_list) { diff --git a/fs/smb/client/inode.c b/fs/smb/client/inode.c index 72d2052a22d4..f23c36d6ae6b 100644 --- a/fs/smb/client/inode.c +++ b/fs/smb/client/inode.c @@ -3278,13 +3278,6 @@ cifs_setattr_unix(struct dentry *direntry, struct iattr *attrs) if (rc) goto out; - if ((attrs->ia_valid & ATTR_SIZE) && - attrs->ia_size != i_size_read(inode)) { - truncate_setsize(inode, attrs->ia_size); - netfs_resize_file(&cifsInode->netfs, attrs->ia_size, true); - fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size); - } - setattr_copy(&nop_mnt_idmap, inode, attrs); mark_inode_dirty(inode); @@ -3496,13 +3489,6 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs) if (rc) goto cifs_setattr_exit; - if ((attrs->ia_valid & ATTR_SIZE) && - attrs->ia_size != i_size_read(inode)) { - truncate_setsize(inode, attrs->ia_size); - netfs_resize_file(&cifsInode->netfs, attrs->ia_size, true); - fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size); - } - setattr_copy(&nop_mnt_idmap, inode, attrs); mark_inode_dirty(inode); diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c index 5aca5b4a9868..46580d6eab62 100644 --- a/fs/smb/client/smb2pdu.c +++ b/fs/smb/client/smb2pdu.c @@ -4454,8 +4454,10 @@ smb2_new_read_req(void **buf, unsigned int *total_len, if (rc) return rc; - if (server == NULL) - return -ECONNABORTED; + if (!server) { + rc = -ECONNABORTED; + goto free_req; + } shdr = &req->hdr; shdr->Id.SyncId.ProcessId = cpu_to_le32(io_parms->pid); @@ -4486,8 +4488,10 @@ smb2_new_read_req(void **buf, unsigned int *total_len, rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter, true, need_invalidate); - if (!rdata->mr) - return -EAGAIN; + if (!rdata->mr) { + rc = -EAGAIN; + goto free_req; + } req->Channel = SMB2_CHANNEL_RDMA_V1_INVALIDATE; if (need_invalidate) @@ -4530,6 +4534,10 @@ smb2_new_read_req(void **buf, unsigned int *total_len, *buf = req; return rc; + +free_req: + cifs_small_buf_release(req); + return rc; } static void diff --git a/fs/smb/server/Kconfig b/fs/smb/server/Kconfig index 12594879cb64..8b91a6323d93 100644 --- a/fs/smb/server/Kconfig +++ b/fs/smb/server/Kconfig @@ -9,7 +9,6 @@ config SMB_SERVER select CRYPTO select CRYPTO_ECB select CRYPTO_LIB_ARC4 - select CRYPTO_LIB_DES select CRYPTO_LIB_MD5 select CRYPTO_LIB_SHA256 select CRYPTO_LIB_SHA512 diff --git a/fs/smb/server/auth.c b/fs/smb/server/auth.c index 3f8258954901..9844aaad7ab3 100644 --- a/fs/smb/server/auth.c +++ b/fs/smb/server/auth.c @@ -24,7 +24,6 @@ #include #include -#include #include "server.h" #include "smb_common.h" @@ -436,6 +435,7 @@ int ksmbd_krb5_authenticate(struct ksmbd_session *sess, char *in_blob, resp_ext = ksmbd_ipc_login_request_ext(resp->login_response.account); user = ksmbd_alloc_user(&resp->login_response, resp_ext); + kvfree(resp_ext); if (!user) { ksmbd_debug(AUTH, "login failure\n"); retval = -ENOMEM; diff --git a/fs/smb/server/connection.c b/fs/smb/server/connection.c index c193b87cfa75..ff6cdd720dd4 100644 --- a/fs/smb/server/connection.c +++ b/fs/smb/server/connection.c @@ -11,6 +11,7 @@ #include "server.h" #include "smb_common.h" #include "mgmt/ksmbd_ida.h" +#include "mgmt/user_session.h" #include "connection.h" #include "transport_tcp.h" #include "transport_rdma.h" @@ -123,6 +124,7 @@ void ksmbd_conn_free(struct ksmbd_conn *conn) kvfree(conn->request_buf); kfree(conn->preauth_info); kfree(conn->mechToken); + ksmbd_preauth_session_destroy(conn); ksmbd_conn_put(conn); } @@ -161,6 +163,7 @@ struct ksmbd_conn *ksmbd_conn_alloc(void) init_waitqueue_head(&conn->r_count_q); INIT_LIST_HEAD(&conn->requests); INIT_LIST_HEAD(&conn->async_requests); + INIT_LIST_HEAD(&conn->preauth_sess_table); spin_lock_init(&conn->request_lock); spin_lock_init(&conn->credits_lock); ida_init(&conn->async_ida); @@ -252,6 +255,22 @@ void ksmbd_all_conn_set_status(u64 sess_id, u32 status) up_read(&conn_list_lock); } +void ksmbd_conn_abort(struct ksmbd_conn *conn) +{ + bool shutdown = false; + + spin_lock(&conn->request_lock); + if (!ksmbd_conn_exiting(conn) && !ksmbd_conn_releasing(conn)) { + ksmbd_conn_set_exiting(conn); + shutdown = true; + } + spin_unlock(&conn->request_lock); + wake_up_all(&conn->req_running_q); + + if (shutdown && conn->transport->ops->shutdown) + conn->transport->ops->shutdown(conn->transport); +} + void ksmbd_conn_wait_idle(struct ksmbd_conn *conn) { wait_event(conn->req_running_q, atomic_read(&conn->req_running) < 2); diff --git a/fs/smb/server/connection.h b/fs/smb/server/connection.h index f8f88651166b..ceb1c1aefbf6 100644 --- a/fs/smb/server/connection.h +++ b/fs/smb/server/connection.h @@ -168,6 +168,7 @@ struct ksmbd_conn *ksmbd_conn_alloc(void); void ksmbd_conn_free(struct ksmbd_conn *conn); struct ksmbd_conn *ksmbd_conn_get(struct ksmbd_conn *conn); void ksmbd_conn_put(struct ksmbd_conn *conn); +void ksmbd_conn_abort(struct ksmbd_conn *conn); int ksmbd_conn_wq_init(void); void ksmbd_conn_wq_destroy(void); bool ksmbd_conn_lookup_dialect(struct ksmbd_conn *c); diff --git a/fs/smb/server/mgmt/share_config.c b/fs/smb/server/mgmt/share_config.c index c9b1108d6e96..1e90fd30bd77 100644 --- a/fs/smb/server/mgmt/share_config.c +++ b/fs/smb/server/mgmt/share_config.c @@ -158,6 +158,11 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work, atomic_set(&share->refcount, 1); INIT_LIST_HEAD(&share->veto_list); share->name = kstrdup(name, KSMBD_DEFAULT_GFP); + if (!share->name) { + kill_share(share); + share = NULL; + goto out; + } if (!test_share_config_flag(share, KSMBD_SHARE_FLAG_PIPE)) { int path_len = PATH_MAX; @@ -199,7 +204,7 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work, share->path = NULL; } } - if (ret || !share->name) { + if (ret) { kill_share(share); share = NULL; goto out; diff --git a/fs/smb/server/mgmt/user_config.c b/fs/smb/server/mgmt/user_config.c index d051ac3bc831..53db14e91bfd 100644 --- a/fs/smb/server/mgmt/user_config.c +++ b/fs/smb/server/mgmt/user_config.c @@ -26,6 +26,7 @@ struct ksmbd_user *ksmbd_login_user(const char *account) resp_ext = ksmbd_ipc_login_request_ext(account); user = ksmbd_alloc_user(resp, resp_ext); + kvfree(resp_ext); out: kvfree(resp); return user; diff --git a/fs/smb/server/mgmt/user_session.c b/fs/smb/server/mgmt/user_session.c index 5bc2f18d68bb..e81591566521 100644 --- a/fs/smb/server/mgmt/user_session.c +++ b/fs/smb/server/mgmt/user_session.c @@ -212,10 +212,21 @@ static void ksmbd_expire_session(struct ksmbd_conn *conn) int ksmbd_session_register(struct ksmbd_conn *conn, struct ksmbd_session *sess) { + int ret; + sess->dialect = conn->dialect; memcpy(sess->ClientGUID, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE); ksmbd_expire_session(conn); - return xa_err(xa_store(&conn->sessions, sess->id, sess, KSMBD_DEFAULT_GFP)); + ret = xa_err(xa_store(&conn->sessions, sess->id, sess, + KSMBD_DEFAULT_GFP)); + if (ret) { + down_write(&sessions_table_lock); + hash_del(&sess->hlist); + up_write(&sessions_table_lock); + ksmbd_user_session_put(sess); + } + + return ret; } static int ksmbd_chann_del(struct ksmbd_conn *conn, struct ksmbd_session *sess) @@ -374,6 +385,17 @@ struct preauth_session *ksmbd_preauth_session_alloc(struct ksmbd_conn *conn, return sess; } +void ksmbd_preauth_session_destroy(struct ksmbd_conn *conn) +{ + struct preauth_session *sess, *tmp; + + list_for_each_entry_safe(sess, tmp, &conn->preauth_sess_table, + preauth_entry) { + list_del(&sess->preauth_entry); + kfree(sess); + } +} + void destroy_previous_session(struct ksmbd_conn *conn, struct ksmbd_user *user, u64 id) { diff --git a/fs/smb/server/mgmt/user_session.h b/fs/smb/server/mgmt/user_session.h index cba7f688f6b5..525650d6876c 100644 --- a/fs/smb/server/mgmt/user_session.h +++ b/fs/smb/server/mgmt/user_session.h @@ -101,6 +101,7 @@ void destroy_previous_session(struct ksmbd_conn *conn, struct ksmbd_user *user, u64 id); struct preauth_session *ksmbd_preauth_session_alloc(struct ksmbd_conn *conn, u64 sess_id); +void ksmbd_preauth_session_destroy(struct ksmbd_conn *conn); struct preauth_session *ksmbd_preauth_session_lookup(struct ksmbd_conn *conn, unsigned long long id); diff --git a/fs/smb/server/server.c b/fs/smb/server/server.c index 9da4525010cc..76515d9777e3 100644 --- a/fs/smb/server/server.c +++ b/fs/smb/server/server.c @@ -170,8 +170,10 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, if (conn->ops->is_transform_hdr && conn->ops->is_transform_hdr(work->request_buf)) { rc = conn->ops->decrypt_req(work); - if (rc < 0) + if (rc < 0) { + ksmbd_conn_abort(conn); return; + } work->encrypted = true; } diff --git a/fs/smb/server/smb2ops.c b/fs/smb/server/smb2ops.c index bcf05caa2304..4c50b7f57f76 100644 --- a/fs/smb/server/smb2ops.c +++ b/fs/smb/server/smb2ops.c @@ -262,8 +262,10 @@ void init_smb3_02_server(struct ksmbd_conn *conn) if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB3_MULTICHANNEL) conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_MULTI_CHANNEL; - if (server_conf.flags & KSMBD_GLOBAL_FLAG_DURABLE_HANDLE) - conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_PERSISTENT_HANDLES; + /* + * Durable handles are in-memory only. Do not advertise persistent + * handles until CA recovery and fencing are implemented. + */ } /** @@ -286,10 +288,7 @@ int init_smb3_11_server(struct ksmbd_conn *conn) if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB3_MULTICHANNEL) conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_MULTI_CHANNEL; - if (server_conf.flags & KSMBD_GLOBAL_FLAG_DURABLE_HANDLE) - conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_PERSISTENT_HANDLES; - - INIT_LIST_HEAD(&conn->preauth_sess_table); + /* See init_smb3_02_server(): persistent handles require CA recovery. */ return 0; } diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c index e532ee89e327..89febfb8fc90 100644 --- a/fs/smb/server/smb2pdu.c +++ b/fs/smb/server/smb2pdu.c @@ -1170,7 +1170,7 @@ int smb2_handle_negotiate(struct ksmbd_work *work) KSMBD_DEFAULT_GFP); if (!conn->preauth_info) { rc = -ENOMEM; - rsp->hdr.Status = STATUS_INVALID_PARAMETER; + rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; goto err_out; } @@ -1267,7 +1267,7 @@ int smb2_handle_negotiate(struct ksmbd_work *work) ksmbd_conn_set_need_setup(conn); err_out: - if (rc) + if (rc && rsp->hdr.Status == STATUS_SUCCESS) rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES; if (!rc) @@ -2049,12 +2049,11 @@ int smb2_tree_connect(struct ksmbd_work *work) write_unlock(&sess->tree_conns_lock); rsp->StructureSize = cpu_to_le16(16); out_err1: - if (server_conf.flags & KSMBD_GLOBAL_FLAG_DURABLE_HANDLE && share && - test_share_config_flag(share, - KSMBD_SHARE_FLAG_CONTINUOUS_AVAILABILITY)) - rsp->Capabilities = SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY; - else - rsp->Capabilities = 0; + /* + * A configured CA share is not continuously available until persistent + * open recovery, ownership fencing, and failover are implemented. + */ + rsp->Capabilities = 0; rsp->Reserved = 0; /* default manual caching */ rsp->ShareFlags = SMB2_SHAREFLAG_MANUAL_CACHING; diff --git a/fs/smb/server/smbacl.c b/fs/smb/server/smbacl.c index d3d0a22620f9..0879012aa63c 100644 --- a/fs/smb/server/smbacl.c +++ b/fs/smb/server/smbacl.c @@ -1471,7 +1471,7 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, DELETE; ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl)); - aces_size = acl_size - sizeof(struct smb_acl); + aces_size = pdacl_size - sizeof(struct smb_acl); for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) { if (aces_size < offsetof(struct smb_ace, sid) + CIFS_SID_BASE_SIZE) @@ -1495,7 +1495,7 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, id_to_sid(uid, sid_type, &sid); ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl)); - aces_size = acl_size - sizeof(struct smb_acl); + aces_size = pdacl_size - sizeof(struct smb_acl); for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) { if (aces_size < offsetof(struct smb_ace, sid) + CIFS_SID_BASE_SIZE) diff --git a/fs/smb/server/transport_ipc.c b/fs/smb/server/transport_ipc.c index 1c5645238bd3..225f6459f378 100644 --- a/fs/smb/server/transport_ipc.c +++ b/fs/smb/server/transport_ipc.c @@ -498,6 +498,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_rpc_command *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_rpc_command)) + return -EINVAL; + if (check_add_overflow(sizeof(struct ksmbd_rpc_command), resp->payload_sz, &msg_sz)) return -EINVAL; @@ -507,6 +510,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_spnego_authen_response *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_spnego_authen_response)) + return -EINVAL; + msg_sz = sizeof(struct ksmbd_spnego_authen_response) + resp->session_key_len + resp->spnego_blob_len; break; @@ -515,6 +521,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_share_config_response *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_share_config_response)) + return -EINVAL; + if (resp->payload_sz) { if (resp->payload_sz < resp->veto_list_sz) return -EINVAL; @@ -529,6 +538,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) { struct ksmbd_login_response_ext *resp = entry->response; + if (entry->msg_sz < sizeof(struct ksmbd_login_response_ext)) + return -EINVAL; + if (resp->ngroups) { if (resp->ngroups < 0 || resp->ngroups > NGROUPS_MAX) { diff --git a/fs/smb/server/transport_tcp.c b/fs/smb/server/transport_tcp.c index a09b07b3b108..78c909d66814 100644 --- a/fs/smb/server/transport_tcp.c +++ b/fs/smb/server/transport_tcp.c @@ -454,6 +454,11 @@ static void ksmbd_tcp_disconnect(struct ksmbd_transport *t) atomic_dec(&active_num_conn); } +static void ksmbd_tcp_shutdown(struct ksmbd_transport *t) +{ + kernel_sock_shutdown(TCP_TRANS(t)->sock, SHUT_RDWR); +} + static void tcp_destroy_socket(struct socket *ksmbd_socket) { int ret; @@ -709,5 +714,6 @@ static const struct ksmbd_transport_ops ksmbd_tcp_transport_ops = { .read = ksmbd_tcp_read, .writev = ksmbd_tcp_writev, .disconnect = ksmbd_tcp_disconnect, + .shutdown = ksmbd_tcp_shutdown, .free_transport = ksmbd_tcp_free_transport, }; diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c index 764eca75291c..216b6e427aa7 100644 --- a/fs/smb/server/vfs.c +++ b/fs/smb/server/vfs.c @@ -327,9 +327,6 @@ static int check_lock_range(struct file *filp, loff_t start, loff_t end, struct file_lock_context *ctx = locks_inode_context(file_inode(filp)); int error = 0; - if (start == end) - return 0; - if (!ctx || list_empty_careful(&ctx->flc_posix)) return 0; diff --git a/fs/smb/server/vfs_cache.c b/fs/smb/server/vfs_cache.c index 8b5f50ac61e2..0b654021bec0 100644 --- a/fs/smb/server/vfs_cache.c +++ b/fs/smb/server/vfs_cache.c @@ -1103,9 +1103,12 @@ void ksmbd_launch_ksmbd_durable_scavenger(void) server_conf.dh_task = kthread_run(ksmbd_durable_scavenger, (void *)NULL, "ksmbd-durable-scavenger"); - if (IS_ERR(server_conf.dh_task)) + if (IS_ERR(server_conf.dh_task)) { pr_err("cannot start conn thread, err : %ld\n", PTR_ERR(server_conf.dh_task)); + server_conf.dh_task = NULL; + durable_scavenger_running = false; + } mutex_unlock(&durable_scavenger_lock); } diff --git a/fs/squashfs/cache.c b/fs/squashfs/cache.c index 92fb857d2c76..c8756f65bc54 100644 --- a/fs/squashfs/cache.c +++ b/fs/squashfs/cache.c @@ -299,7 +299,7 @@ int squashfs_copy_data(void *buffer, struct squashfs_cache_entry *entry, { int remaining = length; - if (length == 0) + if (length == 0 || offset < 0) return 0; else if (buffer == NULL) return min(length, entry->length - offset); diff --git a/fs/stat.c b/fs/stat.c index 6c79661e1b96..8c110555b3d7 100644 --- a/fs/stat.c +++ b/fs/stat.c @@ -53,7 +53,7 @@ void fill_mg_cmtime(struct kstat *stat, u32 request_mask, struct inode *inode) } stat->mtime = inode_get_mtime(inode); - stat->ctime.tv_sec = inode->i_ctime_sec; + stat->ctime.tv_sec = inode_get_ctime_sec(inode); stat->ctime.tv_nsec = (u32)atomic_read(pcn); if (!(stat->ctime.tv_nsec & I_CTIME_QUERIED)) stat->ctime.tv_nsec = ((u32)atomic_fetch_or(I_CTIME_QUERIED, pcn)); diff --git a/fs/super.c b/fs/super.c index 8af2ee9395bc..3fe887425c8e 100644 --- a/fs/super.c +++ b/fs/super.c @@ -2083,7 +2083,7 @@ int freeze_super(struct super_block *sb, enum freeze_holder who, const void *fre int ret; if (!super_lock_excl(sb)) { - WARN_ON_ONCE("Dying superblock while freezing!"); + WARN_ONCE(1, "Dying superblock while freezing!"); return -EINVAL; } atomic_inc(&sb->s_active); @@ -2247,7 +2247,7 @@ int thaw_super(struct super_block *sb, enum freeze_holder who, const void *freeze_owner) { if (!super_lock_excl(sb)) { - WARN_ON_ONCE("Dying superblock while thawing!"); + WARN_ONCE(1, "Dying superblock while thawing!"); return -EINVAL; } return thaw_super_locked(sb, who, freeze_owner); diff --git a/fs/udf/inode.c b/fs/udf/inode.c index 3a184ded12ad..271e3f0d1fb1 100644 --- a/fs/udf/inode.c +++ b/fs/udf/inode.c @@ -337,65 +337,6 @@ const struct address_space_operations udf_aops = { .migrate_folio = buffer_migrate_folio, }; -/* - * Expand file stored in ICB to a normal one-block-file - * - * This function requires i_mutex held - */ -int udf_expand_file_adinicb(struct inode *inode) -{ - struct folio *folio; - struct udf_inode_info *iinfo = UDF_I(inode); - int err; - - WARN_ON_ONCE(!inode_is_locked(inode)); - if (!iinfo->i_lenAlloc) { - down_write(&iinfo->i_data_sem); - if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD)) - iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT; - else - iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG; - up_write(&iinfo->i_data_sem); - mark_inode_dirty(inode); - return 0; - } - - folio = __filemap_get_folio(inode->i_mapping, 0, - FGP_LOCK | FGP_ACCESSED | FGP_CREAT, GFP_KERNEL); - if (IS_ERR(folio)) - return PTR_ERR(folio); - - if (!folio_test_uptodate(folio)) - udf_adinicb_read_folio(folio); - down_write(&iinfo->i_data_sem); - memset(iinfo->i_data + iinfo->i_lenEAttr, 0x00, - iinfo->i_lenAlloc); - iinfo->i_lenAlloc = 0; - if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD)) - iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT; - else - iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG; - folio_mark_dirty(folio); - folio_unlock(folio); - up_write(&iinfo->i_data_sem); - err = filemap_fdatawrite(inode->i_mapping); - if (err) { - /* Restore everything back so that we don't lose data... */ - folio_lock(folio); - down_write(&iinfo->i_data_sem); - memcpy_from_folio(iinfo->i_data + iinfo->i_lenEAttr, - folio, 0, inode->i_size); - folio_unlock(folio); - iinfo->i_alloc_type = ICBTAG_FLAG_AD_IN_ICB; - iinfo->i_lenAlloc = inode->i_size; - up_write(&iinfo->i_data_sem); - } - folio_put(folio); - mark_inode_dirty(inode); - - return err; -} - #define UDF_MAP_CREATE 0x01 /* Mapping can allocate new blocks */ #define UDF_MAP_NOPREALLOC 0x02 /* Do not preallocate blocks */ @@ -456,6 +397,76 @@ static int udf_map_block(struct inode *inode, struct udf_map_rq *map) return ret; } +/* + * Expand file stored in ICB to a normal one-block-file + * + * This function requires i_mutex held + */ +int udf_expand_file_adinicb(struct inode *inode) +{ + struct folio *folio; + struct udf_inode_info *iinfo = UDF_I(inode); + struct udf_map_rq map = { + .lblk = 0, + .iflags = UDF_MAP_CREATE, + }; + int err; + + WARN_ON_ONCE(!inode_is_locked(inode)); + if (!iinfo->i_lenAlloc) { + down_write(&iinfo->i_data_sem); + if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD)) + iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT; + else + iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG; + up_write(&iinfo->i_data_sem); + mark_inode_dirty(inode); + return 0; + } + + folio = __filemap_get_folio(inode->i_mapping, 0, + FGP_LOCK | FGP_ACCESSED | FGP_CREAT, GFP_KERNEL); + if (IS_ERR(folio)) + return PTR_ERR(folio); + + if (!folio_test_uptodate(folio)) + udf_adinicb_read_folio(folio); + down_write(&iinfo->i_data_sem); + memset(iinfo->i_data + iinfo->i_lenEAttr, 0x00, + iinfo->i_lenAlloc); + iinfo->i_lenAlloc = 0; + if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD)) + iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT; + else + iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG; + up_write(&iinfo->i_data_sem); + + /* Allocate the block underlying the data */ + err = udf_map_block(inode, &map); + if (err < 0) + goto restore; + + folio_mark_dirty(folio); + folio_unlock(folio); + err = filemap_fdatawrite(inode->i_mapping); + if (err) { + /* Restore everything back so that we don't lose data... */ + folio_lock(folio); +restore: + down_write(&iinfo->i_data_sem); + memcpy_from_folio(iinfo->i_data + iinfo->i_lenEAttr, + folio, 0, inode->i_size); + iinfo->i_alloc_type = ICBTAG_FLAG_AD_IN_ICB; + iinfo->i_lenAlloc = inode->i_size; + up_write(&iinfo->i_data_sem); + folio_unlock(folio); + } + folio_put(folio); + mark_inode_dirty(inode); + + return err; +} + static int __udf_get_block(struct inode *inode, sector_t block, struct buffer_head *bh_result, int flags) { diff --git a/fs/udf/super.c b/fs/udf/super.c index 5a82ae2af93e..ff170bc0962b 100644 --- a/fs/udf/super.c +++ b/fs/udf/super.c @@ -2048,6 +2048,17 @@ static int udf_load_vrs(struct super_block *sb, struct udf_options *uopt, return 0; } +static void udf_mark_buffer_dirty(struct buffer_head *bh) +{ + /* + * We set buffer uptodate unconditionally here to avoid spurious + * warnings from mark_buffer_dirty() when previous EIO has marked + * the buffer as !uptodate + */ + set_buffer_uptodate(bh); + mark_buffer_dirty(bh); +} + static void udf_finalize_lvid(struct logicalVolIntegrityDesc *lvid) { struct timespec64 ts; @@ -2083,7 +2094,7 @@ static void udf_open_lvid(struct super_block *sb) UDF_SET_FLAG(sb, UDF_FLAG_INCONSISTENT); udf_finalize_lvid(lvid); - mark_buffer_dirty(bh); + udf_mark_buffer_dirty(bh); sbi->s_lvid_dirty = 0; mutex_unlock(&sbi->s_alloc_mutex); /* Make opening of filesystem visible on the media immediately */ @@ -2116,14 +2127,8 @@ static void udf_close_lvid(struct super_block *sb) if (!UDF_QUERY_FLAG(sb, UDF_FLAG_INCONSISTENT)) lvid->integrityType = cpu_to_le32(LVID_INTEGRITY_TYPE_CLOSE); - /* - * We set buffer uptodate unconditionally here to avoid spurious - * warnings from mark_buffer_dirty() when previous EIO has marked - * the buffer as !uptodate - */ - set_buffer_uptodate(bh); udf_finalize_lvid(lvid); - mark_buffer_dirty(bh); + udf_mark_buffer_dirty(bh); sbi->s_lvid_dirty = 0; mutex_unlock(&sbi->s_alloc_mutex); /* Make closing of filesystem visible on the media immediately */ @@ -2405,7 +2410,7 @@ static int udf_sync_fs(struct super_block *sb, int wait) * Blockdevice will be synced later so we don't have to submit * the buffer for IO */ - mark_buffer_dirty(bh); + udf_mark_buffer_dirty(bh); sbi->s_lvid_dirty = 0; } mutex_unlock(&sbi->s_alloc_mutex); diff --git a/fs/udf/symlink.c b/fs/udf/symlink.c index fe03745d09b1..a05d1888a2ba 100644 --- a/fs/udf/symlink.c +++ b/fs/udf/symlink.c @@ -36,6 +36,8 @@ static int udf_pc_to_char(struct super_block *sb, unsigned char *from, /* Reserve one byte for terminating \0 */ tolen--; while (elen < fromlen) { + if (fromlen - elen < sizeof(struct pathComponent)) + return -EIO; pc = (struct pathComponent *)(from + elen); elen += sizeof(struct pathComponent); switch (pc->componentType) { diff --git a/include/acpi/processor.h b/include/acpi/processor.h index d0eccbd920e5..ff864c1cee3a 100644 --- a/include/acpi/processor.h +++ b/include/acpi/processor.h @@ -423,6 +423,8 @@ int acpi_processor_power_init(struct acpi_processor *pr); int acpi_processor_power_exit(struct acpi_processor *pr); int acpi_processor_power_state_has_changed(struct acpi_processor *pr); int acpi_processor_hotplug(struct acpi_processor *pr); +void acpi_processor_register_idle_driver(void); +void acpi_processor_unregister_idle_driver(void); #else static inline int acpi_processor_power_init(struct acpi_processor *pr) { diff --git a/include/drm/bridge/dw_dp.h b/include/drm/bridge/dw_dp.h index d05df49fd884..22105c3e8e4d 100644 --- a/include/drm/bridge/dw_dp.h +++ b/include/drm/bridge/dw_dp.h @@ -11,10 +11,18 @@ struct drm_encoder; struct dw_dp; +enum { + DW_DP_MP_SINGLE_PIXEL, + DW_DP_MP_DUAL_PIXEL, + DW_DP_MP_QUAD_PIXEL, +}; + struct dw_dp_plat_data { u32 max_link_rate; + u8 pixel_mode; }; struct dw_dp *dw_dp_bind(struct device *dev, struct drm_encoder *encoder, const struct dw_dp_plat_data *plat_data); +void dw_dp_unbind(struct dw_dp *dp); #endif /* __DW_DP__ */ diff --git a/include/linux/bio.h b/include/linux/bio.h index 16c1c85613b7..3a9dd300b5ee 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -324,6 +324,8 @@ extern struct bio *bio_split(struct bio *bio, int sectors, gfp_t gfp, struct bio_set *bs); int bio_split_io_at(struct bio *bio, const struct queue_limits *lim, unsigned *segs, unsigned max_bytes, unsigned len_align); +u8 bio_seg_gap(struct request_queue *q, struct bio *prev, struct bio *next, + u8 gaps_bit); /** * bio_next_split - get next @sectors from a bio, splitting if necessary @@ -376,17 +378,27 @@ void submit_bio(struct bio *bio); extern void bio_endio(struct bio *); -static inline void bio_io_error(struct bio *bio) +/** + * bio_endio_status - end I/O on a bio with a specific status + * @bio: bio + * @status: status to set + * + * Set @bio->bi_status to @status and call bio_endio(). + **/ +static inline void bio_endio_status(struct bio *bio, blk_status_t status) { - bio->bi_status = BLK_STS_IOERR; + bio->bi_status = status; bio_endio(bio); } +static inline void bio_io_error(struct bio *bio) +{ + bio_endio_status(bio, BLK_STS_IOERR); +} + static inline void bio_wouldblock_error(struct bio *bio) { - bio_set_flag(bio, BIO_QUIET); - bio->bi_status = BLK_STS_AGAIN; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_AGAIN); } /* diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index b25d12545f46..b54506b3b76d 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -152,6 +152,14 @@ struct request { unsigned short nr_phys_segments; unsigned short nr_integrity_segments; + /* + * The lowest set bit for address gaps between physical segments. This + * provides information necessary for dma optimization opprotunities, + * like for testing if the segments can be coalesced against the + * device's iommu granule. + */ + unsigned char phys_gap_bit; + #ifdef CONFIG_BLK_INLINE_ENCRYPTION struct bio_crypt_ctx *crypt_ctx; struct blk_crypto_keyslot *crypt_keyslot; @@ -208,6 +216,14 @@ struct request { void *end_io_data; }; +/* + * Returns a mask with all bits starting at req->phys_gap_bit set to 1. + */ +static inline unsigned long req_phys_gap_mask(const struct request *req) +{ + return ~(((1 << req->phys_gap_bit) >> 1) - 1); +} + static inline enum req_op req_op(const struct request *req) { return req->cmd_flags & REQ_OP_MASK; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 4e2e3aed32f5..47fe5631d9e2 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -218,6 +218,18 @@ struct bio { enum rw_hint bi_write_hint; u8 bi_write_stream; blk_status_t bi_status; + + /* + * The bvec gap bit indicates the lowest set bit in any address offset + * between all bi_io_vecs. This field is initialized only after the bio + * is split to the hardware limits (see bio_split_io_at()). The value + * may be used to consider DMA optimization when performing that + * mapping. The value is compared to a power of two mask where the + * result depends on any bit set within the mask, so saving the lowest + * bit is sufficient to know if any segment gap collides with the mask. + */ + u8 bi_bvec_gap_bit; + atomic_t __bi_remaining; struct bvec_iter bi_iter; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e264c695f31c..7052507ed451 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -536,7 +536,7 @@ static inline void bpf_obj_memcpy(struct btf_record *rec, if (IS_ERR_OR_NULL(rec)) { if (long_memcpy) - bpf_long_memcpy(dst, src, round_up(size, 8)); + bpf_long_memcpy(dst, src, size); else memcpy(dst, src, size); return; @@ -559,7 +559,7 @@ static inline void copy_map_value(struct bpf_map *map, void *dst, void *src) static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src) { - bpf_obj_memcpy(map->record, dst, src, map->value_size, true); + bpf_obj_memcpy(map->record, dst, src, round_up(map->value_size, 8), true); } static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src) @@ -1044,21 +1044,6 @@ static inline bool bpf_pseudo_func(const struct bpf_insn *insn) return bpf_is_ldimm64(insn) && insn->src_reg == BPF_PSEUDO_FUNC; } -/* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an - * atomic load or store, and false if it is a read-modify-write instruction. - */ -static inline bool -bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn) -{ - switch (atomic_insn->imm) { - case BPF_LOAD_ACQ: - case BPF_STORE_REL: - return true; - default: - return false; - } -} - struct bpf_prog_ops { int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr); diff --git a/include/linux/compiler-context-analysis.h b/include/linux/compiler-context-analysis.h new file mode 100644 index 000000000000..f8af63045281 --- /dev/null +++ b/include/linux/compiler-context-analysis.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Macros and attributes for compiler-based static context analysis. + */ + +#ifndef _LINUX_COMPILER_CONTEXT_ANALYSIS_H +#define _LINUX_COMPILER_CONTEXT_ANALYSIS_H + +#ifdef __CHECKER__ + +/* Sparse context/lock checking support. */ +# define __must_hold(x) __attribute__((context(x,1,1))) +# define __acquires(x) __attribute__((context(x,0,1))) +# define __cond_acquires(x) __attribute__((context(x,0,-1))) +# define __releases(x) __attribute__((context(x,1,0))) +# define __acquire(x) __context__(x,1) +# define __release(x) __context__(x,-1) +# define __cond_lock(x, c) ((c) ? ({ __acquire(x); 1; }) : 0) + +#else /* !__CHECKER__ */ + +# define __must_hold(x) +# define __acquires(x) +# define __cond_acquires(x) +# define __releases(x) +# define __acquire(x) (void)0 +# define __release(x) (void)0 +# define __cond_lock(x, c) (c) + +#endif /* __CHECKER__ */ + +#endif /* _LINUX_COMPILER_CONTEXT_ANALYSIS_H */ diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index 2f18b8a01afe..ba49413f704b 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -37,6 +37,8 @@ # define BTF_TYPE_TAG(value) /* nothing */ #endif +#include + /* sparse defines __CHECKER__; see Documentation/dev-tools/sparse.rst */ #ifdef __CHECKER__ /* address spaces */ @@ -47,14 +49,6 @@ # define __rcu __attribute__((noderef, address_space(__rcu))) static inline void __chk_user_ptr(const volatile void __user *ptr) { } static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } -/* context/locking */ -# define __must_hold(x) __attribute__((context(x,1,1))) -# define __acquires(x) __attribute__((context(x,0,1))) -# define __cond_acquires(x) __attribute__((context(x,0,-1))) -# define __releases(x) __attribute__((context(x,1,0))) -# define __acquire(x) __context__(x,1) -# define __release(x) __context__(x,-1) -# define __cond_lock(x,c) ((c) ? ({ __acquire(x); 1; }) : 0) /* other */ # define __force __attribute__((force)) # define __nocast __attribute__((nocast)) @@ -75,14 +69,6 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { } # define __chk_user_ptr(x) (void)0 # define __chk_io_ptr(x) (void)0 -/* context/locking */ -# define __must_hold(x) -# define __acquires(x) -# define __cond_acquires(x) -# define __releases(x) -# define __acquire(x) (void)0 -# define __release(x) (void)0 -# define __cond_lock(x,c) (c) /* other */ # define __force # define __nocast diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h index d58e329ac0e7..0b0b5fbb6cc1 100644 --- a/include/linux/dma-buf.h +++ b/include/linux/dma-buf.h @@ -582,6 +582,11 @@ void dma_buf_unpin(struct dma_buf_attachment *attach); struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info); int dma_buf_fd(struct dma_buf *dmabuf, int flags); +/* + * This tree has no DMA-BUF fd tracepoint, so publishing a reserved fd only + * requires fd_install(). Call this after all fallible work has succeeded. + */ +#define dma_buf_fd_install(dmabuf, fd) fd_install((fd), (dmabuf)->file) struct dma_buf *dma_buf_get(int fd); void dma_buf_put(struct dma_buf *dmabuf); diff --git a/include/linux/efi.h b/include/linux/efi.h index a98cc39e7aaa..0ec9f3dc596d 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -1228,8 +1228,8 @@ efi_call_acpi_prm_handler(efi_status_t (__efiapi *handler_addr)(u64, void *), /* * efi_runtime_service() function identifiers. - * "NONE" is used by efi_recover_from_page_fault() to check if the page - * fault happened while executing an efi runtime service. + * "NONE" is used by efi_crash_gracefully_on_page_fault() to check if the + * page fault happened while executing an efi runtime service. */ enum efi_rts_ids { EFI_NONE, diff --git a/include/linux/filter.h b/include/linux/filter.h index 2469fd2e4015..260a19063405 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -383,6 +383,37 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn) /* Legacy alias */ #define BPF_STX_XADD(SIZE, DST, SRC, OFF) BPF_ATOMIC_OP(SIZE, BPF_ADD, DST, SRC, OFF) +/* + * Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an + * atomic load or store, and false if it is a read-modify-write instruction. + */ +static inline bool +bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn) +{ + switch (atomic_insn->imm) { + case BPF_LOAD_ACQ: + case BPF_STORE_REL: + return true; + default: + return false; + } +} + +/* + * A load-acquire is the only BPF_STX class instruction that reads into + * dst_reg from src_reg + off16, i.e. it has the operand roles of a BPF_LDX. + * Unlike bpf_atomic_is_load_store(), @insn is not assumed to be a BPF_ATOMIC + * instruction here, so that callers which walk all instruction classes can + * use this directly. + */ +static inline bool bpf_atomic_is_load_acq(const struct bpf_insn *insn) +{ + return BPF_CLASS(insn->code) == BPF_STX && + (BPF_MODE(insn->code) == BPF_ATOMIC || + BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) && + insn->imm == BPF_LOAD_ACQ; +} + /* Memory store, *(uint *) (dst_reg + off16) = imm32 */ #define BPF_ST_MEM(SIZE, DST, OFF, IMM) \ diff --git a/include/linux/firmware/qcom/qcom_scm.h b/include/linux/firmware/qcom/qcom_scm.h index a13f703b16cd..5045f8fe876d 100644 --- a/include/linux/firmware/qcom/qcom_scm.h +++ b/include/linux/firmware/qcom/qcom_scm.h @@ -72,6 +72,20 @@ struct qcom_scm_pas_metadata { ssize_t size; }; +struct qcom_scm_pas_context { + struct device *dev; + u32 pas_id; + phys_addr_t mem_phys; + size_t mem_size; + void *ptr; + dma_addr_t phys; + ssize_t size; +}; + +struct qcom_scm_pas_context *devm_qcom_scm_pas_context_alloc(struct device *dev, + u32 pas_id, + phys_addr_t mem_phys, + size_t mem_size); int qcom_scm_pas_init_image(u32 pas_id, const void *metadata, size_t size, struct qcom_scm_pas_metadata *ctx); void qcom_scm_pas_metadata_release(struct qcom_scm_pas_metadata *ctx); diff --git a/include/linux/fs.h b/include/linux/fs.h index 14f5accc97b7..55ecd9d26b4f 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1783,12 +1783,12 @@ struct timespec64 inode_set_ctime_deleg(struct inode *inode, static inline time64_t inode_get_atime_sec(const struct inode *inode) { - return inode->i_atime_sec; + return READ_ONCE(inode->i_atime_sec); } static inline long inode_get_atime_nsec(const struct inode *inode) { - return inode->i_atime_nsec; + return READ_ONCE(inode->i_atime_nsec); } static inline struct timespec64 inode_get_atime(const struct inode *inode) @@ -1802,8 +1802,8 @@ static inline struct timespec64 inode_get_atime(const struct inode *inode) static inline struct timespec64 inode_set_atime_to_ts(struct inode *inode, struct timespec64 ts) { - inode->i_atime_sec = ts.tv_sec; - inode->i_atime_nsec = ts.tv_nsec; + WRITE_ONCE(inode->i_atime_sec, ts.tv_sec); + WRITE_ONCE(inode->i_atime_nsec, ts.tv_nsec); return ts; } @@ -1818,12 +1818,12 @@ static inline struct timespec64 inode_set_atime(struct inode *inode, static inline time64_t inode_get_mtime_sec(const struct inode *inode) { - return inode->i_mtime_sec; + return READ_ONCE(inode->i_mtime_sec); } static inline long inode_get_mtime_nsec(const struct inode *inode) { - return inode->i_mtime_nsec; + return READ_ONCE(inode->i_mtime_nsec); } static inline struct timespec64 inode_get_mtime(const struct inode *inode) @@ -1836,8 +1836,8 @@ static inline struct timespec64 inode_get_mtime(const struct inode *inode) static inline struct timespec64 inode_set_mtime_to_ts(struct inode *inode, struct timespec64 ts) { - inode->i_mtime_sec = ts.tv_sec; - inode->i_mtime_nsec = ts.tv_nsec; + WRITE_ONCE(inode->i_mtime_sec, ts.tv_sec); + WRITE_ONCE(inode->i_mtime_nsec, ts.tv_nsec); return ts; } @@ -1862,12 +1862,12 @@ static inline struct timespec64 inode_set_mtime(struct inode *inode, static inline time64_t inode_get_ctime_sec(const struct inode *inode) { - return inode->i_ctime_sec; + return READ_ONCE(inode->i_ctime_sec); } static inline long inode_get_ctime_nsec(const struct inode *inode) { - return inode->i_ctime_nsec & ~I_CTIME_QUERIED; + return READ_ONCE(inode->i_ctime_nsec) & ~I_CTIME_QUERIED; } static inline struct timespec64 inode_get_ctime(const struct inode *inode) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index c3b1c74bdd7a..62e85b2c6516 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -797,8 +797,9 @@ unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec); unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec); extern ftrace_func_t ftrace_trace_function; +struct trace_array; -int ftrace_regex_open(struct ftrace_ops *ops, int flag, +int ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag, struct inode *inode, struct file *file); ssize_t ftrace_filter_write(struct file *file, const char __user *ubuf, size_t cnt, loff_t *ppos); @@ -1008,7 +1009,7 @@ static inline unsigned long ftrace_location(unsigned long ip) * have them defined when ftrace is not enabled, but these * functions may still be called. Use a macro instead of inline. */ -#define ftrace_regex_open(ops, flag, inod, file) ({ -ENODEV; }) +#define ftrace_regex_open(tr, ops, flag, inode, file) ({ -ENODEV; }) #define ftrace_set_early_filter(ops, buf, enable) do { } while (0) #define ftrace_set_filter_ip(ops, ip, remove, reset) ({ -ENODEV; }) #define ftrace_set_filter_ips(ops, ips, cnt, remove, reset) ({ -ENODEV; }) diff --git a/include/linux/hwmon.h b/include/linux/hwmon.h index 301a83afbd66..f5330e7ae247 100644 --- a/include/linux/hwmon.h +++ b/include/linux/hwmon.h @@ -13,6 +13,7 @@ #define _HWMON_H_ #include +#include struct device; struct attribute_group; @@ -495,6 +496,8 @@ char *devm_hwmon_sanitize_name(struct device *dev, const char *name); void hwmon_lock(struct device *dev); void hwmon_unlock(struct device *dev); +DEFINE_GUARD(hwmon_lock, struct device *, hwmon_lock(_T), hwmon_unlock(_T)) + /** * hwmon_is_bad_char - Is the char invalid in a hwmon name * @ch: the char to be considered diff --git a/include/linux/i3c/master.h b/include/linux/i3c/master.h index 1aa1efd835b8..df0132048a10 100644 --- a/include/linux/i3c/master.h +++ b/include/linux/i3c/master.h @@ -228,6 +228,8 @@ struct i3c_dev_desc { * every time the I3C device is rediscovered with a different dynamic * address assigned * @bus: I3C bus this device is attached to + * @node: unregistered device list node, only for use by + * i3c_master_register_new_i3c_devs(), it is not protected by a lock * * I3C device object exposed to I3C device drivers. The takes care of linking * this object to the relevant &struct_i3c_dev_desc one. @@ -238,6 +240,7 @@ struct i3c_device { struct device dev; struct i3c_dev_desc *desc; struct i3c_bus *bus; + struct list_head node; }; /* diff --git a/include/linux/ieee80211-uhr.h b/include/linux/ieee80211-uhr.h new file mode 100644 index 000000000000..132acced7d79 --- /dev/null +++ b/include/linux/ieee80211-uhr.h @@ -0,0 +1,220 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * IEEE 802.11 UHR definitions + * + * Copyright (c) 2025-2026 Intel Corporation + */ +#ifndef LINUX_IEEE80211_UHR_H +#define LINUX_IEEE80211_UHR_H + +#include +#include + +#define IEEE80211_UHR_OPER_PARAMS_DPS_ENA 0x0001 +#define IEEE80211_UHR_OPER_PARAMS_NPCA_ENA 0x0002 +#define IEEE80211_UHR_OPER_PARAMS_DBE_ENA 0x0004 +#define IEEE80211_UHR_OPER_PARAMS_PEDCA_ENA 0x0008 + +struct ieee80211_uhr_operation { + __le16 params; + u8 basic_mcs_nss_set[4]; + u8 variable[]; +} __packed; + +#define IEEE80211_UHR_NPCA_PARAMS_PRIMARY_CHAN_OFFS 0x0000000F +#define IEEE80211_UHR_NPCA_PARAMS_MIN_DUR_THRESH 0x000000F0 +#define IEEE80211_UHR_NPCA_PARAMS_SWITCH_DELAY 0x00003F00 +#define IEEE80211_UHR_NPCA_PARAMS_SWITCH_BACK_DELAY 0x000FC000 +#define IEEE80211_UHR_NPCA_PARAMS_INIT_QSRC 0x00300000 +#define IEEE80211_UHR_NPCA_PARAMS_MOPLEN 0x00400000 +#define IEEE80211_UHR_NPCA_PARAMS_DIS_SUBCH_BMAP_PRES 0x00800000 + +struct ieee80211_uhr_npca_info { + __le32 params; + __le16 dis_subch_bmap[]; +} __packed; + +static inline bool ieee80211_uhr_oper_size_ok(const u8 *data, u8 len, + bool beacon) +{ + const struct ieee80211_uhr_operation *oper = (const void *)data; + u8 needed = sizeof(*oper); + + if (len < needed) + return false; + + /* nothing else present in beacons */ + if (beacon) + return true; + + /* FIXME: DPS, DBE, P-EDCA (consider order, also relative to NPCA) */ + + if (oper->params & cpu_to_le16(IEEE80211_UHR_OPER_PARAMS_NPCA_ENA)) { + const struct ieee80211_uhr_npca_info *npca = + (const void *)oper->variable; + + needed += sizeof(*npca); + + if (len < needed) + return false; + + if (npca->params & cpu_to_le32(IEEE80211_UHR_NPCA_PARAMS_DIS_SUBCH_BMAP_PRES)) + needed += sizeof(npca->dis_subch_bmap[0]); + } + + return len >= needed; +} + +/* + * Note: cannot call this on the element coming from a beacon, + * must ensure ieee80211_uhr_oper_size_ok(..., false) first + */ +static inline const struct ieee80211_uhr_npca_info * +ieee80211_uhr_npca_info(const struct ieee80211_uhr_operation *oper) +{ + if (!(oper->params & cpu_to_le16(IEEE80211_UHR_OPER_PARAMS_NPCA_ENA))) + return NULL; + + /* FIXME: DPS */ + + return (const void *)oper->variable; +} + +static inline const __le16 * +ieee80211_uhr_npca_dis_subch_bitmap(const struct ieee80211_uhr_operation *oper) +{ + const struct ieee80211_uhr_npca_info *npca; + + npca = ieee80211_uhr_npca_info(oper); + if (!npca) + return NULL; + if (!(npca->params & cpu_to_le32(IEEE80211_UHR_NPCA_PARAMS_DIS_SUBCH_BMAP_PRES))) + return NULL; + return npca->dis_subch_bmap; +} + +#define IEEE80211_UHR_MAC_CAP0_DPS_SUPP 0x01 +#define IEEE80211_UHR_MAC_CAP0_DPS_ASSIST_SUPP 0x02 +#define IEEE80211_UHR_MAC_CAP0_DPS_AP_STATIC_HCM_SUPP 0x04 +#define IEEE80211_UHR_MAC_CAP0_NPCA_SUPP 0x10 +#define IEEE80211_UHR_MAC_CAP0_ENH_BSR_SUPP 0x20 +#define IEEE80211_UHR_MAC_CAP0_ADD_MAP_TID_SUPP 0x40 +#define IEEE80211_UHR_MAC_CAP0_EOTSP_SUPP 0x80 + +#define IEEE80211_UHR_MAC_CAP1_DSO_SUPP 0x01 +#define IEEE80211_UHR_MAC_CAP1_PEDCA_SUPP 0x02 +#define IEEE80211_UHR_MAC_CAP1_DBE_SUPP 0x04 +#define IEEE80211_UHR_MAC_CAP1_UL_LLI_SUPP 0x08 +#define IEEE80211_UHR_MAC_CAP1_P2P_LLI_SUPP 0x10 +#define IEEE80211_UHR_MAC_CAP1_PUO_SUPP 0x20 +#define IEEE80211_UHR_MAC_CAP1_AP_PUO_SUPP 0x40 +#define IEEE80211_UHR_MAC_CAP1_DUO_SUPP 0x80 + +#define IEEE80211_UHR_MAC_CAP2_OMC_UL_MU_DIS_RX_SUPP 0x01 +#define IEEE80211_UHR_MAC_CAP2_AOM_SUPP 0x02 +#define IEEE80211_UHR_MAC_CAP2_IFCS_LOC_SUPP 0x04 +#define IEEE80211_UHR_MAC_CAP2_UHR_TRS_SUPP 0x08 +#define IEEE80211_UHR_MAC_CAP2_TXSPG_SUPP 0x10 +#define IEEE80211_UHR_MAC_CAP2_TXOP_RET_IN_TXSPG 0x20 +#define IEEE80211_UHR_MAC_CAP2_UHR_OM_PU_TO_LOW 0xC0 + +#define IEEE80211_UHR_MAC_CAP3_UHR_OM_PU_TO_HIGH 0x03 +#define IEEE80211_UHR_MAC_CAP3_PARAM_UPD_ADV_NOTIF_INTV 0x1C +#define IEEE80211_UHR_MAC_CAP3_UPD_IND_TIM_INTV_LOW 0xE0 + +#define IEEE80211_UHR_MAC_CAP4_UPD_IND_TIM_INTV_HIGH 0x03 +#define IEEE80211_UHR_MAC_CAP4_BOUNDED_ESS 0x04 +#define IEEE80211_UHR_MAC_CAP4_BTM_ASSURANCE 0x08 +#define IEEE80211_UHR_MAC_CAP4_CO_BF_SUPP 0x10 + +#define IEEE80211_UHR_MAC_CAP_DBE_MAX_BW 0x07 +#define IEEE80211_UHR_MAC_CAP_DBE_EHT_MCS_MAP_160_PRES 0x08 +#define IEEE80211_UHR_MAC_CAP_DBE_EHT_MCS_MAP_320_PRES 0x10 + +struct ieee80211_uhr_cap_mac { + u8 mac_cap[5]; +} __packed; + +struct ieee80211_uhr_cap { + struct ieee80211_uhr_cap_mac mac; + /* DBE, PHY capabilities */ + u8 variable[]; +} __packed; + +#define IEEE80211_UHR_PHY_CAP_MAX_NSS_RX_SND_NDP_LE80 0x01 +#define IEEE80211_UHR_PHY_CAP_MAX_NSS_RX_DL_MU_LE80 0x02 +#define IEEE80211_UHR_PHY_CAP_MAX_NSS_RX_SND_NDP_160 0x04 +#define IEEE80211_UHR_PHY_CAP_MAX_NSS_RX_DL_MU_160 0x08 +#define IEEE80211_UHR_PHY_CAP_MAX_NSS_RX_SND_NDP_320 0x10 +#define IEEE80211_UHR_PHY_CAP_MAX_NSS_RX_DL_MU_320 0x20 +#define IEEE80211_UHR_PHY_CAP_ELR_RX 0x40 +#define IEEE80211_UHR_PHY_CAP_ELR_TX 0x80 + +struct ieee80211_uhr_cap_phy { + u8 cap; +} __packed; + +static inline bool ieee80211_uhr_capa_size_ok(const u8 *data, u8 len, + bool from_ap) +{ + const struct ieee80211_uhr_cap *cap = (const void *)data; + size_t needed = sizeof(*cap) + sizeof(struct ieee80211_uhr_cap_phy); + + if (len < needed) + return false; + + /* + * A non-AP STA does not include the DBE Capability Parameters field + * in the UHR MAC Capabilities Information field. + */ + if (from_ap && cap->mac.mac_cap[1] & IEEE80211_UHR_MAC_CAP1_DBE_SUPP) { + u8 dbe; + + needed += 1; + if (len < needed) + return false; + + dbe = cap->variable[0]; + + if (dbe & IEEE80211_UHR_MAC_CAP_DBE_EHT_MCS_MAP_160_PRES) + needed += 3; + + if (dbe & IEEE80211_UHR_MAC_CAP_DBE_EHT_MCS_MAP_320_PRES) + needed += 3; + } + + return len >= needed; +} + +static inline const struct ieee80211_uhr_cap_phy * +ieee80211_uhr_phy_cap(const struct ieee80211_uhr_cap *cap, bool from_ap) +{ + u8 offs = 0; + + if (from_ap && cap->mac.mac_cap[1] & IEEE80211_UHR_MAC_CAP1_DBE_SUPP) { + u8 dbe = cap->variable[0]; + + offs += 1; + + if (dbe & IEEE80211_UHR_MAC_CAP_DBE_EHT_MCS_MAP_160_PRES) + offs += 3; + + if (dbe & IEEE80211_UHR_MAC_CAP_DBE_EHT_MCS_MAP_320_PRES) + offs += 3; + } + + return (const void *)&cap->variable[offs]; +} + +#define IEEE80211_SMD_INFO_CAPA_DL_DATA_FWD 0x01 +#define IEEE80211_SMD_INFO_CAPA_MAX_NUM_PREP 0x0E +#define IEEE80211_SMD_INFO_CAPA_TYPE 0x10 +#define IEEE80211_SMD_INFO_CAPA_PTK_PER_AP_MLD 0x20 + +struct ieee80211_smd_info { + u8 id[ETH_ALEN]; + u8 capa; + __le16 timeout; +} __packed; + +#endif /* LINUX_IEEE80211_UHR_H */ diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h index 992a0c09c1e5..2526ccd8a4a0 100644 --- a/include/linux/ieee80211.h +++ b/include/linux/ieee80211.h @@ -9,7 +9,7 @@ * Copyright (c) 2006, Michael Wu * Copyright (c) 2013 - 2014 Intel Mobile Communications GmbH * Copyright (c) 2016 - 2017 Intel Deutschland GmbH - * Copyright (c) 2018 - 2025 Intel Corporation + * Copyright (c) 2018 - 2026 Intel Corporation */ #ifndef LINUX_IEEE80211_H @@ -1381,8 +1381,9 @@ struct ieee80211_mgmt { #define BSS_MEMBERSHIP_SELECTOR_SAE_H2E 123 #define BSS_MEMBERSHIP_SELECTOR_HE_PHY 122 #define BSS_MEMBERSHIP_SELECTOR_EHT_PHY 121 +#define BSS_MEMBERSHIP_SELECTOR_UHR_PHY 120 -#define BSS_MEMBERSHIP_SELECTOR_MIN BSS_MEMBERSHIP_SELECTOR_EHT_PHY +#define BSS_MEMBERSHIP_SELECTOR_MIN BSS_MEMBERSHIP_SELECTOR_UHR_PHY /* mgmt header + 1 byte category code */ #define IEEE80211_MIN_ACTION_SIZE offsetof(struct ieee80211_mgmt, u.action.u) @@ -2117,6 +2118,15 @@ enum ieee80211_eid_ext { WLAN_EID_EXT_BANDWIDTH_INDICATION = 135, WLAN_EID_EXT_KNOWN_STA_IDENTIFCATION = 136, WLAN_EID_EXT_NON_AP_STA_REG_CON = 137, + WLAN_EID_EXT_UHR_OPER = 151, + WLAN_EID_EXT_UHR_CAPA = 152, + WLAN_EID_EXT_MACP = 153, + WLAN_EID_EXT_SMD = 154, + WLAN_EID_EXT_BSS_SMD_TRANS_PARAMS = 155, + WLAN_EID_EXT_CHAN_USAGE = 156, + WLAN_EID_EXT_UHR_MODE_CHG = 157, + WLAN_EID_EXT_UHR_PARAM_UPD = 158, + WLAN_EID_EXT_TXPI = 159, }; /* Action category code */ @@ -3352,6 +3362,22 @@ static inline bool for_each_element_completed(const struct element *element, #define WLAN_RSNX_CAPA_PROTECTED_TWT BIT(4) #define WLAN_RSNX_CAPA_SAE_H2E BIT(5) +/* EBPCC = Enhanced BSS Parameter Change Count */ +#define IEEE80211_ENH_CRIT_UPD_EBPCC 0x0F +#define IEEE80211_ENH_CRIT_UPD_TYPE 0x70 +#define IEEE80211_ENH_CRIT_UPD_TYPE_NO_UHR 0 +#define IEEE80211_ENH_CRIT_UPD_TYPE_UHR 1 +#define IEEE80211_ENH_CRIT_UPD_ALL 0x80 + +/** + * struct ieee80211_enh_crit_upd - enhanced critical update (UHR) + * @v: value of the enhanced critical update data, + * see %IEEE80211_ENH_CRIT_UPD_* to parse the bits + */ +struct ieee80211_enh_crit_upd { + u8 v; +} __packed; + /* * reduced neighbor report, based on Draft P802.11ax_D6.1, * section 9.4.2.170 and accepted contributions. @@ -3370,6 +3396,7 @@ static inline bool for_each_element_completed(const struct element *element, #define IEEE80211_RNR_TBTT_PARAMS_COLOC_ESS 0x10 #define IEEE80211_RNR_TBTT_PARAMS_PROBE_ACTIVE 0x20 #define IEEE80211_RNR_TBTT_PARAMS_COLOC_AP 0x40 +#define IEEE80211_RNR_TBTT_PARAMS_SAME_SMD 0x80 #define IEEE80211_RNR_TBTT_PARAMS_PSD_NO_LIMIT 127 #define IEEE80211_RNR_TBTT_PARAMS_PSD_RESERVED -128 @@ -3422,6 +3449,7 @@ struct ieee80211_tbtt_info_ge_11 { u8 bss_params; s8 psd_20; struct ieee80211_rnr_mld_params mld_params; + struct ieee80211_enh_crit_upd enh_crit_upd; } __packed; /* NAN operation mode, as defined in Wi-Fi Aware (TM) specification Table 81 */ @@ -3445,6 +3473,7 @@ struct ieee80211_tbtt_info_ge_11 { #include "ieee80211-vht.h" #include "ieee80211-he.h" #include "ieee80211-eht.h" +#include "ieee80211-uhr.h" #include "ieee80211-mesh.h" #endif /* LINUX_IEEE80211_H */ diff --git a/include/linux/ima.h b/include/linux/ima.h index abf8923f8fc5..8e08baf16c2f 100644 --- a/include/linux/ima.h +++ b/include/linux/ima.h @@ -11,6 +11,7 @@ #include #include #include +#include #include struct linux_binprm; @@ -73,14 +74,8 @@ int ima_validate_range(phys_addr_t phys, size_t size); #endif #ifdef CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT -extern bool arch_ima_get_secureboot(void); extern const char * const *arch_get_ima_policy(void); #else -static inline bool arch_ima_get_secureboot(void) -{ - return false; -} - static inline const char * const *arch_get_ima_policy(void) { return NULL; diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index d73556f10733..4f1607e00359 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -482,6 +482,7 @@ struct io_ring_ctx { * ONLY core io_uring.c should instantiate this struct. */ struct io_tw_state { + bool cancel; }; /* Alias to use in code that doesn't instantiate struct io_tw_state */ typedef struct io_tw_state io_tw_token_t; @@ -625,7 +626,11 @@ enum { REQ_F_SQE_COPIED = IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT), }; -typedef void (*io_req_tw_func_t)(struct io_kiocb *req, io_tw_token_t tw); +struct io_tw_req { + struct io_kiocb *req; +}; + +typedef void (*io_req_tw_func_t)(struct io_tw_req tw_req, io_tw_token_t tw); struct io_task_work { struct llist_node node; diff --git a/include/linux/libnvdimm.h b/include/linux/libnvdimm.h index 28f086c4a187..d929d83abf3b 100644 --- a/include/linux/libnvdimm.h +++ b/include/linux/libnvdimm.h @@ -126,6 +126,15 @@ struct nd_mapping_desc { struct bio; struct resource; struct nd_region; + +/* + * Provider flush callback return values: + * 0: flush completed synchronously + * <0: flush failed + * >0: flush completion was queued and @bio will be completed later + */ +#define NVDIMM_FLUSH_ASYNC 1 + struct nd_region_desc { struct resource *res; struct nd_mapping_desc *mapping; diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 66f98a3da8d8..2ba53c065467 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -539,7 +539,7 @@ void mas_destroy(struct ma_state *mas); int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries); void *mas_prev(struct ma_state *mas, unsigned long min); -void *mas_prev_range(struct ma_state *mas, unsigned long max); +void *mas_prev_range(struct ma_state *mas, unsigned long min); void *mas_next(struct ma_state *mas, unsigned long max); void *mas_next_range(struct ma_state *mas, unsigned long max); diff --git a/include/linux/memory.h b/include/linux/memory.h index 0c214256216f..1c0c8f3c35fb 100644 --- a/include/linux/memory.h +++ b/include/linux/memory.h @@ -19,6 +19,7 @@ #include #include #include +#include #define MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) @@ -64,10 +65,22 @@ struct memory_group { }; }; +enum memory_block_state { + /* These states are exposed to userspace as text strings in sysfs */ + MEM_ONLINE, /* exposed to userspace */ + MEM_GOING_OFFLINE, /* exposed to userspace */ + MEM_OFFLINE, /* exposed to userspace */ + MEM_GOING_ONLINE, + MEM_CANCEL_ONLINE, + MEM_CANCEL_OFFLINE, + MEM_PREPARE_ONLINE, + MEM_FINISH_OFFLINE, +}; + struct memory_block { unsigned long start_section_nr; - unsigned long state; /* serialized by the dev->lock */ - int online_type; /* for passing data to online routine */ + enum memory_block_state state; /* serialized by the dev->lock */ + enum mmop online_type; /* for passing data to online routine */ int nid; /* NID for this memory block */ /* * The single zone of this memory block if all PFNs of this memory block @@ -89,16 +102,6 @@ int arch_get_memory_phys_device(unsigned long start_pfn); unsigned long memory_block_size_bytes(void); int set_memory_block_size_order(unsigned int order); -/* These states are exposed to userspace as text strings in sysfs */ -#define MEM_ONLINE (1<<0) /* exposed to userspace */ -#define MEM_GOING_OFFLINE (1<<1) /* exposed to userspace */ -#define MEM_OFFLINE (1<<2) /* exposed to userspace */ -#define MEM_GOING_ONLINE (1<<3) -#define MEM_CANCEL_ONLINE (1<<4) -#define MEM_CANCEL_OFFLINE (1<<5) -#define MEM_PREPARE_ONLINE (1<<6) -#define MEM_FINISH_OFFLINE (1<<7) - struct memory_notify { /* * The altmap_start_pfn and altmap_nr_pages fields are designated for diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 23f038a16231..bb01cd267131 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -16,11 +16,8 @@ struct resource; struct vmem_altmap; struct dev_pagemap; -#ifdef CONFIG_MEMORY_HOTPLUG -struct page *pfn_to_online_page(unsigned long pfn); - /* Types for control the zone type of onlined and offlined memory */ -enum { +enum mmop { /* Offline the memory. */ MMOP_OFFLINE = 0, /* Online the memory. Zone depends, see default_zone_for_pfn(). */ @@ -31,6 +28,9 @@ enum { MMOP_ONLINE_MOVABLE, }; +#ifdef CONFIG_MEMORY_HOTPLUG +struct page *pfn_to_online_page(unsigned long pfn); + /* Flags for add_memory() and friends to specify memory hotplug details. */ typedef int __bitwise mhp_t; @@ -302,9 +302,9 @@ static inline void __remove_memory(u64 start, u64 size) {} #ifdef CONFIG_MEMORY_HOTPLUG /* Default online_type (MMOP_*) when new memory blocks are added. */ -extern int mhp_get_default_online_type(void); -extern void mhp_set_default_online_type(int online_type); -extern void __ref free_area_init_core_hotplug(struct pglist_data *pgdat); +extern enum mmop mhp_get_default_online_type(void); +extern void mhp_set_default_online_type(enum mmop online_type); +int __ref free_area_init_core_hotplug(struct pglist_data *pgdat); extern int __add_memory(int nid, u64 start, u64 size, mhp_t mhp_flags); extern int add_memory(int nid, u64 start, u64 size, mhp_t mhp_flags); extern int add_memory_resource(int nid, struct resource *resource, @@ -326,8 +326,8 @@ extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap); extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map, unsigned long pnum); -extern struct zone *zone_for_pfn_range(int online_type, int nid, - struct memory_group *group, unsigned long start_pfn, +extern struct zone *zone_for_pfn_range(enum mmop online_type, + int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages); extern int arch_create_linear_mapping(int nid, u64 start, u64 size, struct mhp_params *params); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 85c2b3d358ec..647dcf144ae9 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -1142,6 +1142,7 @@ int mlx5_cmd_destroy_vport_lag(struct mlx5_core_dev *dev); bool mlx5_lag_is_roce(struct mlx5_core_dev *dev); bool mlx5_lag_is_sriov(struct mlx5_core_dev *dev); bool mlx5_lag_is_active(struct mlx5_core_dev *dev); +int mlx5_lag_query_bond_speed(struct mlx5_core_dev *dev, u32 *speed); bool mlx5_lag_mode_is_hash(struct mlx5_core_dev *dev); bool mlx5_lag_is_master(struct mlx5_core_dev *dev); bool mlx5_lag_is_shared_fdb(struct mlx5_core_dev *dev); diff --git a/include/linux/mlx5/fs.h b/include/linux/mlx5/fs.h index 6ac76a0c3827..7bf2449c53b2 100644 --- a/include/linux/mlx5/fs.h +++ b/include/linux/mlx5/fs.h @@ -116,6 +116,7 @@ enum mlx5_flow_namespace_type { }; enum { + FDB_DROP_ROOT, FDB_BYPASS_PATH, FDB_CRYPTO_INGRESS, FDB_TC_OFFLOAD, diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 61257b866fb2..0250fdbe082c 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1071,7 +1071,9 @@ struct mlx5_ifc_e_switch_cap_bits { u8 esw_shared_ingress_acl[0x1]; u8 esw_uplink_ingress_acl[0x1]; u8 root_ft_on_other_esw[0x1]; - u8 reserved_at_a[0xf]; + u8 reserved_at_a[0x1]; + u8 esw_vport_state_max_tx_speed[0x1]; + u8 reserved_at_c[0xd]; u8 esw_functions_changed[0x1]; u8 reserved_at_1a[0x1]; u8 ecpf_vport_exists[0x1]; @@ -5443,7 +5445,8 @@ struct mlx5_ifc_query_vport_state_out_bits { u8 reserved_at_40[0x20]; - u8 reserved_at_60[0x18]; + u8 max_tx_speed[0x10]; + u8 reserved_at_70[0x8]; u8 admin_state[0x4]; u8 state[0x4]; }; @@ -7776,7 +7779,7 @@ struct mlx5_ifc_modify_vport_state_in_bits { u8 reserved_at_41[0xf]; u8 vport_number[0x10]; - u8 reserved_at_60[0x10]; + u8 max_tx_speed[0x10]; u8 ingress_connect[0x1]; u8 egress_connect[0x1]; u8 ingress_connect_valid[0x1]; diff --git a/include/linux/mlx5/vport.h b/include/linux/mlx5/vport.h index b98aaa471ac2..e53a5d033d72 100644 --- a/include/linux/mlx5/vport.h +++ b/include/linux/mlx5/vport.h @@ -41,6 +41,8 @@ (MLX5_CAP_GEN(mdev, port_type) == MLX5_CAP_PORT_TYPE_ETH) && \ mlx5_core_is_pf(mdev)) +#define MLX5_MAX_TX_SPEED_UNIT 100 + enum { MLX5_CAP_INLINE_MODE_L2, MLX5_CAP_INLINE_MODE_VPORT_CONTEXT, @@ -58,6 +60,11 @@ enum { u8 mlx5_query_vport_state(struct mlx5_core_dev *mdev, u8 opmod, u16 vport); int mlx5_modify_vport_admin_state(struct mlx5_core_dev *mdev, u8 opmod, u16 vport, u8 other_vport, u8 state); +int mlx5_query_vport_max_tx_speed(struct mlx5_core_dev *mdev, u8 op_mod, + u16 vport, u8 other_vport, + u32 *max_tx_speed, u8 *state); +int mlx5_modify_vport_max_tx_speed(struct mlx5_core_dev *mdev, u8 opmod, + u16 vport, u8 other_vport, u16 max_tx_speed); int mlx5_query_nic_vport_mac_address(struct mlx5_core_dev *mdev, u16 vport, bool other, u8 *addr); int mlx5_query_mac_address(struct mlx5_core_dev *mdev, u8 *addr); diff --git a/include/linux/mmc/sdio_ids.h b/include/linux/mmc/sdio_ids.h index 673cbdf43453..dce89c110691 100644 --- a/include/linux/mmc/sdio_ids.h +++ b/include/linux/mmc/sdio_ids.h @@ -111,6 +111,7 @@ #define SDIO_VENDOR_ID_MEDIATEK 0x037a #define SDIO_DEVICE_ID_MEDIATEK_MT7663 0x7663 #define SDIO_DEVICE_ID_MEDIATEK_MT7668 0x7668 +#define SDIO_DEVICE_ID_MEDIATEK_MT7902 0x790a #define SDIO_DEVICE_ID_MEDIATEK_MT7961 0x7961 #define SDIO_VENDOR_ID_MICROCHIP_WILC 0x0296 diff --git a/include/linux/mroute_base.h b/include/linux/mroute_base.h index 0075f6e5c3da..f887ffba95ca 100644 --- a/include/linux/mroute_base.h +++ b/include/linux/mroute_base.h @@ -227,6 +227,7 @@ struct mr_table_ops { /** * struct mr_table - a multicast routing table + * @work: used for table destruction * @list: entry within a list of multicast routing tables * @net: net where this table belongs * @ops: protocol specific operations @@ -244,6 +245,7 @@ struct mr_table_ops { * @mroute_reg_vif_num: PIM-device vif index */ struct mr_table { + struct rcu_work work; struct list_head list; possible_net_t net; struct mr_table_ops ops; @@ -275,6 +277,7 @@ void vif_device_init(struct vif_device *v, unsigned short flags, unsigned short get_iflink_mask); +void mr_table_free(struct mr_table *mrt); struct mr_table * mr_table_alloc(struct net *net, u32 id, struct mr_table_ops *ops, diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h index 972059a016c9..e8a3ae2cebb6 100644 --- a/include/linux/nfs_xdr.h +++ b/include/linux/nfs_xdr.h @@ -351,7 +351,7 @@ struct nfs4_layoutreturn { struct nfs4_xdr_opaque_data ld_private; }; -#define PNFS_LAYOUTSTATS_MAXSIZE 256 +#define PNFS_LAYOUTSTATS_MAXSIZE 384 struct nfs42_layoutstat_args; struct nfs42_layoutstat_devinfo; diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 655d194f8e72..b9d3dbe2564d 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -2332,4 +2332,8 @@ enum nvme_pr_change_ptpl { #define NVME_PR_IGNORE_KEY (1 << 3) +/* Section 8.3.4.5.2 of the NVMe 2.1 */ +#define NVME_AUTH_DHCHAP_MAX_HASH_IDS 30 +#define NVME_AUTH_DHCHAP_MAX_DH_IDS 30 + #endif /* _LINUX_NVME_H */ diff --git a/include/linux/pci.h b/include/linux/pci.h index 505bf6e009b1..148b71af80b3 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -72,12 +72,24 @@ /* return bus from PCI devid = ((u16)bus_number) << 8) | devfn */ #define PCI_BUS_NUM(x) (((x) >> 8) & 0xff) +/* + * PCI_SLOT_ALL_DEVICES indicates a slot that covers all devices on the bus. + * Used for PCIe hotplug where the physical slot is the entire secondary bus, + * and, if ARI Forwarding is enabled, functions may appear to be on multiple + * devices. + */ +#define PCI_SLOT_ALL_DEVICES 0xfeff + +/* Used to identify a slot as a placeholder */ +#define PCI_SLOT_PLACEHOLDER 0xffff + /* pci_slot represents a physical slot */ struct pci_slot { struct pci_bus *bus; /* Bus this slot is on */ struct list_head list; /* Node in list of slots */ struct hotplug_slot *hotplug; /* Hotplug info (move here) */ - unsigned char number; /* PCI_SLOT(pci_dev->devfn) */ + u16 number; /* Device nr, or PCI_SLOT_ALL_DEVICES */ + unsigned int per_func_slot:1; /* Allow per function slot */ struct kobject kobj; }; diff --git a/include/linux/pm_runtime.h b/include/linux/pm_runtime.h index 0b436e15f4cd..911d7a4d32c1 100644 --- a/include/linux/pm_runtime.h +++ b/include/linux/pm_runtime.h @@ -637,6 +637,30 @@ DEFINE_GUARD_COND(pm_runtime_active_auto, _try, DEFINE_GUARD_COND(pm_runtime_active_auto, _try_enabled, pm_runtime_resume_and_get(_T), _RET == 0) +/* ACQUIRE() wrapper macros for the guards defined above. */ + +#define PM_RUNTIME_ACQUIRE(_dev, _var) \ + ACQUIRE(pm_runtime_active_try, _var)(_dev) + +#define PM_RUNTIME_ACQUIRE_AUTOSUSPEND(_dev, _var) \ + ACQUIRE(pm_runtime_active_auto_try, _var)(_dev) + +#define PM_RUNTIME_ACQUIRE_IF_ENABLED(_dev, _var) \ + ACQUIRE(pm_runtime_active_try_enabled, _var)(_dev) + +#define PM_RUNTIME_ACQUIRE_IF_ENABLED_AUTOSUSPEND(_dev, _var) \ + ACQUIRE(pm_runtime_active_auto_try_enabled, _var)(_dev) + +/* + * ACQUIRE_ERR() wrapper macro for guard pm_runtime_active. + * + * Always check PM_RUNTIME_ACQUIRE_ERR() after using one of the + * PM_RUNTIME_ACQUIRE*() macros defined above (yes, it can be used with + * any of them) and if it is nonzero, avoid accessing the given device. + */ +#define PM_RUNTIME_ACQUIRE_ERR(_var_ptr) \ + ACQUIRE_ERR(pm_runtime_active, _var_ptr) + /** * pm_runtime_put_sync - Drop device usage counter and run "idle check" if 0. * @dev: Target device. diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h index b4795698d8c2..de98462d5888 100644 --- a/include/linux/remoteproc.h +++ b/include/linux/remoteproc.h @@ -37,279 +37,13 @@ #include #include +#include #include #include #include #include #include - -/** - * struct resource_table - firmware resource table header - * @ver: version number - * @num: number of resource entries - * @reserved: reserved (must be zero) - * @offset: array of offsets pointing at the various resource entries - * - * A resource table is essentially a list of system resources required - * by the remote processor. It may also include configuration entries. - * If needed, the remote processor firmware should contain this table - * as a dedicated ".resource_table" ELF section. - * - * Some resources entries are mere announcements, where the host is informed - * of specific remoteproc configuration. Other entries require the host to - * do something (e.g. allocate a system resource). Sometimes a negotiation - * is expected, where the firmware requests a resource, and once allocated, - * the host should provide back its details (e.g. address of an allocated - * memory region). - * - * The header of the resource table, as expressed by this structure, - * contains a version number (should we need to change this format in the - * future), the number of available resource entries, and their offsets - * in the table. - * - * Immediately following this header are the resource entries themselves, - * each of which begins with a resource entry header (as described below). - */ -struct resource_table { - u32 ver; - u32 num; - u32 reserved[2]; - u32 offset[]; -} __packed; - -/** - * struct fw_rsc_hdr - firmware resource entry header - * @type: resource type - * @data: resource data - * - * Every resource entry begins with a 'struct fw_rsc_hdr' header providing - * its @type. The content of the entry itself will immediately follow - * this header, and it should be parsed according to the resource type. - */ -struct fw_rsc_hdr { - u32 type; - u8 data[]; -} __packed; - -/** - * enum fw_resource_type - types of resource entries - * - * @RSC_CARVEOUT: request for allocation of a physically contiguous - * memory region. - * @RSC_DEVMEM: request to iommu_map a memory-based peripheral. - * @RSC_TRACE: announces the availability of a trace buffer into which - * the remote processor will be writing logs. - * @RSC_VDEV: declare support for a virtio device, and serve as its - * virtio header. - * @RSC_LAST: just keep this one at the end of standard resources - * @RSC_VENDOR_START: start of the vendor specific resource types range - * @RSC_VENDOR_END: end of the vendor specific resource types range - * - * For more details regarding a specific resource type, please see its - * dedicated structure below. - * - * Please note that these values are used as indices to the rproc_handle_rsc - * lookup table, so please keep them sane. Moreover, @RSC_LAST is used to - * check the validity of an index before the lookup table is accessed, so - * please update it as needed. - */ -enum fw_resource_type { - RSC_CARVEOUT = 0, - RSC_DEVMEM = 1, - RSC_TRACE = 2, - RSC_VDEV = 3, - RSC_LAST = 4, - RSC_VENDOR_START = 128, - RSC_VENDOR_END = 512, -}; - -#define FW_RSC_ADDR_ANY (-1) - -/** - * struct fw_rsc_carveout - physically contiguous memory request - * @da: device address - * @pa: physical address - * @len: length (in bytes) - * @flags: iommu protection flags - * @reserved: reserved (must be zero) - * @name: human-readable name of the requested memory region - * - * This resource entry requests the host to allocate a physically contiguous - * memory region. - * - * These request entries should precede other firmware resource entries, - * as other entries might request placing other data objects inside - * these memory regions (e.g. data/code segments, trace resource entries, ...). - * - * Allocating memory this way helps utilizing the reserved physical memory - * (e.g. CMA) more efficiently, and also minimizes the number of TLB entries - * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB - * pressure is important; it may have a substantial impact on performance. - * - * If the firmware is compiled with static addresses, then @da should specify - * the expected device address of this memory region. If @da is set to - * FW_RSC_ADDR_ANY, then the host will dynamically allocate it, and then - * overwrite @da with the dynamically allocated address. - * - * We will always use @da to negotiate the device addresses, even if it - * isn't using an iommu. In that case, though, it will obviously contain - * physical addresses. - * - * Some remote processors needs to know the allocated physical address - * even if they do use an iommu. This is needed, e.g., if they control - * hardware accelerators which access the physical memory directly (this - * is the case with OMAP4 for instance). In that case, the host will - * overwrite @pa with the dynamically allocated physical address. - * Generally we don't want to expose physical addresses if we don't have to - * (remote processors are generally _not_ trusted), so we might want to - * change this to happen _only_ when explicitly required by the hardware. - * - * @flags is used to provide IOMMU protection flags, and @name should - * (optionally) contain a human readable name of this carveout region - * (mainly for debugging purposes). - */ -struct fw_rsc_carveout { - u32 da; - u32 pa; - u32 len; - u32 flags; - u32 reserved; - u8 name[32]; -} __packed; - -/** - * struct fw_rsc_devmem - iommu mapping request - * @da: device address - * @pa: physical address - * @len: length (in bytes) - * @flags: iommu protection flags - * @reserved: reserved (must be zero) - * @name: human-readable name of the requested region to be mapped - * - * This resource entry requests the host to iommu map a physically contiguous - * memory region. This is needed in case the remote processor requires - * access to certain memory-based peripherals; _never_ use it to access - * regular memory. - * - * This is obviously only needed if the remote processor is accessing memory - * via an iommu. - * - * @da should specify the required device address, @pa should specify - * the physical address we want to map, @len should specify the size of - * the mapping and @flags is the IOMMU protection flags. As always, @name may - * (optionally) contain a human readable name of this mapping (mainly for - * debugging purposes). - * - * Note: at this point we just "trust" those devmem entries to contain valid - * physical addresses, but this isn't safe and will be changed: eventually we - * want remoteproc implementations to provide us ranges of physical addresses - * the firmware is allowed to request, and not allow firmwares to request - * access to physical addresses that are outside those ranges. - */ -struct fw_rsc_devmem { - u32 da; - u32 pa; - u32 len; - u32 flags; - u32 reserved; - u8 name[32]; -} __packed; - -/** - * struct fw_rsc_trace - trace buffer declaration - * @da: device address - * @len: length (in bytes) - * @reserved: reserved (must be zero) - * @name: human-readable name of the trace buffer - * - * This resource entry provides the host information about a trace buffer - * into which the remote processor will write log messages. - * - * @da specifies the device address of the buffer, @len specifies - * its size, and @name may contain a human readable name of the trace buffer. - * - * After booting the remote processor, the trace buffers are exposed to the - * user via debugfs entries (called trace0, trace1, etc..). - */ -struct fw_rsc_trace { - u32 da; - u32 len; - u32 reserved; - u8 name[32]; -} __packed; - -/** - * struct fw_rsc_vdev_vring - vring descriptor entry - * @da: device address - * @align: the alignment between the consumer and producer parts of the vring - * @num: num of buffers supported by this vring (must be power of two) - * @notifyid: a unique rproc-wide notify index for this vring. This notify - * index is used when kicking a remote processor, to let it know that this - * vring is triggered. - * @pa: physical address - * - * This descriptor is not a resource entry by itself; it is part of the - * vdev resource type (see below). - * - * Note that @da should either contain the device address where - * the remote processor is expecting the vring, or indicate that - * dynamically allocation of the vring's device address is supported. - */ -struct fw_rsc_vdev_vring { - u32 da; - u32 align; - u32 num; - u32 notifyid; - u32 pa; -} __packed; - -/** - * struct fw_rsc_vdev - virtio device header - * @id: virtio device id (as in virtio_ids.h) - * @notifyid: a unique rproc-wide notify index for this vdev. This notify - * index is used when kicking a remote processor, to let it know that the - * status/features of this vdev have changes. - * @dfeatures: specifies the virtio device features supported by the firmware - * @gfeatures: a place holder used by the host to write back the - * negotiated features that are supported by both sides. - * @config_len: the size of the virtio config space of this vdev. The config - * space lies in the resource table immediate after this vdev header. - * @status: a place holder where the host will indicate its virtio progress. - * @num_of_vrings: indicates how many vrings are described in this vdev header - * @reserved: reserved (must be zero) - * @vring: an array of @num_of_vrings entries of 'struct fw_rsc_vdev_vring'. - * - * This resource is a virtio device header: it provides information about - * the vdev, and is then used by the host and its peer remote processors - * to negotiate and share certain virtio properties. - * - * By providing this resource entry, the firmware essentially asks remoteproc - * to statically allocate a vdev upon registration of the rproc (dynamic vdev - * allocation is not yet supported). - * - * Note: - * 1. unlike virtualization systems, the term 'host' here means - * the Linux side which is running remoteproc to control the remote - * processors. We use the name 'gfeatures' to comply with virtio's terms, - * though there isn't really any virtualized guest OS here: it's the host - * which is responsible for negotiating the final features. - * Yeah, it's a bit confusing. - * - * 2. immediately following this structure is the virtio config space for - * this vdev (which is specific to the vdev; for more info, read the virtio - * spec). The size of the config space is specified by @config_len. - */ -struct fw_rsc_vdev { - u32 id; - u32 notifyid; - u32 dfeatures; - u32 gfeatures; - u32 config_len; - u8 status; - u8 num_of_vrings; - u8 reserved[2]; - struct fw_rsc_vdev_vring vring[]; -} __packed; +#include struct rproc; @@ -412,7 +146,6 @@ struct rproc_ops { * a message. * @RPROC_RUNNING: device is up and running * @RPROC_CRASHED: device has crashed; need to start recovery - * @RPROC_DELETED: device is deleted * @RPROC_ATTACHED: device has been booted by another entity and the core * has attached to it * @RPROC_DETACHED: device has been booted by another entity and waiting @@ -430,10 +163,9 @@ enum rproc_state { RPROC_SUSPENDED = 1, RPROC_RUNNING = 2, RPROC_CRASHED = 3, - RPROC_DELETED = 4, - RPROC_ATTACHED = 5, - RPROC_DETACHED = 6, - RPROC_LAST = 7, + RPROC_ATTACHED = 4, + RPROC_DETACHED = 5, + RPROC_LAST = 6, }; /** @@ -526,7 +258,10 @@ enum rproc_features { * @subdevs: list of subdevices, to following the running state * @notifyids: idr for dynamically assigning rproc-wide unique notify ids * @index: index of this rproc device + * @attach_work: workqueue for attaching rproc * @crash_handler: workqueue for handling a crash + * @crash_handler_lock: serializes crash handler queueing and deletion + * @deleting: remoteproc deletion has begun * @crash_cnt: crash counter * @recovery_disabled: flag that state if recovery was disabled * @max_notifyid: largest allocated notify id. @@ -568,7 +303,10 @@ struct rproc { struct list_head subdevs; struct idr notifyids; int index; + struct work_struct attach_work; struct work_struct crash_handler; + spinlock_t crash_handler_lock; + bool deleting; unsigned int crash_cnt; bool recovery_disabled; int max_notifyid; diff --git a/include/linux/rsc_table.h b/include/linux/rsc_table.h new file mode 100644 index 000000000000..4cef11a2e3a2 --- /dev/null +++ b/include/linux/rsc_table.h @@ -0,0 +1,364 @@ +/* + * Resource table and its types data structure + * + * Copyright(c) 2011 Texas Instruments, Inc. + * Copyright(c) 2011 Google, Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name Texas Instruments nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef RSC_TABLE_H +#define RSC_TABLE_H + +/** + * struct resource_table - firmware resource table header + * @ver: version number + * @num: number of resource entries + * @reserved: reserved (must be zero) + * @offset: array of offsets pointing at the various resource entries + * + * A resource table is essentially a list of system resources required + * by the remote processor. It may also include configuration entries. + * If needed, the remote processor firmware should contain this table + * as a dedicated ".resource_table" ELF section. + * + * Some resources entries are mere announcements, where the host is informed + * of specific remoteproc configuration. Other entries require the host to + * do something (e.g. allocate a system resource). Sometimes a negotiation + * is expected, where the firmware requests a resource, and once allocated, + * the host should provide back its details (e.g. address of an allocated + * memory region). + * + * The header of the resource table, as expressed by this structure, + * contains a version number (should we need to change this format in the + * future), the number of available resource entries, and their offsets + * in the table. + * + * Immediately following this header are the resource entries themselves, + * each of which begins with a resource entry header (as described below). + */ +struct resource_table { + u32 ver; + u32 num; + u32 reserved[2]; + u32 offset[]; +} __packed; + +/** + * struct fw_rsc_hdr - firmware resource entry header + * @type: resource type + * @data: resource data + * + * Every resource entry begins with a 'struct fw_rsc_hdr' header providing + * its @type. The content of the entry itself will immediately follow + * this header, and it should be parsed according to the resource type. + */ +struct fw_rsc_hdr { + u32 type; + u8 data[]; +} __packed; + +/** + * enum fw_resource_type - types of resource entries + * + * @RSC_CARVEOUT: request for allocation of a physically contiguous + * memory region. + * @RSC_DEVMEM: request to iommu_map a memory-based peripheral. + * @RSC_TRACE: announces the availability of a trace buffer into which + * the remote processor will be writing logs. + * @RSC_VDEV: declare support for a virtio device, and serve as its + * virtio header. + * @RSC_LAST: just keep this one at the end of standard resources + * @RSC_VENDOR_START: start of the vendor specific resource types range + * @RSC_VENDOR_END: end of the vendor specific resource types range + * + * For more details regarding a specific resource type, please see its + * dedicated structure below. + * + * Please note that these values are used as indices to the rproc_handle_rsc + * lookup table, so please keep them sane. Moreover, @RSC_LAST is used to + * check the validity of an index before the lookup table is accessed, so + * please update it as needed. + */ +enum fw_resource_type { + RSC_CARVEOUT = 0, + RSC_DEVMEM = 1, + RSC_TRACE = 2, + RSC_VDEV = 3, + RSC_LAST = 4, + RSC_VENDOR_START = 128, + RSC_VENDOR_END = 512, +}; + +#define FW_RSC_ADDR_ANY (-1) + +/** + * struct fw_rsc_carveout - physically contiguous memory request + * @da: device address + * @pa: physical address + * @len: length (in bytes) + * @flags: iommu protection flags + * @reserved: reserved (must be zero) + * @name: human-readable name of the requested memory region + * + * This resource entry requests the host to allocate a physically contiguous + * memory region. + * + * These request entries should precede other firmware resource entries, + * as other entries might request placing other data objects inside + * these memory regions (e.g. data/code segments, trace resource entries, ...). + * + * Allocating memory this way helps utilizing the reserved physical memory + * (e.g. CMA) more efficiently, and also minimizes the number of TLB entries + * needed to map it (in case @rproc is using an IOMMU). Reducing the TLB + * pressure is important; it may have a substantial impact on performance. + * + * If the firmware is compiled with static addresses, then @da should specify + * the expected device address of this memory region. If @da is set to + * FW_RSC_ADDR_ANY, then the host will dynamically allocate it, and then + * overwrite @da with the dynamically allocated address. + * + * We will always use @da to negotiate the device addresses, even if it + * isn't using an iommu. In that case, though, it will obviously contain + * physical addresses. + * + * Some remote processors needs to know the allocated physical address + * even if they do use an iommu. This is needed, e.g., if they control + * hardware accelerators which access the physical memory directly (this + * is the case with OMAP4 for instance). In that case, the host will + * overwrite @pa with the dynamically allocated physical address. + * Generally we don't want to expose physical addresses if we don't have to + * (remote processors are generally _not_ trusted), so we might want to + * change this to happen _only_ when explicitly required by the hardware. + * + * @flags is used to provide IOMMU protection flags, and @name should + * (optionally) contain a human readable name of this carveout region + * (mainly for debugging purposes). + */ +struct fw_rsc_carveout { + u32 da; + u32 pa; + u32 len; + u32 flags; + u32 reserved; + u8 name[32]; +} __packed; + +/** + * struct fw_rsc_devmem - iommu mapping request + * @da: device address + * @pa: physical address + * @len: length (in bytes) + * @flags: iommu protection flags + * @reserved: reserved (must be zero) + * @name: human-readable name of the requested region to be mapped + * + * This resource entry requests the host to iommu map a physically contiguous + * memory region. This is needed in case the remote processor requires + * access to certain memory-based peripherals; _never_ use it to access + * regular memory. + * + * This is obviously only needed if the remote processor is accessing memory + * via an iommu. + * + * @da should specify the required device address, @pa should specify + * the physical address we want to map, @len should specify the size of + * the mapping and @flags is the IOMMU protection flags. As always, @name may + * (optionally) contain a human readable name of this mapping (mainly for + * debugging purposes). + * + * Note: at this point we just "trust" those devmem entries to contain valid + * physical addresses, but this isn't safe and will be changed: eventually we + * want remoteproc implementations to provide us ranges of physical addresses + * the firmware is allowed to request, and not allow firmwares to request + * access to physical addresses that are outside those ranges. + */ +struct fw_rsc_devmem { + u32 da; + u32 pa; + u32 len; + u32 flags; + u32 reserved; + u8 name[32]; +} __packed; + +/** + * struct fw_rsc_trace - trace buffer declaration + * @da: device address + * @len: length (in bytes) + * @reserved: reserved (must be zero) + * @name: human-readable name of the trace buffer + * + * This resource entry provides the host information about a trace buffer + * into which the remote processor will write log messages. + * + * @da specifies the device address of the buffer, @len specifies + * its size, and @name may contain a human readable name of the trace buffer. + * + * After booting the remote processor, the trace buffers are exposed to the + * user via debugfs entries (called trace0, trace1, etc..). + */ +struct fw_rsc_trace { + u32 da; + u32 len; + u32 reserved; + u8 name[32]; +} __packed; + +/** + * struct fw_rsc_vdev_vring - vring descriptor entry + * @da: device address + * @align: the alignment between the consumer and producer parts of the vring + * @num: num of buffers supported by this vring (must be power of two) + * @notifyid: a unique rproc-wide notify index for this vring. This notify + * index is used when kicking a remote processor, to let it know that this + * vring is triggered. + * @pa: physical address + * + * This descriptor is not a resource entry by itself; it is part of the + * vdev resource type (see below). + * + * Note that @da should either contain the device address where + * the remote processor is expecting the vring, or indicate that + * dynamically allocation of the vring's device address is supported. + */ +struct fw_rsc_vdev_vring { + u32 da; + u32 align; + u32 num; + u32 notifyid; + u32 pa; +} __packed; + +/** + * struct fw_rsc_vdev - virtio device header + * @id: virtio device id (as in virtio_ids.h) + * @notifyid: a unique rproc-wide notify index for this vdev. This notify + * index is used when kicking a remote processor, to let it know that the + * status/features of this vdev have changes. + * @dfeatures: specifies the virtio device features supported by the firmware + * @gfeatures: a place holder used by the host to write back the + * negotiated features that are supported by both sides. + * @config_len: the size of the virtio config space of this vdev. The config + * space lies in the resource table immediate after this vdev header. + * @status: a place holder where the host will indicate its virtio progress. + * @num_of_vrings: indicates how many vrings are described in this vdev header + * @reserved: reserved (must be zero) + * @vring: an array of @num_of_vrings entries of 'struct fw_rsc_vdev_vring'. + * + * This resource is a virtio device header: it provides information about + * the vdev, and is then used by the host and its peer remote processors + * to negotiate and share certain virtio properties. + * + * By providing this resource entry, the firmware essentially asks remoteproc + * to statically allocate a vdev upon registration of the rproc (dynamic vdev + * allocation is not yet supported). + * + * Note: + * 1. unlike virtualization systems, the term 'host' here means + * the Linux side which is running remoteproc to control the remote + * processors. We use the name 'gfeatures' to comply with virtio's terms, + * though there isn't really any virtualized guest OS here: it's the host + * which is responsible for negotiating the final features. + * Yeah, it's a bit confusing. + * + * 2. immediately following this structure is the virtio config space for + * this vdev (which is specific to the vdev; for more info, read the virtio + * spec). The size of the config space is specified by @config_len. + */ +struct fw_rsc_vdev { + u32 id; + u32 notifyid; + u32 dfeatures; + u32 gfeatures; + u32 config_len; + u8 status; + u8 num_of_vrings; + u8 reserved[2]; + struct fw_rsc_vdev_vring vring[]; +} __packed; + +/** + * rsc_table_for_each_entry() - iterate over all entries in a resource table + * @table: pointer to the resource table + * @table_sz: total size of the table buffer in bytes + * @dev: device used for error logging + * @cb: callback invoked for each entry: + * @type - value from enum fw_resource_type + * @rsc - pointer to the entry payload (past struct fw_rsc_hdr) + * @offset - byte offset of the payload within the table; callers + * that write back into the table (e.g. to record a + * dynamically allocated address) use this to locate the + * entry for later update + * @avail - bytes available in the payload + * @data - caller-supplied private pointer + * Return 0 to continue iteration, non-zero to stop. + * @data: private pointer forwarded to @cb on every call + * + * Iterates over every resource entry in @table, performing the standard + * truncation check, and invokes @cb for each one. Iteration stops on the + * first non-zero return from @cb or on a malformed table. + * + * Returns 0 after a complete iteration, -EINVAL if the table is truncated, + * or the first non-zero value returned by @cb. + */ +static inline int rsc_table_for_each_entry(struct resource_table *table, + size_t table_sz, + struct device *dev, + int (*cb)(u32 type, void *rsc, + int offset, int avail, + void *data), + void *data) { + int i, ret; + + for (i = 0; i < table->num; i++) { + u32 offset = table->offset[i]; + struct fw_rsc_hdr *hdr; + int avail, rsc_offset; + void *rsc; + + if (offset < sizeof(*table) || offset >= table_sz || + table_sz - offset < sizeof(*hdr)) { + dev_err(dev, "rsc table is truncated\n"); + return -EINVAL; + } + + hdr = (void *)table + offset; + avail = table_sz - offset - sizeof(*hdr); + rsc_offset = offset + sizeof(*hdr); + rsc = (void *)hdr + sizeof(*hdr); + + ret = cb(hdr->type, rsc, rsc_offset, avail, data); + if (ret) + return ret; + } + + return 0; +} + +#endif /* RSC_TABLE_H */ diff --git a/include/linux/rtsx_usb.h b/include/linux/rtsx_usb.h index 276b509c03e3..0fc5a74700a8 100644 --- a/include/linux/rtsx_usb.h +++ b/include/linux/rtsx_usb.h @@ -61,6 +61,9 @@ struct rtsx_ucr { struct timer_list sg_timer; struct mutex dev_mutex; + + u16 card_status_cache; + bool card_status_valid; }; /* buffer size */ diff --git a/include/linux/sched/rt.h b/include/linux/sched/rt.h index 4e3338103654..922935cc3383 100644 --- a/include/linux/sched/rt.h +++ b/include/linux/sched/rt.h @@ -52,8 +52,10 @@ static inline bool rt_or_dl_task_policy(struct task_struct *tsk) #ifdef CONFIG_RT_MUTEXES extern void rt_mutex_pre_schedule(void); +extern void rt_mutex_futex_pre_schedule(void); extern void rt_mutex_schedule(void); extern void rt_mutex_post_schedule(void); +extern void rt_mutex_futex_post_schedule(void); /* * Must hold either p->pi_lock or task_rq(p)->lock. diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h index 4cc52698e214..8d7e5521f7cd 100644 --- a/include/linux/sched/user.h +++ b/include/linux/sched/user.h @@ -25,7 +25,8 @@ struct user_struct { #if defined(CONFIG_PERF_EVENTS) || defined(CONFIG_BPF_SYSCALL) || \ defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \ - defined(CONFIG_VFIO_PCI_ZDEV_KVM) || IS_ENABLED(CONFIG_IOMMUFD) + defined(CONFIG_VFIO_PCI_ZDEV_KVM) || IS_ENABLED(CONFIG_IOMMUFD) || \ + defined(CONFIG_SECRETMEM) atomic_long_t locked_vm; #endif #ifdef CONFIG_WATCH_QUEUE diff --git a/include/linux/secure_boot.h b/include/linux/secure_boot.h new file mode 100644 index 000000000000..d17e92351567 --- /dev/null +++ b/include/linux/secure_boot.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved. + * + * Author: Coiby Xu + */ + +#ifndef _LINUX_SECURE_BOOT_H +#define _LINUX_SECURE_BOOT_H + +#include + +#ifdef CONFIG_HAVE_ARCH_GET_SECUREBOOT +/* + * Returns true if the platform secure boot is enabled. + * Returns false if disabled or not supported. + */ +bool arch_get_secureboot(void); +#else +static inline bool arch_get_secureboot(void) { return false; } +#endif + +#endif /* _LINUX_SECURE_BOOT_H */ diff --git a/include/linux/serial_core.h b/include/linux/serial_core.h index 110ad4e2aef9..bdc214386e4a 100644 --- a/include/linux/serial_core.h +++ b/include/linux/serial_core.h @@ -533,6 +533,7 @@ struct uart_port { #define UPF_HARD_FLOW ((__force upf_t) (UPF_AUTO_CTS | UPF_AUTO_RTS)) /* Port has hardware-assisted s/w flow control */ #define UPF_SOFT_FLOW ((__force upf_t) BIT_ULL(22)) +/* Deprecated: use uart_set_cons_flow_enabled()/uart_cons_flow_enabled() instead. */ #define UPF_CONS_FLOW ((__force upf_t) BIT_ULL(23)) #define UPF_SHARE_IRQ ((__force upf_t) BIT_ULL(24)) #define UPF_EXAR_EFR ((__force upf_t) BIT_ULL(25)) @@ -567,6 +568,7 @@ struct uart_port { #define UPSTAT_SYNC_FIFO ((__force upstat_t) (1 << 5)) bool hw_stopped; /* sw-assisted CTS flow state */ + bool cons_flow; /* user specified console flow control */ unsigned int mctrl; /* current modem ctrl settings */ unsigned int frame_time; /* frame timing in ns */ unsigned int type; /* port type */ @@ -1163,6 +1165,24 @@ static inline bool uart_softcts_mode(struct uart_port *uport) return ((uport->status & mask) == UPSTAT_CTS_ENABLE); } +static inline void uart_set_cons_flow_enabled(struct uart_port *uport, bool enabled) +{ + uport->cons_flow = enabled; +} + +static inline bool uart_cons_flow_enabled(const struct uart_port *uport) +{ + return uport->cons_flow; +} + +static inline bool uart_console_hwflow_active(struct uart_port *uport) +{ + return uart_console(uport) && + !(uport->rs485.flags & SER_RS485_ENABLED) && + uart_cons_flow_enabled(uport) && + uart_cts_enabled(uport); +} + /* * The following are helper functions for the low level drivers. */ diff --git a/include/linux/soc/airoha/airoha_offload.h b/include/linux/soc/airoha/airoha_offload.h index d4f6e8124a49..7589fccfeef6 100644 --- a/include/linux/soc/airoha/airoha_offload.h +++ b/include/linux/soc/airoha/airoha_offload.h @@ -6,6 +6,7 @@ #ifndef AIROHA_OFFLOAD_H #define AIROHA_OFFLOAD_H +#include #include #include diff --git a/include/linux/soc/qcom/ubwc.h b/include/linux/soc/qcom/ubwc.h index d9dfc9edc1b2..d3bb18990f84 100644 --- a/include/linux/soc/qcom/ubwc.h +++ b/include/linux/soc/qcom/ubwc.h @@ -8,6 +8,7 @@ #define __QCOM_UBWC_H__ #include +#include #include #include diff --git a/include/linux/soundwire/sdw.h b/include/linux/soundwire/sdw.h index e6a3476bcef1..bf2ddd429620 100644 --- a/include/linux/soundwire/sdw.h +++ b/include/linux/soundwire/sdw.h @@ -1093,6 +1093,8 @@ int sdw_slave_get_current_bank(struct sdw_slave *sdev); int sdw_slave_get_scale_index(struct sdw_slave *slave, u8 *base); +int sdw_slave_wait_for_init(struct sdw_slave *slave, int timeout_ms); + /* messaging and data APIs */ int sdw_read(struct sdw_slave *slave, u32 addr); int sdw_write(struct sdw_slave *slave, u32 addr, u8 value); @@ -1136,6 +1138,12 @@ static inline int sdw_slave_get_current_bank(struct sdw_slave *sdev) return -EINVAL; } +static inline int sdw_slave_wait_for_init(struct sdw_slave *slave, int timeout_ms) +{ + WARN_ONCE(1, "SoundWire API is disabled"); + return -EINVAL; +} + /* messaging and data APIs */ static inline int sdw_read(struct sdw_slave *slave, u32 addr) { diff --git a/include/linux/syscore_ops.h b/include/linux/syscore_ops.h index ae4d48e4c970..ac6d71be5c38 100644 --- a/include/linux/syscore_ops.h +++ b/include/linux/syscore_ops.h @@ -11,14 +11,19 @@ #include struct syscore_ops { + int (*suspend)(void *data); + void (*resume)(void *data); + void (*shutdown)(void *data); +}; + +struct syscore { struct list_head node; - int (*suspend)(void); - void (*resume)(void); - void (*shutdown)(void); + const struct syscore_ops *ops; + void *data; }; -extern void register_syscore_ops(struct syscore_ops *ops); -extern void unregister_syscore_ops(struct syscore_ops *ops); +extern void register_syscore(struct syscore *syscore); +extern void unregister_syscore(struct syscore *syscore); #ifdef CONFIG_PM_SLEEP extern int syscore_suspend(void); extern void syscore_resume(void); diff --git a/include/linux/time_namespace.h b/include/linux/time_namespace.h index c514d0e5a45c..06a83e198ded 100644 --- a/include/linux/time_namespace.h +++ b/include/linux/time_namespace.h @@ -8,6 +8,7 @@ #include #include #include +#include struct user_namespace; extern struct user_namespace init_user_ns; @@ -38,8 +39,6 @@ static inline struct time_namespace *to_time_ns(struct ns_common *ns) return container_of(ns, struct time_namespace, ns); } void __init time_ns_init(void); -extern int vdso_join_timens(struct task_struct *task, - struct time_namespace *ns); extern void timens_commit(struct task_struct *tsk, struct time_namespace *ns); static inline struct time_namespace *get_time_ns(struct time_namespace *ns) @@ -117,12 +116,6 @@ static inline void __init time_ns_init(void) { } -static inline int vdso_join_timens(struct task_struct *task, - struct time_namespace *ns) -{ - return 0; -} - static inline void timens_commit(struct task_struct *tsk, struct time_namespace *ns) { @@ -175,4 +168,6 @@ static inline ktime_t timens_ktime_to_host(clockid_t clockid, ktime_t tim) } #endif +DEFINE_FREE(time_ns, struct time_namespace *, if (_T) put_time_ns(_T)) + #endif /* _LINUX_TIMENS_H */ diff --git a/include/linux/usb.h b/include/linux/usb.h index 375ee6e202db..f2adf7634095 100644 --- a/include/linux/usb.h +++ b/include/linux/usb.h @@ -1185,7 +1185,8 @@ extern ssize_t usb_show_dynids(struct usb_dynids *dynids, char *buf); * interface. It may also use usb_set_interface() to specify the * appropriate altsetting. If unwilling to manage the interface, * return -ENODEV, if genuine IO errors occurred, an appropriate - * negative errno value. + * negative errno value. The usb_device_id parameter is only valid during + * probe. * @disconnect: Called when the interface is no longer accessible, usually * because its device has been (or is being) disconnected or the * driver module is being unloaded. diff --git a/include/linux/virtio.h b/include/linux/virtio.h index d3f723b875a7..9918538f59ca 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -204,6 +204,7 @@ int virtio_device_freeze(struct virtio_device *dev); int virtio_device_restore(struct virtio_device *dev); #endif void virtio_reset_device(struct virtio_device *dev); +void virtio_device_shutdown(struct virtio_device *dev); int virtio_device_reset_prepare(struct virtio_device *dev); int virtio_device_reset_done(struct virtio_device *dev); diff --git a/include/linux/virtio_net.h b/include/linux/virtio_net.h index f36d21b5bc19..c381b916c1b5 100644 --- a/include/linux/virtio_net.h +++ b/include/linux/virtio_net.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -179,6 +180,9 @@ static inline int __virtio_net_hdr_to_skb(struct sk_buff *skb, if (skb->ip_summed == CHECKSUM_PARTIAL && skb->csum_offset != offsetof(struct tcphdr, check)) return -EINVAL; + + BUILD_BUG_ON(TCP_MIN_GSO_SIZE * GSO_MAX_SEGS < GSO_MAX_SIZE); + gso_size = max(gso_size, TCP_MIN_GSO_SIZE); break; } diff --git a/include/media/rc-map.h b/include/media/rc-map.h index d90e4611b066..d95ed3e96de2 100644 --- a/include/media/rc-map.h +++ b/include/media/rc-map.h @@ -262,7 +262,6 @@ struct rc_map *rc_map_get(const char *name); #define RC_MAP_GENIUS_TVGO_A11MCE "rc-genius-tvgo-a11mce" #define RC_MAP_GOTVIEW7135 "rc-gotview7135" #define RC_MAP_HAUPPAUGE "rc-hauppauge" -#define RC_MAP_HAUPPAUGE_NEW "rc-hauppauge" #define RC_MAP_HISI_POPLAR "rc-hisi-poplar" #define RC_MAP_HISI_TV_DEMO "rc-hisi-tv-demo" #define RC_MAP_IMON_MCE "rc-imon-mce" @@ -309,7 +308,6 @@ struct rc_map *rc_map_get(const char *name); #define RC_MAP_PROTEUS_2309 "rc-proteus-2309" #define RC_MAP_PURPLETV "rc-purpletv" #define RC_MAP_PV951 "rc-pv951" -#define RC_MAP_RC5_TV "rc-rc5-tv" #define RC_MAP_RC6_MCE "rc-rc6-mce" #define RC_MAP_REAL_AUDIO_220_32_KEYS "rc-real-audio-220-32-keys" #define RC_MAP_REDDO "rc-reddo" diff --git a/include/net/act_api.h b/include/net/act_api.h index 1ef22d70553b..bac933abcd4c 100644 --- a/include/net/act_api.h +++ b/include/net/act_api.h @@ -242,7 +242,7 @@ static inline void tcf_action_update_bstats(struct tc_action *a, static inline void tcf_action_inc_drop_qstats(struct tc_action *a) { if (likely(a->cpu_qstats)) { - qstats_drop_inc(this_cpu_ptr(a->cpu_qstats)); + qstats_cpu_drop_inc(a->cpu_qstats); return; } atomic_inc(&a->tcfa_drops); diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h index a08d8910ba26..9dea729fd232 100644 --- a/include/net/bluetooth/hci_core.h +++ b/include/net/bluetooth/hci_core.h @@ -916,9 +916,9 @@ static inline void hci_discovery_filter_clear(struct hci_dev *hdev) hdev->discovery.result_filtering = false; hdev->discovery.report_invalid_rssi = true; hdev->discovery.rssi = HCI_RSSI_INVALID; - hdev->discovery.uuid_count = 0; spin_lock(&hdev->discovery.lock); + hdev->discovery.uuid_count = 0; kfree(hdev->discovery.uuids); hdev->discovery.uuids = NULL; spin_unlock(&hdev->discovery.lock); diff --git a/include/net/bluetooth/l2cap.h b/include/net/bluetooth/l2cap.h index 265afda2e13b..2f7c367d68ce 100644 --- a/include/net/bluetooth/l2cap.h +++ b/include/net/bluetooth/l2cap.h @@ -702,7 +702,12 @@ struct l2cap_rx_busy { struct l2cap_pinfo { struct bt_sock bt; + + /* With owning sk_socket chan may be read without lock, other access + * should hold lock_sock. + */ struct l2cap_chan *chan; + struct list_head rx_busy; }; diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h index 1509be85139e..634743ba5fa6 100644 --- a/include/net/cfg80211.h +++ b/include/net/cfg80211.h @@ -7,7 +7,7 @@ * Copyright 2006-2010 Johannes Berg * Copyright 2013-2014 Intel Mobile Communications GmbH * Copyright 2015-2017 Intel Deutschland GmbH - * Copyright (C) 2018-2025 Intel Corporation + * Copyright (C) 2018-2026 Intel Corporation */ #include @@ -126,6 +126,7 @@ struct wiphy; * @IEEE80211_CHAN_NO_4MHZ: 4 MHz bandwidth is not permitted on this channel. * @IEEE80211_CHAN_NO_8MHZ: 8 MHz bandwidth is not permitted on this channel. * @IEEE80211_CHAN_NO_16MHZ: 16 MHz bandwidth is not permitted on this channel. + * @IEEE80211_CHAN_NO_UHR: UHR operation is not permitted on this channel. */ enum ieee80211_channel_flags { IEEE80211_CHAN_DISABLED = BIT(0), @@ -143,6 +144,7 @@ enum ieee80211_channel_flags { IEEE80211_CHAN_NO_10MHZ = BIT(12), IEEE80211_CHAN_NO_HE = BIT(13), /* can use free bits here */ + IEEE80211_CHAN_NO_UHR = BIT(18), IEEE80211_CHAN_NO_320MHZ = BIT(19), IEEE80211_CHAN_NO_EHT = BIT(20), IEEE80211_CHAN_DFS_CONCURRENT = BIT(21), @@ -429,6 +431,18 @@ struct ieee80211_sta_eht_cap { u8 eht_ppe_thres[IEEE80211_EHT_PPE_THRES_MAX_LEN]; }; +/** + * struct ieee80211_sta_uhr_cap - STA's UHR capabilities + * @has_uhr: true iff UHR is supported and data is valid + * @mac: fixed MAC capabilities + * @phy: fixed PHY capabilities + */ +struct ieee80211_sta_uhr_cap { + bool has_uhr; + struct ieee80211_uhr_cap_mac mac; + struct ieee80211_uhr_cap_phy phy; +}; + /* sparse defines __CHECKER__; see Documentation/dev-tools/sparse.rst */ #ifdef __CHECKER__ /* @@ -454,6 +468,7 @@ struct ieee80211_sta_eht_cap { * @he_6ghz_capa: HE 6 GHz capabilities, must be filled in for a * 6 GHz band channel (and 0 may be valid value). * @eht_cap: STA's EHT capabilities + * @uhr_cap: STA's UHR capabilities * @vendor_elems: vendor element(s) to advertise * @vendor_elems.data: vendor element(s) data * @vendor_elems.len: vendor element(s) length @@ -463,6 +478,7 @@ struct ieee80211_sband_iftype_data { struct ieee80211_sta_he_cap he_cap; struct ieee80211_he_6ghz_capa he_6ghz_capa; struct ieee80211_sta_eht_cap eht_cap; + struct ieee80211_sta_uhr_cap uhr_cap; struct { const u8 *data; unsigned int len; @@ -704,6 +720,26 @@ ieee80211_get_eht_iftype_cap(const struct ieee80211_supported_band *sband, return NULL; } +/** + * ieee80211_get_uhr_iftype_cap - return UHR capabilities for an sband's iftype + * @sband: the sband to search for the iftype on + * @iftype: enum nl80211_iftype + * + * Return: pointer to the struct ieee80211_sta_uhr_cap, or NULL is none found + */ +static inline const struct ieee80211_sta_uhr_cap * +ieee80211_get_uhr_iftype_cap(const struct ieee80211_supported_band *sband, + enum nl80211_iftype iftype) +{ + const struct ieee80211_sband_iftype_data *data = + ieee80211_get_sband_iftype_data(sband, iftype); + + if (data && data->uhr_cap.has_uhr) + return &data->uhr_cap; + + return NULL; +} + /** * wiphy_read_of_freq_limits - read frequency limits from device tree * @@ -1485,6 +1521,7 @@ struct cfg80211_s1g_short_beacon { * @he_cap: HE capabilities (or %NULL if HE isn't enabled) * @eht_cap: EHT capabilities (or %NULL if EHT isn't enabled) * @eht_oper: EHT operation IE (or %NULL if EHT isn't enabled) + * @uhr_oper: UHR operation (or %NULL if UHR isn't enabled) * @ht_required: stations must support HT * @vht_required: stations must support VHT * @twt_responder: Enable Target Wait Time @@ -1524,6 +1561,7 @@ struct cfg80211_ap_settings { const struct ieee80211_he_operation *he_oper; const struct ieee80211_eht_cap_elem *eht_cap; const struct ieee80211_eht_operation *eht_oper; + const struct ieee80211_uhr_operation *uhr_oper; bool ht_required, vht_required, he_required, sae_h2e_required; bool twt_responder; u32 flags; @@ -1697,6 +1735,8 @@ struct sta_txpwr { * @eht_capa: EHT capabilities of station * @eht_capa_len: the length of the EHT capabilities * @s1g_capa: S1G capabilities of station + * @uhr_capa: UHR capabilities of the station + * @uhr_capa_len: the length of the UHR capabilities */ struct link_station_parameters { const u8 *mld_mac; @@ -1716,6 +1756,8 @@ struct link_station_parameters { const struct ieee80211_eht_cap_elem *eht_capa; u8 eht_capa_len; const struct ieee80211_s1g_cap *s1g_capa; + const struct ieee80211_uhr_cap *uhr_capa; + u8 uhr_capa_len; }; /** @@ -1784,6 +1826,7 @@ struct cfg80211_ttlm_params { * present/updated * @eml_cap: EML capabilities of this station * @link_sta_params: link related params. + * @epp_peer: EPP peer indication */ struct station_parameters { struct net_device *vlan; @@ -1810,6 +1853,7 @@ struct station_parameters { bool eml_cap_present; u16 eml_cap; struct link_station_parameters link_sta_params; + bool epp_peer; }; /** @@ -1895,6 +1939,11 @@ int cfg80211_check_station_change(struct wiphy *wiphy, * @RATE_INFO_FLAGS_EXTENDED_SC_DMG: 60GHz extended SC MCS * @RATE_INFO_FLAGS_EHT_MCS: EHT MCS information * @RATE_INFO_FLAGS_S1G_MCS: MCS field filled with S1G MCS + * @RATE_INFO_FLAGS_UHR_MCS: UHR MCS information + * @RATE_INFO_FLAGS_UHR_ELR_MCS: UHR ELR MCS was used + * (set together with @RATE_INFO_FLAGS_UHR_MCS) + * @RATE_INFO_FLAGS_UHR_IM: UHR Interference Mitigation + * was used */ enum rate_info_flags { RATE_INFO_FLAGS_MCS = BIT(0), @@ -1906,6 +1955,9 @@ enum rate_info_flags { RATE_INFO_FLAGS_EXTENDED_SC_DMG = BIT(6), RATE_INFO_FLAGS_EHT_MCS = BIT(7), RATE_INFO_FLAGS_S1G_MCS = BIT(8), + RATE_INFO_FLAGS_UHR_MCS = BIT(9), + RATE_INFO_FLAGS_UHR_ELR_MCS = BIT(10), + RATE_INFO_FLAGS_UHR_IM = BIT(11), }; /** @@ -1921,7 +1973,7 @@ enum rate_info_flags { * @RATE_INFO_BW_160: 160 MHz bandwidth * @RATE_INFO_BW_HE_RU: bandwidth determined by HE RU allocation * @RATE_INFO_BW_320: 320 MHz bandwidth - * @RATE_INFO_BW_EHT_RU: bandwidth determined by EHT RU allocation + * @RATE_INFO_BW_EHT_RU: bandwidth determined by EHT/UHR RU allocation * @RATE_INFO_BW_1: 1 MHz bandwidth * @RATE_INFO_BW_2: 2 MHz bandwidth * @RATE_INFO_BW_4: 4 MHz bandwidth @@ -1952,7 +2004,7 @@ enum rate_info_bw { * * @flags: bitflag of flags from &enum rate_info_flags * @legacy: bitrate in 100kbit/s for 802.11abg - * @mcs: mcs index if struct describes an HT/VHT/HE/EHT/S1G rate + * @mcs: mcs index if struct describes an HT/VHT/HE/EHT/S1G/UHR rate * @nss: number of streams (VHT & HE only) * @bw: bandwidth (from &enum rate_info_bw) * @he_gi: HE guard interval (from &enum nl80211_he_gi) @@ -3262,6 +3314,7 @@ struct cfg80211_ml_reconf_req { * Drivers shall disable MLO features for the current association if this * flag is not set. * @ASSOC_REQ_SPP_AMSDU: SPP A-MSDUs will be used on this connection (if any) + * @ASSOC_REQ_DISABLE_UHR: Disable UHR */ enum cfg80211_assoc_req_flags { ASSOC_REQ_DISABLE_HT = BIT(0), @@ -3272,6 +3325,7 @@ enum cfg80211_assoc_req_flags { ASSOC_REQ_DISABLE_EHT = BIT(5), CONNECT_REQ_MLO_SUPPORT = BIT(6), ASSOC_REQ_SPP_AMSDU = BIT(7), + ASSOC_REQ_DISABLE_UHR = BIT(8), }; /** @@ -3996,6 +4050,54 @@ struct cfg80211_nan_conf { u16 vendor_elems_len; }; +#define CFG80211_NAN_SCHED_NUM_TIME_SLOTS 32 + +/** + * struct cfg80211_nan_channel - NAN channel configuration + * + * This struct defines a NAN channel configuration + * + * @chandef: the channel definition + * @channel_entry: pointer to the Channel Entry blob as defined in Wi-Fi Aware + * (TM) 4.0 specification Table 100 (Channel Entry format for the NAN + * Availability attribute). + * @rx_nss: number of spatial streams supported on this channel + */ +struct cfg80211_nan_channel { + struct cfg80211_chan_def chandef; + const u8 *channel_entry; + u8 rx_nss; +}; + +/** + * struct cfg80211_nan_local_sched - NAN local schedule + * + * This struct defines NAN local schedule parameters + * + * @schedule: a mapping of time slots to chandef indexes in %nan_channels. + * An unscheduled slot will be set to %NL80211_NAN_SCHED_NOT_AVAIL_SLOT. + * @n_channels: number of channel definitions in %nan_channels. + * @nan_avail_blob: pointer to NAN Availability attribute blob. + * See %NL80211_ATTR_NAN_AVAIL_BLOB for more details. + * @nan_avail_blob_len: length of the @nan_avail_blob in bytes. + * @deferred: if true, the command containing this schedule configuration is a + * request from the device to perform an announced schedule update. This + * means that it needs to send the updated NAN availability to the peers, + * and do the actual switch on the right time (i.e. at the end of the slot + * after the slot in which the updated NAN Availability was sent). + * See %NL80211_ATTR_NAN_SCHED_DEFERRED for more details. + * If false, the schedule is applied immediately. + * @nan_channels: array of NAN channel definitions that can be scheduled. + */ +struct cfg80211_nan_local_sched { + u8 schedule[CFG80211_NAN_SCHED_NUM_TIME_SLOTS]; + u8 n_channels; + const u8 *nan_avail_blob; + u16 nan_avail_blob_len; + bool deferred; + struct cfg80211_nan_channel nan_channels[] __counted_by(n_channels); +}; + /** * enum cfg80211_nan_conf_changes - indicates changed fields in NAN * configuration @@ -4769,6 +4871,12 @@ struct mgmt_frame_regs { * @nan_change_conf: changes NAN configuration. The changed parameters must * be specified in @changes (using &enum cfg80211_nan_conf_changes); * All other parameters must be ignored. + * @nan_set_local_sched: configure the local schedule for NAN. The schedule + * consists of an array of %cfg80211_nan_channel and the schedule itself, + * in which each entry maps each time slot to the channel on which the + * radio should operate on. If the chandef of a NAN channel is not + * changed, the channel entry must also remain unchanged. It is the + * driver's responsibility to verify this. * * @set_multicast_to_unicast: configure multicast to unicast conversion for BSS * @@ -5146,7 +5254,9 @@ struct cfg80211_ops { struct wireless_dev *wdev, struct cfg80211_nan_conf *conf, u32 changes); - + int (*nan_set_local_sched)(struct wiphy *wiphy, + struct wireless_dev *wdev, + struct cfg80211_nan_local_sched *sched); int (*set_multicast_to_unicast)(struct wiphy *wiphy, struct net_device *dev, const bool enabled); @@ -6774,6 +6884,9 @@ struct wireless_dev { } ocb; struct { u8 cluster_id[ETH_ALEN] __aligned(2); + u8 n_channels; + struct cfg80211_chan_def *chandefs; + bool sched_update_pending; } nan; } u; @@ -9900,6 +10013,18 @@ void cfg80211_nan_func_terminated(struct wireless_dev *wdev, enum nl80211_nan_func_term_reason reason, u64 cookie, gfp_t gfp); +/** + * cfg80211_nan_sched_update_done - notify deferred schedule update completion + * @wdev: the wireless device reporting the event + * @success: whether or not the schedule update was successful + * @gfp: allocation flags + * + * This function notifies user space that a deferred local NAN schedule update + * (requested with %NL80211_ATTR_NAN_SCHED_DEFERRED) has been completed. + */ +void cfg80211_nan_sched_update_done(struct wireless_dev *wdev, bool success, + gfp_t gfp); + /* ethtool helper */ void cfg80211_get_drvinfo(struct net_device *dev, struct ethtool_drvinfo *info); @@ -10328,4 +10453,27 @@ cfg80211_s1g_get_primary_sibling(struct wiphy *wiphy, return ieee80211_get_channel_khz(wiphy, sibling_1mhz_khz); } + +/** + * cfg80211_incumbent_signal_notify - Notify userspace of incumbent signal detection + * @wiphy: the wiphy to use + * @chandef: channel definition in which the interference was detected + * @signal_interference_bitmap: bitmap indicating interference across 20 MHz segments + * @gfp: allocation context for message creation and multicast; pass GFP_ATOMIC + * if called from atomic context (e.g. firmware event handler), otherwise + * GFP_KERNEL + * + * Use this function to notify userspace when an incumbent signal is detected on + * the operating channel in the 6 GHz band. The notification includes the + * current channel definition and a bitmap representing interference across + * the operating bandwidth. Each bit in the bitmap corresponds to a 20 MHz + * segment, with the lowest bit representing the lowest frequency segment. + * Punctured sub-channels are included in the bitmap structure but are always + * set to zero since interference detection is not performed on them. + */ +void cfg80211_incumbent_signal_notify(struct wiphy *wiphy, + const struct cfg80211_chan_def *chandef, + u32 signal_interference_bitmap, + gfp_t gfp); + #endif /* __NET_CFG80211_H */ diff --git a/include/net/inetpeer.h b/include/net/inetpeer.h index f475757daafb..414e9adf4c51 100644 --- a/include/net/inetpeer.h +++ b/include/net/inetpeer.h @@ -35,6 +35,7 @@ struct inetpeer_addr { struct inet_peer { struct rb_node rb_node; + u64 hash; struct inetpeer_addr daddr; u32 metrics[RTAX_MAX]; @@ -125,6 +126,9 @@ static inline int inetpeer_addr_cmp(const struct inetpeer_addr *a, { int i, n; + if (a->family != b->family) + return a->family < b->family ? -1 : 1; + if (a->family == AF_INET) n = sizeof(a->a4) / sizeof(u32); else diff --git a/include/net/libeth/xsk.h b/include/net/libeth/xsk.h index 23eb240d5ba7..3d77e72a8cc8 100644 --- a/include/net/libeth/xsk.h +++ b/include/net/libeth/xsk.h @@ -196,7 +196,7 @@ __libeth_xsk_xmit_fill_buf_md(const struct xdp_desc *xdesc, struct libeth_xdp_tx_desc desc; struct xdp_desc_ctx ctx; - ctx = xsk_buff_raw_get_ctx(sq->pool, xdesc->addr); + ctx = xsk_buff_raw_get_ctx(sq->pool, xdesc->addr, xdesc->options); desc = (typeof(desc)){ .addr = ctx.dma, __libeth_xdp_tx_len(xdesc->len), diff --git a/include/net/netns/ipv4.h b/include/net/netns/ipv4.h index 62166da04554..fa49a9598bbb 100644 --- a/include/net/netns/ipv4.h +++ b/include/net/netns/ipv4.h @@ -271,7 +271,7 @@ struct netns_ipv4 { #ifdef CONFIG_IP_MROUTE #ifndef CONFIG_IP_MROUTE_MULTIPLE_TABLES - struct mr_table *mrt; + struct mr_table __rcu *mrt; #else struct list_head mr_tables; struct fib_rules_ops *mr_rules_ops; diff --git a/include/net/pkt_sched.h b/include/net/pkt_sched.h index 8a75c73fc555..f3649ae8cab6 100644 --- a/include/net/pkt_sched.h +++ b/include/net/pkt_sched.h @@ -12,6 +12,7 @@ #define DEFAULT_TX_QUEUE_LEN 1000 #define STAB_SIZE_LOG_MAX 30 +#define QDISC_PKT_LEN_MAX (1 << 20) /* 1 MiB */ struct qdisc_walker { int stop; diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index b15a0cdf101b..cf1923f8cb52 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -967,12 +967,17 @@ static inline void qdisc_qstats_cpu_requeues_inc(struct Qdisc *sch) static inline void __qdisc_qstats_drop(struct Qdisc *sch, int count) { - sch->qstats.drops += count; + WRITE_ONCE(sch->qstats.drops, sch->qstats.drops + count); } static inline void qstats_drop_inc(struct gnet_stats_queue *qstats) { - qstats->drops++; + WRITE_ONCE(qstats->drops, qstats->drops + 1); +} + +static inline void qstats_cpu_drop_inc(struct gnet_stats_queue __percpu *qstats) +{ + this_cpu_inc(qstats->drops); } static inline void qstats_overlimit_inc(struct gnet_stats_queue *qstats) diff --git a/include/net/sock.h b/include/net/sock.h index 9637ee4f55ee..50b9178cc827 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2468,12 +2468,23 @@ int __sk_queue_drop_skb(struct sock *sk, struct sk_buff_head *sk_queue, struct sk_buff *skb)); int __sock_queue_rcv_skb(struct sock *sk, struct sk_buff *skb); -int sock_queue_rcv_skb_reason(struct sock *sk, struct sk_buff *skb, - enum skb_drop_reason *reason); +enum skb_drop_reason +sock_queue_rcv_skb_reason(struct sock *sk, struct sk_buff *skb); static inline int sock_queue_rcv_skb(struct sock *sk, struct sk_buff *skb) { - return sock_queue_rcv_skb_reason(sk, skb, NULL); + enum skb_drop_reason drop_reason = sock_queue_rcv_skb_reason(sk, skb); + + switch (drop_reason) { + case SKB_DROP_REASON_SOCKET_RCVBUFF: + return -ENOMEM; + case SKB_DROP_REASON_PROTO_MEM: + return -ENOBUFS; + case 0: + return 0; + default: + return -EPERM; + } } int sock_queue_err_skb(struct sock *sk, struct sk_buff *skb); diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h index d74ca548d4d6..21c3b757bb88 100644 --- a/include/net/xdp_sock_drv.h +++ b/include/net/xdp_sock_drv.h @@ -236,6 +236,7 @@ static inline void *xsk_buff_raw_get_data(struct xsk_buff_pool *pool, u64 addr) * xsk_buff_raw_get_ctx - get &xdp_desc context * @pool: XSk buff pool desc address belongs to * @addr: desc address (from userspace) + * @options: desc options (from userspace) * * Wrapper for xp_raw_get_ctx() to be used in drivers, see its kdoc for * details. @@ -244,9 +245,9 @@ static inline void *xsk_buff_raw_get_data(struct xsk_buff_pool *pool, u64 addr) * pointer, if it is present (initialized to %NULL otherwise). */ static inline struct xdp_desc_ctx -xsk_buff_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr) +xsk_buff_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr, u32 options) { - return xp_raw_get_ctx(pool, addr); + return xp_raw_get_ctx(pool, addr, options); } #define XDP_TXMD_FLAGS_VALID ( \ @@ -314,18 +315,20 @@ xsk_tx_metadata_request(const struct xsk_buff_pool *pool, } static inline struct xsk_tx_metadata * -__xsk_buff_get_metadata(const struct xsk_buff_pool *pool, void *data) +__xsk_buff_get_metadata(const struct xsk_buff_pool *pool, void *data, + unsigned int options) { - if (!pool->tx_metadata_len) + if (!pool->tx_metadata_len || !(options & XDP_TX_METADATA)) return NULL; return data - pool->tx_metadata_len; } static inline struct xsk_tx_metadata * -xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr) +xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr, u32 options) { - return __xsk_buff_get_metadata(pool, xp_raw_get_data(pool, addr)); + return __xsk_buff_get_metadata(pool, xp_raw_get_data(pool, addr), + options); } static inline void xsk_buff_dma_sync_for_cpu(struct xdp_buff *xdp) @@ -506,7 +509,7 @@ static inline void *xsk_buff_raw_get_data(struct xsk_buff_pool *pool, u64 addr) } static inline struct xdp_desc_ctx -xsk_buff_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr) +xsk_buff_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr, u32 options) { return (struct xdp_desc_ctx){ }; } @@ -526,13 +529,14 @@ xsk_tx_metadata_request(const struct xsk_buff_pool *pool, } static inline struct xsk_tx_metadata * -__xsk_buff_get_metadata(const struct xsk_buff_pool *pool, void *data) +__xsk_buff_get_metadata(const struct xsk_buff_pool *pool, void *data, + unsigned int options) { return NULL; } static inline struct xsk_tx_metadata * -xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr) +xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr, u32 options) { return NULL; } diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h index 92a2358c6ce3..10e564d85a90 100644 --- a/include/net/xsk_buff_pool.h +++ b/include/net/xsk_buff_pool.h @@ -154,7 +154,8 @@ struct xdp_desc_ctx { struct xsk_tx_metadata *meta; }; -struct xdp_desc_ctx xp_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr); +struct xdp_desc_ctx xp_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr, + u32 options); static inline dma_addr_t xp_get_dma(struct xdp_buff_xsk *xskb) { diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index eaeec00ef4c1..7f811e80b63a 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -876,6 +876,20 @@ __attribute_const__ int ib_rate_to_mult(enum ib_rate rate); */ __attribute_const__ int ib_rate_to_mbps(enum ib_rate rate); +struct ib_port_speed_info { + const char *str; + int rate; /* in deci-Gb/sec (100 MBps units) */ +}; + +/** + * ib_port_attr_to_speed_info - Convert port attributes to speed information + * @attr: Port attributes containing active_speed and active_width + * @speed_info: Speed information to return + * + * Returns 0 on success, -EINVAL on error. + */ +int ib_port_attr_to_speed_info(struct ib_port_attr *attr, + struct ib_port_speed_info *speed_info); /** * enum ib_mr_type - memory region type @@ -1870,6 +1884,11 @@ struct ib_dmah { struct ib_mr { struct ib_device *device; + /* + * Due to IB_MR_REREG_PD pd is not a fixed pointer and can change. For a + * user MR, this value should only be read from a system call that holds + * the uobject lock, or the driver should disable in-place REREG_PD. + */ struct ib_pd *pd; u32 lkey; u32 rkey; @@ -2402,6 +2421,8 @@ struct ib_device_ops { int comp_vector); int (*query_port)(struct ib_device *device, u32 port_num, struct ib_port_attr *port_attr); + int (*query_port_speed)(struct ib_device *device, u32 port_num, + u64 *speed); int (*modify_port)(struct ib_device *device, u32 port_num, int port_modify_mask, struct ib_port_modify *port_modify); diff --git a/include/sound/core.h b/include/sound/core.h index 64327e971122..618055761fe0 100644 --- a/include/sound/core.h +++ b/include/sound/core.h @@ -314,6 +314,8 @@ static inline void snd_card_unref(struct snd_card *card) put_device(&card->card_dev); } +DEFINE_FREE(snd_card_unref, struct snd_card *, if (_T) snd_card_unref(_T)) + #define snd_card_set_dev(card, devptr) ((card)->dev = (devptr)) /* device.c */ diff --git a/include/trace/events/erofs.h b/include/trace/events/erofs.h index dad7360f42f9..59f3ef397bff 100644 --- a/include/trace/events/erofs.h +++ b/include/trace/events/erofs.h @@ -26,10 +26,9 @@ struct erofs_map_blocks; #define show_mflags(flags) __print_flags(flags, "", \ { EROFS_MAP_MAPPED, "M" }, \ { EROFS_MAP_META, "I" }, \ - { EROFS_MAP_ENCODED, "E" }, \ - { EROFS_MAP_FULL_MAPPED, "F" }, \ - { EROFS_MAP_FRAGMENT, "R" }, \ - { EROFS_MAP_PARTIAL_REF, "P" }) + { EROFS_MAP_PARTIAL_MAPPED, "T" }, \ + { EROFS_MAP_PARTIAL_REF, "P" }, \ + { EROFS_MAP_FRAGMENT, "R" }) TRACE_EVENT(erofs_lookup, diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index bcad11a787a5..157f11d3fb72 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -181,6 +181,7 @@ enum devlink_sb_threshold_type { enum devlink_eswitch_mode { DEVLINK_ESWITCH_MODE_LEGACY, DEVLINK_ESWITCH_MODE_SWITCHDEV, + DEVLINK_ESWITCH_MODE_SWITCHDEV_INACTIVE, }; enum devlink_eswitch_inline_mode { diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h index 23a062781468..50d67df78911 100644 --- a/include/uapi/linux/if_xdp.h +++ b/include/uapi/linux/if_xdp.h @@ -149,6 +149,7 @@ struct xsk_tx_metadata { __u16 csum_start; /* Offset from csum_start where checksum should be stored. */ __u16 csum_offset; + __u32 reserved; /* XDP_TXMD_FLAGS_LAUNCH_TIME */ /* Launch time in nanosecond against the PTP HW Clock */ diff --git a/include/uapi/linux/nl80211.h b/include/uapi/linux/nl80211.h index 8134f10e4e6c..857d1f66bf35 100644 --- a/include/uapi/linux/nl80211.h +++ b/include/uapi/linux/nl80211.h @@ -1361,6 +1361,26 @@ * user space that the NAN new cluster has been joined. The cluster ID is * indicated by %NL80211_ATTR_MAC. * + * @NL80211_CMD_INCUMBENT_SIGNAL_DETECT: Once any incumbent signal is detected + * on the operating channel in 6 GHz band, userspace is notified with the + * signal interference bitmap using + * %NL80211_ATTR_INCUMBENT_SIGNAL_INTERFERENCE_BITMAP. The current channel + * definition is also sent. + * + * @NL80211_CMD_NAN_SET_LOCAL_SCHED: Set the local NAN schedule. NAN must be + * operational (%NL80211_CMD_START_NAN was executed). Must contain + * %NL80211_ATTR_NAN_TIME_SLOTS and %NL80211_ATTR_NAN_AVAIL_BLOB, but + * %NL80211_ATTR_NAN_CHANNEL is optional (for example in case of a channel + * removal, that channel won't be provided). + * If %NL80211_ATTR_NAN_SCHED_DEFERRED is set, the command is a request + * from the device to perform an announced schedule update. See + * %NL80211_ATTR_NAN_SCHED_DEFERRED for more details. + * If not set, the schedule should be applied immediately. + * @NL80211_CMD_NAN_SCHED_UPDATE_DONE: Event sent to user space to notify that + * a deferred local NAN schedule update (requested with + * %NL80211_CMD_NAN_SET_LOCAL_SCHED and %NL80211_ATTR_NAN_SCHED_DEFERRED) + * has been completed. The presence of %NL80211_ATTR_NAN_SCHED_UPDATE_SUCCESS + * indicates that the update was successful. * @NL80211_CMD_MAX: highest used command number * @__NL80211_CMD_AFTER_LAST: internal use */ @@ -1624,6 +1644,12 @@ enum nl80211_commands { NL80211_CMD_NAN_NEXT_DW_NOTIFICATION, NL80211_CMD_NAN_CLUSTER_JOINED, + NL80211_CMD_INCUMBENT_SIGNAL_DETECT, + + NL80211_CMD_NAN_SET_LOCAL_SCHED, + + NL80211_CMD_NAN_SCHED_UPDATE_DONE, + /* add new commands above here */ /* used to define NL80211_CMD_MAX below */ @@ -2973,6 +2999,77 @@ enum nl80211_commands { * primary channel is 2 MHz wide, and the control channel designates * the 1 MHz primary subchannel within that 2 MHz primary. * + * @NL80211_ATTR_EPP_PEER: A flag attribute to indicate if the peer is an EPP + * STA. Used with %NL80211_CMD_NEW_STA and %NL80211_CMD_ADD_LINK_STA + * + * @NL80211_ATTR_UHR_CAPABILITY: UHR Capability information element (from + * association request when used with NL80211_CMD_NEW_STATION). Can be set + * only if HE/EHT are also available. + * @NL80211_ATTR_DISABLE_UHR: Force UHR capable interfaces to disable + * this feature during association. This is a flag attribute. + * Currently only supported in mac80211 drivers. + * @NL80211_ATTR_NAN_CHANNEL: This is a nested attribute. There can be multiple + * attributes of this type, each one represents a channel definition and + * consists of top-level attributes like %NL80211_ATTR_WIPHY_FREQ. Must + * contain %NL80211_ATTR_NAN_CHANNEL_ENTRY and + * %NL80211_ATTR_NAN_RX_NSS. + * This attribute is used with %NL80211_CMD_NAN_SET_LOCAL_SCHED to specify + * the channel definitions on which the radio needs to operate during + * specific time slots. All of the channel definitions should be mutually + * incompatible. The number of channels should fit the current + * configuration of channels and the possible interface combinations. + * If an existing NAN channel is changed but the chandef isn't, the + * channel entry must also remain unchanged. + * @NL80211_ATTR_NAN_CHANNEL_ENTRY: a byte array of 6 bytes. contains the + * Channel Entry as defined in Wi-Fi Aware (TM) 4.0 specification Table + * 100 (Channel Entry format for the NAN Availability attribute). + * @NL80211_ATTR_NAN_RX_NSS: (u8) RX NSS used for a NAN channel. This is + * used with %NL80211_ATTR_NAN_CHANNEL when configuring NAN channels with + * %NL80211_CMD_NAN_SET_LOCAL_SCHED. + * @NL80211_ATTR_NAN_TIME_SLOTS: an array of u8 values and 32 cells. each value + * maps a time slot to the chandef on which the radio should operate on in + * that time. %NL80211_NAN_SCHED_NOT_AVAIL_SLOT indicates unscheduled. + * The chandef is represented using its index, where the index is the + * sequential number of the %NL80211_ATTR_NAN_CHANNEL attribute within all + * the attributes of this type. + * Each slots spans over 16TUs, hence the entire schedule spans over + * 512TUs. Other slot durations and periods are currently not supported. + * @NL80211_ATTR_NAN_AVAIL_BLOB: (Binary) The NAN Availability attribute blob, + * including the attribute header, as defined in Wi-Fi Aware (TM) 4.0 + * specification Table 93 (NAN Availability attribute format). Required with + * %NL80211_CMD_NAN_SET_LOCAL_SCHED to provide the raw NAN Availability + * attribute. Used by the device to publish Schedule Update NAFs. + * @NL80211_ATTR_NAN_SCHED_DEFERRED: Flag attribute used with + * %NL80211_CMD_NAN_SET_LOCAL_SCHED. When present, the command is a + * request from the device to perform an announced schedule update. This + * means that it needs to send the updated NAN availability to the peers, + * and do the actual switch on the right time (i.e. at the end of the slot + * after the slot in which the updated NAN Availability was sent). Since + * the slots management is done in the device, the update to the peers + * needs to be sent by the device, so it knows the actual switch time. + * If the flag is not set, the schedule should be applied immediately. + * When this flag is set, the total number of NAN channels from both the + * old and new schedules must not exceed the allowed number of local NAN + * channels, because with deferred scheduling the old channels cannot be + * removed before adding the new ones to free up space. + * @NL80211_ATTR_NAN_SCHED_UPDATE_SUCCESS: flag attribute used with + * %NL80211_CMD_NAN_SCHED_UPDATE_DONE to indicate that the deferred + * schedule update completed successfully. If this flag is not present, + * the update failed. + * + * @NL80211_ATTR_INCUMBENT_SIGNAL_INTERFERENCE_BITMAP: u32 attribute specifying + * the signal interference bitmap detected on the operating bandwidth for + * %NL80211_CMD_INCUMBENT_SIGNAL_DETECT. Each bit represents a 20 MHz + * segment, lowest bit corresponds to the lowest 20 MHz segment, in the + * operating bandwidth where the interference is detected. Punctured + * sub-channels are included in the bitmap structure; however, since + * interference detection is not performed on these sub-channels, their + * corresponding bits are consistently set to zero. + * + * @NL80211_ATTR_UHR_OPERATION: Full UHR Operation element, as it appears in + * association response etc., since it's abridged in the beacon. Used + * for START_AP etc. + * * @NUM_NL80211_ATTR: total number of nl80211_attrs available * @NL80211_ATTR_MAX: highest attribute number currently defined * @__NL80211_ATTR_AFTER_LAST: internal use @@ -3541,6 +3638,23 @@ enum nl80211_attrs { NL80211_ATTR_S1G_PRIMARY_2MHZ, + NL80211_ATTR_EPP_PEER, + + NL80211_ATTR_UHR_CAPABILITY, + NL80211_ATTR_DISABLE_UHR, + + NL80211_ATTR_INCUMBENT_SIGNAL_INTERFERENCE_BITMAP, + + NL80211_ATTR_UHR_OPERATION, + + NL80211_ATTR_NAN_CHANNEL, + NL80211_ATTR_NAN_CHANNEL_ENTRY, + NL80211_ATTR_NAN_TIME_SLOTS, + NL80211_ATTR_NAN_RX_NSS, + NL80211_ATTR_NAN_AVAIL_BLOB, + NL80211_ATTR_NAN_SCHED_DEFERRED, + NL80211_ATTR_NAN_SCHED_UPDATE_SUCCESS, + /* add attributes here, update the policy in nl80211.c */ __NL80211_ATTR_AFTER_LAST, @@ -3893,6 +4007,12 @@ enum nl80211_eht_ru_alloc { * @NL80211_RATE_INFO_4_MHZ_WIDTH: 4 MHz S1G rate * @NL80211_RATE_INFO_8_MHZ_WIDTH: 8 MHz S1G rate * @NL80211_RATE_INFO_16_MHZ_WIDTH: 16 MHz S1G rate + * @NL80211_RATE_INFO_UHR_MCS: UHR MCS index (u8, 0-15, 17, 19, 20, 23) + * Note that the other EHT attributes (such as @NL80211_RATE_INFO_EHT_NSS) + * are used in conjunction with this where applicable + * @NL80211_RATE_INFO_UHR_ELR: UHR ELR flag, which restricts NSS to 1, + * MCS to 0 or 1, and GI to %NL80211_RATE_INFO_EHT_GI_1_6. + * @NL80211_RATE_INFO_UHR_IM: UHR Interference Mitigation flag * @__NL80211_RATE_INFO_AFTER_LAST: internal use */ enum nl80211_rate_info { @@ -3926,6 +4046,9 @@ enum nl80211_rate_info { NL80211_RATE_INFO_4_MHZ_WIDTH, NL80211_RATE_INFO_8_MHZ_WIDTH, NL80211_RATE_INFO_16_MHZ_WIDTH, + NL80211_RATE_INFO_UHR_MCS, + NL80211_RATE_INFO_UHR_ELR, + NL80211_RATE_INFO_UHR_IM, /* keep last */ __NL80211_RATE_INFO_AFTER_LAST, @@ -4248,6 +4371,10 @@ enum nl80211_mpath_info { * capabilities element * @NL80211_BAND_IFTYPE_ATTR_EHT_CAP_PPE: EHT PPE thresholds information as * defined in EHT capabilities element + * @NL80211_BAND_IFTYPE_ATTR_UHR_CAP_MAC: UHR MAC capabilities as in UHR + * capabilities element + * @NL80211_BAND_IFTYPE_ATTR_UHR_CAP_PHY: UHR PHY capabilities as in UHR + * capabilities element * @__NL80211_BAND_IFTYPE_ATTR_AFTER_LAST: internal use * @NL80211_BAND_IFTYPE_ATTR_MAX: highest band attribute currently defined */ @@ -4265,6 +4392,8 @@ enum nl80211_band_iftype_attr { NL80211_BAND_IFTYPE_ATTR_EHT_CAP_PHY, NL80211_BAND_IFTYPE_ATTR_EHT_CAP_MCS_SET, NL80211_BAND_IFTYPE_ATTR_EHT_CAP_PPE, + NL80211_BAND_IFTYPE_ATTR_UHR_CAP_MAC, + NL80211_BAND_IFTYPE_ATTR_UHR_CAP_PHY, /* keep last */ __NL80211_BAND_IFTYPE_ATTR_AFTER_LAST, @@ -4444,6 +4573,11 @@ enum nl80211_wmm_rule { * channel in current regulatory domain. * @NL80211_FREQUENCY_ATTR_NO_16MHZ: 16 MHz operation is not allowed on this * channel in current regulatory domain. + * @NL80211_FREQUENCY_ATTR_S1G_NO_PRIMARY: Channel is not permitted for use + * as a primary channel. Does not prevent the channel from existing + * as a non-primary subchannel. Only applicable to S1G channels. + * @NL80211_FREQUENCY_ATTR_NO_UHR: UHR operation is not allowed on this channel + * in current regulatory domain. * @NL80211_FREQUENCY_ATTR_MAX: highest frequency attribute number * currently defined * @__NL80211_FREQUENCY_ATTR_AFTER_LAST: internal use @@ -4492,6 +4626,8 @@ enum nl80211_frequency_attr { NL80211_FREQUENCY_ATTR_NO_4MHZ, NL80211_FREQUENCY_ATTR_NO_8MHZ, NL80211_FREQUENCY_ATTR_NO_16MHZ, + NL80211_FREQUENCY_ATTR_S1G_NO_PRIMARY, + NL80211_FREQUENCY_ATTR_NO_UHR, /* keep last */ __NL80211_FREQUENCY_ATTR_AFTER_LAST, @@ -4705,6 +4841,7 @@ enum nl80211_sched_scan_match_attr { * despite NO_IR configuration. * @NL80211_RRF_ALLOW_20MHZ_ACTIVITY: Allow activity in 20 MHz bandwidth, * despite NO_IR configuration. + * @NL80211_RRF_NO_UHR: UHR operation not allowed */ enum nl80211_reg_rule_flags { NL80211_RRF_NO_OFDM = 1 << 0, @@ -4731,6 +4868,7 @@ enum nl80211_reg_rule_flags { NL80211_RRF_NO_6GHZ_AFC_CLIENT = 1 << 23, NL80211_RRF_ALLOW_6GHZ_VLP_AP = 1 << 24, NL80211_RRF_ALLOW_20MHZ_ACTIVITY = 1 << 25, + NL80211_RRF_NO_UHR = 1 << 26, }; #define NL80211_RRF_PASSIVE_SCAN NL80211_RRF_NO_IR @@ -8432,4 +8570,6 @@ enum nl80211_nan_capabilities { NL80211_NAN_CAPABILITIES_MAX = __NL80211_NAN_CAPABILITIES_LAST - 1, }; +#define NL80211_NAN_SCHED_NOT_AVAIL_SLOT 0xff + #endif /* __LINUX_NL80211_H */ diff --git a/include/uapi/linux/time_types.h b/include/uapi/linux/time_types.h index bcc0002115d3..03a0d8aaadca 100644 --- a/include/uapi/linux/time_types.h +++ b/include/uapi/linux/time_types.h @@ -30,7 +30,7 @@ struct __kernel_old_timeval { struct __kernel_old_timespec { __kernel_old_time_t tv_sec; /* seconds */ - long tv_nsec; /* nanoseconds */ + __kernel_long_t tv_nsec; /* nanoseconds */ }; struct __kernel_old_itimerval { diff --git a/include/uapi/rdma/ib_user_ioctl_cmds.h b/include/uapi/rdma/ib_user_ioctl_cmds.h index de6f5a94f1e3..35da4026f452 100644 --- a/include/uapi/rdma/ib_user_ioctl_cmds.h +++ b/include/uapi/rdma/ib_user_ioctl_cmds.h @@ -73,6 +73,7 @@ enum uverbs_methods_device { UVERBS_METHOD_QUERY_CONTEXT, UVERBS_METHOD_QUERY_GID_TABLE, UVERBS_METHOD_QUERY_GID_ENTRY, + UVERBS_METHOD_QUERY_PORT_SPEED, }; enum uverbs_attrs_invoke_write_cmd_attr_ids { @@ -86,6 +87,11 @@ enum uverbs_attrs_query_port_cmd_attr_ids { UVERBS_ATTR_QUERY_PORT_RESP, }; +enum uverbs_attrs_query_port_speed_cmd_attr_ids { + UVERBS_ATTR_QUERY_PORT_SPEED_PORT_NUM, + UVERBS_ATTR_QUERY_PORT_SPEED_RESP, +}; + enum uverbs_attrs_get_context_attr_ids { UVERBS_ATTR_GET_CONTEXT_NUM_COMP_VECTORS, UVERBS_ATTR_GET_CONTEXT_CORE_SUPPORT, diff --git a/io_uring/futex.c b/io_uring/futex.c index 3e92c2a63960..7f3b97c99692 100644 --- a/io_uring/futex.c +++ b/io_uring/futex.c @@ -41,24 +41,26 @@ void io_futex_cache_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->futex_cache, kfree); } -static void __io_futex_complete(struct io_kiocb *req, io_tw_token_t tw) +static void __io_futex_complete(struct io_tw_req tw_req, io_tw_token_t tw) { - hlist_del_init(&req->hash_node); - io_req_task_complete(req, tw); + hlist_del_init(&tw_req.req->hash_node); + io_req_task_complete(tw_req, tw); } -static void io_futex_complete(struct io_kiocb *req, io_tw_token_t tw) +static void io_futex_complete(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_ring_ctx *ctx = req->ctx; io_tw_lock(ctx, tw); io_cache_free(&ctx->futex_cache, req->async_data); io_req_async_data_clear(req, 0); - __io_futex_complete(req, tw); + __io_futex_complete(tw_req, tw); } -static void io_futexv_complete(struct io_kiocb *req, io_tw_token_t tw) +static void io_futexv_complete(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); struct futex_vector *futexv = req->async_data; @@ -73,7 +75,7 @@ static void io_futexv_complete(struct io_kiocb *req, io_tw_token_t tw) } io_req_async_data_free(req); - __io_futex_complete(req, tw); + __io_futex_complete(tw_req, tw); } static bool io_futexv_claim(struct io_futex *iof) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9784463fb718..fd64ba7a0927 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -268,6 +268,20 @@ static __cold void io_ring_ctx_ref_free(struct percpu_ref *ref) complete(&ctx->ref_comp); } +/* + * Terminate the request if either of these conditions are true: + * + * 1) It's being executed by the original task, but that task is marked + * with PF_EXITING as it's exiting. + * 2) PF_KTHREAD is set, in which case the invoker of the task_work is + * our fallback task_work. + * 3) The ring has been closed and is going away. + */ +static inline bool io_should_terminate_tw(struct io_ring_ctx *ctx) +{ + return (current->flags & (PF_EXITING | PF_KTHREAD)) || percpu_ref_is_dying(&ctx->refs); +} + static __cold void io_fallback_req_func(struct work_struct *work) { struct io_ring_ctx *ctx = container_of(work, struct io_ring_ctx, @@ -278,8 +292,9 @@ static __cold void io_fallback_req_func(struct work_struct *work) percpu_ref_get(&ctx->refs); mutex_lock(&ctx->uring_lock); + ts.cancel = io_should_terminate_tw(ctx); llist_for_each_entry_safe(req, tmp, node, io_task_work.node) - req->io_task_work.func(req, ts); + req->io_task_work.func((struct io_tw_req){req}, ts); io_submit_flush_completions(ctx); mutex_unlock(&ctx->uring_lock); percpu_ref_put(&ctx->refs); @@ -527,9 +542,9 @@ static void io_queue_iowq(struct io_kiocb *req) io_wq_enqueue(tctx->io_wq, &req->work); } -static void io_req_queue_iowq_tw(struct io_kiocb *req, io_tw_token_t tw) +static void io_req_queue_iowq_tw(struct io_tw_req tw_req, io_tw_token_t tw) { - io_queue_iowq(req); + io_queue_iowq(tw_req.req); } void io_req_queue_iowq(struct io_kiocb *req) @@ -1152,10 +1167,11 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, ctx = req->ctx; mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); + ts.cancel = io_should_terminate_tw(ctx); } INDIRECT_CALL_2(req->io_task_work.func, io_poll_task_func, io_req_rw_complete, - req, ts); + (struct io_tw_req){req}, ts); node = next; (*count)++; if (unlikely(need_resched())) { @@ -1210,11 +1226,6 @@ struct llist_node *tctx_task_work_run(struct io_uring_task *tctx, { struct llist_node *node; - if (unlikely(current->flags & PF_EXITING)) { - io_fallback_tw(tctx, true); - return NULL; - } - node = llist_del_all(&tctx->task_list); if (node) { node = llist_reverse_order(node); @@ -1411,7 +1422,7 @@ static int __io_run_local_work_loop(struct llist_node **node, io_task_work.node); INDIRECT_CALL_2(req->io_task_work.func, io_poll_task_func, io_req_rw_complete, - req, tw); + (struct io_tw_req){req}, tw); *node = next; if (++ret >= events) break; @@ -1432,6 +1443,7 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, io_tw_token_t tw, if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_andnot(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); again: + tw.cancel = io_should_terminate_tw(ctx); min_events -= ret; ret = __io_run_local_work_loop(&ctx->retry_llist.first, tw, max_events); if (ctx->retry_llist.first) @@ -1480,18 +1492,21 @@ static int io_run_local_work(struct io_ring_ctx *ctx, int min_events, return ret; } -static void io_req_task_cancel(struct io_kiocb *req, io_tw_token_t tw) +static void io_req_task_cancel(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; + io_tw_lock(req->ctx, tw); io_req_defer_failed(req, req->cqe.res); } -void io_req_task_submit(struct io_kiocb *req, io_tw_token_t tw) +void io_req_task_submit(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_ring_ctx *ctx = req->ctx; io_tw_lock(ctx, tw); - if (unlikely(io_should_terminate_tw(ctx))) + if (unlikely(tw.cancel)) io_req_defer_failed(req, -EFAULT); else if (req->flags & REQ_F_FORCE_ASYNC) io_queue_iowq(req); @@ -1723,9 +1738,9 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_events) return 0; } -void io_req_task_complete(struct io_kiocb *req, io_tw_token_t tw) +void io_req_task_complete(struct io_tw_req tw_req, io_tw_token_t tw) { - io_req_complete_defer(req); + io_req_complete_defer(tw_req.req); } /* diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 05702288465b..042bdd766571 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -160,9 +160,9 @@ struct file *io_file_get_fixed(struct io_kiocb *req, int fd, void __io_req_task_work_add(struct io_kiocb *req, unsigned flags); void io_req_task_work_add_remote(struct io_kiocb *req, unsigned flags); void io_req_task_queue(struct io_kiocb *req); -void io_req_task_complete(struct io_kiocb *req, io_tw_token_t tw); +void io_req_task_complete(struct io_tw_req tw_req, io_tw_token_t tw); void io_req_task_queue_fail(struct io_kiocb *req, int ret); -void io_req_task_submit(struct io_kiocb *req, io_tw_token_t tw); +void io_req_task_submit(struct io_tw_req tw_req, io_tw_token_t tw); struct llist_node *io_handle_tw_list(struct llist_node *node, unsigned int *count, unsigned int max_entries); struct llist_node *tctx_task_work_run(struct io_uring_task *tctx, unsigned int max_entries, unsigned int *count); void tctx_task_work(struct callback_head *cb); @@ -582,19 +582,6 @@ static inline bool io_allowed_run_tw(struct io_ring_ctx *ctx) ctx->submitter_task == current); } -/* - * Terminate the request if either of these conditions are true: - * - * 1) It's being executed by the original task, but that task is marked - * with PF_EXITING as it's exiting. - * 2) PF_KTHREAD is set, in which case the invoker of the task_work is - * our fallback task_work. - */ -static inline bool io_should_terminate_tw(struct io_ring_ctx *ctx) -{ - return (current->flags & (PF_KTHREAD | PF_EXITING)) || percpu_ref_is_dying(&ctx->refs); -} - static inline void io_req_queue_tw_complete(struct io_kiocb *req, s32 res) { io_req_set_res(req, res, 0); diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index bce74a8b64c6..87b4d306cf1b 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -70,8 +70,9 @@ static inline bool io_msg_need_remote(struct io_ring_ctx *target_ctx) return target_ctx->task_complete; } -static void io_msg_tw_complete(struct io_kiocb *req, io_tw_token_t tw) +static void io_msg_tw_complete(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_ring_ctx *ctx = req->ctx; io_add_aux_cqe(ctx, req->cqe.user_data, req->cqe.res, req->cqe.flags); diff --git a/io_uring/notif.c b/io_uring/notif.c index d8ba1165c949..9960bb2a32d5 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -11,8 +11,9 @@ static const struct ubuf_info_ops io_ubuf_ops; -static void io_notif_tw_complete(struct io_kiocb *notif, io_tw_token_t tw) +static void io_notif_tw_complete(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *notif = tw_req.req; struct io_notif_data *nd = io_notif_to_data(notif); struct io_ring_ctx *ctx = notif->ctx; @@ -34,7 +35,7 @@ static void io_notif_tw_complete(struct io_kiocb *notif, io_tw_token_t tw) } nd = nd->next; - io_req_task_complete(notif, tw); + io_req_task_complete((struct io_tw_req){notif}, tw); } while (nd); } diff --git a/io_uring/poll.c b/io_uring/poll.c index 6745bbc96328..f122a0070155 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -225,7 +225,7 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw) { int v; - if (unlikely(io_should_terminate_tw(req->ctx))) + if (unlikely(tw.cancel)) return -ECANCELED; do { @@ -316,8 +316,9 @@ static int io_poll_check_events(struct io_kiocb *req, io_tw_token_t tw) return IOU_POLL_NO_ACTION; } -void io_poll_task_func(struct io_kiocb *req, io_tw_token_t tw) +void io_poll_task_func(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; int ret; ret = io_poll_check_events(req, tw); @@ -338,7 +339,7 @@ void io_poll_task_func(struct io_kiocb *req, io_tw_token_t tw) poll = io_kiocb_to_cmd(req, struct io_poll); req->cqe.res = mangle_poll(req->cqe.res & poll->events); } else if (ret == IOU_POLL_REISSUE) { - io_req_task_submit(req, tw); + io_req_task_submit(tw_req, tw); return; } else if (ret != IOU_POLL_REMOVE_POLL_USE_RES) { req->cqe.res = ret; @@ -346,14 +347,14 @@ void io_poll_task_func(struct io_kiocb *req, io_tw_token_t tw) } io_req_set_res(req, req->cqe.res, 0); - io_req_task_complete(req, tw); + io_req_task_complete(tw_req, tw); } else { io_tw_lock(req->ctx, tw); if (ret == IOU_POLL_REMOVE_POLL_USE_RES) - io_req_task_complete(req, tw); + io_req_task_complete(tw_req, tw); else if (ret == IOU_POLL_DONE || ret == IOU_POLL_REISSUE) - io_req_task_submit(req, tw); + io_req_task_submit(tw_req, tw); else io_req_defer_failed(req, ret); } diff --git a/io_uring/poll.h b/io_uring/poll.h index c8438286dfa0..5647c5138932 100644 --- a/io_uring/poll.h +++ b/io_uring/poll.h @@ -46,4 +46,4 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags); bool io_poll_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx, bool cancel_all); -void io_poll_task_func(struct io_kiocb *req, io_tw_token_t tw); +void io_poll_task_func(struct io_tw_req tw_req, io_tw_token_t tw); diff --git a/io_uring/rw.c b/io_uring/rw.c index 3285a58409df..b78006228cc7 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -574,8 +574,9 @@ static inline int io_fixup_rw_res(struct io_kiocb *req, long res) return res; } -void io_req_rw_complete(struct io_kiocb *req, io_tw_token_t tw) +void io_req_rw_complete(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); struct kiocb *kiocb = &rw->kiocb; @@ -591,7 +592,7 @@ void io_req_rw_complete(struct io_kiocb *req, io_tw_token_t tw) req->cqe.flags |= io_put_kbuf(req, req->cqe.res, NULL); io_req_rw_cleanup(req, 0); - io_req_task_complete(req, tw); + io_req_task_complete(tw_req, tw); } static void io_complete_rw(struct kiocb *kiocb, long res) diff --git a/io_uring/rw.h b/io_uring/rw.h index 129a53fe5482..9bd7fbf70ea9 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -46,7 +46,7 @@ int io_read_fixed(struct io_kiocb *req, unsigned int issue_flags); int io_write_fixed(struct io_kiocb *req, unsigned int issue_flags); void io_readv_writev_cleanup(struct io_kiocb *req); void io_rw_fail(struct io_kiocb *req); -void io_req_rw_complete(struct io_kiocb *req, io_tw_token_t tw); +void io_req_rw_complete(struct io_tw_req tw_req, io_tw_token_t tw); int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags); void io_rw_cache_free(const void *entry); diff --git a/io_uring/timeout.c b/io_uring/timeout.c index dec1d13ca420..3d03e178044b 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -68,8 +68,9 @@ static inline bool io_timeout_finish(struct io_timeout *timeout, static enum hrtimer_restart io_timeout_fn(struct hrtimer *timer); -static void io_timeout_complete(struct io_kiocb *req, io_tw_token_t tw) +static void io_timeout_complete(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_timeout_data *data = req->async_data; struct io_ring_ctx *ctx = req->ctx; @@ -85,7 +86,7 @@ static void io_timeout_complete(struct io_kiocb *req, io_tw_token_t tw) } } - io_req_task_complete(req, tw); + io_req_task_complete(tw_req, tw); } static __cold bool io_flush_killed_timeouts(struct list_head *list, int err) @@ -157,8 +158,10 @@ __cold void io_flush_timeouts(struct io_ring_ctx *ctx) io_flush_killed_timeouts(&list, 0); } -static void io_req_tw_fail_links(struct io_kiocb *link, io_tw_token_t tw) +static void io_req_tw_fail_links(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *link = tw_req.req; + io_tw_lock(link->ctx, tw); while (link) { struct io_kiocb *nxt = link->link; @@ -168,7 +171,7 @@ static void io_req_tw_fail_links(struct io_kiocb *link, io_tw_token_t tw) res = link->cqe.res; link->link = NULL; io_req_set_res(link, res, 0); - io_req_task_complete(link, tw); + io_req_task_complete((struct io_tw_req){link}, tw); link = nxt; } } @@ -317,14 +320,15 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) return 0; } -static void io_req_task_link_timeout(struct io_kiocb *req, io_tw_token_t tw) +static void io_req_task_link_timeout(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_kiocb *prev = timeout->prev; int ret; if (prev) { - if (!io_should_terminate_tw(req->ctx)) { + if (!tw.cancel) { struct io_cancel_data cd = { .ctx = req->ctx, .data = prev->cqe.user_data, @@ -335,11 +339,11 @@ static void io_req_task_link_timeout(struct io_kiocb *req, io_tw_token_t tw) ret = -ECANCELED; } io_req_set_res(req, ret ?: -ETIME, 0); - io_req_task_complete(req, tw); + io_req_task_complete(tw_req, tw); io_put_req(prev); } else { io_req_set_res(req, -ETIME, 0); - io_req_task_complete(req, tw); + io_req_task_complete(tw_req, tw); } } diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index df521ce10b0a..d0fec7bb87d4 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -115,12 +115,13 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, } EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); -static void io_uring_cmd_work(struct io_kiocb *req, io_tw_token_t tw) +static void io_uring_cmd_work(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); unsigned int flags = IO_URING_F_COMPLETE_DEFER; - if (io_should_terminate_tw(req->ctx)) + if (unlikely(tw.cancel)) flags |= IO_URING_F_TASK_DEAD; /* task_work executor checks the deffered list completion */ diff --git a/io_uring/waitid.c b/io_uring/waitid.c index 97cce0bd2d50..2b756017423f 100644 --- a/io_uring/waitid.c +++ b/io_uring/waitid.c @@ -16,7 +16,7 @@ #include "waitid.h" #include "../kernel/exit.h" -static void io_waitid_cb(struct io_kiocb *req, io_tw_token_t tw); +static void io_waitid_cb(struct io_tw_req tw_req, io_tw_token_t tw); #define IO_WAITID_CANCEL_FLAG BIT(31) #define IO_WAITID_REF_MASK GENMASK(30, 0) @@ -109,7 +109,23 @@ static int io_waitid_finish(struct io_kiocb *req, int ret) return ret; } -static void io_waitid_complete(struct io_kiocb *req, int ret) +static void io_waitid_remove_wq(struct io_kiocb *req) +{ + struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); + struct wait_queue_head *head; + + head = smp_load_acquire(&iw->head); + if (head) { + struct io_waitid_async *iwa = req->async_data; + + smp_store_release(&iw->head, NULL); + spin_lock_irq(&head->lock); + list_del_init(&iwa->wo.child_wait.entry); + spin_unlock_irq(&head->lock); + } +} + +static void io_waitid_complete(struct io_kiocb *req, int ret, bool copy_si) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); @@ -119,17 +135,22 @@ static void io_waitid_complete(struct io_kiocb *req, int ret) lockdep_assert_held(&req->ctx->uring_lock); hlist_del_init(&req->hash_node); + io_waitid_remove_wq(req); - ret = io_waitid_finish(req, ret); + if (copy_si) + ret = io_waitid_finish(req, ret); + else + io_waitid_free(req); if (ret < 0) req_set_fail(req); io_req_set_res(req, ret, 0); } -static bool __io_waitid_cancel(struct io_kiocb *req) +static bool __io_waitid_cancel(struct io_kiocb *req, bool copy_si) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); - struct io_waitid_async *iwa = req->async_data; + + lockdep_assert_held(&req->ctx->uring_lock); /* * Mark us canceled regardless of ownership. This will prevent a @@ -141,51 +162,65 @@ static bool __io_waitid_cancel(struct io_kiocb *req) if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK) return false; - spin_lock_irq(&iw->head->lock); - list_del_init(&iwa->wo.child_wait.entry); - spin_unlock_irq(&iw->head->lock); - io_waitid_complete(req, -ECANCELED); + io_waitid_complete(req, -ECANCELED, copy_si); io_req_queue_tw_complete(req, -ECANCELED); return true; } +static bool io_waitid_cancel_cb(struct io_kiocb *req) +{ + return __io_waitid_cancel(req, true); +} + +static bool io_waitid_cancel_nocopy_cb(struct io_kiocb *req) +{ + return __io_waitid_cancel(req, false); +} + int io_waitid_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, unsigned int issue_flags) { - return io_cancel_remove(ctx, cd, issue_flags, &ctx->waitid_list, __io_waitid_cancel); + return io_cancel_remove(ctx, cd, issue_flags, &ctx->waitid_list, io_waitid_cancel_cb); } bool io_waitid_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx, bool cancel_all) { - return io_cancel_remove_all(ctx, tctx, &ctx->waitid_list, cancel_all, __io_waitid_cancel); + return io_cancel_remove_all(ctx, tctx, &ctx->waitid_list, cancel_all, + tctx ? io_waitid_cancel_cb : io_waitid_cancel_nocopy_cb); } static inline bool io_waitid_drop_issue_ref(struct io_kiocb *req) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); - struct io_waitid_async *iwa = req->async_data; if (!atomic_sub_return(1, &iw->refs)) return false; + io_waitid_remove_wq(req); + /* * Wakeup triggered, racing with us. It was prevented from * completing because of that, queue up the tw to do that. */ req->io_task_work.func = io_waitid_cb; io_req_task_work_add(req); - remove_wait_queue(iw->head, &iwa->wo.child_wait); return true; } -static void io_waitid_cb(struct io_kiocb *req, io_tw_token_t tw) +static void io_waitid_cb(struct io_tw_req tw_req, io_tw_token_t tw) { + struct io_kiocb *req = tw_req.req; struct io_waitid_async *iwa = req->async_data; struct io_ring_ctx *ctx = req->ctx; int ret; io_tw_lock(ctx, tw); + if (unlikely(tw.cancel)) { + io_waitid_complete(req, -ECANCELED, false); + io_req_task_complete(tw_req, tw); + return; + } ret = __do_wait(&iwa->wo); @@ -209,13 +244,12 @@ static void io_waitid_cb(struct io_kiocb *req, io_tw_token_t tw) io_waitid_drop_issue_ref(req); return; } - - remove_wait_queue(iw->head, &iwa->wo.child_wait); + /* fall through to complete, will kill waitqueue */ } } - io_waitid_complete(req, ret); - io_req_task_complete(req, tw); + io_waitid_complete(req, ret, true); + io_req_task_complete(tw_req, tw); } static int io_waitid_wait(struct wait_queue_entry *wait, unsigned mode, @@ -231,6 +265,7 @@ static int io_waitid_wait(struct wait_queue_entry *wait, unsigned mode, return 0; list_del_init(&wait->entry); + smp_store_release(&iw->head, NULL); /* cancel is in progress */ if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK) @@ -257,6 +292,7 @@ int io_waitid_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) iw->which = READ_ONCE(sqe->len); iw->upid = READ_ONCE(sqe->fd); iw->options = READ_ONCE(sqe->file_index); + iw->head = NULL; iw->infop = u64_to_user_ptr(READ_ONCE(sqe->addr2)); memset(&iw->info, 0, sizeof(iw->info)); return 0; @@ -288,11 +324,16 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags) * callback. */ io_ring_submit_lock(ctx, issue_flags); + + /* + * iw->head is valid under the ring lock, and as long as the request + * is on the waitid_list where cancelations may find it. + */ + iw->head = ¤t->signal->wait_chldexit; hlist_add_head(&req->hash_node, &ctx->waitid_list); init_waitqueue_func_entry(&iwa->wo.child_wait, io_waitid_wait); iwa->wo.child_wait.private = req->tctx->task; - iw->head = ¤t->signal->wait_chldexit; add_wait_queue(iw->head, &iwa->wo.child_wait); ret = __do_wait(&iwa->wo); @@ -315,7 +356,7 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags) } hlist_del_init(&req->hash_node); - remove_wait_queue(iw->head, &iwa->wo.child_wait); + io_waitid_remove_wq(req); ret = io_waitid_finish(req, ret); io_ring_submit_unlock(ctx, issue_flags); diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index dafa179da0c9..51227c8d2b1e 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -534,6 +534,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt) uaddr = (u32)uaddr; uaddr &= PAGE_MASK; full_uaddr = clear_lo32(arena->user_vm_start) + uaddr; + if (full_uaddr < arena->user_vm_start) + return; uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT)); if (full_uaddr >= uaddr_end) return; diff --git a/kernel/bpf/bpf_lsm.c b/kernel/bpf/bpf_lsm.c index 518c933fe944..48876fe838ab 100644 --- a/kernel/bpf/bpf_lsm.c +++ b/kernel/bpf/bpf_lsm.c @@ -293,7 +293,6 @@ BTF_ID(func, bpf_lsm_bpf_map_create) BTF_ID(func, bpf_lsm_bpf_map_free) BTF_ID(func, bpf_lsm_bpf_prog) BTF_ID(func, bpf_lsm_bpf_prog_load) -BTF_ID(func, bpf_lsm_bpf_prog_free) BTF_ID(func, bpf_lsm_bpf_token_create) BTF_ID(func, bpf_lsm_bpf_token_free) BTF_ID(func, bpf_lsm_bpf_token_cmd) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 7a4618c05407..f0b865afa9d1 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3504,7 +3504,7 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_ { BPF_LIST_NODE, "bpf_list_node", false }, { BPF_RB_ROOT, "bpf_rb_root", false }, { BPF_RB_NODE, "bpf_rb_node", false }, - { BPF_REFCOUNT, "bpf_refcount", false }, + { BPF_REFCOUNT, "bpf_refcount", true }, }; int type = 0, i; const char *name = __btf_name_by_offset(btf, var_type->name_off); @@ -3586,7 +3586,7 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, int info_cnt, - u32 level); + u32 level, u32 *seen_mask); /* Find special fields in the struct type of a field. * @@ -3597,7 +3597,7 @@ static int btf_find_struct_field(const struct btf *btf, static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t, u32 off, u32 nelems, u32 field_mask, struct btf_field_info *info, - int info_cnt, u32 level) + int info_cnt, u32 level, u32 *seen_mask) { int ret, err, i; @@ -3605,7 +3605,7 @@ static int btf_find_nested_struct(const struct btf *btf, const struct btf_type * if (level >= MAX_RESOLVE_DEPTH) return -E2BIG; - ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level); + ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level, seen_mask); if (ret <= 0) return ret; @@ -3662,7 +3662,7 @@ static int btf_find_field_one(const struct btf *btf, if (expected_size && expected_size != sz * nelems) return 0; ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask, - &info[0], info_cnt, level); + &info[0], info_cnt, level, seen_mask); return ret; } @@ -3727,11 +3727,11 @@ static int btf_find_field_one(const struct btf *btf, static int btf_find_struct_field(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, int info_cnt, - u32 level) + u32 level, u32 *seen_mask) { int ret, idx = 0; const struct btf_member *member; - u32 i, off, seen_mask = 0; + u32 i, off; for_each_member(i, t, member) { const struct btf_type *member_type = btf_type_by_id(btf, @@ -3745,7 +3745,7 @@ static int btf_find_struct_field(const struct btf *btf, ret = btf_find_field_one(btf, t, member_type, i, off, 0, - field_mask, &seen_mask, + field_mask, seen_mask, &info[idx], info_cnt - idx, level); if (ret < 0) return ret; @@ -3756,11 +3756,11 @@ static int btf_find_struct_field(const struct btf *btf, static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, - int info_cnt, u32 level) + int info_cnt, u32 level, u32 *seen_mask) { int ret, idx = 0; const struct btf_var_secinfo *vsi; - u32 i, off, seen_mask = 0; + u32 i, off; for_each_vsi(i, t, vsi) { const struct btf_type *var = btf_type_by_id(btf, vsi->type); @@ -3768,7 +3768,7 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, off = vsi->offset; ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size, - field_mask, &seen_mask, + field_mask, seen_mask, &info[idx], info_cnt - idx, level); if (ret < 0) @@ -3782,10 +3782,12 @@ static int btf_find_field(const struct btf *btf, const struct btf_type *t, u32 field_mask, struct btf_field_info *info, int info_cnt) { + u32 seen_mask = 0; + if (__btf_type_is_struct(t)) - return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0); + return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0, &seen_mask); else if (btf_type_is_datasec(t)) - return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0); + return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0, &seen_mask); return -EINVAL; } @@ -4003,7 +4005,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type rec->spin_lock_off = rec->fields[i].offset; break; case BPF_RES_SPIN_LOCK: - WARN_ON_ONCE(rec->spin_lock_off >= 0); + WARN_ON_ONCE(rec->res_spin_lock_off >= 0); /* Cache offset for faster lookup at runtime */ rec->res_spin_lock_off = rec->fields[i].offset; break; diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index fc2f44b60a55..8980373c7dab 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -813,8 +813,10 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *old_prog = NULL; struct bpf_cgroup_storage *storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {}; struct bpf_cgroup_storage *new_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {}; + struct bpf_cgroup_storage *old_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {}; struct bpf_prog *new_prog = prog ? : link->link.prog; enum cgroup_bpf_attach_type atype; + u32 old_flags, old_pl_flags; struct bpf_prog_list *pl; struct hlist_head *progs; int err; @@ -865,6 +867,8 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, if (pl) { old_prog = pl->prog; + old_pl_flags = pl->flags; + bpf_cgroup_storages_assign(old_storage, pl->storage); } else { pl = kmalloc(sizeof(*pl), GFP_KERNEL); if (!pl) { @@ -884,6 +888,7 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, pl->link = link; pl->flags = flags; bpf_cgroup_storages_assign(pl->storage, storage); + old_flags = cgrp->bpf.flags[atype]; cgrp->bpf.flags[atype] = saved_flags; if (type == BPF_LSM_CGROUP) { @@ -915,12 +920,15 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp, if (old_prog) { pl->prog = old_prog; pl->link = NULL; + pl->flags = old_pl_flags; + bpf_cgroup_storages_assign(pl->storage, old_storage); } bpf_cgroup_storages_free(new_storage); if (!old_prog) { hlist_del(&pl->node); kfree(pl); } + cgrp->bpf.flags[atype] = old_flags; return err; } @@ -1018,6 +1026,20 @@ static void replace_effective_prog(struct cgroup *cgrp, } } +static bool cgroup_bpf_storages_compatible(struct bpf_prog *old_prog, + struct bpf_prog *new_prog) +{ + enum bpf_cgroup_storage_type stype; + + for_each_cgroup_storage_type(stype) { + if (old_prog->aux->cgroup_storage[stype] != + new_prog->aux->cgroup_storage[stype]) + return false; + } + + return true; +} + /** * __cgroup_bpf_replace() - Replace link's program and propagate the change * to descendants @@ -1056,6 +1078,9 @@ static int __cgroup_bpf_replace(struct cgroup *cgrp, if (!found) return -ENOENT; + if (!cgroup_bpf_storages_compatible(link->link.prog, new_prog)) + return -EINVAL; + cgrp->bpf.revisions[atype] += 1; old_prog = xchg(&link->link.prog, new_prog); replace_effective_prog(cgrp, atype, pl); @@ -2239,7 +2264,7 @@ int __cgroup_bpf_run_filter_getsockopt_kern(struct sock *sk, int level, if (ret < 0) return ret; - if (ctx.optlen > *optlen) + if (ctx.optlen > *optlen || ctx.optlen < 0) return -EFAULT; /* BPF programs can shrink the buffer, export the modifications. diff --git a/kernel/bpf/cpumask.c b/kernel/bpf/cpumask.c index 9876c5fe6c2a..341f80bb72ff 100644 --- a/kernel/bpf/cpumask.c +++ b/kernel/bpf/cpumask.c @@ -449,12 +449,12 @@ __bpf_kfunc u32 bpf_cpumask_weight(const struct cpumask *cpumask) * @src__sz: Length of the BPF memory region in bytes. * * Return: - * * 0 if the struct cpumask * instance was populated successfully. + * * 0 if the struct bpf_cpumask * instance was populated successfully. * * -EACCES if the memory region is too small to populate the cpumask. * * -EINVAL if the memory region is not aligned to the size of a long * and the architecture does not support efficient unaligned accesses. */ -__bpf_kfunc int bpf_cpumask_populate(struct cpumask *cpumask, void *src, size_t src__sz) +__bpf_kfunc int bpf_cpumask_populate(struct bpf_cpumask *cpumask, void *src, size_t src__sz) { unsigned long source = (unsigned long)src; @@ -467,7 +467,7 @@ __bpf_kfunc int bpf_cpumask_populate(struct cpumask *cpumask, void *src, size_t !IS_ALIGNED(source, sizeof(long))) return -EINVAL; - bitmap_copy(cpumask_bits(cpumask), src, nr_cpu_ids); + bitmap_copy(cpumask_bits(&cpumask->cpumask), src, nr_cpu_ids); return 0; } diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 5aed5659822f..72757f7290ec 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -3960,7 +3960,7 @@ struct bpf_task_work_ctx { struct bpf_map *map; void *map_val; enum task_work_notify_mode mode; - bpf_task_work_callback_t callback_fn; + bpf_callback_t callback_fn; struct rcu_head rcu; } __aligned(8); @@ -4035,7 +4035,8 @@ static void bpf_task_work_callback(struct callback_head *cb) key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx); migrate_disable(); - ctx->callback_fn(ctx->map, key, ctx->map_val); + ctx->callback_fn((u64)(long)ctx->map, (u64)(long)key, + (u64)(long)ctx->map_val, 0, 0); migrate_enable(); bpf_task_work_ctx_reset(ctx); @@ -4148,7 +4149,7 @@ static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work } static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw, - struct bpf_map *map, bpf_task_work_callback_t callback_fn, + struct bpf_map *map, void *callback_fn, struct bpf_prog_aux *aux, enum task_work_notify_mode mode) { struct bpf_prog *prog; @@ -4173,7 +4174,7 @@ static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work } ctx->task = task; - ctx->callback_fn = callback_fn; + ctx->callback_fn = (bpf_callback_t)callback_fn; ctx->prog = prog; ctx->mode = mode; ctx->map = map; diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h index 5d18d7d85bef..1834db20b861 100644 --- a/kernel/bpf/mmap_unlock_work.h +++ b/kernel/bpf/mmap_unlock_work.h @@ -4,12 +4,15 @@ #ifndef __MMAP_UNLOCK_WORK_H__ #define __MMAP_UNLOCK_WORK_H__ +#include +#include #include /* irq_work to run mmap_read_unlock() in irq_work */ struct mmap_unlock_irq_work { struct irq_work irq_work; struct mm_struct *mm; + atomic_t active; }; DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); @@ -18,32 +21,36 @@ DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); * We cannot do mmap_read_unlock() when the irq is disabled, because of * risk to deadlock with rq_lock. To look up vma when the irqs are * disabled, we need to run mmap_read_unlock() in irq_work. We use a - * percpu variable to do the irq_work. If the irq_work is already used - * by another lookup, we fall over. + * percpu variable to do the irq_work. The active flag reserves the slot + * before mmap_read_trylock() and until the irq_work callback consumes mm. */ -static inline bool bpf_mmap_unlock_get_irq_work(struct mmap_unlock_irq_work **work_ptr) +static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void) { - struct mmap_unlock_irq_work *work = NULL; - bool irq_work_busy = false; + struct mmap_unlock_irq_work *work; - if (irqs_disabled()) { - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) { - work = this_cpu_ptr(&mmap_unlock_work); - if (irq_work_is_busy(&work->irq_work)) { - /* cannot queue more up_read, fallback */ - irq_work_busy = true; - } - } else { - /* - * PREEMPT_RT does not allow to trylock mmap sem in - * interrupt disabled context. Force the fallback code. - */ - irq_work_busy = true; - } - } + if (!irqs_disabled()) + return NULL; + + /* + * PREEMPT_RT does not allow to trylock mmap sem in interrupt + * disabled context. Force the fallback code. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + return ERR_PTR(-EBUSY); + + work = this_cpu_ptr(&mmap_unlock_work); + if (irq_work_is_busy(&work->irq_work) || + atomic_cmpxchg_acquire(&work->active, 0, 1)) + return ERR_PTR(-EBUSY); - *work_ptr = work; - return irq_work_busy; + return work; +} + +static inline void +bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work) +{ + if (work) + atomic_set_release(&work->active, 0); } static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm) diff --git a/kernel/bpf/net_namespace.c b/kernel/bpf/net_namespace.c index 8e88201c98bf..6599d771e481 100644 --- a/kernel/bpf/net_namespace.c +++ b/kernel/bpf/net_namespace.c @@ -171,33 +171,28 @@ static int bpf_netns_link_update_prog(struct bpf_link *link, struct net *net; int idx, ret; + guard(mutex)(&netns_bpf_mutex); + if (old_prog && old_prog != link->prog) return -EPERM; if (new_prog->type != link->prog->type) return -EINVAL; - mutex_lock(&netns_bpf_mutex); - net = net_link->net; - if (!net || !check_net(net)) { + if (!net || !check_net(net)) /* Link auto-detached or netns dying */ - ret = -ENOLINK; - goto out_unlock; - } + return -ENOLINK; run_array = rcu_dereference_protected(net->bpf.run_array[type], lockdep_is_held(&netns_bpf_mutex)); idx = link_index(net, type, net_link); ret = bpf_prog_array_update_at(run_array, idx, new_prog); if (ret) - goto out_unlock; + return ret; old_prog = xchg(&link->prog, new_prog); bpf_prog_put(old_prog); - -out_unlock: - mutex_unlock(&netns_bpf_mutex); - return ret; + return 0; } static int bpf_netns_link_fill_info(const struct bpf_link *link, diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index 9a5f94371e50..c1c9dee4dcdd 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -99,8 +99,10 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete) int err = 0; void *ptr; - if (raw_res_spin_lock_irqsave(&qs->lock, flags)) + if (raw_res_spin_lock_irqsave(&qs->lock, flags)) { + memset(value, 0, qs->map.value_size); return -EBUSY; + } if (queue_stack_map_is_empty(qs)) { memset(value, 0, qs->map.value_size); @@ -130,8 +132,10 @@ static long __stack_map_get(struct bpf_map *map, void *value, bool delete) void *ptr; u32 index; - if (raw_res_spin_lock_irqsave(&qs->lock, flags)) + if (raw_res_spin_lock_irqsave(&qs->lock, flags)) { + memset(value, 0, qs->map.value_size); return -EBUSY; + } if (queue_stack_map_is_empty(qs)) { memset(value, 0, qs->map.value_size); diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index d706c4b7f532..bdc524c8ed79 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -426,7 +426,7 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size) prod_pos = rb->producer_pos; new_prod_pos = prod_pos + len; - while (pend_pos < prod_pos) { + while (prod_pos - pend_pos > 0) { hdr = (void *)rb->data + (pend_pos & rb->mask); hdr_len = READ_ONCE(hdr->len); if (hdr_len & BPF_RINGBUF_BUSY_BIT) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 49ef03ab0595..4f33c547a094 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "percpu_freelist.h" #include "mmap_unlock_work.h" @@ -152,6 +153,131 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b : build_id_parse_nofault(vma, build_id, NULL); } +static inline void stack_map_build_id_set_ip(struct bpf_stack_build_id *id) +{ + id->status = BPF_STACK_BUILD_ID_IP; + memset(id->build_id, 0, BUILD_ID_SIZE_MAX); +} + +static inline u64 stack_map_build_id_offset(unsigned long vm_pgoff, + unsigned long vm_start, u64 ip) +{ + return (vm_pgoff << PAGE_SHIFT) + ip - vm_start; +} + +static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id, + u64 offset, + const unsigned char *build_id) +{ + id->status = BPF_STACK_BUILD_ID_VALID; + id->offset = offset; + if (id->build_id != build_id) + memcpy(id->build_id, build_id, BUILD_ID_SIZE_MAX); +} + +struct stack_map_vma_lock { + struct vm_area_struct *vma; + struct mm_struct *mm; +}; + +/* + * Acquire a stable read-side reference on the VMA covering @ip. + * + * With CONFIG_PER_VMA_LOCK=y this returns a VMA with its per-VMA read + * lock held and mmap_lock dropped, so the caller may sleep. + * + * With CONFIG_PER_VMA_LOCK=n it returns a VMA with mmap_lock still + * held; the caller must snapshot any fields it needs and pin vm_file + * with get_file() before stack_map_unlock_vma() drops mmap_lock, as + * the VMA may be split, merged, or freed after that. + * + * Returns NULL on failure, in which case no lock is held. + */ +static struct vm_area_struct * +stack_map_lock_vma(struct stack_map_vma_lock *lock, unsigned long ip) +{ + struct mm_struct *mm = lock->mm; + struct vm_area_struct *vma; + + /* noop under !CONFIG_PER_VMA_LOCK */ + vma = lock_vma_under_rcu(mm, ip); + if (vma) { + lock->vma = vma; + return vma; + } + + /* + * Taking mmap_read_lock() is unsafe here, because the caller BPF + * program might already hold it, causing a deadlock. + */ + if (!mmap_read_trylock(mm)) + return NULL; + + vma = vma_lookup(mm, ip); + if (!vma) { + mmap_read_unlock(mm); + return NULL; + } + +#ifdef CONFIG_PER_VMA_LOCK + if (!vma_start_read_locked(vma)) { + mmap_read_unlock(mm); + return NULL; + } + mmap_read_unlock(mm); +#endif + + lock->vma = vma; + return vma; +} + +static void stack_map_unlock_vma(struct stack_map_vma_lock *lock) +{ +#ifdef CONFIG_PER_VMA_LOCK + vma_end_read(lock->vma); +#else + mmap_read_unlock(lock->mm); +#endif + lock->vma = NULL; +} + +static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *id_offs, + u32 trace_nr) +{ + struct mm_struct *mm = current->mm; + struct stack_map_vma_lock lock = { .mm = mm }; + struct vm_area_struct *vma; + struct file *file; + u64 offset; + u64 ip; + + for (u32 i = 0; i < trace_nr; i++) { + ip = READ_ONCE(id_offs[i].ip); + + vma = stack_map_lock_vma(&lock, ip); + if (!vma) { + stack_map_build_id_set_ip(&id_offs[i]); + continue; + } + if (vma_is_anonymous(vma) || !vma->vm_file) { + stack_map_build_id_set_ip(&id_offs[i]); + stack_map_unlock_vma(&lock); + continue; + } + + file = get_file(vma->vm_file); + offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip); + stack_map_unlock_vma(&lock); + + /* build_id_parse_file() may block on filesystem reads */ + if (build_id_parse_file(file, id_offs[i].build_id, NULL)) + stack_map_build_id_set_ip(&id_offs[i]); + else + stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id); + fput(file); + } +} + /* * Expects all id_offs[i].ip values to be set to correct initial IPs. * They will be subsequently: @@ -165,48 +291,60 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, u32 trace_nr, bool user, bool may_fault) { - int i; - struct mmap_unlock_irq_work *work = NULL; - bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); + struct mmap_unlock_irq_work *work; + bool has_user_ctx = user && current && current->mm; struct vm_area_struct *vma, *prev_vma = NULL; - const char *prev_build_id; + const unsigned char *prev_build_id = NULL; + int i; - /* If the irq_work is in use, fall back to report ips. Same - * fallback is used for kernel stack (!user) on a stackmap with - * build_id. - */ - if (!user || !current || !current->mm || irq_work_busy || - !mmap_read_trylock(current->mm)) { - /* cannot access current->mm, fall back to ips */ - for (i = 0; i < trace_nr; i++) { - id_offs[i].status = BPF_STACK_BUILD_ID_IP; - memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); - } + if (may_fault && has_user_ctx) { + stack_map_get_build_id_offset_sleepable(id_offs, trace_nr); return; } + if (!has_user_ctx) + goto fallback; + + work = bpf_mmap_unlock_guard_get(); + if (IS_ERR(work)) + goto fallback; + + if (!mmap_read_trylock(current->mm)) { + bpf_mmap_unlock_guard_put(work); + goto fallback; + } + for (i = 0; i < trace_nr; i++) { u64 ip = READ_ONCE(id_offs[i].ip); + u64 offset; - if (range_in_vma(prev_vma, ip, ip)) { + if (prev_build_id && range_in_vma(prev_vma, ip, ip)) { vma = prev_vma; - memcpy(id_offs[i].build_id, prev_build_id, BUILD_ID_SIZE_MAX); - goto build_id_valid; + offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip); + stack_map_build_id_set_valid(&id_offs[i], offset, prev_build_id); + continue; } vma = find_vma(current->mm, ip); - if (!vma || fetch_build_id(vma, id_offs[i].build_id, may_fault)) { + if (!vma || vma_is_anonymous(vma) || + fetch_build_id(vma, id_offs[i].build_id, may_fault)) { /* per entry fall back to ips */ - id_offs[i].status = BPF_STACK_BUILD_ID_IP; - memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); + stack_map_build_id_set_ip(&id_offs[i]); + prev_vma = vma; + prev_build_id = NULL; continue; } -build_id_valid: - id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ip - vma->vm_start; - id_offs[i].status = BPF_STACK_BUILD_ID_VALID; + offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip); + stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id); prev_vma = vma; prev_build_id = id_offs[i].build_id; } bpf_mmap_unlock_mm(work, current->mm); + return; + +fallback: + /* cannot access current->mm, fall back to ips */ + for (i = 0; i < trace_nr; i++) + stack_map_build_id_set_ip(&id_offs[i]); } static struct perf_callchain_entry * diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index e92fd2bec98c..bab472152d64 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1593,7 +1593,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) err = security_bpf_map_create(map, attr, token, uattr.is_kernel); if (err) - goto free_map_sec; + goto free_map; err = bpf_map_alloc_id(map); if (err) @@ -3343,9 +3343,10 @@ static const char *bpf_link_type_strs[] = { static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) { const struct bpf_link *link = filp->private_data; - const struct bpf_prog *prog = link->prog; + const struct bpf_prog *prog; enum bpf_link_type type = link->type; char prog_tag[sizeof(prog->tag) * 2 + 1] = { }; + u32 prog_id = 0; if (type < ARRAY_SIZE(bpf_link_type_strs) && bpf_link_type_strs[type]) { if (link->type == BPF_LINK_TYPE_KPROBE_MULTI) @@ -3362,13 +3363,20 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) } seq_printf(m, "link_id:\t%u\n", link->id); + rcu_read_lock(); + prog = READ_ONCE(link->prog); if (prog) { bin2hex(prog_tag, prog->tag, sizeof(prog->tag)); + prog_id = prog->aux->id; + } + rcu_read_unlock(); + + if (prog) { seq_printf(m, "prog_tag:\t%s\n" "prog_id:\t%u\n", prog_tag, - prog->aux->id); + prog_id); } if (link->ops->show_fdinfo) link->ops->show_fdinfo(link, m); @@ -5372,6 +5380,7 @@ static int bpf_link_get_info_by_fd(struct file *file, { struct bpf_link_info __user *uinfo = u64_to_user_ptr(attr->info.info); struct bpf_link_info info; + const struct bpf_prog *prog; u32 info_len = attr->info.info_len; int err; @@ -5386,8 +5395,12 @@ static int bpf_link_get_info_by_fd(struct file *file, info.type = link->type; info.id = link->id; - if (link->prog) - info.prog_id = link->prog->aux->id; + + rcu_read_lock(); + prog = READ_ONCE(link->prog); + if (prog) + info.prog_id = prog->aux->id; + rcu_read_unlock(); if (link->ops->fill_link_info) { err = link->ops->fill_link_info(link, &info); diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index e791ae065c39..13e1aabe6f88 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -753,9 +753,9 @@ static struct bpf_iter_reg task_vma_reg_info = { BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, bpf_callback_t, callback_fn, void *, callback_ctx, u64, flags) { - struct mmap_unlock_irq_work *work = NULL; + struct mmap_unlock_irq_work *work; struct vm_area_struct *vma; - bool irq_work_busy = false; + bool __maybe_unused mmput_needed = false; struct mm_struct *mm; int ret = -ENOENT; @@ -765,14 +765,43 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, if (!task) return -ENOENT; - mm = task->mm; + if (task == current) { + mm = task->mm; + } else { + /* + * Foreign task: pin task->mm against a concurrent exit_mm(). + * Use trylock on alloc_lock instead of get_task_mm()'s + * blocking task_lock() to avoid deadlocking the target task. + */ + if (!IS_ENABLED(CONFIG_MMU)) + return -EOPNOTSUPP; + if (irqs_disabled()) + return -EBUSY; + if (!spin_trylock(&task->alloc_lock)) + return -EBUSY; + mm = task->mm; + if (mm && !(task->flags & PF_KTHREAD)) { + mmget(mm); + mmput_needed = true; + } else { + mm = NULL; + } + spin_unlock(&task->alloc_lock); + } if (!mm) return -ENOENT; - irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); + work = bpf_mmap_unlock_guard_get(); + if (IS_ERR(work)) { + ret = PTR_ERR(work); + goto out; + } - if (irq_work_busy || !mmap_read_trylock(mm)) - return -EBUSY; + if (!mmap_read_trylock(mm)) { + bpf_mmap_unlock_guard_put(work); + ret = -EBUSY; + goto out; + } vma = find_vma(mm, start); @@ -782,6 +811,11 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, ret = 0; } bpf_mmap_unlock_mm(work, mm); +out: +#ifdef CONFIG_MMU + if (mmput_needed) + mmput_async(mm); +#endif return ret; } @@ -1161,6 +1195,8 @@ static void do_mmap_read_unlock(struct irq_work *entry) work = container_of(entry, struct mmap_unlock_irq_work, irq_work); mmap_read_unlock_non_owner(work->mm); + work->mm = NULL; + bpf_mmap_unlock_guard_put(work); } static int __init task_iter_init(void) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 140be10eeb7b..c19eb8a7d396 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6385,6 +6385,30 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno) return reg->type == PTR_TO_ARENA; } +static bool is_load_acq_unsafe(struct bpf_verifier_env *env, int regno, + struct bpf_insn *insn) +{ + const struct bpf_reg_state *reg = reg_state(env, regno); + + /* + * A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the + * verifier, unlike a regular BPF_LDX. The JIT would emit a plain load + * with no exception table entry, so a fault (e.g. NULL deref) crashes + * the kernel instead of being handled. + * + * Reject the source pointer types that a BPF_LDX would have had that + * fault protection applied to, i.e. the ones bpf_convert_ctx_accesses() + * turns into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID and any PTR_UNTRUSTED + * pointer (untrusted btf ids, untrusted MEM_ALLOC, rdonly untrusted + * memory). A PTR_TRUSTED pointer is not among them, is not converted, + * and stays allowed. Same for the other flagged PTR_TO_BTF_ID variants + * (MEM_ALLOC, MEM_RCU, ...), hence the exact match on the base type. + */ + return insn->imm == BPF_LOAD_ACQ && + (reg->type == PTR_TO_BTF_ID || + (type_flag(reg->type) & PTR_UNTRUSTED)); +} + /* Return false if @regno contains a pointer whose type isn't supported for * atomic instruction @insn. */ @@ -6401,7 +6425,8 @@ static bool atomic_ptr_type_ok(struct bpf_verifier_env *env, int regno, return false; if (is_arena_reg(env, regno)) return bpf_jit_supports_insn(insn, true); - + if (is_load_acq_unsafe(env, regno, insn)) + return false; return true; } @@ -7975,7 +8000,7 @@ static int check_atomic_load(struct bpf_verifier_env *env, { int err; - err = check_load_mem(env, insn, true, false, false, "atomic_load"); + err = check_reg_arg(env, insn->src_reg, SRC_OP); if (err) return err; @@ -7986,7 +8011,7 @@ static int check_atomic_load(struct bpf_verifier_env *env, return -EACCES; } - return 0; + return check_load_mem(env, insn, true, false, false, "atomic_load"); } static int check_atomic_store(struct bpf_verifier_env *env, @@ -24406,13 +24431,25 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) struct btf *bpf_get_btf_vmlinux(void) { - if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { + /* Pairs with the smp_store_release() on the parse path below. */ + struct btf *btf = smp_load_acquire(&btf_vmlinux); + + if (!btf && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) { mutex_lock(&bpf_verifier_lock); - if (!btf_vmlinux) - btf_vmlinux = btf_parse_vmlinux(); + btf = btf_vmlinux; + if (!btf) { + btf = btf_parse_vmlinux(); + /* + * Order the parsed BTF contents and the globals the + * parse populated (e.g. bpf_ctx_convert.t) before + * the pointer publication. Pairs with the acquire + * on the lockless fast path above. + */ + smp_store_release(&btf_vmlinux, btf); + } mutex_unlock(&bpf_verifier_lock); } - return btf_vmlinux; + return btf; } /* diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h index 337608f408ce..0eb9ebd4fcb9 100644 --- a/kernel/cgroup/cpuset-internal.h +++ b/kernel/cgroup/cpuset-internal.h @@ -165,7 +165,7 @@ struct cpuset { * number of SCHED_DEADLINE tasks attached to this cpuset, so that we * know when to rebuild associated root domain bandwidth information. */ - int nr_deadline_tasks; + atomic_t nr_deadline_tasks; int nr_migrate_dl_tasks; u64 sum_migrate_dl_bw; diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index 2386090351cd..6ab677c00218 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -145,14 +145,14 @@ void inc_dl_tasks_cs(struct task_struct *p) { struct cpuset *cs = task_cs(p); - cs->nr_deadline_tasks++; + atomic_inc(&cs->nr_deadline_tasks); } void dec_dl_tasks_cs(struct task_struct *p) { struct cpuset *cs = task_cs(p); - cs->nr_deadline_tasks--; + atomic_dec(&cs->nr_deadline_tasks); } static inline bool is_partition_valid(const struct cpuset *cs) @@ -1033,7 +1033,7 @@ static void dl_update_tasks_root_domain(struct cpuset *cs) struct css_task_iter it; struct task_struct *task; - if (cs->nr_deadline_tasks == 0) + if (atomic_read(&cs->nr_deadline_tasks) == 0) return; css_task_iter_start(&cs->css, 0, &it); @@ -1410,7 +1410,46 @@ static bool partition_xcpus_del(int old_prs, struct cpuset *parent, return isolcpus_updated; } -static void update_unbound_workqueue_cpumask(bool isolcpus_updated) +/* + * isolated_cpus_can_update - check for isolated & nohz_full conflicts + * @add_cpus: cpu mask for cpus that are going to be isolated + * @del_cpus: cpu mask for cpus that are no longer isolated, can be NULL + * Return: false if there is conflict, true otherwise + * + * If nohz_full is enabled and we have isolated CPUs, their combination must + * still leave housekeeping CPUs. + * + * TBD: Should consider merging this function into + * prstate_housekeeping_conflict(). + */ +static bool isolated_cpus_can_update(struct cpumask *add_cpus, + struct cpumask *del_cpus) +{ + cpumask_var_t full_hk_cpus; + int res = true; + + if (!housekeeping_enabled(HK_TYPE_KERNEL_NOISE)) + return true; + + if (del_cpus && cpumask_weight_and(del_cpus, + housekeeping_cpumask(HK_TYPE_KERNEL_NOISE))) + return true; + + if (!alloc_cpumask_var(&full_hk_cpus, GFP_KERNEL)) + return false; + + cpumask_and(full_hk_cpus, housekeeping_cpumask(HK_TYPE_KERNEL_NOISE), + housekeeping_cpumask(HK_TYPE_DOMAIN)); + cpumask_andnot(full_hk_cpus, full_hk_cpus, isolated_cpus); + cpumask_and(full_hk_cpus, full_hk_cpus, cpu_active_mask); + if (!cpumask_weight_andnot(full_hk_cpus, add_cpus)) + res = false; + + free_cpumask_var(full_hk_cpus); + return res; +} + +static void update_isolation_cpumasks(bool isolcpus_updated) { int ret; @@ -1568,15 +1607,18 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs, if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) || cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus)) return PERR_INVCPUS; + if ((new_prs == PRS_ISOLATED) && + !isolated_cpus_can_update(tmp->new_cpus, NULL)) + return PERR_HKEEPING; spin_lock_irq(&callback_lock); isolcpus_updated = partition_xcpus_add(new_prs, NULL, tmp->new_cpus); list_add(&cs->remote_sibling, &remote_children); cpumask_copy(cs->effective_xcpus, tmp->new_cpus); spin_unlock_irq(&callback_lock); - update_unbound_workqueue_cpumask(isolcpus_updated); + update_isolation_cpumasks(isolcpus_updated); cpuset_force_rebuild(); - cs->prs_err = 0; + WRITE_ONCE(cs->prs_err, 0); /* * Propagate changes in top_cpuset's effective_cpus down the hierarchy. @@ -1622,7 +1664,7 @@ static void remote_partition_disable(struct cpuset *cs, struct tmpmasks *tmp) compute_excpus(cs, cs->effective_xcpus); reset_partition_data(cs); spin_unlock_irq(&callback_lock); - update_unbound_workqueue_cpumask(isolcpus_updated); + update_isolation_cpumasks(isolcpus_updated); cpuset_force_rebuild(); /* @@ -1655,7 +1697,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus, WARN_ON_ONCE(!cpumask_subset(cs->effective_xcpus, subpartitions_cpus)); if (cpumask_empty(excpus)) { - cs->prs_err = PERR_CPUSEMPTY; + WRITE_ONCE(cs->prs_err, PERR_CPUSEMPTY); goto invalidate; } @@ -1670,10 +1712,13 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus, if (adding) { WARN_ON_ONCE(cpumask_intersects(tmp->addmask, subpartitions_cpus)); if (!capable(CAP_SYS_ADMIN)) - cs->prs_err = PERR_ACCESS; + WRITE_ONCE(cs->prs_err, PERR_ACCESS); else if (cpumask_intersects(tmp->addmask, subpartitions_cpus) || cpumask_subset(top_cpuset.effective_cpus, tmp->addmask)) - cs->prs_err = PERR_NOCPUS; + WRITE_ONCE(cs->prs_err, PERR_NOCPUS); + else if ((prs == PRS_ISOLATED) && + !isolated_cpus_can_update(tmp->addmask, tmp->delmask)) + WRITE_ONCE(cs->prs_err, PERR_HKEEPING); if (cs->prs_err) goto invalidate; } @@ -1691,7 +1736,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus, if (xcpus) cpumask_copy(cs->exclusive_cpus, xcpus); spin_unlock_irq(&callback_lock); - update_unbound_workqueue_cpumask(isolcpus_updated); + update_isolation_cpumasks(isolcpus_updated); if (adding || deleting) cpuset_force_rebuild(); @@ -1775,6 +1820,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, int subparts_delta = 0; int isolcpus_updated = 0; struct cpumask *xcpus = user_xcpus(cs); + int parent_prs = parent->partition_root_state; bool nocpu; lockdep_assert_held(&cpuset_mutex); @@ -1839,6 +1885,10 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, if (prstate_housekeeping_conflict(new_prs, xcpus)) return PERR_HKEEPING; + if ((new_prs == PRS_ISOLATED) && (new_prs != parent_prs) && + !isolated_cpus_can_update(xcpus, NULL)) + return PERR_HKEEPING; + if (tasks_nocpu_error(parent, cs, xcpus)) return PERR_NOCPUS; @@ -1894,6 +1944,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, * * For invalid partition: * delmask = newmask & parent->effective_xcpus + * The partition may become valid soon. */ if (is_partition_invalid(cs)) { adding = false; @@ -1908,6 +1959,23 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, deleting = cpumask_and(tmp->delmask, tmp->delmask, parent->effective_xcpus); } + + /* + * TBD: Invalidate a currently valid child root partition may + * still break isolated_cpus_can_update() rule if parent is an + * isolated partition. + */ + if (is_partition_valid(cs) && (old_prs != parent_prs)) { + if ((parent_prs == PRS_ROOT) && + /* Adding to parent means removing isolated CPUs */ + !isolated_cpus_can_update(tmp->delmask, tmp->addmask)) + part_error = PERR_HKEEPING; + if ((parent_prs == PRS_ISOLATED) && + /* Adding to parent means adding isolated CPUs */ + !isolated_cpus_can_update(tmp->addmask, tmp->delmask)) + part_error = PERR_HKEEPING; + } + /* * The new CPUs to be removed from parent's effective CPUs * must be present. @@ -2049,7 +2117,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, WARN_ON_ONCE(parent->nr_subparts < 0); } spin_unlock_irq(&callback_lock); - update_unbound_workqueue_cpumask(isolcpus_updated); + update_isolation_cpumasks(isolcpus_updated); if ((old_prs != new_prs) && (cmd == partcmd_update)) update_partition_exclusive_flag(cs, new_prs); @@ -2115,13 +2183,13 @@ static void compute_partition_effective_cpumask(struct cpuset *cs, * partition root. */ WARN_ON_ONCE(is_remote_partition(child)); - child->prs_err = 0; + WRITE_ONCE(child->prs_err, 0); if (!cpumask_subset(child->effective_xcpus, cs->effective_xcpus)) - child->prs_err = PERR_INVCPUS; + WRITE_ONCE(child->prs_err, PERR_INVCPUS); else if (populated && cpumask_subset(new_ecpus, child->effective_xcpus)) - child->prs_err = PERR_NOCPUS; + WRITE_ONCE(child->prs_err, PERR_NOCPUS); if (child->prs_err) { int old_prs = child->partition_root_state; @@ -2488,8 +2556,10 @@ static void partition_cpus_change(struct cpuset *cs, struct cpuset *trialcs, return; prs_err = validate_partition(cs, trialcs); - if (prs_err) - trialcs->prs_err = cs->prs_err = prs_err; + if (prs_err) { + WRITE_ONCE(cs->prs_err, prs_err); + trialcs->prs_err = prs_err; + } if (is_remote_partition(cs)) { if (trialcs->prs_err) @@ -3039,7 +3109,11 @@ static int update_prstate(struct cpuset *cs, int new_prs) * A change in load balance state only, no change in cpumasks. * Need to update isolated_cpus. */ - isolcpus_updated = true; + if ((new_prs == PRS_ISOLATED) && + !isolated_cpus_can_update(cs->effective_xcpus, NULL)) + err = PERR_HKEEPING; + else + isolcpus_updated = true; } else { /* * Switching back to member is always allowed even if it @@ -3074,7 +3148,7 @@ static int update_prstate(struct cpuset *cs, int new_prs) else if (isolcpus_updated) isolated_cpus_update(old_prs, new_prs, cs->effective_xcpus); spin_unlock_irq(&callback_lock); - update_unbound_workqueue_cpumask(isolcpus_updated); + update_isolation_cpumasks(isolcpus_updated); /* Force update if switching back to member & update effective_xcpus */ update_cpumasks_hier(cs, &tmpmask, !new_prs); @@ -3313,8 +3387,8 @@ static void cpuset_attach(struct cgroup_taskset *tset) cs->old_mems_allowed = cpuset_attach_nodemask_to; if (cs->nr_migrate_dl_tasks) { - cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks; - oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks; + atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks); + atomic_sub(cs->nr_migrate_dl_tasks, &oldcs->nr_deadline_tasks); reset_migrate_dl_data(cs); } @@ -3942,7 +4016,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) if (remote && (cpumask_empty(subpartitions_cpus) || (cpumask_empty(&new_cpus) && partition_is_populated(cs, NULL)))) { - cs->prs_err = PERR_HOTPLUG; + WRITE_ONCE(cs->prs_err, PERR_HOTPLUG); remote_partition_disable(cs, tmp); compute_effective_cpumask(&new_cpus, cs, parent); remote = false; diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c index b0f0d15085db..7481fbb947d3 100644 --- a/kernel/cpu_pm.c +++ b/kernel/cpu_pm.c @@ -173,7 +173,7 @@ int cpu_cluster_pm_exit(void) EXPORT_SYMBOL_GPL(cpu_cluster_pm_exit); #ifdef CONFIG_PM -static int cpu_pm_suspend(void) +static int cpu_pm_suspend(void *data) { int ret; @@ -185,20 +185,24 @@ static int cpu_pm_suspend(void) return ret; } -static void cpu_pm_resume(void) +static void cpu_pm_resume(void *data) { cpu_cluster_pm_exit(); cpu_pm_exit(); } -static struct syscore_ops cpu_pm_syscore_ops = { +static const struct syscore_ops cpu_pm_syscore_ops = { .suspend = cpu_pm_suspend, .resume = cpu_pm_resume, }; +static struct syscore cpu_pm_syscore = { + .ops = &cpu_pm_syscore_ops, +}; + static int cpu_pm_init(void) { - register_syscore_ops(&cpu_pm_syscore_ops); + register_syscore(&cpu_pm_syscore); return 0; } core_initcall(cpu_pm_init); diff --git a/kernel/crash_dump_dm_crypt.c b/kernel/crash_dump_dm_crypt.c index 9501b0704f19..12e38922ca74 100644 --- a/kernel/crash_dump_dm_crypt.c +++ b/kernel/crash_dump_dm_crypt.c @@ -80,7 +80,6 @@ static int add_key_to_keyring(struct dm_crypt_key *dm_key, kexec_dprintk("Error when adding key"); } - key_ref_put(keyring_ref); return r; } @@ -103,6 +102,7 @@ static int restore_dm_crypt_keys_to_thread_keyring(void) struct dm_crypt_key *key; size_t keys_header_size; key_ref_t keyring_ref; + int ret = 0; u64 addr; /* find the target keyring (which must be writable) */ @@ -117,7 +117,8 @@ static int restore_dm_crypt_keys_to_thread_keyring(void) dm_crypt_keys_read((char *)&key_count, sizeof(key_count), &addr); if (key_count < 0 || key_count > KEY_NUM_MAX) { kexec_dprintk("Failed to read the number of dm-crypt keys\n"); - return -1; + ret = -1; + goto out; } kexec_dprintk("There are %u keys\n", key_count); @@ -125,8 +126,10 @@ static int restore_dm_crypt_keys_to_thread_keyring(void) keys_header_size = get_keys_header_size(key_count); keys_header = kzalloc(keys_header_size, GFP_KERNEL); - if (!keys_header) - return -ENOMEM; + if (!keys_header) { + ret = -ENOMEM; + goto out; + } dm_crypt_keys_read((char *)keys_header, keys_header_size, &addr); @@ -136,7 +139,9 @@ static int restore_dm_crypt_keys_to_thread_keyring(void) add_key_to_keyring(key, keyring_ref); } - return 0; +out: + key_ref_put(keyring_ref); + return ret; } static int read_key_from_user_keying(struct dm_crypt_key *dm_key) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index e27225f8aeb2..294543316622 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -268,9 +268,9 @@ void __init swiotlb_update_mem_attributes(void) } static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t start, - unsigned long nslabs, bool late_alloc, unsigned int nareas) + void *vaddr, unsigned long nslabs, bool late_alloc, + unsigned int nareas) { - void *vaddr = phys_to_virt(start); unsigned long bytes = nslabs << IO_TLB_SHIFT, i; mem->nslabs = nslabs; @@ -411,7 +411,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, return; } - swiotlb_init_io_tlb_pool(mem, __pa(tlb), nslabs, false, nareas); + swiotlb_init_io_tlb_pool(mem, __pa(tlb), tlb, nslabs, false, nareas); add_mem_pool(&io_tlb_default_mem, mem); if (flags & SWIOTLB_VERBOSE) @@ -509,7 +509,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), nslabs, true, + swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), vstart, nslabs, true, nareas); add_mem_pool(&io_tlb_default_mem, mem); @@ -607,25 +607,26 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limit) * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. * @gfp: GFP flags for the allocation. + * @vaddr: Receives the virtual address for the allocated buffer. * * Return: Allocated pages, or %NULL on allocation failure. */ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, - u64 phys_limit, gfp_t gfp) + u64 phys_limit, gfp_t gfp, void **vaddr) { struct page *page; + *vaddr = NULL; + /* * Allocate from the atomic pools if memory is encrypted and * the allocation is atomic, because decrypting may block. */ if (!gfpflags_allow_blocking(gfp) && dev && force_dma_unencrypted(dev)) { - void *vaddr; - if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; - return dma_alloc_from_pool(dev, bytes, &vaddr, gfp, + return dma_alloc_from_pool(dev, bytes, vaddr, gfp, dma_coherent_ok); } @@ -647,6 +648,8 @@ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, return NULL; } + if (page) + *vaddr = phys_to_virt(page_to_phys(page)); return page; } @@ -687,6 +690,7 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev, { struct io_tlb_pool *pool; unsigned int slot_order; + void *tlb_vaddr; struct page *tlb; size_t pool_size; size_t tlb_size; @@ -703,7 +707,8 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev, pool->areas = (void *)pool + sizeof(*pool); tlb_size = nslabs << IO_TLB_SHIFT; - while (!(tlb = swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp))) { + while (!(tlb = swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp, + &tlb_vaddr))) { if (nslabs <= minslabs) goto error_tlb; nslabs = ALIGN(nslabs >> 1, IO_TLB_SEGSIZE); @@ -717,11 +722,12 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev, if (!pool->slots) goto error_slots; - swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), nslabs, true, nareas); + swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), tlb_vaddr, nslabs, + true, nareas); return pool; error_slots: - swiotlb_free_tlb(page_address(tlb), tlb_size); + swiotlb_free_tlb(tlb_vaddr, tlb_size); error_tlb: kfree(pool); error: @@ -1849,7 +1855,8 @@ static int rmem_swiotlb_device_init(struct reserved_mem *rmem, set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_pool(pool, rmem->base, nslabs, + swiotlb_init_io_tlb_pool(pool, rmem->base, phys_to_virt(rmem->base), + nslabs, false, nareas); mem->force_bounce = true; mem->for_alloc = true; diff --git a/kernel/events/core.c b/kernel/events/core.c index 04c9d4915093..76d048a48827 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6737,6 +6737,7 @@ static void perf_mmap_open(struct vm_area_struct *vma) } static void perf_pmu_output_stop(struct perf_event *event); +static void perf_mmap_unaccount(struct vm_area_struct *vma, struct perf_buffer *rb); /* * A buffer can be mmap()ed multiple times; either directly through the same @@ -6752,9 +6753,6 @@ static void perf_mmap_close(struct vm_area_struct *vma) mapped_f unmapped = get_mapped(event, event_unmapped); struct perf_buffer *rb = ring_buffer_get(event); struct user_struct *mmap_user = rb->mmap_user; - int mmap_locked = rb->mmap_locked; - unsigned long size = perf_data_size(rb); - bool detach_rest = false; /* FIXIES vs perf_pmu_unregister() */ if (unmapped) @@ -6785,17 +6783,18 @@ static void perf_mmap_close(struct vm_area_struct *vma) mutex_unlock(&rb->aux_mutex); } - if (refcount_dec_and_test(&rb->mmap_count)) - detach_rest = true; - - if (!refcount_dec_and_mutex_lock(&event->mmap_count, &event->mmap_mutex)) - goto out_put; - - ring_buffer_attach(event, NULL); - mutex_unlock(&event->mmap_mutex); + /* + * Drop references in reverse order of perf_mmap() to prevent + * rb revival after rb->mmap_count reaches zero. + */ + if (refcount_dec_and_mutex_lock(&event->mmap_count, + &event->mmap_mutex)) { + ring_buffer_attach(event, NULL); + mutex_unlock(&event->mmap_mutex); + } /* If there's still other mmap()s of this buffer, we're done. */ - if (!detach_rest) + if (!refcount_dec_and_test(&rb->mmap_count)) goto out_put; /* @@ -6848,11 +6847,7 @@ static void perf_mmap_close(struct vm_area_struct *vma) * Aside from that, this buffer is 'fully' detached and unmapped, * undo the VM accounting. */ - - atomic_long_sub((size >> PAGE_SHIFT) + 1 - mmap_locked, - &mmap_user->locked_vm); - atomic64_sub(mmap_locked, &vma->vm_mm->pinned_vm); - free_uid(mmap_user); + perf_mmap_unaccount(vma, rb); out_put: ring_buffer_put(rb); /* could be last */ @@ -6994,6 +6989,15 @@ static void perf_mmap_account(struct vm_area_struct *vma, long user_extra, long atomic64_add(extra, &vma->vm_mm->pinned_vm); } +static void perf_mmap_unaccount(struct vm_area_struct *vma, struct perf_buffer *rb) +{ + struct user_struct *user = rb->mmap_user; + + atomic_long_sub((perf_data_size(rb) >> PAGE_SHIFT) + 1 - rb->mmap_locked, + &user->locked_vm); + atomic64_sub(rb->mmap_locked, &vma->vm_mm->pinned_vm); +} + static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event, unsigned long nr_pages) { @@ -7056,8 +7060,6 @@ static int perf_mmap_rb(struct vm_area_struct *vma, struct perf_event *event, if (!rb) return -ENOMEM; - refcount_set(&rb->mmap_count, 1); - rb->mmap_user = get_current_user(); rb->mmap_locked = extra; ring_buffer_attach(event, rb); @@ -7207,16 +7209,54 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) mapped(event, vma->vm_mm); /* - * Try to map it into the page table. On fail, invoke - * perf_mmap_close() to undo the above, as the callsite expects - * full cleanup in this case and therefore does not invoke - * vmops::close(). + * Try to map it into the page table. On fail undo the above, + * as the callsite expects full cleanup in this case and + * therefore does not invoke vmops::close(). */ ret = map_range(event->rb, vma); - if (ret) - perf_mmap_close(vma); + if (likely(!ret)) + return 0; + + /* Error path */ + + /* + * If this is the first mmap(), then event->mmap_count should + * be stable at 1. It is only modified by: + * perf_mmap_{open,close}() and perf_mmap(). + * + * The former are not possible because this mmap() hasn't been + * successful yet, and the latter is serialized by + * event->mmap_mutex which we still hold (note that mmap_lock + * is not strictly sufficient here, because the event fd can + * be passed to another process through trivial means like + * fork(), leading to concurrent mmap() from different mm). + * + * Make sure to remove event->rb before releasing + * event->mmap_mutex, such that any concurrent mmap() will not + * attempt use this failed buffer. + */ + if (refcount_read(&event->mmap_count) == 1) { + /* + * Minimal perf_mmap_close(); there can't be AUX or + * other events on account of this being the first. + */ + mapped = get_mapped(event, event_unmapped); + if (mapped) + mapped(event, vma->vm_mm); + perf_mmap_unaccount(vma, event->rb); + ring_buffer_attach(event, NULL); /* drops last rb->refcount */ + refcount_set(&event->mmap_count, 0); + return ret; + } + + /* + * Otherwise this is an already existing buffer, and there is + * no race vs first exposure, so fall-through and call + * perf_mmap_close(). + */ } + perf_mmap_close(vma); return ret; } diff --git a/kernel/events/internal.h b/kernel/events/internal.h index d9cc57083091..c03c4f2eea57 100644 --- a/kernel/events/internal.h +++ b/kernel/events/internal.h @@ -67,6 +67,7 @@ static inline void rb_free_rcu(struct rcu_head *rcu_head) struct perf_buffer *rb; rb = container_of(rcu_head, struct perf_buffer, rcu_head); + free_uid(rb->mmap_user); rb_free(rb); } diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c index 20a905023736..dc2aa2a7109e 100644 --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -340,6 +340,8 @@ ring_buffer_init(struct perf_buffer *rb, long watermark, int flags) rb->paused = 1; mutex_init(&rb->aux_mutex); + rb->mmap_user = get_current_user(); + refcount_set(&rb->mmap_count, 1); } void perf_aux_output_flag(struct perf_output_handle *handle, u64 flags) diff --git a/kernel/futex/core.c b/kernel/futex/core.c index 0ecbd1ad5fcd..69f340cddfd1 100644 --- a/kernel/futex/core.c +++ b/kernel/futex/core.c @@ -125,7 +125,7 @@ late_initcall(fail_futex_debugfs); #endif /* CONFIG_FAIL_FUTEX */ static struct futex_hash_bucket * -__futex_hash(union futex_key *key, struct futex_private_hash *fph); +__futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_private_hash **fph_p); #ifdef CONFIG_FUTEX_PRIVATE_HASH static bool futex_ref_get(struct futex_private_hash *fph); @@ -141,35 +141,14 @@ static bool futex_private_hash_get(struct futex_private_hash *fph) void futex_private_hash_put(struct futex_private_hash *fph) { - struct mm_struct *mm = fph->mm; - - if (futex_ref_put(fph)) - wake_up_var(mm); -} - -/** - * futex_hash_get - Get an additional reference for the local hash. - * @hb: ptr to the private local hash. - * - * Obtain an additional reference for the already obtained hash bucket. The - * caller must already own an reference. - */ -void futex_hash_get(struct futex_hash_bucket *hb) -{ - struct futex_private_hash *fph = hb->priv; + struct mm_struct *mm; if (!fph) return; - WARN_ON_ONCE(!futex_private_hash_get(fph)); -} -void futex_hash_put(struct futex_hash_bucket *hb) -{ - struct futex_private_hash *fph = hb->priv; - - if (!fph) - return; - futex_private_hash_put(fph); + mm = fph->mm; + if (futex_ref_put(fph)) + wake_up_var(mm); } static struct futex_hash_bucket * @@ -177,14 +156,6 @@ __futex_hash_private(union futex_key *key, struct futex_private_hash *fph) { u32 hash; - if (!futex_key_is_private(key)) - return NULL; - - if (!fph) - fph = rcu_dereference(key->private.mm->futex_phash); - if (!fph || !fph->hash_mask) - return NULL; - hash = jhash2((void *)&key->private.address, sizeof(key->private.address) / 4, key->both.offset); @@ -205,13 +176,12 @@ static void futex_rehash_private(struct futex_private_hash *old, spin_lock(&hb_old->lock); plist_for_each_entry_safe(this, tmp, &hb_old->chain, list) { - plist_del(&this->list, &hb_old->chain); futex_hb_waiters_dec(hb_old); WARN_ON_ONCE(this->lock_ptr != &hb_old->lock); - hb_new = __futex_hash(&this->key, new); + hb_new = __futex_hash(&this->key, new, NULL); futex_hb_waiters_inc(hb_new); /* * The new pointer isn't published yet but an already @@ -265,9 +235,8 @@ static void futex_pivot_hash(struct mm_struct *mm) } } -struct futex_private_hash *futex_private_hash(void) +struct futex_private_hash *futex_private_hash(struct mm_struct *mm) { - struct mm_struct *mm = current->mm; /* * Ideally we don't loop. If there is a replacement in progress * then a new private hash is already prepared and a reference can't be @@ -293,18 +262,17 @@ struct futex_private_hash *futex_private_hash(void) goto again; } -struct futex_hash_bucket *futex_hash(union futex_key *key) +struct futex_bucket_ref futex_hash(union futex_key *key) { - struct futex_private_hash *fph; - struct futex_hash_bucket *hb; - again: scoped_guard(rcu) { - hb = __futex_hash(key, NULL); - fph = hb->priv; + struct futex_private_hash *fph = NULL; + struct futex_hash_bucket *hb; + + hb = __futex_hash(key, NULL, &fph); if (!fph || futex_private_hash_get(fph)) - return hb; + return (struct futex_bucket_ref){ .hb = hb, .fph = fph }; } futex_pivot_hash(key->private.mm); goto again; @@ -312,15 +280,9 @@ struct futex_hash_bucket *futex_hash(union futex_key *key) #else /* !CONFIG_FUTEX_PRIVATE_HASH */ -static struct futex_hash_bucket * -__futex_hash_private(union futex_key *key, struct futex_private_hash *fph) -{ - return NULL; -} - -struct futex_hash_bucket *futex_hash(union futex_key *key) +struct futex_bucket_ref futex_hash(union futex_key *key) { - return __futex_hash(key, NULL); + return (struct futex_bucket_ref){ .hb = __futex_hash(key, NULL, NULL), .fph = NULL }; } #endif /* CONFIG_FUTEX_PRIVATE_HASH */ @@ -398,6 +360,8 @@ static int futex_mpol(struct mm_struct *mm, unsigned long addr) * __futex_hash - Return the hash bucket * @key: Pointer to the futex key for which the hash is calculated * @fph: Pointer to private hash if known + * @fph_p: Pointer to a private hash pointer; output for the private hash + * used when set. * * We hash on the keys returned from get_futex_key (see below) and return the * corresponding hash bucket. @@ -406,18 +370,22 @@ static int futex_mpol(struct mm_struct *mm, unsigned long addr) * global hash is returned. */ static struct futex_hash_bucket * -__futex_hash(union futex_key *key, struct futex_private_hash *fph) +__futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_private_hash **fph_p) { int node = key->both.node; u32 hash; - if (node == FUTEX_NO_NODE) { - struct futex_hash_bucket *hb; - - hb = __futex_hash_private(key, fph); - if (hb) - return hb; +#ifdef CONFIG_FUTEX_PRIVATE_HASH + if (node == FUTEX_NO_NODE && futex_key_is_private(key)) { + if (!fph) + fph = rcu_dereference(key->private.mm->futex_phash); + if (fph && fph->hash_mask) { + if (fph_p) + *fph_p = fph; + return __futex_hash_private(key, fph); + } } +#endif hash = jhash2((u32 *)key, offsetof(typeof(*key), both.offset) / sizeof(u32), @@ -1362,7 +1330,7 @@ static void exit_pi_state_list(struct task_struct *curr) * on the mutex. */ WARN_ON(curr != current); - guard(private_hash)(); + guard(private_hash)(current->mm); /* * We are a ZOMBIE and nobody can enqueue itself on * pi_state_list anymore, but we have to be careful @@ -1374,7 +1342,8 @@ static void exit_pi_state_list(struct task_struct *curr) pi_state = list_entry(next, struct futex_pi_state, list); key = pi_state->key; if (1) { - CLASS(hb, hb)(&key); + CLASS(hbr, hbr)(&key); + auto hb = hbr.hb; /* * We can race against put_pi_state() removing itself from the @@ -1556,12 +1525,8 @@ void futex_exec_done(struct task_struct *tsk) tsk->futex_state = FUTEX_STATE_OK; } -static void futex_hash_bucket_init(struct futex_hash_bucket *fhb, - struct futex_private_hash *fph) +static void futex_hash_bucket_init(struct futex_hash_bucket *fhb) { -#ifdef CONFIG_FUTEX_PRIVATE_HASH - fhb->priv = fph; -#endif atomic_set(&fhb->waiters, 0); plist_head_init(&fhb->chain); spin_lock_init(&fhb->lock); @@ -1866,7 +1831,7 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) fph->mm = mm; for (i = 0; i < hash_slots; i++) - futex_hash_bucket_init(&fph->queues[i], fph); + futex_hash_bucket_init(&fph->queues[i]); if (custom) { struct wait_bit_queue_entry __wbq_entry; @@ -2069,7 +2034,7 @@ static int __init futex_init(void) BUG_ON(!table); for (i = 0; i < hashsize; i++) - futex_hash_bucket_init(&table[i], NULL); + futex_hash_bucket_init(&table[i]); futex_queues[n] = table; } diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index d126365a36ca..17893fe1ce40 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -144,7 +144,6 @@ struct futex_hash_bucket { atomic_t waiters; spinlock_t lock; struct plist_head chain; - struct futex_private_hash *priv; } ____cacheline_aligned_in_smp; /* @@ -184,7 +183,7 @@ typedef void (futex_wake_fn)(struct wake_q_head *wake_q, struct futex_q *q); * @requeue_pi_key: the requeue_pi target futex key * @bitset: bitset for the optional bitmasked wakeup * @requeue_state: State field for futex_requeue_pi() - * @drop_hb_ref: Waiter should drop the extra hash bucket reference if true + * @drop_fph: Waiter should drop the extra private hash reference when set * @requeue_wait: RCU wait for futex_requeue_pi() (RT only) * * We use this hashed waitqueue, instead of a normal wait_queue_entry_t, so @@ -211,7 +210,7 @@ struct futex_q { union futex_key *requeue_pi_key; u32 bitset; atomic_t requeue_state; - bool drop_hb_ref; + struct futex_private_hash *drop_fph; #ifdef CONFIG_PREEMPT_RT struct rcuwait requeue_wait; #endif @@ -231,28 +230,29 @@ extern struct hrtimer_sleeper * futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout, int flags, u64 range_ns); -extern struct futex_hash_bucket *futex_hash(union futex_key *key); -#ifdef CONFIG_FUTEX_PRIVATE_HASH -extern void futex_hash_get(struct futex_hash_bucket *hb); -extern void futex_hash_put(struct futex_hash_bucket *hb); +struct futex_bucket_ref { + struct futex_hash_bucket *hb; + struct futex_private_hash *fph; +}; -extern struct futex_private_hash *futex_private_hash(void); +#ifdef CONFIG_FUTEX_PRIVATE_HASH +extern struct futex_private_hash *futex_private_hash(struct mm_struct *mm); extern void futex_private_hash_put(struct futex_private_hash *fph); #else /* !CONFIG_FUTEX_PRIVATE_HASH */ -static inline void futex_hash_get(struct futex_hash_bucket *hb) { } -static inline void futex_hash_put(struct futex_hash_bucket *hb) { } -static inline struct futex_private_hash *futex_private_hash(void) { return NULL; } +static inline struct futex_private_hash *futex_private_hash(struct mm_struct *mm) { return NULL; } static inline void futex_private_hash_put(struct futex_private_hash *fph) { } #endif -DEFINE_CLASS(hb, struct futex_hash_bucket *, - if (_T) futex_hash_put(_T), +extern struct futex_bucket_ref futex_hash(union futex_key *key); + +DEFINE_CLASS(hbr, struct futex_bucket_ref, + if (_T.fph) futex_private_hash_put(_T.fph), futex_hash(key), union futex_key *key); DEFINE_CLASS(private_hash, struct futex_private_hash *, if (_T) futex_private_hash_put(_T), - futex_private_hash(), void); + futex_private_hash(mm), struct mm_struct *mm); /** * futex_match - Check whether two futex keys are equal diff --git a/kernel/futex/pi.c b/kernel/futex/pi.c index 7110453848be..86f972c10fe8 100644 --- a/kernel/futex/pi.c +++ b/kernel/futex/pi.c @@ -1005,7 +1005,8 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl retry_private: if (1) { - CLASS(hb, hb)(&q.key); + CLASS(hbr, hbr)(&q.key); + auto hb = hbr.hb; futex_q_lock(&q, hb); @@ -1063,17 +1064,11 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl * Caution; releasing @hb in-scope. The hb->lock is still locked * while the reference is dropped. The reference can not be dropped * after the unlock because if a user initiated resize is in progress - * then we might need to wake him. This can not be done after the - * rt_mutex_pre_schedule() invocation. The hb will remain valid because - * the thread, performing resize, will block on hb->lock during - * the requeue. + * then we might need to wake him. The hb will remain valid + * because the thread, performing resize, will block on + * hb->lock during the requeue. */ - futex_hash_put(no_free_ptr(hb)); - /* - * Must be done before we enqueue the waiter, here is unfortunately - * under the hb lock, but that *should* work because it does nothing. - */ - rt_mutex_pre_schedule(); + futex_private_hash_put(no_free_ptr(hbr.fph)); rt_mutex_init_waiter(&rt_waiter); @@ -1139,10 +1134,6 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl * the */ futex_q_lockptr_lock(&q); - /* - * Waiter is unqueued. - */ - rt_mutex_post_schedule(); no_block: /* * Fixup the pi_state owner and possibly acquire the lock if we @@ -1158,11 +1149,9 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl futex_unqueue_pi(&q); spin_unlock(q.lock_ptr); - if (q.drop_hb_ref) { - CLASS(hb, hb)(&q.key); - /* Additional reference from futex_unlock_pi() */ - futex_hash_put(hb); - } + + /* Additional reference from futex_unlock_pi() */ + futex_private_hash_put(q.drop_fph); goto out; out_unlock_put_key: @@ -1218,7 +1207,8 @@ int futex_unlock_pi(u32 __user *uaddr, unsigned int flags) if (ret) return ret; - CLASS(hb, hb)(&key); + CLASS(hbr, hbr)(&key); + auto hb = hbr.hb; spin_lock(&hb->lock); retry_hb: @@ -1275,8 +1265,9 @@ int futex_unlock_pi(u32 __user *uaddr, unsigned int flags) * Acquire a reference for the leaving waiter to ensure * valid futex_q::lock_ptr. */ - futex_hash_get(hb); - top_waiter->drop_hb_ref = true; + if (futex_key_is_private(&key)) + top_waiter->drop_fph = futex_private_hash(key.private.mm); + __futex_unqueue(top_waiter); raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock); goto retry_hb; diff --git a/kernel/futex/requeue.c b/kernel/futex/requeue.c index 21df23df8997..b3f4a4bccb12 100644 --- a/kernel/futex/requeue.c +++ b/kernel/futex/requeue.c @@ -249,8 +249,8 @@ void requeue_pi_wake_futex(struct futex_q *q, union futex_key *key, * Acquire a reference for the waiter to ensure valid * futex_q::lock_ptr. */ - futex_hash_get(hb); - q->drop_hb_ref = true; + if (futex_key_is_private(key)) + q->drop_fph = futex_private_hash(key->private.mm); q->lock_ptr = &hb->lock; task = READ_ONCE(q->task); @@ -467,8 +467,10 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags1, retry_private: if (1) { - CLASS(hb, hb1)(&key1); - CLASS(hb, hb2)(&key2); + CLASS(hbr, hbr1)(&key1); + CLASS(hbr, hbr2)(&key2); + auto hb1 = hbr1.hb; + auto hb2 = hbr2.hb; futex_hb_waiters_inc(hb2); double_lock_hb(hb1, hb2); @@ -840,7 +842,8 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags, switch (futex_requeue_pi_wakeup_sync(&q)) { case Q_REQUEUE_PI_IGNORE: { - CLASS(hb, hb)(&q.key); + CLASS(hbr, hbr)(&q.key); + auto hb = hbr.hb; /* The waiter is still on uaddr1 */ spin_lock(&hb->lock); ret = handle_early_requeue_pi_wakeup(hb, &q, to); @@ -910,11 +913,8 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags, default: BUG(); } - if (q.drop_hb_ref) { - CLASS(hb, hb)(&q.key); - /* Additional reference from requeue_pi_wake_futex() */ - futex_hash_put(hb); - } + /* Additional reference from requeue_pi_wake_futex() */ + futex_private_hash_put(q.drop_fph); out: if (to) { diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index e2bbe5509ec2..67eeaabbeb3d 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -169,7 +169,8 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset) if ((flags & FLAGS_STRICT) && !nr_wake) return 0; - CLASS(hb, hb)(&key); + CLASS(hbr, hbr)(&key); + auto hb = hbr.hb; /* Make sure we really have tasks to wakeup */ if (!futex_hb_waiters_pending(hb)) @@ -266,8 +267,10 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, retry_private: if (1) { - CLASS(hb, hb1)(&key1); - CLASS(hb, hb2)(&key2); + CLASS(hbr, hbr1)(&key1); + CLASS(hbr, hbr2)(&key2); + auto hb1 = hbr1.hb; + auto hb2 = hbr2.hb; double_lock_hb(hb1, hb2); op_ret = futex_atomic_op_inuser(op, uaddr2); @@ -409,7 +412,7 @@ int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *woken) * Make sure to have a reference on the private_hash such that we * don't block on rehash after changing the task state below. */ - guard(private_hash)(); + guard(private_hash)(current->mm); /* * Enqueuing multiple futexes is tricky, because we need to enqueue @@ -446,7 +449,8 @@ int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *woken) u32 val = vs[i].w.val; if (1) { - CLASS(hb, hb)(&q->key); + CLASS(hbr, hbr)(&q->key); + auto hb = hbr.hb; futex_q_lock(q, hb); ret = futex_get_value_locked(&uval, uaddr); @@ -620,7 +624,8 @@ int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags, retry_private: if (1) { - CLASS(hb, hb)(&q->key); + CLASS(hbr, hbr)(&q->key); + auto hb = hbr.hb; futex_q_lock(q, hb); diff --git a/kernel/irq/generic-chip.c b/kernel/irq/generic-chip.c index bf59e37d650a..3cd0c40282c0 100644 --- a/kernel/irq/generic-chip.c +++ b/kernel/irq/generic-chip.c @@ -650,7 +650,7 @@ static struct irq_data *irq_gc_get_irq_data(struct irq_chip_generic *gc) } #ifdef CONFIG_PM -static int irq_gc_suspend(void) +static int irq_gc_suspend(void *data) { struct irq_chip_generic *gc; @@ -670,7 +670,7 @@ static int irq_gc_suspend(void) return 0; } -static void irq_gc_resume(void) +static void irq_gc_resume(void *data) { struct irq_chip_generic *gc; @@ -693,7 +693,7 @@ static void irq_gc_resume(void) #define irq_gc_resume NULL #endif -static void irq_gc_shutdown(void) +static void irq_gc_shutdown(void *data) { struct irq_chip_generic *gc; @@ -709,15 +709,19 @@ static void irq_gc_shutdown(void) } } -static struct syscore_ops irq_gc_syscore_ops = { +static const struct syscore_ops irq_gc_syscore_ops = { .suspend = irq_gc_suspend, .resume = irq_gc_resume, .shutdown = irq_gc_shutdown, }; +static struct syscore irq_gc_syscore = { + .ops = &irq_gc_syscore_ops, +}; + static int __init irq_gc_init_ops(void) { - register_syscore_ops(&irq_gc_syscore_ops); + register_syscore(&irq_gc_syscore); return 0; } device_initcall(irq_gc_init_ops); diff --git a/kernel/irq/pm.c b/kernel/irq/pm.c index f7394729cedc..99ff65466d87 100644 --- a/kernel/irq/pm.c +++ b/kernel/irq/pm.c @@ -211,21 +211,26 @@ void rearm_wake_irq(unsigned int irq) /** * irq_pm_syscore_resume - enable interrupt lines early + * @data: syscore context * * Enable all interrupt lines with %IRQF_EARLY_RESUME set. */ -static void irq_pm_syscore_resume(void) +static void irq_pm_syscore_resume(void *data) { resume_irqs(true); } -static struct syscore_ops irq_pm_syscore_ops = { +static const struct syscore_ops irq_pm_syscore_ops = { .resume = irq_pm_syscore_resume, }; +static struct syscore irq_pm_syscore = { + .ops = &irq_pm_syscore_ops, +}; + static int __init irq_pm_init_ops(void) { - register_syscore_ops(&irq_pm_syscore_ops); + register_syscore(&irq_pm_syscore); return 0; } diff --git a/kernel/kcsan/core.c b/kernel/kcsan/core.c index 8a7baf4e332e..2db82661cd60 100644 --- a/kernel/kcsan/core.c +++ b/kernel/kcsan/core.c @@ -585,8 +585,14 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned * information is lost if dirtied by KCSAN. */ kcsan_save_irqtrace(current); - if (!interrupt_watcher) + if (!interrupt_watcher) { local_irq_save(irq_flags); + /* + * NMIs can still fire, disable checking for all interrupt + * contexts. + */ + raw_cpu_ptr(&kcsan_cpu_ctx)->disable_count++; + } watchpoint = insert_watchpoint((unsigned long)ptr, size, is_write); if (watchpoint == NULL) { @@ -699,8 +705,10 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned atomic_long_dec(&kcsan_counters[KCSAN_COUNTER_USED_WATCHPOINTS]); out_unlock: - if (!interrupt_watcher) + if (!interrupt_watcher) { + raw_cpu_ptr(&kcsan_cpu_ctx)->disable_count--; local_irq_restore(irq_flags); + } kcsan_restore_irqtrace(current); ctx->disable_scoped--; diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 2d4c5bab5af8..e0de81114824 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -5437,6 +5437,8 @@ __lock_set_class(struct lockdep_map *lock, const char *name, lock->wait_type_outer, lock->lock_type); class = register_lock_class(lock, subclass, 0); + if (!class) + return 0; hlock->class_idx = class - lock_classes; curr->lockdep_depth = i; diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c index 73af81b99bf6..928860be7a4a 100644 --- a/kernel/locking/rtmutex_api.c +++ b/kernel/locking/rtmutex_api.c @@ -398,6 +398,7 @@ int __sched rt_mutex_wait_proxy_lock(struct rt_mutex_base *lock, { int ret; + rt_mutex_futex_pre_schedule(); raw_spin_lock_irq(&lock->wait_lock); /* sleep on the mutex */ set_current_state(TASK_INTERRUPTIBLE); @@ -408,6 +409,7 @@ int __sched rt_mutex_wait_proxy_lock(struct rt_mutex_base *lock, */ fixup_rt_mutex_waiters(lock, true); raw_spin_unlock_irq(&lock->wait_lock); + rt_mutex_futex_post_schedule(); return ret; } diff --git a/kernel/module/dups.c b/kernel/module/dups.c index bd2149fbe117..c21b675c239b 100644 --- a/kernel/module/dups.c +++ b/kernel/module/dups.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "internal.h" @@ -38,27 +39,37 @@ static bool enable_dups_trace = IS_ENABLED(CONFIG_MODULE_DEBUG_AUTOLOAD_DUPS_TRACE); module_param(enable_dups_trace, bool_enable_only, 0644); -/* - * Protects dup_kmod_reqs list, adds / removals with RCU. - */ +/* A mutex-protected list of active kmod requests. */ static DEFINE_MUTEX(kmod_dup_mutex); static LIST_HEAD(dup_kmod_reqs); struct kmod_dup_req { + refcount_t refcount; struct list_head list; char name[MODULE_NAME_LEN]; struct completion first_req_done; - struct work_struct complete_work; struct delayed_work delete_work; int dup_ret; }; +static void get_kmod_req(struct kmod_dup_req *kmod_req) +{ + refcount_inc(&kmod_req->refcount); +} + +static void put_kmod_req(struct kmod_dup_req *kmod_req) +{ + if (refcount_dec_and_test(&kmod_req->refcount)) + kfree(kmod_req); +} + static struct kmod_dup_req *kmod_dup_request_lookup(char *module_name) { struct kmod_dup_req *kmod_req; - list_for_each_entry_rcu(kmod_req, &dup_kmod_reqs, list, - lockdep_is_held(&kmod_dup_mutex)) { + lockdep_assert_held(&kmod_dup_mutex); + + list_for_each_entry(kmod_req, &dup_kmod_reqs, list) { if (strlen(kmod_req->name) == strlen(module_name) && !memcmp(kmod_req->name, module_name, strlen(module_name))) { return kmod_req; @@ -87,33 +98,10 @@ static void kmod_dup_request_delete(struct work_struct *work) * just returning 0. */ mutex_lock(&kmod_dup_mutex); - list_del_rcu(&kmod_req->list); - synchronize_rcu(); + list_del(&kmod_req->list); mutex_unlock(&kmod_dup_mutex); - kfree(kmod_req); -} -static void kmod_dup_request_complete(struct work_struct *work) -{ - struct kmod_dup_req *kmod_req; - - kmod_req = container_of(work, struct kmod_dup_req, complete_work); - - /* - * This will ensure that the kernel will let all the waiters get - * informed its time to check the return value. It's time to - * go home. - */ - complete_all(&kmod_req->first_req_done); - - /* - * Now that we have allowed prior request_module() calls to go on - * with life, let's schedule deleting this entry. We don't have - * to do it right away, but we *eventually* want to do it so to not - * let this linger forever as this is just a boot optimization for - * possible abuses of vmalloc() incurred by finit_module() thrashing. - */ - queue_delayed_work(system_wq, &kmod_req->delete_work, 60 * HZ); + put_kmod_req(kmod_req); } bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) @@ -129,8 +117,8 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) if (!new_kmod_req) return false; - memcpy(new_kmod_req->name, module_name, strlen(module_name)); - INIT_WORK(&new_kmod_req->complete_work, kmod_dup_request_complete); + refcount_set(&new_kmod_req->refcount, 1); + strscpy(new_kmod_req->name, module_name); INIT_DELAYED_WORK(&new_kmod_req->delete_work, kmod_dup_request_delete); init_completion(&new_kmod_req->first_req_done); @@ -161,10 +149,12 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) * keep tab on duplicates later. */ pr_debug("New request_module() for %s\n", module_name); - list_add_rcu(&new_kmod_req->list, &dup_kmod_reqs); + list_add(&new_kmod_req->list, &dup_kmod_reqs); mutex_unlock(&kmod_dup_mutex); return false; } + + get_kmod_req(kmod_req); mutex_unlock(&kmod_dup_mutex); /* We are dealing with a duplicate request now */ @@ -194,7 +184,7 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) * calls bail out right away. */ *dup_ret = 0; - return true; + goto out; } /* @@ -209,12 +199,14 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) TASK_KILLABLE); if (ret) { *dup_ret = ret; - return true; + goto out; } /* Now the duplicate request has the same exact return value as the first request */ *dup_ret = kmod_req->dup_ret; +out: + put_kmod_req(kmod_req); return true; } @@ -224,24 +216,31 @@ void kmod_dup_request_announce(char *module_name, int ret) mutex_lock(&kmod_dup_mutex); + /* + * Look for a kmod_dup_req previously added in + * kmod_dup_request_exists_wait(). Note that a request_module_nowait() + * without its own kmod_dup_req entry can announce a result of + * a concurrent request_module() call. + */ kmod_req = kmod_dup_request_lookup(module_name); - if (!kmod_req) - goto out; + if (!kmod_req || completion_done(&kmod_req->first_req_done)) { + mutex_unlock(&kmod_dup_mutex); + return; + } kmod_req->dup_ret = ret; - /* - * If we complete() here we may allow duplicate threads - * to continue before the first one that submitted the - * request. We're in no rush also, given that each and - * every bounce back to userspace is slow we avoid that - * with a slight delay here. So queueue up the completion - * and let duplicates suffer, just wait a tad bit longer. - * There is no rush. But we also don't want to hold the - * caller up forever or introduce any boot delays. - */ - queue_work(system_wq, &kmod_req->complete_work); + /* Inform all duplicate waiters to check the return value. */ + complete_all(&kmod_req->first_req_done); -out: mutex_unlock(&kmod_dup_mutex); + + /* + * Now that we have allowed prior request_module() calls to go on + * with life, let's schedule deleting this entry. We don't have + * to do it right away, but we *eventually* want to do it so to not + * let this linger forever as this is just a boot optimization for + * possible abuses of vmalloc() incurred by finit_module() thrashing. + */ + queue_delayed_work(system_dfl_wq, &kmod_req->delete_work, 60 * HZ); } diff --git a/kernel/module/stats.c b/kernel/module/stats.c index 3ba0e98b3c91..2a4e2f670896 100644 --- a/kernel/module/stats.c +++ b/kernel/module/stats.c @@ -253,7 +253,7 @@ int try_add_failed_module(const char *name, enum fail_dup_mod_reason reason) mod_fail = kzalloc(sizeof(*mod_fail), GFP_KERNEL); if (!mod_fail) return -ENOMEM; - memcpy(mod_fail->name, name, strlen(name)); + strscpy(mod_fail->name, name); __set_bit(reason, &mod_fail->dup_fail_mask); atomic_long_inc(&mod_fail->count); list_add_rcu(&mod_fail->list, &dup_failed_modules); diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index e249e5786fbc..b691e2a047b5 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -2798,9 +2798,10 @@ int snapshot_write_next(struct snapshot_handle *handle) return error; error = memory_bm_create(&zero_bm, GFP_ATOMIC, PG_ANY); - if (error) + if (error) { + memory_bm_free(©_bm, PG_UNSAFE_CLEAR); return error; - + } nr_zero_pages = 0; hibernate_restore_protection_begin(); diff --git a/kernel/power/wakelock.c b/kernel/power/wakelock.c index 4e941999a53b..5c2a248cf249 100644 --- a/kernel/power/wakelock.c +++ b/kernel/power/wakelock.c @@ -63,7 +63,7 @@ static unsigned int number_of_wakelocks; static inline bool wakelocks_limit_exceeded(void) { - return number_of_wakelocks > CONFIG_PM_WAKELOCKS_LIMIT; + return number_of_wakelocks >= CONFIG_PM_WAKELOCKS_LIMIT; } static inline void increment_wakelocks_number(void) diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index c27fc7fc64eb..cc3d57be447d 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -3155,6 +3155,106 @@ static inline void printk_kthreads_check_locked(void) { } #endif /* CONFIG_PRINTK */ + +/* + * Print out one record for each console. + * + * @do_cond_resched is set by the caller. It can be true only in schedulable + * context. + * + * @next_seq is set to the sequence number after the last available record. + * The value is valid only when there is at least one usable console and all + * usable consoles were flushed. + * + * @handover will be set to true if a printk waiter has taken over the + * console_lock, in which case the caller is no longer holding the + * console_lock. Otherwise it is set to false. + * + * @any_usable will be set to true if there are any usable consoles. + * + * Returns true when there was at least one usable console and a record was + * flushed. A returned false indicates there were no records to flush for any + * of the consoles. It may also indicate that there were no usable consoles, + * the context has been lost or there is a panic suitation. Regardless the + * reason, the caller should assume it is not useful to immediately try again. + * + * Requires the console_lock. + */ +static bool console_flush_one_record(bool do_cond_resched, u64 *next_seq, bool *handover, + bool *any_usable) +{ + struct console_flush_type ft; + bool any_progress = false; + struct console *con; + int cookie; + + printk_get_console_flush_type(&ft); + + cookie = console_srcu_read_lock(); + for_each_console_srcu(con) { + short flags = console_srcu_read_flags(con); + u64 printk_seq; + bool progress; + + /* + * console_flush_one_record() is only responsible for + * nbcon consoles when the nbcon consoles cannot print via + * their atomic or threaded flushing. + */ + if ((flags & CON_NBCON) && (ft.nbcon_atomic || ft.nbcon_offload)) + continue; + + if (!console_is_usable(con, flags, !do_cond_resched)) + continue; + *any_usable = true; + + if (flags & CON_NBCON) { + progress = nbcon_legacy_emit_next_record(con, handover, cookie, + !do_cond_resched); + } else { + progress = console_emit_next_record(con, handover, cookie); + } + + /* + * If a handover has occurred, the SRCU read lock + * is already released. + */ + if (*handover) + return false; + + /* + * @con can be used here now that it is certain that this + * context is still holding the SRCU read lock. + */ + if (flags & CON_NBCON) + printk_seq = nbcon_seq_read(con); + else + printk_seq = con->seq; + + /* Track the next of the highest seq flushed. */ + if (printk_seq > *next_seq) + *next_seq = printk_seq; + + if (!progress) + continue; + any_progress = true; + + /* Allow panic_cpu to take over the consoles safely. */ + if (panic_on_other_cpu()) + goto abandon; + + if (do_cond_resched) + cond_resched(); + } + console_srcu_read_unlock(cookie); + + return any_progress; + +abandon: + console_srcu_read_unlock(cookie); + return false; +} + /* * Print out all remaining records to all consoles. * @@ -3180,77 +3280,24 @@ static inline void printk_kthreads_check_locked(void) { } */ static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handover) { - struct console_flush_type ft; bool any_usable = false; - struct console *con; bool any_progress; - int cookie; *next_seq = 0; *handover = false; do { - any_progress = false; - - printk_get_console_flush_type(&ft); - - cookie = console_srcu_read_lock(); - for_each_console_srcu(con) { - short flags = console_srcu_read_flags(con); - u64 printk_seq; - bool progress; - - /* - * console_flush_all() is only responsible for nbcon - * consoles when the nbcon consoles cannot print via - * their atomic or threaded flushing. - */ - if ((flags & CON_NBCON) && (ft.nbcon_atomic || ft.nbcon_offload)) - continue; - - if (!console_is_usable(con, flags, !do_cond_resched)) - continue; - any_usable = true; - - if (flags & CON_NBCON) { - progress = nbcon_legacy_emit_next_record(con, handover, cookie, - !do_cond_resched); - printk_seq = nbcon_seq_read(con); - } else { - progress = console_emit_next_record(con, handover, cookie); - printk_seq = con->seq; - } - - /* - * If a handover has occurred, the SRCU read lock - * is already released. - */ - if (*handover) - return false; + any_progress = console_flush_one_record(do_cond_resched, next_seq, handover, + &any_usable); - /* Track the next of the highest seq flushed. */ - if (printk_seq > *next_seq) - *next_seq = printk_seq; + if (*handover) + return false; - if (!progress) - continue; - any_progress = true; - - /* Allow panic_cpu to take over the consoles safely. */ - if (panic_on_other_cpu()) - goto abandon; - - if (do_cond_resched) - cond_resched(); - } - console_srcu_read_unlock(cookie); + if (panic_on_other_cpu()) + return false; } while (any_progress); return any_usable; - -abandon: - console_srcu_read_unlock(cookie); - return false; } static void __console_flush_and_unlock(void) @@ -3660,12 +3707,13 @@ static bool legacy_kthread_create(void) /** * printk_kthreads_shutdown - shutdown all threaded printers + * @data: syscore context * * On system shutdown all threaded printers are stopped. This allows printk * to transition back to atomic printing, thus providing a robust mechanism * for the final shutdown/reboot messages to be output. */ -static void printk_kthreads_shutdown(void) +static void printk_kthreads_shutdown(void *data) { struct console *con; @@ -3687,10 +3735,14 @@ static void printk_kthreads_shutdown(void) console_list_unlock(); } -static struct syscore_ops printk_syscore_ops = { +static const struct syscore_ops printk_syscore_ops = { .shutdown = printk_kthreads_shutdown, }; +static struct syscore printk_syscore = { + .ops = &printk_syscore_ops, +}; + /* * If appropriate, start nbcon kthreads and set @printk_kthreads_running. * If any kthreads fail to start, those consoles are unregistered. @@ -3758,7 +3810,7 @@ static void printk_kthreads_check_locked(void) static int __init printk_set_kthreads_ready(void) { - register_syscore_ops(&printk_syscore_ops); + register_syscore(&printk_syscore); console_list_lock(); printk_kthreads_ready = true; diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index cafb1cc8eff8..41e32b015a64 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -973,7 +973,7 @@ void rcu_all_qs(void) { unsigned long flags; - if (!raw_cpu_read(rcu_data.rcu_urgent_qs)) + if (!READ_ONCE(*raw_cpu_ptr(&rcu_data.rcu_urgent_qs))) return; preempt_disable(); // For CONFIG_PREEMPT_COUNT=y kernels /* Load rcu_urgent_qs before other flags. */ @@ -981,8 +981,8 @@ void rcu_all_qs(void) preempt_enable(); return; } - this_cpu_write(rcu_data.rcu_urgent_qs, false); - if (unlikely(raw_cpu_read(rcu_data.rcu_need_heavy_qs))) { + WRITE_ONCE(*this_cpu_ptr(&rcu_data.rcu_urgent_qs), false); + if (unlikely(READ_ONCE(*this_cpu_ptr(&rcu_data.rcu_need_heavy_qs)))) { local_irq_save(flags); rcu_momentary_eqs(); local_irq_restore(flags); @@ -1002,8 +1002,8 @@ void rcu_note_context_switch(bool preempt) /* Load rcu_urgent_qs before other flags. */ if (!smp_load_acquire(this_cpu_ptr(&rcu_data.rcu_urgent_qs))) goto out; - this_cpu_write(rcu_data.rcu_urgent_qs, false); - if (unlikely(raw_cpu_read(rcu_data.rcu_need_heavy_qs))) + WRITE_ONCE(*this_cpu_ptr(&rcu_data.rcu_urgent_qs), false); + if (unlikely(READ_ONCE(*this_cpu_ptr(&rcu_data.rcu_need_heavy_qs)))) rcu_momentary_eqs(); out: rcu_tasks_qs(current, preempt); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 08ac7da783a6..560f735cb750 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7333,6 +7333,17 @@ void rt_mutex_pre_schedule(void) sched_submit_work(current); } +/* + * Used within the futex syscall context, skips sched_submit_work() because none + * its work will be done. Asserts ensure that it is indeed the case. + */ +void rt_mutex_futex_pre_schedule(void) +{ + lockdep_assert(!(current->flags & (PF_WQ_WORKER | PF_IO_WORKER))); + lockdep_assert(!current->plug); + lockdep_assert(!fetch_and_set(current->sched_rt_mutex, 1)); +} + void rt_mutex_schedule(void) { lockdep_assert(current->sched_rt_mutex); @@ -7345,6 +7356,11 @@ void rt_mutex_post_schedule(void) lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0)); } +void rt_mutex_futex_post_schedule(void) +{ + lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0)); +} + /* * rt_mutex_setprio - set the current priority of a task * @p: task to boost diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c index 374b65940d03..22a1fbccb929 100644 --- a/kernel/sched/cpufreq_schedutil.c +++ b/kernel/sched/cpufreq_schedutil.c @@ -325,7 +325,7 @@ static void sugov_iowait_boost(struct sugov_cpu *sg_cpu, u64 time, * A CPU running a task which woken up after an IO operation can have its * utilization boosted to speed up the completion of those IO operations. * The IO boost value is increased each time a task wakes up from IO, in - * sugov_iowait_apply(), and it's instead decreased by this function, + * sugov_iowait_boost(), and it's instead decreased by this function, * each time an increase has not been requested (!iowait_boost_pending). * * A CPU which also appears to have been idle for at least one tick has also diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index eb4d5e558b6e..4f02f823d104 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4463,7 +4463,8 @@ static inline void update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cfs_rq *gcfs_rq) { long delta_sum, delta_avg = gcfs_rq->avg.runnable_avg - se->avg.runnable_avg; - u32 new_sum, divider; + u64 new_sum; + u32 divider; /* Nothing to update */ if (!delta_avg) @@ -4477,7 +4478,7 @@ update_tg_cfs_runnable(struct cfs_rq *cfs_rq, struct sched_entity *se, struct cf /* Set new sched_entity's runnable */ se->avg.runnable_avg = gcfs_rq->avg.runnable_avg; - new_sum = se->avg.runnable_avg * divider; + new_sum = (u64)se->avg.runnable_avg * divider; delta_sum = (long)new_sum - (long)se->avg.runnable_sum; se->avg.runnable_sum = new_sum; @@ -10679,7 +10680,9 @@ static inline void update_sg_lb_stats(struct lb_env *env, /* Check for a misfit task on the cpu */ if (sgs->group_misfit_task_load < rq->misfit_task_load) { sgs->group_misfit_task_load = rq->misfit_task_load; - *sg_overloaded = 1; + + if (balancing_at_rd) + *sg_overloaded = 1; } } else if (env->idle && sched_reduced_capacity(rq, env->sd)) { /* Check for a task running on a CPU with reduced capacity */ @@ -10745,6 +10748,17 @@ static bool update_sd_pick_busiest(struct lb_env *env, sds->local_stat.group_type != group_has_spare)) return false; + /* + * Candidate sg has no more than one task per CPU and has higher + * per-CPU capacity. Migrating tasks to less capable CPUs may harm + * throughput. Maximize throughput, power/energy consequences are not + * considered. + */ + if ((env->sd->flags & SD_ASYM_CPUCAPACITY) && + (sgs->group_type <= group_fully_busy) && + (capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu)))) + return false; + if (sgs->group_type > busiest->group_type) return true; @@ -10847,17 +10861,6 @@ static bool update_sd_pick_busiest(struct lb_env *env, break; } - /* - * Candidate sg has no more than one task per CPU and has higher - * per-CPU capacity. Migrating tasks to less capable CPUs may harm - * throughput. Maximize throughput, power/energy consequences are not - * considered. - */ - if ((env->sd->flags & SD_ASYM_CPUCAPACITY) && - (sgs->group_type <= group_fully_busy) && - (capacity_greater(sg->sgc->min_capacity, capacity_of(env->dst_cpu)))) - return false; - return true; } diff --git a/kernel/sys.c b/kernel/sys.c index 8b58eece4e58..c8376d7a0741 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2189,7 +2189,7 @@ static int prctl_set_auxv(struct mm_struct *mm, unsigned long addr, BUILD_BUG_ON(sizeof(user_auxv) != sizeof(mm->saved_auxv)); task_lock(current); - memcpy(mm->saved_auxv, user_auxv, len); + memcpy(mm->saved_auxv, user_auxv, sizeof(user_auxv)); task_unlock(current); return 0; diff --git a/kernel/time/Makefile b/kernel/time/Makefile index f7d52d9543cc..662bccb3b7f9 100644 --- a/kernel/time/Makefile +++ b/kernel/time/Makefile @@ -29,6 +29,6 @@ endif obj-$(CONFIG_GENERIC_GETTIMEOFDAY) += vsyscall.o obj-$(CONFIG_DEBUG_FS) += timekeeping_debug.o obj-$(CONFIG_TEST_UDELAY) += test_udelay.o -obj-$(CONFIG_TIME_NS) += namespace.o +obj-$(CONFIG_TIME_NS) += namespace.o namespace_vdso.o obj-$(CONFIG_TEST_CLOCKSOURCE_WATCHDOG) += clocksource-wdtest.o obj-$(CONFIG_TIME_KUNIT_TEST) += time_test.o diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index df7194961658..c2ce6cea2410 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -1482,8 +1482,12 @@ static int __init init_clocksource_sysfs(void) { int error = subsys_system_register(&clocksource_subsys, NULL); - if (!error) - error = device_register(&device_clocksource); + if (error) + return error; + + error = device_register(&device_clocksource); + if (error) + bus_unregister(&clocksource_subsys); return error; } diff --git a/kernel/time/namespace.c b/kernel/time/namespace.c index 5b6997f4dc3d..51d2fcba7b5f 100644 --- a/kernel/time/namespace.c +++ b/kernel/time/namespace.c @@ -18,8 +18,9 @@ #include #include #include +#include -#include +#include "namespace_internal.h" ktime_t do_timens_ktime_to_host(clockid_t clockid, ktime_t tim, struct timens_offsets *ns_offsets) @@ -138,117 +139,7 @@ struct time_namespace *copy_time_ns(u64 flags, return clone_time_ns(user_ns, old_ns); } -static struct timens_offset offset_from_ts(struct timespec64 off) -{ - struct timens_offset ret; - - ret.sec = off.tv_sec; - ret.nsec = off.tv_nsec; - - return ret; -} - -/* - * A time namespace VVAR page has the same layout as the VVAR page which - * contains the system wide VDSO data. - * - * For a normal task the VVAR pages are installed in the normal ordering: - * VVAR - * PVCLOCK - * HVCLOCK - * TIMENS <- Not really required - * - * Now for a timens task the pages are installed in the following order: - * TIMENS - * PVCLOCK - * HVCLOCK - * VVAR - * - * The check for vdso_clock->clock_mode is in the unlikely path of - * the seq begin magic. So for the non-timens case most of the time - * 'seq' is even, so the branch is not taken. - * - * If 'seq' is odd, i.e. a concurrent update is in progress, the extra check - * for vdso_clock->clock_mode is a non-issue. The task is spin waiting for the - * update to finish and for 'seq' to become even anyway. - * - * Timens page has vdso_clock->clock_mode set to VDSO_CLOCKMODE_TIMENS which - * enforces the time namespace handling path. - */ -static void timens_setup_vdso_clock_data(struct vdso_clock *vc, - struct time_namespace *ns) -{ - struct timens_offset *offset = vc->offset; - struct timens_offset monotonic = offset_from_ts(ns->offsets.monotonic); - struct timens_offset boottime = offset_from_ts(ns->offsets.boottime); - - vc->seq = 1; - vc->clock_mode = VDSO_CLOCKMODE_TIMENS; - offset[CLOCK_MONOTONIC] = monotonic; - offset[CLOCK_MONOTONIC_RAW] = monotonic; - offset[CLOCK_MONOTONIC_COARSE] = monotonic; - offset[CLOCK_BOOTTIME] = boottime; - offset[CLOCK_BOOTTIME_ALARM] = boottime; -} - -struct page *find_timens_vvar_page(struct vm_area_struct *vma) -{ - if (likely(vma->vm_mm == current->mm)) - return current->nsproxy->time_ns->vvar_page; - - /* - * VM_PFNMAP | VM_IO protect .fault() handler from being called - * through interfaces like /proc/$pid/mem or - * process_vm_{readv,writev}() as long as there's no .access() - * in special_mapping_vmops(). - * For more details check_vma_flags() and __access_remote_vm() - */ - - WARN(1, "vvar_page accessed remotely"); - - return NULL; -} - -/* - * Protects possibly multiple offsets writers racing each other - * and tasks entering the namespace. - */ -static DEFINE_MUTEX(offset_lock); - -static void timens_set_vvar_page(struct task_struct *task, - struct time_namespace *ns) -{ - struct vdso_time_data *vdata; - struct vdso_clock *vc; - unsigned int i; - - if (ns == &init_time_ns) - return; - - /* Fast-path, taken by every task in namespace except the first. */ - if (likely(ns->frozen_offsets)) - return; - - mutex_lock(&offset_lock); - /* Nothing to-do: vvar_page has been already initialized. */ - if (ns->frozen_offsets) - goto out; - - ns->frozen_offsets = true; - vdata = page_address(ns->vvar_page); - vc = vdata->clock_data; - - for (i = 0; i < CS_BASES; i++) - timens_setup_vdso_clock_data(&vc[i], ns); - - if (IS_ENABLED(CONFIG_POSIX_AUX_CLOCKS)) { - for (i = 0; i < ARRAY_SIZE(vdata->aux_clock_data); i++) - timens_setup_vdso_clock_data(&vdata->aux_clock_data[i], ns); - } - -out: - mutex_unlock(&offset_lock); -} +DEFINE_MUTEX(timens_offset_lock); void free_time_ns(struct time_namespace *ns) { @@ -298,12 +189,6 @@ static void timens_put(struct ns_common *ns) put_time_ns(to_time_ns(ns)); } -void timens_commit(struct task_struct *tsk, struct time_namespace *ns) -{ - timens_set_vvar_page(tsk, ns); - vdso_join_timens(tsk, ns); -} - static int timens_install(struct nsset *nsset, struct ns_common *new) { struct nsproxy *nsproxy = nsset->nsproxy; @@ -367,36 +252,33 @@ static void show_offset(struct seq_file *m, int clockid, struct timespec64 *ts) void proc_timens_show_offsets(struct task_struct *p, struct seq_file *m) { - struct ns_common *ns; - struct time_namespace *time_ns; + struct time_namespace *time_ns __free(time_ns) = NULL; + struct ns_common *ns = timens_for_children_get(p); - ns = timens_for_children_get(p); if (!ns) return; + time_ns = to_time_ns(ns); show_offset(m, CLOCK_MONOTONIC, &time_ns->offsets.monotonic); show_offset(m, CLOCK_BOOTTIME, &time_ns->offsets.boottime); - put_time_ns(time_ns); } int proc_timens_set_offset(struct file *file, struct task_struct *p, struct proc_timens_offset *offsets, int noffsets) { - struct ns_common *ns; - struct time_namespace *time_ns; + struct time_namespace *time_ns __free(time_ns) = NULL; + struct ns_common *ns = timens_for_children_get(p); struct timespec64 tp; int i, err; - ns = timens_for_children_get(p); if (!ns) return -ESRCH; + time_ns = to_time_ns(ns); - if (!file_ns_capable(file, time_ns->user_ns, CAP_SYS_TIME)) { - put_time_ns(time_ns); + if (!file_ns_capable(file, time_ns->user_ns, CAP_SYS_TIME)) return -EPERM; - } for (i = 0; i < noffsets; i++) { struct proc_timens_offset *off = &offsets[i]; @@ -409,15 +291,14 @@ int proc_timens_set_offset(struct file *file, struct task_struct *p, ktime_get_boottime_ts64(&tp); break; default: - err = -EINVAL; - goto out; + return -EINVAL; } - err = -ERANGE; + if (off->val.tv_sec > KTIME_SEC_MAX || off->val.tv_sec < -KTIME_SEC_MAX) + return -ERANGE; - if (off->val.tv_sec > KTIME_SEC_MAX || - off->val.tv_sec < -KTIME_SEC_MAX) - goto out; + if (off->val.tv_nsec < 0 || off->val.tv_nsec >= NSEC_PER_SEC) + return -EINVAL; tp = timespec64_add(tp, off->val); /* @@ -425,10 +306,10 @@ int proc_timens_set_offset(struct file *file, struct task_struct *p, * still unreachable. */ if (tp.tv_sec < 0 || tp.tv_sec > KTIME_SEC_MAX / 2) - goto out; + return -ERANGE; } - mutex_lock(&offset_lock); + mutex_lock(&timens_offset_lock); if (time_ns->frozen_offsets) { err = -EACCES; goto out_unlock; @@ -453,9 +334,7 @@ int proc_timens_set_offset(struct file *file, struct task_struct *p, } out_unlock: - mutex_unlock(&offset_lock); -out: - put_time_ns(time_ns); + mutex_unlock(&timens_offset_lock); return err; } diff --git a/kernel/time/namespace_internal.h b/kernel/time/namespace_internal.h new file mode 100644 index 000000000000..e85da11abb4d --- /dev/null +++ b/kernel/time/namespace_internal.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _TIME_NAMESPACE_INTERNAL_H +#define _TIME_NAMESPACE_INTERNAL_H + +#include + +/* + * Protects possibly multiple offsets writers racing each other + * and tasks entering the namespace. + */ +extern struct mutex timens_offset_lock; + +#endif /* _TIME_NAMESPACE_INTERNAL_H */ diff --git a/kernel/time/namespace_vdso.c b/kernel/time/namespace_vdso.c new file mode 100644 index 000000000000..0e154f901501 --- /dev/null +++ b/kernel/time/namespace_vdso.c @@ -0,0 +1,146 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Author: Andrei Vagin + * Author: Dmitry Safonov + */ + +#include +#include +#include +#include +#include + +#include +#include + +#include "namespace_internal.h" + +static struct timens_offset offset_from_ts(struct timespec64 off) +{ + struct timens_offset ret; + + ret.sec = off.tv_sec; + ret.nsec = off.tv_nsec; + + return ret; +} + +/* + * A time namespace VVAR page has the same layout as the VVAR page which + * contains the system wide VDSO data. + * + * For a normal task the VVAR pages are installed in the normal ordering: + * VVAR + * PVCLOCK + * HVCLOCK + * TIMENS <- Not really required + * + * Now for a timens task the pages are installed in the following order: + * TIMENS + * PVCLOCK + * HVCLOCK + * VVAR + * + * The check for vdso_clock->clock_mode is in the unlikely path of + * the seq begin magic. So for the non-timens case most of the time + * 'seq' is even, so the branch is not taken. + * + * If 'seq' is odd, i.e. a concurrent update is in progress, the extra check + * for vdso_clock->clock_mode is a non-issue. The task is spin waiting for the + * update to finish and for 'seq' to become even anyway. + * + * Timens page has vdso_clock->clock_mode set to VDSO_CLOCKMODE_TIMENS which + * enforces the time namespace handling path. + */ +static void timens_setup_vdso_clock_data(struct vdso_clock *vc, + struct time_namespace *ns) +{ + struct timens_offset *offset = vc->offset; + struct timens_offset monotonic = offset_from_ts(ns->offsets.monotonic); + struct timens_offset boottime = offset_from_ts(ns->offsets.boottime); + + vc->seq = 1; + vc->clock_mode = VDSO_CLOCKMODE_TIMENS; + offset[CLOCK_MONOTONIC] = monotonic; + offset[CLOCK_MONOTONIC_RAW] = monotonic; + offset[CLOCK_MONOTONIC_COARSE] = monotonic; + offset[CLOCK_BOOTTIME] = boottime; + offset[CLOCK_BOOTTIME_ALARM] = boottime; +} + +struct page *find_timens_vvar_page(struct vm_area_struct *vma) +{ + if (likely(vma->vm_mm == current->mm)) + return current->nsproxy->time_ns->vvar_page; + + /* + * VM_PFNMAP | VM_IO protect .fault() handler from being called + * through interfaces like /proc/$pid/mem or + * process_vm_{readv,writev}() as long as there's no .access() + * in special_mapping_vmops(). + * For more details check_vma_flags() and __access_remote_vm() + */ + + WARN(1, "vvar_page accessed remotely"); + + return NULL; +} + +static void timens_set_vvar_page(struct task_struct *task, + struct time_namespace *ns) +{ + struct vdso_time_data *vdata; + struct vdso_clock *vc; + unsigned int i; + + if (ns == &init_time_ns) + return; + + /* Fast-path, taken by every task in namespace except the first. */ + if (likely(ns->frozen_offsets)) + return; + + guard(mutex)(&timens_offset_lock); + /* Nothing to-do: vvar_page has been already initialized. */ + if (ns->frozen_offsets) + return; + + ns->frozen_offsets = true; + vdata = page_address(ns->vvar_page); + vc = vdata->clock_data; + + for (i = 0; i < CS_BASES; i++) + timens_setup_vdso_clock_data(&vc[i], ns); + + if (IS_ENABLED(CONFIG_POSIX_AUX_CLOCKS)) { + for (i = 0; i < ARRAY_SIZE(vdata->aux_clock_data); i++) + timens_setup_vdso_clock_data(&vdata->aux_clock_data[i], ns); + } +} + +/* + * The vvar page layout depends on whether a task belongs to the root or + * non-root time namespace. Whenever a task changes its namespace, the VVAR + * page tables are cleared and then they will be re-faulted with a + * corresponding layout. + * See also the comment near timens_setup_vdso_clock_data() for details. + */ +static int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) +{ + struct mm_struct *mm = task->mm; + struct vm_area_struct *vma; + VMA_ITERATOR(vmi, mm, 0); + + guard(mmap_read_lock)(mm); + for_each_vma(vmi, vma) { + if (vma_is_special_mapping(vma, &vdso_vvar_mapping)) + zap_vma_pages(vma); + } + return 0; +} + +void timens_commit(struct task_struct *tsk, struct time_namespace *ns) +{ + timens_set_vvar_page(tsk, ns); + vdso_join_timens(tsk, ns); +} diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c index 425d429906d0..f3aaef695b8c 100644 --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -296,6 +296,11 @@ int sched_clock_suspend(void) return 0; } +static int sched_clock_syscore_suspend(void *data) +{ + return sched_clock_suspend(); +} + void sched_clock_resume(void) { struct clock_read_data *rd = &cd.read_data[0]; @@ -305,14 +310,23 @@ void sched_clock_resume(void) rd->read_sched_clock = cd.actual_read_sched_clock; } -static struct syscore_ops sched_clock_ops = { - .suspend = sched_clock_suspend, - .resume = sched_clock_resume, +static void sched_clock_syscore_resume(void *data) +{ + sched_clock_resume(); +} + +static const struct syscore_ops sched_clock_syscore_ops = { + .suspend = sched_clock_syscore_suspend, + .resume = sched_clock_syscore_resume, +}; + +static struct syscore sched_clock_syscore = { + .ops = &sched_clock_syscore_ops, }; static int __init sched_clock_syscore_init(void) { - register_syscore_ops(&sched_clock_ops); + register_syscore(&sched_clock_syscore); return 0; } diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 06184f304c6a..377e29ca5a52 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -1995,6 +1995,11 @@ void timekeeping_resume(void) timerfd_resume(); } +static void timekeeping_syscore_resume(void *data) +{ + timekeeping_resume(); +} + int timekeeping_suspend(void) { struct timekeeper *tks = &tk_core.shadow_timekeeper; @@ -2062,15 +2067,24 @@ int timekeeping_suspend(void) return 0; } +static int timekeeping_syscore_suspend(void *data) +{ + return timekeeping_suspend(); +} + /* sysfs resume/suspend bits for timekeeping */ -static struct syscore_ops timekeeping_syscore_ops = { - .resume = timekeeping_resume, - .suspend = timekeeping_suspend, +static const struct syscore_ops timekeeping_syscore_ops = { + .resume = timekeeping_syscore_resume, + .suspend = timekeeping_syscore_suspend, +}; + +static struct syscore timekeeping_syscore = { + .ops = &timekeeping_syscore_ops, }; static int __init timekeeping_init_ops(void) { - register_syscore_ops(&timekeeping_syscore_ops); + register_syscore(&timekeeping_syscore); return 0; } device_initcall(timekeeping_init_ops); @@ -2140,6 +2154,11 @@ static __always_inline void timekeeping_apply_adjustment(struct timekeeper *tk, * xtime_nsec_2 = xtime_nsec_1 - offset * Which simplifies to: * xtime_nsec -= offset + * + * When subtracting offset from xtime_nsec, the same amount + * (in appropriate units) has to be added to ntp_error, in + * order to correctly track the delta between the time + * reported in xtime_nsec, and the intended time. */ if ((mult_adj > 0) && (tk->tkr_mono.mult + mult_adj < mult_adj)) { /* NTP adjustment caused clocksource mult overflow */ @@ -2150,6 +2169,7 @@ static __always_inline void timekeeping_apply_adjustment(struct timekeeper *tk, tk->tkr_mono.mult += mult_adj; tk->xtime_interval += interval; tk->tkr_mono.xtime_nsec -= offset; + tk->ntp_error += offset << tk->ntp_error_shift; } /* @@ -3065,7 +3085,9 @@ static const struct attribute_group aux_clock_enable_attr_group = { static int __init tk_aux_sysfs_init(void) { struct kobject *auxo, *tko = kobject_create_and_add("time", kernel_kobj); + struct kobject *clks[MAX_AUX_CLOCKS]; int ret = -ENOMEM; + int i; if (!tko) return ret; @@ -3074,21 +3096,28 @@ static int __init tk_aux_sysfs_init(void) if (!auxo) goto err_clean; - for (int i = 0; i < MAX_AUX_CLOCKS; i++) { + for (i = 0; i < MAX_AUX_CLOCKS; i++) { char id[2] = { [0] = '0' + i, }; - struct kobject *clk = kobject_create_and_add(id, auxo); + clks[i] = kobject_create_and_add(id, auxo); - if (!clk) { + if (!clks[i]) { ret = -ENOMEM; - goto err_clean; + goto err_clks; } - ret = sysfs_create_group(clk, &aux_clock_enable_attr_group); + ret = sysfs_create_group(clks[i], &aux_clock_enable_attr_group); if (ret) - goto err_clean; + goto err_clk; } return 0; +err_clk: + kobject_put(clks[i]); +err_clks: + while (--i >= 0) { + sysfs_remove_group(clks[i], &aux_clock_enable_attr_group); + kobject_put(clks[i]); + } err_clean: kobject_put(auxo); kobject_put(tko); diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c index 76d896a99d7b..32f338052bc4 100644 --- a/kernel/time/timer_migration.c +++ b/kernel/time/timer_migration.c @@ -1667,8 +1667,10 @@ static int tmigr_setup_groups(unsigned int cpu, unsigned int node, } /* Assert single root without parent */ - if (WARN_ON_ONCE(i >= tmigr_hierarchy_levels)) + if (WARN_ON_ONCE(i >= tmigr_hierarchy_levels)) { + kfree(stack); return -EINVAL; + } for (; i >= start_lvl; i--) { group = stack[i]; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 6f5b6de6baaf..1bb2175ab1af 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -4627,7 +4627,8 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file) /** * ftrace_regex_open - initialize function tracer filter files - * @ops: The ftrace_ops that hold the hash filters + * @tr: The trace_array that holds the ftrace_ops [optional] + * @ops: The ftrace_ops that hold the hash filters [optional] * @flag: The type of filter to process * @inode: The inode, usually passed in to your open routine * @file: The file, usually passed in to your open routine @@ -4641,26 +4642,45 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file) * tracing_lseek() should be used as the lseek routine, and * release must call ftrace_regex_release(). * + * Note, If @tr is not NULL, its reference has to be taken before + * @ops may be referenced. + * If @ops is NULL and @tr is not, then @tr->ops is used. + * If @tr is NULL and @ops is not then @ops->private is uesd for @tr. + * If both @tr and @ops are NULL, then the &global_ops is + * to be used, and @tr will be the global_ops.private pointer. + * * Returns: 0 on success or a negative errno value on failure */ int -ftrace_regex_open(struct ftrace_ops *ops, int flag, +ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag, struct inode *inode, struct file *file) { - struct ftrace_iterator *iter; + struct ftrace_iterator *iter = NULL; struct ftrace_hash *hash; struct list_head *mod_head; - struct trace_array *tr = ops->private; - int ret = -ENOMEM; - - ftrace_ops_init(ops); + int ret = -ENODEV; if (unlikely(ftrace_disabled)) return -ENODEV; + if (!tr) { + if (!ops) + ops = &global_ops; + tr = ops->private; + } + if (tracing_check_open_get_tr(tr)) return -ENODEV; + if (!ops) + ops = tr->ops; + + if (WARN_ON_ONCE(!ops)) + goto out; + + ftrace_ops_init(ops); + + ret = -ENOMEM; iter = kzalloc(sizeof(*iter), GFP_KERNEL); if (!iter) goto out; @@ -4738,21 +4758,19 @@ ftrace_regex_open(struct ftrace_ops *ops, int flag, static int ftrace_filter_open(struct inode *inode, struct file *file) { - struct ftrace_ops *ops = inode->i_private; + struct trace_array *tr = inode->i_private; - /* Checks for tracefs lockdown */ - return ftrace_regex_open(ops, - FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES, - inode, file); + return ftrace_regex_open(tr, NULL, + FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES, + inode, file); } static int ftrace_notrace_open(struct inode *inode, struct file *file) { - struct ftrace_ops *ops = inode->i_private; + struct trace_array *tr = inode->i_private; - /* Checks for tracefs lockdown */ - return ftrace_regex_open(ops, FTRACE_ITER_NOTRACE, + return ftrace_regex_open(tr, NULL, FTRACE_ITER_NOTRACE, inode, file); } @@ -7060,15 +7078,15 @@ static const struct file_operations ftrace_graph_notrace_fops = { }; #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ -void ftrace_create_filter_files(struct ftrace_ops *ops, +void ftrace_create_filter_files(struct trace_array *tr, struct dentry *parent) { trace_create_file("set_ftrace_filter", TRACE_MODE_WRITE, parent, - ops, &ftrace_filter_fops); + tr, &ftrace_filter_fops); trace_create_file("set_ftrace_notrace", TRACE_MODE_WRITE, parent, - ops, &ftrace_notrace_fops); + tr, &ftrace_notrace_fops); } /* @@ -7093,7 +7111,6 @@ void ftrace_destroy_filter_files(struct ftrace_ops *ops) static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer) { - trace_create_file("available_filter_functions", TRACE_MODE_READ, d_tracer, NULL, &ftrace_avail_fops); @@ -7106,7 +7123,7 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer) trace_create_file("touched_functions", TRACE_MODE_READ, d_tracer, NULL, &ftrace_touched_fops); - ftrace_create_filter_files(&global_ops, d_tracer); + ftrace_create_filter_files(NULL, d_tracer); #ifdef CONFIG_FUNCTION_GRAPH_TRACER trace_create_file("set_graph_function", TRACE_MODE_WRITE, d_tracer, diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c index 31c7ef21b526..73018638934a 100644 --- a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c @@ -583,7 +583,6 @@ struct ring_buffer_per_cpu { struct trace_buffer { unsigned flags; - int cpus; atomic_t record_disabled; atomic_t resizing; cpumask_var_t cpumask; @@ -2564,7 +2563,6 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags, init_irq_work(&buffer->irq_work.work, rb_wake_up_waiters); init_waitqueue_head(&buffer->irq_work.waiters); - buffer->cpus = nr_cpu_ids; bsize = sizeof(void *) * nr_cpu_ids; buffer->buffers = kzalloc(ALIGN(bsize, cache_line_size()), @@ -6682,6 +6680,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer, struct buffer_page *reader; unsigned long missed_events; unsigned int commit; + unsigned int size; unsigned int read; u64 save_timestamp; @@ -6716,7 +6715,8 @@ int ring_buffer_read_page(struct trace_buffer *buffer, event = rb_reader_event(cpu_buffer); read = reader->read; - commit = rb_page_size(reader); + commit = rb_page_commit(reader); + size = rb_page_size(reader); /* Check if any events were dropped */ missed_events = cpu_buffer->lost_events; @@ -6728,34 +6728,31 @@ int ring_buffer_read_page(struct trace_buffer *buffer, * we must copy the data from the page to the buffer. * Otherwise, we can simply swap the page with the one passed in. */ - if (read || (len < (commit - read)) || + if (read || (len < (size - read)) || cpu_buffer->reader_page == cpu_buffer->commit_page || cpu_buffer->mapped) { struct buffer_data_page *rpage = cpu_buffer->reader_page->page; unsigned int rpos = read; unsigned int pos = 0; - unsigned int size; + unsigned int event_size; + unsigned int flags = 0; - /* - * If a full page is expected, this can still be returned - * if there's been a previous partial read and the - * rest of the page can be read and the commit page is off - * the reader page. - */ - if (full && - (!read || (len < (commit - read)) || - cpu_buffer->reader_page == cpu_buffer->commit_page)) + /* If a full page is requested, it cannot be the commit page */ + if (full && cpu_buffer->reader_page == cpu_buffer->commit_page) return -1; - if (len > (commit - read)) - len = (commit - read); + if (len > (size - read)) + len = (size - read); /* Always keep the time extend and data together */ - size = rb_event_ts_length(event); + event_size = rb_event_ts_length(event); - if (len < size) + if (len < event_size) return -1; + if (commit & RB_MISSED_EVENTS) + flags = RB_MISSED_EVENTS; + /* save the current timestamp, since the user will need it */ save_timestamp = cpu_buffer->read_stamp; @@ -6767,25 +6764,25 @@ int ring_buffer_read_page(struct trace_buffer *buffer, * one or two events. * We have already ensured there's enough space if this * is a time extend. */ - size = rb_event_length(event); - memcpy(bpage->data + pos, rpage->data + rpos, size); + event_size = rb_event_length(event); + memcpy(bpage->data + pos, rpage->data + rpos, event_size); - len -= size; + len -= event_size; rb_advance_reader(cpu_buffer); rpos = reader->read; - pos += size; + pos += event_size; - if (rpos >= commit) + if (rpos >= size) break; event = rb_reader_event(cpu_buffer); /* Always keep the time extend and data together */ - size = rb_event_ts_length(event); - } while (len >= size); + event_size = rb_event_ts_length(event); + } while (len >= event_size); /* update bpage */ - local_set(&bpage->commit, pos); + local_set(&bpage->commit, pos | flags); bpage->time_stamp = save_timestamp; /* we copied everything to the beginning */ @@ -6810,12 +6807,14 @@ int ring_buffer_read_page(struct trace_buffer *buffer, * on the page. */ if (reader->real_end) - local_set(&bpage->commit, reader->real_end); + local_set(&bpage->commit, reader->real_end | + (commit & RB_MISSED_EVENTS)); } cpu_buffer->lost_events = 0; commit = local_read(&bpage->commit); + size = commit & ~RB_MISSED_MASK; /* * Set a flag in the commit field if we lost events */ @@ -6823,20 +6822,20 @@ int ring_buffer_read_page(struct trace_buffer *buffer, /* If there is room at the end of the page to save the * missed events, then record it there. */ - if (rb_page_capacity(reader) - commit >= sizeof(missed_events)) { - memcpy(&bpage->data[commit], &missed_events, + if (rb_page_capacity(reader) - size >= sizeof(missed_events)) { + memcpy(&bpage->data[size], &missed_events, sizeof(missed_events)); - local_add(RB_MISSED_STORED, &bpage->commit); - commit += sizeof(missed_events); + commit |= RB_MISSED_STORED; + size += sizeof(missed_events); } - local_add(RB_MISSED_EVENTS, &bpage->commit); + local_set(&bpage->commit, commit | RB_MISSED_EVENTS); } /* * This page may be off to user land. Zero it out here. */ - if (commit < rb_page_capacity(reader)) - memset(&bpage->data[commit], 0, rb_page_capacity(reader) - commit); + if (size < rb_page_capacity(reader)) + memset(&bpage->data[size], 0, rb_page_capacity(reader) - size); return read; } diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index e62772a76671..77ad2392954a 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -537,7 +537,7 @@ static struct trace_array global_trace = { .trace_flags = TRACE_DEFAULT_FLAGS, }; -static struct trace_array *printk_trace = &global_trace; +struct trace_array *printk_trace = &global_trace; /* List of trace_arrays interested in the top level trace_marker */ static LIST_HEAD(marker_copies); @@ -1904,10 +1904,7 @@ static ssize_t trace_seq_to_buffer(struct trace_seq *s, void *buf, size_t cnt) unsigned long __read_mostly tracing_thresh; #ifdef CONFIG_TRACER_MAX_TRACE -static const struct file_operations tracing_max_lat_fops; - #ifdef LATENCY_FS_NOTIFY - static struct workqueue_struct *fsnotify_wq; static void latency_fsnotify_workfn(struct work_struct *work) @@ -1924,17 +1921,6 @@ static void latency_fsnotify_workfn_irq(struct irq_work *iwork) queue_work(fsnotify_wq, &tr->fsnotify_work); } -static void trace_create_maxlat_file(struct trace_array *tr, - struct dentry *d_tracer) -{ - INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn); - init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq); - tr->d_max_latency = trace_create_file("tracing_max_latency", - TRACE_MODE_WRITE, - d_tracer, tr, - &tracing_max_lat_fops); -} - __init static int latency_fsnotify_init(void) { fsnotify_wq = alloc_workqueue("tr_max_lat_wq", @@ -1959,14 +1945,22 @@ void latency_fsnotify(struct trace_array *tr) */ irq_work_queue(&tr->fsnotify_irqwork); } +#endif /* !LATENCY_FS_NOTIFY */ -#else /* !LATENCY_FS_NOTIFY */ - -#define trace_create_maxlat_file(tr, d_tracer) \ - trace_create_file("tracing_max_latency", TRACE_MODE_WRITE, \ - d_tracer, tr, &tracing_max_lat_fops) +static const struct file_operations tracing_max_lat_fops; +static void trace_create_maxlat_file(struct trace_array *tr, + struct dentry *d_tracer) +{ +#ifdef LATENCY_FS_NOTIFY + INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn); + init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq); #endif + tr->d_max_latency = trace_create_file("tracing_max_latency", + TRACE_MODE_WRITE, + d_tracer, tr, + &tracing_max_lat_fops); +} /* * Copy the new maximum trace into the separate maximum-trace @@ -2101,7 +2095,9 @@ update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu) __update_max_tr(tr, tsk, cpu); arch_spin_unlock(&tr->max_lock); } - +#else /* !CONFIG_TRACER_MAX_TRACE */ +static inline void trace_create_maxlat_file(struct trace_array *tr, + struct dentry *d_tracer) { } #endif /* CONFIG_TRACER_MAX_TRACE */ struct pipe_wait { @@ -5894,7 +5890,7 @@ trace_event_update_with_eval_map(struct module *mod, map = start; - trace_event_update_all(map, len); + trace_event_update_all(map, len, mod); if (len <= 0) return; @@ -9397,11 +9393,72 @@ trace_options_core_write(struct file *filp, const char __user *ubuf, size_t cnt, return cnt; } +/* + * The tr_index is the address of a trace_array->trace_flags_index[] + * element that holds the index of the trace flag. But since the + * trace_array reference has not been taken yet, it cannot be referenced + * as it could have been freed by a rmdir of the instance the trace_array + * represents. + * + * Search the list of trace_arrays and compare the tr_index to the + * address of the entire trace_array trace_flags_index array for each + * trace_array in the list. If one is matched, then take the reference + * and return it. If not, the trace_array no longer exits. + */ +static int trace_array_options_get(void *tr_index) +{ + struct trace_array *tr; + int ret; + + ret = security_locked_down(LOCKDOWN_TRACEFS); + if (ret) + return ret; + + if (tracing_disabled) + return -ENODEV; + + guard(mutex)(&trace_types_lock); + list_for_each_entry(tr, &ftrace_trace_arrays, list) { + if (tr_index >= (void *)&tr->trace_flags_index[0] && + tr_index < (void *)&tr->trace_flags_index[TRACE_FLAGS_MAX_SIZE]) { + tr->ref++; + return 0; + } + } + return -ENODEV; +} + +static int trace_options_open(struct inode *inode, struct file *filp) +{ + void *tr_index = inode->i_private; + + if (trace_array_options_get(tr_index) < 0) + return -ENODEV; + + filp->private_data = tr_index; + + return 0; +} + +static int trace_options_release(struct inode *inode, struct file *filp) +{ + void *tr_index = filp->private_data; + struct trace_array *tr; + unsigned int index; + + get_tr_index(tr_index, &tr, &index); + + trace_array_put(tr); + + return 0; +} + static const struct file_operations trace_options_core_fops = { - .open = tracing_open_generic, - .read = trace_options_core_read, - .write = trace_options_core_write, - .llseek = generic_file_llseek, + .open = trace_options_open, + .read = trace_options_core_read, + .write = trace_options_core_write, + .llseek = generic_file_llseek, + .release = trace_options_release, }; struct dentry *trace_create_file(const char *name, @@ -10392,9 +10449,7 @@ init_tracer_tracefs(struct trace_array *tr, struct dentry *d_tracer) create_trace_options_dir(tr); -#ifdef CONFIG_TRACER_MAX_TRACE trace_create_maxlat_file(tr, d_tracer); -#endif if (ftrace_create_function_files(tr, d_tracer)) MEM_FAIL(1, "Could not allocate function filter files"); diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 62d5a5f224c4..729b1921f955 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -346,8 +346,8 @@ struct trace_array { spinlock_t snapshot_trigger_lock; unsigned int snapshot; unsigned long max_latency; -#ifdef CONFIG_FSNOTIFY struct dentry *d_max_latency; +#ifdef CONFIG_FSNOTIFY struct work_struct fsnotify_work; struct irq_work fsnotify_irqwork; #endif @@ -479,6 +479,8 @@ extern bool trace_clock_in_ns(struct trace_array *tr); extern unsigned long trace_adjust_address(struct trace_array *tr, unsigned long addr); +extern struct trace_array *printk_trace; + /* * The global tracer (top) should be the first trace array added, * but we check the flag anyway. @@ -1253,7 +1255,7 @@ extern void clear_ftrace_function_probes(struct trace_array *tr); int register_ftrace_command(struct ftrace_func_command *cmd); int unregister_ftrace_command(struct ftrace_func_command *cmd); -void ftrace_create_filter_files(struct ftrace_ops *ops, +void ftrace_create_filter_files(struct trace_array *tr, struct dentry *parent); void ftrace_destroy_filter_files(struct ftrace_ops *ops); @@ -1276,11 +1278,11 @@ static inline void clear_ftrace_function_probes(struct trace_array *tr) { } +#define ftrace_create_filter_files(tr, parent) do { } while (0) /* * The ops parameter passed in is usually undefined. * This must be a macro. */ -#define ftrace_create_filter_files(ops, parent) do { } while (0) #define ftrace_destroy_filter_files(ops) do { } while (0) #endif /* CONFIG_FUNCTION_TRACER && CONFIG_DYNAMIC_FTRACE */ @@ -2134,13 +2136,13 @@ static inline const char *get_syscall_name(int syscall) #ifdef CONFIG_EVENT_TRACING void trace_event_init(void); -void trace_event_update_all(struct trace_eval_map **map, int len); +void trace_event_update_all(struct trace_eval_map **map, int len, struct module *mod); /* Used from boot time tracer */ extern int ftrace_set_clr_event(struct trace_array *tr, char *buf, int set); extern int trigger_process_regex(struct trace_event_file *file, char *buff); #else static inline void __init trace_event_init(void) { } -static inline void trace_event_update_all(struct trace_eval_map **map, int len) { } +static inline void trace_event_update_all(struct trace_eval_map **map, int len, struct module *mod) { } #endif #ifdef CONFIG_TRACER_SNAPSHOT diff --git a/kernel/trace/trace_btf.c b/kernel/trace/trace_btf.c index 5bbdbcbbde3c..43e379801177 100644 --- a/kernel/trace/trace_btf.c +++ b/kernel/trace/trace_btf.c @@ -61,47 +61,50 @@ struct btf_anon_stack { /* * Find a member of data structure/union by name and return it. - * Return NULL if not found, or -EINVAL if parameter is invalid. - * If the member is an member of anonymous union/structure, the offset - * of that anonymous union/structure is stored into @anon_offset. Caller - * can calculate the correct offset from the root data structure by - * adding anon_offset to the member's offset. + * Return NULL if not found, or ERR_PTR(-EINVAL) if parameter is invalid. + * If the member is a member of an anonymous union/structure, the bit offset + * of that anonymous union/structure is stored into @anon_offset. + * If @member_type is non-NULL, the actual containing structure/union type + * of the found member is stored into @member_type. */ const struct btf_member *btf_find_struct_member(struct btf *btf, const struct btf_type *type, const char *member_name, - u32 *anon_offset) + u32 *anon_offset, + const struct btf_type **member_type) { struct btf_anon_stack *anon_stack; const struct btf_member *member; + const struct btf_type *mtype; u32 tid, cur_offset = 0; const char *name; int i, top = 0; + if (!btf_type_is_struct(type)) + return ERR_PTR(-EINVAL); + anon_stack = kcalloc(BTF_ANON_STACK_MAX, sizeof(*anon_stack), GFP_KERNEL); if (!anon_stack) return ERR_PTR(-ENOMEM); retry: - if (!btf_type_is_struct(type)) { - member = ERR_PTR(-EINVAL); - goto out; - } - for_each_member(i, type, member) { if (!member->name_off) { /* Anonymous union/struct: push it for later use */ - if (btf_type_skip_modifiers(btf, member->type, &tid) && + mtype = btf_type_skip_modifiers(btf, member->type, &tid); + if (mtype && btf_type_is_struct(mtype) && top < BTF_ANON_STACK_MAX) { anon_stack[top].tid = tid; - anon_stack[top++].offset = - cur_offset + member->offset; + anon_stack[top++].offset = cur_offset + + __btf_member_bit_offset(type, member); } } else { name = btf_name_by_offset(btf, member->name_off); if (name && !strcmp(member_name, name)) { if (anon_offset) *anon_offset = cur_offset; + if (member_type) + *member_type = type; goto out; } } diff --git a/kernel/trace/trace_btf.h b/kernel/trace/trace_btf.h index 4bc44bc261e6..4bd26bceae23 100644 --- a/kernel/trace/trace_btf.h +++ b/kernel/trace/trace_btf.h @@ -8,4 +8,5 @@ const struct btf_param *btf_get_func_param(const struct btf_type *func_proto, const struct btf_member *btf_find_struct_member(struct btf *btf, const struct btf_type *type, const char *member_name, - u32 *anon_offset); + u32 *anon_offset, + const struct btf_type **member_type); diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c index 0f96eb95ad38..b3ee028f93ef 100644 --- a/kernel/trace/trace_events.c +++ b/kernel/trace/trace_events.c @@ -3402,7 +3402,7 @@ static void update_event_fields(struct trace_event_call *call, } /* Update all events for replacing eval and sanitizing */ -void trace_event_update_all(struct trace_eval_map **map, int len) +void trace_event_update_all(struct trace_eval_map **map, int len, struct module *mod) { struct trace_event_call *call, *p; const char *last_system = NULL; @@ -3414,6 +3414,10 @@ void trace_event_update_all(struct trace_eval_map **map, int len) mutex_lock(&event_mutex); down_write(&trace_event_sem); list_for_each_entry_safe(call, p, &ftrace_events, list) { + + if (mod && call->module != mod) + continue; + /* events are usually grouped together with systems */ if (!last_system || call->class->system != last_system) { first = true; diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index d17c18934445..a56994cfc64b 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -101,7 +101,7 @@ int ftrace_create_function_files(struct trace_array *tr, return ret; } - ftrace_create_filter_files(tr->ops, parent); + ftrace_create_filter_files(tr, parent); return 0; } diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c index 8e1df280d311..95dcfeede94a 100644 --- a/kernel/trace/trace_probe.c +++ b/kernel/trace/trace_probe.c @@ -355,9 +355,8 @@ static bool btf_type_is_char_ptr(struct btf *btf, const struct btf_type *type) { const struct btf_type *real_type; u32 intdata; - s32 tid; - real_type = btf_type_skip_modifiers(btf, type->type, &tid); + real_type = btf_type_skip_modifiers(btf, type->type, NULL); if (!real_type) return false; @@ -374,14 +373,13 @@ static bool btf_type_is_char_array(struct btf *btf, const struct btf_type *type) const struct btf_type *real_type; const struct btf_array *array; u32 intdata; - s32 tid; if (BTF_INFO_KIND(type->info) != BTF_KIND_ARRAY) return false; array = (const struct btf_array *)(type + 1); - real_type = btf_type_skip_modifiers(btf, array->type, &tid); + real_type = btf_type_skip_modifiers(btf, array->type, NULL); intdata = btf_type_int(real_type); return !(BTF_INT_ENCODING(intdata) & BTF_INT_SIGNED) @@ -579,12 +577,12 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type, { struct fetch_insn *code = *pcode; const struct btf_member *field; + const struct btf_type *mtype; u32 bitoffs, anon_offs; bool is_struct = ctx->struct_btf != NULL; struct btf *btf = ctx_btf(ctx); char *next; int is_ptr; - s32 tid; do { if (!is_struct) { @@ -595,7 +593,7 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type, } /* Convert a struct pointer type to a struct type */ - type = btf_type_skip_modifiers(btf, type->type, &tid); + type = btf_type_skip_modifiers(btf, type->type, NULL); if (!type) { trace_probe_log_err(ctx->offset, BAD_BTF_TID); return -EINVAL; @@ -614,7 +612,7 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type, anon_offs = 0; field = btf_find_struct_member(btf, type, fieldname, - &anon_offs); + &anon_offs, &mtype); if (IS_ERR(field)) { trace_probe_log_err(ctx->offset, BAD_BTF_TID); return PTR_ERR(field); @@ -627,7 +625,7 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type, bitoffs += anon_offs; /* Accumulate the bit-offsets of the dot-connected fields */ - if (btf_type_kflag(type)) { + if (btf_type_kflag(mtype)) { bitoffs += BTF_MEMBER_BIT_OFFSET(field->offset); ctx->last_bitsize = BTF_MEMBER_BITFIELD_SIZE(field->offset); } else { @@ -635,7 +633,7 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type, ctx->last_bitsize = 0; } - type = btf_type_skip_modifiers(btf, field->type, &tid); + type = btf_type_skip_modifiers(btf, field->type, NULL); if (!type) { trace_probe_log_err(ctx->offset, BAD_BTF_TID); return -EINVAL; @@ -754,7 +752,7 @@ static int parse_btf_arg(char *varname, return -ENOENT; found: - type = btf_type_skip_modifiers(ctx->btf, tid, &tid); + type = btf_type_skip_modifiers(ctx->btf, tid, NULL); found_type: if (!type) { trace_probe_log_err(ctx->offset, BAD_BTF_TID); diff --git a/kernel/trace/trace_stack.c b/kernel/trace/trace_stack.c index 0aa2514a6593..e7f4e523587d 100644 --- a/kernel/trace/trace_stack.c +++ b/kernel/trace/trace_stack.c @@ -499,7 +499,7 @@ stack_trace_filter_open(struct inode *inode, struct file *file) struct ftrace_ops *ops = inode->i_private; /* Checks for tracefs lockdown */ - return ftrace_regex_open(ops, FTRACE_ITER_FILTER, + return ftrace_regex_open(NULL, ops, FTRACE_ITER_FILTER, inode, file); } diff --git a/lib/interval_tree_test.c b/lib/interval_tree_test.c index 5fd62656f42e..c479d9176b75 100644 --- a/lib/interval_tree_test.c +++ b/lib/interval_tree_test.c @@ -139,13 +139,13 @@ static int intersection_range_check(void) intxn1 = bitmap_alloc(nnodes, GFP_KERNEL); if (!intxn1) { - WARN_ON_ONCE("Failed to allocate intxn1\n"); + WARN_ONCE(1, "Failed to allocate intxn1\n"); return -ENOMEM; } intxn2 = bitmap_alloc(nnodes, GFP_KERNEL); if (!intxn2) { - WARN_ON_ONCE("Failed to allocate intxn2\n"); + WARN_ONCE(1, "Failed to allocate intxn2\n"); bitmap_free(intxn1); return -ENOMEM; } diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 5aa4c9500018..cff42ec10f4b 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -4026,35 +4026,40 @@ int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp, void *entry, unsigned long range_lo, unsigned long range_hi, unsigned long *next, gfp_t gfp) { - unsigned long min = range_lo; - int ret = 0; - - range_lo = max(min, *next); - ret = mas_empty_area(mas, range_lo, range_hi, 1); - if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) { - mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED; - ret = 1; - } - if (ret < 0 && range_lo > min) { - mas_reset(mas); - ret = mas_empty_area(mas, min, range_hi, 1); - if (ret == 0) - ret = 1; - } - if (ret < 0) - return ret; + int ret; + unsigned long min; + min = range_lo; do { + range_lo = max(min, *next); + ret = mas_empty_area(mas, range_lo, range_hi, 1); + if (ret < 0 && range_lo > min) { + mas_reset(mas); + ret = mas_empty_area(mas, min, range_hi, 1); + if (ret == 0) + ret = 1; + } + if (ret < 0) + goto out; + mas_insert(mas, entry); } while (mas_nomem(mas, gfp)); - if (mas_is_err(mas)) - return xa_err(mas->node); + if (mas_is_err(mas)) { + ret = xa_err(mas->node); + goto out; + } + + if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) { + mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED; + ret = 1; + } *startp = mas->index; *next = *startp + 1; if (*next == 0) mas->tree->ma_flags |= MT_FLAGS_ALLOC_WRAPPED; +out: mas_destroy(mas); return ret; } diff --git a/lib/string.c b/lib/string.c index b632c71df1a5..01c9845024b8 100644 --- a/lib/string.c +++ b/lib/string.c @@ -839,7 +839,8 @@ void *memchr_inv(const void *start, int c, size_t bytes) { u8 value = c; u64 value64; - unsigned int words, prefix; + size_t words; + unsigned int prefix; if (bytes <= 16) return check_bytes8(start, value, bytes); diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 00d34a6c6276..ec1df4cd8480 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -392,7 +392,7 @@ static int dmirror_fault(struct dmirror *dmirror, unsigned long start, /* Since the mm is for the mirrored process, get a reference first. */ if (!mmget_not_zero(mm)) - return 0; + return -EFAULT; for (addr = start; addr < end; addr = range.end) { range.start = addr; diff --git a/lib/vdso/datastore.c b/lib/vdso/datastore.c index a565c30c71a0..64d868dad8dd 100644 --- a/lib/vdso/datastore.c +++ b/lib/vdso/datastore.c @@ -103,28 +103,3 @@ struct vm_area_struct *vdso_install_vvar_mapping(struct mm_struct *mm, unsigned VM_PFNMAP | VM_SEALED_SYSMAP, &vdso_vvar_mapping); } - -#ifdef CONFIG_TIME_NS -/* - * The vvar page layout depends on whether a task belongs to the root or - * non-root time namespace. Whenever a task changes its namespace, the VVAR - * page tables are cleared and then they will be re-faulted with a - * corresponding layout. - * See also the comment near timens_setup_vdso_clock_data() for details. - */ -int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) -{ - struct mm_struct *mm = task->mm; - struct vm_area_struct *vma; - VMA_ITERATOR(vmi, mm, 0); - - mmap_read_lock(mm); - for_each_vma(vmi, vma) { - if (vma_is_special_mapping(vma, &vdso_vvar_mapping)) - zap_vma_pages(vma); - } - mmap_read_unlock(mm); - - return 0; -} -#endif diff --git a/mm/damon/sysfs.c b/mm/damon/sysfs.c index 53f99f05eb38..7f52470be36e 100644 --- a/mm/damon/sysfs.c +++ b/mm/damon/sysfs.c @@ -1440,14 +1440,16 @@ static inline bool damon_sysfs_kdamond_running( static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, struct damon_sysfs_context *sys_ctx) { + enum damon_ops_id ops_id; int err; - err = damon_select_ops(ctx, sys_ctx->ops_id); + ops_id = READ_ONCE(sys_ctx->ops_id); + err = damon_select_ops(ctx, ops_id); if (err) return err; ctx->addr_unit = READ_ONCE(sys_ctx->addr_unit); /* addr_unit is respected by only DAMON_OPS_PADDR */ - if (sys_ctx->ops_id == DAMON_OPS_PADDR) + if (ops_id == DAMON_OPS_PADDR) ctx->min_sz_region = max( DAMON_MIN_REGION / ctx->addr_unit, 1); err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 56b85d3d3dbb..bedd9ba223fb 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2311,6 +2311,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, * Use flush_needed to indicate whether the PMD entry * is present, instead of checking pmd_present() again. */ + if (flush_needed && pmd_dirty(orig_pmd)) + folio_mark_dirty(folio); if (flush_needed && pmd_young(orig_pmd) && likely(vma_has_recency(vma))) folio_mark_accessed(folio); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6b3ba903ba39..a11e56ef9923 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -353,12 +353,6 @@ int hugepage_madvise(struct vm_area_struct *vma, #endif *vm_flags &= ~VM_NOHUGEPAGE; *vm_flags |= VM_HUGEPAGE; - /* - * If the vma become good for khugepaged to scan, - * register it here without waiting a page fault that - * may not happen any time soon. - */ - khugepaged_enter_vma(vma, *vm_flags); break; case MADV_NOHUGEPAGE: *vm_flags &= ~VM_HUGEPAGE; diff --git a/mm/madvise.c b/mm/madvise.c index 73f56b18c337..b8c25fea4fab 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -177,6 +177,14 @@ static int madvise_update_vma(vm_flags_t new_flags, /* vm_flags is protected by the mmap_lock held in write mode. */ vma_start_write(vma); vm_flags_reset(vma, new_flags); + /* + * If the vma become good for khugepaged to scan, + * register it here without waiting a page fault that + * may not happen any time soon. + */ + if (new_flags & VM_HUGEPAGE) + khugepaged_enter_vma(vma, new_flags); + if (set_new_anon_name) return replace_anon_vma_name(vma, anon_name); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index b7030bcd9b03..9fdfb2d31765 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -221,7 +221,7 @@ void put_online_mems(void) bool movable_node_enabled = false; static int mhp_default_online_type = -1; -int mhp_get_default_online_type(void) +enum mmop mhp_get_default_online_type(void) { if (mhp_default_online_type >= 0) return mhp_default_online_type; @@ -240,7 +240,7 @@ int mhp_get_default_online_type(void) return mhp_default_online_type; } -void mhp_set_default_online_type(int online_type) +void mhp_set_default_online_type(enum mmop online_type) { mhp_default_online_type = online_type; } @@ -1046,7 +1046,7 @@ static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn return movable_node_enabled ? movable_zone : kernel_zone; } -struct zone *zone_for_pfn_range(int online_type, int nid, +struct zone *zone_for_pfn_range(enum mmop online_type, int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages) { @@ -1279,7 +1279,8 @@ static pg_data_t *hotadd_init_pgdat(int nid) pgdat = NODE_DATA(nid); /* init node's zones as empty zones, we don't have any present pages.*/ - free_area_init_core_hotplug(pgdat); + if (free_area_init_core_hotplug(pgdat)) + return NULL; /* * The node we allocated has no zone fallback lists. For avoiding @@ -2338,7 +2339,7 @@ EXPORT_SYMBOL_GPL(remove_memory); static int try_offline_memory_block(struct memory_block *mem, void *arg) { - uint8_t online_type = MMOP_ONLINE_KERNEL; + enum mmop online_type = MMOP_ONLINE_KERNEL; uint8_t **online_types = arg; struct page *page; int rc; @@ -2371,7 +2372,7 @@ static int try_reonline_memory_block(struct memory_block *mem, void *arg) int rc; if (**online_types != MMOP_OFFLINE) { - mem->online_type = **online_types; + mem->online_type = (enum mmop)**online_types; rc = device_online(&mem->dev); if (rc < 0) pr_warn("%s: Failed to re-online memory: %d", diff --git a/mm/mm_init.c b/mm/mm_init.c index 5789ad8ba9b6..a27ba1c1a719 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1351,7 +1351,7 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, unsigned long node_start_pfn, unsigned long node_end_pfn) { - unsigned long realtotalpages = 0, totalpages = 0; + unsigned long realtotalpages = 0; enum zone_type i; for (i = 0; i < MAX_NR_ZONES; i++) { @@ -1381,11 +1381,10 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, zone->present_early_pages = real_size; #endif - totalpages += spanned; realtotalpages += real_size; } - pgdat->node_spanned_pages = totalpages; + pgdat->node_spanned_pages = node_end_pfn - node_start_pfn; pgdat->node_present_pages = realtotalpages; pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages); } @@ -1561,7 +1560,7 @@ void __init set_pageblock_order(void) * NOTE: this function is only called during memory hotplug */ #ifdef CONFIG_MEMORY_HOTPLUG -void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) +int __ref free_area_init_core_hotplug(struct pglist_data *pgdat) { int nid = pgdat->node_id; enum zone_type z; @@ -1569,8 +1568,14 @@ void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) pgdat_init_internals(pgdat); - if (pgdat->per_cpu_nodestats == &boot_nodestats) - pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat); + if (pgdat->per_cpu_nodestats == &boot_nodestats) { + struct per_cpu_nodestat __percpu *p; + + p = alloc_percpu(struct per_cpu_nodestat); + if (!p) + return -ENOMEM; + pgdat->per_cpu_nodestats = p; + } /* * Reset the nr_zones, order and highest_zoneidx before reuse. @@ -1608,6 +1613,8 @@ void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) zone->present_pages = 0; zone_init_internals(zone, z, nid, 0); } + + return 0; } #endif diff --git a/mm/mremap.c b/mm/mremap.c index 419a0ea0a870..bdf63ffaad3d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -1247,18 +1247,29 @@ static void dontunmap_complete(struct vma_remap_struct *vrm, { unsigned long start = vrm->addr; unsigned long end = vrm->addr + vrm->old_len; - unsigned long old_start = vrm->vma->vm_start; - unsigned long old_end = vrm->vma->vm_end; + struct vm_area_struct *vma = vrm->vma; + unsigned long old_start = vma->vm_start; + unsigned long old_end = vma->vm_end; /* We always clear VM_LOCKED[ONFAULT] on the old VMA. */ - vm_flags_clear(vrm->vma, VM_LOCKED_MASK); + vm_flags_clear(vma, VM_LOCKED_MASK); /* * anon_vma links of the old vma is no longer needed after its page * table has been moved. */ - if (new_vma != vrm->vma && start == old_start && end == old_end) - unlink_anon_vmas(vrm->vma); + if (new_vma != vma && start == old_start && end == old_end) { + const pgoff_t pgoff_unfaulted = vma->vm_start >> PAGE_SHIFT; + + unlink_anon_vmas(vma); + /* + * The VMA is now unfaulted and it is an invariant that + * unfaulted anonymous VMAs have page offset equal to + * vma->vm_start >> PAGE_SHIFT. + */ + if (vma_is_anonymous(vma) && !vma->vm_file) + vma->vm_pgoff = pgoff_unfaulted; + } /* Because we won't unmap we don't need to touch locked_vm. */ } diff --git a/mm/secretmem.c b/mm/secretmem.c index 9b0f5d9ec6f4..932fb32e63b6 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -18,6 +18,8 @@ #include #include #include +#include +#include #include @@ -47,10 +49,69 @@ bool secretmem_active(void) return !!atomic_read(&secretmem_users); } +struct secretmem_inode_state { + struct user_struct *user; + atomic_long_t nr_pages_accounted; +}; + +static bool __secretmem_account_pages(struct user_struct *user, + unsigned long nr_pages) +{ + unsigned long page_limit, cur_pages, new_pages; + + if (!nr_pages) + return true; + + page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; + + cur_pages = atomic_long_read(&user->locked_vm); + do { + new_pages = cur_pages + nr_pages; + if (new_pages > page_limit) + return false; + } while (!atomic_long_try_cmpxchg(&user->locked_vm, + &cur_pages, new_pages)); + return true; +} + +static bool secretmem_account_folio(struct secretmem_inode_state *state, + const struct folio *folio) +{ + const unsigned long nr_pages = folio_nr_pages(folio); + + if (!__secretmem_account_pages(state->user, nr_pages)) + return false; + + atomic_long_add(nr_pages, &state->nr_pages_accounted); + return true; +} + +static void __secretmem_unaccount_pages(struct secretmem_inode_state *state, + unsigned long nr_pages) +{ + atomic_long_sub(nr_pages, &state->user->locked_vm); + atomic_long_sub(nr_pages, &state->nr_pages_accounted); +} + +static void secretmem_unaccount_folio(struct secretmem_inode_state *state, + struct folio *folio) +{ + __secretmem_unaccount_pages(state, folio_nr_pages(folio)); +} + +static void secretmem_unaccount_all_folios(struct secretmem_inode_state *state) +{ + const unsigned long nr_pages_accounted = + atomic_long_read(&state->nr_pages_accounted); + + __secretmem_unaccount_pages(state, nr_pages_accounted); +} + static vm_fault_t secretmem_fault(struct vm_fault *vmf) { struct address_space *mapping = vmf->vma->vm_file->f_mapping; struct inode *inode = file_inode(vmf->vma->vm_file); + struct secretmem_inode_state *state = inode->i_private; pgoff_t offset = vmf->pgoff; gfp_t gfp = vmf->gfp_mask; unsigned long addr; @@ -72,8 +133,15 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) goto out; } + if (!secretmem_account_folio(state, folio)) { + folio_put(folio); + ret = VM_FAULT_SIGBUS; + goto out; + } + err = set_direct_map_invalid_noflush(folio_page(folio, 0)); if (err) { + secretmem_unaccount_folio(state, folio); folio_put(folio); ret = vmf_error(err); goto out; @@ -82,6 +150,7 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) __folio_mark_uptodate(folio); err = filemap_add_folio(mapping, folio, offset, gfp); if (unlikely(err)) { + secretmem_unaccount_folio(state, folio); /* * If a split of large page was required, it * already happened when we marked the page invalid @@ -112,23 +181,30 @@ static const struct vm_operations_struct secretmem_vm_ops = { .fault = secretmem_fault, }; +static void secretmem_destroy_inode_priv(struct inode *inode) +{ + struct secretmem_inode_state *state = inode->i_private; + + secretmem_unaccount_all_folios(state); + free_uid(state->user); + kfree(state); + inode->i_private = NULL; +} + static int secretmem_release(struct inode *inode, struct file *file) { atomic_dec(&secretmem_users); + secretmem_destroy_inode_priv(inode); + return 0; } static int secretmem_mmap_prepare(struct vm_area_desc *desc) { - const unsigned long len = desc->end - desc->start; - if ((desc->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) return -EINVAL; - if (!mlock_future_ok(desc->mm, desc->vm_flags | VM_LOCKED, len)) - return -EAGAIN; - - desc->vm_flags |= VM_LOCKED | VM_DONTDUMP; + desc->vm_flags |= VM_DONTDUMP; desc->vm_ops = &secretmem_vm_ops; return 0; @@ -188,20 +264,40 @@ static const struct inode_operations secretmem_iops = { static struct vfsmount *secretmem_mnt; +static int secretmem_init_inode_priv(struct inode *inode) +{ + struct secretmem_inode_state *state; + + state = kzalloc_obj(*state); + if (!state) + return -ENOMEM; + + state->user = get_uid(current_user()); + inode->i_private = state; + return 0; +} + static struct file *secretmem_file_create(unsigned long flags) { struct file *file; struct inode *inode; const char *anon_name = "[secretmem]"; + int err; inode = anon_inode_make_secure_inode(secretmem_mnt->mnt_sb, anon_name, NULL); if (IS_ERR(inode)) return ERR_CAST(inode); + err = secretmem_init_inode_priv(inode); + if (err) + goto err_free_inode; + file = alloc_file_pseudo(inode, secretmem_mnt, "secretmem", O_RDWR | O_LARGEFILE, &secretmem_fops); - if (IS_ERR(file)) - goto err_free_inode; + if (IS_ERR(file)) { + err = PTR_ERR(file); + goto err_free_priv; + } mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER); mapping_set_unevictable(inode->i_mapping); @@ -216,10 +312,11 @@ static struct file *secretmem_file_create(unsigned long flags) atomic_inc(&secretmem_users); return file; - +err_free_priv: + secretmem_destroy_inode_priv(inode); err_free_inode: iput(inode); - return file; + return ERR_PTR(err); } SYSCALL_DEFINE1(memfd_secret, unsigned int, flags) diff --git a/net/batman-adv/bridge_loop_avoidance.c b/net/batman-adv/bridge_loop_avoidance.c index 1591911bceb4..5f2bb9eba3e1 100644 --- a/net/batman-adv/bridge_loop_avoidance.c +++ b/net/batman-adv/bridge_loop_avoidance.c @@ -694,12 +694,14 @@ static void batadv_bla_add_claim(struct batadv_priv *bat_priv, struct batadv_bla_backbone_gw *old_backbone_gw; struct batadv_bla_claim *claim; struct batadv_bla_claim search_claim; - bool remove_crc = false; int hash_added; + u16 claim_crc; + bool changed; ether_addr_copy(search_claim.addr, mac); search_claim.vid = vid; claim = batadv_claim_hash_find(bat_priv, &search_claim); + claim_crc = crc16(0, mac, ETH_ALEN); /* create a new claim entry if it does not exist yet. */ if (!claim) { @@ -731,43 +733,56 @@ static void batadv_bla_add_claim(struct batadv_priv *bat_priv, kfree(claim); return; } + + spin_lock_bh(&backbone_gw->crc_lock); + backbone_gw->crc ^= claim_crc; + spin_unlock_bh(&backbone_gw->crc_lock); + + WRITE_ONCE(backbone_gw->lasttime, jiffies); + + batadv_claim_put(claim); + return; + } + + WRITE_ONCE(claim->lasttime, jiffies); + + /* replace backbone_gw atomically and adjust reference counters */ + spin_lock_bh(&claim->backbone_lock); + if (claim->backbone_gw != backbone_gw) { + changed = true; + + old_backbone_gw = claim->backbone_gw; + kref_get(&backbone_gw->refcount); + claim->backbone_gw = backbone_gw; } else { - WRITE_ONCE(claim->lasttime, jiffies); - if (claim->backbone_gw == backbone_gw) - /* no need to register a new backbone */ - goto claim_free_ref; + old_backbone_gw = NULL; + changed = false; + } + spin_unlock_bh(&claim->backbone_lock); + if (changed) { batadv_dbg(BATADV_DBG_BLA, bat_priv, "%s(): changing ownership for %pM, vid %d to gw %pM\n", __func__, mac, batadv_print_vid(vid), backbone_gw->orig); - remove_crc = true; - } + /* add claim address to new backbone_gw */ + spin_lock_bh(&backbone_gw->crc_lock); + backbone_gw->crc ^= claim_crc; + spin_unlock_bh(&backbone_gw->crc_lock); - /* replace backbone_gw atomically and adjust reference counters */ - spin_lock_bh(&claim->backbone_lock); - old_backbone_gw = claim->backbone_gw; - kref_get(&backbone_gw->refcount); - claim->backbone_gw = backbone_gw; - spin_unlock_bh(&claim->backbone_lock); + WRITE_ONCE(backbone_gw->lasttime, jiffies); + } - if (remove_crc) { + if (old_backbone_gw) { /* remove claim address from old backbone_gw */ spin_lock_bh(&old_backbone_gw->crc_lock); - old_backbone_gw->crc ^= crc16(0, claim->addr, ETH_ALEN); + old_backbone_gw->crc ^= claim_crc; spin_unlock_bh(&old_backbone_gw->crc_lock); - } - - batadv_backbone_gw_put(old_backbone_gw); - /* add claim address to new backbone_gw */ - spin_lock_bh(&backbone_gw->crc_lock); - backbone_gw->crc ^= crc16(0, claim->addr, ETH_ALEN); - spin_unlock_bh(&backbone_gw->crc_lock); - WRITE_ONCE(backbone_gw->lasttime, jiffies); + batadv_backbone_gw_put(old_backbone_gw); + } -claim_free_ref: batadv_claim_put(claim); } diff --git a/net/batman-adv/distributed-arp-table.c b/net/batman-adv/distributed-arp-table.c index 4bae8df85516..72717e2952ab 100644 --- a/net/batman-adv/distributed-arp-table.c +++ b/net/batman-adv/distributed-arp-table.c @@ -375,18 +375,19 @@ batadv_dat_entry_hash_find(struct batadv_priv *bat_priv, __be32 ip, static void batadv_dat_entry_add(struct batadv_priv *bat_priv, __be32 ip, u8 *mac_addr, unsigned short vid) { + u64 u64_mac = ether_addr_to_u64(mac_addr); struct batadv_dat_entry *dat_entry; int hash_added; dat_entry = batadv_dat_entry_hash_find(bat_priv, ip, vid); /* if this entry is already known, just update it */ if (dat_entry) { - if (!batadv_compare_eth(dat_entry->mac_addr, mac_addr)) - ether_addr_copy(dat_entry->mac_addr, mac_addr); + atomic64_set(&dat_entry->mac_addr, u64_mac); + dat_entry->last_update = jiffies; batadv_dbg(BATADV_DBG_DAT, bat_priv, "Entry updated: %pI4 %pM (vid: %d)\n", - &dat_entry->ip, dat_entry->mac_addr, + &dat_entry->ip, mac_addr, batadv_print_vid(vid)); goto out; } @@ -397,7 +398,7 @@ static void batadv_dat_entry_add(struct batadv_priv *bat_priv, __be32 ip, dat_entry->ip = ip; dat_entry->vid = vid; - ether_addr_copy(dat_entry->mac_addr, mac_addr); + atomic64_set(&dat_entry->mac_addr, u64_mac); dat_entry->last_update = jiffies; kref_init(&dat_entry->refcount); @@ -413,7 +414,7 @@ static void batadv_dat_entry_add(struct batadv_priv *bat_priv, __be32 ip, } batadv_dbg(BATADV_DBG_DAT, bat_priv, "New entry added: %pI4 %pM (vid: %d)\n", - &dat_entry->ip, dat_entry->mac_addr, batadv_print_vid(vid)); + &dat_entry->ip, mac_addr, batadv_print_vid(vid)); out: batadv_dat_entry_put(dat_entry); @@ -868,6 +869,8 @@ batadv_dat_cache_dump_entry(struct sk_buff *msg, u32 portid, struct netlink_callback *cb, struct batadv_dat_entry *dat_entry) { + u8 mac[ETH_ALEN]; + u64 u64_mac; int msecs; void *hdr; @@ -880,11 +883,12 @@ batadv_dat_cache_dump_entry(struct sk_buff *msg, u32 portid, genl_dump_check_consistent(cb, hdr); msecs = jiffies_to_msecs(jiffies - dat_entry->last_update); + u64_mac = atomic64_read(&dat_entry->mac_addr); + u64_to_ether_addr(u64_mac, mac); if (nla_put_in_addr(msg, BATADV_ATTR_DAT_CACHE_IP4ADDRESS, dat_entry->ip) || - nla_put(msg, BATADV_ATTR_DAT_CACHE_HWADDRESS, ETH_ALEN, - dat_entry->mac_addr) || + nla_put(msg, BATADV_ATTR_DAT_CACHE_HWADDRESS, ETH_ALEN, mac) || nla_put_u16(msg, BATADV_ATTR_DAT_CACHE_VID, dat_entry->vid) || nla_put_u32(msg, BATADV_ATTR_LAST_SEEN_MSECS, msecs)) { genlmsg_cancel(msg, hdr); @@ -1151,6 +1155,8 @@ bool batadv_dat_snoop_outgoing_arp_request(struct batadv_priv *bat_priv, struct net_device *mesh_iface = bat_priv->mesh_iface; int hdr_size = 0; unsigned short vid; + u8 mac[ETH_ALEN]; + u64 u64_mac; if (!atomic_read(&bat_priv->distributed_arp_table)) goto out; @@ -1178,6 +1184,9 @@ bool batadv_dat_snoop_outgoing_arp_request(struct batadv_priv *bat_priv, dat_entry = batadv_dat_entry_hash_find(bat_priv, ip_dst, vid); if (dat_entry) { + u64_mac = atomic64_read(&dat_entry->mac_addr); + u64_to_ether_addr(u64_mac, mac); + /* If the ARP request is destined for a local client the local * client will answer itself. DAT would only generate a * duplicate packet. @@ -1186,7 +1195,7 @@ bool batadv_dat_snoop_outgoing_arp_request(struct batadv_priv *bat_priv, * additional DAT answer may trigger kernel warnings about * a packet coming from the wrong port. */ - if (batadv_is_my_client(bat_priv, dat_entry->mac_addr, vid)) { + if (batadv_is_my_client(bat_priv, mac, vid)) { ret = true; goto out; } @@ -1196,18 +1205,16 @@ bool batadv_dat_snoop_outgoing_arp_request(struct batadv_priv *bat_priv, * the backbone gws belonging to our backbone has claimed the * destination. */ - if (!batadv_bla_check_claim(bat_priv, - dat_entry->mac_addr, vid)) { + if (!batadv_bla_check_claim(bat_priv, mac, vid)) { batadv_dbg(BATADV_DBG_DAT, bat_priv, "Device %pM claimed by another backbone gw. Don't send ARP reply!", - dat_entry->mac_addr); + mac); ret = true; goto out; } skb_new = batadv_dat_arp_create_reply(bat_priv, ip_dst, ip_src, - dat_entry->mac_addr, - hw_src, vid); + mac, hw_src, vid); if (!skb_new) goto out; @@ -1249,6 +1256,8 @@ bool batadv_dat_snoop_incoming_arp_request(struct batadv_priv *bat_priv, struct batadv_dat_entry *dat_entry = NULL; bool ret = false; unsigned short vid; + u8 mac[ETH_ALEN]; + u64 u64_mac; int err; if (!atomic_read(&bat_priv->distributed_arp_table)) @@ -1276,8 +1285,11 @@ bool batadv_dat_snoop_incoming_arp_request(struct batadv_priv *bat_priv, if (!dat_entry) goto out; + u64_mac = atomic64_read(&dat_entry->mac_addr); + u64_to_ether_addr(u64_mac, mac); + skb_new = batadv_dat_arp_create_reply(bat_priv, ip_dst, ip_src, - dat_entry->mac_addr, hw_src, vid); + mac, hw_src, vid); if (!skb_new) goto out; @@ -1368,6 +1380,8 @@ bool batadv_dat_snoop_incoming_arp_reply(struct batadv_priv *bat_priv, u8 *hw_src, *hw_dst; bool dropped = false; unsigned short vid; + u8 mac[ETH_ALEN]; + u64 u64_mac; if (!atomic_read(&bat_priv->distributed_arp_table)) goto out; @@ -1396,11 +1410,17 @@ bool batadv_dat_snoop_incoming_arp_reply(struct batadv_priv *bat_priv, * this frame would lead to doubled receive of an ARP reply. */ dat_entry = batadv_dat_entry_hash_find(bat_priv, ip_src, vid); - if (dat_entry && batadv_compare_eth(hw_src, dat_entry->mac_addr)) { - batadv_dbg(BATADV_DBG_DAT, bat_priv, "Doubled ARP reply removed: ARP MSG = [src: %pM-%pI4 dst: %pM-%pI4]; dat_entry: %pM-%pI4\n", - hw_src, &ip_src, hw_dst, &ip_dst, - dat_entry->mac_addr, &dat_entry->ip); - dropped = true; + if (dat_entry) { + u64_mac = atomic64_read(&dat_entry->mac_addr); + u64_to_ether_addr(u64_mac, mac); + + if (batadv_compare_eth(hw_src, mac)) { + batadv_dbg(BATADV_DBG_DAT, bat_priv, + "Doubled ARP reply removed: ARP MSG = [src: %pM-%pI4 dst: %pM-%pI4]; dat_entry: %pM-%pI4\n", + hw_src, &ip_src, hw_dst, &ip_dst, + mac, &dat_entry->ip); + dropped = true; + } } /* Update our internal cache with both the IP addresses the node got diff --git a/net/batman-adv/mesh-interface.c b/net/batman-adv/mesh-interface.c index 0d598bf39bfc..3d49fd7ef3e6 100644 --- a/net/batman-adv/mesh-interface.c +++ b/net/batman-adv/mesh-interface.c @@ -258,6 +258,8 @@ static netdev_tx_t batadv_interface_tx(struct sk_buff *skb, if (batadv_compare_eth(ethhdr->h_dest, ectp_addr)) goto dropped; + batadv_skb_set_priority(skb, 0); + gw_mode = atomic_read(&bat_priv->gw.mode); if (is_multicast_ether_addr(ethhdr->h_dest)) { /* if gw mode is off, broadcast every packet */ @@ -291,6 +293,9 @@ static netdev_tx_t batadv_interface_tx(struct sk_buff *skb, send: if (do_bcast && !is_broadcast_ether_addr(ethhdr->h_dest)) { + /* WARNING batadv_mcast_forw_mode might add more headers + * in front of the skb. and might even reallocate the skb + */ forw_mode = batadv_mcast_forw_mode(bat_priv, skb, vid, &mcast_is_routable); switch (forw_mode) { @@ -308,8 +313,6 @@ static netdev_tx_t batadv_interface_tx(struct sk_buff *skb, } } - batadv_skb_set_priority(skb, 0); - /* ethernet packet should be broadcasted */ if (do_bcast) { primary_if = batadv_primary_if_get_selected(bat_priv); diff --git a/net/batman-adv/types.h b/net/batman-adv/types.h index ac4494f1b8e2..51d04f351fdd 100644 --- a/net/batman-adv/types.h +++ b/net/batman-adv/types.h @@ -2127,7 +2127,7 @@ struct batadv_dat_entry { __be32 ip; /** @mac_addr: the MAC address associated to the stored IPv4 */ - u8 mac_addr[ETH_ALEN]; + atomic64_t mac_addr; /** @vid: the vlan ID associated to this entry */ unsigned short vid; diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c index eb91dc9fbe86..6edc26fe6d2d 100644 --- a/net/bluetooth/hci_conn.c +++ b/net/bluetooth/hci_conn.c @@ -286,8 +286,6 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data) struct hci_cp_enhanced_setup_sync_conn cp; const struct sco_param *param; - kfree(conn_handle); - if (!hci_conn_valid(hdev, conn)) return -ECANCELED; @@ -456,6 +454,15 @@ static bool hci_setup_sync_conn(struct hci_conn *conn, __u16 handle) return true; } +static void hci_enhanced_setup_sync_destroy(struct hci_dev *hdev, void *data, + int err) +{ + struct conn_handle_t *conn_handle = data; + + hci_conn_put(conn_handle->conn); + kfree(conn_handle); +} + bool hci_setup_sync(struct hci_conn *conn, __u16 handle) { int result; @@ -467,12 +474,15 @@ bool hci_setup_sync(struct hci_conn *conn, __u16 handle) if (!conn_handle) return false; - conn_handle->conn = conn; + conn_handle->conn = hci_conn_get(conn); conn_handle->handle = handle; result = hci_cmd_sync_queue(conn->hdev, hci_enhanced_setup_sync, - conn_handle, NULL); - if (result < 0) + conn_handle, + hci_enhanced_setup_sync_destroy); + if (result < 0) { + hci_conn_put(conn); kfree(conn_handle); + } return result == 0; } diff --git a/net/bluetooth/hci_sync.c b/net/bluetooth/hci_sync.c index 27261f8e904e..be5886a7424f 100644 --- a/net/bluetooth/hci_sync.c +++ b/net/bluetooth/hci_sync.c @@ -540,8 +540,6 @@ static int adv_timeout_expire_sync(struct hci_dev *hdev, void *data) { u8 instance = *(u8 *)data; - kfree(data); - hci_clear_adv_instance_sync(hdev, NULL, instance, false); if (list_empty(&hdev->adv_instances)) @@ -550,6 +548,12 @@ static int adv_timeout_expire_sync(struct hci_dev *hdev, void *data) return 0; } +static void adv_timeout_expire_destroy(struct hci_dev *hdev, void *data, + int err) +{ + kfree(data); +} + static void adv_timeout_expire(struct work_struct *work) { u8 *inst_ptr; @@ -570,7 +574,9 @@ static void adv_timeout_expire(struct work_struct *work) goto unlock; *inst_ptr = hdev->cur_adv_instance; - hci_cmd_sync_queue(hdev, adv_timeout_expire_sync, inst_ptr, NULL); + if (hci_cmd_sync_queue(hdev, adv_timeout_expire_sync, inst_ptr, + adv_timeout_expire_destroy) < 0) + kfree(inst_ptr); unlock: hci_dev_unlock(hdev); diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c index 1a916c0f40cc..d332cd999d8c 100644 --- a/net/bluetooth/l2cap_sock.c +++ b/net/bluetooth/l2cap_sock.c @@ -1284,7 +1284,12 @@ static void l2cap_sock_kill(struct sock *sk) BT_DBG("sk %p state %s", sk, state_to_string(sk->sk_state)); + /* Take lock to synchronize against access without owning sk->sk_socket, + * eg. in l2cap_sock_cleanup_listen(). proto_ops etc. don't need lock. + */ + lock_sock(sk); l2cap_sock_put_chan(sk); + release_sock(sk); /* Kill poor orphan */ sock_set_flag(sk, SOCK_DEAD); @@ -1487,14 +1492,10 @@ static void l2cap_sock_cleanup_listen(struct sock *parent) * establish sk_lock -> conn->lock and invert the established * conn->lock -> chan->lock -> sk_lock order (lockdep deadlock). * - * Instead, briefly take the child sk lock to fetch and pin its chan. - * l2cap_conn_del() reaches the chan free only via - * l2cap_chan_del() -> l2cap_sock_teardown_cb(), which itself takes - * the child sk lock; holding it across l2cap_chan_hold_unless_zero() - * therefore guarantees the chan cannot be freed while we read and - * pin it (hold_unless_zero() additionally skips a chan already past - * its last reference). We then drop the sk lock before taking - * chan->lock, so sk and chan locks are never held together. + * Instead, briefly take the child sk lock to synchronize vs. + * l2cap_sock_kill that puts l2cap_pi(sk)->chan. We then drop the sk + * lock before taking chan->lock, so sk and chan locks are never held + * together. * * Since we cannot call l2cap_chan_close() without conn->lock, * schedule l2cap_chan_timeout to close the channel; it already @@ -1504,10 +1505,12 @@ static void l2cap_sock_cleanup_listen(struct sock *parent) struct l2cap_chan *chan; lock_sock_nested(sk, L2CAP_NESTING_NORMAL); - chan = l2cap_chan_hold_unless_zero(l2cap_pi(sk)->chan); + chan = l2cap_pi(sk)->chan; + if (chan) + l2cap_chan_hold(chan); release_sock(sk); if (!chan) { - /* l2cap_conn_del() already tearing this child down */ + /* Already torn down */ sock_put(sk); continue; } @@ -1539,6 +1542,11 @@ static int l2cap_sock_new_connection_cb(struct l2cap_chan *chan, lock_sock(parent); + if (parent->sk_state != BT_LISTEN) { + release_sock(parent); + return -EINVAL; + } + /* Check for backlog size */ if (sk_acceptq_is_full(parent)) { BT_DBG("backlog full %d", parent->sk_ack_backlog); @@ -1703,10 +1711,14 @@ static void l2cap_sock_state_change_cb(struct l2cap_chan *chan, int state, if (!sk) return; + lock_sock(sk); + sk->sk_state = state; if (err) sk->sk_err = err; + + release_sock(sk); } static struct sk_buff *l2cap_sock_alloc_skb_cb(struct l2cap_chan *chan, @@ -1782,6 +1794,8 @@ static void l2cap_sock_resume_cb(struct l2cap_chan *chan) if (!sk) return; + lock_sock(sk); + if (test_and_clear_bit(FLAG_PENDING_SECURITY, &chan->flags)) { sk->sk_state = BT_CONNECTED; chan->state = BT_CONNECTED; @@ -1789,6 +1803,8 @@ static void l2cap_sock_resume_cb(struct l2cap_chan *chan) clear_bit(BT_SK_SUSPEND, &bt_sk(sk)->flags); sk->sk_state_change(sk); + + release_sock(sk); } static void l2cap_sock_set_shutdown_cb(struct l2cap_chan *chan) diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c index 0ef601dbb2e2..01f46a985e2b 100644 --- a/net/bluetooth/mgmt.c +++ b/net/bluetooth/mgmt.c @@ -2434,11 +2434,15 @@ static int send_cancel(struct hci_dev *hdev, void *data) mgmt_cmd_complete(cmd->sk, hdev->id, MGMT_OP_MESH_SEND_CANCEL, 0, NULL, 0); - mgmt_pending_free(cmd); return 0; } +static void send_cancel_destroy(struct hci_dev *hdev, void *data, int err) +{ + mgmt_pending_free(data); +} + static int mesh_send_cancel(struct sock *sk, struct hci_dev *hdev, void *data, u16 len) { @@ -2459,7 +2463,8 @@ static int mesh_send_cancel(struct sock *sk, struct hci_dev *hdev, if (!cmd) err = -ENOMEM; else - err = hci_cmd_sync_queue(hdev, send_cancel, cmd, NULL); + err = hci_cmd_sync_queue(hdev, send_cancel, cmd, + send_cancel_destroy); if (err < 0) { err = mgmt_cmd_status(sk, hdev->id, MGMT_OP_MESH_SEND_CANCEL, @@ -2645,7 +2650,7 @@ static int send_hci_cmd_sync(struct hci_dev *hdev, void *data) if (IS_ERR(skb)) { mgmt_cmd_status(cmd->sk, hdev->id, MGMT_OP_HCI_CMD_SYNC, mgmt_status(PTR_ERR(skb))); - goto done; + return 0; } mgmt_cmd_complete(cmd->sk, hdev->id, MGMT_OP_HCI_CMD_SYNC, 0, @@ -2653,12 +2658,14 @@ static int send_hci_cmd_sync(struct hci_dev *hdev, void *data) kfree_skb(skb); -done: - mgmt_pending_free(cmd); - return 0; } +static void send_hci_cmd_sync_destroy(struct hci_dev *hdev, void *data, int err) +{ + mgmt_pending_free(data); +} + static int mgmt_hci_cmd_sync(struct sock *sk, struct hci_dev *hdev, void *data, u16 len) { @@ -2684,7 +2691,8 @@ static int mgmt_hci_cmd_sync(struct sock *sk, struct hci_dev *hdev, if (!cmd) err = -ENOMEM; else - err = hci_cmd_sync_queue(hdev, send_hci_cmd_sync, cmd, NULL); + err = hci_cmd_sync_queue(hdev, send_hci_cmd_sync, cmd, + send_hci_cmd_sync_destroy); if (err < 0) { err = mgmt_cmd_status(sk, hdev->id, MGMT_OP_HCI_CMD_SYNC, @@ -6088,6 +6096,7 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev, struct mgmt_pending_cmd *cmd; const u16 max_uuid_count = ((U16_MAX - sizeof(*cp)) / 16); u16 uuid_count, expected_len; + u8 (*uuids)[16] = NULL; u8 status; int err; @@ -6164,12 +6173,10 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev, hdev->discovery.result_filtering = true; hdev->discovery.type = cp->type; hdev->discovery.rssi = cp->rssi; - hdev->discovery.uuid_count = uuid_count; if (uuid_count > 0) { - hdev->discovery.uuids = kmemdup(cp->uuids, uuid_count * 16, - GFP_KERNEL); - if (!hdev->discovery.uuids) { + uuids = kmemdup(cp->uuids, uuid_count * sizeof(*uuids), GFP_KERNEL); + if (!uuids) { err = mgmt_cmd_complete(sk, hdev->id, MGMT_OP_START_SERVICE_DISCOVERY, MGMT_STATUS_FAILED, @@ -6179,6 +6186,11 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev, } } + spin_lock(&hdev->discovery.lock); + hdev->discovery.uuids = uuids; + hdev->discovery.uuid_count = uuid_count; + spin_unlock(&hdev->discovery.lock); + err = hci_cmd_sync_queue(hdev, start_discovery_sync, cmd, start_discovery_complete); if (err < 0) { @@ -10357,6 +10369,7 @@ static bool is_filter_match(struct hci_dev *hdev, s8 rssi, u8 *eir, !hci_test_quirk(hdev, HCI_QUIRK_STRICT_DUPLICATE_FILTER)))) return false; + spin_lock(&hdev->discovery.lock); if (hdev->discovery.uuid_count != 0) { /* If a list of UUIDs is provided in filter, results with no * matching UUID should be dropped. @@ -10365,9 +10378,12 @@ static bool is_filter_match(struct hci_dev *hdev, s8 rssi, u8 *eir, hdev->discovery.uuids) && !eir_has_uuids(scan_rsp, scan_rsp_len, hdev->discovery.uuid_count, - hdev->discovery.uuids)) + hdev->discovery.uuids)) { + spin_unlock(&hdev->discovery.lock); return false; + } } + spin_unlock(&hdev->discovery.lock); /* If duplicate filtering does not report RSSI changes, then restart * scanning to ensure updated result with updated RSSI values. diff --git a/net/bluetooth/msft.c b/net/bluetooth/msft.c index dea81c8e8b20..3220c44a2ed4 100644 --- a/net/bluetooth/msft.c +++ b/net/bluetooth/msft.c @@ -165,6 +165,11 @@ static bool read_supported_features(struct hci_dev *hdev, if (rp->sub_opcode != MSFT_OP_READ_SUPPORTED_FEATURES) goto failed; + if (skb->len < sizeof(*rp) + rp->evt_prefix_len) { + bt_dev_err(hdev, "MSFT event prefix length mismatch"); + goto failed; + } + if (rp->evt_prefix_len > 0) { msft->evt_prefix = kmemdup(rp->evt_prefix, rp->evt_prefix_len, GFP_KERNEL); diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c index bdc9a7cd9405..6571b33d553e 100644 --- a/net/bluetooth/rfcomm/core.c +++ b/net/bluetooth/rfcomm/core.c @@ -1458,6 +1458,10 @@ static int rfcomm_apply_pn(struct rfcomm_dlc *d, int cr, struct rfcomm_pn *pn) d->mtu = __le16_to_cpu(pn->mtu); + /* MTU 0 causes an infinite loop when fragmenting in sendmsg */ + if (!d->mtu) + d->mtu = RFCOMM_DEFAULT_MTU; + if (cr && d->mtu > s->mtu) d->mtu = s->mtu; diff --git a/net/bridge/br_netlink_tunnel.c b/net/bridge/br_netlink_tunnel.c index a713668ea34f..e7eceab5b515 100644 --- a/net/bridge/br_netlink_tunnel.c +++ b/net/bridge/br_netlink_tunnel.c @@ -302,7 +302,8 @@ int br_process_vlan_tunnel_info(const struct net_bridge *br, if (!(tinfo_last->flags & BRIDGE_VLAN_INFO_RANGE_BEGIN)) return -EINVAL; - if ((tinfo_curr->vid - tinfo_last->vid) != + if (tinfo_curr->vid < tinfo_last->vid || + (tinfo_curr->vid - tinfo_last->vid) != (tinfo_curr->tunid - tinfo_last->tunid)) return -EINVAL; t = tinfo_last->tunid; diff --git a/net/bridge/br_vlan.c b/net/bridge/br_vlan.c index a4ba725ad1df..76da4f580d4c 100644 --- a/net/bridge/br_vlan.c +++ b/net/bridge/br_vlan.c @@ -1131,7 +1131,7 @@ int __br_vlan_set_default_pvid(struct net_bridge *br, u16 pvid, if (err) goto out; - if (br_vlan_delete(br, old_pvid)) + if (!br_vlan_delete(br, old_pvid)) br_vlan_notify(br, NULL, old_pvid, 0, RTM_DELVLAN); br_vlan_notify(br, NULL, pvid, 0, RTM_NEWVLAN); __set_bit(0, changed); @@ -1153,7 +1153,7 @@ int __br_vlan_set_default_pvid(struct net_bridge *br, u16 pvid, &vlchange, extack); if (err) goto err_port; - if (nbp_vlan_delete(p, old_pvid)) + if (!nbp_vlan_delete(p, old_pvid)) br_vlan_notify(br, p, old_pvid, 0, RTM_DELVLAN); br_vlan_notify(p->br, p, pvid, 0, RTM_NEWVLAN); __set_bit(p->port_no, changed); diff --git a/net/can/bcm.c b/net/can/bcm.c index a819872fddef..67d5e3ecc04f 100644 --- a/net/can/bcm.c +++ b/net/can/bcm.c @@ -396,7 +396,6 @@ static void bcm_send_to_user(struct bcm_op *op, struct bcm_msg_head *head, struct sockaddr_can *addr; struct sock *sk = op->sk; unsigned int datalen = head->nframes * op->cfsiz; - int err; unsigned int *pflags; enum skb_drop_reason reason; @@ -453,8 +452,8 @@ static void bcm_send_to_user(struct bcm_op *op, struct bcm_msg_head *head, addr->can_family = AF_CAN; addr->can_ifindex = op->rx_ifindex; - err = sock_queue_rcv_skb_reason(sk, skb, &reason); - if (err < 0) { + reason = sock_queue_rcv_skb_reason(sk, skb); + if (reason) { struct bcm_sock *bo = bcm_sk(sk); sk_skb_reason_drop(sk, skb, reason); diff --git a/net/can/isotp.c b/net/can/isotp.c index 720b911fe1ed..a77ac3954e41 100644 --- a/net/can/isotp.c +++ b/net/can/isotp.c @@ -357,7 +357,8 @@ static void isotp_rcv_skb(struct sk_buff *skb, struct sock *sk) addr->can_family = AF_CAN; addr->can_ifindex = skb->dev->ifindex; - if (sock_queue_rcv_skb_reason(sk, skb, &reason) < 0) + reason = sock_queue_rcv_skb_reason(sk, skb); + if (reason) sk_skb_reason_drop(sk, skb, reason); } diff --git a/net/can/j1939/socket.c b/net/can/j1939/socket.c index e56d8134938e..70a57afdef1a 100644 --- a/net/can/j1939/socket.c +++ b/net/can/j1939/socket.c @@ -332,7 +332,8 @@ static void j1939_sk_recv_one(struct j1939_sock *jsk, struct sk_buff *oskb) if (skb->sk) skcb->msg_flags |= MSG_DONTROUTE; - if (sock_queue_rcv_skb_reason(&jsk->sk, skb, &reason) < 0) + reason = sock_queue_rcv_skb_reason(&jsk->sk, skb); + if (reason) sk_skb_reason_drop(&jsk->sk, skb, reason); } diff --git a/net/can/raw.c b/net/can/raw.c index c397071bfa8b..dc02defa560d 100644 --- a/net/can/raw.c +++ b/net/can/raw.c @@ -206,7 +206,8 @@ static void raw_rcv(struct sk_buff *oskb, void *data) if (oskb->sk == sk) *pflags |= MSG_CONFIRM; - if (sock_queue_rcv_skb_reason(sk, skb, &reason) < 0) + reason = sock_queue_rcv_skb_reason(sk, skb); + if (reason) sk_skb_reason_drop(sk, skb, reason); } diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c index 5c7435fc6483..4bb238eb8606 100644 --- a/net/ceph/messenger_v2.c +++ b/net/ceph/messenger_v2.c @@ -2144,6 +2144,11 @@ static int process_banner_prefix(struct ceph_connection *con) payload_len = ceph_decode_16(&p); dout("%s con %p payload_len %d\n", __func__, con, payload_len); + if (payload_len < sizeof(u64) + sizeof(u64)) { + con->error_msg = "protocol error, bad banner payload len"; + return -EINVAL; + } + return prepare_read_banner_payload(con, payload_len); } diff --git a/net/core/dev.c b/net/core/dev.c index e37c532c92aa..ae5e8dfd2fc2 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -10241,6 +10241,37 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode, netdev_ops_assert_locked(dev); + if (prog) { + enum bpf_xdp_mode other_mode = mode == XDP_MODE_SKB + ? XDP_MODE_DRV : XDP_MODE_SKB; + bool offload = mode == XDP_MODE_HW; + + if (!offload && dev_xdp_prog(dev, other_mode)) { + NL_SET_ERR_MSG(extack, "Native and generic XDP can't be active at the same time"); + return -EEXIST; + } + if (!offload && bpf_prog_is_offloaded(prog->aux)) { + NL_SET_ERR_MSG(extack, "Using offloaded program without HW_MODE flag is not supported"); + return -EINVAL; + } + if (bpf_prog_is_dev_bound(prog->aux) && !bpf_offload_dev_match(prog, dev)) { + NL_SET_ERR_MSG(extack, "Program bound to different device"); + return -EINVAL; + } + if (bpf_prog_is_dev_bound(prog->aux) && mode == XDP_MODE_SKB) { + NL_SET_ERR_MSG(extack, "Can't attach device-bound programs in generic mode"); + return -EINVAL; + } + if (prog->expected_attach_type == BPF_XDP_DEVMAP) { + NL_SET_ERR_MSG(extack, "BPF_XDP_DEVMAP programs can not be attached to a device"); + return -EINVAL; + } + if (prog->expected_attach_type == BPF_XDP_CPUMAP) { + NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device"); + return -EINVAL; + } + } + if (dev->cfg->hds_config == ETHTOOL_TCP_DATA_SPLIT_ENABLED && prog && !prog->aux->xdp_has_frags) { NL_SET_ERR_MSG(extack, "unable to install XDP to device using tcp-data-split"); @@ -10380,38 +10411,10 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack new_prog = link->link.prog; if (new_prog) { - bool offload = mode == XDP_MODE_HW; - enum bpf_xdp_mode other_mode = mode == XDP_MODE_SKB - ? XDP_MODE_DRV : XDP_MODE_SKB; - if ((flags & XDP_FLAGS_UPDATE_IF_NOEXIST) && cur_prog) { NL_SET_ERR_MSG(extack, "XDP program already attached"); return -EBUSY; } - if (!offload && dev_xdp_prog(dev, other_mode)) { - NL_SET_ERR_MSG(extack, "Native and generic XDP can't be active at the same time"); - return -EEXIST; - } - if (!offload && bpf_prog_is_offloaded(new_prog->aux)) { - NL_SET_ERR_MSG(extack, "Using offloaded program without HW_MODE flag is not supported"); - return -EINVAL; - } - if (bpf_prog_is_dev_bound(new_prog->aux) && !bpf_offload_dev_match(new_prog, dev)) { - NL_SET_ERR_MSG(extack, "Program bound to different device"); - return -EINVAL; - } - if (bpf_prog_is_dev_bound(new_prog->aux) && mode == XDP_MODE_SKB) { - NL_SET_ERR_MSG(extack, "Can't attach device-bound programs in generic mode"); - return -EINVAL; - } - if (new_prog->expected_attach_type == BPF_XDP_DEVMAP) { - NL_SET_ERR_MSG(extack, "BPF_XDP_DEVMAP programs can not be attached to a device"); - return -EINVAL; - } - if (new_prog->expected_attach_type == BPF_XDP_CPUMAP) { - NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device"); - return -EINVAL; - } } /* don't call drivers if the effective program didn't change */ @@ -12079,6 +12082,7 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name, free_percpu(dev->pcpu_refcnt); free_dev: #endif + ref_tracker_dir_exit(&dev->refcnt_tracker); kvfree(dev); return NULL; } diff --git a/net/core/filter.c b/net/core/filter.c index 332f4986376f..48af015981ab 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -8189,10 +8189,8 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) case BPF_CGROUP_UNIX_SENDMSG: case BPF_CGROUP_INET4_GETPEERNAME: case BPF_CGROUP_INET6_GETPEERNAME: - case BPF_CGROUP_UNIX_GETPEERNAME: case BPF_CGROUP_INET4_GETSOCKNAME: case BPF_CGROUP_INET6_GETSOCKNAME: - case BPF_CGROUP_UNIX_GETSOCKNAME: return &bpf_sock_addr_setsockopt_proto; default: return NULL; @@ -8212,10 +8210,8 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) case BPF_CGROUP_UNIX_SENDMSG: case BPF_CGROUP_INET4_GETPEERNAME: case BPF_CGROUP_INET6_GETPEERNAME: - case BPF_CGROUP_UNIX_GETPEERNAME: case BPF_CGROUP_INET4_GETSOCKNAME: case BPF_CGROUP_INET6_GETSOCKNAME: - case BPF_CGROUP_UNIX_GETSOCKNAME: return &bpf_sock_addr_getsockopt_proto; default: return NULL; diff --git a/net/core/lwt_bpf.c b/net/core/lwt_bpf.c index 5fffdd4e71ad..953237bca36b 100644 --- a/net/core/lwt_bpf.c +++ b/net/core/lwt_bpf.c @@ -163,10 +163,10 @@ static int bpf_output(struct net *net, struct sock *sk, struct sk_buff *skb) return dst->lwtstate->orig_output(net, sk, skb); } -static int xmit_check_hhlen(struct sk_buff *skb, int hh_len) +static int xmit_check_headroom(struct sk_buff *skb, int hroom) { - if (skb_headroom(skb) < hh_len) { - int nhead = HH_DATA_ALIGN(hh_len - skb_headroom(skb)); + if (skb_headroom(skb) < hroom) { + int nhead = hroom - skb_headroom(skb); if (pskb_expand_head(skb, nhead, 0, GFP_ATOMIC)) return -ENOMEM; @@ -278,7 +278,7 @@ static int bpf_xmit(struct sk_buff *skb) bpf = bpf_lwt_lwtunnel(dst->lwtstate); if (bpf->xmit.prog) { - int hh_len = dst->dev->hard_header_len; + int hroom = LL_RESERVED_SPACE(dst->dev); __be16 proto = skb->protocol; int ret; @@ -294,9 +294,12 @@ static int bpf_xmit(struct sk_buff *skb) return -EINVAL; } /* If the header was expanded, headroom might be too - * small for L2 header to come, expand as needed. + * small for the L2 header to come, expand as needed. + * neigh_hh_output() copies the cached header in + * HH_DATA_MOD aligned chunks, so match the reservation + * made before LWT xmit. */ - ret = xmit_check_hhlen(skb, hh_len); + ret = xmit_check_headroom(skb, hroom); if (unlikely(ret)) return ret; diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c index b0e0f22d7b21..4bece5eda9d1 100644 --- a/net/core/net_namespace.c +++ b/net/core/net_namespace.c @@ -398,14 +398,8 @@ static __net_init void preinit_net_sysctl(struct net *net) } /* init code that must occur even if setup_net() is not called. */ -static __net_init int preinit_net(struct net *net, struct user_namespace *user_ns) +static __net_init void preinit_net(struct net *net, struct user_namespace *user_ns) { - int ret; - - ret = ns_common_init(net); - if (ret) - return ret; - refcount_set(&net->passive, 1); ref_tracker_dir_init(&net->refcnt_tracker, 128, "net_refcnt"); ref_tracker_dir_init(&net->notrefcnt_tracker, 128, "net_notrefcnt"); @@ -426,7 +420,6 @@ static __net_init int preinit_net(struct net *net, struct user_namespace *user_n INIT_LIST_HEAD(&net->ptype_all); INIT_LIST_HEAD(&net->ptype_specific); preinit_net_sysctl(net); - return 0; } /* @@ -531,8 +524,12 @@ void net_passive_dec(struct net *net) if (refcount_dec_and_test(&net->passive)) { kfree(rcu_access_pointer(net->gen)); +#ifdef CONFIG_REF_TRACKER /* There should not be any trackers left there. */ ref_tracker_dir_exit(&net->notrefcnt_tracker); + if (!net->refcnt_tracker.dead) + ref_tracker_dir_exit(&net->refcnt_tracker); +#endif /* Wait for an extra rcu_barrier() before final free. */ llist_add(&net->defer_free_list, &defer_free_list); @@ -567,12 +564,14 @@ struct net *copy_net_ns(u64 flags, goto dec_ucounts; } - rv = preinit_net(net, user_ns); - if (rv < 0) - goto dec_ucounts; + preinit_net(net, user_ns); net->ucounts = ucounts; get_user_ns(user_ns); + rv = ns_common_init(net); + if (rv) + goto put_userns_no_common; + rv = down_read_killable(&pernet_ops_rwsem); if (rv < 0) goto put_userns; @@ -584,6 +583,7 @@ struct net *copy_net_ns(u64 flags, if (rv < 0) { put_userns: ns_common_free(net); +put_userns_no_common: #ifdef CONFIG_KEYS key_remove_domain(net->key_domain); #endif @@ -1281,7 +1281,8 @@ void __init net_ns_init(void) * This currently cannot fail as the initial network namespace * has a static inode number. */ - if (preinit_net(&init_net, &init_user_ns)) + preinit_net(&init_net, &init_user_ns); + if (ns_common_init(&init_net)) panic("Could not preinitialize the initial network namespace"); down_write(&pernet_ops_rwsem); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index b775b6305fb7..d1fa5640a912 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -482,6 +482,13 @@ static int page_pool_register_dma_index(struct page_pool *pool, if (unlikely(!PP_DMA_INDEX_BITS)) goto out; + /* + * Drivers request GFP flags according to both the current context and + * the device constraints, but the XArray entry itself is by no mean + * used by the device, so remove zone/policy flags. + */ + gfp &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM | __GFP_COMP); + if (in_softirq()) err = xa_alloc(&pool->dma_mapped, &id, netmem_to_page(netmem), PP_DMA_INDEX_LIMIT, gfp); @@ -498,29 +505,40 @@ static int page_pool_register_dma_index(struct page_pool *pool, return err; } -static int page_pool_release_dma_index(struct page_pool *pool, - netmem_ref netmem) +static void __page_pool_unmap_netmem_dma(struct page_pool *pool, + netmem_ref netmem) { struct page *old, *page = netmem_to_page(netmem); unsigned long id; + dma_addr_t dma; - if (unlikely(!PP_DMA_INDEX_BITS)) - return 0; - - id = netmem_get_dma_index(netmem); - if (!id) - return -1; + if (!pool->dma_map) + return; - if (in_softirq()) - old = xa_cmpxchg(&pool->dma_mapped, id, page, NULL, 0); - else - old = xa_cmpxchg_bh(&pool->dma_mapped, id, page, NULL, 0); - if (old != page) - return -1; + /* Cache dma_addr before xa_cmpxchg. The scrub path holds no page ref; + * the unref path calls put_page() regardless of cmpxchg outcome, so + * after the cmpxchg we cannot safely touch netmem fields. + */ + dma = page_pool_get_dma_addr_netmem(netmem); - netmem_set_dma_index(netmem, 0); + if (likely(PP_DMA_INDEX_BITS)) { + id = netmem_get_dma_index(netmem); + if (!id) + return; + + if (in_softirq()) + old = xa_cmpxchg(&pool->dma_mapped, + id, page, NULL, 0); + else + old = xa_cmpxchg_bh(&pool->dma_mapped, + id, page, NULL, 0); + if (old != page) + return; + } - return 0; + dma_unmap_page_attrs(pool->p.dev, dma, + PAGE_SIZE << pool->p.order, pool->p.dma_dir, + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING); } static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem, gfp_t gfp) @@ -726,24 +744,16 @@ void page_pool_clear_pp_info(netmem_ref netmem) static __always_inline void __page_pool_release_netmem_dma(struct page_pool *pool, netmem_ref netmem) { - dma_addr_t dma; - + /* Caller must hold a page ref: __page_pool_unmap_netmem_dma() is + * safe without a ref, but the field clears below require it. + */ if (!pool->dma_map) - /* Always account for inflight pages, even if we didn't - * map them - */ return; - if (page_pool_release_dma_index(pool, netmem)) - return; - - dma = page_pool_get_dma_addr_netmem(netmem); - - /* When page is unmapped, it cannot be returned to our pool */ - dma_unmap_page_attrs(pool->p.dev, dma, - PAGE_SIZE << pool->p.order, pool->p.dma_dir, - DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING); + __page_pool_unmap_netmem_dma(pool, netmem); page_pool_set_dma_addr_netmem(netmem, 0); + if (likely(PP_DMA_INDEX_BITS)) + netmem_set_dma_index(netmem, 0); } /* Disconnects a page (from a page_pool). API users can have a need @@ -1169,8 +1179,9 @@ static void page_pool_scrub(struct page_pool *pool) synchronize_net(); } + /* No page ref, dma-unmap only. */ xa_for_each(&pool->dma_mapped, id, ptr) - __page_pool_release_netmem_dma(pool, page_to_netmem((struct page *)ptr)); + __page_pool_unmap_netmem_dma(pool, page_to_netmem((struct page *)ptr)); } /* No more consumers should exist, but producers could still diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 9c9759106822..03b2e96fe6dc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1941,12 +1941,12 @@ int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask) int i, order, psize, new_frags; u32 d_off; - if (skb_shared(skb) || skb_unclone(skb, gfp_mask)) - return -EINVAL; - if (!skb_frags_readable(skb)) return -EFAULT; + if (skb_shared(skb) || skb_unclone(skb, gfp_mask)) + return -EINVAL; + if (!num_frags) goto release; diff --git a/net/core/sock.c b/net/core/sock.c index b69c574c20ad..1b64ac35073d 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -520,32 +520,24 @@ int __sock_queue_rcv_skb(struct sock *sk, struct sk_buff *skb) } EXPORT_SYMBOL(__sock_queue_rcv_skb); -int sock_queue_rcv_skb_reason(struct sock *sk, struct sk_buff *skb, - enum skb_drop_reason *reason) +enum skb_drop_reason +sock_queue_rcv_skb_reason(struct sock *sk, struct sk_buff *skb) { enum skb_drop_reason drop_reason; int err; err = sk_filter_reason(sk, skb, &drop_reason); if (err) - goto out; + return drop_reason; err = __sock_queue_rcv_skb(sk, skb); switch (err) { case -ENOMEM: - drop_reason = SKB_DROP_REASON_SOCKET_RCVBUFF; - break; + return SKB_DROP_REASON_SOCKET_RCVBUFF; case -ENOBUFS: - drop_reason = SKB_DROP_REASON_PROTO_MEM; - break; - default: - drop_reason = SKB_NOT_DROPPED_YET; - break; + return SKB_DROP_REASON_PROTO_MEM; } -out: - if (reason) - *reason = drop_reason; - return err; + return SKB_NOT_DROPPED_YET; } EXPORT_SYMBOL(sock_queue_rcv_skb_reason); diff --git a/net/devlink/netlink_gen.c b/net/devlink/netlink_gen.c index 9fd00977d59e..5ad435aee29d 100644 --- a/net/devlink/netlink_gen.c +++ b/net/devlink/netlink_gen.c @@ -229,7 +229,7 @@ static const struct nla_policy devlink_eswitch_get_nl_policy[DEVLINK_ATTR_DEV_NA static const struct nla_policy devlink_eswitch_set_nl_policy[DEVLINK_ATTR_ESWITCH_ENCAP_MODE + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, }, - [DEVLINK_ATTR_ESWITCH_MODE] = NLA_POLICY_MAX(NLA_U16, 1), + [DEVLINK_ATTR_ESWITCH_MODE] = NLA_POLICY_MAX(NLA_U16, 2), [DEVLINK_ATTR_ESWITCH_INLINE_MODE] = NLA_POLICY_MAX(NLA_U8, 3), [DEVLINK_ATTR_ESWITCH_ENCAP_MODE] = NLA_POLICY_MAX(NLA_U8, 1), }; diff --git a/net/hsr/hsr_device.c b/net/hsr/hsr_device.c index fd2fea25eff0..82b9e94be12f 100644 --- a/net/hsr/hsr_device.c +++ b/net/hsr/hsr_device.c @@ -820,6 +820,8 @@ int hsr_dev_finalize(struct net_device *hsr_dev, struct net_device *slave[2], hsr_del_ports(hsr); err_add_master: hsr_del_self_node(hsr); + hsr_del_nodes(&hsr->node_db); + hsr_del_nodes(&hsr->proxy_node_db); if (unregister) unregister_netdevice(hsr_dev); diff --git a/net/ife/ife.c b/net/ife/ife.c index 7a75947a31e3..2ddf725d3389 100644 --- a/net/ife/ife.c +++ b/net/ife/ife.c @@ -37,7 +37,7 @@ void *ife_encode(struct sk_buff *skb, u16 metalen) * where ORIGDATA = original ethernet header ... */ int hdrm = metalen + IFE_METAHDRLEN; - int total_push = hdrm + skb->dev->hard_header_len; + int total_push = hdrm + ETH_HLEN; struct ifeheadr *ifehdr; struct ethhdr *iethh; /* inner ether header */ int skboff = 0; @@ -50,9 +50,9 @@ void *ife_encode(struct sk_buff *skb, u16 metalen) iethh = (struct ethhdr *) skb->data; __skb_push(skb, total_push); - memcpy(skb->data, iethh, skb->dev->hard_header_len); + memcpy(skb->data, iethh, ETH_HLEN); skb_reset_mac_header(skb); - skboff += skb->dev->hard_header_len; + skboff += ETH_HLEN; /* total metadata length */ ifehdr = (struct ifeheadr *) (skb->data + skboff); @@ -69,12 +69,12 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen) int total_pull; u16 ifehdrln; - if (!pskb_may_pull(skb, skb->dev->hard_header_len + IFE_METAHDRLEN)) + if (!pskb_may_pull(skb, ETH_HLEN + IFE_METAHDRLEN)) return NULL; - ifehdr = (struct ifeheadr *) (skb->data + skb->dev->hard_header_len); + ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN); ifehdrln = ntohs(ifehdr->metalen); - total_pull = skb->dev->hard_header_len + ifehdrln; + total_pull = ETH_HLEN + ifehdrln; if (unlikely(ifehdrln < 2)) return NULL; @@ -82,7 +82,7 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen) if (unlikely(!pskb_may_pull(skb, total_pull + ETH_HLEN))) return NULL; - ifehdr = (struct ifeheadr *)(skb->data + skb->dev->hard_header_len); + ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN); skb_set_mac_header(skb, total_pull); __skb_pull(skb, total_pull); *metalen = ifehdrln - IFE_METAHDRLEN; diff --git a/net/ipv4/inetpeer.c b/net/ipv4/inetpeer.c index 7b1e0a2d6906..17933ac5b428 100644 --- a/net/ipv4/inetpeer.c +++ b/net/ipv4/inetpeer.c @@ -22,6 +22,7 @@ #include #include #include +#include /* * Theory of operations. @@ -53,6 +54,34 @@ */ static struct kmem_cache *peer_cachep __ro_after_init; +static siphash_aligned_key_t inetpeer_hash_key __read_mostly; + +static u64 inetpeer_addr_hash(const struct inetpeer_addr *a) +{ + net_get_random_once(&inetpeer_hash_key, sizeof(inetpeer_hash_key)); + + if (a->family == AF_INET) + return siphash_2u32((__force u32)a->a4.addr, a->a4.vif, + &inetpeer_hash_key); + + return siphash_4u32((__force u32)a->a6.s6_addr32[0], + (__force u32)a->a6.s6_addr32[1], + (__force u32)a->a6.s6_addr32[2], + (__force u32)a->a6.s6_addr32[3], + &inetpeer_hash_key); +} + +static int inetpeer_entry_cmp(u64 dhash, + const struct inetpeer_addr *daddr, + const struct inet_peer *p) +{ + if (dhash < p->hash) + return -1; + if (dhash > p->hash) + return 1; + + return inetpeer_addr_cmp(daddr, &p->daddr); +} void inet_peer_base_init(struct inet_peer_base *bp) { @@ -86,6 +115,7 @@ void __init inet_initpeers(void) /* Called with rcu_read_lock() or base->lock held */ static struct inet_peer *lookup(const struct inetpeer_addr *daddr, + u64 dhash, struct inet_peer_base *base, unsigned int seq, struct inet_peer *gc_stack[], @@ -107,7 +137,7 @@ static struct inet_peer *lookup(const struct inetpeer_addr *daddr, break; parent = next; p = rb_entry(parent, struct inet_peer, rb_node); - cmp = inetpeer_addr_cmp(daddr, &p->daddr); + cmp = inetpeer_entry_cmp(dhash, daddr, p); if (cmp == 0) { now = jiffies; if (READ_ONCE(p->dtime) != now) @@ -172,6 +202,7 @@ struct inet_peer *inet_getpeer(struct inet_peer_base *base, const struct inetpeer_addr *daddr) { struct inet_peer *p, *gc_stack[PEER_MAX_GC]; + u64 dhash = inetpeer_addr_hash(daddr); struct rb_node **pp, *parent; unsigned int gc_cnt, seq; @@ -179,7 +210,7 @@ struct inet_peer *inet_getpeer(struct inet_peer_base *base, * Because of a concurrent writer, we might not find an existing entry. */ seq = read_seqbegin(&base->lock); - p = lookup(daddr, base, seq, NULL, &gc_cnt, &parent, &pp); + p = lookup(daddr, dhash, base, seq, NULL, &gc_cnt, &parent, &pp); if (p) return p; @@ -191,11 +222,12 @@ struct inet_peer *inet_getpeer(struct inet_peer_base *base, write_seqlock_bh(&base->lock); gc_cnt = 0; - p = lookup(daddr, base, seq, gc_stack, &gc_cnt, &parent, &pp); + p = lookup(daddr, dhash, base, seq, gc_stack, &gc_cnt, &parent, &pp); if (!p) { p = kmem_cache_alloc(peer_cachep, GFP_ATOMIC); if (p) { p->daddr = *daddr; + p->hash = dhash; p->dtime = (__u32)jiffies; refcount_set(&p->refcnt, 1); atomic_set(&p->rid, 0); diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c index 92867b439909..6cf95defb01e 100644 --- a/net/ipv4/ipmr.c +++ b/net/ipv4/ipmr.c @@ -102,7 +102,8 @@ static DEFINE_SPINLOCK(mfc_unres_lock); static struct kmem_cache *mrt_cachep __ro_after_init; static struct mr_table *ipmr_new_table(struct net *net, u32 id); -static void ipmr_free_table(struct mr_table *mrt); +static void ipmr_free_table(struct mr_table *mrt, + struct list_head *dev_kill_list); static void ip_mr_forward(struct net *net, struct mr_table *mrt, struct net_device *dev, struct sk_buff *skb, @@ -112,7 +113,8 @@ static int ipmr_cache_report(const struct mr_table *mrt, static void mroute_netlink_event(struct mr_table *mrt, struct mfc_cache *mfc, int cmd); static void igmpmsg_netlink_event(const struct mr_table *mrt, struct sk_buff *pkt); -static void mroute_clean_tables(struct mr_table *mrt, int flags); +static void mroute_clean_tables(struct mr_table *mrt, int flags, + struct list_head *dev_kill_list); static void ipmr_expire_process(struct timer_list *t); #ifdef CONFIG_IP_MROUTE_MULTIPLE_TABLES @@ -149,16 +151,6 @@ static struct mr_table *__ipmr_get_table(struct net *net, u32 id) return NULL; } -static struct mr_table *ipmr_get_table(struct net *net, u32 id) -{ - struct mr_table *mrt; - - rcu_read_lock(); - mrt = __ipmr_get_table(net, id); - rcu_read_unlock(); - return mrt; -} - static int ipmr_fib_lookup(struct net *net, struct flowi4 *flp4, struct mr_table **mrt) { @@ -250,6 +242,7 @@ static const struct fib_rules_ops __net_initconst ipmr_rules_ops_template = { static int __net_init ipmr_rules_init(struct net *net) { struct fib_rules_ops *ops; + LIST_HEAD(dev_kill_list); struct mr_table *mrt; int err; @@ -273,23 +266,22 @@ static int __net_init ipmr_rules_init(struct net *net) return 0; err2: - rtnl_lock(); - ipmr_free_table(mrt); - rtnl_unlock(); + ipmr_free_table(mrt, &dev_kill_list); err1: fib_rules_unregister(ops); return err; } -static void __net_exit ipmr_rules_exit(struct net *net) +static void __net_exit ipmr_rules_exit_rtnl(struct net *net, + struct list_head *dev_kill_list) { struct mr_table *mrt, *next; - ASSERT_RTNL(); list_for_each_entry_safe(mrt, next, &net->ipv4.mr_tables, list) { - list_del(&mrt->list); - ipmr_free_table(mrt); + list_del_rcu(&mrt->list); + ipmr_free_table(mrt, dev_kill_list); } + fib_rules_unregister(net->ipv4.mr_rules_ops); } @@ -310,28 +302,30 @@ bool ipmr_rule_default(const struct fib_rule *rule) } EXPORT_SYMBOL(ipmr_rule_default); #else -#define ipmr_for_each_table(mrt, net) \ - for (mrt = net->ipv4.mrt; mrt; mrt = NULL) - static struct mr_table *ipmr_mr_table_iter(struct net *net, struct mr_table *mrt) { if (!mrt) - return net->ipv4.mrt; + return rcu_dereference(net->ipv4.mrt); return NULL; } -static struct mr_table *ipmr_get_table(struct net *net, u32 id) +static struct mr_table *__ipmr_get_table(struct net *net, u32 id) { - return net->ipv4.mrt; + return rcu_dereference_check(net->ipv4.mrt, + lockdep_rtnl_is_held() || + !rcu_access_pointer(net->ipv4.mrt)); } -#define __ipmr_get_table ipmr_get_table +#define ipmr_for_each_table(mrt, net) \ + for (mrt = __ipmr_get_table(net, 0); mrt; mrt = NULL) static int ipmr_fib_lookup(struct net *net, struct flowi4 *flp4, struct mr_table **mrt) { - *mrt = net->ipv4.mrt; + *mrt = rcu_dereference(net->ipv4.mrt); + if (!*mrt) + return -EAGAIN; return 0; } @@ -342,15 +336,18 @@ static int __net_init ipmr_rules_init(struct net *net) mrt = ipmr_new_table(net, RT_TABLE_DEFAULT); if (IS_ERR(mrt)) return PTR_ERR(mrt); - net->ipv4.mrt = mrt; + + rcu_assign_pointer(net->ipv4.mrt, mrt); return 0; } -static void __net_exit ipmr_rules_exit(struct net *net) +static void __net_exit ipmr_rules_exit_rtnl(struct net *net, + struct list_head *dev_kill_list) { - ASSERT_RTNL(); - ipmr_free_table(net->ipv4.mrt); - net->ipv4.mrt = NULL; + struct mr_table *mrt = rcu_dereference_protected(net->ipv4.mrt, 1); + + RCU_INIT_POINTER(net->ipv4.mrt, NULL); + ipmr_free_table(mrt, dev_kill_list); } static int ipmr_rules_dump(struct net *net, struct notifier_block *nb, @@ -371,6 +368,17 @@ bool ipmr_rule_default(const struct fib_rule *rule) EXPORT_SYMBOL(ipmr_rule_default); #endif +static struct mr_table *ipmr_get_table(struct net *net, u32 id) +{ + struct mr_table *mrt; + + rcu_read_lock(); + mrt = __ipmr_get_table(net, id); + rcu_read_unlock(); + + return mrt; +} + static inline int ipmr_hash_cmp(struct rhashtable_compare_arg *arg, const void *ptr) { @@ -424,17 +432,21 @@ static struct mr_table *ipmr_new_table(struct net *net, u32 id) ipmr_expire_process, ipmr_new_table_set); } -static void ipmr_free_table(struct mr_table *mrt) +static void ipmr_free_table(struct mr_table *mrt, struct list_head *dev_kill_list) { struct net *net = read_pnet(&mrt->net); + LIST_HEAD(ipmr_dev_kill_list); WARN_ON_ONCE(!mr_can_free_table(net)); - timer_shutdown_sync(&mrt->ipmr_expire_timer); mroute_clean_tables(mrt, MRT_FLUSH_VIFS | MRT_FLUSH_VIFS_STATIC | - MRT_FLUSH_MFC | MRT_FLUSH_MFC_STATIC); - rhltable_destroy(&mrt->mfc_hash); - kfree(mrt); + MRT_FLUSH_MFC | MRT_FLUSH_MFC_STATIC, + &ipmr_dev_kill_list); + timer_shutdown_sync(&mrt->ipmr_expire_timer); + mr_table_free(mrt); + + WARN_ON_ONCE(!net_initialized(net) && !list_empty(&ipmr_dev_kill_list)); + list_splice(&ipmr_dev_kill_list, dev_kill_list); } /* Service routines creating virtual interfaces: DVMRP tunnels and PIMREG */ @@ -518,15 +530,16 @@ static netdev_tx_t reg_vif_xmit(struct sk_buff *skb, struct net_device *dev) }; int err; + rcu_read_lock(); err = ipmr_fib_lookup(net, &fl4, &mrt); if (err < 0) { + rcu_read_unlock(); kfree_skb(skb); return err; } DEV_STATS_ADD(dev, tx_bytes, skb->len); DEV_STATS_INC(dev, tx_packets); - rcu_read_lock(); /* Pairs with WRITE_ONCE() in vif_add() and vif_delete() */ ipmr_cache_report(mrt, skb, READ_ONCE(mrt->mroute_reg_vif_num), @@ -1120,12 +1133,19 @@ static int ipmr_cache_report(const struct mr_table *mrt, static int ipmr_cache_unresolved(struct mr_table *mrt, vifi_t vifi, struct sk_buff *skb, struct net_device *dev) { + struct net *net = read_pnet(&mrt->net); const struct iphdr *iph = ip_hdr(skb); - struct mfc_cache *c; + struct mfc_cache *c = NULL; bool found = false; int err; spin_lock_bh(&mfc_unres_lock); + + if (!check_net(net)) { + err = -EINVAL; + goto err; + } + list_for_each_entry(c, &mrt->mfc_unres_queue, _c.list) { if (c->mfc_mcastgrp == iph->daddr && c->mfc_origin == iph->saddr) { @@ -1138,10 +1158,8 @@ static int ipmr_cache_unresolved(struct mr_table *mrt, vifi_t vifi, /* Create a new entry if allowable */ c = ipmr_cache_alloc_unres(); if (!c) { - spin_unlock_bh(&mfc_unres_lock); - - kfree_skb(skb); - return -ENOBUFS; + err = -ENOBUFS; + goto err; } /* Fill in the new cache entry */ @@ -1151,17 +1169,8 @@ static int ipmr_cache_unresolved(struct mr_table *mrt, vifi_t vifi, /* Reflect first query at mrouted. */ err = ipmr_cache_report(mrt, skb, vifi, IGMPMSG_NOCACHE); - - if (err < 0) { - /* If the report failed throw the cache entry - out - Brad Parker - */ - spin_unlock_bh(&mfc_unres_lock); - - ipmr_cache_free(c); - kfree_skb(skb); - return err; - } + if (err < 0) + goto err; atomic_inc(&mrt->cache_resolve_queue_len); list_add(&c->_c.list, &mrt->mfc_unres_queue); @@ -1174,18 +1183,26 @@ static int ipmr_cache_unresolved(struct mr_table *mrt, vifi_t vifi, /* See if we can append the packet */ if (c->_c.mfc_un.unres.unresolved.qlen > 3) { - kfree_skb(skb); + c = NULL; err = -ENOBUFS; - } else { - if (dev) { - skb->dev = dev; - skb->skb_iif = dev->ifindex; - } - skb_queue_tail(&c->_c.mfc_un.unres.unresolved, skb); - err = 0; + goto err; + } + + if (dev) { + skb->dev = dev; + skb->skb_iif = dev->ifindex; } + skb_queue_tail(&c->_c.mfc_un.unres.unresolved, skb); + spin_unlock_bh(&mfc_unres_lock); + return 0; + +err: + spin_unlock_bh(&mfc_unres_lock); + if (c) + ipmr_cache_free(c); + kfree_skb(skb); return err; } @@ -1293,12 +1310,12 @@ static int ipmr_mfc_add(struct net *net, struct mr_table *mrt, } /* Close the multicast socket, and clear the vif tables etc */ -static void mroute_clean_tables(struct mr_table *mrt, int flags) +static void mroute_clean_tables(struct mr_table *mrt, int flags, + struct list_head *dev_kill_list) { struct net *net = read_pnet(&mrt->net); - struct mr_mfc *c, *tmp; struct mfc_cache *cache; - LIST_HEAD(list); + struct mr_mfc *c, *tmp; int i; /* Shut down all active vif entries */ @@ -1308,9 +1325,8 @@ static void mroute_clean_tables(struct mr_table *mrt, int flags) !(flags & MRT_FLUSH_VIFS_STATIC)) || (!(mrt->vif_table[i].flags & VIFF_STATIC) && !(flags & MRT_FLUSH_VIFS))) continue; - vif_delete(mrt, i, 0, &list); + vif_delete(mrt, i, 0, dev_kill_list); } - unregister_netdevice_many(&list); } /* Wipe the cache */ @@ -1330,7 +1346,7 @@ static void mroute_clean_tables(struct mr_table *mrt, int flags) } if (flags & MRT_FLUSH_MFC) { - if (atomic_read(&mrt->cache_resolve_queue_len) != 0) { + if (atomic_read(&mrt->cache_resolve_queue_len) != 0 || !check_net(net)) { spin_lock_bh(&mfc_unres_lock); list_for_each_entry_safe(c, tmp, &mrt->mfc_unres_queue, list) { list_del(&c->list); @@ -1349,9 +1365,11 @@ static void mroute_clean_tables(struct mr_table *mrt, int flags) static void mrtsock_destruct(struct sock *sk) { struct net *net = sock_net(sk); + LIST_HEAD(dev_kill_list); struct mr_table *mrt; rtnl_lock(); + ipmr_for_each_table(mrt, net) { if (sk == rtnl_dereference(mrt->mroute_sk)) { IPV4_DEVCONF_ALL(net, MC_FORWARDING)--; @@ -1360,9 +1378,13 @@ static void mrtsock_destruct(struct sock *sk) NETCONFA_IFINDEX_ALL, net->ipv4.devconf_all); RCU_INIT_POINTER(mrt->mroute_sk, NULL); - mroute_clean_tables(mrt, MRT_FLUSH_VIFS | MRT_FLUSH_MFC); + mroute_clean_tables(mrt, MRT_FLUSH_VIFS | MRT_FLUSH_MFC, + &dev_kill_list); } } + + unregister_netdevice_many(&dev_kill_list); + rtnl_unlock(); } @@ -1485,7 +1507,9 @@ int ip_mroute_setsockopt(struct sock *sk, int optname, sockptr_t optval, sk == rtnl_dereference(mrt->mroute_sk), parent); break; - case MRT_FLUSH: + case MRT_FLUSH: { + LIST_HEAD(dev_kill_list); + if (optlen != sizeof(val)) { ret = -EINVAL; break; @@ -1494,8 +1518,11 @@ int ip_mroute_setsockopt(struct sock *sk, int optname, sockptr_t optval, ret = -EFAULT; break; } - mroute_clean_tables(mrt, val); + + mroute_clean_tables(mrt, val, &dev_kill_list); + unregister_netdevice_many(&dev_kill_list); break; + } /* Control PIM assert. */ case MRT_ASSERT: if (optlen != sizeof(val)) { @@ -3228,6 +3255,7 @@ static void __net_exit ipmr_notifier_exit(struct net *net) /* Setup for IP multicast routing */ static int __net_init ipmr_net_init(struct net *net) { + LIST_HEAD(dev_kill_list); int err; err = ipmr_notifier_init(net); @@ -3253,9 +3281,7 @@ static int __net_init ipmr_net_init(struct net *net) proc_cache_fail: remove_proc_entry("ip_mr_vif", net->proc_net); proc_vif_fail: - rtnl_lock(); - ipmr_rules_exit(net); - rtnl_unlock(); + ipmr_rules_exit_rtnl(net, &dev_kill_list); #endif ipmr_rules_fail: ipmr_notifier_exit(net); @@ -3272,20 +3298,16 @@ static void __net_exit ipmr_net_exit(struct net *net) ipmr_notifier_exit(net); } -static void __net_exit ipmr_net_exit_batch(struct list_head *net_list) +static void __net_exit ipmr_net_exit_rtnl(struct net *net, + struct list_head *dev_kill_list) { - struct net *net; - - rtnl_lock(); - list_for_each_entry(net, net_list, exit_list) - ipmr_rules_exit(net); - rtnl_unlock(); + ipmr_rules_exit_rtnl(net, dev_kill_list); } static struct pernet_operations ipmr_net_ops = { .init = ipmr_net_init, .exit = ipmr_net_exit, - .exit_batch = ipmr_net_exit_batch, + .exit_rtnl = ipmr_net_exit_rtnl, }; static const struct rtnl_msg_handler ipmr_rtnl_msg_handlers[] __initconst = { diff --git a/net/ipv4/ipmr_base.c b/net/ipv4/ipmr_base.c index 28d77d454d44..e6bb023e8dbd 100644 --- a/net/ipv4/ipmr_base.c +++ b/net/ipv4/ipmr_base.c @@ -28,6 +28,20 @@ void vif_device_init(struct vif_device *v, } EXPORT_SYMBOL(vif_device_init); +static void __mr_free_table(struct work_struct *work) +{ + struct mr_table *mrt = container_of(to_rcu_work(work), + struct mr_table, work); + + rhltable_destroy(&mrt->mfc_hash); + kfree(mrt); +} + +void mr_table_free(struct mr_table *mrt) +{ + queue_rcu_work(system_unbound_wq, &mrt->work); +} + struct mr_table * mr_table_alloc(struct net *net, u32 id, struct mr_table_ops *ops, @@ -50,6 +64,8 @@ mr_table_alloc(struct net *net, u32 id, kfree(mrt); return ERR_PTR(err); } + + INIT_RCU_WORK(&mrt->work, __mr_free_table); INIT_LIST_HEAD(&mrt->mfc_cache_list); INIT_LIST_HEAD(&mrt->mfc_unres_queue); diff --git a/net/ipv4/ping.c b/net/ipv4/ping.c index 7f32753a16f4..21ced929cc9c 100644 --- a/net/ipv4/ping.c +++ b/net/ipv4/ping.c @@ -944,7 +944,8 @@ static enum skb_drop_reason __ping_queue_rcv_skb(struct sock *sk, pr_debug("ping_queue_rcv_skb(sk=%p,sk->num=%d,skb=%p)\n", inet_sk(sk), inet_sk(sk)->inet_num, skb); - if (sock_queue_rcv_skb_reason(sk, skb, &reason) < 0) { + reason = sock_queue_rcv_skb_reason(sk, skb); + if (reason) { sk_skb_reason_drop(sk, skb, reason); pr_debug("ping_queue_rcv_skb -> failed\n"); return reason; diff --git a/net/ipv4/raw.c b/net/ipv4/raw.c index c10cf41a365d..5735ae7cba01 100644 --- a/net/ipv4/raw.c +++ b/net/ipv4/raw.c @@ -308,7 +308,8 @@ static int raw_rcv_skb(struct sock *sk, struct sk_buff *skb) /* Charge it to the socket. */ ipv4_pktinfo_prepare(sk, skb, true); - if (sock_queue_rcv_skb_reason(sk, skb, &reason) < 0) { + reason = sock_queue_rcv_skb_reason(sk, skb); + if (reason) { sk_skb_reason_drop(sk, skb, reason); return NET_RX_DROP; } diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index db80b81a74cb..53700d6f6f28 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1132,8 +1132,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) zc = MSG_SPLICE_PAGES; } - if (!sockc_err && sockc.dmabuf_id && - (!(flags & MSG_ZEROCOPY) || !sock_flag(sk, SOCK_ZEROCOPY))) { + if (!sockc_err && sockc.dmabuf_id && (zc != MSG_ZEROCOPY || !binding)) { err = -EINVAL; goto out_err; } @@ -1203,7 +1202,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) trace_tcp_sendmsg_locked(sk, msg, skb, size_goal); - if (copy <= 0 || !tcp_skb_can_collapse_to(skb)) { + if (copy <= 0 || !tcp_skb_can_collapse_to(skb) || + unlikely(skb_frags_readable(skb) != !binding)) { bool first_skb; new_segment: diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index efa533a362e3..8c10b7d18218 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3519,7 +3519,7 @@ int __tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb, int segs) avail_wnd = cur_mss; } - len = cur_mss * segs; + len = cur_mss * (tcp_urg_mode(tp) ? 1 : segs); if (len > avail_wnd) { len = rounddown(avail_wnd, cur_mss); if (!len) diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c index e047a4680ab0..61e61e36f090 100644 --- a/net/ipv6/ip6mr.c +++ b/net/ipv6/ip6mr.c @@ -1063,10 +1063,14 @@ static void ip6mr_cache_resolve(struct net *net, struct mr_table *mrt, static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt, mifi_t mifi, int assert) { + enum skb_drop_reason reason; struct sock *mroute6_sk; struct sk_buff *skb; struct mrt6msg *msg; - int ret; + + mroute6_sk = rcu_dereference(mrt->mroute_sk); + if (!mroute6_sk) + return -EINVAL; #ifdef CONFIG_IPV6_PIMSM_V2 if (assert == MRT6MSG_WHOLEPKT || assert == MRT6MSG_WRMIFWHOLE) @@ -1136,23 +1140,17 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt, skb->ip_summed = CHECKSUM_UNNECESSARY; } - mroute6_sk = rcu_dereference(mrt->mroute_sk); - if (!mroute6_sk) { - kfree_skb(skb); - return -EINVAL; - } - mrt6msg_netlink_event(mrt, skb); /* Deliver to user space multicast routing algorithms */ - ret = sock_queue_rcv_skb(mroute6_sk, skb); + reason = sock_queue_rcv_skb_reason(mroute6_sk, skb); - if (ret < 0) { - net_warn_ratelimited("mroute6: pending queue full, dropping entries\n"); - kfree_skb(skb); + if (reason) { + sk_skb_reason_drop(mroute6_sk, skb, reason); + return -ENOMEM; } - return ret; + return 0; } /* Queue a packet for resolution. It gets locked cache entry! */ diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c index 6cb8e32a3061..b796d84db5ce 100644 --- a/net/ipv6/raw.c +++ b/net/ipv6/raw.c @@ -368,7 +368,8 @@ static inline int rawv6_rcv_skb(struct sock *sk, struct sk_buff *skb) /* Charge it to the socket. */ skb_dst_drop(skb); - if (sock_queue_rcv_skb_reason(sk, skb, &reason) < 0) { + reason = sock_queue_rcv_skb_reason(sk, skb); + if (reason) { sk_skb_reason_drop(sk, skb, reason); return NET_RX_DROP; } diff --git a/net/ipv6/route.c b/net/ipv6/route.c index 2d94a62ed10a..ec6677bf7dea 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -4831,7 +4831,7 @@ static void rt6_upper_bound_set(struct fib6_info *rt, int *weight, int total) { int upper_bound = -1; - if (!rt6_is_dead(rt)) { + if (total && !rt6_is_dead(rt)) { *weight += rt->fib6_nh->fib_nh_weight; upper_bound = DIV_ROUND_CLOSEST_ULL((u64) (*weight) << 31, total) - 1; diff --git a/net/ipv6/xfrm6_input.c b/net/ipv6/xfrm6_input.c index 699a001ac166..c9e208c57a6b 100644 --- a/net/ipv6/xfrm6_input.c +++ b/net/ipv6/xfrm6_input.c @@ -249,7 +249,7 @@ int xfrm6_input_addr(struct sk_buff *skb, xfrm_address_t *daddr, goto drop; } - if (1 + sp->len == XFRM_MAX_DEPTH) { + if (sp->len >= XFRM_MAX_DEPTH) { XFRM_INC_STATS(net, LINUX_MIB_XFRMINBUFFERERROR); goto drop; } diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index dc126f7f030c..0753af2a5128 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -5,6 +5,7 @@ * Copyright (c) 2016 Tom Herbert */ +#include #include #include #include @@ -390,7 +391,9 @@ static int kcm_parse_func_strparser(struct strparser *strp, struct sk_buff *skb) struct bpf_prog *prog = psock->bpf_prog; int res; + rcu_read_lock(); res = bpf_prog_run_pin_on_cpu(prog, skb); + rcu_read_unlock(); return res; } diff --git a/net/mac80211/link.c b/net/mac80211/link.c index 235e370c2b59..cd02053ff506 100644 --- a/net/mac80211/link.c +++ b/net/mac80211/link.c @@ -343,7 +343,8 @@ static int ieee80211_vif_update_links(struct ieee80211_sub_if_data *sdata, link = links[link_id]; ieee80211_link_init(sdata, link_id, &link->data, &link->conf); ieee80211_link_setup(&link->data); - ieee80211_set_wmm_default(&link->data, true, non_sta); + if (sdata->vif.type != NL80211_IFTYPE_AP_VLAN) + ieee80211_set_wmm_default(&link->data, true, non_sta); } if (new_links == 0) diff --git a/net/mac80211/mlme.c b/net/mac80211/mlme.c index 9ec4125c06d1..8d782c8782f9 100644 --- a/net/mac80211/mlme.c +++ b/net/mac80211/mlme.c @@ -7303,7 +7303,7 @@ ieee80211_mgd_check_cross_link_csa(struct ieee80211_sub_if_data *sdata, prof = (void *)sta_profiles[link_id]; prof_elems = ieee802_11_parse_elems(prof->variable + (prof->sta_info_len - 1), - len - + len - sizeof(*prof) - (prof->sta_info_len - 1), false, NULL); diff --git a/net/mac80211/s1g.c b/net/mac80211/s1g.c index 7702cff84d54..1e2f07090a43 100644 --- a/net/mac80211/s1g.c +++ b/net/mac80211/s1g.c @@ -147,7 +147,7 @@ ieee80211_s1g_tx_twt_setup_fail(struct ieee80211_sub_if_data *sdata, drv_twt_teardown_request(sdata->local, sdata, &sta->sta, flowid); - ieee80211_s1g_send_twt_teardown(sdata, mgmt->sa, sdata->vif.addr, + ieee80211_s1g_send_twt_teardown(sdata, mgmt->da, sdata->vif.addr, flowid); } diff --git a/net/mac80211/spectmgmt.c b/net/mac80211/spectmgmt.c index 7422888d3640..080235ca23fc 100644 --- a/net/mac80211/spectmgmt.c +++ b/net/mac80211/spectmgmt.c @@ -227,7 +227,7 @@ int ieee80211_parse_ch_switch_ie(struct ieee80211_sub_if_data *sdata, { enum nl80211_band new_band = current_band; int new_freq; - u8 new_chan_no = 0, new_op_class = 0; + int new_chan_no = -1, new_op_class = -1; struct ieee80211_channel *new_chan; struct cfg80211_chan_def new_chandef = {}; const struct ieee80211_sec_chan_offs_ie *sec_chan_offs; @@ -256,7 +256,7 @@ int ieee80211_parse_ch_switch_ie(struct ieee80211_sub_if_data *sdata, new_op_class = ext_chansw_elem->new_operating_class; if (!ieee80211_operating_class_to_band(new_op_class, &new_band)) { - new_op_class = 0; + new_op_class = -1; if (!unprot_action) sdata_info(sdata, "cannot understand ECSA IE operating class, %d, ignoring\n", @@ -268,14 +268,14 @@ int ieee80211_parse_ch_switch_ie(struct ieee80211_sub_if_data *sdata, } } - if (!new_op_class && elems->ch_switch_ie) { + if (new_op_class < 0 && elems->ch_switch_ie) { new_chan_no = elems->ch_switch_ie->new_ch_num; csa_ie->count = elems->ch_switch_ie->count; csa_ie->mode = elems->ch_switch_ie->mode; } /* nothing here we understand */ - if (!new_chan_no) + if (new_chan_no < 0) return 1; /* Mesh Channel Switch Parameters Element */ @@ -359,7 +359,8 @@ int ieee80211_parse_ch_switch_ie(struct ieee80211_sub_if_data *sdata, get_unaligned_le16(bwi->info.optional); } else if (!wide_bw_chansw_ie || !wbcs_elem_to_chandef(wide_bw_chansw_ie, &new_chandef)) { - if (!ieee80211_operating_class_to_chandef(new_op_class, new_chan, + if (new_op_class < 0 || + !ieee80211_operating_class_to_chandef(new_op_class, new_chan, &new_chandef)) new_chandef = csa_ie->chanreq.oper; } diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c index c125871adb62..89eb340e982b 100644 --- a/net/mac80211/tx.c +++ b/net/mac80211/tx.c @@ -5140,7 +5140,8 @@ static void ieee80211_set_beacon_cntdwn(struct ieee80211_sub_if_data *sdata, if (sdata->vif.type == NL80211_IFTYPE_AP && resp) { u16 *resp_offsets = resp->cntdwn_counter_offsets; - resp->data[resp_offsets[i]] = count; + if (resp_offsets[i]) + resp->data[resp_offsets[i]] = count; } } } diff --git a/net/netfilter/ipvs/ip_vs_ftp.c b/net/netfilter/ipvs/ip_vs_ftp.c index 206c6700e200..d3be352a5ea8 100644 --- a/net/netfilter/ipvs/ip_vs_ftp.c +++ b/net/netfilter/ipvs/ip_vs_ftp.c @@ -103,7 +103,7 @@ static int ip_vs_ftp_get_addrport(char *data, char *data_limit, char *s, c; unsigned char p[6]; char edelim; - __u16 hport; + __u32 hport; int i = 0; if (data_limit - data < plen) { @@ -145,7 +145,11 @@ static int ip_vs_ftp_get_addrport(char *data, char *data_limit, return -1; c = *data; if (isdigit(c)) { - p[i] = p[i]*10 + c - '0'; + unsigned int val = p[i] * 10 + c - '0'; + + if (val > 255) + return -1; + p[i] = val; } else if (c == ',' && i < 5) { i++; p[i] = 0; @@ -223,6 +227,8 @@ static int ip_vs_ftp_get_addrport(char *data, char *data_limit, if (!isdigit(*s)) break; hport = hport * 10 + *s - '0'; + if (hport > 65535) + return -1; } if (s == data_limit || !hport || *s != edelim) return -1; diff --git a/net/netfilter/nf_nat_sip.c b/net/netfilter/nf_nat_sip.c index a1c41defaf22..6b00c81084fe 100644 --- a/net/netfilter/nf_nat_sip.c +++ b/net/netfilter/nf_nat_sip.c @@ -267,12 +267,17 @@ static unsigned int nf_nat_sip(struct sk_buff *skb, unsigned int protoff, SIP_HDR_CONTACT, &in_header, &matchoff, &matchlen, &addr, &port) > 0) { + int old_len = skb->len, delta; + if (!map_addr(skb, protoff, dataoff, dptr, datalen, matchoff, matchlen, &addr, port)) { nf_ct_helper_log(skb, ct, "cannot mangle contact"); return NF_DROP; } + + delta = (int)skb->len - old_len; + coff += delta; } if (!map_sip_addr(skb, protoff, dataoff, dptr, datalen, SIP_HDR_FROM) || diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c index d01f96d32f28..a3a66b6268cd 100644 --- a/net/netfilter/nf_tables_api.c +++ b/net/netfilter/nf_tables_api.c @@ -10774,10 +10774,6 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb) return -EAGAIN; } - err = nft_flow_rule_offload_commit(net); - if (err < 0) - return err; - /* 1. Allocate space for next generation rules_gen_X[] */ list_for_each_entry_safe(trans, next, &nft_net->commit_list, list) { struct nft_table *table = trans->table; @@ -10802,6 +10798,16 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb) } } + /* must be last, so audit and chain blob set up does not leave hardware + * in consistent state. + */ + err = nft_flow_rule_offload_commit(net); + if (err < 0) { + nf_tables_commit_chain_prepare_cancel(net); + nf_tables_commit_audit_free(&adl); + return err; + } + /* step 2. Make rules_gen_X visible to packet path */ list_for_each_entry(table, &nft_net->tables, list) { list_for_each_entry(chain, &table->chains, list) diff --git a/net/netfilter/xt_cgroup.c b/net/netfilter/xt_cgroup.c index 43d2ae2be628..28e6cd51b2fb 100644 --- a/net/netfilter/xt_cgroup.c +++ b/net/netfilter/xt_cgroup.c @@ -23,7 +23,7 @@ MODULE_DESCRIPTION("Xtables: process control group matching"); MODULE_ALIAS("ipt_cgroup"); MODULE_ALIAS("ip6t_cgroup"); -#define NET_CLS_CLASSID_INVALID_MSG "xt_cgroup: classid invalid without net_cls cgroups\n" +#define NET_CLS_CLASSID_INVALID_MSG "classid invalid without net_cls cgroups\n" static int cgroup_mt_check_v0(const struct xt_mtchk_param *par) { @@ -33,7 +33,7 @@ static int cgroup_mt_check_v0(const struct xt_mtchk_param *par) return -EINVAL; if (!IS_ENABLED(CONFIG_CGROUP_NET_CLASSID)) { - pr_info(NET_CLS_CLASSID_INVALID_MSG); + pr_info_ratelimited(NET_CLS_CLASSID_INVALID_MSG); return -EINVAL; } @@ -49,7 +49,7 @@ static int cgroup_mt_check_v1(const struct xt_mtchk_param *par) return -EINVAL; if (!info->has_path && !info->has_classid) { - pr_info("xt_cgroup: no path or classid specified\n"); + pr_info_ratelimited("no path or classid specified\n"); return -EINVAL; } @@ -59,7 +59,7 @@ static int cgroup_mt_check_v1(const struct xt_mtchk_param *par) } if (info->has_classid && !IS_ENABLED(CONFIG_CGROUP_NET_CLASSID)) { - pr_info(NET_CLS_CLASSID_INVALID_MSG); + pr_info_ratelimited(NET_CLS_CLASSID_INVALID_MSG); return -EINVAL; } @@ -89,7 +89,7 @@ static int cgroup_mt_check_v2(const struct xt_mtchk_param *par) return -EINVAL; if (!info->has_path && !info->has_classid) { - pr_info("xt_cgroup: no path or classid specified\n"); + pr_info_ratelimited("no path or classid specified\n"); return -EINVAL; } @@ -99,7 +99,7 @@ static int cgroup_mt_check_v2(const struct xt_mtchk_param *par) } if (info->has_classid && !IS_ENABLED(CONFIG_CGROUP_NET_CLASSID)) { - pr_info(NET_CLS_CLASSID_INVALID_MSG); + pr_info_ratelimited(NET_CLS_CLASSID_INVALID_MSG); return -EINVAL; } diff --git a/net/netfilter/xt_hl.c b/net/netfilter/xt_hl.c index c1a70f8f0441..59e93d97b507 100644 --- a/net/netfilter/xt_hl.c +++ b/net/netfilter/xt_hl.c @@ -6,6 +6,7 @@ * Hop Limit matching module * (C) 2001-2002 Maciej Soltysiak */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include #include @@ -22,6 +23,18 @@ MODULE_LICENSE("GPL"); MODULE_ALIAS("ipt_ttl"); MODULE_ALIAS("ip6t_hl"); +static int ttl_mt_check(const struct xt_mtchk_param *par) +{ + const struct ipt_ttl_info *info = par->matchinfo; + + if (info->mode > IPT_TTL_GT) { + pr_info_ratelimited("Unknown TTL match mode: %d\n", info->mode); + return -EINVAL; + } + + return 0; +} + static bool ttl_mt(const struct sk_buff *skb, struct xt_action_param *par) { const struct ipt_ttl_info *info = par->matchinfo; @@ -41,6 +54,18 @@ static bool ttl_mt(const struct sk_buff *skb, struct xt_action_param *par) return false; } +static int hl_mt6_check(const struct xt_mtchk_param *par) +{ + const struct ip6t_hl_info *info = par->matchinfo; + + if (info->mode > IP6T_HL_GT) { + pr_info_ratelimited("Unknown Hop Limit match mode: %d\n", info->mode); + return -EINVAL; + } + + return 0; +} + static bool hl_mt6(const struct sk_buff *skb, struct xt_action_param *par) { const struct ip6t_hl_info *info = par->matchinfo; @@ -65,6 +90,7 @@ static struct xt_match hl_mt_reg[] __read_mostly = { .name = "ttl", .revision = 0, .family = NFPROTO_IPV4, + .checkentry = ttl_mt_check, .match = ttl_mt, .matchsize = sizeof(struct ipt_ttl_info), .me = THIS_MODULE, @@ -73,6 +99,7 @@ static struct xt_match hl_mt_reg[] __read_mostly = { .name = "hl", .revision = 0, .family = NFPROTO_IPV6, + .checkentry = hl_mt6_check, .match = hl_mt6, .matchsize = sizeof(struct ip6t_hl_info), .me = THIS_MODULE, diff --git a/net/nfc/digital_core.c b/net/nfc/digital_core.c index dae378f1d52b..b18021253224 100644 --- a/net/nfc/digital_core.c +++ b/net/nfc/digital_core.c @@ -127,7 +127,7 @@ static void digital_wq_cmd_complete(struct work_struct *work) mutex_unlock(&ddev->cmd_lock); - if (!IS_ERR(cmd->resp)) + if (!IS_ERR_OR_NULL(cmd->resp)) print_hex_dump_debug("DIGITAL RX: ", DUMP_PREFIX_NONE, 16, 1, cmd->resp->data, cmd->resp->len, false); diff --git a/net/nfc/llcp_core.c b/net/nfc/llcp_core.c index e671483d28ef..41f8e2f7875a 100644 --- a/net/nfc/llcp_core.c +++ b/net/nfc/llcp_core.c @@ -1288,10 +1288,9 @@ static void nfc_llcp_recv_snl(struct nfc_llcp_local *local, { struct nfc_llcp_sock *llcp_sock; u8 dsap, ssap, type, length, tid, sap; - const u8 *tlv; - u16 tlv_len, offset; + const u8 *tlv, *tlv_end; const char *service_name; - size_t service_name_len; + int service_name_len; struct nfc_llcp_sdp_tlv *sdp; HLIST_HEAD(llc_sdres_list); size_t sdres_tlvs_len; @@ -1307,22 +1306,34 @@ static void nfc_llcp_recv_snl(struct nfc_llcp_local *local, return; } + /* + * Walk the SNL TLV list in the linear part of the skb only, + * bounded by skb_tail_pointer(). Each TLV needs a two-byte + * header (type, length) and its declared length must fit before + * the end; this also keeps the walk safe for very short frames. + */ tlv = &skb->data[LLCP_HEADER_SIZE]; - tlv_len = skb->len - LLCP_HEADER_SIZE; - offset = 0; + tlv_end = skb_tail_pointer(skb); sdres_tlvs_len = 0; - while (offset < tlv_len) { + while (tlv + 2 < tlv_end) { type = tlv[0]; length = tlv[1]; + if (tlv + 2 + length > tlv_end) + break; + switch (type) { case LLCP_TLV_SDREQ: + if (length < 1) + break; + tid = tlv[2]; service_name = (char *) &tlv[3]; service_name_len = length - 1; - pr_debug("Looking for %.16s\n", service_name); + pr_debug("Looking for %.*s\n", service_name_len, + service_name); if (service_name_len == strlen("urn:nfc:sn:sdp") && !strncmp(service_name, "urn:nfc:sn:sdp", @@ -1382,6 +1393,9 @@ static void nfc_llcp_recv_snl(struct nfc_llcp_local *local, break; case LLCP_TLV_SDRES: + if (length != 2) + break; + mutex_lock(&local->sdreq_lock); pr_debug("LLCP_TLV_SDRES: searching tid %d\n", tlv[2]); @@ -1410,7 +1424,6 @@ static void nfc_llcp_recv_snl(struct nfc_llcp_local *local, break; } - offset += length + 2; tlv += length + 2; } diff --git a/net/nfc/llcp_sock.c b/net/nfc/llcp_sock.c index 915929cd724f..507447bbceea 100644 --- a/net/nfc/llcp_sock.c +++ b/net/nfc/llcp_sock.c @@ -319,14 +319,22 @@ static int nfc_llcp_getsockopt(struct socket *sock, int level, int optname, if (get_user(len, optlen)) return -EFAULT; - local = llcp_sock->local; - if (!local) - return -ENODEV; + if (len < 0) + return -EINVAL; + + if (len < sizeof(u32)) + return -EINVAL; len = min_t(u32, len, sizeof(u32)); lock_sock(sk); + local = llcp_sock->local; + if (!local) { + release_sock(sk); + return -ENODEV; + } + switch (optname) { case NFC_LLCP_RW: rw = llcp_sock->rw > LLCP_MAX_RW ? local->rw : llcp_sock->rw; diff --git a/net/nfc/nci/data.c b/net/nfc/nci/data.c index 5f98c73db5af..4253edea5c8d 100644 --- a/net/nfc/nci/data.c +++ b/net/nfc/nci/data.c @@ -46,11 +46,11 @@ void nci_data_exchange_complete(struct nci_dev *ndev, struct sk_buff *skb, timer_delete_sync(&ndev->data_timer); clear_bit(NCI_DATA_EXCHANGE_TO, &ndev->flags); - /* Mark the exchange as done before calling the callback. - * The callback (e.g. rawsock_data_exchange_complete) may - * want to immediately queue another data exchange. - */ - clear_bit(NCI_DATA_EXCHANGE, &ndev->flags); + /* Claim completion atomically -- both close and rx_work may race here */ + if (!test_and_clear_bit(NCI_DATA_EXCHANGE, &ndev->flags)) { + kfree_skb(skb); + return; + } if (cb) { /* forward skb to nfc core */ diff --git a/net/nfc/nci/rsp.c b/net/nfc/nci/rsp.c index 165aa4115166..b0ab4f5acbce 100644 --- a/net/nfc/nci/rsp.c +++ b/net/nfc/nci/rsp.c @@ -50,11 +50,27 @@ static u8 nci_core_init_rsp_packet_v1(struct nci_dev *ndev, const struct nci_core_init_rsp_1 *rsp_1 = (void *)skb->data; const struct nci_core_init_rsp_2 *rsp_2; + /* Ensure that the status field can be accessed. */ + if (skb_headlen(skb) < 1) + return NCI_STATUS_SYNTAX_ERROR; + pr_debug("status 0x%x\n", rsp_1->status); if (rsp_1->status != NCI_STATUS_OK) return rsp_1->status; + /* Success response must contain the full fixed-size header */ + if (skb_headlen(skb) < sizeof(*rsp_1)) + return NCI_STATUS_SYNTAX_ERROR; + + /* Ensure the variable-length rf_interfaces array and trailing + * rsp_2 structure are fully contained within the skb. + */ + if (skb_headlen(skb) < sizeof(*rsp_1) + + rsp_1->num_supported_rf_interfaces + + sizeof(*rsp_2)) + return NCI_STATUS_SYNTAX_ERROR; + ndev->nfcc_features = __le32_to_cpu(rsp_1->nfcc_features); ndev->num_supported_rf_interfaces = rsp_1->num_supported_rf_interfaces; @@ -87,15 +103,25 @@ static u8 nci_core_init_rsp_packet_v2(struct nci_dev *ndev, const struct sk_buff *skb) { const struct nci_core_init_rsp_nci_ver2 *rsp = (void *)skb->data; - const u8 *supported_rf_interface = rsp->supported_rf_interfaces; + const u8 *supported_rf_interface; u8 rf_interface_idx = 0; u8 rf_extension_cnt = 0; + /* Ensure that the status field can be accessed. */ + if (skb_headlen(skb) < 1) + return NCI_STATUS_SYNTAX_ERROR; + pr_debug("status %x\n", rsp->status); if (rsp->status != NCI_STATUS_OK) return rsp->status; + /* Success response must contain the full fixed-size header */ + if (skb_headlen(skb) < sizeof(*rsp)) + return NCI_STATUS_SYNTAX_ERROR; + + supported_rf_interface = rsp->supported_rf_interfaces; + ndev->nfcc_features = __le32_to_cpu(rsp->nfcc_features); ndev->num_supported_rf_interfaces = rsp->num_supported_rf_interfaces; @@ -104,13 +130,22 @@ static u8 nci_core_init_rsp_packet_v2(struct nci_dev *ndev, NCI_MAX_SUPPORTED_RF_INTERFACES); while (rf_interface_idx < ndev->num_supported_rf_interfaces) { - ndev->supported_rf_interfaces[rf_interface_idx++] = *supported_rf_interface++; + /* Each entry: [rf_interface_type (1B)] [ext_count (1B)] [ext...] */ + if (supported_rf_interface + 2 > skb_tail_pointer(skb)) + break; + ndev->supported_rf_interfaces[rf_interface_idx] = *supported_rf_interface++; - /* skip rf extension parameters */ rf_extension_cnt = *supported_rf_interface++; + if (supported_rf_interface + rf_extension_cnt > skb_tail_pointer(skb)) + break; + + /* Only count the entry after full validation */ + rf_interface_idx++; supported_rf_interface += rf_extension_cnt; } + ndev->num_supported_rf_interfaces = rf_interface_idx; + ndev->max_logical_connections = rsp->max_logical_connections; ndev->max_routing_table_size = __le16_to_cpu(rsp->max_routing_table_size); diff --git a/net/phonet/pep.c b/net/phonet/pep.c index 5910dcd26555..36c85811fd08 100644 --- a/net/phonet/pep.c +++ b/net/phonet/pep.c @@ -1116,7 +1116,7 @@ static int pep_getsockopt(struct sock *sk, int level, int optname, len = min_t(unsigned int, sizeof(int), len); if (put_user(len, optlen)) return -EFAULT; - if (put_user(val, (int __user *) optval)) + if (copy_to_user(optval, &val, len)) return -EFAULT; return 0; } diff --git a/net/rds/cong.c b/net/rds/cong.c index 8b689ebbd5b5..c85af56e42e7 100644 --- a/net/rds/cong.c +++ b/net/rds/cong.c @@ -256,9 +256,9 @@ void rds_cong_map_updated(struct rds_cong_map *map, uint64_t portmask) map, &map->m_addr); rds_stats_inc(s_cong_update_received); atomic_inc(&rds_cong_generation); - if (waitqueue_active(&map->m_waitq)) + if (wq_has_sleeper(&map->m_waitq)) wake_up(&map->m_waitq); - if (waitqueue_active(&rds_poll_waitq)) + if (wq_has_sleeper(&rds_poll_waitq)) wake_up_all(&rds_poll_waitq); if (portmask && !list_empty(&rds_cong_monitor)) { diff --git a/net/sched/act_api.c b/net/sched/act_api.c index 3578fd5ac4fb..6fc6a986a799 100644 --- a/net/sched/act_api.c +++ b/net/sched/act_api.c @@ -452,7 +452,10 @@ static size_t tcf_action_shared_attrs_size(const struct tc_action *act) /* TCA_STATS_QUEUE */ + nla_total_size_64bit(sizeof(struct gnet_stats_queue)) + nla_total_size(0) /* TCA_ACT_OPTIONS nested */ - + nla_total_size(sizeof(struct tcf_t)); /* TCA_GACT_TM */ + /* TCA_GACT_TM; actions dump their tcf_t with nla_put_64bit(), + * which may emit an extra NLA_PAD attribute. + */ + + nla_total_size_64bit(sizeof(struct tcf_t)); } static size_t tcf_action_full_attrs_size(size_t sz) @@ -1578,7 +1581,7 @@ void tcf_action_update_stats(struct tc_action *a, u64 bytes, u64 packets, if (a->cpu_bstats) { _bstats_update(this_cpu_ptr(a->cpu_bstats), bytes, packets); - this_cpu_ptr(a->cpu_qstats)->drops += drops; + this_cpu_add(a->cpu_qstats->drops, drops); if (hw) _bstats_update(this_cpu_ptr(a->cpu_bstats_hw), diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c index 284800b7d3c5..06d8f78b7368 100644 --- a/net/sched/act_bpf.c +++ b/net/sched/act_bpf.c @@ -76,7 +76,7 @@ TC_INDIRECT_SCOPE int tcf_bpf_act(struct sk_buff *skb, break; case TC_ACT_SHOT: action = filter_res; - qstats_drop_inc(this_cpu_ptr(prog->common.cpu_qstats)); + qstats_cpu_drop_inc(prog->common.cpu_qstats); break; case TC_ACT_UNSPEC: action = prog->tcf_action; @@ -389,6 +389,31 @@ static void tcf_bpf_cleanup(struct tc_action *act) tcf_bpf_cfg_cleanup(&tmp); } +static size_t tcf_bpf_get_fill_size(const struct tc_action *act) +{ + struct tcf_bpf *prog = to_bpf(act); + size_t size = nla_total_size(sizeof(struct tc_act_bpf)); + + /* bpf_ops and bpf_num_ops are published as separate stores under + * tcf_lock, so take it here as tcf_bpf_dump() does. + */ + spin_lock_bh(&prog->tcf_lock); + if (tcf_bpf_is_ebpf(prog)) { + /* TCA_ACT_BPF_NAME */ + size += nla_total_size(ACT_BPF_NAME_LEN + 1); + size += nla_total_size(sizeof(u32)); /* TCA_ACT_BPF_ID */ + size += nla_total_size(BPF_TAG_SIZE); /* TCA_ACT_BPF_TAG */ + } else { + size += nla_total_size(sizeof(u16)); /* TCA_ACT_BPF_OPS_LEN */ + /* TCA_ACT_BPF_OPS */ + size += nla_total_size(prog->bpf_num_ops * + sizeof(struct sock_filter)); + } + spin_unlock_bh(&prog->tcf_lock); + + return size; +} + static struct tc_action_ops act_bpf_ops __read_mostly = { .kind = "bpf", .id = TCA_ID_BPF, @@ -397,6 +422,7 @@ static struct tc_action_ops act_bpf_ops __read_mostly = { .dump = tcf_bpf_dump, .cleanup = tcf_bpf_cleanup, .init = tcf_bpf_init, + .get_fill_size = tcf_bpf_get_fill_size, .size = sizeof(struct tcf_bpf), }; MODULE_ALIAS_NET_ACT("bpf"); diff --git a/net/sched/act_ct.c b/net/sched/act_ct.c index cdf095818f73..cbac27ba76af 100644 --- a/net/sched/act_ct.c +++ b/net/sched/act_ct.c @@ -1655,6 +1655,51 @@ static int tcf_ct_offload_act_setup(struct tc_action *act, void *entry_data, return 0; } +static size_t tcf_ct_get_fill_size(const struct tc_action *act) +{ + const struct tcf_ct_params *p; + size_t size; + + size = nla_total_size(sizeof(struct tc_ct)) /* TCA_CT_PARMS */ + + nla_total_size(sizeof(u16)); /* TCA_CT_ACTION */ + + rcu_read_lock(); + p = rcu_dereference(to_ct(act)->params); + + if (p->ct_action & TCA_CT_ACT_CLEAR) + goto out; + + /* TCA_CT_MARK, TCA_CT_MARK_MASK */ + if (IS_ENABLED(CONFIG_NF_CONNTRACK_MARK)) + size += nla_total_size(sizeof(p->mark)) + + nla_total_size(sizeof(p->mark_mask)); + + /* TCA_CT_LABELS, TCA_CT_LABELS_MASK */ + if (IS_ENABLED(CONFIG_NF_CONNTRACK_LABELS)) + size += nla_total_size(sizeof(p->labels)) + + nla_total_size(sizeof(p->labels_mask)); + + if (IS_ENABLED(CONFIG_NF_CONNTRACK_ZONES)) + size += nla_total_size(sizeof(p->zone)); /* TCA_CT_ZONE */ + + if (p->ct_action & TCA_CT_ACT_NAT) + /* TCA_CT_NAT_IPV6_{MIN,MAX}, the larger of the two address + * variants, plus TCA_CT_NAT_PORT_{MIN,MAX}. + */ + size += 2 * nla_total_size(sizeof(struct in6_addr)) + + 2 * nla_total_size(sizeof(__be16)); + + /* TCA_CT_HELPER_{NAME,FAMILY,PROTO} */ + if (p->helper) + size += nla_total_size(NF_CT_HELPER_NAME_LEN) + + nla_total_size(sizeof(u8)) + + nla_total_size(sizeof(u8)); +out: + rcu_read_unlock(); + + return size; +} + static struct tc_action_ops act_ct_ops = { .kind = "ct", .id = TCA_ID_CT, @@ -1664,6 +1709,7 @@ static struct tc_action_ops act_ct_ops = { .init = tcf_ct_init, .cleanup = tcf_ct_cleanup, .stats_update = tcf_stats_update, + .get_fill_size = tcf_ct_get_fill_size, .offload_act_setup = tcf_ct_offload_act_setup, .size = sizeof(struct tcf_ct), }; diff --git a/net/sched/act_ctinfo.c b/net/sched/act_ctinfo.c index 71efe04d00b5..0517225c144e 100644 --- a/net/sched/act_ctinfo.c +++ b/net/sched/act_ctinfo.c @@ -355,6 +355,16 @@ static void tcf_ctinfo_cleanup(struct tc_action *a) kfree_rcu(cp, rcu); } +static size_t tcf_ctinfo_get_fill_size(const struct tc_action *act) +{ + return nla_total_size(sizeof(struct tc_ctinfo)) /* TCA_CTINFO_ACT */ + + nla_total_size(sizeof(u16)) /* TCA_CTINFO_ZONE */ + /* TCA_CTINFO_PARMS_{DSCP_MASK,DSCP_STATEMASK,CPMARK_MASK} */ + + 3 * nla_total_size(sizeof(u32)) + /* TCA_CTINFO_STATS_{DSCP_SET,DSCP_ERROR,CPMARK_SET} */ + + 3 * nla_total_size_64bit(sizeof(u64)); +} + static struct tc_action_ops act_ctinfo_ops = { .kind = "ctinfo", .id = TCA_ID_CTINFO, @@ -363,6 +373,7 @@ static struct tc_action_ops act_ctinfo_ops = { .dump = tcf_ctinfo_dump, .init = tcf_ctinfo_init, .cleanup= tcf_ctinfo_cleanup, + .get_fill_size = tcf_ctinfo_get_fill_size, .size = sizeof(struct tcf_ctinfo), }; MODULE_ALIAS_NET_ACT("ctinfo"); diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c index 4ad01d4e820d..b0d66ba80baa 100644 --- a/net/sched/act_ife.c +++ b/net/sched/act_ife.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include @@ -723,11 +724,11 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a, tcf_lastuse_update(&ife->tcf_tm); if (skb_at_tc_ingress(skb)) - skb_push(skb, skb->dev->hard_header_len); + skb_push(skb, ETH_HLEN); tlv_data = ife_decode(skb, &metalen); if (unlikely(!tlv_data)) { - qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats)); + qstats_cpu_drop_inc(ife->common.cpu_qstats); return TC_ACT_SHOT; } @@ -740,7 +741,7 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a, curr_data = ife_tlv_meta_decode(tlv_data, ifehdr_end, &mtype, &dlen, NULL); if (!curr_data) { - qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats)); + qstats_cpu_drop_inc(ife->common.cpu_qstats); return TC_ACT_SHOT; } @@ -755,7 +756,7 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a, } if (WARN_ON(tlv_data != ifehdr_end)) { - qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats)); + qstats_cpu_drop_inc(ife->common.cpu_qstats); return TC_ACT_SHOT; } @@ -795,7 +796,7 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a, where ORIGDATA = original ethernet header ... */ u16 metalen = ife_get_sz(skb, p); - int hdrm = metalen + skb->dev->hard_header_len + IFE_METAHDRLEN; + int hdrm = metalen + ETH_HLEN + IFE_METAHDRLEN; unsigned int skboff = 0; int new_len = skb->len + hdrm; bool exceed_mtu = false; @@ -821,12 +822,12 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a, * so lets be conservative.. */ if ((action == TC_ACT_SHOT) || exceed_mtu) { drop: - qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats)); + qstats_cpu_drop_inc(ife->common.cpu_qstats); return TC_ACT_SHOT; } if (skb_at_tc_ingress(skb)) - skb_push(skb, skb->dev->hard_header_len); + skb_push(skb, ETH_HLEN); ife_meta = ife_encode(skb, metalen); if (!ife_meta) @@ -856,11 +857,27 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a, oethh->h_proto = htons(p->eth_type); if (skb_at_tc_ingress(skb)) - skb_pull(skb, skb->dev->hard_header_len); + skb_pull(skb, ETH_HLEN); return action; } +/* IFE encapsulates the original Ethernet header and, on decode, expects to + * find one, so it can only ever work on skbs that carry one. Loopback carries + * Ethernet header as well, so it qualifies here. + * At ingress, also verify that the L2 header about to be pushed back really + * is an Ethernet header because the skb could've been redirected with mirred + * from a non-Ethernet device. + */ +static bool tcf_ife_is_eth_skb(const struct sk_buff *skb) +{ + if (skb->dev->type != ARPHRD_ETHER && + skb->dev->type != ARPHRD_LOOPBACK) + return false; + + return !skb_at_tc_ingress(skb) || skb->mac_len == ETH_HLEN; +} + TC_INDIRECT_SCOPE int tcf_ife_act(struct sk_buff *skb, const struct tc_action *a, struct tcf_result *res) @@ -869,6 +886,13 @@ TC_INDIRECT_SCOPE int tcf_ife_act(struct sk_buff *skb, struct tcf_ife_params *p; int ret; + if (unlikely(!tcf_ife_is_eth_skb(skb))) { + bstats_update(this_cpu_ptr(ife->common.cpu_bstats), skb); + tcf_lastuse_update(&ife->tcf_tm); + qstats_cpu_drop_inc(ife->common.cpu_qstats); + return TC_ACT_SHOT; + } + p = rcu_dereference_bh(ife->params); if (p->flags & IFE_ENCODE) { ret = tcf_ife_encode(skb, a, res, p); @@ -878,6 +902,28 @@ TC_INDIRECT_SCOPE int tcf_ife_act(struct sk_buff *skb, return tcf_ife_decode(skb, a, res); } +static size_t tcf_ife_get_fill_size(const struct tc_action *act) +{ + struct tcf_ife_info *ife = to_ife(act); + const struct tcf_ife_params *p; + struct tcf_meta_info *e; + size_t size = nla_total_size(sizeof(struct tc_ife)) /* TCA_IFE_PARMS */ + + nla_total_size(ETH_ALEN) /* TCA_IFE_DMAC */ + + nla_total_size(ETH_ALEN) /* TCA_IFE_SMAC */ + + nla_total_size(2) /* TCA_IFE_TYPE */ + + nla_total_size(0); /* TCA_IFE_METALST */ + + rcu_read_lock(); + p = rcu_dereference(ife->params); + if (p) { + list_for_each_entry_rcu(e, &p->metalist, metalist) + size += nla_total_size(sizeof(u32)); + } + rcu_read_unlock(); + + return size; +} + static struct tc_action_ops act_ife_ops = { .kind = "ife", .id = TCA_ID_IFE, @@ -886,6 +932,7 @@ static struct tc_action_ops act_ife_ops = { .dump = tcf_ife_dump, .cleanup = tcf_ife_cleanup, .init = tcf_ife_init, + .get_fill_size = tcf_ife_get_fill_size, .size = sizeof(struct tcf_ife_info), }; MODULE_ALIAS_NET_ACT("ife"); diff --git a/net/sched/act_mpls.c b/net/sched/act_mpls.c index 6654011dcd2b..758a13f53213 100644 --- a/net/sched/act_mpls.c +++ b/net/sched/act_mpls.c @@ -123,7 +123,7 @@ TC_INDIRECT_SCOPE int tcf_mpls_act(struct sk_buff *skb, return p->action; drop: - qstats_drop_inc(this_cpu_ptr(m->common.cpu_qstats)); + qstats_cpu_drop_inc(m->common.cpu_qstats); return TC_ACT_SHOT; } diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c index 5aa91fb145a2..7ce50e303575 100644 --- a/net/sched/act_pedit.c +++ b/net/sched/act_pedit.c @@ -622,6 +622,29 @@ static int tcf_pedit_offload_act_setup(struct tc_action *act, void *entry_data, return 0; } +static size_t tcf_pedit_get_fill_size(const struct tc_action *act) +{ + const struct tcf_pedit_parms *parms; + size_t size; + + rcu_read_lock(); + parms = rcu_dereference(to_pedit(act)->parms); + size = nla_total_size(struct_size_t(struct tc_pedit, keys, + parms->tcfp_nkeys)); + if (parms->tcfp_keys_ex) { + /* TCA_PEDIT_KEYS_EX, holding one TCA_PEDIT_KEY_EX nest with a + * HTYPE and a CMD attribute per key. + */ + size += nla_total_size(0) + + parms->tcfp_nkeys * (nla_total_size(0) + + nla_total_size(sizeof(u16)) + + nla_total_size(sizeof(u16))); + } + rcu_read_unlock(); + + return size; +} + static struct tc_action_ops act_pedit_ops = { .kind = "pedit", .id = TCA_ID_PEDIT, @@ -631,6 +654,7 @@ static struct tc_action_ops act_pedit_ops = { .dump = tcf_pedit_dump, .cleanup = tcf_pedit_cleanup, .init = tcf_pedit_init, + .get_fill_size = tcf_pedit_get_fill_size, .offload_act_setup = tcf_pedit_offload_act_setup, .size = sizeof(struct tcf_pedit), }; diff --git a/net/sched/act_police.c b/net/sched/act_police.c index 1ec08627c9a1..e5fb45ad76bf 100644 --- a/net/sched/act_police.c +++ b/net/sched/act_police.c @@ -316,7 +316,7 @@ TC_INDIRECT_SCOPE int tcf_police_act(struct sk_buff *skb, qstats_overlimit_inc(this_cpu_ptr(police->common.cpu_qstats)); inc_drops: if (ret == TC_ACT_SHOT) - qstats_drop_inc(this_cpu_ptr(police->common.cpu_qstats)); + qstats_cpu_drop_inc(police->common.cpu_qstats); end: return ret; } @@ -490,6 +490,17 @@ static int tcf_police_offload_act_setup(struct tc_action *act, void *entry_data, return 0; } +static size_t tcf_police_get_fill_size(const struct tc_action *act) +{ + return nla_total_size(sizeof(struct tc_police)) /* TCA_POLICE_TBF */ + + nla_total_size_64bit(sizeof(u64)) /* TCA_POLICE_RATE64 */ + + nla_total_size_64bit(sizeof(u64)) /* TCA_POLICE_PEAKRATE64 */ + + nla_total_size_64bit(sizeof(u64)) /* TCA_POLICE_PKTRATE64 */ + + nla_total_size_64bit(sizeof(u64)) /* TCA_POLICE_PKTBURST64 */ + + nla_total_size(sizeof(u32)) /* TCA_POLICE_RESULT */ + + nla_total_size(sizeof(u32)); /* TCA_POLICE_AVRATE */ +} + MODULE_AUTHOR("Alexey Kuznetsov"); MODULE_DESCRIPTION("Policing actions"); MODULE_LICENSE("GPL"); @@ -503,6 +514,7 @@ static struct tc_action_ops act_police_ops = { .dump = tcf_police_dump, .init = tcf_police_init, .cleanup = tcf_police_cleanup, + .get_fill_size = tcf_police_get_fill_size, .offload_act_setup = tcf_police_offload_act_setup, .size = sizeof(struct tcf_police), }; diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c index 2ceb4d141b71..44319a159b55 100644 --- a/net/sched/act_sample.c +++ b/net/sched/act_sample.c @@ -315,6 +315,14 @@ static int tcf_sample_offload_act_setup(struct tc_action *act, void *entry_data, return 0; } +static size_t tcf_sample_get_fill_size(const struct tc_action *act) +{ + return nla_total_size(sizeof(struct tc_sample)) /* TCA_SAMPLE_PARMS */ + + nla_total_size(sizeof(u32)) /* TCA_SAMPLE_RATE */ + + nla_total_size(sizeof(u32)) /* TCA_SAMPLE_TRUNC_SIZE */ + + nla_total_size(sizeof(u32)); /* TCA_SAMPLE_PSAMPLE_GROUP */ +} + static struct tc_action_ops act_sample_ops = { .kind = "sample", .id = TCA_ID_SAMPLE, @@ -324,6 +332,7 @@ static struct tc_action_ops act_sample_ops = { .dump = tcf_sample_dump, .init = tcf_sample_init, .cleanup = tcf_sample_cleanup, + .get_fill_size = tcf_sample_get_fill_size, .get_psample_group = tcf_sample_get_group, .offload_act_setup = tcf_sample_offload_act_setup, .size = sizeof(struct tcf_sample), diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c index 5450c1293eb5..8691fe1e67da 100644 --- a/net/sched/act_skbedit.c +++ b/net/sched/act_skbedit.c @@ -86,7 +86,7 @@ TC_INDIRECT_SCOPE int tcf_skbedit_act(struct sk_buff *skb, return params->action; err: - qstats_drop_inc(this_cpu_ptr(d->common.cpu_qstats)); + qstats_cpu_drop_inc(d->common.cpu_qstats); return TC_ACT_SHOT; } diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c index a9e0c1326e2a..caa3cdfd2d3d 100644 --- a/net/sched/act_skbmod.c +++ b/net/sched/act_skbmod.c @@ -38,7 +38,6 @@ TC_INDIRECT_SCOPE int tcf_skbmod_act(struct sk_buff *skb, if (unlikely(p->action == TC_ACT_SHOT)) goto drop; - max_edit_len = skb_mac_header_len(skb); flags = p->flags; /* tcf_skbmod_init() guarantees "flags" to be one of the following: @@ -51,14 +50,19 @@ TC_INDIRECT_SCOPE int tcf_skbmod_act(struct sk_buff *skb, if (flags == SKBMOD_F_ECN) { switch (skb_protocol(skb, true)) { case cpu_to_be16(ETH_P_IP): + max_edit_len = sizeof(struct iphdr); + break; case cpu_to_be16(ETH_P_IPV6): - max_edit_len += skb_network_header_len(skb); + max_edit_len = sizeof(struct ipv6hdr); break; default: goto out; } - } else if (!skb->dev || skb->dev->type != ARPHRD_ETHER) { - goto out; + max_edit_len += skb_network_offset(skb); + } else { + if (!skb->dev || skb->dev->type != ARPHRD_ETHER) + goto out; + max_edit_len = ETH_HLEN; } err = skb_ensure_writable(skb, max_edit_len); diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c index b14807761d82..ff401ace4f3d 100644 --- a/net/sched/act_tunnel_key.c +++ b/net/sched/act_tunnel_key.c @@ -835,6 +835,85 @@ static int tcf_tunnel_key_offload_act_setup(struct tc_action *act, return 0; } +static size_t +tunnel_key_geneve_opts_fill_size(const struct ip_tunnel_info *info) +{ + const u8 *src = ip_tunnel_info_opts(info); + int len = info->options_len; + size_t size = 0; + + while (len > 0) { + const struct geneve_opt *opt = (const struct geneve_opt *)src; + + /* TCA_TUNNEL_KEY_ENC_OPT_GENEVE_{CLASS,TYPE,DATA} */ + size += nla_total_size(2) + + nla_total_size(1) + + nla_total_size(opt->length * 4); + + len -= sizeof(struct geneve_opt) + opt->length * 4; + src += sizeof(struct geneve_opt) + opt->length * 4; + } + + return size; +} + +static size_t tunnel_key_opts_fill_size(const struct ip_tunnel_info *info) +{ + size_t size; + + if (!info->options_len) + return 0; + + /* TCA_TUNNEL_KEY_ENC_OPTS and the per-protocol nest inside it */ + size = nla_total_size(0) + nla_total_size(0); + + if (test_bit(IP_TUNNEL_GENEVE_OPT_BIT, info->key.tun_flags)) { + size += tunnel_key_geneve_opts_fill_size(info); + } else if (test_bit(IP_TUNNEL_VXLAN_OPT_BIT, info->key.tun_flags)) { + /* TCA_TUNNEL_KEY_ENC_OPT_VXLAN_GBP */ + size += nla_total_size(sizeof(u32)); + } else if (test_bit(IP_TUNNEL_ERSPAN_OPT_BIT, info->key.tun_flags)) { + /* TCA_TUNNEL_KEY_ENC_OPT_ERSPAN_{VER,INDEX,DIR,HWID} */ + size += nla_total_size(sizeof(u8)) + + nla_total_size(sizeof(__be32)) + + nla_total_size(sizeof(u8)) + + nla_total_size(sizeof(u8)); + } + + return size; +} + +static size_t tunnel_key_get_fill_size(const struct tc_action *act) +{ + struct tcf_tunnel_key *t = to_tunnel_key(act); + const struct tcf_tunnel_key_params *params; + /* TCA_TUNNEL_KEY_PARMS */ + size_t size = nla_total_size(sizeof(struct tc_tunnel_key)); + + rcu_read_lock(); + params = rcu_dereference(t->params); + if (params->tcft_action == TCA_TUNNEL_KEY_ACT_SET) { + const struct ip_tunnel_info *info = + ¶ms->tcft_enc_metadata->u.tun_info; + + /* In dump order: TCA_TUNNEL_KEY_ENC_KEY_ID, the IPv6 address + * pair (larger than the IPv4 one), ..._ENC_DST_PORT, + * ..._NO_CSUM, ..._NO_FRAG, the options and ..._ENC_{TOS,TTL}. + */ + size += nla_total_size(sizeof(__be32)) + + 2 * nla_total_size(sizeof(struct in6_addr)) + + nla_total_size(sizeof(__be16)) + + nla_total_size(sizeof(u8)) + + nla_total_size(0) + + tunnel_key_opts_fill_size(info) + + nla_total_size(sizeof(u8)) + + nla_total_size(sizeof(u8)); + } + rcu_read_unlock(); + + return size; +} + static struct tc_action_ops act_tunnel_key_ops = { .kind = "tunnel_key", .id = TCA_ID_TUNNEL_KEY, @@ -843,6 +922,7 @@ static struct tc_action_ops act_tunnel_key_ops = { .dump = tunnel_key_dump, .init = tunnel_key_init, .cleanup = tunnel_key_release, + .get_fill_size = tunnel_key_get_fill_size, .offload_act_setup = tcf_tunnel_key_offload_act_setup, .size = sizeof(struct tcf_tunnel_key), }; diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index 21a70cc9940c..d274514cd84d 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -2248,6 +2248,12 @@ static bool is_ingress_or_clsact(struct tcf_block *block, struct Qdisc *q) return tcf_block_shared(block) || (q && !!(q->flags & TCQ_F_INGRESS)); } +enum tcf_tp_insert_state { + TP_NOT_CREATED = 0, /* did not create and insert a new tp */ + TP_CREATED, /* created and inserted a new tp */ + TP_NOT_OWNED, /* created a proto but failed to insert */ +}; + static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, struct netlink_ext_ack *extack) { @@ -2268,12 +2274,12 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, unsigned long cl; void *fh; int err; - int tp_created; + enum tcf_tp_insert_state tp_state; bool rtnl_held = false; u32 flags; replay: - tp_created = 0; + tp_state = TP_NOT_CREATED; err = nlmsg_parse_deprecated(n, sizeof(*t), tca, TCA_MAX, rtm_tca_policy, extack); @@ -2395,13 +2401,15 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, goto errout_tp; } - tp_created = 1; + tp_state = TP_CREATED; tp = tcf_chain_tp_insert_unique(chain, tp_new, protocol, prio, rtnl_held); if (IS_ERR(tp)) { err = PTR_ERR(tp); goto errout_tp; } + if (tp != tp_new) + tp_state = TP_NOT_OWNED; } else { mutex_unlock(&chain->filter_chain_lock); } @@ -2455,13 +2463,13 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, } errout: - if (err && tp_created) + if (err && tp_state == TP_CREATED) tcf_chain_tp_delete_empty(chain, tp, rtnl_held, NULL); errout_tp: if (chain) { if (tp && !IS_ERR(tp)) tcf_proto_put(tp, rtnl_held, NULL); - if (!tp_created) + if (tp_state == TP_NOT_CREATED) tcf_chain_put(chain); } tcf_block_release(q, block, rtnl_held); diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c index 08decb95648c..efe055a9c669 100644 --- a/net/sched/sch_api.c +++ b/net/sched/sch_api.c @@ -610,8 +610,11 @@ void __qdisc_calculate_pkt_len(struct sk_buff *skb, pkt_len <<= stab->szopts.size_log; out: - if (unlikely(pkt_len < 1)) - pkt_len = 1; + /* A size table can inflate qdisc_pkt_len() beyond any real packet + * (via overhead, the data table, or size_log); cap it so deficit + * schedulers such as DRR/ETS terminate their refill loops. + */ + pkt_len = clamp_t(int, pkt_len, 1, QDISC_PKT_LEN_MAX); qdisc_skb_cb(skb)->pkt_len = pkt_len; } diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c index e03fe7096b35..8ed8ac120693 100644 --- a/net/sched/sch_cake.c +++ b/net/sched/sch_cake.c @@ -1829,7 +1829,7 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch, if (ack) { WRITE_ONCE(b->ack_drops, b->ack_drops + 1); - sch->qstats.drops++; + qdisc_qstats_drop(sch); ack_pkt_len = qdisc_pkt_len(ack); b->bytes += ack_pkt_len; q->buffer_used += skb->truesize - ack->truesize; diff --git a/net/sched/sch_codel.c b/net/sched/sch_codel.c index b284f8eda875..1d2caed1bf5b 100644 --- a/net/sched/sch_codel.c +++ b/net/sched/sch_codel.c @@ -206,7 +206,7 @@ static int codel_init(struct Qdisc *sch, struct nlattr *opt, codel_params_init(&q->params); codel_vars_init(&q->vars); codel_stats_init(&q->stats); - q->params.mtu = psched_mtu(qdisc_dev(sch)); + q->params.mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 256, 1 << 20); if (opt) { int err = codel_change(sch, opt, extack); diff --git a/net/sched/sch_fifo.c b/net/sched/sch_fifo.c index e6bfd39ff339..1b6388d50967 100644 --- a/net/sched/sch_fifo.c +++ b/net/sched/sch_fifo.c @@ -19,7 +19,7 @@ static int bfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free) { - if (likely(sch->qstats.backlog + qdisc_pkt_len(skb) <= + if (likely((u64)sch->qstats.backlog + qdisc_pkt_len(skb) <= READ_ONCE(sch->limit))) return qdisc_enqueue_tail(skb, sch); diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c index 5e4193007994..7599a4f7a3e2 100644 --- a/net/sched/sch_fq.c +++ b/net/sched/sch_fq.c @@ -771,8 +771,12 @@ static struct sk_buff *fq_dequeue(struct Qdisc *sch) * f->time_next_packet was set when prior packet was sent, * and current time (@now) can be too late by tens of us. */ - if (f->time_next_packet) - len -= min(len/2, now - f->time_next_packet); + if (f->time_next_packet) { + s64 drift = now - f->time_next_packet; + + if (drift > 0) + len -= min_t(u64, len / 2, drift); + } f->time_next_packet = now + len; } out: @@ -920,7 +924,7 @@ static int fq_resize(struct Qdisc *sch, u32 log) } static const struct netlink_range_validation iq_range = { - .max = INT_MAX, + .max = 1 << 20, }; static const struct nla_policy fq_policy[TCA_FQ_MAX + 1] = { @@ -1046,14 +1050,10 @@ static int fq_change(struct Qdisc *sch, struct nlattr *opt, nla_get_u32(tb[TCA_FQ_FLOW_PLIMIT])); if (tb[TCA_FQ_QUANTUM]) { - u32 quantum = nla_get_u32(tb[TCA_FQ_QUANTUM]); + u32 quantum = clamp_t(u32, nla_get_u32(tb[TCA_FQ_QUANTUM]), + 256, 1 << 20); - if (quantum > 0 && quantum <= (1 << 20)) { - WRITE_ONCE(q->quantum, quantum); - } else { - NL_SET_ERR_MSG_MOD(extack, "invalid quantum"); - err = -EINVAL; - } + WRITE_ONCE(q->quantum, quantum); } if (tb[TCA_FQ_INITIAL_QUANTUM]) @@ -1166,12 +1166,14 @@ static int fq_init(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { struct fq_sched_data *q = qdisc_priv(sch); + u32 mtu; int i, err; sch->limit = 10000; q->flow_plimit = 100; - q->quantum = 2 * psched_mtu(qdisc_dev(sch)); - q->initial_quantum = 10 * psched_mtu(qdisc_dev(sch)); + mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 1, 1 << 20); + q->quantum = clamp_t(u32, 2 * mtu, 256, 1 << 20); + q->initial_quantum = min_t(u32, 10 * mtu, 1 << 20); q->flow_refill_delay = msecs_to_jiffies(40); q->flow_max_rate = ~0UL; q->time_next_delayed_flow = ~0ULL; diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index 3548985bbd42..3b3401dc5c6f 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -176,7 +176,7 @@ static unsigned int fq_codel_drop(struct Qdisc *sch, unsigned int max_packets, flow->cvars.count += i; q->backlogs[idx] -= len; q->memory_usage -= mem; - sch->qstats.drops += i; + __qdisc_qstats_drop(sch, i); sch->qstats.backlog -= len; sch->q.qlen -= i; return idx; @@ -507,6 +507,7 @@ static int fq_codel_init(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { struct fq_codel_sched_data *q = qdisc_priv(sch); + u32 mtu; int i; int err; @@ -514,13 +515,14 @@ static int fq_codel_init(struct Qdisc *sch, struct nlattr *opt, q->flows_cnt = 1024; q->memory_limit = 32 << 20; /* 32 MBytes */ q->drop_batch_size = 64; - q->quantum = psched_mtu(qdisc_dev(sch)); + mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 256, FQ_CODEL_QUANTUM_MAX); + q->quantum = mtu; INIT_LIST_HEAD(&q->new_flows); INIT_LIST_HEAD(&q->old_flows); codel_params_init(&q->cparams); codel_stats_init(&q->cstats); q->cparams.ecn = true; - q->cparams.mtu = psched_mtu(qdisc_dev(sch)); + q->cparams.mtu = mtu; if (opt) { err = fq_codel_change(sch, opt, extack); diff --git a/net/sched/sch_fq_pie.c b/net/sched/sch_fq_pie.c index 0c7b2095f865..4b829d894af1 100644 --- a/net/sched/sch_fq_pie.c +++ b/net/sched/sch_fq_pie.c @@ -427,7 +427,8 @@ static int fq_pie_init(struct Qdisc *sch, struct nlattr *opt, pie_params_init(&q->p_params); sch->limit = 10 * 1024; q->p_params.limit = sch->limit; - q->quantum = psched_mtu(qdisc_dev(sch)); + q->quantum = clamp_t(u32, psched_mtu(qdisc_dev(sch)), + 256, 1 << 20); q->sch = sch; q->ecn_prob = 10; q->flows_cnt = 1024; diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c index 532fde548b88..9185b361dd78 100644 --- a/net/sched/sch_gred.c +++ b/net/sched/sch_gred.c @@ -179,7 +179,7 @@ static int gred_enqueue(struct sk_buff *skb, struct Qdisc *sch, * if no default DP has been configured. This * allows for DP flows to be left untouched. */ - if (likely(sch->qstats.backlog + qdisc_pkt_len(skb) <= + if (likely((u64)sch->qstats.backlog + qdisc_pkt_len(skb) <= sch->limit)) return qdisc_enqueue_tail(skb, sch); else @@ -244,7 +244,7 @@ static int gred_enqueue(struct sk_buff *skb, struct Qdisc *sch, break; } - if (gred_backlog(t, q, sch) + qdisc_pkt_len(skb) <= q->limit) { + if ((u64)gred_backlog(t, q, sch) + qdisc_pkt_len(skb) <= q->limit) { q->backlog += qdisc_pkt_len(skb); return qdisc_enqueue_tail(skb, sch); } @@ -389,7 +389,7 @@ static int gred_offload_dump_stats(struct Qdisc *sch) packets += u64_stats_read(&hw_stats->stats.bstats[i].packets); sch->qstats.qlen += hw_stats->stats.qstats[i].qlen; sch->qstats.backlog += hw_stats->stats.qstats[i].backlog; - sch->qstats.drops += hw_stats->stats.qstats[i].drops; + __qdisc_qstats_drop(sch, hw_stats->stats.qstats[i].drops); sch->qstats.requeues += hw_stats->stats.qstats[i].requeues; sch->qstats.overlimits += hw_stats->stats.qstats[i].overlimits; } diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c index e1c9d1edc8a9..863a5b3e1d09 100644 --- a/net/sched/sch_hhf.c +++ b/net/sched/sch_hhf.c @@ -623,6 +623,10 @@ static int hhf_init(struct Qdisc *sch, struct nlattr *opt, q->hhf_evict_timeout = HZ; /* 1 sec */ q->hhf_non_hh_weight = 2; + if ((int)q->quantum <= 0 || + (u64)q->quantum * q->hhf_non_hh_weight > INT_MAX) + q->quantum = 256; + if (opt) { int err = hhf_change(sch, opt, extack); diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index db97fdb40351..fc8c54039343 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -223,6 +223,7 @@ static struct htb_class *htb_classify(struct sk_buff *skb, struct Qdisc *sch, struct htb_class *cl; struct tcf_result res; struct tcf_proto *tcf; + unsigned int hops = 0; int result; /* allow to select class by setting skb->priority to valid classid; @@ -265,6 +266,10 @@ static struct htb_class *htb_classify(struct sk_buff *skb, struct Qdisc *sch, if (!cl->level) return cl; /* we hit leaf; return it */ + if (++hops > TC_HTB_MAXDEPTH) { + pr_warn_ratelimited("htb: classify loop detected, dropping packet\n"); + return NULL; + } /* we have got inner class; apply inner filter chain */ tcf = rcu_dereference_bh(cl->filter_list); } @@ -632,13 +637,11 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch, } else { return qdisc_drop(skb, sch, to_free); } -#ifdef CONFIG_NET_CLS_ACT } else if (!cl) { if (ret & __NET_XMIT_BYPASS) qdisc_qstats_drop(sch); __qdisc_drop(skb, to_free); return ret; -#endif } else if ((ret = qdisc_enqueue(skb, cl->leaf.q, to_free)) != NET_XMIT_SUCCESS) { if (net_xmit_drop_count(ret)) { diff --git a/net/sched/sch_plug.c b/net/sched/sch_plug.c index cefb65201e17..b60ddfee6a68 100644 --- a/net/sched/sch_plug.c +++ b/net/sched/sch_plug.c @@ -89,7 +89,7 @@ static int plug_enqueue(struct sk_buff *skb, struct Qdisc *sch, { struct plug_sched_data *q = qdisc_priv(sch); - if (likely(sch->qstats.backlog + skb->len <= q->limit)) { + if (likely((u64)sch->qstats.backlog + skb->len <= q->limit)) { if (!q->unplug_indefinite) q->pkts_current_epoch++; return qdisc_enqueue_tail(skb, sch); diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c index 96eb2f122973..645e40f2f402 100644 --- a/net/sched/sch_sfq.c +++ b/net/sched/sch_sfq.c @@ -796,7 +796,8 @@ static int sfq_init(struct Qdisc *sch, struct nlattr *opt, q->tail = NULL; q->divisor = SFQ_DEFAULT_HASH_DIVISOR; q->maxflows = SFQ_DEFAULT_FLOWS; - q->quantum = psched_mtu(qdisc_dev(sch)); + q->quantum = clamp_t(u32, psched_mtu(qdisc_dev(sch)), + 256, 1 << 20); q->perturb_period = 0; get_random_bytes(&q->perturbation, sizeof(q->perturbation)); diff --git a/net/sched/sch_teql.c b/net/sched/sch_teql.c index 4d463ba5ce72..eb424b20027e 100644 --- a/net/sched/sch_teql.c +++ b/net/sched/sch_teql.c @@ -377,6 +377,7 @@ static netdev_tx_t teql_master_xmit(struct sk_buff *skb, struct net_device *dev) nores = 1; break; } + skb->dev = dev; __skb_pull(skb, skb_network_offset(skb)); } while ((q = rcu_dereference(NEXT_SLAVE(q))) != start); diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 6057a7d9718b..e23390d9326b 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -409,13 +409,13 @@ void smc_sk_init(struct net *net, struct sock *sk, int protocol) "sk_lock-AF_SMC", &smc_key); spin_lock_init(&smc->accept_q_lock); spin_lock_init(&smc->conn.send_lock); - sk->sk_prot->hash(sk); mutex_init(&smc->clcsock_release_lock); smc_init_saved_callbacks(smc); smc->limit_smc_hs = net->smc.limit_smc_hs; smc->use_fallback = false; /* assume rdma capability first */ smc->fallback_rsn = 0; smc_close_init(smc); + sk->sk_prot->hash(sk); } static struct sock *smc_sock_alloc(struct net *net, struct socket *sock, diff --git a/net/smc/smc_inet.c b/net/smc/smc_inet.c index a94084b4a498..520b666fdd8f 100644 --- a/net/smc/smc_inet.c +++ b/net/smc/smc_inet.c @@ -15,13 +15,16 @@ #include "smc_inet.h" #include "smc.h" +#include "smc_close.h" static int smc_inet_init_sock(struct sock *sk); +static void smc_inet_destroy_sock(struct sock *sk); static struct proto smc_inet_prot = { .name = "INET_SMC", .owner = THIS_MODULE, .init = smc_inet_init_sock, + .destroy = smc_inet_destroy_sock, .hash = smc_hash_sk, .unhash = smc_unhash_sk, .release_cb = smc_release_cb, @@ -68,6 +71,7 @@ static struct proto smc_inet6_prot = { .name = "INET6_SMC", .owner = THIS_MODULE, .init = smc_inet_init_sock, + .destroy = smc_inet_destroy_sock, .hash = smc_hash_sk, .unhash = smc_unhash_sk, .release_cb = smc_release_cb, @@ -116,6 +120,18 @@ static int smc_inet_init_sock(struct sock *sk) return smc_create_clcsk(net, sk, sk->sk_family); } +static void smc_inet_destroy_sock(struct sock *sk) +{ + /* The sock is hashed and smc_diag dumps dereference smc->clcsock + * without clcsock_release_lock, while sk_common_release() calls + * .destroy before .unhash. Unhash first, as __smc_release() does, + * so no dump can observe the clcsock being released; the second + * unhash is a no-op. + */ + sk->sk_prot->unhash(sk); + smc_clcsock_release(smc_sk(sk)); +} + int __init smc_inet_init(void) { int rc; diff --git a/net/smc/smc_llc.c b/net/smc/smc_llc.c index 51937265203c..65cf07bb30d8 100644 --- a/net/smc/smc_llc.c +++ b/net/smc/smc_llc.c @@ -341,6 +341,7 @@ int smc_llc_flow_initiate(struct smc_link_group *lgr, void smc_llc_flow_stop(struct smc_link_group *lgr, struct smc_llc_flow *flow) { spin_lock_bh(&lgr->llc_flow_lock); + smc_llc_flow_qentry_del(flow); memset(flow, 0, sizeof(*flow)); flow->type = SMC_LLC_FLOW_NONE; spin_unlock_bh(&lgr->llc_flow_lock); @@ -1945,6 +1946,8 @@ static void smc_llc_event_handler(struct smc_llc_qentry *qentry) if (lgr->llc_flow_lcl.type == SMC_LLC_FLOW_REQ_ADD_LINK) { /* server started add_link processing */ + /* free any qentry stashed in REQ_ADD_LINK state */ + smc_llc_flow_qentry_del(&lgr->llc_flow_lcl); lgr->llc_flow_lcl.type = SMC_LLC_FLOW_ADD_LINK; smc_llc_flow_qentry_set(&lgr->llc_flow_lcl, qentry); diff --git a/net/sunrpc/rpcb_clnt.c b/net/sunrpc/rpcb_clnt.c index 53bcca365fb1..2d74c660de66 100644 --- a/net/sunrpc/rpcb_clnt.c +++ b/net/sunrpc/rpcb_clnt.c @@ -490,6 +490,8 @@ static int rpcb_register_inet4(struct sunrpc_net *sn, int result; map->r_addr = rpc_sockaddr2uaddr(sap, GFP_KERNEL); + if (!map->r_addr) + return -ENOMEM; msg->rpc_proc = &rpcb_procedures4[RPCBPROC_UNSET]; if (port != 0) { @@ -516,6 +518,8 @@ static int rpcb_register_inet6(struct sunrpc_net *sn, int result; map->r_addr = rpc_sockaddr2uaddr(sap, GFP_KERNEL); + if (!map->r_addr) + return -ENOMEM; msg->rpc_proc = &rpcb_procedures4[RPCBPROC_UNSET]; if (port != 0) { diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index a4422cbd3254..2946f6b562fb 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -1202,9 +1202,9 @@ static void xs_save_old_callbacks(struct sock_xprt *transport, struct sock *sk) static void xs_restore_old_callbacks(struct sock_xprt *transport, struct sock *sk) { - sk->sk_data_ready = transport->old_data_ready; + WRITE_ONCE(sk->sk_data_ready, transport->old_data_ready); sk->sk_state_change = transport->old_state_change; - sk->sk_write_space = transport->old_write_space; + WRITE_ONCE(sk->sk_write_space, transport->old_write_space); sk->sk_error_report = transport->old_error_report; } @@ -1664,6 +1664,7 @@ static void xs_udp_do_set_buffer_size(struct rpc_xprt *xprt) { struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt); struct sock *sk = transport->inet; + void (*write_space)(struct sock *sock); if (transport->rcvsize) { sk->sk_userlocks |= SOCK_RCVBUF_LOCK; @@ -1672,7 +1673,8 @@ static void xs_udp_do_set_buffer_size(struct rpc_xprt *xprt) if (transport->sndsize) { sk->sk_userlocks |= SOCK_SNDBUF_LOCK; sk->sk_sndbuf = transport->sndsize * xprt->max_reqs * 2; - sk->sk_write_space(sk); + write_space = READ_ONCE(sk->sk_write_space); + write_space(sk); } } @@ -1988,8 +1990,8 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt, xs_save_old_callbacks(transport, sk); sk->sk_user_data = xprt; - sk->sk_data_ready = xs_data_ready; - sk->sk_write_space = xs_udp_write_space; + WRITE_ONCE(sk->sk_data_ready, xs_data_ready); + WRITE_ONCE(sk->sk_write_space, xs_udp_write_space); sk->sk_state_change = xs_local_state_change; sk->sk_error_report = xs_error_report; sk->sk_use_task_frag = false; @@ -2191,8 +2193,8 @@ static void xs_udp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) xs_save_old_callbacks(transport, sk); sk->sk_user_data = xprt; - sk->sk_data_ready = xs_data_ready; - sk->sk_write_space = xs_udp_write_space; + WRITE_ONCE(sk->sk_data_ready, xs_data_ready); + WRITE_ONCE(sk->sk_write_space, xs_udp_write_space); sk->sk_use_task_frag = false; xprt_set_connected(xprt); @@ -2378,9 +2380,9 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock) xs_save_old_callbacks(transport, sk); sk->sk_user_data = xprt; - sk->sk_data_ready = xs_data_ready; + WRITE_ONCE(sk->sk_data_ready, xs_data_ready); sk->sk_state_change = xs_tcp_state_change; - sk->sk_write_space = xs_tcp_write_space; + WRITE_ONCE(sk->sk_write_space, xs_tcp_write_space); sk->sk_error_report = xs_error_report; sk->sk_use_task_frag = false; diff --git a/net/tls/tls_strp.c b/net/tls/tls_strp.c index c72e88317627..c0611c432776 100644 --- a/net/tls/tls_strp.c +++ b/net/tls/tls_strp.c @@ -431,9 +431,10 @@ static int tls_strp_read_copy(struct tls_strparser *strp, bool qshort) return 0; } -static bool tls_strp_check_queue_ok(struct tls_strparser *strp) +static bool tls_strp_check_queue_ok(struct tls_strparser *strp, + unsigned int len) { - unsigned int len = strp->stm.offset + strp->stm.full_len; + unsigned int remaining = strp->stm.offset + len; struct sk_buff *first, *skb; u32 seq; @@ -444,9 +445,9 @@ static bool tls_strp_check_queue_ok(struct tls_strparser *strp) /* Make sure there's no duplicate data in the queue, * and the decrypted status matches. */ - while (skb->len < len) { + while (skb->len < remaining) { seq += skb->len; - len -= skb->len; + remaining -= skb->len; skb = skb->next; if (TCP_SKB_CB(skb)->seq != seq) @@ -525,6 +526,11 @@ static int tls_strp_read_sock(struct tls_strparser *strp) tls_strp_load_anchor_with_queue(strp, inq); if (!strp->stm.full_len) { + if (inq < TLS_HEADER_SIZE) + return tls_strp_read_copy(strp, true); + if (!tls_strp_check_queue_ok(strp, TLS_HEADER_SIZE)) + return tls_strp_read_copy(strp, false); + sz = tls_rx_msg_size(strp, strp->anchor); if (sz < 0) return sz; @@ -535,7 +541,7 @@ static int tls_strp_read_sock(struct tls_strparser *strp) return tls_strp_read_copy(strp, true); } - if (!tls_strp_check_queue_ok(strp)) + if (!tls_strp_check_queue_ok(strp, strp->stm.full_len)) return tls_strp_read_copy(strp, false); WRITE_ONCE(strp->msg_ready, 1); diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c index d007ce89d19b..4ebf1cab2d97 100644 --- a/net/vmw_vsock/af_vsock.c +++ b/net/vmw_vsock/af_vsock.c @@ -1716,12 +1716,10 @@ static int vsock_connect(struct socket *sock, struct sockaddr *addr, prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE); } - if (sk->sk_err) { - err = -sk->sk_err; + err = sock_error(sk); + if (err) { sk->sk_state = TCP_CLOSE; sock->state = SS_UNCONNECTED; - } else { - err = 0; } out_wait: @@ -1760,10 +1758,10 @@ static int vsock_accept(struct socket *sock, struct socket *newsock, * created upon connection establishment. */ timeout = sock_rcvtimeo(listener, arg->flags & O_NONBLOCK); - prepare_to_wait(sk_sleep(listener), &wait, TASK_INTERRUPTIBLE); while ((connected = vsock_dequeue_accept(listener)) == NULL && - listener->sk_err == 0) { + timeout != 0) { + prepare_to_wait(sk_sleep(listener), &wait, TASK_INTERRUPTIBLE); release_sock(listener); timeout = schedule_timeout(timeout); finish_wait(sk_sleep(listener), &wait); @@ -1772,19 +1770,12 @@ static int vsock_accept(struct socket *sock, struct socket *newsock, if (signal_pending(current)) { err = sock_intr_errno(timeout); goto out; - } else if (timeout == 0) { - err = -EAGAIN; - goto out; } - - prepare_to_wait(sk_sleep(listener), &wait, TASK_INTERRUPTIBLE); } - finish_wait(sk_sleep(listener), &wait); - - if (listener->sk_err) - err = -listener->sk_err; - if (connected) { + if (!connected) { + err = -EAGAIN; + } else { sk_acceptq_removed(listener); lock_sock_nested(connected, SINGLE_DEPTH_NESTING); diff --git a/net/wireless/core.c b/net/wireless/core.c index efbf7b304634..6a9ba151b09f 100644 --- a/net/wireless/core.c +++ b/net/wireless/core.c @@ -5,7 +5,7 @@ * Copyright 2006-2010 Johannes Berg * Copyright 2013-2014 Intel Mobile Communications GmbH * Copyright 2015-2017 Intel Deutschland GmbH - * Copyright (C) 2018-2025 Intel Corporation + * Copyright (C) 2018-2026 Intel Corporation */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt @@ -234,6 +234,7 @@ void cfg80211_stop_p2p_device(struct cfg80211_registered_device *rdev, if (!wdev_running(wdev)) return; + cfg80211_pmsr_wdev_down(wdev); rdev_stop_p2p_device(rdev, wdev); wdev->is_running = false; @@ -251,6 +252,8 @@ void cfg80211_stop_p2p_device(struct cfg80211_registered_device *rdev, void cfg80211_stop_nan(struct cfg80211_registered_device *rdev, struct wireless_dev *wdev) { + struct cfg80211_nan_local_sched empty_sched = {}; + lockdep_assert_held(&rdev->wiphy.mtx); if (WARN_ON(wdev->iftype != NL80211_IFTYPE_NAN)) @@ -259,12 +262,64 @@ void cfg80211_stop_nan(struct cfg80211_registered_device *rdev, if (!wdev_running(wdev)) return; + cfg80211_pmsr_wdev_down(wdev); + + /* + * If there is a scheduled update pending, mark it as canceled, so the + * empty schedule will be accepted + */ + wdev->u.nan.sched_update_pending = false; + + /* Unschedule all */ + cfg80211_nan_set_local_schedule(rdev, wdev, &empty_sched); + rdev_stop_nan(rdev, wdev); wdev->is_running = false; rdev->opencount--; } +int cfg80211_nan_set_local_schedule(struct cfg80211_registered_device *rdev, + struct wireless_dev *wdev, + struct cfg80211_nan_local_sched *sched) +{ + int ret; + + lockdep_assert_held(&rdev->wiphy.mtx); + + if (wdev->iftype != NL80211_IFTYPE_NAN || !wdev_running(wdev)) + return -EINVAL; + + if (wdev->u.nan.sched_update_pending) + return -EBUSY; + + ret = rdev_nan_set_local_sched(rdev, wdev, sched); + if (ret) + return ret; + + wdev->u.nan.sched_update_pending = sched->deferred; + + kfree(wdev->u.nan.chandefs); + wdev->u.nan.chandefs = NULL; + wdev->u.nan.n_channels = 0; + + if (!sched->n_channels) + return 0; + + wdev->u.nan.chandefs = kcalloc(sched->n_channels, + sizeof(*wdev->u.nan.chandefs), + GFP_KERNEL); + if (!wdev->u.nan.chandefs) + return -ENOMEM; + + for (int i = 0; i < sched->n_channels; i++) + wdev->u.nan.chandefs[i] = sched->nan_channels[i].chandef; + + wdev->u.nan.n_channels = sched->n_channels; + + return 0; +} + void cfg80211_shutdown_all_interfaces(struct wiphy *wiphy) { struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy); diff --git a/net/wireless/core.h b/net/wireless/core.h index 9bfd39af1742..1086aa1a44f4 100644 --- a/net/wireless/core.h +++ b/net/wireless/core.h @@ -545,6 +545,10 @@ void cfg80211_stop_p2p_device(struct cfg80211_registered_device *rdev, void cfg80211_stop_nan(struct cfg80211_registered_device *rdev, struct wireless_dev *wdev); +int cfg80211_nan_set_local_schedule(struct cfg80211_registered_device *rdev, + struct wireless_dev *wdev, + struct cfg80211_nan_local_sched *sched); + struct cfg80211_internal_bss * cfg80211_bss_update(struct cfg80211_registered_device *rdev, struct cfg80211_internal_bss *tmp, diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c index 776dcf8835d7..232218818ed7 100644 --- a/net/wireless/nl80211.c +++ b/net/wireless/nl80211.c @@ -332,6 +332,62 @@ static int validate_nan_cluster_id(const struct nlattr *attr, return 0; } +static int validate_nan_avail_blob(const struct nlattr *attr, + struct netlink_ext_ack *extack) +{ + const u8 *data = nla_data(attr); + unsigned int len = nla_len(attr); + u16 attr_len; + + /* Need at least: Attr ID (1) + Length (2) */ + if (len < 3) { + NL_SET_ERR_MSG_FMT(extack, + "NAN Availability: Too short (need at least 3 bytes, have %u)", + len); + return -EINVAL; + } + + if (data[0] != 0x12) { + NL_SET_ERR_MSG_FMT(extack, + "NAN Availability: Invalid Attribute ID 0x%02x (expected 0x12)", + data[0]); + return -EINVAL; + } + + attr_len = get_unaligned_le16(&data[1]); + + if (attr_len != len - 3) { + NL_SET_ERR_MSG_FMT(extack, + "NAN Availability: Length field (%u) doesn't match data length (%u)", + attr_len, len - 3); + return -EINVAL; + } + + return 0; +} + +static int validate_uhr_capa(const struct nlattr *attr, + struct netlink_ext_ack *extack) +{ + const u8 *data = nla_data(attr); + unsigned int len = nla_len(attr); + + if (!ieee80211_uhr_capa_size_ok(data, len, false)) + return -EINVAL; + return 0; +} + +static int validate_uhr_operation(const struct nlattr *attr, + struct netlink_ext_ack *extack) +{ + const u8 *data = nla_data(attr); + unsigned int len = nla_len(attr); + + if (!ieee80211_uhr_oper_size_ok(data, len, false)) + return -EINVAL; + return 0; +} + /* policy for the attributes */ static const struct nla_policy nl80211_policy[NUM_NL80211_ATTR]; @@ -934,6 +990,20 @@ static const struct nla_policy nl80211_policy[NUM_NL80211_ATTR] = { NLA_POLICY_NESTED(nl80211_s1g_short_beacon), [NL80211_ATTR_BSS_PARAM] = { .type = NLA_FLAG }, [NL80211_ATTR_S1G_PRIMARY_2MHZ] = { .type = NLA_FLAG }, + [NL80211_ATTR_EPP_PEER] = { .type = NLA_FLAG }, + [NL80211_ATTR_UHR_CAPABILITY] = + NLA_POLICY_VALIDATE_FN(NLA_BINARY, validate_uhr_capa, 255), + [NL80211_ATTR_DISABLE_UHR] = { .type = NLA_FLAG }, + [NL80211_ATTR_UHR_OPERATION] = + NLA_POLICY_VALIDATE_FN(NLA_BINARY, validate_uhr_operation), + [NL80211_ATTR_NAN_CHANNEL] = NLA_POLICY_NESTED(nl80211_policy), + [NL80211_ATTR_NAN_CHANNEL_ENTRY] = NLA_POLICY_EXACT_LEN(6), + [NL80211_ATTR_NAN_RX_NSS] = { .type = NLA_U8 }, + [NL80211_ATTR_NAN_TIME_SLOTS] = + NLA_POLICY_EXACT_LEN(CFG80211_NAN_SCHED_NUM_TIME_SLOTS), + [NL80211_ATTR_NAN_AVAIL_BLOB] = + NLA_POLICY_VALIDATE_FN(NLA_BINARY, validate_nan_avail_blob), + [NL80211_ATTR_NAN_SCHED_DEFERRED] = { .type = NLA_FLAG }, }; /* policy for the key attributes */ @@ -1316,6 +1386,12 @@ static int nl80211_msg_put_channel(struct sk_buff *msg, struct wiphy *wiphy, if ((chan->flags & IEEE80211_CHAN_NO_16MHZ) && nla_put_flag(msg, NL80211_FREQUENCY_ATTR_NO_16MHZ)) goto nla_put_failure; + if ((chan->flags & IEEE80211_CHAN_S1G_NO_PRIMARY) && + nla_put_flag(msg, NL80211_FREQUENCY_ATTR_S1G_NO_PRIMARY)) + goto nla_put_failure; + if ((chan->flags & IEEE80211_CHAN_NO_UHR) && + nla_put_flag(msg, NL80211_FREQUENCY_ATTR_NO_UHR)) + goto nla_put_failure; } if (nla_put_u32(msg, NL80211_FREQUENCY_ATTR_MAX_TX_POWER, @@ -1949,6 +2025,7 @@ nl80211_send_iftype_data(struct sk_buff *msg, { const struct ieee80211_sta_he_cap *he_cap = &iftdata->he_cap; const struct ieee80211_sta_eht_cap *eht_cap = &iftdata->eht_cap; + const struct ieee80211_sta_uhr_cap *uhr_cap = &iftdata->uhr_cap; if (nl80211_put_iftypes(msg, NL80211_BAND_IFTYPE_ATTR_IFTYPES, iftdata->types_mask)) @@ -2000,6 +2077,14 @@ nl80211_send_iftype_data(struct sk_buff *msg, return -ENOBUFS; } + if (uhr_cap->has_uhr) { + if (nla_put(msg, NL80211_BAND_IFTYPE_ATTR_UHR_CAP_MAC, + sizeof(uhr_cap->mac), &uhr_cap->mac) || + nla_put(msg, NL80211_BAND_IFTYPE_ATTR_UHR_CAP_PHY, + sizeof(uhr_cap->phy), &uhr_cap->phy)) + return -ENOBUFS; + } + if (sband->band == NL80211_BAND_6GHZ && nla_put(msg, NL80211_BAND_IFTYPE_ATTR_HE_6GHZ_CAPA, sizeof(iftdata->he_6ghz_capa), @@ -3513,11 +3598,10 @@ static bool nl80211_can_set_dev_channel(struct wireless_dev *wdev) } static int _nl80211_parse_chandef(struct cfg80211_registered_device *rdev, - struct genl_info *info, bool monitor, + struct netlink_ext_ack *extack, + struct nlattr **attrs, bool monitor, struct cfg80211_chan_def *chandef) { - struct netlink_ext_ack *extack = info->extack; - struct nlattr **attrs = info->attrs; u32 control_freq; if (!attrs[NL80211_ATTR_WIPHY_FREQ]) { @@ -3527,10 +3611,10 @@ static int _nl80211_parse_chandef(struct cfg80211_registered_device *rdev, } control_freq = MHZ_TO_KHZ( - nla_get_u32(info->attrs[NL80211_ATTR_WIPHY_FREQ])); - if (info->attrs[NL80211_ATTR_WIPHY_FREQ_OFFSET]) + nla_get_u32(attrs[NL80211_ATTR_WIPHY_FREQ])); + if (attrs[NL80211_ATTR_WIPHY_FREQ_OFFSET]) control_freq += - nla_get_u32(info->attrs[NL80211_ATTR_WIPHY_FREQ_OFFSET]); + nla_get_u32(attrs[NL80211_ATTR_WIPHY_FREQ_OFFSET]); memset(chandef, 0, sizeof(*chandef)); chandef->chan = ieee80211_get_channel_khz(&rdev->wiphy, control_freq); @@ -3598,40 +3682,43 @@ static int _nl80211_parse_chandef(struct cfg80211_registered_device *rdev, attrs[NL80211_ATTR_S1G_PRIMARY_2MHZ]); } - if (info->attrs[NL80211_ATTR_WIPHY_EDMG_CHANNELS]) { + if (attrs[NL80211_ATTR_WIPHY_EDMG_CHANNELS]) { chandef->edmg.channels = - nla_get_u8(info->attrs[NL80211_ATTR_WIPHY_EDMG_CHANNELS]); + nla_get_u8(attrs[NL80211_ATTR_WIPHY_EDMG_CHANNELS]); - if (info->attrs[NL80211_ATTR_WIPHY_EDMG_BW_CONFIG]) + if (attrs[NL80211_ATTR_WIPHY_EDMG_BW_CONFIG]) chandef->edmg.bw_config = - nla_get_u8(info->attrs[NL80211_ATTR_WIPHY_EDMG_BW_CONFIG]); + nla_get_u8(attrs[NL80211_ATTR_WIPHY_EDMG_BW_CONFIG]); } else { chandef->edmg.bw_config = 0; chandef->edmg.channels = 0; } - if (info->attrs[NL80211_ATTR_PUNCT_BITMAP]) { + if (attrs[NL80211_ATTR_PUNCT_BITMAP]) { chandef->punctured = - nla_get_u32(info->attrs[NL80211_ATTR_PUNCT_BITMAP]); + nla_get_u32(attrs[NL80211_ATTR_PUNCT_BITMAP]); if (chandef->punctured && !wiphy_ext_feature_isset(&rdev->wiphy, NL80211_EXT_FEATURE_PUNCT)) { - NL_SET_ERR_MSG(extack, - "driver doesn't support puncturing"); + NL_SET_ERR_MSG_ATTR(extack, + attrs[NL80211_ATTR_WIPHY_FREQ], + "driver doesn't support puncturing"); return -EINVAL; } } if (!cfg80211_chandef_valid(chandef)) { - NL_SET_ERR_MSG(extack, "invalid channel definition"); + NL_SET_ERR_MSG_ATTR(extack, attrs[NL80211_ATTR_WIPHY_FREQ], + "invalid channel definition"); return -EINVAL; } if (!_cfg80211_chandef_usable(&rdev->wiphy, chandef, IEEE80211_CHAN_DISABLED, monitor ? IEEE80211_CHAN_CAN_MONITOR : 0)) { - NL_SET_ERR_MSG(extack, "(extension) channel is disabled"); + NL_SET_ERR_MSG_ATTR(extack, attrs[NL80211_ATTR_WIPHY_FREQ], + "(extension) channel is disabled"); return -EINVAL; } @@ -3646,10 +3733,11 @@ static int _nl80211_parse_chandef(struct cfg80211_registered_device *rdev, } int nl80211_parse_chandef(struct cfg80211_registered_device *rdev, - struct genl_info *info, + struct netlink_ext_ack *extack, + struct nlattr **attrs, struct cfg80211_chan_def *chandef) { - return _nl80211_parse_chandef(rdev, info, false, chandef); + return _nl80211_parse_chandef(rdev, extack, attrs, false, chandef); } static int __nl80211_set_channel(struct cfg80211_registered_device *rdev, @@ -3676,7 +3764,7 @@ static int __nl80211_set_channel(struct cfg80211_registered_device *rdev, link_id = 0; } - result = _nl80211_parse_chandef(rdev, info, + result = _nl80211_parse_chandef(rdev, info->extack, info->attrs, iftype == NL80211_IFTYPE_MONITOR, &chandef); if (result) @@ -6438,6 +6526,7 @@ static int nl80211_calculate_ap_params(struct cfg80211_ap_settings *params) cap->datalen - 1)) return -EINVAL; } + return 0; } @@ -6561,6 +6650,9 @@ static int nl80211_validate_ap_phy_operation(struct cfg80211_ap_settings *params (channel->flags & IEEE80211_CHAN_NO_EHT)) return -EOPNOTSUPP; + if (params->uhr_oper && (channel->flags & IEEE80211_CHAN_NO_UHR)) + return -EOPNOTSUPP; + return 0; } @@ -6742,7 +6834,8 @@ static int nl80211_start_ap(struct sk_buff *skb, struct genl_info *info) } if (info->attrs[NL80211_ATTR_WIPHY_FREQ]) { - err = nl80211_parse_chandef(rdev, info, ¶ms->chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, + ¶ms->chandef); if (err) goto out; } else if (wdev->valid_links) { @@ -6859,6 +6952,9 @@ static int nl80211_start_ap(struct sk_buff *skb, struct genl_info *info) if (err) goto out; + if (info->attrs[NL80211_ATTR_UHR_OPERATION]) + params->uhr_oper = nla_data(info->attrs[NL80211_ATTR_UHR_OPERATION]); + err = nl80211_validate_ap_phy_operation(params); if (err) goto out; @@ -7143,7 +7239,8 @@ bool nl80211_put_sta_rate(struct sk_buff *msg, struct rate_info *info, int attr) break; case RATE_INFO_BW_EHT_RU: rate_flg = 0; - WARN_ON(!(info->flags & RATE_INFO_FLAGS_EHT_MCS)); + WARN_ON(!(info->flags & RATE_INFO_FLAGS_EHT_MCS) && + !(info->flags & RATE_INFO_FLAGS_UHR_MCS)); break; } @@ -7196,6 +7293,23 @@ bool nl80211_put_sta_rate(struct sk_buff *msg, struct rate_info *info, int attr) nla_put_u8(msg, NL80211_RATE_INFO_EHT_RU_ALLOC, info->eht_ru_alloc)) return false; + } else if (info->flags & RATE_INFO_FLAGS_UHR_MCS) { + if (nla_put_u8(msg, NL80211_RATE_INFO_UHR_MCS, info->mcs)) + return false; + if (nla_put_u8(msg, NL80211_RATE_INFO_EHT_NSS, info->nss)) + return false; + if (nla_put_u8(msg, NL80211_RATE_INFO_EHT_GI, info->eht_gi)) + return false; + if (info->bw == RATE_INFO_BW_EHT_RU && + nla_put_u8(msg, NL80211_RATE_INFO_EHT_RU_ALLOC, + info->eht_ru_alloc)) + return false; + if (info->flags & RATE_INFO_FLAGS_UHR_ELR_MCS && + nla_put_flag(msg, NL80211_RATE_INFO_UHR_ELR)) + return false; + if (info->flags & RATE_INFO_FLAGS_UHR_IM && + nla_put_flag(msg, NL80211_RATE_INFO_UHR_IM)) + return false; } nla_nest_end(msg, rate); @@ -8069,7 +8183,8 @@ int cfg80211_check_station_change(struct wiphy *wiphy, if (params->ext_capab || params->link_sta_params.ht_capa || params->link_sta_params.vht_capa || params->link_sta_params.he_capa || - params->link_sta_params.eht_capa) + params->link_sta_params.eht_capa || + params->link_sta_params.uhr_capa) return -EINVAL; if (params->sta_flags_mask & BIT(NL80211_STA_FLAG_SPP_AMSDU)) return -EINVAL; @@ -8289,6 +8404,16 @@ static int nl80211_set_station_tdls(struct genl_info *info, } } + if (info->attrs[NL80211_ATTR_UHR_CAPABILITY]) { + if (!params->link_sta_params.eht_capa) + return -EINVAL; + + params->link_sta_params.uhr_capa = + nla_data(info->attrs[NL80211_ATTR_UHR_CAPABILITY]); + params->link_sta_params.uhr_capa_len = + nla_len(info->attrs[NL80211_ATTR_UHR_CAPABILITY]); + } + if (info->attrs[NL80211_ATTR_S1G_CAPABILITY]) params->link_sta_params.s1g_capa = nla_data(info->attrs[NL80211_ATTR_S1G_CAPABILITY]); @@ -8609,6 +8734,16 @@ static int nl80211_new_station(struct sk_buff *skb, struct genl_info *info) } } + if (info->attrs[NL80211_ATTR_UHR_CAPABILITY]) { + if (!params.link_sta_params.eht_capa) + return -EINVAL; + + params.link_sta_params.uhr_capa = + nla_data(info->attrs[NL80211_ATTR_UHR_CAPABILITY]); + params.link_sta_params.uhr_capa_len = + nla_len(info->attrs[NL80211_ATTR_UHR_CAPABILITY]); + } + if (info->attrs[NL80211_ATTR_EML_CAPABILITY]) { params.eml_cap_present = true; params.eml_cap = @@ -8668,10 +8803,11 @@ static int nl80211_new_station(struct sk_buff *skb, struct genl_info *info) params.link_sta_params.ht_capa = NULL; params.link_sta_params.vht_capa = NULL; - /* HE and EHT require WME */ + /* HE, EHT and UHR require WME */ if (params.link_sta_params.he_capa_len || params.link_sta_params.he_6ghz_capa || - params.link_sta_params.eht_capa_len) + params.link_sta_params.eht_capa_len || + params.link_sta_params.uhr_capa_len) return -EINVAL; } @@ -8788,6 +8924,10 @@ static int nl80211_new_station(struct sk_buff *skb, struct genl_info *info) goto out; } } + + params.epp_peer = + nla_get_flag(info->attrs[NL80211_ATTR_EPP_PEER]); + err = rdev_add_station(rdev, dev, mac_addr, ¶ms); out: dev_put(params.vlan); @@ -11174,7 +11314,7 @@ static int nl80211_start_radar_detection(struct sk_buff *skb, if (dfs_region == NL80211_DFS_UNSET) return -EINVAL; - err = nl80211_parse_chandef(rdev, info, &chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, &chandef); if (err) return err; @@ -11262,7 +11402,7 @@ static int nl80211_notify_radar_detection(struct sk_buff *skb, return -EINVAL; } - err = nl80211_parse_chandef(rdev, info, &chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, &chandef); if (err) { GENL_SET_ERR_MSG(info, "Unable to extract chandef info"); return err; @@ -11448,7 +11588,8 @@ static int nl80211_channel_switch(struct sk_buff *skb, struct genl_info *info) goto free; skip_beacons: - err = nl80211_parse_chandef(rdev, info, ¶ms.chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, + ¶ms.chandef); if (err) goto free; @@ -12341,6 +12482,9 @@ static int nl80211_associate(struct sk_buff *skb, struct genl_info *info) if (nla_get_flag(info->attrs[NL80211_ATTR_DISABLE_EHT])) req.flags |= ASSOC_REQ_DISABLE_EHT; + if (nla_get_flag(info->attrs[NL80211_ATTR_DISABLE_UHR])) + req.flags |= ASSOC_REQ_DISABLE_UHR; + if (info->attrs[NL80211_ATTR_VHT_CAPABILITY_MASK]) memcpy(&req.vht_capa_mask, nla_data(info->attrs[NL80211_ATTR_VHT_CAPABILITY_MASK]), @@ -12674,7 +12818,8 @@ static int nl80211_join_ibss(struct sk_buff *skb, struct genl_info *info) ibss.ie_len = nla_len(info->attrs[NL80211_ATTR_IE]); } - err = nl80211_parse_chandef(rdev, info, &ibss.chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, + &ibss.chandef); if (err) return err; @@ -13220,6 +13365,9 @@ static int nl80211_connect(struct sk_buff *skb, struct genl_info *info) if (nla_get_flag(info->attrs[NL80211_ATTR_DISABLE_EHT])) connect.flags |= ASSOC_REQ_DISABLE_EHT; + if (nla_get_flag(info->attrs[NL80211_ATTR_DISABLE_UHR])) + connect.flags |= ASSOC_REQ_DISABLE_UHR; + if (info->attrs[NL80211_ATTR_VHT_CAPABILITY_MASK]) memcpy(&connect.vht_capa_mask, nla_data(info->attrs[NL80211_ATTR_VHT_CAPABILITY_MASK]), @@ -13670,7 +13818,7 @@ static int nl80211_remain_on_channel(struct sk_buff *skb, duration > rdev->wiphy.max_remain_on_channel_duration) return -EINVAL; - err = nl80211_parse_chandef(rdev, info, &chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, &chandef); if (err) return err; @@ -13886,7 +14034,8 @@ static int nl80211_tx_mgmt(struct sk_buff *skb, struct genl_info *info) */ chandef.chan = NULL; if (info->attrs[NL80211_ATTR_WIPHY_FREQ]) { - err = nl80211_parse_chandef(rdev, info, &chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, + &chandef); if (err) return err; } @@ -14289,7 +14438,8 @@ static int nl80211_join_ocb(struct sk_buff *skb, struct genl_info *info) struct ocb_setup setup = {}; int err; - err = nl80211_parse_chandef(rdev, info, &setup.chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, + &setup.chandef); if (err) return err; @@ -14364,7 +14514,8 @@ static int nl80211_join_mesh(struct sk_buff *skb, struct genl_info *info) cfg.auto_open_plinks = false; if (info->attrs[NL80211_ATTR_WIPHY_FREQ]) { - err = nl80211_parse_chandef(rdev, info, &setup.chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, + &setup.chandef); if (err) return err; } else { @@ -16246,6 +16397,224 @@ void cfg80211_nan_func_terminated(struct wireless_dev *wdev, } EXPORT_SYMBOL(cfg80211_nan_func_terminated); +void cfg80211_nan_sched_update_done(struct wireless_dev *wdev, bool success, + gfp_t gfp) +{ + struct wiphy *wiphy = wdev->wiphy; + struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy); + struct sk_buff *msg; + void *hdr; + + trace_cfg80211_nan_sched_update_done(wiphy, wdev, success); + + /* Can happen if we stopped NAN */ + if (!wdev->u.nan.sched_update_pending) + return; + + wdev->u.nan.sched_update_pending = false; + + if (!wdev->owner_nlportid) + return; + + msg = nlmsg_new(NLMSG_DEFAULT_SIZE, gfp); + if (!msg) + return; + + hdr = nl80211hdr_put(msg, 0, 0, 0, NL80211_CMD_NAN_SCHED_UPDATE_DONE); + if (!hdr) + goto nla_put_failure; + + if (nla_put_u32(msg, NL80211_ATTR_WIPHY, rdev->wiphy_idx) || + nla_put_u64_64bit(msg, NL80211_ATTR_WDEV, wdev_id(wdev), + NL80211_ATTR_PAD) || + (success && + nla_put_flag(msg, NL80211_ATTR_NAN_SCHED_UPDATE_SUCCESS))) + goto nla_put_failure; + + genlmsg_end(msg, hdr); + + genlmsg_unicast(wiphy_net(wiphy), msg, wdev->owner_nlportid); + + return; + +nla_put_failure: + nlmsg_free(msg); +} +EXPORT_SYMBOL(cfg80211_nan_sched_update_done); + +static int nl80211_parse_nan_channel(struct cfg80211_registered_device *rdev, + struct nlattr *channel, + struct genl_info *info, + struct cfg80211_nan_local_sched *sched, + u8 index) +{ + struct nlattr **channel_parsed __free(kfree) = NULL; + struct cfg80211_chan_def chandef; + u8 n_rx_nss; + int ret; + + channel_parsed = kcalloc(NL80211_ATTR_MAX + 1, sizeof(*channel_parsed), + GFP_KERNEL); + if (!channel_parsed) + return -ENOMEM; + + ret = nla_parse_nested(channel_parsed, NL80211_ATTR_MAX, channel, NULL, + info->extack); + if (ret) + return ret; + + ret = nl80211_parse_chandef(rdev, info->extack, channel_parsed, + &chandef); + if (ret) + return ret; + + if (chandef.chan->band == NL80211_BAND_6GHZ) { + NL_SET_ERR_MSG(info->extack, + "6 GHz band is not supported"); + return -EOPNOTSUPP; + } + + if (!cfg80211_reg_can_beacon(&rdev->wiphy, &chandef, + NL80211_IFTYPE_NAN)) { + NL_SET_ERR_MSG_ATTR(info->extack, channel, + "Channel in NAN schedule is not allowed for NAN operation"); + return -EINVAL; + } + + for (int i = 0; i < index; i++) { + if (cfg80211_chandef_compatible(&sched->nan_channels[i].chandef, + &chandef)) { + NL_SET_ERR_MSG_ATTR(info->extack, channel, + "Channels in NAN schedule must be mutually incompatible"); + return -EINVAL; + } + } + + if (!channel_parsed[NL80211_ATTR_NAN_CHANNEL_ENTRY]) + return -EINVAL; + + sched->nan_channels[index].channel_entry = + nla_data(channel_parsed[NL80211_ATTR_NAN_CHANNEL_ENTRY]); + + if (!channel_parsed[NL80211_ATTR_NAN_RX_NSS]) + return -EINVAL; + + sched->nan_channels[index].rx_nss = + nla_get_u8(channel_parsed[NL80211_ATTR_NAN_RX_NSS]); + + n_rx_nss = u8_get_bits(rdev->wiphy.nan_capa.n_antennas, 0x03); + if (sched->nan_channels[index].rx_nss > n_rx_nss || + !sched->nan_channels[index].rx_nss) { + NL_SET_ERR_MSG_ATTR(info->extack, channel, + "Invalid RX NSS in NAN channel definition"); + return -EINVAL; + } + + sched->nan_channels[index].chandef = chandef; + + return 0; +} + +static bool nl80211_nan_is_sched_empty(struct cfg80211_nan_local_sched *sched) +{ + if (!sched->n_channels) + return true; + + for (int i = 0; i < ARRAY_SIZE(sched->schedule); i++) { + if (sched->schedule[i] != NL80211_NAN_SCHED_NOT_AVAIL_SLOT) + return false; + } + + return true; +} + +static int nl80211_nan_set_local_sched(struct sk_buff *skb, + struct genl_info *info) +{ + struct cfg80211_registered_device *rdev = info->user_ptr[0]; + struct cfg80211_nan_local_sched *sched __free(kfree) = NULL; + struct wireless_dev *wdev = info->user_ptr[1]; + int rem, i = 0, n_channels = 0; + struct nlattr *channel; + bool sched_empty; + + if (wdev->iftype != NL80211_IFTYPE_NAN) + return -EOPNOTSUPP; + + if (!wdev_running(wdev)) + return -ENOTCONN; + + if (!info->attrs[NL80211_ATTR_NAN_TIME_SLOTS]) + return -EINVAL; + + /* First count how many channel attributes we got */ + nlmsg_for_each_attr_type(channel, NL80211_ATTR_NAN_CHANNEL, + info->nlhdr, GENL_HDRLEN, rem) + n_channels++; + + sched = kzalloc(struct_size(sched, nan_channels, n_channels), + GFP_KERNEL); + if (!sched) + return -ENOMEM; + + sched->n_channels = n_channels; + + nlmsg_for_each_attr_type(channel, NL80211_ATTR_NAN_CHANNEL, + info->nlhdr, GENL_HDRLEN, rem) { + int ret = nl80211_parse_nan_channel(rdev, channel, info, sched, + i); + + if (ret) + return ret; + i++; + } + + memcpy(sched->schedule, + nla_data(info->attrs[NL80211_ATTR_NAN_TIME_SLOTS]), + nla_len(info->attrs[NL80211_ATTR_NAN_TIME_SLOTS])); + + for (int slot = 0; slot < ARRAY_SIZE(sched->schedule); slot++) { + if (sched->schedule[slot] != NL80211_NAN_SCHED_NOT_AVAIL_SLOT && + sched->schedule[slot] >= sched->n_channels) { + NL_SET_ERR_MSG(info->extack, + "Invalid time slot in NAN schedule"); + return -EINVAL; + } + } + + sched_empty = nl80211_nan_is_sched_empty(sched); + + sched->deferred = + nla_get_flag(info->attrs[NL80211_ATTR_NAN_SCHED_DEFERRED]); + + if (sched_empty) { + if (sched->deferred) { + NL_SET_ERR_MSG(info->extack, + "Schedule cannot be deferred if all time slots are unavailable"); + return -EINVAL; + } + + if (info->attrs[NL80211_ATTR_NAN_AVAIL_BLOB]) { + NL_SET_ERR_MSG(info->extack, + "NAN Availability blob must be empty if all time slots are unavailable"); + return -EINVAL; + } + } else { + if (!info->attrs[NL80211_ATTR_NAN_AVAIL_BLOB]) { + NL_SET_ERR_MSG(info->extack, + "NAN Availability blob attribute is required"); + return -EINVAL; + } + + sched->nan_avail_blob = + nla_data(info->attrs[NL80211_ATTR_NAN_AVAIL_BLOB]); + sched->nan_avail_blob_len = + nla_len(info->attrs[NL80211_ATTR_NAN_AVAIL_BLOB]); + } + + return cfg80211_nan_set_local_schedule(rdev, wdev, sched); +} + static int nl80211_get_protocol_features(struct sk_buff *skb, struct genl_info *info) { @@ -16836,7 +17205,7 @@ static int nl80211_tdls_channel_switch(struct sk_buff *skb, !info->attrs[NL80211_ATTR_OPER_CLASS]) return -EINVAL; - err = nl80211_parse_chandef(rdev, info, &chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, &chandef); if (err) return err; @@ -17645,6 +18014,16 @@ nl80211_add_mod_link_station(struct sk_buff *skb, struct genl_info *info, } } + if (info->attrs[NL80211_ATTR_UHR_CAPABILITY]) { + if (!params.eht_capa) + return -EINVAL; + + params.uhr_capa = + nla_data(info->attrs[NL80211_ATTR_UHR_CAPABILITY]); + params.uhr_capa_len = + nla_len(info->attrs[NL80211_ATTR_UHR_CAPABILITY]); + } + if (info->attrs[NL80211_ATTR_HE_6GHZ_CAPABILITY]) params.he_6ghz_capa = nla_data(info->attrs[NL80211_ATTR_HE_6GHZ_CAPABILITY]); @@ -19040,6 +19419,12 @@ static const struct genl_small_ops nl80211_small_ops[] = { .flags = GENL_UNS_ADMIN_PERM, .internal_flags = IFLAGS(NL80211_FLAG_NEED_NETDEV_UP), }, + { + .cmd = NL80211_CMD_NAN_SET_LOCAL_SCHED, + .doit = nl80211_nan_set_local_sched, + .flags = GENL_ADMIN_PERM, + .internal_flags = IFLAGS(NL80211_FLAG_NEED_WDEV_UP), + }, }; static struct genl_family nl80211_fam __ro_after_init = { @@ -21000,6 +21385,46 @@ void cfg80211_ch_switch_notify(struct net_device *dev, } EXPORT_SYMBOL(cfg80211_ch_switch_notify); +void cfg80211_incumbent_signal_notify(struct wiphy *wiphy, + const struct cfg80211_chan_def *chandef, + u32 signal_interference_bitmap, + gfp_t gfp) +{ + struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy); + struct sk_buff *msg; + void *hdr; + + trace_cfg80211_incumbent_signal_notify(wiphy, chandef, signal_interference_bitmap); + + msg = nlmsg_new(NLMSG_DEFAULT_SIZE, gfp); + if (!msg) + return; + + hdr = nl80211hdr_put(msg, 0, 0, 0, NL80211_CMD_INCUMBENT_SIGNAL_DETECT); + if (!hdr) + goto nla_put_failure; + + if (nla_put_u32(msg, NL80211_ATTR_WIPHY, rdev->wiphy_idx)) + goto nla_put_failure; + + if (nl80211_send_chandef(msg, chandef)) + goto nla_put_failure; + + if (nla_put_u32(msg, NL80211_ATTR_INCUMBENT_SIGNAL_INTERFERENCE_BITMAP, + signal_interference_bitmap)) + goto nla_put_failure; + + genlmsg_end(msg, hdr); + + genlmsg_multicast_netns(&nl80211_fam, wiphy_net(&rdev->wiphy), msg, 0, + NL80211_MCGRP_MLME, gfp); + return; + +nla_put_failure: + nlmsg_free(msg); +} +EXPORT_SYMBOL(cfg80211_incumbent_signal_notify); + void cfg80211_ch_switch_started_notify(struct net_device *dev, struct cfg80211_chan_def *chandef, unsigned int link_id, u8 count, diff --git a/net/wireless/nl80211.h b/net/wireless/nl80211.h index 5e25782af1e0..048ba92c3e42 100644 --- a/net/wireless/nl80211.h +++ b/net/wireless/nl80211.h @@ -1,7 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* * Portions of this file - * Copyright (C) 2018, 2020-2024 Intel Corporation + * Copyright (C) 2018, 2020-2025 Intel Corporation */ #ifndef __NET_WIRELESS_NL80211_H #define __NET_WIRELESS_NL80211_H @@ -23,7 +23,8 @@ static inline u64 wdev_id(struct wireless_dev *wdev) } int nl80211_parse_chandef(struct cfg80211_registered_device *rdev, - struct genl_info *info, + struct netlink_ext_ack *extack, + struct nlattr **attrs, struct cfg80211_chan_def *chandef); int nl80211_parse_random_mac(struct nlattr **attrs, u8 *mac_addr, u8 *mac_addr_mask); diff --git a/net/wireless/pmsr.c b/net/wireless/pmsr.c index 06563f608487..6882c39b73a4 100644 --- a/net/wireless/pmsr.c +++ b/net/wireless/pmsr.c @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ /* - * Copyright (C) 2018 - 2021, 2023 - 2024 Intel Corporation + * Copyright (C) 2018 - 2021, 2023 - 2026 Intel Corporation */ #include #include "core.h" @@ -224,7 +224,8 @@ static int pmsr_parse_peer(struct cfg80211_registered_device *rdev, if (err) return err; - err = nl80211_parse_chandef(rdev, info, &out->chandef); + err = nl80211_parse_chandef(rdev, info->extack, info->attrs, + &out->chandef); if (err) return err; diff --git a/net/wireless/rdev-ops.h b/net/wireless/rdev-ops.h index ac6884bacf3f..f4c06282b2e1 100644 --- a/net/wireless/rdev-ops.h +++ b/net/wireless/rdev-ops.h @@ -1060,6 +1060,22 @@ rdev_nan_change_conf(struct cfg80211_registered_device *rdev, return ret; } +static inline int +rdev_nan_set_local_sched(struct cfg80211_registered_device *rdev, + struct wireless_dev *wdev, + struct cfg80211_nan_local_sched *sched) +{ + int ret; + + trace_rdev_nan_set_local_sched(&rdev->wiphy, wdev, sched); + if (rdev->ops->nan_set_local_sched) + ret = rdev->ops->nan_set_local_sched(&rdev->wiphy, wdev, sched); + else + ret = -EOPNOTSUPP; + trace_rdev_return_int(&rdev->wiphy, ret); + return ret; +} + static inline int rdev_set_mac_acl(struct cfg80211_registered_device *rdev, struct net_device *dev, struct cfg80211_acl_data *params) diff --git a/net/wireless/reg.c b/net/wireless/reg.c index 73cab51f6379..15e56dfad00f 100644 --- a/net/wireless/reg.c +++ b/net/wireless/reg.c @@ -5,7 +5,7 @@ * Copyright 2008-2011 Luis R. Rodriguez * Copyright 2013-2014 Intel Mobile Communications GmbH * Copyright 2017 Intel Deutschland GmbH - * Copyright (C) 2018 - 2025 Intel Corporation + * Copyright (C) 2018 - 2026 Intel Corporation * * Permission to use, copy, modify, and/or distribute this software for any * purpose with or without fee is hereby granted, provided that the above @@ -1605,6 +1605,8 @@ static u32 map_regdom_flags(u32 rd_flags) channel_flags |= IEEE80211_CHAN_ALLOW_6GHZ_VLP_AP; if (rd_flags & NL80211_RRF_ALLOW_20MHZ_ACTIVITY) channel_flags |= IEEE80211_CHAN_ALLOW_20MHZ_ACTIVITY; + if (rd_flags & NL80211_RRF_NO_UHR) + channel_flags |= IEEE80211_CHAN_NO_UHR; return channel_flags; } diff --git a/net/wireless/trace.h b/net/wireless/trace.h index 2b71f1d867a0..30d3619f702c 100644 --- a/net/wireless/trace.h +++ b/net/wireless/trace.h @@ -2393,6 +2393,27 @@ TRACE_EVENT(rdev_del_nan_func, WIPHY_PR_ARG, WDEV_PR_ARG, __entry->cookie) ); +TRACE_EVENT(rdev_nan_set_local_sched, + TP_PROTO(struct wiphy *wiphy, struct wireless_dev *wdev, + struct cfg80211_nan_local_sched *sched), + TP_ARGS(wiphy, wdev, sched), + TP_STRUCT__entry( + WIPHY_ENTRY + WDEV_ENTRY + __array(u8, schedule, CFG80211_NAN_SCHED_NUM_TIME_SLOTS) + ), + TP_fast_assign( + WIPHY_ASSIGN; + WDEV_ASSIGN; + memcpy(__entry->schedule, sched->schedule, + CFG80211_NAN_SCHED_NUM_TIME_SLOTS); + ), + TP_printk(WIPHY_PR_FMT ", " WDEV_PR_FMT ", schedule: %s", + WIPHY_PR_ARG, WDEV_PR_ARG, + __print_array(__entry->schedule, + CFG80211_NAN_SCHED_NUM_TIME_SLOTS, 1)) +); + TRACE_EVENT(rdev_set_mac_acl, TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, struct cfg80211_acl_data *params), @@ -4222,6 +4243,42 @@ TRACE_EVENT(cfg80211_nan_cluster_joined, WDEV_PR_ARG, __entry->cluster_id, __entry->new_cluster ? " [new]" : "") ); + +TRACE_EVENT(cfg80211_incumbent_signal_notify, + TP_PROTO(struct wiphy *wiphy, + const struct cfg80211_chan_def *chandef, + u32 signal_interference_bitmap), + TP_ARGS(wiphy, chandef, signal_interference_bitmap), + TP_STRUCT__entry( + WIPHY_ENTRY + CHAN_DEF_ENTRY + __field(u32, signal_interference_bitmap) + ), + TP_fast_assign( + WIPHY_ASSIGN; + CHAN_DEF_ASSIGN(chandef); + __entry->signal_interference_bitmap = signal_interference_bitmap; + ), + TP_printk(WIPHY_PR_FMT ", " CHAN_DEF_PR_FMT ", signal_interference_bitmap=0x%x", + WIPHY_PR_ARG, CHAN_DEF_PR_ARG, __entry->signal_interference_bitmap) +); + +TRACE_EVENT(cfg80211_nan_sched_update_done, + TP_PROTO(struct wiphy *wiphy, struct wireless_dev *wdev, bool success), + TP_ARGS(wiphy, wdev, success), + TP_STRUCT__entry( + WIPHY_ENTRY + WDEV_ENTRY + __field(bool, success) + ), + TP_fast_assign( + WIPHY_ASSIGN; + WDEV_ASSIGN; + __entry->success = success; + ), + TP_printk(WIPHY_PR_FMT ", " WDEV_PR_FMT " success=%d", + WIPHY_PR_ARG, WDEV_PR_ARG, __entry->success) +); #endif /* !__RDEV_OPS_TRACE || TRACE_HEADER_MULTI_READ */ #undef TRACE_INCLUDE_PATH diff --git a/net/wireless/util.c b/net/wireless/util.c index 81d6d27d273c..918d8f8468bd 100644 --- a/net/wireless/util.c +++ b/net/wireless/util.c @@ -5,7 +5,7 @@ * Copyright 2007-2009 Johannes Berg * Copyright 2013-2014 Intel Mobile Communications GmbH * Copyright 2017 Intel Deutschland GmbH - * Copyright (C) 2018-2023, 2025 Intel Corporation + * Copyright (C) 2018-2023, 2025-2026 Intel Corporation */ #include #include @@ -1573,26 +1573,30 @@ static u32 cfg80211_calculate_bitrate_he(struct rate_info *rate) return result / 10000; } -static u32 cfg80211_calculate_bitrate_eht(struct rate_info *rate) +static u32 _cfg80211_calculate_bitrate_eht_uhr(struct rate_info *rate) { #define SCALE 6144 - static const u32 mcs_divisors[16] = { - 102399, /* 16.666666... */ - 51201, /* 8.333333... */ - 34134, /* 5.555555... */ - 25599, /* 4.166666... */ - 17067, /* 2.777777... */ - 12801, /* 2.083333... */ - 11377, /* 1.851725... */ - 10239, /* 1.666666... */ - 8532, /* 1.388888... */ - 7680, /* 1.250000... */ - 6828, /* 1.111111... */ - 6144, /* 1.000000... */ - 5690, /* 0.926106... */ - 5120, /* 0.833333... */ - 409600, /* 66.666666... */ - 204800, /* 33.333333... */ + static const u32 mcs_divisors[] = { + [ 0] = 102399, /* 16.666666... */ + [ 1] = 51201, /* 8.333333... */ + [ 2] = 34134, /* 5.555555... */ + [ 3] = 25599, /* 4.166666... */ + [ 4] = 17067, /* 2.777777... */ + [ 5] = 12801, /* 2.083333... */ + [ 6] = 11377, /* 1.851725... */ + [ 7] = 10239, /* 1.666666... */ + [ 8] = 8532, /* 1.388888... */ + [ 9] = 7680, /* 1.250000... */ + [10] = 6828, /* 1.111111... */ + [11] = 6144, /* 1.000000... */ + [12] = 5690, /* 0.926106... */ + [13] = 5120, /* 0.833333... */ + [14] = 409600, /* 66.666666... */ + [15] = 204800, /* 33.333333... */ + [17] = 38400, /* 6.250180... */ + [19] = 19200, /* 3.125090... */ + [20] = 15360, /* 2.500000... */ + [23] = 9600, /* 1.562545... */ }; static const u32 rates_996[3] = { 480388888, 453700000, 408333333 }; static const u32 rates_484[3] = { 229411111, 216666666, 195000000 }; @@ -1603,8 +1607,6 @@ static u32 cfg80211_calculate_bitrate_eht(struct rate_info *rate) u64 tmp; u32 result; - if (WARN_ON_ONCE(rate->mcs > 15)) - return 0; if (WARN_ON_ONCE(rate->eht_gi > NL80211_RATE_INFO_EHT_GI_3_2)) return 0; if (WARN_ON_ONCE(rate->eht_ru_alloc > @@ -1685,7 +1687,7 @@ static u32 cfg80211_calculate_bitrate_eht(struct rate_info *rate) rate->eht_ru_alloc == NL80211_RATE_INFO_EHT_RU_ALLOC_26) result = rates_26[rate->eht_gi]; else { - WARN(1, "invalid EHT MCS: bw:%d, ru:%d\n", + WARN(1, "invalid EHT or UHR MCS: bw:%d, ru:%d\n", rate->bw, rate->eht_ru_alloc); return 0; } @@ -1699,11 +1701,64 @@ static u32 cfg80211_calculate_bitrate_eht(struct rate_info *rate) tmp *= rate->nss; do_div(tmp, 8); + /* and handle interference mitigation - 0.9x */ + if (rate->flags & RATE_INFO_FLAGS_UHR_IM) { + if (WARN(rate->nss != 1 || rate->mcs == 15, + "invalid NSS or MCS for UHR IM\n")) + return 0; + tmp *= 9000; + do_div(tmp, 10000); + } + result = tmp; return result / 10000; } +static u32 cfg80211_calculate_bitrate_eht(struct rate_info *rate) +{ + if (WARN_ONCE(rate->mcs > 15, "bad EHT MCS %d\n", rate->mcs)) + return 0; + + if (WARN_ONCE(rate->flags & (RATE_INFO_FLAGS_UHR_ELR_MCS | + RATE_INFO_FLAGS_UHR_IM), + "bad EHT MCS flags 0x%x\n", rate->flags)) + return 0; + + return _cfg80211_calculate_bitrate_eht_uhr(rate); +} + +static u32 cfg80211_calculate_bitrate_uhr(struct rate_info *rate) +{ + if (rate->flags & RATE_INFO_FLAGS_UHR_ELR_MCS) { + WARN_ONCE(rate->eht_gi != NL80211_RATE_INFO_EHT_GI_1_6, + "bad UHR ELR guard interval %d\n", + rate->eht_gi); + WARN_ONCE(rate->mcs > 1, "bad UHR ELR MCS %d\n", rate->mcs); + WARN_ONCE(rate->nss != 1, "bad UHR ELR NSS %d\n", rate->nss); + WARN_ONCE(rate->bw != RATE_INFO_BW_20, + "bad UHR ELR bandwidth %d\n", + rate->bw); + WARN_ONCE(rate->flags & RATE_INFO_FLAGS_UHR_IM, + "bad UHR MCS flags 0x%x\n", rate->flags); + if (rate->mcs == 0) + return 17; + return 33; + } + + switch (rate->mcs) { + case 0 ... 15: + case 17: + case 19: + case 20: + case 23: + return _cfg80211_calculate_bitrate_eht_uhr(rate); + } + + WARN_ONCE(1, "bad UHR MCS %d\n", rate->mcs); + return 0; +} + static u32 cfg80211_calculate_bitrate_s1g(struct rate_info *rate) { /* For 1, 2, 4, 8 and 16 MHz channels */ @@ -1828,6 +1883,8 @@ u32 cfg80211_calculate_bitrate(struct rate_info *rate) return cfg80211_calculate_bitrate_he(rate); if (rate->flags & RATE_INFO_FLAGS_EHT_MCS) return cfg80211_calculate_bitrate_eht(rate); + if (rate->flags & RATE_INFO_FLAGS_UHR_MCS) + return cfg80211_calculate_bitrate_uhr(rate); if (rate->flags & RATE_INFO_FLAGS_S1G_MCS) return cfg80211_calculate_bitrate_s1g(rate); diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index ff93fd7ac631..2e41770a9988 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -160,6 +160,17 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff_xsk *xskb, u32 len, return 0; } +static void __xsk_rcv_zc_safe(struct xdp_sock *xs, struct xdp_buff_xsk *xskb, + u32 len, u32 flags) +{ + u64 addr; + + addr = xp_get_handle(xskb, xskb->pool); + __xskq_prod_reserve_desc(xs->rx, addr, len, flags); + + xp_release(xskb); +} + static int xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len) { struct xdp_buff_xsk *xskb = container_of(xdp, struct xdp_buff_xsk, xdp); @@ -242,9 +253,11 @@ static int __xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len) u32 frame_size = __xsk_pool_get_rx_frame_size(xs->pool); void *copy_from = xsk_copy_xdp_start(xdp), *copy_to; u32 from_len, meta_len, rem, num_desc; - struct xdp_buff_xsk *xskb; + struct xdp_buff_xsk *xskb, *tmp; struct xdp_buff *xsk_xdp; + LIST_HEAD(xsk_buffs); skb_frag_t *frag; + u32 i; from_len = xdp->data_end - copy_from; meta_len = xdp->data - copy_from; @@ -287,22 +300,45 @@ static int __xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len) frag = &sinfo->frags[0]; } + for (i = 0; i < num_desc; i++) { + xsk_xdp = xsk_buff_alloc(xs->pool); + if (!xsk_xdp) + goto err_alloc; + + xskb = container_of(xsk_xdp, struct xdp_buff_xsk, xdp); + if (unlikely(!list_empty(&xskb->list_node))) + goto err_alloc; + + list_add_tail(&xskb->list_node, &xsk_buffs); + } + do { u32 to_len = frame_size + meta_len; u32 copied; - xsk_xdp = xsk_buff_alloc(xs->pool); + xskb = list_first_entry(&xsk_buffs, struct xdp_buff_xsk, + list_node); + list_del_init(&xskb->list_node); + xsk_xdp = &xskb->xdp; copy_to = xsk_xdp->data - meta_len; copied = xsk_copy_xdp(copy_to, ©_from, to_len, &from_len, &frag, rem); rem -= copied; - xskb = container_of(xsk_xdp, struct xdp_buff_xsk, xdp); - __xsk_rcv_zc(xs, xskb, copied - meta_len, rem ? XDP_PKT_CONTD : 0); + __xsk_rcv_zc_safe(xs, xskb, copied - meta_len, + rem ? XDP_PKT_CONTD : 0); meta_len = 0; } while (rem); return 0; + +err_alloc: + list_for_each_entry_safe(xskb, tmp, &xsk_buffs, list_node) { + list_del_init(&xskb->list_node); + xsk_buff_free(&xskb->xdp); + } + xs->rx_dropped++; + return -ENOMEM; } static bool xsk_tx_writeable(struct xdp_sock *xs) diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c index 772fe1838cfb..6e9c51b6b430 100644 --- a/net/xdp/xsk_buff_pool.c +++ b/net/xdp/xsk_buff_pool.c @@ -767,6 +767,7 @@ EXPORT_SYMBOL(xp_raw_get_dma); * xp_raw_get_ctx - get &xdp_desc context * @pool: XSk buff pool desc address belongs to * @addr: desc address (from userspace) + * @options: desc options (from userspace) * * Helper for getting desc's DMA address and metadata pointer, if present. * Saves one call on hotpath and double calculation of the actual address. @@ -775,14 +776,16 @@ EXPORT_SYMBOL(xp_raw_get_dma); * Return: new &xdp_desc_ctx struct containing desc's DMA address and metadata * pointer, if it is present (initialized to %NULL otherwise). */ -struct xdp_desc_ctx xp_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr) +struct xdp_desc_ctx xp_raw_get_ctx(const struct xsk_buff_pool *pool, u64 addr, + u32 options) { struct xdp_desc_ctx ret; addr = __xp_raw_get_addr(pool, addr); ret.dma = __xp_raw_get_dma(pool, addr); - ret.meta = __xsk_buff_get_metadata(pool, __xp_raw_get_data(pool, addr)); + ret.meta = __xsk_buff_get_metadata(pool, __xp_raw_get_data(pool, addr), + options); return ret; } diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index 1eb8d9f8b104..4f764b5748d2 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -440,20 +440,26 @@ static inline void xskq_prod_write_addr_batch(struct xsk_queue *q, struct xdp_de q->cached_prod = cached_prod; } -static inline int xskq_prod_reserve_desc(struct xsk_queue *q, - u64 addr, u32 len, u32 flags) +static inline void __xskq_prod_reserve_desc(struct xsk_queue *q, + u64 addr, u32 len, u32 flags) { struct xdp_rxtx_ring *ring = (struct xdp_rxtx_ring *)q->ring; u32 idx; - if (xskq_prod_is_full(q)) - return -ENOBUFS; - /* A, matches D */ idx = q->cached_prod++ & q->ring_mask; ring->desc[idx].addr = addr; ring->desc[idx].len = len; ring->desc[idx].options = flags; +} + +static inline int xskq_prod_reserve_desc(struct xsk_queue *q, + u64 addr, u32 len, u32 flags) +{ + if (xskq_prod_is_full(q)) + return -ENOBUFS; + + __xskq_prod_reserve_desc(q, addr, len, flags); return 0; } diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c index 54222fcbd7fd..b152edf50fec 100644 --- a/net/xfrm/xfrm_output.c +++ b/net/xfrm/xfrm_output.c @@ -629,10 +629,8 @@ static int xfrm_dev_direct_output(struct sock *sk, struct xfrm_state *x, nf_reset_ct(skb); err = skb_dst(skb)->ops->local_out(net, sk, skb); - if (unlikely(err != 1)) { - kfree_skb(skb); + if (unlikely(err != 1)) return err; - } /* In transport mode, network destination is * directly reachable, while in tunnel mode, diff --git a/rust/kernel/cpufreq.rs b/rust/kernel/cpufreq.rs index a509d09ca452..1c378350f7ea 100644 --- a/rust/kernel/cpufreq.rs +++ b/rust/kernel/cpufreq.rs @@ -361,23 +361,28 @@ pub fn new() -> Self { } } - /// Adds a new entry to the table. - pub fn add(&mut self, freq: Hertz, flags: u32, driver_data: u32) -> Result { + /// Adds a raw frequency-table entry. + fn push(&mut self, frequency: u32, flags: u32, driver_data: u32) -> Result { // Adds the new entry at the end of the vector. Ok(self.entries.push( bindings::cpufreq_frequency_table { flags, driver_data, - frequency: freq.as_khz() as u32, + frequency, }, GFP_KERNEL, )?) } + /// Adds a new entry to the table. + pub fn add(&mut self, freq: Hertz, flags: u32, driver_data: u32) -> Result { + self.push(freq.as_khz() as u32, flags, driver_data) + } + /// Consumes the [`TableBuilder`] and returns [`TableBox`]. pub fn to_table(mut self) -> Result { // Add last entry to the table. - self.add(Hertz(c_ulong::MAX), 0, 0)?; + self.push(bindings::CPUFREQ_TABLE_END as u32, 0, 0)?; TableBox::new(self.entries) } @@ -816,7 +821,9 @@ fn update_limits(_policy: &mut Policy) { } /// Driver's `bios_limit` callback. - fn bios_limit(_policy: &mut Policy, _limit: &mut u32) -> Result { + /// + /// Returns HW/BIOS max frequency limitations for the CPU. + fn bios_limit(_policy: &mut Policy) -> Result { build_error!(VTABLE_DEFAULT_ERROR) } @@ -1351,9 +1358,12 @@ impl Registration { from_result(|| { let mut policy = PolicyCpu::from_cpu(cpu_id)?; - + let val = T::bios_limit(&mut policy)?; // SAFETY: `limit` is guaranteed by the C code to be valid. - T::bios_limit(&mut policy, &mut (unsafe { *limit })).map(|()| 0) + unsafe { + *limit = val; + } + Ok(0) }) } diff --git a/rust/uapi/uapi_helper.h b/rust/uapi/uapi_helper.h index 06d7d1a2e8da..1c4aa4292dce 100644 --- a/rust/uapi/uapi_helper.h +++ b/rust/uapi/uapi_helper.h @@ -6,11 +6,11 @@ * Sorted alphabetically. */ -#include #include #include #include #include +#include #include #include #include diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c index 1ba1927b548e..164d9dd6fd92 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -320,9 +320,15 @@ static int __init ftrace_direct_init(void) ftrace_set_filter_ip(&direct, (unsigned long) my_ip, 0, 0); ret = register_ftrace_direct(&direct, my_tramp); - if (!ret) - simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn"); - return ret; + if (ret) + return ret; + simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn"); + if (IS_ERR(simple_tsk)) { + unregister_ftrace_direct(&direct, my_tramp, true); + return PTR_ERR(simple_tsk); + } + + return 0; } static void __exit ftrace_direct_exit(void) diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c index 7a7822dfeb50..b03766c6217b 100644 --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -364,9 +364,15 @@ static int __init ftrace_direct_multi_init(void) ret = register_ftrace_direct(&direct, my_tramp); - if (!ret) - simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn"); - return ret; + if (ret) + return ret; + simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn"); + if (IS_ERR(simple_tsk)) { + unregister_ftrace_direct(&direct, my_tramp, true); + return PTR_ERR(simple_tsk); + } + + return 0; } static void __exit ftrace_direct_multi_exit(void) diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c index 010c398f6a70..31f81f25968f 100644 --- a/scripts/mod/modpost.c +++ b/scripts/mod/modpost.c @@ -1570,14 +1570,14 @@ static void read_symbols(const char *modname) struct elf_info info = { }; Elf_Sym *sym; - if (!parse_elf(&info, modname)) - return; - if (!strends(modname, ".o")) { error("%s: filename must be suffixed with .o\n", modname); return; } + if (!parse_elf(&info, modname)) + return; + /* strip trailing .o */ mod = new_module(modname, strlen(modname) - strlen(".o")); diff --git a/scripts/tags.sh b/scripts/tags.sh index 99ce427d9a69..173f5d23c4dc 100755 --- a/scripts/tags.sh +++ b/scripts/tags.sh @@ -100,7 +100,7 @@ all_compiled_sources() { echo include/generated/autoconf.h find $ignore -name "*.cmd" -exec \ - grep -Poh '(?<=^ )\S+|(?<== )\S+[^\\](?=$)' {} \+ | + grep -Poh '(?<=^ )\S+\.([chS]|rs)(?=\s)|(?<== )\S+\.(?1)(?=$)' {} \+ | awk '!a[$0]++' } | xargs realpath -esq $([ -z "$KBUILD_ABS_SRCTREE" ] && echo --relative-to=.) | sort -u diff --git a/security/apparmor/af_unix.c b/security/apparmor/af_unix.c index 1e061345c0b1..834a3b1c2f0a 100644 --- a/security/apparmor/af_unix.c +++ b/security/apparmor/af_unix.c @@ -416,7 +416,7 @@ static int profile_peer_perm(struct aa_profile *profile, u32 request, unix_sk(sk), peer_addr, peer_addrlen, &p, &ad->info); - return fn_for_each_in_ns(peer_label, peerp, + return fn_for_each_in_scope(peer_label, peerp, match_label(profile, rules, state, request, peerp, p, ad)); } diff --git a/security/apparmor/apparmorfs.c b/security/apparmor/apparmorfs.c index 515cc80064c6..4dfca0c6e938 100644 --- a/security/apparmor/apparmorfs.c +++ b/security/apparmor/apparmorfs.c @@ -877,7 +877,7 @@ static ssize_t query_label(char *buf, size_t buf_len, perms = allperms; if (view_only) { - label_for_each_in_ns(i, labels_ns(label), label, profile) { + label_for_each_in_scope(i, labels_ns(label), label, profile) { profile_query_cb(profile, &perms, match_str, match_len); } } else { diff --git a/security/apparmor/domain.c b/security/apparmor/domain.c index cf9feed1306d..8e478ac99ff6 100644 --- a/security/apparmor/domain.c +++ b/security/apparmor/domain.c @@ -116,7 +116,7 @@ static inline aa_state_t match_component(struct aa_profile *profile, * @label: label to check access permissions for * @stack: whether this is a stacking request * @state: state to start match in - * @subns: whether to do permission checks on components in a subns + * @inview: whether to match labels in view or only in scope * @request: permissions to request * @perms: perms struct to set * @@ -128,7 +128,7 @@ static inline aa_state_t match_component(struct aa_profile *profile, */ static int label_compound_match(struct aa_profile *profile, struct aa_label *label, bool stack, - aa_state_t state, bool subns, u32 request, + aa_state_t state, bool inview, u32 request, struct aa_perms *perms) { struct aa_ruleset *rules = profile->label.rules[0]; @@ -136,9 +136,9 @@ static int label_compound_match(struct aa_profile *profile, struct label_it i; struct path_cond cond = { }; - /* find first subcomponent that is visible */ + /* find first subcomponent that is in view and going to be interated with */ label_for_each(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = match_component(profile, tp, stack, state); if (!state) @@ -152,7 +152,7 @@ static int label_compound_match(struct aa_profile *profile, next: label_for_each_cont(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = aa_dfa_match(rules->file->dfa, state, "//&"); state = match_component(profile, tp, false, state); @@ -178,7 +178,7 @@ static int label_compound_match(struct aa_profile *profile, * @label: label to check access permissions for * @stack: whether this is a stacking request * @start: state to start match in - * @subns: whether to do permission checks on components in a subns + * @inview: whether to match labels in view or only in scope * @request: permissions to request * @perms: an initialized perms struct to add accumulation to * @@ -190,7 +190,7 @@ static int label_compound_match(struct aa_profile *profile, */ static int label_components_match(struct aa_profile *profile, struct aa_label *label, bool stack, - aa_state_t start, bool subns, u32 request, + aa_state_t start, bool inview, u32 request, struct aa_perms *perms) { struct aa_ruleset *rules = profile->label.rules[0]; @@ -202,7 +202,7 @@ static int label_components_match(struct aa_profile *profile, /* find first subcomponent to test */ label_for_each(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = match_component(profile, tp, stack, start); if (!state) @@ -219,7 +219,7 @@ static int label_components_match(struct aa_profile *profile, aa_apply_modes_to_perms(profile, &tmp); aa_perms_accum(perms, &tmp); label_for_each_cont(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = match_component(profile, tp, stack, start); if (!state) @@ -246,26 +246,26 @@ static int label_components_match(struct aa_profile *profile, * @label: label to match (NOT NULL) * @stack: whether this is a stacking request * @state: state to start in - * @subns: whether to match subns components + * @inview: whether to match labels in view or only in scope * @request: permission request * @perms: Returns computed perms (NOT NULL) * * Returns: the state the match finished in, may be the none matching state */ static int label_match(struct aa_profile *profile, struct aa_label *label, - bool stack, aa_state_t state, bool subns, u32 request, + bool stack, aa_state_t state, bool inview, u32 request, struct aa_perms *perms) { int error; *perms = nullperms; - error = label_compound_match(profile, label, stack, state, subns, + error = label_compound_match(profile, label, stack, state, inview, request, perms); if (!error) return error; *perms = allperms; - return label_components_match(profile, label, stack, state, subns, + return label_components_match(profile, label, stack, state, inview, request, perms); } @@ -864,6 +864,61 @@ static int profile_onexec(const struct cred *subj_cred, } /* ensure none ns domain transitions are correctly applied with onexec */ +static struct aa_label *label_merge_wrap(struct aa_label *a, struct aa_label *b, + gfp_t gfp) +{ + struct aa_label *label = aa_label_merge(a, b, gfp); + + if (!label) + return ERR_PTR(-ENOMEM); + return label; +} + +static bool is_profile_priv_restricted_to_stack(const struct cred *subj_cred, + struct aa_profile *profile) +{ + if (profile_unconfined(profile) && profile == profile->ns->unconfined && + aa_unprivileged_unconfined_restricted && + /* cap_capable returns false (0) if true, hence true here means + * doesn't have capability and the stack will be restricted + */ + cap_capable(current_cred(), &init_user_ns, CAP_MAC_OVERRIDE, + CAP_OPT_NOAUDIT)) + return true; + return false; +} + +static const char *stack_msg = "change_profile unprivileged unconfined converted to stacking"; + +static struct aa_label *priv_restricted_transition(const struct cred *subj_cred, + struct aa_profile *profile, + const char *op, u32 request, + const char *name, + struct aa_label *transition, + gfp_t gfp) +{ + if (!is_profile_priv_restricted_to_stack(subj_cred, profile)) + return aa_get_newest_label(transition); + + /* transition allowed but only via stack */ + struct aa_label *target = label_merge_wrap(&profile->label, + transition, gfp); + if (IS_ERR_OR_NULL(target)) + return target; + + /* doing this here is less than optimal but good enough until the + * fs mediation rework lands + */ + struct aa_perms perms = { + .allow = request, + .audit = request, + }; + aa_audit_file(subj_cred, profile, &perms, op, + request, name, NULL, target, + subj_cred->euid, stack_msg, 0); + + return target; +} static struct aa_label *handle_onexec(const struct cred *subj_cred, struct aa_label *label, @@ -881,29 +936,36 @@ static struct aa_label *handle_onexec(const struct cred *subj_cred, AA_BUG(!bprm); AA_BUG(!buffer); - /* TODO: determine how much we want to loosen this */ - error = fn_for_each_in_ns(label, profile, + /* TODO: determine how much we want to loosen this + * only check profiles in scope for permission to change at exec + */ + error = fn_for_each_in_scope(label, profile, profile_onexec(subj_cred, profile, onexec, stack, bprm, buffer, cond, unsafe)); if (error) return ERR_PTR(error); - new = fn_label_build_in_ns(label, profile, GFP_KERNEL, - stack ? aa_label_merge(&profile->label, onexec, - GFP_KERNEL) - : aa_get_newest_label(onexec), + new = fn_label_build_in_scope(label, profile, GFP_KERNEL, + stack ? label_merge_wrap(&profile->label, onexec, + GFP_KERNEL) + : priv_restricted_transition(subj_cred, profile, + OP_CHANGE_ONEXEC, AA_MAY_ONEXEC, + bprm->filename, onexec, + GFP_KERNEL), profile_transition(subj_cred, profile, bprm, buffer, cond, unsafe)); - if (new) + AA_BUG(!new); + if (!IS_ERR(new)) return new; /* TODO: get rid of GLOBAL_ROOT_UID */ - error = fn_for_each_in_ns(label, profile, + error = fn_for_each_in_scope(label, profile, aa_audit_file(subj_cred, profile, &nullperms, OP_CHANGE_ONEXEC, AA_MAY_ONEXEC, bprm->filename, NULL, onexec, GLOBAL_ROOT_UID, - "failed to build target label", -ENOMEM)); + "failed to build target label", + PTR_ERR(new))); return ERR_PTR(error); } @@ -966,14 +1028,10 @@ int apparmor_bprm_creds_for_exec(struct linux_binprm *bprm) profile_transition(subj_cred, profile, bprm, buffer, &cond, &unsafe)); - AA_BUG(!new); if (IS_ERR(new)) { error = PTR_ERR(new); goto done; - } else if (!new) { - error = -ENOMEM; - goto done; } /* Policy has specified a domain transitions. If no_new_privs and @@ -1075,8 +1133,8 @@ static struct aa_label *build_change_hat(const struct cred *subj_cred, if (!hat) { error = -ENOENT; if (COMPLAIN_MODE(profile)) { - hat = aa_new_learning_profile(profile, true, name, - GFP_KERNEL); + hat = __aa_new_learning_profile(profile, true, name, + GFP_KERNEL); if (!hat) { info = "failed null profile create"; error = -ENOMEM; @@ -1114,6 +1172,7 @@ static struct aa_label *change_hat(const struct cred *subj_cred, bool sibling = false; const char *name, *info = NULL; int i, error; + bool needput = false; AA_BUG(!label); AA_BUG(!hats); @@ -1126,7 +1185,6 @@ static struct aa_label *change_hat(const struct cred *subj_cred, * the profiles and label, we can rely on the namespaces being live * and avoid incrementing their refcounts while grabbing the lock. */ - label = aa_get_label(label); ns = labels_ns(label); retry: @@ -1134,15 +1192,19 @@ static struct aa_label *change_hat(const struct cred *subj_cred, if (label_is_stale(label)) { new = aa_get_newest_label(label); new_ns = labels_ns(new); + + if (needput) + /* aa_put_label() is safe to call when under lock */ + aa_put_label(label); + label = new; + needput = true; + /* check if replaced with label in parent ns, and lock there */ if (new_ns != ns) { - aa_put_label(new); mutex_unlock(&ns->lock); ns = new_ns; - label = new; + /* retry will bottom out at the root of the tree */ goto retry; } - aa_put_label(label); - label = new; } if (PROFILE_IS_HAT(labels_profile(label))) @@ -1151,9 +1213,10 @@ static struct aa_label *change_hat(const struct cred *subj_cred, /*find first matching hat */ for (i = 0; i < count && !hat; i++) { name = hats[i]; - label_for_each_in_ns(it, labels_ns(label), label, profile) { + label_for_each_in_scope(it, labels_ns(label), label, profile) { if (sibling && PROFILE_IS_HAT(profile)) { - root = aa_get_profile(profile->parent); + root = aa_get_profile(rcu_dereference_protected(profile->parent, + mutex_is_locked(&ns->lock))); } else if (!sibling && !PROFILE_IS_HAT(profile)) { root = aa_get_profile(profile); } else { /* conflicting change type */ @@ -1187,7 +1250,7 @@ static struct aa_label *change_hat(const struct cred *subj_cred, * change_hat. */ name = NULL; - label_for_each_in_ns(it, labels_ns(label), label, profile) { + label_for_each_in_scope(it, labels_ns(label), label, profile) { if (!list_empty(&profile->base.profiles)) { info = "hat not found"; error = -ENOENT; @@ -1198,7 +1261,7 @@ static struct aa_label *change_hat(const struct cred *subj_cred, error = -ECHILD; fail: - label_for_each_in_ns(it, labels_ns(label), label, profile) { + label_for_each_in_scope(it, labels_ns(label), label, profile) { /* * no target as it has failed to be found or built * @@ -1214,19 +1277,20 @@ static struct aa_label *change_hat(const struct cred *subj_cred, } } mutex_unlock(&ns->lock); + if (needput) + aa_put_label(label); return ERR_PTR(error); build: - new = fn_label_build_in_ns(label, profile, GFP_KERNEL, + new = fn_label_build_in_scope(label, profile, GFP_KERNEL, build_change_hat(subj_cred, profile, name, sibling), aa_get_label(&profile->label)); - if (!new) { - info = "label build failed"; - error = -ENOMEM; - goto fail; - } /* else if (IS_ERR) build_change_hat has logged error so return new */ mutex_unlock(&ns->lock); + AA_BUG(!new); + /* return new label or error ptr */ + if (needput) + aa_put_label(label); return new; } @@ -1280,7 +1344,7 @@ int aa_change_hat(const char *hats[], int count, u64 token, int flags) bool empty = true; rcu_read_lock(); - label_for_each_in_ns(i, labels_ns(label), label, profile) { + label_for_each_in_scope(i, labels_ns(label), label, profile) { empty &= list_empty(&profile->base.profiles); } rcu_read_unlock(); @@ -1367,7 +1431,7 @@ int aa_change_hat(const char *hats[], int count, u64 token, int flags) perms.kill = AA_MAY_CHANGEHAT; fail: - fn_for_each_in_ns(label, profile, + fn_for_each_in_scope(label, profile, aa_audit_file(subj_cred, profile, &perms, OP_CHANGE_HAT, AA_MAY_CHANGEHAT, NULL, NULL, target, GLOBAL_ROOT_UID, info, error)); @@ -1399,8 +1463,6 @@ static int change_profile_perms_wrapper(const char *op, const char *name, return error; } -static const char *stack_msg = "change_profile unprivileged unconfined converted to stacking"; - /** * aa_change_profile - perform a one-way profile transition * @fqname: name of profile may include namespace (NOT NULL) @@ -1460,28 +1522,6 @@ int aa_change_profile(const char *fqname, int flags) op = OP_CHANGE_PROFILE; } - /* This should move to a per profile test. Requires pushing build - * into callback - */ - if (!stack && unconfined(label) && - label == &labels_ns(label)->unconfined->label && - aa_unprivileged_unconfined_restricted && - /* TODO: refactor so this check is a fn */ - cap_capable(current_cred(), &init_user_ns, CAP_MAC_OVERRIDE, - CAP_OPT_NOAUDIT)) { - /* regardless of the request in this case apparmor - * stacks against unconfined so admin set policy can't be - * by-passed - */ - stack = true; - perms.audit = request; - (void) fn_for_each_in_ns(label, profile, - aa_audit_file(subj_cred, profile, &perms, op, - request, auditname, NULL, target, - GLOBAL_ROOT_UID, stack_msg, 0)); - perms.audit = 0; - } - if (*fqname == '&') { stack = true; /* don't have label_parse() do stacking */ @@ -1521,7 +1561,7 @@ int aa_change_profile(const char *fqname, int flags) * * if (!stack) { */ - error = fn_for_each_in_ns(label, profile, + error = fn_for_each_in_scope(label, profile, change_profile_perms_wrapper(op, auditname, subj_cred, profile, target, stack, @@ -1535,7 +1575,7 @@ int aa_change_profile(const char *fqname, int flags) check: /* check if tracing task is allowed to trace target domain */ error = may_change_ptraced_domain(subj_cred, target, &info); - if (error && !fn_for_each_in_ns(label, profile, + if (error && !fn_for_each_in_scope(label, profile, COMPLAIN_MODE(profile))) goto audit; @@ -1551,10 +1591,14 @@ int aa_change_profile(const char *fqname, int flags) /* stacking is always a subset, so only check the nonstack case */ if (!stack) { - new = fn_label_build_in_ns(label, profile, GFP_KERNEL, - aa_get_label(target), + new = fn_label_build_in_scope(label, profile, GFP_KERNEL, + priv_restricted_transition(subj_cred, profile, + op, request, + auditname, target, + GFP_KERNEL), aa_get_label(&profile->label)); - if (IS_ERR_OR_NULL(new)) + AA_BUG(!new); + if (IS_ERR(new)) goto build_fail; /* * no new privs prevents domain transitions that would @@ -1578,10 +1622,9 @@ int aa_change_profile(const char *fqname, int flags) goto build_fail; error = aa_replace_current_label(new); } else { - if (new) { - aa_put_label(new); - new = NULL; - } + /* new will be recomputed so at exec time. So discard */ + aa_put_label(new); + new = NULL; /* full transition will be built in exec path */ aa_set_current_onexec(target, stack); @@ -1599,7 +1642,7 @@ int aa_change_profile(const char *fqname, int flags) perms.allow = 0; audit: - error = fn_for_each_in_ns(label, profile, + error = fn_for_each_in_scope(label, profile, aa_audit_file(subj_cred, profile, &perms, op, request, auditname, NULL, new ? new : target, diff --git a/security/apparmor/include/lib.h b/security/apparmor/include/lib.h index 26df19c1df4f..1721782c6f92 100644 --- a/security/apparmor/include/lib.h +++ b/security/apparmor/include/lib.h @@ -76,6 +76,19 @@ int aa_print_debug_params(char *buffer); /* Flag indicating whether initialization completed */ extern int apparmor_initialized; +/* semantic split of scope and view */ +#define aa_in_scope(SUBJ, OBJ) \ + aa_ns_visible(SUBJ, OBJ, false) + +#define aa_in_view(SUBJ, OBJ) \ + aa_ns_visible(SUBJ, OBJ, true) + +#define label_for_each_in_scope(I, NS, L, P) \ + label_for_each_in_ns(I, NS, L, P) + +#define fn_for_each_in_scope(L, P, FN) \ + fn_for_each_in_ns(L, P, FN) + /* fn's in lib */ const char *skipn_spaces(const char *str, size_t n); const char *aa_splitn_fqname(const char *fqname, size_t n, const char **ns_name, @@ -259,7 +272,6 @@ void aa_policy_destroy(struct aa_policy *policy); * * Returns: new label on success * ERR_PTR if build @FN fails - * NULL if label_build fails due to low memory conditions * * @FN must return a label or ERR_PTR on failure. NULL is not allowed */ @@ -275,7 +287,7 @@ void aa_policy_destroy(struct aa_policy *policy); DEFINE_VEC(label, __lvec); \ DEFINE_VEC(profile, __pvec); \ if (vec_setup(label, __lvec, (L)->size, (GFP))) { \ - __new_ = NULL; \ + __new_ = ERR_PTR(-ENOMEM); \ goto __done; \ } \ __j = 0; \ @@ -297,28 +309,29 @@ void aa_policy_destroy(struct aa_policy *policy); if (__count > 1) { \ __new_ = aa_vec_find_or_create_label(__pvec,\ __count, (GFP)); \ - /* only fails if out of Mem */ \ if (!__new_) \ - __new_ = NULL; \ + __new_ = ERR_PTR(-ENOMEM); \ } else \ __new_ = aa_get_label(&__pvec[0]->label); \ vec_cleanup(profile, __pvec, __count); \ } else \ - __new_ = NULL; \ + __new_ = ERR_PTR(-ENOMEM); \ __do_cleanup: \ vec_cleanup(label, __lvec, (L)->size); \ } else { \ (P) = labels_profile(L); \ __new_ = (FN); \ + AA_BUG(!__new_); \ } \ __done: \ - if (!__new_) \ + if (PTR_ERR(__new_)) \ AA_DEBUG(DEBUG_LABEL, "label build failed\n"); \ + AA_BUG(!__new_); \ (__new_); \ }) -#define __fn_build_in_ns(NS, P, NS_FN, OTHER_FN) \ +#define __fn_build_in_scope(NS, P, NS_FN, OTHER_FN) \ ({ \ struct aa_label *__new; \ if ((P)->ns != (NS)) \ @@ -328,10 +341,10 @@ __done: \ (__new); \ }) -#define fn_label_build_in_ns(L, P, GFP, NS_FN, OTHER_FN) \ +#define fn_label_build_in_scope(L, P, GFP, NS_FN, OTHER_FN) \ ({ \ fn_label_build((L), (P), (GFP), \ - __fn_build_in_ns(labels_ns(L), (P), (NS_FN), (OTHER_FN))); \ + __fn_build_in_scope(labels_ns(L), (P), (NS_FN), (OTHER_FN))); \ }) #endif /* __AA_LIB_H */ diff --git a/security/apparmor/include/policy.h b/security/apparmor/include/policy.h index bf105ae9019d..2eaa1bee31bc 100644 --- a/security/apparmor/include/policy.h +++ b/security/apparmor/include/policy.h @@ -271,6 +271,9 @@ struct aa_profile *aa_alloc_profile(const char *name, struct aa_proxy *proxy, gfp_t gfp); struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name, gfp_t gfp); +struct aa_profile *__aa_new_learning_profile(struct aa_profile *parent, + bool hat, const char *base, + gfp_t gfp); struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, const char *base, gfp_t gfp); void aa_free_profile(struct aa_profile *profile); diff --git a/security/apparmor/label.c b/security/apparmor/label.c index d382ac772ca1..5c42cd01ea47 100644 --- a/security/apparmor/label.c +++ b/security/apparmor/label.c @@ -1278,7 +1278,7 @@ static inline aa_state_t match_component(struct aa_profile *profile, * @rules: ruleset to search * @label: label to check access permissions for * @state: state to start match in - * @subns: whether to do permission checks on components in a subns + * @inview: whether to match labels in view or only in scope * @request: permissions to request * @perms: perms struct to set * @@ -1291,7 +1291,7 @@ static inline aa_state_t match_component(struct aa_profile *profile, static int label_compound_match(struct aa_profile *profile, struct aa_ruleset *rules, struct aa_label *label, - aa_state_t state, bool subns, u32 request, + aa_state_t state, bool inview, u32 request, struct aa_perms *perms) { struct aa_profile *tp; @@ -1299,7 +1299,7 @@ static int label_compound_match(struct aa_profile *profile, /* find first subcomponent that is visible */ label_for_each(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = match_component(profile, rules, tp, state); if (!state) @@ -1313,7 +1313,7 @@ static int label_compound_match(struct aa_profile *profile, next: label_for_each_cont(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = aa_dfa_match(rules->policy->dfa, state, "//&"); state = match_component(profile, rules, tp, state); @@ -1334,7 +1334,7 @@ static int label_compound_match(struct aa_profile *profile, * @rules: ruleset to search * @label: label to check access permissions for * @start: state to start match in - * @subns: whether to do permission checks on components in a subns + * @inview: whether to match labels in view or only in scope * @request: permissions to request * @perms: an initialized perms struct to add accumulation to * @@ -1347,7 +1347,7 @@ static int label_compound_match(struct aa_profile *profile, static int label_components_match(struct aa_profile *profile, struct aa_ruleset *rules, struct aa_label *label, aa_state_t start, - bool subns, u32 request, + bool inview, u32 request, struct aa_perms *perms) { struct aa_profile *tp; @@ -1357,7 +1357,7 @@ static int label_components_match(struct aa_profile *profile, /* find first subcomponent to test */ label_for_each(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = match_component(profile, rules, tp, start); if (!state) @@ -1372,7 +1372,7 @@ static int label_components_match(struct aa_profile *profile, tmp = *aa_lookup_perms(rules->policy, state); aa_perms_accum(perms, &tmp); label_for_each_cont(i, label, tp) { - if (!aa_ns_visible(profile->ns, tp->ns, subns)) + if (!aa_ns_visible(profile->ns, tp->ns, inview)) continue; state = match_component(profile, rules, tp, start); if (!state) @@ -1397,24 +1397,24 @@ static int label_components_match(struct aa_profile *profile, * @rules: ruleset to search * @label: label to match (NOT NULL) * @state: state to start in - * @subns: whether to match subns components + * @inview: whether to match labels in view or only in scope * @request: permission request * @perms: Returns computed perms (NOT NULL) * * Returns: the state the match finished in, may be the none matching state */ int aa_label_match(struct aa_profile *profile, struct aa_ruleset *rules, - struct aa_label *label, aa_state_t state, bool subns, + struct aa_label *label, aa_state_t state, bool inview, u32 request, struct aa_perms *perms) { - aa_state_t tmp = label_compound_match(profile, rules, label, state, subns, - request, perms); + aa_state_t tmp = label_compound_match(profile, rules, label, state, + inview, request, perms); if ((perms->allow & request) == request) return tmp; /* failed compound_match try component matches */ *perms = allperms; - return label_components_match(profile, rules, label, state, subns, + return label_components_match(profile, rules, label, state, inview, request, perms); } diff --git a/security/apparmor/lib.c b/security/apparmor/lib.c index 82dbb97ad406..5d53c15e46a9 100644 --- a/security/apparmor/lib.c +++ b/security/apparmor/lib.c @@ -478,24 +478,22 @@ bool aa_policy_init(struct aa_policy *policy, const char *prefix, const char *name, gfp_t gfp) { char *hname; + size_t hname_sz; + INIT_LIST_HEAD(&policy->list); + INIT_LIST_HEAD(&policy->profiles); + hname_sz = (prefix ? strlen(prefix) + 2 : 0) + strlen(name) + 1; /* freed by policy_free */ - if (prefix) { - hname = aa_str_alloc(strlen(prefix) + strlen(name) + 3, gfp); - if (hname) - sprintf(hname, "%s//%s", prefix, name); - } else { - hname = aa_str_alloc(strlen(name) + 1, gfp); - if (hname) - strcpy(hname, name); - } + hname = aa_str_alloc(hname_sz, gfp); if (!hname) return false; + if (prefix) + scnprintf(hname, hname_sz, "%s//%s", prefix, name); + else + strscpy(hname, name, hname_sz); policy->hname = hname; /* base.name is a substring of fqname */ policy->name = basename(policy->hname); - INIT_LIST_HEAD(&policy->list); - INIT_LIST_HEAD(&policy->profiles); return true; } diff --git a/security/apparmor/mount.c b/security/apparmor/mount.c index 523570aa1a5a..2f5d918832c1 100644 --- a/security/apparmor/mount.c +++ b/security/apparmor/mount.c @@ -735,17 +735,11 @@ int aa_pivotroot(const struct cred *subj_cred, struct aa_label *label, build_pivotroot(subj_cred, profile, new_path, new_buffer, old_path, old_buffer)); - if (!target) { - info = "label build failed"; - error = -ENOMEM; - goto fail; - } else if (!IS_ERR(target)) { + AA_BUG(!target); + if (!IS_ERR(target)) { error = aa_replace_current_label(target); - if (error) { - /* TODO: audit target */ - aa_put_label(target); - goto out; - } + if (error) + goto fail; aa_put_label(target); } else /* already audited error */ @@ -763,7 +757,8 @@ int aa_pivotroot(const struct cred *subj_cred, struct aa_label *label, NULL /*new_name */, NULL /* old_name */, NULL, NULL, - 0, NULL, AA_MAY_PIVOTROOT, &nullperms, info, + 0, target->hname, AA_MAY_PIVOTROOT, &nullperms, info, error)); + aa_put_label(target); goto out; } diff --git a/security/apparmor/policy.c b/security/apparmor/policy.c index b0203b30db60..70d6315a059a 100644 --- a/security/apparmor/policy.c +++ b/security/apparmor/policy.c @@ -698,7 +698,7 @@ struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name, } /** - * aa_new_learning_profile - create or find a null-X learning profile + * __aa_new_learning_profile - create or find a null-X learning profile * @parent: profile that caused this profile to be created (NOT NULL) * @hat: true if the null- learning profile is a hat * @base: name to base the null profile off of @@ -715,14 +715,16 @@ struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name, * * Returns: new refcounted profile else NULL on failure */ -struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, - const char *base, gfp_t gfp) +struct aa_profile *__aa_new_learning_profile(struct aa_profile *parent, + bool hat, const char *base, + gfp_t gfp) { struct aa_profile *p, *profile; const char *bname; char *name = NULL; AA_BUG(!parent); + AA_BUG(!mutex_is_locked(&parent->ns->lock)); if (base) { name = kmalloc(strlen(parent->base.hname) + 8 + strlen(base), @@ -754,7 +756,6 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, if (hat) profile->label.flags |= FLAG_HAT; - mutex_lock_nested(&profile->ns->lock, profile->ns->level); p = __find_child(&parent->base.profiles, bname); if (p) { aa_free_profile(profile); @@ -762,7 +763,6 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, } else { __add_profile(&parent->base.profiles, profile); } - mutex_unlock(&profile->ns->lock); /* refcount released by caller */ out: @@ -776,6 +776,18 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, return NULL; } +struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, + const char *base, gfp_t gfp) +{ + struct aa_profile *profile; + + mutex_lock_nested(&parent->ns->lock, parent->ns->level); + profile = __aa_new_learning_profile(parent, hat, base, gfp); + mutex_unlock(&parent->ns->lock); + + return profile; +} + /** * replacement_allowed - test to see if replacement is allowed * @profile: profile to test if it can be replaced (MAYBE NULL) diff --git a/security/integrity/Makefile b/security/integrity/Makefile index 92b63039c654..45dfdedbdad4 100644 --- a/security/integrity/Makefile +++ b/security/integrity/Makefile @@ -18,6 +18,7 @@ integrity-$(CONFIG_LOAD_IPL_KEYS) += platform_certs/load_ipl_s390.o integrity-$(CONFIG_LOAD_PPC_KEYS) += platform_certs/efi_parser.o \ platform_certs/load_powerpc.o \ platform_certs/keyring_handler.o +integrity-$(CONFIG_EFI) += efi_secureboot.o # The relative order of the 'ima' and 'evm' LSMs depends on the order below. obj-$(CONFIG_IMA) += ima/ obj-$(CONFIG_EVM) += evm/ diff --git a/security/integrity/efi_secureboot.c b/security/integrity/efi_secureboot.c new file mode 100644 index 000000000000..bfd4260a83a3 --- /dev/null +++ b/security/integrity/efi_secureboot.c @@ -0,0 +1,56 @@ +// SPDX-License-Identifier: GPL-1.0+ +/* + * Copyright (C) 2018 IBM Corporation + */ +#include +#include +#include + +#ifndef arch_efi_boot_mode +#define arch_efi_boot_mode efi_secureboot_mode_unset +#endif + +static enum efi_secureboot_mode get_sb_mode(void) +{ + enum efi_secureboot_mode mode; + + if (!efi_rt_services_supported(EFI_RT_SUPPORTED_GET_VARIABLE)) { + pr_info("integrity: secureboot mode unknown, no efi\n"); + return efi_secureboot_mode_unknown; + } + + mode = efi_get_secureboot_mode(efi.get_variable); + if (mode == efi_secureboot_mode_disabled) + pr_info("integrity: secureboot mode disabled\n"); + else if (mode == efi_secureboot_mode_unknown) + pr_info("integrity: secureboot mode unknown\n"); + else + pr_info("integrity: secureboot mode enabled\n"); + return mode; +} + +/* + * Query secure boot status + * + * Note don't call this function too early e.g. in __setup hook otherwise the + * kernel may hang when calling efi_get_secureboot_mode. + * + */ +bool arch_get_secureboot(void) +{ + static enum efi_secureboot_mode sb_mode; + static bool initialized; + + if (!initialized && efi_enabled(EFI_BOOT)) { + sb_mode = arch_efi_boot_mode; + + if (sb_mode == efi_secureboot_mode_unset) + sb_mode = get_sb_mode(); + initialized = true; + } + + if (sb_mode == efi_secureboot_mode_enabled) + return true; + else + return false; +} diff --git a/security/integrity/ima/ima_appraise.c b/security/integrity/ima/ima_appraise.c index b7fc3e7ee37e..6a33484fcd6f 100644 --- a/security/integrity/ima/ima_appraise.c +++ b/security/integrity/ima/ima_appraise.c @@ -27,7 +27,7 @@ core_param(ima_appraise, ima_appraise_cmdline_default, charp, 0); void __init ima_appraise_parse_cmdline(void) { const char *str = ima_appraise_cmdline_default; - bool sb_state = arch_ima_get_secureboot(); + bool sb_state = arch_get_secureboot(); int appraisal_state = ima_appraise; if (!str) diff --git a/security/integrity/ima/ima_efi.c b/security/integrity/ima/ima_efi.c index 138029bfcce1..78191879dd98 100644 --- a/security/integrity/ima/ima_efi.c +++ b/security/integrity/ima/ima_efi.c @@ -2,52 +2,9 @@ /* * Copyright (C) 2018 IBM Corporation */ -#include #include #include -#include - -#ifndef arch_ima_efi_boot_mode -#define arch_ima_efi_boot_mode efi_secureboot_mode_unset -#endif - -static enum efi_secureboot_mode get_sb_mode(void) -{ - enum efi_secureboot_mode mode; - - if (!efi_rt_services_supported(EFI_RT_SUPPORTED_GET_VARIABLE)) { - pr_info("ima: secureboot mode unknown, no efi\n"); - return efi_secureboot_mode_unknown; - } - - mode = efi_get_secureboot_mode(efi.get_variable); - if (mode == efi_secureboot_mode_disabled) - pr_info("ima: secureboot mode disabled\n"); - else if (mode == efi_secureboot_mode_unknown) - pr_info("ima: secureboot mode unknown\n"); - else - pr_info("ima: secureboot mode enabled\n"); - return mode; -} - -bool arch_ima_get_secureboot(void) -{ - static enum efi_secureboot_mode sb_mode; - static bool initialized; - - if (!initialized && efi_enabled(EFI_BOOT)) { - sb_mode = arch_ima_efi_boot_mode; - - if (sb_mode == efi_secureboot_mode_unset) - sb_mode = get_sb_mode(); - initialized = true; - } - - if (sb_mode == efi_secureboot_mode_enabled) - return true; - else - return false; -} +#include /* secureboot arch rules */ static const char * const sb_arch_rules[] = { @@ -67,7 +24,7 @@ static const char * const sb_arch_rules[] = { const char * const *arch_get_ima_policy(void) { - if (IS_ENABLED(CONFIG_IMA_ARCH_POLICY) && arch_ima_get_secureboot()) { + if (IS_ENABLED(CONFIG_IMA_ARCH_POLICY) && arch_get_secureboot()) { if (IS_ENABLED(CONFIG_MODULE_SIG)) set_module_sig_enforced(); if (IS_ENABLED(CONFIG_KEXEC_SIG)) diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c index 2f58e8208997..e51b53b7789b 100644 --- a/security/integrity/ima/ima_main.c +++ b/security/integrity/ima/ima_main.c @@ -978,8 +978,7 @@ static int ima_load_data(enum kernel_load_data_id id, bool contents) switch (id) { case LOADING_KEXEC_IMAGE: - if (IS_ENABLED(CONFIG_KEXEC_SIG) - && arch_ima_get_secureboot()) { + if (IS_ENABLED(CONFIG_KEXEC_SIG) && arch_get_secureboot()) { pr_err("impossible to appraise a kernel image without a file descriptor; try using kexec_file_load syscall.\n"); return -EACCES; } diff --git a/security/integrity/integrity.h b/security/integrity/integrity.h index c2c2da691123..17b037b3fd09 100644 --- a/security/integrity/integrity.h +++ b/security/integrity/integrity.h @@ -14,6 +14,7 @@ #include #include +#include #include #include #include diff --git a/security/integrity/platform_certs/load_uefi.c b/security/integrity/platform_certs/load_uefi.c index d1fdd113450a..c0d6948446c3 100644 --- a/security/integrity/platform_certs/load_uefi.c +++ b/security/integrity/platform_certs/load_uefi.c @@ -212,7 +212,7 @@ static int __init load_uefi_certs(void) } /* the MOK/MOKx can not be trusted when secure boot is disabled */ - if (!arch_ima_get_secureboot()) + if (!arch_get_secureboot()) return 0; mokx = get_cert_list(L"MokListXRT", &mok_var, &mokxsize, &status); diff --git a/security/smack/smack.h b/security/smack/smack.h index 759343a6bbae..4f7f4760e432 100644 --- a/security/smack/smack.h +++ b/security/smack/smack.h @@ -303,8 +303,9 @@ int smack_populate_secattr(struct smack_known *skp); * Shared data. */ extern int smack_enabled __initdata; -extern int smack_cipso_direct; -extern int smack_cipso_mapped; +extern u8 smack_cipso_auto_level[2]; +#define smack_cipso_direct (+smack_cipso_auto_level[0]) +#define smack_cipso_mapped (+smack_cipso_auto_level[1]) extern struct smack_known *smack_net_ambient; extern struct smack_known *smack_syslog_label; #ifdef CONFIG_SECURITY_SMACK_BRINGUP diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c index 0d984fdf211a..217304cd20c3 100644 --- a/security/smack/smack_lsm.c +++ b/security/smack/smack_lsm.c @@ -130,12 +130,13 @@ static int smk_bu_note(char *note, struct smack_known *sskp, #define smk_bu_note(note, sskp, oskp, mode, RC) (RC) #endif -#ifdef CONFIG_SECURITY_SMACK_BRINGUP -static int smk_bu_current(char *note, struct smack_known *oskp, - int mode, int rc) +static int +smk_bu_tsk_to_obj(struct task_struct *tsk, const struct task_smack *tsp, + char *note, struct smack_known *oskp, int mode, int rc) { - struct task_smack *tsp = smack_cred(current_cred()); +#ifdef CONFIG_SECURITY_SMACK_BRINGUP char acc[SMK_NUM_ACCESS_TYPE + 1]; + char comm[TASK_COMM_LEN]; if (rc <= 0) return rc; @@ -143,14 +144,22 @@ static int smk_bu_current(char *note, struct smack_known *oskp, rc = 0; smk_bu_mode(mode, acc); + pr_info("Smack %s: (%s %s %s) %s %s\n", smk_bu_mess[rc], - tsp->smk_task->smk_known, oskp->smk_known, - acc, current->comm, note); + smk_of_task(tsp)->smk_known, oskp->smk_known, + acc, get_task_comm(comm, tsk), note); return 0; -} #else -#define smk_bu_current(note, oskp, mode, RC) (RC) + return rc; #endif +} + +static int smk_bu_current(char *note, struct smack_known *oskp, + int mode, int rc) +{ + return smk_bu_tsk_to_obj(current, smack_cred(current_cred()), + note, oskp, mode, rc); +} #ifdef CONFIG_SECURITY_SMACK_BRINGUP static int smk_bu_task(struct task_struct *otp, int mode, int rc) @@ -1312,7 +1321,7 @@ static int smack_inode_getattr(const struct path *path) */ static int smack_inode_xattr_skipcap(const char *name) { - if (strncmp(name, XATTR_SMACK_SUFFIX, strlen(XATTR_SMACK_SUFFIX))) + if (strncmp(name, XATTR_SMACK_SUFFIX, strlen(XATTR_SMACK_SUFFIX)) == 0) return 0; if (strcmp(name, XATTR_NAME_SMACK) == 0 || @@ -3353,14 +3362,20 @@ static int smack_sem_semop(struct kern_ipc_perm *isp, struct sembuf *sops, } /** - * smk_curacc_msq : helper to check if current has access on msq - * @isp : the msq + * smk_tskacc_msq : helper to check if tsk has access on msq + * @tsk: the task that requests access + * @isp : the sysv msg queue permissions * @access : access requested * - * return 0 if current has access, error otherwise + * return 0 if tsk has access, error otherwise */ -static int smk_curacc_msq(struct kern_ipc_perm *isp, int access) +static int +smk_tskacc_msq(struct task_struct *tsk, struct kern_ipc_perm *isp, int access) { + const bool tsk_is_current = (tsk == current); + const struct cred * const tsk_cred = + (tsk_is_current ? current_cred() : get_task_cred(tsk)); + struct task_smack * const tsp = smack_cred(tsk_cred); struct smack_known *msp = smack_of_ipc(isp); struct smk_audit_info ad; int rc; @@ -3369,11 +3384,25 @@ static int smk_curacc_msq(struct kern_ipc_perm *isp, int access) smk_ad_init(&ad, __func__, LSM_AUDIT_DATA_IPC); ad.a.u.ipc_id = isp->id; #endif - rc = smk_curacc(msp, access, &ad); - rc = smk_bu_current("msq", msp, access, rc); + rc = smk_tskacc(tsp, msp, access, &ad); + rc = smk_bu_tsk_to_obj(tsk, tsp, "msq", msp, access, rc); + if (!tsk_is_current) + put_cred(tsk_cred); return rc; } +/** + * smk_curacc_msq : helper to check if current has access on msq + * @isp : the sysv msg queue permissions + * @access : access requested + * + * return 0 if current has access, error otherwise + */ +static int smk_curacc_msq(struct kern_ipc_perm *isp, int access) +{ + return smk_tskacc_msq(current, isp, access); +} + /** * smack_msg_queue_associate - Smack access check for msg_queue * @isp: the object @@ -3441,21 +3470,21 @@ static int smack_msg_queue_msgsnd(struct kern_ipc_perm *isp, struct msg_msg *msg } /** - * smack_msg_queue_msgrcv - Smack access check for msg_queue + * smack_msg_queue_msgrcv - check it target has r/w access to msg_queue * @isp: the object * @msg: unused - * @target: unused + * @target: the task that msgrcv() from the queue * @type: unused * @mode: unused * - * Returns 0 if current has read and write access, error code otherwise + * Returns 0 if target has read and write access, error code otherwise */ static int smack_msg_queue_msgrcv(struct kern_ipc_perm *isp, struct msg_msg *msg, struct task_struct *target, long type, int mode) { - return smk_curacc_msq(isp, MAY_READWRITE); + return smk_tskacc_msq(target, isp, MAY_READWRITE); } /** diff --git a/security/smack/smackfs.c b/security/smack/smackfs.c index d27d9140dda2..32db00102d1d 100644 --- a/security/smack/smackfs.c +++ b/security/smack/smackfs.c @@ -83,18 +83,27 @@ static DEFINE_MUTEX(smk_net6addr_lock); struct smack_known *smack_net_ambient; /* - * This is the level in a CIPSO header that indicates a + * Sensitivity levels for automatically created CIPSO labels. + * See smack_access.c`smack_populate_secattr() + * + * [0] "direct" labeling, label length < SMK_CIPSOLEN(24): * smack label is contained directly in the category set. * It can be reset via smackfs/direct - */ -int smack_cipso_direct = SMACK_CIPSO_DIRECT_DEFAULT; - -/* - * This is the level in a CIPSO header that indicates a + * + * [1] "mapped" labeling, label length >= SMK_CIPSOLEN(24): * secid is contained directly in the category set. * It can be reset via smackfs/mapped */ -int smack_cipso_mapped = SMACK_CIPSO_MAPPED_DEFAULT; +u8 smack_cipso_auto_level[2] = { + SMACK_CIPSO_DIRECT_DEFAULT, + SMACK_CIPSO_MAPPED_DEFAULT, +}; + +static int +smk_cipso_auto_level_idx(const struct file *file) +{ + return (file_inode(file)->i_ino != SMK_DIRECT); +} #ifdef CONFIG_SECURITY_SMACK_BRINGUP /* @@ -1598,24 +1607,17 @@ static ssize_t smk_read_doi(struct file *filp, char __user *buf, static ssize_t smk_write_doi(struct file *file, const char __user *buf, size_t count, loff_t *ppos) { - char temp[80]; - unsigned long u; + int ret; + u32 u; if (!smack_privileged(CAP_MAC_ADMIN)) return -EPERM; - if (count >= sizeof(temp) || count == 0) - return -EINVAL; - - if (copy_from_user(temp, buf, count) != 0) - return -EFAULT; + ret = kstrtou32_from_user(buf, count, 10, &u); + if (unlikely(ret)) + return ret; - temp[count] = '\0'; - - if (kstrtoul(temp, 10, &u)) - return -EINVAL; - - if (u == CIPSO_V4_DOI_UNKNOWN || u > U32_MAX) + if (u == CIPSO_V4_DOI_UNKNOWN) return -EINVAL; return smk_cipso_doi(u, GFP_KERNEL) ? : count; @@ -1628,158 +1630,80 @@ static const struct file_operations smk_doi_ops = { }; /** - * smk_read_direct - read() for /smack/direct - * @filp: file pointer, not actually used + * smk_read_cipso_auto_level - read() for smackfs/direct and smackfs/mapped + * @filp: file pointer * @buf: where to put the result * @count: maximum to send along * @ppos: where to start * * Returns number of bytes read or error code, as appropriate */ -static ssize_t smk_read_direct(struct file *filp, char __user *buf, +static ssize_t smk_read_cipso_auto_level(struct file *filp, char __user *buf, size_t count, loff_t *ppos) { - char temp[80]; - ssize_t rc; + char temp[sizeof "255"]; + int n; if (*ppos != 0) return 0; - sprintf(temp, "%d", smack_cipso_direct); - rc = simple_read_from_buffer(buf, count, ppos, temp, strlen(temp)); - - return rc; + n = sprintf(temp, "%u", (unsigned int)smack_cipso_auto_level[ + smk_cipso_auto_level_idx(filp)]); + return simple_read_from_buffer(buf, count, ppos, temp, n); } /** - * smk_write_direct - write() for /smack/direct - * @file: file pointer, not actually used + * smk_write_cipso_auto_level - write() for smackfs/direct and smackfs/mapped + * @filp: file pointer * @buf: where to get the data from * @count: bytes sent * @ppos: where to start * * Returns number of bytes written or error code, as appropriate */ -static ssize_t smk_write_direct(struct file *file, const char __user *buf, - size_t count, loff_t *ppos) +static ssize_t +smk_write_cipso_auto_level(struct file *filp, const char __user *buf, + size_t count, loff_t *ppos) { - struct smack_known *skp; - char temp[80]; - int i; + int ret, idx; + u8 i, old_lvl; if (!smack_privileged(CAP_MAC_ADMIN)) return -EPERM; - - if (count >= sizeof(temp) || count == 0) - return -EINVAL; - - if (copy_from_user(temp, buf, count) != 0) - return -EFAULT; - - temp[count] = '\0'; - - if (sscanf(temp, "%d", &i) != 1) - return -EINVAL; - /* - * Don't do anything if the value hasn't actually changed. - * If it is changing reset the level on entries that were - * set up to be direct when they were created. + * draft-ietf-cipso-ipsecurity-01 (CIPSO 2.2), 3.4.2.4: + * "Sensitivity Level is 1 octet in length. Its value is from 0 to 255" */ - if (smack_cipso_direct != i) { - mutex_lock(&smack_known_lock); - list_for_each_entry_rcu(skp, &smack_known_list, list) - if (skp->smk_netlabel.attr.mls.lvl == - smack_cipso_direct) - skp->smk_netlabel.attr.mls.lvl = i; - smack_cipso_direct = i; - mutex_unlock(&smack_known_lock); - } - - return count; -} - -static const struct file_operations smk_direct_ops = { - .read = smk_read_direct, - .write = smk_write_direct, - .llseek = default_llseek, -}; - -/** - * smk_read_mapped - read() for /smack/mapped - * @filp: file pointer, not actually used - * @buf: where to put the result - * @count: maximum to send along - * @ppos: where to start - * - * Returns number of bytes read or error code, as appropriate - */ -static ssize_t smk_read_mapped(struct file *filp, char __user *buf, - size_t count, loff_t *ppos) -{ - char temp[80]; - ssize_t rc; - - if (*ppos != 0) - return 0; - - sprintf(temp, "%d", smack_cipso_mapped); - rc = simple_read_from_buffer(buf, count, ppos, temp, strlen(temp)); - - return rc; -} - -/** - * smk_write_mapped - write() for /smack/mapped - * @file: file pointer, not actually used - * @buf: where to get the data from - * @count: bytes sent - * @ppos: where to start - * - * Returns number of bytes written or error code, as appropriate - */ -static ssize_t smk_write_mapped(struct file *file, const char __user *buf, - size_t count, loff_t *ppos) -{ - struct smack_known *skp; - char temp[80]; - int i; - - if (!smack_privileged(CAP_MAC_ADMIN)) - return -EPERM; - - if (count >= sizeof(temp) || count == 0) - return -EINVAL; - - if (copy_from_user(temp, buf, count) != 0) - return -EFAULT; - - temp[count] = '\0'; - - if (sscanf(temp, "%d", &i) != 1) - return -EINVAL; + ret = kstrtou8_from_user(buf, count, 10, &i); + if (unlikely(ret)) + return ret; /* * Don't do anything if the value hasn't actually changed. * If it is changing reset the level on entries that were - * set up to be mapped when they were created. + * set up to be "auto" level when they were created. */ - if (smack_cipso_mapped != i) { + idx = smk_cipso_auto_level_idx(filp); + old_lvl = smack_cipso_auto_level[idx]; + + if (old_lvl != i) { + struct smack_known *skp; mutex_lock(&smack_known_lock); list_for_each_entry_rcu(skp, &smack_known_list, list) if (skp->smk_netlabel.attr.mls.lvl == - smack_cipso_mapped) + old_lvl) skp->smk_netlabel.attr.mls.lvl = i; - smack_cipso_mapped = i; + smack_cipso_auto_level[idx] = i; mutex_unlock(&smack_known_lock); } return count; } -static const struct file_operations smk_mapped_ops = { - .read = smk_read_mapped, - .write = smk_write_mapped, +static const struct file_operations +smk_cipso_auto_level_ops = { + .read = smk_read_cipso_auto_level, + .write = smk_write_cipso_auto_level, .llseek = default_llseek, }; @@ -2179,22 +2103,15 @@ static ssize_t smk_read_logging(struct file *filp, char __user *buf, static ssize_t smk_write_logging(struct file *file, const char __user *buf, size_t count, loff_t *ppos) { - char temp[32]; - int i; + int i, ret; if (!smack_privileged(CAP_MAC_ADMIN)) return -EPERM; - if (count >= sizeof(temp) || count == 0) - return -EINVAL; + ret = kstrtos32_from_user(buf, count, 10, &i); + if (unlikely(ret)) + return ret; - if (copy_from_user(temp, buf, count) != 0) - return -EFAULT; - - temp[count] = '\0'; - - if (sscanf(temp, "%d", &i) != 1) - return -EINVAL; if (i < 0 || i > 3) return -EINVAL; log_policy = i; @@ -2838,22 +2755,15 @@ static ssize_t smk_read_ptrace(struct file *filp, char __user *buf, static ssize_t smk_write_ptrace(struct file *file, const char __user *buf, size_t count, loff_t *ppos) { - char temp[32]; - int i; + int i, ret; if (!smack_privileged(CAP_MAC_ADMIN)) return -EPERM; - if (*ppos != 0 || count >= sizeof(temp) || count == 0) - return -EINVAL; - - if (copy_from_user(temp, buf, count) != 0) - return -EFAULT; - - temp[count] = '\0'; + ret = kstrtos32_from_user(buf, count, 10, &i); + if (unlikely(ret)) + return ret; - if (sscanf(temp, "%d", &i) != 1) - return -EINVAL; if (i < SMACK_PTRACE_DEFAULT || i > SMACK_PTRACE_MAX) return -EINVAL; smack_ptrace_rule = i; @@ -2888,7 +2798,7 @@ static int smk_fill_super(struct super_block *sb, struct fs_context *fc) [SMK_DOI] = { "doi", &smk_doi_ops, S_IRUGO|S_IWUSR}, [SMK_DIRECT] = { - "direct", &smk_direct_ops, S_IRUGO|S_IWUSR}, + "direct", &smk_cipso_auto_level_ops, 0644}, [SMK_AMBIENT] = { "ambient", &smk_ambient_ops, S_IRUGO|S_IWUSR}, [SMK_NET4ADDR] = { @@ -2904,7 +2814,7 @@ static int smk_fill_super(struct super_block *sb, struct fs_context *fc) [SMK_ACCESSES] = { "access", &smk_access_ops, S_IRUGO|S_IWUGO}, [SMK_MAPPED] = { - "mapped", &smk_mapped_ops, S_IRUGO|S_IWUSR}, + "mapped", &smk_cipso_auto_level_ops, 0644}, [SMK_LOAD2] = { "load2", &smk_load2_ops, S_IRUGO|S_IWUSR}, [SMK_LOAD_SELF2] = { diff --git a/sound/core/control.c b/sound/core/control.c index c714f2e5596b..652c09e110f6 100644 --- a/sound/core/control.c +++ b/sound/core/control.c @@ -2219,7 +2219,6 @@ EXPORT_SYMBOL_GPL(snd_ctl_request_layer); */ void snd_ctl_register_layer(struct snd_ctl_layer_ops *lops) { - struct snd_card *card; int card_number; scoped_guard(rwsem_write, &snd_ctl_layer_rwsem) { @@ -2227,11 +2226,12 @@ void snd_ctl_register_layer(struct snd_ctl_layer_ops *lops) snd_ctl_layer = lops; } for (card_number = 0; card_number < SNDRV_CARDS; card_number++) { - card = snd_card_ref(card_number); + struct snd_card *card __free(snd_card_unref) = + snd_card_ref(card_number); + if (card) { scoped_guard(rwsem_read, &card->controls_rwsem) lops->lregister(card); - snd_card_unref(card); } } } diff --git a/sound/core/control_led.c b/sound/core/control_led.c index e33dfcf863cf..ae903af93dd3 100644 --- a/sound/core/control_led.c +++ b/sound/core/control_led.c @@ -240,8 +240,6 @@ static void snd_ctl_led_notify(struct snd_card *card, unsigned int mask, } } -DEFINE_FREE(snd_card_unref, struct snd_card *, if (_T) snd_card_unref(_T)) - static int snd_ctl_led_set_id(int card_number, struct snd_ctl_elem_id *id, unsigned int group, bool set) { @@ -257,6 +255,8 @@ static int snd_ctl_led_set_id(int card_number, struct snd_ctl_elem_id *id, kctl = snd_ctl_find_id(card, id); if (!kctl) return -ENOENT; + if (!kctl->info || !kctl->get) + return -EINVAL; ioff = snd_ctl_get_ioff(kctl, id); vd = &kctl->vd[ioff]; access = vd->access & SNDRV_CTL_ELEM_ACCESS_LED_MASK; @@ -758,18 +758,17 @@ static int __init snd_ctl_led_init(void) static void __exit snd_ctl_led_exit(void) { struct snd_ctl_led *led; - struct snd_card *card; unsigned int group, card_number; snd_ctl_disconnect_layer(&snd_ctl_led_lops); for (card_number = 0; card_number < SNDRV_CARDS; card_number++) { if (!snd_ctl_led_card_valid[card_number]) continue; - card = snd_card_ref(card_number); - if (card) { + struct snd_card *card __free(snd_card_unref) = + snd_card_ref(card_number); + + if (card) snd_ctl_led_sysfs_remove(card); - snd_card_unref(card); - } } for (group = 0; group < MAX_LED; group++) { led = &snd_ctl_leds[group]; diff --git a/sound/core/init.c b/sound/core/init.c index c8f992cff4a4..bc5182c5ce12 100644 --- a/sound/core/init.c +++ b/sound/core/init.c @@ -579,6 +579,8 @@ EXPORT_SYMBOL_GPL(snd_card_disconnect_sync); static int snd_card_do_free(struct snd_card *card) { + bool managed = card->managed; + card->releasing = true; #if IS_ENABLED(CONFIG_SND_MIXER_OSS) if (snd_mixer_oss_notify_callback) @@ -593,7 +595,7 @@ static int snd_card_do_free(struct snd_card *card) } if (card->release_completion) complete(card->release_completion); - if (!card->managed) + if (!managed) kfree(card); return 0; } diff --git a/sound/core/seq/seq_clientmgr.c b/sound/core/seq/seq_clientmgr.c index 0c31b19dd948..e73c1becb6e1 100644 --- a/sound/core/seq/seq_clientmgr.c +++ b/sound/core/seq/seq_clientmgr.c @@ -528,7 +528,7 @@ static int bounce_error_event(struct snd_seq_client *client, struct snd_seq_event *event, int err, int atomic, int hop) { - struct snd_seq_event bounce_ev; + struct snd_seq_event bounce_ev, quoted; int result; if (client == NULL || @@ -544,15 +544,19 @@ static int bounce_error_event(struct snd_seq_client *client, * For user clients, send SNDRV_SEQ_EVENT_BOUNCE with the * original event embedded as variable-length data. This * avoids exposing data.quote.event (a kernel pointer) to - * userspace. The variable-length path in snd_seq_event_dup() - * copies the event data from data.ext.ptr into chained cells, - * and snd_seq_expand_var_event() copies only the data content - * -- never the pointer -- to userspace. + * userspace. Sanitise the embedded copy too - a queued + * variable-length event carries the address of its own + * extension cell, and the payload goes out verbatim. */ + quoted = *event; + if (snd_seq_ev_is_variable("ed)) { + quoted.data.ext.len &= ~SNDRV_SEQ_EXT_MASK; + quoted.data.ext.ptr = NULL; + } bounce_ev.type = SNDRV_SEQ_EVENT_BOUNCE; bounce_ev.flags = SNDRV_SEQ_EVENT_LENGTH_VARIABLE; bounce_ev.data.ext.len = sizeof(struct snd_seq_event); - bounce_ev.data.ext.ptr = (char *)event; + bounce_ev.data.ext.ptr = (char *)"ed; } else { /* * For kernel clients, quote the event pointer directly. diff --git a/sound/core/seq/seq_midi.c b/sound/core/seq/seq_midi.c index 24d485caeb0a..f61f30b91180 100644 --- a/sound/core/seq/seq_midi.c +++ b/sound/core/seq/seq_midi.c @@ -42,9 +42,11 @@ struct seq_midisynth { struct snd_rawmidi *rmidi; int device; int subdevice; + struct snd_rawmidi_substream __rcu *input_substream; + snd_use_lock_t input_use_lock; /* in-flight event_input users */ struct snd_rawmidi_file input_rfile; - spinlock_t output_lock; /* protects output_rfile publication */ snd_use_lock_t output_use_lock; /* in-flight event_input users */ + struct snd_rawmidi_substream __rcu *output_substream; struct snd_rawmidi_file output_rfile; int seq_client; int seq_port; @@ -76,6 +78,14 @@ static void snd_midi_input_event(struct snd_rawmidi_substream *substream) msynth = runtime->private_data; if (msynth == NULL) return; + + scoped_guard(rcu) { + if (rcu_dereference(msynth->input_substream) != substream) + return; + + snd_use_lock_use(&msynth->input_use_lock); + } + memset(&ev, 0, sizeof(ev)); while (runtime->avail > 0) { res = snd_rawmidi_kernel_read(substream, buf, sizeof(buf)); @@ -95,6 +105,8 @@ static void snd_midi_input_event(struct snd_rawmidi_substream *substream) memset(&ev, 0, sizeof(ev)); } } + + snd_use_lock_free(&msynth->input_use_lock); } static int dump_midi(struct snd_rawmidi_substream *substream, const char *buf, int count) @@ -134,8 +146,8 @@ static int event_process_midi(struct snd_seq_event *ev, int direct, if (snd_BUG_ON(!msynth)) return -EINVAL; - scoped_guard(spinlock_irqsave, &msynth->output_lock) { - substream = msynth->output_rfile.output; + scoped_guard(rcu) { + substream = rcu_dereference(msynth->output_substream); if (!substream) return -ENODEV; snd_use_lock_use(&msynth->output_use_lock); @@ -177,7 +189,7 @@ static int snd_seq_midisynth_new(struct seq_midisynth *msynth, msynth->card = card; msynth->device = device; msynth->subdevice = subdevice; - spin_lock_init(&msynth->output_lock); + snd_use_lock_init(&msynth->input_use_lock); snd_use_lock_init(&msynth->output_use_lock); return 0; } @@ -188,28 +200,31 @@ static int midisynth_subscribe(void *private_data, struct snd_seq_port_subscribe int err; struct seq_midisynth *msynth = private_data; struct snd_rawmidi_runtime *runtime; + struct snd_rawmidi_file rfile = {}; struct snd_rawmidi_params params; /* open midi port */ err = snd_rawmidi_kernel_open(msynth->rmidi, msynth->subdevice, SNDRV_RAWMIDI_LFLG_INPUT, - &msynth->input_rfile); + &rfile); if (err < 0) { pr_debug("ALSA: seq_midi: midi input open failed!!!\n"); return err; } - runtime = msynth->input_rfile.input->runtime; + runtime = rfile.input->runtime; memset(¶ms, 0, sizeof(params)); params.avail_min = 1; params.buffer_size = input_buffer_size; - err = snd_rawmidi_input_params(msynth->input_rfile.input, ¶ms); + err = snd_rawmidi_input_params(rfile.input, ¶ms); if (err < 0) { - snd_rawmidi_kernel_release(&msynth->input_rfile); + snd_rawmidi_kernel_release(&rfile); return err; } snd_midi_event_reset_encode(msynth->parser); runtime->event = snd_midi_input_event; runtime->private_data = msynth; + msynth->input_rfile = rfile; + rcu_assign_pointer(msynth->input_substream, rfile.input); snd_rawmidi_kernel_read(msynth->input_rfile.input, NULL, 0); return 0; } @@ -219,10 +234,19 @@ static int midisynth_unsubscribe(void *private_data, struct snd_seq_port_subscri { int err; struct seq_midisynth *msynth = private_data; + struct snd_rawmidi_file rfile; - if (snd_BUG_ON(!msynth->input_rfile.input)) + rcu_assign_pointer(msynth->input_substream, NULL); + synchronize_rcu(); + snd_use_lock_sync(&msynth->input_use_lock); + + rfile = msynth->input_rfile; + msynth->input_rfile = (struct snd_rawmidi_file){}; + + if (snd_BUG_ON(!rfile.input)) return -EINVAL; - err = snd_rawmidi_kernel_release(&msynth->input_rfile); + + err = snd_rawmidi_kernel_release(&rfile); return err; } @@ -252,8 +276,8 @@ static int midisynth_use(void *private_data, struct snd_seq_port_subscribe *info return err; } snd_midi_event_reset_decode(msynth->parser); - scoped_guard(spinlock_irqsave, &msynth->output_lock) - msynth->output_rfile = rfile; + msynth->output_rfile = rfile; + rcu_assign_pointer(msynth->output_substream, rfile.output); return 0; } @@ -261,17 +285,16 @@ static int midisynth_use(void *private_data, struct snd_seq_port_subscribe *info static int midisynth_unuse(void *private_data, struct snd_seq_port_subscribe *info) { struct seq_midisynth *msynth = private_data; - struct snd_rawmidi_file rfile = {}; + struct snd_rawmidi_file rfile; - scoped_guard(spinlock_irqsave, &msynth->output_lock) { - rfile = msynth->output_rfile; - msynth->output_rfile = (struct snd_rawmidi_file){}; - } + rcu_assign_pointer(msynth->output_substream, NULL); + synchronize_rcu(); + snd_use_lock_sync(&msynth->output_use_lock); + rfile = msynth->output_rfile; + msynth->output_rfile = (struct snd_rawmidi_file){}; if (snd_BUG_ON(!rfile.output)) return -EINVAL; - - snd_use_lock_sync(&msynth->output_use_lock); snd_rawmidi_drain_output(rfile.output); return snd_rawmidi_kernel_release(&rfile); } diff --git a/sound/drivers/mtpav.c b/sound/drivers/mtpav.c index d31eadf4be5f..1ed6d4f0cb62 100644 --- a/sound/drivers/mtpav.c +++ b/sound/drivers/mtpav.c @@ -642,9 +642,7 @@ static void snd_mtpav_free(struct snd_card *card) { struct mtpav *crd = card->private_data; - guard(spinlock_irqsave)(&crd->spinlock); - if (crd->istimer > 0) - snd_mtpav_remove_output_timer(crd); + timer_shutdown_sync(&crd->timer); } /* diff --git a/sound/hda/common/proc.c b/sound/hda/common/proc.c index 5f3f61519ba6..2d26b4305bcc 100644 --- a/sound/hda/common/proc.c +++ b/sound/hda/common/proc.c @@ -624,7 +624,7 @@ static void print_conn_list(struct snd_info_buffer *buffer, /* Get Cache connections info */ cache_len = snd_hda_get_conn_list(codec, nid, &list); if (cache_len >= 0 && (cache_len != conn_len || - memcmp(list, conn, conn_len) != 0)) { + memcmp(list, conn, conn_len * sizeof(*conn)) != 0)) { snd_iprintf(buffer, " In-driver Connection: %d\n", cache_len); if (cache_len > 0) { snd_iprintf(buffer, " "); diff --git a/sound/pci/asihpi/hpi6000.c b/sound/pci/asihpi/hpi6000.c index b08578c93c6a..57a730be2674 100644 --- a/sound/pci/asihpi/hpi6000.c +++ b/sound/pci/asihpi/hpi6000.c @@ -537,6 +537,11 @@ static short create_adapter_obj(struct hpi_adapter_obj *pao, hr1.size = sizeof(hr1); error = hpi6000_message_response_sequence(pao, 0, &hm, &hr0); + if (error) { + HPI_DEBUG_LOG(ERROR, "message transport error %d\n", + error); + return error; + } if (hr0.error) { HPI_DEBUG_LOG(DEBUG, "message error %d\n", hr0.error); return hr0.error; diff --git a/sound/pci/ice1712/ice1712.c b/sound/pci/ice1712/ice1712.c index 4cec56769c0f..e20f6bfcbdb8 100644 --- a/sound/pci/ice1712/ice1712.c +++ b/sound/pci/ice1712/ice1712.c @@ -2523,7 +2523,7 @@ static int snd_ice1712_probe(struct pci_dev *pci, const struct pci_device_id *pci_id) { static int dev; - struct snd_card *card; + struct snd_card *card __free(snd_card_unref) = NULL; struct snd_ice1712 *ice; int pcm_dev = 0, err; const struct snd_ice1712_card_info * const *tbl, *c; @@ -2535,8 +2535,8 @@ static int snd_ice1712_probe(struct pci_dev *pci, return -ENOENT; } - err = snd_card_new(&pci->dev, index[dev], id[dev], THIS_MODULE, - sizeof(*ice), &card); + err = snd_devm_card_new(&pci->dev, index[dev], id[dev], THIS_MODULE, + sizeof(*ice), &card); if (err < 0) return err; ice = card->private_data; @@ -2640,6 +2640,7 @@ static int snd_ice1712_probe(struct pci_dev *pci, if (err < 0) return err; pci_set_drvdata(pci, card); + card = NULL; /* probe succeeded, don't release as error */ dev++; return 0; } diff --git a/sound/pci/via82xx_modem.c b/sound/pci/via82xx_modem.c index 6ce2cd88cda6..605a3a0053d9 100644 --- a/sound/pci/via82xx_modem.c +++ b/sound/pci/via82xx_modem.c @@ -575,24 +575,18 @@ static inline unsigned int calc_linear_pos(struct via82xx_modem *chip, viadev->bufsize2, viadev->idx_table[idx].offset, viadev->idx_table[idx].size, count); #endif - if (count && size < count) { + if (! count) + /* bogus count 0 on the DMA boundary? */ + res = viadev->idx_table[idx].offset; + else + /* count register returns full size + * when end of buffer is reached + */ + res = viadev->idx_table[idx].offset + size; + if (check_invalid_pos(viadev, res)) { dev_dbg(chip->card->dev, - "invalid via82xx_cur_ptr, using last valid pointer\n"); + "invalid via82xx_cur_ptr (2), using last valid pointer\n"); res = viadev->lastpos; - } else { - if (! count) - /* bogus count 0 on the DMA boundary? */ - res = viadev->idx_table[idx].offset; - else - /* count register returns full size - * when end of buffer is reached - */ - res = viadev->idx_table[idx].offset + size; - if (check_invalid_pos(viadev, res)) { - dev_dbg(chip->card->dev, - "invalid via82xx_cur_ptr (2), using last valid pointer\n"); - res = viadev->lastpos; - } } } viadev->lastpos = res; /* remember the last position */ diff --git a/sound/soc/apple/mca.c b/sound/soc/apple/mca.c index c4dcb2b54591..17b572ad8b99 100644 --- a/sound/soc/apple/mca.c +++ b/sound/soc/apple/mca.c @@ -210,10 +210,10 @@ static void mca_fe_early_trigger(struct snd_pcm_substream *substream, int cmd, SERDES_STATUS_EN | SERDES_STATUS_RST, SERDES_STATUS_RST); /* - * Experiments suggest that it takes at most ~1 us - * for the bit to clear, so wait 2 us for good measure. + * The SERDES cluster needs a bit of time to reset itself + * and settle before we start poking it. This is... slow... */ - udelay(2); + udelay(25); WARN_ON(readl_relaxed(cl->base + serdes_unit + REG_SERDES_STATUS) & SERDES_STATUS_RST); mca_modify(cl, serdes_conf, SERDES_CONF_SYNC_SEL, diff --git a/sound/soc/codecs/rt700-sdw.c b/sound/soc/codecs/rt700-sdw.c index 44543c0da177..f7bd793e3e67 100644 --- a/sound/soc/codecs/rt700-sdw.c +++ b/sound/soc/codecs/rt700-sdw.c @@ -459,10 +459,8 @@ static int rt700_sdw_remove(struct sdw_slave *slave) { struct rt700_priv *rt700 = dev_get_drvdata(&slave->dev); - if (rt700->hw_init) { - cancel_delayed_work_sync(&rt700->jack_detect_work); - cancel_delayed_work_sync(&rt700->jack_btn_check_work); - } + cancel_delayed_work_sync(&rt700->jack_detect_work); + cancel_delayed_work_sync(&rt700->jack_btn_check_work); pm_runtime_disable(&slave->dev); diff --git a/sound/soc/codecs/tas2783-sdw.c b/sound/soc/codecs/tas2783-sdw.c index e273b80d033e..b7f7b025efd3 100644 --- a/sound/soc/codecs/tas2783-sdw.c +++ b/sound/soc/codecs/tas2783-sdw.c @@ -1175,24 +1175,20 @@ static s32 tas2783_sdca_dev_resume(struct device *dev) { struct sdw_slave *slave = dev_to_sdw_dev(dev); struct tas2783_prv *tas_dev = dev_get_drvdata(dev); - unsigned long t; + int ret; - if (!slave->unattach_request) - goto regmap_sync; + ret = sdw_slave_wait_for_init(slave, TAS2783_PROBE_TIMEOUT); + if (ret) + return ret; - t = wait_for_completion_timeout(&slave->initialization_complete, - msecs_to_jiffies(TAS2783_PROBE_TIMEOUT)); - if (!t) { - dev_err(&slave->dev, "resume: initialization timed out\n"); - sdw_show_ping_status(slave->bus, true); - return -ETIMEDOUT; + regcache_cache_only(tas_dev->regmap, false); + ret = regcache_sync(tas_dev->regmap); + if (ret) { + regcache_cache_only(tas_dev->regmap, true); + regcache_mark_dirty(tas_dev->regmap); + return ret; } - slave->unattach_request = 0; - -regmap_sync: - regcache_cache_only(tas_dev->regmap, false); - regcache_sync(tas_dev->regmap); return 0; } @@ -1259,9 +1255,23 @@ static s32 tas_update_status(struct sdw_slave *slave, if (tas_dev->hw_init || tas_dev->status != SDW_SLAVE_ATTACHED) return 0; - /* updated the cache data to device */ regcache_cache_only(tas_dev->regmap, false); - regcache_sync(tas_dev->regmap); + + /* + * The device is attaching uninitialized: either this is the first + * attach, or it lost power (and with it all register and DSP state) + * while the controller was power-gated during system suspend. The + * cache still holds the pre-suspend values, and tas_io_init() below + * resets the device via TAS2783_SW_RESET anyway, so syncing it back + * is both useless and harmful: later read-modify-write updates would + * compare against stale data and skip the hardware write. + * + * Drop the cache instead, so that subsequent accesses see the real + * hardware state. Syncing after the reset is not an option either: + * the cache accepts registers for which tas2783_sdca_mbq_size() + * returns 0, and writing those back fails with -EINVAL. + */ + regcache_drop_region(tas_dev->regmap, 0, UINT_MAX); /* perform I/O transfers required for Slave initialization */ return tas_io_init(&slave->dev, slave); diff --git a/sound/soc/fsl/fsl-asoc-card.c b/sound/soc/fsl/fsl-asoc-card.c index 71113886e494..f660eeb8e545 100644 --- a/sound/soc/fsl/fsl-asoc-card.c +++ b/sound/soc/fsl/fsl-asoc-card.c @@ -651,8 +651,8 @@ static int fsl_asoc_card_probe(struct platform_device *pdev) cpu_pdev = of_find_device_by_node(cpu_np); if (!cpu_pdev) { - dev_err(&pdev->dev, "failed to find CPU DAI device\n"); - ret = -EINVAL; + ret = dev_err_probe(&pdev->dev, -EPROBE_DEFER, + "failed to find CPU DAI device\n"); goto fail; } diff --git a/sound/soc/fsl/fsl_audmix.c b/sound/soc/fsl/fsl_audmix.c index d9b0bd61755d..c9a7a41ef388 100644 --- a/sound/soc/fsl/fsl_audmix.c +++ b/sound/soc/fsl/fsl_audmix.c @@ -454,6 +454,9 @@ static const struct of_device_id fsl_audmix_ids[] = { }; MODULE_DEVICE_TABLE(of, fsl_audmix_ids); +static int fsl_audmix_runtime_resume(struct device *dev); +static int fsl_audmix_runtime_suspend(struct device *dev); + static int fsl_audmix_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; @@ -485,13 +488,25 @@ static int fsl_audmix_probe(struct platform_device *pdev) spin_lock_init(&priv->lock); platform_set_drvdata(pdev, priv); pm_runtime_enable(dev); + if (!pm_runtime_enabled(dev)) { + ret = fsl_audmix_runtime_resume(dev); + if (ret) + goto err_disable_pm; + } + + ret = pm_runtime_resume_and_get(dev); + if (ret < 0) + goto err_pm_get_sync; + + /* To enable regmap cache only when runtime PM enabled */ + pm_runtime_put(dev); ret = devm_snd_soc_register_component(dev, &fsl_audmix_component, fsl_audmix_dai, ARRAY_SIZE(fsl_audmix_dai)); if (ret) { dev_err(dev, "failed to register ASoC DAI\n"); - goto err_disable_pm; + goto err_pm_get_sync; } /* @@ -503,12 +518,15 @@ static int fsl_audmix_probe(struct platform_device *pdev) if (IS_ERR(priv->pdev)) { ret = PTR_ERR(priv->pdev); dev_err(dev, "failed to register platform: %d\n", ret); - goto err_disable_pm; + goto err_pm_get_sync; } } return 0; +err_pm_get_sync: + if (!pm_runtime_status_suspended(dev)) + fsl_audmix_runtime_suspend(dev); err_disable_pm: pm_runtime_disable(dev); return ret; @@ -519,6 +537,8 @@ static void fsl_audmix_remove(struct platform_device *pdev) struct fsl_audmix *priv = dev_get_drvdata(&pdev->dev); pm_runtime_disable(&pdev->dev); + if (!pm_runtime_status_suspended(&pdev->dev)) + fsl_audmix_runtime_suspend(&pdev->dev); if (priv->pdev) platform_device_unregister(priv->pdev); diff --git a/sound/soc/meson/meson-card-utils.c b/sound/soc/meson/meson-card-utils.c index cdb759b466ad..8617a4661a33 100644 --- a/sound/soc/meson/meson-card-utils.c +++ b/sound/soc/meson/meson-card-utils.c @@ -50,25 +50,20 @@ int meson_card_reallocate_links(struct snd_soc_card *card, num_links * sizeof(*priv->card.dai_link), GFP_KERNEL | __GFP_ZERO); if (!links) - goto err_links; + return -ENOMEM; + + priv->card.dai_link = links; + priv->card.num_links = num_links; ldata = krealloc(priv->link_data, num_links * sizeof(*priv->link_data), GFP_KERNEL | __GFP_ZERO); + /* meson_card_clean_references() will free the links on this error path */ if (!ldata) - goto err_ldata; + return -ENOMEM; - priv->card.dai_link = links; priv->link_data = ldata; - priv->card.num_links = num_links; return 0; - -err_ldata: - kfree(links); -err_links: - dev_err(priv->card.dev, "failed to allocate links\n"); - return -ENOMEM; - } EXPORT_SYMBOL_GPL(meson_card_reallocate_links); diff --git a/sound/soc/pxa/pxa-ssp.c b/sound/soc/pxa/pxa-ssp.c index b8a3cb8b7597..ba8cae8e85ef 100644 --- a/sound/soc/pxa/pxa-ssp.c +++ b/sound/soc/pxa/pxa-ssp.c @@ -769,13 +769,10 @@ static int pxa_ssp_probe(struct snd_soc_dai *dai) goto err_priv; } - priv->extclk = devm_clk_get(dev, "extclk"); + priv->extclk = devm_clk_get_optional(dev, "extclk"); if (IS_ERR(priv->extclk)) { ret = PTR_ERR(priv->extclk); - if (ret == -EPROBE_DEFER) - goto err_priv; - - priv->extclk = NULL; + goto err_priv; } } else { priv->ssp = pxa_ssp_request(dai->id + 1, "SoC audio"); diff --git a/sound/soc/qcom/qdsp6/q6apm.c b/sound/soc/qcom/qdsp6/q6apm.c index 6326649ba31b..16b2e227a6f1 100644 --- a/sound/soc/qcom/qdsp6/q6apm.c +++ b/sound/soc/qcom/qdsp6/q6apm.c @@ -700,14 +700,17 @@ EXPORT_SYMBOL_GPL(q6apm_graph_prepare); int q6apm_graph_start(struct q6apm_graph *graph) { struct audioreach_graph *ar_graph = graph->ar_graph; - int ret = 0; + int ret; - if (ar_graph->start_count == 0) + if (ar_graph->start_count == 0) { ret = audioreach_graph_mgmt_cmd(ar_graph, APM_CMD_GRAPH_START); + if (ret) + return ret; + } ar_graph->start_count++; - return ret; + return 0; } EXPORT_SYMBOL_GPL(q6apm_graph_start); @@ -715,6 +718,9 @@ int q6apm_graph_stop(struct q6apm_graph *graph) { struct audioreach_graph *ar_graph = graph->ar_graph; + if (ar_graph->start_count == 0) + return 0; + if (--ar_graph->start_count > 0) return 0; diff --git a/sound/soc/soc-dapm.c b/sound/soc/soc-dapm.c index 51fb09d56c5a..f640bbe319fa 100644 --- a/sound/soc/soc-dapm.c +++ b/sound/soc/soc-dapm.c @@ -3549,7 +3549,7 @@ int snd_soc_dapm_put_enum_double(struct snd_kcontrol *kcontrol, val = snd_soc_enum_item_to_val(e, item[0]) << e->shift_l; mask = e->mask << e->shift_l; if (e->shift_l != e->shift_r) { - if (item[1] > e->items) + if (item[1] >= e->items) return -EINVAL; val |= snd_soc_enum_item_to_val(e, item[1]) << e->shift_r; mask |= e->mask << e->shift_r; diff --git a/sound/soc/sof/topology.c b/sound/soc/sof/topology.c index d6134df874ee..5ddfa2cc51d1 100644 --- a/sound/soc/sof/topology.c +++ b/sound/soc/sof/topology.c @@ -839,6 +839,7 @@ static int sof_control_load_volume(struct snd_soc_component *scomp, struct snd_soc_tplg_mixer_control *mc = container_of(hdr, struct snd_soc_tplg_mixer_control, hdr); int tlv[SOF_TLV_ITEMS]; + u32 min, max; unsigned int mask; int ret; @@ -846,6 +847,11 @@ static int sof_control_load_volume(struct snd_soc_component *scomp, if (le32_to_cpu(mc->num_channels) > SND_SOC_TPLG_MAX_CHAN) return -EINVAL; + min = le32_to_cpu(mc->min); + max = le32_to_cpu(mc->max); + if (min > max || max >= INT_MAX) + return -EINVAL; + /* * If control has more than 2 channels we need to override the info. This is because even if * ASoC layer has defined topology's max channel count to SND_SOC_TPLG_MAX_CHAN = 8, the @@ -856,12 +862,12 @@ static int sof_control_load_volume(struct snd_soc_component *scomp, kc->info = snd_sof_volume_info; scontrol->comp_id = sdev->next_comp_id; - scontrol->min_volume_step = le32_to_cpu(mc->min); - scontrol->max_volume_step = le32_to_cpu(mc->max); + scontrol->min_volume_step = min; + scontrol->max_volume_step = max; scontrol->num_channels = le32_to_cpu(mc->num_channels); - scontrol->max = le32_to_cpu(mc->max); - if (le32_to_cpu(mc->max) == 1) + scontrol->max = max; + if (max == 1) goto skip; /* extract tlv data */ @@ -871,7 +877,7 @@ static int sof_control_load_volume(struct snd_soc_component *scomp, } /* set up volume table */ - ret = set_up_volume_table(scontrol, tlv, le32_to_cpu(mc->max) + 1); + ret = set_up_volume_table(scontrol, tlv, max + 1); if (ret < 0) { dev_err(scomp->dev, "error: setting up volume table\n"); return ret; @@ -904,7 +910,7 @@ static int sof_control_load_volume(struct snd_soc_component *scomp, return 0; err: - if (le32_to_cpu(mc->max) > 1) + if (max > 1) kfree(scontrol->volume_table); return ret; diff --git a/sound/soc/xilinx/xlnx_formatter_pcm.c b/sound/soc/xilinx/xlnx_formatter_pcm.c index 45b178148b98..4d90e96a677c 100644 --- a/sound/soc/xilinx/xlnx_formatter_pcm.c +++ b/sound/soc/xilinx/xlnx_formatter_pcm.c @@ -384,7 +384,7 @@ static int xlnx_formatter_pcm_open(struct snd_soc_component *component, if (err) { dev_err(component->dev, "Unable to set constraint on period bytes\n"); - return err; + goto error; } /* Resize the buffer bytes as divisible by 64 */ @@ -394,7 +394,7 @@ static int xlnx_formatter_pcm_open(struct snd_soc_component *component, if (err) { dev_err(component->dev, "Unable to set constraint on buffer bytes\n"); - return err; + goto error; } /* Set periods as integer multiple */ @@ -403,7 +403,7 @@ static int xlnx_formatter_pcm_open(struct snd_soc_component *component, if (err < 0) { dev_err(component->dev, "Unable to set constraint on periods to be integer\n"); - return err; + goto error; } /* enable DMA IOC irq */ @@ -412,6 +412,14 @@ static int xlnx_formatter_pcm_open(struct snd_soc_component *component, writel(val, stream_data->mmio + XLNX_AUD_CTRL); return 0; + +error: + if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) + adata->play_stream = NULL; + else + adata->capture_stream = NULL; + kfree(stream_data); + return err; } static int xlnx_formatter_pcm_close(struct snd_soc_component *component, diff --git a/sound/usb/fcp.c b/sound/usb/fcp.c index f94630a508c7..a17bf0028a8a 100644 --- a/sound/usb/fcp.c +++ b/sound/usb/fcp.c @@ -187,10 +187,6 @@ static int fcp_usb(struct usb_mixer_interface *mixer, u32 opcode, { struct fcp_data *private = mixer->private_data; struct usb_device *dev = mixer->chip->dev; - struct fcp_usb_packet *req __free(kfree) = NULL; - struct fcp_usb_packet *resp __free(kfree) = NULL; - size_t req_buf_size = struct_size(req, data, req_size); - size_t resp_buf_size = struct_size(resp, data, resp_size); int retries = 0; const int max_retries = 5; int err; @@ -198,10 +194,14 @@ static int fcp_usb(struct usb_mixer_interface *mixer, u32 opcode, if (!private->urb) return -ENODEV; + struct fcp_usb_packet *req __free(kfree) = NULL; + size_t req_buf_size = struct_size(req, data, req_size); req = kmalloc(req_buf_size, GFP_KERNEL); if (!req) return -ENOMEM; + struct fcp_usb_packet *resp __free(kfree) = NULL; + size_t resp_buf_size = struct_size(resp, data, resp_size); resp = kmalloc(resp_buf_size, GFP_KERNEL); if (!resp) return -ENOMEM; @@ -305,16 +305,17 @@ static int fcp_usb(struct usb_mixer_interface *mixer, u32 opcode, static int fcp_reinit(struct usb_mixer_interface *mixer) { struct fcp_data *private = mixer->private_data; - void *step0_resp __free(kfree) = NULL; - void *step2_resp __free(kfree) = NULL; if (private->urb) return 0; - step0_resp = kmalloc(private->step0_resp_size, GFP_KERNEL); + void *step0_resp __free(kfree) = + kmalloc(private->step0_resp_size, GFP_KERNEL); if (!step0_resp) return -ENOMEM; - step2_resp = kmalloc(private->step2_resp_size, GFP_KERNEL); + + void *step2_resp __free(kfree) = + kmalloc(private->step2_resp_size, GFP_KERNEL); if (!step2_resp) return -ENOMEM; @@ -472,7 +473,6 @@ static int fcp_ioctl_init(struct usb_mixer_interface *mixer, struct fcp_init init; struct usb_device *dev = mixer->chip->dev; struct fcp_data *private = mixer->private_data; - void *resp __free(kfree) = NULL; void *step2_resp; int err, buf_size; @@ -493,7 +493,8 @@ static int fcp_ioctl_init(struct usb_mixer_interface *mixer, /* Allocate response buffer */ buf_size = init.step0_resp_size + init.step2_resp_size; - resp = kmalloc(buf_size, GFP_KERNEL); + void *resp __free(kfree) = + kzalloc(buf_size, GFP_KERNEL); if (!resp) return -ENOMEM; @@ -627,7 +628,6 @@ static int fcp_ioctl_set_meter_map(struct usb_mixer_interface *mixer, { struct fcp_meter_map map; struct fcp_data *private = mixer->private_data; - s16 *tmp_map __free(kfree) = NULL; int err; if (copy_from_user(&map, arg, sizeof(map))) @@ -650,7 +650,8 @@ static int fcp_ioctl_set_meter_map(struct usb_mixer_interface *mixer, return -EINVAL; /* Allocate and copy the map data */ - tmp_map = memdup_array_user(arg->map, map.map_size, sizeof(s16)); + s16 *tmp_map __free(kfree) = + memdup_array_user(arg->map, map.map_size, sizeof(s16)); if (IS_ERR(tmp_map)) return PTR_ERR(tmp_map); @@ -660,17 +661,16 @@ static int fcp_ioctl_set_meter_map(struct usb_mixer_interface *mixer, /* If the control doesn't exist, create it */ if (!private->meter_ctl) { - s16 *new_map __free(kfree) = NULL; - __le32 *meter_levels __free(kfree) = NULL; - /* Allocate buffer for the map */ - new_map = kmalloc_array(map.map_size, sizeof(s16), GFP_KERNEL); + s16 *new_map __free(kfree) = + kmalloc_array(map.map_size, sizeof(s16), GFP_KERNEL); if (!new_map) return -ENOMEM; /* Allocate buffer for reading meter levels */ - meter_levels = kmalloc_array(map.meter_slots, sizeof(__le32), - GFP_KERNEL); + __le32 *meter_levels __free(kfree) = + kmalloc_array(map.meter_slots, sizeof(__le32), + GFP_KERNEL); if (!meter_levels) return -ENOMEM; @@ -1033,6 +1033,8 @@ static int fcp_init(struct usb_mixer_interface *mixer, step0_resp, private->step0_resp_size); if (err < 0) return err; + if (err != private->step0_resp_size) + return -EIO; err = fcp_init_notify(mixer); if (err < 0) diff --git a/sound/usb/mixer_scarlett2.c b/sound/usb/mixer_scarlett2.c index 46535926bb20..133e735dd3e9 100644 --- a/sound/usb/mixer_scarlett2.c +++ b/sound/usb/mixer_scarlett2.c @@ -2499,18 +2499,18 @@ static int scarlett2_usb( { struct scarlett2_data *private = mixer->private_data; struct usb_device *dev = mixer->chip->dev; - struct scarlett2_usb_packet *req __free(kfree) = NULL; - struct scarlett2_usb_packet *resp __free(kfree) = NULL; - size_t req_buf_size = struct_size(req, data, req_size); - size_t resp_buf_size = struct_size(resp, data, resp_size); int retries = 0; const int max_retries = 5; int err; + struct scarlett2_usb_packet *req __free(kfree) = NULL; + size_t req_buf_size = struct_size(req, data, req_size); req = kmalloc(req_buf_size, GFP_KERNEL); if (!req) return -ENOMEM; + struct scarlett2_usb_packet *resp __free(kfree) = NULL; + size_t resp_buf_size = struct_size(resp, data, resp_size); resp = kmalloc(resp_buf_size, GFP_KERNEL); if (!resp) return -ENOMEM; @@ -4065,9 +4065,9 @@ static int scarlett2_input_select_ctl_info( struct scarlett2_data *private = mixer->private_data; int inputs = private->info->gain_input_count; int i, err; - char **values __free(kfree) = NULL; + char **values __free(kfree) = + kcalloc(inputs, sizeof(char *), GFP_KERNEL); - values = kcalloc(inputs, sizeof(char *), GFP_KERNEL); if (!values) return -ENOMEM; @@ -9293,8 +9293,6 @@ static long scarlett2_hwdep_read(struct snd_hwdep *hw, __le32 len; } __packed req; - u8 *resp __free(kfree) = NULL; - /* Flash segment must first be selected */ if (private->flash_write_state != SCARLETT2_FLASH_WRITE_STATE_SELECTED) return -EINVAL; @@ -9332,7 +9330,8 @@ static long scarlett2_hwdep_read(struct snd_hwdep *hw, req.offset = cpu_to_le32(*offset); req.len = cpu_to_le32(count); - resp = kzalloc(count, GFP_KERNEL); + u8 *resp __free(kfree) = + kzalloc(count, GFP_KERNEL); if (!resp) return -ENOMEM; @@ -9480,7 +9479,6 @@ static ssize_t scarlett2_devmap_read( loff_t pos) { struct usb_mixer_interface *mixer = entry->private_data; - u8 *resp_buf __free(kfree) = NULL; const size_t block_size = SCARLETT2_DEVMAP_BLOCK_SIZE; size_t copied = 0; @@ -9490,7 +9488,8 @@ static ssize_t scarlett2_devmap_read( if (pos + count > entry->size) count = entry->size - pos; - resp_buf = kmalloc(block_size, GFP_KERNEL); + u8 *resp_buf __free(kfree) = + kmalloc(block_size, GFP_KERNEL); if (!resp_buf) return -ENOMEM; diff --git a/sound/usb/quirks.c b/sound/usb/quirks.c index 3d81b003a510..5eef4f3c5716 100644 --- a/sound/usb/quirks.c +++ b/sound/usb/quirks.c @@ -555,7 +555,6 @@ int snd_usb_create_quirk(struct snd_usb_audio *chip, static int snd_usb_extigy_boot_quirk(struct usb_device *dev, struct usb_interface *intf) { struct usb_host_config *config = dev->actconfig; - struct usb_device_descriptor *new_device_descriptor __free(kfree) = NULL; int err; if (le16_to_cpu(get_cfg_desc(config)->wTotalLength) == EXTIGY_FIRMWARE_SIZE_OLD || @@ -566,8 +565,8 @@ static int snd_usb_extigy_boot_quirk(struct usb_device *dev, struct usb_interfac 0x10, 0x43, 0x0001, 0x000a, NULL, 0); if (err < 0) dev_dbg(&dev->dev, "error sending boot message: %d\n", err); - - new_device_descriptor = kmalloc(sizeof(*new_device_descriptor), GFP_KERNEL); + struct usb_device_descriptor *new_device_descriptor __free(kfree) = + kmalloc(sizeof(*new_device_descriptor), GFP_KERNEL); if (!new_device_descriptor) return -ENOMEM; err = usb_get_descriptor(dev, USB_DT_DEVICE, 0, @@ -910,7 +909,6 @@ static void mbox2_setup_48_24_magic(struct usb_device *dev) static int snd_usb_mbox2_boot_quirk(struct usb_device *dev) { struct usb_host_config *config = dev->actconfig; - struct usb_device_descriptor *new_device_descriptor __free(kfree) = NULL; int err; u8 bootresponse[0x12]; int fwsize; @@ -945,7 +943,8 @@ static int snd_usb_mbox2_boot_quirk(struct usb_device *dev) dev_dbg(&dev->dev, "device initialised!\n"); - new_device_descriptor = kmalloc(sizeof(*new_device_descriptor), GFP_KERNEL); + struct usb_device_descriptor *new_device_descriptor __free(kfree) = + kmalloc(sizeof(*new_device_descriptor), GFP_KERNEL); if (!new_device_descriptor) return -ENOMEM; @@ -1267,7 +1266,6 @@ static void mbox3_setup_defaults(struct usb_device *dev) static int snd_usb_mbox3_boot_quirk(struct usb_device *dev) { struct usb_host_config *config = dev->actconfig; - struct usb_device_descriptor *new_device_descriptor __free(kfree) = NULL; int err; int descriptor_size; @@ -1280,7 +1278,8 @@ static int snd_usb_mbox3_boot_quirk(struct usb_device *dev) dev_dbg(&dev->dev, "MBOX3: device initialised!\n"); - new_device_descriptor = kmalloc(sizeof(*new_device_descriptor), GFP_KERNEL); + struct usb_device_descriptor *new_device_descriptor __free(kfree) = + kmalloc(sizeof(*new_device_descriptor), GFP_KERNEL); if (!new_device_descriptor) return -ENOMEM; diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c index c9de44a45778..941d73b737de 100644 --- a/tools/bpf/bpftool/map.c +++ b/tools/bpf/bpftool/map.c @@ -659,8 +659,6 @@ static int do_show_subset(int argc, char **argv) show_map_close_json(fds[i], &info); else show_map_close_plain(fds[i], &info); - - close(fds[i]); } if (json_output && nb_fds > 1) jsonw_end_array(json_wtr); /* root array */ @@ -790,6 +788,12 @@ static int maps_have_btf(int *fds, int nb_fds) static struct btf *btf_vmlinux; +static void free_btf_vmlinux(void) +{ + btf__free(btf_vmlinux); + btf_vmlinux = NULL; +} + static int get_map_kv_btf(const struct bpf_map_info *info, struct btf **btf) { int err = 0; @@ -889,7 +893,6 @@ map_dump(int fd, struct bpf_map_info *info, json_writer_t *wtr, exit_free: free(key); free(value); - close(fd); free_map_kv_btf(btf); return err; @@ -938,6 +941,7 @@ static int do_dump(int argc, char **argv) for (i = 0; i < nb_fds; i++) { if (bpf_map_get_info_by_fd(fds[i], &info, &len)) { p_err("can't get map info: %s", strerror(errno)); + err = -1; break; } err = map_dump(fds[i], &info, wtr, nb_fds > 1); @@ -958,7 +962,7 @@ static int do_dump(int argc, char **argv) close(fds[i]); exit_free: free(fds); - btf__free(btf_vmlinux); + free_btf_vmlinux(); return err; } @@ -1049,7 +1053,7 @@ static void print_key_value(struct bpf_map_info *info, void *key, btf_wtr = get_btf_writer(); if (!btf_wtr) { p_info("failed to create json writer for btf. falling back to plain output"); - btf__free(btf); + free_map_kv_btf(btf); btf = NULL; print_entry_plain(info, key, value); } else { @@ -1065,7 +1069,7 @@ static void print_key_value(struct bpf_map_info *info, void *key, } else { print_entry_plain(info, key, value); } - btf__free(btf); + free_map_kv_btf(btf); } static int do_lookup(int argc, char **argv) diff --git a/tools/bpf/bpftool/sign.c b/tools/bpf/bpftool/sign.c index f9b742f4bb10..1257dba8ef2f 100644 --- a/tools/bpf/bpftool/sign.c +++ b/tools/bpf/bpftool/sign.c @@ -175,8 +175,11 @@ int bpftool_prog_sign(struct bpf_load_and_run_opts *opts) goto cleanup; } - EVP_Digest(opts->insns, opts->insns_sz, opts->excl_prog_hash, - &opts->excl_prog_hash_sz, EVP_sha256(), NULL); + if (EVP_Digest(opts->insns, opts->insns_sz, opts->excl_prog_hash, + &opts->excl_prog_hash_sz, EVP_sha256(), NULL) != 1) { + err = -EIO; + goto cleanup; + } bd_out = BIO_new(BIO_s_mem()); if (!bd_out) { diff --git a/tools/bpf/bpftool/struct_ops.c b/tools/bpf/bpftool/struct_ops.c index aa43dead249c..835e5e561f7f 100644 --- a/tools/bpf/bpftool/struct_ops.c +++ b/tools/bpf/bpftool/struct_ops.c @@ -643,6 +643,10 @@ int do_struct_ops(int argc, char **argv) err = cmd_select(cmds, argc, argv, do_help); btf__free(btf_vmlinux); + btf_vmlinux = NULL; + map_info_type = NULL; + map_info_alloc_len = 0; + map_info_type_id = 0; return err; } diff --git a/tools/build/Makefile.feature b/tools/build/Makefile.feature index 32bbe29fe5f6..a0133d52aa8a 100644 --- a/tools/build/Makefile.feature +++ b/tools/build/Makefile.feature @@ -119,9 +119,7 @@ FEATURE_TESTS_EXTRA := \ libbfd-liberty-z \ libopencsd \ libperl \ - cxx \ llvm \ - clang \ libbpf \ libpfm4 \ libdebuginfod \ diff --git a/tools/build/feature/Makefile b/tools/build/feature/Makefile index 95646290cb89..c972304c3211 100644 --- a/tools/build/feature/Makefile +++ b/tools/build/feature/Makefile @@ -58,7 +58,6 @@ FILES= \ test-libbpf.bin \ test-get_cpuid.bin \ test-sdt.bin \ - test-cxx.bin \ test-gettid.bin \ test-jvmti.bin \ test-jvmti-cmlr.bin \ @@ -66,7 +65,6 @@ FILES= \ test-sched_getcpu.bin \ test-setns.bin \ test-libopencsd.bin \ - test-clang.bin \ test-llvm.bin \ test-llvm-perf.bin \ test-libaio.bin \ @@ -330,9 +328,6 @@ $(OUTPUT)test-libbpf.bin: $(OUTPUT)test-sdt.bin: $(BUILD) -$(OUTPUT)test-cxx.bin: - $(BUILDXX) -std=gnu++11 - $(OUTPUT)test-gettid.bin: $(BUILD) @@ -358,15 +353,6 @@ $(OUTPUT)test-llvm-perf.bin: $(shell $(LLVM_CONFIG) --system-libs) \ > $(@:.bin=.make.output) 2>&1 -$(OUTPUT)test-clang.bin: - $(BUILDXX) -std=gnu++17 \ - -I$(shell $(LLVM_CONFIG) --includedir) \ - -L$(shell $(LLVM_CONFIG) --libdir) \ - -Wl,--start-group -lclang-cpp -Wl,--end-group \ - $(shell $(LLVM_CONFIG) --libs Core option) \ - $(shell $(LLVM_CONFIG) --system-libs) \ - > $(@:.bin=.make.output) 2>&1 - -include $(OUTPUT)*.d $(OUTPUT)test-libaio.bin: diff --git a/tools/include/nolibc/arch-powerpc.h b/tools/include/nolibc/arch-powerpc.h index 204564bbcd32..c7d7a72c16dc 100644 --- a/tools/include/nolibc/arch-powerpc.h +++ b/tools/include/nolibc/arch-powerpc.h @@ -23,7 +23,7 @@ */ #define _NOLIBC_SYSCALL_CLOBBERLIST \ - "memory", "cr0", "r12", "r11", "r10", "r9" + "memory", "cr0", "ctr", "xer", "r12", "r11", "r10", "r9" #define my_syscall0(num) \ ({ \ diff --git a/tools/include/nolibc/arch-sparc.h b/tools/include/nolibc/arch-sparc.h index ca420d843e25..a9386a4808a7 100644 --- a/tools/include/nolibc/arch-sparc.h +++ b/tools/include/nolibc/arch-sparc.h @@ -45,7 +45,7 @@ \ __asm__ volatile ( \ _NOLIBC_SYSCALL \ - : "+r"(_arg1) \ + : "=r"(_arg1) \ : "r"(_num) \ : "memory", "cc" \ ); \ diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h index 23a062781468..50d67df78911 100644 --- a/tools/include/uapi/linux/if_xdp.h +++ b/tools/include/uapi/linux/if_xdp.h @@ -149,6 +149,7 @@ struct xsk_tx_metadata { __u16 csum_start; /* Offset from csum_start where checksum should be stored. */ __u16 csum_offset; + __u32 reserved; /* XDP_TXMD_FLAGS_LAUNCH_TIME */ /* Launch time in nanosecond against the PTP HW Clock */ diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 3baa6025ecba..94ee5f52b866 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -12148,13 +12148,14 @@ static const char *arch_specific_lib_paths(void) /* Get full path to program/shared library. */ static int resolve_full_path(const char *file, char *result, size_t result_sz) { - const char *search_paths[3] = {}; + const char *search_paths[4] = {}; int i, perm; if (str_has_sfx(file, ".so") || strstr(file, ".so.")) { search_paths[0] = getenv("LD_LIBRARY_PATH"); search_paths[1] = "/usr/lib64:/usr/lib"; search_paths[2] = arch_specific_lib_paths(); + search_paths[3] = "/lib64:/lib"; perm = R_OK; } else { search_paths[0] = getenv("PATH"); diff --git a/tools/net/sunrpc/xdrgen/templates/C/enum/declaration/enum.j2 b/tools/net/sunrpc/xdrgen/templates/C/enum/declaration/enum.j2 index d1405c7c5354..c7ae506076bb 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/enum/declaration/enum.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/enum/declaration/enum.j2 @@ -1,4 +1,3 @@ {# SPDX-License-Identifier: GPL-2.0 #} - bool xdrgen_decode_{{ name }}(struct xdr_stream *xdr, {{ name }} *ptr); bool xdrgen_encode_{{ name }}(struct xdr_stream *xdr, {{ name }} value); diff --git a/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close.j2 b/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close.j2 index a07586cbee17..446266ad6d17 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close.j2 @@ -1,3 +1,4 @@ {# SPDX-License-Identifier: GPL-2.0 #} }; + typedef enum {{ name }} {{ name }}; diff --git a/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close_be.j2 b/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close_be.j2 index 2c18948bddf7..cfeee2287e68 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close_be.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/enum/definition/close_be.j2 @@ -1,3 +1,4 @@ {# SPDX-License-Identifier: GPL-2.0 #} }; + typedef __be32 {{ name }}; diff --git a/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/string.j2 b/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/string.j2 index cf65b71eaef3..7ddc2bf3edac 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/string.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/string.j2 @@ -2,7 +2,9 @@ {% if annotate %} /* member {{ name }} (variable-length string) */ {% endif %} +{% if maxsize != "0" %} if (value->{{ name }}.len > {{ maxsize }}) return false; +{% endif %} if (xdr_stream_encode_opaque(xdr, value->{{ name }}.data, value->{{ name }}.len) < 0) return false; diff --git a/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/variable_length_opaque.j2 b/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/variable_length_opaque.j2 index 1d477c2d197a..5bf00070ae95 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/variable_length_opaque.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/pointer/encoder/variable_length_opaque.j2 @@ -2,7 +2,9 @@ {% if annotate %} /* member {{ name }} (variable-length opaque) */ {% endif %} +{% if maxsize != "0" %} if (value->{{ name }}.len > {{ maxsize }}) return false; +{% endif %} if (xdr_stream_encode_opaque(xdr, value->{{ name }}.data, value->{{ name }}.len) < 0) return false; diff --git a/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/string.j2 b/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/string.j2 index cf65b71eaef3..7ddc2bf3edac 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/string.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/string.j2 @@ -2,7 +2,9 @@ {% if annotate %} /* member {{ name }} (variable-length string) */ {% endif %} +{% if maxsize != "0" %} if (value->{{ name }}.len > {{ maxsize }}) return false; +{% endif %} if (xdr_stream_encode_opaque(xdr, value->{{ name }}.data, value->{{ name }}.len) < 0) return false; diff --git a/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/variable_length_opaque.j2 b/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/variable_length_opaque.j2 index 1d477c2d197a..5bf00070ae95 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/variable_length_opaque.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/struct/encoder/variable_length_opaque.j2 @@ -2,7 +2,9 @@ {% if annotate %} /* member {{ name }} (variable-length opaque) */ {% endif %} +{% if maxsize != "0" %} if (value->{{ name }}.len > {{ maxsize }}) return false; +{% endif %} if (xdr_stream_encode_opaque(xdr, value->{{ name }}.data, value->{{ name }}.len) < 0) return false; diff --git a/tools/net/sunrpc/xdrgen/templates/C/union/definition/close.j2 b/tools/net/sunrpc/xdrgen/templates/C/union/definition/close.j2 index 01d716d0099e..19ee759d70c6 100644 --- a/tools/net/sunrpc/xdrgen/templates/C/union/definition/close.j2 +++ b/tools/net/sunrpc/xdrgen/templates/C/union/definition/close.j2 @@ -1,8 +1,3 @@ {# SPDX-License-Identifier: GPL-2.0 #} } u; }; -{%- if name in public_apis %} - -bool xdrgen_decode_{{ name }}(struct xdr_stream *xdr, struct {{ name }} *ptr); -bool xdrgen_encode_{{ name }}(struct xdr_stream *xdr, const struct {{ name }} *ptr); -{%- endif -%} diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c index 4cc33452d79b..d8118256243e 100644 --- a/tools/perf/builtin-ftrace.c +++ b/tools/perf/builtin-ftrace.c @@ -71,18 +71,11 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused, static bool check_ftrace_capable(void) { - bool used_root; - - if (perf_cap__capable(CAP_PERFMON, &used_root)) - return true; - - if (!used_root && perf_cap__capable(CAP_SYS_ADMIN, &used_root)) + if (perf_cap__capable(CAP_PERFMON) || + perf_cap__capable(CAP_SYS_ADMIN)) return true; - pr_err("ftrace only works for %s!\n", - used_root ? "root" - : "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" - ); + pr_err("ftrace only works for users with the CAP_PERFMON or CAP_SYS_ADMIN capability!\n"); return false; } diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c index c82b9720c929..87cff5f5a99c 100644 --- a/tools/perf/builtin-record.c +++ b/tools/perf/builtin-record.c @@ -63,6 +63,7 @@ #include #include #include +#include #ifndef HAVE_GETTID #include #endif @@ -652,27 +653,14 @@ static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size) struct record *rec = to; if (record__comp_enabled(rec)) { - struct perf_record_compressed2 *event = map->data; - size_t padding = 0; - u8 pad[8] = {0}; ssize_t compressed = zstd_compress(rec->session, map, map->data, mmap__mmap_len(map), bf, size); if (compressed < 0) return (int)compressed; - bf = event; thread->samples++; - - /* - * The record from `zstd_compress` is not 8 bytes aligned, which would cause asan - * error. We make it aligned here. - */ - event->data_size = compressed - sizeof(struct perf_record_compressed2); - event->header.size = PERF_ALIGN(compressed, sizeof(u64)); - padding = event->header.size - compressed; - return record__write(rec, map, bf, compressed) || - record__write(rec, map, &pad, padding); + return record__write(rec, map, map->data, compressed); } thread->samples++; @@ -1550,18 +1538,36 @@ static void record__adjust_affinity(struct record *rec, struct mmap *map) } } -static size_t process_comp_header(void *record, size_t increment) +/* + * Called once with data_size == 0 to start a record, then once with + * data_size == compressed payload size to finalize and 8-byte-pad it + * (unaligned records trip ASan in the reader). + * Returns the bytes written, or -1 if it won't fit. + */ +static ssize_t process_comp_header(void *record, size_t dst_size, + size_t data_size) { struct perf_record_compressed2 *event = record; size_t size = sizeof(*event); - if (increment) { - event->header.size += increment; - return increment; + if (data_size) { + size_t padding; + + event->data_size = data_size; + event->header.size = PERF_ALIGN(size + data_size, sizeof(u64)); + padding = event->header.size - size - data_size; + if (padding > dst_size) + return -1; + memset(record + size + data_size, 0, padding); + return padding; } + if (size > dst_size) + return -1; + event->header.type = PERF_RECORD_COMPRESSED2; event->header.size = size; + event->data_size = 0; return size; } @@ -1570,7 +1576,12 @@ static ssize_t zstd_compress(struct perf_session *session, struct mmap *map, void *dst, size_t dst_size, void *src, size_t src_size) { ssize_t compressed; - size_t max_record_size = PERF_SAMPLE_MAX_SIZE - sizeof(struct perf_record_compressed2) - 1; + /* + * Reserve space so per-record PERF_ALIGN() padding keeps header.size + * within u16. + */ + size_t max_record_size = PERF_SAMPLE_MAX_SIZE + - sizeof(struct perf_record_compressed2) - sizeof(u64); struct zstd_data *zstd_data = &session->zstd_data; if (map && map->file) @@ -2702,7 +2713,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv) trigger_error(&auxtrace_snapshot_trigger); trigger_error(&switch_output_trigger); err = -1; - goto out_child; + goto out_child_no_flush; } if (auxtrace_record__snapshot_started) { @@ -2849,6 +2860,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv) out_child: record__stop_threads(rec); record__mmap_read_all(rec, true); + goto out_free_threads; +out_child_no_flush: + /* mmap read already failed — retrying would just fail again */ + record__stop_threads(rec); out_free_threads: record__free_thread_data(rec); evlist__finalize_ctlfd(rec->evlist); diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c index 1202ceb425f0..3fc05180b157 100644 --- a/tools/perf/builtin-stat.c +++ b/tools/perf/builtin-stat.c @@ -2637,7 +2637,8 @@ int cmd_stat(int argc, const char **argv) output = fopen(output_name, mode); if (!output) { perror("failed to create output file"); - return -1; + status = -1; + goto out; } if (!stat_config.json_output) { clock_gettime(CLOCK_REALTIME, &tm); @@ -2648,7 +2649,8 @@ int cmd_stat(int argc, const char **argv) output = fdopen(output_fd, mode); if (!output) { perror("Failed opening logfd"); - return -errno; + status = -errno; + goto out; } } @@ -2657,7 +2659,8 @@ int cmd_stat(int argc, const char **argv) parse_options_usage(stat_usage, stat_options, "o", 1); parse_options_usage(NULL, stat_options, "log-fd", 0); parse_options_usage(NULL, stat_options, "interval-clear", 0); - return -1; + status = -1; + goto out; } stat_config.output = output; diff --git a/tools/perf/pmu-events/arch/x86/amdzen5/floating-point.json b/tools/perf/pmu-events/arch/x86/amdzen5/floating-point.json index 9204bfb1d69e..50d38434f8d3 100644 --- a/tools/perf/pmu-events/arch/x86/amdzen5/floating-point.json +++ b/tools/perf/pmu-events/arch/x86/amdzen5/floating-point.json @@ -179,6 +179,30 @@ "BriefDescription": "Retired scalar floating-point blend ops.", "UMask": "0x09" }, + { + "EventName": "fp_ops_retired_by_type.scalar_mov", + "EventCode": "0x0a", + "BriefDescription": "Retired scalar floating-point MOV ops.", + "UMask": "0x0a" + }, + { + "EventName": "fp_ops_retired_by_type.scalar_shuffle", + "EventCode": "0x0a", + "BriefDescription": "Retired scalar floating-point shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", + "UMask": "0x0b" + }, + { + "EventName": "fp_ops_retired_by_type.scalar_bfloat", + "EventCode": "0x0a", + "BriefDescription": "Retired scalar floating-point bfloat ops.", + "UMask": "0x0c" + }, + { + "EventName": "fp_ops_retired_by_type.scalar_logical", + "EventCode": "0x0a", + "BriefDescription": "Retired scalar floating-point logical ops.", + "UMask": "0x0d" + }, { "EventName": "fp_ops_retired_by_type.scalar_other", "EventCode": "0x0a", @@ -245,12 +269,24 @@ "BriefDescription": "Retired vector floating-point blend ops.", "UMask": "0x90" }, + { + "EventName": "fp_ops_retired_by_type.vector_mov", + "EventCode": "0x0a", + "BriefDescription": "Retired vector floating-point MOV ops.", + "UMask": "0xa0" + }, { "EventName": "fp_ops_retired_by_type.vector_shuffle", "EventCode": "0x0a", "BriefDescription": "Retired vector floating-point shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", "UMask": "0xb0" }, + { + "EventName": "fp_ops_retired_by_type.vector_bfloat", + "EventCode": "0x0a", + "BriefDescription": "Retired vector floating-point bfloat ops.", + "UMask": "0xc0" + }, { "EventName": "fp_ops_retired_by_type.vector_logical", "EventCode": "0x0a", @@ -278,7 +314,7 @@ { "EventName": "sse_avx_ops_retired.mmx_add", "EventCode": "0x0b", - "BriefDescription": "Retired MMX integer add.", + "BriefDescription": "Retired MMX integer add ops.", "UMask": "0x01" }, { @@ -299,16 +335,34 @@ "BriefDescription": "Retired MMX integer multiply-accumulate ops.", "UMask": "0x04" }, + { + "EventName": "sse_avx_ops_retired.mmx_aes", + "EventCode": "0x0b", + "BriefDescription": "Retired MMX integer AES ops.", + "UMask": "0x05" + }, + { + "EventName": "sse_avx_ops_retired.mmx_sha", + "EventCode": "0x0b", + "BriefDescription": "Retired MMX integer SHA ops.", + "UMask": "0x06" + }, { "EventName": "sse_avx_ops_retired.mmx_cmp", "EventCode": "0x0b", "BriefDescription": "Retired MMX integer compare ops.", "UMask": "0x07" }, + { + "EventName": "sse_avx_ops_retired.mmx_cvt", + "EventCode": "0x0b", + "BriefDescription": "Retired MMX integer convert or pack ops.", + "UMask": "0x08" + }, { "EventName": "sse_avx_ops_retired.mmx_shift", "EventCode": "0x0b", - "BriefDescription": "Retired MMX integer shift ops.", + "BriefDescription": "Retired MMX integer shift or rotate ops.", "UMask": "0x09" }, { @@ -323,10 +377,17 @@ "BriefDescription": "Retired MMX integer shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", "UMask": "0x0b" }, + { + "EventName": "sse_avx_ops_retired.mmx_vnni", + "EventCode": "0x0b", + "BriefDescription": "Retired MMX integer VNNI ops.", + "UMask": "0x0c" + }, { "EventName": "sse_avx_ops_retired.mmx_pack", "EventCode": "0x0b", - "BriefDescription": "Retired MMX integer pack ops.", + "BriefDescription": "This event is deprecated. Refer to new event sse_avx_ops_retired.mmx_vnni", + "Deprecated": "1", "UMask": "0x0c" }, { @@ -389,16 +450,23 @@ "BriefDescription": "Retired SSE and AVX integer compare ops.", "UMask": "0x70" }, + { + "EventName": "sse_avx_ops_retired.sse_avx_cvt", + "EventCode": "0x0b", + "BriefDescription": "Retired SSE and AVX integer convert or pack ops.", + "UMask": "0x80" + }, { "EventName": "sse_avx_ops_retired.sse_avx_clm", "EventCode": "0x0b", - "BriefDescription": "Retired SSE and AVX integer CLM ops.", + "BriefDescription": "This event is deprecated. Refer to new event sse_avx_ops_retired.sse_avx_cvt", + "Deprecated": "1", "UMask": "0x80" }, { "EventName": "sse_avx_ops_retired.sse_avx_shift", "EventCode": "0x0b", - "BriefDescription": "Retired SSE and AVX integer shift ops.", + "BriefDescription": "Retired SSE and AVX integer shift or rotate ops.", "UMask": "0x90" }, { @@ -413,10 +481,17 @@ "BriefDescription": "Retired SSE and AVX integer shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", "UMask": "0xb0" }, + { + "EventName": "sse_avx_ops_retired.sse_avx_vnni", + "EventCode": "0x0b", + "BriefDescription": "Retired SSE and AVX integer VNNI ops.", + "UMask": "0xc0" + }, { "EventName": "sse_avx_ops_retired.sse_avx_pack", "EventCode": "0x0b", - "BriefDescription": "Retired SSE and AVX integer pack ops.", + "BriefDescription": "This event is deprecated. Refer to new event sse_avx_ops_retired.sse_avx_vnni", + "Deprecated": "1", "UMask": "0xc0" }, { @@ -497,12 +572,24 @@ "BriefDescription": "Retired 128-bit packed floating-point blend ops.", "UMask": "0x09" }, + { + "EventName": "fp_pack_ops_retired.fp128_mov", + "EventCode": "0x0c", + "BriefDescription": "Retired 128-bit packed floating-point MOV ops.", + "UMask": "0x0a" + }, { "EventName": "fp_pack_ops_retired.fp128_shuffle", "EventCode": "0x0c", "BriefDescription": "Retired 128-bit packed floating-point shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", "UMask": "0x0b" }, + { + "EventName": "fp_pack_ops_retired.fp128_bfloat", + "EventCode": "0x0c", + "BriefDescription": "Retired 128-bit packed floating-point bfloat ops.", + "UMask": "0x0c" + }, { "EventName": "fp_pack_ops_retired.fp128_logical", "EventCode": "0x0c", @@ -575,12 +662,24 @@ "BriefDescription": "Retired 256-bit packed floating-point blend ops.", "UMask": "0x90" }, + { + "EventName": "fp_pack_ops_retired.fp256_mov", + "EventCode": "0x0c", + "BriefDescription": "Retired 256-bit packed floating-point MOV ops.", + "UMask": "0xa0" + }, { "EventName": "fp_pack_ops_retired.fp256_shuffle", "EventCode": "0x0c", "BriefDescription": "Retired 256-bit packed floating-point shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", "UMask": "0xb0" }, + { + "EventName": "fp_pack_ops_retired.fp256_bfloat", + "EventCode": "0x0c", + "BriefDescription": "Retired 256-bit packed floating-point bfloat ops.", + "UMask": "0xc0" + }, { "EventName": "fp_pack_ops_retired.fp256_logical", "EventCode": "0x0c", @@ -647,16 +746,23 @@ "BriefDescription": "Retired 128-bit packed integer compare ops.", "UMask": "0x07" }, + { + "EventName": "packed_int_op_type.int128_cvt", + "EventCode": "0x0d", + "BriefDescription": "Retired 128-bit packed integer convert or pack ops.", + "UMask": "0x08" + }, { "EventName": "packed_int_op_type.int128_clm", "EventCode": "0x0d", - "BriefDescription": "Retired 128-bit packed integer CLM ops.", + "BriefDescription": "This event is deprecated. Refer to new event packed_int_op_type.int128_cvt", + "Deprecated": "1", "UMask": "0x08" }, { "EventName": "packed_int_op_type.int128_shift", "EventCode": "0x0d", - "BriefDescription": "Retired 128-bit packed integer shift ops.", + "BriefDescription": "Retired 128-bit packed integer shift or rotate ops.", "UMask": "0x09" }, { @@ -671,10 +777,17 @@ "BriefDescription": "Retired 128-bit packed integer shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", "UMask": "0x0b" }, + { + "EventName": "packed_int_op_type.int128_vnni", + "EventCode": "0x0d", + "BriefDescription": "Retired 128-bit packed integer VNNI ops.", + "UMask": "0x0c" + }, { "EventName": "packed_int_op_type.int128_pack", "EventCode": "0x0d", - "BriefDescription": "Retired 128-bit packed integer pack ops.", + "BriefDescription": "This event is deprecated. Refer to new event packed_int_op_type.int128_vnni", + "Deprecated": "1", "UMask": "0x0c" }, { @@ -719,16 +832,34 @@ "BriefDescription": "Retired 256-bit packed integer multiply-accumulate ops.", "UMask": "0x40" }, + { + "EventName": "packed_int_op_type.int256_aes", + "EventCode": "0x0d", + "BriefDescription": "Retired 256-bit packed integer AES ops.", + "UMask": "0x50" + }, + { + "EventName": "packed_int_op_type.int256_sha", + "EventCode": "0x0d", + "BriefDescription": "Retired 256-bit packed integer SHA ops.", + "UMask": "0x60" + }, { "EventName": "packed_int_op_type.int256_cmp", "EventCode": "0x0d", "BriefDescription": "Retired 256-bit packed integer compare ops.", "UMask": "0x70" }, + { + "EventName": "packed_int_op_type.int256_cvt", + "EventCode": "0x0d", + "BriefDescription": "Retired 256-bit packed integer convert or pack ops.", + "UMask": "0x80" + }, { "EventName": "packed_int_op_type.int256_shift", "EventCode": "0x0d", - "BriefDescription": "Retired 256-bit packed integer shift ops.", + "BriefDescription": "Retired 256-bit packed integer shift or rotate ops.", "UMask": "0x90" }, { @@ -743,10 +874,17 @@ "BriefDescription": "Retired 256-bit packed integer shuffle ops (may include instructions not necessarily thought of as including shuffles e.g. horizontal add, dot product, and certain MOV instructions).", "UMask": "0xb0" }, + { + "EventName": "packed_int_op_type.int256_vnni", + "EventCode": "0x0d", + "BriefDescription": "Retired 256-bit packed integer VNNI ops.", + "UMask": "0xc0" + }, { "EventName": "packed_int_op_type.int256_pack", "EventCode": "0x0d", - "BriefDescription": "Retired 256-bit packed integer pack ops.", + "BriefDescription": "This event is deprecated. Refer to new event packed_int_op_type.int256_vnni", + "Deprecated": "1", "UMask": "0xc0" }, { diff --git a/tools/perf/pmu-events/arch/x86/amdzen5/load-store.json b/tools/perf/pmu-events/arch/x86/amdzen5/load-store.json index 06bbaea15925..b1994539ece8 100644 --- a/tools/perf/pmu-events/arch/x86/amdzen5/load-store.json +++ b/tools/perf/pmu-events/arch/x86/amdzen5/load-store.json @@ -8,9 +8,15 @@ { "EventName": "ls_locks.bus_lock", "EventCode": "0x25", - "BriefDescription": "Retired Lock instructions which caused a bus lock.", + "BriefDescription": "Retired lock instructions which caused a bus lock.", "UMask": "0x01" }, + { + "EventName": "ls_locks.all", + "EventCode": "0x25", + "BriefDescription": "Retired lock instructions of all types.", + "UMask": "0x1f" + }, { "EventName": "ls_ret_cl_flush", "EventCode": "0x26", diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py index 168c044dd7cc..59426ba68de5 100755 --- a/tools/perf/pmu-events/jevents.py +++ b/tools/perf/pmu-events/jevents.py @@ -559,13 +559,14 @@ const struct pmu_table_entry {_pending_events_tblname}[] = {{ def print_pending_metrics() -> None: """Optionally close metrics table.""" - def metric_cmp_key(j: JsonEvent) -> Tuple[bool, str, str]: + def metric_cmp_key(j: JsonEvent) -> Tuple[str, str, str, str]: def fix_none(s: Optional[str]) -> str: if s is None: return '' return s - return (j.desc is not None, fix_none(j.pmu), fix_none(j.metric_name)) + return (fix_none(j.pmu), fix_none(j.metric_name), j.metric_expr.ToPerfJson(), + fix_none(j.desc)) global _pending_metrics if not _pending_metrics: diff --git a/tools/perf/pmu-events/metric.py b/tools/perf/pmu-events/metric.py index 92acd89ed97a..ac322891c4f1 100644 --- a/tools/perf/pmu-events/metric.py +++ b/tools/perf/pmu-events/metric.py @@ -445,7 +445,11 @@ class Metric: def __lt__(self, other): """Sort order.""" - return self.name < other.name + if self.name != other.name: + return self.name < other.name + if not self.expr.Equals(other.expr): + return self.expr.ToPerfJson() < other.expr.ToPerfJson() + return self.description < other.description def AddToMetricGroup(self, group): """Callback used when being added to a MetricGroup.""" diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c index 0d2fb7a4ae5b..a4ee849ad864 100644 --- a/tools/perf/tests/builtin-test.c +++ b/tools/perf/tests/builtin-test.c @@ -10,34 +10,43 @@ #ifdef HAVE_BACKTRACE_SUPPORT #include #endif -#include -#include #include -#include #include -#include +#include + #include -#include +#include "util/term.h" +#include +#include +#include +#include +#include #include +#include +#include +#include +#include + +#include +#include +#include + #include "builtin.h" +#include "color.h" #include "config.h" +#include "debug.h" #include "hist.h" #include "intlist.h" -#include "tests.h" -#include "debug.h" -#include "color.h" -#include -#include #include "string2.h" #include "symbol.h" +#include "tests-scripts.h" +#include "tests.h" #include "util/rlimit.h" #include "util/strbuf.h" -#include -#include -#include -#include +#include "util/term.h" -#include "tests-scripts.h" +static const char *junit_filename; +static struct strbuf junit_xml_buf = STRBUF_INIT; /* * Command line option to not fork the test running in the same process and @@ -48,6 +57,8 @@ static bool dont_fork; static bool sequential; /* Number of times each test is run. */ static unsigned int runs_per_test = 1; +/* Number of lines to include in failure snippet. */ +static unsigned int failure_snippet_lines = 10; const char *dso_to_test; const char *test_objdump_path = "objdump"; @@ -295,6 +306,11 @@ struct child_test { struct test_suite *test; int suite_num; int test_case_num; + struct strbuf err_output; + int result; + bool done; + struct timespec start_time; + struct timespec end_time; }; static jmp_buf run_test_jmp_buf; @@ -334,7 +350,7 @@ static int run_test_child(struct child_process *process) for (size_t i = 0; i < ARRAY_SIZE(signals); i++) signal(signals[i], child_test_sig_handler); - pr_debug("--- start ---\n"); + pr_debug("---- start ----\n"); pr_debug("test child forked, pid %d\n", getpid()); err = test_function(child->test, child->test_case_num)(child->test, child->test_case_num); pr_debug("---- end(%d) ----\n", err); @@ -349,42 +365,362 @@ static int run_test_child(struct child_process *process) #define TEST_RUNNING -3 +static struct pollfd *global_pfds; +static size_t *global_pfd_indices; +static unsigned int summary_tests_passed; +static unsigned int summary_subtests_passed; +static unsigned int summary_tests_skipped; +static unsigned int summary_tests_failed; +static struct strbuf summary_failed_tests_buf = STRBUF_INIT; + +static int strbuf_addstr_safe(struct strbuf *sb, const char *s); +static int __printf(2, 3) strbuf_addf_safe(struct strbuf *sb, const char *fmt, ...); + +static char *xml_escape(const char *str) +{ + struct strbuf buf = STRBUF_INIT; + const char *p; + char *res; + + if (!str) + return strdup(""); + + for (p = str; *p; p++) { + if (*p == '&') + strbuf_addstr(&buf, "&"); + else if (*p == '<') + strbuf_addstr(&buf, "<"); + else if (*p == '>') + strbuf_addstr(&buf, ">"); + else if (*p == '"') + strbuf_addstr(&buf, """); + else if ((unsigned char)*p >= 32 || *p == '\n' || *p == '\t') + strbuf_addch(&buf, *p); + } + res = strbuf_detach(&buf, NULL); + return res ? res : strdup(""); +} + +static int get_term_width(void) +{ + struct winsize ws; + int cols = 80; + int term_width; + + /* + * If output is redirected to a file or piped, we don't need to wrap + * or truncate at all. Use a massive virtually infinite terminal width + * so descriptions are printed in full. + */ + if (!isatty(fileno(debug_file()))) + return 10000; + + get_term_dimensions(&ws); + if (ws.ws_col > 0) + cols = ws.ws_col; + + /* + * Limit description width to fit on a single line. We subtract 35 + * columns of headroom to allocate space for: + * - The suite index prefix: e.g. " 10.100:" (8 characters) plus 1 space separator. + * - The trailing colon (1 character) and space before status (1 character). + * - The longest status results: e.g. "Skip (some metrics failed)" (26 characters) + * or "Running (XX active)" (20 characters). + * + * A minimum description width of 10 is enforced to ensure names are + * legible even on very narrow consoles. + */ + term_width = cols - 35; + if (term_width < 10) + term_width = 10; + + return term_width; +} + +static int get_max_desc_width(int width) +{ + int term_width = get_term_width(); + + return width > term_width ? term_width : width; +} + static int print_test_result(struct test_suite *t, int curr_suite, int curr_test_case, - int result, int width, int running) + int result, int width, int running, + const char *err_output, double elapsed) { + int pad_width = get_max_desc_width(width); + int term_width = get_term_width(); + if (test_suite__num_test_cases(t) > 1) { - int subw = width > 2 ? width - 2 : width; + char prefix[32]; + int len = snprintf(prefix, sizeof(prefix), "%3d.%1d:", + curr_suite + 1, curr_test_case + 1); + int pad = len >= 4 ? pad_width + 4 - len : pad_width; + int trunc = len >= 4 ? term_width + 4 - len : term_width; - pr_info("%3d.%1d: %-*s:", curr_suite + 1, curr_test_case + 1, subw, + pr_info("%s %-*.*s:", prefix, pad, trunc, + test_description(t, curr_test_case)); + } else { + pr_info("%3d: %-*.*s:", curr_suite + 1, pad_width, term_width, test_description(t, curr_test_case)); - } else - pr_info("%3d: %-*s:", curr_suite + 1, width, test_description(t, curr_test_case)); + } switch (result) { case TEST_RUNNING: - color_fprintf(stderr, PERF_COLOR_YELLOW, " Running (%d active)\n", running); + color_fprintf(debug_file(), PERF_COLOR_YELLOW, " Running (%d active)\n", running); break; case TEST_OK: + if (test_suite__num_test_cases(t) > 1) + summary_subtests_passed++; + else + summary_tests_passed++; pr_info(" Ok\n"); break; case TEST_SKIP: { const char *reason = skip_reason(t, curr_test_case); + summary_tests_skipped++; if (reason) - color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (%s)\n", reason); + color_fprintf(debug_file(), PERF_COLOR_YELLOW, " Skip (%s)\n", reason); else - color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip\n"); + color_fprintf(debug_file(), PERF_COLOR_YELLOW, " Skip\n"); } break; case TEST_FAIL: default: - color_fprintf(stderr, PERF_COLOR_RED, " FAILED!\n"); + summary_tests_failed++; + if (test_suite__num_test_cases(t) > 1) + strbuf_addf_safe(&summary_failed_tests_buf, " %3d.%1d: %s\n", + curr_suite + 1, curr_test_case + 1, + test_description(t, curr_test_case)); + else + strbuf_addf_safe(&summary_failed_tests_buf, " %3d: %s\n", + curr_suite + 1, + test_description(t, curr_test_case)); + color_fprintf(debug_file(), PERF_COLOR_RED, " FAILED!\n"); break; } + if (junit_filename && result != TEST_RUNNING) { + const char *classname = t->desc; + const char *testname = test_description(t, curr_test_case); + char *escaped_err = xml_escape(err_output); + char *escaped_class = xml_escape(classname); + char *escaped_test = xml_escape(testname); + + strbuf_addf(&junit_xml_buf, + " \n", + escaped_class, escaped_test, elapsed); + if (result != TEST_OK && result != TEST_SKIP) { + strbuf_addf(&junit_xml_buf, + " \n%s\n \n", + escaped_err); + } else if (result == TEST_SKIP) { + const char *reason = skip_reason(t, curr_test_case); + char *escaped_reason = xml_escape(reason ? reason : "Skip"); + + strbuf_addf(&junit_xml_buf, " \n", + escaped_reason); + free(escaped_reason); + } + strbuf_addstr(&junit_xml_buf, " \n"); + free(escaped_err); + free(escaped_class); + free(escaped_test); + } + return 0; } +static const char * const fail_keywords[] = { + "error", "fail", "segv", "abort", + "signal", "fatal", "panic", "corrupt", NULL +}; + +static const char *find_next_keyword(const char *str, size_t max_len, size_t *kw_len) +{ + const char *best = NULL; + size_t best_len = 0; + int k; + + for (k = 0; fail_keywords[k]; k++) { + const char *s = str; + size_t len = strlen(fail_keywords[k]); + + while ((size_t)(s - str) + len <= max_len) { + size_t i; + + if (best && s >= best) + break; + + for (i = 0; i < len; i++) { + if (tolower(s[i]) != fail_keywords[k][i]) + break; + } + if (i == len) { + if (!best || s < best) { + best = s; + best_len = len; + } + break; + } + s++; + } + } + if (best) { + *kw_len = best_len; + return best; + } + return NULL; +} + +static void print_line_highlighted(FILE *fp, const char *line, size_t len) +{ + const char *s = line; + + while (len > 0) { + size_t kw_len = 0; + const char *match = find_next_keyword(s, len, &kw_len); + + if (!match) { + fwrite(s, 1, len, fp); + break; + } + if (match > s) + fwrite(s, 1, match - s, fp); + if (perf_use_color_default) + fprintf(fp, "%s", PERF_COLOR_RED); + fwrite(match, 1, kw_len, fp); + if (perf_use_color_default) + fprintf(fp, "%s", PERF_COLOR_RESET); + + len -= (match + kw_len) - s; + s = match + kw_len; + } +} + + +static void print_test_failure_snippet(FILE *fp, const char *buf) +{ + size_t num_lines = 0; + size_t max_lines = 128; + const char **lines = calloc(max_lines, sizeof(const char *)); + size_t *line_lens = calloc(max_lines, sizeof(size_t)); + const char *s = buf; + size_t i; + unsigned int picked_count = 0; + bool *pick; + int last_printed = -1; + + if (!lines || !line_lens) { + free(lines); free(line_lens); + fprintf(fp, "%s", buf); + return; + } + + while (*s) { + const char *eol = strchr(s, '\n'); + size_t len; + + if (eol) + len = eol - s + 1; + else + len = strlen(s); + + if (num_lines == max_lines) { + const char **new_lines; + size_t *new_lens; + + max_lines *= 2; + new_lines = realloc(lines, max_lines * sizeof(const char *)); + if (!new_lines) { + free(lines); free(line_lens); + fprintf(fp, "%s", buf); + return; + } + lines = new_lines; + + new_lens = realloc(line_lens, max_lines * sizeof(size_t)); + if (!new_lens) { + free(lines); free(line_lens); + fprintf(fp, "%s", buf); + return; + } + line_lens = new_lens; + } + lines[num_lines] = s; + line_lens[num_lines] = len; + num_lines++; + s += len; + } + + if (num_lines <= failure_snippet_lines) { + for (i = 0; i < num_lines; i++) + print_line_highlighted(fp, lines[i], line_lens[i]); + free(lines); free(line_lens); + return; + } + + pick = calloc(num_lines, sizeof(bool)); + if (!pick) { + for (i = 0; i < num_lines; i++) + print_line_highlighted(fp, lines[i], line_lens[i]); + free(lines); free(line_lens); + return; + } + + /* Pass 0: Always pick the very first line */ + if (num_lines > 0 && picked_count < failure_snippet_lines) { + pick[0] = true; + picked_count++; + } + + /* Pass 1: Pick lines with failure keywords from start (Highest Priority) */ + for (i = 0; i < num_lines && picked_count < failure_snippet_lines; i++) { + size_t dummy; + + if (find_next_keyword(lines[i], line_lens[i], &dummy)) { + if (!pick[i]) { + pick[i] = true; + picked_count++; + } + /* Prioritize getting the immediate next line for context */ + if (i + 1 < num_lines && !pick[i + 1] && + picked_count < failure_snippet_lines) { + pick[i + 1] = true; + picked_count++; + } + } + } + + /* Pass 2: Fill remaining quota from the end backwards */ + i = num_lines; + while (i > 0 && picked_count < failure_snippet_lines) { + i--; + if (!pick[i]) { + pick[i] = true; + picked_count++; + } + } + + for (i = 0; i < num_lines; i++) { + if (!pick[i]) + continue; + if (last_printed != -1 && (int)i > last_printed + 1) { + if (perf_use_color_default) + fprintf(fp, "%s...%s\n", PERF_COLOR_BLUE, PERF_COLOR_RESET); + else + fprintf(fp, "...\n"); + } + print_line_highlighted(fp, lines[i], line_lens[i]); + last_printed = i; + } + + free(pick); + free(lines); + free(line_lens); +} + static void finish_test(struct child_test **child_tests, int running_test, int child_test_num, int width) { @@ -395,6 +731,9 @@ static void finish_test(struct child_test **child_tests, int running_test, int c struct strbuf err_output = STRBUF_INIT; int last_running = -1; int ret; + struct timespec end_time; + double elapsed; + width = get_max_desc_width(width); if (child_test == NULL) { /* Test wasn't started. */ @@ -409,13 +748,14 @@ static void finish_test(struct child_test **child_tests, int running_test, int c * sub test names. */ if (test_suite__num_test_cases(t) > 1 && curr_test_case == 0) - pr_info("%3d: %-*s:\n", curr_suite + 1, width, test_description(t, -1)); + pr_info("%3d: %-*.*s:\n", curr_suite + 1, width, width, + test_description(t, -1)); /* * Busy loop reading from the child's stdout/stderr that are set to be * non-blocking until EOF. */ - if (err > 0) + if (err >= 0) fcntl(err, F_SETFL, O_NONBLOCK); if (verbose > 1) { if (test_suite__num_test_cases(t) > 1) @@ -448,7 +788,7 @@ static void finish_test(struct child_test **child_tests, int running_test, int c fprintf(debug_file(), PERF_COLOR_DELETE_LINE); } print_test_result(t, curr_suite, curr_test_case, TEST_RUNNING, - width, running); + width, running, NULL, 0.0); last_running = running; } } @@ -469,29 +809,338 @@ static void finish_test(struct child_test **child_tests, int running_test, int c if (len > 0) { err_done = false; buf[len] = '\0'; - strbuf_addstr(&err_output, buf); + strbuf_addstr_safe(&err_output, buf); } } } if (err_done) err_done = check_if_command_finished(&child_test->process); } + /* Drain any remaining data from the pipe. */ + if (err >= 0) { + char buf[512]; + ssize_t len; + + while ((len = read(err, buf, sizeof(buf) - 1)) > 0) { + buf[len] = '\0'; + strbuf_addstr_safe(&err_output, buf); + } + } if (perf_use_color_default && last_running != -1) { /* Erase "Running (.. active)" line printed before poll/sleep. */ fprintf(debug_file(), PERF_COLOR_DELETE_LINE); } /* Clean up child process. */ ret = finish_command(&child_test->process); - if (verbose > 1 || (verbose == 1 && ret == TEST_FAIL)) + child_test->process.pid = 0; + if (child_test->err_output.len > 0) { + struct strbuf merged = STRBUF_INIT; + + if (child_test->err_output.buf) + strbuf_addstr_safe(&merged, child_test->err_output.buf); + if (err_output.buf) + strbuf_addstr_safe(&merged, err_output.buf); + strbuf_release(&err_output); + err_output = merged; + } + if (verbose > 1) fprintf(stderr, "%s", err_output.buf); + else if (verbose == 1 && ret == TEST_FAIL) + print_test_failure_snippet(stderr, err_output.buf); + clock_gettime(CLOCK_MONOTONIC, &end_time); + elapsed = (end_time.tv_sec - child_test->start_time.tv_sec) + + (end_time.tv_nsec - child_test->start_time.tv_nsec) / 1000000000.0; + + print_test_result(t, curr_suite, curr_test_case, ret, width, /*running=*/0, + err_output.buf, elapsed); strbuf_release(&err_output); - print_test_result(t, curr_suite, curr_test_case, ret, width, /*running=*/0); + strbuf_release(&child_test->err_output); if (err > 0) close(err); zfree(&child_tests[running_test]); } +static int strbuf_addstr_safe(struct strbuf *sb, const char *s) +{ + sigset_t set, oldset; + int ret; + + sigemptyset(&set); + sigaddset(&set, SIGINT); + sigaddset(&set, SIGTERM); + pthread_sigmask(SIG_BLOCK, &set, &oldset); + ret = strbuf_addstr(sb, s); + pthread_sigmask(SIG_SETMASK, &oldset, NULL); + return ret; +} + +static int __printf(2, 3) strbuf_addf_safe(struct strbuf *sb, const char *fmt, ...) +{ + char buf[1024]; + va_list ap; + int len; + sigset_t set, oldset; + int ret; + + sigemptyset(&set); + sigaddset(&set, SIGINT); + sigaddset(&set, SIGTERM); + sigprocmask(SIG_BLOCK, &set, &oldset); + + va_start(ap, fmt); + len = vsnprintf(buf, sizeof(buf), fmt, ap); + va_end(ap); + + if (len < 0) { + sigprocmask(SIG_SETMASK, &oldset, NULL); + return len; + } + if ((size_t)len >= sizeof(buf)) { + char *dynamic_buf = malloc(len + 1); + + if (!dynamic_buf) { + sigprocmask(SIG_SETMASK, &oldset, NULL); + return -ENOMEM; + } + va_start(ap, fmt); + vsnprintf(dynamic_buf, len + 1, fmt, ap); + va_end(ap); + ret = strbuf_addstr(sb, dynamic_buf); + free(dynamic_buf); + } else { + ret = strbuf_addstr(sb, buf); + } + + sigprocmask(SIG_SETMASK, &oldset, NULL); + return ret; +} + +static void drain_child_process_err(struct child_test *child) +{ + char buf[512]; + ssize_t len; + + while ((len = read(child->process.err, buf, sizeof(buf) - 1)) > 0) { + buf[len] = '\0'; + strbuf_addstr_safe(&child->err_output, buf); + } +} + +static void handle_child_pipe_activity(struct child_test *child, short revents) +{ + if (!revents) + return; + + drain_child_process_err(child); + /* + * If the child closed its end of the pipe (EOF) or encountered + * an error, close the file descriptor immediately and set it + * to -1. This removes it from the pfds array for subsequent + * iterations, preventing a tight CPU busy-loop while waiting + * for the process itself to exit. + */ + if (revents & (POLLHUP | POLLERR | POLLNVAL)) { + close(child->process.err); + child->process.err = -1; + } +} + +static int finish_tests_parallel(struct child_test **child_tests, size_t num_tests, int width) +{ + size_t next_to_print = 0; + struct pollfd *pfds; + size_t *pfd_indices; + size_t num_pfds = 0; + int last_running = -1; + size_t i; + int last_suite_printed = -1; + sigset_t set, oldset; + + width = get_max_desc_width(width); + + sigemptyset(&set); + sigaddset(&set, SIGINT); + sigaddset(&set, SIGTERM); + + pthread_sigmask(SIG_BLOCK, &set, &oldset); + global_pfds = calloc(num_tests, sizeof(*pfds)); + global_pfd_indices = calloc(num_tests, sizeof(*pfd_indices)); + pfds = global_pfds; + pfd_indices = global_pfd_indices; + if (!pfds || !pfd_indices) { + free(pfds); + free(pfd_indices); + global_pfds = NULL; + global_pfd_indices = NULL; + pthread_sigmask(SIG_SETMASK, &oldset, NULL); + return -ENOMEM; + } + pthread_sigmask(SIG_SETMASK, &oldset, NULL); + + for (i = 0; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child) + continue; + strbuf_init(&child->err_output, 0); + if (child->process.err >= 0) + fcntl(child->process.err, F_SETFL, O_NONBLOCK); + } + + while (next_to_print < num_tests) { + size_t running_count = 0; + size_t p; + + while (next_to_print < num_tests && + (!child_tests[next_to_print] || child_tests[next_to_print]->done)) + next_to_print++; + + if (next_to_print >= num_tests) + break; + + num_pfds = 0; + + for (i = next_to_print; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child || child->done) + continue; + + if (!check_if_command_finished(&child->process)) + running_count++; + + if (child->process.err >= 0) { + pfds[num_pfds].fd = child->process.err; + pfds[num_pfds].events = POLLIN | POLLERR | POLLHUP | POLLNVAL; + pfd_indices[num_pfds] = i; + num_pfds++; + } + } + + if (perf_use_color_default && running_count != (size_t)last_running) { + struct child_test *next_child = child_tests[next_to_print]; + + if (last_running != -1) + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + + if (next_child) { + if (test_suite__num_test_cases(next_child->test) > 1 && + last_suite_printed != next_child->suite_num) { + pr_info("%3d: %-*.*s:\n", + next_child->suite_num + 1, + width, width, + test_description( + next_child->test, -1)); + last_suite_printed = next_child->suite_num; + } + print_test_result(next_child->test, next_child->suite_num, + next_child->test_case_num, TEST_RUNNING, width, + running_count, NULL, 0.0); + } + last_running = running_count; + } + + if (num_pfds == 0) { + if (running_count > 0) + usleep(10 * 1000); + } else { + int pret = poll(pfds, num_pfds, 100); + + if (pret > 0) { + for (p = 0; p < num_pfds; p++) { + size_t idx = pfd_indices[p]; + + handle_child_pipe_activity(child_tests[idx], + pfds[p].revents); + } + } + } + + for (i = next_to_print; i < num_tests; i++) { + struct child_test *child = child_tests[i]; + + if (!child || child->done) + continue; + + if (check_if_command_finished(&child->process)) { + if (child->process.err >= 0) { + drain_child_process_err(child); + close(child->process.err); + child->process.err = -1; + } + child->result = finish_command(&child->process); + child->process.pid = 0; + clock_gettime(CLOCK_MONOTONIC, &child->end_time); + child->done = true; + } + } + + while (next_to_print < num_tests) { + struct child_test *child = child_tests[next_to_print]; + double elapsed; + + if (!child) { + next_to_print++; + continue; + } + if (!child->done) + break; + + if (perf_use_color_default && last_running != -1) { + fprintf(debug_file(), PERF_COLOR_DELETE_LINE); + last_running = -1; + } + + if (test_suite__num_test_cases(child->test) > 1 && + last_suite_printed != child->suite_num) { + pr_info("%3d: %-*.*s:\n", child->suite_num + 1, + width, width, + test_description(child->test, -1)); + last_suite_printed = child->suite_num; + } + + if (verbose > 1) { + if (test_suite__num_test_cases(child->test) > 1) { + pr_info("%3d.%1d: %s:\n", child->suite_num + 1, + child->test_case_num + 1, + test_description(child->test, + child->test_case_num)); + } else { + pr_info("%3d: %s:\n", child->suite_num + 1, + test_description(child->test, -1)); + } + } + + if (verbose > 1) + fprintf(stderr, "%s", child->err_output.buf); + else if (verbose == 1 && child->result == TEST_FAIL) + print_test_failure_snippet(stderr, child->err_output.buf); + + elapsed = (child->end_time.tv_sec - child->start_time.tv_sec) + + (child->end_time.tv_nsec - + child->start_time.tv_nsec) / 1000000000.0; + + print_test_result(child->test, child->suite_num, child->test_case_num, + child->result, width, 0, child->err_output.buf, elapsed); + pthread_sigmask(SIG_BLOCK, &set, &oldset); + strbuf_release(&child->err_output); + child_tests[next_to_print] = NULL; + zfree(&child); + pthread_sigmask(SIG_SETMASK, &oldset, NULL); + next_to_print++; + } + } + + pthread_sigmask(SIG_BLOCK, &set, &oldset); + free(global_pfds); + free(global_pfd_indices); + global_pfds = NULL; + global_pfd_indices = NULL; + pthread_sigmask(SIG_SETMASK, &oldset, NULL); + return 0; +} + static int start_test(struct test_suite *test, int curr_suite, int curr_test_case, struct child_test **child, int width, int pass) { @@ -500,11 +1149,18 @@ static int start_test(struct test_suite *test, int curr_suite, int curr_test_cas *child = NULL; if (dont_fork) { if (pass == 1) { + struct timespec start_time, end_time; + double elapsed; + + clock_gettime(CLOCK_MONOTONIC, &start_time); pr_debug("--- start ---\n"); err = test_function(test, curr_test_case)(test, curr_test_case); pr_debug("---- end ----\n"); + clock_gettime(CLOCK_MONOTONIC, &end_time); + elapsed = (end_time.tv_sec - start_time.tv_sec) + + (end_time.tv_nsec - start_time.tv_nsec) / 1000000000.0; print_test_result(test, curr_suite, curr_test_case, err, width, - /*running=*/0); + /*running=*/0, NULL, elapsed); } return 0; } @@ -525,13 +1181,14 @@ static int start_test(struct test_suite *test, int curr_suite, int curr_test_cas (*child)->test_case_num = curr_test_case; (*child)->process.pid = -1; (*child)->process.no_stdin = 1; + (*child)->process.in = -1; + (*child)->process.out = -1; + (*child)->process.err = -1; if (verbose <= 0) { (*child)->process.no_stdout = 1; (*child)->process.no_stderr = 1; } else { (*child)->process.stdout_to_stderr = 1; - (*child)->process.out = -1; - (*child)->process.err = -1; } (*child)->process.no_exec_cmd = run_test_child; if (sequential || pass == 2) { @@ -555,6 +1212,58 @@ static void cmd_test_sig_handler(int sig) siglongjmp(cmd_test_jmp_buf, sig); } +static void print_tests_summary(void) +{ + pr_info("\n=== Test Summary ===\n"); + pr_info("Passed main tests : %u\n", summary_tests_passed); + pr_info("Passed subtests : %u\n", summary_subtests_passed); + pr_info("Skipped tests : %u\n", summary_tests_skipped); + if (summary_tests_failed > 0) { + color_fprintf(debug_file(), PERF_COLOR_RED, "Failed tests : %u\n", + summary_tests_failed); + pr_info("List of failed tests:\n"); + pr_info("%s", summary_failed_tests_buf.buf); + } else { + color_fprintf(debug_file(), PERF_COLOR_GREEN, "Failed tests : 0\n"); + } + + if (junit_filename) { + int fd; + FILE *fp; + + fd = open(junit_filename, O_CREAT | O_TRUNC | O_WRONLY | O_NOFOLLOW, 0644); + if (fd >= 0) { + fp = fdopen(fd, "w"); + if (fp) { + unsigned int total = summary_tests_passed + + summary_subtests_passed + + summary_tests_skipped + + summary_tests_failed; + fprintf(fp, "\n"); + fprintf(fp, "\n"); + fprintf(fp, + " \n", + total, summary_tests_failed, + summary_tests_skipped); + fprintf(fp, "%s", junit_xml_buf.buf); + fprintf(fp, " \n"); + fprintf(fp, "\n"); + fclose(fp); + pr_info("Wrote junit XML output to %s\n", junit_filename); + } else { + close(fd); + pr_err("Failed to associate stream with fd for %s: %s\n", + junit_filename, strerror(errno)); + } + } else { + pr_err("Failed to open %s for writing junit XML output: %s\n", + junit_filename, strerror(errno)); + } + } + strbuf_release(&junit_xml_buf); + strbuf_release(&summary_failed_tests_buf); +} + static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], struct intlist *skiplist) { @@ -632,9 +1341,36 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], } if (intlist__find(skiplist, curr_suite + 1)) { - pr_info("%3d: %-*s:", curr_suite + 1, width, - test_description(*t, -1)); - color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (user override)\n"); + if (pass == 1) { + int pad_width = get_max_desc_width(width); + int term_width = get_term_width(); + + pr_info("%3d: %-*.*s:", curr_suite + 1, + pad_width, term_width, + test_description(*t, -1)); + color_fprintf(debug_file(), PERF_COLOR_YELLOW, + " Skip (user override)\n"); + summary_tests_skipped++; + if (junit_filename) { + char *escaped_class = + xml_escape((const char *) + test_description(*t, -1)); + char *escaped_test = xml_escape("override"); + char *escaped_reason = + xml_escape("user override"); + + strbuf_addf(&junit_xml_buf, + " \n", + escaped_class, escaped_test); + strbuf_addf(&junit_xml_buf, + " \n", + escaped_reason); + strbuf_addstr(&junit_xml_buf, " \n"); + free(escaped_reason); + free(escaped_test); + free(escaped_class); + } + } continue; } @@ -654,8 +1390,9 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], } if (!sequential) { /* Parallel mode starts tests but doesn't finish them. Do that now. */ - for (size_t x = 0; x < num_tests; x++) - finish_test(child_tests, x, num_tests, width); + err = finish_tests_parallel(child_tests, num_tests, width); + if (err) + goto err_out; } } err_out: @@ -666,6 +1403,11 @@ static int __cmd_test(struct test_suite **suites, int argc, const char *argv[], for (size_t x = 0; x < num_tests; x++) finish_test(child_tests, x, num_tests, width); } + print_tests_summary(); + free(global_pfds); + free(global_pfd_indices); + global_pfds = NULL; + global_pfd_indices = NULL; free(child_tests); return err; } @@ -748,10 +1490,21 @@ static struct test_suite **build_suites(void) for (size_t i = 0, j = 0; i < ARRAY_SIZE(suites); i++, j = 0) \ while ((suite = suites[i][j++]) != NULL) - for_each_suite(t) + for_each_suite(t) { + if (t->setup) { + int ret = t->setup(t); + + if (ret < 0) { + errno = -ret; + return NULL; + } + } num_suites++; + } result = calloc(num_suites + 1, sizeof(struct test_suite *)); + if (!result) + return NULL; for (int pass = 1; pass <= 2; pass++) { for_each_suite(t) { @@ -796,6 +1549,10 @@ int cmd_test(int argc, const char **argv) OPT_STRING(0, "dso", &dso_to_test, "dso", "dso to test"), OPT_STRING(0, "objdump", &test_objdump_path, "path", "objdump binary to use for disassembly and annotations"), + OPT_UINTEGER(0, "failure-snippet-lines", &failure_snippet_lines, + "Number of lines to include in failure snippet, default 10"), + OPT_STRING_OPTARG('j', "junit", &junit_filename, "file", + "Generate junit XML output, default test.xml", "test.xml"), OPT_END() }; const char * const test_subcommands[] = { "list", NULL }; @@ -814,6 +1571,8 @@ int cmd_test(int argc, const char **argv) argc = parse_options_subcommand(argc, argv, test_options, test_subcommands, test_usage, 0); if (argc >= 1 && !strcmp(argv[0], "list")) { suites = build_suites(); + if (!suites) + return errno ? -errno : -ENOMEM; ret = perf_test__list(stdout, suites, argc - 1, argv + 1); free(suites); return ret; @@ -846,7 +1605,14 @@ int cmd_test(int argc, const char **argv) rlimit__bump_memlock(); suites = build_suites(); + if (!suites) { + int err = errno; + + intlist__delete(skiplist); + return err ? -err : -ENOMEM; + } ret = __cmd_test(suites, argc, argv, skiplist); free(suites); + intlist__delete(skiplist); return ret; } diff --git a/tools/perf/tests/shell/kvm.sh b/tools/perf/tests/shell/kvm.sh new file mode 100755 index 000000000000..5244c3ddfb1c --- /dev/null +++ b/tools/perf/tests/shell/kvm.sh @@ -0,0 +1,169 @@ +#!/bin/bash +# perf kvm tests +# SPDX-License-Identifier: GPL-2.0 + +set -e + +err=0 +perfdata=$(mktemp /tmp/__perf_kvm_test.perf.data.XXXXX) +qemu_pid_file=$(mktemp /tmp/__perf_kvm_test.qemu.pid.XXXXX) + +cleanup() { + rm -f "${perfdata}" + if [ -f "${qemu_pid_file}" ]; then + if [ -s "${qemu_pid_file}" ]; then + qemu_pid=$(cat "${qemu_pid_file}") + if [ -n "${qemu_pid}" ]; then + kill "${qemu_pid}" 2>/dev/null || true + fi + fi + rm -f "${qemu_pid_file}" + fi + trap - EXIT TERM INT +} + +trap_cleanup() { + echo "Unexpected signal in ${FUNCNAME[1]}" + cleanup + exit 1 +} +trap trap_cleanup EXIT TERM INT + +skip() { + echo "Skip: $1" + cleanup + exit 2 +} + +test_kvm_stat() { + echo "Testing perf kvm stat" + + local duration + local success=false + for duration in 1 2 4 8; do + echo "Recording kvm events for pid ${qemu_pid} (duration ${duration}s)..." + rm -f "${perfdata}" "${perfdata}".old + if ! perf kvm stat record -p "${qemu_pid}" -o "${perfdata}" \ + sleep ${duration} >/dev/null 2>&1; then + echo "perf kvm stat record failed, retrying..." + continue + fi + + if [ -e "${perfdata}" ] && \ + perf kvm -i "${perfdata}" stat report 2>&1 | grep -q "VM-EXIT"; then + success=true + break + fi + echo "No VM-EXIT events found, retrying..." + done + + if [ "$success" = false ]; then + echo "Failed to find VM-EXIT in report" + perf kvm -i "${perfdata}" stat report 2>&1 || true + err=1 + return + fi + + echo "perf kvm stat test [Success]" +} + +test_kvm_record_report() { + echo "Testing perf kvm record/report" + + local duration + local success=false + for duration in 1 2 4 8; do + echo "Recording kvm profile for pid ${qemu_pid} (duration ${duration}s)..." + rm -f "${perfdata}" "${perfdata}".old + + perf kvm --host record -p "${qemu_pid}" -o "${perfdata}" \ + -e cpu-clock sleep ${duration} + + if [ -e "${perfdata}" ] && \ + perf kvm -i "${perfdata}" report --stdio 2>&1 | grep -q "Event count"; then + success=true + break + fi + echo "No samples or report failed, retrying..." + done + + if [ "$success" = false ]; then + echo "Failed to report kvm profile" + perf kvm -i "${perfdata}" report --stdio 2>&1 || true + err=1 + return + fi + + echo "perf kvm record/report test [Success]" +} + +test_kvm_buildid_list() { + echo "Testing perf kvm buildid-list" + + # We reuse the perf.data from the previous record test + if ! perf kvm --host -i "${perfdata}" buildid-list 2>&1 | grep -q "."; then + echo "Failed to list buildids" + perf kvm --host -i "${perfdata}" buildid-list 2>&1 + err=1 + return + fi + + echo "perf kvm buildid-list test [Success]" +} + +setup_qemu() { + # Find qemu + if [ "$(uname -m)" = "x86_64" ]; then + qemu="qemu-system-x86_64" + elif [ "$(uname -m)" = "aarch64" ]; then + qemu="qemu-system-aarch64" + elif [ "$(uname -m)" = "s390x" ]; then + qemu="qemu-system-s390x" + elif [ "$(uname -m)" = "ppc64le" ]; then + qemu="qemu-system-ppc64" + else + qemu="qemu-system-$(uname -m)" + fi + + if ! which -s "$qemu"; then + skip "$qemu not found" + fi + + if [ ! -r /dev/kvm ] || [ ! -w /dev/kvm ]; then + skip "/dev/kvm not accessible" + fi + + if ! perf kvm stat record -o /dev/null -a sleep 0.01 >/dev/null 2>&1; then + skip "No permission to record kvm events" + fi + + echo "Starting $qemu..." + # Start qemu in background, detached, with pidfile + # We use -display none -daemonize and a monitor to keep it alive/controllable if needed + # We don't need a real kernel, just KVM active. + if ! $qemu -enable-kvm -display none -daemonize -pidfile "${qemu_pid_file}" -monitor none; then + echo "Failed to start qemu" + err=1 + return + fi + + # Wait a bit for qemu to start + sleep 1 + qemu_pid=$(cat "${qemu_pid_file}") + + if ! kill -0 "${qemu_pid}" 2>/dev/null; then + echo "Qemu process failed to stay alive" + err=1 + return + fi +} + +setup_qemu +if [ $err -eq 0 ]; then + test_kvm_stat + test_kvm_record_report + test_kvm_buildid_list +fi + +cleanup +exit $err diff --git a/tools/perf/tests/shell/lib/perf_metric_validation.py b/tools/perf/tests/shell/lib/perf_metric_validation.py index dea8ef1977bf..3d52f94f22b9 100644 --- a/tools/perf/tests/shell/lib/perf_metric_validation.py +++ b/tools/perf/tests/shell/lib/perf_metric_validation.py @@ -383,10 +383,13 @@ class Validator: wl = workload.split() command.extend(wl) print(" ".join(command)) - cmd = subprocess.run(command, stderr=subprocess.PIPE, encoding='utf-8') - data = [x+'}' for x in cmd.stderr.split('}\n') if x] - if data[0][0] != '{': - data[0] = data[0][data[0].find('{'):] + cmd = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, encoding='utf-8') + lines = cmd.stderr.splitlines() + cmd.stdout.splitlines() + data = [] + for line in lines: + line = line.strip() + if line.startswith('{') and line.endswith('}'): + data.append(line) return data def collect_perf(self, workload: str): diff --git a/tools/perf/tests/shell/lib/perf_record.sh b/tools/perf/tests/shell/lib/perf_record.sh new file mode 100644 index 000000000000..2b9e11b66dc7 --- /dev/null +++ b/tools/perf/tests/shell/lib/perf_record.sh @@ -0,0 +1,58 @@ +# SPDX-License-Identifier: GPL-2.0 + +PERF_RECORD_LOGS=() + +perf_record_with_retry() { + local perfdata="$1" + local check_cmd="$2" + local testprog_base="$3" + shift 3 + + local logfile + logfile=$(mktemp /tmp/__perf_record_retry.XXXXXX) + PERF_RECORD_LOGS+=("$logfile") + + # Save the e flag state and disable it + local save_e + if [[ $- == *e* ]]; then + save_e="set -e" + else + save_e="set +e" + fi + set +e + + local duration + local first_run=true + local ret=1 + local cmd_prefix="perf record" + if [ -n "${PERF_RECORD_CMD}" ]; then + cmd_prefix="${PERF_RECORD_CMD}" + fi + + for duration in 0.01 0.1 0.3 1.0 2.0; do + rm -f "${perfdata}".old + ${cmd_prefix} "$@" -o "${perfdata}" ${testprog_base} ${duration} > "$logfile" 2>&1 + local record_exit=$? + + if [ "$first_run" = true ] && [ $record_exit -ne 0 ]; then + ret=2 + break + fi + first_run=false + + if [ -e "${perfdata}" ] && eval "${check_cmd}"; then + ret=0 + break + fi + done + + eval "$save_e" + return $ret +} + +perf_record_cleanup() { + for logfile in "${PERF_RECORD_LOGS[@]}"; do + rm -f "$logfile" + done + PERF_RECORD_LOGS=() +} diff --git a/tools/perf/tests/shell/pipe_test.sh b/tools/perf/tests/shell/pipe_test.sh index e459aa99a951..ce68d850c983 100755 --- a/tools/perf/tests/shell/pipe_test.sh +++ b/tools/perf/tests/shell/pipe_test.sh @@ -12,8 +12,8 @@ skip_test_missing_symbol ${sym} data=$(mktemp /tmp/perf.data.XXXXXX) data2=$(mktemp /tmp/perf.data2.XXXXXX) -prog="perf test -w noploop" -[ "$(uname -m)" = "s390x" ] && prog="$prog 3" +prog="perf test -w noploop 0.1" +[ "$(uname -m)" = "s390x" ] && prog="perf test -w noploop 3" err=0 set -e diff --git a/tools/perf/tests/shell/record+zstd_comp_decomp_multi_record.sh b/tools/perf/tests/shell/record+zstd_comp_decomp_multi_record.sh new file mode 100755 index 000000000000..c05ace8214ca --- /dev/null +++ b/tools/perf/tests/shell/record+zstd_comp_decomp_multi_record.sh @@ -0,0 +1,63 @@ +#!/bin/bash +# Zstd perf.data compression/decompression of multi-record data +# SPDX-License-Identifier: GPL-2.0 + +perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX) +recout=$(mktemp /tmp/__perf_test.zstd.rec.XXXXX) +injout=$(mktemp /tmp/__perf_test.zstd.inj.XXXXX) +perf_tool=perf + +cleanup() { + rm -f "${perfdata}" "${perfdata}".old "${perfdata}".decomp "${recout}" "${injout}" +} +trap cleanup EXIT TERM INT + +skip_if_no_z_record() { + $perf_tool record -h 2>&1 | grep -q -- '-z, --compression-level' +} + +collect_z_record() { + echo "Collecting compressed record file:" + [ "$(uname -m)" != s390x ] && gflag='-g' + $perf_tool record -o "${perfdata}" $gflag -z -F max -m 32M --per-thread -- \ + $perf_tool test -w thloop 5 1 \ + >/dev/null 2>"${recout}" +} + +check_record() { + echo "Checking record did not fail to write data:" + if grep -q "failed to write perf data" "${recout}"; then + cat "${recout}" + return 1 + fi +} + +check_decompress() { + echo "Checking compressed file decompresses cleanly:" + if ! $perf_tool inject -i "${perfdata}" -o "${perfdata}".decomp 2>"${injout}"; then + cat "${injout}" + return 1 + fi + if grep -Eqi "decompress|corrupt|failed to process type" "${injout}"; then + cat "${injout}" + return 1 + fi +} + +skip_if_no_z_record || exit 2 +collect_z_record +check_record || exit 1 + +# Need >1 record, else the multi-record path wasn't exercised. +# Skip rather than pass/fail spuriously. +nr=$($perf_tool report -i "${perfdata}" --stats 2>/dev/null | + awk '/COMPRESSED2 events:/ { print $3 }') +if [ -z "${nr}" ] || [ "${nr}" -lt 2 ]; then + echo "less than two compressed records (${nr:-0}), skipping" + exit 2 +fi +echo "Produced ${nr} compressed records" + +check_decompress +err=$? +exit $err diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh index 0f5841c479e7..fc78ca52075c 100755 --- a/tools/perf/tests/shell/record.sh +++ b/tools/perf/tests/shell/record.sh @@ -1,10 +1,13 @@ #!/bin/bash -# perf record tests (exclusive) # SPDX-License-Identifier: GPL-2.0 +# perf record tests set -e shelldir=$(dirname "$0") +. "${shelldir}"/lib/perf_record.sh + + # shellcheck source=lib/waiting.sh . "${shelldir}"/lib/waiting.sh @@ -39,6 +42,7 @@ cleanup() { rm -f "${perfdata}" rm -f "${perfdata}".old rm -f "${script_output}" + perf_record_cleanup trap - EXIT TERM INT } @@ -50,22 +54,20 @@ trap_cleanup() { } trap trap_cleanup EXIT TERM INT +check_per_thread() { + perf report -i "${perfdata}" -q | grep -q "${testsym}" +} + test_per_thread() { echo "Basic --per-thread mode test" - if ! perf record -o /dev/null --quiet ${testprog} 2> /dev/null - then + local ret=0 + perf_record_with_retry "${perfdata}" "check_per_thread" "perf test -w thloop" \ + --per-thread || ret=$? + if [ $ret -eq 2 ]; then echo "Per-thread record [Skipped event not supported]" return - fi - if ! perf record --per-thread -o "${perfdata}" ${testprog} 2> /dev/null - then - echo "Per-thread record [Failed record]" - err=1 - return - fi - if ! perf report -i "${perfdata}" -q | grep -q "${testsym}" - then - echo "Per-thread record [Failed missing output]" + elif [ $ret -eq 1 ]; then + echo "Per-thread record [Failed record or missing output]" err=1 return fi @@ -96,6 +98,10 @@ test_per_thread() { echo "Basic --per-thread mode test [Success]" } +check_register_capture() { + perf script -F ip,sym,iregs -i "${perfdata}" 2>/dev/null | grep -q "DI:" +} + test_register_capture() { echo "Register capture test" if ! perf list pmu | grep -q 'br_inst_retired.near_call' @@ -108,11 +114,12 @@ test_register_capture() { echo "Register capture test [Skipped missing registers]" return fi - if ! perf record -o - --intr-regs=di,r8,dx,cx -e br_inst_retired.near_call \ - -c 1000 --per-thread ${testprog} 2> /dev/null \ - | perf script -F ip,sym,iregs -i - 2> /dev/null \ - | grep -q "DI:" - then + + local ret=0 + perf_record_with_retry "${perfdata}" "check_register_capture" "perf test -w thloop" \ + --intr-regs=di,r8,dx,cx -e br_inst_retired.near_call -c 1000 --per-thread || ret=$? + + if [ $ret -ne 0 ]; then echo "Register capture test [Failed missing output]" err=1 return @@ -120,65 +127,66 @@ test_register_capture() { echo "Register capture test [Success]" } +check_system_wide() { + perf report -i "${perfdata}" -q | grep -q "${testsym}" +} + test_system_wide() { echo "Basic --system-wide mode test" - if ! perf record -aB --synth=no -o "${perfdata}" ${testprog} 2> /dev/null - then + local ret=0 + perf_record_with_retry "${perfdata}" "check_system_wide" "perf test -w thloop" \ + -aB --synth=no || ret=$? + if [ $ret -eq 2 ]; then echo "System-wide record [Skipped not supported]" return - fi - if ! perf report -i "${perfdata}" -q | grep -q "${testsym}" - then + elif [ $ret -eq 1 ]; then echo "System-wide record [Failed missing output]" err=1 return fi - if ! perf record -aB --synth=no -e cpu-clock,cs --threads=cpu \ - -o "${perfdata}" ${testprog} 2> /dev/null - then - echo "System-wide record [Failed record --threads option]" - err=1 - return - fi - if ! perf report -i "${perfdata}" -q | grep -q "${testsym}" - then - echo "System-wide record [Failed --threads missing output]" + + ret=0 + perf_record_with_retry "${perfdata}" "check_system_wide" "perf test -w thloop" \ + -aB --synth=no -e cpu-clock,cs --threads=cpu || ret=$? + if [ $ret -ne 0 ]; then + echo "System-wide record [Failed record --threads option or missing output]" err=1 return fi echo "Basic --system-wide mode test [Success]" } +check_workload() { + perf report -i "${perfdata}" -q | grep -q "${testsym}" +} + test_workload() { echo "Basic target workload test" - if ! perf record -o "${perfdata}" ${testprog} 2> /dev/null - then - echo "Workload record [Failed record]" + local ret=0 + perf_record_with_retry "${perfdata}" "check_workload" "perf test -w thloop" || ret=$? + if [ $ret -ne 0 ]; then + echo "Workload record [Failed record or missing output]" err=1 return fi - if ! perf report -i "${perfdata}" -q | grep -q "${testsym}" - then - echo "Workload record [Failed missing output]" - err=1 - return - fi - if ! perf record -e cpu-clock,cs --threads=package \ - -o "${perfdata}" ${testprog} 2> /dev/null - then - echo "Workload record [Failed record --threads option]" - err=1 - return - fi - if ! perf report -i "${perfdata}" -q | grep -q "${testsym}" - then - echo "Workload record [Failed --threads missing output]" + + ret=0 + perf_record_with_retry "${perfdata}" "check_workload" "perf test -w thloop" \ + -e cpu-clock,cs --threads=package || ret=$? + if [ $ret -ne 0 ]; then + echo "Workload record [Failed record --threads option or missing output]" err=1 return fi echo "Basic target workload test [Success]" } +check_branch_counter() { + perf report -i "${perfdata}" -D -q 2>/dev/null | grep -q "$br_cntr_output" && \ + perf script -i "${perfdata}" -F +brstackinsn,+brcntr 2>/dev/null | \ + grep -q "$br_cntr_script_output" +} + test_branch_counter() { echo "Branch counter test" # Check if the branch counter feature is supported @@ -190,67 +198,60 @@ test_branch_counter() { return fi done - if ! perf record -o "${perfdata}" -e "{branches:p,instructions}" -j any,counter ${testprog} 2> /dev/null - then - echo "Branch counter record test [Failed record]" - err=1 - return - fi - if ! perf report -i "${perfdata}" -D -q | grep -q "$br_cntr_output" - then - echo "Branch counter report test [Failed missing output]" - err=1 - return - fi - if ! perf script -i "${perfdata}" -F +brstackinsn,+brcntr | grep -q "$br_cntr_script_output" - then - echo " Branch counter script test [Failed missing output]" + local ret=0 + perf_record_with_retry "${perfdata}" "check_branch_counter" "perf test -w thloop" \ + -e "{branches:p,instructions}" -j any,counter || ret=$? + if [ $ret -ne 0 ]; then + echo "Branch counter test [Failed record or missing output]" err=1 return fi echo "Branch counter test [Success]" } +check_cgroup() { + perf report -i "${perfdata}" -D 2>/dev/null | grep -q "CGROUP" && \ + perf script -i "${perfdata}" -F cgroup 2>/dev/null | grep -q -v "unknown" +} + test_cgroup() { echo "Cgroup sampling test" - if ! perf record -aB --synth=cgroup --all-cgroups -o "${perfdata}" ${testprog} 2> /dev/null - then + local ret=0 + perf_record_with_retry "${perfdata}" "check_cgroup" "perf test -w thloop" \ + -aB --synth=cgroup --all-cgroups || ret=$? + if [ $ret -eq 2 ]; then echo "Cgroup sampling [Skipped not supported]" return - fi - if ! perf report -i "${perfdata}" -D | grep -q "CGROUP" - then + elif [ $ret -eq 1 ]; then echo "Cgroup sampling [Failed missing output]" err=1 return fi - if ! perf script -i "${perfdata}" -F cgroup | grep -q -v "unknown" - then - echo "Cgroup sampling [Failed cannot resolve cgroup names]" - err=1 - return - fi echo "Cgroup sampling test [Success]" } +check_uid() { + perf report -i "${perfdata}" -q | grep -q "${testsym}" +} + test_uid() { echo "Uid sampling test" - if ! perf record -aB --synth=no --uid "$(id -u)" -o "${perfdata}" ${testprog} \ - > "${script_output}" 2>&1 - then - if grep -q "libbpf.*EPERM" "${script_output}" + local ret=0 + perf_record_with_retry "${perfdata}" "check_uid" "perf test -w thloop" \ + -aB --synth=no --uid "$(id -u)" || ret=$? + if [ $ret -eq 2 ]; then + local logfile="${PERF_RECORD_LOGS[${#PERF_RECORD_LOGS[@]}-1]}" + if grep -q -E "libbpf.*EPERM|Access to performance monitoring" "$logfile" || \ + grep -q -E "Permission denied|Failure to open any events" "$logfile" then echo "Uid sampling [Skipped permissions]" return else echo "Uid sampling [Failed to record]" err=1 - # cat "${script_output}" return fi - fi - if ! perf report -i "${perfdata}" -q | grep -q "${testsym}" - then + elif [ $ret -eq 1 ]; then echo "Uid sampling [Failed missing output]" err=1 return diff --git a/tools/perf/tests/shell/record_lbr.sh b/tools/perf/tests/shell/record_lbr.sh index 78a02e90ece1..8d51afeb437b 100755 --- a/tools/perf/tests/shell/record_lbr.sh +++ b/tools/perf/tests/shell/record_lbr.sh @@ -1,9 +1,12 @@ #!/bin/bash -# perf record LBR tests (exclusive) # SPDX-License-Identifier: GPL-2.0 +# perf record LBR tests set -e +shelldir=$(dirname "$0") +. "${shelldir}"/lib/perf_record.sh + ParanoidAndNotRoot() { [ "$(id -u)" != 0 ] && [ "$(cat /proc/sys/kernel/perf_event_paranoid)" -gt $1 ] } @@ -22,6 +25,7 @@ cleanup() { rm -rf "${perfdata}" rm -rf "${perfdata}".old rm -rf "${perfdata}".txt + perf_record_cleanup trap - EXIT TERM INT } @@ -34,22 +38,28 @@ trap_cleanup() { trap trap_cleanup EXIT TERM INT +check_lbr_callgraph() { + perf report --stitch-lbr -i "${perfdata}" > "${perfdata}".txt 2>&1 +} + lbr_callgraph_test() { test="LBR callgraph" echo "$test" - if ! perf record -e cycles --call-graph lbr -o "${perfdata}" perf test -w thloop - then + set +e + perf_record_with_retry "${perfdata}" "check_lbr_callgraph" "perf test -w thloop" \ + -e cycles --call-graph lbr + local ret=$? + set -e + + if [ $ret -eq 2 ]; then echo "$test [Failed support missing]" if [ $err -eq 0 ] then err=2 fi return - fi - - if ! perf report --stitch-lbr -i "${perfdata}" > "${perfdata}".txt - then + elif [ $ret -eq 1 ]; then cat "${perfdata}".txt echo "$test [Failed in perf report]" err=1 @@ -59,6 +69,12 @@ lbr_callgraph_test() { echo "$test [Success]" } +check_lbr_samples() { + local out + out=$(perf report -D -i "${perfdata}" 2> /dev/null | grep -A1 'PERF_RECORD_SAMPLE') + [ "$(echo "$out" | grep -c 'PERF_RECORD_SAMPLE' || true)" -gt 0 ] +} + lbr_test() { local branch_flags=$1 local test="LBR $2 test" @@ -70,25 +86,27 @@ lbr_test() { local r echo "$test" - if ! perf record -e cycles $branch_flags -o "${perfdata}" perf test -w thloop - then + set +e + perf_record_with_retry "${perfdata}" "check_lbr_samples" "perf test -w thloop" \ + -e cycles $branch_flags + local ret=$? + set -e + + if [ $ret -eq 2 ]; then echo "$test [Failed support missing]" - perf record -e cycles $branch_flags -o "${perfdata}" perf test -w thloop || true if [ $err -eq 0 ] then err=2 fi return - fi - - out=$(perf report -D -i "${perfdata}" 2> /dev/null | grep -A1 'PERF_RECORD_SAMPLE') - sam_nr=$(echo "$out" | grep -c 'PERF_RECORD_SAMPLE' || true) - if [ $sam_nr -eq 0 ] - then + elif [ $ret -eq 1 ]; then echo "$test [Failed no samples captured]" err=1 return fi + + out=$(perf report -D -i "${perfdata}" 2> /dev/null | grep -A1 'PERF_RECORD_SAMPLE') + sam_nr=$(echo "$out" | grep -c 'PERF_RECORD_SAMPLE' || true) echo "$test: $sam_nr samples" bs_nr=$(echo "$out" | grep -c 'branch stack: nr:' || true) diff --git a/tools/perf/tests/shell/stat_all_metrics.sh b/tools/perf/tests/shell/stat_all_metrics.sh index 6fa585a1e34c..feeb34c6fa6d 100755 --- a/tools/perf/tests/shell/stat_all_metrics.sh +++ b/tools/perf/tests/shell/stat_all_metrics.sh @@ -12,93 +12,137 @@ system_wide_flag="-a" if ParanoidAndNotRoot 0 then system_wide_flag="" - test_prog="perf test -w noploop" + test_prog="perf test -w noploop 0.01" fi -err=0 +check_metric() { + local output="$1" + local status="$2" + local metric="$3" + + if [[ $status -ne 0 || ! "$output" =~ ${metric:0:50} ]]; then + return 1 + fi + + if [[ "$output" =~ "" || "$output" =~ "" ]]; then + return 1 + fi + + return 0 +} + +skip=0 +err=3 for m in $(perf list --raw-dump metrics); do echo "Testing $m" result=$(perf stat -M "$m" $system_wide_flag -- $test_prog 2>&1) result_err=$? - if [[ $result_err -eq 0 && "$result" =~ ${m:0:50} ]] - then - # No error result and metric shown. + + if check_metric "$result" $result_err "$m"; then + if [[ "$err" -ne 1 ]] + then + err=0 + fi continue fi - if [[ "$result" =~ "Cannot resolve IDs for" ]] + + if [[ "$result" =~ "Access to performance monitoring and observability operations is limited" || \ + "$result" =~ "in per-thread mode, enable system wide" || \ + "$result" =~ "" || \ + "$result" =~ "Cannot resolve IDs for" || \ + "$result" =~ "No supported events found" || \ + "$result" =~ "FP_ARITH" || \ + "$result" =~ "AMX" || \ + "$result" =~ "PMM" ]] then - echo "Metric contains missing events" - echo $result - err=1 # Fail - continue - elif [[ "$result" =~ \ - "Access to performance monitoring and observability operations is limited" ]] + true + else + result=$(perf stat -M "$m" $system_wide_flag -- perf test -w noploop 0.1 2>&1) + result_err=$? + + if check_metric "$result" $result_err "$m"; then + if [[ "$err" -ne 1 ]] + then + err=0 + fi + continue + fi + fi + + # If retry also failed, determine if we skip, ignore, or fail + if [[ "$result" =~ "Access to performance monitoring and observability operations is limited" ]] then - echo "Permission failure" + echo "[Skipped $m] Permission failure" echo $result if [[ $err -eq 0 ]] then - err=2 # Skip + skip=1 fi continue elif [[ "$result" =~ "in per-thread mode, enable system wide" ]] then - echo "Permissions - need system wide mode" + echo "[Skipped $m] Permissions - need system wide mode" echo $result if [[ $err -eq 0 ]] then - err=2 # Skip + skip=1 fi continue - elif [[ "$result" =~ "" ]] + elif [[ "$result" =~ "" || \ + "$result" =~ "Cannot resolve IDs for" || \ + "$result" =~ "No supported events found" ]] then - echo "Not supported events" + if [[ $(perf list --raw-dump $m) == "Default"* ]] + then + echo "[Ignored $m] failed but as a Default metric this can be expected" + echo $result + continue + fi + echo "[Skipped $m] Not supported events" echo $result if [[ $err -eq 0 ]] then - err=2 # Skip + skip=1 fi continue elif [[ "$result" =~ "" ]] then - echo "Not counted events" + echo "[Skipped $m] Not counted events" echo $result if [[ $err -eq 0 ]] then - err=2 # Skip + skip=1 fi continue elif [[ "$result" =~ "FP_ARITH" || "$result" =~ "AMX" ]] then - echo "FP issues" + echo "[Skipped $m] FP issues" echo $result if [[ $err -eq 0 ]] then - err=2 # Skip + skip=1 fi continue elif [[ "$result" =~ "PMM" ]] then - echo "Optane memory issues" + echo "[Skipped $m] Optane memory issues" echo $result if [[ $err -eq 0 ]] then - err=2 # Skip + skip=1 fi continue fi - # Failed, possibly the workload was too small so retry with something longer. - result=$(perf stat -M "$m" $system_wide_flag -- perf bench internals synthesize 2>&1) - result_err=$? - if [[ $result_err -eq 0 && "$result" =~ ${m:0:50} ]] - then - # No error result and metric shown. - continue - fi - echo "Metric '$m' has non-zero error '$result_err' or not printed in:" + echo "[Failed $m] has non-zero error '$result_err' or not printed/counted in:" echo "$result" err=1 done +# return SKIP only if no success returned +if [[ "$err" -eq 3 && "$skip" -eq 1 ]] +then + err=2 +fi + exit "$err" diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh index f43e28a136d3..11de77ee38ad 100755 --- a/tools/perf/tests/shell/stat_bpf_counters.sh +++ b/tools/perf/tests/shell/stat_bpf_counters.sh @@ -4,21 +4,26 @@ set -e -workload="perf test -w sqrtloop" +# Get the first allowed CPU +CPU=$(taskset -c -p $$ | awk -F': ' '{print $2}' | awk -F'[,-]' '{print $1}') +if [ -z "$CPU" ]; then + CPU=0 +fi +workload=(taskset -c "$CPU" awk 'BEGIN { for (i=0; i<10000000; i++) sum+=i }') -# check whether $2 is within +/- 20% of $1 +# check whether $2 is within +/- 15% of $1 compare_number() { first_num=$1 second_num=$2 - # upper bound is first_num * 120% - upper=$(expr $first_num + $first_num / 5 ) - # lower bound is first_num * 80% - lower=$(expr $first_num - $first_num / 5 ) + # upper bound is first_num * 115% + upper=$(expr $first_num + $first_num / 20 \* 3 ) + # lower bound is first_num * 85% + lower=$(expr $first_num - $first_num / 20 \* 3 ) if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then - echo "The difference between $first_num and $second_num are greater than 20%." + echo "The difference between $first_num and $second_num are greater than 15%." exit 1 fi } @@ -41,8 +46,15 @@ check_counts() test_bpf_counters() { printf "Testing --bpf-counters " - base_instructions=$(perf stat --no-big-num -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}') - bpf_instructions=$(perf stat --no-big-num --bpf-counters -e instructions -- $workload 2>&1 | awk '/instructions/ {print $1}') + base_instructions=$(perf stat --no-big-num -e instructions:u -- "${workload[@]}" 2>&1 | \ + awk -v i=0 -v c=0 '/instructions/ { \ + if ($1 != " 0) printf "%.0f", c; else print "&1 | \ + awk -v i=0 -v c=0 '/instructions/ { \ + if ($1 != " 0) printf "%.0f", c; else print "&1) - base_instructions=$(echo "$stat_output"| awk '/base_instructions/ {print $1}') - bpf_instructions=$(echo "$stat_output"| awk '/bpf_instructions/ {print $1}') + stat_output=$(perf stat --no-big-num \ + -e instructions/name=base_instructions/u,instructions/name=bpf_instructions/bu \ + -- "${workload[@]}" 2>&1) + base_instructions=$(echo "$stat_output"| \ + awk -v i=0 -v c=0 '/base_instructions/ { \ + if ($1 != " 0) printf "%.0f", c; else print " 0) printf "%.0f", c; else print "/dev/null 2>&1; then + echo "Skipping: no permission to record system-wide events (-a)" + exit 2 +fi + + pythonvalidator=$(dirname $0)/lib/perf_metric_validation.py rulefile=$(dirname $0)/lib/perf_metric_validation_rules.json tmpdir=$(mktemp -d /tmp/__perf_test.program.XXXXX) diff --git a/tools/perf/tests/shell/test_brstack.sh b/tools/perf/tests/shell/test_brstack.sh index 85233d435be6..71550e0b37ba 100755 --- a/tools/perf/tests/shell/test_brstack.sh +++ b/tools/perf/tests/shell/test_brstack.sh @@ -38,9 +38,13 @@ is_arm64() { [ "$(uname -m)" = "aarch64" ]; } +has_kaslr_bug() { + [ "$(uname -m)" != "aarch64" ]; +} + check_branches() { if ! tr -s ' ' '\n' < "$TMPDIR/perf.script" | grep -E -m1 -q "$1"; then - echo "Branches missing $1" + echo "ERROR: Branches missing $1" err=1 fi } @@ -48,6 +52,8 @@ check_branches() { test_user_branches() { echo "Testing user branch stack sampling" + start_err=$err + err=0 perf record -o "$TMPDIR/perf.data" --branch-filter any,save_type,u -- ${TESTPROG} > "$TMPDIR/record.txt" 2>&1 perf script -i "$TMPDIR/perf.data" --fields brstacksym > "$TMPDIR/perf.script" @@ -73,59 +79,105 @@ test_user_branches() { perf script -i "$TMPDIR/perf.data" --fields brstack | \ tr ' ' '\n' > "$TMPDIR/perf.script" - # There should be no kernel addresses with the u option, in either - # source or target addresses. - if grep -E -m1 "0x[89a-f][0-9a-f]{15}" $TMPDIR/perf.script; then - echo "ERROR: Kernel address found in user mode" + # There should be no kernel addresses in the target with the u option. + local regex="0x[89a-f][0-9a-f]{15}" + if has_kaslr_bug; then + # If the system has a kaslr bug that may leak kernel addresses + # in the source of something like an ERET/SYSRET. Make the regex + # more specific and just check the target address is in user + # code. + regex="^0x[0-9a-f]{0,16}/0x[89a-f][0-9a-f]{15}/" + fi + if grep -q -E -m1 "$regex" $TMPDIR/perf.script; then + echo "Testing user branch stack sampling [Failed kernel address found in user mode]" err=1 fi # some branch types are still not being tested: # IND COND_CALL COND_RET SYSRET SERROR NO_TX + if [ $err -eq 0 ]; then + echo "Testing user branch stack sampling [Passed]" + err=$start_err + else + echo "Testing user branch stack sampling [Failed]" + fi } test_trap_eret_branches() { echo "Testing trap & eret branches" + if ! is_arm64; then - echo "skip: not arm64" - else + echo "Testing trap & eret branches [Skipped not arm64]" + return + fi + start_err=$err + local ret=1 + for loops in 1000 10000 100000; do + err=0 perf record -o $TMPDIR/perf.data --branch-filter any,save_type,u,k -- \ - perf test -w traploop 1000 + perf test -w traploop $loops > "$TMPDIR/record.txt" 2>&1 perf script -i $TMPDIR/perf.data --fields brstacksym | \ tr ' ' '\n' > $TMPDIR/perf.script # BRBINF.TYPE == TRAP are mapped to PERF_BR_IRQ by the BRBE driver check_branches "^trap_bench\+[^ ]+/[^ ]/IRQ/" check_branches "^[^ ]+/trap_bench\+[^ ]+/ERET/" + if [ $err -eq 0 ]; then + ret=0 + break + fi + done + + if [ $ret -eq 0 ]; then + echo "Testing trap & eret branches [Passed]" + err=$start_err + else + echo "Testing trap & eret branches [Failed]" + err=1 fi } test_kernel_branches() { - echo "Testing that k option only includes kernel source addresses" + echo "Testing kernel branch sampling" - if ! perf record --branch-filter any,k -o- -- true > /dev/null; then - echo "skip: not enough privileges" - else + if ! perf record --branch-filter any,k -o- -- true > "$TMPDIR/record.txt" 2>&1; then + echo "Testing that k option [Skipped not enough privileges]" + return + fi + start_err=$err + local ret=1 + for loops in 1000 10000 100000; do + err=0 perf record -o $TMPDIR/perf.data --branch-filter any,k -- \ - perf bench syscall basic --loop 1000 + perf bench syscall basic --loop $loops > "$TMPDIR/record.txt" 2>&1 perf script -i $TMPDIR/perf.data --fields brstack | \ tr ' ' '\n' > $TMPDIR/perf.script # Example of branch entries: # "0xffffffff93bda241/0xffffffff93bda20f/M/-/-/..." - # Source addresses come first and target address can be either - # userspace or kernel even with k option, as long as the source - # is in kernel. + # Source addresses come first in user or kernel code. Next is the target + # address that must be in the kernel. - #Look for source addresses with top bit set - if ! grep -E -m1 "^0x[89a-f][0-9a-f]{15}" $TMPDIR/perf.script; then - echo "ERROR: Kernel branches missing" + # Look for source addresses with top bit set + if ! grep -q -E -m1 "^0x[89a-f][0-9a-f]{15}" $TMPDIR/perf.script; then err=1 fi - # Look for no source addresses without top bit set - if grep -E -m1 "^0x[0-7][0-9a-f]{0,15}" $TMPDIR/perf.script; then - echo "ERROR: User branches found with kernel filter" + # Look for no target addresses without top bit set + if grep -q -E -m1 "^0x[0-9a-f]{0,16}/0x[0-7][0-9a-f]{1,15}/" \ + $TMPDIR/perf.script; then err=1 fi + if [ $err -eq 0 ]; then + ret=0 + break + fi + done + + if [ $ret -eq 0 ]; then + echo "Testing kernel branch sampling [Passed]" + err=$start_err + else + echo "Testing kernel branch sampling [Failed]" + err=1 fi } @@ -136,14 +188,15 @@ test_filter() { test_filter_expect=$2 echo "Testing branch stack filtering permutation ($test_filter_filter,$test_filter_expect)" - perf record -o "$TMPDIR/perf.data" --branch-filter "$test_filter_filter,save_type,u" -- ${TESTPROG} > "$TMPDIR/record.txt" 2>&1 + perf record -o "$TMPDIR/perf.data" --branch-filter "$test_filter_filter,save_type,u" -- \ + ${TESTPROG} > "$TMPDIR/record.txt" 2>&1 perf script -i "$TMPDIR/perf.data" --fields brstack > "$TMPDIR/perf.script" # fail if we find any branch type that doesn't match any of the expected ones # also consider UNKNOWN branch types (-) if [ ! -s "$TMPDIR/perf.script" ] then - echo "Empty script output" + echo "Testing branch stack filtering [Failed empty script output]" err=1 return fi @@ -154,26 +207,44 @@ test_filter() { > "$TMPDIR/perf.script-filtered" || true if [ -s "$TMPDIR/perf.script-filtered" ] then - echo "Unexpected branch filter in script output" + echo "Testing branch stack filtering [Failed unexpected branch filter]" cat "$TMPDIR/perf.script" err=1 return fi + echo "Testing branch stack filtering [Passed]" } test_syscall() { echo "Testing syscalls" # skip if perf doesn't have enough privileges - if ! perf record --branch-filter any,k -o- -- true > /dev/null; then - echo "skip: not enough privileges" - else + if ! perf record --branch-filter any,k -o- -- true > "$TMPDIR/record.txt" 2>&1; then + echo "Testing syscalls [Skipped: not enough privileges]" + return + fi + start_err=$err + local ret=1 + for loops in 8000 30000 100000; do + err=0 perf record -o $TMPDIR/perf.data --branch-filter \ - any_call,save_type,u,k -c 10000 -- \ - perf bench syscall basic --loop 1000 + any_call,save_type,u,k -c 10007 -- \ + perf bench syscall basic --loop $loops > "$TMPDIR/record.txt" 2>&1 perf script -i $TMPDIR/perf.data --fields brstacksym | \ tr ' ' '\n' > $TMPDIR/perf.script check_branches "getppid[^ ]*/SYSCALL/" + if [ $err -eq 0 ]; then + ret=0 + break + fi + done + + if [ $ret -eq 0 ]; then + echo "Testing syscalls [Passed]" + err=$start_err + else + echo "Testing syscalls [Failed]" + err=1 fi } set -e diff --git a/tools/perf/tests/shell/trace_record_replay.sh b/tools/perf/tests/shell/trace_record_replay.sh index 88d30a03dcec..38fcafcdfb91 100755 --- a/tools/perf/tests/shell/trace_record_replay.sh +++ b/tools/perf/tests/shell/trace_record_replay.sh @@ -6,16 +6,46 @@ # shellcheck source=lib/probe.sh . "$(dirname $0)"/lib/probe.sh +# shellcheck source=lib/perf_record.sh +. "$(dirname $0)"/lib/perf_record.sh skip_if_no_perf_trace || exit 2 [ "$(id -u)" = 0 ] || exit 2 file=$(mktemp /tmp/temporary_file.XXXXX) +err=0 -perf trace record -o ${file} sleep 1 || exit 1 -if ! perf trace -i ${file} 2>&1 | grep nanosleep; then - echo "Failed: cannot find *nanosleep syscall" +cleanup() { + rm -f ${file} + perf_record_cleanup + trap - EXIT INT TERM +} + +trap_cleanup() { + echo "Unexpected signal in ${FUNCNAME[1]}" + cleanup + exit 1 +} +trap trap_cleanup EXIT INT TERM + +check_nanosleep() { + perf trace -i "${file}" 2>&1 | grep -q nanosleep +} + +PERF_RECORD_CMD="perf trace record" perf_record_with_retry "${file}" "check_nanosleep" "sleep" +err=$? + +if [ $err -ne 0 ]; then + if [ $err -eq 2 ]; then + logfile="${PERF_RECORD_LOGS[${#PERF_RECORD_LOGS[@]}-1]}" + echo "perf trace record failed. Log output:" + cat "$logfile" + else + echo "Failed: cannot find *nanosleep syscall" + fi + cleanup exit 1 fi -rm -f ${file} +cleanup +exit 0 diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h index 33de16dde737..c79f3ab93cd4 100644 --- a/tools/perf/tests/tests.h +++ b/tools/perf/tests/tests.h @@ -38,12 +38,14 @@ struct test_case { const char *skip_reason; test_fnptr run_case; bool exclusive; + void *priv; }; struct test_suite { const char *desc; struct test_case *test_cases; void *priv; + int (*setup)(struct test_suite *suite); }; #define DECLARE_SUITE(name) \ diff --git a/tools/perf/ui/browsers/hists.c b/tools/perf/ui/browsers/hists.c index 487c0b08c003..9c371b262b3b 100644 --- a/tools/perf/ui/browsers/hists.c +++ b/tools/perf/ui/browsers/hists.c @@ -3064,15 +3064,15 @@ static int evsel__hists_browse(struct evsel *evsel, int nr_events, const char *h browser->min_pcnt = min_pcnt; hist_browser__update_nr_entries(browser); + memset(options, 0, sizeof(options)); + memset(actions, 0, sizeof(actions)); + browser->pstack = pstack__new(3); if (browser->pstack == NULL) goto out; ui_helpline__push(helpline); - memset(options, 0, sizeof(options)); - memset(actions, 0, sizeof(actions)); - if (symbol_conf.col_width_list_str) perf_hpp__set_user_width(symbol_conf.col_width_list_str); diff --git a/tools/perf/util/arm-spe.c b/tools/perf/util/arm-spe.c index 71be979f5077..cc99b06e5980 100644 --- a/tools/perf/util/arm-spe.c +++ b/tools/perf/util/arm-spe.c @@ -1543,6 +1543,10 @@ static u64 **arm_spe__alloc_metadata(struct perf_record_auxtrace_info *info, hdr_sz = ptr[ARM_SPE_HEADER_SIZE]; *nr_cpu = ptr[ARM_SPE_CPUS_NUM]; + /* nr_cpu is used as a divisor below */ + if (*nr_cpu <= 0) + return NULL; + metadata = calloc(*nr_cpu, sizeof(*metadata)); if (!metadata) return NULL; diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c index 1539c1dc823c..1749359aa9a2 100644 --- a/tools/perf/util/auxtrace.c +++ b/tools/perf/util/auxtrace.c @@ -235,8 +235,12 @@ static int auxtrace_queues__grow(struct auxtrace_queues *queues, { unsigned int nr_queues = queues->nr_queues; struct auxtrace_queue *queue_array; + struct auxtrace_queue *old_array = queues->queue_array; unsigned int i; + if (!new_nr_queues) + return -EINVAL; + if (!nr_queues) nr_queues = AUXTRACE_INIT_NR_QUEUES; @@ -251,16 +255,17 @@ static int auxtrace_queues__grow(struct auxtrace_queues *queues, return -ENOMEM; for (i = 0; i < queues->nr_queues; i++) { - list_splice_tail(&queues->queue_array[i].head, + list_splice_tail(&old_array[i].head, &queue_array[i].head); - queue_array[i].tid = queues->queue_array[i].tid; - queue_array[i].cpu = queues->queue_array[i].cpu; - queue_array[i].set = queues->queue_array[i].set; - queue_array[i].priv = queues->queue_array[i].priv; + queue_array[i].tid = old_array[i].tid; + queue_array[i].cpu = old_array[i].cpu; + queue_array[i].set = old_array[i].set; + queue_array[i].priv = old_array[i].priv; } queues->nr_queues = nr_queues; queues->queue_array = queue_array; + free(old_array); return 0; } diff --git a/tools/perf/util/bpf-event.c b/tools/perf/util/bpf-event.c index 2294336f6e60..195516356f62 100644 --- a/tools/perf/util/bpf-event.c +++ b/tools/perf/util/bpf-event.c @@ -969,7 +969,8 @@ void __bpf_event__print_bpf_prog_info(struct perf_bpil *info_linear, { struct bpf_prog_info *info = &info_linear->info; __u64 required_arrays = (1UL << PERF_BPIL_JITED_KSYMS) | - (1UL << PERF_BPIL_JITED_FUNC_LENS); + (1UL << PERF_BPIL_JITED_FUNC_LENS) | + (1UL << PERF_BPIL_PROG_TAGS); __u32 *prog_lens; __u64 *prog_addrs; char name[KSYM_NAME_LEN]; diff --git a/tools/perf/util/bpf-filter.c b/tools/perf/util/bpf-filter.c index 1a2e7b388d57..bcd81084e342 100644 --- a/tools/perf/util/bpf-filter.c +++ b/tools/perf/util/bpf-filter.c @@ -629,24 +629,20 @@ struct perf_bpf_filter_expr *perf_bpf_filter_expr__new(enum perf_bpf_filter_term static bool check_bpf_filter_capable(void) { - bool used_root; + int fd; - if (perf_cap__capable(CAP_BPF, &used_root)) + if (perf_cap__capable(CAP_BPF)) return true; - if (!used_root) { - /* Check if root already pinned the filter programs and maps */ - int fd = get_pinned_fd("filters"); - - if (fd >= 0) { - close(fd); - return true; - } + /* Check if root already pinned the filter programs and maps */ + fd = get_pinned_fd("filters"); + if (fd >= 0) { + close(fd); + return true; } - pr_err("Error: BPF filter only works for %s!\n" - "\tPlease run 'perf record --setup-filter pin' as root first.\n", - used_root ? "root" : "users with the CAP_BPF capability"); + pr_err("Error: BPF filter only works for users with the CAP_BPF capability!\n" + "\tPlease run 'perf record --setup-filter pin' as root first.\n"); return false; } diff --git a/tools/perf/util/cap.c b/tools/perf/util/cap.c index ac6d1d9a523d..272bd8255ff1 100644 --- a/tools/perf/util/cap.c +++ b/tools/perf/util/cap.c @@ -12,7 +12,7 @@ #define MAX_LINUX_CAPABILITY_U32S _LINUX_CAPABILITY_U32S_3 -bool perf_cap__capable(int cap, bool *used_root) +bool perf_cap__capable(int cap) { struct __user_cap_header_struct header = { .version = _LINUX_CAPABILITY_VERSION_3, @@ -21,7 +21,6 @@ bool perf_cap__capable(int cap, bool *used_root) struct __user_cap_data_struct data[MAX_LINUX_CAPABILITY_U32S] = {}; __u32 cap_val; - *used_root = false; while (syscall(SYS_capget, &header, &data[0]) == -1) { /* Retry, first attempt has set the header.version correctly. */ if (errno == EINVAL && header.version != _LINUX_CAPABILITY_VERSION_3 && @@ -29,7 +28,6 @@ bool perf_cap__capable(int cap, bool *used_root) continue; pr_debug2("capget syscall failed (%m) fall back on root check\n"); - *used_root = true; return geteuid() == 0; } diff --git a/tools/perf/util/cap.h b/tools/perf/util/cap.h index c1b8ac033ccc..bf09fb20c779 100644 --- a/tools/perf/util/cap.h +++ b/tools/perf/util/cap.h @@ -18,7 +18,6 @@ #define CAP_BPF 39 #endif -/* Query if a capability is supported, used_root is set if the fallback root check was used. */ -bool perf_cap__capable(int cap, bool *used_root); +bool perf_cap__capable(int cap); #endif /* __PERF_CAP_H */ diff --git a/tools/perf/util/capstone.c b/tools/perf/util/capstone.c index 2c7feab61b7b..1e2b697c86fc 100644 --- a/tools/perf/util/capstone.c +++ b/tools/perf/util/capstone.c @@ -156,6 +156,7 @@ static void print_capstone_detail(cs_insn *insn, char *buf, size_t len, for (i = 0; i < insn->detail->x86.op_count; i++) { cs_x86_op *op = &insn->detail->x86.operands[i]; u64 orig_addr; + struct map *found_map = NULL; if (op->type != X86_OP_MEM) continue; @@ -171,19 +172,22 @@ static void print_capstone_detail(cs_insn *insn, char *buf, size_t len, if (dso__kernel(map__dso(map))) { /* * The kernel maps can be split into sections, let's - * find the map first and the search the symbol. + * find the map first and then search the symbol. */ - map = maps__find(map__kmaps(map), addr); - if (map == NULL) + found_map = maps__find(map__kmaps(map), addr); + if (found_map == NULL) continue; + map = found_map; } /* convert it to map-relative address for search */ addr = map__map_ip(map, addr); sym = map__find_symbol(map, addr); - if (sym == NULL) + if (sym == NULL) { + map__put(found_map); continue; + } if (addr == sym->start) { scnprintf(buf, len, "\t# %"PRIx64" <%s>", @@ -192,6 +196,7 @@ static void print_capstone_detail(cs_insn *insn, char *buf, size_t len, scnprintf(buf, len, "\t# %"PRIx64" <%s+%#"PRIx64">", orig_addr, sym->name, addr - sym->start); } + map__put(found_map); break; } } diff --git a/tools/perf/util/compress.h b/tools/perf/util/compress.h index 6cfecfca16f2..ec6c38129e24 100644 --- a/tools/perf/util/compress.h +++ b/tools/perf/util/compress.h @@ -54,7 +54,8 @@ int zstd_fini(struct zstd_data *data); ssize_t zstd_compress_stream_to_records(struct zstd_data *data, void *dst, size_t dst_size, void *src, size_t src_size, size_t max_record_size, - size_t process_header(void *record, size_t increment)); + ssize_t process_header(void *record, size_t dst_size, + size_t data_size)); size_t zstd_decompress_stream(struct zstd_data *data, void *src, size_t src_size, void *dst, size_t dst_size); @@ -75,7 +76,8 @@ ssize_t zstd_compress_stream_to_records(struct zstd_data *data __maybe_unused, void *dst __maybe_unused, size_t dst_size __maybe_unused, void *src __maybe_unused, size_t src_size __maybe_unused, size_t max_record_size __maybe_unused, - size_t process_header(void *record, size_t increment) __maybe_unused) + ssize_t process_header(void *record, size_t dst_size, + size_t data_size) __maybe_unused) { return 0; } diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c index 520670348311..15f99c2c2775 100644 --- a/tools/perf/util/cs-etm.c +++ b/tools/perf/util/cs-etm.c @@ -645,6 +645,8 @@ static int cs_etm__init_traceid_queue(struct cs_etm_queue *etmq, queue->tid); tidq->decode_thread = machine__findnew_thread(&etm->session->machines.host, -1, queue->tid); + if (!tidq->frontend_thread || !tidq->decode_thread) + goto out; tidq->packet = zalloc(sizeof(struct cs_etm_packet)); if (!tidq->packet) @@ -679,6 +681,8 @@ static int cs_etm__init_traceid_queue(struct cs_etm_queue *etmq, zfree(&tidq->prev_packet); zfree(&tidq->packet); out: + thread__zput(tidq->frontend_thread); + thread__zput(tidq->decode_thread); return rc; } @@ -1505,8 +1509,7 @@ cs_etm__get_trace(struct cs_etm_queue *etmq) etmq->buf_used = 0; etmq->buf_len = aux_buffer->size; etmq->buf = aux_buffer->data; - - return etmq->buf_len; + return 0; } /* @@ -2062,6 +2065,45 @@ static int cs_etm__end_block(struct cs_etm_queue *etmq, return 0; } + +static int cs_etm__flush_stack_cb(struct thread *thread, + void *data __maybe_unused) +{ + thread_stack__flush(thread); + return 0; +} + +static void cs_etm__flush_machine_stack(struct cs_etm_queue *etmq, pid_t pid) +{ + struct machine *machine; + + machine = machines__find(&etmq->etm->session->machines, pid); + if (machine) + machine__for_each_thread(machine, cs_etm__flush_stack_cb, NULL); +} + +static void cs_etm__flush_all_stack(struct cs_etm_queue *etmq) +{ + enum cs_etm_pid_fmt pid_fmt = cs_etm__get_pid_fmt(etmq); + + if (!etmq->etm->synth_opts.last_branch) + return; + + switch (pid_fmt) { + case CS_ETM_PIDFMT_CTXTID2: + /* Clear the guest stack if virtualization is supported */ + cs_etm__flush_machine_stack(etmq, DEFAULT_GUEST_KERNEL_ID); + fallthrough; + case CS_ETM_PIDFMT_CTXTID: + cs_etm__flush_machine_stack(etmq, HOST_KERNEL_ID); + break; + case CS_ETM_PIDFMT_NONE: + default: + break; + + } +} + /* * cs_etm__get_data_block: Fetch a block from the auxtrace_buffer queue * if need be. @@ -2073,20 +2115,33 @@ static int cs_etm__get_data_block(struct cs_etm_queue *etmq) { int ret; - if (!etmq->buf_len) { - ret = cs_etm__get_trace(etmq); - if (ret <= 0) - return ret; - /* - * We cannot assume consecutive blocks in the data file - * are contiguous, reset the decoder to force re-sync. - */ - ret = cs_etm_decoder__reset(etmq->decoder); - if (ret) - return ret; - } + /* The current block is not finished */ + if (etmq->buf_len) + return 1; + + ret = cs_etm__get_trace(etmq); + if (ret < 0) + return ret; - return etmq->buf_len; + /* No more buffer to read */ + if (!etmq->buf_len) + return 0; + + /* + * We cannot assume consecutive blocks in the data file + * are contiguous, reset the decoder to force re-sync. + */ + ret = cs_etm_decoder__reset(etmq->decoder); + if (ret) + return ret; + + /* + * Since the decoder is reset, this causes a global trace + * discontinuity. Flush all thread stacks. + */ + cs_etm__flush_all_stack(etmq); + + return 1; } static bool cs_etm__is_svc_instr(struct cs_etm_queue *etmq, diff --git a/tools/perf/util/data-convert-json.c b/tools/perf/util/data-convert-json.c index 9dc1e184cf3c..9c96d84df900 100644 --- a/tools/perf/util/data-convert-json.c +++ b/tools/perf/util/data-convert-json.c @@ -242,6 +242,7 @@ static int process_sample_event(const struct perf_tool *tool, trace_seq_init(&s); tep_print_field(&s, sample->raw_data, fields[i]); output_json_key_string(out, true, 3, fields[i]->name, s.buffer); + trace_seq_destroy(&s); i++; } diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c index 6fa104752de2..70daf4fd200e 100644 --- a/tools/perf/util/dso.c +++ b/tools/perf/util/dso.c @@ -394,7 +394,9 @@ int dso__decompress_kmodule_path(struct dso *dso, const char *name, { int fd = decompress_kmodule(dso, name, pathname, len); - close(fd); + /* decompress_kmodule() returns -1 on failure, don't close(-1) */ + if (fd >= 0) + close(fd); return fd >= 0 ? 0 : -1; } @@ -581,9 +583,18 @@ static char *dso__get_filename(struct dso *dso, const char *root_dir, goto out; if (!is_regular_file(name)) { + struct stat st; char *new_name; - if (errno != ENOENT || dso__nsinfo(dso) == NULL) + /* + * errno only reflects the failure reason when stat() itself + * failed: a successful stat() on a non-regular file (e.g. a + * directory) leaves a stale errno, which a previous failed + * iteration of the try_to_open_dso() fallback loop may have + * set to ENOENT. + */ + if (stat(name, &st) == 0 || errno != ENOENT || + dso__nsinfo(dso) == NULL) goto out; new_name = dso__filename_with_chroot(dso, name); @@ -639,10 +650,13 @@ static int __open_dso(struct dso *dso, struct machine *machine) mutex_lock(dso__lock(dso)); name = dso__get_filename(dso, machine ? machine->root_dir : "", &decomp); - if (name) + if (name) { fd = do_open(name); - else + } else { + if (errno == 0) + errno = ENOENT; fd = -errno; + } if (decomp) unlink(name); @@ -999,7 +1013,20 @@ static ssize_t dso_cache__memcpy(struct dso_cache *cache, u64 offset, u8 *data, u64 size, bool out) { u64 cache_offset = offset - cache->offset; - u64 cache_size = min(cache->size - cache_offset, size); + u64 cache_size; + + /* + * The RB tree matches using DSO__DATA_CACHE_SIZE, but a short + * pread may leave cache->size smaller. For a regular file a + * short pread only happens at end-of-file, so an offset past + * the valid data is EOF: return 0, matching what a direct + * pread() at that offset would return, and cached_io() then + * stops its read loop. + */ + if (cache_offset >= cache->size) + return 0; + + cache_size = min(cache->size - cache_offset, size); if (out) memcpy(data, cache->data + cache_offset, cache_size); @@ -1023,7 +1050,7 @@ static ssize_t file_read(struct dso *dso, struct machine *machine, if (dso__data(dso)->fd < 0) { dso__data(dso)->status = DSO_DATA_STATUS_ERROR; - ret = -errno; + ret = dso__data(dso)->fd; goto out; } @@ -1145,8 +1172,8 @@ static int file_size(struct dso *dso, struct machine *machine) try_to_open_dso(dso, machine); if (dso__data(dso)->fd < 0) { - ret = -errno; dso__data(dso)->status = DSO_DATA_STATUS_ERROR; + ret = dso__data(dso)->fd; goto out; } @@ -1936,7 +1963,12 @@ const u8 *dso__read_symbol(struct dso *dso, const char *symfs_filename, errno = SYMBOL_ANNOTATE_ERRNO__BPF_MISSING_BTF; return NULL; } - assert(len <= info_linear->info.jited_prog_len); + if (len > info_linear->info.jited_prog_len) { + pr_debug("BPF symbol length %zu exceeds jited_prog_len %u\n", + len, info_linear->info.jited_prog_len); + errno = SYMBOL_ANNOTATE_ERRNO__BPF_MISSING_BTF; + return NULL; + } *out_buf_len = len; return (const u8 *)(uintptr_t)(info_linear->info.jited_prog_insns); #else diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h index 54e470dd0730..b374dfabf49c 100644 --- a/tools/perf/util/dso.h +++ b/tools/perf/util/dso.h @@ -9,6 +9,7 @@ #include #include #include +#include #include "build-id.h" #include "debuginfo.h" #include "mutex.h" @@ -20,6 +21,40 @@ struct perf_env; #define DSO__NAME_KALLSYMS "[kernel.kallsyms]" #define DSO__NAME_KCORE "[kernel.kcore]" +#define DSO__NAME_GUEST_KALLSYMS "[guest.kernel.kallsyms]" +#define DSO__NAME_GUEST_KALLSYMS_PID_PREFIX "[guest.kernel.kallsyms." + +/* + * Validate names of the form "[guest.kernel.kallsyms.]", where + * is the PID of the guest VM and varies per guest, so it + * cannot be matched with strcmp() against a fixed string. + * + * Every character after the fixed prefix must be a decimal digit, + * with ']' immediately terminating the digit run and nothing + * following it. This rules out '/', "..", or any other character + * being smuggled into the name. + */ +static inline bool is_guest_kallsyms_pid_name(const char *name) +{ + const size_t prefix_len = sizeof(DSO__NAME_GUEST_KALLSYMS_PID_PREFIX) - 1; + size_t digits; + + if (strncmp(name, DSO__NAME_GUEST_KALLSYMS_PID_PREFIX, prefix_len) != 0) + return false; + + digits = strspn(name + prefix_len, "0123456789"); + if (digits == 0) + return false; + + /* ']' must terminate the digit run, with nothing trailing it */ + if (name[prefix_len + digits] != ']') + return false; + + if (name[prefix_len + digits + 1] != '\0') + return false; + + return true; +} /** * enum dso_binary_type - The kind of DSO generally associated with a memory @@ -894,8 +929,28 @@ static inline bool dso__is_kcore(const struct dso *dso) static inline bool dso__is_kallsyms(const struct dso *dso) { enum dso_binary_type bt = dso__binary_type(dso); + const char *name; + + if (bt == DSO_BINARY_TYPE__KALLSYMS || bt == DSO_BINARY_TYPE__GUEST_KALLSYMS) + return true; + + if (bt != DSO_BINARY_TYPE__NOT_FOUND) + return false; + + if (!dso__kernel(dso)) + return false; + + name = dso__long_name(dso); + if (!name) + return false; + + if (!strcmp(name, DSO__NAME_KALLSYMS)) + return true; + + if (!strcmp(name, DSO__NAME_GUEST_KALLSYMS)) + return true; - return bt == DSO_BINARY_TYPE__KALLSYMS || bt == DSO_BINARY_TYPE__GUEST_KALLSYMS; + return is_guest_kallsyms_pid_name(name); } bool dso__is_object_file(const struct dso *dso); diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c index 22e43a57dc95..e01103443b23 100644 --- a/tools/perf/util/evsel.c +++ b/tools/perf/util/evsel.c @@ -646,7 +646,7 @@ struct tep_event *evsel__tp_format(struct evsel *evsel) tp_format = trace_event__tp_format(evsel->tp_sys, evsel->tp_name); if (IS_ERR(tp_format)) { - int err = -PTR_ERR(evsel->tp_format); + int err = -PTR_ERR(tp_format); errno = err; pr_err("Error getting tracepoint format '%s': %m\n", diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h index c64005278687..a4f244a04686 100644 --- a/tools/perf/util/hist.h +++ b/tools/perf/util/hist.h @@ -709,6 +709,8 @@ struct block_hist { struct hist_entry he; }; +#define NO_ADDR 0 + #ifdef HAVE_SLANG_SUPPORT #include "../ui/keysyms.h" void attr_to_script(char *buf, struct perf_event_attr *attr); @@ -746,14 +748,16 @@ int evlist__tui_browse_hists(struct evlist *evlist __maybe_unused, static inline int __hist_entry__tui_annotate(struct hist_entry *he __maybe_unused, struct map_symbol *ms __maybe_unused, struct evsel *evsel __maybe_unused, - struct hist_browser_timer *hbt __maybe_unused) + struct hist_browser_timer *hbt __maybe_unused, + u64 al_addr __maybe_unused) { return 0; } static inline int hist_entry__tui_annotate(struct hist_entry *he __maybe_unused, struct evsel *evsel __maybe_unused, - struct hist_browser_timer *hbt __maybe_unused) + struct hist_browser_timer *hbt __maybe_unused, + u64 al_addr __maybe_unused) { return 0; } diff --git a/tools/perf/util/intel-bts.c b/tools/perf/util/intel-bts.c index 3625c6224750..9f5f378cf774 100644 --- a/tools/perf/util/intel-bts.c +++ b/tools/perf/util/intel-bts.c @@ -831,7 +831,7 @@ int intel_bts_process_auxtrace_info(union perf_event *event, struct perf_session *session) { struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info; - size_t min_sz = sizeof(u64) * INTEL_BTS_SNAPSHOT_MODE; + size_t min_sz = sizeof(u64) * (INTEL_BTS_SNAPSHOT_MODE + 1); struct intel_bts *bts; int err; diff --git a/tools/perf/util/intel-pt.c b/tools/perf/util/intel-pt.c index 9b1011fe4826..fd636f5639ca 100644 --- a/tools/perf/util/intel-pt.c +++ b/tools/perf/util/intel-pt.c @@ -4411,7 +4411,7 @@ int intel_pt_process_auxtrace_info(union perf_event *event, struct perf_session *session) { struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info; - size_t min_sz = sizeof(u64) * INTEL_PT_PER_CPU_MMAPS; + size_t min_sz = sizeof(u64) * (INTEL_PT_PER_CPU_MMAPS + 1); struct intel_pt *pt; void *info_end; __u64 *info; diff --git a/tools/perf/util/libbfd.c b/tools/perf/util/libbfd.c index 63ea3fb53e77..6df72889e507 100644 --- a/tools/perf/util/libbfd.c +++ b/tools/perf/util/libbfd.c @@ -15,6 +15,7 @@ #ifdef HAVE_LIBBPF_SUPPORT #include #include +#include #endif #include #include @@ -510,7 +511,7 @@ int symbol__disassemble_bpf_libbfd(struct symbol *sym __maybe_unused, char tpath[PATH_MAX]; size_t buf_size; int nr_skip = 0; - char *buf; + char *buf = NULL; bfd *bfdf; int ret; FILE *s; @@ -552,6 +553,11 @@ int symbol__disassemble_bpf_libbfd(struct symbol *sym __maybe_unused, info_linear = info_node->info_linear; sub_id = dso__bpf_prog(dso)->sub_id; + /* jited_prog_insns is only valid if bpil_offs_to_addr() converted it */ + if (!(info_linear->arrays & (1UL << PERF_BPIL_JITED_INSNS))) { + ret = SYMBOL_ANNOTATE_ERRNO__BPF_MISSING_BTF; + goto out; + } info.buffer = (void *)(uintptr_t)(info_linear->info.jited_prog_insns); info.buffer_length = info_linear->info.jited_prog_len; @@ -581,6 +587,12 @@ int symbol__disassemble_bpf_libbfd(struct symbol *sym __maybe_unused, if (disassemble == NULL) abort(); + /* jited_ksyms is only valid if bpil_offs_to_addr() converted it */ + if (!(info_linear->arrays & (1UL << PERF_BPIL_JITED_KSYMS))) { + ret = SYMBOL_ANNOTATE_ERRNO__BPF_MISSING_BTF; + goto out; + } + fflush(s); do { const struct bpf_line_info *linfo = NULL; @@ -609,7 +621,7 @@ int symbol__disassemble_bpf_libbfd(struct symbol *sym __maybe_unused, if (!annotate_opts.hide_src_code && srcline) { args->offset = -1; - args->line = strdup(srcline); + args->line = (char *)srcline; args->line_nr = 0; args->fileloc = NULL; args->ms->sym = sym; @@ -634,9 +646,12 @@ int symbol__disassemble_bpf_libbfd(struct symbol *sym __maybe_unused, ret = 0; out: - free(prog_linfo); + bpf_prog_linfo__free(prog_linfo); btf__free(btf); - fclose(s); + if (s) { + fclose(s); + free(buf); + } bfd_close(bfdf); return ret; #else diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c index 9331b4cc19f1..cb138d1bdd7b 100644 --- a/tools/perf/util/machine.c +++ b/tools/perf/util/machine.c @@ -327,16 +327,22 @@ struct machine *machines__findnew(struct machines *machines, pid_t pid) if ((pid != HOST_KERNEL_ID) && (pid != DEFAULT_GUEST_KERNEL_ID) && (symbol_conf.guestmount)) { - snprintf(path, sizeof(path), "%s/%d", symbol_conf.guestmount, pid); + if (snprintf(path, sizeof(path), "%s/%d", + symbol_conf.guestmount, pid) >= (int)sizeof(path)) { + pr_err("Guest path too long for pid %d\n", pid); + machine = NULL; + goto out; + } if (access(path, R_OK)) { static struct strlist *seen; if (!seen) seen = strlist__new(NULL, NULL); - if (!strlist__has_entry(seen, path)) { + if (!seen || !strlist__has_entry(seen, path)) { pr_err("Can't access file %s\n", path); - strlist__add(seen, path); + if (seen) + strlist__add(seen, path); } machine = NULL; goto out; @@ -1229,27 +1235,35 @@ int machines__create_guest_kernel_maps(struct machines *machines) for (i = 0; i < items; i++) { if (!isdigit(namelist[i]->d_name[0])) { /* Filter out . and .. */ + free(namelist[i]); continue; } + errno = 0; pid = (pid_t)strtol(namelist[i]->d_name, &endp, 10); if ((*endp != '\0') || (endp == namelist[i]->d_name) || (errno == ERANGE)) { pr_debug("invalid directory (%s). Skipping.\n", namelist[i]->d_name); + free(namelist[i]); + continue; + } + if (snprintf(path, sizeof(path), "%s/%s/proc/kallsyms", + symbol_conf.guestmount, + namelist[i]->d_name) >= (int)sizeof(path)) { + pr_debug("Guest kallsyms path too long for %s. Skipping.\n", + namelist[i]->d_name); + free(namelist[i]); continue; } - snprintf(path, sizeof(path), "%s/%s/proc/kallsyms", - symbol_conf.guestmount, - namelist[i]->d_name); - ret = access(path, R_OK); - if (ret) { + if (access(path, R_OK)) { pr_debug("Can't access file %s\n", path); - goto failure; + free(namelist[i]); + continue; } machines__create_kernel_maps(machines, pid); + free(namelist[i]); } -failure: free(namelist); } @@ -1390,8 +1404,10 @@ static int maps__set_modules_path_dir(struct maps *maps, char *path, size_t path return -1; } /* Bounds check, should never happen. */ - if (root_len >= path_size) - return -1; + if (root_len >= path_size) { + ret = -1; + goto out; + } path[root_len++] = '/'; while ((dent = io_dir__readdir(&iod)) != NULL) { if (io_dir__is_dir(&iod, dent)) { @@ -1878,7 +1894,8 @@ int machine__process_fork_event(struct machine *machine, union perf_event *event * (fork) event that would have removed the thread was lost. Assume the * latter case and continue on as best we can. */ - if (thread__pid(parent) != (pid_t)event->fork.ppid) { + if (parent != NULL && + thread__pid(parent) != (pid_t)event->fork.ppid) { dump_printf("removing erroneous parent thread %d/%d\n", thread__pid(parent), thread__tid(parent)); machine__remove_thread(machine, parent); diff --git a/tools/perf/util/metricgroup.c b/tools/perf/util/metricgroup.c index 238c3e605bfe..bf98a01c4222 100644 --- a/tools/perf/util/metricgroup.c +++ b/tools/perf/util/metricgroup.c @@ -1614,8 +1614,10 @@ int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp, new_expr->metric_expr = old_expr->metric_expr; new_expr->metric_threshold = old_expr->metric_threshold; new_expr->metric_name = strdup(old_expr->metric_name); - if (!new_expr->metric_name) + if (!new_expr->metric_name) { + free(new_expr); return -ENOMEM; + } new_expr->metric_unit = old_expr->metric_unit; new_expr->runtime = old_expr->runtime; @@ -1627,6 +1629,7 @@ int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp, alloc_size = sizeof(*new_expr->metric_refs); new_expr->metric_refs = calloc(nr + 1, alloc_size); if (!new_expr->metric_refs) { + zfree(&new_expr->metric_name); free(new_expr); return -ENOMEM; } @@ -1643,6 +1646,7 @@ int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp, alloc_size = sizeof(*new_expr->metric_events); new_expr->metric_events = calloc(nr + 1, alloc_size); if (!new_expr->metric_events) { + zfree(&new_expr->metric_name); zfree(&new_expr->metric_refs); free(new_expr); return -ENOMEM; @@ -1653,6 +1657,7 @@ int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp, evsel = old_expr->metric_events[idx]; evsel = evlist__find_evsel(evlist, evsel->core.idx); if (evsel == NULL) { + zfree(&new_expr->metric_name); zfree(&new_expr->metric_events); zfree(&new_expr->metric_refs); free(new_expr); diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c index 779fe1280a56..a65a4d9f8cc8 100644 --- a/tools/perf/util/python.c +++ b/tools/perf/util/python.c @@ -872,11 +872,11 @@ static void pyrf_counts_values__delete(struct pyrf_counts_values *pcounts_values 0, help } static PyMemberDef pyrf_counts_values_members[] = { - counts_values_member_def(val, T_ULONG, "Value of event"), - counts_values_member_def(ena, T_ULONG, "Time for which enabled"), - counts_values_member_def(run, T_ULONG, "Time for which running"), - counts_values_member_def(id, T_ULONG, "Unique ID for an event"), - counts_values_member_def(lost, T_ULONG, "Num of lost samples"), + counts_values_member_def(val, T_ULONGLONG, "Value of event"), + counts_values_member_def(ena, T_ULONGLONG, "Time for which enabled"), + counts_values_member_def(run, T_ULONGLONG, "Time for which running"), + counts_values_member_def(id, T_ULONGLONG, "Unique ID for an event"), + counts_values_member_def(lost, T_ULONGLONG, "Num of lost samples"), { .name = NULL, }, }; @@ -886,8 +886,15 @@ static PyObject *pyrf_counts_values_get_values(struct pyrf_counts_values *self, if (!vals) return NULL; - for (int i = 0; i < 5; i++) - PyList_SetItem(vals, i, PyLong_FromLong(self->values.values[i])); + for (int i = 0; i < 5; i++) { + PyObject *val = PyLong_FromUnsignedLongLong(self->values.values[i]); + + if (!val) { + Py_DECREF(vals); + return NULL; + } + PyList_SetItem(vals, i, val); + } return vals; } @@ -898,19 +905,34 @@ static int pyrf_counts_values_set_values(struct pyrf_counts_values *self, PyObje Py_ssize_t size; PyObject *item = NULL; + if (list == NULL) { + PyErr_SetString(PyExc_TypeError, "cannot delete attribute"); + return -1; + } + if (!PyList_Check(list)) { PyErr_SetString(PyExc_TypeError, "Value assigned must be a list"); return -1; } size = PyList_Size(list); + if (size != 5) { + PyErr_SetString(PyExc_ValueError, "List must have exactly 5 entries"); + return -1; + } + for (Py_ssize_t i = 0; i < size; i++) { + unsigned long long val; + item = PyList_GetItem(list, i); if (!PyLong_Check(item)) { PyErr_SetString(PyExc_TypeError, "List members should be numbers"); return -1; } - self->values.values[i] = PyLong_AsLong(item); + val = PyLong_AsUnsignedLongLong(item); + if (val == (unsigned long long)-1 && PyErr_Occurred()) + return -1; + self->values.values[i] = val; } return 0; @@ -1074,11 +1096,21 @@ static PyObject *pyrf_evsel__open(struct pyrf_evsel *pevsel, &pcpus, &pthreads, &group, &inherit)) return NULL; - if (pthreads != NULL) + if (pthreads != NULL && pthreads != Py_None) { + if (!PyObject_TypeCheck(pthreads, &pyrf_thread_map__type)) { + PyErr_SetString(PyExc_TypeError, "threads must be a thread_map"); + return NULL; + } threads = ((struct pyrf_thread_map *)pthreads)->threads; + } - if (pcpus != NULL) + if (pcpus != NULL && pcpus != Py_None) { + if (!PyObject_TypeCheck(pcpus, &pyrf_cpu_map__type)) { + PyErr_SetString(PyExc_TypeError, "cpus must be a cpu_map"); + return NULL; + } cpus = ((struct pyrf_cpu_map *)pcpus)->cpus; + } evsel->core.attr.inherit = inherit; /* @@ -2034,8 +2066,10 @@ static PyObject *pyrf__parse_events(PyObject *self, PyObject *args) if (!PyArg_ParseTuple(args, "s|OO", &input, &pcpus, &pthreads)) return NULL; - threads = pthreads ? ((struct pyrf_thread_map *)pthreads)->threads : NULL; - cpus = pcpus ? ((struct pyrf_cpu_map *)pcpus)->cpus : NULL; + threads = (pthreads && pthreads != Py_None) ? + ((struct pyrf_thread_map *)pthreads)->threads : NULL; + cpus = (pcpus && pcpus != Py_None) ? + ((struct pyrf_cpu_map *)pcpus)->cpus : NULL; parse_events_error__init(&err); evlist__init(&evlist, cpus, threads); @@ -2062,8 +2096,10 @@ static PyObject *pyrf__parse_metrics(PyObject *self, PyObject *args) if (!PyArg_ParseTuple(args, "s|OO", &input, &pcpus, &pthreads)) return NULL; - threads = pthreads ? ((struct pyrf_thread_map *)pthreads)->threads : NULL; - cpus = pcpus ? ((struct pyrf_cpu_map *)pcpus)->cpus : NULL; + threads = (pthreads && pthreads != Py_None) ? + ((struct pyrf_thread_map *)pthreads)->threads : NULL; + cpus = (pcpus && pcpus != Py_None) ? + ((struct pyrf_cpu_map *)pcpus)->cpus : NULL; evlist__init(&evlist, cpus, threads); ret = metricgroup__parse_groups(&evlist, /*pmu=*/"all", input, @@ -2123,6 +2159,8 @@ static int pyrf__metrics_cb(const struct pmu_metric *pm, Py_XDECREF(dict); return -ENOMEM; } + Py_DECREF(key); + Py_DECREF(value); if (!add_to_dict(dict, "MetricName", pm->metric_name) || !add_to_dict(dict, "PMU", pm->pmu) || diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c index af0df841d640..0d4f1f29b335 100644 --- a/tools/perf/util/symbol.c +++ b/tools/perf/util/symbol.c @@ -2326,8 +2326,7 @@ static bool symbol__read_kptr_restrict(void) { bool value = false; FILE *fp = fopen("/proc/sys/kernel/kptr_restrict", "r"); - bool used_root; - bool cap_syslog = perf_cap__capable(CAP_SYSLOG, &used_root); + bool cap_syslog = perf_cap__capable(CAP_SYSLOG); if (fp != NULL) { char line[8]; diff --git a/tools/perf/util/synthetic-events.c b/tools/perf/util/synthetic-events.c index c5b0ba775098..ef0586cbc103 100644 --- a/tools/perf/util/synthetic-events.c +++ b/tools/perf/util/synthetic-events.c @@ -1058,7 +1058,7 @@ int perf_event__synthesize_threads(const struct perf_tool *tool, else thread_nr = nr_threads_synthesize; - if (thread_nr <= 1) { + if (thread_nr <= 1 || n <= 1) { err = __perf_event__synthesize_threads(tool, process, machine, needs_mmap, mmap_data, @@ -1104,8 +1104,8 @@ int perf_event__synthesize_threads(const struct perf_tool *tool, } err = 0; out_join: - for (i = 0; i < thread_nr; i++) - pthread_join(synthesize_threads[i], NULL); + for (j = 0; j < i; j++) + pthread_join(synthesize_threads[j], NULL); free(args); free_threads: free(synthesize_threads); diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c index c6a0a27b12c2..47d5922efdee 100644 --- a/tools/perf/util/thread-stack.c +++ b/tools/perf/util/thread-stack.c @@ -642,7 +642,7 @@ void thread_stack__br_sample(struct thread *thread, int cpu, sz -= nr; be = &dst->entries[nr]; nr = min(ts->br_stack_pos, sz); - memcpy(be, &src->entries[0], bsz * ts->br_stack_pos); + memcpy(be, &src->entries[0], bsz * nr); } } diff --git a/tools/perf/util/trace-event-read.c b/tools/perf/util/trace-event-read.c index ecbbb93f0185..52ed496d92c3 100644 --- a/tools/perf/util/trace-event-read.c +++ b/tools/perf/util/trace-event-read.c @@ -25,18 +25,18 @@ static int input_fd; static ssize_t trace_data_size; static bool repipe; -static int __do_read(int fd, void *buf, int size) +static ssize_t __do_read(int fd, void *buf, size_t size) { - int rsize = size; + size_t rsize = size; while (size) { - int ret = read(fd, buf, size); + ssize_t ret = read(fd, buf, size); if (ret <= 0) return -1; if (repipe) { - int retw = write(STDOUT_FILENO, buf, ret); + ssize_t retw = write(STDOUT_FILENO, buf, ret); if (retw <= 0 || retw != ret) { pr_debug("repiping input file"); @@ -51,13 +51,13 @@ static int __do_read(int fd, void *buf, int size) return rsize; } -static int do_read(void *data, int size) +static ssize_t do_read(void *data, size_t size) { - int r; + ssize_t r; r = __do_read(input_fd, data, size); if (r <= 0) { - pr_debug("reading input file (size expected=%d received=%d)", + pr_debug("reading input file (size expected=%zu received=%zd)", size, r); return -1; } @@ -68,10 +68,10 @@ static int do_read(void *data, int size) } /* If it fails, the next read will report it */ -static void skip(int size) +static void skip(size_t size) { char buf[BUFSIZ]; - int r; + size_t r; while (size) { r = size > BUFSIZ ? BUFSIZ : size; @@ -127,6 +127,11 @@ static char *read_string(void) } } + if (size >= (int)sizeof(buf) - 1) { + pr_debug("string too long (max %zu bytes)", sizeof(buf) - 1); + goto out; + } + buf[size++] = c; if (!c) @@ -197,7 +202,7 @@ static int read_header_files(struct tep_handle *pevent) unsigned long long size; char *header_page; char buf[BUFSIZ]; - int ret = 0; + ssize_t ret = 0; if (do_read(buf, 12) < 0) return -1; @@ -245,7 +250,7 @@ static int read_header_files(struct tep_handle *pevent) static int read_ftrace_file(struct tep_handle *pevent, unsigned long long size) { - int ret; + ssize_t ret; char *buf; buf = malloc(size); @@ -271,7 +276,7 @@ static int read_ftrace_file(struct tep_handle *pevent, unsigned long long size) static int read_event_file(struct tep_handle *pevent, char *sys, unsigned long long size) { - int ret; + ssize_t ret; char *buf; buf = malloc(size); @@ -317,7 +322,7 @@ static int read_event_files(struct tep_handle *pevent) int systems; int count; int i,x; - int ret; + ssize_t ret; systems = read4(pevent); @@ -345,7 +350,7 @@ static int read_saved_cmdline(struct tep_handle *pevent) { unsigned long long size; char *buf; - int ret; + ssize_t ret; /* it can have 0 size */ size = read8(pevent); diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c index 0f031eb80b4c..452c691991ae 100644 --- a/tools/perf/util/util.c +++ b/tools/perf/util/util.c @@ -331,15 +331,9 @@ int perf_event_paranoid(void) bool perf_event_paranoid_check(int max_level) { - bool used_root; - - if (perf_cap__capable(CAP_SYS_ADMIN, &used_root)) - return true; - - if (!used_root && perf_cap__capable(CAP_PERFMON, &used_root)) - return true; - - return perf_event_paranoid() <= max_level; + return perf_cap__capable(CAP_SYS_ADMIN) || + perf_cap__capable(CAP_PERFMON) || + perf_event_paranoid() <= max_level; } int perf_tip(char **strp, const char *dirpath) diff --git a/tools/perf/util/zstd.c b/tools/perf/util/zstd.c index 57027e0ac7b6..d17726b8fb50 100644 --- a/tools/perf/util/zstd.c +++ b/tools/perf/util/zstd.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include "util/compress.h" #include "util/debug.h" @@ -30,9 +31,11 @@ int zstd_fini(struct zstd_data *data) ssize_t zstd_compress_stream_to_records(struct zstd_data *data, void *dst, size_t dst_size, void *src, size_t src_size, size_t max_record_size, - size_t process_header(void *record, size_t increment)) + ssize_t process_header(void *record, size_t dst_size, + size_t data_size)) { - size_t ret, size, compressed = 0; + size_t ret, compressed = 0; + ssize_t size; ZSTD_inBuffer input = { src, src_size, 0 }; ZSTD_outBuffer output; void *record; @@ -54,7 +57,10 @@ ssize_t zstd_compress_stream_to_records(struct zstd_data *data, void *dst, size_ while (input.pos < input.size) { record = dst; - size = process_header(record, 0); + size = process_header(record, dst_size, 0); + /* Output buffer full — cannot fit even the record header */ + if (size < 0) + goto reset; compressed += size; dst += size; dst_size -= size; @@ -65,17 +71,37 @@ ssize_t zstd_compress_stream_to_records(struct zstd_data *data, void *dst, size_ if (ZSTD_isError(ret)) { pr_err("failed to compress %ld bytes: %s\n", (long)src_size, ZSTD_getErrorName(ret)); - memcpy(dst, src, src_size); - return src_size; + goto reset; } - size = output.pos; - size = process_header(record, size); + compressed += output.pos; + dst += output.pos; + dst_size -= output.pos; + /* + * No progress: ZSTD couldn't emit any bytes into the + * remaining output buffer. Calling process_header + * with output.pos=0 would re-trigger header initialization, + * double-subtracting the header size from dst_size and + * underflowing the unsigned counter. + */ + if (output.pos == 0) + goto reset; + size = process_header(record, dst_size, output.pos); + if (size < 0) + goto reset; compressed += size; dst += size; dst_size -= size; } return compressed; + +reset: + /* Reset so the context is usable if the caller retries */ + ret = ZSTD_initCStream(data->cstream, data->comp_level); + if (ZSTD_isError(ret)) + pr_err("failed to reset compression context: %s\n", + ZSTD_getErrorName(ret)); + return -1; } size_t zstd_decompress_stream(struct zstd_data *data, void *src, size_t src_size, diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h index dd9144624dc9..3ffd14e9c948 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -15,121 +15,17 @@ __ret; \ }) -/* v6.12: 819513666966 ("sched_ext: Add cgroup support") */ -#define __COMPAT_scx_bpf_task_cgroup(p) \ - (bpf_ksym_exists(scx_bpf_task_cgroup) ? \ - scx_bpf_task_cgroup((p)) : NULL) - /* - * v6.13: The verb `dispatch` was too overloaded and confusing. kfuncs are - * renamed to unload the verb. - * - * Build error is triggered if old names are used. New binaries work with both - * new and old names. The compat macros will be removed on v6.15 release. + * v6.15: 950ad93df2fc ("bpf: add kfunc for populating cpumask bits") * - * scx_bpf_dispatch_from_dsq() and friends were added during v6.12 by - * 4c30f5ce4f7a ("sched_ext: Implement scx_bpf_dispatch[_vtime]_from_dsq()"). - * Preserve __COMPAT macros until v6.15. + * Compat macro will be dropped on v6.19 release. */ -void scx_bpf_dispatch___compat(struct task_struct *p, u64 dsq_id, u64 slice, u64 enq_flags) __ksym __weak; -void scx_bpf_dispatch_vtime___compat(struct task_struct *p, u64 dsq_id, u64 slice, u64 vtime, u64 enq_flags) __ksym __weak; -bool scx_bpf_consume___compat(u64 dsq_id) __ksym __weak; -void scx_bpf_dispatch_from_dsq_set_slice___compat(struct bpf_iter_scx_dsq *it__iter, u64 slice) __ksym __weak; -void scx_bpf_dispatch_from_dsq_set_vtime___compat(struct bpf_iter_scx_dsq *it__iter, u64 vtime) __ksym __weak; -bool scx_bpf_dispatch_from_dsq___compat(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak; -bool scx_bpf_dispatch_vtime_from_dsq___compat(struct bpf_iter_scx_dsq *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __ksym __weak; -int bpf_cpumask_populate(struct cpumask *dst, void *src, size_t src__sz) __ksym __weak; - -#define scx_bpf_dsq_insert(p, dsq_id, slice, enq_flags) \ - (bpf_ksym_exists(scx_bpf_dsq_insert) ? \ - scx_bpf_dsq_insert((p), (dsq_id), (slice), (enq_flags)) : \ - scx_bpf_dispatch___compat((p), (dsq_id), (slice), (enq_flags))) - -#define scx_bpf_dsq_insert_vtime(p, dsq_id, slice, vtime, enq_flags) \ - (bpf_ksym_exists(scx_bpf_dsq_insert_vtime) ? \ - scx_bpf_dsq_insert_vtime((p), (dsq_id), (slice), (vtime), (enq_flags)) : \ - scx_bpf_dispatch_vtime___compat((p), (dsq_id), (slice), (vtime), (enq_flags))) - -#define scx_bpf_dsq_move_to_local(dsq_id) \ - (bpf_ksym_exists(scx_bpf_dsq_move_to_local) ? \ - scx_bpf_dsq_move_to_local((dsq_id)) : \ - scx_bpf_consume___compat((dsq_id))) - -#define __COMPAT_scx_bpf_dsq_move_set_slice(it__iter, slice) \ - (bpf_ksym_exists(scx_bpf_dsq_move_set_slice) ? \ - scx_bpf_dsq_move_set_slice((it__iter), (slice)) : \ - (bpf_ksym_exists(scx_bpf_dispatch_from_dsq_set_slice___compat) ? \ - scx_bpf_dispatch_from_dsq_set_slice___compat((it__iter), (slice)) : \ - (void)0)) - -#define __COMPAT_scx_bpf_dsq_move_set_vtime(it__iter, vtime) \ - (bpf_ksym_exists(scx_bpf_dsq_move_set_vtime) ? \ - scx_bpf_dsq_move_set_vtime((it__iter), (vtime)) : \ - (bpf_ksym_exists(scx_bpf_dispatch_from_dsq_set_vtime___compat) ? \ - scx_bpf_dispatch_from_dsq_set_vtime___compat((it__iter), (vtime)) : \ - (void) 0)) - -#define __COMPAT_scx_bpf_dsq_move(it__iter, p, dsq_id, enq_flags) \ - (bpf_ksym_exists(scx_bpf_dsq_move) ? \ - scx_bpf_dsq_move((it__iter), (p), (dsq_id), (enq_flags)) : \ - (bpf_ksym_exists(scx_bpf_dispatch_from_dsq___compat) ? \ - scx_bpf_dispatch_from_dsq___compat((it__iter), (p), (dsq_id), (enq_flags)) : \ - false)) - -#define __COMPAT_scx_bpf_dsq_move_vtime(it__iter, p, dsq_id, enq_flags) \ - (bpf_ksym_exists(scx_bpf_dsq_move_vtime) ? \ - scx_bpf_dsq_move_vtime((it__iter), (p), (dsq_id), (enq_flags)) : \ - (bpf_ksym_exists(scx_bpf_dispatch_vtime_from_dsq___compat) ? \ - scx_bpf_dispatch_vtime_from_dsq___compat((it__iter), (p), (dsq_id), (enq_flags)) : \ - false)) +int bpf_cpumask_populate(struct bpf_cpumask *dst, void *src, size_t src__sz) __ksym __weak; #define __COMPAT_bpf_cpumask_populate(cpumask, src, size__sz) \ (bpf_ksym_exists(bpf_cpumask_populate) ? \ (bpf_cpumask_populate(cpumask, src, size__sz)) : -EOPNOTSUPP) -#define scx_bpf_dispatch(p, dsq_id, slice, enq_flags) \ - _Static_assert(false, "scx_bpf_dispatch() renamed to scx_bpf_dsq_insert()") - -#define scx_bpf_dispatch_vtime(p, dsq_id, slice, vtime, enq_flags) \ - _Static_assert(false, "scx_bpf_dispatch_vtime() renamed to scx_bpf_dsq_insert_vtime()") - -#define scx_bpf_consume(dsq_id) ({ \ - _Static_assert(false, "scx_bpf_consume() renamed to scx_bpf_dsq_move_to_local()"); \ - false; \ -}) - -#define scx_bpf_dispatch_from_dsq_set_slice(it__iter, slice) \ - _Static_assert(false, "scx_bpf_dispatch_from_dsq_set_slice() renamed to scx_bpf_dsq_move_set_slice()") - -#define scx_bpf_dispatch_from_dsq_set_vtime(it__iter, vtime) \ - _Static_assert(false, "scx_bpf_dispatch_from_dsq_set_vtime() renamed to scx_bpf_dsq_move_set_vtime()") - -#define scx_bpf_dispatch_from_dsq(it__iter, p, dsq_id, enq_flags) ({ \ - _Static_assert(false, "scx_bpf_dispatch_from_dsq() renamed to scx_bpf_dsq_move()"); \ - false; \ -}) - -#define scx_bpf_dispatch_vtime_from_dsq(it__iter, p, dsq_id, enq_flags) ({ \ - _Static_assert(false, "scx_bpf_dispatch_vtime_from_dsq() renamed to scx_bpf_dsq_move_vtime()"); \ - false; \ -}) - -#define __COMPAT_scx_bpf_dispatch_from_dsq_set_slice(it__iter, slice) \ - _Static_assert(false, "__COMPAT_scx_bpf_dispatch_from_dsq_set_slice() renamed to __COMPAT_scx_bpf_dsq_move_set_slice()") - -#define __COMPAT_scx_bpf_dispatch_from_dsq_set_vtime(it__iter, vtime) \ - _Static_assert(false, "__COMPAT_scx_bpf_dispatch_from_dsq_set_vtime() renamed to __COMPAT_scx_bpf_dsq_move_set_vtime()") - -#define __COMPAT_scx_bpf_dispatch_from_dsq(it__iter, p, dsq_id, enq_flags) ({ \ - _Static_assert(false, "__COMPAT_scx_bpf_dispatch_from_dsq() renamed to __COMPAT_scx_bpf_dsq_move()"); \ - false; \ -}) - -#define __COMPAT_scx_bpf_dispatch_vtime_from_dsq(it__iter, p, dsq_id, enq_flags) ({ \ - _Static_assert(false, "__COMPAT_scx_bpf_dispatch_vtime_from_dsq() renamed to __COMPAT_scx_bpf_dsq_move_vtime()"); \ - false; \ -}) - /** * __COMPAT_is_enq_cpu_selected - Test if SCX_ENQ_CPU_SELECTED is on * in a compatible way. We will preserve this __COMPAT helper until v6.16. diff --git a/tools/sched_ext/scx_flatcg.bpf.c b/tools/sched_ext/scx_flatcg.bpf.c index 2c720e3ecad5..2e757f3b1353 100644 --- a/tools/sched_ext/scx_flatcg.bpf.c +++ b/tools/sched_ext/scx_flatcg.bpf.c @@ -256,7 +256,7 @@ static void cgrp_cap_budget(struct cgv_node *cgv_node, struct fcg_cgrp_ctx *cgc) * and thus can't be updated and repositioned. Instead, we collect the * vtime deltas separately and apply it asynchronously here. */ - delta = __sync_fetch_and_sub(&cgc->cvtime_delta, cgc->cvtime_delta); + delta = __sync_fetch_and_and(&cgc->cvtime_delta, 0); cvtime = cgv_node->cvtime + delta; /* @@ -382,7 +382,7 @@ void BPF_STRUCT_OPS(fcg_enqueue, struct task_struct *p, u64 enq_flags) return; } - cgrp = __COMPAT_scx_bpf_task_cgroup(p); + cgrp = scx_bpf_task_cgroup(p); cgc = find_cgrp_ctx(cgrp); if (!cgc) goto out_release; @@ -508,7 +508,7 @@ void BPF_STRUCT_OPS(fcg_runnable, struct task_struct *p, u64 enq_flags) { struct cgroup *cgrp; - cgrp = __COMPAT_scx_bpf_task_cgroup(p); + cgrp = scx_bpf_task_cgroup(p); update_active_weight_sums(cgrp, true); bpf_cgroup_release(cgrp); } @@ -521,7 +521,7 @@ void BPF_STRUCT_OPS(fcg_running, struct task_struct *p) if (fifo_sched) return; - cgrp = __COMPAT_scx_bpf_task_cgroup(p); + cgrp = scx_bpf_task_cgroup(p); cgc = find_cgrp_ctx(cgrp); if (cgc) { /* @@ -564,11 +564,12 @@ void BPF_STRUCT_OPS(fcg_stopping, struct task_struct *p, bool runnable) if (!taskc->bypassed_at) return; - cgrp = __COMPAT_scx_bpf_task_cgroup(p); + cgrp = scx_bpf_task_cgroup(p); cgc = find_cgrp_ctx(cgrp); if (cgc) { __sync_fetch_and_add(&cgc->cvtime_delta, - p->se.sum_exec_runtime - taskc->bypassed_at); + (p->se.sum_exec_runtime - taskc->bypassed_at) * + FCG_HWEIGHT_ONE / (cgc->hweight ?: 1)); taskc->bypassed_at = 0; } bpf_cgroup_release(cgrp); @@ -578,7 +579,7 @@ void BPF_STRUCT_OPS(fcg_quiescent, struct task_struct *p, u64 deq_flags) { struct cgroup *cgrp; - cgrp = __COMPAT_scx_bpf_task_cgroup(p); + cgrp = scx_bpf_task_cgroup(p); update_active_weight_sums(cgrp, false); bpf_cgroup_release(cgrp); } @@ -766,10 +767,18 @@ void BPF_STRUCT_OPS(fcg_dispatch, s32 cpu, struct task_struct *prev) * cgroup to execute but the latter needs to be done in a loop * and we can't keep the lock held. Oh well... */ + s64 delta = now - cpuc->cur_at - cgrp_slice_ns; + bpf_spin_lock(&cgv_tree_lock); - __sync_fetch_and_add(&cgc->cvtime_delta, - (cpuc->cur_at + cgrp_slice_ns - now) * - FCG_HWEIGHT_ONE / (cgc->hweight ?: 1)); + /* keep the dividends positive, BPF division is unsigned */ + if (delta >= 0) + __sync_fetch_and_add(&cgc->cvtime_delta, + (u64)delta * FCG_HWEIGHT_ONE / + (cgc->hweight ?: 1)); + else + __sync_fetch_and_sub(&cgc->cvtime_delta, + (u64)-delta * FCG_HWEIGHT_ONE / + (cgc->hweight ?: 1)); bpf_spin_unlock(&cgv_tree_lock); } else { stat_inc(FCG_STAT_CNS_GONE); diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c index 4f65a7550fb2..19984b52323f 100644 --- a/tools/sched_ext/scx_qmap.bpf.c +++ b/tools/sched_ext/scx_qmap.bpf.c @@ -301,7 +301,7 @@ static void update_core_sched_head_seq(struct task_struct *p) * moving them to HIGHPRI_DSQ and then consuming them first. This makes minor * difference only when dsp_batch is larger than 1. * - * scx_bpf_dispatch[_vtime]_from_dsq() are allowed both from ops.dispatch() and + * scx_bpf_dsq_move[_vtime]() are allowed both from ops.dispatch() and * non-rq-lock holding BPF programs. As demonstration, this function is called * from qmap_dispatch() and monitor_timerfn(). */ @@ -320,12 +320,9 @@ static bool dispatch_highpri(bool from_timer) if (tctx->highpri) { /* exercise the set_*() and vtime interface too */ - __COMPAT_scx_bpf_dsq_move_set_slice( - BPF_FOR_EACH_ITER, slice_ns * 2); - __COMPAT_scx_bpf_dsq_move_set_vtime( - BPF_FOR_EACH_ITER, highpri_seq++); - __COMPAT_scx_bpf_dsq_move_vtime( - BPF_FOR_EACH_ITER, p, HIGHPRI_DSQ, 0); + scx_bpf_dsq_move_set_slice(BPF_FOR_EACH_ITER, slice_ns * 2); + scx_bpf_dsq_move_set_vtime(BPF_FOR_EACH_ITER, highpri_seq++); + scx_bpf_dsq_move_vtime(BPF_FOR_EACH_ITER, p, HIGHPRI_DSQ, 0); } } @@ -342,9 +339,8 @@ static bool dispatch_highpri(bool from_timer) else cpu = scx_bpf_pick_any_cpu(p->cpus_ptr, 0); - if (__COMPAT_scx_bpf_dsq_move(BPF_FOR_EACH_ITER, p, - SCX_DSQ_LOCAL_ON | cpu, - SCX_ENQ_PREEMPT)) { + if (scx_bpf_dsq_move(BPF_FOR_EACH_ITER, p, SCX_DSQ_LOCAL_ON | cpu, + SCX_ENQ_PREEMPT)) { if (cpu == this_cpu) { dispatched = true; __sync_fetch_and_add(&nr_expedited_local, 1); diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include index ded48263dd5e..c5bb7bf42f43 100644 --- a/tools/scripts/Makefile.include +++ b/tools/scripts/Makefile.include @@ -59,10 +59,18 @@ LLVM_SUFFIX := $(LLVM) endif $(call allow-override,CC,$(LLVM_PREFIX)clang$(LLVM_SUFFIX)) +$(call allow-override,CLANG,$(LLVM_PREFIX)clang$(LLVM_SUFFIX)) +$(call allow-override,HOSTCC,$(LLVM_PREFIX)clang$(LLVM_SUFFIX)) $(call allow-override,AR,$(LLVM_PREFIX)llvm-ar$(LLVM_SUFFIX)) +$(call allow-override,HOSTAR,$(LLVM_PREFIX)llvm-ar$(LLVM_SUFFIX)) $(call allow-override,LD,$(LLVM_PREFIX)ld.lld$(LLVM_SUFFIX)) +$(call allow-override,HOSTLD,$(LLVM_PREFIX)ld.lld$(LLVM_SUFFIX)) $(call allow-override,CXX,$(LLVM_PREFIX)clang++$(LLVM_SUFFIX)) $(call allow-override,STRIP,$(LLVM_PREFIX)llvm-strip$(LLVM_SUFFIX)) +$(call allow-override,LLVM_STRIP,$(LLVM_PREFIX)llvm-strip$(LLVM_SUFFIX)) +$(call allow-override,LLC,$(LLVM_PREFIX)llc$(LLVM_SUFFIX)) +$(call allow-override,LLVM_CONFIG,$(LLVM_PREFIX)llvm-config$(LLVM_SUFFIX)) +$(call allow-override,LLVM_OBJCOPY,$(LLVM_PREFIX)llvm-objcopy$(LLVM_SUFFIX)) else # Allow setting various cross-compile vars or setting CROSS_COMPILE as a prefix. $(call allow-override,CC,$(CROSS_COMPILE)gcc) @@ -70,26 +78,21 @@ $(call allow-override,AR,$(CROSS_COMPILE)ar) $(call allow-override,LD,$(CROSS_COMPILE)ld) $(call allow-override,CXX,$(CROSS_COMPILE)g++) $(call allow-override,STRIP,$(CROSS_COMPILE)strip) -endif - -CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq "__clang__"; echo $$?) -ifneq ($(LLVM),) -HOSTAR ?= $(LLVM_PREFIX)llvm-ar$(LLVM_SUFFIX) -HOSTCC ?= $(LLVM_PREFIX)clang$(LLVM_SUFFIX) -HOSTLD ?= $(LLVM_PREFIX)ld.lld$(LLVM_SUFFIX) -else -HOSTAR ?= ar -HOSTCC ?= gcc -HOSTLD ?= ld +# Host versions aren't prefixed +$(call allow-override,HOSTAR,ar) +$(call allow-override,HOSTCC,gcc) +$(call allow-override,HOSTLD,ld) + +# Some tools still require Clang, LLC and/or LLVM utils +$(call allow-override,CLANG,clang) +$(call allow-override,LLC,llc) +$(call allow-override,LLVM_CONFIG,llvm-config) +$(call allow-override,LLVM_OBJCOPY,llvm-objcopy) +$(call allow-override,LLVM_STRIP,llvm-strip) endif -# Some tools require Clang, LLC and/or LLVM utils -CLANG ?= clang -LLC ?= llc -LLVM_CONFIG ?= llvm-config -LLVM_OBJCOPY ?= llvm-objcopy -LLVM_STRIP ?= llvm-strip +CC_NO_CLANG := $(shell $(CC) -dM -E -x c /dev/null | grep -Fq "__clang__"; echo $$?) # Some tools require bpftool SYSTEM_BPFTOOL ?= bpftool diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index cd99c1956331..f3392cb3b7af 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -123,7 +123,7 @@ def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) lines.pop() # Filter out any extraneous non-test output that might have gotten mixed in. - return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)] + return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)] def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]: args = ['kunit.action=list_attr'] diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 0eb61de9abd4..a8eb223e57e1 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -896,6 +896,18 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel.assert_called_once_with( args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter='', filter_action=None, timeout=300) + def test_list_tests_with_prefix(self): + want = ['suite.test1', 'suite.test2', 'suite2.test1'] + self.linux_source_mock.run_kernel.return_value = [ + '[ 0.100000] TAP version 14', + '[ 0.200000] suite.test1', + '[ 0.200000] suite.test2', + '[ 0.300000] suite2.test1'] + + got = kunit._list_tests(self.linux_source_mock, + kunit.KunitExecRequest(None, None, None, False, False, '.kunit', 300, 'suite*', '', None, None, 'suite', False, False, False)) + self.assertEqual(got, want) + @mock.patch.object(kunit, '_list_tests') def test_run_isolated_by_suite(self, mock_tests): mock_tests.return_value = ['suite.test1', 'suite.test2', 'suite2.test1'] diff --git a/tools/testing/selftests/arm64/fp/fp-ptrace.c b/tools/testing/selftests/arm64/fp/fp-ptrace.c index a85c19e9524e..c3fcd47e3396 100644 --- a/tools/testing/selftests/arm64/fp/fp-ptrace.c +++ b/tools/testing/selftests/arm64/fp/fp-ptrace.c @@ -65,6 +65,9 @@ /* VL 128..2048 in powers of 2 */ #define MAX_NUM_VLS 5 +/* Sentinel for detecting buffer bytes the kernel did not write */ +#define REGSET_SENTINEL 0xa5 + /* * FPMR bits we can set without doing feature checks to see if values * are valid. @@ -181,6 +184,20 @@ static bool compare_buffer(const char *name, void *out, return false; } +static bool buffer_is_filled(const void *buffer, size_t size, + unsigned char value) +{ + const unsigned char *bytes = buffer; + size_t i; + + for (i = 0; i < size; i++) { + if (bytes[i] != value) + return false; + } + + return true; +} + struct test_config { int sve_vl_in; int sve_vl_expected; @@ -401,6 +418,7 @@ static bool check_ptrace_values_sve(pid_t child, struct test_config *config) struct user_sve_header *sve; struct user_fpsimd_state *fpsimd; struct iovec iov; + size_t buf_size; int ret, vq; bool pass = true; @@ -409,14 +427,16 @@ static bool check_ptrace_values_sve(pid_t child, struct test_config *config) vq = __sve_vq_from_vl(config->sve_vl_in); - iov.iov_len = SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, SVE_PT_REGS_SVE); - iov.iov_base = malloc(iov.iov_len); + buf_size = SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, SVE_PT_REGS_SVE); + iov.iov_len = buf_size; + iov.iov_base = malloc(buf_size); if (!iov.iov_base) { ksft_print_msg("OOM allocating %lu byte SVE buffer\n", iov.iov_len); return false; } + memset(iov.iov_base, REGSET_SENTINEL, buf_size); ret = ptrace(PTRACE_GETREGSET, child, NT_ARM_SVE, &iov); if (ret != 0) { ksft_print_msg("Failed to read initial SVE: %s (%d)\n", @@ -440,10 +460,16 @@ static bool check_ptrace_values_sve(pid_t child, struct test_config *config) } if (svcr_in & SVCR_SM) { - if (sve->size != sizeof(sve)) { + if (sve->size != sizeof(*sve)) { ksft_print_msg("NT_ARM_SVE reports data with PSTATE.SM\n"); pass = false; } + if (!buffer_is_filled(iov.iov_base + sizeof(*sve), + buf_size - sizeof(*sve), REGSET_SENTINEL)) { + ksft_print_msg("NT_ARM_SVE wrote beyond its header with PSTATE.SM\n"); + pass = false; + } + goto out; } else { if (sve->size != SVE_PT_SIZE(vq, sve->flags)) { ksft_print_msg("Mismatch in SVE header size: %d != %lu\n", @@ -485,6 +511,7 @@ static bool check_ptrace_values_ssve(pid_t child, struct test_config *config) struct user_sve_header *sve; struct user_fpsimd_state *fpsimd; struct iovec iov; + size_t buf_size; int ret, vq; bool pass = true; @@ -493,14 +520,16 @@ static bool check_ptrace_values_ssve(pid_t child, struct test_config *config) vq = __sve_vq_from_vl(config->sme_vl_in); - iov.iov_len = SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, SVE_PT_REGS_SVE); - iov.iov_base = malloc(iov.iov_len); + buf_size = SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, SVE_PT_REGS_SVE); + iov.iov_len = buf_size; + iov.iov_base = malloc(buf_size); if (!iov.iov_base) { ksft_print_msg("OOM allocating %lu byte SSVE buffer\n", iov.iov_len); return false; } + memset(iov.iov_base, REGSET_SENTINEL, buf_size); ret = ptrace(PTRACE_GETREGSET, child, NT_ARM_SSVE, &iov); if (ret != 0) { ksft_print_msg("Failed to read initial SSVE: %s (%d)\n", @@ -523,10 +552,16 @@ static bool check_ptrace_values_ssve(pid_t child, struct test_config *config) } if (!(svcr_in & SVCR_SM)) { - if (sve->size != sizeof(sve)) { + if (sve->size != sizeof(*sve)) { ksft_print_msg("NT_ARM_SSVE reports data without PSTATE.SM\n"); pass = false; } + if (!buffer_is_filled(iov.iov_base + sizeof(*sve), + buf_size - sizeof(*sve), REGSET_SENTINEL)) { + ksft_print_msg("NT_ARM_SSVE wrote beyond its header without PSTATE.SM\n"); + pass = false; + } + goto out; } else { if (sve->size != SVE_PT_SIZE(vq, sve->flags)) { ksft_print_msg("Mismatch in SSVE header size: %d != %lu\n", diff --git a/tools/testing/selftests/arm64/fp/sve-test.S b/tools/testing/selftests/arm64/fp/sve-test.S index 80e072f221cd..7ef7835389e7 100644 --- a/tools/testing/selftests/arm64/fp/sve-test.S +++ b/tools/testing/selftests/arm64/fp/sve-test.S @@ -298,15 +298,20 @@ function irritator_handler add x0, x0, #1 str x0, [x2, #ucontext_regs + 8 * 23] +#ifndef SSVE // Corrupt some random Z-regs movi v0.8b, #1 movi v9.16b, #2 movi v31.8b, #3 // And P0 ptrue p0.d -#ifndef SSVE // And FFR wrffr p15.b +#else + // Enter and exit streaming mode, will reset all of the V, Z, P + // and FFR registers that the system has. + smstart_sm + smstop #endif ret diff --git a/tools/testing/selftests/arm64/mte/check_buffer_fill.c b/tools/testing/selftests/arm64/mte/check_buffer_fill.c index ff4e07503349..039b1d7d8566 100644 --- a/tools/testing/selftests/arm64/mte/check_buffer_fill.c +++ b/tools/testing/selftests/arm64/mte/check_buffer_fill.c @@ -406,6 +406,8 @@ int main(int argc, char *argv[]) size_t page_size = getpagesize(); int item = ARRAY_SIZE(sizes); + ksft_print_header(); + sizes[item - 3] = page_size - 1; sizes[item - 2] = page_size; sizes[item - 1] = page_size + 1; diff --git a/tools/testing/selftests/arm64/mte/check_child_memory.c b/tools/testing/selftests/arm64/mte/check_child_memory.c index 5e97ee792e4d..e6a8acca2a94 100644 --- a/tools/testing/selftests/arm64/mte/check_child_memory.c +++ b/tools/testing/selftests/arm64/mte/check_child_memory.c @@ -146,6 +146,8 @@ int main(int argc, char *argv[]) int err; int item = ARRAY_SIZE(sizes); + ksft_print_header(); + page_size = getpagesize(); if (!page_size) { ksft_print_msg("ERR: Unable to get page size\n"); diff --git a/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c b/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c index 325bca0de0f6..d23f154d3288 100644 --- a/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c +++ b/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c @@ -131,6 +131,7 @@ int main(int argc, char *argv[]) if (err) return err; + ksft_print_header(); ksft_set_plan(1); evaluate_test(mte_gcr_fork_test(), diff --git a/tools/testing/selftests/arm64/mte/check_hugetlb_options.c b/tools/testing/selftests/arm64/mte/check_hugetlb_options.c index aad1234c7e0f..23e4a7a9950c 100644 --- a/tools/testing/selftests/arm64/mte/check_hugetlb_options.c +++ b/tools/testing/selftests/arm64/mte/check_hugetlb_options.c @@ -230,6 +230,8 @@ int main(int argc, char *argv[]) void *map_ptr; unsigned long map_size; + ksft_print_header(); + err = mte_default_setup(); if (err) return err; diff --git a/tools/testing/selftests/arm64/mte/check_ksm_options.c b/tools/testing/selftests/arm64/mte/check_ksm_options.c index 0cf5faef1724..4855b737d550 100644 --- a/tools/testing/selftests/arm64/mte/check_ksm_options.c +++ b/tools/testing/selftests/arm64/mte/check_ksm_options.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -22,6 +23,20 @@ static size_t page_sz; static unsigned long ksm_sysfs[5]; +static bool has_merge_across_nodes; + +static bool merge_across_nodes_available(void) +{ + const char *path = PATH_KSM "merge_across_nodes"; + + if (!access(path, R_OK | W_OK)) + return true; + if (errno == ENOENT) + return false; + + ksft_exit_skip("Unable to read and write %s: %s\n", path, + strerror(errno)); +} static unsigned long read_sysfs(char *str) { @@ -56,8 +71,10 @@ static void write_sysfs(char *str, unsigned long val) static void mte_ksm_setup(void) { - ksm_sysfs[0] = read_sysfs(PATH_KSM "merge_across_nodes"); - write_sysfs(PATH_KSM "merge_across_nodes", 1); + if (has_merge_across_nodes) { + ksm_sysfs[0] = read_sysfs(PATH_KSM "merge_across_nodes"); + write_sysfs(PATH_KSM "merge_across_nodes", 1); + } ksm_sysfs[1] = read_sysfs(PATH_KSM "sleep_millisecs"); write_sysfs(PATH_KSM "sleep_millisecs", 0); ksm_sysfs[2] = read_sysfs(PATH_KSM "run"); @@ -70,7 +87,8 @@ static void mte_ksm_setup(void) static void mte_ksm_restore(void) { - write_sysfs(PATH_KSM "merge_across_nodes", ksm_sysfs[0]); + if (has_merge_across_nodes) + write_sysfs(PATH_KSM "merge_across_nodes", ksm_sysfs[0]); write_sysfs(PATH_KSM "sleep_millisecs", ksm_sysfs[1]); write_sysfs(PATH_KSM "run", ksm_sysfs[2]); write_sysfs(PATH_KSM "max_page_sharing", ksm_sysfs[3]); @@ -132,9 +150,16 @@ int main(int argc, char *argv[]) { int err; + ksft_print_header(); + err = mte_default_setup(); if (err) return err; + + if (geteuid() != 0) + ksft_exit_skip("Please run the test as root\n"); + + has_merge_across_nodes = merge_across_nodes_available(); page_sz = getpagesize(); if (!page_sz) { ksft_print_msg("ERR: Unable to get page size\n"); diff --git a/tools/testing/selftests/arm64/mte/check_mmap_options.c b/tools/testing/selftests/arm64/mte/check_mmap_options.c index c100af3012cb..492f2cd41f43 100644 --- a/tools/testing/selftests/arm64/mte/check_mmap_options.c +++ b/tools/testing/selftests/arm64/mte/check_mmap_options.c @@ -945,6 +945,8 @@ int main(int argc, char *argv[]) }, }; + ksft_print_header(); + err = mte_default_setup(); if (err) return err; diff --git a/tools/testing/selftests/arm64/mte/check_prctl.c b/tools/testing/selftests/arm64/mte/check_prctl.c index f7f320defa7b..d16a91117eef 100644 --- a/tools/testing/selftests/arm64/mte/check_prctl.c +++ b/tools/testing/selftests/arm64/mte/check_prctl.c @@ -119,7 +119,7 @@ int main(void) int i; ksft_print_header(); - ksft_set_plan(ARRAY_SIZE(mte_modes)); + ksft_set_plan(ARRAY_SIZE(mte_modes) + 1); check_basic_read(); for (i = 0; i < ARRAY_SIZE(mte_modes); i++) diff --git a/tools/testing/selftests/arm64/mte/check_tags_inclusion.c b/tools/testing/selftests/arm64/mte/check_tags_inclusion.c index 4b764f2a8185..6b4fa6705d7c 100644 --- a/tools/testing/selftests/arm64/mte/check_tags_inclusion.c +++ b/tools/testing/selftests/arm64/mte/check_tags_inclusion.c @@ -175,6 +175,8 @@ int main(int argc, char *argv[]) { int err; + ksft_print_header(); + err = mte_default_setup(); if (err) return err; diff --git a/tools/testing/selftests/arm64/mte/check_user_mem.c b/tools/testing/selftests/arm64/mte/check_user_mem.c index fb7936c4e097..af343aa61732 100644 --- a/tools/testing/selftests/arm64/mte/check_user_mem.c +++ b/tools/testing/selftests/arm64/mte/check_user_mem.c @@ -201,6 +201,8 @@ int main(int argc, char *argv[]) int tag_offsets[] = {page_sz, MT_GRANULE_SIZE}; char test_name[TEST_NAME_MAX]; + ksft_print_header(); + page_sz = getpagesize(); if (!page_sz) { ksft_print_msg("ERR: Unable to get page size\n"); diff --git a/tools/testing/selftests/bpf/README.rst b/tools/testing/selftests/bpf/README.rst index 776fbe3cb8f9..a36afb460e26 100644 --- a/tools/testing/selftests/bpf/README.rst +++ b/tools/testing/selftests/bpf/README.rst @@ -107,12 +107,12 @@ Docker container and local rootfs image. The overall steps are as follows: tools/testing/selftests/bpf/vmtest.sh \ -l -- \ ./test_progs -d \ - \"$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \ + "$(cat tools/testing/selftests/bpf/DENYLIST.riscv64 \ | cut -d'#' -f1 \ | sed -e 's/^[[:space:]]*//' \ -e 's/[[:space:]]*$//' \ | tr -s '\n' ',' \ - )\" + )" Link: https://github.com/pulehui/riscv-bpf-vmtest.git [0] Link: https://github.com/libbpf/ci/blob/main/rootfs/mkrootfs_debian.sh [1] diff --git a/tools/testing/selftests/bpf/network_helpers.c b/tools/testing/selftests/bpf/network_helpers.c index cdf7b6641444..2067e12034ba 100644 --- a/tools/testing/selftests/bpf/network_helpers.c +++ b/tools/testing/selftests/bpf/network_helpers.c @@ -97,7 +97,7 @@ int settimeo(int fd, int timeout_ms) int start_server_addr(int type, const struct sockaddr_storage *addr, socklen_t addrlen, const struct network_helper_opts *opts) { - int fd; + int on = 1, fd; if (!opts) opts = &default_opts; @@ -111,6 +111,12 @@ int start_server_addr(int type, const struct sockaddr_storage *addr, socklen_t a if (settimeo(fd, opts->timeout_ms)) goto error_close; + if ((type & SOCK_TYPE_MASK) == SOCK_STREAM && + setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on))) { + log_err("Failed to enable SO_REUSEADDR"); + goto error_close; + } + if (opts->post_socket_cb && opts->post_socket_cb(fd, opts->cb_opts)) { log_err("Failed to call post_socket_cb"); @@ -122,7 +128,7 @@ int start_server_addr(int type, const struct sockaddr_storage *addr, socklen_t a goto error_close; } - if (type == SOCK_STREAM) { + if ((type & SOCK_TYPE_MASK) == SOCK_STREAM) { if (listen(fd, opts->backlog ? MAX(opts->backlog, 0) : 1) < 0) { log_err("Failed to listed on socket"); goto error_close; diff --git a/tools/testing/selftests/bpf/network_helpers.h b/tools/testing/selftests/bpf/network_helpers.h index ef208eefd571..5ea989bc4009 100644 --- a/tools/testing/selftests/bpf/network_helpers.h +++ b/tools/testing/selftests/bpf/network_helpers.h @@ -25,6 +25,11 @@ typedef __u16 __sum16; #define VIP_NUM 5 #define MAGIC_BYTES 123 +/* include/linux/net.h */ +#ifndef SOCK_TYPE_MASK +#define SOCK_TYPE_MASK 0xf +#endif + struct network_helper_opts { int timeout_ms; int proto; diff --git a/tools/testing/selftests/bpf/prog_tests/global_map_resize.c b/tools/testing/selftests/bpf/prog_tests/global_map_resize.c index 56b5baef35c8..602ce30f1720 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_map_resize.c +++ b/tools/testing/selftests/bpf/prog_tests/global_map_resize.c @@ -23,6 +23,7 @@ static void global_map_resize_bss_subtest(void) struct bpf_map *map; const __u32 desired_sz = sizeof(skel->bss->sum) + sysconf(_SC_PAGE_SIZE) * 2; size_t array_len, actual_sz, new_sz; + int *array; skel = test_global_map_resize__open(); if (!ASSERT_OK_PTR(skel, "test_global_map_resize__open")) @@ -58,10 +59,13 @@ static void global_map_resize_bss_subtest(void) goto teardown; /* fill the newly resized array with ones, - * skipping the first element which was previously set + * skipping the first element which was previously set; + * access through a plain pointer to avoid -Warray-bounds + * since the array was resized beyond its declared length. */ + array = skel->bss->array; for (int i = 1; i < array_len; i++) - skel->bss->array[i] = 1; + array[i] = 1; /* set global const values before loading */ skel->rodata->pid = getpid(); diff --git a/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c b/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c index b6391af5f6f9..39e8a3b8b6af 100644 --- a/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c +++ b/tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c @@ -3,6 +3,7 @@ #include "network_helpers.h" #include "test_progs.h" +#include "test_lwt_ip_encap.skel.h" #define BPF_FILE "test_lwt_ip_encap.bpf.o" @@ -32,6 +33,9 @@ #define IP6_ADDR_8 "fb08::1" #define IP6_ADDR_GRE "fb10::1" +#define IP4_ADDR_VXLAN "172.16.17.100" +#define IP6_ADDR_VXLAN "fb11::1" + #define IP6_ADDR_SRC IP6_ADDR_1 #define IP6_ADDR_DST IP6_ADDR_4 @@ -406,7 +410,8 @@ static int test_gso_fix(const char *ns1, const char *ns3, int family) static int check_ping_ok(const char *ns1) { SYS(fail, "ip netns exec %s ping -c 1 -W1 -I veth1 %s > /dev/null", ns1, IP4_ADDR_DST); - SYS(fail, "ip netns exec %s ping6 -c 1 -W1 -I veth1 %s > /dev/null", ns1, IP6_ADDR_DST); + SYS(fail, "ip netns exec %s %s -c 1 -W1 -I veth1 %s > /dev/null", ns1, + ping_command(AF_INET6), IP6_ADDR_DST); return 0; fail: return -1; @@ -420,7 +425,8 @@ static int check_ping_fails(const char *ns1) if (!ret) return -1; - ret = SYS_NOFAIL("ip netns exec %s ping6 -c 1 -W1 -I veth1 %s", ns1, IP6_ADDR_DST); + ret = SYS_NOFAIL("ip netns exec %s %s -c 1 -W1 -I veth1 %s", ns1, + ping_command(AF_INET6), IP6_ADDR_DST); if (!ret) return -1; @@ -538,3 +544,145 @@ void test_lwt_ip_encap_ipv4(void) if (test__start_subtest("ingress")) lwt_ip_encap(IPV4_ENCAP, INGRESS, ""); } + +/* + * VxLAN Setup/topology: + * + * NS1 (IP*_ADDR_1) NS2 NS3 (IP*_ADDR_4) + * [ping src] + * | top route + * veth1 (LWT encap) <<-- veth2 veth3 <<-- veth4 (ping dst) + * | ^ + * (bottom route) | (inner pkt) + * v bottom route | + * veth5 -->> veth6 veth7 -->> veth8 (vxlan decap) + * (IP*_ADDR_VXLAN) + * + * Add the VxLAN endpoint addresses to NS3's veth8, create standard + * VxLAN decap devices bound to those addresses, and install routes so + * NS1/NS2 can reach the endpoints via the bottom route. NS2 here is to + * make sure the LWT-encap VxLAN packets are routed to NS3 correctly. + */ +static int setup_vxlan_routes(const char *ns3, const char *ns1, const char *ns2) +{ + struct nstoken *nstoken; + + nstoken = open_netns(ns3); + if (!ASSERT_OK_PTR(nstoken, "open ns3 for vxlan")) + return -1; + + SYS(fail_close, "ip a add %s/32 dev veth8", IP4_ADDR_VXLAN); + SYS(fail_close, "ip -6 a add %s/128 dev veth8", IP6_ADDR_VXLAN); + /* + * Standard VxLAN devices to decap the encapsulated packets. The inner + * Ethernet frame uses a broadcast dst MAC so the IP stack accepts it + * without ARP or FDB configuration. + */ + SYS(fail_close, "ip link add vxlan4 type vxlan id 1 dstport 4789 local %s dev veth8 nolearning noudpcsum", + IP4_ADDR_VXLAN); + SYS(fail_close, "ip link set vxlan4 up"); + SYS(fail_close, "ip link add vxlan6 type vxlan id 1 dstport 4789 local %s dev veth8 nolearning udp6zerocsumrx", + IP6_ADDR_VXLAN); + SYS(fail_close, "ip link set vxlan6 up"); + close_netns(nstoken); + + SYS(fail, "ip -n %s route add %s/32 dev veth5 via %s", + ns1, IP4_ADDR_VXLAN, IP4_ADDR_6); + SYS(fail, "ip -n %s route add %s/32 dev veth7 via %s", + ns2, IP4_ADDR_VXLAN, IP4_ADDR_8); + SYS(fail, "ip -n %s -6 route add %s/128 dev veth5 via %s", + ns1, IP6_ADDR_VXLAN, IP6_ADDR_6); + SYS(fail, "ip -n %s -6 route add %s/128 dev veth7 via %s", + ns2, IP6_ADDR_VXLAN, IP6_ADDR_8); + return 0; + +fail_close: + close_netns(nstoken); +fail: + return -1; +} + +static void lwt_ip_encap_vxlan(bool ipv4_encap) +{ + char ns1[NETNS_NAME_SIZE] = NETNS_BASE "-1-"; + char ns2[NETNS_NAME_SIZE] = NETNS_BASE "-2-"; + char ns3[NETNS_NAME_SIZE] = NETNS_BASE "-3-"; + const char *sec = ipv4_encap ? "encap_vxlan" : "encap_vxlan6"; + int expected_offset = ipv4_encap ? (int)sizeof(struct iphdr) + : (int)sizeof(struct ipv6hdr); + struct test_lwt_ip_encap *skel = NULL; + int thdr_offset, err; + + if (!ASSERT_OK(create_ns(ns1, NETNS_NAME_SIZE), "create ns1")) + goto out; + if (!ASSERT_OK(create_ns(ns2, NETNS_NAME_SIZE), "create ns2")) + goto out; + if (!ASSERT_OK(create_ns(ns3, NETNS_NAME_SIZE), "create ns3")) + goto out; + + if (!ASSERT_OK(setup_network(ns1, ns2, ns3, ""), "setup network")) + goto out; + + if (!ASSERT_OK(setup_vxlan_routes(ns3, ns1, ns2), "setup vxlan routes")) + goto out; + + skel = test_lwt_ip_encap__open(); + if (!ASSERT_OK_PTR(skel, "test_lwt_ip_encap__open")) + goto out; + + bpf_program__set_autoload(skel->progs.bpf_lwt_encap_gre, false); + bpf_program__set_autoload(skel->progs.bpf_lwt_encap_gre6, false); + bpf_program__set_autoload(skel->progs.bpf_lwt_encap_vxlan, false); + bpf_program__set_autoload(skel->progs.bpf_lwt_encap_vxlan6, false); + bpf_program__set_autoload(skel->progs.fexit_lwt_push_ip_encap, true); + skel->rodata->tgt_ip_version = ipv4_encap ? 4 : 6; + + err = test_lwt_ip_encap__load(skel); + if (!ASSERT_OK(err, "test_lwt_ip_encap__load")) + goto out; + + err = test_lwt_ip_encap__attach(skel); + if (!ASSERT_OK(err, "test_lwt_ip_encap__attach")) + goto out; + + /* Remove the direct NS2->DST route so packets must go via LWT encap. */ + SYS(out, "ip -n %s route del %s/32 dev veth3", ns2, IP4_ADDR_DST); + SYS(out, "ip -n %s -6 route del %s/128 dev veth3", ns2, IP6_ADDR_DST); + + if (ipv4_encap) + SYS(out, "ip -n %s route add %s encap bpf xmit obj %s sec %s dev veth1", + ns1, IP4_ADDR_DST, BPF_FILE, sec); + else + SYS(out, "ip -n %s -6 route add %s encap bpf xmit obj %s sec %s dev veth1", + ns1, IP6_ADDR_DST, BPF_FILE, sec); + + skel->bss->fexit_triggered = false; + + if (ipv4_encap) + SYS(out, "ip netns exec %s ping -c 1 -W1 %s", ns1, IP4_ADDR_DST); + else + SYS(out, "ip netns exec %s %s -c 1 -W1 %s", ns1, + ping_command(AF_INET6), IP6_ADDR_DST); + + if (!ASSERT_TRUE(skel->bss->fexit_triggered, "fexit_triggered")) + goto out; + + thdr_offset = (int)skel->bss->transport_hdr - (int)skel->bss->network_hdr; + ASSERT_EQ(thdr_offset, expected_offset, "transport_hdr offset"); + +out: + test_lwt_ip_encap__destroy(skel); + SYS_NOFAIL("ip netns del %s", ns1); + SYS_NOFAIL("ip netns del %s", ns2); + SYS_NOFAIL("ip netns del %s", ns3); +} + +void test_lwt_ip_encap_vxlan_ipv4(void) +{ + lwt_ip_encap_vxlan(IPV4_ENCAP); +} + +void test_lwt_ip_encap_vxlan_ipv6(void) +{ + lwt_ip_encap_vxlan(IPV6_ENCAP); +} diff --git a/tools/testing/selftests/bpf/prog_tests/mptcp.c b/tools/testing/selftests/bpf/prog_tests/mptcp.c index 8fade8bdc451..32dfc1c511af 100644 --- a/tools/testing/selftests/bpf/prog_tests/mptcp.c +++ b/tools/testing/selftests/bpf/prog_tests/mptcp.c @@ -264,7 +264,7 @@ static int verify_mptcpify(int server_fd, int client_fd) return err; } -static int run_mptcpify(int cgroup_fd) +static int run_mptcpify(int cgroup_fd, int type) { int server_fd, client_fd, err = 0; struct mptcpify *mptcpify_skel; @@ -280,7 +280,7 @@ static int run_mptcpify(int cgroup_fd) goto out; /* without MPTCP */ - server_fd = start_server(AF_INET, SOCK_STREAM, NULL, 0, 0); + server_fd = start_server(AF_INET, type, NULL, 0, 0); if (!ASSERT_GE(server_fd, 0, "start_server")) { err = -EIO; goto out; @@ -317,7 +317,14 @@ static void test_mptcpify(void) if (!ASSERT_OK_PTR(netns, "netns_new")) goto fail; - ASSERT_OK(run_mptcpify(cgroup_fd), "run_mptcpify"); + ASSERT_OK(run_mptcpify(cgroup_fd, SOCK_STREAM), "run_mptcpify"); + /* userspace sets flags such as SOCK_CLOEXEC together with the type; + * the BPF prog must still upgrade the socket to MPTCP. See + * update_socket_protocol() in net/socket.c, which runs before the + * type is masked with SOCK_TYPE_MASK. + */ + ASSERT_OK(run_mptcpify(cgroup_fd, SOCK_STREAM | SOCK_CLOEXEC), + "run_mptcpify_cloexec"); fail: netns_free(netns); diff --git a/tools/testing/selftests/bpf/prog_tests/sha256.c b/tools/testing/selftests/bpf/prog_tests/sha256.c index 604a0b1423d5..5edbc6194b07 100644 --- a/tools/testing/selftests/bpf/prog_tests/sha256.c +++ b/tools/testing/selftests/bpf/prog_tests/sha256.c @@ -25,10 +25,10 @@ void test_sha256(void) size_t i; data = malloc(MAX_LEN); - if (!ASSERT_OK_PTR(data, "malloc")) + if (!ASSERT_NEQ(data, NULL, "malloc")) goto out; digests = malloc((MAX_LEN + 1) * SHA256_DIGEST_LENGTH); - if (!ASSERT_OK_PTR(digests, "malloc")) + if (!ASSERT_NEQ(digests, NULL, "malloc")) goto out; /* Generate MAX_LEN bytes of "random" data deterministically. */ diff --git a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h index 17db400f0e0d..a972e174f741 100644 --- a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h +++ b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h @@ -8,6 +8,9 @@ #define AF_INET 2 #define AF_INET6 10 +/* include/linux/net.h */ +#define SOCK_TYPE_MASK 0xf + #define SOL_SOCKET 1 #define SO_REUSEADDR 2 #define SO_SNDBUF 7 diff --git a/tools/testing/selftests/bpf/progs/mptcpify.c b/tools/testing/selftests/bpf/progs/mptcpify.c index cbdc730c3a47..e3f8cb54dbe9 100644 --- a/tools/testing/selftests/bpf/progs/mptcpify.c +++ b/tools/testing/selftests/bpf/progs/mptcpify.c @@ -15,7 +15,7 @@ int BPF_PROG(mptcpify, int family, int type, int protocol) return protocol; if ((family == AF_INET || family == AF_INET6) && - type == SOCK_STREAM && + (type & SOCK_TYPE_MASK) == SOCK_STREAM && (!protocol || protocol == IPPROTO_TCP)) { return IPPROTO_MPTCP; } diff --git a/tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c b/tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c index d6cb986e7533..4a934fccf8f5 100644 --- a/tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c +++ b/tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c @@ -1,11 +1,9 @@ // SPDX-License-Identifier: GPL-2.0 -#include +#include "vmlinux.h" #include -#include -#include -#include #include #include +#include struct grehdr { __be16 flags; @@ -64,13 +62,13 @@ int bpf_lwt_encap_gre6(struct __sk_buff *skb) hdr.ip6hdr.nexthdr = 47; /* IPPROTO_GRE */ hdr.ip6hdr.hop_limit = 0x40; /* fb01::1 */ - hdr.ip6hdr.saddr.s6_addr[0] = 0xfb; - hdr.ip6hdr.saddr.s6_addr[1] = 1; - hdr.ip6hdr.saddr.s6_addr[15] = 1; + hdr.ip6hdr.saddr.in6_u.u6_addr8[0] = 0xfb; + hdr.ip6hdr.saddr.in6_u.u6_addr8[1] = 1; + hdr.ip6hdr.saddr.in6_u.u6_addr8[15] = 1; /* fb10::1 */ - hdr.ip6hdr.daddr.s6_addr[0] = 0xfb; - hdr.ip6hdr.daddr.s6_addr[1] = 0x10; - hdr.ip6hdr.daddr.s6_addr[15] = 1; + hdr.ip6hdr.daddr.in6_u.u6_addr8[0] = 0xfb; + hdr.ip6hdr.daddr.in6_u.u6_addr8[1] = 0x10; + hdr.ip6hdr.daddr.in6_u.u6_addr8[15] = 1; hdr.greh.protocol = skb->protocol; @@ -82,4 +80,141 @@ int bpf_lwt_encap_gre6(struct __sk_buff *skb) return BPF_LWT_REROUTE; } +#define VXLAN_PORT 4789 +#define VXLAN_FLAGS 0x08000000 +#define VXLAN_VNI 1 + +#define ETH_ALEN 6 /* Octets in one ethernet addr */ +#define ETH_P_IP 0x0800 /* Internet Protocol packet */ +#define ETH_P_IPV6 0x86DD /* IPv6 over bluebook */ + +static const __u8 bcast[ETH_ALEN] = { + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, +}; + +static const __u8 srcmac[ETH_ALEN] = { + 0x02, 0x00, 0x00, 0x00, 0x00, 0x01, +}; + +SEC("encap_vxlan") +int bpf_lwt_encap_vxlan(struct __sk_buff *skb) +{ + struct encap_hdr { + struct iphdr iph; + struct udphdr udph; + struct vxlanhdr vxh; + struct ethhdr eth; + } __attribute__((__packed__)) hdr; + int err; + + memset(&hdr, 0, sizeof(hdr)); + + hdr.iph.ihl = 5; + hdr.iph.version = 4; + hdr.iph.ttl = 0x40; + hdr.iph.protocol = 17; /* IPPROTO_UDP */ + hdr.iph.tot_len = bpf_htons(skb->len + sizeof(hdr)); +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ + hdr.iph.saddr = 0x640510ac; /* 172.16.5.100 */ + hdr.iph.daddr = 0x641110ac; /* 172.16.17.100 */ +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ + hdr.iph.saddr = 0xac100564; /* 172.16.5.100 */ + hdr.iph.daddr = 0xac101164; /* 172.16.17.100 */ +#else +#error "Fix your compiler's __BYTE_ORDER__?!" +#endif + + hdr.udph.source = bpf_htons(VXLAN_PORT); + hdr.udph.dest = bpf_htons(VXLAN_PORT); + hdr.udph.len = bpf_htons(skb->len + sizeof(hdr.udph) + sizeof(hdr.vxh) + + sizeof(hdr.eth)); + + hdr.vxh.vx_flags = bpf_htonl(VXLAN_FLAGS); + hdr.vxh.vx_vni = bpf_htonl(VXLAN_VNI << 8); + + __builtin_memcpy(hdr.eth.h_dest, bcast, ETH_ALEN); + __builtin_memcpy(hdr.eth.h_source, srcmac, ETH_ALEN); + hdr.eth.h_proto = bpf_htons(ETH_P_IP); + + err = bpf_lwt_push_encap(skb, BPF_LWT_ENCAP_IP, &hdr, sizeof(hdr)); + if (err) + return BPF_DROP; + + return BPF_LWT_REROUTE; +} + +SEC("encap_vxlan6") +int bpf_lwt_encap_vxlan6(struct __sk_buff *skb) +{ + struct encap_hdr { + struct ipv6hdr ip6hdr; + struct udphdr udph; + struct vxlanhdr vxh; + struct ethhdr eth; + } __attribute__((__packed__)) hdr; + int err; + + memset(&hdr, 0, sizeof(hdr)); + + hdr.ip6hdr.version = 6; + hdr.ip6hdr.nexthdr = 17; /* IPPROTO_UDP */ + hdr.ip6hdr.hop_limit = 0x40; + hdr.ip6hdr.payload_len = bpf_htons(skb->len + sizeof(hdr.udph) + sizeof(hdr.vxh) + + sizeof(hdr.eth)); + /* fb05::1 */ + hdr.ip6hdr.saddr.in6_u.u6_addr8[0] = 0xfb; + hdr.ip6hdr.saddr.in6_u.u6_addr8[1] = 0x05; + hdr.ip6hdr.saddr.in6_u.u6_addr8[15] = 1; + /* fb11::1 */ + hdr.ip6hdr.daddr.in6_u.u6_addr8[0] = 0xfb; + hdr.ip6hdr.daddr.in6_u.u6_addr8[1] = 0x11; + hdr.ip6hdr.daddr.in6_u.u6_addr8[15] = 1; + + hdr.udph.source = bpf_htons(VXLAN_PORT); + hdr.udph.dest = bpf_htons(VXLAN_PORT); + hdr.udph.len = bpf_htons(skb->len + sizeof(hdr.udph) + sizeof(hdr.vxh) + + sizeof(hdr.eth)); + + hdr.vxh.vx_flags = bpf_htonl(VXLAN_FLAGS); + hdr.vxh.vx_vni = bpf_htonl(VXLAN_VNI << 8); + + __builtin_memcpy(hdr.eth.h_dest, bcast, ETH_ALEN); + __builtin_memcpy(hdr.eth.h_source, srcmac, ETH_ALEN); + hdr.eth.h_proto = bpf_htons(ETH_P_IPV6); + + err = bpf_lwt_push_encap(skb, BPF_LWT_ENCAP_IP, &hdr, sizeof(hdr)); + if (err) + return BPF_DROP; + + return BPF_LWT_REROUTE; +} + +volatile const int tgt_ip_version; + +__u16 transport_hdr = 0; +__u16 network_hdr = 0; +bool fexit_triggered = false; + +SEC("?fexit/bpf_lwt_push_ip_encap") +int BPF_PROG(fexit_lwt_push_ip_encap, struct sk_buff *skb, void *hdr, u32 len, bool ingress, + int retval) +{ + struct iphdr *iph; + + if (retval || fexit_triggered) + return 0; + + iph = (typeof(iph)) (skb->head + skb->network_header); + if (iph->version != tgt_ip_version) + return 0; + + if ((iph->version == 4 && iph->protocol == 17 /* IPPROTO_UDP */) || + (iph->version == 6 && ((struct ipv6hdr *)iph)->nexthdr == 17 /* IPPROTO_UDP */)) { + fexit_triggered = true; + transport_hdr = skb->transport_header; + network_hdr = skb->network_header; + } + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index 02a85dda30e6..eda3fcf72095 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -550,18 +550,19 @@ bool test__start_subtest(const char *subtest_name) struct test_state *state = env.test_state; struct subtest_state *subtest_state; size_t sub_state_size = sizeof(*subtest_state); + void *tmp; if (env.subtest_state) test__end_subtest(); state->subtest_num++; - state->subtest_states = - realloc(state->subtest_states, - state->subtest_num * sub_state_size); - if (!state->subtest_states) { + tmp = realloc(state->subtest_states, state->subtest_num * sub_state_size); + if (!tmp) { + state->subtest_num--; fprintf(stderr, "Not enough memory to allocate subtest result\n"); return false; } + state->subtest_states = tmp; subtest_state = &state->subtest_states[state->subtest_num - 1]; @@ -1690,7 +1691,7 @@ static void server_main(void) data[i].worker_id = i; data[i].sock_fd = env.worker_socks[i]; rc = pthread_create(&dispatcher_threads[i], NULL, dispatch_thread, &data[i]); - if (rc < 0) { + if (rc) { perror("Failed to launch dispatcher thread"); exit(EXIT_ERR_SETUP_INFRA); } diff --git a/tools/testing/selftests/bpf/test_sockmap.c b/tools/testing/selftests/bpf/test_sockmap.c index 76568db7a664..f8b57f3c3f23 100644 --- a/tools/testing/selftests/bpf/test_sockmap.c +++ b/tools/testing/selftests/bpf/test_sockmap.c @@ -514,7 +514,8 @@ static int msg_alloc_iov(struct msghdr *msg, return 0; unwind_iov: for (i--; i >= 0 ; i--) - free(msg->msg_iov[i].iov_base); + free(iov[i].iov_base); + free(iov); return -ENOMEM; } diff --git a/tools/testing/selftests/bpf/veristat.c b/tools/testing/selftests/bpf/veristat.c index 1be1e353d40a..df4aea8bbe82 100644 --- a/tools/testing/selftests/bpf/veristat.c +++ b/tools/testing/selftests/bpf/veristat.c @@ -513,6 +513,40 @@ static bool is_bpf_obj_file(const char *path) { return err == 0; } +/* Exact filter match */ +static bool name_filter_matches(struct filter *f, const char *filename, const char *prog_name) +{ + if (f->any_glob) + return glob_matches(filename, f->any_glob) || + (prog_name && glob_matches(prog_name, f->any_glob)); + if (f->file_glob && f->prog_glob) + return prog_name && + glob_matches(filename, f->file_glob) && + glob_matches(prog_name, f->prog_glob); + if (f->file_glob) + return glob_matches(filename, f->file_glob); + if (f->prog_glob) + return prog_name && glob_matches(prog_name, f->prog_glob); + return false; +} + +/* Check if the filter does not outright reject the file name */ +static bool name_filter_may_match(struct filter *f, const char *filename) +{ + if (f->file_glob) + return glob_matches(filename, f->file_glob); + /* + * If we don't know program name yet, any_glob filter + * has to assume that current BPF object file might be + * relevant; we'll check again later on after opening + * BPF object file, at which point program name will + * be known finally. + */ + if (f->any_glob || f->prog_glob) + return true; + return false; +} + static bool should_process_file_prog(const char *filename, const char *prog_name) { struct filter *f; @@ -520,16 +554,7 @@ static bool should_process_file_prog(const char *filename, const char *prog_name for (i = 0; i < env.deny_filter_cnt; i++) { f = &env.deny_filters[i]; - if (f->kind != FILTER_NAME) - continue; - - if (f->any_glob && glob_matches(filename, f->any_glob)) - return false; - if (f->any_glob && prog_name && glob_matches(prog_name, f->any_glob)) - return false; - if (f->file_glob && glob_matches(filename, f->file_glob)) - return false; - if (f->prog_glob && prog_name && glob_matches(prog_name, f->prog_glob)) + if (f->kind == FILTER_NAME && name_filter_matches(f, filename, prog_name)) return false; } @@ -539,24 +564,15 @@ static bool should_process_file_prog(const char *filename, const char *prog_name continue; allow_cnt++; - if (f->any_glob) { - if (glob_matches(filename, f->any_glob)) - return true; - /* If we don't know program name yet, any_glob filter - * has to assume that current BPF object file might be - * relevant; we'll check again later on after opening - * BPF object file, at which point program name will - * be known finally. - */ - if (!prog_name || glob_matches(prog_name, f->any_glob)) - return true; - } else { - if (f->file_glob && !glob_matches(filename, f->file_glob)) - continue; - if (f->prog_glob && prog_name && !glob_matches(prog_name, f->prog_glob)) - continue; + if (prog_name && name_filter_matches(f, filename, prog_name)) + return true; + /* + * If there is no prog_name and the file name is not blocked by + * the filter, allow to open the file. Afterwards there would be + * a second refining query with prog_name set. + */ + if (!prog_name && name_filter_may_match(f, filename)) return true; - } } /* if there are no file/prog name allow filters, allow all progs, @@ -702,6 +718,12 @@ static int append_filter(struct filter **filters, int *cnt, const char *str) } } + if ((!f->any_glob && !f->file_glob && !f->prog_glob) || + (f->any_glob && strcmp(f->any_glob, "") == 0)) { + fprintf(stderr, "Invalid filter: '%s'\n", str); + return -EINVAL; + } + *cnt += 1; return 0; } diff --git a/tools/testing/selftests/bpf/vmtest.sh b/tools/testing/selftests/bpf/vmtest.sh index 9ca802285393..6a3d026d76bd 100755 --- a/tools/testing/selftests/bpf/vmtest.sh +++ b/tools/testing/selftests/bpf/vmtest.sh @@ -428,8 +428,17 @@ main() if [[ $# -eq 0 && "${debug_shell}" == "no" ]]; then echo "No command specified, will run ${DEFAULT_COMMAND} in the vm" - else - command="$@" + elif [[ $# -gt 0 ]]; then + # Quote each argument so the command survives into the guest: the + # host expands ${command} into the generated init script, which + # the guest bash then parses as shell source. Without the %q + # escapes an argument with a space or a glob (e.g. -a 'verifier_*') + # is re-split and expanded against /root/bpf there. + # + # Skip this when there is no command: printf '%q ' would still + # apply the format once and emit '', which is not the empty + # command that -s (debug shell) expects. + command=$(printf '%q ' "$@") fi local kconfig_file="${OUTPUT_DIR}/latest.config" diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh index a17256d9f88a..592e533cd4d7 100755 --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh @@ -488,6 +488,7 @@ write_cpu_online() CPU=${1%=*} VAL=${1#*=} CPUFILE=//sys/devices/system/cpu/cpu${CPU}/online + echo $VAL > $CPUFILE || return 1 if [[ $VAL -eq 0 ]] then OFFLINE_CPUS="$OFFLINE_CPUS $CPU" @@ -497,7 +498,6 @@ write_cpu_online() sort | uniq -u) } fi - echo $VAL > $CPUFILE pause 0.05 } @@ -565,7 +565,8 @@ set_ctrl_state() eval $COMM $REDIRECT ;; O*) VAL=${CMD#?} - write_cpu_online $VAL + COMM="write_cpu_online $VAL" + eval $COMM $REDIRECT ;; T*) COMM="echo 0 > $TFILE" eval $COMM $REDIRECT diff --git a/tools/testing/selftests/kselftest_harness.h b/tools/testing/selftests/kselftest_harness.h index 6928915a643b..be7f0ecab770 100644 --- a/tools/testing/selftests/kselftest_harness.h +++ b/tools/testing/selftests/kselftest_harness.h @@ -76,7 +76,7 @@ static inline void __kselftest_memset_safe(void *s, int c, size_t n) memset(s, c, n); } -#define KSELFTEST_PRIO_TEST_F 20000 +#define KSELFTEST_PRIO_TEST 20000 #define KSELFTEST_PRIO_XFAIL 20001 #define TEST_TIMEOUT_DEFAULT 30 @@ -194,7 +194,7 @@ static inline void __kselftest_memset_safe(void *s, int c, size_t n) .fixture = &_fixture_global, \ .termsig = _signal, \ .timeout = TEST_TIMEOUT_DEFAULT, }; \ - static void __attribute__((constructor)) _register_##test_name(void) \ + static void __attribute__((constructor(KSELFTEST_PRIO_TEST))) _register_##test_name(void) \ { \ __register_test(&_##test_name##_object); \ } \ @@ -238,7 +238,7 @@ static inline void __kselftest_memset_safe(void *s, int c, size_t n) FIXTURE_VARIANT(fixture_name); \ static struct __fixture_metadata _##fixture_name##_fixture_object = \ { .name = #fixture_name, }; \ - static void __attribute__((constructor)) \ + static void __attribute__((constructor(KSELFTEST_PRIO_TEST))) \ _register_##fixture_name##_data(void) \ { \ __register_fixture(&_##fixture_name##_fixture_object); \ @@ -364,7 +364,7 @@ static inline void __kselftest_memset_safe(void *s, int c, size_t n) _##fixture_name##_##variant_name##_object = \ { .name = #variant_name, \ .data = &_##fixture_name##_##variant_name##_variant}; \ - static void __attribute__((constructor)) \ + static void __attribute__((constructor(KSELFTEST_PRIO_TEST))) \ _register_##fixture_name##_##variant_name(void) \ { \ __register_fixture_variant(&_##fixture_name##_fixture_object, \ @@ -467,8 +467,8 @@ static inline void __kselftest_memset_safe(void *s, int c, size_t n) !__atomic_test_and_set(_metadata->no_teardown, __ATOMIC_RELAXED)) \ fixture_name##_teardown(_metadata, self, variant); \ } \ - static struct __test_metadata *_##fixture_name##_##test_name##_object; \ - static void __attribute__((constructor(KSELFTEST_PRIO_TEST_F))) \ + static struct __test_metadata *_##fixture_name##_##test_name##_object __maybe_unused; \ + static void __attribute__((constructor(KSELFTEST_PRIO_TEST))) \ _register_##fixture_name##_##test_name(void) \ { \ struct __test_metadata *object = mmap(NULL, sizeof(*object), \ @@ -1315,7 +1315,7 @@ static int test_harness_run(int argc, char **argv) return KSFT_FAIL; } -static void __attribute__((constructor)) __constructor_order_first(void) +static void __attribute__((constructor(KSELFTEST_PRIO_TEST))) __constructor_order_first(void) { __constructor_order_forward = true; } diff --git a/tools/testing/selftests/lsm/common.c b/tools/testing/selftests/lsm/common.c index 9ad258912646..927dce4f04cb 100644 --- a/tools/testing/selftests/lsm/common.c +++ b/tools/testing/selftests/lsm/common.c @@ -76,7 +76,7 @@ int attr_lsm_count(void) return 0; if (read_sysfs_lsms(names, sysconf(_SC_PAGESIZE))) - return 0; + goto out; if (strstr(names, "selinux")) count++; @@ -85,5 +85,7 @@ int attr_lsm_count(void) if (strstr(names, "apparmor")) count++; +out: + free(names); return count; } diff --git a/tools/testing/selftests/mm/cow.c b/tools/testing/selftests/mm/cow.c index 0df61422467d..af8db738423d 100644 --- a/tools/testing/selftests/mm/cow.c +++ b/tools/testing/selftests/mm/cow.c @@ -1699,8 +1699,13 @@ static void run_with_tmpfile(non_anon_test_fn fn, const char *desc) /* File consists of a single page filled with zeroes. */ if (fallocate(fd, 0, 0, pagesize)) { - ksft_perror("fallocate() failed"); - log_test_result(KSFT_FAIL); + if (errno == EOPNOTSUPP) { + ksft_print_msg("fallocate() not supported by filesystem\n"); + log_test_result(KSFT_SKIP); + } else { + ksft_perror("fallocate() failed"); + log_test_result(KSFT_FAIL); + } goto close; } diff --git a/tools/testing/selftests/mm/ksm_tests.c b/tools/testing/selftests/mm/ksm_tests.c index b77462b5c240..e763295b5b2a 100644 --- a/tools/testing/selftests/mm/ksm_tests.c +++ b/tools/testing/selftests/mm/ksm_tests.c @@ -175,12 +175,12 @@ static void *allocate_memory(void *ptr, int prot, int mapping, char data, size_ void *map_ptr = mmap(ptr, map_size, PROT_WRITE, mapping, -1, 0); if (!map_ptr) { - perror("mmap"); + ksft_perror("mmap"); return NULL; } memset(map_ptr, data, map_size); if (mprotect(map_ptr, map_size, prot)) { - perror("mprotect"); + ksft_perror("mprotect"); munmap(map_ptr, map_size); return NULL; } @@ -201,11 +201,11 @@ static int ksm_do_scan(int scan_count, struct timespec start_time, int timeout) if (ksm_read_sysfs(KSM_FP("full_scans"), &cur_scan)) return 1; if (clock_gettime(CLOCK_MONOTONIC_RAW, &cur_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); return 1; } if ((cur_time.tv_sec - start_time.tv_sec) > timeout) { - printf("Scan time limit exceeded\n"); + ksft_print_msg("Scan time limit exceeded\n"); return 1; } } @@ -218,12 +218,12 @@ static int ksm_merge_pages(int merge_type, void *addr, size_t size, { if (merge_type == KSM_MERGE_MADVISE) { if (madvise(addr, size, MADV_MERGEABLE)) { - perror("madvise"); + ksft_perror("madvise"); return 1; } } else if (merge_type == KSM_MERGE_PRCTL) { if (prctl(PR_SET_MEMORY_MERGE, 1, 0, 0, 0)) { - perror("prctl"); + ksft_perror("prctl"); return 1; } } @@ -242,7 +242,7 @@ static int ksm_unmerge_pages(void *addr, size_t size, struct timespec start_time, int timeout) { if (madvise(addr, size, MADV_UNMERGEABLE)) { - perror("madvise"); + ksft_perror("madvise"); return 1; } return 0; @@ -288,8 +288,8 @@ static bool assert_ksm_pages_count(long dupl_page_count) static int ksm_save_def(struct ksm_sysfs *ksm_sysfs) { if (ksm_read_sysfs(KSM_FP("max_page_sharing"), &ksm_sysfs->max_page_sharing) || - numa_available() ? 0 : - ksm_read_sysfs(KSM_FP("merge_across_nodes"), &ksm_sysfs->merge_across_nodes) || + (numa_available() ? 0 : + ksm_read_sysfs(KSM_FP("merge_across_nodes"), &ksm_sysfs->merge_across_nodes)) || ksm_read_sysfs(KSM_FP("sleep_millisecs"), &ksm_sysfs->sleep_millisecs) || ksm_read_sysfs(KSM_FP("pages_to_scan"), &ksm_sysfs->pages_to_scan) || ksm_read_sysfs(KSM_FP("run"), &ksm_sysfs->run) || @@ -304,8 +304,8 @@ static int ksm_save_def(struct ksm_sysfs *ksm_sysfs) static int ksm_restore(struct ksm_sysfs *ksm_sysfs) { if (ksm_write_sysfs(KSM_FP("max_page_sharing"), ksm_sysfs->max_page_sharing) || - numa_available() ? 0 : - ksm_write_sysfs(KSM_FP("merge_across_nodes"), ksm_sysfs->merge_across_nodes) || + (numa_available() ? 0 : + ksm_write_sysfs(KSM_FP("merge_across_nodes"), ksm_sysfs->merge_across_nodes)) || ksm_write_sysfs(KSM_FP("pages_to_scan"), ksm_sysfs->pages_to_scan) || ksm_write_sysfs(KSM_FP("run"), ksm_sysfs->run) || ksm_write_sysfs(KSM_FP("sleep_millisecs"), ksm_sysfs->sleep_millisecs) || @@ -324,7 +324,7 @@ static int check_ksm_merge(int merge_type, int mapping, int prot, struct timespec start_time; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); return KSFT_FAIL; } @@ -338,7 +338,6 @@ static int check_ksm_merge(int merge_type, int mapping, int prot, /* verify that the right number of pages are merged */ if (assert_ksm_pages_count(page_count)) { - printf("OK\n"); munmap(map_ptr, page_size * page_count); if (merge_type == KSM_MERGE_PRCTL) prctl(PR_SET_MEMORY_MERGE, 0, 0, 0, 0); @@ -346,7 +345,6 @@ static int check_ksm_merge(int merge_type, int mapping, int prot, } err_out: - printf("Not OK\n"); munmap(map_ptr, page_size * page_count); return KSFT_FAIL; } @@ -358,7 +356,7 @@ static int check_ksm_unmerge(int merge_type, int mapping, int prot, int timeout, int page_count = 2; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); return KSFT_FAIL; } @@ -380,13 +378,11 @@ static int check_ksm_unmerge(int merge_type, int mapping, int prot, int timeout, /* check that unmerging was successful and 0 pages are currently merged */ if (assert_ksm_pages_count(0)) { - printf("OK\n"); munmap(map_ptr, page_size * page_count); return KSFT_PASS; } err_out: - printf("Not OK\n"); munmap(map_ptr, page_size * page_count); return KSFT_FAIL; } @@ -398,7 +394,7 @@ static int check_ksm_zero_page_merge(int merge_type, int mapping, int prot, long struct timespec start_time; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); return KSFT_FAIL; } @@ -425,12 +421,10 @@ static int check_ksm_zero_page_merge(int merge_type, int mapping, int prot, long else if (!use_zero_pages && !assert_ksm_pages_count(page_count)) goto err_out; - printf("OK\n"); munmap(map_ptr, page_size * page_count); return KSFT_PASS; err_out: - printf("Not OK\n"); munmap(map_ptr, page_size * page_count); return KSFT_FAIL; } @@ -446,9 +440,9 @@ static int get_next_mem_node(int node) mem_node = i % (max_node + 1); node_size = numa_node_size(mem_node, NULL); if (node_size > 0) - break; + return mem_node; } - return mem_node; + return -ENODEV; } static int get_first_mem_node(void) @@ -461,31 +455,33 @@ static int check_ksm_numa_merge(int merge_type, int mapping, int prot, int timeo { void *numa1_map_ptr, *numa2_map_ptr; struct timespec start_time; + int first_node, second_node; int page_count = 2; - int first_node; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); return KSFT_FAIL; } if (numa_available() < 0) { - perror("NUMA support not enabled"); + ksft_print_msg("NUMA support not enabled\n"); return KSFT_SKIP; } - if (numa_num_configured_nodes() <= 1) { - printf("At least 2 NUMA nodes must be available\n"); + first_node = get_first_mem_node(); + second_node = get_next_mem_node(first_node); + + if (second_node < 0) { + ksft_print_msg("At least 2 NUMA nodes with memory must be available\n"); return KSFT_SKIP; } if (ksm_write_sysfs(KSM_FP("merge_across_nodes"), merge_across_nodes)) return KSFT_FAIL; /* allocate 2 pages in 2 different NUMA nodes and fill them with the same data */ - first_node = get_first_mem_node(); numa1_map_ptr = numa_alloc_onnode(page_size, first_node); - numa2_map_ptr = numa_alloc_onnode(page_size, get_next_mem_node(first_node)); + numa2_map_ptr = numa_alloc_onnode(page_size, second_node); if (!numa1_map_ptr || !numa2_map_ptr) { - perror("numa_alloc_onnode"); + ksft_perror("numa_alloc_onnode"); return KSFT_FAIL; } @@ -510,13 +506,11 @@ static int check_ksm_numa_merge(int merge_type, int mapping, int prot, int timeo numa_free(numa1_map_ptr, page_size); numa_free(numa2_map_ptr, page_size); - printf("OK\n"); return KSFT_PASS; err_out: numa_free(numa1_map_ptr, page_size); numa_free(numa2_map_ptr, page_size); - printf("Not OK\n"); return KSFT_FAIL; } @@ -529,7 +523,7 @@ static int ksm_merge_hugepages_time(int merge_type, int mapping, int prot, int pagemap_fd, n_normal_pages, n_huge_pages; if (!thp_is_enabled()) { - printf("Transparent Hugepages not available\n"); + ksft_print_msg("Transparent Hugepages not available\n"); return KSFT_SKIP; } @@ -559,36 +553,35 @@ static int ksm_merge_hugepages_time(int merge_type, int mapping, int prot, else n_huge_pages++; } - printf("Number of normal pages: %d\n", n_normal_pages); - printf("Number of huge pages: %d\n", n_huge_pages); + ksft_print_msg("Number of normal pages: %d\n", n_normal_pages); + ksft_print_msg("Number of huge pages: %d\n", n_huge_pages); memset(map_ptr, '*', len); if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } if (ksm_merge_pages(merge_type, map_ptr, map_size, start_time, timeout)) goto err_out; if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC + (end_time.tv_nsec - start_time.tv_nsec); - printf("Total size: %lu MiB\n", map_size / MB); - printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC, + ksft_print_msg("Total size: %lu MiB\n", map_size / MB); + ksft_print_msg("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC, scan_time_ns % NSEC_PER_SEC); - printf("Average speed: %.3f MiB/s\n", (map_size / MB) / + ksft_print_msg("Average speed: %.3f MiB/s\n", (map_size / MB) / ((double)scan_time_ns / NSEC_PER_SEC)); munmap(map_ptr_orig, len + HPAGE_SIZE); return KSFT_PASS; err_out: - printf("Not OK\n"); munmap(map_ptr_orig, len + HPAGE_SIZE); return KSFT_FAIL; } @@ -606,30 +599,29 @@ static int ksm_merge_time(int merge_type, int mapping, int prot, int timeout, si return KSFT_FAIL; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } if (ksm_merge_pages(merge_type, map_ptr, map_size, start_time, timeout)) goto err_out; if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC + (end_time.tv_nsec - start_time.tv_nsec); - printf("Total size: %lu MiB\n", map_size / MB); - printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC, + ksft_print_msg("Total size: %lu MiB\n", map_size / MB); + ksft_print_msg("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC, scan_time_ns % NSEC_PER_SEC); - printf("Average speed: %.3f MiB/s\n", (map_size / MB) / + ksft_print_msg("Average speed: %.3f MiB/s\n", (map_size / MB) / ((double)scan_time_ns / NSEC_PER_SEC)); munmap(map_ptr, map_size); return KSFT_PASS; err_out: - printf("Not OK\n"); munmap(map_ptr, map_size); return KSFT_FAIL; } @@ -646,37 +638,36 @@ static int ksm_unmerge_time(int merge_type, int mapping, int prot, int timeout, if (!map_ptr) return KSFT_FAIL; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } if (ksm_merge_pages(merge_type, map_ptr, map_size, start_time, timeout)) goto err_out; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } if (ksm_unmerge_pages(map_ptr, map_size, start_time, timeout)) goto err_out; if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC + (end_time.tv_nsec - start_time.tv_nsec); - printf("Total size: %lu MiB\n", map_size / MB); - printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC, + ksft_print_msg("Total size: %lu MiB\n", map_size / MB); + ksft_print_msg("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC, scan_time_ns % NSEC_PER_SEC); - printf("Average speed: %.3f MiB/s\n", (map_size / MB) / + ksft_print_msg("Average speed: %.3f MiB/s\n", (map_size / MB) / ((double)scan_time_ns / NSEC_PER_SEC)); munmap(map_ptr, map_size); return KSFT_PASS; err_out: - printf("Not OK\n"); munmap(map_ptr, map_size); return KSFT_FAIL; } @@ -695,24 +686,24 @@ static int ksm_cow_time(int merge_type, int mapping, int prot, int timeout, size return KSFT_FAIL; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); return KSFT_FAIL; } for (size_t i = 0; i < page_count - 1; i = i + 2) memset(map_ptr + page_size * i, '-', 1); if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); return KSFT_FAIL; } cow_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC + (end_time.tv_nsec - start_time.tv_nsec); - printf("Total size: %lu MiB\n\n", (page_size * page_count) / MB); - printf("Not merged pages:\n"); - printf("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC, + ksft_print_msg("Total size: %lu MiB\n\n", (page_size * page_count) / MB); + ksft_print_msg("Not merged pages:\n"); + ksft_print_msg("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC, cow_time_ns % NSEC_PER_SEC); - printf("Average speed: %.3f MiB/s\n\n", ((page_size * (page_count / 2)) / MB) / + ksft_print_msg("Average speed: %.3f MiB/s\n\n", ((page_size * (page_count / 2)) / MB) / ((double)cow_time_ns / NSEC_PER_SEC)); /* Create 2000 pairs of duplicate pages */ @@ -724,30 +715,29 @@ static int ksm_cow_time(int merge_type, int mapping, int prot, int timeout, size goto err_out; if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } for (size_t i = 0; i < page_count - 1; i = i + 2) memset(map_ptr + page_size * i, '-', 1); if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) { - perror("clock_gettime"); + ksft_perror("clock_gettime"); goto err_out; } cow_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC + (end_time.tv_nsec - start_time.tv_nsec); - printf("Merged pages:\n"); - printf("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC, + ksft_print_msg("Merged pages:\n"); + ksft_print_msg("Total time: %ld.%09ld s\n", cow_time_ns / NSEC_PER_SEC, cow_time_ns % NSEC_PER_SEC); - printf("Average speed: %.3f MiB/s\n", ((page_size * (page_count / 2)) / MB) / + ksft_print_msg("Average speed: %.3f MiB/s\n", ((page_size * (page_count / 2)) / MB) / ((double)cow_time_ns / NSEC_PER_SEC)); munmap(map_ptr, page_size * page_count); return KSFT_PASS; err_out: - printf("Not OK\n"); munmap(map_ptr, page_size * page_count); return KSFT_FAIL; } @@ -765,6 +755,10 @@ int main(int argc, char *argv[]) bool use_zero_pages = KSM_USE_ZERO_PAGES_DEFAULT; bool merge_across_nodes = KSM_MERGE_ACROSS_NODES_DEFAULT; long size_MB = 0; + const char *test_descr = "KSM merging"; + + ksft_print_header(); + ksft_set_plan(1); while ((opt = getopt(argc, argv, "dha:p:l:z:m:s:t:MUZNPCHD")) != -1) { switch (opt) { @@ -773,17 +767,13 @@ int main(int argc, char *argv[]) break; case 'p': page_count = atol(optarg); - if (page_count <= 0) { - printf("The number of pages must be greater than 0\n"); - return KSFT_FAIL; - } + if (page_count <= 0) + ksft_exit_fail_msg("The number of pages must be greater than 0\n"); break; case 'l': ksm_scan_limit_sec = atoi(optarg); - if (ksm_scan_limit_sec <= 0) { - printf("Timeout value must be greater than 0\n"); - return KSFT_FAIL; - } + if (ksm_scan_limit_sec <= 0) + ksft_exit_fail_msg("Timeout value must be greater than 0\n"); break; case 'h': print_help(); @@ -805,19 +795,15 @@ int main(int argc, char *argv[]) break; case 's': size_MB = atoi(optarg); - if (size_MB <= 0) { - printf("Size must be greater than 0\n"); - return KSFT_FAIL; - } + if (size_MB <= 0) + ksft_exit_fail_msg("Size must be greater than 0\n"); break; case 't': { int tmp = atoi(optarg); - if (tmp < 0 || tmp > KSM_MERGE_LAST) { - printf("Invalid merge type\n"); - return KSFT_FAIL; - } + if (tmp < 0 || tmp > KSM_MERGE_LAST) + ksft_exit_fail_msg("Invalid merge type\n"); merge_type = tmp; } break; @@ -845,82 +831,80 @@ int main(int argc, char *argv[]) test_name = KSM_COW_TIME; break; default: - return KSFT_FAIL; + ksft_exit_fail_msg("Unknown option\n"); } } if (prot == 0) prot = str_to_prot(KSM_PROT_STR_DEFAULT); - if (access(KSM_SYSFS_PATH, F_OK)) { - printf("Config KSM not enabled\n"); - return KSFT_SKIP; - } + if (access(KSM_SYSFS_PATH, F_OK)) + ksft_exit_skip("Config KSM not enabled\n"); - if (ksm_save_def(&ksm_sysfs_old)) { - printf("Cannot save default tunables\n"); - return KSFT_FAIL; - } + if (ksm_save_def(&ksm_sysfs_old)) + ksft_exit_fail_msg("Cannot save default tunables\n"); if (ksm_write_sysfs(KSM_FP("run"), 2) || ksm_write_sysfs(KSM_FP("sleep_millisecs"), 0) || - numa_available() ? 0 : - ksm_write_sysfs(KSM_FP("merge_across_nodes"), 1) || + (numa_available() ? 0 : + ksm_write_sysfs(KSM_FP("merge_across_nodes"), 1)) || ksm_write_sysfs(KSM_FP("pages_to_scan"), page_count)) - return KSFT_FAIL; + ksft_exit_fail_msg("Cannot set up KSM tunables\n"); switch (test_name) { case CHECK_KSM_MERGE: + test_descr = "KSM merging"; ret = check_ksm_merge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, page_count, ksm_scan_limit_sec, page_size); break; case CHECK_KSM_UNMERGE: + test_descr = "KSM unmerging"; ret = check_ksm_unmerge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, page_size); break; case CHECK_KSM_ZERO_PAGE_MERGE: + test_descr = "KSM zero page merging"; ret = check_ksm_zero_page_merge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, page_count, ksm_scan_limit_sec, use_zero_pages, page_size); break; case CHECK_KSM_NUMA_MERGE: + test_descr = "KSM NUMA merging"; ret = check_ksm_numa_merge(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, merge_across_nodes, page_size); break; case KSM_MERGE_TIME: - if (size_MB == 0) { - printf("Option '-s' is required.\n"); - return KSFT_FAIL; - } + if (size_MB == 0) + ksft_exit_fail_msg("Option '-s' is required\n"); + test_descr = "KSM merge time"; ret = ksm_merge_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, size_MB); break; case KSM_MERGE_TIME_HUGE_PAGES: - if (size_MB == 0) { - printf("Option '-s' is required.\n"); - return KSFT_FAIL; - } + if (size_MB == 0) + ksft_exit_fail_msg("Option '-s' is required\n"); + test_descr = "KSM merge time with huge pages"; ret = ksm_merge_hugepages_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, size_MB); break; case KSM_UNMERGE_TIME: - if (size_MB == 0) { - printf("Option '-s' is required.\n"); - return KSFT_FAIL; - } + if (size_MB == 0) + ksft_exit_fail_msg("Option '-s' is required\n"); + test_descr = "KSM unmerge time"; ret = ksm_unmerge_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, size_MB); break; case KSM_COW_TIME: + test_descr = "KSM COW time"; ret = ksm_cow_time(merge_type, MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec, page_size); break; } - if (ksm_restore(&ksm_sysfs_old)) { - printf("Cannot restore default tunables\n"); - return KSFT_FAIL; - } + if (ksm_restore(&ksm_sysfs_old)) + ksft_print_msg("Cannot restore default tunables\n"); + + ksft_test_result_report(ret, "%s\n", test_descr); - return ret; + ksft_finished(); } diff --git a/tools/testing/selftests/mm/memfd_secret.c b/tools/testing/selftests/mm/memfd_secret.c index 9a0597310a76..06cffbc3fd77 100644 --- a/tools/testing/selftests/mm/memfd_secret.c +++ b/tools/testing/selftests/mm/memfd_secret.c @@ -57,33 +57,6 @@ static void test_file_apis(int fd) pass("file IO is blocked as expected\n"); } -static void test_mlock_limit(int fd) -{ - size_t len; - char *mem; - - len = mlock_limit_cur; - if (len % page_size != 0) - len = (len/page_size) * page_size; - - mem = mmap(NULL, len, prot, mode, fd, 0); - if (mem == MAP_FAILED) { - fail("unable to mmap secret memory\n"); - return; - } - munmap(mem, len); - - len = mlock_limit_max * 2; - mem = mmap(NULL, len, prot, mode, fd, 0); - if (mem != MAP_FAILED) { - fail("unexpected mlock limit violation\n"); - munmap(mem, len); - return; - } - - pass("mlock limit is respected\n"); -} - static void test_vmsplice(int fd, const char *desc) { ssize_t transferred; @@ -297,7 +270,7 @@ static void prepare(void) strerror(errno)); } -#define NUM_TESTS 6 +#define NUM_TESTS 5 int main(int argc, char *argv[]) { @@ -319,7 +292,6 @@ int main(int argc, char *argv[]) if (ftruncate(fd, page_size)) ksft_exit_fail_msg("ftruncate failed: %s\n", strerror(errno)); - test_mlock_limit(fd); test_file_apis(fd); /* * We have to run the first vmsplice test before any secretmem page was diff --git a/tools/testing/selftests/proc/proc-maps-race.c b/tools/testing/selftests/proc/proc-maps-race.c index a546475db550..7de87221bf27 100644 --- a/tools/testing/selftests/proc/proc-maps-race.c +++ b/tools/testing/selftests/proc/proc-maps-race.c @@ -340,7 +340,8 @@ static bool query_addr_at(int maps_fd, void *addr, static inline bool split_vma(FIXTURE_DATA(proc_maps_race) *self) { - return mmap(self->mod_info->addr, self->page_size, self->mod_info->prot | PROT_EXEC, + /* PROT_NONE differs from both readable neighbors. */ + return mmap(self->mod_info->addr, self->page_size, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) != MAP_FAILED; } diff --git a/tools/testing/selftests/proc/proc-pidns.c b/tools/testing/selftests/proc/proc-pidns.c index 52500597f951..f0fff6991d46 100644 --- a/tools/testing/selftests/proc/proc-pidns.c +++ b/tools/testing/selftests/proc/proc-pidns.c @@ -6,6 +6,7 @@ #include #include +#include #include #include #include diff --git a/tools/testing/selftests/rseq/rseq-x86-thread-pointer.h b/tools/testing/selftests/rseq/rseq-x86-thread-pointer.h index d3133587d996..5a29d6bec51f 100644 --- a/tools/testing/selftests/rseq/rseq-x86-thread-pointer.h +++ b/tools/testing/selftests/rseq/rseq-x86-thread-pointer.h @@ -8,13 +8,11 @@ #ifndef _RSEQ_X86_THREAD_POINTER #define _RSEQ_X86_THREAD_POINTER -#include - #ifdef __cplusplus extern "C" { #endif -#if __GNUC_PREREQ (11, 1) +#if __GNUC__ > 11 || (__GNUC__ == 11 && __GNUC_MINOR__ >= 1) static inline void *rseq_thread_pointer(void) { return __builtin_thread_pointer(); diff --git a/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c index 6f4c3f5a1c5d..7ef9de7b27eb 100644 --- a/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c +++ b/tools/testing/selftests/sched_ext/ddsp_bogus_dsq_fail.bpf.c @@ -14,18 +14,16 @@ s32 BPF_STRUCT_OPS(ddsp_bogus_dsq_fail_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags) { s32 cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0); + if (cpu < 0) + cpu = prev_cpu; - if (cpu >= 0) { - /* - * If we dispatch to a bogus DSQ that will fall back to the - * builtin global DSQ, we fail gracefully. - */ - scx_bpf_dsq_insert_vtime(p, 0xcafef00d, SCX_SLICE_DFL, - p->scx.dsq_vtime, 0); - return cpu; - } - - return prev_cpu; + /* + * If we dispatch to a bogus DSQ that will fall back to the + * builtin global DSQ, we fail gracefully. + */ + scx_bpf_dsq_insert_vtime(p, 0xcafef00d, SCX_SLICE_DFL, + p->scx.dsq_vtime, 0); + return cpu; } void BPF_STRUCT_OPS(ddsp_bogus_dsq_fail_exit, struct scx_exit_info *ei) diff --git a/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c index e4a55027778f..82dca4cdc0a6 100644 --- a/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c +++ b/tools/testing/selftests/sched_ext/ddsp_vtimelocal_fail.bpf.c @@ -14,15 +14,14 @@ s32 BPF_STRUCT_OPS(ddsp_vtimelocal_fail_select_cpu, struct task_struct *p, s32 prev_cpu, u64 wake_flags) { s32 cpu = scx_bpf_pick_idle_cpu(p->cpus_ptr, 0); + if (cpu < 0) + cpu = prev_cpu; - if (cpu >= 0) { - /* Shouldn't be allowed to vtime dispatch to a builtin DSQ. */ - scx_bpf_dsq_insert_vtime(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, - p->scx.dsq_vtime, 0); - return cpu; - } + /* Shouldn't be allowed to vtime dispatch to a builtin DSQ. */ + scx_bpf_dsq_insert_vtime(p, SCX_DSQ_LOCAL, SCX_SLICE_DFL, + p->scx.dsq_vtime, 0); - return prev_cpu; + return cpu; } void BPF_STRUCT_OPS(ddsp_vtimelocal_fail_exit, struct scx_exit_info *ei) diff --git a/tools/testing/selftests/sched_ext/exit.c b/tools/testing/selftests/sched_ext/exit.c index b987611789d1..01b17092d5c8 100644 --- a/tools/testing/selftests/sched_ext/exit.c +++ b/tools/testing/selftests/sched_ext/exit.c @@ -31,6 +31,7 @@ static enum scx_test_status run(void *ctx) continue; skel = exit__open(); + SCX_FAIL_IF(!skel, "Failed to open"); SCX_ENUM_INIT(skel); skel->rodata->exit_point = tc; SCX_FAIL_IF(exit__load(skel), "Failed to load skel"); diff --git a/tools/testing/selftests/sched_ext/prog_run.c b/tools/testing/selftests/sched_ext/prog_run.c index 05974820ca69..1129ec2aaddc 100644 --- a/tools/testing/selftests/sched_ext/prog_run.c +++ b/tools/testing/selftests/sched_ext/prog_run.c @@ -28,7 +28,8 @@ static enum scx_test_status setup(void **ctx) static enum scx_test_status run(void *ctx) { struct prog_run *skel = ctx; - struct bpf_link *link; + struct bpf_link *link = NULL; + enum scx_test_status status = SCX_TEST_PASS; int prog_fd, err = 0; prog_fd = bpf_program__fd(skel->progs.prog_run_syscall); @@ -42,23 +43,40 @@ static enum scx_test_status run(void *ctx) link = bpf_map__attach_struct_ops(skel->maps.prog_run_ops); if (!link) { SCX_ERR("Failed to attach scheduler"); - close(prog_fd); - return SCX_TEST_FAIL; + status = SCX_TEST_FAIL; + goto out; } err = bpf_prog_test_run_opts(prog_fd, &topts); - SCX_EQ(err, 0); + if (err) { + SCX_ERR("BPF_PROG_RUN failed (%d)", err); + status = SCX_TEST_FAIL; + goto out; + } /* Assumes uei.kind is written last */ while (skel->data->uei.kind == EXIT_KIND(SCX_EXIT_NONE)) sched_yield(); - SCX_EQ(skel->data->uei.kind, EXIT_KIND(SCX_EXIT_UNREG_BPF)); - SCX_EQ(skel->data->uei.exit_code, 0xdeadbeef); + if (skel->data->uei.kind != EXIT_KIND(SCX_EXIT_UNREG_BPF)) { + SCX_ERR("Unexpected exit kind: %llu", + (unsigned long long)skel->data->uei.kind); + status = SCX_TEST_FAIL; + goto out; + } + if (skel->data->uei.exit_code != 0xdeadbeef) { + SCX_ERR("Unexpected exit code: %lld", + (long long)skel->data->uei.exit_code); + status = SCX_TEST_FAIL; + goto out; + } + +out: close(prog_fd); - bpf_link__destroy(link); + if (link) + bpf_link__destroy(link); - return SCX_TEST_PASS; + return status; } static void cleanup(void *ctx) diff --git a/tools/testing/selftests/timers/leap-a-day.c b/tools/testing/selftests/timers/leap-a-day.c index 04004a7c0934..e677e31a2211 100644 --- a/tools/testing/selftests/timers/leap-a-day.c +++ b/tools/testing/selftests/timers/leap-a-day.c @@ -9,16 +9,19 @@ * kernel's leap-second behavior, as well as how well applications * handle the leap-second discontinuity. * - * Usage: leap-a-day [-s] [-i ] + * Usage: leap-a-day [-w] [-i ] [-t] * * Options: - * -s: Each iteration, set the date to 10 seconds before midnight GMT. - * This speeds up the number of leapsecond transitions tested, - * but because it calls settimeofday frequently, advancing the - * time by 24 hours every ~16 seconds, it may cause application - * disruption. + * -w: Only set the leap-second flag and wait for the leap second + * each iteration, instead of advancing the time. By default the + * date is set to 10 seconds before midnight GMT, which speeds up + * the number of leapsecond transitions tested, but because it + * calls settimeofday frequently, advancing the time by 24 hours + * every ~16 seconds, it may cause application disruption. * - * -i: Number of iterations to run (default: infinite) + * -i: Number of iterations to run (-1 = infinite, default: 10) + * + * -t: Print TAI time. * * Other notes: Disabling NTP prior to running this is advised, as the two * may conflict in their commands to the kernel. @@ -186,7 +189,7 @@ int main(int argc, char **argv) int opt; /* Process arguments */ - while ((opt = getopt(argc, argv, "sti:")) != -1) { + while ((opt = getopt(argc, argv, "wti:")) != -1) { switch (opt) { case 'w': printf("Only setting leap-flag, not changing time. It could take up to a day for leap to trigger.\n"); diff --git a/tools/testing/selftests/vDSO/vgetrandom-chacha.S b/tools/testing/selftests/vDSO/vgetrandom-chacha.S index a4a82e1c28a9..8c3cbf4dfd6a 100644 --- a/tools/testing/selftests/vDSO/vgetrandom-chacha.S +++ b/tools/testing/selftests/vDSO/vgetrandom-chacha.S @@ -14,7 +14,7 @@ #elif defined(__riscv) && __riscv_xlen == 64 #include "../../../../arch/riscv/kernel/vdso/vgetrandom-chacha.S" #elif defined(__s390x__) -#include "../../../../arch/s390/kernel/vdso64/vgetrandom-chacha.S" +#include "../../../../arch/s390/kernel/vdso/vgetrandom-chacha.S" #elif defined(__x86_64__) #include "../../../../arch/x86/entry/vdso/vgetrandom-chacha.S" #endif diff --git a/tools/testing/selftests/zram/zram_lib.sh b/tools/testing/selftests/zram/zram_lib.sh index 21ec1966de76..0d44d83888f9 100755 --- a/tools/testing/selftests/zram/zram_lib.sh +++ b/tools/testing/selftests/zram/zram_lib.sh @@ -37,7 +37,7 @@ kernel_gte() if [ $kernel_major -gt $major ]; then return 0 - elif [[ $kernel_major -eq $major && $kernel_minor -ge $minor ]]; then + elif [ $kernel_major -eq $major ] && [ $kernel_minor -ge $minor ]; then return 0 fi diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index bec634ac68cb..c16115a06b55 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -5630,7 +5630,7 @@ static int kvm_offline_cpu(unsigned int cpu) return 0; } -static void kvm_shutdown(void) +static void kvm_shutdown(void *data) { /* * Disable hardware virtualization and set kvm_rebooting to indicate @@ -5648,7 +5648,7 @@ static void kvm_shutdown(void) on_each_cpu(kvm_disable_virtualization_cpu, NULL, 1); } -static int kvm_suspend(void) +static int kvm_suspend(void *data) { /* * Secondary CPUs and CPU hotplug are disabled across the suspend/resume @@ -5665,7 +5665,7 @@ static int kvm_suspend(void) return 0; } -static void kvm_resume(void) +static void kvm_resume(void *data) { lockdep_assert_not_held(&kvm_usage_lock); lockdep_assert_irqs_disabled(); @@ -5673,12 +5673,16 @@ static void kvm_resume(void) WARN_ON_ONCE(kvm_enable_virtualization_cpu()); } -static struct syscore_ops kvm_syscore_ops = { +static const struct syscore_ops kvm_syscore_ops = { .suspend = kvm_suspend, .resume = kvm_resume, .shutdown = kvm_shutdown, }; +static struct syscore kvm_syscore = { + .ops = &kvm_syscore_ops, +}; + int kvm_enable_virtualization(void) { int r; @@ -5695,7 +5699,7 @@ int kvm_enable_virtualization(void) if (r) goto err_cpuhp; - register_syscore_ops(&kvm_syscore_ops); + register_syscore(&kvm_syscore); /* * Undo virtualization enabling and bail if the system is going down. @@ -5717,7 +5721,7 @@ int kvm_enable_virtualization(void) return 0; err_rebooting: - unregister_syscore_ops(&kvm_syscore_ops); + unregister_syscore(&kvm_syscore); cpuhp_remove_state(CPUHP_AP_KVM_ONLINE); err_cpuhp: kvm_arch_disable_virtualization(); @@ -5733,7 +5737,7 @@ void kvm_disable_virtualization(void) if (--kvm_usage_count) return; - unregister_syscore_ops(&kvm_syscore_ops); + unregister_syscore(&kvm_syscore); cpuhp_remove_state(CPUHP_AP_KVM_ONLINE); kvm_arch_disable_virtualization(); }