From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org,
torvalds@linux-foundation.org, stable@vger.kernel.org
Cc: lwn@lwn.net, jslaby@suse.cz,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Subject: Re: Linux 6.12.114
Date: Wed, 21 Oct 2026 10:38:08 +0200 [thread overview]
Message-ID: <2026102108-embroider-balsamic-0b6b@gregkh> (raw)
In-Reply-To: <2026102108-stoning-enunciate-650c@gregkh>
diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst
index 1ef5784c1b84..53faeed7c190 100644
--- a/Documentation/RCU/whatisRCU.rst
+++ b/Documentation/RCU/whatisRCU.rst
@@ -971,6 +971,16 @@ unfortunately any spinlock in a ``SLAB_TYPESAFE_BY_RCU`` object must be
initialized after each and every call to kmem_cache_alloc(), which renders
reference-free spinlock acquisition completely unsafe. Therefore, when
using ``SLAB_TYPESAFE_BY_RCU``, make proper use of a reference counter.
+If using refcount_t, the specialized refcount_{add|inc}_not_zero_acquire()
+and refcount_set_release() APIs should be used to ensure correct operation
+ordering when verifying object identity and when initializing newly
+allocated objects. Acquire fence in refcount_{add|inc}_not_zero_acquire()
+ensures that identity checks happen *after* reference count is taken.
+refcount_set_release() should be called after a newly allocated object is
+fully initialized and release fence ensures that new values are visible
+*before* refcount can be successfully taken by other users. Once
+refcount_set_release() is called, the object should be considered visible
+by other tasks.
(Those willing to initialize their locks in a kmem_cache constructor
may also use locking, including cache-friendly sequence locking.)
diff --git a/Documentation/admin-guide/device-mapper/dm-ima.rst b/Documentation/admin-guide/device-mapper/dm-ima.rst
index a4aa50a828e0..2a3b50ffbee4 100644
--- a/Documentation/admin-guide/device-mapper/dm-ima.rst
+++ b/Documentation/admin-guide/device-mapper/dm-ima.rst
@@ -424,7 +424,8 @@ section above) has the following data format for 'integrity' target.
target_attributes := <target_name> "," <target_version> "," <dev_name> "," <start>
<tag_size> "," <mode> "," [<meta_device> ","] [<block_size> ","] <recalculate> ","
- <allow_discards> "," <fix_padding> "," <fix_hmac> "," <legacy_recalculate> ","
+ <allow_discards> "," <allow_discards_keyed> "," <fix_padding> "," <fix_hmac> ","
+ <legacy_recalculate> ","
<journal_sectors> "," <interleave_sectors> "," <buffer_sectors> ";"
target_name := "target_name=integrity"
@@ -438,6 +439,7 @@ section above) has the following data format for 'integrity' target.
block_size := "block_size=" <N>
recalculate := "recalculate=" <yes_no>
allow_discards := "allow_discards=" <yes_no>
+ allow_discards_keyed := "allow_discards_keyed=" <yes_no>
fix_padding := "fix_padding=" <yes_no>
fix_hmac := "fix_hmac=" <yes_no>
legacy_recalculate := "legacy_recalculate=" <yes_no>
@@ -455,7 +457,8 @@ section above) has the following data format for 'integrity' target.
dm_version=4.45.0;
name=integrity1,uuid=,major=253,minor=1,minor_count=1,num_targets=1;
target_index=0,target_begin=0,target_len=7856,target_name=integrity,target_version=1.10.0,
- dev_name=253:0,start=0,tag_size=32,mode=J,recalculate=n,allow_discards=n,fix_padding=n,
+ dev_name=253:0,start=0,tag_size=32,mode=J,recalculate=n,allow_discards=n,
+ allow_discards_keyed=n,fix_padding=n,
fix_hmac=n,legacy_recalculate=n,journal_sectors=88,interleave_sectors=32768,buffer_sectors=128;
diff --git a/Documentation/admin-guide/device-mapper/dm-integrity.rst b/Documentation/admin-guide/device-mapper/dm-integrity.rst
index d8a5f14d0e3c..38fe5c70603b 100644
--- a/Documentation/admin-guide/device-mapper/dm-integrity.rst
+++ b/Documentation/admin-guide/device-mapper/dm-integrity.rst
@@ -185,6 +185,19 @@ allow_discards
Allow block discard requests (a.k.a. TRIM) for the integrity device.
Discards are only allowed to devices using internal hash.
+ A discarded block is marked with a constant filler tag that anyone
+ with raw write access to the backing device can forge without the
+ key. Use allow_discards_keyed instead on new volumes.
+
+allow_discards_keyed
+ Like allow_discards, but marks a discarded block with a keyed
+ checksum of the sector number, HMAC_key(salt || sector), instead of
+ the constant filler tag, so it can't be forged without the
+ integrity key.
+
+ Not compatible with volumes that already have discarded blocks
+ marked the old way; only use on a freshly formatted volume.
+
fix_padding
Use a smaller padding of the tag area that is more
space-efficient. If this option is not present, large padding is
diff --git a/Documentation/core-api/refcount-vs-atomic.rst b/Documentation/core-api/refcount-vs-atomic.rst
index 79a009ce11df..94e628c1eb49 100644
--- a/Documentation/core-api/refcount-vs-atomic.rst
+++ b/Documentation/core-api/refcount-vs-atomic.rst
@@ -86,7 +86,19 @@ Memory ordering guarantee changes:
* none (both fully unordered)
-case 2) - increment-based ops that return no value
+case 2) - non-"Read/Modify/Write" (RMW) ops with release ordering
+-----------------------------------------------------------------
+
+Function changes:
+
+ * atomic_set_release() --> refcount_set_release()
+
+Memory ordering guarantee changes:
+
+ * none (both provide RELEASE ordering)
+
+
+case 3) - increment-based ops that return no value
--------------------------------------------------
Function changes:
@@ -98,7 +110,7 @@ Memory ordering guarantee changes:
* none (both fully unordered)
-case 3) - decrement-based RMW ops that return no value
+case 4) - decrement-based RMW ops that return no value
------------------------------------------------------
Function changes:
@@ -110,7 +122,7 @@ Memory ordering guarantee changes:
* fully unordered --> RELEASE ordering
-case 4) - increment-based RMW ops that return a value
+case 5) - increment-based RMW ops that return a value
-----------------------------------------------------
Function changes:
@@ -126,7 +138,20 @@ Memory ordering guarantees changes:
result of obtaining pointer to the object!
-case 5) - generic dec/sub decrement-based RMW ops that return a value
+case 6) - increment-based RMW ops with acquire ordering that return a value
+---------------------------------------------------------------------------
+
+Function changes:
+
+ * atomic_inc_not_zero() --> refcount_inc_not_zero_acquire()
+ * no atomic counterpart --> refcount_add_not_zero_acquire()
+
+Memory ordering guarantees changes:
+
+ * fully ordered --> ACQUIRE ordering on success
+
+
+case 7) - generic dec/sub decrement-based RMW ops that return a value
---------------------------------------------------------------------
Function changes:
@@ -139,7 +164,7 @@ Memory ordering guarantees changes:
* fully ordered --> RELEASE ordering + ACQUIRE ordering on success
-case 6) other decrement-based RMW ops that return a value
+case 8) other decrement-based RMW ops that return a value
---------------------------------------------------------
Function changes:
@@ -154,7 +179,7 @@ Memory ordering guarantees changes:
.. note:: atomic_add_unless() only provides full order on success.
-case 7) - lock-based RMW
+case 9) - lock-based RMW
------------------------
Function changes:
diff --git a/Documentation/networking/af_xdp.rst b/Documentation/networking/af_xdp.rst
index dceeb0d763aa..e2c5ec61c2bd 100644
--- a/Documentation/networking/af_xdp.rst
+++ b/Documentation/networking/af_xdp.rst
@@ -43,12 +43,13 @@ UMEM also has two rings: the FILL ring and the COMPLETION ring. The
FILL ring is used by the application to send down addr for the kernel
to fill in with RX packet data. References to these frames will then
appear in the RX ring once each packet has been received. The
-COMPLETION ring, on the other hand, contains frame addr that the
-kernel has transmitted completely and can now be used again by user
-space, for either TX or RX. Thus, the frame addrs appearing in the
-COMPLETION ring are addrs that were previously transmitted using the
-TX ring. In summary, the RX and FILL rings are used for the RX path
-and the TX and COMPLETION rings are used for the TX path.
+COMPLETION ring, on the other hand, contains frame addresses from Tx
+descriptors that the kernel has finished processing and that can now be
+used again by user space, for either Tx or Rx. This includes frames whose
+transmission has completed as well as frames referenced by invalid Tx
+descriptors rejected by the kernel. A completion therefore returns
+ownership of a frame to user space, but does not by itself guarantee that
+the packet was successfully transmitted.
The socket is then finally bound with a bind() call to a device and a
specific queue id on that device, and it is not until bind is
@@ -169,14 +170,15 @@ chunks mode, then the incoming addr will be left untouched.
UMEM Completion Ring
~~~~~~~~~~~~~~~~~~~~
-The COMPLETION Ring is used transfer ownership of UMEM frames from
+The COMPLETION Ring is used to transfer ownership of UMEM frames from
kernel-space to user-space. Just like the FILL ring, UMEM indices are
-used.
-
-Frames passed from the kernel to user-space are frames that has been
-sent (TX ring) and can be used by user-space again.
-
-The user application consumes UMEM addrs from this ring.
+used. Frames passed from the kernel to user-space are frames referenced
+by Tx descriptors that the kernel has finished processing and can be
+used by user-space again. This includes both frames whose transmission
+has completed and frames referenced by invalid Tx descriptors that were
+rejected and reclaimed by the kernel. A completion entry does not
+guarantee successful packet transmission. The user application consumes
+UMEM addrs from this ring.
RX Ring
@@ -499,21 +501,25 @@ will be treated as an invalid descriptor.
These are the semantics for producing packets onto AF_XDP Tx ring
consisting of multiple frames:
-* When an invalid descriptor is found, all the other
- descriptors/frames of this packet are marked as invalid and not
- completed. The next descriptor is treated as the start of a new
- packet, even if this was not the intent (because we cannot guess
- the intent). As before, if your program is producing invalid
- descriptors you have a bug that must be fixed.
+* When an invalid descriptor is found, the complete packet is treated as
+ invalid. The kernel consumes descriptors through the descriptor marking
+ the end of the packet and returns all their frame addresses through the
+ COMPLETION ring. A standalone invalid descriptor is treated as a
+ one-descriptor invalid packet. The descriptor following the end of the
+ invalid packet is treated as the start of a new packet. As before, if
+ your program is producing invalid descriptors you have a bug that must
+ be fixed. Rejected descriptors are reported in the ``tx_invalid_descs``
+ statistic.
* Zero length descriptors are treated as invalid descriptors.
* For copy mode, the maximum supported number of frames in a packet is
- equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all
- descriptors accumulated so far are dropped and treated as
- invalid. To produce an application that will work on any system
- regardless of this config setting, limit the number of frags to 18,
- as the minimum value of the config is 17.
+ equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all descriptors
+ through the end of the oversized packet are consumed, treated as invalid,
+ and their frame addresses are returned through the COMPLETION ring. To
+ produce an application that will work on any system regardless of this
+ config setting, limit the number of frags to 18, as the minimum value of
+ the config is 17.
* For zero-copy mode, the limit is up to what the NIC HW
supports. Usually at least five on the NICs we have checked. We
diff --git a/Documentation/networking/net_cachelines/net_device.rst b/Documentation/networking/net_cachelines/net_device.rst
index 22b07c814f4a..ce8666529fc5 100644
--- a/Documentation/networking/net_cachelines/net_device.rst
+++ b/Documentation/networking/net_cachelines/net_device.rst
@@ -166,7 +166,7 @@ unsigned:1 wol_enabled
unsigned:1 threaded - - napi_poll(napi_enable,dev_set_threaded)
unsigned_long:1 see_all_hwtstamp_requests
unsigned_long:1 change_proto_down
-unsigned_long:1 netns_local
+unsigned_long:1 netns_immutable
unsigned_long:1 fcoe_mtu
struct_list_head net_notifier_list
struct_macsec_ops* macsec_ops
diff --git a/Documentation/networking/rxrpc.rst b/Documentation/networking/rxrpc.rst
index e807e18ba32a..2680abd1cb26 100644
--- a/Documentation/networking/rxrpc.rst
+++ b/Documentation/networking/rxrpc.rst
@@ -1172,3 +1172,14 @@ adjusted through sysctls in /proc/net/rxrpc/:
header plus exactly 1412 bytes of data. The terminal packet must contain
a four byte header plus any amount of data. In any event, a jumbo packet
may not exceed rxrpc_rx_mtu in size.
+
+
+API Function Reference
+======================
+
+.. kernel-doc:: net/rxrpc/af_rxrpc.c
+.. kernel-doc:: net/rxrpc/key.c
+.. kernel-doc:: net/rxrpc/peer_object.c
+.. kernel-doc:: net/rxrpc/recvmsg.c
+.. kernel-doc:: net/rxrpc/sendmsg.c
+.. kernel-doc:: net/rxrpc/server_key.c
diff --git a/Documentation/networking/switchdev.rst b/Documentation/networking/switchdev.rst
index f355f0166f1b..2966b7122f05 100644
--- a/Documentation/networking/switchdev.rst
+++ b/Documentation/networking/switchdev.rst
@@ -137,7 +137,7 @@ would be sub-port 0 on port 1 on switch 1.
Port Features
^^^^^^^^^^^^^
-dev->netns_local
+dev->netns_immutable
If the switchdev driver (and device) only supports offloading of the default
network namespace (netns), the driver should set this private flag to prevent
diff --git a/Makefile b/Makefile
index 440ddc25dc9a..fe02bbac2151 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
VERSION = 6
PATCHLEVEL = 12
-SUBLEVEL = 113
+SUBLEVEL = 114
EXTRAVERSION =
NAME = Baby Opossum Posse
@@ -753,12 +753,6 @@ endif # KBUILD_EXTMOD
# Defaults to vmlinux, but the arch makefile usually adds further targets
all: vmlinux
-CFLAGS_GCOV := -fprofile-arcs -ftest-coverage
-ifdef CONFIG_CC_IS_GCC
-CFLAGS_GCOV += -fno-tree-loop-im
-endif
-export CFLAGS_GCOV
-
# The arch Makefiles can override CC_FLAGS_FTRACE. We may also append it later.
ifdef CONFIG_FUNCTION_TRACER
CC_FLAGS_FTRACE := -pg
@@ -1027,6 +1021,27 @@ KBUILD_CFLAGS += -fno-builtin-wcslen
# Ensure compilers do not transform certain loops into calls to wcslen()
KBUILD_CFLAGS += -fno-builtin-wcslen
+CFLAGS_GCOV := -fprofile-arcs -ftest-coverage
+ifdef CONFIG_CC_IS_GCC
+CFLAGS_GCOV += -fno-tree-loop-im
+# Use atomic counter updates to avoid concurrent-access crashes in GCOV.
+# Only enable if -fprofile-update=prefer-atomic does not introduce new
+# undefined symbols (e.g. libatomic calls that the kernel cannot link).
+CFLAGS_GCOV += $(call try-run,\
+ echo 'long long x; void f(void){x++;}' | \
+ $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -w -fprofile-arcs \
+ -ftest-coverage -x c - -c -o "$$TMP.base" && \
+ echo 'long long x; void f(void){x++;}' | \
+ $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -w -fprofile-arcs \
+ -ftest-coverage -fprofile-update=prefer-atomic \
+ -x c - -c -o "$$TMP" && \
+ $(NM) "$$TMP.base" | grep ' U ' > "$$TMP.ubase" || true ; \
+ $(NM) "$$TMP" | grep ' U ' > "$$TMP.utest" || true ; \
+ cmp -s "$$TMP.ubase" "$$TMP.utest",\
+ -fprofile-update=prefer-atomic)
+endif
+export CFLAGS_GCOV
+
# change __FILE__ to the relative path from the srctree
KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
diff --git a/arch/arm/mach-exynos/mcpm-exynos.c b/arch/arm/mach-exynos/mcpm-exynos.c
index fd0dbeb93357..cb7d8a7b14e0 100644
--- a/arch/arm/mach-exynos/mcpm-exynos.c
+++ b/arch/arm/mach-exynos/mcpm-exynos.c
@@ -215,7 +215,7 @@ static const struct of_device_id exynos_dt_mcpm_match[] = {
{},
};
-static void exynos_mcpm_setup_entry_point(void)
+static void exynos_mcpm_setup_entry_point(void *data)
{
/*
* U-Boot SPL is hardcoded to jump to the start of ns_sram_base_addr
@@ -228,10 +228,14 @@ static void exynos_mcpm_setup_entry_point(void)
__raw_writel(__pa_symbol(mcpm_entry_point), ns_sram_base_addr + 8);
}
-static struct syscore_ops exynos_mcpm_syscore_ops = {
+static const struct syscore_ops exynos_mcpm_syscore_ops = {
.resume = exynos_mcpm_setup_entry_point,
};
+static struct syscore exynos_mcpm_syscore = {
+ .ops = &exynos_mcpm_syscore_ops,
+};
+
static int __init exynos_mcpm_init(void)
{
struct device_node *node;
@@ -300,9 +304,9 @@ static int __init exynos_mcpm_init(void)
pmu_raw_writel(value, EXYNOS_COMMON_OPTION(i));
}
- exynos_mcpm_setup_entry_point();
+ exynos_mcpm_setup_entry_point(NULL);
- register_syscore_ops(&exynos_mcpm_syscore_ops);
+ register_syscore(&exynos_mcpm_syscore);
return ret;
}
diff --git a/arch/arm/mach-exynos/suspend.c b/arch/arm/mach-exynos/suspend.c
index cac4e82f6c82..aca767935e20 100644
--- a/arch/arm/mach-exynos/suspend.c
+++ b/arch/arm/mach-exynos/suspend.c
@@ -53,9 +53,9 @@ struct exynos_pm_data {
void (*pm_prepare)(void);
void (*pm_resume_prepare)(void);
- void (*pm_resume)(void);
- int (*pm_suspend)(void);
int (*cpu_suspend)(unsigned long);
+
+ const struct syscore_ops *syscore_ops;
};
/* Used only on Exynos542x/5800 */
@@ -377,7 +377,7 @@ static void exynos5420_pm_prepare(void)
}
-static int exynos_pm_suspend(void)
+static int exynos_pm_suspend(void *data)
{
exynos_pm_central_suspend();
@@ -391,7 +391,7 @@ static int exynos_pm_suspend(void)
return 0;
}
-static int exynos5420_pm_suspend(void)
+static int exynos5420_pm_suspend(void *data)
{
u32 this_cluster;
@@ -409,7 +409,7 @@ static int exynos5420_pm_suspend(void)
return 0;
}
-static void exynos_pm_resume(void)
+static void exynos_pm_resume(void *data)
{
u32 cpuid = read_cpuid_part();
@@ -430,7 +430,7 @@ static void exynos_pm_resume(void)
exynos_set_delayed_reset_assertion(true);
}
-static void exynos3250_pm_resume(void)
+static void exynos3250_pm_resume(void *data)
{
u32 cpuid = read_cpuid_part();
@@ -474,7 +474,7 @@ static void exynos5420_prepare_pm_resume(void)
}
}
-static void exynos5420_pm_resume(void)
+static void exynos5420_pm_resume(void *data)
{
unsigned long tmp;
@@ -597,41 +597,52 @@ static const struct platform_suspend_ops exynos_suspend_ops = {
.valid = suspend_valid_only_mem,
};
+static const struct syscore_ops exynos3250_syscore_ops = {
+ .suspend = exynos_pm_suspend,
+ .resume = exynos3250_pm_resume,
+};
+
static const struct exynos_pm_data exynos3250_pm_data = {
.wkup_irq = exynos3250_wkup_irq,
.wake_disable_mask = ((0xFF << 8) | (0x1F << 1)),
- .pm_suspend = exynos_pm_suspend,
- .pm_resume = exynos3250_pm_resume,
.pm_prepare = exynos3250_pm_prepare,
.cpu_suspend = exynos3250_cpu_suspend,
+ .syscore_ops = &exynos3250_syscore_ops,
+};
+
+static const struct syscore_ops exynos_syscore_ops = {
+ .suspend = exynos_pm_suspend,
+ .resume = exynos_pm_resume,
};
static const struct exynos_pm_data exynos4_pm_data = {
.wkup_irq = exynos4_wkup_irq,
.wake_disable_mask = ((0xFF << 8) | (0x1F << 1)),
- .pm_suspend = exynos_pm_suspend,
- .pm_resume = exynos_pm_resume,
.pm_prepare = exynos_pm_prepare,
.cpu_suspend = exynos_cpu_suspend,
+ .syscore_ops = &exynos_syscore_ops,
};
static const struct exynos_pm_data exynos5250_pm_data = {
.wkup_irq = exynos5250_wkup_irq,
.wake_disable_mask = ((0xFF << 8) | (0x1F << 1)),
- .pm_suspend = exynos_pm_suspend,
- .pm_resume = exynos_pm_resume,
.pm_prepare = exynos_pm_prepare,
.cpu_suspend = exynos_cpu_suspend,
+ .syscore_ops = &exynos_syscore_ops,
+};
+
+static const struct syscore_ops exynos5420_syscore_ops = {
+ .resume = exynos5420_pm_resume,
+ .suspend = exynos5420_pm_suspend,
};
static const struct exynos_pm_data exynos5420_pm_data = {
.wkup_irq = exynos5250_wkup_irq,
.wake_disable_mask = (0x7F << 7) | (0x1F << 1),
.pm_resume_prepare = exynos5420_prepare_pm_resume,
- .pm_resume = exynos5420_pm_resume,
- .pm_suspend = exynos5420_pm_suspend,
.pm_prepare = exynos5420_pm_prepare,
.cpu_suspend = exynos5420_cpu_suspend,
+ .syscore_ops = &exynos5420_syscore_ops,
};
static const struct of_device_id exynos_pmu_of_device_ids[] __initconst = {
@@ -657,7 +668,7 @@ static const struct of_device_id exynos_pmu_of_device_ids[] __initconst = {
{ /*sentinel*/ },
};
-static struct syscore_ops exynos_pm_syscore_ops;
+static struct syscore exynos_pm_syscore;
void __init exynos_pm_init(void)
{
@@ -685,10 +696,9 @@ void __init exynos_pm_init(void)
tmp |= pm_data->wake_disable_mask;
pmu_raw_writel(tmp, S5P_WAKEUP_MASK);
- exynos_pm_syscore_ops.suspend = pm_data->pm_suspend;
- exynos_pm_syscore_ops.resume = pm_data->pm_resume;
+ exynos_pm_syscore.ops = pm_data->syscore_ops;
- register_syscore_ops(&exynos_pm_syscore_ops);
+ register_syscore(&exynos_pm_syscore);
suspend_set_ops(&exynos_suspend_ops);
/*
diff --git a/arch/arm/mach-pxa/generic.h b/arch/arm/mach-pxa/generic.h
index c9c2c46ecead..caad4fca8de3 100644
--- a/arch/arm/mach-pxa/generic.h
+++ b/arch/arm/mach-pxa/generic.h
@@ -34,9 +34,9 @@ extern void __init pxa27x_map_io(void);
extern void __init pxa3xx_init_irq(void);
extern void __init pxa3xx_map_io(void);
-extern struct syscore_ops pxa_irq_syscore_ops;
-extern struct syscore_ops pxa2xx_mfp_syscore_ops;
-extern struct syscore_ops pxa3xx_mfp_syscore_ops;
+extern struct syscore pxa_irq_syscore;
+extern struct syscore pxa2xx_mfp_syscore;
+extern struct syscore pxa3xx_mfp_syscore;
void __init pxa_set_ffuart_info(void *info);
void __init pxa_set_btuart_info(void *info);
diff --git a/arch/arm/mach-pxa/irq.c b/arch/arm/mach-pxa/irq.c
index a9ef71008147..78992c5d45ea 100644
--- a/arch/arm/mach-pxa/irq.c
+++ b/arch/arm/mach-pxa/irq.c
@@ -179,7 +179,7 @@ void __init pxa_init_irq(int irq_nr, int (*fn)(struct irq_data *, unsigned int))
static unsigned long saved_icmr[MAX_INTERNAL_IRQS/32];
static unsigned long saved_ipr[MAX_INTERNAL_IRQS];
-static int pxa_irq_suspend(void)
+static int pxa_irq_suspend(void *data)
{
int i;
@@ -198,7 +198,7 @@ static int pxa_irq_suspend(void)
return 0;
}
-static void pxa_irq_resume(void)
+static void pxa_irq_resume(void *data)
{
int i;
@@ -220,11 +220,15 @@ static void pxa_irq_resume(void)
#define pxa_irq_resume NULL
#endif
-struct syscore_ops pxa_irq_syscore_ops = {
+static const struct syscore_ops pxa_irq_syscore_ops = {
.suspend = pxa_irq_suspend,
.resume = pxa_irq_resume,
};
+struct syscore pxa_irq_syscore = {
+ .ops = &pxa_irq_syscore_ops,
+};
+
#ifdef CONFIG_OF
static const struct of_device_id intc_ids[] __initconst = {
{ .compatible = "marvell,pxa-intc", },
diff --git a/arch/arm/mach-pxa/mfp-pxa2xx.c b/arch/arm/mach-pxa/mfp-pxa2xx.c
index f5a3d890f682..d1347055fbe4 100644
--- a/arch/arm/mach-pxa/mfp-pxa2xx.c
+++ b/arch/arm/mach-pxa/mfp-pxa2xx.c
@@ -346,7 +346,7 @@ static unsigned long saved_gpdr[4];
static unsigned long saved_gplr[4];
static unsigned long saved_pgsr[4];
-static int pxa2xx_mfp_suspend(void)
+static int pxa2xx_mfp_suspend(void *data)
{
int i;
@@ -385,7 +385,7 @@ static int pxa2xx_mfp_suspend(void)
return 0;
}
-static void pxa2xx_mfp_resume(void)
+static void pxa2xx_mfp_resume(void *data)
{
int i;
@@ -404,11 +404,15 @@ static void pxa2xx_mfp_resume(void)
#define pxa2xx_mfp_resume NULL
#endif
-struct syscore_ops pxa2xx_mfp_syscore_ops = {
+static const struct syscore_ops pxa2xx_mfp_syscore_ops = {
.suspend = pxa2xx_mfp_suspend,
.resume = pxa2xx_mfp_resume,
};
+struct syscore pxa2xx_mfp_syscore = {
+ .ops = &pxa2xx_mfp_syscore_ops,
+};
+
static int __init pxa2xx_mfp_init(void)
{
int i;
diff --git a/arch/arm/mach-pxa/mfp-pxa3xx.c b/arch/arm/mach-pxa/mfp-pxa3xx.c
index d16ab7451efe..fe7498fbb62b 100644
--- a/arch/arm/mach-pxa/mfp-pxa3xx.c
+++ b/arch/arm/mach-pxa/mfp-pxa3xx.c
@@ -27,13 +27,13 @@
* a pull-down mode if they're an active low chip select, and we're
* just entering standby.
*/
-static int pxa3xx_mfp_suspend(void)
+static int pxa3xx_mfp_suspend(void *data)
{
mfp_config_lpm();
return 0;
}
-static void pxa3xx_mfp_resume(void)
+static void pxa3xx_mfp_resume(void *data)
{
mfp_config_run();
@@ -49,7 +49,11 @@ static void pxa3xx_mfp_resume(void)
#define pxa3xx_mfp_resume NULL
#endif
-struct syscore_ops pxa3xx_mfp_syscore_ops = {
+static const struct syscore_ops pxa3xx_mfp_syscore_ops = {
.suspend = pxa3xx_mfp_suspend,
.resume = pxa3xx_mfp_resume,
};
+
+struct syscore pxa3xx_mfp_syscore = {
+ .ops = &pxa3xx_mfp_syscore_ops,
+};
diff --git a/arch/arm/mach-pxa/pxa25x.c b/arch/arm/mach-pxa/pxa25x.c
index 03e34841fc00..70509a599814 100644
--- a/arch/arm/mach-pxa/pxa25x.c
+++ b/arch/arm/mach-pxa/pxa25x.c
@@ -235,8 +235,8 @@ static int __init pxa25x_init(void)
pxa25x_init_pm();
- register_syscore_ops(&pxa_irq_syscore_ops);
- register_syscore_ops(&pxa2xx_mfp_syscore_ops);
+ register_syscore(&pxa_irq_syscore);
+ register_syscore(&pxa2xx_mfp_syscore);
if (!of_have_populated_dt()) {
software_node_register(&pxa2xx_gpiochip_node);
diff --git a/arch/arm/mach-pxa/pxa27x.c b/arch/arm/mach-pxa/pxa27x.c
index f8382477d629..ff6361979038 100644
--- a/arch/arm/mach-pxa/pxa27x.c
+++ b/arch/arm/mach-pxa/pxa27x.c
@@ -337,8 +337,8 @@ static int __init pxa27x_init(void)
pxa27x_init_pm();
- register_syscore_ops(&pxa_irq_syscore_ops);
- register_syscore_ops(&pxa2xx_mfp_syscore_ops);
+ register_syscore(&pxa_irq_syscore);
+ register_syscore(&pxa2xx_mfp_syscore);
if (!of_have_populated_dt()) {
software_node_register(&pxa2xx_gpiochip_node);
diff --git a/arch/arm/mach-pxa/pxa3xx.c b/arch/arm/mach-pxa/pxa3xx.c
index 1d1e5713464d..06c578ea658e 100644
--- a/arch/arm/mach-pxa/pxa3xx.c
+++ b/arch/arm/mach-pxa/pxa3xx.c
@@ -424,8 +424,8 @@ static int __init pxa3xx_init(void)
if (cpu_is_pxa320())
enable_irq_wake(IRQ_WAKEUP1);
- register_syscore_ops(&pxa_irq_syscore_ops);
- register_syscore_ops(&pxa3xx_mfp_syscore_ops);
+ register_syscore(&pxa_irq_syscore);
+ register_syscore(&pxa3xx_mfp_syscore);
}
return ret;
diff --git a/arch/arm/mach-pxa/smemc.c b/arch/arm/mach-pxa/smemc.c
index 2d2a321d82f8..fb93a8f28356 100644
--- a/arch/arm/mach-pxa/smemc.c
+++ b/arch/arm/mach-pxa/smemc.c
@@ -18,7 +18,7 @@ static unsigned long msc[2];
static unsigned long sxcnfg, memclkcfg;
static unsigned long csadrcfg[4];
-static int pxa3xx_smemc_suspend(void)
+static int pxa3xx_smemc_suspend(void *data)
{
msc[0] = __raw_readl(MSC0);
msc[1] = __raw_readl(MSC1);
@@ -32,7 +32,7 @@ static int pxa3xx_smemc_suspend(void)
return 0;
}
-static void pxa3xx_smemc_resume(void)
+static void pxa3xx_smemc_resume(void *data)
{
__raw_writel(msc[0], MSC0);
__raw_writel(msc[1], MSC1);
@@ -46,11 +46,15 @@ static void pxa3xx_smemc_resume(void)
__raw_writel(0x2, CSMSADRCFG);
}
-static struct syscore_ops smemc_syscore_ops = {
+static const struct syscore_ops smemc_syscore_ops = {
.suspend = pxa3xx_smemc_suspend,
.resume = pxa3xx_smemc_resume,
};
+static struct syscore smemc_syscore = {
+ .ops = &smemc_syscore_ops,
+};
+
static int __init smemc_init(void)
{
if (cpu_is_pxa3xx()) {
@@ -64,7 +68,7 @@ static int __init smemc_init(void)
*/
__raw_writel(0x2, CSMSADRCFG);
- register_syscore_ops(&smemc_syscore_ops);
+ register_syscore(&smemc_syscore);
}
return 0;
diff --git a/arch/arm/mach-s3c/irq-pm-s3c64xx.c b/arch/arm/mach-s3c/irq-pm-s3c64xx.c
index 4a1e935bada1..ab726c595001 100644
--- a/arch/arm/mach-s3c/irq-pm-s3c64xx.c
+++ b/arch/arm/mach-s3c/irq-pm-s3c64xx.c
@@ -58,7 +58,7 @@ static struct irq_grp_save {
static u32 irq_uart_mask[SERIAL_SAMSUNG_UARTS];
-static int s3c64xx_irq_pm_suspend(void)
+static int s3c64xx_irq_pm_suspend(void *data)
{
struct irq_grp_save *grp = eint_grp_save;
int i;
@@ -79,7 +79,7 @@ static int s3c64xx_irq_pm_suspend(void)
return 0;
}
-static void s3c64xx_irq_pm_resume(void)
+static void s3c64xx_irq_pm_resume(void *data)
{
struct irq_grp_save *grp = eint_grp_save;
int i;
@@ -100,18 +100,22 @@ static void s3c64xx_irq_pm_resume(void)
S3C_PMDBG("%s: IRQ configuration restored\n", __func__);
}
-static struct syscore_ops s3c64xx_irq_syscore_ops = {
+static const struct syscore_ops s3c64xx_irq_syscore_ops = {
.suspend = s3c64xx_irq_pm_suspend,
.resume = s3c64xx_irq_pm_resume,
};
+static struct syscore s3c64xx_irq_syscore = {
+ .ops = &s3c64xx_irq_syscore_ops,
+};
+
static __init int s3c64xx_syscore_init(void)
{
/* Appropriate drivers (pinctrl, uart) handle this when using DT. */
if (of_have_populated_dt() || !soc_is_s3c64xx())
return 0;
- register_syscore_ops(&s3c64xx_irq_syscore_ops);
+ register_syscore(&s3c64xx_irq_syscore);
return 0;
}
diff --git a/arch/arm/mach-s5pv210/pm.c b/arch/arm/mach-s5pv210/pm.c
index 6fa70f787df4..fa270750364c 100644
--- a/arch/arm/mach-s5pv210/pm.c
+++ b/arch/arm/mach-s5pv210/pm.c
@@ -195,20 +195,24 @@ static const struct platform_suspend_ops s5pv210_suspend_ops = {
/*
* Syscore operations used to delay restore of certain registers.
*/
-static void s5pv210_pm_resume(void)
+static void s5pv210_pm_resume(void *data)
{
s3c_pm_do_restore_core(s5pv210_core_save, ARRAY_SIZE(s5pv210_core_save));
}
-static struct syscore_ops s5pv210_pm_syscore_ops = {
+static const struct syscore_ops s5pv210_pm_syscore_ops = {
.resume = s5pv210_pm_resume,
};
+static struct syscore s5pv210_pm_syscore = {
+ .ops = &s5pv210_pm_syscore_ops,
+};
+
/*
* Initialization entry point.
*/
void __init s5pv210_pm_init(void)
{
- register_syscore_ops(&s5pv210_pm_syscore_ops);
+ register_syscore(&s5pv210_pm_syscore);
suspend_set_ops(&s5pv210_suspend_ops);
}
diff --git a/arch/arm/mach-versatile/integrator_ap.c b/arch/arm/mach-versatile/integrator_ap.c
index 4bd6712e9f52..ee90d6619d0d 100644
--- a/arch/arm/mach-versatile/integrator_ap.c
+++ b/arch/arm/mach-versatile/integrator_ap.c
@@ -63,13 +63,13 @@ static void __init ap_map_io(void)
#ifdef CONFIG_PM
static unsigned long ic_irq_enable;
-static int irq_suspend(void)
+static int irq_suspend(void *data)
{
ic_irq_enable = readl(VA_IC_BASE + IRQ_ENABLE);
return 0;
}
-static void irq_resume(void)
+static void irq_resume(void *data)
{
/* disable all irq sources */
cm_clear_irqs();
@@ -83,14 +83,18 @@ static void irq_resume(void)
#define irq_resume NULL
#endif
-static struct syscore_ops irq_syscore_ops = {
+static const struct syscore_ops irq_syscore_ops = {
.suspend = irq_suspend,
.resume = irq_resume,
};
+static struct syscore irq_syscore = {
+ .ops = &irq_syscore_ops,
+};
+
static int __init irq_syscore_init(void)
{
- register_syscore_ops(&irq_syscore_ops);
+ register_syscore(&irq_syscore);
return 0;
}
diff --git a/arch/arm/mm/cache-b15-rac.c b/arch/arm/mm/cache-b15-rac.c
index 6f63b90f9e1a..e7807356dfab 100644
--- a/arch/arm/mm/cache-b15-rac.c
+++ b/arch/arm/mm/cache-b15-rac.c
@@ -256,7 +256,7 @@ static int b15_rac_dead_cpu(unsigned int cpu)
return 0;
}
-static int b15_rac_suspend(void)
+static int b15_rac_suspend(void *data)
{
/* Suspend the read-ahead cache oeprations, forcing our cache
* implementation to fallback to the regular ARMv7 calls.
@@ -271,7 +271,7 @@ static int b15_rac_suspend(void)
return 0;
}
-static void b15_rac_resume(void)
+static void b15_rac_resume(void *data)
{
/* Coming out of a S3 suspend/resume cycle, the read-ahead cache
* register RAC_CONFIG0_REG will be restored to its default value, make
@@ -282,11 +282,15 @@ static void b15_rac_resume(void)
clear_bit(RAC_SUSPENDED, &b15_rac_flags);
}
-static struct syscore_ops b15_rac_syscore_ops = {
+static const struct syscore_ops b15_rac_syscore_ops = {
.suspend = b15_rac_suspend,
.resume = b15_rac_resume,
};
+static struct syscore b15_rac_syscore = {
+ .ops = &b15_rac_syscore_ops,
+};
+
static int __init b15_rac_init(void)
{
struct device_node *dn, *cpu_dn;
@@ -347,7 +351,7 @@ static int __init b15_rac_init(void)
}
if (IS_ENABLED(CONFIG_PM_SLEEP))
- register_syscore_ops(&b15_rac_syscore_ops);
+ register_syscore(&b15_rac_syscore);
spin_lock(&rac_lock);
reg = __raw_readl(b15_rac_base + RAC_CONFIG0_REG);
diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c
index ff1fe2d39869..fd3bf1c9f852 100644
--- a/arch/arm64/kernel/ptrace.c
+++ b/arch/arm64/kernel/ptrace.c
@@ -559,6 +559,42 @@ static int gpr_get(struct task_struct *target,
return membuf_write(&to, uregs, sizeof(*uregs));
}
+static void update_syscall_orig_x0_after_ptrace(struct task_struct *target)
+{
+ struct pt_regs *regs = task_pt_regs(target);
+ struct kernel_siginfo *info = target->last_siginfo;
+
+ /*
+ * Skip the update for NO_SYSCALL (set either by the user or the
+ * tracer), as regs[0] holds the return value (see the comment in
+ * el0_svc_common()) and can be unwound using syscall_rollback().
+ */
+ if (regs->syscallno == NO_SYSCALL)
+ return;
+
+ /* We should only be called when target is in a ptrace stop */
+ if (WARN_ON_ONCE(!info))
+ return;
+
+ /*
+ * For compat tasks, orig_r0 is provided directly through GPR index
+ * 17.
+ */
+ if (is_compat_thread(task_thread_info(target)))
+ return;
+
+ /*
+ * Don't update orig_x0 for a syscall-exit-stop, as x0 now contains the
+ * return value of the system call.
+ */
+ if ((info->si_code & ~0x80) == SIGTRAP &&
+ target->ptrace_message == PTRACE_EVENTMSG_SYSCALL_EXIT) {
+ return;
+ }
+
+ regs->orig_x0 = regs->regs[0];
+}
+
static int gpr_set(struct task_struct *target, const struct user_regset *regset,
unsigned int pos, unsigned int count,
const void *kbuf, const void __user *ubuf)
@@ -574,6 +610,14 @@ static int gpr_set(struct task_struct *target, const struct user_regset *regset,
return -EINVAL;
task_pt_regs(target)->user_regs = newregs;
+
+ /*
+ * Keep orig_x0 authoritative so that seccomp (via
+ * syscall_get_arguments()), audit and the restart path all see the same
+ * first argument the syscall is dispatched with, even if it has been
+ * updated by a tracer.
+ */
+ update_syscall_orig_x0_after_ptrace(target);
return 0;
}
@@ -752,6 +796,12 @@ static int system_call_set(struct task_struct *target,
return ret;
task_pt_regs(target)->syscallno = syscallno;
+
+ /*
+ * Re-sync orig_x0 in case the syscall number has been changed
+ * from NO_SYSCALL.
+ */
+ update_syscall_orig_x0_after_ptrace(target);
return ret;
}
diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c
index 39f0e87a340e..8192bc0bbc87 100644
--- a/arch/arm64/kvm/at.c
+++ b/arch/arm64/kvm/at.c
@@ -1087,7 +1087,8 @@ void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
/* Do the stage-2 translation */
ipa = (par & GENMASK_ULL(47, 12)) | (vaddr & GENMASK_ULL(11, 0));
out.esr = 0;
- ret = kvm_walk_nested_s2(vcpu, ipa, &out);
+ scoped_guard(srcu, &vcpu->kvm->srcu)
+ ret = kvm_walk_nested_s2(vcpu, ipa, &out);
if (ret < 0)
return;
diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
index b217b256853c..3d5e74585505 100644
--- a/arch/arm64/kvm/vgic/vgic-v3.c
+++ b/arch/arm64/kvm/vgic/vgic-v3.c
@@ -416,9 +416,13 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
bool is_pending;
bool stored;
+ irq = vgic_get_irq(kvm, NULL, index);
+ if (!irq)
+ continue;
+
vcpu = irq->target_vcpu;
if (!vcpu)
- continue;
+ goto put_irq;
pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
@@ -429,7 +433,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
if (ptr != last_ptr) {
ret = kvm_read_guest_lock(kvm, ptr, &val, 1);
if (ret)
- goto out;
+ goto put_irq;
last_ptr = ptr;
}
@@ -441,7 +445,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
vgic_v4_get_vlpi_state(irq, &is_pending);
if (stored == is_pending)
- continue;
+ goto put_irq;
if (is_pending)
val |= 1 << bit_nr;
@@ -449,6 +453,8 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
val &= ~(1 << bit_nr);
ret = vgic_write_guest_lock(kvm, ptr, &val, 1);
+put_irq:
+ vgic_put_irq(kvm, irq);
if (ret)
goto out;
}
diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c
index 497b7af5caa4..967d8111642c 100644
--- a/arch/arm64/kvm/vgic/vgic.c
+++ b/arch/arm64/kvm/vgic/vgic.c
@@ -144,6 +144,7 @@ void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu)
list_for_each_entry_safe(irq, tmp, &vgic_cpu->ap_list_head, ap_list) {
if (irq->intid >= VGIC_MIN_LPI) {
raw_spin_lock(&irq->irq_lock);
+ irq->pending_latch = false;
list_del(&irq->ap_list);
irq->vcpu = NULL;
raw_spin_unlock(&irq->irq_lock);
@@ -661,7 +662,11 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
continue;
}
- /* This interrupt looks like it has to be migrated. */
+ /*
+ * This interrupt looks like it has to be migrated,
+ * make sure it is kept alive while locks are dropped.
+ */
+ vgic_get_irq_kref(irq);
raw_spin_unlock(&irq->irq_lock);
raw_spin_unlock(&vgic_cpu->ap_list_lock);
@@ -706,6 +711,8 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
raw_spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock);
raw_spin_unlock(&vcpuA->arch.vgic_cpu.ap_list_lock);
+ vgic_put_irq(vcpu->kvm, irq);
+
if (target_vcpu_needs_kick) {
kvm_make_request(KVM_REQ_IRQ_PENDING, target_vcpu);
kvm_vcpu_kick(target_vcpu);
diff --git a/arch/loongarch/kernel/machine_kexec.c b/arch/loongarch/kernel/machine_kexec.c
index 8ef4e4595d61..182226c67970 100644
--- a/arch/loongarch/kernel/machine_kexec.c
+++ b/arch/loongarch/kernel/machine_kexec.c
@@ -62,6 +62,7 @@ static void kexec_image_info(const struct kimage *kimage)
int machine_kexec_prepare(struct kimage *kimage)
{
int i;
+ char head[8];
char *bootloader = "kexec";
void *cmdline_ptr = (void *)KEXEC_CMDLINE_ADDR;
@@ -72,7 +73,9 @@ int machine_kexec_prepare(struct kimage *kimage)
/* Find the command line */
for (i = 0; i < kimage->nr_segments; i++) {
- if (!strncmp(bootloader, (char __user *)kimage->segment[i].buf, strlen(bootloader))) {
+ if (copy_from_user(head, kimage->segment[i].buf, strlen(bootloader)))
+ continue;
+ if (!strncmp(bootloader, head, strlen(bootloader))) {
if (!copy_from_user(cmdline_ptr, kimage->segment[i].buf, COMMAND_LINE_SIZE))
kimage->arch.cmdline_ptr = (unsigned long)cmdline_ptr;
break;
diff --git a/arch/loongarch/kernel/smp.c b/arch/loongarch/kernel/smp.c
index 780abcd86ca3..e3c3ff251dcb 100644
--- a/arch/loongarch/kernel/smp.c
+++ b/arch/loongarch/kernel/smp.c
@@ -499,28 +499,32 @@ int hibernate_resume_nonboot_cpu_disable(void)
*/
#ifdef CONFIG_PM
-static int loongson_ipi_suspend(void)
+static int loongson_ipi_suspend(void *data)
{
return 0;
}
-static void loongson_ipi_resume(void)
+static void loongson_ipi_resume(void *data)
{
iocsr_write32(0xffffffff, LOONGARCH_IOCSR_IPI_EN);
}
-static struct syscore_ops loongson_ipi_syscore_ops = {
+static const struct syscore_ops loongson_ipi_syscore_ops = {
.resume = loongson_ipi_resume,
.suspend = loongson_ipi_suspend,
};
+static struct syscore loongson_ipi_syscore = {
+ .ops = &loongson_ipi_syscore_ops,
+};
+
/*
* Enable boot cpu ipi before enabling nonboot cpus
* during syscore_resume.
*/
static int __init ipi_pm_init(void)
{
- register_syscore_ops(&loongson_ipi_syscore_ops);
+ register_syscore(&loongson_ipi_syscore);
return 0;
}
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 29a26b5e7416..4797b0bcc4fd 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -210,17 +210,13 @@ bool bpf_jit_supports_far_kfunc_call(void)
static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
{
- int off, tc_ninsn = 0;
+ int off, jmp_offset;
u8 tcc = tail_call_reg(ctx);
u8 a1 = LOONGARCH_GPR_A1;
u8 a2 = LOONGARCH_GPR_A2;
u8 t1 = LOONGARCH_GPR_T1;
u8 t2 = LOONGARCH_GPR_T2;
u8 t3 = LOONGARCH_GPR_T3;
- const int idx0 = ctx->idx;
-
-#define cur_offset (ctx->idx - idx0)
-#define jmp_offset (tc_ninsn - (cur_offset))
/*
* a0: &ctx
@@ -230,12 +226,12 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
* if (index >= array->map.max_entries)
* goto out;
*/
- tc_ninsn = insn ? ctx->offset[insn+1] - ctx->offset[insn] : ctx->offset[0];
emit_zext_32(ctx, a2, true);
off = offsetof(struct bpf_array, map.max_entries);
emit_insn(ctx, ldwu, t1, a1, off);
/* bgeu $a2, $t1, jmp_offset */
+ jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0;
if (emit_tailcall_jmp(ctx, BPF_JGE, a2, t1, jmp_offset) < 0)
goto toofar;
@@ -244,6 +240,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
* goto out;
*/
emit_insn(ctx, addid, REG_TCC, tcc, -1);
+ jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0;
if (emit_tailcall_jmp(ctx, BPF_JSLT, REG_TCC, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
goto toofar;
@@ -256,6 +253,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
off = offsetof(struct bpf_array, ptrs);
emit_insn(ctx, ldd, t2, t2, off);
/* beq $t2, $zero, jmp_offset */
+ jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0;
if (emit_tailcall_jmp(ctx, BPF_JEQ, t2, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
goto toofar;
@@ -269,8 +267,6 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
toofar:
pr_info_once("tail_call: jump too far\n");
return -1;
-#undef cur_offset
-#undef jmp_offset
}
static void emit_atomic(const struct bpf_insn *insn, struct jit_ctx *ctx)
@@ -564,7 +560,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
move_reg(ctx, t1, src);
emit_sext_32(ctx, t1, is32);
emit_insn(ctx, divd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
@@ -581,7 +577,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
emit_sext_32(ctx, t1, is32);
emit_sext_32(ctx, dst, is32);
emit_insn(ctx, divd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
@@ -599,7 +595,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
move_reg(ctx, t1, src);
emit_sext_32(ctx, t1, is32);
emit_insn(ctx, modd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
@@ -616,7 +612,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
emit_sext_32(ctx, t1, is32);
emit_sext_32(ctx, dst, is32);
emit_insn(ctx, modd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
diff --git a/arch/mips/alchemy/common/dbdma.c b/arch/mips/alchemy/common/dbdma.c
index 6a3c890f7bbf..6c2c2010bbae 100644
--- a/arch/mips/alchemy/common/dbdma.c
+++ b/arch/mips/alchemy/common/dbdma.c
@@ -982,7 +982,7 @@ u32 au1xxx_dbdma_put_dscr(u32 chanid, au1x_ddma_desc_t *dscr)
static unsigned long alchemy_dbdma_pm_data[NUM_DBDMA_CHANS + 1][6];
-static int alchemy_dbdma_suspend(void)
+static int alchemy_dbdma_suspend(void *data)
{
int i;
void __iomem *addr;
@@ -1019,7 +1019,7 @@ static int alchemy_dbdma_suspend(void)
return 0;
}
-static void alchemy_dbdma_resume(void)
+static void alchemy_dbdma_resume(void *data)
{
int i;
void __iomem *addr;
@@ -1044,11 +1044,15 @@ static void alchemy_dbdma_resume(void)
}
}
-static struct syscore_ops alchemy_dbdma_syscore_ops = {
+static const struct syscore_ops alchemy_dbdma_syscore_ops = {
.suspend = alchemy_dbdma_suspend,
.resume = alchemy_dbdma_resume,
};
+static struct syscore alchemy_dbdma_syscore = {
+ .ops = &alchemy_dbdma_syscore_ops,
+};
+
static int __init dbdma_setup(unsigned int irq, dbdev_tab_t *idtable)
{
int ret;
@@ -1071,7 +1075,7 @@ static int __init dbdma_setup(unsigned int irq, dbdev_tab_t *idtable)
printk(KERN_ERR "Cannot grab DBDMA interrupt!\n");
else {
dbdma_initialized = 1;
- register_syscore_ops(&alchemy_dbdma_syscore_ops);
+ register_syscore(&alchemy_dbdma_syscore);
}
return ret;
diff --git a/arch/mips/alchemy/common/irq.c b/arch/mips/alchemy/common/irq.c
index da9f9220048f..2403afcd2fb9 100644
--- a/arch/mips/alchemy/common/irq.c
+++ b/arch/mips/alchemy/common/irq.c
@@ -758,7 +758,7 @@ static inline void alchemy_ic_resume_one(void __iomem *base, unsigned long *d)
wmb();
}
-static int alchemy_ic_suspend(void)
+static int alchemy_ic_suspend(void *data)
{
alchemy_ic_suspend_one((void __iomem *)KSEG1ADDR(AU1000_IC0_PHYS_ADDR),
alchemy_gpic_pmdata);
@@ -767,7 +767,7 @@ static int alchemy_ic_suspend(void)
return 0;
}
-static void alchemy_ic_resume(void)
+static void alchemy_ic_resume(void *data)
{
alchemy_ic_resume_one((void __iomem *)KSEG1ADDR(AU1000_IC1_PHYS_ADDR),
&alchemy_gpic_pmdata[7]);
@@ -775,7 +775,7 @@ static void alchemy_ic_resume(void)
alchemy_gpic_pmdata);
}
-static int alchemy_gpic_suspend(void)
+static int alchemy_gpic_suspend(void *data)
{
void __iomem *base = (void __iomem *)KSEG1ADDR(AU1300_GPIC_PHYS_ADDR);
int i;
@@ -806,7 +806,7 @@ static int alchemy_gpic_suspend(void)
return 0;
}
-static void alchemy_gpic_resume(void)
+static void alchemy_gpic_resume(void *data)
{
void __iomem *base = (void __iomem *)KSEG1ADDR(AU1300_GPIC_PHYS_ADDR);
int i;
@@ -837,16 +837,24 @@ static void alchemy_gpic_resume(void)
wmb();
}
-static struct syscore_ops alchemy_ic_pmops = {
+static const struct syscore_ops alchemy_ic_pmops = {
.suspend = alchemy_ic_suspend,
.resume = alchemy_ic_resume,
};
-static struct syscore_ops alchemy_gpic_pmops = {
+static struct syscore alchemy_ic_pm = {
+ .ops = &alchemy_ic_pmops,
+};
+
+static const struct syscore_ops alchemy_gpic_pmops = {
.suspend = alchemy_gpic_suspend,
.resume = alchemy_gpic_resume,
};
+static struct syscore alchemy_gpic_pm = {
+ .ops = &alchemy_gpic_pmops,
+};
+
/******************************************************************************/
/* create chained handlers for the 4 IC requests to the MIPS IRQ ctrl */
@@ -880,7 +888,7 @@ static void __init au1000_init_irq(struct alchemy_irqmap *map)
ic_init((void __iomem *)KSEG1ADDR(AU1000_IC0_PHYS_ADDR));
ic_init((void __iomem *)KSEG1ADDR(AU1000_IC1_PHYS_ADDR));
- register_syscore_ops(&alchemy_ic_pmops);
+ register_syscore(&alchemy_ic_pm);
mips_cpu_irq_init();
/* register all 64 possible IC0+IC1 irq sources as type "none".
@@ -925,7 +933,7 @@ static void __init alchemy_gpic_init_irq(const struct alchemy_irqmap *dints)
int i;
void __iomem *bank_base;
- register_syscore_ops(&alchemy_gpic_pmops);
+ register_syscore(&alchemy_gpic_pm);
mips_cpu_irq_init();
/* disable & ack all possible interrupt sources */
diff --git a/arch/mips/alchemy/common/usb.c b/arch/mips/alchemy/common/usb.c
index 5d618547ebf0..a55f32bf517c 100644
--- a/arch/mips/alchemy/common/usb.c
+++ b/arch/mips/alchemy/common/usb.c
@@ -580,22 +580,26 @@ static void alchemy_usb_pm(int susp)
}
}
-static int alchemy_usb_suspend(void)
+static int alchemy_usb_suspend(void *data)
{
alchemy_usb_pm(1);
return 0;
}
-static void alchemy_usb_resume(void)
+static void alchemy_usb_resume(void *data)
{
alchemy_usb_pm(0);
}
-static struct syscore_ops alchemy_usb_pm_ops = {
+static const struct syscore_ops alchemy_usb_pm_syscore_ops = {
.suspend = alchemy_usb_suspend,
.resume = alchemy_usb_resume,
};
+static struct syscore alchemy_usb_pm_syscore = {
+ .ops = &alchemy_usb_pm_syscore_ops,
+};
+
static int __init alchemy_usb_init(void)
{
int ret = 0;
@@ -620,7 +624,7 @@ static int __init alchemy_usb_init(void)
}
if (!ret)
- register_syscore_ops(&alchemy_usb_pm_ops);
+ register_syscore(&alchemy_usb_pm_syscore);
return ret;
}
diff --git a/arch/mips/pci/pci-alchemy.c b/arch/mips/pci/pci-alchemy.c
index 58625d1b6465..6bfee0f71803 100644
--- a/arch/mips/pci/pci-alchemy.c
+++ b/arch/mips/pci/pci-alchemy.c
@@ -304,7 +304,7 @@ static int alchemy_pci_def_idsel(unsigned int devsel, int assert)
}
/* save PCI controller register contents. */
-static int alchemy_pci_suspend(void)
+static int alchemy_pci_suspend(void *data)
{
struct alchemy_pci_context *ctx = __alchemy_pci_ctx;
if (!ctx)
@@ -326,7 +326,7 @@ static int alchemy_pci_suspend(void)
return 0;
}
-static void alchemy_pci_resume(void)
+static void alchemy_pci_resume(void *data)
{
struct alchemy_pci_context *ctx = __alchemy_pci_ctx;
if (!ctx)
@@ -354,9 +354,13 @@ static void alchemy_pci_resume(void)
alchemy_pci_wired_entry(ctx); /* install it */
}
-static struct syscore_ops alchemy_pci_pmops = {
- .suspend = alchemy_pci_suspend,
- .resume = alchemy_pci_resume,
+static const struct syscore_ops alchemy_pci_syscore_ops = {
+ .suspend = alchemy_pci_suspend,
+ .resume = alchemy_pci_resume,
+};
+
+static struct syscore alchemy_pci_syscore = {
+ .ops = &alchemy_pci_syscore_ops,
};
static int alchemy_pci_probe(struct platform_device *pdev)
@@ -478,7 +482,7 @@ static int alchemy_pci_probe(struct platform_device *pdev)
__alchemy_pci_ctx = ctx;
platform_set_drvdata(pdev, ctx);
- register_syscore_ops(&alchemy_pci_pmops);
+ register_syscore(&alchemy_pci_syscore);
register_pci_controller(&ctx->alchemy_pci_ctrl);
dev_info(&pdev->dev, "PCI controller at %ld MHz\n",
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index 6d98e6f08d4d..a709969277f9 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -1304,6 +1304,21 @@ extern int pudp_test_and_clear_young(struct vm_area_struct *vma,
static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
unsigned long addr, pmd_t *pmdp)
{
+ /*
+ * Non-present PMDs can be migration entries or device-private THP
+ * entries. This can happen at 2 places:
+ * - When the address space is being unmapped zap_huge_pmd(), and we
+ * encounter non-present pmds.
+ * - migrate_vma_collect_huge_pmd() could calls this during migration
+ * of device-private pmd entries.
+ */
+ if (!pmd_present(*pmdp)) {
+ pmd_t old_pmd = READ_ONCE(*pmdp);
+
+ pmd_clear(pmdp);
+ return old_pmd;
+ }
+
if (radix_enabled())
return radix__pmdp_huge_get_and_clear(mm, addr, pmdp);
return hash__pmdp_huge_get_and_clear(mm, addr, pmdp);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 92930b0b5d0e..8ac7c1d3564f 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -91,7 +91,7 @@ static inline bool xive_enabled(void) { return __xive_enabled; }
bool xive_spapr_init(void);
bool xive_native_init(void);
-void xive_smp_probe(void);
+int xive_smp_probe(void);
int xive_smp_prepare_cpu(unsigned int cpu);
void xive_smp_setup_cpu(void);
void xive_smp_disable_cpu(void);
@@ -154,7 +154,7 @@ static inline bool xive_enabled(void) { return false; }
static inline bool xive_spapr_init(void) { return false; }
static inline bool xive_native_init(void) { return false; }
-static inline void xive_smp_probe(void) { }
+static inline int xive_smp_probe(void) { return -EINVAL; }
static inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
static inline void xive_smp_setup_cpu(void) { }
static inline void xive_smp_disable_cpu(void) { }
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index 5a4a75369043..af3fe81c7e67 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -191,16 +191,21 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp, int full)
{
pmd_t pmd;
+ bool was_present = pmd_present(*pmdp);
+
VM_BUG_ON(addr & ~HPAGE_PMD_MASK);
- VM_BUG_ON((pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) &&
- !pmd_devmap(*pmdp)) || !pmd_present(*pmdp));
+ VM_BUG_ON(was_present && !pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp));
+ /*
+ * Check pmdp_huge_get_and_clear() for non-present pmd case.
+ */
pmd = pmdp_huge_get_and_clear(vma->vm_mm, addr, pmdp);
/*
* if it not a fullmm flush, then we can possibly end up converting
* this PMD pte entry to a regular level 0 PTE by a parallel page fault.
- * Make sure we flush the tlb in this case.
+ * Make sure we flush the tlb in this case. TLB flush not needed for
+ * non-present case.
*/
- if (!full)
+ if (was_present && !full)
flush_pmd_tlb_range(vma, addr, addr + HPAGE_PMD_SIZE);
return pmd;
}
diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c
index e89f64a0f24a..2d817d95db80 100644
--- a/arch/powerpc/mm/pgtable-frag.c
+++ b/arch/powerpc/mm/pgtable-frag.c
@@ -25,6 +25,7 @@ void pte_frag_destroy(void *pte_frag)
count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT;
/* We allow PTE_FRAG_NR fragments from a PTE page */
if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) {
+ folio_clear_active(ptdesc_folio(ptdesc));
pagetable_pte_dtor(ptdesc);
pagetable_free(ptdesc);
}
diff --git a/arch/powerpc/platforms/cell/spu_base.c b/arch/powerpc/platforms/cell/spu_base.c
index dea6f0f25897..bbd360320680 100644
--- a/arch/powerpc/platforms/cell/spu_base.c
+++ b/arch/powerpc/platforms/cell/spu_base.c
@@ -727,7 +727,7 @@ static inline void crash_register_spus(struct list_head *list)
}
#endif
-static void spu_shutdown(void)
+static void spu_shutdown(void *data)
{
struct spu *spu;
@@ -739,10 +739,14 @@ static void spu_shutdown(void)
mutex_unlock(&spu_full_list_mutex);
}
-static struct syscore_ops spu_syscore_ops = {
+static const struct syscore_ops spu_syscore_ops = {
.shutdown = spu_shutdown,
};
+static struct syscore spu_syscore = {
+ .ops = &spu_syscore_ops,
+};
+
static int __init init_spu_base(void)
{
int i, ret = 0;
@@ -776,7 +780,7 @@ static int __init init_spu_base(void)
crash_register_spus(&spu_full_list);
mutex_unlock(&spu_full_list_mutex);
spu_add_dev_attr(&dev_attr_stat);
- register_syscore_ops(&spu_syscore_ops);
+ register_syscore(&spu_syscore);
spu_init_affinity();
diff --git a/arch/powerpc/platforms/powermac/pic.c b/arch/powerpc/platforms/powermac/pic.c
index 2202bf77c7a3..582ff2aafc5d 100644
--- a/arch/powerpc/platforms/powermac/pic.c
+++ b/arch/powerpc/platforms/powermac/pic.c
@@ -599,7 +599,7 @@ static int pmacpic_find_viaint(void)
return viaint;
}
-static int pmacpic_suspend(void)
+static int pmacpic_suspend(void *data)
{
int viaint = pmacpic_find_viaint();
@@ -620,7 +620,7 @@ static int pmacpic_suspend(void)
return 0;
}
-static void pmacpic_resume(void)
+static void pmacpic_resume(void *data)
{
int i;
@@ -633,15 +633,19 @@ static void pmacpic_resume(void)
pmac_unmask_irq(irq_get_irq_data(i));
}
-static struct syscore_ops pmacpic_syscore_ops = {
+static const struct syscore_ops pmacpic_syscore_ops = {
.suspend = pmacpic_suspend,
.resume = pmacpic_resume,
};
+static struct syscore pmacpic_syscore = {
+ .ops = &pmacpic_syscore_ops,
+};
+
static int __init init_pmacpic_syscore(void)
{
if (pmac_irq_hw[0])
- register_syscore_ops(&pmacpic_syscore_ops);
+ register_syscore(&pmacpic_syscore);
return 0;
}
diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
index 8f14f0581a21..6fd88c5a55e1 100644
--- a/arch/powerpc/platforms/powernv/smp.c
+++ b/arch/powerpc/platforms/powernv/smp.c
@@ -329,10 +329,12 @@ static void pnv_cause_ipi(int cpu)
static void __init pnv_smp_probe(void)
{
- if (xive_enabled())
- xive_smp_probe();
- else
+ if (xive_enabled()) {
+ if (xive_smp_probe() < 0)
+ return;
+ } else {
xics_smp_probe();
+ }
if (cpu_has_feature(CPU_FTR_DBELL)) {
ic_cause_ipi = smp_ops->cause_ipi;
diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c
index c597711ef20a..0d9b0b901807 100644
--- a/arch/powerpc/platforms/pseries/smp.c
+++ b/arch/powerpc/platforms/pseries/smp.c
@@ -194,10 +194,12 @@ static int pseries_cause_nmi_ipi(int cpu)
static __init void pSeries_smp_probe(void)
{
- if (xive_enabled())
- xive_smp_probe();
- else
+ if (xive_enabled()) {
+ if (xive_smp_probe() < 0)
+ return;
+ } else {
xics_smp_probe();
+ }
/* No doorbell facility, must use the interrupt controller for IPIs */
if (!cpu_has_feature(CPU_FTR_DBELL))
diff --git a/arch/powerpc/sysdev/fsl_lbc.c b/arch/powerpc/sysdev/fsl_lbc.c
index 217cea150987..7ed07232a69a 100644
--- a/arch/powerpc/sysdev/fsl_lbc.c
+++ b/arch/powerpc/sysdev/fsl_lbc.c
@@ -350,7 +350,7 @@ static int fsl_lbc_ctrl_probe(struct platform_device *dev)
#ifdef CONFIG_SUSPEND
/* save lbc registers */
-static int fsl_lbc_syscore_suspend(void)
+static int fsl_lbc_syscore_suspend(void *data)
{
struct fsl_lbc_ctrl *ctrl;
struct fsl_lbc_regs __iomem *lbc;
@@ -374,7 +374,7 @@ static int fsl_lbc_syscore_suspend(void)
}
/* restore lbc registers */
-static void fsl_lbc_syscore_resume(void)
+static void fsl_lbc_syscore_resume(void *data)
{
struct fsl_lbc_ctrl *ctrl;
struct fsl_lbc_regs __iomem *lbc;
@@ -408,10 +408,14 @@ static const struct of_device_id fsl_lbc_match[] = {
};
#ifdef CONFIG_SUSPEND
-static struct syscore_ops lbc_syscore_pm_ops = {
+static const struct syscore_ops lbc_syscore_pm_ops = {
.suspend = fsl_lbc_syscore_suspend,
.resume = fsl_lbc_syscore_resume,
};
+
+static struct syscore lbc_syscore_pm = {
+ .ops = &lbc_syscore_pm_ops,
+};
#endif
static struct platform_driver fsl_lbc_ctrl_driver = {
@@ -425,7 +429,7 @@ static struct platform_driver fsl_lbc_ctrl_driver = {
static int __init fsl_lbc_init(void)
{
#ifdef CONFIG_SUSPEND
- register_syscore_ops(&lbc_syscore_pm_ops);
+ register_syscore(&lbc_syscore_pm);
#endif
return platform_driver_register(&fsl_lbc_ctrl_driver);
}
diff --git a/arch/powerpc/sysdev/fsl_pci.c b/arch/powerpc/sysdev/fsl_pci.c
index ef7707ea0db7..4e501654cb41 100644
--- a/arch/powerpc/sysdev/fsl_pci.c
+++ b/arch/powerpc/sysdev/fsl_pci.c
@@ -1258,7 +1258,7 @@ static void fsl_pci_syscore_do_suspend(struct pci_controller *hose)
send_pme_turnoff_message(hose);
}
-static int fsl_pci_syscore_suspend(void)
+static int fsl_pci_syscore_suspend(void *data)
{
struct pci_controller *hose, *tmp;
@@ -1291,7 +1291,7 @@ static void fsl_pci_syscore_do_resume(struct pci_controller *hose)
setup_pci_atmu(hose);
}
-static void fsl_pci_syscore_resume(void)
+static void fsl_pci_syscore_resume(void *data)
{
struct pci_controller *hose, *tmp;
@@ -1299,10 +1299,14 @@ static void fsl_pci_syscore_resume(void)
fsl_pci_syscore_do_resume(hose);
}
-static struct syscore_ops pci_syscore_pm_ops = {
+static const struct syscore_ops pci_syscore_pm_ops = {
.suspend = fsl_pci_syscore_suspend,
.resume = fsl_pci_syscore_resume,
};
+
+static struct syscore pci_syscore_pm = {
+ .ops = &pci_syscore_pm_ops,
+};
#endif
void fsl_pcibios_fixup_phb(struct pci_controller *phb)
@@ -1359,7 +1363,7 @@ static struct platform_driver fsl_pci_driver = {
static int __init fsl_pci_init(void)
{
#ifdef CONFIG_PM_SLEEP
- register_syscore_ops(&pci_syscore_pm_ops);
+ register_syscore(&pci_syscore_pm);
#endif
return platform_driver_register(&fsl_pci_driver);
}
diff --git a/arch/powerpc/sysdev/ipic.c b/arch/powerpc/sysdev/ipic.c
index 5f69e2d50f26..d7b2db9926f6 100644
--- a/arch/powerpc/sysdev/ipic.c
+++ b/arch/powerpc/sysdev/ipic.c
@@ -817,7 +817,7 @@ static struct {
u32 sercr;
} ipic_saved_state;
-static int ipic_suspend(void)
+static int ipic_suspend(void *data)
{
struct ipic *ipic = primary_ipic;
@@ -848,7 +848,7 @@ static int ipic_suspend(void)
return 0;
}
-static void ipic_resume(void)
+static void ipic_resume(void *data)
{
struct ipic *ipic = primary_ipic;
@@ -870,18 +870,22 @@ static void ipic_resume(void)
#define ipic_resume NULL
#endif
-static struct syscore_ops ipic_syscore_ops = {
+static const struct syscore_ops ipic_syscore_ops = {
.suspend = ipic_suspend,
.resume = ipic_resume,
};
+static struct syscore ipic_syscore = {
+ .ops = &ipic_syscore_ops,
+};
+
static int __init init_ipic_syscore(void)
{
if (!primary_ipic || !primary_ipic->regs)
return -ENODEV;
printk(KERN_DEBUG "Registering ipic system core operations\n");
- register_syscore_ops(&ipic_syscore_ops);
+ register_syscore(&ipic_syscore);
return 0;
}
diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c
index d94cf36b0f65..479dca1d7d44 100644
--- a/arch/powerpc/sysdev/mpic.c
+++ b/arch/powerpc/sysdev/mpic.c
@@ -1943,7 +1943,7 @@ static void mpic_suspend_one(struct mpic *mpic)
}
}
-static int mpic_suspend(void)
+static int mpic_suspend(void *data)
{
struct mpic *mpic = mpics;
@@ -1985,7 +1985,7 @@ static void mpic_resume_one(struct mpic *mpic)
} /* end for loop */
}
-static void mpic_resume(void)
+static void mpic_resume(void *data)
{
struct mpic *mpic = mpics;
@@ -1995,19 +1995,23 @@ static void mpic_resume(void)
}
}
-static struct syscore_ops mpic_syscore_ops = {
+static const struct syscore_ops mpic_syscore_ops = {
.resume = mpic_resume,
.suspend = mpic_suspend,
};
+static struct syscore mpic_syscore = {
+ .ops = &mpic_syscore_ops,
+};
+
static int mpic_init_sys(void)
{
int rc;
- register_syscore_ops(&mpic_syscore_ops);
+ register_syscore(&mpic_syscore);
rc = subsys_system_register(&mpic_subsys, NULL);
if (rc) {
- unregister_syscore_ops(&mpic_syscore_ops);
+ unregister_syscore(&mpic_syscore);
pr_err("mpic: Failed to register subsystem!\n");
return rc;
}
diff --git a/arch/powerpc/sysdev/mpic_timer.c b/arch/powerpc/sysdev/mpic_timer.c
index 7166e2e0baaf..60f5b3934b51 100644
--- a/arch/powerpc/sysdev/mpic_timer.c
+++ b/arch/powerpc/sysdev/mpic_timer.c
@@ -519,7 +519,7 @@ static void __init timer_group_init(struct device_node *np)
kfree(priv);
}
-static void mpic_timer_resume(void)
+static void mpic_timer_resume(void *data)
{
struct timer_group_priv *priv;
@@ -535,10 +535,14 @@ static const struct of_device_id mpic_timer_ids[] = {
{},
};
-static struct syscore_ops mpic_timer_syscore_ops = {
+static const struct syscore_ops mpic_timer_syscore_ops = {
.resume = mpic_timer_resume,
};
+static struct syscore mpic_timer_syscore = {
+ .ops = &mpic_timer_syscore_ops,
+};
+
static int __init mpic_timer_init(void)
{
struct device_node *np = NULL;
@@ -546,7 +550,7 @@ static int __init mpic_timer_init(void)
for_each_matching_node(np, mpic_timer_ids)
timer_group_init(np);
- register_syscore_ops(&mpic_timer_syscore_ops);
+ register_syscore(&mpic_timer_syscore);
if (list_empty(&timer_group_list))
return -ENODEV;
diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
index fa01818c1972..f42ace109593 100644
--- a/arch/powerpc/sysdev/xive/common.c
+++ b/arch/powerpc/sysdev/xive/common.c
@@ -1270,15 +1270,19 @@ noinstr static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
xive_ops->put_ipi(cpu, xc);
}
-void __init xive_smp_probe(void)
+int __init xive_smp_probe(void)
{
+ int ret;
+
smp_ops->cause_ipi = xive_cause_ipi;
/* Register the IPI */
- xive_init_ipis();
+ ret = xive_init_ipis();
+ if (ret < 0)
+ return ret;
/* Allocate and setup IPI for the boot CPU */
- xive_setup_cpu_ipi(smp_processor_id());
+ return xive_setup_cpu_ipi(smp_processor_id());
}
#endif /* CONFIG_SMP */
diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h
index efa1b3519b23..25a1385dcd3f 100644
--- a/arch/riscv/include/asm/processor.h
+++ b/arch/riscv/include/asm/processor.h
@@ -102,6 +102,8 @@ struct thread_struct {
unsigned long s[12]; /* s[0]: frame pointer */
struct __riscv_d_ext_state fstate;
unsigned long bad_cause;
+ unsigned long envcfg;
+ unsigned long sum;
u32 riscv_v_flags;
u32 vstate_ctrl;
struct __riscv_v_ext_state vstate;
diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h
index 7594df37cc9f..9685cd85e57c 100644
--- a/arch/riscv/include/asm/switch_to.h
+++ b/arch/riscv/include/asm/switch_to.h
@@ -70,6 +70,13 @@ static __always_inline bool has_fpu(void) { return false; }
#define __switch_to_fpu(__prev, __next) do { } while (0)
#endif
+static inline void __switch_to_envcfg(struct task_struct *next)
+{
+ asm volatile (ALTERNATIVE("nop", "csrw " __stringify(CSR_ENVCFG) ", %0",
+ 0, RISCV_ISA_EXT_XLINUXENVCFG, 1)
+ :: "r" (next->thread.envcfg) : "memory");
+}
+
extern struct task_struct *__switch_to(struct task_struct *,
struct task_struct *);
@@ -103,6 +110,7 @@ do { \
__switch_to_vector(__prev, __next); \
if (switch_to_should_flush_icache(__next)) \
local_flush_icache_all(); \
+ __switch_to_envcfg(__next); \
((last) = __switch_to(__prev, __next)); \
} while (0)
diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile
index 0678650817d4..7328e62a7b8e 100644
--- a/arch/riscv/kernel/Makefile
+++ b/arch/riscv/kernel/Makefile
@@ -3,12 +3,6 @@
# Makefile for the RISC-V Linux kernel
#
-ifdef CONFIG_FTRACE
-CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_patch.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_sbi.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE)
-endif
CFLAGS_syscall_table.o += $(call cc-disable-warning, override-init)
CFLAGS_compat_syscall_table.o += $(call cc-disable-warning, override-init)
@@ -24,7 +18,6 @@ CFLAGS_sbi_ecall.o := -mcmodel=medany
ifdef CONFIG_FTRACE
CFLAGS_REMOVE_alternative.o = $(CC_FLAGS_FTRACE)
CFLAGS_REMOVE_cpufeature.o = $(CC_FLAGS_FTRACE)
-CFLAGS_REMOVE_sbi_ecall.o = $(CC_FLAGS_FTRACE)
endif
ifdef CONFIG_RELOCATABLE
CFLAGS_alternative.o += -fno-pie
@@ -43,6 +36,14 @@ CFLAGS_sbi_ecall.o += -D__NO_FORTIFY
endif
endif
+ifdef CONFIG_FTRACE
+CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_patch.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_sbi.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_sbi_ecall.o = $(CC_FLAGS_FTRACE)
+endif
+
extra-y += vmlinux.lds
obj-y += head.o
diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c
index dfed9986b45e..1d93658825f7 100644
--- a/arch/riscv/kernel/asm-offsets.c
+++ b/arch/riscv/kernel/asm-offsets.c
@@ -35,6 +35,7 @@ void asm_offsets(void)
OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]);
OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]);
OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]);
+ OFFSET(TASK_THREAD_SUM, task_struct, thread.sum);
OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu);
OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags);
@@ -348,6 +349,10 @@ void asm_offsets(void)
offsetof(struct task_struct, thread.s[11])
- offsetof(struct task_struct, thread.ra)
);
+ DEFINE(TASK_THREAD_SUM_RA,
+ offsetof(struct task_struct, thread.sum)
+ - offsetof(struct task_struct, thread.ra)
+ );
DEFINE(TASK_THREAD_F0_F0,
offsetof(struct task_struct, thread.fstate.f[0])
diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c
index 472de1ed1ad7..248167b802eb 100644
--- a/arch/riscv/kernel/cpufeature.c
+++ b/arch/riscv/kernel/cpufeature.c
@@ -28,6 +28,8 @@
#define NUM_ALPHA_EXTS ('z' - 'a' + 1)
+static bool any_cpu_has_zicboz;
+
unsigned long elf_hwcap __read_mostly;
/* Host ISA bitmap */
@@ -98,6 +100,7 @@ static int riscv_ext_zicboz_validate(const struct riscv_isa_ext_data *data,
pr_err("Zicboz disabled as cboz-block-size present, but is not a power-of-2\n");
return -EINVAL;
}
+ any_cpu_has_zicboz = true;
return 0;
}
@@ -920,8 +923,10 @@ unsigned long riscv_get_elf_hwcap(void)
void riscv_user_isa_enable(void)
{
- if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_ZICBOZ))
- csr_set(CSR_ENVCFG, ENVCFG_CBZE);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICBOZ))
+ current->thread.envcfg |= ENVCFG_CBZE;
+ else if (any_cpu_has_zicboz)
+ pr_warn_once("Zicboz disabled as it is unavailable on some harts\n");
}
#ifdef CONFIG_RISCV_ALTERNATIVE
diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
index fee140a5c211..d8f74743134a 100644
--- a/arch/riscv/kernel/entry.S
+++ b/arch/riscv/kernel/entry.S
@@ -402,9 +402,18 @@ SYM_FUNC_START(__switch_to)
REG_S s9, TASK_THREAD_S9_RA(a3)
REG_S s10, TASK_THREAD_S10_RA(a3)
REG_S s11, TASK_THREAD_S11_RA(a3)
+
+ /* save the user space access flag */
+ csrr s0, CSR_STATUS
+ REG_S s0, TASK_THREAD_SUM_RA(a3)
+
/* Save the kernel shadow call stack pointer */
scs_save_current
/* Restore context from next->thread */
+ REG_L s0, TASK_THREAD_SUM_RA(a4)
+ li s1, SR_SUM
+ and s0, s0, s1
+ csrs CSR_STATUS, s0
REG_L ra, TASK_THREAD_RA_RA(a4)
REG_L sp, TASK_THREAD_SP_RA(a4)
REG_L s0, TASK_THREAD_S0_RA(a4)
diff --git a/arch/riscv/kernel/patch.c b/arch/riscv/kernel/patch.c
index 34ef522f07a8..9c9282f8a3a5 100644
--- a/arch/riscv/kernel/patch.c
+++ b/arch/riscv/kernel/patch.c
@@ -42,19 +42,20 @@ static inline bool is_kernel_exittext(uintptr_t addr)
static __always_inline void *patch_map(void *addr, const unsigned int fixmap)
{
uintptr_t uintaddr = (uintptr_t) addr;
- struct page *page;
+ phys_addr_t phys;
- if (core_kernel_text(uintaddr) || is_kernel_exittext(uintaddr))
- page = phys_to_page(__pa_symbol(addr));
- else if (IS_ENABLED(CONFIG_STRICT_MODULE_RWX))
- page = vmalloc_to_page(addr);
- else
- return addr;
+ if (core_kernel_text(uintaddr) || is_kernel_exittext(uintaddr)) {
+ phys = __pa_symbol(addr);
+ } else if (IS_ENABLED(CONFIG_EXECMEM)) {
+ struct page *page = vmalloc_to_page(addr);
- BUG_ON(!page);
+ BUG_ON(!page);
+ phys = page_to_phys(page) + offset_in_page(addr);
+ } else {
+ return addr;
+ }
- return (void *)set_fixmap_offset(fixmap, page_to_phys(page) +
- offset_in_page(addr));
+ return (void *)set_fixmap_offset(fixmap, phys);
}
static void patch_unmap(int fixmap)
diff --git a/arch/riscv/kernel/suspend.c b/arch/riscv/kernel/suspend.c
index c8cec0cc5833..9a8a0dc035b2 100644
--- a/arch/riscv/kernel/suspend.c
+++ b/arch/riscv/kernel/suspend.c
@@ -14,7 +14,7 @@
void suspend_save_csrs(struct suspend_context *context)
{
- if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_XLINUXENVCFG))
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_XLINUXENVCFG))
context->envcfg = csr_read(CSR_ENVCFG);
context->tvec = csr_read(CSR_TVEC);
context->ie = csr_read(CSR_IE);
@@ -37,7 +37,7 @@ void suspend_save_csrs(struct suspend_context *context)
void suspend_restore_csrs(struct suspend_context *context)
{
csr_write(CSR_SCRATCH, 0);
- if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_XLINUXENVCFG))
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_XLINUXENVCFG))
csr_write(CSR_ENVCFG, context->envcfg);
csr_write(CSR_TVEC, context->tvec);
csr_write(CSR_IE, context->ie);
diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c
index f3427f6de608..fed55b469794 100644
--- a/arch/riscv/kvm/main.c
+++ b/arch/riscv/kvm/main.c
@@ -117,8 +117,8 @@ module_init(riscv_kvm_init);
static void __exit riscv_kvm_exit(void)
{
- kvm_riscv_aia_exit();
-
kvm_exit();
+
+ kvm_riscv_aia_exit();
}
module_exit(riscv_kvm_exit);
diff --git a/arch/riscv/kvm/vcpu_vector.c b/arch/riscv/kvm/vcpu_vector.c
index 8454c1c3655a..eaf88c20508a 100644
--- a/arch/riscv/kvm/vcpu_vector.c
+++ b/arch/riscv/kvm/vcpu_vector.c
@@ -79,8 +79,11 @@ int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu,
cntx->vector.vlenb = riscv_v_vsize / 32;
vcpu->arch.host_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
- if (!vcpu->arch.host_context.vector.datap)
+ if (!vcpu->arch.host_context.vector.datap) {
+ kfree(vcpu->arch.guest_context.vector.datap);
+ vcpu->arch.guest_context.vector.datap = NULL;
return -ENOMEM;
+ }
return 0;
}
diff --git a/arch/riscv/kvm/vmid.c b/arch/riscv/kvm/vmid.c
index ddc98714ce8e..38f30f5a608c 100644
--- a/arch/riscv/kvm/vmid.c
+++ b/arch/riscv/kvm/vmid.c
@@ -26,7 +26,7 @@ void __init kvm_riscv_gstage_vmid_detect(void)
/* Figure-out number of VMID bits in HW */
old = csr_read(CSR_HGATP);
- csr_write(CSR_HGATP, old | HGATP_VMID);
+ csr_write(CSR_HGATP, (kvm_riscv_gstage_mode() << HGATP_MODE_SHIFT) | HGATP_VMID);
vmid_bits = csr_read(CSR_HGATP);
vmid_bits = (vmid_bits & HGATP_VMID) >> HGATP_VMID_SHIFT;
vmid_bits = fls_long(vmid_bits);
diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h
index 9725586f4259..b458957a383e 100644
--- a/arch/s390/include/asm/gmap.h
+++ b/arch/s390/include/asm/gmap.h
@@ -134,8 +134,6 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt,
int fake);
int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt,
int fake);
-int gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long saddr,
- unsigned long *pgt, int *dat_protection, int *fake);
int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte);
void gmap_register_pte_notifier(struct gmap_notifier *);
@@ -152,6 +150,7 @@ int s390_replace_asce(struct gmap *gmap);
void s390_uv_destroy_pfns(unsigned long count, unsigned long *pfns);
int __s390_uv_destroy_range(struct mm_struct *mm, unsigned long start,
unsigned long end, bool interruptible);
+unsigned long *gmap_table_walk(struct gmap *gmap, unsigned long gaddr, int level);
/**
* s390_uv_destroy_range - Destroy a range of pages in the given mm.
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 8e77afbed58e..603b56bfccd3 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -527,6 +527,9 @@ struct kvm_vcpu_stat {
#define PGM_REGION_FIRST_TRANS 0x39
#define PGM_REGION_SECOND_TRANS 0x3a
#define PGM_REGION_THIRD_TRANS 0x3b
+#define PGM_SECURE_STORAGE_ACCESS 0x3d
+#define PGM_NON_SECURE_STORAGE_ACCESS 0x3e
+#define PGM_SECURE_STORAGE_VIOLATION 0x3f
#define PGM_MONITOR 0x40
#define PGM_PER 0x80
#define PGM_CRYPTO_OPERATION 0x119
diff --git a/arch/s390/include/asm/nmi.h b/arch/s390/include/asm/nmi.h
index 227466ce9e41..da5a98183cb6 100644
--- a/arch/s390/include/asm/nmi.h
+++ b/arch/s390/include/asm/nmi.h
@@ -22,6 +22,7 @@
#define MCCK_CODE_SYSTEM_DAMAGE BIT(63)
#define MCCK_CODE_EXT_DAMAGE BIT(63 - 5)
#define MCCK_CODE_CP BIT(63 - 9)
+#define MCCK_CODE_CK BIT(63 - 11)
#define MCCK_CODE_STG_ERROR BIT(63 - 16)
#define MCCK_CODE_STG_KEY_ERROR BIT(63 - 18)
#define MCCK_CODE_STG_DEGRAD BIT(63 - 19)
@@ -33,6 +34,8 @@
#define MCCK_CODE_FC_VALID BIT(63 - 43)
#define MCCK_CODE_CPU_TIMER_VALID BIT(63 - 46)
+#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE | MCCK_CODE_CK)
+
#ifndef __ASSEMBLY__
union mci {
diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index edf346a14820..1b919f1e0ff7 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -215,6 +215,7 @@ extern struct airq_iv *zpci_aif_sbv;
struct zpci_dev *zpci_create_device(u32 fid, u32 fh, enum zpci_state state);
int zpci_add_device(struct zpci_dev *zdev);
int zpci_enable_device(struct zpci_dev *);
+int zpci_reenable_device(struct zpci_dev *zdev);
int zpci_disable_device(struct zpci_dev *);
int zpci_scan_configured_device(struct zpci_dev *zdev, u32 fh);
int zpci_deconfigure_device(struct zpci_dev *zdev);
@@ -243,6 +244,7 @@ void update_uid_checking(bool new);
/* IOMMU Interface */
int zpci_init_iommu(struct zpci_dev *zdev);
void zpci_destroy_iommu(struct zpci_dev *zdev);
+int zpci_iommu_register_ioat(struct zpci_dev *zdev, u8 *status);
#ifdef CONFIG_PCI
static inline bool zpci_use_mio(struct zpci_dev *zdev)
diff --git a/arch/s390/include/asm/pci_insn.h b/arch/s390/include/asm/pci_insn.h
index e5f57cfe1d45..025c6dcbf893 100644
--- a/arch/s390/include/asm/pci_insn.h
+++ b/arch/s390/include/asm/pci_insn.h
@@ -16,11 +16,11 @@
#define ZPCI_PCI_ST_FUNC_NOT_AVAIL 40
#define ZPCI_PCI_ST_ALREADY_IN_RQ_STATE 44
-/* Load/Store return codes */
-#define ZPCI_PCI_LS_OK 0
-#define ZPCI_PCI_LS_ERR 1
-#define ZPCI_PCI_LS_BUSY 2
-#define ZPCI_PCI_LS_INVAL_HANDLE 3
+/* PCI instruction condition codes */
+#define ZPCI_CC_OK 0
+#define ZPCI_CC_ERR 1
+#define ZPCI_CC_BUSY 2
+#define ZPCI_CC_INVAL_HANDLE 3
/* Load/Store address space identifiers */
#define ZPCI_PCIAS_MEMIO_0 0
diff --git a/arch/s390/include/asm/processor.h b/arch/s390/include/asm/processor.h
index ef622c3f88e5..6054d157a8f9 100644
--- a/arch/s390/include/asm/processor.h
+++ b/arch/s390/include/asm/processor.h
@@ -39,6 +39,7 @@
#include <asm/runtime_instr.h>
#include <asm/irqflags.h>
#include <asm/alternative.h>
+#include <asm/fault.h>
struct pcpu {
unsigned long ec_mask; /* bit mask for ec_xxx functions */
@@ -186,10 +187,8 @@ struct thread_struct {
unsigned long hardirq_timer; /* task cputime in hardirq context */
unsigned long softirq_timer; /* task cputime in softirq context */
const sys_call_ptr_t *sys_call_table; /* system call table address */
- unsigned long gmap_addr; /* address of last gmap fault. */
- unsigned int gmap_write_flag; /* gmap fault write indication */
+ union teid gmap_teid; /* address and flags of last gmap fault */
unsigned int gmap_int_code; /* int code of last gmap fault */
- unsigned int gmap_pfault; /* signal of a pending guest pfault */
int ufpu_flags; /* user fpu flags */
int kfpu_flags; /* kernel fpu flags */
diff --git a/arch/s390/include/asm/ptrace.h b/arch/s390/include/asm/ptrace.h
index 2ad9324f6338..788bc4467445 100644
--- a/arch/s390/include/asm/ptrace.h
+++ b/arch/s390/include/asm/ptrace.h
@@ -14,11 +14,13 @@
#define PIF_SYSCALL 0 /* inside a system call */
#define PIF_EXECVE_PGSTE_RESTART 1 /* restart execve for PGSTE binaries */
#define PIF_SYSCALL_RET_SET 2 /* return value was set via ptrace */
+#define PIF_GUEST_FAULT 3 /* indicates program check in sie64a */
#define PIF_FTRACE_FULL_REGS 4 /* all register contents valid (ftrace) */
#define _PIF_SYSCALL BIT(PIF_SYSCALL)
#define _PIF_EXECVE_PGSTE_RESTART BIT(PIF_EXECVE_PGSTE_RESTART)
#define _PIF_SYSCALL_RET_SET BIT(PIF_SYSCALL_RET_SET)
+#define _PIF_GUEST_FAULT BIT(PIF_GUEST_FAULT)
#define _PIF_FTRACE_FULL_REGS BIT(PIF_FTRACE_FULL_REGS)
#define PSW32_MASK_PER _AC(0x40000000, UL)
diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 153d93468b77..7e82dfb3c7aa 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -476,12 +476,12 @@ static inline int is_prot_virt_host(void)
}
int uv_pin_shared(unsigned long paddr);
-int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb);
-int gmap_destroy_page(struct gmap *gmap, unsigned long gaddr);
int uv_destroy_folio(struct folio *folio);
int uv_destroy_pte(pte_t pte);
int uv_convert_from_secure_pte(pte_t pte);
-int gmap_convert_to_secure(struct gmap *gmap, unsigned long gaddr);
+int make_hva_secure(struct mm_struct *mm, unsigned long hva, struct uv_cb_header *uvcb);
+int uv_convert_from_secure(unsigned long paddr);
+int uv_convert_from_secure_folio(struct folio *folio);
void setup_uv(void);
diff --git a/arch/s390/kernel/asm-offsets.c b/arch/s390/kernel/asm-offsets.c
index 8fd98a0f999f..1848d5123000 100644
--- a/arch/s390/kernel/asm-offsets.c
+++ b/arch/s390/kernel/asm-offsets.c
@@ -14,7 +14,6 @@
#include <linux/purgatory.h>
#include <linux/pgtable.h>
#include <linux/ftrace.h>
-#include <asm/gmap.h>
#include <asm/stacktrace.h>
int main(void)
@@ -162,7 +161,6 @@ int main(void)
OFFSET(__LC_PGM_TDB, lowcore, pgm_tdb);
BLANK();
/* gmap/sie offsets */
- OFFSET(__GMAP_ASCE, gmap, asce);
OFFSET(__SIE_PROG0C, kvm_s390_sie_block, prog0c);
OFFSET(__SIE_PROG20, kvm_s390_sie_block, prog20);
/* kexec_sha_region */
diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index 8f5120802115..439950ddc416 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -222,17 +222,6 @@ SYM_FUNC_START(__sie64a)
lctlg %c1,%c1,__LC_KERNEL_ASCE(%r14) # load primary asce
lg %r14,__LC_CURRENT(%r14)
mvi __TI_sie(%r14),0
-# some program checks are suppressing. C code (e.g. do_protection_exception)
-# will rewind the PSW by the ILC, which is often 4 bytes in case of SIE. There
-# are some corner cases (e.g. runtime instrumentation) where ILC is unpredictable.
-# Other instructions between __sie64a and .Lsie_done should not cause program
-# interrupts. So lets use 3 nops as a landing pad for all possible rewinds.
-.Lrewind_pad6:
- nopr 7
-.Lrewind_pad4:
- nopr 7
-.Lrewind_pad2:
- nopr 7
SYM_INNER_LABEL(sie_exit, SYM_L_GLOBAL)
lg %r14,__SF_SIE_SAVEAREA(%r15) # load guest register save area
stmg %r0,%r13,0(%r14) # save guest gprs 0-13
@@ -244,15 +233,6 @@ SYM_INNER_LABEL(sie_exit, SYM_L_GLOBAL)
lmg %r6,%r14,__SF_GPRS(%r15) # restore kernel registers
lg %r2,__SF_SIE_REASON(%r15) # return exit reason code
BR_EX %r14
-.Lsie_fault:
- lghi %r14,-EFAULT
- stg %r14,__SF_SIE_REASON(%r15) # set exit reason code
- j sie_exit
-
- EX_TABLE(.Lrewind_pad6,.Lsie_fault)
- EX_TABLE(.Lrewind_pad4,.Lsie_fault)
- EX_TABLE(.Lrewind_pad2,.Lsie_fault)
- EX_TABLE(sie_exit,.Lsie_fault)
SYM_FUNC_END(__sie64a)
EXPORT_SYMBOL(__sie64a)
EXPORT_SYMBOL(sie_exit)
@@ -328,13 +308,21 @@ SYM_CODE_START(pgm_check_handler)
GET_LC %r13
stpt __LC_SYS_ENTER_TIMER(%r13)
BPOFF
- lgr %r10,%r15
lmg %r8,%r9,__LC_PGM_OLD_PSW(%r13)
+ xgr %r10,%r10
tmhh %r8,0x0001 # coming from user space?
jno .Lpgm_skip_asce
lctlg %c1,%c1,__LC_KERNEL_ASCE(%r13)
j 3f # -> fault in user space
.Lpgm_skip_asce:
+#if IS_ENABLED(CONFIG_KVM)
+ lg %r11,__LC_CURRENT(%r13)
+ tm __TI_sie(%r11),0xff
+ jz 1f
+ BPENTER __SF_SIE_FLAGS(%r15),_TIF_ISOLATE_BP_GUEST
+ SIEEXIT __SF_SIE_CONTROL(%r15),%r13
+ lghi %r10,_PIF_GUEST_FAULT
+#endif
1: tmhh %r8,0x4000 # PER bit set in old PSW ?
jnz 2f # -> enabled, can't be a double fault
tm __LC_PGM_ILC+3(%r13),0x80 # check for per exception
@@ -345,21 +333,12 @@ SYM_CODE_START(pgm_check_handler)
CHECK_VMAP_STACK __LC_SAVE_AREA,%r13,4f
3: lg %r15,__LC_KERNEL_STACK(%r13)
4: la %r11,STACK_FRAME_OVERHEAD(%r15)
- xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
+ stg %r10,__PT_FLAGS(%r11)
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
stmg %r0,%r7,__PT_R0(%r11)
mvc __PT_R8(64,%r11),__LC_SAVE_AREA(%r13)
mvc __PT_LAST_BREAK(8,%r11),__LC_PGM_LAST_BREAK(%r13)
- stctg %c1,%c1,__PT_CR1(%r11)
-#if IS_ENABLED(CONFIG_KVM)
- ltg %r12,__LC_GMAP(%r13)
- jz 5f
- clc __GMAP_ASCE(8,%r12), __PT_CR1(%r11)
- jne 5f
- BPENTER __SF_SIE_FLAGS(%r10),_TIF_ISOLATE_BP_GUEST
- SIEEXIT __SF_SIE_CONTROL(%r10),%r13
-#endif
-5: stmg %r8,%r9,__PT_PSW(%r11)
+ stmg %r8,%r9,__PT_PSW(%r11)
# clear user controlled registers to prevent speculative use
xgr %r0,%r0
xgr %r1,%r1
@@ -368,6 +347,7 @@ SYM_CODE_START(pgm_check_handler)
xgr %r5,%r5
xgr %r6,%r6
xgr %r7,%r7
+ xgr %r12,%r12
lgr %r2,%r11
brasl %r14,__do_pgm_check
tmhh %r8,0x0001 # returning to user space?
diff --git a/arch/s390/kernel/nmi.c b/arch/s390/kernel/nmi.c
index fbd218b6fc8e..e8d65f87a17d 100644
--- a/arch/s390/kernel/nmi.c
+++ b/arch/s390/kernel/nmi.c
@@ -279,8 +279,7 @@ static void notrace s390_backup_mcck_info(struct pt_regs *regs)
sie_page = container_of(sie_block, struct sie_page, sie_block);
mcck_backup = &sie_page->mcck_info;
- mcck_backup->mcic = get_lowcore()->mcck_interruption_code &
- ~(MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE);
+ mcck_backup->mcic = get_lowcore()->mcck_interruption_code & ~MCCK_CODE_NO_GUEST;
mcck_backup->ext_damage_code = get_lowcore()->external_damage_code;
mcck_backup->failing_storage_address = get_lowcore()->failing_storage_address;
}
@@ -292,8 +291,6 @@ NOKPROBE_SYMBOL(s390_backup_mcck_info);
#define ED_STP_ISLAND 6 /* External damage STP island check */
#define ED_STP_SYNC 7 /* External damage STP sync check */
-#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE)
-
/*
* machine check handler.
*/
diff --git a/arch/s390/kernel/traps.c b/arch/s390/kernel/traps.c
index 908bae849843..b746213d3110 100644
--- a/arch/s390/kernel/traps.c
+++ b/arch/s390/kernel/traps.c
@@ -31,6 +31,7 @@
#include <asm/asm-extable.h>
#include <asm/vtime.h>
#include <asm/fpu.h>
+#include <asm/fault.h>
#include "entry.h"
static inline void __user *get_trap_ip(struct pt_regs *regs)
@@ -317,9 +318,24 @@ void noinstr __do_pgm_check(struct pt_regs *regs)
struct lowcore *lc = get_lowcore();
irqentry_state_t state;
unsigned int trapnr;
+ union teid teid;
+ teid.val = lc->trans_exc_code;
regs->int_code = lc->pgm_int_code;
- regs->int_parm_long = lc->trans_exc_code;
+ regs->int_parm_long = teid.val;
+
+ /*
+ * In case of a guest fault, short-circuit the fault handler and return.
+ * This way the sie64a() function will return 0; fault address and
+ * other relevant bits are saved in current->thread.gmap_teid, and
+ * the fault number in current->thread.gmap_int_code. KVM will be
+ * able to use this information to handle the fault.
+ */
+ if (test_pt_regs_flag(regs, PIF_GUEST_FAULT)) {
+ current->thread.gmap_teid.val = regs->int_parm_long;
+ current->thread.gmap_int_code = regs->int_code & 0xffff;
+ return;
+ }
state = irqentry_enter(regs);
@@ -408,8 +424,8 @@ static void (*pgm_check_table[128])(struct pt_regs *regs) = {
[0x3b] = do_dat_exception,
[0x3c] = default_trap_handler,
[0x3d] = do_secure_storage_access,
- [0x3e] = do_non_secure_storage_access,
- [0x3f] = do_secure_storage_violation,
+ [0x3e] = default_trap_handler,
+ [0x3f] = default_trap_handler,
[0x40] = monitor_event_exception,
[0x41 ... 0x7f] = default_trap_handler,
};
@@ -420,5 +436,3 @@ static void (*pgm_check_table[128])(struct pt_regs *regs) = {
__stringify(default_trap_handler))
COND_TRAP(do_secure_storage_access);
-COND_TRAP(do_non_secure_storage_access);
-COND_TRAP(do_secure_storage_violation);
diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
index 9646f773208a..6cf5025c1f58 100644
--- a/arch/s390/kernel/uv.c
+++ b/arch/s390/kernel/uv.c
@@ -19,19 +19,6 @@
#include <asm/sections.h>
#include <asm/uv.h>
-#if !IS_ENABLED(CONFIG_KVM)
-unsigned long __gmap_translate(struct gmap *gmap, unsigned long gaddr)
-{
- return 0;
-}
-
-int gmap_fault(struct gmap *gmap, unsigned long gaddr,
- unsigned int fault_flags)
-{
- return 0;
-}
-#endif
-
/* the bootdata_preserved fields come from ones in arch/s390/boot/uv.c */
int __bootdata_preserved(prot_virt_guest);
EXPORT_SYMBOL(prot_virt_guest);
@@ -159,6 +146,7 @@ int uv_destroy_folio(struct folio *folio)
folio_put(folio);
return rc;
}
+EXPORT_SYMBOL(uv_destroy_folio);
/*
* The present PTE still indirectly holds a folio reference through the mapping.
@@ -175,7 +163,7 @@ int uv_destroy_pte(pte_t pte)
*
* @paddr: Absolute host address of page to be exported
*/
-static int uv_convert_from_secure(unsigned long paddr)
+int uv_convert_from_secure(unsigned long paddr)
{
struct uv_cb_cfs uvcb = {
.header.cmd = UVC_CMD_CONV_FROM_SEC_STOR,
@@ -187,11 +175,12 @@ static int uv_convert_from_secure(unsigned long paddr)
return -EINVAL;
return 0;
}
+EXPORT_SYMBOL_GPL(uv_convert_from_secure);
/*
* The caller must already hold a reference to the folio.
*/
-static int uv_convert_from_secure_folio(struct folio *folio)
+int uv_convert_from_secure_folio(struct folio *folio)
{
int rc;
@@ -206,6 +195,7 @@ static int uv_convert_from_secure_folio(struct folio *folio)
folio_put(folio);
return rc;
}
+EXPORT_SYMBOL_GPL(uv_convert_from_secure_folio);
/*
* The present PTE still indirectly holds a folio reference through the mapping.
@@ -216,6 +206,39 @@ int uv_convert_from_secure_pte(pte_t pte)
return uv_convert_from_secure_folio(pfn_folio(pte_pfn(pte)));
}
+/**
+ * should_export_before_import - Determine whether an export is needed
+ * before an import-like operation
+ * @uvcb: the Ultravisor control block of the UVC to be performed
+ * @mm: the mm of the process
+ *
+ * Returns whether an export is needed before every import-like operation.
+ * This is needed for shared pages, which don't trigger a secure storage
+ * exception when accessed from a different guest.
+ *
+ * Although considered as one, the Unpin Page UVC is not an actual import,
+ * so it is not affected.
+ *
+ * No export is needed also when there is only one protected VM, because the
+ * page cannot belong to the wrong VM in that case (there is no "other VM"
+ * it can belong to).
+ *
+ * Return: true if an export is needed before every import, otherwise false.
+ */
+static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_struct *mm)
+{
+ /*
+ * The misc feature indicates, among other things, that importing a
+ * shared page from a different protected VM will automatically also
+ * transfer its ownership.
+ */
+ if (uv_has_feature(BIT_UV_FEAT_MISC))
+ return false;
+ if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
+ return false;
+ return atomic_read(&mm->context.protected_count) > 1;
+}
+
/*
* Calculate the expected ref_count for a folio that would otherwise have no
* further pins. This was cribbed from similar functions in other places in
@@ -237,13 +260,31 @@ static int expected_folio_refs(struct folio *folio)
return res;
}
-static int make_folio_secure(struct folio *folio, struct uv_cb_header *uvcb)
+/**
+ * __make_folio_secure() - make a folio secure
+ * @folio: the folio to make secure
+ * @uvcb: the uvcb that describes the UVC to be used
+ *
+ * The folio @folio will be made secure if possible, @uvcb will be passed
+ * as-is to the UVC.
+ *
+ * Return: 0 on success;
+ * -EBUSY if the folio is in writeback or has too many references;
+ * -EAGAIN if the UVC needs to be attempted again;
+ * -ENXIO if the address is not mapped;
+ * -EINVAL if the UVC failed for other reasons.
+ *
+ * Context: The caller must hold exactly one extra reference on the folio
+ * (it's the same logic as split_folio()), and the folio must be
+ * locked.
+ */
+static int __make_folio_secure(struct folio *folio, struct uv_cb_header *uvcb)
{
int expected, cc = 0;
if (folio_test_writeback(folio))
- return -EAGAIN;
- expected = expected_folio_refs(folio);
+ return -EBUSY;
+ expected = expected_folio_refs(folio) + 1;
if (!folio_ref_freeze(folio, expected))
return -EBUSY;
set_bit(PG_arch_1, &folio->flags);
@@ -268,250 +309,100 @@ static int make_folio_secure(struct folio *folio, struct uv_cb_header *uvcb)
return uvcb->rc == 0x10a ? -ENXIO : -EINVAL;
}
-/**
- * should_export_before_import - Determine whether an export is needed
- * before an import-like operation
- * @uvcb: the Ultravisor control block of the UVC to be performed
- * @mm: the mm of the process
- *
- * Returns whether an export is needed before every import-like operation.
- * This is needed for shared pages, which don't trigger a secure storage
- * exception when accessed from a different guest.
- *
- * Although considered as one, the Unpin Page UVC is not an actual import,
- * so it is not affected.
- *
- * No export is needed also when there is only one protected VM, because the
- * page cannot belong to the wrong VM in that case (there is no "other VM"
- * it can belong to).
- *
- * Return: true if an export is needed before every import, otherwise false.
- */
-static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_struct *mm)
+static int make_folio_secure(struct mm_struct *mm, struct folio *folio, struct uv_cb_header *uvcb)
{
- /*
- * The misc feature indicates, among other things, that importing a
- * shared page from a different protected VM will automatically also
- * transfer its ownership.
- */
- if (uv_has_feature(BIT_UV_FEAT_MISC))
- return false;
- if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
- return false;
- return atomic_read(&mm->context.protected_count) > 1;
-}
+ int rc;
-/*
- * Drain LRU caches: the local one on first invocation and the ones of all
- * CPUs on successive invocations. Returns "true" on the first invocation.
- */
-static bool drain_lru(bool *drain_lru_called)
-{
- /*
- * If we have tried a local drain and the folio refcount
- * still does not match our expected safe value, try with a
- * system wide drain. This is needed if the pagevecs holding
- * the page are on a different CPU.
- */
- if (*drain_lru_called) {
- lru_add_drain_all();
- /* We give up here, don't retry immediately. */
- return false;
- }
- /*
- * We are here if the folio refcount does not match the
- * expected safe value. The main culprits are usually
- * pagevecs. With lru_add_drain() we drain the pagevecs
- * on the local CPU so that hopefully the refcount will
- * reach the expected safe value.
- */
- lru_add_drain();
- *drain_lru_called = true;
- /* The caller should try again immediately */
- return true;
+ if (!folio_trylock(folio))
+ return -EAGAIN;
+ if (should_export_before_import(uvcb, mm))
+ uv_convert_from_secure(folio_to_phys(folio));
+ rc = __make_folio_secure(folio, uvcb);
+ folio_unlock(folio);
+
+ return rc;
}
-/*
- * Requests the Ultravisor to make a page accessible to a guest.
- * If it's brought in the first time, it will be cleared. If
- * it has been exported before, it will be decrypted and integrity
- * checked.
+/**
+ * s390_wiggle_split_folio() - try to drain extra references to a folio and optionally split.
+ * @mm: the mm containing the folio to work on
+ * @folio: the folio
+ * @split: whether to split a large folio
+ *
+ * Context: Must be called while holding an extra reference to the folio;
+ * the mm lock should not be held.
+ * Return: 0 if the folio was split successfully;
+ * -EAGAIN if the folio was not split successfully but another attempt
+ * can be made, or if @split was set to false;
+ * -EINVAL in case of other errors. See split_folio().
*/
-int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb)
+static int s390_wiggle_split_folio(struct mm_struct *mm, struct folio *folio, bool split)
{
- struct vm_area_struct *vma;
- bool drain_lru_called = false;
- spinlock_t *ptelock;
- unsigned long uaddr;
- struct folio *folio;
- pte_t *ptep;
int rc;
-again:
- rc = -EFAULT;
- mmap_read_lock(gmap->mm);
-
- uaddr = __gmap_translate(gmap, gaddr);
- if (IS_ERR_VALUE(uaddr))
- goto out;
- vma = vma_lookup(gmap->mm, uaddr);
- if (!vma)
- goto out;
- /*
- * Secure pages cannot be huge and userspace should not combine both.
- * In case userspace does it anyway this will result in an -EFAULT for
- * the unpack. The guest is thus never reaching secure mode. If
- * userspace is playing dirty tricky with mapping huge pages later
- * on this will result in a segmentation fault.
- */
- if (is_vm_hugetlb_page(vma))
- goto out;
-
- rc = -ENXIO;
- ptep = get_locked_pte(gmap->mm, uaddr, &ptelock);
- if (!ptep)
- goto out;
- if (pte_present(*ptep) && !(pte_val(*ptep) & _PAGE_INVALID) && pte_write(*ptep)) {
- folio = page_folio(pte_page(*ptep));
- rc = -EAGAIN;
- if (folio_test_large(folio)) {
- rc = -E2BIG;
- } else if (folio_trylock(folio)) {
- if (should_export_before_import(uvcb, gmap->mm))
- uv_convert_from_secure(PFN_PHYS(folio_pfn(folio)));
- rc = make_folio_secure(folio, uvcb);
- folio_unlock(folio);
- }
-
- /*
- * Once we drop the PTL, the folio may get unmapped and
- * freed immediately. We need a temporary reference.
- */
- if (rc == -EAGAIN || rc == -E2BIG)
- folio_get(folio);
- }
- pte_unmap_unlock(ptep, ptelock);
-out:
- mmap_read_unlock(gmap->mm);
-
- switch (rc) {
- case -E2BIG:
+ lockdep_assert_not_held(&mm->mmap_lock);
+ folio_wait_writeback(folio);
+ lru_add_drain_all();
+ if (split) {
folio_lock(folio);
rc = split_folio(folio);
folio_unlock(folio);
- folio_put(folio);
-
- switch (rc) {
- case 0:
- /* Splitting succeeded, try again immediately. */
- goto again;
- case -EAGAIN:
- /* Additional folio references. */
- if (drain_lru(&drain_lru_called))
- goto again;
- return -EAGAIN;
- case -EBUSY:
- /* Unexpected race. */
- return -EAGAIN;
- }
- WARN_ON_ONCE(1);
- return -ENXIO;
- case -EAGAIN:
- /*
- * If we are here because the UVC returned busy or partial
- * completion, this is just a useless check, but it is safe.
- */
- folio_wait_writeback(folio);
- folio_put(folio);
- return -EAGAIN;
- case -EBUSY:
- /* Additional folio references. */
- if (drain_lru(&drain_lru_called))
- goto again;
- return -EAGAIN;
- case -ENXIO:
- if (gmap_fault(gmap, gaddr, FAULT_FLAG_WRITE))
- return -EFAULT;
- return -EAGAIN;
+
+ if (rc != -EBUSY)
+ return rc;
}
- return rc;
+ return -EAGAIN;
}
-EXPORT_SYMBOL_GPL(gmap_make_secure);
-int gmap_convert_to_secure(struct gmap *gmap, unsigned long gaddr)
-{
- struct uv_cb_cts uvcb = {
- .header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
- .header.len = sizeof(uvcb),
- .guest_handle = gmap->guest_handle,
- .gaddr = gaddr,
- };
-
- return gmap_make_secure(gmap, gaddr, &uvcb);
-}
-EXPORT_SYMBOL_GPL(gmap_convert_to_secure);
-
-/**
- * gmap_destroy_page - Destroy a guest page.
- * @gmap: the gmap of the guest
- * @gaddr: the guest address to destroy
- *
- * An attempt will be made to destroy the given guest page. If the attempt
- * fails, an attempt is made to export the page. If both attempts fail, an
- * appropriate error is returned.
- */
-int gmap_destroy_page(struct gmap *gmap, unsigned long gaddr)
+int make_hva_secure(struct mm_struct *mm, unsigned long hva, struct uv_cb_header *uvcb)
{
struct vm_area_struct *vma;
struct folio_walk fw;
- unsigned long uaddr;
struct folio *folio;
int rc;
- rc = -EFAULT;
- mmap_read_lock(gmap->mm);
-
- uaddr = __gmap_translate(gmap, gaddr);
- if (IS_ERR_VALUE(uaddr))
- goto out;
- vma = vma_lookup(gmap->mm, uaddr);
- if (!vma)
- goto out;
- /*
- * Huge pages should not be able to become secure
- */
- if (is_vm_hugetlb_page(vma))
- goto out;
+ mmap_read_lock(mm);
+ vma = vma_lookup(mm, hva);
+ if (!vma) {
+ mmap_read_unlock(mm);
+ return -EFAULT;
+ }
+ folio = folio_walk_start(&fw, vma, hva, 0);
+ if (!folio) {
+ mmap_read_unlock(mm);
+ return -ENXIO;
+ }
- rc = 0;
- folio = folio_walk_start(&fw, vma, uaddr, 0);
- if (!folio)
- goto out;
- /*
- * See gmap_make_secure(): large folios cannot be secure. Small
- * folio implies FW_LEVEL_PTE.
- */
- if (folio_test_large(folio) || !pte_write(fw.pte))
- goto out_walk_end;
- rc = uv_destroy_folio(folio);
+ folio_get(folio);
/*
- * Fault handlers can race; it is possible that two CPUs will fault
- * on the same secure page. One CPU can destroy the page, reboot,
- * re-enter secure mode and import it, while the second CPU was
- * stuck at the beginning of the handler. At some point the second
- * CPU will be able to progress, and it will not be able to destroy
- * the page. In that case we do not want to terminate the process,
- * we instead try to export the page.
+ * Secure pages cannot be huge and userspace should not combine both.
+ * In case userspace does it anyway this will result in an -EFAULT for
+ * the unpack. The guest is thus never reaching secure mode.
+ * If userspace plays dirty tricks and decides to map huge pages at a
+ * later point in time, it will receive a segmentation fault or
+ * KVM_RUN will return -EFAULT.
*/
- if (rc)
- rc = uv_convert_from_secure_folio(folio);
-out_walk_end:
+ if (folio_test_hugetlb(folio))
+ rc = -EFAULT;
+ else if (folio_test_large(folio))
+ rc = -E2BIG;
+ else if (!pte_write(fw.pte) || (pte_val(fw.pte) & _PAGE_INVALID))
+ rc = -ENXIO;
+ else
+ rc = make_folio_secure(mm, folio, uvcb);
folio_walk_end(&fw, vma);
-out:
- mmap_read_unlock(gmap->mm);
+ mmap_read_unlock(mm);
+
+ if (rc == -E2BIG || rc == -EBUSY) {
+ rc = s390_wiggle_split_folio(mm, folio, rc == -E2BIG);
+ if (!rc)
+ rc = -EAGAIN;
+ }
+ folio_put(folio);
+
return rc;
}
-EXPORT_SYMBOL_GPL(gmap_destroy_page);
+EXPORT_SYMBOL_GPL(make_hva_secure);
/*
* To be called with the folio locked or with an extra reference! This will
diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile
index 02217fb4ae10..d972dea657fd 100644
--- a/arch/s390/kvm/Makefile
+++ b/arch/s390/kvm/Makefile
@@ -8,7 +8,7 @@ include $(srctree)/virt/kvm/Makefile.kvm
ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
kvm-y += kvm-s390.o intercept.o interrupt.o priv.o sigp.o
-kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o
+kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o gmap.o
kvm-$(CONFIG_VFIO_PCI_ZDEV_KVM) += pci.o
obj-$(CONFIG_KVM) += kvm.o
diff --git a/arch/s390/kvm/gaccess.c b/arch/s390/kvm/gaccess.c
index 7bc97ebd60d5..448304e662fb 100644
--- a/arch/s390/kvm/gaccess.c
+++ b/arch/s390/kvm/gaccess.c
@@ -16,6 +16,7 @@
#include <asm/gmap.h>
#include <asm/dat-bits.h>
#include "kvm-s390.h"
+#include "gmap.h"
#include "gaccess.h"
/*
@@ -1392,6 +1393,42 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr,
return 0;
}
+/**
+ * shadow_pgt_lookup() - find a shadow page table
+ * @sg: pointer to the shadow guest address space structure
+ * @saddr: the address in the shadow aguest address space
+ * @pgt: parent gmap address of the page table to get shadowed
+ * @dat_protection: if the pgtable is marked as protected by dat
+ * @fake: pgt references contiguous guest memory block, not a pgtable
+ *
+ * Returns 0 if the shadow page table was found and -EAGAIN if the page
+ * table was not found.
+ *
+ * Called with sg->mm->mmap_lock in read.
+ */
+static int shadow_pgt_lookup(struct gmap *sg, unsigned long saddr, unsigned long *pgt,
+ int *dat_protection, int *fake)
+{
+ unsigned long *table;
+ struct page *page;
+ int rc;
+
+ spin_lock(&sg->guest_table_lock);
+ table = gmap_table_walk(sg, saddr, 1); /* get segment pointer */
+ if (table && !(*table & _SEGMENT_ENTRY_INVALID)) {
+ /* Shadow page tables are full pages (pte+pgste) */
+ page = pfn_to_page(*table >> PAGE_SHIFT);
+ *pgt = page->index & ~GMAP_SHADOW_FAKE_TABLE;
+ *dat_protection = !!(*table & _SEGMENT_ENTRY_PROTECT);
+ *fake = !!(page->index & GMAP_SHADOW_FAKE_TABLE);
+ rc = 0;
+ } else {
+ rc = -EAGAIN;
+ }
+ spin_unlock(&sg->guest_table_lock);
+ return rc;
+}
+
/**
* kvm_s390_shadow_fault - handle fault on a shadow page table
* @vcpu: virtual cpu
@@ -1415,6 +1452,9 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg,
int dat_protection, fake;
int rc;
+ if (KVM_BUG_ON(!gmap_is_shadow(sg), vcpu->kvm))
+ return -EFAULT;
+
mmap_read_lock(sg->mm);
/*
* We don't want any guest-2 tables to change - so the parent
@@ -1423,7 +1463,7 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg,
*/
ipte_lock(vcpu->kvm);
- rc = gmap_shadow_pgt_lookup(sg, saddr, &pgt, &dat_protection, &fake);
+ rc = shadow_pgt_lookup(sg, saddr, &pgt, &dat_protection, &fake);
if (rc)
rc = kvm_s390_shadow_tables(sg, saddr, &pgt, &dat_protection,
&fake);
diff --git a/arch/s390/kvm/gmap.c b/arch/s390/kvm/gmap.c
new file mode 100644
index 000000000000..6d8944d1b4a0
--- /dev/null
+++ b/arch/s390/kvm/gmap.c
@@ -0,0 +1,121 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Guest memory management for KVM/s390
+ *
+ * Copyright IBM Corp. 2008, 2020, 2024
+ *
+ * Author(s): Claudio Imbrenda <imbrenda@linux.ibm.com>
+ * Martin Schwidefsky <schwidefsky@de.ibm.com>
+ * David Hildenbrand <david@redhat.com>
+ * Janosch Frank <frankja@linux.vnet.ibm.com>
+ */
+
+#include <linux/compiler.h>
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+#include <linux/pgtable.h>
+#include <linux/pagemap.h>
+
+#include <asm/lowcore.h>
+#include <asm/gmap.h>
+#include <asm/uv.h>
+
+#include "gmap.h"
+
+/**
+ * gmap_make_secure() - make one guest page secure
+ * @gmap: the guest gmap
+ * @gaddr: the guest address that needs to be made secure
+ * @uvcb: the UVCB specifying which operation needs to be performed
+ *
+ * Context: needs to be called with kvm->srcu held.
+ * Return: 0 on success, < 0 in case of error.
+ */
+int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb)
+{
+ struct kvm *kvm = gmap->private;
+ unsigned long vmaddr;
+
+ lockdep_assert_held(&kvm->srcu);
+
+ vmaddr = gfn_to_hva(kvm, gpa_to_gfn(gaddr));
+ if (kvm_is_error_hva(vmaddr))
+ return -EFAULT;
+ return make_hva_secure(gmap->mm, vmaddr, uvcb);
+}
+
+int gmap_convert_to_secure(struct gmap *gmap, unsigned long gaddr)
+{
+ struct uv_cb_cts uvcb = {
+ .header.cmd = UVC_CMD_CONV_TO_SEC_STOR,
+ .header.len = sizeof(uvcb),
+ .guest_handle = gmap->guest_handle,
+ .gaddr = gaddr,
+ };
+
+ return gmap_make_secure(gmap, gaddr, &uvcb);
+}
+
+/**
+ * __gmap_destroy_page() - Destroy a guest page.
+ * @gmap: the gmap of the guest
+ * @page: the page to destroy
+ *
+ * An attempt will be made to destroy the given guest page. If the attempt
+ * fails, an attempt is made to export the page. If both attempts fail, an
+ * appropriate error is returned.
+ *
+ * Context: must be called holding the mm lock for gmap->mm
+ */
+static int __gmap_destroy_page(struct gmap *gmap, struct page *page)
+{
+ struct folio *folio = page_folio(page);
+ int rc;
+
+ /*
+ * See gmap_make_secure(): large folios cannot be secure. Small
+ * folio implies FW_LEVEL_PTE.
+ */
+ if (folio_test_large(folio))
+ return -EFAULT;
+
+ rc = uv_destroy_folio(folio);
+ /*
+ * Fault handlers can race; it is possible that two CPUs will fault
+ * on the same secure page. One CPU can destroy the page, reboot,
+ * re-enter secure mode and import it, while the second CPU was
+ * stuck at the beginning of the handler. At some point the second
+ * CPU will be able to progress, and it will not be able to destroy
+ * the page. In that case we do not want to terminate the process,
+ * we instead try to export the page.
+ */
+ if (rc)
+ rc = uv_convert_from_secure_folio(folio);
+
+ return rc;
+}
+
+/**
+ * gmap_destroy_page() - Destroy a guest page.
+ * @gmap: the gmap of the guest
+ * @gaddr: the guest address to destroy
+ *
+ * An attempt will be made to destroy the given guest page. If the attempt
+ * fails, an attempt is made to export the page. If both attempts fail, an
+ * appropriate error is returned.
+ *
+ * Context: may sleep.
+ */
+int gmap_destroy_page(struct gmap *gmap, unsigned long gaddr)
+{
+ struct page *page;
+ int rc = 0;
+
+ mmap_read_lock(gmap->mm);
+ page = gfn_to_page(gmap->private, gpa_to_gfn(gaddr));
+ if (page)
+ rc = __gmap_destroy_page(gmap, page);
+ kvm_release_page_clean(page);
+ mmap_read_unlock(gmap->mm);
+ return rc;
+}
diff --git a/arch/s390/kvm/gmap.h b/arch/s390/kvm/gmap.h
new file mode 100644
index 000000000000..14431ce978da
--- /dev/null
+++ b/arch/s390/kvm/gmap.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KVM guest address space mapping code
+ *
+ * Copyright IBM Corp. 2007, 2016, 2025
+ * Author(s): Martin Schwidefsky <schwidefsky@de.ibm.com>
+ * Claudio Imbrenda <imbrenda@linux.ibm.com>
+ */
+
+#ifndef ARCH_KVM_S390_GMAP_H
+#define ARCH_KVM_S390_GMAP_H
+
+#define GMAP_SHADOW_FAKE_TABLE 1ULL
+
+int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb);
+int gmap_convert_to_secure(struct gmap *gmap, unsigned long gaddr);
+int gmap_destroy_page(struct gmap *gmap, unsigned long gaddr);
+
+#endif
diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/intercept.c
index f0be263b334c..deb2f2b3e5c3 100644
--- a/arch/s390/kvm/intercept.c
+++ b/arch/s390/kvm/intercept.c
@@ -21,6 +21,7 @@
#include "gaccess.h"
#include "trace.h"
#include "trace-s390.h"
+#include "gmap.h"
u8 kvm_s390_get_ilen(struct kvm_vcpu *vcpu)
{
@@ -367,7 +368,7 @@ static int handle_mvpg_pei(struct kvm_vcpu *vcpu)
reg2, &srcaddr, GACC_FETCH, 0);
if (rc)
return kvm_s390_inject_prog_cond(vcpu, rc);
- rc = kvm_arch_fault_in_page(vcpu, srcaddr, 0);
+ rc = gmap_fault(vcpu->arch.gmap, srcaddr, 0);
if (rc != 0)
return rc;
@@ -376,7 +377,7 @@ static int handle_mvpg_pei(struct kvm_vcpu *vcpu)
reg1, &dstaddr, GACC_STORE, 0);
if (rc)
return kvm_s390_inject_prog_cond(vcpu, rc);
- rc = kvm_arch_fault_in_page(vcpu, dstaddr, 1);
+ rc = gmap_fault(vcpu->arch.gmap, dstaddr, FAULT_FLAG_WRITE);
if (rc != 0)
return rc;
@@ -549,7 +550,7 @@ static int handle_pv_uvc(struct kvm_vcpu *vcpu)
* If the unpin did not succeed, the guest will exit again for the UVC
* and we will retry the unpin.
*/
- if (rc == -EINVAL)
+ if (rc == -EINVAL || rc == -ENXIO)
return 0;
/*
* If we got -EAGAIN here, we simply return it. It will eventually
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index 0f3ad8c80381..1474a776efa3 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -2865,9 +2865,7 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e,
void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
struct mcck_volatile_info *mcck_info)
{
- struct kvm_s390_interrupt_info inti;
struct kvm_s390_irq irq = {};
- struct kvm_s390_mchk_info *mchk;
union mci mci;
__u64 cr14 = 0; /* upper bits are not used */
int rc;
@@ -2880,20 +2878,14 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
if (mci.w)
cr14 |= CR14_WARNING_SUBMASK;
- mchk = mci.ck ? &inti.mchk : &irq.u.mchk;
- mchk->cr14 = cr14;
- mchk->mcic = mcck_info->mcic;
- mchk->ext_damage_code = mcck_info->ext_damage_code;
- mchk->failing_storage_address = mcck_info->failing_storage_address;
- if (mci.ck) {
- /* Inject the floating machine check */
- inti.type = KVM_S390_MCHK;
- rc = __inject_vm(vcpu->kvm, &inti);
- } else {
- /* Inject the machine check to specified vcpu */
- irq.type = KVM_S390_MCHK;
- rc = kvm_s390_inject_vcpu(vcpu, &irq);
- }
+ irq.u.mchk.cr14 = cr14;
+ irq.u.mchk.mcic = mcck_info->mcic;
+ irq.u.mchk.ext_damage_code = mcck_info->ext_damage_code;
+ irq.u.mchk.failing_storage_address = mcck_info->failing_storage_address;
+
+ /* Inject the machine check to specified vcpu */
+ irq.type = KVM_S390_MCHK;
+ rc = kvm_s390_inject_vcpu(vcpu, &irq);
WARN_ON_ONCE(rc);
}
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 0e3b8ee0ec26..7bb5640bb8a7 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -49,6 +49,7 @@
#include "kvm-s390.h"
#include "gaccess.h"
#include "pci.h"
+#include "gmap.h"
#define CREATE_TRACE_POINTS
#include "trace.h"
@@ -4582,22 +4583,6 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo
return 1;
}
-/**
- * kvm_arch_fault_in_page - fault-in guest page if necessary
- * @vcpu: The corresponding virtual cpu
- * @gpa: Guest physical address
- * @writable: Whether the page should be writable or not
- *
- * Make sure that a guest page has been faulted-in on the host.
- *
- * Return: Zero on success, negative error code otherwise.
- */
-long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable)
-{
- return gmap_fault(vcpu->arch.gmap, gpa,
- writable ? FAULT_FLAG_WRITE : 0);
-}
-
static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token,
unsigned long token)
{
@@ -4665,12 +4650,11 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu)
if (!vcpu->arch.gmap->pfault_enabled)
return false;
- hva = gfn_to_hva(vcpu->kvm, gpa_to_gfn(current->thread.gmap_addr));
- hva += current->thread.gmap_addr & ~PAGE_MASK;
+ hva = gfn_to_hva(vcpu->kvm, current->thread.gmap_teid.addr);
if (read_guest_real(vcpu, vcpu->arch.pfault_token, &arch.pfault_token, 8))
return false;
- return kvm_setup_async_pf(vcpu, current->thread.gmap_addr, hva, &arch);
+ return kvm_setup_async_pf(vcpu, current->thread.gmap_teid.addr * PAGE_SIZE, hva, &arch);
}
static int vcpu_pre_run(struct kvm_vcpu *vcpu)
@@ -4708,6 +4692,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
clear_bit(vcpu->vcpu_idx, vcpu->kvm->arch.gisa_int.kicked_mask);
vcpu->arch.sie_block->icptcode = 0;
+ current->thread.gmap_int_code = 0;
cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
VCPU_EVENT(vcpu, 6, "entering sie flags %x", cpuflags);
trace_kvm_s390_sie_enter(vcpu, cpuflags);
@@ -4715,7 +4700,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
return 0;
}
-static int vcpu_post_run_fault_in_sie(struct kvm_vcpu *vcpu)
+static int vcpu_post_run_addressing_exception(struct kvm_vcpu *vcpu)
{
struct kvm_s390_pgm_info pgm_info = {
.code = PGM_ADDRESSING,
@@ -4751,10 +4736,182 @@ static int vcpu_post_run_fault_in_sie(struct kvm_vcpu *vcpu)
return kvm_s390_inject_prog_irq(vcpu, &pgm_info);
}
+static void kvm_s390_assert_primary_as(struct kvm_vcpu *vcpu)
+{
+ KVM_BUG(current->thread.gmap_teid.as != PSW_BITS_AS_PRIMARY, vcpu->kvm,
+ "Unexpected program interrupt 0x%x, TEID 0x%016lx",
+ current->thread.gmap_int_code, current->thread.gmap_teid.val);
+}
+
+/*
+ * __kvm_s390_handle_dat_fault() - handle a dat fault for the gmap of a vcpu
+ * @vcpu: the vCPU whose gmap is to be fixed up
+ * @gfn: the guest frame number used for memslots (including fake memslots)
+ * @gaddr: the gmap address, does not have to match @gfn for ucontrol gmaps
+ * @foll: FOLL_* flags
+ *
+ * Return: 0 on success, < 0 in case of error.
+ * Context: The mm lock must not be held before calling. May sleep.
+ */
+int __kvm_s390_handle_dat_fault(struct kvm_vcpu *vcpu, gfn_t gfn, gpa_t gaddr, unsigned int foll)
+{
+ struct kvm_memory_slot *slot;
+ unsigned int fault_flags;
+ bool writable, unlocked;
+ unsigned long vmaddr;
+ struct page *page;
+ kvm_pfn_t pfn;
+ int rc;
+
+ slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
+ if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
+ return vcpu_post_run_addressing_exception(vcpu);
+
+ fault_flags = foll & FOLL_WRITE ? FAULT_FLAG_WRITE : 0;
+ if (vcpu->arch.gmap->pfault_enabled)
+ foll |= FOLL_NOWAIT;
+ vmaddr = __gfn_to_hva_memslot(slot, gfn);
+
+try_again:
+ pfn = __kvm_faultin_pfn(slot, gfn, foll, &writable, &page);
+
+ /* Access outside memory, inject addressing exception */
+ if (is_noslot_pfn(pfn))
+ return vcpu_post_run_addressing_exception(vcpu);
+ /* Signal pending: try again */
+ if (pfn == KVM_PFN_ERR_SIGPENDING)
+ return -EAGAIN;
+
+ /* Needs I/O, try to setup async pfault (only possible with FOLL_NOWAIT) */
+ if (pfn == KVM_PFN_ERR_NEEDS_IO) {
+ trace_kvm_s390_major_guest_pfault(vcpu);
+ if (kvm_arch_setup_async_pf(vcpu))
+ return 0;
+ vcpu->stat.pfault_sync++;
+ /* Could not setup async pfault, try again synchronously */
+ foll &= ~FOLL_NOWAIT;
+ goto try_again;
+ }
+ /* Any other error */
+ if (is_error_pfn(pfn))
+ return -EFAULT;
+
+ /* Success */
+ mmap_read_lock(vcpu->arch.gmap->mm);
+ /* Mark the userspace PTEs as young and/or dirty, to avoid page fault loops */
+ rc = fixup_user_fault(vcpu->arch.gmap->mm, vmaddr, fault_flags, &unlocked);
+ if (!rc)
+ rc = __gmap_link(vcpu->arch.gmap, gaddr, vmaddr);
+ scoped_guard(spinlock, &vcpu->kvm->mmu_lock) {
+ kvm_release_faultin_page(vcpu->kvm, page, false, writable);
+ }
+ mmap_read_unlock(vcpu->arch.gmap->mm);
+ return rc;
+}
+
+static int vcpu_dat_fault_handler(struct kvm_vcpu *vcpu, unsigned long gaddr, unsigned int foll)
+{
+ unsigned long gaddr_tmp;
+ gfn_t gfn;
+
+ gfn = gpa_to_gfn(gaddr);
+ if (kvm_is_ucontrol(vcpu->kvm)) {
+ /*
+ * This translates the per-vCPU guest address into a
+ * fake guest address, which can then be used with the
+ * fake memslots that are identity mapping userspace.
+ * This allows ucontrol VMs to use the normal fault
+ * resolution path, like normal VMs.
+ */
+ mmap_read_lock(vcpu->arch.gmap->mm);
+ gaddr_tmp = __gmap_translate(vcpu->arch.gmap, gaddr);
+ mmap_read_unlock(vcpu->arch.gmap->mm);
+ if (gaddr_tmp == -EFAULT) {
+ vcpu->run->exit_reason = KVM_EXIT_S390_UCONTROL;
+ vcpu->run->s390_ucontrol.trans_exc_code = gaddr;
+ vcpu->run->s390_ucontrol.pgm_code = PGM_SEGMENT_TRANSLATION;
+ return -EREMOTE;
+ }
+ gfn = gpa_to_gfn(gaddr_tmp);
+ }
+ return __kvm_s390_handle_dat_fault(vcpu, gfn, gaddr, foll);
+}
+
+static int vcpu_post_run_handle_fault(struct kvm_vcpu *vcpu)
+{
+ unsigned int foll = 0;
+ unsigned long gaddr;
+ int rc;
+
+ gaddr = current->thread.gmap_teid.addr * PAGE_SIZE;
+ if (kvm_s390_cur_gmap_fault_is_write())
+ foll = FOLL_WRITE;
+
+ switch (current->thread.gmap_int_code & PGM_INT_CODE_MASK) {
+ case 0:
+ vcpu->stat.exit_null++;
+ break;
+ case PGM_SECURE_STORAGE_ACCESS:
+ case PGM_SECURE_STORAGE_VIOLATION:
+ kvm_s390_assert_primary_as(vcpu);
+ /*
+ * This can happen after a reboot with asynchronous teardown;
+ * the new guest (normal or protected) will run on top of the
+ * previous protected guest. The old pages need to be destroyed
+ * so the new guest can use them.
+ */
+ if (gmap_destroy_page(vcpu->arch.gmap, gaddr)) {
+ /*
+ * Either KVM messed up the secure guest mapping or the
+ * same page is mapped into multiple secure guests.
+ *
+ * This exception is only triggered when a guest 2 is
+ * running and can therefore never occur in kernel
+ * context.
+ */
+ pr_warn_ratelimited("Secure storage violation (%x) in task: %s, pid %d\n",
+ current->thread.gmap_int_code, current->comm,
+ current->pid);
+ send_sig(SIGSEGV, current, 0);
+ }
+ break;
+ case PGM_NON_SECURE_STORAGE_ACCESS:
+ kvm_s390_assert_primary_as(vcpu);
+ /*
+ * This is normal operation; a page belonging to a protected
+ * guest has not been imported yet. Try to import the page into
+ * the protected guest.
+ */
+ rc = gmap_convert_to_secure(vcpu->arch.gmap, gaddr);
+ if (rc == -EINVAL)
+ send_sig(SIGSEGV, current, 0);
+ if (rc != -ENXIO)
+ break;
+ foll = FOLL_WRITE;
+ fallthrough;
+ case PGM_PROTECTION:
+ case PGM_SEGMENT_TRANSLATION:
+ case PGM_PAGE_TRANSLATION:
+ case PGM_ASCE_TYPE:
+ case PGM_REGION_FIRST_TRANS:
+ case PGM_REGION_SECOND_TRANS:
+ case PGM_REGION_THIRD_TRANS:
+ kvm_s390_assert_primary_as(vcpu);
+ return vcpu_dat_fault_handler(vcpu, gaddr, foll);
+ default:
+ KVM_BUG(1, vcpu->kvm, "Unexpected program interrupt 0x%x, TEID 0x%016lx",
+ current->thread.gmap_int_code, current->thread.gmap_teid.val);
+ send_sig(SIGSEGV, current, 0);
+ break;
+ }
+ return 0;
+}
+
static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
{
struct mcck_volatile_info *mcck_info;
struct sie_page *sie_page;
+ int rc;
VCPU_EVENT(vcpu, 6, "exit sie icptcode %d",
vcpu->arch.sie_block->icptcode);
@@ -4776,7 +4933,7 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
}
if (vcpu->arch.sie_block->icptcode > 0) {
- int rc = kvm_handle_sie_intercept(vcpu);
+ rc = kvm_handle_sie_intercept(vcpu);
if (rc != -EOPNOTSUPP)
return rc;
@@ -4785,24 +4942,9 @@ static int vcpu_post_run(struct kvm_vcpu *vcpu, int exit_reason)
vcpu->run->s390_sieic.ipa = vcpu->arch.sie_block->ipa;
vcpu->run->s390_sieic.ipb = vcpu->arch.sie_block->ipb;
return -EREMOTE;
- } else if (exit_reason != -EFAULT) {
- vcpu->stat.exit_null++;
- return 0;
- } else if (kvm_is_ucontrol(vcpu->kvm)) {
- vcpu->run->exit_reason = KVM_EXIT_S390_UCONTROL;
- vcpu->run->s390_ucontrol.trans_exc_code =
- current->thread.gmap_addr;
- vcpu->run->s390_ucontrol.pgm_code = 0x10;
- return -EREMOTE;
- } else if (current->thread.gmap_pfault) {
- trace_kvm_s390_major_guest_pfault(vcpu);
- current->thread.gmap_pfault = 0;
- if (kvm_arch_setup_async_pf(vcpu))
- return 0;
- vcpu->stat.pfault_sync++;
- return kvm_arch_fault_in_page(vcpu, current->thread.gmap_addr, 1);
}
- return vcpu_post_run_fault_in_sie(vcpu);
+
+ return vcpu_post_run_handle_fault(vcpu);
}
#define PSW_INT_MASK (PSW_MASK_EXT | PSW_MASK_IO | PSW_MASK_MCHECK)
@@ -5643,7 +5785,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
}
#endif
case KVM_S390_VCPU_FAULT: {
- r = gmap_fault(vcpu->arch.gmap, arg, 0);
+ idx = srcu_read_lock(&vcpu->kvm->srcu);
+ r = vcpu_dat_fault_handler(vcpu, arg, 0);
+ srcu_read_unlock(&vcpu->kvm->srcu, idx);
break;
}
case KVM_ENABLE_CAP:
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index e680c6bf0c9d..3f72a217ee99 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -394,7 +394,6 @@ int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu);
/* implemented in kvm-s390.c */
int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod);
-long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable);
int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr);
int kvm_s390_vcpu_store_status(struct kvm_vcpu *vcpu, unsigned long addr);
int kvm_s390_vcpu_start(struct kvm_vcpu *vcpu);
@@ -409,6 +408,12 @@ void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu);
void kvm_s390_set_cpu_timer(struct kvm_vcpu *vcpu, __u64 cputm);
__u64 kvm_s390_get_cpu_timer(struct kvm_vcpu *vcpu);
int kvm_s390_cpus_from_pv(struct kvm *kvm, u16 *rc, u16 *rrc);
+int __kvm_s390_handle_dat_fault(struct kvm_vcpu *vcpu, gfn_t gfn, gpa_t gaddr, unsigned int flags);
+
+static inline int kvm_s390_handle_dat_fault(struct kvm_vcpu *vcpu, gpa_t gaddr, unsigned int flags)
+{
+ return __kvm_s390_handle_dat_fault(vcpu, gpa_to_gfn(gaddr), gaddr, flags);
+}
/* implemented in diag.c */
int kvm_s390_handle_diag(struct kvm_vcpu *vcpu);
@@ -529,6 +534,13 @@ static inline int kvm_s390_use_sca_entries(void)
void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
struct mcck_volatile_info *mcck_info);
+static inline bool kvm_s390_cur_gmap_fault_is_write(void)
+{
+ if (current->thread.gmap_int_code == PGM_PROTECTION)
+ return true;
+ return test_facility(75) && (current->thread.gmap_teid.fsi == TEID_FSI_STORE);
+}
+
/**
* kvm_s390_vcpu_crypto_reset_all
*
diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
index d9fcbf41025e..86a8939fc3d9 100644
--- a/arch/s390/kvm/pci.c
+++ b/arch/s390/kvm/pci.c
@@ -502,7 +502,6 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
{
struct zpci_dev *zdev = opaque;
- u8 status;
int rc;
if (!zdev)
@@ -549,13 +548,7 @@ static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
*/
zdev->gisa = (u32)virt_to_phys(&kvm->arch.sie_page2->gisa);
- rc = zpci_enable_device(zdev);
- if (rc)
- goto clear_gisa;
-
- /* Re-register the IOMMU that was already created */
- rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table), &status);
+ rc = zpci_reenable_device(zdev);
if (rc)
goto clear_gisa;
@@ -585,7 +578,6 @@ static void kvm_s390_pci_unregister_kvm(void *opaque)
{
struct zpci_dev *zdev = opaque;
struct kvm *kvm;
- u8 status;
if (!zdev)
return;
@@ -619,12 +611,7 @@ static void kvm_s390_pci_unregister_kvm(void *opaque)
goto out;
}
- if (zpci_enable_device(zdev))
- goto out;
-
- /* Re-register the IOMMU that was already created */
- zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table), &status);
+ zpci_reenable_device(zdev);
out:
spin_lock(&kvm->arch.kzdev_list_lock);
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index 75e81ba26d04..22c012aa5206 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -17,6 +17,7 @@
#include <linux/sched/mm.h>
#include <linux/mmu_notifier.h>
#include "kvm-s390.h"
+#include "gmap.h"
bool kvm_s390_pv_is_protected(struct kvm *kvm)
{
@@ -638,10 +639,28 @@ static int unpack_one(struct kvm *kvm, unsigned long addr, u64 tweak,
.tweak[1] = offset,
};
int ret = gmap_make_secure(kvm->arch.gmap, addr, &uvcb);
+ unsigned long vmaddr;
+ bool unlocked;
*rc = uvcb.header.rc;
*rrc = uvcb.header.rrc;
+ if (ret == -ENXIO) {
+ mmap_read_lock(kvm->mm);
+ vmaddr = gfn_to_hva(kvm, gpa_to_gfn(addr));
+ if (kvm_is_error_hva(vmaddr)) {
+ ret = -EFAULT;
+ } else {
+ ret = fixup_user_fault(kvm->mm, vmaddr, FAULT_FLAG_WRITE, &unlocked);
+ if (!ret)
+ ret = __gmap_link(kvm->arch.gmap, addr, vmaddr);
+ }
+ mmap_read_unlock(kvm->mm);
+ if (!ret)
+ return -EAGAIN;
+ return ret;
+ }
+
if (ret && ret != -EAGAIN)
KVM_UV_EVENT(kvm, 3, "PROTVIRT VM UNPACK: failed addr %llx with rc %x rrc %x",
uvcb.gaddr, *rc, *rrc);
@@ -660,6 +679,8 @@ int kvm_s390_pv_unpack(struct kvm *kvm, unsigned long addr, unsigned long size,
KVM_UV_EVENT(kvm, 3, "PROTVIRT VM UNPACK: start addr %lx size %lx",
addr, size);
+ guard(srcu)(&kvm->srcu);
+
while (offset < size) {
ret = unpack_one(kvm, addr, tweak, offset, rc, rrc);
if (ret == -EAGAIN) {
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 5b5e691b61ac..de9f0dd42197 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -923,19 +923,19 @@ static int handle_fault(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
{
int rc;
- if (current->thread.gmap_int_code == PGM_PROTECTION)
+ if ((current->thread.gmap_int_code & PGM_INT_CODE_MASK) == PGM_PROTECTION)
/* we can directly forward all protection exceptions */
return inject_fault(vcpu, PGM_PROTECTION,
- current->thread.gmap_addr, 1);
+ current->thread.gmap_teid.addr * PAGE_SIZE, 1);
rc = kvm_s390_shadow_fault(vcpu, vsie_page->gmap,
- current->thread.gmap_addr, NULL);
+ current->thread.gmap_teid.addr * PAGE_SIZE, NULL);
if (rc > 0) {
rc = inject_fault(vcpu, rc,
- current->thread.gmap_addr,
- current->thread.gmap_write_flag);
+ current->thread.gmap_teid.addr * PAGE_SIZE,
+ kvm_s390_cur_gmap_fault_is_write());
if (rc >= 0)
- vsie_page->fault_addr = current->thread.gmap_addr;
+ vsie_page->fault_addr = current->thread.gmap_teid.addr * PAGE_SIZE;
}
return rc;
}
@@ -1149,6 +1149,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
* also kick the vSIE.
*/
vcpu->arch.sie_block->prog0c |= PROG_IN_SIE;
+ current->thread.gmap_int_code = 0;
barrier();
if (!kvm_s390_vcpu_sie_inhibited(vcpu))
rc = sie64a(scb_s, vcpu->run->s.regs.gprs, gmap_get_enabled()->asce);
@@ -1173,7 +1174,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
if (rc > 0)
rc = 0; /* we could still have an icpt */
- else if (rc == -EFAULT)
+ else if (current->thread.gmap_int_code)
return handle_fault(vcpu, vsie_page);
switch (scb_s->icptcode) {
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index ad8b0d6b77ea..2d03af60ed88 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -46,12 +46,6 @@
#include <asm/uv.h>
#include "../kernel/entry.h"
-enum fault_type {
- KERNEL_FAULT,
- USER_FAULT,
- GMAP_FAULT,
-};
-
static DEFINE_STATIC_KEY_FALSE(have_store_indication);
static int __init fault_init(void)
@@ -65,28 +59,15 @@ early_initcall(fault_init);
/*
* Find out which address space caused the exception.
*/
-static enum fault_type get_fault_type(struct pt_regs *regs)
+static bool is_kernel_fault(struct pt_regs *regs)
{
union teid teid = { .val = regs->int_parm_long };
- struct gmap *gmap;
-
- if (likely(teid.as == PSW_BITS_AS_PRIMARY)) {
- if (user_mode(regs))
- return USER_FAULT;
- if (!IS_ENABLED(CONFIG_PGSTE))
- return KERNEL_FAULT;
- gmap = (struct gmap *)get_lowcore()->gmap;
- if (gmap && gmap->asce == regs->cr1)
- return GMAP_FAULT;
- return KERNEL_FAULT;
- }
+
+ if (user_mode(regs))
+ return false;
if (teid.as == PSW_BITS_AS_SECONDARY)
- return USER_FAULT;
- /* Access register mode, not used in the kernel */
- if (teid.as == PSW_BITS_AS_ACCREG)
- return USER_FAULT;
- /* Home space -> access via kernel ASCE */
- return KERNEL_FAULT;
+ return false;
+ return true;
}
static unsigned long get_fault_address(struct pt_regs *regs)
@@ -181,21 +162,12 @@ static void dump_fault_info(struct pt_regs *regs)
break;
}
pr_cont("mode while using ");
- switch (get_fault_type(regs)) {
- case USER_FAULT:
- asce = get_lowcore()->user_asce.val;
- pr_cont("user ");
- break;
- case GMAP_FAULT:
- asce = ((struct gmap *)get_lowcore()->gmap)->asce;
- pr_cont("gmap ");
- break;
- case KERNEL_FAULT:
+ if (is_kernel_fault(regs)) {
asce = get_lowcore()->kernel_asce.val;
pr_cont("kernel ");
- break;
- default:
- unreachable();
+ } else {
+ asce = get_lowcore()->user_asce.val;
+ pr_cont("user ");
}
pr_cont("ASCE.\n");
dump_pagetable(asce, get_fault_address(regs));
@@ -230,7 +202,6 @@ static void do_sigsegv(struct pt_regs *regs, int si_code)
static void handle_fault_error_nolock(struct pt_regs *regs, int si_code)
{
- enum fault_type fault_type;
unsigned long address;
bool is_write;
@@ -241,17 +212,15 @@ static void handle_fault_error_nolock(struct pt_regs *regs, int si_code)
}
if (fixup_exception(regs))
return;
- fault_type = get_fault_type(regs);
- if (fault_type == KERNEL_FAULT) {
+ if (is_kernel_fault(regs)) {
address = get_fault_address(regs);
is_write = fault_is_write(regs);
if (kfence_handle_page_fault(address, is_write, regs))
return;
- }
- if (fault_type == KERNEL_FAULT)
pr_alert("Unable to handle kernel pointer dereference in virtual kernel address space\n");
- else
+ } else {
pr_alert("Unable to handle kernel paging request in virtual user address space\n");
+ }
dump_fault_info(regs);
die(regs, "Oops");
}
@@ -285,9 +254,7 @@ static void do_exception(struct pt_regs *regs, int access)
struct vm_area_struct *vma;
unsigned long address;
struct mm_struct *mm;
- enum fault_type type;
unsigned int flags;
- struct gmap *gmap;
vm_fault_t fault;
bool is_write;
@@ -301,16 +268,8 @@ static void do_exception(struct pt_regs *regs, int access)
mm = current->mm;
address = get_fault_address(regs);
is_write = fault_is_write(regs);
- type = get_fault_type(regs);
- switch (type) {
- case KERNEL_FAULT:
+ if (is_kernel_fault(regs) || faulthandler_disabled() || !mm)
return handle_fault_error_nolock(regs, 0);
- case USER_FAULT:
- case GMAP_FAULT:
- if (faulthandler_disabled() || !mm)
- return handle_fault_error_nolock(regs, 0);
- break;
- }
perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
flags = FAULT_FLAG_DEFAULT;
if (user_mode(regs))
@@ -350,18 +309,6 @@ static void do_exception(struct pt_regs *regs, int access)
}
lock_mmap:
mmap_read_lock(mm);
- gmap = NULL;
- if (IS_ENABLED(CONFIG_PGSTE) && type == GMAP_FAULT) {
- gmap = (struct gmap *)get_lowcore()->gmap;
- current->thread.gmap_addr = address;
- current->thread.gmap_write_flag = !!(flags & FAULT_FLAG_WRITE);
- current->thread.gmap_int_code = regs->int_code & 0xffff;
- address = __gmap_translate(gmap, address);
- if (address == -EFAULT)
- return handle_fault_error(regs, SEGV_MAPERR);
- if (gmap->pfault_enabled)
- flags |= FAULT_FLAG_RETRY_NOWAIT;
- }
retry:
vma = find_vma(mm, address);
if (!vma)
@@ -377,50 +324,22 @@ static void do_exception(struct pt_regs *regs, int access)
return handle_fault_error(regs, SEGV_ACCERR);
fault = handle_mm_fault(vma, address, flags, regs);
if (fault_signal_pending(fault, regs)) {
- if (flags & FAULT_FLAG_RETRY_NOWAIT)
- mmap_read_unlock(mm);
if (!user_mode(regs))
handle_fault_error_nolock(regs, 0);
return;
}
/* The fault is fully completed (including releasing mmap lock) */
- if (fault & VM_FAULT_COMPLETED) {
- if (gmap) {
- mmap_read_lock(mm);
- goto gmap;
- }
+ if (fault & VM_FAULT_COMPLETED)
return;
- }
if (unlikely(fault & VM_FAULT_ERROR)) {
mmap_read_unlock(mm);
goto error;
}
if (fault & VM_FAULT_RETRY) {
- if (IS_ENABLED(CONFIG_PGSTE) && gmap && (flags & FAULT_FLAG_RETRY_NOWAIT)) {
- /*
- * FAULT_FLAG_RETRY_NOWAIT has been set,
- * mmap_lock has not been released
- */
- current->thread.gmap_pfault = 1;
- return handle_fault_error(regs, 0);
- }
- flags &= ~FAULT_FLAG_RETRY_NOWAIT;
flags |= FAULT_FLAG_TRIED;
mmap_read_lock(mm);
goto retry;
}
-gmap:
- if (IS_ENABLED(CONFIG_PGSTE) && gmap) {
- address = __gmap_link(gmap, current->thread.gmap_addr,
- address);
- if (address == -EFAULT)
- return handle_fault_error(regs, SEGV_MAPERR);
- if (address == -ENOMEM) {
- fault = VM_FAULT_OOM;
- mmap_read_unlock(mm);
- goto error;
- }
- }
mmap_read_unlock(mm);
return;
error:
@@ -496,7 +415,6 @@ void do_secure_storage_access(struct pt_regs *regs)
struct folio_walk fw;
struct mm_struct *mm;
struct folio *folio;
- struct gmap *gmap;
int rc;
/*
@@ -521,17 +439,24 @@ void do_secure_storage_access(struct pt_regs *regs)
*/
panic("Unexpected PGM 0x3d with TEID bit 61=0");
}
- switch (get_fault_type(regs)) {
- case GMAP_FAULT:
- mm = current->mm;
- gmap = (struct gmap *)get_lowcore()->gmap;
- mmap_read_lock(mm);
- addr = __gmap_translate(gmap, addr);
- mmap_read_unlock(mm);
- if (IS_ERR_VALUE(addr))
- return handle_fault_error_nolock(regs, SEGV_MAPERR);
- fallthrough;
- case USER_FAULT:
+ if (is_kernel_fault(regs)) {
+ folio = phys_to_folio(addr);
+ if (unlikely(!folio_try_get(folio)))
+ return;
+ rc = uv_convert_from_secure(folio_to_phys(folio));
+ if (!rc)
+ clear_bit(PG_arch_1, &folio->flags.f);
+ folio_put(folio);
+ /*
+ * There are some valid fixup types for kernel
+ * accesses to donated secure memory. zeropad is one
+ * of them.
+ */
+ if (rc)
+ return handle_fault_error_nolock(regs, 0);
+ } else {
+ if (faulthandler_disabled())
+ return handle_fault_error_nolock(regs, 0);
mm = current->mm;
mmap_read_lock(mm);
vma = find_vma(mm, addr);
@@ -540,7 +465,7 @@ void do_secure_storage_access(struct pt_regs *regs)
folio = folio_walk_start(&fw, vma, addr, 0);
if (!folio) {
mmap_read_unlock(mm);
- break;
+ return;
}
/* arch_make_folio_accessible() needs a raised refcount. */
folio_get(folio);
@@ -550,56 +475,8 @@ void do_secure_storage_access(struct pt_regs *regs)
if (rc)
send_sig(SIGSEGV, current, 0);
mmap_read_unlock(mm);
- break;
- case KERNEL_FAULT:
- folio = phys_to_folio(addr);
- if (unlikely(!folio_try_get(folio)))
- break;
- rc = arch_make_folio_accessible(folio);
- folio_put(folio);
- if (rc)
- BUG();
- break;
- default:
- unreachable();
}
}
NOKPROBE_SYMBOL(do_secure_storage_access);
-void do_non_secure_storage_access(struct pt_regs *regs)
-{
- struct gmap *gmap = (struct gmap *)get_lowcore()->gmap;
- unsigned long gaddr = get_fault_address(regs);
-
- if (WARN_ON_ONCE(get_fault_type(regs) != GMAP_FAULT))
- return handle_fault_error_nolock(regs, SEGV_MAPERR);
- if (gmap_convert_to_secure(gmap, gaddr) == -EINVAL)
- send_sig(SIGSEGV, current, 0);
-}
-NOKPROBE_SYMBOL(do_non_secure_storage_access);
-
-void do_secure_storage_violation(struct pt_regs *regs)
-{
- struct gmap *gmap = (struct gmap *)get_lowcore()->gmap;
- unsigned long gaddr = get_fault_address(regs);
-
- /*
- * If the VM has been rebooted, its address space might still contain
- * secure pages from the previous boot.
- * Clear the page so it can be reused.
- */
- if (!gmap_destroy_page(gmap, gaddr))
- return;
- /*
- * Either KVM messed up the secure guest mapping or the same
- * page is mapped into multiple secure guests.
- *
- * This exception is only triggered when a guest 2 is running
- * and can therefore never occur in kernel context.
- */
- pr_warn_ratelimited("Secure storage violation in task: %s, pid %d\n",
- current->comm, current->pid);
- send_sig(SIGSEGV, current, 0);
-}
-
#endif /* CONFIG_PGSTE */
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index eb0b51a36be0..719a9adc2ad3 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -635,6 +635,7 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr)
radix_tree_preload_end();
return rc;
}
+EXPORT_SYMBOL(__gmap_link);
/**
* gmap_fault - resolve a fault on a guest address
@@ -803,8 +804,7 @@ static void gmap_call_notifier(struct gmap *gmap, unsigned long start,
*
* Note: Can also be called for shadow gmaps.
*/
-static inline unsigned long *gmap_table_walk(struct gmap *gmap,
- unsigned long gaddr, int level)
+unsigned long *gmap_table_walk(struct gmap *gmap, unsigned long gaddr, int level)
{
const int asce_type = gmap->asce & _ASCE_TYPE_MASK;
unsigned long *table = gmap->table;
@@ -855,6 +855,7 @@ static inline unsigned long *gmap_table_walk(struct gmap *gmap,
}
return table;
}
+EXPORT_SYMBOL(gmap_table_walk);
/**
* gmap_pte_op_walk - walk the gmap page table, get the page table lock
@@ -2002,46 +2003,6 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt,
}
EXPORT_SYMBOL_GPL(gmap_shadow_sgt);
-/**
- * gmap_shadow_pgt_lookup - find a shadow page table
- * @sg: pointer to the shadow guest address space structure
- * @saddr: the address in the shadow aguest address space
- * @pgt: parent gmap address of the page table to get shadowed
- * @dat_protection: if the pgtable is marked as protected by dat
- * @fake: pgt references contiguous guest memory block, not a pgtable
- *
- * Returns 0 if the shadow page table was found and -EAGAIN if the page
- * table was not found.
- *
- * Called with sg->mm->mmap_lock in read.
- */
-int gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long saddr,
- unsigned long *pgt, int *dat_protection,
- int *fake)
-{
- unsigned long *table;
- struct page *page;
- int rc;
-
- BUG_ON(!gmap_is_shadow(sg));
- spin_lock(&sg->guest_table_lock);
- table = gmap_table_walk(sg, saddr, 1); /* get segment pointer */
- if (table && !(*table & _SEGMENT_ENTRY_INVALID)) {
- /* Shadow page tables are full pages (pte+pgste) */
- page = pfn_to_page(*table >> PAGE_SHIFT);
- *pgt = page->index & ~GMAP_SHADOW_FAKE_TABLE;
- *dat_protection = !!(*table & _SEGMENT_ENTRY_PROTECT);
- *fake = !!(page->index & GMAP_SHADOW_FAKE_TABLE);
- rc = 0;
- } else {
- rc = -EAGAIN;
- }
- spin_unlock(&sg->guest_table_lock);
- return rc;
-
-}
-EXPORT_SYMBOL_GPL(gmap_shadow_pgt_lookup);
-
/**
* gmap_shadow_pgt - instantiate a shadow page table
* @sg: pointer to the shadow guest address space structure
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index 2fe5b91c3d90..554492f07b04 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -134,14 +134,13 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
struct zpci_fib fib = {0};
u8 cc;
- WARN_ON_ONCE(iota & 0x3fff);
fib.pba = base;
/* Work around off by one in ISM virt device */
if (zdev->pft == PCI_FUNC_TYPE_ISM && limit > base)
fib.pal = limit + (1 << 12);
else
fib.pal = limit;
- fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
+ fib.iota = iota;
fib.gd = zdev->gisa;
cc = zpci_mod_fc(req, &fib, status);
if (cc)
@@ -709,6 +708,23 @@ int zpci_enable_device(struct zpci_dev *zdev)
}
EXPORT_SYMBOL_GPL(zpci_enable_device);
+int zpci_reenable_device(struct zpci_dev *zdev)
+{
+ u8 status;
+ int rc;
+
+ rc = zpci_enable_device(zdev);
+ if (rc)
+ return rc;
+
+ rc = zpci_iommu_register_ioat(zdev, &status);
+ if (rc)
+ zpci_disable_device(zdev);
+
+ return rc;
+}
+EXPORT_SYMBOL_GPL(zpci_reenable_device);
+
int zpci_disable_device(struct zpci_dev *zdev)
{
u32 fh = zdev->fh;
@@ -758,7 +774,6 @@ EXPORT_SYMBOL_GPL(zpci_disable_device);
*/
int zpci_hot_reset_device(struct zpci_dev *zdev)
{
- u8 status;
int rc;
lockdep_assert_held(&zdev->state_lock);
@@ -777,19 +792,9 @@ int zpci_hot_reset_device(struct zpci_dev *zdev)
return rc;
}
- rc = zpci_enable_device(zdev);
- if (rc)
- return rc;
-
- if (zdev->dma_table)
- rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table), &status);
- if (rc) {
- zpci_disable_device(zdev);
- return rc;
- }
+ rc = zpci_reenable_device(zdev);
- return 0;
+ return rc;
}
/**
diff --git a/arch/s390/pci/pci_sysfs.c b/arch/s390/pci/pci_sysfs.c
index 1f81f6ff7b95..56df97850c94 100644
--- a/arch/s390/pci/pci_sysfs.c
+++ b/arch/s390/pci/pci_sysfs.c
@@ -51,7 +51,6 @@ static DEVICE_ATTR_RO(mio_enabled);
static int _do_recover(struct pci_dev *pdev, struct zpci_dev *zdev)
{
- u8 status;
int ret;
pci_stop_and_remove_bus_device(pdev);
@@ -69,16 +68,8 @@ static int _do_recover(struct pci_dev *pdev, struct zpci_dev *zdev)
return ret;
}
- ret = zpci_enable_device(zdev);
- if (ret)
- return ret;
+ ret = zpci_reenable_device(zdev);
- if (zdev->dma_table) {
- ret = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table), &status);
- if (ret)
- zpci_disable_device(zdev);
- }
return ret;
}
diff --git a/arch/sh/mm/pmb.c b/arch/sh/mm/pmb.c
index 68eb7cc6e564..482eec50f404 100644
--- a/arch/sh/mm/pmb.c
+++ b/arch/sh/mm/pmb.c
@@ -857,7 +857,7 @@ static int __init pmb_debugfs_init(void)
subsys_initcall(pmb_debugfs_init);
#ifdef CONFIG_PM
-static void pmb_syscore_resume(void)
+static void pmb_syscore_resume(void *data)
{
struct pmb_entry *pmbe;
int i;
@@ -874,13 +874,17 @@ static void pmb_syscore_resume(void)
read_unlock(&pmb_rwlock);
}
-static struct syscore_ops pmb_syscore_ops = {
+static const struct syscore_ops pmb_syscore_ops = {
.resume = pmb_syscore_resume,
};
+static struct syscore pmb_syscore = {
+ .ops = &pmb_syscore_ops,
+};
+
static int __init pmb_sysdev_init(void)
{
- register_syscore_ops(&pmb_syscore_ops);
+ register_syscore(&pmb_syscore);
return 0;
}
subsys_initcall(pmb_sysdev_init);
diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig
index 46b53ab06165..0cf89264db08 100644
--- a/arch/x86/crypto/Kconfig
+++ b/arch/x86/crypto/Kconfig
@@ -366,7 +366,7 @@ config CRYPTO_CHACHA20_X86_64
- AVX-512VL (Advanced Vector Extensions-512VL)
config CRYPTO_AEGIS128_AESNI_SSE2
- tristate "AEAD ciphers: AEGIS-128 (AES-NI/SSE2)"
+ tristate "AEAD ciphers: AEGIS-128 (AES-NI/SSE4.1)"
depends on X86 && 64BIT
select CRYPTO_AEAD
select CRYPTO_SIMD
@@ -375,7 +375,7 @@ config CRYPTO_AEGIS128_AESNI_SSE2
Architecture: x86_64 using:
- AES-NI (AES New Instructions)
- - SSE2 (Streaming SIMD Extensions 2)
+ - SSE4.1 (Streaming SIMD Extensions 4.1)
config CRYPTO_NHPOLY1305_SSE2
tristate "Hash functions: NHPoly1305 (SSE2)"
diff --git a/arch/x86/crypto/aegis128-aesni-asm.S b/arch/x86/crypto/aegis128-aesni-asm.S
index 2de859173940..6520d0ed9814 100644
--- a/arch/x86/crypto/aegis128-aesni-asm.S
+++ b/arch/x86/crypto/aegis128-aesni-asm.S
@@ -1,13 +1,12 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * AES-NI + SSE2 implementation of AEGIS-128
+ * AES-NI + SSE4.1 implementation of AEGIS-128
*
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*/
#include <linux/linkage.h>
-#include <linux/cfi_types.h>
#include <asm/frame.h>
#define STATE0 %xmm0
@@ -20,11 +19,6 @@
#define T0 %xmm6
#define T1 %xmm7
-#define STATEP %rdi
-#define LEN %esi
-#define SRC %rdx
-#define DST %rcx
-
.section .rodata.cst16.aegis128_const, "aM", @progbits, 32
.align 16
.Laegis128_const_0:
@@ -73,6 +67,8 @@
* %r9
*/
SYM_FUNC_START_LOCAL(__load_partial)
+ .set LEN, %ecx
+ .set SRC, %rsi
xor %r9d, %r9d
pxor MSG, MSG
@@ -139,6 +135,8 @@ SYM_FUNC_END(__load_partial)
* %r10
*/
SYM_FUNC_START_LOCAL(__store_partial)
+ .set LEN, %ecx
+ .set DST, %rdx
mov LEN, %r8d
mov DST, %r9
@@ -185,16 +183,21 @@ SYM_FUNC_START_LOCAL(__store_partial)
SYM_FUNC_END(__store_partial)
/*
- * void crypto_aegis128_aesni_init(void *state, const void *key, const void *iv);
+ * void aegis128_aesni_init(struct aegis_state *state,
+ * const struct aegis_block *key,
+ * const u8 iv[AEGIS128_NONCE_SIZE]);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_init)
+SYM_FUNC_START(aegis128_aesni_init)
+ .set STATEP, %rdi
+ .set KEYP, %rsi
+ .set IVP, %rdx
FRAME_BEGIN
/* load IV: */
- movdqu (%rdx), T1
+ movdqu (IVP), T1
/* load key: */
- movdqa (%rsi), KEY
+ movdqa (KEYP), KEY
pxor KEY, T1
movdqa T1, STATE0
movdqa KEY, STATE3
@@ -227,13 +230,16 @@ SYM_FUNC_START(crypto_aegis128_aesni_init)
FRAME_END
RET
-SYM_FUNC_END(crypto_aegis128_aesni_init)
+SYM_FUNC_END(aegis128_aesni_init)
/*
- * void crypto_aegis128_aesni_ad(void *state, unsigned int length,
- * const void *data);
+ * void aegis128_aesni_ad(struct aegis_state *state, const u8 *data,
+ * unsigned int len);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_ad)
+SYM_FUNC_START(aegis128_aesni_ad)
+ .set STATEP, %rdi
+ .set SRC, %rsi
+ .set LEN, %edx
FRAME_BEGIN
cmp $0x10, LEN
@@ -379,7 +385,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad)
.Lad_out:
FRAME_END
RET
-SYM_FUNC_END(crypto_aegis128_aesni_ad)
+SYM_FUNC_END(aegis128_aesni_ad)
.macro encrypt_block a s0 s1 s2 s3 s4 i
movdq\a (\i * 0x10)(SRC), MSG
@@ -400,10 +406,14 @@ SYM_FUNC_END(crypto_aegis128_aesni_ad)
.endm
/*
- * void crypto_aegis128_aesni_enc(void *state, unsigned int length,
- * const void *src, void *dst);
+ * void aegis128_aesni_enc(struct aegis_state *state, const u8 *src, u8 *dst,
+ * unsigned int len);
*/
-SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc)
+SYM_FUNC_START(aegis128_aesni_enc)
+ .set STATEP, %rdi
+ .set SRC, %rsi
+ .set DST, %rdx
+ .set LEN, %ecx
FRAME_BEGIN
cmp $0x10, LEN
@@ -494,13 +504,17 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc)
.Lenc_out:
FRAME_END
RET
-SYM_FUNC_END(crypto_aegis128_aesni_enc)
+SYM_FUNC_END(aegis128_aesni_enc)
/*
- * void crypto_aegis128_aesni_enc_tail(void *state, unsigned int length,
- * const void *src, void *dst);
+ * void aegis128_aesni_enc_tail(struct aegis_state *state, const u8 *src,
+ * u8 *dst, unsigned int len);
*/
-SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc_tail)
+SYM_FUNC_START(aegis128_aesni_enc_tail)
+ .set STATEP, %rdi
+ .set SRC, %rsi
+ .set DST, %rdx
+ .set LEN, %ecx
FRAME_BEGIN
/* load the state: */
@@ -534,7 +548,7 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc_tail)
FRAME_END
RET
-SYM_FUNC_END(crypto_aegis128_aesni_enc_tail)
+SYM_FUNC_END(aegis128_aesni_enc_tail)
.macro decrypt_block a s0 s1 s2 s3 s4 i
movdq\a (\i * 0x10)(SRC), MSG
@@ -554,10 +568,14 @@ SYM_FUNC_END(crypto_aegis128_aesni_enc_tail)
.endm
/*
- * void crypto_aegis128_aesni_dec(void *state, unsigned int length,
- * const void *src, void *dst);
+ * void aegis128_aesni_dec(struct aegis_state *state, const u8 *src, u8 *dst,
+ * unsigned int len);
*/
-SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec)
+SYM_FUNC_START(aegis128_aesni_dec)
+ .set STATEP, %rdi
+ .set SRC, %rsi
+ .set DST, %rdx
+ .set LEN, %ecx
FRAME_BEGIN
cmp $0x10, LEN
@@ -648,13 +666,17 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec)
.Ldec_out:
FRAME_END
RET
-SYM_FUNC_END(crypto_aegis128_aesni_dec)
+SYM_FUNC_END(aegis128_aesni_dec)
/*
- * void crypto_aegis128_aesni_dec_tail(void *state, unsigned int length,
- * const void *src, void *dst);
+ * void aegis128_aesni_dec_tail(struct aegis_state *state, const u8 *src,
+ * u8 *dst, unsigned int len);
*/
-SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec_tail)
+SYM_FUNC_START(aegis128_aesni_dec_tail)
+ .set STATEP, %rdi
+ .set SRC, %rsi
+ .set DST, %rdx
+ .set LEN, %ecx
FRAME_BEGIN
/* load the state: */
@@ -698,14 +720,18 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec_tail)
FRAME_END
RET
-SYM_FUNC_END(crypto_aegis128_aesni_dec_tail)
+SYM_FUNC_END(aegis128_aesni_dec_tail)
/*
- * void crypto_aegis128_aesni_final(void *state, void *tag_xor,
- * unsigned int assoclen,
- * unsigned int cryptlen);
+ * void aegis128_aesni_final(struct aegis_state *state,
+ * struct aegis_block *tag_xor,
+ * unsigned int assoclen, unsigned int cryptlen);
*/
-SYM_FUNC_START(crypto_aegis128_aesni_final)
+SYM_FUNC_START(aegis128_aesni_final)
+ .set STATEP, %rdi
+ .set TAG_XOR, %rsi
+ .set ASSOCLEN, %edx
+ .set CRYPTLEN, %ecx
FRAME_BEGIN
/* load the state: */
@@ -716,10 +742,8 @@ SYM_FUNC_START(crypto_aegis128_aesni_final)
movdqu 0x40(STATEP), STATE4
/* prepare length block: */
- movd %edx, MSG
- movd %ecx, T0
- pslldq $8, T0
- pxor T0, MSG
+ movd ASSOCLEN, MSG
+ pinsrd $2, CRYPTLEN, MSG
psllq $3, MSG /* multiply by 8 (to get bit count) */
pxor STATE3, MSG
@@ -734,7 +758,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_final)
aegis128_update; pxor MSG, STATE3
/* xor tag: */
- movdqu (%rsi), MSG
+ movdqu (TAG_XOR), MSG
pxor STATE0, MSG
pxor STATE1, MSG
@@ -742,8 +766,8 @@ SYM_FUNC_START(crypto_aegis128_aesni_final)
pxor STATE3, MSG
pxor STATE4, MSG
- movdqu MSG, (%rsi)
+ movdqu MSG, (TAG_XOR)
FRAME_END
RET
-SYM_FUNC_END(crypto_aegis128_aesni_final)
+SYM_FUNC_END(aegis128_aesni_final)
diff --git a/arch/x86/crypto/aegis128-aesni-glue.c b/arch/x86/crypto/aegis128-aesni-glue.c
index 4623189000d8..f5bbf274a54e 100644
--- a/arch/x86/crypto/aegis128-aesni-glue.c
+++ b/arch/x86/crypto/aegis128-aesni-glue.c
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* The AEGIS-128 Authenticated-Encryption Algorithm
- * Glue for AES-NI + SSE2 implementation
+ * Glue for AES-NI + SSE4.1 implementation
*
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
@@ -23,27 +23,6 @@
#define AEGIS128_MIN_AUTH_SIZE 8
#define AEGIS128_MAX_AUTH_SIZE 16
-asmlinkage void crypto_aegis128_aesni_init(void *state, void *key, void *iv);
-
-asmlinkage void crypto_aegis128_aesni_ad(
- void *state, unsigned int length, const void *data);
-
-asmlinkage void crypto_aegis128_aesni_enc(
- void *state, unsigned int length, const void *src, void *dst);
-
-asmlinkage void crypto_aegis128_aesni_dec(
- void *state, unsigned int length, const void *src, void *dst);
-
-asmlinkage void crypto_aegis128_aesni_enc_tail(
- void *state, unsigned int length, const void *src, void *dst);
-
-asmlinkage void crypto_aegis128_aesni_dec_tail(
- void *state, unsigned int length, const void *src, void *dst);
-
-asmlinkage void crypto_aegis128_aesni_final(
- void *state, void *tag_xor, unsigned int cryptlen,
- unsigned int assoclen);
-
struct aegis_block {
u8 bytes[AEGIS128_BLOCK_SIZE] __aligned(AEGIS128_BLOCK_ALIGN);
};
@@ -56,15 +35,31 @@ struct aegis_ctx {
struct aegis_block key;
};
-struct aegis_crypt_ops {
- int (*skcipher_walk_init)(struct skcipher_walk *walk,
- struct aead_request *req, bool atomic);
+asmlinkage void aegis128_aesni_init(struct aegis_state *state,
+ const struct aegis_block *key,
+ const u8 iv[AEGIS128_NONCE_SIZE]);
- void (*crypt_blocks)(void *state, unsigned int length, const void *src,
- void *dst);
- void (*crypt_tail)(void *state, unsigned int length, const void *src,
- void *dst);
-};
+asmlinkage void aegis128_aesni_ad(struct aegis_state *state, const u8 *data,
+ unsigned int len);
+
+asmlinkage void aegis128_aesni_enc(struct aegis_state *state, const u8 *src,
+ u8 *dst, unsigned int len);
+
+asmlinkage void aegis128_aesni_dec(struct aegis_state *state, const u8 *src,
+ u8 *dst, unsigned int len);
+
+asmlinkage void aegis128_aesni_enc_tail(struct aegis_state *state,
+ const u8 *src, u8 *dst,
+ unsigned int len);
+
+asmlinkage void aegis128_aesni_dec_tail(struct aegis_state *state,
+ const u8 *src, u8 *dst,
+ unsigned int len);
+
+asmlinkage void aegis128_aesni_final(struct aegis_state *state,
+ struct aegis_block *tag_xor,
+ unsigned int assoclen,
+ unsigned int cryptlen);
static void crypto_aegis128_aesni_process_ad(
struct aegis_state *state, struct scatterlist *sg_src,
@@ -85,15 +80,14 @@ static void crypto_aegis128_aesni_process_ad(
if (pos > 0) {
unsigned int fill = AEGIS128_BLOCK_SIZE - pos;
memcpy(buf.bytes + pos, src, fill);
- crypto_aegis128_aesni_ad(state,
- AEGIS128_BLOCK_SIZE,
- buf.bytes);
+ aegis128_aesni_ad(state, buf.bytes,
+ AEGIS128_BLOCK_SIZE);
pos = 0;
left -= fill;
src += fill;
}
- crypto_aegis128_aesni_ad(state, left, src);
+ aegis128_aesni_ad(state, src, left);
src += left & ~(AEGIS128_BLOCK_SIZE - 1);
left &= AEGIS128_BLOCK_SIZE - 1;
@@ -110,26 +104,47 @@ static void crypto_aegis128_aesni_process_ad(
if (pos > 0) {
memset(buf.bytes + pos, 0, AEGIS128_BLOCK_SIZE - pos);
- crypto_aegis128_aesni_ad(state, AEGIS128_BLOCK_SIZE, buf.bytes);
+ aegis128_aesni_ad(state, buf.bytes, AEGIS128_BLOCK_SIZE);
}
}
-static void crypto_aegis128_aesni_process_crypt(
- struct aegis_state *state, struct skcipher_walk *walk,
- const struct aegis_crypt_ops *ops)
+static __always_inline int
+crypto_aegis128_aesni_process_crypt(struct aegis_state *state,
+ struct skcipher_walk *walk, bool enc)
{
+ int err = 0;
+
while (walk->nbytes >= AEGIS128_BLOCK_SIZE) {
- ops->crypt_blocks(state,
- round_down(walk->nbytes, AEGIS128_BLOCK_SIZE),
- walk->src.virt.addr, walk->dst.virt.addr);
- skcipher_walk_done(walk, walk->nbytes % AEGIS128_BLOCK_SIZE);
+ if (enc)
+ aegis128_aesni_enc(state, walk->src.virt.addr,
+ walk->dst.virt.addr,
+ round_down(walk->nbytes,
+ AEGIS128_BLOCK_SIZE));
+ else
+ aegis128_aesni_dec(state, walk->src.virt.addr,
+ walk->dst.virt.addr,
+ round_down(walk->nbytes,
+ AEGIS128_BLOCK_SIZE));
+ kernel_fpu_end();
+ err = skcipher_walk_done(walk,
+ walk->nbytes % AEGIS128_BLOCK_SIZE);
+ kernel_fpu_begin();
}
if (walk->nbytes) {
- ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
- walk->dst.virt.addr);
- skcipher_walk_done(walk, 0);
+ if (enc)
+ aegis128_aesni_enc_tail(state, walk->src.virt.addr,
+ walk->dst.virt.addr,
+ walk->nbytes);
+ else
+ aegis128_aesni_dec_tail(state, walk->src.virt.addr,
+ walk->dst.virt.addr,
+ walk->nbytes);
+ kernel_fpu_end();
+ err = skcipher_walk_done(walk, 0);
+ kernel_fpu_begin();
}
+ return err;
}
static struct aegis_ctx *crypto_aegis128_aesni_ctx(struct crypto_aead *aead)
@@ -162,42 +177,46 @@ static int crypto_aegis128_aesni_setauthsize(struct crypto_aead *tfm,
return 0;
}
-static void crypto_aegis128_aesni_crypt(struct aead_request *req,
- struct aegis_block *tag_xor,
- unsigned int cryptlen,
- const struct aegis_crypt_ops *ops)
+static __always_inline int
+crypto_aegis128_aesni_crypt(struct aead_request *req,
+ struct aegis_block *tag_xor,
+ unsigned int cryptlen, bool enc)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(tfm);
struct skcipher_walk walk;
struct aegis_state state;
+ int err;
- ops->skcipher_walk_init(&walk, req, true);
+ if (enc)
+ err = skcipher_walk_aead_encrypt(&walk, req, false);
+ else
+ err = skcipher_walk_aead_decrypt(&walk, req, false);
+ if (err)
+ return err;
kernel_fpu_begin();
- crypto_aegis128_aesni_init(&state, ctx->key.bytes, req->iv);
+ aegis128_aesni_init(&state, &ctx->key, req->iv);
crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen);
- crypto_aegis128_aesni_process_crypt(&state, &walk, ops);
- crypto_aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
-
+ err = crypto_aegis128_aesni_process_crypt(&state, &walk, enc);
+ if (err == 0)
+ aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end();
+ return err;
}
static int crypto_aegis128_aesni_encrypt(struct aead_request *req)
{
- static const struct aegis_crypt_ops OPS = {
- .skcipher_walk_init = skcipher_walk_aead_encrypt,
- .crypt_blocks = crypto_aegis128_aesni_enc,
- .crypt_tail = crypto_aegis128_aesni_enc_tail,
- };
-
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_block tag = {};
unsigned int authsize = crypto_aead_authsize(tfm);
unsigned int cryptlen = req->cryptlen;
+ int err;
- crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS);
+ err = crypto_aegis128_aesni_crypt(req, &tag, cryptlen, true);
+ if (err)
+ return err;
scatterwalk_map_and_copy(tag.bytes, req->dst,
req->assoclen + cryptlen, authsize, 1);
@@ -208,21 +227,18 @@ static int crypto_aegis128_aesni_decrypt(struct aead_request *req)
{
static const struct aegis_block zeros = {};
- static const struct aegis_crypt_ops OPS = {
- .skcipher_walk_init = skcipher_walk_aead_decrypt,
- .crypt_blocks = crypto_aegis128_aesni_dec,
- .crypt_tail = crypto_aegis128_aesni_dec_tail,
- };
-
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_block tag;
unsigned int authsize = crypto_aead_authsize(tfm);
unsigned int cryptlen = req->cryptlen - authsize;
+ int err;
scatterwalk_map_and_copy(tag.bytes, req->src,
req->assoclen + cryptlen, authsize, 0);
- crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS);
+ err = crypto_aegis128_aesni_crypt(req, &tag, cryptlen, false);
+ if (err)
+ return err;
return crypto_memneq(tag.bytes, zeros.bytes, authsize) ? -EBADMSG : 0;
}
@@ -267,7 +283,7 @@ static struct simd_aead_alg *simd_alg;
static int __init crypto_aegis128_aesni_module_init(void)
{
- if (!boot_cpu_has(X86_FEATURE_XMM2) ||
+ if (!boot_cpu_has(X86_FEATURE_XMM4_1) ||
!boot_cpu_has(X86_FEATURE_AES) ||
!cpu_has_xfeatures(XFEATURE_MASK_SSE, NULL))
return -ENODEV;
@@ -286,6 +302,6 @@ module_exit(crypto_aegis128_aesni_module_exit);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Ondrej Mosnacek <omosnacek@gmail.com>");
-MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm -- AESNI+SSE2 implementation");
+MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm -- AESNI+SSE4.1 implementation");
MODULE_ALIAS_CRYPTO("aegis128");
MODULE_ALIAS_CRYPTO("aegis128-aesni");
diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
index 1022d1dcbe44..5dde66bd94bd 100644
--- a/arch/x86/events/amd/ibs.c
+++ b/arch/x86/events/amd/ibs.c
@@ -1488,26 +1488,30 @@ static int x86_pmu_amd_ibs_starting_cpu(unsigned int cpu)
#ifdef CONFIG_PM
-static int perf_ibs_suspend(void)
+static int perf_ibs_suspend(void *data)
{
clear_APIC_ibs();
return 0;
}
-static void perf_ibs_resume(void)
+static void perf_ibs_resume(void *data)
{
ibs_eilvt_setup();
setup_APIC_ibs();
}
-static struct syscore_ops perf_ibs_syscore_ops = {
+static const struct syscore_ops perf_ibs_syscore_ops = {
.resume = perf_ibs_resume,
.suspend = perf_ibs_suspend,
};
+static struct syscore perf_ibs_syscore = {
+ .ops = &perf_ibs_syscore_ops,
+};
+
static void perf_ibs_pm_init(void)
{
- register_syscore_ops(&perf_ibs_syscore_ops);
+ register_syscore(&perf_ibs_syscore);
}
#else
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 2efb414959e1..35b6c340001d 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2606,6 +2606,27 @@ static void intel_pmu_del_event(struct perf_event *event)
intel_pmu_pebs_del(event);
}
+int __intel_pmu_quiesce(void)
+{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+ int pmu_enabled = cpuc->enabled;
+
+ cpuc->enabled = 0;
+ if (pmu_enabled)
+ intel_pmu_disable_all();
+
+ return pmu_enabled;
+}
+
+void __intel_pmu_resume(int pmu_enabled)
+{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
+
+ cpuc->enabled = pmu_enabled;
+ if (pmu_enabled)
+ intel_pmu_enable_all(0);
+}
+
static int icl_set_topdown_event_period(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
@@ -2784,24 +2805,19 @@ static void intel_pmu_read_event(struct perf_event *event)
{
if (event->hw.flags & (PERF_X86_EVENT_AUTO_RELOAD | PERF_X86_EVENT_TOPDOWN)) {
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
- bool pmu_enabled = cpuc->enabled;
+ int pmu_enabled;
/* Only need to call update_topdown_event() once for group read. */
if (is_metric_event(event) && (cpuc->txn_flags & PERF_PMU_TXN_READ))
return;
- cpuc->enabled = 0;
- if (pmu_enabled)
- intel_pmu_disable_all();
+ pmu_enabled = __intel_pmu_quiesce();
if (is_topdown_count(event))
static_call(intel_pmu_update_topdown_event)(event);
else
intel_pmu_drain_pebs_buffer();
-
- cpuc->enabled = pmu_enabled;
- if (pmu_enabled)
- intel_pmu_enable_all(0);
+ __intel_pmu_resume(pmu_enabled);
return;
}
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 4eac31693210..44c041c6a29b 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -949,8 +949,11 @@ int intel_pmu_drain_bts_buffer(void)
void intel_pmu_drain_pebs_buffer(void)
{
+ struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
struct perf_sample_data data;
+ WARN_ON_ONCE(cpuc->enabled);
+
static_call(x86_pmu_drain_pebs)(NULL, &data);
}
@@ -1448,8 +1451,11 @@ static void intel_pmu_pebs_via_pt_enable(struct perf_event *event)
static inline void intel_pmu_drain_large_pebs(struct cpu_hw_events *cpuc)
{
if (cpuc->n_pebs == cpuc->n_large_pebs &&
- cpuc->n_pebs != cpuc->n_pebs_via_pt)
+ cpuc->n_pebs != cpuc->n_pebs_via_pt) {
+ int enabled = __intel_pmu_quiesce();
intel_pmu_drain_pebs_buffer();
+ __intel_pmu_resume(enabled);
+ }
}
void intel_pmu_pebs_enable(struct perf_event *event)
diff --git a/arch/x86/events/intel/uncore.c b/arch/x86/events/intel/uncore.c
index 2736815e651c..2cc80848e318 100644
--- a/arch/x86/events/intel/uncore.c
+++ b/arch/x86/events/intel/uncore.c
@@ -1535,9 +1535,15 @@ static int uncore_event_cpu_offline(unsigned int cpu)
{
int die, target;
+ /* Clear the references */
+ die = topology_logical_die_id(cpu);
+ uncore_box_unref(uncore_msr_uncores, die);
+ uncore_box_unref(uncore_mmio_uncores, die);
+
/* Check if exiting cpu is used for collecting uncore events */
if (!cpumask_test_and_clear_cpu(cpu, &uncore_cpu_mask))
- goto unref;
+ return 0;
+
/* Find a new cpu to collect uncore events */
target = cpumask_any_but(topology_die_cpumask(cpu), cpu);
@@ -1550,16 +1556,10 @@ static int uncore_event_cpu_offline(unsigned int cpu)
uncore_change_context(uncore_msr_uncores, cpu, target);
uncore_change_context(uncore_mmio_uncores, cpu, target);
uncore_change_context(uncore_pci_uncores, cpu, target);
-
-unref:
- /* Clear the references */
- die = topology_logical_die_id(cpu);
- uncore_box_unref(uncore_msr_uncores, die);
- uncore_box_unref(uncore_mmio_uncores, die);
return 0;
}
-static int allocate_boxes(struct intel_uncore_type **types,
+static void allocate_boxes(struct intel_uncore_type **types,
unsigned int die, unsigned int cpu)
{
struct intel_uncore_box *box, *tmp;
@@ -1588,14 +1588,13 @@ static int allocate_boxes(struct intel_uncore_type **types,
list_del_init(&box->active_list);
box->pmu->boxes[die] = box;
}
- return 0;
+ return;
cleanup:
list_for_each_entry_safe(box, tmp, &allocated, active_list) {
list_del_init(&box->active_list);
kfree(box);
}
- return -ENOMEM;
}
static int uncore_box_ref(struct intel_uncore_type **types,
@@ -1604,11 +1603,7 @@ static int uncore_box_ref(struct intel_uncore_type **types,
struct intel_uncore_type *type;
struct intel_uncore_pmu *pmu;
struct intel_uncore_box *box;
- int i, ret;
-
- ret = allocate_boxes(types, id, cpu);
- if (ret)
- return ret;
+ int i;
for (; *types; types++) {
type = *types;
@@ -1624,27 +1619,26 @@ static int uncore_box_ref(struct intel_uncore_type **types,
static int uncore_event_cpu_online(unsigned int cpu)
{
- int die, target, msr_ret, mmio_ret;
+ int die, target;
die = topology_logical_die_id(cpu);
- msr_ret = uncore_box_ref(uncore_msr_uncores, die, cpu);
- mmio_ret = uncore_box_ref(uncore_mmio_uncores, die, cpu);
+ allocate_boxes(uncore_msr_uncores, die, cpu);
+ allocate_boxes(uncore_mmio_uncores, die, cpu);
/*
* Check if there is an online cpu in the package
* which collects uncore events already.
*/
target = cpumask_any_and(&uncore_cpu_mask, topology_die_cpumask(cpu));
- if (target < nr_cpu_ids)
- return 0;
-
- cpumask_set_cpu(cpu, &uncore_cpu_mask);
-
- if (!msr_ret)
+ if (target >= nr_cpu_ids) {
+ cpumask_set_cpu(cpu, &uncore_cpu_mask);
uncore_change_context(uncore_msr_uncores, -1, cpu);
- if (!mmio_ret)
uncore_change_context(uncore_mmio_uncores, -1, cpu);
- uncore_change_context(uncore_pci_uncores, -1, cpu);
+ uncore_change_context(uncore_pci_uncores, -1, cpu);
+ }
+
+ uncore_box_ref(uncore_msr_uncores, die, cpu);
+ uncore_box_ref(uncore_mmio_uncores, die, cpu);
return 0;
}
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 3878173933be..85afab67414a 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -1560,6 +1560,9 @@ static __always_inline void __intel_pmu_lbr_disable(void)
wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
}
+extern int __intel_pmu_quiesce(void);
+extern void __intel_pmu_resume(int pmu_enabled);
+
int intel_pmu_save_and_restart(struct perf_event *event);
struct event_constraint *
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index 239f612816e6..5d7e5b53e33e 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -316,7 +316,7 @@ static int __init hv_pci_init(void)
return 1;
}
-static int hv_suspend(void)
+static int hv_suspend(void *data)
{
union hv_x64_msr_hypercall_contents hypercall_msr;
int ret;
@@ -343,7 +343,7 @@ static int hv_suspend(void)
return ret;
}
-static void hv_resume(void)
+static void hv_resume(void *data)
{
union hv_x64_msr_hypercall_contents hypercall_msr;
int ret;
@@ -370,11 +370,15 @@ static void hv_resume(void)
}
/* Note: when the ops are called, only CPU0 is online and IRQs are disabled. */
-static struct syscore_ops hv_syscore_ops = {
+static const struct syscore_ops hv_syscore_ops = {
.suspend = hv_suspend,
.resume = hv_resume,
};
+static struct syscore hv_syscore = {
+ .ops = &hv_syscore_ops,
+};
+
static void (* __initdata old_setup_percpu_clockev)(void);
static void __init hv_stimer_setup_percpu_clockev(void)
@@ -604,7 +608,7 @@ void __init hyperv_init(void)
x86_init.pci.arch_init = hv_pci_init;
- register_syscore_ops(&hv_syscore_ops);
+ register_syscore(&hv_syscore);
if (cpuid_ebx(HYPERV_CPUID_FEATURES) & HV_ACCESS_PARTITION_ID)
hv_get_partition_id();
diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index c884deca839b..30a1916b0c4e 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -591,7 +591,7 @@ static void gart_fixup_northbridges(void)
}
}
-static void gart_resume(void)
+static void gart_resume(void *data)
{
pr_info("PCI-DMA: Resuming GART IOMMU\n");
@@ -600,11 +600,15 @@ static void gart_resume(void)
enable_gart_translations();
}
-static struct syscore_ops gart_syscore_ops = {
+static const struct syscore_ops gart_syscore_ops = {
.resume = gart_resume,
};
+static struct syscore gart_syscore = {
+ .ops = &gart_syscore_ops,
+};
+
/*
* Private Northbridge GATT initialization in case we cannot use the
* AGP driver for some reason.
@@ -650,7 +654,7 @@ static __init int init_amd_gatt(struct agp_kern_info *info)
agp_gatt_table = gatt;
- register_syscore_ops(&gart_syscore_ops);
+ register_syscore(&gart_syscore);
flush_gart();
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 21dc37cca20e..65c521d74ab8 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2379,7 +2379,7 @@ static struct {
unsigned int apic_cmci;
} apic_pm_state;
-static int lapic_suspend(void)
+static int lapic_suspend(void *data)
{
unsigned long flags;
int maxlvt;
@@ -2427,7 +2427,7 @@ static int lapic_suspend(void)
return 0;
}
-static void lapic_resume(void)
+static void lapic_resume(void *data)
{
unsigned int l, h;
unsigned long flags;
@@ -2507,11 +2507,15 @@ static void lapic_resume(void)
* are needed on every CPU up until machine_halt/restart/poweroff.
*/
-static struct syscore_ops lapic_syscore_ops = {
+static const struct syscore_ops lapic_syscore_ops = {
.resume = lapic_resume,
.suspend = lapic_suspend,
};
+static struct syscore lapic_syscore = {
+ .ops = &lapic_syscore_ops,
+};
+
static void apic_pm_activate(void)
{
apic_pm_state.active = 1;
@@ -2521,7 +2525,7 @@ static int __init init_lapic_sysfs(void)
{
/* XXX: remove suspend/resume procs if !apic_pm_state.active? */
if (boot_cpu_has(X86_FEATURE_APIC))
- register_syscore_ops(&lapic_syscore_ops);
+ register_syscore(&lapic_syscore);
return 0;
}
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 1029ea4ac8ba..2f478d8ef562 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2308,7 +2308,12 @@ static void resume_ioapic_id(int ioapic_idx)
}
}
-static void ioapic_resume(void)
+static int ioapic_suspend(void *data)
+{
+ return save_ioapic_entries();
+}
+
+static void ioapic_resume(void *data)
{
int ioapic_idx;
@@ -2318,14 +2323,18 @@ static void ioapic_resume(void)
restore_ioapic_entries();
}
-static struct syscore_ops ioapic_syscore_ops = {
- .suspend = save_ioapic_entries,
+static const struct syscore_ops ioapic_syscore_ops = {
+ .suspend = ioapic_suspend,
.resume = ioapic_resume,
};
+static struct syscore ioapic_syscore = {
+ .ops = &ioapic_syscore_ops,
+};
+
static int __init ioapic_init_ops(void)
{
- register_syscore_ops(&ioapic_syscore_ops);
+ register_syscore(&ioapic_syscore);
return 0;
}
diff --git a/arch/x86/kernel/cpu/aperfmperf.c b/arch/x86/kernel/cpu/aperfmperf.c
index f642de2ebdac..e7c7b42459df 100644
--- a/arch/x86/kernel/cpu/aperfmperf.c
+++ b/arch/x86/kernel/cpu/aperfmperf.c
@@ -36,7 +36,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct aperfmperf, cpu_samples) = {
.seq = SEQCNT_ZERO(cpu_samples.seq)
};
-static void init_counter_refs(void)
+static void init_counter_refs(void *data)
{
u64 aperf, mperf;
@@ -288,16 +288,20 @@ static bool __init intel_set_max_freq_ratio(void)
}
#ifdef CONFIG_PM_SLEEP
-static struct syscore_ops freq_invariance_syscore_ops = {
+static const struct syscore_ops freq_invariance_syscore_ops = {
.resume = init_counter_refs,
};
-static void register_freq_invariance_syscore_ops(void)
+static struct syscore freq_invariance_syscore = {
+ .ops = &freq_invariance_syscore_ops,
+};
+
+static void register_freq_invariance_syscore(void)
{
- register_syscore_ops(&freq_invariance_syscore_ops);
+ register_syscore(&freq_invariance_syscore);
}
#else
-static inline void register_freq_invariance_syscore_ops(void) {}
+static inline void register_freq_invariance_syscore(void) {}
#endif
static void freq_invariance_enable(void)
@@ -307,7 +311,7 @@ static void freq_invariance_enable(void)
return;
}
static_branch_enable_cpuslocked(&arch_scale_freq_key);
- register_freq_invariance_syscore_ops();
+ register_freq_invariance_syscore();
pr_info("Estimated ratio of average max frequency by base frequency (times 1024): %llu\n", arch_max_freq_ratio);
}
@@ -534,7 +538,7 @@ static int __init bp_init_aperfmperf(void)
if (!cpu_feature_enabled(X86_FEATURE_APERFMPERF))
return 0;
- init_counter_refs();
+ init_counter_refs(NULL);
bp_init_freq_invariance();
return 0;
}
@@ -543,5 +547,5 @@ early_initcall(bp_init_aperfmperf);
void ap_init_aperfmperf(void)
{
if (cpu_feature_enabled(X86_FEATURE_APERFMPERF))
- init_counter_refs();
+ init_counter_refs(NULL);
}
diff --git a/arch/x86/kernel/cpu/intel_epb.c b/arch/x86/kernel/cpu/intel_epb.c
index 30b1d63b97f3..91b9020d08b6 100644
--- a/arch/x86/kernel/cpu/intel_epb.c
+++ b/arch/x86/kernel/cpu/intel_epb.c
@@ -75,7 +75,7 @@ static u8 energ_perf_values[] = {
[EPB_INDEX_POWERSAVE] = ENERGY_PERF_BIAS_POWERSAVE,
};
-static int intel_epb_save(void)
+static int intel_epb_save(void *data)
{
u64 epb;
@@ -89,7 +89,7 @@ static int intel_epb_save(void)
return 0;
}
-static void intel_epb_restore(void)
+static void intel_epb_restore(void *data)
{
u64 val = this_cpu_read(saved_epb);
u64 epb;
@@ -114,11 +114,15 @@ static void intel_epb_restore(void)
wrmsrl(MSR_IA32_ENERGY_PERF_BIAS, (epb & ~EPB_MASK) | val);
}
-static struct syscore_ops intel_epb_syscore_ops = {
+static const struct syscore_ops intel_epb_syscore_ops = {
.suspend = intel_epb_save,
.resume = intel_epb_restore,
};
+static struct syscore intel_epb_syscore = {
+ .ops = &intel_epb_syscore_ops,
+};
+
static const char * const energy_perf_strings[] = {
[EPB_INDEX_PERFORMANCE] = "performance",
[EPB_INDEX_BALANCE_PERFORMANCE] = "balance-performance",
@@ -185,7 +189,7 @@ static int intel_epb_online(unsigned int cpu)
{
struct device *cpu_dev = get_cpu_device(cpu);
- intel_epb_restore();
+ intel_epb_restore(NULL);
if (!cpuhp_tasks_frozen)
sysfs_merge_group(&cpu_dev->kobj, &intel_epb_attr_group);
@@ -199,7 +203,7 @@ static int intel_epb_offline(unsigned int cpu)
if (!cpuhp_tasks_frozen)
sysfs_unmerge_group(&cpu_dev->kobj, &intel_epb_attr_group);
- intel_epb_save();
+ intel_epb_save(NULL);
return 0;
}
@@ -230,7 +234,7 @@ static __init int intel_epb_init(void)
if (ret < 0)
goto err_out_online;
- register_syscore_ops(&intel_epb_syscore_ops);
+ register_syscore(&intel_epb_syscore);
return 0;
err_out_online:
diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index 7be30b377578..6e6cb9dec9a7 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -2396,13 +2396,13 @@ static void vendor_disable_error_reporting(void)
mce_disable_error_reporting();
}
-static int mce_syscore_suspend(void)
+static int mce_syscore_suspend(void *data)
{
vendor_disable_error_reporting();
return 0;
}
-static void mce_syscore_shutdown(void)
+static void mce_syscore_shutdown(void *data)
{
vendor_disable_error_reporting();
}
@@ -2412,7 +2412,7 @@ static void mce_syscore_shutdown(void)
* Only one CPU is active at this time, the others get re-added later using
* CPU hotplug:
*/
-static void mce_syscore_resume(void)
+static void mce_syscore_resume(void *data)
{
__mcheck_cpu_init_generic();
__mcheck_cpu_init_vendor(raw_cpu_ptr(&cpu_info));
@@ -2420,12 +2420,16 @@ static void mce_syscore_resume(void)
cr4_set_bits(X86_CR4_MCE);
}
-static struct syscore_ops mce_syscore_ops = {
+static const struct syscore_ops mce_syscore_ops = {
.suspend = mce_syscore_suspend,
.shutdown = mce_syscore_shutdown,
.resume = mce_syscore_resume,
};
+static struct syscore mce_syscore = {
+ .ops = &mce_syscore_ops,
+};
+
/*
* mce_device: Sysfs support
*/
@@ -2826,7 +2830,7 @@ static __init int mcheck_init_device(void)
if (err < 0)
goto err_out_online;
- register_syscore_ops(&mce_syscore_ops);
+ register_syscore(&mce_syscore);
return 0;
diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index e8021d3e5882..7809380e81a9 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -774,8 +774,17 @@ void microcode_bsp_resume(void)
reload_early_microcode(cpu);
}
-static struct syscore_ops mc_syscore_ops = {
- .resume = microcode_bsp_resume,
+static void microcode_bsp_syscore_resume(void *data)
+{
+ microcode_bsp_resume();
+}
+
+static const struct syscore_ops mc_syscore_ops = {
+ .resume = microcode_bsp_syscore_resume,
+};
+
+static struct syscore mc_syscore = {
+ .ops = &mc_syscore_ops,
};
static int mc_cpu_online(unsigned int cpu)
@@ -854,7 +863,7 @@ static int __init microcode_init(void)
}
}
- register_syscore_ops(&mc_syscore_ops);
+ register_syscore(&mc_syscore);
cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "x86/microcode:online",
mc_cpu_online, mc_cpu_down_prep);
diff --git a/arch/x86/kernel/cpu/mtrr/legacy.c b/arch/x86/kernel/cpu/mtrr/legacy.c
index d25882fcf181..2415ffaaf02c 100644
--- a/arch/x86/kernel/cpu/mtrr/legacy.c
+++ b/arch/x86/kernel/cpu/mtrr/legacy.c
@@ -41,7 +41,7 @@ struct mtrr_value {
static struct mtrr_value *mtrr_value;
-static int mtrr_save(void)
+static int mtrr_save(void *data)
{
int i;
@@ -56,7 +56,7 @@ static int mtrr_save(void)
return 0;
}
-static void mtrr_restore(void)
+static void mtrr_restore(void *data)
{
int i;
@@ -69,11 +69,15 @@ static void mtrr_restore(void)
}
}
-static struct syscore_ops mtrr_syscore_ops = {
+static const struct syscore_ops mtrr_syscore_ops = {
.suspend = mtrr_save,
.resume = mtrr_restore,
};
+static struct syscore mtrr_syscore = {
+ .ops = &mtrr_syscore_ops,
+};
+
void mtrr_register_syscore(void)
{
mtrr_value = kcalloc(num_var_ranges, sizeof(*mtrr_value), GFP_KERNEL);
@@ -86,5 +90,5 @@ void mtrr_register_syscore(void)
* TBD: is there any system with such CPU which supports
* suspend/resume? If no, we should remove the code.
*/
- register_syscore_ops(&mtrr_syscore_ops);
+ register_syscore(&mtrr_syscore);
}
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 3db8992d1ae5..868e260947a1 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -2447,15 +2447,24 @@ static void rdtgroup_kn_get(struct rdtgroup *rdtgrp, struct kernfs_node *kn)
static void rdtgroup_kn_put(struct rdtgroup *rdtgrp, struct kernfs_node *kn)
{
- if (atomic_dec_and_test(&rdtgrp->waitcount) &&
- (rdtgrp->flags & RDT_DELETED)) {
+ bool needs_free;
+
+ if (!atomic_dec_and_mutex_lock(&rdtgrp->waitcount, &rdtgroup_mutex)) {
+ kernfs_unbreak_active_protection(kn);
+ return;
+ }
+
+ needs_free = rdtgrp->flags & RDT_DELETED;
+
+ mutex_unlock(&rdtgroup_mutex);
+
+ kernfs_unbreak_active_protection(kn);
+
+ if (needs_free) {
if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP ||
rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED)
rdtgroup_pseudo_lock_remove(rdtgrp);
- kernfs_unbreak_active_protection(kn);
rdtgroup_remove(rdtgrp);
- } else {
- kernfs_unbreak_active_protection(kn);
}
}
diff --git a/arch/x86/kernel/cpu/umwait.c b/arch/x86/kernel/cpu/umwait.c
index 2293efd6ffa6..66ed3802bd13 100644
--- a/arch/x86/kernel/cpu/umwait.c
+++ b/arch/x86/kernel/cpu/umwait.c
@@ -86,15 +86,19 @@ static int umwait_cpu_offline(unsigned int cpu)
* trust the firmware nor does it matter if the same value is written
* again.
*/
-static void umwait_syscore_resume(void)
+static void umwait_syscore_resume(void *data)
{
umwait_update_control_msr(NULL);
}
-static struct syscore_ops umwait_syscore_ops = {
+static const struct syscore_ops umwait_syscore_ops = {
.resume = umwait_syscore_resume,
};
+static struct syscore umwait_syscore = {
+ .ops = &umwait_syscore_ops,
+};
+
/* sysfs interface */
/*
@@ -226,7 +230,7 @@ static int __init umwait_init(void)
return ret;
}
- register_syscore_ops(&umwait_syscore_ops);
+ register_syscore(&umwait_syscore);
/*
* Add umwait control interface. Ignore failure, so at least the
diff --git a/arch/x86/kernel/i8237.c b/arch/x86/kernel/i8237.c
index 2cd124ad9380..896d46b44284 100644
--- a/arch/x86/kernel/i8237.c
+++ b/arch/x86/kernel/i8237.c
@@ -19,7 +19,7 @@
* in asm/dma.h.
*/
-static void i8237A_resume(void)
+static void i8237A_resume(void *data)
{
unsigned long flags;
int i;
@@ -41,10 +41,14 @@ static void i8237A_resume(void)
release_dma_lock(flags);
}
-static struct syscore_ops i8237_syscore_ops = {
+static const struct syscore_ops i8237_syscore_ops = {
.resume = i8237A_resume,
};
+static struct syscore i8237_syscore = {
+ .ops = &i8237_syscore_ops,
+};
+
static int __init i8237A_init_ops(void)
{
/*
@@ -70,7 +74,7 @@ static int __init i8237A_init_ops(void)
if (x86_pnpbios_disabled() && dmi_get_bios_year() >= 2017)
return -ENODEV;
- register_syscore_ops(&i8237_syscore_ops);
+ register_syscore(&i8237_syscore);
return 0;
}
device_initcall(i8237A_init_ops);
diff --git a/arch/x86/kernel/i8259.c b/arch/x86/kernel/i8259.c
index c20d1832c481..73200f492e11 100644
--- a/arch/x86/kernel/i8259.c
+++ b/arch/x86/kernel/i8259.c
@@ -246,19 +246,19 @@ static void save_ELCR(char *trigger)
trigger[1] = inb(PIC_ELCR2) & 0xDE;
}
-static void i8259A_resume(void)
+static void i8259A_resume(void *data)
{
init_8259A(i8259A_auto_eoi);
restore_ELCR(irq_trigger);
}
-static int i8259A_suspend(void)
+static int i8259A_suspend(void *data)
{
save_ELCR(irq_trigger);
return 0;
}
-static void i8259A_shutdown(void)
+static void i8259A_shutdown(void *data)
{
/* Put the i8259A into a quiescent state that
* the kernel initialization code can get it
@@ -268,12 +268,16 @@ static void i8259A_shutdown(void)
outb(0xff, PIC_SLAVE_IMR); /* mask all of 8259A-2 */
}
-static struct syscore_ops i8259_syscore_ops = {
+static const struct syscore_ops i8259_syscore_ops = {
.suspend = i8259A_suspend,
.resume = i8259A_resume,
.shutdown = i8259A_shutdown,
};
+static struct syscore i8259_syscore = {
+ .ops = &i8259_syscore_ops,
+};
+
static void mask_8259A(void)
{
unsigned long flags;
@@ -443,7 +447,7 @@ EXPORT_SYMBOL(legacy_pic);
static int __init i8259A_init_ops(void)
{
if (legacy_pic == &default_legacy_pic)
- register_syscore_ops(&i8259_syscore_ops);
+ register_syscore(&i8259_syscore);
return 0;
}
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 46887d60348e..a4c78026eea0 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -720,7 +720,7 @@ static int kvm_cpu_down_prepare(unsigned int cpu)
#endif
-static int kvm_suspend(void)
+static int kvm_suspend(void *data)
{
u64 val = 0;
@@ -734,7 +734,7 @@ static int kvm_suspend(void)
return 0;
}
-static void kvm_resume(void)
+static void kvm_resume(void *data)
{
kvm_cpu_online(raw_smp_processor_id());
@@ -744,11 +744,15 @@ static void kvm_resume(void)
#endif
}
-static struct syscore_ops kvm_syscore_ops = {
+static const struct syscore_ops kvm_syscore_ops = {
.suspend = kvm_suspend,
.resume = kvm_resume,
};
+static struct syscore kvm_syscore = {
+ .ops = &kvm_syscore_ops,
+};
+
static void kvm_pv_guest_cpu_reboot(void *unused)
{
kvm_guest_cpu_offline(true);
@@ -859,7 +863,7 @@ static void __init kvm_guest_init(void)
machine_ops.crash_shutdown = kvm_crash_shutdown;
#endif
- register_syscore_ops(&kvm_syscore_ops);
+ register_syscore(&kvm_syscore);
/*
* Hard lockup detection is enabled by default. Disable it, as guests
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index a7a1486ce270..e2b55b7f6497 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -392,7 +392,7 @@ static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
allow_gbpages = tdp_enabled ? boot_cpu_has(X86_FEATURE_GBPAGES) :
guest_cpuid_has(vcpu, X86_FEATURE_GBPAGES);
if (allow_gbpages)
- kvm_governed_feature_set(vcpu, X86_FEATURE_GBPAGES);
+ guest_cpu_cap_set(vcpu, X86_FEATURE_GBPAGES);
best = kvm_find_cpuid_entry(vcpu, 1);
if (best && apic) {
diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h
index f16a7b2c2adc..1a8cc377d256 100644
--- a/arch/x86/kvm/cpuid.h
+++ b/arch/x86/kvm/cpuid.h
@@ -241,8 +241,8 @@ static __always_inline bool kvm_is_governed_feature(unsigned int x86_feature)
return kvm_governed_feature_index(x86_feature) >= 0;
}
-static __always_inline void kvm_governed_feature_set(struct kvm_vcpu *vcpu,
- unsigned int x86_feature)
+static __always_inline void guest_cpu_cap_set(struct kvm_vcpu *vcpu,
+ unsigned int x86_feature)
{
BUILD_BUG_ON(!kvm_is_governed_feature(x86_feature));
@@ -250,15 +250,15 @@ static __always_inline void kvm_governed_feature_set(struct kvm_vcpu *vcpu,
vcpu->arch.governed_features.enabled);
}
-static __always_inline void kvm_governed_feature_check_and_set(struct kvm_vcpu *vcpu,
- unsigned int x86_feature)
+static __always_inline void guest_cpu_cap_check_and_set(struct kvm_vcpu *vcpu,
+ unsigned int x86_feature)
{
if (kvm_cpu_cap_has(x86_feature) && guest_cpuid_has(vcpu, x86_feature))
- kvm_governed_feature_set(vcpu, x86_feature);
+ guest_cpu_cap_set(vcpu, x86_feature);
}
-static __always_inline bool guest_can_use(struct kvm_vcpu *vcpu,
- unsigned int x86_feature)
+static __always_inline bool guest_cpu_cap_has(struct kvm_vcpu *vcpu,
+ unsigned int x86_feature)
{
BUILD_BUG_ON(!kvm_is_governed_feature(x86_feature));
@@ -268,7 +268,7 @@ static __always_inline bool guest_can_use(struct kvm_vcpu *vcpu,
static inline bool kvm_vcpu_is_legal_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
{
- if (guest_can_use(vcpu, X86_FEATURE_LAM))
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_LAM))
cr3 &= ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57);
return kvm_vcpu_is_legal_gpa(vcpu, cr3);
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index a7d38b8cbd75..6e6d00b3c042 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -994,6 +994,8 @@ static void apic_update_ppr(struct kvm_lapic *apic)
if (__apic_update_ppr(apic, &ppr) &&
apic_has_interrupt_for_ppr(apic, ppr) != -1)
kvm_make_request(KVM_REQ_EVENT, apic->vcpu);
+ else
+ kvm_lapic_update_cr8_intercept(apic->vcpu);
}
void kvm_apic_update_ppr(struct kvm_vcpu *vcpu)
@@ -2608,7 +2610,33 @@ u64 kvm_lapic_get_cr8(struct kvm_vcpu *vcpu)
return (tpr & 0xf0) >> 4;
}
-void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value)
+void kvm_lapic_update_cr8_intercept(struct kvm_vcpu *vcpu)
+{
+ int max_irr, tpr;
+
+ if (!kvm_x86_ops.update_cr8_intercept)
+ return;
+
+ if (!lapic_in_kernel(vcpu))
+ return;
+
+ if (vcpu->arch.apic->apicv_active)
+ return;
+
+ if (!vcpu->arch.apic->vapic_addr)
+ max_irr = kvm_lapic_find_highest_irr(vcpu);
+ else
+ max_irr = -1;
+
+ if (max_irr != -1)
+ max_irr >>= 4;
+
+ tpr = kvm_lapic_get_cr8(vcpu);
+
+ kvm_x86_call(update_cr8_intercept)(vcpu, tpr, max_irr);
+}
+
+static void __kvm_apic_set_base(struct kvm_vcpu *vcpu, u64 value)
{
u64 old_value = vcpu->arch.apic_base;
struct kvm_lapic *apic = vcpu->arch.apic;
@@ -2656,6 +2684,27 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value)
}
}
+int kvm_apic_set_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
+{
+ enum lapic_mode old_mode = kvm_get_apic_mode(vcpu);
+ enum lapic_mode new_mode = kvm_apic_mode(msr_info->data);
+ u64 reserved_bits = kvm_vcpu_reserved_gpa_bits_raw(vcpu) | 0x2ff |
+ (guest_cpuid_has(vcpu, X86_FEATURE_X2APIC) ? 0 : X2APIC_ENABLE);
+
+ if ((msr_info->data & reserved_bits) != 0 || new_mode == LAPIC_MODE_INVALID)
+ return 1;
+ if (!msr_info->host_initiated) {
+ if (old_mode == LAPIC_MODE_X2APIC && new_mode == LAPIC_MODE_XAPIC)
+ return 1;
+ if (old_mode == LAPIC_MODE_DISABLED && new_mode == LAPIC_MODE_X2APIC)
+ return 1;
+ }
+
+ __kvm_apic_set_base(vcpu, msr_info->data);
+ kvm_recalculate_apic_map(vcpu->kvm);
+ return 0;
+}
+
void kvm_apic_update_apicv(struct kvm_vcpu *vcpu)
{
struct kvm_lapic *apic = vcpu->arch.apic;
@@ -2766,7 +2815,14 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)
msr_val = APIC_DEFAULT_PHYS_BASE | MSR_IA32_APICBASE_ENABLE;
if (kvm_vcpu_is_reset_bsp(vcpu))
msr_val |= MSR_IA32_APICBASE_BSP;
- kvm_lapic_set_base(vcpu, msr_val);
+
+ /*
+ * Use the inner helper to avoid an extra recalcuation of the
+ * optimized APIC map if some other task has dirtied the map.
+ * The recalculation needed for this vCPU will be done after
+ * all APIC state has been initialized (see below).
+ */
+ __kvm_apic_set_base(vcpu, msr_val);
}
if (!apic)
@@ -3107,7 +3163,6 @@ int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s)
kvm_x86_call(apicv_pre_state_restore)(vcpu);
- kvm_lapic_set_base(vcpu, vcpu->arch.apic_base);
/* set SPIV separately to get count of SW disabled APICs right */
apic_set_spiv(apic, *((u32 *)(s->regs + APIC_SPIV)));
diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h
index 3aa599db7796..a0668b0b1d36 100644
--- a/arch/x86/kvm/lapic.h
+++ b/arch/x86/kvm/lapic.h
@@ -94,8 +94,8 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu);
void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event);
u64 kvm_lapic_get_cr8(struct kvm_vcpu *vcpu);
void kvm_lapic_set_tpr(struct kvm_vcpu *vcpu, unsigned long cr8);
+void kvm_lapic_update_cr8_intercept(struct kvm_vcpu *vcpu);
void kvm_lapic_set_eoi(struct kvm_vcpu *vcpu);
-void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value);
void kvm_recalculate_apic_map(struct kvm *kvm);
void kvm_apic_set_version(struct kvm_vcpu *vcpu);
void kvm_apic_after_set_mcg_cap(struct kvm_vcpu *vcpu);
@@ -117,7 +117,7 @@ bool kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *src,
struct kvm_lapic_irq *irq, int *r, struct dest_map *dest_map);
void kvm_apic_send_ipi(struct kvm_lapic *apic, u32 icr_low, u32 icr_high);
-int kvm_set_apic_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
+int kvm_apic_set_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
int kvm_apic_get_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s);
int kvm_apic_set_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s);
void kvm_apic_update_hwapic_isr(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 1d9c919c9232..838893f16663 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -126,7 +126,7 @@ static inline unsigned long kvm_get_active_pcid(struct kvm_vcpu *vcpu)
static inline unsigned long kvm_get_active_cr3_lam_bits(struct kvm_vcpu *vcpu)
{
- if (!guest_can_use(vcpu, X86_FEATURE_LAM))
+ if (!guest_cpu_cap_has(vcpu, X86_FEATURE_LAM))
return 0;
return kvm_read_cr3(vcpu) & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57);
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 1a4d5fa11922..ab6d3f565b0a 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5110,7 +5110,7 @@ static void reset_guest_rsvds_bits_mask(struct kvm_vcpu *vcpu,
__reset_rsvds_bits_mask(&context->guest_rsvd_check,
vcpu->arch.reserved_gpa_bits,
context->cpu_role.base.level, is_efer_nx(context),
- guest_can_use(vcpu, X86_FEATURE_GBPAGES),
+ guest_cpu_cap_has(vcpu, X86_FEATURE_GBPAGES),
is_cr4_pse(context),
guest_cpuid_is_amd_compatible(vcpu));
}
@@ -5187,7 +5187,7 @@ static void reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu,
__reset_rsvds_bits_mask(shadow_zero_check, reserved_hpa_bits(),
context->root_role.level,
context->root_role.efer_nx,
- guest_can_use(vcpu, X86_FEATURE_GBPAGES),
+ guest_cpu_cap_has(vcpu, X86_FEATURE_GBPAGES),
is_pse, is_amd);
if (!shadow_me_mask)
diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
index 26ee933ef549..49afacc9b099 100644
--- a/arch/x86/kvm/regs.c
+++ b/arch/x86/kvm/regs.c
@@ -572,7 +572,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
apic_base_msr.data = sregs->apic_base;
apic_base_msr.host_initiated = true;
- if (kvm_set_apic_base(vcpu, &apic_base_msr))
+ if (kvm_apic_set_base(vcpu, &apic_base_msr))
return -EINVAL;
if (vcpu->arch.guest_state_protected)
@@ -591,8 +591,6 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3);
- kvm_set_cr8(vcpu, sregs->cr8);
-
*mmu_reset_needed |= vcpu->arch.efer != sregs->efer;
kvm_x86_call(set_efer)(vcpu, sregs->efer);
@@ -621,7 +619,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR);
kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR);
- update_cr8_intercept(vcpu);
+ kvm_set_cr8(vcpu, sregs->cr8);
/* Older userspace won't unhalt the vcpu on reset. */
if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 &&
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 2d81f1cb4ca7..4aee865718fe 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -111,7 +111,7 @@ static void nested_svm_uninit_mmu_context(struct kvm_vcpu *vcpu)
static bool nested_vmcb_needs_vls_intercept(struct vcpu_svm *svm)
{
- if (!guest_can_use(&svm->vcpu, X86_FEATURE_V_VMSAVE_VMLOAD))
+ if (!guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_V_VMSAVE_VMLOAD))
return true;
if (!nested_npt_enabled(svm))
@@ -610,7 +610,7 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12
vmcb_mark_dirty(vmcb02, VMCB_DR);
}
- if (unlikely(guest_can_use(vcpu, X86_FEATURE_LBRV) &&
+ if (unlikely(guest_cpu_cap_has(vcpu, X86_FEATURE_LBRV) &&
(svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) {
/*
* Reserved bits of DEBUGCTL are ignored. Be consistent with
@@ -621,6 +621,7 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12
} else {
svm_copy_lbrs(vmcb02, vmcb01);
}
+ vmcb_mark_dirty(vmcb02, VMCB_LBR);
svm_update_lbrv(&svm->vcpu);
}
@@ -671,7 +672,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
* exit_int_info, exit_int_info_err, next_rip, insn_len, insn_bytes.
*/
- if (guest_can_use(vcpu, X86_FEATURE_VGIF) &&
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
(svm->nested.ctl.int_ctl & V_GIF_ENABLE_MASK))
int_ctl_vmcb12_bits |= (V_GIF_MASK | V_GIF_ENABLE_MASK);
else
@@ -710,7 +711,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
vmcb02->control.tsc_offset = vcpu->arch.tsc_offset;
- if (guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR) &&
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_TSCRATEMSR) &&
svm->tsc_ratio_msr != kvm_caps.default_tsc_scaling_ratio)
nested_svm_update_tsc_ratio_msr(vcpu);
@@ -724,24 +725,32 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
vmcb02->control.event_inj_err = svm->nested.ctl.event_inj_err;
/*
- * next_rip is consumed on VMRUN as the return address pushed on the
+ * NextRIP is consumed on VMRUN as the return address pushed on the
* stack for injected soft exceptions/interrupts. If nrips is exposed
- * to L1, take it verbatim from vmcb12. If nrips is supported in
- * hardware but not exposed to L1, stuff the actual L2 RIP to emulate
- * what a nrips=0 CPU would do (L1 is responsible for advancing RIP
- * prior to injecting the event).
+ * to L1, take it verbatim from vmcb12.
+ *
+ * If nrips is supported in hardware but not exposed to L1, stuff the
+ * actual L2 RIP to emulate what a nrips=0 CPU would do (L1 is
+ * responsible for advancing RIP prior to injecting the event). This is
+ * only the case for the first L2 run after VMRUN. After that (e.g.
+ * during save/restore), NextRIP is updated by the CPU and/or KVM, and
+ * the value of the L2 RIP from vmcb12 should not be used.
*/
- if (guest_can_use(vcpu, X86_FEATURE_NRIPS))
- vmcb02->control.next_rip = svm->nested.ctl.next_rip;
- else if (boot_cpu_has(X86_FEATURE_NRIPS))
- vmcb02->control.next_rip = vmcb12_rip;
+ if (boot_cpu_has(X86_FEATURE_NRIPS)) {
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS) ||
+ !svm->nested.nested_run_pending)
+ vmcb02->control.next_rip = svm->nested.ctl.next_rip;
+ else
+ vmcb02->control.next_rip = vmcb12_rip;
+ }
svm->nmi_l1_to_l2 = is_evtinj_nmi(vmcb02->control.event_inj);
if (is_evtinj_soft(vmcb02->control.event_inj)) {
svm->soft_int_injected = true;
svm->soft_int_csbase = vmcb12_csbase;
svm->soft_int_old_rip = vmcb12_rip;
- if (guest_can_use(vcpu, X86_FEATURE_NRIPS))
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS) ||
+ !svm->nested.nested_run_pending)
svm->soft_int_next_rip = svm->nested.ctl.next_rip;
else
svm->soft_int_next_rip = vmcb12_rip;
@@ -752,11 +761,11 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
if (!nested_vmcb_needs_vls_intercept(svm))
vmcb02->control.virt_ext |= VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK;
- if (guest_can_use(vcpu, X86_FEATURE_PAUSEFILTER))
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_PAUSEFILTER))
pause_count12 = svm->nested.ctl.pause_filter_count;
else
pause_count12 = 0;
- if (guest_can_use(vcpu, X86_FEATURE_PFTHRESHOLD))
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_PFTHRESHOLD))
pause_thresh12 = svm->nested.ctl.pause_filter_thresh;
else
pause_thresh12 = 0;
@@ -959,6 +968,14 @@ void svm_copy_vmrun_state(struct vmcb_save_area *to_save,
to_save->rsp = from_save->rsp;
to_save->rip = from_save->rip;
to_save->cpl = 0;
+
+ if (kvm_cpu_cap_has(X86_FEATURE_LBRV)) {
+ to_save->dbgctl = from_save->dbgctl & ~DEBUGCTL_RESERVED_BITS;
+ to_save->br_from = from_save->br_from;
+ to_save->br_to = from_save->br_to;
+ to_save->last_excp_from = from_save->last_excp_from;
+ to_save->last_excp_to = from_save->last_excp_to;
+ }
}
void svm_copy_vmloadsave_state(struct vmcb *to_vmcb, struct vmcb *from_vmcb)
@@ -1035,7 +1052,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
if (vmcb12->control.exit_code != SVM_EXIT_ERR)
nested_save_pending_event_to_vmcb12(svm, vmcb12);
- if (guest_can_use(vcpu, X86_FEATURE_NRIPS))
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
vmcb12->control.next_rip = vmcb02->control.next_rip;
vmcb12->control.event_inj = 0;
@@ -1074,11 +1091,13 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
if (!nested_exit_on_intr(svm))
kvm_make_request(KVM_REQ_EVENT, &svm->vcpu);
- if (unlikely(guest_can_use(vcpu, X86_FEATURE_LBRV) &&
- (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK)))
+ if (unlikely(guest_cpu_cap_has(vcpu, X86_FEATURE_LBRV) &&
+ (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) {
svm_copy_lbrs(vmcb12, vmcb02);
- else
+ } else {
svm_copy_lbrs(vmcb01, vmcb02);
+ vmcb_mark_dirty(vmcb01, VMCB_LBR);
+ }
svm_update_lbrv(vcpu);
@@ -1144,9 +1163,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
nested_svm_uninit_mmu_context(vcpu);
- rc = nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true);
- if (rc)
- return 1;
+ if (nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true))
+ kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
/*
* Drop what we picked up for L2 via svm_complete_interrupts() so it
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index eb9c1f9e23bd..c086040ef11d 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -624,7 +624,7 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
unsigned long ulen, unsigned long *n,
- int write)
+ unsigned int flags)
{
struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
unsigned long npages, size;
@@ -668,7 +668,7 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
return ERR_PTR(-ENOMEM);
/* Pin the user virtual address. */
- npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
+ npinned = pin_user_pages_fast(uaddr, npages, flags, pages);
if (npinned != npages) {
pr_err("SEV: Failure locking %lu pages.\n", npages);
ret = -ENOMEM;
@@ -756,7 +756,7 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
vaddr_end = vaddr + size;
/* Lock the user memory. */
- inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1);
+ inpages = sev_pin_memory(kvm, vaddr, size, &npages, FOLL_WRITE);
if (IS_ERR(inpages))
return PTR_ERR(inpages);
@@ -1261,7 +1261,7 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec)
if (IS_ERR(src_p))
return PTR_ERR(src_p);
- dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, 1);
+ dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, FOLL_WRITE);
if (IS_ERR(dst_p)) {
sev_unpin_memory(kvm, src_p, n);
return PTR_ERR(dst_p);
@@ -1328,7 +1328,7 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp)
if (copy_from_user(¶ms, u64_to_user_ptr(argp->data), sizeof(params)))
return -EFAULT;
- pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, 1);
+ pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, FOLL_WRITE);
if (IS_ERR(pages))
return PTR_ERR(pages);
@@ -1810,7 +1810,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
/* Pin guest memory */
guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK,
- PAGE_SIZE, &n, 1);
+ PAGE_SIZE, &n, FOLL_WRITE);
if (IS_ERR(guest_page)) {
ret = PTR_ERR(guest_page);
goto e_free_trans;
@@ -2703,6 +2703,8 @@ int sev_mem_enc_register_region(struct kvm *kvm,
struct enc_region *region;
int ret = 0;
+ guard(mutex)(&kvm->lock);
+
if (!sev_guest(kvm))
return -ENOTTY;
@@ -2717,11 +2719,14 @@ int sev_mem_enc_register_region(struct kvm *kvm,
if (!region)
return -ENOMEM;
- mutex_lock(&kvm->lock);
- region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages, 1);
+ /*
+ * Do NOT specify FOLL_WRITE, as KVM isn't using the pinned pages to
+ * write memory, and FOLL_LONGTERM itself triggers CoW unshare.
+ */
+ region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages,
+ FOLL_LONGTERM);
if (IS_ERR(region->pages)) {
ret = PTR_ERR(region->pages);
- mutex_unlock(&kvm->lock);
goto e_free;
}
@@ -2739,8 +2744,6 @@ int sev_mem_enc_register_region(struct kvm *kvm,
region->size = range->size;
list_add_tail(®ion->list, &sev->regions_list);
- mutex_unlock(&kvm->lock);
-
return ret;
e_free:
@@ -4570,16 +4573,15 @@ static void sev_es_vcpu_after_set_cpuid(struct vcpu_svm *svm)
* For SEV-ES, accesses to MSR_IA32_XSS should not be intercepted if
* the host/guest supports its use.
*
- * guest_can_use() checks a number of requirements on the host/guest to
- * ensure that MSR_IA32_XSS is available, but it might report true even
- * if X86_FEATURE_XSAVES isn't configured in the guest to ensure host
- * MSR_IA32_XSS is always properly restored. For SEV-ES, it is better
- * to further check that the guest CPUID actually supports
- * X86_FEATURE_XSAVES so that accesses to MSR_IA32_XSS by misbehaved
- * guests will still get intercepted and caught in the normal
- * kvm_emulate_rdmsr()/kvm_emulated_wrmsr() paths.
+ * KVM treats the guest as being capable of using XSAVES even if XSAVES
+ * isn't enabled in guest CPUID as there is no intercept for XSAVES,
+ * i.e. the guest can use XSAVES/XRSTOR to read/write XSS if XSAVE is
+ * exposed to the guest and XSAVES is supported in hardware. Condition
+ * full XSS passthrough on the guest being able to use XSAVES *and*
+ * XSAVES being exposed to the guest so that KVM can at least honor
+ * guest CPUID for RDMSR and WRMSR.
*/
- if (guest_can_use(vcpu, X86_FEATURE_XSAVES) &&
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
guest_cpuid_has(vcpu, X86_FEATURE_XSAVES))
set_msr_interception(vcpu, svm->msrpm, MSR_IA32_XSS, 1, 1);
else
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 9408ab965aea..4c3891fb789a 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -990,8 +990,6 @@ void svm_copy_lbrs(struct vmcb *to_vmcb, struct vmcb *from_vmcb)
to_vmcb->save.br_to = from_vmcb->save.br_to;
to_vmcb->save.last_excp_from = from_vmcb->save.last_excp_from;
to_vmcb->save.last_excp_to = from_vmcb->save.last_excp_to;
-
- vmcb_mark_dirty(to_vmcb, VMCB_LBR);
}
static void svm_recalc_lbr_msr_intercepts(struct kvm_vcpu *vcpu)
@@ -1040,7 +1038,7 @@ void svm_update_lbrv(struct kvm_vcpu *vcpu)
struct vcpu_svm *svm = to_svm(vcpu);
bool current_enable_lbrv = svm->vmcb->control.virt_ext & LBR_CTL_ENABLE_MASK;
bool enable_lbrv = (svm->vmcb->save.dbgctl & DEBUGCTLMSR_LBR) ||
- (is_guest_mode(vcpu) && guest_can_use(vcpu, X86_FEATURE_LBRV) &&
+ (is_guest_mode(vcpu) && guest_cpu_cap_has(vcpu, X86_FEATURE_LBRV) &&
(svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK));
if (enable_lbrv && !current_enable_lbrv)
@@ -2922,7 +2920,7 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
switch (msr_info->index) {
case MSR_AMD64_TSC_RATIO:
if (!msr_info->host_initiated &&
- !guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR))
+ !guest_cpu_cap_has(vcpu, X86_FEATURE_TSCRATEMSR))
return 1;
msr_info->data = svm->tsc_ratio_msr;
break;
@@ -2966,19 +2964,19 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
msr_info->data = svm->tsc_aux;
break;
case MSR_IA32_DEBUGCTLMSR:
- msr_info->data = svm->vmcb->save.dbgctl;
+ msr_info->data = lbrv ? svm->vmcb->save.dbgctl : 0;
break;
case MSR_IA32_LASTBRANCHFROMIP:
- msr_info->data = svm->vmcb->save.br_from;
+ msr_info->data = lbrv ? svm->vmcb->save.br_from : 0;
break;
case MSR_IA32_LASTBRANCHTOIP:
- msr_info->data = svm->vmcb->save.br_to;
+ msr_info->data = lbrv ? svm->vmcb->save.br_to : 0;
break;
case MSR_IA32_LASTINTFROMIP:
- msr_info->data = svm->vmcb->save.last_excp_from;
+ msr_info->data = lbrv ? svm->vmcb->save.last_excp_from : 0;
break;
case MSR_IA32_LASTINTTOIP:
- msr_info->data = svm->vmcb->save.last_excp_to;
+ msr_info->data = lbrv ? svm->vmcb->save.last_excp_to : 0;
break;
case MSR_VM_HSAVE_PA:
msr_info->data = svm->nested.hsave_msr;
@@ -3082,7 +3080,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
switch (ecx) {
case MSR_AMD64_TSC_RATIO:
- if (!guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR)) {
+ if (!guest_cpu_cap_has(vcpu, X86_FEATURE_TSCRATEMSR)) {
if (!msr->host_initiated)
return 1;
@@ -3104,7 +3102,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
svm->tsc_ratio_msr = data;
- if (guest_can_use(vcpu, X86_FEATURE_TSCRATEMSR) &&
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_TSCRATEMSR) &&
is_guest_mode(vcpu))
nested_svm_update_tsc_ratio_msr(vcpu);
@@ -3258,6 +3256,38 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
vmcb_mark_dirty(svm->vmcb, VMCB_LBR);
svm_update_lbrv(vcpu);
break;
+ case MSR_IA32_LASTBRANCHFROMIP:
+ if (!lbrv)
+ return KVM_MSR_RET_UNSUPPORTED;
+ if (!msr->host_initiated)
+ return 1;
+ svm->vmcb->save.br_from = data;
+ vmcb_mark_dirty(svm->vmcb, VMCB_LBR);
+ break;
+ case MSR_IA32_LASTBRANCHTOIP:
+ if (!lbrv)
+ return KVM_MSR_RET_UNSUPPORTED;
+ if (!msr->host_initiated)
+ return 1;
+ svm->vmcb->save.br_to = data;
+ vmcb_mark_dirty(svm->vmcb, VMCB_LBR);
+ break;
+ case MSR_IA32_LASTINTFROMIP:
+ if (!lbrv)
+ return KVM_MSR_RET_UNSUPPORTED;
+ if (!msr->host_initiated)
+ return 1;
+ svm->vmcb->save.last_excp_from = data;
+ vmcb_mark_dirty(svm->vmcb, VMCB_LBR);
+ break;
+ case MSR_IA32_LASTINTTOIP:
+ if (!lbrv)
+ return KVM_MSR_RET_UNSUPPORTED;
+ if (!msr->host_initiated)
+ return 1;
+ svm->vmcb->save.last_excp_to = data;
+ vmcb_mark_dirty(svm->vmcb, VMCB_LBR);
+ break;
case MSR_VM_HSAVE_PA:
/*
* Old kernels did not validate the value written to
@@ -4542,11 +4572,11 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
if (boot_cpu_has(X86_FEATURE_XSAVE) &&
boot_cpu_has(X86_FEATURE_XSAVES) &&
guest_cpuid_has(vcpu, X86_FEATURE_XSAVE))
- kvm_governed_feature_set(vcpu, X86_FEATURE_XSAVES);
+ guest_cpu_cap_set(vcpu, X86_FEATURE_XSAVES);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_NRIPS);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_TSCRATEMSR);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_LBRV);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_NRIPS);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_TSCRATEMSR);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_LBRV);
/*
* Intercept VMLOAD if the vCPU model is Intel in order to emulate that
@@ -4554,12 +4584,12 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
* SVM on Intel is bonkers and extremely unlikely to work).
*/
if (!guest_cpuid_is_intel_compatible(vcpu))
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_V_VMSAVE_VMLOAD);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_V_VMSAVE_VMLOAD);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_PAUSEFILTER);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_PFTHRESHOLD);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_VGIF);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_VNMI);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_PAUSEFILTER);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_PFTHRESHOLD);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_VGIF);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_VNMI);
svm_recalc_instruction_intercepts(vcpu, svm);
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 93ad26f3b1a1..19da3ce2b766 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -510,7 +510,7 @@ static inline bool svm_is_intercept(struct vcpu_svm *svm, int bit)
static inline bool nested_vgif_enabled(struct vcpu_svm *svm)
{
- return guest_can_use(&svm->vcpu, X86_FEATURE_VGIF) &&
+ return guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_VGIF) &&
(svm->nested.ctl.int_ctl & V_GIF_ENABLE_MASK);
}
@@ -562,7 +562,7 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)
static inline bool nested_vnmi_enabled(struct vcpu_svm *svm)
{
- return guest_can_use(&svm->vcpu, X86_FEATURE_VNMI) &&
+ return guest_cpu_cap_has(&svm->vcpu, X86_FEATURE_VNMI) &&
(svm->nested.ctl.int_ctl & V_NMI_ENABLE_MASK);
}
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index abda07f41b4a..bd9f41a78ec1 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -3692,6 +3692,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
if (!from_vmentry)
return NVMX_VMENTRY_VMEXIT;
+ kvm_vcpu_unmap(vcpu, &vmx->nested.apic_access_page_map, false);
+ kvm_vcpu_unmap(vcpu, &vmx->nested.virtual_apic_map, true);
+ kvm_vcpu_unmap(vcpu, &vmx->nested.pi_desc_map, true);
+ vmx->nested.pi_desc = NULL;
+
load_vmcs12_host_state(vcpu, vmcs12);
vmcs12->vm_exit_reason = exit_reason.full;
if (enable_shadow_vmcs || nested_vmx_is_evmptr12_valid(vmx))
@@ -6671,7 +6676,7 @@ static int vmx_get_nested_state(struct kvm_vcpu *vcpu,
vmx = to_vmx(vcpu);
vmcs12 = get_vmcs12(vcpu);
- if (guest_can_use(vcpu, X86_FEATURE_VMX) &&
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_VMX) &&
(vmx->nested.vmxon || vmx->nested.smm.vmxon)) {
kvm_state.hdr.vmx.vmxon_pa = vmx->nested.vmxon_ptr;
kvm_state.hdr.vmx.vmcs12_pa = vmx->nested.current_vmptr;
@@ -6812,7 +6817,7 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu,
if (kvm_state->flags & ~KVM_STATE_NESTED_EVMCS)
return -EINVAL;
} else {
- if (!guest_can_use(vcpu, X86_FEATURE_VMX))
+ if (!guest_cpu_cap_has(vcpu, X86_FEATURE_VMX))
return -EINVAL;
if (!page_address_valid(vcpu, kvm_state->hdr.vmx.vmxon_pa))
@@ -6846,7 +6851,7 @@ static int vmx_set_nested_state(struct kvm_vcpu *vcpu,
return -EINVAL;
if ((kvm_state->flags & KVM_STATE_NESTED_EVMCS) &&
- (!guest_can_use(vcpu, X86_FEATURE_VMX) ||
+ (!guest_cpu_cap_has(vcpu, X86_FEATURE_VMX) ||
!vmx->nested.enlightened_vmcs_enabled))
return -EINVAL;
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 84420bfbcb1b..7e098fb60bbf 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -2103,7 +2103,7 @@ int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
[msr_info->index - MSR_IA32_SGXLEPUBKEYHASH0];
break;
case KVM_FIRST_EMULATED_VMX_MSR ... KVM_LAST_EMULATED_VMX_MSR:
- if (!guest_can_use(vcpu, X86_FEATURE_VMX))
+ if (!guest_cpu_cap_has(vcpu, X86_FEATURE_VMX))
return 1;
if (vmx_get_vmx_msr(&vmx->nested.msrs, msr_info->index,
&msr_info->data))
@@ -2422,7 +2422,7 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
case KVM_FIRST_EMULATED_VMX_MSR ... KVM_LAST_EMULATED_VMX_MSR:
if (!msr_info->host_initiated)
return 1; /* they are read-only */
- if (!guest_can_use(vcpu, X86_FEATURE_VMX))
+ if (!guest_cpu_cap_has(vcpu, X86_FEATURE_VMX))
return 1;
return vmx_set_vmx_msr(vcpu, msr_index, data);
case MSR_IA32_RTIT_CTL:
@@ -4608,7 +4608,7 @@ vmx_adjust_secondary_exec_control(struct vcpu_vmx *vmx, u32 *exec_control,
\
if (cpu_has_vmx_##name()) { \
if (kvm_is_governed_feature(X86_FEATURE_##feat_name)) \
- __enabled = guest_can_use(__vcpu, X86_FEATURE_##feat_name); \
+ __enabled = guest_cpu_cap_has(__vcpu, X86_FEATURE_##feat_name); \
else \
__enabled = guest_cpuid_has(__vcpu, X86_FEATURE_##feat_name); \
vmx_adjust_secondary_exec_control(vmx, exec_control, SECONDARY_EXEC_##ctrl_name,\
@@ -7832,10 +7832,10 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
*/
if (boot_cpu_has(X86_FEATURE_XSAVE) &&
guest_cpuid_has(vcpu, X86_FEATURE_XSAVE))
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_XSAVES);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_XSAVES);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_VMX);
- kvm_governed_feature_check_and_set(vcpu, X86_FEATURE_LAM);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_VMX);
+ guest_cpu_cap_check_and_set(vcpu, X86_FEATURE_LAM);
vmx_setup_uret_msrs(vmx);
@@ -7843,7 +7843,7 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
vmcs_set_secondary_exec_control(vmx,
vmx_secondary_exec_control(vmx));
- if (guest_can_use(vcpu, X86_FEATURE_VMX))
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_VMX))
vmx->msr_ia32_feature_control_valid_bits |=
FEAT_CTL_VMX_ENABLED_INSIDE_SMX |
FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX;
@@ -7852,7 +7852,7 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
~(FEAT_CTL_VMX_ENABLED_INSIDE_SMX |
FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX);
- if (guest_can_use(vcpu, X86_FEATURE_VMX))
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_VMX))
nested_vmx_cr_fixed1_bits_update(vcpu);
if (boot_cpu_has(X86_FEATURE_INTEL_PT) &&
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 99549df4c633..23696631599a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -320,6 +320,9 @@ static const u32 msrs_to_save_base[] = {
MSR_IA32_UMWAIT_CONTROL,
MSR_IA32_XFD, MSR_IA32_XFD_ERR,
+ MSR_IA32_DEBUGCTLMSR,
+ MSR_IA32_LASTBRANCHFROMIP, MSR_IA32_LASTBRANCHTOIP,
+ MSR_IA32_LASTINTFROMIP, MSR_IA32_LASTINTTOIP,
};
static const u32 msrs_to_save_pmu[] = {
@@ -655,27 +658,6 @@ static void drop_user_return_notifiers(void)
kvm_on_user_return(&msrs->urn);
}
-int kvm_set_apic_base(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
-{
- enum lapic_mode old_mode = kvm_get_apic_mode(vcpu);
- enum lapic_mode new_mode = kvm_apic_mode(msr_info->data);
- u64 reserved_bits = kvm_vcpu_reserved_gpa_bits_raw(vcpu) | 0x2ff |
- (guest_cpuid_has(vcpu, X86_FEATURE_X2APIC) ? 0 : X2APIC_ENABLE);
-
- if ((msr_info->data & reserved_bits) != 0 || new_mode == LAPIC_MODE_INVALID)
- return 1;
- if (!msr_info->host_initiated) {
- if (old_mode == LAPIC_MODE_X2APIC && new_mode == LAPIC_MODE_XAPIC)
- return 1;
- if (old_mode == LAPIC_MODE_DISABLED && new_mode == LAPIC_MODE_X2APIC)
- return 1;
- }
-
- kvm_lapic_set_base(vcpu, msr_info->data);
- kvm_recalculate_apic_map(vcpu->kvm);
- return 0;
-}
-
/*
* Handle a fault on a hardware virtualization (VMX or SVM) instruction.
*
@@ -1029,7 +1011,7 @@ void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
if (vcpu->arch.xcr0 != kvm_host.xcr0)
xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
- if (guest_can_use(vcpu, X86_FEATURE_XSAVES) &&
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
vcpu->arch.ia32_xss != kvm_host.xss)
wrmsrl(MSR_IA32_XSS, vcpu->arch.ia32_xss);
}
@@ -1060,7 +1042,7 @@ void kvm_load_host_xsave_state(struct kvm_vcpu *vcpu)
if (vcpu->arch.xcr0 != kvm_host.xcr0)
xsetbv(XCR_XFEATURE_ENABLED_MASK, kvm_host.xcr0);
- if (guest_can_use(vcpu, X86_FEATURE_XSAVES) &&
+ if (guest_cpu_cap_has(vcpu, X86_FEATURE_XSAVES) &&
vcpu->arch.ia32_xss != kvm_host.xss)
wrmsrl(MSR_IA32_XSS, kvm_host.xss);
}
@@ -3472,7 +3454,7 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
case MSR_MTRRdefType:
return kvm_mtrr_set_msr(vcpu, msr, data);
case MSR_IA32_APICBASE:
- return kvm_set_apic_base(vcpu, msr_info);
+ return kvm_apic_set_base(vcpu, msr_info);
case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff:
return kvm_x2apic_msr_write(vcpu, msr, data);
case MSR_IA32_TSC_DEADLINE:
@@ -4715,7 +4697,6 @@ static int kvm_vcpu_ioctl_set_lapic(struct kvm_vcpu *vcpu,
r = kvm_apic_set_state(vcpu, s);
if (r)
return r;
- update_cr8_intercept(vcpu);
return 0;
}
@@ -9783,33 +9764,6 @@ static void post_kvm_run_save(struct kvm_vcpu *vcpu)
kvm_run->flags |= KVM_RUN_X86_GUEST_MODE;
}
-void update_cr8_intercept(struct kvm_vcpu *vcpu)
-{
- int max_irr, tpr;
-
- if (!kvm_x86_ops.update_cr8_intercept)
- return;
-
- if (!lapic_in_kernel(vcpu))
- return;
-
- if (vcpu->arch.apic->apicv_active)
- return;
-
- if (!vcpu->arch.apic->vapic_addr)
- max_irr = kvm_lapic_find_highest_irr(vcpu);
- else
- max_irr = -1;
-
- if (max_irr != -1)
- max_irr >>= 4;
-
- tpr = kvm_lapic_get_cr8(vcpu);
-
- kvm_x86_call(update_cr8_intercept)(vcpu, tpr, max_irr);
-}
-
-
int kvm_check_nested_events(struct kvm_vcpu *vcpu)
{
if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) {
@@ -10512,7 +10466,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
kvm_x86_call(enable_irq_window)(vcpu);
if (kvm_lapic_enabled(vcpu)) {
- update_cr8_intercept(vcpu);
+ kvm_lapic_update_cr8_intercept(vcpu);
kvm_lapic_sync_to_vapic(vcpu);
}
}
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index abab4bd96d0a..6517e10551e9 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -288,7 +288,7 @@ static inline bool is_noncanonical_invlpg_address(u64 la, struct kvm_vcpu *vcpu)
static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu,
gva_t gva, gfn_t gfn, unsigned access)
{
- u64 gen = kvm_memslots(vcpu->kvm)->generation;
+ u64 gen = kvm_vcpu_memslots(vcpu)->generation;
if (unlikely(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS))
return;
@@ -305,7 +305,7 @@ static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu,
static inline bool vcpu_match_mmio_gen(struct kvm_vcpu *vcpu)
{
- return vcpu->arch.mmio_gen == kvm_memslots(vcpu->kvm)->generation;
+ return vcpu->arch.mmio_gen == kvm_vcpu_memslots(vcpu)->generation;
}
/*
@@ -641,6 +641,5 @@ int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu,
struct kvm_debugregs *dbgregs);
void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid);
void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu);
-void update_cr8_intercept(struct kvm_vcpu *vcpu);
#endif
diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index 6a75fe1c7a5c..2e7cfaac7b21 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -548,14 +548,21 @@ static void bfq_pd_init(struct blkg_policy_data *pd)
bfqg->rq_pos_tree = RB_ROOT;
}
-static void bfq_pd_free(struct blkg_policy_data *pd)
+static void bfqg_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct bfq_group *bfqg = pd_to_bfqg(pd);
bfqg_stats_exit(&bfqg->stats);
bfqg_put(bfqg);
}
+static void bfq_pd_free(struct blkg_policy_data *pd)
+{
+ call_rcu(&pd->rcu_head, bfqg_release);
+}
+
static void bfq_pd_reset_stats(struct blkg_policy_data *pd)
{
struct bfq_group *bfqg = pd_to_bfqg(pd);
diff --git a/block/bio-integrity.c b/block/bio-integrity.c
index 04bab987b087..c6721b95d034 100644
--- a/block/bio-integrity.c
+++ b/block/bio-integrity.c
@@ -12,6 +12,7 @@
#include <linux/bio.h>
#include <linux/workqueue.h>
#include <linux/slab.h>
+#include <linux/t10-pi.h>
#include "blk.h"
static struct kmem_cache *bip_slab;
@@ -428,7 +429,10 @@ bool bio_integrity_prep(struct bio *bio)
* memory to disk for non-integrity metadata where nothing else
* initializes the memory.
*/
- if (bi->csum_type == BLK_INTEGRITY_CSUM_NONE)
+ if (bi->csum_type == BLK_INTEGRITY_CSUM_NONE ||
+ bi->tuple_size > (bi->csum_type == BLK_INTEGRITY_CSUM_CRC64 ?
+ sizeof(struct crc64_pi_tuple) :
+ sizeof(struct t10_pi_tuple)))
gfp |= __GFP_ZERO;
break;
default:
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index d89da7e067f4..06de4024a9a2 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1561,6 +1561,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
if (queue_is_mq(q))
blk_mq_freeze_queue(q);
+
+ mutex_lock(&q->blkcg_mutex);
retry:
spin_lock_irq(&q->queue_lock);
@@ -1625,6 +1627,7 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
spin_unlock_irq(&q->queue_lock);
out:
+ mutex_unlock(&q->blkcg_mutex);
if (queue_is_mq(q))
blk_mq_unfreeze_queue(q);
if (pinned_blkg)
@@ -2189,7 +2192,7 @@ void blk_cgroup_bio_start(struct bio *bio)
}
u64_stats_update_end_irqrestore(&bis->sync, flags);
- cgroup_rstat_updated(blkcg->css.cgroup, cpu);
+ __cgroup_rstat_updated(blkcg->css.cgroup, cpu);
put_cpu();
}
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 112bf11d0fad..270e58e5f38a 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -140,6 +140,8 @@ struct blkg_policy_data {
struct blkcg_gq *blkg;
int plid;
bool online;
+
+ struct rcu_head rcu_head;
};
/*
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 61a347f504d7..8b775dd95ba5 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -3019,9 +3019,20 @@ static void ioc_pd_init(struct blkg_policy_data *pd)
spin_unlock_irqrestore(&ioc->lock, flags);
}
+static void iocg_release(struct rcu_head *rcu)
+{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
+ struct ioc_gq *iocg = pd_to_iocg(pd);
+
+ free_percpu(iocg->pcpu_stat);
+ kfree(iocg);
+}
+
static void ioc_pd_free(struct blkg_policy_data *pd)
{
struct ioc_gq *iocg = pd_to_iocg(pd);
+ struct blkcg_gq *blkg = pd_to_blkg(pd);
struct ioc *ioc = iocg->ioc;
unsigned long flags;
@@ -3043,8 +3054,14 @@ static void ioc_pd_free(struct blkg_policy_data *pd)
hrtimer_cancel(&iocg->waitq_timer);
}
- free_percpu(iocg->pcpu_stat);
- kfree(iocg);
+
+ /* off ->active_iocgs and timer gone, so nothing can re-arm the delay */
+ iocg->delay = 0;
+ iocg->indelay_since = 0;
+ if (blkg)
+ blkcg_clear_delay(blkg);
+
+ call_rcu(&pd->rcu_head, iocg_release);
}
static void ioc_pd_stat(struct blkg_policy_data *pd, struct seq_file *s)
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index ebb522788d97..6be1c5e020ac 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -1031,13 +1031,30 @@ static void iolatency_pd_offline(struct blkg_policy_data *pd)
iolatency_clear_scaling(blkg);
}
-static void iolatency_pd_free(struct blkg_policy_data *pd)
+static void iolat_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct iolatency_grp *iolat = pd_to_lat(pd);
+
free_percpu(iolat->stats);
kfree(iolat);
}
+static void iolatency_pd_free(struct blkg_policy_data *pd)
+{
+ struct blkcg_gq *blkg = pd_to_blkg(pd);
+
+ /*
+ * Groups throttled as collateral have min_lat_nsec == 0, so
+ * iolatency_pd_offline() leaves their delay set. Drop it here, where
+ * no in-flight bio can re-arm it via check_scale_change().
+ */
+ if (blkg)
+ blkcg_clear_delay(blkg);
+ call_rcu(&pd->rcu_head, iolat_release);
+}
+
static struct cftype iolatency_files[] = {
{
.name = "latency",
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index bdf363868ac0..7a29eeb7b695 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -327,16 +327,25 @@ static void throtl_pd_online(struct blkg_policy_data *pd)
tg_update_has_rules(tg);
}
-static void throtl_pd_free(struct blkg_policy_data *pd)
+static void tg_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct throtl_grp *tg = pd_to_tg(pd);
- del_timer_sync(&tg->service_queue.pending_timer);
blkg_rwstat_exit(&tg->stat_bytes);
blkg_rwstat_exit(&tg->stat_ios);
kfree(tg);
}
+static void throtl_pd_free(struct blkg_policy_data *pd)
+{
+ struct throtl_grp *tg = pd_to_tg(pd);
+
+ del_timer_sync(&tg->service_queue.pending_timer);
+ call_rcu(&pd->rcu_head, tg_release);
+}
+
static struct throtl_grp *
throtl_rb_first(struct throtl_service_queue *parent_sq)
{
diff --git a/drivers/accel/ivpu/ivpu_drv.c b/drivers/accel/ivpu/ivpu_drv.c
index ea2909c4745a..a811da0634e7 100644
--- a/drivers/accel/ivpu/ivpu_drv.c
+++ b/drivers/accel/ivpu/ivpu_drv.c
@@ -446,7 +446,7 @@ static const struct drm_driver driver = {
.postclose = ivpu_postclose,
.gem_create_object = ivpu_gem_create_object,
- .gem_prime_import_sg_table = drm_gem_shmem_prime_import_sg_table,
+ .gem_prime_import = ivpu_gem_prime_import,
.ioctls = ivpu_drm_ioctls,
.num_ioctls = ARRAY_SIZE(ivpu_drm_ioctls),
diff --git a/drivers/accel/ivpu/ivpu_gem.c b/drivers/accel/ivpu/ivpu_gem.c
index b4213e63afb4..3929357a61a3 100644
--- a/drivers/accel/ivpu/ivpu_gem.c
+++ b/drivers/accel/ivpu/ivpu_gem.c
@@ -20,13 +20,15 @@
#include "ivpu_mmu.h"
#include "ivpu_mmu_context.h"
+MODULE_IMPORT_NS(DMA_BUF);
+
static const struct drm_gem_object_funcs ivpu_gem_funcs;
static inline void ivpu_dbg_bo(struct ivpu_device *vdev, struct ivpu_bo *bo, const char *action)
{
ivpu_dbg(vdev, BO,
- "%6s: bo %8p vpu_addr %9llx size %8zu ctx %d has_pages %d dma_mapped %d mmu_mapped %d wc %d imported %d\n",
- action, bo, bo->vpu_addr, ivpu_bo_size(bo), bo->ctx_id,
+ "%6s: bo %8p size %9zu ctx %d vpu_addr %9llx pages %d sgt %d mmu_mapped %d wc %d imported %d\n",
+ action, bo, ivpu_bo_size(bo), bo->ctx_id, bo->vpu_addr,
(bool)bo->base.pages, (bool)bo->base.sgt, bo->mmu_mapped, bo->base.map_wc,
(bool)bo->base.base.import_attach);
}
@@ -41,22 +43,47 @@ static inline void ivpu_bo_unlock(struct ivpu_bo *bo)
dma_resv_unlock(bo->base.base.resv);
}
+static struct sg_table *ivpu_bo_map_attachment(struct ivpu_device *vdev, struct ivpu_bo *bo)
+{
+ struct sg_table *sgt;
+
+ drm_WARN_ON(&vdev->drm, !bo->base.base.import_attach);
+
+ ivpu_bo_lock(bo);
+
+ sgt = bo->base.sgt;
+ if (!sgt) {
+ sgt = dma_buf_map_attachment(bo->base.base.import_attach, DMA_BIDIRECTIONAL);
+ if (IS_ERR(sgt))
+ ivpu_err(vdev, "Failed to map BO in IOMMU: %ld\n", PTR_ERR(sgt));
+ else
+ bo->base.sgt = sgt;
+ }
+
+ ivpu_bo_unlock(bo);
+
+ return sgt;
+}
+
/*
- * ivpu_bo_pin() - pin the backing physical pages and map them to VPU.
+ * ivpu_bo_bind() - pin the backing physical pages and map them to VPU.
*
* This function pins physical memory pages, then maps the physical pages
* to IOMMU address space and finally updates the VPU MMU page tables
* to allow the VPU to translate VPU address to IOMMU address.
*/
-int __must_check ivpu_bo_pin(struct ivpu_bo *bo)
+int __must_check ivpu_bo_bind(struct ivpu_bo *bo)
{
struct ivpu_device *vdev = ivpu_bo_to_vdev(bo);
struct sg_table *sgt;
int ret = 0;
- ivpu_dbg_bo(vdev, bo, "pin");
+ ivpu_dbg_bo(vdev, bo, "bind");
- sgt = drm_gem_shmem_get_pages_sgt(&bo->base);
+ if (bo->base.base.import_attach)
+ sgt = ivpu_bo_map_attachment(vdev, bo);
+ else
+ sgt = drm_gem_shmem_get_pages_sgt(&bo->base);
if (IS_ERR(sgt)) {
ret = PTR_ERR(sgt);
ivpu_err(vdev, "Failed to map BO in IOMMU: %d\n", ret);
@@ -97,7 +124,9 @@ ivpu_bo_alloc_vpu_addr(struct ivpu_bo *bo, struct ivpu_mmu_context *ctx,
ret = ivpu_mmu_context_insert_node(ctx, range, ivpu_bo_size(bo), &bo->mm_node);
if (!ret) {
bo->ctx = ctx;
+ bo->ctx_id = ctx->id;
bo->vpu_addr = bo->mm_node.start;
+ ivpu_dbg_bo(vdev, bo, "vaddr");
} else {
ivpu_err(vdev, "Failed to add BO to context %u: %d\n", ctx->id, ret);
}
@@ -113,7 +142,7 @@ static void ivpu_bo_unbind_locked(struct ivpu_bo *bo)
{
struct ivpu_device *vdev = ivpu_bo_to_vdev(bo);
- lockdep_assert(dma_resv_held(bo->base.base.resv) || !kref_read(&bo->base.base.refcount));
+ dma_resv_assert_held(bo->base.base.resv);
if (bo->mmu_mapped) {
drm_WARN_ON(&vdev->drm, !bo->ctx);
@@ -132,9 +161,14 @@ static void ivpu_bo_unbind_locked(struct ivpu_bo *bo)
return;
if (bo->base.sgt) {
- dma_unmap_sgtable(vdev->drm.dev, bo->base.sgt, DMA_BIDIRECTIONAL, 0);
- sg_free_table(bo->base.sgt);
- kfree(bo->base.sgt);
+ if (bo->base.base.import_attach) {
+ dma_buf_unmap_attachment(bo->base.base.import_attach,
+ bo->base.sgt, DMA_BIDIRECTIONAL);
+ } else {
+ dma_unmap_sgtable(vdev->drm.dev, bo->base.sgt, DMA_BIDIRECTIONAL, 0);
+ sg_free_table(bo->base.sgt);
+ kfree(bo->base.sgt);
+ }
bo->base.sgt = NULL;
}
}
@@ -177,7 +211,49 @@ struct drm_gem_object *ivpu_gem_create_object(struct drm_device *dev, size_t siz
return &bo->base.base;
}
-static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 flags, u32 ctx_id)
+struct drm_gem_object *ivpu_gem_prime_import(struct drm_device *dev,
+ struct dma_buf *dma_buf)
+{
+ struct ivpu_device *vdev = to_ivpu_device(dev);
+ struct device *attach_dev = dev->dev;
+ struct dma_buf_attachment *attach;
+ struct drm_gem_object *obj;
+ struct ivpu_bo *bo;
+ int ret;
+
+ attach = dma_buf_attach(dma_buf, attach_dev);
+ if (IS_ERR(attach))
+ return ERR_CAST(attach);
+
+ get_dma_buf(dma_buf);
+
+ obj = drm_gem_shmem_prime_import_sg_table(dev, attach, NULL);
+ if (IS_ERR(obj)) {
+ ret = PTR_ERR(obj);
+ goto fail_detach;
+ }
+
+ obj->import_attach = attach;
+ obj->resv = dma_buf->resv;
+
+ bo = to_ivpu_bo(obj);
+
+ mutex_lock(&vdev->bo_list_lock);
+ list_add_tail(&bo->bo_list_node, &vdev->bo_list);
+ mutex_unlock(&vdev->bo_list_lock);
+
+ ivpu_dbg(vdev, BO, "import: bo %8p size %9zu\n", bo, ivpu_bo_size(bo));
+
+ return obj;
+
+fail_detach:
+ dma_buf_detach(dma_buf, attach);
+ dma_buf_put(dma_buf);
+
+ return ERR_PTR(ret);
+}
+
+static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 flags)
{
struct drm_gem_shmem_object *shmem;
struct ivpu_bo *bo;
@@ -195,7 +271,6 @@ static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 fla
return ERR_CAST(shmem);
bo = to_ivpu_bo(&shmem->base);
- bo->ctx_id = ctx_id;
bo->base.map_wc = flags & DRM_IVPU_BO_WC;
bo->flags = flags;
@@ -203,7 +278,7 @@ static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 fla
list_add_tail(&bo->bo_list_node, &vdev->bo_list);
mutex_unlock(&vdev->bo_list_lock);
- ivpu_dbg_bo(vdev, bo, "alloc");
+ ivpu_dbg(vdev, BO, " alloc: bo %8p size %9llu\n", bo, size);
return bo;
}
@@ -238,6 +313,8 @@ static void ivpu_gem_bo_free(struct drm_gem_object *obj)
ivpu_dbg_bo(vdev, bo, "free");
+ drm_WARN_ON(&vdev->drm, list_empty(&bo->bo_list_node));
+
mutex_lock(&vdev->bo_list_lock);
list_del(&bo->bo_list_node);
@@ -246,7 +323,9 @@ static void ivpu_gem_bo_free(struct drm_gem_object *obj)
drm_WARN_ON(&vdev->drm, ivpu_bo_size(bo) == 0);
drm_WARN_ON(&vdev->drm, bo->base.vaddr);
+ ivpu_bo_lock(bo);
ivpu_bo_unbind_locked(bo);
+ ivpu_bo_unlock(bo);
mutex_unlock(&vdev->bo_list_lock);
drm_WARN_ON(&vdev->drm, bo->mmu_mapped);
@@ -284,7 +363,7 @@ int ivpu_bo_create_ioctl(struct drm_device *dev, void *data, struct drm_file *fi
if (size == 0)
return -EINVAL;
- bo = ivpu_bo_alloc(vdev, size, args->flags, file_priv->ctx.id);
+ bo = ivpu_bo_alloc(vdev, size, args->flags);
if (IS_ERR(bo)) {
ivpu_err(vdev, "Failed to allocate BO: %pe (ctx %u size %llu flags 0x%x)",
bo, file_priv->ctx.id, args->size, args->flags);
@@ -318,7 +397,7 @@ ivpu_bo_create(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx,
drm_WARN_ON(&vdev->drm, !PAGE_ALIGNED(range->end));
drm_WARN_ON(&vdev->drm, !PAGE_ALIGNED(size));
- bo = ivpu_bo_alloc(vdev, size, flags, IVPU_GLOBAL_CONTEXT_MMU_SSID);
+ bo = ivpu_bo_alloc(vdev, size, flags);
if (IS_ERR(bo)) {
ivpu_err(vdev, "Failed to allocate BO: %pe (vpu_addr 0x%llx size %llu flags 0x%x)",
bo, range->start, size, flags);
@@ -329,7 +408,7 @@ ivpu_bo_create(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx,
if (ret)
goto err_put;
- ret = ivpu_bo_pin(bo);
+ ret = ivpu_bo_bind(bo);
if (ret)
goto err_put;
diff --git a/drivers/accel/ivpu/ivpu_gem.h b/drivers/accel/ivpu/ivpu_gem.h
index df8e65b77ec4..57c9c2db0206 100644
--- a/drivers/accel/ivpu/ivpu_gem.h
+++ b/drivers/accel/ivpu/ivpu_gem.h
@@ -24,10 +24,11 @@ struct ivpu_bo {
bool mmu_mapped;
};
-int ivpu_bo_pin(struct ivpu_bo *bo);
+int ivpu_bo_bind(struct ivpu_bo *bo);
void ivpu_bo_unbind_all_bos_from_context(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx);
struct drm_gem_object *ivpu_gem_create_object(struct drm_device *dev, size_t size);
+struct drm_gem_object *ivpu_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf);
struct ivpu_bo *ivpu_bo_create(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx,
struct ivpu_addr_range *range, u64 size, u32 flags);
struct ivpu_bo *ivpu_bo_create_global(struct ivpu_device *vdev, u64 size, u32 flags);
diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c
index 172502b71b9c..3856d9683ed8 100644
--- a/drivers/accel/ivpu/ivpu_job.c
+++ b/drivers/accel/ivpu/ivpu_job.c
@@ -668,7 +668,7 @@ ivpu_job_prepare_bos_for_submit(struct drm_file *file, struct ivpu_job *job, u32
job->bos[i] = to_ivpu_bo(obj);
- ret = ivpu_bo_pin(job->bos[i]);
+ ret = ivpu_bo_bind(job->bos[i]);
if (ret)
return ret;
}
diff --git a/drivers/accel/qaic/mhi_controller.c b/drivers/accel/qaic/mhi_controller.c
index ada9b1eb0787..8b3a0690ee8f 100644
--- a/drivers/accel/qaic/mhi_controller.c
+++ b/drivers/accel/qaic/mhi_controller.c
@@ -34,7 +34,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -50,7 +49,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -66,7 +64,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -82,7 +79,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -98,7 +94,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -114,7 +109,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -130,7 +124,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -146,7 +139,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -162,7 +154,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -178,7 +169,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -194,7 +184,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -210,7 +199,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -226,7 +214,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -242,7 +229,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -258,7 +244,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -274,7 +259,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -290,7 +274,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -306,7 +289,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -322,7 +304,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -338,7 +319,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -354,7 +334,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -370,7 +349,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -386,7 +364,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
{
@@ -402,7 +379,6 @@ static const struct mhi_channel_config aic100_channels[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
.wake_capable = false,
},
};
diff --git a/drivers/accel/qaic/qaic.h b/drivers/accel/qaic/qaic.h
index 2d7b3af09e28..06aa35b7a484 100644
--- a/drivers/accel/qaic/qaic.h
+++ b/drivers/accel/qaic/qaic.h
@@ -32,6 +32,11 @@
#define to_accel_kdev(qddev) (to_drm(qddev)->accel->kdev) /* Return Linux device of accel node */
#define to_qaic_device(dev) (to_qaic_drm_device((dev))->qdev)
+enum aic_families {
+ FAMILY_AIC100,
+ FAMILY_MAX,
+};
+
enum __packed dev_states {
/* Device is offline or will be very soon */
QAIC_OFFLINE,
@@ -115,10 +120,10 @@ struct qaic_device {
struct pci_dev *pdev;
/* Req. ID of request that will be queued next in MHI control device */
u32 next_seq_num;
- /* Base address of bar 0 */
- void __iomem *bar_0;
- /* Base address of bar 2 */
- void __iomem *bar_2;
+ /* Base address of the MHI bar */
+ void __iomem *bar_mhi;
+ /* Base address of the DBCs bar */
+ void __iomem *bar_dbc;
/* Controller structure for MHI devices */
struct mhi_controller *mhi_cntrl;
/* MHI control channel device */
diff --git a/drivers/accel/qaic/qaic_drv.c b/drivers/accel/qaic/qaic_drv.c
index cb606c4bb851..dda1ca4bcbfc 100644
--- a/drivers/accel/qaic/qaic_drv.c
+++ b/drivers/accel/qaic/qaic_drv.c
@@ -34,13 +34,38 @@
MODULE_IMPORT_NS(DMA_BUF);
-#define PCI_DEV_AIC080 0xa080
-#define PCI_DEV_AIC100 0xa100
+#define PCI_DEVICE_ID_QCOM_AIC080 0xa080
+#define PCI_DEVICE_ID_QCOM_AIC100 0xa100
#define QAIC_NAME "qaic"
#define QAIC_DESC "Qualcomm Cloud AI Accelerators"
#define CNTL_MAJOR 5
#define CNTL_MINOR 0
+struct qaic_device_config {
+ /* Indicates the AIC family the device belongs to */
+ int family;
+ /* A bitmask representing the available BARs */
+ int bar_mask;
+ /* An index value used to identify the MHI controller BAR */
+ unsigned int mhi_bar_idx;
+ /* An index value used to identify the DBCs BAR */
+ unsigned int dbc_bar_idx;
+};
+
+static const struct qaic_device_config aic080_config = {
+ .family = FAMILY_AIC100,
+ .bar_mask = BIT(0) | BIT(2) | BIT(4),
+ .mhi_bar_idx = 0,
+ .dbc_bar_idx = 2,
+};
+
+static const struct qaic_device_config aic100_config = {
+ .family = FAMILY_AIC100,
+ .bar_mask = BIT(0) | BIT(2) | BIT(4),
+ .mhi_bar_idx = 0,
+ .dbc_bar_idx = 2,
+};
+
bool datapath_polling;
module_param(datapath_polling, bool, 0400);
MODULE_PARM_DESC(datapath_polling, "Operate the datapath in polling mode");
@@ -353,7 +378,8 @@ void qaic_dev_reset_clean_local_state(struct qaic_device *qdev)
release_dbc(qdev, i);
}
-static struct qaic_device *create_qdev(struct pci_dev *pdev, const struct pci_device_id *id)
+static struct qaic_device *create_qdev(struct pci_dev *pdev,
+ const struct qaic_device_config *config)
{
struct device *dev = &pdev->dev;
struct qaic_drm_device *qddev;
@@ -366,12 +392,10 @@ static struct qaic_device *create_qdev(struct pci_dev *pdev, const struct pci_de
return NULL;
qdev->dev_state = QAIC_OFFLINE;
- if (id->device == PCI_DEV_AIC080 || id->device == PCI_DEV_AIC100) {
- qdev->num_dbc = 16;
- qdev->dbc = devm_kcalloc(dev, qdev->num_dbc, sizeof(*qdev->dbc), GFP_KERNEL);
- if (!qdev->dbc)
- return NULL;
- }
+ qdev->num_dbc = 16;
+ qdev->dbc = devm_kcalloc(dev, qdev->num_dbc, sizeof(*qdev->dbc), GFP_KERNEL);
+ if (!qdev->dbc)
+ return NULL;
qddev = devm_drm_dev_alloc(&pdev->dev, &qaic_accel_driver, struct qaic_drm_device, drm);
if (IS_ERR(qddev))
@@ -430,7 +454,8 @@ static struct qaic_device *create_qdev(struct pci_dev *pdev, const struct pci_de
return qdev;
}
-static int init_pci(struct qaic_device *qdev, struct pci_dev *pdev)
+static int init_pci(struct qaic_device *qdev, struct pci_dev *pdev,
+ const struct qaic_device_config *config)
{
int bars;
int ret;
@@ -438,9 +463,9 @@ static int init_pci(struct qaic_device *qdev, struct pci_dev *pdev)
bars = pci_select_bars(pdev, IORESOURCE_MEM) & 0x3f;
/* make sure the device has the expected BARs */
- if (bars != (BIT(0) | BIT(2) | BIT(4))) {
- pci_dbg(pdev, "%s: expected BARs 0, 2, and 4 not found in device. Found 0x%x\n",
- __func__, bars);
+ if (bars != config->bar_mask) {
+ pci_dbg(pdev, "%s: expected BARs %#x not found in device. Found %#x\n",
+ __func__, config->bar_mask, bars);
return -EINVAL;
}
@@ -453,13 +478,13 @@ static int init_pci(struct qaic_device *qdev, struct pci_dev *pdev)
return ret;
dma_set_max_seg_size(&pdev->dev, UINT_MAX);
- qdev->bar_0 = devm_ioremap_resource(&pdev->dev, &pdev->resource[0]);
- if (IS_ERR(qdev->bar_0))
- return PTR_ERR(qdev->bar_0);
+ qdev->bar_mhi = devm_ioremap_resource(&pdev->dev, &pdev->resource[config->mhi_bar_idx]);
+ if (IS_ERR(qdev->bar_mhi))
+ return PTR_ERR(qdev->bar_mhi);
- qdev->bar_2 = devm_ioremap_resource(&pdev->dev, &pdev->resource[2]);
- if (IS_ERR(qdev->bar_2))
- return PTR_ERR(qdev->bar_2);
+ qdev->bar_dbc = devm_ioremap_resource(&pdev->dev, &pdev->resource[config->dbc_bar_idx]);
+ if (IS_ERR(qdev->bar_dbc))
+ return PTR_ERR(qdev->bar_dbc);
/* Managed release since we use pcim_enable_device above */
pci_set_master(pdev);
@@ -519,21 +544,22 @@ static int init_msi(struct qaic_device *qdev, struct pci_dev *pdev)
static int qaic_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
+ struct qaic_device_config *config = (struct qaic_device_config *)id->driver_data;
struct qaic_device *qdev;
int mhi_irq;
int ret;
int i;
- qdev = create_qdev(pdev, id);
+ qdev = create_qdev(pdev, config);
if (!qdev)
return -ENOMEM;
- ret = init_pci(qdev, pdev);
+ ret = init_pci(qdev, pdev, config);
if (ret)
return ret;
for (i = 0; i < qdev->num_dbc; ++i)
- qdev->dbc[i].dbc_base = qdev->bar_2 + QAIC_DBC_OFF(i);
+ qdev->dbc[i].dbc_base = qdev->bar_dbc + QAIC_DBC_OFF(i);
mhi_irq = init_msi(qdev, pdev);
if (mhi_irq < 0)
@@ -543,7 +569,7 @@ static int qaic_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
if (ret)
return ret;
- qdev->mhi_cntrl = qaic_mhi_register_controller(pdev, qdev->bar_0, mhi_irq,
+ qdev->mhi_cntrl = qaic_mhi_register_controller(pdev, qdev->bar_mhi, mhi_irq,
qdev->single_msi);
if (IS_ERR(qdev->mhi_cntrl)) {
ret = PTR_ERR(qdev->mhi_cntrl);
@@ -611,8 +637,8 @@ static struct mhi_driver qaic_mhi_driver = {
};
static const struct pci_device_id qaic_ids[] = {
- { PCI_DEVICE(PCI_VENDOR_ID_QCOM, PCI_DEV_AIC080), },
- { PCI_DEVICE(PCI_VENDOR_ID_QCOM, PCI_DEV_AIC100), },
+ { PCI_DEVICE_DATA(QCOM, AIC080, (kernel_ulong_t)&aic080_config), },
+ { PCI_DEVICE_DATA(QCOM, AIC100, (kernel_ulong_t)&aic100_config), },
{ }
};
MODULE_DEVICE_TABLE(pci, qaic_ids);
diff --git a/drivers/accel/qaic/qaic_timesync.c b/drivers/accel/qaic/qaic_timesync.c
index 301f4462d51b..2473c66309d4 100644
--- a/drivers/accel/qaic/qaic_timesync.c
+++ b/drivers/accel/qaic/qaic_timesync.c
@@ -201,7 +201,7 @@ static int qaic_timesync_probe(struct mhi_device *mhi_dev, const struct mhi_devi
goto free_sync_msg;
/* Qtimer register pointer */
- mqtsdev->qtimer_addr = qdev->bar_0 + QTIMER_REG_OFFSET;
+ mqtsdev->qtimer_addr = qdev->bar_mhi + QTIMER_REG_OFFSET;
timer_setup(timer, qaic_timesync_timer, 0);
timer->expires = jiffies + msecs_to_jiffies(timesync_delay_ms);
add_timer(timer);
diff --git a/drivers/acpi/acpi_platform.c b/drivers/acpi/acpi_platform.c
index 48d15dd785f6..affa076a211d 100644
--- a/drivers/acpi/acpi_platform.c
+++ b/drivers/acpi/acpi_platform.c
@@ -71,18 +71,15 @@ static struct notifier_block acpi_platform_notifier = {
.notifier_call = acpi_platform_device_remove_notify,
};
-static void acpi_platform_fill_resource(struct acpi_device *adev,
- const struct resource *src, struct resource *dest)
+static void acpi_platform_fill_resource(struct device *parent,
+ const struct resource *src,
+ struct resource *dest)
{
- struct device *parent;
-
*dest = *src;
-
/*
* If the device has parent we need to take its resources into
* account as well because this device might consume part of those.
*/
- parent = acpi_get_first_physical_node(acpi_dev_parent(adev));
if (parent && dev_is_pci(parent))
dest->parent = pci_find_resource(to_pci_dev(parent), dest);
}
@@ -110,7 +107,8 @@ static unsigned int acpi_platform_resource_count(struct acpi_resource *ares, voi
struct platform_device *acpi_create_platform_device(struct acpi_device *adev,
const struct property_entry *properties)
{
- struct acpi_device *parent = acpi_dev_parent(adev);
+ struct acpi_device *p = acpi_dev_parent(adev);
+ struct device *parent __free(put_device) = acpi_bus_get_primary_device(p);
struct platform_device *pdev = NULL;
struct platform_device_info pdevinfo;
const struct acpi_device_id *match;
@@ -149,7 +147,7 @@ struct platform_device *acpi_create_platform_device(struct acpi_device *adev,
}
count = 0;
list_for_each_entry(rentry, &resource_list, node)
- acpi_platform_fill_resource(adev, rentry->res,
+ acpi_platform_fill_resource(parent, rentry->res,
&resources[count++]);
acpi_dev_free_resource_list(&resource_list);
@@ -161,7 +159,7 @@ struct platform_device *acpi_create_platform_device(struct acpi_device *adev,
* attached to it, that physical device should be the parent of the
* platform device we are about to create.
*/
- pdevinfo.parent = parent ? acpi_get_first_physical_node(parent) : NULL;
+ pdevinfo.parent = parent;
pdevinfo.name = dev_name(&adev->dev);
pdevinfo.id = PLATFORM_DEVID_NONE;
pdevinfo.res = resources;
diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
index 9deb98c0a6fe..da0d6628ee41 100644
--- a/drivers/acpi/bus.c
+++ b/drivers/acpi/bus.c
@@ -747,30 +747,52 @@ static int __init acpi_setup_sb_notify_handler(void)
Device Matching
-------------------------------------------------------------------------- */
+
+static struct device *primary_physical_device(struct acpi_device *adev)
+{
+ struct acpi_device_physical_node *pn;
+
+ pn = list_first_entry_or_null(&adev->physical_node_list,
+ struct acpi_device_physical_node, node);
+ if (pn)
+ return pn->dev;
+
+ return NULL;
+}
+
/**
- * acpi_get_first_physical_node - Get first physical node of an ACPI device
+ * acpi_bus_get_primary_device - Get first physical device for a given ACPI one
+ * @adev: ACPI device to get the first physical device for.
+ *
+ * Find the first physical device for which @adev is the ACPI companion and
+ * reference count it if present.
+ *
+ * Return: Pointer to the first physical counterpart of @adev or NULL if there
+ * are none. Callers are responsible for invoking put_device() on the returned
+ * device.
+ */
+struct device *acpi_bus_get_primary_device(struct acpi_device *adev)
+{
+ if (!adev)
+ return NULL;
+
+ guard(mutex)(&adev->physical_node_lock);
+
+ return get_device(primary_physical_device(adev));
+}
+EXPORT_SYMBOL_GPL(acpi_bus_get_primary_device);
+
+/**
+ * acpi_get_first_physical_node - Find first physical node of an ACPI device
* @adev: ACPI device in question
*
* Return: First physical node of ACPI device @adev
*/
struct device *acpi_get_first_physical_node(struct acpi_device *adev)
{
- struct mutex *physical_node_lock = &adev->physical_node_lock;
- struct device *phys_dev;
-
- mutex_lock(physical_node_lock);
- if (list_empty(&adev->physical_node_list)) {
- phys_dev = NULL;
- } else {
- const struct acpi_device_physical_node *node;
+ guard(mutex)(&adev->physical_node_lock);
- node = list_first_entry(&adev->physical_node_list,
- struct acpi_device_physical_node, node);
-
- phys_dev = node->dev;
- }
- mutex_unlock(physical_node_lock);
- return phys_dev;
+ return primary_physical_device(adev);
}
EXPORT_SYMBOL_GPL(acpi_get_first_physical_node);
diff --git a/drivers/acpi/device_pm.c b/drivers/acpi/device_pm.c
index 3b4d048c4941..dbd4446025ec 100644
--- a/drivers/acpi/device_pm.c
+++ b/drivers/acpi/device_pm.c
@@ -1161,7 +1161,7 @@ EXPORT_SYMBOL_GPL(acpi_subsys_complete);
*/
int acpi_subsys_suspend(struct device *dev)
{
- if (!dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND) ||
+ if (!dev_pm_smart_suspend(dev) ||
acpi_dev_needs_resume(dev, ACPI_COMPANION(dev)))
pm_runtime_resume(dev);
@@ -1320,7 +1320,7 @@ EXPORT_SYMBOL_GPL(acpi_subsys_restore_early);
*/
int acpi_subsys_poweroff(struct device *dev)
{
- if (!dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND) ||
+ if (!dev_pm_smart_suspend(dev) ||
acpi_dev_needs_resume(dev, ACPI_COMPANION(dev)))
pm_runtime_resume(dev);
diff --git a/drivers/acpi/pci_link.c b/drivers/acpi/pci_link.c
index b727db968f33..ce022799e1b3 100644
--- a/drivers/acpi/pci_link.c
+++ b/drivers/acpi/pci_link.c
@@ -761,7 +761,7 @@ static int acpi_pci_link_resume(struct acpi_pci_link *link)
return 0;
}
-static void irqrouter_resume(void)
+static void irqrouter_resume(void *data)
{
struct acpi_pci_link *link;
@@ -888,10 +888,14 @@ static int __init acpi_irq_balance_set(char *str)
__setup("acpi_irq_balance", acpi_irq_balance_set);
-static struct syscore_ops irqrouter_syscore_ops = {
+static const struct syscore_ops irqrouter_syscore_ops = {
.resume = irqrouter_resume,
};
+static struct syscore irqrouter_syscore = {
+ .ops = &irqrouter_syscore_ops,
+};
+
void __init acpi_pci_link_init(void)
{
if (acpi_noirq)
@@ -904,6 +908,6 @@ void __init acpi_pci_link_init(void)
else
acpi_irq_balance = 0;
}
- register_syscore_ops(&irqrouter_syscore_ops);
+ register_syscore(&irqrouter_syscore);
acpi_scan_add_handler(&pci_link_handler);
}
diff --git a/drivers/acpi/sleep.c b/drivers/acpi/sleep.c
index 0b7fa4a8c379..1b9579bb97c2 100644
--- a/drivers/acpi/sleep.c
+++ b/drivers/acpi/sleep.c
@@ -892,13 +892,13 @@ bool acpi_s2idle_wakeup(void)
#ifdef CONFIG_PM_SLEEP
static u32 saved_bm_rld;
-static int acpi_save_bm_rld(void)
+static int acpi_save_bm_rld(void *data)
{
acpi_read_bit_register(ACPI_BITREG_BUS_MASTER_RLD, &saved_bm_rld);
return 0;
}
-static void acpi_restore_bm_rld(void)
+static void acpi_restore_bm_rld(void *data)
{
u32 resumed_bm_rld = 0;
@@ -909,14 +909,18 @@ static void acpi_restore_bm_rld(void)
acpi_write_bit_register(ACPI_BITREG_BUS_MASTER_RLD, saved_bm_rld);
}
-static struct syscore_ops acpi_sleep_syscore_ops = {
+static const struct syscore_ops acpi_sleep_syscore_ops = {
.suspend = acpi_save_bm_rld,
.resume = acpi_restore_bm_rld,
};
+static struct syscore acpi_sleep_syscore = {
+ .ops = &acpi_sleep_syscore_ops,
+};
+
static void acpi_sleep_syscore_init(void)
{
- register_syscore_ops(&acpi_sleep_syscore_ops);
+ register_syscore(&acpi_sleep_syscore);
}
#else
static inline void acpi_sleep_syscore_init(void) {}
diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index caf77faf7a33..fe3ed016b6d1 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -4226,6 +4226,13 @@ static const struct ata_dev_quirks_entry __ata_dev_quirks[] = {
/* Apacer models with LPM issues */
{ "Apacer AS340*", NULL, ATA_QUIRK_NOLPM },
+ /* PNY CS900 (Phison PS3111-S11, DRAM-less) drops the link on DIPM */
+ { "PNY CS900 1TB SSD", NULL, ATA_QUIRK_NOLPM },
+
+ /* Silicon Motion models with LPM issues */
+ { "MD619HXCLDE3TC", "TCVAID", ATA_QUIRK_NOLPM },
+ { "MD619GXCLDE3TC", "TCV35D", ATA_QUIRK_NOLPM },
+
/* These specific Samsung models/firmware-revs do not handle LPM well */
{ "SAMSUNG MZMPC128HBFU-000MV", "CXM14M1Q", ATA_QUIRK_NOLPM },
{ "SAMSUNG SSD PM830 mSATA *", "CXM13D1Q", ATA_QUIRK_NOLPM },
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index d2777255129c..d4f6d82fd908 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -3030,8 +3030,14 @@ static void atapi_qc_complete(struct ata_queued_cmd *qc)
if (qc->cdb[0] == ALLOW_MEDIUM_REMOVAL && qc->dev->sdev)
qc->dev->sdev->locked = 0;
- if (!cmd->result)
- cmd->result = SAM_STAT_CHECK_CONDITION;
+ /*
+ * Report CHECK CONDITION unless the command was terminated
+ * with a host byte set. Only set the status byte, so that the
+ * SCSI midlayer internal byte, which scsi_check_sense() may
+ * have set from the sense data, is preserved.
+ */
+ if (get_host_byte(cmd) == DID_OK)
+ set_status_byte(cmd, SAM_STAT_CHECK_CONDITION);
ata_qc_done(qc);
return;
}
diff --git a/drivers/base/core.c b/drivers/base/core.c
index a6134a5e5e11..130354f898e7 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -533,9 +533,9 @@ static ssize_t auto_remove_on_show(struct device *dev,
struct device_link *link = to_devlink(dev);
const char *output;
- if (link->flags & DL_FLAG_AUTOREMOVE_SUPPLIER)
+ if (device_link_test(link, DL_FLAG_AUTOREMOVE_SUPPLIER))
output = "supplier unbind";
- else if (link->flags & DL_FLAG_AUTOREMOVE_CONSUMER)
+ else if (device_link_test(link, DL_FLAG_AUTOREMOVE_CONSUMER))
output = "consumer unbind";
else
output = "never";
@@ -549,7 +549,7 @@ static ssize_t runtime_pm_show(struct device *dev,
{
struct device_link *link = to_devlink(dev);
- return sysfs_emit(buf, "%d\n", !!(link->flags & DL_FLAG_PM_RUNTIME));
+ return sysfs_emit(buf, "%d\n", device_link_test(link, DL_FLAG_PM_RUNTIME));
}
static DEVICE_ATTR_RO(runtime_pm);
@@ -558,8 +558,7 @@ static ssize_t sync_state_only_show(struct device *dev,
{
struct device_link *link = to_devlink(dev);
- return sysfs_emit(buf, "%d\n",
- !!(link->flags & DL_FLAG_SYNC_STATE_ONLY));
+ return sysfs_emit(buf, "%d\n", device_link_test(link, DL_FLAG_SYNC_STATE_ONLY));
}
static DEVICE_ATTR_RO(sync_state_only);
@@ -865,12 +864,12 @@ struct device_link *device_link_add(struct device *consumer,
if (link->consumer != consumer)
continue;
- if (link->flags & DL_FLAG_INFERRED &&
+ if (device_link_test(link, DL_FLAG_INFERRED) &&
!(flags & DL_FLAG_INFERRED))
link->flags &= ~DL_FLAG_INFERRED;
if (flags & DL_FLAG_PM_RUNTIME) {
- if (!(link->flags & DL_FLAG_PM_RUNTIME)) {
+ if (!device_link_test(link, DL_FLAG_PM_RUNTIME)) {
pm_runtime_new_link(consumer);
link->flags |= DL_FLAG_PM_RUNTIME;
}
@@ -880,8 +879,8 @@ struct device_link *device_link_add(struct device *consumer,
if (flags & DL_FLAG_STATELESS) {
kref_get(&link->kref);
- if (link->flags & DL_FLAG_SYNC_STATE_ONLY &&
- !(link->flags & DL_FLAG_STATELESS)) {
+ if (device_link_test(link, DL_FLAG_SYNC_STATE_ONLY) &&
+ !device_link_test(link, DL_FLAG_STATELESS)) {
link->flags |= DL_FLAG_STATELESS;
goto reorder;
} else {
@@ -896,7 +895,7 @@ struct device_link *device_link_add(struct device *consumer,
* update the existing link to stay around longer.
*/
if (flags & DL_FLAG_AUTOREMOVE_SUPPLIER) {
- if (link->flags & DL_FLAG_AUTOREMOVE_CONSUMER) {
+ if (device_link_test(link, DL_FLAG_AUTOREMOVE_CONSUMER)) {
link->flags &= ~DL_FLAG_AUTOREMOVE_CONSUMER;
link->flags |= DL_FLAG_AUTOREMOVE_SUPPLIER;
}
@@ -904,12 +903,12 @@ struct device_link *device_link_add(struct device *consumer,
link->flags &= ~(DL_FLAG_AUTOREMOVE_CONSUMER |
DL_FLAG_AUTOREMOVE_SUPPLIER);
}
- if (!(link->flags & DL_FLAG_MANAGED)) {
+ if (!device_link_test(link, DL_FLAG_MANAGED)) {
kref_get(&link->kref);
link->flags |= DL_FLAG_MANAGED;
device_link_init_status(link, consumer, supplier);
}
- if (link->flags & DL_FLAG_SYNC_STATE_ONLY &&
+ if (device_link_test(link, DL_FLAG_SYNC_STATE_ONLY) &&
!(flags & DL_FLAG_SYNC_STATE_ONLY)) {
link->flags &= ~DL_FLAG_SYNC_STATE_ONLY;
goto reorder;
@@ -1013,7 +1012,7 @@ static void __device_link_del(struct kref *kref)
static void device_link_put_kref(struct device_link *link)
{
- if (link->flags & DL_FLAG_STATELESS)
+ if (device_link_test(link, DL_FLAG_STATELESS))
kref_put(&link->kref, __device_link_del);
else if (!device_is_registered(link->consumer))
__device_link_del(&link->kref);
@@ -1077,7 +1076,7 @@ static void device_links_missing_supplier(struct device *dev)
if (link->supplier->links.status == DL_DEV_DRIVER_BOUND) {
WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
} else {
- WARN_ON(!(link->flags & DL_FLAG_SYNC_STATE_ONLY));
+ WARN_ON(!device_link_test(link, DL_FLAG_SYNC_STATE_ONLY));
WRITE_ONCE(link->status, DL_STATE_DORMANT);
}
}
@@ -1145,14 +1144,14 @@ int device_links_check_suppliers(struct device *dev)
device_links_write_lock();
list_for_each_entry(link, &dev->links.suppliers, c_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
if (link->status != DL_STATE_AVAILABLE &&
- !(link->flags & DL_FLAG_SYNC_STATE_ONLY)) {
+ !device_link_test(link, DL_FLAG_SYNC_STATE_ONLY)) {
if (dev_is_best_effort(dev) &&
- link->flags & DL_FLAG_INFERRED &&
+ device_link_test(link, DL_FLAG_INFERRED) &&
!link->supplier->can_match) {
ret = -EAGAIN;
continue;
@@ -1201,7 +1200,7 @@ static void __device_links_queue_sync_state(struct device *dev,
return;
list_for_each_entry(link, &dev->links.consumers, s_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
if (link->status != DL_STATE_ACTIVE)
return;
@@ -1341,7 +1340,7 @@ void device_links_force_bind(struct device *dev)
device_links_write_lock();
list_for_each_entry_safe(link, ln, &dev->links.suppliers, c_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
if (link->status != DL_STATE_AVAILABLE) {
@@ -1394,7 +1393,7 @@ void device_links_driver_bound(struct device *dev)
device_links_write_lock();
list_for_each_entry(link, &dev->links.consumers, s_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
/*
@@ -1410,7 +1409,7 @@ void device_links_driver_bound(struct device *dev)
WARN_ON(link->status != DL_STATE_DORMANT);
WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
- if (link->flags & DL_FLAG_AUTOPROBE_CONSUMER)
+ if (device_link_test(link, DL_FLAG_AUTOPROBE_CONSUMER))
driver_deferred_probe_add(link->consumer);
}
@@ -1422,11 +1421,11 @@ void device_links_driver_bound(struct device *dev)
list_for_each_entry_safe(link, ln, &dev->links.suppliers, c_node) {
struct device *supplier;
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
supplier = link->supplier;
- if (link->flags & DL_FLAG_SYNC_STATE_ONLY) {
+ if (device_link_test(link, DL_FLAG_SYNC_STATE_ONLY)) {
/*
* When DL_FLAG_SYNC_STATE_ONLY is set, it means no
* other DL_MANAGED_LINK_FLAGS have been set. So, it's
@@ -1434,7 +1433,7 @@ void device_links_driver_bound(struct device *dev)
*/
device_link_drop_managed(link);
} else if (dev_is_best_effort(dev) &&
- link->flags & DL_FLAG_INFERRED &&
+ device_link_test(link, DL_FLAG_INFERRED) &&
link->status != DL_STATE_CONSUMER_PROBE &&
!link->supplier->can_match) {
/*
@@ -1486,10 +1485,10 @@ static void __device_links_no_driver(struct device *dev)
struct device_link *link, *ln;
list_for_each_entry_safe_reverse(link, ln, &dev->links.suppliers, c_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
- if (link->flags & DL_FLAG_AUTOREMOVE_CONSUMER) {
+ if (device_link_test(link, DL_FLAG_AUTOREMOVE_CONSUMER)) {
device_link_drop_managed(link);
continue;
}
@@ -1501,7 +1500,8 @@ static void __device_links_no_driver(struct device *dev)
if (link->supplier->links.status == DL_DEV_DRIVER_BOUND) {
WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
} else {
- WARN_ON(!(link->flags & DL_FLAG_SYNC_STATE_ONLY));
+ WARN_ON(link->supplier->links.status != DL_DEV_UNBINDING &&
+ !device_link_test(link, DL_FLAG_SYNC_STATE_ONLY));
WRITE_ONCE(link->status, DL_STATE_DORMANT);
}
}
@@ -1526,7 +1526,7 @@ void device_links_no_driver(struct device *dev)
device_links_write_lock();
list_for_each_entry(link, &dev->links.consumers, s_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
/*
@@ -1563,10 +1563,10 @@ void device_links_driver_cleanup(struct device *dev)
device_links_write_lock();
list_for_each_entry_safe(link, ln, &dev->links.consumers, s_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
- WARN_ON(link->flags & DL_FLAG_AUTOREMOVE_CONSUMER);
+ WARN_ON(device_link_test(link, DL_FLAG_AUTOREMOVE_CONSUMER));
WARN_ON(link->status != DL_STATE_SUPPLIER_UNBIND);
/*
@@ -1575,7 +1575,7 @@ void device_links_driver_cleanup(struct device *dev)
* has moved to DL_STATE_SUPPLIER_UNBIND.
*/
if (link->status == DL_STATE_SUPPLIER_UNBIND &&
- link->flags & DL_FLAG_AUTOREMOVE_SUPPLIER)
+ device_link_test(link, DL_FLAG_AUTOREMOVE_SUPPLIER))
device_link_drop_managed(link);
WRITE_ONCE(link->status, DL_STATE_DORMANT);
@@ -1609,7 +1609,7 @@ bool device_links_busy(struct device *dev)
device_links_write_lock();
list_for_each_entry(link, &dev->links.consumers, s_node) {
- if (!(link->flags & DL_FLAG_MANAGED))
+ if (!device_link_test(link, DL_FLAG_MANAGED))
continue;
if (link->status == DL_STATE_CONSUMER_PROBE
@@ -1651,8 +1651,8 @@ void device_links_unbind_consumers(struct device *dev)
list_for_each_entry(link, &dev->links.consumers, s_node) {
enum device_link_state status;
- if (!(link->flags & DL_FLAG_MANAGED) ||
- link->flags & DL_FLAG_SYNC_STATE_ONLY)
+ if (!device_link_test(link, DL_FLAG_MANAGED) ||
+ device_link_test(link, DL_FLAG_SYNC_STATE_ONLY))
continue;
status = link->status;
@@ -1808,7 +1808,7 @@ static void fw_devlink_parse_fwtree(struct fwnode_handle *fwnode)
static void fw_devlink_relax_link(struct device_link *link)
{
- if (!(link->flags & DL_FLAG_INFERRED))
+ if (!device_link_test(link, DL_FLAG_INFERRED))
return;
if (device_link_flag_is_sync_state_only(link->flags))
@@ -1844,7 +1844,7 @@ static int fw_devlink_dev_sync_state(struct device *dev, void *data)
struct device_link *link = to_devlink(dev);
struct device *sup = link->supplier;
- if (!(link->flags & DL_FLAG_MANAGED) ||
+ if (!device_link_test(link, DL_FLAG_MANAGED) ||
link->status == DL_STATE_ACTIVE || sup->state_synced ||
!dev_has_sync_state(sup))
return 0;
@@ -2126,7 +2126,7 @@ static bool __fw_devlink_relax_cycles(struct fwnode_handle *con_handle,
* such due to a cycle.
*/
if (device_link_flag_is_sync_state_only(dev_link->flags) &&
- !(dev_link->flags & DL_FLAG_CYCLE))
+ !device_link_test(dev_link, DL_FLAG_CYCLE))
continue;
if (__fw_devlink_relax_cycles(con_handle,
diff --git a/drivers/base/firmware_loader/main.c b/drivers/base/firmware_loader/main.c
index b4c4a5329570..b894687b059c 100644
--- a/drivers/base/firmware_loader/main.c
+++ b/drivers/base/firmware_loader/main.c
@@ -1621,16 +1621,20 @@ static int fw_pm_notify(struct notifier_block *notify_block,
}
/* stop caching firmware once syscore_suspend is reached */
-static int fw_suspend(void)
+static int fw_suspend(void *data)
{
fw_cache.state = FW_LOADER_NO_CACHE;
return 0;
}
-static struct syscore_ops fw_syscore_ops = {
+static const struct syscore_ops fw_syscore_ops = {
.suspend = fw_suspend,
};
+static struct syscore fw_syscore = {
+ .ops = &fw_syscore_ops,
+};
+
static int __init register_fw_pm_ops(void)
{
int ret;
@@ -1646,14 +1650,14 @@ static int __init register_fw_pm_ops(void)
if (ret)
return ret;
- register_syscore_ops(&fw_syscore_ops);
+ register_syscore(&fw_syscore);
return ret;
}
static inline void unregister_fw_pm_ops(void)
{
- unregister_syscore_ops(&fw_syscore_ops);
+ unregister_syscore(&fw_syscore);
unregister_pm_notifier(&fw_cache.pm_notify);
}
#else
diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index abb16a5bb796..ab240ccf7fd4 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -654,12 +654,12 @@ static void device_resume_noirq(struct device *dev, pm_message_t state, bool asy
* so change its status accordingly.
*
* Otherwise, the device is going to be resumed, so set its PM-runtime
- * status to "active", but do that only if DPM_FLAG_SMART_SUSPEND is set
- * to avoid confusing drivers that don't use it.
+ * status to "active" unless its power.smart_suspend flag is clear, in
+ * which case it is not necessary to update its PM-runtime status.
*/
if (skip_resume)
pm_runtime_set_suspended(dev);
- else if (dev_pm_skip_suspend(dev))
+ else if (dev_pm_smart_suspend(dev))
pm_runtime_set_active(dev);
if (dev->pm_domain) {
@@ -1105,6 +1105,8 @@ static void device_complete(struct device *dev, pm_message_t state)
device_unlock(dev);
out:
+ /* If enabling runtime PM for the device is blocked, unblock it. */
+ pm_runtime_unblock(dev);
pm_runtime_put(dev);
}
@@ -1393,6 +1395,10 @@ static int device_suspend_late(struct device *dev, pm_message_t state, bool asyn
TRACE_DEVICE(dev);
TRACE_SUSPEND(0);
+ /*
+ * Disable runtime PM for the device without checking if there is a
+ * pending resume request for it.
+ */
__pm_runtime_disable(dev, false);
dpm_wait_for_subordinate(dev, async);
@@ -1781,6 +1787,49 @@ int dpm_suspend(pm_message_t state)
return error;
}
+static void device_prepare_smart_suspend(struct device *dev)
+{
+ struct device_link *link;
+ int idx;
+
+ /*
+ * The "smart suspend" feature is enabled for devices whose drivers ask
+ * for it and for devices without PM callbacks unless runtime PM is
+ * disabled and enabling it is blocked for them.
+ *
+ * However, if "smart suspend" is not enabled for the device's parent
+ * or any of its suppliers that take runtime PM into account, it cannot
+ * be enabled for the device either.
+ */
+ dev->power.smart_suspend = (dev->power.no_pm_callbacks ||
+ dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND)) &&
+ !pm_runtime_blocked(dev);
+
+ if (!dev_pm_smart_suspend(dev))
+ return;
+
+ if (dev->parent && !dev_pm_smart_suspend(dev->parent) &&
+ !dev->parent->power.ignore_children && !pm_runtime_blocked(dev->parent)) {
+ dev->power.smart_suspend = false;
+ return;
+ }
+
+ idx = device_links_read_lock();
+
+ list_for_each_entry_rcu_locked(link, &dev->links.suppliers, c_node) {
+ if (!device_link_test(link, DL_FLAG_PM_RUNTIME))
+ continue;
+
+ if (!dev_pm_smart_suspend(link->supplier) &&
+ !pm_runtime_blocked(link->supplier)) {
+ dev->power.smart_suspend = false;
+ break;
+ }
+ }
+
+ device_links_read_unlock(idx);
+}
+
/**
* device_prepare - Prepare a device for system power transition.
* @dev: Device to handle.
@@ -1801,6 +1850,13 @@ static int device_prepare(struct device *dev, pm_message_t state)
* it again during the complete phase.
*/
pm_runtime_get_noresume(dev);
+ /*
+ * If runtime PM is disabled for the device at this point and it has
+ * never been enabled so far, it should not be enabled until this system
+ * suspend-resume cycle is complete, so prepare to trigger a warning on
+ * subsequent attempts to enable it.
+ */
+ pm_runtime_block_if_disabled(dev);
if (dev->power.syscore)
return 0;
@@ -1832,9 +1888,11 @@ static int device_prepare(struct device *dev, pm_message_t state)
if (ret < 0) {
suspend_report_result(dev, callback, ret);
+ pm_runtime_unblock(dev);
pm_runtime_put(dev);
return ret;
}
+ device_prepare_smart_suspend(dev);
/*
* A positive return value from ->prepare() means "this device appears
* to be runtime-suspended and its state is fine, so if it really is
@@ -2010,6 +2068,5 @@ void device_pm_check_callbacks(struct device *dev)
bool dev_pm_skip_suspend(struct device *dev)
{
- return dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND) &&
- pm_runtime_status_suspended(dev);
+ return dev_pm_smart_suspend(dev) && pm_runtime_status_suspended(dev);
}
diff --git a/drivers/base/power/runtime.c b/drivers/base/power/runtime.c
index 167ff6f7a3fe..b0b72fa2a120 100644
--- a/drivers/base/power/runtime.c
+++ b/drivers/base/power/runtime.c
@@ -290,7 +290,7 @@ static int rpm_get_suppliers(struct device *dev)
device_links_read_lock_held()) {
int retval;
- if (!(link->flags & DL_FLAG_PM_RUNTIME))
+ if (!device_link_test(link, DL_FLAG_PM_RUNTIME))
continue;
retval = pm_runtime_get_sync(link->supplier);
@@ -1469,20 +1469,26 @@ int pm_runtime_barrier(struct device *dev)
}
EXPORT_SYMBOL_GPL(pm_runtime_barrier);
-/**
- * __pm_runtime_disable - Disable runtime PM of a device.
- * @dev: Device to handle.
- * @check_resume: If set, check if there's a resume request for the device.
- *
- * Increment power.disable_depth for the device and if it was zero previously,
- * cancel all pending runtime PM requests for the device and wait for all
- * operations in progress to complete. The device can be either active or
- * suspended after its runtime PM has been disabled.
- *
- * If @check_resume is set and there's a resume request pending when
- * __pm_runtime_disable() is called and power.disable_depth is zero, the
- * function will wake up the device before disabling its runtime PM.
- */
+void pm_runtime_block_if_disabled(struct device *dev)
+{
+ spin_lock_irq(&dev->power.lock);
+
+ if (dev->power.disable_depth && dev->power.last_status == RPM_INVALID)
+ dev->power.last_status = RPM_BLOCKED;
+
+ spin_unlock_irq(&dev->power.lock);
+}
+
+void pm_runtime_unblock(struct device *dev)
+{
+ spin_lock_irq(&dev->power.lock);
+
+ if (dev->power.last_status == RPM_BLOCKED)
+ dev->power.last_status = RPM_INVALID;
+
+ spin_unlock_irq(&dev->power.lock);
+}
+
void __pm_runtime_disable(struct device *dev, bool check_resume)
{
spin_lock_irq(&dev->power.lock);
@@ -1541,6 +1547,10 @@ void pm_runtime_enable(struct device *dev)
if (--dev->power.disable_depth > 0)
goto out;
+ if (dev->power.last_status == RPM_BLOCKED) {
+ dev_warn(dev, "Attempt to enable runtime PM when it is blocked\n");
+ dump_stack();
+ }
dev->power.last_status = RPM_INVALID;
dev->power.accounting_timestamp = ktime_get_mono_fast_ns();
@@ -1580,6 +1590,23 @@ int devm_pm_runtime_set_active_enabled(struct device *dev)
}
EXPORT_SYMBOL_GPL(devm_pm_runtime_set_active_enabled);
+bool pm_runtime_blocked(struct device *dev)
+{
+ bool ret;
+
+ /*
+ * dev->power.last_status is a bit field, so in case it is updated via
+ * RMW, read it under the spin lock.
+ */
+ spin_lock_irq(&dev->power.lock);
+
+ ret = dev->power.last_status == RPM_BLOCKED;
+
+ spin_unlock_irq(&dev->power.lock);
+
+ return ret;
+}
+
static void pm_runtime_disable_action(void *data)
{
pm_runtime_dont_use_autosuspend(data);
@@ -1873,7 +1900,7 @@ void pm_runtime_get_suppliers(struct device *dev)
list_for_each_entry_rcu(link, &dev->links.suppliers, c_node,
device_links_read_lock_held())
- if (link->flags & DL_FLAG_PM_RUNTIME) {
+ if (device_link_test(link, DL_FLAG_PM_RUNTIME)) {
link->supplier_preactivated = true;
pm_runtime_get_sync(link->supplier);
}
@@ -1927,7 +1954,7 @@ static void pm_runtime_drop_link_count(struct device *dev)
*/
void pm_runtime_drop_link(struct device_link *link)
{
- if (!(link->flags & DL_FLAG_PM_RUNTIME))
+ if (!device_link_test(link, DL_FLAG_PM_RUNTIME))
return;
pm_runtime_drop_link_count(link->consumer);
diff --git a/drivers/base/syscore.c b/drivers/base/syscore.c
index 13db1f78d2ce..483adb796654 100644
--- a/drivers/base/syscore.c
+++ b/drivers/base/syscore.c
@@ -11,32 +11,32 @@
#include <linux/suspend.h>
#include <trace/events/power.h>
-static LIST_HEAD(syscore_ops_list);
-static DEFINE_MUTEX(syscore_ops_lock);
+static LIST_HEAD(syscore_list);
+static DEFINE_MUTEX(syscore_lock);
/**
- * register_syscore_ops - Register a set of system core operations.
- * @ops: System core operations to register.
+ * register_syscore - Register a set of system core operations.
+ * @syscore: System core operations to register.
*/
-void register_syscore_ops(struct syscore_ops *ops)
+void register_syscore(struct syscore *syscore)
{
- mutex_lock(&syscore_ops_lock);
- list_add_tail(&ops->node, &syscore_ops_list);
- mutex_unlock(&syscore_ops_lock);
+ mutex_lock(&syscore_lock);
+ list_add_tail(&syscore->node, &syscore_list);
+ mutex_unlock(&syscore_lock);
}
-EXPORT_SYMBOL_GPL(register_syscore_ops);
+EXPORT_SYMBOL_GPL(register_syscore);
/**
- * unregister_syscore_ops - Unregister a set of system core operations.
- * @ops: System core operations to unregister.
+ * unregister_syscore - Unregister a set of system core operations.
+ * @syscore: System core operations to unregister.
*/
-void unregister_syscore_ops(struct syscore_ops *ops)
+void unregister_syscore(struct syscore *syscore)
{
- mutex_lock(&syscore_ops_lock);
- list_del(&ops->node);
- mutex_unlock(&syscore_ops_lock);
+ mutex_lock(&syscore_lock);
+ list_del(&syscore->node);
+ mutex_unlock(&syscore_lock);
}
-EXPORT_SYMBOL_GPL(unregister_syscore_ops);
+EXPORT_SYMBOL_GPL(unregister_syscore);
#ifdef CONFIG_PM_SLEEP
/**
@@ -46,7 +46,7 @@ EXPORT_SYMBOL_GPL(unregister_syscore_ops);
*/
int syscore_suspend(void)
{
- struct syscore_ops *ops;
+ struct syscore *syscore;
int ret = 0;
trace_suspend_resume(TPS("syscore_suspend"), 0, true);
@@ -59,25 +59,27 @@ int syscore_suspend(void)
WARN_ONCE(!irqs_disabled(),
"Interrupts enabled before system core suspend.\n");
- list_for_each_entry_reverse(ops, &syscore_ops_list, node)
- if (ops->suspend) {
- pm_pr_dbg("Calling %pS\n", ops->suspend);
- ret = ops->suspend();
+ list_for_each_entry_reverse(syscore, &syscore_list, node)
+ if (syscore->ops->suspend) {
+ pm_pr_dbg("Calling %pS\n", syscore->ops->suspend);
+ ret = syscore->ops->suspend(syscore->data);
if (ret)
goto err_out;
WARN_ONCE(!irqs_disabled(),
- "Interrupts enabled after %pS\n", ops->suspend);
+ "Interrupts enabled after %pS\n",
+ syscore->ops->suspend);
}
trace_suspend_resume(TPS("syscore_suspend"), 0, false);
return 0;
err_out:
- pr_err("PM: System core suspend callback %pS failed.\n", ops->suspend);
+ pr_err("PM: System core suspend callback %pS failed.\n",
+ syscore->ops->suspend);
- list_for_each_entry_continue(ops, &syscore_ops_list, node)
- if (ops->resume)
- ops->resume();
+ list_for_each_entry_continue(syscore, &syscore_list, node)
+ if (syscore->ops->resume)
+ syscore->ops->resume(syscore->data);
return ret;
}
@@ -90,18 +92,19 @@ EXPORT_SYMBOL_GPL(syscore_suspend);
*/
void syscore_resume(void)
{
- struct syscore_ops *ops;
+ struct syscore *syscore;
trace_suspend_resume(TPS("syscore_resume"), 0, true);
WARN_ONCE(!irqs_disabled(),
"Interrupts enabled before system core resume.\n");
- list_for_each_entry(ops, &syscore_ops_list, node)
- if (ops->resume) {
- pm_pr_dbg("Calling %pS\n", ops->resume);
- ops->resume();
+ list_for_each_entry(syscore, &syscore_list, node)
+ if (syscore->ops->resume) {
+ pm_pr_dbg("Calling %pS\n", syscore->ops->resume);
+ syscore->ops->resume(syscore->data);
WARN_ONCE(!irqs_disabled(),
- "Interrupts enabled after %pS\n", ops->resume);
+ "Interrupts enabled after %pS\n",
+ syscore->ops->resume);
}
trace_suspend_resume(TPS("syscore_resume"), 0, false);
}
@@ -113,16 +116,17 @@ EXPORT_SYMBOL_GPL(syscore_resume);
*/
void syscore_shutdown(void)
{
- struct syscore_ops *ops;
+ struct syscore *syscore;
- mutex_lock(&syscore_ops_lock);
+ mutex_lock(&syscore_lock);
- list_for_each_entry_reverse(ops, &syscore_ops_list, node)
- if (ops->shutdown) {
+ list_for_each_entry_reverse(syscore, &syscore_list, node)
+ if (syscore->ops->shutdown) {
if (initcall_debug)
- pr_info("PM: Calling %pS\n", ops->shutdown);
- ops->shutdown();
+ pr_info("PM: Calling %pS\n",
+ syscore->ops->shutdown);
+ syscore->ops->shutdown(syscore->data);
}
- mutex_unlock(&syscore_ops_lock);
+ mutex_unlock(&syscore_lock);
}
diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index 6d5523058bed..085a4d624659 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -915,7 +915,10 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req,
struct iov_iter iter;
const int dir = ITER_DEST;
- import_ubuf(dir, u64_to_user_ptr(io->addr), rq_bytes, &iter);
+ if (import_ubuf(dir, u64_to_user_ptr(io->addr), rq_bytes,
+ &iter) < 0)
+ return 0;
+
return ublk_copy_user_pages(req, 0, &iter, dir);
}
return rq_bytes;
@@ -936,7 +939,10 @@ static int ublk_unmap_io(const struct ublk_queue *ubq,
WARN_ON_ONCE(io->res > rq_bytes);
- import_ubuf(dir, u64_to_user_ptr(io->addr), io->res, &iter);
+ if (import_ubuf(dir, u64_to_user_ptr(io->addr), io->res,
+ &iter) < 0)
+ return 0;
+
return ublk_copy_user_pages(req, 0, &iter, dir);
}
return rq_bytes;
@@ -1552,14 +1558,31 @@ static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq)
return !was_canceled;
}
-static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io,
+static void ublk_cancel_cmd(struct ublk_queue *ubq, unsigned tag,
unsigned int issue_flags)
{
+ struct ublk_io *io = &ubq->ios[tag];
+ struct ublk_device *ub = ubq->dev;
+ struct request *req;
bool done;
if (!(io->flags & UBLK_IO_FLAG_ACTIVE))
return;
+ /*
+ * Don't try to cancel this command if the request is started for
+ * avoiding race between io_uring_cmd_done() and
+ * io_uring_cmd_complete_in_task().
+ *
+ * Either the started request will be aborted via __ublk_abort_rq(),
+ * then this uring_cmd is canceled next time, or it will be done in
+ * task work function ublk_dispatch_req() because io_uring guarantees
+ * that ublk_dispatch_req() is always called
+ */
+ req = blk_mq_tag_to_rq(ub->tag_set.tags[ubq->q_id], tag);
+ if (req && blk_mq_request_started(req) && req->tag == tag)
+ return;
+
spin_lock(&ubq->cancel_lock);
done = !!(io->flags & UBLK_IO_FLAG_CANCELED);
if (!done)
@@ -1582,7 +1605,6 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd,
struct task_struct *task;
struct ublk_device *ub;
bool need_schedule;
- struct ublk_io *io;
if (WARN_ON_ONCE(!ubq))
return;
@@ -1597,9 +1619,8 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd,
ub = ubq->dev;
need_schedule = ublk_abort_requests(ub, ubq);
- io = &ubq->ios[pdu->tag];
- WARN_ON_ONCE(io->cmd != cmd);
- ublk_cancel_cmd(ubq, io, issue_flags);
+ WARN_ON_ONCE(ubq->ios[pdu->tag].cmd != cmd);
+ ublk_cancel_cmd(ubq, pdu->tag, issue_flags);
if (need_schedule) {
if (ublk_nosrv_should_stop_dev(ub))
@@ -1619,7 +1640,7 @@ static void ublk_cancel_queue(struct ublk_queue *ubq)
int i;
for (i = 0; i < ubq->q_depth; i++)
- ublk_cancel_cmd(ubq, &ubq->ios[i], IO_URING_F_UNLOCKED);
+ ublk_cancel_cmd(ubq, i, IO_URING_F_UNLOCKED);
}
/* Cancel all pending commands, must be called after del_gendisk() returns */
diff --git a/drivers/bus/mhi/host/pci_generic.c b/drivers/bus/mhi/host/pci_generic.c
index 833d0c940477..5fc526ddece2 100644
--- a/drivers/bus/mhi/host/pci_generic.c
+++ b/drivers/bus/mhi/host/pci_generic.c
@@ -86,22 +86,6 @@ struct mhi_pci_dev_info {
.doorbell_mode_switch = false, \
}
-#define MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(ch_num, ch_name, el_count, ev_ring) \
- { \
- .num = ch_num, \
- .name = ch_name, \
- .num_elements = el_count, \
- .event_ring = ev_ring, \
- .dir = DMA_FROM_DEVICE, \
- .ee_mask = BIT(MHI_EE_AMSS), \
- .pollcfg = 0, \
- .doorbell = MHI_DB_BRST_DISABLE, \
- .lpm_notify = false, \
- .offload_channel = false, \
- .doorbell_mode_switch = false, \
- .auto_queue = true, \
- }
-
#define MHI_EVENT_CONFIG_CTRL(ev_ring, el_count) \
{ \
.num_elements = el_count, \
@@ -253,7 +237,7 @@ static const struct mhi_channel_config modem_qcom_v1_mhi_channels[] = {
MHI_CHANNEL_CONFIG_UL(14, "QMI", 4, 0),
MHI_CHANNEL_CONFIG_DL(15, "QMI", 4, 0),
MHI_CHANNEL_CONFIG_UL(20, "IPCR", 8, 0),
- MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(21, "IPCR", 8, 0),
+ MHI_CHANNEL_CONFIG_DL(21, "IPCR", 8, 0),
MHI_CHANNEL_CONFIG_UL_FP(34, "FIREHOSE", 32, 0),
MHI_CHANNEL_CONFIG_DL_FP(35, "FIREHOSE", 32, 0),
MHI_CHANNEL_CONFIG_UL(46, "IP_SW0", 64, 2),
@@ -618,7 +602,7 @@ static const struct mhi_channel_config mhi_telit_fn980_hw_v1_channels[] = {
MHI_CHANNEL_CONFIG_UL(14, "QMI", 32, 0),
MHI_CHANNEL_CONFIG_DL(15, "QMI", 32, 0),
MHI_CHANNEL_CONFIG_UL(20, "IPCR", 16, 0),
- MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(21, "IPCR", 16, 0),
+ MHI_CHANNEL_CONFIG_DL(21, "IPCR", 16, 0),
MHI_CHANNEL_CONFIG_HW_UL(100, "IP_HW0", 128, 1),
MHI_CHANNEL_CONFIG_HW_DL(101, "IP_HW0", 128, 2),
};
diff --git a/drivers/bus/mvebu-mbus.c b/drivers/bus/mvebu-mbus.c
index 00cb792bda18..dd94145c9b22 100644
--- a/drivers/bus/mvebu-mbus.c
+++ b/drivers/bus/mvebu-mbus.c
@@ -1006,7 +1006,7 @@ static __init int mvebu_mbus_debugfs_init(void)
}
fs_initcall(mvebu_mbus_debugfs_init);
-static int mvebu_mbus_suspend(void)
+static int mvebu_mbus_suspend(void *data)
{
struct mvebu_mbus_state *s = &mbus_state;
int win;
@@ -1040,7 +1040,7 @@ static int mvebu_mbus_suspend(void)
return 0;
}
-static void mvebu_mbus_resume(void)
+static void mvebu_mbus_resume(void *data)
{
struct mvebu_mbus_state *s = &mbus_state;
int win;
@@ -1069,9 +1069,13 @@ static void mvebu_mbus_resume(void)
}
}
-static struct syscore_ops mvebu_mbus_syscore_ops = {
- .suspend = mvebu_mbus_suspend,
- .resume = mvebu_mbus_resume,
+static const struct syscore_ops mvebu_mbus_syscore_ops = {
+ .suspend = mvebu_mbus_suspend,
+ .resume = mvebu_mbus_resume,
+};
+
+static struct syscore mvebu_mbus_syscore = {
+ .ops = &mvebu_mbus_syscore_ops,
};
static int __init mvebu_mbus_common_init(struct mvebu_mbus_state *mbus,
@@ -1118,7 +1122,7 @@ static int __init mvebu_mbus_common_init(struct mvebu_mbus_state *mbus,
writel(UNIT_SYNC_BARRIER_ALL,
mbus->mbuswins_base + UNIT_SYNC_BARRIER_OFF);
- register_syscore_ops(&mvebu_mbus_syscore_ops);
+ register_syscore(&mvebu_mbus_syscore);
return 0;
}
diff --git a/drivers/char/ipmi/ipmi_ssif.c b/drivers/char/ipmi/ipmi_ssif.c
index 7950f13c1ef7..a19184151323 100644
--- a/drivers/char/ipmi/ipmi_ssif.c
+++ b/drivers/char/ipmi/ipmi_ssif.c
@@ -1292,6 +1292,7 @@ static void shutdown_ssif(void *send_info)
if (ssif_info->thread) {
complete(&ssif_info->wake_thread);
kthread_stop(ssif_info->thread);
+ ssif_info->thread = NULL;
}
}
@@ -1936,6 +1937,25 @@ static int ssif_probe(struct i2c_client *client)
out:
if (rv) {
+ /*
+ * If ipmi_register_smi() starts the interface, it will
+ * call shutdown and that will free the thread and set
+ * it to NULL. Otherwise it must be freed here.
+ */
+ if (ssif_info->thread) {
+ struct task_struct *thread = ssif_info->thread;
+
+ /*
+ * The thread exits on its own once it sees
+ * stopping, so hold a reference across the stop.
+ */
+ get_task_struct(thread);
+ ssif_info->stopping = true;
+ complete(&ssif_info->wake_thread);
+ kthread_stop(thread);
+ put_task_struct(thread);
+ ssif_info->thread = NULL;
+ }
if (addr_info)
addr_info->client = NULL;
diff --git a/drivers/clk/at91/pmc.c b/drivers/clk/at91/pmc.c
index 40bdc1c80ab0..3d5c2085fc9e 100644
--- a/drivers/clk/at91/pmc.c
+++ b/drivers/clk/at91/pmc.c
@@ -115,7 +115,7 @@ struct pmc_data *pmc_data_allocate(unsigned int ncore, unsigned int nsystem,
/* Address in SECURAM that say if we suspend to backup mode. */
static void __iomem *at91_pmc_backup_suspend;
-static int at91_pmc_suspend(void)
+static int at91_pmc_suspend(void *data)
{
unsigned int backup;
@@ -129,7 +129,7 @@ static int at91_pmc_suspend(void)
return clk_save_context();
}
-static void at91_pmc_resume(void)
+static void at91_pmc_resume(void *data)
{
unsigned int backup;
@@ -143,11 +143,15 @@ static void at91_pmc_resume(void)
clk_restore_context();
}
-static struct syscore_ops pmc_syscore_ops = {
+static const struct syscore_ops pmc_syscore_ops = {
.suspend = at91_pmc_suspend,
.resume = at91_pmc_resume,
};
+static struct syscore pmc_syscore = {
+ .ops = &pmc_syscore_ops,
+};
+
static const struct of_device_id pmc_dt_ids[] = {
{ .compatible = "atmel,sama5d2-pmc" },
{ .compatible = "microchip,sama7g5-pmc", },
@@ -184,7 +188,7 @@ static int __init pmc_register_ops(void)
return -ENOMEM;
}
- register_syscore_ops(&pmc_syscore_ops);
+ register_syscore(&pmc_syscore);
return 0;
}
diff --git a/drivers/clk/imx/clk-vf610.c b/drivers/clk/imx/clk-vf610.c
index 9e11f1c7c397..41eb38552a9c 100644
--- a/drivers/clk/imx/clk-vf610.c
+++ b/drivers/clk/imx/clk-vf610.c
@@ -139,7 +139,7 @@ static struct clk * __init vf610_get_fixed_clock(
return clk;
};
-static int vf610_clk_suspend(void)
+static int vf610_clk_suspend(void *data)
{
int i;
@@ -156,7 +156,7 @@ static int vf610_clk_suspend(void)
return 0;
}
-static void vf610_clk_resume(void)
+static void vf610_clk_resume(void *data)
{
int i;
@@ -171,11 +171,15 @@ static void vf610_clk_resume(void)
writel_relaxed(ccgr[i], CCM_CCGRx(i));
}
-static struct syscore_ops vf610_clk_syscore_ops = {
+static const struct syscore_ops vf610_clk_syscore_ops = {
.suspend = vf610_clk_suspend,
.resume = vf610_clk_resume,
};
+static struct syscore vf610_clk_syscore = {
+ .ops = &vf610_clk_syscore_ops,
+};
+
static void __init vf610_clocks_init(struct device_node *ccm_node)
{
struct device_node *np;
@@ -462,7 +466,7 @@ static void __init vf610_clocks_init(struct device_node *ccm_node)
for (i = 0; i < ARRAY_SIZE(clks_init_on); i++)
clk_prepare_enable(clk[clks_init_on[i]]);
- register_syscore_ops(&vf610_clk_syscore_ops);
+ register_syscore(&vf610_clk_syscore);
/* Add the clocks to provider list */
clk_data.clks = clk;
diff --git a/drivers/clk/ingenic/jz4725b-cgu.c b/drivers/clk/ingenic/jz4725b-cgu.c
index 590e9c85cb25..94cee44c854f 100644
--- a/drivers/clk/ingenic/jz4725b-cgu.c
+++ b/drivers/clk/ingenic/jz4725b-cgu.c
@@ -268,6 +268,6 @@ static void __init jz4725b_cgu_init(struct device_node *np)
if (retval)
pr_err("%s: failed to register CGU Clocks\n", __func__);
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
CLK_OF_DECLARE_DRIVER(jz4725b_cgu, "ingenic,jz4725b-cgu", jz4725b_cgu_init);
diff --git a/drivers/clk/ingenic/jz4740-cgu.c b/drivers/clk/ingenic/jz4740-cgu.c
index 3e0a30574ebb..2def3aedc8dd 100644
--- a/drivers/clk/ingenic/jz4740-cgu.c
+++ b/drivers/clk/ingenic/jz4740-cgu.c
@@ -266,6 +266,6 @@ static void __init jz4740_cgu_init(struct device_node *np)
if (retval)
pr_err("%s: failed to register CGU Clocks\n", __func__);
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
CLK_OF_DECLARE_DRIVER(jz4740_cgu, "ingenic,jz4740-cgu", jz4740_cgu_init);
diff --git a/drivers/clk/ingenic/jz4755-cgu.c b/drivers/clk/ingenic/jz4755-cgu.c
index f2c2d848dab7..17cf5dcaece9 100644
--- a/drivers/clk/ingenic/jz4755-cgu.c
+++ b/drivers/clk/ingenic/jz4755-cgu.c
@@ -337,7 +337,7 @@ static void __init jz4755_cgu_init(struct device_node *np)
if (retval)
pr_err("%s: failed to register CGU Clocks\n", __func__);
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
/*
* CGU has some children devices, this is useful for probing children devices
diff --git a/drivers/clk/ingenic/jz4760-cgu.c b/drivers/clk/ingenic/jz4760-cgu.c
index e407f00bd594..372fe4b07992 100644
--- a/drivers/clk/ingenic/jz4760-cgu.c
+++ b/drivers/clk/ingenic/jz4760-cgu.c
@@ -436,7 +436,7 @@ static void __init jz4760_cgu_init(struct device_node *np)
if (retval)
pr_err("%s: failed to register CGU Clocks\n", __func__);
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
/* We only probe via devicetree, no need for a platform driver */
diff --git a/drivers/clk/ingenic/jz4770-cgu.c b/drivers/clk/ingenic/jz4770-cgu.c
index 6ae1740367f9..58f1d3bad677 100644
--- a/drivers/clk/ingenic/jz4770-cgu.c
+++ b/drivers/clk/ingenic/jz4770-cgu.c
@@ -456,7 +456,7 @@ static void __init jz4770_cgu_init(struct device_node *np)
if (retval)
pr_err("%s: failed to register CGU Clocks\n", __func__);
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
/* We only probe via devicetree, no need for a platform driver */
diff --git a/drivers/clk/ingenic/jz4780-cgu.c b/drivers/clk/ingenic/jz4780-cgu.c
index b1dadc0a5e75..a3fb27298bbe 100644
--- a/drivers/clk/ingenic/jz4780-cgu.c
+++ b/drivers/clk/ingenic/jz4780-cgu.c
@@ -803,6 +803,6 @@ static void __init jz4780_cgu_init(struct device_node *np)
return;
}
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
CLK_OF_DECLARE_DRIVER(jz4780_cgu, "ingenic,jz4780-cgu", jz4780_cgu_init);
diff --git a/drivers/clk/ingenic/pm.c b/drivers/clk/ingenic/pm.c
index 341752b640d2..206d5cf2872f 100644
--- a/drivers/clk/ingenic/pm.c
+++ b/drivers/clk/ingenic/pm.c
@@ -15,7 +15,7 @@
static void __iomem * __maybe_unused ingenic_cgu_base;
-static int __maybe_unused ingenic_cgu_pm_suspend(void)
+static int __maybe_unused ingenic_cgu_pm_suspend(void *data)
{
u32 val = readl(ingenic_cgu_base + CGU_REG_LCR);
@@ -24,22 +24,26 @@ static int __maybe_unused ingenic_cgu_pm_suspend(void)
return 0;
}
-static void __maybe_unused ingenic_cgu_pm_resume(void)
+static void __maybe_unused ingenic_cgu_pm_resume(void *data)
{
u32 val = readl(ingenic_cgu_base + CGU_REG_LCR);
writel(val & ~LCR_LOW_POWER_MODE, ingenic_cgu_base + CGU_REG_LCR);
}
-static struct syscore_ops __maybe_unused ingenic_cgu_pm_ops = {
+static const struct syscore_ops __maybe_unused ingenic_cgu_pm_ops = {
.suspend = ingenic_cgu_pm_suspend,
.resume = ingenic_cgu_pm_resume,
};
-void ingenic_cgu_register_syscore_ops(struct ingenic_cgu *cgu)
+static struct syscore __maybe_unused ingenic_cgu_pm = {
+ .ops = &ingenic_cgu_pm_ops,
+};
+
+void ingenic_cgu_register_syscore(struct ingenic_cgu *cgu)
{
if (IS_ENABLED(CONFIG_PM_SLEEP)) {
ingenic_cgu_base = cgu->base;
- register_syscore_ops(&ingenic_cgu_pm_ops);
+ register_syscore(&ingenic_cgu_pm);
}
}
diff --git a/drivers/clk/ingenic/pm.h b/drivers/clk/ingenic/pm.h
index fa7540407b6b..0dcb57dc64cb 100644
--- a/drivers/clk/ingenic/pm.h
+++ b/drivers/clk/ingenic/pm.h
@@ -7,6 +7,6 @@
struct ingenic_cgu;
-void ingenic_cgu_register_syscore_ops(struct ingenic_cgu *cgu);
+void ingenic_cgu_register_syscore(struct ingenic_cgu *cgu);
#endif /* DRIVERS_CLK_INGENIC_PM_H */
diff --git a/drivers/clk/ingenic/tcu.c b/drivers/clk/ingenic/tcu.c
index 7d04ef40b7cf..bc6a51da2072 100644
--- a/drivers/clk/ingenic/tcu.c
+++ b/drivers/clk/ingenic/tcu.c
@@ -455,7 +455,7 @@ static int __init ingenic_tcu_probe(struct device_node *np)
return ret;
}
-static int __maybe_unused tcu_pm_suspend(void)
+static int __maybe_unused tcu_pm_suspend(void *data)
{
struct ingenic_tcu *tcu = ingenic_tcu;
@@ -465,7 +465,7 @@ static int __maybe_unused tcu_pm_suspend(void)
return 0;
}
-static void __maybe_unused tcu_pm_resume(void)
+static void __maybe_unused tcu_pm_resume(void *data)
{
struct ingenic_tcu *tcu = ingenic_tcu;
@@ -473,11 +473,15 @@ static void __maybe_unused tcu_pm_resume(void)
clk_enable(tcu->clk);
}
-static struct syscore_ops __maybe_unused tcu_pm_ops = {
+static const struct syscore_ops __maybe_unused tcu_pm_ops = {
.suspend = tcu_pm_suspend,
.resume = tcu_pm_resume,
};
+static struct syscore __maybe_unused tcu_pm = {
+ .ops = &tcu_pm_ops,
+};
+
static void __init ingenic_tcu_init(struct device_node *np)
{
int ret = ingenic_tcu_probe(np);
@@ -486,7 +490,7 @@ static void __init ingenic_tcu_init(struct device_node *np)
pr_crit("Failed to initialize TCU clocks: %d\n", ret);
if (IS_ENABLED(CONFIG_PM_SLEEP))
- register_syscore_ops(&tcu_pm_ops);
+ register_syscore(&tcu_pm);
}
CLK_OF_DECLARE_DRIVER(jz4740_cgu, "ingenic,jz4740-tcu", ingenic_tcu_init);
diff --git a/drivers/clk/ingenic/x1000-cgu.c b/drivers/clk/ingenic/x1000-cgu.c
index feb03eed4fe8..98f7da93e013 100644
--- a/drivers/clk/ingenic/x1000-cgu.c
+++ b/drivers/clk/ingenic/x1000-cgu.c
@@ -555,7 +555,7 @@ static void __init x1000_cgu_init(struct device_node *np)
return;
}
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
/*
* CGU has some children devices, this is useful for probing children devices
diff --git a/drivers/clk/ingenic/x1830-cgu.c b/drivers/clk/ingenic/x1830-cgu.c
index 0fd46e50a513..acf856e5009e 100644
--- a/drivers/clk/ingenic/x1830-cgu.c
+++ b/drivers/clk/ingenic/x1830-cgu.c
@@ -463,7 +463,7 @@ static void __init x1830_cgu_init(struct device_node *np)
return;
}
- ingenic_cgu_register_syscore_ops(cgu);
+ ingenic_cgu_register_syscore(cgu);
}
/*
* CGU has some children devices, this is useful for probing children devices
diff --git a/drivers/clk/mediatek/clk-pllfh.c b/drivers/clk/mediatek/clk-pllfh.c
index 094ec8a26d66..1e97daeb45e4 100644
--- a/drivers/clk/mediatek/clk-pllfh.c
+++ b/drivers/clk/mediatek/clk-pllfh.c
@@ -194,12 +194,56 @@ static void mtk_clk_unregister_pllfh(struct clk_hw *hw)
kfree(fh);
}
+static void mtk_clk_cleanup_pllfhs(void __iomem *iomem_base,
+ const struct mtk_pll_data *plls, int num_plls,
+ void __iomem *iomem_fhctl_base,
+ struct mtk_pllfh_data *pllfhs, int num_fhs,
+ struct clk_hw_onecell_data *clk_data)
+{
+ void __iomem *base = iomem_base;
+ void __iomem *fhctl_base = iomem_fhctl_base;
+ int i;
+
+ for (i = num_plls - 1; i >= 0; i--) {
+ const struct mtk_pll_data *pll = &plls[i];
+ struct mtk_pllfh_data *pllfh;
+ bool use_fhctl;
+
+ if (IS_ERR_OR_NULL(clk_data->hws[pll->id]))
+ continue;
+
+ pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id);
+ use_fhctl = fhctl_is_supported_and_enabled(pllfh);
+
+ if (!base)
+ base = mtk_clk_pll_get_base(clk_data->hws[pll->id],
+ pll);
+
+ if (use_fhctl) {
+ if (!fhctl_base)
+ fhctl_base = pllfh->state.base;
+ mtk_clk_unregister_pllfh(clk_data->hws[pll->id]);
+ } else {
+ mtk_clk_unregister_pll(clk_data->hws[pll->id]);
+ }
+
+ clk_data->hws[pll->id] = ERR_PTR(-ENOENT);
+ }
+
+ if (fhctl_base)
+ iounmap(fhctl_base);
+
+ if (base)
+ iounmap(base);
+}
+
+
int mtk_clk_register_pllfhs(struct device_node *node,
const struct mtk_pll_data *plls, int num_plls,
struct mtk_pllfh_data *pllfhs, int num_fhs,
struct clk_hw_onecell_data *clk_data)
{
- void __iomem *base;
+ void __iomem *base, *fhctl_base = NULL;
int i;
struct clk_hw *hw;
@@ -235,24 +279,8 @@ int mtk_clk_register_pllfhs(struct device_node *node,
return 0;
err:
- while (--i >= 0) {
- const struct mtk_pll_data *pll = &plls[i];
- struct mtk_pllfh_data *pllfh;
- bool use_fhctl;
-
- pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id);
- use_fhctl = fhctl_is_supported_and_enabled(pllfh);
-
- if (use_fhctl)
- mtk_clk_unregister_pllfh(clk_data->hws[pll->id]);
- else
- mtk_clk_unregister_pll(clk_data->hws[pll->id]);
-
- clk_data->hws[pll->id] = ERR_PTR(-ENOENT);
- }
-
- iounmap(base);
-
+ mtk_clk_cleanup_pllfhs(base, plls, i, fhctl_base, pllfhs, num_fhs,
+ clk_data);
return PTR_ERR(hw);
}
EXPORT_SYMBOL_GPL(mtk_clk_register_pllfhs);
@@ -261,38 +289,10 @@ void mtk_clk_unregister_pllfhs(const struct mtk_pll_data *plls, int num_plls,
struct mtk_pllfh_data *pllfhs, int num_fhs,
struct clk_hw_onecell_data *clk_data)
{
- void __iomem *base = NULL, *fhctl_base = NULL;
- int i;
-
if (!clk_data)
return;
- for (i = num_plls; i > 0; i--) {
- const struct mtk_pll_data *pll = &plls[i - 1];
- struct mtk_pllfh_data *pllfh;
- bool use_fhctl;
-
- if (IS_ERR_OR_NULL(clk_data->hws[pll->id]))
- continue;
-
- pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id);
- use_fhctl = fhctl_is_supported_and_enabled(pllfh);
-
- if (use_fhctl) {
- fhctl_base = pllfh->state.base;
- mtk_clk_unregister_pllfh(clk_data->hws[pll->id]);
- } else {
- base = mtk_clk_pll_get_base(clk_data->hws[pll->id],
- pll);
- mtk_clk_unregister_pll(clk_data->hws[pll->id]);
- }
-
- clk_data->hws[pll->id] = ERR_PTR(-ENOENT);
- }
-
- if (fhctl_base)
- iounmap(fhctl_base);
-
- iounmap(base);
+ mtk_clk_cleanup_pllfhs(NULL, plls, num_plls, NULL, pllfhs,
+ num_fhs, clk_data);
}
EXPORT_SYMBOL_GPL(mtk_clk_unregister_pllfhs);
diff --git a/drivers/clk/mvebu/common.c b/drivers/clk/mvebu/common.c
index 785dbede4835..5adbbd91a6db 100644
--- a/drivers/clk/mvebu/common.c
+++ b/drivers/clk/mvebu/common.c
@@ -215,22 +215,26 @@ static struct clk *clk_gating_get_src(
return ERR_PTR(-ENODEV);
}
-static int mvebu_clk_gating_suspend(void)
+static int mvebu_clk_gating_suspend(void *data)
{
ctrl->saved_reg = readl(ctrl->base);
return 0;
}
-static void mvebu_clk_gating_resume(void)
+static void mvebu_clk_gating_resume(void *data)
{
writel(ctrl->saved_reg, ctrl->base);
}
-static struct syscore_ops clk_gate_syscore_ops = {
+static const struct syscore_ops clk_gate_syscore_ops = {
.suspend = mvebu_clk_gating_suspend,
.resume = mvebu_clk_gating_resume,
};
+static struct syscore clk_gate_syscore = {
+ .ops = &clk_gate_syscore_ops,
+};
+
void __init mvebu_clk_gating_setup(struct device_node *np,
const struct clk_gating_soc_desc *desc)
{
@@ -284,7 +288,7 @@ void __init mvebu_clk_gating_setup(struct device_node *np,
of_clk_add_provider(np, clk_gating_get_src, ctrl);
- register_syscore_ops(&clk_gate_syscore_ops);
+ register_syscore(&clk_gate_syscore);
return;
gates_out:
diff --git a/drivers/clk/rockchip/clk-rk3288.c b/drivers/clk/rockchip/clk-rk3288.c
index 90d329216064..9e64fa5321a2 100644
--- a/drivers/clk/rockchip/clk-rk3288.c
+++ b/drivers/clk/rockchip/clk-rk3288.c
@@ -871,7 +871,7 @@ static const int rk3288_saved_cru_reg_ids[] = {
static u32 rk3288_saved_cru_regs[ARRAY_SIZE(rk3288_saved_cru_reg_ids)];
-static int rk3288_clk_suspend(void)
+static int rk3288_clk_suspend(void *data)
{
int i, reg_id;
@@ -906,7 +906,7 @@ static int rk3288_clk_suspend(void)
return 0;
}
-static void rk3288_clk_resume(void)
+static void rk3288_clk_resume(void *data)
{
int i, reg_id;
@@ -923,11 +923,15 @@ static void rk3288_clk_shutdown(void)
writel_relaxed(0xf3030000, rk3288_cru_base + RK3288_MODE_CON);
}
-static struct syscore_ops rk3288_clk_syscore_ops = {
+static const struct syscore_ops rk3288_clk_syscore_ops = {
.suspend = rk3288_clk_suspend,
.resume = rk3288_clk_resume,
};
+static struct syscore rk3288_clk_syscore = {
+ .ops = &rk3288_clk_syscore_ops,
+};
+
static void __init rk3288_common_init(struct device_node *np,
enum rk3288_variant soc)
{
@@ -976,7 +980,7 @@ static void __init rk3288_common_init(struct device_node *np,
rockchip_register_restart_notifier(ctx, RK3288_GLB_SRST_FST,
rk3288_clk_shutdown);
- register_syscore_ops(&rk3288_clk_syscore_ops);
+ register_syscore(&rk3288_clk_syscore);
rockchip_clk_of_add_provider(np, ctx);
}
diff --git a/drivers/clk/samsung/clk-s5pv210-audss.c b/drivers/clk/samsung/clk-s5pv210-audss.c
index b31c00ea331f..75c7721e4e4e 100644
--- a/drivers/clk/samsung/clk-s5pv210-audss.c
+++ b/drivers/clk/samsung/clk-s5pv210-audss.c
@@ -35,7 +35,7 @@ static unsigned long reg_save[][2] = {
{ASS_CLK_GATE, 0},
};
-static int s5pv210_audss_clk_suspend(void)
+static int s5pv210_audss_clk_suspend(void *data)
{
int i;
@@ -45,7 +45,7 @@ static int s5pv210_audss_clk_suspend(void)
return 0;
}
-static void s5pv210_audss_clk_resume(void)
+static void s5pv210_audss_clk_resume(void *data)
{
int i;
@@ -53,10 +53,14 @@ static void s5pv210_audss_clk_resume(void)
writel(reg_save[i][1], reg_base + reg_save[i][0]);
}
-static struct syscore_ops s5pv210_audss_clk_syscore_ops = {
+static const struct syscore_ops s5pv210_audss_clk_syscore_ops = {
.suspend = s5pv210_audss_clk_suspend,
.resume = s5pv210_audss_clk_resume,
};
+
+static struct syscore s5pv210_audss_clk_syscore = {
+ .ops = &s5pv210_audss_clk_syscore_ops,
+};
#endif /* CONFIG_PM_SLEEP */
/* register s5pv210_audss clocks */
@@ -174,7 +178,7 @@ static int s5pv210_audss_clk_probe(struct platform_device *pdev)
}
#ifdef CONFIG_PM_SLEEP
- register_syscore_ops(&s5pv210_audss_clk_syscore_ops);
+ register_syscore(&s5pv210_audss_clk_syscore);
#endif
return 0;
diff --git a/drivers/clk/samsung/clk.c b/drivers/clk/samsung/clk.c
index e6533513f8ac..47ae261decd6 100644
--- a/drivers/clk/samsung/clk.c
+++ b/drivers/clk/samsung/clk.c
@@ -271,7 +271,7 @@ void __init samsung_clk_of_register_fixed_ext(struct samsung_clk_provider *ctx,
}
#ifdef CONFIG_PM_SLEEP
-static int samsung_clk_suspend(void)
+static int samsung_clk_suspend(void *data)
{
struct samsung_clock_reg_cache *reg_cache;
@@ -284,7 +284,7 @@ static int samsung_clk_suspend(void)
return 0;
}
-static void samsung_clk_resume(void)
+static void samsung_clk_resume(void *data)
{
struct samsung_clock_reg_cache *reg_cache;
@@ -293,11 +293,15 @@ static void samsung_clk_resume(void)
reg_cache->rd_num);
}
-static struct syscore_ops samsung_clk_syscore_ops = {
+static const struct syscore_ops samsung_clk_syscore_ops = {
.suspend = samsung_clk_suspend,
.resume = samsung_clk_resume,
};
+static struct syscore samsung_clk_syscore = {
+ .ops = &samsung_clk_syscore_ops,
+};
+
void samsung_clk_extended_sleep_init(void __iomem *reg_base,
const unsigned long *rdump,
unsigned long nr_rdump,
@@ -316,7 +320,7 @@ void samsung_clk_extended_sleep_init(void __iomem *reg_base,
panic("could not allocate register dump storage.\n");
if (list_empty(&clock_reg_cache_list))
- register_syscore_ops(&samsung_clk_syscore_ops);
+ register_syscore(&samsung_clk_syscore);
reg_cache->reg_base = reg_base;
reg_cache->rd_num = nr_rdump;
diff --git a/drivers/clk/sunxi-ng/ccu-sun50i-h616.c b/drivers/clk/sunxi-ng/ccu-sun50i-h616.c
index 84e406ddf9d1..d4e3838ed90b 100644
--- a/drivers/clk/sunxi-ng/ccu-sun50i-h616.c
+++ b/drivers/clk/sunxi-ng/ccu-sun50i-h616.c
@@ -318,10 +318,16 @@ static SUNXI_CCU_M_WITH_MUX_GATE(gpu0_clk, "gpu0", gpu0_parents, 0x670,
24, 1, /* mux */
BIT(31), /* gate */
CLK_SET_RATE_PARENT);
+
+/*
+ * This clk is needed as a temporary fall back during GPU PLL freq changes.
+ * Set CLK_IS_CRITICAL flag to prevent from being disabled.
+ */
+#define SUN50I_H616_GPU_CLK1_REG 0x674
static SUNXI_CCU_M_WITH_GATE(gpu1_clk, "gpu1", "pll-periph0-2x", 0x674,
0, 2, /* M */
BIT(31),/* gate */
- 0);
+ CLK_IS_CRITICAL);
static SUNXI_CCU_GATE(bus_gpu_clk, "bus-gpu", "psi-ahb1-ahb2",
0x67c, BIT(0), 0);
@@ -1095,11 +1101,37 @@ static const u32 usb2_clk_regs[] = {
SUN50I_H616_USB3_CLK_REG,
};
+static struct ccu_mux_nb sun50i_h616_cpu_nb = {
+ .common = &cpux_clk.common,
+ .cm = &cpux_clk.mux,
+ .delay_us = 1, /* manual doesn't really say */
+ .bypass_index = 4, /* PLL_PERI0@600MHz, as recommended by manual */
+};
+
+static struct ccu_pll_nb sun50i_h616_pll_cpu_nb = {
+ .common = &pll_cpux_clk.common,
+ .enable = BIT(29), /* LOCK_ENABLE */
+ .lock = BIT(28),
+};
+
+static struct ccu_mux_nb sun50i_h616_gpu_nb = {
+ .common = &gpu0_clk.common,
+ .cm = &gpu0_clk.mux,
+ .delay_us = 1, /* manual doesn't really say */
+ .bypass_index = 1, /* GPU_CLK1@400MHz */
+};
+
+static struct ccu_pll_nb sun50i_h616_pll_gpu_nb = {
+ .common = &pll_gpu_clk.common,
+ .enable = BIT(29), /* LOCK_ENABLE */
+ .lock = BIT(28),
+};
+
static int sun50i_h616_ccu_probe(struct platform_device *pdev)
{
void __iomem *reg;
u32 val;
- int i;
+ int ret, i;
reg = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(reg))
@@ -1143,6 +1175,14 @@ static int sun50i_h616_ccu_probe(struct platform_device *pdev)
val &= ~(GENMASK(21, 16) | BIT(0));
writel(val | (11 << 16) | BIT(0), reg + SUN50I_H616_PLL_AUDIO_REG);
+ /*
+ * Set the input-divider for the gpu1 clock to 3, to reach a safe 400 MHz.
+ */
+ val = readl(reg + SUN50I_H616_GPU_CLK1_REG);
+ val &= ~GENMASK(1, 0);
+ val |= 2;
+ writel(val, reg + SUN50I_H616_GPU_CLK1_REG);
+
/*
* First clock parent (osc32K) is unusable for CEC. But since there
* is no good way to force parent switch (both run with same frequency),
@@ -1152,7 +1192,25 @@ static int sun50i_h616_ccu_probe(struct platform_device *pdev)
val |= BIT(24);
writel(val, reg + SUN50I_H616_HDMI_CEC_CLK_REG);
- return devm_sunxi_ccu_probe(&pdev->dev, reg, &sun50i_h616_ccu_desc);
+ ret = devm_sunxi_ccu_probe(&pdev->dev, reg, &sun50i_h616_ccu_desc);
+ if (ret)
+ return ret;
+
+ /* Reparent CPU during CPU PLL rate changes */
+ ccu_mux_notifier_register(pll_cpux_clk.common.hw.clk,
+ &sun50i_h616_cpu_nb);
+
+ /* Re-lock the CPU PLL after any rate changes */
+ ccu_pll_notifier_register(&sun50i_h616_pll_cpu_nb);
+
+ /* Reparent GPU during GPU PLL rate changes */
+ ccu_mux_notifier_register(pll_gpu_clk.common.hw.clk,
+ &sun50i_h616_gpu_nb);
+
+ /* Re-lock the GPU PLL after any rate changes */
+ ccu_pll_notifier_register(&sun50i_h616_pll_gpu_nb);
+
+ return 0;
}
static const struct of_device_id sun50i_h616_ccu_ids[] = {
diff --git a/drivers/clk/tegra/clk-tegra210.c b/drivers/clk/tegra/clk-tegra210.c
index a3488aaac3f7..8a411fc580b4 100644
--- a/drivers/clk/tegra/clk-tegra210.c
+++ b/drivers/clk/tegra/clk-tegra210.c
@@ -3444,7 +3444,7 @@ static void tegra210_disable_cpu_clock(u32 cpu)
static u32 spare_reg_ctx, misc_clk_enb_ctx, clk_msk_arm_ctx;
static u32 cpu_softrst_ctx[3];
-static int tegra210_clk_suspend(void)
+static int tegra210_clk_suspend(void *data)
{
unsigned int i;
@@ -3465,7 +3465,7 @@ static int tegra210_clk_suspend(void)
return 0;
}
-static void tegra210_clk_resume(void)
+static void tegra210_clk_resume(void *data)
{
unsigned int i;
@@ -3523,13 +3523,17 @@ static void tegra210_cpu_clock_resume(void)
}
#endif
-static struct syscore_ops tegra_clk_syscore_ops = {
+static const struct syscore_ops tegra_clk_syscore_ops = {
#ifdef CONFIG_PM_SLEEP
.suspend = tegra210_clk_suspend,
.resume = tegra210_clk_resume,
#endif
};
+static struct syscore tegra_clk_syscore = {
+ .ops = &tegra_clk_syscore_ops,
+};
+
static struct tegra_cpu_car_ops tegra210_cpu_car_ops = {
.wait_for_reset = tegra210_wait_cpu_in_reset,
.disable_clock = tegra210_disable_cpu_clock,
@@ -3813,6 +3817,6 @@ static void __init tegra210_clock_init(struct device_node *np)
tegra_cpu_car_ops = &tegra210_cpu_car_ops;
- register_syscore_ops(&tegra_clk_syscore_ops);
+ register_syscore(&tegra_clk_syscore);
}
CLK_OF_DECLARE(tegra210, "nvidia,tegra210-car", tegra210_clock_init);
diff --git a/drivers/clk/thead/clk-th1520-ap.c b/drivers/clk/thead/clk-th1520-ap.c
index c8ebacc6934a..510670b9c983 100644
--- a/drivers/clk/thead/clk-th1520-ap.c
+++ b/drivers/clk/thead/clk-th1520-ap.c
@@ -493,7 +493,7 @@ static struct ccu_div axi4_cpusys2_aclk = {
.hw.init = CLK_HW_INIT_PARENTS_HW("axi4-cpusys2-aclk",
gmac_pll_clk_parent,
&ccu_div_ops,
- 0),
+ CLK_IS_CRITICAL),
},
};
@@ -515,7 +515,7 @@ static struct ccu_div axi_aclk = {
.hw.init = CLK_HW_INIT_PARENTS_DATA("axi-aclk",
axi_parents,
&ccu_div_ops,
- 0),
+ CLK_IS_CRITICAL),
},
};
@@ -664,7 +664,7 @@ static struct ccu_div apb_pclk = {
.hw.init = CLK_HW_INIT_PARENTS_DATA("apb-pclk",
apb_parents,
&ccu_div_ops,
- CLK_IGNORE_UNUSED),
+ CLK_IS_CRITICAL),
},
};
@@ -695,7 +695,7 @@ static struct ccu_div vi_clk = {
.hw.init = CLK_HW_INIT_PARENTS_HW("vi",
video_pll_clk_parent,
&ccu_div_ops,
- 0),
+ CLK_IS_CRITICAL),
},
};
@@ -720,7 +720,7 @@ static struct ccu_div vo_axi_clk = {
.hw.init = CLK_HW_INIT_PARENTS_HW("vo-axi",
video_pll_clk_parent,
&ccu_div_ops,
- 0),
+ CLK_IS_CRITICAL),
},
};
@@ -745,7 +745,7 @@ static struct ccu_div vp_axi_clk = {
.hw.init = CLK_HW_INIT_PARENTS_HW("vp-axi",
video_pll_clk_parent,
&ccu_div_ops,
- CLK_IGNORE_UNUSED),
+ CLK_IS_CRITICAL),
},
};
@@ -796,23 +796,23 @@ static const struct clk_parent_data emmc_sdio_ref_clk_pd[] = {
static CCU_GATE(CLK_BROM, brom_clk, "brom", ahb2_cpusys_hclk_pd, 0x100, BIT(4), 0);
static CCU_GATE(CLK_BMU, bmu_clk, "bmu", axi4_cpusys2_aclk_pd, 0x100, BIT(5), 0);
static CCU_GATE(CLK_AON2CPU_A2X, aon2cpu_a2x_clk, "aon2cpu-a2x", axi4_cpusys2_aclk_pd,
- 0x134, BIT(8), 0);
+ 0x134, BIT(8), CLK_IS_CRITICAL);
static CCU_GATE(CLK_X2X_CPUSYS, x2x_cpusys_clk, "x2x-cpusys", axi4_cpusys2_aclk_pd,
- 0x134, BIT(7), 0);
+ 0x134, BIT(7), CLK_IS_CRITICAL);
static CCU_GATE(CLK_CPU2AON_X2H, cpu2aon_x2h_clk, "cpu2aon-x2h", axi_aclk_pd,
- 0x138, BIT(8), CLK_IGNORE_UNUSED);
+ 0x138, BIT(8), CLK_IS_CRITICAL);
static CCU_GATE(CLK_CPU2PERI_X2H, cpu2peri_x2h_clk, "cpu2peri-x2h", axi4_cpusys2_aclk_pd,
- 0x140, BIT(9), CLK_IGNORE_UNUSED);
+ 0x140, BIT(9), CLK_IS_CRITICAL);
static CCU_GATE(CLK_PERISYS_APB1_HCLK, perisys_apb1_hclk, "perisys-apb1-hclk", perisys_ahb_hclk_pd,
- 0x150, BIT(9), CLK_IGNORE_UNUSED);
+ 0x150, BIT(9), CLK_IS_CRITICAL);
static CCU_GATE(CLK_PERISYS_APB2_HCLK, perisys_apb2_hclk, "perisys-apb2-hclk", perisys_ahb_hclk_pd,
- 0x150, BIT(10), CLK_IGNORE_UNUSED);
+ 0x150, BIT(10), CLK_IS_CRITICAL);
static CCU_GATE(CLK_PERISYS_APB3_HCLK, perisys_apb3_hclk, "perisys-apb3-hclk", perisys_ahb_hclk_pd,
- 0x150, BIT(11), CLK_IGNORE_UNUSED);
+ 0x150, BIT(11), CLK_IS_CRITICAL);
static CCU_GATE(CLK_PERISYS_APB4_HCLK, perisys_apb4_hclk, "perisys-apb4-hclk", perisys_ahb_hclk_pd,
0x150, BIT(12), 0);
-static CCU_GATE(CLK_NPU_AXI, npu_axi_clk, "npu-axi", axi_aclk_pd, 0x1c8, BIT(5), 0);
-static CCU_GATE(CLK_CPU2VP, cpu2vp_clk, "cpu2vp", axi_aclk_pd, 0x1e0, BIT(13), 0);
+static CCU_GATE(CLK_NPU_AXI, npu_axi_clk, "npu-axi", axi_aclk_pd, 0x1c8, BIT(5), CLK_IS_CRITICAL);
+static CCU_GATE(CLK_CPU2VP, cpu2vp_clk, "cpu2vp", axi_aclk_pd, 0x1e0, BIT(13), CLK_IS_CRITICAL);
static CCU_GATE(CLK_EMMC_SDIO, emmc_sdio_clk, "emmc-sdio", emmc_sdio_ref_clk_pd, 0x204, BIT(30), 0);
static CCU_GATE(CLK_GMAC1, gmac1_clk, "gmac1", gmac_pll_clk_pd, 0x204, BIT(26), 0);
static CCU_GATE(CLK_PADCTRL1, padctrl1_clk, "padctrl1", perisys_apb_pclk_pd, 0x204, BIT(24), 0);
diff --git a/drivers/clocksource/samsung_pwm_timer.c b/drivers/clocksource/samsung_pwm_timer.c
index b9561e3f196c..0544124cf5ce 100644
--- a/drivers/clocksource/samsung_pwm_timer.c
+++ b/drivers/clocksource/samsung_pwm_timer.c
@@ -56,7 +56,7 @@
#define TCON_AUTORELOAD(chan) \
((chan < 5) ? _TCON_AUTORELOAD(chan) : _TCON_AUTORELOAD4(chan))
-DEFINE_SPINLOCK(samsung_pwm_lock);
+DEFINE_RAW_SPINLOCK(samsung_pwm_lock);
EXPORT_SYMBOL(samsung_pwm_lock);
struct samsung_pwm_clocksource {
@@ -87,14 +87,14 @@ static void samsung_timer_set_prescale(unsigned int channel, u16 prescale)
if (channel >= 2)
shift = TCFG0_PRESCALER1_SHIFT;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(pwm.base + REG_TCFG0);
reg &= ~(TCFG0_PRESCALER_MASK << shift);
reg |= (prescale - 1) << shift;
writel(reg, pwm.base + REG_TCFG0);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_timer_set_divisor(unsigned int channel, u8 divisor)
@@ -106,14 +106,14 @@ static void samsung_timer_set_divisor(unsigned int channel, u8 divisor)
bits = (fls(divisor) - 1) - pwm.variant.div_base;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(pwm.base + REG_TCFG1);
reg &= ~(TCFG1_MUX_MASK << shift);
reg |= bits << shift;
writel(reg, pwm.base + REG_TCFG1);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_stop(unsigned int channel)
@@ -124,13 +124,13 @@ static void samsung_time_stop(unsigned int channel)
if (channel > 0)
++channel;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
tcon &= ~TCON_START(channel);
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
@@ -142,7 +142,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
if (tcon_chan > 0)
++tcon_chan;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
@@ -153,7 +153,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
writel_relaxed(tcnt, pwm.base + REG_TCMPB(channel));
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_start(unsigned int channel, bool periodic)
@@ -164,7 +164,7 @@ static void samsung_time_start(unsigned int channel, bool periodic)
if (channel > 0)
++channel;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
@@ -178,7 +178,7 @@ static void samsung_time_start(unsigned int channel, bool periodic)
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int samsung_set_next_event(unsigned long cycles,
diff --git a/drivers/clocksource/timer-armada-370-xp.c b/drivers/clocksource/timer-armada-370-xp.c
index 49735ced52bb..5db0cbecbf6d 100644
--- a/drivers/clocksource/timer-armada-370-xp.c
+++ b/drivers/clocksource/timer-armada-370-xp.c
@@ -208,14 +208,14 @@ static int armada_370_xp_timer_dying_cpu(unsigned int cpu)
static u32 timer0_ctrl_reg, timer0_local_ctrl_reg;
-static int armada_370_xp_timer_suspend(void)
+static int armada_370_xp_timer_suspend(void *data)
{
timer0_ctrl_reg = readl(timer_base + TIMER_CTRL_OFF);
timer0_local_ctrl_reg = readl(local_base + TIMER_CTRL_OFF);
return 0;
}
-static void armada_370_xp_timer_resume(void)
+static void armada_370_xp_timer_resume(void *data)
{
writel(0xffffffff, timer_base + TIMER0_VAL_OFF);
writel(0xffffffff, timer_base + TIMER0_RELOAD_OFF);
@@ -223,11 +223,15 @@ static void armada_370_xp_timer_resume(void)
writel(timer0_local_ctrl_reg, local_base + TIMER_CTRL_OFF);
}
-static struct syscore_ops armada_370_xp_timer_syscore_ops = {
+static const struct syscore_ops armada_370_xp_timer_syscore_ops = {
.suspend = armada_370_xp_timer_suspend,
.resume = armada_370_xp_timer_resume,
};
+static struct syscore armada_370_xp_timer_syscore = {
+ .ops = &armada_370_xp_timer_syscore_ops,
+};
+
static unsigned long armada_370_delay_timer_read(void)
{
return ~readl(timer_base + TIMER0_VAL_OFF);
@@ -325,7 +329,7 @@ static int __init armada_370_xp_timer_common_init(struct device_node *np)
return res;
}
- register_syscore_ops(&armada_370_xp_timer_syscore_ops);
+ register_syscore(&armada_370_xp_timer_syscore);
return 0;
}
diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c
index f3c30037dc8e..4ad8d614acf6 100644
--- a/drivers/cpuidle/cpuidle-psci.c
+++ b/drivers/cpuidle/cpuidle-psci.c
@@ -158,26 +158,30 @@ static void psci_idle_syscore_switch(bool suspend)
}
}
-static int psci_idle_syscore_suspend(void)
+static int psci_idle_syscore_suspend(void *data)
{
psci_idle_syscore_switch(true);
return 0;
}
-static void psci_idle_syscore_resume(void)
+static void psci_idle_syscore_resume(void *data)
{
psci_idle_syscore_switch(false);
}
-static struct syscore_ops psci_idle_syscore_ops = {
+static const struct syscore_ops psci_idle_syscore_ops = {
.suspend = psci_idle_syscore_suspend,
.resume = psci_idle_syscore_resume,
};
+static struct syscore psci_idle_syscore = {
+ .ops = &psci_idle_syscore_ops,
+};
+
static void psci_idle_init_syscore(void)
{
if (psci_cpuidle_use_syscore)
- register_syscore_ops(&psci_idle_syscore_ops);
+ register_syscore(&psci_idle_syscore);
}
static void psci_idle_init_cpuhp(void)
diff --git a/drivers/crypto/hisilicon/hpre/hpre_main.c b/drivers/crypto/hisilicon/hpre/hpre_main.c
index a11fe5e27677..6eeaa00e07b2 100644
--- a/drivers/crypto/hisilicon/hpre/hpre_main.c
+++ b/drivers/crypto/hisilicon/hpre/hpre_main.c
@@ -593,6 +593,8 @@ static void hpre_close_sva_prefetch(struct hisi_qm *qm)
static void hpre_enable_clock_gate(struct hisi_qm *qm)
{
+ unsigned long offset;
+ u8 clusters_num, i;
u32 val;
if (qm->ver < QM_HW_V3)
@@ -606,17 +608,23 @@ static void hpre_enable_clock_gate(struct hisi_qm *qm)
val |= HPRE_PEH_CFG_AUTO_GATE_EN;
writel(val, qm->io_base + HPRE_PEH_CFG_AUTO_GATE);
- val = readl(qm->io_base + HPRE_CLUSTER_DYN_CTL);
- val |= HPRE_CLUSTER_DYN_CTL_EN;
- writel(val, qm->io_base + HPRE_CLUSTER_DYN_CTL);
-
- val = readl_relaxed(qm->io_base + HPRE_CORE_SHB_CFG);
- val |= HPRE_CORE_GATE_EN;
- writel(val, qm->io_base + HPRE_CORE_SHB_CFG);
+ clusters_num = qm->cap_tables.dev_cap_table[HPRE_CLUSTER_NUM_CAP_IDX].cap_val;
+ for (i = 0; i < clusters_num; i++) {
+ offset = (unsigned long)i * HPRE_CLSTR_ADDR_INTRVL;
+ val = readl(qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
+ val |= HPRE_CLUSTER_DYN_CTL_EN;
+ writel(val, qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
+
+ val = readl(qm->io_base + offset + HPRE_CORE_SHB_CFG);
+ val |= HPRE_CORE_GATE_EN;
+ writel(val, qm->io_base + offset + HPRE_CORE_SHB_CFG);
+ }
}
static void hpre_disable_clock_gate(struct hisi_qm *qm)
{
+ unsigned long offset;
+ u8 clusters_num, i;
u32 val;
if (qm->ver < QM_HW_V3)
@@ -630,13 +638,17 @@ static void hpre_disable_clock_gate(struct hisi_qm *qm)
val &= ~HPRE_PEH_CFG_AUTO_GATE_EN;
writel(val, qm->io_base + HPRE_PEH_CFG_AUTO_GATE);
- val = readl(qm->io_base + HPRE_CLUSTER_DYN_CTL);
- val &= ~HPRE_CLUSTER_DYN_CTL_EN;
- writel(val, qm->io_base + HPRE_CLUSTER_DYN_CTL);
-
- val = readl_relaxed(qm->io_base + HPRE_CORE_SHB_CFG);
- val &= ~HPRE_CORE_GATE_EN;
- writel(val, qm->io_base + HPRE_CORE_SHB_CFG);
+ clusters_num = qm->cap_tables.dev_cap_table[HPRE_CLUSTER_NUM_CAP_IDX].cap_val;
+ for (i = 0; i < clusters_num; i++) {
+ offset = (unsigned long)i * HPRE_CLSTR_ADDR_INTRVL;
+ val = readl(qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
+ val &= ~HPRE_CLUSTER_DYN_CTL_EN;
+ writel(val, qm->io_base + offset + HPRE_CLUSTER_DYN_CTL);
+
+ val = readl(qm->io_base + offset + HPRE_CORE_SHB_CFG);
+ val &= ~HPRE_CORE_GATE_EN;
+ writel(val, qm->io_base + offset + HPRE_CORE_SHB_CFG);
+ }
}
static int hpre_set_user_domain_and_cache(struct hisi_qm *qm)
diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index 38c9d633db5f..32882cc9b18e 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -4603,6 +4603,15 @@ void hisi_qm_reset_done(struct pci_dev *pdev)
}
EXPORT_SYMBOL_GPL(hisi_qm_reset_done);
+static irqreturn_t qm_rsvd_irq(int irq, void *data)
+{
+ struct hisi_qm *qm = data;
+
+ dev_info(&qm->pdev->dev, "Reserved interrupt, ignore!\n");
+
+ return IRQ_HANDLED;
+}
+
static irqreturn_t qm_abnormal_irq(int irq, void *data)
{
struct hisi_qm *qm = data;
@@ -4887,7 +4896,7 @@ static void qm_unregister_abnormal_irq(struct hisi_qm *qm)
struct pci_dev *pdev = qm->pdev;
u32 irq_vector, val;
- if (qm->fun_type == QM_HW_VF)
+ if (qm->fun_type == QM_HW_VF && qm->ver < QM_HW_V3)
return;
val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val;
@@ -4904,17 +4913,28 @@ static int qm_register_abnormal_irq(struct hisi_qm *qm)
u32 irq_vector, val;
int ret;
- if (qm->fun_type == QM_HW_VF)
- return 0;
-
val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val;
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_ABN_IRQ_TYPE_MASK))
return 0;
-
irq_vector = val & QM_IRQ_VECTOR_MASK;
+
+ /* For VF, this is a reserved interrupt in V3 version. */
+ if (qm->fun_type == QM_HW_VF) {
+ if (qm->ver < QM_HW_V3)
+ return 0;
+
+ ret = request_irq(pci_irq_vector(pdev, irq_vector), qm_rsvd_irq,
+ IRQF_NO_AUTOEN, qm->dev_name, qm);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to request reserved irq, ret = %d!\n", ret);
+ return ret;
+ }
+ return 0;
+ }
+
ret = request_irq(pci_irq_vector(pdev, irq_vector), qm_abnormal_irq, 0, qm->dev_name, qm);
if (ret)
- dev_err(&qm->pdev->dev, "failed to request abnormal irq, ret = %d", ret);
+ dev_err(&qm->pdev->dev, "failed to request abnormal irq, ret = %d!\n", ret);
return ret;
}
@@ -5253,6 +5273,12 @@ static int hisi_qm_pci_init(struct hisi_qm *qm)
pci_set_master(pdev);
num_vec = qm_get_irq_num(qm);
+ if (!num_vec) {
+ dev_err(dev, "Device irq num is zero!\n");
+ ret = -EINVAL;
+ goto err_get_pci_res;
+ }
+ num_vec = roundup_pow_of_two(num_vec);
ret = pci_alloc_irq_vectors(pdev, num_vec, num_vec, PCI_IRQ_MSI);
if (ret < 0) {
dev_err(dev, "Failed to enable MSI vectors!\n");
diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 1c11bd5f0df0..d95206ed6ab8 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -32,6 +32,7 @@ int cxl_get_poison_by_endpoint(struct cxl_port *port);
struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa);
u64 cxl_dpa_to_hpa(struct cxl_region *cxlr, const struct cxl_memdev *cxlmd,
u64 dpa);
+void kill_regions(struct cxl_root_decoder *cxlrd);
#else
static inline u64 cxl_dpa_to_hpa(struct cxl_region *cxlr,
@@ -58,6 +59,7 @@ static inline int cxl_region_init(void)
static inline void cxl_region_exit(void)
{
}
+static inline void kill_regions(struct cxl_root_decoder *cxlrd) { };
#define CXL_REGION_ATTR(x) NULL
#define CXL_REGION_TYPE(x) NULL
#define SET_CXL_REGION_ATTR(x)
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index b094aac78ca1..a1de6b2d4c29 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -310,6 +310,7 @@ static bool cxl_mem_raw_command_allowed(u16 opcode)
* cxl_payload_from_user_allowed() - Check contents of in_payload.
* @opcode: The mailbox command opcode.
* @payload_in: Pointer to the input payload passed in from user space.
+ * @in_size: Size of @payload_in in bytes.
*
* Return:
* * true - payload_in passes check for @opcode.
@@ -324,12 +325,15 @@ static bool cxl_mem_raw_command_allowed(u16 opcode)
*
* The specific checks are determined by the opcode.
*/
-static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in)
+static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in,
+ size_t in_size)
{
switch (opcode) {
case CXL_MBOX_OP_SET_PARTITION_INFO: {
struct cxl_mbox_set_partition_info *pi = payload_in;
+ if (in_size < sizeof(*pi))
+ return false;
if (pi->flags & CXL_SET_PARTITION_IMMEDIATE_FLAG)
return false;
break;
@@ -337,6 +341,8 @@ static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in)
case CXL_MBOX_OP_CLEAR_LOG: {
const uuid_t *uuid = (uuid_t *)payload_in;
+ if (in_size < sizeof(uuid_t))
+ return false;
/*
* Restrict the ‘Clear log’ action to only apply to
* Vendor debug logs.
@@ -364,7 +370,8 @@ static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox_cmd,
if (IS_ERR(mbox_cmd->payload_in))
return PTR_ERR(mbox_cmd->payload_in);
- if (!cxl_payload_from_user_allowed(opcode, mbox_cmd->payload_in)) {
+ if (!cxl_payload_from_user_allowed(opcode, mbox_cmd->payload_in,
+ in_size)) {
dev_dbg(cxl_mbox->host, "%s: input payload not allowed\n",
cxl_mem_opcode_to_name(opcode));
kvfree(mbox_cmd->payload_in);
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 6eb2e6936141..35aceb7b3804 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -445,6 +445,8 @@ static void cxl_root_decoder_release(struct device *dev)
if (atomic_read(&cxlrd->region_id) >= 0)
memregion_free(atomic_read(&cxlrd->region_id));
+ mutex_destroy(&cxlrd->regions_lock);
+ xa_destroy(&cxlrd->regions);
__cxl_decoder_release(&cxlrd->cxlsd.cxld);
kfree(cxlrd);
}
@@ -1825,7 +1827,8 @@ struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port,
return ERR_PTR(rc);
}
- mutex_init(&cxlrd->range_lock);
+ mutex_init(&cxlrd->regions_lock);
+ xa_init(&cxlrd->regions);
cxld = &cxlsd->cxld;
cxld->dev.type = &cxl_decoder_root_type;
@@ -2009,6 +2012,8 @@ static void cxld_unregister(void *dev)
cxled = to_cxl_endpoint_decoder(dev);
cxl_decoder_kill_region(cxled);
}
+ if (is_root_decoder(dev))
+ kill_regions(to_cxl_root_decoder(dev));
device_unregister(dev);
}
diff --git a/drivers/cxl/core/ras.c b/drivers/cxl/core/ras.c
index 6256b145a517..b0d9e65bae2f 100644
--- a/drivers/cxl/core/ras.c
+++ b/drivers/cxl/core/ras.c
@@ -7,6 +7,10 @@
#include "core.h"
#include "trace.h"
+/* Check that UCE header definition is maintained to keep ABI intact */
+static_assert(CXL_HEADERLOG_TRACE_SIZE_U32 == 128,
+ "rasdaemon ABI requires exactly 128 u32s");
+
void cxl_handle_cor_ras(struct cxl_dev_state *cxlds, void __iomem *ras_base)
{
void __iomem *addr;
@@ -28,12 +32,12 @@ static void header_log_copy(void __iomem *ras_base, u32 *log)
{
void __iomem *addr;
u32 *log_addr;
- int i, log_u32_size = CXL_HEADERLOG_SIZE / sizeof(u32);
+ int i;
addr = ras_base + CXL_RAS_HEADER_LOG_OFFSET;
log_addr = log;
- for (i = 0; i < log_u32_size; i++) {
+ for (i = 0; i < CXL_HEADERLOG_SIZE_U32; i++) {
*log_addr = readl(addr);
log_addr++;
addr += sizeof(u32);
@@ -46,7 +50,7 @@ static void header_log_copy(void __iomem *ras_base, u32 *log)
*/
bool cxl_handle_ras(struct cxl_dev_state *cxlds, void __iomem *ras_base)
{
- u32 hl[CXL_HEADERLOG_SIZE_U32];
+ u32 hl[CXL_HEADERLOG_TRACE_SIZE_U32] = {};
void __iomem *addr;
u32 status;
u32 fe;
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 009c0a262ee8..15253e5c28d6 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -2327,12 +2327,13 @@ static struct cxl_region *to_cxl_region(struct device *dev)
return container_of(dev, struct cxl_region, dev);
}
-static void unregister_region(void *_cxlr)
+static void unregister_region(struct cxl_region *cxlr)
{
- struct cxl_region *cxlr = _cxlr;
+ struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent);
struct cxl_region_params *p = &cxlr->params;
int i;
+ xa_erase(&cxlrd->regions, cxlr->id);
device_del(&cxlr->dev);
/*
@@ -2460,6 +2461,19 @@ static int cxl_region_calculate_adistance(struct notifier_block *nb,
return NOTIFY_STOP;
}
+/* unwind all remaining regions */
+void kill_regions(struct cxl_root_decoder *cxlrd)
+{
+ unsigned long index;
+ struct cxl_region *cxlr;
+
+ guard(mutex)(&cxlrd->regions_lock);
+ /* no more region creation */
+ cxlrd->dead = true;
+ xa_for_each(&cxlrd->regions, index, cxlr)
+ unregister_region(cxlr);
+}
+
/**
* devm_cxl_add_region - Adds a region to a decoder
* @cxlrd: root decoder
@@ -2498,14 +2512,15 @@ static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd,
if (rc)
goto err;
- rc = devm_add_action_or_reset(port->uport_dev, unregister_region, cxlr);
- if (rc)
+ rc = xa_insert(&cxlrd->regions, cxlr->id, cxlr, GFP_KERNEL);
+ if (rc) {
+ unregister_region(cxlr);
return ERR_PTR(rc);
+ }
dev_dbg(port->uport_dev, "%s: created %s\n",
dev_name(&cxlrd->cxlsd.cxld.dev), dev_name(dev));
return cxlr;
-
err:
put_device(dev);
return ERR_PTR(rc);
@@ -2533,6 +2548,9 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
{
int rc;
+ if (cxlrd->dead)
+ return ERR_PTR(-ENXIO);
+
switch (mode) {
case CXL_DECODER_RAM:
case CXL_DECODER_PMEM:
@@ -2566,7 +2584,12 @@ static ssize_t create_pmem_region_store(struct device *dev,
if (rc != 1)
return -EINVAL;
+ rc = mutex_lock_interruptible(&cxlrd->regions_lock);
+ if (rc)
+ return rc;
+
cxlr = __create_region(cxlrd, CXL_DECODER_PMEM, id);
+ mutex_unlock(&cxlrd->regions_lock);
if (IS_ERR(cxlr))
return PTR_ERR(cxlr);
@@ -2586,7 +2609,12 @@ static ssize_t create_ram_region_store(struct device *dev,
if (rc != 1)
return -EINVAL;
+ rc = mutex_lock_interruptible(&cxlrd->regions_lock);
+ if (rc)
+ return rc;
+
cxlr = __create_region(cxlrd, CXL_DECODER_RAM, id);
+ mutex_unlock(&cxlrd->regions_lock);
if (IS_ERR(cxlr))
return PTR_ERR(cxlr);
@@ -2614,33 +2642,32 @@ static ssize_t region_show(struct device *dev, struct device_attribute *attr,
}
DEVICE_ATTR_RO(region);
-static struct cxl_region *
-cxl_find_region_by_name(struct cxl_root_decoder *cxlrd, const char *name)
-{
- struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
- struct device *region_dev;
-
- region_dev = device_find_child_by_name(&cxld->dev, name);
- if (!region_dev)
- return ERR_PTR(-ENODEV);
-
- return to_cxl_region(region_dev);
-}
-
static ssize_t delete_region_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
- struct cxl_port *port = to_cxl_port(dev->parent);
struct cxl_region *cxlr;
+ int rc, id;
- cxlr = cxl_find_region_by_name(cxlrd, buf);
- if (IS_ERR(cxlr))
- return PTR_ERR(cxlr);
+ rc = mutex_lock_interruptible(&cxlrd->regions_lock);
+ if (rc)
+ return rc;
- devm_release_action(port->uport_dev, unregister_region, cxlr);
- put_device(&cxlr->dev);
+ rc = sscanf(buf, "region%d\n", &id);
+ if (rc != 1) {
+ mutex_unlock(&cxlrd->regions_lock);
+ return -EINVAL;
+ }
+
+ cxlr = xa_load(&cxlrd->regions, id);
+ if (!cxlr || !sysfs_streq(buf, dev_name(&cxlr->dev))) {
+ mutex_unlock(&cxlrd->regions_lock);
+ return -ENODEV;
+ }
+
+ unregister_region(cxlr);
+ mutex_unlock(&cxlrd->regions_lock);
return len;
}
@@ -3238,7 +3265,6 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
struct cxl_endpoint_decoder *cxled)
{
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
- struct cxl_port *port = cxlrd_to_port(cxlrd);
struct range *hpa = &cxled->cxld.hpa_range;
struct cxl_region_params *p;
struct cxl_region *cxlr;
@@ -3313,7 +3339,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
err:
up_write(&cxl_region_rwsem);
- devm_release_action(port->uport_dev, unregister_region, cxlr);
+ unregister_region(cxlr);
return ERR_PTR(rc);
}
@@ -3345,7 +3371,7 @@ int cxl_add_to_region(struct cxl_port *root, struct cxl_endpoint_decoder *cxled)
* Ensure that if multiple threads race to construct_region() for @hpa
* one does the construction and the others add to that.
*/
- mutex_lock(&cxlrd->range_lock);
+ guard(mutex)(&cxlrd->regions_lock);
region_dev = device_find_child(&cxlrd->cxlsd.cxld.dev, hpa,
match_region_by_range);
if (!region_dev) {
@@ -3353,7 +3379,6 @@ int cxl_add_to_region(struct cxl_port *root, struct cxl_endpoint_decoder *cxled)
region_dev = &cxlr->dev;
} else
cxlr = to_cxl_region(region_dev);
- mutex_unlock(&cxlrd->range_lock);
rc = PTR_ERR_OR_ZERO(cxlr);
if (rc)
diff --git a/drivers/cxl/core/trace.h b/drivers/cxl/core/trace.h
index 8389a94adb1a..2a7f25719319 100644
--- a/drivers/cxl/core/trace.h
+++ b/drivers/cxl/core/trace.h
@@ -57,7 +57,7 @@ TRACE_EVENT(cxl_aer_uncorrectable_error,
__field(u64, serial)
__field(u32, status)
__field(u32, first_error)
- __array(u32, header_log, CXL_HEADERLOG_SIZE_U32)
+ __array(u32, header_log, CXL_HEADERLOG_TRACE_SIZE_U32)
),
TP_fast_assign(
__assign_str(memdev);
@@ -66,10 +66,14 @@ TRACE_EVENT(cxl_aer_uncorrectable_error,
__entry->status = status;
__entry->first_error = fe;
/*
- * Embed the 512B headerlog data for user app retrieval and
- * parsing, but no need to print this in the trace buffer.
+ * Embed headerlog data for user app retrieval and parsing,
+ * but no need to print in the trace buffer. Only
+ * CXL_HEADERLOG_SIZE_U32 (16) dwords are hardware data;
+ * the remaining entries preserve the 512-byte ABI layout
+ * rasdaemon depends on and are zero-filled by the caller.
*/
- memcpy(__entry->header_log, hl, CXL_HEADERLOG_SIZE);
+ memcpy(__entry->header_log, hl,
+ CXL_HEADERLOG_TRACE_SIZE_U32 * sizeof(u32));
),
TP_printk("memdev=%s host=%s serial=%lld: status: '%s' first_error: '%s'",
__get_str(memdev), __get_str(host), __entry->serial,
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 490394c84aee..e2b7e398d4d5 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -147,8 +147,18 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
#define CXL_RAS_CAP_CONTROL_FE_MASK GENMASK(5, 0)
#define CXL_RAS_HEADER_LOG_OFFSET 0x18
#define CXL_RAS_CAPABILITY_LENGTH 0x58
-#define CXL_HEADERLOG_SIZE SZ_512
-#define CXL_HEADERLOG_SIZE_U32 SZ_512 / sizeof(u32)
+#define CXL_HEADERLOG_SIZE SZ_64
+#define CXL_HEADERLOG_SIZE_U32 (CXL_HEADERLOG_SIZE / sizeof(u32))
+
+/*
+ * The RAS UCE trace event header array was originally sized at SZ_512/sizeof(u32)
+ * = 128 u32s due to a bug. Userspace tools (rasdaemon) have grown a dependency
+ * on that 512-byte layout. Keep the trace array at 128 u32s to preserve the
+ * ABI; only CXL_HEADERLOG_SIZE_U32 (16) dwords are valid hardware data, the
+ * remainder are zero-filled.
+ */
+#define CXL_HEADERLOG_TRACE_SIZE SZ_512
+#define CXL_HEADERLOG_TRACE_SIZE_U32 (CXL_HEADERLOG_TRACE_SIZE / sizeof(u32))
/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
#define CXLDEV_CAP_ARRAY_OFFSET 0x0
@@ -442,7 +452,9 @@ typedef u64 (*cxl_hpa_to_spa_fn)(struct cxl_root_decoder *cxlrd, u64 hpa);
* @region_id: region id for next region provisioning event
* @hpa_to_spa: translate CXL host-physical-address to Platform system-physical-address
* @platform_data: platform specific configuration data
- * @range_lock: sync region autodiscovery by address range
+ * @regions_lock: sync region discovery, construction, and deletion
+ * @regions: regions to remove at root decoder destruct time
+ * @dead: root decoder dead to region creation
* @qos_class: QoS performance class cookie
* @cxlsd: base cxl switch decoder
*/
@@ -451,7 +463,9 @@ struct cxl_root_decoder {
atomic_t region_id;
cxl_hpa_to_spa_fn hpa_to_spa;
void *platform_data;
- struct mutex range_lock;
+ struct mutex regions_lock;
+ struct xarray regions;
+ bool dead;
int qos_class;
struct cxl_switch_decoder cxlsd;
};
diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
index e55136bb525e..4be796248166 100644
--- a/drivers/dma/idxd/init.c
+++ b/drivers/dma/idxd/init.c
@@ -155,18 +155,12 @@ static void idxd_cleanup_interrupts(struct idxd_device *idxd)
static void idxd_clean_wqs(struct idxd_device *idxd)
{
- struct idxd_wq *wq;
struct device *conf_dev;
int i;
for (i = 0; i < idxd->max_wqs; i++) {
- wq = idxd->wqs[i];
- if (idxd->hw.wq_cap.op_config)
- bitmap_free(wq->opcap_bmap);
- kfree(wq->wqcfg);
- conf_dev = wq_confdev(wq);
+ conf_dev = wq_confdev(idxd->wqs[i]);
put_device(conf_dev);
- kfree(wq);
}
bitmap_free(idxd->wq_enable_map);
kfree(idxd->wqs);
@@ -208,7 +202,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "wq%d.%d", idxd->id, wq->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(wq);
goto err_unwind;
}
@@ -222,7 +215,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
wq->wqcfg = kzalloc_node(idxd->wqcfg_size, GFP_KERNEL, dev_to_node(dev));
if (!wq->wqcfg) {
put_device(conf_dev);
- kfree(wq);
rc = -ENOMEM;
goto err_unwind;
}
@@ -230,9 +222,7 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
if (idxd->hw.wq_cap.op_config) {
wq->opcap_bmap = bitmap_zalloc(IDXD_MAX_OPCAP_BITS, GFP_KERNEL);
if (!wq->opcap_bmap) {
- kfree(wq->wqcfg);
put_device(conf_dev);
- kfree(wq);
rc = -ENOMEM;
goto err_unwind;
}
@@ -247,13 +237,8 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
err_unwind:
while (--i >= 0) {
- wq = idxd->wqs[i];
- if (idxd->hw.wq_cap.op_config)
- bitmap_free(wq->opcap_bmap);
- kfree(wq->wqcfg);
- conf_dev = wq_confdev(wq);
+ conf_dev = wq_confdev(idxd->wqs[i]);
put_device(conf_dev);
- kfree(wq);
}
bitmap_free(idxd->wq_enable_map);
@@ -265,15 +250,12 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
static void idxd_clean_engines(struct idxd_device *idxd)
{
- struct idxd_engine *engine;
struct device *conf_dev;
int i;
for (i = 0; i < idxd->max_engines; i++) {
- engine = idxd->engines[i];
- conf_dev = engine_confdev(engine);
+ conf_dev = engine_confdev(idxd->engines[i]);
put_device(conf_dev);
- kfree(engine);
}
kfree(idxd->engines);
}
@@ -308,7 +290,6 @@ static int idxd_setup_engines(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "engine%d.%d", idxd->id, engine->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(engine);
goto err;
}
@@ -319,10 +300,8 @@ static int idxd_setup_engines(struct idxd_device *idxd)
err:
while (--i >= 0) {
- engine = idxd->engines[i];
- conf_dev = engine_confdev(engine);
+ conf_dev = engine_confdev(idxd->engines[i]);
put_device(conf_dev);
- kfree(engine);
}
kfree(idxd->engines);
@@ -331,13 +310,10 @@ static int idxd_setup_engines(struct idxd_device *idxd)
static void idxd_clean_groups(struct idxd_device *idxd)
{
- struct idxd_group *group;
int i;
for (i = 0; i < idxd->max_groups; i++) {
- group = idxd->groups[i];
- put_device(group_confdev(group));
- kfree(group);
+ put_device(group_confdev(idxd->groups[i]));
}
kfree(idxd->groups);
}
@@ -372,7 +348,6 @@ static int idxd_setup_groups(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "group%d.%d", idxd->id, group->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(group);
goto err;
}
@@ -397,7 +372,6 @@ static int idxd_setup_groups(struct idxd_device *idxd)
while (--i >= 0) {
group = idxd->groups[i];
put_device(group_confdev(group));
- kfree(group);
}
kfree(idxd->groups);
diff --git a/drivers/dma/ti/k3-udma-glue.c b/drivers/dma/ti/k3-udma-glue.c
index bf177879cc28..d651983fe381 100644
--- a/drivers/dma/ti/k3-udma-glue.c
+++ b/drivers/dma/ti/k3-udma-glue.c
@@ -84,6 +84,7 @@ struct k3_udma_glue_rx_channel {
struct k3_udma_glue_rx_flow *flows;
u32 flow_num;
u32 flows_ready;
+ bool single_fdq; /* one FDQ for all flows */
};
static void k3_udma_chan_dev_release(struct device *dev)
@@ -970,10 +971,13 @@ k3_udma_glue_request_rx_chn_priv(struct device *dev, const char *name,
ep_cfg = rx_chn->common.ep_config;
- if (xudma_is_pktdma(rx_chn->common.udmax))
+ if (xudma_is_pktdma(rx_chn->common.udmax)) {
rx_chn->udma_rchan_id = ep_cfg->mapped_channel_id;
- else
+ rx_chn->single_fdq = false;
+ } else {
rx_chn->udma_rchan_id = -1;
+ rx_chn->single_fdq = true;
+ }
/* request and cfg UDMAP RX channel */
rx_chn->udma_rchanx = xudma_rchan_get(rx_chn->common.udmax,
@@ -1103,6 +1107,9 @@ k3_udma_glue_request_remote_rx_chn_common(struct k3_udma_glue_rx_channel *rx_chn
rx_chn->common.chan_dev.dma_coherent = true;
dma_coerce_mask_and_coherent(&rx_chn->common.chan_dev,
DMA_BIT_MASK(48));
+ rx_chn->single_fdq = false;
+ } else {
+ rx_chn->single_fdq = true;
}
ret = k3_udma_glue_allocate_rx_flows(rx_chn, cfg);
@@ -1454,7 +1461,7 @@ EXPORT_SYMBOL_GPL(k3_udma_glue_tdown_rx_chn);
void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_num, void *data,
- void (*cleanup)(void *data, dma_addr_t desc_dma), bool skip_fdq)
+ void (*cleanup)(void *data, dma_addr_t desc_dma))
{
struct k3_udma_glue_rx_flow *flow = &rx_chn->flows[flow_num];
struct device *dev = rx_chn->common.dev;
@@ -1466,7 +1473,7 @@ void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn,
dev_dbg(dev, "RX reset flow %u occ_rx %u\n", flow_num, occ_rx);
/* Skip RX FDQ in case one FDQ is used for the set of flows */
- if (skip_fdq)
+ if (rx_chn->single_fdq && flow_num)
goto do_reset;
/*
diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
index 4a89ad8c90a3..39e6625c143c 100644
--- a/drivers/dma/xilinx/xilinx_dma.c
+++ b/drivers/dma/xilinx/xilinx_dma.c
@@ -166,6 +166,8 @@
/* Delay loop counter to prevent hardware failure */
#define XILINX_DMA_LOOP_COUNT 1000000
+/* Delay between polls (avoid a delay of 0 to prevent CPU stalls) */
+#define XILINX_DMA_POLL_DELAY_US 10
/* AXI DMA Specific Registers/Offsets */
#define XILINX_DMA_REG_SRCDSTADDR 0x18
@@ -1315,7 +1317,8 @@ static int xilinx_dma_stop_transfer(struct xilinx_dma_chan *chan)
/* Wait for the hardware to halt */
return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- val & XILINX_DMA_DMASR_HALTED, 0,
+ val & XILINX_DMA_DMASR_HALTED,
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
}
@@ -1330,7 +1333,8 @@ static int xilinx_cdma_stop_transfer(struct xilinx_dma_chan *chan)
u32 val;
return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- val & XILINX_DMA_DMASR_IDLE, 0,
+ val & XILINX_DMA_DMASR_IDLE,
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
}
@@ -1347,7 +1351,8 @@ static void xilinx_dma_start(struct xilinx_dma_chan *chan)
/* Wait for the hardware to start */
err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- !(val & XILINX_DMA_DMASR_HALTED), 0,
+ !(val & XILINX_DMA_DMASR_HALTED),
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
if (err) {
@@ -1780,7 +1785,8 @@ static int xilinx_dma_reset(struct xilinx_dma_chan *chan)
/* Wait for the hardware to finish reset */
err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMACR, tmp,
- !(tmp & XILINX_DMA_DMACR_RESET), 0,
+ !(tmp & XILINX_DMA_DMACR_RESET),
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
if (err) {
diff --git a/drivers/dpll/dpll_core.c b/drivers/dpll/dpll_core.c
index 6fa6a0d8e430..0a8db4dc2578 100644
--- a/drivers/dpll/dpll_core.c
+++ b/drivers/dpll/dpll_core.c
@@ -1009,12 +1009,8 @@ EXPORT_SYMBOL_GPL(dpll_pin_ref_sync_pair_add);
static struct dpll_device_registration *
dpll_device_registration_first(struct dpll_device *dpll)
{
- struct dpll_device_registration *reg;
-
- reg = list_first_entry_or_null((struct list_head *)&dpll->registration_list,
- struct dpll_device_registration, list);
- WARN_ON(!reg);
- return reg;
+ return list_first_entry_or_null((struct list_head *)&dpll->registration_list,
+ struct dpll_device_registration, list);
}
void *dpll_priv(struct dpll_device *dpll)
@@ -1022,6 +1018,8 @@ void *dpll_priv(struct dpll_device *dpll)
struct dpll_device_registration *reg;
reg = dpll_device_registration_first(dpll);
+ if (!reg)
+ return NULL;
return reg->priv;
}
@@ -1030,6 +1028,8 @@ const struct dpll_device_ops *dpll_device_ops(struct dpll_device *dpll)
struct dpll_device_registration *reg;
reg = dpll_device_registration_first(dpll);
+ if (!reg)
+ return NULL;
return reg->ops;
}
diff --git a/drivers/dpll/dpll_netlink.c b/drivers/dpll/dpll_netlink.c
index 270ea8f7248a..3939f0362039 100644
--- a/drivers/dpll/dpll_netlink.c
+++ b/drivers/dpll/dpll_netlink.c
@@ -66,6 +66,22 @@ static bool dpll_pin_available(struct dpll_pin *pin)
return false;
}
+static bool dpll_device_registered(struct dpll_device *dpll)
+{
+ return dpll_device_ops(dpll);
+}
+
+static struct dpll_pin_ref *dpll_pin_first_registered_ref(struct dpll_pin *pin)
+{
+ struct dpll_pin_ref *ref;
+ unsigned long i;
+
+ xa_for_each(&pin->dpll_refs, i, ref)
+ if (dpll_device_registered(ref->dpll))
+ return ref;
+ return NULL;
+}
+
/**
* dpll_msg_add_pin_handle - attach pin handle attribute to a given message
* @msg: pointer to sk_buff message to attach a pin handle
@@ -534,6 +550,8 @@ dpll_msg_add_pin_dplls(struct sk_buff *msg, struct dpll_pin *pin,
int ret;
xa_for_each(&pin->dpll_refs, index, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
attr = nla_nest_start(msg, DPLL_A_PIN_PARENT_DEVICE);
if (!attr)
return -EMSGSIZE;
@@ -571,7 +589,10 @@ dpll_cmd_pin_get_one(struct sk_buff *msg, struct dpll_pin *pin,
int ret;
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
- ASSERT_NOT_NULL(ref);
+ if (!ref || !dpll_device_registered(ref->dpll))
+ ref = dpll_pin_first_registered_ref(pin);
+ if (!ref)
+ return -ENODEV;
ret = dpll_msg_add_pin_handle(msg, pin);
if (ret)
@@ -817,6 +838,8 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
}
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
if (!ops->frequency_set || !ops->frequency_get) {
NL_SET_ERR_MSG(extack, "frequency set not supported by the device");
@@ -824,6 +847,10 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
}
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->frequency_get(pin, dpll_pin_on_dpll_priv(dpll, pin), dpll,
@@ -836,6 +863,8 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
return 0;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->frequency_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -855,6 +884,8 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
xa_for_each(&pin->dpll_refs, i, ref) {
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
if (ops->frequency_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -878,6 +909,8 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
int ret;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
if (!ops->esync_set || !ops->esync_get) {
NL_SET_ERR_MSG(extack,
@@ -886,6 +919,10 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
}
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->esync_get(pin, dpll_pin_on_dpll_priv(dpll, pin), dpll,
@@ -908,6 +945,8 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
xa_for_each(&pin->dpll_refs, i, ref) {
void *pin_dpll_priv;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
pin_dpll_priv = dpll_pin_on_dpll_priv(dpll, pin);
@@ -931,6 +970,8 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
pin_dpll_priv = dpll_pin_on_dpll_priv(dpll, pin);
@@ -966,7 +1007,10 @@ dpll_pin_ref_sync_state_set(struct dpll_pin *pin,
return -EINVAL;
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
- ASSERT_NOT_NULL(ref);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
if (!ops->ref_sync_set || !ops->ref_sync_get) {
NL_SET_ERR_MSG(extack, "reference sync not supported by this pin");
@@ -984,6 +1028,8 @@ dpll_pin_ref_sync_state_set(struct dpll_pin *pin,
if (state == old_state)
return 0;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->ref_sync_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1006,6 +1052,8 @@ dpll_pin_ref_sync_state_set(struct dpll_pin *pin,
xa_for_each(&pin->dpll_refs, i, ref) {
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
if (ops->ref_sync_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1185,6 +1233,8 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
}
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
if (!ops->phase_adjust_set || !ops->phase_adjust_get) {
NL_SET_ERR_MSG(extack, "phase adjust not supported");
@@ -1192,6 +1242,10 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
}
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->phase_adjust_get(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1205,6 +1259,8 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
return 0;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->phase_adjust_set(pin,
@@ -1227,6 +1283,8 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
xa_for_each(&pin->dpll_refs, i, ref) {
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
if (ops->phase_adjust_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1256,7 +1314,7 @@ dpll_pin_parent_device_set(struct dpll_pin *pin, struct nlattr *parent_nest,
return -EINVAL;
}
pdpll_idx = nla_get_u32(tb[DPLL_A_PIN_PARENT_ID]);
- dpll = xa_load(&dpll_device_xa, pdpll_idx);
+ dpll = dpll_device_get_by_id(pdpll_idx);
if (!dpll) {
NL_SET_ERR_MSG(extack, "parent device not found");
return -EINVAL;
@@ -1546,6 +1604,10 @@ int dpll_nl_pin_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
ret = dpll_cmd_pin_get_one(skb, pin, cb->extack);
if (ret) {
genlmsg_cancel(skb, hdr);
+ if (ret == -ENODEV) {
+ ret = 0;
+ continue;
+ }
break;
}
genlmsg_end(skb, hdr);
diff --git a/drivers/edac/igen6_edac.c b/drivers/edac/igen6_edac.c
index 63a61932e67f..28a6f9e66721 100644
--- a/drivers/edac/igen6_edac.c
+++ b/drivers/edac/igen6_edac.c
@@ -127,6 +127,7 @@
static struct res_config {
bool machine_check;
+ /* The number of present memory controllers. */
int num_imc;
u32 imc_base;
u32 cmf_base;
@@ -557,7 +558,7 @@ static struct res_config mtl_p_cfg = {
.err_addr_to_imc_addr = adl_err_addr_to_imc_addr,
};
-static const struct pci_device_id igen6_pci_tbl[] = {
+static struct pci_device_id igen6_pci_tbl[] = {
{ PCI_VDEVICE(INTEL, DID_EHL_SKU5), (kernel_ulong_t)&ehl_cfg },
{ PCI_VDEVICE(INTEL, DID_EHL_SKU6), (kernel_ulong_t)&ehl_cfg },
{ PCI_VDEVICE(INTEL, DID_EHL_SKU7), (kernel_ulong_t)&ehl_cfg },
@@ -784,13 +785,22 @@ static u64 ecclog_read_and_clear(struct igen6_imc *imc)
{
u64 ecclog = readq(imc->window + ECC_ERROR_LOG_OFFSET);
- if (ecclog & (ECC_ERROR_LOG_CE | ECC_ERROR_LOG_UE)) {
- /* Clear CE/UE bits by writing 1s */
- writeq(ecclog, imc->window + ECC_ERROR_LOG_OFFSET);
- return ecclog;
- }
+ /*
+ * Quirk: The ECC_ERROR_LOG register of certain SoCs may contain
+ * the invalid value ~0. This will result in a flood of invalid
+ * error reports in polling mode. Skip it.
+ */
+ if (ecclog == ~0)
+ return 0;
- return 0;
+ /* Neither a CE nor a UE. Skip it.*/
+ if (!(ecclog & (ECC_ERROR_LOG_CE | ECC_ERROR_LOG_UE)))
+ return 0;
+
+ /* Clear CE/UE bits by writing 1s */
+ writeq(ecclog, imc->window + ECC_ERROR_LOG_OFFSET);
+
+ return ecclog;
}
static void errsts_clear(struct igen6_imc *imc)
@@ -1177,23 +1187,40 @@ static int igen6_pci_setup(struct pci_dev *pdev, u64 *mchbar)
return -ENODEV;
}
-static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev)
+static void igen6_check(struct mem_ctl_info *mci)
+{
+ struct igen6_imc *imc = mci->pvt_info;
+ u64 ecclog;
+
+ /* errsts_clear() isn't NMI-safe. Delay it in the IRQ context */
+ ecclog = ecclog_read_and_clear(imc);
+ if (!ecclog)
+ return;
+
+ if (!ecclog_gen_pool_add(imc->mc, ecclog))
+ irq_work_queue(&ecclog_irq_work);
+}
+
+/* Check whether the memory controller is absent. */
+static bool igen6_imc_absent(void __iomem *window)
+{
+ return readl(window + MAD_INTER_CHANNEL_OFFSET) == ~0;
+}
+
+static void imc_release(struct device *dev)
+{
+ /* Nothing to do, the 'imc' owns the 'dev' and will also release it. */
+}
+
+static int igen6_register_mci(int mc, void __iomem *window, struct pci_dev *pdev)
{
struct edac_mc_layer layers[2];
struct mem_ctl_info *mci;
struct igen6_imc *imc;
- void __iomem *window;
int rc;
edac_dbg(2, "\n");
- mchbar += mc * MCHBAR_SIZE;
- window = ioremap(mchbar, MCHBAR_SIZE);
- if (!window) {
- igen6_printk(KERN_ERR, "Failed to ioremap 0x%llx\n", mchbar);
- return -ENODEV;
- }
-
layers[0].type = EDAC_MC_LAYER_CHANNEL;
layers[0].size = NUM_CHANNELS;
layers[0].is_virt_csrow = false;
@@ -1218,9 +1245,12 @@ static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev)
mci->edac_cap = EDAC_FLAG_SECDED;
mci->mod_name = EDAC_MOD_STR;
mci->dev_name = pci_name(pdev);
+ if (edac_op_state == EDAC_OPSTATE_POLL)
+ mci->edac_check = igen6_check;
mci->pvt_info = &igen6_pvt->imc[mc];
imc = mci->pvt_info;
+ imc->dev.release = imc_release;
device_initialize(&imc->dev);
/*
* EDAC core uses mci->pdev(pointer of structure device) as
@@ -1257,7 +1287,6 @@ static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev)
fail2:
edac_mc_free(mci);
fail:
- iounmap(window);
return rc;
}
@@ -1283,6 +1312,58 @@ static void igen6_unregister_mcis(void)
}
}
+static int igen6_register_mcis(struct pci_dev *pdev, u64 mchbar)
+{
+ void __iomem *window;
+ int lmc, pmc, rc;
+ u64 base;
+
+ for (lmc = 0, pmc = 0; pmc < NUM_IMC; pmc++) {
+ base = mchbar + pmc * MCHBAR_SIZE;
+ window = ioremap(base, MCHBAR_SIZE);
+ if (!window) {
+ igen6_printk(KERN_ERR, "Failed to ioremap 0x%llx for mc%d\n", base, pmc);
+ rc = -ENOMEM;
+ goto out_unregister_mcis;
+ }
+
+ if (igen6_imc_absent(window)) {
+ iounmap(window);
+ edac_dbg(2, "Skip absent mc%d\n", pmc);
+ continue;
+ }
+
+ rc = igen6_register_mci(lmc, window, pdev);
+ if (rc)
+ goto out_iounmap;
+
+ /* Done, if all present MCs are detected and registered. */
+ if (++lmc >= res_cfg->num_imc)
+ break;
+ }
+
+ if (!lmc) {
+ igen6_printk(KERN_ERR, "No mc found.\n");
+ return -ENODEV;
+ }
+
+ if (lmc < res_cfg->num_imc) {
+ igen6_printk(KERN_WARNING, "Expected %d mcs, but only %d detected.",
+ res_cfg->num_imc, lmc);
+ res_cfg->num_imc = lmc;
+ }
+
+ return 0;
+
+out_iounmap:
+ iounmap(window);
+
+out_unregister_mcis:
+ igen6_unregister_mcis();
+
+ return rc;
+}
+
static int igen6_mem_slice_setup(u64 mchbar)
{
struct igen6_imc *imc = &igen6_pvt->imc[0];
@@ -1357,10 +1438,29 @@ static void unregister_err_handler(void)
unregister_nmi_handler(NMI_SERR, IGEN6_NMI_NAME);
}
+static void opstate_set(struct res_config *cfg, const struct pci_device_id *ent)
+{
+ /*
+ * Quirk: Certain SoCs' error reporting interrupts don't work.
+ * Force polling mode for them to ensure that memory error
+ * events can be handled.
+ */
+ if (ent->device == DID_ADL_N_SKU4) {
+ edac_op_state = EDAC_OPSTATE_POLL;
+ return;
+ }
+
+ /* Set the mode according to the configuration data. */
+ if (cfg->machine_check)
+ edac_op_state = EDAC_OPSTATE_INT;
+ else
+ edac_op_state = EDAC_OPSTATE_NMI;
+}
+
static int igen6_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
{
u64 mchbar;
- int i, rc;
+ int rc;
edac_dbg(2, "\n");
@@ -1374,11 +1474,11 @@ static int igen6_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
if (rc)
goto fail;
- for (i = 0; i < res_cfg->num_imc; i++) {
- rc = igen6_register_mci(i, mchbar, pdev);
- if (rc)
- goto fail2;
- }
+ opstate_set(res_cfg, ent);
+
+ rc = igen6_register_mcis(pdev, mchbar);
+ if (rc)
+ goto fail;
if (res_cfg->num_imc > 1) {
rc = igen6_mem_slice_setup(mchbar);
@@ -1457,8 +1557,6 @@ static int __init igen6_init(void)
if (owner && strncmp(owner, EDAC_MOD_STR, sizeof(EDAC_MOD_STR)))
return -EBUSY;
- edac_op_state = EDAC_OPSTATE_NMI;
-
rc = pci_register_driver(&igen6_driver);
if (rc)
return rc;
diff --git a/drivers/firewire/ohci.c b/drivers/firewire/ohci.c
index 90fcab1f65bc..4dca78f859f6 100644
--- a/drivers/firewire/ohci.c
+++ b/drivers/firewire/ohci.c
@@ -753,12 +753,14 @@ static void ar_context_link_page(struct ar_context *ctx, unsigned int index)
static void ar_context_release(struct ar_context *ctx)
{
- struct device *dev = ctx->ohci->card.device;
+ struct device *dev;
unsigned int i;
if (!ctx->buffer)
return;
+ dev = ctx->ohci->card.device;
+
vunmap(ctx->buffer);
for (i = 0; i < AR_BUFFERS; i++) {
diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c
index 62222c9cfc3e..04a4a58cb68b 100644
--- a/drivers/firmware/arm_ffa/driver.c
+++ b/drivers/firmware/arm_ffa/driver.c
@@ -613,6 +613,26 @@ static u16 ffa_memory_attributes_get(u32 func_id)
return FFA_MEM_NORMAL | FFA_MEM_WRITE_BACK | FFA_MEM_INNER_SHAREABLE;
}
+static void ffa_emad_impdef_value_init(u32 version, void *dst, void *src)
+{
+ struct ffa_mem_region_attributes *ep_mem_access;
+
+ if (FFA_EMAD_HAS_IMPDEF_FIELD(version))
+ memcpy(dst, src, sizeof(ep_mem_access->impdef_val));
+}
+
+static void
+ffa_mem_region_additional_setup(u32 version, struct ffa_mem_region *mem_region)
+{
+ if (!FFA_MEM_REGION_HAS_EP_MEM_OFFSET(version)) {
+ mem_region->ep_mem_size = 0;
+ } else {
+ mem_region->ep_mem_size = ffa_emad_size_get(version);
+ mem_region->ep_mem_offset = sizeof(*mem_region);
+ memset(mem_region->reserved, 0, 12);
+ }
+}
+
static int
ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
struct ffa_mem_ops_args *args)
@@ -625,33 +645,39 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
struct ffa_composite_mem_region *composite;
struct ffa_mem_region_addr_range *constituents;
struct ffa_mem_region_attributes *ep_mem_access;
- u32 idx, frag_len, length, buf_sz = 0, num_entries = sg_nents(args->sg);
+ u32 idx, frag_len, length, buf_sz = 0, num_entries = sg_nents(args->sg), ep_offset;
+ u32 emad_end, emad_size = ffa_emad_size_get(drv_info->version);
mem_region->tag = args->tag;
mem_region->flags = args->flags;
mem_region->sender_id = drv_info->vm_id;
mem_region->attributes = ffa_memory_attributes_get(func_id);
- ep_mem_access = buffer +
- ffa_mem_desc_offset(buffer, 0, drv_info->version);
+
+ ffa_mem_region_additional_setup(drv_info->version, mem_region);
composite_offset = ffa_mem_desc_offset(buffer, args->nattrs,
drv_info->version);
+ if (composite_offset + sizeof(*composite) > max_fragsize)
+ return -ENXIO;
+
+ for (idx = 0; idx < args->nattrs; idx++) {
+ ep_offset = ffa_mem_desc_offset(buffer, idx, drv_info->version);
+ if (check_add_overflow(ep_offset, emad_size, &emad_end))
+ return -ENXIO;
+
+ if (emad_end > max_fragsize)
+ return -ENXIO;
- for (idx = 0; idx < args->nattrs; idx++, ep_mem_access++) {
+ ep_mem_access = buffer + ep_offset;
+ memset(ep_mem_access, 0, emad_size);
ep_mem_access->receiver = args->attrs[idx].receiver;
ep_mem_access->attrs = args->attrs[idx].attrs;
ep_mem_access->composite_off = composite_offset;
- ep_mem_access->flag = 0;
- ep_mem_access->reserved = 0;
+ ffa_emad_impdef_value_init(drv_info->version,
+ ep_mem_access->impdef_val,
+ args->attrs[idx].impdef_val);
}
mem_region->handle = 0;
mem_region->ep_count = args->nattrs;
- if (drv_info->version <= FFA_VERSION_1_0) {
- mem_region->ep_mem_size = 0;
- } else {
- mem_region->ep_mem_size = sizeof(*ep_mem_access);
- mem_region->ep_mem_offset = sizeof(*mem_region);
- memset(mem_region->reserved, 0, 12);
- }
composite = buffer + composite_offset;
composite->total_pg_cnt = ffa_get_num_pages_sg(args->sg);
@@ -684,7 +710,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
constituents = buffer;
}
- if ((void *)constituents - buffer > max_fragsize) {
+ if ((void *)constituents + sizeof(*constituents) - buffer > max_fragsize) {
pr_err("Memory Region Fragment > Tx Buffer size\n");
return -EFAULT;
}
@@ -693,7 +719,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
constituents->pg_cnt = args->sg->length / FFA_PAGE_SIZE;
constituents->reserved = 0;
constituents++;
- frag_len += sizeof(struct ffa_mem_region_addr_range);
+ frag_len += sizeof(*constituents);
} while ((args->sg = sg_next(args->sg)));
return ffa_transmit_fragment(func_id, addr, buf_sz, frag_len,
diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c
index 771b6da2d83b..8769078e3689 100644
--- a/drivers/firmware/arm_scmi/bus.c
+++ b/drivers/firmware/arm_scmi/bus.c
@@ -249,8 +249,9 @@ static int scmi_match_by_id_table(struct device *dev, void *data)
(id_table->name && !strcmp(sdev->name, id_table->name));
}
-static struct scmi_device *scmi_child_dev_find(struct device *parent,
- int prot_id, const char *name)
+/* Returns a device_find_child() reference which must be dropped by caller. */
+static struct scmi_device *
+scmi_child_dev_find_get(struct device *parent, int prot_id, const char *name)
{
struct scmi_device_id id_table;
struct device *dev;
@@ -262,9 +263,6 @@ static struct scmi_device *scmi_child_dev_find(struct device *parent,
if (!dev)
return NULL;
- /* Drop the refcnt bumped implicitly by device_find_child */
- put_device(dev);
-
return to_scmi_dev(dev);
}
@@ -332,10 +330,19 @@ void scmi_driver_unregister(struct scmi_driver *driver)
}
EXPORT_SYMBOL_GPL(scmi_driver_unregister);
+static void scmi_device_release_resources(struct scmi_device *scmi_dev)
+{
+ if (scmi_dev->id) {
+ ida_free(&scmi_bus_id, scmi_dev->id);
+ scmi_dev->id = 0;
+ }
+}
+
static void scmi_device_release(struct device *dev)
{
struct scmi_device *scmi_dev = to_scmi_dev(dev);
+ scmi_device_release_resources(scmi_dev);
kfree_const(scmi_dev->name);
kfree(scmi_dev);
}
@@ -350,8 +357,9 @@ static void __scmi_device_destroy(struct scmi_device *scmi_dev)
if (scmi_dev->protocol_id == SCMI_PROTOCOL_SYSTEM)
atomic_set(&scmi_syspower_registered, 0);
- ida_free(&scmi_bus_id, scmi_dev->id);
- device_unregister(&scmi_dev->dev);
+ device_del(&scmi_dev->dev);
+ scmi_device_release_resources(scmi_dev);
+ put_device(&scmi_dev->dev);
}
static struct scmi_device *
@@ -368,9 +376,11 @@ __scmi_device_create(struct device_node *np, struct device *parent,
* each DT defined protocol at probe time, and the concurrent
* registration of SCMI drivers.
*/
- scmi_dev = scmi_child_dev_find(parent, protocol, name);
- if (scmi_dev)
+ scmi_dev = scmi_child_dev_find_get(parent, protocol, name);
+ if (scmi_dev) {
+ put_device(&scmi_dev->dev);
return scmi_dev;
+ }
/*
* Ignore any possible subsequent failures while creating the device
@@ -421,8 +431,8 @@ __scmi_device_create(struct device_node *np, struct device *parent,
return scmi_dev;
put_dev:
+ scmi_device_release_resources(scmi_dev);
put_device(&scmi_dev->dev);
- ida_free(&scmi_bus_id, id);
return NULL;
}
@@ -494,9 +504,11 @@ void scmi_device_destroy(struct device *parent, int protocol, const char *name)
{
struct scmi_device *scmi_dev;
- scmi_dev = scmi_child_dev_find(parent, protocol, name);
- if (scmi_dev)
+ scmi_dev = scmi_child_dev_find_get(parent, protocol, name);
+ if (scmi_dev) {
__scmi_device_destroy(scmi_dev);
+ put_device(&scmi_dev->dev);
+ }
}
EXPORT_SYMBOL_GPL(scmi_device_destroy);
diff --git a/drivers/gpio/gpio-latch.c b/drivers/gpio/gpio-latch.c
index d7c3b20c8482..e935b3b34117 100644
--- a/drivers/gpio/gpio-latch.c
+++ b/drivers/gpio/gpio-latch.c
@@ -72,7 +72,7 @@ static int gpio_latch_get_direction(struct gpio_chip *gc, unsigned int offset)
}
static void gpio_latch_set_unlocked(struct gpio_latch_priv *priv,
- void (*set)(struct gpio_desc *desc, int value),
+ int (*set)(struct gpio_desc *desc, int value),
unsigned int offset, bool val)
{
int latch = offset / priv->n_latched_gpios;
diff --git a/drivers/gpio/gpio-mxc.c b/drivers/gpio/gpio-mxc.c
index a8ab78ae7fa3..a61bc17b2708 100644
--- a/drivers/gpio/gpio-mxc.c
+++ b/drivers/gpio/gpio-mxc.c
@@ -667,7 +667,7 @@ static const struct dev_pm_ops mxc_gpio_dev_pm_ops = {
RUNTIME_PM_OPS(mxc_gpio_runtime_suspend, mxc_gpio_runtime_resume, NULL)
};
-static int mxc_gpio_syscore_suspend(void)
+static int mxc_gpio_syscore_suspend(void *data)
{
struct mxc_gpio_port *port;
int ret;
@@ -684,7 +684,7 @@ static int mxc_gpio_syscore_suspend(void)
return 0;
}
-static void mxc_gpio_syscore_resume(void)
+static void mxc_gpio_syscore_resume(void *data)
{
struct mxc_gpio_port *port;
int ret;
@@ -701,11 +701,15 @@ static void mxc_gpio_syscore_resume(void)
}
}
-static struct syscore_ops mxc_gpio_syscore_ops = {
+static const struct syscore_ops mxc_gpio_syscore_ops = {
.suspend = mxc_gpio_syscore_suspend,
.resume = mxc_gpio_syscore_resume,
};
+static struct syscore mxc_gpio_syscore = {
+ .ops = &mxc_gpio_syscore_ops,
+};
+
static struct platform_driver mxc_gpio_driver = {
.driver = {
.name = "gpio-mxc",
@@ -718,7 +722,7 @@ static struct platform_driver mxc_gpio_driver = {
static int __init gpio_mxc_init(void)
{
- register_syscore_ops(&mxc_gpio_syscore_ops);
+ register_syscore(&mxc_gpio_syscore);
return platform_driver_register(&mxc_gpio_driver);
}
diff --git a/drivers/gpio/gpio-pca953x.c b/drivers/gpio/gpio-pca953x.c
index ad9280472bdd..1c7ae71dfe55 100644
--- a/drivers/gpio/gpio-pca953x.c
+++ b/drivers/gpio/gpio-pca953x.c
@@ -525,15 +525,23 @@ static int pca953x_read_regs(struct pca953x_chip *chip, int reg, unsigned long *
return 0;
}
-static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned off)
+static int pca953x_gpio_direction_input_unlocked(struct gpio_chip *gc,
+ unsigned int off)
{
struct pca953x_chip *chip = gpiochip_get_data(gc);
u8 dirreg = chip->recalc_addr(chip, chip->regs->direction, off);
u8 bit = BIT(off % BANK_SZ);
+ return regmap_write_bits(chip->regmap, dirreg, bit, bit);
+}
+
+static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned int off)
+{
+ struct pca953x_chip *chip = gpiochip_get_data(gc);
+
guard(mutex)(&chip->i2c_lock);
- return regmap_write_bits(chip->regmap, dirreg, bit, bit);
+ return pca953x_gpio_direction_input_unlocked(gc, off);
}
static int pca953x_gpio_direction_output(struct gpio_chip *gc,
@@ -759,9 +767,9 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d)
DECLARE_BITMAP(reg_direction, MAX_LINE);
int level;
- if (chip->driver_data & PCA_PCAL) {
- guard(mutex)(&chip->i2c_lock);
+ guard(mutex)(&chip->i2c_lock);
+ if (chip->driver_data & PCA_PCAL) {
/* Enable latch on interrupt-enabled inputs */
pca953x_write_regs(chip, PCAL953X_IN_LATCH, chip->irq_mask);
@@ -782,7 +790,7 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d)
/* Look for any newly setup interrupt */
for_each_set_bit(level, irq_mask, gc->ngpio)
- pca953x_gpio_direction_input(&chip->gpio_chip, level);
+ pca953x_gpio_direction_input_unlocked(&chip->gpio_chip, level);
mutex_unlock(&chip->irq_lock);
}
diff --git a/drivers/gpio/gpio-pxa.c b/drivers/gpio/gpio-pxa.c
index 91cea97255fa..b8c49967e093 100644
--- a/drivers/gpio/gpio-pxa.c
+++ b/drivers/gpio/gpio-pxa.c
@@ -746,7 +746,7 @@ static int __init pxa_gpio_dt_init(void)
device_initcall(pxa_gpio_dt_init);
#ifdef CONFIG_PM
-static int pxa_gpio_suspend(void)
+static int pxa_gpio_suspend(void *data)
{
struct pxa_gpio_chip *pchip = pxa_gpio_chip;
struct pxa_gpio_bank *c;
@@ -767,7 +767,7 @@ static int pxa_gpio_suspend(void)
return 0;
}
-static void pxa_gpio_resume(void)
+static void pxa_gpio_resume(void *data)
{
struct pxa_gpio_chip *pchip = pxa_gpio_chip;
struct pxa_gpio_bank *c;
@@ -791,14 +791,18 @@ static void pxa_gpio_resume(void)
#define pxa_gpio_resume NULL
#endif
-static struct syscore_ops pxa_gpio_syscore_ops = {
+static const struct syscore_ops pxa_gpio_syscore_ops = {
.suspend = pxa_gpio_suspend,
.resume = pxa_gpio_resume,
};
+static struct syscore pxa_gpio_syscore = {
+ .ops = &pxa_gpio_syscore_ops,
+};
+
static int __init pxa_gpio_sysinit(void)
{
- register_syscore_ops(&pxa_gpio_syscore_ops);
+ register_syscore(&pxa_gpio_syscore);
return 0;
}
postcore_initcall(pxa_gpio_sysinit);
diff --git a/drivers/gpio/gpio-sa1100.c b/drivers/gpio/gpio-sa1100.c
index 242dad763ac4..9e3a1199f51b 100644
--- a/drivers/gpio/gpio-sa1100.c
+++ b/drivers/gpio/gpio-sa1100.c
@@ -253,7 +253,7 @@ static void sa1100_gpio_handler(struct irq_desc *desc)
} while (mask);
}
-static int sa1100_gpio_suspend(void)
+static int sa1100_gpio_suspend(void *data)
{
struct sa1100_gpio_chip *sgc = &sa1100_gpio_chip;
@@ -272,19 +272,23 @@ static int sa1100_gpio_suspend(void)
return 0;
}
-static void sa1100_gpio_resume(void)
+static void sa1100_gpio_resume(void *data)
{
sa1100_update_edge_regs(&sa1100_gpio_chip);
}
-static struct syscore_ops sa1100_gpio_syscore_ops = {
+static const struct syscore_ops sa1100_gpio_syscore_ops = {
.suspend = sa1100_gpio_suspend,
.resume = sa1100_gpio_resume,
};
+static struct syscore sa1100_gpio_syscore = {
+ .ops = &sa1100_gpio_syscore_ops,
+};
+
static int __init sa1100_gpio_init_devicefs(void)
{
- register_syscore_ops(&sa1100_gpio_syscore_ops);
+ register_syscore(&sa1100_gpio_syscore);
return 0;
}
diff --git a/drivers/gpio/gpiolib.c b/drivers/gpio/gpiolib.c
index 878f9ab4a098..144f1e174c54 100644
--- a/drivers/gpio/gpiolib.c
+++ b/drivers/gpio/gpiolib.c
@@ -945,6 +945,11 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data,
int base = 0;
int ret;
+ /* Only allow one set() and one set_multiple(). */
+ if ((gc->set && gc->set_rv) ||
+ (gc->set_multiple && gc->set_multiple_rv))
+ return -EINVAL;
+
gdev = kzalloc(sizeof(*gdev), GFP_KERNEL);
if (!gdev)
return -ENOMEM;
@@ -2722,6 +2727,27 @@ int gpiod_direction_input(struct gpio_desc *desc)
}
EXPORT_SYMBOL_GPL(gpiod_direction_input);
+static int gpiochip_set(struct gpio_chip *gc, unsigned int offset, int value)
+{
+ int ret;
+
+ lockdep_assert_held(&gc->gpiodev->srcu);
+
+ if (WARN_ON(unlikely(!gc->set && !gc->set_rv)))
+ return -EOPNOTSUPP;
+
+ if (gc->set_rv) {
+ ret = gc->set_rv(gc, offset, value);
+ if (ret > 0)
+ ret = -EBADE;
+
+ return ret;
+ }
+
+ gc->set(gc, offset, value);
+ return 0;
+}
+
static int gpiod_direction_output_raw_commit(struct gpio_desc *desc, int value)
{
int val = !!value, ret = 0;
@@ -2735,7 +2761,7 @@ static int gpiod_direction_output_raw_commit(struct gpio_desc *desc, int value)
* output-only, but if there is then not even a .set() operation it
* is pretty tricky to drive the output line.
*/
- if (!guard.gc->set && !guard.gc->direction_output) {
+ if (!guard.gc->set && !guard.gc->set_rv && !guard.gc->direction_output) {
gpiod_warn(desc,
"%s: missing set() and direction_output() operations\n",
__func__);
@@ -2758,7 +2784,9 @@ static int gpiod_direction_output_raw_commit(struct gpio_desc *desc, int value)
* If we can't actively set the direction, we are some
* output-only chip, so just drive the output as desired.
*/
- guard.gc->set(guard.gc, gpio_chip_hwgpio(desc), val);
+ ret = gpiochip_set(guard.gc, gpio_chip_hwgpio(desc), val);
+ if (ret)
+ return ret;
}
if (!ret)
@@ -3330,13 +3358,13 @@ EXPORT_SYMBOL_GPL(gpiod_get_array_value);
* @desc: gpio descriptor whose state need to be set.
* @value: Non-zero for setting it HIGH otherwise it will set to LOW.
*/
-static void gpio_set_open_drain_value_commit(struct gpio_desc *desc, bool value)
+static int gpio_set_open_drain_value_commit(struct gpio_desc *desc, bool value)
{
int ret = 0, offset = gpio_chip_hwgpio(desc);
CLASS(gpio_chip_guard, guard)(desc);
if (!guard.gc)
- return;
+ return -ENODEV;
if (value) {
ret = guard.gc->direction_input(guard.gc, offset);
@@ -3350,6 +3378,8 @@ static void gpio_set_open_drain_value_commit(struct gpio_desc *desc, bool value)
gpiod_err(desc,
"%s: Error in set_value for open drain err %d\n",
__func__, ret);
+
+ return ret;
}
/*
@@ -3357,13 +3387,13 @@ static void gpio_set_open_drain_value_commit(struct gpio_desc *desc, bool value)
* @desc: gpio descriptor whose state need to be set.
* @value: Non-zero for setting it HIGH otherwise it will set to LOW.
*/
-static void gpio_set_open_source_value_commit(struct gpio_desc *desc, bool value)
+static int gpio_set_open_source_value_commit(struct gpio_desc *desc, bool value)
{
int ret = 0, offset = gpio_chip_hwgpio(desc);
CLASS(gpio_chip_guard, guard)(desc);
if (!guard.gc)
- return;
+ return -ENODEV;
if (value) {
ret = guard.gc->direction_output(guard.gc, offset, 1);
@@ -3377,16 +3407,18 @@ static void gpio_set_open_source_value_commit(struct gpio_desc *desc, bool value
gpiod_err(desc,
"%s: Error in set_value for open source err %d\n",
__func__, ret);
+
+ return ret;
}
-static void gpiod_set_raw_value_commit(struct gpio_desc *desc, bool value)
+static int gpiod_set_raw_value_commit(struct gpio_desc *desc, bool value)
{
CLASS(gpio_chip_guard, guard)(desc);
if (!guard.gc)
- return;
+ return -ENODEV;
trace_gpio_value(desc_to_gpio(desc), 0, value);
- guard.gc->set(guard.gc, gpio_chip_hwgpio(desc), value);
+ return gpiochip_set(guard.gc, gpio_chip_hwgpio(desc), value);
}
/*
@@ -3398,21 +3430,42 @@ static void gpiod_set_raw_value_commit(struct gpio_desc *desc, bool value)
* defines which outputs are to be changed
* @bits: bit value array; one bit per output; BITS_PER_LONG bits per word
* defines the values the outputs specified by mask are to be set to
+ *
+ * Returns: 0 on success, negative error number on failure.
*/
-static void gpio_chip_set_multiple(struct gpio_chip *gc,
- unsigned long *mask, unsigned long *bits)
+static int gpiochip_set_multiple(struct gpio_chip *gc,
+ unsigned long *mask, unsigned long *bits)
{
+ unsigned int i;
+ int ret = 0;
+
lockdep_assert_held(&gc->gpiodev->srcu);
+ if (WARN_ON(unlikely(!gc->set_multiple && !gc->set_multiple_rv &&
+ !gc->set && !gc->set_rv)))
+ return -EOPNOTSUPP;
+
+ if (gc->set_multiple_rv) {
+ ret = gc->set_multiple_rv(gc, mask, bits);
+ if (ret > 0)
+ ret = -EBADE;
+
+ return ret;
+ }
+
if (gc->set_multiple) {
gc->set_multiple(gc, mask, bits);
- } else {
- unsigned int i;
+ return 0;
+ }
- /* set outputs if the corresponding mask bit is set */
- for_each_set_bit(i, mask, gc->ngpio)
- gc->set(gc, i, test_bit(i, bits));
+ /* set outputs if the corresponding mask bit is set */
+ for_each_set_bit(i, mask, gc->ngpio) {
+ ret = gpiochip_set(gc, i, test_bit(i, bits));
+ if (ret)
+ break;
}
+
+ return ret;
}
int gpiod_set_array_value_complex(bool raw, bool can_sleep,
@@ -3422,7 +3475,7 @@ int gpiod_set_array_value_complex(bool raw, bool can_sleep,
unsigned long *value_bitmap)
{
struct gpio_chip *gc;
- int i = 0;
+ int i = 0, ret;
/*
* Validate array_info against desc_array and its size.
@@ -3445,7 +3498,10 @@ int gpiod_set_array_value_complex(bool raw, bool can_sleep,
bitmap_xor(value_bitmap, value_bitmap,
array_info->invert_mask, array_size);
- gpio_chip_set_multiple(gc, array_info->set_mask, value_bitmap);
+ ret = gpiochip_set_multiple(gc, array_info->set_mask,
+ value_bitmap);
+ if (ret)
+ return ret;
i = find_first_zero_bit(array_info->set_mask, array_size);
if (i == array_size)
@@ -3522,8 +3578,11 @@ int gpiod_set_array_value_complex(bool raw, bool can_sleep,
} while ((i < array_size) &&
gpio_device_chip_cmp(desc_array[i]->gdev, guard.gc));
/* push collected bits to outputs */
- if (count != 0)
- gpio_chip_set_multiple(guard.gc, mask, bits);
+ if (count != 0) {
+ ret = gpiochip_set_multiple(guard.gc, mask, bits);
+ if (ret)
+ return ret;
+ }
if (mask != fastpath_mask)
bitmap_free(mask);
@@ -3544,12 +3603,12 @@ int gpiod_set_array_value_complex(bool raw, bool can_sleep,
* This function can be called from contexts where we cannot sleep, and will
* complain if the GPIO chip functions potentially sleep.
*/
-void gpiod_set_raw_value(struct gpio_desc *desc, int value)
+int gpiod_set_raw_value(struct gpio_desc *desc, int value)
{
- VALIDATE_DESC_VOID(desc);
+ VALIDATE_DESC(desc);
/* Should be using gpiod_set_raw_value_cansleep() */
WARN_ON(desc->gdev->can_sleep);
- gpiod_set_raw_value_commit(desc, value);
+ return gpiod_set_raw_value_commit(desc, value);
}
EXPORT_SYMBOL_GPL(gpiod_set_raw_value);
@@ -3562,16 +3621,17 @@ EXPORT_SYMBOL_GPL(gpiod_set_raw_value);
* different semantic quirks like active low and open drain/source
* handling.
*/
-static void gpiod_set_value_nocheck(struct gpio_desc *desc, int value)
+static int gpiod_set_value_nocheck(struct gpio_desc *desc, int value)
{
if (test_bit(FLAG_ACTIVE_LOW, &desc->flags))
value = !value;
+
if (test_bit(FLAG_OPEN_DRAIN, &desc->flags))
- gpio_set_open_drain_value_commit(desc, value);
+ return gpio_set_open_drain_value_commit(desc, value);
else if (test_bit(FLAG_OPEN_SOURCE, &desc->flags))
- gpio_set_open_source_value_commit(desc, value);
- else
- gpiod_set_raw_value_commit(desc, value);
+ return gpio_set_open_source_value_commit(desc, value);
+
+ return gpiod_set_raw_value_commit(desc, value);
}
/**
@@ -3585,12 +3645,12 @@ static void gpiod_set_value_nocheck(struct gpio_desc *desc, int value)
* This function can be called from contexts where we cannot sleep, and will
* complain if the GPIO chip functions potentially sleep.
*/
-void gpiod_set_value(struct gpio_desc *desc, int value)
+int gpiod_set_value(struct gpio_desc *desc, int value)
{
- VALIDATE_DESC_VOID(desc);
+ VALIDATE_DESC(desc);
/* Should be using gpiod_set_value_cansleep() */
WARN_ON(desc->gdev->can_sleep);
- gpiod_set_value_nocheck(desc, value);
+ return gpiod_set_value_nocheck(desc, value);
}
EXPORT_SYMBOL_GPL(gpiod_set_value);
@@ -4003,11 +4063,11 @@ EXPORT_SYMBOL_GPL(gpiod_get_array_value_cansleep);
*
* This function is to be called from contexts that can sleep.
*/
-void gpiod_set_raw_value_cansleep(struct gpio_desc *desc, int value)
+int gpiod_set_raw_value_cansleep(struct gpio_desc *desc, int value)
{
might_sleep();
- VALIDATE_DESC_VOID(desc);
- gpiod_set_raw_value_commit(desc, value);
+ VALIDATE_DESC(desc);
+ return gpiod_set_raw_value_commit(desc, value);
}
EXPORT_SYMBOL_GPL(gpiod_set_raw_value_cansleep);
@@ -4021,11 +4081,11 @@ EXPORT_SYMBOL_GPL(gpiod_set_raw_value_cansleep);
*
* This function is to be called from contexts that can sleep.
*/
-void gpiod_set_value_cansleep(struct gpio_desc *desc, int value)
+int gpiod_set_value_cansleep(struct gpio_desc *desc, int value)
{
might_sleep();
- VALIDATE_DESC_VOID(desc);
- gpiod_set_value_nocheck(desc, value);
+ VALIDATE_DESC(desc);
+ return gpiod_set_value_nocheck(desc, value);
}
EXPORT_SYMBOL_GPL(gpiod_set_value_cansleep);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index b667da0ec68a..06b80396c6c9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -409,6 +409,7 @@ bool amdgpu_get_bios(struct amdgpu_device *adev);
bool amdgpu_read_bios(struct amdgpu_device *adev);
bool amdgpu_soc15_read_bios_from_rom(struct amdgpu_device *adev,
u8 *bios, u32 length_bytes);
+void amdgpu_bios_release(struct amdgpu_device *adev);
/*
* Clocks
*/
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index a89172a696e4..1a92f04c63ac 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -805,7 +805,7 @@ int amdgpu_amdkfd_send_close_event_drain_irq(struct amdgpu_device *adev,
int amdgpu_amdkfd_check_and_lock_kfd(struct amdgpu_device *adev)
{
- return kgd2kfd_check_and_lock_kfd();
+ return kgd2kfd_check_and_lock_kfd(adev->kfd.dev);
}
void amdgpu_amdkfd_unlock_kfd(struct amdgpu_device *adev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
index d7de6fe40494..d79c431de223 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
@@ -423,7 +423,7 @@ int kgd2kfd_post_reset(struct kfd_dev *kfd);
void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry);
void kgd2kfd_set_sram_ecc_flag(struct kfd_dev *kfd);
void kgd2kfd_smi_event_throttle(struct kfd_dev *kfd, uint64_t throttle_bitmask);
-int kgd2kfd_check_and_lock_kfd(void);
+int kgd2kfd_check_and_lock_kfd(struct kfd_dev *kfd);
void kgd2kfd_unlock_kfd(void);
int kgd2kfd_start_sched(struct kfd_dev *kfd, uint32_t node_id);
int kgd2kfd_stop_sched(struct kfd_dev *kfd, uint32_t node_id);
@@ -489,7 +489,7 @@ void kgd2kfd_smi_event_throttle(struct kfd_dev *kfd, uint64_t throttle_bitmask)
{
}
-static inline int kgd2kfd_check_and_lock_kfd(void)
+static inline int kgd2kfd_check_and_lock_kfd(struct kfd_dev *kfd)
{
return 0;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c
index 0c8975ac5af9..9254b85d4e70 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c
@@ -36,11 +36,19 @@
#include "atombios_encoders.h"
#include "bif/bif_4_1_d.h"
-static void amdgpu_atombios_lookup_i2c_gpio_quirks(struct amdgpu_device *adev,
- ATOM_GPIO_I2C_ASSIGMENT *gpio,
- u8 index)
+/* VBIOS-reported table size is unchecked against the image; cap iterations and
+ * adev->i2c_bus[] indexing to AMDGPU_MAX_I2C_BUS.
+ */
+static int amdgpu_atombios_gpio_i2c_num_entries(uint16_t size)
{
+ u32 bytes;
+
+ if (size < sizeof(ATOM_COMMON_TABLE_HEADER))
+ return 0;
+ bytes = size - sizeof(ATOM_COMMON_TABLE_HEADER);
+ return (int)min_t(u32, bytes / sizeof(ATOM_GPIO_I2C_ASSIGMENT),
+ AMDGPU_MAX_I2C_BUS);
}
static struct amdgpu_i2c_bus_rec amdgpu_atombios_get_bus_rec_for_i2c_gpio(ATOM_GPIO_I2C_ASSIGMENT *gpio)
@@ -103,14 +111,10 @@ struct amdgpu_i2c_bus_rec amdgpu_atombios_lookup_i2c_gpio(struct amdgpu_device *
if (amdgpu_atom_parse_data_header(ctx, index, &size, NULL, NULL, &data_offset)) {
i2c_info = (struct _ATOM_GPIO_I2C_INFO *)(ctx->bios + data_offset);
- num_indices = (size - sizeof(ATOM_COMMON_TABLE_HEADER)) /
- sizeof(ATOM_GPIO_I2C_ASSIGMENT);
+ num_indices = amdgpu_atombios_gpio_i2c_num_entries(size);
gpio = &i2c_info->asGPIO_Info[0];
for (i = 0; i < num_indices; i++) {
-
- amdgpu_atombios_lookup_i2c_gpio_quirks(adev, gpio, i);
-
if (gpio->sucI2cId.ucAccess == id) {
i2c = amdgpu_atombios_get_bus_rec_for_i2c_gpio(gpio);
break;
@@ -137,13 +141,10 @@ void amdgpu_atombios_i2c_init(struct amdgpu_device *adev)
if (amdgpu_atom_parse_data_header(ctx, index, &size, NULL, NULL, &data_offset)) {
i2c_info = (struct _ATOM_GPIO_I2C_INFO *)(ctx->bios + data_offset);
- num_indices = (size - sizeof(ATOM_COMMON_TABLE_HEADER)) /
- sizeof(ATOM_GPIO_I2C_ASSIGMENT);
+ num_indices = amdgpu_atombios_gpio_i2c_num_entries(size);
gpio = &i2c_info->asGPIO_Info[0];
for (i = 0; i < num_indices; i++) {
- amdgpu_atombios_lookup_i2c_gpio_quirks(adev, gpio, i);
-
i2c = amdgpu_atombios_get_bus_rec_for_i2c_gpio(gpio);
if (i2c.valid) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
index e66543089824..dec312b0f3b0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
@@ -48,41 +48,50 @@
/* Check if current bios is an ATOM BIOS.
* Return true if it is ATOM BIOS. Otherwise, return false.
*/
-static bool check_atom_bios(uint8_t *bios, size_t size)
+static bool check_atom_bios(struct amdgpu_device *adev, size_t size)
{
uint16_t tmp, bios_header_start;
+ uint8_t *bios = adev->bios;
if (!bios || size < 0x49) {
- DRM_INFO("vbios mem is null or mem size is wrong\n");
+ dev_dbg(adev->dev, "VBIOS mem is null or mem size is wrong\n");
return false;
}
if (!AMD_IS_VALID_VBIOS(bios)) {
- DRM_INFO("BIOS signature incorrect %x %x\n", bios[0], bios[1]);
+ dev_dbg(adev->dev, "VBIOS signature incorrect %x %x\n", bios[0],
+ bios[1]);
return false;
}
bios_header_start = bios[0x48] | (bios[0x49] << 8);
if (!bios_header_start) {
- DRM_INFO("Can't locate bios header\n");
+ dev_dbg(adev->dev, "Can't locate VBIOS header\n");
return false;
}
tmp = bios_header_start + 4;
if (size < tmp) {
- DRM_INFO("BIOS header is broken\n");
+ dev_dbg(adev->dev, "VBIOS header is broken\n");
return false;
}
if (!memcmp(bios + tmp, "ATOM", 4) ||
!memcmp(bios + tmp, "MOTA", 4)) {
- DRM_DEBUG("ATOMBIOS detected\n");
+ dev_dbg(adev->dev, "ATOMBIOS detected\n");
return true;
}
return false;
}
+void amdgpu_bios_release(struct amdgpu_device *adev)
+{
+ kfree(adev->bios);
+ adev->bios = NULL;
+ adev->bios_size = 0;
+}
+
/* If you boot an IGP board with a discrete card as the primary,
* the IGP rom is not accessible via the rom bar as the IGP rom is
* part of the system bios. On boot, the system bios puts a
@@ -119,8 +128,8 @@ static bool amdgpu_read_bios_from_vram(struct amdgpu_device *adev)
memcpy_fromio(adev->bios, bios, size);
iounmap(bios);
- if (!check_atom_bios(adev->bios, size)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, size)) {
+ amdgpu_bios_release(adev);
return false;
}
@@ -147,8 +156,8 @@ bool amdgpu_read_bios(struct amdgpu_device *adev)
memcpy_fromio(adev->bios, bios, size);
pci_unmap_rom(adev->pdev, bios);
- if (!check_atom_bios(adev->bios, size)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, size)) {
+ amdgpu_bios_release(adev);
return false;
}
@@ -187,8 +196,8 @@ static bool amdgpu_read_bios_from_rom(struct amdgpu_device *adev)
/* read complete BIOS */
amdgpu_asic_read_bios_from_rom(adev, adev->bios, len);
- if (!check_atom_bios(adev->bios, len)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, len)) {
+ amdgpu_bios_release(adev);
return false;
}
@@ -217,14 +226,15 @@ static bool amdgpu_read_platform_bios(struct amdgpu_device *adev)
memcpy_fromio(adev->bios, bios, romlen);
iounmap(bios);
- if (!check_atom_bios(adev->bios, romlen))
+ if (!check_atom_bios(adev, romlen))
goto free_bios;
adev->bios_size = romlen;
return true;
free_bios:
- kfree(adev->bios);
+ amdgpu_bios_release(adev);
+
return false;
}
@@ -325,8 +335,8 @@ static bool amdgpu_atrm_get_bios(struct amdgpu_device *adev)
break;
}
- if (!check_atom_bios(adev->bios, size)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, size)) {
+ amdgpu_bios_release(adev);
return false;
}
adev->bios_size = size;
@@ -391,13 +401,14 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
acpi_size tbl_size;
UEFI_ACPI_VFCT *vfct;
unsigned int offset;
+ bool r = false;
if (!ACPI_SUCCESS(acpi_get_table("VFCT", 1, &hdr)))
return false;
tbl_size = hdr->length;
if (tbl_size < sizeof(UEFI_ACPI_VFCT)) {
dev_info(adev->dev, "ACPI VFCT table present but broken (too short #1),skipping\n");
- return false;
+ goto out;
}
vfct = (UEFI_ACPI_VFCT *)hdr;
@@ -410,13 +421,13 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
offset += sizeof(VFCT_IMAGE_HEADER);
if (offset > tbl_size) {
dev_info(adev->dev, "ACPI VFCT image header truncated,skipping\n");
- return false;
+ goto out;
}
offset += vhdr->ImageLength;
if (offset > tbl_size) {
dev_info(adev->dev, "ACPI VFCT image truncated,skipping\n");
- return false;
+ goto out;
}
if (vhdr->ImageLength &&
@@ -425,17 +436,21 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
vhdr->ImageLength,
GFP_KERNEL);
- if (!check_atom_bios(adev->bios, vhdr->ImageLength)) {
- kfree(adev->bios);
- return false;
+ if (!check_atom_bios(adev, vhdr->ImageLength)) {
+ amdgpu_bios_release(adev);
+ goto out;
}
adev->bios_size = vhdr->ImageLength;
- return true;
+ r = true;
+ goto out;
}
}
dev_info(adev->dev, "ACPI VFCT table present but broken (too short #2),skipping\n");
- return false;
+
+out:
+ acpi_put_table(hdr);
+ return r;
}
#else
static inline bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cd05ddacd811..13363330820a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2849,6 +2849,12 @@ static int amdgpu_device_fw_loading(struct amdgpu_device *adev)
static int amdgpu_device_init_schedulers(struct amdgpu_device *adev)
{
+ struct drm_sched_init_args args = {
+ .ops = &amdgpu_sched_ops,
+ .num_rqs = DRM_SCHED_PRIORITY_COUNT,
+ .timeout_wq = adev->reset_domain->wq,
+ .dev = adev->dev,
+ };
long timeout;
int r, i;
@@ -2874,12 +2880,12 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev)
break;
}
- r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- ring->num_hw_submission, 0,
- timeout, adev->reset_domain->wq,
- ring->sched_score, ring->name,
- adev->dev);
+ args.timeout = timeout;
+ args.credit_limit = ring->num_hw_submission;
+ args.score = ring->sched_score;
+ args.name = ring->name;
+
+ r = drm_sched_init(&ring->sched, &args);
if (r) {
DRM_ERROR("Failed to create scheduler on ring %s.\n",
ring->name);
@@ -4512,8 +4518,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
goto failed;
}
/* init i2c buses */
- if (!amdgpu_device_has_dc_support(adev))
- amdgpu_atombios_i2c_init(adev);
+ amdgpu_i2c_init(adev);
}
}
@@ -4787,14 +4792,12 @@ void amdgpu_device_fini_sw(struct amdgpu_device *adev)
amdgpu_reset_fini(adev);
/* free i2c buses */
- if (!amdgpu_device_has_dc_support(adev))
- amdgpu_i2c_fini(adev);
+ amdgpu_i2c_fini(adev);
if (amdgpu_emu_mode != 1)
amdgpu_atombios_fini(adev);
- kfree(adev->bios);
- adev->bios = NULL;
+ amdgpu_bios_release(adev);
kfree(adev->fru_info);
adev->fru_info = NULL;
@@ -5499,6 +5502,31 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
amdgpu_fence_driver_force_completion(ring, fence);
}
+ /*
+ * MES scheduler rings have no drm scheduler, so they are missed by the
+ * loop above. Realign their polling fence too (one per XCC), otherwise the
+ * first post-reset submission polls forever on a stale seq. sched.ready is
+ * only set while the driver owns the ring.
+ */
+ for (i = 0; i < AMDGPU_MAX_MES_PIPES; i++) {
+ struct amdgpu_ring *mes_ring = &adev->mes.ring[i];
+
+ if (mes_ring->fence_drv.initialized && mes_ring->sched.ready)
+ amdgpu_fence_driver_force_completion(mes_ring, fence);
+ }
+
+ /*
+ * KIQ rings are polling-fence/no_scheduler like MES, so realign their
+ * fence too (one ring per XCC), otherwise the first post-reset KIQ
+ * submission polls forever on a stale seq.
+ */
+ for (i = 0; i < AMDGPU_MAX_GC_INSTANCES; i++) {
+ struct amdgpu_ring *kiq_ring = &adev->gfx.kiq[i].ring;
+
+ if (kiq_ring->fence_drv.initialized && kiq_ring->sched.ready)
+ amdgpu_fence_driver_force_completion(kiq_ring, fence);
+ }
+
amdgpu_fence_driver_isr_toggle(adev, false);
if (job && job->vm)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
index 171603c4ee6c..66e7f899de2a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c
@@ -922,7 +922,7 @@ int amdgpu_gfx_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *r
if (r)
return r;
- if (adev->gfx.cp_ecc_error_irq.funcs) {
+ if (!amdgpu_sriov_vf(adev) && adev->gfx.cp_ecc_error_irq.funcs) {
r = amdgpu_irq_get(adev, &adev->gfx.cp_ecc_error_irq, 0);
if (r)
goto late_fini;
@@ -937,6 +937,21 @@ int amdgpu_gfx_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *r
return r;
}
+void amdgpu_gfx_ras_suspend(struct amdgpu_device *adev,
+ struct ras_common_if *ras_block)
+{
+ if (!amdgpu_sriov_vf(adev) && adev->gfx.cp_ecc_error_irq.funcs)
+ amdgpu_irq_put(adev, &adev->gfx.cp_ecc_error_irq, 0);
+}
+
+void amdgpu_gfx_ras_fini(struct amdgpu_device *adev,
+ struct ras_common_if *ras_block)
+{
+ if (!amdgpu_sriov_vf(adev) && adev->gfx.cp_ecc_error_irq.funcs)
+ amdgpu_irq_put(adev, &adev->gfx.cp_ecc_error_irq, 0);
+ amdgpu_ras_block_late_fini(adev, ras_block);
+}
+
int amdgpu_gfx_ras_sw_init(struct amdgpu_device *adev)
{
int err = 0;
@@ -965,6 +980,12 @@ int amdgpu_gfx_ras_sw_init(struct amdgpu_device *adev)
if (!ras->ras_block.ras_late_init)
ras->ras_block.ras_late_init = amdgpu_gfx_ras_late_init;
+ if (!ras->ras_block.ras_suspend)
+ ras->ras_block.ras_suspend = amdgpu_gfx_ras_suspend;
+
+ if (!ras->ras_block.ras_fini)
+ ras->ras_block.ras_fini = amdgpu_gfx_ras_fini;
+
/* If not defined special ras_cb function, use default ras_cb */
if (!ras->ras_block.ras_cb)
ras->ras_block.ras_cb = amdgpu_gfx_process_ras_data_cb;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
index 5644e10a86a9..e64dcfdbfadd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h
@@ -547,7 +547,8 @@ bool amdgpu_gfx_is_me_queue_enabled(struct amdgpu_device *adev, int me,
void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable);
int amdgpu_get_gfx_off_status(struct amdgpu_device *adev, uint32_t *value);
int amdgpu_gfx_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block);
-void amdgpu_gfx_ras_fini(struct amdgpu_device *adev);
+void amdgpu_gfx_ras_suspend(struct amdgpu_device *adev, struct ras_common_if *ras_block);
+void amdgpu_gfx_ras_fini(struct amdgpu_device *adev, struct ras_common_if *ras_block);
int amdgpu_get_gfx_off_entrycount(struct amdgpu_device *adev, u64 *value);
int amdgpu_get_gfx_off_residency(struct amdgpu_device *adev, u32 *residency);
int amdgpu_set_gfx_off_residency(struct amdgpu_device *adev, bool value);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
index e36fede7f74c..52882d88c290 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
@@ -51,8 +51,6 @@
#include "amdgpu_amdkfd.h"
#include "amdgpu_hmm.h"
-#define MAX_WALK_BYTE (2UL << 30)
-
/**
* amdgpu_hmm_invalidate_gfx - callback to notify about mm change
*
@@ -170,12 +168,14 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
void *owner, struct page **pages,
struct hmm_range **phmm_range)
{
+ const u64 max_bytes = SZ_2G;
+
struct hmm_range *hmm_range;
- unsigned long end;
unsigned long timeout;
- unsigned long i;
unsigned long *pfns;
- int r = 0;
+ unsigned long end;
+ unsigned long i;
+ int r;
hmm_range = kzalloc(sizeof(*hmm_range), GFP_KERNEL);
if (unlikely(!hmm_range))
@@ -196,8 +196,9 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
end = start + npages * PAGE_SIZE;
hmm_range->dev_private_owner = owner;
+ hmm_range->notifier_seq = mmu_interval_read_begin(notifier);
do {
- hmm_range->end = min(hmm_range->start + MAX_WALK_BYTE, end);
+ hmm_range->end = min(hmm_range->start + max_bytes, end);
pr_debug("hmm range: start = 0x%lx, end = 0x%lx",
hmm_range->start, hmm_range->end);
@@ -205,7 +206,6 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
retry:
- hmm_range->notifier_seq = mmu_interval_read_begin(notifier);
r = hmm_range_fault(hmm_range);
if (unlikely(r)) {
if (r == -EBUSY && !time_after(jiffies, timeout))
@@ -215,7 +215,7 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
if (hmm_range->end == end)
break;
- hmm_range->hmm_pfns += MAX_WALK_BYTE >> PAGE_SHIFT;
+ hmm_range->hmm_pfns += max_bytes >> PAGE_SHIFT;
hmm_range->start = hmm_range->end;
} while (hmm_range->end < end);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c
index 00d6211e0fbf..72f73a186dd9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c
@@ -225,13 +225,12 @@ void amdgpu_i2c_destroy(struct amdgpu_i2c_chan *i2c)
kfree(i2c);
}
-/* Add the default buses */
void amdgpu_i2c_init(struct amdgpu_device *adev)
{
- if (amdgpu_hw_i2c)
- DRM_INFO("hw_i2c forced on, you may experience display detection problems!\n");
-
- amdgpu_atombios_i2c_init(adev);
+ if (!adev->is_atom_fw) {
+ if (!amdgpu_device_has_dc_support(adev))
+ amdgpu_atombios_i2c_init(adev);
+ }
}
/* remove all the buses */
@@ -247,22 +246,6 @@ void amdgpu_i2c_fini(struct amdgpu_device *adev)
}
}
-/* Add additional buses */
-void amdgpu_i2c_add(struct amdgpu_device *adev,
- const struct amdgpu_i2c_bus_rec *rec,
- const char *name)
-{
- struct drm_device *dev = adev_to_drm(adev);
- int i;
-
- for (i = 0; i < AMDGPU_MAX_I2C_BUS; i++) {
- if (!adev->i2c_bus[i]) {
- adev->i2c_bus[i] = amdgpu_i2c_create(dev, rec, name);
- return;
- }
- }
-}
-
/* looks up bus based on id */
struct amdgpu_i2c_chan *
amdgpu_i2c_lookup(struct amdgpu_device *adev,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h
index 63c2ff7499e1..1d3d3806e0dd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h
@@ -30,9 +30,6 @@ struct amdgpu_i2c_chan *amdgpu_i2c_create(struct drm_device *dev,
void amdgpu_i2c_destroy(struct amdgpu_i2c_chan *i2c);
void amdgpu_i2c_init(struct amdgpu_device *adev);
void amdgpu_i2c_fini(struct amdgpu_device *adev);
-void amdgpu_i2c_add(struct amdgpu_device *adev,
- const struct amdgpu_i2c_bus_rec *rec,
- const char *name);
struct amdgpu_i2c_chan *
amdgpu_i2c_lookup(struct amdgpu_device *adev,
const struct amdgpu_i2c_bus_rec *i2c_bus);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c
index bf563904b3fa..1784e484798b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c
@@ -90,6 +90,9 @@ struct amdgpu_ras_block_list {
struct list_head node;
struct amdgpu_ras_block_object *ras_obj;
+
+ /* set by ras_late_init, cleared by ras_suspend/ras_fini */
+ bool active;
};
const char *get_ras_block_str(struct ras_common_if *ras_block)
@@ -3874,10 +3877,23 @@ void amdgpu_ras_resume(struct amdgpu_device *adev)
void amdgpu_ras_suspend(struct amdgpu_device *adev)
{
struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
+ struct amdgpu_ras_block_list *node;
+ struct amdgpu_ras_block_object *obj;
if (!adev->ras_enabled || !con)
return;
+ /* run per-block ras_suspend before tearing down the RAS context */
+ list_for_each_entry(node, &adev->ras_list, node) {
+ if (!node->active)
+ continue;
+
+ obj = node->ras_obj;
+ if (obj && obj->ras_suspend)
+ obj->ras_suspend(adev, &obj->ras_comm);
+ node->active = false;
+ }
+
amdgpu_ras_disable_all_features(adev, 0);
/* Make sure all ras objects are disabled. */
if (AMDGPU_RAS_GET_FEATURES(con->features))
@@ -3931,8 +3947,15 @@ int amdgpu_ras_late_init(struct amdgpu_device *adev)
obj->ras_comm.name, r);
return r;
}
- } else
- amdgpu_ras_block_late_init_default(adev, &obj->ras_comm);
+ } else {
+ r = amdgpu_ras_block_late_init_default(adev, &obj->ras_comm);
+ if (r) {
+ dev_err(adev->dev, "%s failed to execute ras_block_late_init_default! ret:%d\n",
+ obj->ras_comm.name, r);
+ return r;
+ }
+ }
+ node->active = true;
}
return 0;
@@ -3966,11 +3989,12 @@ int amdgpu_ras_fini(struct amdgpu_device *adev)
list_for_each_entry_safe(ras_node, tmp, &adev->ras_list, node) {
if (ras_node->ras_obj) {
obj = ras_node->ras_obj;
- if (amdgpu_ras_is_supported(adev, obj->ras_comm.block) &&
- obj->ras_fini)
+ /* fall back to default cleanup if ras_suspend already ran */
+ if (ras_node->active && obj->ras_fini)
obj->ras_fini(adev, &obj->ras_comm);
else
amdgpu_ras_block_late_fini_default(adev, &obj->ras_comm);
+ ras_node->active = false;
}
/* Clear ras blocks from ras_list and free ras block list node */
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h
index 7e7521fedafc..a705b1157d8e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h
@@ -706,6 +706,7 @@ struct amdgpu_ras_block_object {
int (*ras_block_match)(struct amdgpu_ras_block_object *block_obj,
enum amdgpu_ras_block block, uint32_t sub_block_index);
int (*ras_late_init)(struct amdgpu_device *adev, struct ras_common_if *ras_block);
+ void (*ras_suspend)(struct amdgpu_device *adev, struct ras_common_if *ras_block);
void (*ras_fini)(struct amdgpu_device *adev, struct ras_common_if *ras_block);
ras_ih_cb ras_cb;
const struct amdgpu_ras_block_hw_ops *hw_ops;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 60c5e8803f60..93da39a5ce49 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -174,6 +174,8 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,
* @tmz: if we should setup a TMZ enabled mapping
* @size: in number of bytes to map, out number of bytes mapped
* @addr: resulting address inside the MC address space
+ * @vm_needs_flush: out, set true if a GART window was programmed (VMID 0 flush
+ * needed) or false for a direct address
*
* Setup one of the GART windows to access a specific piece of memory or return
* the physical address for local memory.
@@ -182,7 +184,8 @@ static int amdgpu_ttm_map_buffer(struct ttm_buffer_object *bo,
struct ttm_resource *mem,
struct amdgpu_res_cursor *mm_cur,
unsigned int window, struct amdgpu_ring *ring,
- bool tmz, uint64_t *size, uint64_t *addr)
+ bool tmz, uint64_t *size, uint64_t *addr,
+ bool *vm_needs_flush)
{
struct amdgpu_device *adev = ring->adev;
unsigned int offset, num_pages, num_dw, num_bytes;
@@ -203,9 +206,12 @@ static int amdgpu_ttm_map_buffer(struct ttm_buffer_object *bo,
if (!tmz && mem->start != AMDGPU_BO_INVALID_OFFSET) {
*addr = amdgpu_ttm_domain_start(adev, mem->mem_type) +
mm_cur->start;
+ *vm_needs_flush = false;
return 0;
}
+ /* A GART window is programmed below, so its VMID 0 TLB needs a flush */
+ *vm_needs_flush = true;
/*
* If start begins at an offset inside the page, then adjust the size
@@ -314,6 +320,7 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
while (src_mm.remaining) {
uint64_t from, to, cur_size, tiling_flags;
uint32_t num_type, data_format, max_com, write_compress_disable;
+ bool src_vm_flush, dst_vm_flush;
struct dma_fence *next;
/* Never copy more than 256MiB at once to avoid a timeout */
@@ -321,12 +328,12 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
/* Map src to window 0 and dst to window 1. */
r = amdgpu_ttm_map_buffer(src->bo, src->mem, &src_mm,
- 0, ring, tmz, &cur_size, &from);
+ 0, ring, tmz, &cur_size, &from, &src_vm_flush);
if (r)
goto error;
r = amdgpu_ttm_map_buffer(dst->bo, dst->mem, &dst_mm,
- 1, ring, tmz, &cur_size, &to);
+ 1, ring, tmz, &cur_size, &to, &dst_vm_flush);
if (r)
goto error;
@@ -354,7 +361,7 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
}
r = amdgpu_copy_buffer(ring, from, to, cur_size, resv,
- &next, false, true, copy_flags);
+ &next, false, src_vm_flush || dst_vm_flush, copy_flags);
if (r)
goto error;
@@ -2311,6 +2318,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
struct amdgpu_res_cursor cursor;
+ bool vm_needs_flush = false;
u64 addr;
int r = 0;
@@ -2338,12 +2346,12 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
size = min(cursor.size, 256ULL << 20);
r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
- 1, ring, false, &size, &addr);
+ 1, ring, false, &size, &addr, &vm_needs_flush);
if (r)
goto err;
r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
- &next, true, true,
+ &next, vm_needs_flush, true,
AMDGPU_KERNEL_JOB_ID_TTM_CLEAR_BUFFER);
if (r)
goto err;
@@ -2370,6 +2378,7 @@ int amdgpu_fill_buffer(struct amdgpu_bo *bo,
struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
struct dma_fence *fence = NULL;
struct amdgpu_res_cursor dst;
+ bool vm_needs_flush = false;
int r;
if (!adev->mman.buffer_funcs_enabled) {
@@ -2388,12 +2397,12 @@ int amdgpu_fill_buffer(struct amdgpu_bo *bo,
cur_size = min(dst.size, 256ULL << 20);
r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &dst,
- 1, ring, false, &cur_size, &to);
+ 1, ring, false, &cur_size, &to, &vm_needs_flush);
if (r)
goto error;
r = amdgpu_ttm_fill_mem(ring, src_data, to, cur_size, resv,
- &next, true, delayed, k_job_id);
+ &next, vm_needs_flush, delayed, k_job_id);
if (r)
goto error;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h b/drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h
index 6e9eeaeb3de1..b4f9c2f4e92c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h
@@ -28,17 +28,21 @@
#define AMD_SRIOV_MSG_VBIOS_SIZE_KB 64
#define AMD_SRIOV_MSG_DATAEXCHANGE_OFFSET_KB AMD_SRIOV_MSG_VBIOS_SIZE_KB
#define AMD_SRIOV_MSG_DATAEXCHANGE_SIZE_KB 4
-
+#define AMD_SRIOV_MSG_TMR_OFFSET_KB 2048
+#define AMD_SRIOV_MSG_BAD_PAGE_SIZE_KB 2
+#define AMD_SRIOV_RAS_TELEMETRY_SIZE_KB 64
/*
* layout
- * 0 64KB 65KB 66KB
- * | VBIOS | PF2VF | VF2PF | Bad Page | ...
- * | 64KB | 1KB | 1KB |
+ * 0 64KB 65KB 66KB 68KB 132KB
+ * | VBIOS | PF2VF | VF2PF | Bad Page | RAS Telemetry Region | ...
+ * | 64KB | 1KB | 1KB | 2KB | 64KB | ...
*/
+
#define AMD_SRIOV_MSG_SIZE_KB 1
#define AMD_SRIOV_MSG_PF2VF_OFFSET_KB AMD_SRIOV_MSG_DATAEXCHANGE_OFFSET_KB
#define AMD_SRIOV_MSG_VF2PF_OFFSET_KB (AMD_SRIOV_MSG_PF2VF_OFFSET_KB + AMD_SRIOV_MSG_SIZE_KB)
#define AMD_SRIOV_MSG_BAD_PAGE_OFFSET_KB (AMD_SRIOV_MSG_VF2PF_OFFSET_KB + AMD_SRIOV_MSG_SIZE_KB)
+#define AMD_SRIOV_MSG_RAS_TELEMETRY_OFFSET_KB (AMD_SRIOV_MSG_BAD_PAGE_OFFSET_KB + AMD_SRIOV_MSG_BAD_PAGE_SIZE_KB)
/*
* PF2VF history log:
@@ -86,30 +90,59 @@ enum amd_sriov_ucode_engine_id {
union amd_sriov_msg_feature_flags {
struct {
- uint32_t error_log_collect : 1;
- uint32_t host_load_ucodes : 1;
- uint32_t host_flr_vramlost : 1;
- uint32_t mm_bw_management : 1;
- uint32_t pp_one_vf_mode : 1;
- uint32_t reg_indirect_acc : 1;
- uint32_t av1_support : 1;
- uint32_t vcn_rb_decouple : 1;
- uint32_t mes_info_enable : 1;
- uint32_t reserved : 23;
+ uint32_t error_log_collect : 1;
+ uint32_t host_load_ucodes : 1;
+ uint32_t host_flr_vramlost : 1;
+ uint32_t mm_bw_management : 1;
+ uint32_t pp_one_vf_mode : 1;
+ uint32_t reg_indirect_acc : 1;
+ uint32_t av1_support : 1;
+ uint32_t vcn_rb_decouple : 1;
+ uint32_t mes_info_dump_enable : 1;
+ uint32_t ras_caps : 1;
+ uint32_t ras_telemetry : 1;
+ uint32_t reserved : 21;
} flags;
uint32_t all;
};
union amd_sriov_reg_access_flags {
struct {
- uint32_t vf_reg_access_ih : 1;
- uint32_t vf_reg_access_mmhub : 1;
- uint32_t vf_reg_access_gc : 1;
- uint32_t reserved : 29;
+ uint32_t vf_reg_access_ih : 1;
+ uint32_t vf_reg_access_mmhub : 1;
+ uint32_t vf_reg_access_gc : 1;
+ uint32_t reserved : 29;
} flags;
uint32_t all;
};
+union amd_sriov_ras_caps {
+ struct {
+ uint64_t block_umc : 1;
+ uint64_t block_sdma : 1;
+ uint64_t block_gfx : 1;
+ uint64_t block_mmhub : 1;
+ uint64_t block_athub : 1;
+ uint64_t block_pcie_bif : 1;
+ uint64_t block_hdp : 1;
+ uint64_t block_xgmi_wafl : 1;
+ uint64_t block_df : 1;
+ uint64_t block_smn : 1;
+ uint64_t block_sem : 1;
+ uint64_t block_mp0 : 1;
+ uint64_t block_mp1 : 1;
+ uint64_t block_fuse : 1;
+ uint64_t block_mca : 1;
+ uint64_t block_vcn : 1;
+ uint64_t block_jpeg : 1;
+ uint64_t block_ih : 1;
+ uint64_t block_mpio : 1;
+ uint64_t poison_propogation_mode : 1;
+ uint64_t reserved : 44;
+ } bits;
+ uint64_t all;
+};
+
union amd_sriov_msg_os_info {
struct {
uint32_t windows : 1;
@@ -158,7 +191,7 @@ struct amd_sriov_msg_pf2vf_info_header {
uint32_t reserved[2];
};
-#define AMD_SRIOV_MSG_PF2VF_INFO_FILLED_SIZE (49)
+#define AMD_SRIOV_MSG_PF2VF_INFO_FILLED_SIZE (55)
struct amd_sriov_msg_pf2vf_info {
/* header contains size and version */
struct amd_sriov_msg_pf2vf_info_header header;
@@ -211,6 +244,12 @@ struct amd_sriov_msg_pf2vf_info {
uint32_t pcie_atomic_ops_support_flags;
/* Portion of GPU memory occupied by VF. MAX value is 65535, but set to uint32_t to maintain alignment with reserved size */
uint32_t gpu_capacity;
+ /* vf bdf on host pci tree for debug only */
+ uint32_t bdf_on_host;
+ uint32_t more_bp; //Reserved for future use.
+ union amd_sriov_ras_caps ras_en_caps;
+ union amd_sriov_ras_caps ras_telemetry_en_caps;
+
/* reserved */
uint32_t reserved[256 - AMD_SRIOV_MSG_PF2VF_INFO_FILLED_SIZE];
} __packed;
@@ -283,8 +322,12 @@ enum amd_sriov_mailbox_request_message {
MB_REQ_MSG_REL_GPU_FINI_ACCESS,
MB_REQ_MSG_REQ_GPU_RESET_ACCESS,
MB_REQ_MSG_REQ_GPU_INIT_DATA,
+ MB_REQ_MSG_PSP_VF_CMD_RELAY,
MB_REQ_MSG_LOG_VF_ERROR = 200,
+ MB_REQ_MSG_READY_TO_RESET = 201,
+ MB_REQ_MSG_RAS_POISON = 202,
+ MB_REQ_RAS_ERROR_COUNT = 203,
};
/* mailbox message send from host to guest */
@@ -297,10 +340,60 @@ enum amd_sriov_mailbox_response_message {
MB_RES_MSG_FAIL,
MB_RES_MSG_QUERY_ALIVE,
MB_RES_MSG_GPU_INIT_DATA_READY,
+ MB_RES_MSG_RAS_ERROR_COUNT_READY = 11,
MB_RES_MSG_TEXT_MESSAGE = 255
};
+enum amd_sriov_ras_telemetry_gpu_block {
+ RAS_TELEMETRY_GPU_BLOCK_UMC = 0,
+ RAS_TELEMETRY_GPU_BLOCK_SDMA = 1,
+ RAS_TELEMETRY_GPU_BLOCK_GFX = 2,
+ RAS_TELEMETRY_GPU_BLOCK_MMHUB = 3,
+ RAS_TELEMETRY_GPU_BLOCK_ATHUB = 4,
+ RAS_TELEMETRY_GPU_BLOCK_PCIE_BIF = 5,
+ RAS_TELEMETRY_GPU_BLOCK_HDP = 6,
+ RAS_TELEMETRY_GPU_BLOCK_XGMI_WAFL = 7,
+ RAS_TELEMETRY_GPU_BLOCK_DF = 8,
+ RAS_TELEMETRY_GPU_BLOCK_SMN = 9,
+ RAS_TELEMETRY_GPU_BLOCK_SEM = 10,
+ RAS_TELEMETRY_GPU_BLOCK_MP0 = 11,
+ RAS_TELEMETRY_GPU_BLOCK_MP1 = 12,
+ RAS_TELEMETRY_GPU_BLOCK_FUSE = 13,
+ RAS_TELEMETRY_GPU_BLOCK_MCA = 14,
+ RAS_TELEMETRY_GPU_BLOCK_VCN = 15,
+ RAS_TELEMETRY_GPU_BLOCK_JPEG = 16,
+ RAS_TELEMETRY_GPU_BLOCK_IH = 17,
+ RAS_TELEMETRY_GPU_BLOCK_MPIO = 18,
+ RAS_TELEMETRY_GPU_BLOCK_COUNT = 19,
+};
+
+struct amd_sriov_ras_telemetry_header {
+ uint32_t checksum;
+ uint32_t used_size;
+ uint32_t reserved[2];
+};
+
+struct amd_sriov_ras_telemetry_error_count {
+ struct {
+ uint32_t ce_count;
+ uint32_t ue_count;
+ uint32_t de_count;
+ uint32_t ce_overflow_count;
+ uint32_t ue_overflow_count;
+ uint32_t de_overflow_count;
+ uint32_t reserved[6];
+ } block[RAS_TELEMETRY_GPU_BLOCK_COUNT];
+};
+
+struct amdsriov_ras_telemetry {
+ struct amd_sriov_ras_telemetry_header header;
+
+ union {
+ struct amd_sriov_ras_telemetry_error_count error_count;
+ } body;
+};
+
/* version data stored in MAILBOX_MSGBUF_RCV_DW1 for future expansion */
enum amd_sriov_gpu_init_data_version {
GPU_INIT_DATA_READY_V1 = 1,
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
index 4d115538a956..a076a314dc67 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
@@ -3398,10 +3398,19 @@ static int gfx_v12_0_cp_resume(struct amdgpu_device *adev)
gfx_v12_0_cp_gfx_enable(adev, true);
}
- if (adev->enable_mes_kiq && adev->mes.kiq_hw_init)
+ if (adev->enable_mes_kiq && adev->mes.kiq_hw_init) {
r = amdgpu_mes_kiq_hw_init(adev);
- else
+ /*
+ * With MES, GFX KIQ ring is owned by the MES and is never
+ * initialized/used directly by the driver, so it must
+ * not be left flagged as ready. mes_v12_0_hw_init() clears
+ * but clear here if MES init fails
+ */
+ if (r)
+ adev->gfx.kiq[0].ring.sched.ready = false;
+ } else {
r = gfx_v12_0_kiq_resume(adev);
+ }
if (r)
return r;
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
index 06ab40412200..870ec359ed4b 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c
@@ -4018,8 +4018,6 @@ static int gfx_v9_0_hw_fini(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__GFX))
- amdgpu_irq_put(adev, &adev->gfx.cp_ecc_error_irq, 0);
amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);
diff --git a/drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h b/drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h
index 1d099ffb3a5a..9d61d76e1bf9 100644
--- a/drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h
+++ b/drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h
@@ -40,6 +40,7 @@ enum idh_request {
IDH_LOG_VF_ERROR = 200,
IDH_READY_TO_RESET = 201,
IDH_RAS_POISON = 202,
+ IDH_REQ_RAS_ERROR_COUNT = 203,
};
enum idh_event {
@@ -54,6 +55,8 @@ enum idh_event {
IDH_RAS_POISON_READY,
IDH_PF_SOFT_FLR_NOTIFICATION,
IDH_RAS_ERROR_DETECTED,
+ IDH_RAS_ERROR_COUNT_READY = 11,
+
IDH_TEXT_MESSAGE = 255,
};
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c b/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c
index 42edc91f4a78..20ef8835b12b 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c
@@ -402,7 +402,9 @@ static int vcn_v2_5_hw_fini(void *handle)
RREG32_SOC15(VCN, i, mmUVD_STATUS)))
vcn_v2_5_set_powergating_state(adev, AMD_PG_STATE_GATE);
- if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__VCN))
+ /* VF doesn't enable interrupt operations for RAS */
+ if (!amdgpu_sriov_vf(adev) &&
+ amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__VCN))
amdgpu_irq_put(adev, &adev->vcn.inst[i].ras_poison_irq, 0);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
index d86de234765d..e9d43924bcd7 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
@@ -1926,6 +1926,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_ENC,
.align_mask = 0x3f,
.nop = VCN_ENC_CMD_NO_OP,
+ .no_user_fence = true,
.extra_dw = sizeof(struct amdgpu_vcn_rb_metadata),
.get_rptr = vcn_v4_0_unified_ring_get_rptr,
.get_wptr = vcn_v4_0_unified_ring_get_wptr,
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index d2993594c848..f4e40ed231fb 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1403,6 +1403,23 @@ int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
while ((char *)sub_type_hdr + sizeof(struct crat_subtype_generic) <
((char *)crat_image) + image_len) {
+ if (!sub_type_hdr->length ||
+ sub_type_hdr->length < sizeof(struct crat_subtype_generic)) {
+ pr_warn("Invalid CRAT subtype length %u\n",
+ sub_type_hdr->length);
+ ret = -EINVAL;
+ break;
+ }
+
+ /* Validate subtype fits within remaining image */
+ if ((char *)sub_type_hdr + sub_type_hdr->length >
+ (char *)crat_image + image_len) {
+ pr_warn("CRAT subtype length %u exceeds image bounds\n",
+ sub_type_hdr->length);
+ ret = -EINVAL;
+ break;
+ }
+
if (sub_type_hdr->flags & CRAT_SUBTYPE_FLAGS_ENABLED) {
ret = kfd_parse_subtype(sub_type_hdr, device_list);
if (ret)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index b536ec72cffb..fa95bde45f3c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -487,6 +487,7 @@ struct kfd_dev *kgd2kfd_probe(struct amdgpu_device *adev, bool vf)
mutex_init(&kfd->doorbell_mutex);
ida_init(&kfd->doorbell_ida);
+ atomic_set(&kfd->kfd_processes_count, 0);
return kfd;
}
@@ -1430,9 +1431,19 @@ unsigned int kfd_get_num_xgmi_sdma_engines(struct kfd_node *node)
kfd_get_num_sdma_engines(node);
}
-int kgd2kfd_check_and_lock_kfd(void)
+int kgd2kfd_check_and_lock_kfd(struct kfd_dev *kfd)
{
mutex_lock(&kfd_processes_mutex);
+
+ /* kfd_processes_count is per kfd_dev, return -EBUSY without
+ * further check
+ */
+ if (!!atomic_read(&kfd->kfd_processes_count)) {
+ pr_debug("process_wq_release not finished\n");
+ mutex_unlock(&kfd_processes_mutex);
+ return -EBUSY;
+ }
+
if (!hash_empty(kfd_processes_table) || kfd_is_locked()) {
mutex_unlock(&kfd_processes_mutex);
return -EBUSY;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index d03160a463d0..a411ea1a20dc 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -1222,6 +1222,15 @@ static int evict_process_queues_cpsch(struct device_queue_manager *dqm,
}
}
}
+
+ /*
+ * Heavy-weight TLB flush after MES removes queues to ensure
+ * in-flight memory accesses complete before memory is freed/migrated.
+ * HWS does this automatically, MES does not.
+ */
+ if (dqm->dev->kfd->shared_resources.enable_mes)
+ kfd_flush_tlb(pdd);
+
pdd->last_evict_timestamp = get_jiffies_64();
if (!dqm->dev->kfd->shared_resources.enable_mes)
retval = execute_queues_cpsch(dqm,
@@ -3404,8 +3413,11 @@ int suspend_queues(struct kfd_process *p,
if (!per_device_suspended) {
dqm_unlock(dqm);
mutex_unlock(&p->event_mutex);
- if (total_suspended)
+ if (total_suspended) {
amdgpu_amdkfd_debug_mem_fence(dqm->dev->adev);
+ /* Heavy-weight TLB flush after MES suspends queues */
+ kfd_flush_tlb(pdd);
+ }
continue;
}
@@ -3575,6 +3587,12 @@ bool kfd_dqm_is_queue_in_process(struct device_queue_manager *dqm,
dqm_unlock(dqm);
return r;
}
+
+size_t mqd_size_from_queue_type(struct device_queue_manager *dqm, enum kfd_queue_type type)
+{
+ return dqm->mqd_mgrs[get_mqd_type_from_queue_type(type)]->mqd_size;
+}
+
#if defined(CONFIG_DEBUG_FS)
static void seq_reg_dump(struct seq_file *m,
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
index 09ab36f8e8c6..c03b6e6fb368 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
@@ -327,6 +327,8 @@ int debug_refresh_runlist(struct device_queue_manager *dqm);
bool kfd_dqm_is_queue_in_process(struct device_queue_manager *dqm,
struct qcm_process_device *qpd,
int doorbell_off, u32 *queue_format);
+size_t mqd_size_from_queue_type(struct device_queue_manager *dqm,
+ enum kfd_queue_type type);
static inline unsigned int get_sh_mem_bases_32(struct kfd_process_device *pdd)
{
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index ca6c48e3703c..39bc24e86c2a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -39,22 +39,22 @@
#endif
#define dev_fmt(fmt) "kfd_migrate: " fmt
-static uint64_t
-svm_migrate_direct_mapping_addr(struct amdgpu_device *adev, uint64_t addr)
+static u64
+svm_migrate_direct_mapping_addr(struct amdgpu_device *adev, u64 addr)
{
return addr + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM);
}
static int
-svm_migrate_gart_map(struct amdgpu_ring *ring, uint64_t npages,
- dma_addr_t *addr, uint64_t *gart_addr, uint64_t flags)
+svm_migrate_gart_map(struct amdgpu_ring *ring, u64 npages,
+ dma_addr_t *addr, u64 *gart_addr, u64 flags)
{
struct amdgpu_device *adev = ring->adev;
struct amdgpu_job *job;
unsigned int num_dw, num_bytes;
struct dma_fence *fence;
- uint64_t src_addr, dst_addr;
- uint64_t pte_flags;
+ u64 src_addr, dst_addr;
+ u64 pte_flags;
void *cpu_addr;
int r;
@@ -123,16 +123,16 @@ svm_migrate_gart_map(struct amdgpu_ring *ring, uint64_t npages,
static int
svm_migrate_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys,
- uint64_t *vram, uint64_t npages,
+ u64 *vram, u64 npages,
enum MIGRATION_COPY_DIR direction,
struct dma_fence **mfence)
{
- const uint64_t GTT_MAX_PAGES = AMDGPU_GTT_MAX_TRANSFER_SIZE;
+ const u64 GTT_MAX_PAGES = AMDGPU_GTT_MAX_TRANSFER_SIZE;
struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
- uint64_t gart_s, gart_d;
+ u64 gart_s, gart_d;
struct dma_fence *next;
- uint64_t size;
- int r;
+ u64 size;
+ int r = 0;
mutex_lock(&adev->mman.gtt_window_lock);
@@ -252,64 +252,47 @@ svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr)
return page;
}
-static void svm_migrate_put_sys_page(unsigned long addr)
-{
- struct page *page;
-
- page = pfn_to_page(addr >> PAGE_SHIFT);
- unlock_page(page);
- put_page(page);
-}
-
-static unsigned long svm_migrate_unsuccessful_pages(struct migrate_vma *migrate)
-{
- unsigned long upages = 0;
- unsigned long i;
-
- for (i = 0; i < migrate->npages; i++) {
- if (migrate->src[i] & MIGRATE_PFN_VALID &&
- !(migrate->src[i] & MIGRATE_PFN_MIGRATE))
- upages++;
- }
- return upages;
-}
-
-static int
+static long
svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
struct migrate_vma *migrate, struct dma_fence **mfence,
- dma_addr_t *scratch, uint64_t ttm_res_offset)
+ dma_addr_t *scratch, u64 ttm_res_offset)
{
- uint64_t npages = migrate->cpages;
+ u64 npages = migrate->npages;
struct amdgpu_device *adev = node->adev;
struct device *dev = adev->dev;
struct amdgpu_res_cursor cursor;
+ long mpages;
dma_addr_t *src;
- uint64_t *dst;
- uint64_t i, j;
- int r;
+ u64 *dst;
+ u64 i, j;
+ int r = 0;
pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start,
prange->last, ttm_res_offset);
src = scratch;
- dst = (uint64_t *)(scratch + npages);
+ dst = (u64 *)(scratch + npages);
amdgpu_res_first(prange->ttm_res, ttm_res_offset,
npages << PAGE_SHIFT, &cursor);
- for (i = j = 0; i < npages; i++) {
+ mpages = 0;
+ for (i = j = 0; (i < npages) && (mpages < migrate->cpages); i++) {
struct page *spage;
- dst[i] = cursor.start + (j << PAGE_SHIFT);
- migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
- svm_migrate_get_vram_page(prange, migrate->dst[i]);
- migrate->dst[i] = migrate_pfn(migrate->dst[i]);
-
+ if (migrate->src[i] & MIGRATE_PFN_MIGRATE) {
+ dst[i] = cursor.start + (j << PAGE_SHIFT);
+ migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
+ svm_migrate_get_vram_page(prange, migrate->dst[i]);
+ migrate->dst[i] = migrate_pfn(migrate->dst[i]);
+ mpages++;
+ }
spage = migrate_pfn_to_page(migrate->src[i]);
if (spage && !is_zone_device_page(spage)) {
src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
DMA_BIDIRECTIONAL);
r = dma_mapping_error(dev, src[i]);
if (r) {
+ src[i] = 0;
dev_err(dev, "%s: fail %d dma_map_page\n",
__func__, r);
goto out_free_vram_pages;
@@ -334,7 +317,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
pr_debug_ratelimited("dma mapping src to 0x%llx, pfn 0x%lx\n",
src[i] >> PAGE_SHIFT, page_to_pfn(spage));
- if (j >= (cursor.size >> PAGE_SHIFT) - 1 && i < npages - 1) {
+ /* accumulated j + 1 pages reach end of current drm_buddy_block */
+ if (j + 1 >= (cursor.size >> PAGE_SHIFT)) {
r = svm_migrate_copy_memory_gart(adev, src + i - j,
dst + i - j, j + 1,
FROM_RAM_TO_VRAM,
@@ -348,16 +332,21 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
}
}
- r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j,
+ if (j > 0)
+ r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j,
FROM_RAM_TO_VRAM, mfence);
out_free_vram_pages:
if (r) {
pr_debug("failed %d to copy memory to vram\n", r);
- while (i--) {
+ for (i = 0; i < npages && mpages; i++) {
+ if (!dst[i])
+ continue;
svm_migrate_put_vram_page(adev, dst[i]);
migrate->dst[i] = 0;
+ mpages--;
}
+ mpages = r;
}
#ifdef DEBUG_FORCE_MIXED_DOMAINS
@@ -375,22 +364,22 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange,
}
#endif
- return r;
+ return mpages;
}
static long
svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange,
- struct vm_area_struct *vma, uint64_t start,
- uint64_t end, uint32_t trigger, uint64_t ttm_res_offset)
+ struct vm_area_struct *vma, u64 start,
+ u64 end, uint32_t trigger, u64 ttm_res_offset)
{
struct kfd_process *p = container_of(prange->svms, struct kfd_process, svms);
- uint64_t npages = (end - start) >> PAGE_SHIFT;
+ u64 npages = (end - start) >> PAGE_SHIFT;
struct amdgpu_device *adev = node->adev;
struct kfd_process_device *pdd;
struct dma_fence *mfence = NULL;
struct migrate_vma migrate = { 0 };
unsigned long cpages = 0;
- unsigned long mpages = 0;
+ long mpages = 0;
dma_addr_t *scratch;
void *buf;
int r = -ENOMEM;
@@ -403,7 +392,7 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange,
migrate.pgmap_owner = SVM_ADEV_PGMAP_OWNER(adev);
buf = kvcalloc(npages,
- 2 * sizeof(*migrate.src) + sizeof(uint64_t) + sizeof(dma_addr_t),
+ 2 * sizeof(*migrate.src) + sizeof(u64) + sizeof(dma_addr_t),
GFP_KERNEL);
if (!buf)
goto out;
@@ -436,15 +425,17 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange,
else
pr_debug("0x%lx pages collected\n", cpages);
- r = svm_migrate_copy_to_vram(node, prange, &migrate, &mfence, scratch, ttm_res_offset);
+ mpages = svm_migrate_copy_to_vram(node, prange, &migrate, &mfence, scratch, ttm_res_offset);
migrate_vma_pages(&migrate);
svm_migrate_copy_done(adev, mfence);
migrate_vma_finalize(&migrate);
- mpages = cpages - svm_migrate_unsuccessful_pages(&migrate);
- pr_debug("successful/cpages/npages 0x%lx/0x%lx/0x%lx\n",
+ if (mpages >= 0)
+ pr_debug("migrated/collected/requested 0x%lx/0x%lx/0x%lx\n",
mpages, cpages, migrate.npages);
+ else
+ r = mpages;
kfd_smi_event_migration_end(node, p->lead_thread->pid,
start >> PAGE_SHIFT, end >> PAGE_SHIFT,
@@ -455,14 +446,13 @@ svm_migrate_vma_to_vram(struct kfd_node *node, struct svm_range *prange,
out_free:
kvfree(buf);
out:
- if (!r && mpages) {
+ if (!r && mpages > 0) {
pdd = svm_range_get_pdd_by_node(prange, node);
if (pdd)
WRITE_ONCE(pdd->page_in, pdd->page_in + mpages);
-
- return mpages;
}
- return r;
+
+ return r ? r : mpages;
}
/**
@@ -486,7 +476,7 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc,
{
unsigned long addr, start, end;
struct vm_area_struct *vma;
- uint64_t ttm_res_offset;
+ u64 ttm_res_offset;
struct kfd_node *node;
unsigned long mpages = 0;
long r = 0;
@@ -573,17 +563,19 @@ static void svm_migrate_page_free(struct page *page)
}
}
-static int
+static long
svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
struct migrate_vma *migrate, struct dma_fence **mfence,
- dma_addr_t *scratch, uint64_t npages)
+ dma_addr_t *scratch, u64 npages)
{
struct device *dev = adev->dev;
- uint64_t *src;
+ struct page *dpage = NULL;
dma_addr_t *dst;
- struct page *dpage;
- uint64_t i = 0, j;
- uint64_t addr;
+ u64 *src;
+ long mpages;
+
+ u64 i = 0, j;
+ u64 addr;
int r = 0;
pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start,
@@ -591,9 +583,10 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
addr = migrate->start;
- src = (uint64_t *)(scratch + npages);
+ src = (u64 *)(scratch + npages);
dst = scratch;
+ mpages = 0;
for (i = 0, j = 0; i < npages; i++, addr += PAGE_SIZE) {
struct page *spage;
@@ -635,6 +628,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
r = dma_mapping_error(dev, dst[i]);
if (r) {
dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r);
+ dst[i] = 0;
goto out_oom;
}
@@ -642,22 +636,47 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
dst[i] >> PAGE_SHIFT, page_to_pfn(dpage));
migrate->dst[i] = migrate_pfn(page_to_pfn(dpage));
+ mpages++;
+
+ dpage = NULL;
j++;
}
- r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j,
- FROM_VRAM_TO_RAM, mfence);
-
+ if (j > 0)
+ r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j,
+ FROM_VRAM_TO_RAM, mfence);
out_oom:
if (r) {
pr_debug("failed %d copy to ram\n", r);
+
+ /* first release current dpage when dma_map_page fail */
+ if (dpage) {
+ unlock_page(dpage);
+ put_page(dpage);
+ }
+
+ /* release previous allocated sys pages and unmap dma address */
while (i--) {
- svm_migrate_put_sys_page(dst[i]);
+
+ if (dst[i]) {
+ dma_unmap_page(dev, dst[i], PAGE_SIZE,
+ DMA_BIDIRECTIONAL);
+ dst[i] = 0;
+ }
+
+ dpage = migrate_pfn_to_page(migrate->dst[i]);
+ if (!dpage)
+ continue;
+
+ unlock_page(dpage);
+ put_page(dpage);
migrate->dst[i] = 0;
+ mpages--;
}
+ mpages = r;
}
- return r;
+ return mpages;
}
/**
@@ -679,14 +698,13 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
*/
static long
svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
- struct vm_area_struct *vma, uint64_t start, uint64_t end,
+ struct vm_area_struct *vma, u64 start, u64 end,
uint32_t trigger, struct page *fault_page)
{
struct kfd_process *p = container_of(prange->svms, struct kfd_process, svms);
- uint64_t npages = (end - start) >> PAGE_SHIFT;
- unsigned long upages = npages;
+ u64 npages = (end - start) >> PAGE_SHIFT;
unsigned long cpages = 0;
- unsigned long mpages = 0;
+ long mpages = 0;
struct amdgpu_device *adev = node->adev;
struct kfd_process_device *pdd;
struct dma_fence *mfence = NULL;
@@ -706,7 +724,7 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
migrate.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE;
buf = kvcalloc(npages,
- 2 * sizeof(*migrate.src) + sizeof(uint64_t) + sizeof(dma_addr_t),
+ 2 * sizeof(*migrate.src) + sizeof(u64) + sizeof(dma_addr_t),
GFP_KERNEL);
if (!buf)
goto out;
@@ -732,7 +750,6 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
if (!cpages) {
pr_debug("failed collect migrate device pages [0x%lx 0x%lx]\n",
prange->start, prange->last);
- upages = svm_migrate_unsuccessful_pages(&migrate);
goto out_free;
}
if (cpages != npages)
@@ -741,13 +758,15 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
else
pr_debug("0x%lx pages collected\n", cpages);
- r = svm_migrate_copy_to_ram(adev, prange, &migrate, &mfence,
+ mpages = svm_migrate_copy_to_ram(adev, prange, &migrate, &mfence,
scratch, npages);
migrate_vma_pages(&migrate);
- upages = svm_migrate_unsuccessful_pages(&migrate);
- pr_debug("unsuccessful/cpages/npages 0x%lx/0x%lx/0x%lx\n",
- upages, cpages, migrate.npages);
+ if (mpages >= 0)
+ pr_debug("migrated/collected/requested 0x%lx/0x%lx/0x%lx\n",
+ mpages, cpages, migrate.npages);
+ else
+ r = mpages;
svm_migrate_copy_done(adev, mfence);
migrate_vma_finalize(&migrate);
@@ -761,8 +780,7 @@ svm_migrate_vma_to_ram(struct kfd_node *node, struct svm_range *prange,
out_free:
kvfree(buf);
out:
- if (!r && cpages) {
- mpages = cpages - upages;
+ if (!r && mpages > 0) {
pdd = svm_range_get_pdd_by_node(prange, node);
if (pdd)
WRITE_ONCE(pdd->page_out, pdd->page_out + mpages);
@@ -845,6 +863,9 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm,
}
if (r >= 0) {
+ WARN_ONCE(prange->vram_pages < mpages,
+ "Recorded vram pages(0x%llx) should not be less than migration pages(0x%lx).",
+ prange->vram_pages, mpages);
prange->vram_pages -= mpages;
/* prange does not have vram page set its actual_loc to system
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c
index 6a69988b2d4f..194a1c79cf04 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c
@@ -27,6 +27,7 @@
#include <linux/uaccess.h>
#include "kfd_priv.h"
#include "kfd_mqd_manager.h"
+#include "kfd_topology.h"
#include "v9_structs.h"
#include "gc/gc_9_0_offset.h"
#include "gc/gc_9_0_sh_mask.h"
@@ -375,20 +376,26 @@ static int get_wave_state(struct mqd_manager *mm, void *mqd,
static void get_checkpoint_info(struct mqd_manager *mm, void *mqd, u32 *ctl_stack_size)
{
struct v9_mqd *m = get_mqd(mqd);
+ struct kfd_topology_device *topo_dev = kfd_topology_device_by_id(mm->dev->id);
+ u32 bo_size = topo_dev ? ALIGN(topo_dev->node_props.ctl_stack_size, PAGE_SIZE) : 0;
- *ctl_stack_size = m->cp_hqd_cntl_stack_size;
+ *ctl_stack_size = min_t(u32, m->cp_hqd_cntl_stack_size, bo_size);
}
static void checkpoint_mqd(struct mqd_manager *mm, void *mqd, void *mqd_dst, void *ctl_stack_dst)
{
struct v9_mqd *m;
+ struct kfd_topology_device *topo_dev = kfd_topology_device_by_id(mm->dev->id);
+ u32 bo_size = topo_dev ? ALIGN(topo_dev->node_props.ctl_stack_size, PAGE_SIZE) : 0;
+ u32 ctl_stack_copy_size;
/* Control stack is located one page after MQD. */
void *ctl_stack = (void *)((uintptr_t)mqd + PAGE_SIZE);
m = get_mqd(mqd);
+ ctl_stack_copy_size = min_t(u32, m->cp_hqd_cntl_stack_size, bo_size);
memcpy(mqd_dst, m, sizeof(struct v9_mqd));
- memcpy(ctl_stack_dst, ctl_stack, m->cp_hqd_cntl_stack_size);
+ memcpy(ctl_stack_dst, ctl_stack, ctl_stack_copy_size);
}
static void restore_mqd(struct mqd_manager *mm, void **mqd,
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index 74bd33a9bd09..939a6c92351b 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -386,6 +386,8 @@ struct kfd_dev {
/* bitmap for dynamic doorbell allocation from doorbell object */
unsigned long *doorbell_bitmap;
+
+ atomic_t kfd_processes_count;
};
enum kfd_mempool {
@@ -438,7 +440,8 @@ enum kfd_queue_type {
KFD_QUEUE_TYPE_HIQ,
KFD_QUEUE_TYPE_DIQ,
KFD_QUEUE_TYPE_SDMA_XGMI,
- KFD_QUEUE_TYPE_SDMA_BY_ENG_ID
+ KFD_QUEUE_TYPE_SDMA_BY_ENG_ID,
+ KFD_QUEUE_TYPE_MAX,
};
enum kfd_queue_format {
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
index 64f3a0687f8a..b9f00b756c50 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
@@ -1091,6 +1091,8 @@ static void kfd_process_destroy_pdds(struct kfd_process *p)
pdd->runtime_inuse = false;
}
+ atomic_dec(&pdd->dev->kfd->kfd_processes_count);
+
kfree(pdd);
p->pdds[i] = NULL;
}
@@ -1662,6 +1664,8 @@ struct kfd_process_device *kfd_create_process_device_data(struct kfd_node *dev,
/* Init idr used for memory handle translation */
idr_init(&pdd->alloc_idr);
+ atomic_inc(&dev->kfd->kfd_processes_count);
+
return pdd;
}
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
index be3f821dfd56..bf3c7d055fc3 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
@@ -1015,6 +1015,23 @@ int kfd_criu_restore_queue(struct kfd_process *p,
goto exit;
}
+ pdd = kfd_process_device_data_by_id(p, q_data->gpu_id);
+ if (!pdd) {
+ pr_err("Failed to get pdd\n");
+ ret = -EINVAL;
+ goto exit;
+ }
+
+ if (q_data->type >= KFD_QUEUE_TYPE_MAX) {
+ ret = -EINVAL;
+ goto exit;
+ }
+
+ if (q_data->mqd_size != mqd_size_from_queue_type(pdd->dev->dqm, q_data->type)) {
+ ret = -EINVAL;
+ goto exit;
+ }
+
*priv_data_offset += sizeof(*q_data);
q_extra_data_size = (uint64_t)q_data->ctl_stack_size + q_data->mqd_size;
@@ -1037,13 +1054,6 @@ int kfd_criu_restore_queue(struct kfd_process *p,
*priv_data_offset += q_extra_data_size;
- pdd = kfd_process_device_data_by_id(p, q_data->gpu_id);
- if (!pdd) {
- pr_err("Failed to get pdd\n");
- ret = -EINVAL;
- goto exit;
- }
-
/* data stored in this order: mqd, ctl_stack */
mqd = q_extra_data;
ctl_stack = mqd + q_data->mqd_size;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 592953fe9afc..8010ef10e572 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -255,8 +255,10 @@ void kfd_smi_event_update_vmfault(struct kfd_node *dev, uint16_t pasid)
if (task_info) {
/* Report VM faults from user applications, not retry from kernel */
if (task_info->pid)
- kfd_smi_event_add(0, dev, KFD_SMI_EVENT_VMFAULT, KFD_EVENT_FMT_VMFAULT(
- task_info->pid, task_info->task_name));
+ kfd_smi_event_add(task_info->tgid, dev,
+ KFD_SMI_EVENT_VMFAULT,
+ KFD_EVENT_FMT_VMFAULT(task_info->pid,
+ task_info->task_name));
amdgpu_vm_put_task_info(task_info);
}
}
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
index 9682c190e952..db707c2311be 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
@@ -44,7 +44,7 @@
* in amdgpu_dm_kms.h file
*/
-#define AMDGPU_DM_MAX_DISPLAY_INDEX 31
+#define AMDGPU_DM_MAX_DISPLAY_COUNT 32
#define AMDGPU_DM_MAX_CRTC 6
@@ -732,6 +732,7 @@ struct amdgpu_dm_connector {
int sr_skip_count;
bool disallow_edp_enter_psr;
+ bool disallow_edp_enter_replay;
/* Record progress status of mst*/
uint8_t mst_status;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c
index 41812081418e..b4a769216d60 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c
@@ -31,6 +31,7 @@
#include "dc.h"
#include "amdgpu_securedisplay.h"
#include "amdgpu_dm_psr.h"
+#include "amdgpu_dm_replay.h"
static const char *const pipe_crc_sources[] = {
"none",
@@ -216,6 +217,7 @@ int amdgpu_dm_crtc_configure_crc_source(struct drm_crtc *crtc,
{
struct amdgpu_device *adev = drm_to_adev(crtc->dev);
struct dc_stream_state *stream_state = dm_crtc_state->stream;
+ struct amdgpu_dm_connector *aconnector = NULL;
bool enable = amdgpu_dm_is_valid_crc_source(source);
int ret = 0;
@@ -223,11 +225,22 @@ int amdgpu_dm_crtc_configure_crc_source(struct drm_crtc *crtc,
if (!stream_state)
return -EINVAL;
+ /* Get connector from stream */
+ aconnector = (struct amdgpu_dm_connector *)stream_state->dm_stream_context;
+
mutex_lock(&adev->dm.dc_lock);
- /* For PSR1, check that the panel has exited PSR */
- if (stream_state->link->psr_settings.psr_version < DC_PSR_VERSION_SU_1)
- amdgpu_dm_psr_wait_disable(stream_state);
+
+ if (enable) {
+ /* For PSR1, check that the panel has exited PSR */
+ if (stream_state->link->psr_settings.psr_version < DC_PSR_VERSION_SU_1)
+ amdgpu_dm_psr_wait_disable(stream_state);
+
+ /* Set flag to disallow enter replay when CRC source is enabled */
+ if (aconnector)
+ aconnector->disallow_edp_enter_replay = true;
+ amdgpu_dm_replay_disable(stream_state);
+ }
/* Enable or disable CRTC CRC generation */
if (dm_is_crc_source_crtc(source) || source == AMDGPU_DM_PIPE_CRC_SOURCE_NONE) {
@@ -250,6 +263,12 @@ int amdgpu_dm_crtc_configure_crc_source(struct drm_crtc *crtc,
DYN_EXPANSION_AUTO);
}
+ if (!enable) {
+ /* Clear flag to allow enter replay when CRC source is disabled */
+ if (aconnector)
+ aconnector->disallow_edp_enter_replay = false;
+ }
+
unlock:
mutex_unlock(&adev->dm.dc_lock);
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
index 779c2c3ffbc0..824e88fe54c6 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
@@ -479,13 +479,13 @@ static void amdgpu_dm_crtc_reset_state(struct drm_crtc *crtc)
{
struct dm_crtc_state *state;
- if (crtc->state)
- amdgpu_dm_crtc_destroy_state(crtc, crtc->state);
-
state = kzalloc(sizeof(*state), GFP_KERNEL);
- if (WARN_ON(!state))
+ if (!state)
return;
+ if (crtc->state)
+ amdgpu_dm_crtc_destroy_state(crtc, crtc->state);
+
__drm_atomic_helper_crtc_reset(crtc, &state->base);
}
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
index 10ba4d7bf632..930310aff1d6 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
@@ -256,7 +256,7 @@ void hdcp_reset_display(struct hdcp_workqueue *hdcp_work, unsigned int link_inde
cancel_delayed_work(&hdcp_w->property_validate_dwork);
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) {
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) {
hdcp_w->encryption_status[conn_index] =
MOD_HDCP_ENCRYPTION_STATUS_HDCP_OFF;
if (hdcp_w->aconnector[conn_index]) {
@@ -303,7 +303,7 @@ static void event_property_update(struct work_struct *work)
struct drm_connector *connector;
struct drm_connector_state *conn_state;
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) {
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) {
aconnector = hdcp_work->aconnector[conn_index];
if (!aconnector)
@@ -372,7 +372,7 @@ static void event_property_validate(struct work_struct *work)
guard(mutex)(&hdcp_work->mutex);
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX;
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT;
conn_index++) {
aconnector = hdcp_work->aconnector[conn_index];
@@ -749,10 +749,10 @@ struct hdcp_workqueue *hdcp_create_workqueue(struct amdgpu_device *adev,
memset(hdcp_work[i].aconnector, 0,
sizeof(struct amdgpu_dm_connector *) *
- AMDGPU_DM_MAX_DISPLAY_INDEX);
+ AMDGPU_DM_MAX_DISPLAY_COUNT);
memset(hdcp_work[i].encryption_status, 0,
sizeof(enum mod_hdcp_encryption_status) *
- AMDGPU_DM_MAX_DISPLAY_INDEX);
+ AMDGPU_DM_MAX_DISPLAY_COUNT);
}
cp_psp->funcs.update_stream_config = update_config;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
index 69b445b011c8..d18c4cc7d69f 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
@@ -43,7 +43,7 @@ struct hdcp_workqueue {
struct delayed_work callback_dwork;
struct delayed_work watchdog_timer_dwork;
struct delayed_work property_validate_dwork;
- struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_COUNT];
struct mutex mutex;
struct mod_hdcp hdcp;
@@ -51,7 +51,7 @@ struct hdcp_workqueue {
struct mod_hdcp_display display;
struct mod_hdcp_link link;
- enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_COUNT];
/* when display is unplugged from mst hub, connctor will be
* destroyed within dm_dp_mst_connector_destroy. connector
* hdcp perperties, like type, undesired, desired, enabled,
@@ -61,9 +61,9 @@ struct hdcp_workqueue {
* will be retrieved from hdcp_work within dm_dp_mst_get_modes
*/
/* un-desired, desired, enabled */
- unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_COUNT];
/* hdcp1.x, hdcp2.x */
- unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_COUNT];
uint8_t max_link;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c
index a215234151ac..82bc3dcd3f65 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c
@@ -913,8 +913,9 @@ void amdgpu_dm_hpd_init(struct amdgpu_device *adev)
continue;
amdgpu_dm_connector = to_amdgpu_dm_connector(connector);
-
dc_link = amdgpu_dm_connector->dc_link;
+ if (!dc_link)
+ continue;
/*
* Get a base driver irq reference for hpd ints for the lifetime
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
index 97eeabc556a0..a033628c3d54 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
@@ -1415,17 +1415,15 @@ static const struct drm_plane_helper_funcs dm_plane_helper_funcs = {
static void amdgpu_dm_plane_drm_plane_reset(struct drm_plane *plane)
{
- struct dm_plane_state *amdgpu_state = NULL;
-
- if (plane->state)
- plane->funcs->atomic_destroy_state(plane, plane->state);
+ struct dm_plane_state *amdgpu_state;
amdgpu_state = kzalloc(sizeof(*amdgpu_state), GFP_KERNEL);
- WARN_ON(amdgpu_state == NULL);
-
if (!amdgpu_state)
return;
+ if (plane->state)
+ plane->funcs->atomic_destroy_state(plane, plane->state);
+
__drm_atomic_helper_plane_reset(plane, &amdgpu_state->base);
amdgpu_state->degamma_tf = AMDGPU_TRANSFER_FUNCTION_DEFAULT;
amdgpu_state->hdr_mult = AMDGPU_HDR_MULT_DEFAULT;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c
index 41f07f13a7b5..7db927dcfe8d 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c
@@ -153,10 +153,17 @@ bool amdgpu_dm_replay_enable(struct dc_stream_state *stream, bool wait)
{
bool replay_active = true;
struct dc_link *link = NULL;
+ struct amdgpu_dm_connector *aconnector = NULL;
if (stream == NULL)
return false;
+ /* Check if replay is disabled by connector flag */
+ aconnector = (struct amdgpu_dm_connector *)stream->dm_stream_context;
+ if (!aconnector || aconnector->disallow_edp_enter_replay) {
+ return false;
+ }
+
link = stream->link;
if (link) {
diff --git a/drivers/gpu/drm/amd/display/dc/core/dc.c b/drivers/gpu/drm/amd/display/dc/core/dc.c
index 070cca8b521b..ad1cc546ba48 100644
--- a/drivers/gpu/drm/amd/display/dc/core/dc.c
+++ b/drivers/gpu/drm/amd/display/dc/core/dc.c
@@ -1377,7 +1377,7 @@ static void disable_vbios_mode_if_required(
struct dc *dc_create(const struct dc_init_data *init_params)
{
- struct dc *dc = kzalloc(sizeof(*dc), GFP_KERNEL);
+ struct dc *dc = kvzalloc(sizeof(*dc), GFP_KERNEL);
unsigned int full_pipe_count;
if (!dc)
@@ -1425,7 +1425,7 @@ struct dc *dc_create(const struct dc_init_data *init_params)
destruct_dc:
dc_destruct(dc);
- kfree(dc);
+ kvfree(dc);
return NULL;
}
@@ -1474,7 +1474,7 @@ void dc_deinit_callbacks(struct dc *dc)
void dc_destroy(struct dc **dc)
{
dc_destruct(*dc);
- kfree(*dc);
+ kvfree(*dc);
*dc = NULL;
}
diff --git a/drivers/gpu/drm/amd/display/dc/dc_dp_types.h b/drivers/gpu/drm/amd/display/dc/dc_dp_types.h
index 223c3d55544b..b128f1da98c9 100644
--- a/drivers/gpu/drm/amd/display/dc/dc_dp_types.h
+++ b/drivers/gpu/drm/amd/display/dc/dc_dp_types.h
@@ -1112,7 +1112,7 @@ struct dc_lttpr_caps {
union dp_main_link_channel_coding_lttpr_cap main_link_channel_coding;
union dp_128b_132b_supported_lttpr_link_rates supported_128b_132b_rates;
union dp_alpm_lttpr_cap alpm;
- uint8_t aux_rd_interval[MAX_REPEATER_CNT - 1];
+ uint8_t aux_rd_interval[MAX_REPEATER_CNT];
};
struct dc_dongle_dfp_cap_ext {
diff --git a/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c b/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c
index 51d491bffa32..3f656e5dacf8 100644
--- a/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c
+++ b/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c
@@ -457,14 +457,13 @@ void dml21_handle_phantom_streams_planes(const struct dc *dc, struct dc_state *c
main_plane = main_stream_status->plane_states[dc_plane_index];
/* create phantom planes for subvp enabled plane */
- dml21_add_phantom_plane(dml_ctx,
- dc,
- context,
- phantom_stream,
- main_plane,
- &dml_ctx->v21.mode_programming.programming->plane_programming[dml_plane_index]);
-
- phantoms_added = true;
+ if (dml21_add_phantom_plane(dml_ctx,
+ dc,
+ context,
+ phantom_stream,
+ main_plane,
+ &dml_ctx->v21.mode_programming.programming->plane_programming[dml_plane_index]))
+ phantoms_added = true;
}
}
}
diff --git a/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c b/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c
index d18b60c9761b..37dd05d91768 100644
--- a/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c
+++ b/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c
@@ -2449,7 +2449,7 @@ static void calculate_mcache_row_bytes(
DML2_ASSERT(*p->num_mcaches > 0);
}
-static void calculate_mcache_setting(
+static bool calculate_mcache_setting(
struct dml2_core_internal_scratch *scratch,
struct dml2_core_calcs_calculate_mcache_setting_params *p)
{
@@ -2473,7 +2473,7 @@ static void calculate_mcache_setting(
*p->lc_comb_mcache = 0;
if (!p->dcc_enable)
- return;
+ return true;
l->is_dual_plane = dml_is_420(p->source_format) || p->source_format == dml2_rgbe_alpha;
@@ -2509,7 +2509,14 @@ static void calculate_mcache_setting(
l->l_p.mvmpg_per_mcache_lb = &l->mvmpg_per_mcache_lb_l;
calculate_mcache_row_bytes(scratch, &l->l_p);
- dml2_assert(*p->num_mcaches_l > 0);
+ if (*p->num_mcaches_l == 0 ||
+ (p->surf_vert ? l->mvmpg_height_l : l->mvmpg_width_l) == 0) {
+ dml2_printf("DML::%s: degenerate luma viewport (num_mcaches_l=%u mvmpg_%s_l=%u) — mode not supported\n",
+ __func__, *p->num_mcaches_l,
+ p->surf_vert ? "height" : "width",
+ p->surf_vert ? l->mvmpg_height_l : l->mvmpg_width_l);
+ return false;
+ }
if (l->is_dual_plane) {
l->c_p.num_chans = p->num_chans;
@@ -2544,7 +2551,14 @@ static void calculate_mcache_setting(
l->c_p.mvmpg_per_mcache_lb = &l->mvmpg_per_mcache_lb_c;
calculate_mcache_row_bytes(scratch, &l->c_p);
- dml2_assert(*p->num_mcaches_c > 0);
+ if (*p->num_mcaches_c == 0 ||
+ (p->surf_vert ? l->mvmpg_height_c : l->mvmpg_width_c) == 0) {
+ dml2_printf("DML::%s: degenerate chroma viewport (num_mcaches_c=%u mvmpg_%s_c=%u) — mode not supported\n",
+ __func__, *p->num_mcaches_c,
+ p->surf_vert ? "height" : "width",
+ p->surf_vert ? l->mvmpg_height_c : l->mvmpg_width_c);
+ return false;
+ }
}
// Sharing for iMALL access
@@ -2652,6 +2666,7 @@ static void calculate_mcache_setting(
*p->mcache_shift_granularity_l = l->mvmpg_access_width_l;
*p->mcache_shift_granularity_c = l->mvmpg_access_width_c;
+ return true;
}
static void calculate_mall_bw_overhead_factor(
@@ -8481,7 +8496,10 @@ static bool dml_core_mode_support(struct dml2_core_calcs_mode_support_ex *in_out
calculate_mcache_setting_params->mall_comb_mcache_c = &mode_lib->ms.mall_comb_mcache_c[k];
calculate_mcache_setting_params->lc_comb_mcache = &mode_lib->ms.lc_comb_mcache[k];
- calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params);
+ if (!calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params)) {
+ mode_lib->ms.support.ModeSupport = false;
+ return false;
+ }
}
calculate_mall_bw_overhead_factor(
@@ -10483,7 +10501,8 @@ static bool dml_core_mode_programming(struct dml2_core_calcs_mode_programming_ex
calculate_mcache_setting_params->mall_comb_mcache_l = &mode_lib->mp.mall_comb_mcache_l[k];
calculate_mcache_setting_params->mall_comb_mcache_c = &mode_lib->mp.mall_comb_mcache_c[k];
calculate_mcache_setting_params->lc_comb_mcache = &mode_lib->mp.lc_comb_mcache[k];
- calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params);
+ if (!calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params))
+ return false;
}
calculate_mall_bw_overhead_factor(
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c
index c739e5b2c559..074598048389 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c
@@ -1617,3 +1617,55 @@ bool dcn35_is_dp_dig_pixel_rate_div_policy(struct pipe_ctx *pipe_ctx)
return false;
}
+
+static void disable_link_output_symclk_on_tx_off(struct dc_link *link, enum dp_link_encoding link_encoding)
+{
+ struct dc *dc = link->ctx->dc;
+ struct pipe_ctx *pipe_ctx = NULL;
+ uint8_t i;
+
+ for (i = 0; i < MAX_PIPES; i++) {
+ pipe_ctx = &dc->current_state->res_ctx.pipe_ctx[i];
+ if (pipe_ctx->stream && pipe_ctx->stream->link == link && pipe_ctx->top_pipe == NULL) {
+ pipe_ctx->clock_source->funcs->program_pix_clk(
+ pipe_ctx->clock_source,
+ &pipe_ctx->stream_res.pix_clk_params,
+ link_encoding,
+ &pipe_ctx->pll_settings);
+ break;
+ }
+ }
+}
+
+void dcn35_disable_link_output(struct dc_link *link,
+ const struct link_resource *link_res,
+ enum signal_type signal)
+{
+ struct dc *dc = link->ctx->dc;
+ const struct link_hwss *link_hwss = get_link_hwss(link, link_res);
+ struct dmcu *dmcu = dc->res_pool->dmcu;
+
+ if (signal == SIGNAL_TYPE_EDP &&
+ link->dc->hwss.edp_backlight_control &&
+ !link->skip_implict_edp_power_control)
+ link->dc->hwss.edp_backlight_control(link, false);
+ else if (dmcu != NULL && dmcu->funcs->lock_phy)
+ dmcu->funcs->lock_phy(dmcu);
+
+ if (dc_is_tmds_signal(signal) && link->phy_state.symclk_ref_cnts.otg > 0) {
+ disable_link_output_symclk_on_tx_off(link, DP_UNKNOWN_ENCODING);
+ link->phy_state.symclk_state = SYMCLK_ON_TX_OFF;
+ } else {
+ link_hwss->disable_link_output(link, link_res, signal);
+ link->phy_state.symclk_state = SYMCLK_OFF_TX_OFF;
+ }
+ /*
+ * Add the logic to extract BOTH power up and power down sequences
+ * from enable/disable link output and only call edp panel control
+ * in enable_link_dp and disable_link_dp once.
+ */
+ if (dmcu != NULL && dmcu->funcs->unlock_phy)
+ dmcu->funcs->unlock_phy(dmcu);
+
+ dc->link_srv->dp_trace_source_sequence(link, DPCD_SOURCE_SEQ_AFTER_DISABLE_LINK_PHY);
+}
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h
index e27b3609020f..45e762f99119 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h
@@ -99,4 +99,8 @@ void dcn35_set_long_vblank(struct pipe_ctx **pipe_ctx,
bool dcn35_is_dp_dig_pixel_rate_div_policy(struct pipe_ctx *pipe_ctx);
+void dcn35_disable_link_output(struct dc_link *link,
+ const struct link_resource *link_res,
+ enum signal_type signal);
+
#endif /* __DC_HWSS_DCN35_H__ */
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c
index 2bbf1fef94fd..e34f292f5c41 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c
@@ -107,7 +107,7 @@ static const struct hw_sequencer_funcs dcn35_funcs = {
.enable_lvds_link_output = dce110_enable_lvds_link_output,
.enable_tmds_link_output = dce110_enable_tmds_link_output,
.enable_dp_link_output = dce110_enable_dp_link_output,
- .disable_link_output = dcn32_disable_link_output,
+ .disable_link_output = dcn35_disable_link_output,
.z10_restore = dcn35_z10_restore,
.z10_save_init = dcn31_z10_save_init,
.set_disp_pattern_generator = dcn30_set_disp_pattern_generator,
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c
index 768b756d427c..be99798cf6ee 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c
@@ -225,6 +225,7 @@ void dcn401_init_hw(struct dc *dc)
int edp_num;
uint32_t backlight = MAX_BACKLIGHT_LEVEL;
uint32_t user_level = MAX_BACKLIGHT_LEVEL;
+ bool dchub_ref_freq_changed;
int current_dchub_ref_freq = 0;
if (dc->clk_mgr && dc->clk_mgr->funcs && dc->clk_mgr->funcs->init_clocks) {
@@ -445,12 +446,14 @@ void dcn401_init_hw(struct dc *dc)
dc->caps.dmub_caps.mclk_sw = dc->ctx->dmub_srv->dmub->feature_caps.fw_assisted_mclk_switch_ver > 0;
dc->caps.dmub_caps.fams_ver = dc->ctx->dmub_srv->dmub->feature_caps.fw_assisted_mclk_switch_ver;
dc->debug.fams2_config.bits.enable &= dc->ctx->dmub_srv->dmub->feature_caps.fw_assisted_mclk_switch_ver == 2;
- if ((!dc->debug.fams2_config.bits.enable && dc->res_pool->funcs->update_bw_bounding_box)
- || res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000 != current_dchub_ref_freq) {
+ dchub_ref_freq_changed =
+ res_pool->ref_clocks.dchub_ref_clock_inKhz / 1000 != current_dchub_ref_freq;
+ if ((!dc->debug.fams2_config.bits.enable || dchub_ref_freq_changed) &&
+ dc->res_pool->funcs->update_bw_bounding_box &&
+ dc->clk_mgr && dc->clk_mgr->bw_params) {
/* update bounding box if FAMS2 disabled, or if dchub clk has changed */
- if (dc->clk_mgr)
- dc->res_pool->funcs->update_bw_bounding_box(dc,
- dc->clk_mgr->bw_params);
+ dc->res_pool->funcs->update_bw_bounding_box(dc,
+ dc->clk_mgr->bw_params);
}
}
}
diff --git a/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c b/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c
index ccfdca7c83f6..1fa4d32ce0be 100644
--- a/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c
@@ -90,9 +90,14 @@
#include "dml/dcn32/dcn32_fpu.h"
#include "dc_state_priv.h"
+#include "dc_fpu.h"
#include "dml2/dml2_wrapper.h"
+#if !defined(DC_RUN_WITH_PREEMPTION_ENABLED)
+#define DC_RUN_WITH_PREEMPTION_ENABLED(code) code
+#endif
+
#define DC_LOGGER_INIT(logger)
enum dcn32_clk_src_array_id {
@@ -1649,7 +1654,8 @@ static void dcn32_enable_phantom_plane(struct dc *dc,
if (curr_pipe->top_pipe && curr_pipe->top_pipe->plane_state == curr_pipe->plane_state)
phantom_plane = prev_phantom_plane;
else
- phantom_plane = dc_state_create_phantom_plane(dc, context, curr_pipe->plane_state);
+ DC_RUN_WITH_PREEMPTION_ENABLED(phantom_plane =
+ dc_state_create_phantom_plane(dc, context, curr_pipe->plane_state));
if (!phantom_plane)
continue;
diff --git a/drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c b/drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c
index bc5d3758fbf6..7fb81496aac7 100644
--- a/drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c
+++ b/drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c
@@ -1247,6 +1247,14 @@ static bool smu_is_workload_profile_available(struct smu_context *smu,
return smu->workload_map && smu->workload_map[profile].valid_mapping;
}
+static int smu_set_power_dep(struct smu_context *smu, bool enable)
+{
+ if (!smu->ppt_funcs->set_power_dep)
+ return 0;
+
+ return smu->ppt_funcs->set_power_dep(smu, enable);
+}
+
static int smu_sw_init(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
@@ -1312,6 +1320,8 @@ static int smu_sw_init(void *handle)
if (!smu->ppt_funcs->get_fan_control_mode)
smu->adev->pm.no_fan = true;
+ smu_set_power_dep(smu, true);
+
return 0;
}
@@ -1334,6 +1344,8 @@ static int smu_sw_fini(void *handle)
smu_fini_microcode(smu);
+ smu_set_power_dep(smu, false);
+
return 0;
}
diff --git a/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h b/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h
index 2b8a18ce25d9..0145aff45c38 100644
--- a/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h
+++ b/drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h
@@ -566,6 +566,9 @@ struct smu_context {
bool pm_enabled;
bool is_apu;
+ /* Power dependency link from an integrated xHCI controller to the GPU */
+ struct device_link *usb_power_link;
+
uint32_t smc_driver_if_version;
uint32_t smc_fw_if_version;
uint32_t smc_fw_version;
@@ -1445,6 +1448,13 @@ struct pptable_funcs {
*/
int (*set_wbrf_exclusion_ranges)(struct smu_context *smu,
struct freq_band_range *exclusion_ranges);
+ /**
+ * @set_power_dep: Create or destroy a power dependency link
+ * from an integrated xHCI controller to the GPU so that the GPU is
+ * resumed before the USB controller during PM resume. @enable is true
+ * to create the link and false to tear it down.
+ */
+ int (*set_power_dep)(struct smu_context *smu, bool enable);
};
typedef enum {
diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_0_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_0_ppt.c
index 3fdca8a5816b..a710106d2880 100644
--- a/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_0_ppt.c
+++ b/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_0_ppt.c
@@ -1641,6 +1641,50 @@ static int smu_v14_0_common_set_mall_enable(struct smu_context *smu)
return ret;
}
+/*
+ * Link any xHCI controller sharing the GPU's PCIe root port as a consumer
+ * of the GPU so the GPU resumes first, avoiding an xHCI resume race.
+ */
+static int smu_v14_0_0_set_power_dep(struct smu_context *smu, bool enable)
+{
+ struct amdgpu_device *adev = smu->adev;
+ struct pci_dev *gpu_pdev = adev->pdev;
+ struct pci_dev *root_port, *usb_pdev = NULL;
+ struct device_link *link;
+
+ if (!enable) {
+ if (smu->usb_power_link) {
+ device_link_del(smu->usb_power_link);
+ smu->usb_power_link = NULL;
+ }
+ return 0;
+ }
+
+ root_port = pcie_find_root_port(gpu_pdev);
+ while ((usb_pdev = pci_get_class(PCI_CLASS_SERIAL_USB_XHCI, usb_pdev))) {
+ struct pci_dev *usb_root;
+
+ usb_root = pcie_find_root_port(usb_pdev);
+ if (usb_root != root_port)
+ continue;
+
+ /* Create device link: USB (consumer) depends on GPU (supplier) */
+ link = device_link_add(&usb_pdev->dev, &gpu_pdev->dev,
+ DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME);
+ if (link) {
+ smu->usb_power_link = link;
+ drm_info(adev_to_drm(adev), "USB controller %s D0 power state depends on %s\n",
+ pci_name(usb_pdev), pci_name(gpu_pdev));
+ /* Only create one link for the first USB controller found */
+ break;
+ }
+ }
+
+ pci_dev_put(usb_pdev);
+
+ return 0;
+}
+
static const struct pptable_funcs smu_v14_0_0_ppt_funcs = {
.check_fw_status = smu_v14_0_check_fw_status,
.check_fw_version = smu_v14_0_check_fw_version,
@@ -1673,6 +1717,7 @@ static const struct pptable_funcs smu_v14_0_0_ppt_funcs = {
.dpm_set_umsch_mm_enable = smu_v14_0_0_set_umsch_mm_enable,
.get_dpm_clock_table = smu_v14_0_common_get_dpm_table,
.set_mall_enable = smu_v14_0_common_set_mall_enable,
+ .set_power_dep = smu_v14_0_0_set_power_dep,
};
static void smu_v14_0_0_set_smu_mailbox_registers(struct smu_context *smu)
diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c
index e60288af3502..ee4cc6721414 100644
--- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c
+++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c
@@ -134,17 +134,17 @@ int etnaviv_sched_push_job(struct etnaviv_gem_submit *submit)
int etnaviv_sched_init(struct etnaviv_gpu *gpu)
{
- int ret;
-
- ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- etnaviv_hw_jobs_limit, etnaviv_job_hang_limit,
- msecs_to_jiffies(500), NULL, NULL,
- dev_name(gpu->dev), gpu->dev);
- if (ret)
- return ret;
-
- return 0;
+ const struct drm_sched_init_args args = {
+ .ops = &etnaviv_sched_ops,
+ .num_rqs = DRM_SCHED_PRIORITY_COUNT,
+ .credit_limit = etnaviv_hw_jobs_limit,
+ .hang_limit = etnaviv_job_hang_limit,
+ .timeout = msecs_to_jiffies(500),
+ .name = dev_name(gpu->dev),
+ .dev = gpu->dev,
+ };
+
+ return drm_sched_init(&gpu->sched, &args);
}
void etnaviv_sched_fini(struct etnaviv_gpu *gpu)
diff --git a/drivers/gpu/drm/exynos/exynos_drm_drv.h b/drivers/gpu/drm/exynos/exynos_drm_drv.h
index 23646e55f142..06c29ff2aac0 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_drv.h
+++ b/drivers/gpu/drm/exynos/exynos_drm_drv.h
@@ -199,6 +199,7 @@ struct drm_exynos_file_private {
struct exynos_drm_private {
struct device *g2d_dev;
struct device *dma_dev;
+ struct device *vidi_dev;
void *mapping;
/* for atomic commit */
diff --git a/drivers/gpu/drm/exynos/exynos_drm_vidi.c b/drivers/gpu/drm/exynos/exynos_drm_vidi.c
index 8400330dfe3e..684170f44989 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_vidi.c
+++ b/drivers/gpu/drm/exynos/exynos_drm_vidi.c
@@ -231,9 +231,14 @@ ATTRIBUTE_GROUPS(vidi);
int vidi_connection_ioctl(struct drm_device *drm_dev, void *data,
struct drm_file *file_priv)
{
- struct vidi_context *ctx = dev_get_drvdata(drm_dev->dev);
+ struct exynos_drm_private *priv = drm_dev->dev_private;
+ struct device *dev = priv ? priv->vidi_dev : NULL;
+ struct vidi_context *ctx = dev ? dev_get_drvdata(dev) : NULL;
struct drm_exynos_vidi_connection *vidi = data;
+ if (!ctx)
+ return -ENODEV;
+
if (!vidi) {
DRM_DEV_DEBUG_KMS(ctx->dev,
"user data for vidi is null.\n");
@@ -407,6 +412,7 @@ static int vidi_bind(struct device *dev, struct device *master, void *data)
{
struct vidi_context *ctx = dev_get_drvdata(dev);
struct drm_device *drm_dev = data;
+ struct exynos_drm_private *priv = drm_dev->dev_private;
struct drm_encoder *encoder = &ctx->encoder;
struct exynos_drm_plane *exynos_plane;
struct exynos_drm_plane_config plane_config = { 0 };
@@ -414,6 +420,8 @@ static int vidi_bind(struct device *dev, struct device *master, void *data)
int ret;
ctx->drm_dev = drm_dev;
+ if (priv)
+ priv->vidi_dev = dev;
plane_config.pixel_formats = formats;
plane_config.num_pixel_formats = ARRAY_SIZE(formats);
@@ -459,8 +467,12 @@ static int vidi_bind(struct device *dev, struct device *master, void *data)
static void vidi_unbind(struct device *dev, struct device *master, void *data)
{
struct vidi_context *ctx = dev_get_drvdata(dev);
+ struct drm_device *drm_dev = data;
+ struct exynos_drm_private *priv = drm_dev->dev_private;
del_timer_sync(&ctx->timer);
+ if (priv)
+ priv->vidi_dev = NULL;
}
static const struct component_ops vidi_component_ops = {
diff --git a/drivers/gpu/drm/gud/gud_drv.c b/drivers/gpu/drm/gud/gud_drv.c
index f1200a2ef880..065d52895bfb 100644
--- a/drivers/gpu/drm/gud/gud_drv.c
+++ b/drivers/gpu/drm/gud/gud_drv.c
@@ -288,6 +288,8 @@ static int gud_get_properties(struct gud_device *gdrm)
* but mask out any additions on future devices.
*/
val &= GUD_ROTATION_MASK;
+ if (!(val & GUD_ROTATION_0))
+ continue;
ret = drm_plane_create_rotation_property(&gdrm->pipe.plane,
DRM_MODE_ROTATE_0, val);
break;
diff --git a/drivers/gpu/drm/i915/display/g4x_dp.c b/drivers/gpu/drm/i915/display/g4x_dp.c
index 9a5beb91dd4b..724d24ef1c78 100644
--- a/drivers/gpu/drm/i915/display/g4x_dp.c
+++ b/drivers/gpu/drm/i915/display/g4x_dp.c
@@ -169,13 +169,12 @@ static void assert_dp_port(struct intel_dp *intel_dp, bool state)
{
struct intel_display *display = to_intel_display(intel_dp);
struct intel_digital_port *dig_port = dp_to_dig_port(intel_dp);
- struct drm_i915_private *dev_priv = to_i915(dig_port->base.base.dev);
bool cur_state = intel_de_read(display, intel_dp->output_reg) & DP_PORT_EN;
- I915_STATE_WARN(dev_priv, cur_state != state,
- "[ENCODER:%d:%s] state assertion failure (expected %s, current %s)\n",
- dig_port->base.base.base.id, dig_port->base.base.name,
- str_on_off(state), str_on_off(cur_state));
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "[ENCODER:%d:%s] state assertion failure (expected %s, current %s)\n",
+ dig_port->base.base.base.id, dig_port->base.base.name,
+ str_on_off(state), str_on_off(cur_state));
}
#define assert_dp_port_disabled(d) assert_dp_port((d), false)
@@ -184,9 +183,9 @@ static void assert_edp_pll(struct drm_i915_private *dev_priv, bool state)
struct intel_display *display = &dev_priv->display;
bool cur_state = intel_de_read(display, DP_A) & DP_PLL_ENABLE;
- I915_STATE_WARN(dev_priv, cur_state != state,
- "eDP PLL state assertion failure (expected %s, current %s)\n",
- str_on_off(state), str_on_off(cur_state));
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "eDP PLL state assertion failure (expected %s, current %s)\n",
+ str_on_off(state), str_on_off(cur_state));
}
#define assert_edp_pll_enabled(d) assert_edp_pll((d), true)
#define assert_edp_pll_disabled(d) assert_edp_pll((d), false)
diff --git a/drivers/gpu/drm/i915/display/i9xx_wm.c b/drivers/gpu/drm/i915/display/i9xx_wm.c
index 15cda57fbc91..9535e92dcbf4 100644
--- a/drivers/gpu/drm/i915/display/i9xx_wm.c
+++ b/drivers/gpu/drm/i915/display/i9xx_wm.c
@@ -715,10 +715,11 @@ static unsigned int g4x_tlb_miss_wa(int fifo_size, int width, int cpp)
static void g4x_write_wm_values(struct drm_i915_private *dev_priv,
const struct g4x_wm_values *wm)
{
+ struct intel_display *display = &dev_priv->display;
enum pipe pipe;
for_each_pipe(dev_priv, pipe)
- trace_g4x_wm(intel_crtc_for_pipe(dev_priv, pipe), wm);
+ trace_g4x_wm(intel_crtc_for_pipe(display, pipe), wm);
intel_uncore_write(&dev_priv->uncore, DSPFW1(dev_priv),
FW_WM(wm->sr.plane, SR) |
@@ -747,10 +748,11 @@ static void g4x_write_wm_values(struct drm_i915_private *dev_priv,
static void vlv_write_wm_values(struct drm_i915_private *dev_priv,
const struct vlv_wm_values *wm)
{
+ struct intel_display *display = &dev_priv->display;
enum pipe pipe;
for_each_pipe(dev_priv, pipe) {
- trace_vlv_wm(intel_crtc_for_pipe(dev_priv, pipe), wm);
+ trace_vlv_wm(intel_crtc_for_pipe(display, pipe), wm);
intel_uncore_write(&dev_priv->uncore, VLV_DDL(pipe),
(wm->ddl[pipe].plane[PLANE_CURSOR] << DDL_CURSOR_SHIFT) |
@@ -2088,12 +2090,13 @@ static void i965_update_wm(struct drm_i915_private *dev_priv)
static struct intel_crtc *intel_crtc_for_plane(struct drm_i915_private *i915,
enum i9xx_plane_id i9xx_plane)
{
+ struct intel_display *display = &i915->display;
struct intel_plane *plane;
for_each_intel_plane(&i915->drm, plane) {
if (plane->id == PLANE_PRIMARY &&
plane->i9xx_plane == i9xx_plane)
- return intel_crtc_for_pipe(i915, plane->pipe);
+ return intel_crtc_for_pipe(display, plane->pipe);
}
return NULL;
@@ -3716,6 +3719,7 @@ static void g4x_wm_get_hw_state(struct drm_i915_private *dev_priv)
static void g4x_wm_sanitize(struct drm_i915_private *dev_priv)
{
+ struct intel_display *display = &dev_priv->display;
struct intel_plane *plane;
struct intel_crtc *crtc;
@@ -3723,7 +3727,7 @@ static void g4x_wm_sanitize(struct drm_i915_private *dev_priv)
for_each_intel_plane(&dev_priv->drm, plane) {
struct intel_crtc *crtc =
- intel_crtc_for_pipe(dev_priv, plane->pipe);
+ intel_crtc_for_pipe(display, plane->pipe);
struct intel_crtc_state *crtc_state =
to_intel_crtc_state(crtc->base.state);
struct intel_plane_state *plane_state =
@@ -3871,6 +3875,7 @@ static void vlv_wm_get_hw_state(struct drm_i915_private *dev_priv)
static void vlv_wm_sanitize(struct drm_i915_private *dev_priv)
{
+ struct intel_display *display = &dev_priv->display;
struct intel_plane *plane;
struct intel_crtc *crtc;
@@ -3878,7 +3883,7 @@ static void vlv_wm_sanitize(struct drm_i915_private *dev_priv)
for_each_intel_plane(&dev_priv->drm, plane) {
struct intel_crtc *crtc =
- intel_crtc_for_pipe(dev_priv, plane->pipe);
+ intel_crtc_for_pipe(display, plane->pipe);
struct intel_crtc_state *crtc_state =
to_intel_crtc_state(crtc->base.state);
struct intel_plane_state *plane_state =
diff --git a/drivers/gpu/drm/i915/display/intel_atomic_plane.c b/drivers/gpu/drm/i915/display/intel_atomic_plane.c
index 5c2a7987cccb..186735b469c5 100644
--- a/drivers/gpu/drm/i915/display/intel_atomic_plane.c
+++ b/drivers/gpu/drm/i915/display/intel_atomic_plane.c
@@ -710,13 +710,13 @@ intel_crtc_get_plane(struct intel_crtc *crtc, enum plane_id plane_id)
int intel_plane_atomic_check(struct intel_atomic_state *state,
struct intel_plane *plane)
{
- struct drm_i915_private *i915 = to_i915(state->base.dev);
+ struct intel_display *display = to_intel_display(state);
struct intel_plane_state *new_plane_state =
intel_atomic_get_new_plane_state(state, plane);
const struct intel_plane_state *old_plane_state =
intel_atomic_get_old_plane_state(state, plane);
const struct intel_plane_state *new_primary_crtc_plane_state;
- struct intel_crtc *crtc = intel_crtc_for_pipe(i915, plane->pipe);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, plane->pipe);
const struct intel_crtc_state *old_crtc_state =
intel_atomic_get_old_crtc_state(state, crtc);
struct intel_crtc_state *new_crtc_state =
diff --git a/drivers/gpu/drm/i915/display/intel_bios.c b/drivers/gpu/drm/i915/display/intel_bios.c
index 4793d460892b..18e8453a339c 100644
--- a/drivers/gpu/drm/i915/display/intel_bios.c
+++ b/drivers/gpu/drm/i915/display/intel_bios.c
@@ -2815,7 +2815,6 @@ static bool child_device_size_valid(struct intel_display *display, int size)
static void
parse_general_definitions(struct intel_display *display)
{
- struct drm_i915_private *i915 = to_i915(display->drm);
const struct bdb_general_definitions *defs;
struct intel_bios_encoder_data *devdata;
const struct child_device_config *child;
@@ -2840,7 +2839,7 @@ parse_general_definitions(struct intel_display *display)
bus_pin = defs->crt_ddc_gmbus_pin;
drm_dbg_kms(display->drm, "crt_ddc_bus_pin: %d\n", bus_pin);
- if (intel_gmbus_is_valid_pin(i915, bus_pin))
+ if (intel_gmbus_is_valid_pin(display, bus_pin))
display->vbt.crt_ddc_pin = bus_pin;
if (!child_device_size_valid(display, defs->child_dev_size))
@@ -3426,7 +3425,6 @@ bool intel_bios_is_tv_present(struct intel_display *display)
*/
bool intel_bios_is_lvds_present(struct intel_display *display, u8 *i2c_pin)
{
- struct drm_i915_private *i915 = to_i915(display->drm);
const struct intel_bios_encoder_data *devdata;
if (list_empty(&display->vbt.display_devices))
@@ -3443,7 +3441,7 @@ bool intel_bios_is_lvds_present(struct intel_display *display, u8 *i2c_pin)
child->device_type != DEVICE_TYPE_LFP)
continue;
- if (intel_gmbus_is_valid_pin(i915, child->i2c_pin))
+ if (intel_gmbus_is_valid_pin(display, child->i2c_pin))
*i2c_pin = child->i2c_pin;
/* However, we cannot trust the BIOS writers to populate
diff --git a/drivers/gpu/drm/i915/display/intel_cdclk.c b/drivers/gpu/drm/i915/display/intel_cdclk.c
index aa3ba66c5307..51cd5375145f 100644
--- a/drivers/gpu/drm/i915/display/intel_cdclk.c
+++ b/drivers/gpu/drm/i915/display/intel_cdclk.c
@@ -2038,6 +2038,7 @@ static void _bxt_set_cdclk(struct drm_i915_private *dev_priv,
const struct intel_cdclk_config *cdclk_config,
enum pipe pipe)
{
+ struct intel_display *display = &dev_priv->display;
int cdclk = cdclk_config->cdclk;
int vco = cdclk_config->vco;
@@ -2063,7 +2064,7 @@ static void _bxt_set_cdclk(struct drm_i915_private *dev_priv,
intel_de_write(dev_priv, CDCLK_CTL, bxt_cdclk_ctl(dev_priv, cdclk_config, pipe));
if (pipe != INVALID_PIPE)
- intel_crtc_wait_for_next_vblank(intel_crtc_for_pipe(dev_priv, pipe));
+ intel_crtc_wait_for_next_vblank(intel_crtc_for_pipe(display, pipe));
}
static void bxt_set_cdclk(struct drm_i915_private *dev_priv,
@@ -3262,6 +3263,7 @@ static bool intel_cdclk_need_serialize(struct drm_i915_private *i915,
int intel_modeset_calc_cdclk(struct intel_atomic_state *state)
{
+ struct intel_display *display = to_intel_display(state);
struct drm_i915_private *dev_priv = to_i915(state->base.dev);
const struct intel_cdclk_state *old_cdclk_state;
struct intel_cdclk_state *new_cdclk_state;
@@ -3308,7 +3310,7 @@ int intel_modeset_calc_cdclk(struct intel_atomic_state *state)
struct intel_crtc_state *crtc_state;
pipe = ilog2(new_cdclk_state->active_pipes);
- crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ crtc = intel_crtc_for_pipe(display, pipe);
crtc_state = intel_atomic_get_crtc_state(&state->base, crtc);
if (IS_ERR(crtc_state))
diff --git a/drivers/gpu/drm/i915/display/intel_crt.c b/drivers/gpu/drm/i915/display/intel_crt.c
index 835c8b844494..1be55bdb48b9 100644
--- a/drivers/gpu/drm/i915/display/intel_crt.c
+++ b/drivers/gpu/drm/i915/display/intel_crt.c
@@ -81,12 +81,13 @@ static struct intel_crt *intel_attached_crt(struct intel_connector *connector)
return intel_encoder_to_crt(intel_attached_encoder(connector));
}
-bool intel_crt_port_enabled(struct drm_i915_private *dev_priv,
+bool intel_crt_port_enabled(struct intel_display *display,
i915_reg_t adpa_reg, enum pipe *pipe)
{
+ struct drm_i915_private *dev_priv = to_i915(display->drm);
u32 val;
- val = intel_de_read(dev_priv, adpa_reg);
+ val = intel_de_read(display, adpa_reg);
/* asserts want to know the pipe even if the port is disabled */
if (HAS_PCH_CPT(dev_priv))
@@ -100,6 +101,7 @@ bool intel_crt_port_enabled(struct drm_i915_private *dev_priv,
static bool intel_crt_get_hw_state(struct intel_encoder *encoder,
enum pipe *pipe)
{
+ struct intel_display *display = to_intel_display(encoder);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_crt *crt = intel_encoder_to_crt(encoder);
intel_wakeref_t wakeref;
@@ -110,7 +112,7 @@ static bool intel_crt_get_hw_state(struct intel_encoder *encoder,
if (!wakeref)
return false;
- ret = intel_crt_port_enabled(dev_priv, crt->adpa_reg, pipe);
+ ret = intel_crt_port_enabled(display, crt->adpa_reg, pipe);
intel_display_power_put(dev_priv, encoder->power_domain, wakeref);
@@ -119,11 +121,11 @@ static bool intel_crt_get_hw_state(struct intel_encoder *encoder,
static unsigned int intel_crt_get_flags(struct intel_encoder *encoder)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_display *display = to_intel_display(encoder);
struct intel_crt *crt = intel_encoder_to_crt(encoder);
u32 tmp, flags = 0;
- tmp = intel_de_read(dev_priv, crt->adpa_reg);
+ tmp = intel_de_read(display, crt->adpa_reg);
if (tmp & ADPA_HSYNC_ACTIVE_HIGH)
flags |= DRM_MODE_FLAG_PHSYNC;
@@ -168,13 +170,14 @@ static void intel_crt_set_dpms(struct intel_encoder *encoder,
const struct intel_crtc_state *crtc_state,
int mode)
{
+ struct intel_display *display = to_intel_display(encoder);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_crt *crt = intel_encoder_to_crt(encoder);
struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc);
const struct drm_display_mode *adjusted_mode = &crtc_state->hw.adjusted_mode;
u32 adpa;
- if (DISPLAY_VER(dev_priv) >= 5)
+ if (DISPLAY_VER(display) >= 5)
adpa = ADPA_HOTPLUG_BITS;
else
adpa = 0;
@@ -193,7 +196,7 @@ static void intel_crt_set_dpms(struct intel_encoder *encoder,
adpa |= ADPA_PIPE_SEL(crtc->pipe);
if (!HAS_PCH_SPLIT(dev_priv))
- intel_de_write(dev_priv, BCLRPAT(dev_priv, crtc->pipe), 0);
+ intel_de_write(display, BCLRPAT(display, crtc->pipe), 0);
switch (mode) {
case DRM_MODE_DPMS_ON:
@@ -210,7 +213,7 @@ static void intel_crt_set_dpms(struct intel_encoder *encoder,
break;
}
- intel_de_write(dev_priv, crt->adpa_reg, adpa);
+ intel_de_write(display, crt->adpa_reg, adpa);
}
static void intel_disable_crt(struct intel_atomic_state *state,
@@ -241,9 +244,10 @@ static void hsw_disable_crt(struct intel_atomic_state *state,
const struct intel_crtc_state *old_crtc_state,
const struct drm_connector_state *old_conn_state)
{
+ struct intel_display *display = to_intel_display(encoder);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
- drm_WARN_ON(&dev_priv->drm, !old_crtc_state->has_pch_encoder);
+ drm_WARN_ON(display->drm, !old_crtc_state->has_pch_encoder);
intel_set_pch_fifo_underrun_reporting(dev_priv, PIPE_A, false);
}
@@ -253,6 +257,7 @@ static void hsw_post_disable_crt(struct intel_atomic_state *state,
const struct intel_crtc_state *old_crtc_state,
const struct drm_connector_state *old_conn_state)
{
+ struct intel_display *display = to_intel_display(encoder);
struct intel_crtc *crtc = to_intel_crtc(old_crtc_state->uapi.crtc);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
@@ -272,7 +277,7 @@ static void hsw_post_disable_crt(struct intel_atomic_state *state,
hsw_fdi_disable(encoder);
- drm_WARN_ON(&dev_priv->drm, !old_crtc_state->has_pch_encoder);
+ drm_WARN_ON(display->drm, !old_crtc_state->has_pch_encoder);
intel_set_pch_fifo_underrun_reporting(dev_priv, PIPE_A, true);
}
@@ -282,9 +287,10 @@ static void hsw_pre_pll_enable_crt(struct intel_atomic_state *state,
const struct intel_crtc_state *crtc_state,
const struct drm_connector_state *conn_state)
{
+ struct intel_display *display = to_intel_display(encoder);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
- drm_WARN_ON(&dev_priv->drm, !crtc_state->has_pch_encoder);
+ drm_WARN_ON(display->drm, !crtc_state->has_pch_encoder);
intel_set_pch_fifo_underrun_reporting(dev_priv, PIPE_A, false);
}
@@ -294,11 +300,12 @@ static void hsw_pre_enable_crt(struct intel_atomic_state *state,
const struct intel_crtc_state *crtc_state,
const struct drm_connector_state *conn_state)
{
+ struct intel_display *display = to_intel_display(encoder);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc);
enum pipe pipe = crtc->pipe;
- drm_WARN_ON(&dev_priv->drm, !crtc_state->has_pch_encoder);
+ drm_WARN_ON(display->drm, !crtc_state->has_pch_encoder);
intel_set_cpu_fifo_underrun_reporting(dev_priv, pipe, false);
@@ -312,11 +319,12 @@ static void hsw_enable_crt(struct intel_atomic_state *state,
const struct intel_crtc_state *crtc_state,
const struct drm_connector_state *conn_state)
{
+ struct intel_display *display = to_intel_display(encoder);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_crtc *crtc = to_intel_crtc(crtc_state->uapi.crtc);
enum pipe pipe = crtc->pipe;
- drm_WARN_ON(&dev_priv->drm, !crtc_state->has_pch_encoder);
+ drm_WARN_ON(display->drm, !crtc_state->has_pch_encoder);
intel_ddi_enable_transcoder_func(encoder, crtc_state);
@@ -346,9 +354,10 @@ static enum drm_mode_status
intel_crt_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
+ struct intel_display *display = to_intel_display(connector->dev);
struct drm_device *dev = connector->dev;
struct drm_i915_private *dev_priv = to_i915(dev);
- int max_dotclk = dev_priv->display.cdclk.max_dotclk_freq;
+ int max_dotclk = display->cdclk.max_dotclk_freq;
enum drm_mode_status status;
int max_clock;
@@ -367,7 +376,7 @@ intel_crt_mode_valid(struct drm_connector *connector,
* DAC limit supposedly 355 MHz.
*/
max_clock = 270000;
- else if (IS_DISPLAY_VER(dev_priv, 3, 4))
+ else if (IS_DISPLAY_VER(display, 3, 4))
max_clock = 400000;
else
max_clock = 350000;
@@ -428,6 +437,7 @@ static int hsw_crt_compute_config(struct intel_encoder *encoder,
struct intel_crtc_state *pipe_config,
struct drm_connector_state *conn_state)
{
+ struct intel_display *display = to_intel_display(encoder);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct drm_display_mode *adjusted_mode =
&pipe_config->hw.adjusted_mode;
@@ -450,7 +460,7 @@ static int hsw_crt_compute_config(struct intel_encoder *encoder,
if (HAS_PCH_LPT(dev_priv)) {
/* TODO: Check crtc_state->max_link_bpp_x16 instead of bw_constrained */
if (pipe_config->bw_constrained && pipe_config->pipe_bpp < 24) {
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"LPT only supports 24bpp\n");
return -EINVAL;
}
@@ -470,6 +480,7 @@ static int hsw_crt_compute_config(struct intel_encoder *encoder,
static bool ilk_crt_detect_hotplug(struct drm_connector *connector)
{
+ struct intel_display *display = to_intel_display(connector->dev);
struct drm_device *dev = connector->dev;
struct intel_crt *crt = intel_attached_crt(to_intel_connector(connector));
struct drm_i915_private *dev_priv = to_i915(dev);
@@ -483,36 +494,36 @@ static bool ilk_crt_detect_hotplug(struct drm_connector *connector)
crt->force_hotplug_required = false;
- save_adpa = adpa = intel_de_read(dev_priv, crt->adpa_reg);
- drm_dbg_kms(&dev_priv->drm,
+ save_adpa = adpa = intel_de_read(display, crt->adpa_reg);
+ drm_dbg_kms(display->drm,
"trigger hotplug detect cycle: adpa=0x%x\n", adpa);
adpa |= ADPA_CRT_HOTPLUG_FORCE_TRIGGER;
if (turn_off_dac)
adpa &= ~ADPA_DAC_ENABLE;
- intel_de_write(dev_priv, crt->adpa_reg, adpa);
+ intel_de_write(display, crt->adpa_reg, adpa);
- if (intel_de_wait_for_clear(dev_priv,
+ if (intel_de_wait_for_clear(display,
crt->adpa_reg,
ADPA_CRT_HOTPLUG_FORCE_TRIGGER,
1000))
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"timed out waiting for FORCE_TRIGGER");
if (turn_off_dac) {
- intel_de_write(dev_priv, crt->adpa_reg, save_adpa);
- intel_de_posting_read(dev_priv, crt->adpa_reg);
+ intel_de_write(display, crt->adpa_reg, save_adpa);
+ intel_de_posting_read(display, crt->adpa_reg);
}
}
/* Check the status to see if both blue and green are on now */
- adpa = intel_de_read(dev_priv, crt->adpa_reg);
+ adpa = intel_de_read(display, crt->adpa_reg);
if ((adpa & ADPA_CRT_HOTPLUG_MONITOR_MASK) != 0)
ret = true;
else
ret = false;
- drm_dbg_kms(&dev_priv->drm, "ironlake hotplug adpa=0x%x, result %d\n",
+ drm_dbg_kms(display->drm, "ironlake hotplug adpa=0x%x, result %d\n",
adpa, ret);
return ret;
@@ -520,6 +531,7 @@ static bool ilk_crt_detect_hotplug(struct drm_connector *connector)
static bool valleyview_crt_detect_hotplug(struct drm_connector *connector)
{
+ struct intel_display *display = to_intel_display(connector->dev);
struct drm_device *dev = connector->dev;
struct intel_crt *crt = intel_attached_crt(to_intel_connector(connector));
struct drm_i915_private *dev_priv = to_i915(dev);
@@ -542,29 +554,29 @@ static bool valleyview_crt_detect_hotplug(struct drm_connector *connector)
*/
reenable_hpd = intel_hpd_disable(dev_priv, crt->base.hpd_pin);
- save_adpa = adpa = intel_de_read(dev_priv, crt->adpa_reg);
- drm_dbg_kms(&dev_priv->drm,
+ save_adpa = adpa = intel_de_read(display, crt->adpa_reg);
+ drm_dbg_kms(display->drm,
"trigger hotplug detect cycle: adpa=0x%x\n", adpa);
adpa |= ADPA_CRT_HOTPLUG_FORCE_TRIGGER;
- intel_de_write(dev_priv, crt->adpa_reg, adpa);
+ intel_de_write(display, crt->adpa_reg, adpa);
- if (intel_de_wait_for_clear(dev_priv, crt->adpa_reg,
+ if (intel_de_wait_for_clear(display, crt->adpa_reg,
ADPA_CRT_HOTPLUG_FORCE_TRIGGER, 1000)) {
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"timed out waiting for FORCE_TRIGGER");
- intel_de_write(dev_priv, crt->adpa_reg, save_adpa);
+ intel_de_write(display, crt->adpa_reg, save_adpa);
}
/* Check the status to see if both blue and green are on now */
- adpa = intel_de_read(dev_priv, crt->adpa_reg);
+ adpa = intel_de_read(display, crt->adpa_reg);
if ((adpa & ADPA_CRT_HOTPLUG_MONITOR_MASK) != 0)
ret = true;
else
ret = false;
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"valleyview hotplug adpa=0x%x, result %d\n", adpa, ret);
if (reenable_hpd)
@@ -575,6 +587,7 @@ static bool valleyview_crt_detect_hotplug(struct drm_connector *connector)
static bool intel_crt_detect_hotplug(struct drm_connector *connector)
{
+ struct intel_display *display = to_intel_display(connector->dev);
struct drm_device *dev = connector->dev;
struct drm_i915_private *dev_priv = to_i915(dev);
u32 stat;
@@ -603,18 +616,18 @@ static bool intel_crt_detect_hotplug(struct drm_connector *connector)
CRT_HOTPLUG_FORCE_DETECT,
CRT_HOTPLUG_FORCE_DETECT);
/* wait for FORCE_DETECT to go off */
- if (intel_de_wait_for_clear(dev_priv, PORT_HOTPLUG_EN(dev_priv),
+ if (intel_de_wait_for_clear(display, PORT_HOTPLUG_EN(display),
CRT_HOTPLUG_FORCE_DETECT, 1000))
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"timed out waiting for FORCE_DETECT to go off");
}
- stat = intel_de_read(dev_priv, PORT_HOTPLUG_STAT(dev_priv));
+ stat = intel_de_read(display, PORT_HOTPLUG_STAT(display));
if ((stat & CRT_HOTPLUG_MONITOR_MASK) != CRT_HOTPLUG_MONITOR_NONE)
ret = true;
/* clear the interrupt we just generated, if any */
- intel_de_write(dev_priv, PORT_HOTPLUG_STAT(dev_priv),
+ intel_de_write(display, PORT_HOTPLUG_STAT(display),
CRT_HOTPLUG_INT_STATUS);
i915_hotplug_interrupt_update(dev_priv, CRT_HOTPLUG_FORCE_DETECT, 0);
@@ -660,8 +673,7 @@ static int intel_crt_ddc_get_modes(struct drm_connector *connector,
static bool intel_crt_detect_ddc(struct drm_connector *connector)
{
- struct intel_crt *crt = intel_attached_crt(to_intel_connector(connector));
- struct drm_i915_private *dev_priv = to_i915(crt->base.base.dev);
+ struct intel_display *display = to_intel_display(connector->dev);
const struct drm_edid *drm_edid;
bool ret = false;
@@ -674,15 +686,15 @@ static bool intel_crt_detect_ddc(struct drm_connector *connector)
* have to check the EDID input spec of the attached device.
*/
if (drm_edid_is_digital(drm_edid)) {
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"CRT not detected via DDC:0x50 [EDID reports a digital panel]\n");
} else {
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"CRT detected via DDC:0x50 [EDID]\n");
ret = true;
}
} else {
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"CRT not detected via DDC:0x50 [no valid EDID found]\n");
}
@@ -694,8 +706,7 @@ static bool intel_crt_detect_ddc(struct drm_connector *connector)
static enum drm_connector_status
intel_crt_load_detect(struct intel_crt *crt, enum pipe pipe)
{
- struct drm_device *dev = crt->base.base.dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
+ struct intel_display *display = to_intel_display(&crt->base);
enum transcoder cpu_transcoder = (enum transcoder)pipe;
u32 save_bclrpat;
u32 save_vtotal;
@@ -706,14 +717,14 @@ intel_crt_load_detect(struct intel_crt *crt, enum pipe pipe)
u8 st00;
enum drm_connector_status status;
- drm_dbg_kms(&dev_priv->drm, "starting load-detect on CRT\n");
+ drm_dbg_kms(display->drm, "starting load-detect on CRT\n");
- save_bclrpat = intel_de_read(dev_priv,
- BCLRPAT(dev_priv, cpu_transcoder));
- save_vtotal = intel_de_read(dev_priv,
- TRANS_VTOTAL(dev_priv, cpu_transcoder));
- vblank = intel_de_read(dev_priv,
- TRANS_VBLANK(dev_priv, cpu_transcoder));
+ save_bclrpat = intel_de_read(display,
+ BCLRPAT(display, cpu_transcoder));
+ save_vtotal = intel_de_read(display,
+ TRANS_VTOTAL(display, cpu_transcoder));
+ vblank = intel_de_read(display,
+ TRANS_VBLANK(display, cpu_transcoder));
vtotal = REG_FIELD_GET(VTOTAL_MASK, save_vtotal) + 1;
vactive = REG_FIELD_GET(VACTIVE_MASK, save_vtotal) + 1;
@@ -722,25 +733,25 @@ intel_crt_load_detect(struct intel_crt *crt, enum pipe pipe)
vblank_end = REG_FIELD_GET(VBLANK_END_MASK, vblank) + 1;
/* Set the border color to purple. */
- intel_de_write(dev_priv, BCLRPAT(dev_priv, cpu_transcoder), 0x500050);
+ intel_de_write(display, BCLRPAT(display, cpu_transcoder), 0x500050);
- if (DISPLAY_VER(dev_priv) != 2) {
- u32 transconf = intel_de_read(dev_priv,
- TRANSCONF(dev_priv, cpu_transcoder));
+ if (DISPLAY_VER(display) != 2) {
+ u32 transconf = intel_de_read(display,
+ TRANSCONF(display, cpu_transcoder));
- intel_de_write(dev_priv, TRANSCONF(dev_priv, cpu_transcoder),
+ intel_de_write(display, TRANSCONF(display, cpu_transcoder),
transconf | TRANSCONF_FORCE_BORDER);
- intel_de_posting_read(dev_priv,
- TRANSCONF(dev_priv, cpu_transcoder));
+ intel_de_posting_read(display,
+ TRANSCONF(display, cpu_transcoder));
/* Wait for next Vblank to substitue
* border color for Color info */
- intel_crtc_wait_for_next_vblank(intel_crtc_for_pipe(dev_priv, pipe));
- st00 = intel_de_read8(dev_priv, _VGA_MSR_WRITE);
+ intel_crtc_wait_for_next_vblank(intel_crtc_for_pipe(display, pipe));
+ st00 = intel_de_read8(display, _VGA_MSR_WRITE);
status = ((st00 & (1 << 4)) != 0) ?
connector_status_connected :
connector_status_disconnected;
- intel_de_write(dev_priv, TRANSCONF(dev_priv, cpu_transcoder),
+ intel_de_write(display, TRANSCONF(display, cpu_transcoder),
transconf);
} else {
bool restore_vblank = false;
@@ -751,13 +762,13 @@ intel_crt_load_detect(struct intel_crt *crt, enum pipe pipe)
* Yes, this will flicker
*/
if (vblank_start <= vactive && vblank_end >= vtotal) {
- u32 vsync = intel_de_read(dev_priv,
- TRANS_VSYNC(dev_priv, cpu_transcoder));
+ u32 vsync = intel_de_read(display,
+ TRANS_VSYNC(display, cpu_transcoder));
u32 vsync_start = REG_FIELD_GET(VSYNC_START_MASK, vsync) + 1;
vblank_start = vsync_start;
- intel_de_write(dev_priv,
- TRANS_VBLANK(dev_priv, cpu_transcoder),
+ intel_de_write(display,
+ TRANS_VBLANK(display, cpu_transcoder),
VBLANK_START(vblank_start - 1) |
VBLANK_END(vblank_end - 1));
restore_vblank = true;
@@ -771,9 +782,9 @@ intel_crt_load_detect(struct intel_crt *crt, enum pipe pipe)
/*
* Wait for the border to be displayed
*/
- while (intel_de_read(dev_priv, PIPEDSL(dev_priv, pipe)) >= vactive)
+ while (intel_de_read(display, PIPEDSL(display, pipe)) >= vactive)
;
- while ((dsl = intel_de_read(dev_priv, PIPEDSL(dev_priv, pipe))) <= vsample)
+ while ((dsl = intel_de_read(display, PIPEDSL(display, pipe))) <= vsample)
;
/*
* Watch ST00 for an entire scanline
@@ -783,15 +794,15 @@ intel_crt_load_detect(struct intel_crt *crt, enum pipe pipe)
do {
count++;
/* Read the ST00 VGA status register */
- st00 = intel_de_read8(dev_priv, _VGA_MSR_WRITE);
+ st00 = intel_de_read8(display, _VGA_MSR_WRITE);
if (st00 & (1 << 4))
detect++;
- } while ((intel_de_read(dev_priv, PIPEDSL(dev_priv, pipe)) == dsl));
+ } while ((intel_de_read(display, PIPEDSL(display, pipe)) == dsl));
/* restore vblank if necessary */
if (restore_vblank)
- intel_de_write(dev_priv,
- TRANS_VBLANK(dev_priv, cpu_transcoder),
+ intel_de_write(display,
+ TRANS_VBLANK(display, cpu_transcoder),
vblank);
/*
* If more than 3/4 of the scanline detected a monitor,
@@ -805,7 +816,7 @@ intel_crt_load_detect(struct intel_crt *crt, enum pipe pipe)
}
/* Restore previous settings */
- intel_de_write(dev_priv, BCLRPAT(dev_priv, cpu_transcoder),
+ intel_de_write(display, BCLRPAT(display, cpu_transcoder),
save_bclrpat);
return status;
@@ -842,6 +853,7 @@ intel_crt_detect(struct drm_connector *connector,
struct drm_modeset_acquire_ctx *ctx,
bool force)
{
+ struct intel_display *display = to_intel_display(connector->dev);
struct drm_i915_private *dev_priv = to_i915(connector->dev);
struct intel_crt *crt = intel_attached_crt(to_intel_connector(connector));
struct intel_encoder *intel_encoder = &crt->base;
@@ -849,7 +861,7 @@ intel_crt_detect(struct drm_connector *connector,
intel_wakeref_t wakeref;
int status;
- drm_dbg_kms(&dev_priv->drm, "[CONNECTOR:%d:%s] force=%d\n",
+ drm_dbg_kms(display->drm, "[CONNECTOR:%d:%s] force=%d\n",
connector->base.id, connector->name,
force);
@@ -859,7 +871,7 @@ intel_crt_detect(struct drm_connector *connector,
if (!intel_display_driver_check_access(dev_priv))
return connector->status;
- if (dev_priv->display.params.load_detect_test) {
+ if (display->params.load_detect_test) {
wakeref = intel_display_power_get(dev_priv,
intel_encoder->power_domain);
goto load_detect;
@@ -872,18 +884,18 @@ intel_crt_detect(struct drm_connector *connector,
wakeref = intel_display_power_get(dev_priv,
intel_encoder->power_domain);
- if (I915_HAS_HOTPLUG(dev_priv)) {
+ if (I915_HAS_HOTPLUG(display)) {
/* We can not rely on the HPD pin always being correctly wired
* up, for example many KVM do not pass it through, and so
* only trust an assertion that the monitor is connected.
*/
if (intel_crt_detect_hotplug(connector)) {
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"CRT detected via hotplug\n");
status = connector_status_connected;
goto out;
} else
- drm_dbg_kms(&dev_priv->drm,
+ drm_dbg_kms(display->drm,
"CRT not detected via hotplug\n");
}
@@ -896,7 +908,7 @@ intel_crt_detect(struct drm_connector *connector,
* broken monitor (without edid) to work behind a broken kvm (that fails
* to have the right resistors for HP detection) needs to fix this up.
* For now just bail out. */
- if (I915_HAS_HOTPLUG(dev_priv)) {
+ if (I915_HAS_HOTPLUG(display)) {
status = connector_status_disconnected;
goto out;
}
@@ -916,10 +928,10 @@ intel_crt_detect(struct drm_connector *connector,
} else {
if (intel_crt_detect_ddc(connector))
status = connector_status_connected;
- else if (DISPLAY_VER(dev_priv) < 4)
+ else if (DISPLAY_VER(display) < 4)
status = intel_crt_load_detect(crt,
to_intel_crtc(connector->state->crtc)->pipe);
- else if (dev_priv->display.params.load_detect_test)
+ else if (display->params.load_detect_test)
status = connector_status_disconnected;
else
status = connector_status_unknown;
@@ -934,6 +946,7 @@ intel_crt_detect(struct drm_connector *connector,
static int intel_crt_get_modes(struct drm_connector *connector)
{
+ struct intel_display *display = to_intel_display(connector->dev);
struct drm_device *dev = connector->dev;
struct drm_i915_private *dev_priv = to_i915(dev);
struct intel_crt *crt = intel_attached_crt(to_intel_connector(connector));
@@ -953,7 +966,7 @@ static int intel_crt_get_modes(struct drm_connector *connector)
goto out;
/* Try to probe digital port for output in DVI-I -> VGA mode. */
- ddc = intel_gmbus_get_adapter(dev_priv, GMBUS_PIN_DPB);
+ ddc = intel_gmbus_get_adapter(display, GMBUS_PIN_DPB);
ret = intel_crt_ddc_get_modes(connector, ddc);
out:
@@ -964,19 +977,19 @@ static int intel_crt_get_modes(struct drm_connector *connector)
void intel_crt_reset(struct drm_encoder *encoder)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->dev);
+ struct intel_display *display = to_intel_display(encoder->dev);
struct intel_crt *crt = intel_encoder_to_crt(to_intel_encoder(encoder));
- if (DISPLAY_VER(dev_priv) >= 5) {
+ if (DISPLAY_VER(display) >= 5) {
u32 adpa;
- adpa = intel_de_read(dev_priv, crt->adpa_reg);
+ adpa = intel_de_read(display, crt->adpa_reg);
adpa &= ~ADPA_CRT_HOTPLUG_MASK;
adpa |= ADPA_HOTPLUG_BITS;
- intel_de_write(dev_priv, crt->adpa_reg, adpa);
- intel_de_posting_read(dev_priv, crt->adpa_reg);
+ intel_de_write(display, crt->adpa_reg, adpa);
+ intel_de_posting_read(display, crt->adpa_reg);
- drm_dbg_kms(&dev_priv->drm, "crt adpa set to 0x%x\n", adpa);
+ drm_dbg_kms(display->drm, "crt adpa set to 0x%x\n", adpa);
crt->force_hotplug_required = true;
}
@@ -1006,8 +1019,9 @@ static const struct drm_encoder_funcs intel_crt_enc_funcs = {
.destroy = intel_encoder_destroy,
};
-void intel_crt_init(struct drm_i915_private *dev_priv)
+void intel_crt_init(struct intel_display *display)
{
+ struct drm_i915_private *dev_priv = to_i915(display->drm);
struct drm_connector *connector;
struct intel_crt *crt;
struct intel_connector *intel_connector;
@@ -1022,7 +1036,7 @@ void intel_crt_init(struct drm_i915_private *dev_priv)
else
adpa_reg = ADPA;
- adpa = intel_de_read(dev_priv, adpa_reg);
+ adpa = intel_de_read(display, adpa_reg);
if ((adpa & ADPA_DAC_ENABLE) == 0) {
/*
* On some machines (some IVB at least) CRT can be
@@ -1032,11 +1046,11 @@ void intel_crt_init(struct drm_i915_private *dev_priv)
* take. So the only way to tell is attempt to enable
* it and see what happens.
*/
- intel_de_write(dev_priv, adpa_reg,
+ intel_de_write(display, adpa_reg,
adpa | ADPA_DAC_ENABLE | ADPA_HSYNC_CNTL_DISABLE | ADPA_VSYNC_CNTL_DISABLE);
- if ((intel_de_read(dev_priv, adpa_reg) & ADPA_DAC_ENABLE) == 0)
+ if ((intel_de_read(display, adpa_reg) & ADPA_DAC_ENABLE) == 0)
return;
- intel_de_write(dev_priv, adpa_reg, adpa);
+ intel_de_write(display, adpa_reg, adpa);
}
crt = kzalloc(sizeof(struct intel_crt), GFP_KERNEL);
@@ -1049,16 +1063,16 @@ void intel_crt_init(struct drm_i915_private *dev_priv)
return;
}
- ddc_pin = dev_priv->display.vbt.crt_ddc_pin;
+ ddc_pin = display->vbt.crt_ddc_pin;
connector = &intel_connector->base;
crt->connector = intel_connector;
- drm_connector_init_with_ddc(&dev_priv->drm, connector,
+ drm_connector_init_with_ddc(display->drm, connector,
&intel_crt_connector_funcs,
DRM_MODE_CONNECTOR_VGA,
- intel_gmbus_get_adapter(dev_priv, ddc_pin));
+ intel_gmbus_get_adapter(display, ddc_pin));
- drm_encoder_init(&dev_priv->drm, &crt->base.base, &intel_crt_enc_funcs,
+ drm_encoder_init(display->drm, &crt->base.base, &intel_crt_enc_funcs,
DRM_MODE_ENCODER_DAC, "CRT");
intel_connector_attach_encoder(intel_connector, &crt->base);
@@ -1070,14 +1084,14 @@ void intel_crt_init(struct drm_i915_private *dev_priv)
else
crt->base.pipe_mask = ~0;
- if (DISPLAY_VER(dev_priv) != 2)
+ if (DISPLAY_VER(display) != 2)
connector->interlace_allowed = true;
crt->adpa_reg = adpa_reg;
crt->base.power_domain = POWER_DOMAIN_PORT_CRT;
- if (I915_HAS_HOTPLUG(dev_priv) &&
+ if (I915_HAS_HOTPLUG(display) &&
!dmi_check_system(intel_spurious_crt_detect)) {
crt->base.hpd_pin = HPD_CRT;
crt->base.hotplug = intel_encoder_hotplug;
@@ -1087,7 +1101,7 @@ void intel_crt_init(struct drm_i915_private *dev_priv)
}
intel_connector->base.polled = intel_connector->polled;
- if (HAS_DDI(dev_priv)) {
+ if (HAS_DDI(display)) {
assert_port_valid(dev_priv, PORT_E);
crt->base.port = PORT_E;
@@ -1131,8 +1145,8 @@ void intel_crt_init(struct drm_i915_private *dev_priv)
u32 fdi_config = FDI_RX_POLARITY_REVERSED_LPT |
FDI_RX_LINK_REVERSAL_OVERRIDE;
- dev_priv->display.fdi.rx_config = intel_de_read(dev_priv,
- FDI_RX_CTL(PIPE_A)) & fdi_config;
+ display->fdi.rx_config = intel_de_read(display,
+ FDI_RX_CTL(PIPE_A)) & fdi_config;
}
intel_crt_reset(&crt->base.base);
diff --git a/drivers/gpu/drm/i915/display/intel_crt.h b/drivers/gpu/drm/i915/display/intel_crt.h
index fe7690c2b948..e0abfe96a3d2 100644
--- a/drivers/gpu/drm/i915/display/intel_crt.h
+++ b/drivers/gpu/drm/i915/display/intel_crt.h
@@ -10,20 +10,20 @@
enum pipe;
struct drm_encoder;
-struct drm_i915_private;
+struct intel_display;
#ifdef I915
-bool intel_crt_port_enabled(struct drm_i915_private *dev_priv,
+bool intel_crt_port_enabled(struct intel_display *display,
i915_reg_t adpa_reg, enum pipe *pipe);
-void intel_crt_init(struct drm_i915_private *dev_priv);
+void intel_crt_init(struct intel_display *display);
void intel_crt_reset(struct drm_encoder *encoder);
#else
-static inline bool intel_crt_port_enabled(struct drm_i915_private *dev_priv,
+static inline bool intel_crt_port_enabled(struct intel_display *display,
i915_reg_t adpa_reg, enum pipe *pipe)
{
return false;
}
-static inline void intel_crt_init(struct drm_i915_private *dev_priv)
+static inline void intel_crt_init(struct intel_display *display)
{
}
static inline void intel_crt_reset(struct drm_encoder *encoder)
diff --git a/drivers/gpu/drm/i915/display/intel_crtc.c b/drivers/gpu/drm/i915/display/intel_crtc.c
index 1b578cad2813..b2faa92a7ecc 100644
--- a/drivers/gpu/drm/i915/display/intel_crtc.c
+++ b/drivers/gpu/drm/i915/display/intel_crtc.c
@@ -35,11 +35,11 @@
static void assert_vblank_disabled(struct drm_crtc *crtc)
{
- struct drm_i915_private *i915 = to_i915(crtc->dev);
+ struct intel_display *display = to_intel_display(crtc->dev);
- if (I915_STATE_WARN(i915, drm_crtc_vblank_get(crtc) == 0,
- "[CRTC:%d:%s] vblank assertion failure (expected off, current on)\n",
- crtc->base.id, crtc->name))
+ if (INTEL_DISPLAY_STATE_WARN(display, drm_crtc_vblank_get(crtc) == 0,
+ "[CRTC:%d:%s] vblank assertion failure (expected off, current on)\n",
+ crtc->base.id, crtc->name))
drm_crtc_vblank_put(crtc);
}
@@ -48,12 +48,12 @@ struct intel_crtc *intel_first_crtc(struct drm_i915_private *i915)
return to_intel_crtc(drm_crtc_from_index(&i915->drm, 0));
}
-struct intel_crtc *intel_crtc_for_pipe(struct drm_i915_private *i915,
+struct intel_crtc *intel_crtc_for_pipe(struct intel_display *display,
enum pipe pipe)
{
struct intel_crtc *crtc;
- for_each_intel_crtc(&i915->drm, crtc) {
+ for_each_intel_crtc(display->drm, crtc) {
if (crtc->pipe == pipe)
return crtc;
}
@@ -69,7 +69,8 @@ void intel_crtc_wait_for_next_vblank(struct intel_crtc *crtc)
void intel_wait_for_vblank_if_active(struct drm_i915_private *i915,
enum pipe pipe)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(i915, pipe);
+ struct intel_display *display = &i915->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
if (crtc->active)
intel_crtc_wait_for_next_vblank(crtc);
diff --git a/drivers/gpu/drm/i915/display/intel_crtc.h b/drivers/gpu/drm/i915/display/intel_crtc.h
index b615b7ab5ccd..0de8c772df2e 100644
--- a/drivers/gpu/drm/i915/display/intel_crtc.h
+++ b/drivers/gpu/drm/i915/display/intel_crtc.h
@@ -15,6 +15,7 @@ struct drm_i915_private;
struct intel_atomic_state;
struct intel_crtc;
struct intel_crtc_state;
+struct intel_display;
/*
* FIXME: We should instead only take spinlocks once for the entire update
@@ -43,7 +44,7 @@ void intel_pipe_update_end(struct intel_atomic_state *state,
struct intel_crtc *crtc);
void intel_wait_for_vblank_workers(struct intel_atomic_state *state);
struct intel_crtc *intel_first_crtc(struct drm_i915_private *i915);
-struct intel_crtc *intel_crtc_for_pipe(struct drm_i915_private *i915,
+struct intel_crtc *intel_crtc_for_pipe(struct intel_display *display,
enum pipe pipe);
void intel_wait_for_vblank_if_active(struct drm_i915_private *i915,
enum pipe pipe);
diff --git a/drivers/gpu/drm/i915/display/intel_cx0_phy.c b/drivers/gpu/drm/i915/display/intel_cx0_phy.c
index f11309efff33..c7baf4255c19 100644
--- a/drivers/gpu/drm/i915/display/intel_cx0_phy.c
+++ b/drivers/gpu/drm/i915/display/intel_cx0_phy.c
@@ -3246,28 +3246,28 @@ static void intel_c10pll_state_verify(const struct intel_crtc_state *state,
struct intel_encoder *encoder,
struct intel_c10pll_state *mpllb_hw_state)
{
- struct drm_i915_private *i915 = to_i915(crtc->base.dev);
+ struct intel_display *display = to_intel_display(state);
const struct intel_c10pll_state *mpllb_sw_state = &state->dpll_hw_state.cx0pll.c10;
int i;
for (i = 0; i < ARRAY_SIZE(mpllb_sw_state->pll); i++) {
u8 expected = mpllb_sw_state->pll[i];
- I915_STATE_WARN(i915, mpllb_hw_state->pll[i] != expected,
- "[CRTC:%d:%s] mismatch in C10MPLLB: Register[%d] (expected 0x%02x, found 0x%02x)",
- crtc->base.base.id, crtc->base.name, i,
- expected, mpllb_hw_state->pll[i]);
+ INTEL_DISPLAY_STATE_WARN(display, mpllb_hw_state->pll[i] != expected,
+ "[CRTC:%d:%s] mismatch in C10MPLLB: Register[%d] (expected 0x%02x, found 0x%02x)",
+ crtc->base.base.id, crtc->base.name, i,
+ expected, mpllb_hw_state->pll[i]);
}
- I915_STATE_WARN(i915, mpllb_hw_state->tx != mpllb_sw_state->tx,
- "[CRTC:%d:%s] mismatch in C10MPLLB: Register TX0 (expected 0x%02x, found 0x%02x)",
- crtc->base.base.id, crtc->base.name,
- mpllb_sw_state->tx, mpllb_hw_state->tx);
+ INTEL_DISPLAY_STATE_WARN(display, mpllb_hw_state->tx != mpllb_sw_state->tx,
+ "[CRTC:%d:%s] mismatch in C10MPLLB: Register TX0 (expected 0x%02x, found 0x%02x)",
+ crtc->base.base.id, crtc->base.name,
+ mpllb_sw_state->tx, mpllb_hw_state->tx);
- I915_STATE_WARN(i915, mpllb_hw_state->cmn != mpllb_sw_state->cmn,
- "[CRTC:%d:%s] mismatch in C10MPLLB: Register CMN0 (expected 0x%02x, found 0x%02x)",
- crtc->base.base.id, crtc->base.name,
- mpllb_sw_state->cmn, mpllb_hw_state->cmn);
+ INTEL_DISPLAY_STATE_WARN(display, mpllb_hw_state->cmn != mpllb_sw_state->cmn,
+ "[CRTC:%d:%s] mismatch in C10MPLLB: Register CMN0 (expected 0x%02x, found 0x%02x)",
+ crtc->base.base.id, crtc->base.name,
+ mpllb_sw_state->cmn, mpllb_hw_state->cmn);
}
void intel_cx0pll_readout_hw_state(struct intel_encoder *encoder,
@@ -3353,51 +3353,51 @@ static void intel_c20pll_state_verify(const struct intel_crtc_state *state,
struct intel_encoder *encoder,
struct intel_c20pll_state *mpll_hw_state)
{
- struct drm_i915_private *i915 = to_i915(crtc->base.dev);
+ struct intel_display *display = to_intel_display(state);
const struct intel_c20pll_state *mpll_sw_state = &state->dpll_hw_state.cx0pll.c20;
bool sw_use_mpllb = intel_c20phy_use_mpllb(mpll_sw_state);
bool hw_use_mpllb = intel_c20phy_use_mpllb(mpll_hw_state);
int clock = intel_c20pll_calc_port_clock(encoder, mpll_sw_state);
int i;
- I915_STATE_WARN(i915, mpll_hw_state->clock != clock,
- "[CRTC:%d:%s] mismatch in C20: Register CLOCK (expected %d, found %d)",
- crtc->base.base.id, crtc->base.name,
- mpll_sw_state->clock, mpll_hw_state->clock);
+ INTEL_DISPLAY_STATE_WARN(display, mpll_hw_state->clock != clock,
+ "[CRTC:%d:%s] mismatch in C20: Register CLOCK (expected %d, found %d)",
+ crtc->base.base.id, crtc->base.name,
+ mpll_sw_state->clock, mpll_hw_state->clock);
- I915_STATE_WARN(i915, sw_use_mpllb != hw_use_mpllb,
- "[CRTC:%d:%s] mismatch in C20: Register MPLLB selection (expected %d, found %d)",
- crtc->base.base.id, crtc->base.name,
- sw_use_mpllb, hw_use_mpllb);
+ INTEL_DISPLAY_STATE_WARN(display, sw_use_mpllb != hw_use_mpllb,
+ "[CRTC:%d:%s] mismatch in C20: Register MPLLB selection (expected %d, found %d)",
+ crtc->base.base.id, crtc->base.name,
+ sw_use_mpllb, hw_use_mpllb);
if (hw_use_mpllb) {
for (i = 0; i < ARRAY_SIZE(mpll_sw_state->mpllb); i++) {
- I915_STATE_WARN(i915, mpll_hw_state->mpllb[i] != mpll_sw_state->mpllb[i],
- "[CRTC:%d:%s] mismatch in C20MPLLB: Register[%d] (expected 0x%04x, found 0x%04x)",
- crtc->base.base.id, crtc->base.name, i,
- mpll_sw_state->mpllb[i], mpll_hw_state->mpllb[i]);
+ INTEL_DISPLAY_STATE_WARN(display, mpll_hw_state->mpllb[i] != mpll_sw_state->mpllb[i],
+ "[CRTC:%d:%s] mismatch in C20MPLLB: Register[%d] (expected 0x%04x, found 0x%04x)",
+ crtc->base.base.id, crtc->base.name, i,
+ mpll_sw_state->mpllb[i], mpll_hw_state->mpllb[i]);
}
} else {
for (i = 0; i < ARRAY_SIZE(mpll_sw_state->mplla); i++) {
- I915_STATE_WARN(i915, mpll_hw_state->mplla[i] != mpll_sw_state->mplla[i],
- "[CRTC:%d:%s] mismatch in C20MPLLA: Register[%d] (expected 0x%04x, found 0x%04x)",
- crtc->base.base.id, crtc->base.name, i,
- mpll_sw_state->mplla[i], mpll_hw_state->mplla[i]);
+ INTEL_DISPLAY_STATE_WARN(display, mpll_hw_state->mplla[i] != mpll_sw_state->mplla[i],
+ "[CRTC:%d:%s] mismatch in C20MPLLA: Register[%d] (expected 0x%04x, found 0x%04x)",
+ crtc->base.base.id, crtc->base.name, i,
+ mpll_sw_state->mplla[i], mpll_hw_state->mplla[i]);
}
}
for (i = 0; i < ARRAY_SIZE(mpll_sw_state->tx); i++) {
- I915_STATE_WARN(i915, mpll_hw_state->tx[i] != mpll_sw_state->tx[i],
- "[CRTC:%d:%s] mismatch in C20: Register TX[%i] (expected 0x%04x, found 0x%04x)",
- crtc->base.base.id, crtc->base.name, i,
- mpll_sw_state->tx[i], mpll_hw_state->tx[i]);
+ INTEL_DISPLAY_STATE_WARN(display, mpll_hw_state->tx[i] != mpll_sw_state->tx[i],
+ "[CRTC:%d:%s] mismatch in C20: Register TX[%i] (expected 0x%04x, found 0x%04x)",
+ crtc->base.base.id, crtc->base.name, i,
+ mpll_sw_state->tx[i], mpll_hw_state->tx[i]);
}
for (i = 0; i < ARRAY_SIZE(mpll_sw_state->cmn); i++) {
- I915_STATE_WARN(i915, mpll_hw_state->cmn[i] != mpll_sw_state->cmn[i],
- "[CRTC:%d:%s] mismatch in C20: Register CMN[%i] (expected 0x%04x, found 0x%04x)",
- crtc->base.base.id, crtc->base.name, i,
- mpll_sw_state->cmn[i], mpll_hw_state->cmn[i]);
+ INTEL_DISPLAY_STATE_WARN(display, mpll_hw_state->cmn[i] != mpll_sw_state->cmn[i],
+ "[CRTC:%d:%s] mismatch in C20: Register CMN[%i] (expected 0x%04x, found 0x%04x)",
+ crtc->base.base.id, crtc->base.name, i,
+ mpll_sw_state->cmn[i], mpll_hw_state->cmn[i]);
}
}
diff --git a/drivers/gpu/drm/i915/display/intel_display.c b/drivers/gpu/drm/i915/display/intel_display.c
index 2b4552cd1eed..4a3cc3369180 100644
--- a/drivers/gpu/drm/i915/display/intel_display.c
+++ b/drivers/gpu/drm/i915/display/intel_display.c
@@ -292,10 +292,10 @@ u8 intel_crtc_joined_pipe_mask(const struct intel_crtc_state *crtc_state)
struct intel_crtc *intel_primary_crtc(const struct intel_crtc_state *crtc_state)
{
- struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev);
+ struct intel_display *display = to_intel_display(crtc_state);
if (intel_crtc_is_joiner_secondary(crtc_state))
- return intel_crtc_for_pipe(i915, joiner_primary_pipe(crtc_state));
+ return intel_crtc_for_pipe(display, joiner_primary_pipe(crtc_state));
else
return to_intel_crtc(crtc_state->uapi.crtc);
}
@@ -321,6 +321,7 @@ intel_wait_for_pipe_off(const struct intel_crtc_state *old_crtc_state)
void assert_transcoder(struct drm_i915_private *dev_priv,
enum transcoder cpu_transcoder, bool state)
{
+ struct intel_display *display = &dev_priv->display;
bool cur_state;
enum intel_display_power_domain power_domain;
intel_wakeref_t wakeref;
@@ -341,24 +342,24 @@ void assert_transcoder(struct drm_i915_private *dev_priv,
cur_state = false;
}
- I915_STATE_WARN(dev_priv, cur_state != state,
- "transcoder %s assertion failure (expected %s, current %s)\n",
- transcoder_name(cpu_transcoder), str_on_off(state),
- str_on_off(cur_state));
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "transcoder %s assertion failure (expected %s, current %s)\n",
+ transcoder_name(cpu_transcoder), str_on_off(state),
+ str_on_off(cur_state));
}
static void assert_plane(struct intel_plane *plane, bool state)
{
- struct drm_i915_private *i915 = to_i915(plane->base.dev);
+ struct intel_display *display = to_intel_display(plane->base.dev);
enum pipe pipe;
bool cur_state;
cur_state = plane->get_hw_state(plane, &pipe);
- I915_STATE_WARN(i915, cur_state != state,
- "%s assertion failure (expected %s, current %s)\n",
- plane->base.name, str_on_off(state),
- str_on_off(cur_state));
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "%s assertion failure (expected %s, current %s)\n",
+ plane->base.name, str_on_off(state),
+ str_on_off(cur_state));
}
#define assert_plane_enabled(p) assert_plane(p, true)
@@ -1668,6 +1669,7 @@ static void hsw_configure_cpu_transcoder(const struct intel_crtc_state *crtc_sta
static void hsw_crtc_enable(struct intel_atomic_state *state,
struct intel_crtc *crtc)
{
+ struct intel_display *display = to_intel_display(state);
const struct intel_crtc_state *new_crtc_state =
intel_atomic_get_new_crtc_state(state, crtc);
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
@@ -1762,7 +1764,7 @@ static void hsw_crtc_enable(struct intel_atomic_state *state,
hsw_workaround_pipe = pipe_crtc_state->hsw_workaround_pipe;
if (IS_HASWELL(dev_priv) && hsw_workaround_pipe != INVALID_PIPE) {
struct intel_crtc *wa_crtc =
- intel_crtc_for_pipe(dev_priv, hsw_workaround_pipe);
+ intel_crtc_for_pipe(display, hsw_workaround_pipe);
intel_crtc_wait_for_next_vblank(wa_crtc);
intel_crtc_wait_for_next_vblank(wa_crtc);
@@ -3966,7 +3968,7 @@ int intel_crtc_dotclock(const struct intel_crtc_state *pipe_config)
struct drm_display_mode *
intel_encoder_current_mode(struct intel_encoder *encoder)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
+ struct intel_display *display = to_intel_display(encoder);
struct intel_crtc_state *crtc_state;
struct drm_display_mode *mode;
struct intel_crtc *crtc;
@@ -3975,7 +3977,7 @@ intel_encoder_current_mode(struct intel_encoder *encoder)
if (!encoder->get_hw_state(encoder, &pipe))
return NULL;
- crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ crtc = intel_crtc_for_pipe(display, pipe);
mode = kzalloc(sizeof(*mode), GFP_KERNEL);
if (!mode)
@@ -7803,7 +7805,7 @@ void intel_setup_outputs(struct drm_i915_private *dev_priv)
if (HAS_DDI(dev_priv)) {
if (intel_ddi_crt_present(dev_priv))
- intel_crt_init(dev_priv);
+ intel_crt_init(display);
intel_bios_for_each_encoder(display, intel_ddi_init);
@@ -7818,7 +7820,7 @@ void intel_setup_outputs(struct drm_i915_private *dev_priv)
* incorrect sharing of the PPS.
*/
intel_lvds_init(dev_priv);
- intel_crt_init(dev_priv);
+ intel_crt_init(display);
dpd_is_edp = intel_dp_is_port_edp(dev_priv, PORT_D);
@@ -7849,7 +7851,7 @@ void intel_setup_outputs(struct drm_i915_private *dev_priv)
bool has_edp, has_port;
if (IS_VALLEYVIEW(dev_priv) && dev_priv->display.vbt.int_crt_support)
- intel_crt_init(dev_priv);
+ intel_crt_init(display);
/*
* The DP_DETECTED bit is the latched state of the DDC
@@ -7895,14 +7897,14 @@ void intel_setup_outputs(struct drm_i915_private *dev_priv)
vlv_dsi_init(dev_priv);
} else if (IS_PINEVIEW(dev_priv)) {
intel_lvds_init(dev_priv);
- intel_crt_init(dev_priv);
+ intel_crt_init(display);
} else if (IS_DISPLAY_VER(dev_priv, 3, 4)) {
bool found = false;
if (IS_MOBILE(dev_priv))
intel_lvds_init(dev_priv);
- intel_crt_init(dev_priv);
+ intel_crt_init(display);
if (intel_de_read(dev_priv, GEN3_SDVOB) & SDVO_DETECTED) {
drm_dbg_kms(&dev_priv->drm, "probing SDVOB\n");
@@ -7944,7 +7946,7 @@ void intel_setup_outputs(struct drm_i915_private *dev_priv)
if (IS_I85X(dev_priv))
intel_lvds_init(dev_priv);
- intel_crt_init(dev_priv);
+ intel_crt_init(display);
intel_dvo_init(dev_priv);
}
@@ -8260,7 +8262,8 @@ int intel_initial_commit(struct drm_device *dev)
void i830_enable_pipe(struct drm_i915_private *dev_priv, enum pipe pipe)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_display *display = &dev_priv->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
enum transcoder cpu_transcoder = (enum transcoder)pipe;
/* 640x480@60Hz, ~25175 kHz */
struct dpll clock = {
@@ -8341,7 +8344,8 @@ void i830_enable_pipe(struct drm_i915_private *dev_priv, enum pipe pipe)
void i830_disable_pipe(struct drm_i915_private *dev_priv, enum pipe pipe)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_display *display = &dev_priv->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
drm_dbg_kms(&dev_priv->drm, "disabling pipe %c due to force quirk\n",
pipe_name(pipe));
diff --git a/drivers/gpu/drm/i915/display/intel_display.h b/drivers/gpu/drm/i915/display/intel_display.h
index b21d9578d5db..3c4b48697bb1 100644
--- a/drivers/gpu/drm/i915/display/intel_display.h
+++ b/drivers/gpu/drm/i915/display/intel_display.h
@@ -570,18 +570,17 @@ void assert_transcoder(struct drm_i915_private *dev_priv,
bool assert_port_valid(struct drm_i915_private *i915, enum port port);
/*
- * Use I915_STATE_WARN(x) (rather than WARN() and WARN_ON()) for hw state sanity
- * checks to check for unexpected conditions which may not necessarily be a user
- * visible problem. This will either WARN() or DRM_ERROR() depending on the
- * verbose_state_checks module param, to enable distros and users to tailor
- * their preferred amount of i915 abrt spam.
+ * Use INTEL_DISPLAY_STATE_WARN(x) (rather than WARN() and WARN_ON()) for hw
+ * state sanity checks to check for unexpected conditions which may not
+ * necessarily be a user visible problem. This will either drm_WARN() or
+ * drm_err() depending on the verbose_state_checks module param, to enable
+ * distros and users to tailor their preferred amount of i915 abrt spam.
*/
-#define I915_STATE_WARN(__i915, condition, format...) ({ \
- struct drm_device *drm = &(__i915)->drm; \
+#define INTEL_DISPLAY_STATE_WARN(__display, condition, format...) ({ \
int __ret_warn_on = !!(condition); \
if (unlikely(__ret_warn_on)) \
- if (!drm_WARN(drm, __i915->display.params.verbose_state_checks, format)) \
- drm_err(drm, format); \
+ if (!drm_WARN((__display)->drm, (__display)->params.verbose_state_checks, format)) \
+ drm_err((__display)->drm, format); \
unlikely(__ret_warn_on); \
})
diff --git a/drivers/gpu/drm/i915/display/intel_display_driver.c b/drivers/gpu/drm/i915/display/intel_display_driver.c
index 069426d9260b..8d65a26c9a40 100644
--- a/drivers/gpu/drm/i915/display/intel_display_driver.c
+++ b/drivers/gpu/drm/i915/display/intel_display_driver.c
@@ -168,10 +168,11 @@ static void intel_mode_config_cleanup(struct drm_i915_private *i915)
static void intel_plane_possible_crtcs_init(struct drm_i915_private *dev_priv)
{
+ struct intel_display *display = &dev_priv->display;
struct intel_plane *plane;
for_each_intel_plane(&dev_priv->drm, plane) {
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv,
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display,
plane->pipe);
plane->base.possible_crtcs = drm_crtc_mask(&crtc->base);
@@ -219,7 +220,7 @@ int intel_display_driver_probe_noirq(struct drm_i915_private *i915)
intel_bios_init(display);
- ret = intel_vga_register(i915);
+ ret = intel_vga_register(display);
if (ret)
goto cleanup_bios;
@@ -273,7 +274,7 @@ int intel_display_driver_probe_noirq(struct drm_i915_private *i915)
intel_dmc_fini(i915);
intel_power_domains_driver_remove(i915);
cleanup_vga:
- intel_vga_unregister(i915);
+ intel_vga_unregister(display);
cleanup_bios:
intel_bios_driver_remove(display);
@@ -430,7 +431,7 @@ int intel_display_driver_probe_nogem(struct drm_i915_private *i915)
intel_pps_setup(display);
- intel_gmbus_setup(i915);
+ intel_gmbus_setup(display);
drm_dbg_kms(&i915->drm, "%d display pipe%s available.\n",
INTEL_NUM_PIPES(i915),
@@ -456,7 +457,7 @@ int intel_display_driver_probe_nogem(struct drm_i915_private *i915)
intel_hti_init(display);
/* Just disable it once at startup */
- intel_vga_disable(i915);
+ intel_vga_disable(display);
intel_setup_outputs(i915);
ret = intel_dp_tunnel_mgr_init(display);
@@ -606,7 +607,7 @@ void intel_display_driver_remove_noirq(struct drm_i915_private *i915)
intel_overlay_cleanup(i915);
- intel_gmbus_teardown(i915);
+ intel_gmbus_teardown(display);
destroy_workqueue(i915->display.wq.flip);
destroy_workqueue(i915->display.wq.modeset);
@@ -623,7 +624,7 @@ void intel_display_driver_remove_nogem(struct drm_i915_private *i915)
intel_power_domains_driver_remove(i915);
- intel_vga_unregister(i915);
+ intel_vga_unregister(display);
intel_bios_driver_remove(display);
}
@@ -681,12 +682,13 @@ __intel_display_driver_resume(struct drm_i915_private *i915,
struct drm_atomic_state *state,
struct drm_modeset_acquire_ctx *ctx)
{
+ struct intel_display *display = &i915->display;
struct drm_crtc_state *crtc_state;
struct drm_crtc *crtc;
int ret, i;
intel_modeset_setup_hw_state(i915, ctx);
- intel_vga_redisable(i915);
+ intel_vga_redisable(display);
if (!state)
return 0;
diff --git a/drivers/gpu/drm/i915/display/intel_display_irq.c b/drivers/gpu/drm/i915/display/intel_display_irq.c
index 73369847ed66..e2a98395e921 100644
--- a/drivers/gpu/drm/i915/display/intel_display_irq.c
+++ b/drivers/gpu/drm/i915/display/intel_display_irq.c
@@ -27,7 +27,8 @@
static void
intel_handle_vblank(struct drm_i915_private *dev_priv, enum pipe pipe)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_display *display = &dev_priv->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
drm_crtc_handle_vblank(&crtc->base);
}
@@ -305,7 +306,8 @@ static void display_pipe_crc_irq_handler(struct drm_i915_private *dev_priv,
u32 crc2, u32 crc3,
u32 crc4)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_display *display = &dev_priv->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
struct intel_pipe_crc *pipe_crc = &crtc->pipe_crc;
u32 crcs[5] = { crc0, crc1, crc2, crc3, crc4 };
@@ -344,7 +346,8 @@ display_pipe_crc_irq_handler(struct drm_i915_private *dev_priv,
static void flip_done_handler(struct drm_i915_private *i915,
enum pipe pipe)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(i915, pipe);
+ struct intel_display *display = &i915->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
spin_lock(&i915->drm.event_lock);
@@ -548,12 +551,13 @@ void i965_pipestat_irq_handler(struct drm_i915_private *dev_priv,
intel_opregion_asle_intr(display);
if (pipe_stats[0] & PIPE_GMBUS_INTERRUPT_STATUS)
- intel_gmbus_irq_handler(dev_priv);
+ intel_gmbus_irq_handler(display);
}
void valleyview_pipestat_irq_handler(struct drm_i915_private *dev_priv,
u32 pipe_stats[I915_MAX_PIPES])
{
+ struct intel_display *display = &dev_priv->display;
enum pipe pipe;
for_each_pipe(dev_priv, pipe) {
@@ -571,7 +575,7 @@ void valleyview_pipestat_irq_handler(struct drm_i915_private *dev_priv,
}
if (pipe_stats[0] & PIPE_GMBUS_INTERRUPT_STATUS)
- intel_gmbus_irq_handler(dev_priv);
+ intel_gmbus_irq_handler(display);
}
static void ibx_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir)
@@ -593,7 +597,7 @@ static void ibx_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir)
intel_dp_aux_irq_handler(display);
if (pch_iir & SDE_GMBUS)
- intel_gmbus_irq_handler(dev_priv);
+ intel_gmbus_irq_handler(display);
if (pch_iir & SDE_AUDIO_HDCP_MASK)
drm_dbg(&dev_priv->drm, "PCH HDCP audio interrupt\n");
@@ -682,7 +686,7 @@ static void cpt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir)
intel_dp_aux_irq_handler(display);
if (pch_iir & SDE_GMBUS_CPT)
- intel_gmbus_irq_handler(dev_priv);
+ intel_gmbus_irq_handler(display);
if (pch_iir & SDE_AUDIO_CP_REQ_CPT)
drm_dbg(&dev_priv->drm, "Audio CP request interrupt\n");
@@ -1125,7 +1129,7 @@ void gen8_de_irq_handler(struct drm_i915_private *dev_priv, u32 master_ctl)
if ((IS_GEMINILAKE(dev_priv) || IS_BROXTON(dev_priv)) &&
(iir & BXT_DE_PORT_GMBUS)) {
- intel_gmbus_irq_handler(dev_priv);
+ intel_gmbus_irq_handler(display);
found = true;
}
diff --git a/drivers/gpu/drm/i915/display/intel_display_power.c b/drivers/gpu/drm/i915/display/intel_display_power.c
index 85335d8dfec1..be39a0180921 100644
--- a/drivers/gpu/drm/i915/display/intel_display_power.c
+++ b/drivers/gpu/drm/i915/display/intel_display_power.c
@@ -1176,43 +1176,44 @@ static void hsw_assert_cdclk(struct drm_i915_private *dev_priv)
static void assert_can_disable_lcpll(struct drm_i915_private *dev_priv)
{
+ struct intel_display *display = &dev_priv->display;
struct intel_crtc *crtc;
- for_each_intel_crtc(&dev_priv->drm, crtc)
- I915_STATE_WARN(dev_priv, crtc->active,
- "CRTC for pipe %c enabled\n",
- pipe_name(crtc->pipe));
-
- I915_STATE_WARN(dev_priv, intel_de_read(dev_priv, HSW_PWR_WELL_CTL2),
- "Display power well on\n");
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, SPLL_CTL) & SPLL_PLL_ENABLE,
- "SPLL enabled\n");
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, WRPLL_CTL(0)) & WRPLL_PLL_ENABLE,
- "WRPLL1 enabled\n");
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, WRPLL_CTL(1)) & WRPLL_PLL_ENABLE,
- "WRPLL2 enabled\n");
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, PP_STATUS(dev_priv, 0)) & PP_ON,
- "Panel power on\n");
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, BLC_PWM_CPU_CTL2) & BLM_PWM_ENABLE,
- "CPU PWM1 enabled\n");
+ for_each_intel_crtc(display->drm, crtc)
+ INTEL_DISPLAY_STATE_WARN(display, crtc->active,
+ "CRTC for pipe %c enabled\n",
+ pipe_name(crtc->pipe));
+
+ INTEL_DISPLAY_STATE_WARN(display, intel_de_read(display, HSW_PWR_WELL_CTL2),
+ "Display power well on\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, SPLL_CTL) & SPLL_PLL_ENABLE,
+ "SPLL enabled\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, WRPLL_CTL(0)) & WRPLL_PLL_ENABLE,
+ "WRPLL1 enabled\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, WRPLL_CTL(1)) & WRPLL_PLL_ENABLE,
+ "WRPLL2 enabled\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, PP_STATUS(display, 0)) & PP_ON,
+ "Panel power on\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, BLC_PWM_CPU_CTL2) & BLM_PWM_ENABLE,
+ "CPU PWM1 enabled\n");
if (IS_HASWELL(dev_priv))
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, HSW_BLC_PWM2_CTL) & BLM_PWM_ENABLE,
- "CPU PWM2 enabled\n");
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, BLC_PWM_PCH_CTL1) & BLM_PCH_PWM_ENABLE,
- "PCH PWM1 enabled\n");
- I915_STATE_WARN(dev_priv,
- (intel_de_read(dev_priv, UTIL_PIN_CTL) & (UTIL_PIN_ENABLE | UTIL_PIN_MODE_MASK)) == (UTIL_PIN_ENABLE | UTIL_PIN_MODE_PWM),
- "Utility pin enabled in PWM mode\n");
- I915_STATE_WARN(dev_priv,
- intel_de_read(dev_priv, PCH_GTC_CTL) & PCH_GTC_ENABLE,
- "PCH GTC enabled\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, HSW_BLC_PWM2_CTL) & BLM_PWM_ENABLE,
+ "CPU PWM2 enabled\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, BLC_PWM_PCH_CTL1) & BLM_PCH_PWM_ENABLE,
+ "PCH PWM1 enabled\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ (intel_de_read(display, UTIL_PIN_CTL) & (UTIL_PIN_ENABLE | UTIL_PIN_MODE_MASK)) == (UTIL_PIN_ENABLE | UTIL_PIN_MODE_PWM),
+ "Utility pin enabled in PWM mode\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_de_read(display, PCH_GTC_CTL) & PCH_GTC_ENABLE,
+ "PCH GTC enabled\n");
/*
* In theory we can still leave IRQs enabled, as long as only the HPD
@@ -1220,8 +1221,8 @@ static void assert_can_disable_lcpll(struct drm_i915_private *dev_priv)
* gen-specific and since we only disable LCPLL after we fully disable
* the interrupts, the check below should be enough.
*/
- I915_STATE_WARN(dev_priv, intel_irqs_enabled(dev_priv),
- "IRQs enabled\n");
+ INTEL_DISPLAY_STATE_WARN(display, intel_irqs_enabled(dev_priv),
+ "IRQs enabled\n");
}
static u32 hsw_read_dcomp(struct drm_i915_private *dev_priv)
diff --git a/drivers/gpu/drm/i915/display/intel_display_power_well.c b/drivers/gpu/drm/i915/display/intel_display_power_well.c
index 46e9eff12c23..3c1a211e7b96 100644
--- a/drivers/gpu/drm/i915/display/intel_display_power_well.c
+++ b/drivers/gpu/drm/i915/display/intel_display_power_well.c
@@ -187,8 +187,10 @@ int intel_power_well_refcount(struct i915_power_well *power_well)
static void hsw_power_well_post_enable(struct drm_i915_private *dev_priv,
u8 irq_pipe_mask, bool has_vga)
{
+ struct intel_display *display = &dev_priv->display;
+
if (has_vga)
- intel_vga_reset_io_mem(dev_priv);
+ intel_vga_reset_io_mem(display);
if (irq_pipe_mask)
gen8_irq_power_well_post_enable(dev_priv, irq_pipe_mask);
@@ -1232,7 +1234,7 @@ static void vlv_display_power_well_init(struct drm_i915_private *dev_priv)
intel_crt_reset(&encoder->base);
}
- intel_vga_redisable_power_on(dev_priv);
+ intel_vga_redisable_power_on(display);
intel_pps_unlock_regs_wa(display);
}
diff --git a/drivers/gpu/drm/i915/display/intel_display_trace.h b/drivers/gpu/drm/i915/display/intel_display_trace.h
index c734ef1fba3c..7176eefaf505 100644
--- a/drivers/gpu/drm/i915/display/intel_display_trace.h
+++ b/drivers/gpu/drm/i915/display/intel_display_trace.h
@@ -141,7 +141,8 @@ TRACE_EVENT(intel_cpu_fifo_underrun,
),
TP_fast_assign(
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_display *display = &dev_priv->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
__assign_str(dev);
__entry->pipe = pipe;
__entry->frame = intel_crtc_get_vblank_counter(crtc);
@@ -165,8 +166,9 @@ TRACE_EVENT(intel_pch_fifo_underrun,
),
TP_fast_assign(
+ struct intel_display *display = &dev_priv->display;
enum pipe pipe = pch_transcoder;
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
__assign_str(dev);
__entry->pipe = pipe;
__entry->frame = intel_crtc_get_vblank_counter(crtc);
@@ -458,7 +460,8 @@ TRACE_EVENT(intel_fbc_activate,
),
TP_fast_assign(
- struct intel_crtc *crtc = intel_crtc_for_pipe(to_i915(plane->base.dev),
+ struct intel_display *display = to_intel_display(plane->base.dev);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display,
plane->pipe);
__assign_str(dev);
__assign_str(name);
@@ -485,7 +488,8 @@ TRACE_EVENT(intel_fbc_deactivate,
),
TP_fast_assign(
- struct intel_crtc *crtc = intel_crtc_for_pipe(to_i915(plane->base.dev),
+ struct intel_display *display = to_intel_display(plane->base.dev);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display,
plane->pipe);
__assign_str(dev);
__assign_str(name);
@@ -512,7 +516,8 @@ TRACE_EVENT(intel_fbc_nuke,
),
TP_fast_assign(
- struct intel_crtc *crtc = intel_crtc_for_pipe(to_i915(plane->base.dev),
+ struct intel_display *display = to_intel_display(plane->base.dev);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display,
plane->pipe);
__assign_str(dev);
__assign_str(name);
diff --git a/drivers/gpu/drm/i915/display/intel_dp.c b/drivers/gpu/drm/i915/display/intel_dp.c
index 63c335d3d9ce..b1ba117cfce8 100644
--- a/drivers/gpu/drm/i915/display/intel_dp.c
+++ b/drivers/gpu/drm/i915/display/intel_dp.c
@@ -5202,10 +5202,11 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp)
struct intel_encoder *encoder = &dig_port->base;
bool link_ok = true;
bool reprobe_needed = false;
+ int tries = 33;
drm_WARN_ON_ONCE(&i915->drm, intel_dp->mst.active_links < 0);
- for (;;) {
+ while (--tries) {
u8 esi[4] = {};
u8 ack[4] = {};
@@ -5248,6 +5249,11 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp)
if (!link_ok || intel_dp->link.force_retrain)
intel_encoder_link_check_queue_work(encoder, 0);
+ if (!tries) {
+ drm_dbg_kms(&i915->drm, "DPRX ESI not clearing, device may be stuck\n");
+ reprobe_needed = true;
+ }
+
return !reprobe_needed;
}
diff --git a/drivers/gpu/drm/i915/display/intel_dpll.c b/drivers/gpu/drm/i915/display/intel_dpll.c
index 340dfce480b8..0e4a21d35be2 100644
--- a/drivers/gpu/drm/i915/display/intel_dpll.c
+++ b/drivers/gpu/drm/i915/display/intel_dpll.c
@@ -2212,7 +2212,8 @@ void chv_enable_pll(const struct intel_crtc_state *crtc_state)
int vlv_force_pll_on(struct drm_i915_private *dev_priv, enum pipe pipe,
const struct dpll *dpll)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_display *display = &dev_priv->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
struct intel_crtc_state *crtc_state;
crtc_state = intel_crtc_state_alloc(crtc);
@@ -2318,12 +2319,13 @@ void vlv_force_pll_off(struct drm_i915_private *dev_priv, enum pipe pipe)
static void assert_pll(struct drm_i915_private *dev_priv,
enum pipe pipe, bool state)
{
+ struct intel_display *display = &dev_priv->display;
bool cur_state;
- cur_state = intel_de_read(dev_priv, DPLL(dev_priv, pipe)) & DPLL_VCO_ENABLE;
- I915_STATE_WARN(dev_priv, cur_state != state,
- "PLL state assertion failure (expected %s, current %s)\n",
- str_on_off(state), str_on_off(cur_state));
+ cur_state = intel_de_read(display, DPLL(display, pipe)) & DPLL_VCO_ENABLE;
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "PLL state assertion failure (expected %s, current %s)\n",
+ str_on_off(state), str_on_off(cur_state));
}
void assert_pll_enabled(struct drm_i915_private *i915, enum pipe pipe)
diff --git a/drivers/gpu/drm/i915/display/intel_dpll_mgr.c b/drivers/gpu/drm/i915/display/intel_dpll_mgr.c
index f490b2157828..c9eda647919a 100644
--- a/drivers/gpu/drm/i915/display/intel_dpll_mgr.c
+++ b/drivers/gpu/drm/i915/display/intel_dpll_mgr.c
@@ -173,18 +173,19 @@ void assert_shared_dpll(struct drm_i915_private *i915,
struct intel_shared_dpll *pll,
bool state)
{
+ struct intel_display *display = &i915->display;
bool cur_state;
struct intel_dpll_hw_state hw_state;
- if (drm_WARN(&i915->drm, !pll,
+ if (drm_WARN(display->drm, !pll,
"asserting DPLL %s with no DPLL\n", str_on_off(state)))
return;
cur_state = intel_dpll_get_hw_state(i915, pll, &hw_state);
- I915_STATE_WARN(i915, cur_state != state,
- "%s assertion failure (expected %s, current %s)\n",
- pll->info->name, str_on_off(state),
- str_on_off(cur_state));
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "%s assertion failure (expected %s, current %s)\n",
+ pll->info->name, str_on_off(state),
+ str_on_off(cur_state));
}
static enum tc_port icl_pll_id_to_tc_port(enum intel_dpll_id id)
@@ -545,14 +546,15 @@ static bool ibx_pch_dpll_get_hw_state(struct drm_i915_private *i915,
static void ibx_assert_pch_refclk_enabled(struct drm_i915_private *i915)
{
+ struct intel_display *display = &i915->display;
u32 val;
bool enabled;
- val = intel_de_read(i915, PCH_DREF_CONTROL);
+ val = intel_de_read(display, PCH_DREF_CONTROL);
enabled = !!(val & (DREF_SSC_SOURCE_MASK | DREF_NONSPREAD_SOURCE_MASK |
DREF_SUPERSPREAD_SOURCE_MASK));
- I915_STATE_WARN(i915, !enabled,
- "PCH refclk assertion failure, should be active but is disabled\n");
+ INTEL_DISPLAY_STATE_WARN(display, !enabled,
+ "PCH refclk assertion failure, should be active but is disabled\n");
}
static void ibx_pch_dpll_enable(struct drm_i915_private *i915,
@@ -4619,6 +4621,7 @@ verify_single_dpll_state(struct drm_i915_private *i915,
struct intel_crtc *crtc,
const struct intel_crtc_state *new_crtc_state)
{
+ struct intel_display *display = &i915->display;
struct intel_dpll_hw_state dpll_hw_state = {};
u8 pipe_mask;
bool active;
@@ -4626,22 +4629,22 @@ verify_single_dpll_state(struct drm_i915_private *i915,
active = intel_dpll_get_hw_state(i915, pll, &dpll_hw_state);
if (!pll->info->always_on) {
- I915_STATE_WARN(i915, !pll->on && pll->active_mask,
- "%s: pll in active use but not on in sw tracking\n",
- pll->info->name);
- I915_STATE_WARN(i915, pll->on && !pll->active_mask,
- "%s: pll is on but not used by any active pipe\n",
- pll->info->name);
- I915_STATE_WARN(i915, pll->on != active,
- "%s: pll on state mismatch (expected %i, found %i)\n",
- pll->info->name, pll->on, active);
+ INTEL_DISPLAY_STATE_WARN(display, !pll->on && pll->active_mask,
+ "%s: pll in active use but not on in sw tracking\n",
+ pll->info->name);
+ INTEL_DISPLAY_STATE_WARN(display, pll->on && !pll->active_mask,
+ "%s: pll is on but not used by any active pipe\n",
+ pll->info->name);
+ INTEL_DISPLAY_STATE_WARN(display, pll->on != active,
+ "%s: pll on state mismatch (expected %i, found %i)\n",
+ pll->info->name, pll->on, active);
}
if (!crtc) {
- I915_STATE_WARN(i915,
- pll->active_mask & ~pll->state.pipe_mask,
- "%s: more active pll users than references: 0x%x vs 0x%x\n",
- pll->info->name, pll->active_mask, pll->state.pipe_mask);
+ INTEL_DISPLAY_STATE_WARN(display,
+ pll->active_mask & ~pll->state.pipe_mask,
+ "%s: more active pll users than references: 0x%x vs 0x%x\n",
+ pll->info->name, pll->active_mask, pll->state.pipe_mask);
return;
}
@@ -4649,23 +4652,23 @@ verify_single_dpll_state(struct drm_i915_private *i915,
pipe_mask = BIT(crtc->pipe);
if (new_crtc_state->hw.active)
- I915_STATE_WARN(i915, !(pll->active_mask & pipe_mask),
- "%s: pll active mismatch (expected pipe %c in active mask 0x%x)\n",
- pll->info->name, pipe_name(crtc->pipe), pll->active_mask);
+ INTEL_DISPLAY_STATE_WARN(display, !(pll->active_mask & pipe_mask),
+ "%s: pll active mismatch (expected pipe %c in active mask 0x%x)\n",
+ pll->info->name, pipe_name(crtc->pipe), pll->active_mask);
else
- I915_STATE_WARN(i915, pll->active_mask & pipe_mask,
- "%s: pll active mismatch (didn't expect pipe %c in active mask 0x%x)\n",
- pll->info->name, pipe_name(crtc->pipe), pll->active_mask);
+ INTEL_DISPLAY_STATE_WARN(display, pll->active_mask & pipe_mask,
+ "%s: pll active mismatch (didn't expect pipe %c in active mask 0x%x)\n",
+ pll->info->name, pipe_name(crtc->pipe), pll->active_mask);
- I915_STATE_WARN(i915, !(pll->state.pipe_mask & pipe_mask),
- "%s: pll enabled crtcs mismatch (expected 0x%x in 0x%x)\n",
- pll->info->name, pipe_mask, pll->state.pipe_mask);
+ INTEL_DISPLAY_STATE_WARN(display, !(pll->state.pipe_mask & pipe_mask),
+ "%s: pll enabled crtcs mismatch (expected 0x%x in 0x%x)\n",
+ pll->info->name, pipe_mask, pll->state.pipe_mask);
- I915_STATE_WARN(i915,
- pll->on && memcmp(&pll->state.hw_state, &dpll_hw_state,
- sizeof(dpll_hw_state)),
- "%s: pll hw state mismatch\n",
- pll->info->name);
+ INTEL_DISPLAY_STATE_WARN(display,
+ pll->on && memcmp(&pll->state.hw_state, &dpll_hw_state,
+ sizeof(dpll_hw_state)),
+ "%s: pll hw state mismatch\n",
+ pll->info->name);
}
static bool has_alt_port_dpll(const struct intel_shared_dpll *old_pll,
@@ -4678,6 +4681,7 @@ static bool has_alt_port_dpll(const struct intel_shared_dpll *old_pll,
void intel_shared_dpll_state_verify(struct intel_atomic_state *state,
struct intel_crtc *crtc)
{
+ struct intel_display *display = to_intel_display(state);
struct drm_i915_private *i915 = to_i915(state->base.dev);
const struct intel_crtc_state *old_crtc_state =
intel_atomic_get_old_crtc_state(state, crtc);
@@ -4693,16 +4697,16 @@ void intel_shared_dpll_state_verify(struct intel_atomic_state *state,
u8 pipe_mask = BIT(crtc->pipe);
struct intel_shared_dpll *pll = old_crtc_state->shared_dpll;
- I915_STATE_WARN(i915, pll->active_mask & pipe_mask,
- "%s: pll active mismatch (didn't expect pipe %c in active mask (0x%x))\n",
- pll->info->name, pipe_name(crtc->pipe), pll->active_mask);
+ INTEL_DISPLAY_STATE_WARN(display, pll->active_mask & pipe_mask,
+ "%s: pll active mismatch (didn't expect pipe %c in active mask (0x%x))\n",
+ pll->info->name, pipe_name(crtc->pipe), pll->active_mask);
/* TC ports have both MG/TC and TBT PLL referenced simultaneously */
- I915_STATE_WARN(i915, !has_alt_port_dpll(old_crtc_state->shared_dpll,
- new_crtc_state->shared_dpll) &&
- pll->state.pipe_mask & pipe_mask,
- "%s: pll enabled crtcs mismatch (found pipe %c in enabled mask (0x%x))\n",
- pll->info->name, pipe_name(crtc->pipe), pll->state.pipe_mask);
+ INTEL_DISPLAY_STATE_WARN(display, !has_alt_port_dpll(old_crtc_state->shared_dpll,
+ new_crtc_state->shared_dpll) &&
+ pll->state.pipe_mask & pipe_mask,
+ "%s: pll enabled crtcs mismatch (found pipe %c in enabled mask (0x%x))\n",
+ pll->info->name, pipe_name(crtc->pipe), pll->state.pipe_mask);
}
}
diff --git a/drivers/gpu/drm/i915/display/intel_dsb.c b/drivers/gpu/drm/i915/display/intel_dsb.c
index da24e041d269..c39bae1be89e 100644
--- a/drivers/gpu/drm/i915/display/intel_dsb.c
+++ b/drivers/gpu/drm/i915/display/intel_dsb.c
@@ -763,7 +763,7 @@ void intel_dsb_cleanup(struct intel_dsb *dsb)
void intel_dsb_irq_handler(struct intel_display *display,
enum pipe pipe, enum intel_dsb_id dsb_id)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(to_i915(display->drm), pipe);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
u32 tmp, errors;
tmp = intel_de_read_fw(display, DSB_INTERRUPT(pipe, dsb_id));
diff --git a/drivers/gpu/drm/i915/display/intel_dsi_vbt.c b/drivers/gpu/drm/i915/display/intel_dsi_vbt.c
index f0e3be0fe420..e8129a720210 100644
--- a/drivers/gpu/drm/i915/display/intel_dsi_vbt.c
+++ b/drivers/gpu/drm/i915/display/intel_dsi_vbt.c
@@ -323,6 +323,7 @@ enum {
static void icl_native_gpio_set_value(struct drm_i915_private *dev_priv,
int gpio, bool value)
{
+ struct intel_display *display = &dev_priv->display;
int index;
if (drm_WARN_ON(&dev_priv->drm, DISPLAY_VER(dev_priv) == 11 && gpio >= MIPI_RESET_2))
@@ -367,7 +368,7 @@ static void icl_native_gpio_set_value(struct drm_i915_private *dev_priv,
case MIPI_AVEE_EN_2:
index = gpio == MIPI_AVEE_EN_1 ? 1 : 2;
- intel_de_rmw(dev_priv, GPIO(dev_priv, index),
+ intel_de_rmw(display, GPIO(display, index),
GPIO_CLOCK_VAL_OUT,
GPIO_CLOCK_DIR_MASK | GPIO_CLOCK_DIR_OUT |
GPIO_CLOCK_VAL_MASK | (value ? GPIO_CLOCK_VAL_OUT : 0));
@@ -376,7 +377,7 @@ static void icl_native_gpio_set_value(struct drm_i915_private *dev_priv,
case MIPI_VIO_EN_2:
index = gpio == MIPI_VIO_EN_1 ? 1 : 2;
- intel_de_rmw(dev_priv, GPIO(dev_priv, index),
+ intel_de_rmw(display, GPIO(display, index),
GPIO_DATA_VAL_OUT,
GPIO_DATA_DIR_MASK | GPIO_DATA_DIR_OUT |
GPIO_DATA_VAL_MASK | (value ? GPIO_DATA_VAL_OUT : 0));
diff --git a/drivers/gpu/drm/i915/display/intel_dvo.c b/drivers/gpu/drm/i915/display/intel_dvo.c
index 12e7628cbecf..2bb4d829ca3c 100644
--- a/drivers/gpu/drm/i915/display/intel_dvo.c
+++ b/drivers/gpu/drm/i915/display/intel_dvo.c
@@ -416,6 +416,7 @@ static bool intel_dvo_init_dev(struct drm_i915_private *dev_priv,
struct intel_dvo *intel_dvo,
const struct intel_dvo_device *dvo)
{
+ struct intel_display *display = &dev_priv->display;
struct i2c_adapter *i2c;
u32 dpll[I915_MAX_PIPES];
enum pipe pipe;
@@ -427,7 +428,7 @@ static bool intel_dvo_init_dev(struct drm_i915_private *dev_priv,
* special cases, but otherwise default to what's defined
* in the spec.
*/
- if (intel_gmbus_is_valid_pin(dev_priv, dvo->gpio))
+ if (intel_gmbus_is_valid_pin(display, dvo->gpio))
gpio = dvo->gpio;
else if (dvo->type == INTEL_DVO_CHIP_LVDS)
gpio = GMBUS_PIN_SSC;
@@ -439,7 +440,7 @@ static bool intel_dvo_init_dev(struct drm_i915_private *dev_priv,
* It appears that everything is on GPIOE except for panels
* on i830 laptops, which are on GPIOB (DVOA).
*/
- i2c = intel_gmbus_get_adapter(dev_priv, gpio);
+ i2c = intel_gmbus_get_adapter(display, gpio);
intel_dvo->dev = *dvo;
@@ -488,6 +489,7 @@ static bool intel_dvo_probe(struct drm_i915_private *i915,
void intel_dvo_init(struct drm_i915_private *i915)
{
+ struct intel_display *display = &i915->display;
struct intel_connector *connector;
struct intel_encoder *encoder;
struct intel_dvo *intel_dvo;
@@ -548,7 +550,7 @@ void intel_dvo_init(struct drm_i915_private *i915)
drm_connector_init_with_ddc(&i915->drm, &connector->base,
&intel_dvo_connector_funcs,
intel_dvo_connector_type(&intel_dvo->dev),
- intel_gmbus_get_adapter(i915, GMBUS_PIN_DPC));
+ intel_gmbus_get_adapter(display, GMBUS_PIN_DPC));
drm_connector_helper_add(&connector->base,
&intel_dvo_connector_helper_funcs);
diff --git a/drivers/gpu/drm/i915/display/intel_fbc.c b/drivers/gpu/drm/i915/display/intel_fbc.c
index 52b79bacef4d..809b477f907e 100644
--- a/drivers/gpu/drm/i915/display/intel_fbc.c
+++ b/drivers/gpu/drm/i915/display/intel_fbc.c
@@ -378,11 +378,11 @@ static void i8xx_fbc_program_cfb(struct intel_fbc *fbc)
struct drm_i915_private *i915 = to_i915(display->drm);
drm_WARN_ON(display->drm,
- range_overflows_end_t(u64, i915_gem_stolen_area_address(i915),
+ range_end_overflows_t(u64, i915_gem_stolen_area_address(i915),
i915_gem_stolen_node_offset(&fbc->compressed_fb),
U32_MAX));
drm_WARN_ON(display->drm,
- range_overflows_end_t(u64, i915_gem_stolen_area_address(i915),
+ range_end_overflows_t(u64, i915_gem_stolen_area_address(i915),
i915_gem_stolen_node_offset(&fbc->compressed_llb),
U32_MAX));
intel_de_write(i915, FBC_CFB_BASE,
@@ -1792,7 +1792,6 @@ static void intel_fbc_underrun_work_fn(struct work_struct *work)
{
struct intel_fbc *fbc = container_of(work, typeof(*fbc), underrun_work);
struct intel_display *display = fbc->display;
- struct drm_i915_private *i915 = to_i915(display->drm);
mutex_lock(&fbc->lock);
@@ -1805,7 +1804,7 @@ static void intel_fbc_underrun_work_fn(struct work_struct *work)
intel_fbc_deactivate(fbc, "FIFO underrun");
if (!fbc->flip_pending)
- intel_crtc_wait_for_next_vblank(intel_crtc_for_pipe(i915, fbc->state.plane->pipe));
+ intel_crtc_wait_for_next_vblank(intel_crtc_for_pipe(display, fbc->state.plane->pipe));
__intel_fbc_disable(fbc);
out:
mutex_unlock(&fbc->lock);
diff --git a/drivers/gpu/drm/i915/display/intel_fdi.c b/drivers/gpu/drm/i915/display/intel_fdi.c
index 222cd0e1a2bc..98e1a3606227 100644
--- a/drivers/gpu/drm/i915/display/intel_fdi.c
+++ b/drivers/gpu/drm/i915/display/intel_fdi.c
@@ -26,9 +26,10 @@ struct intel_fdi_funcs {
static void assert_fdi_tx(struct drm_i915_private *dev_priv,
enum pipe pipe, bool state)
{
+ struct intel_display *display = &dev_priv->display;
bool cur_state;
- if (HAS_DDI(dev_priv)) {
+ if (HAS_DDI(display)) {
/*
* DDI does not have a specific FDI_TX register.
*
@@ -36,14 +37,14 @@ static void assert_fdi_tx(struct drm_i915_private *dev_priv,
* so pipe->transcoder cast is fine here.
*/
enum transcoder cpu_transcoder = (enum transcoder)pipe;
- cur_state = intel_de_read(dev_priv,
- TRANS_DDI_FUNC_CTL(dev_priv, cpu_transcoder)) & TRANS_DDI_FUNC_ENABLE;
+ cur_state = intel_de_read(display,
+ TRANS_DDI_FUNC_CTL(display, cpu_transcoder)) & TRANS_DDI_FUNC_ENABLE;
} else {
- cur_state = intel_de_read(dev_priv, FDI_TX_CTL(pipe)) & FDI_TX_ENABLE;
+ cur_state = intel_de_read(display, FDI_TX_CTL(pipe)) & FDI_TX_ENABLE;
}
- I915_STATE_WARN(dev_priv, cur_state != state,
- "FDI TX state assertion failure (expected %s, current %s)\n",
- str_on_off(state), str_on_off(cur_state));
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "FDI TX state assertion failure (expected %s, current %s)\n",
+ str_on_off(state), str_on_off(cur_state));
}
void assert_fdi_tx_enabled(struct drm_i915_private *i915, enum pipe pipe)
@@ -59,12 +60,13 @@ void assert_fdi_tx_disabled(struct drm_i915_private *i915, enum pipe pipe)
static void assert_fdi_rx(struct drm_i915_private *dev_priv,
enum pipe pipe, bool state)
{
+ struct intel_display *display = &dev_priv->display;
bool cur_state;
- cur_state = intel_de_read(dev_priv, FDI_RX_CTL(pipe)) & FDI_RX_ENABLE;
- I915_STATE_WARN(dev_priv, cur_state != state,
- "FDI RX state assertion failure (expected %s, current %s)\n",
- str_on_off(state), str_on_off(cur_state));
+ cur_state = intel_de_read(display, FDI_RX_CTL(pipe)) & FDI_RX_ENABLE;
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "FDI RX state assertion failure (expected %s, current %s)\n",
+ str_on_off(state), str_on_off(cur_state));
}
void assert_fdi_rx_enabled(struct drm_i915_private *i915, enum pipe pipe)
@@ -80,6 +82,7 @@ void assert_fdi_rx_disabled(struct drm_i915_private *i915, enum pipe pipe)
void assert_fdi_tx_pll_enabled(struct drm_i915_private *i915,
enum pipe pipe)
{
+ struct intel_display *display = &i915->display;
bool cur_state;
/* ILK FDI PLL is always enabled */
@@ -87,23 +90,24 @@ void assert_fdi_tx_pll_enabled(struct drm_i915_private *i915,
return;
/* On Haswell, DDI ports are responsible for the FDI PLL setup */
- if (HAS_DDI(i915))
+ if (HAS_DDI(display))
return;
- cur_state = intel_de_read(i915, FDI_TX_CTL(pipe)) & FDI_TX_PLL_ENABLE;
- I915_STATE_WARN(i915, !cur_state,
- "FDI TX PLL assertion failure, should be active but is disabled\n");
+ cur_state = intel_de_read(display, FDI_TX_CTL(pipe)) & FDI_TX_PLL_ENABLE;
+ INTEL_DISPLAY_STATE_WARN(display, !cur_state,
+ "FDI TX PLL assertion failure, should be active but is disabled\n");
}
static void assert_fdi_rx_pll(struct drm_i915_private *i915,
enum pipe pipe, bool state)
{
+ struct intel_display *display = &i915->display;
bool cur_state;
- cur_state = intel_de_read(i915, FDI_RX_CTL(pipe)) & FDI_RX_PLL_ENABLE;
- I915_STATE_WARN(i915, cur_state != state,
- "FDI RX PLL assertion failure (expected %s, current %s)\n",
- str_on_off(state), str_on_off(cur_state));
+ cur_state = intel_de_read(display, FDI_RX_CTL(pipe)) & FDI_RX_PLL_ENABLE;
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "FDI RX PLL assertion failure (expected %s, current %s)\n",
+ str_on_off(state), str_on_off(cur_state));
}
void assert_fdi_rx_pll_enabled(struct drm_i915_private *i915, enum pipe pipe)
@@ -137,6 +141,7 @@ void intel_fdi_link_train(struct intel_crtc *crtc,
*/
int intel_fdi_add_affected_crtcs(struct intel_atomic_state *state)
{
+ struct intel_display *display = to_intel_display(state);
struct drm_i915_private *i915 = to_i915(state->base.dev);
const struct intel_crtc_state *old_crtc_state;
const struct intel_crtc_state *new_crtc_state;
@@ -145,7 +150,7 @@ int intel_fdi_add_affected_crtcs(struct intel_atomic_state *state)
if (!IS_IVYBRIDGE(i915) || INTEL_NUM_PIPES(i915) != 3)
return 0;
- crtc = intel_crtc_for_pipe(i915, PIPE_C);
+ crtc = intel_crtc_for_pipe(display, PIPE_C);
new_crtc_state = intel_atomic_get_new_crtc_state(state, crtc);
if (!new_crtc_state)
return 0;
@@ -157,7 +162,7 @@ int intel_fdi_add_affected_crtcs(struct intel_atomic_state *state)
if (!old_crtc_state->fdi_lanes)
return 0;
- crtc = intel_crtc_for_pipe(i915, PIPE_B);
+ crtc = intel_crtc_for_pipe(display, PIPE_B);
new_crtc_state = intel_atomic_get_crtc_state(&state->base, crtc);
if (IS_ERR(new_crtc_state))
return PTR_ERR(new_crtc_state);
@@ -184,6 +189,7 @@ static int ilk_check_fdi_lanes(struct drm_device *dev, enum pipe pipe,
struct intel_crtc_state *pipe_config,
enum pipe *pipe_to_reduce)
{
+ struct intel_display *display = to_intel_display(dev);
struct drm_i915_private *dev_priv = to_i915(dev);
struct drm_atomic_state *state = pipe_config->uapi.state;
struct intel_crtc *other_crtc;
@@ -223,7 +229,7 @@ static int ilk_check_fdi_lanes(struct drm_device *dev, enum pipe pipe,
if (pipe_config->fdi_lanes <= 2)
return 0;
- other_crtc = intel_crtc_for_pipe(dev_priv, PIPE_C);
+ other_crtc = intel_crtc_for_pipe(display, PIPE_C);
other_crtc_state =
intel_atomic_get_crtc_state(state, other_crtc);
if (IS_ERR(other_crtc_state))
@@ -244,7 +250,7 @@ static int ilk_check_fdi_lanes(struct drm_device *dev, enum pipe pipe,
return -EINVAL;
}
- other_crtc = intel_crtc_for_pipe(dev_priv, PIPE_B);
+ other_crtc = intel_crtc_for_pipe(display, PIPE_B);
other_crtc_state =
intel_atomic_get_crtc_state(state, other_crtc);
if (IS_ERR(other_crtc_state))
diff --git a/drivers/gpu/drm/i915/display/intel_fifo_underrun.c b/drivers/gpu/drm/i915/display/intel_fifo_underrun.c
index 8949fbb1cc60..4ffbaf44c21b 100644
--- a/drivers/gpu/drm/i915/display/intel_fifo_underrun.c
+++ b/drivers/gpu/drm/i915/display/intel_fifo_underrun.c
@@ -57,6 +57,7 @@
static bool ivb_can_enable_err_int(struct drm_device *dev)
{
+ struct intel_display *display = to_intel_display(dev);
struct drm_i915_private *dev_priv = to_i915(dev);
struct intel_crtc *crtc;
enum pipe pipe;
@@ -64,7 +65,7 @@ static bool ivb_can_enable_err_int(struct drm_device *dev)
lockdep_assert_held(&dev_priv->irq_lock);
for_each_pipe(dev_priv, pipe) {
- crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ crtc = intel_crtc_for_pipe(display, pipe);
if (crtc->cpu_fifo_underrun_disabled)
return false;
@@ -75,6 +76,7 @@ static bool ivb_can_enable_err_int(struct drm_device *dev)
static bool cpt_can_enable_serr_int(struct drm_device *dev)
{
+ struct intel_display *display = to_intel_display(dev);
struct drm_i915_private *dev_priv = to_i915(dev);
enum pipe pipe;
struct intel_crtc *crtc;
@@ -82,7 +84,7 @@ static bool cpt_can_enable_serr_int(struct drm_device *dev)
lockdep_assert_held(&dev_priv->irq_lock);
for_each_pipe(dev_priv, pipe) {
- crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ crtc = intel_crtc_for_pipe(display, pipe);
if (crtc->pch_fifo_underrun_disabled)
return false;
@@ -282,8 +284,9 @@ static void cpt_set_fifo_underrun_reporting(struct drm_device *dev,
static bool __intel_set_cpu_fifo_underrun_reporting(struct drm_device *dev,
enum pipe pipe, bool enable)
{
+ struct intel_display *display = to_intel_display(dev);
struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
bool old;
lockdep_assert_held(&dev_priv->irq_lock);
@@ -351,8 +354,9 @@ bool intel_set_pch_fifo_underrun_reporting(struct drm_i915_private *dev_priv,
enum pipe pch_transcoder,
bool enable)
{
+ struct intel_display *display = &dev_priv->display;
struct intel_crtc *crtc =
- intel_crtc_for_pipe(dev_priv, pch_transcoder);
+ intel_crtc_for_pipe(display, pch_transcoder);
unsigned long flags;
bool old;
@@ -395,7 +399,8 @@ bool intel_set_pch_fifo_underrun_reporting(struct drm_i915_private *dev_priv,
void intel_cpu_fifo_underrun_irq_handler(struct drm_i915_private *dev_priv,
enum pipe pipe)
{
- struct intel_crtc *crtc = intel_crtc_for_pipe(dev_priv, pipe);
+ struct intel_display *display = &dev_priv->display;
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
u32 underruns = 0;
/* We may be called too early in init, thanks BIOS! */
diff --git a/drivers/gpu/drm/i915/display/intel_gmbus.c b/drivers/gpu/drm/i915/display/intel_gmbus.c
index 88b37deb5c82..3a268e70e308 100644
--- a/drivers/gpu/drm/i915/display/intel_gmbus.c
+++ b/drivers/gpu/drm/i915/display/intel_gmbus.c
@@ -48,7 +48,7 @@ struct intel_gmbus {
u32 reg0;
i915_reg_t gpio_reg;
struct i2c_algo_bit_data bit_algo;
- struct drm_i915_private *i915;
+ struct intel_display *display;
};
enum gmbus_gpio {
@@ -149,9 +149,10 @@ static const struct gmbus_pin gmbus_pins_mtp[] = {
[GMBUS_PIN_12_TC4_ICP] = { "tc4", GPIOM },
};
-static const struct gmbus_pin *get_gmbus_pin(struct drm_i915_private *i915,
+static const struct gmbus_pin *get_gmbus_pin(struct intel_display *display,
unsigned int pin)
{
+ struct drm_i915_private *i915 = to_i915(display->drm);
const struct gmbus_pin *pins;
size_t size;
@@ -173,7 +174,7 @@ static const struct gmbus_pin *get_gmbus_pin(struct drm_i915_private *i915,
} else if (IS_GEMINILAKE(i915) || IS_BROXTON(i915)) {
pins = gmbus_pins_bxt;
size = ARRAY_SIZE(gmbus_pins_bxt);
- } else if (DISPLAY_VER(i915) == 9) {
+ } else if (DISPLAY_VER(display) == 9) {
pins = gmbus_pins_skl;
size = ARRAY_SIZE(gmbus_pins_skl);
} else if (IS_BROADWELL(i915)) {
@@ -190,9 +191,9 @@ static const struct gmbus_pin *get_gmbus_pin(struct drm_i915_private *i915,
return &pins[pin];
}
-bool intel_gmbus_is_valid_pin(struct drm_i915_private *i915, unsigned int pin)
+bool intel_gmbus_is_valid_pin(struct intel_display *display, unsigned int pin)
{
- return get_gmbus_pin(i915, pin);
+ return get_gmbus_pin(display, pin);
}
/* Intel GPIO access functions */
@@ -206,42 +207,45 @@ to_intel_gmbus(struct i2c_adapter *i2c)
}
void
-intel_gmbus_reset(struct drm_i915_private *i915)
+intel_gmbus_reset(struct intel_display *display)
{
- intel_de_write(i915, GMBUS0(i915), 0);
- intel_de_write(i915, GMBUS4(i915), 0);
+ intel_de_write(display, GMBUS0(display), 0);
+ intel_de_write(display, GMBUS4(display), 0);
}
-static void pnv_gmbus_clock_gating(struct drm_i915_private *i915,
+static void pnv_gmbus_clock_gating(struct intel_display *display,
bool enable)
{
/* When using bit bashing for I2C, this bit needs to be set to 1 */
- intel_de_rmw(i915, DSPCLK_GATE_D(i915), PNV_GMBUSUNIT_CLOCK_GATE_DISABLE,
+ intel_de_rmw(display, DSPCLK_GATE_D(display),
+ PNV_GMBUSUNIT_CLOCK_GATE_DISABLE,
!enable ? PNV_GMBUSUNIT_CLOCK_GATE_DISABLE : 0);
}
-static void pch_gmbus_clock_gating(struct drm_i915_private *i915,
+static void pch_gmbus_clock_gating(struct intel_display *display,
bool enable)
{
- intel_de_rmw(i915, SOUTH_DSPCLK_GATE_D, PCH_GMBUSUNIT_CLOCK_GATE_DISABLE,
+ intel_de_rmw(display, SOUTH_DSPCLK_GATE_D,
+ PCH_GMBUSUNIT_CLOCK_GATE_DISABLE,
!enable ? PCH_GMBUSUNIT_CLOCK_GATE_DISABLE : 0);
}
-static void bxt_gmbus_clock_gating(struct drm_i915_private *i915,
+static void bxt_gmbus_clock_gating(struct intel_display *display,
bool enable)
{
- intel_de_rmw(i915, GEN9_CLKGATE_DIS_4, BXT_GMBUS_GATING_DIS,
+ intel_de_rmw(display, GEN9_CLKGATE_DIS_4, BXT_GMBUS_GATING_DIS,
!enable ? BXT_GMBUS_GATING_DIS : 0);
}
static u32 get_reserved(struct intel_gmbus *bus)
{
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
+ struct drm_i915_private *i915 = to_i915(display->drm);
u32 reserved = 0;
/* On most chips, these bits must be preserved in software. */
if (!IS_I830(i915) && !IS_I845G(i915))
- reserved = intel_de_read_notrace(i915, bus->gpio_reg) &
+ reserved = intel_de_read_notrace(display, bus->gpio_reg) &
(GPIO_DATA_PULLUP_DISABLE | GPIO_CLOCK_PULLUP_DISABLE);
return reserved;
@@ -250,31 +254,31 @@ static u32 get_reserved(struct intel_gmbus *bus)
static int get_clock(void *data)
{
struct intel_gmbus *bus = data;
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
u32 reserved = get_reserved(bus);
- intel_de_write_notrace(i915, bus->gpio_reg, reserved | GPIO_CLOCK_DIR_MASK);
- intel_de_write_notrace(i915, bus->gpio_reg, reserved);
+ intel_de_write_notrace(display, bus->gpio_reg, reserved | GPIO_CLOCK_DIR_MASK);
+ intel_de_write_notrace(display, bus->gpio_reg, reserved);
- return (intel_de_read_notrace(i915, bus->gpio_reg) & GPIO_CLOCK_VAL_IN) != 0;
+ return (intel_de_read_notrace(display, bus->gpio_reg) & GPIO_CLOCK_VAL_IN) != 0;
}
static int get_data(void *data)
{
struct intel_gmbus *bus = data;
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
u32 reserved = get_reserved(bus);
- intel_de_write_notrace(i915, bus->gpio_reg, reserved | GPIO_DATA_DIR_MASK);
- intel_de_write_notrace(i915, bus->gpio_reg, reserved);
+ intel_de_write_notrace(display, bus->gpio_reg, reserved | GPIO_DATA_DIR_MASK);
+ intel_de_write_notrace(display, bus->gpio_reg, reserved);
- return (intel_de_read_notrace(i915, bus->gpio_reg) & GPIO_DATA_VAL_IN) != 0;
+ return (intel_de_read_notrace(display, bus->gpio_reg) & GPIO_DATA_VAL_IN) != 0;
}
static void set_clock(void *data, int state_high)
{
struct intel_gmbus *bus = data;
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
u32 reserved = get_reserved(bus);
u32 clock_bits;
@@ -284,14 +288,14 @@ static void set_clock(void *data, int state_high)
clock_bits = GPIO_CLOCK_DIR_OUT | GPIO_CLOCK_DIR_MASK |
GPIO_CLOCK_VAL_MASK;
- intel_de_write_notrace(i915, bus->gpio_reg, reserved | clock_bits);
- intel_de_posting_read(i915, bus->gpio_reg);
+ intel_de_write_notrace(display, bus->gpio_reg, reserved | clock_bits);
+ intel_de_posting_read(display, bus->gpio_reg);
}
static void set_data(void *data, int state_high)
{
struct intel_gmbus *bus = data;
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
u32 reserved = get_reserved(bus);
u32 data_bits;
@@ -301,20 +305,21 @@ static void set_data(void *data, int state_high)
data_bits = GPIO_DATA_DIR_OUT | GPIO_DATA_DIR_MASK |
GPIO_DATA_VAL_MASK;
- intel_de_write_notrace(i915, bus->gpio_reg, reserved | data_bits);
- intel_de_posting_read(i915, bus->gpio_reg);
+ intel_de_write_notrace(display, bus->gpio_reg, reserved | data_bits);
+ intel_de_posting_read(display, bus->gpio_reg);
}
static int
intel_gpio_pre_xfer(struct i2c_adapter *adapter)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
+ struct drm_i915_private *i915 = to_i915(display->drm);
- intel_gmbus_reset(i915);
+ intel_gmbus_reset(display);
if (IS_PINEVIEW(i915))
- pnv_gmbus_clock_gating(i915, false);
+ pnv_gmbus_clock_gating(display, false);
set_data(bus, 1);
set_clock(bus, 1);
@@ -326,13 +331,14 @@ static void
intel_gpio_post_xfer(struct i2c_adapter *adapter)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
+ struct drm_i915_private *i915 = to_i915(display->drm);
set_data(bus, 1);
set_clock(bus, 1);
if (IS_PINEVIEW(i915))
- pnv_gmbus_clock_gating(i915, true);
+ pnv_gmbus_clock_gating(display, true);
}
static void
@@ -355,16 +361,17 @@ intel_gpio_setup(struct intel_gmbus *bus, i915_reg_t gpio_reg)
algo->data = bus;
}
-static bool has_gmbus_irq(struct drm_i915_private *i915)
+static bool has_gmbus_irq(struct intel_display *display)
{
+ struct drm_i915_private *i915 = to_i915(display->drm);
/*
* encoder->shutdown() may want to use GMBUS
* after irqs have already been disabled.
*/
- return HAS_GMBUS_IRQ(i915) && intel_irqs_enabled(i915);
+ return HAS_GMBUS_IRQ(display) && intel_irqs_enabled(i915);
}
-static int gmbus_wait(struct drm_i915_private *i915, u32 status, u32 irq_en)
+static int gmbus_wait(struct intel_display *display, u32 status, u32 irq_en)
{
DEFINE_WAIT(wait);
u32 gmbus2;
@@ -374,21 +381,21 @@ static int gmbus_wait(struct drm_i915_private *i915, u32 status, u32 irq_en)
* we also need to check for NAKs besides the hw ready/idle signal, we
* need to wake up periodically and check that ourselves.
*/
- if (!has_gmbus_irq(i915))
+ if (!has_gmbus_irq(display))
irq_en = 0;
- add_wait_queue(&i915->display.gmbus.wait_queue, &wait);
- intel_de_write_fw(i915, GMBUS4(i915), irq_en);
+ add_wait_queue(&display->gmbus.wait_queue, &wait);
+ intel_de_write_fw(display, GMBUS4(display), irq_en);
status |= GMBUS_SATOER;
- ret = wait_for_us((gmbus2 = intel_de_read_fw(i915, GMBUS2(i915))) & status,
+ ret = wait_for_us((gmbus2 = intel_de_read_fw(display, GMBUS2(display))) & status,
2);
if (ret)
- ret = wait_for((gmbus2 = intel_de_read_fw(i915, GMBUS2(i915))) & status,
+ ret = wait_for((gmbus2 = intel_de_read_fw(display, GMBUS2(display))) & status,
50);
- intel_de_write_fw(i915, GMBUS4(i915), 0);
- remove_wait_queue(&i915->display.gmbus.wait_queue, &wait);
+ intel_de_write_fw(display, GMBUS4(display), 0);
+ remove_wait_queue(&display->gmbus.wait_queue, &wait);
if (gmbus2 & GMBUS_SATOER)
return -ENXIO;
@@ -397,7 +404,7 @@ static int gmbus_wait(struct drm_i915_private *i915, u32 status, u32 irq_en)
}
static int
-gmbus_wait_idle(struct drm_i915_private *i915)
+gmbus_wait_idle(struct intel_display *display)
{
DEFINE_WAIT(wait);
u32 irq_enable;
@@ -405,33 +412,33 @@ gmbus_wait_idle(struct drm_i915_private *i915)
/* Important: The hw handles only the first bit, so set only one! */
irq_enable = 0;
- if (has_gmbus_irq(i915))
+ if (has_gmbus_irq(display))
irq_enable = GMBUS_IDLE_EN;
- add_wait_queue(&i915->display.gmbus.wait_queue, &wait);
- intel_de_write_fw(i915, GMBUS4(i915), irq_enable);
+ add_wait_queue(&display->gmbus.wait_queue, &wait);
+ intel_de_write_fw(display, GMBUS4(display), irq_enable);
- ret = intel_de_wait_fw(i915, GMBUS2(i915), GMBUS_ACTIVE, 0, 10);
+ ret = intel_de_wait_fw(display, GMBUS2(display), GMBUS_ACTIVE, 0, 10);
- intel_de_write_fw(i915, GMBUS4(i915), 0);
- remove_wait_queue(&i915->display.gmbus.wait_queue, &wait);
+ intel_de_write_fw(display, GMBUS4(display), 0);
+ remove_wait_queue(&display->gmbus.wait_queue, &wait);
return ret;
}
-static unsigned int gmbus_max_xfer_size(struct drm_i915_private *i915)
+static unsigned int gmbus_max_xfer_size(struct intel_display *display)
{
- return DISPLAY_VER(i915) >= 9 ? GEN9_GMBUS_BYTE_COUNT_MAX :
+ return DISPLAY_VER(display) >= 9 ? GEN9_GMBUS_BYTE_COUNT_MAX :
GMBUS_BYTE_COUNT_MAX;
}
static int
-gmbus_xfer_read_chunk(struct drm_i915_private *i915,
+gmbus_xfer_read_chunk(struct intel_display *display,
unsigned short addr, u8 *buf, unsigned int len,
u32 gmbus0_reg, u32 gmbus1_index)
{
unsigned int size = len;
- bool burst_read = len > gmbus_max_xfer_size(i915);
+ bool burst_read = len > gmbus_max_xfer_size(display);
bool extra_byte_added = false;
if (burst_read) {
@@ -444,21 +451,21 @@ gmbus_xfer_read_chunk(struct drm_i915_private *i915,
len++;
}
size = len % 256 + 256;
- intel_de_write_fw(i915, GMBUS0(i915),
+ intel_de_write_fw(display, GMBUS0(display),
gmbus0_reg | GMBUS_BYTE_CNT_OVERRIDE);
}
- intel_de_write_fw(i915, GMBUS1(i915),
+ intel_de_write_fw(display, GMBUS1(display),
gmbus1_index | GMBUS_CYCLE_WAIT | (size << GMBUS_BYTE_COUNT_SHIFT) | (addr << GMBUS_SLAVE_ADDR_SHIFT) | GMBUS_SLAVE_READ | GMBUS_SW_RDY);
while (len) {
int ret;
u32 val, loop = 0;
- ret = gmbus_wait(i915, GMBUS_HW_RDY, GMBUS_HW_RDY_EN);
+ ret = gmbus_wait(display, GMBUS_HW_RDY, GMBUS_HW_RDY_EN);
if (ret)
return ret;
- val = intel_de_read_fw(i915, GMBUS3(i915));
+ val = intel_de_read_fw(display, GMBUS3(display));
do {
if (extra_byte_added && len == 1) {
len--;
@@ -471,7 +478,7 @@ gmbus_xfer_read_chunk(struct drm_i915_private *i915,
if (burst_read && len == size - 4)
/* Reset the override bit */
- intel_de_write_fw(i915, GMBUS0(i915), gmbus0_reg);
+ intel_de_write_fw(display, GMBUS0(display), gmbus0_reg);
}
return 0;
@@ -488,9 +495,10 @@ gmbus_xfer_read_chunk(struct drm_i915_private *i915,
#define INTEL_GMBUS_BURST_READ_MAX_LEN 767U
static int
-gmbus_xfer_read(struct drm_i915_private *i915, struct i2c_msg *msg,
+gmbus_xfer_read(struct intel_display *display, struct i2c_msg *msg,
u32 gmbus0_reg, u32 gmbus1_index)
{
+ struct drm_i915_private *i915 = to_i915(display->drm);
u8 *buf = msg->buf;
unsigned int rx_size = msg->len;
unsigned int len;
@@ -500,9 +508,9 @@ gmbus_xfer_read(struct drm_i915_private *i915, struct i2c_msg *msg,
if (HAS_GMBUS_BURST_READ(i915))
len = min(rx_size, INTEL_GMBUS_BURST_READ_MAX_LEN);
else
- len = min(rx_size, gmbus_max_xfer_size(i915));
+ len = min(rx_size, gmbus_max_xfer_size(display));
- ret = gmbus_xfer_read_chunk(i915, msg->addr, buf, len,
+ ret = gmbus_xfer_read_chunk(display, msg->addr, buf, len,
gmbus0_reg, gmbus1_index);
if (ret)
return ret;
@@ -515,7 +523,7 @@ gmbus_xfer_read(struct drm_i915_private *i915, struct i2c_msg *msg,
}
static int
-gmbus_xfer_write_chunk(struct drm_i915_private *i915,
+gmbus_xfer_write_chunk(struct intel_display *display,
unsigned short addr, u8 *buf, unsigned int len,
u32 gmbus1_index)
{
@@ -528,8 +536,8 @@ gmbus_xfer_write_chunk(struct drm_i915_private *i915,
len -= 1;
}
- intel_de_write_fw(i915, GMBUS3(i915), val);
- intel_de_write_fw(i915, GMBUS1(i915),
+ intel_de_write_fw(display, GMBUS3(display), val);
+ intel_de_write_fw(display, GMBUS1(display),
gmbus1_index | GMBUS_CYCLE_WAIT | (chunk_size << GMBUS_BYTE_COUNT_SHIFT) | (addr << GMBUS_SLAVE_ADDR_SHIFT) | GMBUS_SLAVE_WRITE | GMBUS_SW_RDY);
while (len) {
int ret;
@@ -539,9 +547,9 @@ gmbus_xfer_write_chunk(struct drm_i915_private *i915,
val |= *buf++ << (8 * loop);
} while (--len && ++loop < 4);
- intel_de_write_fw(i915, GMBUS3(i915), val);
+ intel_de_write_fw(display, GMBUS3(display), val);
- ret = gmbus_wait(i915, GMBUS_HW_RDY, GMBUS_HW_RDY_EN);
+ ret = gmbus_wait(display, GMBUS_HW_RDY, GMBUS_HW_RDY_EN);
if (ret)
return ret;
}
@@ -550,7 +558,7 @@ gmbus_xfer_write_chunk(struct drm_i915_private *i915,
}
static int
-gmbus_xfer_write(struct drm_i915_private *i915, struct i2c_msg *msg,
+gmbus_xfer_write(struct intel_display *display, struct i2c_msg *msg,
u32 gmbus1_index)
{
u8 *buf = msg->buf;
@@ -559,9 +567,9 @@ gmbus_xfer_write(struct drm_i915_private *i915, struct i2c_msg *msg,
int ret;
do {
- len = min(tx_size, gmbus_max_xfer_size(i915));
+ len = min(tx_size, gmbus_max_xfer_size(display));
- ret = gmbus_xfer_write_chunk(i915, msg->addr, buf, len,
+ ret = gmbus_xfer_write_chunk(display, msg->addr, buf, len,
gmbus1_index);
if (ret)
return ret;
@@ -588,7 +596,7 @@ gmbus_is_index_xfer(struct i2c_msg *msgs, int i, int num)
}
static int
-gmbus_index_xfer(struct drm_i915_private *i915, struct i2c_msg *msgs,
+gmbus_index_xfer(struct intel_display *display, struct i2c_msg *msgs,
u32 gmbus0_reg)
{
u32 gmbus1_index = 0;
@@ -604,17 +612,17 @@ gmbus_index_xfer(struct drm_i915_private *i915, struct i2c_msg *msgs,
/* GMBUS5 holds 16-bit index */
if (gmbus5)
- intel_de_write_fw(i915, GMBUS5(i915), gmbus5);
+ intel_de_write_fw(display, GMBUS5(display), gmbus5);
if (msgs[1].flags & I2C_M_RD)
- ret = gmbus_xfer_read(i915, &msgs[1], gmbus0_reg,
+ ret = gmbus_xfer_read(display, &msgs[1], gmbus0_reg,
gmbus1_index);
else
- ret = gmbus_xfer_write(i915, &msgs[1], gmbus1_index);
+ ret = gmbus_xfer_write(display, &msgs[1], gmbus1_index);
/* Clear GMBUS5 after each index transfer */
if (gmbus5)
- intel_de_write_fw(i915, GMBUS5(i915), 0);
+ intel_de_write_fw(display, GMBUS5(display), 0);
return ret;
}
@@ -624,34 +632,35 @@ do_gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num,
u32 gmbus0_source)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
+ struct drm_i915_private *i915 = to_i915(display->drm);
int i = 0, inc, try = 0;
int ret = 0;
/* Display WA #0868: skl,bxt,kbl,cfl,glk */
if (IS_GEMINILAKE(i915) || IS_BROXTON(i915))
- bxt_gmbus_clock_gating(i915, false);
+ bxt_gmbus_clock_gating(display, false);
else if (HAS_PCH_SPT(i915) || HAS_PCH_CNP(i915))
- pch_gmbus_clock_gating(i915, false);
+ pch_gmbus_clock_gating(display, false);
retry:
- intel_de_write_fw(i915, GMBUS0(i915), gmbus0_source | bus->reg0);
+ intel_de_write_fw(display, GMBUS0(display), gmbus0_source | bus->reg0);
for (; i < num; i += inc) {
inc = 1;
if (gmbus_is_index_xfer(msgs, i, num)) {
- ret = gmbus_index_xfer(i915, &msgs[i],
+ ret = gmbus_index_xfer(display, &msgs[i],
gmbus0_source | bus->reg0);
inc = 2; /* an index transmission is two msgs */
} else if (msgs[i].flags & I2C_M_RD) {
- ret = gmbus_xfer_read(i915, &msgs[i],
+ ret = gmbus_xfer_read(display, &msgs[i],
gmbus0_source | bus->reg0, 0);
} else {
- ret = gmbus_xfer_write(i915, &msgs[i], 0);
+ ret = gmbus_xfer_write(display, &msgs[i], 0);
}
if (!ret)
- ret = gmbus_wait(i915,
+ ret = gmbus_wait(display,
GMBUS_HW_WAIT_PHASE, GMBUS_HW_WAIT_EN);
if (ret == -ETIMEDOUT)
goto timeout;
@@ -663,19 +672,19 @@ do_gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num,
* a STOP on the very first cycle. To simplify the code we
* unconditionally generate the STOP condition with an additional gmbus
* cycle. */
- intel_de_write_fw(i915, GMBUS1(i915), GMBUS_CYCLE_STOP | GMBUS_SW_RDY);
+ intel_de_write_fw(display, GMBUS1(display), GMBUS_CYCLE_STOP | GMBUS_SW_RDY);
/* Mark the GMBUS interface as disabled after waiting for idle.
* We will re-enable it at the start of the next xfer,
* till then let it sleep.
*/
- if (gmbus_wait_idle(i915)) {
- drm_dbg_kms(&i915->drm,
+ if (gmbus_wait_idle(display)) {
+ drm_dbg_kms(display->drm,
"GMBUS [%s] timed out waiting for idle\n",
adapter->name);
ret = -ETIMEDOUT;
}
- intel_de_write_fw(i915, GMBUS0(i915), 0);
+ intel_de_write_fw(display, GMBUS0(display), 0);
ret = ret ?: i;
goto out;
@@ -694,8 +703,8 @@ do_gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num,
* it's slow responding and only answers on the 2nd retry.
*/
ret = -ENXIO;
- if (gmbus_wait_idle(i915)) {
- drm_dbg_kms(&i915->drm,
+ if (gmbus_wait_idle(display)) {
+ drm_dbg_kms(display->drm,
"GMBUS [%s] timed out after NAK\n",
adapter->name);
ret = -ETIMEDOUT;
@@ -705,11 +714,11 @@ do_gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num,
* of resetting the GMBUS controller and so clearing the
* BUS_ERROR raised by the target's NAK.
*/
- intel_de_write_fw(i915, GMBUS1(i915), GMBUS_SW_CLR_INT);
- intel_de_write_fw(i915, GMBUS1(i915), 0);
- intel_de_write_fw(i915, GMBUS0(i915), 0);
+ intel_de_write_fw(display, GMBUS1(display), GMBUS_SW_CLR_INT);
+ intel_de_write_fw(display, GMBUS1(display), 0);
+ intel_de_write_fw(display, GMBUS0(display), 0);
- drm_dbg_kms(&i915->drm, "GMBUS [%s] NAK for addr: %04x %c(%d)\n",
+ drm_dbg_kms(display->drm, "GMBUS [%s] NAK for addr: %04x %c(%d)\n",
adapter->name, msgs[i].addr,
(msgs[i].flags & I2C_M_RD) ? 'r' : 'w', msgs[i].len);
@@ -720,7 +729,7 @@ do_gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num,
* drm_do_probe_ddc_edid, which bails out on the first -ENXIO.
*/
if (ret == -ENXIO && i == 0 && try++ == 0) {
- drm_dbg_kms(&i915->drm,
+ drm_dbg_kms(display->drm,
"GMBUS [%s] NAK on first message, retry\n",
adapter->name);
goto retry;
@@ -729,10 +738,10 @@ do_gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num,
goto out;
timeout:
- drm_dbg_kms(&i915->drm,
+ drm_dbg_kms(display->drm,
"GMBUS [%s] timed out, falling back to bit banging on pin %d\n",
bus->adapter.name, bus->reg0 & 0xff);
- intel_de_write_fw(i915, GMBUS0(i915), 0);
+ intel_de_write_fw(display, GMBUS0(display), 0);
/*
* Hardware may not support GMBUS over these pins? Try GPIO bitbanging
@@ -743,9 +752,9 @@ do_gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num,
out:
/* Display WA #0868: skl,bxt,kbl,cfl,glk */
if (IS_GEMINILAKE(i915) || IS_BROXTON(i915))
- bxt_gmbus_clock_gating(i915, true);
+ bxt_gmbus_clock_gating(display, true);
else if (HAS_PCH_SPT(i915) || HAS_PCH_CNP(i915))
- pch_gmbus_clock_gating(i915, true);
+ pch_gmbus_clock_gating(display, true);
return ret;
}
@@ -754,7 +763,8 @@ static int
gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
+ struct drm_i915_private *i915 = to_i915(display->drm);
intel_wakeref_t wakeref;
int ret;
@@ -778,7 +788,8 @@ gmbus_xfer(struct i2c_adapter *adapter, struct i2c_msg *msgs, int num)
int intel_gmbus_output_aksv(struct i2c_adapter *adapter)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
+ struct drm_i915_private *i915 = to_i915(display->drm);
u8 cmd = DRM_HDCP_DDC_AKSV;
u8 buf[DRM_HDCP_KSV_LEN] = {};
struct i2c_msg msgs[] = {
@@ -799,7 +810,7 @@ int intel_gmbus_output_aksv(struct i2c_adapter *adapter)
int ret;
wakeref = intel_display_power_get(i915, POWER_DOMAIN_GMBUS);
- mutex_lock(&i915->display.gmbus.mutex);
+ mutex_lock(&display->gmbus.mutex);
/*
* In order to output Aksv to the receiver, use an indexed write to
@@ -808,7 +819,7 @@ int intel_gmbus_output_aksv(struct i2c_adapter *adapter)
*/
ret = do_gmbus_xfer(adapter, msgs, ARRAY_SIZE(msgs), GMBUS_AKSV_SELECT);
- mutex_unlock(&i915->display.gmbus.mutex);
+ mutex_unlock(&display->gmbus.mutex);
intel_display_power_put(i915, POWER_DOMAIN_GMBUS, wakeref);
return ret;
@@ -832,27 +843,27 @@ static void gmbus_lock_bus(struct i2c_adapter *adapter,
unsigned int flags)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
- mutex_lock(&i915->display.gmbus.mutex);
+ mutex_lock(&display->gmbus.mutex);
}
static int gmbus_trylock_bus(struct i2c_adapter *adapter,
unsigned int flags)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
- return mutex_trylock(&i915->display.gmbus.mutex);
+ return mutex_trylock(&display->gmbus.mutex);
}
static void gmbus_unlock_bus(struct i2c_adapter *adapter,
unsigned int flags)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
- mutex_unlock(&i915->display.gmbus.mutex);
+ mutex_unlock(&display->gmbus.mutex);
}
static const struct i2c_lock_operations gmbus_lock_ops = {
@@ -863,31 +874,32 @@ static const struct i2c_lock_operations gmbus_lock_ops = {
/**
* intel_gmbus_setup - instantiate all Intel i2c GMBuses
- * @i915: i915 device private
+ * @display: display device
*/
-int intel_gmbus_setup(struct drm_i915_private *i915)
+int intel_gmbus_setup(struct intel_display *display)
{
- struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
+ struct drm_i915_private *i915 = to_i915(display->drm);
+ struct pci_dev *pdev = to_pci_dev(display->drm->dev);
unsigned int pin;
int ret;
if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915))
- i915->display.gmbus.mmio_base = VLV_DISPLAY_BASE;
- else if (!HAS_GMCH(i915))
+ display->gmbus.mmio_base = VLV_DISPLAY_BASE;
+ else if (!HAS_GMCH(display))
/*
* Broxton uses the same PCH offsets for South Display Engine,
* even though it doesn't have a PCH.
*/
- i915->display.gmbus.mmio_base = PCH_DISPLAY_BASE;
+ display->gmbus.mmio_base = PCH_DISPLAY_BASE;
- mutex_init(&i915->display.gmbus.mutex);
- init_waitqueue_head(&i915->display.gmbus.wait_queue);
+ mutex_init(&display->gmbus.mutex);
+ init_waitqueue_head(&display->gmbus.wait_queue);
- for (pin = 0; pin < ARRAY_SIZE(i915->display.gmbus.bus); pin++) {
+ for (pin = 0; pin < ARRAY_SIZE(display->gmbus.bus); pin++) {
const struct gmbus_pin *gmbus_pin;
struct intel_gmbus *bus;
- gmbus_pin = get_gmbus_pin(i915, pin);
+ gmbus_pin = get_gmbus_pin(display, pin);
if (!gmbus_pin)
continue;
@@ -903,7 +915,7 @@ int intel_gmbus_setup(struct drm_i915_private *i915)
"i915 gmbus %s", gmbus_pin->name);
bus->adapter.dev.parent = &pdev->dev;
- bus->i915 = i915;
+ bus->display = display;
bus->adapter.algo = &gmbus_algorithm;
bus->adapter.lock_ops = &gmbus_lock_ops;
@@ -921,7 +933,7 @@ int intel_gmbus_setup(struct drm_i915_private *i915)
if (IS_I830(i915))
bus->force_bit = 1;
- intel_gpio_setup(bus, GPIO(i915, gmbus_pin->gpio));
+ intel_gpio_setup(bus, GPIO(display, gmbus_pin->gpio));
ret = i2c_add_adapter(&bus->adapter);
if (ret) {
@@ -929,43 +941,43 @@ int intel_gmbus_setup(struct drm_i915_private *i915)
goto err;
}
- i915->display.gmbus.bus[pin] = bus;
+ display->gmbus.bus[pin] = bus;
}
- intel_gmbus_reset(i915);
+ intel_gmbus_reset(display);
return 0;
err:
- intel_gmbus_teardown(i915);
+ intel_gmbus_teardown(display);
return ret;
}
-struct i2c_adapter *intel_gmbus_get_adapter(struct drm_i915_private *i915,
+struct i2c_adapter *intel_gmbus_get_adapter(struct intel_display *display,
unsigned int pin)
{
- if (drm_WARN_ON(&i915->drm, pin >= ARRAY_SIZE(i915->display.gmbus.bus) ||
- !i915->display.gmbus.bus[pin]))
+ if (drm_WARN_ON(display->drm, pin >= ARRAY_SIZE(display->gmbus.bus) ||
+ !display->gmbus.bus[pin]))
return NULL;
- return &i915->display.gmbus.bus[pin]->adapter;
+ return &display->gmbus.bus[pin]->adapter;
}
void intel_gmbus_force_bit(struct i2c_adapter *adapter, bool force_bit)
{
struct intel_gmbus *bus = to_intel_gmbus(adapter);
- struct drm_i915_private *i915 = bus->i915;
+ struct intel_display *display = bus->display;
- mutex_lock(&i915->display.gmbus.mutex);
+ mutex_lock(&display->gmbus.mutex);
bus->force_bit += force_bit ? 1 : -1;
- drm_dbg_kms(&i915->drm,
+ drm_dbg_kms(display->drm,
"%sabling bit-banging on %s. force bit now %d\n",
force_bit ? "en" : "dis", adapter->name,
bus->force_bit);
- mutex_unlock(&i915->display.gmbus.mutex);
+ mutex_unlock(&display->gmbus.mutex);
}
bool intel_gmbus_is_forced_bit(struct i2c_adapter *adapter)
@@ -975,25 +987,25 @@ bool intel_gmbus_is_forced_bit(struct i2c_adapter *adapter)
return bus->force_bit;
}
-void intel_gmbus_teardown(struct drm_i915_private *i915)
+void intel_gmbus_teardown(struct intel_display *display)
{
unsigned int pin;
- for (pin = 0; pin < ARRAY_SIZE(i915->display.gmbus.bus); pin++) {
+ for (pin = 0; pin < ARRAY_SIZE(display->gmbus.bus); pin++) {
struct intel_gmbus *bus;
- bus = i915->display.gmbus.bus[pin];
+ bus = display->gmbus.bus[pin];
if (!bus)
continue;
i2c_del_adapter(&bus->adapter);
kfree(bus);
- i915->display.gmbus.bus[pin] = NULL;
+ display->gmbus.bus[pin] = NULL;
}
}
-void intel_gmbus_irq_handler(struct drm_i915_private *i915)
+void intel_gmbus_irq_handler(struct intel_display *display)
{
- wake_up_all(&i915->display.gmbus.wait_queue);
+ wake_up_all(&display->gmbus.wait_queue);
}
diff --git a/drivers/gpu/drm/i915/display/intel_gmbus.h b/drivers/gpu/drm/i915/display/intel_gmbus.h
index 8111eb23e2af..35a200a9efc0 100644
--- a/drivers/gpu/drm/i915/display/intel_gmbus.h
+++ b/drivers/gpu/drm/i915/display/intel_gmbus.h
@@ -8,8 +8,8 @@
#include <linux/types.h>
-struct drm_i915_private;
struct i2c_adapter;
+struct intel_display;
#define GMBUS_PIN_DISABLED 0
#define GMBUS_PIN_SSC 1
@@ -34,18 +34,17 @@ struct i2c_adapter;
#define GMBUS_NUM_PINS 15 /* including 0 */
-int intel_gmbus_setup(struct drm_i915_private *dev_priv);
-void intel_gmbus_teardown(struct drm_i915_private *dev_priv);
-bool intel_gmbus_is_valid_pin(struct drm_i915_private *dev_priv,
- unsigned int pin);
+int intel_gmbus_setup(struct intel_display *display);
+void intel_gmbus_teardown(struct intel_display *display);
+bool intel_gmbus_is_valid_pin(struct intel_display *display, unsigned int pin);
int intel_gmbus_output_aksv(struct i2c_adapter *adapter);
struct i2c_adapter *
-intel_gmbus_get_adapter(struct drm_i915_private *dev_priv, unsigned int pin);
+intel_gmbus_get_adapter(struct intel_display *display, unsigned int pin);
void intel_gmbus_force_bit(struct i2c_adapter *adapter, bool force_bit);
bool intel_gmbus_is_forced_bit(struct i2c_adapter *adapter);
-void intel_gmbus_reset(struct drm_i915_private *dev_priv);
+void intel_gmbus_reset(struct intel_display *display);
-void intel_gmbus_irq_handler(struct drm_i915_private *i915);
+void intel_gmbus_irq_handler(struct intel_display *display);
#endif /* __INTEL_GMBUS_H__ */
diff --git a/drivers/gpu/drm/i915/display/intel_gmbus_regs.h b/drivers/gpu/drm/i915/display/intel_gmbus_regs.h
index 53aacbda983c..59bad1dda6d6 100644
--- a/drivers/gpu/drm/i915/display/intel_gmbus_regs.h
+++ b/drivers/gpu/drm/i915/display/intel_gmbus_regs.h
@@ -8,9 +8,9 @@
#include "i915_reg_defs.h"
-#define GMBUS_MMIO_BASE(__i915) ((__i915)->display.gmbus.mmio_base)
+#define __GMBUS_MMIO_BASE(__display) ((__display)->gmbus.mmio_base)
-#define GPIO(__i915, gpio) _MMIO(GMBUS_MMIO_BASE(__i915) + 0x5010 + 4 * (gpio))
+#define GPIO(__display, gpio) _MMIO(__GMBUS_MMIO_BASE(__display) + 0x5010 + 4 * (gpio))
#define GPIO_CLOCK_DIR_MASK (1 << 0)
#define GPIO_CLOCK_DIR_IN (0 << 1)
#define GPIO_CLOCK_DIR_OUT (1 << 1)
@@ -27,7 +27,7 @@
#define GPIO_DATA_PULLUP_DISABLE (1 << 13)
/* clock/port select */
-#define GMBUS0(__i915) _MMIO(GMBUS_MMIO_BASE(__i915) + 0x5100)
+#define GMBUS0(__display) _MMIO(__GMBUS_MMIO_BASE(__display) + 0x5100)
#define GMBUS_AKSV_SELECT (1 << 11)
#define GMBUS_RATE_100KHZ (0 << 8)
#define GMBUS_RATE_50KHZ (1 << 8)
@@ -37,7 +37,7 @@
#define GMBUS_BYTE_CNT_OVERRIDE (1 << 6)
/* command/status */
-#define GMBUS1(__i915) _MMIO(GMBUS_MMIO_BASE(__i915) + 0x5104)
+#define GMBUS1(__display) _MMIO(__GMBUS_MMIO_BASE(__display) + 0x5104)
#define GMBUS_SW_CLR_INT (1 << 31)
#define GMBUS_SW_RDY (1 << 30)
#define GMBUS_ENT (1 << 29) /* enable timeout */
@@ -54,7 +54,7 @@
#define GMBUS_SLAVE_WRITE (0 << 0)
/* status */
-#define GMBUS2(__i915) _MMIO(GMBUS_MMIO_BASE(__i915) + 0x5108)
+#define GMBUS2(__display) _MMIO(__GMBUS_MMIO_BASE(__display) + 0x5108)
#define GMBUS_INUSE (1 << 15)
#define GMBUS_HW_WAIT_PHASE (1 << 14)
#define GMBUS_STALL_TIMEOUT (1 << 13)
@@ -64,10 +64,10 @@
#define GMBUS_ACTIVE (1 << 9)
/* data buffer bytes 3-0 */
-#define GMBUS3(__i915) _MMIO(GMBUS_MMIO_BASE(__i915) + 0x510c)
+#define GMBUS3(__display) _MMIO(__GMBUS_MMIO_BASE(__display) + 0x510c)
/* interrupt mask (Pineview+) */
-#define GMBUS4(__i915) _MMIO(GMBUS_MMIO_BASE(__i915) + 0x5110)
+#define GMBUS4(__display) _MMIO(__GMBUS_MMIO_BASE(__display) + 0x5110)
#define GMBUS_SLAVE_TIMEOUT_EN (1 << 4)
#define GMBUS_NAK_EN (1 << 3)
#define GMBUS_IDLE_EN (1 << 2)
@@ -75,7 +75,7 @@
#define GMBUS_HW_RDY_EN (1 << 0)
/* byte index */
-#define GMBUS5(__i915) _MMIO(GMBUS_MMIO_BASE(__i915) + 0x5120)
+#define GMBUS5(__display) _MMIO(__GMBUS_MMIO_BASE(__display) + 0x5120)
#define GMBUS_2BYTE_INDEX_EN (1 << 31)
#endif /* __INTEL_GMBUS_REGS_H__ */
diff --git a/drivers/gpu/drm/i915/display/intel_hdmi.c b/drivers/gpu/drm/i915/display/intel_hdmi.c
index c5b2fbaeff89..1f3d099a9711 100644
--- a/drivers/gpu/drm/i915/display/intel_hdmi.c
+++ b/drivers/gpu/drm/i915/display/intel_hdmi.c
@@ -2913,7 +2913,6 @@ static struct intel_encoder *
get_encoder_by_ddc_pin(struct intel_encoder *encoder, u8 ddc_pin)
{
struct intel_display *display = to_intel_display(encoder);
- struct drm_i915_private *i915 = to_i915(encoder->base.dev);
struct intel_encoder *other;
for_each_intel_encoder(display->drm, other) {
@@ -2927,7 +2926,7 @@ get_encoder_by_ddc_pin(struct intel_encoder *encoder, u8 ddc_pin)
connector = enc_to_dig_port(other)->hdmi.attached_connector;
- if (connector && connector->base.ddc == intel_gmbus_get_adapter(i915, ddc_pin))
+ if (connector && connector->base.ddc == intel_gmbus_get_adapter(display, ddc_pin))
return other;
}
@@ -2937,7 +2936,6 @@ get_encoder_by_ddc_pin(struct intel_encoder *encoder, u8 ddc_pin)
static u8 intel_hdmi_ddc_pin(struct intel_encoder *encoder)
{
struct intel_display *display = to_intel_display(encoder);
- struct drm_i915_private *i915 = to_i915(encoder->base.dev);
struct intel_encoder *other;
const char *source;
u8 ddc_pin;
@@ -2950,7 +2948,7 @@ static u8 intel_hdmi_ddc_pin(struct intel_encoder *encoder)
source = "platform default";
}
- if (!intel_gmbus_is_valid_pin(i915, ddc_pin)) {
+ if (!intel_gmbus_is_valid_pin(display, ddc_pin)) {
drm_dbg_kms(display->drm,
"[ENCODER:%d:%s] Invalid DDC pin %d\n",
encoder->base.base.id, encoder->base.name, ddc_pin);
@@ -3048,7 +3046,7 @@ bool intel_hdmi_init_connector(struct intel_digital_port *dig_port,
drm_connector_init_with_ddc(dev, connector,
&intel_hdmi_connector_funcs,
DRM_MODE_CONNECTOR_HDMIA,
- intel_gmbus_get_adapter(dev_priv, ddc_pin));
+ intel_gmbus_get_adapter(display, ddc_pin));
drm_connector_helper_add(connector, &intel_hdmi_connector_helper_funcs);
diff --git a/drivers/gpu/drm/i915/display/intel_hotplug_irq.c b/drivers/gpu/drm/i915/display/intel_hotplug_irq.c
index 2c4e946d5575..b34e759db4ce 100644
--- a/drivers/gpu/drm/i915/display/intel_hotplug_irq.c
+++ b/drivers/gpu/drm/i915/display/intel_hotplug_irq.c
@@ -556,6 +556,7 @@ void xelpdp_pica_irq_handler(struct drm_i915_private *i915, u32 iir)
void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir)
{
+ struct intel_display *display = &dev_priv->display;
u32 ddi_hotplug_trigger = pch_iir & SDE_DDI_HOTPLUG_MASK_ICP;
u32 tc_hotplug_trigger = pch_iir & SDE_TC_HOTPLUG_MASK_ICP;
u32 pin_mask = 0, long_mask = 0;
@@ -589,11 +590,12 @@ void icp_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir)
intel_hpd_irq_handler(dev_priv, pin_mask, long_mask);
if (pch_iir & SDE_GMBUS_ICP)
- intel_gmbus_irq_handler(dev_priv);
+ intel_gmbus_irq_handler(display);
}
void spt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir)
{
+ struct intel_display *display = &dev_priv->display;
u32 hotplug_trigger = pch_iir & SDE_HOTPLUG_MASK_SPT &
~SDE_PORTE_HOTPLUG_SPT;
u32 hotplug2_trigger = pch_iir & SDE_PORTE_HOTPLUG_SPT;
@@ -625,7 +627,7 @@ void spt_irq_handler(struct drm_i915_private *dev_priv, u32 pch_iir)
intel_hpd_irq_handler(dev_priv, pin_mask, long_mask);
if (pch_iir & SDE_GMBUS_CPT)
- intel_gmbus_irq_handler(dev_priv);
+ intel_gmbus_irq_handler(display);
}
void ilk_hpd_irq_handler(struct drm_i915_private *dev_priv, u32 hotplug_trigger)
diff --git a/drivers/gpu/drm/i915/display/intel_link_bw.c b/drivers/gpu/drm/i915/display/intel_link_bw.c
index e7a9b860fac6..c87cd1d16d0a 100644
--- a/drivers/gpu/drm/i915/display/intel_link_bw.c
+++ b/drivers/gpu/drm/i915/display/intel_link_bw.c
@@ -26,7 +26,6 @@ void intel_link_bw_init_limits(struct intel_atomic_state *state,
struct intel_link_bw_limits *limits)
{
struct intel_display *display = to_intel_display(state);
- struct drm_i915_private *i915 = to_i915(state->base.dev);
enum pipe pipe;
limits->force_fec_pipes = 0;
@@ -34,7 +33,7 @@ void intel_link_bw_init_limits(struct intel_atomic_state *state,
for_each_pipe(display, pipe) {
const struct intel_crtc_state *crtc_state =
intel_atomic_get_new_crtc_state(state,
- intel_crtc_for_pipe(i915, pipe));
+ intel_crtc_for_pipe(display, pipe));
if (state->base.duplicated && crtc_state) {
limits->max_bpp_x16[pipe] = crtc_state->max_link_bpp_x16;
diff --git a/drivers/gpu/drm/i915/display/intel_lvds.c b/drivers/gpu/drm/i915/display/intel_lvds.c
index fb4ed9f7855b..a3459f89315f 100644
--- a/drivers/gpu/drm/i915/display/intel_lvds.c
+++ b/drivers/gpu/drm/i915/display/intel_lvds.c
@@ -899,7 +899,7 @@ void intel_lvds_init(struct drm_i915_private *i915)
drm_connector_init_with_ddc(&i915->drm, &connector->base,
&intel_lvds_connector_funcs,
DRM_MODE_CONNECTOR_LVDS,
- intel_gmbus_get_adapter(i915, ddc_pin));
+ intel_gmbus_get_adapter(display, ddc_pin));
drm_encoder_init(&i915->drm, &encoder->base, &intel_lvds_enc_funcs,
DRM_MODE_ENCODER_LVDS, "LVDS");
diff --git a/drivers/gpu/drm/i915/display/intel_modeset_setup.c b/drivers/gpu/drm/i915/display/intel_modeset_setup.c
index 72694dde3c22..fe6a90d78e78 100644
--- a/drivers/gpu/drm/i915/display/intel_modeset_setup.c
+++ b/drivers/gpu/drm/i915/display/intel_modeset_setup.c
@@ -221,6 +221,7 @@ static u8 get_transcoder_pipes(struct drm_i915_private *i915,
static void get_portsync_pipes(struct intel_crtc *crtc,
u8 *master_pipe_mask, u8 *slave_pipes_mask)
{
+ struct intel_display *display = to_intel_display(crtc);
struct drm_i915_private *i915 = to_i915(crtc->base.dev);
struct intel_crtc_state *crtc_state =
to_intel_crtc_state(crtc->base.state);
@@ -243,7 +244,7 @@ static void get_portsync_pipes(struct intel_crtc *crtc,
*master_pipe_mask = get_transcoder_pipes(i915, BIT(master_transcoder));
drm_WARN_ON(&i915->drm, !is_power_of_2(*master_pipe_mask));
- master_crtc = intel_crtc_for_pipe(i915, ffs(*master_pipe_mask) - 1);
+ master_crtc = intel_crtc_for_pipe(display, ffs(*master_pipe_mask) - 1);
master_crtc_state = to_intel_crtc_state(master_crtc->base.state);
*slave_pipes_mask = get_transcoder_pipes(i915, master_crtc_state->sync_mode_slaves_mask);
}
@@ -375,6 +376,7 @@ static void intel_crtc_copy_hw_to_uapi_state(struct intel_crtc_state *crtc_state
static void
intel_sanitize_plane_mapping(struct drm_i915_private *i915)
{
+ struct intel_display *display = &i915->display;
struct intel_crtc *crtc;
if (DISPLAY_VER(i915) >= 4)
@@ -396,7 +398,7 @@ intel_sanitize_plane_mapping(struct drm_i915_private *i915)
"[PLANE:%d:%s] attached to the wrong pipe, disabling plane\n",
plane->base.base.id, plane->base.name);
- plane_crtc = intel_crtc_for_pipe(i915, pipe);
+ plane_crtc = intel_crtc_for_pipe(display, pipe);
intel_plane_disable_noatomic(plane_crtc, plane);
}
}
@@ -662,6 +664,7 @@ static void intel_sanitize_encoder(struct intel_encoder *encoder)
/* FIXME read out full plane state for all planes */
static void readout_plane_state(struct drm_i915_private *i915)
{
+ struct intel_display *display = &i915->display;
struct intel_plane *plane;
struct intel_crtc *crtc;
@@ -674,7 +677,7 @@ static void readout_plane_state(struct drm_i915_private *i915)
visible = plane->get_hw_state(plane, &pipe);
- crtc = intel_crtc_for_pipe(i915, pipe);
+ crtc = intel_crtc_for_pipe(display, pipe);
crtc_state = to_intel_crtc_state(crtc->base.state);
intel_set_plane_visible(crtc_state, plane_state, visible);
@@ -695,6 +698,7 @@ static void readout_plane_state(struct drm_i915_private *i915)
static void intel_modeset_readout_hw_state(struct drm_i915_private *i915)
{
+ struct intel_display *display = &i915->display;
struct intel_cdclk_state *cdclk_state =
to_intel_cdclk_state(i915->display.cdclk.obj.state);
struct intel_dbuf_state *dbuf_state =
@@ -743,7 +747,7 @@ static void intel_modeset_readout_hw_state(struct drm_i915_private *i915)
pipe = 0;
if (encoder->get_hw_state(encoder, &pipe)) {
- crtc = intel_crtc_for_pipe(i915, pipe);
+ crtc = intel_crtc_for_pipe(display, pipe);
crtc_state = to_intel_crtc_state(crtc->base.state);
encoder->base.crtc = &crtc->base;
diff --git a/drivers/gpu/drm/i915/display/intel_modeset_verify.c b/drivers/gpu/drm/i915/display/intel_modeset_verify.c
index 3491db5cad31..bc70e72ccc2e 100644
--- a/drivers/gpu/drm/i915/display/intel_modeset_verify.c
+++ b/drivers/gpu/drm/i915/display/intel_modeset_verify.c
@@ -27,6 +27,7 @@ static void intel_connector_verify_state(const struct intel_crtc_state *crtc_sta
const struct drm_connector_state *conn_state)
{
struct intel_connector *connector = to_intel_connector(conn_state->connector);
+ struct intel_display *display = to_intel_display(connector);
struct drm_i915_private *i915 = to_i915(connector->base.dev);
drm_dbg_kms(&i915->drm, "[CONNECTOR:%d:%s]\n",
@@ -35,29 +36,29 @@ static void intel_connector_verify_state(const struct intel_crtc_state *crtc_sta
if (connector->get_hw_state(connector)) {
struct intel_encoder *encoder = intel_attached_encoder(connector);
- I915_STATE_WARN(i915, !crtc_state,
- "connector enabled without attached crtc\n");
+ INTEL_DISPLAY_STATE_WARN(display, !crtc_state,
+ "connector enabled without attached crtc\n");
if (!crtc_state)
return;
- I915_STATE_WARN(i915, !crtc_state->hw.active,
- "connector is active, but attached crtc isn't\n");
+ INTEL_DISPLAY_STATE_WARN(display, !crtc_state->hw.active,
+ "connector is active, but attached crtc isn't\n");
if (!encoder || encoder->type == INTEL_OUTPUT_DP_MST)
return;
- I915_STATE_WARN(i915,
- conn_state->best_encoder != &encoder->base,
- "atomic encoder doesn't match attached encoder\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ conn_state->best_encoder != &encoder->base,
+ "atomic encoder doesn't match attached encoder\n");
- I915_STATE_WARN(i915, conn_state->crtc != encoder->base.crtc,
- "attached encoder crtc differs from connector crtc\n");
+ INTEL_DISPLAY_STATE_WARN(display, conn_state->crtc != encoder->base.crtc,
+ "attached encoder crtc differs from connector crtc\n");
} else {
- I915_STATE_WARN(i915, crtc_state && crtc_state->hw.active,
- "attached crtc is active, but connector isn't\n");
- I915_STATE_WARN(i915, !crtc_state && conn_state->best_encoder,
- "best encoder set without crtc!\n");
+ INTEL_DISPLAY_STATE_WARN(display, crtc_state && crtc_state->hw.active,
+ "attached crtc is active, but connector isn't\n");
+ INTEL_DISPLAY_STATE_WARN(display, !crtc_state && conn_state->best_encoder,
+ "best encoder set without crtc!\n");
}
}
@@ -65,6 +66,7 @@ static void
verify_connector_state(struct intel_atomic_state *state,
struct intel_crtc *crtc)
{
+ struct intel_display *display = to_intel_display(state);
struct drm_connector *connector;
const struct drm_connector_state *new_conn_state;
int i;
@@ -81,8 +83,8 @@ verify_connector_state(struct intel_atomic_state *state,
intel_connector_verify_state(crtc_state, new_conn_state);
- I915_STATE_WARN(to_i915(connector->dev), new_conn_state->best_encoder != encoder,
- "connector's atomic encoder doesn't match legacy encoder\n");
+ INTEL_DISPLAY_STATE_WARN(display, new_conn_state->best_encoder != encoder,
+ "connector's atomic encoder doesn't match legacy encoder\n");
}
}
@@ -109,6 +111,7 @@ static void intel_pipe_config_sanity_check(const struct intel_crtc_state *crtc_s
static void
verify_encoder_state(struct intel_atomic_state *state)
{
+ struct intel_display *display = to_intel_display(state);
struct drm_i915_private *i915 = to_i915(state->base.dev);
struct intel_encoder *encoder;
struct drm_connector *connector;
@@ -134,25 +137,25 @@ verify_encoder_state(struct intel_atomic_state *state)
found = true;
enabled = true;
- I915_STATE_WARN(i915,
- new_conn_state->crtc != encoder->base.crtc,
- "connector's crtc doesn't match encoder crtc\n");
+ INTEL_DISPLAY_STATE_WARN(display,
+ new_conn_state->crtc != encoder->base.crtc,
+ "connector's crtc doesn't match encoder crtc\n");
}
if (!found)
continue;
- I915_STATE_WARN(i915, !!encoder->base.crtc != enabled,
- "encoder's enabled state mismatch (expected %i, found %i)\n",
- !!encoder->base.crtc, enabled);
+ INTEL_DISPLAY_STATE_WARN(display, !!encoder->base.crtc != enabled,
+ "encoder's enabled state mismatch (expected %i, found %i)\n",
+ !!encoder->base.crtc, enabled);
if (!encoder->base.crtc) {
bool active;
active = encoder->get_hw_state(encoder, &pipe);
- I915_STATE_WARN(i915, active,
- "encoder detached but still enabled on pipe %c.\n",
- pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display, active,
+ "encoder detached but still enabled on pipe %c.\n",
+ pipe_name(pipe));
}
}
}
@@ -161,8 +164,8 @@ static void
verify_crtc_state(struct intel_atomic_state *state,
struct intel_crtc *crtc)
{
- struct drm_device *dev = crtc->base.dev;
- struct drm_i915_private *i915 = to_i915(dev);
+ struct intel_display *display = to_intel_display(state);
+ struct drm_i915_private *i915 = to_i915(display->drm);
const struct intel_crtc_state *sw_crtc_state =
intel_atomic_get_new_crtc_state(state, crtc);
struct intel_crtc_state *hw_crtc_state;
@@ -173,7 +176,7 @@ verify_crtc_state(struct intel_atomic_state *state,
if (!hw_crtc_state)
return;
- drm_dbg_kms(&i915->drm, "[CRTC:%d:%s]\n", crtc->base.base.id,
+ drm_dbg_kms(display->drm, "[CRTC:%d:%s]\n", crtc->base.base.id,
crtc->base.name);
hw_crtc_state->hw.enable = sw_crtc_state->hw.enable;
@@ -184,30 +187,30 @@ verify_crtc_state(struct intel_atomic_state *state,
if (IS_I830(i915) && hw_crtc_state->hw.active)
hw_crtc_state->hw.active = sw_crtc_state->hw.active;
- I915_STATE_WARN(i915,
- sw_crtc_state->hw.active != hw_crtc_state->hw.active,
- "crtc active state doesn't match with hw state (expected %i, found %i)\n",
- sw_crtc_state->hw.active, hw_crtc_state->hw.active);
+ INTEL_DISPLAY_STATE_WARN(display,
+ sw_crtc_state->hw.active != hw_crtc_state->hw.active,
+ "crtc active state doesn't match with hw state (expected %i, found %i)\n",
+ sw_crtc_state->hw.active, hw_crtc_state->hw.active);
- I915_STATE_WARN(i915, crtc->active != sw_crtc_state->hw.active,
- "transitional active state does not match atomic hw state (expected %i, found %i)\n",
- sw_crtc_state->hw.active, crtc->active);
+ INTEL_DISPLAY_STATE_WARN(display, crtc->active != sw_crtc_state->hw.active,
+ "transitional active state does not match atomic hw state (expected %i, found %i)\n",
+ sw_crtc_state->hw.active, crtc->active);
primary_crtc = intel_primary_crtc(sw_crtc_state);
- for_each_encoder_on_crtc(dev, &primary_crtc->base, encoder) {
+ for_each_encoder_on_crtc(display->drm, &primary_crtc->base, encoder) {
enum pipe pipe;
bool active;
active = encoder->get_hw_state(encoder, &pipe);
- I915_STATE_WARN(i915, active != sw_crtc_state->hw.active,
- "[ENCODER:%i] active %i with crtc active %i\n",
- encoder->base.base.id, active,
- sw_crtc_state->hw.active);
+ INTEL_DISPLAY_STATE_WARN(display, active != sw_crtc_state->hw.active,
+ "[ENCODER:%i] active %i with crtc active %i\n",
+ encoder->base.base.id, active,
+ sw_crtc_state->hw.active);
- I915_STATE_WARN(i915, active && primary_crtc->pipe != pipe,
- "Encoder connected to wrong pipe %c\n",
- pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display, active && primary_crtc->pipe != pipe,
+ "Encoder connected to wrong pipe %c\n",
+ pipe_name(pipe));
if (active)
intel_encoder_get_config(encoder, hw_crtc_state);
@@ -220,7 +223,7 @@ verify_crtc_state(struct intel_atomic_state *state,
if (!intel_pipe_config_compare(sw_crtc_state,
hw_crtc_state, false)) {
- I915_STATE_WARN(i915, 1, "pipe state doesn't match!\n");
+ INTEL_DISPLAY_STATE_WARN(display, 1, "pipe state doesn't match!\n");
intel_crtc_state_dump(hw_crtc_state, NULL, "hw state");
intel_crtc_state_dump(sw_crtc_state, NULL, "sw state");
}
diff --git a/drivers/gpu/drm/i915/display/intel_pch_display.c b/drivers/gpu/drm/i915/display/intel_pch_display.c
index f13ab680c2cf..4210de87a0a2 100644
--- a/drivers/gpu/drm/i915/display/intel_pch_display.c
+++ b/drivers/gpu/drm/i915/display/intel_pch_display.c
@@ -39,58 +39,61 @@ static void assert_pch_dp_disabled(struct drm_i915_private *dev_priv,
enum pipe pipe, enum port port,
i915_reg_t dp_reg)
{
+ struct intel_display *display = &dev_priv->display;
enum pipe port_pipe;
bool state;
state = g4x_dp_port_enabled(dev_priv, dp_reg, port, &port_pipe);
- I915_STATE_WARN(dev_priv, state && port_pipe == pipe,
- "PCH DP %c enabled on transcoder %c, should be disabled\n",
- port_name(port), pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display, state && port_pipe == pipe,
+ "PCH DP %c enabled on transcoder %c, should be disabled\n",
+ port_name(port), pipe_name(pipe));
- I915_STATE_WARN(dev_priv,
- HAS_PCH_IBX(dev_priv) && !state && port_pipe == PIPE_B,
- "IBX PCH DP %c still using transcoder B\n",
- port_name(port));
+ INTEL_DISPLAY_STATE_WARN(display,
+ HAS_PCH_IBX(dev_priv) && !state && port_pipe == PIPE_B,
+ "IBX PCH DP %c still using transcoder B\n",
+ port_name(port));
}
static void assert_pch_hdmi_disabled(struct drm_i915_private *dev_priv,
enum pipe pipe, enum port port,
i915_reg_t hdmi_reg)
{
+ struct intel_display *display = &dev_priv->display;
enum pipe port_pipe;
bool state;
state = intel_sdvo_port_enabled(dev_priv, hdmi_reg, &port_pipe);
- I915_STATE_WARN(dev_priv, state && port_pipe == pipe,
- "PCH HDMI %c enabled on transcoder %c, should be disabled\n",
- port_name(port), pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display, state && port_pipe == pipe,
+ "PCH HDMI %c enabled on transcoder %c, should be disabled\n",
+ port_name(port), pipe_name(pipe));
- I915_STATE_WARN(dev_priv,
- HAS_PCH_IBX(dev_priv) && !state && port_pipe == PIPE_B,
- "IBX PCH HDMI %c still using transcoder B\n",
- port_name(port));
+ INTEL_DISPLAY_STATE_WARN(display,
+ HAS_PCH_IBX(dev_priv) && !state && port_pipe == PIPE_B,
+ "IBX PCH HDMI %c still using transcoder B\n",
+ port_name(port));
}
static void assert_pch_ports_disabled(struct drm_i915_private *dev_priv,
enum pipe pipe)
{
+ struct intel_display *display = &dev_priv->display;
enum pipe port_pipe;
assert_pch_dp_disabled(dev_priv, pipe, PORT_B, PCH_DP_B);
assert_pch_dp_disabled(dev_priv, pipe, PORT_C, PCH_DP_C);
assert_pch_dp_disabled(dev_priv, pipe, PORT_D, PCH_DP_D);
- I915_STATE_WARN(dev_priv,
- intel_crt_port_enabled(dev_priv, PCH_ADPA, &port_pipe) && port_pipe == pipe,
- "PCH VGA enabled on transcoder %c, should be disabled\n",
- pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_crt_port_enabled(display, PCH_ADPA, &port_pipe) && port_pipe == pipe,
+ "PCH VGA enabled on transcoder %c, should be disabled\n",
+ pipe_name(pipe));
- I915_STATE_WARN(dev_priv,
- intel_lvds_port_enabled(dev_priv, PCH_LVDS, &port_pipe) && port_pipe == pipe,
- "PCH LVDS enabled on transcoder %c, should be disabled\n",
- pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display,
+ intel_lvds_port_enabled(dev_priv, PCH_LVDS, &port_pipe) && port_pipe == pipe,
+ "PCH LVDS enabled on transcoder %c, should be disabled\n",
+ pipe_name(pipe));
/* PCH SDVOB multiplex with HDMIB */
assert_pch_hdmi_disabled(dev_priv, pipe, PORT_B, PCH_HDMIB);
@@ -101,14 +104,15 @@ static void assert_pch_ports_disabled(struct drm_i915_private *dev_priv,
static void assert_pch_transcoder_disabled(struct drm_i915_private *dev_priv,
enum pipe pipe)
{
+ struct intel_display *display = &dev_priv->display;
u32 val;
bool enabled;
- val = intel_de_read(dev_priv, PCH_TRANSCONF(pipe));
+ val = intel_de_read(display, PCH_TRANSCONF(pipe));
enabled = !!(val & TRANS_ENABLE);
- I915_STATE_WARN(dev_priv, enabled,
- "transcoder assertion failed, should be off on pipe %c but is still active\n",
- pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display, enabled,
+ "transcoder assertion failed, should be off on pipe %c but is still active\n",
+ pipe_name(pipe));
}
static void ibx_sanitize_pch_hdmi_port(struct drm_i915_private *dev_priv,
diff --git a/drivers/gpu/drm/i915/display/intel_pps.c b/drivers/gpu/drm/i915/display/intel_pps.c
index feddc30e3375..22616a6f7509 100644
--- a/drivers/gpu/drm/i915/display/intel_pps.c
+++ b/drivers/gpu/drm/i915/display/intel_pps.c
@@ -800,7 +800,6 @@ bool intel_pps_vdd_on_unlocked(struct intel_dp *intel_dp)
void intel_pps_vdd_on(struct intel_dp *intel_dp)
{
struct intel_display *display = to_intel_display(intel_dp);
- struct drm_i915_private *i915 = to_i915(display->drm);
intel_wakeref_t wakeref;
bool vdd;
@@ -810,10 +809,10 @@ void intel_pps_vdd_on(struct intel_dp *intel_dp)
vdd = false;
with_intel_pps_lock(intel_dp, wakeref)
vdd = intel_pps_vdd_on_unlocked(intel_dp);
- I915_STATE_WARN(i915, !vdd, "[ENCODER:%d:%s] %s VDD already requested on\n",
- dp_to_dig_port(intel_dp)->base.base.base.id,
- dp_to_dig_port(intel_dp)->base.base.name,
- pps_name(intel_dp));
+ INTEL_DISPLAY_STATE_WARN(display, !vdd, "[ENCODER:%d:%s] %s VDD already requested on\n",
+ dp_to_dig_port(intel_dp)->base.base.base.id,
+ dp_to_dig_port(intel_dp)->base.base.name,
+ pps_name(intel_dp));
}
static void intel_pps_vdd_off_sync_unlocked(struct intel_dp *intel_dp)
@@ -920,18 +919,17 @@ static void edp_panel_vdd_schedule_off(struct intel_dp *intel_dp)
void intel_pps_vdd_off_unlocked(struct intel_dp *intel_dp, bool sync)
{
struct intel_display *display = to_intel_display(intel_dp);
- struct drm_i915_private *dev_priv = to_i915(display->drm);
lockdep_assert_held(&display->pps.mutex);
if (!intel_dp_is_edp(intel_dp))
return;
- I915_STATE_WARN(dev_priv, !intel_dp->pps.want_panel_vdd,
- "[ENCODER:%d:%s] %s VDD not forced on",
- dp_to_dig_port(intel_dp)->base.base.base.id,
- dp_to_dig_port(intel_dp)->base.base.name,
- pps_name(intel_dp));
+ INTEL_DISPLAY_STATE_WARN(display, !intel_dp->pps.want_panel_vdd,
+ "[ENCODER:%d:%s] %s VDD not forced on",
+ dp_to_dig_port(intel_dp)->base.base.base.id,
+ dp_to_dig_port(intel_dp)->base.base.name,
+ pps_name(intel_dp));
intel_dp->pps.want_panel_vdd = false;
@@ -1806,7 +1804,7 @@ void assert_pps_unlocked(struct intel_display *display, enum pipe pipe)
((val & PANEL_UNLOCK_MASK) == PANEL_UNLOCK_REGS))
locked = false;
- I915_STATE_WARN(dev_priv, panel_pipe == pipe && locked,
- "panel assertion failure, pipe %c regs locked\n",
- pipe_name(pipe));
+ INTEL_DISPLAY_STATE_WARN(display, panel_pipe == pipe && locked,
+ "panel assertion failure, pipe %c regs locked\n",
+ pipe_name(pipe));
}
diff --git a/drivers/gpu/drm/i915/display/intel_sdvo.c b/drivers/gpu/drm/i915/display/intel_sdvo.c
index 7cc519b402e9..3af944771471 100644
--- a/drivers/gpu/drm/i915/display/intel_sdvo.c
+++ b/drivers/gpu/drm/i915/display/intel_sdvo.c
@@ -2081,10 +2081,10 @@ intel_sdvo_get_edid(struct drm_connector *connector)
static const struct drm_edid *
intel_sdvo_get_analog_edid(struct drm_connector *connector)
{
- struct drm_i915_private *i915 = to_i915(connector->dev);
+ struct intel_display *display = to_intel_display(connector->dev);
struct i2c_adapter *ddc;
- ddc = intel_gmbus_get_adapter(i915, i915->display.vbt.crt_ddc_pin);
+ ddc = intel_gmbus_get_adapter(display, display->vbt.crt_ddc_pin);
if (!ddc)
return NULL;
@@ -2637,6 +2637,7 @@ intel_sdvo_select_ddc_bus(struct intel_sdvo *sdvo,
static void
intel_sdvo_select_i2c_bus(struct intel_sdvo *sdvo)
{
+ struct intel_display *display = to_intel_display(&sdvo->base);
struct drm_i915_private *dev_priv = to_i915(sdvo->base.base.dev);
const struct sdvo_device_mapping *mapping;
u8 pin;
@@ -2647,7 +2648,7 @@ intel_sdvo_select_i2c_bus(struct intel_sdvo *sdvo)
mapping = &dev_priv->display.vbt.sdvo_mappings[1];
if (mapping->initialized &&
- intel_gmbus_is_valid_pin(dev_priv, mapping->i2c_pin))
+ intel_gmbus_is_valid_pin(display, mapping->i2c_pin))
pin = mapping->i2c_pin;
else
pin = GMBUS_PIN_DPB;
@@ -2656,7 +2657,7 @@ intel_sdvo_select_i2c_bus(struct intel_sdvo *sdvo)
sdvo->base.base.base.id, sdvo->base.base.name,
pin, sdvo->target_addr);
- sdvo->i2c = intel_gmbus_get_adapter(dev_priv, pin);
+ sdvo->i2c = intel_gmbus_get_adapter(display, pin);
/*
* With gmbus we should be able to drive sdvo i2c at 2MHz, but somehow
diff --git a/drivers/gpu/drm/i915/display/intel_snps_phy.c b/drivers/gpu/drm/i915/display/intel_snps_phy.c
index e6df1f92def5..4b3a32736fd6 100644
--- a/drivers/gpu/drm/i915/display/intel_snps_phy.c
+++ b/drivers/gpu/drm/i915/display/intel_snps_phy.c
@@ -1997,6 +1997,7 @@ int intel_snps_phy_check_hdmi_link_rate(int clock)
void intel_mpllb_state_verify(struct intel_atomic_state *state,
struct intel_crtc *crtc)
{
+ struct intel_display *display = to_intel_display(state);
struct drm_i915_private *i915 = to_i915(state->base.dev);
const struct intel_crtc_state *new_crtc_state =
intel_atomic_get_new_crtc_state(state, crtc);
@@ -2019,11 +2020,11 @@ void intel_mpllb_state_verify(struct intel_atomic_state *state,
intel_mpllb_readout_hw_state(encoder, &mpllb_hw_state);
#define MPLLB_CHECK(__name) \
- I915_STATE_WARN(i915, mpllb_sw_state->__name != mpllb_hw_state.__name, \
- "[CRTC:%d:%s] mismatch in MPLLB: %s (expected 0x%08x, found 0x%08x)", \
- crtc->base.base.id, crtc->base.name, \
- __stringify(__name), \
- mpllb_sw_state->__name, mpllb_hw_state.__name)
+ INTEL_DISPLAY_STATE_WARN(display, mpllb_sw_state->__name != mpllb_hw_state.__name, \
+ "[CRTC:%d:%s] mismatch in MPLLB: %s (expected 0x%08x, found 0x%08x)", \
+ crtc->base.base.id, crtc->base.name, \
+ __stringify(__name), \
+ mpllb_sw_state->__name, mpllb_hw_state.__name)
MPLLB_CHECK(mpllb_cp);
MPLLB_CHECK(mpllb_div);
diff --git a/drivers/gpu/drm/i915/display/intel_sprite_uapi.c b/drivers/gpu/drm/i915/display/intel_sprite_uapi.c
index 4853c4806004..1d0b84b464c1 100644
--- a/drivers/gpu/drm/i915/display/intel_sprite_uapi.c
+++ b/drivers/gpu/drm/i915/display/intel_sprite_uapi.c
@@ -42,6 +42,7 @@ static void intel_plane_set_ckey(struct intel_plane_state *plane_state,
int intel_sprite_set_colorkey_ioctl(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
+ struct intel_display *display = to_intel_display(dev);
struct drm_i915_private *dev_priv = to_i915(dev);
struct drm_intel_sprite_colorkey *set = data;
struct drm_plane *plane;
@@ -100,7 +101,7 @@ int intel_sprite_set_colorkey_ioctl(struct drm_device *dev, void *data,
*/
if (!ret && has_dst_key_in_primary_plane(dev_priv)) {
struct intel_crtc *crtc =
- intel_crtc_for_pipe(dev_priv,
+ intel_crtc_for_pipe(display,
to_intel_plane(plane)->pipe);
plane_state = drm_atomic_get_plane_state(state,
diff --git a/drivers/gpu/drm/i915/display/intel_vga.c b/drivers/gpu/drm/i915/display/intel_vga.c
index 0b5916c15307..2c76a0176a35 100644
--- a/drivers/gpu/drm/i915/display/intel_vga.c
+++ b/drivers/gpu/drm/i915/display/intel_vga.c
@@ -14,24 +14,26 @@
#include "intel_de.h"
#include "intel_vga.h"
-static i915_reg_t intel_vga_cntrl_reg(struct drm_i915_private *i915)
+static i915_reg_t intel_vga_cntrl_reg(struct intel_display *display)
{
+ struct drm_i915_private *i915 = to_i915(display->drm);
+
if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915))
return VLV_VGACNTRL;
- else if (DISPLAY_VER(i915) >= 5)
+ else if (DISPLAY_VER(display) >= 5)
return CPU_VGACNTRL;
else
return VGACNTRL;
}
/* Disable the VGA plane that we never use */
-void intel_vga_disable(struct drm_i915_private *dev_priv)
+void intel_vga_disable(struct intel_display *display)
{
- struct pci_dev *pdev = to_pci_dev(dev_priv->drm.dev);
- i915_reg_t vga_reg = intel_vga_cntrl_reg(dev_priv);
+ struct pci_dev *pdev = to_pci_dev(display->drm->dev);
+ i915_reg_t vga_reg = intel_vga_cntrl_reg(display);
u8 sr1;
- if (intel_de_read(dev_priv, vga_reg) & VGA_DISP_DISABLE)
+ if (intel_de_read(display, vga_reg) & VGA_DISP_DISABLE)
return;
/* WaEnableVGAAccessThroughIOPort:ctg,elk,ilk,snb,ivb,vlv,hsw */
@@ -42,23 +44,24 @@ void intel_vga_disable(struct drm_i915_private *dev_priv)
vga_put(pdev, VGA_RSRC_LEGACY_IO);
udelay(300);
- intel_de_write(dev_priv, vga_reg, VGA_DISP_DISABLE);
- intel_de_posting_read(dev_priv, vga_reg);
+ intel_de_write(display, vga_reg, VGA_DISP_DISABLE);
+ intel_de_posting_read(display, vga_reg);
}
-void intel_vga_redisable_power_on(struct drm_i915_private *dev_priv)
+void intel_vga_redisable_power_on(struct intel_display *display)
{
- i915_reg_t vga_reg = intel_vga_cntrl_reg(dev_priv);
+ i915_reg_t vga_reg = intel_vga_cntrl_reg(display);
- if (!(intel_de_read(dev_priv, vga_reg) & VGA_DISP_DISABLE)) {
- drm_dbg_kms(&dev_priv->drm,
+ if (!(intel_de_read(display, vga_reg) & VGA_DISP_DISABLE)) {
+ drm_dbg_kms(display->drm,
"Something enabled VGA plane, disabling it\n");
- intel_vga_disable(dev_priv);
+ intel_vga_disable(display);
}
}
-void intel_vga_redisable(struct drm_i915_private *i915)
+void intel_vga_redisable(struct intel_display *display)
{
+ struct drm_i915_private *i915 = to_i915(display->drm);
intel_wakeref_t wakeref;
/*
@@ -74,14 +77,14 @@ void intel_vga_redisable(struct drm_i915_private *i915)
if (!wakeref)
return;
- intel_vga_redisable_power_on(i915);
+ intel_vga_redisable_power_on(display);
intel_display_power_put(i915, POWER_DOMAIN_VGA, wakeref);
}
-void intel_vga_reset_io_mem(struct drm_i915_private *i915)
+void intel_vga_reset_io_mem(struct intel_display *display)
{
- struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
+ struct pci_dev *pdev = to_pci_dev(display->drm->dev);
/*
* After we re-enable the power well, if we touch VGA register 0x3d5
@@ -98,10 +101,10 @@ void intel_vga_reset_io_mem(struct drm_i915_private *i915)
vga_put(pdev, VGA_RSRC_LEGACY_IO);
}
-int intel_vga_register(struct drm_i915_private *i915)
+int intel_vga_register(struct intel_display *display)
{
- struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
+ struct pci_dev *pdev = to_pci_dev(display->drm->dev);
int ret;
/*
@@ -119,9 +122,9 @@ int intel_vga_register(struct drm_i915_private *i915)
return 0;
}
-void intel_vga_unregister(struct drm_i915_private *i915)
+void intel_vga_unregister(struct intel_display *display)
{
- struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
+ struct pci_dev *pdev = to_pci_dev(display->drm->dev);
vga_client_unregister(pdev);
}
diff --git a/drivers/gpu/drm/i915/display/intel_vga.h b/drivers/gpu/drm/i915/display/intel_vga.h
index ba5b55b917f0..824dfc32a199 100644
--- a/drivers/gpu/drm/i915/display/intel_vga.h
+++ b/drivers/gpu/drm/i915/display/intel_vga.h
@@ -6,13 +6,13 @@
#ifndef __INTEL_VGA_H__
#define __INTEL_VGA_H__
-struct drm_i915_private;
+struct intel_display;
-void intel_vga_reset_io_mem(struct drm_i915_private *i915);
-void intel_vga_disable(struct drm_i915_private *i915);
-void intel_vga_redisable(struct drm_i915_private *i915);
-void intel_vga_redisable_power_on(struct drm_i915_private *i915);
-int intel_vga_register(struct drm_i915_private *i915);
-void intel_vga_unregister(struct drm_i915_private *i915);
+void intel_vga_reset_io_mem(struct intel_display *display);
+void intel_vga_disable(struct intel_display *display);
+void intel_vga_redisable(struct intel_display *display);
+void intel_vga_redisable_power_on(struct intel_display *display);
+int intel_vga_register(struct intel_display *display);
+void intel_vga_unregister(struct intel_display *display);
#endif /* __INTEL_VGA_H__ */
diff --git a/drivers/gpu/drm/i915/display/skl_watermark.c b/drivers/gpu/drm/i915/display/skl_watermark.c
index 0c868be53f2f..34068a4d915b 100644
--- a/drivers/gpu/drm/i915/display/skl_watermark.c
+++ b/drivers/gpu/drm/i915/display/skl_watermark.c
@@ -3388,6 +3388,7 @@ static void intel_dbuf_mdclk_min_tracker_update(struct intel_atomic_state *state
static enum pipe intel_mbus_joined_pipe(struct intel_atomic_state *state,
const struct intel_dbuf_state *dbuf_state)
{
+ struct intel_display *display = to_intel_display(state);
struct drm_i915_private *i915 = to_i915(state->base.dev);
enum pipe pipe = ffs(dbuf_state->active_pipes) - 1;
const struct intel_crtc_state *new_crtc_state;
@@ -3396,7 +3397,7 @@ static enum pipe intel_mbus_joined_pipe(struct intel_atomic_state *state,
drm_WARN_ON(&i915->drm, !dbuf_state->joined_mbus);
drm_WARN_ON(&i915->drm, !is_power_of_2(dbuf_state->active_pipes));
- crtc = intel_crtc_for_pipe(i915, pipe);
+ crtc = intel_crtc_for_pipe(display, pipe);
new_crtc_state = intel_atomic_get_new_crtc_state(state, crtc);
if (new_crtc_state && !intel_crtc_needs_modeset(new_crtc_state))
@@ -3458,7 +3459,7 @@ void intel_dbuf_mbus_pre_ddb_update(struct intel_atomic_state *state)
void intel_dbuf_mbus_post_ddb_update(struct intel_atomic_state *state)
{
- struct drm_i915_private *i915 = to_i915(state->base.dev);
+ struct intel_display *display = to_intel_display(state);
const struct intel_dbuf_state *new_dbuf_state =
intel_atomic_get_new_dbuf_state(state);
const struct intel_dbuf_state *old_dbuf_state =
@@ -3477,7 +3478,7 @@ void intel_dbuf_mbus_post_ddb_update(struct intel_atomic_state *state)
intel_dbuf_mbus_join_update(state, pipe);
if (pipe != INVALID_PIPE) {
- struct intel_crtc *crtc = intel_crtc_for_pipe(i915, pipe);
+ struct intel_crtc *crtc = intel_crtc_for_pipe(display, pipe);
intel_crtc_wait_for_next_vblank(crtc);
}
diff --git a/drivers/gpu/drm/i915/display/vlv_dsi_pll.c b/drivers/gpu/drm/i915/display/vlv_dsi_pll.c
index 70c5a13a3c75..59a50647f2c3 100644
--- a/drivers/gpu/drm/i915/display/vlv_dsi_pll.c
+++ b/drivers/gpu/drm/i915/display/vlv_dsi_pll.c
@@ -592,15 +592,16 @@ void bxt_dsi_reset_clocks(struct intel_encoder *encoder, enum port port)
static void assert_dsi_pll(struct drm_i915_private *i915, bool state)
{
+ struct intel_display *display = &i915->display;
bool cur_state;
vlv_cck_get(i915);
cur_state = vlv_cck_read(i915, CCK_REG_DSI_PLL_CONTROL) & DSI_PLL_VCO_EN;
vlv_cck_put(i915);
- I915_STATE_WARN(i915, cur_state != state,
- "DSI PLL state assertion failure (expected %s, current %s)\n",
- str_on_off(state), str_on_off(cur_state));
+ INTEL_DISPLAY_STATE_WARN(display, cur_state != state,
+ "DSI PLL state assertion failure (expected %s, current %s)\n",
+ str_on_off(state), str_on_off(cur_state));
}
void assert_dsi_pll_enabled(struct drm_i915_private *i915)
diff --git a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
index 72090f52fb85..c8501f823991 100644
--- a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
+++ b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
@@ -3935,11 +3935,11 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
struct drm_i915_private *i915 = siblings[0]->i915;
struct virtual_engine *ve;
unsigned int n;
- int err;
+ int err = -ENOMEM;
ve = kzalloc(struct_size(ve, siblings, count), GFP_KERNEL);
if (!ve)
- return ERR_PTR(-ENOMEM);
+ goto err;
ve->base.i915 = i915;
ve->base.gt = siblings[0]->gt;
@@ -3971,10 +3971,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
intel_engine_init_execlists(&ve->base);
ve->base.sched_engine = i915_sched_engine_create(ENGINE_VIRTUAL);
- if (!ve->base.sched_engine) {
- err = -ENOMEM;
- goto err_put;
- }
+ if (!ve->base.sched_engine)
+ goto err_noput;
ve->base.sched_engine->private_data = &ve->base;
ve->base.cops = &virtual_context_ops;
@@ -3990,10 +3988,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
intel_context_init(&ve->context, &ve->base);
ve->base.breadcrumbs = intel_breadcrumbs_create(NULL);
- if (!ve->base.breadcrumbs) {
- err = -ENOMEM;
+ if (!ve->base.breadcrumbs)
goto err_put;
- }
for (n = 0; n < count; n++) {
struct intel_engine_cs *sibling = siblings[n];
@@ -4068,8 +4064,13 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
virtual_engine_initial_hint(ve);
return &ve->context;
+err_noput:
+ kfree(ve);
+ goto err;
+
err_put:
intel_context_put(&ve->context);
+err:
return ERR_PTR(err);
}
diff --git a/drivers/gpu/drm/i915/gt/intel_rc6.c b/drivers/gpu/drm/i915/gt/intel_rc6.c
index 9ca42589da4d..bf38cc5fe872 100644
--- a/drivers/gpu/drm/i915/gt/intel_rc6.c
+++ b/drivers/gpu/drm/i915/gt/intel_rc6.c
@@ -341,7 +341,7 @@ static int vlv_rc6_init(struct intel_rc6 *rc6)
return PTR_ERR(pctx);
}
- GEM_BUG_ON(range_overflows_end_t(u64,
+ GEM_BUG_ON(range_end_overflows_t(u64,
i915->dsm.stolen.start,
pctx->stolen->start,
U32_MAX));
diff --git a/drivers/gpu/drm/i915/i915_suspend.c b/drivers/gpu/drm/i915/i915_suspend.c
index f8373a461f17..f18f1acf2158 100644
--- a/drivers/gpu/drm/i915/i915_suspend.c
+++ b/drivers/gpu/drm/i915/i915_suspend.c
@@ -118,6 +118,7 @@ void i915_save_display(struct drm_i915_private *dev_priv)
void i915_restore_display(struct drm_i915_private *dev_priv)
{
+ struct intel_display *display = &dev_priv->display;
struct pci_dev *pdev = to_pci_dev(dev_priv->drm.dev);
if (!HAS_DISPLAY(dev_priv))
@@ -134,7 +135,7 @@ void i915_restore_display(struct drm_i915_private *dev_priv)
intel_de_write(dev_priv, DSPARB(dev_priv),
dev_priv->regfile.saveDSPARB);
- intel_vga_redisable(dev_priv);
+ intel_vga_redisable(display);
- intel_gmbus_reset(dev_priv);
+ intel_gmbus_reset(display);
}
diff --git a/drivers/gpu/drm/i915/i915_utils.h b/drivers/gpu/drm/i915/i915_utils.h
index 71bdc89bd621..e9b8ec51f6ab 100644
--- a/drivers/gpu/drm/i915/i915_utils.h
+++ b/drivers/gpu/drm/i915/i915_utils.h
@@ -69,30 +69,6 @@ bool i915_error_injected(void);
drm_err(&(i915)->drm, fmt, ##__VA_ARGS__); \
})
-#define range_overflows(start, size, max) ({ \
- typeof(start) start__ = (start); \
- typeof(size) size__ = (size); \
- typeof(max) max__ = (max); \
- (void)(&start__ == &size__); \
- (void)(&start__ == &max__); \
- start__ >= max__ || size__ > max__ - start__; \
-})
-
-#define range_overflows_t(type, start, size, max) \
- range_overflows((type)(start), (type)(size), (type)(max))
-
-#define range_overflows_end(start, size, max) ({ \
- typeof(start) start__ = (start); \
- typeof(size) size__ = (size); \
- typeof(max) max__ = (max); \
- (void)(&start__ == &size__); \
- (void)(&start__ == &max__); \
- start__ > max__ || size__ > max__ - start__; \
-})
-
-#define range_overflows_end_t(type, start, size, max) \
- range_overflows_end((type)(start), (type)(size), (type)(max))
-
#define ptr_mask_bits(ptr, n) ({ \
unsigned long __v = (unsigned long)(ptr); \
(typeof(ptr))(__v & -BIT(n)); \
diff --git a/drivers/gpu/drm/imagination/pvr_queue.c b/drivers/gpu/drm/imagination/pvr_queue.c
index 367d5915130a..b660b85b58fd 100644
--- a/drivers/gpu/drm/imagination/pvr_queue.c
+++ b/drivers/gpu/drm/imagination/pvr_queue.c
@@ -1240,6 +1240,17 @@ struct pvr_queue *pvr_queue_create(struct pvr_context *ctx,
},
};
struct pvr_device *pvr_dev = ctx->pvr_dev;
+ const struct drm_sched_init_args sched_args = {
+ .ops = &pvr_queue_sched_ops,
+ .submit_wq = pvr_dev->sched_wq,
+ .num_rqs = 1,
+ .credit_limit = 64 * 1024,
+ .hang_limit = 1,
+ .timeout = msecs_to_jiffies(500),
+ .timeout_wq = pvr_dev->sched_wq,
+ .name = "pvr-queue",
+ .dev = pvr_dev->base.dev,
+ };
struct drm_gpu_scheduler *sched;
struct pvr_queue *queue;
int ctx_state_size, err;
@@ -1312,12 +1323,7 @@ struct pvr_queue *pvr_queue_create(struct pvr_context *ctx,
queue->timeline_ufo.value = cpu_map;
- err = drm_sched_init(&queue->scheduler,
- &pvr_queue_sched_ops,
- pvr_dev->sched_wq, 1, 64 * 1024, 1,
- msecs_to_jiffies(500),
- pvr_dev->sched_wq, NULL, "pvr-queue",
- pvr_dev->base.dev);
+ err = drm_sched_init(&queue->scheduler, &sched_args);
if (err)
goto err_release_ufo;
diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c
index 00f0b2907cb6..9b21eada6c49 100644
--- a/drivers/gpu/drm/lima/lima_sched.c
+++ b/drivers/gpu/drm/lima/lima_sched.c
@@ -517,18 +517,22 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name)
{
unsigned int timeout = lima_sched_timeout_ms > 0 ?
lima_sched_timeout_ms : 10000;
+ const struct drm_sched_init_args args = {
+ .ops = &lima_sched_ops,
+ .num_rqs = DRM_SCHED_PRIORITY_COUNT,
+ .credit_limit = 1,
+ .hang_limit = lima_job_hang_limit,
+ .timeout = msecs_to_jiffies(timeout),
+ .name = name,
+ .dev = pipe->ldev->dev,
+ };
pipe->fence_context = dma_fence_context_alloc(1);
spin_lock_init(&pipe->fence_lock);
INIT_WORK(&pipe->recover_work, lima_sched_recover_work);
- return drm_sched_init(&pipe->base, &lima_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- 1,
- lima_job_hang_limit,
- msecs_to_jiffies(timeout), NULL,
- NULL, name, pipe->ldev->dev);
+ return drm_sched_init(&pipe->base, &args);
}
void lima_sched_pipe_fini(struct lima_sched_pipe *pipe)
diff --git a/drivers/gpu/drm/mediatek/mtk_dsi.c b/drivers/gpu/drm/mediatek/mtk_dsi.c
index 96dcfbf6ba5a..57e8abc5c5f6 100644
--- a/drivers/gpu/drm/mediatek/mtk_dsi.c
+++ b/drivers/gpu/drm/mediatek/mtk_dsi.c
@@ -1192,6 +1192,7 @@ static int mtk_dsi_probe(struct platform_device *pdev)
if (!dsi)
return -ENOMEM;
+ dsi->bridge.funcs = &mtk_dsi_bridge_funcs;
dsi->driver_data = of_device_get_match_data(dev);
dsi->engine_clk = devm_clk_get(dev, "engine");
@@ -1224,6 +1225,11 @@ static int mtk_dsi_probe(struct platform_device *pdev)
dsi->host.ops = &mtk_dsi_ops;
dsi->host.dev = dev;
+
+ init_waitqueue_head(&dsi->irq_wait_queue);
+
+ platform_set_drvdata(pdev, dsi);
+
ret = mipi_dsi_host_register(&dsi->host);
if (ret < 0)
return dev_err_probe(dev, ret, "Failed to register DSI host\n");
@@ -1235,11 +1241,6 @@ static int mtk_dsi_probe(struct platform_device *pdev)
return dev_err_probe(&pdev->dev, ret, "Failed to request DSI irq\n");
}
- init_waitqueue_head(&dsi->irq_wait_queue);
-
- platform_set_drvdata(pdev, dsi);
-
- dsi->bridge.funcs = &mtk_dsi_bridge_funcs;
dsi->bridge.of_node = dev->of_node;
dsi->bridge.type = DRM_MODE_CONNECTOR_DSI;
diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
index 2407140508d8..4bae3813d90b 100644
--- a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
+++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
@@ -2277,7 +2277,6 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev)
struct msm_drm_private *priv = dev->dev_private;
struct platform_device *pdev = priv->gpu_pdev;
struct adreno_platform_config *config = pdev->dev.platform_data;
- struct device_node *node;
struct a6xx_gpu *a6xx_gpu;
struct adreno_gpu *adreno_gpu;
struct msm_gpu *gpu;
@@ -2296,7 +2295,8 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev)
adreno_gpu->registers = NULL;
/* Check if there is a GMU phandle and set it up */
- node = of_parse_phandle(pdev->dev.of_node, "qcom,gmu", 0);
+ struct device_node *node __free(device_node) =
+ of_parse_phandle(pdev->dev.of_node, "qcom,gmu", 0);
/* FIXME: How do we gracefully handle this? */
BUG_ON(!node);
@@ -2341,7 +2341,6 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev)
ret = a6xx_gmu_wrapper_init(a6xx_gpu, node);
else
ret = a6xx_gmu_init(a6xx_gpu, node);
- of_node_put(node);
if (ret) {
a6xx_destroy(&(a6xx_gpu->base.base));
return ERR_PTR(ret);
diff --git a/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c b/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c
index 6e4e74f9d63d..83efedb810ed 100644
--- a/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c
+++ b/drivers/gpu/drm/msm/disp/mdp4/mdp4_kms.c
@@ -391,7 +391,7 @@ static int mdp4_kms_init(struct drm_device *dev)
ret = mdp_kms_init(&mdp4_kms->base, &kms_funcs);
if (ret) {
DRM_DEV_ERROR(dev->dev, "failed to init kms\n");
- goto fail;
+ return ret;
}
kms = priv->kms;
@@ -402,7 +402,7 @@ static int mdp4_kms_init(struct drm_device *dev)
ret = regulator_enable(mdp4_kms->vdd);
if (ret) {
DRM_DEV_ERROR(dev->dev, "failed to enable regulator vdd: %d\n", ret);
- goto fail;
+ return ret;
}
}
@@ -414,7 +414,7 @@ static int mdp4_kms_init(struct drm_device *dev)
DRM_DEV_ERROR(dev->dev, "unexpected MDP version: v%d.%d\n",
major, minor);
ret = -ENXIO;
- goto fail;
+ return ret;
}
mdp4_kms->rev = minor;
@@ -423,7 +423,7 @@ static int mdp4_kms_init(struct drm_device *dev)
if (!mdp4_kms->lut_clk) {
DRM_DEV_ERROR(dev->dev, "failed to get lut_clk\n");
ret = -ENODEV;
- goto fail;
+ return ret;
}
clk_set_rate(mdp4_kms->lut_clk, max_clk);
}
@@ -445,7 +445,7 @@ static int mdp4_kms_init(struct drm_device *dev)
mmu = msm_iommu_new(&pdev->dev, 0);
if (IS_ERR(mmu)) {
ret = PTR_ERR(mmu);
- goto fail;
+ return ret;
} else if (!mmu) {
DRM_DEV_INFO(dev->dev, "no iommu, fallback to phys "
"contig buffers for scanout\n");
@@ -458,7 +458,7 @@ static int mdp4_kms_init(struct drm_device *dev)
if (!IS_ERR(mmu))
mmu->funcs->destroy(mmu);
ret = PTR_ERR(aspace);
- goto fail;
+ return ret;
}
kms->aspace = aspace;
@@ -467,7 +467,7 @@ static int mdp4_kms_init(struct drm_device *dev)
ret = modeset_init(mdp4_kms);
if (ret) {
DRM_DEV_ERROR(dev->dev, "modeset_init failed: %d\n", ret);
- goto fail;
+ return ret;
}
mdp4_kms->blank_cursor_bo = msm_gem_new(dev, SZ_16K, MSM_BO_WC | MSM_BO_SCANOUT);
@@ -475,14 +475,14 @@ static int mdp4_kms_init(struct drm_device *dev)
ret = PTR_ERR(mdp4_kms->blank_cursor_bo);
DRM_DEV_ERROR(dev->dev, "could not allocate blank-cursor bo: %d\n", ret);
mdp4_kms->blank_cursor_bo = NULL;
- goto fail;
+ return ret;
}
ret = msm_gem_get_and_pin_iova(mdp4_kms->blank_cursor_bo, kms->aspace,
&mdp4_kms->blank_cursor_iova);
if (ret) {
DRM_DEV_ERROR(dev->dev, "could not pin blank-cursor bo: %d\n", ret);
- goto fail;
+ return ret;
}
dev->mode_config.min_width = 0;
@@ -491,12 +491,6 @@ static int mdp4_kms_init(struct drm_device *dev)
dev->mode_config.max_height = 2048;
return 0;
-
-fail:
- if (kms)
- mdp4_destroy(kms);
-
- return ret;
}
static const struct dev_pm_ops mdp4_pm_ops = {
diff --git a/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c b/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c
index 374704cce656..9e4c4531ff52 100644
--- a/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c
+++ b/drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c
@@ -514,7 +514,7 @@ static int mdp5_kms_init(struct drm_device *dev)
ret = mdp_kms_init(&mdp5_kms->base, &kms_funcs);
if (ret) {
DRM_DEV_ERROR(&pdev->dev, "failed to init kms\n");
- goto fail;
+ return ret;
}
config = mdp5_cfg_get_config(mdp5_kms->cfg);
@@ -537,7 +537,7 @@ static int mdp5_kms_init(struct drm_device *dev)
aspace = msm_kms_init_aspace(mdp5_kms->dev);
if (IS_ERR(aspace)) {
ret = PTR_ERR(aspace);
- goto fail;
+ return ret;
}
kms->aspace = aspace;
@@ -547,7 +547,7 @@ static int mdp5_kms_init(struct drm_device *dev)
ret = modeset_init(mdp5_kms);
if (ret) {
DRM_DEV_ERROR(&pdev->dev, "modeset_init failed: %d\n", ret);
- goto fail;
+ return ret;
}
dev->mode_config.min_width = 0;
@@ -559,11 +559,6 @@ static int mdp5_kms_init(struct drm_device *dev)
dev->vblank_disable_immediate = true;
return 0;
-fail:
- if (kms)
- mdp5_kms_destroy(kms);
-
- return ret;
}
static void mdp5_destroy(struct mdp5_kms *mdp5_kms)
diff --git a/drivers/gpu/drm/msm/msm_gpu.c b/drivers/gpu/drm/msm/msm_gpu.c
index a274b8466423..76b44bb8de0d 100644
--- a/drivers/gpu/drm/msm/msm_gpu.c
+++ b/drivers/gpu/drm/msm/msm_gpu.c
@@ -421,25 +421,23 @@ static void recover_worker(struct kthread_work *work)
msm_update_fence(ring->fctx, fence);
}
- if (msm_gpu_active(gpu)) {
- /* retire completed submits, plus the one that hung: */
- retire_submits(gpu);
+ gpu->funcs->recover(gpu);
- gpu->funcs->recover(gpu);
+ /* retire completed submits, plus the one that hung: */
+ retire_submits(gpu);
- /*
- * Replay all remaining submits starting with highest priority
- * ring
- */
- for (i = 0; i < gpu->nr_rings; i++) {
- struct msm_ringbuffer *ring = gpu->rb[i];
- unsigned long flags;
+ /*
+ * Replay all remaining submits starting with highest priority
+ * ring
+ */
+ for (i = 0; i < gpu->nr_rings; i++) {
+ struct msm_ringbuffer *ring = gpu->rb[i];
+ unsigned long flags;
- spin_lock_irqsave(&ring->submit_lock, flags);
- list_for_each_entry(submit, &ring->submits, node)
- gpu->funcs->submit(gpu, submit);
- spin_unlock_irqrestore(&ring->submit_lock, flags);
- }
+ spin_lock_irqsave(&ring->submit_lock, flags);
+ list_for_each_entry(submit, &ring->submits, node)
+ gpu->funcs->submit(gpu, submit);
+ spin_unlock_irqrestore(&ring->submit_lock, flags);
}
pm_runtime_put(&gpu->pdev->dev);
diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c
index 9d6655f96f0c..917768a17a0b 100644
--- a/drivers/gpu/drm/msm/msm_ringbuffer.c
+++ b/drivers/gpu/drm/msm/msm_ringbuffer.c
@@ -59,8 +59,14 @@ static const struct drm_sched_backend_ops msm_sched_ops = {
struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,
void *memptrs, uint64_t memptrs_iova)
{
+ struct drm_sched_init_args args = {
+ .ops = &msm_sched_ops,
+ .num_rqs = DRM_SCHED_PRIORITY_COUNT,
+ .credit_limit = num_hw_submissions,
+ .timeout = MAX_SCHEDULE_TIMEOUT,
+ .dev = gpu->dev->dev,
+ };
struct msm_ringbuffer *ring;
- long sched_timeout;
char name[32];
int ret;
@@ -87,6 +93,7 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,
}
msm_gem_object_set_name(ring->bo, "ring%d", id);
+ args.name = to_msm_bo(ring->bo)->name,
ring->end = ring->start + (MSM_GPU_RINGBUFFER_SZ >> 2);
ring->next = ring->start;
@@ -95,16 +102,10 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,
ring->memptrs = memptrs;
ring->memptrs_iova = memptrs_iova;
- /* currently managing hangcheck ourselves: */
- sched_timeout = MAX_SCHEDULE_TIMEOUT;
-
- ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- num_hw_submissions, 0, sched_timeout,
- NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev);
- if (ret) {
+ ret = drm_sched_init(&ring->sched, &args);
+ if (ret)
goto fail;
- }
+ ring->sched_initialized = true;
INIT_LIST_HEAD(&ring->submits);
spin_lock_init(&ring->submit_lock);
@@ -126,7 +127,8 @@ void msm_ringbuffer_destroy(struct msm_ringbuffer *ring)
if (IS_ERR_OR_NULL(ring))
return;
- drm_sched_fini(&ring->sched);
+ if (ring->sched_initialized)
+ drm_sched_fini(&ring->sched);
msm_fence_context_free(ring->fctx);
diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.h b/drivers/gpu/drm/msm/msm_ringbuffer.h
index 0d6beb8cd39a..72e4de1aafa2 100644
--- a/drivers/gpu/drm/msm/msm_ringbuffer.h
+++ b/drivers/gpu/drm/msm/msm_ringbuffer.h
@@ -52,6 +52,7 @@ struct msm_ringbuffer {
* The job scheduler for this ring.
*/
struct drm_gpu_scheduler sched;
+ bool sched_initialized;
/*
* List of in-flight submits on this ring. Protected by submit_lock.
diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c
index 23f60b7d0e4f..7676d962a959 100644
--- a/drivers/gpu/drm/nouveau/nouveau_sched.c
+++ b/drivers/gpu/drm/nouveau/nouveau_sched.c
@@ -405,7 +405,14 @@ nouveau_sched_init(struct nouveau_sched *sched, struct nouveau_drm *drm,
{
struct drm_gpu_scheduler *drm_sched = &sched->base;
struct drm_sched_entity *entity = &sched->entity;
- const long timeout = msecs_to_jiffies(NOUVEAU_SCHED_JOB_TIMEOUT_MS);
+ struct drm_sched_init_args args = {
+ .ops = &nouveau_sched_ops,
+ .num_rqs = DRM_SCHED_PRIORITY_COUNT,
+ .credit_limit = credit_limit,
+ .timeout = msecs_to_jiffies(NOUVEAU_SCHED_JOB_TIMEOUT_MS),
+ .name = "nouveau_sched",
+ .dev = drm->dev->dev
+ };
int ret;
if (!wq) {
@@ -417,10 +424,9 @@ nouveau_sched_init(struct nouveau_sched *sched, struct nouveau_drm *drm,
sched->wq = wq;
}
- ret = drm_sched_init(drm_sched, &nouveau_sched_ops, wq,
- NOUVEAU_SCHED_PRIORITY_COUNT,
- credit_limit, 0, timeout,
- NULL, NULL, "nouveau_sched", drm->dev->dev);
+ args.submit_wq = wq,
+
+ ret = drm_sched_init(drm_sched, &args);
if (ret)
goto fail_wq;
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c
index 3ad4f6e9a8ac..5952f0abc070 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c
@@ -734,12 +734,18 @@ r535_gsp_rpc_push(struct nvkm_gsp *gsp, void *argv, bool wait, u32 repc)
mutex_lock(&gsp->cmdq.mutex);
if (rpc_size > max_rpc_size) {
const u32 fn = rpc->function;
+ void *next;
- /* Adjust length, and send initial RPC. */
- rpc->length = sizeof(*rpc) + max_rpc_size;
- cmd->checksum = rpc->length;
+ /* Send initial RPC. */
+ next = r535_gsp_rpc_get(gsp, fn, max_rpc_size);
+ if (IS_ERR(next)) {
+ repv = next;
+ goto done;
+ }
- repv = r535_gsp_rpc_send(gsp, argv, false, 0);
+ memcpy(next, argv, max_rpc_size);
+
+ repv = r535_gsp_rpc_send(gsp, next, false, 0);
if (IS_ERR(repv))
goto done;
@@ -749,7 +755,6 @@ r535_gsp_rpc_push(struct nvkm_gsp *gsp, void *argv, bool wait, u32 repc)
/* Remaining chunks sent as CONTINUATION_RECORD RPCs. */
while (rpc_size) {
u32 size = min(rpc_size, max_rpc_size);
- void *next;
next = r535_gsp_rpc_get(gsp, NV_VGPU_MSG_FUNCTION_CONTINUATION_RECORD, size);
if (IS_ERR(next)) {
@@ -777,6 +782,8 @@ r535_gsp_rpc_push(struct nvkm_gsp *gsp, void *argv, bool wait, u32 repc)
} else {
repv = NULL;
}
+ if (!IS_ERR(repv))
+ kvfree(cmd);
} else {
repv = r535_gsp_rpc_send(gsp, argv, wait, repc);
}
@@ -1469,8 +1476,8 @@ r535_gsp_acpi_caps(acpi_handle handle, CAPS_METHOD_DATA *caps)
if (!obj)
goto done;
- if (WARN_ON(obj->type != ACPI_TYPE_BUFFER) ||
- WARN_ON(obj->buffer.length != 4))
+ if (obj->type != ACPI_TYPE_BUFFER ||
+ obj->buffer.length != 4)
goto done;
caps->status = 0;
@@ -1505,8 +1512,8 @@ r535_gsp_acpi_jt(acpi_handle handle, JT_METHOD_DATA *jt)
if (!obj)
goto done;
- if (WARN_ON(obj->type != ACPI_TYPE_BUFFER) ||
- WARN_ON(obj->buffer.length != 4))
+ if (obj->type != ACPI_TYPE_BUFFER ||
+ obj->buffer.length != 4)
goto done;
jt->status = 0;
@@ -1593,8 +1600,8 @@ r535_gsp_acpi_dod(acpi_handle handle, DOD_METHOD_DATA *dod)
_DOD = output.pointer;
- if (WARN_ON(_DOD->type != ACPI_TYPE_PACKAGE) ||
- WARN_ON(_DOD->package.count > ARRAY_SIZE(dod->acpiIdList)))
+ if (_DOD->type != ACPI_TYPE_PACKAGE ||
+ _DOD->package.count > ARRAY_SIZE(dod->acpiIdList))
return;
for (int i = 0; i < _DOD->package.count; i++) {
diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c
index df49d37d0e7e..20b52e17db27 100644
--- a/drivers/gpu/drm/panfrost/panfrost_job.c
+++ b/drivers/gpu/drm/panfrost/panfrost_job.c
@@ -830,8 +830,15 @@ static irqreturn_t panfrost_job_irq_handler(int irq, void *data)
int panfrost_job_init(struct panfrost_device *pfdev)
{
+ struct drm_sched_init_args args = {
+ .ops = &panfrost_sched_ops,
+ .num_rqs = DRM_SCHED_PRIORITY_COUNT,
+ .credit_limit = 2,
+ .timeout = msecs_to_jiffies(JOB_TIMEOUT_MS),
+ .name = "pan_js",
+ .dev = pfdev->dev,
+ };
struct panfrost_job_slot *js;
- unsigned int nentries = 2;
int ret, j;
/* All GPUs have two entries per queue, but without jobchain
@@ -839,7 +846,7 @@ int panfrost_job_init(struct panfrost_device *pfdev)
* so let's just advertise one entry in that case.
*/
if (!panfrost_has_hw_feature(pfdev, HW_FEATURE_JOBCHAIN_DISAMBIGUATION))
- nentries = 1;
+ args.credit_limit = 1;
pfdev->js = js = devm_kzalloc(pfdev->dev, sizeof(*js), GFP_KERNEL);
if (!js)
@@ -865,17 +872,12 @@ int panfrost_job_init(struct panfrost_device *pfdev)
pfdev->reset.wq = alloc_ordered_workqueue("panfrost-reset", 0);
if (!pfdev->reset.wq)
return -ENOMEM;
+ args.timeout_wq = pfdev->reset.wq;
for (j = 0; j < NUM_JOB_SLOTS; j++) {
js->queue[j].fence_context = dma_fence_context_alloc(1);
- ret = drm_sched_init(&js->queue[j].sched,
- &panfrost_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- nentries, 0,
- msecs_to_jiffies(JOB_TIMEOUT_MS),
- pfdev->reset.wq,
- NULL, "pan_js", pfdev->dev);
+ ret = drm_sched_init(&js->queue[j].sched, &args);
if (ret) {
dev_err(pfdev->dev, "Failed to create scheduler: %d.", ret);
goto err_sched;
diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c b/drivers/gpu/drm/panthor/panthor_mmu.c
index 9b6afc9c1da1..30107d28d230 100644
--- a/drivers/gpu/drm/panthor/panthor_mmu.c
+++ b/drivers/gpu/drm/panthor/panthor_mmu.c
@@ -311,6 +311,9 @@ struct panthor_vm {
u64 end;
} kernel_auto_va;
+ /** @user_va_range: Upper boundary of VAs VM users can map objects against. */
+ u64 user_va_range;
+
/** @as: Address space related fields. */
struct {
/**
@@ -2293,6 +2296,16 @@ panthor_vm_create(struct panthor_device *ptdev, bool for_mcu,
u64 full_va_range = 1ull << va_bits;
struct drm_gem_object *dummy_gem;
struct drm_gpu_scheduler *sched;
+ const struct drm_sched_init_args sched_args = {
+ .ops = &panthor_vm_bind_ops,
+ .submit_wq = ptdev->mmu->vm.wq,
+ .num_rqs = 1,
+ .credit_limit = 1,
+ /* Bind operations are synchronous for now, no timeout needed. */
+ .timeout = MAX_SCHEDULE_TIMEOUT,
+ .name = "panthor-vm-bind",
+ .dev = ptdev->base.dev,
+ };
struct io_pgtable_cfg pgtbl_cfg;
u64 mair, min_va, va_range;
struct panthor_vm *vm;
@@ -2323,6 +2336,8 @@ panthor_vm_create(struct panthor_device *ptdev, bool for_mcu,
va_range = full_va_range;
}
+ vm->user_va_range = kernel_va_start;
+
mutex_init(&vm->mm_lock);
drm_mm_init(&vm->mm, kernel_va_start, kernel_va_size);
vm->kernel_auto_va.start = auto_kernel_va_start;
@@ -2350,11 +2365,7 @@ panthor_vm_create(struct panthor_device *ptdev, bool for_mcu,
goto err_mm_takedown;
}
- /* Bind operations are synchronous for now, no timeout needed. */
- ret = drm_sched_init(&vm->sched, &panthor_vm_bind_ops, ptdev->mmu->vm.wq,
- 1, 1, 0,
- MAX_SCHEDULE_TIMEOUT, NULL, NULL,
- "panthor-vm-bind", ptdev->base.dev);
+ ret = drm_sched_init(&vm->sched, &sched_args);
if (ret)
goto err_free_io_pgtable;
@@ -2412,6 +2423,10 @@ panthor_vm_bind_prepare_op_ctx(struct drm_file *file,
if (!IS_ALIGNED(op->va | op->size | op->bo_offset, vm_pgsz))
return -EINVAL;
+ /* We don't allow mappings that overlap with kbo's reserved range */
+ if (range_overflows(op->va, op->size, vm->user_va_range))
+ return -EINVAL;
+
switch (op->flags & DRM_PANTHOR_VM_BIND_OP_TYPE_MASK) {
case DRM_PANTHOR_VM_BIND_OP_TYPE_MAP:
gem = drm_gem_object_lookup(file, op->bo_handle);
diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
index 822b003893a8..e3fbe34924ce 100644
--- a/drivers/gpu/drm/panthor/panthor_sched.c
+++ b/drivers/gpu/drm/panthor/panthor_sched.c
@@ -742,6 +742,12 @@ struct panthor_job_profiling_data {
*/
#define MAX_GROUPS_PER_POOL 128
+/*
+ * Mark added on an entry of group pool Xarray to identify if the group has
+ * been fully initialized and can be accessed elsewhere in the driver code.
+ */
+#define GROUP_REGISTERED XA_MARK_1
+
/**
* struct panthor_group_pool - Group pool
*
@@ -1383,7 +1389,7 @@ cs_slot_process_fault_event_locked(struct panthor_device *ptdev,
u64 cs_extract = queue->iface.output->extract;
struct panthor_job *job;
- spin_lock(&queue->fence_ctx.lock);
+ guard(spinlock_irqsave)(&queue->fence_ctx.lock);
list_for_each_entry(job, &queue->fence_ctx.in_flight_jobs, node) {
if (cs_extract >= job->ringbuf.end)
continue;
@@ -1393,7 +1399,6 @@ cs_slot_process_fault_event_locked(struct panthor_device *ptdev,
dma_fence_set_error(job->done_fence, -EINVAL);
}
- spin_unlock(&queue->fence_ctx.lock);
}
drm_warn(&ptdev->base,
@@ -2051,13 +2056,13 @@ group_term_post_processing(struct panthor_group *group)
if (!queue)
continue;
- spin_lock(&queue->fence_ctx.lock);
- list_for_each_entry_safe(job, tmp, &queue->fence_ctx.in_flight_jobs, node) {
- list_move_tail(&job->node, &faulty_jobs);
- dma_fence_set_error(job->done_fence, err);
- dma_fence_signal_locked(job->done_fence);
+ scoped_guard(spinlock_irqsave, &queue->fence_ctx.lock) {
+ list_for_each_entry_safe(job, tmp, &queue->fence_ctx.in_flight_jobs, node) {
+ list_move_tail(&job->node, &faulty_jobs);
+ dma_fence_set_error(job->done_fence, err);
+ dma_fence_signal_locked(job->done_fence);
+ }
}
- spin_unlock(&queue->fence_ctx.lock);
/* Manually update the syncobj seqno to unblock waiters. */
syncobj = group->syncobjs->kmap + (i * sizeof(*syncobj));
@@ -2896,15 +2901,15 @@ static void group_sync_upd_work(struct work_struct *work)
syncobj = group->syncobjs->kmap + (queue_idx * sizeof(*syncobj));
- spin_lock(&queue->fence_ctx.lock);
- list_for_each_entry_safe(job, job_tmp, &queue->fence_ctx.in_flight_jobs, node) {
- if (syncobj->seqno < job->done_fence->seqno)
- break;
+ scoped_guard(spinlock_irqsave, &queue->fence_ctx.lock) {
+ list_for_each_entry_safe(job, job_tmp, &queue->fence_ctx.in_flight_jobs, node) {
+ if (syncobj->seqno < job->done_fence->seqno)
+ break;
- list_move_tail(&job->node, &done_jobs);
- dma_fence_signal_locked(job->done_fence);
+ list_move_tail(&job->node, &done_jobs);
+ dma_fence_signal_locked(job->done_fence);
+ }
}
- spin_unlock(&queue->fence_ctx.lock);
}
dma_fence_end_signalling(cookie);
@@ -3147,9 +3152,8 @@ queue_run_job(struct drm_sched_job *sched_job)
job->ringbuf.end = job->ringbuf.start + (instrs.count * sizeof(u64));
panthor_job_get(&job->base);
- spin_lock(&queue->fence_ctx.lock);
- list_add_tail(&job->node, &queue->fence_ctx.in_flight_jobs);
- spin_unlock(&queue->fence_ctx.lock);
+ scoped_guard(spinlock_irqsave, &queue->fence_ctx.lock)
+ list_add_tail(&job->node, &queue->fence_ctx.in_flight_jobs);
/* Make sure the ring buffer is updated before the INSERT
* register.
@@ -3286,6 +3290,22 @@ static struct panthor_queue *
group_create_queue(struct panthor_group *group,
const struct drm_panthor_queue_create *args)
{
+ const struct drm_sched_init_args sched_args = {
+ .ops = &panthor_queue_sched_ops,
+ .submit_wq = group->ptdev->scheduler->wq,
+ .num_rqs = 1,
+ /*
+ * The credit limit argument tells us the total number of
+ * instructions across all CS slots in the ringbuffer, with
+ * some jobs requiring twice as many as others, depending on
+ * their profiling status.
+ */
+ .credit_limit = args->ringbuf_size / sizeof(u64),
+ .timeout = msecs_to_jiffies(JOB_TIMEOUT_MS),
+ .timeout_wq = group->ptdev->reset.wq,
+ .name = "panthor-queue",
+ .dev = group->ptdev->base.dev,
+ };
struct drm_gpu_scheduler *drm_sched;
struct panthor_queue *queue;
int ret;
@@ -3356,17 +3376,7 @@ group_create_queue(struct panthor_group *group,
if (ret)
goto err_free_queue;
- /*
- * Credit limit argument tells us the total number of instructions
- * across all CS slots in the ringbuffer, with some jobs requiring
- * twice as many as others, depending on their profiling status.
- */
- ret = drm_sched_init(&queue->scheduler, &panthor_queue_sched_ops,
- group->ptdev->scheduler->wq, 1,
- args->ringbuf_size / sizeof(u64),
- 0, msecs_to_jiffies(JOB_TIMEOUT_MS),
- group->ptdev->reset.wq,
- NULL, "panthor-queue", group->ptdev->base.dev);
+ ret = drm_sched_init(&queue->scheduler, &sched_args);
if (ret)
goto err_free_queue;
@@ -3509,6 +3519,8 @@ int panthor_group_create(struct panthor_file *pfile,
}
mutex_unlock(&sched->reset.lock);
+ xa_set_mark(&gpool->xa, gid, GROUP_REGISTERED);
+
return gid;
err_put_group:
@@ -3523,6 +3535,9 @@ int panthor_group_destroy(struct panthor_file *pfile, u32 group_handle)
struct panthor_scheduler *sched = ptdev->scheduler;
struct panthor_group *group;
+ if (!xa_get_mark(&gpool->xa, group_handle, GROUP_REGISTERED))
+ return -EINVAL;
+
group = xa_erase(&gpool->xa, group_handle);
if (!group)
return -EINVAL;
@@ -3548,12 +3563,12 @@ int panthor_group_destroy(struct panthor_file *pfile, u32 group_handle)
}
static struct panthor_group *group_from_handle(struct panthor_group_pool *pool,
- u32 group_handle)
+ unsigned long group_handle)
{
struct panthor_group *group;
xa_lock(&pool->xa);
- group = group_get(xa_load(&pool->xa, group_handle));
+ group = group_get(xa_find(&pool->xa, &group_handle, group_handle, GROUP_REGISTERED));
xa_unlock(&pool->xa);
return group;
diff --git a/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c b/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c
index a9216a99b2be..5b8da2b6cbd0 100644
--- a/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c
+++ b/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c
@@ -574,6 +574,13 @@ static void rzg2l_mipi_dsi_atomic_disable(struct drm_bridge *bridge,
rzg2l_mipi_dsi_stop_video(dsi);
rzg2l_mipi_dsi_stop_hs_clock(dsi);
+}
+
+static void rzg2l_mipi_dsi_atomic_post_disable(struct drm_bridge *bridge,
+ struct drm_bridge_state *old_bridge_state)
+{
+ struct rzg2l_mipi_dsi *dsi = bridge_to_rzg2l_mipi_dsi(bridge);
+
rzg2l_mipi_dsi_stop(dsi);
}
@@ -598,6 +605,7 @@ static const struct drm_bridge_funcs rzg2l_mipi_dsi_bridge_ops = {
.atomic_reset = drm_atomic_helper_bridge_reset,
.atomic_enable = rzg2l_mipi_dsi_atomic_enable,
.atomic_disable = rzg2l_mipi_dsi_atomic_disable,
+ .atomic_post_disable = rzg2l_mipi_dsi_atomic_post_disable,
.mode_valid = rzg2l_mipi_dsi_bridge_mode_valid,
};
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
index 42521bd2ac16..e3a65056c9c5 100644
--- a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
@@ -2764,22 +2764,29 @@ static int vop2_plane_init(struct vop2 *vop2, struct vop2_win *win,
return 0;
}
-static struct vop2_video_port *find_vp_without_primary(struct vop2 *vop2)
+/*
+ * On RK3566 these windows don't have an independent
+ * framebuffer. They can only share/mirror the framebuffer
+ * with smart0, esmart0 and cluster0 respectively.
+ * And RK3566 share the same vop version with Rk3568, so we
+ * need to use soc_id for identification here.
+ */
+static bool vop2_is_mirror_win(struct vop2_win *win)
{
- int i;
-
- for (i = 0; i < vop2->data->nr_vps; i++) {
- struct vop2_video_port *vp = &vop2->vps[i];
-
- if (!vp->crtc.port)
- continue;
- if (vp->primary_plane)
- continue;
+ struct vop2 *vop2 = win->vop2;
- return vp;
+ if (vop2->data->soc_id == 3566) {
+ switch (win->data->phys_id) {
+ case ROCKCHIP_VOP2_SMART1:
+ case ROCKCHIP_VOP2_ESMART1:
+ case ROCKCHIP_VOP2_CLUSTER1:
+ return true;
+ default:
+ return false;
+ }
+ } else {
+ return false;
}
-
- return NULL;
}
static int vop2_create_crtcs(struct vop2 *vop2)
@@ -2790,7 +2797,9 @@ static int vop2_create_crtcs(struct vop2 *vop2)
struct drm_plane *plane;
struct device_node *port;
struct vop2_video_port *vp;
- int i, nvp, nvps = 0;
+ struct vop2_win *win;
+ u32 possible_crtcs;
+ int i, j, nvp, nvps = 0;
int ret;
for (i = 0; i < vop2_data->nr_vps; i++) {
@@ -2806,10 +2815,9 @@ static int vop2_create_crtcs(struct vop2 *vop2)
snprintf(dclk_name, sizeof(dclk_name), "dclk_vp%d", vp->id);
vp->dclk = devm_clk_get(vop2->dev, dclk_name);
- if (IS_ERR(vp->dclk)) {
- drm_err(vop2->drm, "failed to get %s\n", dclk_name);
- return PTR_ERR(vp->dclk);
- }
+ if (IS_ERR(vp->dclk))
+ return dev_err_probe(drm->dev, PTR_ERR(vp->dclk),
+ "failed to get %s\n", dclk_name);
np = of_graph_get_remote_node(dev->of_node, i, -1);
if (!np) {
@@ -2819,55 +2827,67 @@ static int vop2_create_crtcs(struct vop2 *vop2)
of_node_put(np);
port = of_graph_get_port_by_id(dev->of_node, i);
- if (!port) {
- drm_err(vop2->drm, "no port node found for video_port%d\n", i);
- return -ENOENT;
- }
-
+ if (!port)
+ return dev_err_probe(drm->dev, -ENOENT,
+ "no port node found for video_port%d\n", i);
vp->crtc.port = port;
nvps++;
}
nvp = 0;
- for (i = 0; i < vop2->registered_num_wins; i++) {
- struct vop2_win *win = &vop2->win[i];
- u32 possible_crtcs = 0;
+ /* Register a primary plane for every crtc */
+ for (i = 0; i < vop2_data->nr_vps; i++) {
+ vp = &vop2->vps[i];
- if (vop2->data->soc_id == 3566) {
- /*
- * On RK3566 these windows don't have an independent
- * framebuffer. They share the framebuffer with smart0,
- * esmart0 and cluster0 respectively.
- */
- switch (win->data->phys_id) {
- case ROCKCHIP_VOP2_SMART1:
- case ROCKCHIP_VOP2_ESMART1:
- case ROCKCHIP_VOP2_CLUSTER1:
+ if (!vp->crtc.port)
+ continue;
+
+ for (j = 0; j < vop2->registered_num_wins; j++) {
+ win = &vop2->win[j];
+
+ /* Aready registered as primary plane */
+ if (win->base.type == DRM_PLANE_TYPE_PRIMARY)
+ continue;
+
+ if (vop2_is_mirror_win(win))
continue;
- }
- }
- if (win->type == DRM_PLANE_TYPE_PRIMARY) {
- vp = find_vp_without_primary(vop2);
- if (vp) {
+ if (win->type == DRM_PLANE_TYPE_PRIMARY) {
possible_crtcs = BIT(nvp);
vp->primary_plane = win;
+ ret = vop2_plane_init(vop2, win, possible_crtcs);
+ if (ret)
+ return dev_err_probe(drm->dev, ret,
+ "failed to init primary plane %s\n",
+ win->data->name);
nvp++;
- } else {
- /* change the unused primary window to overlay window */
- win->type = DRM_PLANE_TYPE_OVERLAY;
+ break;
}
}
- if (win->type == DRM_PLANE_TYPE_OVERLAY)
- possible_crtcs = (1 << nvps) - 1;
+ if (!vp->primary_plane)
+ return dev_err_probe(drm->dev, -ENOENT,
+ "no primary plane for vp %d\n", i);
+ }
+
+ /* Register all unused window as overlay plane */
+ for (i = 0; i < vop2->registered_num_wins; i++) {
+ win = &vop2->win[i];
+
+ /* Aready registered as primary plane */
+ if (win->base.type == DRM_PLANE_TYPE_PRIMARY)
+ continue;
+
+ if (vop2_is_mirror_win(win))
+ continue;
+
+ win->type = DRM_PLANE_TYPE_OVERLAY;
+ possible_crtcs = (1 << nvps) - 1;
ret = vop2_plane_init(vop2, win, possible_crtcs);
- if (ret) {
- drm_err(vop2->drm, "failed to init plane %s: %d\n",
- win->data->name, ret);
- return ret;
- }
+ if (ret)
+ return dev_err_probe(drm->dev, ret, "failed to init overlay plane %s\n",
+ win->data->name);
}
for (i = 0; i < vop2_data->nr_vps; i++) {
@@ -2881,10 +2901,9 @@ static int vop2_create_crtcs(struct vop2 *vop2)
ret = drm_crtc_init_with_planes(drm, &vp->crtc, plane, NULL,
&vop2_crtc_funcs,
"video_port%d", vp->id);
- if (ret) {
- drm_err(vop2->drm, "crtc init for video_port%d failed\n", i);
- return ret;
- }
+ if (ret)
+ return dev_err_probe(drm->dev, ret,
+ "crtc init for video_port%d failed\n", i);
drm_crtc_helper_add(&vp->crtc, &vop2_crtc_helper_funcs);
@@ -3206,10 +3225,9 @@ static int vop2_bind(struct device *dev, struct device *master, void *data)
dev_set_drvdata(dev, vop2);
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "vop");
- if (!res) {
- drm_err(vop2->drm, "failed to get vop2 register byname\n");
- return -EINVAL;
- }
+ if (!res)
+ return dev_err_probe(drm->dev, -EINVAL,
+ "failed to get vop2 register byname\n");
vop2->regs = devm_ioremap_resource(dev, res);
if (IS_ERR(vop2->regs))
@@ -3233,56 +3251,54 @@ static int vop2_bind(struct device *dev, struct device *master, void *data)
if (vop2_data->feature & VOP2_FEATURE_HAS_SYS_GRF) {
vop2->sys_grf = syscon_regmap_lookup_by_phandle(dev->of_node, "rockchip,grf");
if (IS_ERR(vop2->sys_grf))
- return dev_err_probe(dev, PTR_ERR(vop2->sys_grf), "cannot get sys_grf");
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->sys_grf),
+ "cannot get sys_grf\n");
}
if (vop2_data->feature & VOP2_FEATURE_HAS_VOP_GRF) {
vop2->vop_grf = syscon_regmap_lookup_by_phandle(dev->of_node, "rockchip,vop-grf");
if (IS_ERR(vop2->vop_grf))
- return dev_err_probe(dev, PTR_ERR(vop2->vop_grf), "cannot get vop_grf");
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->vop_grf),
+ "cannot get vop_grf\n");
}
if (vop2_data->feature & VOP2_FEATURE_HAS_VO1_GRF) {
vop2->vo1_grf = syscon_regmap_lookup_by_phandle(dev->of_node, "rockchip,vo1-grf");
if (IS_ERR(vop2->vo1_grf))
- return dev_err_probe(dev, PTR_ERR(vop2->vo1_grf), "cannot get vo1_grf");
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->vo1_grf),
+ "cannot get vo1_grf\n");
}
if (vop2_data->feature & VOP2_FEATURE_HAS_SYS_PMU) {
vop2->sys_pmu = syscon_regmap_lookup_by_phandle(dev->of_node, "rockchip,pmu");
if (IS_ERR(vop2->sys_pmu))
- return dev_err_probe(dev, PTR_ERR(vop2->sys_pmu), "cannot get sys_pmu");
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->sys_pmu),
+ "cannot get sys_pmu\n");
}
vop2->hclk = devm_clk_get(vop2->dev, "hclk");
- if (IS_ERR(vop2->hclk)) {
- drm_err(vop2->drm, "failed to get hclk source\n");
- return PTR_ERR(vop2->hclk);
- }
+ if (IS_ERR(vop2->hclk))
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->hclk),
+ "failed to get hclk source\n");
vop2->aclk = devm_clk_get(vop2->dev, "aclk");
- if (IS_ERR(vop2->aclk)) {
- drm_err(vop2->drm, "failed to get aclk source\n");
- return PTR_ERR(vop2->aclk);
- }
+ if (IS_ERR(vop2->aclk))
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->aclk),
+ "failed to get aclk source\n");
vop2->pclk = devm_clk_get_optional(vop2->dev, "pclk_vop");
- if (IS_ERR(vop2->pclk)) {
- drm_err(vop2->drm, "failed to get pclk source\n");
- return PTR_ERR(vop2->pclk);
- }
+ if (IS_ERR(vop2->pclk))
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->pclk),
+ "failed to get pclk source\n");
vop2->pll_hdmiphy0 = devm_clk_get_optional(vop2->dev, "pll_hdmiphy0");
- if (IS_ERR(vop2->pll_hdmiphy0)) {
- drm_err(vop2->drm, "failed to get pll_hdmiphy0\n");
- return PTR_ERR(vop2->pll_hdmiphy0);
- }
+ if (IS_ERR(vop2->pll_hdmiphy0))
+ return dev_err_probe(drm->dev, PTR_ERR(vop2->pll_hdmiphy0),
+ "failed to get pll_hdmiphy0\n");
vop2->irq = platform_get_irq(pdev, 0);
- if (vop2->irq < 0) {
- drm_err(vop2->drm, "cannot find irq for vop2\n");
- return vop2->irq;
- }
+ if (vop2->irq < 0)
+ return dev_err_probe(drm->dev, vop2->irq, "cannot find irq for vop2\n");
mutex_init(&vop2->vop2_lock);
diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
index 34ebc2fa3a5a..9a80d0115267 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -1236,40 +1236,24 @@ static void drm_sched_run_job_work(struct work_struct *w)
* drm_sched_init - Init a gpu scheduler instance
*
* @sched: scheduler instance
- * @ops: backend operations for this scheduler
- * @submit_wq: workqueue to use for submission. If NULL, an ordered wq is
- * allocated and used
- * @num_rqs: number of runqueues, one for each priority, up to DRM_SCHED_PRIORITY_COUNT
- * @credit_limit: the number of credits this scheduler can hold from all jobs
- * @hang_limit: number of times to allow a job to hang before dropping it
- * @timeout: timeout value in jiffies for the scheduler
- * @timeout_wq: workqueue to use for timeout work. If NULL, the system_wq is
- * used
- * @score: optional score atomic shared with other schedulers
- * @name: name used for debugging
- * @dev: target &struct device
+ * @args: scheduler initialization arguments
*
* Return 0 on success, otherwise error code.
*/
-int drm_sched_init(struct drm_gpu_scheduler *sched,
- const struct drm_sched_backend_ops *ops,
- struct workqueue_struct *submit_wq,
- u32 num_rqs, u32 credit_limit, unsigned int hang_limit,
- long timeout, struct workqueue_struct *timeout_wq,
- atomic_t *score, const char *name, struct device *dev)
+int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_init_args *args)
{
int i;
- sched->ops = ops;
- sched->credit_limit = credit_limit;
- sched->name = name;
- sched->timeout = timeout;
- sched->timeout_wq = timeout_wq ? : system_wq;
- sched->hang_limit = hang_limit;
- sched->score = score ? score : &sched->_score;
- sched->dev = dev;
+ sched->ops = args->ops;
+ sched->credit_limit = args->credit_limit;
+ sched->name = args->name;
+ sched->timeout = args->timeout;
+ sched->hang_limit = args->hang_limit;
+ sched->timeout_wq = args->timeout_wq ? args->timeout_wq : system_wq;
+ sched->score = args->score ? args->score : &sched->_score;
+ sched->dev = args->dev;
- if (num_rqs > DRM_SCHED_PRIORITY_COUNT) {
+ if (args->num_rqs > DRM_SCHED_PRIORITY_COUNT) {
/* This is a gross violation--tell drivers what the problem is.
*/
drm_err(sched, "%s: num_rqs cannot be greater than DRM_SCHED_PRIORITY_COUNT\n",
@@ -1284,16 +1268,16 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,
return 0;
}
- if (submit_wq) {
- sched->submit_wq = submit_wq;
+ if (args->submit_wq) {
+ sched->submit_wq = args->submit_wq;
sched->own_submit_wq = false;
} else {
#ifdef CONFIG_LOCKDEP
- sched->submit_wq = alloc_ordered_workqueue_lockdep_map(name,
+ sched->submit_wq = alloc_ordered_workqueue_lockdep_map(args->name,
WQ_MEM_RECLAIM,
&drm_sched_lockdep_map);
#else
- sched->submit_wq = alloc_ordered_workqueue(name, WQ_MEM_RECLAIM);
+ sched->submit_wq = alloc_ordered_workqueue(args->name, WQ_MEM_RECLAIM);
#endif
if (!sched->submit_wq)
return -ENOMEM;
@@ -1301,11 +1285,11 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,
sched->own_submit_wq = true;
}
- sched->sched_rq = kmalloc_array(num_rqs, sizeof(*sched->sched_rq),
+ sched->sched_rq = kmalloc_array(args->num_rqs, sizeof(*sched->sched_rq),
GFP_KERNEL | __GFP_ZERO);
if (!sched->sched_rq)
goto Out_check_own;
- sched->num_rqs = num_rqs;
+ sched->num_rqs = args->num_rqs;
for (i = DRM_SCHED_PRIORITY_KERNEL; i < sched->num_rqs; i++) {
sched->sched_rq[i] = kzalloc(sizeof(*sched->sched_rq[i]), GFP_KERNEL);
if (!sched->sched_rq[i])
diff --git a/drivers/gpu/drm/tilcdc/tilcdc_crtc.c b/drivers/gpu/drm/tilcdc/tilcdc_crtc.c
index b5f60b2b2d0e..41802c9bd147 100644
--- a/drivers/gpu/drm/tilcdc/tilcdc_crtc.c
+++ b/drivers/gpu/drm/tilcdc/tilcdc_crtc.c
@@ -586,7 +586,7 @@ static void tilcdc_crtc_recover_work(struct work_struct *work)
drm_modeset_unlock(&crtc->mutex);
}
-static void tilcdc_crtc_destroy(struct drm_crtc *crtc)
+void tilcdc_crtc_destroy(struct drm_crtc *crtc)
{
struct tilcdc_drm_private *priv = crtc->dev->dev_private;
diff --git a/drivers/gpu/drm/tilcdc/tilcdc_drv.c b/drivers/gpu/drm/tilcdc/tilcdc_drv.c
index cd5eefa06060..322e02f5a763 100644
--- a/drivers/gpu/drm/tilcdc/tilcdc_drv.c
+++ b/drivers/gpu/drm/tilcdc/tilcdc_drv.c
@@ -171,8 +171,7 @@ static void tilcdc_fini(struct drm_device *dev)
if (priv->crtc)
tilcdc_crtc_shutdown(priv->crtc);
- if (priv->is_registered)
- drm_dev_unregister(dev);
+ drm_dev_unregister(dev);
drm_kms_helper_poll_fini(dev);
drm_atomic_helper_shutdown(dev);
@@ -219,21 +218,21 @@ static int tilcdc_init(const struct drm_driver *ddrv, struct device *dev)
priv->wq = alloc_ordered_workqueue("tilcdc", 0);
if (!priv->wq) {
ret = -ENOMEM;
- goto init_failed;
+ goto put_drm;
}
priv->mmio = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(priv->mmio)) {
dev_err(dev, "failed to request / ioremap\n");
ret = PTR_ERR(priv->mmio);
- goto init_failed;
+ goto free_wq;
}
priv->clk = clk_get(dev, "fck");
if (IS_ERR(priv->clk)) {
dev_err(dev, "failed to get functional clock\n");
ret = -ENODEV;
- goto init_failed;
+ goto free_wq;
}
pm_runtime_enable(dev);
@@ -312,7 +311,7 @@ static int tilcdc_init(const struct drm_driver *ddrv, struct device *dev)
ret = tilcdc_crtc_create(ddev);
if (ret < 0) {
dev_err(dev, "failed to create crtc\n");
- goto init_failed;
+ goto disable_pm;
}
modeset_init(ddev);
@@ -323,46 +322,46 @@ static int tilcdc_init(const struct drm_driver *ddrv, struct device *dev)
if (ret) {
dev_err(dev, "failed to register cpufreq notifier\n");
priv->freq_transition.notifier_call = NULL;
- goto init_failed;
+ goto destroy_crtc;
}
#endif
if (priv->is_componentized) {
ret = component_bind_all(dev, ddev);
if (ret < 0)
- goto init_failed;
+ goto unregister_cpufreq_notif;
ret = tilcdc_add_component_encoder(ddev);
if (ret < 0)
- goto init_failed;
+ goto unbind_component;
} else {
ret = tilcdc_attach_external_device(ddev);
if (ret)
- goto init_failed;
+ goto unregister_cpufreq_notif;
}
if (!priv->external_connector &&
((priv->num_encoders == 0) || (priv->num_connectors == 0))) {
dev_err(dev, "no encoders/connectors found\n");
ret = -EPROBE_DEFER;
- goto init_failed;
+ goto unbind_component;
}
ret = drm_vblank_init(ddev, 1);
if (ret < 0) {
dev_err(dev, "failed to initialize vblank\n");
- goto init_failed;
+ goto unbind_component;
}
ret = platform_get_irq(pdev, 0);
if (ret < 0)
- goto init_failed;
+ goto unbind_component;
priv->irq = ret;
ret = tilcdc_irq_install(ddev, priv->irq);
if (ret < 0) {
dev_err(dev, "failed to install IRQ handler\n");
- goto init_failed;
+ goto unbind_component;
}
drm_mode_config_reset(ddev);
@@ -371,15 +370,33 @@ static int tilcdc_init(const struct drm_driver *ddrv, struct device *dev)
ret = drm_dev_register(ddev, 0);
if (ret)
- goto init_failed;
- priv->is_registered = true;
+ goto stop_poll;
drm_fbdev_dma_setup(ddev, bpp);
return 0;
-init_failed:
- tilcdc_fini(ddev);
+stop_poll:
+ drm_kms_helper_poll_fini(ddev);
+ tilcdc_irq_uninstall(ddev);
+unbind_component:
+ if (priv->is_componentized)
+ component_unbind_all(dev, ddev);
+unregister_cpufreq_notif:
+#ifdef CONFIG_CPU_FREQ
+ cpufreq_unregister_notifier(&priv->freq_transition,
+ CPUFREQ_TRANSITION_NOTIFIER);
+destroy_crtc:
+#endif
+ tilcdc_crtc_destroy(priv->crtc);
+disable_pm:
+ pm_runtime_disable(dev);
+ clk_put(priv->clk);
+free_wq:
+ destroy_workqueue(priv->wq);
+put_drm:
platform_set_drvdata(pdev, NULL);
+ ddev->dev_private = NULL;
+ drm_dev_put(ddev);
return ret;
}
diff --git a/drivers/gpu/drm/tilcdc/tilcdc_drv.h b/drivers/gpu/drm/tilcdc/tilcdc_drv.h
index b818448c83f6..58b276f82a66 100644
--- a/drivers/gpu/drm/tilcdc/tilcdc_drv.h
+++ b/drivers/gpu/drm/tilcdc/tilcdc_drv.h
@@ -82,7 +82,6 @@ struct tilcdc_drm_private {
struct drm_encoder *external_encoder;
struct drm_connector *external_connector;
- bool is_registered;
bool is_componentized;
bool irq_enabled;
};
@@ -164,6 +163,7 @@ void tilcdc_crtc_set_panel_info(struct drm_crtc *crtc,
void tilcdc_crtc_set_simulate_vesa_sync(struct drm_crtc *crtc,
bool simulate_vesa_sync);
void tilcdc_crtc_shutdown(struct drm_crtc *crtc);
+void tilcdc_crtc_destroy(struct drm_crtc *crtc);
int tilcdc_crtc_update_fb(struct drm_crtc *crtc,
struct drm_framebuffer *fb,
struct drm_pending_vblank_event *event);
diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c
index 5bc656509ee9..70bd566cf6d4 100644
--- a/drivers/gpu/drm/v3d/v3d_sched.c
+++ b/drivers/gpu/drm/v3d/v3d_sched.c
@@ -865,67 +865,54 @@ static const struct drm_sched_backend_ops v3d_cpu_sched_ops = {
.free_job = v3d_sched_job_free
};
+static int
+v3d_queue_sched_init(struct v3d_dev *v3d, const struct drm_sched_backend_ops *ops,
+ enum v3d_queue queue, const char *name)
+{
+ struct drm_sched_init_args args = {
+ .num_rqs = DRM_SCHED_PRIORITY_COUNT,
+ .credit_limit = 1,
+ .timeout = msecs_to_jiffies(500),
+ .dev = v3d->drm.dev,
+ };
+
+ args.ops = ops;
+ args.name = name;
+
+ return drm_sched_init(&v3d->queue[queue].sched, &args);
+}
+
int
v3d_sched_init(struct v3d_dev *v3d)
{
- int hw_jobs_limit = 1;
- int job_hang_limit = 0;
- int hang_limit_ms = 500;
int ret;
- ret = drm_sched_init(&v3d->queue[V3D_BIN].sched,
- &v3d_bin_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_bin", v3d->drm.dev);
+ ret = v3d_queue_sched_init(v3d, &v3d_bin_sched_ops, V3D_BIN, "v3d_bin");
if (ret)
return ret;
- ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched,
- &v3d_render_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_render", v3d->drm.dev);
+ ret = v3d_queue_sched_init(v3d, &v3d_render_sched_ops, V3D_RENDER,
+ "v3d_render");
if (ret)
goto fail;
- ret = drm_sched_init(&v3d->queue[V3D_TFU].sched,
- &v3d_tfu_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_tfu", v3d->drm.dev);
+ ret = v3d_queue_sched_init(v3d, &v3d_tfu_sched_ops, V3D_TFU, "v3d_tfu");
if (ret)
goto fail;
if (v3d_has_csd(v3d)) {
- ret = drm_sched_init(&v3d->queue[V3D_CSD].sched,
- &v3d_csd_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_csd", v3d->drm.dev);
+ ret = v3d_queue_sched_init(v3d, &v3d_csd_sched_ops, V3D_CSD,
+ "v3d_csd");
if (ret)
goto fail;
- ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched,
- &v3d_cache_clean_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_cache_clean", v3d->drm.dev);
+ ret = v3d_queue_sched_init(v3d, &v3d_cache_clean_sched_ops,
+ V3D_CACHE_CLEAN, "v3d_cache_clean");
if (ret)
goto fail;
}
- ret = drm_sched_init(&v3d->queue[V3D_CPU].sched,
- &v3d_cpu_sched_ops, NULL,
- DRM_SCHED_PRIORITY_COUNT,
- 1, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_cpu", v3d->drm.dev);
+ ret = v3d_queue_sched_init(v3d, &v3d_cpu_sched_ops, V3D_CPU, "v3d_cpu");
if (ret)
goto fail;
diff --git a/drivers/gpu/drm/vc4/vc4_bo.c b/drivers/gpu/drm/vc4/vc4_bo.c
index 212c86acca48..57935b5c640a 100644
--- a/drivers/gpu/drm/vc4/vc4_bo.c
+++ b/drivers/gpu/drm/vc4/vc4_bo.c
@@ -1050,7 +1050,7 @@ static void vc4_bo_cache_destroy(struct drm_device *dev, void *unused)
struct vc4_dev *vc4 = to_vc4_dev(dev);
int i;
- del_timer(&vc4->bo_cache.time_timer);
+ timer_shutdown_sync(&vc4->bo_cache.time_timer);
cancel_work_sync(&vc4->bo_cache.time_work);
vc4_bo_cache_purge(dev);
diff --git a/drivers/gpu/drm/vc4/vc4_drv.c b/drivers/gpu/drm/vc4/vc4_drv.c
index 8c104ace3dc6..871d4096b74a 100644
--- a/drivers/gpu/drm/vc4/vc4_drv.c
+++ b/drivers/gpu/drm/vc4/vc4_drv.c
@@ -20,6 +20,7 @@
* driver.
*/
+#include <linux/aperture.h>
#include <linux/clk.h>
#include <linux/component.h>
#include <linux/device.h>
@@ -30,7 +31,6 @@
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
-#include <drm/drm_aperture.h>
#include <drm/drm_atomic_helper.h>
#include <drm/drm_drv.h>
#include <drm/drm_fbdev_dma.h>
@@ -356,9 +356,12 @@ static int vc4_drm_bind(struct device *dev)
}
}
- ret = drm_aperture_remove_framebuffers(driver);
- if (ret)
+ ret = aperture_remove_all_conflicting_devices(driver->name);
+ if (ret) {
+ if (firmware)
+ rpi_firmware_put(firmware);
goto err;
+ }
if (firmware) {
ret = rpi_firmware_property(firmware,
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h
index ca0f7dcc99a2..a597c73a2bf1 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -43,6 +43,8 @@
#include <drm/drm_probe_helper.h>
#include <drm/virtgpu_drm.h>
+#include <xen/xen.h>
+
#define DRIVER_NAME "virtio_gpu"
#define DRIVER_DESC "virtio GPU"
#define DRIVER_DATE "0"
@@ -61,6 +63,24 @@
/* See virtio_gpu_ctx_create. One additional character for NULL terminator. */
#define DEBUG_NAME_MAX_LEN 65
+/*
+ * Whether the host must be told about resource backing pages by DMA address
+ * rather than guest-physical address.
+ *
+ * This mirrors vring_use_map_api() in drivers/virtio/virtio_ring.c, including
+ * its xen_domain() case.
+ */
+static inline bool virtio_gpu_use_dma_api(const struct virtio_device *vdev)
+{
+ if (!virtio_has_dma_quirk(vdev))
+ return true;
+
+ if (xen_domain())
+ return true;
+
+ return false;
+}
+
struct virtio_gpu_object_params {
unsigned long size;
bool dumb;
@@ -89,6 +109,7 @@ struct virtio_gpu_object_params {
struct virtio_gpu_object {
struct drm_gem_shmem_object base;
+ struct sg_table *sgt;
uint32_t hw_res_handle;
bool dumb;
bool created;
@@ -476,6 +497,10 @@ struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
struct drm_gem_object *virtgpu_gem_prime_import_sg_table(
struct drm_device *dev, struct dma_buf_attachment *attach,
struct sg_table *sgt);
+int virtgpu_dma_buf_import_sgt(struct virtio_gpu_mem_entry **ents,
+ unsigned int *nents,
+ struct virtio_gpu_object *bo,
+ struct dma_buf_attachment *attach);
/* virtgpu_debugfs.c */
void virtio_gpu_debugfs_init(struct drm_minor *minor);
diff --git a/drivers/gpu/drm/virtio/virtgpu_object.c b/drivers/gpu/drm/virtio/virtgpu_object.c
index c7e74cf13022..c4f8cdf85f60 100644
--- a/drivers/gpu/drm/virtio/virtgpu_object.c
+++ b/drivers/gpu/drm/virtio/virtgpu_object.c
@@ -137,7 +137,7 @@ static int virtio_gpu_object_shmem_init(struct virtio_gpu_device *vgdev,
struct virtio_gpu_mem_entry **ents,
unsigned int *nents)
{
- bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev);
+ bool use_dma_api = virtio_gpu_use_dma_api(vgdev->vdev);
struct scatterlist *sg;
struct sg_table *pages;
int si;
diff --git a/drivers/gpu/drm/virtio/virtgpu_plane.c b/drivers/gpu/drm/virtio/virtgpu_plane.c
index 7a091beddaef..d59f11b78186 100644
--- a/drivers/gpu/drm/virtio/virtgpu_plane.c
+++ b/drivers/gpu/drm/virtio/virtgpu_plane.c
@@ -26,6 +26,7 @@
#include <drm/drm_atomic_helper.h>
#include <drm/drm_damage_helper.h>
#include <drm/drm_fourcc.h>
+#include <drm/drm_gem_atomic_helper.h>
#include "virtgpu_drv.h"
@@ -276,6 +277,9 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,
vgfb = to_virtio_gpu_framebuffer(new_state->fb);
vgplane_st = to_virtio_gpu_plane_state(new_state);
bo = gem_to_virtio_gpu_obj(vgfb->base.obj[0]);
+
+ drm_gem_plane_helper_prepare_fb(plane, new_state);
+
if (!bo || (plane->type == DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob))
return 0;
diff --git a/drivers/gpu/drm/virtio/virtgpu_prime.c b/drivers/gpu/drm/virtio/virtgpu_prime.c
index 44425f20d91a..8e63aa67270a 100644
--- a/drivers/gpu/drm/virtio/virtgpu_prime.c
+++ b/drivers/gpu/drm/virtio/virtgpu_prime.c
@@ -27,6 +27,8 @@
#include "virtgpu_drv.h"
+MODULE_IMPORT_NS(DMA_BUF);
+
static int virtgpu_virtio_get_uuid(struct dma_buf *buf,
uuid_t *uuid)
{
@@ -142,6 +144,46 @@ struct dma_buf *virtgpu_gem_prime_export(struct drm_gem_object *obj,
return buf;
}
+int virtgpu_dma_buf_import_sgt(struct virtio_gpu_mem_entry **ents,
+ unsigned int *nents,
+ struct virtio_gpu_object *bo,
+ struct dma_buf_attachment *attach)
+{
+ struct scatterlist *sl;
+ struct sg_table *sgt;
+ long i, ret;
+
+ dma_resv_assert_held(attach->dmabuf->resv);
+
+ ret = dma_resv_wait_timeout(attach->dmabuf->resv,
+ DMA_RESV_USAGE_KERNEL,
+ false, MAX_SCHEDULE_TIMEOUT);
+ if (ret <= 0)
+ return ret < 0 ? ret : -ETIMEDOUT;
+
+ sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
+ if (IS_ERR(sgt))
+ return PTR_ERR(sgt);
+
+ *ents = kvmalloc_array(sgt->nents,
+ sizeof(struct virtio_gpu_mem_entry),
+ GFP_KERNEL);
+ if (!(*ents)) {
+ dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
+ return -ENOMEM;
+ }
+
+ *nents = sgt->nents;
+ for_each_sgtable_dma_sg(sgt, sl, i) {
+ (*ents)[i].addr = cpu_to_le64(sg_dma_address(sl));
+ (*ents)[i].length = cpu_to_le32(sg_dma_len(sl));
+ (*ents)[i].padding = 0;
+ }
+
+ bo->sgt = sgt;
+ return 0;
+}
+
struct drm_gem_object *virtgpu_gem_prime_import(struct drm_device *dev,
struct dma_buf *buf)
{
diff --git a/drivers/gpu/drm/virtio/virtgpu_vq.c b/drivers/gpu/drm/virtio/virtgpu_vq.c
index 4369275cb56b..9bf7e69b85aa 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vq.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vq.c
@@ -601,7 +601,7 @@ void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(objs->objs[0]);
struct virtio_gpu_transfer_to_host_2d *cmd_p;
struct virtio_gpu_vbuffer *vbuf;
- bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev);
+ bool use_dma_api = virtio_gpu_use_dma_api(vgdev->vdev);
if (virtio_gpu_is_shmem(bo) && use_dma_api)
dma_sync_sgtable_for_device(vgdev->vdev->dev.parent,
@@ -1020,7 +1020,7 @@ void virtio_gpu_cmd_transfer_to_host_3d(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(objs->objs[0]);
struct virtio_gpu_transfer_host_3d *cmd_p;
struct virtio_gpu_vbuffer *vbuf;
- bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev);
+ bool use_dma_api = virtio_gpu_use_dma_api(vgdev->vdev);
if (virtio_gpu_is_shmem(bo) && use_dma_api)
dma_sync_sgtable_for_device(vgdev->vdev->dev.parent,
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
index 2e84e1029732..1b0b9d2d4e72 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c
@@ -856,8 +856,6 @@ static int vmw_driver_load(struct vmw_private *dev_priv, u32 pci_id)
bool refuse_dma = false;
struct pci_dev *pdev = to_pci_dev(dev_priv->drm.dev);
- dev_priv->drm.dev_private = dev_priv;
-
vmw_sw_context_init(dev_priv);
mutex_init(&dev_priv->cmdbuf_mutex);
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.h b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.h
index 4e2807f5f94c..b21831ef214a 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.h
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.h
@@ -639,7 +639,7 @@ static inline struct vmw_surface *vmw_res_to_srf(struct vmw_resource *res)
static inline struct vmw_private *vmw_priv(struct drm_device *dev)
{
- return (struct vmw_private *)dev->dev_private;
+ return container_of(dev, struct vmw_private, drm);
}
static inline struct vmw_private *vmw_priv_from_ttm(struct ttm_device *bdev)
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
index 5f99f7437ae6..985774f22a13 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
@@ -280,7 +280,7 @@ static void vmw_du_put_cursor_mob(struct vmw_cursor_plane *vcp,
static int vmw_du_get_cursor_mob(struct vmw_cursor_plane *vcp,
struct vmw_plane_state *vps)
{
- struct vmw_private *dev_priv = vcp->base.dev->dev_private;
+ struct vmw_private *dev_priv = vmw_priv(vcp->base.dev);
u32 size = vmw_du_cursor_mob_size(vps->base.crtc_w, vps->base.crtc_h);
u32 i;
u32 cursor_max_dim, mob_max_size;
@@ -382,6 +382,7 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
unsigned long kmap_num;
SVGA3dCopyBox *box;
unsigned box_count;
+ u64 src_extent;
void *virtual;
bool is_iomem;
struct vmw_dma_cmd {
@@ -393,6 +394,8 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
vmw_surface_get_desc(VMW_CURSOR_SNOOP_FORMAT);
const u32 image_pitch = VMW_CURSOR_SNOOP_WIDTH * desc->pitchBytesPerBlock;
+ BUILD_BUG_ON(sizeof(*srf->snooper.image) != desc->pitchBytesPerBlock);
+
cmd = container_of(header, struct vmw_dma_cmd, header);
/* No snooper installed, nothing to copy */
@@ -430,8 +433,19 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
return;
}
+ if (box->w == 0 || box->h == 0)
+ return;
+
+ src_extent = (u64)(box->h - 1) * cmd->dma.guest.pitch +
+ (u64)box->w * desc->pitchBytesPerBlock;
+ if (src_extent > bo->base.size) {
+ DRM_ERROR("Cursor snoop source of %llu bytes exceeds the %zu byte buffer\n",
+ src_extent, bo->base.size);
+ return;
+ }
+
kmap_offset = cmd->dma.guest.ptr.offset >> PAGE_SHIFT;
- kmap_num = (VMW_CURSOR_SNOOP_HEIGHT*image_pitch) >> PAGE_SHIFT;
+ kmap_num = PFN_UP(src_extent);
ret = ttm_bo_reserve(bo, true, false, NULL);
if (unlikely(ret != 0)) {
@@ -445,14 +459,16 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
virtual = ttm_kmap_obj_virtual(&map, &is_iomem);
- if (box->w == VMW_CURSOR_SNOOP_WIDTH && cmd->dma.guest.pitch == image_pitch) {
+ if (box->w == VMW_CURSOR_SNOOP_WIDTH &&
+ box->h == VMW_CURSOR_SNOOP_HEIGHT &&
+ cmd->dma.guest.pitch == image_pitch) {
memcpy(srf->snooper.image, virtual,
VMW_CURSOR_SNOOP_HEIGHT*image_pitch);
} else {
- /* Image is unsigned pointer. */
+ /* The destination stride is in pixels. */
for (i = 0; i < box->h; i++)
- memcpy(srf->snooper.image + i * image_pitch,
- virtual + i * cmd->dma.guest.pitch,
+ memcpy(srf->snooper.image + i * VMW_CURSOR_SNOOP_WIDTH,
+ virtual + (size_t)i * cmd->dma.guest.pitch,
box->w * desc->pitchBytesPerBlock);
}
@@ -519,7 +535,7 @@ void vmw_du_cursor_plane_destroy(struct drm_plane *plane)
struct vmw_cursor_plane *vcp = vmw_plane_to_vcp(plane);
u32 i;
- vmw_cursor_update_position(plane->dev->dev_private, false, 0, 0);
+ vmw_cursor_update_position(vmw_priv(plane->dev), false, 0, 0);
for (i = 0; i < ARRAY_SIZE(vcp->cursor_mobs); i++)
vmw_du_destroy_cursor_mob(&vcp->cursor_mobs[i]);
diff --git a/drivers/gpu/drm/xe/xe_execlist.c b/drivers/gpu/drm/xe/xe_execlist.c
index 6a59165b9569..99bfb50df1ca 100644
--- a/drivers/gpu/drm/xe/xe_execlist.c
+++ b/drivers/gpu/drm/xe/xe_execlist.c
@@ -331,6 +331,15 @@ static const struct drm_sched_backend_ops drm_sched_ops = {
static int execlist_exec_queue_init(struct xe_exec_queue *q)
{
struct drm_gpu_scheduler *sched;
+ const struct drm_sched_init_args args = {
+ .ops = &drm_sched_ops,
+ .num_rqs = 1,
+ .credit_limit = q->lrc[0]->ring.size / MAX_JOB_SIZE_BYTES,
+ .hang_limit = XE_SCHED_HANG_LIMIT,
+ .timeout = XE_SCHED_JOB_TIMEOUT,
+ .name = q->hwe->name,
+ .dev = gt_to_xe(q->gt)->drm.dev,
+ };
struct xe_execlist_exec_queue *exl;
struct xe_device *xe = gt_to_xe(q->gt);
int err;
@@ -345,11 +354,7 @@ static int execlist_exec_queue_init(struct xe_exec_queue *q)
exl->q = q;
- err = drm_sched_init(&exl->sched, &drm_sched_ops, NULL, 1,
- q->lrc[0]->ring.size / MAX_JOB_SIZE_BYTES,
- XE_SCHED_HANG_LIMIT, XE_SCHED_JOB_TIMEOUT,
- NULL, NULL, q->hwe->name,
- gt_to_xe(q->gt)->drm.dev);
+ err = drm_sched_init(&exl->sched, &args);
if (err)
goto err_free;
diff --git a/drivers/gpu/drm/xe/xe_gpu_scheduler.c b/drivers/gpu/drm/xe/xe_gpu_scheduler.c
index 50361b4638f9..869b43a4151d 100644
--- a/drivers/gpu/drm/xe/xe_gpu_scheduler.c
+++ b/drivers/gpu/drm/xe/xe_gpu_scheduler.c
@@ -63,13 +63,24 @@ int xe_sched_init(struct xe_gpu_scheduler *sched,
atomic_t *score, const char *name,
struct device *dev)
{
+ const struct drm_sched_init_args args = {
+ .ops = ops,
+ .submit_wq = submit_wq,
+ .num_rqs = 1,
+ .credit_limit = hw_submission,
+ .hang_limit = hang_limit,
+ .timeout = timeout,
+ .timeout_wq = timeout_wq,
+ .score = score,
+ .name = name,
+ .dev = dev,
+ };
+
sched->ops = xe_ops;
INIT_LIST_HEAD(&sched->msgs);
INIT_WORK(&sched->work_process_msg, xe_sched_process_msg_work);
- return drm_sched_init(&sched->base, ops, submit_wq, 1, hw_submission,
- hang_limit, timeout, timeout_wq, score, name,
- dev);
+ return drm_sched_init(&sched->base, &args);
}
void xe_sched_fini(struct xe_gpu_scheduler *sched)
diff --git a/drivers/gpu/drm/xe/xe_guc.c b/drivers/gpu/drm/xe/xe_guc.c
index bedd9894e182..d4c8a782e7e5 100644
--- a/drivers/gpu/drm/xe/xe_guc.c
+++ b/drivers/gpu/drm/xe/xe_guc.c
@@ -289,10 +289,6 @@ static int xe_guc_realloc_post_hwconfig(struct xe_guc *guc)
if (ret)
return ret;
- ret = xe_managed_bo_reinit_in_vram(xe, tile, &guc->ct.bo);
- if (ret)
- return ret;
-
return 0;
}
@@ -397,6 +393,10 @@ int xe_guc_init_post_hwconfig(struct xe_guc *guc)
if (ret)
return ret;
+ ret = xe_guc_ct_init_post_hwconfig(&guc->ct);
+ if (ret)
+ return ret;
+
guc_init_params_post_hwconfig(guc);
ret = xe_guc_submit_init(guc, ~0);
diff --git a/drivers/gpu/drm/xe/xe_guc_ct.c b/drivers/gpu/drm/xe/xe_guc_ct.c
index 7619b48dbfe4..e9da1b478cb3 100644
--- a/drivers/gpu/drm/xe/xe_guc_ct.c
+++ b/drivers/gpu/drm/xe/xe_guc_ct.c
@@ -36,6 +36,8 @@ static void receive_g2h(struct xe_guc_ct *ct);
static void g2h_worker_func(struct work_struct *w);
static void safe_mode_worker_func(struct work_struct *w);
static void ct_exit_safe_mode(struct xe_guc_ct *ct);
+static void xe_guc_ct_set_state(struct xe_guc_ct *ct,
+ enum xe_guc_ct_state state);
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
enum {
@@ -206,6 +208,13 @@ static void primelockdep(struct xe_guc_ct *ct)
fs_reclaim_release(GFP_KERNEL);
}
+static void guc_action_disable_ct(void *arg)
+{
+ struct xe_guc_ct *ct = arg;
+
+ xe_guc_ct_set_state(ct, XE_GUC_CT_STATE_DISABLED);
+}
+
int xe_guc_ct_init(struct xe_guc_ct *ct)
{
struct xe_device *xe = ct_to_xe(ct);
@@ -252,7 +261,37 @@ int xe_guc_ct_init(struct xe_guc_ct *ct)
xe_gt_assert(gt, ct->state == XE_GUC_CT_STATE_NOT_INITIALIZED);
ct->state = XE_GUC_CT_STATE_DISABLED;
- return 0;
+
+ return devm_add_action_or_reset(xe->drm.dev, guc_action_disable_ct, ct);
+}
+
+/**
+ * xe_guc_ct_init_post_hwconfig - Reinitialize the GuC CTB in VRAM
+ * @ct: the &xe_guc_ct
+ *
+ * Allocate a new BO in VRAM and free the previous BO that was allocated
+ * in system memory (SMEM). Applicable only for DGFX products.
+ *
+ * Return: 0 on success, or a negative errno on failure.
+ */
+int xe_guc_ct_init_post_hwconfig(struct xe_guc_ct *ct)
+{
+ struct xe_device *xe = ct_to_xe(ct);
+ struct xe_gt *gt = ct_to_gt(ct);
+ struct xe_tile *tile = gt_to_tile(gt);
+ int ret;
+
+ xe_assert(xe, !xe_guc_ct_enabled(ct));
+
+ if (!IS_DGFX(xe))
+ return 0;
+
+ ret = xe_managed_bo_reinit_in_vram(xe, tile, &ct->bo);
+ if (ret)
+ return ret;
+
+ devm_release_action(xe->drm.dev, guc_action_disable_ct, ct);
+ return devm_add_action_or_reset(xe->drm.dev, guc_action_disable_ct, ct);
}
#define desc_read(xe_, guc_ctb__, field_) \
diff --git a/drivers/gpu/drm/xe/xe_guc_ct.h b/drivers/gpu/drm/xe/xe_guc_ct.h
index c7ac9407b861..398e3adf49c7 100644
--- a/drivers/gpu/drm/xe/xe_guc_ct.h
+++ b/drivers/gpu/drm/xe/xe_guc_ct.h
@@ -11,6 +11,7 @@
struct drm_printer;
int xe_guc_ct_init(struct xe_guc_ct *ct);
+int xe_guc_ct_init_post_hwconfig(struct xe_guc_ct *ct);
int xe_guc_ct_enable(struct xe_guc_ct *ct);
void xe_guc_ct_disable(struct xe_guc_ct *ct);
void xe_guc_ct_stop(struct xe_guc_ct *ct);
diff --git a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
index a3f421e2adc0..5282d0eb9474 100644
--- a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
+++ b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
@@ -10,6 +10,7 @@
#include <linux/workqueue.h>
#include "xe_gpu_scheduler_types.h"
+#include "xe_hw_fence_types.h"
struct dma_fence;
struct xe_exec_queue;
@@ -24,6 +25,10 @@ struct xe_guc_exec_queue {
struct rcu_head rcu;
/** @sched: GPU scheduler for this xe_exec_queue */
struct xe_gpu_scheduler sched;
+ /**
+ * @name: Scheduler timeline name, kept with @sched until RCU free.
+ */
+ char name[MAX_FENCE_NAME_LEN];
/** @entity: Scheduler entity for this xe_exec_queue */
struct xe_sched_entity entity;
/**
diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
index 51bd4a0f2a18..ed95176b0e1e 100644
--- a/drivers/gpu/drm/xe/xe_guc_submit.c
+++ b/drivers/gpu/drm/xe/xe_guc_submit.c
@@ -1472,7 +1472,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
err = xe_sched_init(&ge->sched, &drm_sched_ops, &xe_sched_ops,
NULL, q->lrc[0]->ring.size / MAX_JOB_SIZE_BYTES, 64,
timeout, guc_to_gt(guc)->ordered_wq, NULL,
- q->name, gt_to_xe(q->gt)->drm.dev);
+ ge->name, gt_to_xe(q->gt)->drm.dev);
if (err)
goto err_free;
@@ -1499,6 +1499,8 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
xe_exec_queue_assign_name(q, q->guc->id);
+ strscpy(ge->name, q->name, sizeof(ge->name));
+
trace_xe_exec_queue_create(q);
return 0;
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index fa0d7c641402..da148df42105 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -2469,7 +2469,7 @@ static void vm_bind_ioctl_ops_unwind(struct xe_vm *vm,
}
static int vma_lock_and_validate(struct drm_exec *exec, struct xe_vma *vma,
- bool validate)
+ bool res_evict, bool validate)
{
struct xe_bo *bo = xe_vma_bo(vma);
int err = 0;
@@ -2478,7 +2478,7 @@ static int vma_lock_and_validate(struct drm_exec *exec, struct xe_vma *vma,
if (!bo->vm)
err = drm_exec_lock_obj(exec, &bo->ttm.base);
if (!err && validate)
- err = xe_bo_validate(bo, xe_vma_vm(vma), true);
+ err = xe_bo_validate(bo, xe_vma_vm(vma), res_evict);
}
return err;
@@ -2500,13 +2500,22 @@ static int check_ufence(struct xe_vma *vma)
}
static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,
- struct xe_vma_op *op)
+ struct xe_vma_ops *vops, struct xe_vma_op *op)
{
int err = 0;
+ bool res_evict;
+
+ /*
+ * We only allow evicting a BO within the VM if it is not part of an
+ * array of binds, as an array of binds can evict another BO within the
+ * bind.
+ */
+ res_evict = !(vops->flags & XE_VMA_OPS_ARRAY_OF_BINDS);
switch (op->base.op) {
case DRM_GPUVA_OP_MAP:
err = vma_lock_and_validate(exec, op->map.vma,
+ res_evict,
!xe_vm_in_fault_mode(vm) ||
op->map.immediate);
break;
@@ -2517,11 +2526,13 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,
err = vma_lock_and_validate(exec,
gpuva_to_vma(op->base.remap.unmap->va),
- false);
+ res_evict, false);
if (!err && op->remap.prev)
- err = vma_lock_and_validate(exec, op->remap.prev, true);
+ err = vma_lock_and_validate(exec, op->remap.prev,
+ res_evict, true);
if (!err && op->remap.next)
- err = vma_lock_and_validate(exec, op->remap.next, true);
+ err = vma_lock_and_validate(exec, op->remap.next,
+ res_evict, true);
break;
case DRM_GPUVA_OP_UNMAP:
err = check_ufence(gpuva_to_vma(op->base.unmap.va));
@@ -2530,7 +2541,7 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,
err = vma_lock_and_validate(exec,
gpuva_to_vma(op->base.unmap.va),
- false);
+ res_evict, false);
break;
case DRM_GPUVA_OP_PREFETCH:
{
@@ -2541,7 +2552,7 @@ static int op_lock_and_prep(struct drm_exec *exec, struct xe_vm *vm,
err = vma_lock_and_validate(exec,
gpuva_to_vma(op->base.prefetch.va),
- false);
+ res_evict, false);
if (!err && !xe_vma_has_no_bo(vma))
err = xe_bo_migrate(xe_vma_bo(vma),
region_to_mem_type[region]);
@@ -2566,7 +2577,7 @@ static int vm_bind_ioctl_ops_lock_and_prep(struct drm_exec *exec,
return err;
list_for_each_entry(op, &vops->list, link) {
- err = op_lock_and_prep(exec, vm, op);
+ err = op_lock_and_prep(exec, vm, vops, op);
if (err)
return err;
}
@@ -3163,6 +3174,8 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
}
xe_vma_ops_init(&vops, vm, q, syncs, num_syncs);
+ if (args->num_binds > 1)
+ vops.flags |= XE_VMA_OPS_ARRAY_OF_BINDS;
for (i = 0; i < args->num_binds; ++i) {
u64 range = bind_ops[i].range;
u64 addr = bind_ops[i].addr;
diff --git a/drivers/gpu/drm/xe/xe_vm_types.h b/drivers/gpu/drm/xe/xe_vm_types.h
index b280e335c5bb..e75dc5044616 100644
--- a/drivers/gpu/drm/xe/xe_vm_types.h
+++ b/drivers/gpu/drm/xe/xe_vm_types.h
@@ -377,6 +377,9 @@ struct xe_vma_ops {
u32 num_syncs;
/** @pt_update_ops: page table update operations */
struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE];
+ /** @flags: signify the properties within xe_vma_ops */
+#define XE_VMA_OPS_ARRAY_OF_BINDS BIT(2)
+ u32 flags;
#ifdef TEST_VM_OPS_ERROR
/** @inject_error: inject error to test error handling */
bool inject_error;
diff --git a/drivers/hid/hid-core.c b/drivers/hid/hid-core.c
index df2e43b27388..52eda362bb1a 100644
--- a/drivers/hid/hid-core.c
+++ b/drivers/hid/hid-core.c
@@ -670,7 +670,11 @@ static int hid_parser_main(struct hid_parser *parser, struct hid_item *item)
ret = hid_add_field(parser, HID_FEATURE_REPORT, data);
break;
default:
- hid_warn(parser->device, "unknown main item tag 0x%x\n", item->tag);
+ if (item->tag >= HID_MAIN_ITEM_TAG_RESERVED_MIN &&
+ item->tag <= HID_MAIN_ITEM_TAG_RESERVED_MAX)
+ hid_warn_ratelimited(parser->device, "reserved main item tag 0x%x\n", item->tag);
+ else
+ hid_warn_ratelimited(parser->device, "unknown main item tag 0x%x\n", item->tag);
ret = 0;
}
@@ -2057,9 +2061,9 @@ int hid_report_raw_event(struct hid_device *hid, enum hid_report_type type, u8 *
}
if (csize < rsize) {
- dbg_hid("report %d is too short, (%d < %d)\n", report->id,
- csize, rsize);
- memset(cdata + csize, 0, rsize - csize);
+ hid_warn_ratelimited(hid, "Event data for report %d was too short (%d vs %d)\n",
+ report->id, rsize, csize);
+ return -EINVAL;
}
if ((hid->claimed & HID_CLAIMED_HIDDEV) && hid->hiddev_report_event)
diff --git a/drivers/hid/hid-logitech-hidpp.c b/drivers/hid/hid-logitech-hidpp.c
index 9926ab6e808e..c84bd26faea9 100644
--- a/drivers/hid/hid-logitech-hidpp.c
+++ b/drivers/hid/hid-logitech-hidpp.c
@@ -2878,18 +2878,19 @@ static int hidpp_ff_init(struct hidpp_device *hidpp,
* ownership to FF core
*/
data = kmemdup(data, sizeof(*data), GFP_KERNEL);
- if (!data)
- return -ENOMEM;
+ if (!data) {
+ error = -ENOMEM;
+ goto err_destroy_ff;
+ }
data->effect_ids = kcalloc(num_slots, sizeof(int), GFP_KERNEL);
if (!data->effect_ids) {
- kfree(data);
- return -ENOMEM;
+ error = -ENOMEM;
+ goto err_free_data;
}
data->wq = create_singlethread_workqueue("hidpp-ff-sendqueue");
if (!data->wq) {
- kfree(data->effect_ids);
- kfree(data);
- return -ENOMEM;
+ error = -ENOMEM;
+ goto err_free_effect_ids;
}
data->hidpp = hidpp;
@@ -2919,6 +2920,14 @@ static int hidpp_ff_init(struct hidpp_device *hidpp,
version);
return 0;
+
+err_free_effect_ids:
+ kfree(data->effect_ids);
+err_free_data:
+ kfree(data);
+err_destroy_ff:
+ input_ff_destroy(dev);
+ return error;
}
/* ************************************************************************** */
diff --git a/drivers/hid/hid-steam.c b/drivers/hid/hid-steam.c
index 0f826acfbf27..4283219275a3 100644
--- a/drivers/hid/hid-steam.c
+++ b/drivers/hid/hid-steam.c
@@ -324,6 +324,13 @@ static int steam_recv_report(struct steam_device *steam,
u8 *buf;
int ret;
+ /*
+ * All reports start with a two byte header.
+ * We must read at least two bytes to get a sensible output.
+ */
+ if (size < 2)
+ return -EINVAL;
+
r = steam->hdev->report_enum[HID_FEATURE_REPORT].report_id_hash[0];
if (!r) {
hid_err(steam->hdev, "No HID_FEATURE_REPORT submitted - nothing to read\n");
@@ -346,10 +353,30 @@ static int steam_recv_report(struct steam_device *steam,
ret = hid_hw_raw_request(steam->hdev, 0x00,
buf, hid_report_len(r) + 1,
HID_FEATURE_REPORT, HID_REQ_GET_REPORT);
- if (ret > 0)
- memcpy(data, buf + 1, min(size, ret - 1));
+ if (ret > 0) {
+ /* Remove the report ID from the return buffer */
+ ret--;
+ size = min(size, ret);
+ memcpy(data, buf + 1, size);
+ }
kfree(buf);
- return ret;
+
+ if (ret < 0) {
+ hid_err(steam->hdev, "%s: error %d\n", __func__, ret);
+ return ret;
+ }
+
+ if (ret < 2) {
+ hid_err(steam->hdev, "%s: reply too short\n", __func__);
+ return -EPROTO;
+ }
+ if (ret < data[1] + 2) {
+ hid_err(steam->hdev, "%s: expected %u bytes, read %i\n",
+ __func__, data[1] + 2, ret);
+ return -EPROTO;
+ }
+ hid_dbg(steam->hdev, "Received report %*ph\n", size, data);
+ return size;
}
static int steam_send_report(struct steam_device *steam,
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index db1c71832f2a..1ddbbc80cf13 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -2776,7 +2776,7 @@ static void hv_crash_handler(struct pt_regs *regs)
hv_synic_disable_regs(cpu);
};
-static int hv_synic_suspend(void)
+static int hv_synic_suspend(void *data)
{
/*
* When we reach here, all the non-boot CPUs have been offlined.
@@ -2803,7 +2803,7 @@ static int hv_synic_suspend(void)
return 0;
}
-static void hv_synic_resume(void)
+static void hv_synic_resume(void *data)
{
hv_synic_enable_regs(0);
@@ -2815,11 +2815,15 @@ static void hv_synic_resume(void)
}
/* The callbacks run only on CPU0, with irqs_disabled. */
-static struct syscore_ops hv_synic_syscore_ops = {
+static const struct syscore_ops hv_synic_syscore_ops = {
.suspend = hv_synic_suspend,
.resume = hv_synic_resume,
};
+static struct syscore hv_synic_syscore = {
+ .ops = &hv_synic_syscore_ops,
+};
+
static int __init hv_acpi_init(void)
{
int ret;
@@ -2828,6 +2832,13 @@ static int __init hv_acpi_init(void)
return -ENODEV;
if (hv_root_partition && !hv_nested)
+ /*
+ * A non-nested root partition does not need VMBus client
+ * functionality. However, the mshv_root module may have
+ * a dependency on the VMBus module as described in
+ * commit 840b740a35bf. Return success so the module
+ * loads even though no VMBus initialization is done.
+ */
return 0;
/*
@@ -2862,7 +2873,7 @@ static int __init hv_acpi_init(void)
hv_setup_kexec_handler(hv_kexec_handler);
hv_setup_crash_handler(hv_crash_handler);
- register_syscore_ops(&hv_synic_syscore_ops);
+ register_syscore(&hv_synic_syscore);
return 0;
@@ -2876,7 +2887,15 @@ static void __exit vmbus_exit(void)
{
int cpu;
- unregister_syscore_ops(&hv_synic_syscore_ops);
+ if (hv_root_partition && !hv_nested)
+ /*
+ * If a non-nested root partition loaded the VMBus module,
+ * hv_acpi_init() did not do any VMBus initialization.
+ * There's nothing to clean up, so just return.
+ */
+ return;
+
+ unregister_syscore(&hv_synic_syscore);
hv_remove_kexec_handler();
hv_remove_crash_handler();
diff --git a/drivers/hwmon/asus_rog_ryujin.c b/drivers/hwmon/asus_rog_ryujin.c
index e5e93a20723c..593950802b39 100644
--- a/drivers/hwmon/asus_rog_ryujin.c
+++ b/drivers/hwmon/asus_rog_ryujin.c
@@ -18,15 +18,25 @@
#define USB_VENDOR_ID_ASUS_ROG 0x0b05
#define USB_PRODUCT_ID_RYUJIN_AIO 0x1988 /* ASUS ROG RYUJIN II 360 */
+struct rog_ryujin_device_info {
+ u8 temp_offset;
+ u8 pump_speed_offset;
+ u8 fan_speed_offset;
+ u8 duty_channel;
+ bool has_controller;
+};
+
+static const struct rog_ryujin_device_info rog_ryujin_ii_360_info = {
+ .temp_offset = 3,
+ .pump_speed_offset = 5,
+ .fan_speed_offset = 7,
+ .duty_channel = 0,
+ .has_controller = true,
+};
+
#define STATUS_VALIDITY 1500 /* ms */
#define MAX_REPORT_LENGTH 65
-/* Cooler status report offsets */
-#define RYUJIN_TEMP_SENSOR_1 3
-#define RYUJIN_TEMP_SENSOR_2 4
-#define RYUJIN_PUMP_SPEED 5
-#define RYUJIN_INTERNAL_FAN_SPEED 7
-
/* Cooler duty report offsets */
#define RYUJIN_PUMP_DUTY 4
#define RYUJIN_INTERNAL_FAN_DUTY 5
@@ -81,10 +91,7 @@ static const char *const rog_ryujin_speed_label[] = {
struct rog_ryujin_data {
struct hid_device *hdev;
struct device *hwmon_dev;
- /* For locking access to buffer */
- struct mutex buffer_lock;
- /* For queueing multiple readers */
- struct mutex status_report_request_mutex;
+ const struct rog_ryujin_device_info *info;
/* For reinitializing the completions below */
spinlock_t status_report_request_lock;
struct completion cooler_status_received;
@@ -116,6 +123,8 @@ static int rog_ryujin_pwm_to_percent(long val)
static umode_t rog_ryujin_is_visible(const void *data,
enum hwmon_sensor_types type, u32 attr, int channel)
{
+ const struct rog_ryujin_data *priv = data;
+
switch (type) {
case hwmon_temp:
switch (attr) {
@@ -127,6 +136,8 @@ static umode_t rog_ryujin_is_visible(const void *data,
}
break;
case hwmon_fan:
+ if (channel >= 2 && !priv->info->has_controller)
+ return 0;
switch (attr) {
case hwmon_fan_label:
case hwmon_fan_input:
@@ -136,6 +147,8 @@ static umode_t rog_ryujin_is_visible(const void *data,
}
break;
case hwmon_pwm:
+ if (channel >= 2 && !priv->info->has_controller)
+ return 0;
switch (attr) {
case hwmon_pwm_input:
return 0644;
@@ -153,18 +166,10 @@ static umode_t rog_ryujin_is_visible(const void *data,
/* Writes the command to the device with the rest of the report filled with zeroes */
static int rog_ryujin_write_expanded(struct rog_ryujin_data *priv, const u8 *cmd, int cmd_length)
{
- int ret;
-
- mutex_lock(&priv->buffer_lock);
-
memcpy_and_pad(priv->buffer, MAX_REPORT_LENGTH, cmd, cmd_length, 0x00);
- ret = hid_hw_output_report(priv->hdev, priv->buffer, MAX_REPORT_LENGTH);
-
- mutex_unlock(&priv->buffer_lock);
- return ret;
+ return hid_hw_output_report(priv->hdev, priv->buffer, MAX_REPORT_LENGTH);
}
-/* Assumes priv->status_report_request_mutex is locked */
static int rog_ryujin_execute_cmd(struct rog_ryujin_data *priv, const u8 *cmd, int cmd_length,
struct completion *status_completion)
{
@@ -196,14 +201,11 @@ static int rog_ryujin_execute_cmd(struct rog_ryujin_data *priv, const u8 *cmd, i
static int rog_ryujin_get_status(struct rog_ryujin_data *priv)
{
- int ret = mutex_lock_interruptible(&priv->status_report_request_mutex);
-
- if (ret < 0)
- return ret;
+ int ret;
if (!time_after(jiffies, priv->updated + msecs_to_jiffies(STATUS_VALIDITY))) {
/* Data is up to date */
- goto unlock_and_return;
+ return 0;
}
/* Retrieve cooler status */
@@ -211,36 +213,34 @@ static int rog_ryujin_get_status(struct rog_ryujin_data *priv)
rog_ryujin_execute_cmd(priv, get_cooler_status_cmd, GET_CMD_LENGTH,
&priv->cooler_status_received);
if (ret < 0)
- goto unlock_and_return;
+ return ret;
- /* Retrieve controller status (speeds) */
- ret =
- rog_ryujin_execute_cmd(priv, get_controller_speed_cmd, GET_CMD_LENGTH,
- &priv->controller_status_received);
- if (ret < 0)
- goto unlock_and_return;
+ if (priv->info->has_controller) {
+ /* Retrieve controller status (speeds) */
+ ret = rog_ryujin_execute_cmd(priv, get_controller_speed_cmd,
+ GET_CMD_LENGTH,
+ &priv->controller_status_received);
+ if (ret < 0)
+ return ret;
+ }
/* Retrieve cooler duty */
ret =
rog_ryujin_execute_cmd(priv, get_cooler_duty_cmd, GET_CMD_LENGTH,
&priv->cooler_duty_received);
if (ret < 0)
- goto unlock_and_return;
+ return ret;
- /* Retrieve controller duty */
- ret =
- rog_ryujin_execute_cmd(priv, get_controller_duty_cmd, GET_CMD_LENGTH,
- &priv->controller_duty_received);
- if (ret < 0)
- goto unlock_and_return;
+ if (priv->info->has_controller) {
+ /* Retrieve controller duty */
+ ret = rog_ryujin_execute_cmd(priv, get_controller_duty_cmd,
+ GET_CMD_LENGTH,
+ &priv->controller_duty_received);
+ if (ret < 0)
+ return ret;
+ }
priv->updated = jiffies;
-
-unlock_and_return:
- mutex_unlock(&priv->status_report_request_mutex);
- if (ret < 0)
- return ret;
-
return 0;
}
@@ -303,16 +303,14 @@ static int rog_ryujin_write_fixed_duty(struct rog_ryujin_data *priv, int channel
* Retrieve cooler duty since both pump and internal fan are set
* together, then write back with one of them modified.
*/
- ret = mutex_lock_interruptible(&priv->status_report_request_mutex);
- if (ret < 0)
- return ret;
ret =
rog_ryujin_execute_cmd(priv, get_cooler_duty_cmd, GET_CMD_LENGTH,
&priv->cooler_duty_received);
if (ret < 0)
- goto unlock_and_return;
+ return ret;
memcpy(set_cmd, set_cooler_duty_cmd, SET_CMD_LENGTH);
+ set_cmd[2] = priv->info->duty_channel;
/* Cooler duties are set as 0-100% */
val = rog_ryujin_pwm_to_percent(val);
@@ -329,11 +327,7 @@ static int rog_ryujin_write_fixed_duty(struct rog_ryujin_data *priv, int channel
set_cmd[RYUJIN_SET_COOLER_FAN_DUTY_OFFSET] = val;
}
- ret = rog_ryujin_execute_cmd(priv, set_cmd, SET_CMD_LENGTH, &priv->cooler_duty_set);
-unlock_and_return:
- mutex_unlock(&priv->status_report_request_mutex);
- if (ret < 0)
- return ret;
+ return rog_ryujin_execute_cmd(priv, set_cmd, SET_CMD_LENGTH, &priv->cooler_duty_set);
} else {
/*
* Controller fan duty (channel == 2). No need to retrieve current
@@ -417,19 +411,29 @@ static int rog_ryujin_raw_event(struct hid_device *hdev, struct hid_report *repo
{
struct rog_ryujin_data *priv = hid_get_drvdata(hdev);
- if (data[0] != RYUJIN_CMD_PREFIX)
+ if (size < 2 || data[0] != RYUJIN_CMD_PREFIX)
return 0;
if (data[1] == RYUJIN_GET_COOLER_STATUS_CMD_RESPONSE) {
+ if (size <= priv->info->temp_offset + 1 ||
+ size <= priv->info->pump_speed_offset + 1 ||
+ size <= priv->info->fan_speed_offset + 1)
+ return 0;
+
/* Received coolant temp and speeds of pump and internal fan */
- priv->temp_input[0] =
- data[RYUJIN_TEMP_SENSOR_1] * 1000 + data[RYUJIN_TEMP_SENSOR_2] * 100;
- priv->speed_input[0] = get_unaligned_le16(data + RYUJIN_PUMP_SPEED);
- priv->speed_input[1] = get_unaligned_le16(data + RYUJIN_INTERNAL_FAN_SPEED);
+ priv->temp_input[0] = data[priv->info->temp_offset] * 1000 +
+ data[priv->info->temp_offset + 1] * 100;
+ priv->speed_input[0] =
+ get_unaligned_le16(data + priv->info->pump_speed_offset);
+ priv->speed_input[1] =
+ get_unaligned_le16(data + priv->info->fan_speed_offset);
if (!completion_done(&priv->cooler_status_received))
complete_all(&priv->cooler_status_received);
} else if (data[1] == RYUJIN_GET_CONTROLLER_SPEED_CMD_RESPONSE) {
+ if (size <= RYUJIN_CONTROLLER_SPEED_3 + 1)
+ return 0;
+
/* Received speeds of four fans attached to the controller */
priv->speed_input[2] = get_unaligned_le16(data + RYUJIN_CONTROLLER_SPEED_1);
priv->speed_input[3] = get_unaligned_le16(data + RYUJIN_CONTROLLER_SPEED_2);
@@ -439,6 +443,9 @@ static int rog_ryujin_raw_event(struct hid_device *hdev, struct hid_report *repo
if (!completion_done(&priv->controller_status_received))
complete_all(&priv->controller_status_received);
} else if (data[1] == RYUJIN_GET_COOLER_DUTY_CMD_RESPONSE) {
+ if (size <= RYUJIN_INTERNAL_FAN_DUTY)
+ return 0;
+
/* Received report for pump and internal fan duties (in %) */
if (data[RYUJIN_PUMP_DUTY] == 0 && data[RYUJIN_INTERNAL_FAN_DUTY] == 0) {
/*
@@ -465,6 +472,9 @@ static int rog_ryujin_raw_event(struct hid_device *hdev, struct hid_report *repo
if (!completion_done(&priv->cooler_duty_received))
complete_all(&priv->cooler_duty_received);
} else if (data[1] == RYUJIN_GET_CONTROLLER_DUTY_CMD_RESPONSE) {
+ if (size <= RYUJIN_CONTROLLER_DUTY)
+ return 0;
+
/* Received report for controller duty for fans (in PWM) */
if (data[RYUJIN_CONTROLLER_DUTY] == 0) {
/*
@@ -499,11 +509,15 @@ static int rog_ryujin_probe(struct hid_device *hdev, const struct hid_device_id
struct rog_ryujin_data *priv;
int ret;
+ if (!id->driver_data)
+ return -EINVAL;
+
priv = devm_kzalloc(&hdev->dev, sizeof(*priv), GFP_KERNEL);
if (!priv)
return -ENOMEM;
priv->hdev = hdev;
+ priv->info = (const struct rog_ryujin_device_info *)id->driver_data;
hid_set_drvdata(hdev, priv);
/*
@@ -538,8 +552,6 @@ static int rog_ryujin_probe(struct hid_device *hdev, const struct hid_device_id
goto fail_and_close;
}
- mutex_init(&priv->status_report_request_mutex);
- mutex_init(&priv->buffer_lock);
spin_lock_init(&priv->status_report_request_lock);
init_completion(&priv->cooler_status_received);
init_completion(&priv->controller_status_received);
@@ -576,7 +588,8 @@ static void rog_ryujin_remove(struct hid_device *hdev)
}
static const struct hid_device_id rog_ryujin_table[] = {
- { HID_USB_DEVICE(USB_VENDOR_ID_ASUS_ROG, USB_PRODUCT_ID_RYUJIN_AIO) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ASUS_ROG, USB_PRODUCT_ID_RYUJIN_AIO),
+ .driver_data = (kernel_ulong_t)&rog_ryujin_ii_360_info },
{ }
};
diff --git a/drivers/hwmon/lm90.c b/drivers/hwmon/lm90.c
index 45d5a2427dcb..581a1d10a1b5 100644
--- a/drivers/hwmon/lm90.c
+++ b/drivers/hwmon/lm90.c
@@ -695,6 +695,7 @@ struct lm90_data {
struct mutex update_lock;
struct delayed_work alert_work;
struct work_struct report_work;
+ bool shutdown; /* true if shutting down */
bool valid; /* true if register values are valid */
bool alarms_valid; /* true if status register values are valid */
unsigned long last_updated; /* in jiffies */
@@ -1113,6 +1114,9 @@ static void lm90_report_alarms(struct work_struct *work)
static int lm90_update_alarms_locked(struct lm90_data *data, bool force)
{
+ if (data->shutdown)
+ return 0;
+
if (force || !data->alarms_valid ||
time_after(jiffies, data->alarms_updated + msecs_to_jiffies(data->update_interval))) {
struct i2c_client *client = data->client;
@@ -2522,15 +2526,23 @@ static void lm90_restore_conf(void *_data)
struct lm90_data *data = _data;
struct i2c_client *client = data->client;
- cancel_delayed_work_sync(&data->alert_work);
- cancel_work_sync(&data->report_work);
-
/* Restore initial configuration */
if (data->flags & LM90_HAVE_CONVRATE)
lm90_write_convrate(data, data->convrate_orig);
lm90_write_reg(client, LM90_REG_CONFIG1, data->config_orig);
}
+static void lm90_stop_work(void *_data)
+{
+ struct lm90_data *data = _data;
+
+ mutex_lock(&data->update_lock);
+ data->shutdown = true;
+ mutex_unlock(&data->update_lock);
+ cancel_delayed_work_sync(&data->alert_work);
+ cancel_work_sync(&data->report_work);
+}
+
static int lm90_init_client(struct i2c_client *client, struct lm90_data *data)
{
struct device_node *np = client->dev.of_node;
@@ -2841,6 +2853,10 @@ static int lm90_probe(struct i2c_client *client)
data->hwmon_dev = hwmon_dev;
+ err = devm_add_action_or_reset(&client->dev, lm90_stop_work, data);
+ if (err)
+ return err;
+
if (client->irq) {
dev_dbg(dev, "IRQ: %d\n", client->irq);
err = devm_request_threaded_irq(dev, client->irq,
@@ -2869,7 +2885,8 @@ static void lm90_alert(struct i2c_client *client, enum i2c_alert_protocol type,
*/
struct lm90_data *data = i2c_get_clientdata(client);
- if ((data->flags & LM90_HAVE_BROKEN_ALERT) &&
+ mutex_lock(&data->update_lock);
+ if (!data->shutdown && (data->flags & LM90_HAVE_BROKEN_ALERT) &&
(data->current_alarms & data->alert_alarms)) {
if (!(data->config & 0x80)) {
dev_dbg(&client->dev, "Disabling ALERT#\n");
@@ -2878,6 +2895,7 @@ static void lm90_alert(struct i2c_client *client, enum i2c_alert_protocol type,
schedule_delayed_work(&data->alert_work,
max_t(int, HZ, msecs_to_jiffies(data->update_interval)));
}
+ mutex_unlock(&data->update_lock);
} else {
dev_dbg(&client->dev, "Everything OK\n");
}
diff --git a/drivers/hwmon/pmbus/pmbus_core.c b/drivers/hwmon/pmbus/pmbus_core.c
index 024d19781737..ea79cff0e8d0 100644
--- a/drivers/hwmon/pmbus/pmbus_core.c
+++ b/drivers/hwmon/pmbus/pmbus_core.c
@@ -35,7 +35,7 @@
struct pmbus_sensor {
struct pmbus_sensor *next;
char name[PMBUS_NAME_SIZE]; /* sysfs sensor name */
- struct device_attribute attribute;
+ struct sensor_device_attribute attribute;
u8 page; /* page number */
u8 phase; /* phase number, 0xff for all phases */
u16 reg; /* register */
@@ -58,7 +58,7 @@ struct pmbus_boolean {
struct pmbus_label {
char name[PMBUS_NAME_SIZE]; /* sysfs label name */
- struct device_attribute attribute;
+ struct sensor_device_attribute attribute;
char label[PMBUS_NAME_SIZE]; /* label */
};
#define to_pmbus_label(_attr) \
@@ -1086,9 +1086,24 @@ static u16 pmbus_data2reg_direct(struct pmbus_data *data,
static u16 pmbus_data2reg_vid(struct pmbus_data *data,
struct pmbus_sensor *sensor, s64 val)
{
- val = clamp_val(val, 500, 1600);
-
- return 2 + DIV_ROUND_CLOSEST_ULL((1600LL - val) * 100LL, 625);
+ switch (data->info->vrm_version[sensor->page]) {
+ case vr12:
+ val = clamp_val(val, 250, 1520);
+ return 1 + DIV_ROUND_CLOSEST_ULL(val - 250, 5);
+ case vr13:
+ val = clamp_val(val, 500, 3040);
+ return 1 + DIV_ROUND_CLOSEST_ULL(val - 500, 10);
+ case imvp9:
+ val = clamp_val(val, 200, 2740);
+ return 1 + DIV_ROUND_CLOSEST_ULL(val - 200, 10);
+ case amd625mv:
+ val = clamp_val(val, 200, 1550);
+ return DIV_ROUND_CLOSEST_ULL((1550LL - val) * 100LL, 625);
+ case vr11:
+ default:
+ val = clamp_val(val, 500, 1600);
+ return 2 + DIV_ROUND_CLOSEST_ULL((1600LL - val) * 100LL, 625);
+ }
}
static u16 pmbus_data2reg(struct pmbus_data *data,
@@ -1223,7 +1238,8 @@ static ssize_t pmbus_show_sensor(struct device *dev,
struct device_attribute *devattr, char *buf)
{
struct i2c_client *client = to_i2c_client(dev->parent);
- struct pmbus_sensor *sensor = to_pmbus_sensor(devattr);
+ struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr);
+ struct pmbus_sensor *sensor = to_pmbus_sensor(attr);
struct pmbus_data *data = i2c_get_clientdata(client);
ssize_t ret;
@@ -1243,7 +1259,8 @@ static ssize_t pmbus_set_sensor(struct device *dev,
{
struct i2c_client *client = to_i2c_client(dev->parent);
struct pmbus_data *data = i2c_get_clientdata(client);
- struct pmbus_sensor *sensor = to_pmbus_sensor(devattr);
+ struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr);
+ struct pmbus_sensor *sensor = to_pmbus_sensor(attr);
ssize_t rv = count;
s64 val;
int ret;
@@ -1266,7 +1283,8 @@ static ssize_t pmbus_set_sensor(struct device *dev,
static ssize_t pmbus_show_label(struct device *dev,
struct device_attribute *da, char *buf)
{
- struct pmbus_label *label = to_pmbus_label(da);
+ struct sensor_device_attribute *attr = to_sensor_dev_attr(da);
+ struct pmbus_label *label = to_pmbus_label(attr);
return sysfs_emit(buf, "%s\n", label->label);
}
@@ -1419,8 +1437,8 @@ static struct pmbus_sensor *pmbus_add_sensor(struct pmbus_data *data,
bool update, bool readonly,
bool writeonly, bool convert)
{
+ struct sensor_device_attribute *a;
struct pmbus_sensor *sensor;
- struct device_attribute *a;
sensor = devm_kzalloc(data->dev, sizeof(*sensor), GFP_KERNEL);
if (!sensor)
@@ -1444,12 +1462,11 @@ static struct pmbus_sensor *pmbus_add_sensor(struct pmbus_data *data,
sensor->update = update;
sensor->convert = convert;
sensor->data = -ENODATA;
- pmbus_dev_attr_init(a, sensor->name,
- readonly ? 0444 : 0644,
- writeonly ? pmbus_show_zero : pmbus_show_sensor,
- pmbus_set_sensor);
+ pmbus_attr_init(a, sensor->name, readonly ? 0444 : 0644,
+ writeonly ? pmbus_show_zero : pmbus_show_sensor,
+ pmbus_set_sensor, -1);
- if (pmbus_add_attribute(data, &a->attr))
+ if (pmbus_add_attribute(data, &a->dev_attr.attr))
return NULL;
sensor->next = data->sensors;
@@ -1466,8 +1483,8 @@ static int pmbus_add_label(struct pmbus_data *data,
const char *name, int seq,
const char *lstring, int index, int phase)
{
+ struct sensor_device_attribute *a;
struct pmbus_label *label;
- struct device_attribute *a;
label = devm_kzalloc(data->dev, sizeof(*label), GFP_KERNEL);
if (!label)
@@ -1492,8 +1509,8 @@ static int pmbus_add_label(struct pmbus_data *data,
lstring, index, phase);
}
- pmbus_dev_attr_init(a, label->name, 0444, pmbus_show_label, NULL);
- return pmbus_add_attribute(data, &a->attr);
+ pmbus_attr_init(a, label->name, 0444, pmbus_show_label, NULL, -1);
+ return pmbus_add_attribute(data, &a->dev_attr.attr);
}
/*
@@ -2381,7 +2398,7 @@ struct pmbus_samples_attr {
struct pmbus_samples_reg {
int page;
struct pmbus_samples_attr *attr;
- struct device_attribute dev_attr;
+ struct sensor_device_attribute attribute;
};
static struct pmbus_samples_attr pmbus_samples_registers[] = {
@@ -2403,14 +2420,15 @@ static struct pmbus_samples_attr pmbus_samples_registers[] = {
}
};
-#define to_samples_reg(x) container_of(x, struct pmbus_samples_reg, dev_attr)
+#define to_samples_reg(x) container_of(x, struct pmbus_samples_reg, attribute)
static ssize_t pmbus_show_samples(struct device *dev,
struct device_attribute *devattr, char *buf)
{
int val;
struct i2c_client *client = to_i2c_client(dev->parent);
- struct pmbus_samples_reg *reg = to_samples_reg(devattr);
+ struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr);
+ struct pmbus_samples_reg *reg = to_samples_reg(attr);
struct pmbus_data *data = i2c_get_clientdata(client);
mutex_lock(&data->update_lock);
@@ -2429,7 +2447,8 @@ static ssize_t pmbus_set_samples(struct device *dev,
int ret;
long val;
struct i2c_client *client = to_i2c_client(dev->parent);
- struct pmbus_samples_reg *reg = to_samples_reg(devattr);
+ struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr);
+ struct pmbus_samples_reg *reg = to_samples_reg(attr);
struct pmbus_data *data = i2c_get_clientdata(client);
if (kstrtol(buf, 0, &val) < 0)
@@ -2445,6 +2464,7 @@ static ssize_t pmbus_set_samples(struct device *dev,
static int pmbus_add_samples_attr(struct pmbus_data *data, int page,
struct pmbus_samples_attr *attr)
{
+ struct sensor_device_attribute *a;
struct pmbus_samples_reg *reg;
reg = devm_kzalloc(data->dev, sizeof(*reg), GFP_KERNEL);
@@ -2454,10 +2474,12 @@ static int pmbus_add_samples_attr(struct pmbus_data *data, int page,
reg->attr = attr;
reg->page = page;
- pmbus_dev_attr_init(®->dev_attr, attr->name, 0644,
- pmbus_show_samples, pmbus_set_samples);
+ a = ®->attribute;
- return pmbus_add_attribute(data, ®->dev_attr.attr);
+ pmbus_attr_init(a, attr->name, 0644,
+ pmbus_show_samples, pmbus_set_samples, -1);
+
+ return pmbus_add_attribute(data, &a->dev_attr.attr);
}
static int pmbus_add_samples_attributes(struct i2c_client *client,
@@ -2926,9 +2948,15 @@ static void pmbus_notify(struct pmbus_data *data, int page, int reg, int flags)
struct device_attribute *da = to_dev_attr(data->group.attrs[i]);
struct sensor_device_attribute *attr = to_sensor_dev_attr(da);
int index = attr->index;
- u16 smask = pb_index_to_mask(index);
- u8 spage = pb_index_to_page(index);
- u16 sreg = pb_index_to_reg(index);
+ u16 smask, sreg;
+ u8 spage;
+
+ if (index == -1)
+ continue;
+
+ smask = pb_index_to_mask(index);
+ spage = pb_index_to_page(index);
+ sreg = pb_index_to_reg(index);
if (reg == sreg && page == spage && (smask & flags)) {
dev_dbg(data->dev, "sysfs notify: %s", da->attr.name);
diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c
index 312abd57ddce..9127bc025552 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
@@ -54,10 +54,14 @@ MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
#define PARAM_PM_SAVE_NEVER 1 /* never save any state */
#define PARAM_PM_SAVE_SELF_HOSTED 2 /* save self-hosted state only */
+/*
+ * Save option for ETM4. ETE, sysreg ETM4s and ACPI boots ignore this option and
+ * will always save.
+ */
static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
module_param(pm_save_enable, int, 0444);
MODULE_PARM_DESC(pm_save_enable,
- "Save/restore state on power down: 1 = never, 2 = self-hosted");
+ "Save/restore state on power down: 1 = never, 2 = self-hosted. MMIO and DT only.");
static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
static void etm4_set_default_config(struct etmv4_config *config);
@@ -694,8 +698,7 @@ static int etm4_parse_event_config(struct coresight_device *csdev,
struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
struct etmv4_config *config = &drvdata->config;
struct perf_event_attr *attr = &event->attr;
- unsigned long cfg_hash;
- int preset, cc_threshold;
+ int cc_threshold;
/* Clear configuration from previous run */
memset(config, 0, sizeof(struct etmv4_config));
@@ -770,20 +773,6 @@ static int etm4_parse_event_config(struct coresight_device *csdev,
/* bit[12], Return stack enable bit */
config->cfg |= TRCCONFIGR_RS;
- /*
- * Set any selected configuration and preset.
- *
- * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset)
- * in the perf attributes defined in coresight-etm-perf.c.
- * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config.
- * A zero configid means no configuration active, preset = 0 means no preset selected.
- */
- if (attr->config2 & GENMASK_ULL(63, 32)) {
- cfg_hash = (u32)(attr->config2 >> 32);
- preset = attr->config & 0xF;
- ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset);
- }
-
/* branch broadcast - enable if selected and supported */
if (attr->config & BIT(ETM_OPT_BRANCH_BROADCAST)) {
if (!drvdata->trcbb) {
@@ -806,8 +795,10 @@ static int etm4_enable_perf(struct coresight_device *csdev,
struct perf_event *event,
struct coresight_trace_id_map *id_map)
{
- int ret = 0, trace_id;
+ int ret = 0, trace_id, preset;
struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+ struct perf_event_attr *attr = &event->attr;
+ unsigned long cfg_hash = 0;
if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id())) {
ret = -EINVAL;
@@ -819,6 +810,22 @@ static int etm4_enable_perf(struct coresight_device *csdev,
if (ret)
goto out;
+ /*
+ * Set any selected configuration and preset.
+ *
+ * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset)
+ * in the perf attributes defined in coresight-etm-perf.c.
+ * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config.
+ * A zero configid means no configuration active, preset = 0 means no preset selected.
+ */
+ if (attr->config2 & GENMASK_ULL(63, 32)) {
+ cfg_hash = (u32)(attr->config2 >> 32);
+ preset = attr->config & 0xF;
+ ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset);
+ if (ret)
+ goto out;
+ }
+
/*
* perf allocates cpu ids as part of _setup_aux() - device needs to use
* the allocated ID. This reads the current version without allocation.
@@ -832,13 +839,20 @@ static int etm4_enable_perf(struct coresight_device *csdev,
dev_err(&drvdata->csdev->dev, "Failed to set trace ID for %s on CPU%d\n",
dev_name(&drvdata->csdev->dev), drvdata->cpu);
ret = -EINVAL;
- goto out;
+ goto err_cfg;
}
drvdata->trcid = (u8)trace_id;
/* And enable it */
ret = etm4_enable_hw(drvdata);
+ if (ret)
+ goto err_cfg;
+ return 0;
+
+err_cfg:
+ if (cfg_hash)
+ cscfg_csdev_disable_active_config(csdev);
out:
return ret;
}
@@ -2114,6 +2128,17 @@ static void etm4_pm_clear(void)
}
}
+static bool etm4x_always_pm_save(struct device *dev, struct csdev_access *csa)
+{
+ /*
+ * Only IO mem ETM devices will benefit from skipping PM save and only
+ * DT has the option to control it, not ACPI. Otherwise system register
+ * based ETMs and ETEs will always lose context on CPU power down, so
+ * always save.
+ */
+ return !csa->io_mem || is_acpi_device_node(dev_fwnode(dev));
+}
+
static int etm4_add_coresight_dev(struct etm4_init_arg *init_arg)
{
int ret;
@@ -2123,6 +2148,7 @@ static int etm4_add_coresight_dev(struct etm4_init_arg *init_arg)
struct coresight_desc desc = { 0 };
u8 major, minor;
char *type_name;
+ bool pm_save;
if (!drvdata)
return -EINVAL;
@@ -2150,6 +2176,21 @@ static int etm4_add_coresight_dev(struct etm4_init_arg *init_arg)
etm4_set_default(&drvdata->config);
+ if (etm4x_always_pm_save(dev, init_arg->csa))
+ pm_save = true;
+ else if (pm_save_enable == PARAM_PM_SAVE_FIRMWARE)
+ pm_save = coresight_loses_context_with_cpu(dev);
+ else
+ pm_save = pm_save_enable != PARAM_PM_SAVE_NEVER;
+
+ if (pm_save) {
+ drvdata->save_state = devm_kmalloc(dev,
+ sizeof(struct etmv4_save_state),
+ GFP_KERNEL);
+ if (!drvdata->save_state)
+ return -ENOMEM;
+ }
+
pdata = coresight_get_platform_data(dev);
if (IS_ERR(pdata))
return PTR_ERR(pdata);
@@ -2206,17 +2247,6 @@ static int etm4_probe(struct device *dev)
if (IS_ERR(drvdata->atclk))
return PTR_ERR(drvdata->atclk);
- if (pm_save_enable == PARAM_PM_SAVE_FIRMWARE)
- pm_save_enable = coresight_loses_context_with_cpu(dev) ?
- PARAM_PM_SAVE_SELF_HOSTED : PARAM_PM_SAVE_NEVER;
-
- if (pm_save_enable != PARAM_PM_SAVE_NEVER) {
- drvdata->save_state = devm_kmalloc(dev,
- sizeof(struct etmv4_save_state), GFP_KERNEL);
- if (!drvdata->save_state)
- return -ENOMEM;
- }
-
spin_lock_init(&drvdata->spinlock);
drvdata->cpu = coresight_get_cpu(dev);
diff --git a/drivers/hwtracing/coresight/coresight-syscfg.c b/drivers/hwtracing/coresight/coresight-syscfg.c
index 30a561d87481..55f36ed1af13 100644
--- a/drivers/hwtracing/coresight/coresight-syscfg.c
+++ b/drivers/hwtracing/coresight/coresight-syscfg.c
@@ -395,6 +395,8 @@ static void cscfg_remove_owned_csdev_configs(struct coresight_device *csdev, voi
if (list_empty(&csdev->config_csdev_list))
return;
+ guard(spinlock_irqsave)(&csdev->cscfg_csdev_lock);
+
list_for_each_entry_safe(config_csdev, tmp, &csdev->config_csdev_list, node) {
if (config_csdev->config_desc->load_owner == load_owner)
list_del(&config_csdev->node);
diff --git a/drivers/hwtracing/coresight/coresight-tmc-core.c b/drivers/hwtracing/coresight/coresight-tmc-core.c
index 96ef2517dd43..6e503ca89202 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-core.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-core.c
@@ -496,7 +496,7 @@ static int __tmc_probe(struct device *dev, struct resource *res)
drvdata->base = base;
desc.access = CSDEV_ACCESS_IOMEM(base);
- spin_lock_init(&drvdata->spinlock);
+ raw_spin_lock_init(&drvdata->spinlock);
devid = readl_relaxed(drvdata->base + CORESIGHT_DEVID);
drvdata->config_type = BMVAL(devid, 6, 7);
@@ -600,7 +600,7 @@ static void tmc_shutdown(struct amba_device *adev)
unsigned long flags;
struct tmc_drvdata *drvdata = amba_get_drvdata(adev);
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (coresight_get_mode(drvdata->csdev) == CS_MODE_DISABLED)
goto out;
@@ -614,7 +614,7 @@ static void tmc_shutdown(struct amba_device *adev)
* the system is going down after this.
*/
out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
}
static void __tmc_remove(struct device *dev)
diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index d4f641cd9de6..fca8e9ffb3cf 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -182,9 +182,9 @@ static int tmc_enable_etf_sink_sysfs(struct coresight_device *csdev)
* If we don't have a buffer release the lock and allocate memory.
* Otherwise keep the lock and move along.
*/
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (!drvdata->buf) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
/* Allocating the memory here while outside of the spinlock */
buf = kzalloc(drvdata->size, GFP_KERNEL);
@@ -192,7 +192,7 @@ static int tmc_enable_etf_sink_sysfs(struct coresight_device *csdev)
return -ENOMEM;
/* Let's try again */
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
}
if (drvdata->reading) {
@@ -235,7 +235,7 @@ static int tmc_enable_etf_sink_sysfs(struct coresight_device *csdev)
used = false;
}
out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
/* Free memory outside the spinlock if need be */
if (!used)
@@ -253,7 +253,7 @@ static int tmc_enable_etf_sink_perf(struct coresight_device *csdev, void *data)
struct perf_output_handle *handle = data;
struct cs_buffers *buf = etm_perf_sink_config(handle);
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
do {
ret = -EINVAL;
if (drvdata->reading)
@@ -296,7 +296,7 @@ static int tmc_enable_etf_sink_perf(struct coresight_device *csdev, void *data)
csdev->refcnt++;
}
} while (0);
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return ret;
}
@@ -331,16 +331,16 @@ static int tmc_disable_etf_sink(struct coresight_device *csdev)
unsigned long flags;
struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (drvdata->reading) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return -EBUSY;
}
csdev->refcnt--;
if (csdev->refcnt) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return -EBUSY;
}
@@ -351,7 +351,7 @@ static int tmc_disable_etf_sink(struct coresight_device *csdev)
drvdata->pid = -1;
coresight_set_mode(csdev, CS_MODE_DISABLED);
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
dev_dbg(&csdev->dev, "TMC-ETB/ETF disabled\n");
return 0;
@@ -366,9 +366,9 @@ static int tmc_enable_etf_link(struct coresight_device *csdev,
struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
bool first_enable = false;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (drvdata->reading) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return -EBUSY;
}
@@ -381,7 +381,7 @@ static int tmc_enable_etf_link(struct coresight_device *csdev,
}
if (!ret)
csdev->refcnt++;
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
if (first_enable)
dev_dbg(&csdev->dev, "TMC-ETF enabled\n");
@@ -396,9 +396,9 @@ static void tmc_disable_etf_link(struct coresight_device *csdev,
struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
bool last_disable = false;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (drvdata->reading) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return;
}
@@ -408,7 +408,7 @@ static void tmc_disable_etf_link(struct coresight_device *csdev,
coresight_set_mode(csdev, CS_MODE_DISABLED);
last_disable = true;
}
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
if (last_disable)
dev_dbg(&csdev->dev, "TMC-ETF disabled\n");
@@ -488,7 +488,7 @@ static unsigned long tmc_update_etf_buffer(struct coresight_device *csdev,
if (WARN_ON_ONCE(coresight_get_mode(csdev) != CS_MODE_PERF))
return 0;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
/* Don't do anything if another tracer is using this sink */
if (csdev->refcnt != 1)
@@ -585,7 +585,7 @@ static unsigned long tmc_update_etf_buffer(struct coresight_device *csdev,
*/
CS_LOCK(drvdata->base);
out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return to_read;
}
@@ -623,7 +623,7 @@ int tmc_read_prepare_etb(struct tmc_drvdata *drvdata)
drvdata->config_type != TMC_CONFIG_TYPE_ETF))
return -EINVAL;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (drvdata->reading) {
ret = -EBUSY;
@@ -655,7 +655,7 @@ int tmc_read_prepare_etb(struct tmc_drvdata *drvdata)
drvdata->reading = true;
out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return ret;
}
@@ -672,14 +672,14 @@ int tmc_read_unprepare_etb(struct tmc_drvdata *drvdata)
drvdata->config_type != TMC_CONFIG_TYPE_ETF))
return -EINVAL;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
/* Re-enable the TMC if need be */
if (coresight_get_mode(drvdata->csdev) == CS_MODE_SYSFS) {
/* There is no point in reading a TMC in HW FIFO mode */
mode = readl_relaxed(drvdata->base + TMC_MODE);
if (mode != TMC_MODE_CIRCULAR_BUFFER) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return -EINVAL;
}
/*
@@ -693,7 +693,7 @@ int tmc_read_unprepare_etb(struct tmc_drvdata *drvdata)
memset(drvdata->buf, 0, drvdata->size);
rc = __tmc_etb_enable_hw(drvdata);
if (rc) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return rc;
}
} else {
@@ -706,7 +706,7 @@ int tmc_read_unprepare_etb(struct tmc_drvdata *drvdata)
}
drvdata->reading = false;
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
/*
* Free allocated memory outside of the spinlock. There is no need
diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c
index a48bb85d0e7f..11ac1a8b9e7a 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
@@ -1176,10 +1176,17 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
* buffer, provided the size matches. Any allocation has to be done
* with the lock released.
*/
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
+
+ /*
+ * If the ETR is already enabled, continue with the existing buffer.
+ */
+ if (coresight_get_mode(csdev) == CS_MODE_SYSFS)
+ goto out;
+
sysfs_buf = READ_ONCE(drvdata->sysfs_buf);
if (!sysfs_buf || (sysfs_buf->size != drvdata->size)) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
/* Allocate memory with the locks released */
free_buf = new_buf = tmc_etr_setup_sysfs_buf(drvdata);
@@ -1187,7 +1194,7 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
return new_buf;
/* Let's try again */
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
}
if (drvdata->reading || coresight_get_mode(csdev) == CS_MODE_PERF) {
@@ -1206,7 +1213,7 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev)
}
out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
/* Free memory outside the spinlock if need be */
if (free_buf)
@@ -1224,7 +1231,20 @@ static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
if (IS_ERR(sysfs_buf))
return PTR_ERR(sysfs_buf);
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
+
+ /*
+ * Since the sysfs buffer allocation and the hardware enablement is not
+ * in the same critical region, it's possible to race with the perf.
+ */
+ if (coresight_get_mode(csdev) == CS_MODE_PERF) {
+ drvdata->sysfs_buf = NULL;
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
+
+ /* Free allocated memory out side of the spinlock */
+ tmc_etr_free_sysfs_buf(sysfs_buf);
+ return -EBUSY;
+ }
/*
* In sysFS mode we can have multiple writers per sink. Since this
@@ -1243,7 +1263,7 @@ static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev)
}
out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
if (!ret)
dev_dbg(&csdev->dev, "TMC-ETR enabled\n");
@@ -1562,17 +1582,17 @@ tmc_update_etr_buffer(struct coresight_device *csdev,
struct etr_perf_buffer *etr_perf = config;
struct etr_buf *etr_buf = etr_perf->etr_buf;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
/* Don't do anything if another tracer is using this sink */
if (csdev->refcnt != 1) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
goto out;
}
if (WARN_ON(drvdata->perf_buf != etr_buf)) {
lost = true;
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
goto out;
}
@@ -1582,7 +1602,7 @@ tmc_update_etr_buffer(struct coresight_device *csdev,
tmc_sync_etr_buf(drvdata);
CS_LOCK(drvdata->base);
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
lost = etr_buf->full;
offset = etr_buf->offset;
@@ -1651,7 +1671,7 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
struct perf_output_handle *handle = data;
struct etr_perf_buffer *etr_perf = etm_perf_sink_config(handle);
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
/* Don't use this sink if it is already claimed by sysFS */
if (coresight_get_mode(csdev) == CS_MODE_SYSFS) {
rc = -EBUSY;
@@ -1691,7 +1711,7 @@ static int tmc_enable_etr_sink_perf(struct coresight_device *csdev, void *data)
}
unlock_out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return rc;
}
@@ -1713,16 +1733,16 @@ static int tmc_disable_etr_sink(struct coresight_device *csdev)
unsigned long flags;
struct tmc_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (drvdata->reading) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return -EBUSY;
}
csdev->refcnt--;
if (csdev->refcnt) {
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return -EBUSY;
}
@@ -1735,7 +1755,7 @@ static int tmc_disable_etr_sink(struct coresight_device *csdev)
/* Reset perf specific data */
drvdata->perf_buf = NULL;
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
dev_dbg(&csdev->dev, "TMC-ETR disabled\n");
return 0;
@@ -1762,7 +1782,7 @@ int tmc_read_prepare_etr(struct tmc_drvdata *drvdata)
if (WARN_ON_ONCE(drvdata->config_type != TMC_CONFIG_TYPE_ETR))
return -EINVAL;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
if (drvdata->reading) {
ret = -EBUSY;
goto out;
@@ -1784,7 +1804,7 @@ int tmc_read_prepare_etr(struct tmc_drvdata *drvdata)
drvdata->reading = true;
out:
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
return ret;
}
@@ -1798,7 +1818,7 @@ int tmc_read_unprepare_etr(struct tmc_drvdata *drvdata)
if (WARN_ON_ONCE(drvdata->config_type != TMC_CONFIG_TYPE_ETR))
return -EINVAL;
- spin_lock_irqsave(&drvdata->spinlock, flags);
+ raw_spin_lock_irqsave(&drvdata->spinlock, flags);
/* RE-enable the TMC if need be */
if (coresight_get_mode(drvdata->csdev) == CS_MODE_SYSFS) {
@@ -1818,7 +1838,7 @@ int tmc_read_unprepare_etr(struct tmc_drvdata *drvdata)
}
drvdata->reading = false;
- spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ raw_spin_unlock_irqrestore(&drvdata->spinlock, flags);
/* Free allocated memory out side of the spinlock */
if (sysfs_buf)
diff --git a/drivers/hwtracing/coresight/coresight-tmc.h b/drivers/hwtracing/coresight/coresight-tmc.h
index 2a53acbb5990..a9fccc3eb423 100644
--- a/drivers/hwtracing/coresight/coresight-tmc.h
+++ b/drivers/hwtracing/coresight/coresight-tmc.h
@@ -197,7 +197,7 @@ struct tmc_drvdata {
void __iomem *base;
struct coresight_device *csdev;
struct miscdevice miscdev;
- spinlock_t spinlock;
+ raw_spinlock_t spinlock;
pid_t pid;
bool reading;
union {
diff --git a/drivers/i3c/master.c b/drivers/i3c/master.c
index 4b23fe33329b..bb15b621c067 100644
--- a/drivers/i3c/master.c
+++ b/drivers/i3c/master.c
@@ -1661,12 +1661,16 @@ static int i3c_master_early_i3c_dev_add(struct i3c_master_controller *master,
static void
i3c_master_register_new_i3c_devs(struct i3c_master_controller *master)
{
+ struct i3c_device *i3cdev, *tmp;
struct i3c_dev_desc *desc;
+ LIST_HEAD(i3c_unreg_devs);
int ret;
if (!master->init_done)
return;
+ i3c_bus_maintenance_lock(&master->bus);
+
i3c_bus_for_each_i3cdev(&master->bus, desc) {
if (desc->dev || !desc->info.dyn_addr || desc == master->this)
continue;
@@ -1687,15 +1691,30 @@ i3c_master_register_new_i3c_devs(struct i3c_master_controller *master)
if (desc->boardinfo)
desc->dev->dev.of_node = desc->boardinfo->of_node;
- ret = device_register(&desc->dev->dev);
- if (ret) {
- dev_err(&master->dev,
- "Failed to add I3C device (err = %d)\n", ret);
- desc->dev->desc = NULL;
- put_device(&desc->dev->dev);
- desc->dev = NULL;
- }
+ list_add_tail(&desc->dev->node, &i3c_unreg_devs);
+ }
+
+ i3c_bus_maintenance_unlock(&master->bus);
+
+ list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) {
+ ret = device_register(&i3cdev->dev);
+ if (ret)
+ dev_err(&master->dev, "Failed to add I3C device (err = %d)\n", ret);
+ else
+ list_del_init(&i3cdev->node);
}
+
+ i3c_bus_maintenance_lock(&master->bus);
+
+ list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) {
+ list_del(&i3cdev->node);
+ desc = i3cdev->desc;
+ i3cdev->desc = NULL;
+ put_device(&i3cdev->dev);
+ desc->dev = NULL;
+ }
+
+ i3c_bus_maintenance_unlock(&master->bus);
}
/**
@@ -1724,9 +1743,7 @@ int i3c_master_do_daa(struct i3c_master_controller *master)
if (ret)
return ret;
- i3c_bus_normaluse_lock(&master->bus);
i3c_master_register_new_i3c_devs(master);
- i3c_bus_normaluse_unlock(&master->bus);
return 0;
}
@@ -2904,9 +2921,7 @@ int i3c_master_register(struct i3c_master_controller *master,
* register I3C devices discovered during the initial DAA.
*/
master->init_done = true;
- i3c_bus_normaluse_lock(&master->bus);
i3c_master_register_new_i3c_devs(master);
- i3c_bus_normaluse_unlock(&master->bus);
return 0;
diff --git a/drivers/i3c/master/mipi-i3c-hci/core.c b/drivers/i3c/master/mipi-i3c-hci/core.c
index 51780354be3b..73a620144c7e 100644
--- a/drivers/i3c/master/mipi-i3c-hci/core.c
+++ b/drivers/i3c/master/mipi-i3c-hci/core.c
@@ -804,6 +804,7 @@ static int i3c_hci_probe(struct platform_device *pdev)
hci = devm_kzalloc(&pdev->dev, sizeof(*hci), GFP_KERNEL);
if (!hci)
return -ENOMEM;
+ mutex_init(&hci->control_mutex);
hci->base_regs = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(hci->base_regs))
return PTR_ERR(hci->base_regs);
diff --git a/drivers/i3c/master/mipi-i3c-hci/dma.c b/drivers/i3c/master/mipi-i3c-hci/dma.c
index 09bc109c4ab1..d61d88b3a47a 100644
--- a/drivers/i3c/master/mipi-i3c-hci/dma.c
+++ b/drivers/i3c/master/mipi-i3c-hci/dma.c
@@ -457,16 +457,25 @@ static bool hci_dma_dequeue_xfer(struct i3c_hci *hci,
struct hci_rh_data *rh = &rings->headers[xfer_list[0].ring_number];
unsigned int i;
bool did_unqueue = false;
-
- /* stop the ring */
- rh_reg_write(RING_CONTROL, RING_CTRL_ABORT);
- if (wait_for_completion_timeout(&rh->op_done, HZ) == 0) {
- /*
- * We're deep in it if ever this condition is ever met.
- * Hardware might still be writing to memory, etc.
- */
- dev_crit(&hci->master.dev, "unable to abort the ring\n");
- WARN_ON(1);
+ u32 ring_status;
+
+ guard(mutex)(&hci->control_mutex);
+
+ ring_status = rh_reg_read(RING_STATUS);
+ if (ring_status & RING_STATUS_RUNNING) {
+ /* stop the ring */
+ reinit_completion(&rh->op_done);
+ rh_reg_write(RING_CONTROL, rh_reg_read(RING_CONTROL) | RING_CTRL_ABORT);
+ wait_for_completion_timeout(&rh->op_done, HZ);
+ ring_status = rh_reg_read(RING_STATUS);
+ if (ring_status & RING_STATUS_RUNNING) {
+ /*
+ * We're deep in it if ever this condition is ever met.
+ * Hardware might still be writing to memory, etc.
+ */
+ dev_crit(&hci->master.dev, "unable to abort the ring\n");
+ WARN_ON(1);
+ }
}
for (i = 0; i < n; i++) {
diff --git a/drivers/i3c/master/mipi-i3c-hci/hci.h b/drivers/i3c/master/mipi-i3c-hci/hci.h
index aaa47ac47381..24e6dd22cf0e 100644
--- a/drivers/i3c/master/mipi-i3c-hci/hci.h
+++ b/drivers/i3c/master/mipi-i3c-hci/hci.h
@@ -48,6 +48,7 @@ struct i3c_hci {
const struct hci_io_ops *io;
void *io_data;
const struct hci_cmd_ops *cmd;
+ struct mutex control_mutex;
atomic_t next_cmd_tid;
u32 caps;
unsigned int quirks;
diff --git a/drivers/i3c/master/svc-i3c-master.c b/drivers/i3c/master/svc-i3c-master.c
index d5cc26955454..705e7b9256f0 100644
--- a/drivers/i3c/master/svc-i3c-master.c
+++ b/drivers/i3c/master/svc-i3c-master.c
@@ -58,6 +58,7 @@
#define SVC_I3C_MSTATUS 0x088
#define SVC_I3C_MSTATUS_STATE(x) FIELD_GET(GENMASK(2, 0), (x))
#define SVC_I3C_MSTATUS_STATE_DAA(x) (SVC_I3C_MSTATUS_STATE(x) == 5)
+#define SVC_I3C_MSTATUS_STATE_SLVREQ(x) (SVC_I3C_MSTATUS_STATE(x) == 1)
#define SVC_I3C_MSTATUS_STATE_IDLE(x) (SVC_I3C_MSTATUS_STATE(x) == 0)
#define SVC_I3C_MSTATUS_BETWEEN(x) FIELD_GET(BIT(4), (x))
#define SVC_I3C_MSTATUS_NACKED(x) FIELD_GET(BIT(5), (x))
@@ -102,6 +103,7 @@
#define SVC_I3C_MDATACTRL_TXTRIG_FIFO_NOT_FULL GENMASK(5, 4)
#define SVC_I3C_MDATACTRL_RXTRIG_FIFO_NOT_EMPTY 0
#define SVC_I3C_MDATACTRL_RXCOUNT(x) FIELD_GET(GENMASK(28, 24), (x))
+#define SVC_I3C_MDATACTRL_TXCOUNT(x) FIELD_GET(GENMASK(20, 16), (x))
#define SVC_I3C_MDATACTRL_TXFULL BIT(30)
#define SVC_I3C_MDATACTRL_RXEMPTY BIT(31)
@@ -113,6 +115,7 @@
#define SVC_I3C_MWDATAHE 0x0BC
#define SVC_I3C_MRDATAB 0x0C0
#define SVC_I3C_MRDATAH 0x0C8
+#define SVC_I3C_MWDATAB1 0x0CC
#define SVC_I3C_MWMSG_SDR 0x0D0
#define SVC_I3C_MRMSG_SDR 0x0D4
#define SVC_I3C_MWMSG_DDR 0x0D8
@@ -133,6 +136,22 @@
#define SVC_I3C_EVENT_IBI GENMASK(7, 0)
#define SVC_I3C_EVENT_HOTJOIN BIT(31)
+/*
+ * SVC_I3C_QUIRK_FIFO_EMPTY:
+ * I3C HW stalls the write transfer if the transmit FIFO becomes empty,
+ * when new data is written to FIFO, I3C HW resumes the transfer but
+ * the first transmitted data bit may have the wrong value.
+ * Workaround:
+ * Fill the FIFO in advance to prevent FIFO from becoming empty.
+ */
+#define SVC_I3C_QUIRK_FIFO_EMPTY BIT(0)
+/*
+ * SVC_I3C_QUIRK_FLASE_SLVSTART:
+ * I3C HW may generate an invalid SlvStart event when emitting a STOP.
+ * If it is a true SlvStart, the MSTATUS state is SLVREQ.
+ */
+#define SVC_I3C_QUIRK_FALSE_SLVSTART BIT(1)
+
struct svc_i3c_cmd {
u8 addr;
bool rnw;
@@ -236,6 +255,11 @@ struct svc_i3c_i2c_dev_data {
struct i3c_generic_ibi_pool *ibi_pool;
};
+static inline bool svc_has_quirk(struct svc_i3c_master *master, u32 quirk)
+{
+ return (master->drvdata->quirks & quirk);
+}
+
static inline bool is_events_enabled(struct svc_i3c_master *master, u32 mask)
{
return !!(master->enabled_events & mask);
@@ -547,6 +571,15 @@ static void svc_i3c_master_ibi_work(struct work_struct *work)
break;
case SVC_I3C_MSTATUS_IBITYPE_MASTER_REQUEST:
svc_i3c_master_emit_stop(master);
+
+ /*
+ * If a target gets stuck holding SDA low, the controller reports a MR.
+ * On NPCM845, emitting STOP may spuriously set SLVSTART, retriggering
+ * the interrupt and re-entering MR handling, leading to an IRQ storm.
+ * Clear SLVSTART after STOP to break the loop.
+ */
+ if (svc_has_quirk(master, SVC_I3C_QUIRK_FALSE_SLVSTART))
+ writel(SVC_I3C_MINT_SLVSTART, master->regs + SVC_I3C_MSTATUS);
break;
default:
break;
@@ -568,6 +601,19 @@ static irqreturn_t svc_i3c_master_irq_handler(int irq, void *dev_id)
/* Clear the interrupt status */
writel(SVC_I3C_MINT_SLVSTART, master->regs + SVC_I3C_MSTATUS);
+ if (svc_has_quirk(master, SVC_I3C_QUIRK_FALSE_SLVSTART)) {
+ /*
+ * Re-read MSTATUS to obtain the latest state and avoid
+ * missing an IBI that arrives after MSTATUS is latched
+ * but before SLVSTART is cleared.
+ */
+ active = readl(master->regs + SVC_I3C_MSTATUS);
+
+ /* Ignore the false event */
+ if (!SVC_I3C_MSTATUS_STATE_SLVREQ(active))
+ return IRQ_HANDLED;
+ }
+
svc_i3c_master_disable_interrupts(master);
/* Handle the interrupt in a non atomic context */
@@ -892,7 +938,7 @@ static int svc_i3c_master_do_daa_locked(struct svc_i3c_master *master,
u8 *addrs, unsigned int *count)
{
u64 prov_id[SVC_I3C_MAX_DEVS] = {}, nacking_prov_id = 0;
- unsigned int dev_nb = 0, last_addr = 0;
+ unsigned int dev_nb = 0, last_addr = 0, dyn_addr = 0;
u32 reg;
int ret, i;
@@ -934,6 +980,26 @@ static int svc_i3c_master_do_daa_locked(struct svc_i3c_master *master,
if (SVC_I3C_MSTATUS_RXPEND(reg)) {
u8 data[6];
+ /*
+ * One slave sends its ID to request for address assignment,
+ * prefilling the dynamic address can reduce SCL clock stalls
+ * and also fix the SVC_I3C_QUIRK_FIFO_EMPTY quirk.
+ *
+ * Ideally, prefilling before the processDAA command is better.
+ * However, it requires an additional check to write the dyn_addr
+ * at the right time because the driver needs to write the processDAA
+ * command twice for one assignment.
+ * Prefilling here is safe and efficient because the FIFO starts
+ * filling within a few hundred nanoseconds, which is significantly
+ * faster compared to the 64 SCL clock cycles.
+ */
+ ret = i3c_master_get_free_addr(&master->base, last_addr + 1);
+ if (ret < 0)
+ break;
+
+ dyn_addr = ret;
+ writel(dyn_addr, master->regs + SVC_I3C_MWDATAB);
+
/*
* We only care about the 48-bit provisioned ID yet to
* be sure a device does not nack an address twice.
@@ -1007,21 +1073,16 @@ static int svc_i3c_master_do_daa_locked(struct svc_i3c_master *master,
if (ret)
break;
- /* Give the slave device a suitable dynamic address */
- ret = i3c_master_get_free_addr(&master->base, last_addr + 1);
- if (ret < 0)
- break;
-
- addrs[dev_nb] = ret;
+ addrs[dev_nb] = dyn_addr;
dev_dbg(master->dev, "DAA: device %d assigned to 0x%02x\n",
dev_nb, addrs[dev_nb]);
-
- writel(addrs[dev_nb], master->regs + SVC_I3C_MWDATAB);
last_addr = addrs[dev_nb++];
}
/* Need manual issue STOP except for Complete condition */
svc_i3c_master_emit_stop(master);
+ svc_i3c_master_flush_fifo(master);
+
return ret;
}
@@ -1218,6 +1279,29 @@ static int svc_i3c_master_xfer(struct svc_i3c_master *master,
SVC_I3C_MCTRL_RDTERM(*actual_len),
master->regs + SVC_I3C_MCTRL);
+ /*
+ * The entire transaction can consist of multiple write transfers.
+ * Prefilling before EmitStartAddr causes the data to be emitted
+ * immediately, becoming part of the previous transfer.
+ * The only way to work around this hardware issue is to let the
+ * FIFO start filling as soon as possible after EmitStartAddr.
+ */
+ if (svc_has_quirk(master, SVC_I3C_QUIRK_FIFO_EMPTY) && !rnw && xfer_len) {
+ u32 space, end, len;
+
+ reg = readl(master->regs + SVC_I3C_MDATACTRL);
+ space = SVC_I3C_FIFO_SIZE - SVC_I3C_MDATACTRL_TXCOUNT(reg);
+ if (space) {
+ end = xfer_len > space ? 0 : SVC_I3C_MWDATAB_END;
+ len = min_t(u32, xfer_len, space);
+ writesb(master->regs + SVC_I3C_MWDATAB1, out, len - 1);
+ /* Mark END bit if this is the last byte */
+ writel(out[len - 1] | end, master->regs + SVC_I3C_MWDATAB);
+ xfer_len -= len;
+ out += len;
+ }
+ }
+
ret = readl_poll_timeout(master->regs + SVC_I3C_MSTATUS, reg,
SVC_I3C_MSTATUS_MCTRLDONE(reg), 0, 1000);
if (ret)
@@ -1304,6 +1388,7 @@ static int svc_i3c_master_xfer(struct svc_i3c_master *master,
emit_stop:
svc_i3c_master_emit_stop(master);
svc_i3c_master_clear_merrwarn(master);
+ svc_i3c_master_flush_fifo(master);
return ret;
}
@@ -1958,7 +2043,10 @@ static const struct dev_pm_ops svc_i3c_pm_ops = {
svc_i3c_runtime_resume, NULL)
};
-static const struct svc_i3c_drvdata npcm845_drvdata = {};
+static const struct svc_i3c_drvdata npcm845_drvdata = {
+ .quirks = SVC_I3C_QUIRK_FIFO_EMPTY |
+ SVC_I3C_QUIRK_FALSE_SLVSTART,
+};
static const struct svc_i3c_drvdata svc_default_drvdata = {};
diff --git a/drivers/infiniband/core/counters.c b/drivers/infiniband/core/counters.c
index 8cbca897fc53..78a76b4fa6f6 100644
--- a/drivers/infiniband/core/counters.c
+++ b/drivers/infiniband/core/counters.c
@@ -190,12 +190,19 @@ static struct rdma_counter *alloc_and_bind(struct ib_device *dev, u32 port,
ret = __rdma_counter_bind_qp(counter, qp);
if (ret)
- goto err_mode;
+ goto err_bind;
rdma_restrack_parent_name(&counter->res, &qp->res);
rdma_restrack_add(&counter->res);
return counter;
+err_bind:
+ mutex_lock(&port_counter->lock);
+ port_counter->num_counters--;
+ if (!port_counter->num_counters &&
+ port_counter->mode.mode == RDMA_COUNTER_MODE_MANUAL)
+ __counter_set_mode(port_counter, RDMA_COUNTER_MODE_NONE, 0);
+ mutex_unlock(&port_counter->lock);
err_mode:
rdma_free_hw_stats_struct(counter->stats);
err_stats:
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 467bdfea6c92..8231fd9b665c 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -1402,6 +1402,9 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd,
}
qplqp->type = (u8)qptype;
qplqp->wqe_mode = bnxt_re_is_var_size_supported(rdev, uctx);
+ if (uctx && qplqp->wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE &&
+ (!ureq->sq_slots || ureq->sq_slots > BNXT_RE_MAX_SQ_SLOTS))
+ return -EINVAL;
if (init_attr->qp_type == IB_QPT_RC) {
qplqp->max_rd_atomic = dev_attr->max_qp_rd_atom;
qplqp->max_dest_rd_atomic = dev_attr->max_qp_init_rd_atom;
@@ -1719,19 +1722,14 @@ int bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata)
ib_srq);
struct bnxt_re_dev *rdev = srq->rdev;
struct bnxt_qplib_srq *qplib_srq = &srq->qplib_srq;
- struct bnxt_qplib_nq *nq = NULL;
- if (qplib_srq->cq)
- nq = qplib_srq->cq->nq;
- if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) {
- free_page((unsigned long)srq->uctx_srq_page);
+ if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT)
hash_del(&srq->hash_entry);
- }
bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq);
+ if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT)
+ free_page((unsigned long)srq->uctx_srq_page);
ib_umem_release(srq->umem);
atomic_dec(&rdev->stats.res.srq_count);
- if (nq)
- nq->budget--;
return 0;
}
@@ -1772,7 +1770,6 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
struct ib_udata *udata)
{
struct bnxt_qplib_dev_attr *dev_attr;
- struct bnxt_qplib_nq *nq = NULL;
struct bnxt_re_ucontext *uctx;
struct bnxt_re_dev *rdev;
struct bnxt_re_srq *srq;
@@ -1818,7 +1815,6 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
srq->qplib_srq.eventq_hw_ring_id = rdev->nqr->nq[0].ring_id;
srq->qplib_srq.sg_info.pgsize = PAGE_SIZE;
srq->qplib_srq.sg_info.pgshft = PAGE_SHIFT;
- nq = &rdev->nqr->nq[0];
if (udata) {
rc = bnxt_re_init_user_srq(rdev, pd, srq, udata);
@@ -1853,8 +1849,6 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
goto fail;
}
}
- if (nq)
- nq->budget++;
active_srqs = atomic_inc_return(&rdev->stats.res.srq_count);
if (active_srqs > rdev->stats.res.srq_watermark)
rdev->stats.res.srq_watermark = active_srqs;
@@ -2984,15 +2978,14 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
nq = cq->qplib_cq.nq;
cctx = rdev->chip_ctx;
- if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT) {
- free_page((unsigned long)cq->uctx_cq_page);
+ if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
hash_del(&cq->hash_entry);
- }
bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
+ if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
+ free_page((unsigned long)cq->uctx_cq_page);
ib_umem_release(cq->umem);
atomic_dec(&rdev->stats.res.cq_count);
- nq->budget--;
kfree(cq->cql);
return 0;
}
@@ -4237,6 +4230,7 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata)
goto fail;
}
spin_lock_init(&uctx->sh_lock);
+ mutex_init(&uctx->wcdpi_lock);
resp.comp_mask = BNXT_RE_UCNTX_CMASK_HAVE_CCTX;
chip_met_rev_num = rdev->chip_ctx->chip_num;
@@ -4435,8 +4429,8 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle *
struct bnxt_re_ucontext *uctx;
struct bnxt_re_dev *rdev;
u64 mmap_offset;
+ u32 dpi = 0;
u32 length;
- u32 dpi;
u64 addr;
int err;
@@ -4453,14 +4447,23 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle *
switch (alloc_type) {
case BNXT_RE_ALLOC_WC_PAGE:
- if (cctx->modes.db_push) {
+ if (cctx->modes.db_push) {
+ mutex_lock(&uctx->wcdpi_lock);
+ /* already allocated — one WC page per context */
+ if (uctx->wcdpi.dbr) {
+ mutex_unlock(&uctx->wcdpi_lock);
+ return -EEXIST;
+ }
if (bnxt_qplib_alloc_dpi(&rdev->qplib_res, &uctx->wcdpi,
- uctx, BNXT_QPLIB_DPI_TYPE_WC))
+ uctx, BNXT_QPLIB_DPI_TYPE_WC)) {
+ mutex_unlock(&uctx->wcdpi_lock);
return -ENOMEM;
+ }
length = PAGE_SIZE;
dpi = uctx->wcdpi.dpi;
addr = (u64)uctx->wcdpi.umdbr;
mmap_flag = BNXT_RE_MMAP_WC_DB;
+ mutex_unlock(&uctx->wcdpi_lock);
} else {
return -EINVAL;
}
@@ -4483,8 +4486,15 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle *
}
entry = bnxt_re_mmap_entry_insert(uctx, addr, mmap_flag, &mmap_offset);
- if (!entry)
+ if (!entry) {
+ if (mmap_flag == BNXT_RE_MMAP_WC_DB) {
+ mutex_lock(&uctx->wcdpi_lock);
+ bnxt_qplib_dealloc_dpi(&rdev->qplib_res, &uctx->wcdpi);
+ uctx->wcdpi.dbr = NULL;
+ mutex_unlock(&uctx->wcdpi_lock);
+ }
return -ENOMEM;
+ }
uobj->object = entry;
uverbs_finalize_uobj_create(attrs, BNXT_RE_ALLOC_PAGE_HANDLE);
@@ -4515,11 +4525,16 @@ static int alloc_page_obj_cleanup(struct ib_uobject *uobject,
switch (entry->mmap_flag) {
case BNXT_RE_MMAP_WC_DB:
- if (uctx && uctx->wcdpi.dbr) {
+ if (uctx) {
struct bnxt_re_dev *rdev = uctx->rdev;
- bnxt_qplib_dealloc_dpi(&rdev->qplib_res, &uctx->wcdpi);
- uctx->wcdpi.dbr = NULL;
+ mutex_lock(&uctx->wcdpi_lock);
+ if (uctx->wcdpi.dbr) {
+ bnxt_qplib_dealloc_dpi(&rdev->qplib_res,
+ &uctx->wcdpi);
+ uctx->wcdpi.dbr = NULL;
+ }
+ mutex_unlock(&uctx->wcdpi_lock);
}
break;
case BNXT_RE_MMAP_DBR_BAR:
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h
index 9cd8f770d1b2..b01b685ee70f 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h
@@ -141,6 +141,7 @@ struct bnxt_re_ucontext {
struct bnxt_re_dev *rdev;
struct bnxt_qplib_dpi dpi;
struct bnxt_qplib_dpi wcdpi;
+ struct mutex wcdpi_lock; /* serialises WC DPI alloc/free */
void *shpg;
spinlock_t sh_lock; /* protect shpg */
struct rdma_user_mmap_entry *shpage_mmap;
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
index fcfef5cbb38d..92f3131a9a82 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
@@ -357,6 +357,7 @@ int bnxt_qplib_modify_cc(struct bnxt_qplib_res *res,
#define BNXT_VAR_MAX_SLOT_ALIGN 256
#define BNXT_VAR_MAX_SGE 13
#define BNXT_RE_MAX_RQ_WQES 65536
+#define BNXT_RE_MAX_SQ_SLOTS 65536
#define BNXT_STATIC_MAX_SGE 6
diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c
index 9c24789bbb54..9ed43d23c183 100644
--- a/drivers/infiniband/hw/cxgb4/mem.c
+++ b/drivers/infiniband/hw/cxgb4/mem.c
@@ -199,7 +199,8 @@ static int _c4iw_write_mem_dma(struct c4iw_rdev *rdev, u32 addr, u32 len,
daddr = dma_map_single(&rdev->lldi.pdev->dev, data, len, DMA_TO_DEVICE);
if (dma_mapping_error(&rdev->lldi.pdev->dev, daddr))
- return -1;
+ return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
+ wr_waitp);
save = daddr;
while (remain > inline_threshold) {
@@ -235,30 +236,12 @@ static int write_adapter_mem(struct c4iw_rdev *rdev, u32 addr, u32 len,
void *data, struct sk_buff *skb,
struct c4iw_wr_wait *wr_waitp)
{
- int ret;
-
- if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl) {
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
- wr_waitp);
- goto out;
- }
-
- if (len <= inline_threshold) {
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
+ if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl ||
+ len <= inline_threshold)
+ return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
wr_waitp);
- goto out;
- }
-
- ret = _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
- if (ret) {
- pr_warn_ratelimited("%s: dma map failure (non fatal)\n",
- pci_name(rdev->lldi.pdev));
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
- wr_waitp);
- }
-out:
- return ret;
+ return _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
}
/*
@@ -280,12 +263,16 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
bool stag_idx_allocated = false;
static atomic_t key;
- if (c4iw_fatal_error(rdev))
+ if (c4iw_fatal_error(rdev)) {
+ kfree_skb(skb);
return -EIO;
+ }
tpt = kmalloc(sizeof(*tpt), GFP_KERNEL);
- if (!tpt)
+ if (!tpt) {
+ kfree_skb(skb);
return -ENOMEM;
+ }
stag_state = stag_state > 0;
stag_idx = (*stag) >> 8;
@@ -297,6 +284,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
rdev->stats.stag.fail++;
mutex_unlock(&rdev->stats.lock);
kfree(tpt);
+ kfree_skb(skb);
return -ENOMEM;
}
mutex_lock(&rdev->stats.lock);
@@ -471,8 +459,10 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
FW_RI_STAG_NSMR, mhp->attr.perms,
mhp->attr.mw_bind_enable, 0, 0, ~0ULL, 0, 0, 0,
NULL, mhp->wr_waitp);
- if (ret)
- goto err_free_skb;
+ if (ret) {
+ kfree_skb(mhp->dereg_skb);
+ goto err_free_wr_wait;
+ }
ret = finish_mem_reg(mhp, stag);
if (ret)
@@ -481,8 +471,6 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
err_dereg_mem:
dereg_mem(&rhp->rdev, mhp->attr.stag, mhp->attr.pbl_size,
mhp->attr.pbl_addr, mhp->dereg_skb, mhp->wr_waitp);
-err_free_skb:
- kfree_skb(mhp->dereg_skb);
err_free_wr_wait:
c4iw_put_wr_wait(mhp->wr_waitp);
err_free_mhp:
diff --git a/drivers/infiniband/hw/efa/efa_com.c b/drivers/infiniband/hw/efa/efa_com.c
index 00e339abc2c1..090279dd0ab9 100644
--- a/drivers/infiniband/hw/efa/efa_com.c
+++ b/drivers/infiniband/hw/efa/efa_com.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause
/*
- * Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved.
+ * Copyright 2018-2026 Amazon.com, Inc. or its affiliates. All rights reserved.
*/
#include "efa_com.h"
@@ -499,7 +499,6 @@ static int efa_com_wait_and_process_admin_cq_polling(struct efa_comp_ctx *comp_c
{
unsigned long timeout;
unsigned long flags;
- int err;
timeout = jiffies + usecs_to_jiffies(aq->completion_timeout);
@@ -519,24 +518,20 @@ static int efa_com_wait_and_process_admin_cq_polling(struct efa_comp_ctx *comp_c
atomic64_inc(&aq->stats.no_completion);
clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
- err = -ETIME;
- goto out;
+ return -ETIME;
}
msleep(aq->poll_interval);
}
- err = efa_com_comp_status_to_errno(comp_ctx->user_cqe->acq_common_descriptor.status);
-out:
- efa_com_put_comp_ctx(aq, comp_ctx);
- return err;
+ return efa_com_comp_status_to_errno(
+ comp_ctx->user_cqe->acq_common_descriptor.status);
}
static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *comp_ctx,
struct efa_com_admin_queue *aq)
{
unsigned long flags;
- int err;
wait_for_completion_timeout(&comp_ctx->wait_event,
usecs_to_jiffies(aq->completion_timeout));
@@ -570,14 +565,11 @@ static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *com
comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc);
clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state);
- err = -ETIME;
- goto out;
+ return -ETIME;
}
- err = efa_com_comp_status_to_errno(comp_ctx->user_cqe->acq_common_descriptor.status);
-out:
- efa_com_put_comp_ctx(aq, comp_ctx);
- return err;
+ return efa_com_comp_status_to_errno(
+ comp_ctx->user_cqe->acq_common_descriptor.status);
}
/*
@@ -644,13 +636,13 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq,
if (err) {
ibdev_err_ratelimited(
aq->efa_dev,
- "Failed to process command %s (opcode %u) comp_status %d err %d\n",
+ "Failed to process command %s (opcode %u) err %d\n",
efa_com_cmd_str(cmd->aq_common_descriptor.opcode),
- cmd->aq_common_descriptor.opcode,
- comp_ctx->user_cqe->acq_common_descriptor.status, err);
+ cmd->aq_common_descriptor.opcode, err);
atomic64_inc(&aq->stats.cmd_err);
}
+ efa_com_put_comp_ctx(aq, comp_ctx);
up(&aq->avail_cmds);
return err;
diff --git a/drivers/infiniband/hw/irdma/hw.c b/drivers/infiniband/hw/irdma/hw.c
index ad50b77282f8..c1e5bcf6ab60 100644
--- a/drivers/infiniband/hw/irdma/hw.c
+++ b/drivers/infiniband/hw/irdma/hw.c
@@ -1001,9 +1001,9 @@ static int irdma_create_cqp(struct irdma_pci_f *rf)
INIT_LIST_HEAD(&cqp->cqp_avail_reqs);
INIT_LIST_HEAD(&cqp->cqp_pending_reqs);
- /* init the waitqueue of the cqp_requests and add them to the list */
+ /* init the completion of the cqp_requests and add them to the list */
for (i = 0; i < sqsize; i++) {
- init_waitqueue_head(&cqp->cqp_requests[i].waitq);
+ init_completion(&cqp->cqp_requests[i].comp);
list_add_tail(&cqp->cqp_requests[i].list, &cqp->cqp_avail_reqs);
}
init_waitqueue_head(&cqp->remove_wq);
@@ -2103,8 +2103,7 @@ void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq)
cqp_request->compl_info.error = info.error;
if (cqp_request->waiting) {
- WRITE_ONCE(cqp_request->request_done, true);
- wake_up(&cqp_request->waitq);
+ complete_all(&cqp_request->comp);
irdma_put_cqp_request(&rf->cqp, cqp_request);
} else {
if (cqp_request->callback_fcn)
diff --git a/drivers/infiniband/hw/irdma/main.h b/drivers/infiniband/hw/irdma/main.h
index e5dc43442dff..47b95efd51ce 100644
--- a/drivers/infiniband/hw/irdma/main.h
+++ b/drivers/infiniband/hw/irdma/main.h
@@ -155,13 +155,12 @@ struct irdma_cqp_compl_info {
struct irdma_cqp_request {
struct cqp_cmds_info info;
- wait_queue_head_t waitq;
+ struct completion comp;
struct list_head list;
refcount_t refcnt;
void (*callback_fcn)(struct irdma_cqp_request *cqp_request);
void *param;
struct irdma_cqp_compl_info compl_info;
- bool request_done; /* READ/WRITE_ONCE macros operate on it */
bool waiting:1;
bool dynamic:1;
};
diff --git a/drivers/infiniband/hw/irdma/utils.c b/drivers/infiniband/hw/irdma/utils.c
index 45c3f82887ef..1a73e1b74074 100644
--- a/drivers/infiniband/hw/irdma/utils.c
+++ b/drivers/infiniband/hw/irdma/utils.c
@@ -445,7 +445,7 @@ struct irdma_cqp_request *irdma_alloc_and_get_cqp_request(struct irdma_cqp *cqp,
if (cqp_request) {
cqp_request->dynamic = true;
if (wait)
- init_waitqueue_head(&cqp_request->waitq);
+ init_completion(&cqp_request->comp);
}
}
if (!cqp_request) {
@@ -482,7 +482,7 @@ void irdma_free_cqp_request(struct irdma_cqp *cqp,
if (cqp_request->dynamic) {
kfree(cqp_request);
} else {
- WRITE_ONCE(cqp_request->request_done, false);
+ reinit_completion(&cqp_request->comp);
cqp_request->callback_fcn = NULL;
cqp_request->waiting = false;
@@ -516,8 +516,7 @@ irdma_free_pending_cqp_request(struct irdma_cqp *cqp,
{
if (cqp_request->waiting) {
cqp_request->compl_info.error = true;
- WRITE_ONCE(cqp_request->request_done, true);
- wake_up(&cqp_request->waitq);
+ complete_all(&cqp_request->comp);
}
wait_event_timeout(cqp->remove_wq,
refcount_read(&cqp_request->refcnt) == 1, 1000);
@@ -571,9 +570,8 @@ static int irdma_wait_event(struct irdma_pci_f *rf,
cqp_timeout.compl_cqp_cmds = atomic64_read(&rf->sc_dev.cqp->completed_ops);
do {
irdma_cqp_ce_handler(rf, &rf->ccq.sc_cq);
- if (wait_event_timeout(cqp_request->waitq,
- READ_ONCE(cqp_request->request_done),
- msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS)))
+ if (wait_for_completion_timeout(&cqp_request->comp,
+ msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS)))
break;
irdma_check_cqp_progress(&cqp_timeout, &rf->sc_dev);
diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
index 97fa34563552..f86249226b60 100644
--- a/drivers/infiniband/hw/irdma/verbs.c
+++ b/drivers/infiniband/hw/irdma/verbs.c
@@ -602,18 +602,18 @@ static int irdma_setup_umode_qp(struct ib_udata *udata,
iwqp->ctx_info.qp_compl_ctx = req.user_compl_ctx;
iwqp->user_mode = 1;
- if (req.user_wqe_bufs) {
- info->qp_uk_init_info.legacy_mode = ucontext->legacy_mode;
- spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
- iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs,
- &ucontext->qp_reg_mem_list);
- spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
- if (!iwqp->iwpbl) {
- ret = -ENODATA;
- ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n");
- return ret;
- }
+ info->qp_uk_init_info.legacy_mode = ucontext->legacy_mode;
+
+ spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
+ iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs,
+ &ucontext->qp_reg_mem_list);
+ spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
+
+ if (!iwqp->iwpbl) {
+ ret = -ENODATA;
+ ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n");
+ return ret;
}
if (!ucontext->use_raw_attrs) {
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 9e04b595c77f..b0b10b8bedfb 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -649,7 +649,7 @@ int mlx5_ib_arm_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags)
{
struct mlx5_core_dev *mdev = to_mdev(ibcq->device)->mdev;
struct mlx5_ib_cq *cq = to_mcq(ibcq);
- void __iomem *uar_page = mdev->priv.uar->map;
+ void __iomem *uar_page = mdev->priv.bfreg.up->map;
unsigned long irq_flags;
int ret = 0;
@@ -924,7 +924,7 @@ static int create_cq_kernel(struct mlx5_ib_dev *dev, struct mlx5_ib_cq *cq,
cq->buf.frag_buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
- *index = dev->mdev->priv.uar->index;
+ *index = dev->mdev->priv.bfreg.up->index;
return 0;
@@ -1021,15 +1021,18 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
if (cq->create_flags & IB_UVERBS_CQ_FLAGS_IGNORE_OVERRUN)
MLX5_SET(cqc, cqc, oi, 1);
+ if (udata) {
+ cq->mcq.comp = mlx5_add_cq_to_tasklet;
+ cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp;
+ } else {
+ cq->mcq.comp = mlx5_ib_cq_comp;
+ }
+
err = mlx5_core_create_cq(dev->mdev, &cq->mcq, cqb, inlen, out, sizeof(out));
if (err)
goto err_cqb;
mlx5_ib_dbg(dev, "cqn 0x%x\n", cq->mcq.cqn);
- if (udata)
- cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp;
- else
- cq->mcq.comp = mlx5_ib_cq_comp;
cq->mcq.event = mlx5_ib_cq_event;
INIT_LIST_HEAD(&cq->wc_list);
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 2d1cb516d845..42a917c2d0d7 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -4502,17 +4502,6 @@ static void mlx5_ib_stage_cong_debugfs_cleanup(struct mlx5_ib_dev *dev)
mlx5_core_native_port_num(dev->mdev) - 1);
}
-static int mlx5_ib_stage_uar_init(struct mlx5_ib_dev *dev)
-{
- dev->mdev->priv.uar = mlx5_get_uars_page(dev->mdev);
- return PTR_ERR_OR_ZERO(dev->mdev->priv.uar);
-}
-
-static void mlx5_ib_stage_uar_cleanup(struct mlx5_ib_dev *dev)
-{
- mlx5_put_uars_page(dev->mdev, dev->mdev->priv.uar);
-}
-
static int mlx5_ib_stage_bfrag_init(struct mlx5_ib_dev *dev)
{
int err;
@@ -4742,9 +4731,6 @@ static const struct mlx5_ib_profile pf_profile = {
STAGE_CREATE(MLX5_IB_STAGE_CONG_DEBUGFS,
mlx5_ib_stage_cong_debugfs_init,
mlx5_ib_stage_cong_debugfs_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_UAR,
- mlx5_ib_stage_uar_init,
- mlx5_ib_stage_uar_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_BFREG,
mlx5_ib_stage_bfrag_init,
mlx5_ib_stage_bfrag_cleanup),
@@ -4805,9 +4791,6 @@ const struct mlx5_ib_profile raw_eth_profile = {
STAGE_CREATE(MLX5_IB_STAGE_CONG_DEBUGFS,
mlx5_ib_stage_cong_debugfs_init,
mlx5_ib_stage_cong_debugfs_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_UAR,
- mlx5_ib_stage_uar_init,
- mlx5_ib_stage_uar_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_BFREG,
mlx5_ib_stage_bfrag_init,
mlx5_ib_stage_bfrag_cleanup),
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 3135519f1cfd..5c0efac217ea 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -975,7 +975,6 @@ enum mlx5_ib_stages {
MLX5_IB_STAGE_ODP,
MLX5_IB_STAGE_COUNTERS,
MLX5_IB_STAGE_CONG_DEBUGFS,
- MLX5_IB_STAGE_UAR,
MLX5_IB_STAGE_BFREG,
MLX5_IB_STAGE_PRE_IB_REG_UMR,
MLX5_IB_STAGE_WHITELIST_UID,
diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c
index 5499025e8a0a..b1750f2a6dcb 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -54,8 +54,10 @@ struct rvt_dev_info *rvt_alloc_device(size_t size, int nports)
return rdi;
rdi->ports = kcalloc(nports, sizeof(*rdi->ports), GFP_KERNEL);
- if (!rdi->ports)
+ if (!rdi->ports) {
ib_dealloc_device(&rdi->ibdev);
+ return NULL;
+ }
return rdi;
}
diff --git a/drivers/infiniband/sw/rxe/rxe.h b/drivers/infiniband/sw/rxe/rxe.h
index fe7f97066732..120e114c2e1e 100644
--- a/drivers/infiniband/sw/rxe/rxe.h
+++ b/drivers/infiniband/sw/rxe/rxe.h
@@ -100,43 +100,6 @@
#define rxe_info_mw(mw, fmt, ...) ibdev_info_ratelimited((mw)->ibmw.device, \
"mw#%d %s: " fmt, (mw)->elem.index, __func__, ##__VA_ARGS__)
-/* responder states */
-enum resp_states {
- RESPST_NONE,
- RESPST_GET_REQ,
- RESPST_CHK_PSN,
- RESPST_CHK_OP_SEQ,
- RESPST_CHK_OP_VALID,
- RESPST_CHK_RESOURCE,
- RESPST_CHK_LENGTH,
- RESPST_CHK_RKEY,
- RESPST_EXECUTE,
- RESPST_READ_REPLY,
- RESPST_ATOMIC_REPLY,
- RESPST_ATOMIC_WRITE_REPLY,
- RESPST_PROCESS_FLUSH,
- RESPST_COMPLETE,
- RESPST_ACKNOWLEDGE,
- RESPST_CLEANUP,
- RESPST_DUPLICATE_REQUEST,
- RESPST_ERR_MALFORMED_WQE,
- RESPST_ERR_UNSUPPORTED_OPCODE,
- RESPST_ERR_MISALIGNED_ATOMIC,
- RESPST_ERR_PSN_OUT_OF_SEQ,
- RESPST_ERR_MISSING_OPCODE_FIRST,
- RESPST_ERR_MISSING_OPCODE_LAST_C,
- RESPST_ERR_MISSING_OPCODE_LAST_D1E,
- RESPST_ERR_TOO_MANY_RDMA_ATM_REQ,
- RESPST_ERR_RNR,
- RESPST_ERR_RKEY_VIOLATION,
- RESPST_ERR_INVALIDATE_RKEY,
- RESPST_ERR_LENGTH,
- RESPST_ERR_CQ_OVERFLOW,
- RESPST_ERROR,
- RESPST_DONE,
- RESPST_EXIT,
-};
-
void rxe_set_mtu(struct rxe_dev *rxe, unsigned int dev_mtu);
int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name,
diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h
index ded46119151b..1c78b9d36da4 100644
--- a/drivers/infiniband/sw/rxe/rxe_loc.h
+++ b/drivers/infiniband/sw/rxe/rxe_loc.h
@@ -58,6 +58,7 @@ int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma);
/* rxe_mr.c */
u8 rxe_get_next_key(u32 last_key);
+void rxe_mr_init(int access, struct rxe_mr *mr);
void rxe_mr_init_dma(int access, struct rxe_mr *mr);
int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
int access, struct rxe_mr *mr);
diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index 83e325b76f67..818f1693bdec 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -46,7 +46,7 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length)
}
}
-static void rxe_mr_init(int access, struct rxe_mr *mr)
+void rxe_mr_init(int access, struct rxe_mr *mr)
{
u32 key = mr->elem.index << 8 | rxe_get_next_key(-1);
@@ -73,14 +73,46 @@ void rxe_mr_init_dma(int access, struct rxe_mr *mr)
mr->ibmr.type = IB_MR_TYPE_DMA;
}
+/*
+ * Convert iova to page_info index. The page_info stores pages of size
+ * PAGE_SIZE, but MRs can have different page sizes. This function
+ * handles the conversion for all cases:
+ *
+ * 1. mr->page_size > PAGE_SIZE:
+ * The MR's iova may not be aligned to mr->page_size. We use the
+ * aligned base (iova & page_mask) as reference, then calculate
+ * which PAGE_SIZE sub-page the iova falls into.
+ *
+ * 2. mr->page_size <= PAGE_SIZE:
+ * Use simple shift arithmetic since each page_info entry corresponds
+ * to one or more MR pages.
+ */
static unsigned long rxe_mr_iova_to_index(struct rxe_mr *mr, u64 iova)
{
- return (iova >> mr->page_shift) - (mr->ibmr.iova >> mr->page_shift);
+ int idx;
+
+ if (mr_page_size(mr) > PAGE_SIZE)
+ idx = (iova - (mr->ibmr.iova & mr->page_mask)) >> PAGE_SHIFT;
+ else
+ idx = (iova >> mr->page_shift) -
+ (mr->ibmr.iova >> mr->page_shift);
+
+ WARN_ON(idx >= mr->nbuf);
+ return idx;
}
+/*
+ * Convert iova to offset within the page_info entry.
+ *
+ * For mr_page_size > PAGE_SIZE, the offset is within the system page.
+ * For mr_page_size <= PAGE_SIZE, the offset is within the MR page size.
+ */
static unsigned long rxe_mr_iova_to_page_offset(struct rxe_mr *mr, u64 iova)
{
- return iova & (mr_page_size(mr) - 1);
+ if (mr_page_size(mr) > PAGE_SIZE)
+ return iova & (PAGE_SIZE - 1);
+ else
+ return iova & (mr_page_size(mr) - 1);
}
static bool is_pmem_page(struct page *pg)
@@ -94,37 +126,69 @@ static bool is_pmem_page(struct page *pg)
static int rxe_mr_fill_pages_from_sgt(struct rxe_mr *mr, struct sg_table *sgt)
{
- XA_STATE(xas, &mr->page_list, 0);
struct sg_page_iter sg_iter;
struct page *page;
bool persistent = !!(mr->access & IB_ACCESS_FLUSH_PERSISTENT);
+ WARN_ON(mr_page_size(mr) != PAGE_SIZE);
+
__sg_page_iter_start(&sg_iter, sgt->sgl, sgt->orig_nents, 0);
if (!__sg_page_iter_next(&sg_iter))
return 0;
- do {
- xas_lock(&xas);
- while (true) {
- page = sg_page_iter_page(&sg_iter);
-
- if (persistent && !is_pmem_page(page)) {
- rxe_dbg_mr(mr, "Page can't be persistent\n");
- xas_set_err(&xas, -EINVAL);
- break;
- }
+ while (true) {
+ page = sg_page_iter_page(&sg_iter);
- xas_store(&xas, page);
- if (xas_error(&xas))
- break;
- xas_next(&xas);
- if (!__sg_page_iter_next(&sg_iter))
- break;
+ if (persistent && !is_pmem_page(page)) {
+ rxe_dbg_mr(mr, "Page can't be persistent\n");
+ return -EINVAL;
}
- xas_unlock(&xas);
- } while (xas_nomem(&xas, GFP_KERNEL));
- return xas_error(&xas);
+ mr->page_info[mr->nbuf].page = page;
+ mr->page_info[mr->nbuf].offset = 0;
+ mr->nbuf++;
+
+ if (!__sg_page_iter_next(&sg_iter))
+ break;
+ }
+
+ return 0;
+}
+
+static int __alloc_mr_page_info(struct rxe_mr *mr, int num_pages)
+{
+ mr->page_info = kcalloc(num_pages, sizeof(struct rxe_mr_page),
+ GFP_KERNEL);
+ if (!mr->page_info)
+ return -ENOMEM;
+
+ mr->max_allowed_buf = num_pages;
+ mr->nbuf = 0;
+
+ return 0;
+}
+
+static int alloc_mr_page_info(struct rxe_mr *mr, int num_pages)
+{
+ int ret;
+
+ WARN_ON(mr->num_buf);
+ ret = __alloc_mr_page_info(mr, num_pages);
+ if (ret)
+ return ret;
+
+ mr->num_buf = num_pages;
+
+ return 0;
+}
+
+static void free_mr_page_info(struct rxe_mr *mr)
+{
+ if (!mr->page_info)
+ return;
+
+ kfree(mr->page_info);
+ mr->page_info = NULL;
}
int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
@@ -135,8 +199,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
rxe_mr_init(access, mr);
- xa_init(&mr->page_list);
-
umem = ib_umem_get(&rxe->ib_dev, start, length, access);
if (IS_ERR(umem)) {
rxe_dbg_mr(mr, "Unable to pin memory region err = %d\n",
@@ -144,46 +206,24 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length,
return PTR_ERR(umem);
}
+ err = alloc_mr_page_info(mr, ib_umem_num_pages(umem));
+ if (err)
+ goto err2;
+
err = rxe_mr_fill_pages_from_sgt(mr, &umem->sgt_append.sgt);
- if (err) {
- ib_umem_release(umem);
- return err;
- }
+ if (err)
+ goto err1;
mr->umem = umem;
mr->ibmr.type = IB_MR_TYPE_USER;
mr->state = RXE_MR_STATE_VALID;
return 0;
-}
-
-static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf)
-{
- XA_STATE(xas, &mr->page_list, 0);
- int i = 0;
- int err;
-
- xa_init(&mr->page_list);
-
- do {
- xas_lock(&xas);
- while (i != num_buf) {
- xas_store(&xas, XA_ZERO_ENTRY);
- if (xas_error(&xas))
- break;
- xas_next(&xas);
- i++;
- }
- xas_unlock(&xas);
- } while (xas_nomem(&xas, GFP_KERNEL));
-
- err = xas_error(&xas);
- if (err)
- return err;
-
- mr->num_buf = num_buf;
-
- return 0;
+err1:
+ free_mr_page_info(mr);
+err2:
+ ib_umem_release(umem);
+ return err;
}
int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr)
@@ -193,7 +233,7 @@ int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr)
/* always allow remote access for FMRs */
rxe_mr_init(RXE_ACCESS_REMOTE, mr);
- err = rxe_mr_alloc(mr, max_pages);
+ err = alloc_mr_page_info(mr, max_pages);
if (err)
goto err1;
@@ -206,26 +246,43 @@ int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr)
return err;
}
+/*
+ * I) MRs with page_size >= PAGE_SIZE,
+ * Split a large MR page (mr->page_size) into multiple PAGE_SIZE
+ * sub-pages and store them in page_info, offset is always 0.
+ *
+ * Called when mr->page_size > PAGE_SIZE. Each call to rxe_set_page()
+ * represents one mr->page_size region, which we must split into
+ * (mr->page_size >> PAGE_SHIFT) individual pages.
+ *
+ * II) MRs with page_size < PAGE_SIZE,
+ * Save each PAGE_SIZE page and its offset within the system page in page_info.
+ */
static int rxe_set_page(struct ib_mr *ibmr, u64 dma_addr)
{
struct rxe_mr *mr = to_rmr(ibmr);
- struct page *page = ib_virt_dma_to_page(dma_addr);
bool persistent = !!(mr->access & IB_ACCESS_FLUSH_PERSISTENT);
- int err;
+ u32 i, pages_per_mr = mr_page_size(mr) >> PAGE_SHIFT;
- if (persistent && !is_pmem_page(page)) {
- rxe_dbg_mr(mr, "Page cannot be persistent\n");
- return -EINVAL;
- }
+ pages_per_mr = MAX(1, pages_per_mr);
- if (unlikely(mr->nbuf == mr->num_buf))
- return -ENOMEM;
+ for (i = 0; i < pages_per_mr; i++) {
+ u64 addr = dma_addr + i * PAGE_SIZE;
+ struct page *sub_page = ib_virt_dma_to_page(addr);
- err = xa_err(xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL));
- if (err)
- return err;
+ if (unlikely(mr->nbuf >= mr->max_allowed_buf))
+ return -ENOMEM;
+
+ if (persistent && !is_pmem_page(sub_page)) {
+ rxe_dbg_mr(mr, "Page cannot be persistent\n");
+ return -EINVAL;
+ }
+
+ mr->page_info[mr->nbuf].page = sub_page;
+ mr->page_info[mr->nbuf].offset = addr & (PAGE_SIZE - 1);
+ mr->nbuf++;
+ }
- mr->nbuf++;
return 0;
}
@@ -235,6 +292,31 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl,
struct rxe_mr *mr = to_rmr(ibmr);
unsigned int page_size = mr_page_size(mr);
+ /*
+ * Ensure page_size and PAGE_SIZE are compatible for mapping.
+ * We require one to be a multiple of the other for correct
+ * iova-to-page conversion.
+ */
+ if (!IS_ALIGNED(page_size, PAGE_SIZE) &&
+ !IS_ALIGNED(PAGE_SIZE, page_size)) {
+ rxe_dbg_mr(mr, "MR page size %u must be compatible with PAGE_SIZE %lu\n",
+ page_size, PAGE_SIZE);
+ return -EINVAL;
+ }
+
+ if (mr_page_size(mr) > PAGE_SIZE) {
+ /* resize page_info if needed */
+ u32 map_mr_pages = (page_size >> PAGE_SHIFT) * mr->num_buf;
+
+ if (map_mr_pages > mr->max_allowed_buf) {
+ rxe_dbg_mr(mr, "requested pages %u exceed max %u\n",
+ map_mr_pages, mr->max_allowed_buf);
+ free_mr_page_info(mr);
+ if (__alloc_mr_page_info(mr, map_mr_pages))
+ return -ENOMEM;
+ }
+ }
+
mr->nbuf = 0;
mr->page_shift = ilog2(page_size);
mr->page_mask = ~((u64)page_size - 1);
@@ -246,30 +328,30 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl,
static int rxe_mr_copy_xarray(struct rxe_mr *mr, u64 iova, void *addr,
unsigned int length, enum rxe_mr_copy_dir dir)
{
- unsigned int page_offset = rxe_mr_iova_to_page_offset(mr, iova);
- unsigned long index = rxe_mr_iova_to_index(mr, iova);
unsigned int bytes;
- struct page *page;
- void *va;
+ u8 *va;
while (length) {
- page = xa_load(&mr->page_list, index);
- if (!page)
+ unsigned long index = rxe_mr_iova_to_index(mr, iova);
+ struct rxe_mr_page *info = &mr->page_info[index];
+ unsigned int page_offset = rxe_mr_iova_to_page_offset(mr, iova);
+
+ if (!info->page)
return -EFAULT;
- bytes = min_t(unsigned int, length,
- mr_page_size(mr) - page_offset);
- va = kmap_local_page(page);
+ page_offset += info->offset;
+ bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset);
+ va = kmap_local_page(info->page);
+
if (dir == RXE_FROM_MR_OBJ)
memcpy(addr, va + page_offset, bytes);
else
memcpy(va + page_offset, addr, bytes);
kunmap_local(va);
- page_offset = 0;
addr += bytes;
+ iova += bytes;
length -= bytes;
- index++;
}
return 0;
@@ -424,9 +506,6 @@ int copy_data(
int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
{
- unsigned int page_offset;
- unsigned long index;
- struct page *page;
unsigned int bytes;
int err;
u8 *va;
@@ -446,15 +525,17 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
return err;
while (length > 0) {
- index = rxe_mr_iova_to_index(mr, iova);
- page = xa_load(&mr->page_list, index);
- page_offset = rxe_mr_iova_to_page_offset(mr, iova);
- if (!page)
+ unsigned long index = rxe_mr_iova_to_index(mr, iova);
+ struct rxe_mr_page *info = &mr->page_info[index];
+ unsigned int page_offset = rxe_mr_iova_to_page_offset(mr, iova);
+
+ if (!info->page)
return -EFAULT;
- bytes = min_t(unsigned int, length,
- mr_page_size(mr) - page_offset);
- va = kmap_local_page(page);
+ page_offset += info->offset;
+ bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset);
+
+ va = kmap_local_page(info->page);
arch_wb_cache_pmem(va + page_offset, bytes);
kunmap_local(va);
@@ -488,6 +569,7 @@ int rxe_mr_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
} else {
unsigned long index;
int err;
+ struct rxe_mr_page *info;
err = mr_check_range(mr, iova, sizeof(value));
if (err) {
@@ -496,9 +578,12 @@ int rxe_mr_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode,
}
page_offset = rxe_mr_iova_to_page_offset(mr, iova);
index = rxe_mr_iova_to_index(mr, iova);
- page = xa_load(&mr->page_list, index);
- if (!page)
+ info = &mr->page_info[index];
+ if (!info->page)
return RESPST_ERR_RKEY_VIOLATION;
+
+ page_offset += info->offset;
+ page = info->page;
}
if (unlikely(page_offset & 0x7)) {
@@ -545,6 +630,7 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value)
} else {
unsigned long index;
int err;
+ struct rxe_mr_page *info;
/* See IBA oA19-28 */
err = mr_check_range(mr, iova, sizeof(value));
@@ -554,9 +640,12 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value)
}
page_offset = rxe_mr_iova_to_page_offset(mr, iova);
index = rxe_mr_iova_to_index(mr, iova);
- page = xa_load(&mr->page_list, index);
- if (!page)
+ info = &mr->page_info[index];
+ if (!info->page)
return RESPST_ERR_RKEY_VIOLATION;
+
+ page_offset += info->offset;
+ page = info->page;
}
/* See IBA A19.4.2 */
@@ -728,5 +817,5 @@ void rxe_mr_cleanup(struct rxe_pool_elem *elem)
ib_umem_release(mr->umem);
if (mr->ibmr.type != IB_MR_TYPE_DMA)
- xa_destroy(&mr->page_list);
+ free_mr_page_info(mr);
}
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h
index 6573ceec0ef5..49bac90a8f27 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.h
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.h
@@ -126,6 +126,43 @@ struct rxe_comp_info {
u32 rnr_retry;
};
+/* responder states */
+enum resp_states {
+ RESPST_NONE,
+ RESPST_GET_REQ,
+ RESPST_CHK_PSN,
+ RESPST_CHK_OP_SEQ,
+ RESPST_CHK_OP_VALID,
+ RESPST_CHK_RESOURCE,
+ RESPST_CHK_LENGTH,
+ RESPST_CHK_RKEY,
+ RESPST_EXECUTE,
+ RESPST_READ_REPLY,
+ RESPST_ATOMIC_REPLY,
+ RESPST_ATOMIC_WRITE_REPLY,
+ RESPST_PROCESS_FLUSH,
+ RESPST_COMPLETE,
+ RESPST_ACKNOWLEDGE,
+ RESPST_CLEANUP,
+ RESPST_DUPLICATE_REQUEST,
+ RESPST_ERR_MALFORMED_WQE,
+ RESPST_ERR_UNSUPPORTED_OPCODE,
+ RESPST_ERR_MISALIGNED_ATOMIC,
+ RESPST_ERR_PSN_OUT_OF_SEQ,
+ RESPST_ERR_MISSING_OPCODE_FIRST,
+ RESPST_ERR_MISSING_OPCODE_LAST_C,
+ RESPST_ERR_MISSING_OPCODE_LAST_D1E,
+ RESPST_ERR_TOO_MANY_RDMA_ATM_REQ,
+ RESPST_ERR_RNR,
+ RESPST_ERR_RKEY_VIOLATION,
+ RESPST_ERR_INVALIDATE_RKEY,
+ RESPST_ERR_LENGTH,
+ RESPST_ERR_CQ_OVERFLOW,
+ RESPST_ERROR,
+ RESPST_DONE,
+ RESPST_EXIT,
+};
+
enum rdatm_res_state {
rdatm_res_state_next,
rdatm_res_state_new,
@@ -298,6 +335,11 @@ static inline int rkey_is_mw(u32 rkey)
return (index >= RXE_MIN_MW_INDEX) && (index <= RXE_MAX_MW_INDEX);
}
+struct rxe_mr_page {
+ struct page *page;
+ unsigned int offset; /* offset in system page */
+};
+
struct rxe_mr {
struct rxe_pool_elem elem;
struct ib_mr ibmr;
@@ -314,10 +356,13 @@ struct rxe_mr {
unsigned int page_shift;
u64 page_mask;
+ /* size of page_info when mr allocated */
u32 num_buf;
+ /* real size of page_info */
+ u32 max_allowed_buf;
u32 nbuf;
- struct xarray page_list;
+ struct rxe_mr_page *page_info;
};
static inline unsigned int mr_page_size(struct rxe_mr *mr)
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index b9a14c3ef440..54aa09e1174d 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -912,7 +912,9 @@ static void free_ga_log(struct amd_iommu *iommu)
{
#ifdef CONFIG_IRQ_REMAP
iommu_free_pages(iommu->ga_log, get_order(GA_LOG_SIZE));
+ iommu->ga_log = NULL;
iommu_free_pages(iommu->ga_log_tail, get_order(8));
+ iommu->ga_log_tail = NULL;
#endif
}
@@ -957,6 +959,9 @@ static int iommu_init_ga_log(struct amd_iommu *iommu)
if (!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir))
return 0;
+ if (iommu->ga_log && iommu->ga_log_tail)
+ return 0;
+
iommu->ga_log = iommu_alloc_pages(GFP_KERNEL, get_order(GA_LOG_SIZE));
if (!iommu->ga_log)
goto err_out;
@@ -1787,13 +1792,8 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h,
else
iommu->mmio_phys_end = MMIO_CNTR_CONF_OFFSET;
- /*
- * Note: GA (128-bit IRTE) mode requires cmpxchg16b supports.
- * GAM also requires GA mode. Therefore, we need to
- * check cmpxchg16b support before enabling it.
- */
- if (!boot_cpu_has(X86_FEATURE_CX16) ||
- ((h->efr_attr & (0x1 << IOMMU_FEAT_GASUP_SHIFT)) == 0))
+ /* GAM requires GA mode. */
+ if ((h->efr_attr & (0x1 << IOMMU_FEAT_GASUP_SHIFT)) == 0)
amd_iommu_guest_ir = AMD_IOMMU_GUEST_IR_LEGACY;
break;
case 0x11:
@@ -1803,13 +1803,8 @@ static int __init init_iommu_one(struct amd_iommu *iommu, struct ivhd_header *h,
else
iommu->mmio_phys_end = MMIO_CNTR_CONF_OFFSET;
- /*
- * Note: GA (128-bit IRTE) mode requires cmpxchg16b supports.
- * XT, GAM also requires GA mode. Therefore, we need to
- * check cmpxchg16b support before enabling them.
- */
- if (!boot_cpu_has(X86_FEATURE_CX16) ||
- ((h->efr_reg & (0x1 << IOMMU_EFR_GASUP_SHIFT)) == 0)) {
+ /* XT and GAM require GA mode. */
+ if ((h->efr_reg & (0x1 << IOMMU_EFR_GASUP_SHIFT)) == 0) {
amd_iommu_guest_ir = AMD_IOMMU_GUEST_IR_LEGACY;
break;
}
@@ -2927,7 +2922,7 @@ static void disable_iommus(void)
* disable suspend until real resume implemented
*/
-static void amd_iommu_resume(void)
+static void amd_iommu_resume(void *data)
{
struct amd_iommu *iommu;
@@ -2941,7 +2936,7 @@ static void amd_iommu_resume(void)
amd_iommu_enable_interrupts();
}
-static int amd_iommu_suspend(void)
+static int amd_iommu_suspend(void *data)
{
/* disable IOMMUs to go out of the way for BIOS */
disable_iommus();
@@ -2949,11 +2944,15 @@ static int amd_iommu_suspend(void)
return 0;
}
-static struct syscore_ops amd_iommu_syscore_ops = {
+static const struct syscore_ops amd_iommu_syscore_ops = {
.suspend = amd_iommu_suspend,
.resume = amd_iommu_resume,
};
+static struct syscore amd_iommu_syscore = {
+ .ops = &amd_iommu_syscore_ops,
+};
+
static void __init free_iommu_resources(void)
{
kmem_cache_destroy(amd_iommu_irq_cache);
@@ -3106,6 +3105,12 @@ static int __init early_amd_iommu_init(void)
return -EINVAL;
}
+ if (!boot_cpu_has(X86_FEATURE_CX16)) {
+ pr_err("Failed to initialize. The CMPXCHG16B feature is required.\n");
+ ret = -EINVAL;
+ goto out;
+ }
+
/*
* Validate checksum here so we don't need to do it when
* we actually parse the table
@@ -3353,7 +3358,7 @@ static int __init state_next(void)
init_state = IOMMU_ENABLED;
break;
case IOMMU_ENABLED:
- register_syscore_ops(&amd_iommu_syscore_ops);
+ register_syscore(&amd_iommu_syscore);
iommu_snp_enable();
amd_iommu_apply_erratum_snp();
@@ -3467,12 +3472,12 @@ int __init amd_iommu_enable(void)
void amd_iommu_disable(void)
{
- amd_iommu_suspend();
+ amd_iommu_suspend(NULL);
}
int amd_iommu_reenable(int mode)
{
- amd_iommu_resume();
+ amd_iommu_resume(NULL);
return 0;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c b/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c
index 2c7b668c8423..3207fa983606 100644
--- a/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c
+++ b/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c
@@ -778,6 +778,12 @@ static int tegra241_cmdqv_init_structures(struct arm_smmu_device *smmu)
ret = tegra241_cmdqv_init_vintf(cmdqv, 0, vintf);
if (ret) {
dev_err(cmdqv->dev, "failed to init vintf0: %d\n", ret);
+ /*
+ * tegra241_cmdqv_init_vintf() failed to publish the vintf0 to
+ * cmdqv->vintfs[], so the probe unwind path that goes through
+ * cmdqv->vintfs[] would miss it. Free it here.
+ */
+ kfree(vintf);
return ret;
}
@@ -799,16 +805,22 @@ static int tegra241_cmdqv_init_structures(struct arm_smmu_device *smmu)
static struct dentry *cmdqv_debugfs_dir;
#endif
-static struct arm_smmu_device *
-__tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
- int irq)
+/*
+ * Probe the CMDQV and reallocate @smmu into the larger cmdqv->smmu.
+ *
+ * devm_krealloc() may relocate and free the original @smmu, so update *smmu to
+ * the new pointer once it succeeds. The error paths after it do the same, so a
+ * caller falling back keeps a live @smmu instead of the freed original.
+ */
+static int __tegra241_cmdqv_probe(struct arm_smmu_device **smmu,
+ struct resource *res, int irq)
{
static const struct arm_smmu_impl_ops init_ops = {
.init_structures = tegra241_cmdqv_init_structures,
.device_remove = tegra241_cmdqv_remove,
};
- struct tegra241_cmdqv *cmdqv = NULL;
- struct arm_smmu_device *new_smmu;
+ struct device *dev = (*smmu)->dev;
+ struct tegra241_cmdqv *cmdqv;
void __iomem *base;
u32 regval;
int ret;
@@ -817,25 +829,28 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
base = ioremap(res->start, resource_size(res));
if (!base) {
- dev_err(smmu->dev, "failed to ioremap\n");
- return NULL;
+ dev_err(dev, "failed to ioremap\n");
+ return -ENOMEM;
}
regval = readl(base + TEGRA241_CMDQV_CONFIG);
if (disable_cmdqv) {
- dev_info(smmu->dev, "Detected disable_cmdqv=true\n");
+ dev_info(dev, "Detected disable_cmdqv=true\n");
writel(regval & ~CMDQV_EN, base + TEGRA241_CMDQV_CONFIG);
+ ret = -ENODEV;
goto iounmap;
}
- cmdqv = devm_krealloc(smmu->dev, smmu, sizeof(*cmdqv), GFP_KERNEL);
- if (!cmdqv)
+ cmdqv = devm_krealloc(dev, *smmu, sizeof(*cmdqv), GFP_KERNEL);
+ if (!cmdqv) {
+ ret = -ENOMEM;
goto iounmap;
- new_smmu = &cmdqv->smmu;
+ }
+ *smmu = &cmdqv->smmu;
cmdqv->irq = irq;
cmdqv->base = base;
- cmdqv->dev = smmu->impl_dev;
+ cmdqv->dev = (*smmu)->impl_dev;
regval = readl_relaxed(REG_CMDQV(cmdqv, PARAM));
cmdqv->num_vintfs = 1 << FIELD_GET(CMDQV_NUM_VINTF_LOG2, regval);
@@ -844,8 +859,10 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
cmdqv->vintfs =
kcalloc(cmdqv->num_vintfs, sizeof(*cmdqv->vintfs), GFP_KERNEL);
- if (!cmdqv->vintfs)
+ if (!cmdqv->vintfs) {
+ ret = -ENOMEM;
goto iounmap;
+ }
ida_init(&cmdqv->vintf_ids);
@@ -874,38 +891,37 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
#endif
/* Provide init-level ops only, until tegra241_cmdqv_init_structures */
- new_smmu->impl_ops = &init_ops;
+ cmdqv->smmu.impl_ops = &init_ops;
- return new_smmu;
+ return 0;
free_vintfs:
ida_destroy(&cmdqv->vintf_ids);
kfree(cmdqv->vintfs);
iounmap:
iounmap(base);
- return NULL;
+ return ret;
}
struct arm_smmu_device *tegra241_cmdqv_probe(struct arm_smmu_device *smmu)
{
- struct arm_smmu_device *new_smmu;
struct resource *res = NULL;
- int irq;
+ int irq, ret;
if (!smmu->dev->of_node)
res = tegra241_cmdqv_find_acpi_resource(smmu->impl_dev, &irq);
if (!res)
goto out_fallback;
- new_smmu = __tegra241_cmdqv_probe(smmu, res, irq);
+ ret = __tegra241_cmdqv_probe(&smmu, res, irq);
kfree(res);
- if (new_smmu)
- return new_smmu;
+ if (!ret)
+ return smmu;
out_fallback:
dev_info(smmu->impl_dev, "Falling back to standard SMMU CMDQ\n");
smmu->options &= ~ARM_SMMU_OPT_TEGRA241_CMDQV;
put_device(smmu->impl_dev);
- return ERR_PTR(-ENODEV);
+ return smmu;
}
diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 6c1e33b84bd5..94fd05381680 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -1921,7 +1921,7 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8
context_clear_present(context);
__iommu_flush_cache(iommu, context, sizeof(*context));
spin_unlock(&iommu->lock);
- intel_context_flush_no_pasid(info, context, did);
+ intel_context_flush_no_pasid(info, context, did, PCI_DEVID(bus, devfn));
context_clear_entry(context);
__iommu_flush_cache(iommu, context, sizeof(*context));
}
@@ -2504,7 +2504,7 @@ static void iommu_flush_all(void)
}
}
-static int iommu_suspend(void)
+static int iommu_suspend(void *data)
{
struct dmar_drhd_unit *drhd;
struct intel_iommu *iommu = NULL;
@@ -2531,7 +2531,7 @@ static int iommu_suspend(void)
return 0;
}
-static void iommu_resume(void)
+static void iommu_resume(void *data)
{
struct dmar_drhd_unit *drhd;
struct intel_iommu *iommu = NULL;
@@ -2562,14 +2562,18 @@ static void iommu_resume(void)
}
}
-static struct syscore_ops iommu_syscore_ops = {
+static const struct syscore_ops iommu_syscore_ops = {
.resume = iommu_resume,
.suspend = iommu_suspend,
};
+static struct syscore iommu_syscore = {
+ .ops = &iommu_syscore_ops,
+};
+
static void __init init_iommu_pm_ops(void)
{
- register_syscore_ops(&iommu_syscore_ops);
+ register_syscore(&iommu_syscore);
}
#else
@@ -4160,7 +4164,7 @@ static int context_flip_pri(struct device_domain_info *info, bool enable)
if (!ecap_coherent(iommu->ecap))
clflush_cache_range(context, sizeof(*context));
- intel_context_flush_no_pasid(info, context, did);
+ intel_context_flush_no_pasid(info, context, did, PCI_DEVID(bus, devfn));
spin_unlock(&iommu->lock);
return 0;
@@ -4335,9 +4339,9 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
}
spin_unlock_irqrestore(&dmar_domain->lock, flags);
- cache_tag_unassign_domain(dmar_domain, dev, pasid);
- domain_detach_iommu(dmar_domain, iommu);
if (!WARN_ON_ONCE(!dev_pasid)) {
+ cache_tag_unassign_domain(dmar_domain, dev, pasid);
+ domain_detach_iommu(dmar_domain, iommu);
intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
kfree(dev_pasid);
}
diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
index b8d09b00f394..f94d739c2291 100644
--- a/drivers/iommu/intel/iommu.h
+++ b/drivers/iommu/intel/iommu.h
@@ -1297,7 +1297,7 @@ void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long start,
unsigned long end);
void intel_context_flush_no_pasid(struct device_domain_info *info,
- struct context_entry *context, u16 did);
+ struct context_entry *context, u16 did, u16 sid);
int intel_iommu_enable_prq(struct intel_iommu *iommu);
int intel_iommu_finish_prq(struct intel_iommu *iommu);
diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c
index 93e9b9243d5f..dbf1e775fd80 100644
--- a/drivers/iommu/intel/pasid.c
+++ b/drivers/iommu/intel/pasid.c
@@ -717,7 +717,7 @@ static void device_pasid_table_teardown(struct device *dev, u8 bus, u8 devfn)
context_clear_present(context);
__iommu_flush_cache(iommu, context, sizeof(*context));
spin_unlock(&iommu->lock);
- intel_context_flush_no_pasid(info, context, did);
+ intel_context_flush_no_pasid(info, context, did, PCI_DEVID(bus, devfn));
context_clear_entry(context);
__iommu_flush_cache(iommu, context, sizeof(*context));
}
@@ -919,9 +919,12 @@ static void __context_flush_dev_iotlb(struct device_domain_info *info)
* This helper can only be used when IOMMU is working in the legacy mode or
* IOMMU is in scalable mode but all PASID table entries of the device are
* non-present.
+ *
+ * @sid identifies the context entry that was modified, which may be a DMA
+ * alias of @info->dev rather than its own requester ID.
*/
void intel_context_flush_no_pasid(struct device_domain_info *info,
- struct context_entry *context, u16 did)
+ struct context_entry *context, u16 did, u16 sid)
{
struct intel_iommu *iommu = info->iommu;
@@ -931,7 +934,7 @@ void intel_context_flush_no_pasid(struct device_domain_info *info,
* when operating in scalable mode. Therefore the @did value doesn't
* matter in scalable mode.
*/
- iommu->flush.flush_context(iommu, did, PCI_DEVID(info->bus, info->devfn),
+ iommu->flush.flush_context(iommu, did, sid,
DMA_CCMD_MASK_NOBIT, DMA_CCMD_DEVICE_INVL);
/*
diff --git a/drivers/iommu/io-pgfault.c b/drivers/iommu/io-pgfault.c
index 8b5926c1452e..afe977235d72 100644
--- a/drivers/iommu/io-pgfault.c
+++ b/drivers/iommu/io-pgfault.c
@@ -332,16 +332,30 @@ void iopf_group_response(struct iopf_group *group,
.code = status,
};
- /* Only send response if there is a fault report pending */
mutex_lock(&fault_param->lock);
- if (!list_empty(&group->pending_node)) {
- ops->page_response(dev, &group->last_fault, &resp);
- list_del_init(&group->pending_node);
- }
+ ops->page_response(dev, &group->last_fault, &resp);
+ list_del_init(&group->pending_node);
mutex_unlock(&fault_param->lock);
}
EXPORT_SYMBOL_GPL(iopf_group_response);
+/**
+ * iopf_group_dequeue - Dequeue a page fault group from the pending list
+ * @group: the group to dequeue
+ *
+ * The fault handler is responsible for responding to the group after
+ * this function returns.
+ */
+void iopf_group_dequeue(struct iopf_group *group)
+{
+ struct iommu_fault_param *fault_param = group->fault_param;
+
+ mutex_lock(&fault_param->lock);
+ list_del_init(&group->pending_node);
+ mutex_unlock(&fault_param->lock);
+}
+EXPORT_SYMBOL_GPL(iopf_group_dequeue);
+
/**
* iopf_queue_discard_partial - Remove all pending partial fault
* @queue: the queue whose partial faults need to be discarded
diff --git a/drivers/iommu/iommufd/fault.c b/drivers/iommu/iommufd/fault.c
index 74e3c93bce6e..bdaecb6e1f7a 100644
--- a/drivers/iommu/iommufd/fault.c
+++ b/drivers/iommu/iommufd/fault.c
@@ -454,6 +454,8 @@ int iommufd_fault_iopf_handler(struct iopf_group *group)
hwpt = group->attach_handle->domain->fault_data;
fault = hwpt->fault;
+ iopf_group_dequeue(group);
+
spin_lock(&fault->lock);
list_add_tail(&group->node, &fault->deliver);
spin_unlock(&fault->lock);
diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
index fbdeded3d48b..067b175231ad 100644
--- a/drivers/iommu/s390-iommu.c
+++ b/drivers/iommu/s390-iommu.c
@@ -31,6 +31,7 @@ struct s390_domain {
unsigned long *dma_table;
spinlock_t list_lock;
struct rcu_head rcu;
+ u8 origin_type;
};
static struct iommu_domain blocking_domain;
@@ -345,6 +346,7 @@ static struct iommu_domain *s390_domain_alloc_paging(struct device *dev)
s390_domain->domain.geometry.force_aperture = true;
s390_domain->domain.geometry.aperture_start = 0;
s390_domain->domain.geometry.aperture_end = ZPCI_TABLE_SIZE_RT - 1;
+ s390_domain->origin_type = ZPCI_TABLE_TYPE_RTX;
spin_lock_init(&s390_domain->list_lock);
INIT_LIST_HEAD_RCU(&s390_domain->devices);
@@ -381,6 +383,61 @@ static void zdev_s390_domain_update(struct zpci_dev *zdev,
spin_unlock_irqrestore(&zdev->dom_lock, flags);
}
+static u64 get_iota_region_flag(struct s390_domain *domain)
+{
+ switch (domain->origin_type) {
+ case ZPCI_TABLE_TYPE_RTX:
+ return ZPCI_IOTA_RTTO_FLAG;
+ case ZPCI_TABLE_TYPE_RSX:
+ return ZPCI_IOTA_RSTO_FLAG;
+ case ZPCI_TABLE_TYPE_RFX:
+ return ZPCI_IOTA_RFTO_FLAG;
+ default:
+ WARN_ONCE(1, "Invalid IOMMU table (%x)\n", domain->origin_type);
+ return 0;
+ }
+}
+
+static int s390_iommu_domain_reg_ioat(struct zpci_dev *zdev,
+ struct iommu_domain *domain, u8 *status)
+{
+ struct s390_domain *s390_domain;
+ int rc = 0;
+ u64 iota;
+
+ switch (domain->type) {
+ case IOMMU_DOMAIN_IDENTITY:
+ rc = zpci_register_ioat(zdev, 0, zdev->start_dma,
+ zdev->end_dma, 0, status);
+ break;
+ case IOMMU_DOMAIN_BLOCKED:
+ /* Nothing to do in this case */
+ break;
+ default:
+ s390_domain = to_s390_domain(domain);
+ iota = virt_to_phys(s390_domain->dma_table) |
+ get_iota_region_flag(s390_domain);
+ rc = zpci_register_ioat(zdev, 0, zdev->start_dma,
+ zdev->end_dma, iota, status);
+ }
+
+ return rc;
+}
+
+int zpci_iommu_register_ioat(struct zpci_dev *zdev, u8 *status)
+{
+ unsigned long flags;
+ int rc;
+
+ spin_lock_irqsave(&zdev->dom_lock, flags);
+
+ rc = s390_iommu_domain_reg_ioat(zdev, zdev->s390_domain, status);
+
+ spin_unlock_irqrestore(&zdev->dom_lock, flags);
+
+ return rc;
+}
+
static int blocking_domain_attach_device(struct iommu_domain *domain,
struct device *dev)
{
@@ -422,9 +479,9 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
blocking_domain_attach_device(&blocking_domain, dev);
/* If we fail now DMA remains blocked via blocking domain */
- cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(s390_domain->dma_table), &status);
- if (cc && status != ZPCI_PCI_ST_FUNC_NOT_AVAIL)
+ cc = s390_iommu_domain_reg_ioat(zdev, domain, &status);
+ if (cc != ZPCI_CC_OK && cc != ZPCI_CC_INVAL_HANDLE &&
+ (cc != ZPCI_CC_ERR || status != ZPCI_PCI_ST_FUNC_NOT_AVAIL))
return -EIO;
zdev->dma_table = s390_domain->dma_table;
zdev_s390_domain_update(zdev, domain);
diff --git a/drivers/irqchip/exynos-combiner.c b/drivers/irqchip/exynos-combiner.c
index 552aa04ff063..87403f180fb4 100644
--- a/drivers/irqchip/exynos-combiner.c
+++ b/drivers/irqchip/exynos-combiner.c
@@ -200,12 +200,13 @@ static void __init combiner_init(void __iomem *combiner_base,
/**
* combiner_suspend - save interrupt combiner state before suspend
+ * @data: syscore context
*
* Save the interrupt enable set register for all combiner groups since
* the state is lost when the system enters into a sleep state.
*
*/
-static int combiner_suspend(void)
+static int combiner_suspend(void *data)
{
int i;
@@ -218,12 +219,13 @@ static int combiner_suspend(void)
/**
* combiner_resume - restore interrupt combiner state after resume
+ * @data: syscore context
*
* Restore the interrupt enable set register for all combiner groups since
* the state is lost when the system enters into a sleep state on suspend.
*
*/
-static void combiner_resume(void)
+static void combiner_resume(void *data)
{
int i;
@@ -240,11 +242,15 @@ static void combiner_resume(void)
#define combiner_resume NULL
#endif
-static struct syscore_ops combiner_syscore_ops = {
+static const struct syscore_ops combiner_syscore_ops = {
.suspend = combiner_suspend,
.resume = combiner_resume,
};
+static struct syscore combiner_syscore = {
+ .ops = &combiner_syscore_ops,
+};
+
static int __init combiner_of_init(struct device_node *np,
struct device_node *parent)
{
@@ -264,7 +270,7 @@ static int __init combiner_of_init(struct device_node *np,
combiner_init(combiner_base, np);
- register_syscore_ops(&combiner_syscore_ops);
+ register_syscore(&combiner_syscore);
return 0;
}
diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c
index d7c5ef248474..fd6ae54dfb97 100644
--- a/drivers/irqchip/irq-armada-370-xp.c
+++ b/drivers/irqchip/irq-armada-370-xp.c
@@ -724,7 +724,7 @@ static void __exception_irq_entry mpic_handle_irq(struct pt_regs *regs)
} while (1);
}
-static int mpic_suspend(void)
+static int mpic_suspend(void *data)
{
struct mpic *mpic = mpic_data;
@@ -733,7 +733,7 @@ static int mpic_suspend(void)
return 0;
}
-static void mpic_resume(void)
+static void mpic_resume(void *data)
{
struct mpic *mpic = mpic_data;
bool src0, src1;
@@ -786,11 +786,15 @@ static void mpic_resume(void)
mpic_ipi_resume(mpic);
}
-static struct syscore_ops mpic_syscore_ops = {
+static const struct syscore_ops mpic_syscore_ops = {
.suspend = mpic_suspend,
.resume = mpic_resume,
};
+static struct syscore mpic_syscore = {
+ .ops = &mpic_syscore_ops,
+};
+
static int __init mpic_map_region(struct device_node *np, int index,
void __iomem **base, phys_addr_t *phys_base)
{
@@ -903,7 +907,7 @@ static int __init mpic_of_init(struct device_node *node, struct device_node *par
mpic_handle_cascade_irq, mpic);
}
- register_syscore_ops(&mpic_syscore_ops);
+ register_syscore(&mpic_syscore);
return 0;
}
diff --git a/drivers/irqchip/irq-bcm7038-l1.c b/drivers/irqchip/irq-bcm7038-l1.c
index bca43fa99c8f..ecd773866311 100644
--- a/drivers/irqchip/irq-bcm7038-l1.c
+++ b/drivers/irqchip/irq-bcm7038-l1.c
@@ -291,7 +291,7 @@ static int bcm7038_l1_init_one(struct device_node *dn, unsigned int idx,
static LIST_HEAD(bcm7038_l1_intcs_list);
static DEFINE_RAW_SPINLOCK(bcm7038_l1_intcs_lock);
-static int bcm7038_l1_suspend(void)
+static int bcm7038_l1_suspend(void *data)
{
struct bcm7038_l1_chip *intc;
int boot_cpu, word;
@@ -317,7 +317,7 @@ static int bcm7038_l1_suspend(void)
return 0;
}
-static void bcm7038_l1_resume(void)
+static void bcm7038_l1_resume(void *data)
{
struct bcm7038_l1_chip *intc;
int boot_cpu, word;
@@ -338,11 +338,15 @@ static void bcm7038_l1_resume(void)
}
}
-static struct syscore_ops bcm7038_l1_syscore_ops = {
+static const struct syscore_ops bcm7038_l1_syscore_ops = {
.suspend = bcm7038_l1_suspend,
.resume = bcm7038_l1_resume,
};
+static struct syscore bcm7038_l1_syscore = {
+ .ops = &bcm7038_l1_syscore_ops,
+};
+
static int bcm7038_l1_set_wake(struct irq_data *d, unsigned int on)
{
struct bcm7038_l1_chip *intc = irq_data_get_irq_chip_data(d);
@@ -429,7 +433,7 @@ static int bcm7038_l1_of_init(struct device_node *dn, struct device_node *parent
raw_spin_unlock(&bcm7038_l1_intcs_lock);
if (list_is_singular(&bcm7038_l1_intcs_list))
- register_syscore_ops(&bcm7038_l1_syscore_ops);
+ register_syscore(&bcm7038_l1_syscore);
#endif
pr_info("registered BCM7038 L1 intc (%pOF, IRQs: %d)\n",
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 299134fd02c6..fdf8f1bd157e 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4894,7 +4894,7 @@ static void its_enable_quirks(struct its_node *its)
its_quirks, its);
}
-static int its_save_disable(void)
+static int its_save_disable(void *data)
{
struct its_node *its;
int err = 0;
@@ -4930,7 +4930,7 @@ static int its_save_disable(void)
return err;
}
-static void its_restore_enable(void)
+static void its_restore_enable(void *data)
{
struct its_node *its;
int ret;
@@ -4990,11 +4990,15 @@ static void its_restore_enable(void)
raw_spin_unlock(&its_lock);
}
-static struct syscore_ops its_syscore_ops = {
+static const struct syscore_ops its_syscore_ops = {
.suspend = its_save_disable,
.resume = its_restore_enable,
};
+static struct syscore its_syscore = {
+ .ops = &its_syscore_ops,
+};
+
static void __init __iomem *its_map_one(struct resource *res, int *err)
{
void __iomem *its_base;
@@ -5772,7 +5776,7 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
}
}
- register_syscore_ops(&its_syscore_ops);
+ register_syscore(&its_syscore);
return 0;
}
diff --git a/drivers/irqchip/irq-i8259.c b/drivers/irqchip/irq-i8259.c
index 115bdcffab24..f133f6176a12 100644
--- a/drivers/irqchip/irq-i8259.c
+++ b/drivers/irqchip/irq-i8259.c
@@ -202,13 +202,13 @@ static void mask_and_ack_8259A(struct irq_data *d)
}
}
-static void i8259A_resume(void)
+static void i8259A_resume(void *data)
{
if (i8259A_auto_eoi >= 0)
init_8259A(i8259A_auto_eoi);
}
-static void i8259A_shutdown(void)
+static void i8259A_shutdown(void *data)
{
/* Put the i8259A into a quiescent state that
* the kernel initialization code can get it
@@ -220,11 +220,15 @@ static void i8259A_shutdown(void)
}
}
-static struct syscore_ops i8259_syscore_ops = {
+static const struct syscore_ops i8259_syscore_ops = {
.resume = i8259A_resume,
.shutdown = i8259A_shutdown,
};
+static struct syscore i8259_syscore = {
+ .ops = &i8259_syscore_ops,
+};
+
static void init_8259A(int auto_eoi)
{
unsigned long flags;
@@ -320,7 +324,7 @@ struct irq_domain * __init __init_i8259_irqs(struct device_node *node)
if (request_irq(irq, no_action, IRQF_NO_THREAD, "cascade", NULL))
pr_err("Failed to register cascade interrupt\n");
- register_syscore_ops(&i8259_syscore_ops);
+ register_syscore(&i8259_syscore);
return domain;
}
diff --git a/drivers/irqchip/irq-imx-gpcv2.c b/drivers/irqchip/irq-imx-gpcv2.c
index 8a0e82067924..2107729cf0dd 100644
--- a/drivers/irqchip/irq-imx-gpcv2.c
+++ b/drivers/irqchip/irq-imx-gpcv2.c
@@ -33,7 +33,7 @@ static void __iomem *gpcv2_idx_to_reg(struct gpcv2_irqchip_data *cd, int i)
return cd->gpc_base + cd->cpu2wakeup + i * 4;
}
-static int gpcv2_wakeup_source_save(void)
+static int gpcv2_wakeup_source_save(void *data)
{
struct gpcv2_irqchip_data *cd;
void __iomem *reg;
@@ -52,7 +52,7 @@ static int gpcv2_wakeup_source_save(void)
return 0;
}
-static void gpcv2_wakeup_source_restore(void)
+static void gpcv2_wakeup_source_restore(void *data)
{
struct gpcv2_irqchip_data *cd;
int i;
@@ -65,9 +65,13 @@ static void gpcv2_wakeup_source_restore(void)
writel_relaxed(cd->saved_irq_mask[i], gpcv2_idx_to_reg(cd, i));
}
-static struct syscore_ops imx_gpcv2_syscore_ops = {
- .suspend = gpcv2_wakeup_source_save,
- .resume = gpcv2_wakeup_source_restore,
+static const struct syscore_ops gpcv2_syscore_ops = {
+ .suspend = gpcv2_wakeup_source_save,
+ .resume = gpcv2_wakeup_source_restore,
+};
+
+static struct syscore gpcv2_syscore = {
+ .ops = &gpcv2_syscore_ops,
};
static int imx_gpcv2_irq_set_wake(struct irq_data *d, unsigned int on)
@@ -276,7 +280,7 @@ static int __init imx_gpcv2_irqchip_init(struct device_node *node,
writel_relaxed(~0x1, cd->gpc_base + cd->cpu2wakeup);
imx_gpcv2_instance = cd;
- register_syscore_ops(&imx_gpcv2_syscore_ops);
+ register_syscore(&gpcv2_syscore);
/*
* Clear the OF_POPULATED flag set in of_irq_init so that
diff --git a/drivers/irqchip/irq-loongson-eiointc.c b/drivers/irqchip/irq-loongson-eiointc.c
index e24db71a8783..e3b49d018a5e 100644
--- a/drivers/irqchip/irq-loongson-eiointc.c
+++ b/drivers/irqchip/irq-loongson-eiointc.c
@@ -306,21 +306,25 @@ static struct irq_domain *acpi_get_vec_parent(int node, struct acpi_vector_group
return NULL;
}
-static int eiointc_suspend(void)
+static int eiointc_suspend(void *data)
{
return 0;
}
-static void eiointc_resume(void)
+static void eiointc_resume(void *data)
{
eiointc_router_init(0);
}
-static struct syscore_ops eiointc_syscore_ops = {
+static const struct syscore_ops eiointc_syscore_ops = {
.suspend = eiointc_suspend,
.resume = eiointc_resume,
};
+static struct syscore eiointc_syscore = {
+ .ops = &eiointc_syscore_ops,
+};
+
static int __init pch_pic_parse_madt(union acpi_subtable_headers *header,
const unsigned long end)
{
@@ -400,7 +404,7 @@ static int __init eiointc_init(struct eiointc_priv *priv, int parent_irq,
irq_set_chained_handler_and_data(parent_irq, eiointc_irq_dispatch, priv);
if (nr_pics == 1) {
- register_syscore_ops(&eiointc_syscore_ops);
+ register_syscore(&eiointc_syscore);
cpuhp_setup_state_nocalls(CPUHP_AP_IRQ_EIOINTC_STARTING,
"irqchip/loongarch/eiointc:starting",
eiointc_router_init, NULL);
diff --git a/drivers/irqchip/irq-loongson-htpic.c b/drivers/irqchip/irq-loongson-htpic.c
index f4abdf156de7..1c691c4be989 100644
--- a/drivers/irqchip/irq-loongson-htpic.c
+++ b/drivers/irqchip/irq-loongson-htpic.c
@@ -71,15 +71,19 @@ static void htpic_reg_init(void)
writel(0xffff, htpic->base + HTINT_EN_OFF);
}
-static void htpic_resume(void)
+static void htpic_resume(void *data)
{
htpic_reg_init();
}
-struct syscore_ops htpic_syscore_ops = {
+static const struct syscore_ops htpic_syscore_ops = {
.resume = htpic_resume,
};
+static struct syscore htpic_syscore = {
+ .ops = &htpic_syscore_ops,
+};
+
static int __init htpic_of_init(struct device_node *node, struct device_node *parent)
{
unsigned int parent_irq[4];
@@ -130,7 +134,7 @@ static int __init htpic_of_init(struct device_node *node, struct device_node *pa
htpic_irq_dispatch, htpic);
}
- register_syscore_ops(&htpic_syscore_ops);
+ register_syscore(&htpic_syscore);
return 0;
diff --git a/drivers/irqchip/irq-loongson-htvec.c b/drivers/irqchip/irq-loongson-htvec.c
index 5da02c7ad0b3..52952cca7c18 100644
--- a/drivers/irqchip/irq-loongson-htvec.c
+++ b/drivers/irqchip/irq-loongson-htvec.c
@@ -159,7 +159,7 @@ static void htvec_reset(struct htvec *priv)
}
}
-static int htvec_suspend(void)
+static int htvec_suspend(void *data)
{
int i;
@@ -169,7 +169,7 @@ static int htvec_suspend(void)
return 0;
}
-static void htvec_resume(void)
+static void htvec_resume(void *data)
{
int i;
@@ -177,11 +177,15 @@ static void htvec_resume(void)
writel(htvec_priv->saved_vec_en[i], htvec_priv->base + HTVEC_EN_OFF + 4 * i);
}
-static struct syscore_ops htvec_syscore_ops = {
+static const struct syscore_ops htvec_syscore_ops = {
.suspend = htvec_suspend,
.resume = htvec_resume,
};
+static struct syscore htvec_syscore = {
+ .ops = &htvec_syscore_ops,
+};
+
static int htvec_init(phys_addr_t addr, unsigned long size,
int num_parents, int parent_irq[], struct fwnode_handle *domain_handle)
{
@@ -214,7 +218,7 @@ static int htvec_init(phys_addr_t addr, unsigned long size,
htvec_priv = priv;
- register_syscore_ops(&htvec_syscore_ops);
+ register_syscore(&htvec_syscore);
return 0;
diff --git a/drivers/irqchip/irq-loongson-pch-lpc.c b/drivers/irqchip/irq-loongson-pch-lpc.c
index 912bf50a5c7c..3a125f3e4287 100644
--- a/drivers/irqchip/irq-loongson-pch-lpc.c
+++ b/drivers/irqchip/irq-loongson-pch-lpc.c
@@ -151,7 +151,7 @@ static int pch_lpc_disabled(struct pch_lpc *priv)
(readl(priv->base + LPC_INT_STS) == 0xffffffff);
}
-static int pch_lpc_suspend(void)
+static int pch_lpc_suspend(void *data)
{
pch_lpc_priv->saved_reg_ctl = readl(pch_lpc_priv->base + LPC_INT_CTL);
pch_lpc_priv->saved_reg_ena = readl(pch_lpc_priv->base + LPC_INT_ENA);
@@ -159,18 +159,22 @@ static int pch_lpc_suspend(void)
return 0;
}
-static void pch_lpc_resume(void)
+static void pch_lpc_resume(void *data)
{
writel(pch_lpc_priv->saved_reg_ctl, pch_lpc_priv->base + LPC_INT_CTL);
writel(pch_lpc_priv->saved_reg_ena, pch_lpc_priv->base + LPC_INT_ENA);
writel(pch_lpc_priv->saved_reg_pol, pch_lpc_priv->base + LPC_INT_POL);
}
-static struct syscore_ops pch_lpc_syscore_ops = {
+static const struct syscore_ops pch_lpc_syscore_ops = {
.suspend = pch_lpc_suspend,
.resume = pch_lpc_resume,
};
+static struct syscore pch_lpc_syscore = {
+ .ops = &pch_lpc_syscore_ops,
+};
+
int __init pch_lpc_acpi_init(struct irq_domain *parent,
struct acpi_madt_lpc_pic *acpi_pchlpc)
{
@@ -222,7 +226,7 @@ int __init pch_lpc_acpi_init(struct irq_domain *parent,
pch_lpc_priv = priv;
pch_lpc_handle = irq_handle;
- register_syscore_ops(&pch_lpc_syscore_ops);
+ register_syscore(&pch_lpc_syscore);
return 0;
diff --git a/drivers/irqchip/irq-loongson-pch-pic.c b/drivers/irqchip/irq-loongson-pch-pic.c
index 69efda35a8e7..75a256e83c5e 100644
--- a/drivers/irqchip/irq-loongson-pch-pic.c
+++ b/drivers/irqchip/irq-loongson-pch-pic.c
@@ -278,7 +278,7 @@ static void pch_pic_reset(struct pch_pic *priv)
}
}
-static int pch_pic_suspend(void)
+static int pch_pic_suspend(void *data)
{
int i, j;
@@ -296,7 +296,7 @@ static int pch_pic_suspend(void)
return 0;
}
-static void pch_pic_resume(void)
+static void pch_pic_resume(void *data)
{
int i, j;
@@ -313,11 +313,15 @@ static void pch_pic_resume(void)
}
}
-static struct syscore_ops pch_pic_syscore_ops = {
+static const struct syscore_ops pch_pic_syscore_ops = {
.suspend = pch_pic_suspend,
.resume = pch_pic_resume,
};
+static struct syscore pch_pic_syscore = {
+ .ops = &pch_pic_syscore_ops,
+};
+
static int pch_pic_init(phys_addr_t addr, unsigned long size, int vec_base,
struct irq_domain *parent_domain, struct fwnode_handle *domain_handle,
u32 gsi_base)
@@ -356,7 +360,7 @@ static int pch_pic_init(phys_addr_t addr, unsigned long size, int vec_base,
pch_pic_priv[nr_pics++] = priv;
if (nr_pics == 1)
- register_syscore_ops(&pch_pic_syscore_ops);
+ register_syscore(&pch_pic_syscore);
return 0;
diff --git a/drivers/irqchip/irq-mchp-eic.c b/drivers/irqchip/irq-mchp-eic.c
index 8a5baa0987a4..2ad3c1d02d52 100644
--- a/drivers/irqchip/irq-mchp-eic.c
+++ b/drivers/irqchip/irq-mchp-eic.c
@@ -109,7 +109,7 @@ static int mchp_eic_irq_set_wake(struct irq_data *d, unsigned int on)
return 0;
}
-static int mchp_eic_irq_suspend(void)
+static int mchp_eic_irq_suspend(void *data)
{
unsigned int hwirq;
@@ -123,7 +123,7 @@ static int mchp_eic_irq_suspend(void)
return 0;
}
-static void mchp_eic_irq_resume(void)
+static void mchp_eic_irq_resume(void *data)
{
unsigned int hwirq;
@@ -135,11 +135,15 @@ static void mchp_eic_irq_resume(void)
MCHP_EIC_SCFG(hwirq));
}
-static struct syscore_ops mchp_eic_syscore_ops = {
+static const struct syscore_ops mchp_eic_syscore_ops = {
.suspend = mchp_eic_irq_suspend,
.resume = mchp_eic_irq_resume,
};
+static struct syscore mchp_eic_syscore = {
+ .ops = &mchp_eic_syscore_ops,
+};
+
static struct irq_chip mchp_eic_chip = {
.name = "eic",
.flags = IRQCHIP_MASK_ON_SUSPEND | IRQCHIP_SET_TYPE_MASKED,
@@ -256,7 +260,7 @@ static int mchp_eic_init(struct device_node *node, struct device_node *parent)
goto clk_unprepare;
}
- register_syscore_ops(&mchp_eic_syscore_ops);
+ register_syscore(&mchp_eic_syscore);
pr_info("%pOF: EIC registered, nr_irqs %u\n", node, MCHP_EIC_NIRQ);
diff --git a/drivers/irqchip/irq-mst-intc.c b/drivers/irqchip/irq-mst-intc.c
index f6133ae28155..d771eb034612 100644
--- a/drivers/irqchip/irq-mst-intc.c
+++ b/drivers/irqchip/irq-mst-intc.c
@@ -143,7 +143,7 @@ static void mst_intc_polarity_restore(struct mst_intc_chip_data *cd)
writew_relaxed(cd->saved_polarity_conf[i], addr + i * 4);
}
-static void mst_irq_resume(void)
+static void mst_irq_resume(void *data)
{
struct mst_intc_chip_data *cd;
@@ -151,7 +151,7 @@ static void mst_irq_resume(void)
mst_intc_polarity_restore(cd);
}
-static int mst_irq_suspend(void)
+static int mst_irq_suspend(void *data)
{
struct mst_intc_chip_data *cd;
@@ -160,14 +160,18 @@ static int mst_irq_suspend(void)
return 0;
}
-static struct syscore_ops mst_irq_syscore_ops = {
+static const struct syscore_ops mst_irq_syscore_ops = {
.suspend = mst_irq_suspend,
.resume = mst_irq_resume,
};
+static struct syscore mst_irq_syscore = {
+ .ops = &mst_irq_syscore_ops,
+};
+
static int __init mst_irq_pm_init(void)
{
- register_syscore_ops(&mst_irq_syscore_ops);
+ register_syscore(&mst_irq_syscore);
return 0;
}
late_initcall(mst_irq_pm_init);
diff --git a/drivers/irqchip/irq-mtk-cirq.c b/drivers/irqchip/irq-mtk-cirq.c
index 76bc0283e3b9..889f014f0451 100644
--- a/drivers/irqchip/irq-mtk-cirq.c
+++ b/drivers/irqchip/irq-mtk-cirq.c
@@ -199,7 +199,7 @@ static const struct irq_domain_ops cirq_domain_ops = {
};
#ifdef CONFIG_PM_SLEEP
-static int mtk_cirq_suspend(void)
+static int mtk_cirq_suspend(void *data)
{
void __iomem *reg;
u32 value, mask;
@@ -257,7 +257,7 @@ static int mtk_cirq_suspend(void)
return 0;
}
-static void mtk_cirq_resume(void)
+static void mtk_cirq_resume(void *data)
{
void __iomem *reg = mtk_cirq_reg(cirq_data, CIRQ_CONTROL);
u32 value;
@@ -272,14 +272,18 @@ static void mtk_cirq_resume(void)
writel_relaxed(value, reg);
}
-static struct syscore_ops mtk_cirq_syscore_ops = {
+static const struct syscore_ops mtk_cirq_syscore_ops = {
.suspend = mtk_cirq_suspend,
.resume = mtk_cirq_resume,
};
+static struct syscore mtk_cirq_syscore = {
+ .ops = &mtk_cirq_syscore_ops,
+};
+
static void mtk_cirq_syscore_init(void)
{
- register_syscore_ops(&mtk_cirq_syscore_ops);
+ register_syscore(&mtk_cirq_syscore);
}
#else
static inline void mtk_cirq_syscore_init(void) {}
diff --git a/drivers/irqchip/irq-renesas-rzg2l.c b/drivers/irqchip/irq-renesas-rzg2l.c
index ac88d5aa46c2..72944e6a1658 100644
--- a/drivers/irqchip/irq-renesas-rzg2l.c
+++ b/drivers/irqchip/irq-renesas-rzg2l.c
@@ -399,7 +399,7 @@ static int rzg2l_irqc_set_type(struct irq_data *d, unsigned int type)
return irq_chip_set_type_parent(d, IRQ_TYPE_LEVEL_HIGH);
}
-static int rzg2l_irqc_irq_suspend(void)
+static int rzg2l_irqc_irq_suspend(void *data)
{
struct rzg2l_irqc_reg_cache *cache = &rzg2l_irqc_data->cache;
void __iomem *base = rzg2l_irqc_data->base;
@@ -411,7 +411,7 @@ static int rzg2l_irqc_irq_suspend(void)
return 0;
}
-static void rzg2l_irqc_irq_resume(void)
+static void rzg2l_irqc_irq_resume(void *data)
{
struct rzg2l_irqc_reg_cache *cache = &rzg2l_irqc_data->cache;
void __iomem *base = rzg2l_irqc_data->base;
@@ -426,11 +426,15 @@ static void rzg2l_irqc_irq_resume(void)
writel_relaxed(cache->iitsr, base + IITSR);
}
-static struct syscore_ops rzg2l_irqc_syscore_ops = {
+static const struct syscore_ops rzg2l_irqc_syscore_ops = {
.suspend = rzg2l_irqc_irq_suspend,
.resume = rzg2l_irqc_irq_resume,
};
+static struct syscore rzg2l_irqc_syscore = {
+ .ops = &rzg2l_irqc_syscore_ops,
+};
+
static const struct irq_chip rzg2l_irqc_chip = {
.name = "rzg2l-irqc",
.irq_eoi = rzg2l_irqc_eoi,
@@ -590,7 +594,7 @@ static int rzg2l_irqc_common_init(struct device_node *node, struct device_node *
goto pm_put;
}
- register_syscore_ops(&rzg2l_irqc_syscore_ops);
+ register_syscore(&rzg2l_irqc_syscore);
/*
* Prevent the cleanup function from invoking put_device by assigning
diff --git a/drivers/irqchip/irq-sa11x0.c b/drivers/irqchip/irq-sa11x0.c
index 9d0b80271949..141c3601d8a5 100644
--- a/drivers/irqchip/irq-sa11x0.c
+++ b/drivers/irqchip/irq-sa11x0.c
@@ -85,7 +85,7 @@ static struct sa1100irq_state {
unsigned int iccr;
} sa1100irq_state;
-static int sa1100irq_suspend(void)
+static int sa1100irq_suspend(void *data)
{
struct sa1100irq_state *st = &sa1100irq_state;
@@ -102,7 +102,7 @@ static int sa1100irq_suspend(void)
return 0;
}
-static void sa1100irq_resume(void)
+static void sa1100irq_resume(void *data)
{
struct sa1100irq_state *st = &sa1100irq_state;
@@ -114,14 +114,18 @@ static void sa1100irq_resume(void)
}
}
-static struct syscore_ops sa1100irq_syscore_ops = {
+static const struct syscore_ops sa1100irq_syscore_ops = {
.suspend = sa1100irq_suspend,
.resume = sa1100irq_resume,
};
+static struct syscore sa1100irq_syscore = {
+ .ops = &sa1100irq_syscore_ops,
+};
+
static int __init sa1100irq_init_devicefs(void)
{
- register_syscore_ops(&sa1100irq_syscore_ops);
+ register_syscore(&sa1100irq_syscore);
return 0;
}
diff --git a/drivers/irqchip/irq-sifive-plic.c b/drivers/irqchip/irq-sifive-plic.c
index b58b3cd807d4..d57e9b25d0b4 100644
--- a/drivers/irqchip/irq-sifive-plic.c
+++ b/drivers/irqchip/irq-sifive-plic.c
@@ -250,7 +250,7 @@ static int plic_irq_set_type(struct irq_data *d, unsigned int type)
return IRQ_SET_MASK_OK;
}
-static int plic_irq_suspend(void)
+static int plic_irq_suspend(void *data)
{
unsigned int i, cpu;
unsigned long flags;
@@ -282,7 +282,7 @@ static int plic_irq_suspend(void)
return 0;
}
-static void plic_irq_resume(void)
+static void plic_irq_resume(void *data)
{
unsigned int i, index, cpu;
unsigned long flags;
@@ -313,11 +313,15 @@ static void plic_irq_resume(void)
}
}
-static struct syscore_ops plic_irq_syscore_ops = {
+static const struct syscore_ops plic_irq_syscore_ops = {
.suspend = plic_irq_suspend,
.resume = plic_irq_resume,
};
+static struct syscore plic_irq_syscore = {
+ .ops = &plic_irq_syscore_ops,
+};
+
static int plic_irqdomain_map(struct irq_domain *d, unsigned int irq,
irq_hw_number_t hwirq)
{
@@ -683,7 +687,7 @@ static int plic_probe(struct fwnode_handle *fwnode)
cpuhp_setup_state(CPUHP_AP_IRQ_SIFIVE_PLIC_STARTING,
"irqchip/sifive/plic:starting",
plic_starting_cpu, plic_dying_cpu);
- register_syscore_ops(&plic_irq_syscore_ops);
+ register_syscore(&plic_irq_syscore);
plic_global_setup_done = true;
}
}
diff --git a/drivers/irqchip/irq-sun6i-r.c b/drivers/irqchip/irq-sun6i-r.c
index 99958d470d62..81e8342d1c9f 100644
--- a/drivers/irqchip/irq-sun6i-r.c
+++ b/drivers/irqchip/irq-sun6i-r.c
@@ -268,7 +268,7 @@ static const struct irq_domain_ops sun6i_r_intc_domain_ops = {
.free = irq_domain_free_irqs_common,
};
-static int sun6i_r_intc_suspend(void)
+static int sun6i_r_intc_suspend(void *data)
{
u32 buf[BITS_TO_U32(MAX(SUN6I_NR_TOP_LEVEL_IRQS, SUN6I_NR_MUX_BITS))];
int i;
@@ -284,7 +284,7 @@ static int sun6i_r_intc_suspend(void)
return 0;
}
-static void sun6i_r_intc_resume(void)
+static void sun6i_r_intc_resume(void *data)
{
int i;
@@ -294,17 +294,21 @@ static void sun6i_r_intc_resume(void)
writel_relaxed(0, base + SUN6I_IRQ_ENABLE(i));
}
-static void sun6i_r_intc_shutdown(void)
+static void sun6i_r_intc_shutdown(void *data)
{
- sun6i_r_intc_suspend();
+ sun6i_r_intc_suspend(data);
}
-static struct syscore_ops sun6i_r_intc_syscore_ops = {
+static const struct syscore_ops sun6i_r_intc_syscore_ops = {
.suspend = sun6i_r_intc_suspend,
.resume = sun6i_r_intc_resume,
.shutdown = sun6i_r_intc_shutdown,
};
+static struct syscore sun6i_r_intc_syscore = {
+ .ops = &sun6i_r_intc_syscore_ops,
+};
+
static int __init sun6i_r_intc_init(struct device_node *node,
struct device_node *parent,
const struct sun6i_r_intc_variant *v)
@@ -346,10 +350,10 @@ static int __init sun6i_r_intc_init(struct device_node *node,
return -ENOMEM;
}
- register_syscore_ops(&sun6i_r_intc_syscore_ops);
+ register_syscore(&sun6i_r_intc_syscore);
sun6i_r_intc_ack_nmi();
- sun6i_r_intc_resume();
+ sun6i_r_intc_resume(NULL);
return 0;
}
diff --git a/drivers/irqchip/irq-tegra.c b/drivers/irqchip/irq-tegra.c
index ad3e2c1b3c87..70ab4dc8f476 100644
--- a/drivers/irqchip/irq-tegra.c
+++ b/drivers/irqchip/irq-tegra.c
@@ -132,7 +132,7 @@ static int tegra_set_wake(struct irq_data *d, unsigned int enable)
return 0;
}
-static int tegra_ictlr_suspend(void)
+static int tegra_ictlr_suspend(void *data)
{
unsigned long flags;
unsigned int i;
@@ -161,7 +161,7 @@ static int tegra_ictlr_suspend(void)
return 0;
}
-static void tegra_ictlr_resume(void)
+static void tegra_ictlr_resume(void *data)
{
unsigned long flags;
unsigned int i;
@@ -184,14 +184,18 @@ static void tegra_ictlr_resume(void)
local_irq_restore(flags);
}
-static struct syscore_ops tegra_ictlr_syscore_ops = {
+static const struct syscore_ops tegra_ictlr_syscore_ops = {
.suspend = tegra_ictlr_suspend,
.resume = tegra_ictlr_resume,
};
+static struct syscore tegra_ictlr_syscore = {
+ .ops = &tegra_ictlr_syscore_ops,
+};
+
static void tegra_ictlr_syscore_init(void)
{
- register_syscore_ops(&tegra_ictlr_syscore_ops);
+ register_syscore(&tegra_ictlr_syscore);
}
#else
#define tegra_set_wake NULL
diff --git a/drivers/irqchip/irq-vic.c b/drivers/irqchip/irq-vic.c
index ea93e7236c4a..e2076ea885cf 100644
--- a/drivers/irqchip/irq-vic.c
+++ b/drivers/irqchip/irq-vic.c
@@ -120,7 +120,7 @@ static void resume_one_vic(struct vic_device *vic)
writel(~vic->soft_int, base + VIC_INT_SOFT_CLEAR);
}
-static void vic_resume(void)
+static void vic_resume(void *data)
{
int id;
@@ -146,7 +146,7 @@ static void suspend_one_vic(struct vic_device *vic)
writel(~vic->resume_irqs, base + VIC_INT_ENABLE_CLEAR);
}
-static int vic_suspend(void)
+static int vic_suspend(void *data)
{
int id;
@@ -156,11 +156,15 @@ static int vic_suspend(void)
return 0;
}
-static struct syscore_ops vic_syscore_ops = {
+static const struct syscore_ops vic_syscore_ops = {
.suspend = vic_suspend,
.resume = vic_resume,
};
+static struct syscore vic_syscore = {
+ .ops = &vic_syscore_ops,
+};
+
/**
* vic_pm_init - initcall to register VIC pm
*
@@ -171,7 +175,7 @@ static struct syscore_ops vic_syscore_ops = {
static int __init vic_pm_init(void)
{
if (vic_id > 0)
- register_syscore_ops(&vic_syscore_ops);
+ register_syscore(&vic_syscore);
return 0;
}
diff --git a/drivers/leds/leds-aw200xx.c b/drivers/leds/leds-aw200xx.c
index f9d9844e0273..7adf77c0a48a 100644
--- a/drivers/leds/leds-aw200xx.c
+++ b/drivers/leds/leds-aw200xx.c
@@ -379,7 +379,7 @@ static void aw200xx_enable(const struct aw200xx *const chip)
static void aw200xx_disable(const struct aw200xx *const chip)
{
- return gpiod_set_value_cansleep(chip->hwen, 0);
+ gpiod_set_value_cansleep(chip->hwen, 0);
}
static int aw200xx_probe_get_display_rows(struct device *dev,
diff --git a/drivers/leds/trigger/ledtrig-cpu.c b/drivers/leds/trigger/ledtrig-cpu.c
index 05848a2fecff..679323c2ccda 100644
--- a/drivers/leds/trigger/ledtrig-cpu.c
+++ b/drivers/leds/trigger/ledtrig-cpu.c
@@ -94,28 +94,32 @@ void ledtrig_cpu(enum cpu_led_event ledevt)
}
EXPORT_SYMBOL(ledtrig_cpu);
-static int ledtrig_cpu_syscore_suspend(void)
+static int ledtrig_cpu_syscore_suspend(void *data)
{
ledtrig_cpu(CPU_LED_STOP);
return 0;
}
-static void ledtrig_cpu_syscore_resume(void)
+static void ledtrig_cpu_syscore_resume(void *data)
{
ledtrig_cpu(CPU_LED_START);
}
-static void ledtrig_cpu_syscore_shutdown(void)
+static void ledtrig_cpu_syscore_shutdown(void *data)
{
ledtrig_cpu(CPU_LED_HALTED);
}
-static struct syscore_ops ledtrig_cpu_syscore_ops = {
+static const struct syscore_ops ledtrig_cpu_syscore_ops = {
.shutdown = ledtrig_cpu_syscore_shutdown,
.suspend = ledtrig_cpu_syscore_suspend,
.resume = ledtrig_cpu_syscore_resume,
};
+static struct syscore ledtrig_cpu_syscore = {
+ .ops = &ledtrig_cpu_syscore_ops,
+};
+
static int ledtrig_online_cpu(unsigned int cpu)
{
ledtrig_cpu(CPU_LED_START);
@@ -157,7 +161,7 @@ static int __init ledtrig_cpu_init(void)
led_trigger_register_simple(trig->name, &trig->_trig);
}
- register_syscore_ops(&ledtrig_cpu_syscore_ops);
+ register_syscore(&ledtrig_cpu_syscore);
ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "leds/trigger:starting",
ledtrig_online_cpu, ledtrig_prepare_down_cpu);
diff --git a/drivers/macintosh/via-pmu.c b/drivers/macintosh/via-pmu.c
index b0f09c70f1ff..5fe47e784d43 100644
--- a/drivers/macintosh/via-pmu.c
+++ b/drivers/macintosh/via-pmu.c
@@ -2600,7 +2600,7 @@ void pmu_blink(int n)
#if defined(CONFIG_SUSPEND) && defined(CONFIG_PPC32)
int pmu_sys_suspended;
-static int pmu_syscore_suspend(void)
+static int pmu_syscore_suspend(void *data)
{
/* Suspend PMU event interrupts */
pmu_suspend();
@@ -2614,7 +2614,7 @@ static int pmu_syscore_suspend(void)
return 0;
}
-static void pmu_syscore_resume(void)
+static void pmu_syscore_resume(void *data)
{
struct adb_request req;
@@ -2634,14 +2634,18 @@ static void pmu_syscore_resume(void)
pmu_sys_suspended = 0;
}
-static struct syscore_ops pmu_syscore_ops = {
+static const struct syscore_ops pmu_syscore_ops = {
.suspend = pmu_syscore_suspend,
.resume = pmu_syscore_resume,
};
+static struct syscore pmu_syscore = {
+ .ops = &pmu_syscore_ops,
+};
+
static int pmu_syscore_register(void)
{
- register_syscore_ops(&pmu_syscore_ops);
+ register_syscore(&pmu_syscore);
return 0;
}
diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c
index 0a7be46701a9..b1313077de7a 100644
--- a/drivers/md/dm-integrity.c
+++ b/drivers/md/dm-integrity.c
@@ -65,6 +65,7 @@
#define SB_VERSION_4 4
#define SB_VERSION_5 5
#define SB_VERSION_6 6
+#define SB_VERSION_7 7
#define SB_SECTORS 8
#define MAX_SECTORS_PER_BLOCK 8
@@ -90,6 +91,7 @@ struct superblock {
#define SB_FLAG_FIXED_PADDING 0x8
#define SB_FLAG_FIXED_HMAC 0x10
#define SB_FLAG_INLINE 0x20
+#define SB_FLAG_DISCARD_KEYED 0x40
#define JOURNAL_ENTRY_ROUNDUP 8
@@ -219,10 +221,12 @@ struct dm_integrity_c {
__u8 log2_blocks_per_bitmap_bit;
unsigned char mode;
+ bool internal_hash;
int failed;
- struct crypto_shash *internal_hash;
+ struct crypto_shash *internal_shash;
+ unsigned int internal_hash_digestsize;
struct dm_target *ti;
@@ -273,6 +277,7 @@ struct dm_integrity_c {
bool recalculate_flag;
bool reset_recalculate_flag;
bool discard;
+ bool discard_keyed;
bool fix_padding;
bool fix_hmac;
bool legacy_recalculate;
@@ -473,7 +478,9 @@ static void wraparound_section(struct dm_integrity_c *ic, unsigned int *sec_ptr)
static void sb_set_version(struct dm_integrity_c *ic)
{
- if (ic->sb->flags & cpu_to_le32(SB_FLAG_INLINE))
+ if (ic->sb->flags & cpu_to_le32(SB_FLAG_DISCARD_KEYED))
+ ic->sb->version = SB_VERSION_7;
+ else if (ic->sb->flags & cpu_to_le32(SB_FLAG_INLINE))
ic->sb->version = SB_VERSION_6;
else if (ic->sb->flags & cpu_to_le32(SB_FLAG_FIXED_HMAC))
ic->sb->version = SB_VERSION_5;
@@ -1404,7 +1411,7 @@ static int dm_integrity_rw_tag(struct dm_integrity_c *ic, unsigned char *tag, se
{
unsigned int hash_offset = 0;
unsigned char mismatch_hash = 0;
- unsigned char mismatch_filler = !ic->discard;
+ unsigned char mismatch_filler = !ic->discard || ic->discard_keyed;
do {
unsigned char *data, *dp;
@@ -1456,7 +1463,7 @@ static int dm_integrity_rw_tag(struct dm_integrity_c *ic, unsigned char *tag, se
}
hash_offset = 0;
mismatch_hash = 0;
- mismatch_filler = !ic->discard;
+ mismatch_filler = !ic->discard || ic->discard_keyed;
}
}
}
@@ -1634,11 +1641,11 @@ static void integrity_sector_checksum(struct dm_integrity_c *ic, sector_t sector
const char *data, char *result)
{
__le64 sector_le = cpu_to_le64(sector);
- SHASH_DESC_ON_STACK(req, ic->internal_hash);
+ SHASH_DESC_ON_STACK(req, ic->internal_shash);
int r;
unsigned int digest_size;
- req->tfm = ic->internal_hash;
+ req->tfm = ic->internal_shash;
r = crypto_shash_init(req);
if (unlikely(r < 0)) {
@@ -1660,10 +1667,12 @@ static void integrity_sector_checksum(struct dm_integrity_c *ic, sector_t sector
goto failed;
}
- r = crypto_shash_update(req, data, ic->sectors_per_block << SECTOR_SHIFT);
- if (unlikely(r < 0)) {
- dm_integrity_io_error(ic, "crypto_shash_update", r);
- goto failed;
+ if (likely(data)) {
+ r = crypto_shash_update(req, data, ic->sectors_per_block << SECTOR_SHIFT);
+ if (unlikely(r < 0)) {
+ dm_integrity_io_error(ic, "crypto_shash_update", r);
+ goto failed;
+ }
}
r = crypto_shash_final(req, result);
@@ -1672,7 +1681,7 @@ static void integrity_sector_checksum(struct dm_integrity_c *ic, sector_t sector
goto failed;
}
- digest_size = crypto_shash_digestsize(ic->internal_hash);
+ digest_size = ic->internal_hash_digestsize;
if (unlikely(digest_size < ic->tag_size))
memset(result + digest_size, 0, ic->tag_size - digest_size);
@@ -1683,6 +1692,53 @@ static void integrity_sector_checksum(struct dm_integrity_c *ic, sector_t sector
get_random_bytes(result, ic->tag_size);
}
+/*
+ * Authenticated marker for a discarded block: HMAC_key(salt || sector), with
+ * no data payload. Because a real data tag's input always covers a full
+ * block, its length differs from this marker's, so the two can never
+ * collide structurally, regardless of block content.
+ */
+static void integrity_discard_checksum(struct dm_integrity_c *ic, sector_t sector, char *result)
+{
+ integrity_sector_checksum(ic, sector, NULL, result);
+}
+
+static void integrity_discard_fill_tags(struct dm_integrity_c *ic, unsigned char *checksums,
+ sector_t *sector, unsigned int blocks)
+{
+ char digest[MAX_T(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)];
+ unsigned int i;
+
+ for (i = 0; i < blocks; i++) {
+ integrity_discard_checksum(ic, *sector, digest);
+ memcpy(checksums + i * ic->tag_size, digest, ic->tag_size);
+ *sector += ic->sectors_per_block;
+ }
+}
+
+static int integrity_recheck_verify_tag(struct dm_integrity_io *dio, char *checksum,
+ char *on_disk_tag, sector_t logical_sector)
+{
+ struct dm_integrity_c *ic = dio->ic;
+ int r;
+
+ if (!ic->discard_keyed)
+ return dm_integrity_rw_tag(ic, checksum, &dio->metadata_block,
+ &dio->metadata_offset, ic->tag_size, TAG_CMP);
+
+ r = dm_integrity_rw_tag(ic, on_disk_tag, &dio->metadata_block,
+ &dio->metadata_offset, ic->tag_size, TAG_READ);
+ if (unlikely(r))
+ return r;
+
+ r = crypto_memneq(on_disk_tag, checksum, ic->tag_size);
+ if (unlikely(r)) {
+ integrity_discard_checksum(ic, logical_sector, checksum);
+ r = crypto_memneq(on_disk_tag, checksum, ic->tag_size);
+ }
+ return r;
+}
+
static noinline void integrity_recheck(struct dm_integrity_io *dio, char *checksum)
{
struct bio *bio = dm_bio_from_per_bio_data(dio, sizeof(struct dm_integrity_io));
@@ -1707,6 +1763,7 @@ static noinline void integrity_recheck(struct dm_integrity_io *dio, char *checks
sector_t alignment;
char *mem;
char *buffer = page_to_virt(page);
+ char on_disk_tag[MAX_T(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)];
int r;
struct dm_io_request io_req;
struct dm_io_region io_loc;
@@ -1734,8 +1791,8 @@ static noinline void integrity_recheck(struct dm_integrity_io *dio, char *checks
}
integrity_sector_checksum(ic, logical_sector, buffer, checksum);
- r = dm_integrity_rw_tag(ic, checksum, &dio->metadata_block,
- &dio->metadata_offset, ic->tag_size, TAG_CMP);
+ r = integrity_recheck_verify_tag(dio, checksum, on_disk_tag,
+ logical_sector);
if (r) {
if (r > 0) {
DMERR_LIMIT("%pg: Checksum failed at sector 0x%llx",
@@ -1772,7 +1829,7 @@ static void integrity_metadata(struct work_struct *w)
if (ic->internal_hash) {
struct bvec_iter iter;
struct bio_vec bv;
- unsigned int digest_size = crypto_shash_digestsize(ic->internal_hash);
+ unsigned int digest_size = ic->internal_hash_digestsize;
struct bio *bio = dm_bio_from_per_bio_data(dio, sizeof(struct dm_integrity_io));
char *checksums;
unsigned int extra_space = unlikely(digest_size > ic->tag_size) ? digest_size - ic->tag_size : 0;
@@ -1800,14 +1857,19 @@ static void integrity_metadata(struct work_struct *w)
if (unlikely(dio->op == REQ_OP_DISCARD)) {
unsigned int bi_size = dio->bio_details.bi_iter.bi_size;
unsigned int max_size = likely(checksums != checksums_onstack) ? PAGE_SIZE : HASH_MAX_DIGESTSIZE;
- unsigned int max_blocks = max_size / ic->tag_size;
+ unsigned int max_blocks = (max_size - extra_space) / ic->tag_size;
+ sector_t sector = dio->range.logical_sector;
- memset(checksums, DISCARD_FILLER, max_size);
+ if (!ic->discard_keyed)
+ memset(checksums, DISCARD_FILLER, max_size);
while (bi_size) {
unsigned int this_step_blocks = bi_size >> (SECTOR_SHIFT + ic->sb->log2_sectors_per_block);
this_step_blocks = min(this_step_blocks, max_blocks);
+ if (ic->discard_keyed)
+ integrity_discard_fill_tags(ic, checksums, §or,
+ this_step_blocks);
r = dm_integrity_rw_tag(ic, checksums, &dio->metadata_block, &dio->metadata_offset,
this_step_blocks * ic->tag_size, TAG_WRITE);
if (unlikely(r)) {
@@ -2120,7 +2182,7 @@ static bool __journal_read_write(struct dm_integrity_io *dio, struct bio *bio,
} while (++s < ic->sectors_per_block);
if (ic->internal_hash) {
- unsigned int digest_size = crypto_shash_digestsize(ic->internal_hash);
+ unsigned int digest_size = ic->internal_hash_digestsize;
if (unlikely(digest_size > ic->tag_size)) {
char checksums_onstack[HASH_MAX_DIGESTSIZE];
@@ -2427,7 +2489,7 @@ static int dm_integrity_map_inline(struct dm_integrity_io *dio, bool from_map)
if (!dio->integrity_payload) {
unsigned digest_size, extra_size;
dio->payload_len = ic->tuple_size * (bio_sectors(bio) >> ic->sb->log2_sectors_per_block);
- digest_size = crypto_shash_digestsize(ic->internal_hash);
+ digest_size = ic->internal_hash_digestsize;
extra_size = unlikely(digest_size > ic->tag_size) ? digest_size - ic->tag_size : 0;
dio->payload_len += extra_size;
dio->integrity_payload = kmalloc(dio->payload_len, GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN);
@@ -2546,7 +2608,7 @@ static void dm_integrity_inline_recheck(struct work_struct *w)
outgoing_data = dio->integrity_payload + PAGE_SIZE;
while (dio->bio_details.bi_iter.bi_size) {
- char digest[HASH_MAX_DIGESTSIZE];
+ char digest[MAX_T(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)];
int r;
struct bio_integrity_payload *bip;
struct bio_vec bv;
@@ -2590,7 +2652,7 @@ static void dm_integrity_inline_recheck(struct work_struct *w)
bio_put(outgoing_bio);
integrity_sector_checksum(ic, dio->bio_details.bi_iter.bi_sector, outgoing_data, digest);
- if (unlikely(crypto_memneq(digest, dio->integrity_payload, min(crypto_shash_digestsize(ic->internal_hash), ic->tag_size)))) {
+ if (unlikely(crypto_memneq(digest, dio->integrity_payload, min(ic->internal_hash_digestsize, ic->tag_size)))) {
DMERR_LIMIT("%pg: Checksum failed at sector 0x%llx",
ic->dev->bdev, dio->bio_details.bi_iter.bi_sector);
atomic64_inc(&ic->number_of_mismatches);
@@ -2624,13 +2686,13 @@ static int dm_integrity_end_io(struct dm_target *ti, struct bio *bio, blk_status
unlikely(dio->integrity_range_locked))
goto skip_check;
while (dio->bio_details.bi_iter.bi_size) {
- char digest[HASH_MAX_DIGESTSIZE];
+ char digest[MAX_T(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)];
struct bio_vec bv = bio_iter_iovec(bio, dio->bio_details.bi_iter);
char *mem = bvec_kmap_local(&bv);
//memset(mem, 0xff, ic->sectors_per_block << SECTOR_SHIFT);
integrity_sector_checksum(ic, dio->bio_details.bi_iter.bi_sector, mem, digest);
if (unlikely(crypto_memneq(digest, dio->integrity_payload + pos,
- min(crypto_shash_digestsize(ic->internal_hash), ic->tag_size)))) {
+ min(ic->internal_hash_digestsize, ic->tag_size)))) {
kunmap_local(mem);
dm_integrity_free_payload(dio);
INIT_WORK(&dio->work, dm_integrity_inline_recheck);
@@ -3012,8 +3074,8 @@ static void integrity_recalc(struct work_struct *w)
goto free_ret;
}
recalc_tags_size = (recalc_sectors >> ic->sb->log2_sectors_per_block) * ic->tag_size;
- if (crypto_shash_digestsize(ic->internal_hash) > ic->tag_size)
- recalc_tags_size += crypto_shash_digestsize(ic->internal_hash) - ic->tag_size;
+ if (ic->internal_hash_digestsize > ic->tag_size)
+ recalc_tags_size += ic->internal_hash_digestsize - ic->tag_size;
recalc_tags = kvmalloc(recalc_tags_size, GFP_NOIO);
if (!recalc_tags) {
vfree(recalc_buffer);
@@ -3172,8 +3234,8 @@ static void integrity_recalc_inline(struct work_struct *w)
}
recalc_tags_size = (recalc_sectors >> ic->sb->log2_sectors_per_block) * ic->tuple_size;
- if (crypto_shash_digestsize(ic->internal_hash) > ic->tuple_size)
- recalc_tags_size += crypto_shash_digestsize(ic->internal_hash) - ic->tuple_size;
+ if (ic->internal_hash_digestsize > ic->tuple_size)
+ recalc_tags_size += ic->internal_hash_digestsize - ic->tuple_size;
recalc_tags = kmalloc(recalc_tags_size, GFP_NOIO | __GFP_NOWARN);
if (!recalc_tags) {
kfree(recalc_buffer);
@@ -3656,6 +3718,8 @@ static void dm_integrity_resume(struct dm_target *ti)
ic->wrote_to_journal = false;
flags = ic->sb->flags & cpu_to_le32(SB_FLAG_RECALCULATING);
+ if (ic->discard_keyed)
+ flags |= cpu_to_le32(SB_FLAG_DISCARD_KEYED);
r = sync_rw_sb(ic, REQ_OP_READ);
if (r)
dm_integrity_io_error(ic, "reading superblock", r);
@@ -3803,7 +3867,8 @@ static void dm_integrity_status(struct dm_target *ti, status_type_t type,
arg_count += ic->sectors_per_block != 1;
arg_count += !!(ic->sb->flags & cpu_to_le32(SB_FLAG_RECALCULATING));
arg_count += ic->reset_recalculate_flag;
- arg_count += ic->discard;
+ arg_count += ic->discard && !ic->discard_keyed;
+ arg_count += ic->discard_keyed;
arg_count += ic->mode == 'J';
arg_count += ic->mode == 'J';
arg_count += ic->mode == 'B';
@@ -3824,8 +3889,10 @@ static void dm_integrity_status(struct dm_target *ti, status_type_t type,
DMEMIT(" recalculate");
if (ic->reset_recalculate_flag)
DMEMIT(" reset_recalculate");
- if (ic->discard)
+ if (ic->discard && !ic->discard_keyed)
DMEMIT(" allow_discards");
+ if (ic->discard_keyed)
+ DMEMIT(" allow_discards_keyed");
DMEMIT(" journal_sectors:%u", ic->initial_sectors - SB_SECTORS);
DMEMIT(" interleave_sectors:%u", 1U << ic->sb->log2_interleave_sectors);
DMEMIT(" buffer_sectors:%u", 1U << ic->log2_buffer_sectors);
@@ -3874,6 +3941,7 @@ static void dm_integrity_status(struct dm_target *ti, status_type_t type,
DMEMIT(",recalculate=%c", (ic->sb->flags & cpu_to_le32(SB_FLAG_RECALCULATING)) ?
'y' : 'n');
DMEMIT(",allow_discards=%c", ic->discard ? 'y' : 'n');
+ DMEMIT(",allow_discards_keyed=%c", ic->discard_keyed ? 'y' : 'n');
DMEMIT(",fix_padding=%c",
((ic->sb->flags & cpu_to_le32(SB_FLAG_FIXED_PADDING)) != 0) ? 'y' : 'n');
DMEMIT(",fix_hmac=%c",
@@ -4035,6 +4103,9 @@ static int initialize_superblock(struct dm_integrity_c *ic,
get_random_bytes(ic->sb->salt, SALT_SIZE);
}
+ if (ic->discard_keyed)
+ ic->sb->flags |= cpu_to_le32(SB_FLAG_DISCARD_KEYED);
+
if (!ic->meta_dev) {
if (ic->fix_padding)
ic->sb->flags |= cpu_to_le32(SB_FLAG_FIXED_PADDING);
@@ -4671,6 +4742,9 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv
ic->reset_recalculate_flag = true;
} else if (!strcmp(opt_string, "allow_discards")) {
ic->discard = true;
+ } else if (!strcmp(opt_string, "allow_discards_keyed")) {
+ ic->discard = true;
+ ic->discard_keyed = true;
} else if (!strcmp(opt_string, "fix_padding")) {
ic->fix_padding = true;
} else if (!strcmp(opt_string, "fix_hmac")) {
@@ -4699,10 +4773,14 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv
buffer_sectors = 1;
ic->log2_buffer_sectors = min((int)__fls(buffer_sectors), 31 - SECTOR_SHIFT);
- r = get_mac(&ic->internal_hash, &ic->internal_hash_alg, &ti->error,
+ r = get_mac(&ic->internal_shash, &ic->internal_hash_alg, &ti->error,
"Invalid internal hash", "Error setting internal hash key");
if (r)
goto bad;
+ if (ic->internal_shash) {
+ ic->internal_hash = true;
+ ic->internal_hash_digestsize = crypto_shash_digestsize(ic->internal_shash);
+ }
r = get_mac(&ic->journal_mac, &ic->journal_mac_alg, &ti->error,
"Invalid journal mac", "Error setting journal mac key");
@@ -4715,7 +4793,7 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv
r = -EINVAL;
goto bad;
}
- ic->tag_size = crypto_shash_digestsize(ic->internal_hash);
+ ic->tag_size = ic->internal_hash_digestsize;
}
if (ic->tag_size > MAX_TAG_SIZE) {
ti->error = "Too big tag size";
@@ -4785,6 +4863,11 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv
ti->error = "Discard can be only used with internal hash";
goto bad;
}
+ if (ic->discard_keyed && !ic->internal_hash_alg.key) {
+ r = -EINVAL;
+ ti->error = "Keyed discard can only be used with keyed internal hash";
+ goto bad;
+ }
ic->autocommit_jiffies = msecs_to_jiffies(sync_msec);
ic->autocommit_msec = sync_msec;
@@ -4911,7 +4994,7 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv
should_write_sb = true;
}
- if (!ic->sb->version || ic->sb->version > SB_VERSION_6) {
+ if (!ic->sb->version || ic->sb->version > SB_VERSION_7) {
r = -EINVAL;
ti->error = "Unknown version";
goto bad;
@@ -4959,6 +5042,11 @@ static int dm_integrity_ctr(struct dm_target *ti, unsigned int argc, char **argv
goto bad;
}
}
+ if (!ic->discard_keyed && (ic->sb->flags & cpu_to_le32(SB_FLAG_DISCARD_KEYED))) {
+ r = -EINVAL;
+ ti->error = "Keyed discard cannot be disabled once enabled";
+ goto bad;
+ }
if (!!(ic->sb->flags & cpu_to_le32(SB_FLAG_HAVE_JOURNAL_MAC)) != !!ic->journal_mac_alg.alg_string) {
r = -EINVAL;
ti->error = "Journal mac mismatch";
@@ -5236,8 +5324,8 @@ static void dm_integrity_dtr(struct dm_target *ti)
if (ic->sb)
free_pages_exact(ic->sb, SB_SECTORS << SECTOR_SHIFT);
- if (ic->internal_hash)
- crypto_free_shash(ic->internal_hash);
+ if (ic->internal_shash)
+ crypto_free_shash(ic->internal_shash);
free_alg(&ic->internal_hash_alg);
if (ic->journal_crypt)
@@ -5254,7 +5342,7 @@ static void dm_integrity_dtr(struct dm_target *ti)
static struct target_type integrity_target = {
.name = "integrity",
- .version = {1, 13, 0},
+ .version = {1, 14, 0},
.module = THIS_MODULE,
.features = DM_TARGET_SINGLETON | DM_TARGET_INTEGRITY,
.ctr = dm_integrity_ctr,
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index 20b8f560a2da..505f8a900ec9 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -1505,6 +1505,18 @@ bool dm_table_has_no_data_devices(struct dm_table *t)
return true;
}
+bool dm_table_is_wildcard(struct dm_table *t)
+{
+ for (unsigned int i = 0; i < t->num_targets; i++) {
+ struct dm_target *ti = dm_table_get_target(t, i);
+
+ if (!dm_target_is_wildcard(ti->type))
+ return false;
+ }
+
+ return true;
+}
+
static int device_not_zoned(struct dm_target *ti, struct dm_dev *dev,
sector_t start, sector_t len, void *data)
{
@@ -1823,6 +1835,19 @@ static bool dm_table_supports_secure_erase(struct dm_table *t)
return true;
}
+bool dm_table_supports_size_change(struct dm_table *t, sector_t old_size,
+ sector_t new_size)
+{
+ if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) && dm_has_zone_plugs(t->md) &&
+ old_size != new_size) {
+ DMWARN("%s: device has zone write plug resources. "
+ "Cannot change size",
+ dm_device_name(t->md));
+ return false;
+ }
+ return true;
+}
+
int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
struct queue_limits *limits)
{
@@ -1864,11 +1889,17 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
dax_write_cache(t->md->dax_dev, true);
/* For a zoned table, setup the zone related queue attributes. */
- if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) &&
- (limits->features & BLK_FEAT_ZONED)) {
- r = dm_set_zones_restrictions(t, q, limits);
- if (r)
- return r;
+ if (IS_ENABLED(CONFIG_BLK_DEV_ZONED)) {
+ if (limits->features & BLK_FEAT_ZONED) {
+ r = dm_set_zones_restrictions(t, q, limits);
+ if (r)
+ return r;
+ } else if (dm_has_zone_plugs(t->md)) {
+ DMWARN("%s: device has zone write plug resources. "
+ "Cannot switch to non-zoned table.",
+ dm_device_name(t->md));
+ return -EINVAL;
+ }
}
r = queue_limits_set(q, limits);
diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c
index 9e9ff481d5a0..b6115ac9c825 100644
--- a/drivers/md/dm-verity-fec.c
+++ b/drivers/md/dm-verity-fec.c
@@ -55,36 +55,6 @@ static int fec_decode_rs8(struct dm_verity *v, struct dm_verity_fec_io *fio,
fio->erasures, 0, NULL);
}
-/*
- * Read error-correcting codes for the requested RS block. Returns a pointer
- * to the data block. Caller is responsible for releasing buf.
- */
-static u8 *fec_read_parity(struct dm_verity *v, u64 rsb, int index,
- unsigned int *offset, unsigned int par_buf_offset,
- struct dm_buffer **buf, unsigned short ioprio)
-{
- u64 position, block, rem;
- u8 *res;
-
- /* We have already part of parity bytes read, skip to the next block */
- if (par_buf_offset)
- index++;
-
- position = (index + rsb) * v->fec->roots;
- block = div64_u64_rem(position, v->fec->io_size, &rem);
- *offset = par_buf_offset ? 0 : (unsigned int)rem;
-
- res = dm_bufio_read_with_ioprio(v->fec->bufio, block, buf, ioprio);
- if (IS_ERR(res)) {
- DMERR("%s: FEC %llu: parity read failed (block %llu): %ld",
- v->data_dev->name, (unsigned long long)rsb,
- (unsigned long long)block, PTR_ERR(res));
- *buf = NULL;
- }
-
- return res;
-}
-
/* Loop over each preallocated buffer slot. */
#define fec_for_each_prealloc_buffer(__i) \
for (__i = 0; __i < DM_VERITY_FEC_BUF_PREALLOC; __i++)
@@ -132,14 +102,28 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_io *io,
{
int r, corrected = 0, res;
struct dm_buffer *buf;
- unsigned int n, i, offset, par_buf_offset = 0;
+ unsigned int n, i, j, parity_pos, to_copy;
u8 *par, *block, par_buf[DM_VERITY_FEC_RSM - DM_VERITY_FEC_MIN_RSN];
+ u64 parity_block;
struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size);
- par = fec_read_parity(v, rsb, block_offset, &offset,
- par_buf_offset, &buf, bio_prio(bio));
- if (IS_ERR(par))
+ /*
+ * Compute the index of the first parity block that will be needed and
+ * the starting position in that block. Then read that block.
+ *
+ * io_size is always a power of 2, but roots might not be. Note that
+ * when it's not, a codeword's parity bytes can span a block boundary.
+ */
+ parity_block = (rsb + block_offset) * v->fec->roots;
+ parity_pos = parity_block & (v->fec->io_size - 1);
+ parity_block >>= v->data_dev_block_bits;
+ par = dm_bufio_read_with_ioprio(v->fec->bufio, parity_block, &buf,
+ bio_prio(bio));
+ if (IS_ERR(par)) {
+ DMERR("%s: FEC %llu: parity read failed (block %llu): %ld",
+ v->data_dev->name, rsb, parity_block, PTR_ERR(par));
return PTR_ERR(par);
+ }
/*
* Decode the RS blocks we have in bufs. Each RS block results in
@@ -147,7 +131,32 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_io *io,
*/
fec_for_each_buffer_rs_block(fio, n, i) {
block = fec_buffer_rs_block(v, fio, n, i);
- memcpy(&par_buf[par_buf_offset], &par[offset], v->fec->roots - par_buf_offset);
+
+ /*
+ * Copy the next 'roots' parity bytes to 'par_buf', reading
+ * another parity block if needed.
+ */
+ to_copy = min(v->fec->io_size - parity_pos, v->fec->roots);
+ for (j = 0; j < to_copy; j++)
+ par_buf[j] = par[parity_pos++];
+ if (to_copy < v->fec->roots) {
+ parity_block++;
+ parity_pos = 0;
+
+ dm_bufio_release(buf);
+ par = dm_bufio_read_with_ioprio(v->fec->bufio,
+ parity_block, &buf,
+ bio_prio(bio));
+ if (IS_ERR(par)) {
+ DMERR("%s: FEC %llu: parity read failed (block %llu): %ld",
+ v->data_dev->name, rsb, parity_block,
+ PTR_ERR(par));
+ return PTR_ERR(par);
+ }
+ for (; j < v->fec->roots; j++)
+ par_buf[j] = par[parity_pos++];
+ }
+
res = fec_decode_rs8(v, fio, block, par_buf, neras);
if (res < 0) {
r = res;
@@ -160,25 +169,6 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_io *io,
block_offset++;
if (block_offset >= 1 << v->data_dev_block_bits)
goto done;
-
- /* Read the next block when we run out of parity bytes */
- offset += (v->fec->roots - par_buf_offset);
- /* Check if parity bytes are split between blocks */
- if (offset < v->fec->io_size && (offset + v->fec->roots) > v->fec->io_size) {
- par_buf_offset = v->fec->io_size - offset;
- memcpy(par_buf, &par[offset], par_buf_offset);
- offset += par_buf_offset;
- } else
- par_buf_offset = 0;
-
- if (offset >= v->fec->io_size) {
- dm_bufio_release(buf);
-
- par = fec_read_parity(v, rsb, block_offset, &offset,
- par_buf_offset, &buf, bio_prio(bio));
- if (IS_ERR(par))
- return PTR_ERR(par);
- }
}
done:
r = corrected;
@@ -273,7 +263,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io,
(unsigned long long)block, PTR_ERR(bbuf));
/* assume the block is corrupted */
- if (neras && *neras <= v->fec->roots)
+ if (neras && *neras < v->fec->roots)
fio->erasures[(*neras)++] = i;
continue;
@@ -291,7 +281,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io,
* skip if we have already found the theoretical
* maximum number (i.e. fec->roots) of erasures
*/
- if (neras && *neras <= v->fec->roots &&
+ if (neras && *neras < v->fec->roots &&
fec_is_erasure(v, io, want_digest, bbuf))
fio->erasures[(*neras)++] = i;
}
diff --git a/drivers/md/dm-zone.c b/drivers/md/dm-zone.c
index 912b9fe1f564..85777d875cc0 100644
--- a/drivers/md/dm-zone.c
+++ b/drivers/md/dm-zone.c
@@ -163,22 +163,22 @@ int dm_revalidate_zones(struct dm_table *t, struct request_queue *q)
{
struct mapped_device *md = t->md;
struct gendisk *disk = md->disk;
+ unsigned int nr_zones = disk->nr_zones;
int ret;
if (!get_capacity(disk))
return 0;
- /* Revalidate only if something changed. */
- if (!disk->nr_zones || disk->nr_zones != md->nr_zones) {
- DMINFO("%s using %s zone append",
- disk->disk_name,
- queue_emulates_zone_append(q) ? "emulated" : "native");
- md->nr_zones = 0;
- }
-
- if (md->nr_zones)
+ /*
+ * Do not revalidate if zone write plug resources have already
+ * been allocated.
+ */
+ if (dm_has_zone_plugs(md))
return 0;
+ DMINFO("%s using %s zone append", disk->disk_name,
+ queue_emulates_zone_append(q) ? "emulated" : "native");
+
/*
* Our table is not live yet. So the call to dm_get_live_table()
* in dm_blk_report_zones() will fail. Set a temporary pointer to
@@ -192,6 +192,7 @@ int dm_revalidate_zones(struct dm_table *t, struct request_queue *q)
if (ret) {
DMERR("Revalidate zones failed %d", ret);
+ disk->nr_zones = nr_zones;
return ret;
}
@@ -401,6 +402,21 @@ int dm_set_zones_restrictions(struct dm_table *t, struct request_queue *q,
return 0;
}
+ if (get_capacity(disk) && dm_has_zone_plugs(t->md)) {
+ if (q->limits.chunk_sectors != lim->chunk_sectors) {
+ DMWARN("%s: device has zone write plug resources. "
+ "Cannot change zone size",
+ disk->disk_name);
+ return -EINVAL;
+ }
+ if (lim->max_zone_append_sectors != 0 &&
+ !dm_table_is_wildcard(t)) {
+ DMWARN("%s: device has zone write plug resources. "
+ "New table must emulate zone append",
+ disk->disk_name);
+ return -EINVAL;
+ }
+ }
/*
* Warn once (when the capacity is not yet set) if the mapped device is
* partially using zone resources of the target devices as that leads to
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 20f6c6ebace0..e23d4c63484b 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2451,6 +2451,12 @@ static struct dm_table *__bind(struct mapped_device *md, struct dm_table *t,
size = dm_table_get_size(t);
old_size = dm_get_size(md);
+
+ if (!dm_table_supports_size_change(t, old_size, size)) {
+ old_map = ERR_PTR(-EINVAL);
+ goto out;
+ }
+
set_capacity(md->disk, size);
ret = dm_table_set_restrictions(t, md->queue, limits);
diff --git a/drivers/md/dm.h b/drivers/md/dm.h
index 8ad782249af8..8887c1f65639 100644
--- a/drivers/md/dm.h
+++ b/drivers/md/dm.h
@@ -58,6 +58,7 @@ void dm_table_event_callback(struct dm_table *t,
void (*fn)(void *), void *context);
struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector);
bool dm_table_has_no_data_devices(struct dm_table *table);
+bool dm_table_is_wildcard(struct dm_table *t);
int dm_calculate_queue_limits(struct dm_table *table,
struct queue_limits *limits);
int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
@@ -73,6 +74,8 @@ struct dm_target *dm_table_get_immutable_target(struct dm_table *t);
struct dm_target *dm_table_get_wildcard_target(struct dm_table *t);
bool dm_table_bio_based(struct dm_table *t);
bool dm_table_request_based(struct dm_table *t);
+bool dm_table_supports_size_change(struct dm_table *t, sector_t old_size,
+ sector_t new_size);
void dm_lock_md_type(struct mapped_device *md);
void dm_unlock_md_type(struct mapped_device *md);
@@ -112,12 +115,14 @@ bool dm_is_zone_write(struct mapped_device *md, struct bio *bio);
int dm_zone_get_reset_bitmap(struct mapped_device *md, struct dm_table *t,
sector_t sector, unsigned int nr_zones,
unsigned long *need_reset);
+#define dm_has_zone_plugs(md) ((md)->disk->zone_wplugs_hash != NULL)
#else
#define dm_blk_report_zones NULL
static inline bool dm_is_zone_write(struct mapped_device *md, struct bio *bio)
{
return false;
}
+#define dm_has_zone_plugs(md) false
#endif
/*
diff --git a/drivers/md/md.c b/drivers/md/md.c
index b3ba51d7da60..db8df74fcf67 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -380,25 +380,24 @@ static bool is_suspended(struct mddev *mddev, struct bio *bio)
bool md_handle_request(struct mddev *mddev, struct bio *bio)
{
check_suspended:
- if (is_suspended(mddev, bio)) {
- DEFINE_WAIT(__wait);
- /* Bail out if REQ_NOWAIT is set for the bio */
- if (bio->bi_opf & REQ_NOWAIT) {
- bio_wouldblock_error(bio);
- return true;
- }
- for (;;) {
- prepare_to_wait(&mddev->sb_wait, &__wait,
- TASK_UNINTERRUPTIBLE);
- if (!is_suspended(mddev, bio))
- break;
- schedule();
+ if (unlikely(md_cloned_bio(mddev, bio))) {
+ /*
+ * This bio is an MD cloned bio and already holds an
+ * active_io reference, so percpu_ref_get() is safe here.
+ */
+ percpu_ref_get(&mddev->active_io);
+ } else {
+ if (is_suspended(mddev, bio)) {
+ /* Bail out if REQ_NOWAIT is set for the bio */
+ if (bio->bi_opf & REQ_NOWAIT) {
+ bio_wouldblock_error(bio);
+ return true;
+ }
+ wait_event(mddev->sb_wait, !is_suspended(mddev, bio));
}
- finish_wait(&mddev->sb_wait, &__wait);
+ if (!percpu_ref_tryget_live(&mddev->active_io))
+ goto check_suspended;
}
- if (!percpu_ref_tryget_live(&mddev->active_io))
- goto check_suspended;
-
if (!mddev->pers->make_request(mddev, bio)) {
percpu_ref_put(&mddev->active_io);
if (!mddev->gendisk && mddev->pers->prepare_suspend)
@@ -665,6 +664,7 @@ static void no_op(struct percpu_ref *r) {}
int mddev_init(struct mddev *mddev)
{
+ int err = 0;
if (percpu_ref_init(&mddev->active_io, active_io_release,
PERCPU_REF_ALLOW_REINIT, GFP_KERNEL))
@@ -672,10 +672,23 @@ int mddev_init(struct mddev *mddev)
if (percpu_ref_init(&mddev->writes_pending, no_op,
PERCPU_REF_ALLOW_REINIT, GFP_KERNEL)) {
- percpu_ref_exit(&mddev->active_io);
- return -ENOMEM;
+ err = -ENOMEM;
+ goto exit_acitve_io;
}
+ err = bioset_init(&mddev->bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
+ if (err)
+ goto exit_writes_pending;
+
+ err = bioset_init(&mddev->sync_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
+ if (err)
+ goto exit_bio_set;
+
+ err = bioset_init(&mddev->io_clone_set, BIO_POOL_SIZE,
+ offsetof(struct md_io_clone, bio_clone), 0);
+ if (err)
+ goto exit_sync_set;
+
/* We want to start with the refcount at zero */
percpu_ref_put(&mddev->writes_pending);
@@ -705,11 +718,24 @@ int mddev_init(struct mddev *mddev)
INIT_WORK(&mddev->del_work, mddev_delayed_delete);
return 0;
+
+exit_sync_set:
+ bioset_exit(&mddev->sync_set);
+exit_bio_set:
+ bioset_exit(&mddev->bio_set);
+exit_writes_pending:
+ percpu_ref_exit(&mddev->writes_pending);
+exit_acitve_io:
+ percpu_ref_exit(&mddev->active_io);
+ return err;
}
EXPORT_SYMBOL_GPL(mddev_init);
void mddev_destroy(struct mddev *mddev)
{
+ bioset_exit(&mddev->bio_set);
+ bioset_exit(&mddev->sync_set);
+ bioset_exit(&mddev->io_clone_set);
percpu_ref_exit(&mddev->active_io);
percpu_ref_exit(&mddev->writes_pending);
}
@@ -6152,25 +6178,6 @@ int md_run(struct mddev *mddev)
sysfs_notify_dirent_safe(rdev->sysfs_state);
nowait = nowait && bdev_nowait(rdev->bdev);
}
-
- if (!bioset_initialized(&mddev->bio_set)) {
- err = bioset_init(&mddev->bio_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
- if (err)
- return err;
- }
- if (!bioset_initialized(&mddev->sync_set)) {
- err = bioset_init(&mddev->sync_set, BIO_POOL_SIZE, 0, BIOSET_NEED_BVECS);
- if (err)
- goto exit_bio_set;
- }
-
- if (!bioset_initialized(&mddev->io_clone_set)) {
- err = bioset_init(&mddev->io_clone_set, BIO_POOL_SIZE,
- offsetof(struct md_io_clone, bio_clone), 0);
- if (err)
- goto exit_sync_set;
- }
-
spin_lock(&pers_lock);
pers = find_pers(mddev->level, mddev->clevel);
if (!pers || !try_module_get(pers->owner)) {
@@ -6181,8 +6188,7 @@ int md_run(struct mddev *mddev)
else
pr_warn("md: personality for level %s is not loaded!\n",
mddev->clevel);
- err = -EINVAL;
- goto abort;
+ return -EINVAL;
}
spin_unlock(&pers_lock);
if (mddev->level != pers->level) {
@@ -6195,8 +6201,7 @@ int md_run(struct mddev *mddev)
pers->start_reshape == NULL) {
/* This personality cannot handle reshaping... */
module_put(pers->owner);
- err = -EINVAL;
- goto abort;
+ return -EINVAL;
}
if (pers->sync_request) {
@@ -6323,12 +6328,6 @@ int md_run(struct mddev *mddev)
mddev->private = NULL;
module_put(pers->owner);
mddev->bitmap_ops->destroy(mddev);
-abort:
- bioset_exit(&mddev->io_clone_set);
-exit_sync_set:
- bioset_exit(&mddev->sync_set);
-exit_bio_set:
- bioset_exit(&mddev->bio_set);
return err;
}
EXPORT_SYMBOL_GPL(md_run);
@@ -6484,12 +6483,14 @@ static void __md_stop_writes(struct mddev *mddev)
{
del_timer_sync(&mddev->safemode_timer);
- if (mddev->pers && mddev->pers->quiesce) {
- mddev->pers->quiesce(mddev, 1);
- mddev->pers->quiesce(mddev, 0);
- }
+ if (md_is_rdwr(mddev) || !mddev_is_dm(mddev)) {
+ if (mddev->pers && mddev->pers->quiesce) {
+ mddev->pers->quiesce(mddev, 1);
+ mddev->pers->quiesce(mddev, 0);
+ }
- mddev->bitmap_ops->flush(mddev);
+ mddev->bitmap_ops->flush(mddev);
+ }
if (md_is_rdwr(mddev) &&
((!mddev->in_sync && !mddev_is_clustered(mddev)) ||
@@ -6544,10 +6545,6 @@ static void __md_stop(struct mddev *mddev)
mddev->to_remove = &md_redundancy_group;
module_put(pers->owner);
clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
-
- bioset_exit(&mddev->bio_set);
- bioset_exit(&mddev->sync_set);
- bioset_exit(&mddev->io_clone_set);
}
void md_stop(struct mddev *mddev)
diff --git a/drivers/md/md.h b/drivers/md/md.h
index bd5d95b1bac7..eba943cbd3b2 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -1001,6 +1001,11 @@ void mddev_update_io_opt(struct mddev *mddev, unsigned int nr_stripes);
extern const struct block_device_operations md_fops;
+static inline bool md_cloned_bio(struct mddev *mddev, struct bio *bio)
+{
+ return bio->bi_pool == &mddev->io_clone_set;
+}
+
/*
* MD devices can be used undeneath by DM, in which case ->gendisk is NULL.
*/
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 32b2eb2a169c..c1ed903fa9c0 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1322,11 +1322,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
bool r1bio_existed = !!r1_bio;
/*
- * If r1_bio is set, we are blocking the raid1d thread
- * so there is a tiny risk of deadlock. So ask for
+ * An md cloned bio indicates we are in the error path.
+ * This is more reliable than checking r1_bio, which might
+ * be NULL even in the error path if a failed bio was split.
+ */
+ bool err_path = md_cloned_bio(mddev, bio);
+
+ /*
+ * If we are in the error path, we are blocking the raid1d
+ * thread so there is a tiny risk of deadlock. So ask for
* emergency memory if needed.
*/
- gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
+ gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
/*
* Still need barrier for READ in case that whole
@@ -1400,7 +1407,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
}
r1_bio->read_disk = rdisk;
- if (!r1bio_existed) {
+ if (likely(!md_cloned_bio(mddev, bio))) {
md_account_bio(mddev, &bio);
r1_bio->master_bio = bio;
}
@@ -1427,7 +1434,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
raid_end_bio_io(r1_bio);
}
-static void raid1_write_request(struct mddev *mddev, struct bio *bio,
+static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
int max_write_sectors)
{
struct r1conf *conf = mddev->private;
@@ -1447,7 +1454,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
DEFINE_WAIT(w);
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
for (;;) {
prepare_to_wait(&conf->wait_barrier,
@@ -1469,7 +1476,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
if (!wait_barrier(conf, bio->bi_iter.bi_sector,
bio->bi_opf & REQ_NOWAIT)) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
retry_write:
@@ -1557,7 +1564,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
mddev_add_trace_msg(mddev, "raid1 wait rdev %d blocked",
blocked_rdev->raid_disk);
@@ -1668,7 +1675,8 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
/* In case raid1d snuck in to freeze_array */
wake_up_barrier(conf);
- return;
+ return true;
+
err_handle:
for (k = 0; k < i; k++) {
if (r1_bio->bios[k]) {
@@ -1680,6 +1688,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
bio->bi_status = errno_to_blk_status(error);
set_bit(R1BIO_Uptodate, &r1_bio->state);
raid_end_bio_io(r1_bio);
+ return false;
}
static bool raid1_make_request(struct mddev *mddev, struct bio *bio)
@@ -1703,8 +1712,9 @@ static bool raid1_make_request(struct mddev *mddev, struct bio *bio)
if (bio_data_dir(bio) == READ)
raid1_read_request(mddev, bio, sectors, NULL);
else {
- md_write_start(mddev,bio);
- raid1_write_request(mddev, bio, sectors);
+ md_write_start(mddev, bio);
+ if (!raid1_write_request(mddev, bio, sectors))
+ md_write_end(mddev);
}
return true;
}
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 468c1dfa9d6f..875759fa86ca 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1143,7 +1143,7 @@ static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf,
}
static void raid10_read_request(struct mddev *mddev, struct bio *bio,
- struct r10bio *r10_bio, bool io_accounting)
+ struct r10bio *r10_bio)
{
struct r10conf *conf = mddev->private;
struct bio *read_bio;
@@ -1152,9 +1152,22 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
char b[BDEVNAME_SIZE];
int slot = r10_bio->read_slot;
struct md_rdev *err_rdev = NULL;
- gfp_t gfp = GFP_NOIO;
int error;
+ /*
+ * An md cloned bio indicates we are in the error path.
+ * This is more reliable than checking slot, which might
+ * be -1 even in the error path if a failed bio was split.
+ */
+ bool err_path = md_cloned_bio(mddev, bio);
+
+ /*
+ * If we are in the error path, we are blocking the raid10d
+ * thread so there is a tiny risk of deadlock. So ask for
+ * emergency memory if needed.
+ */
+ gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
+
if (slot >= 0 && r10_bio->devs[slot].rdev) {
/*
* This is an error retry, but we cannot
@@ -1164,11 +1177,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
* we lose the device name in error messages.
*/
int disk;
- /*
- * As we are blocking raid10, it is a little safer to
- * use __GFP_HIGH.
- */
- gfp = GFP_NOIO | __GFP_HIGH;
disk = r10_bio->devs[slot].devnum;
err_rdev = conf->mirrors[disk].rdev;
@@ -1220,7 +1228,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
}
slot = r10_bio->read_slot;
- if (io_accounting) {
+ if (likely(!md_cloned_bio(mddev, bio))) {
md_account_bio(mddev, &bio);
r10_bio->master_bio = bio;
}
@@ -1349,7 +1357,7 @@ static void wait_blocked_dev(struct mddev *mddev, struct r10bio *r10_bio)
}
}
-static void raid10_write_request(struct mddev *mddev, struct bio *bio,
+static bool raid10_write_request(struct mddev *mddev, struct bio *bio,
struct r10bio *r10_bio)
{
struct r10conf *conf = mddev->private;
@@ -1366,7 +1374,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
/* Bail out if REQ_NOWAIT is set for the bio */
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
for (;;) {
prepare_to_wait(&conf->wait_barrier,
@@ -1382,7 +1390,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
sectors = r10_bio->sectors;
if (!regular_request_wait(mddev, conf, bio, sectors)) {
free_r10bio(r10_bio);
- return;
+ return false;
}
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
@@ -1399,7 +1407,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
if (bio->bi_opf & REQ_NOWAIT) {
allow_barrier(conf);
bio_wouldblock_error(bio);
- return;
+ return false;
}
mddev_add_trace_msg(conf->mddev,
"raid10 wait reshape metadata");
@@ -1507,7 +1515,8 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
raid10_write_one_disk(mddev, r10_bio, bio, true, i);
}
one_write_done(r10_bio);
- return;
+ return true;
+
err_handle:
for (k = 0; k < i; k++) {
int d = r10_bio->devs[k].devnum;
@@ -1527,10 +1536,12 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
bio->bi_status = errno_to_blk_status(error);
set_bit(R10BIO_Uptodate, &r10_bio->state);
raid_end_bio_io(r10_bio);
+ return false;
}
-static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
+static bool __make_request(struct mddev *mddev, struct bio *bio, int sectors)
{
+ bool ret;
struct r10conf *conf = mddev->private;
struct r10bio *r10_bio;
@@ -1546,10 +1557,13 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) *
conf->geo.raid_disks);
+ ret = true;
if (bio_data_dir(bio) == READ)
- raid10_read_request(mddev, bio, r10_bio, true);
+ raid10_read_request(mddev, bio, r10_bio);
else
- raid10_write_request(mddev, bio, r10_bio);
+ ret = raid10_write_request(mddev, bio, r10_bio);
+
+ return ret;
}
static void raid_end_discard_bio(struct r10bio *r10bio)
@@ -1900,7 +1914,8 @@ static bool raid10_make_request(struct mddev *mddev, struct bio *bio)
sectors = chunk_sects -
(bio->bi_iter.bi_sector &
(chunk_sects - 1));
- __make_request(mddev, bio, sectors);
+ if (!__make_request(mddev, bio, sectors))
+ md_write_end(mddev);
/* In case raid10d snuck in to freeze_array */
wake_up_barrier(conf);
@@ -2882,7 +2897,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)
rdev_dec_pending(rdev, mddev);
r10_bio->state = 0;
- raid10_read_request(mddev, r10_bio->master_bio, r10_bio, false);
+ raid10_read_request(mddev, r10_bio->master_bio, r10_bio);
/*
* allow_barrier after re-submit to ensure no sync io
* can be issued while regular io pending.
diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.c b/drivers/media/platform/chips-media/wave5/wave5-helper.c
index d60841c54a80..f5deae948092 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-helper.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-helper.c
@@ -25,6 +25,11 @@ const char *state_to_str(enum vpu_instance_state state)
}
}
+int wave5_kfifo_alloc(struct vpu_instance *inst)
+{
+ return kfifo_alloc(&inst->irq_status, 16 * sizeof(int), GFP_KERNEL);
+}
+
void wave5_cleanup_instance(struct vpu_instance *inst)
{
int i;
@@ -47,7 +52,7 @@ void wave5_cleanup_instance(struct vpu_instance *inst)
v4l2_fh_del(&inst->v4l2_fh);
v4l2_fh_exit(&inst->v4l2_fh);
}
- list_del_init(&inst->list);
+ kfifo_free(&inst->irq_status);
ida_free(&inst->dev->inst_ida, inst->id);
kfree(inst->codec_info);
kfree(inst);
@@ -60,8 +65,31 @@ int wave5_vpu_release_device(struct file *filp,
struct vpu_instance *inst = wave5_to_vpu_inst(filp->private_data);
struct vpu_device *dev = inst->dev;
int ret = 0;
+ unsigned long flags;
+
+ /*
+ * To prevent Null reference exception, the existing irq handler were
+ * separated to two modules.
+ * One is to queue interrupt reason into the irq handler,
+ * the other is irq_thread to call the wave5_vpu_dec_finish_decode
+ * to get decoded frame.
+ * The list of instances should be protected between all flow of the
+ * decoding process, but to protect the list in the irq_handler, spin lock
+ * should be used, and mutex should be used in the irq_thread because spin lock
+ * is not able to be used because mutex is already being used
+ * in the wave5_vpu_dec_finish_decode.
+ * So the spin lock and mutex were used to protect the list in the release function.
+ */
+ ret = mutex_lock_interruptible(&inst->dev->irq_lock);
+ if (ret)
+ return ret;
+ spin_lock_irqsave(&inst->dev->irq_spinlock, flags);
+ list_del_init(&inst->list);
+ spin_unlock_irqrestore(&inst->dev->irq_spinlock, flags);
+ mutex_unlock(&inst->dev->irq_lock);
v4l2_m2m_ctx_release(inst->v4l2_fh.m2m_ctx);
+
if (inst->state != VPU_INST_STATE_NONE) {
u32 fail_res;
diff --git a/drivers/media/platform/chips-media/wave5/wave5-helper.h b/drivers/media/platform/chips-media/wave5/wave5-helper.h
index 6cee1c14d3ce..f84a0365186a 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-helper.h
+++ b/drivers/media/platform/chips-media/wave5/wave5-helper.h
@@ -28,4 +28,5 @@ const struct vpu_format *wave5_find_vpu_fmt_by_idx(unsigned int idx,
const struct vpu_format fmt_list[MAX_FMTS]);
enum wave_std wave5_to_vpu_std(unsigned int v4l2_pix_fmt, enum vpu_instance_type type);
void wave5_return_bufs(struct vb2_queue *q, u32 state);
+int wave5_kfifo_alloc(struct vpu_instance *inst);
#endif
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
index fba3d7cc4cca..19510a363724 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
@@ -1923,6 +1923,11 @@ static int wave5_vpu_open_dec(struct file *filp)
inst->xfer_func = V4L2_XFER_FUNC_DEFAULT;
init_completion(&inst->irq_done);
+ ret = wave5_kfifo_alloc(inst);
+ if (ret) {
+ dev_err(inst->dev->dev, "failed to allocate fifo\n");
+ goto cleanup_inst;
+ }
inst->id = ida_alloc(&inst->dev->inst_ida, GFP_KERNEL);
if (inst->id < 0) {
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
index 0e92eca18236..917264348b55 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
@@ -1705,6 +1705,11 @@ static int wave5_vpu_open_enc(struct file *filp)
inst->frame_rate = 30;
init_completion(&inst->irq_done);
+ ret = wave5_kfifo_alloc(inst);
+ if (ret) {
+ dev_err(inst->dev->dev, "failed to allocate fifo\n");
+ goto cleanup_inst;
+ }
inst->id = ida_alloc(&inst->dev->inst_ida, GFP_KERNEL);
if (inst->id < 0) {
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu.c b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
index 4f7eda0504ab..5ec14a7c70c7 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu.c
@@ -50,8 +50,11 @@ static void wave5_vpu_handle_irq(void *dev_id)
u32 seq_done;
u32 cmd_done;
u32 irq_reason;
- struct vpu_instance *inst;
+ u32 irq_subreason;
+ struct vpu_instance *inst, *tmp;
struct vpu_device *dev = dev_id;
+ int val;
+ unsigned long flags;
irq_reason = wave5_vdi_read_register(dev, W5_VPU_VINT_REASON);
seq_done = wave5_vdi_read_register(dev, W5_RET_SEQ_DONE_INSTANCE_INFO);
@@ -59,7 +62,8 @@ static void wave5_vpu_handle_irq(void *dev_id)
wave5_vdi_write_register(dev, W5_VPU_VINT_REASON_CLR, irq_reason);
wave5_vdi_write_register(dev, W5_VPU_VINT_CLEAR, 0x1);
- list_for_each_entry(inst, &dev->instances, list) {
+ spin_lock_irqsave(&dev->irq_spinlock, flags);
+ list_for_each_entry_safe(inst, tmp, &dev->instances, list) {
if (irq_reason & BIT(INT_WAVE5_INIT_SEQ) ||
irq_reason & BIT(INT_WAVE5_ENC_SET_PARAM)) {
@@ -81,22 +85,54 @@ static void wave5_vpu_handle_irq(void *dev_id)
irq_reason & BIT(INT_WAVE5_ENC_PIC)) {
if (cmd_done & BIT(inst->id)) {
cmd_done &= ~BIT(inst->id);
- wave5_vdi_write_register(dev, W5_RET_QUEUE_CMD_DONE_INST,
- cmd_done);
- inst->ops->finish_process(inst);
+ if (dev->irq >= 0) {
+ irq_subreason =
+ wave5_vdi_read_register(dev, W5_VPU_VINT_REASON);
+ if (!(irq_subreason & BIT(INT_WAVE5_DEC_PIC)))
+ wave5_vdi_write_register(dev,
+ W5_RET_QUEUE_CMD_DONE_INST,
+ cmd_done);
+ }
+ val = BIT(INT_WAVE5_DEC_PIC);
+ kfifo_in(&inst->irq_status, &val, sizeof(int));
}
}
+ }
+ spin_unlock_irqrestore(&dev->irq_spinlock, flags);
+
+ if (dev->irq < 0)
+ up(&dev->irq_sem);
+}
+
+static irqreturn_t wave5_vpu_irq(int irq, void *dev_id)
+{
+ struct vpu_device *dev = dev_id;
- wave5_vpu_clear_interrupt(inst, irq_reason);
+ if (wave5_vdi_read_register(dev, W5_VPU_VPU_INT_STS)) {
+ wave5_vpu_handle_irq(dev);
+ return IRQ_WAKE_THREAD;
}
+
+ return IRQ_HANDLED;
}
static irqreturn_t wave5_vpu_irq_thread(int irq, void *dev_id)
{
struct vpu_device *dev = dev_id;
+ struct vpu_instance *inst, *tmp;
+ int irq_status, ret;
- if (wave5_vdi_read_register(dev, W5_VPU_VPU_INT_STS))
- wave5_vpu_handle_irq(dev);
+ mutex_lock(&dev->irq_lock);
+ list_for_each_entry_safe(inst, tmp, &dev->instances, list) {
+ while (kfifo_len(&inst->irq_status)) {
+ ret = kfifo_out(&inst->irq_status, &irq_status, sizeof(int));
+ if (!ret)
+ break;
+
+ inst->ops->finish_process(inst);
+ }
+ }
+ mutex_unlock(&dev->irq_lock);
return IRQ_HANDLED;
}
@@ -120,6 +156,35 @@ static enum hrtimer_restart wave5_vpu_timer_callback(struct hrtimer *timer)
return HRTIMER_RESTART;
}
+static int irq_thread(void *data)
+{
+ struct vpu_device *dev = (struct vpu_device *)data;
+ struct vpu_instance *inst, *tmp;
+ int irq_status, ret;
+
+ while (!kthread_should_stop()) {
+ if (down_interruptible(&dev->irq_sem))
+ continue;
+
+ if (kthread_should_stop())
+ break;
+
+ mutex_lock(&dev->irq_lock);
+ list_for_each_entry_safe(inst, tmp, &dev->instances, list) {
+ while (kfifo_len(&inst->irq_status)) {
+ ret = kfifo_out(&inst->irq_status, &irq_status, sizeof(int));
+ if (!ret)
+ break;
+
+ inst->ops->finish_process(inst);
+ }
+ }
+ mutex_unlock(&dev->irq_lock);
+ }
+
+ return 0;
+}
+
static int wave5_vpu_load_firmware(struct device *dev, const char *fw_name,
u32 *revision)
{
@@ -184,6 +249,8 @@ static int wave5_vpu_probe(struct platform_device *pdev)
mutex_init(&dev->dev_lock);
mutex_init(&dev->hw_lock);
+ mutex_init(&dev->irq_lock);
+ spin_lock_init(&dev->irq_spinlock);
dev_set_drvdata(&pdev->dev, dev);
dev->dev = &pdev->dev;
@@ -226,9 +293,13 @@ static int wave5_vpu_probe(struct platform_device *pdev)
}
dev->product = wave5_vpu_get_product_id(dev);
+ INIT_LIST_HEAD(&dev->instances);
+
dev->irq = platform_get_irq(pdev, 0);
if (dev->irq < 0) {
dev_err(&pdev->dev, "failed to get irq resource, falling back to polling\n");
+ sema_init(&dev->irq_sem, 1);
+ dev->irq_thread = kthread_run(irq_thread, dev, "irq thread");
hrtimer_init(&dev->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_PINNED);
dev->hrtimer.function = &wave5_vpu_timer_callback;
dev->worker = kthread_create_worker(0, "vpu_irq_thread");
@@ -240,7 +311,7 @@ static int wave5_vpu_probe(struct platform_device *pdev)
dev->vpu_poll_interval = vpu_poll_interval;
kthread_init_work(&dev->work, wave5_vpu_irq_work_fn);
} else {
- ret = devm_request_threaded_irq(&pdev->dev, dev->irq, NULL,
+ ret = devm_request_threaded_irq(&pdev->dev, dev->irq, wave5_vpu_irq,
wave5_vpu_irq_thread, IRQF_ONESHOT, "vpu_irq", dev);
if (ret) {
dev_err(&pdev->dev, "Register interrupt handler, fail: %d\n", ret);
@@ -248,7 +319,6 @@ static int wave5_vpu_probe(struct platform_device *pdev)
}
}
- INIT_LIST_HEAD(&dev->instances);
ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
if (ret) {
dev_err(&pdev->dev, "v4l2_device_register, fail: %d\n", ret);
@@ -310,6 +380,12 @@ static void wave5_vpu_remove(struct platform_device *pdev)
v4l2_device_unregister(&dev->v4l2_dev);
if (dev->irq < 0) {
+ if (dev->irq_thread) {
+ kthread_stop(dev->irq_thread);
+ up(&dev->irq_sem);
+ dev->irq_thread = NULL;
+ }
+
hrtimer_cancel(&dev->hrtimer);
kthread_cancel_work_sync(&dev->work);
kthread_destroy_worker(dev->worker);
@@ -317,6 +393,7 @@ static void wave5_vpu_remove(struct platform_device *pdev)
mutex_destroy(&dev->dev_lock);
mutex_destroy(&dev->hw_lock);
+ mutex_destroy(&dev->irq_lock);
reset_control_assert(dev->resets);
clk_bulk_disable_unprepare(dev->num_clks, dev->clks);
wave5_vdi_release(&pdev->dev);
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h
index 703b2413dafa..70b8324bbb78 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpuapi.h
@@ -8,6 +8,7 @@
#ifndef VPUAPI_H_INCLUDED
#define VPUAPI_H_INCLUDED
+#include <linux/kfifo.h>
#include <linux/idr.h>
#include <linux/genalloc.h>
#include <media/v4l2-device.h>
@@ -747,6 +748,7 @@ struct vpu_device {
struct video_device *video_dev_enc;
struct mutex dev_lock; /* lock for the src, dst v4l2 queues */
struct mutex hw_lock; /* lock hw configurations */
+ struct mutex irq_lock;
int irq;
enum product_id product;
struct vpu_attr attr;
@@ -764,7 +766,10 @@ struct vpu_device {
struct kthread_worker *worker;
int vpu_poll_interval;
int num_clks;
+ struct task_struct *irq_thread;
+ struct semaphore irq_sem; /* signal to irq_thread when interrupt happens*/
struct reset_control *resets;
+ spinlock_t irq_spinlock; /* protect instances list */
};
struct vpu_instance;
@@ -788,6 +793,7 @@ struct vpu_instance {
enum v4l2_ycbcr_encoding ycbcr_enc;
enum v4l2_quantization quantization;
+ struct kfifo irq_status;
enum vpu_instance_state state;
enum vpu_instance_type type;
const struct vpu_instance_ops *ops;
diff --git a/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-bytecap.c b/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-bytecap.c
index b461d86bed54..b1e35df41a9c 100644
--- a/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-bytecap.c
+++ b/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-bytecap.c
@@ -112,6 +112,7 @@ struct dcmipp_bytecap_device {
u32 sequence;
struct media_pipeline pipe;
struct v4l2_subdev *s_subdev;
+ u32 s_subdev_pad_nb;
enum dcmipp_state state;
@@ -337,33 +338,6 @@ static const struct v4l2_ioctl_ops dcmipp_bytecap_ioctl_ops = {
.vidioc_streamoff = vb2_ioctl_streamoff,
};
-static int dcmipp_pipeline_s_stream(struct dcmipp_bytecap_device *vcap,
- int state)
-{
- struct media_pad *pad;
- int ret;
-
- /*
- * Get source subdev - since link is IMMUTABLE, pointer is cached
- * within the dcmipp_bytecap_device structure
- */
- if (!vcap->s_subdev) {
- pad = media_pad_remote_pad_first(&vcap->vdev.entity.pads[0]);
- if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
- return -EINVAL;
- vcap->s_subdev = media_entity_to_v4l2_subdev(pad->entity);
- }
-
- ret = v4l2_subdev_call(vcap->s_subdev, video, s_stream, state);
- if (ret < 0) {
- dev_err(vcap->dev, "failed to %s streaming (%d)\n",
- state ? "start" : "stop", ret);
- return ret;
- }
-
- return 0;
-}
-
static void dcmipp_start_capture(struct dcmipp_bytecap_device *vcap,
struct dcmipp_buf *buf)
{
@@ -395,11 +369,26 @@ static int dcmipp_bytecap_start_streaming(struct vb2_queue *vq,
struct dcmipp_bytecap_device *vcap = vb2_get_drv_priv(vq);
struct media_entity *entity = &vcap->vdev.entity;
struct dcmipp_buf *buf;
+ struct media_pad *pad;
int ret;
vcap->sequence = 0;
memset(&vcap->count, 0, sizeof(vcap->count));
+ /*
+ * Get source subdev - since link is IMMUTABLE, pointer is cached
+ * within the dcmipp_bytecap_device structure
+ */
+ if (!vcap->s_subdev) {
+ pad = media_pad_remote_pad_first(&vcap->vdev.entity.pads[0]);
+ if (!pad || !is_media_entity_v4l2_subdev(pad->entity)) {
+ ret = -EINVAL;
+ goto err_buffer_done;
+ }
+ vcap->s_subdev = media_entity_to_v4l2_subdev(pad->entity);
+ vcap->s_subdev_pad_nb = pad->index;
+ }
+
ret = pm_runtime_resume_and_get(vcap->dev);
if (ret < 0) {
dev_err(vcap->dev, "%s: Failed to start streaming, cannot get sync (%d)\n",
@@ -414,7 +403,8 @@ static int dcmipp_bytecap_start_streaming(struct vb2_queue *vq,
goto err_pm_put;
}
- ret = dcmipp_pipeline_s_stream(vcap, 1);
+ ret = v4l2_subdev_enable_streams(vcap->s_subdev,
+ vcap->s_subdev_pad_nb, BIT_ULL(0));
if (ret)
goto err_media_pipeline_stop;
@@ -482,7 +472,10 @@ static void dcmipp_bytecap_stop_streaming(struct vb2_queue *vq)
int ret;
u32 status;
- dcmipp_pipeline_s_stream(vcap, 0);
+ ret = v4l2_subdev_disable_streams(vcap->s_subdev,
+ vcap->s_subdev_pad_nb, BIT_ULL(0));
+ if (ret)
+ dev_warn(vcap->dev, "Failed to disable stream\n");
/* Stop the media pipeline */
media_pipeline_stop(vcap->vdev.entity.pads);
diff --git a/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-byteproc.c b/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-byteproc.c
index 5a361ad6b023..a19c8235af56 100644
--- a/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-byteproc.c
+++ b/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-byteproc.c
@@ -78,7 +78,6 @@ struct dcmipp_byteproc_device {
struct v4l2_subdev sd;
struct device *dev;
void __iomem *regs;
- bool streaming;
};
static const struct v4l2_mbus_framefmt fmt_default = {
@@ -239,11 +238,10 @@ static int dcmipp_byteproc_set_fmt(struct v4l2_subdev *sd,
struct v4l2_subdev_state *sd_state,
struct v4l2_subdev_format *fmt)
{
- struct dcmipp_byteproc_device *byteproc = v4l2_get_subdevdata(sd);
struct v4l2_mbus_framefmt *mf;
struct v4l2_rect *crop, *compose;
- if (byteproc->streaming)
+ if (v4l2_subdev_is_streaming(sd))
return -EBUSY;
mf = v4l2_subdev_state_get_format(sd_state, fmt->pad);
@@ -382,30 +380,19 @@ static int dcmipp_byteproc_set_selection(struct v4l2_subdev *sd,
return 0;
}
-static const struct v4l2_subdev_pad_ops dcmipp_byteproc_pad_ops = {
- .enum_mbus_code = dcmipp_byteproc_enum_mbus_code,
- .enum_frame_size = dcmipp_byteproc_enum_frame_size,
- .get_fmt = v4l2_subdev_get_fmt,
- .set_fmt = dcmipp_byteproc_set_fmt,
- .get_selection = dcmipp_byteproc_get_selection,
- .set_selection = dcmipp_byteproc_set_selection,
-};
-
static int dcmipp_byteproc_configure_scale_crop
- (struct dcmipp_byteproc_device *byteproc)
+ (struct dcmipp_byteproc_device *byteproc,
+ struct v4l2_subdev_state *state)
{
const struct dcmipp_byteproc_pix_map *vpix;
- struct v4l2_subdev_state *state;
struct v4l2_mbus_framefmt *sink_fmt;
u32 hprediv, vprediv;
struct v4l2_rect *compose, *crop;
u32 val = 0;
- state = v4l2_subdev_lock_and_get_active_state(&byteproc->sd);
sink_fmt = v4l2_subdev_state_get_format(state, 0);
compose = v4l2_subdev_state_get_compose(state, 0);
crop = v4l2_subdev_state_get_crop(state, 1);
- v4l2_subdev_unlock_state(state);
/* find output format bpp */
vpix = dcmipp_byteproc_pix_map_by_code(sink_fmt->code);
@@ -460,48 +447,73 @@ static int dcmipp_byteproc_configure_scale_crop
return 0;
}
-static int dcmipp_byteproc_s_stream(struct v4l2_subdev *sd, int enable)
+static int dcmipp_byteproc_enable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ u32 pad, u64 streams_mask)
{
struct dcmipp_byteproc_device *byteproc = v4l2_get_subdevdata(sd);
struct v4l2_subdev *s_subdev;
- struct media_pad *pad;
- int ret = 0;
+ struct media_pad *s_pad;
+ int ret;
/* Get source subdev */
- pad = media_pad_remote_pad_first(&sd->entity.pads[0]);
- if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
+ s_pad = media_pad_remote_pad_first(&sd->entity.pads[0]);
+ if (!s_pad || !is_media_entity_v4l2_subdev(s_pad->entity))
return -EINVAL;
- s_subdev = media_entity_to_v4l2_subdev(pad->entity);
-
- if (enable) {
- ret = dcmipp_byteproc_configure_scale_crop(byteproc);
- if (ret)
- return ret;
-
- ret = v4l2_subdev_call(s_subdev, video, s_stream, enable);
- if (ret < 0) {
- dev_err(byteproc->dev,
- "failed to start source subdev streaming (%d)\n",
- ret);
- return ret;
- }
- } else {
- ret = v4l2_subdev_call(s_subdev, video, s_stream, enable);
- if (ret < 0) {
- dev_err(byteproc->dev,
- "failed to stop source subdev streaming (%d)\n",
- ret);
- return ret;
- }
+ s_subdev = media_entity_to_v4l2_subdev(s_pad->entity);
+
+ ret = dcmipp_byteproc_configure_scale_crop(byteproc, state);
+ if (ret)
+ return ret;
+
+ ret = v4l2_subdev_enable_streams(s_subdev, s_pad->index, BIT_ULL(0));
+ if (ret < 0) {
+ dev_err(byteproc->dev,
+ "failed to start source subdev streaming (%d)\n", ret);
+ return ret;
}
- byteproc->streaming = enable;
+ return 0;
+}
+
+static int dcmipp_byteproc_disable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ u32 pad, u64 streams_mask)
+{
+ struct dcmipp_byteproc_device *byteproc = v4l2_get_subdevdata(sd);
+ struct v4l2_subdev *s_subdev;
+ struct media_pad *s_pad;
+ int ret;
+
+ /* Get source subdev */
+ s_pad = media_pad_remote_pad_first(&sd->entity.pads[0]);
+ if (!s_pad || !is_media_entity_v4l2_subdev(s_pad->entity))
+ return -EINVAL;
+ s_subdev = media_entity_to_v4l2_subdev(s_pad->entity);
+
+ ret = v4l2_subdev_disable_streams(s_subdev, s_pad->index, BIT_ULL(0));
+ if (ret < 0) {
+ dev_err(byteproc->dev,
+ "failed to start source subdev streaming (%d)\n", ret);
+ return ret;
+ }
return 0;
}
+static const struct v4l2_subdev_pad_ops dcmipp_byteproc_pad_ops = {
+ .enum_mbus_code = dcmipp_byteproc_enum_mbus_code,
+ .enum_frame_size = dcmipp_byteproc_enum_frame_size,
+ .get_fmt = v4l2_subdev_get_fmt,
+ .set_fmt = dcmipp_byteproc_set_fmt,
+ .get_selection = dcmipp_byteproc_get_selection,
+ .set_selection = dcmipp_byteproc_set_selection,
+ .enable_streams = dcmipp_byteproc_enable_streams,
+ .disable_streams = dcmipp_byteproc_disable_streams,
+};
+
static const struct v4l2_subdev_video_ops dcmipp_byteproc_video_ops = {
- .s_stream = dcmipp_byteproc_s_stream,
+ .s_stream = v4l2_subdev_s_stream_helper,
};
static const struct v4l2_subdev_ops dcmipp_byteproc_ops = {
diff --git a/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-parallel.c b/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-parallel.c
index 62c5c3331cfe..823c9da336a7 100644
--- a/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-parallel.c
+++ b/drivers/media/platform/st/stm32/stm32-dcmipp/dcmipp-parallel.c
@@ -129,7 +129,6 @@ struct dcmipp_par_device {
struct v4l2_subdev sd;
struct device *dev;
void __iomem *regs;
- bool streaming;
};
static const struct v4l2_mbus_framefmt fmt_default = {
@@ -230,7 +229,7 @@ static int dcmipp_par_set_fmt(struct v4l2_subdev *sd,
struct dcmipp_par_device *par = v4l2_get_subdevdata(sd);
struct v4l2_mbus_framefmt *mf;
- if (par->streaming)
+ if (v4l2_subdev_is_streaming(sd))
return -EBUSY;
mf = v4l2_subdev_state_get_format(sd_state, fmt->pad);
@@ -261,18 +260,11 @@ static int dcmipp_par_set_fmt(struct v4l2_subdev *sd,
return 0;
}
-static const struct v4l2_subdev_pad_ops dcmipp_par_pad_ops = {
- .enum_mbus_code = dcmipp_par_enum_mbus_code,
- .enum_frame_size = dcmipp_par_enum_frame_size,
- .get_fmt = v4l2_subdev_get_fmt,
- .set_fmt = dcmipp_par_set_fmt,
-};
-
-static int dcmipp_par_configure(struct dcmipp_par_device *par)
+static int dcmipp_par_configure(struct dcmipp_par_device *par,
+ struct v4l2_subdev_state *state)
{
u32 val = 0;
const struct dcmipp_par_pix_map *vpix;
- struct v4l2_subdev_state *state;
struct v4l2_mbus_framefmt *sink_fmt;
struct v4l2_mbus_framefmt *src_fmt;
@@ -306,10 +298,8 @@ static int dcmipp_par_configure(struct dcmipp_par_device *par)
}
/* Set format */
- state = v4l2_subdev_lock_and_get_active_state(&par->sd);
sink_fmt = v4l2_subdev_state_get_format(state, 0);
src_fmt = v4l2_subdev_state_get_format(state, 1);
- v4l2_subdev_unlock_state(state);
vpix = dcmipp_par_pix_map_by_code(sink_fmt->code, src_fmt->code);
if (!vpix) {
@@ -328,55 +318,79 @@ static int dcmipp_par_configure(struct dcmipp_par_device *par)
return 0;
}
-static int dcmipp_par_s_stream(struct v4l2_subdev *sd, int enable)
+static int dcmipp_par_enable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ u32 pad, u64 streams_mask)
{
struct dcmipp_par_device *par =
container_of(sd, struct dcmipp_par_device, sd);
struct v4l2_subdev *s_subdev;
- struct media_pad *pad;
- int ret = 0;
+ struct media_pad *s_pad;
+ int ret;
/* Get source subdev */
- pad = media_pad_remote_pad_first(&sd->entity.pads[0]);
- if (!pad || !is_media_entity_v4l2_subdev(pad->entity))
+ s_pad = media_pad_remote_pad_first(&sd->entity.pads[0]);
+ if (!s_pad || !is_media_entity_v4l2_subdev(s_pad->entity))
return -EINVAL;
- s_subdev = media_entity_to_v4l2_subdev(pad->entity);
-
- if (enable) {
- ret = dcmipp_par_configure(par);
- if (ret)
- return ret;
-
- /* Enable parallel interface */
- reg_set(par, DCMIPP_PRCR, DCMIPP_PRCR_ENABLE);
-
- ret = v4l2_subdev_call(s_subdev, video, s_stream, enable);
- if (ret < 0) {
- dev_err(par->dev,
- "failed to start source subdev streaming (%d)\n",
- ret);
- return ret;
- }
- } else {
- ret = v4l2_subdev_call(s_subdev, video, s_stream, enable);
- if (ret < 0) {
- dev_err(par->dev,
- "failed to stop source subdev streaming (%d)\n",
- ret);
- return ret;
- }
+ s_subdev = media_entity_to_v4l2_subdev(s_pad->entity);
+
+ ret = dcmipp_par_configure(par, state);
+ if (ret)
+ return ret;
+
+ /* Enable parallel interface */
+ reg_set(par, DCMIPP_PRCR, DCMIPP_PRCR_ENABLE);
+
+ ret = v4l2_subdev_enable_streams(s_subdev, s_pad->index, BIT_ULL(0));
+ if (ret < 0) {
+ dev_err(par->dev,
+ "failed to start source subdev streaming (%d)\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
- /* Disable parallel interface */
- reg_clear(par, DCMIPP_PRCR, DCMIPP_PRCR_ENABLE);
+static int dcmipp_par_disable_streams(struct v4l2_subdev *sd,
+ struct v4l2_subdev_state *state,
+ u32 pad, u64 streams_mask)
+{
+ struct dcmipp_par_device *par =
+ container_of(sd, struct dcmipp_par_device, sd);
+ struct v4l2_subdev *s_subdev;
+ struct media_pad *s_pad;
+ int ret;
+
+ /* Get source subdev */
+ s_pad = media_pad_remote_pad_first(&sd->entity.pads[0]);
+ if (!s_pad || !is_media_entity_v4l2_subdev(s_pad->entity))
+ return -EINVAL;
+ s_subdev = media_entity_to_v4l2_subdev(s_pad->entity);
+
+ ret = v4l2_subdev_disable_streams(s_subdev, s_pad->index, BIT_ULL(0));
+ if (ret < 0) {
+ dev_err(par->dev,
+ "failed to stop source subdev streaming (%d)\n", ret);
+ return ret;
}
- par->streaming = enable;
+ /* Disable parallel interface */
+ reg_clear(par, DCMIPP_PRCR, DCMIPP_PRCR_ENABLE);
- return ret;
+ return 0;
}
+static const struct v4l2_subdev_pad_ops dcmipp_par_pad_ops = {
+ .enum_mbus_code = dcmipp_par_enum_mbus_code,
+ .enum_frame_size = dcmipp_par_enum_frame_size,
+ .get_fmt = v4l2_subdev_get_fmt,
+ .set_fmt = dcmipp_par_set_fmt,
+ .enable_streams = dcmipp_par_enable_streams,
+ .disable_streams = dcmipp_par_disable_streams,
+};
+
static const struct v4l2_subdev_video_ops dcmipp_par_video_ops = {
- .s_stream = dcmipp_par_s_stream,
+ .s_stream = v4l2_subdev_s_stream_helper,
};
static const struct v4l2_subdev_ops dcmipp_par_ops = {
diff --git a/drivers/media/platform/verisilicon/hantro.h b/drivers/media/platform/verisilicon/hantro.h
index 811260dc3c77..439f0adb4e81 100644
--- a/drivers/media/platform/verisilicon/hantro.h
+++ b/drivers/media/platform/verisilicon/hantro.h
@@ -77,6 +77,7 @@ struct hantro_irq {
* @double_buffer: core needs double buffering
* @legacy_regs: core uses legacy register set
* @late_postproc: postproc must be set up at the end of the job
+ * @shared_devices: an array of device ids that cannot run concurrently
*/
struct hantro_variant {
unsigned int enc_offset;
@@ -101,6 +102,7 @@ struct hantro_variant {
unsigned int double_buffer : 1;
unsigned int legacy_regs : 1;
unsigned int late_postproc : 1;
+ const struct of_device_id *shared_devices;
};
/**
diff --git a/drivers/media/platform/verisilicon/hantro_drv.c b/drivers/media/platform/verisilicon/hantro_drv.c
index 137ca13eeed7..01d6ab67a8bf 100644
--- a/drivers/media/platform/verisilicon/hantro_drv.c
+++ b/drivers/media/platform/verisilicon/hantro_drv.c
@@ -13,6 +13,7 @@
#include <linux/clk.h>
#include <linux/module.h>
#include <linux/of.h>
+#include <linux/of_platform.h>
#include <linux/platform_device.h>
#include <linux/pm.h>
#include <linux/pm_runtime.h>
@@ -1038,6 +1039,41 @@ static int hantro_disable_multicore(struct hantro_dev *vpu)
return 0;
}
+static struct v4l2_m2m_dev *hantro_get_v4l2_m2m_dev(struct hantro_dev *vpu)
+{
+ struct device_node *node;
+ struct hantro_dev *shared_vpu;
+
+ if (!vpu->variant || !vpu->variant->shared_devices)
+ goto init_new_m2m_dev;
+
+ for_each_matching_node(node, vpu->variant->shared_devices) {
+ struct platform_device *pdev;
+ struct v4l2_m2m_dev *m2m_dev;
+
+ pdev = of_find_device_by_node(node);
+ if (!pdev)
+ continue;
+
+ shared_vpu = platform_get_drvdata(pdev);
+ if (IS_ERR_OR_NULL(shared_vpu) || shared_vpu == vpu) {
+ platform_device_put(pdev);
+ continue;
+ }
+
+ v4l2_m2m_get(shared_vpu->m2m_dev);
+ m2m_dev = shared_vpu->m2m_dev;
+ platform_device_put(pdev);
+
+ of_node_put(node);
+
+ return m2m_dev;
+ }
+
+init_new_m2m_dev:
+ return v4l2_m2m_init(&vpu_m2m_ops);
+}
+
static int hantro_probe(struct platform_device *pdev)
{
const struct of_device_id *match;
@@ -1189,7 +1225,7 @@ static int hantro_probe(struct platform_device *pdev)
}
platform_set_drvdata(pdev, vpu);
- vpu->m2m_dev = v4l2_m2m_init(&vpu_m2m_ops);
+ vpu->m2m_dev = hantro_get_v4l2_m2m_dev(vpu);
if (IS_ERR(vpu->m2m_dev)) {
v4l2_err(&vpu->v4l2_dev, "Failed to init mem2mem device\n");
ret = PTR_ERR(vpu->m2m_dev);
@@ -1228,7 +1264,7 @@ static int hantro_probe(struct platform_device *pdev)
hantro_remove_enc_func(vpu);
err_m2m_rel:
media_device_cleanup(&vpu->mdev);
- v4l2_m2m_release(vpu->m2m_dev);
+ v4l2_m2m_put(vpu->m2m_dev);
err_v4l2_unreg:
v4l2_device_unregister(&vpu->v4l2_dev);
err_clk_unprepare:
@@ -1251,7 +1287,7 @@ static void hantro_remove(struct platform_device *pdev)
hantro_remove_dec_func(vpu);
hantro_remove_enc_func(vpu);
media_device_cleanup(&vpu->mdev);
- v4l2_m2m_release(vpu->m2m_dev);
+ v4l2_m2m_put(vpu->m2m_dev);
v4l2_device_unregister(&vpu->v4l2_dev);
clk_bulk_unprepare(vpu->variant->num_clocks, vpu->clocks);
reset_control_assert(vpu->resets);
diff --git a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
index 74fd985a8aad..de27c142b7fe 100644
--- a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
+++ b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
@@ -361,6 +361,12 @@ const struct hantro_variant imx8mq_vpu_variant = {
.num_regs = ARRAY_SIZE(imx8mq_reg_names)
};
+static const struct of_device_id imx8mq_vpu_shared_resources[] = {
+ { .compatible = "nxp,imx8mq-vpu-g1", },
+ { .compatible = "nxp,imx8mq-vpu-g2", },
+ { /* sentinel */ }
+};
+
const struct hantro_variant imx8mq_vpu_g1_variant = {
.dec_fmts = imx8m_vpu_dec_fmts,
.num_dec_fmts = ARRAY_SIZE(imx8m_vpu_dec_fmts),
@@ -374,6 +380,7 @@ const struct hantro_variant imx8mq_vpu_g1_variant = {
.num_irqs = ARRAY_SIZE(imx8mq_irqs),
.clk_names = imx8mq_g1_clk_names,
.num_clocks = ARRAY_SIZE(imx8mq_g1_clk_names),
+ .shared_devices = imx8mq_vpu_shared_resources,
};
const struct hantro_variant imx8mq_vpu_g2_variant = {
@@ -389,6 +396,7 @@ const struct hantro_variant imx8mq_vpu_g2_variant = {
.num_irqs = ARRAY_SIZE(imx8mq_g2_irqs),
.clk_names = imx8mq_g2_clk_names,
.num_clocks = ARRAY_SIZE(imx8mq_g2_clk_names),
+ .shared_devices = imx8mq_vpu_shared_resources,
};
const struct hantro_variant imx8mm_vpu_g1_variant = {
diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
index eb22d6172462..85006fb18f72 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -90,6 +90,7 @@ static const char * const m2m_entity_name[] = {
* @job_work: worker to run queued jobs.
* @job_queue_flags: flags of the queue status, %QUEUE_PAUSED.
* @m2m_ops: driver callbacks
+ * @kref: device reference count
*/
struct v4l2_m2m_dev {
struct v4l2_m2m_ctx *curr_ctx;
@@ -109,6 +110,8 @@ struct v4l2_m2m_dev {
unsigned long job_queue_flags;
const struct v4l2_m2m_ops *m2m_ops;
+
+ struct kref kref;
};
static struct v4l2_m2m_queue_ctx *get_queue_ctx(struct v4l2_m2m_ctx *m2m_ctx,
@@ -1210,6 +1213,7 @@ struct v4l2_m2m_dev *v4l2_m2m_init(const struct v4l2_m2m_ops *m2m_ops)
INIT_LIST_HEAD(&m2m_dev->job_queue);
spin_lock_init(&m2m_dev->job_spinlock);
INIT_WORK(&m2m_dev->job_work, v4l2_m2m_device_run_work);
+ kref_init(&m2m_dev->kref);
return m2m_dev;
}
@@ -1221,6 +1225,25 @@ void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev)
}
EXPORT_SYMBOL_GPL(v4l2_m2m_release);
+void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev)
+{
+ kref_get(&m2m_dev->kref);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_get);
+
+static void v4l2_m2m_release_from_kref(struct kref *kref)
+{
+ struct v4l2_m2m_dev *m2m_dev = container_of(kref, struct v4l2_m2m_dev, kref);
+
+ v4l2_m2m_release(m2m_dev);
+}
+
+void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev)
+{
+ kref_put(&m2m_dev->kref, v4l2_m2m_release_from_kref);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_put);
+
struct v4l2_m2m_ctx *v4l2_m2m_ctx_init(struct v4l2_m2m_dev *m2m_dev,
void *drv_priv,
int (*queue_init)(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq))
diff --git a/drivers/mfd/intel-lpss.c b/drivers/mfd/intel-lpss.c
index 2a9018112dfc..6901a61a09a4 100644
--- a/drivers/mfd/intel-lpss.c
+++ b/drivers/mfd/intel-lpss.c
@@ -480,7 +480,7 @@ EXPORT_SYMBOL_NS_GPL(intel_lpss_remove, INTEL_LPSS);
static int resume_lpss_device(struct device *dev, void *data)
{
- if (!dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND))
+ if (!dev_pm_smart_suspend(dev))
pm_runtime_resume(dev);
return 0;
diff --git a/drivers/mmc/host/vub300.c b/drivers/mmc/host/vub300.c
index 35393f534de2..63989479ae83 100644
--- a/drivers/mmc/host/vub300.c
+++ b/drivers/mmc/host/vub300.c
@@ -370,6 +370,7 @@ static void vub300_delete(struct kref *kref)
{ /* kref callback - softirq */
struct vub300_mmc_host *vub300 = kref_to_vub300_mmc_host(kref);
struct mmc_host *mmc = vub300->mmc;
+
usb_free_urb(vub300->command_out_urb);
vub300->command_out_urb = NULL;
usb_free_urb(vub300->command_res_urb);
@@ -2114,19 +2115,19 @@ static int vub300_probe(struct usb_interface *interface,
command_out_urb = usb_alloc_urb(0, GFP_KERNEL);
if (!command_out_urb) {
retval = -ENOMEM;
- goto error0;
+ goto err_put_udev;
}
command_res_urb = usb_alloc_urb(0, GFP_KERNEL);
if (!command_res_urb) {
retval = -ENOMEM;
- goto error1;
+ goto err_free_out_urb;
}
/* this also allocates memory for our VUB300 mmc host device */
- mmc = mmc_alloc_host(sizeof(struct vub300_mmc_host), &udev->dev);
+ mmc = mmc_alloc_host(sizeof(*vub300), &udev->dev);
if (!mmc) {
retval = -ENOMEM;
dev_err(&udev->dev, "not enough memory for the mmc_host\n");
- goto error4;
+ goto err_free_res_urb;
}
/* MMC core transfer sizes tunable parameters */
mmc->caps = 0;
@@ -2279,7 +2280,7 @@ static int vub300_probe(struct usb_interface *interface,
dev_err(&vub300->udev->dev,
"Could not find two sets of bulk-in/out endpoint pairs\n");
retval = -EINVAL;
- goto error5;
+ goto err_free_host;
}
retval =
usb_control_msg(vub300->udev, usb_rcvctrlpipe(vub300->udev, 0),
@@ -2288,14 +2289,14 @@ static int vub300_probe(struct usb_interface *interface,
0x0000, 0x0000, &vub300->hc_info,
sizeof(vub300->hc_info), 1000);
if (retval < 0)
- goto error5;
+ goto err_free_host;
retval =
usb_control_msg(vub300->udev, usb_sndctrlpipe(vub300->udev, 0),
SET_ROM_WAIT_STATES,
USB_DIR_OUT | USB_TYPE_VENDOR | USB_RECIP_DEVICE,
firmware_rom_wait_states, 0x0000, NULL, 0, 1000);
if (retval < 0)
- goto error5;
+ goto err_free_host;
dev_info(&vub300->udev->dev,
"operating_mode = %s %s %d MHz %s %d byte USB packets\n",
(mmc->caps & MMC_CAP_SDIO_IRQ) ? "IRQs" : "POLL",
@@ -2310,7 +2311,7 @@ static int vub300_probe(struct usb_interface *interface,
0x0000, 0x0000, &vub300->system_port_status,
sizeof(vub300->system_port_status), 1000);
if (retval < 0) {
- goto error5;
+ goto err_free_host;
} else if (sizeof(vub300->system_port_status) == retval) {
vub300->card_present =
(0x0001 & vub300->system_port_status.port_flags) ? 1 : 0;
@@ -2318,7 +2319,7 @@ static int vub300_probe(struct usb_interface *interface,
(0x0010 & vub300->system_port_status.port_flags) ? 1 : 0;
} else {
retval = -EINVAL;
- goto error5;
+ goto err_free_host;
}
usb_set_intfdata(interface, vub300);
INIT_DELAYED_WORK(&vub300->pollwork, vub300_pollwork_thread);
@@ -2343,23 +2344,29 @@ static int vub300_probe(struct usb_interface *interface,
interface_to_InterfaceNumber(interface));
retval = mmc_add_host(mmc);
if (retval)
- goto error6;
+ goto err_stop_io;
return 0;
-error6:
- del_timer_sync(&vub300->inactivity_timer);
-error5:
+
+err_stop_io:
+ vub300->interface = NULL;
+ kref_put(&vub300->kref, vub300_delete);
+
+ return retval;
+
+err_free_host:
mmc_free_host(mmc);
/*
* and hence also frees vub300
* which is contained at the end of struct mmc
*/
-error4:
+err_free_res_urb:
usb_free_urb(command_res_urb);
-error1:
+err_free_out_urb:
usb_free_urb(command_out_urb);
-error0:
+err_put_udev:
usb_put_dev(udev);
+
return retval;
}
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index ac87ea847807..54199c3babcc 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -3126,7 +3126,7 @@ static void amt_link_setup(struct net_device *dev)
dev->addr_len = 0;
dev->priv_flags |= IFF_NO_QUEUE;
dev->lltx = true;
- dev->netns_local = true;
+ dev->netns_immutable = true;
dev->features |= NETIF_F_GSO_SOFTWARE;
dev->hw_features |= NETIF_F_SG | NETIF_F_HW_CSUM;
dev->hw_features |= NETIF_F_FRAGLIST | NETIF_F_RXCSUM;
@@ -3195,6 +3195,9 @@ static int amt_newlink(struct net *net, struct net_device *dev,
struct amt_dev *amt = netdev_priv(dev);
int err = -EINVAL;
+ if (!net_eq(net, dev_net(dev)))
+ return err;
+
amt->net = net;
amt->mode = nla_get_u32(data[IFLA_AMT_MODE]);
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index d10bbe4c3571..a6f3ed0bd86a 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -6088,7 +6088,7 @@ void bond_setup(struct net_device *bond_dev)
bond_dev->lltx = true;
/* Don't allow bond devices to change network namespaces. */
- bond_dev->netns_local = true;
+ bond_dev->netns_immutable = true;
/* By default, we declare the bond to be fully
* VLAN hardware accelerated capable. Special
diff --git a/drivers/net/ethernet/adi/adin1110.c b/drivers/net/ethernet/adi/adin1110.c
index 4352444ec6f6..71a2397edf2b 100644
--- a/drivers/net/ethernet/adi/adin1110.c
+++ b/drivers/net/ethernet/adi/adin1110.c
@@ -1602,7 +1602,7 @@ static int adin1110_probe_netdevs(struct adin1110_priv *priv)
netdev->netdev_ops = &adin1110_netdev_ops;
netdev->ethtool_ops = &adin1110_ethtool_ops;
netdev->priv_flags |= IFF_UNICAST_FLT;
- netdev->netns_local = true;
+ netdev->netns_immutable = true;
port_priv->phydev = get_phy_device(priv->mii_bus, i + 1, false);
if (IS_ERR(port_priv->phydev)) {
diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c
index 9ebc9788ded7..4589396ab01f 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.c
+++ b/drivers/net/ethernet/airoha/airoha_eth.c
@@ -1777,10 +1777,9 @@ static int airoha_qdma_tx_napi_poll(struct napi_struct *napi, int budget)
WRITE_ONCE(desc->msg1, 0);
if (skb) {
- u16 queue = skb_get_queue_mapping(skb);
struct netdev_queue *txq;
- txq = netdev_get_tx_queue(skb->dev, queue);
+ txq = skb_get_tx_queue(skb->dev, skb);
netdev_tx_completed_queue(txq, 1, skb->len);
if (netif_tx_queue_stopped(txq) &&
q->ndesc - q->queued >= q->free_thr)
@@ -1902,20 +1901,40 @@ static int airoha_qdma_init_tx(struct airoha_qdma *qdma)
static void airoha_qdma_cleanup_tx_queue(struct airoha_queue *q)
{
- struct airoha_eth *eth = q->qdma->eth;
+ struct airoha_qdma *qdma = q->qdma;
+ struct airoha_eth *eth = qdma->eth;
+ int qid = q - &qdma->q_tx[0];
spin_lock_bh(&q->lock);
while (q->queued) {
struct airoha_queue_entry *e = &q->entry[q->tail];
+ struct airoha_qdma_desc *desc = &q->desc[q->tail];
dma_unmap_single(eth->dev, e->dma_addr, e->dma_len,
DMA_TO_DEVICE);
dev_kfree_skb_any(e->skb);
e->skb = NULL;
+ /* Reset DMA descriptor */
+ WRITE_ONCE(desc->ctrl, 0);
+ WRITE_ONCE(desc->addr, 0);
+ WRITE_ONCE(desc->data, 0);
+ WRITE_ONCE(desc->msg0, 0);
+ WRITE_ONCE(desc->msg1, 0);
+ WRITE_ONCE(desc->msg2, 0);
+
q->tail = (q->tail + 1) % q->ndesc;
q->queued--;
}
+
+ /* Set TX_DMA_IDX to TX_CPU_IDX to notify the hw the QDMA TX ring is
+ * empty.
+ */
+ airoha_qdma_rmw(qdma, REG_TX_CPU_IDX(qid), TX_RING_CPU_IDX_MASK,
+ FIELD_PREP(TX_RING_CPU_IDX_MASK, q->head));
+ airoha_qdma_rmw(qdma, REG_TX_DMA_IDX(qid), TX_RING_DMA_IDX_MASK,
+ FIELD_PREP(TX_RING_DMA_IDX_MASK, q->head));
+
spin_unlock_bh(&q->lock);
}
@@ -2559,7 +2578,7 @@ static netdev_tx_t airoha_dev_xmit(struct sk_buff *skb,
spin_lock_bh(&q->lock);
- txq = netdev_get_tx_queue(dev, qid);
+ txq = skb_get_tx_queue(dev, skb);
if (q->queued + nr_frags > q->ndesc) {
/* not enough space in the queue */
netif_tx_stop_queue(txq);
@@ -2593,10 +2612,13 @@ static netdev_tx_t airoha_dev_xmit(struct sk_buff *skb,
WRITE_ONCE(desc->msg1, cpu_to_le32(msg1));
WRITE_ONCE(desc->msg2, cpu_to_le32(0xffff));
- e->skb = i ? NULL : skb;
+ e->skb = i == nr_frags - 1 ? skb : NULL;
e->dma_addr = addr;
e->dma_len = len;
+ if (i == nr_frags - 1)
+ continue;
+
data = skb_frag_address(frag);
len = skb_frag_size(frag);
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index ef2462e94b3f..dc2bc7a2de0e 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -924,6 +924,7 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
mapping += bp->rx_dma_offset;
rx_buf->data = page;
rx_buf->data_ptr = page_address(page) + offset + bp->rx_offset;
+ rx_buf->offset = offset;
} else {
u8 *data = __bnxt_alloc_rx_frag(bp, &mapping, rxr, gfp);
@@ -932,6 +933,7 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
rx_buf->data = data;
rx_buf->data_ptr = data + bp->rx_offset;
+ rx_buf->offset = 0;
}
rx_buf->mapping = mapping;
@@ -953,6 +955,7 @@ void bnxt_reuse_rx_data(struct bnxt_rx_ring_info *rxr, u16 cons, void *data)
prod_rx_buf->data_ptr = cons_rx_buf->data_ptr;
prod_rx_buf->mapping = cons_rx_buf->mapping;
+ prod_rx_buf->offset = cons_rx_buf->offset;
prod_bd = &rxr->rx_desc_ring[RX_RING(bp, prod)][RX_IDX(prod)];
cons_bd = &rxr->rx_desc_ring[RX_RING(bp, cons)][RX_IDX(cons)];
@@ -1090,8 +1093,11 @@ static struct sk_buff *bnxt_rx_multi_page_skb(struct bnxt *bp,
struct page *page = data;
u16 prod = rxr->rx_prod;
struct sk_buff *skb;
+ void *frag_start;
int err;
+ frag_start = page_address(page) + rxr->rx_buf_ring[cons].offset;
+
err = bnxt_alloc_rx_data(bp, rxr, prod, GFP_ATOMIC);
if (unlikely(err)) {
bnxt_reuse_rx_data(rxr, cons, data);
@@ -1100,13 +1106,13 @@ static struct sk_buff *bnxt_rx_multi_page_skb(struct bnxt *bp,
dma_addr -= bp->rx_dma_offset;
dma_sync_single_for_cpu(&bp->pdev->dev, dma_addr, rxr->rx_page_size,
bp->rx_dir);
- skb = napi_build_skb(data_ptr - bp->rx_offset, rxr->rx_page_size);
+ skb = napi_build_skb(frag_start, rxr->rx_page_size);
if (!skb) {
page_pool_recycle_direct(rxr->page_pool, page);
return NULL;
}
skb_mark_for_recycle(skb);
- skb_reserve(skb, bp->rx_offset);
+ skb_reserve(skb, data_ptr - (u8 *)frag_start);
__skb_put(skb, len);
return skb;
@@ -4305,11 +4311,14 @@ static void bnxt_init_one_rx_agg_ring_rxbd(struct bnxt *bp,
type = ((u32)rxr->rx_page_size << RX_BD_LEN_SHIFT) |
RX_BD_TYPE_RX_AGG_BD;
- /* On P7, setting EOP will cause the chip to disable
- * Relaxed Ordering (RO) for TPA data. Disable EOP for
- * potentially higher performance with RO.
+ /* Disable EOP if TPA is enabled to prevent overlapping zero
+ * padding with the next segment's data. On P7_PLUS, EOP will
+ * automatically disable Relaxed Ordering (RO) to prevent
+ * potential data corruption (and may degrade performance). On
+ * older chips, RO will not be automatically disabled and may
+ * cause corruption.
*/
- if (BNXT_CHIP_P5_AND_MINUS(bp) || !(bp->flags & BNXT_FLAG_TPA))
+ if (!(bp->flags & BNXT_FLAG_TPA))
type |= RX_BD_FLAGS_AGG_EOP;
bnxt_init_rxbd_pages(ring, type);
@@ -6674,7 +6683,8 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, struct bnxt_vnic_info *vnic)
#endif
if ((bp->flags & BNXT_FLAG_STRIP_VLAN) || def_vlan)
req->flags |= cpu_to_le32(VNIC_CFG_REQ_FLAGS_VLAN_STRIP_MODE);
- if (vnic->vnic_id == BNXT_VNIC_DEFAULT && bnxt_ulp_registered(bp->edev))
+ if (vnic->vnic_id == BNXT_VNIC_DEFAULT &&
+ bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA]))
req->flags |= cpu_to_le32(bnxt_get_roce_vnic_mode(bp));
return hwrm_req_send(bp, req);
@@ -7758,6 +7768,7 @@ static int bnxt_get_avail_msix(struct bnxt *bp, int num);
static int __bnxt_reserve_rings(struct bnxt *bp)
{
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
struct bnxt_hw_rings hwr = {0};
int rx_rings, old_rx_rings, rc;
int cp = bp->cp_nr_rings;
@@ -7768,7 +7779,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
if (!bnxt_need_reserve_rings(bp))
return 0;
- if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(bp->edev)) {
+ if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(edev)) {
ulp_msix = bnxt_get_avail_msix(bp, bp->ulp_num_msix_want);
if (!ulp_msix)
bnxt_set_ulp_stat_ctxs(bp, 0);
@@ -7821,8 +7832,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
}
rx_rings = min_t(int, rx_rings, hwr.grp);
hwr.cp = min_t(int, hwr.cp, bp->cp_nr_rings);
- if (bnxt_ulp_registered(bp->edev) &&
- hwr.stat > bnxt_get_ulp_stat_ctxs(bp))
+ if (bnxt_ulp_registered(edev) && hwr.stat > bnxt_get_ulp_stat_ctxs(bp))
hwr.stat -= bnxt_get_ulp_stat_ctxs(bp);
hwr.cp = min_t(int, hwr.cp, hwr.stat);
rc = bnxt_trim_rings(bp, &rx_rings, &hwr.tx, hwr.cp, sh);
@@ -7861,7 +7871,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
!netif_is_rxfh_configured(bp->dev))
bnxt_set_dflt_rss_indir_tbl(bp, NULL);
- if (!bnxt_ulp_registered(bp->edev) && BNXT_NEW_RM(bp)) {
+ if (!bnxt_ulp_registered(edev) && BNXT_NEW_RM(bp)) {
int resv_msix, resv_ctx, ulp_ctxs;
struct bnxt_hw_resc *hw_resc;
@@ -8379,7 +8389,7 @@ static int bnxt_hwrm_func_backing_store_qcaps_v2(struct bnxt *bp)
struct hwrm_func_backing_store_qcaps_v2_output *resp;
struct hwrm_func_backing_store_qcaps_v2_input *req;
struct bnxt_ctx_mem_info *ctx = bp->ctx;
- u16 type;
+ u16 type, next_type = 0;
int rc;
rc = hwrm_req_init(bp, req, HWRM_FUNC_BACKING_STORE_QCAPS_V2);
@@ -8395,7 +8405,7 @@ static int bnxt_hwrm_func_backing_store_qcaps_v2(struct bnxt *bp)
resp = hwrm_req_hold(bp, req);
- for (type = 0; type < BNXT_CTX_V2_MAX; ) {
+ for (type = 0; type < BNXT_CTX_V2_MAX; type = next_type) {
struct bnxt_ctx_mem_type *ctxm = &ctx->ctx_arr[type];
u8 init_val, init_off, i;
__le32 *p;
@@ -8406,11 +8416,11 @@ static int bnxt_hwrm_func_backing_store_qcaps_v2(struct bnxt *bp)
if (rc)
goto ctx_done;
flags = le32_to_cpu(resp->flags);
- type = le16_to_cpu(resp->next_valid_type);
+ next_type = le16_to_cpu(resp->next_valid_type);
if (!(flags & FUNC_BACKING_STORE_QCAPS_V2_RESP_FLAGS_TYPE_VALID))
continue;
- ctxm->type = le16_to_cpu(resp->type);
+ ctxm->type = type;
ctxm->entry_size = le16_to_cpu(resp->entry_size);
ctxm->flags = flags;
ctxm->instance_bmap = le32_to_cpu(resp->instance_bit_map);
@@ -11099,6 +11109,7 @@ static void bnxt_clear_int_mode(struct bnxt *bp)
int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init)
{
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
bool irq_cleared = false;
bool irq_change = false;
int tcs = bp->num_tc;
@@ -11108,7 +11119,7 @@ int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init)
if (!bnxt_need_reserve_rings(bp))
return 0;
- if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(bp->edev)) {
+ if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(edev)) {
int ulp_msix = bnxt_get_avail_msix(bp, bp->ulp_num_msix_want);
if (ulp_msix > bp->ulp_num_msix_want)
@@ -13997,7 +14008,7 @@ static void bnxt_fw_echo_reply(struct bnxt *bp)
static void bnxt_ulp_restart(struct bnxt *bp)
{
bnxt_ulp_stop(bp);
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
}
static void bnxt_sp_task(struct work_struct *work)
@@ -14157,7 +14168,7 @@ int bnxt_check_rings(struct bnxt *bp, int tx, int rx, bool sh, int tcs,
hwr.cp_p5 = hwr.tx + rx;
rc = bnxt_hwrm_check_rings(bp, &hwr);
if (!rc && pci_msix_can_alloc_dyn(bp->pdev)) {
- if (!bnxt_ulp_registered(bp->edev)) {
+ if (!bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA])) {
hwr.cp += bnxt_get_ulp_msix_num(bp);
hwr.cp = min_t(int, hwr.cp, bnxt_get_max_func_irqs(bp));
}
@@ -14659,7 +14670,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
bnxt_dl_health_fw_status_update(bp, true);
}
rtnl_unlock();
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
bnxt_reenable_sriov(bp);
rtnl_lock();
bnxt_vf_reps_alloc(bp);
@@ -14681,7 +14692,8 @@ static void bnxt_fw_reset_task(struct work_struct *work)
bnxt_fw_reset_abort(bp, rc);
rtnl_unlock();
ulp_start:
- bnxt_ulp_start(bp, rc);
+ if (!rc)
+ bnxt_ulp_start(bp);
}
static int bnxt_init_board(struct pci_dev *pdev, struct net_device *dev)
@@ -15654,12 +15666,13 @@ static void bnxt_remove_one(struct pci_dev *pdev)
if (BNXT_PF(bp))
bnxt_sriov_disable(bp);
- bnxt_rdma_aux_device_del(bp);
+ bnxt_aux_devices_del(bp);
unregister_netdev(dev);
bnxt_ptp_clear(bp);
- bnxt_rdma_aux_device_uninit(bp);
+ bnxt_aux_devices_uninit(bp);
+ bnxt_auxdev_id_free(bp, bp->auxdev_id);
bnxt_free_l2_filters(bp, true);
bnxt_free_ntp_fltrs(bp, true);
@@ -16283,7 +16296,9 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
__bnxt_set_rx_skb_mode(bp, false);
bnxt_set_tpa_flags(bp);
bnxt_set_ring_params(bp);
- bnxt_rdma_aux_device_init(bp);
+ mutex_init(&bp->auxdev_lock);
+ if (!bnxt_auxdev_id_alloc(bp))
+ bnxt_aux_devices_init(bp);
rc = bnxt_set_dflt_rings(bp, true);
if (rc) {
if (BNXT_VF(bp) && rc == -ENODEV) {
@@ -16345,7 +16360,7 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
bnxt_dl_fw_reporters_create(bp);
- bnxt_rdma_aux_device_add(bp);
+ bnxt_aux_devices_add(bp);
bnxt_print_device_info(bp);
@@ -16353,7 +16368,8 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
return 0;
init_err_cleanup:
- bnxt_rdma_aux_device_uninit(bp);
+ bnxt_aux_devices_uninit(bp);
+ bnxt_auxdev_id_free(bp, bp->auxdev_id);
bnxt_dl_unregister(bp);
init_err_dl:
bnxt_shutdown_tc(bp);
@@ -16481,9 +16497,10 @@ static int bnxt_resume(struct device *device)
resume_exit:
rtnl_unlock();
- bnxt_ulp_start(bp, rc);
- if (!rc)
+ if (!rc) {
+ bnxt_ulp_start(bp);
bnxt_reenable_sriov(bp);
+ }
return rc;
}
@@ -16657,9 +16674,10 @@ static void bnxt_io_resume(struct pci_dev *pdev)
netif_device_attach(netdev);
rtnl_unlock();
- bnxt_ulp_start(bp, err);
- if (!err)
+ if (!err) {
+ bnxt_ulp_start(bp);
bnxt_reenable_sriov(bp);
+ }
}
static const struct pci_error_handlers bnxt_err_handler = {
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index c10f8ad8bfbd..1c094dd2ad25 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -24,12 +24,12 @@
#include <linux/interrupt.h>
#include <linux/rhashtable.h>
#include <linux/crash_dump.h>
-#include <linux/auxiliary_bus.h>
#include <net/devlink.h>
#include <net/dst_metadata.h>
#include <net/xdp.h>
#include <linux/dim.h>
#include <linux/io-64-nonatomic-lo-hi.h>
+#include "bnxt_ulp.h"
#ifdef CONFIG_TEE_BNXT_FW
#include <linux/firmware/broadcom/tee_bnxt_fw.h>
#endif
@@ -894,6 +894,7 @@ struct bnxt_sw_rx_bd {
void *data;
u8 *data_ptr;
dma_addr_t mapping;
+ unsigned int offset;
};
struct bnxt_sw_rx_agg_bd {
@@ -2049,12 +2050,6 @@ struct bnxt_fw_health {
#define BNXT_FW_IF_RETRY 10
#define BNXT_FW_SLOT_RESET_RETRY 4
-struct bnxt_aux_priv {
- struct auxiliary_device aux_dev;
- struct bnxt_en_dev *edev;
- int id;
-};
-
enum board_idx {
BCM57301,
BCM57302,
@@ -2288,8 +2283,8 @@ struct bnxt {
#define BNXT_CHIP_P5_AND_MINUS(bp) \
(BNXT_CHIP_P3(bp) || BNXT_CHIP_P4(bp) || BNXT_CHIP_P5(bp))
- struct bnxt_aux_priv *aux_priv;
- struct bnxt_en_dev *edev;
+ struct bnxt_aux_priv *aux_priv[__BNXT_AUXDEV_MAX];
+ struct bnxt_en_dev *edev[__BNXT_AUXDEV_MAX];
struct bnxt_napi **bnapi;
@@ -2669,6 +2664,13 @@ struct bnxt {
struct bnxt_ctx_pg_info *fw_crash_mem;
u32 fw_crash_len;
+ int auxdev_id;
+ /* synchronize validity checks of available aux devices */
+ struct mutex auxdev_lock;
+ u8 auxdev_state[__BNXT_AUXDEV_MAX];
+#define BNXT_ADEV_STATE_NONE 0
+#define BNXT_ADEV_STATE_INIT 1
+#define BNXT_ADEV_STATE_ADD 2
};
#define BNXT_NUM_RX_RING_STATS 8
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c
index 901fd36757ed..06d3fd9ad7a4 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c
@@ -443,12 +443,12 @@ static int bnxt_dl_reload_down(struct devlink *dl, bool netns_change,
NL_SET_ERR_MSG_MOD(extack,
"reload is unsupported while VFs are allocated or being configured");
rtnl_unlock();
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
return -EOPNOTSUPP;
}
if (bp->dev->reg_state == NETREG_UNREGISTERED) {
rtnl_unlock();
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
return -ENODEV;
}
if (netif_running(bp->dev))
@@ -571,8 +571,8 @@ static int bnxt_dl_reload_up(struct devlink *dl, enum devlink_reload_action acti
dev_close(bp->dev);
}
rtnl_unlock();
- if (action == DEVLINK_RELOAD_ACTION_DRIVER_REINIT)
- bnxt_ulp_start(bp, rc);
+ if (!rc && action == DEVLINK_RELOAD_ACTION_DRIVER_REINIT)
+ bnxt_ulp_start(bp);
return rc;
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
index d883e2e57da0..5bedd6bd5f30 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
@@ -4871,7 +4871,7 @@ static void bnxt_self_test(struct net_device *dev, struct ethtool_test *etest,
memset(buf, 0, sizeof(u64) * bp->num_tests);
if (etest->flags & ETH_TEST_FL_OFFLINE &&
- bnxt_ulp_registered(bp->edev)) {
+ bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA])) {
etest->flags |= ETH_TEST_FL_FAILED;
netdev_warn(dev, "Offline tests cannot be run with RoCE driver loaded\n");
return;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h
index f8ef6f1a1964..5f8de1634378 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h
@@ -42,6 +42,10 @@ struct hwrm_resp_hdr {
#define TLV_TYPE_MODIFY_ROCE_CC_GEN1 0x5UL
#define TLV_TYPE_QUERY_ROCE_CC_GEN2 0x6UL
#define TLV_TYPE_MODIFY_ROCE_CC_GEN2 0x7UL
+#define TLV_TYPE_QUERY_ROCE_CC_GEN1_EXT 0x8UL
+#define TLV_TYPE_MODIFY_ROCE_CC_GEN1_EXT 0x9UL
+#define TLV_TYPE_QUERY_ROCE_CC_GEN2_EXT 0xaUL
+#define TLV_TYPE_MODIFY_ROCE_CC_GEN2_EXT 0xbUL
#define TLV_TYPE_ENGINE_CKV_ALIAS_ECC_PUBLIC_KEY 0x8001UL
#define TLV_TYPE_ENGINE_CKV_IV 0x8003UL
#define TLV_TYPE_ENGINE_CKV_AUTH_TAG 0x8004UL
@@ -509,6 +513,7 @@ struct cmd_nums {
#define HWRM_TFC_IF_TBL_GET 0x399UL
#define HWRM_TFC_TBL_SCOPE_CONFIG_GET 0x39aUL
#define HWRM_TFC_RESC_USAGE_QUERY 0x39bUL
+ #define HWRM_TFC_GLOBAL_ID_FREE 0x39cUL
#define HWRM_SV 0x400UL
#define HWRM_DBG_SERDES_TEST 0xff0eUL
#define HWRM_DBG_LOG_BUFFER_FLUSH 0xff0fUL
@@ -624,8 +629,8 @@ struct hwrm_err_output {
#define HWRM_VERSION_MAJOR 1
#define HWRM_VERSION_MINOR 10
#define HWRM_VERSION_UPDATE 3
-#define HWRM_VERSION_RSVD 68
-#define HWRM_VERSION_STR "1.10.3.68"
+#define HWRM_VERSION_RSVD 85
+#define HWRM_VERSION_STR "1.10.3.85"
/* hwrm_ver_get_input (size:192b/24B) */
struct hwrm_ver_get_input {
@@ -1302,6 +1307,43 @@ struct hwrm_async_event_cmpl_error_report {
#define ASYNC_EVENT_CMPL_ERROR_REPORT_EVENT_DATA1_ERROR_TYPE_SFT 0
};
+/* hwrm_async_event_cmpl_dbg_buf_producer (size:128b/16B) */
+struct hwrm_async_event_cmpl_dbg_buf_producer {
+ __le16 type;
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_TYPE_MASK 0x3fUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_TYPE_SFT 0
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_TYPE_HWRM_ASYNC_EVENT 0x2eUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_TYPE_LAST ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_TYPE_HWRM_ASYNC_EVENT
+ __le16 event_id;
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_ID_DBG_BUF_PRODUCER 0x4cUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_ID_LAST ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_ID_DBG_BUF_PRODUCER
+ __le32 event_data2;
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA2_CURR_OFF_MASK 0xffffffffUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA2_CURR_OFF_SFT 0
+ u8 opaque_v;
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_V 0x1UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_OPAQUE_MASK 0xfeUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_OPAQUE_SFT 1
+ u8 timestamp_lo;
+ __le16 timestamp_hi;
+ __le32 event_data1;
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_MASK 0xffffUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_SFT 0
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_SRT_TRACE 0x0UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_SRT2_TRACE 0x1UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_CRT_TRACE 0x2UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_CRT2_TRACE 0x3UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_RIGP0_TRACE 0x4UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_L2_HWRM_TRACE 0x5UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_ROCE_HWRM_TRACE 0x6UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_CA0_TRACE 0x7UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_CA1_TRACE 0x8UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_CA2_TRACE 0x9UL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_RIGP1_TRACE 0xaUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_AFM_KONG_HWRM_TRACE 0xbUL
+ #define ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_LAST ASYNC_EVENT_CMPL_DBG_BUF_PRODUCER_EVENT_DATA1_TYPE_AFM_KONG_HWRM_TRACE
+};
+
/* hwrm_async_event_cmpl_hwrm_error (size:128b/16B) */
struct hwrm_async_event_cmpl_hwrm_error {
__le16 type;
@@ -1864,7 +1906,10 @@ struct hwrm_func_qcaps_output {
__le32 roce_vf_max_gid;
__le32 flags_ext3;
#define FUNC_QCAPS_RESP_FLAGS_EXT3_RM_RSV_WHILE_ALLOC_CAP 0x1UL
- u8 unused_3[7];
+ #define FUNC_QCAPS_RESP_FLAGS_EXT3_REQUIRE_L2_FILTER 0x2UL
+ #define FUNC_QCAPS_RESP_FLAGS_EXT3_MAX_ROCE_VFS_SUPPORTED 0x4UL
+ __le16 max_roce_vfs;
+ u8 unused_3[5];
u8 valid;
};
@@ -2253,17 +2298,18 @@ struct hwrm_func_cfg_input {
#define FUNC_CFG_REQ_FLAGS2_KTLS_KEY_CTX_ASSETS_TEST 0x1UL
#define FUNC_CFG_REQ_FLAGS2_QUIC_KEY_CTX_ASSETS_TEST 0x2UL
__le32 enables2;
- #define FUNC_CFG_REQ_ENABLES2_KDNET 0x1UL
- #define FUNC_CFG_REQ_ENABLES2_DB_PAGE_SIZE 0x2UL
- #define FUNC_CFG_REQ_ENABLES2_QUIC_TX_KEY_CTXS 0x4UL
- #define FUNC_CFG_REQ_ENABLES2_QUIC_RX_KEY_CTXS 0x8UL
- #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_AV_PER_VF 0x10UL
- #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_CQ_PER_VF 0x20UL
- #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_MRW_PER_VF 0x40UL
- #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_QP_PER_VF 0x80UL
- #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_SRQ_PER_VF 0x100UL
- #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_GID_PER_VF 0x200UL
- #define FUNC_CFG_REQ_ENABLES2_XID_PARTITION_CFG 0x400UL
+ #define FUNC_CFG_REQ_ENABLES2_KDNET 0x1UL
+ #define FUNC_CFG_REQ_ENABLES2_DB_PAGE_SIZE 0x2UL
+ #define FUNC_CFG_REQ_ENABLES2_QUIC_TX_KEY_CTXS 0x4UL
+ #define FUNC_CFG_REQ_ENABLES2_QUIC_RX_KEY_CTXS 0x8UL
+ #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_AV_PER_VF 0x10UL
+ #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_CQ_PER_VF 0x20UL
+ #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_MRW_PER_VF 0x40UL
+ #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_QP_PER_VF 0x80UL
+ #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_SRQ_PER_VF 0x100UL
+ #define FUNC_CFG_REQ_ENABLES2_ROCE_MAX_GID_PER_VF 0x200UL
+ #define FUNC_CFG_REQ_ENABLES2_XID_PARTITION_CFG 0x400UL
+ #define FUNC_CFG_REQ_ENABLES2_PHYSICAL_SLOT_NUMBER 0x800UL
u8 port_kdnet_mode;
#define FUNC_CFG_REQ_PORT_KDNET_MODE_DISABLED 0x0UL
#define FUNC_CFG_REQ_PORT_KDNET_MODE_ENABLED 0x1UL
@@ -2281,7 +2327,7 @@ struct hwrm_func_cfg_input {
#define FUNC_CFG_REQ_DB_PAGE_SIZE_2MB 0x9UL
#define FUNC_CFG_REQ_DB_PAGE_SIZE_4MB 0xaUL
#define FUNC_CFG_REQ_DB_PAGE_SIZE_LAST FUNC_CFG_REQ_DB_PAGE_SIZE_4MB
- u8 unused_1[2];
+ __le16 physical_slot_number;
__le32 num_ktls_tx_key_ctxs;
__le32 num_ktls_rx_key_ctxs;
__le32 num_quic_tx_key_ctxs;
@@ -3683,7 +3729,7 @@ struct hwrm_func_ptp_ext_qcfg_output {
u8 valid;
};
-/* hwrm_func_backing_store_cfg_v2_input (size:448b/56B) */
+/* hwrm_func_backing_store_cfg_v2_input (size:512b/64B) */
struct hwrm_func_backing_store_cfg_v2_input {
__le16 req_type;
__le16 cmpl_ring;
@@ -3721,6 +3767,7 @@ struct hwrm_func_backing_store_cfg_v2_input {
#define FUNC_BACKING_STORE_CFG_V2_REQ_TYPE_CA1_TRACE 0x27UL
#define FUNC_BACKING_STORE_CFG_V2_REQ_TYPE_CA2_TRACE 0x28UL
#define FUNC_BACKING_STORE_CFG_V2_REQ_TYPE_RIGP1_TRACE 0x29UL
+ #define FUNC_BACKING_STORE_CFG_V2_REQ_TYPE_AFM_KONG_HWRM_TRACE 0x2aUL
#define FUNC_BACKING_STORE_CFG_V2_REQ_TYPE_INVALID 0xffffUL
#define FUNC_BACKING_STORE_CFG_V2_REQ_TYPE_LAST FUNC_BACKING_STORE_CFG_V2_REQ_TYPE_INVALID
__le16 instance;
@@ -3752,6 +3799,9 @@ struct hwrm_func_backing_store_cfg_v2_input {
__le32 split_entry_1;
__le32 split_entry_2;
__le32 split_entry_3;
+ __le32 enables;
+ #define FUNC_BACKING_STORE_CFG_V2_REQ_ENABLES_NEXT_BS_OFFSET 0x1UL
+ __le32 next_bs_offset;
};
/* hwrm_func_backing_store_cfg_v2_output (size:128b/16B) */
@@ -3802,6 +3852,7 @@ struct hwrm_func_backing_store_qcfg_v2_input {
#define FUNC_BACKING_STORE_QCFG_V2_REQ_TYPE_CA1_TRACE 0x27UL
#define FUNC_BACKING_STORE_QCFG_V2_REQ_TYPE_CA2_TRACE 0x28UL
#define FUNC_BACKING_STORE_QCFG_V2_REQ_TYPE_RIGP1_TRACE 0x29UL
+ #define FUNC_BACKING_STORE_QCFG_V2_REQ_TYPE_AFM_KONG_HWRM_TRACE 0x2aUL
#define FUNC_BACKING_STORE_QCFG_V2_REQ_TYPE_INVALID 0xffffUL
#define FUNC_BACKING_STORE_QCFG_V2_REQ_TYPE_LAST FUNC_BACKING_STORE_QCFG_V2_REQ_TYPE_INVALID
__le16 instance;
@@ -3963,6 +4014,7 @@ struct hwrm_func_backing_store_qcaps_v2_input {
#define FUNC_BACKING_STORE_QCAPS_V2_REQ_TYPE_CA1_TRACE 0x27UL
#define FUNC_BACKING_STORE_QCAPS_V2_REQ_TYPE_CA2_TRACE 0x28UL
#define FUNC_BACKING_STORE_QCAPS_V2_REQ_TYPE_RIGP1_TRACE 0x29UL
+ #define FUNC_BACKING_STORE_QCAPS_V2_REQ_TYPE_AFM_KONG_HWRM_TRACE 0x2aUL
#define FUNC_BACKING_STORE_QCAPS_V2_REQ_TYPE_INVALID 0xffffUL
#define FUNC_BACKING_STORE_QCAPS_V2_REQ_TYPE_LAST FUNC_BACKING_STORE_QCAPS_V2_REQ_TYPE_INVALID
u8 rsvd[6];
@@ -4005,6 +4057,7 @@ struct hwrm_func_backing_store_qcaps_v2_output {
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_TYPE_CA1_TRACE 0x27UL
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_TYPE_CA2_TRACE 0x28UL
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_TYPE_RIGP1_TRACE 0x29UL
+ #define FUNC_BACKING_STORE_QCAPS_V2_RESP_TYPE_AFM_KONG_HWRM_TRACE 0x2aUL
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_TYPE_INVALID 0xffffUL
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_TYPE_LAST FUNC_BACKING_STORE_QCAPS_V2_RESP_TYPE_INVALID
__le16 entry_size;
@@ -4014,6 +4067,8 @@ struct hwrm_func_backing_store_qcaps_v2_output {
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_FLAGS_DRIVER_MANAGED_MEMORY 0x4UL
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_FLAGS_ROCE_QP_PSEUDO_STATIC_ALLOC 0x8UL
#define FUNC_BACKING_STORE_QCAPS_V2_RESP_FLAGS_FW_DBG_TRACE 0x10UL
+ #define FUNC_BACKING_STORE_QCAPS_V2_RESP_FLAGS_FW_BIN_DBG_TRACE 0x20UL
+ #define FUNC_BACKING_STORE_QCAPS_V2_RESP_FLAGS_NEXT_BS_OFFSET 0x40UL
__le32 instance_bit_map;
u8 ctx_init_value;
u8 ctx_init_offset;
@@ -4034,7 +4089,8 @@ struct hwrm_func_backing_store_qcaps_v2_output {
__le32 split_entry_1;
__le32 split_entry_2;
__le32 split_entry_3;
- u8 rsvd3[3];
+ __le16 max_instance_count;
+ u8 rsvd3;
u8 valid;
};
@@ -4535,11 +4591,12 @@ struct hwrm_port_phy_qcfg_output {
#define PORT_PHY_QCFG_RESP_PHY_TYPE_800G_BASEDR8 0x3dUL
#define PORT_PHY_QCFG_RESP_PHY_TYPE_LAST PORT_PHY_QCFG_RESP_PHY_TYPE_800G_BASEDR8
u8 media_type;
- #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_UNKNOWN 0x0UL
- #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_TP 0x1UL
- #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_DAC 0x2UL
- #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_FIBRE 0x3UL
- #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_LAST PORT_PHY_QCFG_RESP_MEDIA_TYPE_FIBRE
+ #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_UNKNOWN 0x0UL
+ #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_TP 0x1UL
+ #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_DAC 0x2UL
+ #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_FIBRE 0x3UL
+ #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_BACKPLANE 0x4UL
+ #define PORT_PHY_QCFG_RESP_MEDIA_TYPE_LAST PORT_PHY_QCFG_RESP_MEDIA_TYPE_BACKPLANE
u8 xcvr_pkg_type;
#define PORT_PHY_QCFG_RESP_XCVR_PKG_TYPE_XCVR_INTERNAL 0x1UL
#define PORT_PHY_QCFG_RESP_XCVR_PKG_TYPE_XCVR_EXTERNAL 0x2UL
@@ -4654,7 +4711,8 @@ struct hwrm_port_phy_qcfg_output {
#define PORT_PHY_QCFG_RESP_LINK_PARTNER_PAM4_ADV_SPEEDS_100GB 0x2UL
#define PORT_PHY_QCFG_RESP_LINK_PARTNER_PAM4_ADV_SPEEDS_200GB 0x4UL
u8 link_down_reason;
- #define PORT_PHY_QCFG_RESP_LINK_DOWN_REASON_RF 0x1UL
+ #define PORT_PHY_QCFG_RESP_LINK_DOWN_REASON_RF 0x1UL
+ #define PORT_PHY_QCFG_RESP_LINK_DOWN_REASON_OTP_SPEED_VIOLATION 0x2UL
__le16 support_speeds2;
#define PORT_PHY_QCFG_RESP_SUPPORT_SPEEDS2_1GB 0x1UL
#define PORT_PHY_QCFG_RESP_SUPPORT_SPEEDS2_10GB 0x2UL
@@ -9241,20 +9299,22 @@ struct hwrm_fw_set_time_output {
/* hwrm_struct_hdr (size:128b/16B) */
struct hwrm_struct_hdr {
__le16 struct_id;
- #define STRUCT_HDR_STRUCT_ID_LLDP_CFG 0x41bUL
- #define STRUCT_HDR_STRUCT_ID_DCBX_ETS 0x41dUL
- #define STRUCT_HDR_STRUCT_ID_DCBX_PFC 0x41fUL
- #define STRUCT_HDR_STRUCT_ID_DCBX_APP 0x421UL
- #define STRUCT_HDR_STRUCT_ID_DCBX_FEATURE_STATE 0x422UL
- #define STRUCT_HDR_STRUCT_ID_LLDP_GENERIC 0x424UL
- #define STRUCT_HDR_STRUCT_ID_LLDP_DEVICE 0x426UL
- #define STRUCT_HDR_STRUCT_ID_POWER_BKUP 0x427UL
- #define STRUCT_HDR_STRUCT_ID_PEER_MMAP 0x429UL
- #define STRUCT_HDR_STRUCT_ID_AFM_OPAQUE 0x1UL
- #define STRUCT_HDR_STRUCT_ID_PORT_DESCRIPTION 0xaUL
- #define STRUCT_HDR_STRUCT_ID_RSS_V2 0x64UL
- #define STRUCT_HDR_STRUCT_ID_MSIX_PER_VF 0xc8UL
- #define STRUCT_HDR_STRUCT_ID_LAST STRUCT_HDR_STRUCT_ID_MSIX_PER_VF
+ #define STRUCT_HDR_STRUCT_ID_LLDP_CFG 0x41bUL
+ #define STRUCT_HDR_STRUCT_ID_DCBX_ETS 0x41dUL
+ #define STRUCT_HDR_STRUCT_ID_DCBX_PFC 0x41fUL
+ #define STRUCT_HDR_STRUCT_ID_DCBX_APP 0x421UL
+ #define STRUCT_HDR_STRUCT_ID_DCBX_FEATURE_STATE 0x422UL
+ #define STRUCT_HDR_STRUCT_ID_LLDP_GENERIC 0x424UL
+ #define STRUCT_HDR_STRUCT_ID_LLDP_DEVICE 0x426UL
+ #define STRUCT_HDR_STRUCT_ID_POWER_BKUP 0x427UL
+ #define STRUCT_HDR_STRUCT_ID_PEER_MMAP 0x429UL
+ #define STRUCT_HDR_STRUCT_ID_AFM_OPAQUE 0x1UL
+ #define STRUCT_HDR_STRUCT_ID_PORT_DESCRIPTION 0xaUL
+ #define STRUCT_HDR_STRUCT_ID_RSS_V2 0x64UL
+ #define STRUCT_HDR_STRUCT_ID_MSIX_PER_VF 0xc8UL
+ #define STRUCT_HDR_STRUCT_ID_UDCC_RTT_BUCKET_COUNT 0x12cUL
+ #define STRUCT_HDR_STRUCT_ID_UDCC_RTT_BUCKET_BOUND 0x12dUL
+ #define STRUCT_HDR_STRUCT_ID_LAST STRUCT_HDR_STRUCT_ID_UDCC_RTT_BUCKET_BOUND
__le16 len;
u8 version;
u8 count;
@@ -9756,6 +9816,7 @@ struct hwrm_dbg_qcaps_output {
#define DBG_QCAPS_RESP_FLAGS_COREDUMP_HOST_DDR 0x10UL
#define DBG_QCAPS_RESP_FLAGS_COREDUMP_HOST_CAPTURE 0x20UL
#define DBG_QCAPS_RESP_FLAGS_PTRACE 0x40UL
+ #define DBG_QCAPS_RESP_FLAGS_REG_ACCESS_RESTRICTED 0x80UL
u8 unused_1[3];
u8 valid;
};
@@ -9996,6 +10057,43 @@ struct hwrm_dbg_ring_info_get_output {
u8 valid;
};
+/* hwrm_dbg_log_buffer_flush_input (size:192b/24B) */
+struct hwrm_dbg_log_buffer_flush_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+ __le16 type;
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_SRT_TRACE 0x0UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_SRT2_TRACE 0x1UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_CRT_TRACE 0x2UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_CRT2_TRACE 0x3UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_RIGP0_TRACE 0x4UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_L2_HWRM_TRACE 0x5UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_ROCE_HWRM_TRACE 0x6UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_CA0_TRACE 0x7UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_CA1_TRACE 0x8UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_CA2_TRACE 0x9UL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_RIGP1_TRACE 0xaUL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_AFM_KONG_HWRM_TRACE 0xbUL
+ #define DBG_LOG_BUFFER_FLUSH_REQ_TYPE_LAST DBG_LOG_BUFFER_FLUSH_REQ_TYPE_AFM_KONG_HWRM_TRACE
+ u8 unused_1[2];
+ __le32 flags;
+ #define DBG_LOG_BUFFER_FLUSH_REQ_FLAGS_FLUSH_ALL_BUFFERS 0x1UL
+};
+
+/* hwrm_dbg_log_buffer_flush_output (size:128b/16B) */
+struct hwrm_dbg_log_buffer_flush_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+ __le32 current_buffer_offset;
+ u8 unused_1[3];
+ u8 valid;
+};
+
/* hwrm_nvm_read_input (size:320b/40B) */
struct hwrm_nvm_read_input {
__le16 req_type;
@@ -10080,6 +10178,7 @@ struct hwrm_nvm_write_input {
#define NVM_WRITE_REQ_FLAGS_KEEP_ORIG_ACTIVE_IMG 0x1UL
#define NVM_WRITE_REQ_FLAGS_BATCH_MODE 0x2UL
#define NVM_WRITE_REQ_FLAGS_BATCH_LAST 0x4UL
+ #define NVM_WRITE_REQ_FLAGS_SKIP_CRID_CHECK 0x8UL
__le32 dir_item_length;
__le32 offset;
__le32 len;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
index 1867552a8bdb..d806724a45f6 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
@@ -28,9 +28,32 @@
static DEFINE_IDA(bnxt_aux_dev_ids);
+struct bnxt_aux_device {
+ const char *name;
+};
+
+static void bnxt_auxdev_set_state(struct bnxt *bp, int idx, int state)
+{
+ bp->auxdev_state[idx] = state;
+}
+
+static bool bnxt_auxdev_is_init(struct bnxt *bp, int idx)
+{
+ return (bp->auxdev_state[idx] == BNXT_ADEV_STATE_INIT);
+}
+
+static bool bnxt_auxdev_is_active(struct bnxt *bp, int idx)
+{
+ return (bp->auxdev_state[idx] == BNXT_ADEV_STATE_ADD);
+}
+
+static struct bnxt_aux_device bnxt_aux_devices[__BNXT_AUXDEV_MAX] = {{
+ .name = "rdma",
+}};
+
static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent)
{
- struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
int num_msix, i;
if (!edev->ulp_tbl->msix_requested) {
@@ -50,55 +73,69 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent)
int bnxt_get_ulp_msix_num(struct bnxt *bp)
{
- if (bp->edev)
- return bp->edev->ulp_num_msix_vec;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ return edev->ulp_num_msix_vec;
return 0;
}
void bnxt_set_ulp_msix_num(struct bnxt *bp, int num)
{
- if (bp->edev)
- bp->edev->ulp_num_msix_vec = num;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ edev->ulp_num_msix_vec = num;
}
int bnxt_get_ulp_msix_num_in_use(struct bnxt *bp)
{
- if (bnxt_ulp_registered(bp->edev))
- return bp->edev->ulp_num_msix_vec;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (bnxt_ulp_registered(edev))
+ return edev->ulp_num_msix_vec;
return 0;
}
int bnxt_get_ulp_stat_ctxs(struct bnxt *bp)
{
- if (bp->edev)
- return bp->edev->ulp_num_ctxs;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ return edev->ulp_num_ctxs;
return 0;
}
void bnxt_set_ulp_stat_ctxs(struct bnxt *bp, int num_ulp_ctx)
{
- if (bp->edev)
- bp->edev->ulp_num_ctxs = num_ulp_ctx;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ edev->ulp_num_ctxs = num_ulp_ctx;
}
int bnxt_get_ulp_stat_ctxs_in_use(struct bnxt *bp)
{
- if (bnxt_ulp_registered(bp->edev))
- return bp->edev->ulp_num_ctxs;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (bnxt_ulp_registered(edev))
+ return edev->ulp_num_ctxs;
return 0;
}
void bnxt_set_dflt_ulp_stat_ctxs(struct bnxt *bp)
{
- if (bp->edev) {
- bp->edev->ulp_num_ctxs = BNXT_MIN_ROCE_STAT_CTXS;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev) {
+ edev->ulp_num_ctxs = BNXT_MIN_ROCE_STAT_CTXS;
/* Reserve one additional stat_ctx for PF0 (except
* on 1-port NICs) as it also creates one stat_ctx
* for PF1 in case of RoCE bonding.
*/
if (BNXT_PF(bp) && !bp->pf.port_id &&
bp->port_count > 1)
- bp->edev->ulp_num_ctxs++;
+ edev->ulp_num_ctxs++;
}
}
@@ -134,7 +171,7 @@ int bnxt_register_dev(struct bnxt_en_dev *edev,
edev->ulp_tbl->msix_requested = bnxt_get_ulp_msix_num(bp);
- bnxt_fill_msix_vecs(bp, bp->edev->msix_entries);
+ bnxt_fill_msix_vecs(bp, edev->msix_entries);
edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED;
exit:
mutex_unlock(&edev->en_dev_lock);
@@ -223,20 +260,27 @@ EXPORT_SYMBOL(bnxt_send_msg);
void bnxt_ulp_stop(struct bnxt *bp)
{
- struct bnxt_aux_priv *aux_priv = bp->aux_priv;
- struct bnxt_en_dev *edev = bp->edev;
+ int i;
- if (!edev)
- return;
-
- mutex_lock(&edev->en_dev_lock);
- if (!bnxt_ulp_registered(edev) ||
- (edev->flags & BNXT_EN_FLAG_ULP_STOPPED))
- goto ulp_stop_exit;
-
- edev->flags |= BNXT_EN_FLAG_ULP_STOPPED;
- if (aux_priv) {
+ mutex_lock(&bp->auxdev_lock);
+ for (i = 0; i < __BNXT_AUXDEV_MAX; i++) {
+ struct bnxt_aux_priv *aux_priv;
struct auxiliary_device *adev;
+ struct bnxt_en_dev *edev;
+
+ if (!bnxt_auxdev_is_active(bp, i))
+ continue;
+
+ aux_priv = bp->aux_priv[i];
+ edev = bp->edev[i];
+ mutex_lock(&edev->en_dev_lock);
+ if (!bnxt_ulp_registered(edev) ||
+ (edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) {
+ mutex_unlock(&edev->en_dev_lock);
+ continue;
+ }
+
+ edev->flags |= BNXT_EN_FLAG_ULP_STOPPED;
adev = &aux_priv->aux_dev;
if (adev->dev.driver) {
@@ -247,29 +291,35 @@ void bnxt_ulp_stop(struct bnxt *bp)
edev->en_state = bp->state;
adrv->suspend(adev, pm);
}
+ mutex_unlock(&edev->en_dev_lock);
}
-ulp_stop_exit:
- mutex_unlock(&edev->en_dev_lock);
+ mutex_unlock(&bp->auxdev_lock);
}
-void bnxt_ulp_start(struct bnxt *bp, int err)
+void bnxt_ulp_start(struct bnxt *bp)
{
- struct bnxt_aux_priv *aux_priv = bp->aux_priv;
- struct bnxt_en_dev *edev = bp->edev;
+ int i;
- if (!edev || err)
- return;
+ mutex_lock(&bp->auxdev_lock);
+ for (i = 0; i < __BNXT_AUXDEV_MAX; i++) {
+ struct bnxt_aux_priv *aux_priv;
+ struct auxiliary_device *adev;
+ struct bnxt_en_dev *edev;
- mutex_lock(&edev->en_dev_lock);
- if (!bnxt_ulp_registered(edev) ||
- !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED))
- goto ulp_start_exit;
+ if (!bnxt_auxdev_is_active(bp, i))
+ continue;
+
+ aux_priv = bp->aux_priv[i];
+ edev = bp->edev[i];
+ mutex_lock(&edev->en_dev_lock);
+ if (!bnxt_ulp_registered(edev) ||
+ !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) {
+ goto clear_flag_continue;
+ }
- if (edev->ulp_tbl->msix_requested)
- bnxt_fill_msix_vecs(bp, edev->msix_entries);
+ if (edev->ulp_tbl->msix_requested)
+ bnxt_fill_msix_vecs(bp, edev->msix_entries);
- if (aux_priv) {
- struct auxiliary_device *adev;
adev = &aux_priv->aux_dev;
if (adev->dev.driver) {
@@ -279,21 +329,22 @@ void bnxt_ulp_start(struct bnxt *bp, int err)
edev->en_state = bp->state;
adrv->resume(adev);
}
+clear_flag_continue:
+ edev->flags &= ~BNXT_EN_FLAG_ULP_STOPPED;
+ mutex_unlock(&edev->en_dev_lock);
}
-ulp_start_exit:
- edev->flags &= ~BNXT_EN_FLAG_ULP_STOPPED;
- mutex_unlock(&edev->en_dev_lock);
+ mutex_unlock(&bp->auxdev_lock);
}
void bnxt_ulp_irq_stop(struct bnxt *bp)
{
- struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
struct bnxt_ulp_ops *ops;
if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED))
return;
- if (bnxt_ulp_registered(bp->edev)) {
+ if (bnxt_ulp_registered(edev)) {
struct bnxt_ulp *ulp = edev->ulp_tbl;
if (!ulp->msix_requested)
@@ -308,13 +359,13 @@ void bnxt_ulp_irq_stop(struct bnxt *bp)
void bnxt_ulp_irq_restart(struct bnxt *bp, int err)
{
- struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
struct bnxt_ulp_ops *ops;
if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED))
return;
- if (bnxt_ulp_registered(bp->edev)) {
+ if (bnxt_ulp_registered(edev)) {
struct bnxt_ulp *ulp = edev->ulp_tbl;
struct bnxt_msix_entry *ent = NULL;
@@ -355,18 +406,21 @@ int bnxt_register_async_events(struct bnxt_en_dev *edev,
}
EXPORT_SYMBOL(bnxt_register_async_events);
-void bnxt_rdma_aux_device_uninit(struct bnxt *bp)
+void bnxt_aux_devices_uninit(struct bnxt *bp)
{
struct bnxt_aux_priv *aux_priv;
struct auxiliary_device *adev;
-
- /* Skip if no auxiliary device init was done. */
- if (!bp->aux_priv)
- return;
-
- aux_priv = bp->aux_priv;
- adev = &aux_priv->aux_dev;
- auxiliary_device_uninit(adev);
+ int idx;
+
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ if (bnxt_auxdev_is_init(bp, idx)) {
+ aux_priv = bp->aux_priv[idx];
+ adev = &aux_priv->aux_dev;
+ auxiliary_device_uninit(adev);
+ }
+ }
+ mutex_unlock(&bp->auxdev_lock);
}
static void bnxt_aux_dev_release(struct device *dev)
@@ -375,20 +429,25 @@ static void bnxt_aux_dev_release(struct device *dev)
container_of(dev, struct bnxt_aux_priv, aux_dev.dev);
struct bnxt *bp = netdev_priv(aux_priv->edev->net);
- ida_free(&bnxt_aux_dev_ids, aux_priv->id);
kfree(aux_priv->edev->ulp_tbl);
- bp->edev = NULL;
+ bp->edev[aux_priv->id] = NULL;
kfree(aux_priv->edev);
+ bp->aux_priv[aux_priv->id] = NULL;
kfree(aux_priv);
- bp->aux_priv = NULL;
}
-void bnxt_rdma_aux_device_del(struct bnxt *bp)
+void bnxt_aux_devices_del(struct bnxt *bp)
{
- if (!bp->edev)
- return;
+ int idx;
- auxiliary_device_delete(&bp->aux_priv->aux_dev);
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ if (bnxt_auxdev_is_active(bp, idx)) {
+ auxiliary_device_delete(&bp->aux_priv[idx]->aux_dev);
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_INIT);
+ }
+ }
+ mutex_unlock(&bp->auxdev_lock);
}
static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp)
@@ -414,83 +473,107 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp)
edev->bar0 = bp->bar0;
}
-void bnxt_rdma_aux_device_add(struct bnxt *bp)
+void bnxt_aux_devices_add(struct bnxt *bp)
{
struct auxiliary_device *aux_dev;
- int rc;
-
- if (!bp->edev)
- return;
-
- aux_dev = &bp->aux_priv->aux_dev;
- rc = auxiliary_device_add(aux_dev);
- if (rc) {
- netdev_warn(bp->dev, "Failed to add auxiliary device for ROCE\n");
- auxiliary_device_uninit(aux_dev);
- bp->flags &= ~BNXT_FLAG_ROCE_CAP;
+ int rc, idx;
+
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ if (bnxt_auxdev_is_init(bp, idx)) {
+ aux_dev = &bp->aux_priv[idx]->aux_dev;
+ rc = auxiliary_device_add(aux_dev);
+ if (rc) {
+ netdev_warn(bp->dev, "Failed to add auxiliary device for ROCE\n");
+ auxiliary_device_uninit(aux_dev);
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_NONE);
+ if (idx == BNXT_AUXDEV_RDMA)
+ bp->flags &= ~BNXT_FLAG_ROCE_CAP;
+ continue;
+ }
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_ADD);
+ }
}
+ mutex_unlock(&bp->auxdev_lock);
}
-void bnxt_rdma_aux_device_init(struct bnxt *bp)
+void bnxt_aux_devices_init(struct bnxt *bp)
{
struct auxiliary_device *aux_dev;
struct bnxt_aux_priv *aux_priv;
struct bnxt_en_dev *edev;
struct bnxt_ulp *ulp;
- int rc;
-
- if (!(bp->flags & BNXT_FLAG_ROCE_CAP))
- return;
-
- aux_priv = kzalloc(sizeof(*bp->aux_priv), GFP_KERNEL);
- if (!aux_priv)
- goto exit;
-
- aux_priv->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL);
- if (aux_priv->id < 0) {
- netdev_warn(bp->dev,
- "ida alloc failed for ROCE auxiliary device\n");
- kfree(aux_priv);
- goto exit;
- }
-
- aux_dev = &aux_priv->aux_dev;
- aux_dev->id = aux_priv->id;
- aux_dev->name = "rdma";
- aux_dev->dev.parent = &bp->pdev->dev;
- aux_dev->dev.release = bnxt_aux_dev_release;
-
- rc = auxiliary_device_init(aux_dev);
- if (rc) {
- ida_free(&bnxt_aux_dev_ids, aux_priv->id);
+ int rc, idx;
+
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_NONE);
+
+ if (idx == BNXT_AUXDEV_RDMA &&
+ !(bp->flags & BNXT_FLAG_ROCE_CAP))
+ continue;
+
+ aux_priv = kzalloc_obj(*aux_priv);
+ if (!aux_priv)
+ goto next_auxdev;
+
+ edev = kzalloc_obj(*edev);
+ if (!edev)
+ goto aux_priv_free;
+ aux_priv->edev = edev;
+ bnxt_set_edev_info(edev, bp);
+
+ ulp = kzalloc_obj(*ulp);
+ if (!ulp)
+ goto edev_free;
+ edev->ulp_tbl = ulp;
+ aux_priv->id = idx;
+
+ aux_dev = &aux_priv->aux_dev;
+ aux_dev->id = bp->auxdev_id;
+ aux_dev->name = bnxt_aux_devices[idx].name;
+ aux_dev->dev.parent = &bp->pdev->dev;
+ aux_dev->dev.release = bnxt_aux_dev_release;
+
+ rc = auxiliary_device_init(aux_dev);
+ if (rc)
+ goto ulp_free;
+ bp->aux_priv[idx] = aux_priv;
+
+ /* From this point, all cleanup will happen via the .release
+ * callback & any error unwinding will need to include a call
+ * to auxiliary_device_uninit.
+ */
+ bp->edev[idx] = edev;
+ if (idx == BNXT_AUXDEV_RDMA)
+ bp->ulp_num_msix_want = bnxt_set_dflt_ulp_msix(bp);
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_INIT);
+
+ continue;
+ulp_free:
+ kfree(ulp);
+edev_free:
+ kfree(edev);
+aux_priv_free:
kfree(aux_priv);
- goto exit;
+next_auxdev:
+ if (idx == BNXT_AUXDEV_RDMA)
+ bp->flags &= ~BNXT_FLAG_ROCE_CAP;
}
- bp->aux_priv = aux_priv;
-
- /* From this point, all cleanup will happen via the .release callback &
- * any error unwinding will need to include a call to
- * auxiliary_device_uninit.
- */
- edev = kzalloc(sizeof(*edev), GFP_KERNEL);
- if (!edev)
- goto aux_dev_uninit;
-
- aux_priv->edev = edev;
-
- ulp = kzalloc(sizeof(*ulp), GFP_KERNEL);
- if (!ulp)
- goto aux_dev_uninit;
+ mutex_unlock(&bp->auxdev_lock);
+}
- edev->ulp_tbl = ulp;
- bp->edev = edev;
- bnxt_set_edev_info(edev, bp);
- bp->ulp_num_msix_want = bnxt_set_dflt_ulp_msix(bp);
+int bnxt_auxdev_id_alloc(struct bnxt *bp)
+{
+ bp->auxdev_id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL);
+ if (bp->auxdev_id < 0)
+ return bp->auxdev_id;
- return;
+ return 0;
+}
-aux_dev_uninit:
- auxiliary_device_uninit(aux_dev);
-exit:
- bp->flags &= ~BNXT_FLAG_ROCE_CAP;
+void bnxt_auxdev_id_free(struct bnxt *bp, int id)
+{
+ if (bp->auxdev_id >= 0)
+ ida_free(&bnxt_aux_dev_ids, id);
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h
index b76a231ca7da..6e1f26c8c057 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h
@@ -10,6 +10,8 @@
#ifndef BNXT_ULP_H
#define BNXT_ULP_H
+#include <linux/auxiliary_bus.h>
+
#define BNXT_ROCE_ULP 0
#define BNXT_MAX_ULP 1
@@ -23,6 +25,17 @@
struct hwrm_async_event_cmpl;
struct bnxt;
+enum bnxt_auxdev_type {
+ BNXT_AUXDEV_RDMA = 0,
+ __BNXT_AUXDEV_MAX
+};
+
+struct bnxt_aux_priv {
+ struct auxiliary_device aux_dev;
+ struct bnxt_en_dev *edev;
+ int id;
+};
+
struct bnxt_msix_entry {
u32 vector;
u32 ring_idx;
@@ -109,19 +122,21 @@ void bnxt_set_ulp_stat_ctxs(struct bnxt *bp, int num_ctxs);
int bnxt_get_ulp_stat_ctxs_in_use(struct bnxt *bp);
void bnxt_set_dflt_ulp_stat_ctxs(struct bnxt *bp);
void bnxt_ulp_stop(struct bnxt *bp);
-void bnxt_ulp_start(struct bnxt *bp, int err);
+void bnxt_ulp_start(struct bnxt *bp);
void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs);
void bnxt_ulp_irq_stop(struct bnxt *bp);
void bnxt_ulp_irq_restart(struct bnxt *bp, int err);
void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl);
-void bnxt_rdma_aux_device_uninit(struct bnxt *bp);
-void bnxt_rdma_aux_device_del(struct bnxt *bp);
-void bnxt_rdma_aux_device_add(struct bnxt *bp);
-void bnxt_rdma_aux_device_init(struct bnxt *bp);
+void bnxt_aux_devices_uninit(struct bnxt *bp);
+void bnxt_aux_devices_del(struct bnxt *bp);
+void bnxt_aux_devices_add(struct bnxt *bp);
+void bnxt_aux_devices_init(struct bnxt *bp);
int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops,
void *handle);
void bnxt_unregister_dev(struct bnxt_en_dev *edev);
int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg);
int bnxt_register_async_events(struct bnxt_en_dev *edev,
unsigned long *events_bmap, u16 max_id);
+int bnxt_auxdev_id_alloc(struct bnxt *bp);
+void bnxt_auxdev_id_free(struct bnxt *bp, int id);
#endif
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_main.c b/drivers/net/ethernet/cavium/liquidio/lio_main.c
index ebb82767b6e5..630cacd338fd 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_main.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_main.c
@@ -3789,9 +3789,7 @@ static int setup_nic_devices(struct octeon_device *octeon_dev)
static int octeon_enable_sriov(struct octeon_device *oct)
{
unsigned int num_vfs_alloced = oct->sriov_info.num_vfs_alloced;
- struct pci_dev *vfdev;
int err;
- u32 u;
if (OCTEON_CN23XX_PF(oct) && num_vfs_alloced) {
err = pci_enable_sriov(oct->pci_dev,
@@ -3804,23 +3802,6 @@ static int octeon_enable_sriov(struct octeon_device *oct)
return err;
}
oct->sriov_info.sriov_enabled = 1;
-
- /* init lookup table that maps DPI ring number to VF pci_dev
- * struct pointer
- */
- u = 0;
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, NULL);
- while (vfdev) {
- if (vfdev->is_virtfn &&
- (vfdev->physfn == oct->pci_dev)) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] =
- vfdev;
- u += oct->sriov_info.rings_per_vf;
- }
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, vfdev);
- }
}
return num_vfs_alloced;
@@ -3828,8 +3809,6 @@ static int octeon_enable_sriov(struct octeon_device *oct)
static int lio_pci_sriov_disable(struct octeon_device *oct)
{
- int u;
-
if (pci_vfs_assigned(oct->pci_dev)) {
dev_err(&oct->pci_dev->dev, "VFs are still assigned to VMs.\n");
return -EPERM;
@@ -3837,12 +3816,6 @@ static int lio_pci_sriov_disable(struct octeon_device *oct)
pci_disable_sriov(oct->pci_dev);
- u = 0;
- while (u < MAX_POSSIBLE_VFS) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] = NULL;
- u += oct->sriov_info.rings_per_vf;
- }
-
oct->sriov_info.num_vfs_alloced = 0;
dev_info(&oct->pci_dev->dev, "oct->pf_num:%d disabled VFs\n",
oct->pf_num);
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_device.h b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
index d26364c2ac81..555c7ef4c04a 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_device.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
@@ -390,9 +390,6 @@ struct octeon_sriov_info {
struct lio_trusted_vf trusted_vf;
- /*lookup table that maps DPI ring number to VF pci_dev struct pointer*/
- struct pci_dev *dpiring_to_vfpcidev_lut[MAX_POSSIBLE_VFS];
-
u64 vf_macaddr[MAX_POSSIBLE_VFS];
u16 vf_vlantci[MAX_POSSIBLE_VFS];
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
index ad685f5d0a13..697fcdc41e3c 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
@@ -26,6 +26,31 @@
#include "octeon_mailbox.h"
#include "cn23xx_pf_device.h"
+static struct pci_dev *lio_vf_pci_dev_by_qno(struct octeon_device *oct, u32 q_no)
+{
+ struct pci_dev *vfdev = NULL;
+ int vfidx;
+
+ if (!oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ if (q_no % oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ vfidx = q_no / oct->sriov_info.rings_per_vf;
+ if (vfidx >= oct->sriov_info.num_vfs_alloced)
+ return NULL;
+
+ while ((vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
+ OCTEON_CN23XX_VF_VID, vfdev))) {
+ if (pci_physfn(vfdev) && pci_physfn(vfdev) == oct->pci_dev &&
+ pci_iov_vf_id(vfdev) == vfidx)
+ return vfdev;
+ }
+
+ return NULL;
+}
+
/**
* octeon_mbox_read:
* @mbox: Pointer mailbox
@@ -237,6 +262,7 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox,
struct octeon_mbox_cmd *mbox_cmd)
{
struct octeon_device *oct = mbox->oct_dev;
+ struct pci_dev *vfdev;
switch (mbox_cmd->msg.s.cmd) {
case OCTEON_VF_ACTIVE:
@@ -260,7 +286,12 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox,
dev_info(&oct->pci_dev->dev,
"got a request for FLR from VF that owns DPI ring %u\n",
mbox->q_no);
- pcie_flr(oct->sriov_info.dpiring_to_vfpcidev_lut[mbox->q_no]);
+ vfdev = lio_vf_pci_dev_by_qno(oct, mbox->q_no);
+ if (!vfdev)
+ break;
+
+ pcie_flr(vfdev);
+ pci_dev_put(vfdev);
break;
case OCTEON_PF_CHANGED_VF_MACADDR:
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c
index 6d69c19ea7c3..5f9eb030aec9 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c
@@ -505,7 +505,7 @@ static void reset_listen_child(struct sock *child)
chtls_send_reset(child, CPL_ABORT_SEND_RST, skb);
sock_orphan(child);
- INC_ORPHAN_COUNT(child);
+ tcp_orphan_count_inc();
if (child->sk_state == TCP_CLOSE)
inet_csk_destroy_sock(child);
}
@@ -870,7 +870,7 @@ static void do_abort_syn_rcv(struct sock *child, struct sock *parent)
* created only after 3 way handshake is done.
*/
sock_orphan(child);
- INC_ORPHAN_COUNT(child);
+ tcp_orphan_count_inc();
chtls_release_resources(child);
chtls_conn_done(child);
} else {
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.h b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.h
index f61ca657601c..e949e4e58066 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.h
+++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.h
@@ -95,7 +95,6 @@ struct deferred_skb_cb {
#define WSCALE_OK(tp) ((tp)->rx_opt.wscale_ok)
#define TSTAMP_OK(tp) ((tp)->rx_opt.tstamp_ok)
#define SACK_OK(tp) ((tp)->rx_opt.sack_ok)
-#define INC_ORPHAN_COUNT(sk) this_cpu_inc(*(sk)->sk_prot->orphan_count)
/* TLS SKB */
#define skb_ulp_tls_inline(skb) (ULP_SKB_CB(skb)->ulp.tls.ofld)
diff --git a/drivers/net/ethernet/ibm/emac/core.c b/drivers/net/ethernet/ibm/emac/core.c
index a7aef2fb668d..e88117add912 100644
--- a/drivers/net/ethernet/ibm/emac/core.c
+++ b/drivers/net/ethernet/ibm/emac/core.c
@@ -3030,15 +3030,14 @@ static int emac_probe(struct platform_device *ofdev)
if (err)
goto err_gone;
- /* Get interrupts. EMAC irq is mandatory */
- dev->emac_irq = irq_of_parse_and_map(np, 0);
- if (!dev->emac_irq) {
- printk(KERN_ERR "%pOF: Can't map main interrupt\n", np);
- err = -ENODEV;
+ dev->emacp = devm_platform_ioremap_resource(ofdev, 0);
+ if (IS_ERR(dev->emacp)) {
+ err = PTR_ERR(dev->emacp);
goto err_gone;
}
/* Setup error IRQ handler */
+ dev->emac_irq = platform_get_irq(ofdev, 0);
err = devm_request_irq(&ofdev->dev, dev->emac_irq, emac_irq, 0, "EMAC",
dev);
if (err) {
@@ -3049,15 +3048,6 @@ static int emac_probe(struct platform_device *ofdev)
ndev->irq = dev->emac_irq;
- /* Map EMAC regs */
- // TODO : platform_get_resource() and devm_ioremap_resource()
- dev->emacp = devm_of_iomap(&ofdev->dev, np, 0, NULL);
- if (!dev->emacp) {
- dev_err(&ofdev->dev, "can't map device registers");
- err = -ENOMEM;
- goto err_gone;
- }
-
/* Wait for dependent devices */
err = emac_wait_deps(dev);
if (err)
diff --git a/drivers/net/ethernet/intel/ice/devlink/devlink_port.c b/drivers/net/ethernet/intel/ice/devlink/devlink_port.c
index c6779d9dffff..a5859fcc10e6 100644
--- a/drivers/net/ethernet/intel/ice/devlink/devlink_port.c
+++ b/drivers/net/ethernet/intel/ice/devlink/devlink_port.c
@@ -588,8 +588,8 @@ static void ice_dealloc_dynamic_port(struct ice_dynamic_port *dyn_port)
xa_erase(&pf->sf_nums, devlink_port->attrs.pci_sf.sf);
ice_eswitch_detach_sf(pf, dyn_port);
- ice_vsi_free(dyn_port->vsi);
xa_erase(&pf->dyn_ports, dyn_port->vsi->idx);
+ ice_vsi_free(dyn_port->vsi);
kfree(dyn_port);
}
diff --git a/drivers/net/ethernet/intel/ice/ice_base.c b/drivers/net/ethernet/intel/ice/ice_base.c
index 98c3764fed39..8d7a2f44469b 100644
--- a/drivers/net/ethernet/intel/ice/ice_base.c
+++ b/drivers/net/ethernet/intel/ice/ice_base.c
@@ -546,13 +546,16 @@ static int ice_vsi_cfg_rxq(struct ice_rx_ring *ring)
err = __xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,
ring->q_index,
ring->q_vector->napi.napi_id,
- ring->rx_buf_len);
+ ice_get_frame_sz(ring));
if (err)
return err;
}
ice_rx_xsk_pool(ring);
if (ring->xsk_pool) {
+ u32 frag_size =
+ xsk_pool_get_rx_frag_step(ring->xsk_pool);
+
xdp_rxq_info_unreg(&ring->xdp_rxq);
ring->rx_buf_len =
@@ -560,7 +563,7 @@ static int ice_vsi_cfg_rxq(struct ice_rx_ring *ring)
err = __xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,
ring->q_index,
ring->q_vector->napi.napi_id,
- ring->rx_buf_len);
+ frag_size);
if (err)
return err;
err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq,
@@ -578,7 +581,7 @@ static int ice_vsi_cfg_rxq(struct ice_rx_ring *ring)
err = __xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev,
ring->q_index,
ring->q_vector->napi.napi_id,
- ring->rx_buf_len);
+ ice_get_frame_sz(ring));
if (err)
return err;
}
diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.c b/drivers/net/ethernet/intel/ice/ice_ptp.c
index ca0d4ba2b63b..f0dab91ce831 100644
--- a/drivers/net/ethernet/intel/ice/ice_ptp.c
+++ b/drivers/net/ethernet/intel/ice/ice_ptp.c
@@ -2936,7 +2936,13 @@ static int ice_ptp_setup_pf(struct ice_pf *pf)
struct ice_ptp *ctrl_ptp = ice_get_ctrl_ptp(pf);
struct ice_ptp *ptp = &pf->ptp;
- if (WARN_ON(!ctrl_ptp) || ice_get_phy_model(&pf->hw) == ICE_PHY_UNSUP)
+ if (!ctrl_ptp) {
+ dev_info(ice_pf_to_dev(pf),
+ "PTP unavailable: no controlling PF\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (ice_get_phy_model(&pf->hw) == ICE_PHY_UNSUP)
return -ENODEV;
INIT_LIST_HEAD(&ptp->port.list_node);
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c
index 0bb9f0dbf535..16b540258576 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ -2344,6 +2344,9 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
ice_trace(xmit_frame_ring, tx_ring, skb);
+ /* record the location of the first descriptor for this packet */
+ first = &tx_ring->tx_buf[tx_ring->next_to_use];
+
if (unlikely(ipv6_hopopt_jumbo_remove(skb)))
goto out_drop;
@@ -2372,8 +2375,6 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
offload.tx_ring = tx_ring;
- /* record the location of the first descriptor for this packet */
- first = &tx_ring->tx_buf[tx_ring->next_to_use];
first->skb = skb;
first->type = ICE_TX_BUF_SKB;
first->bytecount = max_t(unsigned int, skb->len, ETH_ZLEN);
@@ -2437,6 +2438,7 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
out_drop:
ice_trace(xmit_frame_ring_drop, tx_ring, skb);
dev_kfree_skb_any(skb);
+ first->type = ICE_TX_BUF_EMPTY;
return NETDEV_TX_OK;
}
diff --git a/drivers/net/ethernet/intel/idpf/idpf_dev.c b/drivers/net/ethernet/intel/idpf/idpf_dev.c
index 45272bd9e472..ceea5cb6b854 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_dev.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_dev.c
@@ -84,7 +84,7 @@ static int idpf_intr_reg_init(struct idpf_vport *vport,
if (!reg_vals)
return -ENOMEM;
- num_regs = idpf_get_reg_intr_vecs(vport, reg_vals);
+ num_regs = idpf_get_reg_intr_vecs(vport, reg_vals, total_vecs);
if (num_regs < num_vecs) {
err = -EINVAL;
goto free_reg_vals;
diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ethernet/intel/idpf/idpf_lib.c
index 36777ca10f32..b7da39ba0616 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_lib.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c
@@ -1624,10 +1624,9 @@ void idpf_init_task(struct work_struct *work)
set_bit(IDPF_VPORT_REG_NETDEV, vport_config->flags);
}
- /* As all the required vports are created, clear the reset flag
- * unconditionally here in case we were in reset and the link was down.
- */
+ /* Clear the reset and load bits as all vports are created */
clear_bit(IDPF_HR_RESET_IN_PROG, adapter->flags);
+ clear_bit(IDPF_HR_DRV_LOAD, adapter->flags);
/* Start the statistics task now */
queue_delayed_work(adapter->stats_wq, &adapter->stats_task,
msecs_to_jiffies(10 * (pdev->devfn & 0x07)));
@@ -1641,6 +1640,14 @@ void idpf_init_task(struct work_struct *work)
idpf_vport_dealloc(adapter->vports[index]);
}
}
+ /* Cleanup after vc_core_init, which has no way of knowing the
+ * init task failed on driver load.
+ */
+ if (test_and_clear_bit(IDPF_HR_DRV_LOAD, adapter->flags)) {
+ cancel_delayed_work_sync(&adapter->serv_task);
+ cancel_delayed_work_sync(&adapter->mbx_task);
+ }
+
clear_bit(IDPF_HR_RESET_IN_PROG, adapter->flags);
}
@@ -1790,7 +1797,7 @@ static void idpf_init_hard_reset(struct idpf_adapter *adapter)
dev_info(dev, "Device HW Reset initiated\n");
/* Prepare for reset */
- if (test_and_clear_bit(IDPF_HR_DRV_LOAD, adapter->flags)) {
+ if (test_bit(IDPF_HR_DRV_LOAD, adapter->flags)) {
reg_ops->trigger_reset(adapter, IDPF_HR_DRV_LOAD);
} else if (test_and_clear_bit(IDPF_HR_FUNC_RESET, adapter->flags)) {
bool is_reset = idpf_is_reset_detected(adapter);
diff --git a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c b/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
index 61d6f774e2f6..84dc8b72c64b 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
@@ -84,7 +84,7 @@ static int idpf_vf_intr_reg_init(struct idpf_vport *vport,
if (!reg_vals)
return -ENOMEM;
- num_regs = idpf_get_reg_intr_vecs(vport, reg_vals);
+ num_regs = idpf_get_reg_intr_vecs(vport, reg_vals, total_vecs);
if (num_regs < num_vecs) {
err = -EINVAL;
goto free_reg_vals;
diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
index 4e49e4673dd6..a4c04a7bac97 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
@@ -1004,11 +1004,12 @@ static void idpf_init_avail_queues(struct idpf_adapter *adapter)
* idpf_get_reg_intr_vecs - Get vector queue register offset
* @vport: virtual port structure
* @reg_vals: Register offsets to store in
+ * @num_vecs: number of entries the @reg_vals array can hold
*
* Returns number of registers that got populated
*/
int idpf_get_reg_intr_vecs(struct idpf_vport *vport,
- struct idpf_vec_regs *reg_vals)
+ struct idpf_vec_regs *reg_vals, int num_vecs)
{
struct virtchnl2_vector_chunks *chunks;
struct idpf_vec_regs reg_val;
@@ -1032,7 +1033,7 @@ int idpf_get_reg_intr_vecs(struct idpf_vport *vport,
dynctl_reg_spacing = le32_to_cpu(chunk->dynctl_reg_spacing);
itrn_reg_spacing = le32_to_cpu(chunk->itrn_reg_spacing);
- for (i = 0; i < num_vec; i++) {
+ for (i = 0; i < num_vec && num_regs < num_vecs; i++) {
reg_vals[num_regs].dyn_ctl_reg = reg_val.dyn_ctl_reg;
reg_vals[num_regs].itrn_reg = reg_val.itrn_reg;
reg_vals[num_regs].itrn_index_spacing =
@@ -2999,7 +3000,6 @@ int idpf_vc_core_init(struct idpf_adapter *adapter)
pci_sriov_set_totalvfs(adapter->pdev, idpf_get_max_vfs(adapter));
num_max_vports = idpf_get_max_vports(adapter);
- adapter->max_vports = num_max_vports;
adapter->vports = kcalloc(num_max_vports, sizeof(*adapter->vports),
GFP_KERNEL);
if (!adapter->vports)
@@ -3022,6 +3022,12 @@ int idpf_vc_core_init(struct idpf_adapter *adapter)
goto err_netdev_alloc;
}
+ /* Set max_vports only after vports, netdevs and vport_config buffers
+ * are allocated to make sure max_vport bound loops don't end up
+ * crashing, following allocation errors on init.
+ */
+ adapter->max_vports = num_max_vports;
+
/* Start the mailbox task before requesting vectors. This will ensure
* vector information response from mailbox is handled
*/
diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
index 0bfe44fa0385..83068c65f951 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
+++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
@@ -17,7 +17,7 @@ int idpf_vc_core_init(struct idpf_adapter *adapter);
void idpf_vc_core_deinit(struct idpf_adapter *adapter);
int idpf_get_reg_intr_vecs(struct idpf_vport *vport,
- struct idpf_vec_regs *reg_vals);
+ struct idpf_vec_regs *reg_vals, int num_vecs);
int idpf_queue_reg_init(struct idpf_vport *vport);
int idpf_vport_queue_ids_init(struct idpf_vport *vport);
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c
index e4a4d930dd22..ae5aebfd6be1 100644
--- a/drivers/net/ethernet/intel/igc/igc_main.c
+++ b/drivers/net/ethernet/intel/igc/igc_main.c
@@ -3002,7 +3002,8 @@ static void igc_xdp_xmit_zc(struct igc_ring *ring)
olinfo_status = xdp_desc.len << IGC_ADVTXD_PAYLEN_SHIFT;
dma = xsk_buff_raw_get_dma(pool, xdp_desc.addr);
- meta = xsk_buff_get_metadata(pool, xdp_desc.addr);
+ meta = xsk_buff_get_metadata(pool, xdp_desc.addr,
+ xdp_desc.options);
xsk_buff_raw_dma_sync_for_device(pool, dma, xdp_desc.len);
bi = &ring->tx_buffer_info[ntu];
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 8b8404d8c946..aef70cf119bb 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -3543,7 +3543,8 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter,
}
/* initialize XPS */
- if (!test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) {
+ if (!ring_is_xdp(ring) &&
+ !test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) {
struct ixgbe_q_vector *q_vector = ring->q_vector;
if (q_vector)
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
index d415d5dbc110..b62ed5303278 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
@@ -518,6 +518,18 @@ int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind)
return 0;
}
+int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind)
+{
+ struct cgx *cgx = cgxd;
+
+ if (!is_lmac_valid(cgx, lmac_id))
+ return -ENODEV;
+
+ *pkind = cgx_read(cgx, lmac_id, cgx->mac_ops->rxid_map_offset);
+ *pkind = *pkind & 0x3F;
+ return 0;
+}
+
static u8 cgx_get_lmac_type(void *cgxd, int lmac_id)
{
struct cgx *cgx = cgxd;
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
index 1cf12e5c7da8..e8f010ecb284 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
@@ -141,6 +141,7 @@ int cgx_get_cgxid(void *cgxd);
int cgx_get_lmac_cnt(void *cgxd);
void *cgx_get_pdata(int cgx_id);
int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind);
+int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind);
int cgx_lmac_evh_register(struct cgx_event_cb *cb, void *cgxd, int lmac_id);
int cgx_lmac_evh_unregister(void *cgxd, int lmac_id);
int cgx_get_tx_stats(void *cgxd, int lmac_id, int idx, u64 *tx_stat);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
index 34b2a16e8844..88cf7978e0d7 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
@@ -989,6 +989,8 @@ void npc_read_mcam_entry(struct rvu *rvu, struct npc_mcam *mcam,
int blkaddr, u16 src, struct mcam_entry *entry,
u8 *intf, u8 *ena);
bool is_cgx_config_permitted(struct rvu *rvu, u16 pcifunc);
+bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind);
+bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc);
bool is_mac_feature_supported(struct rvu *rvu, int pf, int feature);
u32 rvu_cgx_get_fifolen(struct rvu *rvu);
void *rvu_first_cgx_pdata(struct rvu *rvu);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
index d0215f5e9a06..c227eb7ad5e8 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
@@ -1351,3 +1351,82 @@ void rvu_mac_reset(struct rvu *rvu, u16 pcifunc)
if (mac_ops->mac_reset(cgxd, lmac, !is_vf(pcifunc)))
dev_err(rvu->dev, "Failed to reset MAC\n");
}
+
+/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds
+ * (HiGig, EDSA, etc.) are in use on the shared LMAC. VFs must not program
+ * NPC_TX_DEF_PKIND on NIX_AF_LFX_TX_PARSE_CFG in that case: the PF owns
+ * parse mode and no separate NPC_TX_HIGIG_PKIND is installed on the VF LF.
+ * TX-parse callers skip the write when denied; rvu_lf_reset() clears each LF
+ * before alloc so the next permitted owner programs NPC_TX_DEF_PKIND.
+ */
+bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc)
+{
+ int pf, err, rxpkind;
+ u8 cgx_id, lmac_id;
+ void *cgxd;
+
+ pf = rvu_get_pf(pcifunc);
+
+ if (!(pcifunc & RVU_PFVF_FUNC_MASK))
+ return true;
+
+ if (!is_pf_cgxmapped(rvu, pf))
+ return true;
+
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ err = cgx_get_pkind(cgxd, lmac_id, &rxpkind);
+ if (err)
+ return false;
+
+ switch (rxpkind) {
+ case NPC_RX_HIGIG_PKIND:
+ case NPC_RX_EDSA_PKIND:
+ return false;
+ default:
+ return true;
+ }
+}
+
+/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds
+ * (HiGig, EDSA, etc.) are in use on the shared LMAC.
+ */
+bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind)
+{
+ int pf, err, rxpkind;
+ u8 cgx_id, lmac_id;
+ struct cgx *cgxd;
+
+ pf = rvu_get_pf(pcifunc);
+
+ if (!is_pf_cgxmapped(rvu, pf))
+ return false;
+
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+
+ mutex_lock(&cgxd->lock);
+ if (!is_vf(pcifunc))
+ goto set;
+
+ err = cgx_get_pkind(cgxd, lmac_id, &rxpkind);
+ if (err)
+ goto err;
+
+ switch (rxpkind) {
+ case NPC_RX_HIGIG_PKIND:
+ case NPC_RX_EDSA_PKIND:
+ goto err;
+ default:
+ break;
+ }
+
+set:
+ cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind);
+ mutex_unlock(&cgxd->lock);
+ return true;
+
+err:
+ mutex_unlock(&cgxd->lock);
+ return false;
+}
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
index 74be2012ec04..e9bcfc730f45 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
@@ -358,8 +358,8 @@ static int nix_interface_init(struct rvu *rvu, u16 pcifunc, int type, int nixlf,
pfvf->tx_chan_cnt = 1;
rsp->tx_link = cgx_id * hw->lmac_per_cgx + lmac_id;
- cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind);
- rvu_npc_set_pkind(rvu, pkind, pfvf);
+ if (rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, pkind))
+ rvu_npc_set_pkind(rvu, pkind, pfvf);
break;
case NIX_INTF_TYPE_LBK:
@@ -1444,11 +1444,13 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu,
struct nix_lf_alloc_req *req,
struct nix_lf_alloc_rsp *rsp)
{
- int nixlf, qints, hwctx_size, intf, err, rc = 0;
+ int nixlf, qints, hwctx_size, intf, err, rc = 0, pf;
struct rvu_hwinfo *hw = rvu->hw;
u16 pcifunc = req->hdr.pcifunc;
+ u8 cgx_id = 0, lmac_id = 0;
struct rvu_block *block;
struct rvu_pfvf *pfvf;
+ struct cgx *cgxd;
u64 cfg, ctx_cfg;
int blkaddr;
@@ -1616,8 +1618,20 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu,
rvu_write64(rvu, blkaddr, NIX_AF_LFX_RX_CFG(nixlf), req->rx_cfg);
/* Configure pkind for TX parse config */
+
+ pf = rvu_get_pf(pcifunc);
cfg = NPC_TX_DEF_PKIND;
- rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+
+ if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) {
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ mutex_lock(&cgxd->lock);
+ if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc))
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+ mutex_unlock(&cgxd->lock);
+ } else {
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+ }
intf = is_lbk_vf(rvu, pcifunc) ? NIX_INTF_TYPE_LBK : NIX_INTF_TYPE_CGX;
if (is_sdp_pfvf(pcifunc))
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
index 5c5376eace10..ba233b075744 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
@@ -16,6 +16,7 @@
#include "cgx.h"
#include "npc_profile.h"
#include "rvu_npc_hash.h"
+#include "lmac_common.h"
#define RSVD_MCAM_ENTRIES_PER_PF 3 /* Broadcast, Promisc and AllMulticast */
#define RSVD_MCAM_ENTRIES_PER_NIXLF 1 /* Ucast for LFs */
@@ -3330,10 +3331,11 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
{
struct rvu_pfvf *pfvf = rvu_get_pfvf(rvu, pcifunc);
- int blkaddr, nixlf, rc, intf_mode;
int pf = rvu_get_pf(pcifunc);
+ int blkaddr, nixlf, rc, intf_mode;
+ u8 cgx_id = 0, lmac_id = 0;
u64 rxpkind, txpkind;
- u8 cgx_id, lmac_id;
+ struct cgx *cgxd;
/* use default pkind to disable edsa/higig */
rxpkind = rvu_npc_get_pkind(rvu, pf);
@@ -3357,12 +3359,8 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
/* rx pkind set req valid only for cgx mapped PFs */
if (!is_cgx_config_permitted(rvu, pcifunc))
return 0;
- rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
-
- rc = cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id,
- rxpkind);
- if (rc)
- return rc;
+ if (!rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, rxpkind))
+ return -EINVAL;
}
if (dir & PKIND_TX) {
@@ -3371,8 +3369,19 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
if (rc)
return rc;
- rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
- txpkind);
+ if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) {
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id,
+ &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ mutex_lock(&cgxd->lock);
+ if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc))
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
+ txpkind);
+ mutex_unlock(&cgxd->lock);
+ } else {
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
+ txpkind);
+ }
}
pfvf->intf_mode = intf_mode;
diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
index e26c830ae1ff..37d36cedd157 100644
--- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
+++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
@@ -685,16 +685,16 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
snprintf(netdev->name, sizeof(netdev->name), "lbk%d", n);
}
+ err = otx2_wq_init(vf);
+ if (err)
+ goto err_ptp_destroy;
+
err = register_netdev(netdev);
if (err) {
dev_err(dev, "Failed to register netdevice\n");
- goto err_ptp_destroy;
+ goto err_wq_destroy;
}
- err = otx2_wq_init(vf);
- if (err)
- goto err_unreg_netdev;
-
otx2vf_set_ethtool_ops(netdev);
err = otx2vf_mcam_flow_init(vf);
@@ -722,6 +722,10 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
otx2_shutdown_tc(vf);
err_unreg_netdev:
unregister_netdev(netdev);
+err_wq_destroy:
+ cancel_work_sync(&vf->reset_task);
+ cancel_work_sync(&vf->rx_mode_work);
+ destroy_workqueue(vf->otx2_wq);
err_ptp_destroy:
otx2_ptp_destroy(vf);
err_detach_rsrc:
@@ -769,11 +773,13 @@ static void otx2vf_remove(struct pci_dev *pdev)
}
#endif
- cancel_work_sync(&vf->reset_task);
otx2_unregister_dl(vf);
unregister_netdev(netdev);
- if (vf->otx2_wq)
+ if (vf->otx2_wq) {
+ cancel_work_sync(&vf->reset_task);
+ cancel_work_sync(&vf->rx_mode_work);
destroy_workqueue(vf->otx2_wq);
+ }
otx2_ptp_destroy(vf);
otx2_mcam_flow_del(vf);
otx2_shutdown_tc(vf);
diff --git a/drivers/net/ethernet/marvell/prestera/prestera_main.c b/drivers/net/ethernet/marvell/prestera/prestera_main.c
index 3f7cc04a873b..ebd42fc263fc 100644
--- a/drivers/net/ethernet/marvell/prestera/prestera_main.c
+++ b/drivers/net/ethernet/marvell/prestera/prestera_main.c
@@ -634,7 +634,7 @@ static int prestera_port_create(struct prestera_switch *sw, u32 id)
goto err_dl_port_register;
dev->features |= NETIF_F_HW_TC;
- dev->netns_local = true;
+ dev->netns_immutable = true;
dev->netdev_ops = &prestera_netdev_ops;
dev->ethtool_ops = &prestera_ethtool_ops;
SET_NETDEV_DEV(dev, sw->dev->dev);
diff --git a/drivers/net/ethernet/mellanox/mlx4/srq.c b/drivers/net/ethernet/mellanox/mlx4/srq.c
index dd890f5d7b72..8711689120f3 100644
--- a/drivers/net/ethernet/mellanox/mlx4/srq.c
+++ b/drivers/net/ethernet/mellanox/mlx4/srq.c
@@ -44,13 +44,14 @@ void mlx4_srq_event(struct mlx4_dev *dev, u32 srqn, int event_type)
{
struct mlx4_srq_table *srq_table = &mlx4_priv(dev)->srq_table;
struct mlx4_srq *srq;
+ unsigned long flags;
- rcu_read_lock();
+ spin_lock_irqsave(&srq_table->lock, flags);
srq = radix_tree_lookup(&srq_table->tree, srqn & (dev->caps.num_srqs - 1));
- rcu_read_unlock();
- if (srq)
- refcount_inc(&srq->refcount);
- else {
+ if (!srq || !refcount_inc_not_zero(&srq->refcount))
+ srq = NULL;
+ spin_unlock_irqrestore(&srq_table->lock, flags);
+ if (!srq) {
mlx4_warn(dev, "Async event for bogus SRQ %08x\n", srqn);
return;
}
@@ -203,8 +204,8 @@ int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, u32 cqn, u16 xrcd,
if (err)
goto err_radix;
- refcount_set(&srq->refcount, 1);
init_completion(&srq->free);
+ refcount_set_release(&srq->refcount, 1);
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cq.c b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
index 4caa1b6f40ba..24dd6318964a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/cq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
@@ -66,8 +66,8 @@ void mlx5_cq_tasklet_cb(struct tasklet_struct *t)
tasklet_schedule(&ctx->task);
}
-static void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq,
- struct mlx5_eqe *eqe)
+void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq,
+ struct mlx5_eqe *eqe)
{
unsigned long flags;
struct mlx5_eq_tasklet *tasklet_ctx = cq->tasklet_ctx.priv;
@@ -84,7 +84,15 @@ static void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq,
}
spin_unlock_irqrestore(&tasklet_ctx->lock, flags);
}
+EXPORT_SYMBOL(mlx5_add_cq_to_tasklet);
+static void mlx5_core_cq_dummy_cb(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe)
+{
+ mlx5_core_err(cq->eq->core.dev,
+ "CQ default completion callback, CQ #%u\n", cq->cqn);
+}
+
+#define MLX5_CQ_INIT_CMD_SN cpu_to_be32(2 << 28)
/* Callers must verify outbox status in case of err */
int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
u32 *in, int inlen, u32 *out, int outlen)
@@ -110,10 +118,19 @@ int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
cq->arm_sn = 0;
cq->eq = eq;
cq->uid = MLX5_GET(create_cq_in, in, uid);
+
+ /* Kernel CQs must set the arm_db address prior to calling
+ * this function, allowing for the proper value to be
+ * initialized. User CQs are responsible for their own
+ * initialization since they do not use the arm_db field.
+ */
+ if (cq->arm_db)
+ *cq->arm_db = MLX5_CQ_INIT_CMD_SN;
+
refcount_set(&cq->refcount, 1);
init_completion(&cq->free);
if (!cq->comp)
- cq->comp = mlx5_add_cq_to_tasklet;
+ cq->comp = mlx5_core_cq_dummy_cb;
/* assuming CQ will be deleted before the EQ */
cq->tasklet_ctx.priv = &eq->tasklet_ctx;
INIT_LIST_HEAD(&cq->tasklet_ctx.list);
@@ -134,7 +151,6 @@ int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
mlx5_core_dbg(dev, "failed adding CP 0x%x to debug file system\n",
cq->cqn);
- cq->uar = dev->priv.uar;
cq->irqn = eq->core.irqn;
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 0e1890616831..d9541cfd6515 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -341,6 +341,7 @@ struct mlx5e_cq {
/* data path - accessed per napi poll */
u16 event_ctr;
struct napi_struct *napi;
+ struct mlx5_uars_page *uar;
struct mlx5_core_cq mcq;
struct mlx5e_ch_stats *ch_stats;
@@ -946,6 +947,18 @@ struct mlx5e_priv {
struct mlx5_devcom_comp_dev *devcom;
};
+static inline u16 mlx5e_stats_nch_read(const struct mlx5e_priv *priv)
+{
+ /* Pairs with smp_store_release in mlx5e_stats_nch_write(). */
+ return smp_load_acquire(&priv->stats_nch);
+}
+
+static inline void mlx5e_stats_nch_write(struct mlx5e_priv *priv, u16 n)
+{
+ /* Pairs with smp_load_acquire in mlx5e_stats_nch_read(). */
+ smp_store_release(&priv->stats_nch, n);
+}
+
struct mlx5e_dev {
struct net_device *netdev;
struct devlink_port dl_port;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c
index 1856d211339f..adfb49ff42aa 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c
@@ -33,9 +33,10 @@ mlx5e_hv_vhca_fill_ring_stats(struct mlx5e_priv *priv, int ch,
static void mlx5e_hv_vhca_fill_stats(struct mlx5e_priv *priv, void *data,
int buf_len)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
int ch, i = 0;
- for (ch = 0; ch < priv->stats_nch; ch++) {
+ for (ch = 0; ch < nch; ch++) {
void *buf = data + i;
if (WARN_ON_ONCE(buf +
@@ -50,8 +51,9 @@ static void mlx5e_hv_vhca_fill_stats(struct mlx5e_priv *priv, void *data,
static int mlx5e_hv_vhca_stats_buf_size(struct mlx5e_priv *priv)
{
- return (sizeof(struct mlx5e_hv_vhca_per_ring_stats) *
- priv->stats_nch);
+ u16 nch = mlx5e_stats_nch_read(priv);
+
+ return sizeof(struct mlx5e_hv_vhca_per_ring_stats) * nch;
}
static int mlx5e_hv_vhca_stats_buf_max_size(struct mlx5e_priv *priv)
@@ -106,7 +108,7 @@ static void mlx5e_hv_vhca_stats_control(struct mlx5_hv_vhca_agent *agent,
sagent = &priv->stats_agent;
block->version = MLX5_HV_VHCA_STATS_VERSION;
- block->rings = priv->stats_nch;
+ block->rings = mlx5e_stats_nch_read(priv);
if (!block->command) {
cancel_delayed_work_sync(&priv->stats_agent.work);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
index 3dac708c0d75..121c836acdf3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
@@ -825,7 +825,7 @@ static void mlx5e_build_common_cq_param(struct mlx5_core_dev *mdev,
{
void *cqc = param->cqc;
- MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
+ MLX5_SET(cqc, cqc, uar_page, mdev->priv.bfreg.up->index);
if (MLX5_CAP_GEN(mdev, cqe_128_always) && cache_line_size() >= 128)
MLX5_SET(cqc, cqc, cqe_sz, CQE_STRIDE_128_PAD);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
index afd654583b6b..ff7074310b6c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
@@ -333,7 +333,7 @@ static int mlx5e_ptp_alloc_txqsq(struct mlx5e_ptp *c, int txq_ix,
sq->mdev = mdev;
sq->ch_ix = MLX5E_PTP_CHANNEL_IX;
sq->txq_ix = txq_ix;
- sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map;
+ sq->uar_map = mdev->priv.bfreg.map;
sq->min_inline_mode = params->tx_min_inline_mode;
sq->hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu);
sq->stats = &c->priv->ptp_stats.sq[tc];
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
index 5ec468268d1a..ee1267e0023a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h
@@ -296,10 +296,7 @@ mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc, void __iomem *uar_map,
static inline void mlx5e_cq_arm(struct mlx5e_cq *cq)
{
- struct mlx5_core_cq *mcq;
-
- mcq = &cq->mcq;
- mlx5_cq_arm(mcq, MLX5_CQ_DB_REQ_NOT, mcq->uar->map, cq->wq.cc);
+ mlx5_cq_arm(&cq->mcq, MLX5_CQ_DB_REQ_NOT, cq->uar->map, cq->wq.cc);
}
static inline struct mlx5e_sq_dma *
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
index a59199ed590d..0cc1d40c40b7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
@@ -97,7 +97,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)
xdptxd.dma_addr = xsk_buff_raw_get_dma(pool, desc.addr);
xdptxd.data = xsk_buff_raw_get_data(pool, desc.addr);
xdptxd.len = desc.len;
- meta = xsk_buff_get_metadata(pool, desc.addr);
+ meta = xsk_buff_get_metadata(pool, desc.addr, desc.options);
xsk_buff_raw_dma_sync_for_device(pool, xdptxd.dma_addr, xdptxd.len);
@@ -112,7 +112,7 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)
} else {
mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo, xdpi);
if (xp_tx_metadata_enabled(sq->xsk_pool)) {
- struct xsk_tx_metadata_compl compl;
+ struct xsk_tx_metadata_compl compl = {};
xsk_tx_metadata_to_compl(meta, &compl);
XSK_TX_COMPL_FITS(void *);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
index 6ed3a32b7e22..e9e36358c39d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
@@ -163,17 +163,11 @@ int mlx5e_create_mdev_resources(struct mlx5_core_dev *mdev, bool create_tises)
goto err_dealloc_transport_domain;
}
- err = mlx5_alloc_bfreg(mdev, &res->bfreg, false, false);
- if (err) {
- mlx5_core_err(mdev, "alloc bfreg failed, %d\n", err);
- goto err_destroy_mkey;
- }
-
if (create_tises) {
err = mlx5e_create_tises(mdev, res->tisn);
if (err) {
mlx5_core_err(mdev, "alloc tises failed, %d\n", err);
- goto err_destroy_bfreg;
+ goto err_destroy_mkey;
}
res->tisn_valid = true;
}
@@ -190,8 +184,6 @@ int mlx5e_create_mdev_resources(struct mlx5_core_dev *mdev, bool create_tises)
return 0;
-err_destroy_bfreg:
- mlx5_free_bfreg(mdev, &res->bfreg);
err_destroy_mkey:
mlx5_core_destroy_mkey(mdev, res->mkey);
err_dealloc_transport_domain:
@@ -209,7 +201,6 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
mdev->mlx5e_res.dek_priv = NULL;
if (res->tisn_valid)
mlx5e_destroy_tises(mdev, res->tisn);
- mlx5_free_bfreg(mdev, &res->bfreg);
mlx5_core_destroy_mkey(mdev, res->mkey);
mlx5_core_dealloc_transport_domain(mdev, res->td.tdn);
mlx5_core_dealloc_pd(mdev, res->pdn);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index c5a2fa415227..1f46eae73319 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1460,7 +1460,7 @@ static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c,
sq->pdev = c->pdev;
sq->mkey_be = c->mkey_be;
sq->channel = c;
- sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map;
+ sq->uar_map = mdev->priv.bfreg.map;
sq->min_inline_mode = params->tx_min_inline_mode;
sq->hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu) - ETH_FCS_LEN;
sq->xsk_pool = xsk_pool;
@@ -1545,7 +1545,7 @@ static int mlx5e_alloc_icosq(struct mlx5e_channel *c,
int err;
sq->channel = c;
- sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map;
+ sq->uar_map = mdev->priv.bfreg.map;
sq->reserved_room = param->stop_room;
param->wq.db_numa_node = cpu_to_node(c->cpu);
@@ -1630,7 +1630,7 @@ static int mlx5e_alloc_txqsq(struct mlx5e_channel *c,
sq->priv = c->priv;
sq->ch_ix = c->ix;
sq->txq_ix = txq_ix;
- sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map;
+ sq->uar_map = mdev->priv.bfreg.map;
sq->min_inline_mode = params->tx_min_inline_mode;
sq->hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu);
sq->max_sq_mpw_wqebbs = mlx5e_get_max_sq_aligned_wqebbs(mdev);
@@ -1708,7 +1708,7 @@ static int mlx5e_create_sq(struct mlx5_core_dev *mdev,
MLX5_SET(sqc, sqc, flush_in_error_en, 1);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
- MLX5_SET(wq, wq, uar_page, mdev->mlx5e_res.hw_objs.bfreg.index);
+ MLX5_SET(wq, wq, uar_page, mdev->priv.bfreg.index);
MLX5_SET(wq, wq, log_wq_pg_sz, csp->wq_ctrl->buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(wq, wq, dbr_addr, csp->wq_ctrl->db.dma);
@@ -2061,6 +2061,7 @@ void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq)
static int mlx5e_alloc_cq_common(struct mlx5_core_dev *mdev,
struct net_device *netdev,
struct workqueue_struct *workqueue,
+ struct mlx5_uars_page *uar,
struct mlx5e_cq_param *param,
struct mlx5e_cq *cq)
{
@@ -2077,7 +2078,6 @@ static int mlx5e_alloc_cq_common(struct mlx5_core_dev *mdev,
mcq->set_ci_db = cq->wq_ctrl.db.db;
mcq->arm_db = cq->wq_ctrl.db.db + 1;
*mcq->set_ci_db = 0;
- *mcq->arm_db = 0;
mcq->vector = param->eq_ix;
mcq->comp = mlx5e_completion_event;
mcq->event = mlx5e_cq_error_event;
@@ -2092,6 +2092,7 @@ static int mlx5e_alloc_cq_common(struct mlx5_core_dev *mdev,
cq->mdev = mdev;
cq->netdev = netdev;
cq->workqueue = workqueue;
+ cq->uar = uar;
return 0;
}
@@ -2107,7 +2108,8 @@ static int mlx5e_alloc_cq(struct mlx5_core_dev *mdev,
param->wq.db_numa_node = ccp->node;
param->eq_ix = ccp->ix;
- err = mlx5e_alloc_cq_common(mdev, ccp->netdev, ccp->wq, param, cq);
+ err = mlx5e_alloc_cq_common(mdev, ccp->netdev, ccp->wq,
+ mdev->priv.bfreg.up, param, cq);
cq->napi = ccp->napi;
cq->ch_stats = ccp->ch_stats;
@@ -2152,7 +2154,7 @@ static int mlx5e_create_cq(struct mlx5e_cq *cq, struct mlx5e_cq_param *param)
MLX5_SET(cqc, cqc, cq_period_mode, mlx5e_cq_period_mode(param->cq_period_mode));
MLX5_SET(cqc, cqc, c_eqn_or_apu_element, eqn);
- MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
+ MLX5_SET(cqc, cqc, uar_page, cq->uar->index);
MLX5_SET(cqc, cqc, log_page_size, cq->wq_ctrl.buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(cqc, cqc, dbr_addr, cq->wq_ctrl.db.dma);
@@ -2603,7 +2605,7 @@ static int mlx5e_channel_stats_alloc(struct mlx5e_priv *priv, int ix, int cpu)
GFP_KERNEL, cpu_to_node(cpu));
if (!priv->channel_stats[ix])
return -ENOMEM;
- priv->stats_nch++;
+ mlx5e_stats_nch_write(priv, priv->stats_nch + 1);
return 0;
}
@@ -3447,7 +3449,8 @@ static int mlx5e_alloc_drop_cq(struct mlx5e_priv *priv,
param->wq.buf_numa_node = dev_to_node(mlx5_core_dma_dev(mdev));
param->wq.db_numa_node = dev_to_node(mlx5_core_dma_dev(mdev));
- return mlx5e_alloc_cq_common(priv->mdev, priv->netdev, priv->wq, param, cq);
+ return mlx5e_alloc_cq_common(priv->mdev, priv->netdev, priv->wq,
+ mdev->priv.bfreg.up, param, cq);
}
int mlx5e_open_drop_rq(struct mlx5e_priv *priv,
@@ -3819,9 +3822,10 @@ static int mlx5e_setup_tc(struct net_device *dev, enum tc_setup_type type,
void mlx5e_fold_sw_stats64(struct mlx5e_priv *priv, struct rtnl_link_stats64 *s)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
int i;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
struct mlx5e_rq_stats *xskrq_stats = &channel_stats->xskrq;
struct mlx5e_rq_stats *rq_stats = &channel_stats->rq;
@@ -4397,9 +4401,9 @@ static netdev_features_t mlx5e_fix_features(struct net_device *netdev,
if (mlx5e_is_uplink_rep(priv)) {
features = mlx5e_fix_uplink_rep_features(netdev, features);
- netdev->netns_local = true;
+ netdev->netns_immutable = true;
} else {
- netdev->netns_local = false;
+ netdev->netns_immutable = false;
}
mutex_unlock(&priv->state_lock);
@@ -5287,7 +5291,7 @@ static void mlx5e_get_queue_stats_rx(struct net_device *dev, int i,
struct mlx5e_rq_stats *rq_stats;
ASSERT_RTNL();
- if (mlx5e_is_uplink_rep(priv) || !priv->stats_nch)
+ if (mlx5e_is_uplink_rep(priv) || !mlx5e_stats_nch_read(priv))
return;
channel_stats = priv->channel_stats[i];
@@ -5307,7 +5311,7 @@ static void mlx5e_get_queue_stats_tx(struct net_device *dev, int i,
struct mlx5e_sq_stats *sq_stats;
ASSERT_RTNL();
- if (!priv->stats_nch)
+ if (!mlx5e_stats_nch_read(priv))
return;
/* no special case needed for ptp htb etc since txq2sq_stats is kept up
@@ -5324,6 +5328,7 @@ static void mlx5e_get_base_stats(struct net_device *dev,
struct netdev_queue_stats_tx *tx)
{
struct mlx5e_priv *priv = netdev_priv(dev);
+ u16 nch = mlx5e_stats_nch_read(priv);
struct mlx5e_ptp *ptp_channel;
int i, tc;
@@ -5333,7 +5338,7 @@ static void mlx5e_get_base_stats(struct net_device *dev,
rx->bytes = 0;
rx->alloc_fail = 0;
- for (i = priv->channels.params.num_channels; i < priv->stats_nch; i++) {
+ for (i = priv->channels.params.num_channels; i < nch; i++) {
struct netdev_queue_stats_rx rx_i = {0};
mlx5e_get_queue_stats_rx(dev, i, &rx_i);
@@ -5358,7 +5363,7 @@ static void mlx5e_get_base_stats(struct net_device *dev,
tx->packets = 0;
tx->bytes = 0;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
/* handle two cases:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
index 763b264721af..e07e2f5f8443 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
@@ -56,7 +56,6 @@
#include "lib/vxlan.h"
#define CREATE_TRACE_POINTS
#include "diag/en_rep_tracepoint.h"
-#include "diag/reporter_vnic.h"
#include "en_accel/ipsec.h"
#include "en/tc/int_port.h"
#include "en/ptp.h"
@@ -904,7 +903,7 @@ static void mlx5e_build_rep_netdev(struct net_device *netdev,
netdev->features |= netdev->hw_features;
- netdev->netns_local = true;
+ netdev->netns_immutable = true;
}
static int mlx5e_init_rep(struct mlx5_core_dev *mdev,
@@ -1412,51 +1411,6 @@ static unsigned int mlx5e_ul_rep_stats_grps_num(struct mlx5e_priv *priv)
return ARRAY_SIZE(mlx5e_ul_rep_stats_grps);
}
-static int
-mlx5e_rep_vnic_reporter_diagnose(struct devlink_health_reporter *reporter,
- struct devlink_fmsg *fmsg,
- struct netlink_ext_ack *extack)
-{
- struct mlx5e_rep_priv *rpriv = devlink_health_reporter_priv(reporter);
- struct mlx5_eswitch_rep *rep = rpriv->rep;
-
- mlx5_reporter_vnic_diagnose_counters(rep->esw->dev, fmsg, rep->vport,
- true);
- return 0;
-}
-
-static const struct devlink_health_reporter_ops mlx5_rep_vnic_reporter_ops = {
- .name = "vnic",
- .diagnose = mlx5e_rep_vnic_reporter_diagnose,
-};
-
-static void mlx5e_rep_vnic_reporter_create(struct mlx5e_priv *priv,
- struct devlink_port *dl_port)
-{
- struct mlx5e_rep_priv *rpriv = priv->ppriv;
- struct devlink_health_reporter *reporter;
-
- reporter = devl_port_health_reporter_create(dl_port,
- &mlx5_rep_vnic_reporter_ops,
- 0, rpriv);
- if (IS_ERR(reporter)) {
- mlx5_core_err(priv->mdev,
- "Failed to create representor vnic reporter, err = %ld\n",
- PTR_ERR(reporter));
- return;
- }
-
- rpriv->rep_vnic_reporter = reporter;
-}
-
-static void mlx5e_rep_vnic_reporter_destroy(struct mlx5e_priv *priv)
-{
- struct mlx5e_rep_priv *rpriv = priv->ppriv;
-
- if (!IS_ERR_OR_NULL(rpriv->rep_vnic_reporter))
- devl_health_reporter_destroy(rpriv->rep_vnic_reporter);
-}
-
static const struct mlx5e_profile mlx5e_rep_profile = {
.init = mlx5e_init_rep,
.cleanup = mlx5e_cleanup_rep,
@@ -1580,10 +1534,8 @@ mlx5e_vport_vf_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
dl_port = mlx5_esw_offloads_devlink_port(dev->priv.eswitch,
rpriv->rep->vport);
- if (!IS_ERR(dl_port)) {
+ if (!IS_ERR(dl_port))
SET_NETDEV_DEVLINK_PORT(netdev, dl_port);
- mlx5e_rep_vnic_reporter_create(priv, dl_port);
- }
err = register_netdev(netdev);
if (err) {
@@ -1596,7 +1548,6 @@ mlx5e_vport_vf_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep)
return 0;
err_detach_netdev:
- mlx5e_rep_vnic_reporter_destroy(priv);
mlx5e_detach_netdev(netdev_priv(netdev));
err_cleanup_profile:
priv->profile->cleanup(priv);
@@ -1654,7 +1605,6 @@ mlx5e_vport_rep_unload(struct mlx5_eswitch_rep *rep)
}
unregister_netdev(netdev);
- mlx5e_rep_vnic_reporter_destroy(priv);
mlx5e_detach_netdev(priv);
priv->profile->cleanup(priv);
mlx5e_destroy_netdev(netdev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h
index 70640fa1ad7b..bcd7b4e814d0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h
@@ -118,7 +118,6 @@ struct mlx5e_rep_priv {
struct rtnl_link_stats64 prev_vf_vport_stats;
struct mlx5_flow_handle *send_to_vport_meta_rule;
struct rhashtable tc_ht;
- struct devlink_health_reporter *rep_vnic_reporter;
};
static inline
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 3fb3d9299528..71f0519be09d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -1732,6 +1732,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi
struct skb_shared_info *sinfo;
u32 frag_consumed_bytes;
struct bpf_prog *prog;
+ u8 nr_frags_free = 0;
struct sk_buff *skb;
dma_addr_t addr;
u32 truesize;
@@ -1774,15 +1775,13 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi
prog = rcu_dereference(rq->xdp_prog);
if (prog) {
- u8 nr_frags_free, old_nr_frags = sinfo->nr_frags;
+ u8 old_nr_frags = sinfo->nr_frags;
if (mlx5e_xdp_handle(rq, prog, mxbuf)) {
if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT,
rq->flags)) {
struct mlx5e_wqe_frag_info *pwi;
- wi -= old_nr_frags - sinfo->nr_frags;
-
for (pwi = head_wi; pwi < wi; pwi++)
pwi->frag_page->frags++;
}
@@ -1790,10 +1789,8 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi
}
nr_frags_free = old_nr_frags - sinfo->nr_frags;
- if (unlikely(nr_frags_free)) {
- wi -= nr_frags_free;
+ if (unlikely(nr_frags_free))
truesize -= nr_frags_free * frag_info->frag_stride;
- }
}
skb = mlx5e_build_linear_skb(
@@ -1809,7 +1806,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi
if (xdp_buff_has_frags(&mxbuf->xdp)) {
/* sinfo->nr_frags is reset by build_skb, calculate again. */
- xdp_update_skb_shared_info(skb, wi - head_wi - 1,
+ xdp_update_skb_shared_info(skb, wi - head_wi - nr_frags_free - 1,
sinfo->xdp_frags_size, truesize,
xdp_buff_is_frag_pfmemalloc(
&mxbuf->xdp));
@@ -2091,14 +2088,13 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
if (prog) {
u8 nr_frags_free, old_nr_frags = sinfo->nr_frags;
+ u8 new_nr_frags;
u32 len;
if (mlx5e_xdp_handle(rq, prog, mxbuf)) {
if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) {
struct mlx5e_frag_page *pfp;
- frag_page -= old_nr_frags - sinfo->nr_frags;
-
for (pfp = head_page; pfp < frag_page; pfp++)
pfp->frags++;
@@ -2108,13 +2104,12 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
return NULL; /* page/packet was consumed by XDP */
}
- nr_frags_free = old_nr_frags - sinfo->nr_frags;
- if (unlikely(nr_frags_free)) {
- frag_page -= nr_frags_free;
+ new_nr_frags = sinfo->nr_frags;
+ nr_frags_free = old_nr_frags - new_nr_frags;
+ if (unlikely(nr_frags_free))
truesize -= (nr_frags_free - 1) * page_size +
ALIGN(pg_consumed_bytes,
BIT(rq->mpwqe.log_stride_sz));
- }
len = mxbuf->xdp.data_end - mxbuf->xdp.data;
@@ -2135,7 +2130,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
struct mlx5e_frag_page *pagep;
/* sinfo->nr_frags is reset by build_skb, calculate again. */
- xdp_update_skb_shared_info(skb, frag_page - head_page,
+ xdp_update_skb_shared_info(skb, new_nr_frags,
sinfo->xdp_frags_size, truesize,
xdp_buff_is_frag_pfmemalloc(
&mxbuf->xdp));
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
index 2f077ad1e810..652cae2feb01 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
@@ -528,6 +528,7 @@ static void mlx5e_stats_update_stats_rq_page_pool(struct mlx5e_channel *c)
static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
{
struct mlx5e_sw_stats *s = &priv->stats.sw;
+ u16 nch = mlx5e_stats_nch_read(priv);
int i;
memset(s, 0, sizeof(*s));
@@ -535,7 +536,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
for (i = 0; i < priv->channels.num; i++) /* for active channels only */
mlx5e_stats_update_stats_rq_page_pool(priv->channels.c[i]);
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats =
priv->channel_stats[i];
@@ -2438,7 +2439,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(ptp) { return; }
static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(channels)
{
- int max_nch = priv->stats_nch;
+ int max_nch = mlx5e_stats_nch_read(priv);
return (NUM_RQ_STATS * max_nch) +
(NUM_CH_STATS * max_nch) +
@@ -2451,8 +2452,8 @@ static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(channels)
static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(channels)
{
+ int max_nch = mlx5e_stats_nch_read(priv);
bool is_xsk = priv->xsk.ever_used;
- int max_nch = priv->stats_nch;
int i, j, tc;
for (i = 0; i < max_nch; i++)
@@ -2484,8 +2485,8 @@ static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(channels)
static MLX5E_DECLARE_STATS_GRP_OP_FILL_STATS(channels)
{
+ int max_nch = mlx5e_stats_nch_read(priv);
bool is_xsk = priv->xsk.ever_used;
- int max_nch = priv->stats_nch;
int i, j, tc;
for (i = 0; i < max_nch; i++)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
index 4241cf07a030..78a6d11713c9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c
@@ -32,9 +32,7 @@ enum {
MLX5_EQ_STATE_ALWAYS_ARMED = 0xb,
};
-enum {
- MLX5_EQ_DOORBEL_OFFSET = 0x40,
-};
+#define MLX5_EQ_DOORBELL_OFFSET 0x40
/* budget must be smaller than MLX5_NUM_SPARE_EQE to guarantee that we update
* the ci before we polled all the entries in the EQ. MLX5_NUM_SPARE_EQE is
@@ -318,7 +316,7 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
eqc = MLX5_ADDR_OF(create_eq_in, in, eq_context_entry);
MLX5_SET(eqc, eqc, log_eq_size, eq->fbc.log_sz);
- MLX5_SET(eqc, eqc, uar_page, priv->uar->index);
+ MLX5_SET(eqc, eqc, uar_page, priv->bfreg.up->index);
MLX5_SET(eqc, eqc, intr, vecidx);
MLX5_SET(eqc, eqc, log_page_size,
eq->frag_buf.page_shift - MLX5_ADAPTER_PAGE_SHIFT);
@@ -331,7 +329,7 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
eq->eqn = MLX5_GET(create_eq_out, out, eq_number);
eq->irqn = pci_irq_vector(dev->pdev, vecidx);
eq->dev = dev;
- eq->doorbell = priv->uar->map + MLX5_EQ_DOORBEL_OFFSET;
+ eq->doorbell = priv->bfreg.up->map + MLX5_EQ_DOORBELL_OFFSET;
err = mlx5_debug_eq_add(dev, eq);
if (err)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c
index b0c97648ffc7..9283b7c46e50 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c
@@ -3,6 +3,26 @@
#include <linux/mlx5/driver.h>
#include "eswitch.h"
+#include "diag/reporter_vnic.h"
+
+static int
+mlx5_esw_rep_vnic_reporter_diagnose(struct devlink_health_reporter *reporter,
+ struct devlink_fmsg *fmsg,
+ struct netlink_ext_ack *extack)
+{
+ struct mlx5_vport *vport = devlink_health_reporter_priv(reporter);
+
+ mlx5_reporter_vnic_diagnose_counters(vport->dev, fmsg, vport->vport,
+ true);
+
+ return 0;
+}
+
+static const
+struct devlink_health_reporter_ops mlx5_esw_rep_vnic_reporter_ops = {
+ .name = "vnic",
+ .diagnose = mlx5_esw_rep_vnic_reporter_diagnose,
+};
static void
mlx5_esw_get_port_parent_id(struct mlx5_core_dev *dev, struct netdev_phys_item_id *ppid)
@@ -153,6 +173,7 @@ static const struct devlink_port_ops mlx5_esw_dl_sf_port_ops = {
int mlx5_esw_offloads_devlink_port_register(struct mlx5_eswitch *esw, struct mlx5_vport *vport)
{
+ struct devlink_health_reporter *reporter;
struct mlx5_core_dev *dev = esw->dev;
const struct devlink_port_ops *ops;
struct mlx5_devlink_port *dl_port;
@@ -182,6 +203,16 @@ int mlx5_esw_offloads_devlink_port_register(struct mlx5_eswitch *esw, struct mlx
if (err)
goto rate_err;
+ reporter = devl_port_health_reporter_create(
+ &dl_port->dl_port, &mlx5_esw_rep_vnic_reporter_ops,
+ 0, vport);
+ if (IS_ERR(reporter))
+ mlx5_core_err(dev,
+ "Failed to create vnic health reporter for vport %d: %ld\n",
+ vport_num, PTR_ERR(reporter));
+ else
+ dl_port->vnic_reporter = reporter;
+
return 0;
rate_err:
@@ -197,6 +228,11 @@ void mlx5_esw_offloads_devlink_port_unregister(struct mlx5_eswitch *esw, struct
return;
dl_port = vport->dl_port;
+ if (dl_port->vnic_reporter) {
+ devl_health_reporter_destroy(dl_port->vnic_reporter);
+ dl_port->vnic_reporter = NULL;
+ }
+
mlx5_esw_qos_vport_update_group(esw, vport, NULL, NULL);
devl_rate_leaf_destroy(&dl_port->dl_port);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
index 63c2b36ce967..af25da01181d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
@@ -179,6 +179,7 @@ struct mlx5_vport;
struct mlx5_devlink_port {
struct devlink_port dl_port;
struct mlx5_vport *vport;
+ struct devlink_health_reporter *vnic_reporter;
};
static inline void mlx5_devlink_port_init(struct mlx5_devlink_port *dl_port,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
index c4de6bf8d1b6..ccef64fb40b6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
@@ -421,6 +421,13 @@ static int mlx5_fpga_conn_create_cq(struct mlx5_fpga_conn *conn, int cq_size)
__be64 *pas;
u32 i;
+ conn->cq.mcq.cqe_sz = 64;
+ conn->cq.mcq.set_ci_db = conn->cq.wq_ctrl.db.db;
+ conn->cq.mcq.arm_db = conn->cq.wq_ctrl.db.db + 1;
+ *conn->cq.mcq.set_ci_db = 0;
+ conn->cq.mcq.vector = 0;
+ conn->cq.mcq.comp = mlx5_fpga_conn_cq_complete;
+
cq_size = roundup_pow_of_two(cq_size);
MLX5_SET(cqc, temp_cqc, log_cq_size, ilog2(cq_size));
@@ -468,16 +475,7 @@ static int mlx5_fpga_conn_create_cq(struct mlx5_fpga_conn *conn, int cq_size)
if (err)
goto err_cqwq;
- conn->cq.mcq.cqe_sz = 64;
- conn->cq.mcq.set_ci_db = conn->cq.wq_ctrl.db.db;
- conn->cq.mcq.arm_db = conn->cq.wq_ctrl.db.db + 1;
- *conn->cq.mcq.set_ci_db = 0;
- *conn->cq.mcq.arm_db = 0;
- conn->cq.mcq.vector = 0;
- conn->cq.mcq.comp = mlx5_fpga_conn_cq_complete;
- conn->cq.mcq.uar = fdev->conn_res.uar;
tasklet_setup(&conn->cq.tasklet, mlx5_fpga_conn_cq_tasklet);
-
mlx5_fpga_dbg(fdev, "Created CQ #0x%x\n", conn->cq.mcq.cqn);
goto out;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
index 0979d672d47f..2c71e2ac40c5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
@@ -114,10 +114,11 @@ void mlx5i_cleanup(struct mlx5e_priv *priv)
static void mlx5i_grp_sw_update_stats(struct mlx5e_priv *priv)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
struct rtnl_link_stats64 s = {};
int i, j;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats;
struct mlx5e_rq_stats *rq_stats;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/aso.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/aso.c
index 58bd749b5e4d..129725159a93 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/aso.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/aso.c
@@ -100,7 +100,7 @@ static int create_aso_cq(struct mlx5_aso_cq *cq, void *cqc_data)
MLX5_SET(cqc, cqc, cq_period_mode, MLX5_CQ_PERIOD_MODE_START_FROM_EQE);
MLX5_SET(cqc, cqc, c_eqn_or_apu_element, eqn);
- MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
+ MLX5_SET(cqc, cqc, uar_page, mdev->priv.bfreg.up->index);
MLX5_SET(cqc, cqc, log_page_size, cq->wq_ctrl.buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(cqc, cqc, dbr_addr, cq->wq_ctrl.db.dma);
@@ -129,7 +129,7 @@ static int mlx5_aso_create_cq(struct mlx5_core_dev *mdev, int numa_node,
return -ENOMEM;
MLX5_SET(cqc, cqc_data, log_cq_size, 1);
- MLX5_SET(cqc, cqc_data, uar_page, mdev->priv.uar->index);
+ MLX5_SET(cqc, cqc_data, uar_page, mdev->priv.bfreg.up->index);
if (MLX5_CAP_GEN(mdev, cqe_128_always) && cache_line_size() >= 128)
MLX5_SET(cqc, cqc_data, cqe_sz, CQE_STRIDE_128_PAD);
@@ -163,7 +163,7 @@ static int mlx5_aso_alloc_sq(struct mlx5_core_dev *mdev, int numa_node,
struct mlx5_wq_param param;
int err;
- sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map;
+ sq->uar_map = mdev->priv.bfreg.map;
param.db_numa_node = numa_node;
param.buf_numa_node = numa_node;
@@ -203,7 +203,7 @@ static int create_aso_sq(struct mlx5_core_dev *mdev, int pdn,
MLX5_SET(sqc, sqc, ts_format, ts_format);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
- MLX5_SET(wq, wq, uar_page, mdev->mlx5e_res.hw_objs.bfreg.index);
+ MLX5_SET(wq, wq, uar_page, mdev->priv.bfreg.index);
MLX5_SET(wq, wq, log_wq_pg_sz, sq->wq_ctrl.buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(wq, wq, dbr_addr, sq->wq_ctrl.db.dma);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index fd5676dfda4a..0496f2e3f2a2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -1321,10 +1321,9 @@ static int mlx5_load(struct mlx5_core_dev *dev)
{
int err;
- dev->priv.uar = mlx5_get_uars_page(dev);
- if (IS_ERR(dev->priv.uar)) {
- mlx5_core_err(dev, "Failed allocating uar, aborting\n");
- err = PTR_ERR(dev->priv.uar);
+ err = mlx5_alloc_bfreg(dev, &dev->priv.bfreg, false, false);
+ if (err) {
+ mlx5_core_err(dev, "Failed allocating bfreg, %d\n", err);
return err;
}
@@ -1432,7 +1431,7 @@ static int mlx5_load(struct mlx5_core_dev *dev)
err_irq_table:
mlx5_pagealloc_stop(dev);
mlx5_events_stop(dev);
- mlx5_put_uars_page(dev, dev->priv.uar);
+ mlx5_free_bfreg(dev, &dev->priv.bfreg);
return err;
}
@@ -1455,7 +1454,7 @@ static void mlx5_unload(struct mlx5_core_dev *dev)
mlx5_irq_table_destroy(dev);
mlx5_pagealloc_stop(dev);
mlx5_events_stop(dev);
- mlx5_put_uars_page(dev, dev->priv.uar);
+ mlx5_free_bfreg(dev, &dev->priv.bfreg);
}
int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c
index f57c84e5128b..645d32936776 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c
@@ -1049,12 +1049,6 @@ static int dr_prepare_qp_to_rts(struct mlx5dr_domain *dmn)
return 0;
}
-static void dr_cq_complete(struct mlx5_core_cq *mcq,
- struct mlx5_eqe *eqe)
-{
- pr_err("CQ completion CQ: #%u\n", mcq->cqn);
-}
-
static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
struct mlx5_uars_page *uar,
size_t ncqe)
@@ -1089,6 +1083,13 @@ static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
cqe->op_own = MLX5_CQE_INVALID << 4 | MLX5_CQE_OWNER_MASK;
}
+ cq->mcq.cqe_sz = 64;
+ cq->mcq.set_ci_db = cq->wq_ctrl.db.db;
+ cq->mcq.arm_db = cq->wq_ctrl.db.db + 1;
+ *cq->mcq.set_ci_db = 0;
+ cq->mcq.vector = 0;
+ cq->mdev = mdev;
+
inlen = MLX5_ST_SZ_BYTES(create_cq_in) +
sizeof(u64) * cq->wq_ctrl.buf.npages;
in = kvzalloc(inlen, GFP_KERNEL);
@@ -1112,28 +1113,12 @@ static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
pas = (__be64 *)MLX5_ADDR_OF(create_cq_in, in, pas);
mlx5_fill_page_frag_array(&cq->wq_ctrl.buf, pas);
- cq->mcq.comp = dr_cq_complete;
-
err = mlx5_core_create_cq(mdev, &cq->mcq, in, inlen, out, sizeof(out));
kvfree(in);
if (err)
goto err_cqwq;
- cq->mcq.cqe_sz = 64;
- cq->mcq.set_ci_db = cq->wq_ctrl.db.db;
- cq->mcq.arm_db = cq->wq_ctrl.db.db + 1;
- *cq->mcq.set_ci_db = 0;
-
- /* set no-zero value, in order to avoid the HW to run db-recovery on
- * CQ that used in polling mode.
- */
- *cq->mcq.arm_db = cpu_to_be32(2 << 28);
-
- cq->mcq.vector = 0;
- cq->mcq.uar = uar;
- cq->mdev = mdev;
-
return cq;
err_cqwq:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.c
index b27bb4106532..4fbe0d9d594d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.c
@@ -11,31 +11,29 @@
/* This is the longest supported action sequence for FDB table:
* DECAP, POP_VLAN, MODIFY, CTR, ASO, PUSH_VLAN, MODIFY, ENCAP, Term.
*/
-static const u32 action_order_arr[MLX5HWS_TABLE_TYPE_MAX][MLX5HWS_ACTION_TYP_MAX] = {
- [MLX5HWS_TABLE_TYPE_FDB] = {
- BIT(MLX5HWS_ACTION_TYP_REMOVE_HEADER) |
- BIT(MLX5HWS_ACTION_TYP_REFORMAT_TNL_L2_TO_L2) |
- BIT(MLX5HWS_ACTION_TYP_REFORMAT_TNL_L3_TO_L2),
- BIT(MLX5HWS_ACTION_TYP_POP_VLAN),
- BIT(MLX5HWS_ACTION_TYP_POP_VLAN),
- BIT(MLX5HWS_ACTION_TYP_MODIFY_HDR),
- BIT(MLX5HWS_ACTION_TYP_PUSH_VLAN),
- BIT(MLX5HWS_ACTION_TYP_PUSH_VLAN),
- BIT(MLX5HWS_ACTION_TYP_INSERT_HEADER) |
- BIT(MLX5HWS_ACTION_TYP_REFORMAT_L2_TO_TNL_L2) |
- BIT(MLX5HWS_ACTION_TYP_REFORMAT_L2_TO_TNL_L3),
- BIT(MLX5HWS_ACTION_TYP_CTR),
- BIT(MLX5HWS_ACTION_TYP_TAG),
- BIT(MLX5HWS_ACTION_TYP_ASO_METER),
- BIT(MLX5HWS_ACTION_TYP_MODIFY_HDR),
- BIT(MLX5HWS_ACTION_TYP_TBL) |
- BIT(MLX5HWS_ACTION_TYP_VPORT) |
- BIT(MLX5HWS_ACTION_TYP_DROP) |
- BIT(MLX5HWS_ACTION_TYP_SAMPLER) |
- BIT(MLX5HWS_ACTION_TYP_RANGE) |
- BIT(MLX5HWS_ACTION_TYP_DEST_ARRAY),
- BIT(MLX5HWS_ACTION_TYP_LAST),
- },
+static const u32 action_order_arr[MLX5HWS_ACTION_TYP_MAX] = {
+ BIT(MLX5HWS_ACTION_TYP_REMOVE_HEADER) |
+ BIT(MLX5HWS_ACTION_TYP_REFORMAT_TNL_L2_TO_L2) |
+ BIT(MLX5HWS_ACTION_TYP_REFORMAT_TNL_L3_TO_L2),
+ BIT(MLX5HWS_ACTION_TYP_POP_VLAN),
+ BIT(MLX5HWS_ACTION_TYP_POP_VLAN),
+ BIT(MLX5HWS_ACTION_TYP_MODIFY_HDR),
+ BIT(MLX5HWS_ACTION_TYP_PUSH_VLAN),
+ BIT(MLX5HWS_ACTION_TYP_PUSH_VLAN),
+ BIT(MLX5HWS_ACTION_TYP_INSERT_HEADER) |
+ BIT(MLX5HWS_ACTION_TYP_REFORMAT_L2_TO_TNL_L2) |
+ BIT(MLX5HWS_ACTION_TYP_REFORMAT_L2_TO_TNL_L3),
+ BIT(MLX5HWS_ACTION_TYP_CTR),
+ BIT(MLX5HWS_ACTION_TYP_TAG),
+ BIT(MLX5HWS_ACTION_TYP_ASO_METER),
+ BIT(MLX5HWS_ACTION_TYP_MODIFY_HDR),
+ BIT(MLX5HWS_ACTION_TYP_TBL) |
+ BIT(MLX5HWS_ACTION_TYP_VPORT) |
+ BIT(MLX5HWS_ACTION_TYP_DROP) |
+ BIT(MLX5HWS_ACTION_TYP_SAMPLER) |
+ BIT(MLX5HWS_ACTION_TYP_RANGE) |
+ BIT(MLX5HWS_ACTION_TYP_DEST_ARRAY),
+ BIT(MLX5HWS_ACTION_TYP_LAST),
};
static const char * const mlx5hws_action_type_str[] = {
@@ -83,8 +81,8 @@ static int hws_action_get_shared_stc_nic(struct mlx5hws_context *ctx,
int ret;
mutex_lock(&ctx->ctrl_lock);
- if (ctx->common_res[tbl_type].shared_stc[stc_type]) {
- ctx->common_res[tbl_type].shared_stc[stc_type]->refcount++;
+ if (ctx->common_res.shared_stc[stc_type]) {
+ ctx->common_res.shared_stc[stc_type]->refcount++;
mutex_unlock(&ctx->ctrl_lock);
return 0;
}
@@ -114,7 +112,7 @@ static int hws_action_get_shared_stc_nic(struct mlx5hws_context *ctx,
mlx5hws_err(ctx, "No such stc_type: %d\n", stc_type);
pr_warn("HWS: Invalid stc_type: %d\n", stc_type);
ret = -EINVAL;
- goto unlock_and_out;
+ goto free_shared_stc;
}
ret = mlx5hws_action_alloc_single_stc(ctx, &stc_attr, tbl_type,
@@ -124,8 +122,8 @@ static int hws_action_get_shared_stc_nic(struct mlx5hws_context *ctx,
goto free_shared_stc;
}
- ctx->common_res[tbl_type].shared_stc[stc_type] = shared_stc;
- ctx->common_res[tbl_type].shared_stc[stc_type]->refcount = 1;
+ ctx->common_res.shared_stc[stc_type] = shared_stc;
+ ctx->common_res.shared_stc[stc_type]->refcount = 1;
mutex_unlock(&ctx->ctrl_lock);
@@ -178,16 +176,16 @@ static void hws_action_put_shared_stc(struct mlx5hws_action *action,
}
mutex_lock(&ctx->ctrl_lock);
- if (--ctx->common_res[tbl_type].shared_stc[stc_type]->refcount) {
+ if (--ctx->common_res.shared_stc[stc_type]->refcount) {
mutex_unlock(&ctx->ctrl_lock);
return;
}
- shared_stc = ctx->common_res[tbl_type].shared_stc[stc_type];
+ shared_stc = ctx->common_res.shared_stc[stc_type];
mlx5hws_action_free_single_stc(ctx, tbl_type, &shared_stc->stc_chunk);
kfree(shared_stc);
- ctx->common_res[tbl_type].shared_stc[stc_type] = NULL;
+ ctx->common_res.shared_stc[stc_type] = NULL;
mutex_unlock(&ctx->ctrl_lock);
}
@@ -206,10 +204,10 @@ bool mlx5hws_action_check_combo(struct mlx5hws_context *ctx,
enum mlx5hws_action_type *user_actions,
enum mlx5hws_table_type table_type)
{
- const u32 *order_arr = action_order_arr[table_type];
+ const u32 *order_arr = action_order_arr;
+ bool valid_combo;
u8 order_idx = 0;
u8 user_idx = 0;
- bool valid_combo;
if (table_type >= MLX5HWS_TABLE_TYPE_MAX) {
mlx5hws_err(ctx, "Invalid table_type %d", table_type);
@@ -240,6 +238,7 @@ hws_action_fixup_stc_attr(struct mlx5hws_context *ctx,
enum mlx5hws_table_type table_type,
bool is_mirror)
{
+ struct mlx5hws_pool *pool;
bool use_fixup = false;
u32 fw_tbl_type;
u32 base_id;
@@ -255,13 +254,11 @@ hws_action_fixup_stc_attr(struct mlx5hws_context *ctx,
use_fixup = true;
break;
}
+ pool = stc_attr->ste_table.ste_pool;
if (!is_mirror)
- base_id = mlx5hws_pool_chunk_get_base_id(stc_attr->ste_table.ste_pool,
- &stc_attr->ste_table.ste);
+ base_id = mlx5hws_pool_get_base_id(pool);
else
- base_id =
- mlx5hws_pool_chunk_get_base_mirror_id(stc_attr->ste_table.ste_pool,
- &stc_attr->ste_table.ste);
+ base_id = mlx5hws_pool_get_base_mirror_id(pool);
*fixup_stc_attr = *stc_attr;
fixup_stc_attr->ste_table.ste_obj_id = base_id;
@@ -321,8 +318,8 @@ int mlx5hws_action_alloc_single_stc(struct mlx5hws_context *ctx,
__must_hold(&ctx->ctrl_lock)
{
struct mlx5hws_cmd_stc_modify_attr cleanup_stc_attr = {0};
- struct mlx5hws_pool *stc_pool = ctx->stc_pool[table_type];
struct mlx5hws_cmd_stc_modify_attr fixup_stc_attr = {0};
+ struct mlx5hws_pool *stc_pool = ctx->stc_pool;
bool use_fixup;
u32 obj_0_id;
int ret;
@@ -339,7 +336,7 @@ __must_hold(&ctx->ctrl_lock)
if (!mlx5hws_context_cap_dynamic_reparse(ctx))
stc_attr->reparse_mode = MLX5_IFC_STC_REPARSE_IGNORE;
- obj_0_id = mlx5hws_pool_chunk_get_base_id(stc_pool, stc);
+ obj_0_id = mlx5hws_pool_get_base_id(stc_pool);
/* According to table/action limitation change the stc_attr */
use_fixup = hws_action_fixup_stc_attr(ctx, stc_attr, &fixup_stc_attr, table_type, false);
@@ -355,7 +352,7 @@ __must_hold(&ctx->ctrl_lock)
if (table_type == MLX5HWS_TABLE_TYPE_FDB) {
u32 obj_1_id;
- obj_1_id = mlx5hws_pool_chunk_get_base_mirror_id(stc_pool, stc);
+ obj_1_id = mlx5hws_pool_get_base_mirror_id(stc_pool);
use_fixup = hws_action_fixup_stc_attr(ctx, stc_attr,
&fixup_stc_attr,
@@ -387,19 +384,19 @@ void mlx5hws_action_free_single_stc(struct mlx5hws_context *ctx,
struct mlx5hws_pool_chunk *stc)
__must_hold(&ctx->ctrl_lock)
{
- struct mlx5hws_pool *stc_pool = ctx->stc_pool[table_type];
struct mlx5hws_cmd_stc_modify_attr stc_attr = {0};
+ struct mlx5hws_pool *stc_pool = ctx->stc_pool;
u32 obj_id;
/* Modify the STC not to point to an object */
stc_attr.action_type = MLX5_IFC_STC_ACTION_TYPE_DROP;
stc_attr.action_offset = MLX5HWS_ACTION_OFFSET_HIT;
stc_attr.stc_offset = stc->offset;
- obj_id = mlx5hws_pool_chunk_get_base_id(stc_pool, stc);
+ obj_id = mlx5hws_pool_get_base_id(stc_pool);
mlx5hws_cmd_stc_modify(ctx->mdev, obj_id, &stc_attr);
if (table_type == MLX5HWS_TABLE_TYPE_FDB) {
- obj_id = mlx5hws_pool_chunk_get_base_mirror_id(stc_pool, stc);
+ obj_id = mlx5hws_pool_get_base_mirror_id(stc_pool);
mlx5hws_cmd_stc_modify(ctx->mdev, obj_id, &stc_attr);
}
@@ -561,7 +558,7 @@ hws_action_create_stcs(struct mlx5hws_action *action, u32 obj_id)
if (action->flags & MLX5HWS_ACTION_FLAG_HWS_FDB) {
ret = mlx5hws_action_alloc_single_stc(ctx, &stc_attr,
MLX5HWS_TABLE_TYPE_FDB,
- &action->stc[MLX5HWS_TABLE_TYPE_FDB]);
+ &action->stc);
if (ret)
goto out_err;
}
@@ -585,7 +582,7 @@ hws_action_destroy_stcs(struct mlx5hws_action *action)
if (action->flags & MLX5HWS_ACTION_FLAG_HWS_FDB)
mlx5hws_action_free_single_stc(ctx, MLX5HWS_TABLE_TYPE_FDB,
- &action->stc[MLX5HWS_TABLE_TYPE_FDB]);
+ &action->stc);
mutex_unlock(&ctx->ctrl_lock);
}
@@ -1582,7 +1579,6 @@ hws_action_create_dest_match_range_table(struct mlx5hws_context *ctx,
u32 miss_ft_id)
{
struct mlx5hws_cmd_rtc_create_attr rtc_attr = {0};
- struct mlx5hws_action_default_stc *default_stc;
struct mlx5hws_matcher_action_ste *table_ste;
struct mlx5hws_pool_attr pool_attr = {0};
struct mlx5hws_pool *ste_pool, *stc_pool;
@@ -1605,7 +1601,6 @@ hws_action_create_dest_match_range_table(struct mlx5hws_context *ctx,
pool_attr.table_type = MLX5HWS_TABLE_TYPE_FDB;
pool_attr.pool_type = MLX5HWS_POOL_TYPE_STE;
- pool_attr.flags = MLX5HWS_POOL_FLAGS_FOR_STE_ACTION_POOL;
pool_attr.alloc_log_sz = 1;
table_ste->pool = mlx5hws_pool_create(ctx, &pool_attr);
if (!table_ste->pool) {
@@ -1630,7 +1625,7 @@ hws_action_create_dest_match_range_table(struct mlx5hws_context *ctx,
rtc_attr.fw_gen_wqe = true;
rtc_attr.is_scnd_range = true;
- obj_id = mlx5hws_pool_chunk_get_base_id(ste_pool, ste);
+ obj_id = mlx5hws_pool_get_base_id(ste_pool);
rtc_attr.pd = ctx->pd_num;
rtc_attr.ste_base = obj_id;
@@ -1639,9 +1634,8 @@ hws_action_create_dest_match_range_table(struct mlx5hws_context *ctx,
rtc_attr.table_type = mlx5hws_table_get_res_fw_ft_type(MLX5HWS_TABLE_TYPE_FDB, false);
/* STC is a single resource (obj_id), use any STC for the ID */
- stc_pool = ctx->stc_pool[MLX5HWS_TABLE_TYPE_FDB];
- default_stc = ctx->common_res[MLX5HWS_TABLE_TYPE_FDB].default_stc;
- obj_id = mlx5hws_pool_chunk_get_base_id(stc_pool, &default_stc->default_hit);
+ stc_pool = ctx->stc_pool;
+ obj_id = mlx5hws_pool_get_base_id(stc_pool);
rtc_attr.stc_base = obj_id;
ret = mlx5hws_cmd_rtc_create(ctx->mdev, &rtc_attr, rtc_0_id);
@@ -1651,11 +1645,11 @@ hws_action_create_dest_match_range_table(struct mlx5hws_context *ctx,
}
/* Create mirror RTC */
- obj_id = mlx5hws_pool_chunk_get_base_mirror_id(ste_pool, ste);
+ obj_id = mlx5hws_pool_get_base_mirror_id(ste_pool);
rtc_attr.ste_base = obj_id;
rtc_attr.table_type = mlx5hws_table_get_res_fw_ft_type(MLX5HWS_TABLE_TYPE_FDB, true);
- obj_id = mlx5hws_pool_chunk_get_base_mirror_id(stc_pool, &default_stc->default_hit);
+ obj_id = mlx5hws_pool_get_base_mirror_id(stc_pool);
rtc_attr.stc_base = obj_id;
ret = mlx5hws_cmd_rtc_create(ctx->mdev, &rtc_attr, rtc_1_id);
@@ -1731,7 +1725,7 @@ hws_action_create_dest_match_range_fill_table(struct mlx5hws_context *ctx,
ste_attr.used_id_rtc_0 = &used_rtc_0_id;
ste_attr.used_id_rtc_1 = &used_rtc_1_id;
- common_res = &ctx->common_res[MLX5HWS_TABLE_TYPE_FDB];
+ common_res = &ctx->common_res;
/* init an empty match STE which will always hit */
ste_attr.wqe_ctrl = &wqe_ctrl;
@@ -1750,7 +1744,7 @@ hws_action_create_dest_match_range_fill_table(struct mlx5hws_context *ctx,
wqe_ctrl.stc_ix[MLX5HWS_ACTION_STC_IDX_CTRL] |=
htonl(MLX5HWS_ACTION_STC_IDX_LAST_COMBO2 << 29);
wqe_ctrl.stc_ix[MLX5HWS_ACTION_STC_IDX_HIT] =
- htonl(hit_ft_action->stc[MLX5HWS_TABLE_TYPE_FDB].offset);
+ htonl(hit_ft_action->stc.offset);
wqe_data_arr = (__force __be32 *)&range_wqe_data;
@@ -1843,7 +1837,7 @@ mlx5hws_action_create_dest_match_range(struct mlx5hws_context *ctx,
stc_attr.ste_table.match_definer_id = ctx->caps->trivial_match_definer;
ret = mlx5hws_action_alloc_single_stc(ctx, &stc_attr, MLX5HWS_TABLE_TYPE_FDB,
- &action->stc[MLX5HWS_TABLE_TYPE_FDB]);
+ &action->stc);
if (ret)
goto error_unlock;
@@ -1970,8 +1964,8 @@ __must_hold(&ctx->ctrl_lock)
struct mlx5hws_action_default_stc *default_stc;
int ret;
- if (ctx->common_res[tbl_type].default_stc) {
- ctx->common_res[tbl_type].default_stc->refcount++;
+ if (ctx->common_res.default_stc) {
+ ctx->common_res.default_stc->refcount++;
return 0;
}
@@ -2023,8 +2017,8 @@ __must_hold(&ctx->ctrl_lock)
goto free_nop_dw7;
}
- ctx->common_res[tbl_type].default_stc = default_stc;
- ctx->common_res[tbl_type].default_stc->refcount++;
+ ctx->common_res.default_stc = default_stc;
+ ctx->common_res.default_stc->refcount++;
return 0;
@@ -2046,9 +2040,7 @@ __must_hold(&ctx->ctrl_lock)
{
struct mlx5hws_action_default_stc *default_stc;
- default_stc = ctx->common_res[tbl_type].default_stc;
-
- default_stc = ctx->common_res[tbl_type].default_stc;
+ default_stc = ctx->common_res.default_stc;
if (--default_stc->refcount)
return;
@@ -2058,7 +2050,7 @@ __must_hold(&ctx->ctrl_lock)
mlx5hws_action_free_single_stc(ctx, tbl_type, &default_stc->nop_dw5);
mlx5hws_action_free_single_stc(ctx, tbl_type, &default_stc->nop_ctr);
kfree(default_stc);
- ctx->common_res[tbl_type].default_stc = NULL;
+ ctx->common_res.default_stc = NULL;
}
static void hws_action_modify_write(struct mlx5hws_send_engine *queue,
@@ -2150,8 +2142,7 @@ hws_action_apply_stc(struct mlx5hws_actions_apply_data *apply,
{
struct mlx5hws_action *action = apply->rule_action[action_idx].action;
- apply->wqe_ctrl->stc_ix[stc_idx] =
- htonl(action->stc[apply->tbl_type].offset);
+ apply->wqe_ctrl->stc_ix[stc_idx] = htonl(action->stc.offset);
}
static void
@@ -2181,7 +2172,7 @@ hws_action_setter_modify_header(struct mlx5hws_actions_apply_data *apply,
rule_action = &apply->rule_action[setter->idx_double];
action = rule_action->action;
- stc_idx = htonl(action->stc[apply->tbl_type].offset);
+ stc_idx = htonl(action->stc.offset);
apply->wqe_ctrl->stc_ix[MLX5HWS_ACTION_STC_IDX_DW6] = stc_idx;
apply->wqe_ctrl->stc_ix[MLX5HWS_ACTION_STC_IDX_DW7] = 0;
@@ -2240,7 +2231,7 @@ hws_action_setter_insert_ptr(struct mlx5hws_actions_apply_data *apply,
apply->wqe_data[MLX5HWS_ACTION_OFFSET_DW6] = 0;
apply->wqe_data[MLX5HWS_ACTION_OFFSET_DW7] = htonl(arg_idx);
- stc_idx = htonl(action->stc[apply->tbl_type].offset);
+ stc_idx = htonl(action->stc.offset);
apply->wqe_ctrl->stc_ix[MLX5HWS_ACTION_STC_IDX_DW6] = stc_idx;
apply->wqe_ctrl->stc_ix[MLX5HWS_ACTION_STC_IDX_DW7] = 0;
@@ -2272,7 +2263,7 @@ hws_action_setter_tnl_l3_to_l2(struct mlx5hws_actions_apply_data *apply,
apply->wqe_data[MLX5HWS_ACTION_OFFSET_DW6] = 0;
apply->wqe_data[MLX5HWS_ACTION_OFFSET_DW7] = htonl(arg_idx);
- stc_idx = htonl(action->stc[apply->tbl_type].offset);
+ stc_idx = htonl(action->stc.offset);
apply->wqe_ctrl->stc_ix[MLX5HWS_ACTION_STC_IDX_DW6] = stc_idx;
apply->wqe_ctrl->stc_ix[MLX5HWS_ACTION_STC_IDX_DW7] = 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.h
index bf5c1b241006..2283eefeb223 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_action.h
@@ -70,12 +70,12 @@ struct mlx5hws_action_default_stc {
struct mlx5hws_pool_chunk nop_dw6;
struct mlx5hws_pool_chunk nop_dw7;
struct mlx5hws_pool_chunk default_hit;
- u32 refcount;
+ u32 refcount; /* protected by context ctrl lock */
};
struct mlx5hws_action_shared_stc {
struct mlx5hws_pool_chunk stc_chunk;
- u32 refcount;
+ u32 refcount; /* protected by context ctrl lock */
};
struct mlx5hws_actions_apply_data {
@@ -124,7 +124,7 @@ struct mlx5hws_action {
struct mlx5hws_context *ctx;
union {
struct {
- struct mlx5hws_pool_chunk stc[MLX5HWS_TABLE_TYPE_MAX];
+ struct mlx5hws_pool_chunk stc;
union {
struct {
u32 pat_id;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.c
index 2c7b14172049..7b8eefead1a0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.c
@@ -359,7 +359,7 @@ void mlx5hws_cmd_set_attr_connect_miss_tbl(struct mlx5hws_context *ctx,
ft_attr->type = fw_ft_type;
ft_attr->table_miss_action = MLX5_IFC_MODIFY_FLOW_TABLE_MISS_ACTION_GOTO_TBL;
- default_miss_tbl = ctx->common_res[type].default_miss->ft_id;
+ default_miss_tbl = ctx->common_res.default_miss->ft_id;
if (!default_miss_tbl) {
pr_warn("HWS: no flow table ID for default miss\n");
return;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.h
index 2fbcf4ff571a..e93955ddfbb8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_cmd.h
@@ -63,7 +63,7 @@ struct mlx5hws_cmd_forward_tbl {
u8 type;
u32 ft_id;
u32 fg_id;
- u32 refcount;
+ u32 refcount; /* protected by context ctrl lock */
};
struct mlx5hws_cmd_rtc_create_attr {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.c
index 00e4fdf4a558..876123b0a573 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.c
@@ -23,7 +23,6 @@ static int hws_context_pools_init(struct mlx5hws_context *ctx)
struct mlx5hws_pool_attr pool_attr = {0};
u8 max_log_sz;
int ret;
- int i;
ret = mlx5hws_pat_init_pattern_cache(&ctx->pattern_cache);
if (ret)
@@ -35,27 +34,20 @@ static int hws_context_pools_init(struct mlx5hws_context *ctx)
/* Create an STC pool per FT type */
pool_attr.pool_type = MLX5HWS_POOL_TYPE_STC;
- pool_attr.flags = MLX5HWS_POOL_FLAGS_FOR_STC_POOL;
max_log_sz = min(MLX5HWS_POOL_STC_LOG_SZ, ctx->caps->stc_alloc_log_max);
pool_attr.alloc_log_sz = max(max_log_sz, ctx->caps->stc_alloc_log_gran);
- for (i = 0; i < MLX5HWS_TABLE_TYPE_MAX; i++) {
- pool_attr.table_type = i;
- ctx->stc_pool[i] = mlx5hws_pool_create(ctx, &pool_attr);
- if (!ctx->stc_pool[i]) {
- mlx5hws_err(ctx, "Failed to allocate STC pool [%d]", i);
- ret = -ENOMEM;
- goto free_stc_pools;
- }
+ pool_attr.table_type = MLX5HWS_TABLE_TYPE_FDB;
+ ctx->stc_pool = mlx5hws_pool_create(ctx, &pool_attr);
+ if (!ctx->stc_pool) {
+ mlx5hws_err(ctx, "Failed to allocate STC pool\n");
+ ret = -ENOMEM;
+ goto uninit_cache;
}
return 0;
-free_stc_pools:
- for (i = 0; i < MLX5HWS_TABLE_TYPE_MAX; i++)
- if (ctx->stc_pool[i])
- mlx5hws_pool_destroy(ctx->stc_pool[i]);
-
+uninit_cache:
mlx5hws_definer_uninit_cache(ctx->definer_cache);
uninit_pat_cache:
mlx5hws_pat_uninit_pattern_cache(ctx->pattern_cache);
@@ -64,12 +56,8 @@ static int hws_context_pools_init(struct mlx5hws_context *ctx)
static void hws_context_pools_uninit(struct mlx5hws_context *ctx)
{
- int i;
-
- for (i = 0; i < MLX5HWS_TABLE_TYPE_MAX; i++) {
- if (ctx->stc_pool[i])
- mlx5hws_pool_destroy(ctx->stc_pool[i]);
- }
+ if (ctx->stc_pool)
+ mlx5hws_pool_destroy(ctx->stc_pool);
mlx5hws_definer_uninit_cache(ctx->definer_cache);
mlx5hws_pat_uninit_pattern_cache(ctx->pattern_cache);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.h
index 8ab548aa402b..6cd3c613eac1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_context.h
@@ -37,8 +37,8 @@ struct mlx5hws_context {
struct mlx5_core_dev *mdev;
struct mlx5hws_cmd_query_caps *caps;
u32 pd_num;
- struct mlx5hws_pool *stc_pool[MLX5HWS_TABLE_TYPE_MAX];
- struct mlx5hws_context_common_res common_res[MLX5HWS_TABLE_TYPE_MAX];
+ struct mlx5hws_pool *stc_pool;
+ struct mlx5hws_context_common_res common_res;
struct mlx5hws_pattern_cache *pattern_cache;
struct mlx5hws_definer_cache *definer_cache;
struct mutex ctrl_lock; /* control lock to protect the whole context */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_debug.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_debug.c
index 2b8c5a4e1c4c..e0a8de812dae 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_debug.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_debug.c
@@ -118,7 +118,6 @@ static int hws_debug_dump_matcher(struct seq_file *f, struct mlx5hws_matcher *ma
{
enum mlx5hws_table_type tbl_type = matcher->tbl->type;
struct mlx5hws_cmd_ft_query_attr ft_attr = {0};
- struct mlx5hws_pool_chunk *ste;
struct mlx5hws_pool *ste_pool;
u64 icm_addr_0 = 0;
u64 icm_addr_1 = 0;
@@ -134,12 +133,11 @@ static int hws_debug_dump_matcher(struct seq_file *f, struct mlx5hws_matcher *ma
matcher->end_ft_id,
matcher->col_matcher ? HWS_PTR_TO_ID(matcher->col_matcher) : 0);
- ste = &matcher->match_ste.ste;
ste_pool = matcher->match_ste.pool;
if (ste_pool) {
- ste_0_id = mlx5hws_pool_chunk_get_base_id(ste_pool, ste);
+ ste_0_id = mlx5hws_pool_get_base_id(ste_pool);
if (tbl_type == MLX5HWS_TABLE_TYPE_FDB)
- ste_1_id = mlx5hws_pool_chunk_get_base_mirror_id(ste_pool, ste);
+ ste_1_id = mlx5hws_pool_get_base_mirror_id(ste_pool);
}
seq_printf(f, ",%d,%d,%d,%d",
@@ -148,12 +146,11 @@ static int hws_debug_dump_matcher(struct seq_file *f, struct mlx5hws_matcher *ma
matcher->match_ste.rtc_1_id,
(int)ste_1_id);
- ste = &matcher->action_ste[0].ste;
- ste_pool = matcher->action_ste[0].pool;
+ ste_pool = matcher->action_ste.pool;
if (ste_pool) {
- ste_0_id = mlx5hws_pool_chunk_get_base_id(ste_pool, ste);
+ ste_0_id = mlx5hws_pool_get_base_id(ste_pool);
if (tbl_type == MLX5HWS_TABLE_TYPE_FDB)
- ste_1_id = mlx5hws_pool_chunk_get_base_mirror_id(ste_pool, ste);
+ ste_1_id = mlx5hws_pool_get_base_mirror_id(ste_pool);
else
ste_1_id = -1;
} else {
@@ -171,10 +168,8 @@ static int hws_debug_dump_matcher(struct seq_file *f, struct mlx5hws_matcher *ma
return ret;
seq_printf(f, ",%d,%d,%d,%d,%d,0x%llx,0x%llx\n",
- matcher->action_ste[0].rtc_0_id,
- (int)ste_0_id,
- matcher->action_ste[0].rtc_1_id,
- (int)ste_1_id,
+ matcher->action_ste.rtc_0_id, (int)ste_0_id,
+ matcher->action_ste.rtc_1_id, (int)ste_1_id,
0,
mlx5hws_debug_icm_to_idx(icm_addr_0),
mlx5hws_debug_icm_to_idx(icm_addr_1));
@@ -368,9 +363,10 @@ static int hws_debug_dump_context_info(struct seq_file *f, struct mlx5hws_contex
static int hws_debug_dump_context_stc_resource(struct seq_file *f,
struct mlx5hws_context *ctx,
- u32 tbl_type,
struct mlx5hws_pool_resource *resource)
{
+ u32 tbl_type = MLX5HWS_TABLE_TYPE_BASE + MLX5HWS_TABLE_TYPE_FDB;
+
seq_printf(f, "%d,0x%llx,%u,%u\n",
MLX5HWS_DEBUG_RES_TYPE_CONTEXT_STC,
HWS_PTR_TO_ID(ctx),
@@ -382,31 +378,25 @@ static int hws_debug_dump_context_stc_resource(struct seq_file *f,
static int hws_debug_dump_context_stc(struct seq_file *f, struct mlx5hws_context *ctx)
{
- struct mlx5hws_pool *stc_pool;
- u32 table_type;
+ struct mlx5hws_pool *stc_pool = ctx->stc_pool;
int ret;
- int i;
- for (i = 0; i < MLX5HWS_TABLE_TYPE_MAX; i++) {
- stc_pool = ctx->stc_pool[i];
- table_type = MLX5HWS_TABLE_TYPE_BASE + i;
+ if (!stc_pool)
+ return 0;
- if (!stc_pool)
- continue;
+ if (stc_pool->resource) {
+ ret = hws_debug_dump_context_stc_resource(f, ctx,
+ stc_pool->resource);
+ if (ret)
+ return ret;
+ }
- if (stc_pool->resource[0]) {
- ret = hws_debug_dump_context_stc_resource(f, ctx, table_type,
- stc_pool->resource[0]);
- if (ret)
- return ret;
- }
+ if (stc_pool->mirror_resource) {
+ struct mlx5hws_pool_resource *res = stc_pool->mirror_resource;
- if (i == MLX5HWS_TABLE_TYPE_FDB && stc_pool->mirror_resource[0]) {
- ret = hws_debug_dump_context_stc_resource(f, ctx, table_type,
- stc_pool->mirror_resource[0]);
- if (ret)
- return ret;
- }
+ ret = hws_debug_dump_context_stc_resource(f, ctx, res);
+ if (ret)
+ return ret;
}
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_definer.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_definer.h
index 2f6a7df4021c..5d186350dc08 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_definer.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_definer.h
@@ -785,7 +785,7 @@ struct mlx5hws_definer_cache {
struct mlx5hws_definer_cache_item {
struct mlx5hws_definer definer;
- u32 refcount;
+ u32 refcount; /* protected by context ctrl lock */
struct list_head list_node;
};
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_internal.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_internal.h
index 5643be1cd5bf..73511a5ab02a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_internal.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_internal.h
@@ -39,7 +39,6 @@
#define mlx5hws_dbg(ctx, arg...) mlx5_core_dbg((ctx)->mdev, ##arg)
#define MLX5HWS_TABLE_TYPE_BASE 2
-#define MLX5HWS_ACTION_STE_IDX_ANY 0
static inline bool is_mem_zero(const u8 *mem, size_t size)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.c
index ce541c60c5b4..21d27fad5a33 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.c
@@ -200,7 +200,7 @@ static void hws_matcher_set_rtc_attr_sz(struct mlx5hws_matcher *matcher,
enum mlx5hws_matcher_rtc_type rtc_type,
bool is_mirror)
{
- struct mlx5hws_pool_chunk *ste = &matcher->action_ste[MLX5HWS_ACTION_STE_IDX_ANY].ste;
+ struct mlx5hws_pool_chunk *ste = &matcher->action_ste.ste;
enum mlx5hws_matcher_flow_src flow_src = matcher->attr.optimize_flow_src;
bool is_match_rtc = rtc_type == HWS_MATCHER_RTC_TYPE_MATCH;
@@ -217,14 +217,12 @@ static void hws_matcher_set_rtc_attr_sz(struct mlx5hws_matcher *matcher,
}
static int hws_matcher_create_rtc(struct mlx5hws_matcher *matcher,
- enum mlx5hws_matcher_rtc_type rtc_type,
- u8 action_ste_selector)
+ enum mlx5hws_matcher_rtc_type rtc_type)
{
struct mlx5hws_matcher_attr *attr = &matcher->attr;
struct mlx5hws_cmd_rtc_create_attr rtc_attr = {0};
struct mlx5hws_match_template *mt = matcher->mt;
struct mlx5hws_context *ctx = matcher->tbl->ctx;
- struct mlx5hws_action_default_stc *default_stc;
struct mlx5hws_matcher_action_ste *action_ste;
struct mlx5hws_table *tbl = matcher->tbl;
struct mlx5hws_pool *ste_pool, *stc_pool;
@@ -267,18 +265,10 @@ static int hws_matcher_create_rtc(struct mlx5hws_matcher *matcher,
rtc_attr.match_definer_0 = ctx->caps->linear_match_definer;
}
}
-
- /* Match pool requires implicit allocation */
- ret = mlx5hws_pool_chunk_alloc(ste_pool, ste);
- if (ret) {
- mlx5hws_err(ctx, "Failed to allocate STE for %s RTC",
- hws_matcher_rtc_type_to_str(rtc_type));
- return ret;
- }
break;
case HWS_MATCHER_RTC_TYPE_STE_ARRAY:
- action_ste = &matcher->action_ste[action_ste_selector];
+ action_ste = &matcher->action_ste;
rtc_0_id = &action_ste->rtc_0_id;
rtc_1_id = &action_ste->rtc_1_id;
@@ -300,7 +290,7 @@ static int hws_matcher_create_rtc(struct mlx5hws_matcher *matcher,
return -EINVAL;
}
- obj_id = mlx5hws_pool_chunk_get_base_id(ste_pool, ste);
+ obj_id = mlx5hws_pool_get_base_id(ste_pool);
rtc_attr.pd = ctx->pd_num;
rtc_attr.ste_base = obj_id;
@@ -310,9 +300,8 @@ static int hws_matcher_create_rtc(struct mlx5hws_matcher *matcher,
hws_matcher_set_rtc_attr_sz(matcher, &rtc_attr, rtc_type, false);
/* STC is a single resource (obj_id), use any STC for the ID */
- stc_pool = ctx->stc_pool[tbl->type];
- default_stc = ctx->common_res[tbl->type].default_stc;
- obj_id = mlx5hws_pool_chunk_get_base_id(stc_pool, &default_stc->default_hit);
+ stc_pool = ctx->stc_pool;
+ obj_id = mlx5hws_pool_get_base_id(stc_pool);
rtc_attr.stc_base = obj_id;
ret = mlx5hws_cmd_rtc_create(ctx->mdev, &rtc_attr, rtc_0_id);
@@ -323,11 +312,11 @@ static int hws_matcher_create_rtc(struct mlx5hws_matcher *matcher,
}
if (tbl->type == MLX5HWS_TABLE_TYPE_FDB) {
- obj_id = mlx5hws_pool_chunk_get_base_mirror_id(ste_pool, ste);
+ obj_id = mlx5hws_pool_get_base_mirror_id(ste_pool);
rtc_attr.ste_base = obj_id;
rtc_attr.table_type = mlx5hws_table_get_res_fw_ft_type(tbl->type, true);
- obj_id = mlx5hws_pool_chunk_get_base_mirror_id(stc_pool, &default_stc->default_hit);
+ obj_id = mlx5hws_pool_get_base_mirror_id(stc_pool);
rtc_attr.stc_base = obj_id;
hws_matcher_set_rtc_attr_sz(matcher, &rtc_attr, rtc_type, true);
@@ -350,28 +339,21 @@ static int hws_matcher_create_rtc(struct mlx5hws_matcher *matcher,
}
static void hws_matcher_destroy_rtc(struct mlx5hws_matcher *matcher,
- enum mlx5hws_matcher_rtc_type rtc_type,
- u8 action_ste_selector)
+ enum mlx5hws_matcher_rtc_type rtc_type)
{
struct mlx5hws_matcher_action_ste *action_ste;
struct mlx5hws_table *tbl = matcher->tbl;
- struct mlx5hws_pool_chunk *ste;
- struct mlx5hws_pool *ste_pool;
u32 rtc_0_id, rtc_1_id;
switch (rtc_type) {
case HWS_MATCHER_RTC_TYPE_MATCH:
rtc_0_id = matcher->match_ste.rtc_0_id;
rtc_1_id = matcher->match_ste.rtc_1_id;
- ste_pool = matcher->match_ste.pool;
- ste = &matcher->match_ste.ste;
break;
case HWS_MATCHER_RTC_TYPE_STE_ARRAY:
- action_ste = &matcher->action_ste[action_ste_selector];
+ action_ste = &matcher->action_ste;
rtc_0_id = action_ste->rtc_0_id;
rtc_1_id = action_ste->rtc_1_id;
- ste_pool = action_ste->pool;
- ste = &action_ste->ste;
break;
default:
return;
@@ -381,8 +363,6 @@ static void hws_matcher_destroy_rtc(struct mlx5hws_matcher *matcher,
mlx5hws_cmd_rtc_destroy(matcher->tbl->ctx->mdev, rtc_1_id);
mlx5hws_cmd_rtc_destroy(matcher->tbl->ctx->mdev, rtc_0_id);
- if (rtc_type == HWS_MATCHER_RTC_TYPE_MATCH)
- mlx5hws_pool_chunk_free(ste_pool, ste);
}
static int
@@ -458,20 +438,13 @@ static int hws_matcher_resize_init(struct mlx5hws_matcher *src_matcher)
if (!resize_data)
return -ENOMEM;
- resize_data->max_stes = src_matcher->action_ste[MLX5HWS_ACTION_STE_IDX_ANY].max_stes;
-
- resize_data->action_ste[0].stc = src_matcher->action_ste[0].stc;
- resize_data->action_ste[0].rtc_0_id = src_matcher->action_ste[0].rtc_0_id;
- resize_data->action_ste[0].rtc_1_id = src_matcher->action_ste[0].rtc_1_id;
- resize_data->action_ste[0].pool = src_matcher->action_ste[0].max_stes ?
- src_matcher->action_ste[0].pool :
- NULL;
- resize_data->action_ste[1].stc = src_matcher->action_ste[1].stc;
- resize_data->action_ste[1].rtc_0_id = src_matcher->action_ste[1].rtc_0_id;
- resize_data->action_ste[1].rtc_1_id = src_matcher->action_ste[1].rtc_1_id;
- resize_data->action_ste[1].pool = src_matcher->action_ste[1].max_stes ?
- src_matcher->action_ste[1].pool :
- NULL;
+ resize_data->max_stes = src_matcher->action_ste.max_stes;
+
+ resize_data->stc = src_matcher->action_ste.stc;
+ resize_data->rtc_0_id = src_matcher->action_ste.rtc_0_id;
+ resize_data->rtc_1_id = src_matcher->action_ste.rtc_1_id;
+ resize_data->pool = src_matcher->action_ste.max_stes ?
+ src_matcher->action_ste.pool : NULL;
/* Place the new resized matcher on the dst matcher's list */
list_add(&resize_data->list_node, &src_matcher->resize_dst->resize_data);
@@ -504,47 +477,65 @@ static void hws_matcher_resize_uninit(struct mlx5hws_matcher *matcher)
if (resize_data->max_stes) {
mlx5hws_action_free_single_stc(matcher->tbl->ctx,
matcher->tbl->type,
- &resize_data->action_ste[1].stc);
- mlx5hws_action_free_single_stc(matcher->tbl->ctx,
- matcher->tbl->type,
- &resize_data->action_ste[0].stc);
+ &resize_data->stc);
- if (matcher->tbl->type == MLX5HWS_TABLE_TYPE_FDB) {
- mlx5hws_cmd_rtc_destroy(matcher->tbl->ctx->mdev,
- resize_data->action_ste[1].rtc_1_id);
+ if (matcher->tbl->type == MLX5HWS_TABLE_TYPE_FDB)
mlx5hws_cmd_rtc_destroy(matcher->tbl->ctx->mdev,
- resize_data->action_ste[0].rtc_1_id);
- }
- mlx5hws_cmd_rtc_destroy(matcher->tbl->ctx->mdev,
- resize_data->action_ste[1].rtc_0_id);
+ resize_data->rtc_1_id);
+
mlx5hws_cmd_rtc_destroy(matcher->tbl->ctx->mdev,
- resize_data->action_ste[0].rtc_0_id);
- if (resize_data->action_ste[MLX5HWS_ACTION_STE_IDX_ANY].pool) {
- mlx5hws_pool_destroy(resize_data->action_ste[1].pool);
- mlx5hws_pool_destroy(resize_data->action_ste[0].pool);
- }
+ resize_data->rtc_0_id);
+
+ if (resize_data->pool)
+ mlx5hws_pool_destroy(resize_data->pool);
}
kfree(resize_data);
}
}
-static int
-hws_matcher_bind_at_idx(struct mlx5hws_matcher *matcher, u8 action_ste_selector)
+static int hws_matcher_bind_at(struct mlx5hws_matcher *matcher)
{
+ bool is_jumbo = mlx5hws_matcher_mt_is_jumbo(matcher->mt);
struct mlx5hws_cmd_stc_modify_attr stc_attr = {0};
struct mlx5hws_matcher_action_ste *action_ste;
struct mlx5hws_table *tbl = matcher->tbl;
struct mlx5hws_pool_attr pool_attr = {0};
struct mlx5hws_context *ctx = tbl->ctx;
- int ret;
+ u32 required_stes;
+ u8 max_stes = 0;
+ int i, ret;
+
+ if (matcher->flags & MLX5HWS_MATCHER_FLAGS_COLLISION)
+ return 0;
+
+ for (i = 0; i < matcher->num_of_at; i++) {
+ struct mlx5hws_action_template *at = &matcher->at[i];
+
+ ret = hws_matcher_check_and_process_at(matcher, at);
+ if (ret) {
+ mlx5hws_err(ctx, "Invalid at %d", i);
+ return ret;
+ }
+
+ required_stes = at->num_of_action_stes - (!is_jumbo || at->only_term);
+ max_stes = max(max_stes, required_stes);
+
+ /* Future: Optimize reparse */
+ }
+
+ /* There are no additional STEs required for matcher */
+ if (!max_stes)
+ return 0;
+
+ matcher->action_ste.max_stes = max_stes;
- action_ste = &matcher->action_ste[action_ste_selector];
+ action_ste = &matcher->action_ste;
/* Allocate action STE mempool */
pool_attr.table_type = tbl->type;
pool_attr.pool_type = MLX5HWS_POOL_TYPE_STE;
- pool_attr.flags = MLX5HWS_POOL_FLAGS_FOR_STE_ACTION_POOL;
+ pool_attr.flags = MLX5HWS_POOL_FLAG_BUDDY;
pool_attr.alloc_log_sz = ilog2(roundup_pow_of_two(action_ste->max_stes)) +
matcher->attr.table.sz_row_log;
hws_matcher_set_pool_attr(&pool_attr, matcher);
@@ -555,7 +546,7 @@ hws_matcher_bind_at_idx(struct mlx5hws_matcher *matcher, u8 action_ste_selector)
}
/* Allocate action RTC */
- ret = hws_matcher_create_rtc(matcher, HWS_MATCHER_RTC_TYPE_STE_ARRAY, action_ste_selector);
+ ret = hws_matcher_create_rtc(matcher, HWS_MATCHER_RTC_TYPE_STE_ARRAY);
if (ret) {
mlx5hws_err(ctx, "Failed to create action RTC\n");
goto free_ste_pool;
@@ -579,18 +570,18 @@ hws_matcher_bind_at_idx(struct mlx5hws_matcher *matcher, u8 action_ste_selector)
return 0;
free_rtc:
- hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_STE_ARRAY, action_ste_selector);
+ hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_STE_ARRAY);
free_ste_pool:
mlx5hws_pool_destroy(action_ste->pool);
return ret;
}
-static void hws_matcher_unbind_at_idx(struct mlx5hws_matcher *matcher, u8 action_ste_selector)
+static void hws_matcher_unbind_at(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_matcher_action_ste *action_ste;
struct mlx5hws_table *tbl = matcher->tbl;
- action_ste = &matcher->action_ste[action_ste_selector];
+ action_ste = &matcher->action_ste;
if (!action_ste->max_stes ||
matcher->flags & MLX5HWS_MATCHER_FLAGS_COLLISION ||
@@ -598,65 +589,10 @@ static void hws_matcher_unbind_at_idx(struct mlx5hws_matcher *matcher, u8 action
return;
mlx5hws_action_free_single_stc(tbl->ctx, tbl->type, &action_ste->stc);
- hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_STE_ARRAY, action_ste_selector);
+ hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_STE_ARRAY);
mlx5hws_pool_destroy(action_ste->pool);
}
-static int hws_matcher_bind_at(struct mlx5hws_matcher *matcher)
-{
- bool is_jumbo = mlx5hws_matcher_mt_is_jumbo(matcher->mt);
- struct mlx5hws_table *tbl = matcher->tbl;
- struct mlx5hws_context *ctx = tbl->ctx;
- u32 required_stes;
- u8 max_stes = 0;
- int i, ret;
-
- if (matcher->flags & MLX5HWS_MATCHER_FLAGS_COLLISION)
- return 0;
-
- for (i = 0; i < matcher->num_of_at; i++) {
- struct mlx5hws_action_template *at = &matcher->at[i];
-
- ret = hws_matcher_check_and_process_at(matcher, at);
- if (ret) {
- mlx5hws_err(ctx, "Invalid at %d", i);
- return ret;
- }
-
- required_stes = at->num_of_action_stes - (!is_jumbo || at->only_term);
- max_stes = max(max_stes, required_stes);
-
- /* Future: Optimize reparse */
- }
-
- /* There are no additional STEs required for matcher */
- if (!max_stes)
- return 0;
-
- matcher->action_ste[0].max_stes = max_stes;
- matcher->action_ste[1].max_stes = max_stes;
-
- ret = hws_matcher_bind_at_idx(matcher, 0);
- if (ret)
- return ret;
-
- ret = hws_matcher_bind_at_idx(matcher, 1);
- if (ret)
- goto free_at_0;
-
- return 0;
-
-free_at_0:
- hws_matcher_unbind_at_idx(matcher, 0);
- return ret;
-}
-
-static void hws_matcher_unbind_at(struct mlx5hws_matcher *matcher)
-{
- hws_matcher_unbind_at_idx(matcher, 1);
- hws_matcher_unbind_at_idx(matcher, 0);
-}
-
static int hws_matcher_bind_mt(struct mlx5hws_matcher *matcher)
{
struct mlx5hws_context *ctx = matcher->tbl->ctx;
@@ -676,7 +612,6 @@ static int hws_matcher_bind_mt(struct mlx5hws_matcher *matcher)
/* Create an STE pool per matcher*/
pool_attr.table_type = matcher->tbl->type;
pool_attr.pool_type = MLX5HWS_POOL_TYPE_STE;
- pool_attr.flags = MLX5HWS_POOL_FLAGS_FOR_MATCHER_STE_POOL;
pool_attr.alloc_log_sz = matcher->attr.table.sz_col_log +
matcher->attr.table.sz_row_log;
hws_matcher_set_pool_attr(&pool_attr, matcher);
@@ -802,7 +737,7 @@ static int hws_matcher_create_and_connect(struct mlx5hws_matcher *matcher)
goto unbind_at;
/* Allocate the RTC for the new matcher */
- ret = hws_matcher_create_rtc(matcher, HWS_MATCHER_RTC_TYPE_MATCH, 0);
+ ret = hws_matcher_create_rtc(matcher, HWS_MATCHER_RTC_TYPE_MATCH);
if (ret)
goto destroy_end_ft;
@@ -814,7 +749,7 @@ static int hws_matcher_create_and_connect(struct mlx5hws_matcher *matcher)
return 0;
destroy_rtc:
- hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_MATCH, 0);
+ hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_MATCH);
destroy_end_ft:
hws_matcher_destroy_end_ft(matcher);
unbind_at:
@@ -828,7 +763,7 @@ static void hws_matcher_destroy_and_disconnect(struct mlx5hws_matcher *matcher)
{
hws_matcher_resize_uninit(matcher);
hws_matcher_disconnect(matcher);
- hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_MATCH, 0);
+ hws_matcher_destroy_rtc(matcher, HWS_MATCHER_RTC_TYPE_MATCH);
hws_matcher_destroy_end_ft(matcher);
hws_matcher_unbind_at(matcher);
hws_matcher_unbind_mt(matcher);
@@ -962,10 +897,9 @@ int mlx5hws_matcher_attach_at(struct mlx5hws_matcher *matcher,
return ret;
required_stes = at->num_of_action_stes - (!is_jumbo || at->only_term);
- if (matcher->action_ste[MLX5HWS_ACTION_STE_IDX_ANY].max_stes < required_stes) {
+ if (matcher->action_ste.max_stes < required_stes) {
mlx5hws_dbg(ctx, "Required STEs [%d] exceeds initial action template STE [%d]\n",
- required_stes,
- matcher->action_ste[MLX5HWS_ACTION_STE_IDX_ANY].max_stes);
+ required_stes, matcher->action_ste.max_stes);
return -ENOMEM;
}
@@ -1149,8 +1083,7 @@ static int hws_matcher_resize_precheck(struct mlx5hws_matcher *src_matcher,
return -EINVAL;
}
- if (src_matcher->action_ste[MLX5HWS_ACTION_STE_IDX_ANY].max_stes >
- dst_matcher->action_ste[0].max_stes) {
+ if (src_matcher->action_ste.max_stes > dst_matcher->action_ste.max_stes) {
mlx5hws_err(ctx, "Src/dst matcher max STEs mismatch\n");
return -EINVAL;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.h
index 125391d1a114..b9075919a4c9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_matcher.h
@@ -52,15 +52,11 @@ struct mlx5hws_matcher_action_ste {
u8 max_stes;
};
-struct mlx5hws_matcher_resize_data_node {
+struct mlx5hws_matcher_resize_data {
struct mlx5hws_pool_chunk stc;
u32 rtc_0_id;
u32 rtc_1_id;
struct mlx5hws_pool *pool;
-};
-
-struct mlx5hws_matcher_resize_data {
- struct mlx5hws_matcher_resize_data_node action_ste[2];
u8 max_stes;
struct list_head list_node;
};
@@ -78,7 +74,7 @@ struct mlx5hws_matcher {
struct mlx5hws_matcher *col_matcher;
struct mlx5hws_matcher *resize_dst;
struct mlx5hws_matcher_match_ste match_ste;
- struct mlx5hws_matcher_action_ste action_ste[2];
+ struct mlx5hws_matcher_action_ste action_ste;
struct list_head list_node;
struct list_head resize_data;
};
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pat_arg.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pat_arg.h
index 27ca93385b08..8ddb51980044 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pat_arg.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pat_arg.h
@@ -31,7 +31,7 @@ struct mlx5hws_pattern_cache_item {
u8 *data;
u16 num_of_actions;
} mh_data;
- u32 refcount;
+ u32 refcount; /* protected by pattern_cache lock */
struct list_head ptrn_list_node;
};
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.c
index a8a63e3278be..a6f542704fe4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.c
@@ -20,15 +20,14 @@ static void hws_pool_free_one_resource(struct mlx5hws_pool_resource *resource)
kfree(resource);
}
-static void hws_pool_resource_free(struct mlx5hws_pool *pool,
- int resource_idx)
+static void hws_pool_resource_free(struct mlx5hws_pool *pool)
{
- hws_pool_free_one_resource(pool->resource[resource_idx]);
- pool->resource[resource_idx] = NULL;
+ hws_pool_free_one_resource(pool->resource);
+ pool->resource = NULL;
if (pool->tbl_type == MLX5HWS_TABLE_TYPE_FDB) {
- hws_pool_free_one_resource(pool->mirror_resource[resource_idx]);
- pool->mirror_resource[resource_idx] = NULL;
+ hws_pool_free_one_resource(pool->mirror_resource);
+ pool->mirror_resource = NULL;
}
}
@@ -61,10 +60,8 @@ hws_pool_create_one_resource(struct mlx5hws_pool *pool, u32 log_range,
ret = -EINVAL;
}
- if (ret) {
- mlx5hws_err(pool->ctx, "Failed to allocate resource objects\n");
+ if (ret)
goto free_resource;
- }
resource->pool = pool;
resource->range = 1 << log_range;
@@ -77,199 +74,115 @@ hws_pool_create_one_resource(struct mlx5hws_pool *pool, u32 log_range,
return NULL;
}
-static int
-hws_pool_resource_alloc(struct mlx5hws_pool *pool, u32 log_range, int idx)
+static int hws_pool_resource_alloc(struct mlx5hws_pool *pool)
{
struct mlx5hws_pool_resource *resource;
u32 fw_ft_type, opt_log_range;
fw_ft_type = mlx5hws_table_get_res_fw_ft_type(pool->tbl_type, false);
- opt_log_range = pool->opt_type == MLX5HWS_POOL_OPTIMIZE_ORIG ? 0 : log_range;
+ opt_log_range = pool->opt_type == MLX5HWS_POOL_OPTIMIZE_ORIG ?
+ 0 : pool->alloc_log_sz;
resource = hws_pool_create_one_resource(pool, opt_log_range, fw_ft_type);
if (!resource) {
- mlx5hws_err(pool->ctx, "Failed allocating resource\n");
+ mlx5hws_err(pool->ctx, "Failed to allocate resource\n");
return -EINVAL;
}
- pool->resource[idx] = resource;
+ pool->resource = resource;
if (pool->tbl_type == MLX5HWS_TABLE_TYPE_FDB) {
struct mlx5hws_pool_resource *mirror_resource;
fw_ft_type = mlx5hws_table_get_res_fw_ft_type(pool->tbl_type, true);
- opt_log_range = pool->opt_type == MLX5HWS_POOL_OPTIMIZE_MIRROR ? 0 : log_range;
+ opt_log_range = pool->opt_type == MLX5HWS_POOL_OPTIMIZE_MIRROR ?
+ 0 : pool->alloc_log_sz;
mirror_resource = hws_pool_create_one_resource(pool, opt_log_range, fw_ft_type);
if (!mirror_resource) {
- mlx5hws_err(pool->ctx, "Failed allocating mirrored resource\n");
+ mlx5hws_err(pool->ctx, "Failed to allocate mirrored resource\n");
hws_pool_free_one_resource(resource);
- pool->resource[idx] = NULL;
+ pool->resource = NULL;
return -EINVAL;
}
- pool->mirror_resource[idx] = mirror_resource;
+ pool->mirror_resource = mirror_resource;
}
return 0;
}
-static unsigned long *hws_pool_create_and_init_bitmap(u32 log_range)
-{
- unsigned long *cur_bmp;
-
- cur_bmp = bitmap_zalloc(1 << log_range, GFP_KERNEL);
- if (!cur_bmp)
- return NULL;
-
- bitmap_fill(cur_bmp, 1 << log_range);
-
- return cur_bmp;
-}
-
-static void hws_pool_buddy_db_put_chunk(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
+static int hws_pool_buddy_init(struct mlx5hws_pool *pool)
{
struct mlx5hws_buddy_mem *buddy;
- buddy = pool->db.buddy_manager->buddies[chunk->resource_idx];
- if (!buddy) {
- mlx5hws_err(pool->ctx, "No such buddy (%d)\n", chunk->resource_idx);
- return;
- }
-
- mlx5hws_buddy_free_mem(buddy, chunk->offset, chunk->order);
-}
-
-static struct mlx5hws_buddy_mem *
-hws_pool_buddy_get_next_buddy(struct mlx5hws_pool *pool, int idx,
- u32 order, bool *is_new_buddy)
-{
- static struct mlx5hws_buddy_mem *buddy;
- u32 new_buddy_size;
-
- buddy = pool->db.buddy_manager->buddies[idx];
- if (buddy)
- return buddy;
-
- new_buddy_size = max(pool->alloc_log_sz, order);
- *is_new_buddy = true;
- buddy = mlx5hws_buddy_create(new_buddy_size);
+ buddy = mlx5hws_buddy_create(pool->alloc_log_sz);
if (!buddy) {
- mlx5hws_err(pool->ctx, "Failed to create buddy order: %d index: %d\n",
- new_buddy_size, idx);
- return NULL;
+ mlx5hws_err(pool->ctx, "Failed to create buddy order: %zu\n",
+ pool->alloc_log_sz);
+ return -ENOMEM;
}
- if (hws_pool_resource_alloc(pool, new_buddy_size, idx) != 0) {
- mlx5hws_err(pool->ctx, "Failed to create resource type: %d: size %d index: %d\n",
- pool->type, new_buddy_size, idx);
+ if (hws_pool_resource_alloc(pool) != 0) {
+ mlx5hws_err(pool->ctx, "Failed to create resource type: %d size %zu\n",
+ pool->type, pool->alloc_log_sz);
mlx5hws_buddy_cleanup(buddy);
- return NULL;
+ kfree(buddy);
+ return -ENOMEM;
}
- pool->db.buddy_manager->buddies[idx] = buddy;
+ pool->db.buddy = buddy;
- return buddy;
+ return 0;
}
-static int hws_pool_buddy_get_mem_chunk(struct mlx5hws_pool *pool,
- int order,
- u32 *buddy_idx,
- int *seg)
+static int hws_pool_buddy_db_get_chunk(struct mlx5hws_pool *pool,
+ struct mlx5hws_pool_chunk *chunk)
{
- struct mlx5hws_buddy_mem *buddy;
- bool new_mem = false;
- int ret = 0;
- int i;
-
- *seg = -1;
-
- /* Find the next free place from the buddy array */
- while (*seg == -1) {
- for (i = 0; i < MLX5HWS_POOL_RESOURCE_ARR_SZ; i++) {
- buddy = hws_pool_buddy_get_next_buddy(pool, i,
- order,
- &new_mem);
- if (!buddy) {
- ret = -ENOMEM;
- goto out;
- }
-
- *seg = mlx5hws_buddy_alloc_mem(buddy, order);
- if (*seg != -1)
- goto found;
-
- if (pool->flags & MLX5HWS_POOL_FLAGS_ONE_RESOURCE) {
- mlx5hws_err(pool->ctx,
- "Fail to allocate seg for one resource pool\n");
- ret = -ENOMEM;
- goto out;
- }
-
- if (new_mem) {
- /* We have new memory pool, should be place for us */
- mlx5hws_err(pool->ctx,
- "No memory for order: %d with buddy no: %d\n",
- order, i);
- ret = -ENOMEM;
- goto out;
- }
- }
+ struct mlx5hws_buddy_mem *buddy = pool->db.buddy;
+
+ if (!buddy) {
+ mlx5hws_err(pool->ctx, "Bad buddy state\n");
+ return -EINVAL;
}
-found:
- *buddy_idx = i;
-out:
- return ret;
+ chunk->offset = mlx5hws_buddy_alloc_mem(buddy, chunk->order);
+ if (chunk->offset >= 0)
+ return 0;
+
+ return -ENOMEM;
}
-static int hws_pool_buddy_db_get_chunk(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
+static void hws_pool_buddy_db_put_chunk(struct mlx5hws_pool *pool,
+ struct mlx5hws_pool_chunk *chunk)
{
- int ret = 0;
+ struct mlx5hws_buddy_mem *buddy;
- /* Go over the buddies and find next free slot */
- ret = hws_pool_buddy_get_mem_chunk(pool, chunk->order,
- &chunk->resource_idx,
- &chunk->offset);
- if (ret)
- mlx5hws_err(pool->ctx, "Failed to get free slot for chunk with order: %d\n",
- chunk->order);
+ buddy = pool->db.buddy;
+ if (!buddy) {
+ mlx5hws_err(pool->ctx, "Bad buddy state\n");
+ return;
+ }
- return ret;
+ mlx5hws_buddy_free_mem(buddy, chunk->offset, chunk->order);
}
static void hws_pool_buddy_db_uninit(struct mlx5hws_pool *pool)
{
struct mlx5hws_buddy_mem *buddy;
- int i;
-
- for (i = 0; i < MLX5HWS_POOL_RESOURCE_ARR_SZ; i++) {
- buddy = pool->db.buddy_manager->buddies[i];
- if (buddy) {
- mlx5hws_buddy_cleanup(buddy);
- kfree(buddy);
- pool->db.buddy_manager->buddies[i] = NULL;
- }
- }
- kfree(pool->db.buddy_manager);
+ buddy = pool->db.buddy;
+ if (buddy) {
+ mlx5hws_buddy_cleanup(buddy);
+ kfree(buddy);
+ pool->db.buddy = NULL;
+ }
}
-static int hws_pool_buddy_db_init(struct mlx5hws_pool *pool, u32 log_range)
+static int hws_pool_buddy_db_init(struct mlx5hws_pool *pool)
{
- pool->db.buddy_manager = kzalloc(sizeof(*pool->db.buddy_manager), GFP_KERNEL);
- if (!pool->db.buddy_manager)
- return -ENOMEM;
-
- if (pool->flags & MLX5HWS_POOL_FLAGS_ALLOC_MEM_ON_CREATE) {
- bool new_buddy;
+ int ret;
- if (!hws_pool_buddy_get_next_buddy(pool, 0, log_range, &new_buddy)) {
- mlx5hws_err(pool->ctx,
- "Failed allocating memory on create log_sz: %d\n", log_range);
- kfree(pool->db.buddy_manager);
- return -ENOMEM;
- }
- }
+ ret = hws_pool_buddy_init(pool);
+ if (ret)
+ return ret;
pool->p_db_uninit = &hws_pool_buddy_db_uninit;
pool->p_get_chunk = &hws_pool_buddy_db_get_chunk;
@@ -278,261 +191,105 @@ static int hws_pool_buddy_db_init(struct mlx5hws_pool *pool, u32 log_range)
return 0;
}
-static int hws_pool_create_resource_on_index(struct mlx5hws_pool *pool,
- u32 alloc_size, int idx)
-{
- int ret = hws_pool_resource_alloc(pool, alloc_size, idx);
-
- if (ret) {
- mlx5hws_err(pool->ctx, "Failed to create resource type: %d: size %d index: %d\n",
- pool->type, alloc_size, idx);
- return ret;
- }
-
- return 0;
-}
-
-static struct mlx5hws_pool_elements *
-hws_pool_element_create_new_elem(struct mlx5hws_pool *pool, u32 order, int idx)
+static unsigned long *hws_pool_create_and_init_bitmap(u32 log_range)
{
- struct mlx5hws_pool_elements *elem;
- u32 alloc_size;
-
- alloc_size = pool->alloc_log_sz;
+ unsigned long *bitmap;
- elem = kzalloc(sizeof(*elem), GFP_KERNEL);
- if (!elem)
+ bitmap = bitmap_zalloc(1 << log_range, GFP_KERNEL);
+ if (!bitmap)
return NULL;
- /* Sharing the same resource, also means that all the elements are with size 1 */
- if ((pool->flags & MLX5HWS_POOL_FLAGS_FIXED_SIZE_OBJECTS) &&
- !(pool->flags & MLX5HWS_POOL_FLAGS_RESOURCE_PER_CHUNK)) {
- /* Currently all chunks in size 1 */
- elem->bitmap = hws_pool_create_and_init_bitmap(alloc_size - order);
- if (!elem->bitmap) {
- mlx5hws_err(pool->ctx,
- "Failed to create bitmap type: %d: size %d index: %d\n",
- pool->type, alloc_size, idx);
- goto free_elem;
- }
-
- elem->log_size = alloc_size - order;
- }
-
- if (hws_pool_create_resource_on_index(pool, alloc_size, idx)) {
- mlx5hws_err(pool->ctx, "Failed to create resource type: %d: size %d index: %d\n",
- pool->type, alloc_size, idx);
- goto free_db;
- }
-
- pool->db.element_manager->elements[idx] = elem;
+ bitmap_fill(bitmap, 1 << log_range);
- return elem;
-
-free_db:
- bitmap_free(elem->bitmap);
-free_elem:
- kfree(elem);
- return NULL;
+ return bitmap;
}
-static int hws_pool_element_find_seg(struct mlx5hws_pool_elements *elem, int *seg)
+static int hws_pool_bitmap_init(struct mlx5hws_pool *pool)
{
- unsigned int segment, size;
+ unsigned long *bitmap;
- size = 1 << elem->log_size;
-
- segment = find_first_bit(elem->bitmap, size);
- if (segment >= size) {
- elem->is_full = true;
+ bitmap = hws_pool_create_and_init_bitmap(pool->alloc_log_sz);
+ if (!bitmap) {
+ mlx5hws_err(pool->ctx, "Failed to create bitmap order: %zu\n",
+ pool->alloc_log_sz);
return -ENOMEM;
}
- bitmap_clear(elem->bitmap, segment, 1);
- *seg = segment;
- return 0;
-}
-
-static int
-hws_pool_onesize_element_get_mem_chunk(struct mlx5hws_pool *pool, u32 order,
- u32 *idx, int *seg)
-{
- struct mlx5hws_pool_elements *elem;
-
- elem = pool->db.element_manager->elements[0];
- if (!elem)
- elem = hws_pool_element_create_new_elem(pool, order, 0);
- if (!elem)
- goto err_no_elem;
-
- if (hws_pool_element_find_seg(elem, seg) != 0) {
- mlx5hws_err(pool->ctx, "No more resources (last request order: %d)\n", order);
+ if (hws_pool_resource_alloc(pool) != 0) {
+ mlx5hws_err(pool->ctx, "Failed to create resource type: %d: size %zu\n",
+ pool->type, pool->alloc_log_sz);
+ bitmap_free(bitmap);
return -ENOMEM;
}
- *idx = 0;
- elem->num_of_elements++;
- return 0;
-
-err_no_elem:
- mlx5hws_err(pool->ctx, "Failed to allocate element for order: %d\n", order);
- return -ENOMEM;
-}
-
-static int
-hws_pool_general_element_get_mem_chunk(struct mlx5hws_pool *pool, u32 order,
- u32 *idx, int *seg)
-{
- int ret, i;
-
- for (i = 0; i < MLX5HWS_POOL_RESOURCE_ARR_SZ; i++) {
- if (!pool->resource[i]) {
- ret = hws_pool_create_resource_on_index(pool, order, i);
- if (ret)
- goto err_no_res;
- *idx = i;
- *seg = 0; /* One memory slot in that element */
- return 0;
- }
- }
+ pool->db.bitmap = bitmap;
- mlx5hws_err(pool->ctx, "No more resources (last request order: %d)\n", order);
- return -ENOMEM;
-
-err_no_res:
- mlx5hws_err(pool->ctx, "Failed to allocate element for order: %d\n", order);
- return -ENOMEM;
+ return 0;
}
-static int hws_pool_general_element_db_get_chunk(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
+static int hws_pool_bitmap_db_get_chunk(struct mlx5hws_pool *pool,
+ struct mlx5hws_pool_chunk *chunk)
{
- int ret;
-
- /* Go over all memory elements and find/allocate free slot */
- ret = hws_pool_general_element_get_mem_chunk(pool, chunk->order,
- &chunk->resource_idx,
- &chunk->offset);
- if (ret)
- mlx5hws_err(pool->ctx, "Failed to get free slot for chunk with order: %d\n",
- chunk->order);
+ unsigned long *bitmap, size;
- return ret;
-}
+ if (chunk->order != 0) {
+ mlx5hws_err(pool->ctx, "Pool only supports order 0 allocs\n");
+ return -EINVAL;
+ }
-static void hws_pool_general_element_db_put_chunk(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
-{
- if (unlikely(!pool->resource[chunk->resource_idx]))
- pr_warn("HWS: invalid resource with index %d\n", chunk->resource_idx);
+ bitmap = pool->db.bitmap;
+ if (!bitmap) {
+ mlx5hws_err(pool->ctx, "Bad bitmap state\n");
+ return -EINVAL;
+ }
- if (pool->flags & MLX5HWS_POOL_FLAGS_RELEASE_FREE_RESOURCE)
- hws_pool_resource_free(pool, chunk->resource_idx);
-}
+ size = 1 << pool->alloc_log_sz;
-static void hws_pool_general_element_db_uninit(struct mlx5hws_pool *pool)
-{
- (void)pool;
-}
+ chunk->offset = find_first_bit(bitmap, size);
+ if (chunk->offset >= size)
+ return -ENOMEM;
-/* This memory management works as the following:
- * - At start doesn't allocate no mem at all.
- * - When new request for chunk arrived:
- * allocate resource and give it.
- * - When free that chunk:
- * the resource is freed.
- */
-static int hws_pool_general_element_db_init(struct mlx5hws_pool *pool)
-{
- pool->p_db_uninit = &hws_pool_general_element_db_uninit;
- pool->p_get_chunk = &hws_pool_general_element_db_get_chunk;
- pool->p_put_chunk = &hws_pool_general_element_db_put_chunk;
+ bitmap_clear(bitmap, chunk->offset, 1);
return 0;
}
-static void hws_onesize_element_db_destroy_element(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_elements *elem,
- struct mlx5hws_pool_chunk *chunk)
-{
- if (unlikely(!pool->resource[chunk->resource_idx]))
- pr_warn("HWS: invalid resource with index %d\n", chunk->resource_idx);
-
- hws_pool_resource_free(pool, chunk->resource_idx);
- kfree(elem);
- pool->db.element_manager->elements[chunk->resource_idx] = NULL;
-}
-
-static void hws_onesize_element_db_put_chunk(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
+static void hws_pool_bitmap_db_put_chunk(struct mlx5hws_pool *pool,
+ struct mlx5hws_pool_chunk *chunk)
{
- struct mlx5hws_pool_elements *elem;
-
- if (unlikely(chunk->resource_idx))
- pr_warn("HWS: invalid resource with index %d\n", chunk->resource_idx);
+ unsigned long *bitmap;
- elem = pool->db.element_manager->elements[chunk->resource_idx];
- if (!elem) {
- mlx5hws_err(pool->ctx, "No such element (%d)\n", chunk->resource_idx);
+ bitmap = pool->db.bitmap;
+ if (!bitmap) {
+ mlx5hws_err(pool->ctx, "Bad bitmap state\n");
return;
}
- bitmap_set(elem->bitmap, chunk->offset, 1);
- elem->is_full = false;
- elem->num_of_elements--;
-
- if (pool->flags & MLX5HWS_POOL_FLAGS_RELEASE_FREE_RESOURCE &&
- !elem->num_of_elements)
- hws_onesize_element_db_destroy_element(pool, elem, chunk);
+ bitmap_set(bitmap, chunk->offset, 1);
}
-static int hws_onesize_element_db_get_chunk(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
+static void hws_pool_bitmap_db_uninit(struct mlx5hws_pool *pool)
{
- int ret = 0;
-
- /* Go over all memory elements and find/allocate free slot */
- ret = hws_pool_onesize_element_get_mem_chunk(pool, chunk->order,
- &chunk->resource_idx,
- &chunk->offset);
- if (ret)
- mlx5hws_err(pool->ctx, "Failed to get free slot for chunk with order: %d\n",
- chunk->order);
+ unsigned long *bitmap;
- return ret;
-}
-
-static void hws_onesize_element_db_uninit(struct mlx5hws_pool *pool)
-{
- struct mlx5hws_pool_elements *elem;
- int i;
-
- for (i = 0; i < MLX5HWS_POOL_RESOURCE_ARR_SZ; i++) {
- elem = pool->db.element_manager->elements[i];
- if (elem) {
- bitmap_free(elem->bitmap);
- kfree(elem);
- pool->db.element_manager->elements[i] = NULL;
- }
+ bitmap = pool->db.bitmap;
+ if (bitmap) {
+ bitmap_free(bitmap);
+ pool->db.bitmap = NULL;
}
- kfree(pool->db.element_manager);
}
-/* This memory management works as the following:
- * - At start doesn't allocate no mem at all.
- * - When new request for chunk arrived:
- * aloocate the first and only slot of memory/resource
- * when it ended return error.
- */
-static int hws_pool_onesize_element_db_init(struct mlx5hws_pool *pool)
+static int hws_pool_bitmap_db_init(struct mlx5hws_pool *pool)
{
- pool->db.element_manager = kzalloc(sizeof(*pool->db.element_manager), GFP_KERNEL);
- if (!pool->db.element_manager)
- return -ENOMEM;
+ int ret;
+
+ ret = hws_pool_bitmap_init(pool);
+ if (ret)
+ return ret;
- pool->p_db_uninit = &hws_onesize_element_db_uninit;
- pool->p_get_chunk = &hws_onesize_element_db_get_chunk;
- pool->p_put_chunk = &hws_onesize_element_db_put_chunk;
+ pool->p_db_uninit = &hws_pool_bitmap_db_uninit;
+ pool->p_get_chunk = &hws_pool_bitmap_db_get_chunk;
+ pool->p_put_chunk = &hws_pool_bitmap_db_put_chunk;
return 0;
}
@@ -542,15 +299,14 @@ static int hws_pool_db_init(struct mlx5hws_pool *pool,
{
int ret;
- if (db_type == MLX5HWS_POOL_DB_TYPE_GENERAL_SIZE)
- ret = hws_pool_general_element_db_init(pool);
- else if (db_type == MLX5HWS_POOL_DB_TYPE_ONE_SIZE_RESOURCE)
- ret = hws_pool_onesize_element_db_init(pool);
+ if (db_type == MLX5HWS_POOL_DB_TYPE_BITMAP)
+ ret = hws_pool_bitmap_db_init(pool);
else
- ret = hws_pool_buddy_db_init(pool, pool->alloc_log_sz);
+ ret = hws_pool_buddy_db_init(pool);
if (ret) {
- mlx5hws_err(pool->ctx, "Failed to init general db : %d (ret: %d)\n", db_type, ret);
+ mlx5hws_err(pool->ctx, "Failed to init pool type: %d (ret: %d)\n",
+ db_type, ret);
return ret;
}
@@ -599,15 +355,10 @@ mlx5hws_pool_create(struct mlx5hws_context *ctx, struct mlx5hws_pool_attr *pool_
pool->tbl_type = pool_attr->table_type;
pool->opt_type = pool_attr->opt_type;
- /* Support general db */
- if (pool->flags == (MLX5HWS_POOL_FLAGS_RELEASE_FREE_RESOURCE |
- MLX5HWS_POOL_FLAGS_RESOURCE_PER_CHUNK))
- res_db_type = MLX5HWS_POOL_DB_TYPE_GENERAL_SIZE;
- else if (pool->flags == (MLX5HWS_POOL_FLAGS_ONE_RESOURCE |
- MLX5HWS_POOL_FLAGS_FIXED_SIZE_OBJECTS))
- res_db_type = MLX5HWS_POOL_DB_TYPE_ONE_SIZE_RESOURCE;
- else
+ if (pool->flags & MLX5HWS_POOL_FLAG_BUDDY)
res_db_type = MLX5HWS_POOL_DB_TYPE_BUDDY;
+ else
+ res_db_type = MLX5HWS_POOL_DB_TYPE_BITMAP;
pool->alloc_log_sz = pool_attr->alloc_log_sz;
@@ -623,18 +374,14 @@ mlx5hws_pool_create(struct mlx5hws_context *ctx, struct mlx5hws_pool_attr *pool_
return NULL;
}
-int mlx5hws_pool_destroy(struct mlx5hws_pool *pool)
+void mlx5hws_pool_destroy(struct mlx5hws_pool *pool)
{
- int i;
-
mutex_destroy(&pool->lock);
- for (i = 0; i < MLX5HWS_POOL_RESOURCE_ARR_SZ; i++)
- if (pool->resource[i])
- hws_pool_resource_free(pool, i);
+ if (pool->resource)
+ hws_pool_resource_free(pool);
hws_pool_db_unint(pool);
kfree(pool);
- return 0;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.h
index 621298b352b2..9a781a87f097 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_pool.h
@@ -6,16 +6,12 @@
#define MLX5HWS_POOL_STC_LOG_SZ 15
-#define MLX5HWS_POOL_RESOURCE_ARR_SZ 100
-
enum mlx5hws_pool_type {
MLX5HWS_POOL_TYPE_STE,
MLX5HWS_POOL_TYPE_STC,
};
struct mlx5hws_pool_chunk {
- u32 resource_idx;
- /* Internal offset, relative to base index */
int offset;
int order;
};
@@ -27,29 +23,10 @@ struct mlx5hws_pool_resource {
};
enum mlx5hws_pool_flags {
- /* Only a one resource in that pool */
- MLX5HWS_POOL_FLAGS_ONE_RESOURCE = 1 << 0,
- MLX5HWS_POOL_FLAGS_RELEASE_FREE_RESOURCE = 1 << 1,
- /* No sharing resources between chunks */
- MLX5HWS_POOL_FLAGS_RESOURCE_PER_CHUNK = 1 << 2,
- /* All objects are in the same size */
- MLX5HWS_POOL_FLAGS_FIXED_SIZE_OBJECTS = 1 << 3,
- /* Managed by buddy allocator */
- MLX5HWS_POOL_FLAGS_BUDDY_MANAGED = 1 << 4,
- /* Allocate pool_type memory on pool creation */
- MLX5HWS_POOL_FLAGS_ALLOC_MEM_ON_CREATE = 1 << 5,
-
- /* These values should be used by the caller */
- MLX5HWS_POOL_FLAGS_FOR_STC_POOL =
- MLX5HWS_POOL_FLAGS_ONE_RESOURCE |
- MLX5HWS_POOL_FLAGS_FIXED_SIZE_OBJECTS,
- MLX5HWS_POOL_FLAGS_FOR_MATCHER_STE_POOL =
- MLX5HWS_POOL_FLAGS_RELEASE_FREE_RESOURCE |
- MLX5HWS_POOL_FLAGS_RESOURCE_PER_CHUNK,
- MLX5HWS_POOL_FLAGS_FOR_STE_ACTION_POOL =
- MLX5HWS_POOL_FLAGS_ONE_RESOURCE |
- MLX5HWS_POOL_FLAGS_BUDDY_MANAGED |
- MLX5HWS_POOL_FLAGS_ALLOC_MEM_ON_CREATE,
+ /* Managed by a buddy allocator. If this is not set only allocations of
+ * order 0 are supported.
+ */
+ MLX5HWS_POOL_FLAG_BUDDY = BIT(0),
};
enum mlx5hws_pool_optimize {
@@ -68,34 +45,17 @@ struct mlx5hws_pool_attr {
};
enum mlx5hws_db_type {
- /* Uses for allocating chunk of big memory, each element has its own resource in the FW*/
- MLX5HWS_POOL_DB_TYPE_GENERAL_SIZE,
- /* One resource only, all the elements are with same one size */
- MLX5HWS_POOL_DB_TYPE_ONE_SIZE_RESOURCE,
- /* Many resources, the memory allocated with buddy mechanism */
+ /* Uses a bitmap, supports only allocations of order 0. */
+ MLX5HWS_POOL_DB_TYPE_BITMAP,
+ /* Entries are managed using a buddy mechanism. */
MLX5HWS_POOL_DB_TYPE_BUDDY,
};
-struct mlx5hws_buddy_manager {
- struct mlx5hws_buddy_mem *buddies[MLX5HWS_POOL_RESOURCE_ARR_SZ];
-};
-
-struct mlx5hws_pool_elements {
- u32 num_of_elements;
- unsigned long *bitmap;
- u32 log_size;
- bool is_full;
-};
-
-struct mlx5hws_element_manager {
- struct mlx5hws_pool_elements *elements[MLX5HWS_POOL_RESOURCE_ARR_SZ];
-};
-
struct mlx5hws_pool_db {
enum mlx5hws_db_type type;
union {
- struct mlx5hws_element_manager *element_manager;
- struct mlx5hws_buddy_manager *buddy_manager;
+ unsigned long *bitmap;
+ struct mlx5hws_buddy_mem *buddy;
};
};
@@ -113,9 +73,8 @@ struct mlx5hws_pool {
size_t alloc_log_sz;
enum mlx5hws_table_type tbl_type;
enum mlx5hws_pool_optimize opt_type;
- struct mlx5hws_pool_resource *resource[MLX5HWS_POOL_RESOURCE_ARR_SZ];
- struct mlx5hws_pool_resource *mirror_resource[MLX5HWS_POOL_RESOURCE_ARR_SZ];
- /* DB */
+ struct mlx5hws_pool_resource *resource;
+ struct mlx5hws_pool_resource *mirror_resource;
struct mlx5hws_pool_db db;
/* Functions */
mlx5hws_pool_unint_db p_db_uninit;
@@ -127,7 +86,7 @@ struct mlx5hws_pool *
mlx5hws_pool_create(struct mlx5hws_context *ctx,
struct mlx5hws_pool_attr *pool_attr);
-int mlx5hws_pool_destroy(struct mlx5hws_pool *pool);
+void mlx5hws_pool_destroy(struct mlx5hws_pool *pool);
int mlx5hws_pool_chunk_alloc(struct mlx5hws_pool *pool,
struct mlx5hws_pool_chunk *chunk);
@@ -135,17 +94,13 @@ int mlx5hws_pool_chunk_alloc(struct mlx5hws_pool *pool,
void mlx5hws_pool_chunk_free(struct mlx5hws_pool *pool,
struct mlx5hws_pool_chunk *chunk);
-static inline u32
-mlx5hws_pool_chunk_get_base_id(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
+static inline u32 mlx5hws_pool_get_base_id(struct mlx5hws_pool *pool)
{
- return pool->resource[chunk->resource_idx]->base_id;
+ return pool->resource->base_id;
}
-static inline u32
-mlx5hws_pool_chunk_get_base_mirror_id(struct mlx5hws_pool *pool,
- struct mlx5hws_pool_chunk *chunk)
+static inline u32 mlx5hws_pool_get_base_mirror_id(struct mlx5hws_pool *pool)
{
- return pool->mirror_resource[chunk->resource_idx]->base_id;
+ return pool->mirror_resource->base_id;
}
#endif /* MLX5HWS_POOL_H_ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.c
index 8a011b958b43..97195c643a58 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.c
@@ -142,14 +142,9 @@ hws_rule_save_resize_info(struct mlx5hws_rule *rule,
return;
}
- rule->resize_info->max_stes =
- rule->matcher->action_ste[MLX5HWS_ACTION_STE_IDX_ANY].max_stes;
- rule->resize_info->action_ste_pool[0] = rule->matcher->action_ste[0].max_stes ?
- rule->matcher->action_ste[0].pool :
- NULL;
- rule->resize_info->action_ste_pool[1] = rule->matcher->action_ste[1].max_stes ?
- rule->matcher->action_ste[1].pool :
- NULL;
+ rule->resize_info->max_stes = rule->matcher->action_ste.max_stes;
+ rule->resize_info->action_ste_pool = rule->matcher->action_ste.max_stes ?
+ rule->matcher->action_ste.pool : NULL;
}
memcpy(rule->resize_info->ctrl_seg, ste_attr->wqe_ctrl,
@@ -204,15 +199,15 @@ hws_rule_load_delete_info(struct mlx5hws_rule *rule,
}
}
-static int hws_rule_alloc_action_ste_idx(struct mlx5hws_rule *rule,
- u8 action_ste_selector)
+static int hws_rule_alloc_action_ste(struct mlx5hws_rule *rule,
+ struct mlx5hws_rule_attr *attr)
{
struct mlx5hws_matcher *matcher = rule->matcher;
struct mlx5hws_matcher_action_ste *action_ste;
struct mlx5hws_pool_chunk ste = {0};
int ret;
- action_ste = &matcher->action_ste[action_ste_selector];
+ action_ste = &matcher->action_ste;
ste.order = ilog2(roundup_pow_of_two(action_ste->max_stes));
ret = mlx5hws_pool_chunk_alloc(action_ste->pool, &ste);
if (unlikely(ret)) {
@@ -225,8 +220,7 @@ static int hws_rule_alloc_action_ste_idx(struct mlx5hws_rule *rule,
return 0;
}
-static void hws_rule_free_action_ste_idx(struct mlx5hws_rule *rule,
- u8 action_ste_selector)
+void mlx5hws_rule_free_action_ste(struct mlx5hws_rule *rule)
{
struct mlx5hws_matcher *matcher = rule->matcher;
struct mlx5hws_pool_chunk ste = {0};
@@ -236,10 +230,10 @@ static void hws_rule_free_action_ste_idx(struct mlx5hws_rule *rule,
if (mlx5hws_matcher_is_resizable(matcher)) {
/* Free the original action pool if rule was resized */
max_stes = rule->resize_info->max_stes;
- pool = rule->resize_info->action_ste_pool[action_ste_selector];
+ pool = rule->resize_info->action_ste_pool;
} else {
- max_stes = matcher->action_ste[action_ste_selector].max_stes;
- pool = matcher->action_ste[action_ste_selector].pool;
+ max_stes = matcher->action_ste.max_stes;
+ pool = matcher->action_ste.pool;
}
/* This release is safe only when the rule match part was deleted */
@@ -249,41 +243,6 @@ static void hws_rule_free_action_ste_idx(struct mlx5hws_rule *rule,
mlx5hws_pool_chunk_free(pool, &ste);
}
-static int hws_rule_alloc_action_ste(struct mlx5hws_rule *rule,
- struct mlx5hws_rule_attr *attr)
-{
- int action_ste_idx;
- int ret;
-
- ret = hws_rule_alloc_action_ste_idx(rule, 0);
- if (unlikely(ret))
- return ret;
-
- action_ste_idx = rule->action_ste_idx;
-
- ret = hws_rule_alloc_action_ste_idx(rule, 1);
- if (unlikely(ret)) {
- hws_rule_free_action_ste_idx(rule, 0);
- return ret;
- }
-
- /* Both pools have to return the same index */
- if (unlikely(rule->action_ste_idx != action_ste_idx)) {
- pr_warn("HWS: allocation of action STE failed - pool indexes mismatch\n");
- return -EINVAL;
- }
-
- return 0;
-}
-
-void mlx5hws_rule_free_action_ste(struct mlx5hws_rule *rule)
-{
- if (rule->action_ste_idx > -1) {
- hws_rule_free_action_ste_idx(rule, 1);
- hws_rule_free_action_ste_idx(rule, 0);
- }
-}
-
static void hws_rule_create_init(struct mlx5hws_rule *rule,
struct mlx5hws_send_ste_attr *ste_attr,
struct mlx5hws_actions_apply_data *apply,
@@ -298,9 +257,6 @@ static void hws_rule_create_init(struct mlx5hws_rule *rule,
/* In update we use these rtc's */
rule->rtc_0 = 0;
rule->rtc_1 = 0;
- rule->action_ste_selector = 0;
- } else {
- rule->action_ste_selector = !rule->action_ste_selector;
}
rule->pending_wqes = 0;
@@ -315,8 +271,8 @@ static void hws_rule_create_init(struct mlx5hws_rule *rule,
/* Init default action apply */
apply->tbl_type = tbl->type;
- apply->common_res = &ctx->common_res[tbl->type];
- apply->jump_to_action_stc = matcher->action_ste[0].stc.offset;
+ apply->common_res = &ctx->common_res;
+ apply->jump_to_action_stc = matcher->action_ste.stc.offset;
apply->require_dep = 0;
}
@@ -333,7 +289,6 @@ static void hws_rule_move_init(struct mlx5hws_rule *rule,
rule->pending_wqes = 0;
rule->action_ste_idx = -1;
- rule->action_ste_selector = 0;
rule->status = MLX5HWS_RULE_STATUS_CREATING;
rule->resize_info->state = MLX5HWS_RULE_RESIZE_STATE_WRITING;
}
@@ -403,10 +358,8 @@ static int hws_rule_create_hws(struct mlx5hws_rule *rule,
}
}
/* Skip RX/TX based on the dep_wqe init */
- ste_attr.rtc_0 = dep_wqe->rtc_0 ?
- matcher->action_ste[rule->action_ste_selector].rtc_0_id : 0;
- ste_attr.rtc_1 = dep_wqe->rtc_1 ?
- matcher->action_ste[rule->action_ste_selector].rtc_1_id : 0;
+ ste_attr.rtc_0 = dep_wqe->rtc_0 ? matcher->action_ste.rtc_0_id : 0;
+ ste_attr.rtc_1 = dep_wqe->rtc_1 ? matcher->action_ste.rtc_1_id : 0;
/* Action STEs are written to a specific index last to first */
ste_attr.direct_index = rule->action_ste_idx + action_stes;
apply.next_direct_idx = ste_attr.direct_index;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.h b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.h
index 495cdd17e9f3..fd2bef87116b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_rule.h
@@ -42,7 +42,7 @@ struct mlx5hws_rule_match_tag {
};
struct mlx5hws_rule_resize_info {
- struct mlx5hws_pool *action_ste_pool[2];
+ struct mlx5hws_pool *action_ste_pool;
u32 rtc_0;
u32 rtc_1;
u32 rule_idx;
@@ -62,7 +62,6 @@ struct mlx5hws_rule {
u32 rtc_1; /* The RTC into which the STE was inserted */
int action_ste_idx; /* STE array index */
u8 status; /* enum mlx5hws_rule_status */
- u8 action_ste_selector; /* For rule update - which action STE is in use */
u8 pending_wqes;
bool skip_delete; /* For complex rules - another rule with same tag
* still exists, so don't actually delete this rule.
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_send.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_send.c
index 08be034bd1e1..6911ce413883 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_send.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_send.c
@@ -566,7 +566,7 @@ static int hws_send_ring_alloc_sq(struct mlx5_core_dev *mdev,
size_t buf_sz;
int err;
- sq->uar_map = mdev->mlx5e_res.hw_objs.bfreg.map;
+ sq->uar_map = mdev->priv.bfreg.map;
sq->mdev = mdev;
param.db_numa_node = numa_node;
@@ -639,7 +639,7 @@ static int hws_send_ring_create_sq(struct mlx5_core_dev *mdev, u32 pdn,
MLX5_SET(sqc, sqc, ts_format, ts_format);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
- MLX5_SET(wq, wq, uar_page, mdev->mlx5e_res.hw_objs.bfreg.index);
+ MLX5_SET(wq, wq, uar_page, mdev->priv.bfreg.index);
MLX5_SET(wq, wq, log_wq_pg_sz, sq->wq_ctrl.buf.page_shift - MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(wq, wq, dbr_addr, sq->wq_ctrl.db.dma);
@@ -748,12 +748,6 @@ static int hws_send_ring_open_sq(struct mlx5hws_context *ctx,
return err;
}
-static void hws_cq_complete(struct mlx5_core_cq *mcq,
- struct mlx5_eqe *eqe)
-{
- pr_err("CQ completion CQ: #%u\n", mcq->cqn);
-}
-
static int hws_send_ring_alloc_cq(struct mlx5_core_dev *mdev,
int numa_node,
struct mlx5hws_send_engine *queue,
@@ -776,7 +770,6 @@ static int hws_send_ring_alloc_cq(struct mlx5_core_dev *mdev,
mcq->cqe_sz = 64;
mcq->set_ci_db = cq->wq_ctrl.db.db;
mcq->arm_db = cq->wq_ctrl.db.db + 1;
- mcq->comp = hws_cq_complete;
for (i = 0; i < mlx5_cqwq_get_size(&cq->wq); i++) {
cqe = mlx5_cqwq_get_wqe(&cq->wq, i);
@@ -815,7 +808,7 @@ static int hws_send_ring_create_cq(struct mlx5_core_dev *mdev,
(__be64 *)MLX5_ADDR_OF(create_cq_in, in, pas));
MLX5_SET(cqc, cqc, c_eqn_or_apu_element, eqn);
- MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
+ MLX5_SET(cqc, cqc, uar_page, mdev->priv.bfreg.up->index);
MLX5_SET(cqc, cqc, log_page_size, cq->wq_ctrl.buf.page_shift - MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(cqc, cqc, dbr_addr, cq->wq_ctrl.db.dma);
@@ -838,8 +831,7 @@ static int hws_send_ring_open_cq(struct mlx5_core_dev *mdev,
if (!cqc_data)
return -ENOMEM;
- MLX5_SET(cqc, cqc_data, uar_page, mdev->priv.uar->index);
- MLX5_SET(cqc, cqc_data, cqe_sz, queue->num_entries);
+ MLX5_SET(cqc, cqc_data, uar_page, mdev->priv.bfreg.up->index);
MLX5_SET(cqc, cqc_data, log_cq_size, ilog2(queue->num_entries));
err = hws_send_ring_alloc_cq(mdev, numa_node, queue, cqc_data, cq);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_table.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_table.c
index 9cbe3d71e6f3..0f413547e89b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_table.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/mlx5hws_table.c
@@ -37,6 +37,7 @@ static void hws_table_set_cap_attr(struct mlx5hws_table *tbl,
}
static int hws_table_up_default_fdb_miss_tbl(struct mlx5hws_table *tbl)
+__must_hold(&tbl->ctx->ctrl_lock)
{
struct mlx5hws_cmd_ft_create_attr ft_attr = {0};
struct mlx5hws_cmd_set_fte_attr fte_attr = {0};
@@ -48,8 +49,8 @@ static int hws_table_up_default_fdb_miss_tbl(struct mlx5hws_table *tbl)
if (tbl->type != MLX5HWS_TABLE_TYPE_FDB)
return 0;
- if (ctx->common_res[tbl_type].default_miss) {
- ctx->common_res[tbl_type].default_miss->refcount++;
+ if (ctx->common_res.default_miss) {
+ ctx->common_res.default_miss->refcount++;
return 0;
}
@@ -70,29 +71,28 @@ static int hws_table_up_default_fdb_miss_tbl(struct mlx5hws_table *tbl)
return -EINVAL;
}
- /* ctx->ctrl_lock must be held here */
- ctx->common_res[tbl_type].default_miss = default_miss;
- ctx->common_res[tbl_type].default_miss->refcount++;
+ ctx->common_res.default_miss = default_miss;
+ ctx->common_res.default_miss->refcount++;
return 0;
}
/* Called under ctx->ctrl_lock */
static void hws_table_down_default_fdb_miss_tbl(struct mlx5hws_table *tbl)
+__must_hold(&tbl->ctx->ctrl_lock)
{
struct mlx5hws_cmd_forward_tbl *default_miss;
struct mlx5hws_context *ctx = tbl->ctx;
- u8 tbl_type = tbl->type;
if (tbl->type != MLX5HWS_TABLE_TYPE_FDB)
return;
- default_miss = ctx->common_res[tbl_type].default_miss;
+ default_miss = ctx->common_res.default_miss;
if (--default_miss->refcount)
return;
mlx5hws_cmd_forward_tbl_destroy(ctx->mdev, default_miss);
- ctx->common_res[tbl_type].default_miss = NULL;
+ ctx->common_res.default_miss = NULL;
}
static int hws_table_connect_to_default_miss_tbl(struct mlx5hws_table *tbl, u32 ft_id)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/wc.c b/drivers/net/ethernet/mellanox/mlx5/core/wc.c
index 740b719e7072..089f7d3712b5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/wc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/wc.c
@@ -94,7 +94,7 @@ static int create_wc_cq(struct mlx5_wc_cq *cq, void *cqc_data)
MLX5_SET(cqc, cqc, cq_period_mode, MLX5_CQ_PERIOD_MODE_START_FROM_EQE);
MLX5_SET(cqc, cqc, c_eqn_or_apu_element, eqn);
- MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
+ MLX5_SET(cqc, cqc, uar_page, mdev->priv.bfreg.up->index);
MLX5_SET(cqc, cqc, log_page_size, cq->wq_ctrl.buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(cqc, cqc, dbr_addr, cq->wq_ctrl.db.dma);
@@ -116,7 +116,7 @@ static int mlx5_wc_create_cq(struct mlx5_core_dev *mdev, struct mlx5_wc_cq *cq)
return -ENOMEM;
MLX5_SET(cqc, cqc, log_cq_size, TEST_WC_LOG_CQ_SZ);
- MLX5_SET(cqc, cqc, uar_page, mdev->priv.uar->index);
+ MLX5_SET(cqc, cqc, uar_page, mdev->priv.bfreg.up->index);
if (MLX5_CAP_GEN(mdev, cqe_128_always) && cache_line_size() >= 128)
MLX5_SET(cqc, cqc, cqe_sz, CQE_STRIDE_128_PAD);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index 81164f12cad3..12377cb1669a 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -1680,7 +1680,7 @@ static int mlxsw_sp_port_create(struct mlxsw_sp *mlxsw_sp, u16 local_port,
NETIF_F_HW_TC;
dev->hw_features |= NETIF_F_HW_TC | NETIF_F_LOOPBACK;
dev->lltx = true;
- dev->netns_local = true;
+ dev->netns_immutable = true;
dev->min_mtu = ETH_MIN_MTU;
dev->max_mtu = MLXSW_PORT_MAX_MTU - MLXSW_PORT_ETH_FRAME_HDR;
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index bd1ae41df552..ef89f3e06a26 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -3006,11 +3006,16 @@ void mana_remove(struct gdma_dev *gd, bool suspending)
struct gdma_context *gc = gd->gdma_context;
struct mana_context *ac = gd->driver_data;
struct mana_port_context *apc;
- struct device *dev = gc->dev;
+ struct device *dev;
struct net_device *ndev;
int err;
int i;
+ if (!gc || !ac)
+ return;
+
+ dev = gc->dev;
+
/* adev currently doesn't support suspending, always remove it */
if (gd->adev)
remove_adev(gd);
diff --git a/drivers/net/ethernet/rocker/rocker_main.c b/drivers/net/ethernet/rocker/rocker_main.c
index 23b20d5fd016..2794f75df8fc 100644
--- a/drivers/net/ethernet/rocker/rocker_main.c
+++ b/drivers/net/ethernet/rocker/rocker_main.c
@@ -2575,7 +2575,7 @@ static int rocker_probe_port(struct rocker *rocker, unsigned int port_number)
rocker_carrier_init(rocker_port);
dev->features |= NETIF_F_SG;
- dev->netns_local = true;
+ dev->netns_immutable = true;
/* MTU range: 68 - 9000 */
dev->min_mtu = ROCKER_PORT_MIN_MTU;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 91c07c04435f..057d23c8d890 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -2521,7 +2521,8 @@ static bool stmmac_xdp_xmit_zc(struct stmmac_priv *priv, u32 queue, u32 budget)
tx_desc = tx_q->dma_tx + entry;
dma_addr = xsk_buff_raw_get_dma(pool, xdp_desc.addr);
- meta = xsk_buff_get_metadata(pool, xdp_desc.addr);
+ meta = xsk_buff_get_metadata(pool, xdp_desc.addr,
+ xdp_desc.options);
xsk_buff_raw_dma_sync_for_device(pool, dma_addr, xdp_desc.len);
tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_XSK_TX;
@@ -2566,6 +2567,7 @@ static bool stmmac_xdp_xmit_zc(struct stmmac_priv *priv, u32 queue, u32 budget)
stmmac_enable_dma_transmission(priv, priv->ioaddr, queue);
+ tx_q->tx_skbuff_dma[entry].xsk_meta.tx_timestamp = NULL;
xsk_tx_metadata_to_compl(meta,
&tx_q->tx_skbuff_dma[entry].xsk_meta);
@@ -4085,6 +4087,7 @@ static bool stmmac_vlan_insert(struct stmmac_priv *priv, struct sk_buff *skb,
/**
* stmmac_tso_allocator - close entry point of the driver
* @priv: driver private structure
+ * @entry: TX queue buffer index
* @des: buffer start address
* @total_len: total length to fill in descriptors
* @last_segment: condition for the last descriptor
@@ -4093,8 +4096,9 @@ static bool stmmac_vlan_insert(struct stmmac_priv *priv, struct sk_buff *skb,
* This function fills descriptor and request new descriptors according to
* buffer length to fill
*/
-static void stmmac_tso_allocator(struct stmmac_priv *priv, dma_addr_t des,
- int total_len, bool last_segment, u32 queue)
+static void stmmac_tso_allocator(struct stmmac_priv *priv, u32 *entry,
+ dma_addr_t des, int total_len,
+ bool last_segment, u32 queue)
{
struct stmmac_tx_queue *tx_q = &priv->dma_conf.tx_queue[queue];
struct dma_desc *desc;
@@ -4106,14 +4110,13 @@ static void stmmac_tso_allocator(struct stmmac_priv *priv, dma_addr_t des,
while (tmp_len > 0) {
dma_addr_t curr_addr;
- tx_q->cur_tx = STMMAC_NEXT_ENTRY(tx_q->cur_tx,
- priv->dma_conf.dma_tx_size);
- WARN_ON(tx_q->tx_skbuff[tx_q->cur_tx]);
+ *entry = STMMAC_NEXT_ENTRY(*entry, priv->dma_conf.dma_tx_size);
+ WARN_ON(tx_q->tx_skbuff[*entry]);
if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc = &tx_q->dma_entx[tx_q->cur_tx].basic;
+ desc = &tx_q->dma_entx[*entry].basic;
else
- desc = &tx_q->dma_tx[tx_q->cur_tx];
+ desc = &tx_q->dma_tx[*entry];
curr_addr = des + (total_len - tmp_len);
stmmac_set_desc_addr(priv, desc, curr_addr);
@@ -4238,7 +4241,7 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct dma_desc *desc, *first, *mss_desc = NULL;
struct stmmac_priv *priv = netdev_priv(dev);
- unsigned int first_entry, tx_packets;
+ unsigned int first_entry, entry, tx_packets;
struct stmmac_txq_stats *txq_stats;
int i, first_tx, nfrags, ndesc;
struct stmmac_tx_queue *tx_q;
@@ -4313,22 +4316,24 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
}
first_entry = tx_q->cur_tx;
- WARN_ON(tx_q->tx_skbuff[first_entry]);
+ entry = first_entry;
+
+ WARN_ON(tx_q->tx_skbuff[entry]);
if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc = &tx_q->dma_entx[first_entry].basic;
+ desc = &tx_q->dma_entx[entry].basic;
else
- desc = &tx_q->dma_tx[first_entry];
+ desc = &tx_q->dma_tx[entry];
first = desc;
/* first descriptor: fill Headers on Buf1 */
des = dma_map_single(priv->device, skb->data, skb_headlen(skb),
DMA_TO_DEVICE);
if (dma_mapping_error(priv->device, des))
- goto dma_map_err;
+ goto error;
stmmac_set_desc_addr(priv, first, des);
- stmmac_tso_allocator(priv, des + proto_hdr_len, pay_len,
+ stmmac_tso_allocator(priv, &entry, des + proto_hdr_len, pay_len,
(nfrags == 0), queue);
/* In case two or more DMA transmit descriptors are allocated for this
@@ -4343,10 +4348,10 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
* this DMA buffer right after the DMA engine completely finishes the
* full buffer transmission.
*/
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf = des;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].len = skb_headlen(skb);
- tx_q->tx_skbuff_dma[tx_q->cur_tx].map_as_page = false;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
+ tx_q->tx_skbuff_dma[entry].buf = des;
+ tx_q->tx_skbuff_dma[entry].len = skb_headlen(skb);
+ tx_q->tx_skbuff_dma[entry].map_as_page = false;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
/* Prepare fragments */
for (i = 0; i < nfrags; i++) {
@@ -4356,16 +4361,17 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
skb_frag_size(frag),
DMA_TO_DEVICE);
if (dma_mapping_error(priv->device, des))
- goto dma_map_err;
+ goto error_dma_unmap;
- stmmac_tso_allocator(priv, des, skb_frag_size(frag),
+ stmmac_tso_allocator(priv, &entry, des, skb_frag_size(frag),
(i == nfrags - 1), queue);
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf = des;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].len = skb_frag_size(frag);
- tx_q->tx_skbuff_dma[tx_q->cur_tx].map_as_page = true;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
+ tx_q->tx_skbuff_dma[entry].buf = des;
+ tx_q->tx_skbuff_dma[entry].len = skb_frag_size(frag);
+ tx_q->tx_skbuff_dma[entry].map_as_page = true;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
}
+ tx_q->cur_tx = entry;
tx_q->tx_skbuff_dma[tx_q->cur_tx].last_segment = true;
@@ -4462,7 +4468,22 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
return NETDEV_TX_OK;
-dma_map_err:
+error_dma_unmap:
+ for (;;) {
+ if (tx_q->tbs & STMMAC_TBS_AVAIL)
+ desc = &tx_q->dma_entx[first_entry].basic;
+ else
+ desc = &tx_q->dma_tx[first_entry];
+ stmmac_release_tx_desc(priv, desc, priv->mode);
+ stmmac_free_tx_buffer(priv, &priv->dma_conf, queue,
+ first_entry);
+ if (first_entry == entry)
+ break;
+
+ first_entry = STMMAC_NEXT_ENTRY(first_entry,
+ priv->dma_conf.dma_tx_size);
+ }
+error:
dev_err(priv->device, "Tx dma map failed\n");
dev_kfree_skb(skb);
priv->xstats.tx_dropped++;
@@ -4903,13 +4924,27 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
if (!priv->sph)
return 0;
- /* Not last descriptor */
- if (status & rx_not_ls)
+ /* For GMAC4, when split header is enabled, in some rare cases, the
+ * hardware does not fill buf2 of the first descriptor with payload.
+ * Thus we cannot assume buf2 is always fully filled if it is not
+ * the last descriptor. Otherwise, the length of buf2 of the second
+ * descriptor will be calculated wrong and cause an oops.
+ *
+ * If this is the last descriptor, 'plen' is the length of the
+ * received packet that was transferred to system memory.
+ * Otherwise, it is the accumulated number of bytes that have been
+ * transferred for the current packet.
+ *
+ * Thus 'plen - len' always gives the correct length of buf2.
+ */
+
+ /* Not GMAC4 and not last descriptor */
+ if (!priv->plat->has_gmac4 && (status & rx_not_ls))
return priv->dma_conf.dma_buf_sz;
+ /* GMAC4 or last descriptor */
plen = stmmac_get_rx_frame_len(priv, p, coe);
- /* Last descriptor */
return plen - len;
}
diff --git a/drivers/net/ethernet/ti/am65-cpsw-nuss.c b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
index 45cea845d4aa..e9d9d69e2ba9 100644
--- a/drivers/net/ethernet/ti/am65-cpsw-nuss.c
+++ b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
@@ -398,35 +398,61 @@ static void am65_cpsw_init_host_port_switch(struct am65_cpsw_common *common);
static void am65_cpsw_init_host_port_emac(struct am65_cpsw_common *common);
static void am65_cpsw_init_port_switch_ale(struct am65_cpsw_port *port);
static void am65_cpsw_init_port_emac_ale(struct am65_cpsw_port *port);
+static inline void am65_cpsw_put_page(struct am65_cpsw_rx_flow *flow,
+ struct page *page,
+ bool allow_direct);
+static void am65_cpsw_nuss_rx_cleanup(void *data, dma_addr_t desc_dma);
-static void am65_cpsw_destroy_xdp_rxqs(struct am65_cpsw_common *common)
+static void am65_cpsw_destroy_rxq(struct am65_cpsw_common *common, int id)
{
struct am65_cpsw_rx_chn *rx_chn = &common->rx_chns;
struct am65_cpsw_rx_flow *flow;
struct xdp_rxq_info *rxq;
- int id, port;
+ int port;
- for (id = 0; id < common->rx_ch_num_flows; id++) {
- flow = &rx_chn->flows[id];
+ flow = &rx_chn->flows[id];
+ napi_disable(&flow->napi_rx);
+ hrtimer_cancel(&flow->rx_hrtimer);
+ k3_udma_glue_reset_rx_chn(rx_chn->rx_chn, id, rx_chn,
+ am65_cpsw_nuss_rx_cleanup);
- for (port = 0; port < common->port_num; port++) {
- if (!common->ports[port].ndev)
- continue;
+ for (port = 0; port < common->port_num; port++) {
+ if (!common->ports[port].ndev)
+ continue;
- rxq = &common->ports[port].xdp_rxq[id];
+ rxq = &common->ports[port].xdp_rxq[id];
- if (xdp_rxq_info_is_reg(rxq))
- xdp_rxq_info_unreg(rxq);
- }
+ if (xdp_rxq_info_is_reg(rxq))
+ xdp_rxq_info_unreg(rxq);
+ }
- if (flow->page_pool) {
- page_pool_destroy(flow->page_pool);
- flow->page_pool = NULL;
- }
+ if (flow->page_pool) {
+ page_pool_destroy(flow->page_pool);
+ flow->page_pool = NULL;
}
}
-static int am65_cpsw_create_xdp_rxqs(struct am65_cpsw_common *common)
+static void am65_cpsw_destroy_rxqs(struct am65_cpsw_common *common)
+{
+ struct am65_cpsw_rx_chn *rx_chn = &common->rx_chns;
+ int id;
+
+ reinit_completion(&common->tdown_complete);
+ k3_udma_glue_tdown_rx_chn(rx_chn->rx_chn, true);
+
+ if (common->pdata.quirks & AM64_CPSW_QUIRK_DMA_RX_TDOWN_IRQ) {
+ id = wait_for_completion_timeout(&common->tdown_complete, msecs_to_jiffies(1000));
+ if (!id)
+ dev_err(common->dev, "rx teardown timeout\n");
+ }
+
+ for (id = common->rx_ch_num_flows - 1; id >= 0; id--)
+ am65_cpsw_destroy_rxq(common, id);
+
+ k3_udma_glue_disable_rx_chn(common->rx_chns.rx_chn);
+}
+
+static int am65_cpsw_create_rxq(struct am65_cpsw_common *common, int id)
{
struct am65_cpsw_rx_chn *rx_chn = &common->rx_chns;
struct page_pool_params pp_params = {
@@ -441,45 +467,92 @@ static int am65_cpsw_create_xdp_rxqs(struct am65_cpsw_common *common)
struct am65_cpsw_rx_flow *flow;
struct xdp_rxq_info *rxq;
struct page_pool *pool;
- int id, port, ret;
+ struct page *page;
+ int port, ret, i;
- for (id = 0; id < common->rx_ch_num_flows; id++) {
- flow = &rx_chn->flows[id];
- pp_params.napi = &flow->napi_rx;
- pool = page_pool_create(&pp_params);
- if (IS_ERR(pool)) {
- ret = PTR_ERR(pool);
+ flow = &rx_chn->flows[id];
+ pp_params.napi = &flow->napi_rx;
+ pool = page_pool_create(&pp_params);
+ if (IS_ERR(pool)) {
+ ret = PTR_ERR(pool);
+ return ret;
+ }
+
+ flow->page_pool = pool;
+
+ /* using same page pool is allowed as no running rx handlers
+ * simultaneously for both ndevs
+ */
+ for (port = 0; port < common->port_num; port++) {
+ if (!common->ports[port].ndev)
+ /* FIXME should we BUG here? */
+ continue;
+
+ rxq = &common->ports[port].xdp_rxq[id];
+ ret = xdp_rxq_info_reg(rxq, common->ports[port].ndev,
+ id, flow->napi_rx.napi_id);
+ if (ret)
+ goto err;
+
+ ret = xdp_rxq_info_reg_mem_model(rxq,
+ MEM_TYPE_PAGE_POOL,
+ pool);
+ if (ret)
+ goto err;
+ }
+
+ for (i = 0; i < AM65_CPSW_MAX_RX_DESC; i++) {
+ page = page_pool_dev_alloc_pages(flow->page_pool);
+ if (!page) {
+ dev_err(common->dev, "cannot allocate page in flow %d\n",
+ id);
+ ret = -ENOMEM;
goto err;
}
- flow->page_pool = pool;
+ ret = am65_cpsw_nuss_rx_push(common, page, id);
+ if (ret < 0) {
+ dev_err(common->dev,
+ "cannot submit page to rx channel flow %d, error %d\n",
+ id, ret);
+ am65_cpsw_put_page(flow, page, false);
+ goto err;
+ }
+ }
- /* using same page pool is allowed as no running rx handlers
- * simultaneously for both ndevs
- */
- for (port = 0; port < common->port_num; port++) {
- if (!common->ports[port].ndev)
- continue;
+ napi_enable(&flow->napi_rx);
+ return 0;
- rxq = &common->ports[port].xdp_rxq[id];
+err:
+ am65_cpsw_destroy_rxq(common, id);
+ return ret;
+}
- ret = xdp_rxq_info_reg(rxq, common->ports[port].ndev,
- id, flow->napi_rx.napi_id);
- if (ret)
- goto err;
+static int am65_cpsw_create_rxqs(struct am65_cpsw_common *common)
+{
+ int id, ret;
- ret = xdp_rxq_info_reg_mem_model(rxq,
- MEM_TYPE_PAGE_POOL,
- pool);
- if (ret)
- goto err;
+ for (id = 0; id < common->rx_ch_num_flows; id++) {
+ ret = am65_cpsw_create_rxq(common, id);
+ if (ret) {
+ dev_err(common->dev, "couldn't create rxq %d: %d\n",
+ id, ret);
+ goto err;
}
}
+ ret = k3_udma_glue_enable_rx_chn(common->rx_chns.rx_chn);
+ if (ret) {
+ dev_err(common->dev, "couldn't enable rx chn: %d\n", ret);
+ goto err;
+ }
+
return 0;
err:
- am65_cpsw_destroy_xdp_rxqs(common);
+ for (--id; id >= 0; id--)
+ am65_cpsw_destroy_rxq(common, id);
+
return ret;
}
@@ -543,7 +616,6 @@ static void am65_cpsw_nuss_rx_cleanup(void *data, dma_addr_t desc_dma)
k3_udma_glue_rx_cppi5_to_dma_addr(rx_chn->rx_chn, &buf_dma);
dma_unmap_single(rx_chn->dma_dev, buf_dma, buf_dma_len, DMA_FROM_DEVICE);
k3_cppi_desc_pool_free(rx_chn->desc_pool, desc_rx);
-
am65_cpsw_put_page(&rx_chn->flows[flow_id], page, false);
}
@@ -625,12 +697,9 @@ static struct sk_buff *am65_cpsw_build_skb(void *page_addr,
static int am65_cpsw_nuss_common_open(struct am65_cpsw_common *common)
{
struct am65_cpsw_host *host_p = am65_common_get_host(common);
- struct am65_cpsw_rx_chn *rx_chn = &common->rx_chns;
struct am65_cpsw_tx_chn *tx_chn = common->tx_chns;
- int port_idx, i, ret, tx, flow_idx;
- struct am65_cpsw_rx_flow *flow;
+ int port_idx, ret, tx;
u32 val, port_mask;
- struct page *page;
if (common->usage_count)
return 0;
@@ -690,47 +759,9 @@ static int am65_cpsw_nuss_common_open(struct am65_cpsw_common *common)
am65_cpsw_qos_tx_p0_rate_init(common);
- ret = am65_cpsw_create_xdp_rxqs(common);
- if (ret) {
- dev_err(common->dev, "Failed to create XDP rx queues\n");
+ ret = am65_cpsw_create_rxqs(common);
+ if (ret)
return ret;
- }
-
- for (flow_idx = 0; flow_idx < common->rx_ch_num_flows; flow_idx++) {
- flow = &rx_chn->flows[flow_idx];
- for (i = 0; i < AM65_CPSW_MAX_RX_DESC; i++) {
- page = page_pool_dev_alloc_pages(flow->page_pool);
- if (!page) {
- dev_err(common->dev, "cannot allocate page in flow %d\n",
- flow_idx);
- ret = -ENOMEM;
- goto fail_rx;
- }
-
- ret = am65_cpsw_nuss_rx_push(common, page, flow_idx);
- if (ret < 0) {
- dev_err(common->dev,
- "cannot submit page to rx channel flow %d, error %d\n",
- flow_idx, ret);
- am65_cpsw_put_page(flow, page, false);
- goto fail_rx;
- }
- }
- }
-
- ret = k3_udma_glue_enable_rx_chn(rx_chn->rx_chn);
- if (ret) {
- dev_err(common->dev, "couldn't enable rx chn: %d\n", ret);
- goto fail_rx;
- }
-
- for (i = 0; i < common->rx_ch_num_flows ; i++) {
- napi_enable(&rx_chn->flows[i].napi_rx);
- if (rx_chn->flows[i].irq_disabled) {
- rx_chn->flows[i].irq_disabled = false;
- enable_irq(rx_chn->flows[i].irq);
- }
- }
for (tx = 0; tx < common->tx_ch_num; tx++) {
ret = k3_udma_glue_enable_tx_chn(tx_chn[tx].tx_chn);
@@ -753,30 +784,13 @@ static int am65_cpsw_nuss_common_open(struct am65_cpsw_common *common)
tx--;
}
- for (flow_idx = 0; i < common->rx_ch_num_flows; flow_idx++) {
- flow = &rx_chn->flows[flow_idx];
- if (!flow->irq_disabled) {
- disable_irq(flow->irq);
- flow->irq_disabled = true;
- }
- napi_disable(&flow->napi_rx);
- }
-
- k3_udma_glue_disable_rx_chn(rx_chn->rx_chn);
-
-fail_rx:
- for (i = 0; i < common->rx_ch_num_flows; i++)
- k3_udma_glue_reset_rx_chn(rx_chn->rx_chn, i, rx_chn,
- am65_cpsw_nuss_rx_cleanup, !!i);
-
- am65_cpsw_destroy_xdp_rxqs(common);
+ am65_cpsw_destroy_rxqs(common);
return ret;
}
static int am65_cpsw_nuss_common_stop(struct am65_cpsw_common *common)
{
- struct am65_cpsw_rx_chn *rx_chn = &common->rx_chns;
struct am65_cpsw_tx_chn *tx_chn = common->tx_chns;
int i;
@@ -810,31 +824,12 @@ static int am65_cpsw_nuss_common_stop(struct am65_cpsw_common *common)
k3_udma_glue_disable_tx_chn(tx_chn[i].tx_chn);
}
- reinit_completion(&common->tdown_complete);
- k3_udma_glue_tdown_rx_chn(rx_chn->rx_chn, true);
-
- if (common->pdata.quirks & AM64_CPSW_QUIRK_DMA_RX_TDOWN_IRQ) {
- i = wait_for_completion_timeout(&common->tdown_complete, msecs_to_jiffies(1000));
- if (!i)
- dev_err(common->dev, "rx teardown timeout\n");
- }
-
- for (i = common->rx_ch_num_flows - 1; i >= 0; i--) {
- napi_disable(&rx_chn->flows[i].napi_rx);
- hrtimer_cancel(&rx_chn->flows[i].rx_hrtimer);
- k3_udma_glue_reset_rx_chn(rx_chn->rx_chn, i, rx_chn,
- am65_cpsw_nuss_rx_cleanup, !!i);
- }
-
- k3_udma_glue_disable_rx_chn(rx_chn->rx_chn);
-
+ am65_cpsw_destroy_rxqs(common);
cpsw_ale_stop(common->ale);
writel(0, common->cpsw_base + AM65_CPSW_REG_CTL);
writel(0, common->cpsw_base + AM65_CPSW_REG_STAT_PORT_EN);
- am65_cpsw_destroy_xdp_rxqs(common);
-
dev_dbg(common->dev, "cpsw_nuss stopped\n");
return 0;
}
@@ -3375,7 +3370,7 @@ static int am65_cpsw_nuss_register_ndevs(struct am65_cpsw_common *common)
for (i = 0; i < common->rx_ch_num_flows; i++)
k3_udma_glue_reset_rx_chn(rx_chan->rx_chn, i,
rx_chan,
- am65_cpsw_nuss_rx_cleanup, !!i);
+ am65_cpsw_nuss_rx_cleanup);
k3_udma_glue_disable_rx_chn(rx_chan->rx_chn);
diff --git a/drivers/net/ethernet/ti/cpsw_new.c b/drivers/net/ethernet/ti/cpsw_new.c
index c2041e46b83d..d6f19cfd10b7 100644
--- a/drivers/net/ethernet/ti/cpsw_new.c
+++ b/drivers/net/ethernet/ti/cpsw_new.c
@@ -1427,7 +1427,7 @@ static int cpsw_create_ports(struct cpsw_common *cpsw)
ndev->features |= NETIF_F_HW_VLAN_CTAG_FILTER |
NETIF_F_HW_VLAN_CTAG_RX | NETIF_F_HW_TC;
- ndev->netns_local = true;
+ ndev->netns_immutable = true;
ndev->xdp_features = NETDEV_XDP_ACT_BASIC |
NETDEV_XDP_ACT_REDIRECT |
diff --git a/drivers/net/ethernet/ti/icssg/icssg_common.c b/drivers/net/ethernet/ti/icssg/icssg_common.c
index 74f0f200a89d..62065416e886 100644
--- a/drivers/net/ethernet/ti/icssg/icssg_common.c
+++ b/drivers/net/ethernet/ti/icssg/icssg_common.c
@@ -955,7 +955,7 @@ void prueth_reset_rx_chan(struct prueth_rx_chn *chn,
for (i = 0; i < num_flows; i++)
k3_udma_glue_reset_rx_chn(chn->rx_chn, i, chn,
- prueth_rx_cleanup, !!i);
+ prueth_rx_cleanup);
if (disable)
k3_udma_glue_disable_rx_chn(chn->rx_chn);
}
diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet.h b/drivers/net/ethernet/xilinx/xilinx_axienet.h
index cbdca0fb8945..f9e8e83cfed9 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet.h
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet.h
@@ -516,8 +516,9 @@ struct skbuf_dma_descriptor {
* @stats_work: Work for reading the hardware statistics counters often enough
* to catch overflows.
* @dma_err_task: Work structure to process Axi DMA errors
- * @stopping: Set when @dma_err_task shouldn't do anything because we are
- * about to stop the device.
+ * @stopping: Set when we are about to stop the device: makes @dma_err_task
+ * a no-op (legacy DMA path) and fences RX descriptor
+ * resubmission in axienet_dma_rx_cb() (dmaengine path).
* @tx_irq: Axidma TX IRQ number
* @rx_irq: Axidma RX IRQ number
* @eth_irq: Ethernet core IRQ number
@@ -542,6 +543,7 @@ struct skbuf_dma_descriptor {
* @tx_ring_tail: TX skb ring buffer tail index.
* @rx_ring_head: RX skb ring buffer head index.
* @rx_ring_tail: RX skb ring buffer tail index.
+ * @rx_submit_lock: Protects RX ring resubmission vs teardown in dmaengine path.
*/
struct axienet_local {
struct net_device *ndev;
@@ -620,6 +622,7 @@ struct axienet_local {
int tx_ring_tail;
int rx_ring_head;
int rx_ring_tail;
+ spinlock_t rx_submit_lock;
};
/**
diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
index 3339c5e1a57a..76970a806549 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
@@ -841,6 +841,7 @@ static void axienet_dma_tx_cb(void *data, const struct dmaengine_result *result)
u64_stats_update_end(&lp->tx_stat_sync);
dma_unmap_sg(lp->dev, skbuf_dma->sgl, skbuf_dma->sg_len, DMA_TO_DEVICE);
dev_consume_skb_any(skbuf_dma->skb);
+ skbuf_dma->skb = NULL;
netif_txq_completed_wake(txq, 1, len,
CIRC_SPACE(lp->tx_ring_head, lp->tx_ring_tail, TX_BD_NUM_MAX),
2);
@@ -1127,6 +1128,7 @@ static void axienet_dma_rx_cb(void *data, const struct dmaengine_result *result)
&meta_max_len);
dma_unmap_single(lp->dev, skbuf_dma->dma_address, lp->max_frm_size,
DMA_FROM_DEVICE);
+ skbuf_dma->skb = NULL;
if (IS_ERR(app_metadata)) {
if (net_ratelimit())
@@ -1149,10 +1151,17 @@ static void axienet_dma_rx_cb(void *data, const struct dmaengine_result *result)
u64_stats_update_end(&lp->rx_stat_sync);
rx_submit:
+ spin_lock(&lp->rx_submit_lock);
+ if (lp->stopping) {
+ spin_unlock(&lp->rx_submit_lock);
+ return;
+ }
+
for (i = 0; i < CIRC_SPACE(lp->rx_ring_head, lp->rx_ring_tail,
RX_BUF_NUM_DEFAULT); i++)
axienet_rx_submit_desc(lp->ndev);
dma_async_issue_pending(lp->rx_chan);
+ spin_unlock(&lp->rx_submit_lock);
}
/**
@@ -1475,6 +1484,7 @@ static int axienet_init_dmaengine(struct net_device *ndev)
lp->tx_ring_head = 0;
lp->rx_ring_tail = 0;
lp->rx_ring_head = 0;
+ lp->stopping = false;
lp->tx_skb_ring = kcalloc(TX_BD_NUM_MAX, sizeof(*lp->tx_skb_ring),
GFP_KERNEL);
if (!lp->tx_skb_ring) {
@@ -1686,20 +1696,42 @@ static int axienet_stop(struct net_device *ndev)
free_irq(lp->rx_irq, ndev);
axienet_dma_bd_release(ndev);
} else {
+ struct skbuf_dma_descriptor *skbuf_dma;
+
+ spin_lock_bh(&lp->rx_submit_lock);
+ lp->stopping = true;
+ spin_unlock_bh(&lp->rx_submit_lock);
+
dmaengine_terminate_sync(lp->tx_chan);
- dmaengine_synchronize(lp->tx_chan);
dmaengine_terminate_sync(lp->rx_chan);
- dmaengine_synchronize(lp->rx_chan);
-
- for (i = 0; i < TX_BD_NUM_MAX; i++)
- kfree(lp->tx_skb_ring[i]);
- kfree(lp->tx_skb_ring);
- for (i = 0; i < RX_BUF_NUM_DEFAULT; i++)
- kfree(lp->rx_skb_ring[i]);
- kfree(lp->rx_skb_ring);
dma_release_channel(lp->rx_chan);
dma_release_channel(lp->tx_chan);
+
+ /* Unmap and free any buffer the terminate did not reclaim, so it
+ * is not leaked; a non-NULL skb marks such a slot.
+ */
+ for (i = 0; i < TX_BD_NUM_MAX; i++) {
+ skbuf_dma = lp->tx_skb_ring[i];
+ if (skbuf_dma && skbuf_dma->skb) {
+ dma_unmap_sg(lp->dev, skbuf_dma->sgl,
+ skbuf_dma->sg_len, DMA_TO_DEVICE);
+ dev_kfree_skb_any(skbuf_dma->skb);
+ }
+ kfree(skbuf_dma);
+ }
+ kfree(lp->tx_skb_ring);
+
+ for (i = 0; i < RX_BUF_NUM_DEFAULT; i++) {
+ skbuf_dma = lp->rx_skb_ring[i];
+ if (skbuf_dma && skbuf_dma->skb) {
+ dma_unmap_single(lp->dev, skbuf_dma->dma_address,
+ lp->max_frm_size, DMA_FROM_DEVICE);
+ dev_kfree_skb_any(skbuf_dma->skb);
+ }
+ kfree(skbuf_dma);
+ }
+ kfree(lp->rx_skb_ring);
}
axienet_iow(lp, XAE_IE_OFFSET, 0);
@@ -2615,6 +2647,7 @@ static int axienet_probe(struct platform_device *pdev)
u64_stats_init(&lp->rx_stat_sync);
u64_stats_init(&lp->tx_stat_sync);
+ spin_lock_init(&lp->rx_submit_lock);
mutex_init(&lp->stats_lock);
seqcount_mutex_init(&lp->hw_stats_seqcount, &lp->stats_lock);
INIT_DEFERRABLE_WORK(&lp->stats_work, axienet_refresh_stats);
diff --git a/drivers/net/loopback.c b/drivers/net/loopback.c
index 491e56b3263f..63b236efb3b7 100644
--- a/drivers/net/loopback.c
+++ b/drivers/net/loopback.c
@@ -172,7 +172,7 @@ static void gen_lo_setup(struct net_device *dev,
dev->flags = IFF_LOOPBACK;
dev->priv_flags |= IFF_LIVE_ADDR_CHANGE | IFF_NO_QUEUE;
dev->lltx = true;
- dev->netns_local = true;
+ dev->netns_immutable = true;
netif_keep_dst(dev);
dev->hw_features = NETIF_F_GSO_SOFTWARE;
dev->features = NETIF_F_SG | NETIF_F_FRAGLIST
diff --git a/drivers/net/net_failover.c b/drivers/net/net_failover.c
index 54c8b9d5b5fc..5b50d9186f12 100644
--- a/drivers/net/net_failover.c
+++ b/drivers/net/net_failover.c
@@ -734,7 +734,7 @@ struct failover *net_failover_create(struct net_device *standby_dev)
failover_dev->lltx = true;
/* Don't allow failover devices to change network namespaces. */
- failover_dev->netns_local = true;
+ failover_dev->netns_immutable = true;
failover_dev->hw_features = FAILOVER_VLAN_FEATURES |
NETIF_F_HW_VLAN_CTAG_TX |
diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c
index 0c47713b6adb..68df8a9e1ecc 100644
--- a/drivers/net/netconsole.c
+++ b/drivers/net/netconsole.c
@@ -748,6 +748,7 @@ static ssize_t userdatum_value_store(struct config_item *item, const char *buf,
if (count > MAX_USERDATA_VALUE_LENGTH)
return -EMSGSIZE;
+ mutex_lock(&netconsole_subsys.su_mutex);
mutex_lock(&dynamic_netconsole_mutex);
ret = strscpy(udm->value, buf, sizeof(udm->value));
@@ -761,6 +762,7 @@ static ssize_t userdatum_value_store(struct config_item *item, const char *buf,
ret = count;
out_unlock:
mutex_unlock(&dynamic_netconsole_mutex);
+ mutex_unlock(&netconsole_subsys.su_mutex);
return ret;
}
diff --git a/drivers/net/tap.c b/drivers/net/tap.c
index b3cb7d5a9585..fcc9d135c5fc 100644
--- a/drivers/net/tap.c
+++ b/drivers/net/tap.c
@@ -645,6 +645,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
int err;
struct virtio_net_hdr vnet_hdr = { 0 };
int vnet_hdr_len = 0;
+ int hdr_len = 0;
int copylen = 0;
int depth;
bool zerocopy = false;
@@ -663,13 +664,13 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
if (!copy_from_iter_full(&vnet_hdr, sizeof(vnet_hdr), from))
goto err;
iov_iter_advance(from, vnet_hdr_len - sizeof(vnet_hdr));
- if ((vnet_hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) &&
- tap16_to_cpu(q, vnet_hdr.csum_start) +
- tap16_to_cpu(q, vnet_hdr.csum_offset) + 2 >
- tap16_to_cpu(q, vnet_hdr.hdr_len))
- vnet_hdr.hdr_len = cpu_to_tap16(q,
- tap16_to_cpu(q, vnet_hdr.csum_start) +
- tap16_to_cpu(q, vnet_hdr.csum_offset) + 2);
+ hdr_len = tap16_to_cpu(q, vnet_hdr.hdr_len);
+ if (vnet_hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
+ hdr_len = max(tap16_to_cpu(q, vnet_hdr.csum_start) +
+ tap16_to_cpu(q, vnet_hdr.csum_offset) + 2,
+ hdr_len);
+ vnet_hdr.hdr_len = cpu_to_tap16(q, hdr_len);
+ }
err = -EINVAL;
if (tap16_to_cpu(q, vnet_hdr.hdr_len) > len)
goto err;
@@ -682,12 +683,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
if (msg_control && sock_flag(&q->sk, SOCK_ZEROCOPY)) {
struct iov_iter i;
- copylen = vnet_hdr.hdr_len ?
- tap16_to_cpu(q, vnet_hdr.hdr_len) : GOODCOPY_LEN;
- if (copylen > good_linear)
- copylen = good_linear;
- else if (copylen < ETH_HLEN)
- copylen = ETH_HLEN;
+ copylen = clamp(hdr_len ?: GOODCOPY_LEN, ETH_HLEN, good_linear);
linear = copylen;
i = *from;
iov_iter_advance(&i, copylen);
@@ -697,11 +693,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
if (!zerocopy) {
copylen = len;
- linear = tap16_to_cpu(q, vnet_hdr.hdr_len);
- if (linear > good_linear)
- linear = good_linear;
- else if (linear < ETH_HLEN)
- linear = ETH_HLEN;
+ linear = clamp(hdr_len, ETH_HLEN, good_linear);
}
skb = tap_alloc_skb(&q->sk, TAP_RESERVE, copylen,
@@ -1015,7 +1007,7 @@ static long tap_ioctl(struct file *file, unsigned int cmd,
unsigned int __user *up = argp;
unsigned short u;
int __user *sp = argp;
- struct sockaddr sa;
+ struct sockaddr sa = {};
int s;
int ret;
@@ -1216,12 +1208,13 @@ static int tap_get_user_xdp(struct tap_queue *q, struct xdp_buff *xdp)
}
}
+ skb_probe_transport_header(skb);
+
/* Move network header to the right position for VLAN tagged packets */
if (eth_type_vlan(skb->protocol) &&
vlan_get_protocol_and_depth(skb, skb->protocol, &depth) != 0)
skb_set_network_header(skb, depth);
- skb_probe_transport_header(skb);
dev_queue_xmit(skb);
rcu_read_unlock();
diff --git a/drivers/net/team/team_core.c b/drivers/net/team/team_core.c
index 3cbaee3ff5fd..1645a7a2e076 100644
--- a/drivers/net/team/team_core.c
+++ b/drivers/net/team/team_core.c
@@ -534,21 +534,23 @@ static void team_adjust_ops(struct team *team)
if (!team->en_port_count || !team_is_mode_set(team) ||
!team->mode->ops->transmit)
- team->ops.transmit = team_dummy_transmit;
+ WRITE_ONCE(team->ops.transmit, team_dummy_transmit);
else
- team->ops.transmit = team->mode->ops->transmit;
+ WRITE_ONCE(team->ops.transmit, team->mode->ops->transmit);
if (!team->en_port_count || !team_is_mode_set(team) ||
!team->mode->ops->receive)
- team->ops.receive = team_dummy_receive;
+ WRITE_ONCE(team->ops.receive, team_dummy_receive);
else
- team->ops.receive = team->mode->ops->receive;
+ WRITE_ONCE(team->ops.receive, team->mode->ops->receive);
}
/*
- * We can benefit from the fact that it's ensured no port is present
- * at the time of mode change. Therefore no packets are in fly so there's no
- * need to set mode operations in any special way.
+ * team_change_mode() ensures no ports are present during mode change,
+ * but lockless readers can still reach team_xmit(). Avoid touching
+ * transmit/receive -- they are already set to dummies by
+ * team_adjust_ops() since no ports are enabled. synchronize_net()
+ * drains in-flight readers before destroying old mode state.
*/
static int __team_change_mode(struct team *team,
const struct team_mode *new_mode)
@@ -557,9 +559,21 @@ static int __team_change_mode(struct team *team,
if (team_is_mode_set(team)) {
void (*exit_op)(struct team *team) = team->ops.exit;
- /* Clear ops area so no callback is called any longer */
- memset(&team->ops, 0, sizeof(struct team_mode_ops));
- team_adjust_ops(team);
+ /* Clear cold-path ops used only under RTNL. transmit and
+ * receive are already dummies (no ports) so leave them
+ * alone -- overwriting them is the source of the race.
+ */
+ team->ops.init = NULL;
+ team->ops.exit = NULL;
+ team->ops.port_enter = NULL;
+ team->ops.port_leave = NULL;
+ team->ops.port_change_dev_addr = NULL;
+ team->ops.port_tx_disabled = NULL;
+
+ /* Wait for in-flight readers before tearing down mode
+ * state they may reference.
+ */
+ synchronize_net();
if (exit_op)
exit_op(team);
@@ -582,7 +596,12 @@ static int __team_change_mode(struct team *team,
}
team->mode = new_mode;
- memcpy(&team->ops, new_mode->ops, sizeof(struct team_mode_ops));
+ team->ops.init = new_mode->ops->init;
+ team->ops.exit = new_mode->ops->exit;
+ team->ops.port_enter = new_mode->ops->port_enter;
+ team->ops.port_leave = new_mode->ops->port_leave;
+ team->ops.port_change_dev_addr = new_mode->ops->port_change_dev_addr;
+ team->ops.port_tx_disabled = new_mode->ops->port_tx_disabled;
team_adjust_ops(team);
return 0;
@@ -743,7 +762,7 @@ static rx_handler_result_t team_handle_frame(struct sk_buff **pskb)
/* allow exact match delivery for disabled ports */
res = RX_HANDLER_EXACT;
} else {
- res = team->ops.receive(team, port, skb);
+ res = READ_ONCE(team->ops.receive)(team, port, skb);
}
if (res == RX_HANDLER_ANOTHER) {
struct team_pcpu_stats *pcpu_stats;
@@ -802,7 +821,8 @@ static struct list_head *__team_get_qom_list(struct team *team, u16 queue_id)
/*
* note: already called with rcu_read_lock
*/
-static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb)
+static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb,
+ bool *tx_success)
{
struct list_head *qom_list;
struct team_port *port;
@@ -810,11 +830,11 @@ static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb)
if (!team->queue_override_enabled || !skb->queue_mapping)
return false;
qom_list = __team_get_qom_list(team, skb->queue_mapping);
- list_for_each_entry_rcu(port, qom_list, qom_list) {
- if (!team_dev_queue_xmit(team, port, skb))
- return true;
- }
- return false;
+ port = list_first_or_null_rcu(qom_list, struct team_port, qom_list);
+ if (!port)
+ return false;
+ *tx_success = !team_dev_queue_xmit(team, port, skb);
+ return true;
}
static void __team_queue_override_port_del(struct team *team,
@@ -944,8 +964,6 @@ static void team_port_enable(struct team *team,
team_port_index_hash(team, port->index));
team_adjust_ops(team);
team_queue_override_port_add(team, port);
- if (team->ops.port_enabled)
- team->ops.port_enabled(team, port);
team_notify_peers(team);
team_mcast_rejoin(team);
team_lower_state_changed(port);
@@ -970,8 +988,8 @@ static void team_port_disable(struct team *team,
{
if (!team_port_enabled(port))
return;
- if (team->ops.port_disabled)
- team->ops.port_disabled(team, port);
+ if (team->ops.port_tx_disabled)
+ team->ops.port_tx_disabled(team, port);
hlist_del_rcu(&port->hlist);
__reconstruct_port_hlist(team, port->index);
port->index = -1;
@@ -1347,7 +1365,7 @@ static int team_port_add(struct team *team, struct net_device *port_dev,
static void __team_port_change_port_removed(struct team_port *port);
-static int team_port_del(struct team *team, struct net_device *port_dev)
+static int team_port_del(struct team *team, struct net_device *port_dev, bool unregister)
{
struct net_device *dev = team->dev;
struct team_port *port;
@@ -1385,7 +1403,10 @@ static int team_port_del(struct team *team, struct net_device *port_dev)
__team_port_change_port_removed(port);
team_port_set_orig_dev_addr(port);
- dev_set_mtu(port_dev, port->orig.mtu);
+ if (unregister)
+ __dev_set_mtu(port_dev, port->orig.mtu);
+ else
+ dev_set_mtu(port_dev, port->orig.mtu);
kfree_rcu(port, rcu);
netdev_info(dev, "Port device %s removed\n", portname);
__team_compute_features(team);
@@ -1690,7 +1711,7 @@ static void team_uninit(struct net_device *dev)
mutex_lock(&team->lock);
list_for_each_entry_safe(port, tmp, &team->port_list, list)
- team_port_del(team, port->dev);
+ team_port_del(team, port->dev, false);
__team_change_mode(team, NULL); /* cleanup */
__team_options_unregister(team, team_options, ARRAY_SIZE(team_options));
@@ -1736,9 +1757,8 @@ static netdev_tx_t team_xmit(struct sk_buff *skb, struct net_device *dev)
bool tx_success;
unsigned int len = skb->len;
- tx_success = team_queue_override_transmit(team, skb);
- if (!tx_success)
- tx_success = team->ops.transmit(team, skb);
+ if (!team_queue_override_transmit(team, skb, &tx_success))
+ tx_success = READ_ONCE(team->ops.transmit)(team, skb);
if (tx_success) {
struct team_pcpu_stats *pcpu_stats;
@@ -2002,13 +2022,14 @@ static int team_add_slave(struct net_device *dev, struct net_device *port_dev,
return err;
}
-static int team_del_slave(struct net_device *dev, struct net_device *port_dev)
+static int __team_del_slave(struct net_device *dev, struct net_device *port_dev,
+ bool unregister)
{
struct team *team = netdev_priv(dev);
int err;
mutex_lock(&team->lock);
- err = team_port_del(team, port_dev);
+ err = team_port_del(team, port_dev, unregister);
mutex_unlock(&team->lock);
if (err)
@@ -2024,6 +2045,16 @@ static int team_del_slave(struct net_device *dev, struct net_device *port_dev)
return err;
}
+static int team_del_slave(struct net_device *dev, struct net_device *port_dev)
+{
+ return __team_del_slave(dev, port_dev, false);
+}
+
+static int team_del_slave_on_unregister(struct net_device *dev, struct net_device *port_dev)
+{
+ return __team_del_slave(dev, port_dev, true);
+}
+
static netdev_features_t team_fix_features(struct net_device *dev,
netdev_features_t features)
{
@@ -2274,7 +2305,7 @@ static void team_setup(struct net_device *dev)
dev->lltx = true;
/* Don't allow team devices to change network namespaces. */
- dev->netns_local = true;
+ dev->netns_immutable = true;
dev->features |= NETIF_F_GRO;
@@ -3065,7 +3096,7 @@ static int team_device_event(struct notifier_block *unused,
!!netif_oper_up(port->dev));
break;
case NETDEV_UNREGISTER:
- team_del_slave(port->team->dev, dev);
+ team_del_slave_on_unregister(port->team->dev, dev);
break;
case NETDEV_FEAT_CHANGE:
if (!port->team->notifier_ctx) {
diff --git a/drivers/net/team/team_mode_loadbalance.c b/drivers/net/team/team_mode_loadbalance.c
index 00f8989c29c0..3ca419884185 100644
--- a/drivers/net/team/team_mode_loadbalance.c
+++ b/drivers/net/team/team_mode_loadbalance.c
@@ -658,7 +658,7 @@ static void lb_port_leave(struct team *team, struct team_port *port)
free_percpu(lb_port_priv->pcpu_stats);
}
-static void lb_port_disabled(struct team *team, struct team_port *port)
+static void lb_port_tx_disabled(struct team *team, struct team_port *port)
{
lb_tx_hash_to_port_mapping_null_port(team, port);
}
@@ -668,7 +668,7 @@ static const struct team_mode_ops lb_mode_ops = {
.exit = lb_exit,
.port_enter = lb_port_enter,
.port_leave = lb_port_leave,
- .port_disabled = lb_port_disabled,
+ .port_tx_disabled = lb_port_tx_disabled,
.receive = lb_receive,
.transmit = lb_transmit,
};
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index cb640d29d791..2c62371666cd 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -409,9 +409,6 @@ struct virtnet_info {
u16 rss_indir_table_size;
u32 rss_hash_types_supported;
u32 rss_hash_types_saved;
- struct virtio_net_rss_config_hdr *rss_hdr;
- struct virtio_net_rss_config_trailer rss_trailer;
- u8 rss_hash_key_data[VIRTIO_NET_RSS_MAX_KEY_SIZE];
/* Has control virtqueue */
bool has_cvq;
@@ -470,7 +467,16 @@ struct virtnet_info {
struct failover *failover;
u64 device_stats_cap;
+
+ struct virtio_net_rss_config_hdr *rss_hdr;
+
+ /* Must be last as it ends in a flexible-array member. */
+ TRAILING_OVERLAP(struct virtio_net_rss_config_trailer, rss_trailer, hash_key_data,
+ u8 rss_hash_key_data[VIRTIO_NET_RSS_MAX_KEY_SIZE];
+ );
};
+static_assert(offsetof(struct virtnet_info, rss_trailer.hash_key_data) ==
+ offsetof(struct virtnet_info, rss_hash_key_data));
struct padded_vnet_hdr {
struct virtio_net_hdr_v1_hash hdr;
@@ -3214,17 +3220,31 @@ static void virtnet_rx_resume(struct virtnet_info *vi, struct receive_queue *rq)
static int virtnet_rx_resize(struct virtnet_info *vi,
struct receive_queue *rq, u32 ring_num)
{
+ unsigned int old_ring_num = virtqueue_get_vring_size(rq->vq);
+ struct xdp_buff **tmp_xsk_buffs = NULL;
int err, qindex;
qindex = rq - vi->rq;
+ if (rq->xsk_pool && ring_num > old_ring_num) {
+ tmp_xsk_buffs = kvzalloc_objs(*tmp_xsk_buffs, ring_num);
+ if (!tmp_xsk_buffs)
+ return -ENOMEM;
+ }
+
virtnet_rx_pause(vi, rq);
err = virtqueue_resize(rq->vq, ring_num, virtnet_rq_unmap_free_buf, NULL);
+
+ /* virtqueue_resize may have changed the size even if err != 0 */
+ if (tmp_xsk_buffs && virtqueue_get_vring_size(rq->vq) > old_ring_num)
+ swap(rq->xsk_buffs, tmp_xsk_buffs);
+
if (err)
netdev_err(vi->dev, "resize rx fail: rx queue index: %d err: %d\n", qindex, err);
virtnet_rx_resume(vi, rq);
+ kvfree(tmp_xsk_buffs);
return err;
}
diff --git a/drivers/net/vrf.c b/drivers/net/vrf.c
index c071b3ccd8cb..e48a15ad4a16 100644
--- a/drivers/net/vrf.c
+++ b/drivers/net/vrf.c
@@ -1624,7 +1624,7 @@ static void vrf_setup(struct net_device *dev)
dev->lltx = true;
/* don't allow vrf devices to change network namespaces. */
- dev->netns_local = true;
+ dev->netns_immutable = true;
/* does not make sense for a VLAN to be added to a vrf device */
dev->features |= NETIF_F_VLAN_CHALLENGED;
diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
index f612def46c3c..e70a0b657570 100644
--- a/drivers/net/vxlan/vxlan_core.c
+++ b/drivers/net/vxlan/vxlan_core.c
@@ -1917,6 +1917,7 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
if (n) {
struct vxlan_rdst *rdst = NULL;
+ u8 ha[ETH_ALEN] __aligned(2);
struct vxlan_fdb *f;
struct sk_buff *reply;
@@ -1925,8 +1926,10 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
+ neigh_ha_snapshot(ha, n, n->dev);
+
rcu_read_lock();
- f = vxlan_find_mac_tx(vxlan, n->ha, vni);
+ f = vxlan_find_mac_tx(vxlan, ha, vni);
if (f)
rdst = first_remote_rcu(f);
if (rdst && vxlan_addr_any(&rdst->remote_ip)) {
@@ -1938,7 +1941,7 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
rcu_read_unlock();
reply = arp_create(ARPOP_REPLY, ETH_P_ARP, sip, dev, tip, sha,
- n->ha, sha);
+ ha, sha);
neigh_release(n);
@@ -1971,7 +1974,8 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
#if IS_ENABLED(CONFIG_IPV6)
static struct sk_buff *vxlan_na_create(struct sk_buff *request,
- struct neighbour *n, bool isrouter)
+ struct neighbour *n, u8 *ha,
+ bool isrouter)
{
struct net_device *dev = request->dev;
struct sk_buff *reply;
@@ -2019,7 +2023,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
/* Ethernet header */
ether_addr_copy(eth_hdr(reply)->h_dest, daddr);
- ether_addr_copy(eth_hdr(reply)->h_source, n->ha);
+ ether_addr_copy(eth_hdr(reply)->h_source, ha);
eth_hdr(reply)->h_proto = htons(ETH_P_IPV6);
reply->protocol = htons(ETH_P_IPV6);
@@ -2048,7 +2052,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
na->icmph.icmp6_override = 1;
na->icmph.icmp6_solicited = 1;
na->target = ns->target;
- ether_addr_copy(&na->opt[2], n->ha);
+ ether_addr_copy(&na->opt[2], ha);
na->opt[0] = ND_OPT_TARGET_LL_ADDR;
na->opt[1] = na_olen >> 3;
@@ -2091,6 +2095,7 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
if (n) {
struct vxlan_rdst *rdst = NULL;
+ u8 ha[ETH_ALEN] __aligned(2);
struct vxlan_fdb *f;
struct sk_buff *reply;
@@ -2099,7 +2104,8 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
- f = vxlan_find_mac_tx(vxlan, n->ha, vni);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = vxlan_find_mac_tx(vxlan, ha, vni);
if (f)
rdst = first_remote_rcu(f);
if (rdst && vxlan_addr_any(&rdst->remote_ip)) {
@@ -2108,7 +2114,7 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
- reply = vxlan_na_create(skb, n,
+ reply = vxlan_na_create(skb, n, ha,
!!(f ? f->flags & NTF_ROUTER : 0));
neigh_release(n);
@@ -2402,7 +2408,7 @@ void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
int addr_family;
__u8 tos, ttl;
int ifindex;
- int err;
+ int err = 0;
u32 flags = vxlan->cfg.flags;
bool use_cache;
bool udp_sum = false;
@@ -2504,11 +2510,16 @@ void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
rcu_read_lock();
if (addr_family == AF_INET) {
- struct vxlan_sock *sock4 = rcu_dereference(vxlan->vn4_sock);
+ struct vxlan_sock *sock4;
struct rtable *rt;
__be16 df = 0;
__be32 saddr;
+ sock4 = rcu_dereference(vxlan->vn4_sock);
+ if (unlikely(!sock4)) {
+ goto tx_error;
+ }
+
if (!ifindex)
ifindex = sock4->sock->sk->sk_bound_dev_if;
@@ -2576,9 +2587,14 @@ void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev,
src_port, dst_port, xnet, !udp_sum);
#if IS_ENABLED(CONFIG_IPV6)
} else {
- struct vxlan_sock *sock6 = rcu_dereference(vxlan->vn6_sock);
+ struct vxlan_sock *sock6;
struct in6_addr saddr;
+ sock6 = rcu_dereference(vxlan->vn6_sock);
+ if (unlikely(!sock6)) {
+ goto tx_error;
+ }
+
if (!ifindex)
ifindex = sock6->sock->sk->sk_bound_dev_if;
diff --git a/drivers/net/wireguard/device.c b/drivers/net/wireguard/device.c
index 45e9b908dbfb..055ea3877156 100644
--- a/drivers/net/wireguard/device.c
+++ b/drivers/net/wireguard/device.c
@@ -406,12 +406,11 @@ static struct rtnl_link_ops link_ops __read_mostly = {
.newlink = wg_newlink,
};
-static void wg_netns_pre_exit(struct net *net)
+static void __net_exit wg_netns_exit_rtnl(struct net *net, struct list_head *dev_kill_list)
{
struct wg_device *wg;
struct wg_peer *peer;
- rtnl_lock();
list_for_each_entry(wg, &device_list, device_list) {
if (rcu_access_pointer(wg->creating_net) == net) {
pr_debug("%s: Creating namespace exiting\n", wg->dev->name);
@@ -424,11 +423,10 @@ static void wg_netns_pre_exit(struct net *net)
mutex_unlock(&wg->device_update_lock);
}
}
- rtnl_unlock();
}
-static struct pernet_operations pernet_ops = {
- .pre_exit = wg_netns_pre_exit
+static struct pernet_operations pernet_ops __read_mostly = {
+ .exit_rtnl = wg_netns_exit_rtnl
};
int __init wg_device_init(void)
diff --git a/drivers/net/wireless/ath/ath11k/mhi.c b/drivers/net/wireless/ath/ath11k/mhi.c
index 6974a551883f..593f24602fc4 100644
--- a/drivers/net/wireless/ath/ath11k/mhi.c
+++ b/drivers/net/wireless/ath/ath11k/mhi.c
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: BSD-3-Clause-Clear
/*
* Copyright (c) 2020 The Linux Foundation. All rights reserved.
- * Copyright (c) 2021-2024 Qualcomm Innovation Center, Inc. All rights reserved.
+ * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
*/
#include <linux/msi.h>
@@ -34,7 +34,6 @@ static const struct mhi_channel_config ath11k_mhi_channels_qca6390[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
},
{
.num = 21,
@@ -48,7 +47,6 @@ static const struct mhi_channel_config ath11k_mhi_channels_qca6390[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = true,
},
};
@@ -99,7 +97,6 @@ static const struct mhi_channel_config ath11k_mhi_channels_qcn9074[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
},
{
.num = 21,
@@ -113,7 +110,6 @@ static const struct mhi_channel_config ath11k_mhi_channels_qcn9074[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = true,
},
};
@@ -286,8 +282,10 @@ static void ath11k_mhi_op_status_cb(struct mhi_controller *mhi_cntrl,
break;
}
+ spin_lock_bh(&ab->base_lock);
if (!(test_bit(ATH11K_FLAG_UNREGISTERING, &ab->dev_flags)))
queue_work(ab->workqueue_aux, &ab->reset_work);
+ spin_unlock_bh(&ab->base_lock);
break;
default:
diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c
index 6b0e52a6fb89..18957d450080 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -1023,6 +1023,13 @@ static void ath11k_pci_shutdown(struct pci_dev *pdev)
struct ath11k_pci *ab_pci = ath11k_pci_priv(ab);
ath11k_pci_set_irq_affinity_hint(ab_pci, NULL);
+
+ spin_lock_bh(&ab->base_lock);
+ set_bit(ATH11K_FLAG_UNREGISTERING, &ab->dev_flags);
+ spin_unlock_bh(&ab->base_lock);
+
+ cancel_work_sync(&ab->reset_work);
+
ath11k_pci_power_down(ab);
}
diff --git a/drivers/net/wireless/ath/ath12k/core.h b/drivers/net/wireless/ath/ath12k/core.h
index 3faf3430effb..3567150b49ff 100644
--- a/drivers/net/wireless/ath/ath12k/core.h
+++ b/drivers/net/wireless/ath/ath12k/core.h
@@ -452,7 +452,7 @@ struct ath12k_sta {
u32 smps;
enum hal_pn_type pn_type;
- struct work_struct update_wk;
+ struct wiphy_work update_wk;
struct rate_info txrate;
struct rate_info last_txrate;
u64 rx_duration;
diff --git a/drivers/net/wireless/ath/ath12k/dp_mon.c b/drivers/net/wireless/ath/ath12k/dp_mon.c
index 7bfd323cdf24..ad276f860505 100644
--- a/drivers/net/wireless/ath/ath12k/dp_mon.c
+++ b/drivers/net/wireless/ath/ath12k/dp_mon.c
@@ -10,11 +10,10 @@
#include "dp_tx.h"
#include "peer.h"
-static void ath12k_dp_mon_rx_handle_ofdma_info(void *rx_tlv,
- struct hal_rx_user_status *rx_user_status)
+static void
+ath12k_dp_mon_rx_handle_ofdma_info(const struct hal_rx_ppdu_end_user_stats *ppdu_end_user,
+ struct hal_rx_user_status *rx_user_status)
{
- struct hal_rx_ppdu_end_user_stats *ppdu_end_user = rx_tlv;
-
rx_user_status->ul_ofdma_user_v0_word0 =
__le32_to_cpu(ppdu_end_user->usr_resp_ref);
rx_user_status->ul_ofdma_user_v0_word1 =
@@ -38,7 +37,7 @@ ath12k_dp_mon_rx_populate_byte_count(const struct hal_rx_ppdu_end_user_stats *st
}
static void
-ath12k_dp_mon_rx_populate_mu_user_info(void *rx_tlv,
+ath12k_dp_mon_rx_populate_mu_user_info(const struct hal_rx_ppdu_end_user_stats *rx_tlv,
struct hal_rx_mon_ppdu_info *ppdu_info,
struct hal_rx_user_status *rx_user_status)
{
@@ -76,11 +75,9 @@ ath12k_dp_mon_rx_populate_mu_user_info(void *rx_tlv,
ath12k_dp_mon_rx_populate_byte_count(rx_tlv, ppdu_info, rx_user_status);
}
-static void ath12k_dp_mon_parse_vht_sig_a(u8 *tlv_data,
+static void ath12k_dp_mon_parse_vht_sig_a(const struct hal_rx_vht_sig_a_info *vht_sig,
struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_vht_sig_a_info *vht_sig =
- (struct hal_rx_vht_sig_a_info *)tlv_data;
u32 nsts, group_id, info0, info1;
u8 gi_setting;
@@ -122,11 +119,9 @@ static void ath12k_dp_mon_parse_vht_sig_a(u8 *tlv_data,
u32_get_bits(info1, HAL_RX_VHT_SIG_A_INFO_INFO1_SU_MU_CODING);
}
-static void ath12k_dp_mon_parse_ht_sig(u8 *tlv_data,
+static void ath12k_dp_mon_parse_ht_sig(const struct hal_rx_ht_sig_info *ht_sig,
struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_ht_sig_info *ht_sig =
- (struct hal_rx_ht_sig_info *)tlv_data;
u32 info0 = __le32_to_cpu(ht_sig->info0);
u32 info1 = __le32_to_cpu(ht_sig->info1);
@@ -139,11 +134,9 @@ static void ath12k_dp_mon_parse_ht_sig(u8 *tlv_data,
ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU;
}
-static void ath12k_dp_mon_parse_l_sig_b(u8 *tlv_data,
+static void ath12k_dp_mon_parse_l_sig_b(const struct hal_rx_lsig_b_info *lsigb,
struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_lsig_b_info *lsigb =
- (struct hal_rx_lsig_b_info *)tlv_data;
u32 info0 = __le32_to_cpu(lsigb->info0);
u8 rate;
@@ -173,11 +166,9 @@ static void ath12k_dp_mon_parse_l_sig_b(u8 *tlv_data,
ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU;
}
-static void ath12k_dp_mon_parse_l_sig_a(u8 *tlv_data,
+static void ath12k_dp_mon_parse_l_sig_a(const struct hal_rx_lsig_a_info *lsiga,
struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_lsig_a_info *lsiga =
- (struct hal_rx_lsig_a_info *)tlv_data;
u32 info0 = __le32_to_cpu(lsiga->info0);
u8 rate;
@@ -215,14 +206,13 @@ static void ath12k_dp_mon_parse_l_sig_a(u8 *tlv_data,
ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU;
}
-static void ath12k_dp_mon_parse_he_sig_b2_ofdma(u8 *tlv_data,
- struct hal_rx_mon_ppdu_info *ppdu_info)
+static void
+ath12k_dp_mon_parse_he_sig_b2_ofdma(const struct hal_rx_he_sig_b2_ofdma_info *ofdma,
+ struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_he_sig_b2_ofdma_info *he_sig_b2_ofdma =
- (struct hal_rx_he_sig_b2_ofdma_info *)tlv_data;
u32 info0, value;
- info0 = __le32_to_cpu(he_sig_b2_ofdma->info0);
+ info0 = __le32_to_cpu(ofdma->info0);
ppdu_info->he_data1 |= HE_MCS_KNOWN | HE_DCM_KNOWN | HE_CODING_KNOWN;
@@ -253,11 +243,10 @@ static void ath12k_dp_mon_parse_he_sig_b2_ofdma(u8 *tlv_data,
ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_OFDMA;
}
-static void ath12k_dp_mon_parse_he_sig_b2_mu(u8 *tlv_data,
- struct hal_rx_mon_ppdu_info *ppdu_info)
+static void
+ath12k_dp_mon_parse_he_sig_b2_mu(const struct hal_rx_he_sig_b2_mu_info *he_sig_b2_mu,
+ struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_he_sig_b2_mu_info *he_sig_b2_mu =
- (struct hal_rx_he_sig_b2_mu_info *)tlv_data;
u32 info0, value;
info0 = __le32_to_cpu(he_sig_b2_mu->info0);
@@ -280,11 +269,10 @@ static void ath12k_dp_mon_parse_he_sig_b2_mu(u8 *tlv_data,
ppdu_info->nss = u32_get_bits(info0, HAL_RX_HE_SIG_B2_MU_INFO_INFO0_STA_NSTS);
}
-static void ath12k_dp_mon_parse_he_sig_b1_mu(u8 *tlv_data,
- struct hal_rx_mon_ppdu_info *ppdu_info)
+static void
+ath12k_dp_mon_parse_he_sig_b1_mu(const struct hal_rx_he_sig_b1_mu_info *he_sig_b1_mu,
+ struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_he_sig_b1_mu_info *he_sig_b1_mu =
- (struct hal_rx_he_sig_b1_mu_info *)tlv_data;
u32 info0 = __le32_to_cpu(he_sig_b1_mu->info0);
u16 ru_tones;
@@ -295,11 +283,10 @@ static void ath12k_dp_mon_parse_he_sig_b1_mu(u8 *tlv_data,
ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_MIMO;
}
-static void ath12k_dp_mon_parse_he_sig_mu(u8 *tlv_data,
- struct hal_rx_mon_ppdu_info *ppdu_info)
+static void
+ath12k_dp_mon_parse_he_sig_mu(const struct hal_rx_he_sig_a_mu_dl_info *he_sig_a_mu_dl,
+ struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_he_sig_a_mu_dl_info *he_sig_a_mu_dl =
- (struct hal_rx_he_sig_a_mu_dl_info *)tlv_data;
u32 info0, info1, value;
u16 he_gi = 0, he_ltf = 0;
@@ -430,11 +417,9 @@ static void ath12k_dp_mon_parse_he_sig_mu(u8 *tlv_data,
ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_MIMO;
}
-static void ath12k_dp_mon_parse_he_sig_su(u8 *tlv_data,
+static void ath12k_dp_mon_parse_he_sig_su(const struct hal_rx_he_sig_a_su_info *he_sig_a,
struct hal_rx_mon_ppdu_info *ppdu_info)
{
- struct hal_rx_he_sig_a_su_info *he_sig_a =
- (struct hal_rx_he_sig_a_su_info *)tlv_data;
u32 info0, info1, value;
u32 dcm;
u8 he_dcm = 0, he_stbc = 0;
@@ -583,15 +568,15 @@ static void ath12k_dp_mon_parse_he_sig_su(u8 *tlv_data,
static enum hal_rx_mon_status
ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
struct ath12k_mon_data *pmon,
- u32 tlv_tag, u8 *tlv_data, u32 userid)
+ u32 tlv_tag, const void *tlv_data,
+ u32 userid)
{
struct hal_rx_mon_ppdu_info *ppdu_info = &pmon->mon_ppdu_info;
u32 info[7];
switch (tlv_tag) {
case HAL_RX_PPDU_START: {
- struct hal_rx_ppdu_start *ppdu_start =
- (struct hal_rx_ppdu_start *)tlv_data;
+ const struct hal_rx_ppdu_start *ppdu_start = tlv_data;
info[0] = __le32_to_cpu(ppdu_start->info0);
@@ -610,8 +595,8 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
break;
}
case HAL_RX_PPDU_END_USER_STATS: {
- struct hal_rx_ppdu_end_user_stats *eu_stats =
- (struct hal_rx_ppdu_end_user_stats *)tlv_data;
+ const struct hal_rx_ppdu_end_user_stats *eu_stats = tlv_data;
+ u32 tid_bitmap;
info[0] = __le32_to_cpu(eu_stats->info0);
info[1] = __le32_to_cpu(eu_stats->info1);
@@ -624,10 +609,9 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
u32_get_bits(info[2], HAL_RX_PPDU_END_USER_STATS_INFO2_AST_INDEX);
ppdu_info->fc_valid =
u32_get_bits(info[1], HAL_RX_PPDU_END_USER_STATS_INFO1_FC_VALID);
- ppdu_info->tid =
- ffs(u32_get_bits(info[6],
- HAL_RX_PPDU_END_USER_STATS_INFO6_TID_BITMAP)
- - 1);
+ tid_bitmap = u32_get_bits(info[6],
+ HAL_RX_PPDU_END_USER_STATS_INFO6_TID_BITMAP);
+ ppdu_info->tid = ffs(tid_bitmap) - 1;
ppdu_info->tcp_msdu_count =
u32_get_bits(info[4],
HAL_RX_PPDU_END_USER_STATS_INFO4_TCP_MSDU_CNT);
@@ -673,8 +657,8 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
ppdu_info->num_users += 1;
- ath12k_dp_mon_rx_handle_ofdma_info(tlv_data, rxuser_stats);
- ath12k_dp_mon_rx_populate_mu_user_info(tlv_data, ppdu_info,
+ ath12k_dp_mon_rx_handle_ofdma_info(eu_stats, rxuser_stats);
+ ath12k_dp_mon_rx_populate_mu_user_info(eu_stats, ppdu_info,
rxuser_stats);
}
ppdu_info->mpdu_fcs_ok_bitmap[0] = __le32_to_cpu(eu_stats->rsvd1[0]);
@@ -682,8 +666,8 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
break;
}
case HAL_RX_PPDU_END_USER_STATS_EXT: {
- struct hal_rx_ppdu_end_user_stats_ext *eu_stats =
- (struct hal_rx_ppdu_end_user_stats_ext *)tlv_data;
+ const struct hal_rx_ppdu_end_user_stats_ext *eu_stats = tlv_data;
+
ppdu_info->mpdu_fcs_ok_bitmap[2] = __le32_to_cpu(eu_stats->info1);
ppdu_info->mpdu_fcs_ok_bitmap[3] = __le32_to_cpu(eu_stats->info2);
ppdu_info->mpdu_fcs_ok_bitmap[4] = __le32_to_cpu(eu_stats->info3);
@@ -729,40 +713,25 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
break;
case HAL_PHYRX_RSSI_LEGACY: {
- struct hal_rx_phyrx_rssi_legacy_info *rssi =
- (struct hal_rx_phyrx_rssi_legacy_info *)tlv_data;
- u32 reception_type = 0;
- u32 rssi_legacy_info = __le32_to_cpu(rssi->rsvd[0]);
+ const struct hal_rx_phyrx_rssi_legacy_info *rssi = tlv_data;
info[0] = __le32_to_cpu(rssi->info0);
+ info[1] = __le32_to_cpu(rssi->info1);
/* TODO: Please note that the combined rssi will not be accurate
* in MU case. Rssi in MU needs to be retrieved from
* PHYRX_OTHER_RECEIVE_INFO TLV.
*/
ppdu_info->rssi_comb =
- u32_get_bits(info[0],
- HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RSSI_COMB);
- reception_type =
- u32_get_bits(rssi_legacy_info,
- HAL_RX_PHYRX_RSSI_LEGACY_INFO_RSVD1_RECEPTION);
-
- switch (reception_type) {
- case HAL_RECEPTION_TYPE_ULOFMDA:
- ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_OFDMA;
- break;
- case HAL_RECEPTION_TYPE_ULMIMO:
- ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_MIMO;
- break;
- default:
- ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU;
- break;
- }
+ u32_get_bits(info[1],
+ HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO1_RSSI_COMB);
+
+ ppdu_info->bw = u32_get_bits(info[0],
+ HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RX_BW);
break;
}
case HAL_RXPCU_PPDU_END_INFO: {
- struct hal_rx_ppdu_end_duration *ppdu_rx_duration =
- (struct hal_rx_ppdu_end_duration *)tlv_data;
+ const struct hal_rx_ppdu_end_duration *ppdu_rx_duration = tlv_data;
info[0] = __le32_to_cpu(ppdu_rx_duration->info0);
ppdu_info->rx_duration =
@@ -773,9 +742,7 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
break;
}
case HAL_RX_MPDU_START: {
- struct hal_rx_mpdu_start *mpdu_start =
- (struct hal_rx_mpdu_start *)tlv_data;
- struct dp_mon_mpdu *mon_mpdu = pmon->mon_mpdu;
+ const struct hal_rx_mpdu_start *mpdu_start = tlv_data;
u16 peer_id;
info[1] = __le32_to_cpu(mpdu_start->info1);
@@ -792,67 +759,17 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab,
u32_get_bits(info[0], HAL_RX_MPDU_START_INFO0_PPDU_ID);
}
- mon_mpdu = kzalloc(sizeof(*mon_mpdu), GFP_ATOMIC);
- if (!mon_mpdu)
- return HAL_RX_MON_STATUS_PPDU_NOT_DONE;
-
break;
}
case HAL_RX_MSDU_START:
/* TODO: add msdu start parsing logic */
break;
- case HAL_MON_BUF_ADDR: {
- struct dp_rxdma_mon_ring *buf_ring = &ab->dp.rxdma_mon_buf_ring;
- struct dp_mon_packet_info *packet_info =
- (struct dp_mon_packet_info *)tlv_data;
- int buf_id = u32_get_bits(packet_info->cookie,
- DP_RXDMA_BUF_COOKIE_BUF_ID);
- struct sk_buff *msdu;
- struct dp_mon_mpdu *mon_mpdu = pmon->mon_mpdu;
- struct ath12k_skb_rxcb *rxcb;
-
- spin_lock_bh(&buf_ring->idr_lock);
- msdu = idr_remove(&buf_ring->bufs_idr, buf_id);
- spin_unlock_bh(&buf_ring->idr_lock);
-
- if (unlikely(!msdu)) {
- ath12k_warn(ab, "monitor destination with invalid buf_id %d\n",
- buf_id);
- return HAL_RX_MON_STATUS_PPDU_NOT_DONE;
- }
-
- rxcb = ATH12K_SKB_RXCB(msdu);
- dma_unmap_single(ab->dev, rxcb->paddr,
- msdu->len + skb_tailroom(msdu),
- DMA_FROM_DEVICE);
-
- if (mon_mpdu->tail)
- mon_mpdu->tail->next = msdu;
- else
- mon_mpdu->tail = msdu;
-
- ath12k_dp_mon_buf_replenish(ab, buf_ring, 1);
-
- break;
- }
- case HAL_RX_MSDU_END: {
- struct rx_msdu_end_qcn9274 *msdu_end =
- (struct rx_msdu_end_qcn9274 *)tlv_data;
- bool is_first_msdu_in_mpdu;
- u16 msdu_end_info;
-
- msdu_end_info = __le16_to_cpu(msdu_end->info5);
- is_first_msdu_in_mpdu = u32_get_bits(msdu_end_info,
- RX_MSDU_END_INFO5_FIRST_MSDU);
- if (is_first_msdu_in_mpdu) {
- pmon->mon_mpdu->head = pmon->mon_mpdu->tail;
- pmon->mon_mpdu->tail = NULL;
- }
- break;
- }
+ case HAL_MON_BUF_ADDR:
+ return HAL_RX_MON_STATUS_BUF_ADDR;
+ case HAL_RX_MSDU_END:
+ return HAL_RX_MON_STATUS_MSDU_END;
case HAL_RX_MPDU_END:
- list_add_tail(&pmon->mon_mpdu->list, &pmon->dp_rx_mon_mpdu_list);
- break;
+ return HAL_RX_MON_STATUS_MPDU_END;
case HAL_DUMMY:
return HAL_RX_MON_STATUS_BUF_DONE;
case HAL_RX_PPDU_END_STATUS_DONE:
@@ -1230,7 +1147,10 @@ ath12k_dp_mon_parse_rx_dest(struct ath12k_base *ab, struct ath12k_mon_data *pmon
if ((ptr - skb->data) >= DP_RX_BUFFER_SIZE)
break;
- } while (hal_status == HAL_RX_MON_STATUS_PPDU_NOT_DONE);
+ } while ((hal_status == HAL_RX_MON_STATUS_PPDU_NOT_DONE) ||
+ (hal_status == HAL_RX_MON_STATUS_BUF_ADDR) ||
+ (hal_status == HAL_RX_MON_STATUS_MPDU_END) ||
+ (hal_status == HAL_RX_MON_STATUS_MSDU_END));
return hal_status;
}
@@ -1602,7 +1522,7 @@ ath12k_dp_mon_tx_gen_prot_frame(struct dp_mon_tx_ppdu_info *tx_ppdu_info)
static enum dp_mon_tx_tlv_status
ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
struct ath12k_mon_data *pmon,
- u16 tlv_tag, u8 *tlv_data, u32 userid)
+ u16 tlv_tag, const void *tlv_data, u32 userid)
{
struct dp_mon_tx_ppdu_info *tx_ppdu_info;
enum dp_mon_tx_tlv_status status = DP_MON_TX_STATUS_PPDU_NOT_DONE;
@@ -1612,8 +1532,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
switch (tlv_tag) {
case HAL_TX_FES_SETUP: {
- struct hal_tx_fes_setup *tx_fes_setup =
- (struct hal_tx_fes_setup *)tlv_data;
+ const struct hal_tx_fes_setup *tx_fes_setup = tlv_data;
info[0] = __le32_to_cpu(tx_fes_setup->info0);
tx_ppdu_info->ppdu_id = __le32_to_cpu(tx_fes_setup->schedule_id);
@@ -1624,8 +1543,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_TX_FES_STATUS_END: {
- struct hal_tx_fes_status_end *tx_fes_status_end =
- (struct hal_tx_fes_status_end *)tlv_data;
+ const struct hal_tx_fes_status_end *tx_fes_status_end = tlv_data;
u32 tst_15_0, tst_31_16;
info[0] = __le32_to_cpu(tx_fes_status_end->info0);
@@ -1642,8 +1560,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_RX_RESPONSE_REQUIRED_INFO: {
- struct hal_rx_resp_req_info *rx_resp_req_info =
- (struct hal_rx_resp_req_info *)tlv_data;
+ const struct hal_rx_resp_req_info *rx_resp_req_info = tlv_data;
u32 addr_32;
u16 addr_16;
@@ -1688,8 +1605,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_PCU_PPDU_SETUP_INIT: {
- struct hal_tx_pcu_ppdu_setup_init *ppdu_setup =
- (struct hal_tx_pcu_ppdu_setup_init *)tlv_data;
+ const struct hal_tx_pcu_ppdu_setup_init *ppdu_setup = tlv_data;
u32 addr_32;
u16 addr_16;
@@ -1735,8 +1651,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_TX_QUEUE_EXTENSION: {
- struct hal_tx_queue_exten *tx_q_exten =
- (struct hal_tx_queue_exten *)tlv_data;
+ const struct hal_tx_queue_exten *tx_q_exten = tlv_data;
info[0] = __le32_to_cpu(tx_q_exten->info0);
@@ -1748,8 +1663,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_TX_FES_STATUS_START: {
- struct hal_tx_fes_status_start *tx_fes_start =
- (struct hal_tx_fes_status_start *)tlv_data;
+ const struct hal_tx_fes_status_start *tx_fes_start = tlv_data;
info[0] = __le32_to_cpu(tx_fes_start->info0);
@@ -1760,8 +1674,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_TX_FES_STATUS_PROT: {
- struct hal_tx_fes_status_prot *tx_fes_status =
- (struct hal_tx_fes_status_prot *)tlv_data;
+ const struct hal_tx_fes_status_prot *tx_fes_status = tlv_data;
u32 start_timestamp;
u32 end_timestamp;
@@ -1788,8 +1701,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
case HAL_TX_FES_STATUS_START_PPDU:
case HAL_TX_FES_STATUS_START_PROT: {
- struct hal_tx_fes_status_start_prot *tx_fes_stat_start =
- (struct hal_tx_fes_status_start_prot *)tlv_data;
+ const struct hal_tx_fes_status_start_prot *tx_fes_stat_start = tlv_data;
u64 ppdu_ts;
info[0] = __le32_to_cpu(tx_fes_stat_start->info0);
@@ -1804,8 +1716,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_TX_FES_STATUS_USER_PPDU: {
- struct hal_tx_fes_status_user_ppdu *tx_fes_usr_ppdu =
- (struct hal_tx_fes_status_user_ppdu *)tlv_data;
+ const struct hal_tx_fes_status_user_ppdu *tx_fes_usr_ppdu = tlv_data;
info[0] = __le32_to_cpu(tx_fes_usr_ppdu->info0);
@@ -1848,8 +1759,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
break;
case HAL_RX_FRAME_BITMAP_ACK: {
- struct hal_rx_frame_bitmap_ack *fbm_ack =
- (struct hal_rx_frame_bitmap_ack *)tlv_data;
+ const struct hal_rx_frame_bitmap_ack *fbm_ack = tlv_data;
u32 addr_32;
u16 addr_16;
@@ -1867,8 +1777,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab,
}
case HAL_MACTX_PHY_DESC: {
- struct hal_tx_phy_desc *tx_phy_desc =
- (struct hal_tx_phy_desc *)tlv_data;
+ const struct hal_tx_phy_desc *tx_phy_desc = tlv_data;
info[0] = __le32_to_cpu(tx_phy_desc->info0);
info[1] = __le32_to_cpu(tx_phy_desc->info1);
diff --git a/drivers/net/wireless/ath/ath12k/dp_rx.c b/drivers/net/wireless/ath/ath12k/dp_rx.c
index 1c0d796ffc7a..7b4e01041df9 100644
--- a/drivers/net/wireless/ath/ath12k/dp_rx.c
+++ b/drivers/net/wireless/ath/ath12k/dp_rx.c
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: BSD-3-Clause-Clear
/*
* Copyright (c) 2018-2021 The Linux Foundation. All rights reserved.
- * Copyright (c) 2021-2024 Qualcomm Innovation Center, Inc. All rights reserved.
+ * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
*/
#include <linux/ieee80211.h>
@@ -2905,8 +2905,10 @@ static int ath12k_dp_rx_h_verify_tkip_mic(struct ath12k *ar, struct ath12k_peer
RX_FLAG_IV_STRIPPED | RX_FLAG_DECRYPTED;
skb_pull(msdu, hal_rx_desc_sz);
- if (unlikely(!ath12k_dp_rx_check_nwifi_hdr_len_valid(ab, rx_desc, msdu)))
+ if (unlikely(!ath12k_dp_rx_check_nwifi_hdr_len_valid(ab, rx_desc, msdu))) {
+ dev_kfree_skb_any(msdu);
return -EINVAL;
+ }
ath12k_dp_rx_h_ppdu(ar, rx_desc, rxs);
ath12k_dp_rx_h_undecap(ar, msdu, rx_desc,
@@ -3441,6 +3443,48 @@ ath12k_dp_process_rx_err_buf(struct ath12k *ar, struct hal_reo_dest_ring *desc,
return 0;
}
+static int ath12k_dp_h_msdu_buffer_type(struct ath12k_base *ab,
+ struct list_head *list,
+ struct hal_reo_dest_ring *desc)
+{
+ struct ath12k_rx_desc_info *desc_info;
+ struct ath12k_skb_rxcb *rxcb;
+ struct sk_buff *msdu;
+ u64 desc_va;
+
+ desc_va = (u64)le32_to_cpu(desc->buf_va_hi) << 32 |
+ le32_to_cpu(desc->buf_va_lo);
+ desc_info = (struct ath12k_rx_desc_info *)(uintptr_t)desc_va;
+ if (!desc_info) {
+ u32 cookie;
+
+ cookie = le32_get_bits(desc->buf_addr_info.info1,
+ BUFFER_ADDR_INFO1_SW_COOKIE);
+ desc_info = ath12k_dp_get_rx_desc(ab, cookie);
+ if (!desc_info) {
+ ath12k_warn(ab, "Invalid cookie in manual descriptor retrieval: 0x%x\n",
+ cookie);
+ return -EINVAL;
+ }
+ }
+
+ if (desc_info->magic != ATH12K_DP_RX_DESC_MAGIC) {
+ ath12k_warn(ab, "rx exception, magic check failed with value: %u\n",
+ desc_info->magic);
+ return -EINVAL;
+ }
+
+ msdu = desc_info->skb;
+ desc_info->skb = NULL;
+ list_add_tail(&desc_info->list, list);
+ rxcb = ATH12K_SKB_RXCB(msdu);
+ dma_unmap_single(ab->dev, rxcb->paddr, msdu->len + skb_tailroom(msdu),
+ DMA_FROM_DEVICE);
+ dev_kfree_skb_any(msdu);
+
+ return 0;
+}
+
int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi,
int budget)
{
@@ -3481,6 +3525,19 @@ int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi,
drop = false;
ab->soc_stats.err_ring_pkts++;
+ /* Below case is added to handle data packet from un-associated clients.
+ * As it is expected that AST lookup will fail for
+ * un-associated station's data packets.
+ */
+ if (le32_get_bits(reo_desc->info0, HAL_REO_DEST_RING_INFO0_BUFFER_TYPE) ==
+ HAL_REO_DEST_RING_BUFFER_TYPE_MSDU) {
+ if (!ath12k_dp_h_msdu_buffer_type(ab,
+ &rx_desc_used_list,
+ reo_desc))
+ tot_n_bufs_reaped++;
+ goto next_desc;
+ }
+
ret = ath12k_hal_desc_reo_parse_err(ab, reo_desc, &paddr,
&desc_bank);
if (ret) {
@@ -3529,6 +3586,7 @@ int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi,
tot_n_bufs_reaped++;
}
+next_desc:
if (tot_n_bufs_reaped >= quota) {
tot_n_bufs_reaped = quota;
goto exit;
diff --git a/drivers/net/wireless/ath/ath12k/dp_tx.c b/drivers/net/wireless/ath/ath12k/dp_tx.c
index 21e07b5cee57..f08ab5a4112c 100644
--- a/drivers/net/wireless/ath/ath12k/dp_tx.c
+++ b/drivers/net/wireless/ath/ath12k/dp_tx.c
@@ -10,10 +10,9 @@
#include "hw.h"
static enum hal_tcl_encap_type
-ath12k_dp_tx_get_encap_type(struct ath12k_vif *arvif, struct sk_buff *skb)
+ath12k_dp_tx_get_encap_type(struct ath12k_base *ab, struct sk_buff *skb)
{
struct ieee80211_tx_info *tx_info = IEEE80211_SKB_CB(skb);
- struct ath12k_base *ab = arvif->ar->ab;
if (test_bit(ATH12K_FLAG_RAW_MODE, &ab->dev_flags))
return HAL_TCL_ENCAP_TYPE_RAW;
@@ -289,7 +288,7 @@ int ath12k_dp_tx(struct ath12k *ar, struct ath12k_vif *arvif,
msdu_ext_desc = true;
}
- ti.encap_type = ath12k_dp_tx_get_encap_type(arvif, skb);
+ ti.encap_type = ath12k_dp_tx_get_encap_type(ab, skb);
ti.addr_search_flags = arvif->hal_addr_search_flags;
ti.search_type = arvif->search_type;
ti.type = HAL_TCL_DESC_TYPE_BUFFER;
diff --git a/drivers/net/wireless/ath/ath12k/hal_rx.c b/drivers/net/wireless/ath/ath12k/hal_rx.c
index f7c1aaa3b5d4..ac820333047f 100644
--- a/drivers/net/wireless/ath/ath12k/hal_rx.c
+++ b/drivers/net/wireless/ath/ath12k/hal_rx.c
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: BSD-3-Clause-Clear
/*
* Copyright (c) 2018-2021 The Linux Foundation. All rights reserved.
- * Copyright (c) 2021-2024 Qualcomm Innovation Center, Inc. All rights reserved.
+ * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
*/
#include "debug.h"
@@ -320,7 +320,7 @@ int ath12k_hal_desc_reo_parse_err(struct ath12k_base *ab,
{
enum hal_reo_dest_ring_push_reason push_reason;
enum hal_reo_dest_ring_error_code err_code;
- u32 cookie, val;
+ u32 cookie;
push_reason = le32_get_bits(desc->info0,
HAL_REO_DEST_RING_INFO0_PUSH_REASON);
@@ -335,12 +335,6 @@ int ath12k_hal_desc_reo_parse_err(struct ath12k_base *ab,
return -EINVAL;
}
- val = le32_get_bits(desc->info0, HAL_REO_DEST_RING_INFO0_BUFFER_TYPE);
- if (val != HAL_REO_DEST_RING_BUFFER_TYPE_LINK_DESC) {
- ath12k_warn(ab, "expected buffer type link_desc");
- return -EINVAL;
- }
-
ath12k_hal_rx_reo_ent_paddr_get(ab, &desc->buf_addr_info, paddr, &cookie);
*desc_bank = u32_get_bits(cookie, DP_LINK_DESC_BANK_MASK);
diff --git a/drivers/net/wireless/ath/ath12k/hal_rx.h b/drivers/net/wireless/ath/ath12k/hal_rx.h
index 8c37cbc01b1c..33fb765628c5 100644
--- a/drivers/net/wireless/ath/ath12k/hal_rx.h
+++ b/drivers/net/wireless/ath/ath12k/hal_rx.h
@@ -108,6 +108,9 @@ enum hal_rx_mon_status {
HAL_RX_MON_STATUS_PPDU_NOT_DONE,
HAL_RX_MON_STATUS_PPDU_DONE,
HAL_RX_MON_STATUS_BUF_DONE,
+ HAL_RX_MON_STATUS_BUF_ADDR,
+ HAL_RX_MON_STATUS_MPDU_END,
+ HAL_RX_MON_STATUS_MSDU_END,
};
#define HAL_RX_MAX_MPDU 256
@@ -437,12 +440,15 @@ enum hal_rx_ul_reception_type {
HAL_RECEPTION_TYPE_FRAMELESS
};
-#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RSSI_COMB GENMASK(15, 8)
-#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_RSVD1_RECEPTION GENMASK(3, 0)
+#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RECEPTION GENMASK(3, 0)
+#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RX_BW GENMASK(7, 5)
+#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO1_RSSI_COMB GENMASK(15, 8)
struct hal_rx_phyrx_rssi_legacy_info {
- __le32 rsvd[35];
__le32 info0;
+ __le32 rsvd0[39];
+ __le32 info1;
+ __le32 rsvd1;
} __packed;
#define HAL_RX_MPDU_START_INFO0_PPDU_ID GENMASK(31, 16)
diff --git a/drivers/net/wireless/ath/ath12k/mac.c b/drivers/net/wireless/ath/ath12k/mac.c
index 8e8defddc8fa..6a42c63307c8 100644
--- a/drivers/net/wireless/ath/ath12k/mac.c
+++ b/drivers/net/wireless/ath/ath12k/mac.c
@@ -4271,7 +4271,7 @@ static int ath12k_station_disassoc(struct ath12k *ar,
return 0;
}
-static void ath12k_sta_rc_update_wk(struct work_struct *wk)
+static void ath12k_sta_rc_update_wk(struct wiphy *wiphy, struct wiphy_work *wk)
{
struct ath12k *ar;
struct ath12k_vif *arvif;
@@ -4287,6 +4287,8 @@ static void ath12k_sta_rc_update_wk(struct work_struct *wk)
struct ath12k_wmi_peer_assoc_arg peer_arg;
enum wmi_phy_mode peer_phymode;
+ lockdep_assert_wiphy(wiphy);
+
arsta = container_of(wk, struct ath12k_sta, update_wk);
sta = container_of((void *)arsta, struct ieee80211_sta, drv_priv);
arvif = arsta->arvif;
@@ -4480,10 +4482,12 @@ static int ath12k_mac_station_add(struct ath12k *ar,
goto exit;
}
- arsta->rx_stats = kzalloc(sizeof(*arsta->rx_stats), GFP_KERNEL);
if (!arsta->rx_stats) {
- ret = -ENOMEM;
- goto dec_num_station;
+ arsta->rx_stats = kzalloc(sizeof(*arsta->rx_stats), GFP_KERNEL);
+ if (!arsta->rx_stats) {
+ ret = -ENOMEM;
+ goto dec_num_station;
+ }
}
peer_param.vdev_id = arvif->vdev_id;
@@ -4581,11 +4585,6 @@ static int ath12k_mac_op_sta_state(struct ieee80211_hw *hw,
struct ath12k_peer *peer;
int ret = 0;
- /* cancel must be done outside the mutex to avoid deadlock */
- if ((old_state == IEEE80211_STA_NONE &&
- new_state == IEEE80211_STA_NOTEXIST))
- cancel_work_sync(&arsta->update_wk);
-
ar = ath12k_get_ar_by_vif(hw, vif);
if (!ar) {
WARN_ON_ONCE(1);
@@ -4598,7 +4597,7 @@ static int ath12k_mac_op_sta_state(struct ieee80211_hw *hw,
new_state == IEEE80211_STA_NONE) {
memset(arsta, 0, sizeof(*arsta));
arsta->arvif = arvif;
- INIT_WORK(&arsta->update_wk, ath12k_sta_rc_update_wk);
+ wiphy_work_init(&arsta->update_wk, ath12k_sta_rc_update_wk);
ret = ath12k_mac_station_add(ar, vif, sta);
if (ret)
@@ -4606,6 +4605,8 @@ static int ath12k_mac_op_sta_state(struct ieee80211_hw *hw,
sta->addr, arvif->vdev_id);
} else if ((old_state == IEEE80211_STA_NONE &&
new_state == IEEE80211_STA_NOTEXIST)) {
+ wiphy_work_cancel(hw->wiphy, &arsta->update_wk);
+
if (arvif->vdev_type == WMI_VDEV_TYPE_STA) {
ath12k_bss_disassoc(ar, arvif);
ret = ath12k_mac_vdev_stop(arvif);
@@ -4809,7 +4810,7 @@ static void ath12k_mac_op_sta_rc_update(struct ieee80211_hw *hw,
spin_unlock_bh(&ar->data_lock);
- ieee80211_queue_work(hw, &arsta->update_wk);
+ wiphy_work_queue(hw->wiphy, &arsta->update_wk);
}
static int ath12k_conf_tx_uapsd(struct ath12k_vif *arvif,
@@ -8119,7 +8120,7 @@ static void ath12k_mac_set_bitrate_mask_iter(void *data,
arsta->changed |= IEEE80211_RC_SUPP_RATES_CHANGED;
spin_unlock_bh(&ar->data_lock);
- ieee80211_queue_work(ath12k_ar_to_hw(ar), &arsta->update_wk);
+ wiphy_work_queue(ath12k_ar_to_hw(ar)->wiphy, &arsta->update_wk);
}
static void ath12k_mac_disable_peer_fixed_rate(void *data,
diff --git a/drivers/net/wireless/ath/ath12k/mhi.c b/drivers/net/wireless/ath/ath12k/mhi.c
index df96b0f91f54..af303b9e8acc 100644
--- a/drivers/net/wireless/ath/ath12k/mhi.c
+++ b/drivers/net/wireless/ath/ath12k/mhi.c
@@ -31,7 +31,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_qcn9274[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
},
{
.num = 21,
@@ -45,7 +44,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_qcn9274[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = true,
},
};
@@ -96,7 +94,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_wcn7850[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
},
{
.num = 21,
@@ -110,7 +107,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_wcn7850[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = true,
},
};
diff --git a/drivers/net/wireless/ath/ath12k/wmi.c b/drivers/net/wireless/ath/ath12k/wmi.c
index 598a69f9e6df..ca359eecb42a 100644
--- a/drivers/net/wireless/ath/ath12k/wmi.c
+++ b/drivers/net/wireless/ath/ath12k/wmi.c
@@ -171,6 +171,8 @@ static const struct ath12k_wmi_tlv_policy ath12k_wmi_tlv_policies[] = {
.min_len = sizeof(struct ath12k_wmi_p2p_noa_info) },
[WMI_TAG_P2P_NOA_EVENT] = {
.min_len = sizeof(struct wmi_p2p_noa_event) },
+ [WMI_TAG_PDEV_CSA_SWITCH_COUNT_STATUS_EVENT] = {
+ .min_len = sizeof(struct ath12k_wmi_pdev_csa_event) },
};
static __le32 ath12k_wmi_tlv_hdr(u32 cmd, u32 len)
@@ -384,6 +386,13 @@ ath12k_wmi_tlv_parse_alloc(struct ath12k_base *ab,
return tb;
}
+static u32 ath12k_wmi_tlv_data_len(const void *data)
+{
+ const struct wmi_tlv *tlv = (const struct wmi_tlv *)data - 1;
+
+ return le32_get_bits(tlv->header, WMI_TLV_LEN);
+}
+
static int ath12k_wmi_cmd_send_nowait(struct ath12k_wmi_pdev *wmi, struct sk_buff *skb,
u32 cmd_id)
{
@@ -6735,6 +6744,8 @@ ath12k_wmi_process_csa_switch_count_event(struct ath12k_base *ab,
const struct ath12k_wmi_pdev_csa_event *ev,
const u32 *vdev_ids)
{
+ u32 vdev_ids_len = ath12k_wmi_tlv_data_len(vdev_ids);
+ u32 num_vdevs = le32_to_cpu(ev->num_vdevs);
int i;
struct ath12k_vif *arvif;
@@ -6742,8 +6753,14 @@ ath12k_wmi_process_csa_switch_count_event(struct ath12k_base *ab,
if (ev->current_switch_count)
return;
+ if (num_vdevs > vdev_ids_len / sizeof(*vdev_ids)) {
+ ath12k_warn(ab, "csa switch count num_vdevs %u exceeds tlv array length %u\n",
+ num_vdevs, vdev_ids_len);
+ return;
+ }
+
rcu_read_lock();
- for (i = 0; i < le32_to_cpu(ev->num_vdevs); i++) {
+ for (i = 0; i < num_vdevs; i++) {
arvif = ath12k_mac_get_arvif_by_vdev_id(ab, vdev_ids[i]);
if (!arvif) {
diff --git a/drivers/net/wireless/ath/ath9k/wmi.c b/drivers/net/wireless/ath/ath9k/wmi.c
index 610717fb83f5..805ad31edba2 100644
--- a/drivers/net/wireless/ath/ath9k/wmi.c
+++ b/drivers/net/wireless/ath/ath9k/wmi.c
@@ -206,16 +206,9 @@ static void ath9k_wmi_rsp_callback(struct wmi *wmi, struct sk_buff *skb)
{
skb_pull(skb, sizeof(struct wmi_cmd_hdr));
- if (wmi->cmd_rsp_buf && wmi->cmd_rsp_len) {
- if (skb->len < wmi->cmd_rsp_len) {
- wmi->cmd_rsp_status = -EMSGSIZE;
- goto complete;
- }
+ if (wmi->cmd_rsp_buf != NULL && wmi->cmd_rsp_len != 0)
memcpy(wmi->cmd_rsp_buf, skb->data, wmi->cmd_rsp_len);
- }
- wmi->cmd_rsp_status = 0;
-complete:
complete(&wmi->cmd_wait);
}
@@ -307,7 +300,6 @@ static int ath9k_wmi_cmd_issue(struct wmi *wmi,
/* record the rsp buffer and length */
wmi->cmd_rsp_buf = rsp_buf;
wmi->cmd_rsp_len = rsp_len;
- wmi->cmd_rsp_status = 0;
wmi->last_seq_id = wmi->tx_seq_id;
spin_unlock_irqrestore(&wmi->wmi_lock, flags);
@@ -364,13 +356,9 @@ int ath9k_wmi_cmd(struct wmi *wmi, enum wmi_cmd_id cmd_id,
return -ETIMEDOUT;
}
- spin_lock_irqsave(&wmi->wmi_lock, flags);
- ret = wmi->cmd_rsp_status;
- spin_unlock_irqrestore(&wmi->wmi_lock, flags);
-
mutex_unlock(&wmi->op_mutex);
- return ret;
+ return 0;
out:
ath_dbg(common, WMI, "WMI failure for: %s\n", wmi_cmd_to_name(cmd_id));
diff --git a/drivers/net/wireless/ath/ath9k/wmi.h b/drivers/net/wireless/ath/ath9k/wmi.h
index 0bb9d31f1a1c..5c3b710b8f31 100644
--- a/drivers/net/wireless/ath/ath9k/wmi.h
+++ b/drivers/net/wireless/ath/ath9k/wmi.h
@@ -157,7 +157,6 @@ struct wmi {
u16 tx_seq_id;
u8 *cmd_rsp_buf;
u32 cmd_rsp_len;
- int cmd_rsp_status;
bool stopped;
struct list_head pending_tx_events;
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
index ef08b4163c93..c5fa6f61634a 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
@@ -1281,6 +1281,9 @@ static s32 brcmf_p2p_abort_action_frame(struct brcmf_cfg80211_info *cfg)
brcmf_dbg(TRACE, "Enter\n");
vif = p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif;
+ if (!vif)
+ vif = p2p->bss_idx[P2PAPI_BSSCFG_PRIMARY].vif;
+
err = brcmf_fil_bsscfg_data_set(vif->ifp, "actframe_abort", &int_val,
sizeof(s32));
if (err)
@@ -1817,6 +1820,7 @@ bool brcmf_p2p_send_action_frame(struct brcmf_if *ifp,
/* validate channel and p2p ies */
if (config_af_params.search_channel &&
IS_P2P_SOCIAL_CHANNEL(le32_to_cpu(af_params->channel)) &&
+ p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif &&
p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif->saved_ie.probe_req_ie_len) {
afx_hdl = &p2p->afx_hdl;
afx_hdl->peer_listen_chan = le32_to_cpu(af_params->channel);
diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/main.c b/drivers/net/wireless/intel/iwlwifi/dvm/main.c
index b8713ebd7190..e9b1a1b09eb2 100644
--- a/drivers/net/wireless/intel/iwlwifi/dvm/main.c
+++ b/drivers/net/wireless/intel/iwlwifi/dvm/main.c
@@ -1245,7 +1245,7 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
STATISTICS_NOTIFICATION,
REPLY_TX,
};
- int i;
+ int i, err;
/************************
* 1. Allocating HW data
@@ -1253,6 +1253,7 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
hw = iwl_alloc_all();
if (!hw) {
pr_err("%s: Cannot allocate network device\n", trans->name);
+ err = -ENOMEM;
goto out;
}
@@ -1303,8 +1304,10 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
break;
}
- if (WARN_ON(!priv->lib))
+ if (WARN_ON(!priv->lib)) {
+ err = -ENODEV;
goto out_free_hw;
+ }
/*
* Populate the state variables that the transport layer needs
@@ -1381,14 +1384,16 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
IWL_INFO(priv, "Detected %s, REV=0x%X\n",
priv->trans->name, priv->trans->hw_rev);
- if (iwl_trans_start_hw(priv->trans))
- goto out_free_hw;
+ err = iwl_trans_start_hw(priv->trans);
+ if (err)
+ goto out_leave_trans;
/* Read the EEPROM */
- if (iwl_read_eeprom(priv->trans, &priv->eeprom_blob,
- &priv->eeprom_blob_size)) {
+ err = iwl_read_eeprom(priv->trans, &priv->eeprom_blob,
+ &priv->eeprom_blob_size);
+ if (err) {
IWL_ERR(priv, "Unable to init EEPROM\n");
- goto out_free_hw;
+ goto out_leave_trans;
}
/* Reset chip to save power until we load uCode during "up". */
@@ -1397,13 +1402,17 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
priv->nvm_data = iwl_parse_eeprom_data(priv->trans, priv->cfg,
priv->eeprom_blob,
priv->eeprom_blob_size);
- if (!priv->nvm_data)
+ if (!priv->nvm_data) {
+ err = -ENOMEM;
goto out_free_eeprom_blob;
+ }
- if (iwl_nvm_check_version(priv->nvm_data, priv->trans))
+ err = iwl_nvm_check_version(priv->nvm_data, priv->trans);
+ if (err)
goto out_free_eeprom;
- if (iwl_eeprom_init_hw_params(priv))
+ err = iwl_eeprom_init_hw_params(priv);
+ if (err)
goto out_free_eeprom;
/* extract MAC Address */
@@ -1450,7 +1459,8 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
atomic_set(&priv->queue_stop_count[i], 0);
}
- if (iwl_init_drv(priv))
+ err = iwl_init_drv(priv);
+ if (err)
goto out_free_eeprom;
/* At this point both hw and priv are initialized. */
@@ -1458,7 +1468,8 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
/********************
* 6. Setup services
********************/
- if (iwl_setup_deferred_work(priv))
+ err = iwl_setup_deferred_work(priv);
+ if (err)
goto out_uninit_drv;
iwl_setup_rx_handlers(priv);
@@ -1486,7 +1497,8 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
*
* 7. Setup and register with mac80211 and debugfs
**************************************************/
- if (iwlagn_mac_setup_register(priv, &fw->ucode_capa))
+ err = iwlagn_mac_setup_register(priv, &fw->ucode_capa);
+ if (err)
goto out_destroy_workqueue;
iwl_dbgfs_register(priv, dbgfs_dir);
@@ -1500,15 +1512,16 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
priv->workqueue = NULL;
out_uninit_drv:
iwl_uninit_drv(priv);
-out_free_eeprom_blob:
- kfree(priv->eeprom_blob);
out_free_eeprom:
kfree(priv->nvm_data);
+out_free_eeprom_blob:
+ kfree(priv->eeprom_blob);
+out_leave_trans:
+ iwl_trans_op_mode_leave(priv->trans);
out_free_hw:
ieee80211_free_hw(priv->hw);
out:
- op_mode = NULL;
- return op_mode;
+ return ERR_PTR(err);
}
static void iwl_op_mode_dvm_stop(struct iwl_op_mode *op_mode)
diff --git a/drivers/net/wireless/intel/iwlwifi/fw/smem.c b/drivers/net/wireless/intel/iwlwifi/fw/smem.c
index 3f1272014daf..50ebc81a3748 100644
--- a/drivers/net/wireless/intel/iwlwifi/fw/smem.c
+++ b/drivers/net/wireless/intel/iwlwifi/fw/smem.c
@@ -6,6 +6,7 @@
*/
#include "iwl-drv.h"
#include "runtime.h"
+#include "dbg.h"
#include "fw/api/commands.h"
static void iwl_parse_shared_mem_22000(struct iwl_fw_runtime *fwrt,
@@ -17,7 +18,9 @@ static void iwl_parse_shared_mem_22000(struct iwl_fw_runtime *fwrt,
u8 api_ver = iwl_fw_lookup_notif_ver(fwrt->fw, SYSTEM_GROUP,
SHARED_MEM_CFG_CMD, 0);
- if (WARN_ON(lmac_num > ARRAY_SIZE(mem_cfg->lmac_smem)))
+ /* Note: notification has 3 entries, but we only expect 2 */
+ if (IWL_FW_CHECK(fwrt, lmac_num > ARRAY_SIZE(fwrt->smem_cfg.lmac),
+ "FW advertises %d LMACs\n", lmac_num))
return;
fwrt->smem_cfg.num_lmacs = lmac_num;
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c
index c7c21c79a743..8654f17a1eb8 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c
@@ -1438,7 +1438,7 @@ _iwl_op_mode_start(struct iwl_drv *drv, struct iwlwifi_opmode_table *op)
op_mode = ops->start(drv->trans, drv->trans->cfg,
&drv->fw, dbgfs_dir);
- if (op_mode)
+ if (!IS_ERR(op_mode))
return op_mode;
if (test_bit(STATUS_TRANS_DEAD, &drv->trans->status))
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/binding.c b/drivers/net/wireless/intel/iwlwifi/mvm/binding.c
index 458b97930059..0812522edea0 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/binding.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/binding.c
@@ -2,7 +2,7 @@
/*
* Copyright (C) 2012-2014, 2020 Intel Corporation
* Copyright (C) 2016 Intel Deutschland GmbH
- * Copyright (C) 2022 Intel Corporation
+ * Copyright (C) 2022, 2024, 2026 Intel Corporation
*/
#include <net/mac80211.h>
#include "fw-api.h"
@@ -76,6 +76,9 @@ static void iwl_mvm_iface_iterator(void *_data, u8 *mac,
if (vif == data->ignore_vif)
return;
+ if (vif->type == NL80211_IFTYPE_P2P_DEVICE && !mvmvif->p2p_in_binding)
+ return;
+
if (mvmvif->deflink.phy_ctxt != data->phyctxt)
return;
@@ -158,9 +161,8 @@ int iwl_mvm_binding_remove_vif(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
ret = iwl_mvm_binding_update(mvm, vif, mvmvif->deflink.phy_ctxt,
false);
- if (!ret)
- if (iwl_mvm_sf_update(mvm, vif, true))
- IWL_ERR(mvm, "Failed to update SF state\n");
+ if (!ret && iwl_mvm_sf_update(mvm, vif, true))
+ IWL_ERR(mvm, "Failed to update SF state\n");
return ret;
}
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw.c b/drivers/net/wireless/intel/iwlwifi/mvm/fw.c
index 4bad012472e1..968cb35d6408 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw.c
@@ -490,7 +490,8 @@ static void iwl_mvm_uats_init(struct iwl_mvm *mvm)
.dataflags[0] = IWL_HCMD_DFL_NOCOPY,
};
- if (mvm->trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_AX210) {
+ if (mvm->trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_AX210 ||
+ !mvm->trans->cfg->uhb_supported) {
IWL_DEBUG_RADIO(mvm, "UATS feature is not supported\n");
return;
}
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/link.c b/drivers/net/wireless/intel/iwlwifi/mvm/link.c
index 2b0652168002..e886200924bd 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/link.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/link.c
@@ -410,9 +410,8 @@ int iwl_mvm_remove_link(struct iwl_mvm *mvm, struct ieee80211_vif *vif,
ret = iwl_mvm_link_cmd_send(mvm, &cmd, FW_CTXT_ACTION_REMOVE);
- if (!ret)
- if (iwl_mvm_sf_update(mvm, vif, true))
- IWL_ERR(mvm, "Failed to update SF state\n");
+ if (!ret && iwl_mvm_sf_update(mvm, vif, true))
+ IWL_ERR(mvm, "Failed to update SF state\n");
return ret;
}
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
index 6ac421567705..91d0d2dfef89 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
@@ -1098,6 +1098,7 @@ static void iwl_mvm_cleanup_iterator(void *data, u8 *mac,
spin_unlock_bh(&mvm->time_event_lock);
mvmvif->roc_activity = ROC_NUM_ACTIVITIES;
+ mvmvif->p2p_in_binding = false;
mvmvif->bf_enabled = false;
mvmvif->ba_enabled = false;
@@ -4786,6 +4787,7 @@ static int iwl_mvm_add_aux_sta_for_hs20(struct iwl_mvm *mvm, u32 lmac_id)
static int iwl_mvm_roc_link(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
int ret;
lockdep_assert_held(&mvm->mutex);
@@ -4794,10 +4796,18 @@ static int iwl_mvm_roc_link(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
if (WARN(ret, "Failed binding P2P_DEVICE\n"))
return ret;
+ mvmvif->p2p_in_binding = true;
+
/* The station and queue allocation must be done only after the binding
* is done, as otherwise the FW might incorrectly configure its state.
*/
- return iwl_mvm_add_p2p_bcast_sta(mvm, vif);
+ ret = iwl_mvm_add_p2p_bcast_sta(mvm, vif);
+ if (ret) {
+ iwl_mvm_binding_remove_vif(mvm, vif);
+ mvmvif->p2p_in_binding = false;
+ }
+
+ return ret;
}
static int iwl_mvm_roc(struct ieee80211_hw *hw,
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h
index 67d188409ca3..4e1dbfb3705e 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h
@@ -454,6 +454,8 @@ struct iwl_mvm_esr_exit {
* @unblock_esr_tpt_wk: work for unblocking EMLSR when tpt is high enough.
* @roc_activity: currently running ROC activity for this vif (or
* ROC_NUM_ACTIVITIES if no activity is running).
+ * @p2p_in_binding: indicates that this P2P-Device interface should be
+ * added to the binding, i.e. is running ROC right now
* @session_prot_connection_loss: the connection was lost due to session
* protection ending without receiving a beacon, so we need to now
* protect the deauth separately
@@ -565,6 +567,7 @@ struct iwl_mvm_vif {
struct iwl_mvm_time_event_data time_event_data;
struct iwl_mvm_time_event_data hs_time_event_data;
enum iwl_roc_activity roc_activity;
+ bool p2p_in_binding;
/* TCP Checksum Offload */
netdev_features_t features;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c
index 96de7edf1792..52fd6b2093c9 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c
@@ -1287,6 +1287,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
size_t scan_size;
u32 min_backoff;
struct iwl_mvm_csme_conn_info *csme_conn_info __maybe_unused;
+ int err;
/*
* We use IWL_STATION_COUNT_MAX to check the validity of the station
@@ -1304,7 +1305,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
iwl_mvm_has_mld_api(fw) ? &iwl_mvm_mld_hw_ops :
&iwl_mvm_hw_ops);
if (!hw)
- return NULL;
+ return ERR_PTR(-ENOMEM);
if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ)
max_agg = 512;
@@ -1348,8 +1349,10 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
trans->rx_mpdu_cmd_hdr_size =
sizeof(struct iwl_rx_mpdu_res_start);
- if (WARN_ON(trans->num_rx_queues > 1))
+ if (WARN_ON(trans->num_rx_queues > 1)) {
+ err = -EINVAL;
goto out_free;
+ }
}
mvm->fw_restart = iwlwifi_mod_params.fw_restart ? -1 : 0;
@@ -1426,8 +1429,10 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
iwl_fw_lookup_notif_ver(mvm->fw, LOCATION_GROUP,
TOF_RANGE_RESPONSE_NOTIF, 5);
/* we only support up to version 9 */
- if (WARN_ON_ONCE(mvm->cmd_ver.range_resp > 9))
+ if (WARN_ON_ONCE(mvm->cmd_ver.range_resp > 9)) {
+ err = -EINVAL;
goto out_free;
+ }
/*
* Populate the state variables that the transport layer needs
@@ -1490,6 +1495,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
mvm->phy_db = iwl_phy_db_init(trans);
if (!mvm->phy_db) {
IWL_ERR(mvm, "Cannot init phy_db\n");
+ err = -ENOMEM;
goto out_free;
}
@@ -1502,8 +1508,10 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
scan_size = iwl_mvm_scan_size(mvm);
mvm->scan_cmd = kmalloc(scan_size, GFP_KERNEL);
- if (!mvm->scan_cmd)
+ if (!mvm->scan_cmd) {
+ err = -ENOMEM;
goto out_free;
+ }
mvm->scan_cmd_size = scan_size;
/* invalidate ids to prevent accidental removal of sta_id 0 */
@@ -1532,7 +1540,8 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
iwl_mvm_mei_scan_filter_init(&mvm->mei_scan_filter);
- if (iwl_mvm_start_get_nvm(mvm)) {
+ err = iwl_mvm_start_get_nvm(mvm);
+ if (err) {
/*
* Getting NVM failed while CSME is the owner, but we are
* registered to MEI, we'll get the NVM later when it'll be
@@ -1545,7 +1554,8 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
}
- if (iwl_mvm_start_post_nvm(mvm))
+ err = iwl_mvm_start_post_nvm(mvm);
+ if (err)
goto out_thermal_exit;
return op_mode;
@@ -1565,7 +1575,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg,
iwl_trans_op_mode_leave(trans);
ieee80211_free_hw(mvm->hw);
- return NULL;
+ return ERR_PTR(err);
}
void iwl_mvm_stop_device(struct iwl_mvm *mvm)
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c b/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c
index c0b58e4a4cc1..bdd05062824d 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2012-2014, 2018-2024 Intel Corporation
+ * Copyright (C) 2012-2014, 2018-2024, 2026 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2017 Intel Deutschland GmbH
*/
@@ -90,6 +90,7 @@ static void iwl_mvm_cleanup_roc(struct iwl_mvm *mvm)
} else {
iwl_mvm_rm_p2p_bcast_sta(mvm, vif);
iwl_mvm_binding_remove_vif(mvm, vif);
+ mvmvif->p2p_in_binding = false;
}
/* Do not remove the PHY context as removing and adding
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c
index 8903a5692dfb..a822448ad5ea 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c
@@ -139,9 +139,9 @@ void _iwl_trans_pcie_gen2_stop_device(struct iwl_trans *trans)
if (trans_pcie->is_down)
return;
- if (trans->state >= IWL_TRANS_FW_STARTED)
- if (trans_pcie->fw_reset_handshake)
- iwl_trans_pcie_fw_reset_handshake(trans);
+ if (trans->state >= IWL_TRANS_FW_STARTED &&
+ trans_pcie->fw_reset_handshake)
+ iwl_trans_pcie_fw_reset_handshake(trans);
trans_pcie->is_down = true;
diff --git a/drivers/net/wireless/mediatek/mt76/dma.c b/drivers/net/wireless/mediatek/mt76/dma.c
index e02741c5892e..872d39d46a33 100644
--- a/drivers/net/wireless/mediatek/mt76/dma.c
+++ b/drivers/net/wireless/mediatek/mt76/dma.c
@@ -769,7 +769,11 @@ mt76_dma_rx_cleanup(struct mt76_dev *dev, struct mt76_queue *q)
if (!buf)
break;
- if (!mt76_queue_is_wed_rro(q))
+ if (mtk_wed_device_active(&dev->mmio.wed) &&
+ mt76_queue_is_wed_rro(q))
+ continue;
+
+ if (!mt76_queue_is_wed_rro_ind(q))
mt76_put_page_pool_buf(buf, false);
} while (1);
@@ -1033,10 +1037,6 @@ void mt76_dma_cleanup(struct mt76_dev *dev)
mt76_for_each_q_rx(dev, i) {
struct mt76_queue *q = &dev->q_rx[i];
- if (mtk_wed_device_active(&dev->mmio.wed) &&
- mt76_queue_is_wed_rro(q))
- continue;
-
netif_napi_del(&dev->napi[i]);
mt76_dma_rx_cleanup(dev, q);
diff --git a/drivers/net/wireless/mediatek/mt76/mac80211.c b/drivers/net/wireless/mediatek/mt76/mac80211.c
index e1856664ca5d..cd02dff7ab34 100644
--- a/drivers/net/wireless/mediatek/mt76/mac80211.c
+++ b/drivers/net/wireless/mediatek/mt76/mac80211.c
@@ -791,6 +791,43 @@ void mt76_free_device(struct mt76_dev *dev)
}
EXPORT_SYMBOL_GPL(mt76_free_device);
+static void mt76_reset_phy(struct mt76_phy *phy)
+{
+ if (!phy)
+ return;
+
+ INIT_LIST_HEAD(&phy->tx_list);
+}
+
+void mt76_reset_device(struct mt76_dev *dev)
+{
+ int i;
+
+ rcu_read_lock();
+ for (i = 0; i < ARRAY_SIZE(dev->wcid); i++) {
+ struct mt76_wcid *wcid;
+
+ wcid = rcu_dereference(dev->wcid[i]);
+ if (!wcid)
+ continue;
+
+ wcid->sta = 0;
+ mt76_wcid_cleanup(dev, wcid);
+ rcu_assign_pointer(dev->wcid[i], NULL);
+ }
+ rcu_read_unlock();
+
+ INIT_LIST_HEAD(&dev->wcid_list);
+ INIT_LIST_HEAD(&dev->sta_poll_list);
+ dev->vif_mask = 0;
+ memset(dev->wcid_mask, 0, sizeof(dev->wcid_mask));
+
+ mt76_reset_phy(&dev->phy);
+ for (i = 0; i < ARRAY_SIZE(dev->phys); i++)
+ mt76_reset_phy(dev->phys[i]);
+}
+EXPORT_SYMBOL_GPL(mt76_reset_device);
+
static void mt76_rx_release_amsdu(struct mt76_phy *phy, enum mt76_rxq_id q)
{
struct sk_buff *skb = phy->rx_amsdu[q].head;
@@ -1449,6 +1486,7 @@ mt76_sta_add(struct mt76_phy *phy, struct ieee80211_vif *vif,
{
struct mt76_wcid *wcid = (struct mt76_wcid *)sta->drv_priv;
struct mt76_dev *dev = phy->dev;
+ struct mt76_wcid *published;
int ret;
int i;
@@ -1468,13 +1506,22 @@ mt76_sta_add(struct mt76_phy *phy, struct ieee80211_vif *vif,
mtxq->wcid = wcid->idx;
}
- ewma_signal_init(&wcid->rssi);
- if (phy->band_idx == MT_BAND1)
- mt76_wcid_mask_set(dev->wcid_phy_mask, wcid->idx);
- wcid->phy_idx = phy->band_idx;
- rcu_assign_pointer(dev->wcid[wcid->idx], wcid);
+ published = rcu_dereference_protected(dev->wcid[wcid->idx],
+ lockdep_is_held(&dev->mutex));
+ if (published != wcid) {
+ WARN_ON_ONCE(published);
+ ewma_signal_init(&wcid->rssi);
+ if (phy->band_idx == MT_BAND1)
+ mt76_wcid_mask_set(dev->wcid_phy_mask, wcid->idx);
+ wcid->phy_idx = phy->band_idx;
+ rcu_assign_pointer(dev->wcid[wcid->idx], wcid);
+ mt76_wcid_init(wcid);
+ } else {
+ if (phy->band_idx == MT_BAND1)
+ mt76_wcid_mask_set(dev->wcid_phy_mask, wcid->idx);
+ wcid->phy_idx = phy->band_idx;
+ }
- mt76_wcid_init(wcid);
out:
mutex_unlock(&dev->mutex);
diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h
index daa78f52c15e..8addbffa97c5 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76.h
+++ b/drivers/net/wireless/mediatek/mt76/mt76.h
@@ -1161,6 +1161,15 @@ static inline int mt76_wed_dma_setup(struct mt76_dev *dev, struct mt76_queue *q,
for (i = 0; i < ARRAY_SIZE((dev)->q_rx); i++) \
if ((dev)->q_rx[i].ndesc)
+static inline struct mt76_dev *mt76_wed_to_dev(struct mtk_wed_device *wed)
+{
+#ifdef CONFIG_NET_MEDIATEK_SOC_WED
+ if (wed->wlan.hif2)
+ return container_of(wed, struct mt76_dev, mmio.wed_hif2);
+#endif /* CONFIG_NET_MEDIATEK_SOC_WED */
+ return container_of(wed, struct mt76_dev, mmio.wed);
+}
+
struct mt76_dev *mt76_alloc_device(struct device *pdev, unsigned int size,
const struct ieee80211_ops *ops,
const struct mt76_driver_ops *drv_ops);
@@ -1168,6 +1177,7 @@ int mt76_register_device(struct mt76_dev *dev, bool vht,
struct ieee80211_rate *rates, int n_rates);
void mt76_unregister_device(struct mt76_dev *dev);
void mt76_free_device(struct mt76_dev *dev);
+void mt76_reset_device(struct mt76_dev *dev);
void mt76_unregister_phy(struct mt76_phy *phy);
struct mt76_phy *mt76_alloc_phy(struct mt76_dev *dev, unsigned int size,
diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
index f60fe27ab3f9..800d1073df27 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
@@ -1220,6 +1220,9 @@ int mt76_connac_mcu_uni_add_dev(struct mt76_phy *phy,
len = enable ? sizeof(dev_req) : sizeof(basic_req);
err = mt76_mcu_send_msg(dev, cmd, data, len, true);
+ if (err && cmd == MCU_UNI_CMD(BSS_INFO_UPDATE))
+ err = mt76_connac_mcu_bss_deact_err(dev, err, enable);
+
if (err < 0)
return err;
@@ -1227,7 +1230,11 @@ int mt76_connac_mcu_uni_add_dev(struct mt76_phy *phy,
data = enable ? (void *)&basic_req : (void *)&dev_req;
len = enable ? sizeof(basic_req) : sizeof(dev_req);
- return mt76_mcu_send_msg(dev, cmd, data, len, true);
+ err = mt76_mcu_send_msg(dev, cmd, data, len, true);
+ if (err && cmd == MCU_UNI_CMD(BSS_INFO_UPDATE))
+ err = mt76_connac_mcu_bss_deact_err(dev, err, enable);
+
+ return err;
}
EXPORT_SYMBOL_GPL(mt76_connac_mcu_uni_add_dev);
diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h
index 6901971f5da6..051523090ea7 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h
+++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h
@@ -1876,6 +1876,29 @@ mt76_connac_mcu_get_wlan_idx(struct mt76_dev *dev, struct mt76_wcid *wcid,
}
}
+#define MT76_CONNAC_MCU_STATUS_WLAN_FAILURE 0xc0000001
+
+static inline int
+mt76_connac_mcu_bss_deact_err(struct mt76_dev *mdev, int err, bool enable)
+{
+ if (err != (int)MT76_CONNAC_MCU_STATUS_WLAN_FAILURE)
+ return err;
+
+ /* Ignore wlan_failure state false alarm when deactivating an
+ * inactive network. It does not harm the firmware state.
+ */
+ if (!enable) {
+ dev_dbg(mdev->dev,
+ "ignore wlan_failure when bss is deactivated\n");
+ return 0;
+ }
+
+ dev_warn(mdev->dev,
+ "wlan_failure when bss is activated\n");
+
+ return err;
+}
+
struct sk_buff *
__mt76_connac_mcu_alloc_sta_req(struct mt76_dev *dev, struct mt76_vif *mvif,
struct mt76_wcid *wcid, int len);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/init.c b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
index 9b0f326c231c..cd18540dc98a 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/init.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
@@ -1288,6 +1288,8 @@ int mt7915_register_device(struct mt7915_dev *dev)
void mt7915_unregister_device(struct mt7915_dev *dev)
{
cancel_work_sync(&dev->dump_work);
+ cancel_work_sync(&dev->reset_work);
+ cancel_work_sync(&dev->rc_work);
mt7915_unregister_ext_phy(dev);
mt7915_coredump_unregister(dev);
mt7915_unregister_thermal(&dev->phy);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c
index f165e61ed28f..147c0fbfbeb8 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c
@@ -1458,17 +1458,15 @@ mt7915_mac_full_reset(struct mt7915_dev *dev)
if (i == 10)
dev_err(dev->mt76.dev, "chip full reset failed\n");
- spin_lock_bh(&dev->mt76.sta_poll_lock);
- while (!list_empty(&dev->mt76.sta_poll_list))
- list_del_init(dev->mt76.sta_poll_list.next);
- spin_unlock_bh(&dev->mt76.sta_poll_lock);
-
- memset(dev->mt76.wcid_mask, 0, sizeof(dev->mt76.wcid_mask));
- dev->mt76.vif_mask = 0;
dev->phy.omac_mask = 0;
if (phy2)
phy2->omac_mask = 0;
+ mt76_reset_device(&dev->mt76);
+
+ INIT_LIST_HEAD(&dev->sta_rc_list);
+ INIT_LIST_HEAD(&dev->twt_list);
+
i = mt76_wcid_alloc(dev->mt76.wcid_mask, MT7915_WTBL_STA);
dev->mt76.global_wcid.idx = i;
dev->recovery.hw_full_reset = false;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/main.c b/drivers/net/wireless/mediatek/mt76/mt7925/main.c
index deb226cf5413..9a52607dc5a8 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7925/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7925/main.c
@@ -473,12 +473,16 @@ void mt7925_roc_abort_sync(struct mt792x_dev *dev)
{
struct mt792x_phy *phy = &dev->phy;
+ if (!test_and_clear_bit(MT76_STATE_ROC, &phy->mt76->state))
+ return;
+
del_timer_sync(&phy->roc_timer);
- cancel_work_sync(&phy->roc_work);
- if (test_and_clear_bit(MT76_STATE_ROC, &phy->mt76->state))
- ieee80211_iterate_interfaces(mt76_hw(dev),
- IEEE80211_IFACE_ITER_RESUME_ALL,
- mt7925_roc_iter, (void *)phy);
+
+ cancel_work(&phy->roc_work);
+
+ ieee80211_iterate_interfaces(mt76_hw(dev),
+ IEEE80211_IFACE_ITER_RESUME_ALL,
+ mt7925_roc_iter, (void *)phy);
}
EXPORT_SYMBOL_GPL(mt7925_roc_abort_sync);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c
index 789acc53d8a8..d6437a0fff6a 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7925/mcu.c
@@ -2788,6 +2788,7 @@ int mt7925_mcu_add_bss_info(struct mt792x_phy *phy,
struct mt792x_dev *dev = phy->dev;
struct mt792x_link_sta *mlink_bc;
struct sk_buff *skb;
+ int err;
skb = __mt7925_mcu_alloc_bss_req(&dev->mt76, &mconf->mt76,
MT7925_BSS_UPDATE_MAX_SIZE);
@@ -2813,8 +2814,9 @@ int mt7925_mcu_add_bss_info(struct mt792x_phy *phy,
if (enable)
mt7925_mcu_bss_rlm_tlv(skb, phy->mt76, link_conf, ctx);
- return mt76_mcu_skb_send_msg(&dev->mt76, skb,
- MCU_UNI_CMD(BSS_INFO_UPDATE), true);
+ err = mt76_mcu_skb_send_msg(&dev->mt76, skb,
+ MCU_UNI_CMD(BSS_INFO_UPDATE), true);
+ return mt76_connac_mcu_bss_deact_err(&dev->mt76, err, enable);
}
int mt7925_mcu_set_dbdc(struct mt76_phy *phy, bool enable)
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/init.c b/drivers/net/wireless/mediatek/mt76/mt7996/init.c
index d3d63f07d7b0..6da42d941165 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/init.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/init.c
@@ -1405,6 +1405,8 @@ void mt7996_unregister_device(struct mt7996_dev *dev)
{
cancel_work_sync(&dev->dump_work);
cancel_work_sync(&dev->wed_rro.work);
+ cancel_work_sync(&dev->reset_work);
+ cancel_work_sync(&dev->rc_work);
mt7996_unregister_phy(mt7996_phy3(dev), MT_BAND2);
mt7996_unregister_phy(mt7996_phy2(dev), MT_BAND1);
mt7996_unregister_thermal(&dev->phy);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c
index 24e0346ea3d8..89d8a22e967c 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c
@@ -1916,6 +1916,9 @@ void mt7996_mac_reset_work(struct work_struct *work)
set_bit(MT76_RESET, &phy3->mt76->state);
cancel_delayed_work_sync(&phy3->mt76->mac_work);
}
+
+ mutex_lock(&dev->mt76.mutex);
+
mt76_worker_disable(&dev->mt76.tx_worker);
mt76_for_each_q_rx(&dev->mt76, i) {
if (mtk_wed_device_active(&dev->mt76.mmio.wed) &&
@@ -1926,8 +1929,6 @@ void mt7996_mac_reset_work(struct work_struct *work)
}
napi_disable(&dev->mt76.tx_napi);
- mutex_lock(&dev->mt76.mutex);
-
mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_DMA_STOPPED);
if (mt7996_wait_reset_state(dev, MT_MCU_CMD_RESET_DONE)) {
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c b/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c
index 01d4b54a1acc..ba1667ac1d36 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/mmio.c
@@ -312,6 +312,9 @@ int mt7996_mmio_wed_init(struct mt7996_dev *dev, void *pdev_ptr,
if (!wed_enable)
return 0;
+ if (hif2 && !mtk_wed_device_active(&dev->mt76.mmio.wed))
+ return 0;
+
dev->has_rro = true;
hif1_ofs = MT_WFDMA0_PCIE1(0) - MT_WFDMA0(0);
@@ -410,6 +413,7 @@ int mt7996_mmio_wed_init(struct mt7996_dev *dev, void *pdev_ptr,
wed->wlan.nbuf = MT7996_HW_TOKEN_SIZE;
wed->wlan.token_start = MT7996_TOKEN_SIZE - wed->wlan.nbuf;
+ wed->wlan.hif2 = hif2;
wed->wlan.amsdu_max_subframes = 8;
wed->wlan.amsdu_max_len = 1536;
diff --git a/drivers/net/wireless/mediatek/mt76/wed.c b/drivers/net/wireless/mediatek/mt76/wed.c
index f89e4537555c..6798b56bc555 100644
--- a/drivers/net/wireless/mediatek/mt76/wed.c
+++ b/drivers/net/wireless/mediatek/mt76/wed.c
@@ -8,7 +8,7 @@
void mt76_wed_release_rx_buf(struct mtk_wed_device *wed)
{
- struct mt76_dev *dev = container_of(wed, struct mt76_dev, mmio.wed);
+ struct mt76_dev *dev = mt76_wed_to_dev(wed);
int i;
for (i = 0; i < dev->rx_token_size; i++) {
@@ -31,8 +31,8 @@ EXPORT_SYMBOL_GPL(mt76_wed_release_rx_buf);
#ifdef CONFIG_NET_MEDIATEK_SOC_WED
u32 mt76_wed_init_rx_buf(struct mtk_wed_device *wed, int size)
{
- struct mt76_dev *dev = container_of(wed, struct mt76_dev, mmio.wed);
struct mtk_wed_bm_desc *desc = wed->rx_buf_ring.desc;
+ struct mt76_dev *dev = mt76_wed_to_dev(wed);
struct mt76_queue *q = &dev->q_rx[MT_RXQ_MAIN];
int i, len = SKB_WITH_OVERHEAD(q->buf_size);
struct mt76_txwi_cache *t = NULL;
@@ -84,7 +84,7 @@ EXPORT_SYMBOL_GPL(mt76_wed_init_rx_buf);
int mt76_wed_offload_enable(struct mtk_wed_device *wed)
{
- struct mt76_dev *dev = container_of(wed, struct mt76_dev, mmio.wed);
+ struct mt76_dev *dev = mt76_wed_to_dev(wed);
spin_lock_bh(&dev->token_lock);
dev->token_size = wed->wlan.token_start;
@@ -168,7 +168,7 @@ EXPORT_SYMBOL_GPL(mt76_wed_dma_setup);
void mt76_wed_offload_disable(struct mtk_wed_device *wed)
{
- struct mt76_dev *dev = container_of(wed, struct mt76_dev, mmio.wed);
+ struct mt76_dev *dev = mt76_wed_to_dev(wed);
spin_lock_bh(&dev->token_lock);
dev->token_size = dev->drv->token_size;
@@ -178,7 +178,7 @@ EXPORT_SYMBOL_GPL(mt76_wed_offload_disable);
void mt76_wed_reset_complete(struct mtk_wed_device *wed)
{
- struct mt76_dev *dev = container_of(wed, struct mt76_dev, mmio.wed);
+ struct mt76_dev *dev = mt76_wed_to_dev(wed);
complete(&dev->mmio.wed_reset_complete);
}
diff --git a/drivers/net/wireless/realtek/rtw89/pci.c b/drivers/net/wireless/realtek/rtw89/pci.c
index 48415d108045..d24dcbf26caa 100644
--- a/drivers/net/wireless/realtek/rtw89/pci.c
+++ b/drivers/net/wireless/realtek/rtw89/pci.c
@@ -578,11 +578,16 @@ static void rtw89_pci_release_rpp(struct rtw89_dev *rtwdev,
tx_status = le32_get_bits(rpp->dword, RTW89_PCI_RPP_TX_STATUS);
txch = rtw89_core_get_ch_dma(rtwdev, qsel);
- if (txch == RTW89_TXCH_CH12) {
+ if (unlikely(txch == RTW89_TXCH_CH12)) {
rtw89_warn(rtwdev, "should no fwcmd release report\n");
return;
}
+ if (unlikely(seq >= RTW89_PCI_TXWD_NUM_MAX)) {
+ rtw89_warn(rtwdev, "invalid seq %d\n", seq);
+ return;
+ }
+
tx_ring = &rtwpci->tx_rings[txch];
wd_ring = &tx_ring->wd_ring;
txwd = &wd_ring->pages[seq];
@@ -3086,17 +3091,26 @@ static bool rtw89_pci_is_dac_compatible_bridge(struct rtw89_dev *rtwdev)
return false;
}
-static void rtw89_pci_cfg_dac(struct rtw89_dev *rtwdev)
+static int rtw89_pci_cfg_dac(struct rtw89_dev *rtwdev, bool force)
{
struct rtw89_pci *rtwpci = (struct rtw89_pci *)rtwdev->priv;
+ struct pci_dev *pdev = rtwpci->pdev;
+ int ret;
+ u8 val;
- if (!rtwpci->enable_dac)
- return;
+ if (!rtwpci->enable_dac && !force)
+ return 0;
if (!rtw89_pci_chip_is_manual_dac(rtwdev))
- return;
+ return 0;
+
+ /* Configure DAC only via PCI config API, not DBI interfaces */
+ ret = pci_read_config_byte(pdev, RTW89_PCIE_L1_CTRL, &val);
+ if (ret)
+ return ret;
- rtw89_pci_config_byte_set(rtwdev, RTW89_PCIE_L1_CTRL, RTW89_PCIE_BIT_EN_64BITS);
+ val |= RTW89_PCIE_BIT_EN_64BITS;
+ return pci_write_config_byte(pdev, RTW89_PCIE_L1_CTRL, val);
}
static int rtw89_pci_setup_mapping(struct rtw89_dev *rtwdev,
@@ -3114,13 +3128,16 @@ static int rtw89_pci_setup_mapping(struct rtw89_dev *rtwdev,
}
if (!rtw89_pci_is_dac_compatible_bridge(rtwdev))
- goto no_dac;
+ goto try_dac_done;
ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(36));
if (!ret) {
- rtwpci->enable_dac = true;
- rtw89_pci_cfg_dac(rtwdev);
- } else {
+ ret = rtw89_pci_cfg_dac(rtwdev, true);
+ if (!ret) {
+ rtwpci->enable_dac = true;
+ goto try_dac_done;
+ }
+
ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
if (ret) {
rtw89_err(rtwdev,
@@ -3128,7 +3145,7 @@ static int rtw89_pci_setup_mapping(struct rtw89_dev *rtwdev,
goto err_release_regions;
}
}
-no_dac:
+try_dac_done:
resource_len = pci_resource_len(pdev, bar_id);
rtwpci->mmap = pci_iomap(pdev, bar_id, resource_len);
@@ -4267,7 +4284,7 @@ static int __maybe_unused rtw89_pci_resume(struct device *dev)
rtw89_pci_hci_ldo(rtwdev);
rtw89_pci_l2_hci_ldo(rtwdev);
rtw89_pci_disable_eq(rtwdev);
- rtw89_pci_cfg_dac(rtwdev);
+ rtw89_pci_cfg_dac(rtwdev, false);
rtw89_pci_filter_out(rtwdev);
rtw89_pci_link_cfg(rtwdev);
rtw89_pci_l1ss_cfg(rtwdev);
diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index b9e465db2015..398b0e0ff842 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -211,6 +211,7 @@ struct ntb_transport_mw {
void *alloc_addr;
void *virt_addr;
dma_addr_t dma_addr;
+ dma_addr_t original_dma_addr;
};
struct ntb_transport_client_dev {
@@ -793,14 +794,14 @@ static void ntb_transport_msi_desc_changed(void *data)
static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw)
{
struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
- struct pci_dev *pdev = nt->ndev->pdev;
+ struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
if (!mw->virt_addr)
return;
ntb_mw_clear_trans(nt->ndev, PIDX, num_mw);
- dma_free_coherent(&pdev->dev, mw->alloc_size,
- mw->alloc_addr, mw->dma_addr);
+ dma_free_coherent(dma_dev, mw->alloc_size,
+ mw->alloc_addr, mw->original_dma_addr);
mw->xlat_size = 0;
mw->buff_size = 0;
mw->alloc_size = 0;
@@ -836,6 +837,7 @@ static int ntb_alloc_mw_buffer(struct ntb_transport_mw *mw,
return -ENOMEM;
}
virt_addr = alloc_addr;
+ mw->original_dma_addr = dma_addr;
/*
* we must ensure that the memory address allocated is BAR size
@@ -869,7 +871,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
resource_size_t size)
{
struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
- struct pci_dev *pdev = nt->ndev->pdev;
+ struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
size_t xlat_size, buff_size;
resource_size_t xlat_align;
resource_size_t xlat_align_size;
@@ -898,12 +900,12 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
mw->buff_size = buff_size;
mw->alloc_size = buff_size;
- rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+ rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
if (rc) {
mw->alloc_size *= 2;
- rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+ rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
if (rc) {
- dev_err(&pdev->dev,
+ dev_err(dma_dev,
"Unable to alloc aligned MW buff\n");
mw->xlat_size = 0;
mw->buff_size = 0;
@@ -916,7 +918,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
rc = ntb_mw_set_trans(nt->ndev, PIDX, num_mw, mw->dma_addr,
mw->xlat_size);
if (rc) {
- dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw);
+ dev_err(dma_dev, "Unable to set mw%d translation", num_mw);
ntb_free_mw(nt, num_mw);
return -EIO;
}
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 0f1956bb6d3c..72a5de1f8252 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -828,7 +828,7 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask)
void nvmet_execute_set_features(struct nvmet_req *req)
{
- struct nvmet_subsys *subsys = nvmet_req_subsys(req);
+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
u16 status = 0;
@@ -847,7 +847,7 @@ void nvmet_execute_set_features(struct nvmet_req *req)
break;
}
nvmet_set_result(req,
- (subsys->max_qid - 1) | ((subsys->max_qid - 1) << 16));
+ (ctrl->max_qid - 1) | ((ctrl->max_qid - 1) << 16));
break;
case NVME_FEAT_KATO:
status = nvmet_set_feat_kato(req);
@@ -902,7 +902,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req)
void nvmet_execute_get_features(struct nvmet_req *req)
{
- struct nvmet_subsys *subsys = nvmet_req_subsys(req);
+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u16 status = 0;
@@ -939,7 +939,7 @@ void nvmet_execute_get_features(struct nvmet_req *req)
break;
case NVME_FEAT_NUM_QUEUES:
nvmet_set_result(req,
- (subsys->max_qid-1) | ((subsys->max_qid-1) << 16));
+ (ctrl->max_qid-1) | ((ctrl->max_qid-1) << 16));
break;
case NVME_FEAT_KATO:
nvmet_get_feat_kato(req);
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 5950635d6f58..fa011c16c9b1 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1479,21 +1479,6 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
if (!ctrl->changed_ns_list)
goto out_free_ctrl;
- ctrl->sqs = kcalloc(subsys->max_qid + 1,
- sizeof(struct nvmet_sq *),
- GFP_KERNEL);
- if (!ctrl->sqs)
- goto out_free_changed_ns_list;
-
- ret = ida_alloc_range(&cntlid_ida,
- subsys->cntlid_min, subsys->cntlid_max,
- GFP_KERNEL);
- if (ret < 0) {
- status = NVME_SC_CONNECT_CTRL_BUSY | NVME_STATUS_DNR;
- goto out_free_sqs;
- }
- ctrl->cntlid = ret;
-
/*
* Discovery controllers may use some arbitrary high value
* in order to cleanup stale discovery sessions
@@ -1507,13 +1492,33 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
ctrl->err_counter = 0;
spin_lock_init(&ctrl->error_lock);
- nvmet_start_keep_alive_timer(ctrl);
-
+ down_read(&nvmet_config_sem);
mutex_lock(&subsys->lock);
+
+ ctrl->max_qid = subsys->max_qid;
+
+ ctrl->sqs = kcalloc(ctrl->max_qid + 1,
+ sizeof(struct nvmet_sq *),
+ GFP_KERNEL);
+ if (!ctrl->sqs)
+ goto out_free_changed_ns_list;
+
+ ret = ida_alloc_range(&cntlid_ida,
+ subsys->cntlid_min, subsys->cntlid_max,
+ GFP_KERNEL);
+ if (ret < 0) {
+ status = NVME_SC_CONNECT_CTRL_BUSY | NVME_STATUS_DNR;
+ goto out_free_sqs;
+ }
+ ctrl->cntlid = ret;
+
list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
nvmet_setup_p2p_ns_map(ctrl, req);
nvmet_debugfs_ctrl_setup(ctrl);
mutex_unlock(&subsys->lock);
+ up_read(&nvmet_config_sem);
+
+ nvmet_start_keep_alive_timer(ctrl);
*ctrlp = ctrl;
return 0;
@@ -1521,6 +1526,8 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
out_free_sqs:
kfree(ctrl->sqs);
out_free_changed_ns_list:
+ mutex_unlock(&subsys->lock);
+ up_read(&nvmet_config_sem);
kfree(ctrl->changed_ns_list);
out_free_ctrl:
kfree(ctrl);
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index a593003c1c20..66d0dbf7ac73 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -350,7 +350,7 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
goto out;
}
- if (unlikely(qid > ctrl->subsys->max_qid)) {
+ if (unlikely(qid > ctrl->max_qid)) {
pr_warn("invalid queue id (%d)\n", qid);
status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
req->cqe->result.u32 = IPO_IATTR_CONNECT_SQE(qid);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 25bba2d09762..408c3cda4743 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -204,6 +204,7 @@ struct nvmet_ctrl {
uuid_t hostid;
u16 cntlid;
+ u16 max_qid;
u32 kato;
struct nvmet_port *port;
@@ -642,6 +643,11 @@ static inline struct nvmet_subsys *nvmet_req_subsys(struct nvmet_req *req)
return req->sq->ctrl->subsys;
}
+static inline struct nvmet_ctrl *nvmet_req_ctrl(struct nvmet_req *req)
+{
+ return req->sq->ctrl;
+}
+
static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys)
{
return subsys->type != NVME_NQN_NVME;
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 5509737bb4cb..fcb1a3dbff01 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1906,10 +1906,12 @@ static void nvmet_tcp_tls_handshake_done(void *data, int status,
spin_unlock_bh(&queue->state_lock);
cancel_delayed_work_sync(&queue->tls_handshake_tmo_work);
+
+ if (!status)
+ status = nvmet_tcp_set_queue_sock(queue);
+
if (status)
nvmet_tcp_schedule_release_queue(queue);
- else
- nvmet_tcp_set_queue_sock(queue);
kref_put(&queue->kref, nvmet_tcp_release_queue);
}
diff --git a/drivers/opp/core.c b/drivers/opp/core.c
index b5df41ce3aff..46baa5c36c0d 100644
--- a/drivers/opp/core.c
+++ b/drivers/opp/core.c
@@ -2026,11 +2026,10 @@ int _opp_add(struct device *dev, struct dev_pm_opp *new_opp,
return ret;
}
- list_add(&new_opp->node, head);
- mutex_unlock(&opp_table->lock);
-
new_opp->opp_table = opp_table;
kref_init(&new_opp->kref);
+ list_add(&new_opp->node, head);
+ mutex_unlock(&opp_table->lock);
opp_debug_create_one(new_opp, opp_table);
diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index bdb3e10f947a..b7f4510c284c 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -13,6 +13,7 @@
#include <linux/ioport.h>
#include <linux/of.h>
#include <linux/of_platform.h>
+#include <linux/platform_device.h>
#include <linux/proc_fs.h>
#include <linux/slab.h>
@@ -329,6 +330,7 @@ void __weak pcibios_bus_add_device(struct pci_dev *pdev) { }
void pci_bus_add_device(struct pci_dev *dev)
{
struct device_node *dn = dev->dev.of_node;
+ struct platform_device *pdev;
int retval;
/*
@@ -354,11 +356,22 @@ void pci_bus_add_device(struct pci_dev *dev)
pci_dev_assign_added(dev, true);
if (dev_of_node(&dev->dev) && pci_is_bridge(dev)) {
- retval = of_platform_populate(dev_of_node(&dev->dev), NULL, NULL,
- &dev->dev);
- if (retval)
- pci_err(dev, "failed to populate child OF nodes (%d)\n",
- retval);
+ for_each_available_child_of_node_scoped(dn, child) {
+ /*
+ * First check whether the pwrctl device needs to be
+ * created or not. This is decided based on at least
+ * one of the power supplies being defined in the
+ * devicetree node of the device.
+ */
+ if (!of_pci_supply_present(child)) {
+ pci_dbg(dev, "skipping OF node: %s\n", child->name);
+ continue;
+ }
+
+ pdev = of_platform_device_create(child, NULL, &dev->dev);
+ if (!pdev)
+ pci_err(dev, "failed to create OF node: %s\n", child->name);
+ }
}
}
EXPORT_SYMBOL_GPL(pci_bus_add_device);
diff --git a/drivers/pci/endpoint/functions/pci-epf-ntb.c b/drivers/pci/endpoint/functions/pci-epf-ntb.c
index d12cf226f328..040c32e438be 100644
--- a/drivers/pci/endpoint/functions/pci-epf-ntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-ntb.c
@@ -2075,6 +2075,11 @@ static int __init epf_ntb_init(void)
kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM |
WQ_HIGHPRI, 0);
+ if (!kpcintb_workqueue) {
+ pr_err("Failed to allocate kpcintb workqueue\n");
+ return -ENOMEM;
+ }
+
ret = pci_epf_register_driver(&epf_ntb_driver);
if (ret) {
destroy_workqueue(kpcintb_workqueue);
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index f9786637d21c..963a1a1f983d 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -1418,6 +1418,11 @@ static int __init epf_ntb_init(void)
kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM |
WQ_HIGHPRI, 0);
+ if (!kpcintb_workqueue) {
+ pr_err("Failed to allocate kpcintb workqueue\n");
+ return -ENOMEM;
+ }
+
ret = pci_epf_register_driver(&epf_ntb_driver);
if (ret) {
destroy_workqueue(kpcintb_workqueue);
diff --git a/drivers/pci/of.c b/drivers/pci/of.c
index dacea3fc5128..52f770bcc481 100644
--- a/drivers/pci/of.c
+++ b/drivers/pci/of.c
@@ -728,6 +728,33 @@ void of_pci_make_dev_node(struct pci_dev *pdev)
}
#endif
+/**
+ * of_pci_supply_present() - Check if the power supply is present for the PCI
+ * device
+ * @np: Device tree node
+ *
+ * Check if the power supply for the PCI device is present in the device tree
+ * node or not.
+ *
+ * Return: true if at least one power supply exists; false otherwise.
+ */
+bool of_pci_supply_present(struct device_node *np)
+{
+ struct property *prop;
+ char *supply;
+
+ if (!np)
+ return false;
+
+ for_each_property_of_node(np, prop) {
+ supply = strrchr(prop->name, '-');
+ if (supply && !strcmp(supply, "-supply"))
+ return true;
+ }
+
+ return false;
+}
+
#endif /* CONFIG_PCI */
/**
diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
index 0eb246a07c87..f4828252e222 100644
--- a/drivers/pci/pci-driver.c
+++ b/drivers/pci/pci-driver.c
@@ -29,6 +29,47 @@ struct pci_dynid {
struct pci_device_id id;
};
+/**
+ * do_pci_add_dynid - Add a new PCI device ID to this driver and re-probe
+ * @drv: target PCI driver
+ * @id: ID to be added
+ * @check_dup: whether to check if matching ID is already present
+ *
+ * Add a new dynamic PCI device ID to this driver and causes the driver to
+ * probe for all devices again. @drv must have been registered prior to calling
+ * this function.
+ *
+ * Context: Does GFP_KERNEL allocation.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+static int do_pci_add_dynid(struct pci_driver *drv,
+ const struct pci_device_id *id,
+ bool check_dup)
+{
+ struct pci_dynid *dynid, *existing_dynid;
+
+ dynid = kzalloc(sizeof(*dynid), GFP_KERNEL);
+ if (!dynid)
+ return -ENOMEM;
+
+ dynid->id = *id;
+
+ scoped_guard(spinlock, &drv->dynids.lock) {
+ if (check_dup) {
+ list_for_each_entry(existing_dynid, &drv->dynids.list, node) {
+ if (pci_match_one_id(&existing_dynid->id, id)) {
+ kfree(dynid);
+ return -EEXIST;
+ }
+ }
+ }
+ list_add_tail(&dynid->node, &drv->dynids.list);
+ }
+
+ return driver_attach(&drv->driver);
+}
+
/**
* pci_add_dynid - add a new PCI device ID to this driver and re-probe devices
* @drv: target pci driver
@@ -56,25 +97,17 @@ int pci_add_dynid(struct pci_driver *drv,
unsigned int class, unsigned int class_mask,
unsigned long driver_data)
{
- struct pci_dynid *dynid;
-
- dynid = kzalloc(sizeof(*dynid), GFP_KERNEL);
- if (!dynid)
- return -ENOMEM;
-
- dynid->id.vendor = vendor;
- dynid->id.device = device;
- dynid->id.subvendor = subvendor;
- dynid->id.subdevice = subdevice;
- dynid->id.class = class;
- dynid->id.class_mask = class_mask;
- dynid->id.driver_data = driver_data;
-
- spin_lock(&drv->dynids.lock);
- list_add_tail(&dynid->node, &drv->dynids.list);
- spin_unlock(&drv->dynids.lock);
-
- return driver_attach(&drv->driver);
+ struct pci_device_id id = {
+ .vendor = vendor,
+ .device = device,
+ .subvendor = subvendor,
+ .subdevice = subdevice,
+ .class = class,
+ .class_mask = class_mask,
+ .driver_data = driver_data,
+ };
+
+ return do_pci_add_dynid(drv, &id, false);
}
EXPORT_SYMBOL_GPL(pci_add_dynid);
@@ -90,6 +123,31 @@ static void pci_free_dynids(struct pci_driver *drv)
spin_unlock(&drv->dynids.lock);
}
+/**
+ * do_pci_match_id - See if a PCI ID matches a given pci_id table
+ * @ids: array of PCI device ID structures to search in
+ * @dev_id: the actual PCI device ID structure to match against.
+ * @include_override_only: also match against device ID entries marked as
+ * override only.
+ *
+ * Return: the matching pci_device_id structure or %NULL if there is no match.
+ */
+static const struct pci_device_id *
+do_pci_match_id(const struct pci_device_id *ids,
+ const struct pci_device_id *dev_id,
+ bool include_override_only)
+{
+ if (ids) {
+ while (ids->vendor || ids->subvendor || ids->class_mask) {
+ if ((!ids->override_only || include_override_only) &&
+ pci_match_one_id(ids, dev_id))
+ return ids;
+ ids++;
+ }
+ }
+ return NULL;
+}
+
/**
* pci_match_id - See if a PCI device matches a given pci_id table
* @ids: array of PCI device ID structures to search in
@@ -105,14 +163,9 @@ static void pci_free_dynids(struct pci_driver *drv)
const struct pci_device_id *pci_match_id(const struct pci_device_id *ids,
struct pci_dev *dev)
{
- if (ids) {
- while (ids->vendor || ids->subvendor || ids->class_mask) {
- if (pci_match_one_device(ids, dev))
- return ids;
- ids++;
- }
- }
- return NULL;
+ struct pci_device_id dev_id = pci_id_from_device(dev);
+
+ return do_pci_match_id(ids, &dev_id, true);
}
EXPORT_SYMBOL(pci_match_id);
@@ -127,6 +180,7 @@ static const struct pci_device_id pci_device_id_any = {
* pci_match_device - See if a device matches a driver's list of IDs
* @drv: the PCI driver to match against
* @dev: the PCI device structure to match against
+ * @id_copy: place to store copy of pci_device_id for dynamic ID
*
* Used by a driver to check whether a PCI device is in its list of
* supported devices or in the dynids list, which may have been augmented
@@ -134,10 +188,11 @@ static const struct pci_device_id pci_device_id_any = {
* structure or %NULL if there is no match.
*/
static const struct pci_device_id *pci_match_device(struct pci_driver *drv,
- struct pci_dev *dev)
+ struct pci_dev *dev,
+ struct pci_device_id *id_copy)
{
- struct pci_dynid *dynid;
- const struct pci_device_id *found_id = NULL, *ids;
+ const struct pci_device_id *found_id = NULL;
+ struct pci_device_id dev_id;
int ret;
/* When driver_override is set, only bind to the matching driver */
@@ -145,45 +200,29 @@ static const struct pci_device_id *pci_match_device(struct pci_driver *drv,
if (ret == 0)
return NULL;
+ dev_id = pci_id_from_device(dev);
/* Look at the dynamic ids first, before the static ones */
- spin_lock(&drv->dynids.lock);
- list_for_each_entry(dynid, &drv->dynids.list, node) {
- if (pci_match_one_device(&dynid->id, dev)) {
- found_id = &dynid->id;
- break;
+ scoped_guard(spinlock, &drv->dynids.lock) {
+ struct pci_dynid *dynid;
+
+ list_for_each_entry(dynid, &drv->dynids.list, node) {
+ if (pci_match_one_id(&dynid->id, &dev_id)) {
+ *id_copy = dynid->id;
+ return id_copy;
+ }
}
}
- spin_unlock(&drv->dynids.lock);
+ found_id = do_pci_match_id(drv->id_table, &dev_id, ret > 0);
if (found_id)
return found_id;
- for (ids = drv->id_table; (found_id = pci_match_id(ids, dev));
- ids = found_id + 1) {
- /*
- * The match table is split based on driver_override.
- * In case override_only was set, enforce driver_override
- * matching.
- */
- if (found_id->override_only) {
- if (ret > 0)
- return found_id;
- } else {
- return found_id;
- }
- }
-
/* driver_override will always match, send a dummy id */
if (ret > 0)
return &pci_device_id_any;
return NULL;
}
-static void _pci_free_device(struct device *dev)
-{
- kfree(to_pci_dev(dev));
-}
-
/**
* new_id_store - sysfs frontend to pci_add_dynid()
* @driver: target device driver
@@ -197,38 +236,22 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
{
struct pci_driver *pdrv = to_pci_driver(driver);
const struct pci_device_id *ids = pdrv->id_table;
- u32 vendor, device, subvendor = PCI_ANY_ID,
- subdevice = PCI_ANY_ID, class = 0, class_mask = 0;
- unsigned long driver_data = 0;
+ struct pci_device_id id = {
+ .subvendor = PCI_ANY_ID,
+ .subdevice = PCI_ANY_ID
+ };
int fields;
int retval = 0;
fields = sscanf(buf, "%x %x %x %x %x %x %lx",
- &vendor, &device, &subvendor, &subdevice,
- &class, &class_mask, &driver_data);
+ &id.vendor, &id.device, &id.subvendor, &id.subdevice,
+ &id.class, &id.class_mask, &id.driver_data);
if (fields < 2)
return -EINVAL;
if (fields != 7) {
- struct pci_dev *pdev = kzalloc(sizeof(*pdev), GFP_KERNEL);
- if (!pdev)
- return -ENOMEM;
-
- pdev->vendor = vendor;
- pdev->device = device;
- pdev->subsystem_vendor = subvendor;
- pdev->subsystem_device = subdevice;
- pdev->class = class;
- pdev->dev.release = _pci_free_device;
-
- device_initialize(&pdev->dev);
- if (pci_match_device(pdrv, pdev))
- retval = -EEXIST;
-
- put_device(&pdev->dev);
-
- if (retval)
- return retval;
+ if (do_pci_match_id(pdrv->id_table, &id, false))
+ return -EEXIST;
}
/* Only accept driver_data values that match an existing id_table
@@ -236,7 +259,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
if (ids) {
retval = -EINVAL;
while (ids->vendor || ids->subvendor || ids->class_mask) {
- if (driver_data == ids->driver_data) {
+ if (id.driver_data == ids->driver_data) {
retval = 0;
break;
}
@@ -246,8 +269,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
return retval;
}
- retval = pci_add_dynid(pdrv, vendor, device, subvendor, subdevice,
- class, class_mask, driver_data);
+ retval = do_pci_add_dynid(pdrv, &id, fields != 7);
if (retval)
return retval;
return count;
@@ -416,12 +438,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
static int __pci_device_probe(struct pci_driver *drv, struct pci_dev *pci_dev)
{
const struct pci_device_id *id;
+ struct pci_device_id id_copy;
int error = 0;
if (drv->probe) {
error = -ENODEV;
- id = pci_match_device(drv, pci_dev);
+ id = pci_match_device(drv, pci_dev, &id_copy);
if (id)
error = pci_call_probe(drv, pci_dev, id);
}
@@ -816,8 +839,7 @@ static int pci_pm_suspend(struct device *dev)
* suspend callbacks can cope with runtime-suspended devices, it is
* better to resume the device from runtime suspend here.
*/
- if (!dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND) ||
- pci_dev_need_resume(pci_dev)) {
+ if (!dev_pm_smart_suspend(dev) || pci_dev_need_resume(pci_dev)) {
pm_runtime_resume(dev);
pci_dev->state_saved = false;
} else {
@@ -1158,8 +1180,7 @@ static int pci_pm_poweroff(struct device *dev)
}
/* The reason to do that is the same as in pci_pm_suspend(). */
- if (!dev_pm_test_driver_flags(dev, DPM_FLAG_SMART_SUSPEND) ||
- pci_dev_need_resume(pci_dev)) {
+ if (!dev_pm_smart_suspend(dev) || pci_dev_need_resume(pci_dev)) {
pm_runtime_resume(dev);
pci_dev->state_saved = false;
} else {
@@ -1515,12 +1536,13 @@ static int pci_bus_match(struct device *dev, const struct device_driver *drv)
struct pci_dev *pci_dev = to_pci_dev(dev);
struct pci_driver *pci_drv;
const struct pci_device_id *found_id;
+ struct pci_device_id id_copy;
if (!pci_dev->match_driver)
return 0;
pci_drv = (struct pci_driver *)to_pci_driver(drv);
- found_id = pci_match_device(pci_drv, pci_dev);
+ found_id = pci_match_device(pci_drv, pci_dev, &id_copy);
if (found_id)
return 1;
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index cdc0c78b36f5..f60d0eb60064 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -289,21 +289,40 @@ extern unsigned long pci_hotplug_mmio_pref_size;
extern unsigned long pci_hotplug_bus_size;
/**
- * pci_match_one_device - Tell if a PCI device structure has a matching
- * PCI device id structure
- * @id: single PCI device id structure to match
- * @dev: the PCI device structure to match against
+ * pci_id_from_device - Obtain a pci_device_id from a PCI device
+ * @dev: the PCI device
*
- * Returns the matching pci_device_id structure or %NULL if there is no match.
+ * Return: a pci_device_id filled.
*/
-static inline const struct pci_device_id *
-pci_match_one_device(const struct pci_device_id *id, const struct pci_dev *dev)
+static inline struct pci_device_id pci_id_from_device(const struct pci_dev *dev)
{
- if ((id->vendor == PCI_ANY_ID || id->vendor == dev->vendor) &&
- (id->device == PCI_ANY_ID || id->device == dev->device) &&
- (id->subvendor == PCI_ANY_ID || id->subvendor == dev->subsystem_vendor) &&
- (id->subdevice == PCI_ANY_ID || id->subdevice == dev->subsystem_device) &&
- !((id->class ^ dev->class) & id->class_mask))
+ return (struct pci_device_id) {
+ .vendor = dev->vendor,
+ .device = dev->device,
+ .subvendor = dev->subsystem_vendor,
+ .subdevice = dev->subsystem_device,
+ .class = dev->class,
+ };
+}
+
+/**
+ * pci_match_one_id - Tell if a PCI device ID matches a needle PCI device ID
+ * @id: single PCI device id structure to match against (needle)
+ * @dev_id: the actual ID from the PCI device
+ *
+ * ID can be retrieved from device using pci_id_from_device().
+ *
+ * Return: the matching pci_device_id structure or %NULL if there is no match.
+ */
+static inline const struct pci_device_id *
+pci_match_one_id(const struct pci_device_id *id,
+ const struct pci_device_id *dev_id)
+{
+ if ((id->vendor == PCI_ANY_ID || id->vendor == dev_id->vendor) &&
+ (id->device == PCI_ANY_ID || id->device == dev_id->device) &&
+ (id->subvendor == PCI_ANY_ID || id->subvendor == dev_id->subvendor) &&
+ (id->subdevice == PCI_ANY_ID || id->subdevice == dev_id->subdevice) &&
+ !((id->class ^ dev_id->class) & id->class_mask))
return id;
return NULL;
}
@@ -775,6 +794,7 @@ void pci_set_bus_of_node(struct pci_bus *bus);
void pci_release_bus_of_node(struct pci_bus *bus);
int devm_of_pci_bridge_init(struct device *dev, struct pci_host_bridge *bridge);
+bool of_pci_supply_present(struct device_node *np);
#else
static inline int
@@ -822,6 +842,10 @@ static inline int devm_of_pci_bridge_init(struct device *dev, struct pci_host_br
return 0;
}
+static inline bool of_pci_supply_present(struct device_node *np)
+{
+ return false;
+}
#endif /* CONFIG_OF */
struct of_changeset;
diff --git a/drivers/pci/pwrctl/core.c b/drivers/pci/pwrctl/core.c
index 01d913b60316..316c1bc05d4d 100644
--- a/drivers/pci/pwrctl/core.c
+++ b/drivers/pci/pwrctl/core.c
@@ -110,6 +110,8 @@ EXPORT_SYMBOL_GPL(pci_pwrctl_device_set_ready);
*/
void pci_pwrctl_device_unset_ready(struct pci_pwrctl *pwrctl)
{
+ cancel_work_sync(&pwrctl->work);
+
/*
* We don't have to delete the link here. Typically, this function
* is only called when the power control device is being detached. If
diff --git a/drivers/pci/search.c b/drivers/pci/search.c
index 53840634fbfc..705ab725321a 100644
--- a/drivers/pci/search.c
+++ b/drivers/pci/search.c
@@ -243,8 +243,10 @@ static int match_pci_dev_by_id(struct device *dev, const void *data)
{
struct pci_dev *pdev = to_pci_dev(dev);
const struct pci_device_id *id = data;
+ struct pci_device_id dev_id;
- if (pci_match_one_device(id, pdev))
+ dev_id = pci_id_from_device(pdev);
+ if (pci_match_one_id(id, &dev_id))
return 1;
return 0;
}
@@ -368,9 +370,9 @@ EXPORT_SYMBOL(pci_get_class);
* @class: search for a PCI device with this base class code
* @from: Previous PCI device found in search, or %NULL for new search.
*
- * Iterates through the list of known PCI devices. If a PCI device is found
+ * Iterate through the list of known PCI devices. If a PCI device is found
* with a matching base class code, the reference count to the device is
- * incremented. See pci_match_one_device() to figure out how does this works.
+ * incremented. See pci_match_one_id() to figure out how this works.
* A new search is initiated by passing %NULL as the @from argument.
* Otherwise if @from is not %NULL, searches continue from next device on the
* global list. The reference count for @from is always decremented if it is
diff --git a/drivers/perf/dwc_pcie_pmu.c b/drivers/perf/dwc_pcie_pmu.c
index 7dbda36884c8..bde0a0a06cff 100644
--- a/drivers/perf/dwc_pcie_pmu.c
+++ b/drivers/perf/dwc_pcie_pmu.c
@@ -20,7 +20,6 @@
#include <linux/sysfs.h>
#include <linux/types.h>
-#define DWC_PCIE_VSEC_RAS_DES_ID 0x02
#define DWC_PCIE_EVENT_CNT_CTL 0x8
/*
@@ -101,13 +100,21 @@ struct dwc_pcie_dev_info {
struct list_head dev_node;
};
-struct dwc_pcie_vendor_id {
- int vendor_id;
+struct dwc_pcie_pmu_vsec_id {
+ u16 vendor_id;
+ u16 vsec_id;
+ u8 vsec_rev;
};
-static const struct dwc_pcie_vendor_id dwc_pcie_vendor_ids[] = {
- {.vendor_id = PCI_VENDOR_ID_ALIBABA },
- {.vendor_id = PCI_VENDOR_ID_QCOM },
+/*
+ * VSEC IDs are allocated by the vendor, so a given ID may mean different
+ * things to different vendors. See PCIe r6.0, sec 7.9.5.2.
+ */
+static const struct dwc_pcie_pmu_vsec_id dwc_pcie_pmu_vsec_ids[] = {
+ { .vendor_id = PCI_VENDOR_ID_ALIBABA,
+ .vsec_id = 0x02, .vsec_rev = 0x4 },
+ { .vendor_id = PCI_VENDOR_ID_QCOM,
+ .vsec_id = 0x02, .vsec_rev = 0x4 },
{} /* terminator */
};
@@ -519,31 +526,28 @@ static void dwc_pcie_unregister_pmu(void *data)
perf_pmu_unregister(&pcie_pmu->pmu);
}
-static bool dwc_pcie_match_des_cap(struct pci_dev *pdev)
+static u16 dwc_pcie_des_cap(struct pci_dev *pdev)
{
- const struct dwc_pcie_vendor_id *vid;
- u16 vsec = 0;
+ const struct dwc_pcie_pmu_vsec_id *vid;
+ u16 vsec;
u32 val;
if (!pci_is_pcie(pdev) || !(pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT))
- return false;
+ return 0;
- for (vid = dwc_pcie_vendor_ids; vid->vendor_id; vid++) {
+ for (vid = dwc_pcie_pmu_vsec_ids; vid->vendor_id; vid++) {
vsec = pci_find_vsec_capability(pdev, vid->vendor_id,
- DWC_PCIE_VSEC_RAS_DES_ID);
- if (vsec)
- break;
+ vid->vsec_id);
+ if (vsec) {
+ pci_read_config_dword(pdev, vsec + PCI_VNDR_HEADER,
+ &val);
+ if (PCI_VNDR_HEADER_REV(val) == vid->vsec_rev) {
+ pci_dbg(pdev, "Detected PCIe Vendor-Specific Extended Capability RAS DES\n");
+ return vsec;
+ }
+ }
}
- if (!vsec)
- return false;
-
- pci_read_config_dword(pdev, vsec + PCI_VNDR_HEADER, &val);
- if (PCI_VNDR_HEADER_REV(val) != 0x04)
- return false;
-
- pci_dbg(pdev,
- "Detected PCIe Vendor-Specific Extended Capability RAS DES\n");
- return true;
+ return 0;
}
static void dwc_pcie_unregister_dev(struct dwc_pcie_dev_info *dev_info)
@@ -560,9 +564,7 @@ static int dwc_pcie_register_dev(struct pci_dev *pdev)
u32 sbdf;
sbdf = (pci_domain_nr(pdev->bus) << 16) | PCI_DEVID(pdev->bus->number, pdev->devfn);
- plat_dev = platform_device_register_data(NULL, "dwc_pcie_pmu", sbdf,
- pdev, sizeof(*pdev));
-
+ plat_dev = platform_device_register_simple("dwc_pcie_pmu", sbdf, NULL, 0);
if (IS_ERR(plat_dev))
return PTR_ERR(plat_dev);
@@ -589,7 +591,7 @@ static int dwc_pcie_pmu_notifier(struct notifier_block *nb,
switch (action) {
case BUS_NOTIFY_ADD_DEVICE:
- if (!dwc_pcie_match_des_cap(pdev))
+ if (!dwc_pcie_des_cap(pdev))
return NOTIFY_DONE;
if (dwc_pcie_register_dev(pdev))
return NOTIFY_BAD;
@@ -611,17 +613,26 @@ static struct notifier_block dwc_pcie_pmu_nb = {
static int dwc_pcie_pmu_probe(struct platform_device *plat_dev)
{
- struct pci_dev *pdev = plat_dev->dev.platform_data;
+ struct pci_dev *pdev;
struct dwc_pcie_pmu *pcie_pmu;
char *name;
- u32 sbdf, val;
+ u32 sbdf;
u16 vsec;
int ret;
- vsec = pci_find_vsec_capability(pdev, pdev->vendor,
- DWC_PCIE_VSEC_RAS_DES_ID);
- pci_read_config_dword(pdev, vsec + PCI_VNDR_HEADER, &val);
sbdf = plat_dev->id;
+ pdev = pci_get_domain_bus_and_slot(sbdf >> 16, PCI_BUS_NUM(sbdf & 0xffff),
+ sbdf & 0xff);
+ if (!pdev) {
+ pr_err("No pdev found for the sbdf 0x%x\n", sbdf);
+ return -ENODEV;
+ }
+
+ vsec = dwc_pcie_des_cap(pdev);
+ if (!vsec)
+ return -ENODEV;
+
+ pci_dev_put(pdev);
name = devm_kasprintf(&plat_dev->dev, GFP_KERNEL, "dwc_rootport_%x", sbdf);
if (!name)
return -ENOMEM;
@@ -636,7 +647,7 @@ static int dwc_pcie_pmu_probe(struct platform_device *plat_dev)
pcie_pmu->on_cpu = -1;
pcie_pmu->pmu = (struct pmu){
.name = name,
- .parent = &pdev->dev,
+ .parent = &plat_dev->dev,
.module = THIS_MODULE,
.attr_groups = dwc_pcie_attr_groups,
.capabilities = PERF_PMU_CAP_NO_EXCLUDE,
@@ -741,7 +752,7 @@ static int __init dwc_pcie_pmu_init(void)
int ret;
for_each_pci_dev(pdev) {
- if (!dwc_pcie_match_des_cap(pdev))
+ if (!dwc_pcie_des_cap(pdev))
continue;
ret = dwc_pcie_register_dev(pdev);
diff --git a/drivers/pinctrl/mediatek/pinctrl-moore.c b/drivers/pinctrl/mediatek/pinctrl-moore.c
index aad4891223d3..f24df94c4918 100644
--- a/drivers/pinctrl/mediatek/pinctrl-moore.c
+++ b/drivers/pinctrl/mediatek/pinctrl-moore.c
@@ -573,7 +573,7 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
chip->base = -1;
chip->ngpio = hw->soc->npins;
- ret = gpiochip_add_data(chip, hw);
+ ret = devm_gpiochip_add_data(hw->dev, chip, hw);
if (ret < 0)
return ret;
@@ -587,10 +587,8 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
if (!of_property_present(hw->dev->of_node, "gpio-ranges")) {
ret = gpiochip_add_pin_range(chip, dev_name(hw->dev), 0, 0,
chip->ngpio);
- if (ret < 0) {
- gpiochip_remove(chip);
+ if (ret < 0)
return ret;
- }
}
return 0;
diff --git a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
index efc9b07e6265..cee2dffd57e7 100644
--- a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
+++ b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
@@ -1118,30 +1118,24 @@ int mtk_pctrl_init(struct platform_device *pdev,
pctl->chip->parent = &pdev->dev;
pctl->chip->base = -1;
- ret = gpiochip_add_data(pctl->chip, pctl);
+ ret = devm_gpiochip_add_data(&pdev->dev, pctl->chip, pctl);
if (ret)
return -EINVAL;
/* Register the GPIO to pin mappings. */
ret = gpiochip_add_pin_range(pctl->chip, dev_name(&pdev->dev),
0, 0, pctl->devdata->npins);
- if (ret) {
- ret = -EINVAL;
- goto chip_error;
- }
+ if (ret)
+ return -EINVAL;
/* Only initialize EINT if we have EINT pins */
if (data->eint_hw.ap_num > 0) {
ret = mtk_eint_init(pctl, pdev);
if (ret)
- goto chip_error;
+ return ret;
}
return 0;
-
-chip_error:
- gpiochip_remove(pctl->chip);
- return ret;
}
int mtk_pctrl_common_probe(struct platform_device *pdev)
diff --git a/drivers/pinctrl/mediatek/pinctrl-paris.c b/drivers/pinctrl/mediatek/pinctrl-paris.c
index 87e958d827bf..e07ffe1a38bc 100644
--- a/drivers/pinctrl/mediatek/pinctrl-paris.c
+++ b/drivers/pinctrl/mediatek/pinctrl-paris.c
@@ -771,7 +771,6 @@ static const struct pinmux_ops mtk_pmxops = {
.get_function_name = mtk_pmx_get_func_name,
.get_function_groups = mtk_pmx_get_func_groups,
.set_mux = mtk_pmx_set_mux,
- .gpio_set_direction = mtk_pinmux_gpio_set_direction,
.gpio_request_enable = mtk_pinmux_gpio_request_enable,
};
@@ -840,9 +839,6 @@ static int mtk_gpio_get_direction(struct gpio_chip *chip, unsigned int gpio)
const struct mtk_pin_desc *desc;
int value, err;
- if (gpio >= hw->soc->npins)
- return -EINVAL;
-
/*
* "Virtual" GPIOs are always and only used for interrupts
* Since they are only used for interrupts, they are always inputs
@@ -868,9 +864,6 @@ static int mtk_gpio_get(struct gpio_chip *chip, unsigned int gpio)
const struct mtk_pin_desc *desc;
int value, err;
- if (gpio >= hw->soc->npins)
- return -EINVAL;
-
desc = (const struct mtk_pin_desc *)&hw->soc->pins[gpio];
err = mtk_hw_get_value(hw, desc, PINCTRL_PIN_REG_DI, &value);
@@ -880,40 +873,34 @@ static int mtk_gpio_get(struct gpio_chip *chip, unsigned int gpio)
return !!value;
}
-static void mtk_gpio_set(struct gpio_chip *chip, unsigned int gpio, int value)
+static int mtk_gpio_set(struct gpio_chip *chip, unsigned int gpio, int value)
{
struct mtk_pinctrl *hw = gpiochip_get_data(chip);
const struct mtk_pin_desc *desc;
- if (gpio >= hw->soc->npins)
- return;
-
desc = (const struct mtk_pin_desc *)&hw->soc->pins[gpio];
- mtk_hw_set_value(hw, desc, PINCTRL_PIN_REG_DO, !!value);
+ return mtk_hw_set_value(hw, desc, PINCTRL_PIN_REG_DO, !!value);
}
static int mtk_gpio_direction_input(struct gpio_chip *chip, unsigned int gpio)
{
struct mtk_pinctrl *hw = gpiochip_get_data(chip);
- if (gpio >= hw->soc->npins)
- return -EINVAL;
-
- return pinctrl_gpio_direction_input(chip, gpio);
+ return mtk_pinmux_gpio_set_direction(hw->pctrl, NULL, gpio, true);
}
static int mtk_gpio_direction_output(struct gpio_chip *chip, unsigned int gpio,
int value)
{
struct mtk_pinctrl *hw = gpiochip_get_data(chip);
+ int ret;
- if (gpio >= hw->soc->npins)
- return -EINVAL;
-
- mtk_gpio_set(chip, gpio, value);
+ ret = mtk_gpio_set(chip, gpio, value);
+ if (ret)
+ return ret;
- return pinctrl_gpio_direction_output(chip, gpio);
+ return mtk_pinmux_gpio_set_direction(hw->pctrl, NULL, gpio, false);
}
static int mtk_gpio_to_irq(struct gpio_chip *chip, unsigned int offset)
@@ -964,13 +951,13 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
chip->direction_input = mtk_gpio_direction_input;
chip->direction_output = mtk_gpio_direction_output;
chip->get = mtk_gpio_get;
- chip->set = mtk_gpio_set;
+ chip->set_rv = mtk_gpio_set;
chip->to_irq = mtk_gpio_to_irq;
chip->set_config = mtk_gpio_set_config;
chip->base = -1;
chip->ngpio = hw->soc->npins;
- ret = gpiochip_add_data(chip, hw);
+ ret = devm_gpiochip_add_data(hw->dev, chip, hw);
if (ret < 0)
return ret;
diff --git a/drivers/platform/x86/amd/pmf/Kconfig b/drivers/platform/x86/amd/pmf/Kconfig
index f4fa8bd8bda8..99d67cdbd91e 100644
--- a/drivers/platform/x86/amd/pmf/Kconfig
+++ b/drivers/platform/x86/amd/pmf/Kconfig
@@ -11,6 +11,7 @@ config AMD_PMF
select ACPI_PLATFORM_PROFILE
depends on TEE && AMDTEE
depends on AMD_SFH_HID
+ depends on HAS_IOMEM
help
This driver provides support for the AMD Platform Management Framework.
The goal is to enhance end user experience by making AMD PCs smarter,
diff --git a/drivers/platform/x86/amd/pmf/acpi.c b/drivers/platform/x86/amd/pmf/acpi.c
index d5b496433d69..1b9c7acf0ddf 100644
--- a/drivers/platform/x86/amd/pmf/acpi.c
+++ b/drivers/platform/x86/amd/pmf/acpi.c
@@ -433,37 +433,29 @@ int apmf_install_handler(struct amd_pmf_dev *pmf_dev)
return 0;
}
-static acpi_status apmf_walk_resources(struct acpi_resource *res, void *data)
+int apmf_check_smart_pc(struct amd_pmf_dev *pmf_dev)
{
- struct amd_pmf_dev *dev = data;
+ struct platform_device *pdev = to_platform_device(pmf_dev->dev);
- switch (res->type) {
- case ACPI_RESOURCE_TYPE_ADDRESS64:
- dev->policy_addr = res->data.address64.address.minimum;
- dev->policy_sz = res->data.address64.address.address_length;
- break;
- case ACPI_RESOURCE_TYPE_FIXED_MEMORY32:
- dev->policy_addr = res->data.fixed_memory32.address;
- dev->policy_sz = res->data.fixed_memory32.address_length;
- break;
- }
-
- if (!dev->policy_addr || dev->policy_sz > POLICY_BUF_MAX_SZ || dev->policy_sz == 0) {
- pr_err("Incorrect Policy params, possibly a SBIOS bug\n");
- return AE_ERROR;
+ pmf_dev->res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!pmf_dev->res) {
+ dev_dbg(pmf_dev->dev, "Failed to get I/O memory resource\n");
+ return -EINVAL;
}
- return AE_OK;
-}
-
-int apmf_check_smart_pc(struct amd_pmf_dev *pmf_dev)
-{
- acpi_handle ahandle = ACPI_HANDLE(pmf_dev->dev);
- acpi_status status;
-
- status = acpi_walk_resources(ahandle, METHOD_NAME__CRS, apmf_walk_resources, pmf_dev);
- if (ACPI_FAILURE(status)) {
- dev_dbg(pmf_dev->dev, "acpi_walk_resources failed :%d\n", status);
+ pmf_dev->policy_addr = pmf_dev->res->start;
+ /*
+ * We cannot use resource_size() here because it adds an extra byte to round off the size.
+ * In the case of PMF ResourceTemplate(), this rounding is already handled within the _CRS.
+ * Using resource_size() would increase the resource size by 1, causing a mismatch with the
+ * length field and leading to issues. Therefore, simply use end-start of the ACPI resource
+ * to obtain the actual length.
+ */
+ pmf_dev->policy_sz = pmf_dev->res->end - pmf_dev->res->start;
+
+ if (!pmf_dev->policy_addr || pmf_dev->policy_sz > POLICY_BUF_MAX_SZ ||
+ pmf_dev->policy_sz == 0) {
+ dev_err(pmf_dev->dev, "Incorrect policy params, possibly a SBIOS bug\n");
return -EINVAL;
}
diff --git a/drivers/platform/x86/amd/pmf/core.c b/drivers/platform/x86/amd/pmf/core.c
index 2e352e475b56..496fea1389fc 100644
--- a/drivers/platform/x86/amd/pmf/core.c
+++ b/drivers/platform/x86/amd/pmf/core.c
@@ -282,7 +282,7 @@ int amd_pmf_set_dram_addr(struct amd_pmf_dev *dev, bool alloc_buffer)
dev_err(dev->dev, "Invalid CPU id: 0x%x", dev->cpu_id);
}
- dev->buf = kzalloc(dev->mtable_size, GFP_KERNEL);
+ dev->buf = devm_kzalloc(dev->dev, dev->mtable_size, GFP_KERNEL);
if (!dev->buf)
return -ENOMEM;
}
@@ -530,6 +530,7 @@ static int amd_pmf_probe(struct platform_device *pdev)
mutex_init(&dev->lock);
mutex_init(&dev->update_mutex);
+ mutex_init(&dev->cb_mutex);
amd_pmf_quirks_init(dev);
apmf_acpi_init(dev);
@@ -556,7 +557,7 @@ static void amd_pmf_remove(struct platform_device *pdev)
amd_pmf_dbgfs_unregister(dev);
mutex_destroy(&dev->lock);
mutex_destroy(&dev->update_mutex);
- kfree(dev->buf);
+ mutex_destroy(&dev->cb_mutex);
}
static const struct attribute_group *amd_pmf_driver_groups[] = {
diff --git a/drivers/platform/x86/amd/pmf/pmf.h b/drivers/platform/x86/amd/pmf/pmf.h
index b857ef449890..8e9a07a082b7 100644
--- a/drivers/platform/x86/amd/pmf/pmf.h
+++ b/drivers/platform/x86/amd/pmf/pmf.h
@@ -13,6 +13,7 @@
#include <linux/acpi.h>
#include <linux/input.h>
+#include <linux/platform_device.h>
#include <linux/platform_profile.h>
#define POLICY_BUF_MAX_SZ 0x4b000
@@ -364,19 +365,21 @@ struct amd_pmf_dev {
/* Smart PC solution builder */
struct dentry *esbin;
unsigned char *policy_buf;
- u32 policy_sz;
+ resource_size_t policy_sz;
struct tee_context *tee_ctx;
struct tee_shm *fw_shm_pool;
u32 session_id;
void *shbuf;
struct delayed_work pb_work;
struct pmf_action_table *prev_data;
- u64 policy_addr;
+ resource_size_t policy_addr;
void __iomem *policy_base;
bool smart_pc_enabled;
u16 pmf_if_version;
struct input_dev *pmf_idev;
size_t mtable_size;
+ struct resource *res;
+ struct mutex cb_mutex;
};
struct apmf_sps_prop_granular_v2 {
diff --git a/drivers/platform/x86/amd/pmf/tee-if.c b/drivers/platform/x86/amd/pmf/tee-if.c
index 6254423d05b0..7789d44c61c4 100644
--- a/drivers/platform/x86/amd/pmf/tee-if.c
+++ b/drivers/platform/x86/amd/pmf/tee-if.c
@@ -256,7 +256,7 @@ static int amd_pmf_invoke_cmd_init(struct amd_pmf_dev *dev)
return -ENODEV;
}
- dev_dbg(dev->dev, "Policy Binary size: %u bytes\n", dev->policy_sz);
+ dev_dbg(dev->dev, "Policy Binary size: %llu bytes\n", (unsigned long long)dev->policy_sz);
memset(dev->shbuf, 0, dev->policy_sz);
ta_sm = dev->shbuf;
in = &ta_sm->pmf_input.init_table;
@@ -352,30 +352,28 @@ static ssize_t amd_pmf_get_pb_data(struct file *filp, const char __user *buf,
return -EINVAL;
/* re-alloc to the new buffer length of the policy binary */
- new_policy_buf = memdup_user(buf, length);
- if (IS_ERR(new_policy_buf))
- return PTR_ERR(new_policy_buf);
+ new_policy_buf = devm_kzalloc(dev->dev, length, GFP_KERNEL);
+ if (!new_policy_buf)
+ return -ENOMEM;
+
+ if (copy_from_user(new_policy_buf, buf, length)) {
+ devm_kfree(dev->dev, new_policy_buf);
+ return -EFAULT;
+ }
- kfree(dev->policy_buf);
+ devm_kfree(dev->dev, dev->policy_buf);
dev->policy_buf = new_policy_buf;
dev->policy_sz = length;
- if (!amd_pmf_pb_valid(dev)) {
- ret = -EINVAL;
- goto cleanup;
- }
+ if (!amd_pmf_pb_valid(dev))
+ return -EINVAL;
amd_pmf_hex_dump_pb(dev);
ret = amd_pmf_start_policy_engine(dev);
if (ret < 0)
- goto cleanup;
+ return ret;
return length;
-
-cleanup:
- kfree(dev->policy_buf);
- dev->policy_buf = NULL;
- return ret;
}
static const struct file_operations pb_fops = {
@@ -528,16 +526,16 @@ int amd_pmf_init_smart_pc(struct amd_pmf_dev *dev)
if (ret)
goto err_cancel_work;
- dev->policy_base = devm_ioremap(dev->dev, dev->policy_addr, dev->policy_sz);
- if (!dev->policy_base) {
- ret = -ENOMEM;
- goto err_free_dram_buf;
+ dev->policy_base = devm_ioremap_resource(dev->dev, dev->res);
+ if (IS_ERR(dev->policy_base)) {
+ ret = PTR_ERR(dev->policy_base);
+ goto err_cancel_work;
}
- dev->policy_buf = kzalloc(dev->policy_sz, GFP_KERNEL);
+ dev->policy_buf = devm_kzalloc(dev->dev, dev->policy_sz, GFP_KERNEL);
if (!dev->policy_buf) {
ret = -ENOMEM;
- goto err_free_dram_buf;
+ goto err_cancel_work;
}
memcpy_fromio(dev->policy_buf, dev->policy_base, dev->policy_sz);
@@ -545,21 +543,21 @@ int amd_pmf_init_smart_pc(struct amd_pmf_dev *dev)
if (!amd_pmf_pb_valid(dev)) {
dev_info(dev->dev, "No Smart PC policy present\n");
ret = -EINVAL;
- goto err_free_policy;
+ goto err_cancel_work;
}
amd_pmf_hex_dump_pb(dev);
- dev->prev_data = kzalloc(sizeof(*dev->prev_data), GFP_KERNEL);
+ dev->prev_data = devm_kzalloc(dev->dev, sizeof(*dev->prev_data), GFP_KERNEL);
if (!dev->prev_data) {
ret = -ENOMEM;
- goto err_free_policy;
+ goto err_cancel_work;
}
for (i = 0; i < ARRAY_SIZE(amd_pmf_ta_uuid); i++) {
ret = amd_pmf_tee_init(dev, &amd_pmf_ta_uuid[i]);
if (ret)
- goto err_free_prev_data;
+ goto err_cancel_work;
ret = amd_pmf_start_policy_engine(dev);
switch (ret) {
@@ -574,7 +572,7 @@ int amd_pmf_init_smart_pc(struct amd_pmf_dev *dev)
default:
ret = -EINVAL;
amd_pmf_tee_deinit(dev);
- goto err_free_prev_data;
+ goto err_cancel_work;
}
if (status)
@@ -583,7 +581,7 @@ int amd_pmf_init_smart_pc(struct amd_pmf_dev *dev)
if (!status && !pb_side_load) {
ret = -EINVAL;
- goto err_free_prev_data;
+ goto err_cancel_work;
}
if (pb_side_load)
@@ -599,12 +597,6 @@ int amd_pmf_init_smart_pc(struct amd_pmf_dev *dev)
if (pb_side_load && dev->esbin)
amd_pmf_remove_pb(dev);
amd_pmf_tee_deinit(dev);
-err_free_prev_data:
- kfree(dev->prev_data);
-err_free_policy:
- kfree(dev->policy_buf);
-err_free_dram_buf:
- kfree(dev->buf);
err_cancel_work:
cancel_delayed_work_sync(&dev->pb_work);
@@ -620,11 +612,5 @@ void amd_pmf_deinit_smart_pc(struct amd_pmf_dev *dev)
amd_pmf_remove_pb(dev);
cancel_delayed_work_sync(&dev->pb_work);
- kfree(dev->prev_data);
- dev->prev_data = NULL;
- kfree(dev->policy_buf);
- dev->policy_buf = NULL;
- kfree(dev->buf);
- dev->buf = NULL;
amd_pmf_tee_deinit(dev);
}
diff --git a/drivers/platform/x86/asus-wmi.c b/drivers/platform/x86/asus-wmi.c
index 5d701fde07df..c8a8e30f8d73 100644
--- a/drivers/platform/x86/asus-wmi.c
+++ b/drivers/platform/x86/asus-wmi.c
@@ -4840,20 +4840,20 @@ static int asus_wmi_add(struct platform_device *pdev)
return 0;
fail_wmi_handler:
+ asus_screenpad_exit(asus);
+fail_screenpad:
asus_wmi_backlight_exit(asus);
fail_backlight:
asus_wmi_rfkill_exit(asus);
-fail_screenpad:
- asus_screenpad_exit(asus);
fail_rfkill:
asus_wmi_led_exit(asus);
fail_leds:
+fail_custom_fan_curve:
fail_hwmon:
asus_wmi_input_exit(asus);
fail_input:
asus_wmi_sysfs_exit(asus->platform_device);
fail_sysfs:
-fail_custom_fan_curve:
fail_platform_profile_setup:
if (asus->platform_profile_support)
platform_profile_remove();
diff --git a/drivers/platform/x86/dell/dell-wmi-base.c b/drivers/platform/x86/dell/dell-wmi-base.c
index 713ec2c12a04..4838992ee5f9 100644
--- a/drivers/platform/x86/dell/dell-wmi-base.c
+++ b/drivers/platform/x86/dell/dell-wmi-base.c
@@ -442,7 +442,7 @@ static int dell_wmi_process_key(struct wmi_device *wdev, int type, int code, u16
SW_TABLET_MODE, !buffer[0]);
return 1;
} else if (type == 0x0012 && code == 0x000d && remaining > 0) {
- value = (buffer[2] == 2);
+ value = (buffer[0] == 2);
used = 1;
}
diff --git a/drivers/pmdomain/imx/imx93-blk-ctrl.c b/drivers/pmdomain/imx/imx93-blk-ctrl.c
index b10348ac10f0..89886bdeb6cd 100644
--- a/drivers/pmdomain/imx/imx93-blk-ctrl.c
+++ b/drivers/pmdomain/imx/imx93-blk-ctrl.c
@@ -47,6 +47,8 @@
#define PRIO(X) (X)
+#define BLK_CTRL_NO_PARENT UINT_MAX
+
struct imx93_blk_ctrl_domain;
struct imx93_blk_ctrl {
@@ -73,6 +75,7 @@ struct imx93_blk_ctrl_domain_data {
int num_clks;
u32 rst_mask;
u32 clk_mask;
+ u32 parent;
int num_qos;
struct imx93_blk_ctrl_qos qos[DOMAIN_MAX_QOS];
};
@@ -287,18 +290,46 @@ static int imx93_blk_ctrl_probe(struct platform_device *pdev)
bc->onecell_data.domains[i] = &domain->genpd;
}
+ for (i = 0; i < bc_data->num_domains; i++) {
+ struct imx93_blk_ctrl_domain *domain = &bc->domains[i];
+ const struct imx93_blk_ctrl_domain_data *data = domain->data;
+
+ if (data->parent == BLK_CTRL_NO_PARENT)
+ continue;
+
+ ret = pm_genpd_add_subdomain(&bc->domains[data->parent].genpd,
+ &domain->genpd);
+ if (ret) {
+ dev_err_probe(dev, ret, "failed to add subdomain %s\n",
+ domain->genpd.name);
+ goto cleanup_subdomains;
+ }
+ }
+
pm_runtime_enable(dev);
ret = of_genpd_add_provider_onecell(dev->of_node, &bc->onecell_data);
if (ret) {
dev_err_probe(dev, ret, "failed to add power domain provider\n");
- goto cleanup_pds;
+ goto cleanup_subdomains;
}
dev_set_drvdata(dev, bc);
return 0;
+cleanup_subdomains:
+ for (i--; i >= 0; i--) {
+ const struct imx93_blk_ctrl_domain_data *data = bc->domains[i].data;
+
+ if (data->parent == BLK_CTRL_NO_PARENT)
+ continue;
+
+ pm_genpd_remove_subdomain(&bc->domains[data->parent].genpd,
+ &bc->domains[i].genpd);
+ }
+ i = bc_data->num_domains;
+
cleanup_pds:
for (i--; i >= 0; i--)
pm_genpd_remove(&bc->domains[i].genpd);
@@ -315,6 +346,16 @@ static void imx93_blk_ctrl_remove(struct platform_device *pdev)
pm_runtime_disable(&pdev->dev);
+ for (i = 0; i < bc->onecell_data.num_domains; i++) {
+ struct imx93_blk_ctrl_domain *domain = &bc->domains[i];
+
+ if (domain->data->parent == BLK_CTRL_NO_PARENT)
+ continue;
+
+ pm_genpd_remove_subdomain(&bc->domains[domain->data->parent].genpd,
+ &domain->genpd);
+ }
+
for (i = 0; i < bc->onecell_data.num_domains; i++) {
struct imx93_blk_ctrl_domain *domain = &bc->domains[i];
@@ -327,8 +368,9 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.name = "mediablk-mipi-dsi",
.clk_names = (const char *[]){ "dsi" },
.num_clks = 1,
- .rst_mask = BIT(11) | BIT(12),
- .clk_mask = BIT(11) | BIT(12),
+ .rst_mask = BIT(11),
+ .clk_mask = BIT(11),
+ .parent = IMX93_MEDIABLK_PD_MIPI_PHY,
},
[IMX93_MEDIABLK_PD_MIPI_CSI] = {
.name = "mediablk-mipi-csi",
@@ -336,6 +378,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 2,
.rst_mask = BIT(9) | BIT(10),
.clk_mask = BIT(9) | BIT(10),
+ .parent = IMX93_MEDIABLK_PD_MIPI_PHY,
},
[IMX93_MEDIABLK_PD_PXP] = {
.name = "mediablk-pxp",
@@ -343,6 +386,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 1,
.rst_mask = BIT(7) | BIT(8),
.clk_mask = BIT(7) | BIT(8),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 2,
.qos = {
{
@@ -364,6 +408,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 2,
.rst_mask = BIT(4) | BIT(5) | BIT(6),
.clk_mask = BIT(4) | BIT(5) | BIT(6),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 1,
.qos = {
{
@@ -380,6 +425,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 1,
.rst_mask = BIT(2) | BIT(3),
.clk_mask = BIT(2) | BIT(3),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 4,
.qos = {
{
@@ -405,6 +451,14 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
}
}
},
+ [IMX93_MEDIABLK_PD_MIPI_PHY] = {
+ .name = "mediablk-mipi-phy",
+ .clk_names = NULL,
+ .num_clks = 0,
+ .rst_mask = BIT(12),
+ .clk_mask = BIT(12),
+ .parent = BLK_CTRL_NO_PARENT,
+ },
};
static const struct regmap_range imx93_media_blk_ctl_yes_ranges[] = {
diff --git a/drivers/power/reset/sc27xx-poweroff.c b/drivers/power/reset/sc27xx-poweroff.c
index 90287c31992c..393bd1c33b73 100644
--- a/drivers/power/reset/sc27xx-poweroff.c
+++ b/drivers/power/reset/sc27xx-poweroff.c
@@ -28,7 +28,7 @@ static struct regmap *regmap;
* taking cpus down to avoid racing regmap or spi mutex lock when poweroff
* system through PMIC.
*/
-static void sc27xx_poweroff_shutdown(void)
+static void sc27xx_poweroff_shutdown(void *data)
{
#ifdef CONFIG_HOTPLUG_CPU
int cpu;
@@ -40,10 +40,14 @@ static void sc27xx_poweroff_shutdown(void)
#endif
}
-static struct syscore_ops poweroff_syscore_ops = {
+static const struct syscore_ops poweroff_syscore_ops = {
.shutdown = sc27xx_poweroff_shutdown,
};
+static struct syscore poweroff_syscore = {
+ .ops = &poweroff_syscore_ops,
+};
+
static void sc27xx_poweroff_do_poweroff(void)
{
/* Disable the external subsys connection's power firstly */
@@ -62,7 +66,7 @@ static int sc27xx_poweroff_probe(struct platform_device *pdev)
return -ENODEV;
pm_power_off = sc27xx_poweroff_do_poweroff;
- register_syscore_ops(&poweroff_syscore_ops);
+ register_syscore(&poweroff_syscore);
return 0;
}
diff --git a/drivers/ptp/ptp_ocp.c b/drivers/ptp/ptp_ocp.c
index 42263c68d5e0..e4187eb6fd77 100644
--- a/drivers/ptp/ptp_ocp.c
+++ b/drivers/ptp/ptp_ocp.c
@@ -4558,8 +4558,7 @@ ptp_ocp_detach(struct ptp_ocp *bp)
ptp_ocp_debugfs_remove_device(bp);
ptp_ocp_detach_sysfs(bp);
ptp_ocp_attr_group_del(bp);
- if (timer_pending(&bp->watchdog))
- del_timer_sync(&bp->watchdog);
+ timer_delete_sync(&bp->watchdog);
if (bp->ts0)
ptp_ocp_unregister_ext(bp->ts0);
if (bp->ts1)
diff --git a/drivers/pwm/pwm-samsung.c b/drivers/pwm/pwm-samsung.c
index 951b38ff5f8e..14fb460a4565 100644
--- a/drivers/pwm/pwm-samsung.c
+++ b/drivers/pwm/pwm-samsung.c
@@ -102,7 +102,7 @@ struct samsung_pwm_chip {
* IP. Should this change, both drivers will need to be modified to
* properly synchronize accesses to particular instances.
*/
-static DEFINE_SPINLOCK(samsung_pwm_lock);
+static DEFINE_RAW_SPINLOCK(samsung_pwm_lock);
#endif
static inline
@@ -141,14 +141,14 @@ static void pwm_samsung_set_divisor(struct samsung_pwm_chip *our_chip,
bits = (fls(divisor) - 1) - our_chip->variant.div_base;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(our_chip->base + REG_TCFG1);
reg &= ~(TCFG1_MUX_MASK << shift);
reg |= bits << shift;
writel(reg, our_chip->base + REG_TCFG1);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int pwm_samsung_is_tdiv(struct samsung_pwm_chip *our_chip, unsigned int chan)
@@ -249,7 +249,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm)
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
@@ -263,7 +263,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm)
our_chip->disabled_mask &= ~BIT(pwm->hwpwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
return 0;
}
@@ -275,7 +275,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm)
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
tcon &= ~TCON_AUTORELOAD(tcon_chan);
@@ -290,7 +290,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm)
our_chip->disabled_mask |= BIT(pwm->hwpwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void pwm_samsung_manual_update(struct samsung_pwm_chip *our_chip,
@@ -298,11 +298,11 @@ static void pwm_samsung_manual_update(struct samsung_pwm_chip *our_chip,
{
unsigned long flags;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
__pwm_samsung_manual_update(our_chip, pwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int __pwm_samsung_config(struct pwm_chip *chip, struct pwm_device *pwm,
@@ -390,7 +390,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *our_chip,
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
@@ -404,7 +404,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *our_chip,
writel(tcon, our_chip->base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int pwm_samsung_set_polarity(struct pwm_chip *chip,
diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
index 62f8548fb46a..85bb276b8025 100644
--- a/drivers/remoteproc/Kconfig
+++ b/drivers/remoteproc/Kconfig
@@ -214,7 +214,7 @@ config QCOM_Q6V5_MSS
handled by QCOM_Q6V5_PAS driver.
config QCOM_Q6V5_PAS
- tristate "Qualcomm Hexagon v5 Peripheral Authentication Service support"
+ tristate "Qualcomm Peripheral Authentication Service support"
depends on OF && ARCH_QCOM
depends on QCOM_SMEM
depends on RPMSG_QCOM_SMD || RPMSG_QCOM_SMD=n
@@ -229,11 +229,10 @@ config QCOM_Q6V5_PAS
select QCOM_RPROC_COMMON
select QCOM_SCM
help
- Say y here to support the TrustZone based Peripheral Image Loader
- for the Qualcomm Hexagon v5 based remote processors. This is commonly
- used to control subsystems such as ADSP (Audio DSP),
- CDSP (Compute DSP), MPSS (Modem Peripheral SubSystem), and
- SLPI (Sensor Low Power Island).
+ Say y here to support the TrustZone based Peripheral Image Loader for
+ the Qualcomm remote processors. This is commonly used to control
+ subsystems such as ADSP (Audio DSP), CDSP (Compute DSP), MPSS (Modem
+ Peripheral SubSystem), and SLPI (Sensor Low Power Island).
config QCOM_Q6V5_WCSS
tristate "Qualcomm Hexagon based WCSS Peripheral Image Loader"
diff --git a/drivers/remoteproc/da8xx_remoteproc.c b/drivers/remoteproc/da8xx_remoteproc.c
index 8770d0cf1255..fe0179f4044f 100644
--- a/drivers/remoteproc/da8xx_remoteproc.c
+++ b/drivers/remoteproc/da8xx_remoteproc.c
@@ -378,7 +378,7 @@ MODULE_DEVICE_TABLE(of, davinci_rproc_of_match);
static struct platform_driver da8xx_rproc_driver = {
.probe = da8xx_rproc_probe,
- .remove_new = da8xx_rproc_remove,
+ .remove = da8xx_rproc_remove,
.driver = {
.name = "davinci-rproc",
.of_match_table = of_match_ptr(davinci_rproc_of_match),
diff --git a/drivers/remoteproc/imx_dsp_rproc.c b/drivers/remoteproc/imx_dsp_rproc.c
index 713a57200cf0..427c200a175c 100644
--- a/drivers/remoteproc/imx_dsp_rproc.c
+++ b/drivers/remoteproc/imx_dsp_rproc.c
@@ -1267,7 +1267,7 @@ MODULE_DEVICE_TABLE(of, imx_dsp_rproc_of_match);
static struct platform_driver imx_dsp_rproc_driver = {
.probe = imx_dsp_rproc_probe,
- .remove_new = imx_dsp_rproc_remove,
+ .remove = imx_dsp_rproc_remove,
.driver = {
.name = "imx-dsp-rproc",
.of_match_table = imx_dsp_rproc_of_match,
diff --git a/drivers/remoteproc/imx_rproc.c b/drivers/remoteproc/imx_rproc.c
index 2797b7e97142..4f73b1ea05ff 100644
--- a/drivers/remoteproc/imx_rproc.c
+++ b/drivers/remoteproc/imx_rproc.c
@@ -1202,7 +1202,7 @@ MODULE_DEVICE_TABLE(of, imx_rproc_of_match);
static struct platform_driver imx_rproc_driver = {
.probe = imx_rproc_probe,
- .remove_new = imx_rproc_remove,
+ .remove = imx_rproc_remove,
.driver = {
.name = "imx-rproc",
.of_match_table = imx_rproc_of_match,
diff --git a/drivers/remoteproc/keystone_remoteproc.c b/drivers/remoteproc/keystone_remoteproc.c
index 8f0f7a4cfef2..6e54093d1732 100644
--- a/drivers/remoteproc/keystone_remoteproc.c
+++ b/drivers/remoteproc/keystone_remoteproc.c
@@ -490,7 +490,7 @@ MODULE_DEVICE_TABLE(of, keystone_rproc_of_match);
static struct platform_driver keystone_rproc_driver = {
.probe = keystone_rproc_probe,
- .remove_new = keystone_rproc_remove,
+ .remove = keystone_rproc_remove,
.driver = {
.name = "keystone-rproc",
.of_match_table = keystone_rproc_of_match,
diff --git a/drivers/remoteproc/meson_mx_ao_arc.c b/drivers/remoteproc/meson_mx_ao_arc.c
index f6744b538323..7dfdf11b0036 100644
--- a/drivers/remoteproc/meson_mx_ao_arc.c
+++ b/drivers/remoteproc/meson_mx_ao_arc.c
@@ -246,7 +246,7 @@ MODULE_DEVICE_TABLE(of, meson_mx_ao_arc_rproc_match);
static struct platform_driver meson_mx_ao_arc_rproc_driver = {
.probe = meson_mx_ao_arc_rproc_probe,
- .remove_new = meson_mx_ao_arc_rproc_remove,
+ .remove = meson_mx_ao_arc_rproc_remove,
.driver = {
.name = "meson-mx-ao-arc-rproc",
.of_match_table = meson_mx_ao_arc_rproc_match,
diff --git a/drivers/remoteproc/mtk_scp.c b/drivers/remoteproc/mtk_scp.c
index 81ca0fc3820f..b3613802730c 100644
--- a/drivers/remoteproc/mtk_scp.c
+++ b/drivers/remoteproc/mtk_scp.c
@@ -1591,7 +1591,7 @@ static const struct dev_pm_ops scp_pm_ops = {
static struct platform_driver mtk_scp_driver = {
.probe = scp_probe,
- .remove_new = scp_remove,
+ .remove = scp_remove,
.driver = {
.name = "mtk-scp",
.of_match_table = mtk_scp_of_match,
diff --git a/drivers/remoteproc/pru_rproc.c b/drivers/remoteproc/pru_rproc.c
index 7ffbae209a31..985daf6945dd 100644
--- a/drivers/remoteproc/pru_rproc.c
+++ b/drivers/remoteproc/pru_rproc.c
@@ -1133,7 +1133,7 @@ static struct platform_driver pru_rproc_driver = {
.suppress_bind_attrs = true,
},
.probe = pru_rproc_probe,
- .remove_new = pru_rproc_remove,
+ .remove = pru_rproc_remove,
};
module_platform_driver(pru_rproc_driver);
diff --git a/drivers/remoteproc/qcom_q6v5_adsp.c b/drivers/remoteproc/qcom_q6v5_adsp.c
index 034594026812..13c9898447b2 100644
--- a/drivers/remoteproc/qcom_q6v5_adsp.c
+++ b/drivers/remoteproc/qcom_q6v5_adsp.c
@@ -848,7 +848,7 @@ MODULE_DEVICE_TABLE(of, adsp_of_match);
static struct platform_driver adsp_pil_driver = {
.probe = adsp_probe,
- .remove_new = adsp_remove,
+ .remove = adsp_remove,
.driver = {
.name = "qcom_q6v5_adsp",
.of_match_table = adsp_of_match,
diff --git a/drivers/remoteproc/qcom_q6v5_mss.c b/drivers/remoteproc/qcom_q6v5_mss.c
index e19081d53022..5ddc146435cd 100644
--- a/drivers/remoteproc/qcom_q6v5_mss.c
+++ b/drivers/remoteproc/qcom_q6v5_mss.c
@@ -2546,7 +2546,7 @@ MODULE_DEVICE_TABLE(of, q6v5_of_match);
static struct platform_driver q6v5_driver = {
.probe = q6v5_probe,
- .remove_new = q6v5_remove,
+ .remove = q6v5_remove,
.driver = {
.name = "qcom-q6v5-mss",
.of_match_table = q6v5_of_match,
diff --git a/drivers/remoteproc/qcom_q6v5_pas.c b/drivers/remoteproc/qcom_q6v5_pas.c
index 23f928323bcf..bf4f242f1995 100644
--- a/drivers/remoteproc/qcom_q6v5_pas.c
+++ b/drivers/remoteproc/qcom_q6v5_pas.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
- * Qualcomm ADSP/SLPI Peripheral Image Loader for MSM8974 and MSM8996
+ * Qualcomm Peripheral Authentication Service remoteproc driver
*
* Copyright (C) 2016 Linaro Ltd
* Copyright (C) 2014 Sony Mobile Communications AB
@@ -31,17 +31,18 @@
#include "qcom_q6v5.h"
#include "remoteproc_internal.h"
-#define ADSP_DECRYPT_SHUTDOWN_DELAY_MS 100
+#define QCOM_PAS_DECRYPT_SHUTDOWN_DELAY_MS 100
#define MAX_ASSIGN_COUNT 3
-struct adsp_data {
+struct qcom_pas_data {
int crash_reason_smem;
const char *firmware_name;
const char *dtb_firmware_name;
int pas_id;
int dtb_pas_id;
int lite_pas_id;
+ int lite_dtb_pas_id;
unsigned int minidump_id;
bool auto_boot;
bool decrypt_shutdown;
@@ -60,7 +61,7 @@ struct adsp_data {
int region_assign_vmid;
};
-struct qcom_adsp {
+struct qcom_pas {
struct device *dev;
struct rproc *rproc;
@@ -80,6 +81,7 @@ struct qcom_adsp {
int pas_id;
int dtb_pas_id;
int lite_pas_id;
+ int lite_dtb_pas_id;
unsigned int minidump_id;
int crash_reason_smem;
unsigned int smem_host_id;
@@ -119,36 +121,37 @@ struct qcom_adsp {
struct qcom_scm_pas_context *dtb_pas_ctx;
};
-static void adsp_segment_dump(struct rproc *rproc, struct rproc_dump_segment *segment,
- void *dest, size_t offset, size_t size)
+static void qcom_pas_segment_dump(struct rproc *rproc,
+ struct rproc_dump_segment *segment,
+ void *dest, size_t offset, size_t size)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
int total_offset;
- total_offset = segment->da + segment->offset + offset - adsp->mem_phys;
- if (total_offset < 0 || total_offset + size > adsp->mem_size) {
- dev_err(adsp->dev,
+ total_offset = segment->da + segment->offset + offset - pas->mem_phys;
+ if (total_offset < 0 || total_offset + size > pas->mem_size) {
+ dev_err(pas->dev,
"invalid copy request for segment %pad with offset %zu and size %zu)\n",
&segment->da, offset, size);
memset(dest, 0xff, size);
return;
}
- memcpy_fromio(dest, adsp->mem_region + total_offset, size);
+ memcpy_fromio(dest, pas->mem_region + total_offset, size);
}
-static void adsp_minidump(struct rproc *rproc)
+static void qcom_pas_minidump(struct rproc *rproc)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
if (rproc->dump_conf == RPROC_COREDUMP_DISABLED)
return;
- qcom_minidump(rproc, adsp->minidump_id, adsp_segment_dump);
+ qcom_minidump(rproc, pas->minidump_id, qcom_pas_segment_dump);
}
-static int adsp_pds_enable(struct qcom_adsp *adsp, struct device **pds,
- size_t pd_count)
+static int qcom_pas_pds_enable(struct qcom_pas *pas, struct device **pds,
+ size_t pd_count)
{
int ret;
int i;
@@ -174,8 +177,8 @@ static int adsp_pds_enable(struct qcom_adsp *adsp, struct device **pds,
return ret;
};
-static void adsp_pds_disable(struct qcom_adsp *adsp, struct device **pds,
- size_t pd_count)
+static void qcom_pas_pds_disable(struct qcom_pas *pas, struct device **pds,
+ size_t pd_count)
{
int i;
@@ -185,46 +188,48 @@ static void adsp_pds_disable(struct qcom_adsp *adsp, struct device **pds,
}
}
-static int adsp_shutdown_poll_decrypt(struct qcom_adsp *adsp)
+static int qcom_pas_shutdown_poll_decrypt(struct qcom_pas *pas)
{
unsigned int retry_num = 50;
int ret;
do {
- msleep(ADSP_DECRYPT_SHUTDOWN_DELAY_MS);
- ret = qcom_scm_pas_shutdown(adsp->pas_id);
+ msleep(QCOM_PAS_DECRYPT_SHUTDOWN_DELAY_MS);
+ ret = qcom_scm_pas_shutdown(pas->pas_id);
} while (ret == -EINVAL && --retry_num);
return ret;
}
-static int adsp_unprepare(struct rproc *rproc)
+static int qcom_pas_unprepare(struct rproc *rproc)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
/*
- * adsp_load() did pass the PAS context to the SCM driver for storing
+ * qcom_pas_load() did pass the PAS context to the SCM driver for storing
* metadata context. It might have been released already if
* auth_and_reset() was successful, but in other cases clean it up
* here.
*/
- qcom_pas_metadata_release(adsp->pas_ctx);
- if (adsp->dtb_pas_id)
- qcom_pas_metadata_release(adsp->dtb_pas_ctx);
+ qcom_pas_metadata_release(pas->pas_ctx);
+ if (pas->dtb_pas_id)
+ qcom_pas_metadata_release(pas->dtb_pas_ctx);
return 0;
}
-static int adsp_load(struct rproc *rproc, const struct firmware *fw)
+static int qcom_pas_load(struct rproc *rproc, const struct firmware *fw)
{
- struct qcom_adsp *pas = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
int ret;
- /* Store firmware handle to be used in adsp_start() */
+ /* Store firmware handle to be used in qcom_pas_start() */
pas->firmware = fw;
if (pas->lite_pas_id)
ret = qcom_scm_pas_shutdown(pas->lite_pas_id);
+ if (pas->lite_dtb_pas_id)
+ qcom_scm_pas_shutdown(pas->lite_dtb_pas_id);
if (pas->dtb_pas_id) {
ret = request_firmware(&pas->dtb_firmware, pas->dtb_firmware_name, pas->dev);
@@ -257,248 +262,238 @@ static int adsp_load(struct rproc *rproc, const struct firmware *fw)
return 0;
}
-static int adsp_start(struct rproc *rproc)
+static int qcom_pas_start(struct rproc *rproc)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
int ret;
- ret = qcom_q6v5_prepare(&adsp->q6v5);
+ ret = qcom_q6v5_prepare(&pas->q6v5);
if (ret)
return ret;
- ret = adsp_pds_enable(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
+ ret = qcom_pas_pds_enable(pas, pas->proxy_pds, pas->proxy_pd_count);
if (ret < 0)
goto disable_irqs;
- ret = clk_prepare_enable(adsp->xo);
+ ret = clk_prepare_enable(pas->xo);
if (ret)
goto disable_proxy_pds;
- ret = clk_prepare_enable(adsp->aggre2_clk);
+ ret = clk_prepare_enable(pas->aggre2_clk);
if (ret)
goto disable_xo_clk;
- if (adsp->cx_supply) {
- ret = regulator_enable(adsp->cx_supply);
+ if (pas->cx_supply) {
+ ret = regulator_enable(pas->cx_supply);
if (ret)
goto disable_aggre2_clk;
}
- if (adsp->px_supply) {
- ret = regulator_enable(adsp->px_supply);
+ if (pas->px_supply) {
+ ret = regulator_enable(pas->px_supply);
if (ret)
goto disable_cx_supply;
}
- if (adsp->dtb_pas_id) {
- ret = qcom_scm_pas_auth_and_reset(adsp->dtb_pas_id);
+ if (pas->dtb_pas_id) {
+ ret = qcom_scm_pas_auth_and_reset(pas->dtb_pas_id);
if (ret) {
- dev_err(adsp->dev,
+ dev_err(pas->dev,
"failed to authenticate dtb image and release reset\n");
goto disable_px_supply;
}
}
- ret = qcom_mdt_pas_init(adsp->dev, adsp->firmware, rproc->firmware, adsp->pas_id,
- adsp->mem_phys, adsp->pas_ctx);
+ ret = qcom_mdt_pas_init(pas->dev, pas->firmware, rproc->firmware, pas->pas_id,
+ pas->mem_phys, pas->pas_ctx);
if (ret)
goto disable_px_supply;
- ret = qcom_mdt_load_no_init(adsp->dev, adsp->firmware, rproc->firmware, adsp->pas_id,
- adsp->mem_region, adsp->mem_phys, adsp->mem_size,
- &adsp->mem_reloc);
+ ret = qcom_mdt_load_no_init(pas->dev, pas->firmware, rproc->firmware, pas->pas_id,
+ pas->mem_region, pas->mem_phys, pas->mem_size,
+ &pas->mem_reloc);
if (ret)
goto release_pas_metadata;
- qcom_pil_info_store(adsp->info_name, adsp->mem_phys, adsp->mem_size);
+ qcom_pil_info_store(pas->info_name, pas->mem_phys, pas->mem_size);
- ret = qcom_scm_pas_auth_and_reset(adsp->pas_id);
+ ret = qcom_scm_pas_auth_and_reset(pas->pas_id);
if (ret) {
- dev_err(adsp->dev,
+ dev_err(pas->dev,
"failed to authenticate image and release reset\n");
goto release_pas_metadata;
}
- ret = qcom_q6v5_wait_for_start(&adsp->q6v5, msecs_to_jiffies(5000));
+ ret = qcom_q6v5_wait_for_start(&pas->q6v5, msecs_to_jiffies(5000));
if (ret == -ETIMEDOUT) {
- dev_err(adsp->dev, "start timed out\n");
- qcom_scm_pas_shutdown(adsp->pas_id);
+ dev_err(pas->dev, "start timed out\n");
+ qcom_scm_pas_shutdown(pas->pas_id);
goto release_pas_metadata;
}
- qcom_pas_metadata_release(adsp->pas_ctx);
- if (adsp->dtb_pas_id)
- qcom_pas_metadata_release(adsp->dtb_pas_ctx);
+ qcom_pas_metadata_release(pas->pas_ctx);
+ if (pas->dtb_pas_id)
+ qcom_pas_metadata_release(pas->dtb_pas_ctx);
- /* Remove pointer to the loaded firmware, only valid in adsp_load() & adsp_start() */
- adsp->firmware = NULL;
+ /* firmware is used to pass reference from qcom_pas_start(), drop it now */
+ pas->firmware = NULL;
return 0;
release_pas_metadata:
- qcom_pas_metadata_release(adsp->pas_ctx);
- if (adsp->dtb_pas_id)
- qcom_pas_metadata_release(adsp->dtb_pas_ctx);
+ qcom_pas_metadata_release(pas->pas_ctx);
+ if (pas->dtb_pas_id)
+ qcom_pas_metadata_release(pas->dtb_pas_ctx);
disable_px_supply:
- if (adsp->px_supply)
- regulator_disable(adsp->px_supply);
+ if (pas->px_supply)
+ regulator_disable(pas->px_supply);
disable_cx_supply:
- if (adsp->cx_supply)
- regulator_disable(adsp->cx_supply);
+ if (pas->cx_supply)
+ regulator_disable(pas->cx_supply);
disable_aggre2_clk:
- clk_disable_unprepare(adsp->aggre2_clk);
+ clk_disable_unprepare(pas->aggre2_clk);
disable_xo_clk:
- clk_disable_unprepare(adsp->xo);
+ clk_disable_unprepare(pas->xo);
disable_proxy_pds:
- adsp_pds_disable(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
+ qcom_pas_pds_disable(pas, pas->proxy_pds, pas->proxy_pd_count);
disable_irqs:
- qcom_q6v5_unprepare(&adsp->q6v5);
+ qcom_q6v5_unprepare(&pas->q6v5);
- /* Remove pointer to the loaded firmware, only valid in adsp_load() & adsp_start() */
- adsp->firmware = NULL;
+ /* firmware is used to pass reference from qcom_pas_start(), drop it now */
+ pas->firmware = NULL;
return ret;
}
static void qcom_pas_handover(struct qcom_q6v5 *q6v5)
{
- struct qcom_adsp *adsp = container_of(q6v5, struct qcom_adsp, q6v5);
-
- if (adsp->px_supply)
- regulator_disable(adsp->px_supply);
- if (adsp->cx_supply)
- regulator_disable(adsp->cx_supply);
- clk_disable_unprepare(adsp->aggre2_clk);
- clk_disable_unprepare(adsp->xo);
- adsp_pds_disable(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
+ struct qcom_pas *pas = container_of(q6v5, struct qcom_pas, q6v5);
+
+ if (pas->px_supply)
+ regulator_disable(pas->px_supply);
+ if (pas->cx_supply)
+ regulator_disable(pas->cx_supply);
+ clk_disable_unprepare(pas->aggre2_clk);
+ clk_disable_unprepare(pas->xo);
+ qcom_pas_pds_disable(pas, pas->proxy_pds, pas->proxy_pd_count);
}
-static int adsp_stop(struct rproc *rproc)
+static int qcom_pas_stop(struct rproc *rproc)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
int handover;
int ret;
- ret = qcom_q6v5_request_stop(&adsp->q6v5, adsp->sysmon);
+ ret = qcom_q6v5_request_stop(&pas->q6v5, pas->sysmon);
if (ret == -ETIMEDOUT)
- dev_err(adsp->dev, "timed out on wait\n");
+ dev_err(pas->dev, "timed out on wait\n");
- ret = qcom_scm_pas_shutdown(adsp->pas_id);
- if (ret && adsp->decrypt_shutdown)
- ret = adsp_shutdown_poll_decrypt(adsp);
+ ret = qcom_scm_pas_shutdown(pas->pas_id);
+ if (ret && pas->decrypt_shutdown)
+ ret = qcom_pas_shutdown_poll_decrypt(pas);
if (ret)
- dev_err(adsp->dev, "failed to shutdown: %d\n", ret);
+ dev_err(pas->dev, "failed to shutdown: %d\n", ret);
- if (adsp->dtb_pas_id) {
- ret = qcom_scm_pas_shutdown(adsp->dtb_pas_id);
+ if (pas->dtb_pas_id) {
+ ret = qcom_scm_pas_shutdown(pas->dtb_pas_id);
if (ret)
- dev_err(adsp->dev, "failed to shutdown dtb: %d\n", ret);
+ dev_err(pas->dev, "failed to shutdown dtb: %d\n", ret);
}
- handover = qcom_q6v5_unprepare(&adsp->q6v5);
+ handover = qcom_q6v5_unprepare(&pas->q6v5);
if (handover)
- qcom_pas_handover(&adsp->q6v5);
+ qcom_pas_handover(&pas->q6v5);
- if (adsp->smem_host_id)
- ret = qcom_smem_bust_hwspin_lock_by_host(adsp->smem_host_id);
+ if (pas->smem_host_id)
+ ret = qcom_smem_bust_hwspin_lock_by_host(pas->smem_host_id);
return ret;
}
-static void *adsp_da_to_va(struct rproc *rproc, u64 da, size_t len, bool *is_iomem)
+static void *qcom_pas_da_to_va(struct rproc *rproc, u64 da, size_t len, bool *is_iomem)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
int offset;
- offset = da - adsp->mem_reloc;
- if (offset < 0 || offset + len > adsp->mem_size)
+ offset = da - pas->mem_reloc;
+ if (offset < 0 || offset + len > pas->mem_size)
return NULL;
if (is_iomem)
*is_iomem = true;
- return adsp->mem_region + offset;
+ return pas->mem_region + offset;
}
-static unsigned long adsp_panic(struct rproc *rproc)
+static unsigned long qcom_pas_panic(struct rproc *rproc)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_pas *pas = rproc->priv;
- return qcom_q6v5_panic(&adsp->q6v5);
+ return qcom_q6v5_panic(&pas->q6v5);
}
-static const struct rproc_ops adsp_ops = {
- .unprepare = adsp_unprepare,
- .start = adsp_start,
- .stop = adsp_stop,
- .da_to_va = adsp_da_to_va,
+static const struct rproc_ops qcom_pas_ops = {
+ .unprepare = qcom_pas_unprepare,
+ .start = qcom_pas_start,
+ .stop = qcom_pas_stop,
+ .da_to_va = qcom_pas_da_to_va,
.parse_fw = qcom_register_dump_segments,
- .load = adsp_load,
- .panic = adsp_panic,
+ .load = qcom_pas_load,
+ .panic = qcom_pas_panic,
};
-static const struct rproc_ops adsp_minidump_ops = {
- .unprepare = adsp_unprepare,
- .start = adsp_start,
- .stop = adsp_stop,
- .da_to_va = adsp_da_to_va,
+static const struct rproc_ops qcom_pas_minidump_ops = {
+ .unprepare = qcom_pas_unprepare,
+ .start = qcom_pas_start,
+ .stop = qcom_pas_stop,
+ .da_to_va = qcom_pas_da_to_va,
.parse_fw = qcom_register_dump_segments,
- .load = adsp_load,
- .panic = adsp_panic,
- .coredump = adsp_minidump,
+ .load = qcom_pas_load,
+ .panic = qcom_pas_panic,
+ .coredump = qcom_pas_minidump,
};
-static int adsp_init_clock(struct qcom_adsp *adsp)
+static int qcom_pas_init_clock(struct qcom_pas *pas)
{
- int ret;
-
- adsp->xo = devm_clk_get(adsp->dev, "xo");
- if (IS_ERR(adsp->xo)) {
- ret = PTR_ERR(adsp->xo);
- if (ret != -EPROBE_DEFER)
- dev_err(adsp->dev, "failed to get xo clock");
- return ret;
- }
+ pas->xo = devm_clk_get(pas->dev, "xo");
+ if (IS_ERR(pas->xo))
+ return dev_err_probe(pas->dev, PTR_ERR(pas->xo),
+ "failed to get xo clock");
- adsp->aggre2_clk = devm_clk_get_optional(adsp->dev, "aggre2");
- if (IS_ERR(adsp->aggre2_clk)) {
- ret = PTR_ERR(adsp->aggre2_clk);
- if (ret != -EPROBE_DEFER)
- dev_err(adsp->dev,
- "failed to get aggre2 clock");
- return ret;
- }
+ pas->aggre2_clk = devm_clk_get_optional(pas->dev, "aggre2");
+ if (IS_ERR(pas->aggre2_clk))
+ return dev_err_probe(pas->dev, PTR_ERR(pas->aggre2_clk),
+ "failed to get aggre2 clock");
return 0;
}
-static int adsp_init_regulator(struct qcom_adsp *adsp)
+static int qcom_pas_init_regulator(struct qcom_pas *pas)
{
- adsp->cx_supply = devm_regulator_get_optional(adsp->dev, "cx");
- if (IS_ERR(adsp->cx_supply)) {
- if (PTR_ERR(adsp->cx_supply) == -ENODEV)
- adsp->cx_supply = NULL;
+ pas->cx_supply = devm_regulator_get_optional(pas->dev, "cx");
+ if (IS_ERR(pas->cx_supply)) {
+ if (PTR_ERR(pas->cx_supply) == -ENODEV)
+ pas->cx_supply = NULL;
else
- return PTR_ERR(adsp->cx_supply);
+ return PTR_ERR(pas->cx_supply);
}
- if (adsp->cx_supply)
- regulator_set_load(adsp->cx_supply, 100000);
+ if (pas->cx_supply)
+ regulator_set_load(pas->cx_supply, 100000);
- adsp->px_supply = devm_regulator_get_optional(adsp->dev, "px");
- if (IS_ERR(adsp->px_supply)) {
- if (PTR_ERR(adsp->px_supply) == -ENODEV)
- adsp->px_supply = NULL;
+ pas->px_supply = devm_regulator_get_optional(pas->dev, "px");
+ if (IS_ERR(pas->px_supply)) {
+ if (PTR_ERR(pas->px_supply) == -ENODEV)
+ pas->px_supply = NULL;
else
- return PTR_ERR(adsp->px_supply);
+ return PTR_ERR(pas->px_supply);
}
return 0;
}
-static int adsp_pds_attach(struct device *dev, struct device **devs,
- char **pd_names)
+static int qcom_pas_pds_attach(struct device *dev, struct device **devs, char **pd_names)
{
size_t num_pds = 0;
int ret;
@@ -534,10 +529,9 @@ static int adsp_pds_attach(struct device *dev, struct device **devs,
return ret;
};
-static void adsp_pds_detach(struct qcom_adsp *adsp, struct device **pds,
- size_t pd_count)
+static void qcom_pas_pds_detach(struct qcom_pas *pas, struct device **pds, size_t pd_count)
{
- struct device *dev = adsp->dev;
+ struct device *dev = pas->dev;
int i;
/* Handle single power domain */
@@ -550,62 +544,62 @@ static void adsp_pds_detach(struct qcom_adsp *adsp, struct device **pds,
dev_pm_domain_detach(pds[i], false);
}
-static int adsp_alloc_memory_region(struct qcom_adsp *adsp)
+static int qcom_pas_alloc_memory_region(struct qcom_pas *pas)
{
struct reserved_mem *rmem;
struct device_node *node;
- node = of_parse_phandle(adsp->dev->of_node, "memory-region", 0);
+ node = of_parse_phandle(pas->dev->of_node, "memory-region", 0);
if (!node) {
- dev_err(adsp->dev, "no memory-region specified\n");
+ dev_err(pas->dev, "no memory-region specified\n");
return -EINVAL;
}
rmem = of_reserved_mem_lookup(node);
of_node_put(node);
if (!rmem) {
- dev_err(adsp->dev, "unable to resolve memory-region\n");
+ dev_err(pas->dev, "unable to resolve memory-region\n");
return -EINVAL;
}
- adsp->mem_phys = adsp->mem_reloc = rmem->base;
- adsp->mem_size = rmem->size;
- adsp->mem_region = devm_ioremap_wc(adsp->dev, adsp->mem_phys, adsp->mem_size);
- if (!adsp->mem_region) {
- dev_err(adsp->dev, "unable to map memory region: %pa+%zx\n",
- &rmem->base, adsp->mem_size);
+ pas->mem_phys = pas->mem_reloc = rmem->base;
+ pas->mem_size = rmem->size;
+ pas->mem_region = devm_ioremap_wc(pas->dev, pas->mem_phys, pas->mem_size);
+ if (!pas->mem_region) {
+ dev_err(pas->dev, "unable to map memory region: %pa+%zx\n",
+ &rmem->base, pas->mem_size);
return -EBUSY;
}
- if (!adsp->dtb_pas_id)
+ if (!pas->dtb_pas_id)
return 0;
- node = of_parse_phandle(adsp->dev->of_node, "memory-region", 1);
+ node = of_parse_phandle(pas->dev->of_node, "memory-region", 1);
if (!node) {
- dev_err(adsp->dev, "no dtb memory-region specified\n");
+ dev_err(pas->dev, "no dtb memory-region specified\n");
return -EINVAL;
}
rmem = of_reserved_mem_lookup(node);
of_node_put(node);
if (!rmem) {
- dev_err(adsp->dev, "unable to resolve dtb memory-region\n");
+ dev_err(pas->dev, "unable to resolve dtb memory-region\n");
return -EINVAL;
}
- adsp->dtb_mem_phys = adsp->dtb_mem_reloc = rmem->base;
- adsp->dtb_mem_size = rmem->size;
- adsp->dtb_mem_region = devm_ioremap_wc(adsp->dev, adsp->dtb_mem_phys, adsp->dtb_mem_size);
- if (!adsp->dtb_mem_region) {
- dev_err(adsp->dev, "unable to map dtb memory region: %pa+%zx\n",
- &rmem->base, adsp->dtb_mem_size);
+ pas->dtb_mem_phys = pas->dtb_mem_reloc = rmem->base;
+ pas->dtb_mem_size = rmem->size;
+ pas->dtb_mem_region = devm_ioremap_wc(pas->dev, pas->dtb_mem_phys, pas->dtb_mem_size);
+ if (!pas->dtb_mem_region) {
+ dev_err(pas->dev, "unable to map dtb memory region: %pa+%zx\n",
+ &rmem->base, pas->dtb_mem_size);
return -EBUSY;
}
return 0;
}
-static int adsp_assign_memory_region(struct qcom_adsp *adsp)
+static int qcom_pas_assign_memory_region(struct qcom_pas *pas)
{
struct qcom_scm_vmperm perm[MAX_ASSIGN_COUNT];
struct device_node *node;
@@ -613,45 +607,45 @@ static int adsp_assign_memory_region(struct qcom_adsp *adsp)
int offset;
int ret;
- if (!adsp->region_assign_idx)
+ if (!pas->region_assign_idx)
return 0;
- for (offset = 0; offset < adsp->region_assign_count; ++offset) {
+ for (offset = 0; offset < pas->region_assign_count; ++offset) {
struct reserved_mem *rmem = NULL;
- node = of_parse_phandle(adsp->dev->of_node, "memory-region",
- adsp->region_assign_idx + offset);
+ node = of_parse_phandle(pas->dev->of_node, "memory-region",
+ pas->region_assign_idx + offset);
if (node)
rmem = of_reserved_mem_lookup(node);
of_node_put(node);
if (!rmem) {
- dev_err(adsp->dev, "unable to resolve shareable memory-region index %d\n",
+ dev_err(pas->dev, "unable to resolve shareable memory-region index %d\n",
offset);
return -EINVAL;
}
- if (adsp->region_assign_shared) {
+ if (pas->region_assign_shared) {
perm[0].vmid = QCOM_SCM_VMID_HLOS;
perm[0].perm = QCOM_SCM_PERM_RW;
- perm[1].vmid = adsp->region_assign_vmid;
+ perm[1].vmid = pas->region_assign_vmid;
perm[1].perm = QCOM_SCM_PERM_RW;
perm_size = 2;
} else {
- perm[0].vmid = adsp->region_assign_vmid;
+ perm[0].vmid = pas->region_assign_vmid;
perm[0].perm = QCOM_SCM_PERM_RW;
perm_size = 1;
}
- adsp->region_assign_phys[offset] = rmem->base;
- adsp->region_assign_size[offset] = rmem->size;
- adsp->region_assign_owners[offset] = BIT(QCOM_SCM_VMID_HLOS);
+ pas->region_assign_phys[offset] = rmem->base;
+ pas->region_assign_size[offset] = rmem->size;
+ pas->region_assign_owners[offset] = BIT(QCOM_SCM_VMID_HLOS);
- ret = qcom_scm_assign_mem(adsp->region_assign_phys[offset],
- adsp->region_assign_size[offset],
- &adsp->region_assign_owners[offset],
+ ret = qcom_scm_assign_mem(pas->region_assign_phys[offset],
+ pas->region_assign_size[offset],
+ &pas->region_assign_owners[offset],
perm, perm_size);
if (ret < 0) {
- dev_err(adsp->dev, "assign memory %d failed\n", offset);
+ dev_err(pas->dev, "assign memory %d failed\n", offset);
return ret;
}
}
@@ -659,35 +653,35 @@ static int adsp_assign_memory_region(struct qcom_adsp *adsp)
return 0;
}
-static void adsp_unassign_memory_region(struct qcom_adsp *adsp)
+static void qcom_pas_unassign_memory_region(struct qcom_pas *pas)
{
struct qcom_scm_vmperm perm;
int offset;
int ret;
- if (!adsp->region_assign_idx || adsp->region_assign_shared)
+ if (!pas->region_assign_idx || pas->region_assign_shared)
return;
- for (offset = 0; offset < adsp->region_assign_count; ++offset) {
+ for (offset = 0; offset < pas->region_assign_count; ++offset) {
perm.vmid = QCOM_SCM_VMID_HLOS;
perm.perm = QCOM_SCM_PERM_RW;
- ret = qcom_scm_assign_mem(adsp->region_assign_phys[offset],
- adsp->region_assign_size[offset],
- &adsp->region_assign_owners[offset],
+ ret = qcom_scm_assign_mem(pas->region_assign_phys[offset],
+ pas->region_assign_size[offset],
+ &pas->region_assign_owners[offset],
&perm, 1);
if (ret < 0)
- dev_err(adsp->dev, "unassign memory %d failed\n", offset);
+ dev_err(pas->dev, "unassign memory %d failed\n", offset);
}
}
-static int adsp_probe(struct platform_device *pdev)
+static int qcom_pas_probe(struct platform_device *pdev)
{
- const struct adsp_data *desc;
- struct qcom_adsp *adsp;
+ const struct qcom_pas_data *desc;
+ struct qcom_pas *pas;
struct rproc *rproc;
const char *fw_name, *dtb_fw_name = NULL;
- const struct rproc_ops *ops = &adsp_ops;
+ const struct rproc_ops *ops = &qcom_pas_ops;
int ret;
desc = of_device_get_match_data(&pdev->dev);
@@ -712,9 +706,9 @@ static int adsp_probe(struct platform_device *pdev)
}
if (desc->minidump_id)
- ops = &adsp_minidump_ops;
+ ops = &qcom_pas_minidump_ops;
- rproc = devm_rproc_alloc(&pdev->dev, pdev->name, ops, fw_name, sizeof(*adsp));
+ rproc = devm_rproc_alloc(&pdev->dev, desc->sysmon_name, ops, fw_name, sizeof(*pas));
if (!rproc) {
dev_err(&pdev->dev, "unable to allocate remoteproc\n");
@@ -724,88 +718,86 @@ static int adsp_probe(struct platform_device *pdev)
rproc->auto_boot = desc->auto_boot;
rproc_coredump_set_elf_info(rproc, ELFCLASS32, EM_NONE);
- adsp = rproc->priv;
- adsp->dev = &pdev->dev;
- adsp->rproc = rproc;
- adsp->minidump_id = desc->minidump_id;
- adsp->pas_id = desc->pas_id;
- adsp->lite_pas_id = desc->lite_pas_id;
- adsp->info_name = desc->sysmon_name;
- adsp->smem_host_id = desc->smem_host_id;
- adsp->decrypt_shutdown = desc->decrypt_shutdown;
- adsp->region_assign_idx = desc->region_assign_idx;
- adsp->region_assign_count = min_t(int, MAX_ASSIGN_COUNT, desc->region_assign_count);
- adsp->region_assign_vmid = desc->region_assign_vmid;
- adsp->region_assign_shared = desc->region_assign_shared;
+ pas = rproc->priv;
+ pas->dev = &pdev->dev;
+ pas->rproc = rproc;
+ pas->minidump_id = desc->minidump_id;
+ pas->pas_id = desc->pas_id;
+ pas->lite_pas_id = desc->lite_pas_id;
+ pas->lite_dtb_pas_id = desc->lite_dtb_pas_id;
+ pas->info_name = desc->sysmon_name;
+ pas->smem_host_id = desc->smem_host_id;
+ pas->decrypt_shutdown = desc->decrypt_shutdown;
+ pas->region_assign_idx = desc->region_assign_idx;
+ pas->region_assign_count = min_t(int, MAX_ASSIGN_COUNT, desc->region_assign_count);
+ pas->region_assign_vmid = desc->region_assign_vmid;
+ pas->region_assign_shared = desc->region_assign_shared;
if (dtb_fw_name) {
- adsp->dtb_firmware_name = dtb_fw_name;
- adsp->dtb_pas_id = desc->dtb_pas_id;
+ pas->dtb_firmware_name = dtb_fw_name;
+ pas->dtb_pas_id = desc->dtb_pas_id;
}
- platform_set_drvdata(pdev, adsp);
+ platform_set_drvdata(pdev, pas);
- ret = device_init_wakeup(adsp->dev, true);
+ ret = device_init_wakeup(pas->dev, true);
if (ret)
goto free_rproc;
- ret = adsp_alloc_memory_region(adsp);
+ ret = qcom_pas_alloc_memory_region(pas);
if (ret)
goto free_rproc;
- ret = adsp_assign_memory_region(adsp);
+ ret = qcom_pas_assign_memory_region(pas);
if (ret)
goto free_rproc;
- ret = adsp_init_clock(adsp);
+ ret = qcom_pas_init_clock(pas);
if (ret)
goto unassign_mem;
- ret = adsp_init_regulator(adsp);
+ ret = qcom_pas_init_regulator(pas);
if (ret)
goto unassign_mem;
- ret = adsp_pds_attach(&pdev->dev, adsp->proxy_pds,
- desc->proxy_pd_names);
+ ret = qcom_pas_pds_attach(&pdev->dev, pas->proxy_pds, desc->proxy_pd_names);
if (ret < 0)
goto unassign_mem;
- adsp->proxy_pd_count = ret;
+ pas->proxy_pd_count = ret;
- ret = qcom_q6v5_init(&adsp->q6v5, pdev, rproc, desc->crash_reason_smem, desc->load_state,
- qcom_pas_handover);
+ ret = qcom_q6v5_init(&pas->q6v5, pdev, rproc, desc->crash_reason_smem,
+ desc->load_state, qcom_pas_handover);
if (ret)
goto detach_proxy_pds;
- qcom_add_glink_subdev(rproc, &adsp->glink_subdev, desc->ssr_name);
- qcom_add_smd_subdev(rproc, &adsp->smd_subdev);
- qcom_add_pdm_subdev(rproc, &adsp->pdm_subdev);
- adsp->sysmon = qcom_add_sysmon_subdev(rproc,
- desc->sysmon_name,
- desc->ssctl_id);
- if (IS_ERR(adsp->sysmon)) {
- ret = PTR_ERR(adsp->sysmon);
+ qcom_add_glink_subdev(rproc, &pas->glink_subdev, desc->ssr_name);
+ qcom_add_smd_subdev(rproc, &pas->smd_subdev);
+ qcom_add_pdm_subdev(rproc, &pas->pdm_subdev);
+ pas->sysmon = qcom_add_sysmon_subdev(rproc, desc->sysmon_name, desc->ssctl_id);
+ if (IS_ERR(pas->sysmon)) {
+ ret = PTR_ERR(pas->sysmon);
goto deinit_remove_pdm_smd_glink;
}
- qcom_add_ssr_subdev(rproc, &adsp->ssr_subdev, desc->ssr_name);
+ qcom_add_ssr_subdev(rproc, &pas->ssr_subdev, desc->ssr_name);
- adsp->pas_ctx = devm_kzalloc(adsp->dev, sizeof(*adsp->pas_ctx), GFP_KERNEL);
- if (!adsp->pas_ctx) {
+ pas->pas_ctx = devm_kzalloc(pas->dev, sizeof(*pas->pas_ctx), GFP_KERNEL);
+ if (!pas->pas_ctx) {
ret = -ENOMEM;
goto remove_ssr_sysmon;
}
- adsp->pas_ctx->dev = adsp->dev;
- adsp->pas_ctx->pas_id = adsp->pas_id;
- adsp->pas_ctx->mem_phys = adsp->mem_phys;
- adsp->pas_ctx->mem_size = adsp->mem_size;
+ pas->pas_ctx->dev = pas->dev;
+ pas->pas_ctx->pas_id = pas->pas_id;
+ pas->pas_ctx->mem_phys = pas->mem_phys;
+ pas->pas_ctx->mem_size = pas->mem_size;
- adsp->dtb_pas_ctx = devm_kzalloc(adsp->dev, sizeof(*adsp->dtb_pas_ctx), GFP_KERNEL);
- if (!adsp->dtb_pas_ctx) {
+ pas->dtb_pas_ctx = devm_kzalloc(pas->dev, sizeof(*pas->dtb_pas_ctx), GFP_KERNEL);
+ if (!pas->dtb_pas_ctx) {
ret = -ENOMEM;
goto remove_ssr_sysmon;
}
- adsp->dtb_pas_ctx->dev = adsp->dev;
- adsp->dtb_pas_ctx->pas_id = adsp->dtb_pas_id;
- adsp->dtb_pas_ctx->mem_phys = adsp->dtb_mem_phys;
- adsp->dtb_pas_ctx->mem_size = adsp->dtb_mem_size;
+ pas->dtb_pas_ctx->dev = pas->dev;
+ pas->dtb_pas_ctx->pas_id = pas->dtb_pas_id;
+ pas->dtb_pas_ctx->mem_phys = pas->dtb_mem_phys;
+ pas->dtb_pas_ctx->mem_size = pas->dtb_mem_size;
ret = rproc_add(rproc);
if (ret)
@@ -814,41 +806,41 @@ static int adsp_probe(struct platform_device *pdev)
return 0;
remove_ssr_sysmon:
- qcom_remove_ssr_subdev(rproc, &adsp->ssr_subdev);
- qcom_remove_sysmon_subdev(adsp->sysmon);
+ qcom_remove_ssr_subdev(rproc, &pas->ssr_subdev);
+ qcom_remove_sysmon_subdev(pas->sysmon);
deinit_remove_pdm_smd_glink:
- qcom_remove_pdm_subdev(rproc, &adsp->pdm_subdev);
- qcom_remove_smd_subdev(rproc, &adsp->smd_subdev);
- qcom_remove_glink_subdev(rproc, &adsp->glink_subdev);
- qcom_q6v5_deinit(&adsp->q6v5);
+ qcom_remove_pdm_subdev(rproc, &pas->pdm_subdev);
+ qcom_remove_smd_subdev(rproc, &pas->smd_subdev);
+ qcom_remove_glink_subdev(rproc, &pas->glink_subdev);
+ qcom_q6v5_deinit(&pas->q6v5);
detach_proxy_pds:
- adsp_pds_detach(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
+ qcom_pas_pds_detach(pas, pas->proxy_pds, pas->proxy_pd_count);
unassign_mem:
- adsp_unassign_memory_region(adsp);
+ qcom_pas_unassign_memory_region(pas);
free_rproc:
- device_init_wakeup(adsp->dev, false);
+ device_init_wakeup(pas->dev, false);
return ret;
}
-static void adsp_remove(struct platform_device *pdev)
+static void qcom_pas_remove(struct platform_device *pdev)
{
- struct qcom_adsp *adsp = platform_get_drvdata(pdev);
-
- rproc_del(adsp->rproc);
-
- qcom_q6v5_deinit(&adsp->q6v5);
- adsp_unassign_memory_region(adsp);
- qcom_remove_glink_subdev(adsp->rproc, &adsp->glink_subdev);
- qcom_remove_sysmon_subdev(adsp->sysmon);
- qcom_remove_smd_subdev(adsp->rproc, &adsp->smd_subdev);
- qcom_remove_pdm_subdev(adsp->rproc, &adsp->pdm_subdev);
- qcom_remove_ssr_subdev(adsp->rproc, &adsp->ssr_subdev);
- adsp_pds_detach(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
- device_init_wakeup(adsp->dev, false);
+ struct qcom_pas *pas = platform_get_drvdata(pdev);
+
+ rproc_del(pas->rproc);
+
+ qcom_q6v5_deinit(&pas->q6v5);
+ qcom_pas_unassign_memory_region(pas);
+ qcom_remove_glink_subdev(pas->rproc, &pas->glink_subdev);
+ qcom_remove_sysmon_subdev(pas->sysmon);
+ qcom_remove_smd_subdev(pas->rproc, &pas->smd_subdev);
+ qcom_remove_pdm_subdev(pas->rproc, &pas->pdm_subdev);
+ qcom_remove_ssr_subdev(pas->rproc, &pas->ssr_subdev);
+ qcom_pas_pds_detach(pas, pas->proxy_pds, pas->proxy_pd_count);
+ device_init_wakeup(pas->dev, false);
}
-static const struct adsp_data adsp_resource_init = {
+static const struct qcom_pas_data adsp_resource_init = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
@@ -858,7 +850,7 @@ static const struct adsp_data adsp_resource_init = {
.ssctl_id = 0x14,
};
-static const struct adsp_data sa8775p_adsp_resource = {
+static const struct qcom_pas_data sa8775p_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mbn",
.pas_id = 1,
@@ -875,7 +867,7 @@ static const struct adsp_data sa8775p_adsp_resource = {
.ssctl_id = 0x14,
};
-static const struct adsp_data sdm845_adsp_resource_init = {
+static const struct qcom_pas_data sdm845_adsp_resource_init = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
@@ -886,7 +878,7 @@ static const struct adsp_data sdm845_adsp_resource_init = {
.ssctl_id = 0x14,
};
-static const struct adsp_data sm6350_adsp_resource = {
+static const struct qcom_pas_data sm6350_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
@@ -902,7 +894,7 @@ static const struct adsp_data sm6350_adsp_resource = {
.ssctl_id = 0x14,
};
-static const struct adsp_data sm6375_mpss_resource = {
+static const struct qcom_pas_data sm6375_mpss_resource = {
.crash_reason_smem = 421,
.firmware_name = "modem.mdt",
.pas_id = 4,
@@ -917,7 +909,7 @@ static const struct adsp_data sm6375_mpss_resource = {
.ssctl_id = 0x12,
};
-static const struct adsp_data sm8150_adsp_resource = {
+static const struct qcom_pas_data sm8150_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
@@ -932,7 +924,7 @@ static const struct adsp_data sm8150_adsp_resource = {
.ssctl_id = 0x14,
};
-static const struct adsp_data sm8250_adsp_resource = {
+static const struct qcom_pas_data sm8250_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
@@ -949,7 +941,7 @@ static const struct adsp_data sm8250_adsp_resource = {
.ssctl_id = 0x14,
};
-static const struct adsp_data sm8350_adsp_resource = {
+static const struct qcom_pas_data sm8350_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
@@ -965,7 +957,7 @@ static const struct adsp_data sm8350_adsp_resource = {
.ssctl_id = 0x14,
};
-static const struct adsp_data msm8996_adsp_resource = {
+static const struct qcom_pas_data msm8996_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
@@ -979,7 +971,7 @@ static const struct adsp_data msm8996_adsp_resource = {
.ssctl_id = 0x14,
};
-static const struct adsp_data cdsp_resource_init = {
+static const struct qcom_pas_data cdsp_resource_init = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
@@ -989,7 +981,7 @@ static const struct adsp_data cdsp_resource_init = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sa8775p_cdsp0_resource = {
+static const struct qcom_pas_data sa8775p_cdsp0_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp0.mbn",
.pas_id = 18,
@@ -1007,7 +999,7 @@ static const struct adsp_data sa8775p_cdsp0_resource = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sa8775p_cdsp1_resource = {
+static const struct qcom_pas_data sa8775p_cdsp1_resource = {
.crash_reason_smem = 633,
.firmware_name = "cdsp1.mbn",
.pas_id = 30,
@@ -1025,7 +1017,7 @@ static const struct adsp_data sa8775p_cdsp1_resource = {
.ssctl_id = 0x20,
};
-static const struct adsp_data sdm845_cdsp_resource_init = {
+static const struct qcom_pas_data sdm845_cdsp_resource_init = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
@@ -1036,7 +1028,7 @@ static const struct adsp_data sdm845_cdsp_resource_init = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sm6350_cdsp_resource = {
+static const struct qcom_pas_data sm6350_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
@@ -1052,7 +1044,7 @@ static const struct adsp_data sm6350_cdsp_resource = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sm8150_cdsp_resource = {
+static const struct qcom_pas_data sm8150_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
@@ -1067,7 +1059,7 @@ static const struct adsp_data sm8150_cdsp_resource = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sm8250_cdsp_resource = {
+static const struct qcom_pas_data sm8250_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
@@ -1082,7 +1074,7 @@ static const struct adsp_data sm8250_cdsp_resource = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sc8280xp_nsp0_resource = {
+static const struct qcom_pas_data sc8280xp_nsp0_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
@@ -1096,7 +1088,7 @@ static const struct adsp_data sc8280xp_nsp0_resource = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sc8280xp_nsp1_resource = {
+static const struct qcom_pas_data sc8280xp_nsp1_resource = {
.crash_reason_smem = 633,
.firmware_name = "cdsp.mdt",
.pas_id = 30,
@@ -1110,13 +1102,14 @@ static const struct adsp_data sc8280xp_nsp1_resource = {
.ssctl_id = 0x20,
};
-static const struct adsp_data x1e80100_adsp_resource = {
+static const struct qcom_pas_data x1e80100_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.dtb_firmware_name = "adsp_dtb.mdt",
.pas_id = 1,
.dtb_pas_id = 0x24,
.lite_pas_id = 0x1f,
+ .lite_dtb_pas_id = 0x29,
.minidump_id = 5,
.auto_boot = true,
.proxy_pd_names = (char*[]){
@@ -1130,7 +1123,7 @@ static const struct adsp_data x1e80100_adsp_resource = {
.ssctl_id = 0x14,
};
-static const struct adsp_data x1e80100_cdsp_resource = {
+static const struct qcom_pas_data x1e80100_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.dtb_firmware_name = "cdsp_dtb.mdt",
@@ -1150,7 +1143,7 @@ static const struct adsp_data x1e80100_cdsp_resource = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sm8350_cdsp_resource = {
+static const struct qcom_pas_data sm8350_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
@@ -1167,7 +1160,7 @@ static const struct adsp_data sm8350_cdsp_resource = {
.ssctl_id = 0x17,
};
-static const struct adsp_data sa8775p_gpdsp0_resource = {
+static const struct qcom_pas_data sa8775p_gpdsp0_resource = {
.crash_reason_smem = 640,
.firmware_name = "gpdsp0.mbn",
.pas_id = 39,
@@ -1184,7 +1177,7 @@ static const struct adsp_data sa8775p_gpdsp0_resource = {
.ssctl_id = 0x21,
};
-static const struct adsp_data sa8775p_gpdsp1_resource = {
+static const struct qcom_pas_data sa8775p_gpdsp1_resource = {
.crash_reason_smem = 641,
.firmware_name = "gpdsp1.mbn",
.pas_id = 40,
@@ -1201,7 +1194,7 @@ static const struct adsp_data sa8775p_gpdsp1_resource = {
.ssctl_id = 0x22,
};
-static const struct adsp_data mpss_resource_init = {
+static const struct qcom_pas_data mpss_resource_init = {
.crash_reason_smem = 421,
.firmware_name = "modem.mdt",
.pas_id = 4,
@@ -1218,7 +1211,7 @@ static const struct adsp_data mpss_resource_init = {
.ssctl_id = 0x12,
};
-static const struct adsp_data sc8180x_mpss_resource = {
+static const struct qcom_pas_data sc8180x_mpss_resource = {
.crash_reason_smem = 421,
.firmware_name = "modem.mdt",
.pas_id = 4,
@@ -1233,7 +1226,7 @@ static const struct adsp_data sc8180x_mpss_resource = {
.ssctl_id = 0x12,
};
-static const struct adsp_data msm8996_slpi_resource_init = {
+static const struct qcom_pas_data msm8996_slpi_resource_init = {
.crash_reason_smem = 424,
.firmware_name = "slpi.mdt",
.pas_id = 12,
@@ -1247,7 +1240,7 @@ static const struct adsp_data msm8996_slpi_resource_init = {
.ssctl_id = 0x16,
};
-static const struct adsp_data sdm845_slpi_resource_init = {
+static const struct qcom_pas_data sdm845_slpi_resource_init = {
.crash_reason_smem = 424,
.firmware_name = "slpi.mdt",
.pas_id = 12,
@@ -1263,7 +1256,7 @@ static const struct adsp_data sdm845_slpi_resource_init = {
.ssctl_id = 0x16,
};
-static const struct adsp_data wcss_resource_init = {
+static const struct qcom_pas_data wcss_resource_init = {
.crash_reason_smem = 421,
.firmware_name = "wcnss.mdt",
.pas_id = 6,
@@ -1273,7 +1266,7 @@ static const struct adsp_data wcss_resource_init = {
.ssctl_id = 0x12,
};
-static const struct adsp_data sdx55_mpss_resource = {
+static const struct qcom_pas_data sdx55_mpss_resource = {
.crash_reason_smem = 421,
.firmware_name = "modem.mdt",
.pas_id = 4,
@@ -1288,7 +1281,7 @@ static const struct adsp_data sdx55_mpss_resource = {
.ssctl_id = 0x22,
};
-static const struct adsp_data sm8450_mpss_resource = {
+static const struct qcom_pas_data sm8450_mpss_resource = {
.crash_reason_smem = 421,
.firmware_name = "modem.mdt",
.pas_id = 4,
@@ -1306,7 +1299,7 @@ static const struct adsp_data sm8450_mpss_resource = {
.ssctl_id = 0x12,
};
-static const struct adsp_data sm8550_adsp_resource = {
+static const struct qcom_pas_data sm8550_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.dtb_firmware_name = "adsp_dtb.mdt",
@@ -1326,7 +1319,7 @@ static const struct adsp_data sm8550_adsp_resource = {
.smem_host_id = 2,
};
-static const struct adsp_data sm8550_cdsp_resource = {
+static const struct qcom_pas_data sm8550_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.dtb_firmware_name = "cdsp_dtb.mdt",
@@ -1347,7 +1340,7 @@ static const struct adsp_data sm8550_cdsp_resource = {
.smem_host_id = 5,
};
-static const struct adsp_data sm8550_mpss_resource = {
+static const struct qcom_pas_data sm8550_mpss_resource = {
.crash_reason_smem = 421,
.firmware_name = "modem.mdt",
.dtb_firmware_name = "modem_dtb.mdt",
@@ -1371,7 +1364,7 @@ static const struct adsp_data sm8550_mpss_resource = {
.region_assign_vmid = QCOM_SCM_VMID_MSS_MSA,
};
-static const struct adsp_data sc7280_wpss_resource = {
+static const struct qcom_pas_data sc7280_wpss_resource = {
.crash_reason_smem = 626,
.firmware_name = "wpss.mdt",
.pas_id = 6,
@@ -1388,7 +1381,7 @@ static const struct adsp_data sc7280_wpss_resource = {
.ssctl_id = 0x19,
};
-static const struct adsp_data sm8650_cdsp_resource = {
+static const struct qcom_pas_data sm8650_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.dtb_firmware_name = "cdsp_dtb.mdt",
@@ -1413,7 +1406,7 @@ static const struct adsp_data sm8650_cdsp_resource = {
.region_assign_vmid = QCOM_SCM_VMID_CDSP,
};
-static const struct adsp_data sm8650_mpss_resource = {
+static const struct qcom_pas_data sm8650_mpss_resource = {
.crash_reason_smem = 421,
.firmware_name = "modem.mdt",
.dtb_firmware_name = "modem_dtb.mdt",
@@ -1437,7 +1430,7 @@ static const struct adsp_data sm8650_mpss_resource = {
.region_assign_vmid = QCOM_SCM_VMID_MSS_MSA,
};
-static const struct of_device_id adsp_of_match[] = {
+static const struct of_device_id qcom_pas_of_match[] = {
{ .compatible = "qcom,msm8226-adsp-pil", .data = &msm8996_adsp_resource},
{ .compatible = "qcom,msm8953-adsp-pil", .data = &msm8996_adsp_resource},
{ .compatible = "qcom,msm8974-adsp-pil", .data = &adsp_resource_init},
@@ -1506,17 +1499,17 @@ static const struct of_device_id adsp_of_match[] = {
{ .compatible = "qcom,x1e80100-cdsp-pas", .data = &x1e80100_cdsp_resource},
{ },
};
-MODULE_DEVICE_TABLE(of, adsp_of_match);
+MODULE_DEVICE_TABLE(of, qcom_pas_of_match);
-static struct platform_driver adsp_driver = {
- .probe = adsp_probe,
- .remove_new = adsp_remove,
+static struct platform_driver qcom_pas_driver = {
+ .probe = qcom_pas_probe,
+ .remove = qcom_pas_remove,
.driver = {
.name = "qcom_q6v5_pas",
- .of_match_table = adsp_of_match,
+ .of_match_table = qcom_pas_of_match,
},
};
-module_platform_driver(adsp_driver);
-MODULE_DESCRIPTION("Qualcomm Hexagon v5 Peripheral Authentication Service driver");
+module_platform_driver(qcom_pas_driver);
+MODULE_DESCRIPTION("Qualcomm Peripheral Authentication Service remoteproc driver");
MODULE_LICENSE("GPL v2");
diff --git a/drivers/remoteproc/qcom_q6v5_wcss.c b/drivers/remoteproc/qcom_q6v5_wcss.c
index c560b81b7263..a89faabc11ce 100644
--- a/drivers/remoteproc/qcom_q6v5_wcss.c
+++ b/drivers/remoteproc/qcom_q6v5_wcss.c
@@ -1111,7 +1111,7 @@ MODULE_DEVICE_TABLE(of, q6v5_wcss_of_match);
static struct platform_driver q6v5_wcss_driver = {
.probe = q6v5_wcss_probe,
- .remove_new = q6v5_wcss_remove,
+ .remove = q6v5_wcss_remove,
.driver = {
.name = "qcom-q6v5-wcss-pil",
.of_match_table = q6v5_wcss_of_match,
diff --git a/drivers/remoteproc/qcom_wcnss.c b/drivers/remoteproc/qcom_wcnss.c
index a2ae6adf0053..0a0eb1d7da82 100644
--- a/drivers/remoteproc/qcom_wcnss.c
+++ b/drivers/remoteproc/qcom_wcnss.c
@@ -441,25 +441,31 @@ static void wcnss_release_pds(struct qcom_wcnss *wcnss)
}
static int wcnss_init_regulators(struct qcom_wcnss *wcnss,
- const struct wcnss_vreg_info *info,
- int num_vregs, int num_pd_vregs)
+ const struct wcnss_data *data)
{
+ const struct wcnss_vreg_info *info = data->vregs;
struct regulator_bulk_data *bulk;
+ size_t i, possible_pds = 0, num_vregs = data->num_vregs;
int ret;
- int i;
+
+ for (i = 0; i < WCNSS_MAX_PDS; i++)
+ if (data->pd_names[i])
+ possible_pds++;
/*
* If attaching the power domains suceeded we can skip requesting
* the regulators for the power domains. For old device trees we need to
* reserve extra space to manage them through the regulator interface.
*/
- if (wcnss->num_pds) {
+ if (possible_pds >= num_vregs) {
+ /* Do nothing if vregs do not include PD regulators (pronto-v3) */
+ } else if (wcnss->num_pds) {
info += wcnss->num_pds;
/* Handle single power domain case */
- if (wcnss->num_pds < num_pd_vregs)
- num_vregs += num_pd_vregs - wcnss->num_pds;
+ if (wcnss->num_pds < data->num_pd_vregs)
+ num_vregs += data->num_pd_vregs - wcnss->num_pds;
} else {
- num_vregs += num_pd_vregs;
+ num_vregs += data->num_pd_vregs;
}
bulk = devm_kcalloc(wcnss->dev,
@@ -612,8 +618,7 @@ static int wcnss_probe(struct platform_device *pdev)
if (ret && (ret != -ENODATA || !data->num_pd_vregs))
return ret;
- ret = wcnss_init_regulators(wcnss, data->vregs, data->num_vregs,
- data->num_pd_vregs);
+ ret = wcnss_init_regulators(wcnss, data);
if (ret)
goto detach_pds;
@@ -702,7 +707,7 @@ MODULE_DEVICE_TABLE(of, wcnss_of_match);
static struct platform_driver wcnss_driver = {
.probe = wcnss_probe,
- .remove_new = wcnss_remove,
+ .remove = wcnss_remove,
.driver = {
.name = "qcom-wcnss-pil",
.of_match_table = wcnss_of_match,
diff --git a/drivers/remoteproc/rcar_rproc.c b/drivers/remoteproc/rcar_rproc.c
index cc17e8421f65..921d853594f4 100644
--- a/drivers/remoteproc/rcar_rproc.c
+++ b/drivers/remoteproc/rcar_rproc.c
@@ -214,7 +214,7 @@ MODULE_DEVICE_TABLE(of, rcar_rproc_of_match);
static struct platform_driver rcar_rproc_driver = {
.probe = rcar_rproc_probe,
- .remove_new = rcar_rproc_remove,
+ .remove = rcar_rproc_remove,
.driver = {
.name = "rcar-rproc",
.of_match_table = rcar_rproc_of_match,
diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c
index d3f39009b28e..25a655f33ec0 100644
--- a/drivers/remoteproc/remoteproc_virtio.c
+++ b/drivers/remoteproc/remoteproc_virtio.c
@@ -593,7 +593,7 @@ static void rproc_virtio_remove(struct platform_device *pdev)
/* Platform driver */
static struct platform_driver rproc_virtio_driver = {
.probe = rproc_virtio_probe,
- .remove_new = rproc_virtio_remove,
+ .remove = rproc_virtio_remove,
.driver = {
.name = "rproc-virtio",
},
diff --git a/drivers/remoteproc/st_remoteproc.c b/drivers/remoteproc/st_remoteproc.c
index 1340be9d0110..5df99bae7131 100644
--- a/drivers/remoteproc/st_remoteproc.c
+++ b/drivers/remoteproc/st_remoteproc.c
@@ -457,7 +457,7 @@ static void st_rproc_remove(struct platform_device *pdev)
static struct platform_driver st_rproc_driver = {
.probe = st_rproc_probe,
- .remove_new = st_rproc_remove,
+ .remove = st_rproc_remove,
.driver = {
.name = "st-rproc",
.of_match_table = of_match_ptr(st_rproc_match),
diff --git a/drivers/remoteproc/stm32_rproc.c b/drivers/remoteproc/stm32_rproc.c
index 8c7f7950b80e..b02b36a3f515 100644
--- a/drivers/remoteproc/stm32_rproc.c
+++ b/drivers/remoteproc/stm32_rproc.c
@@ -946,7 +946,7 @@ static DEFINE_SIMPLE_DEV_PM_OPS(stm32_rproc_pm_ops,
static struct platform_driver stm32_rproc_driver = {
.probe = stm32_rproc_probe,
- .remove_new = stm32_rproc_remove,
+ .remove = stm32_rproc_remove,
.driver = {
.name = "stm32-rproc",
.pm = pm_ptr(&stm32_rproc_pm_ops),
diff --git a/drivers/remoteproc/ti_k3_dsp_remoteproc.c b/drivers/remoteproc/ti_k3_dsp_remoteproc.c
index 73be3d216791..f48146e8077f 100644
--- a/drivers/remoteproc/ti_k3_dsp_remoteproc.c
+++ b/drivers/remoteproc/ti_k3_dsp_remoteproc.c
@@ -783,7 +783,7 @@ MODULE_DEVICE_TABLE(of, k3_dsp_of_match);
static struct platform_driver k3_dsp_rproc_driver = {
.probe = k3_dsp_rproc_probe,
- .remove_new = k3_dsp_rproc_remove,
+ .remove = k3_dsp_rproc_remove,
.driver = {
.name = "k3-dsp-rproc",
.of_match_table = k3_dsp_of_match,
diff --git a/drivers/remoteproc/wkup_m3_rproc.c b/drivers/remoteproc/wkup_m3_rproc.c
index c39bd2bf2c1e..35c2145b12db 100644
--- a/drivers/remoteproc/wkup_m3_rproc.c
+++ b/drivers/remoteproc/wkup_m3_rproc.c
@@ -251,7 +251,7 @@ static const struct dev_pm_ops wkup_m3_rproc_pm_ops = {
static struct platform_driver wkup_m3_rproc_driver = {
.probe = wkup_m3_rproc_probe,
- .remove_new = wkup_m3_rproc_remove,
+ .remove = wkup_m3_rproc_remove,
.driver = {
.name = "wkup_m3_rproc",
.of_match_table = wkup_m3_rproc_of_match,
diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c
index e7068016a986..5ed62f46b364 100644
--- a/drivers/s390/crypto/ap_bus.c
+++ b/drivers/s390/crypto/ap_bus.c
@@ -40,6 +40,7 @@
#include <linux/module.h>
#include <asm/uv.h>
#include <asm/chsc.h>
+#include <linux/mempool.h>
#include "ap_bus.h"
#include "ap_debug.h"
@@ -101,6 +102,27 @@ static struct ap_config_info *const ap_qci_info_old = &qci[1];
*/
debug_info_t *ap_dbf_info;
+/*
+ * There is a need for a do-not-allocate-memory path through the AP bus
+ * layer. The pkey layer may be triggered via the in-kernel interface from
+ * a protected key crypto algorithm (namely PAES) to convert a secure key
+ * into a protected key. This happens in a workqueue context, so sleeping
+ * is allowed but memory allocations causing IO operations are not permitted.
+ * To accomplish this, an AP message memory pool with pre-allocated space
+ * is established. When ap_init_apmsg() with use_mempool set to true is
+ * called, instead of kmalloc() the ap message buffer is allocated from
+ * the ap_msg_pool. This pool only holds a limited amount of buffers:
+ * ap_msg_pool_min_items with the item size AP_DEFAULT_MAX_MSG_SIZE and
+ * exactly one of these items (if available) is returned if ap_init_apmsg()
+ * with the use_mempool arg set to true is called. When this pool is exhausted
+ * and use_mempool is set true, ap_init_apmsg() returns -ENOMEM without
+ * any attempt to allocate memory and the caller has to deal with that.
+ */
+static mempool_t *ap_msg_pool;
+static unsigned int ap_msg_pool_min_items = 8;
+module_param_named(msgpool_min_items, ap_msg_pool_min_items, uint, 0440);
+MODULE_PARM_DESC(msgpool_min_items, "AP message pool minimal items");
+
/*
* AP bus rescan related things.
*/
@@ -546,6 +568,48 @@ static void ap_poll_thread_stop(void)
#define is_card_dev(x) ((x)->parent == ap_root_device)
#define is_queue_dev(x) ((x)->parent != ap_root_device)
+/*
+ * ap_init_apmsg() - Initialize ap_message.
+ */
+int ap_init_apmsg(struct ap_message *ap_msg, u32 flags)
+{
+ unsigned int maxmsgsize;
+
+ memset(ap_msg, 0, sizeof(*ap_msg));
+ ap_msg->flags = flags;
+
+ if (flags & AP_MSG_FLAG_MEMPOOL) {
+ ap_msg->msg = mempool_alloc_preallocated(ap_msg_pool);
+ if (!ap_msg->msg)
+ return -ENOMEM;
+ ap_msg->bufsize = AP_DEFAULT_MAX_MSG_SIZE;
+ return 0;
+ }
+
+ maxmsgsize = atomic_read(&ap_max_msg_size);
+ ap_msg->msg = kmalloc(maxmsgsize, GFP_KERNEL);
+ if (!ap_msg->msg)
+ return -ENOMEM;
+ ap_msg->bufsize = maxmsgsize;
+
+ return 0;
+}
+EXPORT_SYMBOL(ap_init_apmsg);
+
+/*
+ * ap_release_apmsg() - Release ap_message.
+ */
+void ap_release_apmsg(struct ap_message *ap_msg)
+{
+ if (ap_msg->flags & AP_MSG_FLAG_MEMPOOL) {
+ memzero_explicit(ap_msg->msg, ap_msg->bufsize);
+ mempool_free(ap_msg->msg, ap_msg_pool);
+ } else {
+ kfree_sensitive(ap_msg->msg);
+ }
+}
+EXPORT_SYMBOL(ap_release_apmsg);
+
/**
* ap_bus_match()
* @dev: Pointer to device
@@ -2431,6 +2495,14 @@ static int __init ap_module_init(void)
/* init ap_queue hashtable */
hash_init(ap_queues);
+ /* create ap msg buffer memory pool */
+ ap_msg_pool = mempool_create_kmalloc_pool(ap_msg_pool_min_items,
+ AP_DEFAULT_MAX_MSG_SIZE);
+ if (!ap_msg_pool) {
+ rc = -ENOMEM;
+ goto out;
+ }
+
/* set up the AP permissions (ioctls, ap and aq masks) */
ap_perms_init();
@@ -2477,6 +2549,7 @@ static int __init ap_module_init(void)
out_bus:
bus_unregister(&ap_bus_type);
out:
+ mempool_destroy(ap_msg_pool);
ap_debug_exit();
return rc;
}
@@ -2487,6 +2560,7 @@ static void __exit ap_module_exit(void)
ap_irq_exit();
root_device_unregister(ap_root_device);
bus_unregister(&ap_bus_type);
+ mempool_destroy(ap_msg_pool);
ap_debug_exit();
}
diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h
index 6111913c858c..4b7ffa840563 100644
--- a/drivers/s390/crypto/ap_bus.h
+++ b/drivers/s390/crypto/ap_bus.h
@@ -214,6 +214,11 @@ struct ap_queue {
typedef enum ap_sm_wait (ap_func_t)(struct ap_queue *queue);
+struct ap_response_type {
+ struct completion work;
+ int type;
+};
+
struct ap_message {
struct list_head list; /* Request queueing. */
unsigned long psmid; /* Message id. */
@@ -222,7 +227,7 @@ struct ap_message {
size_t bufsize; /* allocated msg buffer size */
u16 flags; /* Flags, see AP_MSG_FLAG_xxx */
int rc; /* Return code for this message */
- void *private; /* ap driver private pointer. */
+ struct ap_response_type response;
/* receive is called from tasklet context */
void (*receive)(struct ap_queue *, struct ap_message *,
struct ap_message *);
@@ -231,27 +236,10 @@ struct ap_message {
#define AP_MSG_FLAG_SPECIAL 0x0001 /* flag msg as 'special' with NQAP */
#define AP_MSG_FLAG_USAGE 0x0002 /* CCA, EP11: usage (no admin) msg */
#define AP_MSG_FLAG_ADMIN 0x0004 /* CCA, EP11: admin (=control) msg */
+#define AP_MSG_FLAG_MEMPOOL 0x0008 /* ap msg buffer allocated via mempool */
-/**
- * ap_init_message() - Initialize ap_message.
- * Initialize a message before using. Otherwise this might result in
- * unexpected behaviour.
- */
-static inline void ap_init_message(struct ap_message *ap_msg)
-{
- memset(ap_msg, 0, sizeof(*ap_msg));
-}
-
-/**
- * ap_release_message() - Release ap_message.
- * Releases all memory used internal within the ap_message struct
- * Currently this is the message and private field.
- */
-static inline void ap_release_message(struct ap_message *ap_msg)
-{
- kfree_sensitive(ap_msg->msg);
- kfree_sensitive(ap_msg->private);
-}
+int ap_init_apmsg(struct ap_message *ap_msg, u32 flags);
+void ap_release_apmsg(struct ap_message *ap_msg);
enum ap_sm_wait ap_sm_event(struct ap_queue *aq, enum ap_sm_event event);
enum ap_sm_wait ap_sm_event_loop(struct ap_queue *aq, enum ap_sm_event event);
diff --git a/drivers/s390/crypto/pkey_cca.c b/drivers/s390/crypto/pkey_cca.c
index 937051381720..6c4dac5fa256 100644
--- a/drivers/s390/crypto/pkey_cca.c
+++ b/drivers/s390/crypto/pkey_cca.c
@@ -227,7 +227,7 @@ static int cca_key2protkey(const struct pkey_apqn *apqns, size_t nr_apqns,
/* CCA AES data key */
if (keylen != sizeof(struct secaeskeytoken))
return -EINVAL;
- if (cca_check_secaeskeytoken(pkey_dbf_info, 3, key, 0))
+ if (cca_check_secaeskeytoken(pkey_dbf_info, 3, key, keylen, 0))
return -EINVAL;
} else if (hdr->type == TOKTYPE_CCA_INTERNAL &&
hdr->version == TOKVER_CCA_VLSC) {
@@ -235,12 +235,10 @@ static int cca_key2protkey(const struct pkey_apqn *apqns, size_t nr_apqns,
if (keylen < hdr->len || keylen > MAXCCAVLSCTOKENSIZE)
return -EINVAL;
if (cca_check_secaescipherkey(pkey_dbf_info,
- 3, key, 0, 1))
+ 3, key, keylen, 0, 1))
return -EINVAL;
} else if (hdr->type == TOKTYPE_CCA_INTERNAL_PKA) {
/* CCA ECC (private) key */
- if (keylen < sizeof(struct eccprivkeytoken))
- return -EINVAL;
if (cca_check_sececckeytoken(pkey_dbf_info, 3, key, keylen, 1))
return -EINVAL;
} else {
@@ -474,7 +472,7 @@ static int cca_verifykey(const u8 *key, u32 keylen,
hdr->version == TOKVER_CCA_AES) {
struct secaeskeytoken *t = (struct secaeskeytoken *)key;
- rc = cca_check_secaeskeytoken(pkey_dbf_info, 3, key, 0);
+ rc = cca_check_secaeskeytoken(pkey_dbf_info, 3, key, keylen, 0);
if (rc)
goto out;
*keytype = PKEY_TYPE_CCA_DATA;
@@ -501,7 +499,8 @@ static int cca_verifykey(const u8 *key, u32 keylen,
hdr->version == TOKVER_CCA_VLSC) {
struct cipherkeytoken *t = (struct cipherkeytoken *)key;
- rc = cca_check_secaescipherkey(pkey_dbf_info, 3, key, 0, 1);
+ rc = cca_check_secaescipherkey(pkey_dbf_info, 3,
+ key, keylen, 0, 1);
if (rc)
goto out;
*keytype = PKEY_TYPE_CCA_CIPHER;
diff --git a/drivers/s390/crypto/zcrypt_api.c b/drivers/s390/crypto/zcrypt_api.c
index 6bfebc3ef0a7..0ded84744b1e 100644
--- a/drivers/s390/crypto/zcrypt_api.c
+++ b/drivers/s390/crypto/zcrypt_api.c
@@ -642,16 +642,17 @@ static long zcrypt_rsa_modexpo(struct ap_perms *perms,
struct zcrypt_queue *zq, *pref_zq;
struct ap_message ap_msg;
unsigned int wgt = 0, pref_wgt = 0;
- unsigned int func_code;
- int cpen, qpen, qid = 0, rc = -ENODEV;
+ unsigned int func_code = 0;
+ int cpen, qpen, qid = 0, rc;
struct module *mod;
trace_s390_zcrypt_req(mex, TP_ICARSAMODEXPO);
- ap_init_message(&ap_msg);
+ rc = ap_init_apmsg(&ap_msg, 0);
+ if (rc)
+ goto out;
if (mex->outputdatalength < mex->inputdatalength) {
- func_code = 0;
rc = -EINVAL;
goto out;
}
@@ -728,7 +729,7 @@ static long zcrypt_rsa_modexpo(struct ap_perms *perms,
spin_unlock(&zcrypt_list_lock);
out:
- ap_release_message(&ap_msg);
+ ap_release_apmsg(&ap_msg);
if (tr) {
tr->last_rc = rc;
tr->last_qid = qid;
@@ -746,16 +747,17 @@ static long zcrypt_rsa_crt(struct ap_perms *perms,
struct zcrypt_queue *zq, *pref_zq;
struct ap_message ap_msg;
unsigned int wgt = 0, pref_wgt = 0;
- unsigned int func_code;
- int cpen, qpen, qid = 0, rc = -ENODEV;
+ unsigned int func_code = 0;
+ int cpen, qpen, qid = 0, rc;
struct module *mod;
trace_s390_zcrypt_req(crt, TP_ICARSACRT);
- ap_init_message(&ap_msg);
+ rc = ap_init_apmsg(&ap_msg, 0);
+ if (rc)
+ goto out;
if (crt->outputdatalength < crt->inputdatalength) {
- func_code = 0;
rc = -EINVAL;
goto out;
}
@@ -832,7 +834,7 @@ static long zcrypt_rsa_crt(struct ap_perms *perms,
spin_unlock(&zcrypt_list_lock);
out:
- ap_release_message(&ap_msg);
+ ap_release_apmsg(&ap_msg);
if (tr) {
tr->last_rc = rc;
tr->last_qid = qid;
@@ -842,23 +844,27 @@ static long zcrypt_rsa_crt(struct ap_perms *perms,
return rc;
}
-static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
+static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms,
struct zcrypt_track *tr,
struct ica_xcRB *xcrb)
{
- struct zcrypt_card *zc, *pref_zc;
+ bool userspace = xflags & ZCRYPT_XFLAG_USERSPACE;
+ unsigned int domain, func_code = 0;
+ unsigned int wgt = 0, pref_wgt = 0;
struct zcrypt_queue *zq, *pref_zq;
+ struct zcrypt_card *zc, *pref_zc;
+ int cpen, qpen, qid = 0, rc;
struct ap_message ap_msg;
- unsigned int wgt = 0, pref_wgt = 0;
- unsigned int func_code;
- unsigned short *domain, tdom;
- int cpen, qpen, qid = 0, rc = -ENODEV;
struct module *mod;
trace_s390_zcrypt_req(xcrb, TB_ZSECSENDCPRB);
xcrb->status = 0;
- ap_init_message(&ap_msg);
+
+ rc = ap_init_apmsg(&ap_msg, xflags & ZCRYPT_XFLAG_NOMEMALLOC ?
+ AP_MSG_FLAG_MEMPOOL : 0);
+ if (rc)
+ goto out;
rc = prep_cca_ap_msg(userspace, xcrb, &ap_msg, &func_code, &domain);
if (rc)
@@ -866,10 +872,10 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
print_hex_dump_debug("ccareq: ", DUMP_PREFIX_ADDRESS, 16, 1,
ap_msg.msg, ap_msg.len, false);
- tdom = *domain;
- if (perms != &ap_perms && tdom < AP_DOMAINS) {
+ if (perms != &ap_perms && domain < AP_DOMAINS) {
if (ap_msg.flags & AP_MSG_FLAG_ADMIN) {
- if (!test_bit_inv(tdom, perms->adm)) {
+ domain = array_index_nospec(domain, AP_DOMAINS);
+ if (!test_bit_inv(domain, perms->adm)) {
rc = -ENODEV;
goto out;
}
@@ -882,10 +888,10 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
* If a valid target domain is set and this domain is NOT a usage
* domain but a control only domain, autoselect target domain.
*/
- if (tdom < AP_DOMAINS &&
- !ap_test_config_usage_domain(tdom) &&
- ap_test_config_ctrl_domain(tdom))
- tdom = AUTOSEL_DOM;
+ if (domain < AP_DOMAINS &&
+ !ap_test_config_usage_domain(domain) &&
+ ap_test_config_ctrl_domain(domain))
+ domain = AUTOSEL_DOM;
pref_zc = NULL;
pref_zq = NULL;
@@ -917,8 +923,8 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
/* check for device usable and eligible */
if (!zq->online || !zq->ops->send_cprb ||
!ap_queue_usable(zq->queue) ||
- (tdom != AUTOSEL_DOM &&
- tdom != AP_QID_QUEUE(zq->queue->qid)))
+ (domain != AUTOSEL_DOM &&
+ domain != AP_QID_QUEUE(zq->queue->qid)))
continue;
/* check if device node has admission for this queue */
if (!zcrypt_check_queue(perms,
@@ -941,16 +947,11 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
if (!pref_zq) {
pr_debug("no match for address %02x.%04x => ENODEV\n",
- xcrb->user_defined, *domain);
+ xcrb->user_defined, domain);
rc = -ENODEV;
goto out;
}
- /* in case of auto select, provide the correct domain */
- qid = pref_zq->queue->qid;
- if (*domain == AUTOSEL_DOM)
- *domain = AP_QID_QUEUE(qid);
-
rc = pref_zq->ops->send_cprb(userspace, pref_zq, xcrb, &ap_msg);
if (!rc) {
print_hex_dump_debug("ccarpl: ", DUMP_PREFIX_ADDRESS, 16, 1,
@@ -962,7 +963,7 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
spin_unlock(&zcrypt_list_lock);
out:
- ap_release_message(&ap_msg);
+ ap_release_apmsg(&ap_msg);
if (tr) {
tr->last_rc = rc;
tr->last_qid = qid;
@@ -972,7 +973,7 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
return rc;
}
-long zcrypt_send_cprb(struct ica_xcRB *xcrb)
+long zcrypt_send_cprb(struct ica_xcRB *xcrb, u32 xflags)
{
struct zcrypt_track tr;
int rc;
@@ -980,13 +981,13 @@ long zcrypt_send_cprb(struct ica_xcRB *xcrb)
memset(&tr, 0, sizeof(tr));
do {
- rc = _zcrypt_send_cprb(false, &ap_perms, &tr, xcrb);
+ rc = _zcrypt_send_cprb(xflags, &ap_perms, &tr, xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
/* on ENODEV failure: retry once again after a requested rescan */
if (rc == -ENODEV && zcrypt_process_rescan())
do {
- rc = _zcrypt_send_cprb(false, &ap_perms, &tr, xcrb);
+ rc = _zcrypt_send_cprb(xflags, &ap_perms, &tr, xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
@@ -1024,23 +1025,27 @@ static bool is_desired_ep11_queue(unsigned int dev_qid,
return false;
}
-static long _zcrypt_send_ep11_cprb(bool userspace, struct ap_perms *perms,
+static long _zcrypt_send_ep11_cprb(u32 xflags, struct ap_perms *perms,
struct zcrypt_track *tr,
struct ep11_urb *xcrb)
{
+ bool userspace = xflags & ZCRYPT_XFLAG_USERSPACE;
struct zcrypt_card *zc, *pref_zc;
struct zcrypt_queue *zq, *pref_zq;
struct ep11_target_dev *targets;
unsigned short target_num;
unsigned int wgt = 0, pref_wgt = 0;
- unsigned int func_code, domain;
+ unsigned int func_code = 0, domain;
struct ap_message ap_msg;
- int cpen, qpen, qid = 0, rc = -ENODEV;
+ int cpen, qpen, qid = 0, rc;
struct module *mod;
trace_s390_zcrypt_req(xcrb, TP_ZSENDEP11CPRB);
- ap_init_message(&ap_msg);
+ rc = ap_init_apmsg(&ap_msg, xflags & ZCRYPT_XFLAG_NOMEMALLOC ?
+ AP_MSG_FLAG_MEMPOOL : 0);
+ if (rc)
+ goto out;
target_num = (unsigned short)xcrb->targets_num;
@@ -1073,6 +1078,7 @@ static long _zcrypt_send_ep11_cprb(bool userspace, struct ap_perms *perms,
if (perms != &ap_perms && domain < AP_DOMAINS) {
if (ap_msg.flags & AP_MSG_FLAG_ADMIN) {
+ domain = array_index_nospec(domain, AP_DOMAINS);
if (!test_bit_inv(domain, perms->adm)) {
rc = -ENODEV;
goto out_free;
@@ -1164,7 +1170,7 @@ static long _zcrypt_send_ep11_cprb(bool userspace, struct ap_perms *perms,
out_free:
kfree(targets);
out:
- ap_release_message(&ap_msg);
+ ap_release_apmsg(&ap_msg);
if (tr) {
tr->last_rc = rc;
tr->last_qid = qid;
@@ -1174,7 +1180,7 @@ static long _zcrypt_send_ep11_cprb(bool userspace, struct ap_perms *perms,
return rc;
}
-long zcrypt_send_ep11_cprb(struct ep11_urb *xcrb)
+long zcrypt_send_ep11_cprb(struct ep11_urb *xcrb, u32 xflags)
{
struct zcrypt_track tr;
int rc;
@@ -1182,13 +1188,13 @@ long zcrypt_send_ep11_cprb(struct ep11_urb *xcrb)
memset(&tr, 0, sizeof(tr));
do {
- rc = _zcrypt_send_ep11_cprb(false, &ap_perms, &tr, xcrb);
+ rc = _zcrypt_send_ep11_cprb(xflags, &ap_perms, &tr, xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
/* on ENODEV failure: retry once again after a requested rescan */
if (rc == -ENODEV && zcrypt_process_rescan())
do {
- rc = _zcrypt_send_ep11_cprb(false, &ap_perms, &tr, xcrb);
+ rc = _zcrypt_send_ep11_cprb(xflags, &ap_perms, &tr, xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
@@ -1204,16 +1210,17 @@ static long zcrypt_rng(char *buffer)
struct zcrypt_card *zc, *pref_zc;
struct zcrypt_queue *zq, *pref_zq;
unsigned int wgt = 0, pref_wgt = 0;
- unsigned int func_code;
+ unsigned int func_code = 0;
struct ap_message ap_msg;
- unsigned int domain;
int qid = 0, rc = -ENODEV;
struct module *mod;
trace_s390_zcrypt_req(buffer, TP_HWRNGCPRB);
- ap_init_message(&ap_msg);
- rc = prep_rng_ap_msg(&ap_msg, &func_code, &domain);
+ rc = ap_init_apmsg(&ap_msg, 0);
+ if (rc)
+ goto out;
+ rc = prep_rng_ap_msg(&ap_msg, &func_code, NULL);
if (rc)
goto out;
@@ -1258,7 +1265,7 @@ static long zcrypt_rng(char *buffer)
spin_unlock(&zcrypt_list_lock);
out:
- ap_release_message(&ap_msg);
+ ap_release_apmsg(&ap_msg);
trace_s390_zcrypt_rep(buffer, func_code, rc,
AP_QID_CARD(qid), AP_QID_QUEUE(qid));
return rc;
@@ -1523,6 +1530,7 @@ static int zsecsendcprb_ioctl(struct ap_perms *perms, unsigned long arg)
int rc;
struct ica_xcRB xcrb;
struct zcrypt_track tr;
+ u32 xflags = ZCRYPT_XFLAG_USERSPACE;
struct ica_xcRB __user *uxcrb = (void __user *)arg;
memset(&tr, 0, sizeof(tr));
@@ -1530,13 +1538,13 @@ static int zsecsendcprb_ioctl(struct ap_perms *perms, unsigned long arg)
return -EFAULT;
do {
- rc = _zcrypt_send_cprb(true, perms, &tr, &xcrb);
+ rc = _zcrypt_send_cprb(xflags, perms, &tr, &xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
/* on ENODEV failure: retry once again after a requested rescan */
if (rc == -ENODEV && zcrypt_process_rescan())
do {
- rc = _zcrypt_send_cprb(true, perms, &tr, &xcrb);
+ rc = _zcrypt_send_cprb(xflags, perms, &tr, &xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
@@ -1553,6 +1561,7 @@ static int zsendep11cprb_ioctl(struct ap_perms *perms, unsigned long arg)
int rc;
struct ep11_urb xcrb;
struct zcrypt_track tr;
+ u32 xflags = ZCRYPT_XFLAG_USERSPACE;
struct ep11_urb __user *uxcrb = (void __user *)arg;
memset(&tr, 0, sizeof(tr));
@@ -1560,13 +1569,13 @@ static int zsendep11cprb_ioctl(struct ap_perms *perms, unsigned long arg)
return -EFAULT;
do {
- rc = _zcrypt_send_ep11_cprb(true, perms, &tr, &xcrb);
+ rc = _zcrypt_send_ep11_cprb(xflags, perms, &tr, &xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
/* on ENODEV failure: retry once again after a requested rescan */
if (rc == -ENODEV && zcrypt_process_rescan())
do {
- rc = _zcrypt_send_ep11_cprb(true, perms, &tr, &xcrb);
+ rc = _zcrypt_send_ep11_cprb(xflags, perms, &tr, &xcrb);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
@@ -1827,6 +1836,7 @@ static long trans_xcrb32(struct ap_perms *perms, struct file *filp,
unsigned int cmd, unsigned long arg)
{
struct compat_ica_xcrb __user *uxcrb32 = compat_ptr(arg);
+ u32 xflags = ZCRYPT_XFLAG_USERSPACE;
struct compat_ica_xcrb xcrb32;
struct zcrypt_track tr;
struct ica_xcRB xcrb64;
@@ -1856,13 +1866,13 @@ static long trans_xcrb32(struct ap_perms *perms, struct file *filp,
xcrb64.priority_window = xcrb32.priority_window;
xcrb64.status = xcrb32.status;
do {
- rc = _zcrypt_send_cprb(true, perms, &tr, &xcrb64);
+ rc = _zcrypt_send_cprb(xflags, perms, &tr, &xcrb64);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
/* on ENODEV failure: retry once again after a requested rescan */
if (rc == -ENODEV && zcrypt_process_rescan())
do {
- rc = _zcrypt_send_cprb(true, perms, &tr, &xcrb64);
+ rc = _zcrypt_send_cprb(xflags, perms, &tr, &xcrb64);
} while (rc == -EAGAIN && ++tr.again_counter < TRACK_AGAIN_MAX);
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
diff --git a/drivers/s390/crypto/zcrypt_api.h b/drivers/s390/crypto/zcrypt_api.h
index 4ed481df57ca..44f158329545 100644
--- a/drivers/s390/crypto/zcrypt_api.h
+++ b/drivers/s390/crypto/zcrypt_api.h
@@ -76,6 +76,13 @@ struct zcrypt_track {
#define TRACK_AGAIN_CARD_WEIGHT_PENALTY 1000
#define TRACK_AGAIN_QUEUE_WEIGHT_PENALTY 10000
+/*
+ * xflags - to be used with zcrypt_send_cprb() and
+ * zcrypt_send_ep11_cprb() for the xflags parameter.
+ */
+#define ZCRYPT_XFLAG_USERSPACE 0x0001 /* data ptrs address userspace */
+#define ZCRYPT_XFLAG_NOMEMALLOC 0x0002 /* do not allocate memory via kmalloc */
+
struct zcrypt_ops {
long (*rsa_modexpo)(struct zcrypt_queue *, struct ica_rsa_modexpo *,
struct ap_message *);
@@ -161,8 +168,8 @@ void zcrypt_msgtype_unregister(struct zcrypt_ops *);
struct zcrypt_ops *zcrypt_msgtype(unsigned char *, int);
int zcrypt_api_init(void);
void zcrypt_api_exit(void);
-long zcrypt_send_cprb(struct ica_xcRB *xcRB);
-long zcrypt_send_ep11_cprb(struct ep11_urb *urb);
+long zcrypt_send_cprb(struct ica_xcRB *xcRB, u32 xflags);
+long zcrypt_send_ep11_cprb(struct ep11_urb *urb, u32 xflags);
void zcrypt_device_status_mask_ext(struct zcrypt_device_status_ext *devstatus);
int zcrypt_device_status_ext(int card, int queue,
struct zcrypt_device_status_ext *devstatus);
diff --git a/drivers/s390/crypto/zcrypt_ccamisc.c b/drivers/s390/crypto/zcrypt_ccamisc.c
index 48d36d5a8711..f46819cfa5f0 100644
--- a/drivers/s390/crypto/zcrypt_ccamisc.c
+++ b/drivers/s390/crypto/zcrypt_ccamisc.c
@@ -46,12 +46,18 @@ static DEFINE_SPINLOCK(cca_info_list_lock);
* also checked. Returns 0 on success or errno value on failure.
*/
int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize)
+ const u8 *token, u32 keysize, int keybitsize)
{
struct secaeskeytoken *t = (struct secaeskeytoken *)token;
#define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__)
+ if (keysize < sizeof(*t)) {
+ if (dbg)
+ DBF("%s keysize %u < min token size %zu\n",
+ __func__, keysize, sizeof(*t));
+ return -EINVAL;
+ }
if (t->type != TOKTYPE_CCA_INTERNAL) {
if (dbg)
DBF("%s token check failed, type 0x%02x != 0x%02x\n",
@@ -85,14 +91,20 @@ EXPORT_SYMBOL(cca_check_secaeskeytoken);
* Returns 0 on success or errno value on failure.
*/
int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize,
- int checkcpacfexport)
+ const u8 *token, u32 keysize,
+ int keybitsize, int checkcpacfexport)
{
struct cipherkeytoken *t = (struct cipherkeytoken *)token;
bool keybitsizeok = true;
#define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__)
+ if (keysize < sizeof(*t)) {
+ if (dbg)
+ DBF("%s keysize %u < min token size %zu\n",
+ __func__, keysize, sizeof(*t));
+ return -EINVAL;
+ }
if (t->type != TOKTYPE_CCA_INTERNAL) {
if (dbg)
DBF("%s token check failed, type 0x%02x != 0x%02x\n",
@@ -105,6 +117,18 @@ int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl,
__func__, (int)t->version, TOKVER_CCA_VLSC);
return -EINVAL;
}
+ if (t->len > keysize) {
+ if (dbg)
+ DBF("%s token check failed, len %d > keysize %u\n",
+ __func__, (int)t->len, keysize);
+ return -EINVAL;
+ }
+ if (t->len < sizeof(*t)) {
+ if (dbg)
+ DBF("%s token check failed, len %d < min token size %zu\n",
+ __func__, (int)t->len, sizeof(*t));
+ return -EINVAL;
+ }
if (t->algtype != 0x02) {
if (dbg)
DBF("%s token check failed, algtype 0x%02x != 0x02\n",
@@ -179,6 +203,12 @@ int cca_check_sececckeytoken(debug_info_t *dbg, int dbflvl,
#define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__)
+ if (keysize < sizeof(*t)) {
+ if (dbg)
+ DBF("%s keysize %u < min token size %zu\n",
+ __func__, keysize, sizeof(*t));
+ return -EINVAL;
+ }
if (t->type != TOKTYPE_CCA_INTERNAL_PKA) {
if (dbg)
DBF("%s token check failed, type 0x%02x != 0x%02x\n",
@@ -191,6 +221,12 @@ int cca_check_sececckeytoken(debug_info_t *dbg, int dbflvl,
__func__, (int)t->len, keysize);
return -EINVAL;
}
+ if (t->len < sizeof(*t)) {
+ if (dbg)
+ DBF("%s token check failed, len %d < min token size %zu\n",
+ __func__, (int)t->len, sizeof(*t));
+ return -EINVAL;
+ }
if (t->secid != 0x20) {
if (dbg)
DBF("%s token check failed, secid 0x%02x != 0x20\n",
@@ -379,7 +415,7 @@ int cca_genseckey(u16 cardnr, u16 domain,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, errno %d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -414,7 +450,8 @@ int cca_genseckey(u16 cardnr, u16 domain,
/* check secure key token */
rc = cca_check_secaeskeytoken(zcrypt_dbf_info, DBF_ERR,
- prepparm->lv3.keyblock.tok, 8 * keysize);
+ prepparm->lv3.keyblock.tok,
+ seckeysize, 8 * keysize);
if (rc) {
rc = -EIO;
goto out;
@@ -517,7 +554,7 @@ int cca_clr2seckey(u16 cardnr, u16 domain, u32 keybitsize,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -552,7 +589,8 @@ int cca_clr2seckey(u16 cardnr, u16 domain, u32 keybitsize,
/* check secure key token */
rc = cca_check_secaeskeytoken(zcrypt_dbf_info, DBF_ERR,
- prepparm->lv3.keyblock.tok, 8 * keysize);
+ prepparm->lv3.keyblock.tok,
+ seckeysize, 8 * keysize);
if (rc) {
rc = -EIO;
goto out;
@@ -644,7 +682,7 @@ int cca_sec2protkey(u16 cardnr, u16 domain,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -811,6 +849,7 @@ int cca_gencipherkey(u16 cardnr, u16 domain, u32 keybitsize, u32 keygenflags,
} kb;
} __packed * prepparm;
struct cipherkeytoken *t;
+ u32 keybuflen;
/* get already prepared memory for 2 cprbs with param block each */
rc = alloc_and_prep_cprbmem(PARMBSIZE, &mem, &preqcblk, &prepcblk);
@@ -872,7 +911,7 @@ int cca_gencipherkey(u16 cardnr, u16 domain, u32 keybitsize, u32 keygenflags,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -904,23 +943,28 @@ int cca_gencipherkey(u16 cardnr, u16 domain, u32 keybitsize, u32 keygenflags,
}
/* and some checks on the generated key */
+ t = (struct cipherkeytoken *)prepparm->kb.tlv1.gen_key;
+ if (prepparm->kb.tlv1.len < 2 * sizeof(uint16_t) + sizeof(*t)) {
+ rc = -EIO;
+ goto out;
+ }
+ keybuflen = prepparm->kb.tlv1.len - 2 * sizeof(uint16_t);
rc = cca_check_secaescipherkey(zcrypt_dbf_info, DBF_ERR,
prepparm->kb.tlv1.gen_key,
- keybitsize, 1);
+ keybuflen, keybitsize, 1);
if (rc) {
rc = -EIO;
goto out;
}
/* copy the generated vlsc key token */
- t = (struct cipherkeytoken *)prepparm->kb.tlv1.gen_key;
if (keybuf) {
- if (*keybufsize >= t->len)
- memcpy(keybuf, t, t->len);
+ if (*keybufsize >= keybuflen)
+ memcpy(keybuf, t, keybuflen);
else
rc = -EINVAL;
}
- *keybufsize = t->len;
+ *keybufsize = keybuflen;
out:
free_cprbmem(mem, PARMBSIZE, 0);
@@ -1039,7 +1083,7 @@ static int _ip_cprb_helper(u16 cardnr, u16 domain,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -1258,7 +1302,7 @@ int cca_cipher2protkey(u16 cardnr, u16 domain, const u8 *ckey,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -1424,7 +1468,7 @@ int cca_ecc2protkey(u16 cardnr, u16 domain, const u8 *key,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -1538,7 +1582,7 @@ int cca_query_crypto_facility(u16 cardnr, u16 domain,
prep_xcrb(&xcrb, cardnr, preqcblk, prepcblk);
/* forward xcrb with request CPRB and reply CPRB to zcrypt dd */
- rc = zcrypt_send_cprb(&xcrb);
+ rc = zcrypt_send_cprb(&xcrb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_cprb (cardnr=%d domain=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
diff --git a/drivers/s390/crypto/zcrypt_ccamisc.h b/drivers/s390/crypto/zcrypt_ccamisc.h
index aed7e8384542..bdddebcaad78 100644
--- a/drivers/s390/crypto/zcrypt_ccamisc.h
+++ b/drivers/s390/crypto/zcrypt_ccamisc.h
@@ -135,7 +135,7 @@ struct eccprivkeytoken {
* also checked. Returns 0 on success or errno value on failure.
*/
int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize);
+ const u8 *token, u32 keysize, int keybitsize);
/*
* Simple check if the token is a valid CCA secure AES cipher key
@@ -145,8 +145,8 @@ int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl,
* Returns 0 on success or errno value on failure.
*/
int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize,
- int checkcpacfexport);
+ const u8 *token, u32 keysize,
+ int keybitsize, int checkcpacfexport);
/*
* Simple check if the token is a valid CCA secure ECC private
diff --git a/drivers/s390/crypto/zcrypt_ep11misc.c b/drivers/s390/crypto/zcrypt_ep11misc.c
index cb7e6da43602..b60e262bcaa3 100644
--- a/drivers/s390/crypto/zcrypt_ep11misc.c
+++ b/drivers/s390/crypto/zcrypt_ep11misc.c
@@ -636,7 +636,7 @@ static int ep11_query_info(u16 cardnr, u16 domain, u32 query_type,
req, sizeof(*req) + sizeof(*req_pl),
rep, sizeof(*rep) + sizeof(*rep_pl) + buflen);
- rc = zcrypt_send_ep11_cprb(urb);
+ rc = zcrypt_send_ep11_cprb(urb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_ep11_cprb(card=%d dom=%d) failed, rc=%d\n",
__func__, (int)cardnr, (int)domain, rc);
@@ -892,7 +892,7 @@ static int _ep11_genaeskey(u16 card, u16 domain,
req, sizeof(*req) + req_pl_size,
rep, sizeof(*rep) + sizeof(*rep_pl));
- rc = zcrypt_send_ep11_cprb(urb);
+ rc = zcrypt_send_ep11_cprb(urb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_ep11_cprb(card=%d dom=%d) failed, rc=%d\n",
__func__, (int)card, (int)domain, rc);
@@ -1049,7 +1049,7 @@ static int ep11_cryptsingle(u16 card, u16 domain,
req, sizeof(*req) + req_pl_size,
rep, sizeof(*rep) + rep_pl_size);
- rc = zcrypt_send_ep11_cprb(urb);
+ rc = zcrypt_send_ep11_cprb(urb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_ep11_cprb(card=%d dom=%d) failed, rc=%d\n",
__func__, (int)card, (int)domain, rc);
@@ -1212,7 +1212,7 @@ static int _ep11_unwrapkey(u16 card, u16 domain,
req, sizeof(*req) + req_pl_size,
rep, sizeof(*rep) + sizeof(*rep_pl));
- rc = zcrypt_send_ep11_cprb(urb);
+ rc = zcrypt_send_ep11_cprb(urb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_ep11_cprb(card=%d dom=%d) failed, rc=%d\n",
__func__, (int)card, (int)domain, rc);
@@ -1372,7 +1372,7 @@ static int _ep11_wrapkey(u16 card, u16 domain,
req, sizeof(*req) + req_pl_size,
rep, sizeof(*rep) + sizeof(*rep_pl));
- rc = zcrypt_send_ep11_cprb(urb);
+ rc = zcrypt_send_ep11_cprb(urb, 0);
if (rc) {
ZCRYPT_DBF_ERR("%s zcrypt_send_ep11_cprb(card=%d dom=%d) failed, rc=%d\n",
__func__, (int)card, (int)domain, rc);
diff --git a/drivers/s390/crypto/zcrypt_msgtype50.c b/drivers/s390/crypto/zcrypt_msgtype50.c
index adc65eddaa1e..fc0a2a053dc2 100644
--- a/drivers/s390/crypto/zcrypt_msgtype50.c
+++ b/drivers/s390/crypto/zcrypt_msgtype50.c
@@ -438,7 +438,7 @@ static void zcrypt_msgtype50_receive(struct ap_queue *aq,
msg->len = sizeof(error_reply);
}
out:
- complete((struct completion *)msg->private);
+ complete(&msg->response.work);
}
static atomic_t zcrypt_step = ATOMIC_INIT(0);
@@ -449,30 +449,30 @@ static atomic_t zcrypt_step = ATOMIC_INIT(0);
* @zq: pointer to zcrypt_queue structure that identifies the
* CEXxA device to the request distributor
* @mex: pointer to the modexpo request buffer
+ * This function assumes that ap_msg has been initialized with
+ * ap_init_apmsg() and thus a valid buffer with the size of
+ * ap_msg->bufsize is available within ap_msg. Also the caller has
+ * to make sure ap_release_apmsg() is always called even on failure.
*/
static long zcrypt_msgtype50_modexpo(struct zcrypt_queue *zq,
struct ica_rsa_modexpo *mex,
struct ap_message *ap_msg)
{
- struct completion work;
int rc;
- ap_msg->bufsize = MSGTYPE50_CRB3_MAX_MSG_SIZE;
- ap_msg->msg = kmalloc(ap_msg->bufsize, GFP_KERNEL);
- if (!ap_msg->msg)
- return -ENOMEM;
+ if (ap_msg->bufsize < MSGTYPE50_CRB3_MAX_MSG_SIZE)
+ return -EMSGSIZE;
ap_msg->receive = zcrypt_msgtype50_receive;
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
- ap_msg->private = &work;
rc = ICAMEX_msg_to_type50MEX_msg(zq, ap_msg, mex);
if (rc)
goto out;
- init_completion(&work);
+ init_completion(&ap_msg->response.work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
goto out;
- rc = wait_for_completion_interruptible(&work);
+ rc = wait_for_completion_interruptible(&ap_msg->response.work);
if (rc == 0) {
rc = ap_msg->rc;
if (rc == 0)
@@ -485,7 +485,6 @@ static long zcrypt_msgtype50_modexpo(struct zcrypt_queue *zq,
}
out:
- ap_msg->private = NULL;
if (rc)
pr_debug("send me cprb at dev=%02x.%04x rc=%d\n",
AP_QID_CARD(zq->queue->qid),
@@ -499,30 +498,30 @@ static long zcrypt_msgtype50_modexpo(struct zcrypt_queue *zq,
* @zq: pointer to zcrypt_queue structure that identifies the
* CEXxA device to the request distributor
* @crt: pointer to the modexpoc_crt request buffer
+ * This function assumes that ap_msg has been initialized with
+ * ap_init_apmsg() and thus a valid buffer with the size of
+ * ap_msg->bufsize is available within ap_msg. Also the caller has
+ * to make sure ap_release_apmsg() is always called even on failure.
*/
static long zcrypt_msgtype50_modexpo_crt(struct zcrypt_queue *zq,
struct ica_rsa_modexpo_crt *crt,
struct ap_message *ap_msg)
{
- struct completion work;
int rc;
- ap_msg->bufsize = MSGTYPE50_CRB3_MAX_MSG_SIZE;
- ap_msg->msg = kmalloc(ap_msg->bufsize, GFP_KERNEL);
- if (!ap_msg->msg)
- return -ENOMEM;
+ if (ap_msg->bufsize < MSGTYPE50_CRB3_MAX_MSG_SIZE)
+ return -EMSGSIZE;
ap_msg->receive = zcrypt_msgtype50_receive;
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
- ap_msg->private = &work;
rc = ICACRT_msg_to_type50CRT_msg(zq, ap_msg, crt);
if (rc)
goto out;
- init_completion(&work);
+ init_completion(&ap_msg->response.work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
goto out;
- rc = wait_for_completion_interruptible(&work);
+ rc = wait_for_completion_interruptible(&ap_msg->response.work);
if (rc == 0) {
rc = ap_msg->rc;
if (rc == 0)
@@ -535,7 +534,6 @@ static long zcrypt_msgtype50_modexpo_crt(struct zcrypt_queue *zq,
}
out:
- ap_msg->private = NULL;
if (rc)
pr_debug("send crt cprb at dev=%02x.%04x rc=%d\n",
AP_QID_CARD(zq->queue->qid),
diff --git a/drivers/s390/crypto/zcrypt_msgtype6.c b/drivers/s390/crypto/zcrypt_msgtype6.c
index b64c9d9fc613..257838470559 100644
--- a/drivers/s390/crypto/zcrypt_msgtype6.c
+++ b/drivers/s390/crypto/zcrypt_msgtype6.c
@@ -31,11 +31,6 @@
#define CEIL4(x) ((((x) + 3) / 4) * 4)
-struct response_type {
- struct completion work;
- int type;
-};
-
#define CEXXC_RESPONSE_TYPE_ICA 0
#define CEXXC_RESPONSE_TYPE_XCRB 1
#define CEXXC_RESPONSE_TYPE_EP11 2
@@ -334,7 +329,7 @@ struct type86_fmt2_msg {
static int xcrb_msg_to_type6cprb_msgx(bool userspace, struct ap_message *ap_msg,
struct ica_xcRB *xcrb,
unsigned int *fcode,
- unsigned short **dom)
+ unsigned int *domain)
{
static struct type6_hdr static_type6_hdrX = {
.type = 0x06,
@@ -418,7 +413,8 @@ static int xcrb_msg_to_type6cprb_msgx(bool userspace, struct ap_message *ap_msg,
sizeof(msg->hdr.function_code));
*fcode = (msg->hdr.function_code[0] << 8) | msg->hdr.function_code[1];
- *dom = (unsigned short *)&msg->cprbx.domain;
+ if (domain)
+ *domain = msg->cprbx.domain;
/* check subfunction, US and AU need special flag with NQAP */
if (memcmp(function_code, "US", 2) == 0 ||
@@ -535,7 +531,8 @@ static int xcrb_msg_to_type6_ep11cprb_msgx(bool userspace, struct ap_message *ap
else
ap_msg->flags |= AP_MSG_FLAG_USAGE;
- *domain = msg->cprbx.target_id;
+ if (domain)
+ *domain = msg->cprbx.target_id;
return 0;
}
@@ -856,7 +853,7 @@ static void zcrypt_msgtype6_receive(struct ap_queue *aq,
.type = TYPE82_RSP_CODE,
.reply_code = REP82_ERROR_MACHINE_FAILURE,
};
- struct response_type *resp_type = msg->private;
+ struct ap_response_type *resp_type = &msg->response;
struct type86x_reply *t86r;
int len;
@@ -920,7 +917,7 @@ static void zcrypt_msgtype6_receive_ep11(struct ap_queue *aq,
.type = TYPE82_RSP_CODE,
.reply_code = REP82_ERROR_MACHINE_FAILURE,
};
- struct response_type *resp_type = msg->private;
+ struct ap_response_type *resp_type = &msg->response;
struct type86_ep11_reply *t86r;
int len;
@@ -959,6 +956,10 @@ static atomic_t zcrypt_step = ATOMIC_INIT(0);
/*
* The request distributor calls this function if it picked the CEXxC
* device to handle a modexpo request.
+ * This function assumes that ap_msg has been initialized with
+ * ap_init_apmsg() and thus a valid buffer with the size of
+ * ap_msg->bufsize is available within ap_msg. Also the caller has
+ * to make sure ap_release_apmsg() is always called even on failure.
* @zq: pointer to zcrypt_queue structure that identifies the
* CEXxC device to the request distributor
* @mex: pointer to the modexpo request buffer
@@ -967,27 +968,21 @@ static long zcrypt_msgtype6_modexpo(struct zcrypt_queue *zq,
struct ica_rsa_modexpo *mex,
struct ap_message *ap_msg)
{
- struct response_type resp_type = {
- .type = CEXXC_RESPONSE_TYPE_ICA,
- };
+ struct ap_response_type *resp_type = &ap_msg->response;
int rc;
- ap_msg->msg = (void *)get_zeroed_page(GFP_KERNEL);
- if (!ap_msg->msg)
- return -ENOMEM;
- ap_msg->bufsize = PAGE_SIZE;
ap_msg->receive = zcrypt_msgtype6_receive;
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
- ap_msg->private = &resp_type;
rc = icamex_msg_to_type6mex_msgx(zq, ap_msg, mex);
if (rc)
- goto out_free;
- init_completion(&resp_type.work);
+ goto out;
+ resp_type->type = CEXXC_RESPONSE_TYPE_ICA;
+ init_completion(&resp_type->work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
- goto out_free;
- rc = wait_for_completion_interruptible(&resp_type.work);
+ goto out;
+ rc = wait_for_completion_interruptible(&resp_type->work);
if (rc == 0) {
rc = ap_msg->rc;
if (rc == 0)
@@ -999,16 +994,17 @@ static long zcrypt_msgtype6_modexpo(struct zcrypt_queue *zq,
ap_cancel_message(zq->queue, ap_msg);
}
-out_free:
- free_page((unsigned long)ap_msg->msg);
- ap_msg->private = NULL;
- ap_msg->msg = NULL;
+out:
return rc;
}
/*
* The request distributor calls this function if it picked the CEXxC
* device to handle a modexpo_crt request.
+ * This function assumes that ap_msg has been initialized with
+ * ap_init_apmsg() and thus a valid buffer with the size of
+ * ap_msg->bufsize is available within ap_msg. Also the caller has
+ * to make sure ap_release_apmsg() is always called even on failure.
* @zq: pointer to zcrypt_queue structure that identifies the
* CEXxC device to the request distributor
* @crt: pointer to the modexpoc_crt request buffer
@@ -1017,27 +1013,21 @@ static long zcrypt_msgtype6_modexpo_crt(struct zcrypt_queue *zq,
struct ica_rsa_modexpo_crt *crt,
struct ap_message *ap_msg)
{
- struct response_type resp_type = {
- .type = CEXXC_RESPONSE_TYPE_ICA,
- };
+ struct ap_response_type *resp_type = &ap_msg->response;
int rc;
- ap_msg->msg = (void *)get_zeroed_page(GFP_KERNEL);
- if (!ap_msg->msg)
- return -ENOMEM;
- ap_msg->bufsize = PAGE_SIZE;
ap_msg->receive = zcrypt_msgtype6_receive;
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
- ap_msg->private = &resp_type;
rc = icacrt_msg_to_type6crt_msgx(zq, ap_msg, crt);
if (rc)
- goto out_free;
- init_completion(&resp_type.work);
+ goto out;
+ resp_type->type = CEXXC_RESPONSE_TYPE_ICA;
+ init_completion(&resp_type->work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
- goto out_free;
- rc = wait_for_completion_interruptible(&resp_type.work);
+ goto out;
+ rc = wait_for_completion_interruptible(&resp_type->work);
if (rc == 0) {
rc = ap_msg->rc;
if (rc == 0)
@@ -1049,10 +1039,7 @@ static long zcrypt_msgtype6_modexpo_crt(struct zcrypt_queue *zq,
ap_cancel_message(zq->queue, ap_msg);
}
-out_free:
- free_page((unsigned long)ap_msg->msg);
- ap_msg->private = NULL;
- ap_msg->msg = NULL;
+out:
return rc;
}
@@ -1061,29 +1048,23 @@ static long zcrypt_msgtype6_modexpo_crt(struct zcrypt_queue *zq,
* Prepare a CCA AP msg: fetch the required data from userspace,
* prepare the AP msg, fill some info into the ap_message struct,
* extract some data from the CPRB and give back to the caller.
- * This function allocates memory and needs an ap_msg prepared
- * by the caller with ap_init_message(). Also the caller has to
- * make sure ap_release_message() is always called even on failure.
+ * This function assumes that ap_msg has been initialized with
+ * ap_init_apmsg() and thus a valid buffer with the size of
+ * ap_msg->bufsize is available within ap_msg. Also the caller has
+ * to make sure ap_release_apmsg() is always called even on failure.
*/
int prep_cca_ap_msg(bool userspace, struct ica_xcRB *xcrb,
struct ap_message *ap_msg,
- unsigned int *func_code, unsigned short **dom)
+ unsigned int *func_code, unsigned int *domain)
{
- struct response_type resp_type = {
- .type = CEXXC_RESPONSE_TYPE_XCRB,
- };
+ struct ap_response_type *resp_type = &ap_msg->response;
- ap_msg->bufsize = atomic_read(&ap_max_msg_size);
- ap_msg->msg = kmalloc(ap_msg->bufsize, GFP_KERNEL);
- if (!ap_msg->msg)
- return -ENOMEM;
ap_msg->receive = zcrypt_msgtype6_receive;
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
- ap_msg->private = kmemdup(&resp_type, sizeof(resp_type), GFP_KERNEL);
- if (!ap_msg->private)
- return -ENOMEM;
- return xcrb_msg_to_type6cprb_msgx(userspace, ap_msg, xcrb, func_code, dom);
+ resp_type->type = CEXXC_RESPONSE_TYPE_XCRB;
+ return xcrb_msg_to_type6cprb_msgx(userspace, ap_msg,
+ xcrb, func_code, domain);
}
/*
@@ -1097,7 +1078,7 @@ static long zcrypt_msgtype6_send_cprb(bool userspace, struct zcrypt_queue *zq,
struct ica_xcRB *xcrb,
struct ap_message *ap_msg)
{
- struct response_type *rtype = ap_msg->private;
+ struct ap_response_type *resp_type = &ap_msg->response;
struct {
struct type6_hdr hdr;
struct CPRBX cprbx;
@@ -1128,11 +1109,14 @@ static long zcrypt_msgtype6_send_cprb(bool userspace, struct zcrypt_queue *zq,
msg->hdr.fromcardlen1 -= delta;
}
- init_completion(&rtype->work);
+ /* update domain field within the CPRB struct */
+ msg->cprbx.domain = AP_QID_QUEUE(zq->queue->qid);
+
+ init_completion(&resp_type->work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
goto out;
- rc = wait_for_completion_interruptible(&rtype->work);
+ rc = wait_for_completion_interruptible(&resp_type->work);
if (rc == 0) {
rc = ap_msg->rc;
if (rc == 0)
@@ -1158,28 +1142,21 @@ static long zcrypt_msgtype6_send_cprb(bool userspace, struct zcrypt_queue *zq,
* Prepare an EP11 AP msg: fetch the required data from userspace,
* prepare the AP msg, fill some info into the ap_message struct,
* extract some data from the CPRB and give back to the caller.
- * This function allocates memory and needs an ap_msg prepared
- * by the caller with ap_init_message(). Also the caller has to
- * make sure ap_release_message() is always called even on failure.
+ * This function assumes that ap_msg has been initialized with
+ * ap_init_apmsg() and thus a valid buffer with the size of
+ * ap_msg->bufsize is available within ap_msg. Also the caller has
+ * to make sure ap_release_apmsg() is always called even on failure.
*/
int prep_ep11_ap_msg(bool userspace, struct ep11_urb *xcrb,
struct ap_message *ap_msg,
unsigned int *func_code, unsigned int *domain)
{
- struct response_type resp_type = {
- .type = CEXXC_RESPONSE_TYPE_EP11,
- };
+ struct ap_response_type *resp_type = &ap_msg->response;
- ap_msg->bufsize = atomic_read(&ap_max_msg_size);
- ap_msg->msg = kmalloc(ap_msg->bufsize, GFP_KERNEL);
- if (!ap_msg->msg)
- return -ENOMEM;
ap_msg->receive = zcrypt_msgtype6_receive_ep11;
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
- ap_msg->private = kmemdup(&resp_type, sizeof(resp_type), GFP_KERNEL);
- if (!ap_msg->private)
- return -ENOMEM;
+ resp_type->type = CEXXC_RESPONSE_TYPE_EP11;
return xcrb_msg_to_type6_ep11cprb_msgx(userspace, ap_msg, xcrb,
func_code, domain);
}
@@ -1197,7 +1174,7 @@ static long zcrypt_msgtype6_send_ep11_cprb(bool userspace, struct zcrypt_queue *
{
int rc;
unsigned int lfmt;
- struct response_type *rtype = ap_msg->private;
+ struct ap_response_type *resp_type = &ap_msg->response;
struct {
struct type6_hdr hdr;
struct ep11_cprb cprbx;
@@ -1240,8 +1217,7 @@ static long zcrypt_msgtype6_send_ep11_cprb(bool userspace, struct zcrypt_queue *
lfmt = 1; /* length format #1 */
}
payload_hdr = (struct pld_hdr *)((&msg->pld_lenfmt) + lfmt);
- payload_hdr->dom_val = (unsigned int)
- AP_QID_QUEUE(zq->queue->qid);
+ payload_hdr->dom_val = AP_QID_QUEUE(zq->queue->qid);
}
/*
@@ -1251,11 +1227,11 @@ static long zcrypt_msgtype6_send_ep11_cprb(bool userspace, struct zcrypt_queue *
msg->hdr.fromcardlen1 = zq->reply.bufsize -
sizeof(struct type86_hdr) - sizeof(struct type86_fmt2_ext);
- init_completion(&rtype->work);
+ init_completion(&resp_type->work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
goto out;
- rc = wait_for_completion_interruptible(&rtype->work);
+ rc = wait_for_completion_interruptible(&resp_type->work);
if (rc == 0) {
rc = ap_msg->rc;
if (rc == 0)
@@ -1276,23 +1252,75 @@ static long zcrypt_msgtype6_send_ep11_cprb(bool userspace, struct zcrypt_queue *
return rc;
}
+/**
+ * Prepare a type6 CPRB message for random number generation
+ *
+ * @ap_dev: AP device pointer
+ * @ap_msg: pointer to AP message
+ */
+static inline void rng_type6cprb_msgx(struct ap_message *ap_msg,
+ unsigned int random_number_length,
+ unsigned int *domain)
+{
+ struct {
+ struct type6_hdr hdr;
+ struct CPRBX cprbx;
+ char function_code[2];
+ short int rule_length;
+ char rule[8];
+ short int verb_length;
+ short int key_length;
+ } __packed * msg = ap_msg->msg;
+ static struct type6_hdr static_type6_hdrX = {
+ .type = 0x06,
+ .offset1 = 0x00000058,
+ .agent_id = {'C', 'A'},
+ .function_code = {'R', 'L'},
+ .tocardlen1 = sizeof(*msg) - sizeof(msg->hdr),
+ .fromcardlen1 = sizeof(*msg) - sizeof(msg->hdr),
+ };
+ static struct CPRBX local_cprbx = {
+ .cprb_len = 0x00dc,
+ .cprb_ver_id = 0x02,
+ .func_id = {0x54, 0x32},
+ .req_parml = sizeof(*msg) - sizeof(msg->hdr) -
+ sizeof(msg->cprbx),
+ .rpl_msgbl = sizeof(*msg) - sizeof(msg->hdr),
+ };
+
+ msg->hdr = static_type6_hdrX;
+ msg->hdr.fromcardlen2 = random_number_length;
+ msg->cprbx = local_cprbx;
+ msg->cprbx.rpl_datal = random_number_length;
+ memcpy(msg->function_code, msg->hdr.function_code, 0x02);
+ msg->rule_length = 0x0a;
+ memcpy(msg->rule, "RANDOM ", 8);
+ msg->verb_length = 0x02;
+ msg->key_length = 0x02;
+ ap_msg->len = sizeof(*msg);
+ if (domain)
+ *domain = msg->cprbx.domain;
+}
+
+/*
+ * Prepare a CEXXC get random request ap message.
+ * This function assumes that ap_msg has been initialized with
+ * ap_init_apmsg() and thus a valid buffer with the size of
+ * ap_max_msg_size is available within ap_msg. Also the caller has
+ * to make sure ap_release_apmsg() is always called even on failure.
+ */
int prep_rng_ap_msg(struct ap_message *ap_msg, int *func_code,
unsigned int *domain)
{
- struct response_type resp_type = {
- .type = CEXXC_RESPONSE_TYPE_XCRB,
- };
+ struct ap_response_type *resp_type = &ap_msg->response;
- ap_msg->bufsize = AP_DEFAULT_MAX_MSG_SIZE;
- ap_msg->msg = kmalloc(ap_msg->bufsize, GFP_KERNEL);
- if (!ap_msg->msg)
- return -ENOMEM;
+ if (ap_msg->bufsize < AP_DEFAULT_MAX_MSG_SIZE)
+ return -EMSGSIZE;
ap_msg->receive = zcrypt_msgtype6_receive;
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
- ap_msg->private = kmemdup(&resp_type, sizeof(resp_type), GFP_KERNEL);
- if (!ap_msg->private)
- return -ENOMEM;
+
+ resp_type->type = CEXXC_RESPONSE_TYPE_XCRB;
rng_type6cprb_msgx(ap_msg, ZCRYPT_RNG_BUFFER_SIZE, domain);
@@ -1319,16 +1347,16 @@ static long zcrypt_msgtype6_rng(struct zcrypt_queue *zq,
short int verb_length;
short int key_length;
} __packed * msg = ap_msg->msg;
- struct response_type *rtype = ap_msg->private;
+ struct ap_response_type *resp_type = &ap_msg->response;
int rc;
msg->cprbx.domain = AP_QID_QUEUE(zq->queue->qid);
- init_completion(&rtype->work);
+ init_completion(&resp_type->work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
goto out;
- rc = wait_for_completion_interruptible(&rtype->work);
+ rc = wait_for_completion_interruptible(&resp_type->work);
if (rc == 0) {
rc = ap_msg->rc;
if (rc == 0)
diff --git a/drivers/s390/crypto/zcrypt_msgtype6.h b/drivers/s390/crypto/zcrypt_msgtype6.h
index 6f5ced8d6cda..9cbe6e2b2d33 100644
--- a/drivers/s390/crypto/zcrypt_msgtype6.h
+++ b/drivers/s390/crypto/zcrypt_msgtype6.h
@@ -96,7 +96,7 @@ struct type86_fmt2_ext {
int prep_cca_ap_msg(bool userspace, struct ica_xcRB *xcrb,
struct ap_message *ap_msg,
- unsigned int *fc, unsigned short **dom);
+ unsigned int *fc, unsigned int *dom);
int prep_ep11_ap_msg(bool userspace, struct ep11_urb *xcrb,
struct ap_message *ap_msg,
unsigned int *fc, unsigned int *dom);
@@ -110,55 +110,6 @@ int prep_rng_ap_msg(struct ap_message *ap_msg,
int speed_idx_cca(int);
int speed_idx_ep11(int);
-/**
- * Prepare a type6 CPRB message for random number generation
- *
- * @ap_dev: AP device pointer
- * @ap_msg: pointer to AP message
- */
-static inline void rng_type6cprb_msgx(struct ap_message *ap_msg,
- unsigned int random_number_length,
- unsigned int *domain)
-{
- struct {
- struct type6_hdr hdr;
- struct CPRBX cprbx;
- char function_code[2];
- short int rule_length;
- char rule[8];
- short int verb_length;
- short int key_length;
- } __packed * msg = ap_msg->msg;
- static struct type6_hdr static_type6_hdrX = {
- .type = 0x06,
- .offset1 = 0x00000058,
- .agent_id = {'C', 'A'},
- .function_code = {'R', 'L'},
- .tocardlen1 = sizeof(*msg) - sizeof(msg->hdr),
- .fromcardlen1 = sizeof(*msg) - sizeof(msg->hdr),
- };
- static struct CPRBX local_cprbx = {
- .cprb_len = 0x00dc,
- .cprb_ver_id = 0x02,
- .func_id = {0x54, 0x32},
- .req_parml = sizeof(*msg) - sizeof(msg->hdr) -
- sizeof(msg->cprbx),
- .rpl_msgbl = sizeof(*msg) - sizeof(msg->hdr),
- };
-
- msg->hdr = static_type6_hdrX;
- msg->hdr.fromcardlen2 = random_number_length;
- msg->cprbx = local_cprbx;
- msg->cprbx.rpl_datal = random_number_length;
- memcpy(msg->function_code, msg->hdr.function_code, 0x02);
- msg->rule_length = 0x0a;
- memcpy(msg->rule, "RANDOM ", 8);
- msg->verb_length = 0x02;
- msg->key_length = 0x02;
- ap_msg->len = sizeof(*msg);
- *domain = (unsigned short)msg->cprbx.domain;
-}
-
void zcrypt_msgtype6_init(void);
void zcrypt_msgtype6_exit(void);
diff --git a/drivers/s390/net/ism_drv.c b/drivers/s390/net/ism_drv.c
index f2264cf6f40d..c728095fc505 100644
--- a/drivers/s390/net/ism_drv.c
+++ b/drivers/s390/net/ism_drv.c
@@ -244,13 +244,16 @@ static int unregister_sba(struct ism_dev *ism)
if (ret && ret != ISM_ERROR)
return -EIO;
+ return 0;
+}
+
+static void ism_free_sba(struct ism_dev *ism)
+{
dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
ism->sba, ism->sba_dma_addr);
ism->sba = NULL;
ism->sba_dma_addr = 0;
-
- return 0;
}
static int unregister_ieq(struct ism_dev *ism)
@@ -264,13 +267,16 @@ static int unregister_ieq(struct ism_dev *ism)
if (ret && ret != ISM_ERROR)
return -EIO;
+ return 0;
+}
+
+static void ism_free_ieq(struct ism_dev *ism)
+{
dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
ism->ieq, ism->ieq_dma_addr);
ism->ieq = NULL;
ism->ieq_dma_addr = 0;
-
- return 0;
}
static int ism_read_local_gid(struct ism_dev *ism)
@@ -580,8 +586,10 @@ static int ism_dev_init(struct ism_dev *ism)
unreg_ieq:
unregister_ieq(ism);
+ ism_free_ieq(ism);
unreg_sba:
unregister_sba(ism);
+ ism_free_sba(ism);
free_irq:
free_irq(pci_irq_vector(pdev, 0), ism);
free_client_arr:
@@ -678,9 +686,13 @@ static void ism_dev_exit(struct ism_dev *ism)
if (ism_v2_capable)
ism_del_vlan_id(ism, ISM_RESERVED_VLANID);
+ /* ism will only generate new IRQs while ieq & sba are registered */
unregister_ieq(ism);
unregister_sba(ism);
+ /* drain ongoing irpt handlers */
free_irq(pci_irq_vector(pdev, 0), ism);
+ ism_free_ieq(ism);
+ ism_free_sba(ism);
kfree(ism->sba_client_arr);
pci_free_irq_vectors(pdev);
list_del_init(&ism->list);
diff --git a/drivers/scsi/ch.c b/drivers/scsi/ch.c
index fa07a6f54003..c2c018236517 100644
--- a/drivers/scsi/ch.c
+++ b/drivers/scsi/ch.c
@@ -112,7 +112,6 @@ typedef struct {
int minor;
char name[8];
struct scsi_device *device;
- struct scsi_device **dt; /* ptrs to data transfer elements */
u_int firsts[CH_TYPES];
u_int counts[CH_TYPES];
u_int voltags;
@@ -363,16 +362,10 @@ ch_readconfig(scsi_changer *ch)
vendor_labels[i]);
}
- /* look up the devices of the data transfer elements */
- ch->dt = kcalloc(ch->counts[CHET_DT], sizeof(*ch->dt),
- GFP_KERNEL);
-
- if (!ch->dt) {
- kfree(buffer);
- return -ENOMEM;
- }
-
+ /* report the devices of the data transfer elements */
for (elem = 0; elem < ch->counts[CHET_DT]; elem++) {
+ struct scsi_device *sdev;
+
id = -1;
lun = 0;
if (elem < CH_DT_MAX && -1 != dt_id[elem]) {
@@ -388,10 +381,8 @@ ch_readconfig(scsi_changer *ch)
VPRINTK(KERN_INFO, "dt 0x%x: ",elem+ch->firsts[CHET_DT]);
if (data[6] & 0x80) {
VPRINTK(KERN_CONT, "not this SCSI bus\n");
- ch->dt[elem] = NULL;
} else if (0 == (data[6] & 0x30)) {
VPRINTK(KERN_CONT, "ID/LUN unknown\n");
- ch->dt[elem] = NULL;
} else {
id = ch->device->id;
lun = 0;
@@ -401,18 +392,16 @@ ch_readconfig(scsi_changer *ch)
}
if (-1 != id) {
VPRINTK(KERN_CONT, "ID %i, LUN %i, ",id,lun);
- ch->dt[elem] =
- scsi_device_lookup(ch->device->host,
- ch->device->channel,
- id,lun);
- if (!ch->dt[elem]) {
+ sdev = scsi_device_lookup(ch->device->host,
+ ch->device->channel,
+ id, lun);
+ if (!sdev) {
/* should not happen */
VPRINTK(KERN_CONT, "Huh? device not found!\n");
} else {
VPRINTK(KERN_CONT, "name: %8.8s %16.16s %4.4s\n",
- ch->dt[elem]->vendor,
- ch->dt[elem]->model,
- ch->dt[elem]->rev);
+ sdev->vendor, sdev->model, sdev->rev);
+ scsi_device_put(sdev);
}
}
}
@@ -576,7 +565,6 @@ static void ch_destroy(struct kref *ref)
scsi_changer *ch = container_of(ref, scsi_changer, ref);
ch->device = NULL;
- kfree(ch->dt);
kfree(ch);
}
diff --git a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
index b7dd4efca0c7..64aad320a04a 100644
--- a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
+++ b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
@@ -2978,7 +2978,7 @@ static int device_configure_v3_hw(struct scsi_device *sdev,
return 0;
if (!device_link_add(&sdev->sdev_gendev, dev,
- DL_FLAG_PM_RUNTIME | DL_FLAG_RPM_ACTIVE)) {
+ DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME | DL_FLAG_RPM_ACTIVE)) {
if (pm_runtime_enabled(dev)) {
dev_info(dev, "add device link failed, disable runtime PM for the host\n");
pm_runtime_disable(dev);
@@ -2988,6 +2988,15 @@ static int device_configure_v3_hw(struct scsi_device *sdev,
return 0;
}
+static void hisi_sas_sdev_destroy(struct scsi_device *sdev)
+{
+ struct Scsi_Host *shost = dev_to_shost(&sdev->sdev_gendev);
+ struct hisi_hba *hisi_hba = shost_priv(shost);
+ struct device *dev = hisi_hba->dev;
+
+ device_link_remove(&sdev->sdev_gendev, dev);
+}
+
static struct attribute *host_v3_hw_attrs[] = {
&dev_attr_phy_event_threshold.attr,
&dev_attr_intr_conv_v3_hw.attr,
@@ -3402,6 +3411,7 @@ static const struct scsi_host_template sht_v3_hw = {
.sg_tablesize = HISI_SAS_SGE_PAGE_CNT,
.sg_prot_tablesize = HISI_SAS_SGE_PAGE_CNT,
.slave_alloc = hisi_sas_slave_alloc,
+ .slave_destroy = hisi_sas_sdev_destroy,
.shost_groups = host_v3_hw_groups,
.sdev_groups = sdev_groups_v3_hw,
.tag_alloc_policy = BLK_TAG_ALLOC_RR,
diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c
index 17231a8dab98..c8a44969a6c3 100644
--- a/drivers/scsi/qla2xxx/qla_iocb.c
+++ b/drivers/scsi/qla2xxx/qla_iocb.c
@@ -2758,7 +2758,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode,
if (!elsio->u.els_logo.els_logo_pyld) {
/* ref: INIT */
kref_put(&sp->cmd_kref, qla2x00_sp_release);
- qla2x00_free_fcport(fcport);
return QLA_FUNCTION_FAILED;
}
@@ -2783,7 +2782,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode,
if (rval != QLA_SUCCESS) {
/* ref: INIT */
kref_put(&sp->cmd_kref, qla2x00_sp_release);
- qla2x00_free_fcport(fcport);
return QLA_FUNCTION_FAILED;
}
diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index 457fb3458ca7..56330f6a396e 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -1888,10 +1888,26 @@ __qla2x00_abort_all_cmds(struct qla_qpair *qp, int res)
continue;
}
cmd = (struct qla_tgt_cmd *)sp;
- cmd->aborted = 1;
+
+ if (cmd->sg_mapped)
+ qlt_unmap_sg(vha, cmd);
+
+ if (cmd->state == QLA_TGT_STATE_NEED_DATA) {
+ cmd->aborted = 1;
+ cmd->write_data_transferred = 0;
+ cmd->state = QLA_TGT_STATE_DATA_IN;
+ ha->tgt.tgt_ops->handle_data(cmd);
+ } else {
+ ha->tgt.tgt_ops->free_cmd(cmd);
+ }
break;
case TYPE_TGT_TMCMD:
- /* Skip task management functions. */
+ /*
+ * Currently, only ABTS response gets on the
+ * outstanding_cmds[]
+ */
+ ha->tgt.tgt_ops->free_mcmd(
+ (struct qla_tgt_mgmt_cmd *) sp);
break;
default:
break;
diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
index 11eadb3bd36e..a1f913fb84cb 100644
--- a/drivers/scsi/qla2xxx/qla_target.c
+++ b/drivers/scsi/qla2xxx/qla_target.c
@@ -2487,7 +2487,7 @@ static int qlt_pci_map_calc_cnt(struct qla_tgt_prm *prm)
return -1;
}
-static void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd)
+void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd)
{
struct qla_hw_data *ha;
struct qla_qpair *qpair;
@@ -3817,9 +3817,6 @@ int qlt_abort_cmd(struct qla_tgt_cmd *cmd)
spin_lock_irqsave(&cmd->cmd_lock, flags);
if (cmd->aborted) {
- if (cmd->sg_mapped)
- qlt_unmap_sg(vha, cmd);
-
spin_unlock_irqrestore(&cmd->cmd_lock, flags);
/*
* It's normal to see 2 calls in this path:
diff --git a/drivers/scsi/qla2xxx/qla_target.h b/drivers/scsi/qla2xxx/qla_target.h
index 354fca2e7feb..3644ab140ff9 100644
--- a/drivers/scsi/qla2xxx/qla_target.h
+++ b/drivers/scsi/qla2xxx/qla_target.h
@@ -1059,6 +1059,7 @@ extern int qlt_abort_cmd(struct qla_tgt_cmd *);
extern void qlt_xmit_tm_rsp(struct qla_tgt_mgmt_cmd *);
extern void qlt_free_mcmd(struct qla_tgt_mgmt_cmd *);
extern void qlt_free_cmd(struct qla_tgt_cmd *cmd);
+extern void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd);
extern void qlt_async_event(uint16_t, struct scsi_qla_host *, uint16_t *);
extern void qlt_enable_vha(struct scsi_qla_host *);
extern void qlt_vport_create(struct scsi_qla_host *, struct qla_hw_data *);
diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 3ecdcb21040c..94b873ec7344 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -2338,6 +2338,7 @@ static void scsi_unjam_host(struct Scsi_Host *shost)
int scsi_error_handler(void *data)
{
struct Scsi_Host *shost = data;
+ bool eh_noresume;
/*
* We use TASK_INTERRUPTIBLE so that the thread is not
@@ -2379,7 +2380,8 @@ int scsi_error_handler(void *data)
* what we need to do to get it up and online again (if we can).
* If we fail, we end up taking the thing offline.
*/
- if (!shost->eh_noresume && scsi_autopm_get_host(shost) != 0) {
+ eh_noresume = READ_ONCE(shost->eh_noresume);
+ if (!eh_noresume && scsi_autopm_get_host(shost) != 0) {
SCSI_LOG_ERROR_RECOVERY(1,
shost_printk(KERN_ERR, shost,
"scsi_eh_%d: unable to autoresume\n",
@@ -2403,7 +2405,7 @@ int scsi_error_handler(void *data)
* which are still online.
*/
scsi_restart_operations(shost);
- if (!shost->eh_noresume)
+ if (!eh_noresume)
scsi_autopm_put_host(shost);
}
__set_current_state(TASK_RUNNING);
diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index 7def1f64f1ec..e217cb38d55e 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -102,8 +102,6 @@ MODULE_ALIAS_SCSI_DEVICE(TYPE_ZBC);
#define SD_MINORS 16
-static void sd_config_discard(struct scsi_disk *sdkp, struct queue_limits *lim,
- unsigned int mode);
static void sd_config_write_same(struct scsi_disk *sdkp,
struct queue_limits *lim);
static int sd_revalidate_disk(struct gendisk *);
@@ -121,6 +119,62 @@ static const char *sd_cache_types[] = {
"write back, no read (daft)"
};
+static void sd_disable_discard(struct scsi_disk *sdkp)
+{
+ sdkp->provisioning_mode = SD_LBP_DISABLE;
+ blk_queue_disable_discard(sdkp->disk->queue);
+}
+
+static void sd_config_discard(struct scsi_disk *sdkp, struct queue_limits *lim,
+ unsigned int mode)
+{
+ unsigned int logical_block_size = sdkp->device->sector_size;
+ unsigned int max_blocks = 0;
+
+ lim->discard_alignment = sdkp->unmap_alignment * logical_block_size;
+ lim->discard_granularity = max(sdkp->physical_block_size,
+ sdkp->unmap_granularity * logical_block_size);
+ sdkp->provisioning_mode = mode;
+
+ switch (mode) {
+
+ case SD_LBP_FULL:
+ case SD_LBP_DISABLE:
+ break;
+
+ case SD_LBP_UNMAP:
+ max_blocks = min_not_zero(sdkp->max_unmap_blocks,
+ (u32)SD_MAX_WS16_BLOCKS);
+ break;
+
+ case SD_LBP_WS16:
+ if (sdkp->device->unmap_limit_for_ws)
+ max_blocks = sdkp->max_unmap_blocks;
+ else
+ max_blocks = sdkp->max_ws_blocks;
+
+ max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS16_BLOCKS);
+ break;
+
+ case SD_LBP_WS10:
+ if (sdkp->device->unmap_limit_for_ws)
+ max_blocks = sdkp->max_unmap_blocks;
+ else
+ max_blocks = sdkp->max_ws_blocks;
+
+ max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS10_BLOCKS);
+ break;
+
+ case SD_LBP_ZERO:
+ max_blocks = min_not_zero(sdkp->max_ws_blocks,
+ (u32)SD_MAX_WS10_BLOCKS);
+ break;
+ }
+
+ lim->max_hw_discard_sectors = max_blocks *
+ (logical_block_size >> SECTOR_SHIFT);
+}
+
static void sd_set_flush_flag(struct scsi_disk *sdkp,
struct queue_limits *lim)
{
@@ -836,60 +890,15 @@ static unsigned char sd_setup_protect_cmnd(struct scsi_cmnd *scmd,
return protect;
}
-static void sd_disable_discard(struct scsi_disk *sdkp)
-{
- sdkp->provisioning_mode = SD_LBP_DISABLE;
- blk_queue_disable_discard(sdkp->disk->queue);
-}
-
-static void sd_config_discard(struct scsi_disk *sdkp, struct queue_limits *lim,
- unsigned int mode)
+static void sd_uninit_command(struct scsi_cmnd *cmd)
{
- unsigned int logical_block_size = sdkp->device->sector_size;
- unsigned int max_blocks = 0;
-
- lim->discard_alignment = sdkp->unmap_alignment * logical_block_size;
- lim->discard_granularity = max(sdkp->physical_block_size,
- sdkp->unmap_granularity * logical_block_size);
- sdkp->provisioning_mode = mode;
-
- switch (mode) {
-
- case SD_LBP_FULL:
- case SD_LBP_DISABLE:
- break;
-
- case SD_LBP_UNMAP:
- max_blocks = min_not_zero(sdkp->max_unmap_blocks,
- (u32)SD_MAX_WS16_BLOCKS);
- break;
-
- case SD_LBP_WS16:
- if (sdkp->device->unmap_limit_for_ws)
- max_blocks = sdkp->max_unmap_blocks;
- else
- max_blocks = sdkp->max_ws_blocks;
-
- max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS16_BLOCKS);
- break;
-
- case SD_LBP_WS10:
- if (sdkp->device->unmap_limit_for_ws)
- max_blocks = sdkp->max_unmap_blocks;
- else
- max_blocks = sdkp->max_ws_blocks;
-
- max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS10_BLOCKS);
- break;
+ struct request *rq = scsi_cmd_to_rq(cmd);
- case SD_LBP_ZERO:
- max_blocks = min_not_zero(sdkp->max_ws_blocks,
- (u32)SD_MAX_WS10_BLOCKS);
- break;
- }
+ if (!(rq->rq_flags & RQF_SPECIAL_PAYLOAD))
+ return;
- lim->max_hw_discard_sectors = max_blocks *
- (logical_block_size >> SECTOR_SHIFT);
+ mempool_free(rq->special_vec.bv_page, sd_page_pool);
+ rq->rq_flags &= ~RQF_SPECIAL_PAYLOAD;
}
static void *sd_set_special_bvec(struct request *rq, unsigned int data_len)
@@ -914,6 +923,7 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd)
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
unsigned int data_len = 24;
char *buf;
+ blk_status_t ret;
buf = sd_set_special_bvec(rq, data_len);
if (!buf)
@@ -932,7 +942,10 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd)
cmd->transfersize = data_len;
rq->timeout = SD_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static void sd_config_atomic(struct scsi_disk *sdkp, struct queue_limits *lim)
@@ -1002,6 +1015,7 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd,
u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq));
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
u32 data_len = sdp->sector_size;
+ blk_status_t ret;
if (!sd_set_special_bvec(rq, data_len))
return BLK_STS_RESOURCE;
@@ -1017,7 +1031,10 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd,
cmd->transfersize = data_len;
rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
@@ -1029,6 +1046,7 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq));
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
u32 data_len = sdp->sector_size;
+ blk_status_t ret;
if (!sd_set_special_bvec(rq, data_len))
return BLK_STS_RESOURCE;
@@ -1044,7 +1062,10 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
cmd->transfersize = data_len;
rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static blk_status_t sd_setup_write_zeroes_cmnd(struct scsi_cmnd *cmd)
@@ -1468,14 +1489,6 @@ static blk_status_t sd_init_command(struct scsi_cmnd *cmd)
}
}
-static void sd_uninit_command(struct scsi_cmnd *SCpnt)
-{
- struct request *rq = scsi_cmd_to_rq(SCpnt);
-
- if (rq->rq_flags & RQF_SPECIAL_PAYLOAD)
- mempool_free(rq->special_vec.bv_page, sd_page_pool);
-}
-
static bool sd_need_revalidate(struct gendisk *disk, struct scsi_disk *sdkp)
{
if (sdkp->device->removable || sdkp->write_prot) {
diff --git a/drivers/sh/clk/core.c b/drivers/sh/clk/core.c
index 7a73f5e4a1fc..f02e12dfa5f6 100644
--- a/drivers/sh/clk/core.c
+++ b/drivers/sh/clk/core.c
@@ -569,7 +569,7 @@ long clk_round_rate(struct clk *clk, unsigned long rate)
EXPORT_SYMBOL_GPL(clk_round_rate);
#ifdef CONFIG_PM
-static void clks_core_resume(void)
+static void clks_core_resume(void *data)
{
struct clk *clkp;
@@ -588,13 +588,17 @@ static void clks_core_resume(void)
}
}
-static struct syscore_ops clks_syscore_ops = {
+static const struct syscore_ops clks_syscore_ops = {
.resume = clks_core_resume,
};
+static struct syscore clks_syscore = {
+ .ops = &clks_syscore_ops,
+};
+
static int __init clk_syscore_init(void)
{
- register_syscore_ops(&clks_syscore_ops);
+ register_syscore(&clks_syscore);
return 0;
}
diff --git a/drivers/sh/intc/core.c b/drivers/sh/intc/core.c
index ea571eeb3078..3dde703b7766 100644
--- a/drivers/sh/intc/core.c
+++ b/drivers/sh/intc/core.c
@@ -394,7 +394,7 @@ int __init register_intc_controller(struct intc_desc *desc)
return -ENOMEM;
}
-static int intc_suspend(void)
+static int intc_suspend(void *data)
{
struct intc_desc_int *d;
@@ -420,7 +420,7 @@ static int intc_suspend(void)
return 0;
}
-static void intc_resume(void)
+static void intc_resume(void *data)
{
struct intc_desc_int *d;
@@ -450,11 +450,15 @@ static void intc_resume(void)
}
}
-struct syscore_ops intc_syscore_ops = {
+static const struct syscore_ops intc_syscore_ops = {
.suspend = intc_suspend,
.resume = intc_resume,
};
+static struct syscore intc_syscore = {
+ .ops = &intc_syscore_ops,
+};
+
const struct bus_type intc_subsys = {
.name = "intc",
.dev_name = "intc",
@@ -477,7 +481,7 @@ static int __init register_intc_devs(void)
struct intc_desc_int *d;
int error;
- register_syscore_ops(&intc_syscore_ops);
+ register_syscore(&intc_syscore);
error = subsys_system_register(&intc_subsys, NULL);
if (!error) {
diff --git a/drivers/soc/aspeed/aspeed-lpc-snoop.c b/drivers/soc/aspeed/aspeed-lpc-snoop.c
index fc3a2c41cc10..35f58eb0f134 100644
--- a/drivers/soc/aspeed/aspeed-lpc-snoop.c
+++ b/drivers/soc/aspeed/aspeed-lpc-snoop.c
@@ -11,6 +11,7 @@
*/
#include <linux/bitops.h>
+#include <linux/cleanup.h>
#include <linux/clk.h>
#include <linux/interrupt.h>
#include <linux/fs.h>
@@ -59,6 +60,7 @@ struct aspeed_lpc_snoop_model_data {
struct aspeed_lpc_snoop_channel {
bool enabled;
+ spinlock_t lock;
struct kfifo fifo;
wait_queue_head_t wq;
struct miscdevice miscdev;
@@ -82,6 +84,7 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer,
size_t count, loff_t *ppos)
{
struct aspeed_lpc_snoop_channel *chan = snoop_file_to_chan(file);
+ u8 *buf __free(kfree) = NULL;
unsigned int copied;
int ret = 0;
@@ -93,9 +96,16 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer,
if (ret == -ERESTARTSYS)
return -EINTR;
}
- ret = kfifo_to_user(&chan->fifo, buffer, count, &copied);
- if (ret)
- return ret;
+
+ count = min_t(size_t, count, SNOOP_FIFO_SIZE);
+
+ buf = kmalloc(count, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ copied = kfifo_out_spinlocked(&chan->fifo, buf, count, &chan->lock);
+ if (copied && copy_to_user(buffer, buf, copied))
+ return -EFAULT;
return copied;
}
@@ -119,11 +129,13 @@ static const struct file_operations snoop_fops = {
/* Save a byte to a FIFO and discard the oldest byte if FIFO is full */
static void put_fifo_with_discard(struct aspeed_lpc_snoop_channel *chan, u8 val)
{
- if (!kfifo_initialized(&chan->fifo))
- return;
- if (kfifo_is_full(&chan->fifo))
- kfifo_skip(&chan->fifo);
- kfifo_put(&chan->fifo, val);
+ scoped_guard(spinlock, &chan->lock) {
+ if (!kfifo_initialized(&chan->fifo))
+ return;
+ if (kfifo_is_full(&chan->fifo))
+ kfifo_skip(&chan->fifo);
+ kfifo_put(&chan->fifo, val);
+ }
wake_up_interruptible(&chan->wq);
}
@@ -292,6 +304,9 @@ static int aspeed_lpc_snoop_probe(struct platform_device *pdev)
if (!lpc_snoop)
return -ENOMEM;
+ spin_lock_init(&lpc_snoop->chan[0].lock);
+ spin_lock_init(&lpc_snoop->chan[1].lock);
+
np = pdev->dev.parent->of_node;
if (!of_device_is_compatible(np, "aspeed,ast2400-lpc-v2") &&
!of_device_is_compatible(np, "aspeed,ast2500-lpc-v2") &&
diff --git a/drivers/soc/bcm/brcmstb/biuctrl.c b/drivers/soc/bcm/brcmstb/biuctrl.c
index 364ddbe365c2..bd830649b60d 100644
--- a/drivers/soc/bcm/brcmstb/biuctrl.c
+++ b/drivers/soc/bcm/brcmstb/biuctrl.c
@@ -298,7 +298,7 @@ static int __init setup_hifcpubiuctrl_regs(struct device_node *np)
#ifdef CONFIG_PM_SLEEP
static u32 cpubiuctrl_reg_save[NUM_CPU_BIUCTRL_REGS];
-static int brcmstb_cpu_credit_reg_suspend(void)
+static int brcmstb_cpu_credit_reg_suspend(void *data)
{
unsigned int i;
@@ -311,7 +311,7 @@ static int brcmstb_cpu_credit_reg_suspend(void)
return 0;
}
-static void brcmstb_cpu_credit_reg_resume(void)
+static void brcmstb_cpu_credit_reg_resume(void *data)
{
unsigned int i;
@@ -322,10 +322,14 @@ static void brcmstb_cpu_credit_reg_resume(void)
cbc_writel(cpubiuctrl_reg_save[i], i);
}
-static struct syscore_ops brcmstb_cpu_credit_syscore_ops = {
+static const struct syscore_ops brcmstb_cpu_credit_syscore_ops = {
.suspend = brcmstb_cpu_credit_reg_suspend,
.resume = brcmstb_cpu_credit_reg_resume,
};
+
+static struct syscore brcmstb_cpu_credit_syscore = {
+ .ops = &brcmstb_cpu_credit_syscore_ops,
+};
#endif
@@ -354,7 +358,7 @@ static int __init brcmstb_biuctrl_init(void)
a72_b53_rac_enable_all(np);
mcp_a72_b53_set();
#ifdef CONFIG_PM_SLEEP
- register_syscore_ops(&brcmstb_cpu_credit_syscore_ops);
+ register_syscore(&brcmstb_cpu_credit_syscore);
#endif
ret = 0;
out_put:
diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
index 0fdccd736209..806167724b26 100644
--- a/drivers/soc/tegra/pmc.c
+++ b/drivers/soc/tegra/pmc.c
@@ -28,6 +28,7 @@
#include <linux/iopoll.h>
#include <linux/irqdomain.h>
#include <linux/irq.h>
+#include <linux/irq_work.h>
#include <linux/kernel.h>
#include <linux/of_address.h>
#include <linux/of_clk.h>
@@ -423,6 +424,9 @@ struct tegra_pmc_soc {
* @wake_sw_status_map: Bitmap to hold raw status of wakes without mask
* @wake_cntrl_level_map: Bitmap to hold wake levels to be programmed in
* cntrl register associated with each wake during system suspend.
+ * @syscore: syscore suspend/resume callbacks
+ * @wake_work: IRQ work handler for processing wake-up events.
+ * @wake_status: Status of wake-up events.
*/
struct tegra_pmc {
struct device *dev;
@@ -467,7 +471,11 @@ struct tegra_pmc {
unsigned long *wake_type_dual_edge_map;
unsigned long *wake_sw_status_map;
unsigned long *wake_cntrl_level_map;
- struct syscore_ops syscore;
+ struct syscore syscore;
+
+ /* Pending wake IRQ processing */
+ struct irq_work wake_work;
+ u32 *wake_status;
};
static struct tegra_pmc *pmc = &(struct tegra_pmc) {
@@ -1905,6 +1913,50 @@ static int tegra_pmc_parse_dt(struct tegra_pmc *pmc, struct device_node *np)
return 0;
}
+/* translate sc7 wake sources back into IRQs to catch edge triggered wakeups */
+static void tegra186_pmc_wake_handler(struct irq_work *work)
+{
+ struct tegra_pmc *pmc = container_of(work, struct tegra_pmc, wake_work);
+ unsigned int i, wake;
+
+ for (i = 0; i < pmc->soc->max_wake_vectors; i++) {
+ unsigned long status = pmc->wake_status[i];
+
+ for_each_set_bit(wake, &status, 32) {
+ irq_hw_number_t hwirq = wake + (i * 32);
+ struct irq_desc *desc;
+ unsigned int irq;
+
+ irq = irq_find_mapping(pmc->domain, hwirq);
+ if (!irq) {
+ dev_warn(pmc->dev,
+ "No IRQ found for WAKE#%lu!\n",
+ hwirq);
+ continue;
+ }
+
+ dev_dbg(pmc->dev,
+ "Resume caused by WAKE#%lu mapped to IRQ#%u\n",
+ hwirq, irq);
+
+ desc = irq_to_desc(irq);
+ if (!desc) {
+ dev_warn(pmc->dev,
+ "No descriptor found for IRQ#%u\n",
+ irq);
+ continue;
+ }
+
+ if (!desc->action || !desc->action->name)
+ continue;
+
+ generic_handle_irq(irq);
+ }
+
+ pmc->wake_status[i] = 0;
+ }
+}
+
static int tegra_pmc_init(struct tegra_pmc *pmc)
{
if (pmc->soc->max_wake_events > 0) {
@@ -1923,6 +1975,18 @@ static int tegra_pmc_init(struct tegra_pmc *pmc)
pmc->wake_cntrl_level_map = bitmap_zalloc(pmc->soc->max_wake_events, GFP_KERNEL);
if (!pmc->wake_cntrl_level_map)
return -ENOMEM;
+
+ pmc->wake_status = kcalloc(pmc->soc->max_wake_vectors, sizeof(u32), GFP_KERNEL);
+ if (!pmc->wake_status)
+ return -ENOMEM;
+
+ /*
+ * Initialize IRQ work for processing wake IRQs. Must use
+ * HARD_IRQ variant to run in hard IRQ context on PREEMPT_RT
+ * because we call generic_handle_irq() which requires hard
+ * IRQ context.
+ */
+ pmc->wake_work = IRQ_WORK_INIT_HARD(tegra186_pmc_wake_handler);
}
if (pmc->soc->init)
@@ -3122,47 +3186,30 @@ static void wke_clear_wake_status(struct tegra_pmc *pmc)
}
}
-/* translate sc7 wake sources back into IRQs to catch edge triggered wakeups */
-static void tegra186_pmc_process_wake_events(struct tegra_pmc *pmc, unsigned int index,
- unsigned long status)
+static void tegra186_pmc_wake_syscore_resume(void *data)
{
- unsigned int wake;
-
- dev_dbg(pmc->dev, "Wake[%d:%d] status=%#lx\n", (index * 32) + 31, index * 32, status);
-
- for_each_set_bit(wake, &status, 32) {
- irq_hw_number_t hwirq = wake + 32 * index;
- struct irq_desc *desc;
- unsigned int irq;
-
- irq = irq_find_mapping(pmc->domain, hwirq);
-
- desc = irq_to_desc(irq);
- if (!desc || !desc->action || !desc->action->name) {
- dev_dbg(pmc->dev, "Resume caused by WAKE%ld, IRQ %d\n", hwirq, irq);
- continue;
- }
-
- dev_dbg(pmc->dev, "Resume caused by WAKE%ld, %s\n", hwirq, desc->action->name);
- generic_handle_irq(irq);
- }
-}
-
-static void tegra186_pmc_wake_syscore_resume(void)
-{
- u32 status, mask;
unsigned int i;
+ u32 mask;
for (i = 0; i < pmc->soc->max_wake_vectors; i++) {
mask = readl(pmc->wake + WAKE_AOWAKE_TIER2_ROUTING(i));
- status = readl(pmc->wake + WAKE_AOWAKE_STATUS_R(i)) & mask;
-
- tegra186_pmc_process_wake_events(pmc, i, status);
+ pmc->wake_status[i] = readl(pmc->wake + WAKE_AOWAKE_STATUS_R(i)) & mask;
}
+
+ /* Schedule IRQ work to process wake IRQs (if any) */
+ irq_work_queue(&pmc->wake_work);
}
-static int tegra186_pmc_wake_syscore_suspend(void)
+static int tegra186_pmc_wake_syscore_suspend(void *data)
{
+ unsigned int i;
+
+ /* Check if there are unhandled wake IRQs */
+ for (i = 0; i < pmc->soc->max_wake_vectors; i++)
+ if (pmc->wake_status[i])
+ dev_warn(pmc->dev,
+ "Unhandled wake IRQs pending vector[%u]: 0x%x\n",
+ i, pmc->wake_status[i]);
wke_read_sw_wake_status(pmc);
/* flip the wakeup trigger for dual-edge triggered pads
@@ -3180,6 +3227,11 @@ static int tegra186_pmc_wake_syscore_suspend(void)
return 0;
}
+static const struct syscore_ops tegra186_pmc_wake_syscore_ops = {
+ .suspend = tegra186_pmc_wake_syscore_suspend,
+ .resume = tegra186_pmc_wake_syscore_resume,
+};
+
#if defined(CONFIG_PM_SLEEP) && defined(CONFIG_ARM)
static int tegra_pmc_suspend(struct device *dev)
{
@@ -3830,10 +3882,8 @@ static const struct tegra_pmc_regs tegra186_pmc_regs = {
static void tegra186_pmc_init(struct tegra_pmc *pmc)
{
- pmc->syscore.suspend = tegra186_pmc_wake_syscore_suspend;
- pmc->syscore.resume = tegra186_pmc_wake_syscore_resume;
-
- register_syscore_ops(&pmc->syscore);
+ pmc->syscore.ops = &tegra186_pmc_wake_syscore_ops;
+ register_syscore(&pmc->syscore);
}
static void tegra186_pmc_setup_irq_polarity(struct tegra_pmc *pmc,
diff --git a/drivers/soc/xilinx/zynqmp_power.c b/drivers/soc/xilinx/zynqmp_power.c
index 84ef7c87dfaa..92f1ab1d269b 100644
--- a/drivers/soc/xilinx/zynqmp_power.c
+++ b/drivers/soc/xilinx/zynqmp_power.c
@@ -303,18 +303,18 @@ static int zynqmp_pm_probe(struct platform_device *pdev)
* is not available to use) or -ENODEV(Xilinx Event Manager not compiled),
* then use ipi-mailbox or interrupt method.
*/
+ zynqmp_pm_init_suspend_work = devm_kzalloc(&pdev->dev,
+ sizeof(struct zynqmp_pm_work_struct),
+ GFP_KERNEL);
+ if (!zynqmp_pm_init_suspend_work)
+ return -ENOMEM;
+
+ INIT_WORK(&zynqmp_pm_init_suspend_work->callback_work,
+ zynqmp_pm_init_suspend_work_fn);
+
ret = register_event(&pdev->dev, PM_INIT_SUSPEND_CB, 0, 0, false,
suspend_event_callback);
if (!ret) {
- zynqmp_pm_init_suspend_work = devm_kzalloc(&pdev->dev,
- sizeof(struct zynqmp_pm_work_struct),
- GFP_KERNEL);
- if (!zynqmp_pm_init_suspend_work)
- return -ENOMEM;
-
- INIT_WORK(&zynqmp_pm_init_suspend_work->callback_work,
- zynqmp_pm_init_suspend_work_fn);
-
ret = zynqmp_pm_get_family_info(&pm_family_code, &pm_sub_family_code);
if (ret < 0)
return ret;
@@ -324,14 +324,6 @@ static int zynqmp_pm_probe(struct platform_device *pdev)
else
node_id = PM_DEV_ACPU_0;
- ret = register_event(&pdev->dev, PM_NOTIFY_CB, node_id, EVENT_SUBSYSTEM_RESTART,
- false, subsystem_restart_event_callback);
- if (ret) {
- dev_err(&pdev->dev, "Failed to Register with Xilinx Event manager %d\n",
- ret);
- return ret;
- }
-
zynqmp_pm_init_restart_work = devm_kzalloc(&pdev->dev,
sizeof(struct zynqmp_pm_work_struct),
GFP_KERNEL);
@@ -340,19 +332,18 @@ static int zynqmp_pm_probe(struct platform_device *pdev)
INIT_WORK(&zynqmp_pm_init_restart_work->callback_work,
zynqmp_pm_subsystem_restart_work_fn);
+
+ ret = register_event(&pdev->dev, PM_NOTIFY_CB, node_id, EVENT_SUBSYSTEM_RESTART,
+ false, subsystem_restart_event_callback);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to Register with Xilinx Event manager %d\n",
+ ret);
+ return ret;
+ }
} else if (ret != -EACCES && ret != -ENODEV) {
dev_err(&pdev->dev, "Failed to Register with Xilinx Event manager %d\n", ret);
return ret;
} else if (of_property_present(pdev->dev.of_node, "mboxes")) {
- zynqmp_pm_init_suspend_work =
- devm_kzalloc(&pdev->dev,
- sizeof(struct zynqmp_pm_work_struct),
- GFP_KERNEL);
- if (!zynqmp_pm_init_suspend_work)
- return -ENOMEM;
-
- INIT_WORK(&zynqmp_pm_init_suspend_work->callback_work,
- zynqmp_pm_init_suspend_work_fn);
client = devm_kzalloc(&pdev->dev, sizeof(*client), GFP_KERNEL);
if (!client)
return -ENOMEM;
diff --git a/drivers/soundwire/generic_bandwidth_allocation.c b/drivers/soundwire/generic_bandwidth_allocation.c
index b9316207c3ab..45a2f80ed809 100644
--- a/drivers/soundwire/generic_bandwidth_allocation.c
+++ b/drivers/soundwire/generic_bandwidth_allocation.c
@@ -199,28 +199,25 @@ static int sdw_add_element_group_count(struct sdw_group *group,
int num = group->count;
int i;
- for (i = 0; i <= num; i++) {
+ for (i = 0; i < num; i++) {
if (rate == group->rates[i])
- break;
-
- if (i != num)
- continue;
-
- if (group->count >= group->max_size) {
- unsigned int *rates;
+ return 0;
+ }
- group->max_size += 1;
- rates = krealloc(group->rates,
- (sizeof(int) * group->max_size),
- GFP_KERNEL);
- if (!rates)
- return -ENOMEM;
- group->rates = rates;
- }
+ if (group->count >= group->max_size) {
+ unsigned int *rates;
- group->rates[group->count++] = rate;
+ group->max_size += 1;
+ rates = krealloc(group->rates,
+ (sizeof(int) * group->max_size),
+ GFP_KERNEL);
+ if (!rates)
+ return -ENOMEM;
+ group->rates = rates;
}
+ group->rates[group->count++] = rate;
+
return 0;
}
diff --git a/drivers/soundwire/intel_auxdevice.c b/drivers/soundwire/intel_auxdevice.c
index 078f9eaf5161..ae1dfce163e2 100644
--- a/drivers/soundwire/intel_auxdevice.c
+++ b/drivers/soundwire/intel_auxdevice.c
@@ -345,9 +345,6 @@ static int intel_link_probe(struct auxiliary_device *auxdev,
/* use generic bandwidth allocation algorithm */
sdw->cdns.bus.compute_params = sdw_compute_params;
- /* avoid resuming from pm_runtime suspend if it's not required */
- dev_pm_set_driver_flags(dev, DPM_FLAG_SMART_SUSPEND);
-
ret = sdw_bus_master_add(bus, dev, dev->fwnode);
if (ret) {
dev_err(dev, "sdw_bus_master_add fail: %d\n", ret);
@@ -632,7 +629,10 @@ static int __maybe_unused intel_suspend(struct device *dev)
return 0;
}
- if (pm_runtime_suspended(dev)) {
+ /* Prevent runtime PM from racing with the code below. */
+ pm_runtime_disable(dev);
+
+ if (pm_runtime_status_suspended(dev)) {
dev_dbg(dev, "pm_runtime status: suspended\n");
clock_stop_quirks = sdw->link_res->clock_stop_quirks;
@@ -640,7 +640,7 @@ static int __maybe_unused intel_suspend(struct device *dev)
if ((clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET) ||
!clock_stop_quirks) {
- if (pm_runtime_suspended(dev->parent)) {
+ if (pm_runtime_status_suspended(dev->parent)) {
/*
* paranoia check: this should not happen with the .prepare
* resume to full power
@@ -707,7 +707,6 @@ static int __maybe_unused intel_resume(struct device *dev)
struct sdw_cdns *cdns = dev_get_drvdata(dev);
struct sdw_intel *sdw = cdns_to_intel(cdns);
struct sdw_bus *bus = &cdns->bus;
- int link_flags;
int ret;
if (bus->prop.hw_disabled || !sdw->startup_done) {
@@ -716,23 +715,6 @@ static int __maybe_unused intel_resume(struct device *dev)
return 0;
}
- if (pm_runtime_suspended(dev)) {
- dev_dbg(dev, "pm_runtime status was suspended, forcing active\n");
-
- /* follow required sequence from runtime_pm.rst */
- pm_runtime_disable(dev);
- pm_runtime_set_active(dev);
- pm_runtime_mark_last_busy(dev);
- pm_runtime_enable(dev);
-
- pm_runtime_resume(bus->dev);
-
- link_flags = md_flags >> (bus->link_id * 8);
-
- if (!(link_flags & SDW_INTEL_MASTER_DISABLE_PM_RUNTIME_IDLE))
- pm_runtime_idle(dev);
- }
-
ret = sdw_intel_link_power_up(sdw);
if (ret) {
dev_err(dev, "%s failed: %d\n", __func__, ret);
@@ -752,6 +734,14 @@ static int __maybe_unused intel_resume(struct device *dev)
return ret;
}
+ /*
+ * Runtime PM has been disabled in intel_suspend(), so set the status
+ * to active because the device has just been resumed and re-enable
+ * runtime PM.
+ */
+ pm_runtime_set_active(dev);
+ pm_runtime_enable(dev);
+
/*
* after system resume, the pm_runtime suspend() may kick in
* during the enumeration, before any children device force the
diff --git a/drivers/spi/spi-ch341.c b/drivers/spi/spi-ch341.c
index 850b428b8f64..3eaa8f176f63 100644
--- a/drivers/spi/spi-ch341.c
+++ b/drivers/spi/spi-ch341.c
@@ -152,7 +152,7 @@ static int ch341_probe(struct usb_interface *intf,
if (ret)
return ret;
- ctrl = devm_spi_alloc_master(&udev->dev, sizeof(struct ch341_spi_dev));
+ ctrl = devm_spi_alloc_host(&intf->dev, sizeof(struct ch341_spi_dev));
if (!ctrl)
return -ENOMEM;
@@ -163,7 +163,7 @@ static int ch341_probe(struct usb_interface *intf,
ch341->read_pipe = usb_rcvbulkpipe(udev, usb_endpoint_num(in));
ch341->rx_len = usb_endpoint_maxp(in);
- ch341->rx_buf = devm_kzalloc(&udev->dev, ch341->rx_len, GFP_KERNEL);
+ ch341->rx_buf = devm_kzalloc(&intf->dev, ch341->rx_len, GFP_KERNEL);
if (!ch341->rx_buf)
return -ENOMEM;
@@ -171,8 +171,7 @@ static int ch341_probe(struct usb_interface *intf,
if (!ch341->rx_urb)
return -ENOMEM;
- ch341->tx_buf =
- devm_kzalloc(&udev->dev, CH341_PACKET_LENGTH, GFP_KERNEL);
+ ch341->tx_buf = devm_kzalloc(&intf->dev, CH341_PACKET_LENGTH, GFP_KERNEL);
if (!ch341->tx_buf) {
ret = -ENOMEM;
goto err_free_urb;
diff --git a/drivers/spi/spi-microchip-core-qspi.c b/drivers/spi/spi-microchip-core-qspi.c
index 6aba933fcc0b..2b5271fa870b 100644
--- a/drivers/spi/spi-microchip-core-qspi.c
+++ b/drivers/spi/spi-microchip-core-qspi.c
@@ -74,6 +74,13 @@
#define STATUS_FLAGSX4 BIT(8)
#define STATUS_MASK GENMASK(8, 0)
+/*
+ * QSPI Direct Access register defines
+ */
+#define DIRECT_ACCESS_EN_SSEL BIT(0)
+#define DIRECT_ACCESS_OP_SSEL BIT(1)
+#define DIRECT_ACCESS_OP_SSEL_SHIFT 1
+
#define BYTESUPPER_MASK GENMASK(31, 16)
#define BYTESLOWER_MASK GENMASK(15, 0)
@@ -158,6 +165,38 @@ static int mchp_coreqspi_set_mode(struct mchp_coreqspi *qspi, const struct spi_m
return 0;
}
+static void mchp_coreqspi_set_cs(struct spi_device *spi, bool enable)
+{
+ struct mchp_coreqspi *qspi = spi_controller_get_devdata(spi->controller);
+ u32 val;
+
+ val = readl(qspi->regs + REG_DIRECT_ACCESS);
+
+ val &= ~DIRECT_ACCESS_OP_SSEL;
+ val |= !enable << DIRECT_ACCESS_OP_SSEL_SHIFT;
+
+ writel(val, qspi->regs + REG_DIRECT_ACCESS);
+}
+
+static int mchp_coreqspi_setup(struct spi_device *spi)
+{
+ struct mchp_coreqspi *qspi = spi_controller_get_devdata(spi->controller);
+ u32 val;
+
+ /*
+ * Active low devices need to be specifically set to their inactive
+ * states during probe.
+ */
+ if (spi->mode & SPI_CS_HIGH)
+ return 0;
+
+ val = readl(qspi->regs + REG_DIRECT_ACCESS);
+ val |= DIRECT_ACCESS_OP_SSEL;
+ writel(val, qspi->regs + REG_DIRECT_ACCESS);
+
+ return 0;
+}
+
static inline void mchp_coreqspi_read_op(struct mchp_coreqspi *qspi)
{
u32 control, data;
@@ -194,7 +233,7 @@ static inline void mchp_coreqspi_read_op(struct mchp_coreqspi *qspi)
}
}
-static inline void mchp_coreqspi_write_op(struct mchp_coreqspi *qspi, bool word)
+static inline void mchp_coreqspi_write_op(struct mchp_coreqspi *qspi)
{
u32 control, data;
@@ -265,7 +304,8 @@ static irqreturn_t mchp_coreqspi_isr(int irq, void *dev_id)
return ret;
}
-static int mchp_coreqspi_setup_clock(struct mchp_coreqspi *qspi, struct spi_device *spi)
+static int mchp_coreqspi_setup_clock(struct mchp_coreqspi *qspi, struct spi_device *spi,
+ const struct spi_mem_op *op)
{
unsigned long clk_hz;
u32 control, baud_rate_val = 0;
@@ -274,11 +314,11 @@ static int mchp_coreqspi_setup_clock(struct mchp_coreqspi *qspi, struct spi_devi
if (!clk_hz)
return -EINVAL;
- baud_rate_val = DIV_ROUND_UP(clk_hz, 2 * spi->max_speed_hz);
+ baud_rate_val = DIV_ROUND_UP(clk_hz, 2 * op->max_freq);
if (baud_rate_val > MAX_DIVIDER || baud_rate_val < MIN_DIVIDER) {
dev_err(&spi->dev,
"could not configure the clock for spi clock %d Hz & system clock %ld Hz\n",
- spi->max_speed_hz, clk_hz);
+ op->max_freq, clk_hz);
return -EINVAL;
}
@@ -298,19 +338,6 @@ static int mchp_coreqspi_setup_clock(struct mchp_coreqspi *qspi, struct spi_devi
return 0;
}
-static int mchp_coreqspi_setup_op(struct spi_device *spi_dev)
-{
- struct spi_controller *ctlr = spi_dev->controller;
- struct mchp_coreqspi *qspi = spi_controller_get_devdata(ctlr);
- u32 control = readl_relaxed(qspi->regs + REG_CONTROL);
-
- control |= (CONTROL_MASTER | CONTROL_ENABLE);
- control &= ~CONTROL_CLKIDLE;
- writel_relaxed(control, qspi->regs + REG_CONTROL);
-
- return 0;
-}
-
static inline void mchp_coreqspi_config_op(struct mchp_coreqspi *qspi, const struct spi_mem_op *op)
{
u32 idle_cycles = 0;
@@ -399,7 +426,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o
if (err)
goto error;
- err = mchp_coreqspi_setup_clock(qspi, mem->spi);
+ err = mchp_coreqspi_setup_clock(qspi, mem->spi, op);
if (err)
goto error;
@@ -409,12 +436,13 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o
reinit_completion(&qspi->data_completion);
mchp_coreqspi_config_op(qspi, op);
+ mchp_coreqspi_set_cs(mem->spi, true);
if (op->cmd.opcode) {
qspi->txbuf = &opcode;
qspi->rxbuf = NULL;
qspi->tx_len = op->cmd.nbytes;
qspi->rx_len = 0;
- mchp_coreqspi_write_op(qspi, false);
+ mchp_coreqspi_write_op(qspi);
}
qspi->txbuf = &opaddr[0];
@@ -425,7 +453,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o
qspi->rxbuf = NULL;
qspi->tx_len = op->addr.nbytes;
qspi->rx_len = 0;
- mchp_coreqspi_write_op(qspi, false);
+ mchp_coreqspi_write_op(qspi);
}
if (op->data.nbytes) {
@@ -434,7 +462,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o
qspi->rxbuf = NULL;
qspi->rx_len = 0;
qspi->tx_len = op->data.nbytes;
- mchp_coreqspi_write_op(qspi, true);
+ mchp_coreqspi_write_op(qspi);
} else {
qspi->txbuf = NULL;
qspi->rxbuf = (u8 *)op->data.buf.in;
@@ -449,6 +477,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o
err = -ETIMEDOUT;
error:
+ mchp_coreqspi_set_cs(mem->spi, false);
mutex_unlock(&qspi->op_lock);
mchp_coreqspi_disable_ints(qspi);
@@ -498,6 +527,10 @@ static const struct spi_controller_mem_ops mchp_coreqspi_mem_ops = {
.exec_op = mchp_coreqspi_exec_op,
};
+static const struct spi_controller_mem_caps mchp_coreqspi_mem_caps = {
+ .per_op_freq = true,
+};
+
static int mchp_coreqspi_probe(struct platform_device *pdev)
{
struct spi_controller *ctlr;
@@ -505,6 +538,7 @@ static int mchp_coreqspi_probe(struct platform_device *pdev)
struct device *dev = &pdev->dev;
struct device_node *np = dev->of_node;
int ret;
+ u32 num_cs, val;
ctlr = devm_spi_alloc_host(&pdev->dev, sizeof(*qspi));
if (!ctlr)
@@ -538,12 +572,34 @@ static int mchp_coreqspi_probe(struct platform_device *pdev)
return ret;
}
+ /*
+ * The IP core only has a single CS, any more have to be provided via
+ * gpios
+ */
+ if (of_property_read_u32(pdev->dev.of_node, "num-cs", &num_cs))
+ num_cs = 1;
+
+ ctlr->num_chipselect = num_cs;
+
ctlr->bits_per_word_mask = SPI_BPW_MASK(8);
ctlr->mem_ops = &mchp_coreqspi_mem_ops;
- ctlr->setup = mchp_coreqspi_setup_op;
+ ctlr->mem_caps = &mchp_coreqspi_mem_caps;
ctlr->mode_bits = SPI_CPOL | SPI_CPHA | SPI_RX_DUAL | SPI_RX_QUAD |
SPI_TX_DUAL | SPI_TX_QUAD;
ctlr->dev.of_node = np;
+ ctlr->min_speed_hz = clk_get_rate(qspi->clk) / 30;
+ ctlr->setup = mchp_coreqspi_setup;
+
+ val = readl_relaxed(qspi->regs + REG_CONTROL);
+ val |= (CONTROL_MASTER | CONTROL_ENABLE);
+ writel_relaxed(val, qspi->regs + REG_CONTROL);
+
+ /*
+ * Put cs into software controlled mode
+ */
+ val = readl_relaxed(qspi->regs + REG_DIRECT_ACCESS);
+ val |= DIRECT_ACCESS_EN_SSEL;
+ writel(val, qspi->regs + REG_DIRECT_ACCESS);
ret = spi_register_controller(ctlr);
if (ret)
diff --git a/drivers/spi/spi-mpc52xx.c b/drivers/spi/spi-mpc52xx.c
index a4991f3bc9b6..a067e73d9d4d 100644
--- a/drivers/spi/spi-mpc52xx.c
+++ b/drivers/spi/spi-mpc52xx.c
@@ -475,13 +475,15 @@ static int mpc52xx_spi_probe(struct platform_device *op)
if (ms->irq0 && ms->irq1) {
rc = request_irq(ms->irq0, mpc52xx_spi_irq, 0,
"mpc5200-spi-modf", ms);
- rc |= request_irq(ms->irq1, mpc52xx_spi_irq, 0,
- "mpc5200-spi-spif", ms);
- if (rc) {
- free_irq(ms->irq0, ms);
- free_irq(ms->irq1, ms);
- ms->irq0 = ms->irq1 = 0;
+ if (rc == 0) {
+ rc = request_irq(ms->irq1, mpc52xx_spi_irq, 0,
+ "mpc5200-spi-spif", ms);
+ if (rc)
+ free_irq(ms->irq0, ms);
}
+
+ if (rc)
+ ms->irq0 = ms->irq1 = 0;
} else {
/* operate in polled mode */
ms->irq0 = ms->irq1 = 0;
@@ -501,8 +503,10 @@ static int mpc52xx_spi_probe(struct platform_device *op)
err_register:
dev_err(&ms->host->dev, "initialization failed\n");
- free_irq(ms->irq0, ms);
- free_irq(ms->irq1, ms);
+ if (ms->irq0) {
+ free_irq(ms->irq0, ms);
+ free_irq(ms->irq1, ms);
+ }
cancel_work_sync(&ms->work);
err_gpio:
while (i-- > 0)
@@ -525,8 +529,10 @@ static void mpc52xx_spi_remove(struct platform_device *op)
spi_unregister_controller(host);
- free_irq(ms->irq0, ms);
- free_irq(ms->irq1, ms);
+ if (ms->irq0) {
+ free_irq(ms->irq0, ms);
+ free_irq(ms->irq1, ms);
+ }
cancel_work_sync(&ms->work);
diff --git a/drivers/staging/greybus/raw.c b/drivers/staging/greybus/raw.c
index 71de6776739c..d0fd15af22ba 100644
--- a/drivers/staging/greybus/raw.c
+++ b/drivers/staging/greybus/raw.c
@@ -21,9 +21,10 @@ struct gb_raw {
struct list_head list;
int list_data;
struct mutex list_lock;
- dev_t dev;
+ struct rw_semaphore disconnect_lock;
+ bool disconnected;
struct cdev cdev;
- struct device *device;
+ struct device dev;
};
struct raw_data {
@@ -148,6 +149,15 @@ static int gb_raw_send(struct gb_raw *raw, u32 len, const char __user *data)
return retval;
}
+static void raw_dev_release(struct device *dev)
+{
+ struct gb_raw *raw = container_of(dev, struct gb_raw, dev);
+
+ ida_free(&minors, MINOR(raw->dev.devt));
+
+ kfree(raw);
+}
+
static int gb_raw_probe(struct gb_bundle *bundle,
const struct greybus_bundle_id *id)
{
@@ -164,63 +174,59 @@ static int gb_raw_probe(struct gb_bundle *bundle,
if (cport_desc->protocol_id != GREYBUS_PROTOCOL_RAW)
return -ENODEV;
+ minor = ida_alloc(&minors, GFP_KERNEL);
+ if (minor < 0)
+ return minor;
+
raw = kzalloc(sizeof(*raw), GFP_KERNEL);
- if (!raw)
+ if (!raw) {
+ ida_free(&minors, minor);
return -ENOMEM;
+ }
+
+ device_initialize(&raw->dev);
+ raw->dev.devt = MKDEV(raw_major, minor);
+ raw->dev.class = &raw_class;
+ raw->dev.parent = &bundle->dev;
+ raw->dev.release = raw_dev_release;
+ retval = dev_set_name(&raw->dev, "gb!raw%d", minor);
+ if (retval)
+ goto error_put_device;
connection = gb_connection_create(bundle, le16_to_cpu(cport_desc->id),
gb_raw_request_handler);
if (IS_ERR(connection)) {
retval = PTR_ERR(connection);
- goto error_free;
+ goto error_put_device;
}
INIT_LIST_HEAD(&raw->list);
mutex_init(&raw->list_lock);
+ init_rwsem(&raw->disconnect_lock);
raw->connection = connection;
greybus_set_drvdata(bundle, raw);
- minor = ida_alloc(&minors, GFP_KERNEL);
- if (minor < 0) {
- retval = minor;
- goto error_connection_destroy;
- }
-
- raw->dev = MKDEV(raw_major, minor);
cdev_init(&raw->cdev, &raw_fops);
retval = gb_connection_enable(connection);
if (retval)
- goto error_remove_ida;
+ goto error_connection_destroy;
- retval = cdev_add(&raw->cdev, raw->dev, 1);
+ retval = cdev_device_add(&raw->cdev, &raw->dev);
if (retval)
goto error_connection_disable;
- raw->device = device_create(&raw_class, &connection->bundle->dev,
- raw->dev, raw, "gb!raw%d", minor);
- if (IS_ERR(raw->device)) {
- retval = PTR_ERR(raw->device);
- goto error_del_cdev;
- }
-
return 0;
-error_del_cdev:
- cdev_del(&raw->cdev);
-
error_connection_disable:
gb_connection_disable(connection);
-error_remove_ida:
- ida_free(&minors, minor);
-
error_connection_destroy:
gb_connection_destroy(connection);
-error_free:
- kfree(raw);
+error_put_device:
+ put_device(&raw->dev);
return retval;
}
@@ -231,11 +237,13 @@ static void gb_raw_disconnect(struct gb_bundle *bundle)
struct raw_data *raw_data;
struct raw_data *temp;
- // FIXME - handle removing a connection when the char device node is open.
- device_destroy(&raw_class, raw->dev);
- cdev_del(&raw->cdev);
+ cdev_device_del(&raw->cdev, &raw->dev);
+
+ down_write(&raw->disconnect_lock);
+ raw->disconnected = true;
+ up_write(&raw->disconnect_lock);
+
gb_connection_disable(connection);
- ida_free(&minors, MINOR(raw->dev));
gb_connection_destroy(connection);
mutex_lock(&raw->list_lock);
@@ -244,8 +252,7 @@ static void gb_raw_disconnect(struct gb_bundle *bundle)
kfree(raw_data);
}
mutex_unlock(&raw->list_lock);
-
- kfree(raw);
+ put_device(&raw->dev);
}
/*
@@ -278,11 +285,22 @@ static ssize_t raw_write(struct file *file, const char __user *buf,
if (count > MAX_PACKET_SIZE)
return -E2BIG;
+ down_read(&raw->disconnect_lock);
+
+ if (raw->disconnected) {
+ retval = -ENODEV;
+ goto exit;
+ }
+
retval = gb_raw_send(raw, count, buf);
if (retval)
- return retval;
+ goto exit;
+
+ retval = count;
+exit:
+ up_read(&raw->disconnect_lock);
- return count;
+ return retval;
}
static ssize_t raw_read(struct file *file, char __user *buf, size_t count,
diff --git a/drivers/staging/most/Kconfig b/drivers/staging/most/Kconfig
index 6f420cbcdcff..e89658df6f12 100644
--- a/drivers/staging/most/Kconfig
+++ b/drivers/staging/most/Kconfig
@@ -24,6 +24,4 @@ source "drivers/staging/most/video/Kconfig"
source "drivers/staging/most/dim2/Kconfig"
-source "drivers/staging/most/i2c/Kconfig"
-
endif
diff --git a/drivers/staging/most/Makefile b/drivers/staging/most/Makefile
index 8b3fc5a7af51..e45084df7803 100644
--- a/drivers/staging/most/Makefile
+++ b/drivers/staging/most/Makefile
@@ -3,4 +3,3 @@
obj-$(CONFIG_MOST_NET) += net/
obj-$(CONFIG_MOST_VIDEO) += video/
obj-$(CONFIG_MOST_DIM2) += dim2/
-obj-$(CONFIG_MOST_I2C) += i2c/
diff --git a/drivers/staging/most/i2c/Kconfig b/drivers/staging/most/i2c/Kconfig
deleted file mode 100644
index ff64283cbad1..000000000000
--- a/drivers/staging/most/i2c/Kconfig
+++ /dev/null
@@ -1,13 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-#
-# MOST I2C configuration
-#
-
-config MOST_I2C
- tristate "I2C"
- depends on I2C
- help
- Say Y here if you want to connect via I2C to network transceiver.
-
- To compile this driver as a module, choose M here: the
- module will be called most_i2c.
diff --git a/drivers/staging/most/i2c/Makefile b/drivers/staging/most/i2c/Makefile
deleted file mode 100644
index 71099dd0f85b..000000000000
--- a/drivers/staging/most/i2c/Makefile
+++ /dev/null
@@ -1,4 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_MOST_I2C) += most_i2c.o
-
-most_i2c-objs := i2c.o
diff --git a/drivers/staging/most/i2c/i2c.c b/drivers/staging/most/i2c/i2c.c
deleted file mode 100644
index ce869280a056..000000000000
--- a/drivers/staging/most/i2c/i2c.c
+++ /dev/null
@@ -1,374 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * i2c.c - Hardware Dependent Module for I2C Interface
- *
- * Copyright (C) 2013-2015, Microchip Technology Germany II GmbH & Co. KG
- */
-
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/slab.h>
-#include <linux/i2c.h>
-#include <linux/interrupt.h>
-#include <linux/err.h>
-#include <linux/most.h>
-
-enum { CH_RX, CH_TX, NUM_CHANNELS };
-
-#define MAX_BUFFERS_CONTROL 32
-#define MAX_BUF_SIZE_CONTROL 256
-
-/**
- * list_first_mbo - get the first mbo from a list
- * @ptr: the list head to take the mbo from.
- */
-#define list_first_mbo(ptr) \
- list_first_entry(ptr, struct mbo, list)
-
-static unsigned int polling_rate;
-module_param(polling_rate, uint, 0644);
-MODULE_PARM_DESC(polling_rate, "Polling rate [Hz]. Default = 0 (use IRQ)");
-
-struct hdm_i2c {
- struct most_interface most_iface;
- struct most_channel_capability capabilities[NUM_CHANNELS];
- struct i2c_client *client;
- struct rx {
- struct delayed_work dwork;
- struct list_head list;
- bool int_disabled;
- unsigned int delay;
- } rx;
- char name[64];
-};
-
-static inline struct hdm_i2c *to_hdm(struct most_interface *iface)
-{
- return container_of(iface, struct hdm_i2c, most_iface);
-}
-
-static irqreturn_t most_irq_handler(int, void *);
-static void pending_rx_work(struct work_struct *);
-
-/**
- * configure_channel - called from MOST core to configure a channel
- * @most_iface: interface the channel belongs to
- * @ch_idx: channel to be configured
- * @channel_config: structure that holds the configuration information
- *
- * Return 0 on success, negative on failure.
- *
- * Receives configuration information from MOST core and initialize the
- * corresponding channel.
- */
-static int configure_channel(struct most_interface *most_iface,
- int ch_idx,
- struct most_channel_config *channel_config)
-{
- int ret;
- struct hdm_i2c *dev = to_hdm(most_iface);
- unsigned int delay, pr;
-
- BUG_ON(ch_idx < 0 || ch_idx >= NUM_CHANNELS);
-
- if (channel_config->data_type != MOST_CH_CONTROL) {
- pr_err("bad data type for channel %d\n", ch_idx);
- return -EPERM;
- }
-
- if (channel_config->direction != dev->capabilities[ch_idx].direction) {
- pr_err("bad direction for channel %d\n", ch_idx);
- return -EPERM;
- }
-
- if (channel_config->direction == MOST_CH_RX) {
- if (!polling_rate) {
- if (dev->client->irq <= 0) {
- pr_err("bad irq: %d\n", dev->client->irq);
- return -ENOENT;
- }
- dev->rx.int_disabled = false;
- ret = request_irq(dev->client->irq, most_irq_handler, 0,
- dev->client->name, dev);
- if (ret) {
- pr_err("request_irq(%d) failed: %d\n",
- dev->client->irq, ret);
- return ret;
- }
- } else {
- delay = msecs_to_jiffies(MSEC_PER_SEC / polling_rate);
- dev->rx.delay = delay ? delay : 1;
- pr = MSEC_PER_SEC / jiffies_to_msecs(dev->rx.delay);
- pr_info("polling rate is %u Hz\n", pr);
- }
- }
-
- return 0;
-}
-
-/**
- * enqueue - called from MOST core to enqueue a buffer for data transfer
- * @most_iface: intended interface
- * @ch_idx: ID of the channel the buffer is intended for
- * @mbo: pointer to the buffer object
- *
- * Return 0 on success, negative on failure.
- *
- * Transmit the data over I2C if it is a "write" request or push the buffer into
- * list if it is an "read" request
- */
-static int enqueue(struct most_interface *most_iface,
- int ch_idx, struct mbo *mbo)
-{
- struct hdm_i2c *dev = to_hdm(most_iface);
- int ret;
-
- BUG_ON(ch_idx < 0 || ch_idx >= NUM_CHANNELS);
-
- if (ch_idx == CH_RX) {
- /* RX */
- if (!polling_rate)
- disable_irq(dev->client->irq);
- cancel_delayed_work_sync(&dev->rx.dwork);
- list_add_tail(&mbo->list, &dev->rx.list);
- if (dev->rx.int_disabled || polling_rate)
- pending_rx_work(&dev->rx.dwork.work);
- if (!polling_rate)
- enable_irq(dev->client->irq);
- } else {
- /* TX */
- ret = i2c_master_send(dev->client, mbo->virt_address,
- mbo->buffer_length);
- if (ret <= 0) {
- mbo->processed_length = 0;
- mbo->status = MBO_E_INVAL;
- } else {
- mbo->processed_length = mbo->buffer_length;
- mbo->status = MBO_SUCCESS;
- }
- mbo->complete(mbo);
- }
-
- return 0;
-}
-
-/**
- * poison_channel - called from MOST core to poison buffers of a channel
- * @most_iface: pointer to the interface the channel to be poisoned belongs to
- * @ch_idx: corresponding channel ID
- *
- * Return 0 on success, negative on failure.
- *
- * If channel direction is RX, complete the buffers in list with
- * status MBO_E_CLOSE
- */
-static int poison_channel(struct most_interface *most_iface,
- int ch_idx)
-{
- struct hdm_i2c *dev = to_hdm(most_iface);
- struct mbo *mbo;
-
- BUG_ON(ch_idx < 0 || ch_idx >= NUM_CHANNELS);
-
- if (ch_idx == CH_RX) {
- if (!polling_rate)
- free_irq(dev->client->irq, dev);
- cancel_delayed_work_sync(&dev->rx.dwork);
-
- while (!list_empty(&dev->rx.list)) {
- mbo = list_first_mbo(&dev->rx.list);
- list_del(&mbo->list);
-
- mbo->processed_length = 0;
- mbo->status = MBO_E_CLOSE;
- mbo->complete(mbo);
- }
- }
-
- return 0;
-}
-
-static void do_rx_work(struct hdm_i2c *dev)
-{
- struct mbo *mbo;
- unsigned char msg[MAX_BUF_SIZE_CONTROL];
- int ret;
- u16 pml, data_size;
-
- /* Read PML (2 bytes) */
- ret = i2c_master_recv(dev->client, msg, 2);
- if (ret <= 0) {
- pr_err("Failed to receive PML\n");
- return;
- }
-
- pml = (msg[0] << 8) | msg[1];
- if (!pml)
- return;
-
- data_size = pml + 2;
-
- /* Read the whole message, including PML */
- ret = i2c_master_recv(dev->client, msg, data_size);
- if (ret <= 0) {
- pr_err("Failed to receive a Port Message\n");
- return;
- }
-
- mbo = list_first_mbo(&dev->rx.list);
- list_del(&mbo->list);
-
- mbo->processed_length = min(data_size, mbo->buffer_length);
- memcpy(mbo->virt_address, msg, mbo->processed_length);
- mbo->status = MBO_SUCCESS;
- mbo->complete(mbo);
-}
-
-/**
- * pending_rx_work - Read pending messages through I2C
- * @work: definition of this work item
- *
- * Invoked by the Interrupt Service Routine, most_irq_handler()
- */
-static void pending_rx_work(struct work_struct *work)
-{
- struct hdm_i2c *dev = container_of(work, struct hdm_i2c, rx.dwork.work);
-
- if (list_empty(&dev->rx.list))
- return;
-
- do_rx_work(dev);
-
- if (polling_rate) {
- schedule_delayed_work(&dev->rx.dwork, dev->rx.delay);
- } else {
- dev->rx.int_disabled = false;
- enable_irq(dev->client->irq);
- }
-}
-
-/*
- * most_irq_handler - Interrupt Service Routine
- * @irq: irq number
- * @_dev: private data
- *
- * Schedules a delayed work
- *
- * By default the interrupt line behavior is Active Low. Once an interrupt is
- * generated by the device, until driver clears the interrupt (by reading
- * the PMP message), device keeps the interrupt line in low state. Since i2c
- * read is done in work queue, the interrupt line must be disabled temporarily
- * to avoid ISR being called repeatedly. Re-enable the interrupt in workqueue,
- * after reading the message.
- *
- * Note: If we use the interrupt line in Falling edge mode, there is a
- * possibility to miss interrupts when ISR is getting executed.
- *
- */
-static irqreturn_t most_irq_handler(int irq, void *_dev)
-{
- struct hdm_i2c *dev = _dev;
-
- disable_irq_nosync(irq);
- dev->rx.int_disabled = true;
- schedule_delayed_work(&dev->rx.dwork, 0);
-
- return IRQ_HANDLED;
-}
-
-/*
- * i2c_probe - i2c probe handler
- * @client: i2c client device structure
- * @id: i2c client device id
- *
- * Return 0 on success, negative on failure.
- *
- * Register the i2c client device as a MOST interface
- */
-static int i2c_probe(struct i2c_client *client)
-{
- struct hdm_i2c *dev;
- int ret, i;
-
- dev = kzalloc(sizeof(*dev), GFP_KERNEL);
- if (!dev)
- return -ENOMEM;
-
- /* ID format: i2c-<bus>-<address> */
- snprintf(dev->name, sizeof(dev->name), "i2c-%d-%04x",
- client->adapter->nr, client->addr);
-
- for (i = 0; i < NUM_CHANNELS; i++) {
- dev->capabilities[i].data_type = MOST_CH_CONTROL;
- dev->capabilities[i].num_buffers_packet = MAX_BUFFERS_CONTROL;
- dev->capabilities[i].buffer_size_packet = MAX_BUF_SIZE_CONTROL;
- }
- dev->capabilities[CH_RX].direction = MOST_CH_RX;
- dev->capabilities[CH_RX].name_suffix = "rx";
- dev->capabilities[CH_TX].direction = MOST_CH_TX;
- dev->capabilities[CH_TX].name_suffix = "tx";
-
- dev->most_iface.interface = ITYPE_I2C;
- dev->most_iface.description = dev->name;
- dev->most_iface.num_channels = NUM_CHANNELS;
- dev->most_iface.channel_vector = dev->capabilities;
- dev->most_iface.configure = configure_channel;
- dev->most_iface.enqueue = enqueue;
- dev->most_iface.poison_channel = poison_channel;
-
- INIT_LIST_HEAD(&dev->rx.list);
-
- INIT_DELAYED_WORK(&dev->rx.dwork, pending_rx_work);
-
- dev->client = client;
- i2c_set_clientdata(client, dev);
-
- ret = most_register_interface(&dev->most_iface);
- if (ret) {
- pr_err("Failed to register i2c as a MOST interface\n");
- kfree(dev);
- return ret;
- }
-
- return 0;
-}
-
-/*
- * i2c_remove - i2c remove handler
- * @client: i2c client device structure
- *
- * Return 0 on success.
- *
- * Unregister the i2c client device as a MOST interface
- */
-static void i2c_remove(struct i2c_client *client)
-{
- struct hdm_i2c *dev = i2c_get_clientdata(client);
-
- most_deregister_interface(&dev->most_iface);
- kfree(dev);
-}
-
-static const struct i2c_device_id i2c_id[] = {
- { "most_i2c", 0 },
- { }, /* Terminating entry */
-};
-
-MODULE_DEVICE_TABLE(i2c, i2c_id);
-
-static struct i2c_driver i2c_driver = {
- .driver = {
- .name = "hdm_i2c",
- },
- .probe = i2c_probe,
- .remove = i2c_remove,
- .id_table = i2c_id,
-};
-
-module_i2c_driver(i2c_driver);
-
-MODULE_AUTHOR("Andrey Shvetsov <andrey.shvetsov@k2l.de>");
-MODULE_DESCRIPTION("I2C Hardware Dependent Module");
-MODULE_LICENSE("GPL");
diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme.c b/drivers/staging/rtl8723bs/core/rtw_mlme.c
index 506ba55f15b7..e2d5a40085f6 100644
--- a/drivers/staging/rtl8723bs/core/rtw_mlme.c
+++ b/drivers/staging/rtl8723bs/core/rtw_mlme.c
@@ -590,6 +590,8 @@ int rtw_is_desired_network(struct adapter *adapter, struct wlan_network *pnetwor
privacy = pnetwork->network.privacy;
if (check_fwstate(pmlmepriv, WIFI_UNDER_WPS)) {
+ if (pnetwork->network.ie_length < _FIXED_IE_LENGTH_)
+ return false;
if (rtw_get_wps_ie(pnetwork->network.ies+_FIXED_IE_LENGTH_, pnetwork->network.ie_length-_FIXED_IE_LENGTH_, NULL, &wps_ielen))
return true;
else
@@ -604,11 +606,15 @@ int rtw_is_desired_network(struct adapter *adapter, struct wlan_network *pnetwor
bselected = false;
if (psecuritypriv->ndisauthtype == Ndis802_11AuthModeWPA2PSK) {
- p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_));
- if (p && ie_len > 0)
- bselected = true;
- else
+ if (pnetwork->network.ie_length < _BEACON_IE_OFFSET_) {
bselected = false;
+ } else {
+ p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_));
+ if (p && ie_len > 0)
+ bselected = true;
+ else
+ bselected = false;
+ }
}
}
diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c
index 22c29d1955f5..740868a156dc 100644
--- a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c
+++ b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c
@@ -2258,8 +2258,11 @@ void issue_beacon(struct adapter *padapter, int timeout_ms)
spin_unlock_bh(&pmlmepriv->bcn_update_lock);
- if ((pattrib->pktlen + TXDESC_SIZE) > 512)
+ if ((pattrib->pktlen + TXDESC_SIZE) > 512) {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
pattrib->last_txcmdsz = pattrib->pktlen;
@@ -2322,8 +2325,11 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p
pframe += pattrib->hdrlen;
- if (cur_network->ie_length > MAX_IE_SZ)
+ if (cur_network->ie_length > MAX_IE_SZ) {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
if ((pmlmeinfo->state&0x03) == WIFI_FW_AP_STATE) {
pwps_ie = rtw_get_wps_ie(cur_network->ies+_FIXED_IE_LENGTH_, cur_network->ie_length-_FIXED_IE_LENGTH_, NULL, &wps_ielen);
@@ -2370,8 +2376,11 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p
u8 *ies = pmgntframe->buf_addr+TXDESC_OFFSET+sizeof(struct ieee80211_hdr_3addr);
buf = rtw_zmalloc(MAX_IE_SZ);
- if (!buf)
+ if (!buf) {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
ssid_ie = rtw_get_ie(ies+_FIXED_IE_LENGTH_, WLAN_EID_SSID, &ssid_ielen,
(pframe-ies)-_FIXED_IE_LENGTH_);
@@ -2764,10 +2773,13 @@ void issue_asocrsp(struct adapter *padapter, unsigned short status, struct sta_i
SetSeqNum(pwlanhdr, pmlmeext->mgnt_seq);
pmlmeext->mgnt_seq++;
- if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP))
+ if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP)) {
SetFrameSubType(pwlanhdr, pkt_type);
- else
+ } else {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
pattrib->hdrlen = sizeof(struct ieee80211_hdr_3addr);
pattrib->pktlen += pattrib->hdrlen;
diff --git a/drivers/thermal/intel/intel_hfi.c b/drivers/thermal/intel/intel_hfi.c
index 5b18a46a10b0..e71cc0a11d01 100644
--- a/drivers/thermal/intel/intel_hfi.c
+++ b/drivers/thermal/intel/intel_hfi.c
@@ -592,7 +592,7 @@ static void hfi_disable_instance(void *ptr)
hfi_disable();
}
-static void hfi_syscore_resume(void)
+static void hfi_syscore_resume(void *data)
{
/* This code runs only on the boot CPU. */
struct hfi_cpu_info *info = &per_cpu(hfi_cpu_info, 0);
@@ -603,7 +603,7 @@ static void hfi_syscore_resume(void)
hfi_enable_instance(hfi_instance);
}
-static int hfi_syscore_suspend(void)
+static int hfi_syscore_suspend(void *data)
{
/* No locking needed. There is no concurrency with CPU offline. */
hfi_disable();
@@ -611,11 +611,15 @@ static int hfi_syscore_suspend(void)
return 0;
}
-static struct syscore_ops hfi_pm_ops = {
+static const struct syscore_ops hfi_pm_ops = {
.resume = hfi_syscore_resume,
.suspend = hfi_syscore_suspend,
};
+static struct syscore hfi_pm = {
+ .ops = &hfi_pm_ops,
+};
+
static int hfi_thermal_notify(struct notifier_block *nb, unsigned long state,
void *_notify)
{
@@ -710,7 +714,7 @@ void __init intel_hfi_init(void)
if (thermal_genl_register_notifier(&hfi_thermal_nb))
goto err_nl_notif;
- register_syscore_ops(&hfi_pm_ops);
+ register_syscore(&hfi_pm);
return;
diff --git a/drivers/thermal/thermal_hwmon.c b/drivers/thermal/thermal_hwmon.c
index 3162a1a8cbb6..fcbad98d3e1b 100644
--- a/drivers/thermal/thermal_hwmon.c
+++ b/drivers/thermal/thermal_hwmon.c
@@ -98,34 +98,12 @@ thermal_hwmon_lookup_by_type(const struct thermal_zone_device *tz)
struct thermal_hwmon_device *hwmon;
char type[THERMAL_NAME_LENGTH];
- mutex_lock(&thermal_hwmon_list_lock);
list_for_each_entry(hwmon, &thermal_hwmon_list, node) {
strcpy(type, tz->type);
strreplace(type, '-', '_');
- if (!strcmp(hwmon->type, type)) {
- mutex_unlock(&thermal_hwmon_list_lock);
+ if (!strcmp(hwmon->type, type))
return hwmon;
- }
}
- mutex_unlock(&thermal_hwmon_list_lock);
-
- return NULL;
-}
-
-/* Find the temperature input matching a given thermal zone */
-static struct thermal_hwmon_temp *
-thermal_hwmon_lookup_temp(const struct thermal_hwmon_device *hwmon,
- const struct thermal_zone_device *tz)
-{
- struct thermal_hwmon_temp *temp;
-
- mutex_lock(&thermal_hwmon_list_lock);
- list_for_each_entry(temp, &hwmon->tz_list, hwmon_node)
- if (temp->tz == tz) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return temp;
- }
- mutex_unlock(&thermal_hwmon_list_lock);
return NULL;
}
@@ -141,7 +119,9 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
struct thermal_hwmon_device *hwmon;
struct thermal_hwmon_temp *temp;
int new_hwmon_device = 1;
- int result;
+ int result = 0;
+
+ mutex_lock(&thermal_hwmon_list_lock);
hwmon = thermal_hwmon_lookup_by_type(tz);
if (hwmon) {
@@ -150,8 +130,10 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
}
hwmon = kzalloc(sizeof(*hwmon), GFP_KERNEL);
- if (!hwmon)
- return -ENOMEM;
+ if (!hwmon) {
+ result = -ENOMEM;
+ goto unlock;
+ }
INIT_LIST_HEAD(&hwmon->tz_list);
strscpy(hwmon->type, tz->type, THERMAL_NAME_LENGTH);
@@ -199,24 +181,24 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
temp->temp_crit_present = true;
}
- mutex_lock(&thermal_hwmon_list_lock);
if (new_hwmon_device)
list_add_tail(&hwmon->node, &thermal_hwmon_list);
list_add_tail(&temp->hwmon_node, &hwmon->tz_list);
- mutex_unlock(&thermal_hwmon_list_lock);
- return 0;
+ goto unlock;
- unregister_input:
+unregister_input:
device_remove_file(hwmon->device, &temp->temp_input.attr);
- free_temp_mem:
+free_temp_mem:
kfree(temp);
- unregister_name:
+unregister_name:
if (new_hwmon_device)
hwmon_device_unregister(hwmon->device);
- free_mem:
+free_mem:
if (new_hwmon_device)
kfree(hwmon);
+unlock:
+ mutex_unlock(&thermal_hwmon_list_lock);
return result;
}
@@ -224,8 +206,11 @@ EXPORT_SYMBOL_GPL(thermal_add_hwmon_sysfs);
void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz)
{
+ struct thermal_hwmon_temp *temp, *entry;
struct thermal_hwmon_device *hwmon;
- struct thermal_hwmon_temp *temp;
+ bool unregister;
+
+ guard(mutex)(&thermal_hwmon_list_lock);
hwmon = thermal_hwmon_lookup_by_type(tz);
if (unlikely(!hwmon)) {
@@ -234,29 +219,25 @@ void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz)
return;
}
- temp = thermal_hwmon_lookup_temp(hwmon, tz);
- if (unlikely(!temp)) {
- /* Should never happen... */
- dev_dbg(&tz->device, "temperature input lookup failed!\n");
- return;
- }
+ unregister = hwmon->device->parent == &tz->device;
- device_remove_file(hwmon->device, &temp->temp_input.attr);
- if (temp->temp_crit_present)
- device_remove_file(hwmon->device, &temp->temp_crit.attr);
+ list_for_each_entry_safe_reverse(temp, entry, &hwmon->tz_list, hwmon_node) {
+ if (!unregister && temp->tz != tz)
+ continue;
- mutex_lock(&thermal_hwmon_list_lock);
- list_del(&temp->hwmon_node);
- kfree(temp);
- if (!list_empty(&hwmon->tz_list)) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return;
+ device_remove_file(hwmon->device, &temp->temp_input.attr);
+ if (temp->temp_crit_present)
+ device_remove_file(hwmon->device, &temp->temp_crit.attr);
+
+ list_del(&temp->hwmon_node);
+ kfree(temp);
}
- list_del(&hwmon->node);
- mutex_unlock(&thermal_hwmon_list_lock);
- hwmon_device_unregister(hwmon->device);
- kfree(hwmon);
+ if (unregister) {
+ list_del(&hwmon->node);
+ hwmon_device_unregister(hwmon->device);
+ kfree(hwmon);
+ }
}
EXPORT_SYMBOL_GPL(thermal_remove_hwmon_sysfs);
diff --git a/drivers/tty/serial/8250/8250_core.c b/drivers/tty/serial/8250/8250_core.c
index 68baf75bdadc..f35a060a73a7 100644
--- a/drivers/tty/serial/8250/8250_core.c
+++ b/drivers/tty/serial/8250/8250_core.c
@@ -715,139 +715,140 @@ int serial8250_register_8250_port(const struct uart_8250_port *up)
nr_uarts++;
}
- if (uart->port.type != PORT_8250_CIR) {
- struct mctrl_gpios *gpios;
-
- if (uart->port.dev)
- uart_remove_one_port(&serial8250_reg, &uart->port);
-
- uart->port.ctrl_id = up->port.ctrl_id;
- uart->port.port_id = up->port.port_id;
- uart->port.iobase = up->port.iobase;
- uart->port.membase = up->port.membase;
- uart->port.irq = up->port.irq;
- uart->port.irqflags = up->port.irqflags;
- uart->port.uartclk = up->port.uartclk;
- uart->port.fifosize = up->port.fifosize;
- uart->port.regshift = up->port.regshift;
- uart->port.iotype = up->port.iotype;
- uart->port.flags = up->port.flags | UPF_BOOT_AUTOCONF;
- uart->bugs = up->bugs;
- uart->port.mapbase = up->port.mapbase;
- uart->port.mapsize = up->port.mapsize;
- uart->port.private_data = up->port.private_data;
- uart->tx_loadsz = up->tx_loadsz;
- uart->capabilities = up->capabilities;
- uart->port.throttle = up->port.throttle;
- uart->port.unthrottle = up->port.unthrottle;
- uart->port.rs485_config = up->port.rs485_config;
- uart->port.rs485_supported = up->port.rs485_supported;
- uart->port.rs485 = up->port.rs485;
- uart->rs485_start_tx = up->rs485_start_tx;
- uart->rs485_stop_tx = up->rs485_stop_tx;
- uart->lsr_save_mask = up->lsr_save_mask;
- uart->dma = up->dma;
-
- /* Take tx_loadsz from fifosize if it wasn't set separately */
- if (uart->port.fifosize && !uart->tx_loadsz)
- uart->tx_loadsz = uart->port.fifosize;
-
- if (up->port.dev) {
- uart->port.dev = up->port.dev;
- ret = uart_get_rs485_mode(&uart->port);
- if (ret)
- goto err;
- }
+ if (uart->port.type == PORT_8250_CIR) {
+ ret = -ENODEV;
+ goto unlock;
+ }
- if (up->port.flags & UPF_FIXED_TYPE)
- uart->port.type = up->port.type;
+ if (uart->port.dev)
+ uart_remove_one_port(&serial8250_reg, &uart->port);
+
+ uart->port.ctrl_id = up->port.ctrl_id;
+ uart->port.port_id = up->port.port_id;
+ uart->port.iobase = up->port.iobase;
+ uart->port.membase = up->port.membase;
+ uart->port.irq = up->port.irq;
+ uart->port.irqflags = up->port.irqflags;
+ uart->port.uartclk = up->port.uartclk;
+ uart->port.fifosize = up->port.fifosize;
+ uart->port.regshift = up->port.regshift;
+ uart->port.iotype = up->port.iotype;
+ uart->port.flags = up->port.flags | UPF_BOOT_AUTOCONF;
+ uart->bugs = up->bugs;
+ uart->port.mapbase = up->port.mapbase;
+ uart->port.mapsize = up->port.mapsize;
+ uart->port.private_data = up->port.private_data;
+ uart->tx_loadsz = up->tx_loadsz;
+ uart->capabilities = up->capabilities;
+ uart->port.throttle = up->port.throttle;
+ uart->port.unthrottle = up->port.unthrottle;
+ uart->port.rs485_config = up->port.rs485_config;
+ uart->port.rs485_supported = up->port.rs485_supported;
+ uart->port.rs485 = up->port.rs485;
+ uart->rs485_start_tx = up->rs485_start_tx;
+ uart->rs485_stop_tx = up->rs485_stop_tx;
+ uart->lsr_save_mask = up->lsr_save_mask;
+ uart->dma = up->dma;
+
+ /* Take tx_loadsz from fifosize if it wasn't set separately */
+ if (uart->port.fifosize && !uart->tx_loadsz)
+ uart->tx_loadsz = uart->port.fifosize;
+
+ if (up->port.dev) {
+ uart->port.dev = up->port.dev;
+ ret = uart_get_rs485_mode(&uart->port);
+ if (ret)
+ goto err;
+ }
- /*
- * Only call mctrl_gpio_init(), if the device has no ACPI
- * companion device
- */
- if (!has_acpi_companion(uart->port.dev)) {
- gpios = mctrl_gpio_init(&uart->port, 0);
- if (IS_ERR(gpios)) {
- ret = PTR_ERR(gpios);
- goto err;
- } else {
- uart->gpios = gpios;
- }
- }
+ if (up->port.flags & UPF_FIXED_TYPE)
+ uart->port.type = up->port.type;
- serial8250_set_defaults(uart);
-
- /* Possibly override default I/O functions. */
- if (up->port.serial_in)
- uart->port.serial_in = up->port.serial_in;
- if (up->port.serial_out)
- uart->port.serial_out = up->port.serial_out;
- if (up->port.handle_irq)
- uart->port.handle_irq = up->port.handle_irq;
- /* Possibly override set_termios call */
- if (up->port.set_termios)
- uart->port.set_termios = up->port.set_termios;
- if (up->port.set_ldisc)
- uart->port.set_ldisc = up->port.set_ldisc;
- if (up->port.get_mctrl)
- uart->port.get_mctrl = up->port.get_mctrl;
- if (up->port.set_mctrl)
- uart->port.set_mctrl = up->port.set_mctrl;
- if (up->port.get_divisor)
- uart->port.get_divisor = up->port.get_divisor;
- if (up->port.set_divisor)
- uart->port.set_divisor = up->port.set_divisor;
- if (up->port.startup)
- uart->port.startup = up->port.startup;
- if (up->port.shutdown)
- uart->port.shutdown = up->port.shutdown;
- if (up->port.pm)
- uart->port.pm = up->port.pm;
- if (up->port.handle_break)
- uart->port.handle_break = up->port.handle_break;
- if (up->dl_read)
- uart->dl_read = up->dl_read;
- if (up->dl_write)
- uart->dl_write = up->dl_write;
-
- if (uart->port.type != PORT_8250_CIR) {
- if (uart_console_registered(&uart->port))
- pm_runtime_get_sync(uart->port.dev);
-
- if (serial8250_isa_config != NULL)
- serial8250_isa_config(0, &uart->port,
- &uart->capabilities);
-
- serial8250_apply_quirks(uart);
- ret = uart_add_one_port(&serial8250_reg,
- &uart->port);
- if (ret)
- goto err;
-
- ret = uart->port.line;
+ /*
+ * Only call mctrl_gpio_init(), if the device has no ACPI
+ * companion device
+ */
+ if (uart->port.dev && !has_acpi_companion(uart->port.dev)) {
+ struct mctrl_gpios *gpios = mctrl_gpio_init(&uart->port, 0);
+ if (IS_ERR(gpios)) {
+ ret = PTR_ERR(gpios);
+ goto err;
} else {
- dev_info(uart->port.dev,
- "skipping CIR port at 0x%lx / 0x%llx, IRQ %d\n",
- uart->port.iobase,
- (unsigned long long)uart->port.mapbase,
- uart->port.irq);
-
- ret = 0;
+ uart->gpios = gpios;
}
+ }
- if (!uart->lsr_save_mask)
- uart->lsr_save_mask = LSR_SAVE_FLAGS; /* Use default LSR mask */
+ serial8250_set_defaults(uart);
+
+ /* Possibly override default I/O functions. */
+ if (up->port.serial_in)
+ uart->port.serial_in = up->port.serial_in;
+ if (up->port.serial_out)
+ uart->port.serial_out = up->port.serial_out;
+ if (up->port.handle_irq)
+ uart->port.handle_irq = up->port.handle_irq;
+ /* Possibly override set_termios call */
+ if (up->port.set_termios)
+ uart->port.set_termios = up->port.set_termios;
+ if (up->port.set_ldisc)
+ uart->port.set_ldisc = up->port.set_ldisc;
+ if (up->port.get_mctrl)
+ uart->port.get_mctrl = up->port.get_mctrl;
+ if (up->port.set_mctrl)
+ uart->port.set_mctrl = up->port.set_mctrl;
+ if (up->port.get_divisor)
+ uart->port.get_divisor = up->port.get_divisor;
+ if (up->port.set_divisor)
+ uart->port.set_divisor = up->port.set_divisor;
+ if (up->port.startup)
+ uart->port.startup = up->port.startup;
+ if (up->port.shutdown)
+ uart->port.shutdown = up->port.shutdown;
+ if (up->port.pm)
+ uart->port.pm = up->port.pm;
+ if (up->port.handle_break)
+ uart->port.handle_break = up->port.handle_break;
+ if (up->dl_read)
+ uart->dl_read = up->dl_read;
+ if (up->dl_write)
+ uart->dl_write = up->dl_write;
- /* Initialise interrupt backoff work if required */
- if (up->overrun_backoff_time_ms > 0) {
- uart->overrun_backoff_time_ms =
- up->overrun_backoff_time_ms;
- INIT_DELAYED_WORK(&uart->overrun_backoff,
- serial_8250_overrun_backoff_work);
- } else {
- uart->overrun_backoff_time_ms = 0;
- }
+ if (uart->port.type != PORT_8250_CIR) {
+ if (uart_console_registered(&uart->port))
+ pm_runtime_get_sync(uart->port.dev);
+
+ if (serial8250_isa_config != NULL)
+ serial8250_isa_config(0, &uart->port,
+ &uart->capabilities);
+
+ serial8250_apply_quirks(uart);
+ ret = uart_add_one_port(&serial8250_reg,
+ &uart->port);
+ if (ret)
+ goto err;
+
+ ret = uart->port.line;
+ } else {
+ dev_info(uart->port.dev,
+ "skipping CIR port at 0x%lx / 0x%llx, IRQ %d\n",
+ uart->port.iobase,
+ (unsigned long long)uart->port.mapbase,
+ uart->port.irq);
+
+ ret = 0;
+ }
+
+ if (!uart->lsr_save_mask)
+ uart->lsr_save_mask = LSR_SAVE_FLAGS; /* Use default LSR mask */
+
+ /* Initialise interrupt backoff work if required */
+ if (up->overrun_backoff_time_ms > 0) {
+ uart->overrun_backoff_time_ms =
+ up->overrun_backoff_time_ms;
+ INIT_DELAYED_WORK(&uart->overrun_backoff,
+ serial_8250_overrun_backoff_work);
+ } else {
+ uart->overrun_backoff_time_ms = 0;
}
unlock:
diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
index 2a33ad1e039d..230df69c9f93 100644
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -75,22 +75,23 @@ static inline void uart_port_deref(struct uart_port *uport)
wake_up(&uport->state->remove_wait);
}
-#define uart_port_lock(state, flags) \
- ({ \
- struct uart_port *__uport = uart_port_ref(state); \
- if (__uport) \
- uart_port_lock_irqsave(__uport, &flags); \
- __uport; \
- })
-
-#define uart_port_unlock(uport, flags) \
- ({ \
- struct uart_port *__uport = uport; \
- if (__uport) { \
- uart_port_unlock_irqrestore(__uport, flags); \
- uart_port_deref(__uport); \
- } \
- })
+static inline struct uart_port *uart_port_ref_lock(struct uart_state *state, unsigned long *flags)
+{
+ struct uart_port *uport = uart_port_ref(state);
+
+ if (uport)
+ uart_port_lock_irqsave(uport, flags);
+
+ return uport;
+}
+
+static inline void uart_port_unlock_deref(struct uart_port *uport, unsigned long flags)
+{
+ if (uport) {
+ uart_port_unlock_irqrestore(uport, flags);
+ uart_port_deref(uport);
+ }
+}
static inline struct uart_port *uart_port_check(struct uart_state *state)
{
@@ -127,10 +128,10 @@ static void uart_stop(struct tty_struct *tty)
struct uart_port *port;
unsigned long flags;
- port = uart_port_lock(state, flags);
+ port = uart_port_ref_lock(state, &flags);
if (port)
port->ops->stop_tx(port);
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
}
static void __uart_start(struct uart_state *state)
@@ -168,9 +169,9 @@ static void uart_start(struct tty_struct *tty)
struct uart_port *port;
unsigned long flags;
- port = uart_port_lock(state, flags);
+ port = uart_port_ref_lock(state, &flags);
__uart_start(state);
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
}
static void
@@ -258,14 +259,14 @@ static int uart_alloc_xmit_buf(struct tty_port *port)
if (!page)
return -ENOMEM;
- uport = uart_port_lock(state, flags);
+ uport = uart_port_ref_lock(state, &flags);
if (!state->port.xmit_buf) {
state->port.xmit_buf = (unsigned char *)page;
kfifo_init(&state->port.xmit_fifo, state->port.xmit_buf,
PAGE_SIZE);
- uart_port_unlock(uport, flags);
+ uart_port_unlock_deref(uport, flags);
} else {
- uart_port_unlock(uport, flags);
+ uart_port_unlock_deref(uport, flags);
/*
* Do not free() the page under the port lock, see
* uart_free_xmit_buf().
@@ -289,11 +290,11 @@ static void uart_free_xmit_buf(struct tty_port *port)
* console driver may need to allocate/free a debug object, which
* can end up in printk() recursion.
*/
- uport = uart_port_lock(state, flags);
+ uport = uart_port_ref_lock(state, &flags);
xmit_buf = port->xmit_buf;
port->xmit_buf = NULL;
INIT_KFIFO(port->xmit_fifo);
- uart_port_unlock(uport, flags);
+ uart_port_unlock_deref(uport, flags);
free_page((unsigned long)xmit_buf);
}
@@ -592,15 +593,15 @@ static int uart_put_char(struct tty_struct *tty, u8 c)
unsigned long flags;
int ret = 0;
- port = uart_port_lock(state, flags);
+ port = uart_port_ref_lock(state, &flags);
if (!state->port.xmit_buf) {
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
return 0;
}
if (port)
ret = kfifo_put(&state->port.xmit_fifo, c);
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
return ret;
}
@@ -623,9 +624,9 @@ static ssize_t uart_write(struct tty_struct *tty, const u8 *buf, size_t count)
if (WARN_ON(!state))
return -EL3HLT;
- port = uart_port_lock(state, flags);
+ port = uart_port_ref_lock(state, &flags);
if (!state->port.xmit_buf) {
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
return 0;
}
@@ -633,7 +634,7 @@ static ssize_t uart_write(struct tty_struct *tty, const u8 *buf, size_t count)
ret = kfifo_in(&state->port.xmit_fifo, buf, count);
__uart_start(state);
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
return ret;
}
@@ -644,9 +645,12 @@ static unsigned int uart_write_room(struct tty_struct *tty)
unsigned long flags;
unsigned int ret;
- port = uart_port_lock(state, flags);
- ret = kfifo_avail(&state->port.xmit_fifo);
- uart_port_unlock(port, flags);
+ port = uart_port_ref_lock(state, &flags);
+ if (!state->port.xmit_buf)
+ ret = 0;
+ else
+ ret = kfifo_avail(&state->port.xmit_fifo);
+ uart_port_unlock_deref(port, flags);
return ret;
}
@@ -657,9 +661,9 @@ static unsigned int uart_chars_in_buffer(struct tty_struct *tty)
unsigned long flags;
unsigned int ret;
- port = uart_port_lock(state, flags);
+ port = uart_port_ref_lock(state, &flags);
ret = kfifo_len(&state->port.xmit_fifo);
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
return ret;
}
@@ -678,13 +682,13 @@ static void uart_flush_buffer(struct tty_struct *tty)
pr_debug("uart_flush_buffer(%d) called\n", tty->index);
- port = uart_port_lock(state, flags);
+ port = uart_port_ref_lock(state, &flags);
if (!port)
return;
kfifo_reset(&state->port.xmit_fifo);
if (port->ops->flush_buffer)
port->ops->flush_buffer(port);
- uart_port_unlock(port, flags);
+ uart_port_unlock_deref(port, flags);
tty_port_tty_wakeup(&state->port);
}
@@ -2857,8 +2861,10 @@ int uart_register_driver(struct uart_driver *drv)
for (i = 0; i < drv->nr; i++)
tty_port_destroy(&drv->state[i].port);
tty_driver_kref_put(normal);
+ drv->tty_driver = NULL;
out_kfree:
kfree(drv->state);
+ drv->state = NULL;
out:
return retval;
}
@@ -3138,7 +3144,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
struct tty_port *port;
int ret = 0;
struct device *tty_dev;
- int num_groups;
if (uport->line >= drv->nr)
return -EINVAL;
@@ -3152,6 +3157,25 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
goto out;
}
+ uport->name = kasprintf(GFP_KERNEL, "%s%u", drv->dev_name,
+ drv->tty_driver->name_base + uport->line);
+ if (!uport->name) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ /*
+ * uart_configure_port() may set uport->attr_group and register the
+ * console. Allocate room for both groups and a NULL terminator first.
+ */
+ uport->tty_groups = kcalloc(3, sizeof(*uport->tty_groups), GFP_KERNEL);
+ if (!uport->tty_groups) {
+ kfree(uport->name);
+ ret = -ENOMEM;
+ goto out;
+ }
+ uport->tty_groups[0] = &tty_dev_attr_group;
+
/* Link the port to the driver state table and vice versa */
atomic_set(&state->refcount, 1);
init_waitqueue_head(&state->remove_wait);
@@ -3168,12 +3192,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
state->pm_state = UART_PM_STATE_UNDEFINED;
uart_port_set_cons(uport, drv->cons);
uport->minor = drv->tty_driver->minor_start + uport->line;
- uport->name = kasprintf(GFP_KERNEL, "%s%u", drv->dev_name,
- drv->tty_driver->name_base + uport->line);
- if (!uport->name) {
- ret = -ENOMEM;
- goto out;
- }
if (uport->cons && uport->dev)
of_console_check(uport->dev->of_node, uport->cons->name, uport->line);
@@ -3183,17 +3201,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
port->console = uart_console(uport);
- num_groups = 2;
- if (uport->attr_group)
- num_groups++;
-
- uport->tty_groups = kcalloc(num_groups, sizeof(*uport->tty_groups),
- GFP_KERNEL);
- if (!uport->tty_groups) {
- ret = -ENOMEM;
- goto out;
- }
- uport->tty_groups[0] = &tty_dev_attr_group;
if (uport->attr_group)
uport->tty_groups[1] = uport->attr_group;
diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c
index 07e1e901dea5..dd2e2508df4c 100644
--- a/drivers/tty/tty_io.c
+++ b/drivers/tty/tty_io.c
@@ -3341,7 +3341,7 @@ EXPORT_SYMBOL_GPL(tty_register_device_attr);
void tty_unregister_device(struct tty_driver *driver, unsigned index)
{
device_destroy(&tty_class, MKDEV(driver->major, driver->minor_start) + index);
- if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) {
+ if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC) && driver->cdevs[index]) {
cdev_del(driver->cdevs[index]);
driver->cdevs[index] = NULL;
}
diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c
index 5e6197a6af5e..b6514a8b268f 100644
--- a/drivers/ufs/core/ufs-mcq.c
+++ b/drivers/ufs/core/ufs-mcq.c
@@ -299,19 +299,34 @@ static int ufshcd_mcq_get_tag(struct ufs_hba *hba, struct cq_entry *cqe)
return div_u64(addr, ufshcd_get_ucd_size(hba));
}
+static void ufshcd_mcq_compl_cqe(struct ufs_hba *hba, struct cq_entry *cqe)
+{
+ int tag = ufshcd_mcq_get_tag(hba, cqe);
+
+ ufshcd_compl_one_cqe(hba, tag, cqe);
+ /* After processing the CQE, mark it as an empty (invalid) entry. */
+ cqe->command_desc_base_addr = 0;
+}
+
static void ufshcd_mcq_process_cqe(struct ufs_hba *hba,
struct ufs_hw_queue *hwq)
{
struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq);
- int tag = ufshcd_mcq_get_tag(hba, cqe);
- if (cqe->command_desc_base_addr) {
- ufshcd_compl_one_cqe(hba, tag, cqe);
- /* After processed the cqe, mark it empty (invalid) entry */
- cqe->command_desc_base_addr = 0;
- }
+ if (cqe->command_desc_base_addr)
+ ufshcd_mcq_compl_cqe(hba, cqe);
+ else
+ dev_err(hba->dev, "Abnormal CQ entry!\n");
}
+/*
+ * This function is called from the UFS error handler with the UFS host
+ * controller disabled (HCE = 0). Reading host controller registers, e.g. the
+ * CQ tail pointer (CQTPy), may not be safe with the host controller disabled.
+ * Hence, iterate over all completion queue entries. This won't result in
+ * double completions because ufshcd_mcq_compl_cqe() clears a CQE after it
+ * has been processed.
+ */
void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,
struct ufs_hw_queue *hwq)
{
@@ -320,13 +335,13 @@ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,
spin_lock_irqsave(&hwq->cq_lock, flags);
while (entries > 0) {
- ufshcd_mcq_process_cqe(hba, hwq);
+ struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq);
+
+ if (cqe->command_desc_base_addr)
+ ufshcd_mcq_compl_cqe(hba, cqe);
ufshcd_mcq_inc_cq_head_slot(hwq);
entries--;
}
-
- ufshcd_mcq_update_cq_tail_slot(hwq);
- hwq->cq_head_slot = hwq->cq_tail_slot;
spin_unlock_irqrestore(&hwq->cq_lock, flags);
}
diff --git a/drivers/ufs/core/ufs-sysfs.c b/drivers/ufs/core/ufs-sysfs.c
index f8397ef3cf8d..5d4f23bb6ab1 100644
--- a/drivers/ufs/core/ufs-sysfs.c
+++ b/drivers/ufs/core/ufs-sysfs.c
@@ -426,6 +426,8 @@ static ssize_t pm_qos_enable_show(struct device *dev,
{
struct ufs_hba *hba = dev_get_drvdata(dev);
+ guard(mutex)(&hba->pm_qos_mutex);
+
return sysfs_emit(buf, "%d\n", hba->pm_qos_enabled);
}
diff --git a/drivers/ufs/core/ufshcd-priv.h b/drivers/ufs/core/ufshcd-priv.h
index 786f20ef2238..8a799d9758f5 100644
--- a/drivers/ufs/core/ufshcd-priv.h
+++ b/drivers/ufs/core/ufshcd-priv.h
@@ -78,11 +78,17 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag);
void ufshcd_release_scsi_cmd(struct ufs_hba *hba,
struct ufshcd_lrb *lrbp);
-#define SD_ASCII_STD true
-#define SD_RAW false
-int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
- u8 **buf, bool ascii);
+/**
+ * enum ufs_descr_fmt - UFS string descriptor format
+ * @SD_RAW: Raw UTF-16 format
+ * @SD_ASCII_STD: Convert to null-terminated ASCII string
+ */
+enum ufs_descr_fmt {
+ SD_RAW = 0,
+ SD_ASCII_STD = 1,
+};
+int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, u8 **buf, enum ufs_descr_fmt fmt);
int ufshcd_send_uic_cmd(struct ufs_hba *hba, struct uic_command *uic_cmd);
int ufshcd_send_bsg_uic_cmd(struct ufs_hba *hba, struct uic_command *uic_cmd);
diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 22f4f5f08e74..c8f007f5a059 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -1023,6 +1023,7 @@ EXPORT_SYMBOL_GPL(ufshcd_is_hba_active);
*/
void ufshcd_pm_qos_init(struct ufs_hba *hba)
{
+ guard(mutex)(&hba->pm_qos_mutex);
if (hba->pm_qos_enabled)
return;
@@ -1039,6 +1040,8 @@ void ufshcd_pm_qos_init(struct ufs_hba *hba)
*/
void ufshcd_pm_qos_exit(struct ufs_hba *hba)
{
+ guard(mutex)(&hba->pm_qos_mutex);
+
if (!hba->pm_qos_enabled)
return;
@@ -1053,6 +1056,8 @@ void ufshcd_pm_qos_exit(struct ufs_hba *hba)
*/
static void ufshcd_pm_qos_update(struct ufs_hba *hba, bool on)
{
+ guard(mutex)(&hba->pm_qos_mutex);
+
if (!hba->pm_qos_enabled)
return;
@@ -3733,20 +3738,18 @@ static inline char ufshcd_remove_non_printable(u8 ch)
* @desc_index: descriptor index
* @buf: pointer to buffer where descriptor would be read,
* the caller should free the memory.
- * @ascii: if true convert from unicode to ascii characters
- * null terminated string.
+ * @fmt: if %SD_ASCII_STD, convert from UTF-16 to ASCII
*
* Return:
* * string size on success.
* * -ENOMEM: on allocation failure
* * -EINVAL: on a wrong parameter
*/
-int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
- u8 **buf, bool ascii)
+int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, u8 **buf, enum ufs_descr_fmt fmt)
{
struct uc_string_id *uc_str;
u8 *str;
- int ret;
+ int ret, uc_len;
if (!buf)
return -EINVAL;
@@ -3771,11 +3774,19 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
goto out;
}
- if (ascii) {
+ uc_len = uc_str->len - QUERY_DESC_HDR_SIZE;
+ if (uc_len % sizeof(*uc_str->uc)) {
+ dev_err(hba->dev, "String Desc has an odd UTF-16 payload length\n");
+ str = NULL;
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (fmt == SD_ASCII_STD) {
ssize_t ascii_len;
int i;
- /* remove header and divide by 2 to move from UTF16 to UTF8 */
- ascii_len = (uc_str->len - QUERY_DESC_HDR_SIZE) / 2 + 1;
+ /* Allow up to three UTF-8 bytes per UTF-16 code unit plus a NUL. */
+ ascii_len = uc_len / sizeof(*uc_str->uc) * 3 + 1;
str = kzalloc(ascii_len, GFP_KERNEL);
if (!str) {
ret = -ENOMEM;
@@ -3787,7 +3798,7 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
* we need to convert to utf-8 so it can be displayed
*/
ret = utf16s_to_utf8s(uc_str->uc,
- uc_str->len - QUERY_DESC_HDR_SIZE,
+ uc_len / sizeof(*uc_str->uc),
UTF16_BIG_ENDIAN, str, ascii_len - 1);
/* replace non-printable or non-ASCII characters with spaces */
@@ -3797,11 +3808,17 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
str[ret++] = '\0';
} else {
- str = kmemdup(uc_str->uc, uc_str->len, GFP_KERNEL);
+ /*
+ * Keep the bLength-sized raw output for the RPMB device ID ABI.
+ * The two bytes beyond the UTF-16 payload are explicitly zeroed
+ * instead of being read past the descriptor buffer.
+ */
+ str = kzalloc(uc_str->len, GFP_KERNEL);
if (!str) {
ret = -ENOMEM;
goto out;
}
+ memcpy(str, uc_str->uc, uc_len);
ret = uc_str->len;
}
out:
@@ -5685,7 +5702,17 @@ static void ufshcd_mcq_compl_pending_transfer(struct ufs_hba *hba,
struct ufshcd_lrb *lrbp;
struct scsi_cmnd *cmd;
unsigned long flags;
- int tag;
+ int tag, i;
+
+ for (i = 0; i < hba->nr_hw_queues; i++) {
+ if (force_compl)
+ ufshcd_mcq_compl_all_cqes_lock(hba, &hba->uhq[i]);
+ else
+ ufshcd_mcq_poll_cqe_lock(hba, &hba->uhq[i]);
+ }
+
+ if (!force_compl)
+ return;
for (tag = 0; tag < hba->nutrs; tag++) {
lrbp = &hba->lrb[tag];
@@ -5698,22 +5725,17 @@ static void ufshcd_mcq_compl_pending_transfer(struct ufs_hba *hba,
if (!hwq)
continue;
- if (force_compl) {
- ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
- /*
- * For those cmds of which the cqes are not present
- * in the cq, complete them explicitly.
- */
- spin_lock_irqsave(&hwq->cq_lock, flags);
- if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) {
- set_host_byte(cmd, DID_REQUEUE);
- ufshcd_release_scsi_cmd(hba, lrbp);
- scsi_done(cmd);
- }
- spin_unlock_irqrestore(&hwq->cq_lock, flags);
- } else {
- ufshcd_mcq_poll_cqe_lock(hba, hwq);
+ /*
+ * For those cmds of which the cqes are not present
+ * in the cq, complete them explicitly.
+ */
+ spin_lock_irqsave(&hwq->cq_lock, flags);
+ if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) {
+ set_host_byte(cmd, DID_REQUEUE);
+ ufshcd_release_scsi_cmd(hba, lrbp);
+ scsi_done(cmd);
}
+ spin_unlock_irqrestore(&hwq->cq_lock, flags);
}
}
@@ -9551,7 +9573,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
* we are functional while we are here, skip host resume in error
* handling context.
*/
- hba->host->eh_noresume = 1;
+ WRITE_ONCE(hba->host->eh_noresume, 1);
/*
* Current function would be generally called from the power management
@@ -9573,7 +9595,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
}
scsi_device_put(sdp);
- hba->host->eh_noresume = 0;
+ WRITE_ONCE(hba->host->eh_noresume, 0);
return ret;
}
@@ -10540,6 +10562,9 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
*/
spin_lock_init(&hba->clk_gating.lock);
+ /* Initialize mutex for PM QoS request synchronization */
+ mutex_init(&hba->pm_qos_mutex);
+
/*
* Set the default power management level for runtime and system PM.
* Host controller drivers can override them in their
@@ -10625,6 +10650,7 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
mutex_init(&hba->ee_ctrl_mutex);
mutex_init(&hba->wb_mutex);
+
init_rwsem(&hba->clk_scaling_lock);
ufshcd_init_clk_gating(hba);
diff --git a/drivers/ufs/host/ufs-qcom.c b/drivers/ufs/host/ufs-qcom.c
index a715f377d0a8..ba760aab3a7d 100644
--- a/drivers/ufs/host/ufs-qcom.c
+++ b/drivers/ufs/host/ufs-qcom.c
@@ -561,25 +561,29 @@ static int ufs_qcom_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op,
enum ufs_notify_change_status status)
{
struct ufs_qcom_host *host = ufshcd_get_variant(hba);
- struct phy *phy = host->generic_phy;
if (status == PRE_CHANGE)
return 0;
- if (ufs_qcom_is_link_off(hba)) {
- /*
- * Disable the tx/rx lane symbol clocks before PHY is
- * powered down as the PLL source should be disabled
- * after downstream clocks are disabled.
- */
+ if (!ufs_qcom_is_link_active(hba))
ufs_qcom_disable_lane_clks(host);
- phy_power_off(phy);
- /* reset the connected UFS device during power down */
- ufs_qcom_device_reset_ctrl(hba, true);
- } else if (!ufs_qcom_is_link_active(hba)) {
- ufs_qcom_disable_lane_clks(host);
+ /* reset the connected UFS device during power down */
+ if (ufs_qcom_is_link_off(hba) && host->device_reset) {
+ ufs_qcom_device_reset_ctrl(hba, true);
+ /*
+ * After sending the SSU command, asserting the rst_n
+ * line causes the device firmware to wake up and
+ * execute its reset routine.
+ *
+ * During this process, the device may draw current
+ * beyond the permissible limit for low-power mode (LPM).
+ * A 10ms delay, based on experimental observations,
+ * allows the UFS device to complete its hardware reset
+ * before transitioning the power rail to LPM.
+ */
+ usleep_range(10000, 11000);
}
return ufs_qcom_ice_suspend(host);
@@ -588,26 +592,11 @@ static int ufs_qcom_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op,
static int ufs_qcom_resume(struct ufs_hba *hba, enum ufs_pm_op pm_op)
{
struct ufs_qcom_host *host = ufshcd_get_variant(hba);
- struct phy *phy = host->generic_phy;
int err;
- if (ufs_qcom_is_link_off(hba)) {
- err = phy_power_on(phy);
- if (err) {
- dev_err(hba->dev, "%s: failed PHY power on: %d\n",
- __func__, err);
- return err;
- }
-
- err = ufs_qcom_enable_lane_clks(host);
- if (err)
- return err;
-
- } else if (!ufs_qcom_is_link_active(hba)) {
- err = ufs_qcom_enable_lane_clks(host);
- if (err)
- return err;
- }
+ err = ufs_qcom_enable_lane_clks(host);
+ if (err)
+ return err;
return ufs_qcom_ice_resume(host);
}
@@ -933,12 +922,20 @@ static void ufs_qcom_set_caps(struct ufs_hba *hba)
* @on: If true, enable clocks else disable them.
* @status: PRE_CHANGE or POST_CHANGE notify
*
+ * There are certain clocks which comes from the PHY so it needs
+ * to be managed together along with controller clocks which also
+ * provides a better power saving. Hence keep phy_power_off/on calls
+ * in ufs_qcom_setup_clocks, so that PHY's regulators & clks can be
+ * turned on/off along with UFS's clocks.
+ *
* Return: 0 on success, non-zero on failure.
*/
static int ufs_qcom_setup_clocks(struct ufs_hba *hba, bool on,
enum ufs_notify_change_status status)
{
struct ufs_qcom_host *host = ufshcd_get_variant(hba);
+ struct phy *phy;
+ int err;
/*
* In case ufs_qcom_init() is not yet done, simply ignore.
@@ -948,6 +945,8 @@ static int ufs_qcom_setup_clocks(struct ufs_hba *hba, bool on,
if (!host)
return 0;
+ phy = host->generic_phy;
+
switch (status) {
case PRE_CHANGE:
if (on) {
@@ -957,10 +956,22 @@ static int ufs_qcom_setup_clocks(struct ufs_hba *hba, bool on,
/* disable device ref_clk */
ufs_qcom_dev_ref_clk_ctrl(host, false);
}
+
+ err = phy_power_off(phy);
+ if (err) {
+ dev_err(hba->dev, "phy power off failed, ret=%d\n", err);
+ return err;
+ }
}
break;
case POST_CHANGE:
if (on) {
+ err = phy_power_on(phy);
+ if (err) {
+ dev_err(hba->dev, "phy power on failed, ret = %d\n", err);
+ return err;
+ }
+
/* enable the device ref clock for HS mode*/
if (ufshcd_is_hs_mode(&hba->pwr_info))
ufs_qcom_dev_ref_clk_ctrl(host, true);
diff --git a/drivers/usb/gadget/function/f_fs.c b/drivers/usb/gadget/function/f_fs.c
index 489054d36953..8bad59b4242c 100644
--- a/drivers/usb/gadget/function/f_fs.c
+++ b/drivers/usb/gadget/function/f_fs.c
@@ -649,13 +649,22 @@ static ssize_t ffs_ep0_read(struct file *file, char __user *buf,
static int ffs_ep0_open(struct inode *inode, struct file *file)
{
- struct ffs_data *ffs = inode->i_private;
+ struct ffs_data *ffs = inode->i_sb->s_fs_info;
+ int ret;
- if (ffs->state == FFS_CLOSING)
- return -EBUSY;
+ /* Acquire mutex */
+ ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK);
+ if (ret < 0)
+ return ret;
- file->private_data = ffs;
ffs_data_opened(ffs);
+ if (ffs->state == FFS_CLOSING) {
+ ffs_data_closed(ffs);
+ mutex_unlock(&ffs->mutex);
+ return -EBUSY;
+ }
+ mutex_unlock(&ffs->mutex);
+ file->private_data = ffs;
return stream_open(inode, file);
}
@@ -1212,14 +1221,34 @@ static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data)
static int
ffs_epfile_open(struct inode *inode, struct file *file)
{
- struct ffs_epfile *epfile = inode->i_private;
+ struct ffs_data *ffs = inode->i_sb->s_fs_info;
+ struct ffs_epfile *epfile;
+ int ret;
- if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
+ /* Acquire mutex */
+ ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK);
+ if (ret < 0)
+ return ret;
+
+ if (!atomic_inc_not_zero(&ffs->opened)) {
+ mutex_unlock(&ffs->mutex);
return -ENODEV;
+ }
+ ffs_data_get(ffs);
+ /*
+ * we want the state to be FFS_ACTIVE; FFS_ACTIVE alone is
+ * not enough, though - we might have been through FFS_CLOSING
+ * and back to FFS_ACTIVE, with our file already removed.
+ */
+ epfile = smp_load_acquire(&inode->i_private);
+ if (unlikely(ffs->state != FFS_ACTIVE || !epfile)) {
+ mutex_unlock(&ffs->mutex);
+ ffs_data_closed(ffs);
+ return -ENODEV;
+ }
+ mutex_unlock(&ffs->mutex);
file->private_data = epfile;
- ffs_data_opened(epfile->ffs);
-
return stream_open(inode, file);
}
@@ -1368,7 +1397,7 @@ static void ffs_dmabuf_put(struct dma_buf_attachment *attach)
static int
ffs_epfile_release(struct inode *inode, struct file *file)
{
- struct ffs_epfile *epfile = inode->i_private;
+ struct ffs_epfile *epfile = file->private_data;
struct ffs_dmabuf_priv *priv, *tmp;
struct ffs_data *ffs = epfile->ffs;
@@ -2413,6 +2442,11 @@ static int ffs_epfiles_create(struct ffs_data *ffs)
return 0;
}
+static void clear_one(struct dentry *dentry)
+{
+ smp_store_release(&dentry->d_inode->i_private, NULL);
+}
+
static void ffs_epfiles_destroy(struct ffs_epfile *epfiles, unsigned count)
{
struct ffs_epfile *epfile = epfiles;
@@ -2421,8 +2455,7 @@ static void ffs_epfiles_destroy(struct ffs_epfile *epfiles, unsigned count)
BUG_ON(mutex_is_locked(&epfile->mutex));
__ffs_epfile_read_buffer_free(epfile);
if (epfile->dentry) {
- d_delete(epfile->dentry);
- dput(epfile->dentry);
+ simple_recursive_removal(epfile->dentry, clear_one);
epfile->dentry = NULL;
}
}
diff --git a/drivers/usb/gadget/function/f_hid.c b/drivers/usb/gadget/function/f_hid.c
index 2f4a5d22aca4..b5702ccb7b2f 100644
--- a/drivers/usb/gadget/function/f_hid.c
+++ b/drivers/usb/gadget/function/f_hid.c
@@ -1272,8 +1272,10 @@ static int hidg_bind(struct usb_configuration *c, struct usb_function *f)
/* create char device */
hidg->cdev = cdev_alloc();
- if (!hidg->cdev)
+ if (!hidg->cdev) {
+ status = -ENOMEM;
goto fail_free_all;
+ }
hidg->cdev->ops = &f_hidg_fops;
status = cdev_device_add(hidg->cdev, &hidg->dev);
diff --git a/drivers/usb/gadget/function/f_uac1.c b/drivers/usb/gadget/function/f_uac1.c
index c87e74afc881..0a247cca3488 100644
--- a/drivers/usb/gadget/function/f_uac1.c
+++ b/drivers/usb/gadget/function/f_uac1.c
@@ -1594,7 +1594,8 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
const char *page, size_t len) \
{ \
struct f_uac1_opts *opts = to_f_uac1_opts(item); \
- char *split_page = NULL; \
+ char *buf = NULL; \
+ char *split_page; \
int ret = -EINVAL; \
char *token; \
u32 num; \
@@ -1608,18 +1609,22 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
\
i = 0; \
memset(opts->name##s, 0x00, sizeof(opts->name##s)); \
- split_page = kstrdup(page, GFP_KERNEL); \
+ buf = kstrdup(page, GFP_KERNEL); \
+ split_page = buf; \
while ((token = strsep(&split_page, ",")) != NULL) { \
ret = kstrtou32(token, 0, &num); \
if (ret) \
goto end; \
- \
+ if (i >= UAC_MAX_RATES) { \
+ ret = -EINVAL; \
+ goto end; \
+ } \
opts->name##s[i++] = num; \
ret = len; \
}; \
\
end: \
- kfree(split_page); \
+ kfree(buf); \
mutex_unlock(&opts->lock); \
return ret; \
} \
diff --git a/drivers/usb/gadget/function/f_uac2.c b/drivers/usb/gadget/function/f_uac2.c
index 9b324821c93b..e45db4444c49 100644
--- a/drivers/usb/gadget/function/f_uac2.c
+++ b/drivers/usb/gadget/function/f_uac2.c
@@ -2012,7 +2012,8 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \
const char *page, size_t len) \
{ \
struct f_uac2_opts *opts = to_f_uac2_opts(item); \
- char *split_page = NULL; \
+ char *buf = NULL; \
+ char *split_page; \
int ret = -EINVAL; \
char *token; \
u32 num; \
@@ -2026,18 +2027,22 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \
\
i = 0; \
memset(opts->name##s, 0x00, sizeof(opts->name##s)); \
- split_page = kstrdup(page, GFP_KERNEL); \
+ buf = kstrdup(page, GFP_KERNEL); \
+ split_page = buf; \
while ((token = strsep(&split_page, ",")) != NULL) { \
ret = kstrtou32(token, 0, &num); \
if (ret) \
goto end; \
- \
+ if (i >= UAC_MAX_RATES) { \
+ ret = -EINVAL; \
+ goto end; \
+ } \
opts->name##s[i++] = num; \
ret = len; \
}; \
\
end: \
- kfree(split_page); \
+ kfree(buf); \
mutex_unlock(&opts->lock); \
return ret; \
} \
diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
index 2cedf7e2dbc4..42f2f44b383c 100644
--- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
@@ -11,6 +11,8 @@
#define MLX5V_ETH_HARD_MTU (ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN)
+extern int mlx5_vdpa_max_iotlb_entries;
+
struct mlx5_vdpa_direct_mr {
u64 start;
u64 end;
diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
index 25e2c234bcef..a5b5feb74f26 100644
--- a/drivers/vdpa/mlx5/core/mr.c
+++ b/drivers/vdpa/mlx5/core/mr.c
@@ -778,6 +778,9 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev,
{
int err;
+ if (mlx5_vdpa_max_iotlb_entries < 2)
+ return -EINVAL;
+
if (iotlb)
err = create_user_mr(mvdev, mr, iotlb);
else
@@ -786,7 +789,7 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev,
if (err)
return err;
- mr->iotlb = vhost_iotlb_alloc(0, 0);
+ mr->iotlb = vhost_iotlb_alloc(mlx5_vdpa_max_iotlb_entries, 0);
if (!mr->iotlb) {
err = -ENOMEM;
goto err_mr;
diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c
index aeae31d0cefa..28a4d7a35bf4 100644
--- a/drivers/vdpa/mlx5/core/resources.c
+++ b/drivers/vdpa/mlx5/core/resources.c
@@ -3,8 +3,14 @@
#include <linux/iova.h>
#include <linux/mlx5/driver.h>
+#include <linux/moduleparam.h>
#include "mlx5_vdpa.h"
+int mlx5_vdpa_max_iotlb_entries = 2048;
+module_param_named(max_iotlb_entries, mlx5_vdpa_max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
static int alloc_pd(struct mlx5_vdpa_dev *dev, u32 *pdn, u16 uid)
{
struct mlx5_core_dev *mdev = dev->mdev;
@@ -229,7 +235,10 @@ int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey)
static int init_ctrl_vq(struct mlx5_vdpa_dev *mvdev)
{
- mvdev->cvq.iotlb = vhost_iotlb_alloc(0, 0);
+ if (mlx5_vdpa_max_iotlb_entries < 2)
+ return -EINVAL;
+
+ mvdev->cvq.iotlb = vhost_iotlb_alloc(mlx5_vdpa_max_iotlb_entries, 0);
if (!mvdev->cvq.iotlb)
return -ENOMEM;
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index 6b8e9da15ed0..ac4aa2e29fcc 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
@@ -573,6 +573,8 @@ static int cq_create(struct mlx5_vdpa_net *ndev, u16 idx, u32 num_ent)
vcq->mcq.set_ci_db = vcq->db.db;
vcq->mcq.arm_db = vcq->db.db + 1;
vcq->mcq.cqe_sz = 64;
+ vcq->mcq.comp = mlx5_vdpa_cq_comp;
+ vcq->cqe = num_ent;
err = cq_frag_buf_alloc(ndev, &vcq->buf, num_ent);
if (err)
@@ -612,10 +614,6 @@ static int cq_create(struct mlx5_vdpa_net *ndev, u16 idx, u32 num_ent)
if (err)
goto err_vec;
- vcq->mcq.comp = mlx5_vdpa_cq_comp;
- vcq->cqe = num_ent;
- vcq->mcq.set_ci_db = vcq->db.db;
- vcq->mcq.arm_db = vcq->db.db + 1;
mlx5_cq_arm(&mvq->cq.mcq, MLX5_CQ_DB_REQ_NOT, uar_page, mvq->cq.mcq.cons_index);
kfree(in);
return 0;
diff --git a/drivers/vdpa/vdpa.c b/drivers/vdpa/vdpa.c
index 8a372b51c21a..0fb00c8bde5e 100644
--- a/drivers/vdpa/vdpa.c
+++ b/drivers/vdpa/vdpa.c
@@ -67,57 +67,20 @@ static void vdpa_dev_remove(struct device *d)
static int vdpa_dev_match(struct device *dev, const struct device_driver *drv)
{
- struct vdpa_device *vdev = dev_to_vdpa(dev);
+ int ret;
/* Check override first, and if set, only use the named driver */
- if (vdev->driver_override)
- return strcmp(vdev->driver_override, drv->name) == 0;
+ ret = device_match_driver_override(dev, drv);
+ if (ret >= 0)
+ return ret;
/* Currently devices must be supported by all vDPA bus drivers */
return 1;
}
-static ssize_t driver_override_store(struct device *dev,
- struct device_attribute *attr,
- const char *buf, size_t count)
-{
- struct vdpa_device *vdev = dev_to_vdpa(dev);
- int ret;
-
- ret = driver_set_override(dev, &vdev->driver_override, buf, count);
- if (ret)
- return ret;
-
- return count;
-}
-
-static ssize_t driver_override_show(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- struct vdpa_device *vdev = dev_to_vdpa(dev);
- ssize_t len;
-
- device_lock(dev);
- len = sysfs_emit(buf, "%s\n", vdev->driver_override);
- device_unlock(dev);
-
- return len;
-}
-static DEVICE_ATTR_RW(driver_override);
-
-static struct attribute *vdpa_dev_attrs[] = {
- &dev_attr_driver_override.attr,
- NULL,
-};
-
-static const struct attribute_group vdpa_dev_group = {
- .attrs = vdpa_dev_attrs,
-};
-__ATTRIBUTE_GROUPS(vdpa_dev);
-
static const struct bus_type vdpa_bus = {
.name = "vdpa",
- .dev_groups = vdpa_dev_groups,
+ .driver_override = true,
.match = vdpa_dev_match,
.probe = vdpa_dev_probe,
.remove = vdpa_dev_remove,
@@ -132,7 +95,6 @@ static void vdpa_release_dev(struct device *d)
ops->free(vdev);
ida_free(&vdpa_index_ida, vdev->index);
- kfree(vdev->driver_override);
kfree(vdev);
}
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index 825690c6fed1..3bcec8c1e2ab 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -34,7 +34,7 @@ MODULE_PARM_DESC(batch_mapping, "Batched mapping 1 -Enable; 0 - Disable");
static int max_iotlb_entries = 2048;
module_param(max_iotlb_entries, int, 0444);
MODULE_PARM_DESC(max_iotlb_entries,
- "Maximum number of iotlb entries for each address space. 0 means unlimited. (default: 2048)");
+ "Maximum number of iotlb entries for each address space. (default: 2048)");
static bool use_va = true;
module_param(use_va, bool, 0444);
@@ -201,6 +201,8 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr,
if (!dev_attr->alloc_size)
return ERR_PTR(-EINVAL);
+ if (max_iotlb_entries < 2)
+ return ERR_PTR(-EINVAL);
if (config->mask & BIT_ULL(VDPA_ATTR_DEV_FEATURES)) {
if (config->device_features &
@@ -266,8 +268,10 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr,
for (i = 0; i < vdpasim->dev_attr.nas; i++) {
vhost_iotlb_init(&vdpasim->iommu[i], max_iotlb_entries, 0);
- vhost_iotlb_add_range(&vdpasim->iommu[i], 0, ULONG_MAX, 0,
- VHOST_MAP_RW);
+ ret = vhost_iotlb_add_range(&vdpasim->iommu[i], 0, ULONG_MAX,
+ 0, VHOST_MAP_RW);
+ if (ret)
+ goto err_iommu;
vdpasim->iommu_pt[i] = true;
}
diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c
index ec743bed361c..e8b837df328f 100644
--- a/drivers/vdpa/vdpa_user/iova_domain.c
+++ b/drivers/vdpa/vdpa_user/iova_domain.c
@@ -12,11 +12,17 @@
#include <linux/file.h>
#include <linux/anon_inodes.h>
#include <linux/highmem.h>
+#include <linux/moduleparam.h>
#include <linux/vmalloc.h>
#include <linux/vdpa.h>
#include "iova_domain.h"
+static int max_iotlb_entries = 2048;
+module_param(max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
static int vduse_iotlb_add_range(struct vduse_iova_domain *domain,
u64 start, u64 last,
u64 addr, unsigned int perm,
@@ -622,11 +628,14 @@ vduse_domain_create(unsigned long iova_limit, size_t bounce_size)
if (iova_limit <= bounce_size)
return NULL;
+ if (max_iotlb_entries <= 0)
+ return NULL;
+
domain = kzalloc(sizeof(*domain), GFP_KERNEL);
if (!domain)
return NULL;
- domain->iotlb = vhost_iotlb_alloc(0, 0);
+ domain->iotlb = vhost_iotlb_alloc(max_iotlb_entries, 0);
if (!domain->iotlb)
goto err_iotlb;
diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 4fbfc51407a3..fa391515bab2 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -1756,7 +1756,9 @@ static bool vduse_validate_config(struct vduse_dev_config *config)
sizeof(config->reserved)))
return false;
- if (config->vq_align > PAGE_SIZE)
+ if (config->vq_align < VRING_USED_ALIGN_SIZE ||
+ !is_power_of_2(config->vq_align) ||
+ config->vq_align > PAGE_SIZE)
return false;
if (config->config_size > PAGE_SIZE)
diff --git a/drivers/vhost/iotlb.c b/drivers/vhost/iotlb.c
index ea61330a3431..a6228dd24931 100644
--- a/drivers/vhost/iotlb.c
+++ b/drivers/vhost/iotlb.c
@@ -20,6 +20,14 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map,
rb, __u64, __subtree_last,
START, LAST, static inline, vhost_iotlb_itree);
+static void vhost_iotlb_map_unlink(struct vhost_iotlb *iotlb,
+ struct vhost_iotlb_map *map)
+{
+ vhost_iotlb_itree_remove(map, &iotlb->root);
+ list_del(&map->link);
+ iotlb->nmaps--;
+}
+
/**
* vhost_iotlb_map_free - remove a map node and free it
* @iotlb: the IOTLB
@@ -28,10 +36,8 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map,
void vhost_iotlb_map_free(struct vhost_iotlb *iotlb,
struct vhost_iotlb_map *map)
{
- vhost_iotlb_itree_remove(map, &iotlb->root);
- list_del(&map->link);
+ vhost_iotlb_map_unlink(iotlb, map);
kfree(map);
- iotlb->nmaps--;
}
EXPORT_SYMBOL_GPL(vhost_iotlb_map_free);
@@ -57,14 +63,25 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb,
if (last < start)
return -EFAULT;
+ if (!iotlb->limit)
+ return -EINVAL;
+
/* If the range being mapped is [0, ULONG_MAX], split it into two entries
* otherwise its size would overflow u64.
*/
if (start == 0 && last == ULONG_MAX) {
u64 mid = last / 2;
- int err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr,
- perm, opaque);
+ int err;
+
+ if (iotlb->limit < 2)
+ return -ENOSPC;
+ if (!(iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) &&
+ iotlb->nmaps > iotlb->limit - 2)
+ return -ENOSPC;
+
+ err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr,
+ perm, opaque);
if (err)
return err;
@@ -72,17 +89,19 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb,
start = mid + 1;
}
- if (iotlb->limit &&
- iotlb->nmaps == iotlb->limit &&
- iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) {
- map = list_first_entry(&iotlb->list, typeof(*map), link);
- vhost_iotlb_map_free(iotlb, map);
+ if (iotlb->nmaps >= iotlb->limit) {
+ if (iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) {
+ map = list_first_entry(&iotlb->list, typeof(*map), link);
+ vhost_iotlb_map_unlink(iotlb, map);
+ } else {
+ return -ENOSPC;
+ }
+ } else {
+ map = kmalloc(sizeof(*map), GFP_ATOMIC);
+ if (!map)
+ return -ENOMEM;
}
- map = kmalloc(sizeof(*map), GFP_ATOMIC);
- if (!map)
- return -ENOMEM;
-
map->start = start;
map->size = last - start + 1;
map->last = last;
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index c4c013960547..e64e3a663366 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1637,6 +1637,8 @@ static int vhost_net_set_features(struct vhost_net *n, u64 features)
if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
if (vhost_init_device_iotlb(&n->dev))
goto out_unlock;
+ } else {
+ vhost_clear_device_iotlb(&n->dev);
}
for (i = 0; i < VHOST_NET_VQ_MAX; ++i) {
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 789bd8da5505..a940718745de 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -34,6 +34,11 @@ enum {
#define VHOST_VDPA_DEV_MAX (1U << MINORBITS)
+static int max_iotlb_entries = 2048;
+module_param(max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
#define VHOST_VDPA_IOTLB_BUCKETS 16
struct vhost_vdpa_as {
@@ -112,12 +117,14 @@ static struct vhost_vdpa_as *vhost_vdpa_alloc_as(struct vhost_vdpa *v, u32 asid)
if (asid >= v->vdpa->nas)
return NULL;
+ if (max_iotlb_entries <= 0)
+ return NULL;
as = kmalloc(sizeof(*as), GFP_KERNEL);
if (!as)
return NULL;
- vhost_iotlb_init(&as->iotlb, 0, 0);
+ vhost_iotlb_init(&as->iotlb, max_iotlb_entries, 0);
as->id = asid;
hlist_add_head(&as->hash_link, head);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index f966fc094f12..9d5ea96f67ce 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -344,6 +344,17 @@ static void __vhost_vq_meta_reset(struct vhost_virtqueue *vq)
vq->meta_iotlb[j] = NULL;
}
+/* Caller must hold the virtqueue mutex. */
+static void vhost_vq_invalidate_access(struct vhost_virtqueue *vq)
+{
+ vq->desc = NULL;
+ vq->avail = NULL;
+ vq->used = NULL;
+ vq->log_used = false;
+ vq->log_addr = -1ull;
+ __vhost_vq_meta_reset(vq);
+}
+
static void vhost_vq_meta_reset(struct vhost_dev *d)
{
int i;
@@ -1135,6 +1146,9 @@ EXPORT_SYMBOL_GPL(vhost_dev_set_owner);
static struct vhost_iotlb *iotlb_alloc(void)
{
+ if (max_iotlb_entries <= 0)
+ return NULL;
+
return vhost_iotlb_alloc(max_iotlb_entries,
VHOST_IOTLB_FLAG_RETIRE);
}
@@ -1909,6 +1923,13 @@ int vq_meta_prefetch(struct vhost_virtqueue *vq)
{
unsigned int num = vq->num;
+ /*
+ * vhost_vq_invalidate_access() clears all three addresses together.
+ * A single zero address may be a valid GIOVA in IOTLB mode.
+ */
+ if (!vq->desc && !vq->avail && !vq->used)
+ return 0;
+
if (!vq->iotlb)
return 1;
@@ -1979,6 +2000,8 @@ static long vhost_set_memory(struct vhost_dev *d, struct vhost_memory __user *m)
return -EOPNOTSUPP;
if (mem.nregions > max_mem_regions)
return -E2BIG;
+ if (max_iotlb_entries <= 0)
+ return -EINVAL;
newmem = kvzalloc(struct_size(newmem, regions, mem.nregions),
GFP_KERNEL);
if (!newmem)
@@ -2276,11 +2299,48 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
}
EXPORT_SYMBOL_GPL(vhost_vring_ioctl);
+/* Caller must hold the device mutex. */
+void vhost_clear_device_iotlb(struct vhost_dev *d)
+{
+ struct vhost_iotlb *iotlb;
+ int i;
+
+ iotlb = d->iotlb;
+ if (!iotlb)
+ return;
+
+ vhost_dev_lock_vqs(d);
+
+ /*
+ * vhost_dev_lock_vqs() takes all VQ mutexes in index order. Drop the
+ * device-wide view while they are held, then clear each per-VQ view
+ * and its cached ring access before releasing the locks. Workers
+ * cannot observe a mixed address-space state during this handoff.
+ */
+ d->iotlb = NULL;
+
+ for (i = 0; i < d->nvqs; ++i) {
+ struct vhost_virtqueue *vq = d->vqs[i];
+
+ vq->iotlb = NULL;
+ vhost_vq_invalidate_access(vq);
+ }
+
+ vhost_dev_unlock_vqs(d);
+ vhost_clear_msg(d);
+ vhost_iotlb_free(iotlb);
+ wake_up_interruptible_poll(&d->wait, EPOLLIN | EPOLLRDNORM);
+}
+EXPORT_SYMBOL_GPL(vhost_clear_device_iotlb);
+
int vhost_init_device_iotlb(struct vhost_dev *d)
{
struct vhost_iotlb *niotlb, *oiotlb;
int i;
+ if (max_iotlb_entries <= 0)
+ return -EINVAL;
+
niotlb = iotlb_alloc();
if (!niotlb)
return -ENOMEM;
@@ -2293,7 +2353,10 @@ int vhost_init_device_iotlb(struct vhost_dev *d)
mutex_lock(&vq->mutex);
vq->iotlb = niotlb;
- __vhost_vq_meta_reset(vq);
+ if (oiotlb)
+ __vhost_vq_meta_reset(vq);
+ else
+ vhost_vq_invalidate_access(vq);
mutex_unlock(&vq->mutex);
}
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index ab704d84fb34..2bab74fe0807 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -264,6 +264,7 @@ ssize_t vhost_chr_read_iter(struct vhost_dev *dev, struct iov_iter *to,
int noblock);
ssize_t vhost_chr_write_iter(struct vhost_dev *dev,
struct iov_iter *from);
+void vhost_clear_device_iotlb(struct vhost_dev *d);
int vhost_init_device_iotlb(struct vhost_dev *d);
void vhost_iotlb_map_free(struct vhost_iotlb *iotlb,
diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
index 78cc66fbb3dd..65297f245b27 100644
--- a/drivers/vhost/vsock.c
+++ b/drivers/vhost/vsock.c
@@ -819,6 +819,8 @@ static int vhost_vsock_set_features(struct vhost_vsock *vsock, u64 features)
if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
if (vhost_init_device_iotlb(&vsock->dev))
goto err;
+ } else {
+ vhost_clear_device_iotlb(&vsock->dev);
}
vsock->seqpacket_allow = features & (1ULL << VIRTIO_VSOCK_F_SEQPACKET);
diff --git a/drivers/video/fbdev/core/fb_chrdev.c b/drivers/video/fbdev/core/fb_chrdev.c
index 4ebd16b7e3b8..54f926fb411b 100644
--- a/drivers/video/fbdev/core/fb_chrdev.c
+++ b/drivers/video/fbdev/core/fb_chrdev.c
@@ -85,11 +85,7 @@ static long do_fb_ioctl(struct fb_info *info, unsigned int cmd,
var.activate &= ~FB_ACTIVATE_KD_TEXT;
console_lock();
lock_fb_info(info);
- ret = fbcon_modechange_possible(info, &var);
- if (!ret)
- ret = fb_set_var(info, &var);
- if (!ret)
- fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL);
+ ret = fb_set_var_from_user(info, &var);
unlock_fb_info(info);
console_unlock();
if (!ret && copy_to_user(argp, &var, sizeof(var)))
diff --git a/drivers/video/fbdev/core/fbcon.c b/drivers/video/fbdev/core/fbcon.c
index 024cad45af93..5ce1aba2e761 100644
--- a/drivers/video/fbdev/core/fbcon.c
+++ b/drivers/video/fbdev/core/fbcon.c
@@ -2754,7 +2754,6 @@ void fbcon_update_vcs(struct fb_info *info, bool all)
else
fbcon_modechanged(info);
}
-EXPORT_SYMBOL(fbcon_update_vcs);
/* let fbcon check if it supports a new screen resolution */
int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *var)
@@ -2782,7 +2781,6 @@ int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *va
return 0;
}
-EXPORT_SYMBOL_GPL(fbcon_modechange_possible);
int fbcon_mode_deleted(struct fb_info *info,
struct fb_videomode *mode)
diff --git a/drivers/video/fbdev/core/fbmem.c b/drivers/video/fbdev/core/fbmem.c
index a689b390fce3..044a8f327cc2 100644
--- a/drivers/video/fbdev/core/fbmem.c
+++ b/drivers/video/fbdev/core/fbmem.c
@@ -341,6 +341,19 @@ fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var)
}
EXPORT_SYMBOL(fb_set_var);
+int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var)
+{
+ int ret = fbcon_modechange_possible(info, var);
+
+ if (!ret)
+ ret = fb_set_var(info, var);
+ if (!ret)
+ fbcon_update_vcs(info, var->activate & FB_ACTIVATE_ALL);
+
+ return ret;
+}
+EXPORT_SYMBOL(fb_set_var_from_user);
+
int
fb_blank(struct fb_info *info, int blank)
{
diff --git a/drivers/video/fbdev/core/fbsysfs.c b/drivers/video/fbdev/core/fbsysfs.c
index 10c3cfb959f4..572bf45d9d98 100644
--- a/drivers/video/fbdev/core/fbsysfs.c
+++ b/drivers/video/fbdev/core/fbsysfs.c
@@ -22,9 +22,7 @@ static int activate(struct fb_info *fb_info, struct fb_var_screeninfo *var)
var->activate |= FB_ACTIVATE_FORCE;
console_lock();
lock_fb_info(fb_info);
- err = fb_set_var(fb_info, var);
- if (!err)
- fbcon_update_vcs(fb_info, var->activate & FB_ACTIVATE_ALL);
+ err = fb_set_var_from_user(fb_info, var);
unlock_fb_info(fb_info);
console_unlock();
if (err)
diff --git a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
index 211f23648686..2b99a58ad675 100644
--- a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
+++ b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
@@ -1099,7 +1099,11 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
rg = omapfb_get_mem_region(ofbi->region);
- start = omapfb_get_region_paddr(ofbi);
+ if (ofbi->rotation_type == OMAP_DSS_ROT_VRFB)
+ start = rg->vrfb.paddr[0];
+ else
+ start = rg->paddr;
+
len = fix->smem_len;
DBG("user mmap region start %lx, len %d, off %lx\n", start, len,
@@ -1109,6 +1113,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
vma->vm_ops = &mmap_user_ops;
vma->vm_private_data = rg;
+ atomic_inc(&rg->map_count);
+
r = vm_iomap_memory(vma, start, len);
if (r)
goto error;
@@ -1121,7 +1127,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
return 0;
error:
- omapfb_put_mem_region(ofbi->region);
+ atomic_dec(&rg->map_count);
+ omapfb_put_mem_region(rg);
return r;
}
diff --git a/drivers/video/fbdev/ps3fb.c b/drivers/video/fbdev/ps3fb.c
index dbcda307f6a6..1376d19b19ae 100644
--- a/drivers/video/fbdev/ps3fb.c
+++ b/drivers/video/fbdev/ps3fb.c
@@ -29,7 +29,6 @@
#include <linux/freezer.h>
#include <linux/uaccess.h>
#include <linux/fb.h>
-#include <linux/fbcon.h>
#include <linux/init.h>
#include <asm/cell-regs.h>
@@ -830,9 +829,7 @@ static int ps3fb_ioctl(struct fb_info *info, unsigned int cmd,
/* Force, in case only special bits changed */
var.activate |= FB_ACTIVATE_FORCE;
par->new_mode_id = val;
- retval = fb_set_var(info, &var);
- if (!retval)
- fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL);
+ retval = fb_set_var_from_user(info, &var);
console_unlock();
}
break;
diff --git a/drivers/video/fbdev/sh_mobile_lcdcfb.c b/drivers/video/fbdev/sh_mobile_lcdcfb.c
index 6b37b188af31..84aa2086d5ae 100644
--- a/drivers/video/fbdev/sh_mobile_lcdcfb.c
+++ b/drivers/video/fbdev/sh_mobile_lcdcfb.c
@@ -15,7 +15,6 @@
#include <linux/ctype.h>
#include <linux/dma-mapping.h>
#include <linux/delay.h>
-#include <linux/fbcon.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/ioctl.h>
@@ -1769,11 +1768,9 @@ static void sh_mobile_fb_reconfig(struct fb_info *info)
var.height = ch->display.height;
var.activate = FB_ACTIVATE_NOW;
- if (fb_set_var(info, &var) < 0)
+ if (fb_set_var_from_user(info, &var) < 0)
/* Couldn't reconfigure, hopefully, can continue as before */
return;
-
- fbcon_update_vcs(info, true);
}
/*
diff --git a/drivers/virt/acrn/irqfd.c b/drivers/virt/acrn/irqfd.c
index 9994d818bb7e..1b1a33d702c1 100644
--- a/drivers/virt/acrn/irqfd.c
+++ b/drivers/virt/acrn/irqfd.c
@@ -16,7 +16,8 @@
#include "acrn_drv.h"
-static LIST_HEAD(acrn_irqfd_clients);
+/* Cleanup work has been queued; set via test_and_set_bit(). */
+#define HSM_IRQFD_FLAG_SHUTDOWN 0
/**
* struct hsm_irqfd - Properties of HSM irqfd
@@ -27,6 +28,7 @@ static LIST_HEAD(acrn_irqfd_clients);
* @list: Entry within &acrn_vm.irqfds of irqfds of a VM
* @pt: Structure for select/poll on the associated eventfd
* @msi: MSI data
+ * @flags: Internal lifecycle flags (HSM_IRQFD_FLAG_*)
*/
struct hsm_irqfd {
struct acrn_vm *vm;
@@ -36,6 +38,7 @@ struct hsm_irqfd {
struct list_head list;
poll_table pt;
struct acrn_msi_entry msi;
+ unsigned long flags;
};
static void acrn_irqfd_inject(struct hsm_irqfd *irqfd)
@@ -46,30 +49,29 @@ static void acrn_irqfd_inject(struct hsm_irqfd *irqfd)
irqfd->msi.msi_data);
}
-static void hsm_irqfd_shutdown(struct hsm_irqfd *irqfd)
+/* Queue the cleanup work at most once. Safe from atomic context. */
+static void hsm_irqfd_queue_shutdown(struct hsm_irqfd *irqfd)
{
- u64 cnt;
-
- lockdep_assert_held(&irqfd->vm->irqfds_lock);
-
- /* remove from wait queue */
- list_del_init(&irqfd->list);
- eventfd_ctx_remove_wait_queue(irqfd->eventfd, &irqfd->wait, &cnt);
- eventfd_ctx_put(irqfd->eventfd);
- kfree(irqfd);
+ if (!test_and_set_bit(HSM_IRQFD_FLAG_SHUTDOWN, &irqfd->flags))
+ queue_work(irqfd->vm->irqfd_wq, &irqfd->shutdown);
}
+/* Sole owner of @irqfd: unhook waitqueue, drop eventfd ref, free. */
static void hsm_irqfd_shutdown_work(struct work_struct *work)
{
- struct hsm_irqfd *irqfd;
- struct acrn_vm *vm;
+ struct hsm_irqfd *irqfd = container_of(work, struct hsm_irqfd,
+ shutdown);
+ struct acrn_vm *vm = irqfd->vm;
+ u64 cnt;
- irqfd = container_of(work, struct hsm_irqfd, shutdown);
- vm = irqfd->vm;
mutex_lock(&vm->irqfds_lock);
if (!list_empty(&irqfd->list))
- hsm_irqfd_shutdown(irqfd);
+ list_del_init(&irqfd->list);
mutex_unlock(&vm->irqfds_lock);
+
+ eventfd_ctx_remove_wait_queue(irqfd->eventfd, &irqfd->wait, &cnt);
+ eventfd_ctx_put(irqfd->eventfd);
+ kfree(irqfd);
}
/* Called with wqh->lock held and interrupts disabled */
@@ -78,17 +80,16 @@ static int hsm_irqfd_wakeup(wait_queue_entry_t *wait, unsigned int mode,
{
unsigned long poll_bits = (unsigned long)key;
struct hsm_irqfd *irqfd;
- struct acrn_vm *vm;
irqfd = container_of(wait, struct hsm_irqfd, wait);
- vm = irqfd->vm;
+
if (poll_bits & POLLIN)
/* An event has been signaled, inject an interrupt */
acrn_irqfd_inject(irqfd);
if (poll_bits & POLLHUP)
- /* Do shutdown work in thread to hold wqh->lock */
- queue_work(vm->irqfd_wq, &irqfd->shutdown);
+ /* Defer teardown to the cleanup work; can't sleep here. */
+ hsm_irqfd_queue_shutdown(irqfd);
return 0;
}
@@ -145,6 +146,12 @@ static int acrn_irqfd_assign(struct acrn_vm *vm, struct acrn_irqfd *args)
init_waitqueue_func_entry(&irqfd->wait, hsm_irqfd_wakeup);
init_poll_funcptr(&irqfd->pt, hsm_irqfd_poll_func);
+ /*
+ * Hold irqfds_lock across waitqueue install and list_add so the
+ * irqfd is not visible to deassign/deinit before its waitqueue
+ * entry is in place, and any racing EPOLLHUP cleanup work blocks
+ * on irqfds_lock until publication completes.
+ */
mutex_lock(&vm->irqfds_lock);
list_for_each_entry(tmp, &vm->irqfds, list) {
if (irqfd->eventfd != tmp->eventfd)
@@ -153,14 +160,12 @@ static int acrn_irqfd_assign(struct acrn_vm *vm, struct acrn_irqfd *args)
mutex_unlock(&vm->irqfds_lock);
goto fail;
}
- list_add_tail(&irqfd->list, &vm->irqfds);
- mutex_unlock(&vm->irqfds_lock);
- /* Check the pending event in this stage */
events = vfs_poll(fd_file(f), &irqfd->pt);
-
+ list_add_tail(&irqfd->list, &vm->irqfds);
if (events & EPOLLIN)
acrn_irqfd_inject(irqfd);
+ mutex_unlock(&vm->irqfds_lock);
fdput(f);
return 0;
@@ -187,13 +192,17 @@ static int acrn_irqfd_deassign(struct acrn_vm *vm,
mutex_lock(&vm->irqfds_lock);
list_for_each_entry_safe(irqfd, tmp, &vm->irqfds, list) {
if (irqfd->eventfd == eventfd) {
- hsm_irqfd_shutdown(irqfd);
+ list_del_init(&irqfd->list);
+ hsm_irqfd_queue_shutdown(irqfd);
break;
}
}
mutex_unlock(&vm->irqfds_lock);
eventfd_ctx_put(eventfd);
+ /* Wait for cleanup work to finish so the eventfd is fully detached. */
+ flush_workqueue(vm->irqfd_wq);
+
return 0;
}
@@ -226,9 +235,15 @@ void acrn_irqfd_deinit(struct acrn_vm *vm)
struct hsm_irqfd *irqfd, *next;
dev_dbg(acrn_dev.this_device, "VM %u irqfd deinit.\n", vm->vmid);
- destroy_workqueue(vm->irqfd_wq);
+
mutex_lock(&vm->irqfds_lock);
- list_for_each_entry_safe(irqfd, next, &vm->irqfds, list)
- hsm_irqfd_shutdown(irqfd);
+ list_for_each_entry_safe(irqfd, next, &vm->irqfds, list) {
+ list_del_init(&irqfd->list);
+ hsm_irqfd_queue_shutdown(irqfd);
+ }
mutex_unlock(&vm->irqfds_lock);
+
+ /* Drain all cleanup work before tearing the workqueue down. */
+ flush_workqueue(vm->irqfd_wq);
+ destroy_workqueue(vm->irqfd_wq);
}
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 1b193d08a6ba..148ce610ccb8 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -377,6 +377,34 @@ static void virtio_dev_remove(struct device *_d)
of_node_put(dev->dev.of_node);
}
+static void virtio_dev_shutdown(struct device *_d)
+{
+ struct virtio_device *dev = dev_to_virtio(_d);
+ struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+
+ /*
+ * Stop accesses to or from the device.
+ * We only need to do it if there's a driver - no accesses otherwise.
+ */
+ if (!drv)
+ return;
+
+ /*
+ * Some devices get wedged if you kick them after they are
+ * reset. Mark all vqs as broken to make sure we don't.
+ */
+ virtio_break_device(dev);
+ /*
+ * Guarantee that any callback will see vq->broken as true.
+ */
+ virtio_synchronize_cbs(dev);
+ /*
+ * As IOMMUs are reset on shutdown, this will block device access to memory.
+ * Some devices get wedged if this happens, so reset to make sure it does not.
+ */
+ dev->config->reset(dev);
+}
+
static const struct bus_type virtio_bus = {
.name = "virtio",
.match = virtio_dev_match,
@@ -384,6 +412,7 @@ static const struct bus_type virtio_bus = {
.uevent = virtio_uevent,
.probe = virtio_dev_probe,
.remove = virtio_dev_remove,
+ .shutdown = virtio_dev_shutdown,
};
int __register_virtio_driver(struct virtio_driver *driver, struct module *owner)
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index c0276979675d..2dfc815c54f6 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -1412,7 +1412,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq,
struct scatterlist *sg;
unsigned int i, n, c, descs_used, err_idx;
__le16 head_flags, flags;
- u16 head, id, prev, curr, avail_used_flags;
+ u16 head, id, prev, curr, avail_used_flags, unpub_flags;
int err;
START_USE(vq);
@@ -1535,15 +1535,30 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq,
curr = vq->free_head;
vq->packed.avail_used_flags = avail_used_flags;
+ unpub_flags = avail_used_flags ^ (1 << VRING_PACKED_DESC_F_AVAIL |
+ 1 << VRING_PACKED_DESC_F_USED);
for (n = 0; n < total_sg; n++) {
if (i == err_idx)
break;
+ /*
+ * The mapping loop made every descriptor but the head
+ * available. Stamp the previous wrap counter's AVAIL and USED
+ * bits on those, so that a later and shorter chain at this head
+ * does not leave one of them available beyond its own last
+ * descriptor. Marking them used instead would hand
+ * is_used_desc_packed() a completion we never made.
+ */
+ if (i != head)
+ desc[i].flags = cpu_to_le16(unpub_flags);
vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr]);
curr = vq->packed.desc_extra[curr].next;
i++;
- if (i >= vq->packed.vring.num)
+ if (i >= vq->packed.vring.num) {
i = 0;
+ unpub_flags ^= 1 << VRING_PACKED_DESC_F_AVAIL |
+ 1 << VRING_PACKED_DESC_F_USED;
+ }
}
END_USE(vq);
diff --git a/drivers/watchdog/at91sam9_wdt.c b/drivers/watchdog/at91sam9_wdt.c
index 2c6474cb858b..fe14f00662c5 100644
--- a/drivers/watchdog/at91sam9_wdt.c
+++ b/drivers/watchdog/at91sam9_wdt.c
@@ -242,7 +242,7 @@ static int at91_wdt_init(struct platform_device *pdev, struct at91wdt *wdt)
return 0;
out_stop_timer:
- del_timer(&wdt->timer);
+ timer_shutdown_sync(&wdt->timer);
return err;
}
@@ -378,7 +378,7 @@ static void at91wdt_remove(struct platform_device *pdev)
watchdog_unregister_device(&wdt->wdd);
pr_warn("I quit now, hardware will probably reboot!\n");
- del_timer(&wdt->timer);
+ timer_shutdown_sync(&wdt->timer);
}
#if defined(CONFIG_OF)
diff --git a/drivers/xen/pvcalls-front.c b/drivers/xen/pvcalls-front.c
index b72ee9379d77..d2bdd43703e8 100644
--- a/drivers/xen/pvcalls-front.c
+++ b/drivers/xen/pvcalls-front.c
@@ -32,6 +32,7 @@ struct pvcalls_bedata {
struct xen_pvcalls_front_ring ring;
grant_ref_t ref;
int irq;
+ bool disabled;
struct list_head socket_mappings;
spinlock_t socket_lock;
@@ -131,6 +132,20 @@ static inline int get_request(struct pvcalls_bedata *bedata, int *req_id)
return 0;
}
+/*
+ * Wait for the backend's response to req_id, or for the frontend to be
+ * disabled because the backend violated the wire protocol. Returns 0 once
+ * the response has arrived, or -EIO if the frontend was disabled.
+ */
+static int pvcalls_front_wait_rsp(struct pvcalls_bedata *bedata, u32 req_id)
+{
+ wait_event(bedata->inflight_req,
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled));
+
+ return READ_ONCE(bedata->disabled) ? -EIO : 0;
+}
+
static bool pvcalls_front_write_todo(struct sock_mapping *map)
{
struct pvcalls_data_intf *intf = map->active.ring;
@@ -168,7 +183,8 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
struct pvcalls_bedata *bedata;
struct xen_pvcalls_response *rsp;
uint8_t *src, *dst;
- int req_id = 0, more = 0, done = 0;
+ u32 req_id = 0;
+ int more = 0, done = 0;
if (dev == NULL)
return IRQ_HANDLED;
@@ -179,12 +195,31 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
pvcalls_exit();
return IRQ_HANDLED;
}
+ if (READ_ONCE(bedata->disabled)) {
+ pvcalls_exit();
+ return IRQ_HANDLED;
+ }
again:
while (RING_HAS_UNCONSUMED_RESPONSES(&bedata->ring)) {
rsp = RING_GET_RESPONSE(&bedata->ring, bedata->ring.rsp_cons);
req_id = rsp->req_id;
+ if (req_id >= PVCALLS_NR_RSP_PER_RING) {
+ /*
+ * The backend supplied a req_id that would index
+ * bedata->rsp[] out of bounds: a protocol violation
+ * from a malicious or buggy backend. Log once, stop
+ * trusting this backend and disable the frontend rather
+ * than silently dropping the response and continuing.
+ */
+ pr_err_once("pvcalls: backend sent out-of-range req_id %u, disabling frontend\n",
+ req_id);
+ WRITE_ONCE(bedata->disabled, true);
+ bedata->ring.rsp_cons++;
+ done = 1;
+ break;
+ }
if (rsp->cmd == PVCALLS_POLL) {
struct sock_mapping *map = (struct sock_mapping *)(uintptr_t)
rsp->u.poll.id;
@@ -217,7 +252,7 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
}
RING_FINAL_CHECK_FOR_RESPONSES(&bedata->ring, more);
- if (more)
+ if (more && !READ_ONCE(bedata->disabled))
goto again;
if (done)
wake_up(&bedata->inflight_req);
@@ -330,8 +365,11 @@ int pvcalls_front_socket(struct socket *sock)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit();
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -476,8 +514,11 @@ int pvcalls_front_connect(struct socket *sock, struct sockaddr *addr,
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -707,8 +748,11 @@ int pvcalls_front_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -756,8 +800,11 @@ int pvcalls_front_listen(struct socket *sock, int backlog)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -813,6 +860,14 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock, int flags)
}
}
+ if (READ_ONCE(bedata->disabled)) {
+ clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
+ (void *)&map->passive.flags);
+ wake_up(&map->passive.inflight_accept_req);
+ pvcalls_exit_sock(sock);
+ return -EIO;
+ }
+
map2 = kzalloc(sizeof(*map2), GFP_KERNEL);
if (map2 == NULL) {
clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
@@ -873,10 +928,18 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock, int flags)
}
if (wait_event_interruptible(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id)) {
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled))) {
pvcalls_exit_sock(sock);
return -EINTR;
}
+ if (READ_ONCE(bedata->disabled)) {
+ clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
+ (void *)&map->passive.flags);
+ wake_up(&map->passive.inflight_accept_req);
+ pvcalls_exit_sock(sock);
+ return -EIO;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -1045,7 +1108,8 @@ int pvcalls_front_release(struct socket *sock)
notify_remote_via_irq(bedata->irq);
wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled));
if (map->active_socket) {
/*
diff --git a/drivers/xen/xen-acpi-processor.c b/drivers/xen/xen-acpi-processor.c
index 520756159d3d..8d1860bd5d57 100644
--- a/drivers/xen/xen-acpi-processor.c
+++ b/drivers/xen/xen-acpi-processor.c
@@ -492,7 +492,7 @@ static void xen_acpi_processor_resume_worker(struct work_struct *dummy)
pr_info("ACPI data upload failed, error = %d\n", rc);
}
-static void xen_acpi_processor_resume(void)
+static void xen_acpi_processor_resume(void *data)
{
static DECLARE_WORK(wq, xen_acpi_processor_resume_worker);
@@ -506,10 +506,14 @@ static void xen_acpi_processor_resume(void)
schedule_work(&wq);
}
-static struct syscore_ops xap_syscore_ops = {
+static const struct syscore_ops xap_syscore_ops = {
.resume = xen_acpi_processor_resume,
};
+static struct syscore xap_syscore = {
+ .ops = &xap_syscore_ops,
+};
+
static int __init xen_acpi_processor_init(void)
{
int i;
@@ -560,7 +564,7 @@ static int __init xen_acpi_processor_init(void)
if (rc)
goto err_unregister;
- register_syscore_ops(&xap_syscore_ops);
+ register_syscore(&xap_syscore);
return 0;
err_unregister:
@@ -577,7 +581,7 @@ static void __exit xen_acpi_processor_exit(void)
{
int i;
- unregister_syscore_ops(&xap_syscore_ops);
+ unregister_syscore(&xap_syscore);
bitmap_free(acpi_ids_done);
bitmap_free(acpi_id_present);
bitmap_free(acpi_id_cst_present);
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index 5bf1e5caaa62..b346d4f3eb9d 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -96,7 +96,7 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq)
subreq->transferred += total;
subreq->error = err;
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
}
/**
diff --git a/fs/afs/addr_list.c b/fs/afs/addr_list.c
index f2361a135107..48d68f5aa7b4 100644
--- a/fs/afs/addr_list.c
+++ b/fs/afs/addr_list.c
@@ -298,8 +298,8 @@ int afs_merge_fs_addr4(struct afs_net *net, struct afs_addr_list *alist,
srx.transport.sin.sin_addr.s_addr = xdr;
peer = rxrpc_kernel_lookup_peer(net->socket, &srx, GFP_KERNEL);
- if (!peer)
- return -ENOMEM;
+ if (IS_ERR(peer))
+ return PTR_ERR(peer);
for (i = 0; i < alist->nr_ipv4; i++) {
if (peer == alist->addrs[i].peer) {
@@ -342,8 +342,8 @@ int afs_merge_fs_addr6(struct afs_net *net, struct afs_addr_list *alist,
memcpy(&srx.transport.sin6.sin6_addr, xdr, 16);
peer = rxrpc_kernel_lookup_peer(net->socket, &srx, GFP_KERNEL);
- if (!peer)
- return -ENOMEM;
+ if (IS_ERR(peer))
+ return PTR_ERR(peer);
for (i = alist->nr_ipv4; i < alist->nr_addrs; i++) {
if (peer == alist->addrs[i].peer) {
diff --git a/fs/afs/file.c b/fs/afs/file.c
index 56248a078bca..f717168da4ab 100644
--- a/fs/afs/file.c
+++ b/fs/afs/file.c
@@ -247,7 +247,7 @@ static void afs_fetch_data_notify(struct afs_operation *op)
if (req->pos + req->actual_len >= req->file_size)
__set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
subreq->error = error;
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
req->subreq = NULL;
} else if (req->done) {
req->done(req);
@@ -304,7 +304,7 @@ int afs_fetch_data(struct afs_vnode *vnode, struct afs_read *req)
if (IS_ERR(op)) {
if (req->subreq) {
req->subreq->error = PTR_ERR(op);
- netfs_read_subreq_terminated(req->subreq, false);
+ netfs_read_subreq_terminated(req->subreq);
}
return PTR_ERR(op);
}
@@ -325,7 +325,7 @@ static void afs_read_worker(struct work_struct *work)
fsreq = afs_alloc_read(GFP_NOFS);
if (!fsreq) {
subreq->error = -ENOMEM;
- return netfs_read_subreq_terminated(subreq, false);
+ return netfs_read_subreq_terminated(subreq);
}
fsreq->subreq = subreq;
diff --git a/fs/afs/fsclient.c b/fs/afs/fsclient.c
index 1f8a7878e9d9..b171078773f0 100644
--- a/fs/afs/fsclient.c
+++ b/fs/afs/fsclient.c
@@ -352,7 +352,7 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call)
ret = afs_extract_data(call, true);
if (req->subreq) {
req->subreq->transferred += count_before - call->iov_len;
- netfs_read_subreq_progress(req->subreq, false);
+ netfs_read_subreq_progress(req->subreq);
}
if (ret < 0)
return ret;
diff --git a/fs/afs/inode.c b/fs/afs/inode.c
index c4f92aaa2527..9b9605971ec5 100644
--- a/fs/afs/inode.c
+++ b/fs/afs/inode.c
@@ -647,6 +647,7 @@ void afs_evict_inode(struct inode *inode)
ASSERTCMP(inode->i_ino, ==, vnode->fid.vnode);
+ flush_delayed_work(&vnode->lock_work);
netfs_wait_for_outstanding_io(inode);
truncate_inode_pages_final(&inode->i_data);
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index beaedbb3a01c..a1090863ea46 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -1336,6 +1336,7 @@ extern int __net_init afs_open_socket(struct afs_net *);
extern void __net_exit afs_close_socket(struct afs_net *);
extern void afs_charge_preallocation(struct work_struct *);
extern void afs_put_call(struct afs_call *);
+extern struct afs_call *afs_get_call(struct afs_call *call, enum afs_call_trace why);
void afs_deferred_put_call(struct afs_call *call);
void afs_make_call(struct afs_call *call, gfp_t gfp);
void afs_wait_for_call_to_complete(struct afs_call *call);
@@ -1362,7 +1363,7 @@ static inline void afs_make_op_call(struct afs_operation *op, struct afs_call *c
{
struct afs_addr_list *alist = op->estate->addresses;
- op->call = call;
+ op->call = afs_get_call(call, afs_call_trace_get);
op->type = call->type;
call->op = op;
call->key = op->key;
@@ -1370,6 +1371,7 @@ static inline void afs_make_op_call(struct afs_operation *op, struct afs_call *c
call->peer = rxrpc_kernel_get_peer(alist->addrs[op->addr_index].peer);
call->service_id = op->server->service_id;
afs_make_call(call, gfp);
+ afs_put_call(call);
}
static inline void afs_extract_begin(struct afs_call *call, void *buf, size_t size)
diff --git a/fs/afs/rxrpc.c b/fs/afs/rxrpc.c
index c35c2332c5b7..12bafd03fd59 100644
--- a/fs/afs/rxrpc.c
+++ b/fs/afs/rxrpc.c
@@ -247,8 +247,8 @@ void afs_deferred_put_call(struct afs_call *call)
schedule_work(&call->free_work);
}
-static struct afs_call *afs_get_call(struct afs_call *call,
- enum afs_call_trace why)
+struct afs_call *afs_get_call(struct afs_call *call,
+ enum afs_call_trace why)
{
int r;
diff --git a/fs/afs/super.c b/fs/afs/super.c
index 89787b0e3c13..32bb1b7a498d 100644
--- a/fs/afs/super.c
+++ b/fs/afs/super.c
@@ -660,7 +660,6 @@ static void afs_i_init_once(void *_vnode)
INIT_LIST_HEAD(&vnode->wb_keys);
INIT_LIST_HEAD(&vnode->pending_locks);
INIT_LIST_HEAD(&vnode->granted_locks);
- INIT_DELAYED_WORK(&vnode->lock_work, afs_lock_work);
INIT_LIST_HEAD(&vnode->cb_mmap_link);
seqlock_init(&vnode->cb_lock);
}
@@ -692,6 +691,7 @@ static struct inode *afs_alloc_inode(struct super_block *sb)
init_rwsem(&vnode->rmdir_lock);
INIT_WORK(&vnode->cb_work, afs_invalidate_mmap_work);
+ INIT_DELAYED_WORK(&vnode->lock_work, afs_lock_work);
_leave(" = %p", &vnode->netfs.inode);
return &vnode->netfs.inode;
diff --git a/fs/afs/yfsclient.c b/fs/afs/yfsclient.c
index 362845f9aaae..667b78b48543 100644
--- a/fs/afs/yfsclient.c
+++ b/fs/afs/yfsclient.c
@@ -398,7 +398,7 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call)
ret = afs_extract_data(call, true);
if (req->subreq) {
req->subreq->transferred += count_before - call->iov_len;
- netfs_read_subreq_progress(req->subreq, false);
+ netfs_read_subreq_progress(req->subreq);
}
if (ret < 0)
return ret;
diff --git a/fs/btrfs/acl.c b/fs/btrfs/acl.c
index e0ba00d64ea0..d1c67faacc18 100644
--- a/fs/btrfs/acl.c
+++ b/fs/btrfs/acl.c
@@ -14,6 +14,7 @@
#include "ctree.h"
#include "xattr.h"
#include "acl.h"
+#include "btrfs_inode.h"
struct posix_acl *btrfs_get_acl(struct inode *inode, int type, bool rcu)
{
@@ -116,6 +117,9 @@ int btrfs_set_acl(struct mnt_idmap *idmap, struct dentry *dentry,
struct inode *inode = d_inode(dentry);
umode_t old_mode = inode->i_mode;
+ if (btrfs_root_readonly(BTRFS_I(inode)->root))
+ return -EROFS;
+
if (type == ACL_TYPE_ACCESS && acl) {
ret = posix_acl_update_mode(idmap, inode,
&inode->i_mode, &acl);
diff --git a/fs/btrfs/bio.c b/fs/btrfs/bio.c
index 5fd0b39d8c70..6225f9d045ef 100644
--- a/fs/btrfs/bio.c
+++ b/fs/btrfs/bio.c
@@ -812,9 +812,15 @@ int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start,
struct bio bio;
int ret = 0;
- ASSERT(!(fs_info->sb->s_flags & SB_RDONLY));
BUG_ON(!mirror_num);
+ /*
+ * The fs either mounted RO or hit critical errors, no need
+ * to continue repairing.
+ */
+ if (unlikely(sb_rdonly(fs_info->sb)))
+ return 0;
+
if (btrfs_repair_one_zone(fs_info, logical))
return 0;
diff --git a/fs/btrfs/block-rsv.c b/fs/btrfs/block-rsv.c
index a578bd3b32bb..f111500a2c46 100644
--- a/fs/btrfs/block-rsv.c
+++ b/fs/btrfs/block-rsv.c
@@ -326,10 +326,25 @@ void btrfs_block_rsv_add_bytes(struct btrfs_block_rsv *block_rsv,
void btrfs_update_global_block_rsv(struct btrfs_fs_info *fs_info)
{
struct btrfs_block_rsv *block_rsv = &fs_info->global_block_rsv;
- struct btrfs_space_info *sinfo = block_rsv->space_info;
+ struct btrfs_space_info *sinfo;
struct btrfs_root *root, *tmp;
- u64 num_bytes = btrfs_root_used(&fs_info->tree_root->root_item);
unsigned int min_items = 1;
+ u64 num_bytes;
+
+ /*
+ * A full read-only mount (rescue options) cannot start transactions,
+ * so the global reserve is never consumed. Mark it as full and skip
+ * the accounting.
+ */
+ if (btrfs_is_full_ro(fs_info)) {
+ spin_lock(&block_rsv->lock);
+ block_rsv->full = true;
+ spin_unlock(&block_rsv->lock);
+ return;
+ }
+
+ sinfo = block_rsv->space_info;
+ num_bytes = btrfs_root_used(&fs_info->tree_root->root_item);
/*
* The global block rsv is based on the size of the extent tree, the
diff --git a/fs/btrfs/defrag.c b/fs/btrfs/defrag.c
index d8e8b87c8bc7..727a016b2025 100644
--- a/fs/btrfs/defrag.c
+++ b/fs/btrfs/defrag.c
@@ -1153,26 +1153,21 @@ static_assert(PAGE_ALIGNED(CLUSTER_SIZE));
*
* - Extent bits are locked
*/
-static int defrag_one_locked_target(struct btrfs_inode *inode,
- struct defrag_target_range *target,
- struct folio **folios, int nr_pages,
- struct extent_state **cached_state)
+static void defrag_one_locked_target(struct btrfs_inode *inode,
+ struct defrag_target_range *target,
+ struct folio **folios, int nr_pages,
+ struct extent_state **cached_state)
{
struct btrfs_fs_info *fs_info = inode->root->fs_info;
- struct extent_changeset *data_reserved = NULL;
const u64 start = target->start;
const u64 len = target->len;
unsigned long last_index = (start + len - 1) >> PAGE_SHIFT;
unsigned long start_index = start >> PAGE_SHIFT;
unsigned long first_index = folios[0]->index;
- int ret = 0;
int i;
ASSERT(last_index - first_index + 1 <= nr_pages);
- ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len);
- if (ret < 0)
- return ret;
clear_extent_bit(&inode->io_tree, start, start + len - 1,
EXTENT_DELALLOC | EXTENT_DO_ACCOUNTING |
EXTENT_DEFRAG, cached_state);
@@ -1184,10 +1179,6 @@ static int defrag_one_locked_target(struct btrfs_inode *inode,
folio_clear_checked(folios[i]);
btrfs_folio_clamp_set_dirty(fs_info, folios[i], start, len);
}
- btrfs_delalloc_release_extents(inode, len);
- extent_changeset_free(data_reserved);
-
- return ret;
}
static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
@@ -1203,6 +1194,8 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
u64 last_index = (start + len - 1) >> PAGE_SHIFT;
u64 start_index = start >> PAGE_SHIFT;
unsigned int nr_pages = last_index - start_index + 1;
+ struct extent_changeset *data_reserved = NULL;
+ u64 last_defrag_end = start;
int ret = 0;
int i;
@@ -1213,6 +1206,22 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
if (!folios)
return -ENOMEM;
+ /*
+ * Reserve delalloc space before locking the range and before locking
+ * and dirtying any folios - otherwise we could deadlock, for example
+ * after defrag of one range we dirty folios and keep them locked when
+ * we move to the next range, so reserving delalloc space right before
+ * each range could trigger flushing of delalloc and deadlock on the
+ * extent lock or trigger a transaction commit with flushoncommit, which
+ * can either deadlock on the lock of a folio made dirty in the previous
+ * range or the extent lock.
+ */
+ ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len);
+ if (ret < 0) {
+ kfree(folios);
+ return ret;
+ }
+
/* Prepare all pages */
for (i = 0; i < nr_pages; i++) {
folios[i] = defrag_prepare_one_folio(inode, start_index + i);
@@ -1243,10 +1252,11 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
goto btrfs_unlock_extent;
list_for_each_entry(entry, &target_list, list) {
- ret = defrag_one_locked_target(inode, entry, folios, nr_pages,
- &cached_state);
- if (ret < 0)
- break;
+ defrag_one_locked_target(inode, entry, folios, nr_pages, &cached_state);
+ if (entry->start > last_defrag_end)
+ btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end,
+ entry->start - last_defrag_end, true);
+ last_defrag_end = entry->start + entry->len;
}
list_for_each_entry_safe(entry, tmp, &target_list, list) {
@@ -1263,6 +1273,12 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
folio_put(folios[i]);
}
kfree(folios);
+ btrfs_delalloc_release_extents(inode, len);
+ if (last_defrag_end < start + len)
+ btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end,
+ start + len - last_defrag_end, true);
+ extent_changeset_free(data_reserved);
+
return ret;
}
diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
index cab94d141f66..55d2a3089ac1 100644
--- a/fs/btrfs/delayed-ref.c
+++ b/fs/btrfs/delayed-ref.c
@@ -837,9 +837,13 @@ static void init_delayed_ref_head(struct btrfs_delayed_ref_head *head_ref,
}
/*
- * helper function to actually insert a head node into the rbtree.
- * this does all the dirty work in terms of maintaining the correct
- * overall modification count.
+ * Helper function to actually insert a head node into the rbtree. This does all
+ * the dirty work in terms of maintaining the correct overall modification
+ * count.
+ *
+ * The caller is responsible for calling kfree() on @qrecord. More specifically,
+ * if this function reports that it did not insert it as noted in
+ * @qrecord_inserted_ret, then it's safe to call kfree() on it.
*
* Returns an error pointer in case of an error.
*/
@@ -852,12 +856,25 @@ add_delayed_ref_head(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info = trans->fs_info;
struct btrfs_delayed_ref_head *existing;
struct btrfs_delayed_ref_root *delayed_refs;
- bool qrecord_inserted = false;
+
+ /*
+ * If 'qrecord_inserted_ret' is provided, then the first thing we need
+ * to do is to initialize it to false just in case we have an exit
+ * before trying to insert the record.
+ */
+ if (qrecord_inserted_ret)
+ *qrecord_inserted_ret = false;
delayed_refs = &trans->transaction->delayed_refs;
/* Record qgroup extent info if provided */
if (qrecord) {
+ /*
+ * Setting 'qrecord' but not 'qrecord_inserted_ret' will likely
+ * result in a memory leakage.
+ */
+ ASSERT(qrecord_inserted_ret != NULL);
+
int ret;
ret = btrfs_qgroup_trace_extent_nolock(fs_info, delayed_refs, qrecord);
@@ -865,12 +882,10 @@ add_delayed_ref_head(struct btrfs_trans_handle *trans,
/* Clean up if insertion fails or item exists. */
xa_release(&delayed_refs->dirty_extents,
qrecord->bytenr >> fs_info->sectorsize_bits);
- /* Caller responsible for freeing qrecord on error. */
if (ret < 0)
return ERR_PTR(ret);
- kfree(qrecord);
- } else {
- qrecord_inserted = true;
+ } else if (qrecord_inserted_ret) {
+ *qrecord_inserted_ret = true;
}
}
@@ -902,8 +917,6 @@ add_delayed_ref_head(struct btrfs_trans_handle *trans,
delayed_refs->num_heads_ready++;
atomic_inc(&delayed_refs->num_entries);
}
- if (qrecord_inserted_ret)
- *qrecord_inserted_ret = qrecord_inserted;
return head_ref;
}
@@ -1075,6 +1088,8 @@ static int add_delayed_ref(struct btrfs_trans_handle *trans,
if (qrecord_inserted)
return btrfs_qgroup_trace_extent_post(trans, record);
+
+ kfree(record);
return 0;
free_record:
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index b0035d13286a..d1f245ef1911 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -272,14 +272,15 @@ blk_status_t btree_csum_one_bio(struct btrfs_bio *bbio)
return BLK_STS_IOERR;
/*
- * If an extent_buffer is marked as EXTENT_BUFFER_ZONED_ZEROOUT, don't
- * checksum it but zero-out its content. This is done to preserve
- * ordering of I/O without unnecessarily writing out data.
+ * An extent_buffer marked EXTENT_BUFFER_ZONED_ZEROOUT is written out as
+ * zeros to preserve ordering of I/O without persisting the now
+ * unnecessary block. The bio is fed from the shared zero page (see
+ * write_one_eb()), so there is nothing to checksum here. Crucially, the
+ * buffer's own content is left intact: it may still be referenced, e.g.
+ * btrfs_free_tree_block() reads its header to add a delayed reference.
*/
- if (test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags)) {
- memzero_extent_buffer(eb, 0, eb->len);
+ if (test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags))
return BLK_STS_OK;
- }
if (WARN_ON_ONCE(found_start != eb->start))
return BLK_STS_IOERR;
@@ -3169,15 +3170,6 @@ int btrfs_check_features(struct btrfs_fs_info *fs_info, bool is_rw_mount)
return 0;
}
-static bool fs_is_full_ro(const struct btrfs_fs_info *fs_info)
-{
- if (!sb_rdonly(fs_info->sb))
- return false;
- if (unlikely(fs_info->mount_opt & BTRFS_MOUNT_FULL_RO_MASK))
- return true;
- return false;
-}
-
int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_devices)
{
u32 sectorsize;
@@ -3287,7 +3279,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
WRITE_ONCE(fs_info->fs_error, -EUCLEAN);
/* If the fs has any rescue options, no transaction is allowed. */
- if (fs_is_full_ro(fs_info))
+ if (btrfs_is_full_ro(fs_info))
WRITE_ONCE(fs_info->fs_error, -EROFS);
/* Set up fs_info before parsing mount options */
@@ -4693,6 +4685,7 @@ static void btrfs_destroy_marked_extents(struct btrfs_fs_info *fs_info,
free_extent_buffer_stale(eb);
}
}
+ extent_io_tree_release(dirty_pages);
}
static void btrfs_destroy_pinned_extent(struct btrfs_fs_info *fs_info,
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 399c7e10bc14..a821ea34674f 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -1750,32 +1750,36 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans,
struct btrfs_delayed_extent_op *extent_op,
bool insert_reserved)
{
+ struct btrfs_fs_info *fs_info = trans->fs_info;
int ret = 0;
if (TRANS_ABORTED(trans)) {
if (insert_reserved) {
- btrfs_pin_extent(trans, node->bytenr, node->num_bytes, 1);
- free_head_ref_squota_rsv(trans->fs_info, href);
+ btrfs_pin_extent(trans, node->bytenr, node->num_bytes);
+ free_head_ref_squota_rsv(fs_info, href);
}
return 0;
}
if (node->type == BTRFS_TREE_BLOCK_REF_KEY ||
- node->type == BTRFS_SHARED_BLOCK_REF_KEY)
+ node->type == BTRFS_SHARED_BLOCK_REF_KEY) {
ret = run_delayed_tree_ref(trans, href, node, extent_op,
insert_reserved);
- else if (node->type == BTRFS_EXTENT_DATA_REF_KEY ||
- node->type == BTRFS_SHARED_DATA_REF_KEY)
+ } else if (node->type == BTRFS_EXTENT_DATA_REF_KEY ||
+ node->type == BTRFS_SHARED_DATA_REF_KEY) {
ret = run_delayed_data_ref(trans, href, node, extent_op,
insert_reserved);
- else if (node->type == BTRFS_EXTENT_OWNER_REF_KEY)
+ } else if (node->type == BTRFS_EXTENT_OWNER_REF_KEY) {
ret = 0;
- else
- BUG();
+ } else {
+ ret = -EUCLEAN;
+ btrfs_err(fs_info, "unexpected delayed ref node type: %u", node->type);
+ }
+
if (ret && insert_reserved)
- btrfs_pin_extent(trans, node->bytenr, node->num_bytes, 1);
+ btrfs_pin_extent(trans, node->bytenr, node->num_bytes);
if (ret < 0)
- btrfs_err(trans->fs_info,
+ btrfs_err(fs_info,
"failed to run delayed ref for logical %llu num_bytes %llu type %u action %u ref_mod %d: %d",
node->bytenr, node->num_bytes, node->type,
node->action, node->ref_mod, ret);
@@ -1914,7 +1918,7 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans,
spin_unlock(&delayed_refs->lock);
if (head->must_insert_reserved) {
- btrfs_pin_extent(trans, head->bytenr, head->num_bytes, 1);
+ btrfs_pin_extent(trans, head->bytenr, head->num_bytes);
if (head->is_data) {
struct btrfs_root *csum_root;
@@ -2612,15 +2616,14 @@ static int pin_down_extent(struct btrfs_trans_handle *trans,
return 0;
}
-int btrfs_pin_extent(struct btrfs_trans_handle *trans,
- u64 bytenr, u64 num_bytes, int reserved)
+int btrfs_pin_extent(struct btrfs_trans_handle *trans, u64 bytenr, u64 num_bytes)
{
struct btrfs_block_group *cache;
cache = btrfs_lookup_block_group(trans->fs_info, bytenr);
BUG_ON(!cache); /* Logic error */
- pin_down_extent(trans, cache, bytenr, num_bytes, reserved);
+ pin_down_extent(trans, cache, bytenr, num_bytes, 1);
btrfs_put_block_group(cache);
return 0;
@@ -3531,7 +3534,7 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans, struct btrfs_ref *ref)
* tree, just update pinning info and exit early.
*/
if (ref->ref_root == BTRFS_TREE_LOG_OBJECTID) {
- btrfs_pin_extent(trans, ref->bytenr, ref->num_bytes, 1);
+ btrfs_pin_extent(trans, ref->bytenr, ref->num_bytes);
ret = 0;
} else if (ref->type == BTRFS_REF_METADATA) {
ret = btrfs_add_delayed_tree_ref(trans, ref, NULL);
@@ -5006,7 +5009,7 @@ int btrfs_alloc_logged_file_extent(struct btrfs_trans_handle *trans,
ret = alloc_reserved_file_extent(trans, 0, root_objectid, 0, owner,
offset, ins, 1, root_objectid);
if (ret)
- btrfs_pin_extent(trans, ins->objectid, ins->offset, 1);
+ btrfs_pin_extent(trans, ins->objectid, ins->offset);
ret = btrfs_record_squota_delta(fs_info, &delta);
btrfs_put_block_group(block_group);
return ret;
diff --git a/fs/btrfs/extent-tree.h b/fs/btrfs/extent-tree.h
index 82c054dbc369..4eef608c82f0 100644
--- a/fs/btrfs/extent-tree.h
+++ b/fs/btrfs/extent-tree.h
@@ -110,8 +110,7 @@ int btrfs_lookup_extent_info(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info, u64 bytenr,
u64 offset, int metadata, u64 *refs, u64 *flags,
u64 *owner_root);
-int btrfs_pin_extent(struct btrfs_trans_handle *trans, u64 bytenr, u64 num,
- int reserved);
+int btrfs_pin_extent(struct btrfs_trans_handle *trans, u64 bytenr, u64 num);
int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,
const struct extent_buffer *eb);
int btrfs_exclude_logged_extents(struct extent_buffer *eb);
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 8753ce416c5c..dbb81c58a2e3 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -1839,18 +1839,21 @@ static void end_bbio_meta_write(struct btrfs_bio *bbio)
struct extent_buffer *eb = bbio->private;
struct btrfs_fs_info *fs_info = eb->fs_info;
bool uptodate = !bbio->bio.bi_status;
- struct folio_iter fi;
u32 bio_offset = 0;
if (!uptodate)
set_btree_ioerr(eb);
- bio_for_each_folio_all(fi, &bbio->bio) {
- u64 start = eb->start + bio_offset;
- struct folio *folio = fi.folio;
- u32 len = fi.length;
+ /*
+ * Clear writeback on the buffer's own folios. The bio may carry the
+ * shared zero page instead (EXTENT_BUFFER_ZONED_ZEROOUT), so iterate
+ * the extent buffer folios rather than the bio folios.
+ */
+ for (int i = 0; i < num_extent_folios(eb); i++) {
+ u32 len = min(eb->len - bio_offset, (u32)eb->folio_size);
- btrfs_folio_clear_writeback(fs_info, folio, start, len);
+ btrfs_folio_clear_writeback(fs_info, eb->folios[i],
+ eb->start + bio_offset, len);
bio_offset += len;
}
@@ -1895,7 +1898,8 @@ static noinline_for_stack void write_one_eb(struct extent_buffer *eb,
struct btrfs_fs_info *fs_info = eb->fs_info;
struct btrfs_bio *bbio;
- prepare_eb_write(eb);
+ if (!test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags))
+ prepare_eb_write(eb);
bbio = btrfs_bio_alloc(INLINE_EXTENT_BUFFER_PAGES,
REQ_OP_WRITE | REQ_META | wbc_to_write_flags(wbc),
@@ -1916,9 +1920,23 @@ static noinline_for_stack void write_one_eb(struct extent_buffer *eb,
folio_clear_dirty_for_io(folio);
wbc->nr_to_write--;
}
- ret = bio_add_folio(&bbio->bio, folio, eb->len,
- eb->start - folio_pos(folio));
- ASSERT(ret);
+ if (test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags)) {
+ u32 off = 0;
+
+ while (off < eb->len) {
+ u32 add = min_t(u32, PAGE_SIZE, eb->len - off);
+
+ ret = bio_add_folio(&bbio->bio,
+ page_folio(ZERO_PAGE(0)),
+ add, 0);
+ ASSERT(ret);
+ off += add;
+ }
+ } else {
+ ret = bio_add_folio(&bbio->bio, folio, eb->len,
+ eb->start - folio_pos(folio));
+ ASSERT(ret);
+ }
wbc_account_cgroup_owner(wbc, folio, eb->len);
folio_unlock(folio);
} else {
@@ -1931,8 +1949,24 @@ static noinline_for_stack void write_one_eb(struct extent_buffer *eb,
folio_lock(folio);
folio_clear_dirty_for_io(folio);
folio_start_writeback(folio);
- ret = bio_add_folio(&bbio->bio, folio, eb->folio_size, 0);
- ASSERT(ret);
+ if (test_bit(EXTENT_BUFFER_ZONED_ZEROOUT, &eb->bflags)) {
+ u32 off = 0;
+
+ while (off < eb->folio_size) {
+ u32 add = min_t(u32, PAGE_SIZE,
+ eb->folio_size - off);
+
+ ret = bio_add_folio(&bbio->bio,
+ page_folio(ZERO_PAGE(0)),
+ add, 0);
+ ASSERT(ret);
+ off += add;
+ }
+ } else {
+ ret = bio_add_folio(&bbio->bio, folio,
+ eb->folio_size, 0);
+ ASSERT(ret);
+ }
wbc_account_cgroup_owner(wbc, folio, eb->folio_size);
wbc->nr_to_write -= folio_nr_pages(folio);
folio_unlock(folio);
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 5627e2f7dd64..c885c9a22109 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -969,70 +969,64 @@ static noinline int prepare_pages(struct inode *inode, struct page **pages,
/*
* This function locks the extent and properly waits for data=ordered extents
- * to finish before allowing the pages to be modified if need.
+ * to finish before allowing the pages to be modified.
*
* The return value:
- * 1 - the extent is locked
- * 0 - the extent is not locked, and everything is OK
+ * 0 - the extent is locked
* -EAGAIN - need re-prepare the pages
* the other < 0 number - Something wrong happens
*/
static noinline int
-lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages,
- size_t num_pages, loff_t pos,
- size_t write_bytes,
- u64 *lockstart, u64 *lockend, bool nowait,
- struct extent_state **cached_state)
+lock_and_cleanup_extent(struct btrfs_inode *inode, struct page **pages,
+ size_t num_pages, loff_t pos,
+ size_t write_bytes,
+ u64 *lockstart, u64 *lockend, bool nowait,
+ struct extent_state **cached_state)
{
struct btrfs_fs_info *fs_info = inode->root->fs_info;
+ struct btrfs_ordered_extent *ordered;
u64 start_pos;
u64 last_pos;
int i;
- int ret = 0;
start_pos = round_down(pos, fs_info->sectorsize);
last_pos = round_up(pos + write_bytes, fs_info->sectorsize) - 1;
- if (start_pos < inode->vfs_inode.i_size) {
- struct btrfs_ordered_extent *ordered;
-
- if (nowait) {
- if (!btrfs_try_lock_extent(&inode->io_tree, start_pos, last_pos,
- cached_state)) {
- for (i = 0; i < num_pages; i++) {
- unlock_page(pages[i]);
- put_page(pages[i]);
- pages[i] = NULL;
- }
-
- return -EAGAIN;
- }
- } else {
- btrfs_lock_extent(&inode->io_tree, start_pos, last_pos, cached_state);
- }
-
- ordered = btrfs_lookup_ordered_range(inode, start_pos,
- last_pos - start_pos + 1);
- if (ordered &&
- ordered->file_offset + ordered->num_bytes > start_pos &&
- ordered->file_offset <= last_pos) {
- btrfs_unlock_extent(&inode->io_tree, start_pos, last_pos,
- cached_state);
+ if (nowait) {
+ if (!btrfs_try_lock_extent(&inode->io_tree, start_pos, last_pos,
+ cached_state)) {
for (i = 0; i < num_pages; i++) {
unlock_page(pages[i]);
put_page(pages[i]);
+ pages[i] = NULL;
}
- btrfs_start_ordered_extent(ordered);
- btrfs_put_ordered_extent(ordered);
+
return -EAGAIN;
}
- if (ordered)
- btrfs_put_ordered_extent(ordered);
+ } else {
+ btrfs_lock_extent(&inode->io_tree, start_pos, last_pos, cached_state);
+ }
- *lockstart = start_pos;
- *lockend = last_pos;
- ret = 1;
+ ordered = btrfs_lookup_ordered_range(inode, start_pos,
+ last_pos - start_pos + 1);
+ if (ordered &&
+ ordered->file_offset + ordered->num_bytes > start_pos &&
+ ordered->file_offset <= last_pos) {
+ btrfs_unlock_extent(&inode->io_tree, start_pos, last_pos,
+ cached_state);
+ for (i = 0; i < num_pages; i++) {
+ unlock_page(pages[i]);
+ put_page(pages[i]);
+ }
+ btrfs_start_ordered_extent(ordered);
+ btrfs_put_ordered_extent(ordered);
+ return -EAGAIN;
}
+ if (ordered)
+ btrfs_put_ordered_extent(ordered);
+
+ *lockstart = start_pos;
+ *lockend = last_pos;
/*
* We should be called after prepare_pages() which should have locked
@@ -1041,7 +1035,7 @@ lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages,
for (i = 0; i < num_pages; i++)
WARN_ON(!PageLocked(pages[i]));
- return ret;
+ return 0;
}
/*
@@ -1242,7 +1236,6 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *i)
size_t copied;
size_t dirty_sectors;
size_t num_sectors;
- int extents_locked;
/*
* Fault pages before locking them in prepare_pages
@@ -1326,17 +1319,15 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *i)
break;
}
- extents_locked = lock_and_cleanup_extent_if_need(
- BTRFS_I(inode), pages,
+ ret = lock_and_cleanup_extent(BTRFS_I(inode), pages,
num_pages, pos, write_bytes, &lockstart,
&lockend, nowait, &cached_state);
- if (extents_locked < 0) {
- if (!nowait && extents_locked == -EAGAIN)
+ if (ret < 0) {
+ if (!nowait)
goto again;
btrfs_delalloc_release_extents(BTRFS_I(inode),
reserve_bytes);
- ret = extents_locked;
break;
}
@@ -1389,18 +1380,8 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *i)
dirty_pages, pos, copied,
&cached_state, only_release_metadata);
- /*
- * If we have not locked the extent range, because the range's
- * start offset is >= i_size, we might still have a non-NULL
- * cached extent state, acquired while marking the extent range
- * as delalloc through btrfs_dirty_pages(). Therefore free any
- * possible cached extent state to avoid a memory leak.
- */
- if (extents_locked)
- btrfs_unlock_extent(&BTRFS_I(inode)->io_tree, lockstart,
- lockend, &cached_state);
- else
- free_extent_state(cached_state);
+ btrfs_unlock_extent(&BTRFS_I(inode)->io_tree, lockstart,
+ lockend, &cached_state);
btrfs_delalloc_release_extents(BTRFS_I(inode), reserve_bytes);
if (ret) {
diff --git a/fs/btrfs/fs.h b/fs/btrfs/fs.h
index 90b104dd38c2..de10a95707c8 100644
--- a/fs/btrfs/fs.h
+++ b/fs/btrfs/fs.h
@@ -1044,6 +1044,15 @@ void __btrfs_clear_fs_compat_ro(struct btrfs_fs_info *fs_info, u64 flag,
#define btrfs_test_opt(fs_info, opt) ((fs_info)->mount_opt & \
BTRFS_MOUNT_##opt)
+static inline bool btrfs_is_full_ro(const struct btrfs_fs_info *fs_info)
+{
+ if (!sb_rdonly(fs_info->sb))
+ return false;
+ if (unlikely(fs_info->mount_opt & BTRFS_MOUNT_FULL_RO_MASK))
+ return true;
+ return false;
+}
+
static inline int btrfs_fs_closing(const struct btrfs_fs_info *fs_info)
{
/* Do it this way so we only ever do one test_bit in the normal case. */
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 68bea533bb49..9f3881db076f 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -3854,12 +3854,13 @@ static int btrfs_read_locked_inode(struct inode *inode,
btrfs_inode_split_flags(btrfs_inode_flags(leaf, inode_item),
&BTRFS_I(inode)->flags, &BTRFS_I(inode)->ro_flags);
+
+cache_index:
if (BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM)
mapping_clear_stable_writes(inode->i_mapping);
else
mapping_set_stable_writes(inode->i_mapping);
-cache_index:
/*
* If we were modified in the current generation and evicted from memory
* and then re-read we need to do a full sync since we don't have any
diff --git a/fs/btrfs/lzo.c b/fs/btrfs/lzo.c
index 72856f6775f7..f2097e7cab43 100644
--- a/fs/btrfs/lzo.c
+++ b/fs/btrfs/lzo.c
@@ -400,6 +400,17 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
return -EIO;
}
+ /* The segment must not extend beyond the compressed input. */
+ if (unlikely(cur_in + seg_len > cb->compressed_len)) {
+ struct btrfs_inode *inode = cb->bbio.inode;
+
+ btrfs_err(fs_info,
+ "lzo segment overflows compressed input, root %llu inode %llu offset %llu cur_in %u len %u compressed len %u",
+ btrfs_root_id(inode->root), btrfs_ino(inode),
+ cb->start, cur_in, seg_len, cb->compressed_len);
+ return -EUCLEAN;
+ }
+
/* Copy the compressed segment payload into workspace */
copy_compressed_segment(cb, workspace->cbuf, seg_len, &cur_in);
@@ -449,7 +460,7 @@ int lzo_decompress(struct list_head *ws, const u8 *data_in,
size_t max_segment_len = WORKSPACE_BUF_LENGTH;
int ret = 0;
- if (srclen < LZO_LEN || srclen > max_segment_len + LZO_LEN * 2)
+ if (srclen <= LZO_LEN * 2 || srclen > max_segment_len + LZO_LEN * 2)
return -EUCLEAN;
in_len = read_compress_length(data_in);
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index 5b158eb25d18..57552c414b43 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -4371,12 +4371,13 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode,
struct ulist_node *unode;
struct ulist_iterator uiter;
struct extent_changeset changeset;
+ const u32 sectorsize = root->fs_info->sectorsize;
+ const u64 aligned_start = round_down(start, sectorsize);
+ const u64 aligned_len = round_up(start + len, sectorsize) - aligned_start;
u64 freed = 0;
int ret;
extent_changeset_init(&changeset);
- len = round_up(start + len, root->fs_info->sectorsize);
- start = round_down(start, root->fs_info->sectorsize);
ULIST_ITER_INIT(&uiter);
while ((unode = ulist_next(&reserved->range_changed, &uiter))) {
@@ -4388,12 +4389,15 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode,
extent_changeset_release(&changeset);
- /* Only free range in range [start, start + len) */
- if (range_start >= start + len ||
- range_start + range_len <= start)
+ /*
+ * Only free the range within
+ * [aligned_start, aligned_start + aligned_len).
+ */
+ if (range_start >= aligned_start + aligned_len ||
+ range_start + range_len <= aligned_start)
continue;
- free_start = max(range_start, start);
- free_len = min(start + len, range_start + range_len) -
+ free_start = max(range_start, aligned_start);
+ free_len = min(aligned_start + aligned_len, range_start + range_len) -
free_start;
/*
* TODO: To also modify reserved->ranges_reserved to reflect
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 76842ed54335..4cf852f53184 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -1281,14 +1281,13 @@ static int btrfs_write_and_wait_transaction(struct btrfs_trans_handle *trans)
blk_finish_plug(&plug);
ret2 = btrfs_wait_extents(fs_info, dirty_pages);
- extent_io_tree_release(&trans->transaction->dirty_pages);
-
if (ret)
return ret;
- else if (ret2)
+ if (ret2)
return ret2;
- else
- return 0;
+
+ extent_io_tree_release(&trans->transaction->dirty_pages);
+ return 0;
}
/*
diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index 983771a2a456..ef2794fd28b6 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -4276,44 +4276,45 @@ static void fill_inode_item(struct btrfs_trans_handle *trans,
struct inode *inode, int log_inode_only,
u64 logged_isize)
{
- struct btrfs_map_token token;
+ u64 gen = BTRFS_I(inode)->generation;
u64 flags;
- btrfs_init_map_token(&token, leaf);
-
if (log_inode_only) {
- /* set the generation to zero so the recover code
- * can tell the difference between an logging
- * just to say 'this inode exists' and a logging
- * to say 'update this inode with these values'
+ /*
+ * Set the generation to zero so the recover code can tell the
+ * difference between a logging just to say 'this inode exists'
+ * and a logging to say 'update this inode with these values'.
+ * But only if the inode was not already logged before.
+ * We access ->logged_trans directly since it was already set
+ * up in the call chain by btrfs_log_inode(), and data_race()
+ * to avoid false alerts from KCSAN and since it was set already
+ * and one can set it to 0 since that only happens on eviction
+ * and we are holding a ref on the inode.
*/
- btrfs_set_token_inode_generation(&token, item, 0);
- btrfs_set_token_inode_size(&token, item, logged_isize);
+ ASSERT(data_race(BTRFS_I(inode)->logged_trans) > 0);
+ if (data_race(BTRFS_I(inode)->logged_trans) < trans->transid)
+ gen = 0;
+
+ btrfs_set_inode_size(leaf, item, logged_isize);
} else {
- btrfs_set_token_inode_generation(&token, item,
- BTRFS_I(inode)->generation);
- btrfs_set_token_inode_size(&token, item, inode->i_size);
+ btrfs_set_inode_size(leaf, item, inode->i_size);
}
- btrfs_set_token_inode_uid(&token, item, i_uid_read(inode));
- btrfs_set_token_inode_gid(&token, item, i_gid_read(inode));
- btrfs_set_token_inode_mode(&token, item, inode->i_mode);
- btrfs_set_token_inode_nlink(&token, item, inode->i_nlink);
+ btrfs_set_inode_generation(leaf, item, gen);
+
+ btrfs_set_inode_uid(leaf, item, i_uid_read(inode));
+ btrfs_set_inode_gid(leaf, item, i_gid_read(inode));
+ btrfs_set_inode_mode(leaf, item, inode->i_mode);
+ btrfs_set_inode_nlink(leaf, item, inode->i_nlink);
- btrfs_set_token_timespec_sec(&token, &item->atime,
- inode_get_atime_sec(inode));
- btrfs_set_token_timespec_nsec(&token, &item->atime,
- inode_get_atime_nsec(inode));
+ btrfs_set_timespec_sec(leaf, &item->atime, inode_get_atime_sec(inode));
+ btrfs_set_timespec_nsec(leaf, &item->atime, inode_get_atime_nsec(inode));
- btrfs_set_token_timespec_sec(&token, &item->mtime,
- inode_get_mtime_sec(inode));
- btrfs_set_token_timespec_nsec(&token, &item->mtime,
- inode_get_mtime_nsec(inode));
+ btrfs_set_timespec_sec(leaf, &item->mtime, inode_get_mtime_sec(inode));
+ btrfs_set_timespec_nsec(leaf, &item->mtime, inode_get_mtime_nsec(inode));
- btrfs_set_token_timespec_sec(&token, &item->ctime,
- inode_get_ctime_sec(inode));
- btrfs_set_token_timespec_nsec(&token, &item->ctime,
- inode_get_ctime_nsec(inode));
+ btrfs_set_timespec_sec(leaf, &item->ctime, inode_get_ctime_sec(inode));
+ btrfs_set_timespec_nsec(leaf, &item->ctime, inode_get_ctime_nsec(inode));
btrfs_set_timespec_sec(leaf, &item->otime, BTRFS_I(inode)->i_otime_sec);
btrfs_set_timespec_nsec(leaf, &item->otime, BTRFS_I(inode)->i_otime_nsec);
@@ -4327,13 +4328,13 @@ static void fill_inode_item(struct btrfs_trans_handle *trans,
* inode item in subvolume tree as needed (see overwrite_item()).
*/
- btrfs_set_token_inode_sequence(&token, item, inode_peek_iversion(inode));
- btrfs_set_token_inode_transid(&token, item, trans->transid);
- btrfs_set_token_inode_rdev(&token, item, inode->i_rdev);
+ btrfs_set_inode_sequence(leaf, item, inode_peek_iversion(inode));
+ btrfs_set_inode_transid(leaf, item, trans->transid);
+ btrfs_set_inode_rdev(leaf, item, inode->i_rdev);
flags = btrfs_inode_combine_flags(BTRFS_I(inode)->flags,
BTRFS_I(inode)->ro_flags);
- btrfs_set_token_inode_flags(&token, item, flags);
- btrfs_set_token_inode_block_group(&token, item, 0);
+ btrfs_set_inode_flags(leaf, item, flags);
+ btrfs_set_inode_block_group(leaf, item, 0);
}
static int log_inode_item(struct btrfs_trans_handle *trans,
@@ -5110,42 +5111,63 @@ static int btrfs_log_changed_extents(struct btrfs_trans_handle *trans,
return 0;
}
-static int logged_inode_size(struct btrfs_root *log, struct btrfs_inode *inode,
- struct btrfs_path *path, u64 *size_ret)
+static int get_inode_size_to_log(struct btrfs_trans_handle *trans,
+ struct btrfs_inode *inode,
+ struct btrfs_path *path, u64 *size_ret)
{
struct btrfs_key key;
+ struct btrfs_inode_item *item;
int ret;
key.objectid = btrfs_ino(inode);
key.type = BTRFS_INODE_ITEM_KEY;
key.offset = 0;
- ret = btrfs_search_slot(NULL, log, &key, path, 0, 0);
- if (ret < 0) {
- return ret;
- } else if (ret > 0) {
- *size_ret = 0;
- } else {
- struct btrfs_inode_item *item;
+ /*
+ * Our caller called inode_logged(), so logged_trans is up to date.
+ * Use data_race() to silence any warning from KCSAN. Once logged_trans
+ * is set, it can only be reset to 0 after inode eviction.
+ */
+ if (data_race(inode->logged_trans) == trans->transid) {
+ ret = btrfs_search_slot(NULL, inode->root->log_root, &key, path, 0, 0);
+ } else if (inode->generation < trans->transid) {
+ path->search_commit_root = true;
+ path->skip_locking = true;
+ ret = btrfs_search_slot(NULL, inode->root, &key, path, 0, 0);
+ path->search_commit_root = false;
+ path->skip_locking = false;
- item = btrfs_item_ptr(path->nodes[0], path->slots[0],
- struct btrfs_inode_item);
- *size_ret = btrfs_inode_size(path->nodes[0], item);
- /*
- * If the in-memory inode's i_size is smaller then the inode
- * size stored in the btree, return the inode's i_size, so
- * that we get a correct inode size after replaying the log
- * when before a power failure we had a shrinking truncate
- * followed by addition of a new name (rename / new hard link).
- * Otherwise return the inode size from the btree, to avoid
- * data loss when replaying a log due to previously doing a
- * write that expands the inode's size and logging a new name
- * immediately after.
- */
- if (*size_ret > inode->vfs_inode.i_size)
- *size_ret = inode->vfs_inode.i_size;
+ } else {
+ *size_ret = 0;
+ return 0;
}
+ /*
+ * If the inode was logged before or is from a past transaction, then
+ * its inode item must exist in the log root or in the commit root.
+ */
+ ASSERT(ret <= 0);
+ if (WARN_ON_ONCE(ret > 0))
+ ret = -ENOENT;
+
+ if (ret < 0)
+ return ret;
+
+ item = btrfs_item_ptr(path->nodes[0], path->slots[0],
+ struct btrfs_inode_item);
+ *size_ret = btrfs_inode_size(path->nodes[0], item);
+ /*
+ * If the in-memory inode's i_size is smaller then the inode size stored
+ * in the btree, return the inode's i_size, so that we get a correct
+ * inode size after replaying the log when before a power failure we had
+ * a shrinking truncate followed by addition of a new name (rename / new
+ * hard link). Otherwise return the inode size from the btree, to avoid
+ * data loss when replaying a log due to previously doing a write that
+ * expands the inode's size and logging a new name immediately after.
+ */
+ if (*size_ret > inode->vfs_inode.i_size)
+ *size_ret = inode->vfs_inode.i_size;
+
btrfs_release_path(path);
return 0;
}
@@ -6663,7 +6685,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans,
ret = drop_inode_items(trans, log, path, inode,
BTRFS_XATTR_ITEM_KEY);
} else {
- if (inode_only == LOG_INODE_EXISTS && ctx->logged_before) {
+ if (inode_only == LOG_INODE_EXISTS) {
/*
* Make sure the new inode item we write to the log has
* the same isize as the current one (if it exists).
@@ -6677,7 +6699,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans,
* (zeroes), as if an expanding truncate happened,
* instead of getting a file of 4Kb only.
*/
- ret = logged_inode_size(log, inode, path, &logged_isize);
+ ret = get_inode_size_to_log(trans, inode, path, &logged_isize);
if (ret)
goto out_unlock;
}
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index de0d9a450c3c..6d9df90de938 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -1483,30 +1483,158 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags,
}
/*
- * Try to find a chunk that intersects [start, start + len] range and when one
- * such is found, record the end of it in *start
+ * Find the first pending extent intersecting a range.
+ *
+ * @device: the device to search
+ * @start: start of the range to check
+ * @len: length of the range to check
+ * @pending_start: output pointer for the start of the found pending extent
+ * @pending_end: output pointer for the end of the found pending extent (inclusive)
+ *
+ * Search for a pending chunk allocation that intersects the half-open range
+ * [start, start + len).
+ *
+ * Return: true if a pending extent was found, false otherwise.
+ * If the return value is true, store the first pending extent in
+ * [*pending_start, *pending_end]. Otherwise, the two output variables
+ * may still be modified, to something outside the range and should not
+ * be used.
*/
-static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
- u64 len)
+static bool first_pending_extent(struct btrfs_device *device, u64 start, u64 len,
+ u64 *pending_start, u64 *pending_end)
{
- u64 physical_start, physical_end;
-
lockdep_assert_held(&device->fs_info->chunk_mutex);
- if (find_first_extent_bit(&device->alloc_state, *start,
- &physical_start, &physical_end,
+ if (find_first_extent_bit(&device->alloc_state, start,
+ pending_start, pending_end,
CHUNK_ALLOCATED, NULL)) {
- if (in_range(physical_start, *start, len) ||
- in_range(*start, physical_start,
- physical_end + 1 - physical_start)) {
- *start = physical_end + 1;
+ if (in_range(*pending_start, start, len) ||
+ in_range(start, *pending_start, *pending_end + 1 - *pending_start)) {
return true;
}
}
return false;
}
+/*
+ * Find the first real hole accounting for pending extents.
+ *
+ * @device: the device containing the candidate hole
+ * @start: input/output pointer for the hole start position
+ * @len: input/output pointer for the hole length
+ * @min_hole_size: the size of hole we are looking for
+ *
+ * Given a potential hole specified by [*start, *start + *len), check for pending
+ * chunk allocations within that range. If pending extents are found, the hole is
+ * adjusted to represent the first true free space that is large enough when
+ * accounting for pending chunks.
+ *
+ * Note that this function must handle various cases involving non consecutive
+ * pending extents.
+ *
+ * Returns: true if a suitable hole was found and false otherwise.
+ * If the return value is true, then *start and *len are set to represent the hole.
+ * If the return value is false, then *start is set to the largest hole we
+ * found and *len is set to its length.
+ * If there are no holes at all, then *start is set to the end of the range and
+ * *len is set to 0.
+ */
+static bool find_hole_in_pending_extents(struct btrfs_device *device, u64 *start,
+ u64 *len, u64 min_hole_size)
+{
+ u64 pending_start, pending_end;
+ u64 end;
+ u64 max_hole_start = 0;
+ u64 max_hole_len = 0;
+
+ lockdep_assert_held(&device->fs_info->chunk_mutex);
+
+ if (*len == 0)
+ return false;
+
+ end = *start + *len - 1;
+
+ /*
+ * Loop until we either see a large enough hole or check every pending
+ * extent overlapping the candidate hole.
+ * At every hole that we observe, record it if it is the new max.
+ * At the end of the iteration, set the output variables to the max hole.
+ */
+ while (true) {
+ if (first_pending_extent(device, *start, *len, &pending_start, &pending_end)) {
+ /*
+ * Case 1: the pending extent overlaps the start of
+ * candidate hole. That means the true hole is after the
+ * pending extent, but we need to find the next pending
+ * extent to properly size the hole. In the next loop,
+ * we will reduce to case 2 or 3.
+ * e.g.,
+ *
+ * |----pending A----| real hole |----pending B----|
+ * | candidate hole |
+ * *start end
+ */
+ if (pending_start <= *start) {
+ *start = pending_end + 1;
+ goto next;
+ }
+ /*
+ * Case 2: The pending extent starts after *start (and overlaps
+ * [*start, end), so the first hole just goes up to the start
+ * of the pending extent.
+ * e.g.,
+ *
+ * | real hole |----pending A----|
+ * | candidate hole |
+ * *start end
+ */
+ *len = pending_start - *start;
+ if (*len > max_hole_len) {
+ max_hole_start = *start;
+ max_hole_len = *len;
+ }
+ if (*len >= min_hole_size)
+ break;
+ /*
+ * If the hole wasn't big enough, then we advance past
+ * the pending extent and keep looking.
+ */
+ *start = pending_end + 1;
+ goto next;
+ } else {
+ /*
+ * Case 3: There is no pending extent overlapping the
+ * range [*start, *start + *len - 1], so the only remaining
+ * hole is the remaining range.
+ * e.g.,
+ *
+ * | candidate hole |
+ * | real hole |
+ * *start end
+ */
+
+ if (*len > max_hole_len) {
+ max_hole_start = *start;
+ max_hole_len = *len;
+ }
+ break;
+ }
+next:
+ if (*start > end)
+ break;
+ *len = end - *start + 1;
+ }
+ if (max_hole_len) {
+ *start = max_hole_start;
+ *len = max_hole_len;
+ } else {
+ *start = end + 1;
+ *len = 0;
+ }
+ return max_hole_len >= min_hole_size;
+}
+
static u64 dev_extent_search_start(struct btrfs_device *device)
{
switch (device->fs_devices->chunk_alloc_policy) {
@@ -1569,58 +1697,56 @@ static bool dev_extent_hole_check_zoned(struct btrfs_device *device,
}
/*
- * Check if specified hole is suitable for allocation.
+ * Validate and adjust a hole for chunk allocation
*
- * @device: the device which we have the hole
- * @hole_start: starting position of the hole
- * @hole_size: the size of the hole
- * @num_bytes: the size of the free space that we need
+ * @device: the device containing the candidate hole
+ * @hole_start: input/output pointer for the hole start position
+ * @hole_size: input/output pointer for the hole size
+ * @num_bytes: minimum allocation size required
*
- * This function may modify @hole_start and @hole_size to reflect the suitable
- * position for allocation. Returns 1 if hole position is updated, 0 otherwise.
+ * Check if the specified hole is suitable for allocation and adjust it if
+ * necessary. The hole may be modified to skip over pending chunk allocations
+ * and to satisfy stricter zoned requirements on zoned filesystems.
+ *
+ * For regular (non-zoned) allocation, if the hole after adjustment is smaller
+ * than @num_bytes, the search continues past additional pending extents until
+ * either a sufficiently large hole is found or no more pending extents exist.
+ *
+ * Return: true if a suitable hole was found and false otherwise.
+ * If the return value is true, then *hole_start and *hole_size are set to
+ * represent the hole we found.
+ * If the return value is false, then *hole_start is set to the largest
+ * hole we found and *hole_size is set to its length.
+ * If there are no holes at all, then *hole_start is set to the end of the range
+ * and *hole_size is set to 0.
*/
static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start,
u64 *hole_size, u64 num_bytes)
{
- bool changed = false;
- u64 hole_end = *hole_start + *hole_size;
+ bool found = false;
+ const u64 hole_end = *hole_start + *hole_size - 1;
- for (;;) {
- /*
- * Check before we set max_hole_start, otherwise we could end up
- * sending back this offset anyway.
- */
- if (contains_pending_extent(device, hole_start, *hole_size)) {
- if (hole_end >= *hole_start)
- *hole_size = hole_end - *hole_start;
- else
- *hole_size = 0;
- changed = true;
- }
+ ASSERT(*hole_size > 0);
- switch (device->fs_devices->chunk_alloc_policy) {
- case BTRFS_CHUNK_ALLOC_REGULAR:
- /* No extra check */
- break;
- case BTRFS_CHUNK_ALLOC_ZONED:
- if (dev_extent_hole_check_zoned(device, hole_start,
- hole_size, num_bytes)) {
- changed = true;
- /*
- * The changed hole can contain pending extent.
- * Loop again to check that.
- */
- continue;
- }
- break;
- default:
- BUG();
- }
+again:
+ *hole_size = hole_end - *hole_start + 1;
+ found = find_hole_in_pending_extents(device, hole_start, hole_size, num_bytes);
+ if (!found)
+ return found;
+ ASSERT(*hole_size >= num_bytes);
+ switch (device->fs_devices->chunk_alloc_policy) {
+ case BTRFS_CHUNK_ALLOC_REGULAR:
+ return found;
+ case BTRFS_CHUNK_ALLOC_ZONED:
+ if (dev_extent_hole_check_zoned(device, hole_start, hole_size, num_bytes))
+ goto again;
break;
+ default:
+ BUG();
}
- return changed;
+ return found;
}
/*
@@ -1679,7 +1805,7 @@ static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes,
ret = -ENOMEM;
goto out;
}
-again:
+
if (search_start >= search_end ||
test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state)) {
ret = -ENOSPC;
@@ -1766,11 +1892,7 @@ static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes,
*/
if (search_end > search_start) {
hole_size = search_end - search_start;
- if (dev_extent_hole_check(device, &search_start, &hole_size,
- num_bytes)) {
- btrfs_release_path(path);
- goto again;
- }
+ dev_extent_hole_check(device, &search_start, &hole_size, num_bytes);
if (hole_size > max_hole_size) {
max_hole_start = search_start;
@@ -4870,6 +4992,7 @@ int btrfs_shrink_device(struct btrfs_device *device, u64 new_size)
u64 diff;
u64 start;
u64 free_diff = 0;
+ u64 pending_start, pending_end;
new_size = round_down(new_size, fs_info->sectorsize);
start = new_size;
@@ -4915,7 +5038,7 @@ int btrfs_shrink_device(struct btrfs_device *device, u64 new_size)
* in-memory chunks are synced to disk so that the loop below sees them
* and relocates them accordingly.
*/
- if (contains_pending_extent(device, &start, diff)) {
+ if (first_pending_extent(device, start, diff, &pending_start, &pending_end)) {
mutex_unlock(&fs_info->chunk_mutex);
ret = btrfs_commit_transaction(trans);
if (ret)
diff --git a/fs/btrfs/zstd.c b/fs/btrfs/zstd.c
index c9ea37fabf65..732a84c0f6b1 100644
--- a/fs/btrfs/zstd.c
+++ b/fs/btrfs/zstd.c
@@ -290,8 +290,17 @@ struct list_head *zstd_get_workspace(unsigned int level)
DEFINE_WAIT(wait);
prepare_to_wait(&wsm.wait, &wait, TASK_UNINTERRUPTIBLE);
- schedule();
+ /*
+ * Re-check after being queued: zstd_put_workspace() only wakes
+ * a queue that already has a sleeper, so a workspace returned
+ * since the failed allocation woke nobody.
+ */
+ ws = zstd_find_workspace(level);
+ if (!ws)
+ schedule();
finish_wait(&wsm.wait, &wait);
+ if (ws)
+ return ws;
goto again;
}
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index e5ea8fb6d9a6..5e0f83adc459 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -255,7 +255,7 @@ static void finish_netfs_read(struct ceph_osd_request *req)
}
subreq->error = err;
trace_netfs_sreq(subreq, netfs_sreq_trace_io_progress);
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
iput(req->r_inode);
ceph_dec_osd_stopping_blocker(fsc->mdsc);
}
@@ -317,7 +317,7 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq)
out:
subreq->error = err;
trace_netfs_sreq(subreq, netfs_sreq_trace_io_progress);
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
return true;
}
@@ -440,7 +440,7 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq)
ceph_osdc_put_request(req);
if (err) {
subreq->error = err;
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
}
doutc(cl, "%llx.%llx result %d\n", ceph_vinop(inode), err);
}
@@ -572,7 +572,36 @@ struct ceph_writeback_ctl
u64 truncate_size;
u32 truncate_seq;
bool size_stable;
+
bool head_snapc;
+ struct ceph_snap_context *snapc;
+ struct ceph_snap_context *last_snapc;
+
+ bool done;
+ bool should_loop;
+ bool range_whole;
+ pgoff_t start_index;
+ pgoff_t index;
+ pgoff_t end;
+ xa_mark_t tag;
+
+ pgoff_t strip_unit_end;
+ unsigned int wsize;
+ unsigned int nr_folios;
+ unsigned int max_pages;
+ unsigned int locked_pages;
+
+ int op_idx;
+ int num_ops;
+ u64 offset;
+ u64 len;
+
+ struct folio_batch fbatch;
+ unsigned int processed_in_fbatch;
+
+ bool from_pool;
+ struct page **pages;
+ struct page **data_pages;
};
/*
@@ -937,36 +966,13 @@ static void writepages_finish(struct ceph_osd_request *req)
ceph_dec_osd_stopping_blocker(fsc->mdsc);
}
-/*
- * initiate async writeback
- */
-static int ceph_writepages_start(struct address_space *mapping,
- struct writeback_control *wbc)
+static inline
+bool is_forced_umount(struct address_space *mapping)
{
struct inode *inode = mapping->host;
struct ceph_inode_info *ci = ceph_inode(inode);
struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
struct ceph_client *cl = fsc->client;
- struct ceph_vino vino = ceph_vino(inode);
- pgoff_t index, start_index, end = -1;
- struct ceph_snap_context *snapc = NULL, *last_snapc = NULL, *pgsnapc;
- struct folio_batch fbatch;
- int rc = 0;
- unsigned int wsize = i_blocksize(inode);
- struct ceph_osd_request *req = NULL;
- struct ceph_writeback_ctl ceph_wbc;
- bool should_loop, range_whole = false;
- bool done = false;
- bool caching = ceph_is_cache_enabled(inode);
- xa_mark_t tag;
-
- if (wbc->sync_mode == WB_SYNC_NONE &&
- fsc->write_congested)
- return 0;
-
- doutc(cl, "%llx.%llx (mode=%s)\n", ceph_vinop(inode),
- wbc->sync_mode == WB_SYNC_NONE ? "NONE" :
- (wbc->sync_mode == WB_SYNC_ALL ? "ALL" : "HOLD"));
if (ceph_inode_is_shutdown(inode)) {
if (ci->i_wrbuffer_ref > 0) {
@@ -975,388 +981,735 @@ static int ceph_writepages_start(struct address_space *mapping,
ceph_vinop(inode), ceph_ino(inode));
}
mapping_set_error(mapping, -EIO);
- return -EIO; /* we're in a forced umount, don't write! */
+ return true;
}
+
+ return false;
+}
+
+static inline
+unsigned int ceph_define_write_size(struct address_space *mapping)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+ unsigned int wsize = i_blocksize(inode);
+
if (fsc->mount_options->wsize < wsize)
wsize = fsc->mount_options->wsize;
- folio_batch_init(&fbatch);
+ return wsize;
+}
+
+static inline
+void ceph_folio_batch_init(struct ceph_writeback_ctl *ceph_wbc)
+{
+ folio_batch_init(&ceph_wbc->fbatch);
+ ceph_wbc->processed_in_fbatch = 0;
+}
+
+static inline
+void ceph_folio_batch_reinit(struct ceph_writeback_ctl *ceph_wbc)
+{
+ folio_batch_release(&ceph_wbc->fbatch);
+ ceph_folio_batch_init(ceph_wbc);
+}
+
+static inline
+void ceph_init_writeback_ctl(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct ceph_writeback_ctl *ceph_wbc)
+{
+ ceph_wbc->snapc = NULL;
+ ceph_wbc->last_snapc = NULL;
+
+ ceph_wbc->strip_unit_end = 0;
+ ceph_wbc->wsize = ceph_define_write_size(mapping);
- start_index = wbc->range_cyclic ? mapping->writeback_index : 0;
- index = start_index;
+ ceph_wbc->nr_folios = 0;
+ ceph_wbc->max_pages = 0;
+ ceph_wbc->locked_pages = 0;
+
+ ceph_wbc->done = false;
+ ceph_wbc->should_loop = false;
+ ceph_wbc->range_whole = false;
+
+ ceph_wbc->start_index = wbc->range_cyclic ? mapping->writeback_index : 0;
+ ceph_wbc->index = ceph_wbc->start_index;
+ ceph_wbc->end = -1;
if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages) {
- tag = PAGECACHE_TAG_TOWRITE;
+ ceph_wbc->tag = PAGECACHE_TAG_TOWRITE;
} else {
- tag = PAGECACHE_TAG_DIRTY;
+ ceph_wbc->tag = PAGECACHE_TAG_DIRTY;
}
-retry:
+
+ ceph_wbc->op_idx = -1;
+ ceph_wbc->num_ops = 0;
+ ceph_wbc->offset = 0;
+ ceph_wbc->len = 0;
+ ceph_wbc->from_pool = false;
+
+ ceph_folio_batch_init(ceph_wbc);
+
+ ceph_wbc->pages = NULL;
+ ceph_wbc->data_pages = NULL;
+}
+
+static inline
+int ceph_define_writeback_range(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct ceph_writeback_ctl *ceph_wbc)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+ struct ceph_client *cl = fsc->client;
+
/* find oldest snap context with dirty data */
- snapc = get_oldest_context(inode, &ceph_wbc, NULL);
- if (!snapc) {
+ ceph_wbc->snapc = get_oldest_context(inode, ceph_wbc, NULL);
+ if (!ceph_wbc->snapc) {
/* hmm, why does writepages get called when there
is no dirty data? */
doutc(cl, " no snap context with dirty data?\n");
- goto out;
+ return -ENODATA;
}
- doutc(cl, " oldest snapc is %p seq %lld (%d snaps)\n", snapc,
- snapc->seq, snapc->num_snaps);
- should_loop = false;
- if (ceph_wbc.head_snapc && snapc != last_snapc) {
+ doutc(cl, " oldest snapc is %p seq %lld (%d snaps)\n",
+ ceph_wbc->snapc, ceph_wbc->snapc->seq,
+ ceph_wbc->snapc->num_snaps);
+
+ ceph_wbc->should_loop = false;
+
+ if (ceph_wbc->head_snapc && ceph_wbc->snapc != ceph_wbc->last_snapc) {
/* where to start/end? */
if (wbc->range_cyclic) {
- index = start_index;
- end = -1;
- if (index > 0)
- should_loop = true;
- doutc(cl, " cyclic, start at %lu\n", index);
+ ceph_wbc->index = ceph_wbc->start_index;
+ ceph_wbc->end = -1;
+ if (ceph_wbc->index > 0)
+ ceph_wbc->should_loop = true;
+ doutc(cl, " cyclic, start at %lu\n", ceph_wbc->index);
} else {
- index = wbc->range_start >> PAGE_SHIFT;
- end = wbc->range_end >> PAGE_SHIFT;
+ ceph_wbc->index = wbc->range_start >> PAGE_SHIFT;
+ ceph_wbc->end = wbc->range_end >> PAGE_SHIFT;
if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
- range_whole = true;
- doutc(cl, " not cyclic, %lu to %lu\n", index, end);
+ ceph_wbc->range_whole = true;
+ doutc(cl, " not cyclic, %lu to %lu\n",
+ ceph_wbc->index, ceph_wbc->end);
}
- } else if (!ceph_wbc.head_snapc) {
+ } else if (!ceph_wbc->head_snapc) {
/* Do not respect wbc->range_{start,end}. Dirty pages
* in that range can be associated with newer snapc.
* They are not writeable until we write all dirty pages
* associated with 'snapc' get written */
- if (index > 0)
- should_loop = true;
+ if (ceph_wbc->index > 0)
+ ceph_wbc->should_loop = true;
doutc(cl, " non-head snapc, range whole\n");
}
- if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages)
- tag_pages_for_writeback(mapping, index, end);
+ ceph_put_snap_context(ceph_wbc->last_snapc);
+ ceph_wbc->last_snapc = ceph_wbc->snapc;
- ceph_put_snap_context(last_snapc);
- last_snapc = snapc;
+ return 0;
+}
- while (!done && index <= end) {
- int num_ops = 0, op_idx;
- unsigned i, nr_folios, max_pages, locked_pages = 0;
- struct page **pages = NULL, **data_pages;
- struct page *page;
- pgoff_t strip_unit_end = 0;
- u64 offset = 0, len = 0;
- bool from_pool = false;
+static inline
+bool has_writeback_done(struct ceph_writeback_ctl *ceph_wbc)
+{
+ return ceph_wbc->done && ceph_wbc->index > ceph_wbc->end;
+}
- max_pages = wsize >> PAGE_SHIFT;
+static inline
+bool can_next_page_be_processed(struct ceph_writeback_ctl *ceph_wbc,
+ unsigned index)
+{
+ return index < ceph_wbc->nr_folios &&
+ ceph_wbc->locked_pages < ceph_wbc->max_pages;
+}
-get_more_pages:
- nr_folios = filemap_get_folios_tag(mapping, &index,
- end, tag, &fbatch);
- doutc(cl, "pagevec_lookup_range_tag got %d\n", nr_folios);
- if (!nr_folios && !locked_pages)
- break;
- for (i = 0; i < nr_folios && locked_pages < max_pages; i++) {
- page = &fbatch.folios[i]->page;
- doutc(cl, "? %p idx %lu\n", page, page->index);
- if (locked_pages == 0)
- lock_page(page); /* first page */
- else if (!trylock_page(page))
- break;
+static
+int ceph_check_page_before_write(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct ceph_writeback_ctl *ceph_wbc,
+ struct folio *folio)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+ struct ceph_client *cl = fsc->client;
+ struct ceph_snap_context *pgsnapc;
+ struct page *page = &folio->page;
- /* only dirty pages, or our accounting breaks */
- if (unlikely(!PageDirty(page)) ||
- unlikely(page->mapping != mapping)) {
- doutc(cl, "!dirty or !mapping %p\n", page);
- unlock_page(page);
- continue;
- }
- /* only if matching snap context */
- pgsnapc = page_snap_context(page);
- if (pgsnapc != snapc) {
- doutc(cl, "page snapc %p %lld != oldest %p %lld\n",
- pgsnapc, pgsnapc->seq, snapc, snapc->seq);
- if (!should_loop &&
- !ceph_wbc.head_snapc &&
- wbc->sync_mode != WB_SYNC_NONE)
- should_loop = true;
- unlock_page(page);
- continue;
+ /* only dirty pages, or our accounting breaks */
+ if (unlikely(!PageDirty(page)) || unlikely(page->mapping != mapping)) {
+ doutc(cl, "!dirty or !mapping %p\n", page);
+ return -ENODATA;
+ }
+
+ /* only if matching snap context */
+ pgsnapc = page_snap_context(page);
+ if (pgsnapc != ceph_wbc->snapc) {
+ doutc(cl, "page snapc %p %lld != oldest %p %lld\n",
+ pgsnapc, pgsnapc->seq,
+ ceph_wbc->snapc, ceph_wbc->snapc->seq);
+
+ if (!ceph_wbc->should_loop && !ceph_wbc->head_snapc &&
+ wbc->sync_mode != WB_SYNC_NONE)
+ ceph_wbc->should_loop = true;
+
+ return -ENODATA;
+ }
+
+ if (page_offset(page) >= ceph_wbc->i_size) {
+ doutc(cl, "folio at %lu beyond eof %llu\n",
+ folio->index, ceph_wbc->i_size);
+
+ if ((ceph_wbc->size_stable ||
+ folio_pos(folio) >= i_size_read(inode)) &&
+ folio_clear_dirty_for_io(folio))
+ folio_invalidate(folio, 0, folio_size(folio));
+
+ return -ENODATA;
+ }
+
+ if (ceph_wbc->strip_unit_end &&
+ (page->index > ceph_wbc->strip_unit_end)) {
+ doutc(cl, "end of strip unit %p\n", page);
+ return -E2BIG;
+ }
+
+ return 0;
+}
+
+static inline
+void __ceph_allocate_page_array(struct ceph_writeback_ctl *ceph_wbc,
+ unsigned int max_pages)
+{
+ ceph_wbc->pages = kmalloc_array(max_pages,
+ sizeof(*ceph_wbc->pages),
+ GFP_NOFS);
+ if (!ceph_wbc->pages) {
+ ceph_wbc->from_pool = true;
+ ceph_wbc->pages = mempool_alloc(ceph_wb_pagevec_pool, GFP_NOFS);
+ BUG_ON(!ceph_wbc->pages);
+ }
+}
+
+static inline
+void ceph_allocate_page_array(struct address_space *mapping,
+ struct ceph_writeback_ctl *ceph_wbc,
+ struct page *page)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_inode_info *ci = ceph_inode(inode);
+ u64 objnum;
+ u64 objoff;
+ u32 xlen;
+
+ /* prepare async write request */
+ ceph_wbc->offset = (u64)page_offset(page);
+ ceph_calc_file_object_mapping(&ci->i_layout,
+ ceph_wbc->offset, ceph_wbc->wsize,
+ &objnum, &objoff, &xlen);
+
+ ceph_wbc->num_ops = 1;
+ ceph_wbc->strip_unit_end = page->index + ((xlen - 1) >> PAGE_SHIFT);
+
+ BUG_ON(ceph_wbc->pages);
+ ceph_wbc->max_pages = calc_pages_for(0, (u64)xlen);
+ __ceph_allocate_page_array(ceph_wbc, ceph_wbc->max_pages);
+
+ ceph_wbc->len = 0;
+}
+
+static inline
+bool is_folio_index_contiguous(const struct ceph_writeback_ctl *ceph_wbc,
+ const struct folio *folio)
+{
+ return folio->index == (ceph_wbc->offset + ceph_wbc->len) >> PAGE_SHIFT;
+}
+
+static inline
+bool is_num_ops_too_big(struct ceph_writeback_ctl *ceph_wbc)
+{
+ return ceph_wbc->num_ops >=
+ (ceph_wbc->from_pool ? CEPH_OSD_SLAB_OPS : CEPH_OSD_MAX_OPS);
+}
+
+static inline
+bool is_write_congestion_happened(struct ceph_fs_client *fsc)
+{
+ return atomic_long_inc_return(&fsc->writeback_count) >
+ CONGESTION_ON_THRESH(fsc->mount_options->congestion_kb);
+}
+
+static inline
+int ceph_move_dirty_page_in_page_array(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct ceph_writeback_ctl *ceph_wbc,
+ struct page *page)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+ struct ceph_client *cl = fsc->client;
+ struct page **pages = ceph_wbc->pages;
+ unsigned int index = ceph_wbc->locked_pages;
+ gfp_t gfp_flags = ceph_wbc->locked_pages ? GFP_NOWAIT : GFP_NOFS;
+
+ if (IS_ENCRYPTED(inode)) {
+ pages[index] = fscrypt_encrypt_pagecache_blocks(page,
+ PAGE_SIZE,
+ 0,
+ gfp_flags);
+ if (IS_ERR(pages[index])) {
+ int err = PTR_ERR(pages[index]);
+
+ if (err == -EINVAL) {
+ pr_err_client(cl, "inode->i_blkbits=%hhu\n",
+ inode->i_blkbits);
}
- if (page_offset(page) >= ceph_wbc.i_size) {
- struct folio *folio = page_folio(page);
-
- doutc(cl, "folio at %lu beyond eof %llu\n",
- folio->index, ceph_wbc.i_size);
- if ((ceph_wbc.size_stable ||
- folio_pos(folio) >= i_size_read(inode)) &&
- folio_clear_dirty_for_io(folio))
- folio_invalidate(folio, 0,
- folio_size(folio));
+
+ /* better not fail on first page! */
+ BUG_ON(ceph_wbc->locked_pages == 0);
+
+ pages[index] = NULL;
+ return err;
+ }
+ } else {
+ pages[index] = page;
+ }
+
+ ceph_wbc->locked_pages++;
+
+ return 0;
+}
+
+static
+int ceph_process_folio_batch(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct ceph_writeback_ctl *ceph_wbc)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+ struct ceph_client *cl = fsc->client;
+ struct folio *folio = NULL;
+ unsigned i;
+ int rc = 0;
+
+ for (i = 0; can_next_page_be_processed(ceph_wbc, i); i++) {
+ folio = ceph_wbc->fbatch.folios[i];
+
+ if (!folio)
+ continue;
+
+ doutc(cl, "? %p idx %lu, folio_test_writeback %#x, "
+ "folio_test_dirty %#x, folio_test_locked %#x\n",
+ folio, folio->index, folio_test_writeback(folio),
+ folio_test_dirty(folio),
+ folio_test_locked(folio));
+
+ if (folio_test_writeback(folio) ||
+ folio_test_private_2(folio) /* [DEPRECATED] */) {
+ doutc(cl, "waiting on writeback %p\n", folio);
+ folio_wait_writeback(folio);
+ folio_wait_private_2(folio); /* [DEPRECATED] */
+ continue;
+ }
+
+ if (ceph_wbc->locked_pages == 0)
+ folio_lock(folio);
+ else if (!folio_trylock(folio))
+ break;
+
+ rc = ceph_check_page_before_write(mapping, wbc,
+ ceph_wbc, folio);
+ if (rc == -ENODATA) {
+ rc = 0;
+ folio_unlock(folio);
+ ceph_wbc->fbatch.folios[i] = NULL;
+ continue;
+ } else if (rc == -E2BIG) {
+ rc = 0;
+ folio_unlock(folio);
+ break;
+ }
+
+ if (!folio_clear_dirty_for_io(folio)) {
+ doutc(cl, "%p !folio_clear_dirty_for_io\n", folio);
+ folio_unlock(folio);
+ ceph_wbc->fbatch.folios[i] = NULL;
+ continue;
+ }
+
+ /*
+ * We have something to write. If this is
+ * the first locked page this time through,
+ * calculate max possible write size and
+ * allocate a page array
+ */
+ if (ceph_wbc->locked_pages == 0) {
+ ceph_allocate_page_array(mapping, ceph_wbc, &folio->page);
+ } else if (!is_folio_index_contiguous(ceph_wbc, folio)) {
+ if (is_num_ops_too_big(ceph_wbc)) {
+ folio_redirty_for_writepage(wbc, folio);
folio_unlock(folio);
- continue;
- }
- if (strip_unit_end && (page->index > strip_unit_end)) {
- doutc(cl, "end of strip unit %p\n", page);
- unlock_page(page);
break;
}
- if (PageWriteback(page) ||
- PagePrivate2(page) /* [DEPRECATED] */) {
- if (wbc->sync_mode == WB_SYNC_NONE) {
- doutc(cl, "%p under writeback\n", page);
- unlock_page(page);
- continue;
- }
- doutc(cl, "waiting on writeback %p\n", page);
- wait_on_page_writeback(page);
- folio_wait_private_2(page_folio(page)); /* [DEPRECATED] */
- }
- if (!clear_page_dirty_for_io(page)) {
- doutc(cl, "%p !clear_page_dirty_for_io\n", page);
- unlock_page(page);
- continue;
- }
+ ceph_wbc->num_ops++;
+ ceph_wbc->offset = (u64)folio_pos(folio);
+ ceph_wbc->len = 0;
+ }
- /*
- * We have something to write. If this is
- * the first locked page this time through,
- * calculate max possinle write size and
- * allocate a page array
- */
- if (locked_pages == 0) {
- u64 objnum;
- u64 objoff;
- u32 xlen;
-
- /* prepare async write request */
- offset = (u64)page_offset(page);
- ceph_calc_file_object_mapping(&ci->i_layout,
- offset, wsize,
- &objnum, &objoff,
- &xlen);
- len = xlen;
-
- num_ops = 1;
- strip_unit_end = page->index +
- ((len - 1) >> PAGE_SHIFT);
-
- BUG_ON(pages);
- max_pages = calc_pages_for(0, (u64)len);
- pages = kmalloc_array(max_pages,
- sizeof(*pages),
- GFP_NOFS);
- if (!pages) {
- from_pool = true;
- pages = mempool_alloc(ceph_wb_pagevec_pool, GFP_NOFS);
- BUG_ON(!pages);
- }
-
- len = 0;
- } else if (page->index !=
- (offset + len) >> PAGE_SHIFT) {
- if (num_ops >= (from_pool ? CEPH_OSD_SLAB_OPS :
- CEPH_OSD_MAX_OPS)) {
- redirty_page_for_writepage(wbc, page);
- unlock_page(page);
- break;
- }
-
- num_ops++;
- offset = (u64)page_offset(page);
- len = 0;
- }
+ /* note position of first page in fbatch */
+ doutc(cl, "%llx.%llx will write folio %p idx %lu\n",
+ ceph_vinop(inode), folio, folio->index);
- /* note position of first page in fbatch */
- doutc(cl, "%llx.%llx will write page %p idx %lu\n",
- ceph_vinop(inode), page, page->index);
-
- if (atomic_long_inc_return(&fsc->writeback_count) >
- CONGESTION_ON_THRESH(
- fsc->mount_options->congestion_kb))
- fsc->write_congested = true;
-
- if (IS_ENCRYPTED(inode)) {
- pages[locked_pages] =
- fscrypt_encrypt_pagecache_blocks(page,
- PAGE_SIZE, 0,
- locked_pages ? GFP_NOWAIT : GFP_NOFS);
- if (IS_ERR(pages[locked_pages])) {
- if (PTR_ERR(pages[locked_pages]) == -EINVAL)
- pr_err_client(cl,
- "inode->i_blkbits=%hhu\n",
- inode->i_blkbits);
- /* better not fail on first page! */
- BUG_ON(locked_pages == 0);
- pages[locked_pages] = NULL;
- redirty_page_for_writepage(wbc, page);
- unlock_page(page);
- break;
- }
- ++locked_pages;
- } else {
- pages[locked_pages++] = page;
- }
+ fsc->write_congested = is_write_congestion_happened(fsc);
+
+ rc = ceph_move_dirty_page_in_page_array(mapping, wbc,
+ ceph_wbc, &folio->page);
+ if (rc) {
+ /* Did we just begin a new contiguous op? Nevermind! */
+ if (ceph_wbc->len == 0)
+ ceph_wbc->num_ops--;
- fbatch.folios[i] = NULL;
- len += thp_size(page);
+ folio_redirty_for_writepage(wbc, folio);
+ folio_unlock(folio);
+ break;
}
- /* did we get anything? */
- if (!locked_pages)
- goto release_folios;
- if (i) {
- unsigned j, n = 0;
- /* shift unused page to beginning of fbatch */
- for (j = 0; j < nr_folios; j++) {
- if (!fbatch.folios[j])
- continue;
- if (n < j)
- fbatch.folios[n] = fbatch.folios[j];
- n++;
- }
- fbatch.nr = n;
+ ceph_wbc->fbatch.folios[i] = NULL;
+ ceph_wbc->len += folio_size(folio);
+ }
- if (nr_folios && i == nr_folios &&
- locked_pages < max_pages) {
- doutc(cl, "reached end fbatch, trying for more\n");
- folio_batch_release(&fbatch);
- goto get_more_pages;
- }
+ ceph_wbc->processed_in_fbatch = i;
+
+ return rc;
+}
+
+static inline
+void ceph_shift_unused_folios_left(struct folio_batch *fbatch)
+{
+ unsigned j, n = 0;
+
+ /* shift unused page to beginning of fbatch */
+ for (j = 0; j < folio_batch_count(fbatch); j++) {
+ if (!fbatch->folios[j])
+ continue;
+
+ if (n < j) {
+ fbatch->folios[n] = fbatch->folios[j];
}
+ n++;
+ }
+
+ fbatch->nr = n;
+}
+
+static
+int ceph_submit_write(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct ceph_writeback_ctl *ceph_wbc)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_inode_info *ci = ceph_inode(inode);
+ struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+ struct ceph_client *cl = fsc->client;
+ struct ceph_vino vino = ceph_vino(inode);
+ struct ceph_osd_request *req = NULL;
+ struct page *page = NULL;
+ bool caching = ceph_is_cache_enabled(inode);
+ u64 offset;
+ u64 len;
+ unsigned i;
+
new_request:
- offset = ceph_fscrypt_page_offset(pages[0]);
- len = wsize;
+ offset = ceph_fscrypt_page_offset(ceph_wbc->pages[0]);
+ len = ceph_wbc->wsize;
+ req = ceph_osdc_new_request(&fsc->client->osdc,
+ &ci->i_layout, vino,
+ offset, &len, 0, ceph_wbc->num_ops,
+ CEPH_OSD_OP_WRITE, CEPH_OSD_FLAG_WRITE,
+ ceph_wbc->snapc, ceph_wbc->truncate_seq,
+ ceph_wbc->truncate_size, false);
+ if (IS_ERR(req)) {
req = ceph_osdc_new_request(&fsc->client->osdc,
- &ci->i_layout, vino,
- offset, &len, 0, num_ops,
- CEPH_OSD_OP_WRITE, CEPH_OSD_FLAG_WRITE,
- snapc, ceph_wbc.truncate_seq,
- ceph_wbc.truncate_size, false);
- if (IS_ERR(req)) {
- req = ceph_osdc_new_request(&fsc->client->osdc,
- &ci->i_layout, vino,
- offset, &len, 0,
- min(num_ops,
- CEPH_OSD_SLAB_OPS),
- CEPH_OSD_OP_WRITE,
- CEPH_OSD_FLAG_WRITE,
- snapc, ceph_wbc.truncate_seq,
- ceph_wbc.truncate_size, true);
- BUG_ON(IS_ERR(req));
+ &ci->i_layout, vino,
+ offset, &len, 0,
+ min(ceph_wbc->num_ops,
+ CEPH_OSD_SLAB_OPS),
+ CEPH_OSD_OP_WRITE,
+ CEPH_OSD_FLAG_WRITE,
+ ceph_wbc->snapc,
+ ceph_wbc->truncate_seq,
+ ceph_wbc->truncate_size,
+ true);
+ BUG_ON(IS_ERR(req));
+ }
+
+ page = ceph_wbc->pages[ceph_wbc->locked_pages - 1];
+ BUG_ON(len < ceph_fscrypt_page_offset(page) + thp_size(page) - offset);
+
+ if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
+ for (i = 0; i < folio_batch_count(&ceph_wbc->fbatch); i++) {
+ struct folio *folio = ceph_wbc->fbatch.folios[i];
+
+ if (!folio)
+ continue;
+
+ page = &folio->page;
+ redirty_page_for_writepage(wbc, page);
+ unlock_page(page);
}
- BUG_ON(len < ceph_fscrypt_page_offset(pages[locked_pages - 1]) +
- thp_size(pages[locked_pages - 1]) - offset);
- if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
- rc = -EIO;
- goto release_folios;
+ for (i = 0; i < ceph_wbc->locked_pages; i++) {
+ page = ceph_fscrypt_pagecache_page(ceph_wbc->pages[i]);
+
+ if (!page)
+ continue;
+
+ redirty_page_for_writepage(wbc, page);
+ unlock_page(page);
}
- req->r_callback = writepages_finish;
- req->r_inode = inode;
-
- /* Format the osd request message and submit the write */
- len = 0;
- data_pages = pages;
- op_idx = 0;
- for (i = 0; i < locked_pages; i++) {
- struct page *page = ceph_fscrypt_pagecache_page(pages[i]);
-
- u64 cur_offset = page_offset(page);
- /*
- * Discontinuity in page range? Ceph can handle that by just passing
- * multiple extents in the write op.
- */
- if (offset + len != cur_offset) {
- /* If it's full, stop here */
- if (op_idx + 1 == req->r_num_ops)
- break;
-
- /* Kick off an fscache write with what we have so far. */
- ceph_fscache_write_to_cache(inode, offset, len, caching);
-
- /* Start a new extent */
- osd_req_op_extent_dup_last(req, op_idx,
- cur_offset - offset);
- doutc(cl, "got pages at %llu~%llu\n", offset,
- len);
- osd_req_op_extent_osd_data_pages(req, op_idx,
- data_pages, len, 0,
- from_pool, false);
- osd_req_op_extent_update(req, op_idx, len);
-
- len = 0;
- offset = cur_offset;
- data_pages = pages + i;
- op_idx++;
- }
- set_page_writeback(page);
- if (caching)
- ceph_set_page_fscache(page);
- len += thp_size(page);
+ ceph_osdc_put_request(req);
+ return -EIO;
+ }
+
+ req->r_callback = writepages_finish;
+ req->r_inode = inode;
+
+ /* Format the osd request message and submit the write */
+ len = 0;
+ ceph_wbc->data_pages = ceph_wbc->pages;
+ ceph_wbc->op_idx = 0;
+ for (i = 0; i < ceph_wbc->locked_pages; i++) {
+ u64 cur_offset;
+
+ page = ceph_fscrypt_pagecache_page(ceph_wbc->pages[i]);
+ cur_offset = page_offset(page);
+
+ /*
+ * Discontinuity in page range? Ceph can handle that by just passing
+ * multiple extents in the write op.
+ */
+ if (offset + len != cur_offset) {
+ /* If it's full, stop here */
+ if (ceph_wbc->op_idx + 1 == req->r_num_ops)
+ break;
+
+ /* Kick off an fscache write with what we have so far. */
+ ceph_fscache_write_to_cache(inode, offset, len, caching);
+
+ /* Start a new extent */
+ osd_req_op_extent_dup_last(req, ceph_wbc->op_idx,
+ cur_offset - offset);
+
+ doutc(cl, "got pages at %llu~%llu\n", offset, len);
+
+ osd_req_op_extent_osd_data_pages(req, ceph_wbc->op_idx,
+ ceph_wbc->data_pages,
+ len, 0,
+ ceph_wbc->from_pool,
+ false);
+ osd_req_op_extent_update(req, ceph_wbc->op_idx, len);
+
+ len = 0;
+ offset = cur_offset;
+ ceph_wbc->data_pages = ceph_wbc->pages + i;
+ ceph_wbc->op_idx++;
}
- ceph_fscache_write_to_cache(inode, offset, len, caching);
-
- if (ceph_wbc.size_stable) {
- len = min(len, ceph_wbc.i_size - offset);
- } else if (i == locked_pages) {
- /* writepages_finish() clears writeback pages
- * according to the data length, so make sure
- * data length covers all locked pages */
- u64 min_len = len + 1 - thp_size(page);
- len = get_writepages_data_length(inode, pages[i - 1],
- offset);
- len = max(len, min_len);
+
+ set_page_writeback(page);
+
+ if (caching)
+ ceph_set_page_fscache(page);
+
+ len += thp_size(page);
+ }
+
+ ceph_fscache_write_to_cache(inode, offset, len, caching);
+
+ if (ceph_wbc->size_stable) {
+ len = min(len, ceph_wbc->i_size - offset);
+ } else if (i == ceph_wbc->locked_pages) {
+ /* writepages_finish() clears writeback pages
+ * according to the data length, so make sure
+ * data length covers all locked pages */
+ u64 min_len = len + 1 - thp_size(page);
+ len = get_writepages_data_length(inode,
+ ceph_wbc->pages[i - 1],
+ offset);
+ len = max(len, min_len);
+ }
+
+ if (IS_ENCRYPTED(inode))
+ len = round_up(len, CEPH_FSCRYPT_BLOCK_SIZE);
+
+ doutc(cl, "got pages at %llu~%llu\n", offset, len);
+
+ if (IS_ENCRYPTED(inode) &&
+ ((offset | len) & ~CEPH_FSCRYPT_BLOCK_MASK)) {
+ pr_warn_client(cl,
+ "bad encrypted write offset=%lld len=%llu\n",
+ offset, len);
+ }
+
+ osd_req_op_extent_osd_data_pages(req, ceph_wbc->op_idx,
+ ceph_wbc->data_pages, len,
+ 0, ceph_wbc->from_pool, false);
+ osd_req_op_extent_update(req, ceph_wbc->op_idx, len);
+
+ BUG_ON(ceph_wbc->op_idx + 1 != req->r_num_ops);
+
+ ceph_wbc->from_pool = false;
+ if (i < ceph_wbc->locked_pages) {
+ BUG_ON(ceph_wbc->num_ops <= req->r_num_ops);
+ ceph_wbc->num_ops -= req->r_num_ops;
+ ceph_wbc->locked_pages -= i;
+
+ /* allocate new pages array for next request */
+ ceph_wbc->data_pages = ceph_wbc->pages;
+ __ceph_allocate_page_array(ceph_wbc, ceph_wbc->locked_pages);
+ memcpy(ceph_wbc->pages, ceph_wbc->data_pages + i,
+ ceph_wbc->locked_pages * sizeof(*ceph_wbc->pages));
+ memset(ceph_wbc->data_pages + i, 0,
+ ceph_wbc->locked_pages * sizeof(*ceph_wbc->pages));
+ } else {
+ BUG_ON(ceph_wbc->num_ops != req->r_num_ops);
+ /* request message now owns the pages array */
+ ceph_wbc->pages = NULL;
+ }
+
+ req->r_mtime = inode_get_mtime(inode);
+ ceph_osdc_start_request(&fsc->client->osdc, req);
+ req = NULL;
+
+ wbc->nr_to_write -= i;
+ if (ceph_wbc->pages)
+ goto new_request;
+
+ return 0;
+}
+
+static
+void ceph_wait_until_current_writes_complete(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct ceph_writeback_ctl *ceph_wbc)
+{
+ struct page *page;
+ unsigned i, nr;
+
+ if (wbc->sync_mode != WB_SYNC_NONE &&
+ ceph_wbc->start_index == 0 && /* all dirty pages were checked */
+ !ceph_wbc->head_snapc) {
+ ceph_wbc->index = 0;
+
+ while ((ceph_wbc->index <= ceph_wbc->end) &&
+ (nr = filemap_get_folios_tag(mapping,
+ &ceph_wbc->index,
+ (pgoff_t)-1,
+ PAGECACHE_TAG_WRITEBACK,
+ &ceph_wbc->fbatch))) {
+ for (i = 0; i < nr; i++) {
+ page = &ceph_wbc->fbatch.folios[i]->page;
+ if (page_snap_context(page) != ceph_wbc->snapc)
+ continue;
+ wait_on_page_writeback(page);
+ }
+
+ folio_batch_release(&ceph_wbc->fbatch);
+ cond_resched();
}
- if (IS_ENCRYPTED(inode))
- len = round_up(len, CEPH_FSCRYPT_BLOCK_SIZE);
+ }
+}
+
+/*
+ * initiate async writeback
+ */
+static int ceph_writepages_start(struct address_space *mapping,
+ struct writeback_control *wbc)
+{
+ struct inode *inode = mapping->host;
+ struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode);
+ struct ceph_client *cl = fsc->client;
+ struct ceph_writeback_ctl ceph_wbc;
+ int rc = 0;
- doutc(cl, "got pages at %llu~%llu\n", offset, len);
+ if (wbc->sync_mode == WB_SYNC_NONE && fsc->write_congested)
+ return 0;
- if (IS_ENCRYPTED(inode) &&
- ((offset | len) & ~CEPH_FSCRYPT_BLOCK_MASK))
- pr_warn_client(cl,
- "bad encrypted write offset=%lld len=%llu\n",
- offset, len);
-
- osd_req_op_extent_osd_data_pages(req, op_idx, data_pages, len,
- 0, from_pool, false);
- osd_req_op_extent_update(req, op_idx, len);
-
- BUG_ON(op_idx + 1 != req->r_num_ops);
-
- from_pool = false;
- if (i < locked_pages) {
- BUG_ON(num_ops <= req->r_num_ops);
- num_ops -= req->r_num_ops;
- locked_pages -= i;
-
- /* allocate new pages array for next request */
- data_pages = pages;
- pages = kmalloc_array(locked_pages, sizeof(*pages),
- GFP_NOFS);
- if (!pages) {
- from_pool = true;
- pages = mempool_alloc(ceph_wb_pagevec_pool, GFP_NOFS);
- BUG_ON(!pages);
+ doutc(cl, "%llx.%llx (mode=%s)\n", ceph_vinop(inode),
+ wbc->sync_mode == WB_SYNC_NONE ? "NONE" :
+ (wbc->sync_mode == WB_SYNC_ALL ? "ALL" : "HOLD"));
+
+ if (is_forced_umount(mapping)) {
+ /* we're in a forced umount, don't write! */
+ return -EIO;
+ }
+
+ ceph_init_writeback_ctl(mapping, wbc, &ceph_wbc);
+
+retry:
+ rc = ceph_define_writeback_range(mapping, wbc, &ceph_wbc);
+ if (rc == -ENODATA) {
+ /* hmm, why does writepages get called when there
+ is no dirty data? */
+ rc = 0;
+ goto out;
+ }
+
+ if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages)
+ tag_pages_for_writeback(mapping, ceph_wbc.index, ceph_wbc.end);
+
+ while (!has_writeback_done(&ceph_wbc)) {
+ ceph_wbc.locked_pages = 0;
+ ceph_wbc.max_pages = ceph_wbc.wsize >> PAGE_SHIFT;
+
+get_more_pages:
+ ceph_folio_batch_reinit(&ceph_wbc);
+
+ ceph_wbc.nr_folios = filemap_get_folios_tag(mapping,
+ &ceph_wbc.index,
+ ceph_wbc.end,
+ ceph_wbc.tag,
+ &ceph_wbc.fbatch);
+ doutc(cl, "pagevec_lookup_range_tag for tag %#x got %d\n",
+ ceph_wbc.tag, ceph_wbc.nr_folios);
+
+ if (!ceph_wbc.nr_folios && !ceph_wbc.locked_pages)
+ break;
+
+process_folio_batch:
+ rc = ceph_process_folio_batch(mapping, wbc, &ceph_wbc);
+ ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
+ if (rc)
+ goto release_folios;
+
+ /* did we get anything? */
+ if (!ceph_wbc.locked_pages)
+ goto release_folios;
+
+ if (ceph_wbc.processed_in_fbatch) {
+ if (folio_batch_count(&ceph_wbc.fbatch) == 0 &&
+ ceph_wbc.locked_pages < ceph_wbc.max_pages) {
+ doutc(cl, "reached end fbatch, trying for more\n");
+ goto get_more_pages;
}
- memcpy(pages, data_pages + i,
- locked_pages * sizeof(*pages));
- memset(data_pages + i, 0,
- locked_pages * sizeof(*pages));
- } else {
- BUG_ON(num_ops != req->r_num_ops);
- index = pages[i - 1]->index + 1;
- /* request message now owns the pages array */
- pages = NULL;
}
- req->r_mtime = inode_get_mtime(inode);
- ceph_osdc_start_request(&fsc->client->osdc, req);
- req = NULL;
+ rc = ceph_submit_write(mapping, wbc, &ceph_wbc);
+ if (rc)
+ goto release_folios;
- wbc->nr_to_write -= i;
- if (pages)
- goto new_request;
+ ceph_wbc.locked_pages = 0;
+ ceph_wbc.strip_unit_end = 0;
+
+ if (folio_batch_count(&ceph_wbc.fbatch) > 0) {
+ ceph_wbc.nr_folios =
+ folio_batch_count(&ceph_wbc.fbatch);
+ goto process_folio_batch;
+ }
/*
* We stop writing back only if we are not doing
@@ -1365,61 +1718,41 @@ static int ceph_writepages_start(struct address_space *mapping,
* we tagged for writeback prior to entering this loop.
*/
if (wbc->nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE)
- done = true;
+ ceph_wbc.done = true;
release_folios:
doutc(cl, "folio_batch release on %d folios (%p)\n",
- (int)fbatch.nr, fbatch.nr ? fbatch.folios[0] : NULL);
- folio_batch_release(&fbatch);
+ (int)ceph_wbc.fbatch.nr,
+ ceph_wbc.fbatch.nr ? ceph_wbc.fbatch.folios[0] : NULL);
+ folio_batch_release(&ceph_wbc.fbatch);
}
- if (should_loop && !done) {
+ if (ceph_wbc.should_loop && !ceph_wbc.done) {
/* more to do; loop back to beginning of file */
doutc(cl, "looping back to beginning of file\n");
- end = start_index - 1; /* OK even when start_index == 0 */
+ /* OK even when start_index == 0 */
+ ceph_wbc.end = ceph_wbc.start_index - 1;
/* to write dirty pages associated with next snapc,
* we need to wait until current writes complete */
- if (wbc->sync_mode != WB_SYNC_NONE &&
- start_index == 0 && /* all dirty pages were checked */
- !ceph_wbc.head_snapc) {
- struct page *page;
- unsigned i, nr;
- index = 0;
- while ((index <= end) &&
- (nr = filemap_get_folios_tag(mapping, &index,
- (pgoff_t)-1,
- PAGECACHE_TAG_WRITEBACK,
- &fbatch))) {
- for (i = 0; i < nr; i++) {
- page = &fbatch.folios[i]->page;
- if (page_snap_context(page) != snapc)
- continue;
- wait_on_page_writeback(page);
- }
- folio_batch_release(&fbatch);
- cond_resched();
- }
- }
+ ceph_wait_until_current_writes_complete(mapping, wbc, &ceph_wbc);
- start_index = 0;
- index = 0;
+ ceph_wbc.start_index = 0;
+ ceph_wbc.index = 0;
goto retry;
}
- if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
- mapping->writeback_index = index;
+ if (wbc->range_cyclic || (ceph_wbc.range_whole && wbc->nr_to_write > 0))
+ mapping->writeback_index = ceph_wbc.index;
out:
- ceph_osdc_put_request(req);
- ceph_put_snap_context(last_snapc);
+ ceph_put_snap_context(ceph_wbc.last_snapc);
doutc(cl, "%llx.%llx dend - startone, rc = %d\n", ceph_vinop(inode),
rc);
+
return rc;
}
-
-
/*
* See if a given @snapc is either writeable, or already written.
*/
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index bc1d4386b349..6694aac76a2d 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -574,10 +574,36 @@ static int parse_reply_info_filelock(void **p, void *end,
#define DELEGATED_INO_AVAILABLE xa_mk_value(1)
+static int ceph_insert_deleg_ino(struct ceph_mds_session *s, u64 ino)
+{
+ struct ceph_client *cl = s->s_mdsc->fsc->client;
+ int err;
+
+ /*
+ * Cap how many delegated inodes a single session may hold. This is
+ * the only place that grows the count, so atomic_add_unless() bounds
+ * it at exactly CEPH_MAX_DELEG_INOS; s_num_deleg_inos can never exceed
+ * that.
+ */
+ if (!atomic_add_unless(&s->s_num_deleg_inos, 1, CEPH_MAX_DELEG_INOS)) {
+ pr_warn_ratelimited_client(cl,
+ "MDS session already holds %d delegated inodes\n",
+ CEPH_MAX_DELEG_INOS);
+ return -EOVERFLOW;
+ }
+
+ err = xa_insert(&s->s_delegated_inos, ino, DELEGATED_INO_AVAILABLE,
+ GFP_KERNEL);
+ if (err)
+ atomic_dec(&s->s_num_deleg_inos);
+ return err;
+}
+
static int ceph_parse_deleg_inos(void **p, void *end,
struct ceph_mds_session *s)
{
struct ceph_client *cl = s->s_mdsc->fsc->client;
+ u64 msg_deleg_inos = 0;
u32 sets;
ceph_decode_32_safe(p, end, sets, bad);
@@ -595,16 +621,34 @@ static int ceph_parse_deleg_inos(void **p, void *end,
start, len);
continue;
}
+
+ /*
+ * Bound the number of inodes one reply may delegate.
+ * ceph_insert_deleg_ino() separately caps the per-session
+ * population, so this only has to stop one reply from spinning
+ * the insert loop under an attacker-controlled len.
+ */
+ if (len > (u64)CEPH_MAX_DELEG_INOS ||
+ msg_deleg_inos > (u64)CEPH_MAX_DELEG_INOS - len) {
+ pr_warn_ratelimited_client(cl,
+ "MDS reply delegates too many inodes (have %llu, +%llu, max %d)\n",
+ msg_deleg_inos, len, CEPH_MAX_DELEG_INOS);
+ return -EIO;
+ }
+ msg_deleg_inos += len;
+
while (len--) {
- int err = xa_insert(&s->s_delegated_inos, start++,
- DELEGATED_INO_AVAILABLE,
- GFP_KERNEL);
+ int err = ceph_insert_deleg_ino(s, start++);
+
if (!err) {
doutc(cl, "added delegated inode 0x%llx\n", start - 1);
} else if (err == -EBUSY) {
pr_warn_client(cl,
"MDS delegated inode 0x%llx more than once.\n",
start - 1);
+ } else if (err == -EOVERFLOW) {
+ /* ceph_insert_deleg_ino() already warned. */
+ return -EIO;
} else {
return err;
}
@@ -622,16 +666,17 @@ u64 ceph_get_deleg_ino(struct ceph_mds_session *s)
xa_for_each(&s->s_delegated_inos, ino, val) {
val = xa_erase(&s->s_delegated_inos, ino);
- if (val == DELEGATED_INO_AVAILABLE)
+ if (val == DELEGATED_INO_AVAILABLE) {
+ atomic_dec(&s->s_num_deleg_inos);
return ino;
+ }
}
return 0;
}
int ceph_restore_deleg_ino(struct ceph_mds_session *s, u64 ino)
{
- return xa_insert(&s->s_delegated_inos, ino, DELEGATED_INO_AVAILABLE,
- GFP_KERNEL);
+ return ceph_insert_deleg_ino(s, ino);
}
#else /* BITS_PER_LONG == 64 */
/*
@@ -1015,6 +1060,7 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc,
INIT_LIST_HEAD(&s->s_waiting);
INIT_LIST_HEAD(&s->s_unsafe);
xa_init(&s->s_delegated_inos);
+ atomic_set(&s->s_num_deleg_inos, 0);
INIT_LIST_HEAD(&s->s_cap_releases);
INIT_WORK(&s->s_cap_release_work, ceph_cap_release_work);
@@ -4923,6 +4969,7 @@ static int send_mds_reconnect(struct ceph_mds_client *mdsc,
/* Serialized by s_mutex against concurrent ceph_get_deleg_ino(). */
xa_destroy(&session->s_delegated_inos);
+ atomic_set(&session->s_num_deleg_inos, 0);
if (session->s_state == CEPH_MDS_SESSION_CLOSED ||
session->s_state == CEPH_MDS_SESSION_REJECTED) {
pr_info_client(cl, "mds%d skipping reconnect, session %s\n",
@@ -5172,9 +5219,11 @@ static void check_new_map(struct ceph_mds_client *mdsc,
ceph_mdsmap_get_addr(newmap, i),
sizeof(struct ceph_entity_addr))) {
/* just close it */
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
mutex_lock(&s->s_mutex);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
ceph_con_close(&s->s_con);
mutex_unlock(&s->s_mutex);
s->s_state = CEPH_MDS_SESSION_RESTARTING;
@@ -5189,6 +5238,7 @@ static void check_new_map(struct ceph_mds_client *mdsc,
newstate >= CEPH_MDS_STATE_RECONNECT) {
int rc;
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
clear_bit(i, targets);
rc = send_mds_reconnect(mdsc, s);
@@ -5197,6 +5247,7 @@ static void check_new_map(struct ceph_mds_client *mdsc,
"mds%d reconnect failed: %d\n",
i, rc);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
}
/*
@@ -5209,9 +5260,11 @@ static void check_new_map(struct ceph_mds_client *mdsc,
pr_info_client(cl, "mds%d recovery completed\n",
s->s_mds);
kick_requests(mdsc, i);
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
mutex_lock(&s->s_mutex);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
ceph_kick_flushing_caps(mdsc, s);
mutex_unlock(&s->s_mutex);
wake_up_session_caps(s, RECONNECT);
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index 0e82e61e19c4..a1629e61e75b 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -252,6 +252,7 @@ struct ceph_mds_session {
struct list_head s_waiting; /* waiting requests */
struct list_head s_unsafe; /* unsafe requests */
struct xarray s_delegated_inos;
+ atomic_t s_num_deleg_inos;
};
/*
diff --git a/fs/ceph/super.h b/fs/ceph/super.h
index c0763e2f54d7..83ddd61e1647 100644
--- a/fs/ceph/super.h
+++ b/fs/ceph/super.h
@@ -623,6 +623,15 @@ static inline int ceph_ino_compare(struct inode *inode, void *data)
#define CEPH_MDS_INO_LOG_OFFSET (2 * CEPH_MAX_MDS)
#define CEPH_INO_SYSTEM_BASE ((6*CEPH_MAX_MDS) + (CEPH_MAX_MDS * CEPH_NUM_STRAY))
+/*
+ * Upper bound on the number of delegated inodes a single MDS session may
+ * hold. The MDS normally hands out a small preallocation window (the
+ * userspace mds_client_prealloc_inos option defaults to 1000) and refills
+ * it as the client consumes entries. This leaves generous headroom while
+ * bounding the CPU and memory a malformed delegation interval can consume.
+ */
+#define CEPH_MAX_DELEG_INOS 8192
+
static inline bool ceph_vino_is_reserved(const struct ceph_vino vino)
{
if (vino.ino >= CEPH_INO_SYSTEM_BASE ||
diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c
index 39695addce07..933347416513 100644
--- a/fs/erofs/decompressor.c
+++ b/fs/erofs/decompressor.c
@@ -7,11 +7,6 @@
#include "compress.h"
#include <linux/lz4.h>
-#ifndef LZ4_DISTANCE_MAX /* history window size */
-#define LZ4_DISTANCE_MAX 65535 /* set to maximum value by default */
-#endif
-
-#define LZ4_MAX_DISTANCE_PAGES (DIV_ROUND_UP(LZ4_DISTANCE_MAX, PAGE_SIZE) + 1)
#ifndef LZ4_DECOMPRESS_INPLACE_MARGIN
#define LZ4_DECOMPRESS_INPLACE_MARGIN(srcsize) (((srcsize) >> 8) + 32)
#endif
@@ -21,15 +16,12 @@ static int z_erofs_load_lz4_config(struct super_block *sb,
{
struct erofs_sb_info *sbi = EROFS_SB(sb);
struct z_erofs_lz4_cfgs *lz4 = data;
- u16 distance;
if (lz4) {
if (size < sizeof(struct z_erofs_lz4_cfgs)) {
erofs_err(sb, "invalid lz4 cfgs, size=%u", size);
return -EINVAL;
}
- distance = le16_to_cpu(lz4->max_distance);
-
sbi->lz4.max_pclusterblks = le16_to_cpu(lz4->max_pclusterblks);
if (!sbi->lz4.max_pclusterblks) {
sbi->lz4.max_pclusterblks = 1; /* reserved case */
@@ -40,48 +32,27 @@ static int z_erofs_load_lz4_config(struct super_block *sb,
return -EINVAL;
}
} else {
- distance = le16_to_cpu(dsb->u1.lz4_max_distance);
sbi->lz4.max_pclusterblks = 1;
}
-
- sbi->lz4.max_distance_pages = distance ?
- DIV_ROUND_UP(distance, PAGE_SIZE) + 1 :
- LZ4_MAX_DISTANCE_PAGES;
return z_erofs_gbuf_growsize(sbi->lz4.max_pclusterblks);
}
/*
- * Fill all gaps with bounce pages if it's a sparse page list. Also check if
- * all physical pages are consecutive, which can be seen for moderate CR.
+ * Fill all gaps with bounce pages if it's a sparse page list (for example some
+ * folios are already uptodate and thus can be mapped into userspace). Also
+ * check if pages are physically consecutive, which can be seen for moderate CR.
*/
-static int z_erofs_lz4_prepare_dstpages(struct z_erofs_decompress_req *rq,
- struct page **pagepool)
+static int z_erofs_oneshot_prepare_dstpages(struct z_erofs_decompress_req *rq,
+ struct page **pagepool)
{
- struct page *availables[LZ4_MAX_DISTANCE_PAGES] = { NULL };
- unsigned long bounced[DIV_ROUND_UP(LZ4_MAX_DISTANCE_PAGES,
- BITS_PER_LONG)] = { 0 };
- unsigned int lz4_max_distance_pages =
- EROFS_SB(rq->sb)->lz4.max_distance_pages;
void *kaddr = NULL;
- unsigned int i, j, top;
-
- top = 0;
- for (i = j = 0; i < rq->outpages; ++i, ++j) {
- struct page *const page = rq->out[i];
- struct page *victim;
-
- if (j >= lz4_max_distance_pages)
- j = 0;
+ unsigned int i;
- /* 'valid' bounced can only be tested after a complete round */
- if (!rq->fillgaps && test_bit(j, bounced)) {
- DBG_BUGON(i < lz4_max_distance_pages);
- DBG_BUGON(top >= lz4_max_distance_pages);
- availables[top++] = rq->out[i - lz4_max_distance_pages];
- }
+ for (i = 0; i < rq->outpages; ++i) {
+ struct page *page, *victim;
+ page = rq->out[i];
if (page) {
- __clear_bit(j, bounced);
if (!PageHighMem(page)) {
if (!i) {
kaddr = page_address(page);
@@ -93,21 +64,14 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_decompress_req *rq,
continue;
}
}
- kaddr = NULL;
- continue;
- }
- kaddr = NULL;
- __set_bit(j, bounced);
-
- if (top) {
- victim = availables[--top];
} else {
victim = __erofs_allocpage(pagepool, rq->gfp, true);
if (!victim)
return -ENOMEM;
set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE);
+ rq->out[i] = victim;
}
- rq->out[i] = victim;
+ kaddr = NULL;
}
return kaddr ? 1 : 0;
}
@@ -279,7 +243,7 @@ static int z_erofs_lz4_decompress(struct z_erofs_decompress_req *rq,
dst_maptype = 0;
} else {
/* general decoding path which can be used for all cases */
- ret = z_erofs_lz4_prepare_dstpages(rq, pagepool);
+ ret = z_erofs_oneshot_prepare_dstpages(rq, pagepool);
if (ret < 0)
return ret;
if (ret > 0) {
@@ -312,6 +276,7 @@ static int z_erofs_transform_plain(struct z_erofs_decompress_req *rq,
return -EOPNOTSUPP;
if (rq->alg == Z_EROFS_COMPRESSION_INTERLACED) {
cur = bs - (rq->pageofs_out & (bs - 1));
+ DBG_BUGON(rq->pageofs_in & (bs - 1));
pi = (rq->pageofs_in + rq->inputsize - cur) & ~PAGE_MASK;
cur = min(cur, rq->outputsize);
if (cur && rq->out[0]) {
diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h
index 89dfb3736daa..ed023f14acd3 100644
--- a/fs/erofs/internal.h
+++ b/fs/erofs/internal.h
@@ -77,12 +77,8 @@ struct erofs_dev_context {
bool flatdev;
};
-/* all filesystem-wide lz4 configurations */
struct erofs_sb_lz4_info {
- /* # of pages needed for EROFS lz4 rolling decompression */
- u16 max_distance_pages;
- /* maximum possible blocks for pclusters in the filesystem */
- u16 max_pclusterblks;
+ u16 max_pclusterblks; /* maximum physical blocks for LZ4 pclusters */
};
struct erofs_domain {
diff --git a/fs/erofs/xattr.c b/fs/erofs/xattr.c
index 60d2cf26e837..9753df4e95a2 100644
--- a/fs/erofs/xattr.c
+++ b/fs/erofs/xattr.c
@@ -47,6 +47,8 @@ static int erofs_init_inode_xattrs(struct inode *inode)
if (wait_on_bit_lock(&vi->flags, EROFS_I_BL_XATTR_BIT, TASK_KILLABLE))
return -ERESTARTSYS;
+ it.buf = __EROFS_BUF_INITIALIZER;
+
/* someone has initialized xattrs for us? */
if (test_bit(EROFS_I_EA_INITED_BIT, &vi->flags))
goto out_unlock;
@@ -76,7 +78,6 @@ static int erofs_init_inode_xattrs(struct inode *inode)
goto out_unlock;
}
- it.buf = __EROFS_BUF_INITIALIZER;
erofs_init_metabuf(&it.buf, sb);
it.pos = erofs_iloc(inode) + vi->inode_isize;
@@ -93,7 +94,6 @@ static int erofs_init_inode_xattrs(struct inode *inode)
vi->xattr_shared_xattrs = kmalloc_array(vi->xattr_shared_count,
sizeof(uint), GFP_KERNEL);
if (!vi->xattr_shared_xattrs) {
- erofs_put_metabuf(&it.buf);
ret = -ENOMEM;
goto out_unlock;
}
@@ -112,13 +112,13 @@ static int erofs_init_inode_xattrs(struct inode *inode)
vi->xattr_shared_xattrs[i] = le32_to_cpu(*(__le32 *)it.kaddr);
it.pos += sizeof(__le32);
}
- erofs_put_metabuf(&it.buf);
/* paired with smp_mb() at the beginning of the function. */
smp_mb();
set_bit(EROFS_I_EA_INITED_BIT, &vi->flags);
out_unlock:
+ erofs_put_metabuf(&it.buf);
clear_and_wake_up_bit(EROFS_I_BL_XATTR_BIT, &vi->flags);
return ret;
}
diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c
index d36562c4afff..08720efbb1a1 100644
--- a/fs/erofs/zdata.c
+++ b/fs/erofs/zdata.c
@@ -1237,7 +1237,7 @@ static int z_erofs_decompress_pcluster(struct z_erofs_backend *be, bool eio)
unsigned int pclusterpages = z_erofs_pclusterpages(pcl);
const struct z_erofs_decompressor *decomp =
z_erofs_decomp[pcl->algorithmformat];
- int i, j, jtop, err2, err = eio ? -EIO : 0;
+ int i, err2, err = eio ? -EIO : 0;
struct page *page;
bool overlapped;
bool try_free = true;
@@ -1314,7 +1314,6 @@ static int z_erofs_decompress_pcluster(struct z_erofs_backend *be, bool eio)
be->compressed_pages >= be->onstack_pages + Z_EROFS_ONSTACK_PAGES)
kvfree(be->compressed_pages);
- jtop = 0;
z_erofs_fill_other_copies(be, err);
for (i = 0; i < be->nr_pages; ++i) {
page = be->decompressed_pages[i];
@@ -1322,22 +1321,11 @@ static int z_erofs_decompress_pcluster(struct z_erofs_backend *be, bool eio)
continue;
DBG_BUGON(z_erofs_page_is_invalidated(page));
- if (!z_erofs_is_shortlived_page(page)) {
+ if (!z_erofs_is_shortlived_page(page))
erofs_onlinefolio_end(page_folio(page), err, true);
- continue;
- }
- if (pcl->algorithmformat != Z_EROFS_COMPRESSION_LZ4) {
+ else
erofs_pagepool_add(be->pagepool, page);
- continue;
- }
- for (j = 0; j < jtop && be->decompressed_pages[j] != page; ++j)
- ;
- if (j >= jtop) /* this bounce page is newly detected */
- be->decompressed_pages[jtop++] = page;
}
- while (jtop)
- erofs_pagepool_add(be->pagepool,
- be->decompressed_pages[--jtop]);
if (be->decompressed_pages != be->onstack_pages)
kvfree(be->decompressed_pages);
diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c
index af1046908c62..eb14dda612a9 100644
--- a/fs/erofs/zmap.c
+++ b/fs/erofs/zmap.c
@@ -477,9 +477,11 @@ static int z_erofs_do_map_blocks(struct inode *inode,
err = -EFSCORRUPTED;
goto unmap_out;
}
- afmt = vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER ?
- Z_EROFS_COMPRESSION_INTERLACED :
- Z_EROFS_COMPRESSION_SHIFTED;
+ if ((vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER) &&
+ !(map->m_flags & EROFS_MAP_META))
+ afmt = Z_EROFS_COMPRESSION_INTERLACED;
+ else
+ afmt = Z_EROFS_COMPRESSION_SHIFTED;
} else {
afmt = m.headtype == Z_EROFS_LCLUSTER_TYPE_HEAD2 ?
vi->z_algorithmtype[1] : vi->z_algorithmtype[0];
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 3fcf8191da42..f3149022257a 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -1300,11 +1300,10 @@ static int ext2_setsize(struct inode *inode, loff_t newsize)
filemap_invalidate_unlock(inode->i_mapping);
inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode));
+ mark_inode_dirty(inode);
if (inode_needs_sync(inode)) {
sync_mapping_buffers(inode->i_mapping);
sync_inode_metadata(inode, 1);
- } else {
- mark_inode_dirty(inode);
}
return 0;
diff --git a/fs/ext2/xattr.c b/fs/ext2/xattr.c
index c885dcc3bd0d..5574b601ae09 100644
--- a/fs/ext2/xattr.c
+++ b/fs/ext2/xattr.c
@@ -777,6 +777,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh,
/* Update the inode. */
EXT2_I(inode)->i_file_acl = new_bh ? new_bh->b_blocknr : 0;
inode_set_ctime_current(inode);
+ mark_inode_dirty(inode);
if (IS_SYNC(inode)) {
error = sync_inode_metadata(inode, 1);
/* In case sync failed due to ENOSPC the inode was actually
@@ -789,8 +790,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh,
}
goto cleanup;
}
- } else
- mark_inode_dirty(inode);
+ }
error = 0;
if (old_bh && old_bh != new_bh) {
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index fc6e437d3e81..21de942bb6fb 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -2984,7 +2984,8 @@ typedef enum {
EXT4_IGET_SPECIAL = 0x0001, /* OK to iget a system inode */
EXT4_IGET_HANDLE = 0x0002, /* Inode # is from a handle */
EXT4_IGET_BAD = 0x0004, /* Allow to iget a bad inode */
- EXT4_IGET_EA_INODE = 0x0008 /* Inode should contain an EA value */
+ EXT4_IGET_EA_INODE = 0x0008, /* Inode should contain an EA value */
+ EXT4_IGET_NOWAIT = 0x0010 /* Non-blocking lookup (skip if freeing) */
} ext4_iget_flags;
extern struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 24b5925a7c7e..822d81d51f23 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -4783,6 +4783,20 @@ static int check_igot_inode(struct inode *inode, ext4_iget_flags flags,
return -EFSCORRUPTED;
}
+static int ext4_iget_match(struct inode *inode, unsigned long ino, void *data)
+{
+ if (inode->i_ino != ino)
+ return 0;
+ spin_lock(&inode->i_lock);
+ if (inode->i_state & (I_FREEING | I_WILL_FREE | I_CREATING)) {
+ spin_unlock(&inode->i_lock);
+ return -1;
+ }
+ __iget(inode);
+ spin_unlock(&inode->i_lock);
+ return 1;
+}
+
struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
ext4_iget_flags flags, const char *function,
unsigned int line)
@@ -4816,9 +4830,24 @@ struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
return ERR_PTR(-EFSCORRUPTED);
}
- inode = iget_locked(sb, ino);
- if (!inode)
- return ERR_PTR(-ENOMEM);
+ if (flags & EXT4_IGET_NOWAIT) {
+ inode = find_inode_nowait(sb, ino, ext4_iget_match, NULL);
+ if (!inode)
+ return ERR_PTR(-ENOENT);
+
+ if (inode->i_state & I_NEW)
+ wait_on_inode(inode);
+
+ if (unlikely(inode_unhashed(inode))) {
+ iput(inode);
+ return ERR_PTR(-ENOENT);
+ }
+ } else {
+ inode = iget_locked(sb, ino);
+ if (!inode)
+ return ERR_PTR(-ENOMEM);
+ }
+
if (!(inode->i_state & I_NEW)) {
ret = check_igot_inode(inode, flags, function, line);
if (ret) {
diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index 7f1af018641c..6887c22bca0c 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -1550,7 +1550,7 @@ ext4_xattr_inode_cache_find(struct inode *inode, const void *value,
while (ce) {
ea_inode = ext4_iget(inode->i_sb, ce->e_value,
- EXT4_IGET_EA_INODE);
+ EXT4_IGET_EA_INODE | EXT4_IGET_NOWAIT);
if (IS_ERR(ea_inode))
goto next_entry;
ext4_xattr_inode_set_class(ea_inode);
diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 6322bf2e1622..dc9540400caa 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -1249,7 +1249,7 @@ static int block_operations(struct f2fs_sb_info *sbi)
retry_flush_quotas:
f2fs_lock_all(sbi);
if (__need_flush_quota(sbi)) {
- int locked;
+ bool need_lock = sbi->umount_lock_holder != current;
if (++cnt > DEFAULT_RETRY_QUOTA_FLUSH_COUNT) {
set_sbi_flag(sbi, SBI_QUOTA_SKIP_FLUSH);
@@ -1258,11 +1258,13 @@ static int block_operations(struct f2fs_sb_info *sbi)
}
f2fs_unlock_all(sbi);
- /* only failed during mount/umount/freeze/quotactl */
- locked = down_read_trylock(&sbi->sb->s_umount);
- f2fs_quota_sync(sbi->sb, -1);
- if (locked)
+ /* don't grab s_umount lock during mount/umount/remount/freeze/quotactl */
+ if (!need_lock) {
+ f2fs_do_quota_sync(sbi->sb, -1);
+ } else if (down_read_trylock(&sbi->sb->s_umount)) {
+ f2fs_do_quota_sync(sbi->sb, -1);
up_read(&sbi->sb->s_umount);
+ }
cond_resched();
goto retry_flush_quotas;
}
@@ -1870,7 +1872,8 @@ int f2fs_issue_checkpoint(struct f2fs_sb_info *sbi)
struct cp_control cpc;
cpc.reason = __get_cp_reason(sbi);
- if (!test_opt(sbi, MERGE_CHECKPOINT) || cpc.reason != CP_SYNC) {
+ if (!test_opt(sbi, MERGE_CHECKPOINT) || cpc.reason != CP_SYNC ||
+ sbi->umount_lock_holder == current) {
int ret;
f2fs_down_write(&sbi->gc_lock);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index f8154a431fc9..4bbaae47b184 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -3593,7 +3593,7 @@ static int __reserve_data_block(struct inode *inode, pgoff_t index,
static int prepare_atomic_write_begin(struct f2fs_sb_info *sbi,
struct folio *folio, loff_t pos, unsigned int len,
- block_t *blk_addr, bool *node_changed, bool *use_cow)
+ block_t *blk_addr, bool *node_changed)
{
struct inode *inode = folio->mapping->host;
struct inode *cow_inode = F2FS_I(inode)->cow_inode;
@@ -3608,14 +3608,14 @@ static int prepare_atomic_write_begin(struct f2fs_sb_info *sbi,
/* Look for the block in COW inode first */
err = __find_data_block(cow_inode, index, blk_addr);
- if (err) {
+ if (err)
return err;
- } else if (__is_valid_data_blkaddr(*blk_addr)) {
- *use_cow = true;
+
+ if (__is_valid_data_blkaddr(*blk_addr))
return 0;
- } else if (*blk_addr == NEW_ADDR) {
+
+ if (*blk_addr == NEW_ADDR)
cow_has_reserved_block = true;
- }
if (is_inode_flag_set(inode, FI_ATOMIC_REPLACE))
goto reserve_block;
@@ -3648,7 +3648,6 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
struct folio *folio;
pgoff_t index = pos >> PAGE_SHIFT;
bool need_balance = false;
- bool use_cow = false;
block_t blkaddr = NULL_ADDR;
int err = 0;
@@ -3710,7 +3709,7 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
if (f2fs_is_atomic_file(inode))
err = prepare_atomic_write_begin(sbi, folio, pos, len,
- &blkaddr, &need_balance, &use_cow);
+ &blkaddr, &need_balance);
else
err = prepare_write_begin(sbi, folio, pos, len,
&blkaddr, &need_balance);
@@ -3750,9 +3749,15 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
err = -EFSCORRUPTED;
goto put_folio;
}
- err = f2fs_submit_page_read(use_cow ?
- F2FS_I(inode)->cow_inode : inode,
- folio, blkaddr, 0, true);
+ /*
+ * Although the block may be stored in the COW inode, the folio
+ * belongs to @inode and its data was encrypted (or not) using
+ * @inode's context (see f2fs_encrypt_one_page()). Read with
+ * @inode so the post-read decryption decision matches the
+ * folio's owner; otherwise an unencrypted @inode whose COW inode
+ * is encrypted hits a NULL ->i_crypt_info on decryption.
+ */
+ err = f2fs_submit_page_read(inode, folio, blkaddr, 0, true);
if (err)
goto put_folio;
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 12ca7ba9d6dc..81c2a566f1d4 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -1708,6 +1708,7 @@ struct f2fs_sb_info {
unsigned int nquota_files; /* # of quota sysfile */
struct f2fs_rwsem quota_sem; /* blocking cp for flags */
+ struct task_struct *umount_lock_holder; /* s_umount lock holder */
/* # of pages, see count_type */
atomic_t nr_pages[NR_COUNT_TYPE];
@@ -3728,7 +3729,7 @@ int f2fs_inode_dirtied(struct inode *inode, bool sync);
void f2fs_inode_synced(struct inode *inode);
int f2fs_dquot_initialize(struct inode *inode);
int f2fs_enable_quota_files(struct f2fs_sb_info *sbi, bool rdonly);
-int f2fs_quota_sync(struct super_block *sb, int type);
+int f2fs_do_quota_sync(struct super_block *sb, int type);
loff_t max_file_blocks(struct inode *inode);
void f2fs_quota_off_umount(struct super_block *sb);
void f2fs_save_errors(struct f2fs_sb_info *sbi, unsigned char flag);
diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
index 6c831276d81d..6924876f1c2f 100644
--- a/fs/f2fs/gc.c
+++ b/fs/f2fs/gc.c
@@ -1222,7 +1222,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
.encrypted_page = NULL,
.in_list = 0,
};
- int err;
+ int err = 0;
f2fs_down_read(&F2FS_I(inode)->i_sem);
if (f2fs_is_cow_file(inode)) {
@@ -1287,6 +1287,9 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
goto put_page;
}
+ if (PageUptodate(fio.encrypted_page))
+ goto put_encrypted_page;
+
err = f2fs_submit_page_bio(&fio);
if (err)
goto put_encrypted_page;
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index 0898ce801f28..8a11623dbf1e 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -419,6 +419,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
struct nat_entry *e;
bool need_update = true;
+ f2fs_down_read(&sbi->node_write);
f2fs_down_read(&nm_i->nat_tree_lock);
e = __lookup_nat_cache(nm_i, ino);
if (e && get_nat_flag(e, HAS_LAST_FSYNC) &&
@@ -426,6 +427,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
get_nat_flag(e, HAS_FSYNCED_INODE)))
need_update = false;
f2fs_up_read(&nm_i->nat_tree_lock);
+ f2fs_up_read(&sbi->node_write);
return need_update;
}
@@ -1643,9 +1645,10 @@ static struct page *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino)
return last_page;
}
-static int __write_node_page(struct page *page, bool atomic, bool *submitted,
- struct writeback_control *wbc, bool do_balance,
- enum iostat_type io_type, unsigned int *seq_id)
+static int __write_node_page(struct page *page, bool atomic, bool do_fsync,
+ bool *submitted, struct writeback_control *wbc,
+ bool do_balance, enum iostat_type io_type,
+ unsigned int *seq_id)
{
struct f2fs_sb_info *sbi = F2FS_P_SB(page);
struct folio *folio = page_folio(page);
@@ -1715,13 +1718,14 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
goto redirty_out;
}
- if (atomic) {
- if (!test_opt(sbi, NOBARRIER))
- fio.op_flags |= REQ_PREFLUSH | REQ_FUA;
- if (IS_INODE(page))
- set_dentry_mark(page,
+ if (atomic && !test_opt(sbi, NOBARRIER))
+ fio.op_flags |= REQ_PREFLUSH | REQ_FUA;
+
+ set_dentry_mark(page, false);
+ set_fsync_mark(page, do_fsync);
+ if (IS_INODE(page) && (atomic || is_fsync_dnode(page)))
+ set_dentry_mark(page,
f2fs_need_dentry_mark(sbi, ino_of_node(page)));
- }
/* should add to global list before clearing PAGECACHE status */
if (f2fs_in_warm_node_list(sbi, page)) {
@@ -1781,7 +1785,7 @@ int f2fs_move_node_page(struct page *node_page, int gc_type)
goto out_page;
}
- if (__write_node_page(node_page, false, NULL,
+ if (__write_node_page(node_page, false, false, NULL,
&wbc, false, FS_GC_NODE_IO, NULL)) {
err = -EAGAIN;
unlock_page(node_page);
@@ -1802,7 +1806,7 @@ int f2fs_move_node_page(struct page *node_page, int gc_type)
static int f2fs_write_node_page(struct page *page,
struct writeback_control *wbc)
{
- return __write_node_page(page, false, NULL, wbc, false,
+ return __write_node_page(page, false, false, NULL, wbc, false,
FS_NODE_IO, NULL);
}
@@ -1836,6 +1840,7 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
for (i = 0; i < nr_folios; i++) {
struct page *page = &fbatch.folios[i]->page;
bool submitted = false;
+ bool do_fsync = false;
if (unlikely(f2fs_cp_error(sbi))) {
f2fs_put_page(last_page, 0);
@@ -1866,19 +1871,13 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
f2fs_wait_on_page_writeback(page, NODE, true, true);
- set_fsync_mark(page, 0);
- set_dentry_mark(page, 0);
-
if (!atomic || page == last_page) {
- set_fsync_mark(page, 1);
+ do_fsync = true;
percpu_counter_inc(&sbi->rf_node_block_count);
if (IS_INODE(page)) {
if (is_inode_flag_set(inode,
FI_DIRTY_INODE))
f2fs_update_inode(inode, page);
- if (!atomic)
- set_dentry_mark(page,
- f2fs_need_dentry_mark(sbi, ino));
}
/* may be written by other thread */
if (!PageDirty(page))
@@ -1890,8 +1889,9 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
ret = __write_node_page(page, atomic &&
page == last_page,
- &submitted, wbc, true,
- FS_NODE_IO, seq_id);
+ do_fsync, &submitted,
+ wbc, true, FS_NODE_IO,
+ seq_id);
if (ret) {
unlock_page(page);
f2fs_put_page(last_page, 0);
@@ -1916,6 +1916,11 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
f2fs_debug(sbi, "Retry to write fsync mark: ino=%u, idx=%lx",
ino, page_folio(last_page)->index);
lock_page(last_page);
+ if (unlikely(last_page->mapping != NODE_MAPPING(sbi))) {
+ f2fs_put_page(last_page, 1);
+ ret = -EAGAIN;
+ goto out;
+ }
f2fs_wait_on_page_writeback(last_page, NODE, true, true);
set_page_dirty(last_page);
unlock_page(last_page);
@@ -2098,10 +2103,7 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi,
if (!clear_page_dirty_for_io(page))
goto continue_unlock;
- set_fsync_mark(page, 0);
- set_dentry_mark(page, 0);
-
- ret = __write_node_page(page, false, &submitted,
+ ret = __write_node_page(page, false, false, &submitted,
wbc, do_balance, io_type, NULL);
if (ret)
unlock_page(page);
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index f7acb1998b2c..c278ce784205 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -1736,22 +1736,28 @@ int f2fs_sync_fs(struct super_block *sb, int sync)
static int f2fs_freeze(struct super_block *sb)
{
+ struct f2fs_sb_info *sbi = F2FS_SB(sb);
+
if (f2fs_readonly(sb))
return 0;
/* IO error happened before */
- if (unlikely(f2fs_cp_error(F2FS_SB(sb))))
+ if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
/* must be clean, since sync_filesystem() was already called */
- if (is_sbi_flag_set(F2FS_SB(sb), SBI_IS_DIRTY))
+ if (is_sbi_flag_set(sbi, SBI_IS_DIRTY))
return -EINVAL;
+ sbi->umount_lock_holder = current;
+
/* Let's flush checkpoints and stop the thread. */
- f2fs_flush_ckpt_thread(F2FS_SB(sb));
+ f2fs_flush_ckpt_thread(sbi);
+
+ sbi->umount_lock_holder = NULL;
/* to avoid deadlock on f2fs_evict_inode->SB_FREEZE_FS */
- set_sbi_flag(F2FS_SB(sb), SBI_IS_FREEZING);
+ set_sbi_flag(sbi, SBI_IS_FREEZING);
return 0;
}
@@ -2363,6 +2369,8 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
org_mount_opt = sbi->mount_opt;
old_sb_flags = sb->s_flags;
+ sbi->umount_lock_holder = current;
+
#ifdef CONFIG_QUOTA
org_mount_opt.s_jquota_fmt = F2FS_OPTION(sbi).s_jquota_fmt;
for (i = 0; i < MAXQUOTAS; i++) {
@@ -2593,6 +2601,8 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
limit_reserve_root(sbi);
adjust_unusable_cap_perc(sbi);
*flags = (*flags & ~SB_LAZYTIME) | (sb->s_flags & SB_LAZYTIME);
+
+ sbi->umount_lock_holder = NULL;
return 0;
restore_checkpoint:
if (need_enable_checkpoint) {
@@ -2634,6 +2644,8 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
#endif
sbi->mount_opt = org_mount_opt;
sb->s_flags = old_sb_flags;
+
+ sbi->umount_lock_holder = NULL;
return err;
}
@@ -2950,7 +2962,7 @@ static int f2fs_quota_sync_file(struct f2fs_sb_info *sbi, int type)
return ret;
}
-int f2fs_quota_sync(struct super_block *sb, int type)
+int f2fs_do_quota_sync(struct super_block *sb, int type)
{
struct f2fs_sb_info *sbi = F2FS_SB(sb);
struct quota_info *dqopt = sb_dqopt(sb);
@@ -2998,11 +3010,21 @@ int f2fs_quota_sync(struct super_block *sb, int type)
return ret;
}
+static int f2fs_quota_sync(struct super_block *sb, int type)
+{
+ int ret;
+
+ F2FS_SB(sb)->umount_lock_holder = current;
+ ret = f2fs_do_quota_sync(sb, type);
+ F2FS_SB(sb)->umount_lock_holder = NULL;
+ return ret;
+}
+
static int f2fs_quota_on(struct super_block *sb, int type, int format_id,
const struct path *path)
{
struct inode *inode;
- int err;
+ int err = 0;
/* if quota sysfile exists, deny enabling quota with specific file */
if (f2fs_sb_has_quota_ino(F2FS_SB(sb))) {
@@ -3013,31 +3035,34 @@ static int f2fs_quota_on(struct super_block *sb, int type, int format_id,
if (path->dentry->d_sb != sb)
return -EXDEV;
- err = f2fs_quota_sync(sb, type);
+ F2FS_SB(sb)->umount_lock_holder = current;
+
+ err = f2fs_do_quota_sync(sb, type);
if (err)
- return err;
+ goto out;
inode = d_inode(path->dentry);
err = filemap_fdatawrite(inode->i_mapping);
if (err)
- return err;
+ goto out;
err = filemap_fdatawait(inode->i_mapping);
if (err)
- return err;
+ goto out;
err = dquot_quota_on(sb, type, format_id, path);
if (err)
- return err;
+ goto out;
inode_lock(inode);
F2FS_I(inode)->i_flags |= F2FS_QUOTA_DEFAULT_FL;
f2fs_set_inode_flags(inode);
inode_unlock(inode);
f2fs_mark_inode_dirty_sync(inode, false);
-
- return 0;
+out:
+ F2FS_SB(sb)->umount_lock_holder = NULL;
+ return err;
}
static int __f2fs_quota_off(struct super_block *sb, int type)
@@ -3048,7 +3073,7 @@ static int __f2fs_quota_off(struct super_block *sb, int type)
if (!inode || !igrab(inode))
return dquot_quota_off(sb, type);
- err = f2fs_quota_sync(sb, type);
+ err = f2fs_do_quota_sync(sb, type);
if (err)
goto out_put;
@@ -3071,6 +3096,8 @@ static int f2fs_quota_off(struct super_block *sb, int type)
struct f2fs_sb_info *sbi = F2FS_SB(sb);
int err;
+ F2FS_SB(sb)->umount_lock_holder = current;
+
err = __f2fs_quota_off(sb, type);
/*
@@ -3080,6 +3107,9 @@ static int f2fs_quota_off(struct super_block *sb, int type)
*/
if (is_journalled_quota(sbi))
set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
+
+ F2FS_SB(sb)->umount_lock_holder = NULL;
+
return err;
}
@@ -3212,7 +3242,7 @@ int f2fs_dquot_initialize(struct inode *inode)
return 0;
}
-int f2fs_quota_sync(struct super_block *sb, int type)
+int f2fs_do_quota_sync(struct super_block *sb, int type)
{
return 0;
}
@@ -4760,6 +4790,7 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
if (err)
goto free_compress_inode;
+ sbi->umount_lock_holder = current;
#ifdef CONFIG_QUOTA
/* Enable quota usage during mount */
if (f2fs_sb_has_quota_ino(sbi) && !f2fs_readonly(sb)) {
@@ -4892,6 +4923,8 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
f2fs_update_time(sbi, CP_TIME);
f2fs_update_time(sbi, REQ_TIME);
clear_sbi_flag(sbi, SBI_CP_DISABLED_QUICK);
+
+ sbi->umount_lock_holder = NULL;
return 0;
sync_free_meta:
@@ -4997,6 +5030,8 @@ static void kill_f2fs_super(struct super_block *sb)
struct f2fs_sb_info *sbi = F2FS_SB(sb);
if (sb->s_root) {
+ sbi->umount_lock_holder = current;
+
set_sbi_flag(sbi, SBI_IS_CLOSE);
f2fs_stop_gc_thread(sbi);
f2fs_stop_discard_thread(sbi);
diff --git a/fs/fcntl.c b/fs/fcntl.c
index c4cdee6a7800..cb40e1f661eb 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -374,6 +374,9 @@ static long fcntl_set_rw_hint(struct file *file, unsigned int cmd,
u64 __user *argp = (u64 __user *)arg;
u64 hint;
+ /* nothing is ever written to it */
+ if (IS_IMMUTABLE(inode))
+ return -EPERM;
if (copy_from_user(&hint, argp, sizeof(hint)))
return -EFAULT;
if (!rw_hint_valid(hint))
diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c
index 4c6d1f15a6a8..243971150973 100644
--- a/fs/gfs2/ops_fstype.c
+++ b/fs/gfs2/ops_fstype.c
@@ -1281,7 +1281,6 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc)
if (error) {
gfs2_freeze_unlock(sdp);
- gfs2_destroy_threads(sdp);
fs_err(sdp, "can't make FS RW: %d\n", error);
goto fail_per_node;
}
@@ -1291,6 +1290,7 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc)
fail_per_node:
init_per_node(sdp, UNDO);
+ gfs2_destroy_threads(sdp);
fail_inodes:
init_inodes(sdp, UNDO);
fail_sb:
diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c
index 3d5d4b844621..8628121c1ab1 100644
--- a/fs/gfs2/super.c
+++ b/fs/gfs2/super.c
@@ -157,8 +157,10 @@ int gfs2_make_fs_rw(struct gfs2_sbd *sdp)
gfs2_log_pointers_init(sdp, head.lh_blkno);
error = gfs2_quota_init(sdp);
- if (!error && gfs2_withdrawing_or_withdrawn(sdp))
+ if (!error && gfs2_withdrawing_or_withdrawn(sdp)) {
+ gfs2_quota_cleanup(sdp);
error = -EIO;
+ }
if (!error)
set_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
return error;
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 4703fa7343e7..02a5fc5105e7 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -1858,8 +1858,12 @@ static int iomap_add_to_ioend(struct iomap_writepage_ctx *wpc,
* appending writes.
*/
wpc->ioend->io_size += len;
- if (wpc->ioend->io_offset + wpc->ioend->io_size > end_pos)
- wpc->ioend->io_size = end_pos - wpc->ioend->io_offset;
+ if (wpc->ioend->io_offset + wpc->ioend->io_size > end_pos) {
+ if (wpc->ioend->io_offset >= end_pos)
+ wpc->ioend->io_size = 0;
+ else
+ wpc->ioend->io_size = end_pos - wpc->ioend->io_offset;
+ }
wbc_account_cgroup_owner(wbc, folio, len);
return 0;
diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c
index 4ec22c2f2ea3..87f20d1edb2c 100644
--- a/fs/lockd/svc.c
+++ b/fs/lockd/svc.c
@@ -47,7 +47,7 @@
static struct svc_program nlmsvc_program;
-const struct nlmsvc_binding *nlmsvc_ops;
+const struct nlmsvc_binding __rcu *nlmsvc_ops;
EXPORT_SYMBOL_GPL(nlmsvc_ops);
static DEFINE_MUTEX(nlmsvc_mutex);
@@ -142,7 +142,7 @@ lockd(void *vrqstp)
nlmsvc_retry_blocked(rqstp);
svc_recv(rqstp);
}
- if (nlmsvc_ops)
+ if (rcu_access_pointer(nlmsvc_ops))
nlmsvc_invalidate_all();
nlm_shutdown_hosts();
cancel_delayed_work_sync(&ln->grace_period_end);
diff --git a/fs/lockd/svc4proc.c b/fs/lockd/svc4proc.c
index 3f633bb24623..ea3b689f8b93 100644
--- a/fs/lockd/svc4proc.c
+++ b/fs/lockd/svc4proc.c
@@ -29,7 +29,7 @@ nlm4svc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp,
__be32 error = 0;
/* nfsd callbacks must have been installed for this procedure */
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return nlm_lck_denied_nolocks;
if (lock->lock_start > OFFSET_MAX ||
@@ -487,7 +487,7 @@ nlm4svc_proc_granted_res(struct svc_rqst *rqstp)
{
struct nlm_res *argp = rqstp->rq_argp;
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return rpc_success;
dprintk("lockd: GRANTED_RES called\n");
diff --git a/fs/lockd/svcproc.c b/fs/lockd/svcproc.c
index 8cda2aba82d2..44da63392a09 100644
--- a/fs/lockd/svcproc.c
+++ b/fs/lockd/svcproc.c
@@ -59,7 +59,7 @@ nlmsvc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp,
__be32 error = 0;
/* nfsd callbacks must have been installed for this procedure */
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return nlm_lck_denied_nolocks;
/* Obtain host handle */
@@ -521,7 +521,7 @@ nlmsvc_proc_granted_res(struct svc_rqst *rqstp)
{
struct nlm_res *argp = rqstp->rq_argp;
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return rpc_success;
dprintk("lockd: GRANTED_RES called\n");
diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
index 90c6527ad712..3e3cd67e167a 100644
--- a/fs/lockd/svcsubs.c
+++ b/fs/lockd/svcsubs.c
@@ -86,12 +86,23 @@ int lock_to_openmode(struct file_lock *lock)
static __be32 nlm_do_fopen(struct svc_rqst *rqstp,
struct nlm_file *file, int mode)
{
+ const struct nlmsvc_binding *ops;
struct file **fp = &file->f_file[mode];
__be32 nfserr;
if (*fp)
return 0;
- nfserr = nlmsvc_ops->fopen(rqstp, &file->f_handle, fp, mode);
+
+ rcu_read_lock();
+ ops = rcu_dereference(nlmsvc_ops);
+ if (!ops || !try_module_get(ops->owner)) {
+ rcu_read_unlock();
+ return nlm_lck_denied_nolocks;
+ }
+ rcu_read_unlock();
+
+ nfserr = ops->fopen(rqstp, &file->f_handle, fp, mode);
+ module_put(ops->owner);
if (nfserr)
dprintk("lockd: open failed (error %d)\n", nfserr);
return nfserr;
@@ -159,6 +170,33 @@ nlm_lookup_file(struct svc_rqst *rqstp, struct nlm_file **result,
goto out_unlock;
}
+/*
+ * Release the struct file references held by a nlm_file.
+ */
+static void nlm_release_files(struct nlm_file *file)
+{
+ const struct nlmsvc_binding *ops;
+ bool have_ops;
+
+ rcu_read_lock();
+ ops = rcu_dereference(nlmsvc_ops);
+ have_ops = ops && try_module_get(ops->owner);
+ rcu_read_unlock();
+
+ if (have_ops) {
+ if (file->f_file[O_RDONLY])
+ ops->fclose(file->f_file[O_RDONLY]);
+ if (file->f_file[O_WRONLY])
+ ops->fclose(file->f_file[O_WRONLY]);
+ module_put(ops->owner);
+ } else {
+ if (file->f_file[O_RDONLY])
+ fput(file->f_file[O_RDONLY]);
+ if (file->f_file[O_WRONLY])
+ fput(file->f_file[O_WRONLY]);
+ }
+}
+
/*
* Delete a file after having released all locks, blocks and shares
*/
@@ -168,10 +206,7 @@ nlm_delete_file(struct nlm_file *file)
nlm_debug_print_file("closing file", file);
if (!hlist_unhashed(&file->f_list)) {
hlist_del(&file->f_list);
- if (file->f_file[O_RDONLY])
- nlmsvc_ops->fclose(file->f_file[O_RDONLY]);
- if (file->f_file[O_WRONLY])
- nlmsvc_ops->fclose(file->f_file[O_WRONLY]);
+ nlm_release_files(file);
kfree(file);
} else {
printk(KERN_WARNING "lockd: attempt to release unknown file!\n");
diff --git a/fs/namespace.c b/fs/namespace.c
index f87bdff66166..036d56df2828 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -1311,7 +1311,7 @@ static struct mount *clone_mnt(struct mount *old, struct dentry *root,
}
mnt->mnt.mnt_flags = old->mnt.mnt_flags;
- mnt->mnt.mnt_flags &= ~(MNT_WRITE_HOLD|MNT_MARKED|MNT_INTERNAL);
+ mnt->mnt.mnt_flags &= ~(MNT_WRITE_HOLD|MNT_MARKED|MNT_INTERNAL|MNT_UMOUNT);
atomic_inc(&sb->s_active);
mnt->mnt.mnt_idmap = mnt_idmap_get(mnt_idmap(&old->mnt));
@@ -1778,6 +1778,8 @@ static void umount_tree(struct mount *mnt, enum umount_tree_flags how)
/* Gather the mounts to umount */
for (p = mnt; p; p = next_mnt(p, mnt)) {
+ /* A mount is unmounted once. */
+ WARN_ON_ONCE(p->mnt.mnt_flags & MNT_UMOUNT);
p->mnt.mnt_flags |= MNT_UMOUNT;
if (mnt_ns_attached(p))
move_from_ns(p, &tmp_list);
@@ -1955,6 +1957,9 @@ static int do_umount(struct mount *mnt, int flags)
* detach_mounts allows lazily unmounting those mounts instead of
* leaking them.
*
+ * The dentry is unhashed before the mounts go so that no lookup finds
+ * what they covered. The caller removes it for good afterwards.
+ *
* The caller may hold dentry->d_inode->i_mutex.
*/
void __detach_mounts(struct dentry *dentry)
@@ -1968,6 +1973,8 @@ void __detach_mounts(struct dentry *dentry)
if (!mp)
goto out_unlock;
+ /* the name goes first, what covered it goes second */
+ d_drop(dentry);
event++;
while (!hlist_empty(&mp->m_list)) {
mnt = hlist_entry(mp->m_list.first, struct mount, mnt_mp_list);
@@ -2487,15 +2494,31 @@ static int attach_recursive_mnt(struct mount *source_mnt,
struct mountpoint *dest_mp,
enum mnt_tree_flags_t flags)
{
- struct user_namespace *user_ns = current->nsproxy->mnt_ns->user_ns;
HLIST_HEAD(tree_list);
struct mnt_namespace *ns = top_mnt->mnt_ns;
+ struct user_namespace *user_ns = ns->user_ns;
struct mountpoint *smp;
struct mount *child, *dest_mnt, *p;
struct hlist_node *n;
int err = 0;
bool moving = flags & MNT_TREE_MOVE, beneath = flags & MNT_TREE_BENEATH;
+ /*
+ * A caller in an unprivileged mount namespaces may trigger an
+ * automount and propagate locked mounts into privileged mount
+ * namespaces. Take ownership from the target mount namespace.
+ * It's equivalent for everything but the automount case.
+ *
+ * Detached trees in anonymous mount namespaces by be handed
+ * over via SCM_RIGHTS or inherited in other ways on purpose
+ * the attaching task's mount namespace is authoritative, not
+ * the creator of the detached tree.
+ */
+ if (is_anon_ns(ns))
+ user_ns = current->nsproxy->mnt_ns->user_ns;
+ else
+ user_ns = ns->user_ns;
+
/*
* Preallocate a mountpoint in case the new mounts need to be
* mounted beneath mounts on the same mountpoint.
@@ -4467,14 +4490,15 @@ SYSCALL_DEFINE2(pivot_root, const char __user *, new_root,
new_mnt = real_mount(new.mnt);
root_mnt = real_mount(root.mnt);
old_mnt = real_mount(old.mnt);
+ /* only a mounted mount has a parent that namespace_sem pins */
+ if (!check_mnt(root_mnt) || !check_mnt(new_mnt))
+ goto out4;
ex_parent = new_mnt->mnt_parent;
root_parent = root_mnt->mnt_parent;
if (IS_MNT_SHARED(old_mnt) ||
IS_MNT_SHARED(ex_parent) ||
IS_MNT_SHARED(root_parent))
goto out4;
- if (!check_mnt(root_mnt) || !check_mnt(new_mnt))
- goto out4;
if (new_mnt->mnt.mnt_flags & MNT_LOCKED)
goto out4;
error = -ENOENT;
@@ -4713,6 +4737,13 @@ static int do_mount_setattr(struct path *path, struct mount_kattr *kattr)
* propagation.
*/
namespace_lock();
+ /* invent_group_ids() walks the tree, only walk a mounted one */
+ if (!is_mounted(&mnt->mnt) ||
+ ((mnt_has_parent(mnt) || !is_anon_ns(mnt->mnt_ns)) &&
+ !check_mnt(mnt))) {
+ namespace_unlock();
+ return -EINVAL;
+ }
if (kattr->propagation == MS_SHARED) {
err = invent_group_ids(mnt, kattr->recurse);
if (err) {
@@ -4725,6 +4756,7 @@ static int do_mount_setattr(struct path *path, struct mount_kattr *kattr)
err = -EINVAL;
lock_mount_hash();
+ /* Checked under namespace_sem already if the propagation changes. */
/* Ensure that this isn't anything purely vfs internal. */
if (!is_mounted(&mnt->mnt))
goto out;
diff --git a/fs/netfs/Makefile b/fs/netfs/Makefile
index d08b0bfb6756..7492c4aa331e 100644
--- a/fs/netfs/Makefile
+++ b/fs/netfs/Makefile
@@ -13,6 +13,7 @@ netfs-y := \
read_collect.o \
read_pgpriv2.o \
read_retry.o \
+ rolling_buffer.o \
write_collect.o \
write_issue.o
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 562c59f42133..68ad1ece65a0 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -63,37 +63,6 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in
return fscache_begin_read_operation(&rreq->cache_resources, netfs_i_cookie(ctx));
}
-/*
- * Decant the list of folios to read into a rolling buffer.
- */
-static size_t netfs_load_buffer_from_ra(struct netfs_io_request *rreq,
- struct folio_queue *folioq,
- struct folio_batch *put_batch)
-{
- unsigned int order, nr;
- size_t size = 0;
-
- nr = __readahead_batch(rreq->ractl, (struct page **)folioq->vec.folios,
- ARRAY_SIZE(folioq->vec.folios));
- folioq->vec.nr = nr;
- for (int i = 0; i < nr; i++) {
- struct folio *folio = folioq_folio(folioq, i);
-
- trace_netfs_folio(folio, netfs_folio_trace_read);
- order = folio_order(folio);
- folioq->orders[i] = order;
- size += PAGE_SIZE << order;
-
- if (!folio_batch_add(put_batch, folio))
- folio_batch_release(put_batch);
- }
-
- for (int i = nr; i < folioq_nr_slots(folioq); i++)
- folioq_clear(folioq, i);
-
- return size;
-}
-
/*
* netfs_prepare_read_iterator - Prepare the subreq iterator for I/O
* @subreq: The subrequest to be set up
@@ -128,19 +97,14 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
folio_batch_init(&put_batch);
while (rreq->submitted < subreq->start + rsize) {
- struct folio_queue *tail = rreq->buffer_tail, *new;
- size_t added;
-
- new = kmalloc(sizeof(*new), GFP_NOFS);
- if (!new)
- return -ENOMEM;
- netfs_stat(&netfs_n_folioq);
- folioq_init(new);
- new->prev = tail;
- tail->next = new;
- rreq->buffer_tail = new;
- added = netfs_load_buffer_from_ra(rreq, new, &put_batch);
- rreq->iter.count += added;
+ ssize_t added;
+
+ added = rolling_buffer_load_from_ra(&rreq->buffer, rreq->ractl,
+ &put_batch);
+ if (added < 0) {
+ folio_batch_release(&put_batch);
+ return added;
+ }
rreq->submitted += added;
}
folio_batch_release(&put_batch);
@@ -148,7 +112,7 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
subreq->len = rsize;
if (unlikely(rreq->io_streams[0].sreq_max_segs)) {
- size_t limit = netfs_limit_iter(&rreq->iter, 0, rsize,
+ size_t limit = netfs_limit_iter(&rreq->buffer.iter, 0, rsize,
rreq->io_streams[0].sreq_max_segs);
if (limit < rsize) {
@@ -157,20 +121,16 @@ static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq)
}
}
- subreq->io_iter = rreq->iter;
+ subreq->io_iter = rreq->buffer.iter;
if (iov_iter_is_folioq(&subreq->io_iter)) {
- if (subreq->io_iter.folioq_slot >= folioq_nr_slots(subreq->io_iter.folioq)) {
- subreq->io_iter.folioq = subreq->io_iter.folioq->next;
- subreq->io_iter.folioq_slot = 0;
- }
subreq->curr_folioq = (struct folio_queue *)subreq->io_iter.folioq;
subreq->curr_folioq_slot = subreq->io_iter.folioq_slot;
subreq->curr_folio_order = subreq->curr_folioq->orders[subreq->curr_folioq_slot];
}
iov_iter_truncate(&subreq->io_iter, subreq->len);
- iov_iter_advance(&rreq->iter, subreq->len);
+ rolling_buffer_advance(&rreq->buffer, subreq->len);
return subreq->len;
}
@@ -196,7 +156,7 @@ static void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error
} else {
subreq->error = transferred_or_error;
}
- netfs_read_subreq_terminated(subreq, was_async);
+ schedule_work(&subreq->work);
}
/*
@@ -297,7 +257,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
goto prep_iter_failed;
__set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags);
subreq->error = 0;
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
goto done;
}
@@ -329,7 +289,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq)
} while (size > 0);
if (atomic_dec_and_test(&rreq->nr_outstanding))
- netfs_rreq_terminated(rreq, false);
+ netfs_rreq_terminated(rreq);
/* Defer error return as we may need to wait for outstanding I/O. */
cmpxchg(&rreq->error, 0, ret);
@@ -353,34 +313,6 @@ static int netfs_wait_for_read(struct netfs_io_request *rreq)
return ret;
}
-/*
- * Set up the initial folioq of buffer folios in the rolling buffer and set the
- * iterator to refer to it.
- */
-static int netfs_prime_buffer(struct netfs_io_request *rreq)
-{
- struct folio_queue *folioq;
- struct folio_batch put_batch;
- size_t added;
-
- folioq = kmalloc(sizeof(*folioq), GFP_KERNEL);
- if (!folioq)
- return -ENOMEM;
- netfs_stat(&netfs_n_folioq);
- folioq_init(folioq);
- rreq->buffer = folioq;
- rreq->buffer_tail = folioq;
- rreq->submitted = rreq->start;
- iov_iter_folio_queue(&rreq->iter, ITER_DEST, folioq, 0, 0, 0);
-
- folio_batch_init(&put_batch);
- added = netfs_load_buffer_from_ra(rreq, folioq, &put_batch);
- folio_batch_release(&put_batch);
- rreq->iter.count += added;
- rreq->submitted += added;
- return 0;
-}
-
/**
* netfs_readahead - Helper to manage a read request
* @ractl: The description of the readahead request
@@ -420,7 +352,8 @@ void netfs_readahead(struct readahead_control *ractl)
netfs_rreq_expand(rreq, ractl);
rreq->ractl = ractl;
- if (netfs_prime_buffer(rreq) < 0)
+ rreq->submitted = rreq->start;
+ if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST, rreq->gfp) < 0)
goto cleanup_free;
netfs_read_to_pagecache(rreq);
@@ -436,23 +369,18 @@ EXPORT_SYMBOL(netfs_readahead);
/*
* Create a rolling buffer with a single occupying folio.
*/
-static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio)
+static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio,
+ unsigned int rollbuf_flags)
{
- struct folio_queue *folioq;
+ ssize_t added;
- folioq = kmalloc(sizeof(*folioq), GFP_KERNEL);
- if (!folioq)
+ if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST, rreq->gfp) < 0)
return -ENOMEM;
- netfs_stat(&netfs_n_folioq);
- folioq_init(folioq);
- folioq_append(folioq, folio);
- BUG_ON(folioq_folio(folioq, 0) != folio);
- BUG_ON(folioq_folio_order(folioq, 0) != folio_order(folio));
- rreq->buffer = folioq;
- rreq->buffer_tail = folioq;
- rreq->submitted = rreq->start + rreq->len;
- iov_iter_folio_queue(&rreq->iter, ITER_DEST, folioq, 0, 0, rreq->len);
+ added = rolling_buffer_append(&rreq->buffer, folio, rollbuf_flags, rreq->gfp);
+ if (added < 0)
+ return added;
+ rreq->submitted = rreq->start + added;
rreq->ractl = (struct readahead_control *)1UL;
return 0;
}
@@ -520,7 +448,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio)
}
if (to < flen)
bvec_set_folio(&bvec[i++], folio, flen - to, to);
- iov_iter_bvec(&rreq->iter, ITER_DEST, bvec, i, rreq->len);
+ iov_iter_bvec(&rreq->buffer.iter, ITER_DEST, bvec, i, rreq->len);
rreq->submitted = rreq->start + flen;
netfs_read_to_pagecache(rreq);
@@ -590,7 +518,7 @@ int netfs_read_folio(struct file *file, struct folio *folio)
trace_netfs_read(rreq, rreq->start, rreq->len, netfs_read_trace_readpage);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio);
+ ret = netfs_create_singular_buffer(rreq, folio, 0);
if (ret < 0)
goto discard;
@@ -747,15 +675,15 @@ int netfs_write_begin(struct netfs_inode *ctx,
trace_netfs_read(rreq, pos, len, netfs_read_trace_write_begin);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio);
+ ret = netfs_create_singular_buffer(rreq, folio, 0);
if (ret < 0)
goto error_put;
netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
+ netfs_put_request(rreq, false, netfs_rreq_trace_put_return);
if (ret < 0)
goto error;
- netfs_put_request(rreq, false, netfs_rreq_trace_put_return);
have_folio:
ret = folio_wait_private_2_killable(folio);
@@ -812,11 +740,10 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio,
trace_netfs_read(rreq, start, flen, netfs_read_trace_prefetch_for_write);
/* Set up the output buffer */
- ret = netfs_create_singular_buffer(rreq, folio);
+ ret = netfs_create_singular_buffer(rreq, folio, NETFS_ROLLBUF_PAGECACHE_MARK);
if (ret < 0)
goto error_put;
- folioq_mark2(rreq->buffer, 0);
netfs_read_to_pagecache(rreq);
ret = netfs_wait_for_read(rreq);
netfs_put_request(rreq, false, netfs_rreq_trace_put_return);
diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c
index 917b7edc34ef..2c79db143432 100644
--- a/fs/netfs/direct_read.c
+++ b/fs/netfs/direct_read.c
@@ -25,7 +25,7 @@ static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)
subreq->len = rsize;
if (unlikely(rreq->io_streams[0].sreq_max_segs)) {
- size_t limit = netfs_limit_iter(&rreq->iter, 0, rsize,
+ size_t limit = netfs_limit_iter(&rreq->buffer.iter, 0, rsize,
rreq->io_streams[0].sreq_max_segs);
if (limit < rsize) {
@@ -36,9 +36,9 @@ static void netfs_prepare_dio_read_iterator(struct netfs_io_subrequest *subreq)
trace_netfs_sreq(subreq, netfs_sreq_trace_prepare);
- subreq->io_iter = rreq->iter;
+ subreq->io_iter = rreq->buffer.iter;
iov_iter_truncate(&subreq->io_iter, subreq->len);
- iov_iter_advance(&rreq->iter, subreq->len);
+ iov_iter_advance(&rreq->buffer.iter, subreq->len);
}
/*
@@ -100,7 +100,7 @@ static int netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq)
} while (size > 0);
if (atomic_dec_and_test(&rreq->nr_outstanding))
- netfs_rreq_terminated(rreq, false);
+ netfs_rreq_terminated(rreq);
return ret;
}
@@ -199,15 +199,15 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i
* the request.
*/
if (user_backed_iter(iter)) {
- ret = netfs_extract_user_iter(iter, rreq->len, &rreq->iter, 0);
+ ret = netfs_extract_user_iter(iter, rreq->len, &rreq->buffer.iter, 0);
if (ret < 0)
goto out;
- rreq->direct_bv = (struct bio_vec *)rreq->iter.bvec;
+ rreq->direct_bv = (struct bio_vec *)rreq->buffer.iter.bvec;
rreq->direct_bv_count = ret;
rreq->direct_bv_unpin = iov_iter_extract_will_pin(iter);
- rreq->len = iov_iter_count(&rreq->iter);
+ rreq->len = iov_iter_count(&rreq->buffer.iter);
} else {
- rreq->iter = *iter;
+ rreq->buffer.iter = *iter;
rreq->len = orig_count;
rreq->direct_bv_unpin = false;
iov_iter_advance(iter, orig_count);
diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c
index 59f1b36a43c7..0f8fb01ec84f 100644
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@ -72,12 +72,12 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
* request.
*/
if (user_backed_iter(iter)) {
- n = netfs_extract_user_iter(iter, len, &wreq->iter, 0);
+ n = netfs_extract_user_iter(iter, len, &wreq->buffer.iter, 0);
if (n < 0) {
ret = n;
goto out;
}
- wreq->direct_bv = (struct bio_vec *)wreq->iter.bvec;
+ wreq->direct_bv = (struct bio_vec *)wreq->buffer.iter.bvec;
wreq->direct_bv_count = n;
wreq->direct_bv_unpin = iov_iter_extract_will_pin(iter);
} else {
@@ -86,10 +86,8 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
* (eg. a bio_vec array) will persist till the end of
* the op.
*/
- wreq->iter = *iter;
+ wreq->buffer.iter = *iter;
}
-
- wreq->io_iter = wreq->iter;
}
__set_bit(NETFS_RREQ_USE_IO_ITER, &wreq->flags);
@@ -101,7 +99,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter *
__set_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags);
if (async)
wreq->iocb = iocb;
- wreq->len = iov_iter_count(&wreq->io_iter);
+ wreq->len = iov_iter_count(&wreq->buffer.iter);
wreq->cleanup = netfs_cleanup_dio_write;
ret = netfs_unbuffered_write(wreq, is_sync_kiocb(iocb), wreq->len);
if (ret < 0) {
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index 668c4b3f7813..b8b78c9be293 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -34,6 +34,7 @@ extern struct list_head netfs_io_requests;
extern spinlock_t netfs_proc_lock;
extern mempool_t netfs_request_pool;
extern mempool_t netfs_subrequest_pool;
+extern mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
static inline void netfs_proc_add_rreq(struct netfs_io_request *rreq)
@@ -58,11 +59,8 @@ static inline void netfs_proc_del_rreq(struct netfs_io_request *rreq) {}
/*
* misc.c
*/
-struct folio_queue *netfs_buffer_make_space(struct netfs_io_request *rreq);
-int netfs_buffer_append_folio(struct netfs_io_request *rreq, struct folio *folio,
- bool needs_put);
-struct folio_queue *netfs_delete_buffer_head(struct netfs_io_request *wreq);
-void netfs_clear_buffer(struct netfs_io_request *rreq);
+struct folio_queue *netfs_buffer_make_space(struct netfs_io_request *rreq,
+ enum netfs_folioq_trace trace);
void netfs_reset_iter(struct netfs_io_subrequest *subreq);
int netfs_clear_stale_pre_isize(struct inode *inode, unsigned long long from,
unsigned long long to, bool nowait);
@@ -89,8 +87,7 @@ static inline void netfs_see_request(struct netfs_io_request *rreq,
/*
* read_collect.c
*/
-void netfs_read_termination_worker(struct work_struct *work);
-void netfs_rreq_terminated(struct netfs_io_request *rreq, bool was_async);
+void netfs_rreq_terminated(struct netfs_io_request *rreq);
/*
* read_pgpriv2.c
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 3a8433e802cc..98ff6d2e62ec 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -28,6 +28,7 @@ static struct kmem_cache *netfs_request_slab;
static struct kmem_cache *netfs_subrequest_slab;
mempool_t netfs_request_pool;
mempool_t netfs_subrequest_pool;
+mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
LIST_HEAD(netfs_io_requests);
@@ -105,6 +106,9 @@ static int __init netfs_init(void)
{
int ret = -ENOMEM;
+ if (mempool_init_kmalloc_pool(&netfs_folioq_pool, 100, sizeof(struct folio_queue)) < 0)
+ goto error_folioq_pool;
+
netfs_request_slab = kmem_cache_create("netfs_request",
sizeof(struct netfs_io_request), 0,
SLAB_HWCACHE_ALIGN | SLAB_ACCOUNT,
@@ -157,6 +161,8 @@ static int __init netfs_init(void)
error_reqpool:
kmem_cache_destroy(netfs_request_slab);
error_req:
+ mempool_exit(&netfs_folioq_pool);
+error_folioq_pool:
return ret;
}
fs_initcall(netfs_init);
@@ -169,5 +175,6 @@ static void __exit netfs_exit(void)
kmem_cache_destroy(netfs_subrequest_slab);
mempool_exit(&netfs_request_pool);
kmem_cache_destroy(netfs_request_slab);
+ mempool_exit(&netfs_folioq_pool);
}
module_exit(netfs_exit);
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index f5f7c0de3a7c..b9ed14ae338e 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -9,114 +9,6 @@
#include <linux/rmap.h>
#include "internal.h"
-/*
- * Make sure there's space in the rolling queue.
- */
-struct folio_queue *netfs_buffer_make_space(struct netfs_io_request *rreq)
-{
- struct folio_queue *tail = rreq->buffer_tail, *prev;
- unsigned int prev_nr_slots = 0;
-
- if (WARN_ON_ONCE(!rreq->buffer && tail) ||
- WARN_ON_ONCE(rreq->buffer && !tail))
- return ERR_PTR(-EIO);
-
- prev = tail;
- if (prev) {
- if (!folioq_full(tail))
- return tail;
- prev_nr_slots = folioq_nr_slots(tail);
- }
-
- tail = kmalloc(sizeof(*tail), GFP_NOFS);
- if (!tail)
- return ERR_PTR(-ENOMEM);
- netfs_stat(&netfs_n_folioq);
- folioq_init(tail);
- tail->prev = prev;
- if (prev)
- /* [!] NOTE: After we set prev->next, the consumer is entirely
- * at liberty to delete prev.
- */
- WRITE_ONCE(prev->next, tail);
-
- rreq->buffer_tail = tail;
- if (!rreq->buffer) {
- rreq->buffer = tail;
- iov_iter_folio_queue(&rreq->io_iter, ITER_SOURCE, tail, 0, 0, 0);
- } else {
- /* Make sure we don't leave the master iterator pointing to a
- * block that might get immediately consumed.
- */
- if (rreq->io_iter.folioq == prev &&
- rreq->io_iter.folioq_slot == prev_nr_slots) {
- rreq->io_iter.folioq = tail;
- rreq->io_iter.folioq_slot = 0;
- }
- }
- rreq->buffer_tail_slot = 0;
- return tail;
-}
-
-/*
- * Append a folio to the rolling queue.
- */
-int netfs_buffer_append_folio(struct netfs_io_request *rreq, struct folio *folio,
- bool needs_put)
-{
- struct folio_queue *tail;
- unsigned int slot, order = folio_order(folio);
-
- tail = netfs_buffer_make_space(rreq);
- if (IS_ERR(tail))
- return PTR_ERR(tail);
-
- rreq->io_iter.count += PAGE_SIZE << order;
-
- slot = folioq_append(tail, folio);
- /* Store the counter after setting the slot. */
- smp_store_release(&rreq->buffer_tail_slot, slot);
- return 0;
-}
-
-/*
- * Delete the head of a rolling queue.
- */
-struct folio_queue *netfs_delete_buffer_head(struct netfs_io_request *wreq)
-{
- struct folio_queue *head = wreq->buffer, *next = head->next;
-
- if (next)
- next->prev = NULL;
- netfs_stat_d(&netfs_n_folioq);
- kfree(head);
- wreq->buffer = next;
- return next;
-}
-
-/*
- * Clear out a rolling queue.
- */
-void netfs_clear_buffer(struct netfs_io_request *rreq)
-{
- struct folio_queue *p;
-
- while ((p = rreq->buffer)) {
- rreq->buffer = p->next;
- for (int slot = 0; slot < folioq_count(p); slot++) {
- struct folio *folio = folioq_folio(p, slot);
- if (!folio)
- continue;
- if (folioq_is_marked(p, slot)) {
- trace_netfs_folio(folio, netfs_folio_trace_put);
- folio_put(folio);
- }
- }
- netfs_stat_d(&netfs_n_folioq);
- kfree(p);
- }
-}
-
/*
* Reset the subrequest iterator to refer just to the region remaining to be
* read. The iterator may or may not have been advanced by socket ops or
@@ -315,7 +207,7 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp)
if (folio_test_dirty(folio))
return false;
- end = umin(folio_pos(folio) + folio_size(folio), i_size_read(&ctx->inode));
+ end = umin(folio_pos(folio) + folio_size(folio), ctx->remote_i_size);
if (end > ctx->zero_point)
ctx->zero_point = end;
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index eb8411bd0a74..788bdaf7332f 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -7,7 +7,6 @@
#include <linux/slab.h>
#include <linux/mempool.h>
-#include <linux/delay.h>
#include "internal.h"
/*
@@ -24,16 +23,22 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
struct netfs_io_request *rreq;
mempool_t *mempool = ctx->ops->request_pool ?: &netfs_request_pool;
struct kmem_cache *cache = mempool->pool_data;
+ gfp_t gfp = GFP_KERNEL;
int ret;
- for (;;) {
- rreq = mempool_alloc(mempool, GFP_KERNEL);
- if (rreq)
- break;
- msleep(10);
+ /* Writeback is part of memory reclaim and must not fail due to ENOMEM. */
+ if (origin == NETFS_WRITEBACK) {
+ gfp = GFP_NOFS; /* Allows use of mempools. */
+
+ rreq = mempool_alloc(mempool, gfp);
+ } else {
+ rreq = mempool->alloc(gfp, mempool->pool_data);
+ if (!rreq)
+ return ERR_PTR(-ENOMEM);
}
memset(rreq, 0, kmem_cache_size(cache));
+ rreq->gfp = gfp;
rreq->start = start;
rreq->len = len;
rreq->origin = origin;
@@ -56,7 +61,7 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
origin == NETFS_READ_GAPS ||
origin == NETFS_READ_FOR_WRITE ||
origin == NETFS_DIO_READ)
- INIT_WORK(&rreq->work, netfs_read_termination_worker);
+ INIT_WORK(&rreq->work, NULL);
else
INIT_WORK(&rreq->work, netfs_write_collection_worker);
@@ -143,7 +148,7 @@ static void netfs_free_request(struct work_struct *work)
}
kvfree(rreq->direct_bv);
}
- netfs_clear_buffer(rreq);
+ rolling_buffer_clear(&rreq->buffer);
if (atomic_dec_and_test(&ictx->io_count))
wake_up_var(&ictx->io_count);
@@ -182,13 +187,13 @@ struct netfs_io_subrequest *netfs_alloc_subrequest(struct netfs_io_request *rreq
mempool_t *mempool = rreq->netfs_ops->subrequest_pool ?: &netfs_subrequest_pool;
struct kmem_cache *cache = mempool->pool_data;
- for (;;) {
- subreq = mempool_alloc(rreq->netfs_ops->subrequest_pool ?: &netfs_subrequest_pool,
- GFP_KERNEL);
- if (subreq)
- break;
- msleep(10);
- }
+ /* The write-side callers don't handle failure, so only reads may fail. */
+ if (rreq->gfp == GFP_KERNEL && rreq->origin <= NETFS_DIO_READ)
+ subreq = mempool->alloc(rreq->gfp, mempool->pool_data);
+ else
+ subreq = mempool_alloc(mempool, rreq->gfp);
+ if (!subreq)
+ return NULL;
memset(subreq, 0, kmem_cache_size(cache));
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 782d7b730196..63f9ddf4435f 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -87,7 +87,7 @@ static void netfs_unlock_read_folio(struct netfs_io_subrequest *subreq,
* Unlock any folios that are now completely read. Returns true if the
* subrequest is removed from the list.
*/
-static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was_async)
+static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq)
{
struct netfs_io_subrequest *prev, *next;
struct netfs_io_request *rreq = subreq->rreq;
@@ -103,6 +103,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was
subreq->transferred, subreq->len))
subreq->transferred = subreq->len;
+ trace_netfs_folioq(folioq, netfs_trace_folioq_read_progress);
next_folio:
fsize = PAGE_SIZE << subreq->curr_folio_order;
fpos = round_down(subreq->start + subreq->consumed, fsize);
@@ -119,9 +120,11 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was
if (folioq) {
struct folio *folio = folioq_folio(folioq, slot);
- pr_err("folioq: orders=%02x%02x%02x%02x\n",
+ pr_err("folioq: fq=%x orders=%02x%02x%02x%02x %px\n",
+ folioq->debug_id,
folioq->orders[0], folioq->orders[1],
- folioq->orders[2], folioq->orders[3]);
+ folioq->orders[2], folioq->orders[3],
+ folioq);
if (folio)
pr_err("folio: %llx-%llx ix=%llx o=%u qo=%u\n",
fpos, fend - 1, folio_pos(folio), folio_order(folio),
@@ -222,12 +225,12 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was
slot = 0;
folioq = folioq->next;
subreq->curr_folioq = folioq;
+ trace_netfs_folioq(folioq, netfs_trace_folioq_read_progress);
}
subreq->curr_folioq_slot = slot;
if (folioq && folioq_folio(folioq, slot))
subreq->curr_folio_order = folioq->orders[slot];
- if (!was_async)
- cond_resched();
+ cond_resched();
goto next_folio;
}
@@ -365,7 +368,7 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq)
* Note that we're in normal kernel thread context at this point, possibly
* running on a workqueue.
*/
-static void netfs_rreq_assess(struct netfs_io_request *rreq)
+void netfs_rreq_terminated(struct netfs_io_request *rreq)
{
trace_netfs_rreq(rreq, netfs_rreq_trace_assess);
@@ -391,56 +394,29 @@ static void netfs_rreq_assess(struct netfs_io_request *rreq)
netfs_pgpriv2_write_to_the_cache(rreq);
}
-void netfs_read_termination_worker(struct work_struct *work)
-{
- struct netfs_io_request *rreq =
- container_of(work, struct netfs_io_request, work);
- netfs_see_request(rreq, netfs_rreq_trace_see_work);
- netfs_rreq_assess(rreq);
- netfs_put_request(rreq, false, netfs_rreq_trace_put_work_complete);
-}
-
-/*
- * Handle the completion of all outstanding I/O operations on a read request.
- * We inherit a ref from the caller.
- */
-void netfs_rreq_terminated(struct netfs_io_request *rreq, bool was_async)
-{
- if (!was_async)
- return netfs_rreq_assess(rreq);
- if (!work_pending(&rreq->work)) {
- netfs_get_request(rreq, netfs_rreq_trace_get_work);
- if (!queue_work(system_unbound_wq, &rreq->work))
- netfs_put_request(rreq, was_async, netfs_rreq_trace_put_work_nq);
- }
-}
-
/**
* netfs_read_subreq_progress - Note progress of a read operation.
* @subreq: The read request that has terminated.
- * @was_async: True if we're in an asynchronous context.
*
* This tells the read side of netfs lib that a contributory I/O operation has
* made some progress and that it may be possible to unlock some folios.
*
* Before calling, the filesystem should update subreq->transferred to track
* the amount of data copied into the output buffer.
- *
- * If @was_async is true, the caller might be running in softirq or interrupt
- * context and we can't sleep.
*/
-void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq,
- bool was_async)
+void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
+ might_sleep();
+
trace_netfs_sreq(subreq, netfs_sreq_trace_progress);
if (subreq->transferred > subreq->consumed &&
(rreq->origin == NETFS_READAHEAD ||
rreq->origin == NETFS_READPAGE ||
rreq->origin == NETFS_READ_FOR_WRITE)) {
- netfs_consume_read_data(subreq, was_async);
+ netfs_consume_read_data(subreq);
__clear_bit(NETFS_SREQ_NO_PROGRESS, &subreq->flags);
}
}
@@ -449,7 +425,6 @@ EXPORT_SYMBOL(netfs_read_subreq_progress);
/**
* netfs_read_subreq_terminated - Note the termination of an I/O operation.
* @subreq: The I/O request that has terminated.
- * @was_async: True if we're in an asynchronous context.
*
* This tells the read helper that a contributory I/O operation has terminated,
* one way or another, and that it should integrate the results.
@@ -463,7 +438,7 @@ EXPORT_SYMBOL(netfs_read_subreq_progress);
* Before calling, the filesystem should update subreq->transferred to track
* the amount of data copied into the output buffer.
*/
-void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq, bool was_async)
+void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq)
{
struct netfs_io_request *rreq = subreq->rreq;
@@ -497,7 +472,7 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq, bool was_a
(rreq->origin == NETFS_READAHEAD ||
rreq->origin == NETFS_READPAGE ||
rreq->origin == NETFS_READ_FOR_WRITE)) {
- netfs_consume_read_data(subreq, was_async);
+ netfs_consume_read_data(subreq);
__clear_bit(NETFS_SREQ_NO_PROGRESS, &subreq->flags);
}
rreq->transferred += subreq->transferred;
@@ -539,9 +514,9 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq, bool was_a
}
if (atomic_dec_and_test(&rreq->nr_outstanding))
- netfs_rreq_terminated(rreq, was_async);
+ netfs_rreq_terminated(rreq);
- netfs_put_subrequest(subreq, was_async, netfs_sreq_trace_put_terminated);
+ netfs_put_subrequest(subreq, false, netfs_sreq_trace_put_terminated);
}
EXPORT_SYMBOL(netfs_read_subreq_terminated);
@@ -557,6 +532,6 @@ void netfs_read_subreq_termination_worker(struct work_struct *work)
struct netfs_io_subrequest *subreq =
container_of(work, struct netfs_io_subrequest, work);
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
}
EXPORT_SYMBOL(netfs_read_subreq_termination_worker);
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index e72f5e674834..d29d5ca59458 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -34,8 +34,9 @@ void netfs_pgpriv2_mark_copy_to_cache(struct netfs_io_subrequest *subreq,
* [DEPRECATED] Cancel PG_private_2 on all marked folios in the event of an
* unrecoverable error.
*/
-static void netfs_pgpriv2_cancel(struct folio_queue *folioq)
+static void netfs_pgpriv2_cancel(struct rolling_buffer *buffer)
{
+ struct folio_queue *folioq = buffer->tail;
struct folio *folio;
int slot;
@@ -94,7 +95,7 @@ static int netfs_pgpriv2_copy_folio(struct netfs_io_request *wreq, struct folio
trace_netfs_folio(folio, netfs_folio_trace_store_copy);
/* Attach the folio to the rolling buffer. */
- if (netfs_buffer_append_folio(wreq, folio, false) < 0)
+ if (rolling_buffer_append(&wreq->buffer, folio, 0, wreq->gfp) < 0)
return -ENOMEM;
cache->submit_extendable_to = fsize;
@@ -109,7 +110,7 @@ static int netfs_pgpriv2_copy_folio(struct netfs_io_request *wreq, struct folio
do {
ssize_t part;
- wreq->io_iter.iov_offset = cache->submit_off;
+ wreq->buffer.iter.iov_offset = cache->submit_off;
atomic64_set(&wreq->issued_to, fpos + cache->submit_off);
cache->submit_extendable_to = fsize - cache->submit_off;
@@ -122,8 +123,8 @@ static int netfs_pgpriv2_copy_folio(struct netfs_io_request *wreq, struct folio
cache->submit_len -= part;
} while (cache->submit_len > 0);
- wreq->io_iter.iov_offset = 0;
- iov_iter_advance(&wreq->io_iter, fsize);
+ wreq->buffer.iter.iov_offset = 0;
+ rolling_buffer_advance(&wreq->buffer, fsize);
atomic64_set(&wreq->issued_to, fpos + fsize);
if (flen < fsize)
@@ -151,7 +152,7 @@ void netfs_pgpriv2_write_to_the_cache(struct netfs_io_request *rreq)
goto couldnt_start;
/* Need the first folio to be able to set up the op. */
- for (folioq = rreq->buffer; folioq; folioq = folioq->next) {
+ for (folioq = rreq->buffer.tail; folioq; folioq = folioq->next) {
if (folioq->marks3) {
slot = __ffs(folioq->marks3);
break;
@@ -199,7 +200,7 @@ void netfs_pgpriv2_write_to_the_cache(struct netfs_io_request *rreq)
netfs_put_request(wreq, false, netfs_rreq_trace_put_return);
_leave(" = %d", error);
couldnt_start:
- netfs_pgpriv2_cancel(rreq->buffer);
+ netfs_pgpriv2_cancel(&rreq->buffer);
}
/*
@@ -208,13 +209,13 @@ void netfs_pgpriv2_write_to_the_cache(struct netfs_io_request *rreq)
*/
bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)
{
- struct folio_queue *folioq = wreq->buffer;
+ struct folio_queue *folioq = wreq->buffer.tail;
unsigned long long collected_to = wreq->collected_to;
- unsigned int slot = wreq->buffer_head_slot;
+ unsigned int slot = wreq->buffer.first_tail_slot;
bool made_progress = false;
if (slot >= folioq_nr_slots(folioq)) {
- folioq = netfs_delete_buffer_head(wreq);
+ folioq = rolling_buffer_delete_spent(&wreq->buffer);
slot = 0;
}
@@ -253,9 +254,9 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)
folioq_clear(folioq, slot);
slot++;
if (slot >= folioq_nr_slots(folioq)) {
- if (READ_ONCE(wreq->buffer_tail) == folioq)
- break;
- folioq = netfs_delete_buffer_head(wreq);
+ folioq = rolling_buffer_delete_spent(&wreq->buffer);
+ if (!folioq)
+ goto done;
slot = 0;
}
@@ -263,7 +264,8 @@ bool netfs_pgpriv2_unlock_copied_folios(struct netfs_io_request *wreq)
break;
}
- wreq->buffer = folioq;
- wreq->buffer_head_slot = slot;
+ wreq->buffer.tail = folioq;
+done:
+ wreq->buffer.first_tail_slot = slot;
return made_progress;
}
diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c
index a5487b00ec56..c195dded0552 100644
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@ -233,7 +233,7 @@ void netfs_retry_reads(struct netfs_io_request *rreq)
netfs_retry_read_subrequests(rreq);
if (atomic_dec_and_test(&rreq->nr_outstanding))
- netfs_rreq_terminated(rreq, false);
+ netfs_rreq_terminated(rreq);
}
/*
@@ -244,7 +244,7 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq)
{
struct folio_queue *p;
- for (p = rreq->buffer; p; p = p->next) {
+ for (p = rreq->buffer.tail; p; p = p->next) {
for (int slot = 0; slot < folioq_count(p); slot++) {
struct folio *folio = folioq_folio(p, slot);
diff --git a/fs/netfs/rolling_buffer.c b/fs/netfs/rolling_buffer.c
new file mode 100644
index 000000000000..7c768ca5ff59
--- /dev/null
+++ b/fs/netfs/rolling_buffer.c
@@ -0,0 +1,230 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/* Rolling buffer helpers
+ *
+ * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#include <linux/bitops.h>
+#include <linux/mempool.h>
+#include <linux/pagemap.h>
+#include <linux/rolling_buffer.h>
+#include <linux/slab.h>
+#include "internal.h"
+
+static atomic_t debug_ids;
+
+/**
+ * netfs_folioq_alloc - Allocate a folio_queue struct
+ * @rreq_id: Associated debugging ID for tracing purposes
+ * @gfp: Allocation constraints
+ * @trace: Trace tag to indicate the purpose of the allocation
+ *
+ * Allocate, initialise and account the folio_queue struct and log a trace line
+ * to mark the allocation.
+ */
+struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,
+ unsigned int /*enum netfs_folioq_trace*/ trace)
+{
+ struct folio_queue *fq;
+
+ if (gfp == GFP_KERNEL)
+ fq = netfs_folioq_pool.alloc(gfp, netfs_folioq_pool.pool_data);
+ else
+ fq = mempool_alloc(&netfs_folioq_pool, gfp);
+ if (fq) {
+ netfs_stat(&netfs_n_folioq);
+ folioq_init(fq, rreq_id);
+ fq->debug_id = atomic_inc_return(&debug_ids);
+ trace_netfs_folioq(fq, trace);
+ }
+ return fq;
+}
+EXPORT_SYMBOL(netfs_folioq_alloc);
+
+/**
+ * netfs_folioq_free - Free a folio_queue struct
+ * @folioq: The object to free
+ * @trace: Trace tag to indicate which free
+ *
+ * Free and unaccount the folio_queue struct.
+ */
+void netfs_folioq_free(struct folio_queue *folioq,
+ unsigned int /*enum netfs_trace_folioq*/ trace)
+{
+ trace_netfs_folioq(folioq, trace);
+ netfs_stat_d(&netfs_n_folioq);
+ mempool_free(folioq, &netfs_folioq_pool);
+}
+EXPORT_SYMBOL(netfs_folioq_free);
+
+/*
+ * Initialise a rolling buffer. We allocate an empty folio queue struct to so
+ * that the pointers can be independently driven by the producer and the
+ * consumer.
+ */
+int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
+ unsigned int direction, gfp_t gfp)
+{
+ struct folio_queue *fq;
+
+ fq = netfs_folioq_alloc(rreq_id, gfp, netfs_trace_folioq_rollbuf_init);
+ if (!fq)
+ return -ENOMEM;
+
+ roll->head = fq;
+ roll->tail = fq;
+ iov_iter_folio_queue(&roll->iter, direction, fq, 0, 0, 0);
+ return 0;
+}
+
+/*
+ * Add another folio_queue to a rolling buffer if there's no space left.
+ */
+int rolling_buffer_make_space(struct rolling_buffer *roll, gfp_t gfp)
+{
+ struct folio_queue *fq, *head = roll->head;
+
+ if (!folioq_full(head))
+ return 0;
+
+ fq = netfs_folioq_alloc(head->rreq_id, gfp, netfs_trace_folioq_make_space);
+ if (!fq)
+ return -ENOMEM;
+ fq->prev = head;
+
+ roll->head = fq;
+ if (folioq_full(head)) {
+ /* Make sure we don't leave the master iterator pointing to a
+ * block that might get immediately consumed.
+ */
+ if (roll->iter.folioq == head &&
+ roll->iter.folioq_slot == folioq_nr_slots(head)) {
+ roll->iter.folioq = fq;
+ roll->iter.folioq_slot = 0;
+ }
+ }
+
+ /* Make sure the initialisation is stored before the next pointer.
+ *
+ * [!] NOTE: After we set head->next, the consumer is at liberty to
+ * immediately delete the old head.
+ */
+ smp_store_release(&head->next, fq);
+ return 0;
+}
+
+/*
+ * Decant the list of folios to read into a rolling buffer.
+ */
+ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
+ struct readahead_control *ractl,
+ struct folio_batch *put_batch)
+{
+ struct folio_queue *fq;
+ struct page **vec;
+ int nr, ix, to;
+ ssize_t size = 0;
+
+ if (rolling_buffer_make_space(roll, GFP_KERNEL) < 0)
+ return -ENOMEM;
+
+ fq = roll->head;
+ vec = (struct page **)fq->vec.folios;
+ nr = __readahead_batch(ractl, vec + folio_batch_count(&fq->vec),
+ folio_batch_space(&fq->vec));
+ ix = fq->vec.nr;
+ to = ix + nr;
+ fq->vec.nr = to;
+ for (; ix < to; ix++) {
+ struct folio *folio = folioq_folio(fq, ix);
+ unsigned int order = folio_order(folio);
+
+ fq->orders[ix] = order;
+ size += PAGE_SIZE << order;
+ trace_netfs_folio(folio, netfs_folio_trace_read);
+ if (!folio_batch_add(put_batch, folio))
+ folio_batch_release(put_batch);
+ }
+ WRITE_ONCE(roll->iter.count, roll->iter.count + size);
+
+ /* Store the counter after setting the slot. */
+ smp_store_release(&roll->next_head_slot, to);
+
+ for (; ix < folioq_nr_slots(fq); ix++)
+ folioq_clear(fq, ix);
+
+ return size;
+}
+
+/*
+ * Append a folio to the rolling buffer.
+ */
+ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
+ unsigned int flags, gfp_t gfp)
+{
+ ssize_t size = folio_size(folio);
+ int slot;
+
+ if (rolling_buffer_make_space(roll, gfp) < 0)
+ return -ENOMEM;
+
+ slot = folioq_append(roll->head, folio);
+ if (flags & ROLLBUF_MARK_1)
+ folioq_mark(roll->head, slot);
+ if (flags & ROLLBUF_MARK_2)
+ folioq_mark2(roll->head, slot);
+
+ WRITE_ONCE(roll->iter.count, roll->iter.count + size);
+
+ /* Store the counter after setting the slot. */
+ smp_store_release(&roll->next_head_slot, slot);
+ return size;
+}
+
+/*
+ * Delete a spent buffer from a rolling queue and return the next in line. We
+ * don't return the last buffer to keep the pointers independent, but return
+ * NULL instead.
+ */
+struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll)
+{
+ struct folio_queue *spent = roll->tail, *next = READ_ONCE(spent->next);
+
+ if (!next)
+ return NULL;
+ next->prev = NULL;
+ netfs_folioq_free(spent, netfs_trace_folioq_delete);
+ roll->tail = next;
+ return next;
+}
+
+/*
+ * Clear out a rolling queue. Folios that have mark 1 set are put.
+ */
+void rolling_buffer_clear(struct rolling_buffer *roll)
+{
+ struct folio_batch fbatch;
+ struct folio_queue *p;
+
+ folio_batch_init(&fbatch);
+
+ while ((p = roll->tail)) {
+ roll->tail = p->next;
+ for (int slot = 0; slot < folioq_count(p); slot++) {
+ struct folio *folio = folioq_folio(p, slot);
+
+ if (!folio)
+ continue;
+ if (folioq_is_marked(p, slot)) {
+ trace_netfs_folio(folio, netfs_folio_trace_put);
+ if (!folio_batch_add(&fbatch, folio))
+ folio_batch_release(&fbatch);
+ }
+ }
+
+ netfs_folioq_free(p, netfs_trace_folioq_clear);
+ }
+
+ folio_batch_release(&fbatch);
+}
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index c349867d74c3..53790d0dc334 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -83,9 +83,9 @@ int netfs_folio_written_back(struct folio *folio)
static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
unsigned int *notes)
{
- struct folio_queue *folioq = wreq->buffer;
+ struct folio_queue *folioq = wreq->buffer.tail;
unsigned long long collected_to = wreq->collected_to;
- unsigned int slot = wreq->buffer_head_slot;
+ unsigned int slot = wreq->buffer.first_tail_slot;
if (wreq->origin == NETFS_PGPRIV2_COPY_TO_CACHE) {
if (netfs_pgpriv2_unlock_copied_folios(wreq))
@@ -94,7 +94,9 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
}
if (slot >= folioq_nr_slots(folioq)) {
- folioq = netfs_delete_buffer_head(wreq);
+ folioq = rolling_buffer_delete_spent(&wreq->buffer);
+ if (!folioq)
+ return;
slot = 0;
}
@@ -134,9 +136,9 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
folioq_clear(folioq, slot);
slot++;
if (slot >= folioq_nr_slots(folioq)) {
- if (READ_ONCE(wreq->buffer_tail) == folioq)
- break;
- folioq = netfs_delete_buffer_head(wreq);
+ folioq = rolling_buffer_delete_spent(&wreq->buffer);
+ if (!folioq)
+ goto done;
slot = 0;
}
@@ -144,8 +146,9 @@ static void netfs_writeback_unlock_folios(struct netfs_io_request *wreq,
break;
}
- wreq->buffer = folioq;
- wreq->buffer_head_slot = slot;
+ wreq->buffer.tail = folioq;
+done:
+ wreq->buffer.first_tail_slot = slot;
}
/*
@@ -263,7 +266,9 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
list_for_each_entry_safe_from(subreq, tmp,
&stream->subrequests, rreq_link) {
trace_netfs_sreq(subreq, netfs_sreq_trace_discard);
+ spin_lock_bh(&wreq->lock);
list_del(&subreq->rreq_link);
+ spin_unlock_bh(&wreq->lock);
netfs_put_subrequest(subreq, false, netfs_sreq_trace_put_done);
if (subreq == to)
break;
@@ -287,8 +292,10 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq,
netfs_sreq_trace_new);
trace_netfs_sreq(subreq, netfs_sreq_trace_split);
+ spin_lock_bh(&wreq->lock);
list_add(&subreq->rreq_link, &to->rreq_link);
- to = list_next_entry(to, rreq_link);
+ spin_unlock_bh(&wreq->lock);
+ to = subreq;
trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
stream->sreq_max_len = len;
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index 7431c0a4252c..b115f5285f43 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -105,8 +105,10 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
_enter("R=%x", wreq->debug_id);
ictx = netfs_inode(wreq->inode);
- if (is_buffered && netfs_is_cache_enabled(ictx))
+ if (is_buffered)
fscache_begin_write_operation(&wreq->cache_resources, netfs_i_cookie(ictx));
+ if (rolling_buffer_init(&wreq->buffer, wreq->debug_id, ITER_SOURCE, wreq->gfp) < 0)
+ goto nomem;
wreq->cleaned_to = wreq->start;
@@ -129,6 +131,10 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
}
return wreq;
+nomem:
+ wreq->error = -ENOMEM;
+ netfs_put_request(wreq, false, netfs_rreq_trace_put_failed);
+ return ERR_PTR(-ENOMEM);
}
/**
@@ -153,16 +159,15 @@ static void netfs_prepare_write(struct netfs_io_request *wreq,
loff_t start)
{
struct netfs_io_subrequest *subreq;
- struct iov_iter *wreq_iter = &wreq->io_iter;
+ struct iov_iter *wreq_iter = &wreq->buffer.iter;
/* Make sure we don't point the iterator at a used-up folio_queue
* struct being used as a placeholder to prevent the queue from
* collapsing. In such a case, extend the queue.
*/
if (iov_iter_is_folioq(wreq_iter) &&
- wreq_iter->folioq_slot >= folioq_nr_slots(wreq_iter->folioq)) {
- netfs_buffer_make_space(wreq);
- }
+ wreq_iter->folioq_slot >= folioq_nr_slots(wreq_iter->folioq))
+ rolling_buffer_make_space(&wreq->buffer, wreq->gfp);
subreq = netfs_alloc_subrequest(wreq);
subreq->source = stream->source;
@@ -325,6 +330,9 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
_enter("");
+ if (rolling_buffer_make_space(&wreq->buffer, wreq->gfp) < 0)
+ return -ENOMEM;
+
/* netfs_perform_write() may shift i_size around the page or from out
* of the page to beyond it, but cannot move i_size into or through the
* page since we have it locked.
@@ -424,7 +432,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
}
/* Attach the folio to the rolling buffer. */
- netfs_buffer_append_folio(wreq, folio, false);
+ rolling_buffer_append(&wreq->buffer, folio, 0, wreq->gfp);
/* Move the submission point forward to allow for write-streaming data
* not starting at the front of the page. We don't do write-streaming
@@ -471,7 +479,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
/* Advance the iterator(s). */
if (stream->submit_off > iter_off) {
- iov_iter_advance(&wreq->io_iter, stream->submit_off - iter_off);
+ rolling_buffer_advance(&wreq->buffer, stream->submit_off - iter_off);
iter_off = stream->submit_off;
}
@@ -489,7 +497,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
}
if (fsize > iter_off)
- iov_iter_advance(&wreq->io_iter, fsize - iter_off);
+ rolling_buffer_advance(&wreq->buffer, fsize - iter_off);
atomic64_set(&wreq->issued_to, fpos + fsize);
if (!debug)
@@ -574,6 +582,10 @@ int netfs_writepages(struct address_space *mapping,
}
error = netfs_write_folio(wreq, wbc, folio);
+ if (error == -ENOMEM) {
+ folio_redirty_for_writepage(wbc, folio);
+ folio_unlock(folio);
+ }
} while ((folio = writeback_iter(mapping, wbc, folio, &error)));
netfs_end_issue_write(wreq);
@@ -635,7 +647,7 @@ int netfs_advance_writethrough(struct netfs_io_request *wreq, struct writeback_c
int ret;
_enter("R=%x ic=%zu ws=%u cp=%zu tp=%u",
- wreq->debug_id, wreq->iter.count, wreq->wsize, copied, to_page_end);
+ wreq->debug_id, wreq->buffer.iter.count, wreq->wsize, copied, to_page_end);
/* The folio is locked. */
@@ -724,7 +736,7 @@ int netfs_unbuffered_write(struct netfs_io_request *wreq, bool may_wait, size_t
part = netfs_advance_write(wreq, upload, start, len, false);
start += part;
len -= part;
- iov_iter_advance(&wreq->io_iter, part);
+ rolling_buffer_advance(&wreq->buffer, part);
if (test_bit(NETFS_RREQ_PAUSE, &wreq->flags)) {
trace_netfs_rreq(wreq, netfs_rreq_trace_wait_pause);
wait_on_bit(&wreq->flags, NETFS_RREQ_PAUSE, TASK_UNINTERRUPTIBLE);
diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c
index 4254ba3ee7c5..ada04a7230c2 100644
--- a/fs/nfs/callback_xdr.c
+++ b/fs/nfs/callback_xdr.c
@@ -422,9 +422,9 @@ static __be32 decode_cb_sequence_args(struct svc_rqst *rqstp,
args->csa_nrclists = ntohl(*p++);
args->csa_rclists = NULL;
if (args->csa_nrclists) {
- args->csa_rclists = kmalloc_array(args->csa_nrclists,
- sizeof(*args->csa_rclists),
- GFP_KERNEL);
+ args->csa_rclists = kcalloc(args->csa_nrclists,
+ sizeof(*args->csa_rclists),
+ GFP_KERNEL);
if (unlikely(args->csa_rclists == NULL))
return htonl(NFS4ERR_RESOURCE);
diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index c1bbc358833d..14a5f3cc3ef5 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -455,11 +455,14 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
struct nfs_inode *nfsi = NFS_I(inode);
struct nfs_delegation *delegation, *old_delegation;
struct nfs_delegation *freeme = NULL;
+ bool orphaned = false;
int status = 0;
delegation = kmalloc(sizeof(*delegation), GFP_KERNEL_ACCOUNT);
- if (delegation == NULL)
+ if (delegation == NULL) {
+ nfs4_proc_delegreturn(inode, cred, stateid, NULL, 0);
return -ENOMEM;
+ }
nfs4_stateid_copy(&delegation->stateid, stateid);
refcount_set(&delegation->refcount, 1);
delegation->type = type;
@@ -507,12 +510,16 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
goto out;
}
if (test_and_set_bit(NFS_DELEGATION_RETURNING,
- &old_delegation->flags))
+ &old_delegation->flags)) {
+ orphaned = true;
goto out;
+ }
}
freeme = nfs_detach_delegation_locked(nfsi, old_delegation, clp);
- if (freeme == NULL)
+ if (freeme == NULL) {
+ orphaned = true;
goto out;
+ }
add_new:
/*
* If we didn't revalidate the change attribute before setting
@@ -544,8 +551,11 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
nfs_update_delegated_mtime(inode);
out:
spin_unlock(&clp->cl_lock);
- if (delegation != NULL)
+ if (delegation != NULL) {
+ if (orphaned)
+ nfs_do_return_delegation(inode, delegation, 0);
__nfs_free_delegation(delegation);
+ }
if (freeme != NULL) {
nfs_do_return_delegation(inode, freeme, 0);
nfs_free_delegation(freeme);
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 7de5575f4bee..61705e7f0be7 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -1154,7 +1154,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
struct pnfs_layout_hdr *lo = lseg->pls_layout;
struct inode *inode = lo->plh_inode;
struct nfs4_deviceid_node *devid = FF_LAYOUT_DEVID_NODE(lseg, idx);
- struct nfs4_slot_table *tbl = &clp->cl_session->fc_slot_table;
+ struct nfs4_slot_table *tbl = nfs4_has_session(clp) ?
+ &clp->cl_session->fc_slot_table : clp->cl_slot_tbl;
switch (op_status) {
case NFS4_OK:
diff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c
index be14d30608f6..e278a1ad1ca3 100644
--- a/fs/nfs/fscache.c
+++ b/fs/nfs/fscache.c
@@ -316,7 +316,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq)
netfs = nfs_netfs_alloc(sreq);
if (!netfs) {
sreq->error = -ENOMEM;
- return netfs_read_subreq_terminated(sreq, false);
+ return netfs_read_subreq_terminated(sreq);
}
pgio.pg_netfs = netfs; /* used in completion */
diff --git a/fs/nfs/fscache.h b/fs/nfs/fscache.h
index 1d86f7cc7195..9d86868f4998 100644
--- a/fs/nfs/fscache.h
+++ b/fs/nfs/fscache.h
@@ -75,7 +75,7 @@ static inline void nfs_netfs_put(struct nfs_netfs_io_data *netfs)
netfs->sreq->transferred = min_t(s64, netfs->sreq->len,
atomic64_read(&netfs->transferred));
netfs->sreq->error = netfs->error;
- netfs_read_subreq_terminated(netfs->sreq, false);
+ netfs_read_subreq_terminated(netfs->sreq);
kfree(netfs);
}
static inline void nfs_netfs_inode_init(struct nfs_inode *nfsi)
diff --git a/fs/nfs/localio.c b/fs/nfs/localio.c
index c59f205b5130..fe76b8a9f2e6 100644
--- a/fs/nfs/localio.c
+++ b/fs/nfs/localio.c
@@ -270,6 +270,18 @@ nfs_bvec_alloc_and_import_pagevec(struct page **pagevec,
return bvec;
}
+/*
+ * Ensure all page cache allocations are done from GFP_NOFS context to
+ * prevent direct reclaim recursion back into NFS via nfs_writepages.
+ */
+static void
+nfs_local_mapping_set_gfp_nofs_context(struct address_space *m)
+{
+ gfp_t gfp_mask = mapping_gfp_mask(m);
+
+ mapping_set_gfp_mask(m, (gfp_mask & ~(__GFP_FS)));
+}
+
static void
nfs_local_iocb_free(struct nfs_local_kiocb *iocb)
{
@@ -293,6 +305,7 @@ nfs_local_iocb_alloc(struct nfs_pgio_header *hdr,
return NULL;
}
+ nfs_local_mapping_set_gfp_nofs_context(file->f_mapping);
if (localio_O_DIRECT_semantics &&
test_bit(NFS_IOHDR_ODIRECT, &hdr->flags)) {
iocb->kiocb.ki_filp = file;
@@ -734,6 +747,8 @@ nfs_local_run_commit(struct file *filp, struct nfs_commit_data *data)
end = LLONG_MAX;
}
+ nfs_local_mapping_set_gfp_nofs_context(filp->f_mapping);
+
dprintk("%s: commit %llu - %llu\n", __func__, start, end);
return vfs_fsync_range(filp, start, end, 0);
}
diff --git a/fs/nfs/nfs4session.c b/fs/nfs/nfs4session.c
index 5db460476bf2..c59e710eb7cc 100644
--- a/fs/nfs/nfs4session.c
+++ b/fs/nfs/nfs4session.c
@@ -634,16 +634,22 @@ int nfs4_init_ds_session(struct nfs_client *clp, unsigned long lease_time)
int ret;
spin_lock(&clp->cl_lock);
- if (test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) {
- /*
- * Do not set NFS_CS_CHECK_LEASE_TIME instead set the
- * DS lease to be equal to the MDS lease.
- */
+ /*
+ * Do not set NFS_CS_CHECK_LEASE_TIME instead set the
+ * DS lease to be equal to the MDS lease.
+ *
+ * A v4.0 DS has no session, so seed the lease every time.
+ */
+ if (!session ||
+ test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) {
clp->cl_lease_time = lease_time;
clp->cl_last_renewal = jiffies;
}
spin_unlock(&clp->cl_lock);
+ if (!session)
+ return 0;
+
ret = nfs41_check_session_ready(clp);
if (ret)
return ret;
diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c
index 1eee2dc936d9..a08974e268b8 100644
--- a/fs/nfsd/filecache.c
+++ b/fs/nfsd/filecache.c
@@ -67,11 +67,6 @@ static DEFINE_PER_CPU(unsigned long, nfsd_file_releases);
static DEFINE_PER_CPU(unsigned long, nfsd_file_total_age);
static DEFINE_PER_CPU(unsigned long, nfsd_file_evictions);
-struct nfsd_fcache_disposal {
- spinlock_t lock;
- struct list_head freeme;
-};
-
static struct kmem_cache *nfsd_file_slab;
static struct kmem_cache *nfsd_file_mark_slab;
static struct list_lru nfsd_file_lru;
@@ -437,11 +432,12 @@ nfsd_file_dispose_list(struct list_head *dispose)
}
/**
- * nfsd_file_dispose_list_delayed - move list of dead files to net's freeme list
+ * nfsd_file_dispose_list_delayed - queue dead files for nfsd thread disposal
* @dispose: list of nfsd_files to be disposed
*
- * Transfers each file to the "freeme" list for its nfsd_net, to eventually
- * be disposed of by the per-net garbage collector.
+ * Transfers each file to the dispose list in its nfsd_net and wakes an nfsd
+ * thread to do the actual close. This keeps the cost of fput() in the nfsd
+ * threads rather than in the shrinker or GC worker.
*
* All callers must hold nfsd_gc_lock, so that nfsd_file_cache_shutdown_net()
* can synchronize against them before draining the per-net dispose list.
@@ -456,12 +452,11 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose)
struct nfsd_file *nf = list_first_entry(dispose,
struct nfsd_file, nf_gc);
struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id);
- struct nfsd_fcache_disposal *l = nn->fcache_disposal;
struct svc_serv *serv;
- spin_lock(&l->lock);
- list_move_tail(&nf->nf_gc, &l->freeme);
- spin_unlock(&l->lock);
+ spin_lock(&nn->fcache_dispose_lock);
+ list_move_tail(&nf->nf_gc, &nn->fcache_dispose_list);
+ spin_unlock(&nn->fcache_dispose_lock);
/*
* The filecache laundrette is shut down after the
@@ -485,17 +480,15 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose)
*/
void nfsd_file_net_dispose(struct nfsd_net *nn)
{
- struct nfsd_fcache_disposal *l = nn->fcache_disposal;
-
- if (!list_empty(&l->freeme)) {
+ if (!list_empty(&nn->fcache_dispose_list)) {
LIST_HEAD(dispose);
int i;
- spin_lock(&l->lock);
- for (i = 0; i < 8 && !list_empty(&l->freeme); i++)
- list_move(l->freeme.next, &dispose);
- spin_unlock(&l->lock);
- if (!list_empty(&l->freeme)) {
+ spin_lock(&nn->fcache_dispose_lock);
+ for (i = 0; i < 8 && !list_empty(&nn->fcache_dispose_list); i++)
+ list_move(nn->fcache_dispose_list.next, &dispose);
+ spin_unlock(&nn->fcache_dispose_lock);
+ if (!list_empty(&nn->fcache_dispose_list)) {
/*
* Wake up another thread to share the work
* *before* doing any actual disposing.
@@ -685,11 +678,11 @@ nfsd_file_queue_for_close(struct inode *inode, struct list_head *dispose)
}
/**
- * nfsd_file_close_inode - attempt a delayed close of a nfsd_file
+ * nfsd_file_close_inode - attempt a deferred close of a nfsd_file
* @inode: inode of the file to attempt to remove
*
* Close out any open nfsd_files that can be reaped for @inode. The
- * actual freeing is deferred to the dispose_list_delayed infrastructure.
+ * actual freeing is deferred to the nfsd service threads.
*
* This is used by the fsnotify callbacks and setlease notifier.
*/
@@ -892,42 +885,14 @@ __nfsd_file_cache_purge(struct net *net)
nfsd_file_dispose_list(&dispose);
}
-static struct nfsd_fcache_disposal *
-nfsd_alloc_fcache_disposal(void)
-{
- struct nfsd_fcache_disposal *l;
-
- l = kmalloc(sizeof(*l), GFP_KERNEL);
- if (!l)
- return NULL;
- spin_lock_init(&l->lock);
- INIT_LIST_HEAD(&l->freeme);
- return l;
-}
-
-static void
-nfsd_free_fcache_disposal(struct nfsd_fcache_disposal *l)
-{
- nfsd_file_dispose_list(&l->freeme);
- kfree(l);
-}
-
-static void
-nfsd_free_fcache_disposal_net(struct net *net)
-{
- struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- struct nfsd_fcache_disposal *l = nn->fcache_disposal;
-
- nfsd_free_fcache_disposal(l);
-}
-
int
nfsd_file_cache_start_net(struct net *net)
{
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- nn->fcache_disposal = nfsd_alloc_fcache_disposal();
- return nn->fcache_disposal ? 0 : -ENOMEM;
+ spin_lock_init(&nn->fcache_dispose_lock);
+ INIT_LIST_HEAD(&nn->fcache_dispose_list);
+ return 0;
}
/**
@@ -946,6 +911,8 @@ nfsd_file_cache_purge(struct net *net)
void
nfsd_file_cache_shutdown_net(struct net *net)
{
+ struct nfsd_net *nn = net_generic(net, nfsd_net_id);
+
nfsd_file_cache_purge(net);
/*
* Ensure any in-progress shrinker, GC, or fsnotify/lease callback
@@ -955,7 +922,7 @@ nfsd_file_cache_shutdown_net(struct net *net)
*/
spin_lock(&nfsd_gc_lock);
spin_unlock(&nfsd_gc_lock);
- nfsd_free_fcache_disposal_net(net);
+ nfsd_file_dispose_list(&nn->fcache_dispose_list);
}
void
diff --git a/fs/nfsd/lockd.c b/fs/nfsd/lockd.c
index 6b042218668b..e933788e7a35 100644
--- a/fs/nfsd/lockd.c
+++ b/fs/nfsd/lockd.c
@@ -88,6 +88,7 @@ nlm_fclose(struct file *filp)
}
static const struct nlmsvc_binding nfsd_nlm_ops = {
+ .owner = THIS_MODULE,
.fopen = nlm_fopen, /* open file for locking */
.fclose = nlm_fclose, /* close file */
};
@@ -96,11 +97,12 @@ void
nfsd_lockd_init(void)
{
dprintk("nfsd: initializing lockd\n");
- nlmsvc_ops = &nfsd_nlm_ops;
+ rcu_assign_pointer(nlmsvc_ops, &nfsd_nlm_ops);
}
void
nfsd_lockd_shutdown(void)
{
- nlmsvc_ops = NULL;
+ RCU_INIT_POINTER(nlmsvc_ops, NULL);
+ synchronize_rcu();
}
diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
index b203a15a909d..a42d36b85b22 100644
--- a/fs/nfsd/netns.h
+++ b/fs/nfsd/netns.h
@@ -27,6 +27,16 @@
struct cld_net;
struct nfsd4_client_tracking_ops;
+enum nfsd_net_flag {
+ NFSD_NET_GRACE_ENDED,
+ NFSD_NET_GRACE_END_FORCED,
+ NFSD_NET_IN_GRACE,
+ NFSD_NET_SOMEBODY_RECLAIMED,
+ NFSD_NET_TRACK_RECLAIM_COMPLETES,
+ NFSD_NET_UP,
+ NFSD_NET_LOCKD_UP,
+};
+
enum {
/* cache misses due only to checksum comparison failures */
NFSD_STATS_PAYLOAD_MISSES,
@@ -65,8 +75,7 @@ struct nfsd_net {
struct cache_detail *nametoid_cache;
struct lock_manager nfsd4_manager;
- bool grace_ended;
- bool grace_end_forced;
+ unsigned long flags;
bool client_tracking_active;
time64_t boot_time;
time64_t boot_time_bt; /* same instant in CLOCK_BOOTTIME */
@@ -124,19 +133,13 @@ struct nfsd_net {
spinlock_t blocked_locks_lock;
struct file *rec_file;
- bool in_grace;
const struct nfsd4_client_tracking_ops *client_tracking_ops;
time64_t nfsd4_lease;
time64_t nfsd4_grace;
- bool somebody_reclaimed;
- bool track_reclaim_completes;
atomic_t nr_reclaim_complete;
- bool nfsd_net_up;
- bool lockd_up;
-
seqlock_t writeverf_lock;
unsigned char writeverf[8];
@@ -216,7 +219,8 @@ struct nfsd_net {
/* utsname taken from the process that starts the server */
char nfsd_name[UNX_MAXNODENAME+1];
- struct nfsd_fcache_disposal *fcache_disposal;
+ spinlock_t fcache_dispose_lock;
+ struct list_head fcache_dispose_list;
siphash_key_t siphash_key;
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index efc7cd0ae73b..f94c01d23ea7 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -627,7 +627,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
pr_warn("nfsd4_process_open2 failed to open newly-created file: status=%u\n",
be32_to_cpu(status));
if (reclaim && !status)
- nn->somebody_reclaimed = true;
+ set_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags);
out:
if (open->op_filp) {
fput(open->op_filp);
diff --git a/fs/nfsd/nfs4recover.c b/fs/nfsd/nfs4recover.c
index 41bc53288a14..d49bc03afc2f 100644
--- a/fs/nfsd/nfs4recover.c
+++ b/fs/nfsd/nfs4recover.c
@@ -229,7 +229,7 @@ nfsd4_create_clid_dir(struct nfs4_client *clp)
out_unlock:
inode_unlock(d_inode(dir));
if (status == 0) {
- if (nn->in_grace)
+ if (test_bit(NFSD_NET_IN_GRACE, &nn->flags))
__nfsd4_create_reclaim_record_grace(clp, dname, nn);
vfs_fsync(nn->rec_file, 0);
} else {
@@ -391,7 +391,7 @@ nfsd4_remove_clid_dir(struct nfs4_client *clp)
nfs4_reset_creds(original_cred);
if (status == 0) {
vfs_fsync(nn->rec_file, 0);
- if (nn->in_grace)
+ if (test_bit(NFSD_NET_IN_GRACE, &nn->flags))
__nfsd4_remove_reclaim_record_grace(dname,
HEXDIR_LEN, nn);
}
@@ -449,7 +449,7 @@ nfsd4_recdir_purge_old(struct nfsd_net *nn)
{
int status;
- nn->in_grace = false;
+ clear_bit(NFSD_NET_IN_GRACE, &nn->flags);
if (!nn->rec_file)
return;
status = mnt_want_write_file(nn->rec_file);
@@ -531,7 +531,7 @@ nfsd4_init_recdir(struct net *net)
nfs4_reset_creds(original_cred);
if (!status)
- nn->in_grace = true;
+ set_bit(NFSD_NET_IN_GRACE, &nn->flags);
return status;
}
@@ -1510,7 +1510,7 @@ nfs4_cld_state_init(struct net *net)
INIT_LIST_HEAD(&nn->reclaim_str_hashtbl[i]);
nn->reclaim_str_hashtbl_size = 0;
init_rwsem(&nn->reclaim_str_hashtbl_lock);
- nn->track_reclaim_completes = true;
+ set_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags);
atomic_set(&nn->nr_reclaim_complete, 0);
return 0;
@@ -1521,7 +1521,7 @@ nfs4_cld_state_shutdown(struct net *net)
{
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- nn->track_reclaim_completes = false;
+ clear_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags);
kfree(nn->reclaim_str_hashtbl);
}
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 0e74cf87fc54..c9a55105fa8e 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -2608,7 +2608,7 @@ static void inc_reclaim_complete(struct nfs4_client *clp)
{
struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id);
- if (!nn->track_reclaim_completes)
+ if (!test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags))
return;
down_read(&nn->reclaim_str_hashtbl_lock);
@@ -4863,8 +4863,6 @@ nfsd4_init_leases_net(struct nfsd_net *nn)
nn->nfsd4_lease = 90; /* default lease time */
nn->nfsd4_grace = 90;
- nn->somebody_reclaimed = false;
- nn->track_reclaim_completes = false;
nn->clverifier_counter = get_random_u32();
nn->clientid_base = get_random_u32();
nn->clientid_counter = nn->clientid_base + 1;
@@ -6467,12 +6465,21 @@ nfsd4_renew(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
static void
nfsd4_end_grace(struct nfsd_net *nn)
{
- /* do nothing if grace period already ended */
- if (nn->grace_ended)
+ /*
+ * nfsd4_end_grace() can be entered concurrently from the
+ * laundromat workqueue and from an nfsd compound thread
+ * handling RECLAIM_COMPLETE. Without serialization, both
+ * callers can observe NFSD_NET_GRACE_ENDED clear and proceed
+ * into nfsd4_record_grace_done(). For tracking ops whose
+ * grace_done drains reclaim_str_hashtbl, that results in
+ * list corruption and a double free of every
+ * nfs4_client_reclaim entry. Use an atomic test-and-set so
+ * exactly one caller proceeds.
+ */
+ if (test_and_set_bit(NFSD_NET_GRACE_ENDED, &nn->flags))
return;
trace_nfsd_grace_complete(nn);
- nn->grace_ended = true;
/*
* If the server goes down again right now, an NFSv4
* client will still be allowed to reclaim after it comes back up,
@@ -6514,11 +6521,12 @@ bool nfsd4_force_end_grace(struct nfsd_net *nn)
if (!nn->client_tracking_ops)
return false;
spin_lock(&nn->client_lock);
- if (nn->grace_ended || !nn->client_tracking_active) {
+ if (test_bit(NFSD_NET_GRACE_ENDED, &nn->flags) ||
+ !nn->client_tracking_active) {
spin_unlock(&nn->client_lock);
return false;
}
- WRITE_ONCE(nn->grace_end_forced, true);
+ set_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags);
mod_delayed_work(laundry_wq, &nn->laundromat_work, 0);
spin_unlock(&nn->client_lock);
return true;
@@ -6533,9 +6541,9 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)
time64_t double_grace_period_end = nn->boot_time_bt +
2 * nn->nfsd4_lease;
- if (READ_ONCE(nn->grace_end_forced))
+ if (test_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags))
return false;
- if (nn->track_reclaim_completes) {
+ if (test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags)) {
int size;
down_read(&nn->reclaim_str_hashtbl_lock);
@@ -6544,9 +6552,9 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)
if (atomic_read(&nn->nr_reclaim_complete) == size)
return false;
}
- if (!nn->somebody_reclaimed)
+ if (!test_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags))
return false;
- nn->somebody_reclaimed = false;
+ clear_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags);
/*
* If we've given them *two* lease times to reclaim, and they're
* still not done, give up:
@@ -8404,7 +8412,7 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid);
status = 0;
if (lock->lk_reclaim)
- nn->somebody_reclaimed = true;
+ set_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags);
break;
case FILE_LOCK_DEFERRED:
kref_put(&nbl->nbl_kref, free_nbl);
@@ -8974,8 +8982,8 @@ static int nfs4_state_create_net(struct net *net)
nn->unconf_name_tree = RB_ROOT;
nn->boot_time = ktime_get_real_seconds();
nn->boot_time_bt = ktime_get_boottime_seconds();
- nn->grace_ended = false;
- nn->grace_end_forced = false;
+ clear_bit(NFSD_NET_GRACE_ENDED, &nn->flags);
+ clear_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags);
nn->client_tracking_active = false;
nn->nfsd4_manager.block_opens = true;
INIT_LIST_HEAD(&nn->nfsd4_manager.list);
@@ -9062,7 +9070,8 @@ nfs4_state_start_net(struct net *net)
spin_lock(&nn->client_lock);
nn->client_tracking_active = true;
spin_unlock(&nn->client_lock);
- if (nn->track_reclaim_completes && nn->reclaim_str_hashtbl_size == 0)
+ if (test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags) &&
+ nn->reclaim_str_hashtbl_size == 0)
goto skip_grace;
printk(KERN_INFO "NFSD: starting %lld-second grace period (net %x)\n",
nn->nfsd4_grace, net->ns.inum);
diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index 16fd95a23e11..5646e07b74c3 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -301,7 +301,7 @@ static ssize_t write_unlock_fs(struct file *file, char *buf, size_t size)
error = nlmsvc_unlock_all_by_sb(path.dentry->d_sb);
mutex_lock(&nfsd_mutex);
nn = net_generic(netns(file), nfsd_net_id);
- if (nn->nfsd_net_up)
+ if (test_bit(NFSD_NET_UP, &nn->flags))
nfsd4_revoke_states(nn, path.dentry->d_sb);
else
error = -EINVAL;
@@ -1150,7 +1150,7 @@ static ssize_t write_v4_end_grace(struct file *file, char *buf, size_t size)
}
return scnprintf(buf, SIMPLE_TRANSACTION_LIMIT, "%c\n",
- nn->grace_ended ? 'Y' : 'N');
+ test_bit(NFSD_NET_GRACE_ENDED, &nn->flags) ? 'Y' : 'N');
}
#endif
diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 421f474b75ce..1927fbf26d56 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -381,7 +381,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
int ret;
- if (nn->nfsd_net_up)
+ if (test_bit(NFSD_NET_UP, &nn->flags))
return 0;
ret = nfsd_startup_generic();
@@ -391,11 +391,11 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
if (ret)
goto out_socks;
- if (nfsd_needs_lockd(nn) && !nn->lockd_up) {
+ if (nfsd_needs_lockd(nn) && !test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) {
ret = lockd_up(net, cred);
if (ret)
goto out_socks;
- nn->lockd_up = true;
+ set_bit(NFSD_NET_LOCKD_UP, &nn->flags);
}
ret = nfsd_file_cache_start_net(net);
@@ -413,7 +413,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
if (ret)
goto out_reply_cache;
- nn->nfsd_net_up = true;
+ set_bit(NFSD_NET_UP, &nn->flags);
return 0;
out_reply_cache:
@@ -421,9 +421,9 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
out_filecache:
nfsd_file_cache_shutdown_net(net);
out_lockd:
- if (nn->lockd_up) {
+ if (test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) {
lockd_down(net);
- nn->lockd_up = false;
+ clear_bit(NFSD_NET_LOCKD_UP, &nn->flags);
}
out_socks:
nfsd_shutdown_generic();
@@ -434,7 +434,7 @@ static void nfsd_shutdown_net(struct net *net)
{
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- if (nn->nfsd_net_up) {
+ if (test_bit(NFSD_NET_UP, &nn->flags)) {
percpu_ref_kill_and_confirm(&nn->nfsd_net_ref, nfsd_net_done);
wait_for_completion(&nn->nfsd_net_confirm_done);
@@ -442,18 +442,18 @@ static void nfsd_shutdown_net(struct net *net)
nfs4_state_shutdown_net(net);
nfsd_reply_cache_shutdown(nn);
nfsd_file_cache_shutdown_net(net);
- if (nn->lockd_up) {
+ if (test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) {
lockd_down(net);
- nn->lockd_up = false;
+ clear_bit(NFSD_NET_LOCKD_UP, &nn->flags);
}
wait_for_completion(&nn->nfsd_net_free_done);
}
percpu_ref_exit(&nn->nfsd_net_ref);
- if (nn->nfsd_net_up)
+ if (test_bit(NFSD_NET_UP, &nn->flags))
nfsd_shutdown_generic();
- nn->nfsd_net_up = false;
+ clear_bit(NFSD_NET_UP, &nn->flags);
}
static DEFINE_SPINLOCK(nfsd_notifier_lock);
@@ -974,6 +974,20 @@ nfsd(void *vrqstp)
return 0;
}
+/*
+ * Set rq_status_counter back to an even value, indicating that the rqstp
+ * fields are no longer meaningful to a lockless reader. This pairs with the
+ * odd-valued store made once the request has been decoded, and must run on
+ * every return path that follows it so that the seq-lock like protocol used
+ * by nfsd_nl_rpc_status_get_dumpit() is not left permanently odd. The store
+ * also advances the counter so a concurrent reader detects the transition.
+ */
+static void nfsd_status_counter_set_idle(struct svc_rqst *rqstp)
+{
+ smp_store_release(&rqstp->rq_status_counter,
+ (rqstp->rq_status_counter | 1) + 1);
+}
+
/**
* nfsd_dispatch - Process an NFS or NFSACL or LOCALIO Request
* @rqstp: incoming request
@@ -1035,14 +1049,9 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
if (!proc->pc_encode(rqstp, &rqstp->rq_res_stream))
goto out_encode_err;
- /*
- * Release rq_status_counter setting it to an even value after the rpc
- * request has been properly processed.
- */
- smp_store_release(&rqstp->rq_status_counter, rqstp->rq_status_counter + 1);
-
nfsd_cache_update(rqstp, rp, rqstp->rq_cachetype, nfs_reply);
out_cached_reply:
+ nfsd_status_counter_set_idle(rqstp);
return 1;
out_decode_err:
@@ -1053,12 +1062,14 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
out_update_drop:
nfsd_cache_update(rqstp, rp, RC_NOCACHE, NULL);
out_dropit:
+ nfsd_status_counter_set_idle(rqstp);
return 0;
out_encode_err:
trace_nfsd_cant_encode_err(rqstp);
nfsd_cache_update(rqstp, rp, RC_NOCACHE, NULL);
*statp = rpc_system_err;
+ nfsd_status_counter_set_idle(rqstp);
return 1;
}
diff --git a/fs/ntfs3/bitmap.c b/fs/ntfs3/bitmap.c
index 29b585f443f3..65d05e6a0566 100644
--- a/fs/ntfs3/bitmap.c
+++ b/fs/ntfs3/bitmap.c
@@ -710,20 +710,17 @@ int wnd_set_free(struct wnd_bitmap *wnd, size_t bit, size_t bits)
{
int err = 0;
struct super_block *sb = wnd->sb;
- size_t bits0 = bits;
u32 wbits = 8 * sb->s_blocksize;
size_t iw = bit >> (sb->s_blocksize_bits + 3);
u32 wbit = bit & (wbits - 1);
struct buffer_head *bh;
+ u32 op;
- while (iw < wnd->nwnd && bits) {
- u32 tail, op;
-
+ for (; iw < wnd->nwnd && bits; iw++, bit += op, bits -= op, wbit = 0) {
if (iw + 1 == wnd->nwnd)
wbits = wnd->bits_last;
- tail = wbits - wbit;
- op = min_t(u32, tail, bits);
+ op = min_t(u32, wbits - wbit, bits);
bh = wnd_map(wnd, iw);
if (IS_ERR(bh)) {
@@ -736,20 +733,15 @@ int wnd_set_free(struct wnd_bitmap *wnd, size_t bit, size_t bits)
ntfs_bitmap_clear_le(bh->b_data, wbit, op);
wnd->free_bits[iw] += op;
+ wnd->total_zeroes += op;
set_buffer_uptodate(bh);
mark_buffer_dirty(bh);
unlock_buffer(bh);
put_bh(bh);
- wnd->total_zeroes += op;
- bits -= op;
- wbit = 0;
- iw += 1;
+ wnd_add_free_ext(wnd, bit, op, false);
}
-
- wnd_add_free_ext(wnd, bit, bits0, false);
-
return err;
}
@@ -760,20 +752,17 @@ int wnd_set_used(struct wnd_bitmap *wnd, size_t bit, size_t bits)
{
int err = 0;
struct super_block *sb = wnd->sb;
- size_t bits0 = bits;
size_t iw = bit >> (sb->s_blocksize_bits + 3);
u32 wbits = 8 * sb->s_blocksize;
u32 wbit = bit & (wbits - 1);
struct buffer_head *bh;
+ u32 op;
- while (iw < wnd->nwnd && bits) {
- u32 tail, op;
-
+ for (; iw < wnd->nwnd && bits; iw++, bit += op, bits -= op, wbit = 0) {
if (unlikely(iw + 1 == wnd->nwnd))
wbits = wnd->bits_last;
- tail = wbits - wbit;
- op = min_t(u32, tail, bits);
+ op = min_t(u32, wbits - wbit, bits);
bh = wnd_map(wnd, iw);
if (IS_ERR(bh)) {
@@ -785,21 +774,16 @@ int wnd_set_used(struct wnd_bitmap *wnd, size_t bit, size_t bits)
ntfs_bitmap_set_le(bh->b_data, wbit, op);
wnd->free_bits[iw] -= op;
+ wnd->total_zeroes -= op;
set_buffer_uptodate(bh);
mark_buffer_dirty(bh);
unlock_buffer(bh);
put_bh(bh);
- wnd->total_zeroes -= op;
- bits -= op;
- wbit = 0;
- iw += 1;
+ if (!RB_EMPTY_ROOT(&wnd->start_tree))
+ wnd_remove_free_ext(wnd, bit, op);
}
-
- if (!RB_EMPTY_ROOT(&wnd->start_tree))
- wnd_remove_free_ext(wnd, bit, bits0);
-
return err;
}
@@ -852,15 +836,13 @@ static bool wnd_is_free_hlp(struct wnd_bitmap *wnd, size_t bit, size_t bits)
size_t iw = bit >> (sb->s_blocksize_bits + 3);
u32 wbits = 8 * sb->s_blocksize;
u32 wbit = bit & (wbits - 1);
+ u32 op;
- while (iw < wnd->nwnd && bits) {
- u32 tail, op;
-
+ for (; iw < wnd->nwnd && bits; iw++, bits -= op, wbit = 0) {
if (unlikely(iw + 1 == wnd->nwnd))
wbits = wnd->bits_last;
- tail = wbits - wbit;
- op = min_t(u32, tail, bits);
+ op = min_t(u32, wbits - wbit, bits);
if (wbits != wnd->free_bits[iw]) {
bool ret;
@@ -875,10 +857,6 @@ static bool wnd_is_free_hlp(struct wnd_bitmap *wnd, size_t bit, size_t bits)
if (!ret)
return false;
}
-
- bits -= op;
- wbit = 0;
- iw += 1;
}
return true;
@@ -928,6 +906,7 @@ bool wnd_is_used(struct wnd_bitmap *wnd, size_t bit, size_t bits)
size_t iw = bit >> (sb->s_blocksize_bits + 3);
u32 wbits = 8 * sb->s_blocksize;
u32 wbit = bit & (wbits - 1);
+ u32 op;
size_t end;
struct rb_node *n;
struct e_node *e;
@@ -945,14 +924,11 @@ bool wnd_is_used(struct wnd_bitmap *wnd, size_t bit, size_t bits)
return false;
use_wnd:
- while (iw < wnd->nwnd && bits) {
- u32 tail, op;
-
+ for (; iw < wnd->nwnd && bits; iw++, bits -= op, wbit = 0) {
if (unlikely(iw + 1 == wnd->nwnd))
wbits = wnd->bits_last;
- tail = wbits - wbit;
- op = min_t(u32, tail, bits);
+ op = min_t(u32, wbits - wbit, bits);
if (wnd->free_bits[iw]) {
bool ret;
@@ -966,10 +942,6 @@ bool wnd_is_used(struct wnd_bitmap *wnd, size_t bit, size_t bits)
if (!ret)
goto out;
}
-
- bits -= op;
- wbit = 0;
- iw += 1;
}
ret = true;
diff --git a/fs/ntfs3/frecord.c b/fs/ntfs3/frecord.c
index 45e8d73d38f8..ab8615abb981 100644
--- a/fs/ntfs3/frecord.c
+++ b/fs/ntfs3/frecord.c
@@ -1977,7 +1977,6 @@ int ni_fiemap(struct ntfs_inode *ni, struct fiemap_extent_info *fieinfo,
if (end > alloc_size)
end = alloc_size;
-
while (vbo < end) {
if (idx == -1) {
ok = run_lookup_entry(&run, vcn, &lcn, &clen, &idx);
diff --git a/fs/ntfs3/fsntfs.c b/fs/ntfs3/fsntfs.c
index 170836749de6..c9ff8293fea7 100644
--- a/fs/ntfs3/fsntfs.c
+++ b/fs/ntfs3/fsntfs.c
@@ -2716,4 +2716,4 @@ int ntfs_set_label(struct ntfs_sb_info *sbi, u8 *label, int len)
out:
kfree(uni);
return err;
-}
\ No newline at end of file
+}
diff --git a/fs/ntfs3/run.c b/fs/ntfs3/run.c
index 12f93f5a3d9b..66baaada02e9 100644
--- a/fs/ntfs3/run.c
+++ b/fs/ntfs3/run.c
@@ -1133,14 +1133,17 @@ int run_unpack_ex(struct runs_tree *run, struct ntfs_sb_info *sbi, CLST ino,
err = wnd_set_used_safe(wnd, lcn, len, &done);
if (zone) {
/* Restore zone. Lock mft run. */
- struct rw_semaphore *lock;
- lock = is_mounted(sbi) ? &sbi->mft.ni->file.run_lock :
- NULL;
- if (lock)
- down_read(lock);
- ntfs_refresh_zone(sbi);
- if (lock)
- up_read(lock);
+ struct rw_semaphore *lock =
+ is_mounted(sbi) ? &sbi->mft.ni->file.run_lock :
+ NULL;
+ if (lock) {
+ if (down_read_trylock(lock)) {
+ ntfs_refresh_zone(sbi);
+ up_read(lock);
+ }
+ } else {
+ ntfs_refresh_zone(sbi);
+ }
}
up_write(&wnd->rw_lock);
if (err)
diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c
index 6170e202e56e..9a21b720eeff 100644
--- a/fs/ocfs2/cluster/heartbeat.c
+++ b/fs/ocfs2/cluster/heartbeat.c
@@ -14,6 +14,7 @@
#include <linux/file.h>
#include <linux/kthread.h>
#include <linux/configfs.h>
+#include <linux/mutex.h>
#include <linux/random.h>
#include <linux/crc32.h>
#include <linux/time.h>
@@ -265,6 +266,9 @@ struct o2hb_region {
/* Message key for negotiate timeout message. */
unsigned int hr_key;
struct list_head hr_handler_list;
+ /* Serializes timeout arming against failed-start and teardown. */
+ struct mutex hr_arming_mutex;
+ bool hr_stopping;
/* last hb status, 0 for success, other value for error. */
int hr_last_hb_status;
@@ -329,9 +333,14 @@ static void o2hb_write_timeout(struct work_struct *work)
static void o2hb_arm_timeout(struct o2hb_region *reg)
{
+ mutex_lock(®->hr_arming_mutex);
+
+ if (reg->hr_stopping)
+ goto out_unlock;
+
/* Arm writeout only after thread reaches steady state */
if (atomic_read(®->hr_steady_iterations) != 0)
- return;
+ goto out_unlock;
mlog(ML_HEARTBEAT, "Queue write timeout for %u ms\n",
O2HB_MAX_WRITE_TIMEOUT_MS);
@@ -350,6 +359,18 @@ static void o2hb_arm_timeout(struct o2hb_region *reg)
schedule_delayed_work(®->hr_nego_timeout_work,
msecs_to_jiffies(O2HB_NEGO_TIMEOUT_MS));
bitmap_zero(reg->hr_nego_node_bitmap, O2NM_MAX_NODES);
+
+out_unlock:
+ mutex_unlock(®->hr_arming_mutex);
+}
+
+static void o2hb_queue_nego_timeout(struct o2hb_region *reg,
+ unsigned long delay)
+{
+ mutex_lock(®->hr_arming_mutex);
+ if (!reg->hr_stopping)
+ schedule_delayed_work(®->hr_nego_timeout_work, delay);
+ mutex_unlock(®->hr_arming_mutex);
}
static void o2hb_disarm_timeout(struct o2hb_region *reg)
@@ -358,6 +379,19 @@ static void o2hb_disarm_timeout(struct o2hb_region *reg)
cancel_delayed_work_sync(®->hr_nego_timeout_work);
}
+static void o2hb_set_region_stopping(struct o2hb_region *reg, bool stopping)
+{
+ mutex_lock(®->hr_arming_mutex);
+ reg->hr_stopping = stopping;
+ mutex_unlock(®->hr_arming_mutex);
+}
+
+static void o2hb_quiesce_timeout(struct o2hb_region *reg)
+{
+ o2hb_set_region_stopping(reg, true);
+ o2hb_disarm_timeout(reg);
+}
+
static int o2hb_send_nego_msg(int key, int type, u8 target, u8 node_num)
{
struct o2hb_nego_msg msg;
@@ -407,8 +441,7 @@ static void o2hb_nego_timeout(struct work_struct *work)
/* check negotiate bitmap every second to do timeout
* approve decision.
*/
- schedule_delayed_work(®->hr_nego_timeout_work,
- msecs_to_jiffies(1000));
+ o2hb_queue_nego_timeout(reg, msecs_to_jiffies(1000));
return;
}
@@ -1531,33 +1564,47 @@ static struct o2hb_region *to_o2hb_region(struct config_item *item)
return item ? container_of(item, struct o2hb_region, hr_item) : NULL;
}
-/* drop_item only drops its ref after killing the thread, nothing should
- * be using the region anymore. this has to clean up any state that
- * attributes might have built up. */
-static void o2hb_region_release(struct config_item *item)
+static void o2hb_unmap_slot_data(struct o2hb_region *reg)
{
int i;
struct page *page;
- struct o2hb_region *reg = to_o2hb_region(item);
-
- mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg_bdev(reg));
-
- kfree(reg->hr_tmp_block);
if (reg->hr_slot_data) {
for (i = 0; i < reg->hr_num_pages; i++) {
page = reg->hr_slot_data[i];
- if (page)
+ if (page) {
__free_page(page);
+ reg->hr_slot_data[i] = NULL;
+ }
}
kfree(reg->hr_slot_data);
+ reg->hr_slot_data = NULL;
}
+ kfree(reg->hr_slots);
+ reg->hr_slots = NULL;
+
+ kfree(reg->hr_tmp_block);
+ reg->hr_tmp_block = NULL;
+}
+
+/* drop_item only drops its ref after killing the thread, nothing should
+ * be using the region anymore. this has to clean up any state that
+ * attributes might have built up.
+ */
+static void o2hb_region_release(struct config_item *item)
+{
+ struct o2hb_region *reg = to_o2hb_region(item);
+
+ mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg_bdev(reg));
+
+ o2hb_quiesce_timeout(reg);
+ o2net_unregister_and_flush_handler_list(®->hr_handler_list);
+ o2hb_unmap_slot_data(reg);
+
if (reg->hr_bdev_file)
fput(reg->hr_bdev_file);
- kfree(reg->hr_slots);
-
debugfs_remove_recursive(reg->hr_debug_dir);
kfree(reg->hr_db_livenodes);
kfree(reg->hr_db_regnum);
@@ -1568,7 +1615,6 @@ static void o2hb_region_release(struct config_item *item)
list_del(®->hr_all_item);
spin_unlock(&o2hb_live_lock);
- o2net_unregister_handler_list(®->hr_handler_list);
kfree(reg);
}
@@ -1708,6 +1754,7 @@ static void o2hb_init_region_params(struct o2hb_region *reg)
static int o2hb_map_slot_data(struct o2hb_region *reg)
{
int i, j;
+ int ret = -ENOMEM;
unsigned int last_slot;
unsigned int spp = reg->hr_slots_per_page;
struct page *page;
@@ -1715,15 +1762,15 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
struct o2hb_disk_slot *slot;
reg->hr_tmp_block = kmalloc(reg->hr_block_bytes, GFP_KERNEL);
- if (reg->hr_tmp_block == NULL)
- return -ENOMEM;
+ if (!reg->hr_tmp_block)
+ goto out;
reg->hr_slots = kcalloc(reg->hr_blocks,
sizeof(struct o2hb_disk_slot), GFP_KERNEL);
- if (reg->hr_slots == NULL)
- return -ENOMEM;
+ if (!reg->hr_slots)
+ goto out;
- for(i = 0; i < reg->hr_blocks; i++) {
+ for (i = 0; i < reg->hr_blocks; i++) {
slot = ®->hr_slots[i];
slot->ds_node_num = i;
INIT_LIST_HEAD(&slot->ds_live_item);
@@ -1738,12 +1785,12 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
reg->hr_slot_data = kcalloc(reg->hr_num_pages, sizeof(struct page *),
GFP_KERNEL);
if (!reg->hr_slot_data)
- return -ENOMEM;
+ goto out;
- for(i = 0; i < reg->hr_num_pages; i++) {
+ for (i = 0; i < reg->hr_num_pages; i++) {
page = alloc_page(GFP_KERNEL);
if (!page)
- return -ENOMEM;
+ goto out;
reg->hr_slot_data[i] = page;
@@ -1763,6 +1810,10 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
}
return 0;
+
+out:
+ o2hb_unmap_slot_data(reg);
+ return ret;
}
/* Read in all the slots available and populate the tracking
@@ -1854,9 +1905,11 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
"blocksize %u incorrect for device, expected %d",
reg->hr_block_bytes, sectsize);
ret = -EINVAL;
- goto out3;
+ goto out;
}
+ reg->hr_aborted_start = 0;
+ reg->hr_node_deleted = 0;
o2hb_init_region_params(reg);
/* Generation of zero is invalid */
@@ -1868,18 +1921,15 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
ret = o2hb_map_slot_data(reg);
if (ret) {
mlog_errno(ret);
- goto out3;
+ goto out;
}
ret = o2hb_populate_slot_data(reg);
if (ret) {
mlog_errno(ret);
- goto out3;
+ goto out;
}
- INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout);
- INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout);
-
/*
* A node is considered live after it has beat LIVE_THRESHOLD
* times. We're not steady until we've given them a chance
@@ -1899,13 +1949,14 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
atomic_set(®->hr_steady_iterations, live_threshold);
/* unsteady_iterations is triple the steady_iterations */
atomic_set(®->hr_unsteady_iterations, (live_threshold * 3));
+ o2hb_set_region_stopping(reg, false);
hb_task = kthread_run(o2hb_thread, reg, "o2hb-%s",
reg->hr_item.ci_name);
if (IS_ERR(hb_task)) {
ret = PTR_ERR(hb_task);
mlog_errno(ret);
- goto out3;
+ goto out;
}
spin_lock(&o2hb_live_lock);
@@ -1922,12 +1973,12 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
if (reg->hr_aborted_start) {
ret = -EIO;
- goto out3;
+ goto out;
}
if (reg->hr_node_deleted) {
ret = -EINVAL;
- goto out3;
+ goto out;
}
/* Ok, we were woken. Make sure it wasn't by drop_item() */
@@ -1946,8 +1997,20 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
printk(KERN_NOTICE "o2hb: Heartbeat started on region %s (%pg)\n",
config_item_name(®->hr_item), reg_bdev(reg));
-out3:
+out:
if (ret < 0) {
+ o2hb_quiesce_timeout(reg);
+
+ spin_lock(&o2hb_live_lock);
+ hb_task = reg->hr_task;
+ reg->hr_task = NULL;
+ spin_unlock(&o2hb_live_lock);
+
+ if (hb_task)
+ kthread_stop(hb_task);
+
+ o2hb_unmap_slot_data(reg);
+
fput(reg->hr_bdev_file);
reg->hr_bdev_file = NULL;
}
@@ -2079,6 +2142,10 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g
*/
reg->hr_key = crc32_le(reg->hr_region_num + O2NM_MAX_REGIONS,
name, strlen(name));
+ mutex_init(®->hr_arming_mutex);
+ reg->hr_stopping = true;
+ INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout);
+ INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout);
INIT_LIST_HEAD(®->hr_handler_list);
ret = o2net_register_handler(O2HB_NEGO_TIMEOUT_MSG, reg->hr_key,
sizeof(struct o2hb_nego_msg),
@@ -2099,7 +2166,7 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g
return ®->hr_item;
unregister_handler:
- o2net_unregister_handler_list(®->hr_handler_list);
+ o2net_unregister_and_flush_handler_list(®->hr_handler_list);
remove_item:
spin_lock(&o2hb_live_lock);
list_del(®->hr_all_item);
@@ -2118,6 +2185,8 @@ static void o2hb_heartbeat_group_drop_item(struct config_group *group,
struct o2hb_region *reg = to_o2hb_region(item);
int quorum_region = 0;
+ o2hb_quiesce_timeout(reg);
+
/* stop the thread when the user removes the region dir */
spin_lock(&o2hb_live_lock);
hb_task = reg->hr_task;
diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c
index 15b726a3a1fc..dd4fb83d9cd8 100644
--- a/fs/ocfs2/cluster/tcp.c
+++ b/fs/ocfs2/cluster/tcp.c
@@ -38,6 +38,8 @@
*/
#include <linux/kernel.h>
+#include <linux/completion.h>
+#include <linux/mutex.h>
#include <linux/sched/mm.h>
#include <linux/jiffies.h>
#include <linux/slab.h>
@@ -101,10 +103,15 @@ static struct socket *o2net_listen_sock;
* o2net_wq. teardown detaches the callbacks before destroying the workqueue.
* quorum work is queued as sock containers are shutdown.. stop_listening
* tears down all the node's sock containers, preventing future shutdowns
- * and queued quroum work, before canceling delayed quorum work and
- * destroying the work queue.
+ * and queued quorum work, before canceling delayed quorum work and
+ * destroying the work queue. Handler teardown can also race local listener
+ * shutdown, so keep a waitable destroying pointer until the old ordered
+ * queue has finished draining.
*/
static struct workqueue_struct *o2net_wq;
+static struct workqueue_struct *o2net_wq_destroying;
+static DEFINE_MUTEX(o2net_wq_mutex);
+static DECLARE_COMPLETION(o2net_wq_destroyed);
/* Heartbeat callbacks stay registered across local-node off/on. */
static bool o2net_listening;
static struct work_struct o2net_listen_work;
@@ -886,6 +893,27 @@ void o2net_unregister_handler_list(struct list_head *list)
}
EXPORT_SYMBOL_GPL(o2net_unregister_handler_list);
+static void o2net_flush_wq(void)
+{
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ wait_for_completion(&o2net_wq_destroyed);
+ return;
+ }
+
+ if (o2net_wq)
+ flush_workqueue(o2net_wq);
+ mutex_unlock(&o2net_wq_mutex);
+}
+
+void o2net_unregister_and_flush_handler_list(struct list_head *list)
+{
+ o2net_unregister_handler_list(list);
+ o2net_flush_wq();
+}
+EXPORT_SYMBOL_GPL(o2net_unregister_and_flush_handler_list);
+
static struct o2net_msg_handler *o2net_handler_get(u32 msg_type, u32 key)
{
struct o2net_msg_handler *nmh;
@@ -1716,12 +1744,10 @@ void o2net_disconnect_node(struct o2nm_node *node)
o2net_set_nn_state(nn, NULL, 0, -ENOTCONN);
spin_unlock(&nn->nn_lock);
- if (o2net_wq) {
- cancel_delayed_work(&nn->nn_connect_expired);
- cancel_delayed_work(&nn->nn_connect_work);
- cancel_delayed_work(&nn->nn_still_up);
- flush_workqueue(o2net_wq);
- }
+ cancel_delayed_work(&nn->nn_connect_expired);
+ cancel_delayed_work(&nn->nn_connect_work);
+ cancel_delayed_work(&nn->nn_still_up);
+ o2net_flush_wq();
}
static void o2net_hb_node_down_cb(struct o2nm_node *node, int node_num,
@@ -2066,6 +2092,36 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port)
return ret;
}
+static void o2net_destroy_wq(void)
+{
+ struct workqueue_struct *wq;
+
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ wait_for_completion(&o2net_wq_destroyed);
+ return;
+ }
+
+ wq = o2net_wq;
+ if (!wq) {
+ mutex_unlock(&o2net_wq_mutex);
+ return;
+ }
+
+ reinit_completion(&o2net_wq_destroyed);
+ o2net_wq_destroying = wq;
+ mutex_unlock(&o2net_wq_mutex);
+
+ destroy_workqueue(wq);
+
+ mutex_lock(&o2net_wq_mutex);
+ o2net_wq = NULL;
+ o2net_wq_destroying = NULL;
+ complete_all(&o2net_wq_destroyed);
+ mutex_unlock(&o2net_wq_mutex);
+}
+
/*
* called from node manager when we should bring up our network listening
* socket. node manager handles all the serialization to only call this
@@ -2076,24 +2132,44 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port)
int o2net_start_listening(struct o2nm_node *node)
{
int ret = 0;
+ struct workqueue_struct *wq;
if (WARN_ON_ONCE(READ_ONCE(o2net_listening)))
return -EBUSY;
- BUG_ON(o2net_wq != NULL);
+
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ return -EBUSY;
+ }
+ if (WARN_ON_ONCE(o2net_wq)) {
+ mutex_unlock(&o2net_wq_mutex);
+ return -EBUSY;
+ }
+ mutex_unlock(&o2net_wq_mutex);
+
BUG_ON(o2net_listen_sock != NULL);
mlog(ML_KTHREAD, "starting o2net thread...\n");
- o2net_wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM);
- if (o2net_wq == NULL) {
+ wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM);
+ if (!wq) {
mlog(ML_ERROR, "unable to launch o2net thread\n");
return -ENOMEM; /* ? */
}
+ mutex_lock(&o2net_wq_mutex);
+ if (unlikely(o2net_wq_destroying || o2net_wq)) {
+ mutex_unlock(&o2net_wq_mutex);
+ destroy_workqueue(wq);
+ return -EBUSY;
+ }
+ o2net_wq = wq;
+ mutex_unlock(&o2net_wq_mutex);
+
ret = o2net_open_listening_sock(node->nd_ipv4_address,
node->nd_ipv4_port);
if (ret) {
- destroy_workqueue(o2net_wq);
- o2net_wq = NULL;
+ o2net_destroy_wq();
} else
o2quo_conn_up(node->nd_num);
@@ -2129,8 +2205,7 @@ void o2net_stop_listening(struct o2nm_node *node)
/* finish all work and tear down the work queue */
mlog(ML_KTHREAD, "waiting for o2net thread to exit....\n");
- destroy_workqueue(o2net_wq);
- o2net_wq = NULL;
+ o2net_destroy_wq();
sock_release(o2net_listen_sock);
o2net_listen_sock = NULL;
diff --git a/fs/ocfs2/cluster/tcp.h b/fs/ocfs2/cluster/tcp.h
index 2e86d42b5faf..a11bcee28947 100644
--- a/fs/ocfs2/cluster/tcp.h
+++ b/fs/ocfs2/cluster/tcp.h
@@ -89,6 +89,7 @@ int o2net_register_handler(u32 msg_type, u32 key, u32 max_len,
o2net_post_msg_handler_func *post_func,
struct list_head *unreg_list);
void o2net_unregister_handler_list(struct list_head *list);
+void o2net_unregister_and_flush_handler_list(struct list_head *list);
void o2net_fill_node_map(unsigned long *map, unsigned bytes);
diff --git a/fs/ocfs2/inode.c b/fs/ocfs2/inode.c
index 9a56c57f9ded..ce78ef998217 100644
--- a/fs/ocfs2/inode.c
+++ b/fs/ocfs2/inode.c
@@ -233,6 +233,22 @@ struct inode *ocfs2_iget(struct ocfs2_super *osb, u64 blkno, unsigned flags,
return inode;
}
+static int ocfs2_dinode_has_extents(struct ocfs2_dinode *di)
+{
+ /* inodes flagged with other stuff in id2 */
+ if (le32_to_cpu(di->i_flags) &
+ (OCFS2_SUPER_BLOCK_FL | OCFS2_LOCAL_ALLOC_FL | OCFS2_CHAIN_FL |
+ OCFS2_DEALLOC_FL))
+ return 0;
+ /* i_flags doesn't indicate when id2 is a fast symlink */
+ if (S_ISLNK(le16_to_cpu(di->i_mode)) && le64_to_cpu(di->i_size) &&
+ !le32_to_cpu(di->i_clusters))
+ return 0;
+ if (le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_DATA_FL)
+ return 0;
+
+ return 1;
+}
/*
* here's how inodes get read from disk:
@@ -1452,6 +1468,22 @@ int ocfs2_validate_inode_block(struct super_block *sb,
goto bail;
}
+ if ((le32_to_cpu(di->i_flags) & OCFS2_DIO_ORPHANED_FL) &&
+ le16_to_cpu(di->i_dio_orphaned_slot) >= OCFS2_SB(sb)->max_slots) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: DIO orphaned slot %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->i_dio_orphaned_slot));
+ goto bail;
+ }
+
+ if ((le32_to_cpu(di->i_flags) & OCFS2_ORPHANED_FL) &&
+ le16_to_cpu(di->i_orphaned_slot) >= OCFS2_SB(sb)->max_slots) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: orphaned slot %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->i_orphaned_slot));
+ goto bail;
+ }
+
/*
* Reject dinodes whose i_mode does not name one of the seven
* canonical POSIX file types. ocfs2_populate_inode() copies
@@ -1532,6 +1564,10 @@ int ocfs2_validate_inode_block(struct super_block *sb,
goto bail;
}
+ rc = ocfs2_validate_inode_xattr(sb, bh->b_blocknr, di);
+ if (rc)
+ goto bail;
+
if (le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_DATA_FL) {
struct ocfs2_inline_data *data = &di->id2.i_data;
@@ -1563,6 +1599,93 @@ int ocfs2_validate_inode_block(struct super_block *sb,
}
}
+ if (le16_to_cpu(di->i_suballoc_slot) != (u16)OCFS2_INVALID_SLOT &&
+ (u32)le16_to_cpu(di->i_suballoc_slot) > OCFS2_SB(sb)->max_slots - 1) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: suballoc slot %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->i_suballoc_slot));
+ goto bail;
+ }
+
+ if (S_ISLNK(le16_to_cpu(di->i_mode)) &&
+ !le32_to_cpu(di->i_clusters)) {
+ int max_inline = ocfs2_fast_symlink_chars(sb);
+ u64 i_size = le64_to_cpu(di->i_size);
+
+ if (i_size >= max_inline) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode #%llu: fast symlink i_size %llu exceeds max %d\n",
+ (unsigned long long)bh->b_blocknr,
+ (unsigned long long)i_size,
+ max_inline - 1);
+ goto bail;
+ }
+
+ if (strnlen((char *)di->id2.i_symlink, i_size + 1) != i_size) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode #%llu: fast symlink is not NUL-terminated at i_size %llu\n",
+ (unsigned long long)bh->b_blocknr,
+ (unsigned long long)i_size);
+ goto bail;
+ }
+ }
+
+ if (le32_to_cpu(di->i_flags) & OCFS2_CHAIN_FL) {
+ struct ocfs2_chain_list *cl = &di->id2.i_chain;
+
+ if (le16_to_cpu(cl->cl_count) != ocfs2_chain_recs_per_inode(sb)) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: chain list count %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(cl->cl_count));
+ goto bail;
+ }
+ if (le16_to_cpu(cl->cl_next_free_rec) > le16_to_cpu(cl->cl_count)) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: chain list index %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(cl->cl_next_free_rec));
+ goto bail;
+ }
+ }
+
+ if ((le16_to_cpu(di->i_dyn_features) & OCFS2_HAS_REFCOUNT_FL) &&
+ !di->i_refcount_loc) {
+ rc = ocfs2_error(sb, "Inode #%llu has refcount flag but no i_refcount_loc\n",
+ (unsigned long long)bh->b_blocknr);
+ goto bail;
+ }
+
+ if (ocfs2_dinode_has_extents(di)) {
+ struct ocfs2_extent_list *el = &di->id2.i_list;
+ u16 count = le16_to_cpu(el->l_count);
+ u16 next_free = le16_to_cpu(el->l_next_free_rec);
+
+ if (count == 0) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode %llu: extent list l_count is zero\n",
+ (unsigned long long)bh->b_blocknr);
+ goto bail;
+ }
+ /*
+ * The exact capacity depends on i_xattr_inline_size, another
+ * unvalidated on-disk field. Inline xattrs only shrink the
+ * list, so the no-xattr maximum is a safe upper bound that a
+ * valid l_count never exceeds.
+ */
+ if (count > ocfs2_extent_recs_per_inode(sb)) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode %llu: extent list l_count %u exceeds max %u\n",
+ (unsigned long long)bh->b_blocknr, count,
+ ocfs2_extent_recs_per_inode(sb));
+ goto bail;
+ }
+ if (next_free > count) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode %llu: extent list l_next_free_rec %u exceeds l_count %u\n",
+ (unsigned long long)bh->b_blocknr, next_free, count);
+ goto bail;
+ }
+ }
+
rc = 0;
bail:
@@ -1728,6 +1851,16 @@ static int ocfs2_filecheck_repair_inode_block(struct super_block *sb,
le32_to_cpu(di->i_fs_generation));
}
+ if (ocfs2_dinode_has_extents(di) &&
+ le16_to_cpu(di->id2.i_list.l_next_free_rec) > le16_to_cpu(di->id2.i_list.l_count)) {
+ di->id2.i_list.l_next_free_rec = di->id2.i_list.l_count;
+ changed = 1;
+ mlog(ML_ERROR,
+ "Filecheck: reset dinode #%llu: l_next_free_rec to %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->id2.i_list.l_next_free_rec));
+ }
+
if (changed || ocfs2_validate_meta_ecc(sb, bh->b_data, &di->i_check)) {
ocfs2_compute_meta_ecc(sb, bh->b_data, &di->i_check);
mark_buffer_dirty(bh);
diff --git a/fs/ocfs2/xattr.c b/fs/ocfs2/xattr.c
index 3cc679e5f1a1..5c88f6f80085 100644
--- a/fs/ocfs2/xattr.c
+++ b/fs/ocfs2/xattr.c
@@ -386,6 +386,12 @@ static int ocfs2_init_xattr_bucket(struct ocfs2_xattr_bucket *bucket,
return rc;
}
+static int ocfs2_validate_xattr_entries_flat(struct super_block *sb, u64 blkno,
+ struct ocfs2_xattr_header *xh,
+ size_t region_size);
+static int ocfs2_validate_xattr_bucket(struct ocfs2_xattr_bucket *bucket,
+ u64 blkno);
+
/* Read the xattr bucket at xb_blkno */
static int ocfs2_read_xattr_bucket(struct ocfs2_xattr_bucket *bucket,
u64 xb_blkno)
@@ -404,6 +410,8 @@ static int ocfs2_read_xattr_bucket(struct ocfs2_xattr_bucket *bucket,
spin_unlock(&OCFS2_SB(bucket->bu_inode->i_sb)->osb_xattr_lock);
if (rc)
mlog_errno(rc);
+ else
+ rc = ocfs2_validate_xattr_bucket(bucket, xb_blkno);
}
if (rc)
@@ -505,6 +513,22 @@ static int ocfs2_validate_xattr_block(struct super_block *sb,
le32_to_cpu(xb->xb_fs_generation));
}
+ if (!(le16_to_cpu(xb->xb_flags) & OCFS2_XATTR_INDEXED)) {
+ size_t region_offset =
+ offsetof(struct ocfs2_xattr_block, xb_attrs.xb_header);
+
+ if (bh->b_size < region_offset)
+ return ocfs2_error(sb,
+ "Invalid xattr block %llu: block size %zu is too small\n",
+ (unsigned long long)bh->b_blocknr,
+ bh->b_size);
+
+ return ocfs2_validate_xattr_entries_flat(sb, bh->b_blocknr,
+ &xb->xb_attrs.xb_header,
+ bh->b_size -
+ region_offset);
+ }
+
return 0;
}
@@ -953,6 +977,197 @@ static int ocfs2_xattr_list_entries(struct inode *inode,
return result;
}
+static int ocfs2_validate_xattr_entries_flat(struct super_block *sb, u64 blkno,
+ struct ocfs2_xattr_header *xh,
+ size_t region_size)
+{
+ u16 xattr_count = le16_to_cpu(xh->xh_count);
+ size_t entries_limit = region_size;
+ size_t nv_limit = region_size;
+ size_t max_entries;
+ int i;
+
+ if (region_size < sizeof(*xh))
+ return ocfs2_error(sb,
+ "Invalid xattr in block %llu: region size %zu is too small\n",
+ (unsigned long long)blkno, region_size);
+
+ max_entries = (entries_limit - sizeof(*xh)) /
+ sizeof(struct ocfs2_xattr_entry);
+
+ if (xattr_count > max_entries)
+ return ocfs2_error(sb,
+ "Invalid xattr in block %llu: entry count %u exceeds maximum %zu\n",
+ (unsigned long long)blkno,
+ xattr_count, max_entries);
+
+ for (i = 0; i < xattr_count; i++) {
+ struct ocfs2_xattr_entry *xe = &xh->xh_entries[i];
+ size_t name_offset = le16_to_cpu(xe->xe_name_offset);
+ size_t value_offset;
+
+ if (name_offset > nv_limit ||
+ xe->xe_name_len > nv_limit - name_offset)
+ return ocfs2_error(sb,
+ "Invalid xattr in block %llu: entry %d name is out of bounds\n",
+ (unsigned long long)blkno, i);
+
+ value_offset = name_offset + OCFS2_XATTR_SIZE(xe->xe_name_len);
+ if (value_offset > nv_limit)
+ return ocfs2_error(sb,
+ "Invalid xattr in block %llu: entry %d value starts out of bounds\n",
+ (unsigned long long)blkno, i);
+
+ if (ocfs2_xattr_is_local(xe)) {
+ if (le64_to_cpu(xe->xe_value_size) >
+ nv_limit - value_offset)
+ return ocfs2_error(sb,
+ "Invalid xattr in block %llu: entry %d value is out of bounds\n",
+ (unsigned long long)blkno,
+ i);
+ } else if (sizeof(struct ocfs2_xattr_value_root) >
+ nv_limit - value_offset) {
+ return ocfs2_error(sb,
+ "Invalid xattr in block %llu: entry %d value root is out of bounds\n",
+ (unsigned long long)blkno, i);
+ }
+ }
+
+ return 0;
+}
+
+static int ocfs2_xattr_ibody_lookup_header_raw(struct super_block *sb,
+ u64 blkno,
+ struct ocfs2_dinode *di,
+ struct ocfs2_xattr_header **header,
+ u16 *inline_size_ret)
+{
+ struct ocfs2_xattr_header *xh;
+ u16 xattr_count;
+ size_t max_entries;
+ u16 inline_size = le16_to_cpu(di->i_xattr_inline_size);
+
+ if (inline_size > sb->s_blocksize ||
+ inline_size < sizeof(struct ocfs2_xattr_header)) {
+ ocfs2_error(sb,
+ "Invalid inode %llu: xattr inline size %u\n",
+ (unsigned long long)blkno, inline_size);
+ return -EFSCORRUPTED;
+ }
+
+ xh = (struct ocfs2_xattr_header *)
+ ((void *)di + sb->s_blocksize - inline_size);
+
+ xattr_count = le16_to_cpu(xh->xh_count);
+ max_entries = (inline_size - sizeof(struct ocfs2_xattr_header)) /
+ sizeof(struct ocfs2_xattr_entry);
+
+ if (xattr_count > max_entries) {
+ ocfs2_error(sb,
+ "xattr entry count %u exceeds maximum %zu in inode %llu\n",
+ xattr_count, max_entries,
+ (unsigned long long)blkno);
+ return -EFSCORRUPTED;
+ }
+
+ *header = xh;
+ if (inline_size_ret)
+ *inline_size_ret = inline_size;
+
+ return 0;
+}
+
+int ocfs2_validate_inode_xattr(struct super_block *sb, u64 blkno,
+ struct ocfs2_dinode *di)
+{
+ struct ocfs2_xattr_header *xh;
+ u16 inline_size;
+ int ret;
+
+ if (!(le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_XATTR_FL))
+ return 0;
+
+ ret = ocfs2_xattr_ibody_lookup_header_raw(sb, blkno, di, &xh,
+ &inline_size);
+ if (ret)
+ return ret;
+
+ return ocfs2_validate_xattr_entries_flat(sb, blkno, xh, inline_size);
+}
+
+static int ocfs2_validate_xattr_bucket(struct ocfs2_xattr_bucket *bucket,
+ u64 blkno)
+{
+ struct super_block *sb = bucket->bu_inode->i_sb;
+ struct ocfs2_xattr_header *xh = bucket_xh(bucket);
+ u16 xattr_count = le16_to_cpu(xh->xh_count);
+ size_t region_size = (size_t)sb->s_blocksize * bucket->bu_blocks;
+ size_t entries_limit = sb->s_blocksize;
+ size_t nv_limit = sb->s_blocksize;
+ size_t max_entries;
+ int i;
+
+ if (region_size < sizeof(*xh))
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: region size %zu is too small\n",
+ (unsigned long long)blkno, region_size);
+
+ if (entries_limit < sizeof(*xh))
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: entries limit %zu is too small\n",
+ (unsigned long long)blkno,
+ entries_limit);
+
+ max_entries = (entries_limit - sizeof(*xh)) /
+ sizeof(struct ocfs2_xattr_entry);
+
+ if (xattr_count > max_entries)
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: entry count %u exceeds maximum %zu\n",
+ (unsigned long long)blkno,
+ xattr_count, max_entries);
+
+ for (i = 0; i < xattr_count; i++) {
+ struct ocfs2_xattr_entry *xe = &xh->xh_entries[i];
+ size_t name_offset = le16_to_cpu(xe->xe_name_offset);
+ size_t block_off = name_offset >> sb->s_blocksize_bits;
+ size_t block_offset = name_offset % nv_limit;
+ size_t value_offset;
+
+ if (name_offset >= region_size || block_off >= bucket->bu_blocks)
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: entry %d name is out of bounds\n",
+ (unsigned long long)blkno, i);
+
+ if (xe->xe_name_len > nv_limit - block_offset)
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: entry %d name crosses block boundary\n",
+ (unsigned long long)blkno, i);
+
+ value_offset = block_offset + OCFS2_XATTR_SIZE(xe->xe_name_len);
+ if (value_offset > nv_limit)
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: entry %d value starts out of bounds\n",
+ (unsigned long long)blkno, i);
+
+ if (ocfs2_xattr_is_local(xe)) {
+ if (le64_to_cpu(xe->xe_value_size) >
+ nv_limit - value_offset)
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: entry %d value is out of bounds\n",
+ (unsigned long long)blkno,
+ i);
+ } else if (sizeof(struct ocfs2_xattr_value_root) >
+ nv_limit - value_offset) {
+ return ocfs2_error(sb,
+ "Invalid xattr bucket %llu: entry %d value root is out of bounds\n",
+ (unsigned long long)blkno, i);
+ }
+ }
+
+ return 0;
+}
+
int ocfs2_has_inline_xattr_value_outside(struct inode *inode,
struct ocfs2_dinode *di)
{
diff --git a/fs/ocfs2/xattr.h b/fs/ocfs2/xattr.h
index aeba323e14b3..887cc1a18b1a 100644
--- a/fs/ocfs2/xattr.h
+++ b/fs/ocfs2/xattr.h
@@ -43,6 +43,8 @@ int ocfs2_xattr_set_handle(handle_t *, struct inode *, struct buffer_head *,
struct ocfs2_alloc_context *);
int ocfs2_has_inline_xattr_value_outside(struct inode *inode,
struct ocfs2_dinode *di);
+int ocfs2_validate_inode_xattr(struct super_block *sb, u64 blkno,
+ struct ocfs2_dinode *di);
int ocfs2_xattr_remove(struct inode *, struct buffer_head *);
int ocfs2_init_security_get(struct inode *, struct inode *,
const struct qstr *,
diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
index 3765e1defa19..3a1f120b2af6 100644
--- a/fs/overlayfs/file.c
+++ b/fs/overlayfs/file.c
@@ -488,6 +488,7 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t len, unsigned int flags, enum ovl_copyop op)
{
+ struct inode *inode_in = file_inode(file_in);
struct inode *inode_out = file_inode(file_out);
struct fd real_in, real_out;
const struct cred *old_cred;
@@ -512,7 +513,20 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in,
goto out_unlock;
}
- old_cred = ovl_override_creds(file_inode(file_out)->i_sb);
+ /*
+ * For cross-sb copy, vfs_copy_file_range() will verify read access with
+ * the mounter creds of the dest fs mounter, so we need to explicitly
+ * verify read access with the source mounter creds.
+ */
+ if (unlikely(inode_in->i_sb != inode_out->i_sb)) {
+ old_cred = ovl_override_creds(inode_in->i_sb);
+ ret = rw_verify_area(READ, fd_file(real_in), &pos_in, len);
+ revert_creds(old_cred);
+ if (unlikely(ret))
+ goto out_fdput;
+ }
+
+ old_cred = ovl_override_creds(inode_out->i_sb);
switch (op) {
case OVL_COPY:
ret = vfs_copy_file_range(fd_file(real_in), pos_in,
@@ -535,6 +549,7 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in,
/* Update size */
ovl_file_modified(file_out);
+out_fdput:
fdput(real_in);
fdput(real_out);
diff --git a/fs/smb/client/cifsacl.c b/fs/smb/client/cifsacl.c
index 102a7abd7274..a9157dbc40bb 100644
--- a/fs/smb/client/cifsacl.c
+++ b/fs/smb/client/cifsacl.c
@@ -1315,6 +1315,7 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
{
int rc = 0;
struct smb_sid *owner_sid_ptr, *group_sid_ptr;
+ unsigned int sbflags = cifs_sb->mnt_cifs_flags;
struct smb_acl *dacl_ptr; /* no need for SACL ptr */
char *end_of_acl = ((char *)pntsd) + acl_len;
__u32 dacloffset;
@@ -1332,16 +1333,21 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
le32_to_cpu(pntsd->gsidoffset),
le32_to_cpu(pntsd->sacloffset), dacloffset);
/* cifs_dump_mem("owner_sid: ", owner_sid_ptr, 64); */
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+
rc = parse_sid(owner_sid_ptr, end_of_acl);
if (rc) {
cifs_dbg(FYI, "%s: Error %d parsing Owner SID\n", __func__, rc);
return rc;
}
- rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER);
- if (rc) {
- cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n",
- __func__, rc);
- return rc;
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID)) {
+ rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER);
+ if (rc) {
+ cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n",
+ __func__, rc);
+ return rc;
+ }
}
rc = parse_sid(group_sid_ptr, end_of_acl);
@@ -1350,11 +1356,13 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
__func__, rc);
return rc;
}
- rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP);
- if (rc) {
- cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n",
- __func__, rc);
- return rc;
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID)) {
+ rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP);
+ if (rc) {
+ cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n",
+ __func__, rc);
+ return rc;
+ }
}
if (dacloffset) {
diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c
index 804a7e2f2c6d..23ffe878dc3e 100644
--- a/fs/smb/client/cifsfs.c
+++ b/fs/smb/client/cifsfs.c
@@ -434,6 +434,7 @@ cifs_alloc_inode(struct super_block *sb)
return NULL;
cifs_inode->cifsAttrs = 0x20; /* default */
cifs_inode->time = 0;
+ cifs_inode->time_last_write = 0;
/*
* Until the file is open and we have gotten oplock info back from the
* server, can not assume caching of file data or metadata.
diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h
index bbb06822d60d..0fbb3cd0daf5 100644
--- a/fs/smb/client/cifsglob.h
+++ b/fs/smb/client/cifsglob.h
@@ -1535,6 +1535,7 @@ struct cifsInodeInfo {
spinlock_t writers_lock;
unsigned int writers; /* Number of writers on this inode */
unsigned long time; /* jiffies of last update of inode */
+ unsigned long time_last_write; /* jiffies of last writable close or truncate */
u64 uniqueid; /* server inode number */
u64 createtime; /* creation time on server */
__u8 lease_key[SMB2_LEASE_KEY_SIZE]; /* lease key for this inode */
@@ -2195,7 +2196,7 @@ static inline int cifs_get_num_sgs(const struct smb_rqst *rqst,
struct kvec *iov = &rqst[i].rq_iov[j];
addr = (unsigned long)iov->iov_base + skip;
- if (unlikely(is_vmalloc_addr((void *)addr))) {
+ if (is_vmalloc_or_module_addr((void *)addr)) {
len = iov->iov_len - skip;
nents += DIV_ROUND_UP(offset_in_page(addr) + len,
PAGE_SIZE);
@@ -2222,7 +2223,7 @@ static inline void cifs_sg_set_buf(struct sg_table *sgtable,
unsigned int off = offset_in_page(addr);
addr &= PAGE_MASK;
- if (unlikely(is_vmalloc_addr((void *)addr))) {
+ if (is_vmalloc_or_module_addr((void *)addr)) {
do {
unsigned int len = min_t(unsigned int, buflen, PAGE_SIZE - off);
diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c
index a7261fb38379..7db347fc8faa 100644
--- a/fs/smb/client/connect.c
+++ b/fs/smb/client/connect.c
@@ -3895,6 +3895,10 @@ CIFSTCon(const unsigned int xid, struct cifs_ses *ses,
tcon->tid = smb_buffer_response->Tid;
bcc_ptr = pByteArea(smb_buffer_response);
bytes_left = get_bcc(smb_buffer_response);
+ if (bytes_left < 2) {
+ rc = -EIO;
+ goto out;
+ }
length = strnlen(bcc_ptr, bytes_left - 2);
if (smb_buffer->Flags2 & SMBFLG2_UNICODE)
is_unicode = true;
@@ -3950,6 +3954,7 @@ CIFSTCon(const unsigned int xid, struct cifs_ses *ses,
reset_cifs_unix_caps(xid, tcon, NULL, NULL);
}
}
+out:
cifs_buf_release(smb_buffer);
return rc;
}
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index ee3cc2b2fda4..b6eea70554d7 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -231,7 +231,7 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq)
failed:
add_credits_and_wake_if(rdata->server, &rdata->credits, 0);
subreq->error = rc;
- netfs_read_subreq_terminated(subreq, false);
+ netfs_read_subreq_terminated(subreq);
}
/*
@@ -1325,11 +1325,21 @@ void smb2_deferred_work_close(struct work_struct *work)
{
struct cifsFileInfo *cfile = container_of(work,
struct cifsFileInfo, deferred.work);
+ struct cifsInodeInfo *cinode = CIFS_I(d_inode(cfile->dentry));
- spin_lock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
+ spin_lock(&cinode->deferred_lock);
cifs_del_deferred_close(cfile);
cfile->deferred_close_scheduled = false;
- spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
+ spin_unlock(&cinode->deferred_lock);
+ /*
+ * Refresh time_last_write immediately before the actual server close
+ * so the protection window is anchored to the real close time, not
+ * the earlier userspace close time stored by cifs_close().
+ */
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cinode->time_last_write, jiffies);
+ }
_cifsFileInfo_put(cfile, true, false);
}
@@ -1358,6 +1368,10 @@ int cifs_close(struct inode *inode, struct file *file)
if (file->private_data != NULL) {
cfile = file->private_data;
file->private_data = NULL;
+ if (file->f_mode & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cinode->time_last_write, jiffies);
+ }
dclose = kmalloc(sizeof(struct cifs_deferred_close), GFP_KERNEL);
if ((cfile->status_file_deleted == false) &&
(smb2_can_defer_close(inode, dclose))) {
@@ -3046,15 +3060,27 @@ static int is_inode_writable(struct cifsInodeInfo *cifs_inode)
bool is_size_safe_to_change(struct cifsInodeInfo *cifsInode, __u64 end_of_file,
bool from_readdir)
{
+ struct cifs_sb_info *cifs_sb;
+ unsigned long tlw;
+
if (!cifsInode)
return true;
+ cifs_sb = CIFS_SB(cifsInode->netfs.inode.i_sb);
+
if (is_inode_writable(cifsInode) ||
((cifsInode->oplock & CIFS_CACHE_RW_FLG) != 0 && from_readdir)) {
/* This inode is open for write at least once */
- struct cifs_sb_info *cifs_sb;
- cifs_sb = CIFS_SB(cifsInode->netfs.inode.i_sb);
+ /*
+ * Readdir data is unreliable when we have writable handles or
+ * an exclusive lease -- never allow it to change i_size, even
+ * on direct-IO mounts where the server's directory metadata
+ * can still lag behind the actual file state.
+ */
+ if (from_readdir)
+ return false;
+
if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_DIRECT_IO) {
/* since no page cache to corrupt on directio
we can change size safely */
@@ -3065,8 +3091,40 @@ bool is_size_safe_to_change(struct cifsInodeInfo *cifsInode, __u64 end_of_file,
return true;
return false;
- } else
- return true;
+ }
+
+ /*
+ * No writable handles open. Check whether we are within the attribute
+ * cache validity window of a recent local modification.
+ *
+ * For the close() path: cifs_close() calls smp_store_release() on
+ * time_last_write before _cifsFileInfo_put() removes the handle under
+ * open_file_lock. That spin_unlock() is a store-release that pairs
+ * with the spin_lock() (load-acquire) in is_inode_writable() above,
+ * so if is_inode_writable() returned false the smp_load_acquire()
+ * below is guaranteed to observe any time_last_write update from a
+ * concurrent close().
+ *
+ * For the setattr/truncate paths: those callers use smp_store_release()
+ * directly; the smp_load_acquire() below pairs with that store. There
+ * is no shared lock between setattr and readdir, so this relies on
+ * acquire-release semantics alone. The store propagation latency on
+ * weakly-ordered architectures (nanoseconds) is negligible relative to
+ * the acregmax window (seconds) and the readdir RPC round-trip
+ * (milliseconds), making this a sound design choice in practice.
+ *
+ * time_last_write == 0 means the inode has never been written locally;
+ * skip the window check to avoid false positives near boot time when
+ * jiffies is still close to INITIAL_JIFFIES on 32-bit systems.
+ */
+ if (from_readdir) {
+ /* Pairs with smp_store_release() at close and truncate sites. */
+ tlw = smp_load_acquire(&cifsInode->time_last_write);
+ if (tlw && time_before(jiffies, tlw + cifs_sb->ctx->acregmax))
+ return false;
+ }
+
+ return true;
}
void cifs_oplock_break(struct work_struct *work)
diff --git a/fs/smb/client/inode.c b/fs/smb/client/inode.c
index 7c4828580076..05f525621e0f 100644
--- a/fs/smb/client/inode.c
+++ b/fs/smb/client/inode.c
@@ -242,6 +242,8 @@ cifs_fattr_to_inode(struct inode *inode, struct cifs_fattr *fattr,
* calculating num blocks.
*/
inode->i_blocks = (512 - 1 + fattr->cf_bytes) >> 9;
+ } else if (from_readdir && i_size_read(inode) != fattr->cf_eof) {
+ cifs_i->time = 0;
}
if (S_ISLNK(fattr->cf_mode) && fattr->cf_symlink_target) {
@@ -831,6 +833,7 @@ static void smb311_posix_info_to_fattr(struct cifs_fattr *fattr,
struct smb311_posix_qinfo *info = &data->posix_fi;
struct cifs_sb_info *cifs_sb = CIFS_SB(sb);
struct cifs_tcon *tcon = cifs_sb_master_tcon(cifs_sb);
+ unsigned int sbflags = cifs_sb->mnt_cifs_flags;
memset(fattr, 0, sizeof(*fattr));
@@ -875,8 +878,12 @@ static void smb311_posix_info_to_fattr(struct cifs_fattr *fattr,
fattr->cf_symlink_target = data->symlink_target;
data->symlink_target = NULL;
}
- sid_to_id(cifs_sb, &data->posix_owner, fattr, SIDOWNER);
- sid_to_id(cifs_sb, &data->posix_group, fattr, SIDGROUP);
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID))
+ sid_to_id(cifs_sb, &data->posix_owner, fattr, SIDOWNER);
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID))
+ sid_to_id(cifs_sb, &data->posix_group, fattr, SIDGROUP);
cifs_dbg(FYI, "POSIX query info: mode 0x%x uniqueid 0x%llx nlink %d\n",
fattr->cf_mode, fattr->cf_uniqueid, fattr->cf_nlink);
@@ -2957,13 +2964,17 @@ int cifs_truncate_page(struct address_space *mapping, loff_t from)
void cifs_setsize(struct inode *inode, loff_t offset)
{
struct cifsInodeInfo *cifs_i = CIFS_I(inode);
+ loff_t old_size;
spin_lock(&inode->i_lock);
+ old_size = i_size_read(inode);
i_size_write(inode, offset);
spin_unlock(&inode->i_lock);
/* Cached inode must be refreshed on truncate */
cifs_i->time = 0;
+ if (offset > old_size)
+ pagecache_isize_extended(inode, old_size, offset);
truncate_pagecache(inode, offset);
netfs_wait_for_outstanding_io(inode);
fscache_resize_cookie(cifs_inode_cookie(inode), offset);
@@ -3188,6 +3199,8 @@ cifs_setattr_unix(struct dentry *direntry, struct iattr *attrs)
if ((attrs->ia_valid & ATTR_SIZE) &&
attrs->ia_size != i_size_read(inode)) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cifsInode->time_last_write, jiffies);
truncate_setsize(inode, attrs->ia_size);
netfs_resize_file(&cifsInode->netfs, attrs->ia_size, true);
fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size);
@@ -3296,7 +3309,8 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs)
gid = attrs->ia_gid;
if ((cifs_sb->mnt_cifs_flags & CIFS_MOUNT_CIFS_ACL) ||
- (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MODE_FROM_SID)) {
+ (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MODE_FROM_SID) ||
+ cifs_sb_master_tcon(cifs_sb)->posix_extensions) {
if (uid_valid(uid) || gid_valid(gid)) {
mode = NO_CHANGE_64;
rc = id_mode_to_cifs_acl(inode, full_path, &mode,
@@ -3406,6 +3420,8 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs)
if ((attrs->ia_valid & ATTR_SIZE) &&
attrs->ia_size != i_size_read(inode)) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cifsInode->time_last_write, jiffies);
truncate_setsize(inode, attrs->ia_size);
netfs_resize_file(&cifsInode->netfs, attrs->ia_size, true);
fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size);
diff --git a/fs/smb/client/misc.c b/fs/smb/client/misc.c
index 5ac8faeda4f7..41c1c18fb0f1 100644
--- a/fs/smb/client/misc.c
+++ b/fs/smb/client/misc.c
@@ -772,7 +772,7 @@ cifs_del_deferred_close(struct cifsFileInfo *cfile)
void
cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
{
- struct cifsFileInfo *cfile = NULL;
+ struct cifsFileInfo *cfile = NULL, *failed_cfile = NULL;
struct file_list *tmp_list, *tmp_next_list;
LIST_HEAD(file_head);
@@ -788,8 +788,10 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
spin_unlock(&cifs_inode->deferred_lock);
tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
@@ -797,8 +799,24 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
}
spin_unlock(&cifs_inode->open_file_lock);
+ if (failed_cfile) {
+ if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(failed_cfile, false, false);
+ }
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
- _cifsFileInfo_put(tmp_list->cfile, false, false);
+ struct cifsFileInfo *cfile = tmp_list->cfile;
+
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(cfile, false, false);
list_del(&tmp_list->list);
kfree(tmp_list);
}
@@ -807,7 +825,7 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
void
cifs_close_all_deferred_files(struct cifs_tcon *tcon)
{
- struct cifsFileInfo *cfile;
+ struct cifsFileInfo *cfile, *failed_cfile = NULL;
struct file_list *tmp_list, *tmp_next_list;
LIST_HEAD(file_head);
@@ -820,8 +838,10 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon)
spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
@@ -829,8 +849,24 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon)
}
spin_unlock(&tcon->open_file_lock);
+ if (failed_cfile) {
+ if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(failed_cfile, true, false);
+ }
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
- _cifsFileInfo_put(tmp_list->cfile, true, false);
+ struct cifsFileInfo *cfile = tmp_list->cfile;
+
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(cfile, true, false);
list_del(&tmp_list->list);
kfree(tmp_list);
}
@@ -877,7 +913,7 @@ void cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon,
struct dentry *dentry)
{
struct file_list *tmp_list, *tmp_next_list;
- struct cifsFileInfo *cfile;
+ struct cifsFileInfo *cfile, *failed_cfile = NULL;
LIST_HEAD(file_head);
spin_lock(&tcon->open_file_lock);
@@ -890,16 +926,34 @@ void cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon,
spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
}
spin_unlock(&tcon->open_file_lock);
+ if (failed_cfile) {
+ if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(failed_cfile, true, false);
+ }
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
- _cifsFileInfo_put(tmp_list->cfile, true, false);
+ struct cifsFileInfo *cfile = tmp_list->cfile;
+
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(cfile, true, false);
list_del(&tmp_list->list);
kfree(tmp_list);
}
diff --git a/fs/smb/client/readdir.c b/fs/smb/client/readdir.c
index 6b619506b7a1..f1f7a718f5d5 100644
--- a/fs/smb/client/readdir.c
+++ b/fs/smb/client/readdir.c
@@ -241,9 +241,11 @@ static void
cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info,
struct cifs_sb_info *cifs_sb)
{
+ unsigned int sbflags = cifs_sb->mnt_cifs_flags;
struct smb2_posix_info_parsed parsed;
+ int rc;
- posix_info_parse(info, NULL, &parsed);
+ rc = posix_info_parse(info, NULL, &parsed);
memset(fattr, 0, sizeof(*fattr));
fattr->cf_uniqueid = le64_to_cpu(info->Inode);
@@ -280,8 +282,17 @@ cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info,
le32_to_cpu(info->ReparseTag),
le32_to_cpu(info->Mode));
- sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER);
- sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP);
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+ if (rc < 0) {
+ cifs_dbg(VFS, "%s: failed to parse SIDs: %d\n",
+ __func__, rc);
+ } else {
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID))
+ sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER);
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID))
+ sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP);
+ }
}
static void __dir_info_to_fattr(struct cifs_fattr *fattr, const void *info)
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 55484d38ffe9..6ef0fcd9a05c 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -2842,7 +2842,7 @@ smb2_set_next_command(struct cifs_tcon *tcon, struct smb_rqst *rqst)
struct cifs_ses *ses = tcon->ses;
struct TCP_Server_Info *server = ses->server;
unsigned long len = smb_rqst_len(server, rqst);
- int i, num_padding;
+ int num_padding;
shdr = (struct smb2_hdr *)(rqst->rq_iov[0].iov_base);
if (shdr == NULL) {
@@ -2851,28 +2851,16 @@ smb2_set_next_command(struct cifs_tcon *tcon, struct smb_rqst *rqst)
}
/* SMB headers in a compound are 8 byte aligned. */
-
- /* No padding needed */
- if (!(len & 7))
- goto finished;
+ if (IS_ALIGNED(len, 8))
+ goto out;
num_padding = 8 - (len & 7);
- if (!smb3_encryption_required(tcon)) {
- /*
- * If we do not have encryption then we can just add an extra
- * iov for the padding.
- */
- rqst->rq_iov[rqst->rq_nvec].iov_base = smb2_padding;
- rqst->rq_iov[rqst->rq_nvec].iov_len = num_padding;
- rqst->rq_nvec++;
- len += num_padding;
- } else {
+ if (smb3_encryption_required(tcon)) {
+ int i;
+
/*
- * We can not add a small padding iov for the encryption case
- * because the encryption framework can not handle the padding
- * iovs.
- * We have to flatten this into a single buffer and add
- * the padding to it.
+ * Flatten request into a single buffer with required padding as
+ * the encryption layer can't handle the padding iovs.
*/
for (i = 1; i < rqst->rq_nvec; i++) {
memcpy(rqst->rq_iov[0].iov_base +
@@ -2884,11 +2872,14 @@ smb2_set_next_command(struct cifs_tcon *tcon, struct smb_rqst *rqst)
memset(rqst->rq_iov[0].iov_base + rqst->rq_iov[0].iov_len,
0, num_padding);
rqst->rq_iov[0].iov_len += num_padding;
- len += num_padding;
rqst->rq_nvec = 1;
+ } else {
+ rqst->rq_iov[rqst->rq_nvec].iov_base = smb2_padding;
+ rqst->rq_iov[rqst->rq_nvec].iov_len = num_padding;
+ rqst->rq_nvec++;
}
-
- finished:
+ len += num_padding;
+out:
shdr->NextCommand = cpu_to_le32(len);
}
@@ -3533,6 +3524,9 @@ static long smb3_zero_range(struct file *file, struct cifs_tcon *tcon,
if (keep_size == false && !CIFS_CACHE_READ(cifsi))
goto zero_range_exit;
+ fscache_invalidate(cifs_inode_cookie(inode), NULL,
+ i_size_read(inode), 0);
+
rc = smb3_zero_data(file, tcon, offset, len, xid);
if (rc < 0)
goto zero_range_exit;
@@ -3600,6 +3594,8 @@ static long smb3_punch_hole(struct file *file, struct cifs_tcon *tcon,
*/
truncate_pagecache_range(inode, offset, offset + len - 1);
netfs_wait_for_outstanding_io(inode);
+ fscache_invalidate(cifs_inode_cookie(inode), NULL,
+ i_size_read(inode), 0);
cifs_dbg(FYI, "Offset %lld len %lld\n", offset, len);
@@ -3945,6 +3941,7 @@ static long smb3_collapse_range(struct file *file, struct cifs_tcon *tcon,
truncate_pagecache_range(inode, off, old_eof);
ictx->zero_point = old_eof;
netfs_wait_for_outstanding_io(inode);
+ fscache_invalidate(cifs_inode_cookie(inode), NULL, old_eof, 0);
rc = __smb2_copychunk_range(xid, cfile, cfile, off + len,
old_eof - off - len, off);
@@ -4007,6 +4004,7 @@ static long smb3_insert_range(struct file *file, struct cifs_tcon *tcon,
goto out_2;
truncate_pagecache_range(inode, off, old_eof);
netfs_wait_for_outstanding_io(inode);
+ fscache_invalidate(cifs_inode_cookie(inode), NULL, old_eof, 0);
rc = SMB2_set_eof(xid, tcon, cfile->fid.persistent_fid,
cfile->fid.volatile_fid, cfile->pid, new_eof);
@@ -4722,7 +4720,7 @@ static struct folio_queue *cifs_alloc_folioq_buffer(ssize_t size)
p = kmalloc(sizeof(*p), GFP_NOFS);
if (!p)
goto nomem;
- folioq_init(p);
+ folioq_init(p, 0);
if (tail) {
tail->next = p;
p->prev = tail;
diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c
index 1dbf8d28bdf7..d7d0f60b8540 100644
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@ -3357,15 +3357,6 @@ SMB2_ioctl_init(struct cifs_tcon *tcon, struct TCP_Server_Info *server,
return rc;
if (indatalen) {
- unsigned int len;
-
- if (WARN_ON_ONCE(smb3_encryption_required(tcon) &&
- (check_add_overflow(total_len - 1,
- ALIGN(indatalen, 8), &len) ||
- len > MAX_CIFS_SMALL_BUFFER_SIZE))) {
- cifs_small_buf_release(req);
- return -EIO;
- }
/*
* indatalen is usually small at a couple of bytes max, so
* just allocate through generic pool
diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index 07f71a9481a3..c4c1435e8114 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -460,7 +460,10 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
struct smbd_response *response =
container_of(wc->wr_cqe, struct smbd_response, cqe);
struct smbd_connection *info = response->info;
- int data_length = 0;
+ struct smbdirect_socket_parameters *sp = &info->socket.parameters;
+ u32 data_offset = 0;
+ u32 data_length = 0;
+ u32 remaining_data_length = 0;
log_rdma_recv(INFO, "response=0x%p type=%d wc status=%d wc opcode %d byte_len=%d pkey_index=%u\n",
response, response->type, wc->status, wc->opcode,
@@ -492,7 +495,22 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
/* SMBD data transfer packet */
case SMBD_TRANSFER_DATA:
data_transfer = smbd_response_payload(response);
+
+ if (wc->byte_len <
+ offsetof(struct smbdirect_data_transfer, padding))
+ goto error;
+
+ remaining_data_length = le32_to_cpu(data_transfer->remaining_data_length);
+ data_offset = le32_to_cpu(data_transfer->data_offset);
data_length = le32_to_cpu(data_transfer->data_length);
+ if (wc->byte_len < data_offset ||
+ (u64)wc->byte_len < (u64)data_offset + data_length)
+ goto error;
+
+ if (remaining_data_length > sp->max_fragmented_recv_size ||
+ data_length > sp->max_fragmented_recv_size ||
+ (u64)remaining_data_length + (u64)data_length > (u64)sp->max_fragmented_recv_size)
+ goto error;
if (data_length) {
if (info->full_packet_received)
diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c
index 48d382eaadcb..9fa0c4161bb8 100644
--- a/fs/smb/client/transport.c
+++ b/fs/smb/client/transport.c
@@ -425,19 +425,16 @@ __smb_send_rqst(struct TCP_Server_Info *server, int num_rqst,
return rc;
}
-struct send_req_vars {
- struct smb2_transform_hdr tr_hdr;
- struct smb_rqst rqst[MAX_COMPOUND];
- struct kvec iov;
-};
-
static int
smb_send_rqst(struct TCP_Server_Info *server, int num_rqst,
struct smb_rqst *rqst, int flags)
{
- struct send_req_vars *vars;
- struct smb_rqst *cur_rqst;
- struct kvec *iov;
+ struct smb2_transform_hdr tr_hdr;
+ struct smb_rqst new_rqst[MAX_COMPOUND] = {};
+ struct kvec iov = {
+ .iov_base = &tr_hdr,
+ .iov_len = sizeof(tr_hdr),
+ };
int rc;
if (flags & CIFS_COMPRESS_REQ)
@@ -454,26 +451,15 @@ smb_send_rqst(struct TCP_Server_Info *server, int num_rqst,
return -EIO;
}
- vars = kzalloc(sizeof(*vars), GFP_NOFS);
- if (!vars)
- return -ENOMEM;
- cur_rqst = vars->rqst;
- iov = &vars->iov;
-
- iov->iov_base = &vars->tr_hdr;
- iov->iov_len = sizeof(vars->tr_hdr);
- cur_rqst[0].rq_iov = iov;
- cur_rqst[0].rq_nvec = 1;
+ new_rqst[0].rq_iov = &iov;
+ new_rqst[0].rq_nvec = 1;
rc = server->ops->init_transform_rq(server, num_rqst + 1,
- &cur_rqst[0], rqst);
- if (rc)
- goto out;
-
- rc = __smb_send_rqst(server, num_rqst + 1, &cur_rqst[0]);
- smb3_free_compound_rqst(num_rqst, &cur_rqst[1]);
-out:
- kfree(vars);
+ new_rqst, rqst);
+ if (!rc) {
+ rc = __smb_send_rqst(server, num_rqst + 1, new_rqst);
+ smb3_free_compound_rqst(num_rqst, &new_rqst[1]);
+ }
return rc;
}
diff --git a/fs/smb/server/auth.c b/fs/smb/server/auth.c
index d0f940367623..97ffce1ff417 100644
--- a/fs/smb/server/auth.c
+++ b/fs/smb/server/auth.c
@@ -278,15 +278,15 @@ int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess,
ksmbd_release_crypto_ctx(ctx);
ctx = NULL;
- rc = ksmbd_gen_sess_key(sess, ntlmv2_hash, ntlmv2_rsp);
- if (rc) {
- ksmbd_debug(AUTH, "Could not generate sess key\n");
+ if (crypto_memneq(ntlmv2->ntlmv2_hash, ntlmv2_rsp,
+ CIFS_HMAC_MD5_HASH_SIZE)) {
+ rc = -EINVAL;
goto out;
}
- if (crypto_memneq(ntlmv2->ntlmv2_hash, ntlmv2_rsp,
- CIFS_HMAC_MD5_HASH_SIZE))
- rc = -EINVAL;
+ rc = ksmbd_gen_sess_key(sess, ntlmv2_hash, ntlmv2_rsp);
+ if (rc)
+ ksmbd_debug(AUTH, "Could not generate sess key\n");
out:
if (ctx)
ksmbd_release_crypto_ctx(ctx);
@@ -348,6 +348,8 @@ int ksmbd_decode_ntlmssp_auth_blob(struct authenticate_message *authblob,
nt_len - CIFS_ENCPWD_SIZE,
domain_name, conn->ntlmssp.cryptkey);
kfree(domain_name);
+ if (ret)
+ return ret;
/* The recovered secondary session key */
if (conn->ntlmssp.client_flags & NTLMSSP_NEGOTIATE_KEY_XCH) {
diff --git a/fs/smb/server/connection.c b/fs/smb/server/connection.c
index c5f9705825b8..898e1832ce05 100644
--- a/fs/smb/server/connection.c
+++ b/fs/smb/server/connection.c
@@ -242,15 +242,37 @@ void ksmbd_conn_unlock(struct ksmbd_conn *conn)
mutex_unlock(&conn->srv_mutex);
}
-void ksmbd_all_conn_set_status(u64 sess_id, u32 status)
+static bool ksmbd_session_is_bound_to_conn(struct ksmbd_session *sess,
+ struct ksmbd_conn *conn)
+{
+ bool found;
+
+ rcu_read_lock();
+ found = xa_load(&conn->sessions, sess->id) == sess;
+ rcu_read_unlock();
+ if (found)
+ return true;
+
+ down_read(&sess->chann_lock);
+ found = xa_load(&sess->ksmbd_chann_list, (long)conn);
+ up_read(&sess->chann_lock);
+ return found;
+}
+
+void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status)
{
struct ksmbd_conn *conn;
int bkt;
down_read(&conn_list_lock);
hash_for_each(conn_list, bkt, conn, hlist) {
- if (conn->binding || xa_load(&conn->sessions, sess_id))
- WRITE_ONCE(conn->status, status);
+ if (ksmbd_session_is_bound_to_conn(sess, conn)) {
+ spin_lock(&conn->request_lock);
+ if (!ksmbd_conn_exiting(conn) &&
+ !ksmbd_conn_releasing(conn))
+ WRITE_ONCE(conn->status, status);
+ spin_unlock(&conn->request_lock);
+ }
}
up_read(&conn_list_lock);
}
@@ -276,7 +298,8 @@ void ksmbd_conn_wait_idle(struct ksmbd_conn *conn)
wait_event(conn->req_running_q, atomic_read(&conn->req_running) < 2);
}
-int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id)
+int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn,
+ struct ksmbd_session *sess)
{
struct ksmbd_conn *conn;
int rc, retry_count = 0, max_timeout = 120;
@@ -288,7 +311,7 @@ int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id)
down_read(&conn_list_lock);
hash_for_each(conn_list, bkt, conn, hlist) {
- if (conn->binding || xa_load(&conn->sessions, sess_id)) {
+ if (ksmbd_session_is_bound_to_conn(sess, conn)) {
rcount = (conn == curr_conn) ? 2 : 1;
if (atomic_read(&conn->req_running) >= rcount) {
rc = wait_event_timeout(conn->req_running_q,
@@ -609,8 +632,10 @@ static void stop_sessions(void)
* handler exited its receive loop for an unrelated
* reason).
*/
- if (READ_ONCE(conn->status) != KSMBD_SESS_RELEASING)
+ spin_lock(&conn->request_lock);
+ if (!ksmbd_conn_releasing(conn))
ksmbd_conn_set_exiting(conn);
+ spin_unlock(&conn->request_lock);
target = conn;
break;
}
diff --git a/fs/smb/server/connection.h b/fs/smb/server/connection.h
index 96ef44133051..1ddc53d3a8ab 100644
--- a/fs/smb/server/connection.h
+++ b/fs/smb/server/connection.h
@@ -20,6 +20,8 @@
#include "smb_common.h"
#include "ksmbd_work.h"
+struct ksmbd_session;
+
#define KSMBD_SOCKET_BACKLOG 16
enum {
@@ -161,7 +163,8 @@ extern struct rw_semaphore conn_list_lock;
bool ksmbd_conn_alive(struct ksmbd_conn *conn);
void ksmbd_conn_wait_idle(struct ksmbd_conn *conn);
-int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id);
+int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn,
+ struct ksmbd_session *sess);
struct ksmbd_conn *ksmbd_conn_alloc(void);
void ksmbd_conn_free(struct ksmbd_conn *conn);
struct ksmbd_conn *ksmbd_conn_get(struct ksmbd_conn *conn);
@@ -266,5 +269,5 @@ static inline void ksmbd_conn_set_releasing(struct ksmbd_conn *conn)
WRITE_ONCE(conn->status, KSMBD_SESS_RELEASING);
}
-void ksmbd_all_conn_set_status(u64 sess_id, u32 status);
+void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status);
#endif /* __CONNECTION_H__ */
diff --git a/fs/smb/server/mgmt/tree_connect.c b/fs/smb/server/mgmt/tree_connect.c
index d3483d9c757c..5f75f19f9f9f 100644
--- a/fs/smb/server/mgmt/tree_connect.c
+++ b/fs/smb/server/mgmt/tree_connect.c
@@ -77,7 +77,7 @@ ksmbd_tree_conn_connect(struct ksmbd_work *work, const char *share_name)
tree_conn->share_conf = sc;
tree_conn->t_state = TREE_NEW;
status.tree_conn = tree_conn;
- atomic_set(&tree_conn->refcount, 1);
+ atomic_set(&tree_conn->refcount, 2);
ret = xa_err(xa_store(&sess->tree_conns, tree_conn->id, tree_conn,
KSMBD_DEFAULT_GFP));
@@ -99,8 +99,10 @@ ksmbd_tree_conn_connect(struct ksmbd_work *work, const char *share_name)
void ksmbd_tree_connect_put(struct ksmbd_tree_connect *tcon)
{
- if (atomic_dec_and_test(&tcon->refcount))
+ if (atomic_dec_and_test(&tcon->refcount)) {
+ ksmbd_share_config_put(tcon->share_conf);
kfree(tcon);
+ }
}
int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
@@ -109,14 +111,21 @@ int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
int ret;
write_lock(&sess->tree_conns_lock);
+ if (tree_conn->t_state == TREE_DISCONNECTED ||
+ xa_load(&sess->tree_conns, tree_conn->id) != tree_conn) {
+ write_unlock(&sess->tree_conns_lock);
+ return -ENOENT;
+ }
+ tree_conn->t_state = TREE_DISCONNECTED;
xa_erase(&sess->tree_conns, tree_conn->id);
write_unlock(&sess->tree_conns_lock);
ret = ksmbd_ipc_tree_disconnect_request(sess->id, tree_conn->id);
ksmbd_release_tree_conn_id(sess, tree_conn->id);
- ksmbd_share_config_put(tree_conn->share_conf);
- if (atomic_dec_and_test(&tree_conn->refcount))
+ if (atomic_dec_and_test(&tree_conn->refcount)) {
+ ksmbd_share_config_put(tree_conn->share_conf);
kfree(tree_conn);
+ }
return ret;
}
@@ -148,15 +157,6 @@ int ksmbd_tree_conn_session_logoff(struct ksmbd_session *sess)
return -EINVAL;
xa_for_each(&sess->tree_conns, id, tc) {
- write_lock(&sess->tree_conns_lock);
- if (tc->t_state == TREE_DISCONNECTED) {
- write_unlock(&sess->tree_conns_lock);
- ret = -ENOENT;
- continue;
- }
- tc->t_state = TREE_DISCONNECTED;
- write_unlock(&sess->tree_conns_lock);
-
ret |= ksmbd_tree_conn_disconnect(sess, tc);
}
xa_destroy(&sess->tree_conns);
diff --git a/fs/smb/server/mgmt/user_config.c b/fs/smb/server/mgmt/user_config.c
index 53db14e91bfd..0e5fcd6f95c1 100644
--- a/fs/smb/server/mgmt/user_config.c
+++ b/fs/smb/server/mgmt/user_config.c
@@ -37,6 +37,17 @@ struct ksmbd_user *ksmbd_alloc_user(struct ksmbd_login_response *resp,
{
struct ksmbd_user *user;
+ /*
+ * resp->hash_sz is a __u16 taken from the mountd IPC login response but
+ * resp->hash[] is only KSMBD_REQ_MAX_HASH_SZ bytes. A malformed or
+ * malicious response can set hash_sz far beyond that (up to 65535),
+ * making the memcpy() below read past the response object
+ * (slab-out-of-bounds in ksmbd_alloc_user()). Reject any oversized
+ * hash rather than trust the length.
+ */
+ if (resp->hash_sz > sizeof(resp->hash))
+ return NULL;
+
user = kmalloc(sizeof(struct ksmbd_user), KSMBD_DEFAULT_GFP);
if (!user)
return NULL;
diff --git a/fs/smb/server/mgmt/user_session.c b/fs/smb/server/mgmt/user_session.c
index 2b605664ea3d..0c65a862f858 100644
--- a/fs/smb/server/mgmt/user_session.c
+++ b/fs/smb/server/mgmt/user_session.c
@@ -413,16 +413,16 @@ void destroy_previous_session(struct ksmbd_conn *conn,
memcmp(user->passkey, prev_user->passkey, user->passkey_sz))
goto out;
- ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_RECONNECT);
- err = ksmbd_conn_wait_idle_sess_id(conn, id);
+ ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_RECONNECT);
+ err = ksmbd_conn_wait_idle_sess(conn, prev_sess);
if (err) {
- ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP);
+ ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP);
goto out;
}
ksmbd_destroy_file_table(prev_sess);
prev_sess->state = SMB2_SESSION_EXPIRED;
- ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP);
+ ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP);
ksmbd_launch_ksmbd_durable_scavenger();
out:
up_write(&conn->session_lock);
diff --git a/fs/smb/server/oplock.c b/fs/smb/server/oplock.c
index 6183bd52ca9d..08e3e4504b03 100644
--- a/fs/smb/server/oplock.c
+++ b/fs/smb/server/oplock.c
@@ -56,6 +56,7 @@ static struct oplock_info *alloc_opinfo(struct ksmbd_work *work,
opinfo->conn = ksmbd_conn_get(work->conn);
opinfo->level = SMB2_OPLOCK_LEVEL_NONE;
opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock_init(&opinfo->state_lock);
opinfo->pending_break = 0;
opinfo->fid = id;
opinfo->Tid = Tid;
@@ -424,14 +425,23 @@ void close_id_del_oplock(struct ksmbd_file *fp)
opinfo_del(opinfo);
rcu_assign_pointer(fp->f_opinfo, NULL);
- if (opinfo->op_state == OPLOCK_ACK_WAIT) {
- opinfo->op_state = OPLOCK_CLOSING;
- wake_up_interruptible_all(&opinfo->oplock_q);
- if (opinfo->is_lease) {
- atomic_set(&opinfo->breaking_cnt, 0);
- wake_up_interruptible_all(&opinfo->oplock_brk);
- }
- }
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state == OPLOCK_ACK_WAIT && opinfo->is_lease)
+ atomic_set(&opinfo->breaking_cnt, 0);
+ /*
+ * An opinfo that has been removed from the inode list is terminal. Keep
+ * this transition and releasing pending_break under state_lock. a breaker
+ * takes the same lock before it acquires pending_break or sets ACK_WAIT.
+ */
+ opinfo->op_state = OPLOCK_CLOSING;
+ clear_bit_unlock(0, &opinfo->pending_break);
+ spin_unlock(&opinfo->state_lock);
+ wake_up_interruptible_all(&opinfo->oplock_q);
+ if (opinfo->is_lease)
+ wake_up_interruptible_all(&opinfo->oplock_brk);
+ /* memory barrier is needed for wake_up_bit() */
+ smp_mb__after_atomic();
+ wake_up_bit(&opinfo->pending_break, 0);
opinfo_count_dec(fp);
atomic_dec(&opinfo->refcount);
@@ -610,10 +620,16 @@ static void wait_for_break_ack(struct oplock_info *opinfo)
/* is this a timeout ? */
if (!rc) {
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state == OPLOCK_CLOSING) {
+ spin_unlock(&opinfo->state_lock);
+ return;
+ }
if (opinfo->is_lease)
opinfo->o_lease->state = SMB2_LEASE_NONE_LE;
opinfo->level = SMB2_OPLOCK_LEVEL_NONE;
opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
}
}
@@ -625,17 +641,46 @@ static void wake_up_oplock_break(struct oplock_info *opinfo)
wake_up_bit(&opinfo->pending_break, 0);
}
+static bool oplock_break_set_ack_wait(struct oplock_info *opinfo)
+{
+ bool ret = false;
+
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING) {
+ opinfo->op_state = OPLOCK_ACK_WAIT;
+ ret = true;
+ }
+ spin_unlock(&opinfo->state_lock);
+
+ return ret;
+}
+
static int oplock_break_pending(struct oplock_info *opinfo, int req_op_level)
{
- while (test_and_set_bit(0, &opinfo->pending_break)) {
+ for (;;) {
+ bool closing;
+
+ spin_lock(&opinfo->state_lock);
+ closing = opinfo->op_state == OPLOCK_CLOSING;
+ if (!closing && !test_and_set_bit(0, &opinfo->pending_break)) {
+ spin_unlock(&opinfo->state_lock);
+ break;
+ }
+ spin_unlock(&opinfo->state_lock);
+ if (closing)
+ return -ENOENT;
+
wait_on_bit(&opinfo->pending_break, 0, TASK_UNINTERRUPTIBLE);
/* Not immediately break to none. */
opinfo->open_trunc = 0;
- if (opinfo->op_state == OPLOCK_CLOSING)
+ spin_lock(&opinfo->state_lock);
+ closing = opinfo->op_state == OPLOCK_CLOSING;
+ spin_unlock(&opinfo->state_lock);
+ if (closing)
return -ENOENT;
- else if (opinfo->level <= req_op_level) {
+ if (opinfo->level <= req_op_level) {
if (opinfo->is_lease == false)
return 1;
@@ -1005,7 +1050,11 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci,
release_async_work(in_work);
}
- brk_opinfo->op_state = OPLOCK_ACK_WAIT;
+ if (!oplock_break_set_ack_wait(brk_opinfo)) {
+ atomic_dec_if_positive(&brk_opinfo->breaking_cnt);
+ wake_up_oplock_break(brk_opinfo);
+ return -ENOENT;
+ }
} else
atomic_dec(&brk_opinfo->breaking_cnt);
} else {
@@ -1014,8 +1063,12 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci,
return err < 0 ? err : 0;
if (brk_opinfo->level == SMB2_OPLOCK_LEVEL_BATCH ||
- brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE)
- brk_opinfo->op_state = OPLOCK_ACK_WAIT;
+ brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) {
+ if (!oplock_break_set_ack_wait(brk_opinfo)) {
+ wake_up_oplock_break(brk_opinfo);
+ return -ENOENT;
+ }
+ }
}
if (brk_opinfo->is_lease)
diff --git a/fs/smb/server/oplock.h b/fs/smb/server/oplock.h
index 795a9119dad9..6e8680c06db9 100644
--- a/fs/smb/server/oplock.h
+++ b/fs/smb/server/oplock.h
@@ -59,6 +59,7 @@ struct oplock_info {
struct ksmbd_file *o_fp;
int level;
int op_state;
+ spinlock_t state_lock;
unsigned long pending_break;
u64 fid;
atomic_t breaking_cnt;
diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c
index 5795a19bb4cb..2b06ff081a9b 100644
--- a/fs/smb/server/smb2pdu.c
+++ b/fs/smb/server/smb2pdu.c
@@ -1613,7 +1613,7 @@ static int krb5_authenticate(struct ksmbd_work *work,
in_len = le16_to_cpu(req->SecurityBufferLength);
out_blob = (char *)&rsp->hdr.ProtocolId +
le16_to_cpu(rsp->SecurityBufferOffset);
- out_len = work->response_sz -
+ out_len = work->response_sz - work->next_smb2_rsp_hdr_off -
(le16_to_cpu(rsp->SecurityBufferOffset) + 4);
retval = ksmbd_krb5_authenticate(sess, in_blob, in_len,
@@ -2001,6 +2001,7 @@ int smb2_tree_connect(struct ksmbd_work *work)
struct ksmbd_session *sess = work->sess;
char *treename = NULL, *name = NULL;
struct ksmbd_tree_conn_status status;
+ struct ksmbd_tree_connect *tree_conn = NULL;
struct ksmbd_share_config *share = NULL;
int rc = -EINVAL;
@@ -2025,9 +2026,10 @@ int smb2_tree_connect(struct ksmbd_work *work)
name, treename);
status = ksmbd_tree_conn_connect(work, name);
- if (status.ret == KSMBD_TREE_CONN_STATUS_OK)
+ if (status.ret == KSMBD_TREE_CONN_STATUS_OK) {
+ tree_conn = status.tree_conn;
rsp->hdr.Id.SyncId.TreeId = cpu_to_le32(status.tree_conn->id);
- else
+ } else
goto out_err1;
share = status.tree_conn->share_conf;
@@ -2059,8 +2061,15 @@ int smb2_tree_connect(struct ksmbd_work *work)
status.tree_conn->posix_extensions = true;
write_lock(&sess->tree_conns_lock);
- status.tree_conn->t_state = TREE_CONNECTED;
+ if (status.tree_conn->t_state == TREE_DISCONNECTED) {
+ status.ret = KSMBD_TREE_CONN_STATUS_ERROR;
+ share = NULL;
+ } else {
+ status.tree_conn->t_state = TREE_CONNECTED;
+ }
write_unlock(&sess->tree_conns_lock);
+ if (status.ret != KSMBD_TREE_CONN_STATUS_OK)
+ goto out_err1;
rsp->StructureSize = cpu_to_le16(16);
out_err1:
/*
@@ -2073,8 +2082,13 @@ int smb2_tree_connect(struct ksmbd_work *work)
rsp->ShareFlags = SMB2_SHAREFLAG_MANUAL_CACHING;
rc = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_tree_connect_rsp));
- if (rc)
+ if (rc) {
+ if (status.ret == KSMBD_TREE_CONN_STATUS_OK) {
+ ksmbd_tree_conn_disconnect(sess, status.tree_conn);
+ status.tree_conn = NULL;
+ }
status.ret = KSMBD_TREE_CONN_STATUS_NOMEM;
+ }
if (!IS_ERR(treename))
kfree(treename);
@@ -2110,6 +2124,9 @@ int smb2_tree_connect(struct ksmbd_work *work)
if (status.ret != KSMBD_TREE_CONN_STATUS_OK)
smb2_set_err_rsp(work);
+ if (tree_conn)
+ ksmbd_tree_connect_put(tree_conn);
+
return rc;
}
@@ -2213,17 +2230,6 @@ int smb2_tree_disconnect(struct ksmbd_work *work)
ksmbd_close_tree_conn_fds(work);
- write_lock(&sess->tree_conns_lock);
- if (tcon->t_state == TREE_DISCONNECTED) {
- write_unlock(&sess->tree_conns_lock);
- rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED;
- err = -ENOENT;
- goto err_out;
- }
-
- tcon->t_state = TREE_DISCONNECTED;
- write_unlock(&sess->tree_conns_lock);
-
err = ksmbd_tree_conn_disconnect(sess, tcon);
if (err) {
rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED;
@@ -2258,7 +2264,6 @@ int smb2_session_logoff(struct ksmbd_work *work)
struct ksmbd_session *sess = work->sess;
struct smb2_logoff_req *req;
struct smb2_logoff_rsp *rsp;
- u64 sess_id;
int err;
WORK_BUFFERS(work, req, rsp);
@@ -2272,8 +2277,7 @@ int smb2_session_logoff(struct ksmbd_work *work)
smb2_set_err_rsp(work);
return -ENOENT;
}
- sess_id = le64_to_cpu(req->hdr.SessionId);
- ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_RECONNECT);
+ ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_RECONNECT);
ksmbd_conn_unlock(conn);
ksmbd_close_session_fds(work);
@@ -2290,7 +2294,7 @@ int smb2_session_logoff(struct ksmbd_work *work)
sess->state = SMB2_SESSION_EXPIRED;
up_write(&conn->session_lock);
- ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_SETUP);
+ ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_SETUP);
rsp->StructureSize = cpu_to_le16(4);
err = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_logoff_rsp));
@@ -3265,6 +3269,7 @@ int smb2_open(struct ksmbd_work *work)
if (file_present && !(req->CreateOptions & FILE_DELETE_ON_CLOSE_LE)) {
rc = smb_check_perm_dacl(conn, &path, &daccess,
+ req->DesiredAccess,
sess->user->uid);
if (rc)
goto err_out;
@@ -3809,8 +3814,12 @@ int smb2_open(struct ksmbd_work *work)
rsp->hdr.Status = STATUS_INVALID_PARAMETER;
else if (rc == -EOPNOTSUPP)
rsp->hdr.Status = STATUS_NOT_SUPPORTED;
- else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV)
- rsp->hdr.Status = STATUS_ACCESS_DENIED;
+ else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV) {
+ if (req->DesiredAccess & FILE_ACCESS_SYSTEM_SECURITY_LE)
+ rsp->hdr.Status = STATUS_PRIVILEGE_NOT_HELD;
+ else
+ rsp->hdr.Status = STATUS_ACCESS_DENIED;
+ }
else if (rc == -ENOENT)
rsp->hdr.Status = STATUS_OBJECT_NAME_INVALID;
else if (rc == -EPERM)
@@ -7520,6 +7529,12 @@ static void smb2_remove_blocked_lock(void **argv)
locks_wake_up(flock);
}
+static void smb2_free_lock(struct file_lock *flock)
+{
+ ksmbd_vfs_posix_lock_unblock(flock);
+ locks_free_lock(flock);
+}
+
static inline bool lock_defer_pending(struct file_lock *fl)
{
/* check pending lock waiters */
@@ -7673,14 +7688,14 @@ int smb2_lock(struct ksmbd_work *work)
cmp_lock->end == smb_lock->end &&
!lock_defer_pending(cmp_lock->fl)) {
nolock = 0;
- list_del(&cmp_lock->flist);
- list_del(&cmp_lock->clist);
+ list_del_init(&cmp_lock->flist);
+ list_del_init(&cmp_lock->clist);
cmp_lock->conn = NULL;
spin_unlock(&conn->llist_lock);
up_read(&conn_list_lock);
ksmbd_conn_put(conn);
- locks_free_lock(cmp_lock->fl);
+ smb2_free_lock(cmp_lock->fl);
kfree(cmp_lock);
goto out_check_cl;
}
@@ -7762,11 +7777,12 @@ int smb2_lock(struct ksmbd_work *work)
ksmbd_debug(SMB,
"would have to wait for getting lock\n");
- list_add(&smb_lock->llist, &rollback_list);
argv = kmalloc(sizeof(void *), KSMBD_DEFAULT_GFP);
if (!argv) {
err = -ENOMEM;
+ smb2_free_lock(flock);
+ kfree(smb_lock);
goto out;
}
argv[0] = flock;
@@ -7777,8 +7793,11 @@ int smb2_lock(struct ksmbd_work *work)
if (rc) {
kfree(argv);
err = -ENOMEM;
+ smb2_free_lock(flock);
+ kfree(smb_lock);
goto out;
}
+ list_add(&smb_lock->llist, &rollback_list);
spin_lock(&fp->f_lock);
list_add(&work->fp_entry, &fp->blocked_works);
spin_unlock(&fp->f_lock);
@@ -7811,16 +7830,10 @@ int smb2_lock(struct ksmbd_work *work)
rsp->hdr.Status =
STATUS_RANGE_NOT_LOCKED;
kfree(smb_lock);
- goto out2;
+ /* rollback_list may still hold earlier grants */
+ goto out;
} else if (!rc) {
list_add(&smb_lock->llist, &rollback_list);
- smb_lock->conn = ksmbd_conn_get(work->conn);
- spin_lock(&work->conn->llist_lock);
- list_add_tail(&smb_lock->clist,
- &work->conn->lock_list);
- list_add_tail(&smb_lock->flist,
- &fp->lock_list);
- spin_unlock(&work->conn->llist_lock);
ksmbd_debug(SMB, "successful in taking lock\n");
} else {
locks_free_lock(flock);
@@ -7842,6 +7855,19 @@ int smb2_lock(struct ksmbd_work *work)
if (err)
goto out;
+ /* publish only once the whole batch has committed */
+ if (!list_empty(&rollback_list)) {
+ spin_lock(&work->conn->llist_lock);
+ list_for_each_entry(smb_lock, &rollback_list, llist) {
+ smb_lock->conn = ksmbd_conn_get(work->conn);
+ list_add_tail(&smb_lock->clist,
+ &work->conn->lock_list);
+ list_add_tail(&smb_lock->flist,
+ &fp->lock_list);
+ }
+ spin_unlock(&work->conn->llist_lock);
+ }
+
ksmbd_fd_put(work, fp);
return 0;
@@ -7869,16 +7895,7 @@ int smb2_lock(struct ksmbd_work *work)
}
list_del(&smb_lock->llist);
- conn = smb_lock->conn;
- spin_lock(&conn->llist_lock);
- if (!list_empty(&smb_lock->flist))
- list_del(&smb_lock->flist);
- list_del(&smb_lock->clist);
- smb_lock->conn = NULL;
- spin_unlock(&conn->llist_lock);
- ksmbd_conn_put(conn);
-
- locks_free_lock(smb_lock->fl);
+ smb2_free_lock(smb_lock->fl);
if (rlock)
locks_free_lock(rlock);
kfree(smb_lock);
@@ -8815,7 +8832,10 @@ static void smb20_oplock_break_ack(struct ksmbd_work *work)
smb2_set_err_rsp(work);
out:
- opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING)
+ opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
wake_up_interruptible_all(&opinfo->oplock_q);
opinfo_put(opinfo);
ksmbd_fd_put(work, fp);
@@ -8901,9 +8921,12 @@ static void smb21_lease_break_ack(struct ksmbd_work *work)
if (ret)
goto err_out;
- opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING)
+ opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
wake_up_interruptible_all(&opinfo->oplock_q);
- atomic_dec(&opinfo->breaking_cnt);
+ atomic_dec_if_positive(&opinfo->breaking_cnt);
wake_up_interruptible_all(&opinfo->oplock_brk);
opinfo_put(opinfo);
return;
diff --git a/fs/smb/server/smbacl.c b/fs/smb/server/smbacl.c
index 2a46646b3a0e..a4447c9c660d 100644
--- a/fs/smb/server/smbacl.c
+++ b/fs/smb/server/smbacl.c
@@ -1445,7 +1445,7 @@ bool smb_inherit_flags(int flags, bool is_dir)
}
int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
- __le32 *pdaccess, int uid)
+ __le32 *pdaccess, __le32 raw_daccess, int uid)
{
struct mnt_idmap *idmap = mnt_idmap(path->mnt);
struct smb_ntsd *pntsd = NULL;
@@ -1455,10 +1455,11 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
unsigned int dacl_offset;
size_t dacl_struct_end;
struct smb_sid sid;
- int granted = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE);
+ int requested = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE);
+ int granted = requested;
struct smb_ace *ace;
int i, found = 0;
- unsigned int access_bits = 0;
+ unsigned int access_bits = 0, denied = 0;
struct smb_ace *others_ace = NULL;
struct posix_acl_entry *pa_entry;
unsigned int sid_type = SIDOWNER;
@@ -1492,10 +1493,13 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
goto err_out;
}
+ if (!uid)
+ sid_type = SIDUNIX_USER;
+ id_to_sid(uid, sid_type, &sid);
+
if (*pdaccess & FILE_MAXIMAL_ACCESS_LE) {
- granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
+ access_bits = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
DELETE;
-
ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl));
aces_size = pdacl_size - sizeof(struct smb_acl);
for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) {
@@ -1508,18 +1512,41 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
CIFS_SID_BASE_SIZE)
break;
aces_size -= ace_size;
- granted |= le32_to_cpu(ace->access_req);
+
+ if (ace->sid.num_subauth > SID_MAX_SUB_AUTHORITIES ||
+ ace_size < offsetof(struct smb_ace, sid) +
+ CIFS_SID_BASE_SIZE +
+ sizeof(__le32) * ace->sid.num_subauth)
+ break;
+
+ if (ace->flags & INHERIT_ONLY_ACE ||
+ (compare_sids(&sid, &ace->sid) &&
+ compare_sids(&sid_unix_NFS_mode, &ace->sid) &&
+ compare_sids(&sid_everyone, &ace->sid) &&
+ compare_sids(&sid_authusers, &ace->sid)))
+ goto next_ace;
+
+ switch (ace->type) {
+ case ACCESS_ALLOWED_ACE_TYPE:
+ access_bits |= le32_to_cpu(ace->access_req);
+ break;
+ case ACCESS_DENIED_ACE_TYPE:
+ case ACCESS_DENIED_CALLBACK_ACE_TYPE:
+ denied |= ~access_bits &
+ le32_to_cpu(ace->access_req);
+ break;
+ }
+next_ace:
ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size));
}
-
- if (!pdacl->num_aces)
- granted = GENERIC_ALL_FLAGS;
+ access_bits &= ~denied;
+ if ((raw_daccess & FILE_GENERIC_EXECUTE_LE) &&
+ S_ISREG(d_inode(path->dentry)->i_mode) &&
+ (access_bits & GENERIC_READ_FLAGS) == GENERIC_READ_FLAGS)
+ access_bits |= FILE_EXECUTE;
+ granted = requested | access_bits;
}
- if (!uid)
- sid_type = SIDUNIX_USER;
- id_to_sid(uid, sid_type, &sid);
-
ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl));
aces_size = pdacl_size - sizeof(struct smb_acl);
for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) {
@@ -1543,25 +1570,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
found = 1;
break;
}
- if (!compare_sids(&sid_everyone, &ace->sid))
+ if (!compare_sids(&sid_everyone, &ace->sid) ||
+ !compare_sids(&sid_authusers, &ace->sid))
others_ace = ace;
ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size));
}
- if (*pdaccess & FILE_MAXIMAL_ACCESS_LE && found) {
- granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
- DELETE;
-
- granted |= le32_to_cpu(ace->access_req);
-
- if (!pdacl->num_aces)
- granted = GENERIC_ALL_FLAGS;
- }
-
if (IS_ENABLED(CONFIG_FS_POSIX_ACL)) {
posix_acls = get_inode_acl(d_inode(path->dentry), ACL_TYPE_ACCESS);
- if (!IS_ERR_OR_NULL(posix_acls) && !found) {
+ if (!IS_ERR_OR_NULL(posix_acls) && !found && !others_ace) {
unsigned int id = -1;
pa_entry = posix_acls->a_entries;
@@ -1599,14 +1617,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
}
}
- switch (ace->type) {
- case ACCESS_ALLOWED_ACE_TYPE:
- access_bits = le32_to_cpu(ace->access_req);
- break;
- case ACCESS_DENIED_ACE_TYPE:
- case ACCESS_DENIED_CALLBACK_ACE_TYPE:
- access_bits = le32_to_cpu(~ace->access_req);
- break;
+ if (!(*pdaccess & FILE_MAXIMAL_ACCESS_LE)) {
+ switch (ace->type) {
+ case ACCESS_ALLOWED_ACE_TYPE:
+ access_bits = le32_to_cpu(ace->access_req);
+ break;
+ case ACCESS_DENIED_ACE_TYPE:
+ case ACCESS_DENIED_CALLBACK_ACE_TYPE:
+ access_bits = le32_to_cpu(~ace->access_req);
+ break;
+ }
}
check_access_bits:
diff --git a/fs/smb/server/smbacl.h b/fs/smb/server/smbacl.h
index ab21ba2cd4df..bf0312bfa040 100644
--- a/fs/smb/server/smbacl.h
+++ b/fs/smb/server/smbacl.h
@@ -95,7 +95,7 @@ bool smb_inherit_flags(int flags, bool is_dir);
int smb_inherit_dacl(struct ksmbd_conn *conn, const struct path *path,
unsigned int uid, unsigned int gid);
int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
- __le32 *pdaccess, int uid);
+ __le32 *pdaccess, __le32 raw_daccess, int uid);
int set_info_sec(struct ksmbd_conn *conn, struct ksmbd_tree_connect *tcon,
const struct path *path, struct smb_ntsd *pntsd, int ntsd_len,
bool type_check, bool get_write);
diff --git a/fs/smb/server/transport_rdma.c b/fs/smb/server/transport_rdma.c
index 3976009596d9..585617595727 100644
--- a/fs/smb/server/transport_rdma.c
+++ b/fs/smb/server/transport_rdma.c
@@ -1977,9 +1977,9 @@ static int smb_direct_prepare(struct ksmbd_transport *t)
goto out;
req = (struct smb_direct_negotiate_req *)recvmsg->packet;
- st->max_recv_size = min_t(int, st->max_recv_size,
+ st->max_recv_size = min_t(u32, st->max_recv_size,
le32_to_cpu(req->preferred_send_size));
- st->max_send_size = min_t(int, st->max_send_size,
+ st->max_send_size = min_t(u32, st->max_send_size,
le32_to_cpu(req->max_receive_size));
st->max_fragmented_send_size =
le32_to_cpu(req->max_fragmented_size);
diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c
index 7785bc10aff1..967f66997486 100644
--- a/fs/smb/server/vfs.c
+++ b/fs/smb/server/vfs.c
@@ -1245,6 +1245,8 @@ int ksmbd_vfs_kern_path_locked(struct ksmbd_work *work, char *name,
path);
if (!err)
return 0;
+ if (err != -ENOENT)
+ return err;
if (caseless) {
char *filepath;
diff --git a/fs/smb/server/vfs_cache.c b/fs/smb/server/vfs_cache.c
index ce6e9886df8b..970b8f185aac 100644
--- a/fs/smb/server/vfs_cache.c
+++ b/fs/smb/server/vfs_cache.c
@@ -388,8 +388,9 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp)
if (!IS_ERR_OR_NULL(filp))
fput(filp);
- /* because the reference count of fp is 0, it is guaranteed that
- * there are not accesses to fp->lock_list.
+ /*
+ * The zero fp reference count serializes access to fp->lock_list, but
+ * the VFS may still have blocked requests chained below these locks.
*/
list_for_each_entry_safe(smb_lock, tmp_lock, &fp->lock_list, flist) {
struct ksmbd_conn *conn = smb_lock->conn;
@@ -402,7 +403,8 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp)
ksmbd_conn_put(conn);
}
- list_del(&smb_lock->flist);
+ list_del_init(&smb_lock->flist);
+ ksmbd_vfs_posix_lock_unblock(smb_lock->fl);
locks_free_lock(smb_lock->fl);
kfree(smb_lock);
}
diff --git a/fs/sysfs/group.c b/fs/sysfs/group.c
index f2b7283e6fa0..e1b6c38285f7 100644
--- a/fs/sysfs/group.c
+++ b/fs/sysfs/group.c
@@ -495,17 +495,26 @@ int compat_only_sysfs_link_entry_to_kobj(struct kobject *kobj,
}
EXPORT_SYMBOL_GPL(compat_only_sysfs_link_entry_to_kobj);
-static int sysfs_group_attrs_change_owner(struct kernfs_node *grp_kn,
+static int sysfs_group_attrs_change_owner(struct kobject *kobj,
+ struct kernfs_node *grp_kn,
const struct attribute_group *grp,
struct iattr *newattrs)
{
struct kernfs_node *kn;
- int error;
+ int error, i;
+ umode_t mode;
if (grp->attrs) {
struct attribute *const *attr;
- for (attr = grp->attrs; *attr; attr++) {
+ for (i = 0, attr = grp->attrs; *attr; i++, attr++) {
+ if (grp->is_visible) {
+ mode = grp->is_visible(kobj, *attr, i);
+ if (mode & SYSFS_GROUP_INVISIBLE)
+ break;
+ if (!mode)
+ continue;
+ }
kn = kernfs_find_and_get(grp_kn, (*attr)->name);
if (!kn)
return -ENOENT;
@@ -520,7 +529,14 @@ static int sysfs_group_attrs_change_owner(struct kernfs_node *grp_kn,
if (grp->bin_attrs) {
struct bin_attribute *const *bin_attr;
- for (bin_attr = grp->bin_attrs; *bin_attr; bin_attr++) {
+ for (i = 0, bin_attr = grp->bin_attrs; *bin_attr; i++, bin_attr++) {
+ if (grp->is_bin_visible) {
+ mode = grp->is_bin_visible(kobj, *bin_attr, i);
+ if (mode & SYSFS_GROUP_INVISIBLE)
+ break;
+ if (!mode)
+ continue;
+ }
kn = kernfs_find_and_get(grp_kn, (*bin_attr)->attr.name);
if (!kn)
return -ENOENT;
@@ -570,7 +586,7 @@ int sysfs_group_change_owner(struct kobject *kobj,
error = kernfs_setattr(grp_kn, &newattrs);
if (!error)
- error = sysfs_group_attrs_change_owner(grp_kn, grp, &newattrs);
+ error = sysfs_group_attrs_change_owner(kobj, grp_kn, grp, &newattrs);
kernfs_put(grp_kn);
diff --git a/fs/tracefs/event_inode.c b/fs/tracefs/event_inode.c
index a7b082a665bd..abb0ea57601b 100644
--- a/fs/tracefs/event_inode.c
+++ b/fs/tracefs/event_inode.c
@@ -124,7 +124,17 @@ static inline void put_ei(struct eventfs_inode *ei)
static inline void free_ei(struct eventfs_inode *ei)
{
if (ei) {
+ /* The ei should have no children if it is being freed. */
+ WARN_ON_ONCE(!list_empty(&ei->children));
ei->is_freed = 1;
+ /*
+ * The SRCU iteration has a smp_rmb() to make sure it
+ * sees a child (that may have already been freed)
+ * before it reads is_free. If is_free is set, it must
+ * not use the child it acquired from ei->children, as
+ * the list may be used for SRCU.
+ */
+ smp_wmb();
put_ei(ei);
}
}
@@ -649,6 +659,20 @@ static int eventfs_iterate(struct file *file, struct dir_context *ctx)
list_for_each_entry_srcu(ei_child, &ei->children, list,
srcu_read_lock_held(&eventfs_srcu)) {
+ /*
+ * If the ei is being freed, then the ei->children may be
+ * being used as the rcu list, which means the next element
+ * may be garbage. The ei->is_free is set before switching
+ * the ei->children over to ei->rcu. The read memory barrier
+ * here makes sure the ei_child is read before is_free is
+ * updated.
+ *
+ * Matches the smp_wmb() in free_ei()
+ */
+ smp_rmb();
+ if (ei->is_freed)
+ return -EINVAL;
+
if (c > 0) {
c--;
continue;
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index a4a7f8431aff..c61481d04c8e 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -46,11 +46,11 @@ struct eventfs_attr {
* @ino: The saved inode number
*/
struct eventfs_inode {
+ struct list_head list;
union {
- struct list_head list;
+ struct list_head children;
struct rcu_head rcu;
};
- struct list_head children;
const struct eventfs_entry *entries;
const char *name;
struct eventfs_attr *entry_attrs;
diff --git a/fs/xfs/libxfs/xfs_alloc.c b/fs/xfs/libxfs/xfs_alloc.c
index 3633f4d28c3b..c9a211b7564b 100644
--- a/fs/xfs/libxfs/xfs_alloc.c
+++ b/fs/xfs/libxfs/xfs_alloc.c
@@ -3435,16 +3435,41 @@ xfs_alloc_read_agf(
set_bit(XFS_AGSTATE_AGF_INIT, &pag->pag_opstate);
}
+
#ifdef DEBUG
- else if (!xfs_is_shutdown(pag->pag_mount)) {
- ASSERT(pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks));
- ASSERT(pag->pagf_btreeblks == be32_to_cpu(agf->agf_btreeblks));
- ASSERT(pag->pagf_flcount == be32_to_cpu(agf->agf_flcount));
- ASSERT(pag->pagf_longest == be32_to_cpu(agf->agf_longest));
- ASSERT(pag->pagf_bno_level == be32_to_cpu(agf->agf_bno_level));
- ASSERT(pag->pagf_cnt_level == be32_to_cpu(agf->agf_cnt_level));
+ /*
+ * It's possible for the AGF to be out of sync if the block device is
+ * silently dropping writes. This can happen in fstests with dmflakey
+ * enabled, which allows the buffer to be cleaned and reclaimed by
+ * memory pressure and then re-read from disk here. We will get a
+ * stale version of the AGF from disk, and nothing good can happen from
+ * here. Hence if we detect this situation, immediately shut down the
+ * filesystem.
+ *
+ * This can also happen if we are already in the middle of a forced
+ * shutdown, so don't bother checking if we are already shut down.
+ */
+ if (!xfs_is_shutdown(pag->pag_mount)) {
+ bool ok = true;
+
+ ok &= pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks);
+ ok &= pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks);
+ ok &= pag->pagf_btreeblks == be32_to_cpu(agf->agf_btreeblks);
+ ok &= pag->pagf_flcount == be32_to_cpu(agf->agf_flcount);
+ ok &= pag->pagf_longest == be32_to_cpu(agf->agf_longest);
+ ok &= pag->pagf_bno_level == be32_to_cpu(agf->agf_bno_level);
+ ok &= pag->pagf_cnt_level == be32_to_cpu(agf->agf_cnt_level);
+
+ if (XFS_IS_CORRUPT(pag->pag_mount, !ok)) {
+ xfs_ag_mark_sick(pag, XFS_SICK_AG_AGF);
+ xfs_trans_brelse(tp, agfbp);
+ xfs_force_shutdown(pag->pag_mount,
+ SHUTDOWN_CORRUPT_ONDISK);
+ return -EFSCORRUPTED;
+ }
}
-#endif
+#endif /* DEBUG */
+
if (agfbpp)
*agfbpp = agfbp;
else
diff --git a/fs/xfs/libxfs/xfs_da_btree.c b/fs/xfs/libxfs/xfs_da_btree.c
index 90fbdef23def..ca5f99c0878f 100644
--- a/fs/xfs/libxfs/xfs_da_btree.c
+++ b/fs/xfs/libxfs/xfs_da_btree.c
@@ -1506,21 +1506,20 @@ xfs_da3_fixhashpath(
}
/*
- * Remove an entry from an intermediate node.
+ * Internal implementation to remove an entry from an intermediate node.
*/
STATIC void
-xfs_da3_node_remove(
- struct xfs_da_state *state,
- struct xfs_da_state_blk *drop_blk)
+__xfs_da3_node_remove(
+ struct xfs_trans *tp,
+ struct xfs_inode *dp,
+ struct xfs_da_geometry *geo,
+ struct xfs_da_state_blk *drop_blk)
{
struct xfs_da_intnode *node;
struct xfs_da3_icnode_hdr nodehdr;
struct xfs_da_node_entry *btree;
int index;
int tmp;
- struct xfs_inode *dp = state->args->dp;
-
- trace_xfs_da_node_remove(state->args);
node = drop_blk->bp->b_addr;
xfs_da3_node_hdr_from_disk(dp->i_mount, &nodehdr, node);
@@ -1536,17 +1535,17 @@ xfs_da3_node_remove(
tmp = nodehdr.count - index - 1;
tmp *= (uint)sizeof(xfs_da_node_entry_t);
memmove(&btree[index], &btree[index + 1], tmp);
- xfs_trans_log_buf(state->args->trans, drop_blk->bp,
+ xfs_trans_log_buf(tp, drop_blk->bp,
XFS_DA_LOGRANGE(node, &btree[index], tmp));
index = nodehdr.count - 1;
}
memset(&btree[index], 0, sizeof(xfs_da_node_entry_t));
- xfs_trans_log_buf(state->args->trans, drop_blk->bp,
+ xfs_trans_log_buf(tp, drop_blk->bp,
XFS_DA_LOGRANGE(node, &btree[index], sizeof(btree[index])));
nodehdr.count -= 1;
xfs_da3_node_hdr_to_disk(dp->i_mount, node, &nodehdr);
- xfs_trans_log_buf(state->args->trans, drop_blk->bp,
- XFS_DA_LOGRANGE(node, &node->hdr, state->args->geo->node_hdr_size));
+ xfs_trans_log_buf(tp, drop_blk->bp,
+ XFS_DA_LOGRANGE(node, &node->hdr, geo->node_hdr_size));
/*
* Copy the last hash value from the block to propagate upwards.
@@ -1554,6 +1553,38 @@ xfs_da3_node_remove(
drop_blk->hashval = be32_to_cpu(btree[index - 1].hashval);
}
+/*
+ * Remove an entry from an intermediate node.
+ */
+STATIC void
+xfs_da3_node_remove(
+ struct xfs_da_state *state,
+ struct xfs_da_state_blk *drop_blk)
+{
+ trace_xfs_da_node_remove(state->args);
+
+ __xfs_da3_node_remove(state->args->trans, state->args->dp,
+ state->args->geo, drop_blk);
+}
+
+/*
+ * Remove an entry from an intermediate attr node at the specified index.
+ */
+void
+xfs_attr3_node_entry_remove(
+ struct xfs_trans *tp,
+ struct xfs_inode *dp,
+ struct xfs_buf *bp,
+ int index)
+{
+ struct xfs_da_state_blk blk = {
+ .index = index,
+ .bp = bp,
+ };
+
+ __xfs_da3_node_remove(tp, dp, dp->i_mount->m_attr_geo, &blk);
+}
+
/*
* Unbalance the elements between two intermediate nodes,
* move all Btree elements from one node into another.
diff --git a/fs/xfs/libxfs/xfs_da_btree.h b/fs/xfs/libxfs/xfs_da_btree.h
index cd2e213f5e74..a718b1ceb0aa 100644
--- a/fs/xfs/libxfs/xfs_da_btree.h
+++ b/fs/xfs/libxfs/xfs_da_btree.h
@@ -184,6 +184,8 @@ int xfs_da3_split(xfs_da_state_t *state);
int xfs_da3_join(xfs_da_state_t *state);
void xfs_da3_fixhashpath(struct xfs_da_state *state,
struct xfs_da_state_path *path_to_to_fix);
+void xfs_attr3_node_entry_remove(struct xfs_trans *tp, struct xfs_inode *dp,
+ struct xfs_buf *bp, int index);
/*
* Routines used for finding things in the Btree.
diff --git a/fs/xfs/libxfs/xfs_exchmaps.c b/fs/xfs/libxfs/xfs_exchmaps.c
index 68b9ddf07501..d900aff4fbea 100644
--- a/fs/xfs/libxfs/xfs_exchmaps.c
+++ b/fs/xfs/libxfs/xfs_exchmaps.c
@@ -49,7 +49,7 @@ struct xfs_exchmaps_adjacent {
.right2 = { .br_startblock = HOLESTARTBLOCK }, \
}
-/* Information to reset reflink flag / CoW fork state after an exchange. */
+/* Information to reset CoW fork state after an exchange. */
/*
* If the reflink flag is set on either inode, make sure it has an incore CoW
@@ -127,9 +127,7 @@ xmi_has_more_exchange_work(const struct xfs_exchmaps_intent *xmi)
static inline bool
xmi_has_postop_work(const struct xfs_exchmaps_intent *xmi)
{
- return xmi->xmi_flags & (XFS_EXCHMAPS_CLEAR_INO1_REFLINK |
- XFS_EXCHMAPS_CLEAR_INO2_REFLINK |
- __XFS_EXCHMAPS_INO2_SHORTFORM);
+ return xmi->xmi_flags & __XFS_EXCHMAPS_INO2_SHORTFORM;
}
/* Check all mappings to make sure we can actually exchange them. */
@@ -525,18 +523,6 @@ xfs_exchmaps_link_to_sf(
return error;
}
-/* Clear the reflink flag after an exchange. */
-static inline void
-xfs_exchmaps_clear_reflink(
- struct xfs_trans *tp,
- struct xfs_inode *ip)
-{
- trace_xfs_reflink_unset_inode_flag(ip);
-
- ip->i_diflags2 &= ~XFS_DIFLAG2_REFLINK;
- xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
-}
-
/* Finish whatever work might come after an exchange operation. */
static int
xfs_exchmaps_do_postop_work(
@@ -557,16 +543,6 @@ xfs_exchmaps_do_postop_work(
return error;
}
- if (xmi->xmi_flags & XFS_EXCHMAPS_CLEAR_INO1_REFLINK) {
- xfs_exchmaps_clear_reflink(tp, xmi->xmi_ip1);
- xmi->xmi_flags &= ~XFS_EXCHMAPS_CLEAR_INO1_REFLINK;
- }
-
- if (xmi->xmi_flags & XFS_EXCHMAPS_CLEAR_INO2_REFLINK) {
- xfs_exchmaps_clear_reflink(tp, xmi->xmi_ip2);
- xmi->xmi_flags &= ~XFS_EXCHMAPS_CLEAR_INO2_REFLINK;
- }
-
return 0;
}
@@ -946,56 +922,14 @@ xfs_exchmaps_intent_destroy_cache(void)
}
/*
- * Decide if we will exchange the reflink flags between the two files after the
- * exchange. The only time we want to do this is if we're exchanging all
- * mappings under EOF and the inode reflink flags have different states.
+ * Allocate and initialize a new incore intent item from a request.
*/
-static inline bool
-xmi_can_exchange_reflink_flags(
- const struct xfs_exchmaps_req *req,
- unsigned int reflink_state)
-{
- struct xfs_mount *mp = req->ip1->i_mount;
-
- /*
- * The INO1_WRITTEN optimization can skip exchanging hole and
- * unwritten mappings, which means we cannot guarantee that all
- * shared extents actually moved to the other file. Clearing the
- * reflink flag of an inode that still holds shared extents breaks
- * the CoW write path, so refuse to exchange the flags in that case.
- */
- if (req->flags & XFS_EXCHMAPS_INO1_WRITTEN)
- return false;
-
- /*
- * The INO1_WRITTEN optimization can skip exchanging hole and
- * unwritten mappings, which means we cannot guarantee that all
- * shared extents actually moved to the other file. Clearing the
- * reflink flag of an inode that still holds shared extents breaks
- * the CoW write path, so refuse to exchange the flags in that case.
- */
- if (req->flags & XFS_EXCHMAPS_INO1_WRITTEN)
- return false;
-
- if (hweight32(reflink_state) != 1)
- return false;
- if (req->startoff1 != 0 || req->startoff2 != 0)
- return false;
- if (req->blockcount != XFS_B_TO_FSB(mp, req->ip1->i_disk_size))
- return false;
- if (req->blockcount != XFS_B_TO_FSB(mp, req->ip2->i_disk_size))
- return false;
- return true;
-}
-
-
-/* Allocate and initialize a new incore intent item from a request. */
struct xfs_exchmaps_intent *
xfs_exchmaps_init_intent(
const struct xfs_exchmaps_req *req)
{
struct xfs_exchmaps_intent *xmi;
- unsigned int rs = 0;
+ struct xfs_mount *mp = req->ip1->i_mount;
xmi = kmem_cache_zalloc(xfs_exchmaps_intent_cache,
GFP_NOFS | __GFP_NOFAIL);
@@ -1013,27 +947,18 @@ xfs_exchmaps_init_intent(
return xmi;
}
- if (req->flags & XFS_EXCHMAPS_SET_SIZES) {
- xmi->xmi_flags |= XFS_EXCHMAPS_SET_SIZES;
- xmi->xmi_isize1 = req->ip2->i_disk_size;
- xmi->xmi_isize2 = req->ip1->i_disk_size;
- }
-
- /* Record the state of each inode's reflink flag before the op. */
- if (xfs_is_reflink_inode(req->ip1))
- rs |= 1;
- if (xfs_is_reflink_inode(req->ip2))
- rs |= 2;
-
/*
- * Figure out if we're clearing the reflink flags (which effectively
- * exchanges them) after the operation.
+ * If the caller wanted, set each file's size to that file's exchange
+ * offset + length exchanged from the other file because the ranges in
+ * each file might be different lengths.
*/
- if (xmi_can_exchange_reflink_flags(req, rs)) {
- if (rs & 1)
- xmi->xmi_flags |= XFS_EXCHMAPS_CLEAR_INO1_REFLINK;
- if (rs & 2)
- xmi->xmi_flags |= XFS_EXCHMAPS_CLEAR_INO2_REFLINK;
+ if (req->flags & XFS_EXCHMAPS_SET_SIZES) {
+ loff_t off1 = XFS_FSB_TO_B(mp, xmi->xmi_startoff1);
+ loff_t off2 = XFS_FSB_TO_B(mp, xmi->xmi_startoff2);
+
+ xmi->xmi_flags |= XFS_EXCHMAPS_SET_SIZES;
+ xmi->xmi_isize1 = off1 + (req->ip2->i_disk_size - off2);
+ xmi->xmi_isize2 = off2 + (req->ip1->i_disk_size - off1);
}
if (S_ISDIR(VFS_I(xmi->xmi_ip2)->i_mode) ||
diff --git a/fs/xfs/libxfs/xfs_ialloc.c b/fs/xfs/libxfs/xfs_ialloc.c
index 0fdb2a30e50d..482540a7e8e8 100644
--- a/fs/xfs/libxfs/xfs_ialloc.c
+++ b/fs/xfs/libxfs/xfs_ialloc.c
@@ -2785,12 +2785,35 @@ xfs_ialloc_read_agi(
set_bit(XFS_AGSTATE_AGI_INIT, &pag->pag_opstate);
}
+#ifdef DEBUG
/*
- * It's possible for these to be out of sync if
- * we are in the middle of a forced shutdown.
+ * It's possible for the AGF to be out of sync if the block device is
+ * silently dropping writes. This can happen in fstests with dmflakey
+ * enabled, which allows the buffer to be cleaned and reclaimed by
+ * memory pressure and then re-read from disk here. We will get a
+ * stale version of the AGF from disk, and nothing good can happen from
+ * here. Hence if we detect this situation, immediately shut down the
+ * filesystem.
+ *
+ * This can also happen if we are already in the middle of a forced
+ * shutdown, so don't bother checking if we are already shut down.
*/
- ASSERT(pag->pagi_freecount == be32_to_cpu(agi->agi_freecount) ||
- xfs_is_shutdown(pag->pag_mount));
+ if (!xfs_is_shutdown(pag->pag_mount)) {
+ bool ok = true;
+
+ ok &= pag->pagi_freecount == be32_to_cpu(agi->agi_freecount);
+ ok &= pag->pagi_count == be32_to_cpu(agi->agi_count);
+
+ if (XFS_IS_CORRUPT(pag->pag_mount, !ok)) {
+ xfs_ag_mark_sick(pag, XFS_SICK_AG_AGI);
+ xfs_trans_brelse(tp, agibp);
+ xfs_force_shutdown(pag->pag_mount,
+ SHUTDOWN_CORRUPT_ONDISK);
+ return -EFSCORRUPTED;
+ }
+ }
+#endif /* DEBUG */
+
if (agibpp)
*agibpp = agibp;
else
diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h
index a151ea6c0f72..114286c37c3d 100644
--- a/fs/xfs/libxfs/xfs_log_format.h
+++ b/fs/xfs/libxfs/xfs_log_format.h
@@ -948,10 +948,8 @@ struct xfs_xmi_log_format {
*/
#define XFS_EXCHMAPS_INO1_WRITTEN (1ULL << 2)
-/* Clear the reflink flag from inode1 after the operation. */
+/* No longer set, but older intents may carry them; keep them valid. */
#define XFS_EXCHMAPS_CLEAR_INO1_REFLINK (1ULL << 3)
-
-/* Clear the reflink flag from inode2 after the operation. */
#define XFS_EXCHMAPS_CLEAR_INO2_REFLINK (1ULL << 4)
#define XFS_EXCHMAPS_LOGGED_FLAGS (XFS_EXCHMAPS_ATTR_FORK | \
diff --git a/fs/xfs/xfs_attr_inactive.c b/fs/xfs/xfs_attr_inactive.c
index 24fb12986a56..2305ddad8a13 100644
--- a/fs/xfs/xfs_attr_inactive.c
+++ b/fs/xfs/xfs_attr_inactive.c
@@ -140,7 +140,7 @@ xfs_attr3_node_inactive(
xfs_daddr_t parent_blkno, child_blkno;
struct xfs_buf *child_bp;
struct xfs_da3_icnode_hdr ichdr;
- int error, i;
+ int error;
/*
* Since this code is recursive (gasp!) we must protect ourselves.
@@ -152,7 +152,7 @@ xfs_attr3_node_inactive(
return -EFSCORRUPTED;
}
- xfs_da3_node_hdr_from_disk(dp->i_mount, &ichdr, bp->b_addr);
+ xfs_da3_node_hdr_from_disk(mp, &ichdr, bp->b_addr);
parent_blkno = xfs_buf_daddr(bp);
if (!ichdr.count) {
xfs_trans_brelse(*trans, bp);
@@ -167,7 +167,7 @@ xfs_attr3_node_inactive(
* over the leaves removing all of them. If this is higher up
* in the tree, recurse downward.
*/
- for (i = 0; i < ichdr.count; i++) {
+ while (ichdr.count > 0) {
/*
* Read the subsidiary block to see what we have to work with.
* Don't do this in a transaction. This is a depth-first
@@ -218,29 +218,32 @@ xfs_attr3_node_inactive(
xfs_trans_binval(*trans, child_bp);
child_bp = NULL;
+ error = xfs_da3_node_read_mapped(*trans, dp,
+ parent_blkno, &bp, XFS_ATTR_FORK);
+ if (error)
+ return error;
+
/*
- * If we're not done, re-read the parent to get the next
- * child block number.
+ * Remove entry from parent node, prevents being indexed to.
*/
- if (i + 1 < ichdr.count) {
- struct xfs_da3_icnode_hdr phdr;
+ xfs_attr3_node_entry_remove(*trans, dp, bp, 0);
+
+ xfs_da3_node_hdr_from_disk(mp, &ichdr, bp->b_addr);
+ bp = NULL;
- error = xfs_da3_node_read_mapped(*trans, dp,
- parent_blkno, &bp, XFS_ATTR_FORK);
+ if (ichdr.count > 0) {
+ /*
+ * If we're not done, get the next child block number.
+ */
+ child_fsb = be32_to_cpu(ichdr.btree[0].before);
+
+ /*
+ * Atomically commit the whole invalidate stuff.
+ */
+ error = xfs_trans_roll_inode(trans, dp);
if (error)
return error;
- xfs_da3_node_hdr_from_disk(dp->i_mount, &phdr,
- bp->b_addr);
- child_fsb = be32_to_cpu(phdr.btree[i + 1].before);
- xfs_trans_brelse(*trans, bp);
- bp = NULL;
}
- /*
- * Atomically commit the whole invalidate stuff.
- */
- error = xfs_trans_roll_inode(trans, dp);
- if (error)
- return error;
}
return 0;
@@ -257,10 +260,8 @@ xfs_attr3_root_inactive(
struct xfs_trans **trans,
struct xfs_inode *dp)
{
- struct xfs_mount *mp = dp->i_mount;
struct xfs_da_blkinfo *info;
struct xfs_buf *bp;
- xfs_daddr_t blkno;
int error;
/*
@@ -272,7 +273,6 @@ xfs_attr3_root_inactive(
error = xfs_da3_node_read(*trans, dp, 0, &bp, XFS_ATTR_FORK);
if (error)
return error;
- blkno = xfs_buf_daddr(bp);
/*
* Invalidate the tree, even if the "tree" is only a single leaf block.
@@ -283,10 +283,26 @@ xfs_attr3_root_inactive(
case cpu_to_be16(XFS_DA_NODE_MAGIC):
case cpu_to_be16(XFS_DA3_NODE_MAGIC):
error = xfs_attr3_node_inactive(trans, dp, bp, 1);
+ /*
+ * Empty root node block are not allowed, convert it to leaf.
+ */
+ if (!error)
+ error = xfs_attr3_leaf_init(*trans, dp, 0);
+ if (!error)
+ error = xfs_trans_roll_inode(trans, dp);
break;
case cpu_to_be16(XFS_ATTR_LEAF_MAGIC):
case cpu_to_be16(XFS_ATTR3_LEAF_MAGIC):
error = xfs_attr3_leaf_inactive(trans, dp, bp);
+ /*
+ * Reinit the leaf before truncating extents so that a crash
+ * mid-truncation leaves an empty leaf rather than one with
+ * entries that may reference freed remote value blocks.
+ */
+ if (!error)
+ error = xfs_attr3_leaf_init(*trans, dp, 0);
+ if (!error)
+ error = xfs_trans_roll_inode(trans, dp);
break;
default:
xfs_dirattr_mark_sick(dp, XFS_ATTR_FORK);
@@ -295,26 +311,6 @@ xfs_attr3_root_inactive(
xfs_trans_brelse(*trans, bp);
break;
}
- if (error)
- return error;
-
- /*
- * Invalidate the incore copy of the root block.
- */
- error = xfs_trans_get_buf(*trans, mp->m_ddev_targp, blkno,
- XFS_FSB_TO_BB(mp, mp->m_attr_geo->fsbcount), 0, &bp);
- if (error)
- return error;
- error = bp->b_error;
- if (error) {
- xfs_trans_brelse(*trans, bp);
- return error;
- }
- xfs_trans_binval(*trans, bp); /* remove from cache */
- /*
- * Commit the invalidate and start the next transaction.
- */
- error = xfs_trans_roll_inode(trans, dp);
return error;
}
@@ -333,6 +329,7 @@ xfs_attr_inactive(
{
struct xfs_trans *trans;
struct xfs_mount *mp;
+ struct xfs_buf *bp;
int lock_mode = XFS_ILOCK_SHARED;
int error = 0;
@@ -368,10 +365,27 @@ xfs_attr_inactive(
* removal below.
*/
if (dp->i_af.if_nextents > 0) {
+ /*
+ * Invalidate and truncate all blocks but leave the root block.
+ */
error = xfs_attr3_root_inactive(&trans, dp);
if (error)
goto out_cancel;
+ error = xfs_itruncate_extents(&trans, dp, XFS_ATTR_FORK,
+ XFS_FSB_TO_B(mp, mp->m_attr_geo->fsbcount));
+ if (error)
+ goto out_cancel;
+
+ /*
+ * Invalidate and truncate the root block and ensure that the
+ * operation is completed within a single transaction.
+ */
+ error = xfs_da_get_buf(trans, dp, 0, &bp, XFS_ATTR_FORK);
+ if (error)
+ goto out_cancel;
+
+ xfs_trans_binval(trans, bp);
error = xfs_itruncate_extents(&trans, dp, XFS_ATTR_FORK, 0);
if (error)
goto out_cancel;
diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c
index 95f8a09f96ae..97ff767da598 100644
--- a/fs/xfs/xfs_bmap_util.c
+++ b/fs/xfs/xfs_bmap_util.c
@@ -1504,7 +1504,6 @@ xfs_swap_extents(
struct xfs_bstat *sbp = &sxp->sx_stat;
int src_log_flags, target_log_flags;
int error = 0;
- uint64_t f;
int resblks = 0;
unsigned int flags = 0;
struct timespec64 ctime, mtime;
@@ -1648,14 +1647,12 @@ xfs_swap_extents(
if (error)
goto out_trans_cancel;
- /* Do we have to swap reflink flags? */
- if ((ip->i_diflags2 & XFS_DIFLAG2_REFLINK) ^
- (tip->i_diflags2 & XFS_DIFLAG2_REFLINK)) {
- f = ip->i_diflags2 & XFS_DIFLAG2_REFLINK;
- ip->i_diflags2 &= ~XFS_DIFLAG2_REFLINK;
- ip->i_diflags2 |= tip->i_diflags2 & XFS_DIFLAG2_REFLINK;
- tip->i_diflags2 &= ~XFS_DIFLAG2_REFLINK;
- tip->i_diflags2 |= f & XFS_DIFLAG2_REFLINK;
+ /* Either inode may now own shared blocks; flag both, with CoW forks. */
+ if ((ip->i_diflags2 | tip->i_diflags2) & XFS_DIFLAG2_REFLINK) {
+ ip->i_diflags2 |= XFS_DIFLAG2_REFLINK;
+ tip->i_diflags2 |= XFS_DIFLAG2_REFLINK;
+ xfs_ifork_init_cow(ip);
+ xfs_ifork_init_cow(tip);
}
/* Swap the cow forks. */
diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c
index 1f6b2888bd41..571cf8256672 100644
--- a/fs/xfs/xfs_buf.c
+++ b/fs/xfs/xfs_buf.c
@@ -319,12 +319,14 @@ xfs_buf_free(
ASSERT(list_empty(&bp->b_lru));
- if (xfs_buftarg_is_mem(bp->b_target))
- xmbuf_unmap_page(bp);
- else if (bp->b_flags & _XBF_PAGES)
+ if (xfs_buftarg_is_mem(bp->b_target)) {
+ if (bp->b_addr)
+ xmbuf_unmap_page(bp);
+ } else if (bp->b_flags & _XBF_PAGES) {
xfs_buf_free_pages(bp);
- else if (bp->b_flags & _XBF_KMEM)
+ } else if (bp->b_flags & _XBF_KMEM) {
kfree(bp->b_addr);
+ }
call_rcu(&bp->b_rcu, xfs_buf_free_callback);
}
@@ -2389,61 +2391,6 @@ xfs_buf_delwri_submit(
return error;
}
-/*
- * Push a single buffer on a delwri queue.
- *
- * The purpose of this function is to submit a single buffer of a delwri queue
- * and return with the buffer still on the original queue. The waiting delwri
- * buffer submission infrastructure guarantees transfer of the delwri queue
- * buffer reference to a temporary wait list. We reuse this infrastructure to
- * transfer the buffer back to the original queue.
- *
- * Note the buffer transitions from the queued state, to the submitted and wait
- * listed state and back to the queued state during this call. The buffer
- * locking and queue management logic between _delwri_pushbuf() and
- * _delwri_queue() guarantee that the buffer cannot be queued to another list
- * before returning.
- */
-int
-xfs_buf_delwri_pushbuf(
- struct xfs_buf *bp,
- struct list_head *buffer_list)
-{
- LIST_HEAD (submit_list);
- int error;
-
- ASSERT(bp->b_flags & _XBF_DELWRI_Q);
-
- trace_xfs_buf_delwri_pushbuf(bp, _RET_IP_);
-
- /*
- * Isolate the buffer to a new local list so we can submit it for I/O
- * independently from the rest of the original list.
- */
- xfs_buf_lock(bp);
- list_move(&bp->b_list, &submit_list);
- xfs_buf_unlock(bp);
-
- /*
- * Delwri submission clears the DELWRI_Q buffer flag and returns with
- * the buffer on the wait list with the original reference. Rather than
- * bounce the buffer from a local wait list back to the original list
- * after I/O completion, reuse the original list as the wait list.
- */
- xfs_buf_delwri_submit_buffers(&submit_list, buffer_list);
-
- /*
- * The buffer is now locked, under I/O and wait listed on the original
- * delwri queue. Wait for I/O completion, restore the DELWRI_Q flag and
- * return with the buffer unlocked and on the original queue.
- */
- error = xfs_buf_iowait(bp);
- bp->b_flags |= _XBF_DELWRI_Q;
- xfs_buf_unlock(bp);
-
- return error;
-}
-
void xfs_buf_set_ref(struct xfs_buf *bp, int lru_ref)
{
/*
diff --git a/fs/xfs/xfs_buf.h b/fs/xfs/xfs_buf.h
index 209a389f2abc..3fd9bc72f8b6 100644
--- a/fs/xfs/xfs_buf.h
+++ b/fs/xfs/xfs_buf.h
@@ -334,7 +334,6 @@ extern bool xfs_buf_delwri_queue(struct xfs_buf *, struct list_head *);
void xfs_buf_delwri_queue_here(struct xfs_buf *bp, struct list_head *bl);
extern int xfs_buf_delwri_submit(struct list_head *);
extern int xfs_buf_delwri_submit_nowait(struct list_head *);
-extern int xfs_buf_delwri_pushbuf(struct xfs_buf *, struct list_head *);
static inline xfs_daddr_t xfs_buf_daddr(struct xfs_buf *bp)
{
diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c
index b746a1a0f8dd..aa1108e81c22 100644
--- a/fs/xfs/xfs_buf_item.c
+++ b/fs/xfs/xfs_buf_item.c
@@ -32,6 +32,61 @@ static inline struct xfs_buf_log_item *BUF_ITEM(struct xfs_log_item *lip)
return container_of(lip, struct xfs_buf_log_item, bli_item);
}
+static void
+xfs_buf_item_get_format(
+ struct xfs_buf_log_item *bip,
+ int count)
+{
+ ASSERT(bip->bli_formats == NULL);
+ bip->bli_format_count = count;
+
+ if (count == 1) {
+ bip->bli_formats = &bip->__bli_format;
+ return;
+ }
+
+ bip->bli_formats = kzalloc(count * sizeof(struct xfs_buf_log_format),
+ GFP_KERNEL | __GFP_NOFAIL);
+}
+
+static void
+xfs_buf_item_free_format(
+ struct xfs_buf_log_item *bip)
+{
+ if (bip->bli_formats != &bip->__bli_format) {
+ kfree(bip->bli_formats);
+ bip->bli_formats = NULL;
+ }
+}
+
+static void
+xfs_buf_item_free(
+ struct xfs_buf_log_item *bip)
+{
+ xfs_buf_item_free_format(bip);
+ kvfree(bip->bli_item.li_lv_shadow);
+ kmem_cache_free(xfs_buf_item_cache, bip);
+}
+
+/*
+ * xfs_buf_item_relse() is called when the buf log item is no longer needed.
+ */
+static void
+xfs_buf_item_relse(
+ struct xfs_buf_log_item *bip)
+{
+ struct xfs_buf *bp = bip->bli_buf;
+
+ trace_xfs_buf_item_relse(bp, _RET_IP_);
+
+ ASSERT(!test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags));
+ ASSERT(atomic_read(&bip->bli_refcount) == 0);
+
+ bp->b_log_item = NULL;
+ xfs_buf_rele(bp);
+ xfs_buf_item_free(bip);
+}
+
/* Is this log iovec plausibly large enough to contain the buffer log format? */
bool
xfs_buf_log_check_iovec(
@@ -484,6 +539,42 @@ xfs_buf_item_pin(
atomic_inc(&bip->bli_buf->b_pin_count);
}
+/*
+ * For a stale BLI, process all the necessary completions that must be
+ * performed when the final BLI reference goes away. The buffer will be
+ * referenced and locked here - we return to the caller with the buffer still
+ * referenced and locked for them to finalise processing of the buffer.
+ */
+static void
+xfs_buf_item_finish_stale(
+ struct xfs_buf_log_item *bip)
+{
+ struct xfs_buf *bp = bip->bli_buf;
+ struct xfs_log_item *lip = &bip->bli_item;
+
+ ASSERT(bip->bli_flags & XFS_BLI_STALE);
+ ASSERT(xfs_buf_islocked(bp));
+ ASSERT(bp->b_flags & XBF_STALE);
+ ASSERT(bip->__bli_format.blf_flags & XFS_BLF_CANCEL);
+ ASSERT(list_empty(&lip->li_trans));
+ ASSERT(!bp->b_transp);
+
+ if (bip->bli_flags & XFS_BLI_STALE_INODE) {
+ xfs_buf_item_done(bp);
+ xfs_buf_inode_iodone(bp);
+ ASSERT(list_empty(&bp->b_li_list));
+ return;
+ }
+
+ /*
+ * We may or may not be on the AIL here, xfs_trans_ail_delete() will do
+ * the right thing regardless of the situation in which we are called.
+ */
+ xfs_trans_ail_delete(lip, SHUTDOWN_LOG_IO_ERROR);
+ xfs_buf_item_relse(bip);
+ ASSERT(bp->b_log_item == NULL);
+}
+
/*
* This is called to unpin the buffer associated with the buf log item which was
* previously pinned with a call to xfs_buf_item_pin(). We enter this function
@@ -533,13 +624,6 @@ xfs_buf_item_unpin(
}
if (stale) {
- ASSERT(bip->bli_flags & XFS_BLI_STALE);
- ASSERT(xfs_buf_islocked(bp));
- ASSERT(bp->b_flags & XBF_STALE);
- ASSERT(bip->__bli_format.blf_flags & XFS_BLF_CANCEL);
- ASSERT(list_empty(&lip->li_trans));
- ASSERT(!bp->b_transp);
-
trace_xfs_buf_item_unpin_stale(bip);
/*
@@ -550,22 +634,7 @@ xfs_buf_item_unpin(
* processing is complete.
*/
xfs_buf_rele(bp);
-
- /*
- * If we get called here because of an IO error, we may or may
- * not have the item on the AIL. xfs_trans_ail_delete() will
- * take care of that situation. xfs_trans_ail_delete() drops
- * the AIL lock.
- */
- if (bip->bli_flags & XFS_BLI_STALE_INODE) {
- xfs_buf_item_done(bp);
- xfs_buf_inode_iodone(bp);
- ASSERT(list_empty(&bp->b_li_list));
- } else {
- xfs_trans_ail_delete(lip, SHUTDOWN_LOG_IO_ERROR);
- xfs_buf_item_relse(bp);
- ASSERT(bp->b_log_item == NULL);
- }
+ xfs_buf_item_finish_stale(bip);
xfs_buf_relse(bp);
return;
}
@@ -638,43 +707,42 @@ xfs_buf_item_push(
* Drop the buffer log item refcount and take appropriate action. This helper
* determines whether the bli must be freed or not, since a decrement to zero
* does not necessarily mean the bli is unused.
- *
- * Return true if the bli is freed, false otherwise.
*/
-bool
+void
xfs_buf_item_put(
struct xfs_buf_log_item *bip)
{
- struct xfs_log_item *lip = &bip->bli_item;
- bool aborted;
- bool dirty;
+
+ ASSERT(xfs_buf_islocked(bip->bli_buf));
/* drop the bli ref and return if it wasn't the last one */
if (!atomic_dec_and_test(&bip->bli_refcount))
- return false;
+ return;
- /*
- * We dropped the last ref and must free the item if clean or aborted.
- * If the bli is dirty and non-aborted, the buffer was clean in the
- * transaction but still awaiting writeback from previous changes. In
- * that case, the bli is freed on buffer writeback completion.
- */
- aborted = test_bit(XFS_LI_ABORTED, &lip->li_flags) ||
- xlog_is_shutdown(lip->li_log);
- dirty = bip->bli_flags & XFS_BLI_DIRTY;
- if (dirty && !aborted)
- return false;
+ /* If the BLI is in the AIL, then it is still dirty and in use */
+ if (test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags)) {
+ ASSERT(bip->bli_flags & XFS_BLI_DIRTY);
+ return;
+ }
/*
- * The bli is aborted or clean. An aborted item may be in the AIL
- * regardless of dirty state. For example, consider an aborted
- * transaction that invalidated a dirty bli and cleared the dirty
- * state.
+ * In shutdown conditions, we can be asked to free a dirty BLI that
+ * isn't in the AIL. This can occur due to a checkpoint aborting a BLI
+ * instead of inserting it into the AIL at checkpoint IO completion. If
+ * there's another bli reference (e.g. a btree cursor holds a clean
+ * reference) and it is released via xfs_trans_brelse(), we can get here
+ * with that aborted, dirty BLI. In this case, it is safe to free the
+ * dirty BLI immediately, as it is not in the AIL and there are no
+ * other references to it.
+ *
+ * We should never get here with a stale BLI via that path as
+ * xfs_trans_brelse() specifically holds onto stale buffers rather than
+ * releasing them.
*/
- if (aborted)
- xfs_trans_ail_delete(lip, 0);
- xfs_buf_item_relse(bip->bli_buf);
- return true;
+ ASSERT(!(bip->bli_flags & XFS_BLI_DIRTY) ||
+ test_bit(XFS_LI_ABORTED, &bip->bli_item.li_flags));
+ ASSERT(!(bip->bli_flags & XFS_BLI_STALE));
+ xfs_buf_item_relse(bip);
}
/*
@@ -695,6 +763,15 @@ xfs_buf_item_put(
* if necessary but do not unlock the buffer. This is for support of
* xfs_trans_bhold(). Make sure the XFS_BLI_HOLD field is cleared if we don't
* free the item.
+ *
+ * If the XFS_BLI_STALE flag is set, the last reference to the BLI *must*
+ * perform a completion abort of any objects attached to the buffer for IO
+ * tracking purposes. This generally only happens in shutdown situations,
+ * normally xfs_buf_item_unpin() will drop the last BLI reference and perform
+ * completion processing. However, because transaction completion can race with
+ * checkpoint completion during a shutdown, this release context may end up
+ * being the last active reference to the BLI and so needs to perform this
+ * cleanup.
*/
STATIC void
xfs_buf_item_release(
@@ -702,18 +779,19 @@ xfs_buf_item_release(
{
struct xfs_buf_log_item *bip = BUF_ITEM(lip);
struct xfs_buf *bp = bip->bli_buf;
- bool released;
bool hold = bip->bli_flags & XFS_BLI_HOLD;
bool stale = bip->bli_flags & XFS_BLI_STALE;
-#if defined(DEBUG) || defined(XFS_WARN)
- bool ordered = bip->bli_flags & XFS_BLI_ORDERED;
- bool dirty = bip->bli_flags & XFS_BLI_DIRTY;
bool aborted = test_bit(XFS_LI_ABORTED,
&lip->li_flags);
+ bool dirty = bip->bli_flags & XFS_BLI_DIRTY;
+#if defined(DEBUG) || defined(XFS_WARN)
+ bool ordered = bip->bli_flags & XFS_BLI_ORDERED;
#endif
trace_xfs_buf_item_release(bip);
+ ASSERT(xfs_buf_islocked(bp));
+
/*
* The bli dirty state should match whether the blf has logged segments
* except for ordered buffers, where only the bli should be dirty.
@@ -729,16 +807,56 @@ xfs_buf_item_release(
bp->b_transp = NULL;
bip->bli_flags &= ~(XFS_BLI_LOGGED | XFS_BLI_HOLD | XFS_BLI_ORDERED);
+ /* If there are other references, then we have nothing to do. */
+ if (!atomic_dec_and_test(&bip->bli_refcount))
+ goto out_release;
+
/*
- * Unref the item and unlock the buffer unless held or stale. Stale
- * buffers remain locked until final unpin unless the bli is freed by
- * the unref call. The latter implies shutdown because buffer
- * invalidation dirties the bli and transaction.
+ * Stale buffer completion frees the BLI, unlocks and releases the
+ * buffer. Neither the BLI or buffer are safe to reference after this
+ * call, so there's nothing more we need to do here.
+ *
+ * If we get here with a stale buffer and references to the BLI remain,
+ * we must not unlock the buffer as the last BLI reference owns lock
+ * context, not us.
*/
- released = xfs_buf_item_put(bip);
- if (hold || (stale && !released))
+ if (stale) {
+ xfs_buf_item_finish_stale(bip);
+ xfs_buf_relse(bp);
+ ASSERT(!hold);
+ return;
+ }
+
+ /*
+ * Dirty or clean, aborted items are done and need to be removed from
+ * the AIL and released. This frees the BLI, but leaves the buffer
+ * locked and referenced.
+ */
+ if (aborted || xlog_is_shutdown(lip->li_log)) {
+ ASSERT(list_empty(&bip->bli_buf->b_li_list));
+ xfs_buf_item_done(bp);
+ goto out_release;
+ }
+
+ /*
+ * Clean, unreferenced BLIs can be immediately freed, leaving the buffer
+ * locked and referenced.
+ *
+ * Dirty, unreferenced BLIs *must* be in the AIL awaiting writeback.
+ */
+ if (!dirty)
+ xfs_buf_item_relse(bip);
+ else
+ ASSERT(test_bit(XFS_LI_IN_AIL, &lip->li_flags));
+
+ /* Not safe to reference the BLI from here */
+out_release:
+ /*
+ * If we get here with a stale buffer, we must not unlock the
+ * buffer as the last BLI reference owns lock context, not us.
+ */
+ if (stale || hold)
return;
- ASSERT(!stale || aborted);
xfs_buf_relse(bp);
}
@@ -824,33 +942,6 @@ static const struct xfs_item_ops xfs_buf_item_ops = {
.iop_push = xfs_buf_item_push,
};
-STATIC void
-xfs_buf_item_get_format(
- struct xfs_buf_log_item *bip,
- int count)
-{
- ASSERT(bip->bli_formats == NULL);
- bip->bli_format_count = count;
-
- if (count == 1) {
- bip->bli_formats = &bip->__bli_format;
- return;
- }
-
- bip->bli_formats = kzalloc(count * sizeof(struct xfs_buf_log_format),
- GFP_KERNEL | __GFP_NOFAIL);
-}
-
-STATIC void
-xfs_buf_item_free_format(
- struct xfs_buf_log_item *bip)
-{
- if (bip->bli_formats != &bip->__bli_format) {
- kfree(bip->bli_formats);
- bip->bli_formats = NULL;
- }
-}
-
/*
* Allocate a new buf log item to go with the given buffer.
* Set the buffer's b_log_item field to point to the new
@@ -1072,34 +1163,6 @@ xfs_buf_item_dirty_format(
return false;
}
-STATIC void
-xfs_buf_item_free(
- struct xfs_buf_log_item *bip)
-{
- xfs_buf_item_free_format(bip);
- kvfree(bip->bli_item.li_lv_shadow);
- kmem_cache_free(xfs_buf_item_cache, bip);
-}
-
-/*
- * xfs_buf_item_relse() is called when the buf log item is no longer needed.
- */
-void
-xfs_buf_item_relse(
- struct xfs_buf *bp)
-{
- struct xfs_buf_log_item *bip = bp->b_log_item;
-
- trace_xfs_buf_item_relse(bp, _RET_IP_);
- ASSERT(!test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags));
-
- if (atomic_read(&bip->bli_refcount))
- return;
- bp->b_log_item = NULL;
- xfs_buf_rele(bp);
- xfs_buf_item_free(bip);
-}
-
void
xfs_buf_item_done(
struct xfs_buf *bp)
@@ -1119,5 +1182,5 @@ xfs_buf_item_done(
xfs_trans_ail_delete(&bp->b_log_item->bli_item,
(bp->b_flags & _XBF_LOGRECOVERY) ? 0 :
SHUTDOWN_CORRUPT_INCORE);
- xfs_buf_item_relse(bp);
+ xfs_buf_item_relse(bp->b_log_item);
}
diff --git a/fs/xfs/xfs_buf_item.h b/fs/xfs/xfs_buf_item.h
index 7ddda0aa9dac..141c3780a36b 100644
--- a/fs/xfs/xfs_buf_item.h
+++ b/fs/xfs/xfs_buf_item.h
@@ -49,8 +49,7 @@ struct xfs_buf_log_item {
int xfs_buf_item_init(struct xfs_buf *, struct xfs_mount *);
void xfs_buf_item_done(struct xfs_buf *bp);
-void xfs_buf_item_relse(struct xfs_buf *);
-bool xfs_buf_item_put(struct xfs_buf_log_item *);
+void xfs_buf_item_put(struct xfs_buf_log_item *bip);
void xfs_buf_item_log(struct xfs_buf_log_item *, uint, uint);
bool xfs_buf_item_dirty_format(struct xfs_buf_log_item *);
void xfs_buf_inode_iodone(struct xfs_buf *);
diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c
index 511e7b0d5871..1dd4dde91736 100644
--- a/fs/xfs/xfs_dquot.c
+++ b/fs/xfs/xfs_dquot.c
@@ -1405,11 +1405,9 @@ xfs_qm_dqflush(
ASSERT(XFS_DQ_IS_LOCKED(dqp));
ASSERT(!completion_done(&dqp->q_flush));
+ ASSERT(atomic_read(&dqp->q_pincount) == 0);
trace_xfs_dqflush(dqp);
-
- xfs_qm_dqunpin_wait(dqp);
-
fa = xfs_qm_dqflush_check(dqp);
if (fa) {
xfs_alert(mp, "corrupt dquot ID 0x%x in memory at %pS",
diff --git a/fs/xfs/xfs_exchrange.c b/fs/xfs/xfs_exchrange.c
index f54ed30b76e4..449090cbb169 100644
--- a/fs/xfs/xfs_exchrange.c
+++ b/fs/xfs/xfs_exchrange.c
@@ -287,17 +287,20 @@ xfs_exchrange_mappings(
goto out_unlock;
/*
- * If the caller wanted us to exchange the contents of two complete
- * files of unequal length, exchange the incore sizes now. This should
- * be safe because we flushed both files' page caches, exchanged all
- * the mappings, and updated the ondisk sizes.
+ * If the caller wanted, set each file's size to that file's exchange
+ * offset + length exchanged from the other file because the ranges in
+ * each file might be different lengths. This should be safe because
+ * we flushed both files' page caches, exchanged all the mappings, and
+ * updated the ondisk sizes.
*/
if (fxr->flags & XFS_EXCHANGE_RANGE_TO_EOF) {
- loff_t temp;
+ loff_t old_ip1_size = i_size_read(VFS_I(ip1));
+ loff_t old_ip2_size = i_size_read(VFS_I(ip2));
- temp = i_size_read(VFS_I(ip2));
- i_size_write(VFS_I(ip2), i_size_read(VFS_I(ip1)));
- i_size_write(VFS_I(ip1), temp);
+ i_size_write(VFS_I(ip2), fxr->file2_offset +
+ (old_ip1_size - fxr->file1_offset));
+ i_size_write(VFS_I(ip1), fxr->file1_offset +
+ (old_ip2_size - fxr->file2_offset));
}
out_unlock:
@@ -442,6 +445,30 @@ xfs_exchange_range_checks(
return blen == fxr->length ? 0 : -EINVAL;
}
+/* Flush all relevant dirty pagecache near the range to be exchanged. */
+static inline int
+xfs_exchange_range_flush(
+ struct xfs_inode *ip,
+ const struct xfs_exchrange *fxr,
+ loff_t offset)
+{
+ /*
+ * If TO_EOF is set, the ondisk file size update logic depends on the
+ * ondisk file size matching the incore file size. Flush everything.
+ */
+ if (fxr->flags & XFS_EXCHANGE_RANGE_TO_EOF)
+ return filemap_write_and_wait(VFS_I(ip)->i_mapping);
+
+ /*
+ * Because we're updating the ondisk mappings, flush all dirty data
+ * between the ondisk size and the exchange offset to reduce the chance
+ * of zeroed file contents in that gap after a crash.
+ */
+ return filemap_write_and_wait_range(VFS_I(ip)->i_mapping,
+ min(offset, ip->i_disk_size),
+ offset + fxr->length - 1);
+}
+
/*
* Check that the two inodes are eligible for range exchanges, the ranges make
* sense, and then flush all dirty data. Caller must ensure that the inodes
@@ -467,15 +494,11 @@ xfs_exchange_range_prep(
if (!same_inode)
inode_dio_wait(inode2);
- error = filemap_write_and_wait_range(inode1->i_mapping,
- fxr->file1_offset,
- fxr->file1_offset + fxr->length - 1);
+ error = xfs_exchange_range_flush(XFS_I(inode1), fxr, fxr->file1_offset);
if (error)
return error;
- error = filemap_write_and_wait_range(inode2->i_mapping,
- fxr->file2_offset,
- fxr->file2_offset + fxr->length - 1);
+ error = xfs_exchange_range_flush(XFS_I(inode2), fxr, fxr->file2_offset);
if (error)
return error;
diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
index 08689839dd77..e22d103f5f82 100644
--- a/fs/xfs/xfs_icache.c
+++ b/fs/xfs/xfs_icache.c
@@ -923,7 +923,15 @@ xfs_reclaim_inode(
*/
if (xlog_is_shutdown(ip->i_mount->m_log)) {
xfs_iunpin_wait(ip);
+ /*
+ * Avoid a ABBA deadlock on the inode cluster buffer vs
+ * concurrent xfs_ifree_cluster() trying to mark the inode
+ * stale. We don't need the inode locked to run the flush abort
+ * code, but the flush abort needs to lock the cluster buffer.
+ */
+ xfs_iunlock(ip, XFS_ILOCK_EXCL);
xfs_iflush_shutdown_abort(ip);
+ xfs_ilock(ip, XFS_ILOCK_EXCL);
goto reclaim;
}
if (xfs_ipincount(ip))
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 266fcc789098..3b487341348a 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -1624,7 +1624,7 @@ xfs_ifree_mark_inode_stale(
iip = ip->i_itemp;
if (__xfs_iflags_test(ip, XFS_IFLUSHING)) {
ASSERT(!list_empty(&iip->ili_item.li_bio_list));
- ASSERT(iip->ili_last_fields);
+ ASSERT(iip->ili_last_fields || xlog_is_shutdown(mp->m_log));
goto out_iunlock;
}
diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
index 36ed81eb0d6d..43775ad8ce78 100644
--- a/fs/xfs/xfs_qm.c
+++ b/fs/xfs/xfs_qm.c
@@ -132,6 +132,7 @@ xfs_qm_dqpurge(
dqp->q_flags |= XFS_DQFLAG_FREEING;
+ xfs_qm_dqunpin_wait(dqp);
xfs_dqflock(dqp);
/*
@@ -430,6 +431,7 @@ xfs_qm_dquot_isolate(
struct xfs_dquot *dqp = container_of(item,
struct xfs_dquot, q_lru);
struct xfs_qm_isolate *isol = arg;
+ enum lru_status ret = LRU_SKIP;
if (!xfs_dqlock_nowait(dqp))
goto out_miss_busy;
@@ -442,6 +444,16 @@ xfs_qm_dquot_isolate(
if (dqp->q_flags & XFS_DQFLAG_FREEING)
goto out_miss_unlock;
+ /*
+ * If the dquot is pinned or dirty, rotate it to the end of the LRU to
+ * give some time for it to be cleaned before we try to isolate it
+ * again.
+ */
+ ret = LRU_ROTATE;
+ if (XFS_DQ_IS_DIRTY(dqp) || atomic_read(&dqp->q_pincount) > 0) {
+ goto out_miss_unlock;
+ }
+
/*
* This dquot has acquired a reference in the meantime remove it from
* the freelist and try again.
@@ -457,41 +469,14 @@ xfs_qm_dquot_isolate(
}
/*
- * If the dquot is dirty, flush it. If it's already being flushed, just
- * skip it so there is time for the IO to complete before we try to
- * reclaim it again on the next LRU pass.
+ * The dquot may still be under IO, in which case the flush lock will be
+ * held. If we can't get the flush lock now, just skip over the dquot as
+ * if it was dirty.
*/
if (!xfs_dqflock_nowait(dqp))
goto out_miss_unlock;
- if (XFS_DQ_IS_DIRTY(dqp)) {
- struct xfs_buf *bp = NULL;
- int error;
-
- trace_xfs_dqreclaim_dirty(dqp);
-
- /* we have to drop the LRU lock to flush the dquot */
- spin_unlock(lru_lock);
-
- error = xfs_dquot_use_attached_buf(dqp, &bp);
- if (!bp || error == -EAGAIN) {
- xfs_dqfunlock(dqp);
- goto out_unlock_dirty;
- }
-
- /*
- * dqflush completes dqflock on error, and the delwri ioend
- * does it on success.
- */
- error = xfs_qm_dqflush(dqp, bp);
- if (error)
- goto out_unlock_dirty;
-
- xfs_buf_delwri_queue(bp, &isol->buffers);
- xfs_buf_relse(bp);
- goto out_unlock_dirty;
- }
-
+ ASSERT(!XFS_DQ_IS_DIRTY(dqp));
xfs_dquot_detach_buf(dqp);
xfs_dqfunlock(dqp);
@@ -513,14 +498,7 @@ xfs_qm_dquot_isolate(
out_miss_busy:
trace_xfs_dqreclaim_busy(dqp);
XFS_STATS_INC(dqp->q_mount, xs_qm_dqreclaim_misses);
- return LRU_SKIP;
-
-out_unlock_dirty:
- trace_xfs_dqreclaim_busy(dqp);
- XFS_STATS_INC(dqp->q_mount, xs_qm_dqreclaim_misses);
- xfs_dqunlock(dqp);
- spin_lock(lru_lock);
- return LRU_RETRY;
+ return ret;
}
static unsigned long
@@ -1275,7 +1253,6 @@ xfs_qm_flush_one(
struct xfs_dquot *dqp,
void *data)
{
- struct xfs_mount *mp = dqp->q_mount;
struct list_head *buffer_list = data;
struct xfs_buf *bp = NULL;
int error = 0;
@@ -1286,34 +1263,8 @@ xfs_qm_flush_one(
if (!XFS_DQ_IS_DIRTY(dqp))
goto out_unlock;
- /*
- * The only way the dquot is already flush locked by the time quotacheck
- * gets here is if reclaim flushed it before the dqadjust walk dirtied
- * it for the final time. Quotacheck collects all dquot bufs in the
- * local delwri queue before dquots are dirtied, so reclaim can't have
- * possibly queued it for I/O. The only way out is to push the buffer to
- * cycle the flush lock.
- */
- if (!xfs_dqflock_nowait(dqp)) {
- /* buf is pinned in-core by delwri list */
- error = xfs_buf_incore(mp->m_ddev_targp, dqp->q_blkno,
- mp->m_quotainfo->qi_dqchunklen, 0, &bp);
- if (error)
- goto out_unlock;
-
- if (!(bp->b_flags & _XBF_DELWRI_Q)) {
- error = -EAGAIN;
- xfs_buf_relse(bp);
- goto out_unlock;
- }
- xfs_buf_unlock(bp);
-
- xfs_buf_delwri_pushbuf(bp, buffer_list);
- xfs_buf_rele(bp);
-
- error = -EAGAIN;
- goto out_unlock;
- }
+ xfs_qm_dqunpin_wait(dqp);
+ xfs_dqflock(dqp);
error = xfs_dquot_use_attached_buf(dqp, &bp);
if (error)
diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h
index a53a57b4e365..af12e4c7ecd2 100644
--- a/fs/xfs/xfs_trace.h
+++ b/fs/xfs/xfs_trace.h
@@ -493,7 +493,6 @@ DEFINE_BUF_EVENT(xfs_buf_iowait_done);
DEFINE_BUF_EVENT(xfs_buf_delwri_queue);
DEFINE_BUF_EVENT(xfs_buf_delwri_queued);
DEFINE_BUF_EVENT(xfs_buf_delwri_split);
-DEFINE_BUF_EVENT(xfs_buf_delwri_pushbuf);
DEFINE_BUF_EVENT(xfs_buf_get_uncached);
DEFINE_BUF_EVENT(xfs_buf_item_relse);
DEFINE_BUF_EVENT(xfs_buf_iodone_async);
@@ -1098,7 +1097,6 @@ DEFINE_EVENT(xfs_dquot_class, name, \
TP_ARGS(dqp))
DEFINE_DQUOT_EVENT(xfs_dqadjust);
DEFINE_DQUOT_EVENT(xfs_dqreclaim_want);
-DEFINE_DQUOT_EVENT(xfs_dqreclaim_dirty);
DEFINE_DQUOT_EVENT(xfs_dqreclaim_busy);
DEFINE_DQUOT_EVENT(xfs_dqreclaim_done);
DEFINE_DQUOT_EVENT(xfs_dqattach_found);
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 2aa34fcc10a1..b8cf06904fa2 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -677,6 +677,7 @@ void acpi_bus_unregister_driver(struct acpi_driver *driver);
int acpi_bus_scan(acpi_handle handle);
void acpi_bus_trim(struct acpi_device *start);
acpi_status acpi_bus_get_ejd(acpi_handle handle, acpi_handle * ejd);
+struct device *acpi_bus_get_primary_device(struct acpi_device *adev);
int acpi_match_device_ids(struct acpi_device *device,
const struct acpi_device_id *ids);
void acpi_set_modalias(struct acpi_device *adev, const char *default_id,
@@ -1000,6 +1001,11 @@ int acpi_scan_add_dep(acpi_handle handle, struct acpi_handle_list *dep_devices);
u32 arch_acpi_add_auto_dep(acpi_handle handle);
#else /* CONFIG_ACPI */
+static inline struct device *acpi_bus_get_primary_device(struct acpi_device *adev)
+{
+ return NULL;
+}
+
static inline int register_acpi_bus_type(void *bus) { return 0; }
static inline int unregister_acpi_bus_type(void *bus) { return 0; }
diff --git a/include/clocksource/samsung_pwm.h b/include/clocksource/samsung_pwm.h
index 9b435caa95fe..36f6f246e559 100644
--- a/include/clocksource/samsung_pwm.h
+++ b/include/clocksource/samsung_pwm.h
@@ -15,7 +15,7 @@
* spinlock is not shared between both drivers.
*/
#ifdef CONFIG_CLKSRC_SAMSUNG_PWM
-extern spinlock_t samsung_pwm_lock;
+extern raw_spinlock_t samsung_pwm_lock;
#endif
struct samsung_pwm_variant {
diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h
index 4246ed5b9703..be2644f96da8 100644
--- a/include/drm/gpu_scheduler.h
+++ b/include/drm/gpu_scheduler.h
@@ -550,12 +550,38 @@ struct drm_gpu_scheduler {
struct device *dev;
};
+/**
+ * struct drm_sched_init_args - parameters for initializing a DRM GPU scheduler
+ *
+ * @ops: backend operations provided by the driver
+ * @submit_wq: workqueue to use for submission. If NULL, an ordered wq is
+ * allocated and used.
+ * @num_rqs: Number of run-queues. This may be at most DRM_SCHED_PRIORITY_COUNT,
+ * as there's usually one run-queue per priority, but may be less.
+ * @credit_limit: the number of credits this scheduler can hold from all jobs
+ * @hang_limit: number of times to allow a job to hang before dropping it.
+ * This mechanism is DEPRECATED. Set it to 0.
+ * @timeout: timeout value in jiffies for submitted jobs.
+ * @timeout_wq: workqueue to use for timeout work. If NULL, the system_wq is used.
+ * @score: score atomic shared with other schedulers. May be NULL.
+ * @name: name (typically the driver's name). Used for debugging
+ * @dev: associated device. Used for debugging
+ */
+struct drm_sched_init_args {
+ const struct drm_sched_backend_ops *ops;
+ struct workqueue_struct *submit_wq;
+ struct workqueue_struct *timeout_wq;
+ u32 num_rqs;
+ u32 credit_limit;
+ unsigned int hang_limit;
+ long timeout;
+ atomic_t *score;
+ const char *name;
+ struct device *dev;
+};
+
int drm_sched_init(struct drm_gpu_scheduler *sched,
- const struct drm_sched_backend_ops *ops,
- struct workqueue_struct *submit_wq,
- u32 num_rqs, u32 credit_limit, unsigned int hang_limit,
- long timeout, struct workqueue_struct *timeout_wq,
- atomic_t *score, const char *name, struct device *dev);
+ const struct drm_sched_init_args *args);
void drm_sched_fini(struct drm_gpu_scheduler *sched);
int drm_sched_job_init(struct drm_sched_job *job,
diff --git a/include/linux/arm_ffa.h b/include/linux/arm_ffa.h
index 53f2837ce7df..c1d511e85a79 100644
--- a/include/linux/arm_ffa.h
+++ b/include/linux/arm_ffa.h
@@ -112,6 +112,7 @@
FIELD_PREP(FFA_MINOR_VERSION_MASK, (minor)))
#define FFA_VERSION_1_0 FFA_PACK_VERSION_INFO(1, 0)
#define FFA_VERSION_1_1 FFA_PACK_VERSION_INFO(1, 1)
+#define FFA_VERSION_1_2 FFA_PACK_VERSION_INFO(1, 2)
/**
* FF-A specification mentions explicitly about '4K pages'. This should
@@ -326,6 +327,7 @@ struct ffa_mem_region_attributes {
* an `struct ffa_mem_region_addr_range`.
*/
u32 composite_off;
+ u8 impdef_val[16];
u64 reserved;
};
@@ -405,18 +407,34 @@ struct ffa_mem_region {
#define CONSTITUENTS_OFFSET(x) \
(offsetof(struct ffa_composite_mem_region, constituents[x]))
+#define FFA_EMAD_HAS_IMPDEF_FIELD(version) ((version) >= FFA_VERSION_1_2)
+#define FFA_MEM_REGION_HAS_EP_MEM_OFFSET(version) ((version) > FFA_VERSION_1_0)
+
+static inline u32 ffa_emad_size_get(u32 ffa_version)
+{
+ u32 sz;
+ struct ffa_mem_region_attributes *ep_mem_access;
+
+ if (FFA_EMAD_HAS_IMPDEF_FIELD(ffa_version))
+ sz = sizeof(*ep_mem_access);
+ else
+ sz = sizeof(*ep_mem_access) - sizeof(ep_mem_access->impdef_val);
+
+ return sz;
+}
+
static inline u32
ffa_mem_desc_offset(struct ffa_mem_region *buf, int count, u32 ffa_version)
{
- u32 offset = count * sizeof(struct ffa_mem_region_attributes);
+ u32 offset = count * ffa_emad_size_get(ffa_version);
/*
* Earlier to v1.1, the endpoint memory descriptor array started at
* offset 32(i.e. offset of ep_mem_offset in the current structure)
*/
- if (ffa_version <= FFA_VERSION_1_0)
+ if (!FFA_MEM_REGION_HAS_EP_MEM_OFFSET(ffa_version))
offset += offsetof(struct ffa_mem_region, ep_mem_offset);
else
- offset += sizeof(struct ffa_mem_region);
+ offset += buf->ep_mem_offset;
return offset;
}
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index a63fd54ac6b4..f43aa1cd1b0b 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1323,6 +1323,8 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
+void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags);
+
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
* indirection with a direct call to the bpf program. If the architecture does
@@ -1431,6 +1433,7 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
{
return false;
}
+static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {}
#endif
struct bpf_func_info_aux {
@@ -2655,7 +2658,7 @@ int btf_struct_access(struct bpf_verifier_log *log,
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
- bool strict);
+ bool strict, bool walk_flex_arrays);
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
@@ -2999,6 +3002,11 @@ static inline void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
}
#endif /* CONFIG_BPF_SYSCALL */
+static inline bool bpf_net_capable(void)
+{
+ return capable(CAP_NET_ADMIN) || capable(CAP_SYS_ADMIN);
+}
+
static __always_inline int
bpf_probe_read_kernel_common(void *dst, u32 size, const void *unsafe_ptr)
{
diff --git a/include/linux/bpf_mprog.h b/include/linux/bpf_mprog.h
index 929225f7b095..0b9f4caeeb0a 100644
--- a/include/linux/bpf_mprog.h
+++ b/include/linux/bpf_mprog.h
@@ -340,4 +340,14 @@ static inline bool bpf_mprog_supported(enum bpf_prog_type type)
return false;
}
}
+
+static inline bool bpf_mprog_detach_empty(enum bpf_prog_type type)
+{
+ switch (type) {
+ case BPF_PROG_TYPE_SCHED_CLS:
+ return bpf_net_capable();
+ default:
+ return false;
+ }
+}
#endif /* __BPF_MPROG_H */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 48581c4def02..a0468b6f40b3 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -604,6 +604,8 @@ struct bpf_insn_aux_data {
* accepts callback function as a parameter.
*/
bool calls_callback;
+ /* true if some jump or call instruction targets this instruction */
+ bool jump_target;
};
#define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 2a510dbbfe62..66a283527fa9 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -688,6 +688,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen)
/*
* cgroup scalable recursive statistics.
*/
+void __cgroup_rstat_updated(struct cgroup *cgrp, int cpu);
void cgroup_rstat_updated(struct cgroup *cgrp, int cpu);
void cgroup_rstat_flush(struct cgroup *cgrp);
void cgroup_rstat_flush_hold(struct cgroup *cgrp);
diff --git a/include/linux/device.h b/include/linux/device.h
index 7969d02fe628..cc2ef7f0539c 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -994,6 +994,15 @@ static inline bool dev_pm_test_driver_flags(struct device *dev, u32 flags)
return !!(dev->power.driver_flags & flags);
}
+static inline bool dev_pm_smart_suspend(struct device *dev)
+{
+#ifdef CONFIG_PM_SLEEP
+ return dev->power.smart_suspend;
+#else
+ return false;
+#endif
+}
+
static inline void device_lock(struct device *dev)
{
mutex_lock(&dev->mutex);
@@ -1253,6 +1262,11 @@ void device_links_supplier_sync_state_pause(void);
void device_links_supplier_sync_state_resume(void);
void device_link_wait_removal(void);
+static inline bool device_link_test(const struct device_link *link, u32 flags)
+{
+ return !!(link->flags & flags);
+}
+
/* Create alias, so I can be autoloaded. */
#define MODULE_ALIAS_CHARDEV(major,minor) \
MODULE_ALIAS("char-major-" __stringify(major) "-" __stringify(minor))
diff --git a/include/linux/dma/k3-udma-glue.h b/include/linux/dma/k3-udma-glue.h
index 2dea217629d0..5d43881e6fb7 100644
--- a/include/linux/dma/k3-udma-glue.h
+++ b/include/linux/dma/k3-udma-glue.h
@@ -138,8 +138,7 @@ int k3_udma_glue_rx_get_irq(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_num);
void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_num, void *data,
- void (*cleanup)(void *data, dma_addr_t desc_dma),
- bool skip_fdq);
+ void (*cleanup)(void *data, dma_addr_t desc_dma));
int k3_udma_glue_rx_flow_enable(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_idx);
int k3_udma_glue_rx_flow_disable(struct k3_udma_glue_rx_channel *rx_chn,
diff --git a/include/linux/fb.h b/include/linux/fb.h
index d87d6547c6e4..8f41d449fb9d 100644
--- a/include/linux/fb.h
+++ b/include/linux/fb.h
@@ -537,6 +537,8 @@ extern int fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var);
extern int fb_pan_display(struct fb_info *info, struct fb_var_screeninfo *var);
extern int fb_blank(struct fb_info *info, int blank);
+int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var);
+
/*
* Helpers for framebuffers in I/O memory
*/
diff --git a/include/linux/folio_queue.h b/include/linux/folio_queue.h
index 3abe614ef5f0..4d3f8074c137 100644
--- a/include/linux/folio_queue.h
+++ b/include/linux/folio_queue.h
@@ -37,16 +37,20 @@ struct folio_queue {
#if PAGEVEC_SIZE > BITS_PER_LONG
#error marks is not big enough
#endif
+ unsigned int rreq_id;
+ unsigned int debug_id;
};
/**
* folioq_init - Initialise a folio queue segment
* @folioq: The segment to initialise
+ * @rreq_id: The request identifier to use in tracelines.
*
- * Initialise a folio queue segment. Note that the folio pointers are
- * left uninitialised.
+ * Initialise a folio queue segment and set an identifier to be used in traces.
+ *
+ * Note that the folio pointers are left uninitialised.
*/
-static inline void folioq_init(struct folio_queue *folioq)
+static inline void folioq_init(struct folio_queue *folioq, unsigned int rreq_id)
{
folio_batch_init(&folioq->vec);
folioq->next = NULL;
@@ -54,6 +58,8 @@ static inline void folioq_init(struct folio_queue *folioq)
folioq->marks = 0;
folioq->marks2 = 0;
folioq->marks3 = 0;
+ folioq->rreq_id = rreq_id;
+ folioq->debug_id = 0;
}
/**
diff --git a/include/linux/gpio.h b/include/linux/gpio.h
index 2d105be7bbc3..8bd6e6ac98c1 100644
--- a/include/linux/gpio.h
+++ b/include/linux/gpio.h
@@ -94,7 +94,7 @@ static inline int gpio_get_value_cansleep(unsigned gpio)
}
static inline void gpio_set_value_cansleep(unsigned gpio, int value)
{
- return gpiod_set_raw_value_cansleep(gpio_to_desc(gpio), value);
+ gpiod_set_raw_value_cansleep(gpio_to_desc(gpio), value);
}
static inline int gpio_get_value(unsigned gpio)
@@ -103,7 +103,7 @@ static inline int gpio_get_value(unsigned gpio)
}
static inline void gpio_set_value(unsigned gpio, int value)
{
- return gpiod_set_raw_value(gpio_to_desc(gpio), value);
+ gpiod_set_raw_value(gpio_to_desc(gpio), value);
}
static inline int gpio_to_irq(unsigned gpio)
diff --git a/include/linux/gpio/consumer.h b/include/linux/gpio/consumer.h
index db2dfbae8edb..4a87c24686e3 100644
--- a/include/linux/gpio/consumer.h
+++ b/include/linux/gpio/consumer.h
@@ -119,7 +119,7 @@ int gpiod_get_array_value(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
unsigned long *value_bitmap);
-void gpiod_set_value(struct gpio_desc *desc, int value);
+int gpiod_set_value(struct gpio_desc *desc, int value);
int gpiod_set_array_value(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
@@ -129,7 +129,7 @@ int gpiod_get_raw_array_value(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
unsigned long *value_bitmap);
-void gpiod_set_raw_value(struct gpio_desc *desc, int value);
+int gpiod_set_raw_value(struct gpio_desc *desc, int value);
int gpiod_set_raw_array_value(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
@@ -141,7 +141,7 @@ int gpiod_get_array_value_cansleep(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
unsigned long *value_bitmap);
-void gpiod_set_value_cansleep(struct gpio_desc *desc, int value);
+int gpiod_set_value_cansleep(struct gpio_desc *desc, int value);
int gpiod_set_array_value_cansleep(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
@@ -151,7 +151,7 @@ int gpiod_get_raw_array_value_cansleep(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
unsigned long *value_bitmap);
-void gpiod_set_raw_value_cansleep(struct gpio_desc *desc, int value);
+int gpiod_set_raw_value_cansleep(struct gpio_desc *desc, int value);
int gpiod_set_raw_array_value_cansleep(unsigned int array_size,
struct gpio_desc **desc_array,
struct gpio_array *array_info,
@@ -375,10 +375,11 @@ static inline int gpiod_get_array_value(unsigned int array_size,
WARN_ON(desc_array);
return 0;
}
-static inline void gpiod_set_value(struct gpio_desc *desc, int value)
+static inline int gpiod_set_value(struct gpio_desc *desc, int value)
{
/* GPIO can never have been requested */
WARN_ON(desc);
+ return 0;
}
static inline int gpiod_set_array_value(unsigned int array_size,
struct gpio_desc **desc_array,
@@ -404,10 +405,11 @@ static inline int gpiod_get_raw_array_value(unsigned int array_size,
WARN_ON(desc_array);
return 0;
}
-static inline void gpiod_set_raw_value(struct gpio_desc *desc, int value)
+static inline int gpiod_set_raw_value(struct gpio_desc *desc, int value)
{
/* GPIO can never have been requested */
WARN_ON(desc);
+ return 0;
}
static inline int gpiod_set_raw_array_value(unsigned int array_size,
struct gpio_desc **desc_array,
@@ -434,10 +436,11 @@ static inline int gpiod_get_array_value_cansleep(unsigned int array_size,
WARN_ON(desc_array);
return 0;
}
-static inline void gpiod_set_value_cansleep(struct gpio_desc *desc, int value)
+static inline int gpiod_set_value_cansleep(struct gpio_desc *desc, int value)
{
/* GPIO can never have been requested */
WARN_ON(desc);
+ return 0;
}
static inline int gpiod_set_array_value_cansleep(unsigned int array_size,
struct gpio_desc **desc_array,
@@ -463,11 +466,12 @@ static inline int gpiod_get_raw_array_value_cansleep(unsigned int array_size,
WARN_ON(desc_array);
return 0;
}
-static inline void gpiod_set_raw_value_cansleep(struct gpio_desc *desc,
- int value)
+static inline int gpiod_set_raw_value_cansleep(struct gpio_desc *desc,
+ int value)
{
/* GPIO can never have been requested */
WARN_ON(desc);
+ return 0;
}
static inline int gpiod_set_raw_array_value_cansleep(unsigned int array_size,
struct gpio_desc **desc_array,
diff --git a/include/linux/gpio/driver.h b/include/linux/gpio/driver.h
index 2dd7cb9cc270..ac42f0164d5f 100644
--- a/include/linux/gpio/driver.h
+++ b/include/linux/gpio/driver.h
@@ -346,6 +346,10 @@ struct gpio_irq_chip {
* stores them in "bits", returns 0 on success or negative error
* @set: assigns output value for signal "offset"
* @set_multiple: assigns output values for multiple signals defined by "mask"
+ * @set_rv: assigns output value for signal "offset", returns 0 on success or
+ * negative error value
+ * @set_multiple_rv: assigns output values for multiple signals defined by
+ * "mask", returns 0 on success or negative error value
* @set_config: optional hook for all kinds of settings. Uses the same
* packed config format as generic pinconf.
* @to_irq: optional hook supporting non-static gpiod_to_irq() mappings;
@@ -441,6 +445,12 @@ struct gpio_chip {
void (*set_multiple)(struct gpio_chip *gc,
unsigned long *mask,
unsigned long *bits);
+ int (*set_rv)(struct gpio_chip *gc,
+ unsigned int offset,
+ int value);
+ int (*set_multiple_rv)(struct gpio_chip *gc,
+ unsigned long *mask,
+ unsigned long *bits);
int (*set_config)(struct gpio_chip *gc,
unsigned int offset,
unsigned long config);
diff --git a/include/linux/gpu_buddy.h b/include/linux/gpu_buddy.h
index 2d45ce054f26..db0a23405ec9 100644
--- a/include/linux/gpu_buddy.h
+++ b/include/linux/gpu_buddy.h
@@ -14,15 +14,6 @@
#include <drm/drm_print.h>
-#define range_overflows(start, size, max) ({ \
- typeof(start) start__ = (start); \
- typeof(size) size__ = (size); \
- typeof(max) max__ = (max); \
- (void)(&start__ == &size__); \
- (void)(&start__ == &max__); \
- start__ >= max__ || size__ > max__ - start__; \
-})
-
#define GPU_BUDDY_RANGE_ALLOCATION BIT(0)
#define GPU_BUDDY_TOPDOWN_ALLOCATION BIT(1)
#define GPU_BUDDY_CONTIGUOUS_ALLOCATION BIT(2)
diff --git a/include/linux/hid.h b/include/linux/hid.h
index e2734b9d3ee0..2edaa8cbdd4a 100644
--- a/include/linux/hid.h
+++ b/include/linux/hid.h
@@ -81,6 +81,8 @@ struct hid_item {
#define HID_MAIN_ITEM_TAG_FEATURE 11
#define HID_MAIN_ITEM_TAG_BEGIN_COLLECTION 10
#define HID_MAIN_ITEM_TAG_END_COLLECTION 12
+#define HID_MAIN_ITEM_TAG_RESERVED_MIN 13
+#define HID_MAIN_ITEM_TAG_RESERVED_MAX 15
/*
* HID report descriptor main item contents
@@ -1260,6 +1262,8 @@ void hid_quirks_exit(__u16 bus);
dev_notice(&(hid)->dev, fmt, ##__VA_ARGS__)
#define hid_warn(hid, fmt, ...) \
dev_warn(&(hid)->dev, fmt, ##__VA_ARGS__)
+#define hid_warn_ratelimited(hid, fmt, ...) \
+ dev_warn_ratelimited(&(hid)->dev, fmt, ##__VA_ARGS__)
#define hid_info(hid, fmt, ...) \
dev_info(&(hid)->dev, fmt, ##__VA_ARGS__)
#define hid_dbg(hid, fmt, ...) \
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index dfea039a1dc8..7eaabacdf110 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -172,6 +172,8 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio);
extern int sysctl_hugetlb_shm_group;
extern struct list_head huge_boot_pages[MAX_NUMNODES];
+void hugetlb_bootmem_struct_page_init(void);
+
/* arch callbacks */
#ifndef CONFIG_HIGHPTE
@@ -1265,6 +1267,10 @@ static inline bool hugetlbfs_pagecache_present(
{
return false;
}
+
+static inline void hugetlb_bootmem_struct_page_init(void)
+{
+}
#endif /* CONFIG_HUGETLB_PAGE */
static inline spinlock_t *huge_pte_lock(struct hstate *h,
diff --git a/include/linux/i3c/master.h b/include/linux/i3c/master.h
index 414a8acde018..a17433c2458a 100644
--- a/include/linux/i3c/master.h
+++ b/include/linux/i3c/master.h
@@ -228,6 +228,8 @@ struct i3c_dev_desc {
* every time the I3C device is rediscovered with a different dynamic
* address assigned
* @bus: I3C bus this device is attached to
+ * @node: unregistered device list node, only for use by
+ * i3c_master_register_new_i3c_devs(), it is not protected by a lock
*
* I3C device object exposed to I3C device drivers. The takes care of linking
* this object to the relevant &struct_i3c_dev_desc one.
@@ -238,6 +240,7 @@ struct i3c_device {
struct device dev;
struct i3c_dev_desc *desc;
struct i3c_bus *bus;
+ struct list_head node;
};
/*
diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h
index 85bf3ac6db57..59e7c0a81cee 100644
--- a/include/linux/ieee80211.h
+++ b/include/linux/ieee80211.h
@@ -5348,7 +5348,7 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len)
const struct ieee80211_multi_link_elem *mle = (const void *)data;
u8 fixed = sizeof(*mle);
u8 common = 0;
- bool check_common_len = false;
+ u8 common_len;
u16 control;
if (!data || len < fixed)
@@ -5359,7 +5359,6 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len)
switch (u16_get_bits(control, IEEE80211_ML_CONTROL_TYPE)) {
case IEEE80211_ML_CONTROL_TYPE_BASIC:
common += sizeof(struct ieee80211_mle_basic_common_info);
- check_common_len = true;
if (control & IEEE80211_MLC_BASIC_PRES_LINK_ID)
common += 1;
if (control & IEEE80211_MLC_BASIC_PRES_BSS_PARAM_CH_CNT)
@@ -5377,15 +5376,14 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len)
common += sizeof(struct ieee80211_mle_preq_common_info);
if (control & IEEE80211_MLC_PREQ_PRES_MLD_ID)
common += 1;
- check_common_len = true;
break;
case IEEE80211_ML_CONTROL_TYPE_RECONF:
+ common += 1;
if (control & IEEE80211_MLC_RECONF_PRES_MLD_MAC_ADDR)
common += ETH_ALEN;
break;
case IEEE80211_ML_CONTROL_TYPE_TDLS:
common += sizeof(struct ieee80211_mle_tdls_common_info);
- check_common_len = true;
break;
case IEEE80211_ML_CONTROL_TYPE_PRIO_ACCESS:
common = ETH_ALEN + 1;
@@ -5398,11 +5396,9 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len)
if (len < fixed + common)
return false;
- if (!check_common_len)
- return true;
+ common_len = mle->variable[0];
- /* if present, common length is the first octet there */
- return mle->variable[0] >= common;
+ return common_len >= common && common_len <= len - fixed;
}
/**
diff --git a/include/linux/if_team.h b/include/linux/if_team.h
index cdc684e04a2f..06fec0a78db5 100644
--- a/include/linux/if_team.h
+++ b/include/linux/if_team.h
@@ -121,8 +121,7 @@ struct team_mode_ops {
int (*port_enter)(struct team *team, struct team_port *port);
void (*port_leave)(struct team *team, struct team_port *port);
void (*port_change_dev_addr)(struct team *team, struct team_port *port);
- void (*port_enabled)(struct team *team, struct team_port *port);
- void (*port_disabled)(struct team *team, struct team_port *port);
+ void (*port_tx_disabled)(struct team *team, struct team_port *port);
};
extern int team_modeop_port_enter(struct team *team, struct team_port *port);
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index 6d47d0d3fde0..7711d9920141 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -17,6 +17,14 @@ enum {
* It's also ignored unless IORING_SETUP_DEFER_TASKRUN is set.
*/
IOU_F_TWQ_LAZY_WAKE = 1,
+
+ /*
+ * Set when task_work is queued from a waitqueue wakeup handler, where
+ * an arbitrary provider waitqueue lock is held. Signaling the CQ ring
+ * eventfd inline from there can recurse back into that lock through
+ * epoll, so the eventfd signal must be deferred.
+ */
+ IOU_F_TWQ_IN_WAKE = 2,
};
enum io_uring_cmd_flags {
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index dc7ceb9406ed..929b9d0d90cd 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -1566,6 +1566,7 @@ void iopf_free_group(struct iopf_group *group);
int iommu_report_device_fault(struct device *dev, struct iopf_fault *evt);
void iopf_group_response(struct iopf_group *group,
enum iommu_page_response_code status);
+void iopf_group_dequeue(struct iopf_group *group);
#else
static inline int
iopf_queue_add_device(struct iopf_queue *queue, struct device *dev)
@@ -1611,5 +1612,9 @@ static inline void iopf_group_response(struct iopf_group *group,
enum iommu_page_response_code status)
{
}
+
+static inline void iopf_group_dequeue(struct iopf_group *group)
+{
+}
#endif /* CONFIG_IOMMU_IOPF */
#endif /* __LINUX_IOMMU_H */
diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
index 136f2980cba3..c5afd053ae32 100644
--- a/include/linux/irq_work.h
+++ b/include/linux/irq_work.h
@@ -2,8 +2,9 @@
#ifndef _LINUX_IRQ_WORK_H
#define _LINUX_IRQ_WORK_H
-#include <linux/smp_types.h>
+#include <linux/irq_work_types.h>
#include <linux/rcuwait.h>
+#include <linux/smp_types.h>
/*
* An entry can be in one of four states:
@@ -14,12 +15,6 @@
* busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed
*/
-struct irq_work {
- struct __call_single_node node;
- void (*func)(struct irq_work *);
- struct rcuwait irqwait;
-};
-
#define __IRQ_WORK_INIT(_func, _flags) (struct irq_work){ \
.node = { .u_flags = (_flags), }, \
.func = (_func), \
diff --git a/include/linux/irq_work_types.h b/include/linux/irq_work_types.h
new file mode 100644
index 000000000000..73abec5bb06e
--- /dev/null
+++ b/include/linux/irq_work_types.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_IRQ_WORK_TYPES_H
+#define _LINUX_IRQ_WORK_TYPES_H
+
+#include <linux/smp_types.h>
+#include <linux/types.h>
+
+struct irq_work {
+ struct __call_single_node node;
+ void (*func)(struct irq_work *);
+ struct rcuwait irqwait;
+};
+
+#endif
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 5814bf8298b2..ff10e5740a4c 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -565,7 +565,7 @@ static inline void kasan_init_hw_tags(void) { }
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
void kasan_populate_early_vm_area_shadow(void *start, unsigned long size);
-int kasan_populate_vmalloc(unsigned long addr, unsigned long size);
+int kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask);
void kasan_release_vmalloc(unsigned long start, unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end,
@@ -577,7 +577,7 @@ static inline void kasan_populate_early_vm_area_shadow(void *start,
unsigned long size)
{ }
static inline int kasan_populate_vmalloc(unsigned long start,
- unsigned long size)
+ unsigned long size, gfp_t gfp_mask)
{
return 0;
}
@@ -634,7 +634,7 @@ static __always_inline void kasan_vrealloc(const void *start,
static inline void kasan_populate_early_vm_area_shadow(void *start,
unsigned long size) { }
static inline int kasan_populate_vmalloc(unsigned long start,
- unsigned long size)
+ unsigned long size, gfp_t gfp_mask)
{
return 0;
}
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index b4235e99f0a9..2b7068685ff2 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2528,4 +2528,12 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
struct kvm_pre_fault_memory *range);
#endif
+#ifdef CONFIG_KVM_GENERIC_HARDWARE_ENABLING
+int kvm_enable_virtualization(void);
+void kvm_disable_virtualization(void);
+#else
+static inline int kvm_enable_virtualization(void) { return 0; }
+static inline void kvm_disable_virtualization(void) { }
+#endif
+
#endif
diff --git a/include/linux/lockd/bind.h b/include/linux/lockd/bind.h
index c53c81242e72..e25943e485c5 100644
--- a/include/linux/lockd/bind.h
+++ b/include/linux/lockd/bind.h
@@ -21,11 +21,17 @@
struct svc_rqst;
struct rpc_task;
struct rpc_clnt;
+struct module;
-/*
- * This is the set of functions for lockd->nfsd communication
+/**
+ * struct nlmsvc_binding - lockd -> nfsd callback table
+ * @owner: module that provides this binding.
+ * @fopen: open a file by NFS file handle on behalf of an NLM request.
+ * @fclose: close a file that was previously opened via @fopen.
+ * Implementations MUST be semantically equivalent to fput().
*/
struct nlmsvc_binding {
+ struct module *owner;
__be32 (*fopen)(struct svc_rqst *,
struct nfs_fh *,
struct file **,
@@ -33,7 +39,7 @@ struct nlmsvc_binding {
void (*fclose)(struct file *);
};
-extern const struct nlmsvc_binding *nlmsvc_ops;
+extern const struct nlmsvc_binding __rcu *nlmsvc_ops;
/*
* Similar to nfs_client_initdata, but without the NFS-specific
diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
index 991526039ccb..9d47cdc727ad 100644
--- a/include/linux/mlx5/cq.h
+++ b/include/linux/mlx5/cq.h
@@ -41,7 +41,6 @@ struct mlx5_core_cq {
int cqe_sz;
__be32 *set_ci_db;
__be32 *arm_db;
- struct mlx5_uars_page *uar;
refcount_t refcount;
struct completion free;
unsigned vector;
@@ -184,6 +183,7 @@ static inline void mlx5_cq_put(struct mlx5_core_cq *cq)
complete(&cq->free);
}
+void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe);
int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
u32 *in, int inlen, u32 *out, int outlen);
int mlx5_core_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 2e3324578f2e..aa38f0f28150 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -637,7 +637,7 @@ struct mlx5_priv {
struct mlx5_ft_pool *ft_pool;
struct mlx5_bfreg_data bfregs;
- struct mlx5_uars_page *uar;
+ struct mlx5_sq_bfreg bfreg;
#ifdef CONFIG_MLX5_SF
struct mlx5_vhca_state_notifier *vhca_state_notifier;
struct mlx5_sf_dev_table *sf_dev_table;
@@ -683,7 +683,6 @@ struct mlx5e_resources {
u32 pdn;
struct mlx5_td td;
u32 mkey;
- struct mlx5_sq_bfreg bfreg;
#define MLX5_MAX_NUM_TC 8
u32 tisn[MLX5_MAX_PORTS][MLX5_MAX_NUM_TC];
bool tisn_valid;
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index f4fe593c1400..622dfe2545d7 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -155,6 +155,11 @@ static inline int folio_lru_refs(struct folio *folio)
return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + workingset;
}
+static inline void folio_clear_lru_refs(struct folio *folio)
+{
+ set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0);
+}
+
static inline int folio_lru_gen(struct folio *folio)
{
unsigned long flags = READ_ONCE(folio->flags);
@@ -222,6 +227,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
{
unsigned long seq;
unsigned long flags;
+ unsigned long mask;
int gen = folio_lru_gen(folio);
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
@@ -257,7 +263,14 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
gen = lru_gen_from_seq(seq);
flags = (gen + 1UL) << LRU_GEN_PGOFF;
/* see the comment on MIN_NR_GENS about PG_active */
- set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags);
+ mask = LRU_GEN_MASK;
+ /*
+ * Don't clear PG_workingset here because it can affect PSI accounting
+ * if the activation is due to workingset refault.
+ */
+ if (folio_test_active(folio))
+ mask |= LRU_REFS_MASK | BIT(PG_referenced) | BIT(PG_active);
+ set_mask_bits(&folio->flags, mask, flags);
lru_gen_update_size(lruvec, folio, -1, gen);
/* for folio_rotate_reclaimable() */
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 8f932c4350fc..9d5248554152 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -403,6 +403,8 @@ enum {
NR_LRU_GEN_CAPS
};
+#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset))
+
#define MIN_LRU_BATCH BITS_PER_LONG
#define MAX_LRU_BATCH (MIN_LRU_BATCH * 64)
diff --git a/include/linux/mount.h b/include/linux/mount.h
index 04213d8ef837..fdc14f5d154d 100644
--- a/include/linux/mount.h
+++ b/include/linux/mount.h
@@ -50,7 +50,8 @@ struct path;
#define MNT_ATIME_MASK (MNT_NOATIME | MNT_NODIRATIME | MNT_RELATIME )
#define MNT_INTERNAL_FLAGS (MNT_SHARED | MNT_WRITE_HOLD | MNT_INTERNAL | \
- MNT_DOOMED | MNT_SYNC_UMOUNT | MNT_MARKED)
+ MNT_DOOMED | MNT_SYNC_UMOUNT | MNT_MARKED | \
+ MNT_UMOUNT)
#define MNT_INTERNAL 0x4000
diff --git a/include/linux/net.h b/include/linux/net.h
index b75bc534c1b3..c6d92428c71a 100644
--- a/include/linux/net.h
+++ b/include/linux/net.h
@@ -23,9 +23,30 @@
#include <linux/fs.h>
#include <linux/mm.h>
#include <linux/sockptr.h>
+#include <linux/uio.h>
#include <uapi/linux/net.h>
+/**
+ * struct sockopt - socket option value container
+ * @iter_in: iov_iter for reading optval with the content from the caller.
+ * Use copy_from_iter() given this iov direction is ITER_SOURCE
+ * @iter_out: iov_iter for protocols to update optval data to userspace
+ * Use _copy_to_iter() given iov direction is ITER_DEST
+ * @optlen: serves as both input (buffer size) and output (returned data size).
+ *
+ * Type-safe wrapper for socket option data that works with both
+ * user and kernel buffers.
+ *
+ * The optlen field allows callbacks to return a specific length value
+ * independent of the bytes written via copy_to_iter().
+ */
+typedef struct sockopt {
+ struct iov_iter iter_in;
+ struct iov_iter iter_out;
+ int optlen;
+} sockopt_t;
+
struct poll_table_struct;
struct pipe_inode_info;
struct inode;
@@ -194,6 +215,8 @@ struct proto_ops {
unsigned int optlen);
int (*getsockopt)(struct socket *sock, int level,
int optname, char __user *optval, int __user *optlen);
+ int (*getsockopt_iter)(struct socket *sock, int level,
+ int optname, sockopt_t *opt);
void (*show_fdinfo)(struct seq_file *m, struct socket *sock);
int (*sendmsg) (struct socket *sock, struct msghdr *m,
size_t total_len);
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 0aa536df9cb5..4d1e0b91402d 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1896,6 +1896,8 @@ enum netdev_reg_state {
* @qdisc_hash: qdisc hash table
* @watchdog_timeo: Represents the timeout that is used by
* the watchdog (see dev_watchdog())
+ * @watchdog_lock: protect watchdog_ref_held
+ * @watchdog_ref_held: True if the watchdog device ref is taken.
* @watchdog_timer: List of timers
*
* @proto_down_reason: reason a netdev interface is held down
@@ -1974,7 +1976,7 @@ enum netdev_reg_state {
* regardless of source, even if those aren't
* HWTSTAMP_SOURCE_NETDEV
* @change_proto_down: device supports setting carrier via IFLA_PROTO_DOWN
- * @netns_local: interface can't change network namespaces
+ * @netns_immutable: interface can't change network namespaces
* @fcoe_mtu: device supports maximum FCoE MTU, 2158 bytes
*
* @net_notifier_list: List of per-net netdev notifier block
@@ -2285,6 +2287,8 @@ struct net_device {
/* These may be needed for future network-power-down code. */
struct timer_list watchdog_timer;
int watchdog_timeo;
+ spinlock_t watchdog_lock;
+ bool watchdog_ref_held;
u32 proto_down_reason;
@@ -2369,7 +2373,7 @@ struct net_device {
/* priv_flags_slow, ungrouped to save space */
unsigned long see_all_hwtstamp_requests:1;
unsigned long change_proto_down:1;
- unsigned long netns_local:1;
+ unsigned long netns_immutable:1;
unsigned long fcoe_mtu:1;
struct list_head net_notifier_list;
@@ -4018,12 +4022,13 @@ int dev_change_flags(struct net_device *dev, unsigned int flags,
int dev_set_alias(struct net_device *, const char *, size_t);
int dev_get_alias(const struct net_device *, char *, size_t);
int __dev_change_net_namespace(struct net_device *dev, struct net *net,
- const char *pat, int new_ifindex);
+ const char *pat, int new_ifindex,
+ struct netlink_ext_ack *extack);
static inline
int dev_change_net_namespace(struct net_device *dev, struct net *net,
const char *pat)
{
- return __dev_change_net_namespace(dev, net, pat, 0);
+ return __dev_change_net_namespace(dev, net, pat, 0, NULL);
}
int __dev_set_mtu(struct net_device *, int);
int dev_set_mtu(struct net_device *, int);
diff --git a/include/linux/netfilter.h b/include/linux/netfilter.h
index 2b8aac2c70ad..9b3357d8d8c3 100644
--- a/include/linux/netfilter.h
+++ b/include/linux/netfilter.h
@@ -95,6 +95,8 @@ enum nf_hook_ops_type {
};
struct nf_hook_ops {
+ struct rcu_head rcu;
+
/* User fills in from here down. */
nf_hookfn *hook;
struct net_device *dev;
diff --git a/include/linux/netfilter/x_tables.h b/include/linux/netfilter/x_tables.h
index 5897f3dbaf7c..3716b42a4ad6 100644
--- a/include/linux/netfilter/x_tables.h
+++ b/include/linux/netfilter/x_tables.h
@@ -307,6 +307,7 @@ struct xt_counters *xt_counters_alloc(unsigned int counters);
struct xt_table *xt_register_table(struct net *net,
const struct xt_table *table,
+ const struct nf_hook_ops *template_ops,
struct xt_table_info *bootstrap,
struct xt_table_info *newinfo);
void *xt_unregister_table(struct xt_table *table);
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index f8e0d9fa745a..cab466964141 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -18,9 +18,11 @@
#include <linux/fs.h>
#include <linux/pagemap.h>
#include <linux/uio.h>
+#include <linux/rolling_buffer.h>
enum netfs_sreq_ref_trace;
typedef struct mempool_s mempool_t;
+struct folio_queue;
/**
* folio_start_private_2 - Start an fscache write on a folio. [DEPRECATED]
@@ -238,13 +240,13 @@ struct netfs_io_request {
struct netfs_io_stream io_streams[2]; /* Streams of parallel I/O operations */
#define NR_IO_STREAMS 2 //wreq->nr_io_streams
struct netfs_group *group; /* Writeback group being written back */
- struct folio_queue *buffer; /* Head of I/O buffer */
- struct folio_queue *buffer_tail; /* Tail of I/O buffer */
- struct iov_iter iter; /* Unencrypted-side iterator */
- struct iov_iter io_iter; /* I/O (Encrypted-side) iterator */
+ struct rolling_buffer buffer; /* Unencrypted buffer */
+#define NETFS_ROLLBUF_PUT_MARK ROLLBUF_MARK_1
+#define NETFS_ROLLBUF_PAGECACHE_MARK ROLLBUF_MARK_2
void *netfs_priv; /* Private data for the netfs */
void *netfs_priv2; /* Private data for the netfs */
struct bio_vec *direct_bv; /* DIO buffer list (when handling iovec-iter) */
+ gfp_t gfp; /* GFP flags to use */
unsigned int direct_bv_count; /* Number of elements in direct_bv[] */
unsigned int debug_id;
unsigned int rsize; /* Maximum read size (0 for none) */
@@ -259,8 +261,6 @@ struct netfs_io_request {
long error; /* 0 or error that occurred */
enum netfs_io_origin origin; /* Origin of the request */
bool direct_bv_unpin; /* T if direct_bv[] must be unpinned */
- u8 buffer_head_slot; /* First slot in ->buffer */
- u8 buffer_tail_slot; /* Next slot in ->buffer_tail */
unsigned long long i_size; /* Size of the file */
unsigned long long start; /* Start position */
atomic64_t issued_to; /* Write issuer folio cursor */
@@ -431,8 +431,8 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp);
vm_fault_t netfs_page_mkwrite(struct vm_fault *vmf, struct netfs_group *netfs_group);
/* (Sub)request management API. */
-void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq, bool was_async);
-void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq, bool was_async);
+void netfs_read_subreq_progress(struct netfs_io_subrequest *subreq);
+void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq);
void netfs_read_subreq_termination_worker(struct work_struct *work);
void netfs_get_subrequest(struct netfs_io_subrequest *subreq,
enum netfs_sreq_ref_trace what);
@@ -455,6 +455,12 @@ void netfs_end_io_write(struct inode *inode);
int netfs_start_io_direct(struct inode *inode);
void netfs_end_io_direct(struct inode *inode);
+/* Miscellaneous APIs. */
+struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,
+ unsigned int trace /*enum netfs_folioq_trace*/);
+void netfs_folioq_free(struct folio_queue *folioq,
+ unsigned int trace /*enum netfs_trace_folioq*/);
+
/**
* netfs_inode - Get the netfs inode context from the inode
* @inode: The inode to query
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index 191b524e5c0d..5c29e904a9c5 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -256,6 +256,7 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
* @msg_clear_mask: See ntb_msg_clear_mask().
* @msg_read: See ntb_msg_read().
* @peer_msg_write: See ntb_peer_msg_write().
+ * @get_dma_dev: See ntb_get_dma_dev().
*/
struct ntb_dev_ops {
int (*port_number)(struct ntb_dev *ntb);
@@ -329,6 +330,7 @@ struct ntb_dev_ops {
int (*msg_clear_mask)(struct ntb_dev *ntb, u64 mask_bits);
u32 (*msg_read)(struct ntb_dev *ntb, int *pidx, int midx);
int (*peer_msg_write)(struct ntb_dev *ntb, int pidx, int midx, u32 msg);
+ struct device *(*get_dma_dev)(struct ntb_dev *ntb);
};
static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
@@ -391,6 +393,8 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
/* !ops->msg_clear_mask == !ops->msg_count && */
!ops->msg_read == !ops->msg_count &&
!ops->peer_msg_write == !ops->msg_count &&
+
+ /* ops->get_dma_dev is optional */
1;
}
@@ -1563,6 +1567,26 @@ static inline int ntb_peer_msg_write(struct ntb_dev *ntb, int pidx, int midx,
return ntb->ops->peer_msg_write(ntb, pidx, midx, msg);
}
+/**
+ * ntb_get_dma_dev() - get the device to use for DMA allocations/mappings
+ * @ntb: NTB device context.
+ *
+ * Return a struct device suitable for DMA API allocations and mappings.
+ * This is typically the parent of the NTB device, but may be overridden by a
+ * driver by implementing .get_dma_dev().
+ *
+ * Drivers that implement .get_dma_dev() must return a non-NULL pointer.
+ *
+ * Return: device pointer to use for DMA operations.
+ */
+static inline struct device *ntb_get_dma_dev(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->get_dma_dev)
+ return ntb->dev.parent;
+
+ return ntb->ops->get_dma_dev(ntb);
+}
+
/**
* ntb_peer_resource_idx() - get a resource index for a given peer idx
* @ntb: NTB device context.
diff --git a/include/linux/overflow.h b/include/linux/overflow.h
index 89e9d6049883..456b3b09e3e9 100644
--- a/include/linux/overflow.h
+++ b/include/linux/overflow.h
@@ -238,6 +238,76 @@ static inline bool __must_check __must_check_overflow(bool overflow)
__overflows_type_constexpr(n, T), \
__overflows_type(n, T))
+/**
+ * range_overflows() - Check if a range is out of bounds
+ * @start: Start of the range.
+ * @size: Size of the range.
+ * @max: Exclusive upper boundary.
+ *
+ * A strict check to determine if the range [@start, @start + @size) is
+ * invalid with respect to the allowable range [0, @max). Any range
+ * starting at or beyond @max is considered an overflow, even if @size is 0.
+ *
+ * Returns: true if the range is out of bounds.
+ */
+#define range_overflows(start, size, max) ({ \
+ typeof(start) start__ = (start); \
+ typeof(size) size__ = (size); \
+ typeof(max) max__ = (max); \
+ (void)(&start__ == &size__); \
+ (void)(&start__ == &max__); \
+ start__ >= max__ || size__ > max__ - start__; \
+})
+
+/**
+ * range_overflows_t() - Check if a range is out of bounds
+ * @type: Data type to use.
+ * @start: Start of the range.
+ * @size: Size of the range.
+ * @max: Exclusive upper boundary.
+ *
+ * Same as range_overflows() but forcing the parameters to @type.
+ *
+ * Returns: true if the range is out of bounds.
+ */
+#define range_overflows_t(type, start, size, max) \
+ range_overflows((type)(start), (type)(size), (type)(max))
+
+/**
+ * range_end_overflows() - Check if a range's endpoint is out of bounds
+ * @start: Start of the range.
+ * @size: Size of the range.
+ * @max: Exclusive upper boundary.
+ *
+ * Checks only if the endpoint of a range (@start + @size) exceeds @max.
+ * Unlike range_overflows(), a zero-sized range at the boundary (@start == @max)
+ * is not considered an overflow. Useful for iterator-style checks.
+ *
+ * Returns: true if the endpoint exceeds the boundary.
+ */
+#define range_end_overflows(start, size, max) ({ \
+ typeof(start) start__ = (start); \
+ typeof(size) size__ = (size); \
+ typeof(max) max__ = (max); \
+ (void)(&start__ == &size__); \
+ (void)(&start__ == &max__); \
+ start__ > max__ || size__ > max__ - start__; \
+})
+
+/**
+ * range_end_overflows_t() - Check if a range's endpoint is out of bounds
+ * @type: Data type to use.
+ * @start: Start of the range.
+ * @size: Size of the range.
+ * @max: Exclusive upper boundary.
+ *
+ * Same as range_end_overflows() but forcing the parameters to @type.
+ *
+ * Returns: true if the endpoint exceeds the boundary.
+ */
+#define range_end_overflows_t(type, start, size, max) \
+ range_end_overflows((type)(start), (type)(size), (type)(max))
+
/**
* castable_to_type - like __same_type(), but also allows for casted literals
*
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 4e8c6d0511c5..53171d714ac6 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -543,7 +543,7 @@ FOLIO_FLAG(swapbacked, FOLIO_HEAD_PAGE)
* - PG_private and PG_private_2 cause release_folio() and co to be invoked
*/
PAGEFLAG(Private, private, PF_ANY)
-PAGEFLAG(Private2, private_2, PF_ANY) TESTSCFLAG(Private2, private_2, PF_ANY)
+FOLIO_FLAG(private_2, FOLIO_HEAD_PAGE)
/* owner_2 can be set on tail pages for anon memory */
FOLIO_FLAG(owner_2, FOLIO_HEAD_PAGE)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index ee27f1ba01b8..f3051fb5a5e8 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -929,6 +929,7 @@ struct module;
* function returns zero when the driver chooses to
* take "ownership" of the device or an error code
* (negative number) otherwise.
+ * The pci_device_id parameter is only valid during probe.
* The probe function always gets called from process
* context, so it can sleep.
* @remove: The remove() function gets called whenever a device
diff --git a/include/linux/pm.h b/include/linux/pm.h
index 97b0e23363c8..6bc559807526 100644
--- a/include/linux/pm.h
+++ b/include/linux/pm.h
@@ -600,6 +600,7 @@ enum rpm_status {
RPM_RESUMING,
RPM_SUSPENDED,
RPM_SUSPENDING,
+ RPM_BLOCKED,
};
/*
@@ -682,6 +683,7 @@ struct dev_pm_info {
bool syscore:1;
bool no_pm_callbacks:1; /* Owned by the PM core */
bool async_in_progress:1; /* Owned by the PM core */
+ bool smart_suspend:1; /* Owned by the PM core */
bool must_resume:1; /* Owned by the PM core */
bool may_skip_resume:1; /* Set by subsystems */
#else
diff --git a/include/linux/pm_runtime.h b/include/linux/pm_runtime.h
index 142ab39a3b38..41db4d628d25 100644
--- a/include/linux/pm_runtime.h
+++ b/include/linux/pm_runtime.h
@@ -78,8 +78,11 @@ extern int pm_runtime_get_if_in_use(struct device *dev);
extern int pm_schedule_suspend(struct device *dev, unsigned int delay);
extern int __pm_runtime_set_status(struct device *dev, unsigned int status);
extern int pm_runtime_barrier(struct device *dev);
+extern void pm_runtime_block_if_disabled(struct device *dev);
+extern void pm_runtime_unblock(struct device *dev);
extern void pm_runtime_enable(struct device *dev);
extern void __pm_runtime_disable(struct device *dev, bool check_resume);
+extern bool pm_runtime_blocked(struct device *dev);
extern void pm_runtime_allow(struct device *dev);
extern void pm_runtime_forbid(struct device *dev);
extern void pm_runtime_no_callbacks(struct device *dev);
@@ -275,8 +278,11 @@ static inline int pm_runtime_get_if_active(struct device *dev)
static inline int __pm_runtime_set_status(struct device *dev,
unsigned int status) { return 0; }
static inline int pm_runtime_barrier(struct device *dev) { return 0; }
+static inline void pm_runtime_block_if_disabled(struct device *dev) {}
+static inline void pm_runtime_unblock(struct device *dev) {}
static inline void pm_runtime_enable(struct device *dev) {}
static inline void __pm_runtime_disable(struct device *dev, bool c) {}
+static inline bool pm_runtime_blocked(struct device *dev) { return true; }
static inline void pm_runtime_allow(struct device *dev) {}
static inline void pm_runtime_forbid(struct device *dev) {}
@@ -562,11 +568,18 @@ static inline int pm_runtime_set_suspended(struct device *dev)
* pm_runtime_disable - Disable runtime PM for a device.
* @dev: Target device.
*
- * Prevent the runtime PM framework from working with @dev (by incrementing its
- * "blocking" counter).
+ * Prevent the runtime PM framework from working with @dev by incrementing its
+ * "disable" counter.
*
- * For each invocation of this function for @dev there must be a matching
- * pm_runtime_enable() call in order for runtime PM to be enabled for it.
+ * If the counter is zero when this function runs and there is a pending runtime
+ * resume request for @dev, it will be resumed. If the counter is still zero at
+ * that point, all of the pending runtime PM requests for @dev will be canceled
+ * and all runtime PM operations in progress involving it will be waited for to
+ * complete.
+ *
+ * For each invocation of this function for @dev, there must be a matching
+ * pm_runtime_enable() call, so that runtime PM is eventually enabled for it
+ * again.
*/
static inline void pm_runtime_disable(struct device *dev)
{
diff --git a/include/linux/psi.h b/include/linux/psi.h
index e0745873e3f2..7966e3ac03b9 100644
--- a/include/linux/psi.h
+++ b/include/linux/psi.h
@@ -25,7 +25,9 @@ void psi_memstall_leave(unsigned long *flags);
int psi_show(struct seq_file *s, struct psi_group *group, enum psi_res res);
struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
enum psi_res res, struct file *file,
- struct kernfs_open_file *of);
+ struct kernfs_open_file *of,
+ bool *need_rtpoll_worker);
+int psi_trigger_create_rtpoll_worker(struct psi_group *group);
void psi_trigger_destroy(struct psi_trigger *t);
__poll_t psi_trigger_poll(void **trigger_ptr, struct file *file,
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 0844ab328851..05cc0d07f589 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -971,6 +971,20 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
preempt_enable_notrace();
}
+static __always_inline void rcu_read_lock_dont_migrate(void)
+{
+ if (IS_ENABLED(CONFIG_PREEMPT_RCU))
+ migrate_disable();
+ rcu_read_lock();
+}
+
+static inline void rcu_read_unlock_migrate(void)
+{
+ rcu_read_unlock();
+ if (IS_ENABLED(CONFIG_PREEMPT_RCU))
+ migrate_enable();
+}
+
/**
* RCU_INIT_POINTER() - initialize an RCU protected pointer
* @p: The pointer to be initialized.
diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h
index 27343424225c..9ad134a04b41 100644
--- a/include/linux/rcuwait.h
+++ b/include/linux/rcuwait.h
@@ -4,18 +4,7 @@
#include <linux/rcupdate.h>
#include <linux/sched/signal.h>
-
-/*
- * rcuwait provides a way of blocking and waking up a single
- * task in an rcu-safe manner.
- *
- * The only time @task is non-nil is when a user is blocked (or
- * checking if it needs to) on a condition, and reset as soon as we
- * know that the condition has succeeded and are awoken.
- */
-struct rcuwait {
- struct task_struct __rcu *task;
-};
+#include <linux/types.h>
#define __RCUWAIT_INITIALIZER(name) \
{ .task = NULL, }
diff --git a/include/linux/refcount.h b/include/linux/refcount.h
index 35f039ecb272..4589d2e7bfea 100644
--- a/include/linux/refcount.h
+++ b/include/linux/refcount.h
@@ -87,6 +87,15 @@
* The decrements dec_and_test() and sub_and_test() also provide acquire
* ordering on success.
*
+ * refcount_{add|inc}_not_zero_acquire() and refcount_set_release() provide
+ * acquire and release ordering for cases when the memory occupied by the
+ * object might be reused to store another object. This is important for the
+ * cases where secondary validation is required to detect such reuse, e.g.
+ * SLAB_TYPESAFE_BY_RCU. The secondary validation checks have to happen after
+ * the refcount is taken, hence acquire order is necessary. Similarly, when the
+ * object is initialized, all stores to its attributes should be visible before
+ * the refcount is set, otherwise a stale attribute value might be used by
+ * another task which succeeds in taking a refcount to the new object.
*/
#ifndef _LINUX_REFCOUNT_H
@@ -125,6 +134,31 @@ static inline void refcount_set(refcount_t *r, int n)
atomic_set(&r->refs, n);
}
+/**
+ * refcount_set_release - set a refcount's value with release ordering
+ * @r: the refcount
+ * @n: value to which the refcount will be set
+ *
+ * This function should be used when memory occupied by the object might be
+ * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
+ *
+ * Provides release memory ordering which will order previous memory operations
+ * against this store. This ensures all updates to this object are visible
+ * once the refcount is set and stale values from the object previously
+ * occupying this memory are overwritten with new ones.
+ *
+ * This function should be called only after new object is fully initialized.
+ * After this call the object should be considered visible to other tasks even
+ * if it was not yet added into an object collection normally used to discover
+ * it. This is because other tasks might have discovered the object previously
+ * occupying the same memory and after memory reuse they can succeed in taking
+ * refcount to the new object and start using it.
+ */
+static inline void refcount_set_release(refcount_t *r, int n)
+{
+ atomic_set_release(&r->refs, n);
+}
+
/**
* refcount_read - get a refcount's value
* @r: the refcount
@@ -178,6 +212,52 @@ static inline __must_check bool refcount_add_not_zero(int i, refcount_t *r)
return __refcount_add_not_zero(i, r, NULL);
}
+static inline __must_check __signed_wrap
+bool __refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp)
+{
+ int old = refcount_read(r);
+
+ do {
+ if (!old)
+ break;
+ } while (!atomic_try_cmpxchg_acquire(&r->refs, &old, old + i));
+
+ if (oldp)
+ *oldp = old;
+
+ if (unlikely(old < 0 || old + i < 0))
+ refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF);
+
+ return old;
+}
+
+/**
+ * refcount_add_not_zero_acquire - add a value to a refcount with acquire ordering unless it is 0
+ *
+ * @i: the value to add to the refcount
+ * @r: the refcount
+ *
+ * Will saturate at REFCOUNT_SATURATED and WARN.
+ *
+ * This function should be used when memory occupied by the object might be
+ * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
+ *
+ * Provides acquire memory ordering on success, it is assumed the caller has
+ * guaranteed the object memory to be stable (RCU, etc.). It does provide a
+ * control dependency and thereby orders future stores. See the comment on top.
+ *
+ * Use of this function is not recommended for the normal reference counting
+ * use case in which references are taken and released one at a time. In these
+ * cases, refcount_inc_not_zero_acquire() should instead be used to increment a
+ * reference count.
+ *
+ * Return: false if the passed refcount is 0, true otherwise
+ */
+static inline __must_check bool refcount_add_not_zero_acquire(int i, refcount_t *r)
+{
+ return __refcount_add_not_zero_acquire(i, r, NULL);
+}
+
static inline __signed_wrap
void __refcount_add(int i, refcount_t *r, int *oldp)
{
@@ -236,6 +316,32 @@ static inline __must_check bool refcount_inc_not_zero(refcount_t *r)
return __refcount_inc_not_zero(r, NULL);
}
+static inline __must_check bool __refcount_inc_not_zero_acquire(refcount_t *r, int *oldp)
+{
+ return __refcount_add_not_zero_acquire(1, r, oldp);
+}
+
+/**
+ * refcount_inc_not_zero_acquire - increment a refcount with acquire ordering unless it is 0
+ * @r: the refcount to increment
+ *
+ * Similar to refcount_inc_not_zero(), but provides acquire memory ordering on
+ * success.
+ *
+ * This function should be used when memory occupied by the object might be
+ * reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
+ *
+ * Provides acquire memory ordering on success, it is assumed the caller has
+ * guaranteed the object memory to be stable (RCU, etc.). It does provide a
+ * control dependency and thereby orders future stores. See the comment on top.
+ *
+ * Return: true if the increment was successful, false otherwise
+ */
+static inline __must_check bool refcount_inc_not_zero_acquire(refcount_t *r)
+{
+ return __refcount_inc_not_zero_acquire(r, NULL);
+}
+
static inline void __refcount_inc(refcount_t *r, int *oldp)
{
__refcount_add(1, r, oldp);
diff --git a/include/linux/ring_buffer.h b/include/linux/ring_buffer.h
index 03934249b91e..f39289bce1a0 100644
--- a/include/linux/ring_buffer.h
+++ b/include/linux/ring_buffer.h
@@ -214,14 +214,15 @@ bool ring_buffer_time_stamp_abs(struct trace_buffer *buffer);
size_t ring_buffer_nr_dirty_pages(struct trace_buffer *buffer, int cpu);
struct buffer_data_read_page;
-struct buffer_data_read_page *
-ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu);
+int ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu,
+ struct buffer_data_read_page **rpage);
void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
struct buffer_data_read_page *page);
int ring_buffer_read_page(struct trace_buffer *buffer,
struct buffer_data_read_page *data_page,
size_t len, int cpu, int full);
void *ring_buffer_read_page_data(struct buffer_data_read_page *page);
+unsigned int ring_buffer_read_page_size(struct buffer_data_read_page *rpage);
struct trace_seq;
diff --git a/include/linux/rolling_buffer.h b/include/linux/rolling_buffer.h
new file mode 100644
index 000000000000..9e5dad29669c
--- /dev/null
+++ b/include/linux/rolling_buffer.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* Rolling buffer of folios
+ *
+ * Copyright (C) 2024 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#ifndef _ROLLING_BUFFER_H
+#define _ROLLING_BUFFER_H
+
+#include <linux/folio_queue.h>
+#include <linux/uio.h>
+
+/*
+ * Rolling buffer. Whilst the buffer is live and in use, folios and folio
+ * queue segments can be added to one end by one thread and removed from the
+ * other end by another thread. The buffer isn't allowed to be empty; it must
+ * always have at least one folio_queue in it so that neither side has to
+ * modify both queue pointers.
+ *
+ * The iterator in the buffer is extended as buffers are inserted. It can be
+ * snapshotted to use a segment of the buffer.
+ */
+struct rolling_buffer {
+ struct folio_queue *head; /* Producer's insertion point */
+ struct folio_queue *tail; /* Consumer's removal point */
+ struct iov_iter iter; /* Iterator tracking what's left in the buffer */
+ u8 next_head_slot; /* Next slot in ->head */
+ u8 first_tail_slot; /* First slot in ->tail */
+};
+
+/*
+ * Snapshot of a rolling buffer.
+ */
+struct rolling_buffer_snapshot {
+ struct folio_queue *curr_folioq; /* Queue segment in which current folio resides */
+ unsigned char curr_slot; /* Folio currently being read */
+ unsigned char curr_order; /* Order of folio */
+};
+
+/* Marks to store per-folio in the internal folio_queue structs. */
+#define ROLLBUF_MARK_1 BIT(0)
+#define ROLLBUF_MARK_2 BIT(1)
+
+int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
+ unsigned int direction, gfp_t gfp);
+int rolling_buffer_make_space(struct rolling_buffer *roll, gfp_t gfp);
+ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
+ struct readahead_control *ractl,
+ struct folio_batch *put_batch);
+ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
+ unsigned int flags, gfp_t gfp);
+struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll);
+void rolling_buffer_clear(struct rolling_buffer *roll);
+
+static inline void rolling_buffer_advance(struct rolling_buffer *roll, size_t amount)
+{
+ iov_iter_advance(&roll->iter, amount);
+}
+
+#endif /* _ROLLING_BUFFER_H */
diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h
index ce8464324d42..ae2b34cc10ee 100644
--- a/include/linux/skmsg.h
+++ b/include/linux/skmsg.h
@@ -561,20 +561,6 @@ static inline bool sk_psock_strp_enabled(struct sk_psock *psock)
return !!psock->saved_data_ready;
}
-/* for tcp only, sk is locked */
-static inline ssize_t sk_psock_msg_inq(struct sock *sk)
-{
- struct sk_psock *psock;
- ssize_t inq = 0;
-
- psock = sk_psock_get(sk);
- if (likely(psock)) {
- inq = sk_psock_get_msg_len_nolock(psock);
- sk_psock_put(sk, psock);
- }
- return inq;
-}
-
/* for udp only, sk is not locked */
static inline ssize_t sk_msg_first_len(struct sock *sk)
{
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 28eba6cbb767..7b58cc184430 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -125,6 +125,15 @@ enum _slab_flag_bits {
* rcu_read_lock before reading the address, then rcu_read_unlock after
* taking the spinlock within the structure expected at that address.
*
+ * Note that object identity check has to be done *after* acquiring a
+ * reference, therefore user has to ensure proper ordering for loads.
+ * Similarly, when initializing objects allocated with SLAB_TYPESAFE_BY_RCU,
+ * the newly allocated object has to be fully initialized *before* its
+ * refcount gets initialized and proper ordering for stores is required.
+ * refcount_{add|inc}_not_zero_acquire() and refcount_set_release() are
+ * designed with the proper fences required for reference counting objects
+ * allocated with SLAB_TYPESAFE_BY_RCU.
+ *
* Note that it is not possible to acquire a lock within a structure
* allocated with SLAB_TYPESAFE_BY_RCU without first acquiring a reference
* as described above. The reason is that SLAB_TYPESAFE_BY_RCU pages
diff --git a/include/linux/soc/mediatek/mtk_wed.h b/include/linux/soc/mediatek/mtk_wed.h
index a476648858a6..9b569b1300b9 100644
--- a/include/linux/soc/mediatek/mtk_wed.h
+++ b/include/linux/soc/mediatek/mtk_wed.h
@@ -154,6 +154,7 @@ struct mtk_wed_device {
bool wcid_512;
bool hw_rro;
bool msi;
+ bool hif2;
u16 token_start;
unsigned int nbuf;
diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h
index 4d96bbdb45f0..6f39d3d8ef87 100644
--- a/include/linux/srcutiny.h
+++ b/include/linux/srcutiny.h
@@ -11,6 +11,7 @@
#ifndef _LINUX_SRCU_TINY_H
#define _LINUX_SRCU_TINY_H
+#include <linux/irq_work_types.h>
#include <linux/swait.h>
struct srcu_struct {
@@ -24,18 +25,21 @@ struct srcu_struct {
struct rcu_head *srcu_cb_head; /* Pending callbacks: Head. */
struct rcu_head **srcu_cb_tail; /* Pending callbacks: Tail. */
struct work_struct srcu_work; /* For driving grace periods. */
+ struct irq_work srcu_irq_work; /* Defer schedule_work() to irq work. */
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lockdep_map dep_map;
#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
};
void srcu_drive_gp(struct work_struct *wp);
+void srcu_tiny_irq_work(struct irq_work *irq_work);
#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \
{ \
.srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \
.srcu_cb_tail = &name.srcu_cb_head, \
.srcu_work = __WORK_INITIALIZER(name.srcu_work, srcu_drive_gp), \
+ .srcu_irq_work = { .func = srcu_tiny_irq_work }, \
__SRCU_DEP_MAP_INIT(name) \
}
diff --git a/include/linux/stddef.h b/include/linux/stddef.h
index 929d67710cc5..dab49e2ec8c0 100644
--- a/include/linux/stddef.h
+++ b/include/linux/stddef.h
@@ -93,4 +93,24 @@ enum {
#define DECLARE_FLEX_ARRAY(TYPE, NAME) \
__DECLARE_FLEX_ARRAY(TYPE, NAME)
+/**
+ * TRAILING_OVERLAP() - Overlap a flexible-array member with trailing members.
+ *
+ * Creates a union between a flexible-array member (FAM) in a struct and a set
+ * of additional members that would otherwise follow it.
+ *
+ * @TYPE: Flexible structure type name, including "struct" keyword.
+ * @NAME: Name for a variable to define.
+ * @FAM: The flexible-array member within @TYPE
+ * @MEMBERS: Trailing overlapping members.
+ */
+#define TRAILING_OVERLAP(TYPE, NAME, FAM, MEMBERS) \
+ union { \
+ TYPE NAME; \
+ struct { \
+ unsigned char __offset_to_##FAM[offsetof(TYPE, FAM)]; \
+ MEMBERS \
+ }; \
+ }
+
#endif
diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h
index 1016f2feddc4..65ce45d95f1c 100644
--- a/include/linux/sunrpc/svc_rdma.h
+++ b/include/linux/sunrpc/svc_rdma.h
@@ -213,6 +213,7 @@ struct svc_rdma_recv_ctxt {
*/
struct svc_rdma_write_info {
struct svcxprt_rdma *wi_rdma;
+ struct list_head wi_list;
const struct svc_rdma_chunk *wi_chunk;
@@ -241,7 +242,10 @@ struct svc_rdma_send_ctxt {
struct ib_cqe sc_cqe;
struct xdr_buf sc_hdrbuf;
struct xdr_stream sc_stream;
+
+ struct list_head sc_write_info_list;
struct svc_rdma_write_info sc_reply_info;
+
void *sc_xprt_buf;
int sc_page_count;
int sc_cur_sge_no;
@@ -273,11 +277,14 @@ extern void svc_rdma_cc_init(struct svcxprt_rdma *rdma,
extern void svc_rdma_cc_release(struct svcxprt_rdma *rdma,
struct svc_rdma_chunk_ctxt *cc,
enum dma_data_direction dir);
+extern void svc_rdma_write_chunk_release(struct svcxprt_rdma *rdma,
+ struct svc_rdma_send_ctxt *ctxt);
extern void svc_rdma_reply_chunk_release(struct svcxprt_rdma *rdma,
struct svc_rdma_send_ctxt *ctxt);
-extern int svc_rdma_send_write_list(struct svcxprt_rdma *rdma,
- const struct svc_rdma_recv_ctxt *rctxt,
- const struct xdr_buf *xdr);
+extern int svc_rdma_prepare_write_list(struct svcxprt_rdma *rdma,
+ const struct svc_rdma_recv_ctxt *rctxt,
+ struct svc_rdma_send_ctxt *sctxt,
+ const struct xdr_buf *xdr);
extern int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma,
const struct svc_rdma_pcl *write_pcl,
const struct svc_rdma_pcl *reply_pcl,
@@ -310,6 +317,7 @@ extern int svc_rdma_result_payload(struct svc_rqst *rqstp, unsigned int offset,
unsigned int length);
/* svc_rdma_transport.c */
+extern void svc_rdma_xprt_deferred_close(struct svcxprt_rdma *rdma);
extern struct svc_xprt_class svc_rdma_class;
#ifdef CONFIG_SUNRPC_BACKCHANNEL
extern struct svc_xprt_class svc_rdma_bc_class;
diff --git a/include/linux/syscore_ops.h b/include/linux/syscore_ops.h
index ae4d48e4c970..ac6d71be5c38 100644
--- a/include/linux/syscore_ops.h
+++ b/include/linux/syscore_ops.h
@@ -11,14 +11,19 @@
#include <linux/list.h>
struct syscore_ops {
+ int (*suspend)(void *data);
+ void (*resume)(void *data);
+ void (*shutdown)(void *data);
+};
+
+struct syscore {
struct list_head node;
- int (*suspend)(void);
- void (*resume)(void);
- void (*shutdown)(void);
+ const struct syscore_ops *ops;
+ void *data;
};
-extern void register_syscore_ops(struct syscore_ops *ops);
-extern void unregister_syscore_ops(struct syscore_ops *ops);
+extern void register_syscore(struct syscore *syscore);
+extern void unregister_syscore(struct syscore *syscore);
#ifdef CONFIG_PM_SLEEP
extern int syscore_suspend(void);
extern void syscore_resume(void);
diff --git a/include/linux/types.h b/include/linux/types.h
index 2d7b9ae8714c..f1356a9a5730 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -248,5 +248,17 @@ typedef void (*swap_func_t)(void *a, void *b, int size);
typedef int (*cmp_r_func_t)(const void *a, const void *b, const void *priv);
typedef int (*cmp_func_t)(const void *a, const void *b);
+/*
+ * rcuwait provides a way of blocking and waking up a single
+ * task in an rcu-safe manner.
+ *
+ * The only time @task is non-nil is when a user is blocked (or
+ * checking if it needs to) on a condition, and reset as soon as we
+ * know that the condition has succeeded and are awoken.
+ */
+struct rcuwait {
+ struct task_struct __rcu *task;
+};
+
#endif /* __ASSEMBLY__ */
#endif /* _LINUX_TYPES_H */
diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
index 2e7a30fe6b92..c1da2f6ee5d3 100644
--- a/include/linux/vdpa.h
+++ b/include/linux/vdpa.h
@@ -71,9 +71,6 @@ struct vdpa_mgmt_dev;
* struct vdpa_device - representation of a vDPA device
* @dev: underlying device
* @dma_dev: the actual device that is performing DMA
- * @driver_override: driver name to force a match; do not set directly,
- * because core frees it; use driver_set_override() to
- * set or clear it.
* @config: the configuration ops for this device.
* @cf_lock: Protects get and set access to configuration layout.
* @index: device index
@@ -88,7 +85,6 @@ struct vdpa_mgmt_dev;
struct vdpa_device {
struct device dev;
struct device *dma_dev;
- const char *driver_override;
const struct vdpa_config_ops *config;
struct rw_semaphore cf_lock; /* Protects get/set config */
unsigned int index;
diff --git a/include/media/v4l2-mem2mem.h b/include/media/v4l2-mem2mem.h
index 2e55a13ed3bb..1f10f63ef380 100644
--- a/include/media/v4l2-mem2mem.h
+++ b/include/media/v4l2-mem2mem.h
@@ -544,6 +544,27 @@ v4l2_m2m_register_media_controller(struct v4l2_m2m_dev *m2m_dev,
*/
void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev);
+/**
+ * v4l2_m2m_get() - take a reference to the m2m_dev structure
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * This is used to share the M2M device across multiple devices. This
+ * can be used to avoid scheduling two hardware nodes concurrently.
+ */
+void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_put() - remove a reference to the m2m_dev structure
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Once the M2M device has no more references, v4l2_m2m_release() will be
+ * called automatically. Users of this method should never call
+ * v4l2_m2m_release() directly. See v4l2_m2m_get() for more details.
+ */
+void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev);
+
/**
* v4l2_m2m_ctx_init() - allocate and initialize a m2m context
*
diff --git a/include/net/act_api.h b/include/net/act_api.h
index 76c31ce80fe5..29b72635abe7 100644
--- a/include/net/act_api.h
+++ b/include/net/act_api.h
@@ -238,7 +238,7 @@ static inline void tcf_action_update_bstats(struct tc_action *a,
static inline void tcf_action_inc_drop_qstats(struct tc_action *a)
{
if (likely(a->cpu_qstats)) {
- qstats_drop_inc(this_cpu_ptr(a->cpu_qstats));
+ qstats_cpu_drop_inc(a->cpu_qstats);
return;
}
spin_lock(&a->tcfa_lock);
diff --git a/include/net/ax25.h b/include/net/ax25.h
index 4ee141aae0a2..a7bba42dde15 100644
--- a/include/net/ax25.h
+++ b/include/net/ax25.h
@@ -418,7 +418,6 @@ void ax25_rt_device_down(struct net_device *);
int ax25_rt_ioctl(unsigned int, void __user *);
extern const struct seq_operations ax25_rt_seqops;
ax25_route *ax25_get_route(ax25_address *addr, struct net_device *dev);
-int ax25_rt_autobind(ax25_cb *, ax25_address *);
struct sk_buff *ax25_rt_build_path(struct sk_buff *, ax25_address *,
ax25_address *, ax25_digi *);
void ax25_rt_free(void);
diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h
index 763f003ac29e..f846865db0ca 100644
--- a/include/net/bluetooth/hci_core.h
+++ b/include/net/bluetooth/hci_core.h
@@ -751,6 +751,8 @@ struct hci_conn {
struct dentry *debugfs;
struct hci_dev *hdev;
+
+ spinlock_t proto_lock; /* lock guarding protocol data */
void *l2cap_data;
void *sco_data;
void *iso_data;
diff --git a/include/net/bluetooth/rfcomm.h b/include/net/bluetooth/rfcomm.h
index 7e40d3c5f5a9..e8e82869b281 100644
--- a/include/net/bluetooth/rfcomm.h
+++ b/include/net/bluetooth/rfcomm.h
@@ -210,6 +210,7 @@ struct rfcomm_dlc {
#define RFCOMM_AUTH_REJECT 7
#define RFCOMM_DEFER_SETUP 8
#define RFCOMM_ENC_DROP 9
+#define RFCOMM_CLOSED 10
/* Scheduling flags and events */
#define RFCOMM_SCHED_WAKEUP 31
diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h
index ede91e4709ee..83558dec5132 100644
--- a/include/net/cfg80211.h
+++ b/include/net/cfg80211.h
@@ -2049,6 +2049,99 @@ struct cfg80211_tid_stats {
#define IEEE80211_MAX_CHAINS 4
+/**
+ * struct link_station_info - link station information
+ *
+ * Link station information filled by driver for get_station() and
+ * dump_station().
+ * @filled: bit flag of flags using the bits of &enum nl80211_sta_info to
+ * indicate the relevant values in this struct for them
+ * @connected_time: time(in secs) since a link of station is last connected
+ * @inactive_time: time since last activity for link station(tx/rx)
+ * in milliseconds
+ * @assoc_at: bootime (ns) of the last association of link of station
+ * @rx_bytes: bytes (size of MPDUs) received from this link of station
+ * @tx_bytes: bytes (size of MPDUs) transmitted to this link of station
+ * @signal: The signal strength, type depends on the wiphy's signal_type.
+ * For CFG80211_SIGNAL_TYPE_MBM, value is expressed in _dBm_.
+ * @signal_avg: Average signal strength, type depends on the wiphy's
+ * signal_type. For CFG80211_SIGNAL_TYPE_MBM, value is expressed in _dBm_
+ * @chains: bitmask for filled values in @chain_signal, @chain_signal_avg
+ * @chain_signal: per-chain signal strength of last received packet in dBm
+ * @chain_signal_avg: per-chain signal strength average in dBm
+ * @txrate: current unicast bitrate from this link of station
+ * @rxrate: current unicast bitrate to this link of station
+ * @rx_packets: packets (MSDUs & MMPDUs) received from this link of station
+ * @tx_packets: packets (MSDUs & MMPDUs) transmitted to this link of station
+ * @tx_retries: cumulative retry counts (MPDUs) for this link of station
+ * @tx_failed: number of failed transmissions (MPDUs) (retries exceeded, no ACK)
+ * @rx_dropped_misc: Dropped for un-specified reason.
+ * @bss_param: current BSS parameters
+ * @beacon_loss_count: Number of times beacon loss event has triggered.
+ * @expected_throughput: expected throughput in kbps (including 802.11 headers)
+ * towards this station.
+ * @rx_beacon: number of beacons received from this peer
+ * @rx_beacon_signal_avg: signal strength average (in dBm) for beacons received
+ * from this peer
+ * @rx_duration: aggregate PPDU duration(usecs) for all the frames from a peer
+ * @tx_duration: aggregate PPDU duration(usecs) for all the frames to a peer
+ * @airtime_weight: current airtime scheduling weight
+ * @pertid: per-TID statistics, see &struct cfg80211_tid_stats, using the last
+ * (IEEE80211_NUM_TIDS) index for MSDUs not encapsulated in QoS-MPDUs.
+ * Note that this doesn't use the @filled bit, but is used if non-NULL.
+ * @ack_signal: signal strength (in dBm) of the last ACK frame.
+ * @avg_ack_signal: average rssi value of ack packet for the no of msdu's has
+ * been sent.
+ * @rx_mpdu_count: number of MPDUs received from this station
+ * @fcs_err_count: number of packets (MPDUs) received from this station with
+ * an FCS error. This counter should be incremented only when TA of the
+ * received packet with an FCS error matches the peer MAC address.
+ * @addr: For MLO STA connection, filled with address of the link of station.
+ */
+struct link_station_info {
+ u64 filled;
+ u32 connected_time;
+ u32 inactive_time;
+ u64 assoc_at;
+ u64 rx_bytes;
+ u64 tx_bytes;
+ s8 signal;
+ s8 signal_avg;
+
+ u8 chains;
+ s8 chain_signal[IEEE80211_MAX_CHAINS];
+ s8 chain_signal_avg[IEEE80211_MAX_CHAINS];
+
+ struct rate_info txrate;
+ struct rate_info rxrate;
+ u32 rx_packets;
+ u32 tx_packets;
+ u32 tx_retries;
+ u32 tx_failed;
+ u32 rx_dropped_misc;
+ struct sta_bss_parameters bss_param;
+
+ u32 beacon_loss_count;
+
+ u32 expected_throughput;
+
+ u64 tx_duration;
+ u64 rx_duration;
+ u64 rx_beacon;
+ u8 rx_beacon_signal_avg;
+
+ u16 airtime_weight;
+
+ s8 ack_signal;
+ s8 avg_ack_signal;
+ struct cfg80211_tid_stats *pertid;
+
+ u32 rx_mpdu_count;
+ u32 fcs_err_count;
+
+ u8 addr[ETH_ALEN] __aligned(2);
+};
+
/**
* struct station_info - station information
*
@@ -2133,6 +2226,11 @@ struct cfg80211_tid_stats {
* dump_station() callbacks. User space needs this information to determine
* the accepted and rejected affiliated links of the connected station.
* @assoc_resp_ies_len: Length of @assoc_resp_ies buffer in octets.
+ * @valid_links: bitmap of valid links, or 0 for non-MLO. Drivers fill this
+ * information in cfg80211_new_sta(), cfg80211_del_sta_sinfo(),
+ * get_station() and dump_station() callbacks.
+ * @links: reference to Link sta entries for MLO STA, all link specific
+ * information is accessed through links[link_id].
*/
struct station_info {
u64 filled;
@@ -2198,6 +2296,9 @@ struct station_info {
u8 mld_addr[ETH_ALEN] __aligned(2);
const u8 *assoc_resp_ies;
size_t assoc_resp_ies_len;
+
+ u16 valid_links;
+ struct link_station_info *links[IEEE80211_MLD_MAX_NUM_LINKS];
};
/**
@@ -6389,7 +6490,7 @@ struct wireless_dev {
enum ieee80211_bss_type conn_bss_type;
u32 conn_owner_nlportid;
- struct work_struct disconnect_wk;
+ struct wiphy_work disconnect_wk;
u8 disconnect_bssid[ETH_ALEN];
struct list_head event_list;
diff --git a/include/net/dropreason-core.h b/include/net/dropreason-core.h
index 07826c85d31c..31e0cc1b64ff 100644
--- a/include/net/dropreason-core.h
+++ b/include/net/dropreason-core.h
@@ -60,12 +60,16 @@
FN(TC_EGRESS) \
FN(SECURITY_HOOK) \
FN(QDISC_DROP) \
+ FN(FQ_BAND_LIMIT) \
+ FN(FQ_HORIZON_LIMIT) \
+ FN(FQ_FLOW_LIMIT) \
FN(CPU_BACKLOG) \
FN(XDP) \
FN(TC_INGRESS) \
FN(UNHANDLED_PROTO) \
FN(SKB_CSUM) \
FN(SKB_GSO_SEG) \
+ FN(SKB_BAD_GSO) \
FN(SKB_UCOPY_FAULT) \
FN(DEV_HDR) \
FN(DEV_READY) \
@@ -315,6 +319,21 @@ enum skb_drop_reason {
* failed to enqueue to current qdisc)
*/
SKB_DROP_REASON_QDISC_DROP,
+ /**
+ * @SKB_DROP_REASON_FQ_BAND_LIMIT: dropped by fq qdisc when per band
+ * limit is reached.
+ */
+ SKB_DROP_REASON_FQ_BAND_LIMIT,
+ /**
+ * @SKB_DROP_REASON_FQ_HORIZON_LIMIT: dropped by fq qdisc when packet
+ * timestamp is too far in the future.
+ */
+ SKB_DROP_REASON_FQ_HORIZON_LIMIT,
+ /**
+ * @SKB_DROP_REASON_FQ_FLOW_LIMIT: dropped by fq qdisc when a flow
+ * exceeds its limits.
+ */
+ SKB_DROP_REASON_FQ_FLOW_LIMIT,
/**
* @SKB_DROP_REASON_CPU_BACKLOG: failed to enqueue the skb to the per CPU
* backlog queue. This can be caused by backlog queue full (see
@@ -331,6 +350,8 @@ enum skb_drop_reason {
SKB_DROP_REASON_SKB_CSUM,
/** @SKB_DROP_REASON_SKB_GSO_SEG: gso segmentation error */
SKB_DROP_REASON_SKB_GSO_SEG,
+ /** @SKB_DROP_REASON_SKB_BAD_GSO: malicious gso packet. */
+ SKB_DROP_REASON_SKB_BAD_GSO,
/**
* @SKB_DROP_REASON_SKB_UCOPY_FAULT: failed to copy data from user space,
* e.g., via zerocopy_sg_from_iter() or skb_orphan_frags_rx()
diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h
index 35f012f62015..d2bfc9f67082 100644
--- a/include/net/ip_vs.h
+++ b/include/net/ip_vs.h
@@ -1713,7 +1713,7 @@ static inline char ip_vs_fwd_tag(struct ip_vs_conn *cp)
return fwd;
}
-void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
+bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct ip_vs_conn *cp, int dir, unsigned int toff,
bool has_ports, struct ip_vs_iphdr *ciph);
diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h
index 47181fd749b2..f659d1efcd3c 100644
--- a/include/net/net_namespace.h
+++ b/include/net/net_namespace.h
@@ -470,6 +470,8 @@ struct pernet_operations {
void (*exit)(struct net *net);
void (*exit_batch)(struct list_head *net_exit_list);
/* Following method is called with RTNL held. */
+ void (*exit_rtnl)(struct net *net,
+ struct list_head *dev_kill_list);
void (*exit_batch_rtnl)(struct list_head *net_exit_list,
struct list_head *dev_kill_list);
unsigned int * const id;
diff --git a/include/net/netfilter/nf_conntrack_expect.h b/include/net/netfilter/nf_conntrack_expect.h
index 80f50fd0f7ad..be4a120d549e 100644
--- a/include/net/netfilter/nf_conntrack_expect.h
+++ b/include/net/netfilter/nf_conntrack_expect.h
@@ -54,8 +54,8 @@ struct nf_conntrack_expect {
/* The conntrack of the master connection */
struct nf_conn *master;
- /* Timer function; deletes the expectation. */
- struct timer_list timeout;
+ /* jiffies32 when this expectation expires */
+ u32 timeout;
#if IS_ENABLED(CONFIG_NF_NAT)
union nf_inet_addr saved_addr;
@@ -69,6 +69,14 @@ struct nf_conntrack_expect {
struct rcu_head rcu;
};
+static inline bool nf_ct_exp_is_expired(const struct nf_conntrack_expect *exp)
+{
+ if (READ_ONCE(exp->flags) & NF_CT_EXPECT_DEAD)
+ return true;
+
+ return (__s32)(READ_ONCE(exp->timeout) - nfct_time_stamp) <= 0;
+}
+
static inline struct net *nf_ct_exp_net(struct nf_conntrack_expect *exp)
{
return read_pnet(&exp->net);
@@ -130,7 +138,6 @@ static inline void nf_ct_unlink_expect(struct nf_conntrack_expect *exp)
void nf_ct_remove_expectations(struct nf_conn *ct);
void nf_ct_unexpect_related(struct nf_conntrack_expect *exp);
-bool nf_ct_remove_expect(struct nf_conntrack_expect *exp);
void nf_ct_expect_iterate_destroy(bool (*iter)(struct nf_conntrack_expect *e, void *data), void *data);
void nf_ct_expect_iterate_net(struct net *net,
@@ -153,5 +160,8 @@ static inline int nf_ct_expect_related(struct nf_conntrack_expect *expect,
return nf_ct_expect_related_report(expect, 0, 0, flags);
}
+struct nf_conn_help;
+void nf_ct_expectation_gc(struct nf_conn_help *master_help);
+
#endif /*_NF_CONNTRACK_EXPECT_H*/
diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h
index 923b2c0d6e84..eaddf2aacb2f 100644
--- a/include/net/netfilter/nf_tables.h
+++ b/include/net/netfilter/nf_tables.h
@@ -177,6 +177,13 @@ static inline u64 nft_reg_load64(const u32 *sreg)
return get_unaligned((u64 *)sreg);
}
+static inline bool nft_reg_overlap(u8 src, u8 dst, u32 len)
+{
+ unsigned int n = DIV_ROUND_UP(len, sizeof(u32));
+
+ return src != dst && src < dst + n && dst < src + n;
+}
+
static inline void nft_data_copy(u32 *dst, const struct nft_data *src,
unsigned int len)
{
@@ -306,11 +313,13 @@ static inline void *nft_elem_priv_cast(const struct nft_elem_priv *priv)
* @NFT_ITER_UNSPEC: unspecified, to catch errors
* @NFT_ITER_READ: read-only iteration over set elements
* @NFT_ITER_UPDATE: iteration under mutex to update set element state
+ * @NFT_ITER_UPDATE_CLONE: clone set before iteration under mutex to update element
*/
enum nft_iter_type {
NFT_ITER_UNSPEC,
NFT_ITER_READ,
NFT_ITER_UPDATE,
+ NFT_ITER_UPDATE_CLONE,
};
struct nft_set;
@@ -858,8 +867,6 @@ struct nft_elem_priv *nft_set_elem_init(const struct nft_set *set,
const u32 *key, const u32 *key_end,
const u32 *data,
u64 timeout, u64 expiration, gfp_t gfp);
-int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set,
- struct nft_expr *expr_array[]);
void nft_set_elem_expr_destroy(const struct nft_ctx *ctx,
struct nft_set_elem_expr *elem_expr);
void nft_set_elem_destroy(const struct nft_set *set,
@@ -1765,28 +1772,25 @@ enum nft_trans_elem_flags {
NFT_TRANS_UPD_EXPIRATION = (1 << 1),
};
-struct nft_trans_elem {
- struct nft_trans nft_trans;
- struct nft_set *set;
- struct nft_elem_priv *elem_priv;
+struct nft_trans_one_elem {
+ struct nft_elem_priv *priv;
u64 timeout;
u64 expiration;
u8 update_flags;
+};
+
+struct nft_trans_elem {
+ struct nft_trans nft_trans;
+ struct nft_set *set;
bool bound;
+ unsigned int nelems;
+ struct nft_trans_one_elem elems[] __counted_by(nelems);
};
#define nft_trans_container_elem(t) \
container_of(t, struct nft_trans_elem, nft_trans)
#define nft_trans_elem_set(trans) \
nft_trans_container_elem(trans)->set
-#define nft_trans_elem_priv(trans) \
- nft_trans_container_elem(trans)->elem_priv
-#define nft_trans_elem_update_flags(trans) \
- nft_trans_container_elem(trans)->update_flags
-#define nft_trans_elem_timeout(trans) \
- nft_trans_container_elem(trans)->timeout
-#define nft_trans_elem_expiration(trans) \
- nft_trans_container_elem(trans)->expiration
#define nft_trans_elem_set_bound(trans) \
nft_trans_container_elem(trans)->bound
diff --git a/include/net/netns/mpls.h b/include/net/netns/mpls.h
index 19ad2574b267..2073cbac2afb 100644
--- a/include/net/netns/mpls.h
+++ b/include/net/netns/mpls.h
@@ -16,6 +16,8 @@ struct netns_mpls {
int default_ttl;
size_t platform_labels;
struct mpls_route __rcu * __rcu *platform_label;
+ struct mutex platform_mutex;
+ seqcount_mutex_t platform_label_seq;
struct ctl_table_header *ctl;
};
diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h
index ea2c69fdc839..28ce971bddc9 100644
--- a/include/net/pkt_cls.h
+++ b/include/net/pkt_cls.h
@@ -156,8 +156,20 @@ static inline int tcf_classify(struct sk_buff *skb,
{
return TC_ACT_UNSPEC;
}
-
#endif
+static inline int tcf_classify_qdisc(struct sk_buff *skb,
+ const struct tcf_proto *tp,
+ struct tcf_result *res, bool compat_mode)
+{
+ int ret = tcf_classify(skb, NULL, tp, res, compat_mode);
+
+ /* TC_ACT_REDIRECT from qdisc filter chains is not supported.
+ * Use BPF via tcx or mirred redirect instead.
+ */
+ if (unlikely(ret == TC_ACT_REDIRECT))
+ ret = TC_ACT_SHOT;
+ return ret;
+}
static inline unsigned long
__cls_set_class(unsigned long *clp, unsigned long cl)
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index 4f38a7a5dd6a..5f83b4970f79 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -976,12 +976,17 @@ static inline void qdisc_qstats_cpu_requeues_inc(struct Qdisc *sch)
static inline void __qdisc_qstats_drop(struct Qdisc *sch, int count)
{
- sch->qstats.drops += count;
+ WRITE_ONCE(sch->qstats.drops, sch->qstats.drops + count);
}
static inline void qstats_drop_inc(struct gnet_stats_queue *qstats)
{
- qstats->drops++;
+ WRITE_ONCE(qstats->drops, qstats->drops + 1);
+}
+
+static inline void qstats_cpu_drop_inc(struct gnet_stats_queue __percpu *qstats)
+{
+ this_cpu_inc(qstats->drops);
}
static inline void qstats_overlimit_inc(struct gnet_stats_queue *qstats)
@@ -1094,12 +1099,17 @@ static inline struct sk_buff *qdisc_dequeue_internal(struct Qdisc *sch, bool dir
skb = __skb_dequeue(&sch->gso_skb);
if (skb) {
sch->q.qlen--;
+ qdisc_qstats_backlog_dec(sch, skb);
return skb;
}
- if (direct)
- return __qdisc_dequeue_head(&sch->q);
- else
+ if (direct) {
+ skb = __qdisc_dequeue_head(&sch->q);
+ if (skb)
+ qdisc_qstats_backlog_dec(sch, skb);
+ return skb;
+ } else {
return sch->dequeue(sch);
+ }
}
static inline struct sk_buff *qdisc_dequeue_head(struct Qdisc *sch)
@@ -1315,6 +1325,14 @@ static inline int qdisc_drop(struct sk_buff *skb, struct Qdisc *sch,
return NET_XMIT_DROP;
}
+static inline int qdisc_drop_reason(struct sk_buff *skb, struct Qdisc *sch,
+ struct sk_buff **to_free,
+ enum skb_drop_reason reason)
+{
+ tcf_set_drop_reason(skb, reason);
+ return qdisc_drop(skb, sch, to_free);
+}
+
static inline int qdisc_drop_all(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
diff --git a/include/net/sctp/auth.h b/include/net/sctp/auth.h
index d4b3b2dcd15b..8fcc8bb1439c 100644
--- a/include/net/sctp/auth.h
+++ b/include/net/sctp/auth.h
@@ -23,6 +23,7 @@ struct sctp_association;
struct sctp_authkey;
struct sctp_hmacalgo;
struct crypto_shash;
+struct sctp_cookie;
/*
* Define a generic struct that will hold all the info
@@ -81,6 +82,8 @@ int sctp_auth_init_hmacs(struct sctp_endpoint *ep, gfp_t gfp);
void sctp_auth_destroy_hmacs(struct crypto_shash *auth_hmacs[]);
struct sctp_hmac *sctp_auth_get_hmac(__u16 hmac_id);
struct sctp_hmac *sctp_auth_asoc_get_hmac(const struct sctp_association *asoc);
+bool sctp_auth_verify_cookie_params(const struct sctp_endpoint *ep,
+ const struct sctp_cookie *cookie);
void sctp_auth_asoc_set_default_hmac(struct sctp_association *asoc,
struct sctp_hmac_algo_param *hmacs);
int sctp_auth_asoc_verify_hmac_id(const struct sctp_association *asoc,
diff --git a/include/net/tcp.h b/include/net/tcp.h
index ad9af706085a..77f6c40ead43 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -54,7 +54,15 @@ extern struct inet_hashinfo tcp_hashinfo;
DECLARE_PER_CPU(unsigned int, tcp_orphan_count);
int tcp_orphan_count_sum(void);
-DECLARE_PER_CPU(u32, tcp_tw_isn);
+static inline void tcp_orphan_count_inc(void)
+{
+ this_cpu_inc(tcp_orphan_count);
+}
+
+static inline void tcp_orphan_count_dec(void)
+{
+ this_cpu_dec(tcp_orphan_count);
+}
void tcp_time_wait(struct sock *sk, int state, int timeo);
@@ -937,10 +945,13 @@ struct tcp_skb_cb {
__u32 seq; /* Starting sequence number */
__u32 end_seq; /* SEQ + FIN + SYN + datalen */
union {
- /* Note :
+ /* Notes :
+ * tcp_tw_isn is used in input path only
+ * (isn chosen by tcp_timewait_state_process())
* tcp_gso_segs/size are used in write queue only,
* cf tcp_skb_pcount()/tcp_skb_mss()
*/
+ u32 tcp_tw_isn;
struct {
u16 tcp_gso_segs;
u16 tcp_gso_size;
diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h
index 98491d774d82..02a0896f8525 100644
--- a/include/net/xdp_sock.h
+++ b/include/net/xdp_sock.h
@@ -80,6 +80,7 @@ struct xdp_sock {
* call of __xsk_generic_xmit().
*/
struct sk_buff *skb;
+ bool drain_cont;
struct list_head map_list;
/* Protects map_list */
diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h
index 8d254e4b3571..71e219a74974 100644
--- a/include/net/xdp_sock_drv.h
+++ b/include/net/xdp_sock_drv.h
@@ -232,11 +232,11 @@ xsk_buff_valid_tx_metadata(const struct xsk_tx_metadata *meta)
return !(meta->flags & ~XDP_TXMD_FLAGS_VALID);
}
-static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr)
+static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr, u32 options)
{
struct xsk_tx_metadata *meta;
- if (!pool->tx_metadata_len)
+ if (!pool->tx_metadata_len || !(options & XDP_TX_METADATA))
return NULL;
meta = xp_raw_get_data(pool, addr) - pool->tx_metadata_len;
@@ -426,7 +426,7 @@ static inline bool xsk_buff_valid_tx_metadata(struct xsk_tx_metadata *meta)
return false;
}
-static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr)
+static inline struct xsk_tx_metadata *xsk_buff_get_metadata(struct xsk_buff_pool *pool, u64 addr, u32 options)
{
return NULL;
}
diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h
index 42e9058d8568..4fce4bd826b9 100644
--- a/include/net/xsk_buff_pool.h
+++ b/include/net/xsk_buff_pool.h
@@ -78,6 +78,9 @@ struct xsk_buff_pool {
u32 chunk_size;
u32 chunk_shift;
u32 frame_len;
+ u32 tx_descs_nentries;
+ u32 reclaim_descs;
+ u32 tx_zc_pending_descs;
u8 tx_metadata_len; /* inherited from umem */
u8 cached_need_wakeup;
bool uses_need_wakeup;
@@ -101,12 +104,14 @@ struct xsk_buff_pool {
/* AF_XDP core. */
struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
- struct xdp_umem *umem);
+ struct xdp_umem *umem,
+ u32 max_segs);
int xp_assign_dev(struct xsk_buff_pool *pool, struct net_device *dev,
u16 queue_id, u16 flags);
int xp_assign_dev_shared(struct xsk_buff_pool *pool, struct xdp_sock *umem_xs,
struct net_device *dev, u16 queue_id);
-int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs);
+int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs,
+ u32 max_segs);
void xp_destroy(struct xsk_buff_pool *pool);
void xp_get_pool(struct xsk_buff_pool *pool);
bool xp_put_pool(struct xsk_buff_pool *pool);
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 990747f803fe..28ab13c6ac61 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -633,6 +633,10 @@ struct Scsi_Host {
*/
unsigned nr_hw_queues;
unsigned nr_maps;
+
+ /* Don't resume host in EH */
+ bool eh_noresume;
+
unsigned active_mode:2;
/*
@@ -654,9 +658,6 @@ struct Scsi_Host {
/* Asynchronous scan in progress */
unsigned async_scan:1;
- /* Don't resume host in EH */
- unsigned eh_noresume:1;
-
/* The controller does not support WRITE SAME */
unsigned no_write_same:1;
diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h
index 1d9b068bb175..f68fdb7ca7c0 100644
--- a/include/trace/events/netfs.h
+++ b/include/trace/events/netfs.h
@@ -199,6 +199,18 @@
EM(netfs_trace_donate_to_next, "to-next") \
E_(netfs_trace_donate_to_deferred_next, "defer-next")
+#define netfs_folioq_traces \
+ EM(netfs_trace_folioq_alloc_append_folio, "alloc-apf") \
+ EM(netfs_trace_folioq_alloc_read_prep, "alloc-r-prep") \
+ EM(netfs_trace_folioq_alloc_read_prime, "alloc-r-prime") \
+ EM(netfs_trace_folioq_alloc_read_sing, "alloc-r-sing") \
+ EM(netfs_trace_folioq_clear, "clear") \
+ EM(netfs_trace_folioq_delete, "delete") \
+ EM(netfs_trace_folioq_make_space, "make-space") \
+ EM(netfs_trace_folioq_prep_write, "prep-wr") \
+ EM(netfs_trace_folioq_rollbuf_init, "roll-init") \
+ E_(netfs_trace_folioq_read_progress, "r-progress")
+
#ifndef __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY
#define __NETFS_DECLARE_TRACE_ENUMS_ONCE_ONLY
@@ -217,6 +229,7 @@ enum netfs_sreq_ref_trace { netfs_sreq_ref_traces } __mode(byte);
enum netfs_folio_trace { netfs_folio_traces } __mode(byte);
enum netfs_collect_contig_trace { netfs_collect_contig_traces } __mode(byte);
enum netfs_donate_trace { netfs_donate_traces } __mode(byte);
+enum netfs_folioq_trace { netfs_folioq_traces } __mode(byte);
#endif
@@ -240,6 +253,7 @@ netfs_sreq_ref_traces;
netfs_folio_traces;
netfs_collect_contig_traces;
netfs_donate_traces;
+netfs_folioq_traces;
/*
* Now redefine the EM() and E_() macros to map the enums to the strings that
@@ -258,13 +272,13 @@ TRACE_EVENT(netfs_read,
TP_ARGS(rreq, start, len, what),
TP_STRUCT__entry(
- __field(unsigned int, rreq )
- __field(unsigned int, cookie )
- __field(loff_t, i_size )
- __field(loff_t, start )
- __field(size_t, len )
- __field(enum netfs_read_trace, what )
- __field(unsigned int, netfs_inode )
+ __field(unsigned int, rreq)
+ __field(unsigned int, cookie)
+ __field(loff_t, i_size)
+ __field(loff_t, start)
+ __field(size_t, len)
+ __field(enum netfs_read_trace, what)
+ __field(unsigned int, netfs_inode)
),
TP_fast_assign(
@@ -292,10 +306,10 @@ TRACE_EVENT(netfs_rreq,
TP_ARGS(rreq, what),
TP_STRUCT__entry(
- __field(unsigned int, rreq )
- __field(unsigned int, flags )
- __field(enum netfs_io_origin, origin )
- __field(enum netfs_rreq_trace, what )
+ __field(unsigned int, rreq)
+ __field(unsigned int, flags)
+ __field(enum netfs_io_origin, origin)
+ __field(enum netfs_rreq_trace, what)
),
TP_fast_assign(
@@ -319,15 +333,16 @@ TRACE_EVENT(netfs_sreq,
TP_ARGS(sreq, what),
TP_STRUCT__entry(
- __field(unsigned int, rreq )
- __field(unsigned short, index )
- __field(short, error )
- __field(unsigned short, flags )
- __field(enum netfs_io_source, source )
- __field(enum netfs_sreq_trace, what )
- __field(size_t, len )
- __field(size_t, transferred )
- __field(loff_t, start )
+ __field(unsigned int, rreq)
+ __field(unsigned short, index)
+ __field(short, error)
+ __field(unsigned short, flags)
+ __field(enum netfs_io_source, source)
+ __field(enum netfs_sreq_trace, what)
+ __field(u8, slot)
+ __field(size_t, len)
+ __field(size_t, transferred)
+ __field(loff_t, start)
),
TP_fast_assign(
@@ -340,15 +355,16 @@ TRACE_EVENT(netfs_sreq,
__entry->len = sreq->len;
__entry->transferred = sreq->transferred;
__entry->start = sreq->start;
+ __entry->slot = sreq->curr_folioq_slot;
),
- TP_printk("R=%08x[%x] %s %s f=%02x s=%llx %zx/%zx e=%d",
+ TP_printk("R=%08x[%x] %s %s f=%02x s=%llx %zx/%zx s=%u e=%d",
__entry->rreq, __entry->index,
__print_symbolic(__entry->source, netfs_sreq_sources),
__print_symbolic(__entry->what, netfs_sreq_traces),
__entry->flags,
__entry->start, __entry->transferred, __entry->len,
- __entry->error)
+ __entry->slot, __entry->error)
);
TRACE_EVENT(netfs_failure,
@@ -359,15 +375,15 @@ TRACE_EVENT(netfs_failure,
TP_ARGS(rreq, sreq, error, what),
TP_STRUCT__entry(
- __field(unsigned int, rreq )
- __field(short, index )
- __field(short, error )
- __field(unsigned short, flags )
- __field(enum netfs_io_source, source )
- __field(enum netfs_failure, what )
- __field(size_t, len )
- __field(size_t, transferred )
- __field(loff_t, start )
+ __field(unsigned int, rreq)
+ __field(short, index)
+ __field(short, error)
+ __field(unsigned short, flags)
+ __field(enum netfs_io_source, source)
+ __field(enum netfs_failure, what)
+ __field(size_t, len)
+ __field(size_t, transferred)
+ __field(loff_t, start)
),
TP_fast_assign(
@@ -398,9 +414,9 @@ TRACE_EVENT(netfs_rreq_ref,
TP_ARGS(rreq_debug_id, ref, what),
TP_STRUCT__entry(
- __field(unsigned int, rreq )
- __field(int, ref )
- __field(enum netfs_rreq_ref_trace, what )
+ __field(unsigned int, rreq)
+ __field(int, ref)
+ __field(enum netfs_rreq_ref_trace, what)
),
TP_fast_assign(
@@ -422,10 +438,10 @@ TRACE_EVENT(netfs_sreq_ref,
TP_ARGS(rreq_debug_id, subreq_debug_index, ref, what),
TP_STRUCT__entry(
- __field(unsigned int, rreq )
- __field(unsigned int, subreq )
- __field(int, ref )
- __field(enum netfs_sreq_ref_trace, what )
+ __field(unsigned int, rreq)
+ __field(unsigned int, subreq)
+ __field(int, ref)
+ __field(enum netfs_sreq_ref_trace, what)
),
TP_fast_assign(
@@ -473,10 +489,10 @@ TRACE_EVENT(netfs_write_iter,
TP_ARGS(iocb, from),
TP_STRUCT__entry(
- __field(unsigned long long, start )
- __field(size_t, len )
- __field(unsigned int, flags )
- __field(unsigned int, ino )
+ __field(unsigned long long, start)
+ __field(size_t, len)
+ __field(unsigned int, flags)
+ __field(unsigned int, ino)
),
TP_fast_assign(
@@ -497,12 +513,12 @@ TRACE_EVENT(netfs_write,
TP_ARGS(wreq, what),
TP_STRUCT__entry(
- __field(unsigned int, wreq )
- __field(unsigned int, cookie )
- __field(unsigned int, ino )
- __field(enum netfs_write_trace, what )
- __field(unsigned long long, start )
- __field(unsigned long long, len )
+ __field(unsigned int, wreq)
+ __field(unsigned int, cookie)
+ __field(unsigned int, ino)
+ __field(enum netfs_write_trace, what)
+ __field(unsigned long long, start)
+ __field(unsigned long long, len)
),
TP_fast_assign(
@@ -530,10 +546,10 @@ TRACE_EVENT(netfs_collect,
TP_ARGS(wreq),
TP_STRUCT__entry(
- __field(unsigned int, wreq )
- __field(unsigned int, len )
- __field(unsigned long long, transferred )
- __field(unsigned long long, start )
+ __field(unsigned int, wreq)
+ __field(unsigned int, len)
+ __field(unsigned long long, transferred)
+ __field(unsigned long long, start)
),
TP_fast_assign(
@@ -556,12 +572,12 @@ TRACE_EVENT(netfs_collect_sreq,
TP_ARGS(wreq, subreq),
TP_STRUCT__entry(
- __field(unsigned int, wreq )
- __field(unsigned int, subreq )
- __field(unsigned int, stream )
- __field(unsigned int, len )
- __field(unsigned int, transferred )
- __field(unsigned long long, start )
+ __field(unsigned int, wreq)
+ __field(unsigned int, subreq)
+ __field(unsigned int, stream)
+ __field(unsigned int, len)
+ __field(unsigned int, transferred)
+ __field(unsigned long long, start)
),
TP_fast_assign(
@@ -587,11 +603,11 @@ TRACE_EVENT(netfs_collect_folio,
TP_ARGS(wreq, folio, fend, collected_to),
TP_STRUCT__entry(
- __field(unsigned int, wreq )
- __field(unsigned long, index )
- __field(unsigned long long, fend )
- __field(unsigned long long, cleaned_to )
- __field(unsigned long long, collected_to )
+ __field(unsigned int, wreq)
+ __field(unsigned long, index)
+ __field(unsigned long long, fend)
+ __field(unsigned long long, cleaned_to)
+ __field(unsigned long long, collected_to)
),
TP_fast_assign(
@@ -616,10 +632,10 @@ TRACE_EVENT(netfs_collect_state,
TP_ARGS(wreq, collected_to, notes),
TP_STRUCT__entry(
- __field(unsigned int, wreq )
- __field(unsigned int, notes )
- __field(unsigned long long, collected_to )
- __field(unsigned long long, cleaned_to )
+ __field(unsigned int, wreq)
+ __field(unsigned int, notes)
+ __field(unsigned long long, collected_to)
+ __field(unsigned long long, cleaned_to)
),
TP_fast_assign(
@@ -753,6 +769,29 @@ TRACE_EVENT(netfs_donate,
__entry->amount)
);
+TRACE_EVENT(netfs_folioq,
+ TP_PROTO(const struct folio_queue *fq,
+ enum netfs_folioq_trace trace),
+
+ TP_ARGS(fq, trace),
+
+ TP_STRUCT__entry(
+ __field(unsigned int, rreq)
+ __field(unsigned int, id)
+ __field(enum netfs_folioq_trace, trace)
+ ),
+
+ TP_fast_assign(
+ __entry->rreq = fq ? fq->rreq_id : 0;
+ __entry->id = fq ? fq->debug_id : 0;
+ __entry->trace = trace;
+ ),
+
+ TP_printk("R=%08x fq=%x %s",
+ __entry->rreq, __entry->id,
+ __print_symbolic(__entry->trace, netfs_folioq_traces))
+ );
+
#undef EM
#undef E_
#endif /* _TRACE_NETFS_H */
diff --git a/include/trace/events/rv.h b/include/trace/events/rv.h
index 56592da9301c..add8b64a1104 100644
--- a/include/trace/events/rv.h
+++ b/include/trace/events/rv.h
@@ -16,24 +16,24 @@ DECLARE_EVENT_CLASS(event_da_monitor,
TP_ARGS(state, event, next_state, final_state),
TP_STRUCT__entry(
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
- __array( char, next_state, MAX_DA_NAME_LEN )
- __field( bool, final_state )
+ __string( state, state )
+ __string( event, event )
+ __string( next_state, next_state )
+ __field( bool, final_state )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
- memcpy(__entry->next_state, next_state, MAX_DA_NAME_LEN);
- __entry->final_state = final_state;
+ __assign_str(state);
+ __assign_str(event);
+ __assign_str(next_state);
+ __entry->final_state = final_state;
),
- TP_printk("%s x %s -> %s %s",
- __entry->state,
- __entry->event,
- __entry->next_state,
- __entry->final_state ? "(final)" : "")
+ TP_printk("%s x %s -> %s%s",
+ __get_str(state),
+ __get_str(event),
+ __get_str(next_state),
+ __entry->final_state ? " (final)" : "")
);
DECLARE_EVENT_CLASS(error_da_monitor,
@@ -43,18 +43,18 @@ DECLARE_EVENT_CLASS(error_da_monitor,
TP_ARGS(state, event),
TP_STRUCT__entry(
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
+ __string( state, state )
+ __string( event, event )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
+ __assign_str(state);
+ __assign_str(event);
),
TP_printk("event %s not expected in the state %s",
- __entry->event,
- __entry->state)
+ __get_str(event),
+ __get_str(state))
);
#ifdef CONFIG_RV_MON_WIP
@@ -76,27 +76,27 @@ DECLARE_EVENT_CLASS(event_da_monitor_id,
TP_ARGS(id, state, event, next_state, final_state),
TP_STRUCT__entry(
- __field( int, id )
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
- __array( char, next_state, MAX_DA_NAME_LEN )
- __field( bool, final_state )
+ __field( int, id )
+ __string( state, state )
+ __string( event, event )
+ __string( next_state, next_state )
+ __field( bool, final_state )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
- memcpy(__entry->next_state, next_state, MAX_DA_NAME_LEN);
- __entry->id = id;
- __entry->final_state = final_state;
+ __assign_str(state);
+ __assign_str(event);
+ __assign_str(next_state);
+ __entry->id = id;
+ __entry->final_state = final_state;
),
- TP_printk("%d: %s x %s -> %s %s",
+ TP_printk("%d: %s x %s -> %s%s",
__entry->id,
- __entry->state,
- __entry->event,
- __entry->next_state,
- __entry->final_state ? "(final)" : "")
+ __get_str(state),
+ __get_str(event),
+ __get_str(next_state),
+ __entry->final_state ? " (final)" : "")
);
DECLARE_EVENT_CLASS(error_da_monitor_id,
@@ -106,21 +106,21 @@ DECLARE_EVENT_CLASS(error_da_monitor_id,
TP_ARGS(id, state, event),
TP_STRUCT__entry(
- __field( int, id )
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
+ __field( int, id )
+ __string( state, state )
+ __string( event, event )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
- __entry->id = id;
+ __assign_str(state);
+ __assign_str(event);
+ __entry->id = id;
),
TP_printk("%d: event %s not expected in the state %s",
__entry->id,
- __entry->event,
- __entry->state)
+ __get_str(event),
+ __get_str(state))
);
#ifdef CONFIG_RV_MON_WWNR
diff --git a/include/uapi/linux/netfilter/nf_conntrack_common.h b/include/uapi/linux/netfilter/nf_conntrack_common.h
index 56b6b60a814f..ee51045ae1d6 100644
--- a/include/uapi/linux/netfilter/nf_conntrack_common.h
+++ b/include/uapi/linux/netfilter/nf_conntrack_common.h
@@ -160,6 +160,7 @@ enum ip_conntrack_expect_events {
#define NF_CT_EXPECT_USERSPACE 0x4
#ifdef __KERNEL__
+#define NF_CT_EXPECT_DEAD 0x8
#define NF_CT_EXPECT_MASK (NF_CT_EXPECT_PERMANENT | NF_CT_EXPECT_INACTIVE | \
NF_CT_EXPECT_USERSPACE)
#endif
diff --git a/include/ufs/ufshcd.h b/include/ufs/ufshcd.h
index e68ce42eaff4..70dcc13d136d 100644
--- a/include/ufs/ufshcd.h
+++ b/include/ufs/ufshcd.h
@@ -968,6 +968,7 @@ enum ufshcd_mcq_opr {
* @ufs_rtc_update_work: A work for UFS RTC periodic update
* @pm_qos_req: PM QoS request handle
* @pm_qos_enabled: flag to check if pm qos is enabled
+ * @pm_qos_mutex: synchronizes PM QoS request and status updates
*/
struct ufs_hba {
void __iomem *mmio_base;
@@ -1138,6 +1139,8 @@ struct ufs_hba {
struct delayed_work ufs_rtc_update_work;
struct pm_qos_request pm_qos_req;
bool pm_qos_enabled;
+ /* synchronizes PM QoS request and status updates */
+ struct mutex pm_qos_mutex;
};
/**
@@ -1437,10 +1440,6 @@ static inline int ufshcd_disable_host_tx_lcc(struct ufs_hba *hba)
void ufshcd_auto_hibern8_update(struct ufs_hba *hba, u32 ahit);
void ufshcd_fixup_dev_quirks(struct ufs_hba *hba,
const struct ufs_dev_quirk *fixups);
-#define SD_ASCII_STD true
-#define SD_RAW false
-int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
- u8 **buf, bool ascii);
void ufshcd_hold(struct ufs_hba *hba);
void ufshcd_release(struct ufs_hba *hba);
diff --git a/io_uring/eventfd.c b/io_uring/eventfd.c
index ffc4bd17d078..6a6ec9d4d640 100644
--- a/io_uring/eventfd.c
+++ b/io_uring/eventfd.c
@@ -41,7 +41,7 @@ static void io_eventfd_do_signal(struct rcu_head *rcu)
call_rcu(&ev_fd->rcu, io_eventfd_free);
}
-void io_eventfd_signal(struct io_ring_ctx *ctx)
+void io_eventfd_signal(struct io_ring_ctx *ctx, bool defer)
{
struct io_ev_fd *ev_fd = NULL;
@@ -68,7 +68,7 @@ void io_eventfd_signal(struct io_ring_ctx *ctx)
if (ev_fd->eventfd_async && !io_wq_current_is_worker())
goto out;
- if (likely(eventfd_signal_allowed())) {
+ if (likely(!defer && eventfd_signal_allowed())) {
eventfd_signal_mask(ev_fd->cq_ev_fd, EPOLL_URING_WAKE);
} else {
if (!atomic_fetch_or(BIT(IO_EVENTFD_OP_SIGNAL_BIT), &ev_fd->ops)) {
@@ -101,7 +101,7 @@ void io_eventfd_flush_signal(struct io_ring_ctx *ctx)
if (skip)
return;
- io_eventfd_signal(ctx);
+ io_eventfd_signal(ctx, false);
}
int io_eventfd_register(struct io_ring_ctx *ctx, void __user *arg,
diff --git a/io_uring/eventfd.h b/io_uring/eventfd.h
index d394f49c6321..c93f32d59407 100644
--- a/io_uring/eventfd.h
+++ b/io_uring/eventfd.h
@@ -5,4 +5,4 @@ int io_eventfd_register(struct io_ring_ctx *ctx, void __user *arg,
int io_eventfd_unregister(struct io_ring_ctx *ctx);
void io_eventfd_flush_signal(struct io_ring_ctx *ctx);
-void io_eventfd_signal(struct io_ring_ctx *ctx);
+void io_eventfd_signal(struct io_ring_ctx *ctx, bool defer);
diff --git a/io_uring/futex.c b/io_uring/futex.c
index a3d2b700b480..589769761cea 100644
--- a/io_uring/futex.c
+++ b/io_uring/futex.c
@@ -201,7 +201,7 @@ static void io_futex_wakev_fn(struct wake_q_head *wake_q, struct futex_q *q)
io_req_set_res(req, 0, 0);
req->io_task_work.func = io_futexv_complete;
- io_req_task_work_add(req);
+ __io_req_task_work_add(req, IOU_F_TWQ_IN_WAKE);
}
int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
@@ -248,7 +248,7 @@ static void io_futex_wake_fn(struct wake_q_head *wake_q, struct futex_q *q)
io_req_set_res(req, 0, 0);
req->io_task_work.func = io_futex_complete;
- io_req_task_work_add(req);
+ __io_req_task_work_add(req, IOU_F_TWQ_IN_WAKE);
}
static struct io_futex_data *io_alloc_ifd(struct io_ring_ctx *ctx)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 1e7963cc30d3..0076c7599c94 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1201,7 +1201,7 @@ static inline void io_req_local_work_add(struct io_kiocb *req,
if (ctx->flags & IORING_SETUP_TASKRUN_FLAG)
atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags);
if (ctx->has_evfd)
- io_eventfd_signal(ctx);
+ io_eventfd_signal(ctx, flags & IOU_F_TWQ_IN_WAKE);
}
nr_wait = atomic_read(&ctx->cq_wait_nr);
diff --git a/io_uring/napi.c b/io_uring/napi.c
index fa959fd32042..5542d78a4bd8 100644
--- a/io_uring/napi.c
+++ b/io_uring/napi.c
@@ -38,58 +38,54 @@ static inline ktime_t net_to_ktime(unsigned long t)
return ns_to_ktime(t << 10);
}
-void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock)
+int __io_napi_add_id(struct io_ring_ctx *ctx, unsigned int napi_id)
{
struct hlist_head *hash_list;
- unsigned int napi_id;
- struct sock *sk;
struct io_napi_entry *e;
- sk = sock->sk;
- if (!sk)
- return;
-
- napi_id = READ_ONCE(sk->sk_napi_id);
-
/* Non-NAPI IDs can be rejected. */
if (napi_id < MIN_NAPI_ID)
- return;
+ return -EINVAL;
hash_list = &ctx->napi_ht[hash_min(napi_id, HASH_BITS(ctx->napi_ht))];
- rcu_read_lock();
- e = io_napi_hash_find(hash_list, napi_id);
- if (e) {
- e->timeout = jiffies + NAPI_TIMEOUT;
- rcu_read_unlock();
- return;
+ scoped_guard(rcu) {
+ e = io_napi_hash_find(hash_list, napi_id);
+ if (e) {
+ WRITE_ONCE(e->timeout, jiffies + NAPI_TIMEOUT);
+ return -EEXIST;
+ }
}
- rcu_read_unlock();
e = kmalloc(sizeof(*e), GFP_NOWAIT);
if (!e)
- return;
+ return -ENOMEM;
e->napi_id = napi_id;
e->timeout = jiffies + NAPI_TIMEOUT;
+ /*
+ * guard(spinlock) is not used to manually unlock it before calling
+ * kfree()
+ */
spin_lock(&ctx->napi_lock);
if (unlikely(io_napi_hash_find(hash_list, napi_id))) {
spin_unlock(&ctx->napi_lock);
kfree(e);
- return;
+ return -EEXIST;
}
hlist_add_tail_rcu(&e->node, hash_list);
list_add_tail_rcu(&e->list, &ctx->napi_list);
spin_unlock(&ctx->napi_lock);
+ return 0;
}
static void __io_napi_remove_stale(struct io_ring_ctx *ctx)
{
struct io_napi_entry *e;
- spin_lock(&ctx->napi_lock);
+ guard(spinlock)(&ctx->napi_lock);
/*
* list_for_each_entry_safe() is not required as long as:
* 1. list_del_rcu() does not reset the deleted node next pointer
@@ -103,7 +99,6 @@ static void __io_napi_remove_stale(struct io_ring_ctx *ctx)
kfree_rcu(e, rcu);
}
}
- spin_unlock(&ctx->napi_lock);
}
static inline void io_napi_remove_stale(struct io_ring_ctx *ctx, bool is_stale)
@@ -142,20 +137,17 @@ static bool io_napi_busy_loop_should_end(void *data,
}
static bool __io_napi_do_busy_loop(struct io_ring_ctx *ctx,
+ bool (*loop_end)(void *, unsigned long),
void *loop_end_arg)
{
struct io_napi_entry *e;
- bool (*loop_end)(void *, unsigned long) = NULL;
bool is_stale = false;
- if (loop_end_arg)
- loop_end = io_napi_busy_loop_should_end;
-
list_for_each_entry_rcu(e, &ctx->napi_list, list) {
napi_busy_loop_rcu(e->napi_id, loop_end, loop_end_arg,
ctx->napi_prefer_busy_poll, BUSY_POLL_BUDGET);
- if (time_after(jiffies, e->timeout))
+ if (time_after(jiffies, READ_ONCE(e->timeout)))
is_stale = true;
}
@@ -166,20 +158,25 @@ static void io_napi_blocking_busy_loop(struct io_ring_ctx *ctx,
struct io_wait_queue *iowq)
{
unsigned long start_time = busy_loop_current_time();
+ bool (*loop_end)(void *, unsigned long) = NULL;
void *loop_end_arg = NULL;
bool is_stale = false;
/* Singular lists use a different napi loop end check function and are
* only executed once.
*/
- if (list_is_singular(&ctx->napi_list))
+ if (list_is_singular(&ctx->napi_list)) {
+ loop_end = io_napi_busy_loop_should_end;
loop_end_arg = iowq;
+ }
- rcu_read_lock();
- do {
- is_stale = __io_napi_do_busy_loop(ctx, loop_end_arg);
- } while (!io_napi_busy_loop_should_end(iowq, start_time) && !loop_end_arg);
- rcu_read_unlock();
+ scoped_guard(rcu) {
+ do {
+ is_stale = __io_napi_do_busy_loop(ctx, loop_end,
+ loop_end_arg);
+ } while (!io_napi_busy_loop_should_end(iowq, start_time) &&
+ !loop_end_arg);
+ }
io_napi_remove_stale(ctx, is_stale);
}
@@ -210,13 +207,12 @@ void io_napi_free(struct io_ring_ctx *ctx)
{
struct io_napi_entry *e;
- spin_lock(&ctx->napi_lock);
+ guard(spinlock)(&ctx->napi_lock);
list_for_each_entry(e, &ctx->napi_list, list) {
hash_del_rcu(&e->node);
kfree_rcu(e, rcu);
}
INIT_LIST_HEAD_RCU(&ctx->napi_list);
- spin_unlock(&ctx->napi_lock);
}
/*
@@ -244,6 +240,8 @@ int io_register_napi(struct io_ring_ctx *ctx, void __user *arg)
if (copy_to_user(arg, &curr, sizeof(curr)))
return -EFAULT;
+ /* cap NAPI at 10 msec of spin time */
+ napi.busy_poll_to = min(10000, napi.busy_poll_to);
WRITE_ONCE(ctx->napi_busy_poll_dt, napi.busy_poll_to * NSEC_PER_USEC);
WRITE_ONCE(ctx->napi_prefer_busy_poll, !!napi.prefer_busy_poll);
WRITE_ONCE(ctx->napi_enabled, true);
@@ -312,9 +310,9 @@ int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx)
if (list_empty_careful(&ctx->napi_list))
return 0;
- rcu_read_lock();
- is_stale = __io_napi_do_busy_loop(ctx, NULL);
- rcu_read_unlock();
+ scoped_guard(rcu) {
+ is_stale = __io_napi_do_busy_loop(ctx, NULL, NULL);
+ }
io_napi_remove_stale(ctx, is_stale);
return 1;
diff --git a/io_uring/napi.h b/io_uring/napi.h
index fd275ef0456d..4ae622f37b30 100644
--- a/io_uring/napi.h
+++ b/io_uring/napi.h
@@ -15,7 +15,7 @@ void io_napi_free(struct io_ring_ctx *ctx);
int io_register_napi(struct io_ring_ctx *ctx, void __user *arg);
int io_unregister_napi(struct io_ring_ctx *ctx, void __user *arg);
-void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock);
+int __io_napi_add_id(struct io_ring_ctx *ctx, unsigned int napi_id);
void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq);
int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx);
@@ -48,8 +48,8 @@ static inline void io_napi_add(struct io_kiocb *req)
return;
sock = sock_from_file(req->file);
- if (sock)
- __io_napi_add(ctx, sock);
+ if (sock && sock->sk)
+ __io_napi_add_id(ctx, READ_ONCE(sock->sk->sk_napi_id));
}
#else
diff --git a/io_uring/poll.c b/io_uring/poll.c
index 002f1ae830b8..1a6179ac491b 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -231,9 +231,9 @@ enum {
IOU_POLL_REQUEUE = 4,
};
-static void __io_poll_execute(struct io_kiocb *req, int mask)
+static void __io_poll_execute(struct io_kiocb *req, int mask, unsigned tw_flags)
{
- unsigned flags = 0;
+ unsigned flags = tw_flags;
io_req_set_res(req, mask, 0);
req->io_task_work.func = io_poll_task_func;
@@ -241,14 +241,15 @@ static void __io_poll_execute(struct io_kiocb *req, int mask)
trace_io_uring_task_add(req, mask);
if (!(req->flags & REQ_F_POLL_NO_LAZY))
- flags = IOU_F_TWQ_LAZY_WAKE;
+ flags |= IOU_F_TWQ_LAZY_WAKE;
__io_req_task_work_add(req, flags);
}
-static inline void io_poll_execute(struct io_kiocb *req, int res)
+static inline void io_poll_execute(struct io_kiocb *req, int res,
+ unsigned tw_flags)
{
if (io_poll_get_ownership(req))
- __io_poll_execute(req, res);
+ __io_poll_execute(req, res, tw_flags);
}
/*
@@ -363,7 +364,7 @@ void io_poll_task_func(struct io_kiocb *req, struct io_tw_state *ts)
if (ret == IOU_POLL_NO_ACTION) {
return;
} else if (ret == IOU_POLL_REQUEUE) {
- __io_poll_execute(req, 0);
+ __io_poll_execute(req, 0, 0);
return;
}
io_poll_remove_entries(req);
@@ -401,7 +402,7 @@ static void io_poll_cancel_req(struct io_kiocb *req)
{
io_poll_mark_cancelled(req);
/* kick tw, which should complete the request */
- io_poll_execute(req, 0);
+ io_poll_execute(req, 0, 0);
}
#define IO_ASYNC_POLL_COMMON (EPOLLONESHOT | EPOLLPRI)
@@ -410,7 +411,7 @@ static __cold int io_pollfree_wake(struct io_kiocb *req, struct io_poll *poll)
{
io_poll_mark_cancelled(req);
/* we have to kick tw in case it's not already */
- io_poll_execute(req, 0);
+ io_poll_execute(req, 0, IOU_F_TWQ_IN_WAKE);
/*
* If the waitqueue is being freed early but someone is already
@@ -465,7 +466,7 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync,
else
req->flags &= ~REQ_F_SINGLE_POLL;
}
- __io_poll_execute(req, mask);
+ __io_poll_execute(req, mask, IOU_F_TWQ_IN_WAKE);
}
return 1;
}
@@ -657,7 +658,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req,
if (mask && (poll->events & EPOLLET) &&
io_poll_can_finish_inline(req, ipt)) {
- __io_poll_execute(req, mask);
+ __io_poll_execute(req, mask, 0);
return 0;
}
io_napi_add(req);
@@ -668,7 +669,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req,
* poll was waken up, queue up a tw, it'll deal with it.
*/
if (atomic_cmpxchg(&req->poll_refs, 1, 0) != 1)
- __io_poll_execute(req, 0);
+ __io_poll_execute(req, 0, 0);
}
return 0;
}
diff --git a/io_uring/rw.c b/io_uring/rw.c
index d615c61bbbd2..c8a536ce2fe3 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -156,22 +156,22 @@ static void io_rw_iovec_free(struct io_async_rw *rw)
}
}
-static void io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags)
+static bool io_rw_recycle(struct io_kiocb *req, unsigned int issue_flags)
{
struct io_async_rw *rw = req->async_data;
struct iovec *iov;
- if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) {
- io_rw_iovec_free(rw);
- return;
- }
+ if (unlikely(issue_flags & IO_URING_F_UNLOCKED))
+ return false;
iov = rw->free_iovec;
if (io_alloc_cache_put(&req->ctx->rw_cache, rw)) {
if (iov)
kasan_mempool_poison_object(iov);
req->async_data = NULL;
req->flags &= ~REQ_F_ASYNC_DATA;
+ return true;
}
+ return false;
}
static void io_req_rw_cleanup(struct io_kiocb *req, unsigned int issue_flags)
@@ -205,7 +205,11 @@ static void io_req_rw_cleanup(struct io_kiocb *req, unsigned int issue_flags)
*/
if (!(req->flags & REQ_F_REFCOUNT)) {
req->flags &= ~REQ_F_NEED_CLEANUP;
- io_rw_recycle(req, issue_flags);
+ if (!io_rw_recycle(req, issue_flags)) {
+ struct io_async_rw *rw = req->async_data;
+
+ io_rw_iovec_free(rw);
+ }
}
}
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index 7f3a6c2d9f80..4475b02bafba 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -298,7 +298,7 @@ static int io_waitid_wait(struct wait_queue_entry *wait, unsigned mode,
return 1;
req->io_task_work.func = io_waitid_cb;
- io_req_task_work_add(req);
+ __io_req_task_work_add(req, IOU_F_TWQ_IN_WAKE);
return 1;
}
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 2285b27ce68c..582128fbc1d0 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -422,6 +422,18 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
goto errout;
}
+ /*
+ * A >8 byte return value is passed back in a register pair,
+ * which the struct_ops trampoline does not preserve (only
+ * 8 bytes of the return value are saved and restored).
+ */
+ if (st_ops->func_models[i].ret_size > 8) {
+ pr_warn("func ptr %s in struct %s has a >8 byte return value, which is not supported\n",
+ mname, st_ops->name);
+ err = -EOPNOTSUPP;
+ goto errout;
+ }
+
stub_func_addr = *(void **)(st_ops->cfi_stubs + moff);
err = prepare_arg_info(btf, st_ops->name, mname,
func_proto, stub_func_addr,
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 5f3b1162208b..e8a79d92d4af 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -3518,7 +3518,15 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_
field_mask_test_name(BPF_LIST_NODE, "bpf_list_node");
field_mask_test_name(BPF_RB_ROOT, "bpf_rb_root");
field_mask_test_name(BPF_RB_NODE, "bpf_rb_node");
- field_mask_test_name(BPF_REFCOUNT, "bpf_refcount");
+ if (field_mask & BPF_REFCOUNT) {
+ if (!strcmp(name, "bpf_refcount")) {
+ if (*seen_mask & BPF_REFCOUNT)
+ return -E2BIG;
+ *seen_mask |= BPF_REFCOUNT;
+ type = BPF_REFCOUNT;
+ goto end;
+ }
+ }
/* Only return BPF_KPTR when all other types with matchable names fail */
if (field_mask & BPF_KPTR && !__btf_type_is_struct(var_type)) {
@@ -6550,6 +6558,9 @@ static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {
/* rxrpc */
{ "rxrpc_recvdata", 0x1 },
{ "rxrpc_resend", 0x10 },
+ /* signal */
+ { "signal_generate", 0x20 },
+ { "signal_deliver", 0x20 },
/* skb */
{"kfree_skb", 0x1000},
/* sunrpc */
@@ -6833,7 +6844,7 @@ enum bpf_struct_walk_result {
static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, int off, int size,
u32 *next_btf_id, enum bpf_type_flag *flag,
- const char **field_name)
+ const char **field_name, bool walk_flex_arrays)
{
u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
const struct btf_type *mtype, *elem_type = NULL;
@@ -6860,11 +6871,14 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
*flag |= PTR_UNTRUSTED;
if (off + size > t->size) {
+ struct btf_array *array_elem;
+
+ if (!walk_flex_arrays)
+ goto error;
+
/* If the last element is a variable size array, we may
* need to relax the rule.
*/
- struct btf_array *array_elem;
-
if (vlen == 0)
goto error;
@@ -7125,7 +7139,8 @@ int btf_struct_access(struct bpf_verifier_log *log,
t = btf_type_by_id(btf, id);
do {
- err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, field_name);
+ err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag,
+ field_name, !type_is_alloc(reg->type));
switch (err) {
case WALK_PTR:
@@ -7181,7 +7196,7 @@ bool btf_types_are_same(const struct btf *btf1, u32 id1,
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
- bool strict)
+ bool strict, bool walk_flex_arrays)
{
const struct btf_type *type;
enum bpf_type_flag flag = 0;
@@ -7200,7 +7215,8 @@ bool btf_struct_ids_match(struct bpf_verifier_log *log,
type = btf_type_by_id(btf, id);
if (!type)
return false;
- err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL);
+ err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL,
+ walk_flex_arrays);
if (err != WALK_STRUCT)
return false;
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 3126ec5097f1..617f1b8c3670 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -2100,9 +2100,7 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head,
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
- migrate_disable();
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
- migrate_enable();
}
}
@@ -2123,6 +2121,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
+ struct bpf_rb_node_kern *node;
struct rb_node *pos, *n;
void *obj;
@@ -2131,17 +2130,21 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
+ bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
+ node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
+ WRITE_ONCE(node->owner, BPF_PTR_POISON);
+ }
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
-
-
- migrate_disable();
+ node = rb_entry(pos, struct bpf_rb_node_kern, rb_node);
+ RB_CLEAR_NODE(pos);
+ /* Ensure __bpf_rbtree_add() sees the node as unlinked. */
+ smp_store_release(&node->owner, NULL);
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
- migrate_enable();
}
}
diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h
index 5d18d7d85bef..1834db20b861 100644
--- a/kernel/bpf/mmap_unlock_work.h
+++ b/kernel/bpf/mmap_unlock_work.h
@@ -4,12 +4,15 @@
#ifndef __MMAP_UNLOCK_WORK_H__
#define __MMAP_UNLOCK_WORK_H__
+#include <linux/atomic.h>
+#include <linux/err.h>
#include <linux/irq_work.h>
/* irq_work to run mmap_read_unlock() in irq_work */
struct mmap_unlock_irq_work {
struct irq_work irq_work;
struct mm_struct *mm;
+ atomic_t active;
};
DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
@@ -18,32 +21,36 @@ DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
* We cannot do mmap_read_unlock() when the irq is disabled, because of
* risk to deadlock with rq_lock. To look up vma when the irqs are
* disabled, we need to run mmap_read_unlock() in irq_work. We use a
- * percpu variable to do the irq_work. If the irq_work is already used
- * by another lookup, we fall over.
+ * percpu variable to do the irq_work. The active flag reserves the slot
+ * before mmap_read_trylock() and until the irq_work callback consumes mm.
*/
-static inline bool bpf_mmap_unlock_get_irq_work(struct mmap_unlock_irq_work **work_ptr)
+static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void)
{
- struct mmap_unlock_irq_work *work = NULL;
- bool irq_work_busy = false;
+ struct mmap_unlock_irq_work *work;
- if (irqs_disabled()) {
- if (!IS_ENABLED(CONFIG_PREEMPT_RT)) {
- work = this_cpu_ptr(&mmap_unlock_work);
- if (irq_work_is_busy(&work->irq_work)) {
- /* cannot queue more up_read, fallback */
- irq_work_busy = true;
- }
- } else {
- /*
- * PREEMPT_RT does not allow to trylock mmap sem in
- * interrupt disabled context. Force the fallback code.
- */
- irq_work_busy = true;
- }
- }
+ if (!irqs_disabled())
+ return NULL;
+
+ /*
+ * PREEMPT_RT does not allow to trylock mmap sem in interrupt
+ * disabled context. Force the fallback code.
+ */
+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
+ return ERR_PTR(-EBUSY);
+
+ work = this_cpu_ptr(&mmap_unlock_work);
+ if (irq_work_is_busy(&work->irq_work) ||
+ atomic_cmpxchg_acquire(&work->active, 0, 1))
+ return ERR_PTR(-EBUSY);
- *work_ptr = work;
- return irq_work_busy;
+ return work;
+}
+
+static inline void
+bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work)
+{
+ if (work)
+ atomic_set_release(&work->active, 0);
}
static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm)
diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c
index d869f51ea93a..36384342d7d8 100644
--- a/kernel/bpf/queue_stack_maps.c
+++ b/kernel/bpf/queue_stack_maps.c
@@ -99,8 +99,10 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete)
void *ptr;
if (in_nmi()) {
- if (!raw_spin_trylock_irqsave(&qs->lock, flags))
+ if (!raw_spin_trylock_irqsave(&qs->lock, flags)) {
+ memset(value, 0, qs->map.value_size);
return -EBUSY;
+ }
} else {
raw_spin_lock_irqsave(&qs->lock, flags);
}
@@ -134,8 +136,10 @@ static long __stack_map_get(struct bpf_map *map, void *value, bool delete)
u32 index;
if (in_nmi()) {
- if (!raw_spin_trylock_irqsave(&qs->lock, flags))
+ if (!raw_spin_trylock_irqsave(&qs->lock, flags)) {
+ memset(value, 0, qs->map.value_size);
return -EBUSY;
+ }
} else {
raw_spin_lock_irqsave(&qs->lock, flags);
}
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
index d0c7c5713dc4..a43b2d8b13e0 100644
--- a/kernel/bpf/stackmap.c
+++ b/kernel/bpf/stackmap.c
@@ -166,23 +166,20 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
u32 trace_nr, bool user, bool may_fault)
{
int i;
- struct mmap_unlock_irq_work *work = NULL;
- bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
+ struct mmap_unlock_irq_work *work;
struct vm_area_struct *vma, *prev_vma = NULL;
const char *prev_build_id;
- /* If the irq_work is in use, fall back to report ips. Same
- * fallback is used for kernel stack (!user) on a stackmap with
- * build_id.
- */
- if (!user || !current || !current->mm || irq_work_busy ||
- !mmap_read_trylock(current->mm)) {
- /* cannot access current->mm, fall back to ips */
- for (i = 0; i < trace_nr; i++) {
- id_offs[i].status = BPF_STACK_BUILD_ID_IP;
- memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
- }
- return;
+ if (!user || !current || !current->mm)
+ goto fallback;
+
+ work = bpf_mmap_unlock_guard_get();
+ if (IS_ERR(work))
+ goto fallback;
+
+ if (!mmap_read_trylock(current->mm)) {
+ bpf_mmap_unlock_guard_put(work);
+ goto fallback;
}
for (i = 0; i < trace_nr; i++) {
@@ -207,6 +204,14 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
prev_build_id = id_offs[i].build_id;
}
bpf_mmap_unlock_mm(work, current->mm);
+ return;
+
+fallback:
+ /* cannot access current->mm, fall back to ips */
+ for (i = 0; i < trace_nr; i++) {
+ id_offs[i].status = BPF_STACK_BUILD_ID_IP;
+ memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
+ }
}
static struct perf_callchain_entry *
@@ -483,19 +488,55 @@ const struct bpf_func_proto bpf_get_stackid_proto_pe = {
.arg3_type = ARG_ANYTHING,
};
+static u32 callchain_store(struct perf_callchain_entry *trace, void *buf,
+ u32 elem_size, u64 flags)
+{
+ bool user_build_id = flags & BPF_F_USER_BUILD_ID;
+ u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
+ u32 trace_nr, copy_len;
+ u64 *ips;
+
+ trace_nr = trace->nr - skip;
+ copy_len = trace_nr * elem_size;
+
+ ips = trace->ip + skip;
+ if (user_build_id) {
+ struct bpf_stack_build_id *id_offs = buf;
+
+ for (u32 i = 0; i < trace_nr; i++)
+ id_offs[i].ip = ips[i];
+ } else {
+ memcpy(buf, ips, copy_len);
+ }
+ return trace_nr;
+}
+
+static long callchain_finalize(void *buf, u32 size, u32 trace_nr, u32 elem_size,
+ u64 flags, bool may_fault)
+{
+ bool user_build_id = flags & BPF_F_USER_BUILD_ID;
+ bool user = flags & BPF_F_USER_STACK;
+ u32 copy_len = trace_nr * elem_size;
+
+ if (user_build_id)
+ stack_map_get_build_id_offset(buf, trace_nr, user, may_fault);
+
+ if (size > copy_len)
+ memset(buf + copy_len, 0, size - copy_len);
+ return copy_len;
+}
+
static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
- struct perf_callchain_entry *trace_in,
void *buf, u32 size, u64 flags, bool may_fault)
{
- u32 trace_nr, copy_len, elem_size, max_depth;
bool user_build_id = flags & BPF_F_USER_BUILD_ID;
bool crosstask = task && task != current;
u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
bool user = flags & BPF_F_USER_STACK;
struct perf_callchain_entry *trace;
+ u32 trace_nr, elem_size, max_depth;
bool kernel = !user;
int err = -EINVAL;
- u64 *ips;
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_USER_BUILD_ID)))
@@ -525,10 +566,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
if (may_fault)
rcu_read_lock(); /* need RCU for perf's callchain below */
- if (trace_in) {
- trace = trace_in;
- trace->nr = min_t(u32, trace->nr, max_depth);
- } else if (kernel && task) {
+ if (kernel && task) {
trace = get_callchain_entry_for_task(task, max_depth);
} else {
trace = get_perf_callchain(regs, kernel, user, max_depth,
@@ -542,31 +580,14 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
goto err_fault;
}
- trace_nr = trace->nr - skip;
- copy_len = trace_nr * elem_size;
-
- ips = trace->ip + skip;
- if (user_build_id) {
- struct bpf_stack_build_id *id_offs = buf;
- u32 i;
-
- for (i = 0; i < trace_nr; i++)
- id_offs[i].ip = ips[i];
- } else {
- memcpy(buf, ips, copy_len);
- }
+ trace_nr = callchain_store(trace, buf, elem_size, flags);
- /* trace/ips should not be dereferenced after this point */
+ /* trace should not be dereferenced after this point */
if (may_fault)
rcu_read_unlock();
preempt_enable();
- if (user_build_id)
- stack_map_get_build_id_offset(buf, trace_nr, user, may_fault);
-
- if (size > copy_len)
- memset(buf + copy_len, 0, size - copy_len);
- return copy_len;
+ return callchain_finalize(buf, size, trace_nr, elem_size, flags, may_fault);
err_fault:
err = -EFAULT;
@@ -578,7 +599,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
BPF_CALL_4(bpf_get_stack, struct pt_regs *, regs, void *, buf, u32, size,
u64, flags)
{
- return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */);
+ return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */);
}
const struct bpf_func_proto bpf_get_stack_proto = {
@@ -594,7 +615,7 @@ const struct bpf_func_proto bpf_get_stack_proto = {
BPF_CALL_4(bpf_get_stack_sleepable, struct pt_regs *, regs, void *, buf, u32, size,
u64, flags)
{
- return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, true /* may_fault */);
+ return __bpf_get_stack(regs, NULL, buf, size, flags, true /* may_fault */);
}
const struct bpf_func_proto bpf_get_stack_sleepable_proto = {
@@ -614,14 +635,17 @@ static long __bpf_get_task_stack(struct task_struct *task, void *buf, u32 size,
struct pt_regs *regs;
long res = -EINVAL;
- if (!try_get_task_stack(task))
+ if (!try_get_task_stack(task)) {
+ memset(buf, 0, size);
return -EFAULT;
+ }
regs = task_pt_regs(task);
if (regs)
- res = __bpf_get_stack(regs, task, NULL, buf, size, flags, may_fault);
+ res = __bpf_get_stack(regs, task, buf, size, flags, may_fault);
+ else
+ memset(buf, 0, size);
put_task_stack(task);
-
return res;
}
@@ -660,6 +684,32 @@ const struct bpf_func_proto bpf_get_task_stack_sleepable_proto = {
.arg4_type = ARG_ANYTHING,
};
+static int __bpf_get_stack_pe(struct perf_callchain_entry *trace, void *buf, u32 size,
+ u64 flags)
+{
+ bool user_build_id = flags & BPF_F_USER_BUILD_ID;
+ u64 skip = flags & BPF_F_SKIP_FIELD_MASK;
+ bool user = flags & BPF_F_USER_STACK;
+ u32 elem_size, max_depth, nr_trace;
+ bool kernel = !user;
+
+ if (kernel && user_build_id)
+ return -EINVAL;
+
+ elem_size = user_build_id ? sizeof(struct bpf_stack_build_id) : sizeof(u64);
+ if (unlikely(size % elem_size))
+ return -EINVAL;
+
+ max_depth = stack_map_calculate_max_depth(size, elem_size, flags);
+ trace->nr = min_t(u32, trace->nr, max_depth);
+
+ if (trace->nr < skip)
+ return -EFAULT;
+
+ nr_trace = callchain_store(trace, buf, elem_size, flags);
+ return callchain_finalize(buf, size, nr_trace, elem_size, flags, false /* !may_fault */);
+}
+
BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
void *, buf, u32, size, u64, flags)
{
@@ -671,7 +721,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
__u64 nr_kernel;
if (!(event->attr.sample_type & PERF_SAMPLE_CALLCHAIN))
- return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */);
+ return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */);
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_USER_BUILD_ID)))
@@ -691,7 +741,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
__u64 nr = trace->nr;
trace->nr = nr_kernel;
- err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */);
+ err = __bpf_get_stack_pe(trace, buf, size, flags);
/* restore nr */
trace->nr = nr;
@@ -701,14 +751,13 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
skip += nr_kernel;
if (skip > BPF_F_SKIP_FIELD_MASK)
goto clear;
-
flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip;
- err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */);
+ err = __bpf_get_stack_pe(trace, buf, size, flags);
}
- return err;
clear:
- memset(buf, 0, size);
+ if (err < 0)
+ memset(buf, 0, size);
return err;
}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 857285a48d43..854cc29b47c8 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -707,11 +707,9 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj)
if (!btf_is_kernel(field->kptr.btf)) {
pointee_struct_meta = btf_find_struct_meta(field->kptr.btf,
field->kptr.btf_id);
- migrate_disable();
__bpf_obj_drop_impl(xchgd_field, pointee_struct_meta ?
pointee_struct_meta->record : NULL,
fields[i].type == BPF_KPTR_PERCPU);
- migrate_enable();
} else {
field->kptr.dtor(xchgd_field);
}
@@ -1206,11 +1204,6 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token,
return ret;
}
-static bool bpf_net_capable(void)
-{
- return capable(CAP_NET_ADMIN) || capable(CAP_SYS_ADMIN);
-}
-
#define BPF_MAP_CREATE_LAST_FIELD map_token_fd
/* called via syscall */
static int map_create(union bpf_attr *attr)
@@ -4177,6 +4170,8 @@ static int bpf_prog_detach(const union bpf_attr *attr)
prog = bpf_prog_get_type(attr->attach_bpf_fd, ptype);
if (IS_ERR(prog))
return PTR_ERR(prog);
+ } else if (!bpf_mprog_detach_empty(ptype)) {
+ return -EPERM;
}
} else if (is_cgroup_prog_type(ptype, 0, false)) {
if (attr->attach_flags || attr->relative_fd)
@@ -5902,6 +5897,7 @@ EXPORT_SYMBOL_NS(kern_sys_bpf, BPF_INTERNAL);
static const struct bpf_func_proto bpf_sys_bpf_proto = {
.func = bpf_sys_bpf,
.gpl_only = false,
+ .might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
@@ -5927,6 +5923,7 @@ BPF_CALL_1(bpf_sys_close, u32, fd)
static const struct bpf_func_proto bpf_sys_close_proto = {
.func = bpf_sys_close,
.gpl_only = false,
+ .might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c
index f3a7a96a77d7..1b36cbc3d25c 100644
--- a/kernel/bpf/task_iter.c
+++ b/kernel/bpf/task_iter.c
@@ -757,9 +757,8 @@ static struct bpf_iter_reg task_vma_reg_info = {
BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start,
bpf_callback_t, callback_fn, void *, callback_ctx, u64, flags)
{
- struct mmap_unlock_irq_work *work = NULL;
+ struct mmap_unlock_irq_work *work;
struct vm_area_struct *vma;
- bool irq_work_busy = false;
bool __maybe_unused mmput_needed = false;
struct mm_struct *mm;
int ret = -ENOENT;
@@ -796,9 +795,14 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start,
if (!mm)
return -ENOENT;
- irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
+ work = bpf_mmap_unlock_guard_get();
+ if (IS_ERR(work)) {
+ ret = PTR_ERR(work);
+ goto out;
+ }
- if (irq_work_busy || !mmap_read_trylock(mm)) {
+ if (!mmap_read_trylock(mm)) {
+ bpf_mmap_unlock_guard_put(work);
ret = -EBUSY;
goto out;
}
@@ -1195,6 +1199,8 @@ static void do_mmap_read_unlock(struct irq_work *entry)
work = container_of(entry, struct mmap_unlock_irq_work, irq_work);
mmap_read_unlock_non_owner(work->mm);
+ work->mm = NULL;
+ bpf_mmap_unlock_guard_put(work);
}
static int __init task_iter_init(void)
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 894cd6f205f5..76fdd4ee9225 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -397,6 +397,13 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size)
return ERR_PTR(err);
}
+void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags)
+{
+ mutex_lock(&tr->mutex);
+ tr->flags |= flags;
+ mutex_unlock(&tr->mutex);
+}
+
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex)
{
struct bpf_tramp_image *im;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 2bcd098f774b..ecd8595bd545 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -395,9 +395,18 @@ static bool reg_not_null(const struct bpf_reg_state *reg)
if (type_may_be_null(type))
return false;
+ /*
+ * The types below guarantee a non-NULL base, an unbounded offset can
+ * still wrap base + offset to zero.
+ */
+ if (reg->smin_value <= -BPF_MAX_VAR_OFF || reg->smax_value >= BPF_MAX_VAR_OFF)
+ return false;
+
type = base_type(type);
return type == PTR_TO_SOCKET ||
type == PTR_TO_TCP_SOCK ||
+ type == PTR_TO_XDP_SOCK ||
+ type == PTR_TO_BUF ||
type == PTR_TO_MAP_VALUE ||
type == PTR_TO_MAP_KEY ||
type == PTR_TO_SOCK_COMMON ||
@@ -571,13 +580,25 @@ static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id,
return ref_obj_uses > 1;
}
+/*
+ * Matches BPF_PROBE_ATOMIC too: convert_ctx_accesses() rewrites arena
+ * atomics before opt_subreg_zext_lo32_rnd_hi32() runs.
+ */
static bool is_cmpxchg_insn(const struct bpf_insn *insn)
{
return BPF_CLASS(insn->code) == BPF_STX &&
- BPF_MODE(insn->code) == BPF_ATOMIC &&
+ (BPF_MODE(insn->code) == BPF_ATOMIC ||
+ BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) &&
insn->imm == BPF_CMPXCHG;
}
+static bool is_atomic_fetch_insn(const struct bpf_insn *insn)
+{
+ return BPF_CLASS(insn->code) == BPF_STX &&
+ BPF_MODE(insn->code) == BPF_ATOMIC &&
+ (insn->imm & BPF_FETCH);
+}
+
static int __get_spi(s32 off)
{
return (-off - 1) / BPF_REG_SIZE;
@@ -3517,6 +3538,16 @@ static bool is_jmp_point(struct bpf_verifier_env *env, int insn_idx)
return env->insn_aux_data[insn_idx].jmp_point;
}
+static void mark_jump_target(struct bpf_verifier_env *env, int idx)
+{
+ env->insn_aux_data[idx].jump_target = true;
+}
+
+static bool bpf_is_jump_target(struct bpf_verifier_env *env, int insn_idx)
+{
+ return env->insn_aux_data[insn_idx].jump_target;
+}
+
#define LR_FRAMENO_BITS 3
#define LR_SPI_BITS 6
#define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)
@@ -3968,10 +3999,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* dreg still needs precision before this insn
*/
}
- } else if (class == BPF_LDX) {
- if (!bt_is_reg_set(bt, dreg))
+ } else if (class == BPF_LDX || is_atomic_fetch_insn(insn)) {
+ u32 load_reg = dreg;
+
+ /*
+ * Atomic fetch operation writes the old value into
+ * a register (sreg or r0) and if it was tracked for
+ * precision, propagate to the stack slot like we do
+ * in regular ldx.
+ */
+ if (is_atomic_fetch_insn(insn))
+ load_reg = insn->imm == BPF_CMPXCHG ?
+ BPF_REG_0 : sreg;
+
+ if (!bt_is_reg_set(bt, load_reg))
return 0;
- bt_clear_reg(bt, dreg);
+ bt_clear_reg(bt, load_reg);
/* scalars can only be spilled into stack w/o losing precision.
* Load from any other memory can be zero extended.
@@ -4107,37 +4150,34 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -EFAULT;
}
} else if (opcode == BPF_EXIT) {
- bool r0_precise;
+ bool from_subprog_call, r0_precise;
+
+ /* BPF_EXIT in subprog or callback always returns
+ * right after the call instruction, so by checking
+ * whether the instruction at subseq_idx-1 is subprog
+ * call or not we can distinguish actual exit from
+ * *subprog* from exit from *callback*. In the former
+ * case, we need to propagate r0 precision, if
+ * necessary. In the former we never do that.
+ */
+ from_subprog_call = subseq_idx - 1 >= 0 &&
+ bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+ r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
* In case of a regular function call, instructions giving
* precision to registers R1-R5 should have been found already.
- * In case of a callback, it is ok to have R1-R5 marked for
- * backtracking, as these registers are set by the function
- * invoking callback.
+ * In case of a callback from bpf_loop(), R{1,4} in the calling
+ * frame would be set as precise and that is correct.
*/
- if (subseq_idx >= 0 && calls_callback(env, subseq_idx))
- for (i = BPF_REG_1; i <= BPF_REG_5; i++)
- bt_clear_reg(bt, i);
- if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
+ if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
verbose(env, "BUG regs %x\n", bt_reg_mask(bt));
WARN_ONCE(1, "verifier backtracking bug");
return -EFAULT;
}
- /* BPF_EXIT in subprog or callback always returns
- * right after the call instruction, so by checking
- * whether the instruction at subseq_idx-1 is subprog
- * call or not we can distinguish actual exit from
- * *subprog* from exit from *callback*. In the former
- * case, we need to propagate r0 precision, if
- * necessary. In the former we never do that.
- */
- r0_precise = subseq_idx - 1 >= 0 &&
- bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
- bt_is_reg_set(bt, BPF_REG_0);
-
bt_clear_reg(bt, BPF_REG_0);
if (bt_subprog_enter(bt))
return -EFAULT;
@@ -4169,16 +4209,29 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
*/
}
} else if (class == BPF_LD) {
- if (!bt_is_reg_set(bt, dreg))
- return 0;
- bt_clear_reg(bt, dreg);
/* It's ld_imm64 or ld_abs or ld_ind.
* For ld_imm64 no further tracking of precision
* into parent is necessary
*/
- if (mode == BPF_IND || mode == BPF_ABS)
- /* to be analyzed */
- return -ENOTSUPP;
+ if (mode == BPF_IMM) {
+ bt_clear_reg(bt, dreg);
+ return 0;
+ }
+ /*
+ * BPF_{IND,ABS} are modelled as two branches:
+ * - fallthrough;
+ * - implicit subprogram exit.
+ * It is necessary to switch current frame if
+ * implicit subprogram exit branch is backtracked.
+ */
+ if (mode == BPF_IND || mode == BPF_ABS) {
+ if (bt_is_reg_set(bt, dreg))
+ return -ENOTSUPP;
+ if (subseq_idx != idx + 1)
+ if (bt_subprog_enter(bt))
+ return -EFAULT;
+ return 0;
+ }
}
/* Propagate precision marks to linked registers, to account for
* registers marked as precise in this function.
@@ -5551,7 +5604,8 @@ static int map_kptr_match_type(struct bpf_verifier_env *env,
*/
if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off,
kptr_field->kptr.btf, kptr_field->kptr.btf_id,
- kptr_field->type != BPF_KPTR_UNREF))
+ kptr_field->type != BPF_KPTR_UNREF,
+ !type_is_alloc(reg->type)))
goto bad_type;
return 0;
bad_type:
@@ -5684,8 +5738,15 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,
kptr_field->kptr.btf_id, btf_ld_kptr_type(env, kptr_field));
} else if (class == BPF_STX) {
val_reg = reg_state(env, value_regno);
- if (!register_is_null(val_reg) &&
- map_kptr_match_type(env, kptr_field, val_reg, value_regno))
+ if (register_is_null(val_reg)) {
+ /*
+ * This store is valid only because the scalar is known to be
+ * zero. Mark it precise so another scalar cannot be pruned
+ * against this state.
+ */
+ return mark_chain_precision(env, value_regno);
+ }
+ if (map_kptr_match_type(env, kptr_field, val_reg, value_regno))
return -EACCES;
} else if (class == BPF_ST) {
if (insn->imm) {
@@ -5986,6 +6047,26 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno)
return reg->type == PTR_TO_ARENA;
}
+/* Return false if @regno contains a pointer whose type isn't supported for
+ * atomic instruction @insn.
+ */
+static bool atomic_ptr_type_ok(struct bpf_verifier_env *env, int regno,
+ struct bpf_insn *insn)
+{
+ if (is_ctx_reg(env, regno))
+ return false;
+ if (is_pkt_reg(env, regno))
+ return false;
+ if (is_flow_key_reg(env, regno))
+ return false;
+ if (is_sk_reg(env, regno))
+ return false;
+ if (is_arena_reg(env, regno))
+ return bpf_jit_supports_insn(insn, true);
+
+ return true;
+}
+
static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = {
#ifdef CONFIG_NET
[PTR_TO_SOCKET] = &btf_sock_ids[BTF_SOCK_TYPE_SOCK],
@@ -6203,6 +6284,17 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
if (!priv_stack_supported)
subprog[idx].priv_stack_mode = NO_PRIV_STACK;
process_func:
+ if (subprog[idx].has_ld_abs) {
+ bool in_cb = subprog[idx].is_cb;
+
+ for (j = 0; j < frame && !in_cb; j++)
+ in_cb = subprog[ret_prog[j]].is_cb;
+ if (in_cb) {
+ verbose(env, "cannot use BPF_LD_[ABS|IND] within callback\n");
+ return -EINVAL;
+ }
+ }
+
/* protect against potential stack overflow that might happen when
* bpf2bpf calls get combined with tailcalls. Limit the caller's stack
* depth for such case down to 256 so that the worst case scenario
@@ -6868,6 +6960,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
return -EACCES;
}
+ if (atype != BPF_READ && (type_flag(reg->type) & PTR_UNTRUSTED)) {
+ verbose(env, "only read is supported\n");
+ return -EACCES;
+ }
+
if (env->ops->btf_struct_access && !type_is_alloc(reg->type) && atype == BPF_WRITE) {
if (!btf_is_kernel(reg->btf)) {
verbose(env, "verifier internal error: reg->btf must be kernel btf\n");
@@ -6876,8 +6973,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
ret = env->ops->btf_struct_access(&env->log, reg, off, size);
} else {
/* Writes are permitted with default btf_struct_access for
- * program allocated objects (which always have ref_obj_id > 0),
- * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC.
+ * program allocated objects (which always have ref_obj_id > 0).
*/
if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) {
verbose(env, "only read is supported\n");
@@ -7383,27 +7479,72 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type type,
bool allow_trust_mismatch);
-static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_insn *insn)
+static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
+ bool strict_alignment_once, bool is_ldsx,
+ bool allow_trust_mismatch, const char *ctx)
{
- int load_reg;
+ struct bpf_reg_state *regs = cur_regs(env);
+ enum bpf_reg_type src_reg_type;
int err;
- switch (insn->imm) {
- case BPF_ADD:
- case BPF_ADD | BPF_FETCH:
- case BPF_AND:
- case BPF_AND | BPF_FETCH:
- case BPF_OR:
- case BPF_OR | BPF_FETCH:
- case BPF_XOR:
- case BPF_XOR | BPF_FETCH:
- case BPF_XCHG:
- case BPF_CMPXCHG:
- break;
- default:
- verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n", insn->imm);
- return -EINVAL;
- }
+ /* check src operand */
+ err = check_reg_arg(env, insn->src_reg, SRC_OP);
+ if (err)
+ return err;
+
+ /* check dst operand */
+ err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
+ if (err)
+ return err;
+
+ src_reg_type = regs[insn->src_reg].type;
+
+ /* Check if (src_reg + off) is readable. The state of dst_reg will be
+ * updated by this call.
+ */
+ err = check_mem_access(env, env->insn_idx, insn->src_reg, insn->off,
+ BPF_SIZE(insn->code), BPF_READ, insn->dst_reg,
+ strict_alignment_once, is_ldsx);
+ err = err ?: save_aux_ptr_type(env, src_reg_type,
+ allow_trust_mismatch);
+ err = err ?: reg_bounds_sanity_check(env, ®s[insn->dst_reg], ctx);
+
+ return err;
+}
+
+static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
+ bool strict_alignment_once)
+{
+ struct bpf_reg_state *regs = cur_regs(env);
+ enum bpf_reg_type dst_reg_type;
+ int err;
+
+ /* check src1 operand */
+ err = check_reg_arg(env, insn->src_reg, SRC_OP);
+ if (err)
+ return err;
+
+ /* check src2 operand */
+ err = check_reg_arg(env, insn->dst_reg, SRC_OP);
+ if (err)
+ return err;
+
+ dst_reg_type = regs[insn->dst_reg].type;
+
+ /* Check if (dst_reg + off) is writeable. */
+ err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,
+ BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg,
+ strict_alignment_once, false);
+ err = err ?: save_aux_ptr_type(env, dst_reg_type, false);
+
+ return err;
+}
+
+static int check_atomic_rmw(struct bpf_verifier_env *env,
+ struct bpf_insn *insn)
+{
+ int load_reg;
+ int err;
if (BPF_SIZE(insn->code) != BPF_W && BPF_SIZE(insn->code) != BPF_DW) {
verbose(env, "invalid atomic operand size\n");
@@ -7439,11 +7580,7 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i
return -EACCES;
}
- if (is_ctx_reg(env, insn->dst_reg) ||
- is_pkt_reg(env, insn->dst_reg) ||
- is_flow_key_reg(env, insn->dst_reg) ||
- is_sk_reg(env, insn->dst_reg) ||
- (is_arena_reg(env, insn->dst_reg) && !bpf_jit_supports_insn(insn, true))) {
+ if (!atomic_ptr_type_ok(env, insn->dst_reg, insn)) {
verbose(env, "BPF_ATOMIC stores into R%d %s is not allowed\n",
insn->dst_reg,
reg_type_str(env, reg_state(env, insn->dst_reg)->type));
@@ -7470,28 +7607,47 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i
/* Check whether we can read the memory, with second call for fetch
* case to simulate the register fill.
*/
- err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off,
+ err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_READ, -1, true, false);
if (!err && load_reg >= 0)
- err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off,
- BPF_SIZE(insn->code), BPF_READ, load_reg,
- true, false);
+ err = check_mem_access(env, env->insn_idx, insn->dst_reg,
+ insn->off, BPF_SIZE(insn->code),
+ BPF_READ, load_reg, true, false);
if (err)
return err;
- if (is_arena_reg(env, insn->dst_reg)) {
- err = save_aux_ptr_type(env, PTR_TO_ARENA, false);
- if (err)
- return err;
- }
+ err = save_aux_ptr_type(env, reg_state(env, insn->dst_reg)->type, false);
+ if (err)
+ return err;
/* Check whether we can write into the same memory. */
- err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off,
+ err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_WRITE, -1, true, false);
if (err)
return err;
return 0;
}
+static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ switch (insn->imm) {
+ case BPF_ADD:
+ case BPF_ADD | BPF_FETCH:
+ case BPF_AND:
+ case BPF_AND | BPF_FETCH:
+ case BPF_OR:
+ case BPF_OR | BPF_FETCH:
+ case BPF_XOR:
+ case BPF_XOR | BPF_FETCH:
+ case BPF_XCHG:
+ case BPF_CMPXCHG:
+ return check_atomic_rmw(env, insn);
+ default:
+ verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n",
+ insn->imm);
+ return -EINVAL;
+ }
+}
+
/* When register 'regno' is used to read the stack (either directly or through
* a helper function) make sure that it's within stack boundary and, depending
* on the access type and privileges, that all elements of the stack are
@@ -8792,7 +8948,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off,
btf_vmlinux, *arg_btf_id,
- strict_type_match)) {
+ strict_type_match, !type_is_alloc(reg->type))) {
verbose(env, "R%d is of type %s but %s is expected\n",
regno, btf_type_name(reg->btf, reg->btf_id),
btf_type_name(btf_vmlinux, *arg_btf_id));
@@ -9068,11 +9224,15 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
return err;
}
- if (register_is_null(reg) && type_may_be_null(arg_type))
+ if (register_is_null(reg) && type_may_be_null(arg_type)) {
/* A NULL register has a SCALAR_VALUE type, so skip
* type checking.
*/
+ err = mark_chain_precision(env, regno);
+ if (err)
+ return err;
goto skip_type_check;
+ }
/* arg_btf_id and arg_size are in a union. */
if (base_type(arg_type) == ARG_PTR_TO_BTF_ID ||
@@ -10095,10 +10255,12 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,
callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY;
__mark_reg_known_zero(&callee->regs[BPF_REG_2]);
callee->regs[BPF_REG_2].map_ptr = caller->regs[BPF_REG_1].map_ptr;
+ callee->regs[BPF_REG_2].map_uid = caller->regs[BPF_REG_1].map_uid;
callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE;
__mark_reg_known_zero(&callee->regs[BPF_REG_3]);
callee->regs[BPF_REG_3].map_ptr = caller->regs[BPF_REG_1].map_ptr;
+ callee->regs[BPF_REG_3].map_uid = caller->regs[BPF_REG_1].map_uid;
/* pointer to stack or null */
callee->regs[BPF_REG_4] = caller->regs[BPF_REG_3];
@@ -10176,6 +10338,7 @@ static int set_timer_callback_state(struct bpf_verifier_env *env,
int insn_idx)
{
struct bpf_map *map_ptr = caller->regs[BPF_REG_1].map_ptr;
+ u32 map_uid = caller->regs[BPF_REG_1].map_uid;
/* bpf_timer_set_callback(struct bpf_timer *timer, void *callback_fn);
* callback_fn(struct bpf_map *map, void *key, void *value);
@@ -10183,14 +10346,17 @@ static int set_timer_callback_state(struct bpf_verifier_env *env,
callee->regs[BPF_REG_1].type = CONST_PTR_TO_MAP;
__mark_reg_known_zero(&callee->regs[BPF_REG_1]);
callee->regs[BPF_REG_1].map_ptr = map_ptr;
+ callee->regs[BPF_REG_1].map_uid = map_uid;
callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY;
__mark_reg_known_zero(&callee->regs[BPF_REG_2]);
callee->regs[BPF_REG_2].map_ptr = map_ptr;
+ callee->regs[BPF_REG_2].map_uid = map_uid;
callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE;
__mark_reg_known_zero(&callee->regs[BPF_REG_3]);
callee->regs[BPF_REG_3].map_ptr = map_ptr;
+ callee->regs[BPF_REG_3].map_uid = map_uid;
/* unused */
__mark_reg_not_init(env, &callee->regs[BPF_REG_4]);
@@ -10285,9 +10451,10 @@ static int set_rbtree_add_callback_state(struct bpf_verifier_env *env,
static bool is_rbtree_lock_required_kfunc(u32 btf_id);
-/* Are we currently verifying the callback for a rbtree helper that must
- * be called with lock held? If so, no need to complain about unreleased
- * lock
+/*
+ * Are we currently verifying the callback for an rbtree kfunc that must
+ * be called with a lock held, or one of that callback's subprogs? If so,
+ * no need to complain about an unreleased lock.
*/
static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env)
{
@@ -10295,17 +10462,19 @@ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env)
struct bpf_insn *insn = env->prog->insnsi;
struct bpf_func_state *callee;
int kfunc_btf_id;
+ u32 frame;
- if (!state->curframe)
- return false;
-
- callee = state->frame[state->curframe];
+ for (frame = state->curframe; frame; frame--) {
+ callee = state->frame[frame];
+ if (!callee->in_callback_fn)
+ continue;
- if (!callee->in_callback_fn)
- return false;
+ kfunc_btf_id = insn[callee->callsite].imm;
+ if (is_rbtree_lock_required_kfunc(kfunc_btf_id))
+ return true;
+ }
- kfunc_btf_id = insn[callee->callsite].imm;
- return is_rbtree_lock_required_kfunc(kfunc_btf_id);
+ return false;
}
static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg,
@@ -10857,6 +11026,9 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
verbose(env, "get_local_storage() doesn't support non-zero flags\n");
return -EINVAL;
}
+ err = mark_chain_precision(env, BPF_REG_2);
+ if (err)
+ return err;
break;
case BPF_FUNC_for_each_map_elem:
err = push_callback_call(env, insn, insn_idx, meta.subprogno,
@@ -11755,7 +11927,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,
reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, ®_ref_id);
reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off);
- struct_same = btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match);
+ struct_same = btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id,
+ strict_type_match, !type_is_alloc(reg->type));
/* If kfunc is accepting a projection type (ie. __sk_buff), it cannot
* actually use it -- it must cast to the underlying type. So we allow
* caller to pass in the underlying type.
@@ -12111,7 +12284,8 @@ __process_kf_arg_ptr_to_graph_node(struct bpf_verifier_env *env,
et = btf_type_by_id(field->graph_root.btf, field->graph_root.value_btf_id);
t = btf_type_by_id(reg->btf, reg->btf_id);
if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, 0, field->graph_root.btf,
- field->graph_root.value_btf_id, true)) {
+ field->graph_root.value_btf_id, true,
+ !type_is_alloc(reg->type))) {
verbose(env, "operation on %s expects arg#1 %s at offset=%d "
"in struct %s, but arg is at offset=%d in struct %s\n",
btf_field_type_name(head_field_type),
@@ -12248,6 +12422,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
}
meta->r0_size = reg->var_off.value;
+ if (meta->r0_size > U32_MAX) {
+ verbose(env, "R%d rdonly/rdwr_buf_size exceeds u32 max\n",
+ regno);
+ return -EINVAL;
+ }
ret = mark_chain_precision(env, regno);
if (ret)
return ret;
@@ -12601,7 +12780,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
verbose(env, "arg#%d is neither owning or non-owning ref\n", i);
return -EINVAL;
}
- if (!type_is_non_owning_ref(reg->type))
+ if (!type_is_non_owning_ref(reg->type) && reg->ref_obj_id)
meta->arg_owning_ref = true;
rec = reg_btf_record(reg);
@@ -15880,7 +16059,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
*/
if (!is_jmp32 && BPF_SRC(insn->code) == BPF_X &&
__is_pointer_value(false, src_reg) && __is_pointer_value(false, dst_reg) &&
- type_may_be_null(src_reg->type) != type_may_be_null(dst_reg->type) &&
base_type(src_reg->type) != PTR_TO_BTF_ID &&
base_type(dst_reg->type) != PTR_TO_BTF_ID) {
eq_branch_regs = NULL;
@@ -15896,9 +16074,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
break;
}
if (eq_branch_regs) {
- if (type_may_be_null(src_reg->type))
+ /* src == dst && dst != NULL => src != NULL */
+ if (reg_not_null(dst_reg) && type_may_be_null(src_reg->type))
mark_ptr_not_null_reg(&eq_branch_regs[insn->src_reg]);
- else
+ /* src == dst && src != NULL => dst != NULL */
+ if (reg_not_null(src_reg) && type_may_be_null(dst_reg->type))
mark_ptr_not_null_reg(&eq_branch_regs[insn->dst_reg]);
}
}
@@ -16064,6 +16244,7 @@ static bool may_access_skb(enum bpf_prog_type type)
*/
static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
{
+ struct bpf_verifier_state *state = env->cur_state;
struct bpf_reg_state *regs = cur_regs(env);
static const int ctx_reg = BPF_REG_6;
u8 mode = BPF_MODE(insn->code);
@@ -16074,6 +16255,13 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
return -EINVAL;
}
+ for (i = state->curframe; i; i--) {
+ if (state->frame[i]->in_callback_fn) {
+ verbose(env, "cannot use BPF_LD_[ABS|IND] within callback\n");
+ return -EINVAL;
+ }
+ }
+
if (!env->ops->gen_ld_abs) {
verbose(env, "bpf verifier is misconfigured\n");
return -EINVAL;
@@ -16469,6 +16657,7 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
/* mark branch target for state pruning */
mark_prune_point(env, w);
mark_jmp_point(env, w);
+ mark_jump_target(env, w);
}
if (insn_state[w] == 0) {
@@ -16657,6 +16846,10 @@ static bool is_fastcall_kfunc_call(struct bpf_kfunc_call_arg_meta *meta)
* r0 = *(u64 *)(r10 - 8); r0 += r1;
* r0 += r1; exit;
* exit;
+ *
+ * Both uses of the marks assume that a pattern is entered at its first
+ * spill and thus executes as a unit, hence a pattern is not grown past
+ * an instruction targeted by a jump.
*/
static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,
struct bpf_subprog_info *subprog,
@@ -16711,6 +16904,10 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,
for (i = 1, off = lowest_off; i <= ARRAY_SIZE(caller_saved); ++i, off += BPF_REG_SIZE) {
if (insn_idx - i < 0 || insn_idx + i >= env->prog->len)
break;
+ /* stx/ldx/call must not be a jump targets, a jump to the first stx is fine */
+ if (bpf_is_jump_target(env, insn_idx - i + 1) ||
+ bpf_is_jump_target(env, insn_idx + i))
+ break;
stx = &insns[insn_idx - i];
ldx = &insns[insn_idx + i];
/* must be a stack spill/fill pair */
@@ -16876,6 +17073,7 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
mark_prune_point(env, t + off + 1);
mark_jmp_point(env, t + off + 1);
+ mark_jump_target(env, t + off + 1);
return ret;
@@ -17954,7 +18152,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* infinite loop check triggering, see
* iter_active_depths_differ()
*/
- if (old_reg->iter.btf != cur_reg->iter.btf ||
+ if (old_reg->type != cur_reg->type ||
+ old_reg->iter.btf != cur_reg->iter.btf ||
old_reg->iter.btf_id != cur_reg->iter.btf_id ||
old_reg->iter.state != cur_reg->iter.state ||
/* ignore {old_reg,cur_reg}->iter.depth, see above */
@@ -18870,37 +19069,18 @@ static int do_check(struct bpf_verifier_env *env)
return err;
} else if (class == BPF_LDX) {
- enum bpf_reg_type src_reg_type;
-
- /* check for reserved fields is already done */
-
- /* check src operand */
- err = check_reg_arg(env, insn->src_reg, SRC_OP);
- if (err)
- return err;
+ bool is_ldsx = BPF_MODE(insn->code) == BPF_MEMSX;
- err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
- if (err)
- return err;
-
- src_reg_type = regs[insn->src_reg].type;
-
- /* check that memory (src_reg + off) is readable,
- * the state of dst_reg will be updated by this func
+ /* Check for reserved fields is already done in
+ * resolve_pseudo_ldimm64().
*/
- err = check_mem_access(env, env->insn_idx, insn->src_reg,
- insn->off, BPF_SIZE(insn->code),
- BPF_READ, insn->dst_reg, false,
- BPF_MODE(insn->code) == BPF_MEMSX);
- err = err ?: save_aux_ptr_type(env, src_reg_type, true);
- err = err ?: reg_bounds_sanity_check(env, ®s[insn->dst_reg], "ldx");
+ err = check_load_mem(env, insn, false, is_ldsx, true,
+ "ldx");
if (err)
return err;
} else if (class == BPF_STX) {
- enum bpf_reg_type dst_reg_type;
-
if (BPF_MODE(insn->code) == BPF_ATOMIC) {
- err = check_atomic(env, env->insn_idx, insn);
+ err = check_atomic(env, insn);
if (err)
return err;
env->insn_idx++;
@@ -18912,25 +19092,7 @@ static int do_check(struct bpf_verifier_env *env)
return -EINVAL;
}
- /* check src1 operand */
- err = check_reg_arg(env, insn->src_reg, SRC_OP);
- if (err)
- return err;
- /* check src2 operand */
- err = check_reg_arg(env, insn->dst_reg, SRC_OP);
- if (err)
- return err;
-
- dst_reg_type = regs[insn->dst_reg].type;
-
- /* check that memory (dst_reg + off) is writeable */
- err = check_mem_access(env, env->insn_idx, insn->dst_reg,
- insn->off, BPF_SIZE(insn->code),
- BPF_WRITE, insn->src_reg, false, false);
- if (err)
- return err;
-
- err = save_aux_ptr_type(env, dst_reg_type, false);
+ err = check_store_reg(env, insn, false);
if (err)
return err;
} else if (class == BPF_ST) {
@@ -19024,20 +19186,30 @@ static int do_check(struct bpf_verifier_env *env)
return -EINVAL;
}
process_bpf_exit_full:
- if (env->cur_state->active_lock.ptr && !env->cur_state->curframe) {
- verbose(env, "bpf_spin_unlock is missing\n");
+ if (env->cur_state->active_lock.ptr &&
+ (exception_exit || !env->cur_state->curframe)) {
+ verbose(env, exception_exit ?
+ "bpf_throw cannot be used inside bpf_spin_lock-ed region\n" :
+ "bpf_spin_unlock is missing\n");
return -EINVAL;
}
- if (env->cur_state->active_rcu_lock && !env->cur_state->curframe) {
- verbose(env, "bpf_rcu_read_unlock is missing\n");
+ if (env->cur_state->active_rcu_lock &&
+ (exception_exit || !env->cur_state->curframe)) {
+ verbose(env, exception_exit ?
+ "bpf_throw cannot be used inside bpf_rcu_read_lock-ed region\n" :
+ "bpf_rcu_read_unlock is missing\n");
return -EINVAL;
}
- if (env->cur_state->active_preempt_lock && !env->cur_state->curframe) {
- verbose(env, "%d bpf_preempt_enable%s missing\n",
- env->cur_state->active_preempt_lock,
- env->cur_state->active_preempt_lock == 1 ? " is" : "(s) are");
+ if (env->cur_state->active_preempt_lock &&
+ (exception_exit || !env->cur_state->curframe)) {
+ if (exception_exit)
+ verbose(env, "bpf_throw cannot be used inside bpf_preempt_disable-ed region\n");
+ else
+ verbose(env, "%d bpf_preempt_enable%s missing\n",
+ env->cur_state->active_preempt_lock,
+ env->cur_state->active_preempt_lock == 1 ? " is" : "(s) are");
return -EINVAL;
}
@@ -22346,6 +22518,17 @@ static int check_non_sleepable_error_inject(u32 btf_id)
return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id);
}
+static bool attach_uses_trampoline_retval(enum bpf_attach_type type)
+{
+ switch (type) {
+ case BPF_MODIFY_RETURN:
+ case BPF_TRACE_FEXIT:
+ return true;
+ default:
+ return false;
+ }
+}
+
int bpf_check_attach_target(struct bpf_verifier_log *log,
const struct bpf_prog *prog,
const struct bpf_prog *tgt_prog,
@@ -22580,6 +22763,14 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
if (ret < 0)
return ret;
+ if (tgt_info->fmodel.ret_size > 8 &&
+ attach_uses_trampoline_retval(prog->expected_attach_type)) {
+ bpf_log(log,
+ "Attach to function %s with a >8 byte return value is not supported for this attach type\n",
+ tname);
+ return -EOPNOTSUPP;
+ }
+
if (tgt_prog) {
if (subprog == 0)
addr = (long) tgt_prog->bpf_func;
@@ -22726,8 +22917,10 @@ static bool can_be_sleepable(struct bpf_prog *prog)
return false;
}
}
- return prog->type == BPF_PROG_TYPE_LSM ||
- prog->type == BPF_PROG_TYPE_KPROBE /* only for uprobes */ ||
+ if (prog->type == BPF_PROG_TYPE_LSM)
+ return prog->expected_attach_type != BPF_LSM_CGROUP;
+
+ return prog->type == BPF_PROG_TYPE_KPROBE /* only for uprobes */ ||
prog->type == BPF_PROG_TYPE_STRUCT_OPS;
}
@@ -22814,7 +23007,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -ENOMEM;
if (tgt_prog && tgt_prog->aux->tail_call_reachable)
- tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX;
+ bpf_trampoline_set_flags(tr, BPF_TRAMP_F_TAIL_CALL_CTX);
prog->aux->dst_trampoline = tr;
return 0;
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index ce012aca272a..97d9818db244 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -3881,33 +3881,62 @@ static int cgroup_cpu_pressure_show(struct seq_file *seq, void *v)
static ssize_t pressure_write(struct kernfs_open_file *of, char *buf,
size_t nbytes, enum psi_res res)
{
- struct cgroup_file_ctx *ctx = of->priv;
+ struct cgroup_file_ctx *ctx;
struct psi_trigger *new;
struct cgroup *cgrp;
struct psi_group *psi;
+ bool need_rtpoll_worker;
+ ssize_t ret = 0;
cgrp = cgroup_kn_lock_live(of->kn, false);
if (!cgrp)
return -ENODEV;
- cgroup_get(cgrp);
- cgroup_kn_unlock(of->kn);
+ ctx = of->priv;
+ if (!ctx) {
+ ret = -ENODEV;
+ goto out_unlock;
+ }
/* Allow only one trigger per file descriptor */
if (ctx->psi.trigger) {
- cgroup_put(cgrp);
- return -EBUSY;
+ ret = -EBUSY;
+ goto out_unlock;
}
psi = cgroup_psi(cgrp);
- new = psi_trigger_create(psi, buf, res, of->file, of);
+ new = psi_trigger_create(psi, buf, res, of->file, of,
+ &need_rtpoll_worker);
if (IS_ERR(new)) {
- cgroup_put(cgrp);
- return PTR_ERR(new);
+ ret = PTR_ERR(new);
+ goto out_unlock;
+ }
+
+ /*
+ * The worker fork must run with neither cgroup_mutex nor the file's
+ * kernfs active reference held. The latter is broken since
+ * cgroup_kn_lock_live(). @of->priv may be released while unlocked, so
+ * recheck before publishing @new.
+ */
+ if (need_rtpoll_worker) {
+ cgroup_unlock();
+ ret = psi_trigger_create_rtpoll_worker(psi);
+ cgroup_lock();
+
+ if (!ret && !of->priv)
+ ret = -ENODEV;
+ if (ret) {
+ psi_trigger_destroy(new);
+ goto out_unlock;
+ }
}
smp_store_release(&ctx->psi.trigger, new);
- cgroup_put(cgrp);
+
+out_unlock:
+ cgroup_kn_unlock(of->kn);
+ if (ret)
+ return ret;
return nbytes;
}
diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c
index c4ce2f5a9745..d8bdebdeac02 100644
--- a/kernel/cgroup/rstat.c
+++ b/kernel/cgroup/rstat.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
#include "cgroup-internal.h"
+#include <linux/cpumask.h>
#include <linux/sched/cputime.h>
#include <linux/bpf.h>
@@ -74,7 +75,7 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu,
}
/**
- * cgroup_rstat_updated - keep track of updated rstat_cpu
+ * __cgroup_rstat_updated - keep track of updated rstat_cpu
* @cgrp: target cgroup
* @cpu: cpu on which rstat_cpu was updated
*
@@ -82,7 +83,7 @@ void _cgroup_rstat_cpu_unlock(raw_spinlock_t *cpu_lock, int cpu,
* rstat_cpu->updated_children list. See the comment on top of
* cgroup_rstat_cpu definition for details.
*/
-__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
+void __cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
{
raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu);
unsigned long flags;
@@ -129,6 +130,18 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
_cgroup_rstat_cpu_unlock(cpu_lock, cpu, cgrp, flags, true);
}
+/*
+ * BPF-facing wrapper for __cgroup_rstat_updated(). Validate the
+ * caller-provided CPU before passing it to the internal rstat updater.
+ */
+__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
+{
+ if (unlikely(cpu < 0 || cpu >= nr_cpu_ids || !cpu_possible(cpu)))
+ return;
+
+ __cgroup_rstat_updated(cgrp, cpu);
+}
+
/**
* cgroup_rstat_push_children - push children cgroups into the given list
* @head: current head of the list (= subtree root)
@@ -511,7 +524,7 @@ static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp,
unsigned long flags)
{
u64_stats_update_end_irqrestore(&rstatc->bsync, flags);
- cgroup_rstat_updated(cgrp, smp_processor_id());
+ __cgroup_rstat_updated(cgrp, smp_processor_id());
put_cpu_ptr(rstatc);
}
diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c
index b0f0d15085db..7481fbb947d3 100644
--- a/kernel/cpu_pm.c
+++ b/kernel/cpu_pm.c
@@ -173,7 +173,7 @@ int cpu_cluster_pm_exit(void)
EXPORT_SYMBOL_GPL(cpu_cluster_pm_exit);
#ifdef CONFIG_PM
-static int cpu_pm_suspend(void)
+static int cpu_pm_suspend(void *data)
{
int ret;
@@ -185,20 +185,24 @@ static int cpu_pm_suspend(void)
return ret;
}
-static void cpu_pm_resume(void)
+static void cpu_pm_resume(void *data)
{
cpu_cluster_pm_exit();
cpu_pm_exit();
}
-static struct syscore_ops cpu_pm_syscore_ops = {
+static const struct syscore_ops cpu_pm_syscore_ops = {
.suspend = cpu_pm_suspend,
.resume = cpu_pm_resume,
};
+static struct syscore cpu_pm_syscore = {
+ .ops = &cpu_pm_syscore_ops,
+};
+
static int cpu_pm_init(void)
{
- register_syscore_ops(&cpu_pm_syscore_ops);
+ register_syscore(&cpu_pm_syscore);
return 0;
}
core_initcall(cpu_pm_init);
diff --git a/kernel/irq/debugfs.c b/kernel/irq/debugfs.c
index c6ffb97966be..8bcde325b05c 100644
--- a/kernel/irq/debugfs.c
+++ b/kernel/irq/debugfs.c
@@ -231,7 +231,7 @@ void irq_add_debugfs_entry(unsigned int irq, struct irq_desc *desc)
if (!irq_dir || !desc || desc->debugfs_file)
return;
- sprintf(name, "%d", irq);
+ sprintf(name, "%u", irq);
desc->debugfs_file = debugfs_create_file(name, 0644, irq_dir, desc,
&dfs_irq_ops);
}
diff --git a/kernel/irq/generic-chip.c b/kernel/irq/generic-chip.c
index 32ffcbb87fa1..a083f497fb0d 100644
--- a/kernel/irq/generic-chip.c
+++ b/kernel/irq/generic-chip.c
@@ -664,7 +664,7 @@ static struct irq_data *irq_gc_get_irq_data(struct irq_chip_generic *gc)
}
#ifdef CONFIG_PM
-static int irq_gc_suspend(void)
+static int irq_gc_suspend(void *data)
{
struct irq_chip_generic *gc;
@@ -684,7 +684,7 @@ static int irq_gc_suspend(void)
return 0;
}
-static void irq_gc_resume(void)
+static void irq_gc_resume(void *data)
{
struct irq_chip_generic *gc;
@@ -707,7 +707,7 @@ static void irq_gc_resume(void)
#define irq_gc_resume NULL
#endif
-static void irq_gc_shutdown(void)
+static void irq_gc_shutdown(void *data)
{
struct irq_chip_generic *gc;
@@ -723,15 +723,19 @@ static void irq_gc_shutdown(void)
}
}
-static struct syscore_ops irq_gc_syscore_ops = {
+static const struct syscore_ops irq_gc_syscore_ops = {
.suspend = irq_gc_suspend,
.resume = irq_gc_resume,
.shutdown = irq_gc_shutdown,
};
+static struct syscore irq_gc_syscore = {
+ .ops = &irq_gc_syscore_ops,
+};
+
static int __init irq_gc_init_ops(void)
{
- register_syscore_ops(&irq_gc_syscore_ops);
+ register_syscore(&irq_gc_syscore);
return 0;
}
device_initcall(irq_gc_init_ops);
diff --git a/kernel/irq/pm.c b/kernel/irq/pm.c
index c556bc49d213..633799741131 100644
--- a/kernel/irq/pm.c
+++ b/kernel/irq/pm.c
@@ -227,21 +227,26 @@ void rearm_wake_irq(unsigned int irq)
/**
* irq_pm_syscore_resume - enable interrupt lines early
+ * @data: syscore context
*
* Enable all interrupt lines with %IRQF_EARLY_RESUME set.
*/
-static void irq_pm_syscore_resume(void)
+static void irq_pm_syscore_resume(void *data)
{
resume_irqs(true);
}
-static struct syscore_ops irq_pm_syscore_ops = {
+static const struct syscore_ops irq_pm_syscore_ops = {
.resume = irq_pm_syscore_resume,
};
+static struct syscore irq_pm_syscore = {
+ .ops = &irq_pm_syscore_ops,
+};
+
static int __init irq_pm_init_ops(void)
{
- register_syscore_ops(&irq_pm_syscore_ops);
+ register_syscore(&irq_pm_syscore);
return 0;
}
diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
index 9081ada81c3d..2ad08b9a4101 100644
--- a/kernel/irq/proc.c
+++ b/kernel/irq/proc.c
@@ -10,6 +10,7 @@
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/interrupt.h>
+#include <linux/kernel.h>
#include <linux/kernel_stat.h>
#include <linux/mutex.h>
@@ -81,20 +82,18 @@ static int show_irq_affinity(int type, struct seq_file *m)
static int irq_affinity_hint_proc_show(struct seq_file *m, void *v)
{
struct irq_desc *desc = irq_to_desc((long)m->private);
- unsigned long flags;
cpumask_var_t mask;
if (!zalloc_cpumask_var(&mask, GFP_KERNEL))
return -ENOMEM;
- raw_spin_lock_irqsave(&desc->lock, flags);
- if (desc->affinity_hint)
- cpumask_copy(mask, desc->affinity_hint);
- raw_spin_unlock_irqrestore(&desc->lock, flags);
+ scoped_guard(raw_spinlock_irq, &desc->lock) {
+ if (desc->affinity_hint)
+ cpumask_copy(mask, desc->affinity_hint);
+ }
seq_printf(m, "%*pb\n", cpumask_pr_args(mask));
free_cpumask_var(mask);
-
return 0;
}
@@ -295,32 +294,26 @@ static int irq_spurious_proc_show(struct seq_file *m, void *v)
#define MAX_NAMELEN 128
-static int name_unique(unsigned int irq, struct irqaction *new_action)
+static bool name_unique(unsigned int irq, struct irqaction *new_action)
{
struct irq_desc *desc = irq_to_desc(irq);
struct irqaction *action;
- unsigned long flags;
- int ret = 1;
- raw_spin_lock_irqsave(&desc->lock, flags);
+ guard(raw_spinlock_irq)(&desc->lock);
for_each_action_of_desc(desc, action) {
if ((action != new_action) && action->name &&
- !strcmp(new_action->name, action->name)) {
- ret = 0;
- break;
- }
+ !strcmp(new_action->name, action->name))
+ return false;
}
- raw_spin_unlock_irqrestore(&desc->lock, flags);
- return ret;
+ return true;
}
void register_handler_proc(unsigned int irq, struct irqaction *action)
{
- char name [MAX_NAMELEN];
+ char name[MAX_NAMELEN];
struct irq_desc *desc = irq_to_desc(irq);
- if (!desc->dir || action->dir || !action->name ||
- !name_unique(irq, action))
+ if (!desc->dir || action->dir || !action->name || !name_unique(irq, action))
return;
snprintf(name, MAX_NAMELEN, "%s", action->name);
@@ -331,7 +324,7 @@ void register_handler_proc(unsigned int irq, struct irqaction *action)
#undef MAX_NAMELEN
-#define MAX_NAMELEN 10
+#define MAX_NAMELEN 11
void register_irq_proc(unsigned int irq, struct irq_desc *desc)
{
@@ -347,17 +340,16 @@ void register_irq_proc(unsigned int irq, struct irq_desc *desc)
* added, not when the descriptor is created, so multiple
* tasks might try to register at the same time.
*/
- mutex_lock(®ister_lock);
+ guard(mutex)(®ister_lock);
if (desc->dir)
- goto out_unlock;
-
- sprintf(name, "%d", irq);
+ return;
/* create /proc/irq/1234 */
+ snprintf(name, MAX_NAMELEN, "%u", irq);
desc->dir = proc_mkdir(name, root_irq_dir);
if (!desc->dir)
- goto out_unlock;
+ return;
#ifdef CONFIG_SMP
umode_t umode = S_IRUGO;
@@ -366,31 +358,27 @@ void register_irq_proc(unsigned int irq, struct irq_desc *desc)
umode |= S_IWUSR;
/* create /proc/irq/<irq>/smp_affinity */
- proc_create_data("smp_affinity", umode, desc->dir,
- &irq_affinity_proc_ops, irqp);
+ proc_create_data("smp_affinity", umode, desc->dir, &irq_affinity_proc_ops, irqp);
/* create /proc/irq/<irq>/affinity_hint */
proc_create_single_data("affinity_hint", 0444, desc->dir,
- irq_affinity_hint_proc_show, irqp);
+ irq_affinity_hint_proc_show, irqp);
/* create /proc/irq/<irq>/smp_affinity_list */
proc_create_data("smp_affinity_list", umode, desc->dir,
&irq_affinity_list_proc_ops, irqp);
- proc_create_single_data("node", 0444, desc->dir, irq_node_proc_show,
- irqp);
+ proc_create_single_data("node", 0444, desc->dir, irq_node_proc_show, irqp);
# ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
proc_create_single_data("effective_affinity", 0444, desc->dir,
- irq_effective_aff_proc_show, irqp);
+ irq_effective_aff_proc_show, irqp);
proc_create_single_data("effective_affinity_list", 0444, desc->dir,
- irq_effective_aff_list_proc_show, irqp);
+ irq_effective_aff_list_proc_show, irqp);
# endif
#endif
proc_create_single_data("spurious", 0444, desc->dir,
- irq_spurious_proc_show, (void *)(long)irq);
+ irq_spurious_proc_show, (void *)(long)irq);
-out_unlock:
- mutex_unlock(®ister_lock);
}
void unregister_irq_proc(unsigned int irq, struct irq_desc *desc)
@@ -411,7 +399,7 @@ void unregister_irq_proc(unsigned int irq, struct irq_desc *desc)
#endif
remove_proc_entry("spurious", desc->dir);
- sprintf(name, "%u", irq);
+ snprintf(name, MAX_NAMELEN, "%u", irq);
remove_proc_entry(name, root_irq_dir);
}
@@ -467,7 +455,6 @@ int show_interrupts(struct seq_file *p, void *v)
int i = *(loff_t *) v, j;
struct irqaction *action;
struct irq_desc *desc;
- unsigned long flags;
if (i > ACTUAL_NR_IRQS)
return 0;
@@ -486,19 +473,19 @@ int show_interrupts(struct seq_file *p, void *v)
seq_putc(p, '\n');
}
- rcu_read_lock();
+ guard(rcu)();
desc = irq_to_desc(i);
if (!desc || irq_settings_is_hidden(desc))
- goto outsparse;
+ return 0;
if (!desc->action || irq_desc_is_chained(desc) || !desc->kstat_irqs)
- goto outsparse;
+ return 0;
seq_printf(p, "%*d: ", prec, i);
for_each_online_cpu(j)
seq_printf(p, "%10u ", desc->kstat_irqs ? per_cpu(desc->kstat_irqs->cnt, j) : 0);
- raw_spin_lock_irqsave(&desc->lock, flags);
+ guard(raw_spinlock_irq)(&desc->lock);
if (desc->irq_data.chip) {
if (desc->irq_data.chip->irq_print_chip)
desc->irq_data.chip->irq_print_chip(&desc->irq_data, p);
@@ -527,9 +514,6 @@ int show_interrupts(struct seq_file *p, void *v)
}
seq_putc(p, '\n');
- raw_spin_unlock_irqrestore(&desc->lock, flags);
-outsparse:
- rcu_read_unlock();
return 0;
}
#endif
diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
index df7769c4fb0f..a947b1a75cce 100644
--- a/kernel/printk/nbcon.c
+++ b/kernel/printk/nbcon.c
@@ -1333,7 +1333,7 @@ bool nbcon_kthread_create(struct console *con)
return true;
kt = kthread_run(nbcon_kthread_func, con, "pr/%s%d", con->name, con->index);
- if (WARN_ON(IS_ERR(kt))) {
+ if (IS_ERR(kt)) {
con_printk(KERN_ERR, con, "failed to start printing thread\n");
return false;
}
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 37f1d4637d90..d5c1dff1617c 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -3638,7 +3638,7 @@ static bool legacy_kthread_create(void)
lockdep_assert_console_list_lock_held();
kt = kthread_run(legacy_kthread_func, NULL, "pr/legacy");
- if (WARN_ON(IS_ERR(kt))) {
+ if (IS_ERR(kt)) {
pr_err("failed to start legacy printing thread\n");
return false;
}
@@ -3656,12 +3656,13 @@ static bool legacy_kthread_create(void)
/**
* printk_kthreads_shutdown - shutdown all threaded printers
+ * @data: syscore context
*
* On system shutdown all threaded printers are stopped. This allows printk
* to transition back to atomic printing, thus providing a robust mechanism
* for the final shutdown/reboot messages to be output.
*/
-static void printk_kthreads_shutdown(void)
+static void printk_kthreads_shutdown(void *data)
{
struct console *con;
@@ -3683,10 +3684,14 @@ static void printk_kthreads_shutdown(void)
console_list_unlock();
}
-static struct syscore_ops printk_syscore_ops = {
+static const struct syscore_ops printk_syscore_ops = {
.shutdown = printk_kthreads_shutdown,
};
+static struct syscore printk_syscore = {
+ .ops = &printk_syscore_ops,
+};
+
/*
* If appropriate, start nbcon kthreads and set @printk_kthreads_running.
* If any kthreads fail to start, those consoles are unregistered.
@@ -3753,7 +3758,7 @@ static void printk_kthreads_check_locked(void)
static int __init printk_set_kthreads_ready(void)
{
- register_syscore_ops(&printk_syscore_ops);
+ register_syscore(&printk_syscore);
console_list_lock();
printk_kthreads_ready = true;
diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c
index 4dcbf8aa80ff..ddea40b68e5f 100644
--- a/kernel/rcu/srcutiny.c
+++ b/kernel/rcu/srcutiny.c
@@ -9,6 +9,7 @@
*/
#include <linux/export.h>
+#include <linux/irq_work.h>
#include <linux/mutex.h>
#include <linux/preempt.h>
#include <linux/rcupdate_wait.h>
@@ -37,6 +38,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
ssp->srcu_idx_max = 0;
INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
INIT_LIST_HEAD(&ssp->srcu_work.entry);
+ init_irq_work(&ssp->srcu_irq_work, srcu_tiny_irq_work);
return 0;
}
@@ -80,6 +82,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
void cleanup_srcu_struct(struct srcu_struct *ssp)
{
WARN_ON(ssp->srcu_lock_nesting[0] || ssp->srcu_lock_nesting[1]);
+ irq_work_sync(&ssp->srcu_irq_work);
flush_work(&ssp->srcu_work);
WARN_ON(ssp->srcu_gp_running);
WARN_ON(ssp->srcu_gp_waiting);
@@ -168,6 +171,20 @@ void srcu_drive_gp(struct work_struct *wp)
}
EXPORT_SYMBOL_GPL(srcu_drive_gp);
+/*
+ * Use an irq_work to defer schedule_work() to avoid acquiring the workqueue
+ * pool->lock while the caller might hold scheduler locks, causing lockdep
+ * splats due to workqueue_init() doing a wakeup.
+ */
+void srcu_tiny_irq_work(struct irq_work *irq_work)
+{
+ struct srcu_struct *ssp;
+
+ ssp = container_of(irq_work, struct srcu_struct, srcu_irq_work);
+ schedule_work(&ssp->srcu_work);
+}
+EXPORT_SYMBOL_GPL(srcu_tiny_irq_work);
+
static void srcu_gp_start_if_needed(struct srcu_struct *ssp)
{
unsigned long cookie;
@@ -181,7 +198,7 @@ static void srcu_gp_start_if_needed(struct srcu_struct *ssp)
WRITE_ONCE(ssp->srcu_idx_max, cookie);
if (!READ_ONCE(ssp->srcu_gp_running)) {
if (likely(srcu_init_done))
- schedule_work(&ssp->srcu_work);
+ irq_work_queue(&ssp->srcu_irq_work);
else if (list_empty(&ssp->srcu_work.entry))
list_add(&ssp->srcu_work.entry, &srcu_boot_list);
}
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 3a1983ef278f..876adf5568ed 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -2113,8 +2113,8 @@ static void set_task_runnable(struct rq *rq, struct task_struct *p)
}
/*
- * list_add_tail() must be used. scx_ops_bypass() depends on tasks being
- * appened to the runnable_list.
+ * list_add_tail() must be used. scx_ops_bypass() and rq_offline_scx()
+ * depend on tasks being appended to the runnable_list.
*/
list_add_tail(&p->scx.runnable_node, &rq->scx.runnable_list);
}
@@ -2823,7 +2823,14 @@ static int balance_one(struct rq *rq, struct task_struct *prev)
if (consume_global_dsq(rq))
goto has_tasks;
- if (!SCX_HAS_OP(dispatch) || scx_rq_bypassing(rq) || !scx_rq_online(rq))
+ /*
+ * scx_rq_online() can't be used. Its cpu_active() test goes false
+ * before CPU hotplug waits for an RCU grace period, and
+ * rq_offline_scx() moves this CPU's tasks to the local DSQ only after
+ * the wait. The grace period can depend on those tasks running.
+ */
+ if (!SCX_HAS_OP(dispatch) || scx_rq_bypassing(rq) ||
+ !(rq->scx.flags & SCX_RQ_ONLINE))
goto no_tasks;
dspc->rq = rq;
@@ -3078,9 +3085,14 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
* sched_class, %SCX_OPS_ENQ_LAST must be set. Tell
* ops.enqueue() that @p is the only one available for this cpu,
* which should trigger an explicit follow-up scheduling event.
+ *
+ * Core scheduling can force this CPU idle while @p stays
+ * runnable. @p's cookie then won't match the core's, so skip
+ * the warning in that case.
*/
if (sched_class_above(&ext_sched_class, next->sched_class)) {
- WARN_ON_ONCE(!static_branch_unlikely(&scx_ops_enq_last));
+ WARN_ON_ONCE(sched_cpu_cookie_match(rq, p) &&
+ !static_branch_unlikely(&scx_ops_enq_last));
do_enqueue_task(rq, p, SCX_ENQ_LAST, -1);
} else {
do_enqueue_task(rq, p, 0, -1);
@@ -3514,7 +3526,27 @@ static void rq_online_scx(struct rq *rq)
static void rq_offline_scx(struct rq *rq)
{
+ struct task_struct *p, *n;
+
rq->scx.flags &= ~SCX_RQ_ONLINE;
+
+ /* sched domain rebuilds call rq_offline with the CPU staying alive */
+ if (cpu_active(cpu_of(rq)))
+ return;
+
+ /*
+ * An offline CPU no longer calls ops.dispatch(). Re-enqueue its tasks
+ * onto the local DSQ so that they run here and balance_push() moves
+ * them off.
+ */
+ list_for_each_entry_safe_reverse(p, n, &rq->scx.runnable_list, scx.runnable_node) {
+ struct sched_enq_and_set_ctx ctx;
+
+ if (p->scx.dsq == &rq->scx.local_dsq)
+ continue;
+ sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK, &ctx);
+ sched_enq_and_set_task(&ctx);
+ }
}
#else /* CONFIG_SMP */
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index a6fc7ea51d2e..7550f609e444 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -1289,9 +1289,44 @@ int psi_show(struct seq_file *m, struct psi_group *group, enum psi_res res)
return 0;
}
+/*
+ * Create @group's rtpoll worker after psi_trigger_create() reported the need
+ * for one. kthread creation depends on the whole fork path and we don't want
+ * all of that nested inside cgroup_mutex, so the caller must drop it and any
+ * other lock that forks can wait behind. If two callers race, the loser stops
+ * its never-woken kthread.
+ */
+int psi_trigger_create_rtpoll_worker(struct psi_group *group)
+{
+ struct task_struct *task;
+
+ task = kthread_create(psi_rtpoll_worker, group, "psimon");
+ if (IS_ERR(task))
+ return PTR_ERR(task);
+
+ scoped_guard(mutex, &group->rtpoll_trigger_lock) {
+ if (!rcu_access_pointer(group->rtpoll_task)) {
+ atomic_set(&group->rtpoll_wakeup, 0);
+ wake_up_process(task);
+ rcu_assign_pointer(group->rtpoll_task, task);
+
+ /*
+ * Poll once to catch up on scheduling attempts dropped
+ * while there was no rtpoll worker.
+ */
+ psi_schedule_rtpoll_work(group, 1, true);
+ return 0;
+ }
+ }
+
+ kthread_stop(task);
+ return 0;
+}
+
struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
enum psi_res res, struct file *file,
- struct kernfs_open_file *of)
+ struct kernfs_open_file *of,
+ bool *need_rtpoll_worker)
{
struct psi_trigger *t;
enum psi_states state;
@@ -1299,6 +1334,8 @@ struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
bool privileged;
u32 window_us;
+ *need_rtpoll_worker = false;
+
if (static_branch_likely(&psi_disabled))
return ERR_PTR(-EOPNOTSUPP);
@@ -1359,26 +1396,14 @@ struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
if (privileged) {
mutex_lock(&group->rtpoll_trigger_lock);
- if (!rcu_access_pointer(group->rtpoll_task)) {
- struct task_struct *task;
-
- task = kthread_create(psi_rtpoll_worker, group, "psimon");
- if (IS_ERR(task)) {
- kfree(t);
- mutex_unlock(&group->rtpoll_trigger_lock);
- return ERR_CAST(task);
- }
- atomic_set(&group->rtpoll_wakeup, 0);
- wake_up_process(task);
- rcu_assign_pointer(group->rtpoll_task, task);
- }
-
list_add(&t->node, &group->rtpoll_triggers);
group->rtpoll_min_period = min(group->rtpoll_min_period,
div_u64(t->win.size, UPDATES_PER_WINDOW));
group->rtpoll_nr_triggers[t->state]++;
group->rtpoll_states |= (1 << t->state);
+ *need_rtpoll_worker = !rcu_access_pointer(group->rtpoll_task);
+
mutex_unlock(&group->rtpoll_trigger_lock);
} else {
mutex_lock(&group->avgs_lock);
@@ -1538,6 +1563,8 @@ static ssize_t psi_write(struct file *file, const char __user *user_buf,
size_t buf_size;
struct seq_file *seq;
struct psi_trigger *new;
+ bool need_rtpoll_worker;
+ int ret;
if (static_branch_likely(&psi_disabled))
return -EOPNOTSUPP;
@@ -1562,12 +1589,22 @@ static ssize_t psi_write(struct file *file, const char __user *user_buf,
return -EBUSY;
}
- new = psi_trigger_create(&psi_system, buf, res, file, NULL);
+ new = psi_trigger_create(&psi_system, buf, res, file, NULL,
+ &need_rtpoll_worker);
if (IS_ERR(new)) {
mutex_unlock(&seq->lock);
return PTR_ERR(new);
}
+ if (need_rtpoll_worker) {
+ ret = psi_trigger_create_rtpoll_worker(&psi_system);
+ if (ret) {
+ psi_trigger_destroy(new);
+ mutex_unlock(&seq->lock);
+ return ret;
+ }
+ }
+
smp_store_release(&seq->private, new);
mutex_unlock(&seq->lock);
diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c
index f9cb7896c1b9..b51b39a109e0 100644
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -723,6 +723,23 @@ int __sched_setscheduler(struct task_struct *p,
__setscheduler_params(p, attr);
p->sched_class = next_class;
p->prio = newprio;
+#ifdef CONFIG_RT_MUTEXES
+ /*
+ * In case a DEADLINE task (either proper or boosted) gets
+ * setscheduled to a lower priority class, check if it neeeds to
+ * inherit parameters from a potential pi_task. In that case make
+ * sure replenishment happens with the next enqueue.
+ */
+
+ if (dl_prio(newprio) && !dl_policy(policy)) {
+ struct task_struct *pi_task = rt_mutex_get_top_task(p);
+
+ if (pi_task) {
+ p->dl.pi_se = pi_task->dl.pi_se;
+ queue_flags |= ENQUEUE_REPLENISH;
+ }
+ }
+#endif /* CONFIG_RT_MUTEXES */
}
__setscheduler_uclamp(p, attr);
check_class_changing(rq, p, prev_class);
diff --git a/kernel/time/namespace.c b/kernel/time/namespace.c
index 0775b9ec952a..09a4cc85a9fb 100644
--- a/kernel/time/namespace.c
+++ b/kernel/time/namespace.c
@@ -410,6 +410,11 @@ int proc_timens_set_offset(struct file *file, struct task_struct *p,
goto out;
}
+ if (off->val.tv_nsec < 0 || off->val.tv_nsec >= NSEC_PER_SEC) {
+ err = -EINVAL;
+ goto out;
+ }
+
err = -ERANGE;
if (off->val.tv_sec > KTIME_SEC_MAX ||
diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c
index 68d6c1190ac7..8aacdbde3a10 100644
--- a/kernel/time/sched_clock.c
+++ b/kernel/time/sched_clock.c
@@ -283,6 +283,11 @@ int sched_clock_suspend(void)
return 0;
}
+static int sched_clock_syscore_suspend(void *data)
+{
+ return sched_clock_suspend();
+}
+
void sched_clock_resume(void)
{
struct clock_read_data *rd = &cd.read_data[0];
@@ -292,14 +297,23 @@ void sched_clock_resume(void)
rd->read_sched_clock = cd.actual_read_sched_clock;
}
-static struct syscore_ops sched_clock_ops = {
- .suspend = sched_clock_suspend,
- .resume = sched_clock_resume,
+static void sched_clock_syscore_resume(void *data)
+{
+ sched_clock_resume();
+}
+
+static const struct syscore_ops sched_clock_syscore_ops = {
+ .suspend = sched_clock_syscore_suspend,
+ .resume = sched_clock_syscore_resume,
+};
+
+static struct syscore sched_clock_syscore = {
+ .ops = &sched_clock_syscore_ops,
};
static int __init sched_clock_syscore_init(void)
{
- register_syscore_ops(&sched_clock_ops);
+ register_syscore(&sched_clock_syscore);
return 0;
}
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 4d8927aeca97..626fbb2a1a8c 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -1868,6 +1868,11 @@ void timekeeping_resume(void)
timerfd_resume();
}
+static void timekeeping_syscore_resume(void *data)
+{
+ timekeeping_resume();
+}
+
int timekeeping_suspend(void)
{
struct timekeeper *tk = &tk_core.timekeeper;
@@ -1937,15 +1942,24 @@ int timekeeping_suspend(void)
return 0;
}
+static int timekeeping_syscore_suspend(void *data)
+{
+ return timekeeping_suspend();
+}
+
/* sysfs resume/suspend bits for timekeeping */
-static struct syscore_ops timekeeping_syscore_ops = {
- .resume = timekeeping_resume,
- .suspend = timekeeping_suspend,
+static const struct syscore_ops timekeeping_syscore_ops = {
+ .resume = timekeeping_syscore_resume,
+ .suspend = timekeeping_syscore_suspend,
+};
+
+static struct syscore timekeeping_syscore = {
+ .ops = &timekeeping_syscore_ops,
};
static int __init timekeeping_init_ops(void)
{
- register_syscore_ops(&timekeeping_syscore_ops);
+ register_syscore(&timekeeping_syscore);
return 0;
}
device_initcall(timekeeping_init_ops);
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index c54e7e910b43..6e84f2f1eb20 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -347,6 +347,11 @@ struct buffer_data_read_page {
struct buffer_data_page *data; /* actual data, stored in this page */
};
+static __always_inline unsigned int rb_read_page_capacity(struct buffer_data_read_page *rpage)
+{
+ return (PAGE_SIZE << rpage->order) - BUF_PAGE_HDR_SIZE;
+}
+
/*
* Note, the buffer_page list must be first. The buffer pages
* are allocated in cache lines, which means that each buffer
@@ -1888,6 +1893,8 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
if (!meta || !meta->head_buffer)
return;
+ head_page = cpu_buffer->head_page;
+
/* Do the reader page first */
ret = rb_validate_buffer(cpu_buffer->reader_page->page, cpu_buffer->cpu);
if (ret < 0) {
@@ -1898,8 +1905,6 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
entry_bytes += local_read(&cpu_buffer->reader_page->page->commit);
local_set(&cpu_buffer->reader_page->entries, ret);
- head_page = cpu_buffer->head_page;
-
/* If the commit_buffer is the reader page, update the commit page */
if (meta->commit_buffer == (unsigned long)cpu_buffer->reader_page->page) {
cpu_buffer->commit_page = cpu_buffer->reader_page;
@@ -6391,56 +6396,78 @@ EXPORT_SYMBOL_GPL(ring_buffer_swap_cpu);
* ring_buffer_alloc_read_page - allocate a page to read from buffer
* @buffer: the buffer to allocate for.
* @cpu: the cpu buffer to allocate.
+ * @rpage: pointer to pass in an already allocated page (can be NULL)
+ * and returns the allocated page.
*
- * This function is used in conjunction with ring_buffer_read_page.
+ * This function is used in conjunction with ring_buffer_read_page().
* When reading a full page from the ring buffer, these functions
* can be used to speed up the process. The calling function should
* allocate a few pages first with this function. Then when it
* needs to get pages from the ring buffer, it passes the result
- * of this function into ring_buffer_read_page, which will swap
+ * of this function into ring_buffer_read_page(), which will swap
* the page that was allocated, with the read page of the buffer.
*
+ * If @rpage is provided, and it has a different order than the current
+ * subbuffer order, its payload will be freed and re-allocated. If it
+ * already matches the order, it is simply returned.
+ *
* Returns:
- * The page allocated, or ERR_PTR
+ * 0 on success, < 0 on error
*/
-struct buffer_data_read_page *
-ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)
+int ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu,
+ struct buffer_data_read_page **rpage)
{
struct ring_buffer_per_cpu *cpu_buffer;
- struct buffer_data_read_page *bpage = NULL;
unsigned long flags;
+ unsigned int order;
if (!cpumask_test_cpu(cpu, buffer->cpumask))
- return ERR_PTR(-ENODEV);
+ return -ENODEV;
- bpage = kzalloc(sizeof(*bpage), GFP_KERNEL);
- if (!bpage)
- return ERR_PTR(-ENOMEM);
+ if (!rpage)
+ return -EINVAL;
+
+ order = READ_ONCE(buffer->subbuf_order);
+
+ if (*rpage) {
+ if ((*rpage)->order == order)
+ return 0;
+
+ /* We can reuse rpage, but we discard the payload */
+ free_pages((unsigned long)(*rpage)->data, (*rpage)->order);
+ (*rpage)->data = NULL;
+ } else {
+ *rpage = kzalloc_obj(**rpage);
+ if (!*rpage)
+ return -ENOMEM;
+ }
- bpage->order = buffer->subbuf_order;
+ (*rpage)->order = order;
cpu_buffer = buffer->buffers[cpu];
+
local_irq_save(flags);
arch_spin_lock(&cpu_buffer->lock);
if (cpu_buffer->free_page.data) {
- *bpage = cpu_buffer->free_page;
+ **rpage = cpu_buffer->free_page;
cpu_buffer->free_page.data = NULL;
}
arch_spin_unlock(&cpu_buffer->lock);
local_irq_restore(flags);
- if (bpage->data) {
- rb_init_data_page(bpage->data);
+ if ((*rpage)->data) {
+ rb_init_data_page((*rpage)->data);
} else {
- bpage->data = alloc_cpu_data(cpu, bpage->order);
- if (!bpage->data) {
- kfree(bpage);
- return ERR_PTR(-ENOMEM);
+ (*rpage)->data = alloc_cpu_data(cpu, (*rpage)->order);
+ if (!(*rpage)->data) {
+ kfree(*rpage);
+ *rpage = NULL;
+ return -ENOMEM;
}
}
- return bpage;
+ return 0;
}
EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);
@@ -6448,21 +6475,30 @@ EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);
* ring_buffer_free_read_page - free an allocated read page
* @buffer: the buffer the page was allocate for
* @cpu: the cpu buffer the page came from
- * @data_page: the page to free
+ * @rpage: the buffer_data_read_page to free
*
* Free a page allocated from ring_buffer_alloc_read_page.
*/
void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
- struct buffer_data_read_page *data_page)
+ struct buffer_data_read_page *rpage)
{
struct ring_buffer_per_cpu *cpu_buffer;
- struct buffer_data_page *bpage = data_page->data;
- struct page *page = virt_to_page(bpage);
+ struct buffer_data_page *bpage;
unsigned long flags;
+ struct page *page;
if (!buffer || !buffer->buffers || !buffer->buffers[cpu])
return;
+ if (!rpage)
+ return;
+
+ bpage = rpage->data;
+ if (!bpage)
+ goto out;
+
+ page = virt_to_page(bpage);
+
cpu_buffer = buffer->buffers[cpu];
/*
@@ -6470,14 +6506,14 @@ void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
* is different from the subbuffer order of the buffer -
* we can't reuse it
*/
- if (page_ref_count(page) > 1 || data_page->order != buffer->subbuf_order)
+ if (page_ref_count(page) > 1 || rpage->order != READ_ONCE(buffer->subbuf_order))
goto out;
local_irq_save(flags);
arch_spin_lock(&cpu_buffer->lock);
if (!cpu_buffer->free_page.data) {
- cpu_buffer->free_page = *data_page;
+ cpu_buffer->free_page = *rpage;
bpage = NULL;
}
@@ -6485,8 +6521,8 @@ void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
local_irq_restore(flags);
out:
- free_pages((unsigned long)bpage, data_page->order);
- kfree(data_page);
+ free_pages((unsigned long)bpage, rpage->order);
+ kfree(rpage);
}
EXPORT_SYMBOL_GPL(ring_buffer_free_read_page);
@@ -6557,10 +6593,9 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
if (!bpage)
return -1;
- guard(raw_spinlock_irqsave)(&cpu_buffer->reader_lock);
+ len = min_t(size_t, len, rb_read_page_capacity(data_page));
- if (data_page->order != cpu_buffer->reader_page->order)
- return -1;
+ guard(raw_spinlock_irqsave)(&cpu_buffer->reader_lock);
reader = rb_get_reader_page(cpu_buffer);
if (!reader)
@@ -6575,16 +6610,18 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
/* Check if any events were dropped */
missed_events = cpu_buffer->lost_events;
- /*
- * If this page has been partially read or
- * if len is not big enough to read the rest of the page or
- * a writer is still on the page, then
- * we must copy the data from the page to the buffer.
- * Otherwise, we can simply swap the page with the one passed in.
- */
+ /*
+ * It is not possible to swap the reader page if:
+ * - It has been partially read
+ * - len is not big enough to read it entirely
+ * - A writer is still on it
+ * - The ring buffer is static
+ * - The order doesn't match
+ */
if (read || (len < (size - read)) ||
cpu_buffer->reader_page == cpu_buffer->commit_page ||
- cpu_buffer->mapped) {
+ cpu_buffer->mapped ||
+ data_page->order != reader->order) {
struct buffer_data_page *rpage = cpu_buffer->reader_page->page;
unsigned int rpos = read;
unsigned int pos = 0;
@@ -6676,7 +6713,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
/* If there is room at the end of the page to save the
* missed events, then record it there.
*/
- if (rb_page_capacity(reader) - size >= sizeof(missed_events)) {
+ if (rb_read_page_capacity(data_page) - size >= sizeof(missed_events)) {
memcpy(&bpage->data[size], &missed_events,
sizeof(missed_events));
commit |= RB_MISSED_STORED;
@@ -6688,8 +6725,8 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
/*
* This page may be off to user land. Zero it out here.
*/
- if (size < rb_page_capacity(reader))
- memset(&bpage->data[size], 0, rb_page_capacity(reader) - size);
+ if (size < rb_read_page_capacity(data_page))
+ memset(&bpage->data[size], 0, rb_read_page_capacity(data_page) - size);
return read;
}
@@ -6707,6 +6744,18 @@ void *ring_buffer_read_page_data(struct buffer_data_read_page *page)
}
EXPORT_SYMBOL_GPL(ring_buffer_read_page_data);
+/**
+ * ring_buffer_read_page_size - get size of the read page.
+ * @rpage: the page to get the size from
+ *
+ * Returns size of the page in bytes.
+ */
+unsigned int ring_buffer_read_page_size(struct buffer_data_read_page *rpage)
+{
+ return rpage ? PAGE_SIZE << rpage->order : 0;
+}
+EXPORT_SYMBOL_GPL(ring_buffer_read_page_size);
+
/**
* ring_buffer_subbuf_size_get - get size of the sub buffer.
* @buffer: the buffer to get the sub buffer size from
@@ -6800,7 +6849,7 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
/* Make sure all commits have finished */
synchronize_rcu();
- buffer->subbuf_order = order;
+ WRITE_ONCE(buffer->subbuf_order, order);
/* Make sure all new buffers are allocated, before deleting the old ones */
for_each_buffer_cpu(buffer, cpu) {
@@ -6904,7 +6953,7 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
return 0;
error:
- buffer->subbuf_order = old_order;
+ WRITE_ONCE(buffer->subbuf_order, old_order);
atomic_dec(&buffer->record_disabled);
diff --git a/kernel/trace/ring_buffer_benchmark.c b/kernel/trace/ring_buffer_benchmark.c
index 008187ebd7fe..56f4082e14c8 100644
--- a/kernel/trace/ring_buffer_benchmark.c
+++ b/kernel/trace/ring_buffer_benchmark.c
@@ -104,7 +104,7 @@ static enum event_status read_event(int cpu)
static enum event_status read_page(int cpu)
{
- struct buffer_data_read_page *bpage;
+ struct buffer_data_read_page *bpage = NULL;
struct ring_buffer_event *event;
struct rb_page *rpage;
unsigned long commit;
@@ -114,8 +114,8 @@ static enum event_status read_page(int cpu)
int inc;
int i;
- bpage = ring_buffer_alloc_read_page(buffer, cpu);
- if (IS_ERR(bpage))
+ ret = ring_buffer_alloc_read_page(buffer, cpu, &bpage);
+ if (ret < 0)
return EVENT_DROPPED;
page_size = ring_buffer_subbuf_size_get(buffer);
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index ed1014292c08..115963797fab 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -7177,11 +7177,10 @@ int tracing_set_filter_buffering(struct trace_array *tr, bool set)
}
struct ftrace_buffer_info {
- struct trace_iterator iter;
- void *spare;
- unsigned int spare_cpu;
- unsigned int spare_size;
- unsigned int read;
+ struct trace_iterator iter;
+ struct buffer_data_read_page *spare;
+ unsigned int spare_cpu;
+ unsigned int read;
};
#ifdef CONFIG_TRACER_SNAPSHOT
@@ -7890,8 +7889,8 @@ tracing_buffers_read(struct file *filp, char __user *ubuf,
{
struct ftrace_buffer_info *info = filp->private_data;
struct trace_iterator *iter = &info->iter;
+ unsigned int spare_size;
void *trace_data;
- int page_size;
ssize_t ret = 0;
ssize_t size;
@@ -7903,36 +7902,22 @@ tracing_buffers_read(struct file *filp, char __user *ubuf,
return -EBUSY;
#endif
- page_size = ring_buffer_subbuf_size_get(iter->array_buffer->buffer);
+ spare_size = ring_buffer_read_page_size(info->spare);
- /* Make sure the spare matches the current sub buffer size */
- if (info->spare) {
- if (page_size != info->spare_size) {
- ring_buffer_free_read_page(iter->array_buffer->buffer,
- info->spare_cpu, info->spare);
- info->spare = NULL;
- }
- }
+again:
+ /* Do we have previous read data to read? */
+ if (info->read < spare_size)
+ goto read;
- if (!info->spare) {
- info->spare = ring_buffer_alloc_read_page(iter->array_buffer->buffer,
- iter->cpu_file);
- if (IS_ERR(info->spare)) {
- ret = PTR_ERR(info->spare);
- info->spare = NULL;
- } else {
- info->spare_cpu = iter->cpu_file;
- info->spare_size = page_size;
- }
- }
- if (!info->spare)
+ ret = ring_buffer_alloc_read_page(iter->array_buffer->buffer, iter->cpu_file,
+ &info->spare);
+ if (ret)
return ret;
- /* Do we have previous read data to read? */
- if (info->read < page_size)
- goto read;
+ spare_size = ring_buffer_read_page_size(info->spare);
+ info->read = spare_size;
+ info->spare_cpu = iter->cpu_file;
- again:
trace_access_lock(iter->cpu_file);
ret = ring_buffer_read_page(iter->array_buffer->buffer,
info->spare,
@@ -7955,8 +7940,9 @@ tracing_buffers_read(struct file *filp, char __user *ubuf,
}
info->read = 0;
+
read:
- size = page_size - info->read;
+ size = spare_size - info->read;
if (size > count)
size = count;
trace_data = ring_buffer_read_page_data(info->spare);
@@ -7997,9 +7983,7 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
__trace_array_put(iter->tr);
- if (info->spare)
- ring_buffer_free_read_page(iter->array_buffer->buffer,
- info->spare_cpu, info->spare);
+ ring_buffer_free_read_page(iter->array_buffer->buffer, info->spare_cpu, info->spare);
kvfree(info);
mutex_unlock(&trace_types_lock);
@@ -8008,17 +7992,17 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
}
struct buffer_ref {
- struct trace_buffer *buffer;
- void *page;
- int cpu;
- refcount_t refcount;
+ struct trace_buffer *buffer;
+ struct buffer_data_read_page *rpage;
+ int cpu;
+ refcount_t refcount;
};
static void buffer_ref_release(struct buffer_ref *ref)
{
if (!refcount_dec_and_test(&ref->refcount))
return;
- ring_buffer_free_read_page(ref->buffer, ref->cpu, ref->page);
+ ring_buffer_free_read_page(ref->buffer, ref->cpu, ref->rpage);
kfree(ref);
}
@@ -8078,9 +8062,9 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
.ops = &buffer_pipe_buf_ops,
.spd_release = buffer_spd_release,
};
+ unsigned int page_size = 0;
struct buffer_ref *ref;
bool woken = false;
- int page_size;
int entries, i;
ssize_t ret = 0;
@@ -8089,16 +8073,6 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
return -EBUSY;
#endif
- page_size = ring_buffer_subbuf_size_get(iter->array_buffer->buffer);
- if (*ppos & (page_size - 1))
- return -EINVAL;
-
- if (len & (page_size - 1)) {
- if (len < page_size)
- return -EINVAL;
- len &= (~(page_size - 1));
- }
-
if (splice_grow_spd(pipe, &spd))
return -ENOMEM;
@@ -8118,25 +8092,39 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
refcount_set(&ref->refcount, 1);
ref->buffer = iter->array_buffer->buffer;
- ref->page = ring_buffer_alloc_read_page(ref->buffer, iter->cpu_file);
- if (IS_ERR(ref->page)) {
- ret = PTR_ERR(ref->page);
- ref->page = NULL;
+
+ ret = ring_buffer_alloc_read_page(ref->buffer, iter->cpu_file, &ref->rpage);
+ if (ret) {
kfree(ref);
break;
}
ref->cpu = iter->cpu_file;
- r = ring_buffer_read_page(ref->buffer, ref->page,
- len, iter->cpu_file, 1);
+ page_size = ring_buffer_read_page_size(ref->rpage);
+
+ r = -EINVAL;
+ if (IS_ALIGNED(*ppos, page_size) && len >= page_size) {
+ r = ring_buffer_read_page(ref->buffer, ref->rpage, len, iter->cpu_file, 1);
+ } else if (!i) {
+ /*
+ * If this fails to read on the first iteration, it
+ * means the length was too small and an error should
+ * be returned to user space. Otherwise, at least
+ * one sub-buffer was successfully read but this failed
+ * due to either the length was unaligned or the
+ * subbuf order changed. Either case, do not report
+ * an error.
+ */
+ ret = -EINVAL;
+ }
+
if (r < 0) {
- ring_buffer_free_read_page(ref->buffer, ref->cpu,
- ref->page);
+ ring_buffer_free_read_page(ref->buffer, ref->cpu, ref->rpage);
kfree(ref);
break;
}
- page = virt_to_page(ring_buffer_read_page_data(ref->page));
+ page = virt_to_page(ring_buffer_read_page_data(ref->rpage));
spd.pages[i] = page;
spd.partial[i].len = page_size;
diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c
index 9c6ca26ffb92..95201c0daa8f 100644
--- a/kernel/trace/trace_probe.c
+++ b/kernel/trace/trace_probe.c
@@ -622,7 +622,7 @@ static int parse_btf_arg(char *varname,
int i, is_ptr, ret;
u32 tid;
- if (WARN_ON_ONCE(!ctx->funcname))
+ if (!ctx->funcname)
return -EINVAL;
is_ptr = split_next_field(varname, &field, ctx);
diff --git a/lib/kunit_iov_iter.c b/lib/kunit_iov_iter.c
index 13e15687675a..10a560feb66e 100644
--- a/lib/kunit_iov_iter.c
+++ b/lib/kunit_iov_iter.c
@@ -392,7 +392,7 @@ static void __init iov_kunit_load_folioq(struct kunit *test,
if (folioq_full(p)) {
p->next = kzalloc(sizeof(struct folio_queue), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, p->next);
- folioq_init(p->next);
+ folioq_init(p->next, 0);
p->next->prev = p;
p = p->next;
}
@@ -409,7 +409,7 @@ static struct folio_queue *iov_kunit_create_folioq(struct kunit *test)
folioq = kzalloc(sizeof(struct folio_queue), GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, folioq);
kunit_add_action_or_reset(test, iov_kunit_destroy_folioq, folioq);
- folioq_init(folioq);
+ folioq_init(folioq, 0);
return folioq;
}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1a76a21724d4..348fb144e63b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2112,6 +2112,9 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
zap_deposited_table(tlb->mm, pmd);
add_mm_counter(tlb->mm, mm_counter_file(folio),
-HPAGE_PMD_NR);
+
+ if (pmd_present(orig_pmd) && pmd_dirty(orig_pmd))
+ folio_mark_dirty(folio);
}
spin_unlock(ptl);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 7fccb866dab9..68e0cc0748ca 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3260,6 +3260,10 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid)
struct huge_bootmem_page *m;
struct hstate *h = NULL, *prev_h = NULL;
+ /* The lists are only initialized once a gigantic hstate was allocated. */
+ if (!huge_boot_pages[nid].next)
+ return;
+
list_for_each_entry(m, &huge_boot_pages[nid], list) {
struct page *page = virt_to_page(m);
struct folio *folio = (void *)page;
@@ -3302,7 +3306,7 @@ static void __init gather_bootmem_prealloc_parallel(unsigned long start,
gather_bootmem_prealloc_node(nid);
}
-static void __init gather_bootmem_prealloc(void)
+void __init hugetlb_bootmem_struct_page_init(void)
{
struct padata_mt_job job = {
.thread_fn = gather_bootmem_prealloc_parallel,
@@ -3457,7 +3461,7 @@ static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h)
* - For gigantic pages, this is called early in the boot process and
* pages are allocated from memblock allocated or something similar.
* Gigantic pages are actually added to pools later with the routine
- * gather_bootmem_prealloc.
+ * hugetlb_bootmem_struct_page_init.
* - For non-gigantic pages, this is called later in the boot process after
* all of mm is up and functional. Pages are allocated from buddy and
* then added to hugetlb pools.
@@ -4496,7 +4500,6 @@ static int __init hugetlb_init(void)
hugetlb_cma_check();
hugetlb_init_hstates();
- gather_bootmem_prealloc();
report_hugepages();
hugetlb_sysfs_init();
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 95496e935843..52e68a7df2c8 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -24,8 +24,8 @@
*
* @remap_pte: called for each lowest-level entry (PTE).
* @nr_walked: the number of walked pte.
- * @reuse_page: the page which is reused for the tail vmemmap pages.
- * @reuse_addr: the virtual address of the @reuse_page page.
+ * @vmemmap_head: the page to be installed as first in the vmemmap range
+ * @vmemmap_tail: the page to be installed as non-first in the vmemmap range
* @vmemmap_pages: the list head of the vmemmap pages that can be freed
* or is mapped from.
* @flags: used to modify behavior in vmemmap page table walking
@@ -34,11 +34,13 @@
struct vmemmap_remap_walk {
void (*remap_pte)(pte_t *pte, unsigned long addr,
struct vmemmap_remap_walk *walk);
+
unsigned long nr_walked;
- struct page *reuse_page;
- unsigned long reuse_addr;
+ struct page *vmemmap_head;
+ struct page *vmemmap_tail;
struct list_head *vmemmap_pages;
+
/* Skip the TLB flush when we split the PMD */
#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0)
/* Skip the TLB flush when we remap the PTE */
@@ -140,14 +142,7 @@ static int vmemmap_pte_entry(pte_t *pte, unsigned long addr,
{
struct vmemmap_remap_walk *vmemmap_walk = walk->private;
- /*
- * The reuse_page is found 'first' in page table walking before
- * starting remapping.
- */
- if (!vmemmap_walk->reuse_page)
- vmemmap_walk->reuse_page = pte_page(ptep_get(pte));
- else
- vmemmap_walk->remap_pte(pte, addr, vmemmap_walk);
+ vmemmap_walk->remap_pte(pte, addr, vmemmap_walk);
vmemmap_walk->nr_walked++;
return 0;
@@ -207,18 +202,14 @@ static void free_vmemmap_page_list(struct list_head *list)
static void vmemmap_remap_pte(pte_t *pte, unsigned long addr,
struct vmemmap_remap_walk *walk)
{
- /*
- * Remap the tail pages as read-only to catch illegal write operation
- * to the tail pages.
- */
- pgprot_t pgprot = PAGE_KERNEL_RO;
struct page *page = pte_page(ptep_get(pte));
pte_t entry;
/* Remapping the head page requires r/w */
- if (unlikely(addr == walk->reuse_addr)) {
- pgprot = PAGE_KERNEL;
- list_del(&walk->reuse_page->lru);
+ if (unlikely(walk->nr_walked == 0 && walk->vmemmap_head)) {
+ VM_WARN_ON_ONCE(!PageHead((const struct page *)addr));
+
+ list_del(&walk->vmemmap_head->lru);
/*
* Makes sure that preceding stores to the page contents from
@@ -226,53 +217,47 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr,
* write.
*/
smp_wmb();
+
+ entry = mk_pte(walk->vmemmap_head, PAGE_KERNEL);
+ } else {
+ VM_WARN_ON_ONCE(!PageTail((const struct page *)addr));
+
+ /*
+ * Remap the tail pages as read-only to catch illegal write
+ * operation to the tail pages.
+ */
+ entry = mk_pte(walk->vmemmap_tail, PAGE_KERNEL_RO);
}
- entry = mk_pte(walk->reuse_page, pgprot);
list_add(&page->lru, walk->vmemmap_pages);
set_pte_at(&init_mm, addr, pte, entry);
}
-/*
- * How many struct page structs need to be reset. When we reuse the head
- * struct page, the special metadata (e.g. page->flags or page->mapping)
- * cannot copy to the tail struct page structs. The invalid value will be
- * checked in the free_tail_page_prepare(). In order to avoid the message
- * of "corrupted mapping in tail page". We need to reset at least 4 (one
- * head struct page struct and three tail struct page structs) struct page
- * structs.
- */
-#define NR_RESET_STRUCT_PAGE 4
-
-static inline void reset_struct_pages(struct page *start)
-{
- struct page *from = start + NR_RESET_STRUCT_PAGE;
-
- BUILD_BUG_ON(NR_RESET_STRUCT_PAGE * 2 > PAGE_SIZE / sizeof(struct page));
- memcpy(start, from, sizeof(*from) * NR_RESET_STRUCT_PAGE);
-}
-
static void vmemmap_restore_pte(pte_t *pte, unsigned long addr,
struct vmemmap_remap_walk *walk)
{
- pgprot_t pgprot = PAGE_KERNEL;
- struct page *page;
- void *to;
+ struct page *src = pte_page(ptep_get(pte)), *dst;
- BUG_ON(pte_page(ptep_get(pte)) != walk->reuse_page);
+ /*
+ * When rolling back vmemmap_remap_free(), keep the copied head page
+ * mapping and restore only PTEs currently pointing at the shared tail
+ * page.
+ */
+ if (walk->vmemmap_tail && walk->vmemmap_tail != src)
+ return;
- page = list_first_entry(walk->vmemmap_pages, struct page, lru);
- list_del(&page->lru);
- to = page_to_virt(page);
- copy_page(to, (void *)walk->reuse_addr);
- reset_struct_pages(to);
+ VM_WARN_ON_ONCE(PageHead((const struct page *)addr));
+
+ dst = list_first_entry(walk->vmemmap_pages, struct page, lru);
+ list_del(&dst->lru);
+ copy_page(page_to_virt(dst), page_to_virt(src));
/*
* Makes sure that preceding stores to the page contents become visible
* before the set_pte_at() write.
*/
smp_wmb();
- set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot));
+ set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL));
}
/**
@@ -282,33 +267,28 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr,
* to remap.
* @end: end address of the vmemmap virtual address range that we want to
* remap.
- * @reuse: reuse address.
- *
* Return: %0 on success, negative error code otherwise.
*/
-static int vmemmap_remap_split(unsigned long start, unsigned long end,
- unsigned long reuse)
+static int vmemmap_remap_split(unsigned long start, unsigned long end)
{
struct vmemmap_remap_walk walk = {
.remap_pte = NULL,
.flags = VMEMMAP_SPLIT_NO_TLB_FLUSH,
};
- /* See the comment in the vmemmap_remap_free(). */
- BUG_ON(start - reuse != PAGE_SIZE);
-
- return vmemmap_remap_range(reuse, end, &walk);
+ return vmemmap_remap_range(start, end, &walk);
}
/**
* vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end)
- * to the page which @reuse is mapped to, then free vmemmap
- * which the range are mapped to.
+ * to use @vmemmap_head/tail, then free vmemmap which
+ * the range are mapped to.
* @start: start address of the vmemmap virtual address range that we want
* to remap.
* @end: end address of the vmemmap virtual address range that we want to
* remap.
- * @reuse: reuse address.
+ * @vmemmap_head: the page to be installed as first in the vmemmap range
+ * @vmemmap_tail: the page to be installed as non-first in the vmemmap range
* @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers
* responsibility to free pages.
* @flags: modifications to vmemmap_remap_walk flags
@@ -316,69 +296,39 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end,
* Return: %0 on success, negative error code otherwise.
*/
static int vmemmap_remap_free(unsigned long start, unsigned long end,
- unsigned long reuse,
+ struct page *vmemmap_head,
+ struct page *vmemmap_tail,
struct list_head *vmemmap_pages,
unsigned long flags)
{
int ret;
struct vmemmap_remap_walk walk = {
.remap_pte = vmemmap_remap_pte,
- .reuse_addr = reuse,
+ .vmemmap_head = vmemmap_head,
+ .vmemmap_tail = vmemmap_tail,
.vmemmap_pages = vmemmap_pages,
.flags = flags,
};
- int nid = page_to_nid((struct page *)reuse);
- gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN;
- /*
- * Allocate a new head vmemmap page to avoid breaking a contiguous
- * block of struct page memory when freeing it back to page allocator
- * in free_vmemmap_page_list(). This will allow the likely contiguous
- * struct page backing memory to be kept contiguous and allowing for
- * more allocations of hugepages. Fallback to the currently
- * mapped head page in case should it fail to allocate.
- */
- walk.reuse_page = alloc_pages_node(nid, gfp_mask, 0);
- if (walk.reuse_page) {
- copy_page(page_to_virt(walk.reuse_page),
- (void *)walk.reuse_addr);
- list_add(&walk.reuse_page->lru, vmemmap_pages);
- memmap_pages_add(1);
- }
+ ret = vmemmap_remap_range(start, end, &walk);
+ if (!ret || !walk.nr_walked)
+ return ret;
+
+ end = start + walk.nr_walked * PAGE_SIZE;
/*
- * In order to make remapping routine most efficient for the huge pages,
- * the routine of vmemmap page table walking has the following rules
- * (see more details from the vmemmap_pte_range()):
- *
- * - The range [@start, @end) and the range [@reuse, @reuse + PAGE_SIZE)
- * should be continuous.
- * - The @reuse address is part of the range [@reuse, @end) that we are
- * walking which is passed to vmemmap_remap_range().
- * - The @reuse address is the first in the complete range.
- *
- * So we need to make sure that @start and @reuse meet the above rules.
+ * vmemmap_pages contains pages from the previous vmemmap_remap_range()
+ * call which failed. These are pages which were removed from
+ * the vmemmap. They will be restored in the following call.
*/
- BUG_ON(start - reuse != PAGE_SIZE);
+ walk = (struct vmemmap_remap_walk) {
+ .remap_pte = vmemmap_restore_pte,
+ .vmemmap_tail = vmemmap_tail,
+ .vmemmap_pages = vmemmap_pages,
+ .flags = 0,
+ };
- ret = vmemmap_remap_range(reuse, end, &walk);
- if (ret && walk.nr_walked) {
- end = reuse + walk.nr_walked * PAGE_SIZE;
- /*
- * vmemmap_pages contains pages from the previous
- * vmemmap_remap_range call which failed. These
- * are pages which were removed from the vmemmap.
- * They will be restored in the following call.
- */
- walk = (struct vmemmap_remap_walk) {
- .remap_pte = vmemmap_restore_pte,
- .reuse_addr = reuse,
- .vmemmap_pages = vmemmap_pages,
- .flags = 0,
- };
-
- vmemmap_remap_range(reuse, end, &walk);
- }
+ vmemmap_remap_range(start, end, &walk);
return ret;
}
@@ -415,29 +365,24 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end,
* to remap.
* @end: end address of the vmemmap virtual address range that we want to
* remap.
- * @reuse: reuse address.
* @flags: modifications to vmemmap_remap_walk flags
*
* Return: %0 on success, negative error code otherwise.
*/
static int vmemmap_remap_alloc(unsigned long start, unsigned long end,
- unsigned long reuse, unsigned long flags)
+ unsigned long flags)
{
LIST_HEAD(vmemmap_pages);
struct vmemmap_remap_walk walk = {
.remap_pte = vmemmap_restore_pte,
- .reuse_addr = reuse,
.vmemmap_pages = &vmemmap_pages,
.flags = flags,
};
- /* See the comment in the vmemmap_remap_free(). */
- BUG_ON(start - reuse != PAGE_SIZE);
-
if (alloc_vmemmap_page_list(start, end, &vmemmap_pages))
return -ENOMEM;
- return vmemmap_remap_range(reuse, end, &walk);
+ return vmemmap_remap_range(start, end, &walk);
}
DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key);
@@ -450,8 +395,7 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h,
struct folio *folio, unsigned long flags)
{
int ret;
- unsigned long vmemmap_start = (unsigned long)&folio->page, vmemmap_end;
- unsigned long vmemmap_reuse;
+ unsigned long vmemmap_start, vmemmap_end;
VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(folio), folio);
VM_WARN_ON_ONCE_FOLIO(folio_ref_count(folio), folio);
@@ -462,18 +406,18 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h,
if (flags & VMEMMAP_SYNCHRONIZE_RCU)
synchronize_rcu();
+ vmemmap_start = (unsigned long)&folio->page;
vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
- vmemmap_reuse = vmemmap_start;
+
vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;
/*
* The pages which the vmemmap virtual address range [@vmemmap_start,
- * @vmemmap_end) are mapped to are freed to the buddy allocator, and
- * the range is mapped to the page which @vmemmap_reuse is mapped to.
+ * @vmemmap_end) are mapped to are freed to the buddy allocator.
* When a HugeTLB page is freed to the buddy allocator, previously
* discarded vmemmap pages must be allocated and remapping.
*/
- ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags);
+ ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, flags);
if (!ret) {
folio_clear_hugetlb_vmemmap_optimized(folio);
static_branch_dec(&hugetlb_optimize_vmemmap_key);
@@ -561,9 +505,9 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
struct list_head *vmemmap_pages,
unsigned long flags)
{
- int ret = 0;
- unsigned long vmemmap_start = (unsigned long)&folio->page, vmemmap_end;
- unsigned long vmemmap_reuse;
+ unsigned long vmemmap_start, vmemmap_end;
+ struct page *vmemmap_head, *vmemmap_tail;
+ int nid, ret = 0;
VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(folio), folio);
VM_WARN_ON_ONCE_FOLIO(folio_ref_count(folio), folio);
@@ -588,18 +532,30 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
*/
folio_set_hugetlb_vmemmap_optimized(folio);
+ nid = folio_nid(folio);
+ vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);
+ if (!vmemmap_head) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ copy_page(page_to_virt(vmemmap_head), folio);
+ list_add(&vmemmap_head->lru, vmemmap_pages);
+ memmap_pages_add(1);
+
+ vmemmap_tail = vmemmap_head;
+ vmemmap_start = (unsigned long)&folio->page;
vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
- vmemmap_reuse = vmemmap_start;
- vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;
/*
- * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end)
- * to the page which @vmemmap_reuse is mapped to. Add pages previously
- * mapping the range to vmemmap_pages list so that they can be freed by
- * the caller.
+ * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end).
+ * Add pages previously mapping the range to vmemmap_pages list so that
+ * they can be freed by the caller.
*/
- ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse,
+ ret = vmemmap_remap_free(vmemmap_start, vmemmap_end,
+ vmemmap_head, vmemmap_tail,
vmemmap_pages, flags);
+out:
if (ret) {
static_branch_dec(&hugetlb_optimize_vmemmap_key);
folio_clear_hugetlb_vmemmap_optimized(folio);
@@ -628,21 +584,19 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
static int hugetlb_vmemmap_split_folio(const struct hstate *h, struct folio *folio)
{
- unsigned long vmemmap_start = (unsigned long)&folio->page, vmemmap_end;
- unsigned long vmemmap_reuse;
+ unsigned long vmemmap_start, vmemmap_end;
if (!vmemmap_should_optimize_folio(h, folio))
return 0;
+ vmemmap_start = (unsigned long)&folio->page;
vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h);
- vmemmap_reuse = vmemmap_start;
- vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE;
/*
* Split PMDs on the vmemmap virtual address range [@vmemmap_start,
* @vmemmap_end]
*/
- return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse);
+ return vmemmap_remap_split(vmemmap_start, vmemmap_end);
}
void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list)
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index b18cfa5e2cb7..53240f7bf4e8 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -292,34 +292,117 @@ void __init __weak kasan_populate_early_vm_area_shadow(void *start,
{
}
+struct vmalloc_populate_data {
+ unsigned long start;
+ struct page **pages;
+};
+
static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
- void *unused)
+ void *_data)
{
- unsigned long page;
+ struct vmalloc_populate_data *data = _data;
+ struct page *page;
pte_t pte;
+ int index;
if (likely(!pte_none(ptep_get(ptep))))
return 0;
- page = __get_free_page(GFP_KERNEL);
- if (!page)
- return -ENOMEM;
-
- __memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
- pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
+ index = PFN_DOWN(addr - data->start);
+ page = data->pages[index];
+ __memset(page_to_virt(page), KASAN_VMALLOC_INVALID, PAGE_SIZE);
+ pte = pfn_pte(page_to_pfn(page), PAGE_KERNEL);
spin_lock(&init_mm.page_table_lock);
if (likely(pte_none(ptep_get(ptep)))) {
set_pte_at(&init_mm, addr, ptep, pte);
- page = 0;
+ data->pages[index] = NULL;
}
spin_unlock(&init_mm.page_table_lock);
- if (page)
- free_page(page);
+
+ return 0;
+}
+
+static void ___free_pages_bulk(struct page **pages, int nr_pages)
+{
+ int i;
+
+ for (i = 0; i < nr_pages; i++) {
+ if (pages[i]) {
+ __free_pages(pages[i], 0);
+ pages[i] = NULL;
+ }
+ }
+}
+
+static int ___alloc_pages_bulk(struct page **pages, int nr_pages, gfp_t gfp_mask)
+{
+ unsigned long nr_populated, nr_total = nr_pages;
+ struct page **page_array = pages;
+
+ while (nr_pages) {
+ nr_populated = alloc_pages_bulk_array(gfp_mask, nr_pages, pages);
+ if (!nr_populated) {
+ ___free_pages_bulk(page_array, nr_total - nr_pages);
+ return -ENOMEM;
+ }
+ pages += nr_populated;
+ nr_pages -= nr_populated;
+ }
+
return 0;
}
-int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
+static int __kasan_populate_vmalloc(unsigned long start, unsigned long end, gfp_t gfp_mask)
+{
+ unsigned long nr_pages, nr_total = PFN_UP(end - start);
+ struct vmalloc_populate_data data;
+ unsigned int flags;
+ int ret = 0;
+
+ data.pages = (struct page **)__get_free_page(gfp_mask | __GFP_ZERO);
+ if (!data.pages)
+ return -ENOMEM;
+
+ while (nr_total) {
+ nr_pages = min(nr_total, PAGE_SIZE / sizeof(data.pages[0]));
+ ret = ___alloc_pages_bulk(data.pages, nr_pages, gfp_mask);
+ if (ret)
+ break;
+
+ data.start = start;
+
+ /*
+ * page tables allocations ignore external gfp mask, enforce it
+ * by the scope API
+ */
+ if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)
+ flags = memalloc_nofs_save();
+ else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == 0)
+ flags = memalloc_noio_save();
+
+ ret = apply_to_page_range(&init_mm, start, nr_pages * PAGE_SIZE,
+ kasan_populate_vmalloc_pte, &data);
+
+ if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)
+ memalloc_nofs_restore(flags);
+ else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == 0)
+ memalloc_noio_restore(flags);
+
+ ___free_pages_bulk(data.pages, nr_pages);
+ if (ret)
+ break;
+
+ start += nr_pages * PAGE_SIZE;
+ nr_total -= nr_pages;
+ }
+
+ free_page((unsigned long)data.pages);
+
+ return ret;
+}
+
+int kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask)
{
unsigned long shadow_start, shadow_end;
int ret;
@@ -348,9 +431,7 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
shadow_start = PAGE_ALIGN_DOWN(shadow_start);
shadow_end = PAGE_ALIGN(shadow_end);
- ret = apply_to_page_range(&init_mm, shadow_start,
- shadow_end - shadow_start,
- kasan_populate_vmalloc_pte, NULL);
+ ret = __kasan_populate_vmalloc(shadow_start, shadow_end, gfp_mask);
if (ret)
return ret;
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 50a1fd9087fa..8b0635ed1d20 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -569,7 +569,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val)
if (!val)
return;
- cgroup_rstat_updated(memcg->css.cgroup, cpu);
+ __cgroup_rstat_updated(memcg->css.cgroup, cpu);
statc = this_cpu_ptr(memcg->vmstats_percpu);
for (; statc; statc = statc->parent) {
stats_updates = READ_ONCE(statc->stats_updates) + abs(val);
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 65f997ddd740..77f5f62628b7 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -29,6 +29,7 @@
#include <linux/cma.h>
#include <linux/crash_dump.h>
#include <linux/execmem.h>
+#include <linux/hugetlb.h>
#include <linux/vmstat.h>
#include "internal.h"
#include "slab.h"
@@ -2320,6 +2321,7 @@ void __init page_alloc_init_late(void)
/* Reinit limits that are based on free pages after the kernel is up */
files_maxfiles_init();
#endif
+ hugetlb_bootmem_struct_page_init();
/* Accounting of total+free memory is stable at this point. */
mem_init_print_info();
diff --git a/mm/mremap.c b/mm/mremap.c
index 9685ea816fde..8b507e228a27 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -281,15 +281,6 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
if (WARN_ON_ONCE(!pmd_none(*new_pmd)))
return false;
- /* If this pmd belongs to a uffd vma with remap events disabled, we need
- * to ensure that the uffd-wp state is cleared from all pgtables. This
- * means recursing into lower page tables in move_page_tables(), and we
- * can reuse the existing code if we simply treat the entry as "not
- * moved".
- */
- if (vma_has_uffd_without_event_remap(vma))
- return false;
-
/*
* We don't have to worry about the ordering of src and dst
* ptlocks because exclusive mmap_lock prevents deadlock.
@@ -345,15 +336,6 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr,
if (WARN_ON_ONCE(!pud_none(*new_pud)))
return false;
- /* If this pud belongs to a uffd vma with remap events disabled, we need
- * to ensure that the uffd-wp state is cleared from all pgtables. This
- * means recursing into lower page tables in move_page_tables(), and we
- * can reuse the existing code if we simply treat the entry as "not
- * moved".
- */
- if (vma_has_uffd_without_event_remap(vma))
- return false;
-
/*
* We don't have to worry about the ordering of src and dst
* ptlocks because exclusive mmap_lock prevents deadlock.
@@ -587,6 +569,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
struct mmu_notifier_range range;
pmd_t *old_pmd, *new_pmd;
pud_t *old_pud, *new_pud;
+ bool uffd_supports_move;
if (!len)
return 0;
@@ -597,6 +580,22 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
return move_hugetlb_page_tables(vma, new_vma, old_addr,
new_addr, len);
+ /*
+ * If we are moving a VMA that has uffd-wp registered but with
+ * remap events disabled (new VMA will not be registered with uffd), we
+ * need to ensure that the uffd-wp state is cleared from all pgtables.
+ * This means recursing into lower page tables in move_page_tables().
+ *
+ * We might get called with VMAs reversed when recovering from a
+ * failed page table move. In that case, the
+ * "old"-but-actually-"originally new" VMA during recovery will not have
+ * a uffd context. Recursing into lower page tables during the original
+ * move but not during the recovery move will cause trouble, because we
+ * run into already-existing page tables. So check both VMAs.
+ */
+ uffd_supports_move = !vma_has_uffd_without_event_remap(vma) &&
+ !vma_has_uffd_without_event_remap(new_vma);
+
/*
* If possible, realign addresses to PMD boundary for faster copy.
* Only realign if the mremap copying hits a PMD boundary.
@@ -631,7 +630,8 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
/* We ignore and continue on error? */
continue;
}
- } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) && extent == PUD_SIZE) {
+ } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PUD) &&
+ extent == PUD_SIZE && uffd_supports_move) {
if (move_pgt_entry(NORMAL_PUD, vma, old_addr, new_addr,
old_pud, new_pud, true))
@@ -654,7 +654,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
continue;
split_huge_pmd(vma, old_pmd, old_addr);
} else if (IS_ENABLED(CONFIG_HAVE_MOVE_PMD) &&
- extent == PMD_SIZE) {
+ extent == PMD_SIZE && uffd_supports_move) {
/*
* If the extent is PMD-sized, try to speed the move by
* moving at the PMD level if possible.
@@ -702,6 +702,8 @@ static unsigned long move_vma(struct vm_area_struct *vma,
int err = 0;
bool need_rmap_locks;
struct vma_iterator vmi;
+ const bool split_before = vma->vm_start != old_addr;
+ const bool split_after = vma->vm_end != old_addr + old_len;
/*
* We'd prefer to avoid failure later on in do_munmap:
@@ -714,9 +716,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,
to_account = new_len;
if (vma->vm_ops && vma->vm_ops->may_split) {
- if (vma->vm_start != old_addr)
+ if (split_before)
err = vma->vm_ops->may_split(vma, old_addr);
- if (!err && vma->vm_end != old_addr + old_len)
+ if (!err && split_after)
err = vma->vm_ops->may_split(vma, old_addr + old_len);
if (err)
return err;
@@ -734,6 +736,21 @@ static unsigned long move_vma(struct vm_area_struct *vma,
if (err)
return err;
+ /*
+ * To account mlock()'d pages correctly in the MREMAP_DONTUNMAP
+ * case perform any split ahead of time for an mlock()'d VMA.
+ */
+ if ((flags & MREMAP_DONTUNMAP) && (vma->vm_flags & VM_LOCKED)) {
+ vma_iter_init(&vmi, mm, old_addr);
+
+ if (split_before)
+ err = split_vma(&vmi, vma, old_addr, 1);
+ if (!err && split_after)
+ err = split_vma(&vmi, vma, old_addr + old_len, 0);
+ if (err)
+ return err;
+ }
+
if (vm_flags & VM_ACCOUNT) {
if (security_vm_enough_memory_mm(mm, to_account >> PAGE_SHIFT))
return -ENOMEM;
@@ -742,7 +759,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
vma_start_write(vma);
new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT);
new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff,
- &need_rmap_locks);
+ &need_rmap_locks, flags & MREMAP_DONTUNMAP);
if (!new_vma) {
if (vm_flags & VM_ACCOUNT)
vm_unacct_memory(to_account >> PAGE_SHIFT);
@@ -803,6 +820,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,
untrack_pfn_clear(vma);
if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) {
+ /* Self-merge is disallowed. */
+ VM_WARN_ON_ONCE(new_vma == vma);
+
/* We always clear VM_LOCKED[ONFAULT] on the old vma */
vm_flags_clear(vma, VM_LOCKED_MASK);
@@ -810,7 +830,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
* anon_vma links of the old vma is no longer needed after its page
* table has been moved.
*/
- if (new_vma != vma && vma->vm_start == old_addr &&
+ if (vma->vm_start == old_addr &&
vma->vm_end == (old_addr + old_len)) {
const pgoff_t pgoff_unfaulted = vma->vm_start >> PAGE_SHIFT;
diff --git a/mm/page_owner.c b/mm/page_owner.c
index f92cec789eee..5d1c946c0008 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -501,6 +501,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
{
#ifdef CONFIG_MEMCG
unsigned long memcg_data;
+ struct obj_cgroup *objcg;
struct mem_cgroup *memcg;
bool online;
char name[80];
@@ -510,11 +511,14 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
if (!memcg_data)
goto out_unlock;
- if (memcg_data & MEMCG_DATA_OBJEXTS)
+ if (memcg_data & MEMCG_DATA_OBJEXTS) {
ret += scnprintf(kbuf + ret, count - ret,
"Slab cache page\n");
+ goto out_unlock;
+ }
- memcg = page_memcg_check(page);
+ objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK);
+ memcg = objcg ? obj_cgroup_memcg(objcg) : NULL;
if (!memcg)
goto out_unlock;
@@ -522,7 +526,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret,
cgroup_name(memcg->css.cgroup, name, sizeof(name));
ret += scnprintf(kbuf + ret, count - ret,
"Charged %sto %smemcg %s\n",
- PageMemcgKmem(page) ? "(via objcg) " : "",
+ (memcg_data & MEMCG_DATA_KMEM) ? "(via objcg) " : "",
online ? "" : "offline ",
name);
out_unlock:
diff --git a/mm/page_table_check.c b/mm/page_table_check.c
index ee52927e9e66..2efa910eef23 100644
--- a/mm/page_table_check.c
+++ b/mm/page_table_check.c
@@ -165,18 +165,29 @@ void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte)
if (&init_mm == mm)
return;
- if (pte_user_accessible_page(pte)) {
+ if (pte_user_accessible_page(pte) && !pte_special(pte)) {
page_table_check_clear(pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT);
}
}
EXPORT_SYMBOL(__page_table_check_pte_clear);
+static inline bool page_table_check_huge_zero_pmd(pmd_t pmd)
+{
+ unsigned long pfn = pmd_pfn(pmd);
+
+ if (!pfn_valid(pfn))
+ return false;
+
+ return is_huge_zero_folio(page_folio(pfn_to_page(pfn)));
+}
+
void __page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd)
{
if (&init_mm == mm)
return;
- if (pmd_user_accessible_page(pmd)) {
+ if (pmd_user_accessible_page(pmd) &&
+ !page_table_check_huge_zero_pmd(pmd)) {
page_table_check_clear(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT);
}
}
@@ -221,7 +232,7 @@ void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte,
for (i = 0; i < nr; i++)
__page_table_check_pte_clear(mm, ptep_get(ptep + i));
- if (pte_user_accessible_page(pte))
+ if (pte_user_accessible_page(pte) && !pte_special(pte))
page_table_check_set(pte_pfn(pte), nr, pte_write(pte));
}
EXPORT_SYMBOL(__page_table_check_ptes_set);
@@ -246,7 +257,8 @@ void __page_table_check_pmd_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd)
page_table_check_pmd_flags(pmd);
__page_table_check_pmd_clear(mm, *pmdp);
- if (pmd_user_accessible_page(pmd)) {
+ if (pmd_user_accessible_page(pmd) &&
+ !page_table_check_huge_zero_pmd(pmd)) {
page_table_check_set(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT,
pmd_write(pmd));
}
diff --git a/mm/vma.c b/mm/vma.c
index 3df4cf796c7f..66b0472b7d8a 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -236,10 +236,13 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi,
if (vp->file) {
i_mmap_unlock_write(vp->mapping);
- uprobe_mmap(vp->vma);
- if (vp->adj_next)
- uprobe_mmap(vp->adj_next);
+ if (!vp->skip_vma_uprobe) {
+ uprobe_mmap(vp->vma);
+
+ if (vp->adj_next)
+ uprobe_mmap(vp->adj_next);
+ }
}
if (vp->remove) {
@@ -457,8 +460,8 @@ static int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
* Split a vma into two pieces at address 'addr', a new vma is allocated
* either for the first part or the tail.
*/
-static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
- unsigned long addr, int new_below)
+int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
+ unsigned long addr, int new_below)
{
if (vma->vm_mm->map_count >= sysctl_max_map_count)
return -ENOMEM;
@@ -592,6 +595,7 @@ static int commit_merge(struct vma_merge_struct *vmg,
struct vma_prepare vp;
init_multi_vma_prep(&vp, vmg->vma, adjust, remove, remove2);
+ vp.skip_vma_uprobe = vmg->skip_vma_uprobe;
VM_WARN_ON(vp.anon_vma && adjust && adjust->anon_vma &&
vp.anon_vma != adjust->anon_vma);
@@ -1641,7 +1645,7 @@ int vma_link(struct mm_struct *mm, struct vm_area_struct *vma)
*/
struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
unsigned long addr, unsigned long len, pgoff_t pgoff,
- bool *need_rmap_locks)
+ bool *need_rmap_locks, bool keep_source)
{
struct vm_area_struct *vma = *vmap;
unsigned long vma_start = vma->vm_start;
@@ -1660,6 +1664,14 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
faulted_in_anon_vma = false;
}
+ /*
+ * If the VMA we are copying might contain a uprobe PTE, ensure
+ * that we do not establish one upon merge. Otherwise, when mremap()
+ * moves page tables, it will orphan the newly created PTE.
+ */
+ if (vma->vm_file)
+ vmg.skip_vma_uprobe = true;
+
new_vma = find_vma_prev(mm, addr, &vmg.prev);
if (new_vma && new_vma->vm_start < addr + len)
return NULL; /* should never get here */
@@ -1667,6 +1679,21 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
vmg.vma = NULL; /* New VMA range. */
vmg.pgoff = pgoff;
vmg.next = vma_iter_next_rewind(&vmi, NULL);
+
+ /*
+ * If the original VMA is kept (MREMAP_DONTUNMAP), the source and
+ * destination VMA must be treated distinctly.
+ *
+ * A merge violates this, so in this case disallow a self-merge.
+ */
+ if (!faulted_in_anon_vma && keep_source) {
+ if (vmg.prev == vma)
+ vmg.prev = NULL;
+ if (vmg.next == vma)
+ vmg.next = NULL;
+ faulted_in_anon_vma = true;
+ }
+
new_vma = vma_merge_new_range(&vmg);
if (new_vma) {
diff --git a/mm/vma.h b/mm/vma.h
index 729fe3741e89..f7efbe41c44c 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -19,6 +19,8 @@ struct vma_prepare {
struct vm_area_struct *insert;
struct vm_area_struct *remove;
struct vm_area_struct *remove2;
+
+ bool skip_vma_uprobe :1;
};
struct unlink_vma_file_batch {
@@ -93,6 +95,11 @@ struct vma_merge_struct {
* execute the merge, returning NULL.
*/
bool give_up_on_oom :1;
+
+ /*
+ * If set, skip uprobe_mmap upon merged vma.
+ */
+ bool skip_vma_uprobe :1;
};
static inline bool vmg_nomem(struct vma_merge_struct *vmg)
@@ -333,7 +340,7 @@ int vma_link(struct mm_struct *mm, struct vm_area_struct *vma);
struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
unsigned long addr, unsigned long len, pgoff_t pgoff,
- bool *need_rmap_locks);
+ bool *need_rmap_locks, bool keep_source);
struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *vma);
@@ -343,6 +350,11 @@ bool vma_wants_writenotify(struct vm_area_struct *vma, pgprot_t vm_page_prot);
int mm_take_all_locks(struct mm_struct *mm);
void mm_drop_all_locks(struct mm_struct *mm);
+#ifdef CONFIG_MMU
+int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,
+ unsigned long addr, int new_below);
+#endif
+
static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma)
{
/*
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index e0066c100b79..b11552177f2d 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -2025,6 +2025,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
if (unlikely(!vmap_initialized))
return ERR_PTR(-EBUSY);
+ /* Only reclaim behaviour flags are relevant. */
+ gfp_mask = gfp_mask & GFP_RECLAIM_MASK;
might_sleep();
/*
@@ -2037,8 +2039,6 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
*/
va = node_alloc(size, align, vstart, vend, &addr, &vn_id);
if (!va) {
- gfp_mask = gfp_mask & GFP_RECLAIM_MASK;
-
va = kmem_cache_alloc_node(vmap_area_cachep, gfp_mask, node);
if (unlikely(!va))
return ERR_PTR(-ENOMEM);
@@ -2088,7 +2088,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
BUG_ON(va->va_start < vstart);
BUG_ON(va->va_end > vend);
- ret = kasan_populate_vmalloc(addr, size);
+ ret = kasan_populate_vmalloc(addr, size, gfp_mask);
if (ret) {
free_vmap_area(va);
return ERR_PTR(ret);
@@ -4844,7 +4844,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
/* populate the kasan shadow space */
for (area = 0; area < nr_vms; area++) {
- if (kasan_populate_vmalloc(vas[area]->va_start, sizes[area]))
+ if (kasan_populate_vmalloc(vas[area]->va_start, sizes[area], GFP_KERNEL))
goto err_free_shadow;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 4c13524afaaa..f9b5fbd072db 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2625,8 +2625,6 @@ static bool should_clear_pmd_young(void)
* shorthand helpers
******************************************************************************/
-#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset))
-
#define DEFINE_MAX_SEQ(lruvec) \
unsigned long max_seq = READ_ONCE((lruvec)->lrugen.max_seq)
@@ -4157,8 +4155,10 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw)
old_gen = folio_lru_gen(folio);
if (old_gen < 0)
folio_set_referenced(folio);
- else if (old_gen != new_gen)
+ else if (old_gen != new_gen) {
+ folio_clear_lru_refs(folio);
folio_activate(folio);
+ }
}
arch_leave_lazy_mmu_mode();
@@ -4312,8 +4312,6 @@ void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid)
static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_control *sc,
int tier_idx)
{
- bool success;
- bool dirty, writeback;
int gen = folio_lru_gen(folio);
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
@@ -4324,15 +4322,9 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
- /* unevictable */
- if (!folio_evictable(folio)) {
- success = lru_gen_del_folio(lruvec, folio, true);
- VM_WARN_ON_ONCE_FOLIO(!success, folio);
- folio_set_unevictable(folio);
- lruvec_add_folio(lruvec, folio);
- __count_vm_events(UNEVICTABLE_PGCULLED, delta);
- return true;
- }
+ /* unevictable: let it through and the generic path will cull it */
+ if (!folio_evictable(folio))
+ return false;
/* promoted */
if (gen != lru_gen_from_seq(lrugen->min_seq[type])) {
@@ -4359,22 +4351,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
return true;
}
- dirty = folio_test_dirty(folio);
- writeback = folio_test_writeback(folio);
- if (type == LRU_GEN_FILE && dirty) {
- sc->nr.file_taken += delta;
- if (!writeback)
- sc->nr.unqueued_dirty += delta;
- }
-
- /* waiting for writeback */
- if (folio_test_locked(folio) || writeback ||
- (type == LRU_GEN_FILE && dirty)) {
- gen = folio_inc_gen(lruvec, folio, true);
- list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
- return true;
- }
-
return false;
}
@@ -4384,7 +4360,7 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
/* swap constrained */
if (!(sc->gfp_mask & __GFP_IO) &&
- (folio_test_dirty(folio) ||
+ ((folio_test_dirty(folio) && !folio_is_file_lru(folio)) ||
(folio_test_anon(folio) && !folio_test_swapcache(folio))))
return false;
@@ -4400,10 +4376,9 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
/* see the comment on MAX_NR_TIERS */
if (!folio_test_referenced(folio))
- set_mask_bits(&folio->flags, LRU_REFS_MASK | LRU_REFS_FLAGS, 0);
+ folio_clear_lru_refs(folio);
/* for shrink_folio_list() */
- folio_clear_reclaim(folio);
folio_clear_referenced(folio);
success = lru_gen_del_folio(lruvec, folio, true);
@@ -4627,11 +4602,9 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap
type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON);
list_for_each_entry_safe_reverse(folio, next, &list, lru) {
- if (!folio_evictable(folio)) {
- list_del(&folio->lru);
- folio_putback_lru(folio);
+ /* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */
+ if (!folio_evictable(folio))
continue;
- }
if (folio_test_reclaim(folio) &&
(folio_test_dirty(folio) || folio_test_writeback(folio))) {
diff --git a/net/ax25/af_ax25.c b/net/ax25/af_ax25.c
index 9f3b8b682adb..3ee7dba34310 100644
--- a/net/ax25/af_ax25.c
+++ b/net/ax25/af_ax25.c
@@ -1270,28 +1270,18 @@ static int __must_check ax25_connect(struct socket *sock,
}
}
- /*
- * Must bind first - autobinding in this may or may not work. If
- * the socket is already bound, check to see if the device has
- * been filled in, error if it hasn't.
- */
+ /* Must bind first - autobinding does not work. */
if (sock_flag(sk, SOCK_ZAPPED)) {
- /* check if we can remove this feature. It is broken. */
- printk(KERN_WARNING "ax25_connect(): %s uses autobind, please contact jreuter@yaina.de\n",
- current->comm);
- if ((err = ax25_rt_autobind(ax25, &fsa->fsa_ax25.sax25_call)) < 0) {
- kfree(digi);
- goto out_release;
- }
+ kfree(digi);
+ err = -EINVAL;
+ goto out_release;
+ }
- ax25_fillin_cb(ax25, ax25->ax25_dev);
- ax25_cb_add(ax25);
- } else {
- if (ax25->ax25_dev == NULL) {
- kfree(digi);
- err = -EHOSTUNREACH;
- goto out_release;
- }
+ /* Check to see if the device has been filled in, error if it hasn't. */
+ if (ax25->ax25_dev == NULL) {
+ kfree(digi);
+ err = -EHOSTUNREACH;
+ goto out_release;
}
if (sk->sk_type == SOCK_SEQPACKET &&
diff --git a/net/ax25/ax25_route.c b/net/ax25/ax25_route.c
index 69de75db0c9c..10577434f40b 100644
--- a/net/ax25/ax25_route.c
+++ b/net/ax25/ax25_route.c
@@ -373,80 +373,6 @@ ax25_route *ax25_get_route(ax25_address *addr, struct net_device *dev)
return ax25_rt;
}
-/*
- * Adjust path: If you specify a default route and want to connect
- * a target on the digipeater path but w/o having a special route
- * set before, the path has to be truncated from your target on.
- */
-static inline void ax25_adjust_path(ax25_address *addr, ax25_digi *digipeat)
-{
- int k;
-
- for (k = 0; k < digipeat->ndigi; k++) {
- if (ax25cmp(addr, &digipeat->calls[k]) == 0)
- break;
- }
-
- digipeat->ndigi = k;
-}
-
-
-/*
- * Find which interface to use.
- */
-int ax25_rt_autobind(ax25_cb *ax25, ax25_address *addr)
-{
- ax25_uid_assoc *user;
- ax25_route *ax25_rt;
- int err = 0;
-
- ax25_route_lock_use();
- ax25_rt = ax25_get_route(addr, NULL);
- if (!ax25_rt) {
- ax25_route_lock_unuse();
- return -EHOSTUNREACH;
- }
- rcu_read_lock();
- if ((ax25->ax25_dev = ax25_dev_ax25dev(ax25_rt->dev)) == NULL) {
- err = -EHOSTUNREACH;
- goto put;
- }
-
- user = ax25_findbyuid(current_euid());
- if (user) {
- ax25->source_addr = user->call;
- ax25_uid_put(user);
- } else {
- if (ax25_uid_policy && !capable(CAP_NET_BIND_SERVICE)) {
- err = -EPERM;
- goto put;
- }
- ax25->source_addr = *(ax25_address *)ax25->ax25_dev->dev->dev_addr;
- }
-
- if (ax25_rt->digipeat != NULL) {
- ax25->digipeat = kmemdup(ax25_rt->digipeat, sizeof(ax25_digi),
- GFP_ATOMIC);
- if (ax25->digipeat == NULL) {
- err = -ENOMEM;
- goto put;
- }
- ax25_adjust_path(addr, ax25->digipeat);
- }
-
- if (ax25->sk != NULL) {
- local_bh_disable();
- bh_lock_sock(ax25->sk);
- sock_reset_flag(ax25->sk, SOCK_ZAPPED);
- bh_unlock_sock(ax25->sk);
- local_bh_enable();
- }
-
-put:
- rcu_read_unlock();
- ax25_route_lock_unuse();
- return err;
-}
struct sk_buff *ax25_rt_build_path(struct sk_buff *skb, ax25_address *src,
ax25_address *dest, ax25_digi *digi)
diff --git a/net/batman-adv/soft-interface.c b/net/batman-adv/soft-interface.c
index 47715a612320..72fbf415dc2b 100644
--- a/net/batman-adv/soft-interface.c
+++ b/net/batman-adv/soft-interface.c
@@ -1031,7 +1031,7 @@ static void batadv_softif_init_early(struct net_device *dev)
dev->features |= NETIF_F_HW_VLAN_CTAG_FILTER;
dev->priv_flags |= IFF_NO_QUEUE;
dev->lltx = true;
- dev->netns_local = true;
+ dev->netns_immutable = true;
/* can't call min_mtu, because the needed variables
* have not been initialized yet
diff --git a/net/bluetooth/6lowpan.c b/net/bluetooth/6lowpan.c
index 89de21ec62a1..1cb37fbb668f 100644
--- a/net/bluetooth/6lowpan.c
+++ b/net/bluetooth/6lowpan.c
@@ -1065,23 +1065,15 @@ static void disconnect_all_peers(void)
} while (nchans);
}
-struct set_enable {
- struct work_struct work;
- bool flag;
-};
-
-static void do_enable_set(struct work_struct *work)
+static void do_enable_set(bool flag)
{
- struct set_enable *set_enable = container_of(work,
- struct set_enable, work);
-
- if (!set_enable->flag || enable_6lowpan != set_enable->flag)
+ if (!flag || enable_6lowpan != flag)
/* Disconnect existing connections if 6lowpan is
* disabled
*/
disconnect_all_peers();
- enable_6lowpan = set_enable->flag;
+ enable_6lowpan = flag;
mutex_lock(&set_lock);
if (listen_chan) {
@@ -1091,22 +1083,11 @@ static void do_enable_set(struct work_struct *work)
listen_chan = bt_6lowpan_listen();
mutex_unlock(&set_lock);
-
- kfree(set_enable);
}
static int lowpan_enable_set(void *data, u64 val)
{
- struct set_enable *set_enable;
-
- set_enable = kzalloc(sizeof(*set_enable), GFP_KERNEL);
- if (!set_enable)
- return -ENOMEM;
-
- set_enable->flag = !!val;
- INIT_WORK(&set_enable->work, do_enable_set);
-
- schedule_work(&set_enable->work);
+ do_enable_set(!!val);
return 0;
}
diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c
index 48d28c79e299..602b5be7eccc 100644
--- a/net/bluetooth/hci_conn.c
+++ b/net/bluetooth/hci_conn.c
@@ -1052,6 +1052,8 @@ static struct hci_conn *__hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t
INIT_DELAYED_WORK(&conn->idle_work, hci_conn_idle);
INIT_DELAYED_WORK(&conn->le_conn_timeout, le_conn_timeout);
+ spin_lock_init(&conn->proto_lock);
+
atomic_set(&conn->refcnt, 0);
hci_dev_hold(hdev);
diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c
index 95900c232563..ff3ccb33da0c 100644
--- a/net/bluetooth/hci_core.c
+++ b/net/bluetooth/hci_core.c
@@ -4155,9 +4155,11 @@ static int hci_send_cmd_sync(struct hci_dev *hdev, struct sk_buff *skb)
bt_dev_dbg(hdev, "skb %p", skb);
+ hci_dev_lock(hdev);
kfree_skb(hdev->sent_cmd);
hdev->sent_cmd = skb_clone(skb, GFP_KERNEL);
+ hci_dev_unlock(hdev);
if (!hdev->sent_cmd) {
skb_queue_head(&hdev->cmd_q, skb);
queue_work(hdev->workqueue, &hdev->cmd_work);
@@ -4178,8 +4180,10 @@ static int hci_send_cmd_sync(struct hci_dev *hdev, struct sk_buff *skb)
if (READ_ONCE(hdev->req_status) == HCI_REQ_PEND &&
!hci_dev_test_and_set_flag(hdev, HCI_CMD_PENDING)) {
+ hci_dev_lock(hdev);
kfree_skb(hdev->req_skb);
hdev->req_skb = skb_get(hdev->sent_cmd);
+ hci_dev_unlock(hdev);
}
return err;
diff --git a/net/bluetooth/hci_event.c b/net/bluetooth/hci_event.c
index 53732f8ddf63..cbb1c3ee0752 100644
--- a/net/bluetooth/hci_event.c
+++ b/net/bluetooth/hci_event.c
@@ -80,6 +80,10 @@ static void *hci_le_ev_skb_pull(struct hci_dev *hdev, struct sk_buff *skb,
return data;
}
+static void hci_store_wake_reason(struct hci_dev *hdev,
+ const bdaddr_t *bdaddr, u8 addr_type)
+ __must_hold(&hdev->lock);
+
static u8 hci_cc_inquiry_cancel(struct hci_dev *hdev, void *data,
struct sk_buff *skb)
{
@@ -3060,6 +3064,7 @@ static void hci_conn_complete_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
/* Check for existing connection:
*
@@ -3234,6 +3239,10 @@ static void hci_conn_request_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "bdaddr %pMR type 0x%x", &ev->bdaddr, ev->link_type);
+ hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
+ hci_dev_unlock(hdev);
+
/* Reject incoming connection from device with same BD ADDR against
* CVE-2020-26555
*/
@@ -4955,6 +4964,7 @@ static void hci_sync_conn_complete_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
conn = hci_conn_hash_lookup_ba(hdev, ev->link_type, &ev->bdaddr);
if (!conn) {
@@ -5656,6 +5666,7 @@ static void le_conn_complete_evt(struct hci_dev *hdev, u8 status,
u8 addr_type;
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, bdaddr, bdaddr_type);
/* Advertising stops when a connection is created. On a failed
* connection it keeps running, so leave the state bit alone.
@@ -6275,6 +6286,8 @@ static void hci_le_adv_report_evt(struct hci_dev *hdev, void *data,
info->length + 1))
break;
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
if (info->length <= max_adv_len(hdev)) {
rssi = info->data[info->length];
process_adv_report(hdev, info->type, &info->bdaddr,
@@ -6363,6 +6376,8 @@ static void hci_le_ext_adv_report_evt(struct hci_dev *hdev, void *data,
info->length))
break;
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
evt_type = __le16_to_cpu(info->type) & LE_EXT_ADV_EVT_TYPE_MASK;
legacy_evt_type = ext_evt_type_to_legacy(hdev, evt_type);
@@ -6418,6 +6433,7 @@ static void hci_le_pa_sync_estabilished_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", ev->status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, ev->bdaddr_type);
hci_dev_clear_flag(hdev, HCI_PA_SYNC);
@@ -6712,6 +6728,8 @@ static void hci_le_direct_adv_report_evt(struct hci_dev *hdev, void *data,
for (i = 0; i < ev->num; i++) {
struct hci_ev_le_direct_adv_info *info = &ev->info[i];
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
process_adv_report(hdev, info->type, &info->bdaddr,
info->bdaddr_type, &info->direct_addr,
info->direct_addr_type, HCI_ADV_PHY_1M, 0,
@@ -7297,73 +7315,29 @@ static bool hci_get_cmd_complete(struct hci_dev *hdev, u16 opcode,
return true;
}
-static void hci_store_wake_reason(struct hci_dev *hdev, u8 event,
- struct sk_buff *skb)
+static void hci_store_wake_reason(struct hci_dev *hdev,
+ const bdaddr_t *bdaddr, u8 addr_type)
+ __must_hold(&hdev->lock)
{
- struct hci_ev_le_advertising_info *adv;
- struct hci_ev_le_direct_adv_info *direct_adv;
- struct hci_ev_le_ext_adv_info *ext_adv;
- const struct hci_ev_conn_complete *conn_complete = (void *)skb->data;
- const struct hci_ev_conn_request *conn_request = (void *)skb->data;
-
- hci_dev_lock(hdev);
+ lockdep_assert_held(&hdev->lock);
/* If we are currently suspended and this is the first BT event seen,
* save the wake reason associated with the event.
*/
if (!hdev->suspended || hdev->wake_reason)
- goto unlock;
+ return;
+
+ if (!bdaddr) {
+ hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED;
+ return;
+ }
/* Default to remote wake. Values for wake_reason are documented in the
* Bluez mgmt api docs.
*/
hdev->wake_reason = MGMT_WAKE_REASON_REMOTE_WAKE;
-
- /* Once configured for remote wakeup, we should only wake up for
- * reconnections. It's useful to see which device is waking us up so
- * keep track of the bdaddr of the connection event that woke us up.
- */
- if (event == HCI_EV_CONN_REQUEST) {
- bacpy(&hdev->wake_addr, &conn_request->bdaddr);
- hdev->wake_addr_type = BDADDR_BREDR;
- } else if (event == HCI_EV_CONN_COMPLETE) {
- bacpy(&hdev->wake_addr, &conn_complete->bdaddr);
- hdev->wake_addr_type = BDADDR_BREDR;
- } else if (event == HCI_EV_LE_META) {
- struct hci_ev_le_meta *le_ev = (void *)skb->data;
- u8 subevent = le_ev->subevent;
- u8 *ptr = &skb->data[sizeof(*le_ev)];
- u8 num_reports = *ptr;
-
- if ((subevent == HCI_EV_LE_ADVERTISING_REPORT ||
- subevent == HCI_EV_LE_DIRECT_ADV_REPORT ||
- subevent == HCI_EV_LE_EXT_ADV_REPORT) &&
- num_reports) {
- adv = (void *)(ptr + 1);
- direct_adv = (void *)(ptr + 1);
- ext_adv = (void *)(ptr + 1);
-
- switch (subevent) {
- case HCI_EV_LE_ADVERTISING_REPORT:
- bacpy(&hdev->wake_addr, &adv->bdaddr);
- hdev->wake_addr_type = adv->bdaddr_type;
- break;
- case HCI_EV_LE_DIRECT_ADV_REPORT:
- bacpy(&hdev->wake_addr, &direct_adv->bdaddr);
- hdev->wake_addr_type = direct_adv->bdaddr_type;
- break;
- case HCI_EV_LE_EXT_ADV_REPORT:
- bacpy(&hdev->wake_addr, &ext_adv->bdaddr);
- hdev->wake_addr_type = ext_adv->bdaddr_type;
- break;
- }
- }
- } else {
- hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED;
- }
-
-unlock:
- hci_dev_unlock(hdev);
+ bacpy(&hdev->wake_addr, bdaddr);
+ hdev->wake_addr_type = addr_type;
}
#define HCI_EV_VL(_op, _func, _min_len, _max_len) \
@@ -7610,14 +7584,15 @@ void hci_event_packet(struct hci_dev *hdev, struct sk_buff *skb)
skb_pull(skb, HCI_EVENT_HDR_SIZE);
- /* Store wake reason if we're suspended */
- hci_store_wake_reason(hdev, event, skb);
-
bt_dev_dbg(hdev, "event 0x%2.2x", event);
hci_event_func(hdev, event, skb, &opcode, &status, &req_complete,
&req_complete_skb);
+ hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, NULL, 0);
+ hci_dev_unlock(hdev);
+
if (req_complete) {
req_complete(hdev, status, opcode);
} else if (req_complete_skb) {
diff --git a/net/bluetooth/hci_sync.c b/net/bluetooth/hci_sync.c
index f4dea6fffa72..de4db563e247 100644
--- a/net/bluetooth/hci_sync.c
+++ b/net/bluetooth/hci_sync.c
@@ -32,8 +32,10 @@ static void hci_cmd_sync_complete(struct hci_dev *hdev, u8 result, u16 opcode,
WRITE_ONCE(hdev->req_status, HCI_REQ_DONE);
/* Free the request command so it is not used as response */
+ hci_dev_lock(hdev);
kfree_skb(hdev->req_skb);
hdev->req_skb = NULL;
+ hci_dev_unlock(hdev);
if (skb) {
struct sock *sk = hci_skb_sk(skb);
@@ -792,7 +794,7 @@ int hci_cmd_sync_queue_once(struct hci_dev *hdev, hci_cmd_sync_work_func_t func,
void *data, hci_cmd_sync_work_destroy_t destroy)
{
if (hci_cmd_sync_lookup_entry(hdev, func, data, destroy))
- return 0;
+ return -EEXIST;
return hci_cmd_sync_queue(hdev, func, data, destroy);
}
@@ -1075,14 +1077,19 @@ static int hci_set_random_addr_sync(struct hci_dev *hdev, bdaddr_t *rpa)
* In this kind of scenario skip the update and let the random
* address be updated at the next cycle.
*/
+ rcu_read_lock();
+
if (bacmp(&hdev->random_addr, BDADDR_ANY) &&
(hci_dev_test_flag(hdev, HCI_LE_ADV) ||
hci_lookup_le_connect(hdev))) {
bt_dev_dbg(hdev, "Deferring random address update");
hci_dev_set_flag(hdev, HCI_RPA_EXPIRED);
+ rcu_read_unlock();
return 0;
}
+ rcu_read_unlock();
+
return __hci_cmd_sync_status(hdev, HCI_OP_LE_SET_RANDOM_ADDR,
6, rpa, HCI_CMD_TIMEOUT);
}
@@ -2755,12 +2762,17 @@ static int hci_pause_addr_resolution(struct hci_dev *hdev)
/* Cannot disable addr resolution if scanning is enabled or
* when initiating an LE connection.
*/
+ rcu_read_lock();
+
if (hci_dev_test_flag(hdev, HCI_LE_SCAN) ||
hci_lookup_le_connect(hdev)) {
+ rcu_read_unlock();
bt_dev_err(hdev, "Command not allowed when scan/LE connect");
return -EPERM;
}
+ rcu_read_unlock();
+
/* Cannot disable addr resolution if advertising is enabled. */
err = hci_pause_advertising_sync(hdev);
if (err) {
@@ -2898,6 +2910,8 @@ static u8 hci_update_accept_list_sync(struct hci_dev *hdev)
if (hci_dev_test_flag(hdev, HCI_PA_SYNC)) {
struct hci_conn *conn;
+ rcu_read_lock();
+
conn = hci_conn_hash_lookup_create_pa_sync(hdev);
if (conn) {
struct conn_params pa;
@@ -2907,6 +2921,8 @@ static u8 hci_update_accept_list_sync(struct hci_dev *hdev)
bacpy(&pa.addr, &conn->dst);
pa.addr_type = conn->dst_type;
+ rcu_read_unlock();
+
/* Clear first since there could be addresses left
* behind.
*/
@@ -2916,6 +2932,8 @@ static u8 hci_update_accept_list_sync(struct hci_dev *hdev)
err = hci_le_add_accept_list_sync(hdev, &pa,
&num_entries);
goto done;
+ } else {
+ rcu_read_unlock();
}
}
@@ -2926,10 +2944,13 @@ static u8 hci_update_accept_list_sync(struct hci_dev *hdev)
* the controller.
*/
list_for_each_entry_safe(b, t, &hdev->le_accept_list, list) {
- if (hci_conn_hash_lookup_le(hdev, &b->bdaddr, b->bdaddr_type))
+ rcu_read_lock();
+
+ if (hci_conn_hash_lookup_le(hdev, &b->bdaddr, b->bdaddr_type)) {
+ rcu_read_unlock();
continue;
+ }
- /* Pointers not dereferenced, no locks needed */
pend_conn = hci_pend_le_action_lookup(&hdev->pend_le_conns,
&b->bdaddr,
b->bdaddr_type);
@@ -2937,6 +2958,8 @@ static u8 hci_update_accept_list_sync(struct hci_dev *hdev)
&b->bdaddr,
b->bdaddr_type);
+ rcu_read_unlock();
+
/* If the device is not likely to connect or report,
* remove it from the acceptlist.
*/
@@ -3058,13 +3081,21 @@ static int hci_le_set_ext_scan_param_sync(struct hci_dev *hdev, u8 type,
*/
if (hci_dev_test_flag(hdev, HCI_PA_SYNC)) {
struct hci_cp_le_add_to_accept_list *sent;
+ bdaddr_t bdaddr;
+ hci_dev_lock(hdev);
sent = hci_sent_cmd_data(hdev, HCI_OP_LE_ADD_TO_ACCEPT_LIST);
+ if (sent)
+ bacpy(&bdaddr, &sent->bdaddr);
+ hci_dev_unlock(hdev);
+
if (sent) {
struct hci_conn *conn;
+ rcu_read_lock();
+
conn = hci_conn_hash_lookup_ba(hdev, PA_LINK,
- &sent->bdaddr);
+ &bdaddr);
if (conn) {
struct bt_iso_qos *qos = &conn->iso_qos;
@@ -3087,8 +3118,12 @@ static int hci_le_set_ext_scan_param_sync(struct hci_dev *hdev, u8 type,
phy++;
}
+ rcu_read_unlock();
+
if (num_phy)
goto done;
+ } else {
+ rcu_read_unlock();
}
}
}
@@ -3339,12 +3374,16 @@ int hci_update_passive_scan_sync(struct hci_dev *hdev)
/* If there is at least one pending LE connection, we should
* keep the background scan running.
*/
+ bool exists;
/* If controller is connecting, we should not start scanning
* since some controllers are not able to scan and connect at
* the same time.
*/
- if (hci_lookup_le_connect(hdev))
+ rcu_read_lock();
+ exists = hci_lookup_le_connect(hdev);
+ rcu_read_unlock();
+ if (exists)
return 0;
bt_dev_dbg(hdev, "start background scanning");
@@ -3375,6 +3414,8 @@ static int update_passive_scan_sync(struct hci_dev *hdev, void *data)
int hci_update_passive_scan(struct hci_dev *hdev)
{
+ int err;
+
/* Only queue if it would have any effect */
if (!test_bit(HCI_UP, &hdev->flags) ||
test_bit(HCI_INIT, &hdev->flags) ||
@@ -3384,8 +3425,9 @@ int hci_update_passive_scan(struct hci_dev *hdev)
hci_dev_test_flag(hdev, HCI_UNREGISTER))
return 0;
- return hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL,
- NULL);
+ err = hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL,
+ NULL);
+ return (err == -EEXIST) ? 0 : err;
}
int hci_write_sc_support_sync(struct hci_dev *hdev, u8 val)
@@ -3559,6 +3601,7 @@ int hci_write_fast_connectable_sync(struct hci_dev *hdev, bool enable)
}
static bool disconnected_accept_list_entries(struct hci_dev *hdev)
+ __must_hold(&hdev->lock)
{
struct bdaddr_list *b;
@@ -3599,12 +3642,16 @@ int hci_update_scan_sync(struct hci_dev *hdev)
if (hdev->scanning_paused)
return 0;
+ hci_dev_lock(hdev);
+
if (hci_dev_test_flag(hdev, HCI_CONNECTABLE) ||
disconnected_accept_list_entries(hdev))
scan = SCAN_PAGE;
else
scan = SCAN_DISABLED;
+ hci_dev_unlock(hdev);
+
if (hci_dev_test_flag(hdev, HCI_DISCOVERABLE))
scan |= SCAN_INQUIRY;
@@ -7102,10 +7149,24 @@ static int hci_acl_create_conn_sync(struct hci_dev *hdev, void *data)
return err;
}
+static void hci_acl_create_conn_sync_complete(struct hci_dev *hdev, void *data,
+ int err)
+{
+ struct hci_conn *conn = data;
+
+ hci_conn_put(conn);
+}
+
int hci_connect_acl_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_acl_create_conn_sync, conn,
- NULL);
+ int err;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_acl_create_conn_sync,
+ hci_conn_get(conn),
+ hci_acl_create_conn_sync_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
static void create_le_conn_complete(struct hci_dev *hdev, void *data, int err)
@@ -7115,34 +7176,47 @@ static void create_le_conn_complete(struct hci_dev *hdev, void *data, int err)
bt_dev_dbg(hdev, "err %d", err);
if (err == -ECANCELED)
- return;
+ goto done;
hci_dev_lock(hdev);
if (!hci_conn_valid(hdev, conn))
- goto done;
+ goto unlock;
if (!err) {
hci_connect_le_scan_cleanup(conn, 0x00);
- goto done;
+ goto unlock;
}
- /* Check if connection is still pending */
- if (conn != hci_lookup_le_connect(hdev))
- goto done;
+ /* Check if this connection is still pending.
+ *
+ * hci_lookup_le_connect() returns only the first LE connection
+ * in BT_CONNECT, which is not necessarily this one when two are
+ * pending at once, so ask the connection itself.
+ */
+ if (conn->state != BT_CONNECT)
+ goto unlock;
/* Flush to make sure we send create conn cancel command if needed */
flush_delayed_work(&conn->le_conn_timeout);
hci_conn_failed(conn, bt_status(err));
-done:
+unlock:
hci_dev_unlock(hdev);
+done:
+ hci_conn_put(conn);
}
int hci_connect_le_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_le_create_conn_sync, conn,
- create_le_conn_complete);
+ int err;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_le_create_conn_sync,
+ hci_conn_get(conn),
+ create_le_conn_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
static int hci_acl_cancel_create_conn_sync(struct hci_dev *hdev,
@@ -7264,7 +7338,7 @@ static void create_pa_complete(struct hci_dev *hdev, void *data, int err)
bt_dev_dbg(hdev, "err %d", err);
if (err == -ECANCELED)
- return;
+ goto done;
hci_dev_lock(hdev);
@@ -7288,6 +7362,8 @@ static void create_pa_complete(struct hci_dev *hdev, void *data, int err)
unlock:
hci_dev_unlock(hdev);
+done:
+ hci_conn_put(conn);
}
static int hci_le_pa_create_sync(struct hci_dev *hdev, void *data)
@@ -7375,8 +7451,14 @@ static int hci_le_pa_create_sync(struct hci_dev *hdev, void *data)
int hci_connect_pa_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_le_pa_create_sync, conn,
- create_pa_complete);
+ int err;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_le_pa_create_sync,
+ hci_conn_get(conn),
+ create_pa_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
static void create_big_complete(struct hci_dev *hdev, void *data, int err)
@@ -7386,10 +7468,12 @@ static void create_big_complete(struct hci_dev *hdev, void *data, int err)
bt_dev_dbg(hdev, "err %d", err);
if (err == -ECANCELED)
- return;
+ goto done;
- if (hci_conn_valid(hdev, conn))
- clear_bit(HCI_CONN_CREATE_BIG_SYNC, &conn->flags);
+ clear_bit(HCI_CONN_CREATE_BIG_SYNC, &conn->flags);
+
+done:
+ hci_conn_put(conn);
}
static int hci_le_big_create_sync(struct hci_dev *hdev, void *data)
@@ -7439,6 +7523,15 @@ static int hci_le_big_create_sync(struct hci_dev *hdev, void *data)
int hci_connect_big_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_le_big_create_sync, conn,
- create_big_complete);
+ int err;
+
+ if (!conn)
+ return 0;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_le_big_create_sync,
+ hci_conn_get(conn),
+ create_big_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
diff --git a/net/bluetooth/hci_sysfs.c b/net/bluetooth/hci_sysfs.c
index 60350c6723cb..feb4f1302b37 100644
--- a/net/bluetooth/hci_sysfs.c
+++ b/net/bluetooth/hci_sysfs.c
@@ -13,7 +13,10 @@ static const struct class bt_class = {
static void bt_link_release(struct device *dev)
{
struct hci_conn *conn = to_hci_conn(dev);
+ struct device *parent = dev->parent;
+
kfree(conn);
+ put_device(parent);
}
static const struct device_type bt_link = {
@@ -21,6 +24,16 @@ static const struct device_type bt_link = {
.release = bt_link_release,
};
+/*
+ * The rfcomm tty device will possibly retain even when conn
+ * is down, and sysfs doesn't support move zombie device,
+ * so we should move the device before conn device is destroyed.
+ */
+static int __match_tty(struct device *dev, void *data)
+{
+ return !strncmp(dev_name(dev), "rfcomm", 6);
+}
+
void hci_conn_init_sysfs(struct hci_conn *conn)
{
struct hci_dev *hdev = conn->hdev;
@@ -29,7 +42,7 @@ void hci_conn_init_sysfs(struct hci_conn *conn)
conn->dev.type = &bt_link;
conn->dev.class = &bt_class;
- conn->dev.parent = &hdev->dev;
+ conn->dev.parent = get_device(&hdev->dev);
device_initialize(&conn->dev);
}
@@ -69,7 +82,7 @@ void hci_conn_del_sysfs(struct hci_conn *conn)
while (1) {
struct device *dev;
- dev = device_find_any_child(&conn->dev);
+ dev = device_find_child(&conn->dev, NULL, __match_tty);
if (!dev)
break;
device_move(dev, NULL, DPM_ORDER_DEV_LAST);
diff --git a/net/bluetooth/iso.c b/net/bluetooth/iso.c
index 5b930d0e5a04..98261072aa05 100644
--- a/net/bluetooth/iso.c
+++ b/net/bluetooth/iso.c
@@ -1397,6 +1397,7 @@ static int iso_sock_getname(struct socket *sock, struct sockaddr *addr,
BT_DBG("sock %p, sk %p", sock, sk);
+ memset(sa, 0, sizeof(struct sockaddr_iso));
addr->sa_family = AF_BLUETOOTH;
if (peer) {
@@ -1980,23 +1981,17 @@ static void iso_sock_ready(struct sock *sk)
{
BT_DBG("sk %p", sk);
- if (!sk)
- return;
-
- lock_sock(sk);
+ lockdep_assert(lockdep_sock_is_held(sk));
switch (sk->sk_state) {
case BT_DISCONN:
case BT_CLOSED:
- release_sock(sk);
return;
}
iso_sock_clear_timer(sk);
sk->sk_state = BT_CONNECTED;
sk->sk_state_change(sk);
-
- release_sock(sk);
}
static bool iso_match_big(struct sock *sk, void *data)
@@ -2014,7 +2009,7 @@ static bool iso_match_pa_sync_flag(struct sock *sk, void *data)
static void iso_conn_ready(struct iso_conn *conn)
{
struct sock *parent = NULL;
- struct sock *sk = conn->sk;
+ struct sock *sk;
struct hci_ev_le_big_sync_estabilished *ev = NULL;
struct hci_ev_le_pa_sync_established *ev2 = NULL;
struct hci_ev_le_per_adv_report *ev3 = NULL;
@@ -2022,8 +2017,26 @@ static void iso_conn_ready(struct iso_conn *conn)
BT_DBG("conn %p", conn);
+ iso_conn_lock(conn);
+ sk = iso_sock_hold(conn);
+ iso_conn_unlock(conn);
+
if (sk) {
- iso_sock_ready(conn->sk);
+ lock_sock(sk);
+
+ /* conn->sk may have become NULL if racing with sk close, but
+ * due to held hdev->lock, it can't become different sk.
+ */
+ if (!conn->sk) {
+ release_sock(sk);
+ sock_put(sk);
+ return;
+ }
+
+ iso_sock_ready(sk);
+
+ release_sock(sk);
+ sock_put(sk);
} else {
hcon = conn->hcon;
if (!hcon)
@@ -2281,9 +2294,11 @@ int iso_connect_ind(struct hci_dev *hdev, bdaddr_t *bdaddr, __u8 *flags)
if (!sk)
goto done;
- hcon = iso_pi(sk)->conn->hcon;
+ lock_sock(sk);
+
+ hcon = iso_pi(sk)->conn ? iso_pi(sk)->conn->hcon : NULL;
if (!hcon)
- goto done;
+ goto release3;
if (ev3->data_status == LE_PA_DATA_TRUNCATED) {
/* The controller was unable to retrieve PA data. */
@@ -2291,12 +2306,12 @@ int iso_connect_ind(struct hci_dev *hdev, bdaddr_t *bdaddr, __u8 *flags)
HCI_MAX_PER_AD_TOT_LEN);
hcon->le_per_adv_data_len = 0;
hcon->le_per_adv_data_offset = 0;
- goto done;
+ goto release3;
}
if (hcon->le_per_adv_data_offset + ev3->length >
HCI_MAX_PER_AD_TOT_LEN)
- goto done;
+ goto release3;
memcpy(hcon->le_per_adv_data + hcon->le_per_adv_data_offset,
ev3->data, ev3->length);
@@ -2315,7 +2330,7 @@ int iso_connect_ind(struct hci_dev *hdev, bdaddr_t *bdaddr, __u8 *flags)
&base_len);
if (!base || base_len > BASE_MAX_LENGTH)
- goto done;
+ goto release3;
lock_sock(sk);
memcpy(iso_pi(sk)->base, base, base_len);
@@ -2327,6 +2342,9 @@ int iso_connect_ind(struct hci_dev *hdev, bdaddr_t *bdaddr, __u8 *flags)
*/
hcon->le_per_adv_data_len = 0;
}
+
+release3:
+ release_sock(sk);
} else {
sk = iso_get_sock(&hdev->bdaddr, BDADDR_ANY,
BT_LISTEN, NULL, NULL);
diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c
index 12d39f1e822b..61914f280d6f 100644
--- a/net/bluetooth/l2cap_core.c
+++ b/net/bluetooth/l2cap_core.c
@@ -1843,7 +1843,10 @@ static void l2cap_conn_del(struct hci_conn *hcon, int err)
hci_chan_del(conn->hchan);
conn->hchan = NULL;
+ spin_lock(&hcon->proto_lock);
hcon->l2cap_data = NULL;
+ spin_unlock(&hcon->proto_lock);
+
mutex_unlock(&conn->lock);
l2cap_conn_put(conn);
}
@@ -7167,8 +7170,6 @@ static struct l2cap_conn *l2cap_conn_add(struct hci_conn *hcon)
}
kref_init(&conn->ref);
- hcon->l2cap_data = conn;
- conn->hcon = hci_conn_get(hcon);
conn->hchan = hchan;
BT_DBG("hcon %p conn %p hchan %p", hcon, conn, hchan);
@@ -7197,6 +7198,11 @@ static struct l2cap_conn *l2cap_conn_add(struct hci_conn *hcon)
conn->disc_reason = HCI_ERROR_REMOTE_USER_TERM;
+ spin_lock(&hcon->proto_lock);
+ conn->hcon = hci_conn_get(hcon);
+ hcon->l2cap_data = conn;
+ spin_unlock(&hcon->proto_lock);
+
return conn;
}
@@ -7585,13 +7591,18 @@ static void l2cap_connect_cfm(struct hci_conn *hcon, u8 status)
int l2cap_disconn_ind(struct hci_conn *hcon)
{
- struct l2cap_conn *conn = hcon->l2cap_data;
+ struct l2cap_conn *conn;
+ int ret = HCI_ERROR_REMOTE_USER_TERM;
BT_DBG("hcon %p", hcon);
- if (!conn)
- return HCI_ERROR_REMOTE_USER_TERM;
- return conn->disc_reason;
+ spin_lock(&hcon->proto_lock);
+ conn = hcon->l2cap_data;
+ if (conn)
+ ret = conn->disc_reason;
+ spin_unlock(&hcon->proto_lock);
+
+ return ret;
}
static void l2cap_disconn_cfm(struct hci_conn *hcon, u8 reason)
diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c
index d5b1b5a231a1..ef3af57e4637 100644
--- a/net/bluetooth/l2cap_sock.c
+++ b/net/bluetooth/l2cap_sock.c
@@ -31,6 +31,7 @@
#include <linux/export.h>
#include <linux/filter.h>
#include <linux/sched/signal.h>
+#include <linux/uio.h>
#include <net/bluetooth/bluetooth.h>
#include <net/bluetooth/hci_core.h>
@@ -438,11 +439,26 @@ static int l2cap_get_mode(struct l2cap_chan *chan)
return -EINVAL;
}
+static struct l2cap_conn *l2cap_chan_conn(struct l2cap_chan *chan)
+{
+ lockdep_assert_held(&chan->lock);
+
+ /* l2cap_conn_del() sets FLAG_DEL while holding chan->lock before
+ * conn->hcon is deleted. If not set and conn is non-NULL, conn->hcon
+ * remains alive during this chan->lock critical section.
+ */
+ if (test_bit(FLAG_DEL, &chan->flags))
+ return NULL;
+
+ return chan->conn;
+}
+
static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
- char __user *optval, int __user *optlen)
+ sockopt_t *sopt)
{
struct sock *sk = sock->sk;
struct l2cap_chan *chan = l2cap_pi(sk)->chan;
+ struct l2cap_conn *conn;
struct l2cap_options opts;
struct l2cap_conninfo cinfo;
int err = 0;
@@ -451,9 +467,9 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
BT_DBG("sk %p", sk);
- if (get_user(len, optlen))
- return -EFAULT;
+ len = sopt->optlen;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -494,7 +510,7 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
BT_DBG("mode 0x%2.2x", chan->mode);
len = min(len, sizeof(opts));
- if (copy_to_user(optval, (char *) &opts, len))
+ if (copy_to_iter(&opts, len, &sopt->iter_out) != len)
err = -EFAULT;
break;
@@ -526,7 +542,8 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
if (test_bit(FLAG_FORCE_RELIABLE, &chan->flags))
opt |= L2CAP_LM_RELIABLE;
- if (put_user(opt, (u32 __user *) optval))
+ if (copy_to_iter(&opt, sizeof(opt), &sopt->iter_out) !=
+ sizeof(opt))
err = -EFAULT;
break;
@@ -539,12 +556,18 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
break;
}
+ conn = l2cap_chan_conn(chan);
+ if (!conn) {
+ err = -ENOTCONN;
+ break;
+ }
+
memset(&cinfo, 0, sizeof(cinfo));
- cinfo.hci_handle = chan->conn->hcon->handle;
- memcpy(cinfo.dev_class, chan->conn->hcon->dev_class, 3);
+ cinfo.hci_handle = conn->hcon->handle;
+ memcpy(cinfo.dev_class, conn->hcon->dev_class, 3);
len = min(len, sizeof(cinfo));
- if (copy_to_user(optval, (char *) &cinfo, len))
+ if (copy_to_iter(&cinfo, len, &sopt->iter_out) != len)
err = -EFAULT;
break;
@@ -555,30 +578,35 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
+
return err;
}
static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
- char __user *optval, int __user *optlen)
+ sockopt_t *sopt)
{
struct sock *sk = sock->sk;
struct l2cap_chan *chan = l2cap_pi(sk)->chan;
+ struct l2cap_conn *conn;
struct bt_security sec;
struct bt_power pwr;
- u32 phys;
int len, mode, err = 0;
+ u32 opt;
+ u16 mtu;
+ u8 mval;
BT_DBG("sk %p", sk);
if (level == SOL_L2CAP)
- return l2cap_sock_getsockopt_old(sock, optname, optval, optlen);
+ return l2cap_sock_getsockopt_old(sock, optname, sopt);
if (level != SOL_BLUETOOTH)
return -ENOPROTOOPT;
- if (get_user(len, optlen))
- return -EFAULT;
+ len = sopt->optlen;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -590,18 +618,20 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
}
+ conn = l2cap_chan_conn(chan);
+
memset(&sec, 0, sizeof(sec));
- if (chan->conn) {
- sec.level = chan->conn->hcon->sec_level;
+ if (conn) {
+ sec.level = conn->hcon->sec_level;
if (sk->sk_state == BT_CONNECTED)
- sec.key_size = chan->conn->hcon->enc_key_size;
+ sec.key_size = conn->hcon->enc_key_size;
} else {
sec.level = chan->sec_level;
}
len = min_t(unsigned int, len, sizeof(sec));
- if (copy_to_user(optval, (char *) &sec, len))
+ if (copy_to_iter(&sec, len, &sopt->iter_out) != len)
err = -EFAULT;
break;
@@ -612,15 +642,17 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
}
- if (put_user(test_bit(BT_SK_DEFER_SETUP, &bt_sk(sk)->flags),
- (u32 __user *) optval))
+ opt = test_bit(BT_SK_DEFER_SETUP, &bt_sk(sk)->flags);
+ if (copy_to_iter(&opt, sizeof(opt), &sopt->iter_out) !=
+ sizeof(opt))
err = -EFAULT;
break;
case BT_FLUSHABLE:
- if (put_user(test_bit(FLAG_FLUSHABLE, &chan->flags),
- (u32 __user *) optval))
+ opt = test_bit(FLAG_FLUSHABLE, &chan->flags);
+ if (copy_to_iter(&opt, sizeof(opt), &sopt->iter_out) !=
+ sizeof(opt))
err = -EFAULT;
break;
@@ -635,13 +667,15 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
pwr.force_active = test_bit(FLAG_FORCE_ACTIVE, &chan->flags);
len = min_t(unsigned int, len, sizeof(pwr));
- if (copy_to_user(optval, (char *) &pwr, len))
+ if (copy_to_iter(&pwr, len, &sopt->iter_out) != len)
err = -EFAULT;
break;
case BT_CHANNEL_POLICY:
- if (put_user(chan->chan_policy, (u32 __user *) optval))
+ opt = chan->chan_policy;
+ if (copy_to_iter(&opt, sizeof(opt), &sopt->iter_out) !=
+ sizeof(opt))
err = -EFAULT;
break;
@@ -656,7 +690,9 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
}
- if (put_user(chan->omtu, (u16 __user *) optval))
+ mtu = chan->omtu;
+ if (copy_to_iter(&mtu, sizeof(mtu), &sopt->iter_out) !=
+ sizeof(mtu))
err = -EFAULT;
break;
@@ -666,19 +702,24 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
}
- if (put_user(chan->imtu, (u16 __user *) optval))
+ mtu = chan->imtu;
+ if (copy_to_iter(&mtu, sizeof(mtu), &sopt->iter_out) !=
+ sizeof(mtu))
err = -EFAULT;
break;
case BT_PHY:
- if (sk->sk_state != BT_CONNECTED) {
+ conn = l2cap_chan_conn(chan);
+
+ if (sk->sk_state != BT_CONNECTED || !conn) {
err = -ENOTCONN;
break;
}
- phys = hci_conn_get_phy(chan->conn->hcon);
+ opt = hci_conn_get_phy(conn->hcon);
- if (put_user(phys, (u32 __user *) optval))
+ if (copy_to_iter(&opt, sizeof(opt), &sopt->iter_out) !=
+ sizeof(opt))
err = -EFAULT;
break;
@@ -699,7 +740,9 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
}
- if (put_user(mode, (u8 __user *) optval))
+ mval = mode;
+ if (copy_to_iter(&mval, sizeof(mval), &sopt->iter_out) !=
+ sizeof(mval))
err = -EFAULT;
break;
@@ -709,6 +752,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -739,6 +783,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname,
BT_DBG("sk %p", sk);
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -840,6 +885,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -903,6 +949,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
if (level != SOL_BLUETOOTH)
return -ENOPROTOOPT;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -928,11 +975,10 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
chan->sec_level = sec.level;
- if (!chan->conn)
+ conn = l2cap_chan_conn(chan);
+ if (!conn)
break;
- conn = chan->conn;
-
/* change security for LE channels */
if (chan->scid == L2CAP_CID_ATT) {
if (smp_conn_security(conn->hcon, sec.level)) {
@@ -987,7 +1033,8 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
}
if (opt == BT_FLUSHABLE_OFF) {
- conn = chan->conn;
+ conn = l2cap_chan_conn(chan);
+
/* proceed further only when we have l2cap_conn and
No Flush support in the LM */
if (!conn || !lmp_no_flush_capable(conn->hcon->hdev)) {
@@ -1111,6 +1158,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -2019,7 +2067,7 @@ static const struct proto_ops l2cap_sock_ops = {
.socketpair = sock_no_socketpair,
.shutdown = l2cap_sock_shutdown,
.setsockopt = l2cap_sock_setsockopt,
- .getsockopt = l2cap_sock_getsockopt
+ .getsockopt_iter = l2cap_sock_getsockopt
};
static const struct net_proto_family l2cap_sock_family_ops = {
diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c
index 6c57f5a766c1..b2d9bd743ea2 100644
--- a/net/bluetooth/rfcomm/core.c
+++ b/net/bluetooth/rfcomm/core.c
@@ -65,10 +65,9 @@ static void rfcomm_make_uih(struct sk_buff *skb, u8 addr);
static void rfcomm_process_connect(struct rfcomm_session *s);
-static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src,
- bdaddr_t *dst,
- u8 sec_level,
- int *err);
+static struct socket *rfcomm_session_connect(bdaddr_t *src, bdaddr_t *dst,
+ u8 sec_level, int *err);
+static struct rfcomm_session *rfcomm_session_add(struct socket *sock, int state);
static struct rfcomm_session *rfcomm_session_get(bdaddr_t *src, bdaddr_t *dst);
static struct rfcomm_session *rfcomm_session_del(struct rfcomm_session *s);
@@ -368,28 +367,17 @@ static int rfcomm_check_channel(u8 channel)
return channel < 1 || channel > 30;
}
-static int __rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst, u8 channel)
+static int __rfcomm_dlc_open(struct rfcomm_dlc *d, struct rfcomm_session *s,
+ u8 channel)
{
- struct rfcomm_session *s;
- int err = 0;
u8 dlci;
- BT_DBG("dlc %p state %ld %pMR -> %pMR channel %d",
- d, d->state, src, dst, channel);
-
- if (rfcomm_check_channel(channel))
- return -EINVAL;
+ BT_DBG("dlc %p state %ld session %p channel %d",
+ d, d->state, s, channel);
if (d->state != BT_OPEN && d->state != BT_CLOSED)
return 0;
- s = rfcomm_session_get(src, dst);
- if (!s) {
- s = rfcomm_session_create(src, dst, d->sec_level, &err);
- if (!s)
- return err;
- }
-
dlci = __dlci(__session_dir(s), channel);
/* Check if DLCI already exists */
@@ -424,14 +412,84 @@ static int __rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst,
int rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst, u8 channel)
{
- int r;
+ struct rfcomm_session *s;
+ struct socket *sock;
+ int err;
+
+ BT_DBG("dlc %p state %ld %pMR -> %pMR channel %d",
+ d, d->state, src, dst, channel);
+
+ if (rfcomm_check_channel(channel))
+ return -EINVAL;
rfcomm_lock();
- r = __rfcomm_dlc_open(d, src, dst, channel);
+ clear_bit(RFCOMM_CLOSED, &d->flags);
+ /* Do not page the remote device for a DLC that cannot be opened
+ * anyway. __rfcomm_dlc_open() looks at the state again once the
+ * lock has been re-acquired below.
+ */
+ if (d->state != BT_OPEN && d->state != BT_CLOSED) {
+ rfcomm_unlock();
+ return 0;
+ }
+
+ s = rfcomm_session_get(src, dst);
+ if (s) {
+ err = __rfcomm_dlc_open(d, s, channel);
+ rfcomm_unlock();
+ return err;
+ }
rfcomm_unlock();
- return r;
+
+ /* There is no session for this pair yet. kernel_connect() ends up in
+ * l2cap_chan_connect(), which takes hdev->lock, and the HCI event
+ * path takes rfcomm_mutex while holding hdev->lock, so the socket has
+ * to be connected with rfcomm_mutex released.
+ */
+ sock = rfcomm_session_connect(src, dst, d->sec_level, &err);
+ if (!sock)
+ return err;
+
+ rfcomm_lock();
+
+ /* The DLC may have been closed while the socket was connecting. It
+ * was not on a session, so rfcomm_dlc_close() could only mark it;
+ * attaching it now would leave it with no owner.
+ */
+ if (test_bit(RFCOMM_CLOSED, &d->flags)) {
+ rfcomm_unlock();
+ sock_release(sock);
+ return -ECONNRESET;
+ }
+
+ /* Another opener may have added a session for the same pair in the
+ * meantime; that one is used and this socket is dropped.
+ */
+ s = rfcomm_session_get(src, dst);
+ if (!s) {
+ s = rfcomm_session_add(sock, BT_BOUND);
+ if (s) {
+ s->initiator = 1;
+ sock = NULL;
+ }
+ }
+
+ err = s ? __rfcomm_dlc_open(d, s, channel) : -ENOMEM;
+
+ rfcomm_unlock();
+
+ if (sock)
+ sock_release(sock);
+
+ /* Over an existing ACL link the connection can complete before the
+ * session reaches the list, and that wakeup is then lost, so let
+ * krfcommd look at the socket state now.
+ */
+ rfcomm_schedule();
+
+ return err;
}
static void __rfcomm_dlc_disconn(struct rfcomm_dlc *d)
@@ -511,8 +569,14 @@ int rfcomm_dlc_close(struct rfcomm_dlc *d, int err)
rfcomm_lock();
s = d->session;
- if (!s)
+ if (!s) {
+ /* Not on a session yet: rfcomm_dlc_open() may be connecting a
+ * socket for it with rfcomm_mutex released. Leave a mark so
+ * that it does not attach the DLC once it holds the lock again.
+ */
+ set_bit(RFCOMM_CLOSED, &d->flags);
goto no_session;
+ }
/* after waiting on the mutex check the session still exists
* then check the dlc still exists
@@ -760,12 +824,12 @@ static struct rfcomm_session *rfcomm_session_close(struct rfcomm_session *s,
return rfcomm_session_del(s);
}
-static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src,
- bdaddr_t *dst,
- u8 sec_level,
- int *err)
+/* Creates the L2CAP socket a new session will run on and starts connecting
+ * it. Must be called with rfcomm_mutex released.
+ */
+static struct socket *rfcomm_session_connect(bdaddr_t *src, bdaddr_t *dst,
+ u8 sec_level, int *err)
{
- struct rfcomm_session *s = NULL;
struct sockaddr_l2 addr;
struct socket *sock;
struct sock *sk;
@@ -795,24 +859,16 @@ static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src,
l2cap_pi(sk)->chan->mode = L2CAP_MODE_ERTM;
release_sock(sk);
- s = rfcomm_session_add(sock, BT_BOUND);
- if (!s) {
- *err = -ENOMEM;
- goto failed;
- }
-
- s->initiator = 1;
-
bacpy(&addr.l2_bdaddr, dst);
addr.l2_family = AF_BLUETOOTH;
addr.l2_psm = cpu_to_le16(L2CAP_PSM_RFCOMM);
addr.l2_cid = 0;
addr.l2_bdaddr_type = BDADDR_BREDR;
*err = kernel_connect(sock, (struct sockaddr *) &addr, sizeof(addr), O_NONBLOCK);
- if (*err == 0 || *err == -EINPROGRESS)
- return s;
+ if (*err && *err != -EINPROGRESS)
+ goto failed;
- return rfcomm_session_del(s);
+ return sock;
failed:
sock_release(sock);
diff --git a/net/bridge/br_arp_nd_proxy.c b/net/bridge/br_arp_nd_proxy.c
index f2d6643d2c0a..e1c6a54e9a0c 100644
--- a/net/bridge/br_arp_nd_proxy.c
+++ b/net/bridge/br_arp_nd_proxy.c
@@ -191,13 +191,15 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br,
n = neigh_lookup(&arp_tbl, &tip, vlandev);
if (n) {
struct net_bridge_fdb_entry *f;
+ u8 ha[ETH_ALEN] __aligned(2);
if (!(READ_ONCE(n->nud_state) & NUD_VALID)) {
neigh_release(n);
return;
}
- f = br_fdb_find_rcu(br, n->ha, vid);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = br_fdb_find_rcu(br, ha, vid);
if (f) {
const struct net_bridge_port *dst = READ_ONCE(f->dst);
bool replied = false;
@@ -207,10 +209,10 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br,
br_is_neigh_suppress_enabled(dst, vid)) {
if (!vid)
br_arp_send(br, p, skb->dev, sip, tip,
- sha, n->ha, sha, 0, 0);
+ sha, ha, sha, 0, 0);
else
br_arp_send(br, p, skb->dev, sip, tip,
- sha, n->ha, sha,
+ sha, ha, sha,
skb->vlan_proto,
skb_vlan_tag_get(skb));
replied = true;
@@ -248,7 +250,7 @@ struct nd_msg *br_is_nd_neigh_msg(struct sk_buff *skb, struct nd_msg *msg)
}
static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
- struct sk_buff *request, struct neighbour *n,
+ struct sk_buff *request, struct neighbour *n, u8 *ha,
__be16 vlan_proto, u16 vlan_tci)
{
struct net_device *dev = request->dev;
@@ -306,7 +308,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
ipv6_eth_mc_map(&in6addr_linklocal_allnodes, eth_hdr(reply)->h_dest);
else
ether_addr_copy(eth_hdr(reply)->h_dest, daddr);
- ether_addr_copy(eth_hdr(reply)->h_source, n->ha);
+ ether_addr_copy(eth_hdr(reply)->h_source, ha);
eth_hdr(reply)->h_proto = htons(ETH_P_IPV6);
reply->protocol = htons(ETH_P_IPV6);
@@ -336,7 +338,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
na->icmph.icmp6_override = 1;
na->icmph.icmp6_solicited = dad ? 0 : 1;
na->target = ns->target;
- ether_addr_copy(&na->opt[2], n->ha);
+ ether_addr_copy(&na->opt[2], ha);
na->opt[0] = ND_OPT_TARGET_LL_ADDR;
na->opt[1] = na_olen >> 3;
@@ -365,7 +367,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
__vlan_hwaccel_put_tag(reply, vlan_proto, vlan_tci);
netdev_dbg(dev, "nd send dev %s dst %pI6 dst_hw %pM src %pI6 src_hw %pM\n",
- dev->name, &pip6->daddr, daddr, &pip6->saddr, n->ha);
+ dev->name, &pip6->daddr, daddr, &pip6->saddr, ha);
if (p) {
dev_queue_xmit(reply);
@@ -462,24 +464,26 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br,
n = neigh_lookup(ipv6_stub->nd_tbl, &msg->target, vlandev);
if (n) {
struct net_bridge_fdb_entry *f;
+ u8 ha[ETH_ALEN] __aligned(2);
if (!(READ_ONCE(n->nud_state) & NUD_VALID)) {
neigh_release(n);
return;
}
- f = br_fdb_find_rcu(br, n->ha, vid);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = br_fdb_find_rcu(br, ha, vid);
if (f) {
const struct net_bridge_port *dst = READ_ONCE(f->dst);
bool replied = false;
if (br_is_neigh_suppress_enabled(dst, vid)) {
if (vid != 0)
- br_nd_send(br, p, skb, n,
+ br_nd_send(br, p, skb, n, ha,
skb->vlan_proto,
skb_vlan_tag_get(skb));
else
- br_nd_send(br, p, skb, n, 0, 0);
+ br_nd_send(br, p, skb, n, ha, 0, 0);
replied = true;
}
diff --git a/net/bridge/br_device.c b/net/bridge/br_device.c
index 446761da4e54..339b9be9653d 100644
--- a/net/bridge/br_device.c
+++ b/net/bridge/br_device.c
@@ -488,7 +488,7 @@ void br_dev_setup(struct net_device *dev)
SET_NETDEV_DEVTYPE(dev, &br_type);
dev->priv_flags = IFF_EBRIDGE | IFF_NO_QUEUE;
dev->lltx = true;
- dev->netns_local = true;
+ dev->netns_immutable = true;
dev->features = COMMON_FEATURES | NETIF_F_HW_VLAN_CTAG_TX |
NETIF_F_HW_VLAN_STAG_TX;
diff --git a/net/bridge/netfilter/ebtable_broute.c b/net/bridge/netfilter/ebtable_broute.c
index 741360219552..e6f9e343b41f 100644
--- a/net/bridge/netfilter/ebtable_broute.c
+++ b/net/bridge/netfilter/ebtable_broute.c
@@ -128,8 +128,8 @@ static int __init ebtable_broute_init(void)
static void __exit ebtable_broute_fini(void)
{
- unregister_pernet_subsys(&broute_net_ops);
ebt_unregister_template(&broute_table);
+ unregister_pernet_subsys(&broute_net_ops);
}
module_init(ebtable_broute_init);
diff --git a/net/bridge/netfilter/ebtable_filter.c b/net/bridge/netfilter/ebtable_filter.c
index dacd81b12e62..02b6501c15a5 100644
--- a/net/bridge/netfilter/ebtable_filter.c
+++ b/net/bridge/netfilter/ebtable_filter.c
@@ -109,8 +109,8 @@ static int __init ebtable_filter_init(void)
static void __exit ebtable_filter_fini(void)
{
- unregister_pernet_subsys(&frame_filter_net_ops);
ebt_unregister_template(&frame_filter);
+ unregister_pernet_subsys(&frame_filter_net_ops);
}
module_init(ebtable_filter_init);
diff --git a/net/bridge/netfilter/ebtable_nat.c b/net/bridge/netfilter/ebtable_nat.c
index 0f2a8c6118d4..9985a82555c4 100644
--- a/net/bridge/netfilter/ebtable_nat.c
+++ b/net/bridge/netfilter/ebtable_nat.c
@@ -109,8 +109,8 @@ static int __init ebtable_nat_init(void)
static void __exit ebtable_nat_fini(void)
{
- unregister_pernet_subsys(&frame_nat_net_ops);
ebt_unregister_template(&frame_nat);
+ unregister_pernet_subsys(&frame_nat_net_ops);
}
module_init(ebtable_nat_init);
diff --git a/net/bridge/netfilter/ebtables.c b/net/bridge/netfilter/ebtables.c
index ab5eca1c9981..96176aa88111 100644
--- a/net/bridge/netfilter/ebtables.c
+++ b/net/bridge/netfilter/ebtables.c
@@ -42,6 +42,7 @@
struct ebt_pernet {
struct list_head tables;
+ struct list_head dead_tables;
};
struct ebt_template {
@@ -1164,11 +1165,6 @@ static int do_replace(struct net *net, sockptr_t arg, unsigned int len)
static void __ebt_unregister_table(struct net *net, struct ebt_table *table)
{
- mutex_lock(&ebt_mutex);
- list_del(&table->list);
- mutex_unlock(&ebt_mutex);
- audit_log_nfcfg(table->name, AF_BRIDGE, table->private->nentries,
- AUDIT_XT_OP_UNREGISTER, GFP_KERNEL);
EBT_ENTRY_ITERATE(table->private->entries, table->private->entries_size,
ebt_cleanup_entry, net, NULL);
if (table->private->nentries)
@@ -1269,13 +1265,15 @@ int ebt_register_table(struct net *net, const struct ebt_table *input_table,
for (i = 0; i < num_ops; i++)
ops[i].priv = table;
- list_add(&table->list, &ebt_net->tables);
- mutex_unlock(&ebt_mutex);
-
table->ops = ops;
ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret)
+ if (ret) {
+ synchronize_rcu();
__ebt_unregister_table(net, table);
+ } else {
+ list_add(&table->list, &ebt_net->tables);
+ }
+ mutex_unlock(&ebt_mutex);
audit_log_nfcfg(repl->name, AF_BRIDGE, repl->nentries,
AUDIT_XT_OP_REGISTER, GFP_KERNEL);
@@ -1341,7 +1339,7 @@ void ebt_unregister_template(const struct ebt_table *t)
}
EXPORT_SYMBOL(ebt_unregister_template);
-static struct ebt_table *__ebt_find_table(struct net *net, const char *name)
+void ebt_unregister_table_pre_exit(struct net *net, const char *name)
{
struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
struct ebt_table *t;
@@ -1350,30 +1348,36 @@ static struct ebt_table *__ebt_find_table(struct net *net, const char *name)
list_for_each_entry(t, &ebt_net->tables, list) {
if (strcmp(t->name, name) == 0) {
+ list_move(&t->list, &ebt_net->dead_tables);
mutex_unlock(&ebt_mutex);
- return t;
+ nf_unregister_net_hooks(net, t->ops, hweight32(t->valid_hooks));
+ return;
}
}
mutex_unlock(&ebt_mutex);
- return NULL;
-}
-
-void ebt_unregister_table_pre_exit(struct net *net, const char *name)
-{
- struct ebt_table *table = __ebt_find_table(net, name);
-
- if (table)
- nf_unregister_net_hooks(net, table->ops, hweight32(table->valid_hooks));
}
EXPORT_SYMBOL(ebt_unregister_table_pre_exit);
void ebt_unregister_table(struct net *net, const char *name)
{
- struct ebt_table *table = __ebt_find_table(net, name);
+ struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
+ struct ebt_table *t;
- if (table)
- __ebt_unregister_table(net, table);
+ mutex_lock(&ebt_mutex);
+
+ list_for_each_entry(t, &ebt_net->dead_tables, list) {
+ if (strcmp(t->name, name) == 0) {
+ list_del(&t->list);
+ audit_log_nfcfg(t->name, AF_BRIDGE, t->private->nentries,
+ AUDIT_XT_OP_UNREGISTER, GFP_KERNEL);
+ __ebt_unregister_table(net, t);
+ mutex_unlock(&ebt_mutex);
+ return;
+ }
+ }
+
+ mutex_unlock(&ebt_mutex);
}
/* userspace just supplied us with counters */
@@ -2594,11 +2598,21 @@ static int __net_init ebt_pernet_init(struct net *net)
struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
INIT_LIST_HEAD(&ebt_net->tables);
+ INIT_LIST_HEAD(&ebt_net->dead_tables);
return 0;
}
+static void __net_exit ebt_pernet_exit(struct net *net)
+{
+ struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
+
+ WARN_ON_ONCE(!list_empty(&ebt_net->tables));
+ WARN_ON_ONCE(!list_empty(&ebt_net->dead_tables));
+}
+
static struct pernet_operations ebt_net_ops = {
.init = ebt_pernet_init,
+ .exit = ebt_pernet_exit,
.id = &ebt_pernet_id,
.size = sizeof(struct ebt_pernet),
};
@@ -2607,19 +2621,20 @@ static int __init ebtables_init(void)
{
int ret;
- ret = xt_register_target(&ebt_standard_target);
+ ret = register_pernet_subsys(&ebt_net_ops);
if (ret < 0)
return ret;
- ret = nf_register_sockopt(&ebt_sockopts);
+
+ ret = xt_register_target(&ebt_standard_target);
if (ret < 0) {
- xt_unregister_target(&ebt_standard_target);
+ unregister_pernet_subsys(&ebt_net_ops);
return ret;
}
- ret = register_pernet_subsys(&ebt_net_ops);
+ ret = nf_register_sockopt(&ebt_sockopts);
if (ret < 0) {
- nf_unregister_sockopt(&ebt_sockopts);
xt_unregister_target(&ebt_standard_target);
+ unregister_pernet_subsys(&ebt_net_ops);
return ret;
}
diff --git a/net/ceph/auth_x.c b/net/ceph/auth_x.c
index c83559f015e4..f805a813308f 100644
--- a/net/ceph/auth_x.c
+++ b/net/ceph/auth_x.c
@@ -86,6 +86,11 @@ static int __ceph_x_decrypt(struct ceph_crypto_key *secret, void *p,
if (ret)
return ret;
+ if (plaintext_len < sizeof(*hdr)) {
+ pr_err("%s plaintext too small %d\n", __func__, plaintext_len);
+ return -EINVAL;
+ }
+
if (le64_to_cpu(hdr->magic) != CEPHX_ENC_MAGIC) {
pr_err("%s bad magic\n", __func__);
return -EINVAL;
diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c
index bc01b3aa6b0f..a09bc55ebd75 100644
--- a/net/core/bpf_sk_storage.c
+++ b/net/core/bpf_sk_storage.c
@@ -50,15 +50,14 @@ void bpf_sk_storage_free(struct sock *sk)
{
struct bpf_local_storage *sk_storage;
- rcu_read_lock();
+ rcu_read_lock_dont_migrate();
sk_storage = rcu_dereference(sk->sk_bpf_storage);
- if (!sk_storage) {
- rcu_read_unlock();
- return;
- }
+ if (!sk_storage)
+ goto out;
bpf_local_storage_destroy(sk_storage);
- rcu_read_unlock();
+out:
+ rcu_read_unlock_migrate();
}
static void bpf_sk_storage_map_free(struct bpf_map *map)
@@ -158,9 +157,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)
struct bpf_local_storage_elem *selem;
int ret = 0;
- RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
-
- rcu_read_lock();
+ rcu_read_lock_dont_migrate();
sk_storage = rcu_dereference(sk->sk_bpf_storage);
if (!sk_storage || hlist_empty(&sk_storage->list))
@@ -211,7 +208,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)
}
out:
- rcu_read_unlock();
+ rcu_read_unlock_migrate();
/* In case of an error, don't free anything explicitly here, the
* caller is responsible to call bpf_sk_storage_free.
diff --git a/net/core/dev.c b/net/core/dev.c
index 48f099f9e804..397fd0f9cc7a 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3862,49 +3862,62 @@ struct sk_buff *validate_xmit_skb_list(struct sk_buff *skb, struct net_device *d
}
EXPORT_SYMBOL_GPL(validate_xmit_skb_list);
-static void qdisc_pkt_len_init(struct sk_buff *skb)
+static enum skb_drop_reason qdisc_pkt_len_init(struct sk_buff *skb)
{
- const struct skb_shared_info *shinfo = skb_shinfo(skb);
+ struct skb_shared_info *shinfo = skb_shinfo(skb);
+ unsigned int hdr_len, tlen;
+ u16 gso_segs;
qdisc_skb_cb(skb)->pkt_len = skb->len;
+ if (!shinfo->gso_size)
+ return SKB_NOT_DROPPED_YET;
+
+ gso_segs = shinfo->gso_segs;
/* To get more precise estimation of bytes sent on wire,
* we add to pkt_len the headers size of all segments
*/
- if (shinfo->gso_size && skb_transport_header_was_set(skb)) {
- u16 gso_segs = shinfo->gso_segs;
- unsigned int hdr_len;
- /* mac layer + network layer */
+ /* mac layer + network layer */
+ if (!skb->encapsulation) {
+ if (unlikely(!skb_transport_header_was_set(skb)))
+ return SKB_NOT_DROPPED_YET;
hdr_len = skb_transport_offset(skb);
+ } else {
+ hdr_len = skb_inner_transport_offset(skb);
+ }
+ /* + transport layer */
+ if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6))) {
+ const struct tcphdr *th;
- /* + transport layer */
- if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6))) {
- const struct tcphdr *th;
- struct tcphdr _tcphdr;
-
- th = skb_header_pointer(skb, hdr_len,
- sizeof(_tcphdr), &_tcphdr);
- if (likely(th))
- hdr_len += __tcp_hdrlen(th);
- } else if (shinfo->gso_type & SKB_GSO_UDP_L4) {
- struct udphdr _udphdr;
+ if (!pskb_may_pull(skb, hdr_len + sizeof(struct tcphdr)))
+ return SKB_DROP_REASON_SKB_BAD_GSO;
- if (skb_header_pointer(skb, hdr_len,
- sizeof(_udphdr), &_udphdr))
- hdr_len += sizeof(struct udphdr);
- }
+ th = (const struct tcphdr *)(skb->data + hdr_len);
+ tlen = __tcp_hdrlen(th);
+ if (tlen < sizeof(*th))
+ return SKB_DROP_REASON_SKB_BAD_GSO;
+ hdr_len += tlen;
+ if (!pskb_may_pull(skb, hdr_len))
+ return SKB_DROP_REASON_SKB_BAD_GSO;
+ } else if (shinfo->gso_type & SKB_GSO_UDP_L4) {
+ if (!pskb_may_pull(skb, hdr_len + sizeof(struct udphdr)))
+ return SKB_DROP_REASON_SKB_BAD_GSO;
+ hdr_len += sizeof(struct udphdr);
+ }
- if (unlikely(shinfo->gso_type & SKB_GSO_DODGY)) {
- int payload = skb->len - hdr_len;
+ /* prior pskb_may_pull() might have changed skb->head. */
+ shinfo = skb_shinfo(skb);
+ if (unlikely(shinfo->gso_type & SKB_GSO_DODGY)) {
+ int payload = skb->len - hdr_len;
- /* Malicious packet. */
- if (payload <= 0)
- return;
- gso_segs = DIV_ROUND_UP(payload, shinfo->gso_size);
- }
- qdisc_skb_cb(skb)->pkt_len += (gso_segs - 1) * hdr_len;
+ /* Malicious packet. */
+ if (payload <= 0)
+ return SKB_DROP_REASON_SKB_BAD_GSO;
+ gso_segs = DIV_ROUND_UP(payload, shinfo->gso_size);
}
+ qdisc_skb_cb(skb)->pkt_len += (gso_segs - 1) * hdr_len;
+ return SKB_NOT_DROPPED_YET;
}
static int dev_qdisc_enqueue(struct sk_buff *skb, struct Qdisc *q,
@@ -4485,6 +4498,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
struct Qdisc *q;
int rc = -ENOMEM;
bool again = false;
+ enum skb_drop_reason reason;
skb_reset_mac_header(skb);
skb_assert_len(skb);
@@ -4492,6 +4506,12 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_SCHED_TSTAMP))
__skb_tstamp_tx(skb, NULL, NULL, skb->sk, SCM_TSTAMP_SCHED);
+ reason = qdisc_pkt_len_init(skb);
+ if (unlikely(reason)) {
+ dev_core_stats_tx_dropped_inc(dev);
+ kfree_skb_reason(skb, reason);
+ return -EINVAL;
+ }
/* Disable soft irqs for various locks below. Also
* stops preemption for RCU.
*/
@@ -4499,7 +4519,6 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
skb_update_prio(skb);
- qdisc_pkt_len_init(skb);
tcx_set_ingress(skb, false);
#ifdef CONFIG_NET_EGRESS
if (static_branch_unlikely(&egress_needed_key)) {
@@ -5022,10 +5041,6 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
int max_backlog;
u32 tail;
- reason = SKB_DROP_REASON_DEV_READY;
- if (!netif_running(skb->dev))
- goto bad_dev;
-
reason = SKB_DROP_REASON_CPU_BACKLOG;
sd = &per_cpu(softnet_data, cpu);
@@ -5036,6 +5051,11 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
backlog_lock_irq_save(sd, &flags);
qlen = skb_queue_len(&sd->input_pkt_queue);
if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) {
+ if (unlikely(!netif_running(skb->dev))) {
+ backlog_unlock_irq_restore(sd, &flags);
+ reason = SKB_DROP_REASON_DEV_READY;
+ goto bad_dev;
+ }
if (!qlen) {
/* Schedule NAPI for backlog device. We can use
* non atomic operation as we own the queue lock.
@@ -9607,6 +9627,37 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode,
return -EBUSY;
}
+ if (prog) {
+ enum bpf_xdp_mode other_mode = mode == XDP_MODE_SKB
+ ? XDP_MODE_DRV : XDP_MODE_SKB;
+ bool offload = mode == XDP_MODE_HW;
+
+ if (!offload && dev_xdp_prog(dev, other_mode)) {
+ NL_SET_ERR_MSG(extack, "Native and generic XDP can't be active at the same time");
+ return -EEXIST;
+ }
+ if (!offload && bpf_prog_is_offloaded(prog->aux)) {
+ NL_SET_ERR_MSG(extack, "Using offloaded program without HW_MODE flag is not supported");
+ return -EINVAL;
+ }
+ if (bpf_prog_is_dev_bound(prog->aux) && !bpf_offload_dev_match(prog, dev)) {
+ NL_SET_ERR_MSG(extack, "Program bound to different device");
+ return -EINVAL;
+ }
+ if (bpf_prog_is_dev_bound(prog->aux) && mode == XDP_MODE_SKB) {
+ NL_SET_ERR_MSG(extack, "Can't attach device-bound programs in generic mode");
+ return -EINVAL;
+ }
+ if (prog->expected_attach_type == BPF_XDP_DEVMAP) {
+ NL_SET_ERR_MSG(extack, "BPF_XDP_DEVMAP programs can not be attached to a device");
+ return -EINVAL;
+ }
+ if (prog->expected_attach_type == BPF_XDP_CPUMAP) {
+ NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device");
+ return -EINVAL;
+ }
+ }
+
memset(&xdp, 0, sizeof(xdp));
xdp.command = mode == XDP_MODE_HW ? XDP_SETUP_PROG_HW : XDP_SETUP_PROG;
xdp.extack = extack;
@@ -10459,7 +10510,8 @@ static int netif_alloc_netdev_queues(struct net_device *dev)
netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
spin_lock_init(&dev->tx_global_lock);
-
+ spin_lock_init(&dev->watchdog_lock);
+ dev->watchdog_ref_held = false;
return 0;
}
@@ -11677,6 +11729,7 @@ EXPORT_SYMBOL(unregister_netdev);
* is already taken in the destination network namespace.
* @new_ifindex: If not zero, specifies device index in the target
* namespace.
+ * @extack: netlink extended ack
*
* This function shuts down a device interface and moves it
* to a new network namespace. On success 0 is returned, on
@@ -11686,9 +11739,10 @@ EXPORT_SYMBOL(unregister_netdev);
*/
int __dev_change_net_namespace(struct net_device *dev, struct net *net,
- const char *pat, int new_ifindex)
+ const char *pat, int new_ifindex,
+ struct netlink_ext_ack *extack)
{
- struct netdev_name_node *name_node;
+ struct netdev_name_node *name_node, *tmp;
struct net *net_old = dev_net(dev);
char new_name[IFNAMSIZ] = {};
int err, new_nsid;
@@ -11697,12 +11751,16 @@ int __dev_change_net_namespace(struct net_device *dev, struct net *net,
/* Don't allow namespace local devices to be moved. */
err = -EINVAL;
- if (dev->netns_local)
+ if (dev->netns_immutable) {
+ NL_SET_ERR_MSG(extack, "The interface netns is immutable");
goto out;
+ }
/* Ensure the device has been registered */
- if (dev->reg_state != NETREG_REGISTERED)
+ if (dev->reg_state != NETREG_REGISTERED) {
+ NL_SET_ERR_MSG(extack, "The interface isn't registered");
goto out;
+ }
/* Get out if there is nothing todo */
err = 0;
@@ -11715,30 +11773,55 @@ int __dev_change_net_namespace(struct net_device *dev, struct net *net,
err = -EEXIST;
if (netdev_name_in_use(net, dev->name)) {
/* We get here if we can't use the current device name */
- if (!pat)
+ if (!pat) {
+ NL_SET_ERR_MSG(extack,
+ "An interface with the same name exists in the target netns");
goto out;
+ }
err = dev_prep_valid_name(net, dev, pat, new_name, EEXIST);
- if (err < 0)
+ if (err < 0) {
+ NL_SET_ERR_MSG_FMT(extack,
+ "Unable to use '%s' for the new interface name in the target netns",
+ pat);
goto out;
+ }
}
/* Check that none of the altnames conflicts. */
err = -EEXIST;
- netdev_for_each_altname(dev, name_node)
- if (netdev_name_in_use(net, name_node->name))
- goto out;
+ netdev_for_each_altname_safe(dev, name_node, tmp) {
+ if (!netdev_name_in_use(net, name_node->name))
+ continue;
+
+ if (!check_net(net_old)) {
+ __netdev_name_node_alt_destroy(name_node);
+ continue;
+ }
+
+ NL_SET_ERR_MSG_FMT(extack,
+ "An interface with the altname %s exists in the target netns",
+ name_node->name);
+ goto out;
+ }
/* Check that new_ifindex isn't used yet. */
if (new_ifindex) {
err = dev_index_reserve(net, new_ifindex);
- if (err < 0)
+ if (err < 0) {
+ NL_SET_ERR_MSG_FMT(extack,
+ "The ifindex %d is not available in the target netns",
+ new_ifindex);
goto out;
+ }
} else {
/* If there is an ifindex conflict assign a new one */
err = dev_index_reserve(net, dev->ifindex);
if (err == -EBUSY)
err = dev_index_reserve(net, 0);
- if (err < 0)
+ if (err < 0) {
+ NL_SET_ERR_MSG(extack,
+ "Unable to allocate a new ifindex in the target netns");
goto out;
+ }
new_ifindex = err;
}
@@ -12155,7 +12238,6 @@ static struct pernet_operations __net_initdata netdev_net_ops = {
static void __net_exit default_device_exit_net(struct net *net)
{
- struct netdev_name_node *name_node, *tmp;
struct net_device *dev, *aux;
/*
* Push all migratable network devices back to the
@@ -12167,7 +12249,7 @@ static void __net_exit default_device_exit_net(struct net *net)
char fb_name[IFNAMSIZ];
/* Ignore unmoveable devices (i.e. loopback) */
- if (dev->netns_local)
+ if (dev->netns_immutable)
continue;
/* Leave virtual devices for the generic cleanup */
@@ -12179,10 +12261,6 @@ static void __net_exit default_device_exit_net(struct net *net)
if (netdev_name_in_use(&init_net, fb_name))
snprintf(fb_name, IFNAMSIZ, "dev%%d");
- netdev_for_each_altname_safe(dev, name_node, tmp)
- if (netdev_name_in_use(&init_net, name_node->name))
- __netdev_name_node_alt_destroy(name_node);
-
err = dev_change_net_namespace(dev, &init_net, fb_name);
if (err) {
pr_emerg("%s: failed to move %s to init_net: %d\n",
diff --git a/net/core/filter.c b/net/core/filter.c
index a680cf27ad82..ce290de4b165 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -7475,7 +7475,7 @@ BPF_CALL_5(bpf_tcp_check_syncookie, struct sock *, sk, void *, iph, u32, iph_len
return -EINVAL;
/* sk_listener() allows TCP_NEW_SYN_RECV, which makes no sense here. */
- if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN)
+ if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP)
return -EINVAL;
if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies))
@@ -7548,7 +7548,7 @@ BPF_CALL_5(bpf_tcp_gen_syncookie, struct sock *, sk, void *, iph, u32, iph_len,
if (unlikely(!sk || th_len < sizeof(*th) || th_len != th->doff * 4))
return -EINVAL;
- if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN)
+ if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP)
return -EINVAL;
if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies))
diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c
index 613b2b07bad9..1b5d151f434e 100644
--- a/net/core/net_namespace.c
+++ b/net/core/net_namespace.c
@@ -164,16 +164,51 @@ static void ops_pre_exit_list(const struct pernet_operations *ops,
}
}
+static void ops_exit_rtnl_list(const struct list_head *ops_list,
+ const struct pernet_operations *ops,
+ struct list_head *net_exit_list)
+{
+ const struct pernet_operations *saved_ops = ops;
+ LIST_HEAD(dev_kill_list);
+ struct net *net;
+
+ rtnl_lock();
+
+ list_for_each_entry(net, net_exit_list, exit_list) {
+ __rtnl_net_lock(net);
+
+ ops = saved_ops;
+ list_for_each_entry_continue_reverse(ops, ops_list, list) {
+ if (ops->exit_rtnl)
+ ops->exit_rtnl(net, &dev_kill_list);
+ }
+
+ __rtnl_net_unlock(net);
+ }
+
+ ops = saved_ops;
+ list_for_each_entry_continue_reverse(ops, ops_list, list) {
+ if (ops->exit_batch_rtnl)
+ ops->exit_batch_rtnl(net_exit_list, &dev_kill_list);
+ }
+
+ unregister_netdevice_many(&dev_kill_list);
+
+ rtnl_unlock();
+}
+
static void ops_exit_list(const struct pernet_operations *ops,
struct list_head *net_exit_list)
{
- struct net *net;
if (ops->exit) {
+ struct net *net;
+
list_for_each_entry(net, net_exit_list, exit_list) {
ops->exit(net);
cond_resched();
}
}
+
if (ops->exit_batch)
ops->exit_batch(net_exit_list);
}
@@ -189,6 +224,54 @@ static void ops_free_list(const struct pernet_operations *ops,
}
}
+static void ops_undo_list(const struct list_head *ops_list,
+ const struct pernet_operations *ops,
+ struct list_head *net_exit_list,
+ bool expedite_rcu, bool hold_rtnl)
+{
+ const struct pernet_operations *saved_ops;
+
+ if (!ops)
+ ops = list_entry(ops_list, typeof(*ops), list);
+
+ saved_ops = ops;
+
+ list_for_each_entry_continue_reverse(ops, ops_list, list)
+ ops_pre_exit_list(ops, net_exit_list);
+
+ /* Another CPU might be rcu-iterating the list, wait for it.
+ * This needs to be before calling the exit() notifiers, so the
+ * rcu_barrier() after ops_undo_list() isn't sufficient alone.
+ * Also the pre_exit() and exit() methods need this barrier.
+ */
+ if (expedite_rcu)
+ synchronize_rcu_expedited();
+ else
+ synchronize_rcu();
+
+ if (hold_rtnl)
+ ops_exit_rtnl_list(ops_list, saved_ops, net_exit_list);
+
+ ops = saved_ops;
+ list_for_each_entry_continue_reverse(ops, ops_list, list)
+ ops_exit_list(ops, net_exit_list);
+
+ ops = saved_ops;
+ list_for_each_entry_continue_reverse(ops, ops_list, list)
+ ops_free_list(ops, net_exit_list);
+}
+
+static void ops_undo_single(struct pernet_operations *ops,
+ struct list_head *net_exit_list)
+{
+ bool hold_rtnl = ops->exit_rtnl || ops->exit_batch_rtnl;
+ LIST_HEAD(ops_list);
+
+ list_add(&ops->list, &ops_list);
+ ops_undo_list(&ops_list, NULL, net_exit_list, false, hold_rtnl);
+ list_del(&ops->list);
+}
+
/* should be called with nsid_lock held */
static int alloc_netid(struct net *net, struct net *peer, int reqid)
{
@@ -349,9 +432,8 @@ static __net_init void preinit_net(struct net *net, struct user_namespace *user_
static __net_init int setup_net(struct net *net)
{
/* Must be called with pernet_ops_rwsem held */
- const struct pernet_operations *ops, *saved_ops;
+ const struct pernet_operations *ops;
LIST_HEAD(net_exit_list);
- LIST_HEAD(dev_kill_list);
int error = 0;
preempt_disable();
@@ -374,29 +456,7 @@ static __net_init int setup_net(struct net *net)
* for the pernet modules whose init functions did not fail.
*/
list_add(&net->exit_list, &net_exit_list);
- saved_ops = ops;
- list_for_each_entry_continue_reverse(ops, &pernet_list, list)
- ops_pre_exit_list(ops, &net_exit_list);
-
- synchronize_rcu();
-
- ops = saved_ops;
- rtnl_lock();
- list_for_each_entry_continue_reverse(ops, &pernet_list, list) {
- if (ops->exit_batch_rtnl)
- ops->exit_batch_rtnl(&net_exit_list, &dev_kill_list);
- }
- unregister_netdevice_many(&dev_kill_list);
- rtnl_unlock();
-
- ops = saved_ops;
- list_for_each_entry_continue_reverse(ops, &pernet_list, list)
- ops_exit_list(ops, &net_exit_list);
-
- ops = saved_ops;
- list_for_each_entry_continue_reverse(ops, &pernet_list, list)
- ops_free_list(ops, &net_exit_list);
-
+ ops_undo_list(&pernet_list, ops, &net_exit_list, false, true);
rcu_barrier();
goto out;
}
@@ -594,11 +654,9 @@ static LLIST_HEAD(cleanup_list);
static void cleanup_net(struct work_struct *work)
{
- const struct pernet_operations *ops;
- struct net *net, *tmp, *last;
struct llist_node *net_kill_list;
+ struct net *net, *tmp, *last;
LIST_HEAD(net_exit_list);
- LIST_HEAD(dev_kill_list);
/* Atomically snapshot the list of namespaces to cleanup */
net_kill_list = llist_del_all(&cleanup_list);
@@ -627,33 +685,7 @@ static void cleanup_net(struct work_struct *work)
list_add_tail(&net->exit_list, &net_exit_list);
}
- /* Run all of the network namespace pre_exit methods */
- list_for_each_entry_reverse(ops, &pernet_list, list)
- ops_pre_exit_list(ops, &net_exit_list);
-
- /*
- * Another CPU might be rcu-iterating the list, wait for it.
- * This needs to be before calling the exit() notifiers, so
- * the rcu_barrier() below isn't sufficient alone.
- * Also the pre_exit() and exit() methods need this barrier.
- */
- synchronize_rcu_expedited();
-
- rtnl_lock();
- list_for_each_entry_reverse(ops, &pernet_list, list) {
- if (ops->exit_batch_rtnl)
- ops->exit_batch_rtnl(&net_exit_list, &dev_kill_list);
- }
- unregister_netdevice_many(&dev_kill_list);
- rtnl_unlock();
-
- /* Run all of the network namespace exit methods */
- list_for_each_entry_reverse(ops, &pernet_list, list)
- ops_exit_list(ops, &net_exit_list);
-
- /* Free the net generic variables */
- list_for_each_entry_reverse(ops, &pernet_list, list)
- ops_free_list(ops, &net_exit_list);
+ ops_undo_list(&pernet_list, NULL, &net_exit_list, true, true);
up_read(&pernet_ops_rwsem);
@@ -1230,31 +1262,13 @@ void __init net_ns_init(void)
RTNL_FLAG_DUMP_UNLOCKED);
}
-static void free_exit_list(struct pernet_operations *ops, struct list_head *net_exit_list)
-{
- ops_pre_exit_list(ops, net_exit_list);
- synchronize_rcu();
-
- if (ops->exit_batch_rtnl) {
- LIST_HEAD(dev_kill_list);
-
- rtnl_lock();
- ops->exit_batch_rtnl(net_exit_list, &dev_kill_list);
- unregister_netdevice_many(&dev_kill_list);
- rtnl_unlock();
- }
- ops_exit_list(ops, net_exit_list);
-
- ops_free_list(ops, net_exit_list);
-}
-
#ifdef CONFIG_NET_NS
static int __register_pernet_operations(struct list_head *list,
struct pernet_operations *ops)
{
+ LIST_HEAD(net_exit_list);
struct net *net;
int error;
- LIST_HEAD(net_exit_list);
list_add_tail(&ops->list, list);
if (ops->init || ops->id) {
@@ -1273,21 +1287,21 @@ static int __register_pernet_operations(struct list_head *list,
out_undo:
/* If I have an error cleanup all namespaces I initialized */
list_del(&ops->list);
- free_exit_list(ops, &net_exit_list);
+ ops_undo_single(ops, &net_exit_list);
return error;
}
static void __unregister_pernet_operations(struct pernet_operations *ops)
{
- struct net *net;
LIST_HEAD(net_exit_list);
+ struct net *net;
- list_del(&ops->list);
/* See comment in __register_pernet_operations() */
for_each_net(net)
list_add_tail(&net->exit_list, &net_exit_list);
- free_exit_list(ops, &net_exit_list);
+ list_del(&ops->list);
+ ops_undo_single(ops, &net_exit_list);
}
#else
@@ -1309,8 +1323,9 @@ static void __unregister_pernet_operations(struct pernet_operations *ops)
list_del(&ops->list);
} else {
LIST_HEAD(net_exit_list);
+
list_add(&init_net.exit_list, &net_exit_list);
- free_exit_list(ops, &net_exit_list);
+ ops_undo_single(ops, &net_exit_list);
}
}
diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index 23b0e673fd83..159a6270d5fb 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -2932,7 +2932,7 @@ static int do_setlink(const struct sk_buff *skb,
else
new_ifindex = 0;
- err = __dev_change_net_namespace(dev, net, pat, new_ifindex);
+ err = __dev_change_net_namespace(dev, net, pat, new_ifindex, extack);
put_net(net);
if (err)
goto errout;
diff --git a/net/core/sock.c b/net/core/sock.c
index 75db38fdfd35..181a66488ec2 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2411,6 +2411,9 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority)
sock_copy(newsk, sk);
newsk->sk_prot_creator = prot;
+#ifdef CONFIG_BPF_SYSCALL
+ RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
+#endif
/* SANITY */
if (likely(newsk->sk_net_refcnt)) {
diff --git a/net/ethtool/module.c b/net/ethtool/module.c
index 5a08c320b466..e01275103bcd 100644
--- a/net/ethtool/module.c
+++ b/net/ethtool/module.c
@@ -425,10 +425,11 @@ int ethnl_act_module_fw_flash(struct sk_buff *skb, struct genl_info *info)
ret = ethnl_module_fw_flash_validate(dev, info->extack);
if (ret < 0)
- goto out_rtnl;
+ goto out_complete;
ret = module_flash_fw(dev, tb, skb, info);
+out_complete:
ethnl_ops_complete(dev);
out_rtnl:
diff --git a/net/ethtool/rss.c b/net/ethtool/rss.c
index 3570d58c5cca..b543827ef373 100644
--- a/net/ethtool/rss.c
+++ b/net/ethtool/rss.c
@@ -88,8 +88,10 @@ rss_prepare_get(const struct rss_req_info *request, struct net_device *dev,
rxfh.key = data->hkey;
ret = ops->get_rxfh(dev, &rxfh);
- if (ret)
+ if (ret) {
+ kfree(rss_config);
goto out_ops;
+ }
data->hfunc = rxfh.hfunc;
data->input_xfrm = rxfh.input_xfrm;
diff --git a/net/handshake/handshake-test.c b/net/handshake/handshake-test.c
index a331b308aaa2..d66b133f59d1 100644
--- a/net/handshake/handshake-test.c
+++ b/net/handshake/handshake-test.c
@@ -375,6 +375,10 @@ static void handshake_req_cancel_test2(struct kunit *test)
/* Pretend to accept this request */
next = handshake_req_next(hn, HANDSHAKE_HANDLER_CLASS_TLSHD);
KUNIT_ASSERT_PTR_EQ(test, req, next);
+ /* Simulate FD_PREPARE() consuming the file reference handed
+ * off by handshake_req_next(); see handshake_nl_accept_doit().
+ */
+ fput(filp);
/* Act */
result = handshake_req_cancel(sock->sk);
@@ -417,6 +421,10 @@ static void handshake_req_cancel_test3(struct kunit *test)
/* Pretend to accept this request */
next = handshake_req_next(hn, HANDSHAKE_HANDLER_CLASS_TLSHD);
KUNIT_ASSERT_PTR_EQ(test, req, next);
+ /* Simulate FD_PREPARE() consuming the file reference handed
+ * off by handshake_req_next(); see handshake_nl_accept_doit().
+ */
+ fput(filp);
/* Pretend to complete this request */
handshake_complete(next, -ETIMEDOUT, NULL);
diff --git a/net/handshake/netlink.c b/net/handshake/netlink.c
index e49041cc0f9d..d36f069549f9 100644
--- a/net/handshake/netlink.c
+++ b/net/handshake/netlink.c
@@ -92,7 +92,6 @@ int handshake_nl_accept_doit(struct sk_buff *skb, struct genl_info *info)
struct net *net = sock_net(skb->sk);
struct handshake_net *hn = handshake_pernet(net);
struct handshake_req *req = NULL;
- struct socket *sock;
int class, fd, err;
err = -EOPNOTSUPP;
@@ -109,9 +108,9 @@ int handshake_nl_accept_doit(struct sk_buff *skb, struct genl_info *info)
if (!req)
goto out_status;
- sock = req->hr_sk->sk_socket;
fd = get_unused_fd_flags(O_CLOEXEC);
if (fd < 0) {
+ fput(req->hr_file); /* drop ref from handshake_req_next() */
err = fd;
goto out_complete;
}
@@ -119,10 +118,11 @@ int handshake_nl_accept_doit(struct sk_buff *skb, struct genl_info *info)
err = req->hr_proto->hp_accept(req, info, fd);
if (err) {
put_unused_fd(fd);
+ fput(req->hr_file); /* drop ref from handshake_req_next() */
goto out_complete;
}
- fd_install(fd, get_file(sock->file));
+ fd_install(fd, req->hr_file);
trace_handshake_cmd_accept(net, req, req->hr_sk, fd);
return 0;
diff --git a/net/handshake/request.c b/net/handshake/request.c
index 96f80e0df67b..670e0f979382 100644
--- a/net/handshake/request.c
+++ b/net/handshake/request.c
@@ -181,6 +181,17 @@ static bool remove_pending(struct handshake_net *hn, struct handshake_req *req)
return ret;
}
+/**
+ * handshake_req_next - Return the next queued handshake request
+ * @hn: per-net handshake state
+ * @class: handler class to match
+ *
+ * On a non-NULL return, the caller owns an extra reference
+ * on @req->hr_file. FD_PREPARE() consumes it on success; on
+ * the FD_PREPARE() failure path the caller must fput() it.
+ *
+ * Return: pointer to a removed handshake_req, or NULL.
+ */
struct handshake_req *handshake_req_next(struct handshake_net *hn, int class)
{
struct handshake_req *req, *pos;
@@ -191,6 +202,13 @@ struct handshake_req *handshake_req_next(struct handshake_net *hn, int class)
if (pos->hr_proto->hp_handler_class != class)
continue;
__remove_pending_locked(hn, pos);
+ /* Hand off a file reference to the accept side under
+ * hn_lock. A concurrent handshake_req_cancel() can drop
+ * hr_file before accept reaches FD_PREPARE(); this extra
+ * reference keeps the file alive until FD_PREPARE() takes
+ * ownership.
+ */
+ get_file(pos->hr_file);
req = pos;
break;
}
diff --git a/net/hsr/hsr_device.c b/net/hsr/hsr_device.c
index 988d35866d04..3ef9e8e8f3a9 100644
--- a/net/hsr/hsr_device.c
+++ b/net/hsr/hsr_device.c
@@ -653,7 +653,7 @@ void hsr_dev_setup(struct net_device *dev)
/* Not sure about this. Taken from bridge code. netdevice.h says
* it means "Does not change network namespaces".
*/
- dev->netns_local = true;
+ dev->netns_immutable = true;
dev->needs_free_netdev = true;
diff --git a/net/ieee802154/6lowpan/core.c b/net/ieee802154/6lowpan/core.c
index b200e1a03c17..75ad8638c2d7 100644
--- a/net/ieee802154/6lowpan/core.c
+++ b/net/ieee802154/6lowpan/core.c
@@ -116,7 +116,7 @@ static void lowpan_setup(struct net_device *ldev)
ldev->netdev_ops = &lowpan_netdev_ops;
ldev->header_ops = &lowpan_header_ops;
ldev->needs_free_netdev = true;
- ldev->netns_local = true;
+ ldev->netns_immutable = true;
}
static int lowpan_validate(struct nlattr *tb[], struct nlattr *data[],
diff --git a/net/ieee802154/core.c b/net/ieee802154/core.c
index 4a4379c828b9..edfbd57aebc2 100644
--- a/net/ieee802154/core.c
+++ b/net/ieee802154/core.c
@@ -226,11 +226,11 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev,
list_for_each_entry(wpan_dev, &rdev->wpan_dev_list, list) {
if (!wpan_dev->netdev)
continue;
- wpan_dev->netdev->netns_local = false;
+ wpan_dev->netdev->netns_immutable = false;
err = dev_change_net_namespace(wpan_dev->netdev, net, "wpan%d");
if (err)
break;
- wpan_dev->netdev->netns_local = true;
+ wpan_dev->netdev->netns_immutable = true;
}
if (err) {
@@ -242,11 +242,11 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev,
list) {
if (!wpan_dev->netdev)
continue;
- wpan_dev->netdev->netns_local = false;
+ wpan_dev->netdev->netns_immutable = false;
err = dev_change_net_namespace(wpan_dev->netdev, net,
"wpan%d");
- WARN_ON(err);
- wpan_dev->netdev->netns_local = true;
+ WARN_ON(err && err != -ENOMEM);
+ wpan_dev->netdev->netns_immutable = true;
}
return err;
@@ -255,7 +255,7 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev,
wpan_phy_net_set(&rdev->wpan_phy, net);
err = device_rename(&rdev->wpan_phy.dev, dev_name(&rdev->wpan_phy.dev));
- WARN_ON(err);
+ WARN_ON(err && err != -ENOMEM);
return 0;
}
@@ -291,7 +291,7 @@ static int cfg802154_netdev_notifier_call(struct notifier_block *nb,
switch (state) {
/* TODO NETDEV_DEVTYPE */
case NETDEV_REGISTER:
- dev->netns_local = true;
+ dev->netns_immutable = true;
wpan_dev->identifier = ++rdev->wpan_dev_id;
list_add_rcu(&wpan_dev->list, &rdev->wpan_dev_list);
rdev->devlist_generation++;
diff --git a/net/ife/ife.c b/net/ife/ife.c
index 7a75947a31e3..2ddf725d3389 100644
--- a/net/ife/ife.c
+++ b/net/ife/ife.c
@@ -37,7 +37,7 @@ void *ife_encode(struct sk_buff *skb, u16 metalen)
* where ORIGDATA = original ethernet header ...
*/
int hdrm = metalen + IFE_METAHDRLEN;
- int total_push = hdrm + skb->dev->hard_header_len;
+ int total_push = hdrm + ETH_HLEN;
struct ifeheadr *ifehdr;
struct ethhdr *iethh; /* inner ether header */
int skboff = 0;
@@ -50,9 +50,9 @@ void *ife_encode(struct sk_buff *skb, u16 metalen)
iethh = (struct ethhdr *) skb->data;
__skb_push(skb, total_push);
- memcpy(skb->data, iethh, skb->dev->hard_header_len);
+ memcpy(skb->data, iethh, ETH_HLEN);
skb_reset_mac_header(skb);
- skboff += skb->dev->hard_header_len;
+ skboff += ETH_HLEN;
/* total metadata length */
ifehdr = (struct ifeheadr *) (skb->data + skboff);
@@ -69,12 +69,12 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen)
int total_pull;
u16 ifehdrln;
- if (!pskb_may_pull(skb, skb->dev->hard_header_len + IFE_METAHDRLEN))
+ if (!pskb_may_pull(skb, ETH_HLEN + IFE_METAHDRLEN))
return NULL;
- ifehdr = (struct ifeheadr *) (skb->data + skb->dev->hard_header_len);
+ ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN);
ifehdrln = ntohs(ifehdr->metalen);
- total_pull = skb->dev->hard_header_len + ifehdrln;
+ total_pull = ETH_HLEN + ifehdrln;
if (unlikely(ifehdrln < 2))
return NULL;
@@ -82,7 +82,7 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen)
if (unlikely(!pskb_may_pull(skb, total_pull + ETH_HLEN)))
return NULL;
- ifehdr = (struct ifeheadr *)(skb->data + skb->dev->hard_header_len);
+ ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN);
skb_set_mac_header(skb, total_pull);
__skb_pull(skb, total_pull);
*metalen = ifehdrln - IFE_METAHDRLEN;
diff --git a/net/ipv4/igmp.c b/net/ipv4/igmp.c
index 3f503d60a7db..e827537ec15b 100644
--- a/net/ipv4/igmp.c
+++ b/net/ipv4/igmp.c
@@ -220,13 +220,18 @@ static void ip_sf_list_clear_all(struct ip_sf_list *psf)
static void igmp_stop_timer(struct ip_mc_list *im)
{
+ bool put = false;
+
spin_lock_bh(&im->lock);
if (del_timer(&im->timer))
- refcount_dec(&im->refcnt);
+ put = true;
im->tm_running = 0;
im->reporter = 0;
im->unsolicit_count = 0;
spin_unlock_bh(&im->lock);
+
+ if (put)
+ ip_ma_put(im);
}
/* It must be called with locked im->lock */
@@ -269,6 +274,8 @@ static void igmp_ifc_start_timer(struct in_device *in_dev, int delay)
static void igmp_mod_timer(struct ip_mc_list *im, int max_delay)
{
+ bool put = false;
+
spin_lock_bh(&im->lock);
im->unsolicit_count = 0;
if (del_timer(&im->timer)) {
@@ -278,10 +285,13 @@ static void igmp_mod_timer(struct ip_mc_list *im, int max_delay)
spin_unlock_bh(&im->lock);
return;
}
- refcount_dec(&im->refcnt);
+ put = true;
}
igmp_start_timer(im, max_delay);
spin_unlock_bh(&im->lock);
+
+ if (put)
+ ip_ma_put(im);
}
diff --git a/net/ipv4/ip_tunnel.c b/net/ipv4/ip_tunnel.c
index 87cc6dca894c..ed7bcd70f0c8 100644
--- a/net/ipv4/ip_tunnel.c
+++ b/net/ipv4/ip_tunnel.c
@@ -1148,7 +1148,7 @@ int ip_tunnel_init_net(struct net *net, unsigned int ip_tnl_net_id,
* Allowing to move it to another netns is clearly unsafe.
*/
if (!IS_ERR(itn->fb_tunnel_dev)) {
- itn->fb_tunnel_dev->netns_local = true;
+ itn->fb_tunnel_dev->netns_immutable = true;
itn->fb_tunnel_dev->mtu = ip_tunnel_bind_dev(itn->fb_tunnel_dev);
ip_tunnel_add(itn, netdev_priv(itn->fb_tunnel_dev));
itn->type = itn->fb_tunnel_dev->type;
diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c
index e37853d70db6..c3aa02536251 100644
--- a/net/ipv4/ipmr.c
+++ b/net/ipv4/ipmr.c
@@ -549,7 +549,7 @@ static void reg_vif_setup(struct net_device *dev)
dev->flags = IFF_NOARP;
dev->netdev_ops = ®_vif_netdev_ops;
dev->needs_free_netdev = true;
- dev->netns_local = true;
+ dev->netns_immutable = true;
}
static struct net_device *ipmr_reg_vif(struct net *net, struct mr_table *mrt)
diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c
index b752c9eac998..c68ffde6d615 100644
--- a/net/ipv4/netfilter/arp_tables.c
+++ b/net/ipv4/netfilter/arp_tables.c
@@ -1519,13 +1519,11 @@ int arpt_register_table(struct net *net,
const struct arpt_replace *repl,
const struct nf_hook_ops *template_ops)
{
- struct nf_hook_ops *ops;
- unsigned int num_ops;
- int ret, i;
- struct xt_table_info *newinfo;
struct xt_table_info bootstrap = {0};
- void *loc_cpu_entry;
+ struct xt_table_info *newinfo;
struct xt_table *new_table;
+ void *loc_cpu_entry;
+ int ret;
newinfo = xt_alloc_table_info(repl->size);
if (!newinfo)
@@ -1540,7 +1538,7 @@ int arpt_register_table(struct net *net,
return ret;
}
- new_table = xt_register_table(net, table, &bootstrap, newinfo);
+ new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo);
if (IS_ERR(new_table)) {
struct arpt_entry *iter;
@@ -1550,31 +1548,6 @@ int arpt_register_table(struct net *net,
return PTR_ERR(new_table);
}
- num_ops = hweight32(table->valid_hooks);
- if (num_ops == 0) {
- ret = -EINVAL;
- goto out_free;
- }
-
- ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
- if (!ops) {
- ret = -ENOMEM;
- goto out_free;
- }
-
- for (i = 0; i < num_ops; i++)
- ops[i].priv = new_table;
-
- new_table->ops = ops;
-
- ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret != 0)
- goto out_free;
-
- return ret;
-
-out_free:
- __arpt_unregister_table(net, new_table);
return ret;
}
diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index 0ba456c4c634..2411ef91d296 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -1721,13 +1721,11 @@ int ipt_register_table(struct net *net, const struct xt_table *table,
const struct ipt_replace *repl,
const struct nf_hook_ops *template_ops)
{
- struct nf_hook_ops *ops;
- unsigned int num_ops;
- int ret, i;
- struct xt_table_info *newinfo;
struct xt_table_info bootstrap = {0};
- void *loc_cpu_entry;
+ struct xt_table_info *newinfo;
struct xt_table *new_table;
+ void *loc_cpu_entry;
+ int ret;
newinfo = xt_alloc_table_info(repl->size);
if (!newinfo)
@@ -1742,7 +1740,7 @@ int ipt_register_table(struct net *net, const struct xt_table *table,
return ret;
}
- new_table = xt_register_table(net, table, &bootstrap, newinfo);
+ new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo);
if (IS_ERR(new_table)) {
struct ipt_entry *iter;
@@ -1752,37 +1750,6 @@ int ipt_register_table(struct net *net, const struct xt_table *table,
return PTR_ERR(new_table);
}
- /* No template? No need to do anything. This is used by 'nat' table, it registers
- * with the nat core instead of the netfilter core.
- */
- if (!template_ops)
- return 0;
-
- num_ops = hweight32(table->valid_hooks);
- if (num_ops == 0) {
- ret = -EINVAL;
- goto out_free;
- }
-
- ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
- if (!ops) {
- ret = -ENOMEM;
- goto out_free;
- }
-
- for (i = 0; i < num_ops; i++)
- ops[i].priv = new_table;
-
- new_table->ops = ops;
-
- ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret != 0)
- goto out_free;
-
- return ret;
-
-out_free:
- __ipt_unregister_table(net, new_table);
return ret;
}
diff --git a/net/ipv4/netfilter/iptable_nat.c b/net/ipv4/netfilter/iptable_nat.c
index a5db7c67d61b..625a1ca13b1b 100644
--- a/net/ipv4/netfilter/iptable_nat.c
+++ b/net/ipv4/netfilter/iptable_nat.c
@@ -79,7 +79,7 @@ static int ipt_nat_register_lookups(struct net *net)
while (i)
nf_nat_ipv4_unregister_fn(net, &ops[--i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
return ret;
}
}
@@ -100,7 +100,7 @@ static void ipt_nat_unregister_lookups(struct net *net)
for (i = 0; i < ARRAY_SIZE(nf_nat_ipv4_ops); i++)
nf_nat_ipv4_unregister_fn(net, &ops[i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
}
static int iptable_nat_table_init(struct net *net)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 21c64ca9bb9b..7df7ce86b690 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -297,9 +297,6 @@ enum {
DEFINE_PER_CPU(unsigned int, tcp_orphan_count);
EXPORT_PER_CPU_SYMBOL_GPL(tcp_orphan_count);
-DEFINE_PER_CPU(u32, tcp_tw_isn);
-EXPORT_PER_CPU_SYMBOL_GPL(tcp_tw_isn);
-
long sysctl_tcp_mem[3] __read_mostly;
EXPORT_IPV6_MOD(sysctl_tcp_mem);
@@ -3163,7 +3160,7 @@ void __tcp_close(struct sock *sk, long timeout)
/* remove backlog if any, without releasing ownership. */
__release_sock(sk);
- this_cpu_inc(tcp_orphan_count);
+ tcp_orphan_count_inc();
/* Have we already been destroyed by a softirq or backlog? */
if (state != TCP_CLOSE && sk->sk_state == TCP_CLOSE)
diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c
index 0e7e12117913..4d3315ff4522 100644
--- a/net/ipv4/tcp_bpf.c
+++ b/net/ipv4/tcp_bpf.c
@@ -335,6 +335,7 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk,
static int tcp_bpf_ioctl(struct sock *sk, int cmd, int *karg)
{
+ struct sk_psock *psock;
bool slow;
if (cmd != SIOCINQ)
@@ -345,7 +346,21 @@ static int tcp_bpf_ioctl(struct sock *sk, int cmd, int *karg)
return -EINVAL;
slow = lock_sock_fast(sk);
- *karg = sk_psock_msg_inq(sk);
+ psock = sk_psock_get(sk);
+ if (unlikely(!psock)) {
+ unlock_sock_fast(sk, slow);
+ return tcp_ioctl(sk, cmd, karg);
+ }
+ *karg = sk_psock_get_msg_len_nolock(psock);
+ /* Without a verdict program, ingress data is never diverted to
+ * ingress_msg: it stays in sk_receive_queue and is read through
+ * the fallback to tcp_recvmsg(), so account for it like
+ * tcp_ioctl() does.
+ */
+ if (!READ_ONCE(psock->progs.stream_verdict) &&
+ !READ_ONCE(psock->progs.skb_verdict))
+ *karg += tcp_inq(sk);
+ sk_psock_put(sk, psock);
unlock_sock_fast(sk, slow);
return 0;
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 5c581d4821c1..96eb6d242141 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -7242,6 +7242,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
struct sock *sk, struct sk_buff *skb)
{
struct tcp_fastopen_cookie foc = { .len = -1 };
+ u32 isn = TCP_SKB_CB(skb)->tcp_tw_isn;
struct tcp_options_received tmp_opt;
struct tcp_sock *tp = tcp_sk(sk);
struct net *net = sock_net(sk);
@@ -7252,20 +7253,16 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
struct dst_entry *dst;
struct flowi fl;
u8 syncookies;
- u32 isn;
#ifdef CONFIG_TCP_AO
const struct tcp_ao_hdr *aoh;
#endif
- isn = __this_cpu_read(tcp_tw_isn);
- if (isn) {
- /* TW buckets are converted to open requests without
- * limitations, they conserve resources and peer is
- * evidently real one.
- */
- __this_cpu_write(tcp_tw_isn, 0);
- } else {
+ /* If isn is non-zero, this SYN originally matched a TIME_WAIT socket.
+ * TW sockets are converted to open requests without limitations,
+ * we skip the queue limits and syncookie checks in the block below.
+ */
+ if (!isn) {
syncookies = READ_ONCE(net->ipv4.sysctl_tcp_syncookies);
if (syncookies == 2 || inet_csk_reqsk_queue_is_full(sk)) {
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 48af4b0d77f2..c32ec598930b 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -2311,6 +2311,7 @@ int tcp_v4_rcv(struct sk_buff *skb)
}
}
+ isn = 0;
process:
if (static_branch_unlikely(&ip4_min_ttl)) {
/* min_ttl can be changed concurrently from do_ip_setsockopt() */
@@ -2340,6 +2341,7 @@ int tcp_v4_rcv(struct sk_buff *skb)
th = (const struct tcphdr *)skb->data;
iph = ip_hdr(skb);
tcp_v4_fill_cb(skb, iph, th);
+ TCP_SKB_CB(skb)->tcp_tw_isn = isn;
skb->dev = NULL;
@@ -2424,7 +2426,6 @@ int tcp_v4_rcv(struct sk_buff *skb)
sk = sk2;
tcp_v4_restore_cb(skb);
refcounted = false;
- __this_cpu_write(tcp_tw_isn, isn);
goto process;
}
}
diff --git a/net/ipv4/udp_bpf.c b/net/ipv4/udp_bpf.c
index 779a3a03762f..b07774034a91 100644
--- a/net/ipv4/udp_bpf.c
+++ b/net/ipv4/udp_bpf.c
@@ -52,7 +52,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
ret = udp_msg_has_data(sk, psock);
if (!ret) {
+ release_sock(sk);
wait_woken(&wait, TASK_INTERRUPTIBLE, timeo);
+ lock_sock(sk);
ret = udp_msg_has_data(sk, psock);
}
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -81,6 +83,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
goto out;
}
+ lock_sock(sk);
msg_bytes_ready:
copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
if (!copied) {
@@ -92,11 +95,17 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
if (data) {
if (psock_has_data(psock))
goto msg_bytes_ready;
+
+ release_sock(sk);
+
ret = sk_udp_recvmsg(sk, msg, len, flags, addr_len);
goto out;
}
copied = -EAGAIN;
}
+
+ release_sock(sk);
+
ret = copied;
out:
sk_psock_put(sk, psock);
diff --git a/net/ipv6/ioam6.c b/net/ipv6/ioam6.c
index bea2cf016307..04e02ec689b9 100644
--- a/net/ipv6/ioam6.c
+++ b/net/ipv6/ioam6.c
@@ -714,7 +714,9 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
struct ioam6_schema *sc,
unsigned int sclen, bool is_input)
{
- struct net_device *dev = skb_dst_dev(skb);
+ /* Note: skb_dst_dev_rcu() can't be NULL at this point. */
+ struct net_device *dev = skb_dst_dev_rcu(skb);
+ struct inet6_dev *i_skb_dev, *idev;
struct timespec64 ts;
ktime_t tstamp;
u64 raw64;
@@ -725,13 +727,16 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
data = trace->data + trace->remlen * 4 - trace->nodelen * 4 - sclen * 4;
+ i_skb_dev = skb->dev ? __in6_dev_get(skb->dev) : NULL;
+ idev = __in6_dev_get(dev);
+
/* hop_lim and node_id */
if (trace->type.bit0) {
byte = ipv6_hdr(skb)->hop_limit;
if (is_input)
byte--;
- raw32 = dev_net(dev)->ipv6.sysctl.ioam6_id;
+ raw32 = READ_ONCE(dev_net(dev)->ipv6.sysctl.ioam6_id);
*(__be32 *)data = cpu_to_be32((byte << 24) | raw32);
data += sizeof(__be32);
@@ -739,18 +744,18 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
/* ingress_if_id and egress_if_id */
if (trace->type.bit1) {
- if (!skb->dev)
+ if (!i_skb_dev)
raw16 = IOAM6_U16_UNAVAILABLE;
else
- raw16 = (__force u16)READ_ONCE(__in6_dev_get(skb->dev)->cnf.ioam6_id);
+ raw16 = (__force u16)READ_ONCE(i_skb_dev->cnf.ioam6_id);
*(__be16 *)data = cpu_to_be16(raw16);
data += sizeof(__be16);
- if (dev->flags & IFF_LOOPBACK)
+ if ((dev->flags & IFF_LOOPBACK) || !idev)
raw16 = IOAM6_U16_UNAVAILABLE;
else
- raw16 = (__force u16)READ_ONCE(__in6_dev_get(dev)->cnf.ioam6_id);
+ raw16 = (__force u16)READ_ONCE(idev->cnf.ioam6_id);
*(__be16 *)data = cpu_to_be16(raw16);
data += sizeof(__be16);
@@ -802,12 +807,16 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
struct Qdisc *qdisc;
__u32 qlen, backlog;
- if (dev->flags & IFF_LOOPBACK) {
+ if (dev->flags & IFF_LOOPBACK ||
+ skb_get_queue_mapping(skb) >= dev->num_tx_queues) {
*(__be32 *)data = cpu_to_be32(IOAM6_U32_UNAVAILABLE);
} else {
queue = skb_get_tx_queue(dev, skb);
qdisc = rcu_dereference(queue->qdisc);
+
+ spin_lock_bh(qdisc_lock(qdisc));
qdisc_qstats_qlen_backlog(qdisc, &qlen, &backlog);
+ spin_unlock_bh(qdisc_lock(qdisc));
*(__be32 *)data = cpu_to_be32(backlog);
}
@@ -826,7 +835,7 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
if (is_input)
byte--;
- raw64 = dev_net(dev)->ipv6.sysctl.ioam6_id_wide;
+ raw64 = READ_ONCE(dev_net(dev)->ipv6.sysctl.ioam6_id_wide);
*(__be64 *)data = cpu_to_be64(((u64)byte << 56) | raw64);
data += sizeof(__be64);
@@ -834,18 +843,18 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
/* ingress_if_id and egress_if_id (wide) */
if (trace->type.bit9) {
- if (!skb->dev)
+ if (!i_skb_dev)
raw32 = IOAM6_U32_UNAVAILABLE;
else
- raw32 = READ_ONCE(__in6_dev_get(skb->dev)->cnf.ioam6_id_wide);
+ raw32 = READ_ONCE(i_skb_dev->cnf.ioam6_id_wide);
*(__be32 *)data = cpu_to_be32(raw32);
data += sizeof(__be32);
- if (dev->flags & IFF_LOOPBACK)
+ if ((dev->flags & IFF_LOOPBACK) || !idev)
raw32 = IOAM6_U32_UNAVAILABLE;
else
- raw32 = READ_ONCE(__in6_dev_get(dev)->cnf.ioam6_id_wide);
+ raw32 = READ_ONCE(idev->cnf.ioam6_id_wide);
*(__be32 *)data = cpu_to_be32(raw32);
data += sizeof(__be32);
diff --git a/net/ipv6/ip6_gre.c b/net/ipv6/ip6_gre.c
index ce1ba9d2af40..b737731dbcba 100644
--- a/net/ipv6/ip6_gre.c
+++ b/net/ipv6/ip6_gre.c
@@ -951,7 +951,6 @@ static netdev_tx_t ip6erspan_tunnel_xmit(struct sk_buff *skb,
{
struct ip_tunnel_info *tun_info = NULL;
struct ip6_tnl *t = netdev_priv(dev);
- struct dst_entry *dst = skb_dst(skb);
IP_TUNNEL_DECLARE_FLAGS(flags) = { };
bool truncate = false;
int encap_limit = -1;
@@ -1087,12 +1086,6 @@ static netdev_tx_t ip6erspan_tunnel_xmit(struct sk_buff *skb,
gre_build_header(skb, 8, flags, proto, 0,
htonl(atomic_fetch_inc(&t->o_seqno)));
- /* TooBig packet may have updated dst->dev's mtu */
- if (!t->parms.collect_md && dst) {
- mtu = READ_ONCE(dst_dev(dst)->mtu);
- if (dst6_mtu(dst) > mtu)
- dst->ops->update_pmtu(dst, NULL, skb, mtu, false);
- }
err = ip6_tnl_xmit(skb, dev, dsfield, &fl6, encap_limit, &mtu,
NEXTHDR_GRE);
if (err != 0) {
@@ -1631,7 +1624,7 @@ static int __net_init ip6gre_init_net(struct net *net)
/* FB netdevice is special: we have one, and only one per netns.
* Allowing to move it to another netns is clearly unsafe.
*/
- ign->fb_tunnel_dev->netns_local = true;
+ ign->fb_tunnel_dev->netns_immutable = true;
ip6gre_fb_tunnel_init(ign->fb_tunnel_dev);
ign->fb_tunnel_dev->rtnl_link_ops = &ip6gre_link_ops;
diff --git a/net/ipv6/ip6_tunnel.c b/net/ipv6/ip6_tunnel.c
index 7eb2e67842f4..4053ba5b1a13 100644
--- a/net/ipv6/ip6_tunnel.c
+++ b/net/ipv6/ip6_tunnel.c
@@ -2264,7 +2264,7 @@ static int __net_init ip6_tnl_init_net(struct net *net)
/* FB netdevice is special: we have one, and only one per netns.
* Allowing to move it to another netns is clearly unsafe.
*/
- ip6n->fb_tnl_dev->netns_local = true;
+ ip6n->fb_tnl_dev->netns_immutable = true;
err = ip6_fb_tnl_dev_init(ip6n->fb_tnl_dev);
if (err < 0)
diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
index 3880662b6bc3..a5ad9fb38966 100644
--- a/net/ipv6/ip6_vti.c
+++ b/net/ipv6/ip6_vti.c
@@ -1162,7 +1162,7 @@ static int __net_init vti6_init_net(struct net *net)
goto err_alloc_dev;
dev_net_set(ip6n->fb_tnl_dev, net);
ip6n->fb_tnl_dev->rtnl_link_ops = &vti6_link_ops;
- ip6n->fb_tnl_dev->netns_local = true;
+ ip6n->fb_tnl_dev->netns_immutable = true;
err = vti6_fb_tnl_dev_init(ip6n->fb_tnl_dev);
if (err < 0)
diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c
index ac0120b75ff2..47afc8cef688 100644
--- a/net/ipv6/ip6mr.c
+++ b/net/ipv6/ip6mr.c
@@ -654,7 +654,7 @@ static void reg_vif_setup(struct net_device *dev)
dev->flags = IFF_NOARP;
dev->netdev_ops = ®_vif_netdev_ops;
dev->needs_free_netdev = true;
- dev->netns_local = true;
+ dev->netns_immutable = true;
}
static struct net_device *ip6mr_reg_vif(struct net *net, struct mr_table *mrt)
@@ -1128,7 +1128,6 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt,
msg->im6_src = ipv6_hdr(pkt)->saddr;
msg->im6_dst = ipv6_hdr(pkt)->daddr;
- skb_dst_set(skb, dst_clone(skb_dst(pkt)));
skb->ip_summed = CHECKSUM_UNNECESSARY;
}
@@ -1138,6 +1137,7 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt,
return -EINVAL;
}
+ skb_dst_drop(skb);
mrt6msg_netlink_event(mrt, skb);
/* Deliver to user space multicast routing algorithms */
@@ -1217,6 +1217,7 @@ static int ip6mr_cache_unresolved(struct mr_table *mrt, mifi_t mifi,
skb->dev = dev;
skb->skb_iif = dev->ifindex;
}
+ skb_dst_drop(skb);
skb_queue_tail(&c->_c.mfc_un.unres.unresolved, skb);
err = 0;
}
diff --git a/net/ipv6/mcast.c b/net/ipv6/mcast.c
index d0eefadea5c3..9326dd3976bf 100644
--- a/net/ipv6/mcast.c
+++ b/net/ipv6/mcast.c
@@ -229,7 +229,8 @@ static int __ipv6_sock_mc_join(struct sock *sk, int ifindex,
return err;
}
- mc_lst->next = np->ipv6_mc_list;
+ rcu_assign_pointer(mc_lst->next,
+ sock_dereference(np->ipv6_mc_list, sk));
rcu_assign_pointer(np->ipv6_mc_list, mc_lst);
return 0;
@@ -289,7 +290,8 @@ int ipv6_sock_mc_drop(struct sock *sk, int ifindex, const struct in6_addr *addr)
lnk = &mc_lst->next) {
if ((ifindex == 0 || mc_lst->ifindex == ifindex) &&
ipv6_addr_equal(&mc_lst->addr, addr)) {
- *lnk = mc_lst->next;
+ rcu_assign_pointer(*lnk,
+ sock_dereference(mc_lst->next, sk));
__ipv6_sock_mc_drop(sk, mc_lst);
return 0;
}
@@ -343,7 +345,8 @@ void __ipv6_sock_mc_close(struct sock *sk)
while ((mc_lst = sock_dereference(np->ipv6_mc_list, sk)) != NULL) {
struct net_device *dev;
- np->ipv6_mc_list = mc_lst->next;
+ rcu_assign_pointer(np->ipv6_mc_list,
+ sock_dereference(mc_lst->next, sk));
dev = __dev_get_by_index(net, mc_lst->ifindex);
if (dev) {
@@ -826,9 +829,11 @@ static void mld_del_delrec(struct inet6_dev *idev, struct ifmcaddr6 *im)
}
if (pmc) {
if (pmc_prev)
- rcu_assign_pointer(pmc_prev->next, pmc->next);
+ rcu_assign_pointer(pmc_prev->next,
+ mc_dereference(pmc->next, idev));
else
- rcu_assign_pointer(idev->mc_tomb, pmc->next);
+ rcu_assign_pointer(idev->mc_tomb,
+ mc_dereference(pmc->next, idev));
}
if (pmc) {
@@ -986,7 +991,7 @@ static int __ipv6_dev_mc_inc(struct net_device *dev,
return -ENOMEM;
}
- rcu_assign_pointer(mc->next, idev->mc_list);
+ rcu_assign_pointer(mc->next, mc_dereference(idev->mc_list, idev));
rcu_assign_pointer(idev->mc_list, mc);
mca_get(mc);
@@ -1018,7 +1023,8 @@ int __ipv6_dev_mc_dec(struct inet6_dev *idev, const struct in6_addr *addr)
map = &ma->next) {
if (ipv6_addr_equal(&ma->mca_addr, addr)) {
if (--ma->mca_users == 0) {
- *map = ma->next;
+ rcu_assign_pointer(*map,
+ mc_dereference(ma->next, idev));
igmp6_group_dropped(ma);
ip6_mc_clear_src(ma);
diff --git a/net/ipv6/netfilter/ip6_tables.c b/net/ipv6/netfilter/ip6_tables.c
index f4e01f451d57..551d3ca1036d 100644
--- a/net/ipv6/netfilter/ip6_tables.c
+++ b/net/ipv6/netfilter/ip6_tables.c
@@ -1735,13 +1735,11 @@ int ip6t_register_table(struct net *net, const struct xt_table *table,
const struct ip6t_replace *repl,
const struct nf_hook_ops *template_ops)
{
- struct nf_hook_ops *ops;
- unsigned int num_ops;
- int ret, i;
- struct xt_table_info *newinfo;
struct xt_table_info bootstrap = {0};
- void *loc_cpu_entry;
+ struct xt_table_info *newinfo;
struct xt_table *new_table;
+ void *loc_cpu_entry;
+ int ret;
newinfo = xt_alloc_table_info(repl->size);
if (!newinfo)
@@ -1756,7 +1754,7 @@ int ip6t_register_table(struct net *net, const struct xt_table *table,
return ret;
}
- new_table = xt_register_table(net, table, &bootstrap, newinfo);
+ new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo);
if (IS_ERR(new_table)) {
struct ip6t_entry *iter;
@@ -1766,34 +1764,6 @@ int ip6t_register_table(struct net *net, const struct xt_table *table,
return PTR_ERR(new_table);
}
- if (!template_ops)
- return 0;
-
- num_ops = hweight32(table->valid_hooks);
- if (num_ops == 0) {
- ret = -EINVAL;
- goto out_free;
- }
-
- ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
- if (!ops) {
- ret = -ENOMEM;
- goto out_free;
- }
-
- for (i = 0; i < num_ops; i++)
- ops[i].priv = new_table;
-
- new_table->ops = ops;
-
- ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret != 0)
- goto out_free;
-
- return ret;
-
-out_free:
- __ip6t_unregister_table(net, new_table);
return ret;
}
diff --git a/net/ipv6/netfilter/ip6table_nat.c b/net/ipv6/netfilter/ip6table_nat.c
index e119d4f090cc..5be723232df8 100644
--- a/net/ipv6/netfilter/ip6table_nat.c
+++ b/net/ipv6/netfilter/ip6table_nat.c
@@ -81,7 +81,7 @@ static int ip6t_nat_register_lookups(struct net *net)
while (i)
nf_nat_ipv6_unregister_fn(net, &ops[--i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
return ret;
}
}
@@ -102,7 +102,7 @@ static void ip6t_nat_unregister_lookups(struct net *net)
for (i = 0; i < ARRAY_SIZE(nf_nat_ipv6_ops); i++)
nf_nat_ipv6_unregister_fn(net, &ops[i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
}
static int ip6table_nat_table_init(struct net *net)
diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
index 1f980e827d04..614f7267dc9f 100644
--- a/net/ipv6/sit.c
+++ b/net/ipv6/sit.c
@@ -1861,7 +1861,7 @@ static int __net_init sit_init_net(struct net *net)
/* FB netdevice is special: we have one, and only one per netns.
* Allowing to move it to another netns is clearly unsafe.
*/
- sitn->fb_tunnel_dev->netns_local = true;
+ sitn->fb_tunnel_dev->netns_immutable = true;
t = netdev_priv(sitn->fb_tunnel_dev);
t->net = net;
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 69e37c3d4b1c..ae2a91e3f15c 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1853,6 +1853,7 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb)
}
}
+ isn = 0;
process:
if (static_branch_unlikely(&ip6_min_hopcount)) {
/* min_hopcount can be changed concurrently from do_ipv6_setsockopt() */
@@ -1882,6 +1883,7 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb)
th = (const struct tcphdr *)skb->data;
hdr = ipv6_hdr(skb);
tcp_v6_fill_cb(skb, hdr, th);
+ TCP_SKB_CB(skb)->tcp_tw_isn = isn;
skb->dev = NULL;
@@ -1968,7 +1970,6 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb)
sk = sk2;
tcp_v6_restore_cb(skb);
refcounted = false;
- __this_cpu_write(tcp_tw_isn, isn);
goto process;
}
}
diff --git a/net/mac80211/debug.h b/net/mac80211/debug.h
index 35a8ba25fa57..5b81998cb0c9 100644
--- a/net/mac80211/debug.h
+++ b/net/mac80211/debug.h
@@ -1,7 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Portions
- * Copyright (C) 2022 - 2023 Intel Corporation
+ * Copyright (C) 2022 - 2024 Intel Corporation
*/
#ifndef __MAC80211_DEBUG_H
#define __MAC80211_DEBUG_H
@@ -152,6 +152,14 @@ do { \
else \
_sdata_err((link)->sdata, fmt, ##__VA_ARGS__); \
} while (0)
+#define link_id_info(sdata, link_id, fmt, ...) \
+ do { \
+ if (ieee80211_vif_is_mld(&sdata->vif)) \
+ _sdata_info(sdata, "[link %d] " fmt, link_id, \
+ ##__VA_ARGS__); \
+ else \
+ _sdata_info(sdata, fmt, ##__VA_ARGS__); \
+ } while (0)
#define _link_id_dbg(print, sdata, link_id, fmt, ...) \
do { \
if (ieee80211_vif_is_mld(&(sdata)->vif)) \
diff --git a/net/mac80211/mlme.c b/net/mac80211/mlme.c
index fc02194233f8..7fd8fbe3400d 100644
--- a/net/mac80211/mlme.c
+++ b/net/mac80211/mlme.c
@@ -345,6 +345,7 @@ ieee80211_determine_ap_chan(struct ieee80211_sub_if_data *sdata,
static bool
ieee80211_verify_peer_he_mcs_support(struct ieee80211_sub_if_data *sdata,
+ int link_id,
const struct ieee80211_he_cap_elem *he_cap,
const struct ieee80211_he_operation *he_op)
{
@@ -372,9 +373,9 @@ ieee80211_verify_peer_he_mcs_support(struct ieee80211_sub_if_data *sdata,
*/
if ((mcs_80_map_tx & 0x3) == IEEE80211_HE_MCS_NOT_SUPPORTED ||
(mcs_80_map_rx & 0x3) == IEEE80211_HE_MCS_NOT_SUPPORTED) {
- sdata_info(sdata,
- "Missing mandatory rates for 1 Nss, rx 0x%x, tx 0x%x, disable HE\n",
- mcs_80_map_tx, mcs_80_map_rx);
+ link_id_info(sdata, link_id,
+ "Missing mandatory rates for 1 Nss, rx 0x%x, tx 0x%x, disable HE\n",
+ mcs_80_map_tx, mcs_80_map_rx);
return false;
}
@@ -418,9 +419,9 @@ ieee80211_verify_peer_he_mcs_support(struct ieee80211_sub_if_data *sdata,
if (ap_rx_val == IEEE80211_HE_MCS_NOT_SUPPORTED ||
ap_tx_val == IEEE80211_HE_MCS_NOT_SUPPORTED ||
ap_rx_val < ap_op_val || ap_tx_val < ap_op_val) {
- sdata_info(sdata,
- "Invalid rates for %d Nss, rx %d, tx %d oper %d, disable HE\n",
- nss, ap_rx_val, ap_rx_val, ap_op_val);
+ link_id_info(sdata, link_id,
+ "Invalid rates for %d Nss, rx %d, tx %d oper %d, disable HE\n",
+ nss, ap_rx_val, ap_tx_val, ap_op_val);
return false;
}
}
@@ -871,8 +872,8 @@ ieee80211_determine_chan_mode(struct ieee80211_sub_if_data *sdata,
return elems;
case NL80211_BAND_6GHZ:
if (ap_mode < IEEE80211_CONN_MODE_HE) {
- sdata_info(sdata,
- "Rejecting non-HE 6/7 GHz connection");
+ link_id_info(sdata, link_id,
+ "Rejecting non-HE 6/7 GHz connection");
ret = -EINVAL;
goto free;
}
@@ -916,7 +917,10 @@ ieee80211_determine_chan_mode(struct ieee80211_sub_if_data *sdata,
while (!ieee80211_chandef_usable(sdata, &chanreq->oper,
IEEE80211_CHAN_DISABLED)) {
- if (WARN_ON(chanreq->oper.width == NL80211_CHAN_WIDTH_20_NOHT)) {
+ if (chanreq->oper.width == NL80211_CHAN_WIDTH_20_NOHT) {
+ link_id_info(sdata, link_id,
+ "unusable channel (%d MHz) for connection\n",
+ chanreq->oper.chan->center_freq);
ret = -EINVAL;
goto free;
}
@@ -943,16 +947,18 @@ ieee80211_determine_chan_mode(struct ieee80211_sub_if_data *sdata,
}
if (chanreq->oper.width != ap_chandef->width || ap_mode != conn->mode)
- sdata_info(sdata,
- "regulatory prevented using AP config, downgraded\n");
+ link_id_info(sdata, link_id,
+ "regulatory prevented using AP config, downgraded\n");
if (conn->mode >= IEEE80211_CONN_MODE_HE &&
- (!ieee80211_verify_peer_he_mcs_support(sdata, (void *)elems->he_cap,
+ (!ieee80211_verify_peer_he_mcs_support(sdata, link_id,
+ (void *)elems->he_cap,
elems->he_operation) ||
!ieee80211_verify_sta_he_mcs_support(sdata, sband,
elems->he_operation))) {
conn->mode = IEEE80211_CONN_MODE_VHT;
- sdata_info(sdata, "required MCSes not supported, disabling HE\n");
+ link_id_info(sdata, link_id,
+ "required MCSes not supported, disabling HE\n");
}
if (conn->mode >= IEEE80211_CONN_MODE_EHT &&
@@ -962,7 +968,8 @@ ieee80211_determine_chan_mode(struct ieee80211_sub_if_data *sdata,
conn->bw_limit = min_t(enum ieee80211_conn_bw_limit,
conn->bw_limit,
IEEE80211_CONN_BW_LIMIT_160);
- sdata_info(sdata, "required MCSes not supported, disabling EHT\n");
+ link_id_info(sdata, link_id,
+ "required MCSes not supported, disabling EHT\n");
}
/* the mode can only decrease, so this must terminate */
diff --git a/net/mac80211/rx.c b/net/mac80211/rx.c
index 7e3bcd709c57..261add479030 100644
--- a/net/mac80211/rx.c
+++ b/net/mac80211/rx.c
@@ -1425,6 +1425,9 @@ ieee80211_rx_h_check_dup(struct ieee80211_rx_data *rx)
if (status->flag & RX_FLAG_DUP_VALIDATED)
return RX_CONTINUE;
+ if (ieee80211_is_ext(hdr->frame_control))
+ return RX_CONTINUE;
+
/*
* Drop duplicate 802.11 retransmissions
* (IEEE 802.11-2012: 9.3.2.10 "Duplicate detection and recovery")
@@ -4447,8 +4450,12 @@ static bool ieee80211_accept_frame(struct ieee80211_rx_data *rx)
struct ieee80211_hdr *hdr = (void *)skb->data;
struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(skb);
u8 *bssid = ieee80211_get_bssid(hdr, skb->len, sdata->vif.type);
- bool multicast = is_multicast_ether_addr(hdr->addr1) ||
- ieee80211_is_s1g_beacon(hdr->frame_control);
+ bool multicast;
+
+ if (ieee80211_is_s1g_beacon(hdr->frame_control))
+ return sdata->vif.type == NL80211_IFTYPE_STATION && bssid;
+
+ multicast = is_multicast_ether_addr(hdr->addr1);
switch (sdata->vif.type) {
case NL80211_IFTYPE_STATION:
@@ -4897,6 +4904,7 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
u8 sa[ETH_ALEN];
} addrs __aligned(2);
struct ieee80211_sta_rx_stats *stats;
+ u32 encoded_rate;
/* for parallel-rx, we need to have DUP_VALIDATED, otherwise we write
* to a common data structure; drivers can implement that per queue
@@ -4968,6 +4976,11 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
/* after this point, don't punt to the slowpath! */
+ if (fast_rx->uses_rss)
+ stats = this_cpu_ptr(rx->link_sta->pcpu_rx_stats);
+ else
+ stats = &rx->link_sta->rx_stats;
+
if (rx->key && !(status->flag & RX_FLAG_MIC_STRIPPED) &&
pskb_trim(skb, skb->len - fast_rx->icv_len))
goto drop;
@@ -4999,9 +5012,14 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
/* push the addresses in front */
memcpy(skb_push(skb, sizeof(addrs)), &addrs, sizeof(addrs));
+ /* capture before mesh forward may memset or free skb->cb */
+ encoded_rate = sta_stats_encode_rate(status);
+
res = ieee80211_rx_mesh_data(rx->sdata, rx->sta, rx->skb);
switch (res) {
case RX_QUEUED:
+ stats->last_rx = jiffies;
+ stats->last_rate = encoded_rate;
return true;
case RX_CONTINUE:
break;
@@ -5015,11 +5033,6 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
drop:
dev_kfree_skb(skb);
- if (fast_rx->uses_rss)
- stats = this_cpu_ptr(rx->link_sta->pcpu_rx_stats);
- else
- stats = &rx->link_sta->rx_stats;
-
stats->dropped++;
return true;
}
@@ -5081,6 +5094,11 @@ static bool ieee80211_prepare_and_rx_handle(struct ieee80211_rx_data *rx,
hdr = (struct ieee80211_hdr *)rx->skb->data;
}
+ if (ieee80211_is_s1g_beacon(hdr->frame_control)) {
+ ieee80211_invoke_rx_handlers(rx);
+ return true;
+ }
+
if (unlikely(rx->sta && rx->sta->sta.mlo) &&
is_unicast_ether_addr(hdr->addr1) &&
!ieee80211_is_probe_resp(hdr->frame_control) &&
@@ -5165,6 +5183,13 @@ static bool ieee80211_rx_for_interface(struct ieee80211_rx_data *rx,
struct sta_info *sta;
int link_id = -1;
+ if (ieee80211_is_s1g_beacon(hdr->frame_control)) {
+ if (!ieee80211_rx_data_set_sta(rx, NULL, -1))
+ return false;
+
+ return ieee80211_prepare_and_rx_handle(rx, skb, consume);
+ }
+
/*
* Look up link station first, in case there's a
* chance that they might have a link address that
@@ -5224,6 +5249,17 @@ static void __ieee80211_rx_handle_packet(struct ieee80211_hw *hw,
err = -ENOBUFS;
else
err = skb_linearize(skb);
+ } else if (ieee80211_is_s1g_beacon(fc)) {
+ size_t s1g_hdr_len = offsetof(struct ieee80211_ext,
+ u.s1g_beacon.variable) +
+ ieee80211_s1g_optional_len(fc);
+
+ if (skb->len < s1g_hdr_len)
+ err = -ENOBUFS;
+ else
+ err = skb_linearize(skb);
+ } else if (ieee80211_is_ext(fc)) {
+ err = -EINVAL;
} else {
err = !pskb_may_pull(skb, ieee80211_hdrlen(fc));
}
diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c
index 122697fbd84b..09f212134c82 100644
--- a/net/mac80211/sta_info.c
+++ b/net/mac80211/sta_info.c
@@ -355,6 +355,15 @@ static void sta_info_free_link(struct link_sta_info *link_sta)
free_percpu(link_sta->pcpu_rx_stats);
}
+static void sta_link_free_rcu(struct rcu_head *head)
+{
+ struct sta_link_alloc *alloc =
+ container_of(head, struct sta_link_alloc, rcu_head);
+
+ sta_info_free_link(&alloc->info);
+ kfree(alloc);
+}
+
static void sta_remove_link(struct sta_info *sta, unsigned int link_id,
bool unhash)
{
@@ -379,10 +388,8 @@ static void sta_remove_link(struct sta_info *sta, unsigned int link_id,
sta->sta.valid_links &= ~BIT(link_id);
RCU_INIT_POINTER(sta->link[link_id], NULL);
RCU_INIT_POINTER(sta->sta.link[link_id], NULL);
- if (alloc) {
- sta_info_free_link(&alloc->info);
- kfree_rcu(alloc, rcu_head);
- }
+ if (alloc)
+ call_rcu(&alloc->rcu_head, sta_link_free_rcu);
ieee80211_sta_recalc_aggregates(&sta->sta);
}
@@ -2573,6 +2580,39 @@ static inline u64 sta_get_stats_bytes(struct ieee80211_sta_rx_stats *rxstats)
return value;
}
+#ifdef CONFIG_MAC80211_MESH
+static void sta_set_mesh_sinfo(struct sta_info *sta,
+ struct station_info *sinfo)
+{
+ struct ieee80211_local *local = sta->sdata->local;
+
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_LLID) |
+ BIT_ULL(NL80211_STA_INFO_PLID) |
+ BIT_ULL(NL80211_STA_INFO_PLINK_STATE) |
+ BIT_ULL(NL80211_STA_INFO_LOCAL_PM) |
+ BIT_ULL(NL80211_STA_INFO_PEER_PM) |
+ BIT_ULL(NL80211_STA_INFO_NONPEER_PM) |
+ BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_GATE) |
+ BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_AS);
+
+ sinfo->llid = sta->mesh->llid;
+ sinfo->plid = sta->mesh->plid;
+ sinfo->plink_state = sta->mesh->plink_state;
+ if (test_sta_flag(sta, WLAN_STA_TOFFSET_KNOWN)) {
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_T_OFFSET);
+ sinfo->t_offset = sta->mesh->t_offset;
+ }
+ sinfo->local_pm = sta->mesh->local_pm;
+ sinfo->peer_pm = sta->mesh->peer_pm;
+ sinfo->nonpeer_pm = sta->mesh->nonpeer_pm;
+ sinfo->connected_to_gate = sta->mesh->connected_to_gate;
+ sinfo->connected_to_as = sta->mesh->connected_to_as;
+
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_LINK_METRIC);
+ sinfo->airtime_link_metric = airtime_link_metric_get(local, sta);
+}
+#endif
+
void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo,
bool tidstats)
{
@@ -2757,31 +2797,10 @@ void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo,
sta_set_tidstats(sta, &sinfo->pertid[i], i);
}
- if (ieee80211_vif_is_mesh(&sdata->vif)) {
#ifdef CONFIG_MAC80211_MESH
- sinfo->filled |= BIT_ULL(NL80211_STA_INFO_LLID) |
- BIT_ULL(NL80211_STA_INFO_PLID) |
- BIT_ULL(NL80211_STA_INFO_PLINK_STATE) |
- BIT_ULL(NL80211_STA_INFO_LOCAL_PM) |
- BIT_ULL(NL80211_STA_INFO_PEER_PM) |
- BIT_ULL(NL80211_STA_INFO_NONPEER_PM) |
- BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_GATE) |
- BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_AS);
-
- sinfo->llid = sta->mesh->llid;
- sinfo->plid = sta->mesh->plid;
- sinfo->plink_state = sta->mesh->plink_state;
- if (test_sta_flag(sta, WLAN_STA_TOFFSET_KNOWN)) {
- sinfo->filled |= BIT_ULL(NL80211_STA_INFO_T_OFFSET);
- sinfo->t_offset = sta->mesh->t_offset;
- }
- sinfo->local_pm = sta->mesh->local_pm;
- sinfo->peer_pm = sta->mesh->peer_pm;
- sinfo->nonpeer_pm = sta->mesh->nonpeer_pm;
- sinfo->connected_to_gate = sta->mesh->connected_to_gate;
- sinfo->connected_to_as = sta->mesh->connected_to_as;
+ if (ieee80211_vif_is_mesh(&sdata->vif))
+ sta_set_mesh_sinfo(sta, sinfo);
#endif
- }
sinfo->bss_param.flags = 0;
if (sdata->vif.bss_conf.use_cts_prot)
@@ -2837,12 +2856,6 @@ void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo,
sinfo->filled |=
BIT_ULL(NL80211_STA_INFO_ACK_SIGNAL_AVG);
}
-
- if (ieee80211_vif_is_mesh(&sdata->vif)) {
- sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_LINK_METRIC);
- sinfo->airtime_link_metric =
- airtime_link_metric_get(local, sta);
- }
}
u32 sta_get_expected_throughput(struct sta_info *sta)
diff --git a/net/mac80211/util.c b/net/mac80211/util.c
index 2dfb9c089b47..091a1f51eb15 100644
--- a/net/mac80211/util.c
+++ b/net/mac80211/util.c
@@ -73,6 +73,9 @@ u8 *ieee80211_get_bssid(struct ieee80211_hdr *hdr, size_t len,
if (ieee80211_is_s1g_beacon(fc)) {
struct ieee80211_ext *ext = (void *) hdr;
+ if (len < offsetofend(struct ieee80211_ext, u.s1g_beacon.sa))
+ return NULL;
+
return ext->u.s1g_beacon.sa;
}
diff --git a/net/mpls/af_mpls.c b/net/mpls/af_mpls.c
index 82909a59d410..70035b1a52c4 100644
--- a/net/mpls/af_mpls.c
+++ b/net/mpls/af_mpls.c
@@ -75,16 +75,39 @@ static void rtmsg_lfib(int event, u32 label, struct mpls_route *rt,
struct nlmsghdr *nlh, struct net *net, u32 portid,
unsigned int nlm_flags);
-static struct mpls_route *mpls_route_input_rcu(struct net *net, unsigned index)
+static struct mpls_route *mpls_route_input(struct net *net, unsigned int index)
{
- struct mpls_route *rt = NULL;
+ struct mpls_route __rcu **platform_label;
- if (index < net->mpls.platform_labels) {
- struct mpls_route __rcu **platform_label =
- rcu_dereference_rtnl(net->mpls.platform_label);
- rt = rcu_dereference_rtnl(platform_label[index]);
- }
- return rt;
+ platform_label = mpls_dereference(net, net->mpls.platform_label);
+ return mpls_dereference(net, platform_label[index]);
+}
+
+static struct mpls_route __rcu **mpls_platform_label_rcu(struct net *net, size_t *platform_labels)
+{
+ struct mpls_route __rcu **platform_label;
+ unsigned int sequence;
+
+ do {
+ sequence = read_seqcount_begin(&net->mpls.platform_label_seq);
+ platform_label = rcu_dereference(net->mpls.platform_label);
+ *platform_labels = net->mpls.platform_labels;
+ } while (read_seqcount_retry(&net->mpls.platform_label_seq, sequence));
+
+ return platform_label;
+}
+
+static struct mpls_route *mpls_route_input_rcu(struct net *net, unsigned int index)
+{
+ struct mpls_route __rcu **platform_label;
+ size_t platform_labels;
+
+ platform_label = mpls_platform_label_rcu(net, &platform_labels);
+
+ if (index >= platform_labels)
+ return NULL;
+
+ return rcu_dereference(platform_label[index]);
}
bool mpls_output_possible(const struct net_device *dev)
@@ -135,7 +158,7 @@ void mpls_stats_inc_outucastpkts(struct net_device *dev,
struct mpls_dev *mdev;
if (skb->protocol == htons(ETH_P_MPLS_UC)) {
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_rcu(dev);
if (mdev)
MPLS_INC_STATS_LEN(mdev, skb->len,
tx_packets,
@@ -359,7 +382,7 @@ static int mpls_forward(struct sk_buff *skb, struct net_device *dev,
/* Careful this entire function runs inside of an rcu critical section */
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_rcu(dev);
if (!mdev)
goto drop;
@@ -468,7 +491,7 @@ static int mpls_forward(struct sk_buff *skb, struct net_device *dev,
return 0;
tx_err:
- out_mdev = out_dev ? mpls_dev_get(out_dev) : NULL;
+ out_mdev = out_dev ? mpls_dev_rcu(out_dev) : NULL;
if (out_mdev)
MPLS_INC_STATS(out_mdev, tx_errors);
goto drop;
@@ -532,10 +555,23 @@ static struct mpls_route *mpls_rt_alloc(u8 num_nh, u8 max_alen, u8 max_labels)
return rt;
}
+static void mpls_rt_free_rcu(struct rcu_head *head)
+{
+ struct mpls_route *rt;
+
+ rt = container_of(head, struct mpls_route, rt_rcu);
+
+ change_nexthops(rt) {
+ netdev_put(nh->nh_dev, &nh->nh_dev_tracker);
+ } endfor_nexthops(rt);
+
+ kfree(rt);
+}
+
static void mpls_rt_free(struct mpls_route *rt)
{
if (rt)
- kfree_rcu(rt, rt_rcu);
+ call_rcu(&rt->rt_rcu, mpls_rt_free_rcu);
}
static void mpls_notify_route(struct net *net, unsigned index,
@@ -559,10 +595,8 @@ static void mpls_route_update(struct net *net, unsigned index,
struct mpls_route __rcu **platform_label;
struct mpls_route *rt;
- ASSERT_RTNL();
-
- platform_label = rtnl_dereference(net->mpls.platform_label);
- rt = rtnl_dereference(platform_label[index]);
+ platform_label = mpls_dereference(net, net->mpls.platform_label);
+ rt = mpls_dereference(net, platform_label[index]);
rcu_assign_pointer(platform_label[index], new);
mpls_notify_route(net, index, rt, new, info);
@@ -589,6 +623,7 @@ static unsigned find_free_label(struct net *net)
#if IS_ENABLED(CONFIG_INET)
static struct net_device *inet_fib_lookup_dev(struct net *net,
+ struct mpls_nh *nh,
const void *addr)
{
struct net_device *dev;
@@ -601,14 +636,14 @@ static struct net_device *inet_fib_lookup_dev(struct net *net,
return ERR_CAST(rt);
dev = rt->dst.dev;
- dev_hold(dev);
-
+ netdev_hold(dev, &nh->nh_dev_tracker, GFP_KERNEL);
ip_rt_put(rt);
return dev;
}
#else
static struct net_device *inet_fib_lookup_dev(struct net *net,
+ struct mpls_nh *nh,
const void *addr)
{
return ERR_PTR(-EAFNOSUPPORT);
@@ -617,6 +652,7 @@ static struct net_device *inet_fib_lookup_dev(struct net *net,
#if IS_ENABLED(CONFIG_IPV6)
static struct net_device *inet6_fib_lookup_dev(struct net *net,
+ struct mpls_nh *nh,
const void *addr)
{
struct net_device *dev;
@@ -633,13 +669,14 @@ static struct net_device *inet6_fib_lookup_dev(struct net *net,
return ERR_CAST(dst);
dev = dst->dev;
- dev_hold(dev);
+ netdev_hold(dev, &nh->nh_dev_tracker, GFP_KERNEL);
dst_release(dst);
return dev;
}
#else
static struct net_device *inet6_fib_lookup_dev(struct net *net,
+ struct mpls_nh *nh,
const void *addr)
{
return ERR_PTR(-EAFNOSUPPORT);
@@ -655,16 +692,17 @@ static struct net_device *find_outdev(struct net *net,
if (!oif) {
switch (nh->nh_via_table) {
case NEIGH_ARP_TABLE:
- dev = inet_fib_lookup_dev(net, mpls_nh_via(rt, nh));
+ dev = inet_fib_lookup_dev(net, nh, mpls_nh_via(rt, nh));
break;
case NEIGH_ND_TABLE:
- dev = inet6_fib_lookup_dev(net, mpls_nh_via(rt, nh));
+ dev = inet6_fib_lookup_dev(net, nh, mpls_nh_via(rt, nh));
break;
case NEIGH_LINK_TABLE:
break;
}
} else {
- dev = dev_get_by_index(net, oif);
+ dev = netdev_get_by_index(net, oif,
+ &nh->nh_dev_tracker, GFP_KERNEL);
}
if (!dev)
@@ -673,8 +711,7 @@ static struct net_device *find_outdev(struct net *net,
if (IS_ERR(dev))
return dev;
- /* The caller is holding rtnl anyways, so release the dev reference */
- dev_put(dev);
+ nh->nh_dev = dev;
return dev;
}
@@ -688,20 +725,17 @@ static int mpls_nh_assign_dev(struct net *net, struct mpls_route *rt,
dev = find_outdev(net, rt, nh, oif);
if (IS_ERR(dev)) {
err = PTR_ERR(dev);
- dev = NULL;
goto errout;
}
/* Ensure this is a supported device */
err = -EINVAL;
- if (!mpls_dev_get(dev))
- goto errout;
+ if (!mpls_dev_get(net, dev))
+ goto errout_put;
if ((nh->nh_via_table == NEIGH_LINK_TABLE) &&
(dev->addr_len != nh->nh_via_alen))
- goto errout;
-
- nh->nh_dev = dev;
+ goto errout_put;
if (!(dev->flags & IFF_UP)) {
nh->nh_flags |= RTNH_F_DEAD;
@@ -715,6 +749,9 @@ static int mpls_nh_assign_dev(struct net *net, struct mpls_route *rt,
return 0;
+errout_put:
+ netdev_put(nh->nh_dev, &nh->nh_dev_tracker);
+ nh->nh_dev = NULL;
errout:
return err;
}
@@ -936,6 +973,7 @@ static int mpls_nh_build_multi(struct mpls_route_config *cfg,
return 0;
errout:
+ rt->rt_nhn = nhs;
return err;
}
@@ -1105,7 +1143,7 @@ static int mpls_fill_stats_af(struct sk_buff *skb,
struct mpls_dev *mdev;
struct nlattr *nla;
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_rcu(dev);
if (!mdev)
return -ENODATA;
@@ -1125,7 +1163,7 @@ static size_t mpls_get_stats_af_size(const struct net_device *dev)
{
struct mpls_dev *mdev;
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_rcu(dev);
if (!mdev)
return 0;
@@ -1275,7 +1313,7 @@ static int mpls_netconf_get_devconf(struct sk_buff *in_skb,
if (!dev)
goto errout;
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_get(net, dev);
if (!mdev)
goto errout;
@@ -1328,7 +1366,7 @@ static int mpls_netconf_dump_devconf(struct sk_buff *skb,
rcu_read_lock();
for_each_netdev_dump(net, dev, ctx->ifindex) {
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_rcu(dev);
if (!mdev)
continue;
err = mpls_netconf_fill_devconf(skb, mdev,
@@ -1440,8 +1478,6 @@ static struct mpls_dev *mpls_add_dev(struct net_device *dev)
int err = -ENOMEM;
int i;
- ASSERT_RTNL();
-
mdev = kzalloc(sizeof(*mdev), GFP_KERNEL);
if (!mdev)
return ERR_PTR(err);
@@ -1526,8 +1562,12 @@ static int mpls_ifdown(struct net_device *dev, int event)
change_nexthops(rt) {
unsigned int nh_flags = nh->nh_flags;
- if (nh->nh_dev != dev)
+ if (nh->nh_dev != dev) {
+ if (nh_del)
+ netdev_hold(nh->nh_dev, &nh->nh_dev_tracker,
+ GFP_KERNEL);
goto next;
+ }
switch (event) {
case NETDEV_DOWN:
@@ -1594,28 +1634,33 @@ static int mpls_dev_notify(struct notifier_block *this, unsigned long event,
void *ptr)
{
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+ struct net *net = dev_net(dev);
struct mpls_dev *mdev;
unsigned int flags;
int err;
+ mutex_lock(&net->mpls.platform_mutex);
+
if (event == NETDEV_REGISTER) {
mdev = mpls_add_dev(dev);
- if (IS_ERR(mdev))
- return notifier_from_errno(PTR_ERR(mdev));
+ if (IS_ERR(mdev)) {
+ err = PTR_ERR(mdev);
+ goto err;
+ }
- return NOTIFY_OK;
+ goto out;
}
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_get(net, dev);
if (!mdev)
- return NOTIFY_OK;
+ goto out;
switch (event) {
case NETDEV_DOWN:
err = mpls_ifdown(dev, event);
if (err)
- return notifier_from_errno(err);
+ goto err;
break;
case NETDEV_UP:
flags = dev_get_flags(dev);
@@ -1631,14 +1676,15 @@ static int mpls_dev_notify(struct notifier_block *this, unsigned long event,
} else {
err = mpls_ifdown(dev, event);
if (err)
- return notifier_from_errno(err);
+ goto err;
}
break;
case NETDEV_UNREGISTER:
err = mpls_ifdown(dev, event);
if (err)
- return notifier_from_errno(err);
- mdev = mpls_dev_get(dev);
+ goto err;
+
+ mdev = mpls_dev_get(net, dev);
if (mdev) {
mpls_dev_sysctl_unregister(dev, mdev);
RCU_INIT_POINTER(dev->mpls_ptr, NULL);
@@ -1646,16 +1692,23 @@ static int mpls_dev_notify(struct notifier_block *this, unsigned long event,
}
break;
case NETDEV_CHANGENAME:
- mdev = mpls_dev_get(dev);
+ mdev = mpls_dev_get(net, dev);
if (mdev) {
mpls_dev_sysctl_unregister(dev, mdev);
err = mpls_dev_sysctl_register(dev, mdev);
if (err)
- return notifier_from_errno(err);
+ goto err;
}
break;
}
+
+out:
+ mutex_unlock(&net->mpls.platform_mutex);
return NOTIFY_OK;
+
+err:
+ mutex_unlock(&net->mpls.platform_mutex);
+ return notifier_from_errno(err);
}
static struct notifier_block mpls_dev_notifier = {
@@ -1930,6 +1983,7 @@ static int rtm_to_route_config(struct sk_buff *skb,
static int mpls_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
struct netlink_ext_ack *extack)
{
+ struct net *net = sock_net(skb->sk);
struct mpls_route_config *cfg;
int err;
@@ -1941,7 +1995,9 @@ static int mpls_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
if (err < 0)
goto out;
+ mutex_lock(&net->mpls.platform_mutex);
err = mpls_route_del(cfg, extack);
+ mutex_unlock(&net->mpls.platform_mutex);
out:
kfree(cfg);
@@ -1952,6 +2008,7 @@ static int mpls_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
static int mpls_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
struct netlink_ext_ack *extack)
{
+ struct net *net = sock_net(skb->sk);
struct mpls_route_config *cfg;
int err;
@@ -1963,7 +2020,9 @@ static int mpls_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
if (err < 0)
goto out;
+ mutex_lock(&net->mpls.platform_mutex);
err = mpls_route_add(cfg, extack);
+ mutex_unlock(&net->mpls.platform_mutex);
out:
kfree(cfg);
@@ -2129,7 +2188,7 @@ static int mpls_valid_fib_dump_req(struct net *net, const struct nlmsghdr *nlh,
continue;
ifindex = nla_get_u32(tb[i]);
- filter->dev = __dev_get_by_index(net, ifindex);
+ filter->dev = dev_get_by_index_rcu(net, ifindex);
if (!filter->dev)
return -ENODEV;
filter->filter_set = 1;
@@ -2167,20 +2226,19 @@ static int mpls_dump_routes(struct sk_buff *skb, struct netlink_callback *cb)
struct net *net = sock_net(skb->sk);
struct mpls_route __rcu **platform_label;
struct fib_dump_filter filter = {
- .rtnl_held = true,
+ .rtnl_held = false,
};
unsigned int flags = NLM_F_MULTI;
size_t platform_labels;
unsigned int index;
+ int err;
- ASSERT_RTNL();
+ rcu_read_lock();
if (cb->strict_check) {
- int err;
-
err = mpls_valid_fib_dump_req(net, nlh, &filter, cb);
if (err < 0)
- return err;
+ goto err;
/* for MPLS, there is only 1 table with fixed type and flags.
* If either are set in the filter then return nothing.
@@ -2188,15 +2246,14 @@ static int mpls_dump_routes(struct sk_buff *skb, struct netlink_callback *cb)
if ((filter.table_id && filter.table_id != RT_TABLE_MAIN) ||
(filter.rt_type && filter.rt_type != RTN_UNICAST) ||
filter.flags)
- return skb->len;
+ goto unlock;
}
index = cb->args[0];
if (index < MPLS_LABEL_FIRST_UNRESERVED)
index = MPLS_LABEL_FIRST_UNRESERVED;
- platform_label = rtnl_dereference(net->mpls.platform_label);
- platform_labels = net->mpls.platform_labels;
+ platform_label = mpls_platform_label_rcu(net, &platform_labels);
if (filter.filter_set)
flags |= NLM_F_DUMP_FILTERED;
@@ -2204,7 +2261,7 @@ static int mpls_dump_routes(struct sk_buff *skb, struct netlink_callback *cb)
for (; index < platform_labels; index++) {
struct mpls_route *rt;
- rt = rtnl_dereference(platform_label[index]);
+ rt = rcu_dereference(platform_label[index]);
if (!rt)
continue;
@@ -2219,7 +2276,13 @@ static int mpls_dump_routes(struct sk_buff *skb, struct netlink_callback *cb)
}
cb->args[0] = index;
+unlock:
+ rcu_read_unlock();
return skb->len;
+
+err:
+ rcu_read_unlock();
+ return err;
}
static inline size_t lfib_nlmsg_size(struct mpls_route *rt)
@@ -2350,18 +2413,20 @@ static int mpls_getroute(struct sk_buff *in_skb, struct nlmsghdr *in_nlh,
u32 portid = NETLINK_CB(in_skb).portid;
u32 in_label = LABEL_NOT_SPECIFIED;
struct nlattr *tb[RTA_MAX + 1];
+ struct mpls_route *rt = NULL;
u32 labels[MAX_NEW_LABELS];
struct mpls_shim_hdr *hdr;
unsigned int hdr_size = 0;
const struct mpls_nh *nh;
struct net_device *dev;
- struct mpls_route *rt;
struct rtmsg *rtm, *r;
struct nlmsghdr *nlh;
struct sk_buff *skb;
u8 n_labels;
int err;
+ mutex_lock(&net->mpls.platform_mutex);
+
err = mpls_valid_getroute_req(in_skb, in_nlh, tb, extack);
if (err < 0)
goto errout;
@@ -2383,7 +2448,8 @@ static int mpls_getroute(struct sk_buff *in_skb, struct nlmsghdr *in_nlh,
}
}
- rt = mpls_route_input_rcu(net, in_label);
+ if (in_label < net->mpls.platform_labels)
+ rt = mpls_route_input(net, in_label);
if (!rt) {
err = -ENETUNREACH;
goto errout;
@@ -2404,7 +2470,8 @@ static int mpls_getroute(struct sk_buff *in_skb, struct nlmsghdr *in_nlh,
goto errout_free;
}
- return rtnl_unicast(skb, net, portid);
+ err = rtnl_unicast(skb, net, portid);
+ goto errout;
}
if (tb[RTA_NEWDST]) {
@@ -2497,12 +2564,14 @@ static int mpls_getroute(struct sk_buff *in_skb, struct nlmsghdr *in_nlh,
err = rtnl_unicast(skb, net, portid);
errout:
+ mutex_unlock(&net->mpls.platform_mutex);
return err;
nla_put_failure:
nlmsg_cancel(skb, nlh);
err = -EMSGSIZE;
errout_free:
+ mutex_unlock(&net->mpls.platform_mutex);
kfree_skb(skb);
return err;
}
@@ -2525,10 +2594,13 @@ static int resize_platform_label_table(struct net *net, size_t limit)
/* In case the predefined labels need to be populated */
if (limit > MPLS_LABEL_IPV4NULL) {
struct net_device *lo = net->loopback_dev;
+
rt0 = mpls_rt_alloc(1, lo->addr_len, 0);
if (IS_ERR(rt0))
goto nort0;
+
rt0->rt_nh->nh_dev = lo;
+ netdev_hold(lo, &rt0->rt_nh->nh_dev_tracker, GFP_KERNEL);
rt0->rt_protocol = RTPROT_KERNEL;
rt0->rt_payload_type = MPT_IPV4;
rt0->rt_ttl_propagate = MPLS_TTL_PROP_DEFAULT;
@@ -2539,10 +2611,13 @@ static int resize_platform_label_table(struct net *net, size_t limit)
}
if (limit > MPLS_LABEL_IPV6NULL) {
struct net_device *lo = net->loopback_dev;
+
rt2 = mpls_rt_alloc(1, lo->addr_len, 0);
if (IS_ERR(rt2))
goto nort2;
+
rt2->rt_nh->nh_dev = lo;
+ netdev_hold(lo, &rt2->rt_nh->nh_dev_tracker, GFP_KERNEL);
rt2->rt_protocol = RTPROT_KERNEL;
rt2->rt_payload_type = MPT_IPV6;
rt2->rt_ttl_propagate = MPLS_TTL_PROP_DEFAULT;
@@ -2552,9 +2627,10 @@ static int resize_platform_label_table(struct net *net, size_t limit)
lo->addr_len);
}
- rtnl_lock();
+ mutex_lock(&net->mpls.platform_mutex);
+
/* Remember the original table */
- old = rtnl_dereference(net->mpls.platform_label);
+ old = mpls_dereference(net, net->mpls.platform_label);
old_limit = net->mpls.platform_labels;
/* Free any labels beyond the new table */
@@ -2582,10 +2658,14 @@ static int resize_platform_label_table(struct net *net, size_t limit)
}
/* Update the global pointers */
+ local_bh_disable();
+ write_seqcount_begin(&net->mpls.platform_label_seq);
net->mpls.platform_labels = limit;
rcu_assign_pointer(net->mpls.platform_label, labels);
+ write_seqcount_end(&net->mpls.platform_label_seq);
+ local_bh_enable();
- rtnl_unlock();
+ mutex_unlock(&net->mpls.platform_mutex);
mpls_rt_free(rt2);
mpls_rt_free(rt0);
@@ -2658,12 +2738,15 @@ static const struct ctl_table mpls_table[] = {
},
};
-static int mpls_net_init(struct net *net)
+static __net_init int mpls_net_init(struct net *net)
{
size_t table_size = ARRAY_SIZE(mpls_table);
struct ctl_table *table;
int i;
+ mutex_init(&net->mpls.platform_mutex);
+ seqcount_mutex_init(&net->mpls.platform_label_seq, &net->mpls.platform_mutex);
+
net->mpls.platform_labels = 0;
net->mpls.platform_label = NULL;
net->mpls.ip_ttl_propagate = 1;
@@ -2689,7 +2772,7 @@ static int mpls_net_init(struct net *net)
return 0;
}
-static void mpls_net_exit(struct net *net)
+static __net_exit void mpls_net_exit(struct net *net)
{
struct mpls_route __rcu **platform_label;
size_t platform_labels;
@@ -2709,16 +2792,20 @@ static void mpls_net_exit(struct net *net)
* As such no additional rcu synchronization is necessary when
* freeing the platform_label table.
*/
- rtnl_lock();
- platform_label = rtnl_dereference(net->mpls.platform_label);
+ mutex_lock(&net->mpls.platform_mutex);
+
+ platform_label = mpls_dereference(net, net->mpls.platform_label);
platform_labels = net->mpls.platform_labels;
+
for (index = 0; index < platform_labels; index++) {
- struct mpls_route *rt = rtnl_dereference(platform_label[index]);
- RCU_INIT_POINTER(platform_label[index], NULL);
+ struct mpls_route *rt;
+
+ rt = mpls_dereference(net, platform_label[index]);
mpls_notify_route(net, index, rt, NULL, NULL);
mpls_rt_free(rt);
}
- rtnl_unlock();
+
+ mutex_unlock(&net->mpls.platform_mutex);
kvfree(platform_label);
}
@@ -2737,7 +2824,8 @@ static struct rtnl_af_ops mpls_af_ops __read_mostly = {
static const struct rtnl_msg_handler mpls_rtnl_msg_handlers[] __initdata_or_module = {
{THIS_MODULE, PF_MPLS, RTM_NEWROUTE, mpls_rtm_newroute, NULL, 0},
{THIS_MODULE, PF_MPLS, RTM_DELROUTE, mpls_rtm_delroute, NULL, 0},
- {THIS_MODULE, PF_MPLS, RTM_GETROUTE, mpls_getroute, mpls_dump_routes, 0},
+ {THIS_MODULE, PF_MPLS, RTM_GETROUTE, mpls_getroute, mpls_dump_routes,
+ RTNL_FLAG_DUMP_UNLOCKED},
{THIS_MODULE, PF_MPLS, RTM_GETNETCONF,
mpls_netconf_get_devconf, mpls_netconf_dump_devconf,
RTNL_FLAG_DUMP_UNLOCKED},
diff --git a/net/mpls/internal.h b/net/mpls/internal.h
index b9f492ddf93b..8ddc8b7e6cbd 100644
--- a/net/mpls/internal.h
+++ b/net/mpls/internal.h
@@ -88,6 +88,7 @@ enum mpls_payload_type {
struct mpls_nh { /* next hop label forwarding entry */
struct net_device *nh_dev;
+ netdevice_tracker nh_dev_tracker;
/* nh_flags is accessed under RCU in the packet path; it is
* modified handling netdev events with rtnl lock held
@@ -184,9 +185,20 @@ static inline struct mpls_entry_decoded mpls_entry_decode(struct mpls_shim_hdr *
return result;
}
-static inline struct mpls_dev *mpls_dev_get(const struct net_device *dev)
+#define mpls_dereference(net, p) \
+ rcu_dereference_protected( \
+ (p), \
+ lockdep_is_held(&(net)->mpls.platform_mutex))
+
+static inline struct mpls_dev *mpls_dev_rcu(const struct net_device *dev)
+{
+ return rcu_dereference(dev->mpls_ptr);
+}
+
+static inline struct mpls_dev *mpls_dev_get(const struct net *net,
+ const struct net_device *dev)
{
- return rcu_dereference_rtnl(dev->mpls_ptr);
+ return mpls_dereference(net, dev->mpls_ptr);
}
int nla_put_labels(struct sk_buff *skb, int attrtype, u8 labels,
diff --git a/net/mpls/mpls_iptunnel.c b/net/mpls/mpls_iptunnel.c
index 6e73da94af7f..33872b4b9007 100644
--- a/net/mpls/mpls_iptunnel.c
+++ b/net/mpls/mpls_iptunnel.c
@@ -153,7 +153,7 @@ static int mpls_xmit(struct sk_buff *skb)
return LWTUNNEL_XMIT_DONE;
drop:
- out_mdev = out_dev ? mpls_dev_get(out_dev) : NULL;
+ out_mdev = out_dev ? mpls_dev_rcu(out_dev) : NULL;
if (out_mdev)
MPLS_INC_STATS(out_mdev, tx_errors);
kfree_skb(skb);
diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index 4e30a903f4bd..401159a66075 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -745,7 +745,7 @@ static int ip_vs_route_me_harder(struct netns_ipvs *ipvs, int af,
* Packet has been made sufficiently writable in caller
* - inout: 1=in->out, 0=out->in
*/
-void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
+bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct ip_vs_conn *cp, int inout, unsigned int toff,
bool has_ports, struct ip_vs_iphdr *ciph)
{
@@ -753,6 +753,11 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct icmphdr *icmph = (struct icmphdr *)(skb->data + toff);
struct iphdr *cih = (struct iphdr *)(icmph + 1);
+ /* Before now we may used ihl from skb frag, revalidate it after
+ * copying it into skb head to prevent out-of-bounds access
+ */
+ if (cih->ihl * 4 != ciph->len - ciph->off)
+ return false;
if (inout) {
iph->saddr = cp->vaddr.ip;
ip_send_check(iph);
@@ -786,6 +791,7 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
else
IP_VS_DBG_PKT(11, AF_INET, pp, skb, ciph->off,
"Forwarding altered incoming ICMP");
+ return true;
}
#ifdef CONFIG_IP_VS_IPV6
@@ -877,7 +883,8 @@ static int handle_response_icmp(int af, struct sk_buff *skb,
ip_vs_nat_icmp_v6(skb, pp, cp, 1, toff, has_ports, ciph);
else
#endif
- ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph);
+ if (!ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph))
+ goto out;
if (ip_vs_route_me_harder(cp->ipvs, af, skb, hooknum))
goto out;
diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index 240378e37741..e0c2820be57e 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -1577,7 +1577,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
if (skb_cow(skb, rt->dst.dev->hard_header_len))
goto tx_error;
- ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph);
+ if (!ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph))
+ goto tx_error;
/* Another hack: avoid icmp_send in ip_fragment */
skb->ignore_df = 1;
diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c
index 4a136fc3a9c0..c3f1d5dcba2c 100644
--- a/net/netfilter/nf_conntrack_bpf.c
+++ b/net/netfilter/nf_conntrack_bpf.c
@@ -64,6 +64,15 @@ enum {
NF_BPF_CT_OPTS_SZ = 16,
};
+static void *bpf_ct_opts_result(struct bpf_ct_opts *opts, u32 opts__sz, void *ret)
+{
+ if (!IS_ERR(ret))
+ return ret;
+ if (opts && opts__sz >= offsetofend(struct bpf_ct_opts, error))
+ opts->error = PTR_ERR(ret);
+ return NULL;
+}
+
static int bpf_nf_ct_tuple_parse(struct bpf_sock_tuple *bpf_tuple,
u32 tuple_len, u8 protonum, u8 dir,
struct nf_conntrack_tuple *tuple)
@@ -112,44 +121,56 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple,
struct nf_conntrack_tuple otuple, rtuple;
struct nf_conntrack_zone ct_zone;
struct nf_conn *ct;
+ u8 ct_zone_dir = 0;
+ u16 ct_zone_id;
+ s32 netns_id;
+ u8 l4proto;
int err;
if (!opts || !bpf_tuple)
return ERR_PTR(-EINVAL);
if (!(opts_len == NF_BPF_CT_OPTS_SZ || opts_len == 12))
return ERR_PTR(-EINVAL);
+
+ netns_id = READ_ONCE(opts->netns_id);
+ l4proto = READ_ONCE(opts->l4proto);
+ ct_zone_id = READ_ONCE(opts->ct_zone_id);
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->reserved[0] || opts->reserved[1] || opts->reserved[2])
+ ct_zone_dir = READ_ONCE(opts->ct_zone_dir);
+ if (READ_ONCE(opts->reserved[0]) ||
+ READ_ONCE(opts->reserved[1]) ||
+ READ_ONCE(opts->reserved[2]))
return ERR_PTR(-EINVAL);
} else {
- if (opts->ct_zone_id)
+ if (ct_zone_id)
return ERR_PTR(-EINVAL);
}
- if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS))
+ if (unlikely(netns_id < BPF_F_CURRENT_NETNS))
return ERR_PTR(-EINVAL);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_ORIGINAL, &otuple);
if (err < 0)
return ERR_PTR(err);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_REPLY, &rtuple);
if (err < 0)
return ERR_PTR(err);
- if (opts->netns_id >= 0) {
- net = get_net_ns_by_id(net, opts->netns_id);
+ if (netns_id >= 0) {
+ net = get_net_ns_by_id(net, netns_id);
if (unlikely(!net))
return ERR_PTR(-ENONET);
}
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->ct_zone_dir == 0)
- opts->ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
- nf_ct_zone_init(&ct_zone,
- opts->ct_zone_id, opts->ct_zone_dir, 0);
+ if (ct_zone_dir == 0) {
+ ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
+ opts->ct_zone_dir = ct_zone_dir;
+ }
+ nf_ct_zone_init(&ct_zone, ct_zone_id, ct_zone_dir, 0);
} else {
ct_zone = nf_ct_zone_dflt;
}
@@ -163,7 +184,7 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple,
__nf_ct_set_timeout(ct, timeout * HZ);
out:
- if (opts->netns_id >= 0)
+ if (netns_id >= 0)
put_net(net);
return ct;
@@ -178,46 +199,58 @@ static struct nf_conn *__bpf_nf_ct_lookup(struct net *net,
struct nf_conntrack_tuple tuple;
struct nf_conntrack_zone ct_zone;
struct nf_conn *ct;
+ u8 ct_zone_dir = 0;
+ u16 ct_zone_id;
+ s32 netns_id;
+ u8 l4proto;
int err;
if (!opts || !bpf_tuple)
return ERR_PTR(-EINVAL);
if (!(opts_len == NF_BPF_CT_OPTS_SZ || opts_len == 12))
return ERR_PTR(-EINVAL);
+
+ netns_id = READ_ONCE(opts->netns_id);
+ l4proto = READ_ONCE(opts->l4proto);
+ ct_zone_id = READ_ONCE(opts->ct_zone_id);
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->reserved[0] || opts->reserved[1] || opts->reserved[2])
+ ct_zone_dir = READ_ONCE(opts->ct_zone_dir);
+ if (READ_ONCE(opts->reserved[0]) ||
+ READ_ONCE(opts->reserved[1]) ||
+ READ_ONCE(opts->reserved[2]))
return ERR_PTR(-EINVAL);
} else {
- if (opts->ct_zone_id)
+ if (ct_zone_id)
return ERR_PTR(-EINVAL);
}
- if (unlikely(opts->l4proto != IPPROTO_TCP && opts->l4proto != IPPROTO_UDP))
+ if (unlikely(l4proto != IPPROTO_TCP && l4proto != IPPROTO_UDP))
return ERR_PTR(-EPROTO);
- if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS))
+ if (unlikely(netns_id < BPF_F_CURRENT_NETNS))
return ERR_PTR(-EINVAL);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_ORIGINAL, &tuple);
if (err < 0)
return ERR_PTR(err);
- if (opts->netns_id >= 0) {
- net = get_net_ns_by_id(net, opts->netns_id);
+ if (netns_id >= 0) {
+ net = get_net_ns_by_id(net, netns_id);
if (unlikely(!net))
return ERR_PTR(-ENONET);
}
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->ct_zone_dir == 0)
- opts->ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
- nf_ct_zone_init(&ct_zone,
- opts->ct_zone_id, opts->ct_zone_dir, 0);
+ if (ct_zone_dir == 0) {
+ ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
+ opts->ct_zone_dir = ct_zone_dir;
+ }
+ nf_ct_zone_init(&ct_zone, ct_zone_id, ct_zone_dir, 0);
} else {
ct_zone = nf_ct_zone_dflt;
}
hash = nf_conntrack_find_get(net, &ct_zone, &tuple);
- if (opts->netns_id >= 0)
+ if (netns_id >= 0)
put_net(net);
if (!hash)
return ERR_PTR(-ENOENT);
@@ -298,13 +331,7 @@ bpf_xdp_ct_alloc(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
nfct = __bpf_nf_ct_alloc_entry(dev_net(ctx->rxq->dev), bpf_tuple, tuple__sz,
opts, opts__sz, 10);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
-
- return (struct nf_conn___init *)nfct;
+ return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_xdp_ct_lookup - Lookup CT entry for the given tuple, and acquire a
@@ -333,12 +360,7 @@ bpf_xdp_ct_lookup(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
caller_net = dev_net(ctx->rxq->dev);
nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
- return nfct;
+ return bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_skb_ct_alloc - Allocate a new CT entry
@@ -366,13 +388,7 @@ bpf_skb_ct_alloc(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk);
nfct = __bpf_nf_ct_alloc_entry(net, bpf_tuple, tuple__sz, opts, opts__sz, 10);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
-
- return (struct nf_conn___init *)nfct;
+ return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_skb_ct_lookup - Lookup CT entry for the given tuple, and acquire a
@@ -401,12 +417,7 @@ bpf_skb_ct_lookup(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
caller_net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk);
nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
- return nfct;
+ return bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_ct_insert_entry - Add the provided entry into a CT map
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index f9a60576d119..22b4ef2e0d60 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -1467,6 +1467,31 @@ static bool gc_worker_can_early_drop(const struct nf_conn *ct)
return false;
}
+static void nf_ct_help_gc(struct nf_conn *ct)
+{
+ struct nf_conn_help *help;
+
+ if (!refcount_inc_not_zero(&ct->ct_general.use))
+ return;
+
+ /* load ->status after refcount increase */
+ smp_acquire__after_ctrl_dep();
+
+ if (!nf_ct_is_confirmed(ct) || nf_ct_is_dying(ct)) {
+ nf_ct_put(ct);
+ return;
+ }
+
+ /* re-check helper due to SLAB_TYPESAFE_BY_RCU */
+ if (test_bit(IPS_HELPER_BIT, &ct->status)) {
+ help = nfct_help(ct);
+ if (help)
+ nf_ct_expectation_gc(help);
+ }
+
+ nf_ct_put(ct);
+}
+
static void gc_worker(struct work_struct *work)
{
unsigned int i, hashsz, nf_conntrack_max95 = 0;
@@ -1539,7 +1564,13 @@ static void gc_worker(struct work_struct *work)
expires = (expires - (long)next_run) / ++count;
next_run += expires;
- if (nf_conntrack_max95 == 0 || gc_worker_skip_ct(tmp))
+ if (gc_worker_skip_ct(tmp))
+ continue;
+
+ if (test_bit(IPS_HELPER_BIT, &tmp->status))
+ nf_ct_help_gc(tmp);
+
+ if (nf_conntrack_max95 == 0)
continue;
net = nf_ct_net(tmp);
diff --git a/net/netfilter/nf_conntrack_expect.c b/net/netfilter/nf_conntrack_expect.c
index a45288fcb041..f9352dada84e 100644
--- a/net/netfilter/nf_conntrack_expect.c
+++ b/net/netfilter/nf_conntrack_expect.c
@@ -43,6 +43,24 @@ unsigned int nf_ct_expect_max __read_mostly;
static struct kmem_cache *nf_ct_expect_cachep __read_mostly;
static siphash_aligned_key_t nf_ct_expect_hashrnd;
+void nf_ct_expectation_gc(struct nf_conn_help *master_help)
+{
+ struct nf_conntrack_expect *exp;
+ struct hlist_node *next;
+
+ if (hlist_empty(&master_help->expectations))
+ return;
+
+ spin_lock_bh(&nf_conntrack_expect_lock);
+ hlist_for_each_entry_safe(exp, next, &master_help->expectations, lnode) {
+ if (!nf_ct_exp_is_expired(exp))
+ continue;
+
+ nf_ct_unlink_expect(exp);
+ }
+ spin_unlock_bh(&nf_conntrack_expect_lock);
+}
+
/* nf_conntrack_expect helper functions */
void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp,
u32 portid, int report)
@@ -53,7 +71,6 @@ void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp,
lockdep_nfct_expect_lock_held();
WARN_ON(!master_help);
- WARN_ON(timer_pending(&exp->timeout));
hlist_del_rcu(&exp->hnode);
@@ -70,16 +87,6 @@ void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp,
}
EXPORT_SYMBOL_GPL(nf_ct_unlink_expect_report);
-static void nf_ct_expectation_timed_out(struct timer_list *t)
-{
- struct nf_conntrack_expect *exp = from_timer(exp, t, timeout);
-
- spin_lock_bh(&nf_conntrack_expect_lock);
- nf_ct_unlink_expect(exp);
- spin_unlock_bh(&nf_conntrack_expect_lock);
- nf_ct_expect_put(exp);
-}
-
static unsigned int nf_ct_expect_dst_hash(const struct net *n, const struct nf_conntrack_tuple *tuple)
{
struct {
@@ -117,19 +124,6 @@ nf_ct_exp_equal(const struct nf_conntrack_tuple *tuple,
nf_ct_exp_zone_equal_any(i, zone);
}
-bool nf_ct_remove_expect(struct nf_conntrack_expect *exp)
-{
- lockdep_nfct_expect_lock_held();
-
- if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
- return true;
- }
- return false;
-}
-EXPORT_SYMBOL_GPL(nf_ct_remove_expect);
-
struct nf_conntrack_expect *
__nf_ct_expect_find(struct net *net,
const struct nf_conntrack_zone *zone,
@@ -144,6 +138,8 @@ __nf_ct_expect_find(struct net *net,
h = nf_ct_expect_dst_hash(net, tuple);
hlist_for_each_entry_rcu(i, &nf_ct_expect_hash[h], hnode) {
+ if (nf_ct_exp_is_expired(i))
+ continue;
if (nf_ct_exp_equal(tuple, i, zone, net))
return i;
}
@@ -178,6 +174,7 @@ nf_ct_find_expectation(struct net *net,
{
struct nf_conntrack_net *cnet = nf_ct_pernet(net);
struct nf_conntrack_expect *i, *exp = NULL;
+ struct hlist_node *next;
unsigned int h;
lockdep_nfct_expect_lock_held();
@@ -186,7 +183,11 @@ nf_ct_find_expectation(struct net *net,
return NULL;
h = nf_ct_expect_dst_hash(net, tuple);
- hlist_for_each_entry(i, &nf_ct_expect_hash[h], hnode) {
+ hlist_for_each_entry_safe(i, next, &nf_ct_expect_hash[h], hnode) {
+ if (nf_ct_exp_is_expired(i)) {
+ nf_ct_unlink_expect(i);
+ continue;
+ }
if (!(i->flags & NF_CT_EXPECT_INACTIVE) &&
nf_ct_exp_equal(tuple, i, zone, net)) {
exp = i;
@@ -196,13 +197,16 @@ nf_ct_find_expectation(struct net *net,
if (!exp)
return NULL;
+ if (!refcount_inc_not_zero(&exp->use))
+ return NULL;
+
/* If master is not in hash table yet (ie. packet hasn't left
this machine yet), how can other end know about expected?
Hence these are not the droids you are looking for (if
master ct never got confirmed, we'd hold a reference to it
and weird things would happen to future packets). */
if (!nf_ct_is_confirmed(exp->master))
- return NULL;
+ goto err_release_exp;
/* Avoid race with other CPUs, that for exp->master ct, is
* about to invoke ->destroy(), or nf_ct_delete() via timeout
@@ -214,18 +218,17 @@ nf_ct_find_expectation(struct net *net,
*/
if (unlikely(nf_ct_is_dying(exp->master) ||
!refcount_inc_not_zero(&exp->master->ct_general.use)))
- return NULL;
+ goto err_release_exp;
- if (exp->flags & NF_CT_EXPECT_PERMANENT || !unlink) {
- refcount_inc(&exp->use);
- return exp;
- } else if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect(exp);
+ if (exp->flags & NF_CT_EXPECT_PERMANENT || !unlink)
return exp;
- }
- /* Undo exp->master refcnt increase, if del_timer() failed */
- nf_ct_put(exp->master);
+ nf_ct_unlink_expect(exp);
+
+ return exp;
+
+err_release_exp:
+ nf_ct_expect_put(exp);
return NULL;
}
@@ -241,9 +244,8 @@ void nf_ct_remove_expectations(struct nf_conn *ct)
return;
spin_lock_bh(&nf_conntrack_expect_lock);
- hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) {
- nf_ct_remove_expect(exp);
- }
+ hlist_for_each_entry_safe(exp, next, &help->expectations, lnode)
+ nf_ct_unlink_expect(exp);
spin_unlock_bh(&nf_conntrack_expect_lock);
}
EXPORT_SYMBOL_GPL(nf_ct_remove_expectations);
@@ -292,7 +294,7 @@ static bool master_matches(const struct nf_conntrack_expect *a,
void nf_ct_unexpect_related(struct nf_conntrack_expect *exp)
{
spin_lock_bh(&nf_conntrack_expect_lock);
- nf_ct_remove_expect(exp);
+ WRITE_ONCE(exp->flags, exp->flags | NF_CT_EXPECT_DEAD);
spin_unlock_bh(&nf_conntrack_expect_lock);
}
EXPORT_SYMBOL_GPL(nf_ct_unexpect_related);
@@ -308,6 +310,7 @@ struct nf_conntrack_expect *nf_ct_expect_alloc(struct nf_conn *me)
if (!new)
return NULL;
+ new->timeout = nfct_time_stamp;
new->master = me;
refcount_set(&new->use, 1);
return new;
@@ -414,17 +417,12 @@ static void nf_ct_expect_insert(struct nf_conntrack_expect *exp)
struct net *net = nf_ct_exp_net(exp);
unsigned int h = nf_ct_expect_dst_hash(net, &exp->tuple);
- /* two references : one for hash insert, one for the timer */
- refcount_add(2, &exp->use);
+ refcount_inc(&exp->use);
- timer_setup(&exp->timeout, nf_ct_expectation_timed_out, 0);
helper = rcu_dereference_protected(master_help->helper,
lockdep_is_held(&nf_conntrack_expect_lock));
- if (helper) {
- exp->timeout.expires = jiffies +
- helper->expect_policy[exp->class].timeout * HZ;
- }
- add_timer(&exp->timeout);
+ if (helper)
+ exp->timeout += helper->expect_policy[exp->class].timeout * HZ;
hlist_add_head_rcu(&exp->lnode, &master_help->expectations);
master_help->expecting[exp->class]++;
@@ -436,20 +434,26 @@ static void nf_ct_expect_insert(struct nf_conntrack_expect *exp)
NF_CT_STAT_INC(net, expect_create);
}
-/* Race with expectations being used means we could have none to find; OK. */
-static void evict_oldest_expect(struct nf_conn *master,
- struct nf_conntrack_expect *new)
+static void evict_oldest_expect(struct nf_conn_help *master_help,
+ struct nf_conntrack_expect *new,
+ const struct nf_conntrack_expect_policy *p)
{
- struct nf_conn_help *master_help = nfct_help(master);
struct nf_conntrack_expect *exp, *last = NULL;
+ struct hlist_node *next;
- hlist_for_each_entry(exp, &master_help->expectations, lnode) {
+ hlist_for_each_entry_safe(exp, next, &master_help->expectations, lnode) {
+ if (nf_ct_exp_is_expired(exp)) {
+ nf_ct_unlink_expect(exp);
+ continue;
+ }
if (exp->class == new->class)
last = exp;
}
- if (last)
- nf_ct_remove_expect(last);
+ /* Still worth to evict oldest expectation after garbage collection? */
+ if (last &&
+ master_help->expecting[last->class] >= p->max_expected)
+ nf_ct_unlink_expect(last);
}
static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect,
@@ -474,14 +478,18 @@ static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect,
}
h = nf_ct_expect_dst_hash(net, &expect->tuple);
hlist_for_each_entry_safe(i, next, &nf_ct_expect_hash[h], hnode) {
+ if (nf_ct_exp_is_expired(i)) {
+ nf_ct_unlink_expect(i);
+ continue;
+ }
if (master_matches(i, expect, flags) &&
expect_matches(i, expect)) {
if (i->class != expect->class ||
i->master != expect->master)
return -EALREADY;
- if (nf_ct_remove_expect(i))
- break;
+ nf_ct_unlink_expect(i);
+ break;
} else if (expect_clash(i, expect)) {
ret = -EBUSY;
goto out;
@@ -493,14 +501,8 @@ static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect,
if (helper) {
p = &helper->expect_policy[expect->class];
if (p->max_expected &&
- master_help->expecting[expect->class] >= p->max_expected) {
- evict_oldest_expect(master, expect);
- if (master_help->expecting[expect->class]
- >= p->max_expected) {
- ret = -EMFILE;
- goto out;
- }
- }
+ master_help->expecting[expect->class] >= p->max_expected)
+ evict_oldest_expect(master_help, expect, p);
}
cnet = nf_ct_pernet(net);
@@ -547,10 +549,8 @@ void nf_ct_expect_iterate_destroy(bool (*iter)(struct nf_conntrack_expect *e, vo
hlist_for_each_entry_safe(exp, next,
&nf_ct_expect_hash[i],
hnode) {
- if (iter(exp, data) && del_timer(&exp->timeout)) {
+ if (iter(exp, data))
nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
- }
}
}
@@ -577,10 +577,8 @@ void nf_ct_expect_iterate_net(struct net *net,
if (!net_eq(nf_ct_exp_net(exp), net))
continue;
- if (iter(exp, data) && del_timer(&exp->timeout)) {
+ if (iter(exp, data))
nf_ct_unlink_expect_report(exp, portid, report);
- nf_ct_expect_put(exp);
- }
}
}
@@ -657,17 +655,17 @@ static int exp_seq_show(struct seq_file *s, void *v)
struct net *net = seq_file_net(s);
struct hlist_node *n = v;
char *delim = "";
+ __s32 timeout;
expect = hlist_entry(n, struct nf_conntrack_expect, hnode);
if (!net_eq(nf_ct_exp_net(expect), net))
return 0;
+ if (nf_ct_exp_is_expired(expect))
+ return 0;
- if (expect->timeout.function)
- seq_printf(s, "%ld ", timer_pending(&expect->timeout)
- ? (long)(expect->timeout.expires - jiffies)/HZ : 0);
- else
- seq_puts(s, "- ");
+ timeout = (__s32)(READ_ONCE(expect->timeout) - nfct_time_stamp) / HZ;
+ seq_printf(s, "%d ", timeout > 0 ? timeout : 0);
seq_printf(s, "l3proto = %u proto=%u ",
expect->tuple.src.l3num,
expect->tuple.dst.protonum);
diff --git a/net/netfilter/nf_conntrack_h323_main.c b/net/netfilter/nf_conntrack_h323_main.c
index a116c6629e2b..279201f8a18f 100644
--- a/net/netfilter/nf_conntrack_h323_main.c
+++ b/net/netfilter/nf_conntrack_h323_main.c
@@ -1396,8 +1396,8 @@ static int process_rcf(struct sk_buff *skb, struct nf_conn *ct,
"timeout to %u seconds for",
info->timeout);
nf_ct_dump_tuple(&exp->tuple);
- mod_timer_pending(&exp->timeout,
- jiffies + info->timeout * HZ);
+ WRITE_ONCE(exp->timeout,
+ nfct_time_stamp + (info->timeout * HZ));
}
spin_unlock_bh(&nf_conntrack_expect_lock);
}
diff --git a/net/netfilter/nf_conntrack_helper.c b/net/netfilter/nf_conntrack_helper.c
index ea0cdb7ec915..2cef826e20fb 100644
--- a/net/netfilter/nf_conntrack_helper.c
+++ b/net/netfilter/nf_conntrack_helper.c
@@ -180,10 +180,10 @@ nf_ct_helper_ext_add(struct nf_conn *ct, gfp_t gfp)
struct nf_conn_help *help;
help = nf_ct_ext_add(ct, NF_CT_EXT_HELPER, gfp);
- if (help)
+ if (help) {
+ __set_bit(IPS_HELPER_BIT, &ct->status);
INIT_HLIST_HEAD(&help->expectations);
- else
- pr_debug("failed to add helper extension area");
+ }
return help;
}
EXPORT_SYMBOL_GPL(nf_ct_helper_ext_add);
@@ -202,10 +202,8 @@ int __nf_ct_try_assign_helper(struct nf_conn *ct, struct nf_conn *tmpl,
return 0;
help = nfct_help(tmpl);
- if (help != NULL) {
+ if (help)
helper = rcu_dereference(help->helper);
- set_bit(IPS_HELPER_BIT, &ct->status);
- }
help = nfct_help(ct);
diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c
index 8bb32965edb8..60a4fb408896 100644
--- a/net/netfilter/nf_conntrack_netlink.c
+++ b/net/netfilter/nf_conntrack_netlink.c
@@ -3001,8 +3001,8 @@ static int
ctnetlink_exp_dump_expect(struct sk_buff *skb,
const struct nf_conntrack_expect *exp)
{
+ __s32 timeout = (__s32)(READ_ONCE(exp->timeout) - nfct_time_stamp) / HZ;
struct nf_conn *master = exp->master;
- long timeout = ((long)exp->timeout.expires - (long)jiffies) / HZ;
struct nf_conntrack_helper *helper;
#if IS_ENABLED(CONFIG_NF_NAT)
struct nlattr *nest_parms;
@@ -3045,7 +3045,7 @@ ctnetlink_exp_dump_expect(struct sk_buff *skb,
#endif
if (nla_put_be32(skb, CTA_EXPECT_TIMEOUT, htonl(timeout)) ||
nla_put_be32(skb, CTA_EXPECT_ID, nf_expect_get_id(exp)) ||
- nla_put_be32(skb, CTA_EXPECT_FLAGS, htonl(exp->flags)) ||
+ nla_put_be32(skb, CTA_EXPECT_FLAGS, htonl(exp->flags & NF_CT_EXPECT_MASK)) ||
nla_put_be32(skb, CTA_EXPECT_CLASS, htonl(exp->class)))
goto nla_put_failure;
@@ -3165,6 +3165,9 @@ ctnetlink_exp_dump_table(struct sk_buff *skb, struct netlink_callback *cb)
restart:
hlist_for_each_entry_rcu(exp, &nf_ct_expect_hash[cb->args[0]],
hnode) {
+ if (nf_ct_exp_is_expired(exp))
+ continue;
+
if (l3proto && exp->tuple.src.l3num != l3proto)
continue;
@@ -3444,11 +3447,8 @@ static int ctnetlink_del_expect(struct sk_buff *skb,
}
/* after list removal, usage count == 1 */
- if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect_report(exp, NETLINK_CB(skb).portid,
- nlmsg_report(info->nlh));
- nf_ct_expect_put(exp);
- }
+ nf_ct_unlink_expect_report(exp, NETLINK_CB(skb).portid,
+ nlmsg_report(info->nlh));
spin_unlock_bh(&nf_conntrack_expect_lock);
/* have to put what we 'get' above.
* after this line usage count == 0 */
@@ -3472,14 +3472,10 @@ static int
ctnetlink_change_expect(struct nf_conntrack_expect *x,
const struct nlattr * const cda[])
{
- if (cda[CTA_EXPECT_TIMEOUT]) {
- if (!del_timer(&x->timeout))
- return -ETIME;
+ if (cda[CTA_EXPECT_TIMEOUT])
+ WRITE_ONCE(x->timeout, nfct_time_stamp +
+ ntohl(nla_get_be32(cda[CTA_EXPECT_TIMEOUT])) * HZ);
- x->timeout.expires = jiffies +
- ntohl(nla_get_be32(cda[CTA_EXPECT_TIMEOUT])) * HZ;
- add_timer(&x->timeout);
- }
return 0;
}
diff --git a/net/netfilter/nf_conntrack_sip.c b/net/netfilter/nf_conntrack_sip.c
index 5b75e9bc99e9..449a6e53feae 100644
--- a/net/netfilter/nf_conntrack_sip.c
+++ b/net/netfilter/nf_conntrack_sip.c
@@ -894,11 +894,10 @@ static int refresh_signalling_expectation(struct nf_conn *ct,
exp->tuple.dst.protonum != proto ||
exp->tuple.dst.u.udp.port != port)
continue;
- if (mod_timer_pending(&exp->timeout, jiffies + expires * HZ)) {
- exp->flags &= ~NF_CT_EXPECT_INACTIVE;
- found = 1;
- break;
- }
+ WRITE_ONCE(exp->timeout, nfct_time_stamp + (expires * HZ));
+ WRITE_ONCE(exp->flags, exp->flags & ~NF_CT_EXPECT_INACTIVE);
+ found = 1;
+ break;
}
spin_unlock_bh(&nf_conntrack_expect_lock);
return found;
@@ -914,8 +913,7 @@ static void flush_expectations(struct nf_conn *ct, bool media)
hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) {
if ((exp->class != SIP_EXPECT_SIGNALLING) ^ media)
continue;
- if (!nf_ct_remove_expect(exp))
- continue;
+ nf_ct_unlink_expect(exp);
if (!media)
break;
}
@@ -1385,7 +1383,6 @@ static int process_register_request(struct sk_buff *skb, unsigned int protoff,
nf_ct_expect_init(exp, SIP_EXPECT_SIGNALLING, nf_ct_l3num(ct),
saddr, &daddr, proto, NULL, &port);
- exp->timeout.expires = sip_timeout * HZ;
rcu_assign_pointer(exp->assign_helper, helper);
exp->flags = NF_CT_EXPECT_PERMANENT | NF_CT_EXPECT_INACTIVE;
diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c
index 98edcaa37b38..e9d960a1180b 100644
--- a/net/netfilter/nf_flow_table_ip.c
+++ b/net/netfilter/nf_flow_table_ip.c
@@ -268,6 +268,7 @@ static unsigned int nf_flow_xmit_xfrm(struct sk_buff *skb,
struct dst_entry *dst)
{
skb_orphan(skb);
+ skb_dst_drop(skb);
skb_dst_set_noref(skb, dst);
dst_output(state->net, state->sk, skb);
return NF_STOLEN;
@@ -453,6 +454,7 @@ nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb,
outdev = rt->dst.dev;
skb->dev = outdev;
nexthop = rt_nexthop(rt, flow->tuplehash[!dir].tuple.src_v4.s_addr);
+ skb_dst_drop(skb);
skb_dst_set_noref(skb, &rt->dst);
neigh_xmit(NEIGH_ARP_TABLE, outdev, &nexthop, skb);
ret = NF_STOLEN;
@@ -748,6 +750,7 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
outdev = rt->dst.dev;
skb->dev = outdev;
nexthop = rt6_nexthop(rt, &flow->tuplehash[!dir].tuple.src_v6);
+ skb_dst_drop(skb);
skb_dst_set_noref(skb, &rt->dst);
neigh_xmit(NEIGH_ND_TABLE, outdev, nexthop, skb);
ret = NF_STOLEN;
diff --git a/net/netfilter/nf_nat_core.c b/net/netfilter/nf_nat_core.c
index 34e1ff930e28..109c12fdc57d 100644
--- a/net/netfilter/nf_nat_core.c
+++ b/net/netfilter/nf_nat_core.c
@@ -1247,9 +1247,11 @@ int nf_nat_register_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops,
ret = nf_register_net_hooks(net, nat_ops, ops_count);
if (ret < 0) {
mutex_unlock(&nf_nat_proto_mutex);
- for (i = 0; i < ops_count; i++)
- kfree(nat_ops[i].priv);
- kfree(nat_ops);
+ for (i = 0; i < ops_count; i++) {
+ priv = nat_ops[i].priv;
+ kfree_rcu(priv, rcu_head);
+ }
+ kfree_rcu(nat_ops, rcu);
return ret;
}
@@ -1313,7 +1315,7 @@ void nf_nat_unregister_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops,
}
nat_proto_net->nat_hook_ops = NULL;
- kfree(nat_ops);
+ kfree_rcu(nat_ops, rcu);
}
unlock:
mutex_unlock(&nf_nat_proto_mutex);
diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c
index e4325686953e..4b6cb990ab74 100644
--- a/net/netfilter/nf_tables_api.c
+++ b/net/netfilter/nf_tables_api.c
@@ -350,7 +350,7 @@ static void nft_netdev_unregister_hooks(struct net *net,
list_for_each_entry_safe(hook, next, hook_list, list) {
nf_unregister_net_hook(net, &hook->ops);
if (release_netdev) {
- list_del(&hook->list);
+ list_del_rcu(&hook->list);
kfree_rcu(hook, rcu);
}
}
@@ -440,6 +440,17 @@ static void nft_trans_commit_list_add_tail(struct net *net, struct nft_trans *tr
}
}
+static void nft_trans_commit_list_add_elem(struct net *net, struct nft_trans *trans,
+ gfp_t gfp)
+{
+ WARN_ON_ONCE(trans->msg_type != NFT_MSG_NEWSETELEM &&
+ trans->msg_type != NFT_MSG_DELSETELEM);
+
+ might_alloc(gfp);
+
+ nft_trans_commit_list_add_tail(net, trans);
+}
+
static int nft_trans_table_add(struct nft_ctx *ctx, int msg_type)
{
struct nft_trans *trans;
@@ -701,6 +712,11 @@ static void nft_map_catchall_deactivate(const struct nft_ctx *ctx,
}
}
+/* Use NFT_ITER_UPDATE iterator even if this may be called from the preparation
+ * phase, the set clone might already exist from a previous command, or it might
+ * be a set that is going away and does not require a clone. The netns and
+ * netlink release paths also need to work on the live set.
+ */
static void nft_map_deactivate(const struct nft_ctx *ctx, struct nft_set *set)
{
struct nft_set_iter iter = {
@@ -6352,17 +6368,21 @@ static void nf_tables_setelem_notify(const struct nft_ctx *ctx,
nfnetlink_set_err(net, portid, NFNLGRP_NFTABLES, -ENOBUFS);
}
-static struct nft_trans *nft_trans_elem_alloc(struct nft_ctx *ctx,
+static struct nft_trans *nft_trans_elem_alloc(const struct nft_ctx *ctx,
int msg_type,
struct nft_set *set)
{
+ struct nft_trans_elem *te;
struct nft_trans *trans;
- trans = nft_trans_alloc(ctx, msg_type, sizeof(struct nft_trans_elem));
+ trans = nft_trans_alloc(ctx, msg_type, struct_size(te, elems, 1));
if (trans == NULL)
return NULL;
- nft_trans_elem_set(trans) = set;
+ te = nft_trans_container_elem(trans);
+ te->nelems = 1;
+ te->set = set;
+
return trans;
}
@@ -6484,28 +6504,51 @@ void nft_set_elem_expr_destroy(const struct nft_ctx *ctx,
}
/* Drop references and destroy. Called from gc, dynset and abort path. */
-void nft_set_elem_destroy(const struct nft_set *set,
- const struct nft_elem_priv *elem_priv,
- bool destroy_expr)
+static void __nft_set_elem_destroy(const struct nft_ctx *ctx,
+ const struct nft_set *set,
+ const struct nft_elem_priv *elem_priv,
+ bool destroy_expr)
{
struct nft_set_ext *ext = nft_set_elem_ext(set, elem_priv);
- struct nft_ctx ctx = {
- .net = read_pnet(&set->net),
- .family = set->table->family,
- };
nft_data_release(nft_set_ext_key(ext), NFT_DATA_VALUE);
if (nft_set_ext_exists(ext, NFT_SET_EXT_DATA))
nft_data_release(nft_set_ext_data(ext), set->dtype);
if (destroy_expr && nft_set_ext_exists(ext, NFT_SET_EXT_EXPRESSIONS))
- nft_set_elem_expr_destroy(&ctx, nft_set_ext_expr(ext));
+ nft_set_elem_expr_destroy(ctx, nft_set_ext_expr(ext));
if (nft_set_ext_exists(ext, NFT_SET_EXT_OBJREF))
nft_use_dec(&(*nft_set_ext_obj(ext))->use);
kfree(elem_priv);
}
+
+/* Drop references and destroy. Called from gc and dynset. */
+void nft_set_elem_destroy(const struct nft_set *set,
+ const struct nft_elem_priv *elem_priv,
+ bool destroy_expr)
+{
+ struct nft_ctx ctx = {
+ .net = read_pnet(&set->net),
+ .family = set->table->family,
+ };
+
+ __nft_set_elem_destroy(&ctx, set, elem_priv, destroy_expr);
+}
EXPORT_SYMBOL_GPL(nft_set_elem_destroy);
+/* Drop references and destroy. Called from abort path. */
+static void nft_trans_set_elem_destroy(const struct nft_ctx *ctx, struct nft_trans_elem *te)
+{
+ int i;
+
+ for (i = 0; i < te->nelems; i++) {
+ if (te->elems[i].update_flags)
+ continue;
+
+ __nft_set_elem_destroy(ctx, te->set, te->elems[i].priv, true);
+ }
+}
+
/* Destroy element. References have been already dropped in the preparation
* path via nft_setelem_data_deactivate().
*/
@@ -6521,39 +6564,20 @@ void nf_tables_set_elem_destroy(const struct nft_ctx *ctx,
kfree(elem_priv);
}
-int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set,
- struct nft_expr *expr_array[])
+static void nft_trans_elems_destroy(const struct nft_ctx *ctx,
+ const struct nft_trans_elem *te)
{
- struct nft_expr *expr;
- int err, i, k;
-
- for (i = 0; i < set->num_exprs; i++) {
- expr = kzalloc(set->exprs[i]->ops->size, GFP_KERNEL_ACCOUNT);
- if (!expr)
- goto err_expr;
-
- err = nft_expr_clone(expr, set->exprs[i], GFP_KERNEL_ACCOUNT);
- if (err < 0) {
- kfree(expr);
- goto err_expr;
- }
- expr_array[i] = expr;
- }
-
- return 0;
-
-err_expr:
- for (k = i - 1; k >= 0; k--)
- nft_expr_destroy(ctx, expr_array[k]);
+ int i;
- return -ENOMEM;
+ for (i = 0; i < te->nelems; i++)
+ nf_tables_set_elem_destroy(ctx, te->set, te->elems[i].priv);
}
static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
const struct nft_set_ext_tmpl *tmpl,
const struct nft_set_ext *ext,
struct nft_expr *expr_array[],
- u32 num_exprs)
+ u32 num_exprs, bool override_exprs)
{
struct nft_set_elem_expr *elem_expr = nft_set_ext_expr(ext);
u32 len = sizeof(struct nft_set_elem_expr);
@@ -6576,7 +6600,8 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
goto err_elem_expr_setup;
elem_expr->size += expr_array[i]->ops->size;
- nft_expr_destroy(ctx, expr_array[i]);
+ if (override_exprs)
+ nft_expr_destroy(ctx, expr_array[i]);
expr_array[i] = NULL;
}
@@ -6584,7 +6609,9 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
err_elem_expr_setup:
for (; i < num_exprs; i++) {
- nft_expr_destroy(ctx, expr_array[i]);
+ if (override_exprs)
+ nft_expr_destroy(ctx, expr_array[i]);
+
expr_array[i] = NULL;
}
@@ -6680,6 +6707,36 @@ static void nft_setelem_activate(struct net *net, struct nft_set *set,
}
}
+static void nft_trans_elem_update(const struct nft_set *set,
+ const struct nft_trans_one_elem *elem)
+{
+ const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv);
+
+ if (elem->update_flags & NFT_TRANS_UPD_TIMEOUT)
+ WRITE_ONCE(nft_set_ext_timeout(ext)->timeout, elem->timeout);
+
+ if (elem->update_flags & NFT_TRANS_UPD_EXPIRATION)
+ WRITE_ONCE(nft_set_ext_timeout(ext)->expiration, get_jiffies_64() + elem->expiration);
+}
+
+static void nft_trans_elems_add(const struct nft_ctx *ctx,
+ struct nft_trans_elem *te)
+{
+ int i;
+
+ for (i = 0; i < te->nelems; i++) {
+ const struct nft_trans_one_elem *elem = &te->elems[i];
+
+ if (elem->update_flags)
+ nft_trans_elem_update(te->set, elem);
+ else
+ nft_setelem_activate(ctx->net, te->set, elem->priv);
+
+ nf_tables_setelem_notify(ctx, te->set, elem->priv,
+ NFT_MSG_NEWSETELEM);
+ }
+}
+
static int nft_setelem_catchall_deactivate(const struct net *net,
struct nft_set *set,
struct nft_set_elem *elem)
@@ -6765,6 +6822,24 @@ static void nft_setelem_remove(const struct net *net,
set->ops->remove(net, set, elem_priv);
}
+static void nft_trans_elems_remove(const struct nft_ctx *ctx,
+ const struct nft_trans_elem *te)
+{
+ int i;
+
+ for (i = 0; i < te->nelems; i++) {
+ nf_tables_setelem_notify(ctx, te->set,
+ te->elems[i].priv,
+ te->nft_trans.msg_type);
+
+ nft_setelem_remove(ctx->net, te->set, te->elems[i].priv);
+ if (!nft_setelem_is_catchall(te->set, te->elems[i].priv)) {
+ atomic_dec(&te->set->nelems);
+ te->set->ndeact--;
+ }
+ }
+}
+
static bool nft_setelem_valid_key_end(const struct nft_set *set,
struct nlattr **nla, u32 flags)
{
@@ -6818,6 +6893,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
struct nft_set_binding *binding;
struct nft_elem_priv *elem_priv;
struct nft_object *obj = NULL;
+ bool override_exprs = false;
struct nft_userdata *udata;
struct nft_data_desc desc;
enum nft_registers dreg;
@@ -6924,6 +7000,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
expr_array[0] = expr;
num_exprs = 1;
+ override_exprs = true;
if (set->num_exprs && set->exprs[0]->ops != expr->ops) {
err = -EOPNOTSUPP;
@@ -6952,6 +7029,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
}
expr_array[i] = expr;
num_exprs++;
+ override_exprs = true;
if (set->num_exprs && expr->ops != set->exprs[i]->ops) {
err = -EOPNOTSUPP;
@@ -6965,9 +7043,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
}
} else if (set->num_exprs > 0 &&
!(flags & NFT_SET_ELEM_INTERVAL_END)) {
- err = nft_set_elem_expr_clone(ctx, set, expr_array);
- if (err < 0)
- goto err_set_elem_expr_clone;
+ for (i = 0; i < set->num_exprs; i++)
+ expr_array[i] = set->exprs[i];
num_exprs = set->num_exprs;
}
@@ -7106,7 +7183,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
udata->len = ulen - 1;
nla_memcpy(&udata->data, nla[NFTA_SET_ELEM_USERDATA], ulen);
}
- err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs);
+ err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs,
+ override_exprs);
if (err < 0)
goto err_elem_free;
@@ -7146,23 +7224,27 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
else if (!(nlmsg_flags & NLM_F_EXCL)) {
err = 0;
if (nft_set_ext_exists(ext2, NFT_SET_EXT_TIMEOUT)) {
+ struct nft_trans_one_elem *update;
+
+ update = &nft_trans_container_elem(trans)->elems[0];
+
update_flags = 0;
if (timeout != nft_set_ext_timeout(ext2)->timeout) {
- nft_trans_elem_timeout(trans) = timeout;
+ update->timeout = timeout;
if (expiration == 0)
expiration = timeout;
update_flags |= NFT_TRANS_UPD_TIMEOUT;
}
if (expiration) {
- nft_trans_elem_expiration(trans) = expiration;
+ update->expiration = expiration;
update_flags |= NFT_TRANS_UPD_EXPIRATION;
}
if (update_flags) {
- nft_trans_elem_priv(trans) = elem_priv;
- nft_trans_elem_update_flags(trans) = update_flags;
- nft_trans_commit_list_add_tail(ctx->net, trans);
+ update->priv = elem_priv;
+ update->update_flags = update_flags;
+ nft_trans_commit_list_add_elem(ctx->net, trans, GFP_KERNEL);
goto err_set_size;
}
}
@@ -7181,8 +7263,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
goto err_element_clash;
}
- nft_trans_elem_priv(trans) = elem.priv;
- nft_trans_commit_list_add_tail(ctx->net, trans);
+ nft_trans_container_elem(trans)->elems[0].priv = elem.priv;
+ nft_trans_commit_list_add_elem(ctx->net, trans, GFP_KERNEL);
return set_full ? -ENFILE : 0;
@@ -7204,9 +7286,11 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
err_parse_key:
nft_data_release(&elem.key.val, NFT_DATA_VALUE);
err_set_elem_expr:
- for (i = 0; i < num_exprs && expr_array[i]; i++)
- nft_expr_destroy(ctx, expr_array[i]);
-err_set_elem_expr_clone:
+ if (override_exprs) {
+ for (i = 0; i < num_exprs && expr_array[i]; i++)
+ nft_expr_destroy(ctx, expr_array[i]);
+ }
+
return err;
}
@@ -7321,6 +7405,50 @@ void nft_setelem_data_deactivate(const struct net *net,
nft_use_dec(&(*nft_set_ext_obj(ext))->use);
}
+/* similar to nft_trans_elems_remove, but called from abort path to undo newsetelem.
+ * No notifications and no ndeact changes.
+ *
+ * Returns true if set had been added to (i.e., elements need to be removed again).
+ */
+static bool nft_trans_elems_new_abort(const struct nft_ctx *ctx,
+ const struct nft_trans_elem *te)
+{
+ bool removed = false;
+ int i;
+
+ for (i = 0; i < te->nelems; i++) {
+ if (te->elems[i].update_flags)
+ continue;
+
+ if (!te->set->ops->abort || nft_setelem_is_catchall(te->set, te->elems[i].priv))
+ nft_setelem_remove(ctx->net, te->set, te->elems[i].priv);
+
+ if (!nft_setelem_is_catchall(te->set, te->elems[i].priv))
+ atomic_dec(&te->set->nelems);
+
+ removed = true;
+ }
+
+ return removed;
+}
+
+/* Called from abort path to undo DELSETELEM/DESTROYSETELEM. */
+static void nft_trans_elems_destroy_abort(const struct nft_ctx *ctx,
+ const struct nft_trans_elem *te)
+{
+ int i;
+
+ for (i = 0; i < te->nelems; i++) {
+ if (!nft_setelem_active_next(ctx->net, te->set, te->elems[i].priv)) {
+ nft_setelem_data_activate(ctx->net, te->set, te->elems[i].priv);
+ nft_setelem_activate(ctx->net, te->set, te->elems[i].priv);
+ }
+
+ if (!nft_setelem_is_catchall(te->set, te->elems[i].priv))
+ te->set->ndeact--;
+ }
+}
+
static int nft_del_setelem(struct nft_ctx *ctx, struct nft_set *set,
const struct nlattr *attr)
{
@@ -7400,8 +7528,8 @@ static int nft_del_setelem(struct nft_ctx *ctx, struct nft_set *set,
nft_setelem_data_deactivate(ctx->net, set, elem.priv);
- nft_trans_elem_priv(trans) = elem.priv;
- nft_trans_commit_list_add_tail(ctx->net, trans);
+ nft_trans_container_elem(trans)->elems[0].priv = elem.priv;
+ nft_trans_commit_list_add_elem(ctx->net, trans, GFP_KERNEL);
return 0;
fail_ops:
@@ -7427,7 +7555,8 @@ static int nft_setelem_flush(const struct nft_ctx *ctx,
return 0;
trans = nft_trans_alloc_gfp(ctx, NFT_MSG_DELSETELEM,
- sizeof(struct nft_trans_elem), GFP_ATOMIC);
+ struct_size_t(struct nft_trans_elem, elems, 1),
+ GFP_ATOMIC);
if (!trans)
return -ENOMEM;
@@ -7436,8 +7565,9 @@ static int nft_setelem_flush(const struct nft_ctx *ctx,
nft_setelem_data_deactivate(ctx->net, set, elem_priv);
nft_trans_elem_set(trans) = set;
- nft_trans_elem_priv(trans) = elem_priv;
- nft_trans_commit_list_add_tail(ctx->net, trans);
+ nft_trans_container_elem(trans)->nelems = 1;
+ nft_trans_container_elem(trans)->elems[0].priv = elem_priv;
+ nft_trans_commit_list_add_elem(ctx->net, trans, GFP_ATOMIC);
return 0;
}
@@ -7448,15 +7578,13 @@ static int __nft_set_catchall_flush(const struct nft_ctx *ctx,
{
struct nft_trans *trans;
- trans = nft_trans_alloc_gfp(ctx, NFT_MSG_DELSETELEM,
- sizeof(struct nft_trans_elem), GFP_KERNEL);
+ trans = nft_trans_elem_alloc(ctx, NFT_MSG_DELSETELEM, set);
if (!trans)
return -ENOMEM;
nft_setelem_data_deactivate(ctx->net, set, elem_priv);
- nft_trans_elem_set(trans) = set;
- nft_trans_elem_priv(trans) = elem_priv;
- nft_trans_commit_list_add_tail(ctx->net, trans);
+ nft_trans_container_elem(trans)->elems[0].priv = elem_priv;
+ nft_trans_commit_list_add_elem(ctx->net, trans, GFP_KERNEL);
return 0;
}
@@ -7485,9 +7613,12 @@ static int nft_set_catchall_flush(const struct nft_ctx *ctx,
static int nft_set_flush(struct nft_ctx *ctx, struct nft_set *set, u8 genmask)
{
+ /* The set backend might need to clone the set, do it now from the
+ * preparation phase, use NFT_ITER_UPDATE_CLONE iterator type.
+ */
struct nft_set_iter iter = {
.genmask = genmask,
- .type = NFT_ITER_UPDATE,
+ .type = NFT_ITER_UPDATE_CLONE,
.fn = nft_setelem_flush,
};
@@ -8490,7 +8621,7 @@ static void __nft_unregister_flowtable_net_hooks(struct net *net,
flowtable->data.type->setup(&flowtable->data, hook->ops.dev,
FLOW_BLOCK_UNBIND);
if (release_netdev) {
- list_del(&hook->list);
+ list_del_rcu(&hook->list);
kfree_rcu(hook, rcu);
}
}
@@ -9602,9 +9733,7 @@ static void nft_commit_release(struct nft_trans *trans)
break;
case NFT_MSG_DELSETELEM:
case NFT_MSG_DESTROYSETELEM:
- nf_tables_set_elem_destroy(&ctx,
- nft_trans_elem_set(trans),
- nft_trans_elem_priv(trans));
+ nft_trans_elems_destroy(&ctx, nft_trans_container_elem(trans));
break;
case NFT_MSG_DELOBJ:
case NFT_MSG_DESTROYOBJ:
@@ -10447,25 +10576,8 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
case NFT_MSG_NEWSETELEM:
te = nft_trans_container_elem(trans);
- if (te->update_flags) {
- const struct nft_set_ext *ext =
- nft_set_elem_ext(te->set, te->elem_priv);
-
- if (te->update_flags & NFT_TRANS_UPD_TIMEOUT) {
- WRITE_ONCE(nft_set_ext_timeout(ext)->timeout,
- te->timeout);
- }
- if (te->update_flags & NFT_TRANS_UPD_EXPIRATION) {
- WRITE_ONCE(nft_set_ext_timeout(ext)->expiration,
- get_jiffies_64() + te->expiration);
- }
- } else {
- nft_setelem_activate(net, te->set, te->elem_priv);
- }
+ nft_trans_elems_add(&ctx, te);
- nf_tables_setelem_notify(&ctx, te->set,
- te->elem_priv,
- NFT_MSG_NEWSETELEM);
if (te->set->ops->commit &&
list_empty(&te->set->pending_update)) {
list_add_tail(&te->set->pending_update,
@@ -10477,14 +10589,8 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
case NFT_MSG_DESTROYSETELEM:
te = nft_trans_container_elem(trans);
- nf_tables_setelem_notify(&ctx, te->set,
- te->elem_priv,
- trans->msg_type);
- nft_setelem_remove(net, te->set, te->elem_priv);
- if (!nft_setelem_is_catchall(te->set, te->elem_priv)) {
- atomic_dec(&te->set->nelems);
- te->set->ndeact--;
- }
+ nft_trans_elems_remove(&ctx, te);
+
if (te->set->ops->commit &&
list_empty(&te->set->pending_update)) {
list_add_tail(&te->set->pending_update,
@@ -10606,8 +10712,7 @@ static void nf_tables_abort_release(struct nft_trans *trans)
nft_set_destroy(&ctx, nft_trans_set(trans));
break;
case NFT_MSG_NEWSETELEM:
- nft_set_elem_destroy(nft_trans_elem_set(trans),
- nft_trans_elem_priv(trans), true);
+ nft_trans_set_elem_destroy(&ctx, nft_trans_container_elem(trans));
break;
case NFT_MSG_NEWOBJ:
nft_obj_destroy(&ctx, nft_trans_obj(trans));
@@ -10764,18 +10869,15 @@ static int __nf_tables_abort(struct net *net, enum nfnl_abort_action action)
nft_trans_destroy(trans);
break;
case NFT_MSG_NEWSETELEM:
- if (nft_trans_elem_update_flags(trans) ||
- nft_trans_elem_set_bound(trans)) {
+ if (nft_trans_elem_set_bound(trans)) {
nft_trans_destroy(trans);
break;
}
te = nft_trans_container_elem(trans);
- if (!te->set->ops->abort ||
- nft_setelem_is_catchall(te->set, te->elem_priv))
- nft_setelem_remove(net, te->set, te->elem_priv);
-
- if (!nft_setelem_is_catchall(te->set, te->elem_priv))
- atomic_dec(&te->set->nelems);
+ if (!nft_trans_elems_new_abort(&ctx, te)) {
+ nft_trans_destroy(trans);
+ break;
+ }
if (te->set->ops->abort &&
list_empty(&te->set->pending_update)) {
@@ -10787,12 +10889,7 @@ static int __nf_tables_abort(struct net *net, enum nfnl_abort_action action)
case NFT_MSG_DESTROYSETELEM:
te = nft_trans_container_elem(trans);
- if (!nft_setelem_active_next(net, te->set, te->elem_priv)) {
- nft_setelem_data_activate(net, te->set, te->elem_priv);
- nft_setelem_activate(net, te->set, te->elem_priv);
- }
- if (!nft_setelem_is_catchall(te->set, te->elem_priv))
- te->set->ndeact--;
+ nft_trans_elems_destroy_abort(&ctx, te);
if (te->set->ops->abort &&
list_empty(&te->set->pending_update)) {
diff --git a/net/netfilter/nft_bitwise.c b/net/netfilter/nft_bitwise.c
index 0fc21b729eba..b6e6143989a9 100644
--- a/net/netfilter/nft_bitwise.c
+++ b/net/netfilter/nft_bitwise.c
@@ -42,8 +42,10 @@ static void nft_bitwise_eval_lshift(u32 *dst, const u32 *src,
u32 carry = 0;
for (i = DIV_ROUND_UP(priv->len, sizeof(u32)); i > 0; i--) {
- dst[i - 1] = (src[i - 1] << shift) | carry;
- carry = src[i - 1] >> (BITS_PER_TYPE(u32) - shift);
+ u32 tmp_src = src[i - 1];
+
+ dst[i - 1] = (tmp_src << shift) | carry;
+ carry = tmp_src >> (BITS_PER_TYPE(u32) - shift);
}
}
@@ -55,8 +57,10 @@ static void nft_bitwise_eval_rshift(u32 *dst, const u32 *src,
u32 carry = 0;
for (i = 0; i < DIV_ROUND_UP(priv->len, sizeof(u32)); i++) {
- dst[i] = carry | (src[i] >> shift);
- carry = src[i] << (BITS_PER_TYPE(u32) - shift);
+ u32 tmp_src = src[i];
+
+ dst[i] = carry | (tmp_src >> shift);
+ carry = tmp_src << (BITS_PER_TYPE(u32) - shift);
}
}
@@ -183,6 +187,9 @@ static int nft_bitwise_init(const struct nft_ctx *ctx,
if (err < 0)
return err;
+ if (nft_reg_overlap(priv->sreg, priv->dreg, priv->len))
+ return -EINVAL;
+
if (tb[NFTA_BITWISE_OP]) {
priv->op = ntohl(nla_get_be32(tb[NFTA_BITWISE_OP]));
switch (priv->op) {
diff --git a/net/netfilter/nft_byteorder.c b/net/netfilter/nft_byteorder.c
index 744878773dac..74548b4c114b 100644
--- a/net/netfilter/nft_byteorder.c
+++ b/net/netfilter/nft_byteorder.c
@@ -144,9 +144,16 @@ static int nft_byteorder_init(const struct nft_ctx *ctx,
if (err < 0)
return err;
- return nft_parse_register_store(ctx, tb[NFTA_BYTEORDER_DREG],
- &priv->dreg, NULL, NFT_DATA_VALUE,
- priv->len);
+ err = nft_parse_register_store(ctx, tb[NFTA_BYTEORDER_DREG],
+ &priv->dreg, NULL, NFT_DATA_VALUE,
+ priv->len);
+ if (err < 0)
+ return err;
+
+ if (nft_reg_overlap(priv->sreg, priv->dreg, priv->len))
+ return -EINVAL;
+
+ return 0;
}
static int nft_byteorder_dump(struct sk_buff *skb,
diff --git a/net/netfilter/nft_ct.c b/net/netfilter/nft_ct.c
index c238c97a9053..07a879f66fab 100644
--- a/net/netfilter/nft_ct.c
+++ b/net/netfilter/nft_ct.c
@@ -1128,7 +1128,6 @@ static void nft_ct_helper_obj_eval(struct nft_object *obj,
help = nf_ct_helper_ext_add(ct, GFP_ATOMIC);
if (help) {
rcu_assign_pointer(help->helper, to_assign);
- set_bit(IPS_HELPER_BIT, &ct->status);
if ((ct->status & IPS_NAT_MASK) && !nfct_seqadj(ct))
if (!nfct_seqadj_ext_add(ct))
@@ -1310,7 +1309,7 @@ static void nft_ct_expect_obj_eval(struct nft_object *obj,
&ct->tuplehash[!dir].tuple.src.u3,
&ct->tuplehash[!dir].tuple.dst.u3,
priv->l4proto, NULL, &priv->dport);
- exp->timeout.expires = jiffies + priv->timeout * HZ;
+ exp->timeout += priv->timeout * HZ;
if (nf_ct_expect_related(exp, 0) != 0)
regs->verdict.code = NF_DROP;
diff --git a/net/netfilter/nft_dynset.c b/net/netfilter/nft_dynset.c
index 0d78de9d2d75..59b14e6a2ca8 100644
--- a/net/netfilter/nft_dynset.c
+++ b/net/netfilter/nft_dynset.c
@@ -19,7 +19,8 @@ struct nft_dynset {
u8 sreg_key;
u8 sreg_data;
bool invert;
- bool expr;
+ bool expr:1,
+ override_exprs:1;
u8 num_exprs;
u64 timeout;
struct nft_expr *expr_array[NFT_SET_EXPR_MAX];
@@ -257,6 +258,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
priv->num_exprs++;
priv->expr_array[0] = dynset_expr;
+ priv->override_exprs = true;
if (set->num_exprs > 1 ||
(set->num_exprs == 1 &&
@@ -289,6 +291,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
}
priv->expr_array[i] = dynset_expr;
priv->num_exprs++;
+ priv->override_exprs = true;
if (set->num_exprs) {
if (i >= set->num_exprs) {
@@ -307,9 +310,8 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
goto err_expr_free;
}
} else if (set->num_exprs > 0) {
- err = nft_set_elem_expr_clone(ctx, set, priv->expr_array);
- if (err < 0)
- return err;
+ for (i = 0; i < set->num_exprs; i++)
+ priv->expr_array[i] = set->exprs[i];
priv->num_exprs = set->num_exprs;
}
@@ -339,8 +341,10 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
return 0;
err_expr_free:
- for (i = 0; i < priv->num_exprs; i++)
- nft_expr_destroy(ctx, priv->expr_array[i]);
+ if (priv->override_exprs) {
+ for (i = 0; i < priv->num_exprs; i++)
+ nft_expr_destroy(ctx, priv->expr_array[i]);
+ }
return err;
}
@@ -367,9 +371,10 @@ static void nft_dynset_destroy(const struct nft_ctx *ctx,
struct nft_dynset *priv = nft_expr_priv(expr);
int i;
- for (i = 0; i < priv->num_exprs; i++)
- nft_expr_destroy(ctx, priv->expr_array[i]);
-
+ if (priv->override_exprs) {
+ for (i = 0; i < priv->num_exprs; i++)
+ nft_expr_destroy(ctx, priv->expr_array[i]);
+ }
nf_tables_destroy_set(ctx, priv->set);
}
@@ -393,7 +398,7 @@ static int nft_dynset_dump(struct sk_buff *skb,
nf_jiffies64_to_msecs(priv->timeout),
NFTA_DYNSET_PAD))
goto nla_put_failure;
- if (priv->set->num_exprs == 0) {
+ if (priv->set->num_exprs == 0 || priv->override_exprs) {
if (priv->num_exprs == 1) {
if (nft_expr_dump(skb, NFTA_DYNSET_EXPR,
priv->expr_array[0], reset))
diff --git a/net/netfilter/nft_flow_offload.c b/net/netfilter/nft_flow_offload.c
index 4ddf3eb00225..99d3454f9e3b 100644
--- a/net/netfilter/nft_flow_offload.c
+++ b/net/netfilter/nft_flow_offload.c
@@ -150,10 +150,11 @@ static void nft_dev_path_info(const struct net_device_path_stack *stack,
info->num_encaps++;
break;
case DEV_PATH_BR_VLAN_UNTAG:
- if (WARN_ON_ONCE(info->num_encaps-- == 0)) {
+ if (info->num_encaps == 0) {
info->indev = NULL;
break;
}
+ info->num_encaps--;
break;
case DEV_PATH_BR_VLAN_KEEP:
break;
diff --git a/net/netfilter/nft_set_pipapo.c b/net/netfilter/nft_set_pipapo.c
index 9d2cfdc94a52..96cd9e17bd64 100644
--- a/net/netfilter/nft_set_pipapo.c
+++ b/net/netfilter/nft_set_pipapo.c
@@ -2188,13 +2188,20 @@ static void nft_pipapo_walk(const struct nft_ctx *ctx, struct nft_set *set,
const struct nft_pipapo_match *m;
switch (iter->type) {
- case NFT_ITER_UPDATE:
+ case NFT_ITER_UPDATE_CLONE:
m = pipapo_maybe_clone(set);
if (!m) {
iter->err = -ENOMEM;
return;
}
-
+ nft_pipapo_do_walk(ctx, set, m, iter);
+ break;
+ case NFT_ITER_UPDATE:
+ if (priv->clone)
+ m = priv->clone;
+ else
+ m = rcu_dereference_protected(priv->match,
+ nft_pipapo_transaction_mutex_held(set));
nft_pipapo_do_walk(ctx, set, m, iter);
break;
case NFT_ITER_READ:
diff --git a/net/netfilter/x_tables.c b/net/netfilter/x_tables.c
index efe7b7d71e7f..fb777d2c7375 100644
--- a/net/netfilter/x_tables.c
+++ b/net/netfilter/x_tables.c
@@ -1404,11 +1404,9 @@ struct xt_counters *xt_counters_alloc(unsigned int counters)
}
EXPORT_SYMBOL(xt_counters_alloc);
-struct xt_table_info *
-xt_replace_table(struct xt_table *table,
- unsigned int num_counters,
- struct xt_table_info *newinfo,
- int *error)
+static struct xt_table_info *
+do_replace_table(struct xt_table *table, unsigned int num_counters,
+ struct xt_table_info *newinfo, int *error)
{
struct xt_table_info *private;
unsigned int cpu;
@@ -1463,30 +1461,54 @@ xt_replace_table(struct xt_table *table,
}
}
- audit_log_nfcfg(table->name, table->af, private->number,
- !private->number ? AUDIT_XT_OP_REGISTER :
- AUDIT_XT_OP_REPLACE,
- GFP_KERNEL);
+ return private;
+}
+
+struct xt_table_info *
+xt_replace_table(struct xt_table *table, unsigned int num_counters,
+ struct xt_table_info *newinfo,
+ int *error)
+{
+ struct xt_table_info *private;
+
+ private = do_replace_table(table, num_counters, newinfo, error);
+ if (private)
+ audit_log_nfcfg(table->name, table->af, private->number,
+ AUDIT_XT_OP_REPLACE,
+ GFP_KERNEL);
+
return private;
}
EXPORT_SYMBOL_GPL(xt_replace_table);
struct xt_table *xt_register_table(struct net *net,
const struct xt_table *input_table,
+ const struct nf_hook_ops *template_ops,
struct xt_table_info *bootstrap,
struct xt_table_info *newinfo)
{
struct xt_pernet *xt_net = net_generic(net, xt_pernet_id);
+ struct xt_table *t, *table = NULL;
+ struct nf_hook_ops *ops = NULL;
struct xt_table_info *private;
- struct xt_table *t, *table;
- int ret;
+ unsigned int num_ops;
+ int ret = -EINVAL;
+
+ num_ops = hweight32(input_table->valid_hooks);
+ if (num_ops == 0)
+ goto out;
+
+ ret = -ENOMEM;
+ if (template_ops) {
+ ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
+ if (!ops)
+ goto out;
+ }
/* Don't add one object to multiple lists. */
table = kmemdup(input_table, sizeof(struct xt_table), GFP_KERNEL);
- if (!table) {
- ret = -ENOMEM;
+ if (!table)
goto out;
- }
mutex_lock(&xt[table->af].mutex);
/* Don't autoload: we'd eat our tail... */
@@ -1500,7 +1522,7 @@ struct xt_table *xt_register_table(struct net *net,
/* Simplifies replace_table code. */
table->private = bootstrap;
- if (!xt_replace_table(table, 0, newinfo, &ret))
+ if (!do_replace_table(table, 0, newinfo, &ret))
goto unlock;
private = table->private;
@@ -1509,14 +1531,37 @@ struct xt_table *xt_register_table(struct net *net,
/* save number of initial entries */
private->initial_entries = private->number;
+ if (ops) {
+ int i;
+
+ for (i = 0; i < num_ops; i++)
+ ops[i].priv = table;
+
+ ret = nf_register_net_hooks(net, ops, num_ops);
+ if (ret != 0) {
+ mutex_unlock(&xt[table->af].mutex);
+ /* nf_register_net_hooks() might have published a
+ * base chain before internal error unwind.
+ */
+ synchronize_rcu();
+ goto out;
+ }
+
+ table->ops = ops;
+ }
+
+ audit_log_nfcfg(table->name, table->af, private->number,
+ AUDIT_XT_OP_REGISTER, GFP_KERNEL);
+
list_add(&table->list, &xt_net->tables[table->af]);
mutex_unlock(&xt[table->af].mutex);
return table;
unlock:
mutex_unlock(&xt[table->af].mutex);
- kfree(table);
out:
+ kfree(table);
+ kfree(ops);
return ERR_PTR(ret);
}
EXPORT_SYMBOL_GPL(xt_register_table);
diff --git a/net/openvswitch/vport-internal_dev.c b/net/openvswitch/vport-internal_dev.c
index 5858d65ea1a9..7b05a423a9ab 100644
--- a/net/openvswitch/vport-internal_dev.c
+++ b/net/openvswitch/vport-internal_dev.c
@@ -149,7 +149,7 @@ static struct vport *internal_dev_create(const struct vport_parms *parms)
/* Restrict bridge port to current netns. */
if (vport->port_no == OVSP_LOCAL)
- vport->dev->netns_local = true;
+ vport->dev->netns_immutable = true;
rtnl_lock();
err = register_netdevice(vport->dev);
diff --git a/net/phonet/socket.c b/net/phonet/socket.c
index 5ce0b3ee5def..11250d2d1dda 100644
--- a/net/phonet/socket.c
+++ b/net/phonet/socket.c
@@ -208,9 +208,15 @@ static int pn_socket_autobind(struct socket *sock)
sa.spn_family = AF_PHONET;
err = pn_socket_bind(sock, (struct sockaddr *)&sa,
sizeof(struct sockaddr_pn));
- if (err != -EINVAL)
+ /*
+ * pn_socket_bind() also returns -EINVAL when sk_state != TCP_CLOSE
+ * without a prior bind, so -EINVAL alone is not sufficient to infer
+ * that the socket was already bound. Only treat it as "already
+ * bound" when the port is non-zero; otherwise propagate the error
+ * instead of crashing the kernel.
+ */
+ if (err != -EINVAL || unlikely(!pn_port(pn_sk(sock->sk)->sobject)))
return err;
- BUG_ON(!pn_port(pn_sk(sock->sk)->sobject));
return 0; /* socket was already bound */
}
diff --git a/net/qrtr/mhi.c b/net/qrtr/mhi.c
index 69f53625a049..80e341d2f8a4 100644
--- a/net/qrtr/mhi.c
+++ b/net/qrtr/mhi.c
@@ -24,13 +24,25 @@ static void qcom_mhi_qrtr_dl_callback(struct mhi_device *mhi_dev,
struct qrtr_mhi_dev *qdev = dev_get_drvdata(&mhi_dev->dev);
int rc;
- if (!qdev || mhi_res->transaction_status)
+ if (!qdev || (mhi_res->transaction_status && mhi_res->transaction_status != -ENOTCONN))
return;
+ /* Channel got reset. So just free the buffer */
+ if (mhi_res->transaction_status == -ENOTCONN) {
+ devm_kfree(&mhi_dev->dev, mhi_res->buf_addr);
+ return;
+ }
+
rc = qrtr_endpoint_post(&qdev->ep, mhi_res->buf_addr,
mhi_res->bytes_xferd);
if (rc == -EINVAL)
dev_err(qdev->dev, "invalid ipcrouter packet\n");
+
+ /* Done with the buffer, now recycle it for future use */
+ rc = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, mhi_res->buf_addr,
+ mhi_dev->mhi_cntrl->buffer_len, MHI_EOT);
+ if (rc)
+ dev_err(&mhi_dev->dev, "Failed to recycle the buffer: %d\n", rc);
}
/* From QRTR to MHI */
@@ -72,6 +84,29 @@ static int qcom_mhi_qrtr_send(struct qrtr_endpoint *ep, struct sk_buff *skb)
return rc;
}
+static int qcom_mhi_qrtr_queue_dl_buffers(struct mhi_device *mhi_dev)
+{
+ u32 free_desc;
+ void *buf;
+ int ret;
+
+ free_desc = mhi_get_free_desc_count(mhi_dev, DMA_FROM_DEVICE);
+ while (free_desc--) {
+ buf = devm_kmalloc(&mhi_dev->dev, mhi_dev->mhi_cntrl->buffer_len, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, mhi_dev->mhi_cntrl->buffer_len,
+ MHI_EOT);
+ if (ret) {
+ dev_err(&mhi_dev->dev, "Failed to queue buffer: %d\n", ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
static int qcom_mhi_qrtr_probe(struct mhi_device *mhi_dev,
const struct mhi_device_id *id)
{
@@ -87,20 +122,30 @@ static int qcom_mhi_qrtr_probe(struct mhi_device *mhi_dev,
qdev->ep.xmit = qcom_mhi_qrtr_send;
dev_set_drvdata(&mhi_dev->dev, qdev);
- rc = qrtr_endpoint_register(&qdev->ep, QRTR_EP_NID_AUTO);
- if (rc)
- return rc;
/* start channels */
- rc = mhi_prepare_for_transfer_autoqueue(mhi_dev);
- if (rc) {
- qrtr_endpoint_unregister(&qdev->ep);
+ rc = mhi_prepare_for_transfer(mhi_dev);
+ if (rc)
return rc;
- }
+
+ rc = qrtr_endpoint_register(&qdev->ep, QRTR_EP_NID_AUTO);
+ if (rc)
+ goto err_unprepare;
+
+ rc = qcom_mhi_qrtr_queue_dl_buffers(mhi_dev);
+ if (rc)
+ goto err_unregister;
dev_dbg(qdev->dev, "Qualcomm MHI QRTR driver probed\n");
return 0;
+
+err_unregister:
+ qrtr_endpoint_unregister(&qdev->ep);
+err_unprepare:
+ mhi_unprepare_from_transfer(mhi_dev);
+
+ return rc;
}
static void qcom_mhi_qrtr_remove(struct mhi_device *mhi_dev)
@@ -151,11 +196,13 @@ static int __maybe_unused qcom_mhi_qrtr_pm_resume_early(struct device *dev)
if (state == MHI_STATE_M3)
return 0;
- rc = mhi_prepare_for_transfer_autoqueue(mhi_dev);
- if (rc)
+ rc = mhi_prepare_for_transfer(mhi_dev);
+ if (rc) {
dev_err(dev, "failed to prepare for autoqueue transfer %d\n", rc);
+ return rc;
+ }
- return rc;
+ return qcom_mhi_qrtr_queue_dl_buffers(mhi_dev);
}
static const struct dev_pm_ops qcom_mhi_qrtr_pm_ops = {
diff --git a/net/rxrpc/af_rxrpc.c b/net/rxrpc/af_rxrpc.c
index ec4fe6af218f..0d86c7c37754 100644
--- a/net/rxrpc/af_rxrpc.c
+++ b/net/rxrpc/af_rxrpc.c
@@ -265,11 +265,15 @@ static int rxrpc_listen(struct socket *sock, int backlog)
* @gfp: Allocation flags
*
* Lookup or create a remote transport endpoint record for the specified
- * address and return it with a ref held.
+ * address.
+ *
+ * Return: The peer record found with a reference or a negative error code if
+ * the address is invalid or unsupported.
*/
struct rxrpc_peer *rxrpc_kernel_lookup_peer(struct socket *sock,
struct sockaddr_rxrpc *srx, gfp_t gfp)
{
+ struct rxrpc_peer *peer;
struct rxrpc_sock *rx = rxrpc_sk(sock->sk);
int ret;
@@ -277,15 +281,18 @@ struct rxrpc_peer *rxrpc_kernel_lookup_peer(struct socket *sock,
if (ret < 0)
return ERR_PTR(ret);
- return rxrpc_lookup_peer(rx->local, srx, gfp);
+ peer = rxrpc_lookup_peer(rx->local, srx, gfp);
+ return peer ?: ERR_PTR(-ENOMEM);
}
EXPORT_SYMBOL(rxrpc_kernel_lookup_peer);
/**
* rxrpc_kernel_get_peer - Get a reference on a peer
- * @peer: The peer to get a reference on.
+ * @peer: The peer to get a reference on (may be NULL).
+ *
+ * Get a reference for a remote peer record (if not NULL).
*
- * Get a record for the remote peer in a call.
+ * Return: The @peer argument.
*/
struct rxrpc_peer *rxrpc_kernel_get_peer(struct rxrpc_peer *peer)
{
@@ -296,6 +303,8 @@ EXPORT_SYMBOL(rxrpc_kernel_get_peer);
/**
* rxrpc_kernel_put_peer - Allow a kernel app to drop a peer reference
* @peer: The peer to drop a ref on
+ *
+ * Drop a reference on a peer record.
*/
void rxrpc_kernel_put_peer(struct rxrpc_peer *peer)
{
@@ -320,10 +329,12 @@ EXPORT_SYMBOL(rxrpc_kernel_put_peer);
*
* Allow a kernel service to begin a call on the nominated socket. This just
* sets up all the internal tracking structures and allocates connection and
- * call IDs as appropriate. The call to be used is returned.
+ * call IDs as appropriate.
*
* The default socket destination address and security may be overridden by
* supplying @srx and @key.
+ *
+ * Return: The new call or an error code.
*/
struct rxrpc_call *rxrpc_kernel_begin_call(struct socket *sock,
struct rxrpc_peer *peer,
@@ -437,6 +448,8 @@ EXPORT_SYMBOL(rxrpc_kernel_put_call);
*
* Allow a kernel service to find out whether a call is still alive - whether
* it has completed successfully and all received data has been consumed.
+ *
+ * Return: %true if the call is still ongoing and %false if it has completed.
*/
bool rxrpc_kernel_check_life(const struct socket *sock,
const struct rxrpc_call *call)
@@ -456,6 +469,8 @@ EXPORT_SYMBOL(rxrpc_kernel_check_life);
*
* Allow a kernel service to retrieve the epoch value from a service call to
* see if the client at the other end rebooted.
+ *
+ * Return: The epoch of the call's connection.
*/
u32 rxrpc_kernel_get_epoch(struct socket *sock, struct rxrpc_call *call)
{
diff --git a/net/rxrpc/key.c b/net/rxrpc/key.c
index 73659197edff..1dbf7851f0fc 100644
--- a/net/rxrpc/key.c
+++ b/net/rxrpc/key.c
@@ -536,6 +536,8 @@ EXPORT_SYMBOL(rxrpc_get_server_data_key);
*
* Generate a null RxRPC key that can be used to indicate anonymous security is
* required for a particular domain.
+ *
+ * Return: The new key or a negative error code.
*/
struct key *rxrpc_get_null_key(const char *keyname)
{
diff --git a/net/rxrpc/peer_object.c b/net/rxrpc/peer_object.c
index 3af39fc67d34..e329be68434d 100644
--- a/net/rxrpc/peer_object.c
+++ b/net/rxrpc/peer_object.c
@@ -463,6 +463,8 @@ void rxrpc_destroy_all_peers(struct rxrpc_net *rxnet)
* @call: The call to query
*
* Get a record for the remote peer in a call.
+ *
+ * Return: The call's peer record.
*/
struct rxrpc_peer *rxrpc_kernel_get_call_peer(struct socket *sock, struct rxrpc_call *call)
{
@@ -474,7 +476,9 @@ EXPORT_SYMBOL(rxrpc_kernel_get_call_peer);
* rxrpc_kernel_get_srtt - Get a call's peer smoothed RTT
* @peer: The peer to query
*
- * Get the call's peer smoothed RTT in uS or UINT_MAX if we have no samples.
+ * Get the call's peer smoothed RTT.
+ *
+ * Return: The RTT in uS or %UINT_MAX if we have no samples.
*/
unsigned int rxrpc_kernel_get_srtt(const struct rxrpc_peer *peer)
{
@@ -487,7 +491,10 @@ EXPORT_SYMBOL(rxrpc_kernel_get_srtt);
* @peer: The peer to query
*
* Get a pointer to the address from a peer record. The caller is responsible
- * for making sure that the address is not deallocated.
+ * for making sure that the address is not deallocated. A fake address will be
+ * substituted if %peer in NULL.
+ *
+ * Return: The rxrpc address record or a fake record.
*/
const struct sockaddr_rxrpc *rxrpc_kernel_remote_srx(const struct rxrpc_peer *peer)
{
@@ -500,7 +507,10 @@ EXPORT_SYMBOL(rxrpc_kernel_remote_srx);
* @peer: The peer to query
*
* Get a pointer to the transport address from a peer record. The caller is
- * responsible for making sure that the address is not deallocated.
+ * responsible for making sure that the address is not deallocated. A fake
+ * address will be substituted if %peer in NULL.
+ *
+ * Return: The transport address record or a fake record.
*/
const struct sockaddr *rxrpc_kernel_remote_addr(const struct rxrpc_peer *peer)
{
@@ -515,7 +525,9 @@ EXPORT_SYMBOL(rxrpc_kernel_remote_addr);
* @app_data: The data to set
*
* Set the app-specific data on a peer. AF_RXRPC makes no effort to retain
- * anything the data might refer to. The previous app_data is returned.
+ * anything the data might refer to.
+ *
+ * Return: The previous app_data.
*/
unsigned long rxrpc_kernel_set_peer_data(struct rxrpc_peer *peer, unsigned long app_data)
{
@@ -528,6 +540,8 @@ EXPORT_SYMBOL(rxrpc_kernel_set_peer_data);
* @peer: The peer to query
*
* Retrieve the app-specific data from a peer.
+ *
+ * Return: The peer's app data.
*/
unsigned long rxrpc_kernel_get_peer_data(const struct rxrpc_peer *peer)
{
diff --git a/net/rxrpc/proc.c b/net/rxrpc/proc.c
index 2dd9567cb1cf..7a2279cd11a7 100644
--- a/net/rxrpc/proc.c
+++ b/net/rxrpc/proc.c
@@ -10,6 +10,10 @@
#include <net/af_rxrpc.h>
#include "ar-internal.h"
+#define RXRPC_PROC_ADDRBUF_SIZE \
+ (sizeof("[xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255]") + \
+ sizeof(":12345"))
+
static const char *const rxrpc_conn_states[RXRPC_CONN__NR_STATES] = {
[RXRPC_CONN_UNUSED] = "Unused ",
[RXRPC_CONN_CLIENT_UNSECURED] = "ClUnsec ",
@@ -53,7 +57,7 @@ static int rxrpc_call_seq_show(struct seq_file *seq, void *v)
struct rxrpc_net *rxnet = rxrpc_net(seq_file_net(seq));
enum rxrpc_call_state state;
rxrpc_seq_t acks_hard_ack;
- char lbuff[50], rbuff[50];
+ char lbuff[RXRPC_PROC_ADDRBUF_SIZE], rbuff[RXRPC_PROC_ADDRBUF_SIZE];
long timeout = 0;
if (v == &rxnet->calls) {
@@ -69,11 +73,11 @@ static int rxrpc_call_seq_show(struct seq_file *seq, void *v)
local = call->local;
if (local)
- sprintf(lbuff, "%pISpc", &local->srx.transport);
+ scnprintf(lbuff, sizeof(lbuff), "%pISpc", &local->srx.transport);
else
strcpy(lbuff, "no_local");
- sprintf(rbuff, "%pISpc", &call->dest_srx.transport);
+ scnprintf(rbuff, sizeof(rbuff), "%pISpc", &call->dest_srx.transport);
state = rxrpc_call_state(call);
if (state != RXRPC_CALL_SERVER_PREALLOC)
@@ -142,7 +146,7 @@ static int rxrpc_connection_seq_show(struct seq_file *seq, void *v)
struct rxrpc_connection *conn;
struct rxrpc_net *rxnet = rxrpc_net(seq_file_net(seq));
const char *state;
- char lbuff[50], rbuff[50];
+ char lbuff[RXRPC_PROC_ADDRBUF_SIZE], rbuff[RXRPC_PROC_ADDRBUF_SIZE];
if (v == &rxnet->conn_proc_list) {
seq_puts(seq,
@@ -161,8 +165,8 @@ static int rxrpc_connection_seq_show(struct seq_file *seq, void *v)
goto print;
}
- sprintf(lbuff, "%pISpc", &conn->local->srx.transport);
- sprintf(rbuff, "%pISpc", &conn->peer->srx.transport);
+ scnprintf(lbuff, sizeof(lbuff), "%pISpc", &conn->local->srx.transport);
+ scnprintf(rbuff, sizeof(rbuff), "%pISpc", &conn->peer->srx.transport);
print:
state = rxrpc_is_conn_aborted(conn) ?
rxrpc_call_completions[conn->completion] :
@@ -228,7 +232,7 @@ static int rxrpc_bundle_seq_show(struct seq_file *seq, void *v)
{
struct rxrpc_bundle *bundle;
struct rxrpc_net *rxnet = rxrpc_net(seq_file_net(seq));
- char lbuff[50], rbuff[50];
+ char lbuff[RXRPC_PROC_ADDRBUF_SIZE], rbuff[RXRPC_PROC_ADDRBUF_SIZE];
if (v == &rxnet->bundle_proc_list) {
seq_puts(seq,
@@ -242,8 +246,8 @@ static int rxrpc_bundle_seq_show(struct seq_file *seq, void *v)
bundle = list_entry(v, struct rxrpc_bundle, proc_link);
- sprintf(lbuff, "%pISpc", &bundle->local->srx.transport);
- sprintf(rbuff, "%pISpc", &bundle->peer->srx.transport);
+ scnprintf(lbuff, sizeof(lbuff), "%pISpc", &bundle->local->srx.transport);
+ scnprintf(rbuff, sizeof(rbuff), "%pISpc", &bundle->peer->srx.transport);
seq_printf(seq,
"UDP %-47.47s %-47.47s %4x %3u %3d"
" %c%c%c %08x | %08x %08x %08x %08x %08x\n",
@@ -279,7 +283,7 @@ static int rxrpc_peer_seq_show(struct seq_file *seq, void *v)
{
struct rxrpc_peer *peer;
time64_t now;
- char lbuff[50], rbuff[50];
+ char lbuff[RXRPC_PROC_ADDRBUF_SIZE], rbuff[RXRPC_PROC_ADDRBUF_SIZE];
if (v == SEQ_START_TOKEN) {
seq_puts(seq,
@@ -292,9 +296,9 @@ static int rxrpc_peer_seq_show(struct seq_file *seq, void *v)
peer = list_entry(v, struct rxrpc_peer, hash_link);
- sprintf(lbuff, "%pISpc", &peer->local->srx.transport);
+ scnprintf(lbuff, sizeof(lbuff), "%pISpc", &peer->local->srx.transport);
- sprintf(rbuff, "%pISpc", &peer->srx.transport);
+ scnprintf(rbuff, sizeof(rbuff), "%pISpc", &peer->srx.transport);
now = ktime_get_seconds();
seq_printf(seq,
@@ -404,7 +408,7 @@ const struct seq_operations rxrpc_peer_seq_ops = {
static int rxrpc_local_seq_show(struct seq_file *seq, void *v)
{
struct rxrpc_local *local;
- char lbuff[50];
+ char lbuff[RXRPC_PROC_ADDRBUF_SIZE];
if (v == SEQ_START_TOKEN) {
seq_puts(seq,
@@ -415,7 +419,7 @@ static int rxrpc_local_seq_show(struct seq_file *seq, void *v)
local = hlist_entry(v, struct rxrpc_local, link);
- sprintf(lbuff, "%pISpc", &local->srx.transport);
+ scnprintf(lbuff, sizeof(lbuff), "%pISpc", &local->srx.transport);
seq_printf(seq,
"UDP %-47.47s %3u %3u %3u\n",
diff --git a/net/rxrpc/recvmsg.c b/net/rxrpc/recvmsg.c
index beb0eb426592..bb140d7005c7 100644
--- a/net/rxrpc/recvmsg.c
+++ b/net/rxrpc/recvmsg.c
@@ -546,14 +546,14 @@ int rxrpc_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
* @_service: Where to store the actual service ID (may be upgraded)
*
* Allow a kernel service to receive data and pick up information about the
- * state of a call. Returns 0 if got what was asked for and there's more
- * available, 1 if we got what was asked for and we're at the end of the data
- * and -EAGAIN if we need more data.
+ * state of a call. Note that *@_abort should also be initialised to %0.
*
- * Note that we may return -EAGAIN to drain empty packets at the end of the
- * data, even if we've already copied over the requested data.
+ * Note that we may return %-EAGAIN to drain empty packets at the end
+ * of the data, even if we've already copied over the requested data.
*
- * *_abort should also be initialised to 0.
+ * Return: %0 if got what was asked for and there's more available, %1
+ * if we got what was asked for and we're at the end of the data and
+ * %-EAGAIN if we need more data.
*/
int rxrpc_kernel_recv_data(struct socket *sock, struct rxrpc_call *call,
struct iov_iter *iter, size_t *_len,
diff --git a/net/rxrpc/sendmsg.c b/net/rxrpc/sendmsg.c
index 1d8065468e90..c9b85848e825 100644
--- a/net/rxrpc/sendmsg.c
+++ b/net/rxrpc/sendmsg.c
@@ -745,6 +745,8 @@ int rxrpc_do_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, size_t len)
* appropriate to sending data. No control data should be supplied in @msg,
* nor should an address be supplied. MSG_MORE should be flagged if there's
* more data to come, otherwise this data will end the transmission phase.
+ *
+ * Return: %0 if successful and a negative error code otherwise.
*/
int rxrpc_kernel_send_data(struct socket *sock, struct rxrpc_call *call,
struct msghdr *msg, size_t len,
@@ -780,8 +782,9 @@ EXPORT_SYMBOL(rxrpc_kernel_send_data);
* @error: Local error value
* @why: Indication as to why.
*
- * Allow a kernel service to abort a call, if it's still in an abortable state
- * and return true if the call was aborted, false if it was already complete.
+ * Allow a kernel service to abort a call if it's still in an abortable state.
+ *
+ * Return: %true if the call was aborted, %false if it was already complete.
*/
bool rxrpc_kernel_abort_call(struct socket *sock, struct rxrpc_call *call,
u32 abort_code, int error, enum rxrpc_abort_reason why)
diff --git a/net/rxrpc/server_key.c b/net/rxrpc/server_key.c
index 9d95a6055b70..7501fc7b38b8 100644
--- a/net/rxrpc/server_key.c
+++ b/net/rxrpc/server_key.c
@@ -155,6 +155,8 @@ int rxrpc_server_keyring(struct rxrpc_sock *rx, sockptr_t optval, int optlen)
*
* Set the server security keyring on an rxrpc socket. This is used to provide
* the encryption keys for a kernel service.
+ *
+ * Return: %0 if successful and a negative error code otherwise.
*/
int rxrpc_sock_set_security_keyring(struct sock *sk, struct key *keyring)
{
diff --git a/net/sched/act_api.c b/net/sched/act_api.c
index ca851146f695..94feb3961dcd 100644
--- a/net/sched/act_api.c
+++ b/net/sched/act_api.c
@@ -1570,7 +1570,7 @@ void tcf_action_update_stats(struct tc_action *a, u64 bytes, u64 packets,
if (a->cpu_bstats) {
_bstats_update(this_cpu_ptr(a->cpu_bstats), bytes, packets);
- this_cpu_ptr(a->cpu_qstats)->drops += drops;
+ this_cpu_add(a->cpu_qstats->drops, drops);
if (hw)
_bstats_update(this_cpu_ptr(a->cpu_bstats_hw),
diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c
index e99b9269ea1a..06d8f78b7368 100644
--- a/net/sched/act_bpf.c
+++ b/net/sched/act_bpf.c
@@ -76,7 +76,7 @@ TC_INDIRECT_SCOPE int tcf_bpf_act(struct sk_buff *skb,
break;
case TC_ACT_SHOT:
action = filter_res;
- qstats_drop_inc(this_cpu_ptr(prog->common.cpu_qstats));
+ qstats_cpu_drop_inc(prog->common.cpu_qstats);
break;
case TC_ACT_UNSPEC:
action = prog->tcf_action;
diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
index a90d7959fc43..b0d66ba80baa 100644
--- a/net/sched/act_ife.c
+++ b/net/sched/act_ife.c
@@ -28,6 +28,7 @@
#include <uapi/linux/tc_act/tc_ife.h>
#include <net/tc_act/tc_ife.h>
#include <linux/etherdevice.h>
+#include <linux/if_arp.h>
#include <net/ife.h>
#include <net/tc_wrapper.h>
@@ -723,11 +724,11 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a,
tcf_lastuse_update(&ife->tcf_tm);
if (skb_at_tc_ingress(skb))
- skb_push(skb, skb->dev->hard_header_len);
+ skb_push(skb, ETH_HLEN);
tlv_data = ife_decode(skb, &metalen);
if (unlikely(!tlv_data)) {
- qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats));
+ qstats_cpu_drop_inc(ife->common.cpu_qstats);
return TC_ACT_SHOT;
}
@@ -740,7 +741,7 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a,
curr_data = ife_tlv_meta_decode(tlv_data, ifehdr_end, &mtype,
&dlen, NULL);
if (!curr_data) {
- qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats));
+ qstats_cpu_drop_inc(ife->common.cpu_qstats);
return TC_ACT_SHOT;
}
@@ -755,7 +756,7 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a,
}
if (WARN_ON(tlv_data != ifehdr_end)) {
- qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats));
+ qstats_cpu_drop_inc(ife->common.cpu_qstats);
return TC_ACT_SHOT;
}
@@ -795,7 +796,7 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a,
where ORIGDATA = original ethernet header ...
*/
u16 metalen = ife_get_sz(skb, p);
- int hdrm = metalen + skb->dev->hard_header_len + IFE_METAHDRLEN;
+ int hdrm = metalen + ETH_HLEN + IFE_METAHDRLEN;
unsigned int skboff = 0;
int new_len = skb->len + hdrm;
bool exceed_mtu = false;
@@ -821,12 +822,12 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a,
* so lets be conservative.. */
if ((action == TC_ACT_SHOT) || exceed_mtu) {
drop:
- qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats));
+ qstats_cpu_drop_inc(ife->common.cpu_qstats);
return TC_ACT_SHOT;
}
if (skb_at_tc_ingress(skb))
- skb_push(skb, skb->dev->hard_header_len);
+ skb_push(skb, ETH_HLEN);
ife_meta = ife_encode(skb, metalen);
if (!ife_meta)
@@ -856,11 +857,27 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a,
oethh->h_proto = htons(p->eth_type);
if (skb_at_tc_ingress(skb))
- skb_pull(skb, skb->dev->hard_header_len);
+ skb_pull(skb, ETH_HLEN);
return action;
}
+/* IFE encapsulates the original Ethernet header and, on decode, expects to
+ * find one, so it can only ever work on skbs that carry one. Loopback carries
+ * Ethernet header as well, so it qualifies here.
+ * At ingress, also verify that the L2 header about to be pushed back really
+ * is an Ethernet header because the skb could've been redirected with mirred
+ * from a non-Ethernet device.
+ */
+static bool tcf_ife_is_eth_skb(const struct sk_buff *skb)
+{
+ if (skb->dev->type != ARPHRD_ETHER &&
+ skb->dev->type != ARPHRD_LOOPBACK)
+ return false;
+
+ return !skb_at_tc_ingress(skb) || skb->mac_len == ETH_HLEN;
+}
+
TC_INDIRECT_SCOPE int tcf_ife_act(struct sk_buff *skb,
const struct tc_action *a,
struct tcf_result *res)
@@ -869,6 +886,13 @@ TC_INDIRECT_SCOPE int tcf_ife_act(struct sk_buff *skb,
struct tcf_ife_params *p;
int ret;
+ if (unlikely(!tcf_ife_is_eth_skb(skb))) {
+ bstats_update(this_cpu_ptr(ife->common.cpu_bstats), skb);
+ tcf_lastuse_update(&ife->tcf_tm);
+ qstats_cpu_drop_inc(ife->common.cpu_qstats);
+ return TC_ACT_SHOT;
+ }
+
p = rcu_dereference_bh(ife->params);
if (p->flags & IFE_ENCODE) {
ret = tcf_ife_encode(skb, a, res, p);
diff --git a/net/sched/act_mpls.c b/net/sched/act_mpls.c
index 44a37a71ae92..24d9cb7c1c4d 100644
--- a/net/sched/act_mpls.c
+++ b/net/sched/act_mpls.c
@@ -125,7 +125,7 @@ TC_INDIRECT_SCOPE int tcf_mpls_act(struct sk_buff *skb,
return ret;
drop:
- qstats_drop_inc(this_cpu_ptr(m->common.cpu_qstats));
+ qstats_cpu_drop_inc(m->common.cpu_qstats);
return TC_ACT_SHOT;
}
diff --git a/net/sched/act_police.c b/net/sched/act_police.c
index 1ad8a7948690..098eddcd36b6 100644
--- a/net/sched/act_police.c
+++ b/net/sched/act_police.c
@@ -317,7 +317,7 @@ TC_INDIRECT_SCOPE int tcf_police_act(struct sk_buff *skb,
qstats_overlimit_inc(this_cpu_ptr(police->common.cpu_qstats));
inc_drops:
if (ret == TC_ACT_SHOT)
- qstats_drop_inc(this_cpu_ptr(police->common.cpu_qstats));
+ qstats_cpu_drop_inc(police->common.cpu_qstats);
end:
return ret;
}
diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c
index 4cddf2674373..c52f25e0c0a0 100644
--- a/net/sched/act_skbedit.c
+++ b/net/sched/act_skbedit.c
@@ -85,7 +85,7 @@ TC_INDIRECT_SCOPE int tcf_skbedit_act(struct sk_buff *skb,
return params->action;
err:
- qstats_drop_inc(this_cpu_ptr(d->common.cpu_qstats));
+ qstats_cpu_drop_inc(d->common.cpu_qstats);
return TC_ACT_SHOT;
}
diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c
index 748be0e76a49..1e2dccd6cc4e 100644
--- a/net/sched/cls_api.c
+++ b/net/sched/cls_api.c
@@ -3375,7 +3375,7 @@ int tcf_exts_init_ex(struct tcf_exts *exts, struct net *net, int action,
*/
exts->net = net;
exts->actions = kcalloc(TCA_ACT_MAX_PRIO, sizeof(struct tc_action *),
- GFP_KERNEL);
+ GFP_KERNEL_ACCOUNT);
if (!exts->actions)
return -ENOMEM;
#endif
@@ -4071,9 +4071,7 @@ struct sk_buff *tcf_qevent_handle(struct tcf_qevent *qe, struct Qdisc *sch, stru
fl = rcu_dereference_bh(qe->filter_chain);
- switch (tcf_classify(skb, NULL, fl, &cl_res, false)) {
- case TC_ACT_REDIRECT:
- fallthrough;
+ switch (tcf_classify_qdisc(skb, fl, &cl_res, false)) {
case TC_ACT_SHOT:
qdisc_qstats_drop(sch);
__qdisc_drop(skb, to_free);
diff --git a/net/sched/cls_basic.c b/net/sched/cls_basic.c
index ecfaa4f9a04e..d5ef0a66a73f 100644
--- a/net/sched/cls_basic.c
+++ b/net/sched/cls_basic.c
@@ -193,7 +193,7 @@ static int basic_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
@@ -212,9 +212,11 @@ static int basic_change(struct net *net, struct sk_buff *in_skb,
if (err)
goto errout;
fnew->handle = handle;
- fnew->pf = alloc_percpu(struct tc_basic_pcnt);
+ fnew->pf = alloc_percpu_gfp(struct tc_basic_pcnt, GFP_KERNEL_ACCOUNT);
if (!fnew->pf) {
err = -ENOMEM;
+ if (!fold)
+ idr_remove(&head->handle_idr, fnew->handle);
goto errout;
}
diff --git a/net/sched/cls_bpf.c b/net/sched/cls_bpf.c
index a6003fa35c7c..66b525a85287 100644
--- a/net/sched/cls_bpf.c
+++ b/net/sched/cls_bpf.c
@@ -352,7 +352,7 @@ static int cls_bpf_prog_from_ops(struct nlattr **tb, struct cls_bpf_prog *prog)
if (bpf_size != nla_len(tb[TCA_BPF_OPS]))
return -EINVAL;
- bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL);
+ bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL_ACCOUNT);
if (bpf_ops == NULL)
return -ENOMEM;
@@ -403,7 +403,7 @@ static int cls_bpf_prog_from_efd(struct nlattr **tb, struct cls_bpf_prog *prog,
}
if (tb[TCA_BPF_NAME]) {
- name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL);
+ name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL_ACCOUNT);
if (!name) {
bpf_prog_put(fp);
return -ENOMEM;
@@ -443,7 +443,7 @@ static int cls_bpf_change(struct net *net, struct sk_buff *in_skb,
if (ret < 0)
return ret;
- prog = kzalloc(sizeof(*prog), GFP_KERNEL);
+ prog = kzalloc(sizeof(*prog), GFP_KERNEL_ACCOUNT);
if (!prog)
return -ENOBUFS;
diff --git a/net/sched/cls_cgroup.c b/net/sched/cls_cgroup.c
index 424252982d6a..8f6563e50e63 100644
--- a/net/sched/cls_cgroup.c
+++ b/net/sched/cls_cgroup.c
@@ -95,7 +95,7 @@ static int cls_cgroup_change(struct net *net, struct sk_buff *in_skb,
if (head && handle != head->handle)
return -ENOENT;
- new = kzalloc(sizeof(*head), GFP_KERNEL);
+ new = kzalloc(sizeof(*head), GFP_KERNEL_ACCOUNT);
if (!new)
return -ENOBUFS;
diff --git a/net/sched/cls_flow.c b/net/sched/cls_flow.c
index a8b900a27d2f..334f8fc82ddc 100644
--- a/net/sched/cls_flow.c
+++ b/net/sched/cls_flow.c
@@ -438,7 +438,7 @@ static int flow_change(struct net *net, struct sk_buff *in_skb,
return -EOPNOTSUPP;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c
index 099ff6a3e1f5..5e61bf334ed8 100644
--- a/net/sched/cls_flower.c
+++ b/net/sched/cls_flower.c
@@ -2237,7 +2237,7 @@ static struct fl_flow_mask *fl_create_new_mask(struct cls_fl_head *head,
struct fl_flow_mask *newmask;
int err;
- newmask = kzalloc(sizeof(*newmask), GFP_KERNEL);
+ newmask = kzalloc(sizeof(*newmask), GFP_KERNEL_ACCOUNT);
if (!newmask)
return ERR_PTR(-ENOMEM);
@@ -2398,7 +2398,7 @@ static int fl_change(struct net *net, struct sk_buff *in_skb,
goto errout_tb;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew) {
err = -ENOBUFS;
goto errout_tb;
diff --git a/net/sched/cls_fw.c b/net/sched/cls_fw.c
index fd9c6c2815a1..e32eecdf675f 100644
--- a/net/sched/cls_fw.c
+++ b/net/sched/cls_fw.c
@@ -276,7 +276,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb,
if (f->id != handle && handle)
return -EINVAL;
- fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL);
+ fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
@@ -330,7 +330,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb,
rcu_assign_pointer(tp->root, head);
}
- f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL);
+ f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT);
if (f == NULL)
return -ENOBUFS;
diff --git a/net/sched/cls_matchall.c b/net/sched/cls_matchall.c
index f03bf5da39ee..762ba74aa7be 100644
--- a/net/sched/cls_matchall.c
+++ b/net/sched/cls_matchall.c
@@ -189,7 +189,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
}
- new = kzalloc(sizeof(*new), GFP_KERNEL);
+ new = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT);
if (!new)
return -ENOBUFS;
@@ -201,7 +201,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb,
handle = 1;
new->handle = handle;
new->flags = userflags;
- new->pf = alloc_percpu(struct tc_matchall_pcnt);
+ new->pf = alloc_percpu_gfp(struct tc_matchall_pcnt, GFP_KERNEL_ACCOUNT);
if (!new->pf) {
err = -ENOMEM;
goto err_alloc_percpu;
diff --git a/net/sched/cls_route.c b/net/sched/cls_route.c
index 6d5b734e7c48..1a41237d07aa 100644
--- a/net/sched/cls_route.c
+++ b/net/sched/cls_route.c
@@ -455,7 +455,7 @@ static int route4_set_parms(struct net *net, struct tcf_proto *tp,
h1 = to_hash(nhandle);
b = rtnl_dereference(head->table[h1]);
if (!b) {
- b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL);
+ b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL_ACCOUNT);
if (b == NULL)
return -ENOBUFS;
@@ -530,7 +530,7 @@ static int route4_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
err = -ENOBUFS;
- f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL);
+ f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL_ACCOUNT);
if (!f)
goto errout;
diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c
index e5e5533f866d..ac344aabe859 100644
--- a/net/sched/cls_u32.c
+++ b/net/sched/cls_u32.c
@@ -841,7 +841,7 @@ static struct tc_u_knode *u32_init_knode(struct net *net, struct tcf_proto *tp,
struct tc_u32_sel *s = &n->sel;
struct tc_u_knode *new;
- new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL);
+ new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL_ACCOUNT);
if (!new)
return NULL;
@@ -1132,15 +1132,23 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
goto erridr;
}
- n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL);
+ if (s->offshift >= 16) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offshift must be less than 16");
+ err = -EINVAL;
+ goto erridr;
+ }
+
+ n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL_ACCOUNT);
if (n == NULL) {
err = -ENOBUFS;
goto erridr;
}
#ifdef CONFIG_CLS_U32_PERF
- n->pf = __alloc_percpu(struct_size(n->pf, kcnts, s->nkeys),
- __alignof__(struct tc_u32_pcnt));
+ n->pf = __alloc_percpu_gfp(struct_size(n->pf, kcnts, s->nkeys),
+ __alignof__(struct tc_u32_pcnt),
+ GFP_KERNEL_ACCOUNT);
if (!n->pf) {
err = -ENOBUFS;
goto errfree;
@@ -1162,7 +1170,7 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
goto errout;
#ifdef CONFIG_CLS_U32_MARK
- n->pcpu_success = alloc_percpu(u32);
+ n->pcpu_success = alloc_percpu_gfp(u32, GFP_KERNEL_ACCOUNT);
if (!n->pcpu_success) {
err = -ENOMEM;
goto errout;
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index 43908ec23169..a60aae42398a 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -1704,7 +1704,7 @@ static u32 cake_classify(struct Qdisc *sch, struct cake_tin_data **t,
goto hash;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
@@ -1824,7 +1824,7 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch,
if (ack) {
b->ack_drops++;
- sch->qstats.drops++;
+ qdisc_qstats_drop(sch);
ack_pkt_len = qdisc_pkt_len(ack);
b->bytes += ack_pkt_len;
q->buffer_used += skb->truesize - ack->truesize;
diff --git a/net/sched/sch_codel.c b/net/sched/sch_codel.c
index f12a8f491108..a491250caf33 100644
--- a/net/sched/sch_codel.c
+++ b/net/sched/sch_codel.c
@@ -138,9 +138,9 @@ static const struct nla_policy codel_policy[TCA_CODEL_MAX + 1] = {
static int codel_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct codel_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_CODEL_MAX + 1];
- unsigned int qlen, dropped = 0;
int err;
err = nla_parse_nested_deprecated(tb, TCA_CODEL_MAX, opt,
@@ -179,15 +179,17 @@ static int codel_change(struct Qdisc *sch, struct nlattr *opt,
WRITE_ONCE(q->params.ecn,
!!nla_get_u32(tb[TCA_CODEL_ECN]));
- qlen = sch->q.qlen;
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, true);
- dropped += qdisc_pkt_len(skb);
- qdisc_qstats_backlog_dec(sch, skb);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_qdisc_drop(skb, sch);
}
- qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen, dropped);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_drr.c b/net/sched/sch_drr.c
index ba802db27993..8035ffeef2e9 100644
--- a/net/sched/sch_drr.c
+++ b/net/sched/sch_drr.c
@@ -313,7 +313,7 @@ static struct drr_class *drr_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
fl = rcu_dereference_bh(q->filter_list);
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_ets.c b/net/sched/sch_ets.c
index 27e549646dee..03f5bb812a27 100644
--- a/net/sched/sch_ets.c
+++ b/net/sched/sch_ets.c
@@ -389,7 +389,7 @@ static struct ets_class *ets_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
if (TC_H_MAJ(skb->priority) != sch->handle) {
fl = rcu_dereference_bh(q->filter_list);
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
index 6b85bb2a4f8a..cf70468b6760 100644
--- a/net/sched/sch_fq.c
+++ b/net/sched/sch_fq.c
@@ -535,6 +535,8 @@ static bool fq_packet_beyond_horizon(const struct sk_buff *skb,
return unlikely((s64)skb->tstamp > (s64)(now + q->horizon));
}
+#define FQDR(reason) SKB_DROP_REASON_FQ_##reason
+
static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
@@ -546,7 +548,8 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
band = fq_prio2band(q->prio2band, skb->priority & TC_PRIO_MAX);
if (unlikely(q->band_pkt_count[band] >= sch->limit)) {
q->stat_band_drops[band]++;
- return qdisc_drop(skb, sch, to_free);
+ return qdisc_drop_reason(skb, sch, to_free,
+ FQDR(BAND_LIMIT));
}
now = ktime_get_ns();
@@ -556,8 +559,9 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
/* Check if packet timestamp is too far in the future. */
if (fq_packet_beyond_horizon(skb, q, now)) {
if (q->horizon_drop) {
- q->stat_horizon_drops++;
- return qdisc_drop(skb, sch, to_free);
+ q->stat_horizon_drops++;
+ return qdisc_drop_reason(skb, sch, to_free,
+ FQDR(HORIZON_LIMIT));
}
q->stat_horizon_caps++;
skb->tstamp = now + q->horizon;
@@ -570,7 +574,8 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
if (f != &q->internal) {
if (unlikely(f->qlen >= q->flow_plimit)) {
q->stat_flows_plimit++;
- return qdisc_drop(skb, sch, to_free);
+ return qdisc_drop_reason(skb, sch, to_free,
+ FQDR(FLOW_LIMIT));
}
if (fq_flow_is_detached(f)) {
@@ -595,6 +600,7 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
return NET_XMIT_SUCCESS;
}
+#undef FQDR
static void fq_check_throttled(struct fq_sched_data *q, u64 now)
{
@@ -1002,11 +1008,11 @@ static int fq_load_priomap(struct fq_sched_data *q,
static int fq_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct fq_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_FQ_MAX + 1];
- int err, drop_count = 0;
- unsigned drop_len = 0;
u32 fq_log;
+ int err;
err = nla_parse_nested_deprecated(tb, TCA_FQ_MAX, opt, fq_policy,
NULL);
@@ -1109,16 +1115,18 @@ static int fq_change(struct Qdisc *sch, struct nlattr *opt,
err = fq_resize(sch, fq_log);
sch_tree_lock(sch);
}
+
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
if (!skb)
break;
- drop_len += qdisc_pkt_len(skb);
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
- drop_count++;
}
- qdisc_tree_reduce_backlog(sch, drop_count, drop_len);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return err;
diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c
index 4b0196eed7a9..d41a55863d16 100644
--- a/net/sched/sch_fq_codel.c
+++ b/net/sched/sch_fq_codel.c
@@ -91,7 +91,7 @@ static unsigned int fq_codel_classify(struct sk_buff *skb, struct Qdisc *sch,
return fq_codel_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
@@ -175,7 +175,7 @@ static unsigned int fq_codel_drop(struct Qdisc *sch, unsigned int max_packets,
flow->cvars.count += i;
q->backlogs[idx] -= len;
q->memory_usage -= mem;
- sch->qstats.drops += i;
+ __qdisc_qstats_drop(sch, i);
sch->qstats.backlog -= len;
sch->q.qlen -= i;
return idx;
@@ -402,6 +402,7 @@ static const struct nla_policy fq_codel_policy[TCA_FQ_CODEL_MAX + 1] = {
static int fq_codel_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct fq_codel_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_FQ_CODEL_MAX + 1];
u32 quantum = 0;
@@ -479,13 +480,14 @@ static int fq_codel_change(struct Qdisc *sch, struct nlattr *opt,
q->memory_usage > q->memory_limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
- q->cstats.drop_len += qdisc_pkt_len(skb);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
- q->cstats.drop_count++;
}
- qdisc_tree_reduce_backlog(sch, q->cstats.drop_count, q->cstats.drop_len);
- q->cstats.drop_count = 0;
- q->cstats.drop_len = 0;
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_fq_pie.c b/net/sched/sch_fq_pie.c
index 4b96e26cb420..a3c41e81fa80 100644
--- a/net/sched/sch_fq_pie.c
+++ b/net/sched/sch_fq_pie.c
@@ -96,7 +96,7 @@ static unsigned int fq_pie_classify(struct sk_buff *skb, struct Qdisc *sch,
return fq_pie_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
@@ -285,10 +285,9 @@ static struct sk_buff *fq_pie_qdisc_dequeue(struct Qdisc *sch)
static int fq_pie_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct fq_pie_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_FQ_PIE_MAX + 1];
- unsigned int len_dropped = 0;
- unsigned int num_dropped = 0;
int err;
err = nla_parse_nested(tb, TCA_FQ_PIE_MAX, opt, fq_pie_policy, extack);
@@ -367,11 +366,14 @@ static int fq_pie_change(struct Qdisc *sch, struct nlattr *opt,
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
- len_dropped += qdisc_pkt_len(skb);
- num_dropped += 1;
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
}
- qdisc_tree_reduce_backlog(sch, num_dropped, len_dropped);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 6a61bd0c081c..681e0ab6019f 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -542,28 +542,43 @@ static void dev_watchdog(struct timer_list *t)
dev->netdev_ops->ndo_tx_timeout(dev, i);
netif_unfreeze_queues(dev);
}
- if (!mod_timer(&dev->watchdog_timer,
- round_jiffies(oldest_start +
- dev->watchdog_timeo)))
- release = false;
+ spin_lock(&dev->watchdog_lock);
+ mod_timer(&dev->watchdog_timer,
+ round_jiffies(oldest_start +
+ dev->watchdog_timeo));
+ release = false;
+ spin_unlock(&dev->watchdog_lock);
}
}
spin_unlock(&dev->tx_global_lock);
- if (release)
+ spin_lock(&dev->watchdog_lock);
+ if (timer_pending(&dev->watchdog_timer))
+ release = false;
+ if (release && dev->watchdog_ref_held) {
netdev_put(dev, &dev->watchdog_dev_tracker);
+ dev->watchdog_ref_held = false;
+ }
+ spin_unlock(&dev->watchdog_lock);
}
void __netdev_watchdog_up(struct net_device *dev)
{
- if (dev->netdev_ops->ndo_tx_timeout) {
- if (dev->watchdog_timeo <= 0)
- dev->watchdog_timeo = 5*HZ;
- if (!mod_timer(&dev->watchdog_timer,
- round_jiffies(jiffies + dev->watchdog_timeo)))
+ if (!dev->netdev_ops->ndo_tx_timeout)
+ return;
+ if (dev->watchdog_timeo <= 0)
+ dev->watchdog_timeo = 5*HZ;
+
+ spin_lock_bh(&dev->watchdog_lock);
+ if (!mod_timer(&dev->watchdog_timer,
+ round_jiffies(jiffies + dev->watchdog_timeo))) {
+ if (!dev->watchdog_ref_held) {
netdev_hold(dev, &dev->watchdog_dev_tracker,
GFP_ATOMIC);
+ dev->watchdog_ref_held = true;
+ }
}
+ spin_unlock_bh(&dev->watchdog_lock);
}
EXPORT_SYMBOL_GPL(__netdev_watchdog_up);
@@ -575,8 +590,14 @@ static void dev_watchdog_up(struct net_device *dev)
static void dev_watchdog_down(struct net_device *dev)
{
netif_tx_lock_bh(dev);
- if (del_timer(&dev->watchdog_timer))
+
+ spin_lock(&dev->watchdog_lock);
+ if (timer_delete(&dev->watchdog_timer)) {
netdev_put(dev, &dev->watchdog_dev_tracker);
+ dev->watchdog_ref_held = false;
+ }
+ spin_unlock(&dev->watchdog_lock);
+
netif_tx_unlock_bh(dev);
}
@@ -593,8 +614,6 @@ void netif_carrier_on(struct net_device *dev)
return;
atomic_inc(&dev->carrier_up_count);
linkwatch_fire_event(dev);
- if (netif_running(dev))
- __netdev_watchdog_up(dev);
}
}
EXPORT_SYMBOL(netif_carrier_on);
diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c
index 53b348b83c91..26103178a4ce 100644
--- a/net/sched/sch_gred.c
+++ b/net/sched/sch_gred.c
@@ -389,7 +389,7 @@ static int gred_offload_dump_stats(struct Qdisc *sch)
packets += u64_stats_read(&hw_stats->stats.bstats[i].packets);
sch->qstats.qlen += hw_stats->stats.qstats[i].qlen;
sch->qstats.backlog += hw_stats->stats.qstats[i].backlog;
- sch->qstats.drops += hw_stats->stats.qstats[i].drops;
+ __qdisc_qstats_drop(sch, hw_stats->stats.qstats[i].drops);
sch->qstats.requeues += hw_stats->stats.qstats[i].requeues;
sch->qstats.overlimits += hw_stats->stats.qstats[i].overlimits;
}
diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c
index df3501dfe01a..b7c7121c3611 100644
--- a/net/sched/sch_hfsc.c
+++ b/net/sched/sch_hfsc.c
@@ -1154,7 +1154,7 @@ hfsc_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
head = &q->root;
drift = HFSC_CLASSIFY_MAX_DRIFT;
tcf = rcu_dereference_bh(q->root.filter_list);
- while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) {
+ while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
case TC_ACT_QUEUED:
diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c
index 7d7bc1d7e049..d6a60ae8938c 100644
--- a/net/sched/sch_hhf.c
+++ b/net/sched/sch_hhf.c
@@ -536,9 +536,9 @@ static const struct nla_policy hhf_policy[TCA_HHF_MAX + 1] = {
static int hhf_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct hhf_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_HHF_MAX + 1];
- unsigned int qlen, prev_backlog;
int err;
u64 non_hh_quantum;
u32 new_quantum = q->quantum;
@@ -589,15 +589,17 @@ static int hhf_change(struct Qdisc *sch, struct nlattr *opt,
usecs_to_jiffies(us));
}
- qlen = sch->q.qlen;
- prev_backlog = sch->qstats.backlog;
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
}
- qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen,
- prev_backlog - sch->qstats.backlog);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 36cc9a343b54..e44a5ef99a83 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -243,7 +243,7 @@ static struct htb_class *htb_classify(struct sk_buff *skb, struct Qdisc *sch,
}
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) {
+ while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
case TC_ACT_QUEUED:
diff --git a/net/sched/sch_multiq.c b/net/sched/sch_multiq.c
index e50a37b7cec1..7962fc0292a5 100644
--- a/net/sched/sch_multiq.c
+++ b/net/sched/sch_multiq.c
@@ -36,7 +36,7 @@ multiq_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
int err;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_pie.c b/net/sched/sch_pie.c
index 793a155e27f9..d697adba1a94 100644
--- a/net/sched/sch_pie.c
+++ b/net/sched/sch_pie.c
@@ -138,9 +138,9 @@ static const struct nla_policy pie_policy[TCA_PIE_MAX + 1] = {
static int pie_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct pie_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_PIE_MAX + 1];
- unsigned int qlen, dropped = 0;
int err;
err = nla_parse_nested_deprecated(tb, TCA_PIE_MAX, opt, pie_policy,
@@ -190,15 +190,17 @@ static int pie_change(struct Qdisc *sch, struct nlattr *opt,
nla_get_u32(tb[TCA_PIE_DQ_RATE_ESTIMATOR]));
/* Drop excess packets if new limit is lower */
- qlen = sch->q.qlen;
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, true);
- dropped += qdisc_pkt_len(skb);
- qdisc_qstats_backlog_dec(sch, skb);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_qdisc_drop(skb, sch);
}
- qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen, dropped);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_prio.c b/net/sched/sch_prio.c
index 9e2b9a490db2..6f4388cf1048 100644
--- a/net/sched/sch_prio.c
+++ b/net/sched/sch_prio.c
@@ -39,7 +39,7 @@ prio_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
if (TC_H_MAJ(skb->priority) != sch->handle) {
fl = rcu_dereference_bh(q->filter_list);
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c
index c7c8e8dde31d..70a08f195010 100644
--- a/net/sched/sch_qfq.c
+++ b/net/sched/sch_qfq.c
@@ -712,7 +712,7 @@ static struct qfq_class *qfq_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
fl = rcu_dereference_bh(q->filter_list);
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_sfb.c b/net/sched/sch_sfb.c
index 9a2edaf8352a..cd2fba8caa64 100644
--- a/net/sched/sch_sfb.c
+++ b/net/sched/sch_sfb.c
@@ -260,7 +260,7 @@ static bool sfb_classify(struct sk_buff *skb, struct tcf_proto *fl,
struct tcf_result res;
int result;
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
index cc7d8e484238..4fd6a9f4e6da 100644
--- a/net/sched/sch_sfq.c
+++ b/net/sched/sch_sfq.c
@@ -171,7 +171,7 @@ static unsigned int sfq_classify(struct sk_buff *skb, struct Qdisc *sch,
return sfq_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sctp/auth.c b/net/sctp/auth.c
index 8320764b16b9..1a722ae146a2 100644
--- a/net/sctp/auth.c
+++ b/net/sctp/auth.c
@@ -376,6 +376,82 @@ int sctp_auth_asoc_copy_shkeys(const struct sctp_endpoint *ep,
return -ENOMEM;
}
+static bool sctp_auth_chunk_id_forbidden(__u8 chunk_id)
+{
+ switch (chunk_id) {
+ case SCTP_CID_INIT:
+ case SCTP_CID_INIT_ACK:
+ case SCTP_CID_SHUTDOWN_COMPLETE:
+ case SCTP_CID_AUTH:
+ return true;
+ default:
+ return false;
+ }
+}
+
+/* Verify AUTH parameters copied from a state cookie before they are restored
+ * into an association. When cookie authentication is disabled these fields
+ * are peer-controlled, so they must satisfy the same constraints as locally
+ * generated AUTH parameters.
+ */
+bool sctp_auth_verify_cookie_params(const struct sctp_endpoint *ep,
+ const struct sctp_cookie *cookie)
+{
+ const struct sctp_paramhdr *random;
+ const struct sctp_hmac_algo_param *hmacs;
+ const struct sctp_chunks_param *chunks;
+ u16 hmacs_len, chunks_len;
+ u16 n_hmacs, n_chunks, i;
+ bool has_sha1 = false;
+
+ if (sctp_sk(ep->base.sk)->hmac || !ep->auth_enable)
+ return true;
+
+ random = (const struct sctp_paramhdr *)cookie->auth_random;
+ if (random->type != SCTP_PARAM_RANDOM ||
+ ntohs(random->length) != sizeof(*random) + SCTP_AUTH_RANDOM_LENGTH)
+ return false;
+
+ hmacs = (const struct sctp_hmac_algo_param *)cookie->auth_hmacs;
+ hmacs_len = ntohs(hmacs->param_hdr.length);
+ if (hmacs->param_hdr.type != SCTP_PARAM_HMAC_ALGO ||
+ hmacs_len < sizeof(struct sctp_paramhdr) +
+ sizeof(hmacs->hmac_ids[0]) ||
+ hmacs_len > sizeof(cookie->auth_hmacs) ||
+ (hmacs_len - sizeof(struct sctp_paramhdr)) %
+ sizeof(hmacs->hmac_ids[0]))
+ return false;
+
+ n_hmacs = (hmacs_len - sizeof(struct sctp_paramhdr)) /
+ sizeof(hmacs->hmac_ids[0]);
+ for (i = 0; i < n_hmacs; i++) {
+ u16 hmac_id = ntohs(hmacs->hmac_ids[i]);
+
+ if (hmac_id > SCTP_AUTH_HMAC_ID_MAX ||
+ !sctp_hmac_list[hmac_id].hmac_name)
+ return false;
+ if (hmac_id == SCTP_AUTH_HMAC_ID_SHA1)
+ has_sha1 = true;
+ }
+ if (!has_sha1)
+ return false;
+
+ chunks = (const struct sctp_chunks_param *)cookie->auth_chunks;
+ chunks_len = ntohs(chunks->param_hdr.length);
+ if (chunks->param_hdr.type != SCTP_PARAM_CHUNKS ||
+ chunks_len < sizeof(struct sctp_paramhdr) ||
+ chunks_len > sizeof(cookie->auth_chunks))
+ return false;
+
+ n_chunks = chunks_len - sizeof(struct sctp_paramhdr);
+ for (i = 0; i < n_chunks; i++) {
+ if (sctp_auth_chunk_id_forbidden(chunks->chunks[i]))
+ return false;
+ }
+
+ return true;
+}
+
/* Public interface to create the association shared key.
* See code above for the algorithm.
diff --git a/net/sctp/bind_addr.c b/net/sctp/bind_addr.c
index 6b95d3ba8fe1..0947b276d1e0 100644
--- a/net/sctp/bind_addr.c
+++ b/net/sctp/bind_addr.c
@@ -275,6 +275,16 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list,
param = (struct sctp_paramhdr *)raw_addr_list;
rawaddr = (union sctp_addr_param *)raw_addr_list;
+ if (addrs_len < sizeof(*param)) {
+ retval = -EINVAL;
+ goto out_err;
+ }
+ len = ntohs(param->length);
+ if (addrs_len < len) {
+ retval = -EINVAL;
+ goto out_err;
+ }
+
af = sctp_get_af_specific(param_type2af(param->type));
if (unlikely(!af) ||
!af->from_addr_param(&addr, rawaddr, htons(port), 0)) {
@@ -291,7 +301,6 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list,
goto out_err;
next:
- len = ntohs(param->length);
addrs_len -= len;
raw_addr_list += len;
}
diff --git a/net/sctp/protocol.c b/net/sctp/protocol.c
index c6d74fa517cd..66ede52383f8 100644
--- a/net/sctp/protocol.c
+++ b/net/sctp/protocol.c
@@ -693,8 +693,9 @@ static void sctp_free_addr_wq(struct net *net)
struct sctp_sockaddr_entry *addrw;
struct sctp_sockaddr_entry *temp;
+ timer_shutdown_sync(&net->sctp.addr_wq_timer);
+
spin_lock_bh(&net->sctp.addr_wq_lock);
- del_timer(&net->sctp.addr_wq_timer);
list_for_each_entry_safe(addrw, temp, &net->sctp.addr_waitq, list) {
list_del(&addrw->list);
kfree(addrw);
diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c
index 2366dd0298eb..1bbc041c0644 100644
--- a/net/sctp/sm_make_chunk.c
+++ b/net/sctp/sm_make_chunk.c
@@ -1750,8 +1750,9 @@ struct sctp_association *sctp_unpack_cookie(
struct sk_buff *skb = chunk->skb;
struct sctp_cookie *bear_cookie;
__u8 *digest = ep->digest;
+ struct sctp_chunkhdr *ch;
+ unsigned int len, chlen;
enum sctp_scope scope;
- unsigned int len;
ktime_t kt;
/* Header size is static data prior to the actual cookie, including
@@ -1779,6 +1780,17 @@ struct sctp_association *sctp_unpack_cookie(
cookie = chunk->subh.cookie_hdr;
bear_cookie = &cookie->c;
+ ch = (struct sctp_chunkhdr *)(bear_cookie + 1);
+ if (ch->type != SCTP_CID_INIT)
+ goto malformed;
+ chlen = ntohs(ch->length);
+ if (chlen < sizeof(struct sctp_init_chunk))
+ goto malformed;
+ if (chlen > len - fixed_size)
+ goto malformed;
+ if (bear_cookie->raw_addr_list_len > len - fixed_size - chlen)
+ goto malformed;
+
if (!sctp_sk(ep->base.sk)->hmac)
goto no_hmac;
@@ -1871,6 +1883,9 @@ struct sctp_association *sctp_unpack_cookie(
/* Set up our peer's port number. */
retval->peer.port = ntohs(chunk->sctp_hdr->source);
+ if (!sctp_auth_verify_cookie_params(ep, bear_cookie))
+ goto malformed;
+
/* Populate the association from the cookie. */
memcpy(&retval->c, bear_cookie, sizeof(*bear_cookie));
@@ -2328,7 +2343,8 @@ int sctp_verify_init(struct net *net, const struct sctp_endpoint *ep,
* VIOLATION error. We build the ERROR chunk here and let the normal
* error handling code build and send the packet.
*/
- if (param.v != (void *)chunk->chunk_end)
+ if (param.v != (void *)peer_init +
+ SCTP_PAD4(ntohs(peer_init->chunk_hdr.length)))
return sctp_process_inv_paramlength(asoc, param.p, chunk, errp);
/* The only missing mandatory param possible today is
diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index ee4269b75e36..b16bb0ebb552 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -708,11 +708,12 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net,
struct sctp_cmd_seq *commands)
{
struct sctp_ulpevent *ev, *ai_ev = NULL, *auth_ev = NULL;
+ struct sctp_chunk *err_chk_p = NULL;
struct sctp_association *new_asoc;
struct sctp_init_chunk *peer_init;
struct sctp_chunk *chunk = arg;
- struct sctp_chunk *err_chk_p;
struct sctp_chunk *repl;
+ enum sctp_cid cid;
struct sock *sk;
int error = 0;
@@ -786,6 +787,19 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net,
}
}
+ peer_init = (struct sctp_init_chunk *)(chunk->subh.cookie_hdr + 1);
+ cid = peer_init->chunk_hdr.type;
+ if (!sctp_sk(sk)->hmac &&
+ !sctp_verify_init(net, ep, asoc, cid, peer_init, chunk,
+ &err_chk_p)) {
+ sctp_association_free(new_asoc);
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
+ }
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+
if (security_sctp_assoc_request(new_asoc, chunk->head_skb ?: chunk->skb)) {
sctp_association_free(new_asoc);
return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
@@ -799,7 +813,6 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net,
/* This is a brand-new association, so these are not yet side
* effects--it is safe to run them here.
*/
- peer_init = (struct sctp_init_chunk *)(chunk->subh.cookie_hdr + 1);
if (!sctp_process_init(new_asoc, chunk,
&chunk->subh.cookie_hdr->c.peer_addr,
peer_init, GFP_ATOMIC))
@@ -2216,10 +2229,12 @@ enum sctp_disposition sctp_sf_do_5_2_4_dupcook(
void *arg,
struct sctp_cmd_seq *commands)
{
+ struct sctp_chunk *err_chk_p = NULL;
struct sctp_association *new_asoc;
+ struct sctp_init_chunk *peer_init;
struct sctp_chunk *chunk = arg;
enum sctp_disposition retval;
- struct sctp_chunk *err_chk_p;
+ enum sctp_cid cid;
int error = 0;
char action;
@@ -2288,6 +2303,21 @@ enum sctp_disposition sctp_sf_do_5_2_4_dupcook(
switch (action) {
case 'A': /* Association restart. */
case 'B': /* Collision case B. */
+ peer_init = (struct sctp_init_chunk *)
+ (chunk->subh.cookie_hdr + 1);
+ cid = peer_init->chunk_hdr.type;
+ if (!sctp_sk(ep->base.sk)->hmac &&
+ !sctp_verify_init(net, ep, asoc, cid, peer_init, chunk,
+ &err_chk_p)) {
+ sctp_association_free(new_asoc);
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg,
+ commands);
+ }
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+ fallthrough;
case 'D': /* Collision case D. */
/* Update socket peer label if first association. */
if (security_sctp_assoc_request((struct sctp_association *)asoc,
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index 17ce4d32f64f..6f8c9b79e7bf 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -3380,11 +3380,10 @@ int smc_create_clcsk(struct net *net, struct sock *sk, int family)
return 0;
}
-static int __smc_create(struct net *net, struct socket *sock, int protocol,
- int kern, struct socket *clcsock)
+static int smc_create(struct net *net, struct socket *sock, int protocol,
+ int kern)
{
int family = (protocol == SMCPROTO_SMC6) ? PF_INET6 : PF_INET;
- struct smc_sock *smc;
struct sock *sk;
int rc;
@@ -3403,15 +3402,7 @@ static int __smc_create(struct net *net, struct socket *sock, int protocol,
if (!sk)
goto out;
- /* create internal TCP socket for CLC handshake and fallback */
- smc = smc_sk(sk);
-
- rc = 0;
- if (clcsock)
- smc->clcsock = clcsock;
- else
- rc = smc_create_clcsk(net, sk, family);
-
+ rc = smc_create_clcsk(net, sk, family);
if (rc) {
sk_common_release(sk);
sock->sk = NULL;
@@ -3420,76 +3411,12 @@ static int __smc_create(struct net *net, struct socket *sock, int protocol,
return rc;
}
-static int smc_create(struct net *net, struct socket *sock, int protocol,
- int kern)
-{
- return __smc_create(net, sock, protocol, kern, NULL);
-}
-
static const struct net_proto_family smc_sock_family_ops = {
.family = PF_SMC,
.owner = THIS_MODULE,
.create = smc_create,
};
-static int smc_ulp_init(struct sock *sk)
-{
- struct socket *tcp = sk->sk_socket;
- struct net *net = sock_net(sk);
- struct socket *smcsock;
- int protocol, ret;
-
- /* only TCP can be replaced */
- if (tcp->type != SOCK_STREAM || sk->sk_protocol != IPPROTO_TCP ||
- (sk->sk_family != AF_INET && sk->sk_family != AF_INET6))
- return -ESOCKTNOSUPPORT;
- /* don't handle wq now */
- if (tcp->state != SS_UNCONNECTED || !tcp->file || tcp->wq.fasync_list)
- return -ENOTCONN;
-
- if (sk->sk_family == AF_INET)
- protocol = SMCPROTO_SMC;
- else
- protocol = SMCPROTO_SMC6;
-
- smcsock = sock_alloc();
- if (!smcsock)
- return -ENFILE;
-
- smcsock->type = SOCK_STREAM;
- __module_get(THIS_MODULE); /* tried in __tcp_ulp_find_autoload */
- ret = __smc_create(net, smcsock, protocol, 1, tcp);
- if (ret) {
- sock_release(smcsock); /* module_put() which ops won't be NULL */
- return ret;
- }
-
- /* replace tcp socket to smc */
- smcsock->file = tcp->file;
- smcsock->file->private_data = smcsock;
- smcsock->file->f_inode = SOCK_INODE(smcsock); /* replace inode when sock_close */
- smcsock->file->f_path.dentry->d_inode = SOCK_INODE(smcsock); /* dput() in __fput */
- tcp->file = NULL;
-
- return ret;
-}
-
-static void smc_ulp_clone(const struct request_sock *req, struct sock *newsk,
- const gfp_t priority)
-{
- struct inet_connection_sock *icsk = inet_csk(newsk);
-
- /* don't inherit ulp ops to child when listen */
- icsk->icsk_ulp_ops = NULL;
-}
-
-static struct tcp_ulp_ops smc_ulp_ops __read_mostly = {
- .name = "smc",
- .owner = THIS_MODULE,
- .init = smc_ulp_init,
- .clone = smc_ulp_clone,
-};
-
unsigned int smc_net_id;
static __net_init int smc_net_init(struct net *net)
@@ -3617,20 +3544,13 @@ static int __init smc_init(void)
goto out_ib;
}
- rc = tcp_register_ulp(&smc_ulp_ops);
- if (rc) {
- pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc);
- goto out_lo;
- }
rc = smc_inet_init();
if (rc) {
pr_err("%s: smc_inet_init fails with %d\n", __func__, rc);
- goto out_ulp;
+ goto out_lo;
}
static_branch_enable(&tcp_have_smc);
return 0;
-out_ulp:
- tcp_unregister_ulp(&smc_ulp_ops);
out_lo:
smc_loopback_exit();
out_ib:
@@ -3668,7 +3588,6 @@ static void __exit smc_exit(void)
{
static_branch_disable(&tcp_have_smc);
smc_inet_exit();
- tcp_unregister_ulp(&smc_ulp_ops);
sock_unregister(PF_SMC);
smc_core_exit();
smc_loopback_exit();
@@ -3694,7 +3613,6 @@ MODULE_AUTHOR("Ursula Braun <ubraun@linux.vnet.ibm.com>");
MODULE_DESCRIPTION("smc socket address family");
MODULE_LICENSE("GPL");
MODULE_ALIAS_NETPROTO(PF_SMC);
-MODULE_ALIAS_TCP_ULP("smc");
/* 256 for IPPROTO_SMC and 1 for SOCK_STREAM */
MODULE_ALIAS_NET_PF_PROTO_TYPE(PF_INET, 256, 1);
#if IS_ENABLED(CONFIG_IPV6)
diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
index 0bf9e4075f03..cb72ff920578 100644
--- a/net/smc/smc_loopback.c
+++ b/net/smc/smc_loopback.c
@@ -117,14 +117,9 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
return rc;
}
-static void __smc_lo_unregister_dmb(struct smc_lo_dev *ldev,
- struct smc_lo_dmb_node *dmb_node)
+static void smc_lo_free_dmb(struct smc_lo_dev *ldev,
+ struct smc_lo_dmb_node *dmb_node)
{
- /* remove dmb from hash table */
- write_lock_bh(&ldev->dmb_ht_lock);
- hash_del(&dmb_node->list);
- write_unlock_bh(&ldev->dmb_ht_lock);
-
clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
folio_put(virt_to_folio(dmb_node->cpu_addr));
kfree(dmb_node);
@@ -137,9 +132,10 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
{
struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
struct smc_lo_dev *ldev = smcd->priv;
+ bool last;
/* find dmb from hash table */
- read_lock_bh(&ldev->dmb_ht_lock);
+ write_lock_bh(&ldev->dmb_ht_lock);
hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) {
if (tmp_node->token == dmb->dmb_tok) {
dmb_node = tmp_node;
@@ -147,13 +143,16 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
}
}
if (!dmb_node) {
- read_unlock_bh(&ldev->dmb_ht_lock);
+ write_unlock_bh(&ldev->dmb_ht_lock);
return -EINVAL;
}
- read_unlock_bh(&ldev->dmb_ht_lock);
+ last = refcount_dec_and_test(&dmb_node->refcnt);
+ if (last)
+ hash_del(&dmb_node->list);
+ write_unlock_bh(&ldev->dmb_ht_lock);
- if (refcount_dec_and_test(&dmb_node->refcnt))
- __smc_lo_unregister_dmb(ldev, dmb_node);
+ if (last)
+ smc_lo_free_dmb(ldev, dmb_node);
return 0;
}
@@ -179,14 +178,9 @@ static int smc_lo_attach_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
read_unlock_bh(&ldev->dmb_ht_lock);
return -EINVAL;
}
+ refcount_inc(&dmb_node->refcnt);
read_unlock_bh(&ldev->dmb_ht_lock);
- if (!refcount_inc_not_zero(&dmb_node->refcnt))
- /* the dmb is being unregistered, but has
- * not been removed from the hash table.
- */
- return -EINVAL;
-
/* provide dmb information */
dmb->sba_idx = dmb_node->sba_idx;
dmb->dmb_tok = dmb_node->token;
@@ -200,9 +194,10 @@ static int smc_lo_detach_dmb(struct smcd_dev *smcd, u64 token)
{
struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
struct smc_lo_dev *ldev = smcd->priv;
+ bool last;
/* find dmb_node according to dmb->dmb_tok */
- read_lock_bh(&ldev->dmb_ht_lock);
+ write_lock_bh(&ldev->dmb_ht_lock);
hash_for_each_possible(ldev->dmb_ht, tmp_node, list, token) {
if (tmp_node->token == token) {
dmb_node = tmp_node;
@@ -210,13 +205,17 @@ static int smc_lo_detach_dmb(struct smcd_dev *smcd, u64 token)
}
}
if (!dmb_node) {
- read_unlock_bh(&ldev->dmb_ht_lock);
+ write_unlock_bh(&ldev->dmb_ht_lock);
return -EINVAL;
}
- read_unlock_bh(&ldev->dmb_ht_lock);
+ last = refcount_dec_and_test(&dmb_node->refcnt);
+ if (last)
+ hash_del(&dmb_node->list);
+ write_unlock_bh(&ldev->dmb_ht_lock);
+
+ if (last)
+ smc_lo_free_dmb(ldev, dmb_node);
- if (refcount_dec_and_test(&dmb_node->refcnt))
- __smc_lo_unregister_dmb(ldev, dmb_node);
return 0;
}
diff --git a/net/sunrpc/auth_gss/gss_krb5_unseal.c b/net/sunrpc/auth_gss/gss_krb5_unseal.c
index ef0e6af9fc95..711697e43f22 100644
--- a/net/sunrpc/auth_gss/gss_krb5_unseal.c
+++ b/net/sunrpc/auth_gss/gss_krb5_unseal.c
@@ -86,6 +86,9 @@ gss_krb5_verify_mic_v2(struct krb5_ctx *ctx, struct xdr_buf *message_buffer,
dprintk("RPC: %s\n", __func__);
+ if (read_token->len < GSS_KRB5_TOK_HDR_LEN + ctx->gk5e->cksumlength)
+ return GSS_S_DEFECTIVE_TOKEN;
+
memcpy(&be16_ptr, (char *) ptr, 2);
if (be16_to_cpu(be16_ptr) != KG2_TOK_MIC)
return GSS_S_DEFECTIVE_TOKEN;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
index 68a0aa627455..533ad339054b 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
@@ -376,7 +376,7 @@ static void svc_rdma_wc_receive(struct ib_cq *cq, struct ib_wc *wc)
trace_svcrdma_wc_recv_err(wc, &ctxt->rc_cid);
dropped:
svc_rdma_recv_ctxt_put(rdma, ctxt);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -430,11 +430,14 @@ static void svc_rdma_build_arg_xdr(struct svc_rqst *rqstp,
* to the first byte past the Read list. rc_read_pcl and
* rc_call_pcl cl_count fields are set to the number of
* Read segments in the list.
- * %false: Read list is corrupt. @rctxt's xdr_stream is left in an
- * unknown state.
+ * %false: Read list is corrupt or exceeds the page budget. @rctxt's
+ * xdr_stream is left in an unknown state.
*/
static bool xdr_count_read_segments(struct svc_rdma_recv_ctxt *rctxt, __be32 *p)
{
+ unsigned int maxlen = (rctxt->rc_maxpages - 2) << PAGE_SHIFT;
+ unsigned int total_len = 0;
+
rctxt->rc_call_pcl.cl_count = 0;
rctxt->rc_read_pcl.cl_count = 0;
while (xdr_item_is_present(p)) {
@@ -448,6 +451,12 @@ static bool xdr_count_read_segments(struct svc_rdma_recv_ctxt *rctxt, __be32 *p)
xdr_decode_read_segment(p, &position, &handle,
&length, &offset);
+ if (length > maxlen)
+ return false;
+ total_len += length;
+ if (PAGE_ALIGN(total_len + rctxt->rc_byte_len -
+ xdr_stream_pos(&rctxt->rc_stream)) > maxlen)
+ return false;
if (position) {
if (position & 3)
return false;
@@ -1010,7 +1019,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)
if (ret == -EINVAL)
svc_rdma_send_error(rdma_xprt, ctxt, ret);
svc_rdma_recv_ctxt_put(rdma_xprt, ctxt);
- svc_xprt_deferred_close(xprt);
+ svc_rdma_xprt_deferred_close(rdma_xprt);
return ret;
}
return 0;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_rw.c b/net/sunrpc/xprtrdma/svc_rdma_rw.c
index 448d3ee936a5..4b1571d70039 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_rw.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_rw.c
@@ -230,6 +230,28 @@ static void svc_rdma_write_info_free(struct svc_rdma_write_info *info)
queue_work(svcrdma_wq, &info->wi_work);
}
+/**
+ * svc_rdma_write_chunk_release - Release Write chunk I/O resources
+ * @rdma: controlling transport
+ * @ctxt: Send context that is being released
+ *
+ * Write chunk resources remain live until Send completion because
+ * Write WRs are chained to the Send WR. This function releases all
+ * write_info structures accumulated on @ctxt->sc_write_info_list.
+ */
+void svc_rdma_write_chunk_release(struct svcxprt_rdma *rdma,
+ struct svc_rdma_send_ctxt *ctxt)
+{
+ struct svc_rdma_write_info *info;
+
+ while (!list_empty(&ctxt->sc_write_info_list)) {
+ info = list_first_entry(&ctxt->sc_write_info_list,
+ struct svc_rdma_write_info, wi_list);
+ list_del(&info->wi_list);
+ svc_rdma_write_info_free(info);
+ }
+}
+
/**
* svc_rdma_reply_chunk_release - Release Reply chunk I/O resources
* @rdma: controlling transport
@@ -270,7 +292,7 @@ static void svc_rdma_reply_done(struct ib_cq *cq, struct ib_wc *wc)
trace_svcrdma_wc_reply_err(wc, &cc->cc_cid);
}
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -286,13 +308,11 @@ static void svc_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc)
struct ib_cqe *cqe = wc->wr_cqe;
struct svc_rdma_chunk_ctxt *cc =
container_of(cqe, struct svc_rdma_chunk_ctxt, cc_cqe);
- struct svc_rdma_write_info *info =
- container_of(cc, struct svc_rdma_write_info, wi_cc);
switch (wc->status) {
case IB_WC_SUCCESS:
trace_svcrdma_wc_write(&cc->cc_cid);
- break;
+ return;
case IB_WC_WR_FLUSH_ERR:
trace_svcrdma_wc_write_flush(wc, &cc->cc_cid);
break;
@@ -300,12 +320,11 @@ static void svc_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc)
trace_svcrdma_wc_write_err(wc, &cc->cc_cid);
}
- svc_rdma_wake_send_waiters(rdma, cc->cc_sqecount);
-
- if (unlikely(wc->status != IB_WC_SUCCESS))
- svc_xprt_deferred_close(&rdma->sc_xprt);
-
- svc_rdma_write_info_free(info);
+ /* The RDMA Write has flushed, so the client won't get
+ * some of the outgoing RPC message. Signal the loss
+ * to the client by closing the connection.
+ */
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -350,7 +369,7 @@ static void svc_rdma_wc_read_done(struct ib_cq *cq, struct ib_wc *wc)
*/
svc_rdma_cc_release(rdma, cc, DMA_FROM_DEVICE);
svc_rdma_recv_ctxt_put(rdma, ctxt);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/*
@@ -403,7 +422,7 @@ static int svc_rdma_post_chunk_ctxt(struct svcxprt_rdma *rdma,
} while (1);
trace_svcrdma_sq_post_err(rdma, &cc->cc_cid, ret);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
/* If even one was posted, there will be a completion. */
if (bad_wr != first_wr)
@@ -601,13 +620,27 @@ static int svc_rdma_xb_write(const struct xdr_buf *xdr, void *data)
return xdr->len;
}
-static int svc_rdma_send_write_chunk(struct svcxprt_rdma *rdma,
- const struct svc_rdma_chunk *chunk,
- const struct xdr_buf *xdr)
+/*
+ * svc_rdma_prepare_write_chunk - Link Write WRs for @chunk onto @sctxt's chain
+ *
+ * Write WRs are prepended to the Send WR chain so that a single
+ * ib_post_send() posts both RDMA Writes and the final Send. Only
+ * the first WR in each chunk gets a CQE for error detection;
+ * subsequent WRs complete without individual completion events.
+ * The Send WR's signaled completion indicates all chained
+ * operations have finished.
+ */
+static int svc_rdma_prepare_write_chunk(struct svcxprt_rdma *rdma,
+ struct svc_rdma_send_ctxt *sctxt,
+ const struct svc_rdma_chunk *chunk,
+ const struct xdr_buf *xdr)
{
struct svc_rdma_write_info *info;
struct svc_rdma_chunk_ctxt *cc;
+ struct ib_send_wr *first_wr;
struct xdr_buf payload;
+ struct list_head *pos;
+ struct ib_cqe *cqe;
int ret;
if (xdr_buf_subsegment(xdr, &payload, chunk->ch_position,
@@ -623,10 +656,25 @@ static int svc_rdma_send_write_chunk(struct svcxprt_rdma *rdma,
if (ret != payload.len)
goto out_err;
- trace_svcrdma_post_write_chunk(&cc->cc_cid, cc->cc_sqecount);
- ret = svc_rdma_post_chunk_ctxt(rdma, cc);
- if (ret < 0)
+ ret = -EINVAL;
+ if (unlikely(sctxt->sc_sqecount + cc->cc_sqecount > rdma->sc_sq_depth))
goto out_err;
+
+ first_wr = sctxt->sc_wr_chain;
+ cqe = &cc->cc_cqe;
+ list_for_each(pos, &cc->cc_rwctxts) {
+ struct svc_rdma_rw_ctxt *rwc;
+
+ rwc = list_entry(pos, struct svc_rdma_rw_ctxt, rw_list);
+ first_wr = rdma_rw_ctx_wrs(&rwc->rw_ctx, rdma->sc_qp,
+ rdma->sc_port_num, cqe, first_wr);
+ cqe = NULL;
+ }
+ sctxt->sc_wr_chain = first_wr;
+ sctxt->sc_sqecount += cc->cc_sqecount;
+ list_add(&info->wi_list, &sctxt->sc_write_info_list);
+
+ trace_svcrdma_post_write_chunk(&cc->cc_cid, cc->cc_sqecount);
return 0;
out_err:
@@ -635,17 +683,19 @@ static int svc_rdma_send_write_chunk(struct svcxprt_rdma *rdma,
}
/**
- * svc_rdma_send_write_list - Send all chunks on the Write list
+ * svc_rdma_prepare_write_list - Construct WR chain for sending Write list
* @rdma: controlling RDMA transport
* @rctxt: Write list provisioned by the client
+ * @sctxt: Send WR resources
* @xdr: xdr_buf containing an RPC Reply message
*
- * Returns zero on success, or a negative errno if one or more
- * Write chunks could not be sent.
+ * Returns zero on success, or a negative errno if WR chain
+ * construction fails for one or more Write chunks.
*/
-int svc_rdma_send_write_list(struct svcxprt_rdma *rdma,
- const struct svc_rdma_recv_ctxt *rctxt,
- const struct xdr_buf *xdr)
+int svc_rdma_prepare_write_list(struct svcxprt_rdma *rdma,
+ const struct svc_rdma_recv_ctxt *rctxt,
+ struct svc_rdma_send_ctxt *sctxt,
+ const struct xdr_buf *xdr)
{
struct svc_rdma_chunk *chunk;
int ret;
@@ -653,7 +703,7 @@ int svc_rdma_send_write_list(struct svcxprt_rdma *rdma,
pcl_for_each_chunk(chunk, &rctxt->rc_write_pcl) {
if (!chunk->ch_payload_length)
break;
- ret = svc_rdma_send_write_chunk(rdma, chunk, xdr);
+ ret = svc_rdma_prepare_write_chunk(rdma, sctxt, chunk, xdr);
if (ret < 0)
return ret;
}
diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
index 02aed4932f57..3b623817862d 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
@@ -142,6 +142,7 @@ svc_rdma_send_ctxt_alloc(struct svcxprt_rdma *rdma)
ctxt->sc_send_wr.sg_list = ctxt->sc_sges;
ctxt->sc_send_wr.send_flags = IB_SEND_SIGNALED;
ctxt->sc_cqe.done = svc_rdma_wc_send;
+ INIT_LIST_HEAD(&ctxt->sc_write_info_list);
ctxt->sc_xprt_buf = buffer;
xdr_buf_init(&ctxt->sc_hdrbuf, ctxt->sc_xprt_buf,
rdma->sc_max_req_size);
@@ -227,6 +228,7 @@ static void svc_rdma_send_ctxt_release(struct svcxprt_rdma *rdma,
struct ib_device *device = rdma->sc_cm_id->device;
unsigned int i;
+ svc_rdma_write_chunk_release(rdma, ctxt);
svc_rdma_reply_chunk_release(rdma, ctxt);
if (ctxt->sc_page_count)
@@ -314,7 +316,7 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)
else
trace_svcrdma_wc_send_flush(wc, &ctxt->sc_cid);
svc_rdma_send_ctxt_put(rdma, ctxt);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -375,7 +377,7 @@ int svc_rdma_post_send(struct svcxprt_rdma *rdma,
ret = ib_post_send(rdma->sc_qp, first_wr, &bad_wr);
if (ret) {
trace_svcrdma_sq_post_err(rdma, &cid, ret);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
/* If even one WR was posted, there will be a
* Send completion that bumps sc_sq_avail.
@@ -983,6 +985,12 @@ void svc_rdma_send_error_msg(struct svcxprt_rdma *rdma,
sctxt->sc_send_wr.num_sge = 1;
sctxt->sc_send_wr.opcode = IB_WR_SEND;
sctxt->sc_sges[0].length = sctxt->sc_hdrbuf.len;
+
+ /* Ensure only the error message is posted, not any previously
+ * prepared Write chunk WRs.
+ */
+ sctxt->sc_wr_chain = &sctxt->sc_send_wr;
+ sctxt->sc_sqecount = 1;
if (svc_rdma_post_send(rdma, sctxt))
goto put_ctxt;
return;
@@ -1030,7 +1038,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
if (!p)
goto put_ctxt;
- ret = svc_rdma_send_write_list(rdma, rctxt, &rqstp->rq_res);
+ ret = svc_rdma_prepare_write_list(rdma, rctxt, sctxt, &rqstp->rq_res);
if (ret < 0)
goto put_ctxt;
@@ -1079,7 +1087,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
svc_rdma_send_ctxt_put(rdma, sctxt);
drop_connection:
trace_svcrdma_send_err(rqstp, ret);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
return -ENOTCONN;
}
diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
index 1f9d14e46c9d..b6044a1911ee 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
@@ -99,10 +99,24 @@ struct svc_xprt_class svc_rdma_class = {
.xcl_ident = XPRT_TRANSPORT_RDMA,
};
+/**
+ * svc_rdma_xprt_deferred_close - Close an RDMA transport (deferred)
+ * @rdma: transport to close
+ */
+void svc_rdma_xprt_deferred_close(struct svcxprt_rdma *rdma)
+{
+ svc_xprt_deferred_close(&rdma->sc_xprt);
+
+ /* Release threads parked on sc_send_wait. */
+ wake_up_all(&rdma->sc_send_wait);
+}
+
/* QP event handler */
static void qp_event_handler(struct ib_event *event, void *context)
{
struct svc_xprt *xprt = context;
+ struct svcxprt_rdma *rdma =
+ container_of(xprt, struct svcxprt_rdma, sc_xprt);
trace_svcrdma_qp_error(event, (struct sockaddr *)&xprt->xpt_remote);
switch (event->event) {
@@ -120,7 +134,7 @@ static void qp_event_handler(struct ib_event *event, void *context)
case IB_EVENT_QP_ACCESS_ERR:
case IB_EVENT_DEVICE_FATAL:
default:
- svc_xprt_deferred_close(xprt);
+ svc_rdma_xprt_deferred_close(rdma);
break;
}
}
@@ -311,6 +325,7 @@ static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id,
if (IS_ERR(listen_id)) {
pr_err("Listener dead, address change failed for device %s\n",
cma_id->device->name);
+ cma_xprt->sc_cm_id = NULL;
} else
cma_xprt->sc_cm_id = listen_id;
return 1;
@@ -344,7 +359,7 @@ static int svc_rdma_cma_handler(struct rdma_cm_id *cma_id,
svc_xprt_enqueue(xprt);
break;
case RDMA_CM_EVENT_DISCONNECTED:
- svc_xprt_deferred_close(xprt);
+ svc_rdma_xprt_deferred_close(rdma);
break;
default:
break;
@@ -616,7 +631,16 @@ static void svc_rdma_detach(struct svc_xprt *xprt)
struct svcxprt_rdma *rdma =
container_of(xprt, struct svcxprt_rdma, sc_xprt);
- rdma_disconnect(rdma->sc_cm_id);
+ if (rdma->sc_cm_id)
+ rdma_disconnect(rdma->sc_cm_id);
+
+ /*
+ * Most close paths go through svc_rdma_xprt_deferred_close(),
+ * which wakes the SQ waitqueue. svc_xprt_close() reaches
+ * detach without that helper, so wake any threads parked on
+ * sc_send_wait here as well.
+ */
+ wake_up_all(&rdma->sc_send_wait);
}
/**
diff --git a/net/tipc/core.c b/net/tipc/core.c
index 1ddecea1df6e..315975c3be81 100644
--- a/net/tipc/core.c
+++ b/net/tipc/core.c
@@ -45,6 +45,7 @@
#include "crypto.h"
#include <linux/module.h>
+#include <linux/wait_bit.h>
/* configurable TIPC parameters */
unsigned int tipc_net_id __read_mostly;
@@ -118,8 +119,7 @@ static void __net_exit tipc_exit_net(struct net *net)
#ifdef CONFIG_TIPC_CRYPTO
tipc_crypto_stop(&tipc_net(net)->crypto_tx);
#endif
- while (atomic_read(&tn->wq_count))
- cond_resched();
+ wait_var_event(&tn->wq_count, atomic_read(&tn->wq_count) == 0);
}
static void __net_exit tipc_pernet_pre_exit(struct net *net)
diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c
index 1b8d6bbf8a8e..17385de8bfcd 100644
--- a/net/tipc/udp_media.c
+++ b/net/tipc/udp_media.c
@@ -40,6 +40,7 @@
#include <linux/igmp.h>
#include <linux/kernel.h>
#include <linux/workqueue.h>
+#include <linux/wait_bit.h>
#include <linux/list.h>
#include <net/sock.h>
#include <net/ip.h>
@@ -94,6 +95,8 @@ struct udp_replicast {
* @ifindex: local address scope
* @work: used to schedule deferred work on a bearer
* @rcast: associated udp_replicast container
+ * @rcast_lock: serialize updates to @rcast.list against concurrent updaters
+ * @disabled: bearer is being torn down; reject further @rcast.list additions
*/
struct udp_bearer {
struct tipc_bearer __rcu *bearer;
@@ -101,6 +104,8 @@ struct udp_bearer {
u32 ifindex;
struct work_struct work;
struct udp_replicast rcast;
+ spinlock_t rcast_lock;
+ bool disabled;
};
static int tipc_udp_is_mcast_addr(struct udp_media_addr *addr)
@@ -279,26 +284,6 @@ static int tipc_udp_send_msg(struct net *net, struct sk_buff *skb,
return err;
}
-static bool tipc_udp_is_known_peer(struct tipc_bearer *b,
- struct udp_media_addr *addr)
-{
- struct udp_replicast *rcast, *tmp;
- struct udp_bearer *ub;
-
- ub = rcu_dereference_rtnl(b->media_ptr);
- if (!ub) {
- pr_err_ratelimited("UDP bearer instance not found\n");
- return false;
- }
-
- list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) {
- if (!memcmp(&rcast->addr, addr, sizeof(struct udp_media_addr)))
- return true;
- }
-
- return false;
-}
-
static int tipc_udp_rcast_add(struct tipc_bearer *b,
struct udp_media_addr *addr)
{
@@ -309,16 +294,34 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b,
if (!ub)
return -ENODEV;
+ spin_lock_bh(&ub->rcast_lock);
+ if (ub->disabled) {
+ spin_unlock_bh(&ub->rcast_lock);
+ return 0;
+ }
+ list_for_each_entry(rcast, &ub->rcast.list, list) {
+ if (!memcmp(&rcast->addr, addr, sizeof(*addr))) {
+ spin_unlock_bh(&ub->rcast_lock);
+ return 0;
+ }
+ }
+
rcast = kmalloc(sizeof(*rcast), GFP_ATOMIC);
- if (!rcast)
+ if (!rcast) {
+ spin_unlock_bh(&ub->rcast_lock);
return -ENOMEM;
+ }
if (dst_cache_init(&rcast->dst_cache, GFP_ATOMIC)) {
+ spin_unlock_bh(&ub->rcast_lock);
kfree(rcast);
return -ENOMEM;
}
memcpy(&rcast->addr, addr, sizeof(struct udp_media_addr));
+ list_add_rcu(&rcast->list, &ub->rcast.list);
+ b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT;
+ spin_unlock_bh(&ub->rcast_lock);
if (ntohs(addr->proto) == ETH_P_IP)
pr_info("New replicast peer: %pI4\n", &rcast->addr.ipv4);
@@ -326,8 +329,6 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b,
else if (ntohs(addr->proto) == ETH_P_IPV6)
pr_info("New replicast peer: %pI6\n", &rcast->addr.ipv6);
#endif
- b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT;
- list_add_rcu(&rcast->list, &ub->rcast.list);
return 0;
}
@@ -362,9 +363,6 @@ static int tipc_udp_rcast_disc(struct tipc_bearer *b, struct sk_buff *skb)
return 0;
}
- if (likely(tipc_udp_is_known_peer(b, &src)))
- return 0;
-
return tipc_udp_rcast_add(b, &src);
}
@@ -646,9 +644,6 @@ int tipc_udp_nl_bearer_add(struct tipc_bearer *b, struct nlattr *attr)
return -EINVAL;
}
- if (tipc_udp_is_known_peer(b, &addr))
- return 0;
-
return tipc_udp_rcast_add(b, &addr);
}
@@ -680,6 +675,7 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b,
return -ENOMEM;
INIT_LIST_HEAD(&ub->rcast.list);
+ spin_lock_init(&ub->rcast_lock);
if (!attrs[TIPC_NLA_BEARER_UDP_OPTS])
goto err;
@@ -802,6 +798,14 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b,
return err;
}
+static void rcast_free_rcu(struct rcu_head *rcu)
+{
+ struct udp_replicast *rcast = container_of(rcu, struct udp_replicast, rcu);
+
+ dst_cache_destroy(&rcast->dst_cache);
+ kfree(rcast);
+}
+
/* cleanup_bearer - break the socket/bearer association */
static void cleanup_bearer(struct work_struct *work)
{
@@ -809,20 +813,23 @@ static void cleanup_bearer(struct work_struct *work)
struct udp_replicast *rcast, *tmp;
struct tipc_net *tn;
+ spin_lock_bh(&ub->rcast_lock);
list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) {
- dst_cache_destroy(&rcast->dst_cache);
list_del_rcu(&rcast->list);
- kfree_rcu(rcast, rcu);
+ call_rcu_hurry(&rcast->rcu, rcast_free_rcu);
}
+ ub->disabled = true;
+ spin_unlock_bh(&ub->rcast_lock);
tn = tipc_net(sock_net(ub->ubsock->sk));
- dst_cache_destroy(&ub->rcast.dst_cache);
udp_tunnel_sock_release(ub->ubsock);
- /* Note: could use a call_rcu() to avoid another synchronize_net() */
synchronize_net();
- atomic_dec(&tn->wq_count);
+
+ dst_cache_destroy(&ub->rcast.dst_cache);
+ if (atomic_dec_and_test(&tn->wq_count))
+ wake_up_var(&tn->wq_count);
kfree(ub);
}
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 323ac8b1ad0c..dad093a3a793 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1753,6 +1753,8 @@ tls_decrypt_sw(struct sock *sk, struct tls_context *tls_ctx,
/* If opportunistic TLS 1.3 ZC failed retry without ZC */
if (unlikely(darg->zc && prot->version == TLS_1_3_VERSION &&
darg->tail != TLS_RECORD_TYPE_DATA)) {
+ iov_iter_revert(&msg->msg_iter, strp_msg(darg->skb)->full_len -
+ prot->overhead_size);
darg->zc = false;
if (!darg->tail)
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXNOPADVIOL);
diff --git a/net/wireless/core.c b/net/wireless/core.c
index 7c278b869439..a0c0de494842 100644
--- a/net/wireless/core.c
+++ b/net/wireless/core.c
@@ -165,11 +165,11 @@ int cfg80211_switch_netns(struct cfg80211_registered_device *rdev,
list_for_each_entry(wdev, &rdev->wiphy.wdev_list, list) {
if (!wdev->netdev)
continue;
- wdev->netdev->netns_local = false;
+ wdev->netdev->netns_immutable = false;
err = dev_change_net_namespace(wdev->netdev, net, "wlan%d");
if (err)
break;
- wdev->netdev->netns_local = true;
+ wdev->netdev->netns_immutable = true;
}
if (err) {
@@ -181,11 +181,11 @@ int cfg80211_switch_netns(struct cfg80211_registered_device *rdev,
list) {
if (!wdev->netdev)
continue;
- wdev->netdev->netns_local = false;
+ wdev->netdev->netns_immutable = false;
err = dev_change_net_namespace(wdev->netdev, net,
"wlan%d");
WARN_ON(err);
- wdev->netdev->netns_local = true;
+ wdev->netdev->netns_immutable = true;
}
return err;
@@ -237,6 +237,7 @@ void cfg80211_stop_p2p_device(struct cfg80211_registered_device *rdev,
if (!wdev_running(wdev))
return;
+ cfg80211_pmsr_wdev_down(wdev);
rdev_stop_p2p_device(rdev, wdev);
wdev->is_running = false;
@@ -262,6 +263,8 @@ void cfg80211_stop_nan(struct cfg80211_registered_device *rdev,
if (!wdev_running(wdev))
return;
+ cfg80211_pmsr_wdev_down(wdev);
+
rdev_stop_nan(rdev, wdev);
wdev->is_running = false;
@@ -1280,6 +1283,7 @@ static void _cfg80211_unregister_wdev(struct wireless_dev *wdev,
list_del_rcu(&wdev->list);
synchronize_net();
rdev->devlist_generation++;
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
cfg80211_mlme_purge_registrations(wdev);
@@ -1463,7 +1467,7 @@ void cfg80211_init_wdev(struct wireless_dev *wdev)
wdev->iftype == NL80211_IFTYPE_ADHOC) && !wdev->use_4addr)
wdev->netdev->priv_flags |= IFF_DONT_BRIDGE;
- INIT_WORK(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
+ wiphy_work_init(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
}
void cfg80211_register_wdev(struct cfg80211_registered_device *rdev,
@@ -1545,7 +1549,7 @@ static int cfg80211_netdev_notifier_call(struct notifier_block *nb,
SET_NETDEV_DEVTYPE(dev, &wiphy_type);
wdev->netdev = dev;
/* can only change netns with wiphy */
- dev->netns_local = true;
+ dev->netns_immutable = true;
cfg80211_init_wdev(wdev);
break;
@@ -1571,9 +1575,9 @@ static int cfg80211_netdev_notifier_call(struct notifier_block *nb,
scoped_guard(wiphy, &rdev->wiphy) {
cfg80211_leave(rdev, wdev);
cfg80211_remove_links(wdev);
+ /* since we just did cfg80211_leave() nothing to do there */
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
}
- /* since we just did cfg80211_leave() nothing to do there */
- cancel_work_sync(&wdev->disconnect_wk);
break;
case NETDEV_DOWN:
wiphy_lock(&rdev->wiphy);
diff --git a/net/wireless/core.h b/net/wireless/core.h
index b02b456cfca3..505acfe4a16d 100644
--- a/net/wireless/core.h
+++ b/net/wireless/core.h
@@ -418,7 +418,7 @@ void __cfg80211_port_authorized(struct wireless_dev *wdev, const u8 *peer_addr,
const u8 *td_bitmap, u8 td_bitmap_len);
int cfg80211_mgd_wext_connect(struct cfg80211_registered_device *rdev,
struct wireless_dev *wdev);
-void cfg80211_autodisconnect_wk(struct work_struct *work);
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work);
/* SME implementation */
void cfg80211_conn_work(struct work_struct *work);
diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c
index 6261befa1e94..c5df80b54707 100644
--- a/net/wireless/nl80211.c
+++ b/net/wireless/nl80211.c
@@ -16171,15 +16171,19 @@ static int nl80211_color_change(struct sk_buff *skb, struct genl_info *info)
params.count = nla_get_u8(info->attrs[NL80211_ATTR_COLOR_CHANGE_COUNT]);
params.color = nla_get_u8(info->attrs[NL80211_ATTR_COLOR_CHANGE_COLOR]);
- err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next,
- info->extack);
- if (err)
- return err;
+ params.link_id = nl80211_link_id(info->attrs);
+ if (!wdev->links[params.link_id].ap.beacon_interval)
+ return -EINVAL;
tb = kcalloc(NL80211_ATTR_MAX + 1, sizeof(*tb), GFP_KERNEL);
if (!tb)
return -ENOMEM;
+ err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next,
+ info->extack);
+ if (err)
+ goto out;
+
err = nla_parse_nested(tb, NL80211_ATTR_MAX,
info->attrs[NL80211_ATTR_COLOR_CHANGE_ELEMS],
nl80211_policy, info->extack);
@@ -16236,7 +16240,6 @@ static int nl80211_color_change(struct sk_buff *skb, struct genl_info *info)
params.counter_offset_presp = offset;
}
- params.link_id = nl80211_link_id(info->attrs);
err = rdev_color_change(rdev, dev, ¶ms);
out:
@@ -20153,7 +20156,8 @@ static int nl80211_netlink_notify(struct notifier_block * nb,
wdev->nl_owner_dead = true;
schedule_work(&rdev->destroy_work);
} else if (wdev->conn_owner_nlportid == notify->portid) {
- schedule_work(&wdev->disconnect_wk);
+ wiphy_work_queue(wdev->wiphy,
+ &wdev->disconnect_wk);
}
cfg80211_release_pmsr(wdev, notify->portid);
diff --git a/net/wireless/scan.c b/net/wireless/scan.c
index 6fab9abb47b2..890e5f77de6b 100644
--- a/net/wireless/scan.c
+++ b/net/wireless/scan.c
@@ -1030,6 +1030,7 @@ int cfg80211_scan(struct cfg80211_registered_device *rdev)
struct cfg80211_scan_request *request;
struct cfg80211_scan_request *rdev_req = rdev->scan_req;
u32 n_channels = 0, idx, i;
+ int err;
if (!(rdev->wiphy.flags & WIPHY_FLAG_SPLIT_SCAN_6GHZ))
return rdev_scan(rdev, rdev_req);
@@ -1056,8 +1057,14 @@ int cfg80211_scan(struct cfg80211_registered_device *rdev)
}
rdev_req->scan_6ghz = false;
+ err = rdev_scan(rdev, request);
+ if (err) {
+ kfree(request);
+ return err;
+ }
+
rdev->int_scan_req = request;
- return rdev_scan(rdev, request);
+ return 0;
}
void ___cfg80211_scan_done(struct cfg80211_registered_device *rdev,
@@ -3567,8 +3574,10 @@ int cfg80211_wext_siwscan(struct net_device *dev,
/* translate "Scan for SSID" request */
if (wreq) {
if (wrqu->data.flags & IW_SCAN_THIS_ESSID) {
- if (wreq->essid_len > IEEE80211_MAX_SSID_LEN)
- return -EINVAL;
+ if (wreq->essid_len > IEEE80211_MAX_SSID_LEN) {
+ err = -EINVAL;
+ goto out;
+ }
memcpy(creq->ssids[0].ssid, wreq->essid, wreq->essid_len);
creq->ssids[0].ssid_len = wreq->essid_len;
}
diff --git a/net/wireless/sme.c b/net/wireless/sme.c
index ab57648ca577..059b01bdb630 100644
--- a/net/wireless/sme.c
+++ b/net/wireless/sme.c
@@ -1578,13 +1578,11 @@ int cfg80211_disconnect(struct cfg80211_registered_device *rdev,
* Used to clean up after the connection / connection attempt owner socket
* disconnects
*/
-void cfg80211_autodisconnect_wk(struct work_struct *work)
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work)
{
struct wireless_dev *wdev =
container_of(work, struct wireless_dev, disconnect_wk);
- struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
-
- guard(wiphy)(wdev->wiphy);
+ struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
if (wdev->conn_owner_nlportid) {
switch (wdev->iftype) {
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index 36170e10deda..8dd43f06d6ed 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -440,6 +440,23 @@ void __xsk_map_flush(struct list_head *flush_list)
void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries)
{
+ u32 reclaim_descs = READ_ONCE(pool->reclaim_descs);
+
+ if (unlikely(reclaim_descs)) {
+ u32 pending_descs = READ_ONCE(pool->tx_zc_pending_descs);
+
+ if (nb_entries < pending_descs) {
+ WRITE_ONCE(pool->tx_zc_pending_descs,
+ pending_descs - nb_entries);
+ xskq_prod_submit_n(pool->cq, nb_entries);
+ return;
+ }
+
+ WRITE_ONCE(pool->tx_zc_pending_descs, 0);
+ nb_entries += reclaim_descs;
+ WRITE_ONCE(pool->reclaim_descs, 0);
+ }
+
xskq_prod_submit_n(pool->cq, nb_entries);
}
EXPORT_SYMBOL(xsk_tx_completed);
@@ -518,24 +535,158 @@ static u32 xsk_tx_peek_release_fallback(struct xsk_buff_pool *pool, u32 max_entr
return nb_pkts;
}
+static void xsk_tx_commit_batch(struct xsk_buff_pool *pool,
+ struct xsk_tx_batch *batch)
+{
+ u32 nb_descs = xsk_tx_batch_cq_descs(batch);
+ u32 cq_cached_prod;
+
+ if (!nb_descs)
+ return;
+
+ cq_cached_prod = pool->cq->cached_prod;
+ xskq_prod_write_addr_batch(pool->cq, pool->tx_descs, nb_descs);
+
+ if (unlikely(batch->reclaim_descs)) {
+ u32 cq_pending_descs;
+
+ /* CQ is positional. Descriptors already written but not
+ * submitted must complete before any reclaim-only descriptors
+ * appended below.
+ */
+ cq_pending_descs = cq_cached_prod -
+ READ_ONCE(pool->cq->ring->producer);
+
+ WRITE_ONCE(pool->tx_zc_pending_descs,
+ batch->tx_descs + cq_pending_descs);
+ WRITE_ONCE(pool->reclaim_descs, batch->reclaim_descs);
+ if (unlikely(!pool->tx_zc_pending_descs))
+ xsk_tx_completed(pool, 0);
+ }
+}
+
+static struct xsk_tx_batch
+__xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, struct xdp_sock *xs,
+ struct xdp_desc *descs, u32 max_descs)
+{
+ struct xsk_tx_batch batch = {};
+ u32 entries;
+
+ entries = xskq_cons_nb_entries(xs->tx, max_descs);
+ if (!entries)
+ return batch;
+
+ batch = xskq_cons_read_desc_batch(xs, pool, descs, max_descs);
+ if (!xsk_tx_batch_cq_descs(&batch)) {
+ xs->tx->queue_empty_descs++;
+ } else {
+ __xskq_cons_release(xs->tx);
+ xs->sk.sk_write_space(&xs->sk);
+ }
+ return batch;
+}
+
+static struct xsk_tx_batch
+xsk_tx_peek_release_shared_desc_batch(struct xsk_buff_pool *pool, u32 max_descs)
+{
+ u32 cq_descs_before, cq_descs_after;
+ struct xsk_tx_batch sum_batch = {};
+ bool budget_exhausted;
+ u32 per_socket_budget;
+ struct xdp_sock *xs;
+
+ /* The fairness quota must allow one maximum-sized valid packet. */
+ per_socket_budget = max_t(u32, MAX_PER_SOCKET_BUDGET,
+ pool->netdev->xdp_zc_max_segs);
+
+again:
+ budget_exhausted = false;
+ cq_descs_before = xsk_tx_batch_cq_descs(&sum_batch);
+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
+ u32 budget, budget_left, offset, remaining, used;
+ struct xsk_tx_batch curr_batch;
+
+ /* Once reclaim-only descriptors have been appended to the CQ
+ * address area, do not append driver-visible Tx descriptors
+ * from another socket after them. xsk_tx_completed() relies on
+ * all driver-visible descriptors preceding all reclaim-only
+ * descriptors in CQ order.
+ */
+ if (sum_batch.reclaim_descs)
+ break;
+
+ /* be gentle when playing with pool->tx_descs */
+ offset = xsk_tx_batch_cq_descs(&sum_batch);
+ if (offset >= max_descs)
+ break;
+
+ if (xs->tx_budget_spent >= per_socket_budget) {
+ if (xskq_cons_nb_entries(xs->tx, 1))
+ budget_exhausted = true;
+ continue;
+ }
+
+ budget_left = per_socket_budget - xs->tx_budget_spent;
+ remaining = max_descs - offset;
+ budget = min(remaining, budget_left);
+
+ curr_batch = __xsk_tx_peek_release_desc_batch(pool, xs,
+ pool->tx_descs + offset,
+ budget);
+ used = xsk_tx_batch_cq_descs(&curr_batch);
+ if (!used) {
+ if (curr_batch.budget_limited && budget_left < remaining)
+ budget_exhausted = true;
+ continue;
+ }
+
+ xs->tx_budget_spent += used;
+ sum_batch.tx_descs += curr_batch.tx_descs;
+ sum_batch.reclaim_descs = curr_batch.reclaim_descs;
+ }
+
+ cq_descs_after = xsk_tx_batch_cq_descs(&sum_batch);
+
+ if (sum_batch.reclaim_descs || cq_descs_after >= max_descs)
+ return sum_batch;
+
+ /* Continue filling the batch while this pass made progress */
+ if (cq_descs_before != cq_descs_after)
+ goto again;
+
+ if (!budget_exhausted)
+ return sum_batch;
+
+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list)
+ xs->tx_budget_spent = 0;
+ goto again;
+}
+
u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 nb_pkts)
{
+ struct xsk_tx_batch batch = {};
struct xdp_sock *xs;
+ bool umem_shared;
rcu_read_lock();
- if (!list_is_singular(&pool->xsk_tx_list)) {
- /* Fallback to the non-batched version */
- rcu_read_unlock();
- return xsk_tx_peek_release_fallback(pool, nb_pkts);
- }
+ if (unlikely(READ_ONCE(pool->reclaim_descs)))
+ goto out;
- xs = list_first_or_null_rcu(&pool->xsk_tx_list, struct xdp_sock, tx_list);
- if (!xs) {
- nb_pkts = 0;
+ xs = list_first_or_null_rcu(&pool->xsk_tx_list, struct xdp_sock,
+ tx_list);
+ if (!xs)
goto out;
- }
- nb_pkts = xskq_cons_nb_entries(xs->tx, nb_pkts);
+ nb_pkts = min(nb_pkts, pool->tx_descs_nentries);
+ if (!nb_pkts)
+ goto out;
+
+ umem_shared = !list_is_singular(&pool->xsk_tx_list);
+
+ if (umem_shared && !(pool->umem->flags & XDP_UMEM_SG_FLAG)) {
+ rcu_read_unlock();
+ return xsk_tx_peek_release_fallback(pool, nb_pkts);
+ }
/* This is the backpressure mechanism for the Tx path. Try to
* reserve space in the completion queue for all packets, but
@@ -547,19 +698,16 @@ u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 nb_pkts)
if (!nb_pkts)
goto out;
- nb_pkts = xskq_cons_read_desc_batch(xs->tx, pool, nb_pkts);
- if (!nb_pkts) {
- xs->tx->queue_empty_descs++;
- goto out;
- }
-
- __xskq_cons_release(xs->tx);
- xskq_prod_write_addr_batch(pool->cq, pool->tx_descs, nb_pkts);
- xs->sk.sk_write_space(&xs->sk);
+ batch = umem_shared ?
+ xsk_tx_peek_release_shared_desc_batch(pool, nb_pkts) :
+ __xsk_tx_peek_release_desc_batch(pool, xs,
+ pool->tx_descs,
+ nb_pkts);
+ xsk_tx_commit_batch(pool, &batch);
out:
rcu_read_unlock();
- return nb_pkts;
+ return batch.tx_descs;
}
EXPORT_SYMBOL(xsk_tx_peek_release_desc_batch);
@@ -600,6 +748,28 @@ static void xsk_cq_cancel_locked(struct xdp_sock *xs, u32 n)
spin_unlock_irqrestore(&xs->pool->cq_lock, flags);
}
+static int xsk_cq_submit_addr_single_locked(struct xsk_buff_pool *pool,
+ struct xdp_desc *desc)
+{
+ struct xsk_queue *cq = pool->cq;
+ unsigned long flags;
+ int err = -EAGAIN;
+
+ spin_lock_irqsave(&pool->cq_lock, flags);
+ /* Addresses are written at reserve time and publishing is positional,
+ * so the entry can only be published if no earlier Tx is still in
+ * flight in front of it. The caller retries later otherwise.
+ */
+ if (cq->cached_prod == READ_ONCE(cq->ring->producer)) {
+ err = xskq_prod_reserve_addr(cq, desc->addr);
+ if (!err)
+ xskq_prod_submit_n(cq, 1);
+ }
+ spin_unlock_irqrestore(&pool->cq_lock, flags);
+
+ return err;
+}
+
static u32 xsk_get_num_desc(struct sk_buff *skb)
{
return skb ? (long)skb_shinfo(skb)->destructor_arg : 0;
@@ -822,12 +992,13 @@ static int __xsk_generic_xmit(struct sock *sk)
{
struct xdp_sock *xs = xdp_sk(sk);
u32 max_batch = TX_BATCH_SIZE;
- bool sent_frame = false;
struct xdp_desc desc;
struct sk_buff *skb;
+ u32 cached_cons;
int err = 0;
mutex_lock(&xs->mutex);
+ cached_cons = xs->tx->cached_cons;
/* Since we dropped the RCU read lock, the socket state might have changed. */
if (unlikely(!xsk_is_bound(xs))) {
@@ -844,6 +1015,17 @@ static int __xsk_generic_xmit(struct sock *sk)
goto out;
}
+ if (unlikely(xs->drain_cont)) {
+ if (xsk_cq_submit_addr_single_locked(xs->pool, &desc)) {
+ err = -EAGAIN;
+ goto out;
+ }
+ xs->tx->invalid_descs++;
+ xskq_cons_release(xs->tx);
+ xs->drain_cont = xp_mb_desc(&desc);
+ continue;
+ }
+
/* This is the backpressure mechanism for the Tx path.
* Reserve space in the completion queue and only proceed
* if there is space in it. This avoids having to implement
@@ -857,6 +1039,8 @@ static int __xsk_generic_xmit(struct sock *sk)
err = PTR_ERR(skb);
if (err != -EOVERFLOW)
goto out;
+ if (xp_mb_desc(&desc))
+ xs->drain_cont = true;
err = 0;
continue;
}
@@ -885,18 +1069,29 @@ static int __xsk_generic_xmit(struct sock *sk)
goto out;
}
- sent_frame = true;
xs->skb = NULL;
}
if (xskq_has_descs(xs->tx)) {
+ bool drain = xs->skb || xs->drain_cont || xp_mb_desc(&desc);
+
if (xs->skb)
xsk_drop_skb(xs->skb);
+
+ err = xsk_cq_submit_addr_single_locked(xs->pool, &desc);
+ if (err) {
+ xs->tx->invalid_descs--;
+ xs->drain_cont = drain;
+ err = -EAGAIN;
+ goto out;
+ }
+
xskq_cons_release(xs->tx);
+ xs->drain_cont = xp_mb_desc(&desc);
}
out:
- if (sent_frame)
+ if (xs->tx->cached_cons != cached_cons)
if (xsk_tx_writeable(xs))
sk->sk_write_space(sk);
@@ -1270,7 +1465,8 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
* and/or device.
*/
xs->pool = xp_create_and_assign_umem(xs,
- umem_xs->umem);
+ umem_xs->umem,
+ dev->xdp_zc_max_segs);
if (!xs->pool) {
err = -ENOMEM;
sockfd_put(sock);
@@ -1302,7 +1498,8 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
* utilizes
*/
if (xs->tx && !xs->pool->tx_descs) {
- err = xp_alloc_tx_descs(xs->pool, xs);
+ err = xp_alloc_tx_descs(xs->pool, xs,
+ dev->xdp_zc_max_segs);
if (err) {
xp_put_pool(xs->pool);
xs->pool = NULL;
@@ -1320,7 +1517,9 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
goto out_unlock;
} else {
/* This xsk has its own umem. */
- xs->pool = xp_create_and_assign_umem(xs, xs->umem);
+ xs->pool = xp_create_and_assign_umem(xs, xs->umem,
+ dev->xdp_zc_max_segs);
+
if (!xs->pool) {
err = -ENOMEM;
goto out_unlock;
diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c
index a22cedd280cb..925e0e62491d 100644
--- a/net/xdp/xsk_buff_pool.c
+++ b/net/xdp/xsk_buff_pool.c
@@ -44,18 +44,22 @@ void xp_destroy(struct xsk_buff_pool *pool)
kvfree(pool);
}
-int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs)
+int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs,
+ u32 max_segs)
{
- pool->tx_descs = kvcalloc(xs->tx->nentries, sizeof(*pool->tx_descs),
- GFP_KERNEL);
+ u32 nentries = max(xs->tx->nentries, max_segs);
+
+ pool->tx_descs = kvcalloc(nentries, sizeof(*pool->tx_descs), GFP_KERNEL);
if (!pool->tx_descs)
return -ENOMEM;
+ pool->tx_descs_nentries = nentries;
return 0;
}
struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
- struct xdp_umem *umem)
+ struct xdp_umem *umem,
+ u32 max_segs)
{
bool unaligned = umem->flags & XDP_UMEM_UNALIGNED_CHUNK_FLAG;
struct xsk_buff_pool *pool;
@@ -72,7 +76,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
goto out;
if (xs->tx)
- if (xp_alloc_tx_descs(pool, xs))
+ if (xp_alloc_tx_descs(pool, xs, max_segs))
goto out;
pool->chunk_mask = ~((u64)umem->chunk_size - 1);
diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h
index 2ec64c185f8f..43d4de9afcf9 100644
--- a/net/xdp/xsk_queue.h
+++ b/net/xdp/xsk_queue.h
@@ -53,6 +53,17 @@ struct parsed_desc {
u32 valid;
};
+struct xsk_tx_batch {
+ u32 tx_descs;
+ u32 reclaim_descs;
+ bool budget_limited;
+};
+
+static inline u32 xsk_tx_batch_cq_descs(const struct xsk_tx_batch *batch)
+{
+ return batch->tx_descs + batch->reclaim_descs;
+}
+
/* The structure of the shared state of the rings are a simple
* circular buffer, as outlined in
* Documentation/core-api/circular-buffers.rst. For the Rx and
@@ -258,17 +269,18 @@ static inline void parse_desc(struct xsk_queue *q, struct xsk_buff_pool *pool,
parsed->mb = xp_mb_desc(desc);
}
-static inline
-u32 xskq_cons_read_desc_batch(struct xsk_queue *q, struct xsk_buff_pool *pool,
- u32 max)
+static inline struct xsk_tx_batch
+xskq_cons_read_desc_batch(struct xdp_sock *xs, struct xsk_buff_pool *pool,
+ struct xdp_desc *descs, u32 max)
{
- u32 cached_cons = q->cached_cons, nb_entries = 0;
- struct xdp_desc *descs = pool->tx_descs;
- u32 total_descs = 0, nr_frags = 0;
+ bool drain = READ_ONCE(xs->drain_cont);
+ u32 cached_cons, nb_entries = 0;
+ struct xsk_tx_batch batch = {};
+ struct xsk_queue *q = xs->tx;
+ u32 nr_frags = 0;
+
+ cached_cons = q->cached_cons;
- /* track first entry, if stumble upon *any* invalid descriptor, rewind
- * current packet that consists of frags and stop the processing
- */
while (cached_cons != q->cached_prod && nb_entries < max) {
struct xdp_rxtx_ring *ring = (struct xdp_rxtx_ring *)q->ring;
u32 idx = cached_cons & q->ring_mask;
@@ -278,25 +290,42 @@ u32 xskq_cons_read_desc_batch(struct xsk_queue *q, struct xsk_buff_pool *pool,
cached_cons++;
parse_desc(q, pool, &descs[nb_entries], &parsed);
if (unlikely(!parsed.valid))
- break;
+ drain = true;
+
+ nr_frags++;
+ nb_entries++;
if (likely(!parsed.mb)) {
- total_descs += (nr_frags + 1);
- nr_frags = 0;
- } else {
- nr_frags++;
- if (nr_frags == pool->netdev->xdp_zc_max_segs) {
+ if (unlikely(drain)) {
+ batch.reclaim_descs = nr_frags;
+ WRITE_ONCE(xs->drain_cont, false);
nr_frags = 0;
break;
}
+
+ batch.tx_descs += nr_frags;
+ nr_frags = 0;
+ continue;
+ }
+
+ if (nr_frags == pool->netdev->xdp_zc_max_segs)
+ drain = true;
+ }
+
+ if (nr_frags) {
+ if (drain) {
+ batch.reclaim_descs = nr_frags;
+ WRITE_ONCE(xs->drain_cont, true);
+ } else {
+ if (nb_entries == max)
+ batch.budget_limited = true;
+ cached_cons -= nr_frags;
}
- nb_entries++;
}
- cached_cons -= nr_frags;
/* Release valid plus any invalid entries */
xskq_cons_release_n(q, cached_cons - q->cached_cons);
- return total_descs;
+ return batch;
}
/* Functions for consumers */
diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c
index 7744ed8e4c83..084cc90613d6 100644
--- a/net/xfrm/xfrm_device.c
+++ b/net/xfrm/xfrm_device.c
@@ -226,6 +226,14 @@ struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t featur
pskb = skb2;
}
+ /* skb_gso_segment() set skb->prev to the last segment, but async
+ * crypto may have stolen it above without updating ->prev. Repoint
+ * it at the last retained segment so validate_xmit_skb_list() does
+ * not chain onto a segment now owned by the crypto engine.
+ */
+ if (skb)
+ skb->prev = pskb;
+
return skb;
}
EXPORT_SYMBOL_GPL(validate_xmit_xfrm);
diff --git a/security/apparmor/domain.c b/security/apparmor/domain.c
index ced513197ed5..8685c10e6754 100644
--- a/security/apparmor/domain.c
+++ b/security/apparmor/domain.c
@@ -812,6 +812,52 @@ static int profile_onexec(const struct cred *subj_cred,
/* ensure none ns domain transitions are correctly applied with onexec */
+static bool is_profile_priv_restricted_to_stack(const struct cred *subj_cred,
+ struct aa_profile *profile)
+{
+ if (profile_unconfined(profile) && profile == profile->ns->unconfined &&
+ aa_unprivileged_unconfined_restricted &&
+ /* cap_capable returns false (0) if true, hence true here means
+ * doesn't have capability and the stack will be restricted
+ */
+ cap_capable(current_cred(), &init_user_ns, CAP_MAC_OVERRIDE,
+ CAP_OPT_NOAUDIT))
+ return true;
+ return false;
+}
+
+static const char *stack_msg = "change_profile unprivileged unconfined converted to stacking";
+
+static struct aa_label *priv_restricted_transition(const struct cred *subj_cred,
+ struct aa_profile *profile,
+ const char *op, u32 request,
+ const char *name,
+ struct aa_label *transition,
+ gfp_t gfp)
+{
+ if (!is_profile_priv_restricted_to_stack(subj_cred, profile))
+ return aa_get_newest_label(transition);
+
+ /* transition allowed but only via stack */
+ struct aa_label *target = aa_label_merge(&profile->label,
+ transition, gfp);
+ if (!target)
+ return ERR_PTR(-ENOMEM);
+
+ /* doing this here is less than optimal but good enough until the
+ * fs mediation rework lands
+ */
+ struct aa_perms perms = {
+ .allow = request,
+ .audit = request,
+ };
+ aa_audit_file(subj_cred, profile, &perms, op,
+ request, name, NULL, target,
+ subj_cred->euid, stack_msg, 0);
+
+ return target;
+}
+
static struct aa_label *handle_onexec(const struct cred *subj_cred,
struct aa_label *label,
struct aa_label *onexec, bool stack,
@@ -835,7 +881,10 @@ static struct aa_label *handle_onexec(const struct cred *subj_cred,
if (error)
return ERR_PTR(error);
new = fn_label_build_in_ns(label, profile, GFP_KERNEL,
- aa_get_newest_label(onexec),
+ priv_restricted_transition(subj_cred, profile,
+ OP_CHANGE_ONEXEC, AA_MAY_ONEXEC,
+ bprm->filename, onexec,
+ GFP_KERNEL),
profile_transition(subj_cred, profile, bprm,
buffer,
cond, unsafe));
@@ -1036,8 +1085,8 @@ static struct aa_label *build_change_hat(const struct cred *subj_cred,
if (!hat) {
error = -ENOENT;
if (COMPLAIN_MODE(profile)) {
- hat = aa_new_learning_profile(profile, true, name,
- GFP_KERNEL);
+ hat = __aa_new_learning_profile(profile, true, name,
+ GFP_KERNEL);
if (!hat) {
info = "failed null profile create";
error = -ENOMEM;
@@ -1075,6 +1124,7 @@ static struct aa_label *change_hat(const struct cred *subj_cred,
bool sibling = false;
const char *name, *info = NULL;
int i, error;
+ bool needput = false;
AA_BUG(!label);
AA_BUG(!hats);
@@ -1087,7 +1137,6 @@ static struct aa_label *change_hat(const struct cred *subj_cred,
* the profiles and label, we can rely on the namespaces being live
* and avoid incrementing their refcounts while grabbing the lock.
*/
- label = aa_get_label(label);
ns = labels_ns(label);
retry:
@@ -1095,15 +1144,19 @@ static struct aa_label *change_hat(const struct cred *subj_cred,
if (label_is_stale(label)) {
new = aa_get_newest_label(label);
new_ns = labels_ns(new);
+
+ if (needput)
+ /* aa_put_label() is safe to call when under lock */
+ aa_put_label(label);
+ label = new;
+ needput = true;
+ /* check if replaced with label in parent ns, and lock there */
if (new_ns != ns) {
- aa_put_label(new);
mutex_unlock(&ns->lock);
ns = new_ns;
- label = new;
+ /* retry will bottom out at the root of the tree */
goto retry;
}
- aa_put_label(label);
- label = new;
}
if (PROFILE_IS_HAT(labels_profile(label)))
@@ -1114,7 +1167,8 @@ static struct aa_label *change_hat(const struct cred *subj_cred,
name = hats[i];
label_for_each_in_ns(it, labels_ns(label), label, profile) {
if (sibling && PROFILE_IS_HAT(profile)) {
- root = aa_get_profile(profile->parent);
+ root = aa_get_profile(rcu_dereference_protected(profile->parent,
+ mutex_is_locked(&ns->lock)));
} else if (!sibling && !PROFILE_IS_HAT(profile)) {
root = aa_get_profile(profile);
} else { /* conflicting change type */
@@ -1175,6 +1229,8 @@ static struct aa_label *change_hat(const struct cred *subj_cred,
}
}
mutex_unlock(&ns->lock);
+ if (needput)
+ aa_put_label(label);
return ERR_PTR(error);
build:
@@ -1188,6 +1244,8 @@ static struct aa_label *change_hat(const struct cred *subj_cred,
goto fail;
} /* else if (IS_ERR) build_change_hat has logged error so return new */
mutex_unlock(&ns->lock);
+ if (needput)
+ aa_put_label(label);
return new;
}
@@ -1345,8 +1403,6 @@ static int change_profile_perms_wrapper(const char *op, const char *name,
return error;
}
-static const char *stack_msg = "change_profile unprivileged unconfined converted to stacking";
-
/**
* aa_change_profile - perform a one-way profile transition
* @fqname: name of profile may include namespace (NOT NULL)
@@ -1406,28 +1462,6 @@ int aa_change_profile(const char *fqname, int flags)
op = OP_CHANGE_PROFILE;
}
- /* This should move to a per profile test. Requires pushing build
- * into callback
- */
- if (!stack && unconfined(label) &&
- label == &labels_ns(label)->unconfined->label &&
- aa_unprivileged_unconfined_restricted &&
- /* TODO: refactor so this check is a fn */
- cap_capable(current_cred(), &init_user_ns, CAP_MAC_OVERRIDE,
- CAP_OPT_NOAUDIT)) {
- /* regardless of the request in this case apparmor
- * stacks against unconfined so admin set policy can't be
- * by-passed
- */
- stack = true;
- perms.audit = request;
- (void) fn_for_each_in_ns(label, profile,
- aa_audit_file(subj_cred, profile, &perms, op,
- request, auditname, NULL, target,
- GLOBAL_ROOT_UID, stack_msg, 0));
- perms.audit = 0;
- }
-
if (*fqname == '&') {
stack = true;
/* don't have label_parse() do stacking */
@@ -1498,7 +1532,10 @@ int aa_change_profile(const char *fqname, int flags)
/* stacking is always a subset, so only check the nonstack case */
if (!stack) {
new = fn_label_build_in_ns(label, profile, GFP_KERNEL,
- aa_get_label(target),
+ priv_restricted_transition(subj_cred, profile,
+ op, request,
+ auditname, target,
+ GFP_KERNEL),
aa_get_label(&profile->label));
if (IS_ERR_OR_NULL(new))
goto build_fail;
diff --git a/security/apparmor/include/policy.h b/security/apparmor/include/policy.h
index dbd6af6034f1..e85824c90f43 100644
--- a/security/apparmor/include/policy.h
+++ b/security/apparmor/include/policy.h
@@ -258,6 +258,9 @@ struct aa_profile *aa_alloc_profile(const char *name, struct aa_proxy *proxy,
gfp_t gfp);
struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name,
gfp_t gfp);
+struct aa_profile *__aa_new_learning_profile(struct aa_profile *parent,
+ bool hat, const char *base,
+ gfp_t gfp);
struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat,
const char *base, gfp_t gfp);
void aa_free_profile(struct aa_profile *profile);
diff --git a/security/apparmor/policy.c b/security/apparmor/policy.c
index da39bd937c2a..b3a602c9c23d 100644
--- a/security/apparmor/policy.c
+++ b/security/apparmor/policy.c
@@ -667,7 +667,7 @@ struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name,
}
/**
- * aa_new_learning_profile - create or find a null-X learning profile
+ * __aa_new_learning_profile - create or find a null-X learning profile
* @parent: profile that caused this profile to be created (NOT NULL)
* @hat: true if the null- learning profile is a hat
* @base: name to base the null profile off of
@@ -684,14 +684,16 @@ struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name,
*
* Returns: new refcounted profile else NULL on failure
*/
-struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat,
- const char *base, gfp_t gfp)
+struct aa_profile *__aa_new_learning_profile(struct aa_profile *parent,
+ bool hat, const char *base,
+ gfp_t gfp)
{
struct aa_profile *p, *profile;
const char *bname;
char *name = NULL;
AA_BUG(!parent);
+ AA_BUG(!mutex_is_locked(&parent->ns->lock));
if (base) {
name = kmalloc(strlen(parent->base.hname) + 8 + strlen(base),
@@ -723,7 +725,6 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat,
if (hat)
profile->label.flags |= FLAG_HAT;
- mutex_lock_nested(&profile->ns->lock, profile->ns->level);
p = __find_child(&parent->base.profiles, bname);
if (p) {
aa_free_profile(profile);
@@ -731,7 +732,6 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat,
} else {
__add_profile(&parent->base.profiles, profile);
}
- mutex_unlock(&profile->ns->lock);
/* refcount released by caller */
out:
@@ -745,6 +745,18 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat,
return NULL;
}
+struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat,
+ const char *base, gfp_t gfp)
+{
+ struct aa_profile *profile;
+
+ mutex_lock_nested(&parent->ns->lock, parent->ns->level);
+ profile = __aa_new_learning_profile(parent, hat, base, gfp);
+ mutex_unlock(&parent->ns->lock);
+
+ return profile;
+}
+
/**
* replacement_allowed - test to see if replacement is allowed
* @profile: profile to test if it can be replaced (MAYBE NULL)
diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c
index ec3030614ab2..1ee472482c8a 100644
--- a/security/selinux/selinuxfs.c
+++ b/security/selinux/selinuxfs.c
@@ -74,7 +74,6 @@ struct selinux_fs_info {
int *bool_pending_values;
struct dentry *class_dir;
unsigned long last_class_ino;
- bool policy_opened;
struct dentry *policycap_dir;
unsigned long last_ino;
struct super_block *sb;
@@ -340,44 +339,31 @@ struct policy_load_memory {
static int sel_open_policy(struct inode *inode, struct file *filp)
{
- struct selinux_fs_info *fsi = inode->i_sb->s_fs_info;
struct policy_load_memory *plm = NULL;
int rc;
- BUG_ON(filp->private_data);
-
- mutex_lock(&selinux_state.policy_mutex);
-
rc = avc_has_perm(current_sid(), SECINITSID_SECURITY,
SECCLASS_SECURITY, SECURITY__READ_POLICY, NULL);
if (rc)
- goto err;
-
- rc = -EBUSY;
- if (fsi->policy_opened)
- goto err;
+ return rc;
- rc = -ENOMEM;
plm = kzalloc(sizeof(*plm), GFP_KERNEL);
if (!plm)
- goto err;
+ return -ENOMEM;
+ mutex_lock(&selinux_state.policy_mutex);
rc = security_read_policy(&plm->data, &plm->len);
if (rc)
goto err;
-
if ((size_t)i_size_read(inode) != plm->len) {
inode_lock(inode);
i_size_write(inode, plm->len);
inode_unlock(inode);
}
-
- fsi->policy_opened = 1;
+ mutex_unlock(&selinux_state.policy_mutex);
filp->private_data = plm;
- mutex_unlock(&selinux_state.policy_mutex);
-
return 0;
err:
mutex_unlock(&selinux_state.policy_mutex);
@@ -390,13 +376,8 @@ static int sel_open_policy(struct inode *inode, struct file *filp)
static int sel_release_policy(struct inode *inode, struct file *filp)
{
- struct selinux_fs_info *fsi = inode->i_sb->s_fs_info;
struct policy_load_memory *plm = filp->private_data;
- BUG_ON(!plm);
-
- fsi->policy_opened = 0;
-
vfree(plm->data);
kfree(plm);
diff --git a/sound/core/seq/oss/seq_oss_event.c b/sound/core/seq/oss/seq_oss_event.c
index 7b7c925dd3aa..4699c084d018 100644
--- a/sound/core/seq/oss/seq_oss_event.c
+++ b/sound/core/seq/oss/seq_oss_event.c
@@ -39,8 +39,10 @@ static int set_echo_event(struct seq_oss_devinfo *dp, union evrec *rec, struct s
*/
int
-snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd_seq_event *ev)
+snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q,
+ struct snd_seq_event *ev, snd_use_lock_t **lockp)
{
+ *lockp = NULL;
switch (q->s.code) {
case SEQ_EXTENDED:
return extended_event(dp, q, ev);
@@ -69,7 +71,7 @@ snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd
if (snd_seq_oss_midi_open(dp, q->s.dev, SNDRV_SEQ_OSS_FILE_WRITE))
break;
if (snd_seq_oss_midi_filemode(dp, q->s.dev) & SNDRV_SEQ_OSS_FILE_WRITE)
- return snd_seq_oss_midi_putc(dp, q->s.dev, q->s.parm1, ev);
+ return snd_seq_oss_midi_putc(dp, q->s.dev, q->s.parm1, ev, lockp);
break;
case SEQ_ECHO:
diff --git a/sound/core/seq/oss/seq_oss_event.h b/sound/core/seq/oss/seq_oss_event.h
index b4f723949a17..a4524e51d0e9 100644
--- a/sound/core/seq/oss/seq_oss_event.h
+++ b/sound/core/seq/oss/seq_oss_event.h
@@ -91,7 +91,8 @@ union evrec {
#define ev_is_long(ev) ((ev)->s.code >= 128)
#define ev_length(ev) ((ev)->s.code >= 128 ? LONG_EVENT_SIZE : SHORT_EVENT_SIZE)
-int snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd_seq_event *ev);
+int snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q,
+ struct snd_seq_event *ev, snd_use_lock_t **lockp);
int snd_seq_oss_process_timer_event(struct seq_oss_timer *rec, union evrec *q);
int snd_seq_oss_event_input(struct snd_seq_event *ev, int direct, void *private_data, int atomic, int hop);
diff --git a/sound/core/seq/oss/seq_oss_ioctl.c b/sound/core/seq/oss/seq_oss_ioctl.c
index ccf682689ec9..ce7a69d52b30 100644
--- a/sound/core/seq/oss/seq_oss_ioctl.c
+++ b/sound/core/seq/oss/seq_oss_ioctl.c
@@ -45,14 +45,17 @@ static int snd_seq_oss_oob_user(struct seq_oss_devinfo *dp, void __user *arg)
{
unsigned char ev[8];
struct snd_seq_event tmpev;
+ snd_use_lock_t *lock = NULL;
if (copy_from_user(ev, arg, 8))
return -EFAULT;
memset(&tmpev, 0, sizeof(tmpev));
snd_seq_oss_fill_addr(dp, &tmpev, dp->addr.client, dp->addr.port);
tmpev.time.tick = 0;
- if (! snd_seq_oss_process_event(dp, (union evrec *)ev, &tmpev)) {
+ if (!snd_seq_oss_process_event(dp, (union evrec *)ev, &tmpev, &lock)) {
snd_seq_oss_dispatch(dp, &tmpev, 0, 0);
+ if (lock)
+ snd_use_lock_free(lock);
}
return 0;
}
diff --git a/sound/core/seq/oss/seq_oss_midi.c b/sound/core/seq/oss/seq_oss_midi.c
index f8e247d9e5c9..f1ec7f41dcfe 100644
--- a/sound/core/seq/oss/seq_oss_midi.c
+++ b/sound/core/seq/oss/seq_oss_midi.c
@@ -634,7 +634,8 @@ send_midi_event(struct seq_oss_devinfo *dp, struct snd_seq_event *ev, struct seq
* non-zero : invalid - ignored
*/
int
-snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, struct snd_seq_event *ev)
+snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c,
+ struct snd_seq_event *ev, snd_use_lock_t **lockp)
{
struct seq_oss_midi *mdev;
@@ -643,6 +644,9 @@ snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, stru
return -ENODEV;
if (snd_midi_event_encode_byte(mdev->coder, c, ev)) {
snd_seq_oss_fill_addr(dp, ev, mdev->client, mdev->port);
+ /* the caller must release this later */
+ *lockp = &mdev->use_lock;
+ snd_use_lock_use(*lockp);
snd_use_lock_free(&mdev->use_lock);
return 0;
}
diff --git a/sound/core/seq/oss/seq_oss_midi.h b/sound/core/seq/oss/seq_oss_midi.h
index bcc1683773df..4819d4170bf6 100644
--- a/sound/core/seq/oss/seq_oss_midi.h
+++ b/sound/core/seq/oss/seq_oss_midi.h
@@ -26,7 +26,7 @@ void snd_seq_oss_midi_open_all(struct seq_oss_devinfo *dp, int file_mode);
int snd_seq_oss_midi_close(struct seq_oss_devinfo *dp, int dev);
void snd_seq_oss_midi_reset(struct seq_oss_devinfo *dp, int dev);
int snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c,
- struct snd_seq_event *ev);
+ struct snd_seq_event *ev, snd_use_lock_t **lockp);
int snd_seq_oss_midi_input(struct snd_seq_event *ev, int direct, void *private);
int snd_seq_oss_midi_filemode(struct seq_oss_devinfo *dp, int dev);
int snd_seq_oss_midi_make_info(struct seq_oss_devinfo *dp, int dev, struct midi_info *inf);
diff --git a/sound/core/seq/oss/seq_oss_readq.c b/sound/core/seq/oss/seq_oss_readq.c
index f0db5d3dcba4..b7cdb1c61f26 100644
--- a/sound/core/seq/oss/seq_oss_readq.c
+++ b/sound/core/seq/oss/seq_oss_readq.c
@@ -73,13 +73,17 @@ snd_seq_oss_readq_delete(struct seq_oss_readq *q)
void
snd_seq_oss_readq_clear(struct seq_oss_readq *q)
{
- if (q->qlen) {
- q->qlen = 0;
- q->head = q->tail = 0;
+ scoped_guard(spinlock_irqsave, &q->lock) {
+ if (q->qlen) {
+ q->qlen = 0;
+ q->head = 0;
+ q->tail = 0;
+ }
+ q->input_time = (unsigned long)-1;
}
+
/* if someone sleeping, wake'em up */
wake_up(&q->midi_sleep);
- q->input_time = (unsigned long)-1;
}
/*
@@ -136,28 +140,37 @@ int snd_seq_oss_readq_sysex(struct seq_oss_readq *q, int dev,
/*
* copy an event to input queue:
* return zero if enqueued
+ * caller must hold lock
*/
-int
-snd_seq_oss_readq_put_event(struct seq_oss_readq *q, union evrec *ev)
+static int snd_seq_oss_readq_put_event_locked(struct seq_oss_readq *q,
+ union evrec *ev)
{
- unsigned long flags;
-
- spin_lock_irqsave(&q->lock, flags);
- if (q->qlen >= q->maxlen - 1) {
- spin_unlock_irqrestore(&q->lock, flags);
+ if (q->qlen >= q->maxlen - 1)
return -ENOMEM;
- }
memcpy(&q->q[q->tail], ev, sizeof(*ev));
q->tail = (q->tail + 1) % q->maxlen;
q->qlen++;
- /* wake up sleeper */
- wake_up(&q->midi_sleep);
+ return 0;
+}
- spin_unlock_irqrestore(&q->lock, flags);
+/*
+ * copy an event to input queue:
+ * return zero if enqueued
+ */
+int
+snd_seq_oss_readq_put_event(struct seq_oss_readq *q, union evrec *ev)
+{
+ int rc;
- return 0;
+ scoped_guard(spinlock_irqsave, &q->lock) {
+ rc = snd_seq_oss_readq_put_event_locked(q, ev);
+ if (!rc)
+ wake_up(&q->midi_sleep);
+ }
+
+ return rc;
}
@@ -215,23 +228,31 @@ snd_seq_oss_readq_poll(struct seq_oss_readq *q, struct file *file, poll_table *w
int
snd_seq_oss_readq_put_timestamp(struct seq_oss_readq *q, unsigned long curt, int seq_mode)
{
- if (curt != q->input_time) {
- union evrec rec;
- memset(&rec, 0, sizeof(rec));
- switch (seq_mode) {
- case SNDRV_SEQ_OSS_MODE_SYNTH:
- rec.echo = (curt << 8) | SEQ_WAIT;
- snd_seq_oss_readq_put_event(q, &rec);
- break;
- case SNDRV_SEQ_OSS_MODE_MUSIC:
- rec.t.code = EV_TIMING;
- rec.t.cmd = TMR_WAIT_ABS;
- rec.t.time = curt;
- snd_seq_oss_readq_put_event(q, &rec);
- break;
+ int queued = 0;
+
+ scoped_guard(spinlock_irqsave, &q->lock) {
+ if (curt != q->input_time) {
+ union evrec rec;
+
+ memset(&rec, 0, sizeof(rec));
+ switch (seq_mode) {
+ case SNDRV_SEQ_OSS_MODE_SYNTH:
+ rec.echo = (curt << 8) | SEQ_WAIT;
+ queued = !snd_seq_oss_readq_put_event_locked(q, &rec);
+ break;
+ case SNDRV_SEQ_OSS_MODE_MUSIC:
+ rec.t.code = EV_TIMING;
+ rec.t.cmd = TMR_WAIT_ABS;
+ rec.t.time = curt;
+ queued = !snd_seq_oss_readq_put_event_locked(q, &rec);
+ break;
+ }
+ q->input_time = curt;
}
- q->input_time = curt;
}
+ if (queued)
+ wake_up(&q->midi_sleep);
+
return 0;
}
diff --git a/sound/core/seq/oss/seq_oss_rw.c b/sound/core/seq/oss/seq_oss_rw.c
index 45bb45859770..b7147ac78ee8 100644
--- a/sound/core/seq/oss/seq_oss_rw.c
+++ b/sound/core/seq/oss/seq_oss_rw.c
@@ -154,6 +154,7 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt)
{
int rc = 0;
struct snd_seq_event event;
+ snd_use_lock_t *lock = NULL;
/* if this is a timing event, process the current time */
if (snd_seq_oss_process_timer_event(dp->timer, rec))
@@ -165,7 +166,7 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt)
event.type = SNDRV_SEQ_EVENT_NOTEOFF;
snd_seq_oss_fill_addr(dp, &event, dp->addr.client, dp->addr.port);
- if (snd_seq_oss_process_event(dp, rec, &event))
+ if (snd_seq_oss_process_event(dp, rec, &event, &lock))
return 0; /* invalid event - no need to insert queue */
event.time.tick = snd_seq_oss_timer_cur_tick(dp->timer);
@@ -174,6 +175,8 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt)
else
rc = snd_seq_kernel_client_enqueue(dp->cseq, &event, opt,
!is_nonblock_mode(dp->file_mode));
+ if (lock)
+ snd_use_lock_free(lock);
return rc;
}
diff --git a/sound/core/seq/seq_clientmgr.c b/sound/core/seq/seq_clientmgr.c
index 35f099e02ca7..86316cf027ae 100644
--- a/sound/core/seq/seq_clientmgr.c
+++ b/sound/core/seq/seq_clientmgr.c
@@ -463,7 +463,7 @@ static ssize_t snd_seq_read(struct file *file, char __user *buf, size_t count,
cell = NULL;
err = 0;
- snd_seq_fifo_lock(fifo);
+ guard(snd_seq_fifo)(fifo);
if (IS_ENABLED(CONFIG_SND_SEQ_UMP) && client->midi_version > 0)
aligned_size = sizeof(struct snd_seq_ump_event);
@@ -522,7 +522,6 @@ static ssize_t snd_seq_read(struct file *file, char __user *buf, size_t count,
if (err == -EAGAIN && result > 0)
err = 0;
}
- snd_seq_fifo_unlock(fifo);
return (err < 0) ? err : result;
}
diff --git a/sound/core/seq/seq_fifo.c b/sound/core/seq/seq_fifo.c
index 3a10b081f129..c3dfe04c3883 100644
--- a/sound/core/seq/seq_fifo.c
+++ b/sound/core/seq/seq_fifo.c
@@ -101,37 +101,48 @@ int snd_seq_fifo_event_in(struct snd_seq_fifo *f,
struct snd_seq_event *event)
{
struct snd_seq_event_cell *cell;
+ struct snd_seq_pool *pool;
+ bool linked;
int err;
if (snd_BUG_ON(!f))
return -EINVAL;
- snd_use_lock_use(&f->use_lock);
- err = snd_seq_event_dup(f->pool, event, &cell, 1, NULL, NULL); /* always non-blocking */
+ guard(snd_seq_fifo)(f);
+retry:
+ pool = READ_ONCE(f->pool);
+ err = snd_seq_event_dup(pool, event, &cell, 1, NULL, NULL); /* always non-blocking */
if (err < 0) {
if ((err == -ENOMEM) || (err == -EAGAIN))
atomic_inc(&f->overflow);
- snd_use_lock_free(&f->use_lock);
return err;
}
/* append new cells to fifo */
+ linked = false;
scoped_guard(spinlock_irqsave, &f->lock) {
- if (f->tail != NULL)
- f->tail->next = cell;
- f->tail = cell;
- if (f->head == NULL)
- f->head = cell;
- cell->next = NULL;
- f->cells++;
+ if (cell->pool == f->pool) {
+ if (f->tail)
+ f->tail->next = cell;
+ f->tail = cell;
+ if (!f->head)
+ f->head = cell;
+ cell->next = NULL;
+ f->cells++;
+ linked = true;
+ }
+ }
+
+ if (!linked) {
+ /* Retry against the replacement pool after resize publishes it. */
+ snd_seq_cell_free(cell);
+ goto retry;
}
/* wakeup client */
if (waitqueue_active(&f->input_sleep))
wake_up(&f->input_sleep);
- snd_use_lock_free(&f->use_lock);
-
return 0; /* success */
}
@@ -197,13 +208,21 @@ int snd_seq_fifo_cell_out(struct snd_seq_fifo *f,
void snd_seq_fifo_cell_putback(struct snd_seq_fifo *f,
struct snd_seq_event_cell *cell)
{
+ bool linked = false;
+
if (cell) {
- guard(spinlock_irqsave)(&f->lock);
- cell->next = f->head;
- f->head = cell;
- if (!f->tail)
- f->tail = cell;
- f->cells++;
+ scoped_guard(spinlock_irqsave, &f->lock) {
+ if (cell->pool == f->pool) {
+ cell->next = f->head;
+ f->head = cell;
+ if (!f->tail)
+ f->tail = cell;
+ f->cells++;
+ linked = true;
+ }
+ }
+ if (!linked)
+ snd_seq_cell_free(cell);
}
}
@@ -239,7 +258,7 @@ int snd_seq_fifo_resize(struct snd_seq_fifo *f, int poolsize)
oldpool = f->pool;
oldhead = f->head;
/* exchange pools */
- f->pool = newpool;
+ WRITE_ONCE(f->pool, newpool);
f->head = NULL;
f->tail = NULL;
f->cells = 0;
@@ -263,14 +282,10 @@ int snd_seq_fifo_resize(struct snd_seq_fifo *f, int poolsize)
/* get the number of unused cells safely */
int snd_seq_fifo_unused_cells(struct snd_seq_fifo *f)
{
- int cells;
-
if (!f)
return 0;
- snd_use_lock_use(&f->use_lock);
- scoped_guard(spinlock_irqsave, &f->lock)
- cells = snd_seq_unused_cells(f->pool);
- snd_use_lock_free(&f->use_lock);
- return cells;
+ guard(snd_seq_fifo)(f);
+ guard(spinlock_irqsave)(&f->lock);
+ return snd_seq_unused_cells(f->pool);
}
diff --git a/sound/core/seq/seq_fifo.h b/sound/core/seq/seq_fifo.h
index b56a7b897c9c..4c9c49127746 100644
--- a/sound/core/seq/seq_fifo.h
+++ b/sound/core/seq/seq_fifo.h
@@ -37,6 +37,7 @@ int snd_seq_fifo_event_in(struct snd_seq_fifo *f, struct snd_seq_event *event);
/* lock fifo from release */
#define snd_seq_fifo_lock(fifo) snd_use_lock_use(&(fifo)->use_lock)
#define snd_seq_fifo_unlock(fifo) snd_use_lock_free(&(fifo)->use_lock)
+DEFINE_GUARD(snd_seq_fifo, struct snd_seq_fifo *, snd_seq_fifo_lock(_T), snd_seq_fifo_unlock(_T))
/* get a cell from fifo - fifo should be locked */
int snd_seq_fifo_cell_out(struct snd_seq_fifo *f, struct snd_seq_event_cell **cellp, int nonblock);
diff --git a/sound/drivers/mtpav.c b/sound/drivers/mtpav.c
index 946184a2a758..dc67bb3d4639 100644
--- a/sound/drivers/mtpav.c
+++ b/sound/drivers/mtpav.c
@@ -657,12 +657,8 @@ static int snd_mtpav_get_RAWMIDI(struct mtpav *mcard)
static void snd_mtpav_free(struct snd_card *card)
{
struct mtpav *crd = card->private_data;
- unsigned long flags;
- spin_lock_irqsave(&crd->spinlock, flags);
- if (crd->istimer > 0)
- snd_mtpav_remove_output_timer(crd);
- spin_unlock_irqrestore(&crd->spinlock, flags);
+ timer_shutdown_sync(&crd->timer);
}
/*
diff --git a/sound/soc/amd/acp/acp-mach-common.c b/sound/soc/amd/acp/acp-mach-common.c
index 6c0a92d76b54..75143f24a5ce 100644
--- a/sound/soc/amd/acp/acp-mach-common.c
+++ b/sound/soc/amd/acp/acp-mach-common.c
@@ -133,8 +133,13 @@ static int acp_card_rt5682_init(struct snd_soc_pcm_runtime *rtd)
if (drvdata->hs_codec_id != RT5682)
return -EINVAL;
- drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk");
- drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk");
+ drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk");
+ if (IS_ERR(drvdata->wclk))
+ return PTR_ERR(drvdata->wclk);
+
+ drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk");
+ if (IS_ERR(drvdata->bclk))
+ return PTR_ERR(drvdata->bclk);
ret = snd_soc_dapm_new_controls(&card->dapm, rt5682_widgets,
ARRAY_SIZE(rt5682_widgets));
@@ -375,8 +380,13 @@ static int acp_card_rt5682s_init(struct snd_soc_pcm_runtime *rtd)
return -EINVAL;
if (!drvdata->soc_mclk) {
- drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk");
- drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk");
+ drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk");
+ if (IS_ERR(drvdata->wclk))
+ return PTR_ERR(drvdata->wclk);
+
+ drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk");
+ if (IS_ERR(drvdata->bclk))
+ return PTR_ERR(drvdata->bclk);
}
ret = snd_soc_dapm_new_controls(&card->dapm, rt5682s_widgets,
diff --git a/sound/soc/intel/avs/avs.h b/sound/soc/intel/avs/avs.h
index eca6ec0428bb..bd63676aa7aa 100644
--- a/sound/soc/intel/avs/avs.h
+++ b/sound/soc/intel/avs/avs.h
@@ -339,7 +339,7 @@ struct avs_soc_component {
extern const struct snd_soc_dai_ops avs_dai_fe_ops;
int avs_soc_component_register(struct device *dev, const char *name,
- const struct snd_soc_component_driver *drv,
+ struct snd_soc_component_driver *drv,
struct snd_soc_dai_driver *cpu_dais, int num_cpu_dais);
int avs_dmic_platform_register(struct avs_dev *adev, const char *name);
int avs_i2s_platform_register(struct avs_dev *adev, const char *name, unsigned long port_mask,
diff --git a/sound/soc/intel/avs/path.c b/sound/soc/intel/avs/path.c
index f31d5e2caa7b..05892b0176ac 100644
--- a/sound/soc/intel/avs/path.c
+++ b/sound/soc/intel/avs/path.c
@@ -548,15 +548,10 @@ static int avs_path_module_type_create(struct avs_dev *adev, struct avs_path_mod
static int avs_path_module_send_init_configs(struct avs_dev *adev, struct avs_path_module *mod)
{
- struct avs_soc_component *acomp;
-
- acomp = to_avs_soc_component(mod->template->owner->owner->owner->owner->comp);
-
- u32 num_ids = mod->template->num_config_ids;
- u32 *ids = mod->template->config_ids;
+ struct avs_tplg_module *template = mod->template;
- for (int i = 0; i < num_ids; i++) {
- struct avs_tplg_init_config *config = &acomp->tplg->init_configs[ids[i]];
+ for (int i = 0; i < template->num_init_configs; i++) {
+ struct avs_tplg_init_config *config = template->init_configs[i];
size_t len = config->length;
void *data = config->data;
u32 param = config->param;
diff --git a/sound/soc/intel/avs/pcm.c b/sound/soc/intel/avs/pcm.c
index b63a537c3fae..81fe4f0218c1 100644
--- a/sound/soc/intel/avs/pcm.c
+++ b/sound/soc/intel/avs/pcm.c
@@ -9,6 +9,7 @@
#include <linux/cleanup.h>
#include <linux/debugfs.h>
#include <linux/device.h>
+#include <linux/module.h>
#include <sound/hda_register.h>
#include <sound/hdaudio_ext.h>
#include <sound/pcm_params.h>
@@ -21,6 +22,10 @@
#include "topology.h"
#include "../../codecs/hda.h"
+static bool obsolete_card_names;
+module_param_named(obsolete_card_names, obsolete_card_names, bool, 0444);
+MODULE_PARM_DESC(obsolete_card_names, "Use obsolete PCM device numbering 0=no, 1=yes");
+
struct avs_dma_data {
struct avs_tplg_path_template *template;
struct avs_path *path;
@@ -1274,7 +1279,7 @@ static int avs_component_construct(struct snd_soc_component *component,
return 0;
}
-static const struct snd_soc_component_driver avs_component_driver = {
+static struct snd_soc_component_driver avs_component_driver = {
.name = "avs-pcm",
.probe = avs_component_probe,
.remove = avs_component_remove,
@@ -1289,7 +1294,7 @@ static const struct snd_soc_component_driver avs_component_driver = {
};
int avs_soc_component_register(struct device *dev, const char *name,
- const struct snd_soc_component_driver *drv,
+ struct snd_soc_component_driver *drv,
struct snd_soc_dai_driver *cpu_dais, int num_cpu_dais)
{
struct avs_soc_component *acomp;
@@ -1299,14 +1304,18 @@ int avs_soc_component_register(struct device *dev, const char *name,
if (!acomp)
return -ENOMEM;
+ acomp->base.name = devm_kstrdup(dev, name, GFP_KERNEL);
+ if (!acomp->base.name)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&acomp->node);
+
+ drv->use_dai_pcm_id = !obsolete_card_names;
+
ret = snd_soc_component_initialize(&acomp->base, drv, dev);
if (ret < 0)
return ret;
- /* force name change after ASoC is done with its init */
- acomp->base.name = name;
- INIT_LIST_HEAD(&acomp->node);
-
return snd_soc_add_component(&acomp->base, cpu_dais, num_cpu_dais);
}
@@ -1630,7 +1639,7 @@ static int avs_component_hda_open(struct snd_soc_component *component,
return 0;
}
-static const struct snd_soc_component_driver avs_hda_component_driver = {
+static struct snd_soc_component_driver avs_hda_component_driver = {
.name = "avs-hda-pcm",
.probe = avs_component_hda_probe,
.remove = avs_component_hda_remove,
diff --git a/sound/soc/intel/avs/probes.c b/sound/soc/intel/avs/probes.c
index f0b010956303..a42736b9aa55 100644
--- a/sound/soc/intel/avs/probes.c
+++ b/sound/soc/intel/avs/probes.c
@@ -284,7 +284,7 @@ static struct snd_soc_dai_driver probe_cpu_dais[] = {
},
};
-static const struct snd_soc_component_driver avs_probe_component_driver = {
+static struct snd_soc_component_driver avs_probe_component_driver = {
.name = "avs-probe-compr",
.compress_ops = &avs_probe_compress_ops,
.module_get_upon_open = 1, /* increment refcount when a stream is opened */
diff --git a/sound/soc/intel/avs/topology.c b/sound/soc/intel/avs/topology.c
index 9d49e248cfb2..88618351b75e 100644
--- a/sound/soc/intel/avs/topology.c
+++ b/sound/soc/intel/avs/topology.c
@@ -350,6 +350,7 @@ AVS_DEFINE_PTR_PARSER(modcfg_base, struct avs_tplg_modcfg_base, modcfgs_base);
AVS_DEFINE_PTR_PARSER(modcfg_ext, struct avs_tplg_modcfg_ext, modcfgs_ext);
AVS_DEFINE_PTR_PARSER(pplcfg, struct avs_tplg_pplcfg, pplcfgs);
AVS_DEFINE_PTR_PARSER(binding, struct avs_tplg_binding, bindings);
+AVS_DEFINE_PTR_PARSER(init_config, struct avs_tplg_init_config, init_configs);
static int
parse_audio_format_bitfield(struct snd_soc_component *comp, void *elem, void *object, u32 offset)
@@ -1121,7 +1122,7 @@ static const struct avs_tplg_token_parser module_parsers[] = {
{
.token = AVS_TKN_MOD_INIT_CONFIG_NUM_IDS_U32,
.type = SND_SOC_TPLG_TUPLE_TYPE_WORD,
- .offset = offsetof(struct avs_tplg_module, num_config_ids),
+ .offset = offsetof(struct avs_tplg_module, num_init_configs),
.parse = avs_parse_byte_token,
},
};
@@ -1131,10 +1132,32 @@ static const struct avs_tplg_token_parser init_config_parsers[] = {
.token = AVS_TKN_MOD_INIT_CONFIG_ID_U32,
.type = SND_SOC_TPLG_TUPLE_TYPE_WORD,
.offset = 0,
- .parse = avs_parse_word_token,
+ .parse = avs_parse_init_config_ptr,
},
};
+static int avs_tplg_module_init_configs(struct snd_soc_component *comp,
+ struct avs_tplg_module *module,
+ struct snd_soc_tplg_vendor_array *tuples, u32 block_size)
+{
+ struct avs_tplg_init_config **cfgs;
+ int ret;
+
+ if (!module->num_init_configs)
+ return -EINVAL;
+
+ cfgs = devm_kcalloc(comp->card->dev, module->num_init_configs, sizeof(*cfgs), GFP_KERNEL);
+ if (!cfgs)
+ return -ENOMEM;
+
+ ret = parse_dictionary_entries(comp, tuples, block_size, cfgs, module->num_init_configs,
+ sizeof(*cfgs), AVS_TKN_MOD_INIT_CONFIG_ID_U32,
+ init_config_parsers, ARRAY_SIZE(init_config_parsers));
+ if (!ret)
+ module->init_configs = cfgs;
+ return ret;
+}
+
static struct avs_tplg_module *
avs_tplg_module_create(struct snd_soc_component *comp, struct avs_tplg_pipeline *owner,
struct snd_soc_tplg_vendor_array *tuples, u32 block_size)
@@ -1161,27 +1184,11 @@ avs_tplg_module_create(struct snd_soc_component *comp, struct avs_tplg_pipeline
block_size -= esize;
/* Parse trailing config ids if any. */
if (block_size) {
- u32 num_config_ids = module->num_config_ids;
- u32 *config_ids;
-
- if (!num_config_ids)
- return ERR_PTR(-EINVAL);
-
- config_ids = devm_kcalloc(comp->card->dev, num_config_ids, sizeof(*config_ids),
- GFP_KERNEL);
- if (!config_ids)
- return ERR_PTR(-ENOMEM);
-
tuples = avs_tplg_vendor_array_at(tuples, esize);
- ret = parse_dictionary_entries(comp, tuples, block_size,
- config_ids, num_config_ids, sizeof(*config_ids),
- AVS_TKN_MOD_INIT_CONFIG_ID_U32,
- init_config_parsers,
- ARRAY_SIZE(init_config_parsers));
+
+ ret = avs_tplg_module_init_configs(comp, module, tuples, block_size);
if (ret)
return ERR_PTR(ret);
-
- module->config_ids = config_ids;
}
module->owner = owner;
diff --git a/sound/soc/intel/avs/topology.h b/sound/soc/intel/avs/topology.h
index 409abdb23977..5d68418da8d8 100644
--- a/sound/soc/intel/avs/topology.h
+++ b/sound/soc/intel/avs/topology.h
@@ -194,8 +194,8 @@ struct avs_tplg_module {
u8 domain;
struct avs_tplg_modcfg_ext *cfg_ext;
u32 ctl_id;
- u32 num_config_ids;
- u32 *config_ids;
+ u32 num_init_configs;
+ struct avs_tplg_init_config **init_configs;
struct avs_tplg_pipeline *owner;
/* Pipeline modules management. */
diff --git a/sound/soc/qcom/qdsp6/q6asm.c b/sound/soc/qcom/qdsp6/q6asm.c
index 67e9ca18883c..60c81edd945d 100644
--- a/sound/soc/qcom/qdsp6/q6asm.c
+++ b/sound/soc/qcom/qdsp6/q6asm.c
@@ -654,6 +654,11 @@ static int32_t q6asm_stream_callback(struct apr_device *adev,
goto done;
}
break;
+ case ASM_DATA_CMD_EOS:
+ case ASM_DATA_CMD_READ_V2:
+ case ASM_DATA_CMD_WRITE_V2:
+ /* response as result of close stream */
+ goto done;
default:
dev_err(ac->dev, "command[0x%x] not expecting rsp\n",
result->opcode);
diff --git a/sound/soc/sof/ipc3-pcm.c b/sound/soc/sof/ipc3-pcm.c
index 1c1b8f595367..90ef5d99f626 100644
--- a/sound/soc/sof/ipc3-pcm.c
+++ b/sound/soc/sof/ipc3-pcm.c
@@ -117,22 +117,23 @@ static int sof_ipc3_pcm_hw_params(struct snd_soc_component *component,
if (platform_params->cont_update_posn)
pcm.params.cont_update_posn = 1;
- dev_dbg(component->dev, "stream_tag %d", pcm.params.stream_tag);
+ spcm_dbg(spcm, substream->stream, "stream_tag %d\n",
+ pcm.params.stream_tag);
/* send hw_params IPC to the DSP */
ret = sof_ipc_tx_message(sdev->ipc, &pcm, sizeof(pcm),
&ipc_params_reply, sizeof(ipc_params_reply));
if (ret < 0) {
- dev_err(component->dev, "HW params ipc failed for stream %d\n",
- pcm.params.stream_tag);
+ spcm_err(spcm, substream->stream,
+ "STREAM_PCM_PARAMS ipc failed for stream_tag %d\n",
+ pcm.params.stream_tag);
return ret;
}
ret = snd_sof_set_stream_data_offset(sdev, &spcm->stream[substream->stream],
ipc_params_reply.posn_offset);
if (ret < 0) {
- dev_err(component->dev, "%s: invalid stream data offset for PCM %d\n",
- __func__, spcm->pcm.pcm_id);
+ spcm_err(spcm, substream->stream, "invalid stream data offset\n");
return ret;
}
@@ -171,7 +172,7 @@ static int sof_ipc3_pcm_trigger(struct snd_soc_component *component,
stream.hdr.cmd |= SOF_IPC_STREAM_TRIG_STOP;
break;
default:
- dev_err(component->dev, "Unhandled trigger cmd %d\n", cmd);
+ spcm_err(spcm, substream->stream, "Unhandled trigger cmd %d\n", cmd);
return -EINVAL;
}
diff --git a/sound/soc/sof/ipc3-topology.c b/sound/soc/sof/ipc3-topology.c
index 19bbae725d83..f449362a2905 100644
--- a/sound/soc/sof/ipc3-topology.c
+++ b/sound/soc/sof/ipc3-topology.c
@@ -2386,28 +2386,16 @@ static int sof_ipc3_set_up_all_pipelines(struct snd_sof_dev *sdev, bool verify)
static int sof_tear_down_left_over_pipelines(struct snd_sof_dev *sdev)
{
struct snd_sof_widget *swidget;
- struct snd_sof_pcm *spcm;
- int dir, ret;
+ int ret;
/*
* free all PCMs and their associated DAPM widgets if their connected DAPM widget
* list is not NULL. This should only be true for paused streams at this point.
* This is equivalent to the handling of FE DAI suspend trigger for running streams.
*/
- list_for_each_entry(spcm, &sdev->pcm_list, list) {
- for_each_pcm_streams(dir) {
- struct snd_pcm_substream *substream = spcm->stream[dir].substream;
-
- if (!substream || !substream->runtime || spcm->stream[dir].suspend_ignored)
- continue;
-
- if (spcm->stream[dir].list) {
- ret = sof_pcm_stream_free(sdev, substream, spcm, dir, true);
- if (ret < 0)
- return ret;
- }
- }
- }
+ ret = sof_pcm_free_all_streams(sdev);
+ if (ret)
+ return ret;
/*
* free any left over DAI widgets. This is equivalent to the handling of suspend trigger
diff --git a/sound/soc/sof/ipc4-pcm.c b/sound/soc/sof/ipc4-pcm.c
index 5e877dc921b1..79b4aed4d26c 100644
--- a/sound/soc/sof/ipc4-pcm.c
+++ b/sound/soc/sof/ipc4-pcm.c
@@ -324,7 +324,7 @@ static int sof_ipc4_chain_dma_trigger(struct snd_sof_dev *sdev,
set_fifo_size = false;
break;
default:
- dev_err(sdev->dev, "Unexpected state %d", state);
+ spcm_err(spcm, direction, "Unexpected pipeline state %d\n", state);
return -EINVAL;
}
@@ -344,8 +344,8 @@ static int sof_ipc4_chain_dma_trigger(struct snd_sof_dev *sdev,
struct sof_ipc4_pipeline *pipeline = pipe_widget->private;
if (!pipeline->use_chain_dma) {
- dev_err(sdev->dev,
- "All pipelines in chained DMA stream should have use_chain_dma attribute set.");
+ spcm_err(spcm, direction,
+ "All pipelines in chained DMA path should have use_chain_dma attribute set.");
return -EINVAL;
}
@@ -400,12 +400,12 @@ static int sof_ipc4_trigger_pipelines(struct snd_soc_component *component,
int ret;
int i;
- dev_dbg(sdev->dev, "trigger cmd: %d state: %d\n", cmd, state);
-
spcm = snd_sof_find_spcm_dai(component, rtd);
if (!spcm)
return -EINVAL;
+ spcm_dbg(spcm, substream->stream, "cmd: %d, state: %d\n", cmd, state);
+
pipeline_list = &spcm->stream[substream->stream].pipeline_list;
/* nothing to trigger if the list is empty */
@@ -500,8 +500,20 @@ static int sof_ipc4_trigger_pipelines(struct snd_soc_component *component,
*/
ret = sof_ipc4_set_multi_pipeline_state(sdev, SOF_IPC4_PIPE_PAUSED, trigger_list);
if (ret < 0) {
- dev_err(sdev->dev, "failed to pause all pipelines\n");
- goto free;
+ spcm_err(spcm, substream->stream, "failed to pause all pipelines\n");
+ /*
+ * workaround: if the firmware is crashed or the IPC timed out
+ * while setting the pipeline state we must ignore the error
+ * code and proceed to set adjust the local pipeline states.
+ *
+ * If the firmware is crashed we will not send IPC messages
+ * and we are going to see errors printed, but the state of the
+ * widgets will be correct for the next boot.
+ */
+ if (sdev->fw_state != SOF_FW_CRASHED && ret != -ETIMEDOUT)
+ goto free;
+
+ ret = 0;
}
/* update PAUSED state for all pipelines just triggered */
@@ -529,16 +541,19 @@ static int sof_ipc4_trigger_pipelines(struct snd_soc_component *component,
/* else set the RUNNING/RESET state in the DSP */
ret = sof_ipc4_set_multi_pipeline_state(sdev, state, trigger_list);
if (ret < 0) {
- dev_err(sdev->dev, "failed to set final state %d for all pipelines\n", state);
+ spcm_err(spcm, substream->stream,
+ "failed to set final state %d for all pipelines\n",
+ state);
/*
- * workaround: if the firmware is crashed while setting the
- * pipelines to reset state we must ignore the error code and
- * reset it to 0.
- * Since the firmware is crashed we will not send IPC messages
+ * workaround: if the firmware is crashed or the IPC timed out
+ * while setting the pipeline state we must ignore the error
+ * code and proceed to set adjust the local pipeline states.
+ *
+ * If the firmware is crashed we will not send IPC messages
* and we are going to see errors printed, but the state of the
* widgets will be correct for the next boot.
*/
- if (sdev->fw_state != SOF_FW_CRASHED || state != SOF_IPC4_PIPE_RESET)
+ if (sdev->fw_state != SOF_FW_CRASHED && ret != -ETIMEDOUT)
goto free;
ret = 0;
diff --git a/sound/soc/sof/ipc4-topology.c b/sound/soc/sof/ipc4-topology.c
index fabd364d3316..6af0cc154f3d 100644
--- a/sound/soc/sof/ipc4-topology.c
+++ b/sound/soc/sof/ipc4-topology.c
@@ -3345,9 +3345,6 @@ static int sof_ipc4_dai_get_param(struct snd_sof_dev *sdev, struct snd_sof_dai *
static int sof_ipc4_tear_down_all_pipelines(struct snd_sof_dev *sdev, bool verify)
{
- struct snd_sof_pcm *spcm;
- int dir, ret;
-
/*
* This function is called during system suspend, we need to make sure
* that all streams have been freed up.
@@ -3359,21 +3356,8 @@ static int sof_ipc4_tear_down_all_pipelines(struct snd_sof_dev *sdev, bool verif
*
* This will also make sure that paused streams handled correctly.
*/
- list_for_each_entry(spcm, &sdev->pcm_list, list) {
- for_each_pcm_streams(dir) {
- struct snd_pcm_substream *substream = spcm->stream[dir].substream;
-
- if (!substream || !substream->runtime || spcm->stream[dir].suspend_ignored)
- continue;
- if (spcm->stream[dir].list) {
- ret = sof_pcm_stream_free(sdev, substream, spcm, dir, true);
- if (ret < 0)
- return ret;
- }
- }
- }
- return 0;
+ return sof_pcm_free_all_streams(sdev);
}
static int sof_ipc4_link_setup(struct snd_sof_dev *sdev, struct snd_soc_dai_link *link)
diff --git a/sound/soc/sof/pcm.c b/sound/soc/sof/pcm.c
index c5c6353f18ce..d584a72e6f52 100644
--- a/sound/soc/sof/pcm.c
+++ b/sound/soc/sof/pcm.c
@@ -99,8 +99,8 @@ sof_pcm_setup_connected_widgets(struct snd_sof_dev *sdev, struct snd_soc_pcm_run
ret = snd_soc_dapm_dai_get_connected_widgets(dai, dir, &list,
dpcm_end_walk_at_be);
if (ret < 0) {
- dev_err(sdev->dev, "error: dai %s has no valid %s path\n", dai->name,
- snd_pcm_direction_name(dir));
+ spcm_err(spcm, dir, "dai %s has no valid %s path\n",
+ dai->name, snd_pcm_direction_name(dir));
return ret;
}
@@ -108,8 +108,7 @@ sof_pcm_setup_connected_widgets(struct snd_sof_dev *sdev, struct snd_soc_pcm_run
ret = sof_widget_list_setup(sdev, spcm, params, platform_params, dir);
if (ret < 0) {
- dev_err(sdev->dev, "error: failed widget list set up for pcm %d dir %d\n",
- spcm->pcm.pcm_id, dir);
+ spcm_err(spcm, dir, "Widget list set up failed\n");
spcm->stream[dir].list = NULL;
snd_soc_dapm_dai_free_widgets(&list);
return ret;
@@ -139,6 +138,8 @@ static int sof_pcm_hw_params(struct snd_soc_component *component,
if (!spcm)
return -EINVAL;
+ spcm_dbg(spcm, substream->stream, "Entry: hw_params\n");
+
/*
* Handle repeated calls to hw_params() without free_pcm() in
* between. At least ALSA OSS emulation depends on this.
@@ -151,12 +152,9 @@ static int sof_pcm_hw_params(struct snd_soc_component *component,
spcm->prepared[substream->stream] = false;
}
- dev_dbg(component->dev, "pcm: hw params stream %d dir %d\n",
- spcm->pcm.pcm_id, substream->stream);
-
ret = snd_sof_pcm_platform_hw_params(sdev, substream, params, &platform_params);
if (ret < 0) {
- dev_err(component->dev, "platform hw params failed\n");
+ spcm_err(spcm, substream->stream, "platform hw params failed\n");
return ret;
}
@@ -191,6 +189,84 @@ static int sof_pcm_hw_params(struct snd_soc_component *component,
return 0;
}
+static int sof_pcm_stream_free(struct snd_sof_dev *sdev,
+ struct snd_pcm_substream *substream,
+ struct snd_sof_pcm *spcm, int dir,
+ bool free_widget_list)
+{
+ const struct sof_ipc_pcm_ops *pcm_ops = sof_ipc_get_ops(sdev, pcm);
+ int ret;
+ int err = 0;
+
+ if (spcm->prepared[substream->stream]) {
+ /* stop DMA first if needed */
+ if (pcm_ops && pcm_ops->platform_stop_during_hw_free)
+ snd_sof_pcm_platform_trigger(sdev, substream,
+ SNDRV_PCM_TRIGGER_STOP);
+
+ /* free PCM in the DSP */
+ if (pcm_ops && pcm_ops->hw_free) {
+ ret = pcm_ops->hw_free(sdev->component, substream);
+ if (ret < 0) {
+ spcm_err(spcm, substream->stream,
+ "pcm_ops->hw_free failed %d\n", ret);
+ err = ret;
+ }
+ }
+
+ spcm->prepared[substream->stream] = false;
+ spcm->pending_stop[substream->stream] = false;
+ }
+
+ /* reset the DMA */
+ ret = snd_sof_pcm_platform_hw_free(sdev, substream);
+ if (ret < 0) {
+ spcm_err(spcm, substream->stream,
+ "platform hw free failed %d\n", ret);
+ if (!err)
+ err = ret;
+ }
+
+ /* free widget list */
+ if (free_widget_list) {
+ ret = sof_widget_list_free(sdev, spcm, dir);
+ if (ret < 0) {
+ spcm_err(spcm, substream->stream,
+ "sof_widget_list_free failed %d\n", ret);
+ if (!err)
+ err = ret;
+ }
+ }
+
+ return err;
+}
+
+int sof_pcm_free_all_streams(struct snd_sof_dev *sdev)
+{
+ struct snd_pcm_substream *substream;
+ struct snd_sof_pcm *spcm;
+ int dir, ret;
+
+ list_for_each_entry(spcm, &sdev->pcm_list, list) {
+ for_each_pcm_streams(dir) {
+ substream = spcm->stream[dir].substream;
+
+ if (!substream || !substream->runtime ||
+ spcm->stream[dir].suspend_ignored)
+ continue;
+
+ if (spcm->stream[dir].list) {
+ ret = sof_pcm_stream_free(sdev, substream, spcm,
+ dir, true);
+ if (ret < 0)
+ return ret;
+ }
+ }
+ }
+
+ return 0;
+}
+
static int sof_pcm_hw_free(struct snd_soc_component *component,
struct snd_pcm_substream *substream)
{
@@ -207,8 +283,7 @@ static int sof_pcm_hw_free(struct snd_soc_component *component,
if (!spcm)
return -EINVAL;
- dev_dbg(component->dev, "pcm: free stream %d dir %d\n",
- spcm->pcm.pcm_id, substream->stream);
+ spcm_dbg(spcm, substream->stream, "Entry: hw_free\n");
ret = sof_pcm_stream_free(sdev, substream, spcm, substream->stream, true);
@@ -233,6 +308,8 @@ static int sof_pcm_prepare(struct snd_soc_component *component,
if (!spcm)
return -EINVAL;
+ spcm_dbg(spcm, substream->stream, "Entry: prepare\n");
+
if (spcm->prepared[substream->stream]) {
if (!spcm->pending_stop[substream->stream])
return 0;
@@ -246,15 +323,12 @@ static int sof_pcm_prepare(struct snd_soc_component *component,
return ret;
}
- dev_dbg(component->dev, "pcm: prepare stream %d dir %d\n",
- spcm->pcm.pcm_id, substream->stream);
-
/* set hw_params */
ret = sof_pcm_hw_params(component,
substream, &spcm->params[substream->stream]);
if (ret < 0) {
- dev_err(component->dev,
- "error: set pcm hw_params after resume\n");
+ spcm_err(spcm, substream->stream,
+ "failed to set hw_params after resume\n");
return ret;
}
@@ -284,8 +358,7 @@ static int sof_pcm_trigger(struct snd_soc_component *component,
if (!spcm)
return -EINVAL;
- dev_dbg(component->dev, "pcm: trigger stream %d dir %d cmd %d\n",
- spcm->pcm.pcm_id, substream->stream, cmd);
+ spcm_dbg(spcm, substream->stream, "Entry: trigger (cmd: %d)\n", cmd);
spcm->pending_stop[substream->stream] = false;
@@ -334,7 +407,7 @@ static int sof_pcm_trigger(struct snd_soc_component *component,
reset_hw_params = true;
break;
default:
- dev_err(component->dev, "Unhandled trigger cmd %d\n", cmd);
+ spcm_err(spcm, substream->stream, "Unhandled trigger cmd %d\n", cmd);
return -EINVAL;
}
@@ -436,9 +509,7 @@ static int sof_pcm_open(struct snd_soc_component *component,
if (!spcm)
return -EINVAL;
- dev_dbg(component->dev, "pcm: open stream %d dir %d\n",
- spcm->pcm.pcm_id, substream->stream);
-
+ spcm_dbg(spcm, substream->stream, "Entry: open\n");
caps = &spcm->pcm.caps[substream->stream];
@@ -458,15 +529,6 @@ static int sof_pcm_open(struct snd_soc_component *component,
*/
runtime->hw.buffer_bytes_max = le32_to_cpu(caps->buffer_size_max);
- dev_dbg(component->dev, "period min %zd max %zd bytes\n",
- runtime->hw.period_bytes_min,
- runtime->hw.period_bytes_max);
- dev_dbg(component->dev, "period count %d max %d\n",
- runtime->hw.periods_min,
- runtime->hw.periods_max);
- dev_dbg(component->dev, "buffer max %zd bytes\n",
- runtime->hw.buffer_bytes_max);
-
/* set wait time - TODO: come from topology */
substream->wait_time = 500;
@@ -476,10 +538,19 @@ static int sof_pcm_open(struct snd_soc_component *component,
spcm->prepared[substream->stream] = false;
ret = snd_sof_pcm_platform_open(sdev, substream);
- if (ret < 0)
- dev_err(component->dev, "error: pcm open failed %d\n", ret);
+ if (ret < 0) {
+ spcm_err(spcm, substream->stream,
+ "platform pcm open failed %d\n", ret);
+ return ret;
+ }
- return ret;
+ spcm_dbg(spcm, substream->stream, "period bytes min %zd, max %zd\n",
+ runtime->hw.period_bytes_min, runtime->hw.period_bytes_max);
+ spcm_dbg(spcm, substream->stream, "period count min %d, max %d\n",
+ runtime->hw.periods_min, runtime->hw.periods_max);
+ spcm_dbg(spcm, substream->stream, "buffer bytes max %zd\n", runtime->hw.buffer_bytes_max);
+
+ return 0;
}
static int sof_pcm_close(struct snd_soc_component *component,
@@ -498,13 +569,12 @@ static int sof_pcm_close(struct snd_soc_component *component,
if (!spcm)
return -EINVAL;
- dev_dbg(component->dev, "pcm: close stream %d dir %d\n",
- spcm->pcm.pcm_id, substream->stream);
+ spcm_dbg(spcm, substream->stream, "Entry: close\n");
err = snd_sof_pcm_platform_close(sdev, substream);
if (err < 0) {
- dev_err(component->dev, "error: pcm close failed %d\n",
- err);
+ spcm_err(spcm, substream->stream,
+ "platform pcm close failed %d\n", err);
/*
* keep going, no point in preventing the close
* from happening
@@ -538,7 +608,8 @@ static int sof_pcm_new(struct snd_soc_component *component,
return 0;
}
- dev_dbg(component->dev, "creating new PCM %s\n", spcm->pcm.pcm_name);
+ dev_dbg(spcm->scomp->dev, "pcm%u (%s): Entry: pcm_construct\n",
+ spcm->pcm.pcm_id, spcm->pcm.pcm_name);
/* do we need to pre-allocate playback audio buffer pages */
if (!spcm->pcm.playback)
@@ -546,16 +617,15 @@ static int sof_pcm_new(struct snd_soc_component *component,
caps = &spcm->pcm.caps[stream];
- /* pre-allocate playback audio buffer pages */
- dev_dbg(component->dev,
- "spcm: allocate %s playback DMA buffer size 0x%x max 0x%x\n",
- caps->name, caps->buffer_size_min, caps->buffer_size_max);
-
if (!pcm->streams[stream].substream) {
- dev_err(component->dev, "error: NULL playback substream!\n");
+ spcm_err(spcm, stream, "NULL playback substream!\n");
return -EINVAL;
}
+ /* pre-allocate playback audio buffer pages */
+ spcm_dbg(spcm, stream, "allocate %s playback DMA buffer size 0x%x max 0x%x\n",
+ caps->name, caps->buffer_size_min, caps->buffer_size_max);
+
snd_pcm_set_managed_buffer(pcm->streams[stream].substream,
SNDRV_DMA_TYPE_DEV_SG, sdev->dev,
0, le32_to_cpu(caps->buffer_size_max));
@@ -568,16 +638,15 @@ static int sof_pcm_new(struct snd_soc_component *component,
caps = &spcm->pcm.caps[stream];
- /* pre-allocate capture audio buffer pages */
- dev_dbg(component->dev,
- "spcm: allocate %s capture DMA buffer size 0x%x max 0x%x\n",
- caps->name, caps->buffer_size_min, caps->buffer_size_max);
-
if (!pcm->streams[stream].substream) {
- dev_err(component->dev, "error: NULL capture substream!\n");
+ spcm_err(spcm, stream, "NULL capture substream!\n");
return -EINVAL;
}
+ /* pre-allocate capture audio buffer pages */
+ spcm_dbg(spcm, stream, "allocate %s capture DMA buffer size 0x%x max 0x%x\n",
+ caps->name, caps->buffer_size_min, caps->buffer_size_max);
+
snd_pcm_set_managed_buffer(pcm->streams[stream].substream,
SNDRV_DMA_TYPE_DEV_SG, sdev->dev,
0, le32_to_cpu(caps->buffer_size_max));
diff --git a/sound/soc/sof/sof-audio.c b/sound/soc/sof/sof-audio.c
index f0a4e146d798..ef7d2f4b31c8 100644
--- a/sound/soc/sof/sof-audio.c
+++ b/sound/soc/sof/sof-audio.c
@@ -828,55 +828,6 @@ bool snd_sof_stream_suspend_ignored(struct snd_sof_dev *sdev)
return false;
}
-int sof_pcm_stream_free(struct snd_sof_dev *sdev, struct snd_pcm_substream *substream,
- struct snd_sof_pcm *spcm, int dir, bool free_widget_list)
-{
- const struct sof_ipc_pcm_ops *pcm_ops = sof_ipc_get_ops(sdev, pcm);
- int ret;
- int err = 0;
-
- if (spcm->prepared[substream->stream]) {
- /* stop DMA first if needed */
- if (pcm_ops && pcm_ops->platform_stop_during_hw_free)
- snd_sof_pcm_platform_trigger(sdev, substream, SNDRV_PCM_TRIGGER_STOP);
-
- /* free PCM in the DSP */
- if (pcm_ops && pcm_ops->hw_free) {
- ret = pcm_ops->hw_free(sdev->component, substream);
- if (ret < 0) {
- dev_err(sdev->dev, "%s: pcm_ops hw_free failed %d\n",
- __func__, ret);
- err = ret;
- }
- }
-
- spcm->prepared[substream->stream] = false;
- spcm->pending_stop[substream->stream] = false;
- }
-
- /* reset the DMA */
- ret = snd_sof_pcm_platform_hw_free(sdev, substream);
- if (ret < 0) {
- dev_err(sdev->dev, "%s: platform hw free failed %d\n",
- __func__, ret);
- if (!err)
- err = ret;
- }
-
- /* free widget list */
- if (free_widget_list) {
- ret = sof_widget_list_free(sdev, spcm, dir);
- if (ret < 0) {
- dev_err(sdev->dev, "%s: sof_widget_list_free failed %d\n",
- __func__, ret);
- if (!err)
- err = ret;
- }
- }
-
- return err;
-}
-
/*
* Generic object lookup APIs.
*/
diff --git a/sound/soc/sof/sof-audio.h b/sound/soc/sof/sof-audio.h
index 01b819dd8498..a0b1bfcc093a 100644
--- a/sound/soc/sof/sof-audio.h
+++ b/sound/soc/sof/sof-audio.h
@@ -616,6 +616,20 @@ struct snd_sof_pcm *snd_sof_find_spcm_comp(struct snd_soc_component *scomp,
void snd_sof_pcm_period_elapsed(struct snd_pcm_substream *substream);
void snd_sof_pcm_init_elapsed_work(struct work_struct *work);
+/*
+ * snd_sof_pcm specific wrappers for dev_dbg() and dev_err() to provide
+ * consistent and useful prints.
+ */
+#define spcm_dbg(__spcm, __dir, __fmt, ...) \
+ dev_dbg((__spcm)->scomp->dev, "pcm%u (%s), dir %d: " __fmt, \
+ (__spcm)->pcm.pcm_id, (__spcm)->pcm.pcm_name, __dir, \
+ ##__VA_ARGS__)
+
+#define spcm_err(__spcm, __dir, __fmt, ...) \
+ dev_err((__spcm)->scomp->dev, "%s: pcm%u (%s), dir %d: " __fmt, \
+ __func__, (__spcm)->pcm.pcm_id, (__spcm)->pcm.pcm_name, __dir, \
+ ##__VA_ARGS__)
+
#if IS_ENABLED(CONFIG_SND_SOC_SOF_COMPRESS)
void snd_sof_compr_fragment_elapsed(struct snd_compr_stream *cstream);
void snd_sof_compr_init_elapsed_work(struct work_struct *work);
@@ -648,8 +662,7 @@ int sof_widget_list_setup(struct snd_sof_dev *sdev, struct snd_sof_pcm *spcm,
int sof_widget_list_free(struct snd_sof_dev *sdev, struct snd_sof_pcm *spcm, int dir);
int sof_pcm_dsp_pcm_free(struct snd_pcm_substream *substream, struct snd_sof_dev *sdev,
struct snd_sof_pcm *spcm);
-int sof_pcm_stream_free(struct snd_sof_dev *sdev, struct snd_pcm_substream *substream,
- struct snd_sof_pcm *spcm, int dir, bool free_widget_list);
+int sof_pcm_free_all_streams(struct snd_sof_dev *sdev);
int get_token_u32(void *elem, void *object, u32 offset);
int get_token_u16(void *elem, void *object, u32 offset);
int get_token_comp_format(void *elem, void *object, u32 offset);
diff --git a/sound/usb/pcm.c b/sound/usb/pcm.c
index 08bf535ed163..3a4666e3bf27 100644
--- a/sound/usb/pcm.c
+++ b/sound/usb/pcm.c
@@ -1505,7 +1505,7 @@ static int prepare_playback_urb(struct snd_usb_substream *subs,
for (i = 0; i < ctx->packets; i++) {
counts = snd_usb_endpoint_next_packet_size(ep, ctx, i, avail);
- if (counts < 0)
+ if (counts < 0 || frames + counts >= ep->max_urb_frames)
break;
/* set up descriptor */
urb->iso_frame_desc[i].offset = frames * stride;
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index 9cceb6521143..d7f1c492e3dd 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -8,6 +8,7 @@
#include "bpf_experimental.h"
extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
#define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8)))
@@ -131,7 +132,7 @@ int reject_subprog_with_lock(void *ctx)
}
SEC("?tc")
-__failure __msg("bpf_rcu_read_unlock is missing")
+__failure __msg("bpf_throw cannot be used inside bpf_rcu_read_lock-ed region")
int reject_with_rcu_read_lock(void *ctx)
{
bpf_rcu_read_lock();
@@ -147,11 +148,13 @@ __noinline static int throwing_subprog(struct __sk_buff *ctx)
}
SEC("?tc")
-__failure __msg("bpf_rcu_read_unlock is missing")
+__failure __msg("bpf_throw cannot be used inside bpf_rcu_read_lock-ed region")
int reject_subprog_with_rcu_read_lock(void *ctx)
{
bpf_rcu_read_lock();
- return throwing_subprog(ctx);
+ throwing_subprog(ctx);
+ bpf_rcu_read_unlock();
+ return 0;
}
static bool rbless(struct bpf_rb_node *n1, const struct bpf_rb_node *n2)
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
index 9d415f7ce599..8fcbb2ef5a53 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
@@ -287,9 +287,9 @@ __msg("17: (b7) r0 = 0")
__msg("18: (95) exit")
__msg("returning from callee:")
__msg("to caller at 9:")
-__msg("frame 0: propagating r1,r4")
+__msg("frame 0: propagating r1,r3,r4")
__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r4 stack= before 18: (95) exit")
+__msg("mark_precise: frame0: regs=r1,r3,r4 stack= before 18: (95) exit")
__msg("from 18 to 9: safe")
__naked int callback_result_precise(void)
{
@@ -419,9 +419,9 @@ __msg("to caller at 9:")
/* r1, r4 are always precise for bpf_loop(),
* r6 was marked before backtracking to callback body.
*/
-__msg("frame 0: propagating r1,r4,r6")
+__msg("frame 0: propagating r1,r3,r4,r6")
__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r4,r6 stack= before 16: (95) exit")
+__msg("mark_precise: frame0: regs=r1,r3,r4,r6 stack= before 16: (95) exit")
__msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0")
__msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop")
__msg("mark_precise: frame0: parent state regs= stack=:")
@@ -575,9 +575,9 @@ __msg("to caller at 10:")
/* r1, r4 are always precise for bpf_loop(),
* fp-8 was marked before backtracking to callback body.
*/
-__msg("frame 0: propagating r1,r4,fp-8")
+__msg("frame 0: propagating r1,r3,r4,fp-8")
__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r4 stack=-8 before 18: (95) exit")
+__msg("mark_precise: frame0: regs=r1,r3,r4 stack=-8 before 18: (95) exit")
__msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0")
__msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181")
__msg("mark_precise: frame0: parent state regs= stack=:")
diff --git a/tools/testing/selftests/net/forwarding/README b/tools/testing/selftests/net/forwarding/README
index a652429bfd53..7b41cff993ad 100644
--- a/tools/testing/selftests/net/forwarding/README
+++ b/tools/testing/selftests/net/forwarding/README
@@ -6,7 +6,7 @@ to easily create and test complex environments.
Unfortunately, these namespaces can not be used with actual switching
ASICs, as their ports can not be migrated to other network namespaces
-(dev->netns_local) and most of them probably do not support the
+(dev->netns_immutable) and most of them probably do not support the
L1-separation provided by namespaces.
However, a similar kind of flexibility can be achieved by using VRFs and
diff --git a/tools/testing/selftests/net/tls.c b/tools/testing/selftests/net/tls.c
index f27a12d2a2c9..9388e77a1b21 100644
--- a/tools/testing/selftests/net/tls.c
+++ b/tools/testing/selftests/net/tls.c
@@ -1649,6 +1649,521 @@ TEST_F(tls, recv_efault)
EXPECT_EQ(memcmp(rec2, recv_mem + 9, ret - 9), 0);
}
+#define TLS_RECORD_TYPE_HANDSHAKE 0x16
+
+TEST_F(tls_basic, recvmsg_nopad_retry_iov)
+{
+ char payload[32];
+ char first_iov[sizeof(payload)];
+ char later_iov[sizeof(payload) * 2];
+ char expected_later_iov[sizeof(later_iov)];
+ char cbuf[CMSG_SPACE(sizeof(char))];
+ struct tls_crypto_info_keys tls13;
+ struct iovec iov[] = {
+ { .iov_base = first_iov, .iov_len = sizeof(first_iov) },
+ { .iov_base = later_iov, .iov_len = sizeof(later_iov) },
+ };
+ struct msghdr msg = {
+ .msg_iov = iov,
+ .msg_iovlen = ARRAY_SIZE(iov),
+ .msg_control = cbuf,
+ .msg_controllen = sizeof(cbuf),
+ };
+ int one = 1;
+ int ret;
+ int i;
+
+ if (self->notls)
+ SKIP(return, "no TLS support");
+
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls13, 0);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls13, tls13.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls13, tls13.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX_EXPECT_NO_PAD,
+ &one, sizeof(one));
+ ASSERT_EQ(ret, 0);
+
+ for (i = 0; i < sizeof(payload); i++)
+ payload[i] = 0x40 + i;
+ memset(first_iov, 0xa5, sizeof(first_iov));
+ memset(later_iov, 0x5a, sizeof(later_iov));
+ memset(expected_later_iov, 0x5a, sizeof(expected_later_iov));
+
+ /* A control record forces optimistic TLS 1.3 RX to retry. */
+ ret = tls_send_cmsg(self->fd, TLS_RECORD_TYPE_HANDSHAKE,
+ payload, sizeof(payload), 0);
+ ASSERT_EQ(ret, sizeof(payload));
+
+ ret = recvmsg(self->cfd, &msg, 0);
+ ASSERT_EQ(ret, sizeof(payload));
+ EXPECT_EQ(memcmp(first_iov, payload, sizeof(payload)), 0);
+ EXPECT_EQ(memcmp(later_iov, expected_later_iov,
+ sizeof(later_iov)), 0);
+}
+
+/* key_update, length 1, update_not_requested */
+static const char key_update_msg[] = "\x18\x00\x00\x01\x00";
+static void tls_send_keyupdate(struct __test_metadata *_metadata, int fd)
+{
+ size_t len = sizeof(key_update_msg);
+
+ EXPECT_EQ(tls_send_cmsg(fd, TLS_RECORD_TYPE_HANDSHAKE,
+ (char *)key_update_msg, len, 0),
+ len);
+}
+
+static void tls_recv_keyupdate(struct __test_metadata *_metadata, int fd, int flags)
+{
+ char buf[100];
+
+ EXPECT_EQ(tls_recv_cmsg(_metadata, fd, TLS_RECORD_TYPE_HANDSHAKE, buf, sizeof(buf), flags),
+ sizeof(key_update_msg));
+ EXPECT_EQ(memcmp(buf, key_update_msg, sizeof(key_update_msg)), 0);
+}
+
+/* set the key to 0 then 1 for RX, immediately to 1 for TX */
+TEST_F(tls_basic, rekey_rx)
+{
+ struct tls_crypto_info_keys tls12_0, tls12_1;
+ char const *test_str = "test_message";
+ int send_len = strlen(test_str) + 1;
+ char buf[20];
+ int ret;
+
+ if (self->notls)
+ return;
+
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_0, 0);
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_1, 1);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_1, tls12_1.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_0, tls12_0.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_1, tls12_1.len);
+ EXPECT_EQ(ret, 0);
+
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str, send_len), 0);
+}
+
+/* set the key to 0 then 1 for TX, immediately to 1 for RX */
+TEST_F(tls_basic, rekey_tx)
+{
+ struct tls_crypto_info_keys tls12_0, tls12_1;
+ char const *test_str = "test_message";
+ int send_len = strlen(test_str) + 1;
+ char buf[20];
+ int ret;
+
+ if (self->notls)
+ return;
+
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_0, 0);
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_1, 1);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_0, tls12_0.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_1, tls12_1.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_1, tls12_1.len);
+ EXPECT_EQ(ret, 0);
+
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str, send_len), 0);
+}
+
+TEST_F(tls, rekey)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ char const *test_str_2 = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ /* initial send/recv */
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* send after rekey */
+ send_len = strlen(test_str_2) + 1;
+ EXPECT_EQ(send(self->fd, test_str_2, send_len, 0), send_len);
+
+ /* can't receive the KeyUpdate without a control message */
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), -1);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* recv blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), 0), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* recv non-blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_DONTWAIT), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* update RX key */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ /* recv after rekey */
+ EXPECT_NE(recv(self->cfd, buf, send_len, 0), -1);
+ EXPECT_EQ(memcmp(buf, test_str_2, send_len), 0);
+}
+
+TEST_F(tls, rekey_fail)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ char const *test_str_2 = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+
+ /* initial send/recv */
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+
+ if (variant->tls_version != TLS_1_3_VERSION) {
+ /* just check that rekey is not supported and return */
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EBUSY);
+ return;
+ }
+
+ /* successful update */
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* invalid update: change of version */
+ tls_crypto_info_init(TLS_1_2_VERSION, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* invalid update (RX socket): change of version */
+ tls_crypto_info_init(TLS_1_2_VERSION, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* invalid update: change of cipher */
+ if (variant->cipher_type == TLS_CIPHER_AES_GCM_256)
+ tls_crypto_info_init(variant->tls_version, TLS_CIPHER_CHACHA20_POLY1305, &tls12, 1);
+ else
+ tls_crypto_info_init(variant->tls_version, TLS_CIPHER_AES_GCM_256, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* send after rekey, the invalid updates shouldn't have an effect */
+ send_len = strlen(test_str_2) + 1;
+ EXPECT_EQ(send(self->fd, test_str_2, send_len, 0), send_len);
+
+ /* can't receive the KeyUpdate without a control message */
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), -1);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* recv blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), 0), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* recv non-blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_DONTWAIT), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* update RX key */
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ /* recv after rekey */
+ EXPECT_NE(recv(self->cfd, buf, send_len, 0), -1);
+ EXPECT_EQ(memcmp(buf, test_str_2, send_len), 0);
+}
+
+TEST_F(tls, rekey_peek)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_PEEK), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ /* can't receive the KeyUpdate without a control message */
+ EXPECT_EQ(recv(self->cfd, buf, send_len, MSG_PEEK), -1);
+
+ /* peek KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, MSG_PEEK);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* update RX key */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+}
+
+TEST_F(tls, splice_rekey)
+{
+ int send_len = TLS_PAYLOAD_MAX_LEN / 2;
+ char mem_send[TLS_PAYLOAD_MAX_LEN];
+ char mem_recv[TLS_PAYLOAD_MAX_LEN];
+ struct tls_crypto_info_keys tls12;
+ int p[2];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ memrnd(mem_send, sizeof(mem_send));
+
+ ASSERT_GE(pipe(p), 0);
+ EXPECT_EQ(send(self->fd, mem_send, send_len, 0), send_len);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(send(self->fd, mem_send, send_len, 0), send_len);
+
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len);
+ EXPECT_EQ(read(p[0], mem_recv, send_len), send_len);
+ EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0);
+
+ /* can't splice the KeyUpdate */
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* peek KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, MSG_PEEK);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* can't splice before updating the key */
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* update RX key */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len);
+ EXPECT_EQ(read(p[0], mem_recv, send_len), send_len);
+ EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0);
+}
+
+TEST_F(tls, rekey_peek_splice)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+ char mem_recv[TLS_PAYLOAD_MAX_LEN];
+ int p[2];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ ASSERT_GE(pipe(p), 0);
+
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_PEEK), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len);
+ EXPECT_EQ(read(p[0], mem_recv, send_len), send_len);
+ EXPECT_EQ(memcmp(mem_recv, test_str_1, send_len), 0);
+}
+
+TEST_F(tls, rekey_getsockopt)
+{
+ struct tls_crypto_info_keys tls12;
+ struct tls_crypto_info_keys tls12_get;
+ socklen_t len;
+
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+}
+
+TEST_F(tls, rekey_poll_pending)
+{
+ char const *test_str = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ struct pollfd pfd = { };
+ int send_len;
+ int ret;
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* send immediately after rekey */
+ send_len = strlen(test_str) + 1;
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+
+ /* key hasn't been updated, expect cfd to be non-readable */
+ pfd.fd = self->cfd;
+ pfd.events = POLLIN;
+ EXPECT_EQ(poll(&pfd, 1, 0), 0);
+
+ ret = fork();
+ ASSERT_GE(ret, 0);
+
+ if (ret) {
+ int pid2, status;
+
+ /* wait before installing the new key */
+ sleep(1);
+
+ /* update RX key while poll() is sleeping */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ pid2 = wait(&status);
+ EXPECT_EQ(pid2, ret);
+ EXPECT_EQ(status, 0);
+ } else {
+ pfd.fd = self->cfd;
+ pfd.events = POLLIN;
+ EXPECT_EQ(poll(&pfd, 1, 5000), 1);
+
+ exit(!__test_passed(_metadata));
+ }
+}
+
+TEST_F(tls, rekey_poll_delay)
+{
+ char const *test_str = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ struct pollfd pfd = { };
+ int send_len;
+ int ret;
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ ret = fork();
+ ASSERT_GE(ret, 0);
+
+ if (ret) {
+ int pid2, status;
+
+ /* wait before installing the new key */
+ sleep(1);
+
+ /* update RX key while poll() is sleeping */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ sleep(1);
+ send_len = strlen(test_str) + 1;
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+
+ pid2 = wait(&status);
+ EXPECT_EQ(pid2, ret);
+ EXPECT_EQ(status, 0);
+ } else {
+ pfd.fd = self->cfd;
+ pfd.events = POLLIN;
+ EXPECT_EQ(poll(&pfd, 1, 5000), 1);
+ exit(!__test_passed(_metadata));
+ }
+}
+
FIXTURE(tls_err)
{
int fd, cfd;
diff --git a/tools/testing/vma/vma.c b/tools/testing/vma/vma.c
index 9074aaced9c5..46edffe9c55a 100644
--- a/tools/testing/vma/vma.c
+++ b/tools/testing/vma/vma.c
@@ -1501,7 +1501,7 @@ static bool test_copy_vma(void)
/* Move backwards and do not merge. */
vma = alloc_and_link_vma(&mm, 0x3000, 0x5000, 3, flags);
- vma_new = copy_vma(&vma, 0, 0x2000, 0, &need_locks);
+ vma_new = copy_vma(&vma, 0, 0x2000, 0, &need_locks, false);
ASSERT_NE(vma_new, vma);
ASSERT_EQ(vma_new->vm_start, 0);
@@ -1514,7 +1514,7 @@ static bool test_copy_vma(void)
vma = alloc_and_link_vma(&mm, 0, 0x2000, 0, flags);
vma_next = alloc_and_link_vma(&mm, 0x6000, 0x8000, 6, flags);
- vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, &need_locks);
+ vma_new = copy_vma(&vma, 0x4000, 0x2000, 4, &need_locks, false);
ASSERT_EQ(vma_new, vma_next);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index e2c7e3d91840..71afb0ca0cc9 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -473,15 +473,16 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
}
int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
- unsigned int fd, loff_t offset)
+ unsigned int fd, u64 offset)
{
- loff_t size = slot->npages << PAGE_SHIFT;
+ u64 size = slot->npages << PAGE_SHIFT;
unsigned long start, end;
struct kvm_gmem *gmem;
struct inode *inode;
struct file *file;
int r = -EINVAL;
+ BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset));
BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
file = fget(fd);
@@ -497,8 +498,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
inode = file_inode(file);
- if (offset < 0 || !PAGE_ALIGNED(offset) ||
- offset + size > i_size_read(inode))
+ if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode))
goto err;
filemap_invalidate_lock(inode->i_mapping);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 15ad929ec0d0..55ec5956c28b 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -136,8 +136,6 @@ static int kvm_no_compat_open(struct inode *inode, struct file *file)
#define KVM_COMPAT(c) .compat_ioctl = kvm_no_compat_ioctl, \
.open = kvm_no_compat_open
#endif
-static int kvm_enable_virtualization(void);
-static void kvm_disable_virtualization(void);
static void kvm_io_bus_destroy(struct kvm_io_bus *bus);
@@ -5664,7 +5662,7 @@ static int kvm_offline_cpu(unsigned int cpu)
return 0;
}
-static void kvm_shutdown(void)
+static void kvm_shutdown(void *data)
{
/*
* Disable hardware virtualization and set kvm_rebooting to indicate
@@ -5682,7 +5680,7 @@ static void kvm_shutdown(void)
on_each_cpu(kvm_disable_virtualization_cpu, NULL, 1);
}
-static int kvm_suspend(void)
+static int kvm_suspend(void *data)
{
/*
* Secondary CPUs and CPU hotplug are disabled across the suspend/resume
@@ -5699,7 +5697,7 @@ static int kvm_suspend(void)
return 0;
}
-static void kvm_resume(void)
+static void kvm_resume(void *data)
{
lockdep_assert_not_held(&kvm_usage_lock);
lockdep_assert_irqs_disabled();
@@ -5707,13 +5705,17 @@ static void kvm_resume(void)
WARN_ON_ONCE(kvm_enable_virtualization_cpu());
}
-static struct syscore_ops kvm_syscore_ops = {
+static const struct syscore_ops kvm_syscore_ops = {
.suspend = kvm_suspend,
.resume = kvm_resume,
.shutdown = kvm_shutdown,
};
-static int kvm_enable_virtualization(void)
+static struct syscore kvm_syscore = {
+ .ops = &kvm_syscore_ops,
+};
+
+int kvm_enable_virtualization(void)
{
int r;
@@ -5729,7 +5731,7 @@ static int kvm_enable_virtualization(void)
if (r)
goto err_cpuhp;
- register_syscore_ops(&kvm_syscore_ops);
+ register_syscore(&kvm_syscore);
/*
* Undo virtualization enabling and bail if the system is going down.
@@ -5751,25 +5753,27 @@ static int kvm_enable_virtualization(void)
return 0;
err_rebooting:
- unregister_syscore_ops(&kvm_syscore_ops);
+ unregister_syscore(&kvm_syscore);
cpuhp_remove_state(CPUHP_AP_KVM_ONLINE);
err_cpuhp:
kvm_arch_disable_virtualization();
--kvm_usage_count;
return r;
}
+EXPORT_SYMBOL_GPL(kvm_enable_virtualization);
-static void kvm_disable_virtualization(void)
+void kvm_disable_virtualization(void)
{
guard(mutex)(&kvm_usage_lock);
if (--kvm_usage_count)
return;
- unregister_syscore_ops(&kvm_syscore_ops);
+ unregister_syscore(&kvm_syscore);
cpuhp_remove_state(CPUHP_AP_KVM_ONLINE);
kvm_arch_disable_virtualization();
}
+EXPORT_SYMBOL_GPL(kvm_disable_virtualization);
static int kvm_init_virtualization(void)
{
@@ -5785,21 +5789,11 @@ static void kvm_uninit_virtualization(void)
kvm_disable_virtualization();
}
#else /* CONFIG_KVM_GENERIC_HARDWARE_ENABLING */
-static int kvm_enable_virtualization(void)
-{
- return 0;
-}
-
static int kvm_init_virtualization(void)
{
return 0;
}
-static void kvm_disable_virtualization(void)
-{
-
-}
-
static void kvm_uninit_virtualization(void)
{
diff --git a/virt/kvm/kvm_mm.h b/virt/kvm/kvm_mm.h
index 715f19669d01..15b2c728f7dd 100644
--- a/virt/kvm/kvm_mm.h
+++ b/virt/kvm/kvm_mm.h
@@ -3,6 +3,9 @@
#ifndef __KVM_MM_H__
#define __KVM_MM_H__ 1
+#include <linux/kvm.h>
+#include <linux/kvm_types.h>
+
/*
* Architectures can choose whether to use an rwlock or spinlock
* for the mmu_lock. These macros, for use in common code
@@ -39,7 +42,7 @@ static inline void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm,
void kvm_gmem_init(struct module *module);
int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args);
int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
- unsigned int fd, loff_t offset);
+ unsigned int fd, u64 offset);
void kvm_gmem_unbind(struct kvm_memory_slot *slot);
#else
static inline void kvm_gmem_init(struct module *module)
@@ -49,7 +52,7 @@ static inline void kvm_gmem_init(struct module *module)
static inline int kvm_gmem_bind(struct kvm *kvm,
struct kvm_memory_slot *slot,
- unsigned int fd, loff_t offset)
+ unsigned int fd, u64 offset)
{
WARN_ON_ONCE(1);
return -EIO;
prev parent reply other threads:[~2026-10-21 8:39 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-21 8:38 Greg Kroah-Hartman
2026-10-21 8:38 ` Greg Kroah-Hartman [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2026102108-embroider-balsamic-0b6b@gregkh \
--to=gregkh@linuxfoundation.org \
--cc=akpm@linux-foundation.org \
--cc=jslaby@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=lwn@lwn.net \
--cc=stable@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®