* Re: Linux 6.18.57
2026-10-15 10:37 Linux 6.18.57 Greg Kroah-Hartman
@ 2026-10-15 10:37 ` Greg Kroah-Hartman
0 siblings, 0 replies; 2+ messages in thread
From: Greg Kroah-Hartman @ 2026-10-15 10:37 UTC (permalink / raw)
To: linux-kernel, akpm, torvalds, stable; +Cc: lwn, jslaby, Greg Kroah-Hartman
diff --git a/Documentation/driver-api/driver-model/devres.rst b/Documentation/driver-api/driver-model/devres.rst
index 2b36ebde9cec..012f42a48f85 100644
--- a/Documentation/driver-api/driver-model/devres.rst
+++ b/Documentation/driver-api/driver-model/devres.rst
@@ -466,3 +466,7 @@ SPI
WATCHDOG
devm_watchdog_register_device()
+
+WORKQUEUE
+ devm_alloc_workqueue()
+ devm_alloc_ordered_workqueue()
diff --git a/Documentation/netlink/netlink-raw.yaml b/Documentation/netlink/netlink-raw.yaml
index f7c905c99936..be8b3717a640 100644
--- a/Documentation/netlink/netlink-raw.yaml
+++ b/Documentation/netlink/netlink-raw.yaml
@@ -19,6 +19,12 @@ $defs:
type: [ string, integer ]
pattern: ^[0-9A-Za-z_-]+( - 1)?$
minimum: 0
+ len-or-limit:
+ # literal int, const name, or limit based on fixed-width type
+ # e.g. u8-min, u16-max, etc.
+ type: [ string, integer ]
+ pattern: ^[0-9A-Za-z_-]+$
+ minimum: 0
# Schema for specs
title: Protocol
@@ -279,7 +285,10 @@ properties:
type: string
min:
description: Min value for an integer attribute.
- type: integer
+ $ref: '#/$defs/len-or-limit'
+ max:
+ description: Max value for an integer attribute.
+ $ref: '#/$defs/len-or-limit'
min-len:
description: Min length for a binary attribute.
$ref: '#/$defs/len-or-define'
diff --git a/Documentation/netlink/specs/conntrack.yaml b/Documentation/netlink/specs/conntrack.yaml
index bef528633b17..db7cddcda50a 100644
--- a/Documentation/netlink/specs/conntrack.yaml
+++ b/Documentation/netlink/specs/conntrack.yaml
@@ -457,7 +457,7 @@ attribute-sets:
name: labels
type: binary
-
- name: labels mask
+ name: labels-mask
type: binary
-
name: synproxy
diff --git a/Documentation/netlink/specs/nftables.yaml b/Documentation/netlink/specs/nftables.yaml
index cce88819ba71..17ad707fa0d5 100644
--- a/Documentation/netlink/specs/nftables.yaml
+++ b/Documentation/netlink/specs/nftables.yaml
@@ -915,7 +915,7 @@ attribute-sets:
type: string
doc: Name of set to use
-
- name: set id
+ name: set-id
type: u32
byte-order: big-endian
doc: ID of set to use
diff --git a/Documentation/networking/af_xdp.rst b/Documentation/networking/af_xdp.rst
index 50d92084a49c..cc3f0d16b28f 100644
--- a/Documentation/networking/af_xdp.rst
+++ b/Documentation/networking/af_xdp.rst
@@ -43,12 +43,13 @@ UMEM also has two rings: the FILL ring and the COMPLETION ring. The
FILL ring is used by the application to send down addr for the kernel
to fill in with RX packet data. References to these frames will then
appear in the RX ring once each packet has been received. The
-COMPLETION ring, on the other hand, contains frame addr that the
-kernel has transmitted completely and can now be used again by user
-space, for either TX or RX. Thus, the frame addrs appearing in the
-COMPLETION ring are addrs that were previously transmitted using the
-TX ring. In summary, the RX and FILL rings are used for the RX path
-and the TX and COMPLETION rings are used for the TX path.
+COMPLETION ring, on the other hand, contains frame addresses from Tx
+descriptors that the kernel has finished processing and that can now be
+used again by user space, for either Tx or Rx. This includes frames whose
+transmission has completed as well as frames referenced by invalid Tx
+descriptors rejected by the kernel. A completion therefore returns
+ownership of a frame to user space, but does not by itself guarantee that
+the packet was successfully transmitted.
The socket is then finally bound with a bind() call to a device and a
specific queue id on that device, and it is not until bind is
@@ -169,14 +170,15 @@ chunks mode, then the incoming addr will be left untouched.
UMEM Completion Ring
~~~~~~~~~~~~~~~~~~~~
-The COMPLETION Ring is used transfer ownership of UMEM frames from
+The COMPLETION Ring is used to transfer ownership of UMEM frames from
kernel-space to user-space. Just like the FILL ring, UMEM indices are
-used.
-
-Frames passed from the kernel to user-space are frames that has been
-sent (TX ring) and can be used by user-space again.
-
-The user application consumes UMEM addrs from this ring.
+used. Frames passed from the kernel to user-space are frames referenced
+by Tx descriptors that the kernel has finished processing and can be
+used by user-space again. This includes both frames whose transmission
+has completed and frames referenced by invalid Tx descriptors that were
+rejected and reclaimed by the kernel. A completion entry does not
+guarantee successful packet transmission. The user application consumes
+UMEM addrs from this ring.
RX Ring
@@ -504,21 +506,25 @@ will be treated as an invalid descriptor.
These are the semantics for producing packets onto AF_XDP Tx ring
consisting of multiple frames:
-* When an invalid descriptor is found, all the other
- descriptors/frames of this packet are marked as invalid and not
- completed. The next descriptor is treated as the start of a new
- packet, even if this was not the intent (because we cannot guess
- the intent). As before, if your program is producing invalid
- descriptors you have a bug that must be fixed.
+* When an invalid descriptor is found, the complete packet is treated as
+ invalid. The kernel consumes descriptors through the descriptor marking
+ the end of the packet and returns all their frame addresses through the
+ COMPLETION ring. A standalone invalid descriptor is treated as a
+ one-descriptor invalid packet. The descriptor following the end of the
+ invalid packet is treated as the start of a new packet. As before, if
+ your program is producing invalid descriptors you have a bug that must
+ be fixed. Rejected descriptors are reported in the ``tx_invalid_descs``
+ statistic.
* Zero length descriptors are treated as invalid descriptors.
* For copy mode, the maximum supported number of frames in a packet is
- equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all
- descriptors accumulated so far are dropped and treated as
- invalid. To produce an application that will work on any system
- regardless of this config setting, limit the number of frags to 18,
- as the minimum value of the config is 17.
+ equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all descriptors
+ through the end of the oversized packet are consumed, treated as invalid,
+ and their frame addresses are returned through the COMPLETION ring. To
+ produce an application that will work on any system regardless of this
+ config setting, limit the number of frags to 18, as the minimum value of
+ the config is 17.
* For zero-copy mode, the limit is up to what the NIC HW
supports. Usually at least five on the NICs we have checked. We
diff --git a/Documentation/networking/strparser.rst b/Documentation/networking/strparser.rst
index 8dc6bb04c710..80ee14ed61f8 100644
--- a/Documentation/networking/strparser.rst
+++ b/Documentation/networking/strparser.rst
@@ -77,7 +77,7 @@ Functions
strp_done is called to release any resources held by the stream
parser instance. This must be called after the stream processor
- has been stopped.
+ has been stopped. The socket lock must not be held.
::
diff --git a/Makefile b/Makefile
index 3af976c5dd34..5b57b2f093ea 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
VERSION = 6
PATCHLEVEL = 18
-SUBLEVEL = 56
+SUBLEVEL = 57
EXTRAVERSION =
NAME = Baby Opossum Posse
diff --git a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
index f7fb0a0562ec..8fefb7486770 100644
--- a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
+++ b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
@@ -1779,7 +1779,7 @@ esdhc0_clksync_pins: iic5-sdhc-clk-pins {
};
dspi2_miso_mosi_pins: iic5-spi3-pins {
- pinctrl-single,bits = <0x3 (0x2 << 9) (0x7 << 9)>;
+ pinctrl-single,bits = <0x0 (0x3 << 9) (0x7 << 9)>;
};
i2c5_pins: iic6-i2c-pins {
@@ -1799,7 +1799,7 @@ fspi_data74_pins: xspi1-data74-pins {
};
gpio1_31_28_pins: xspi1-data74-gpio-pins {
- pinctrl-single,bits = <0x0 0x1 (0x7 << 15)>;
+ pinctrl-single,bits = <0x0 (0x1 << 15) (0x7 << 15)>;
};
fspi_data30_pins: xspi1-data30-pins {
@@ -1807,7 +1807,7 @@ fspi_data30_pins: xspi1-data30-pins {
};
gpio1_27_24_pins: xspi1-data30-gpio-pins {
- pinctrl-single,bits = <0x0 0x1 (0x7 << 18)>;
+ pinctrl-single,bits = <0x0 (0x1 << 18) (0x7 << 18)>;
};
fspi_dqs_sck_cs10_pins: xspi1-base-pins {
@@ -1815,7 +1815,7 @@ fspi_dqs_sck_cs10_pins: xspi1-base-pins {
};
gpio1_23_20_pins: xspi1-base-gpio-pins {
- pinctrl-single,bits = <0x0 0x1 (0x7 << 21)>;
+ pinctrl-single,bits = <0x0 (0x1 << 21) (0x7 << 21)>;
};
esdhc0_cmd_data30_clk_vsel_pins: sdhc1-base-sdhc-vsel-pins {
@@ -1864,12 +1864,19 @@ i2c6_i2c7_pins: iic8-iic7-i2c-pins {
};
/* RCWSR14 */
+ /*
+ * IIC1_PMUX is a single bit at RCWSR14[10], but bit-per-mux
+ * uses the mask for the field to clear before applying the
+ * value.
+ * So bits 11 and 12, which are reserved, are written as 0
+ * on every mux switch, it is harmless.
+ */
i2c0_pins: iic1-i2c-pins {
- pinctrl-single,bits = <0x8 0x0 (0x1 << 10)>;
+ pinctrl-single,bits = <0x8 0x0 (0x7 << 10)>;
};
gpio0_3_2_pins: iic1-gpio-pins {
- pinctrl-single,bits = <0x8 (0x1 << 10) (0x1 << 10)>;
+ pinctrl-single,bits = <0x8 (0x1 << 10) (0x7 << 10)>;
};
};
diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index 48f686e4bba2..b26ceeedefa8 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -412,8 +412,9 @@
mov x2, xzr
mrs x1, id_aa64dfr0_el1
ubfx x1, x1, #ID_AA64DFR0_EL1_PMUVer_SHIFT, #4
- cmp x1, #ID_AA64DFR0_EL1_PMUVer_V3P9
- b.lt .Lskip_pmuv3p9_\@
+ cmp x1, #ID_AA64DFR0_EL1_PMUVer_IMP_DEF
+ ccmp x1, #ID_AA64DFR0_EL1_PMUVer_V3P9, #8, ne
+ b.lt .Lskip_pmuv3p9_\@ // Skip if < PMUv3p9 or IMP_DEF
orr x0, x0, #HDFGRTR2_EL2_nPMICNTR_EL0
orr x0, x0, #HDFGRTR2_EL2_nPMICFILTR_EL0
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index ca26460bde65..419e37112a6f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -493,7 +493,6 @@ enum vcpu_sysreg {
ACTLR_EL2, /* Auxiliary Control Register (EL2) */
CPTR_EL2, /* Architectural Feature Trap Register (EL2) */
HACR_EL2, /* Hypervisor Auxiliary Control Register */
- ZCR_EL2, /* SVE Control Register (EL2) */
TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */
TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */
TCR_EL2, /* Translation Control Register (EL2) */
@@ -524,6 +523,7 @@ enum vcpu_sysreg {
SCTLR2_EL2, /* System Control Register 2 (EL2) */
MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */
CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */
+ ZCR_EL2, /* SVE Control Register (EL2) */
/* Any VNCR-capable reg goes after this point */
MARKER(__VNCR_START__),
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 76f32e424065..7b574009f4bd 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -25,6 +25,9 @@ KASAN_SANITIZE_stacktrace.o := n
KCOV_INSTRUMENT_entry-common.o := n
KCOV_INSTRUMENT_idle.o := n
+# Softirq stack switching can run outside interrupt context.
+KCOV_INSTRUMENT_irq.o := n
+
# Object file lists.
obj-y := debug-monitors.o entry.o irq.o fpsimd.o \
entry-common.o entry-fpsimd.o process.o ptrace.o \
diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c
index ceb047e7d49f..8e7f8dedb1cc 100644
--- a/arch/arm64/kernel/ptrace.c
+++ b/arch/arm64/kernel/ptrace.c
@@ -560,6 +560,42 @@ static int gpr_get(struct task_struct *target,
return membuf_write(&to, uregs, sizeof(*uregs));
}
+static void update_syscall_orig_x0_after_ptrace(struct task_struct *target)
+{
+ struct pt_regs *regs = task_pt_regs(target);
+ struct kernel_siginfo *info = target->last_siginfo;
+
+ /*
+ * Skip the update for NO_SYSCALL (set either by the user or the
+ * tracer), as regs[0] holds the return value (see the comment in
+ * el0_svc_common()) and can be unwound using syscall_rollback().
+ */
+ if (regs->syscallno == NO_SYSCALL)
+ return;
+
+ /* We should only be called when target is in a ptrace stop */
+ if (WARN_ON_ONCE(!info))
+ return;
+
+ /*
+ * For compat tasks, orig_r0 is provided directly through GPR index
+ * 17.
+ */
+ if (is_compat_thread(task_thread_info(target)))
+ return;
+
+ /*
+ * Don't update orig_x0 for a syscall-exit-stop, as x0 now contains the
+ * return value of the system call.
+ */
+ if ((info->si_code & ~0x80) == SIGTRAP &&
+ target->ptrace_message == PTRACE_EVENTMSG_SYSCALL_EXIT) {
+ return;
+ }
+
+ regs->orig_x0 = regs->regs[0];
+}
+
static int gpr_set(struct task_struct *target, const struct user_regset *regset,
unsigned int pos, unsigned int count,
const void *kbuf, const void __user *ubuf)
@@ -575,6 +611,14 @@ static int gpr_set(struct task_struct *target, const struct user_regset *regset,
return -EINVAL;
task_pt_regs(target)->user_regs = newregs;
+
+ /*
+ * Keep orig_x0 authoritative so that seccomp (via
+ * syscall_get_arguments()), audit and the restart path all see the same
+ * first argument the syscall is dispatched with, even if it has been
+ * updated by a tracer.
+ */
+ update_syscall_orig_x0_after_ptrace(target);
return 0;
}
@@ -753,6 +797,12 @@ static int system_call_set(struct task_struct *target,
return ret;
task_pt_regs(target)->syscallno = syscallno;
+
+ /*
+ * Re-sync orig_x0 in case the syscall number has been changed
+ * from NO_SYSCALL.
+ */
+ update_syscall_orig_x0_after_ptrace(target);
return ret;
}
diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index e1ad60d3f367..b5c8a8e9adfe 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -438,11 +438,13 @@ static inline bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code)
static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)
{
+ u64 zcr_el2 = vcpu_sve_max_vq(vcpu) - 1;
+
/*
* The vCPU's saved SVE state layout always matches the max VL of the
* vCPU. Start off with the max VL so we can load the SVE state.
*/
- sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2);
+ sve_cond_update_zcr_vq(zcr_el2, SYS_ZCR_EL2);
__sve_restore_state(vcpu_sve_pffr(vcpu),
&vcpu->arch.ctxt.fp_regs.fpsr,
true);
@@ -452,8 +454,10 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)
* nested guest, as the guest hypervisor could select a smaller VL. Slap
* that into hardware before wrapping up.
*/
- if (is_nested_ctxt(vcpu))
- sve_cond_update_zcr_vq(__vcpu_sys_reg(vcpu, ZCR_EL2), SYS_ZCR_EL2);
+ if (is_nested_ctxt(vcpu)) {
+ zcr_el2 = min(zcr_el2, __vcpu_sys_reg(vcpu, ZCR_EL2));
+ sve_cond_update_zcr_vq(zcr_el2, SYS_ZCR_EL2);
+ }
write_sysreg_el1(__vcpu_sys_reg(vcpu, vcpu_sve_zcr_elx(vcpu)), SYS_ZCR);
}
@@ -477,11 +481,11 @@ static inline void fpsimd_lazy_switch_to_guest(struct kvm_vcpu *vcpu)
return;
if (vcpu_has_sve(vcpu)) {
+ zcr_el2 = vcpu_sve_max_vq(vcpu) - 1;
+
/* A guest hypervisor may restrict the effective max VL. */
if (is_nested_ctxt(vcpu))
- zcr_el2 = __vcpu_sys_reg(vcpu, ZCR_EL2);
- else
- zcr_el2 = vcpu_sve_max_vq(vcpu) - 1;
+ zcr_el2 = min(zcr_el2, __vcpu_sys_reg(vcpu, ZCR_EL2));
write_sysreg_el2(zcr_el2, SYS_ZCR);
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 3184fff22442..92853a343c42 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -1840,6 +1840,10 @@ int kvm_init_nv_sysregs(struct kvm_vcpu *vcpu)
/* VNCR_EL2 */
set_sysreg_masks(kvm, VNCR_EL2, VNCR_EL2_RES0, VNCR_EL2_RES1);
+ /* ZCR_EL2 - bits 8:4 are RAZ/WI so treat them as RES0 */
+ set_sysreg_masks(kvm, ZCR_EL2, ZCR_ELx_RES0 | GENMASK_ULL(8, 4),
+ ZCR_ELx_RES1);
+
out:
for (enum vcpu_sysreg sr = __SANITISED_REG_START__; sr < NR_SYS_REGS; sr++)
__vcpu_rmw_sys_reg(vcpu, sr, |=, 0);
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 308ba3dba809..bf8b6d599008 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2750,21 +2750,16 @@ static bool access_zcr_el2(struct kvm_vcpu *vcpu,
struct sys_reg_params *p,
const struct sys_reg_desc *r)
{
- unsigned int vq;
-
if (guest_hyp_sve_traps_enabled(vcpu)) {
kvm_inject_nested_sve_trap(vcpu);
return false;
}
- if (!p->is_write) {
+ if (!p->is_write)
p->regval = __vcpu_sys_reg(vcpu, ZCR_EL2);
- return true;
- }
+ else
+ __vcpu_assign_sys_reg(vcpu, ZCR_EL2, p->regval);
- vq = SYS_FIELD_GET(ZCR_ELx, LEN, p->regval) + 1;
- vq = min(vq, vcpu_sve_max_vq(vcpu));
- __vcpu_assign_sys_reg(vcpu, ZCR_EL2, vq - 1);
return true;
}
diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c
index e3c182352ffc..7ce07c8eb5b7 100644
--- a/arch/arm64/kvm/vgic/vgic-its.c
+++ b/arch/arm64/kvm/vgic/vgic-its.c
@@ -116,18 +116,26 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
kfree(irq);
irq = oldirq;
} else {
- ret = xa_err(__xa_store(&dist->lpi_xa, intid, irq, 0));
- }
-
- xa_unlock_irqrestore(&dist->lpi_xa, flags);
+ /*
+ * The entry is either empty or contains a dead LPI (refcount=0)
+ * from the deferred release path, pending cleanup by
+ * vgic_release_deleted_lpis(). Evict and free it if present.
+ */
+ oldirq = __xa_store(&dist->lpi_xa, intid, irq, 0);
+ ret = xa_err(oldirq);
+ if (ret) {
+ xa_unlock_irqrestore(&dist->lpi_xa, flags);
+ kfree(irq);
- if (ret) {
- xa_release(&dist->lpi_xa, intid);
- kfree(irq);
+ return ERR_PTR(ret);
+ }
- return ERR_PTR(ret);
+ if (oldirq && !WARN_ON_ONCE(refcount_read(&oldirq->refcount)))
+ kfree_rcu(oldirq, rcu);
}
+ xa_unlock_irqrestore(&dist->lpi_xa, flags);
+
/*
* We "cache" the configuration table entries in our struct vgic_irq's.
* However we only have those structs for mapped IRQs, so we read in
diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c
index 5ba673289ef4..7c8ec4482415 100644
--- a/arch/arm64/kvm/vgic/vgic.c
+++ b/arch/arm64/kvm/vgic/vgic.c
@@ -131,11 +131,7 @@ static __must_check bool __vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
static __must_check bool vgic_put_irq_norelease(struct kvm *kvm, struct vgic_irq *irq)
{
- if (!__vgic_put_irq(kvm, irq))
- return false;
-
- irq->pending_release = true;
- return true;
+ return __vgic_put_irq(kvm, irq);
}
void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
@@ -152,12 +148,14 @@ void vgic_put_irq(struct kvm *kvm, struct vgic_irq *irq)
guard(spinlock_irqsave)(&dist->lpi_xa.xa_lock);
}
- if (!__vgic_put_irq(kvm, irq))
+ if (irq->intid < VGIC_MIN_LPI)
return;
- xa_lock_irqsave(&dist->lpi_xa, flags);
- vgic_release_lpi_locked(dist, irq);
- xa_unlock_irqrestore(&dist->lpi_xa, flags);
+ if (refcount_dec_and_lock_irqsave(&irq->refcount,
+ &dist->lpi_xa.xa_lock, &flags)) {
+ vgic_release_lpi_locked(dist, irq);
+ xa_unlock_irqrestore(&dist->lpi_xa, flags);
+ }
}
static void vgic_release_deleted_lpis(struct kvm *kvm)
@@ -169,7 +167,7 @@ static void vgic_release_deleted_lpis(struct kvm *kvm)
xa_lock_irqsave(&dist->lpi_xa, flags);
xa_for_each(&dist->lpi_xa, intid, irq) {
- if (irq->pending_release)
+ if (!refcount_read(&irq->refcount))
vgic_release_lpi_locked(dist, irq);
}
diff --git a/arch/loongarch/include/asm/kvm_host.h b/arch/loongarch/include/asm/kvm_host.h
index 735396787b14..72ee4cdf5acc 100644
--- a/arch/loongarch/include/asm/kvm_host.h
+++ b/arch/loongarch/include/asm/kvm_host.h
@@ -126,6 +126,8 @@ struct kvm_arch {
struct kvm_phyid_map *phyid_map;
/* Enabled PV features */
unsigned long pv_features;
+ /* Supported KVM features */
+ unsigned long kvm_features;
s64 time_offset;
struct kvm_context __percpu *vmcs;
@@ -293,6 +295,12 @@ static inline int kvm_get_pmu_num(struct kvm_vcpu_arch *arch)
return (arch->cpucfg[6] & CPUCFG6_PMNUM) >> CPUCFG6_PMNUM_SHIFT;
}
+/* Check whether KVM support this feature (VMM may disable it) */
+static inline bool kvm_vm_support(struct kvm_arch *arch, int feature)
+{
+ return !!(arch->kvm_features & BIT_ULL(feature));
+}
+
bool kvm_arch_pmi_in_guest(struct kvm_vcpu *vcpu);
/* Debug: dump vcpu state */
diff --git a/arch/loongarch/include/asm/loongarch.h b/arch/loongarch/include/asm/loongarch.h
index 3de03cb864b2..58a4a3b6b035 100644
--- a/arch/loongarch/include/asm/loongarch.h
+++ b/arch/loongarch/include/asm/loongarch.h
@@ -511,6 +511,8 @@
#define CSR_GCFG_GPERF_SHIFT 24
#define CSR_GCFG_GPERF_WIDTH 3
#define CSR_GCFG_GPERF (_ULCAST_(0x7) << CSR_GCFG_GPERF_SHIFT)
+#define CSR_GCFG_GPMP_SHIFT 23
+#define CSR_GCFG_GPMP (_ULCAST_(0x1) << CSR_GCFG_GPMP_SHIFT)
#define CSR_GCFG_GCI_SHIFT 20
#define CSR_GCFG_GCI_WIDTH 2
#define CSR_GCFG_GCI (_ULCAST_(0x3) << CSR_GCFG_GCI_SHIFT)
diff --git a/arch/loongarch/kvm/intc/pch_pic.c b/arch/loongarch/kvm/intc/pch_pic.c
index c0adabfd5079..ab336cde24f6 100644
--- a/arch/loongarch/kvm/intc/pch_pic.c
+++ b/arch/loongarch/kvm/intc/pch_pic.c
@@ -69,6 +69,9 @@ void pch_pic_set_irq(struct loongarch_pch_pic *s, int irq, int level)
/* msi irq handler */
void pch_msi_set_irq(struct kvm *kvm, int irq, int level)
{
+ if (irq < 0 || irq >= EIOINTC_IRQS)
+ return;
+
eiointc_set_irq(kvm->arch.eiointc, irq, level);
}
diff --git a/arch/loongarch/kvm/vm.c b/arch/loongarch/kvm/vm.c
index 890d27e37dde..99f5d4b7c930 100644
--- a/arch/loongarch/kvm/vm.c
+++ b/arch/loongarch/kvm/vm.c
@@ -6,6 +6,7 @@
#include <linux/kvm_host.h>
#include <asm/kvm_mmu.h>
#include <asm/kvm_vcpu.h>
+#include <asm/kvm_csr.h>
#include <asm/kvm_eiointc.h>
#include <asm/kvm_pch_pic.h>
@@ -24,6 +25,23 @@ const struct kvm_stats_header kvm_vm_stats_header = {
sizeof(kvm_vm_stats_desc),
};
+static void kvm_vm_init_features(struct kvm *kvm)
+{
+ unsigned long val;
+
+ val = read_csr_gcfg();
+ if (val & CSR_GCFG_GPMP)
+ kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PMU);
+
+ /* Enable all PV features by default */
+ kvm->arch.pv_features = BIT(KVM_FEATURE_IPI);
+ kvm->arch.kvm_features = BIT(KVM_LOONGARCH_VM_FEAT_PV_IPI);
+ if (kvm_pvtime_supported()) {
+ kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME);
+ kvm->arch.kvm_features |= BIT(KVM_LOONGARCH_VM_FEAT_PV_STEALTIME);
+ }
+}
+
int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
{
int i;
@@ -43,11 +61,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
spin_lock_init(&kvm->arch.pv_setting_lock);
kvm_init_vmcs(kvm);
-
- /* Enable all PV features by default */
- kvm->arch.pv_features = BIT(KVM_FEATURE_IPI);
- if (kvm_pvtime_supported())
- kvm->arch.pv_features |= BIT(KVM_FEATURE_STEAL_TIME);
+ kvm_vm_init_features(kvm);
/*
* cpu_vabits means user address space only (a half of total).
@@ -137,18 +151,14 @@ static int kvm_vm_feature_has_attr(struct kvm *kvm, struct kvm_device_attr *attr
if (cpu_has_lbt_mips)
return 0;
return -ENXIO;
- case KVM_LOONGARCH_VM_FEAT_PMU:
- if (cpu_has_pmp)
+ case KVM_LOONGARCH_VM_FEAT_PTW:
+ if (cpu_has_ptw)
return 0;
return -ENXIO;
+ case KVM_LOONGARCH_VM_FEAT_PMU:
case KVM_LOONGARCH_VM_FEAT_PV_IPI:
- return 0;
case KVM_LOONGARCH_VM_FEAT_PV_STEALTIME:
- if (kvm_pvtime_supported())
- return 0;
- return -ENXIO;
- case KVM_LOONGARCH_VM_FEAT_PTW:
- if (cpu_has_ptw)
+ if (kvm_vm_support(&kvm->arch, attr->attr))
return 0;
return -ENXIO;
default:
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 5b5de4f91eb9..3fb764bc5e1b 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -620,7 +620,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
move_reg(ctx, t1, src);
emit_sext_32(ctx, t1, is32);
emit_insn(ctx, divd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
@@ -637,7 +637,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
emit_sext_32(ctx, t1, is32);
emit_sext_32(ctx, dst, is32);
emit_insn(ctx, divd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
@@ -655,7 +655,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
move_reg(ctx, t1, src);
emit_sext_32(ctx, t1, is32);
emit_insn(ctx, modd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
@@ -672,7 +672,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
emit_sext_32(ctx, t1, is32);
emit_sext_32(ctx, dst, is32);
emit_insn(ctx, modd, dst, dst, t1);
- emit_sext_32(ctx, dst, is32);
+ emit_zext_32(ctx, dst, is32);
}
break;
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index aac8ce30cd3b..7a218159ff15 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -1284,6 +1284,21 @@ extern int pudp_test_and_clear_young(struct vm_area_struct *vma,
static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
unsigned long addr, pmd_t *pmdp)
{
+ /*
+ * Non-present PMDs can be migration entries or device-private THP
+ * entries. This can happen at 2 places:
+ * - When the address space is being unmapped zap_huge_pmd(), and we
+ * encounter non-present pmds.
+ * - migrate_vma_collect_huge_pmd() could calls this during migration
+ * of device-private pmd entries.
+ */
+ if (!pmd_present(*pmdp)) {
+ pmd_t old_pmd = READ_ONCE(*pmdp);
+
+ pmd_clear(pmdp);
+ return old_pmd;
+ }
+
if (radix_enabled())
return radix__pmdp_huge_get_and_clear(mm, addr, pmdp);
return hash__pmdp_huge_get_and_clear(mm, addr, pmdp);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index efb0f5effcc6..4e3e3358993c 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -91,7 +91,7 @@ static inline bool xive_enabled(void) { return __xive_enabled; }
bool xive_spapr_init(void);
bool xive_native_init(void);
-void xive_smp_probe(void);
+int xive_smp_probe(void);
int xive_smp_prepare_cpu(unsigned int cpu);
void xive_smp_setup_cpu(void);
void xive_smp_disable_cpu(void);
@@ -153,7 +153,7 @@ static inline bool xive_enabled(void) { return false; }
static inline bool xive_spapr_init(void) { return false; }
static inline bool xive_native_init(void) { return false; }
-static inline void xive_smp_probe(void) { }
+static inline int xive_smp_probe(void) { return -EINVAL; }
static inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
static inline void xive_smp_setup_cpu(void) { }
static inline void xive_smp_disable_cpu(void) { }
diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c
index 0a7d14f651e8..a0d7ca3c598c 100644
--- a/arch/powerpc/kernel/iommu.c
+++ b/arch/powerpc/kernel/iommu.c
@@ -1156,7 +1156,8 @@ EXPORT_SYMBOL_GPL(iommu_add_device);
*/
static int
spapr_tce_platform_iommu_attach_dev(struct iommu_domain *platform_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct iommu_table_group *table_group;
@@ -1189,7 +1190,7 @@ static struct iommu_domain spapr_tce_platform_domain = {
static int
spapr_tce_blocked_iommu_attach_dev(struct iommu_domain *platform_domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct iommu_group *grp = iommu_group_get(dev);
struct iommu_table_group *table_group;
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index c9431ae7f78a..47c7d2d9a053 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -202,16 +202,21 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp, int full)
{
pmd_t pmd;
+ bool was_present = pmd_present(*pmdp);
+
VM_BUG_ON(addr & ~HPAGE_PMD_MASK);
- VM_BUG_ON((pmd_present(*pmdp) && !pmd_trans_huge(*pmdp)) ||
- !pmd_present(*pmdp));
+ VM_BUG_ON(was_present && !pmd_trans_huge(*pmdp));
+ /*
+ * Check pmdp_huge_get_and_clear() for non-present pmd case.
+ */
pmd = pmdp_huge_get_and_clear(vma->vm_mm, addr, pmdp);
/*
* if it not a fullmm flush, then we can possibly end up converting
* this PMD pte entry to a regular level 0 PTE by a parallel page fault.
- * Make sure we flush the tlb in this case.
+ * Make sure we flush the tlb in this case. TLB flush not needed for
+ * non-present case.
*/
- if (!full)
+ if (was_present && !full)
flush_pmd_tlb_range(vma, addr, addr + HPAGE_PMD_SIZE);
return pmd;
}
diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
index 8f41ef364fc6..b1201dbafcaf 100644
--- a/arch/powerpc/platforms/powernv/smp.c
+++ b/arch/powerpc/platforms/powernv/smp.c
@@ -332,10 +332,12 @@ static void pnv_cause_ipi(int cpu)
static void __init pnv_smp_probe(void)
{
- if (xive_enabled())
- xive_smp_probe();
- else
+ if (xive_enabled()) {
+ if (xive_smp_probe() < 0)
+ return;
+ } else {
xics_smp_probe();
+ }
if (cpu_has_feature(CPU_FTR_DBELL)) {
ic_cause_ipi = smp_ops->cause_ipi;
diff --git a/arch/powerpc/platforms/pseries/papr-hvpipe.c b/arch/powerpc/platforms/pseries/papr-hvpipe.c
index 431f6a3da4c5..a238db095b4e 100644
--- a/arch/powerpc/platforms/pseries/papr-hvpipe.c
+++ b/arch/powerpc/platforms/pseries/papr-hvpipe.c
@@ -449,13 +449,14 @@ static int papr_hvpipe_handle_release(struct inode *inode,
struct file *file)
{
struct hvpipe_source_info *src_info;
+ unsigned long flags;
/*
* Hold the lock, remove source from src_list, reset the
* hvpipe status and release the lock to prevent any race
* with message event IRQ.
*/
- spin_lock(&hvpipe_src_list_lock);
+ spin_lock_irqsave(&hvpipe_src_list_lock, flags);
src_info = file->private_data;
list_del(&src_info->list);
file->private_data = NULL;
@@ -466,10 +467,10 @@ static int papr_hvpipe_handle_release(struct inode *inode,
*/
if (src_info->hvpipe_status & HVPIPE_MSG_AVAILABLE) {
src_info->hvpipe_status = 0;
- spin_unlock(&hvpipe_src_list_lock);
+ spin_unlock_irqrestore(&hvpipe_src_list_lock, flags);
hvpipe_rtas_recv_msg(NULL, 0);
} else
- spin_unlock(&hvpipe_src_list_lock);
+ spin_unlock_irqrestore(&hvpipe_src_list_lock, flags);
kfree(src_info);
return 0;
@@ -484,24 +485,22 @@ static const struct file_operations papr_hvpipe_handle_ops = {
static int papr_hvpipe_dev_create_handle(u32 srcID)
{
- struct hvpipe_source_info *src_info;
- struct file *file;
- long err;
- int fd;
+ struct hvpipe_source_info *src_info __free(kfree) = NULL;
+ unsigned long flags;
- spin_lock(&hvpipe_src_list_lock);
+ spin_lock_irqsave(&hvpipe_src_list_lock, flags);
/*
* Do not allow more than one process communicates with
* each source.
*/
src_info = hvpipe_find_source(srcID);
if (src_info) {
- spin_unlock(&hvpipe_src_list_lock);
+ spin_unlock_irqrestore(&hvpipe_src_list_lock, flags);
pr_err("pid(%d) is already using the source(%d)\n",
src_info->tsk->pid, srcID);
return -EALREADY;
}
- spin_unlock(&hvpipe_src_list_lock);
+ spin_unlock_irqrestore(&hvpipe_src_list_lock, flags);
src_info = kzalloc(sizeof(*src_info), GFP_KERNEL_ACCOUNT);
if (!src_info)
@@ -511,44 +510,26 @@ static int papr_hvpipe_dev_create_handle(u32 srcID)
src_info->tsk = current;
init_waitqueue_head(&src_info->recv_wqh);
- fd = get_unused_fd_flags(O_RDONLY | O_CLOEXEC);
- if (fd < 0) {
- err = fd;
- goto free_buf;
- }
+ FD_PREPARE(fdf, O_RDONLY | O_CLOEXEC,
+ anon_inode_getfile("[papr-hvpipe]", &papr_hvpipe_handle_ops,
+ (void *)src_info, O_RDWR));
+ if (fdf.err)
+ return fdf.err;
- file = anon_inode_getfile("[papr-hvpipe]",
- &papr_hvpipe_handle_ops, (void *)src_info,
- O_RDWR);
- if (IS_ERR(file)) {
- err = PTR_ERR(file);
- goto free_fd;
- }
-
- spin_lock(&hvpipe_src_list_lock);
+ retain_and_null_ptr(src_info);
+ spin_lock_irqsave(&hvpipe_src_list_lock, flags);
/*
* If two processes are executing ioctl() for the same
* source ID concurrently, prevent the second process to
* acquire FD.
*/
if (hvpipe_find_source(srcID)) {
- spin_unlock(&hvpipe_src_list_lock);
- err = -EALREADY;
- goto free_file;
+ spin_unlock_irqrestore(&hvpipe_src_list_lock, flags);
+ return -EALREADY;
}
list_add(&src_info->list, &hvpipe_src_list);
- spin_unlock(&hvpipe_src_list_lock);
-
- fd_install(fd, file);
- return fd;
-
-free_file:
- fput(file);
-free_fd:
- put_unused_fd(fd);
-free_buf:
- kfree(src_info);
- return err;
+ spin_unlock_irqrestore(&hvpipe_src_list_lock, flags);
+ return fd_publish(fdf);
}
/*
diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c
index db99725e752b..14cd0634eeca 100644
--- a/arch/powerpc/platforms/pseries/smp.c
+++ b/arch/powerpc/platforms/pseries/smp.c
@@ -194,10 +194,12 @@ static int pseries_cause_nmi_ipi(int cpu)
static __init void pSeries_smp_probe(void)
{
- if (xive_enabled())
- xive_smp_probe();
- else
+ if (xive_enabled()) {
+ if (xive_smp_probe() < 0)
+ return;
+ } else {
xics_smp_probe();
+ }
/* No doorbell facility, must use the interrupt controller for IPIs */
if (!cpu_has_feature(CPU_FTR_DBELL))
diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
index 6bc373cfef8c..b74cee368c04 100644
--- a/arch/powerpc/sysdev/xive/common.c
+++ b/arch/powerpc/sysdev/xive/common.c
@@ -1274,15 +1274,19 @@ noinstr static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
xive_ops->put_ipi(cpu, xc);
}
-void __init xive_smp_probe(void)
+int __init xive_smp_probe(void)
{
+ int ret;
+
smp_ops->cause_ipi = xive_cause_ipi;
/* Register the IPI */
- xive_init_ipis();
+ ret = xive_init_ipis();
+ if (ret < 0)
+ return ret;
/* Allocate and setup IPI for the boot CPU */
- xive_setup_cpu_ipi(smp_processor_id());
+ return xive_setup_cpu_ipi(smp_processor_id());
}
#endif /* CONFIG_SMP */
diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata
index aca9b0cfcfec..6f230aebd496 100644
--- a/arch/riscv/Kconfig.errata
+++ b/arch/riscv/Kconfig.errata
@@ -145,6 +145,7 @@ config ERRATA_THEAD_PMU
config ERRATA_THEAD_GHOSTWRITE
bool "Apply T-Head Ghostwrite errata"
depends on ERRATA_THEAD && RISCV_ISA_XTHEADVECTOR
+ select RISCV_ALTERNATIVE_EARLY
default y
help
The T-Head C9xx cores have a vulnerability in the xtheadvector
diff --git a/arch/riscv/include/asm/futex.h b/arch/riscv/include/asm/futex.h
index 90c86b115e00..6f3bc9bdb85e 100644
--- a/arch/riscv/include/asm/futex.h
+++ b/arch/riscv/include/asm/futex.h
@@ -40,6 +40,7 @@ arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
if (!access_ok(uaddr, sizeof(u32)))
return -EFAULT;
+ uaddr = untagged_addr(uaddr);
switch (op) {
case FUTEX_OP_SET:
@@ -82,6 +83,7 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
if (!access_ok(uaddr, sizeof(u32)))
return -EFAULT;
+ uaddr = untagged_addr(uaddr);
__enable_user_access();
__asm__ __volatile__ (
diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h
index ab7714aa93bd..016108a28e2d 100644
--- a/arch/riscv/include/asm/scs.h
+++ b/arch/riscv/include/asm/scs.h
@@ -4,6 +4,7 @@
#ifdef __ASSEMBLER__
#include <asm/asm-offsets.h>
+#include <asm/csr.h>
#ifdef CONFIG_SHADOW_CALL_STACK
@@ -23,9 +24,17 @@
REG_L gp, TASK_TI_SCS_SP(tp)
.endm
-/* Load task_scs_sp(current) to gp, but only if tp has changed. */
-.macro scs_load_current_if_task_changed prev
- beq \prev, tp, _skip_scs
+/*
+ * Load task_scs_sp(current) to gp, but only when the trap came from U-mode.
+ * The source privilege level is taken from the saved sstatus.SPP bit (written
+ * by hardware on trap entry), not by comparing tp. tp is a user-writable
+ * register, so gating the shadow call stack reload on it let a user task that
+ * set tp == ¤t skip the reload and run the kernel with an attacker
+ * controlled gp (the shadow call stack pointer).
+ */
+.macro scs_load_current_if_from_user status, tmp
+ andi \tmp, \status, SR_SPP
+ bnez \tmp, _skip_scs
scs_load_current
_skip_scs:
.endm
@@ -43,7 +52,7 @@
.endm
.macro scs_load_current
.endm
-.macro scs_load_current_if_task_changed prev
+.macro scs_load_current_if_from_user status, tmp
.endm
.macro scs_save_current
.endm
diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h
index b61786d43c20..c5c7bc455cb2 100644
--- a/arch/riscv/include/asm/vector.h
+++ b/arch/riscv/include/asm/vector.h
@@ -131,8 +131,8 @@ static __always_inline void __vstate_csr_save(struct __riscv_v_ext_state *dest)
"csrr %0, " __stringify(CSR_VSTART) "\n\t"
"csrr %1, " __stringify(CSR_VTYPE) "\n\t"
"csrr %2, " __stringify(CSR_VL) "\n\t"
- : "=r" (dest->vstart), "=r" (dest->vtype), "=r" (dest->vl),
- "=r" (dest->vcsr) : :);
+ : "=r" (dest->vstart), "=r" (dest->vtype), "=r" (dest->vl)
+ : :);
if (has_xtheadvector()) {
unsigned long status;
@@ -224,7 +224,7 @@ static inline void __riscv_v_vstate_save(struct __riscv_v_ext_state *save_to,
"add %1, %1, %0\n\t"
"vse8.v v24, (%1)\n\t"
".option pop\n\t"
- : "=&r" (vl) : "r" (datap) : "memory");
+ : "=&r" (vl), "+r" (datap) : : "memory");
}
riscv_v_disable();
}
@@ -260,7 +260,7 @@ static inline void __riscv_v_vstate_restore(struct __riscv_v_ext_state *restore_
"add %1, %1, %0\n\t"
"vle8.v v24, (%1)\n\t"
".option pop\n\t"
- : "=&r" (vl) : "r" (datap) : "memory");
+ : "=&r" (vl), "+r" (datap) : : "memory");
}
__vstate_csr_restore(restore_from);
riscv_v_disable();
diff --git a/arch/riscv/include/uapi/asm/vendor/mips.h b/arch/riscv/include/uapi/asm/vendor/mips.h
index e65ab268b265..a31c23afe74f 100644
--- a/arch/riscv/include/uapi/asm/vendor/mips.h
+++ b/arch/riscv/include/uapi/asm/vendor/mips.h
@@ -1,3 +1,5 @@
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#define RISCV_HWPROBE_VENDOR_EXT_XMIPSEXECTL BIT(0)
+#include <linux/const.h>
+
+#define RISCV_HWPROBE_VENDOR_EXT_XMIPSEXECTL _BITULL(0)
diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
index 7549f03ef18f..78f6ea7ba9a7 100644
--- a/arch/riscv/kernel/entry.S
+++ b/arch/riscv/kernel/entry.S
@@ -174,7 +174,7 @@ SYM_CODE_START(handle_exception)
load_global_pointer
/* Load the kernel shadow call stack pointer if coming from userspace */
- scs_load_current_if_task_changed s5
+ scs_load_current_if_from_user s1, s5
#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
move a0, sp
diff --git a/arch/riscv/kvm/vcpu_pmu.c b/arch/riscv/kvm/vcpu_pmu.c
index b374d92c9aa6..be0c7ecc38e7 100644
--- a/arch/riscv/kvm/vcpu_pmu.c
+++ b/arch/riscv/kvm/vcpu_pmu.c
@@ -10,7 +10,10 @@
#include <linux/errno.h>
#include <linux/err.h>
#include <linux/kvm_host.h>
+#include <linux/nospec.h>
+#include <linux/overflow.h>
#include <linux/perf/riscv_pmu.h>
+#include <linux/slab.h>
#include <asm/csr.h>
#include <asm/kvm_vcpu_sbi.h>
#include <asm/kvm_vcpu_pmu.h>
@@ -87,7 +90,8 @@ static void kvm_pmu_release_perf_event(struct kvm_pmc *pmc)
static u64 kvm_pmu_get_perf_event_hw_config(u32 sbi_event_code)
{
- return hw_event_perf_map[sbi_event_code];
+ return hw_event_perf_map[array_index_nospec(sbi_event_code,
+ SBI_PMU_HW_GENERAL_MAX)];
}
static u64 kvm_pmu_get_perf_event_cache_config(u32 sbi_event_code)
@@ -218,6 +222,7 @@ static int pmu_fw_ctr_read_hi(struct kvm_vcpu *vcpu, unsigned long cidx,
return -EINVAL;
}
+ cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS);
pmc = &kvpmu->pmc[cidx];
if (pmc->cinfo.type != SBI_PMU_CTR_TYPE_FW)
@@ -244,6 +249,7 @@ static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
return -EINVAL;
}
+ cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS);
pmc = &kvpmu->pmc[cidx];
if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
@@ -457,13 +463,14 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
unsigned long flags, struct kvm_vcpu_sbi_return *retdata)
{
struct riscv_pmu_event_info *einfo = NULL;
- int shmem_size = num_events * sizeof(*einfo);
+ size_t shmem_size;
gpa_t shmem;
u32 eidx, etype;
u64 econfig;
int ret;
- if (flags != 0 || (saddr_low & (SZ_16 - 1) || num_events == 0)) {
+ if (flags != 0 || (saddr_low & (SZ_16 - 1)) || num_events == 0 ||
+ check_mul_overflow(num_events, sizeof(*einfo), &shmem_size)) {
ret = SBI_ERR_INVALID_PARAM;
goto out;
}
@@ -478,7 +485,8 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
}
}
- einfo = kzalloc(shmem_size, GFP_KERNEL);
+ einfo = kvcalloc(num_events, sizeof(*einfo),
+ GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
if (!einfo) {
ret = SBI_ERR_FAILURE;
goto out;
@@ -490,7 +498,7 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
goto free_mem;
}
- for (int i = 0; i < num_events; i++) {
+ for (unsigned long i = 0; i < num_events; i++) {
eidx = einfo[i].event_idx;
etype = kvm_pmu_get_perf_event_type(eidx);
econfig = kvm_pmu_get_perf_event_config(eidx, einfo[i].event_data);
@@ -506,7 +514,7 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
ret = 0;
free_mem:
- kfree(einfo);
+ kvfree(einfo);
out:
retdata->err_val = ret;
@@ -533,6 +541,7 @@ int kvm_riscv_vcpu_pmu_ctr_info(struct kvm_vcpu *vcpu, unsigned long cidx,
return 0;
}
+ cidx = array_index_nospec(cidx, RISCV_KVM_MAX_COUNTERS);
retdata->out_val = kvpmu->pmc[cidx].cinfo.value;
return 0;
@@ -567,7 +576,8 @@ int kvm_riscv_vcpu_pmu_ctr_start(struct kvm_vcpu *vcpu, unsigned long ctr_base,
}
/* Start the counters that have been configured and requested by the guest */
for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
- pmc_index = i + ctr_base;
+ pmc_index = array_index_nospec(i + ctr_base,
+ RISCV_KVM_MAX_COUNTERS);
if (!test_bit(pmc_index, kvpmu->pmc_in_use))
continue;
/* The guest started the counter again. Reset the overflow status */
@@ -637,7 +647,8 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
/* Stop the counters that have been configured and requested by the guest */
for_each_set_bit(i, &ctr_mask, RISCV_MAX_COUNTERS) {
- pmc_index = i + ctr_base;
+ pmc_index = array_index_nospec(i + ctr_base,
+ RISCV_KVM_MAX_COUNTERS);
if (!test_bit(pmc_index, kvpmu->pmc_in_use))
continue;
pmc = &kvpmu->pmc[pmc_index];
@@ -769,6 +780,7 @@ int kvm_riscv_vcpu_pmu_ctr_cfg_match(struct kvm_vcpu *vcpu, unsigned long ctr_ba
}
}
+ ctr_idx = array_index_nospec(ctr_idx, RISCV_KVM_MAX_COUNTERS);
pmc = &kvpmu->pmc[ctr_idx];
pmc->idx = ctr_idx;
diff --git a/arch/riscv/mm/extable.c b/arch/riscv/mm/extable.c
index dd1530af3ef1..e252eba55ede 100644
--- a/arch/riscv/mm/extable.c
+++ b/arch/riscv/mm/extable.c
@@ -68,8 +68,8 @@ ex_handler_load_unaligned_zeropad(const struct exception_table_entry *ex,
addr = regs_get_gpr(regs, reg_addr * sizeof(unsigned long));
- offset = addr & 0x7UL;
- addr &= ~0x7UL;
+ offset = addr & (sizeof(data) - 1);
+ addr &= ~(sizeof(data) - 1);
data = *(unsigned long *)addr >> (offset * 8);
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 4091a776e37a..f4c85555b7e4 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -458,6 +458,8 @@
#define X86_FEATURE_NO_NESTED_DATA_BP (20*32+ 0) /* No Nested Data Breakpoints */
#define X86_FEATURE_WRMSR_XX_BASE_NS (20*32+ 1) /* WRMSR to {FS,GS,KERNEL_GS}_BASE is non-serializing */
#define X86_FEATURE_LFENCE_RDTSC (20*32+ 2) /* LFENCE always serializing / synchronizes RDTSC */
+
+#define X86_FEATURE_TLBSYNC_SFW_NO (20*32+ 4) /* TLBI Erratum #1718 fixed in microcode */
#define X86_FEATURE_VERW_CLEAR (20*32+ 5) /* The memory form of VERW mitigates TSA */
#define X86_FEATURE_NULL_SEL_CLR_BASE (20*32+ 6) /* Null Selector Clears Base */
@@ -556,4 +558,6 @@
#define X86_BUG_ITS_NATIVE_ONLY X86_BUG( 1*32+ 8) /* "its_native_only" CPU is affected by ITS, VMX is not affected */
#define X86_BUG_TSA X86_BUG( 1*32+ 9) /* "tsa" CPU is affected by Transient Scheduler Attacks */
#define X86_BUG_VMSCAPE X86_BUG( 1*32+10) /* "vmscape" CPU is affected by VMSCAPE attacks from guests */
+
+#define X86_BUG_TLBI X86_BUG( 1*32+12) /* "tlbi" AMD TLBI Erratum #1718 */
#endif /* _ASM_X86_CPUFEATURES_H */
diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index fdf178443f85..617c5a44a96d 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -133,6 +133,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)
KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from)
KVM_X86_OP_OPTIONAL(vm_move_enc_context_from)
KVM_X86_OP_OPTIONAL(guest_memory_reclaimed)
+KVM_X86_OP_OPTIONAL(reload_vmsa)
KVM_X86_OP(get_feature_msr)
KVM_X86_OP(check_emulate_instruction)
KVM_X86_OP(apic_init_signal_blocked)
@@ -147,6 +148,9 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level)
KVM_X86_OP_OPTIONAL(gmem_invalidate)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
+#endif
#undef KVM_X86_OP
#undef KVM_X86_OP_OPTIONAL
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index b2cf3431e259..334962398566 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -127,6 +127,8 @@
KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQ_HV_TLB_FLUSH \
KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+#define KVM_REQ_VMSA_PAGE_RELOAD \
+ KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \
KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
@@ -1926,6 +1928,7 @@ struct kvm_x86_ops {
int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
int (*vm_move_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
void (*guest_memory_reclaimed)(struct kvm *kvm);
+ void (*reload_vmsa)(struct kvm_vcpu *vcpu);
int (*get_feature_msr)(u32 msr, u64 *data);
@@ -1950,6 +1953,9 @@ struct kvm_x86_ops {
void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+ void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
+#endif
int (*gmem_max_mapping_level)(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
};
diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h
index 866ea78ba156..eba925c85f52 100644
--- a/arch/x86/include/asm/tlb.h
+++ b/arch/x86/include/asm/tlb.h
@@ -110,6 +110,12 @@ static inline void __tlbsync(void)
/* TLBSYNC: supported in binutils >= 0.36. */
asm volatile(".byte 0x0f, 0x01, 0xff" ::: "memory");
+
+ if (!static_cpu_has_bug(X86_BUG_TLBI))
+ return;
+
+ /* TLBSYNC: supported in binutils >= 0.36. */
+ asm volatile(".byte 0x0f, 0x01, 0xff" ::: "memory");
}
#else
/* Some compilers (I'm looking at you clang!) simply can't do DCE */
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index b8efe8691b2a..ca4265a1f4bb 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -1050,6 +1050,10 @@ static void init_amd_zen5(struct cpuinfo_x86 *c)
msr_clear_bit(MSR_AMD64_CPUID_FN_7, 18);
pr_emerg_once("RDSEED32 is broken. Disabling the corresponding CPUID bit.\n");
}
+
+ if (cpu_has(c, X86_FEATURE_INVLPGB) &&
+ !cpu_has(c, X86_FEATURE_TLBSYNC_SFW_NO))
+ set_cpu_bug(c, X86_BUG_TLBI);
}
static void init_amd(struct cpuinfo_x86 *c)
diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index ae347e383785..cff725275217 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -276,15 +276,9 @@ static bool is_prefix_bad(struct insn *insn)
return false;
}
-static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool x86_64)
+static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn)
{
- enum insn_mode m = x86_64 ? INSN_MODE_64 : INSN_MODE_32;
u32 volatile *good_insns;
- int ret;
-
- ret = insn_decode(insn, auprobe->insn, sizeof(auprobe->insn), m);
- if (ret < 0)
- return -ENOEXEC;
if (is_prefix_bad(insn))
return -ENOTSUPP;
@@ -293,7 +287,7 @@ static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool
if (insn_masking_exception(insn))
return -ENOTSUPP;
- if (x86_64)
+ if (insn->x86_64)
good_insns = good_insns_64;
else
good_insns = good_insns_32;
@@ -636,9 +630,29 @@ struct uprobe_trampoline {
unsigned long vaddr;
};
+
+#define LEA_INSN_SIZE 5
+#define OPT_INSN_SIZE (LEA_INSN_SIZE + CALL_INSN_SIZE)
+#define REDZONE_SIZE 0x80
+
+static const u8 lea_rsp[] = { 0x48, 0x8d, 0x64, 0x24, 0x80 };
+
+static bool is_opt_insns(const uprobe_opcode_t *insn)
+{
+ return !memcmp(insn, lea_rsp, LEA_INSN_SIZE) &&
+ insn[LEA_INSN_SIZE] == CALL_INSN_OPCODE;
+}
+
+static bool is_swbp_opt_insns(uprobe_opcode_t *insn)
+{
+ return is_swbp_insn(&insn[0]) &&
+ !memcmp(&insn[1], &lea_rsp[1], LEA_INSN_SIZE - 1) &&
+ insn[LEA_INSN_SIZE] == CALL_INSN_OPCODE;
+}
+
static bool is_reachable_by_call(unsigned long vtramp, unsigned long vaddr)
{
- long delta = (long)(vaddr + 5 - vtramp);
+ long delta = (long)(vaddr + OPT_INSN_SIZE - vtramp);
return delta >= INT_MIN && delta <= INT_MAX;
}
@@ -651,7 +665,7 @@ static unsigned long find_nearest_trampoline(unsigned long vaddr)
};
unsigned long low_limit, high_limit;
unsigned long low_tramp, high_tramp;
- unsigned long call_end = vaddr + 5;
+ unsigned long call_end = vaddr + OPT_INSN_SIZE;
if (check_add_overflow(call_end, INT_MIN, &low_limit))
low_limit = PAGE_SIZE;
@@ -810,7 +824,7 @@ SYSCALL_DEFINE0(uprobe)
/* Allow execution only from uprobe trampolines. */
if (!in_uprobe_trampoline(regs->ip))
- return -ENXIO;
+ return -EPROTO;
err = copy_from_user(&args, (void __user *)regs->sp, sizeof(args));
if (err)
@@ -826,8 +840,8 @@ SYSCALL_DEFINE0(uprobe)
regs->ax = args.ax;
regs->r11 = args.r11;
regs->cx = args.cx;
- regs->ip = args.retaddr - 5;
- regs->sp += sizeof(args);
+ regs->ip = args.retaddr - OPT_INSN_SIZE;
+ regs->sp += sizeof(args) + REDZONE_SIZE;
regs->orig_ax = -1;
sp = regs->sp;
@@ -844,12 +858,12 @@ SYSCALL_DEFINE0(uprobe)
*/
if (regs->sp != sp) {
/* skip the trampoline call */
- if (args.retaddr - 5 == regs->ip)
- regs->ip += 5;
+ if (args.retaddr - OPT_INSN_SIZE == regs->ip)
+ regs->ip += OPT_INSN_SIZE;
return regs->ax;
}
- regs->sp -= sizeof(args);
+ regs->sp -= sizeof(args) + REDZONE_SIZE;
/* for the case uprobe_consumer has changed ax/r11/cx */
args.ax = regs->ax;
@@ -857,7 +871,7 @@ SYSCALL_DEFINE0(uprobe)
args.cx = regs->cx;
/* keep return address unless we are instructed otherwise */
- if (args.retaddr - 5 != regs->ip)
+ if (args.retaddr - OPT_INSN_SIZE != regs->ip)
args.retaddr = regs->ip;
if (shstk_push(args.retaddr) == -EFAULT)
@@ -891,7 +905,7 @@ asm (
"pop %rax\n"
"pop %r11\n"
"pop %rcx\n"
- "ret\n"
+ "ret $" __stringify(REDZONE_SIZE) "\n"
"int3\n"
".balign " __stringify(PAGE_SIZE) "\n"
".popsection\n"
@@ -909,7 +923,8 @@ late_initcall(arch_uprobes_init);
enum {
EXPECT_SWBP,
- EXPECT_CALL,
+ EXPECT_OPTIMIZED,
+ EXPECT_SWBP_OPTIMIZED,
};
struct write_opcode_ctx {
@@ -917,30 +932,29 @@ struct write_opcode_ctx {
int expect;
};
-static int is_call_insn(uprobe_opcode_t *insn)
-{
- return *insn == CALL_INSN_OPCODE;
-}
-
/*
- * Verification callback used by int3_update uprobe_write calls to make sure
- * the underlying instruction is as expected - either int3 or call.
+ * Verification callback used by uprobe_write calls to make sure the underlying
+ * instruction is in the expected stage of the INT3 update sequence.
*/
static int verify_insn(struct page *page, unsigned long vaddr, uprobe_opcode_t *new_opcode,
int nbytes, void *data)
{
struct write_opcode_ctx *ctx = data;
- uprobe_opcode_t old_opcode[5];
+ uprobe_opcode_t old_opcode[OPT_INSN_SIZE];
- uprobe_copy_from_page(page, ctx->base, (uprobe_opcode_t *) &old_opcode, 5);
+ uprobe_copy_from_page(page, ctx->base, old_opcode, OPT_INSN_SIZE);
switch (ctx->expect) {
case EXPECT_SWBP:
if (is_swbp_insn(&old_opcode[0]))
return 1;
break;
- case EXPECT_CALL:
- if (is_call_insn(&old_opcode[0]))
+ case EXPECT_OPTIMIZED:
+ if (is_opt_insns(&old_opcode[0]))
+ return 1;
+ break;
+ case EXPECT_SWBP_OPTIMIZED:
+ if (is_swbp_opt_insns(&old_opcode[0]))
return 1;
break;
}
@@ -949,48 +963,122 @@ static int verify_insn(struct page *page, unsigned long vaddr, uprobe_opcode_t *
}
/*
- * Modify multi-byte instructions by using INT3 breakpoints on SMP.
+ * Modify the optimized instruction by using INT3 breakpoints on SMP.
* We completely avoid using stop_machine() here, and achieve the
* synchronization using INT3 breakpoints and SMP cross-calls.
* (borrowed comment from smp_text_poke_batch_finish)
*
- * The way it is done:
- * - Add an INT3 trap to the address that will be patched
- * - SMP sync all CPUs
- * - Update all but the first byte of the patched range
- * - SMP sync all CPUs
- * - Replace the first byte (INT3) by the first byte of the replacing opcode
- * - SMP sync all CPUs
+ * For optimization (int3_update_optimize):
+ * 1) Start with the uprobe INT3 trap already installed
+ * 2) Update everything but the first byte
+ * 3) Replace the first INT3 by the first byte of the LEA instruction
+ *
+ * For unoptimization (int3_update_unoptimize):
+ * 1) Start with the optimized uprobe lea/call instructions
+ * 2) Add an INT3 trap to the address that will be patched
+ * 3) Restore the NOP bytes before the call opcode
+ * 4) Replace the first INT3 by the first byte of the NOP instruction
+ *
+ * Note that unoptimization deliberately keeps the call opcode and displacement
+ * in bytes 5..9. Those bytes become operands of the restored 10-byte NOP.
+ *
+ * Since there is only a single target uprobe-trampoline for the given nop10
+ * instruction address, the CALL instruction will not be changed across
+ * unoptimization/optimization cycles.
+ * Therefore, any task that is preempted at the CALL instruction is guaranteed
+ * to observe that CALL and not anything else.
*/
-static int int3_update(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
- unsigned long vaddr, char *insn, bool optimize)
+static int int3_update_optimize(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
+ unsigned long vaddr, uprobe_opcode_t *insn)
{
- uprobe_opcode_t int3 = UPROBE_SWBP_INSN;
struct write_opcode_ctx ctx = {
.base = vaddr,
};
int err;
/*
- * Write int3 trap.
+ * 1) Initial state after set_swbp() installed the uprobe:
+ * cc 2e 0f 1f 84 00 00 00 00 00
*
- * The swbp_optimize path comes with breakpoint already installed,
- * so we can skip this step for optimize == true.
+ * After a previous unoptimization bytes 5..9 may still contain the
+ * old call instruction, which remains valid for threads already there.
*/
- if (!optimize) {
- ctx.expect = EXPECT_CALL;
- err = uprobe_write(auprobe, vma, vaddr, &int3, 1, verify_insn,
- true /* is_register */, false /* do_update_ref_ctr */,
- &ctx);
- if (err)
- return err;
- }
+ smp_text_poke_sync_each_cpu();
+
+ /*
+ * 2) Rewrite the LEA tail and call displacement:
+ * cc [8d 64 24 80 e8 d0 d1 d2 d3]
+ */
+ ctx.expect = EXPECT_SWBP;
+ err = uprobe_write(auprobe, vma, vaddr + 1, insn + 1,
+ OPT_INSN_SIZE - 1, verify_insn,
+ true /* is_register */, false /* do_update_ref_ctr */,
+ &ctx);
+ if (err)
+ return err;
+
+ smp_text_poke_sync_each_cpu();
+
+ /*
+ * 3) Publish the first LEA byte:
+ * [48] 8d 64 24 80 e8 d0 d1 d2 d3
+ *
+ * From offset 0 this is:
+ * lea -0x80(%rsp), %rsp
+ * call <uprobe-trampoline>
+ */
+ ctx.expect = EXPECT_SWBP_OPTIMIZED;
+ err = uprobe_write(auprobe, vma, vaddr, insn, 1, verify_insn,
+ true /* is_register */, false /* do_update_ref_ctr */,
+ &ctx);
+ if (err)
+ goto error;
smp_text_poke_sync_each_cpu();
+ return 0;
- /* Write all but the first byte of the patched range. */
+error:
+ /*
+ * In all intermediate states byte 0 is INT3, so EXPECT_SWBP covers every
+ * case. Restore NOP bytes 1..4, but keep the valid CALL at bytes 5..9
+ * for a thread that had already executed the LEA before a previous
+ * unoptimization.
+ */
ctx.expect = EXPECT_SWBP;
- err = uprobe_write(auprobe, vma, vaddr + 1, insn + 1, 4, verify_insn,
+ uprobe_write(auprobe, vma, vaddr + 1, auprobe->insn + 1,
+ LEA_INSN_SIZE - 1, verify_insn, true, false, &ctx);
+ smp_text_poke_sync_each_cpu();
+ return err;
+}
+
+static int int3_update_unoptimize(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
+ unsigned long vaddr, uprobe_opcode_t *insn)
+{
+ uprobe_opcode_t int3 = UPROBE_SWBP_INSN;
+ struct write_opcode_ctx ctx = {
+ .base = vaddr,
+ .expect = EXPECT_OPTIMIZED,
+ };
+ int err;
+
+ /*
+ * Note the first two uprobe_write calls use is_register=true, because they
+ * are intermediate patching states while the probe is still active, so
+ * we force the exclusive anonymous page for the update.
+ * Also we use do_update_ref_ctr=false because refctr was already updated by
+ * the initial int3 install.
+ *
+ * The last uprobe_write to nop10 instruction is called with is_register=false
+ * and do_update_ref_ctr=true to trigger the refctr update and to instruct
+ * uprobe_write to zap the anonymous page if it now matches the file page.
+ *
+ * 1) Initial optimized state:
+ * 48 8d 64 24 80 e8 d0 d1 d2 d3
+ *
+ * 2) Trap new entries before restoring the NOP bytes:
+ * [cc] 8d 64 24 80 e8 d0 d1 d2 d3
+ */
+ err = uprobe_write(auprobe, vma, vaddr, &int3, 1, verify_insn,
true /* is_register */, false /* do_update_ref_ctr */,
&ctx);
if (err)
@@ -999,13 +1087,31 @@ static int int3_update(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
smp_text_poke_sync_each_cpu();
/*
- * Write first byte.
+ * 3) Restore bytes 1..4 of the original NOP while keeping byte 0 trapped
+ * and byte 5 as CALL:
+ * cc [2e 0f 1f 84] e8 d0 d1 d2 d3
+ */
+ ctx.expect = EXPECT_SWBP_OPTIMIZED;
+ err = uprobe_write(auprobe, vma, vaddr + 1, insn + 1,
+ LEA_INSN_SIZE - 1, verify_insn,
+ true /* is_register */, false /* do_update_ref_ctr */,
+ &ctx);
+ if (err)
+ return err;
+
+ smp_text_poke_sync_each_cpu();
+
+ /*
+ * 4) Publish the first byte of the original NOP:
+ * [66] 2e 0f 1f 84 e8 d0 d1 d2 d3
*
- * The swbp_unoptimize needs to finish uprobe removal together
- * with ref_ctr update, using uprobe_write with proper flags.
+ * From offset 0 this is the restored 10-byte NOP; the CALL opcode and
+ * displacement are now only NOP operands. Offset 5 still decodes as
+ * CALL for a thread that was already there.
*/
+ ctx.expect = EXPECT_SWBP;
err = uprobe_write(auprobe, vma, vaddr, insn, 1, verify_insn,
- optimize /* is_register */, !optimize /* do_update_ref_ctr */,
+ false /* is_register */, true /* do_update_ref_ctr */,
&ctx);
if (err)
return err;
@@ -1017,17 +1123,25 @@ static int int3_update(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
static int swbp_optimize(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
unsigned long vaddr, unsigned long tramp)
{
- u8 call[5];
+ u8 insn[OPT_INSN_SIZE], *call = &insn[LEA_INSN_SIZE];
- __text_gen_insn(call, CALL_INSN_OPCODE, (const void *) vaddr,
+ /*
+ * We have nop10 instruction (with first byte overwritten to int3),
+ * changing it to:
+ * lea -0x80(%rsp), %rsp
+ * call tramp
+ */
+ memcpy(insn, lea_rsp, LEA_INSN_SIZE);
+ __text_gen_insn(call, CALL_INSN_OPCODE,
+ (const void *) (vaddr + LEA_INSN_SIZE),
(const void *) tramp, CALL_INSN_SIZE);
- return int3_update(auprobe, vma, vaddr, call, true /* optimize */);
+ return int3_update_optimize(auprobe, vma, vaddr, insn);
}
static int swbp_unoptimize(struct arch_uprobe *auprobe, struct vm_area_struct *vma,
unsigned long vaddr)
{
- return int3_update(auprobe, vma, vaddr, auprobe->insn, false /* optimize */);
+ return int3_update_unoptimize(auprobe, vma, vaddr, auprobe->insn);
}
static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst, int len)
@@ -1049,19 +1163,19 @@ static bool __is_optimized(struct mm_struct *mm, uprobe_opcode_t *insn, unsigned
struct __packed __arch_relative_insn {
u8 op;
s32 raddr;
- } *call = (struct __arch_relative_insn *) insn;
+ } *call = (struct __arch_relative_insn *)(insn + LEA_INSN_SIZE);
- if (!is_call_insn(insn))
+ if (!is_opt_insns(insn))
return false;
- return __in_uprobe_trampoline(mm, vaddr + 5 + call->raddr);
+ return __in_uprobe_trampoline(mm, vaddr + OPT_INSN_SIZE + call->raddr);
}
static int is_optimized(struct mm_struct *mm, unsigned long vaddr)
{
- uprobe_opcode_t insn[5];
+ uprobe_opcode_t insn[OPT_INSN_SIZE];
int err;
- err = copy_from_vaddr(mm, vaddr, &insn, 5);
+ err = copy_from_vaddr(mm, vaddr, &insn, OPT_INSN_SIZE);
if (err)
return err;
return __is_optimized(mm, (uprobe_opcode_t *)&insn, vaddr);
@@ -1131,7 +1245,7 @@ static int __arch_uprobe_optimize(struct arch_uprobe *auprobe, struct mm_struct
void arch_uprobe_optimize(struct arch_uprobe *auprobe, unsigned long vaddr)
{
struct mm_struct *mm = current->mm;
- uprobe_opcode_t insn[5];
+ uprobe_opcode_t insn[OPT_INSN_SIZE];
if (!should_optimize(auprobe))
return;
@@ -1142,7 +1256,7 @@ void arch_uprobe_optimize(struct arch_uprobe *auprobe, unsigned long vaddr)
* Check if some other thread already optimized the uprobe for us,
* if it's the case just go away silently.
*/
- if (copy_from_vaddr(mm, vaddr, &insn, 5))
+ if (copy_from_vaddr(mm, vaddr, &insn, OPT_INSN_SIZE))
goto unlock;
if (!is_swbp_insn((uprobe_opcode_t*) &insn))
goto unlock;
@@ -1181,16 +1295,32 @@ static bool insn_is_nopl(struct insn *insn)
return true;
}
+static bool is_optimizable_nop10(struct insn *insn)
+{
+ static const u8 nop10_prefix[] = {
+ 0x66, 0x2e, 0x0f, 0x1f, 0x84
+ };
+
+ /*
+ * Restrict this to the 10-byte NOP form whose last 5 bytes are
+ * SIB/displacement operands. Unoptimization keeps the call opcode and
+ * displacement in those bytes, so other NOP encodings are not safe.
+ */
+ return insn->length == OPT_INSN_SIZE &&
+ (insn_is_nop(insn) || insn_is_nopl(insn)) &&
+ !memcmp(insn->kaddr, nop10_prefix, ARRAY_SIZE(nop10_prefix));
+}
+
static bool can_optimize(struct insn *insn, unsigned long vaddr)
{
- if (!insn->x86_64 || insn->length != 5)
+ if (!insn->x86_64)
return false;
- if (!insn_is_nop(insn) && !insn_is_nopl(insn))
+ if (!is_optimizable_nop10(insn))
return false;
/* We can't do cross page atomic writes yet. */
- return PAGE_SIZE - (vaddr & ~PAGE_MASK) >= 5;
+ return PAGE_SIZE - (vaddr & ~PAGE_MASK) >= OPT_INSN_SIZE;
}
#else /* 32-bit: */
/*
@@ -1577,16 +1707,26 @@ static int push_setup_xol_ops(struct arch_uprobe *auprobe, struct insn *insn)
*/
int arch_uprobe_analyze_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long addr)
{
+ enum insn_mode m = is_64bit_mm(mm) ? INSN_MODE_64 : INSN_MODE_32;
u8 fix_ip_or_call = UPROBE_FIX_IP;
struct insn insn;
int ret;
- ret = uprobe_init_insn(auprobe, &insn, is_64bit_mm(mm));
- if (ret)
- return ret;
+ ret = insn_decode(&insn, auprobe->insn, sizeof(auprobe->insn), m);
+ if (ret < 0)
+ return -ENOEXEC;
- if (can_optimize(&insn, addr))
+ /*
+ * No need to check instruction in uprobe_init_insn in case we
+ * are on top of optimizable nop10.
+ */
+ if (can_optimize(&insn, addr)) {
set_bit(ARCH_UPROBE_FLAG_CAN_OPTIMIZE, &auprobe->flags);
+ } else {
+ ret = uprobe_init_insn(auprobe, &insn);
+ if (ret)
+ return ret;
+ }
ret = branch_setup_xol_ops(auprobe, &insn);
if (ret != -ENOSYS)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 66e18702e5a3..205d314a7583 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -7375,6 +7375,11 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
write_lock(&kvm->mmu_lock);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+ if (slot->gmem.file)
+ kvm_arch_gmem_invalidate_range(kvm, &range);
+#endif
+
if (zap_all) {
__kvm_mmu_zap_all_fast_front_half(kvm);
} else {
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index df54df7ae201..892b6fd48a0d 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1165,12 +1165,11 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu)
return 0;
}
-int nested_svm_vmexit(struct vcpu_svm *svm)
+void nested_svm_vmexit(struct vcpu_svm *svm)
{
struct kvm_vcpu *vcpu = &svm->vcpu;
struct vmcb *vmcb01 = svm->vmcb01.ptr;
struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
- int rc;
if (nested_svm_vmexit_update_vmcb12(vcpu))
kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
@@ -1286,9 +1285,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
nested_svm_uninit_mmu_context(vcpu);
- rc = nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true);
- if (rc)
- return 1;
+ if (nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true))
+ kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
/* Drop tracking for L1->L2 injected NMIs and soft IRQs */
svm->nmi_l1_to_l2 = false;
@@ -1317,8 +1315,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
*/
if (kvm_apicv_activated(vcpu->kvm))
__kvm_vcpu_update_apicv(vcpu);
-
- return 0;
}
static void nested_svm_triple_fault(struct kvm_vcpu *vcpu)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 2b45481f3a9c..582762bec879 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4039,19 +4039,25 @@ static int snp_begin_psc(struct vcpu_svm *svm)
BUG();
}
-static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
+static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
{
struct vcpu_svm *svm = to_svm(vcpu);
struct kvm_memory_slot *slot;
+ struct kvm *kvm = vcpu->kvm;
gfn_t gfn = gpa_to_gfn(gpa);
+ unsigned long mmu_seq;
struct page *page;
kvm_pfn_t pfn;
lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex);
- /* Clear use of the VMSA. */
+ /*
+ * Clear use of the VMSA. Ensure snp_guest_vmsa_gpa is written exactly
+ * once, as it is read locklessly when responding to gfn invalidations.
+ * Pairs with the READ_ONCE() in sev_gmem_invalidate_range().
+ */
svm->vmcb->control.vmsa_pa = INVALID_PAGE;
- svm->sev_es.snp_guest_vmsa_gpa = INVALID_PAGE;
+ WRITE_ONCE(svm->sev_es.snp_guest_vmsa_gpa, INVALID_PAGE);
/*
* When replacing the VMSA during SEV-SNP AP creation,
@@ -4066,6 +4072,9 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
if (!slot)
return;
+ mmu_seq = kvm->mmu_invalidate_seq;
+ smp_rmb();
+
/*
* The new VMSA will be private memory guest memory, so retrieve the
* PFN from the gmem backend.
@@ -4084,15 +4093,20 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
*/
svm->sev_es.snp_has_guest_vmsa = true;
- /* Use the new VMSA */
- svm->sev_es.snp_guest_vmsa_gpa = gpa;
- svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
-
+ read_lock(&kvm->mmu_lock);
/*
- * gmem pages aren't currently migratable, but if this ever changes
- * then care should be taken to ensure svm->sev_es.vmsa is pinned
- * through some other means.
+ * Save the guest-provided GPA. If retry is needed, then KVM will try
+ * again with the same GPA. If the VMSA is usable, then KVM needs to
+ * track the GPA so that the VMSA can be reloaded if the backing page
+ * for the GPA is invalidated.
*/
+ svm->sev_es.snp_guest_vmsa_gpa = gpa;
+ if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
+ kvm_make_request(KVM_REQ_VMSA_PAGE_RELOAD, vcpu);
+ else
+ svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
+ read_unlock(&kvm->mmu_lock);
+
kvm_release_page_clean(page);
}
@@ -4118,7 +4132,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
gpa = svm->sev_es.snp_pending_vmsa_gpa;
svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
- sev_snp_reload_vmsa(vcpu, gpa);
+ __sev_snp_reload_vmsa(vcpu, gpa);
/*
* Mark the vCPU as runnable for CREATE requests, indicated by a valid
@@ -4130,6 +4144,15 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
}
+void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_sev_es_state *sev_es = &to_svm(vcpu)->sev_es;
+
+ guard(mutex)(&sev_es->snp_vmsa_mutex);
+
+ __sev_snp_reload_vmsa(vcpu, sev_es->snp_guest_vmsa_gpa);
+}
+
static int sev_snp_ap_creation(struct vcpu_svm *svm)
{
struct kvm_sev_info *sev = to_kvm_sev_info(svm->vcpu.kvm);
@@ -5166,6 +5189,41 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
}
}
+void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+ struct kvm_vcpu *vcpu;
+ unsigned long i;
+
+ lockdep_assert_held_write(&kvm->mmu_lock);
+
+ /*
+ * An unstable result for "is SNP" is a-ok here, thanks to mmu_lock.
+ * The vCPU's VMSA GPA is invalidated before the vCPU is made visible
+ * to other tasks, and can only become valid while holding mmu_lock,
+ * after the VM is fully committed to being an SNP VM.
+ */
+ if (!sev_snp_guest(kvm))
+ return;
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ /*
+ * Read snp_guest_vmsa_gpa without taking the vCPU's VMSA mutex
+ * (or its generic mutex) as mmu_lock is held, i.e. this task
+ * can't sleep. The VMSA is invalidated outside of mmu_lock,
+ * but can only become valid inside of mmu_lock, i.e. the below
+ * can get false positives, but not false negatives. A false
+ * positive is benign, as a spurious request simply forces the
+ * vCPU to re-establish its VMSA.
+ */
+ gpa_t gpa = READ_ONCE(to_svm(vcpu)->sev_es.snp_guest_vmsa_gpa);
+
+ if (VALID_PAGE(gpa) &&
+ gpa_to_gfn(gpa) >= range->start &&
+ gpa_to_gfn(gpa) < range->end)
+ kvm_make_request_and_kick(KVM_REQ_VMSA_PAGE_RELOAD, vcpu);
+ }
+}
+
int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private)
{
int level, rc;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index ba7b0a67ad29..335b214aa371 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2152,13 +2152,9 @@ static int emulate_svm_instr(struct kvm_vcpu *vcpu, int opcode)
[SVM_INSTR_VMSAVE] = vmsave_interception,
};
struct vcpu_svm *svm = to_svm(vcpu);
- int ret;
if (is_guest_mode(vcpu)) {
- /* Returns '1' or -errno on failure, '0' on success. */
- ret = nested_svm_simple_vmexit(svm, guest_mode_exit_codes[opcode]);
- if (ret)
- return ret;
+ nested_svm_simple_vmexit(svm, guest_mode_exit_codes[opcode]);
return 1;
}
return svm_instr_handlers[opcode](vcpu);
@@ -4787,7 +4783,6 @@ static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram)
{
struct vcpu_svm *svm = to_svm(vcpu);
struct kvm_host_map map_save;
- int ret;
if (!is_guest_mode(vcpu))
return 0;
@@ -4807,9 +4802,7 @@ static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram)
svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP];
svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP];
- ret = nested_svm_simple_vmexit(svm, SVM_EXIT_SW);
- if (ret)
- return ret;
+ nested_svm_simple_vmexit(svm, SVM_EXIT_SW);
/*
* KVM uses VMCB01 to store L1 host state while L2 runs but
@@ -5249,12 +5242,14 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
.mem_enc_register_region = sev_mem_enc_register_region,
.mem_enc_unregister_region = sev_mem_enc_unregister_region,
.guest_memory_reclaimed = sev_guest_memory_reclaimed,
+ .reload_vmsa = sev_snp_reload_vmsa,
.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
.vm_move_enc_context_from = sev_vm_move_enc_context_from,
.gmem_prepare = sev_gmem_prepare,
.gmem_invalidate = sev_gmem_invalidate,
+ .gmem_invalidate_range = sev_gmem_invalidate_range,
.gmem_max_mapping_level = sev_gmem_max_mapping_level,
#endif
.check_emulate_instruction = svm_check_emulate_instruction,
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 85f64d9ebe59..df081ae666f0 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -771,15 +771,15 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu);
void svm_copy_vmrun_state(struct vmcb_save_area *to_save,
struct vmcb_save_area *from_save);
void svm_copy_vmloadsave_state(struct vmcb *to_vmcb, struct vmcb *from_vmcb);
-int nested_svm_vmexit(struct vcpu_svm *svm);
+void nested_svm_vmexit(struct vcpu_svm *svm);
-static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
+static inline void nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
{
svm->vmcb->control.exit_code = exit_code;
svm->vmcb->control.exit_code_hi = 0;
svm->vmcb->control.exit_info_1 = 0;
svm->vmcb->control.exit_info_2 = 0;
- return nested_svm_vmexit(svm);
+ nested_svm_vmexit(svm);
}
int nested_svm_exit_handled(struct vcpu_svm *svm);
@@ -869,6 +869,7 @@ static inline struct page *snp_safe_alloc_page(void)
{
return snp_safe_alloc_page_node(numa_node_id(), GFP_KERNEL_ACCOUNT);
}
+void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu);
int sev_vcpu_create(struct kvm_vcpu *vcpu);
void sev_free_vcpu(struct kvm_vcpu *vcpu);
@@ -882,6 +883,7 @@ extern unsigned int max_sev_asid;
void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ae33393c9b47..907efeed6c91 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10854,6 +10854,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
goto out;
}
}
+ if (kvm_check_request(KVM_REQ_VMSA_PAGE_RELOAD, vcpu))
+ kvm_x86_call(reload_vmsa)(vcpu);
}
if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
@@ -13279,6 +13281,10 @@ void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
{
kvm_x86_call(gmem_invalidate)(start, end);
}
+void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+ kvm_x86_call(gmem_invalidate_range)(kvm, range);
+}
#endif
#endif
diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index 6a75fe1c7a5c..2e7cfaac7b21 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -548,14 +548,21 @@ static void bfq_pd_init(struct blkg_policy_data *pd)
bfqg->rq_pos_tree = RB_ROOT;
}
-static void bfq_pd_free(struct blkg_policy_data *pd)
+static void bfqg_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct bfq_group *bfqg = pd_to_bfqg(pd);
bfqg_stats_exit(&bfqg->stats);
bfqg_put(bfqg);
}
+static void bfq_pd_free(struct blkg_policy_data *pd)
+{
+ call_rcu(&pd->rcu_head, bfqg_release);
+}
+
static void bfq_pd_reset_stats(struct blkg_policy_data *pd)
{
struct bfq_group *bfqg = pd_to_bfqg(pd);
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 1cce3294634d..fd206d1fa3c9 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -140,6 +140,8 @@ struct blkg_policy_data {
struct blkcg_gq *blkg;
int plid;
bool online;
+
+ struct rcu_head rcu_head;
};
/*
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index ecf496912505..d0b13580c6b6 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -3017,9 +3017,20 @@ static void ioc_pd_init(struct blkg_policy_data *pd)
spin_unlock_irqrestore(&ioc->lock, flags);
}
+static void iocg_release(struct rcu_head *rcu)
+{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
+ struct ioc_gq *iocg = pd_to_iocg(pd);
+
+ free_percpu(iocg->pcpu_stat);
+ kfree(iocg);
+}
+
static void ioc_pd_free(struct blkg_policy_data *pd)
{
struct ioc_gq *iocg = pd_to_iocg(pd);
+ struct blkcg_gq *blkg = pd_to_blkg(pd);
struct ioc *ioc = iocg->ioc;
unsigned long flags;
@@ -3041,8 +3052,14 @@ static void ioc_pd_free(struct blkg_policy_data *pd)
hrtimer_cancel(&iocg->waitq_timer);
}
- free_percpu(iocg->pcpu_stat);
- kfree(iocg);
+
+ /* off ->active_iocgs and timer gone, so nothing can re-arm the delay */
+ iocg->delay = 0;
+ iocg->indelay_since = 0;
+ if (blkg)
+ blkcg_clear_delay(blkg);
+
+ call_rcu(&pd->rcu_head, iocg_release);
}
static void ioc_pd_stat(struct blkg_policy_data *pd, struct seq_file *s)
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index 45bd18f68541..17e530b93c48 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -1031,13 +1031,30 @@ static void iolatency_pd_offline(struct blkg_policy_data *pd)
iolatency_clear_scaling(blkg);
}
-static void iolatency_pd_free(struct blkg_policy_data *pd)
+static void iolat_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct iolatency_grp *iolat = pd_to_lat(pd);
+
free_percpu(iolat->stats);
kfree(iolat);
}
+static void iolatency_pd_free(struct blkg_policy_data *pd)
+{
+ struct blkcg_gq *blkg = pd_to_blkg(pd);
+
+ /*
+ * Groups throttled as collateral have min_lat_nsec == 0, so
+ * iolatency_pd_offline() leaves their delay set. Drop it here, where
+ * no in-flight bio can re-arm it via check_scale_change().
+ */
+ if (blkg)
+ blkcg_clear_delay(blkg);
+ call_rcu(&pd->rcu_head, iolat_release);
+}
+
static struct cftype iolatency_files[] = {
{
.name = "latency",
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index c19d052a8f2f..64e78c79a7b4 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -358,16 +358,25 @@ static void throtl_pd_online(struct blkg_policy_data *pd)
tg_update_has_rules(tg);
}
-static void throtl_pd_free(struct blkg_policy_data *pd)
+static void tg_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct throtl_grp *tg = pd_to_tg(pd);
- timer_delete_sync(&tg->service_queue.pending_timer);
blkg_rwstat_exit(&tg->stat_bytes);
blkg_rwstat_exit(&tg->stat_ios);
kfree(tg);
}
+static void throtl_pd_free(struct blkg_policy_data *pd)
+{
+ struct throtl_grp *tg = pd_to_tg(pd);
+
+ timer_delete_sync(&tg->service_queue.pending_timer);
+ call_rcu(&pd->rcu_head, tg_release);
+}
+
static struct throtl_grp *
throtl_rb_first(struct throtl_service_queue *parent_sq)
{
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 3c3cb965fd74..33adc1882276 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -145,7 +145,7 @@ void af_alg_release_parent(struct sock *sk)
}
EXPORT_SYMBOL_GPL(af_alg_release_parent);
-static int alg_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int alg_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
const u32 allowed = CRYPTO_ALG_KERN_DRIVER_ONLY;
struct sock *sk = sock->sk;
diff --git a/crypto/drbg.c b/crypto/drbg.c
index dc8996789da6..51c36cfa8eff 100644
--- a/crypto/drbg.c
+++ b/crypto/drbg.c
@@ -1986,7 +1986,6 @@ static int drbg_kcapi_seed(struct crypto_rng *tfm,
*/
static inline int __init drbg_healthcheck_sanity(void)
{
- int len = 0;
#define OUTBUFLEN 16
unsigned char buf[OUTBUFLEN];
struct drbg_state *drbg = NULL;
@@ -2030,12 +2029,12 @@ static inline int __init drbg_healthcheck_sanity(void)
max_addtllen = drbg_max_addtl(drbg);
max_request_bytes = drbg_max_request_bytes(drbg);
drbg_string_fill(&addtl, buf, max_addtllen + 1);
- /* overflow addtllen with additonal info string */
- len = drbg_generate(drbg, buf, OUTBUFLEN, &addtl);
- BUG_ON(0 < len);
+ /* overflow addtllen with additional info string */
+ ret = drbg_generate(drbg, buf, OUTBUFLEN, &addtl);
+ BUG_ON(ret == 0);
/* overflow max_bits */
- len = drbg_generate(drbg, buf, (max_request_bytes + 1), NULL);
- BUG_ON(0 < len);
+ ret = drbg_generate(drbg, buf, max_request_bytes + 1, NULL);
+ BUG_ON(ret == 0);
/* overflow max addtllen with personalization string */
ret = drbg_seed(drbg, &addtl, false);
diff --git a/drivers/accel/amdxdna/Makefile b/drivers/accel/amdxdna/Makefile
index 6797dac65efa..6344aaf523fa 100644
--- a/drivers/accel/amdxdna/Makefile
+++ b/drivers/accel/amdxdna/Makefile
@@ -14,6 +14,7 @@ amdxdna-y := \
amdxdna_mailbox.o \
amdxdna_mailbox_helper.o \
amdxdna_pci_drv.o \
+ amdxdna_pm.o \
amdxdna_sysfs.o \
amdxdna_ubuf.o \
npu1_regs.o \
diff --git a/drivers/accel/amdxdna/aie2_ctx.c b/drivers/accel/amdxdna/aie2_ctx.c
index 9ec3d90c2d20..a87b24654c36 100644
--- a/drivers/accel/amdxdna/aie2_ctx.c
+++ b/drivers/accel/amdxdna/aie2_ctx.c
@@ -21,6 +21,7 @@
#include "amdxdna_gem.h"
#include "amdxdna_mailbox.h"
#include "amdxdna_pci_drv.h"
+#include "amdxdna_pm.h"
static bool force_cmdlist;
module_param(force_cmdlist, bool, 0600);
@@ -88,7 +89,7 @@ static int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hw
goto out;
}
- ret = aie2_config_cu(hwctx);
+ ret = aie2_config_cu(hwctx, NULL);
if (ret) {
XDNA_ERR(xdna, "Config cu failed, ret %d", ret);
goto out;
@@ -167,14 +168,11 @@ static int aie2_hwctx_resume_cb(struct amdxdna_hwctx *hwctx, void *arg)
int aie2_hwctx_resume(struct amdxdna_client *client)
{
- struct amdxdna_dev *xdna = client->xdna;
-
/*
* The resume path cannot guarantee that mailbox channel can be
* regenerated. If this happen, when submit message to this
* mailbox channel, error will return.
*/
- drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock));
return amdxdna_hwctx_walk(client, NULL, aie2_hwctx_resume_cb);
}
@@ -184,6 +182,8 @@ aie2_sched_notify(struct amdxdna_sched_job *job)
struct dma_fence *fence = job->fence;
trace_xdna_job(&job->base, job->hwctx->name, "signaled fence", job->seq);
+
+ amdxdna_pm_suspend_put(job->hwctx->client->xdna);
job->hwctx->priv->completed++;
dma_fence_signal(fence);
@@ -204,13 +204,13 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size)
cmd_abo = job->cmd_bo;
if (unlikely(job->job_timeout)) {
- amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_TIMEOUT);
+ amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_TIMEOUT);
ret = -EINVAL;
goto out;
}
if (unlikely(!data) || unlikely(size != sizeof(u32))) {
- amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ABORT);
+ amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT);
ret = -EINVAL;
goto out;
}
@@ -220,7 +220,7 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size)
if (status == AIE2_STATUS_SUCCESS)
amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_COMPLETED);
else
- amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ERROR);
+ amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ERROR);
out:
aie2_sched_notify(job);
@@ -264,13 +264,13 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size)
cmd_abo = job->cmd_bo;
if (unlikely(job->job_timeout)) {
- amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_TIMEOUT);
+ amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_TIMEOUT);
ret = -EINVAL;
goto out;
}
if (unlikely(!data) || unlikely(size != sizeof(u32) * 3)) {
- amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ABORT);
+ amdxdna_cmd_set_error(cmd_abo, job, 0, ERT_CMD_STATE_ABORT);
ret = -EINVAL;
goto out;
}
@@ -290,19 +290,12 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size)
fail_cmd_idx, fail_cmd_status);
if (fail_cmd_status == AIE2_STATUS_SUCCESS) {
- amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ABORT);
+ amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ABORT);
ret = -EINVAL;
- goto out;
+ } else {
+ amdxdna_cmd_set_error(cmd_abo, job, fail_cmd_idx, ERT_CMD_STATE_ERROR);
}
- amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ERROR);
-
- if (amdxdna_cmd_get_op(cmd_abo) == ERT_CMD_CHAIN) {
- struct amdxdna_cmd_chain *cc = amdxdna_cmd_get_payload(cmd_abo, NULL);
- cc->error_index = fail_cmd_idx;
- if (cc->error_index >= cc->command_count)
- cc->error_index = 0;
- }
out:
aie2_sched_notify(job);
return ret;
@@ -544,7 +537,7 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
.num_rqs = DRM_SCHED_PRIORITY_COUNT,
.credit_limit = HWCTX_MAX_CMDS,
.timeout = msecs_to_jiffies(HWCTX_MAX_TIMEOUT),
- .name = hwctx->name,
+ .name = "amdxdna_js",
.dev = xdna->ddev.dev,
};
struct drm_gpu_scheduler *sched;
@@ -623,10 +616,14 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
goto free_entity;
}
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret)
+ goto free_col_list;
+
ret = aie2_alloc_resource(hwctx);
if (ret) {
XDNA_ERR(xdna, "Alloc hw resource failed, ret %d", ret);
- goto free_col_list;
+ goto suspend_put;
}
ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id,
@@ -641,6 +638,7 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
XDNA_ERR(xdna, "Create syncobj failed, ret %d", ret);
goto release_resource;
}
+ amdxdna_pm_suspend_put(xdna);
hwctx->status = HWCTX_STAT_INIT;
ndev = xdna->dev_handle;
@@ -653,6 +651,8 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx)
release_resource:
aie2_release_resource(hwctx);
+suspend_put:
+ amdxdna_pm_suspend_put(xdna);
free_col_list:
kfree(hwctx->col_list);
free_entity:
@@ -715,6 +715,14 @@ void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx)
kfree(hwctx->cus);
}
+static int aie2_config_cu_resp_handler(void *handle, void __iomem *data, size_t size)
+{
+ struct amdxdna_hwctx *hwctx = handle;
+
+ amdxdna_pm_suspend_put(hwctx->client->xdna);
+ return 0;
+}
+
static int aie2_hwctx_cu_config(struct amdxdna_hwctx *hwctx, void *buf, u32 size)
{
struct amdxdna_hwctx_param_config_cu *config = buf;
@@ -746,10 +754,14 @@ static int aie2_hwctx_cu_config(struct amdxdna_hwctx *hwctx, void *buf, u32 size
if (!hwctx->cus)
return -ENOMEM;
- ret = aie2_config_cu(hwctx);
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret)
+ goto free_cus;
+
+ ret = aie2_config_cu(hwctx, aie2_config_cu_resp_handler);
if (ret) {
XDNA_ERR(xdna, "Config CU to firmware failed, ret %d", ret);
- goto free_cus;
+ goto pm_suspend_put;
}
wmb(); /* To avoid locking in command submit when check status */
@@ -757,6 +769,8 @@ static int aie2_hwctx_cu_config(struct amdxdna_hwctx *hwctx, void *buf, u32 size
return 0;
+pm_suspend_put:
+ amdxdna_pm_suspend_put(xdna);
free_cus:
kfree(hwctx->cus);
hwctx->cus = NULL;
@@ -824,8 +838,6 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
up_write(&xdna->notifier_lock);
- XDNA_DBG(xdna, "populate memory range %lx %lx",
- mapp->vma->vm_start, mapp->vma->vm_end);
mm = mapp->notifier.mm;
if (!mmget_not_zero(mm)) {
amdxdna_umap_put(mapp);
@@ -899,11 +911,15 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
goto free_chain;
}
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret)
+ goto cleanup_job;
+
retry:
ret = drm_gem_lock_reservations(job->bos, job->bo_cnt, &acquire_ctx);
if (ret) {
XDNA_WARN(xdna, "Failed to lock BOs, ret %d", ret);
- goto cleanup_job;
+ goto suspend_put;
}
for (i = 0; i < job->bo_cnt; i++) {
@@ -911,7 +927,7 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
if (ret) {
XDNA_WARN(xdna, "Failed to reserve fences %d", ret);
drm_gem_unlock_reservations(job->bos, job->bo_cnt, &acquire_ctx);
- goto cleanup_job;
+ goto suspend_put;
}
}
@@ -926,12 +942,12 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
} else if (time_after(jiffies, timeout)) {
ret = -ETIME;
- goto cleanup_job;
+ goto suspend_put;
}
ret = aie2_populate_range(abo);
if (ret)
- goto cleanup_job;
+ goto suspend_put;
goto retry;
}
}
@@ -957,6 +973,8 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
return 0;
+suspend_put:
+ amdxdna_pm_suspend_put(xdna);
cleanup_job:
drm_sched_job_cleanup(&job->base);
free_chain:
diff --git a/drivers/accel/amdxdna/aie2_message.c b/drivers/accel/amdxdna/aie2_message.c
index 9caad083543d..9b3a0706a786 100644
--- a/drivers/accel/amdxdna/aie2_message.c
+++ b/drivers/accel/amdxdna/aie2_message.c
@@ -37,13 +37,9 @@ static int aie2_send_mgmt_msg_wait(struct amdxdna_dev_hdl *ndev,
if (!ndev->mgmt_chann)
return -ENODEV;
- drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock));
ret = xdna_send_msg_wait(xdna, ndev->mgmt_chann, msg);
- if (ret == -ETIME) {
- xdna_mailbox_stop_channel(ndev->mgmt_chann);
- xdna_mailbox_destroy_channel(ndev->mgmt_chann);
- ndev->mgmt_chann = NULL;
- }
+ if (ret == -ETIME)
+ aie2_destroy_mgmt_chann(ndev);
if (!ret && *hdl->data != AIE2_STATUS_SUCCESS) {
XDNA_ERR(xdna, "command opcode 0x%x failed, status 0x%x",
@@ -54,6 +50,16 @@ static int aie2_send_mgmt_msg_wait(struct amdxdna_dev_hdl *ndev,
return ret;
}
+void aie2_destroy_mgmt_chann(struct amdxdna_dev_hdl *ndev)
+{
+ if (!ndev->mgmt_chann)
+ return;
+
+ xdna_mailbox_stop_channel(ndev->mgmt_chann);
+ xdna_mailbox_destroy_channel(ndev->mgmt_chann);
+ ndev->mgmt_chann = NULL;
+}
+
int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev)
{
DECLARE_AIE2_MSG(suspend, MSG_OP_SUSPEND);
@@ -184,6 +190,19 @@ int aie2_query_firmware_version(struct amdxdna_dev_hdl *ndev,
return 0;
}
+static int aie2_destroy_context_req(struct amdxdna_dev_hdl *ndev, u32 id)
+{
+ DECLARE_AIE2_MSG(destroy_ctx, MSG_OP_DESTROY_CONTEXT);
+ struct amdxdna_dev *xdna = ndev->xdna;
+ int ret;
+
+ req.context_id = id;
+ ret = aie2_send_mgmt_msg_wait(ndev, &msg);
+ if (ret)
+ XDNA_WARN(xdna, "Destroy context failed, ret %d", ret);
+
+ return ret;
+}
int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx)
{
DECLARE_AIE2_MSG(create_ctx, MSG_OP_CREATE_CONTEXT);
@@ -206,7 +225,8 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct
return ret;
hwctx->fw_ctx_id = resp.context_id;
- WARN_ONCE(hwctx->fw_ctx_id == -1, "Unexpected context id");
+ if (WARN_ON_ONCE(hwctx->fw_ctx_id == -1))
+ return -EINVAL;
cq_pair = &resp.cq_pair[0];
x2i.mb_head_ptr_reg = AIE2_MBOX_OFF(ndev, cq_pair->x2i_q.head_addr);
@@ -221,50 +241,45 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct
ret = pci_irq_vector(to_pci_dev(xdna->ddev.dev), resp.msix_id);
if (ret == -EINVAL) {
- XDNA_ERR(xdna, "not able to create channel");
- goto out_destroy_context;
+ XDNA_ERR(xdna, "Alloc IRQ failed %d", ret);
+ goto del_ctx_req;
}
intr_reg = i2x.mb_head_ptr_reg + 4;
hwctx->priv->mbox_chann = xdna_mailbox_create_channel(ndev->mbox, &x2i, &i2x,
intr_reg, ret);
if (!hwctx->priv->mbox_chann) {
- XDNA_ERR(xdna, "not able to create channel");
+ XDNA_ERR(xdna, "Not able to create channel");
ret = -EINVAL;
- goto out_destroy_context;
+ goto del_ctx_req;
}
- XDNA_DBG(xdna, "%s mailbox channel irq: %d, msix_id: %d",
- hwctx->name, ret, resp.msix_id);
- XDNA_DBG(xdna, "%s created fw ctx %d pasid %d", hwctx->name,
- hwctx->fw_ctx_id, hwctx->client->pasid);
+ XDNA_DBG(xdna, "Mailbox channel irq: %d, msix_id: %d", ret, resp.msix_id);
+ XDNA_DBG(xdna, "Created fw ctx %d pasid %d", hwctx->fw_ctx_id, hwctx->client->pasid);
return 0;
-out_destroy_context:
- aie2_destroy_context(ndev, hwctx);
+del_ctx_req:
+ aie2_destroy_context_req(ndev, hwctx->fw_ctx_id);
+ hwctx->fw_ctx_id = -1;
return ret;
}
int aie2_destroy_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx)
{
- DECLARE_AIE2_MSG(destroy_ctx, MSG_OP_DESTROY_CONTEXT);
struct amdxdna_dev *xdna = ndev->xdna;
int ret;
if (hwctx->fw_ctx_id == -1)
return 0;
- xdna_mailbox_stop_channel(hwctx->priv->mbox_chann);
-
- req.context_id = hwctx->fw_ctx_id;
- ret = aie2_send_mgmt_msg_wait(ndev, &msg);
- if (ret)
- XDNA_WARN(xdna, "%s destroy context failed, ret %d", hwctx->name, ret);
+ if (!hwctx->priv->mbox_chann)
+ return 0;
+ xdna_mailbox_stop_channel(hwctx->priv->mbox_chann);
+ ret = aie2_destroy_context_req(ndev, hwctx->fw_ctx_id);
xdna_mailbox_destroy_channel(hwctx->priv->mbox_chann);
- XDNA_DBG(xdna, "%s destroyed fw ctx %d", hwctx->name,
- hwctx->fw_ctx_id);
+ XDNA_DBG(xdna, "Destroyed fw ctx %d", hwctx->fw_ctx_id);
hwctx->priv->mbox_chann = NULL;
hwctx->fw_ctx_id = -1;
@@ -377,15 +392,17 @@ int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr,
return xdna_mailbox_send_msg(ndev->mgmt_chann, &msg, TX_TIMEOUT);
}
-int aie2_config_cu(struct amdxdna_hwctx *hwctx)
+int aie2_config_cu(struct amdxdna_hwctx *hwctx,
+ int (*notify_cb)(void *, void __iomem *, size_t))
{
struct mailbox_channel *chann = hwctx->priv->mbox_chann;
struct amdxdna_dev *xdna = hwctx->client->xdna;
u32 shift = xdna->dev_info->dev_mem_buf_shift;
- DECLARE_AIE2_MSG(config_cu, MSG_OP_CONFIG_CU);
+ struct config_cu_req req = { 0 };
+ struct xdna_mailbox_msg msg;
struct drm_gem_object *gobj;
struct amdxdna_gem_obj *abo;
- int ret, i;
+ int i;
if (!chann)
return -ENODEV;
@@ -423,18 +440,12 @@ int aie2_config_cu(struct amdxdna_hwctx *hwctx)
}
req.num_cus = hwctx->cus->num_cus;
- ret = xdna_send_msg_wait(xdna, chann, &msg);
- if (ret == -ETIME)
- aie2_destroy_context(xdna->dev_handle, hwctx);
-
- if (resp.status == AIE2_STATUS_SUCCESS) {
- XDNA_DBG(xdna, "Configure %d CUs, ret %d", req.num_cus, ret);
- return 0;
- }
-
- XDNA_ERR(xdna, "Command opcode 0x%x failed, status 0x%x ret %d",
- msg.opcode, resp.status, ret);
- return ret;
+ msg.send_data = (u8 *)&req;
+ msg.send_size = sizeof(req);
+ msg.handle = hwctx;
+ msg.opcode = MSG_OP_CONFIG_CU;
+ msg.notify_cb = notify_cb;
+ return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT);
}
int aie2_execbuf(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
diff --git a/drivers/accel/amdxdna/aie2_pci.c b/drivers/accel/amdxdna/aie2_pci.c
index 6e07793bbeac..4a7d1feac6a5 100644
--- a/drivers/accel/amdxdna/aie2_pci.c
+++ b/drivers/accel/amdxdna/aie2_pci.c
@@ -26,6 +26,7 @@
#include "amdxdna_gem.h"
#include "amdxdna_mailbox.h"
#include "amdxdna_pci_drv.h"
+#include "amdxdna_pm.h"
static int aie2_max_col = XRS_MAX_COL;
module_param(aie2_max_col, uint, 0600);
@@ -224,15 +225,6 @@ static int aie2_mgmt_fw_init(struct amdxdna_dev_hdl *ndev)
return ret;
}
- if (!ndev->async_events)
- return 0;
-
- ret = aie2_error_async_events_send(ndev);
- if (ret) {
- XDNA_ERR(ndev->xdna, "Send async events failed");
- return ret;
- }
-
return 0;
}
@@ -258,6 +250,8 @@ static int aie2_mgmt_fw_query(struct amdxdna_dev_hdl *ndev)
return ret;
}
+ ndev->total_col = min(aie2_max_col, ndev->metadata.cols);
+
return 0;
}
@@ -312,7 +306,7 @@ static int aie2_xrs_set_dft_dpm_level(struct drm_device *ddev, u32 dpm_level)
if (ndev->pw_mode != POWER_MODE_DEFAULT || ndev->dpm_level == dpm_level)
return 0;
- return ndev->priv->hw_ops.set_dpm(ndev, dpm_level);
+ return aie2_pm_set_dpm(ndev, dpm_level);
}
static struct xrs_action_ops aie2_xrs_actions = {
@@ -332,13 +326,12 @@ static void aie2_hw_stop(struct amdxdna_dev *xdna)
}
aie2_mgmt_fw_fini(ndev);
- xdna_mailbox_stop_channel(ndev->mgmt_chann);
- xdna_mailbox_destroy_channel(ndev->mgmt_chann);
- ndev->mgmt_chann = NULL;
+ aie2_destroy_mgmt_chann(ndev);
drmm_kfree(&xdna->ddev, ndev->mbox);
ndev->mbox = NULL;
aie2_psp_stop(ndev->psp_hdl);
aie2_smu_fini(ndev);
+ aie2_error_async_events_free(ndev);
pci_disable_device(pdev);
ndev->dev_status = AIE2_DEV_INIT;
@@ -425,13 +418,24 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
goto destroy_mgmt_chann;
}
+ ret = aie2_mgmt_fw_query(ndev);
+ if (ret) {
+ XDNA_ERR(xdna, "failed to query fw, ret %d", ret);
+ goto destroy_mgmt_chann;
+ }
+
+ ret = aie2_error_async_events_alloc(ndev);
+ if (ret) {
+ XDNA_ERR(xdna, "Allocate async events failed, ret %d", ret);
+ goto destroy_mgmt_chann;
+ }
+
ndev->dev_status = AIE2_DEV_START;
return 0;
destroy_mgmt_chann:
- xdna_mailbox_stop_channel(ndev->mgmt_chann);
- xdna_mailbox_destroy_channel(ndev->mgmt_chann);
+ aie2_destroy_mgmt_chann(ndev);
stop_psp:
aie2_psp_stop(ndev->psp_hdl);
fini_smu:
@@ -460,7 +464,6 @@ static int aie2_hw_resume(struct amdxdna_dev *xdna)
struct amdxdna_client *client;
int ret;
- guard(mutex)(&xdna->dev_lock);
ret = aie2_hw_start(xdna);
if (ret) {
XDNA_ERR(xdna, "Start hardware failed, %d", ret);
@@ -571,13 +574,6 @@ static int aie2_init(struct amdxdna_dev *xdna)
goto release_fw;
}
- ret = aie2_mgmt_fw_query(ndev);
- if (ret) {
- XDNA_ERR(xdna, "Query firmware failed, ret %d", ret);
- goto stop_hw;
- }
- ndev->total_col = min(aie2_max_col, ndev->metadata.cols);
-
xrs_cfg.clk_list.num_levels = ndev->max_dpm_level + 1;
for (i = 0; i < xrs_cfg.clk_list.num_levels; i++)
xrs_cfg.clk_list.cu_clk_list[i] = ndev->priv->dpm_clk_tbl[i].hclk;
@@ -593,30 +589,10 @@ static int aie2_init(struct amdxdna_dev *xdna)
goto stop_hw;
}
- ret = aie2_error_async_events_alloc(ndev);
- if (ret) {
- XDNA_ERR(xdna, "Allocate async events failed, ret %d", ret);
- goto stop_hw;
- }
-
- ret = aie2_error_async_events_send(ndev);
- if (ret) {
- XDNA_ERR(xdna, "Send async events failed, ret %d", ret);
- goto async_event_free;
- }
-
- /* Issue a command to make sure firmware handled async events */
- ret = aie2_query_firmware_version(ndev, &ndev->xdna->fw_ver);
- if (ret) {
- XDNA_ERR(xdna, "Re-query firmware version failed");
- goto async_event_free;
- }
-
release_firmware(fw);
+ amdxdna_pm_init(xdna);
return 0;
-async_event_free:
- aie2_error_async_events_free(ndev);
stop_hw:
aie2_hw_stop(xdna);
release_fw:
@@ -627,10 +603,8 @@ static int aie2_init(struct amdxdna_dev *xdna)
static void aie2_fini(struct amdxdna_dev *xdna)
{
- struct amdxdna_dev_hdl *ndev = xdna->dev_handle;
-
+ amdxdna_pm_fini(xdna);
aie2_hw_stop(xdna);
- aie2_error_async_events_free(ndev);
}
static int aie2_get_aie_status(struct amdxdna_client *client,
@@ -862,6 +836,10 @@ static int aie2_get_info(struct amdxdna_client *client, struct amdxdna_drm_get_i
if (!drm_dev_enter(&xdna->ddev, &idx))
return -ENODEV;
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret)
+ goto dev_exit;
+
switch (args->param) {
case DRM_AMDXDNA_QUERY_AIE_STATUS:
ret = aie2_get_aie_status(client, args);
@@ -888,8 +866,11 @@ static int aie2_get_info(struct amdxdna_client *client, struct amdxdna_drm_get_i
XDNA_ERR(xdna, "Not supported request parameter %u", args->param);
ret = -EOPNOTSUPP;
}
+
+ amdxdna_pm_suspend_put(xdna);
XDNA_DBG(xdna, "Got param %d", args->param);
+dev_exit:
drm_dev_exit(idx);
return ret;
}
@@ -938,6 +919,10 @@ static int aie2_get_array(struct amdxdna_client *client,
if (!drm_dev_enter(&xdna->ddev, &idx))
return -ENODEV;
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret)
+ goto dev_exit;
+
switch (args->param) {
case DRM_AMDXDNA_HW_CONTEXT_ALL:
ret = aie2_query_ctx_status_array(client, args);
@@ -946,8 +931,11 @@ static int aie2_get_array(struct amdxdna_client *client,
XDNA_ERR(xdna, "Not supported request parameter %u", args->param);
ret = -EOPNOTSUPP;
}
+
+ amdxdna_pm_suspend_put(xdna);
XDNA_DBG(xdna, "Got param %d", args->param);
+dev_exit:
drm_dev_exit(idx);
return ret;
}
@@ -986,6 +974,10 @@ static int aie2_set_state(struct amdxdna_client *client,
if (!drm_dev_enter(&xdna->ddev, &idx))
return -ENODEV;
+ ret = amdxdna_pm_resume_get(xdna);
+ if (ret)
+ goto dev_exit;
+
switch (args->param) {
case DRM_AMDXDNA_SET_POWER_MODE:
ret = aie2_set_power_mode(client, args);
@@ -996,6 +988,8 @@ static int aie2_set_state(struct amdxdna_client *client,
break;
}
+ amdxdna_pm_suspend_put(xdna);
+dev_exit:
drm_dev_exit(idx);
return ret;
}
diff --git a/drivers/accel/amdxdna/aie2_pci.h b/drivers/accel/amdxdna/aie2_pci.h
index 91a8e948f82a..7d56f4bca3d6 100644
--- a/drivers/accel/amdxdna/aie2_pci.h
+++ b/drivers/accel/amdxdna/aie2_pci.h
@@ -182,6 +182,8 @@ struct amdxdna_dev_hdl {
u32 clk_gating;
u32 npuclk_freq;
u32 hclk_freq;
+ u32 max_tops;
+ u32 curr_tops;
/* Mailbox and the management channel */
struct mailbox *mbox;
@@ -217,6 +219,7 @@ struct amdxdna_dev_priv {
u32 mbox_dev_addr;
/* If mbox_size is 0, use BAR size. See MBOX_SIZE macro */
u32 mbox_size;
+ u32 hwctx_limit;
u32 sram_dev_addr;
struct aie2_bar_off_pair sram_offs[SRAM_MAX_INDEX];
struct aie2_bar_off_pair psp_regs_off[PSP_MAX_REGS];
@@ -244,6 +247,7 @@ int npu4_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level);
/* aie2_pm.c */
int aie2_pm_init(struct amdxdna_dev_hdl *ndev);
int aie2_pm_set_mode(struct amdxdna_dev_hdl *ndev, enum amdxdna_power_mode_type target);
+int aie2_pm_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level);
/* aie2_psp.c */
struct psp_device *aie2m_psp_create(struct drm_device *ddev, struct psp_config *conf);
@@ -257,6 +261,7 @@ int aie2_error_async_events_send(struct amdxdna_dev_hdl *ndev);
int aie2_error_async_msg_thread(void *data);
/* aie2_message.c */
+void aie2_destroy_mgmt_chann(struct amdxdna_dev_hdl *ndev);
int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev);
int aie2_resume_fw(struct amdxdna_dev_hdl *ndev);
int aie2_set_runtime_cfg(struct amdxdna_dev_hdl *ndev, u32 type, u64 value);
@@ -272,7 +277,8 @@ int aie2_map_host_buf(struct amdxdna_dev_hdl *ndev, u32 context_id, u64 addr, u6
int aie2_query_status(struct amdxdna_dev_hdl *ndev, char __user *buf, u32 size, u32 *cols_filled);
int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr, u32 size,
void *handle, int (*cb)(void*, void __iomem *, size_t));
-int aie2_config_cu(struct amdxdna_hwctx *hwctx);
+int aie2_config_cu(struct amdxdna_hwctx *hwctx,
+ int (*notify_cb)(void *, void __iomem *, size_t));
int aie2_execbuf(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job,
int (*notify_cb)(void *, void __iomem *, size_t));
int aie2_cmdlist_single_execbuf(struct amdxdna_hwctx *hwctx,
diff --git a/drivers/accel/amdxdna/aie2_pm.c b/drivers/accel/amdxdna/aie2_pm.c
index 426c38fce848..afcd6d4683e5 100644
--- a/drivers/accel/amdxdna/aie2_pm.c
+++ b/drivers/accel/amdxdna/aie2_pm.c
@@ -10,6 +10,7 @@
#include "aie2_pci.h"
#include "amdxdna_pci_drv.h"
+#include "amdxdna_pm.h"
#define AIE2_CLK_GATING_ENABLE 1
#define AIE2_CLK_GATING_DISABLE 0
@@ -26,6 +27,20 @@ static int aie2_pm_set_clk_gating(struct amdxdna_dev_hdl *ndev, u32 val)
return 0;
}
+int aie2_pm_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level)
+{
+ int ret;
+
+ ret = amdxdna_pm_resume_get(ndev->xdna);
+ if (ret)
+ return ret;
+
+ ret = ndev->priv->hw_ops.set_dpm(ndev, dpm_level);
+ amdxdna_pm_suspend_put(ndev->xdna);
+
+ return ret;
+}
+
int aie2_pm_init(struct amdxdna_dev_hdl *ndev)
{
int ret;
@@ -94,7 +109,7 @@ int aie2_pm_set_mode(struct amdxdna_dev_hdl *ndev, enum amdxdna_power_mode_type
return -EOPNOTSUPP;
}
- ret = ndev->priv->hw_ops.set_dpm(ndev, dpm_level);
+ ret = aie2_pm_set_dpm(ndev, dpm_level);
if (ret)
return ret;
diff --git a/drivers/accel/amdxdna/aie2_smu.c b/drivers/accel/amdxdna/aie2_smu.c
index d303701b0ded..28d3512efc2a 100644
--- a/drivers/accel/amdxdna/aie2_smu.c
+++ b/drivers/accel/amdxdna/aie2_smu.c
@@ -22,6 +22,13 @@
#define AIE2_SMU_SET_SOFT_DPMLEVEL 0x7
#define AIE2_SMU_SET_HARD_DPMLEVEL 0x8
+#define NPU4_DPM_TOPS(ndev, dpm_level) \
+({ \
+ typeof(ndev) _ndev = ndev; \
+ (4096 * (_ndev)->total_col * \
+ (_ndev)->priv->dpm_clk_tbl[dpm_level].hclk / 1000000); \
+})
+
static int aie2_smu_exec(struct amdxdna_dev_hdl *ndev, u32 reg_cmd,
u32 reg_arg, u32 *out)
{
@@ -75,8 +82,11 @@ int npu1_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level)
ndev->priv->dpm_clk_tbl[dpm_level].hclk, ret);
return ret;
}
+
ndev->hclk_freq = freq;
ndev->dpm_level = dpm_level;
+ ndev->max_tops = 2 * ndev->total_col;
+ ndev->curr_tops = ndev->max_tops * freq / 1028;
XDNA_DBG(ndev->xdna, "MP-NPU clock %d, H clock %d\n",
ndev->npuclk_freq, ndev->hclk_freq);
@@ -105,6 +115,8 @@ int npu4_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level)
ndev->npuclk_freq = ndev->priv->dpm_clk_tbl[dpm_level].npuclk;
ndev->hclk_freq = ndev->priv->dpm_clk_tbl[dpm_level].hclk;
ndev->dpm_level = dpm_level;
+ ndev->max_tops = NPU4_DPM_TOPS(ndev, ndev->max_dpm_level);
+ ndev->curr_tops = NPU4_DPM_TOPS(ndev, dpm_level);
XDNA_DBG(ndev->xdna, "MP-NPU clock %d, H clock %d\n",
ndev->npuclk_freq, ndev->hclk_freq);
diff --git a/drivers/accel/amdxdna/amdxdna_ctx.c b/drivers/accel/amdxdna/amdxdna_ctx.c
index a49c059e6ed4..feac65f2cab7 100644
--- a/drivers/accel/amdxdna/amdxdna_ctx.c
+++ b/drivers/accel/amdxdna/amdxdna_ctx.c
@@ -135,6 +135,33 @@ int amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo)
return -1;
}
+int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo,
+ struct amdxdna_sched_job *job, u32 cmd_idx,
+ enum ert_cmd_state error_state)
+{
+ struct amdxdna_client *client = job->hwctx->client;
+ struct amdxdna_cmd *cmd = abo->mem.kva;
+ struct amdxdna_cmd_chain *cc = NULL;
+
+ cmd->header &= ~AMDXDNA_CMD_STATE;
+ cmd->header |= FIELD_PREP(AMDXDNA_CMD_STATE, error_state);
+
+ if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) {
+ cc = amdxdna_cmd_get_payload(abo, NULL);
+ cc->error_index = (cmd_idx < cc->command_count) ? cmd_idx : 0;
+ abo = amdxdna_gem_get_obj(client, cc->data[cc->error_index], AMDXDNA_BO_CMD);
+ if (!abo)
+ return -EINVAL;
+ cmd = abo->mem.kva;
+ }
+
+ memset(cmd->data, 0xff, abo->mem.size - sizeof(*cmd));
+ if (cc)
+ amdxdna_gem_put_obj(abo);
+
+ return 0;
+}
+
/*
* This should be called in close() and remove(). DO NOT call in other syscalls.
* This guarantee that when hwctx and resources will be released, if user
@@ -164,19 +191,14 @@ int amdxdna_drm_create_hwctx_ioctl(struct drm_device *dev, void *data, struct dr
if (args->ext || args->ext_flags)
return -EINVAL;
- if (!drm_dev_enter(dev, &idx))
- return -ENODEV;
-
hwctx = kzalloc(sizeof(*hwctx), GFP_KERNEL);
- if (!hwctx) {
- ret = -ENOMEM;
- goto exit;
- }
+ if (!hwctx)
+ return -ENOMEM;
if (copy_from_user(&hwctx->qos, u64_to_user_ptr(args->qos_p), sizeof(hwctx->qos))) {
XDNA_ERR(xdna, "Access QoS info failed");
- ret = -EFAULT;
- goto free_hwctx;
+ kfree(hwctx);
+ return -EFAULT;
}
hwctx->client = client;
@@ -184,30 +206,36 @@ int amdxdna_drm_create_hwctx_ioctl(struct drm_device *dev, void *data, struct dr
hwctx->num_tiles = args->num_tiles;
hwctx->mem_size = args->mem_size;
hwctx->max_opc = args->max_opc;
- ret = xa_alloc_cyclic(&client->hwctx_xa, &hwctx->id, hwctx,
- XA_LIMIT(AMDXDNA_INVALID_CTX_HANDLE + 1, MAX_HWCTX_ID),
- &client->next_hwctxid, GFP_KERNEL);
- if (ret < 0) {
- XDNA_ERR(xdna, "Allocate hwctx ID failed, ret %d", ret);
+
+ guard(mutex)(&xdna->dev_lock);
+
+ if (!drm_dev_enter(dev, &idx)) {
+ ret = -ENODEV;
goto free_hwctx;
}
- hwctx->name = kasprintf(GFP_KERNEL, "hwctx.%d.%d", client->pid, hwctx->id);
+ ret = xdna->dev_info->ops->hwctx_init(hwctx);
+ if (ret) {
+ XDNA_ERR(xdna, "Init hwctx failed, ret %d", ret);
+ goto dev_exit;
+ }
+
+ hwctx->name = kasprintf(GFP_KERNEL, "hwctx.%d.%d", client->pid, hwctx->fw_ctx_id);
if (!hwctx->name) {
ret = -ENOMEM;
- goto rm_id;
+ goto fini_hwctx;
}
- mutex_lock(&xdna->dev_lock);
- ret = xdna->dev_info->ops->hwctx_init(hwctx);
- if (ret) {
- mutex_unlock(&xdna->dev_lock);
- XDNA_ERR(xdna, "Init hwctx failed, ret %d", ret);
+ ret = xa_alloc_cyclic(&client->hwctx_xa, &hwctx->id, hwctx,
+ XA_LIMIT(AMDXDNA_INVALID_CTX_HANDLE + 1, MAX_HWCTX_ID),
+ &client->next_hwctxid, GFP_KERNEL);
+ if (ret < 0) {
+ XDNA_ERR(xdna, "Allocate hwctx ID failed, ret %d", ret);
goto free_name;
}
+
args->handle = hwctx->id;
args->syncobj_handle = hwctx->syncobj_hdl;
- mutex_unlock(&xdna->dev_lock);
atomic64_set(&hwctx->job_submit_cnt, 0);
atomic64_set(&hwctx->job_free_cnt, 0);
@@ -217,12 +245,12 @@ int amdxdna_drm_create_hwctx_ioctl(struct drm_device *dev, void *data, struct dr
free_name:
kfree(hwctx->name);
-rm_id:
- xa_erase(&client->hwctx_xa, hwctx->id);
+fini_hwctx:
+ xdna->dev_info->ops->hwctx_fini(hwctx);
+dev_exit:
+ drm_dev_exit(idx);
free_hwctx:
kfree(hwctx);
-exit:
- drm_dev_exit(idx);
return ret;
}
@@ -422,7 +450,14 @@ int amdxdna_cmd_submit(struct amdxdna_client *client,
goto free_job;
}
} else {
- job->cmd_bo = NULL;
+ /*
+ * A command submission with the invalid handle would leave
+ * job->cmd_bo NULL and later fault when the scheduler
+ * dereferences it in amdxdna_cmd_set_state().
+ */
+ XDNA_DBG(xdna, "Command BO handle required for user submission");
+ ret = -EINVAL;
+ goto free_job;
}
ret = amdxdna_arg_bos_lookup(client, job, arg_bo_hdls, arg_bo_cnt);
@@ -440,11 +475,6 @@ int amdxdna_cmd_submit(struct amdxdna_client *client,
goto unlock_srcu;
}
- if (hwctx->status != HWCTX_STAT_READY) {
- XDNA_ERR(xdna, "HW Context is not ready");
- ret = -EINVAL;
- goto unlock_srcu;
- }
job->hwctx = hwctx;
job->mm = current->mm;
diff --git a/drivers/accel/amdxdna/amdxdna_ctx.h b/drivers/accel/amdxdna/amdxdna_ctx.h
index 383f3f15fac8..d54ccf1a3b6f 100644
--- a/drivers/accel/amdxdna/amdxdna_ctx.h
+++ b/drivers/accel/amdxdna/amdxdna_ctx.h
@@ -139,6 +139,9 @@ amdxdna_cmd_get_state(struct amdxdna_gem_obj *abo)
void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size);
int amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo);
+int amdxdna_cmd_set_error(struct amdxdna_gem_obj *abo,
+ struct amdxdna_sched_job *job, u32 cmd_idx,
+ enum ert_cmd_state error_state);
void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job);
void amdxdna_hwctx_remove_all(struct amdxdna_client *client);
diff --git a/drivers/accel/amdxdna/amdxdna_gem.c b/drivers/accel/amdxdna/amdxdna_gem.c
index ca747457cec7..147c4c66be4c 100644
--- a/drivers/accel/amdxdna/amdxdna_gem.c
+++ b/drivers/accel/amdxdna/amdxdna_gem.c
@@ -108,7 +108,7 @@ static bool amdxdna_hmm_invalidate(struct mmu_interval_notifier *mni,
xdna = to_xdna_dev(to_gobj(abo)->dev);
XDNA_DBG(xdna, "Invalidating range 0x%lx, 0x%lx, type %d",
- mapp->vma->vm_start, mapp->vma->vm_end, abo->type);
+ mapp->range.start, mapp->range.end, abo->type);
if (!mmu_notifier_range_blockable(range))
return false;
@@ -137,15 +137,23 @@ static const struct mmu_interval_notifier_ops amdxdna_hmm_ops = {
.invalidate = amdxdna_hmm_invalidate,
};
+static inline bool compare_range(struct amdxdna_umap *mapp,
+ struct mm_struct *mm,
+ unsigned long start, unsigned long end)
+{
+ return (!mapp->unmapped && mapp->notifier.mm == mm &&
+ mapp->range.start == start && mapp->range.end == end);
+}
+
static void amdxdna_hmm_unregister(struct amdxdna_gem_obj *abo,
struct vm_area_struct *vma)
{
struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev);
struct amdxdna_umap *mapp;
- down_read(&xdna->notifier_lock);
+ down_write(&xdna->notifier_lock);
list_for_each_entry(mapp, &abo->mem.umap_list, node) {
- if (!vma || mapp->vma == vma) {
+ if (!vma || compare_range(mapp, vma->vm_mm, vma->vm_start, vma->vm_end)) {
if (!mapp->unmapped) {
queue_work(xdna->notifier_wq, &mapp->hmm_unreg_work);
mapp->unmapped = true;
@@ -154,22 +162,22 @@ static void amdxdna_hmm_unregister(struct amdxdna_gem_obj *abo,
break;
}
}
- up_read(&xdna->notifier_lock);
+ up_write(&xdna->notifier_lock);
}
static void amdxdna_umap_release(struct kref *ref)
{
struct amdxdna_umap *mapp = container_of(ref, struct amdxdna_umap, refcnt);
- struct vm_area_struct *vma = mapp->vma;
+ struct amdxdna_gem_obj *abo = mapp->abo;
struct amdxdna_dev *xdna;
mmu_interval_notifier_remove(&mapp->notifier);
- if (is_import_bo(mapp->abo) && vma->vm_file && vma->vm_file->f_mapping)
- mapping_clear_unevictable(vma->vm_file->f_mapping);
xdna = to_xdna_dev(to_gobj(mapp->abo)->dev);
down_write(&xdna->notifier_lock);
list_del(&mapp->node);
+ if (list_empty(&abo->mem.umap_list))
+ abo->mem.userptr = AMDXDNA_INVALID_ADDR;
up_write(&xdna->notifier_lock);
kvfree(mapp->range.hmm_pfns);
@@ -202,7 +210,16 @@ static int amdxdna_hmm_register(struct amdxdna_gem_obj *abo,
if (!xdna->dev_info->ops->hmm_invalidate)
return 0;
- mapp = kzalloc(sizeof(*mapp), GFP_KERNEL);
+ down_read(&xdna->notifier_lock);
+ list_for_each_entry(mapp, &abo->mem.umap_list, node) {
+ if (compare_range(mapp, current->mm, addr, addr + len)) {
+ up_read(&xdna->notifier_lock);
+ return 0;
+ }
+ }
+ up_read(&xdna->notifier_lock);
+
+ mapp = kzalloc_obj(*mapp);
if (!mapp)
return -ENOMEM;
@@ -228,17 +245,14 @@ static int amdxdna_hmm_register(struct amdxdna_gem_obj *abo,
mapp->range.start = vma->vm_start;
mapp->range.end = vma->vm_end;
mapp->range.default_flags = HMM_PFN_REQ_FAULT;
- mapp->vma = vma;
mapp->abo = abo;
kref_init(&mapp->refcnt);
- if (abo->mem.userptr == AMDXDNA_INVALID_ADDR)
- abo->mem.userptr = addr;
INIT_WORK(&mapp->hmm_unreg_work, amdxdna_hmm_unreg_work);
- if (is_import_bo(abo) && vma->vm_file && vma->vm_file->f_mapping)
- mapping_set_unevictable(vma->vm_file->f_mapping);
down_write(&xdna->notifier_lock);
+ if (list_empty(&abo->mem.umap_list))
+ abo->mem.userptr = addr;
list_add_tail(&mapp->node, &abo->mem.umap_list);
up_write(&xdna->notifier_lock);
@@ -265,6 +279,23 @@ static void amdxdna_gem_dev_obj_free(struct drm_gem_object *gobj)
amdxdna_gem_destroy_obj(abo);
}
+static void amdxdna_mark_mapp_invalid(struct amdxdna_gem_obj *abo,
+ struct vm_area_struct *vma)
+{
+ struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev);
+ struct amdxdna_umap *mapp;
+
+ down_write(&xdna->notifier_lock);
+ abo->mem.map_invalid = true;
+ list_for_each_entry(mapp, &abo->mem.umap_list, node) {
+ if (compare_range(mapp, vma->vm_mm, vma->vm_start, vma->vm_end)) {
+ mapp->invalid = true;
+ break;
+ }
+ }
+ up_write(&xdna->notifier_lock);
+}
+
static int amdxdna_insert_pages(struct amdxdna_gem_obj *abo,
struct vm_area_struct *vma)
{
@@ -279,26 +310,17 @@ static int amdxdna_insert_pages(struct amdxdna_gem_obj *abo,
XDNA_ERR(xdna, "Failed shmem mmap %d", ret);
return ret;
}
-
- /* The buffer is based on memory pages. Fix the flag. */
- vm_flags_mod(vma, VM_MIXEDMAP, VM_PFNMAP);
- ret = vm_insert_pages(vma, vma->vm_start, abo->base.pages,
- &num_pages);
+ } else {
+ vma->vm_private_data = NULL;
+ vma->vm_ops = NULL;
+ ret = dma_buf_mmap(abo->dma_buf, vma, 0);
if (ret) {
- XDNA_ERR(xdna, "Failed insert pages %d", ret);
- vma->vm_ops->close(vma);
+ XDNA_ERR(xdna, "Failed to mmap dma buf %d", ret);
return ret;
}
- return 0;
- }
-
- vma->vm_private_data = NULL;
- vma->vm_ops = NULL;
- ret = dma_buf_mmap(abo->dma_buf, vma, 0);
- if (ret) {
- XDNA_ERR(xdna, "Failed to mmap dma buf %d", ret);
- return ret;
+ /* Drop the reference drm_gem_mmap_obj() acquired.*/
+ drm_gem_object_put(to_gobj(abo));
}
do {
@@ -307,17 +329,14 @@ static int amdxdna_insert_pages(struct amdxdna_gem_obj *abo,
fault_ret = handle_mm_fault(vma, vma->vm_start + offset,
FAULT_FLAG_WRITE, NULL);
if (fault_ret & VM_FAULT_ERROR) {
- vma->vm_ops->close(vma);
XDNA_ERR(xdna, "Fault in page failed");
- return -EFAULT;
+ amdxdna_mark_mapp_invalid(abo, vma);
+ break;
}
offset += PAGE_SIZE;
} while (--num_pages);
- /* Drop the reference drm_gem_mmap_obj() acquired.*/
- drm_gem_object_put(to_gobj(abo));
-
return 0;
}
@@ -934,6 +953,7 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev,
struct amdxdna_drm_sync_bo *args = data;
struct amdxdna_gem_obj *abo;
struct drm_gem_object *gobj;
+ u64 end, size;
int ret;
gobj = drm_gem_object_lookup(filp, args->handle);
@@ -943,6 +963,13 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev,
}
abo = to_xdna_obj(gobj);
+ if (args->offset >= abo->mem.size ||
+ check_add_overflow(args->offset, args->size, &end)) {
+ ret = -EINVAL;
+ goto put_obj;
+ }
+ size = min(abo->mem.size, end) - args->offset;
+
ret = amdxdna_gem_pin(abo);
if (ret) {
XDNA_ERR(xdna, "Pin BO %d failed, ret %d", args->handle, ret);
@@ -955,7 +982,7 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev,
if (is_import_bo(abo))
drm_clflush_sg(abo->base.sgt);
else if (abo->mem.kva)
- drm_clflush_virt_range(abo->mem.kva + args->offset, args->size);
+ drm_clflush_virt_range(abo->mem.kva + args->offset, size);
else if (abo->base.pages)
drm_clflush_pages(abo->base.pages, gobj->size >> PAGE_SHIFT);
else
diff --git a/drivers/accel/amdxdna/amdxdna_gem.h b/drivers/accel/amdxdna/amdxdna_gem.h
index ae29db94a9d3..b3c814c0bb7e 100644
--- a/drivers/accel/amdxdna/amdxdna_gem.h
+++ b/drivers/accel/amdxdna/amdxdna_gem.h
@@ -9,7 +9,6 @@
#include <linux/hmm.h>
struct amdxdna_umap {
- struct vm_area_struct *vma;
struct mmu_interval_notifier notifier;
struct hmm_range range;
struct work_struct hmm_unreg_work;
diff --git a/drivers/accel/amdxdna/amdxdna_mailbox.c b/drivers/accel/amdxdna/amdxdna_mailbox.c
index 2bacb89cd80c..469242ed8224 100644
--- a/drivers/accel/amdxdna/amdxdna_mailbox.c
+++ b/drivers/accel/amdxdna/amdxdna_mailbox.c
@@ -178,7 +178,8 @@ static void mailbox_release_msg(struct mailbox_channel *mb_chann,
{
MB_DBG(mb_chann, "msg_id 0x%x msg opcode 0x%x",
mb_msg->pkg.header.id, mb_msg->pkg.header.opcode);
- mb_msg->notify_cb(mb_msg->handle, NULL, 0);
+ if (mb_msg->notify_cb)
+ mb_msg->notify_cb(mb_msg->handle, NULL, 0);
kfree(mb_msg);
}
@@ -237,7 +238,7 @@ mailbox_get_resp(struct mailbox_channel *mb_chann, struct xdna_msg_header *heade
{
struct mailbox_msg *mb_msg;
int msg_id;
- int ret;
+ int ret = 0;
msg_id = header->id;
if (!mailbox_validate_msgid(msg_id)) {
@@ -254,9 +255,11 @@ mailbox_get_resp(struct mailbox_channel *mb_chann, struct xdna_msg_header *heade
MB_DBG(mb_chann, "opcode 0x%x size %d id 0x%x",
header->opcode, header->total_size, header->id);
- ret = mb_msg->notify_cb(mb_msg->handle, data, header->total_size);
- if (unlikely(ret))
- MB_ERR(mb_chann, "Message callback ret %d", ret);
+ if (mb_msg->notify_cb) {
+ ret = mb_msg->notify_cb(mb_msg->handle, data, header->total_size);
+ if (unlikely(ret))
+ MB_ERR(mb_chann, "Message callback ret %d", ret);
+ }
kfree(mb_msg);
return ret;
diff --git a/drivers/accel/amdxdna/amdxdna_pci_drv.c b/drivers/accel/amdxdna/amdxdna_pci_drv.c
index 88643e28af84..7d98a84b74cd 100644
--- a/drivers/accel/amdxdna/amdxdna_pci_drv.c
+++ b/drivers/accel/amdxdna/amdxdna_pci_drv.c
@@ -13,13 +13,11 @@
#include <drm/gpu_scheduler.h>
#include <linux/iommu.h>
#include <linux/pci.h>
-#include <linux/pm_runtime.h>
#include "amdxdna_ctx.h"
#include "amdxdna_gem.h"
#include "amdxdna_pci_drv.h"
-
-#define AMDXDNA_AUTOSUSPEND_DELAY 5000 /* milliseconds */
+#include "amdxdna_pm.h"
MODULE_FIRMWARE("amdnpu/1502_00/npu.sbin");
MODULE_FIRMWARE("amdnpu/17f0_10/npu.sbin");
@@ -61,17 +59,9 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp)
struct amdxdna_client *client;
int ret;
- ret = pm_runtime_resume_and_get(ddev->dev);
- if (ret) {
- XDNA_ERR(xdna, "Failed to get rpm, ret %d", ret);
- return ret;
- }
-
client = kzalloc(sizeof(*client), GFP_KERNEL);
- if (!client) {
- ret = -ENOMEM;
- goto put_rpm;
- }
+ if (!client)
+ return -ENOMEM;
client->pid = pid_nr(rcu_access_pointer(filp->pid));
client->xdna = xdna;
@@ -108,9 +98,6 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp)
iommu_sva_unbind_device(client->sva);
failed:
kfree(client);
-put_rpm:
- pm_runtime_mark_last_busy(ddev->dev);
- pm_runtime_put_autosuspend(ddev->dev);
return ret;
}
@@ -133,8 +120,6 @@ static void amdxdna_drm_close(struct drm_device *ddev, struct drm_file *filp)
XDNA_DBG(xdna, "pid %d closed", client->pid);
kfree(client);
- pm_runtime_mark_last_busy(ddev->dev);
- pm_runtime_put_autosuspend(ddev->dev);
}
static int amdxdna_flush(struct file *f, fl_owner_t id)
@@ -313,19 +298,12 @@ static int amdxdna_probe(struct pci_dev *pdev, const struct pci_device_id *id)
goto failed_dev_fini;
}
- pm_runtime_set_autosuspend_delay(dev, AMDXDNA_AUTOSUSPEND_DELAY);
- pm_runtime_use_autosuspend(dev);
- pm_runtime_allow(dev);
-
ret = drm_dev_register(&xdna->ddev, 0);
if (ret) {
XDNA_ERR(xdna, "DRM register failed, ret %d", ret);
- pm_runtime_forbid(dev);
goto failed_sysfs_fini;
}
- pm_runtime_mark_last_busy(dev);
- pm_runtime_put_autosuspend(dev);
return 0;
failed_sysfs_fini:
@@ -342,14 +320,10 @@ static int amdxdna_probe(struct pci_dev *pdev, const struct pci_device_id *id)
static void amdxdna_remove(struct pci_dev *pdev)
{
struct amdxdna_dev *xdna = pci_get_drvdata(pdev);
- struct device *dev = &pdev->dev;
struct amdxdna_client *client;
destroy_workqueue(xdna->notifier_wq);
- pm_runtime_get_noresume(dev);
- pm_runtime_forbid(dev);
-
drm_dev_unplug(&xdna->ddev);
amdxdna_sysfs_fini(xdna);
@@ -368,29 +342,9 @@ static void amdxdna_remove(struct pci_dev *pdev)
mutex_unlock(&xdna->dev_lock);
}
-static int amdxdna_pmops_suspend(struct device *dev)
-{
- struct amdxdna_dev *xdna = pci_get_drvdata(to_pci_dev(dev));
-
- if (!xdna->dev_info->ops->suspend)
- return -EOPNOTSUPP;
-
- return xdna->dev_info->ops->suspend(xdna);
-}
-
-static int amdxdna_pmops_resume(struct device *dev)
-{
- struct amdxdna_dev *xdna = pci_get_drvdata(to_pci_dev(dev));
-
- if (!xdna->dev_info->ops->resume)
- return -EOPNOTSUPP;
-
- return xdna->dev_info->ops->resume(xdna);
-}
-
static const struct dev_pm_ops amdxdna_pm_ops = {
- SYSTEM_SLEEP_PM_OPS(amdxdna_pmops_suspend, amdxdna_pmops_resume)
- RUNTIME_PM_OPS(amdxdna_pmops_suspend, amdxdna_pmops_resume, NULL)
+ SYSTEM_SLEEP_PM_OPS(amdxdna_pm_suspend, amdxdna_pm_resume)
+ RUNTIME_PM_OPS(amdxdna_pm_suspend, amdxdna_pm_resume, NULL)
};
static struct pci_driver amdxdna_pci_driver = {
diff --git a/drivers/accel/amdxdna/amdxdna_pci_drv.h b/drivers/accel/amdxdna/amdxdna_pci_drv.h
index 64009ca24982..171873457a3b 100644
--- a/drivers/accel/amdxdna/amdxdna_pci_drv.h
+++ b/drivers/accel/amdxdna/amdxdna_pci_drv.h
@@ -6,6 +6,7 @@
#ifndef _AMDXDNA_PCI_DRV_H_
#define _AMDXDNA_PCI_DRV_H_
+#include <drm/drm_print.h>
#include <linux/workqueue.h>
#include <linux/xarray.h>
diff --git a/drivers/accel/amdxdna/amdxdna_pm.c b/drivers/accel/amdxdna/amdxdna_pm.c
new file mode 100644
index 000000000000..d024d480521c
--- /dev/null
+++ b/drivers/accel/amdxdna/amdxdna_pm.c
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2025, Advanced Micro Devices, Inc.
+ */
+
+#include <drm/amdxdna_accel.h>
+#include <drm/drm_drv.h>
+#include <linux/pm_runtime.h>
+
+#include "amdxdna_pm.h"
+
+#define AMDXDNA_AUTOSUSPEND_DELAY 5000 /* milliseconds */
+
+int amdxdna_pm_suspend(struct device *dev)
+{
+ struct amdxdna_dev *xdna = to_xdna_dev(dev_get_drvdata(dev));
+ int ret = -EOPNOTSUPP;
+
+ if (xdna->dev_info->ops->suspend)
+ ret = xdna->dev_info->ops->suspend(xdna);
+
+ XDNA_DBG(xdna, "Suspend done ret %d", ret);
+ return ret;
+}
+
+int amdxdna_pm_resume(struct device *dev)
+{
+ struct amdxdna_dev *xdna = to_xdna_dev(dev_get_drvdata(dev));
+ int ret = -EOPNOTSUPP;
+
+ if (xdna->dev_info->ops->resume)
+ ret = xdna->dev_info->ops->resume(xdna);
+
+ XDNA_DBG(xdna, "Resume done ret %d", ret);
+ return ret;
+}
+
+int amdxdna_pm_resume_get(struct amdxdna_dev *xdna)
+{
+ struct device *dev = xdna->ddev.dev;
+ int ret;
+
+ ret = pm_runtime_resume_and_get(dev);
+ if (ret) {
+ XDNA_ERR(xdna, "Resume failed: %d", ret);
+ pm_runtime_set_suspended(dev);
+ }
+
+ return ret;
+}
+
+void amdxdna_pm_suspend_put(struct amdxdna_dev *xdna)
+{
+ struct device *dev = xdna->ddev.dev;
+
+ pm_runtime_put_autosuspend(dev);
+}
+
+void amdxdna_pm_init(struct amdxdna_dev *xdna)
+{
+ struct device *dev = xdna->ddev.dev;
+
+ pm_runtime_set_active(dev);
+ pm_runtime_set_autosuspend_delay(dev, AMDXDNA_AUTOSUSPEND_DELAY);
+ pm_runtime_use_autosuspend(dev);
+ pm_runtime_allow(dev);
+ pm_runtime_put_autosuspend(dev);
+}
+
+void amdxdna_pm_fini(struct amdxdna_dev *xdna)
+{
+ struct device *dev = xdna->ddev.dev;
+
+ pm_runtime_get_noresume(dev);
+ pm_runtime_forbid(dev);
+}
diff --git a/drivers/accel/amdxdna/amdxdna_pm.h b/drivers/accel/amdxdna/amdxdna_pm.h
new file mode 100644
index 000000000000..77b2d6e45570
--- /dev/null
+++ b/drivers/accel/amdxdna/amdxdna_pm.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2025, Advanced Micro Devices, Inc.
+ */
+
+#ifndef _AMDXDNA_PM_H_
+#define _AMDXDNA_PM_H_
+
+#include "amdxdna_pci_drv.h"
+
+int amdxdna_pm_suspend(struct device *dev);
+int amdxdna_pm_resume(struct device *dev);
+int amdxdna_pm_resume_get(struct amdxdna_dev *xdna);
+void amdxdna_pm_suspend_put(struct amdxdna_dev *xdna);
+void amdxdna_pm_init(struct amdxdna_dev *xdna);
+void amdxdna_pm_fini(struct amdxdna_dev *xdna);
+
+#endif /* _AMDXDNA_PM_H_ */
diff --git a/drivers/accel/amdxdna/npu1_regs.c b/drivers/accel/amdxdna/npu1_regs.c
index e4f6dac7d00f..8edee15076b0 100644
--- a/drivers/accel/amdxdna/npu1_regs.c
+++ b/drivers/accel/amdxdna/npu1_regs.c
@@ -72,6 +72,7 @@ static const struct amdxdna_dev_priv npu1_dev_priv = {
.mbox_dev_addr = NPU1_MBOX_BAR_BASE,
.mbox_size = 0, /* Use BAR size */
.sram_dev_addr = NPU1_SRAM_BAR_BASE,
+ .hwctx_limit = 6,
.sram_offs = {
DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU1_SRAM, MPNPU_SRAM_X2I_MAILBOX_0),
DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU1_SRAM, MPNPU_SRAM_I2X_MAILBOX_15),
diff --git a/drivers/accel/amdxdna/npu2_regs.c b/drivers/accel/amdxdna/npu2_regs.c
index a081cac75ee0..050e2c20dccb 100644
--- a/drivers/accel/amdxdna/npu2_regs.c
+++ b/drivers/accel/amdxdna/npu2_regs.c
@@ -71,6 +71,7 @@ static const struct amdxdna_dev_priv npu2_dev_priv = {
.mbox_dev_addr = NPU2_MBOX_BAR_BASE,
.mbox_size = 0, /* Use BAR size */
.sram_dev_addr = NPU2_SRAM_BAR_BASE,
+ .hwctx_limit = 16,
.sram_offs = {
DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU2_SRAM, MPNPU_SRAM_X2I_MAILBOX_0),
DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU2_SRAM, MPNPU_SRAM_X2I_MAILBOX_15),
diff --git a/drivers/accel/amdxdna/npu4_regs.c b/drivers/accel/amdxdna/npu4_regs.c
index 9f2e33182ec6..00303c36b302 100644
--- a/drivers/accel/amdxdna/npu4_regs.c
+++ b/drivers/accel/amdxdna/npu4_regs.c
@@ -92,6 +92,7 @@ static const struct amdxdna_dev_priv npu4_dev_priv = {
.mbox_dev_addr = NPU4_MBOX_BAR_BASE,
.mbox_size = 0, /* Use BAR size */
.sram_dev_addr = NPU4_SRAM_BAR_BASE,
+ .hwctx_limit = 16,
.sram_offs = {
DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU4_SRAM, MPNPU_SRAM_X2I_MAILBOX_0),
DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU4_SRAM, MPNPU_SRAM_X2I_MAILBOX_15),
diff --git a/drivers/accel/amdxdna/npu5_regs.c b/drivers/accel/amdxdna/npu5_regs.c
index 5f1cf83461c4..b6999490020e 100644
--- a/drivers/accel/amdxdna/npu5_regs.c
+++ b/drivers/accel/amdxdna/npu5_regs.c
@@ -71,6 +71,7 @@ static const struct amdxdna_dev_priv npu5_dev_priv = {
.mbox_dev_addr = NPU5_MBOX_BAR_BASE,
.mbox_size = 0, /* Use BAR size */
.sram_dev_addr = NPU5_SRAM_BAR_BASE,
+ .hwctx_limit = 16,
.sram_offs = {
DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU5_SRAM, MPNPU_SRAM_X2I_MAILBOX_0),
DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU5_SRAM, MPNPU_SRAM_X2I_MAILBOX_15),
diff --git a/drivers/accel/amdxdna/npu6_regs.c b/drivers/accel/amdxdna/npu6_regs.c
index 94a7005685a7..27fda4da2f81 100644
--- a/drivers/accel/amdxdna/npu6_regs.c
+++ b/drivers/accel/amdxdna/npu6_regs.c
@@ -71,6 +71,7 @@ static const struct amdxdna_dev_priv npu6_dev_priv = {
.mbox_dev_addr = NPU6_MBOX_BAR_BASE,
.mbox_size = 0, /* Use BAR size */
.sram_dev_addr = NPU6_SRAM_BAR_BASE,
+ .hwctx_limit = 16,
.sram_offs = {
DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU6_SRAM, MPNPU_SRAM_X2I_MAILBOX_0),
DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU6_SRAM, MPNPU_SRAM_X2I_MAILBOX_15),
diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
index 222a6bc61644..16bd24125a23 100644
--- a/drivers/acpi/bus.c
+++ b/drivers/acpi/bus.c
@@ -747,30 +747,52 @@ static int __init acpi_setup_sb_notify_handler(void)
Device Matching
-------------------------------------------------------------------------- */
+
+static struct device *primary_physical_device(struct acpi_device *adev)
+{
+ struct acpi_device_physical_node *pn;
+
+ pn = list_first_entry_or_null(&adev->physical_node_list,
+ struct acpi_device_physical_node, node);
+ if (pn)
+ return pn->dev;
+
+ return NULL;
+}
+
/**
- * acpi_get_first_physical_node - Get first physical node of an ACPI device
+ * acpi_bus_get_primary_device - Get first physical device for a given ACPI one
+ * @adev: ACPI device to get the first physical device for.
+ *
+ * Find the first physical device for which @adev is the ACPI companion and
+ * reference count it if present.
+ *
+ * Return: Pointer to the first physical counterpart of @adev or NULL if there
+ * are none. Callers are responsible for invoking put_device() on the returned
+ * device.
+ */
+struct device *acpi_bus_get_primary_device(struct acpi_device *adev)
+{
+ if (!adev)
+ return NULL;
+
+ guard(mutex)(&adev->physical_node_lock);
+
+ return get_device(primary_physical_device(adev));
+}
+EXPORT_SYMBOL_GPL(acpi_bus_get_primary_device);
+
+/**
+ * acpi_get_first_physical_node - Find first physical node of an ACPI device
* @adev: ACPI device in question
*
* Return: First physical node of ACPI device @adev
*/
struct device *acpi_get_first_physical_node(struct acpi_device *adev)
{
- struct mutex *physical_node_lock = &adev->physical_node_lock;
- struct device *phys_dev;
-
- mutex_lock(physical_node_lock);
- if (list_empty(&adev->physical_node_list)) {
- phys_dev = NULL;
- } else {
- const struct acpi_device_physical_node *node;
+ guard(mutex)(&adev->physical_node_lock);
- node = list_first_entry(&adev->physical_node_list,
- struct acpi_device_physical_node, node);
-
- phys_dev = node->dev;
- }
- mutex_unlock(physical_node_lock);
- return phys_dev;
+ return primary_physical_device(adev);
}
EXPORT_SYMBOL_GPL(acpi_get_first_physical_node);
diff --git a/drivers/acpi/button.c b/drivers/acpi/button.c
index 3c6dd9b4ba0a..384caea90ce5 100644
--- a/drivers/acpi/button.c
+++ b/drivers/acpi/button.c
@@ -19,6 +19,7 @@
#include <linux/slab.h>
#include <linux/acpi.h>
#include <linux/dmi.h>
+#include <linux/platform_device.h>
#include <acpi/button.h>
#define ACPI_BUTTON_CLASS "button"
@@ -145,8 +146,8 @@ static const struct dmi_system_id dmi_lid_quirks[] = {
{}
};
-static int acpi_button_add(struct acpi_device *device);
-static void acpi_button_remove(struct acpi_device *device);
+static int acpi_button_probe(struct platform_device *pdev);
+static void acpi_button_remove(struct platform_device *pdev);
#ifdef CONFIG_PM_SLEEP
static int acpi_button_suspend(struct device *dev);
@@ -157,18 +158,19 @@ static int acpi_button_resume(struct device *dev);
#endif
static SIMPLE_DEV_PM_OPS(acpi_button_pm, acpi_button_suspend, acpi_button_resume);
-static struct acpi_driver acpi_button_driver = {
- .name = "button",
- .class = ACPI_BUTTON_CLASS,
- .ids = button_device_ids,
- .ops = {
- .add = acpi_button_add,
- .remove = acpi_button_remove,
+static struct platform_driver acpi_button_driver = {
+ .probe = acpi_button_probe,
+ .remove = acpi_button_remove,
+ .driver = {
+ .name = "acpi-button",
+ .acpi_match_table = button_device_ids,
+ .pm = &acpi_button_pm,
},
- .drv.pm = &acpi_button_pm,
};
struct acpi_button {
+ struct acpi_device *adev;
+ struct platform_device *pdev;
unsigned int type;
struct input_dev *input;
char phys[32]; /* for input device */
@@ -202,9 +204,9 @@ static int acpi_lid_evaluate_state(struct acpi_device *device)
return lid_state ? 1 : 0;
}
-static int acpi_lid_notify_state(struct acpi_device *device, int state)
+static int acpi_lid_notify_state(struct acpi_button *button, int state)
{
- struct acpi_button *button = acpi_driver_data(device);
+ struct acpi_device *device = button->adev;
ktime_t next_report;
bool do_update;
@@ -287,18 +289,18 @@ static int acpi_lid_notify_state(struct acpi_device *device, int state)
static int __maybe_unused acpi_button_state_seq_show(struct seq_file *seq,
void *offset)
{
- struct acpi_device *device = seq->private;
+ struct acpi_button *button = seq->private;
int state;
- state = acpi_lid_evaluate_state(device);
+ state = acpi_lid_evaluate_state(button->adev);
seq_printf(seq, "state: %s\n",
state < 0 ? "unsupported" : (state ? "open" : "closed"));
return 0;
}
-static int acpi_button_add_fs(struct acpi_device *device)
+static int acpi_button_add_fs(struct acpi_button *button)
{
- struct acpi_button *button = acpi_driver_data(device);
+ struct acpi_device *device = button->adev;
struct proc_dir_entry *entry = NULL;
int ret = 0;
@@ -333,7 +335,7 @@ static int acpi_button_add_fs(struct acpi_device *device)
/* create /proc/acpi/button/lid/LID/state */
entry = proc_create_single_data(ACPI_BUTTON_FILE_STATE, S_IRUGO,
acpi_device_dir(device), acpi_button_state_seq_show,
- device);
+ button);
if (!entry) {
ret = -ENODEV;
goto remove_dev_dir;
@@ -355,9 +357,9 @@ static int acpi_button_add_fs(struct acpi_device *device)
goto done;
}
-static int acpi_button_remove_fs(struct acpi_device *device)
+static int acpi_button_remove_fs(struct acpi_button *button)
{
- struct acpi_button *button = acpi_driver_data(device);
+ struct acpi_device *device = button->adev;
if (button->type != ACPI_BUTTON_TYPE_LID)
return 0;
@@ -385,9 +387,10 @@ int acpi_lid_open(void)
}
EXPORT_SYMBOL(acpi_lid_open);
-static int acpi_lid_update_state(struct acpi_device *device,
+static int acpi_lid_update_state(struct acpi_button *button,
bool signal_wakeup)
{
+ struct acpi_device *device = button->adev;
int state;
state = acpi_lid_evaluate_state(device);
@@ -395,21 +398,19 @@ static int acpi_lid_update_state(struct acpi_device *device,
return state;
if (state && signal_wakeup)
- acpi_pm_wakeup_event(&device->dev);
+ acpi_pm_wakeup_event(&button->pdev->dev);
- return acpi_lid_notify_state(device, state);
+ return acpi_lid_notify_state(button, state);
}
-static void acpi_lid_initialize_state(struct acpi_device *device)
+static void acpi_lid_initialize_state(struct acpi_button *button)
{
- struct acpi_button *button = acpi_driver_data(device);
-
switch (lid_init_state) {
case ACPI_BUTTON_LID_INIT_OPEN:
- (void)acpi_lid_notify_state(device, 1);
+ (void)acpi_lid_notify_state(button, 1);
break;
case ACPI_BUTTON_LID_INIT_METHOD:
- (void)acpi_lid_update_state(device, false);
+ (void)acpi_lid_update_state(button, false);
break;
case ACPI_BUTTON_LID_INIT_IGNORE:
default:
@@ -421,8 +422,8 @@ static void acpi_lid_initialize_state(struct acpi_device *device)
static void acpi_lid_notify(acpi_handle handle, u32 event, void *data)
{
- struct acpi_device *device = data;
- struct acpi_button *button;
+ struct acpi_button *button = data;
+ struct acpi_device *device = button->adev;
if (event != ACPI_BUTTON_NOTIFY_STATUS) {
acpi_handle_debug(device->handle, "Unsupported event [0x%x]\n",
@@ -430,17 +431,16 @@ static void acpi_lid_notify(acpi_handle handle, u32 event, void *data)
return;
}
- button = acpi_driver_data(device);
if (!button->lid_state_initialized)
return;
- acpi_lid_update_state(device, true);
+ acpi_lid_update_state(button, true);
}
static void acpi_button_notify(acpi_handle handle, u32 event, void *data)
{
- struct acpi_device *device = data;
- struct acpi_button *button;
+ struct acpi_button *button = data;
+ struct acpi_device *device = button->adev;
struct input_dev *input;
int keycode;
@@ -455,9 +455,8 @@ static void acpi_button_notify(acpi_handle handle, u32 event, void *data)
return;
}
- acpi_pm_wakeup_event(&device->dev);
+ acpi_pm_wakeup_event(&button->pdev->dev);
- button = acpi_driver_data(device);
if (button->suspended || event == ACPI_BUTTON_NOTIFY_WAKE)
return;
@@ -488,8 +487,7 @@ static u32 acpi_button_event(void *data)
#ifdef CONFIG_PM_SLEEP
static int acpi_button_suspend(struct device *dev)
{
- struct acpi_device *device = to_acpi_device(dev);
- struct acpi_button *button = acpi_driver_data(device);
+ struct acpi_button *button = dev_get_drvdata(dev);
button->suspended = true;
return 0;
@@ -497,15 +495,15 @@ static int acpi_button_suspend(struct device *dev)
static int acpi_button_resume(struct device *dev)
{
+ struct acpi_button *button = dev_get_drvdata(dev);
+ struct acpi_device *device = ACPI_COMPANION(dev);
struct input_dev *input;
- struct acpi_device *device = to_acpi_device(dev);
- struct acpi_button *button = acpi_driver_data(device);
button->suspended = false;
if (button->type == ACPI_BUTTON_TYPE_LID) {
button->last_state = !!acpi_lid_evaluate_state(device);
button->last_time = ktime_get();
- acpi_lid_initialize_state(device);
+ acpi_lid_initialize_state(button);
}
if (button->type == ACPI_BUTTON_TYPE_POWER) {
@@ -521,18 +519,19 @@ static int acpi_button_resume(struct device *dev)
static int acpi_lid_input_open(struct input_dev *input)
{
- struct acpi_device *device = input_get_drvdata(input);
- struct acpi_button *button = acpi_driver_data(device);
+ struct acpi_button *button = input_get_drvdata(input);
+ struct acpi_device *device = button->adev;
button->last_state = !!acpi_lid_evaluate_state(device);
button->last_time = ktime_get();
- acpi_lid_initialize_state(device);
+ acpi_lid_initialize_state(button);
return 0;
}
-static int acpi_button_add(struct acpi_device *device)
+static int acpi_button_probe(struct platform_device *pdev)
{
+ struct acpi_device *device = ACPI_COMPANION(&pdev->dev);
acpi_notify_handler handler;
struct acpi_button *button;
struct input_dev *input;
@@ -549,8 +548,10 @@ static int acpi_button_add(struct acpi_device *device)
if (!button)
return -ENOMEM;
- device->driver_data = button;
+ platform_set_drvdata(pdev, button);
+ button->pdev = pdev;
+ button->adev = device;
button->input = input = input_allocate_device();
if (!input) {
error = -ENOMEM;
@@ -587,7 +588,7 @@ static int acpi_button_add(struct acpi_device *device)
}
if (!error)
- error = acpi_button_add_fs(device);
+ error = acpi_button_add_fs(button);
if (error) {
input_free_device(input);
@@ -600,7 +601,7 @@ static int acpi_button_add(struct acpi_device *device)
input->phys = button->phys;
input->id.bustype = BUS_HOST;
input->id.product = button->type;
- input->dev.parent = &device->dev;
+ input->dev.parent = &pdev->dev;
switch (button->type) {
case ACPI_BUTTON_TYPE_POWER:
@@ -617,7 +618,7 @@ static int acpi_button_add(struct acpi_device *device)
break;
}
- input_set_drvdata(input, device);
+ input_set_drvdata(input, button);
error = input_register_device(input);
if (error) {
input_free_device(input);
@@ -628,17 +629,17 @@ static int acpi_button_add(struct acpi_device *device)
case ACPI_BUS_TYPE_POWER_BUTTON:
status = acpi_install_fixed_event_handler(ACPI_EVENT_POWER_BUTTON,
acpi_button_event,
- device);
+ button);
break;
case ACPI_BUS_TYPE_SLEEP_BUTTON:
status = acpi_install_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON,
acpi_button_event,
- device);
+ button);
break;
default:
status = acpi_install_notify_handler(device->handle,
ACPI_ALL_NOTIFY, handler,
- device);
+ button);
break;
}
if (ACPI_FAILURE(status)) {
@@ -654,24 +655,25 @@ static int acpi_button_add(struct acpi_device *device)
lid_device = device;
}
- device_init_wakeup(&device->dev, true);
+ device_init_wakeup(&pdev->dev, true);
pr_info("%s [%s]\n", name, acpi_device_bid(device));
return 0;
err_input_unregister:
input_unregister_device(input);
err_remove_fs:
- acpi_button_remove_fs(device);
+ acpi_button_remove_fs(button);
err_free_button:
kfree(button);
return error;
}
-static void acpi_button_remove(struct acpi_device *device)
+static void acpi_button_remove(struct platform_device *pdev)
{
- struct acpi_button *button = acpi_driver_data(device);
+ struct acpi_button *button = platform_get_drvdata(pdev);
+ struct acpi_device *adev = button->adev;
- switch (device->device_type) {
+ switch (adev->device_type) {
case ACPI_BUS_TYPE_POWER_BUTTON:
acpi_remove_fixed_event_handler(ACPI_EVENT_POWER_BUTTON,
acpi_button_event);
@@ -681,7 +683,7 @@ static void acpi_button_remove(struct acpi_device *device)
acpi_button_event);
break;
default:
- acpi_remove_notify_handler(device->handle, ACPI_DEVICE_NOTIFY,
+ acpi_remove_notify_handler(adev->handle, ACPI_ALL_NOTIFY,
button->type == ACPI_BUTTON_TYPE_LID ?
acpi_lid_notify :
acpi_button_notify);
@@ -689,7 +691,7 @@ static void acpi_button_remove(struct acpi_device *device)
}
acpi_os_wait_events_complete();
- acpi_button_remove_fs(device);
+ acpi_button_remove_fs(button);
input_unregister_device(button->input);
kfree(button);
}
@@ -728,7 +730,7 @@ module_param_call(lid_init_state,
NULL, 0644);
MODULE_PARM_DESC(lid_init_state, "Behavior for reporting LID initial state");
-static int acpi_button_register_driver(struct acpi_driver *driver)
+static int __init acpi_button_init(void)
{
const struct dmi_system_id *dmi_id;
@@ -744,20 +746,20 @@ static int acpi_button_register_driver(struct acpi_driver *driver)
* Modules such as nouveau.ko and i915.ko have a link time dependency
* on acpi_lid_open(), and would therefore not be loadable on ACPI
* capable kernels booted in non-ACPI mode if the return value of
- * acpi_bus_register_driver() is returned from here with ACPI disabled
+ * platform_driver_register() is returned from here with ACPI disabled
* when this driver is built as a module.
*/
if (acpi_disabled)
return 0;
- return acpi_bus_register_driver(driver);
+ return platform_driver_register(&acpi_button_driver);
}
-static void acpi_button_unregister_driver(struct acpi_driver *driver)
+static void __exit acpi_button_exit(void)
{
if (!acpi_disabled)
- acpi_bus_unregister_driver(driver);
+ platform_driver_unregister(&acpi_button_driver);
}
-module_driver(acpi_button_driver, acpi_button_register_driver,
- acpi_button_unregister_driver);
+module_init(acpi_button_init);
+module_exit(acpi_button_exit);
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 32fa1c1ded60..7f9dd2704f63 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1005,15 +1005,11 @@ static int acpi_bus_extract_wakeup_device_power_package(struct acpi_device *dev)
return err;
}
-/* Do not use a button for S5 wakeup */
-#define ACPI_AVOID_WAKE_FROM_S5 BIT(0)
-
static bool acpi_wakeup_gpe_init(struct acpi_device *device)
{
static const struct acpi_device_id button_device_ids[] = {
- {"PNP0C0C", 0}, /* Power button */
- {"PNP0C0D", ACPI_AVOID_WAKE_FROM_S5}, /* Lid */
- {"PNP0C0E", ACPI_AVOID_WAKE_FROM_S5}, /* Sleep button */
+ {"PNP0C0D", 0}, /* Lid */
+ {"PNP0C0E", 0}, /* Sleep button */
{"", 0},
};
struct acpi_device_wakeup *wakeup = &device->wakeup;
@@ -1022,16 +1018,9 @@ static bool acpi_wakeup_gpe_init(struct acpi_device *device)
wakeup->flags.notifier_present = 0;
- /* Power button, Lid switch always enable wakeup */
match = acpi_match_acpi_device(button_device_ids, device);
- if (match) {
- if ((match->driver_data & ACPI_AVOID_WAKE_FROM_S5) &&
- wakeup->sleep_state == ACPI_STATE_S5)
- wakeup->sleep_state = ACPI_STATE_S4;
- acpi_mark_gpe_for_wake(wakeup->gpe_device, wakeup->gpe_number);
- device_set_wakeup_capable(&device->dev, true);
- return true;
- }
+ if (match && wakeup->sleep_state == ACPI_STATE_S5)
+ wakeup->sleep_state = ACPI_STATE_S4;
status = acpi_setup_gpe_for_wake(device->handle, wakeup->gpe_device,
wakeup->gpe_number);
diff --git a/drivers/android/binder/node.rs b/drivers/android/binder/node.rs
index b76ecff80f91..ee798e9d6c69 100644
--- a/drivers/android/binder/node.rs
+++ b/drivers/android/binder/node.rs
@@ -49,9 +49,9 @@
/// about to drop the weak reference, then the strong increment could be processed after the
/// other thread has already exited, which would be too late.
///
-/// Note that trying to create a `ListArc` to the node can succeed even if `has_normal_push` is
+/// Note that trying to create a `ListArc` to the node can succeed even if `has_pushed_node` is
/// set. This is because another thread might just have popped the node from a todo list, but not
-/// yet called `do_work`. However, if `has_normal_push` is false, then creating a `ListArc` should
+/// yet called `do_work`. However, if `has_pushed_node` is false, then creating a `ListArc` should
/// always succeed.
///
/// Like the other fields in `NodeInner`, the delivery state is protected by the process lock.
@@ -725,7 +725,21 @@ fn do_work(
self.do_work_locked(writer, owner_inner)
}
- fn cancel(self: DArc<Self>) {}
+ fn cancel(self: DArc<Self>) {
+ let _drop_outside_lock;
+ let mut owner_inner = self.owner.inner.lock();
+
+ // We only do something on BINDER_THREAD_EXIT, not process exit.
+ if owner_inner.is_dead {
+ return;
+ }
+
+ // If BINDER_THREAD_EXIT is invoked on a thread with a pending node refcount update, we
+ // should move ourselves to ensure the refcount update is still delivered.
+ if let Some(node) = ListArc::try_from_arc_borrow(self.as_arc_borrow()) {
+ _drop_outside_lock = owner_inner.push_work(node);
+ }
+ }
fn should_sync_wakeup(&self) -> bool {
false
diff --git a/drivers/android/binder/node/wrapper.rs b/drivers/android/binder/node/wrapper.rs
index 43294c050502..aeda07ff7218 100644
--- a/drivers/android/binder/node/wrapper.rs
+++ b/drivers/android/binder/node/wrapper.rs
@@ -57,7 +57,39 @@ fn do_work(
node.do_work_locked(writer, owner_inner)
}
- fn cancel(self: DArc<Self>) {}
+ fn cancel(self: DArc<Self>) {
+ let _drop_outside_lock;
+ let node = &self.node;
+ let mut owner_inner = node.owner.inner.lock();
+
+ // We only do something on BINDER_THREAD_EXIT, not process exit.
+ if owner_inner.is_dead {
+ return;
+ }
+
+ // We transfer the responsibility of the node refcount update to the scheduled Node because
+ // NodeWrapper has no way to re-create the ListArc.
+ let inner = node.inner.access_mut(&mut owner_inner);
+
+ let ds = &mut inner.delivery_state;
+ assert!(ds.has_pushed_wrapper);
+ assert!(ds.has_strong_zero2one);
+ ds.has_pushed_wrapper = false;
+
+ // We are changing the state to one where the Node is the strong zero2one update instead of
+ // the wrapper.
+ ds.has_weak_zero2one = false;
+
+ if !ds.has_pushed_node {
+ if let Some(node2) = ListArc::try_from_arc_borrow(node.as_arc_borrow()) {
+ ds.has_pushed_node = true;
+ _drop_outside_lock = owner_inner.push_work(node2);
+ } else {
+ // This can't actually happen.
+ ds.has_strong_zero2one = false;
+ }
+ }
+ }
fn should_sync_wakeup(&self) -> bool {
false
diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index ef7f93aca36d..cd9c96560ddd 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -4281,6 +4281,13 @@ static const struct ata_dev_quirks_entry __ata_dev_quirks[] = {
/* Apacer models with LPM issues */
{ "Apacer AS340*", NULL, ATA_QUIRK_NOLPM },
+ /* PNY CS900 (Phison PS3111-S11, DRAM-less) drops the link on DIPM */
+ { "PNY CS900 1TB SSD", NULL, ATA_QUIRK_NOLPM },
+
+ /* Silicon Motion models with LPM issues */
+ { "MD619HXCLDE3TC", "TCVAID", ATA_QUIRK_NOLPM },
+ { "MD619GXCLDE3TC", "TCV35D", ATA_QUIRK_NOLPM },
+
/* These specific Samsung models/firmware-revs do not handle LPM well */
{ "SAMSUNG MZMPC128HBFU-000MV", "CXM14M1Q", ATA_QUIRK_NOLPM },
{ "SAMSUNG SSD PM830 mSATA *", "CXM13D1Q", ATA_QUIRK_NOLPM },
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index 14e68ca9784b..c780a7f15738 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -3018,8 +3018,14 @@ static void atapi_qc_complete(struct ata_queued_cmd *qc)
if (qc->cdb[0] == ALLOW_MEDIUM_REMOVAL && qc->dev->sdev)
qc->dev->sdev->locked = 0;
- if (!cmd->result)
- cmd->result = SAM_STAT_CHECK_CONDITION;
+ /*
+ * Report CHECK CONDITION unless the command was terminated
+ * with a host byte set. Only set the status byte, so that the
+ * SCSI midlayer internal byte, which scsi_check_sense() may
+ * have set from the sense data, is preserved.
+ */
+ if (get_host_byte(cmd) == DID_OK)
+ set_status_byte(cmd, SAM_STAT_CHECK_CONDITION);
ata_qc_done(qc);
return;
}
diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index acac73b75c7c..ede589d1149e 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -450,7 +450,7 @@ static struct socket *drbd_try_connect(struct drbd_connection *connection)
* a free one dynamically.
*/
what = "bind before connect";
- err = sock->ops->bind(sock, (struct sockaddr *) &src_in6, my_addr_len);
+ err = sock->ops->bind(sock, (struct sockaddr_unsized *) &src_in6, my_addr_len);
if (err < 0)
goto out;
@@ -537,7 +537,7 @@ static int prepare_listen_socket(struct drbd_connection *connection, struct acce
drbd_setbufsize(s_listen, sndbuf_size, rcvbuf_size);
what = "bind before listen";
- err = s_listen->ops->bind(s_listen, (struct sockaddr *)&my_addr, my_addr_len);
+ err = s_listen->ops->bind(s_listen, (struct sockaddr_unsized *)&my_addr, my_addr_len);
if (err < 0)
goto out;
diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index a11805b29b63..f44512ed1217 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -2141,30 +2141,28 @@ static void ublk_stop_dev(struct ublk_device *ub)
/* reset ublk io_uring queue & io flags */
static void ublk_reset_io_flags(struct ublk_device *ub)
{
- int i, j;
-
- for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
- struct ublk_queue *ubq = ublk_get_queue(ub, i);
+ int i;
- /* UBLK_IO_FLAG_CANCELED can be cleared now */
- spin_lock(&ubq->cancel_lock);
- for (j = 0; j < ubq->q_depth; j++)
- ubq->ios[j].flags &= ~UBLK_IO_FLAG_CANCELED;
- spin_unlock(&ubq->cancel_lock);
- ubq->fail_io = false;
- }
+ for (i = 0; i < ub->dev_info.nr_hw_queues; i++)
+ ublk_get_queue(ub, i)->fail_io = false;
mutex_lock(&ub->cancel_mutex);
ublk_set_canceling(ub, false);
mutex_unlock(&ub->cancel_mutex);
}
/* device can only be started after all IOs are ready */
-static void ublk_mark_io_ready(struct ublk_device *ub)
+static void ublk_mark_io_ready(struct ublk_device *ub, struct ublk_queue *ubq,
+ struct ublk_io *io)
__must_hold(&ub->mutex)
{
if (!ub->unprivileged_daemons && !capable(CAP_SYS_ADMIN))
ub->unprivileged_daemons = true;
+ /* UBLK_IO_FLAG_CANCELED can be cleared now */
+ spin_lock(&ubq->cancel_lock);
+ io->flags &= ~UBLK_IO_FLAG_CANCELED;
+ spin_unlock(&ubq->cancel_lock);
+
ub->nr_io_ready++;
if (ublk_dev_ready(ub)) {
/*
@@ -2190,25 +2188,34 @@ static inline int ublk_check_cmd_op(u32 cmd_op)
return 0;
}
-static inline int ublk_set_auto_buf_reg(struct ublk_io *io, struct io_uring_cmd *cmd)
+/* Must run before ublk_fill_io_cmd(). */
+static inline int ublk_validate_io_buf(const struct ublk_device *ub,
+ struct io_uring_cmd *cmd,
+ struct ublk_auto_buf_reg *buf)
{
- struct ublk_auto_buf_reg buf;
-
- buf = ublk_sqe_addr_to_auto_buf_reg(READ_ONCE(cmd->sqe->addr));
+ if (!ublk_dev_support_auto_buf_reg(ub))
+ return 0;
- if (buf.reserved0 || buf.reserved1)
+ *buf = ublk_sqe_addr_to_auto_buf_reg(READ_ONCE(cmd->sqe->addr));
+ if (buf->reserved0 || buf->reserved1)
return -EINVAL;
-
- if (buf.flags & ~UBLK_AUTO_BUF_REG_F_MASK)
+ if (buf->flags & ~UBLK_AUTO_BUF_REG_F_MASK)
return -EINVAL;
- io->buf.auto_reg = buf;
return 0;
}
-static int ublk_handle_auto_buf_reg(struct ublk_io *io,
- struct io_uring_cmd *cmd,
- u16 *buf_idx)
+static inline void ublk_apply_io_buf(const struct ublk_device *ub,
+ struct ublk_io *io,
+ struct io_uring_cmd *cmd,
+ unsigned long buf_addr,
+ const struct ublk_auto_buf_reg *auto_buf,
+ u16 *buf_idx)
{
+ if (!ublk_dev_support_auto_buf_reg(ub)) {
+ io->buf.addr = buf_addr;
+ return;
+ }
+
if (io->flags & UBLK_IO_FLAG_AUTO_BUF_REG) {
io->flags &= ~UBLK_IO_FLAG_AUTO_BUF_REG;
@@ -2222,11 +2229,12 @@ static int ublk_handle_auto_buf_reg(struct ublk_io *io,
* responsibility for unregistering the buffer, otherwise
* this ublk request gets stuck.
*/
- if (io->buf_ctx_handle == io_uring_cmd_ctx_handle(cmd))
+ if (buf_idx &&
+ io->buf_ctx_handle == io_uring_cmd_ctx_handle(cmd))
*buf_idx = io->buf.auto_reg.index;
}
- return ublk_set_auto_buf_reg(io, cmd);
+ io->buf.auto_reg = *auto_buf;
}
/* Once we return, `io->req` can't be used any more */
@@ -2243,18 +2251,6 @@ ublk_fill_io_cmd(struct ublk_io *io, struct io_uring_cmd *cmd)
return req;
}
-static inline int
-ublk_config_io_buf(const struct ublk_device *ub, struct ublk_io *io,
- struct io_uring_cmd *cmd, unsigned long buf_addr,
- u16 *buf_idx)
-{
- if (ublk_dev_support_auto_buf_reg(ub))
- return ublk_handle_auto_buf_reg(io, cmd, buf_idx);
-
- io->buf.addr = buf_addr;
- return 0;
-}
-
static inline void ublk_prep_cancel(struct io_uring_cmd *cmd,
unsigned int issue_flags,
struct ublk_queue *ubq, unsigned int tag)
@@ -2370,9 +2366,10 @@ static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr)
}
static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
- struct ublk_io *io, __u64 buf_addr)
+ struct ublk_queue *ubq, struct ublk_io *io, __u64 buf_addr)
{
- int ret = 0;
+ struct ublk_auto_buf_reg auto_buf;
+ int ret;
/*
* When handling FETCH command for setting up ublk uring queue,
@@ -2380,6 +2377,10 @@ static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
* FETCH, so it is fine even for IO_URING_F_NONBLOCK.
*/
mutex_lock(&ub->mutex);
+ ret = ublk_validate_io_buf(ub, cmd, &auto_buf);
+ if (ret)
+ goto out;
+
/* UBLK_IO_FETCH_REQ is only allowed before dev is setup */
if (ublk_dev_ready(ub)) {
ret = -EBUSY;
@@ -2395,12 +2396,10 @@ static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
WARN_ON_ONCE(io->flags & UBLK_IO_FLAG_OWNED_BY_SRV);
ublk_fill_io_cmd(io, cmd);
- ret = ublk_config_io_buf(ub, io, cmd, buf_addr, NULL);
- if (ret)
- goto out;
+ ublk_apply_io_buf(ub, io, cmd, buf_addr, &auto_buf, NULL);
WRITE_ONCE(io->task, get_task_struct(current));
- ublk_mark_io_ready(ub);
+ ublk_mark_io_ready(ub, ubq, io);
out:
mutex_unlock(&ub->mutex);
return ret;
@@ -2505,7 +2504,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
ret = ublk_check_fetch_buf(ub, addr);
if (ret)
goto out;
- ret = ublk_fetch(cmd, ub, io, addr);
+ ret = ublk_fetch(cmd, ub, ubq, io, addr);
if (ret)
goto out;
@@ -2543,13 +2542,18 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
case UBLK_IO_REGISTER_IO_BUF:
return ublk_daemon_register_io_buf(cmd, ub, q_id, tag, io, addr,
issue_flags);
- case UBLK_IO_COMMIT_AND_FETCH_REQ:
+ case UBLK_IO_COMMIT_AND_FETCH_REQ: {
+ struct ublk_auto_buf_reg auto_buf;
+
ret = ublk_check_commit_and_fetch(ub, io, addr);
+ if (ret)
+ goto out;
+ ret = ublk_validate_io_buf(ub, cmd, &auto_buf);
if (ret)
goto out;
io->res = result;
req = ublk_fill_io_cmd(io, cmd);
- ret = ublk_config_io_buf(ub, io, cmd, addr, &buf_idx);
+ ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx);
if (buf_idx != UBLK_INVALID_BUF_IDX)
io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
compl = ublk_need_complete_req(ub, io);
@@ -2559,10 +2563,8 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
req->__sector = addr;
if (compl)
__ublk_complete_rq(req, io, ublk_dev_need_map_io(ub));
-
- if (ret)
- goto out;
break;
+ }
case UBLK_IO_NEED_GET_DATA:
/*
* ublk_get_data() may fail and fallback to requeue, so keep
@@ -2570,8 +2572,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
* request
*/
req = ublk_fill_io_cmd(io, cmd);
- ret = ublk_config_io_buf(ub, io, cmd, addr, NULL);
- WARN_ON_ONCE(ret);
+ io->buf.addr = addr;
if (likely(ublk_get_data(ubq, io, req))) {
__ublk_prep_compl_io_cmd(io, req);
return UBLK_IO_RES_OK;
diff --git a/drivers/block/zloop.c b/drivers/block/zloop.c
index 45088c163286..9d21259e9731 100644
--- a/drivers/block/zloop.c
+++ b/drivers/block/zloop.c
@@ -568,6 +568,27 @@ static void zloop_cmd_workfn(struct work_struct *work)
current->flags = orig_flags;
}
+static void zloop_fill_zero_rq(struct request *rq, unsigned int zero_from)
+{
+ struct req_iterator iter;
+ struct bio_vec bv;
+ unsigned int pos = 0;
+ unsigned int end, off;
+
+ rq_for_each_segment(bv, rq, iter) {
+ end = pos + bv.bv_len;
+ if (end > zero_from) {
+ if (pos < zero_from)
+ off = zero_from - pos;
+ else
+ off = 0;
+ memzero_page(bv.bv_page, bv.bv_offset + off,
+ bv.bv_len - off);
+ }
+ pos = end;
+ }
+}
+
static void zloop_complete_rq(struct request *rq)
{
struct zloop_cmd *cmd = blk_mq_rq_to_pdu(rq);
@@ -582,13 +603,9 @@ static void zloop_complete_rq(struct request *rq)
pr_err("Zone %u: failed read sector %llu, %llu sectors\n",
zone_no, cmd->sector, cmd->nr_sectors);
- if (cmd->ret >= 0 && cmd->ret != blk_rq_bytes(rq)) {
- /* short read */
- struct bio *bio;
-
- __rq_for_each_bio(bio, rq)
- zero_fill_bio(bio);
- }
+ /* Short read. Zero out the unread tail beyond cmd->ret. */
+ if (cmd->ret >= 0 && cmd->ret != blk_rq_bytes(rq))
+ zloop_fill_zero_rq(rq, cmd->ret);
break;
case REQ_OP_WRITE:
case REQ_OP_ZONE_APPEND:
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 5a08e8bebd6d..ea5a9b81a310 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -948,8 +948,8 @@ static ssize_t writeback_store(struct device *dev,
const char *buf, size_t len)
{
struct zram *zram = dev_to_zram(dev);
- u64 nr_pages = zram->disksize >> PAGE_SHIFT;
- unsigned long lo = 0, hi = nr_pages;
+ u64 nr_pages;
+ unsigned long lo = 0, hi;
struct zram_pp_ctl *ctl = NULL;
char *args, *param, *val;
ssize_t ret = len;
@@ -972,6 +972,9 @@ static ssize_t writeback_store(struct device *dev,
goto release_init_lock;
}
+ nr_pages = zram->disksize >> PAGE_SHIFT;
+ hi = nr_pages;
+
ctl = init_pp_ctl();
if (!ctl) {
ret = -ENOMEM;
diff --git a/drivers/bluetooth/btintel_pcie.c b/drivers/bluetooth/btintel_pcie.c
index cee774032106..3e8c894fbf64 100644
--- a/drivers/bluetooth/btintel_pcie.c
+++ b/drivers/bluetooth/btintel_pcie.c
@@ -358,7 +358,7 @@ static int btintel_pcie_send_sync(struct btintel_pcie_data *data,
tfd_index = data->ia.tr_hia[BTINTEL_PCIE_TXQ_NUM];
- if (tfd_index > txq->count)
+ if (tfd_index >= txq->count)
return -ERANGE;
if (skb->len > BTINTEL_PCIE_BUFFER_SIZE - BTINTEL_PCIE_HCI_TYPE_LEN) {
@@ -1540,7 +1540,7 @@ static irqreturn_t btintel_pcie_irq_msix_handler(int irq, void *dev_id)
static int btintel_pcie_setup_irq(struct btintel_pcie_data *data)
{
int err;
- int num_irqs, i;
+ int num_irqs, i, j;
for (i = 0; i < BTINTEL_PCIE_MSIX_VEC_MAX; i++)
data->msix_entries[i].entry = i;
@@ -1569,6 +1569,10 @@ static int btintel_pcie_setup_irq(struct btintel_pcie_data *data)
KBUILD_MODNAME,
msix_entry);
if (err) {
+ for (j = 0; j < i; j++)
+ devm_free_irq(&data->pdev->dev,
+ data->msix_entries[j].vector,
+ &data->msix_entries[j]);
pci_free_irq_vectors(data->pdev);
data->alloc_vecs = 0;
return err;
@@ -1639,6 +1643,13 @@ static int btintel_pcie_config_pcie(struct pci_dev *pdev,
if (IS_ERR(data->base_addr))
return PTR_ERR(data->base_addr);
+ /* Do shared hardware reset to ensure a clean start before
+ * configuring interrupts.
+ */
+ err = btintel_pcie_reset_bt(data);
+ if (err)
+ return err;
+
err = btintel_pcie_setup_irq(data);
if (err)
return err;
@@ -2531,7 +2542,7 @@ static void btintel_pcie_remove(struct pci_dev *pdev)
struct msix_entry *msix_entry;
msix_entry = &data->msix_entries[i];
- free_irq(msix_entry->vector, msix_entry);
+ devm_free_irq(&pdev->dev, msix_entry->vector, msix_entry);
}
pci_free_irq_vectors(pdev);
diff --git a/drivers/clocksource/samsung_pwm_timer.c b/drivers/clocksource/samsung_pwm_timer.c
index b9561e3f196c..0544124cf5ce 100644
--- a/drivers/clocksource/samsung_pwm_timer.c
+++ b/drivers/clocksource/samsung_pwm_timer.c
@@ -56,7 +56,7 @@
#define TCON_AUTORELOAD(chan) \
((chan < 5) ? _TCON_AUTORELOAD(chan) : _TCON_AUTORELOAD4(chan))
-DEFINE_SPINLOCK(samsung_pwm_lock);
+DEFINE_RAW_SPINLOCK(samsung_pwm_lock);
EXPORT_SYMBOL(samsung_pwm_lock);
struct samsung_pwm_clocksource {
@@ -87,14 +87,14 @@ static void samsung_timer_set_prescale(unsigned int channel, u16 prescale)
if (channel >= 2)
shift = TCFG0_PRESCALER1_SHIFT;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(pwm.base + REG_TCFG0);
reg &= ~(TCFG0_PRESCALER_MASK << shift);
reg |= (prescale - 1) << shift;
writel(reg, pwm.base + REG_TCFG0);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_timer_set_divisor(unsigned int channel, u8 divisor)
@@ -106,14 +106,14 @@ static void samsung_timer_set_divisor(unsigned int channel, u8 divisor)
bits = (fls(divisor) - 1) - pwm.variant.div_base;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(pwm.base + REG_TCFG1);
reg &= ~(TCFG1_MUX_MASK << shift);
reg |= bits << shift;
writel(reg, pwm.base + REG_TCFG1);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_stop(unsigned int channel)
@@ -124,13 +124,13 @@ static void samsung_time_stop(unsigned int channel)
if (channel > 0)
++channel;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
tcon &= ~TCON_START(channel);
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
@@ -142,7 +142,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
if (tcon_chan > 0)
++tcon_chan;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
@@ -153,7 +153,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
writel_relaxed(tcnt, pwm.base + REG_TCMPB(channel));
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_start(unsigned int channel, bool periodic)
@@ -164,7 +164,7 @@ static void samsung_time_start(unsigned int channel, bool periodic)
if (channel > 0)
++channel;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
@@ -178,7 +178,7 @@ static void samsung_time_start(unsigned int channel, bool periodic)
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int samsung_set_next_event(unsigned long cycles,
diff --git a/drivers/cxl/core/features.c b/drivers/cxl/core/features.c
index 6e25223d3409..bb73f205f16c 100644
--- a/drivers/cxl/core/features.c
+++ b/drivers/cxl/core/features.c
@@ -239,6 +239,8 @@ size_t cxl_get_feature(struct cxl_mailbox *cxl_mbox, const uuid_t *feat_uuid,
uuid_copy(&pi.uuid, feat_uuid);
pi.selection = selection;
+
+ guard(mutex)(&cxl_mbox->feat_mutex);
do {
data_to_rd_size = min(feat_out_size - data_rcvd_size,
cxl_mbox->payload_size);
@@ -313,6 +315,7 @@ int cxl_set_feature(struct cxl_mailbox *cxl_mbox,
data_in_size = cxl_mbox->payload_size - hdr_size;
}
+ guard(mutex)(&cxl_mbox->feat_mutex);
do {
int rc;
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index e1c23aae2a2e..76731576ebfc 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -1515,6 +1515,7 @@ int cxl_mailbox_init(struct cxl_mailbox *cxl_mbox, struct device *host)
cxl_mbox->host = host;
mutex_init(&cxl_mbox->mbox_mutex);
+ mutex_init(&cxl_mbox->feat_mutex);
rcuwait_init(&cxl_mbox->mbox_wait);
return 0;
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 2b9e166ea4e0..8d2dc4b74178 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1995,7 +1995,7 @@ struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port,
return ERR_PTR(rc);
}
- mutex_init(&cxlrd->range_lock);
+ mutex_init(&cxlrd->regions_lock);
cxld = &cxlsd->cxld;
cxld->dev.type = &cxl_decoder_root_type;
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index e840ac002f9b..148168319a26 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -2629,6 +2629,10 @@ static ssize_t create_region_store(struct device *dev, const char *buf,
if (rc != 1)
return -EINVAL;
+ ACQUIRE(mutex_intr, regions_lock)(&cxlrd->regions_lock);
+ if ((rc = ACQUIRE_ERR(mutex_intr, ®ions_lock)))
+ return rc;
+
cxlr = __create_region(cxlrd, mode, id);
if (IS_ERR(cxlr))
return PTR_ERR(cxlr);
@@ -2688,6 +2692,11 @@ static ssize_t delete_region_store(struct device *dev,
struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev);
struct cxl_port *port = to_cxl_port(dev->parent);
struct cxl_region *cxlr;
+ int rc;
+
+ ACQUIRE(mutex_intr, regions_lock)(&cxlrd->regions_lock);
+ if ((rc = ACQUIRE_ERR(mutex_intr, ®ions_lock)))
+ return rc;
cxlr = cxl_find_region_by_name(cxlrd, buf);
if (IS_ERR(cxlr))
@@ -3575,12 +3584,11 @@ int cxl_add_to_region(struct cxl_endpoint_decoder *cxled)
* Ensure that if multiple threads race to construct_region() for @hpa
* one does the construction and the others add to that.
*/
- mutex_lock(&cxlrd->range_lock);
+ guard(mutex)(&cxlrd->regions_lock);
struct cxl_region *cxlr __free(put_cxl_region) =
cxl_find_region_by_range(cxlrd, hpa);
if (!cxlr)
cxlr = construct_region(cxlrd, cxled);
- mutex_unlock(&cxlrd->range_lock);
rc = PTR_ERR_OR_ZERO(cxlr);
if (rc)
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 091c52111d94..b7933a892af7 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -449,7 +449,7 @@ struct cxl_rd_ops {
* @cache_size: extended linear cache size if exists, otherwise zero.
* @region_id: region id for next region provisioning event
* @platform_data: platform specific configuration data
- * @range_lock: sync region autodiscovery by address range
+ * @regions_lock: sync region discovery, construction, and deletion
* @qos_class: QoS performance class cookie
* @ops: CXL root decoder operations
* @cxlsd: base cxl switch decoder
@@ -459,7 +459,7 @@ struct cxl_root_decoder {
resource_size_t cache_size;
atomic_t region_id;
void *platform_data;
- struct mutex range_lock;
+ struct mutex regions_lock;
int qos_class;
struct cxl_rd_ops *ops;
struct cxl_switch_decoder cxlsd;
diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
index afba88f9c3e4..ab4efebb3862 100644
--- a/drivers/dma/idxd/init.c
+++ b/drivers/dma/idxd/init.c
@@ -159,18 +159,12 @@ static void idxd_cleanup_interrupts(struct idxd_device *idxd)
static void idxd_clean_wqs(struct idxd_device *idxd)
{
- struct idxd_wq *wq;
struct device *conf_dev;
int i;
for (i = 0; i < idxd->max_wqs; i++) {
- wq = idxd->wqs[i];
- if (idxd->hw.wq_cap.op_config)
- bitmap_free(wq->opcap_bmap);
- kfree(wq->wqcfg);
- conf_dev = wq_confdev(wq);
+ conf_dev = wq_confdev(idxd->wqs[i]);
put_device(conf_dev);
- kfree(wq);
}
bitmap_free(idxd->wq_enable_map);
kfree(idxd->wqs);
@@ -212,7 +206,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "wq%d.%d", idxd->id, wq->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(wq);
goto err_unwind;
}
@@ -226,7 +219,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
wq->wqcfg = kzalloc_node(idxd->wqcfg_size, GFP_KERNEL, dev_to_node(dev));
if (!wq->wqcfg) {
put_device(conf_dev);
- kfree(wq);
rc = -ENOMEM;
goto err_unwind;
}
@@ -234,9 +226,7 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
if (idxd->hw.wq_cap.op_config) {
wq->opcap_bmap = bitmap_zalloc(IDXD_MAX_OPCAP_BITS, GFP_KERNEL);
if (!wq->opcap_bmap) {
- kfree(wq->wqcfg);
put_device(conf_dev);
- kfree(wq);
rc = -ENOMEM;
goto err_unwind;
}
@@ -251,13 +241,8 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
err_unwind:
while (--i >= 0) {
- wq = idxd->wqs[i];
- if (idxd->hw.wq_cap.op_config)
- bitmap_free(wq->opcap_bmap);
- kfree(wq->wqcfg);
- conf_dev = wq_confdev(wq);
+ conf_dev = wq_confdev(idxd->wqs[i]);
put_device(conf_dev);
- kfree(wq);
}
bitmap_free(idxd->wq_enable_map);
@@ -269,15 +254,12 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
static void idxd_clean_engines(struct idxd_device *idxd)
{
- struct idxd_engine *engine;
struct device *conf_dev;
int i;
for (i = 0; i < idxd->max_engines; i++) {
- engine = idxd->engines[i];
- conf_dev = engine_confdev(engine);
+ conf_dev = engine_confdev(idxd->engines[i]);
put_device(conf_dev);
- kfree(engine);
}
kfree(idxd->engines);
}
@@ -312,7 +294,6 @@ static int idxd_setup_engines(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "engine%d.%d", idxd->id, engine->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(engine);
goto err;
}
@@ -323,10 +304,8 @@ static int idxd_setup_engines(struct idxd_device *idxd)
err:
while (--i >= 0) {
- engine = idxd->engines[i];
- conf_dev = engine_confdev(engine);
+ conf_dev = engine_confdev(idxd->engines[i]);
put_device(conf_dev);
- kfree(engine);
}
kfree(idxd->engines);
@@ -335,13 +314,10 @@ static int idxd_setup_engines(struct idxd_device *idxd)
static void idxd_clean_groups(struct idxd_device *idxd)
{
- struct idxd_group *group;
int i;
for (i = 0; i < idxd->max_groups; i++) {
- group = idxd->groups[i];
- put_device(group_confdev(group));
- kfree(group);
+ put_device(group_confdev(idxd->groups[i]));
}
kfree(idxd->groups);
}
@@ -376,7 +352,6 @@ static int idxd_setup_groups(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "group%d.%d", idxd->id, group->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(group);
goto err;
}
@@ -401,7 +376,6 @@ static int idxd_setup_groups(struct idxd_device *idxd)
while (--i >= 0) {
group = idxd->groups[i];
put_device(group_confdev(group));
- kfree(group);
}
kfree(idxd->groups);
diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
index 8b010e0e0c5c..94d2e5398dd6 100644
--- a/drivers/dma/xilinx/xilinx_dma.c
+++ b/drivers/dma/xilinx/xilinx_dma.c
@@ -167,6 +167,8 @@
/* Delay loop counter to prevent hardware failure */
#define XILINX_DMA_LOOP_COUNT 1000000
+/* Delay between polls (avoid a delay of 0 to prevent CPU stalls) */
+#define XILINX_DMA_POLL_DELAY_US 10
/* AXI DMA Specific Registers/Offsets */
#define XILINX_DMA_REG_SRCDSTADDR 0x18
@@ -1316,7 +1318,8 @@ static int xilinx_dma_stop_transfer(struct xilinx_dma_chan *chan)
/* Wait for the hardware to halt */
return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- val & XILINX_DMA_DMASR_HALTED, 0,
+ val & XILINX_DMA_DMASR_HALTED,
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
}
@@ -1331,7 +1334,8 @@ static int xilinx_cdma_stop_transfer(struct xilinx_dma_chan *chan)
u32 val;
return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- val & XILINX_DMA_DMASR_IDLE, 0,
+ val & XILINX_DMA_DMASR_IDLE,
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
}
@@ -1348,7 +1352,8 @@ static void xilinx_dma_start(struct xilinx_dma_chan *chan)
/* Wait for the hardware to start */
err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- !(val & XILINX_DMA_DMASR_HALTED), 0,
+ !(val & XILINX_DMA_DMASR_HALTED),
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
if (err) {
@@ -1783,7 +1788,8 @@ static int xilinx_dma_reset(struct xilinx_dma_chan *chan)
/* Wait for the hardware to finish reset */
err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMACR, tmp,
- !(tmp & XILINX_DMA_DMACR_RESET), 0,
+ !(tmp & XILINX_DMA_DMACR_RESET),
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
if (err) {
diff --git a/drivers/dpll/dpll_core.c b/drivers/dpll/dpll_core.c
index 6fa6a0d8e430..0a8db4dc2578 100644
--- a/drivers/dpll/dpll_core.c
+++ b/drivers/dpll/dpll_core.c
@@ -1009,12 +1009,8 @@ EXPORT_SYMBOL_GPL(dpll_pin_ref_sync_pair_add);
static struct dpll_device_registration *
dpll_device_registration_first(struct dpll_device *dpll)
{
- struct dpll_device_registration *reg;
-
- reg = list_first_entry_or_null((struct list_head *)&dpll->registration_list,
- struct dpll_device_registration, list);
- WARN_ON(!reg);
- return reg;
+ return list_first_entry_or_null((struct list_head *)&dpll->registration_list,
+ struct dpll_device_registration, list);
}
void *dpll_priv(struct dpll_device *dpll)
@@ -1022,6 +1018,8 @@ void *dpll_priv(struct dpll_device *dpll)
struct dpll_device_registration *reg;
reg = dpll_device_registration_first(dpll);
+ if (!reg)
+ return NULL;
return reg->priv;
}
@@ -1030,6 +1028,8 @@ const struct dpll_device_ops *dpll_device_ops(struct dpll_device *dpll)
struct dpll_device_registration *reg;
reg = dpll_device_registration_first(dpll);
+ if (!reg)
+ return NULL;
return reg->ops;
}
diff --git a/drivers/dpll/dpll_netlink.c b/drivers/dpll/dpll_netlink.c
index d8917150477e..d3dc16b29ca4 100644
--- a/drivers/dpll/dpll_netlink.c
+++ b/drivers/dpll/dpll_netlink.c
@@ -66,6 +66,22 @@ static bool dpll_pin_available(struct dpll_pin *pin)
return false;
}
+static bool dpll_device_registered(struct dpll_device *dpll)
+{
+ return dpll_device_ops(dpll);
+}
+
+static struct dpll_pin_ref *dpll_pin_first_registered_ref(struct dpll_pin *pin)
+{
+ struct dpll_pin_ref *ref;
+ unsigned long i;
+
+ xa_for_each(&pin->dpll_refs, i, ref)
+ if (dpll_device_registered(ref->dpll))
+ return ref;
+ return NULL;
+}
+
/**
* dpll_msg_add_pin_handle - attach pin handle attribute to a given message
* @msg: pointer to sk_buff message to attach a pin handle
@@ -575,6 +591,8 @@ dpll_msg_add_pin_dplls(struct sk_buff *msg, struct dpll_pin *pin,
int ret;
xa_for_each(&pin->dpll_refs, index, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
attr = nla_nest_start(msg, DPLL_A_PIN_PARENT_DEVICE);
if (!attr)
return -EMSGSIZE;
@@ -612,7 +630,10 @@ dpll_cmd_pin_get_one(struct sk_buff *msg, struct dpll_pin *pin,
int ret;
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
- ASSERT_NOT_NULL(ref);
+ if (!ref || !dpll_device_registered(ref->dpll))
+ ref = dpll_pin_first_registered_ref(pin);
+ if (!ref)
+ return -ENODEV;
ret = dpll_msg_add_pin_handle(msg, pin);
if (ret)
@@ -920,6 +941,8 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
}
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
if (!ops->frequency_set || !ops->frequency_get) {
NL_SET_ERR_MSG(extack, "frequency set not supported by the device");
@@ -927,6 +950,10 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
}
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->frequency_get(pin, dpll_pin_on_dpll_priv(dpll, pin), dpll,
@@ -939,6 +966,8 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
return 0;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->frequency_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -958,6 +987,8 @@ dpll_pin_freq_set(struct dpll_pin *pin, struct nlattr *a,
xa_for_each(&pin->dpll_refs, i, ref) {
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
if (ops->frequency_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -981,6 +1012,8 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
int ret;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
if (!ops->esync_set || !ops->esync_get) {
NL_SET_ERR_MSG(extack,
@@ -989,6 +1022,10 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
}
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->esync_get(pin, dpll_pin_on_dpll_priv(dpll, pin), dpll,
@@ -1011,6 +1048,8 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
xa_for_each(&pin->dpll_refs, i, ref) {
void *pin_dpll_priv;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
pin_dpll_priv = dpll_pin_on_dpll_priv(dpll, pin);
@@ -1034,6 +1073,8 @@ dpll_pin_esync_set(struct dpll_pin *pin, struct nlattr *a,
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
pin_dpll_priv = dpll_pin_on_dpll_priv(dpll, pin);
@@ -1069,7 +1110,10 @@ dpll_pin_ref_sync_state_set(struct dpll_pin *pin,
return -EINVAL;
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
- ASSERT_NOT_NULL(ref);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
if (!ops->ref_sync_set || !ops->ref_sync_get) {
NL_SET_ERR_MSG(extack, "reference sync not supported by this pin");
@@ -1087,6 +1131,8 @@ dpll_pin_ref_sync_state_set(struct dpll_pin *pin,
if (state == old_state)
return 0;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->ref_sync_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1109,6 +1155,8 @@ dpll_pin_ref_sync_state_set(struct dpll_pin *pin,
xa_for_each(&pin->dpll_refs, i, ref) {
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
if (ops->ref_sync_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1294,6 +1342,8 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
}
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
if (!ops->phase_adjust_set || !ops->phase_adjust_get) {
NL_SET_ERR_MSG(extack, "phase adjust not supported");
@@ -1301,6 +1351,10 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
}
}
ref = dpll_xa_ref_dpll_first(&pin->dpll_refs);
+ if (!ref || !dpll_device_registered(ref->dpll)) {
+ NL_SET_ERR_MSG(extack, "pin owner dpll not found");
+ return -ENODEV;
+ }
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->phase_adjust_get(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1314,6 +1368,8 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
return 0;
xa_for_each(&pin->dpll_refs, i, ref) {
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
ret = ops->phase_adjust_set(pin,
@@ -1336,6 +1392,8 @@ dpll_pin_phase_adj_set(struct dpll_pin *pin, struct nlattr *phase_adj_attr,
xa_for_each(&pin->dpll_refs, i, ref) {
if (ref == failed)
break;
+ if (!dpll_device_registered(ref->dpll))
+ continue;
ops = dpll_pin_ops(ref);
dpll = ref->dpll;
if (ops->phase_adjust_set(pin, dpll_pin_on_dpll_priv(dpll, pin),
@@ -1365,7 +1423,7 @@ dpll_pin_parent_device_set(struct dpll_pin *pin, struct nlattr *parent_nest,
return -EINVAL;
}
pdpll_idx = nla_get_u32(tb[DPLL_A_PIN_PARENT_ID]);
- dpll = xa_load(&dpll_device_xa, pdpll_idx);
+ dpll = dpll_device_get_by_id(pdpll_idx);
if (!dpll) {
NL_SET_ERR_MSG(extack, "parent device not found");
return -EINVAL;
@@ -1657,6 +1715,10 @@ int dpll_nl_pin_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb)
ret = dpll_cmd_pin_get_one(skb, pin, cb->extack);
if (ret) {
genlmsg_cancel(skb, hdr);
+ if (ret == -ENODEV) {
+ ret = 0;
+ continue;
+ }
break;
}
genlmsg_end(skb, hdr);
diff --git a/drivers/edac/amd64_edac.c b/drivers/edac/amd64_edac.c
index 8bec21dbc757..35717758fc72 100644
--- a/drivers/edac/amd64_edac.c
+++ b/drivers/edac/amd64_edac.c
@@ -2796,13 +2796,13 @@ static inline void decode_bus_error(int node_id, struct mce *m)
* the instance_id. For example, instance_id=0xYXXXXX where Y is the channel
* number.
*
- * For DRAM ECC errors, the Chip Select number is given in bits [2:0] of
+ * For DRAM ECC errors, the Chip Select number is given in bits [1:0] of
* the MCA_SYND[ErrorInformation] field.
*/
static void umc_get_err_info(struct mce *m, struct err_info *err)
{
err->channel = (m->ipid & GENMASK(31, 0)) >> 20;
- err->csrow = m->synd & 0x7;
+ err->csrow = m->synd & 0x3;
}
static void decode_umc_error(int node_id, struct mce *m)
diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c
index 290047b46d51..1115e8398668 100644
--- a/drivers/firmware/arm_scmi/bus.c
+++ b/drivers/firmware/arm_scmi/bus.c
@@ -7,7 +7,6 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/atomic.h>
#include <linux/types.h>
#include <linux/module.h>
#include <linux/of.h>
@@ -33,8 +32,8 @@ struct scmi_requested_dev {
struct list_head node;
};
-/* Track globally the creation of SCMI SystemPower related devices */
-static atomic_t scmi_syspower_registered = ATOMIC_INIT(0);
+/* Track globally the SCMI SystemPower protocol device. */
+static struct scmi_device *scmi_syspower_registered;
/**
* scmi_protocol_device_request - Helper to request a device
@@ -259,8 +258,9 @@ static int scmi_match_by_id_table(struct device *dev, const void *data)
return __scmi_dev_match_by_id_table(scmi_dev, id_table, false);
}
-static struct scmi_device *scmi_child_dev_find(struct device *parent,
- int prot_id, const char *name)
+/* Returns a device_find_child() reference which must be dropped by caller. */
+static struct scmi_device *
+scmi_child_dev_find_get(struct device *parent, int prot_id, const char *name)
{
struct scmi_device_id id_table[2] = { 0 };
struct device *dev;
@@ -272,9 +272,6 @@ static struct scmi_device *scmi_child_dev_find(struct device *parent,
if (!dev)
return NULL;
- /* Drop the refcnt bumped implicitly by device_find_child */
- put_device(dev);
-
return to_scmi_dev(dev);
}
@@ -416,10 +413,23 @@ void scmi_driver_unregister(struct scmi_driver *driver)
}
EXPORT_SYMBOL_GPL(scmi_driver_unregister);
+static void scmi_device_release_resources(struct scmi_device *scmi_dev)
+{
+ if (scmi_dev->protocol_id == SCMI_PROTOCOL_SYSTEM)
+ cmpxchg(&scmi_syspower_registered, scmi_dev, NULL);
+
+ if (scmi_dev->id) {
+ ida_free(&scmi_bus_id, scmi_dev->id);
+ scmi_dev->id = 0;
+ }
+}
+
static void scmi_device_release(struct device *dev)
{
struct scmi_device *scmi_dev = to_scmi_dev(dev);
+ scmi_device_release_resources(scmi_dev);
+ of_node_put(dev->of_node);
kfree_const(scmi_dev->name);
kfree(scmi_dev);
}
@@ -431,11 +441,9 @@ static void __scmi_device_destroy(struct scmi_device *scmi_dev)
dev_name(&scmi_dev->dev), scmi_dev->protocol_id,
scmi_dev->name);
- if (scmi_dev->protocol_id == SCMI_PROTOCOL_SYSTEM)
- atomic_set(&scmi_syspower_registered, 0);
-
- ida_free(&scmi_bus_id, scmi_dev->id);
- device_unregister(&scmi_dev->dev);
+ device_del(&scmi_dev->dev);
+ scmi_device_release_resources(scmi_dev);
+ put_device(&scmi_dev->dev);
}
static struct scmi_device *
@@ -444,6 +452,7 @@ __scmi_device_create(struct device_node *np, struct device *parent,
{
int id, retval;
struct scmi_device *scmi_dev;
+ bool syspower = (protocol == SCMI_PROTOCOL_SYSTEM);
/*
* If the same protocol/name device already exist under the same parent
@@ -452,45 +461,41 @@ __scmi_device_create(struct device_node *np, struct device *parent,
* each DT defined protocol at probe time, and the concurrent
* registration of SCMI drivers.
*/
- scmi_dev = scmi_child_dev_find(parent, protocol, name);
- if (scmi_dev)
+ scmi_dev = scmi_child_dev_find_get(parent, protocol, name);
+ if (scmi_dev) {
+ put_device(&scmi_dev->dev);
return scmi_dev;
+ }
+
+ scmi_dev = kzalloc_obj(*scmi_dev);
+ if (!scmi_dev)
+ return NULL;
+
+ scmi_dev->protocol_id = protocol;
/*
- * Ignore any possible subsequent failures while creating the device
- * since we are doomed anyway at that point; not using a mutex which
- * spans across this whole function to keep things simple and to avoid
- * to serialize all the __scmi_device_create calls across possibly
- * different SCMI server instances (parent)
+ * Reserve the singleton SystemPower protocol device using the device
+ * pointer itself, so delayed release of an older device cannot clear
+ * a reservation owned by a newer device.
*/
- if (protocol == SCMI_PROTOCOL_SYSTEM &&
- atomic_cmpxchg(&scmi_syspower_registered, 0, 1)) {
+ if (syspower && cmpxchg(&scmi_syspower_registered, NULL, scmi_dev)) {
dev_warn(parent,
"SCMI SystemPower protocol device must be unique !\n");
+ kfree(scmi_dev);
return NULL;
}
- scmi_dev = kzalloc(sizeof(*scmi_dev), GFP_KERNEL);
- if (!scmi_dev)
- return NULL;
-
scmi_dev->name = kstrdup_const(name ?: "unknown", GFP_KERNEL);
- if (!scmi_dev->name) {
- kfree(scmi_dev);
- return NULL;
- }
+ if (!scmi_dev->name)
+ goto free_dev;
id = ida_alloc_min(&scmi_bus_id, 1, GFP_KERNEL);
- if (id < 0) {
- kfree_const(scmi_dev->name);
- kfree(scmi_dev);
- return NULL;
- }
+ if (id < 0)
+ goto free_name;
scmi_dev->id = id;
- scmi_dev->protocol_id = protocol;
scmi_dev->dev.parent = parent;
- device_set_node(&scmi_dev->dev, of_fwnode_handle(np));
+ device_set_node(&scmi_dev->dev, of_fwnode_handle(of_node_get(np)));
scmi_dev->dev.bus = &scmi_bus_type;
scmi_dev->dev.release = scmi_device_release;
dev_set_name(&scmi_dev->dev, "scmi_dev.%d", id);
@@ -504,8 +509,14 @@ __scmi_device_create(struct device_node *np, struct device *parent,
return scmi_dev;
put_dev:
+ scmi_device_release_resources(scmi_dev);
put_device(&scmi_dev->dev);
- ida_free(&scmi_bus_id, id);
+ return NULL;
+free_name:
+ kfree_const(scmi_dev->name);
+free_dev:
+ scmi_device_release_resources(scmi_dev);
+ kfree(scmi_dev);
return NULL;
}
@@ -586,9 +597,11 @@ void scmi_device_destroy(struct device *parent, int protocol, const char *name)
{
struct scmi_device *scmi_dev;
- scmi_dev = scmi_child_dev_find(parent, protocol, name);
- if (scmi_dev)
+ scmi_dev = scmi_child_dev_find_get(parent, protocol, name);
+ if (scmi_dev) {
__scmi_device_destroy(scmi_dev);
+ put_device(&scmi_dev->dev);
+ }
}
EXPORT_SYMBOL_GPL(scmi_device_destroy);
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 5a9d6df50679..9e81d7c9a449 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -2703,14 +2703,12 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node,
devm_kfree(info->dev, cinfo);
return -EINVAL;
}
- of_node_get(of_node);
cinfo->id = prot_id;
cinfo->dev = &tdev->dev;
cinfo->handle = &info->handle;
ret = info->desc->ops->chan_setup(cinfo, info->dev, tx);
if (ret) {
- of_node_put(of_node);
scmi_device_destroy(info->dev, prot_id, name);
devm_kfree(info->dev, cinfo);
return ret;
@@ -2734,7 +2732,6 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node,
/* Destroy channel and device only if created by this call. */
if (tdev) {
info->desc->ops->chan_free(prot_id, cinfo, idr);
- of_node_put(of_node);
scmi_device_destroy(info->dev, prot_id, name);
devm_kfree(info->dev, cinfo);
}
@@ -2820,7 +2817,6 @@ static int scmi_chan_destroy(int id, void *p, void *data)
struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
struct scmi_device *sdev = to_scmi_dev(cinfo->dev);
- of_node_put(cinfo->dev->of_node);
scmi_device_destroy(info->dev, cinfo->id, sdev->name);
cinfo->dev = NULL;
}
diff --git a/drivers/firmware/xilinx/zynqmp.c b/drivers/firmware/xilinx/zynqmp.c
index dc9c5d32108d..cae5ddc145a2 100644
--- a/drivers/firmware/xilinx/zynqmp.c
+++ b/drivers/firmware/xilinx/zynqmp.c
@@ -72,6 +72,15 @@ struct pm_api_feature_data {
struct hlist_node hentry;
};
+struct platform_fw_data {
+ /*
+ * Family code for platform.
+ */
+ const u32 family_code;
+};
+
+static struct platform_fw_data *active_platform_fw_data;
+
static const struct mfd_cell firmware_devs[] = {
{
.name = "zynqmp_power_controller",
@@ -465,8 +474,6 @@ EXPORT_SYMBOL_GPL(zynqmp_pm_invoke_fn);
static u32 pm_api_version;
static u32 pm_tz_version;
-static u32 pm_family_code;
-static u32 pm_sub_family_code;
int zynqmp_pm_register_sgi(u32 sgi_num, u32 reset)
{
@@ -533,32 +540,18 @@ EXPORT_SYMBOL_GPL(zynqmp_pm_get_chipid);
/**
* zynqmp_pm_get_family_info() - Get family info of platform
* @family: Returned family code value
- * @subfamily: Returned sub-family code value
*
* Return: Returns status, either success or error+reason
*/
-int zynqmp_pm_get_family_info(u32 *family, u32 *subfamily)
+int zynqmp_pm_get_family_info(u32 *family)
{
- u32 ret_payload[PAYLOAD_ARG_CNT];
- u32 idcode;
- int ret;
-
- /* Check is family or sub-family code already received */
- if (pm_family_code && pm_sub_family_code) {
- *family = pm_family_code;
- *subfamily = pm_sub_family_code;
- return 0;
- }
+ if (!active_platform_fw_data)
+ return -ENODEV;
- ret = zynqmp_pm_invoke_fn(PM_GET_CHIPID, ret_payload, 0);
- if (ret < 0)
- return ret;
+ if (!family)
+ return -EINVAL;
- idcode = ret_payload[1];
- pm_family_code = FIELD_GET(FAMILY_CODE_MASK, idcode);
- pm_sub_family_code = FIELD_GET(SUB_FAMILY_CODE_MASK, idcode);
- *family = pm_family_code;
- *subfamily = pm_sub_family_code;
+ *family = active_platform_fw_data->family_code;
return 0;
}
@@ -1239,8 +1232,13 @@ int zynqmp_pm_pinctrl_set_config(const u32 pin, const u32 param,
u32 value)
{
int ret;
+ u32 pm_family_code;
+
+ ret = zynqmp_pm_get_family_info(&pm_family_code);
+ if (ret)
+ return ret;
- if (pm_family_code == ZYNQMP_FAMILY_CODE &&
+ if (pm_family_code == PM_ZYNQMP_FAMILY_CODE &&
param == PM_PINCTRL_CONFIG_TRI_STATE) {
ret = zynqmp_pm_feature(PM_PINCTRL_CONFIG_PARAM_SET);
if (ret < PM_PINCTRL_PARAM_SET_VERSION) {
@@ -2008,12 +2006,18 @@ static int zynqmp_firmware_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct zynqmp_devinfo *devinfo;
+ u32 pm_family_code;
int ret;
ret = get_set_conduit_method(dev->of_node);
if (ret)
return ret;
+ /* Get platform-specific firmware data from device tree match */
+ active_platform_fw_data = (struct platform_fw_data *)device_get_match_data(dev);
+ if (!active_platform_fw_data)
+ return -EINVAL;
+
/* Get SiP SVC version number */
ret = zynqmp_pm_get_sip_svc_version(&sip_svc_version);
if (ret)
@@ -2046,8 +2050,8 @@ static int zynqmp_firmware_probe(struct platform_device *pdev)
pr_info("%s Platform Management API v%d.%d\n", __func__,
pm_api_version >> 16, pm_api_version & 0xFFFF);
- /* Get the Family code and sub family code of platform */
- ret = zynqmp_pm_get_family_info(&pm_family_code, &pm_sub_family_code);
+ /* Get the Family code of platform */
+ ret = zynqmp_pm_get_family_info(&pm_family_code);
if (ret < 0)
return ret;
@@ -2074,7 +2078,7 @@ static int zynqmp_firmware_probe(struct platform_device *pdev)
zynqmp_pm_api_debugfs_init();
- if (pm_family_code == VERSAL_FAMILY_CODE) {
+ if (pm_family_code != PM_ZYNQMP_FAMILY_CODE) {
em_dev = platform_device_register_data(&pdev->dev, "xlnx_event_manager",
-1, NULL, 0);
if (IS_ERR(em_dev))
@@ -2114,9 +2118,17 @@ static void zynqmp_firmware_sync_state(struct device *dev)
dev_warn(dev, "failed to release power management to firmware\n");
}
+static const struct platform_fw_data platform_fw_data_versal = {
+ .family_code = PM_VERSAL_FAMILY_CODE,
+};
+
+static const struct platform_fw_data platform_fw_data_zynqmp = {
+ .family_code = PM_ZYNQMP_FAMILY_CODE,
+};
+
static const struct of_device_id zynqmp_firmware_of_match[] = {
- {.compatible = "xlnx,zynqmp-firmware"},
- {.compatible = "xlnx,versal-firmware"},
+ {.compatible = "xlnx,zynqmp-firmware", .data = &platform_fw_data_zynqmp},
+ {.compatible = "xlnx,versal-firmware", .data = &platform_fw_data_versal},
{},
};
MODULE_DEVICE_TABLE(of, zynqmp_firmware_of_match);
diff --git a/drivers/gpio/gpio-exar.c b/drivers/gpio/gpio-exar.c
index 9053662f1817..f629cc526f3e 100644
--- a/drivers/gpio/gpio-exar.c
+++ b/drivers/gpio/gpio-exar.c
@@ -199,6 +199,7 @@ static int gpio_exar_probe(struct platform_device *pdev)
index = ida_alloc(&ida_index, GFP_KERNEL);
if (index < 0)
return index;
+ exar_gpio->index = index;
ret = devm_add_action_or_reset(dev, exar_devm_ida_free, exar_gpio);
if (ret)
@@ -214,7 +215,6 @@ static int gpio_exar_probe(struct platform_device *pdev)
exar_gpio->gpio_chip.set = exar_set_value;
exar_gpio->gpio_chip.base = -1;
exar_gpio->gpio_chip.ngpio = ngpios;
- exar_gpio->index = index;
exar_gpio->first_pin = first_pin;
ret = devm_gpiochip_add_data(dev, &exar_gpio->gpio_chip, exar_gpio);
diff --git a/drivers/gpio/gpio-mpsse.c b/drivers/gpio/gpio-mpsse.c
index 120b27183b1d..dac478ef33e1 100644
--- a/drivers/gpio/gpio-mpsse.c
+++ b/drivers/gpio/gpio-mpsse.c
@@ -10,6 +10,7 @@
#include <linux/cleanup.h>
#include <linux/gpio/driver.h>
#include <linux/mutex.h>
+#include <linux/rcupdate.h>
#include <linux/spinlock.h>
#include <linux/usb.h>
@@ -24,6 +25,7 @@ struct mpsse_priv {
raw_spinlock_t irq_spin; /* protects worker list */
atomic_t irq_type[16]; /* pin -> edge detection type */
atomic_t irq_enabled;
+ atomic_t dying; /* no new workers after disconnect */
int id;
u8 gpio_outputs[2]; /* Output states for GPIOs [L, H] */
@@ -46,6 +48,7 @@ struct mpsse_worker {
atomic_t cancelled;
struct list_head list; /* linked list */
struct list_head destroy; /* teardown linked list */
+ struct rcu_head rcu; /* deferred free for dying path */
};
struct bulk_desc {
@@ -515,10 +518,24 @@ static void gpio_mpsse_irq_enable(struct irq_data *irqd)
worker->priv = priv;
INIT_LIST_HEAD(&worker->list);
INIT_WORK(&worker->work, gpio_mpsse_poll);
- schedule_work(&worker->work);
- scoped_guard(raw_spinlock_irqsave, &priv->irq_spin)
+ scoped_guard(raw_spinlock_irqsave, &priv->irq_spin) {
+ if (atomic_read(&priv->dying)) {
+ /*
+ * Cannot kfree() here: this callback
+ * runs with the IRQ descriptor's raw
+ * spinlock held and kfree() may sleep
+ * on PREEMPT_RT. The worker is not
+ * yet published, so the deferred free
+ * is unobservable.
+ */
+ kfree_rcu(worker, rcu);
+ return;
+ }
+
list_add(&worker->list, &priv->workers);
+ schedule_work(&worker->work);
+ }
}
}
@@ -705,6 +722,9 @@ static void gpio_mpsse_disconnect(struct usb_interface *intf)
{
struct mpsse_priv *priv = usb_get_intfdata(intf);
+ scoped_guard(raw_spinlock_irqsave, &priv->irq_spin)
+ atomic_set(&priv->dying, 1);
+
/*
* Lock prevents double-free of worker from here and the teardown
* step at the beginning of gpio_mpsse_poll
diff --git a/drivers/gpio/gpio-pca953x.c b/drivers/gpio/gpio-pca953x.c
index b18a2d0f6984..4eeac12ad0ed 100644
--- a/drivers/gpio/gpio-pca953x.c
+++ b/drivers/gpio/gpio-pca953x.c
@@ -605,20 +605,28 @@ static int pca953x_read_regs(struct pca953x_chip *chip, int reg, unsigned long *
return 0;
}
-static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned off)
+static int pca953x_gpio_direction_input_unlocked(struct gpio_chip *gc,
+ unsigned int off)
{
struct pca953x_chip *chip = gpiochip_get_data(gc);
u8 dirreg = chip->recalc_addr(chip, chip->regs->direction, off);
u8 bit = pca953x_get_bit_mask(chip, off);
- guard(mutex)(&chip->i2c_lock);
-
if (PCA_CHIP_TYPE(chip->driver_data) == TCA6418_TYPE)
return regmap_update_bits(chip->regmap, dirreg, bit, 0);
return regmap_update_bits(chip->regmap, dirreg, bit, bit);
}
+static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned int off)
+{
+ struct pca953x_chip *chip = gpiochip_get_data(gc);
+
+ guard(mutex)(&chip->i2c_lock);
+
+ return pca953x_gpio_direction_input_unlocked(gc, off);
+}
+
static int pca953x_gpio_direction_output(struct gpio_chip *gc,
unsigned off, int val)
{
@@ -856,9 +864,9 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d)
DECLARE_BITMAP(reg_direction, MAX_LINE);
int level;
- if (chip->driver_data & PCA_PCAL) {
- guard(mutex)(&chip->i2c_lock);
+ guard(mutex)(&chip->i2c_lock);
+ if (chip->driver_data & PCA_PCAL) {
/* Enable latch on interrupt-enabled inputs */
pca953x_write_regs(chip, PCAL953X_IN_LATCH, chip->irq_mask);
@@ -879,7 +887,7 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d)
/* Look for any newly setup interrupt */
for_each_set_bit(level, irq_mask, gc->ngpio)
- pca953x_gpio_direction_input(&chip->gpio_chip, level);
+ pca953x_gpio_direction_input_unlocked(&chip->gpio_chip, level);
mutex_unlock(&chip->irq_lock);
}
diff --git a/drivers/gpio/gpio-xilinx.c b/drivers/gpio/gpio-xilinx.c
index 83675ac81077..db9eb3344eea 100644
--- a/drivers/gpio/gpio-xilinx.c
+++ b/drivers/gpio/gpio-xilinx.c
@@ -271,14 +271,7 @@ static void xgpio_save_regs(struct xgpio_instance *chip)
static int xgpio_request(struct gpio_chip *chip, unsigned int offset)
{
- int ret;
-
- ret = pm_runtime_get_sync(chip->parent);
- /*
- * If the device is already active pm_runtime_get() will return 1 on
- * success, but gpio_request still needs to return 0.
- */
- return ret < 0 ? ret : 0;
+ return pm_runtime_resume_and_get(chip->parent);
}
static void xgpio_free(struct gpio_chip *chip, unsigned int offset)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index 5e8d0d6b55ab..23171bcc947f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -567,14 +567,18 @@ int amdgpu_amdkfd_get_dmabuf_info(struct amdgpu_device *adev, int dma_buf_fd,
/* first get metadata_size by buffer = NULL */
r = amdgpu_bo_get_metadata(bo, NULL, 0,
metadata_size, NULL);
+ if (r)
+ goto out_put;
/* user buf_size is bigger than bo metadata_size
* allocate a buf at kernel space and copy */
if (*metadata_size <= buffer_size) {
*metadata_buffer = kzalloc(*metadata_size, GFP_KERNEL);
- if (!*metadata_buffer)
- return -ENOMEM;
+ if (!*metadata_buffer) {
+ r = -ENOMEM;
+ goto out_put;
+ }
r = amdgpu_bo_get_metadata(bo, *metadata_buffer, *metadata_size,
NULL, &metadata_flags);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 3b3f404295b4..cda4696d2dd4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -114,6 +114,20 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f,
*pos &= (1UL << 22) - 1;
+ if (size > PAGE_SIZE)
+ return -EINVAL;
+
+ /*
+ * Access the user buffer only while none of the locks below are held:
+ * a fault takes mmap_lock, and grbm_idx_mutex and srbm_mutex nest
+ * inside it.
+ */
+ u32 *data __free(kfree) = kmalloc(size, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+ if (!read && copy_from_user(data, buf, size))
+ return -EFAULT;
+
r = pm_runtime_get_sync(adev_to_drm(adev)->dev);
if (r < 0) {
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
@@ -146,28 +160,16 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f,
mutex_lock(&adev->pm.mutex);
while (size) {
- uint32_t value;
-
- if (read) {
- value = RREG32(*pos >> 2);
- r = put_user(value, (uint32_t *)buf);
- } else {
- r = get_user(value, (uint32_t *)buf);
- if (!r)
- amdgpu_mm_wreg_mmio_rlc(adev, *pos >> 2, value, 0);
- }
- if (r) {
- result = r;
- goto end;
- }
+ if (read)
+ data[result >> 2] = RREG32(*pos >> 2);
+ else
+ amdgpu_mm_wreg_mmio_rlc(adev, *pos >> 2, data[result >> 2], 0);
result += 4;
- buf += 4;
*pos += 4;
size -= 4;
}
-end:
if (use_bank) {
amdgpu_gfx_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff, 0);
mutex_unlock(&adev->grbm_idx_mutex);
@@ -183,6 +185,10 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f,
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
amdgpu_virt_disable_access_debugfs(adev);
+
+ if (read && copy_to_user(buf, data, result))
+ return -EFAULT;
+
return result;
}
@@ -233,11 +239,24 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off
struct amdgpu_device *adev = rd->adev;
ssize_t result = 0;
int r;
- uint32_t value;
if (size & 0x3 || offset & 0x3)
return -EINVAL;
+ if (size > PAGE_SIZE)
+ return -EINVAL;
+
+ /*
+ * Access the user buffer only while none of the locks below are held:
+ * a fault takes mmap_lock, and grbm_idx_mutex and srbm_mutex nest
+ * inside it.
+ */
+ u32 *data __free(kfree) = kmalloc(size, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+ if (write_en && copy_from_user(data, buf, size))
+ return -EFAULT;
+
r = pm_runtime_get_sync(adev_to_drm(adev)->dev);
if (r < 0) {
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
@@ -277,24 +296,16 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off
mutex_lock(&adev->pm.mutex);
while (size) {
- if (!write_en) {
- value = RREG32(offset >> 2);
- r = put_user(value, (uint32_t *)buf);
- } else {
- r = get_user(value, (uint32_t *)buf);
- if (!r)
- amdgpu_mm_wreg_mmio_rlc(adev, offset >> 2, value, rd->id.xcc_id);
- }
- if (r) {
- result = r;
- goto end;
- }
+ if (!write_en)
+ data[result >> 2] = RREG32(offset >> 2);
+ else
+ amdgpu_mm_wreg_mmio_rlc(adev, offset >> 2, data[result >> 2],
+ rd->id.xcc_id);
offset += 4;
size -= 4;
result += 4;
- buf += 4;
}
-end:
+
if (rd->id.use_grbm) {
amdgpu_gfx_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff, rd->id.xcc_id);
mutex_unlock(&adev->grbm_idx_mutex);
@@ -314,6 +325,10 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
amdgpu_virt_disable_access_debugfs(adev);
+
+ if (!write_en && copy_to_user(buf, data, result))
+ return -EFAULT;
+
return result;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index b47989692bba..88dda7dd2cd1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -5920,6 +5920,31 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
amdgpu_fence_driver_force_completion(ring, fence);
}
+ /*
+ * MES scheduler rings have no drm scheduler, so they are missed by the
+ * loop above. Realign their polling fence too, otherwise the
+ * first post-reset submission polls forever on a stale seq. sched.ready is
+ * only set while the driver owns the ring.
+ */
+ for (i = 0; i < AMDGPU_MAX_MES_PIPES; i++) {
+ struct amdgpu_ring *mes_ring = &adev->mes.ring[i];
+
+ if (mes_ring->fence_drv.initialized && mes_ring->sched.ready)
+ amdgpu_fence_driver_force_completion(mes_ring, fence);
+ }
+
+ /*
+ * KIQ rings are polling-fence/no_scheduler like MES, so realign their
+ * fence too (one ring per XCC), otherwise the first post-reset KIQ
+ * submission polls forever on a stale seq.
+ */
+ for (i = 0; i < AMDGPU_MAX_GC_INSTANCES; i++) {
+ struct amdgpu_ring *kiq_ring = &adev->gfx.kiq[i].ring;
+
+ if (kiq_ring->fence_drv.initialized && kiq_ring->sched.ready)
+ amdgpu_fence_driver_force_completion(kiq_ring, fence);
+ }
+
amdgpu_fence_driver_isr_toggle(adev, false);
if (job && job->vm)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
index b97eb1fef73d..70f4014012ff 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
@@ -1022,20 +1022,17 @@ void amdgpu_gmc_get_vbios_allocations(struct amdgpu_device *adev)
case CHIP_RENOIR:
adev->mman.keep_stolen_vga_memory = true;
break;
- case CHIP_POLARIS10:
- case CHIP_POLARIS11:
- case CHIP_POLARIS12:
- /* MacBookPros with switchable graphics put VRAM at 0 when
- * the iGPU is enabled which results in cursor issues if
- * the cursor ends up at 0. Reserve vram at 0 in that case.
- */
- if (adev->gmc.vram_start == 0)
- adev->mman.keep_stolen_vga_memory = true;
- break;
default:
adev->mman.keep_stolen_vga_memory = false;
break;
}
+ /* MacBookPros with switchable graphics put VRAM at 0 when
+ * the iGPU is enabled which results in cursor issues if
+ * the cursor ends up at 0. Reserve vram at 0 in that case.
+ */
+ if ((adev->pdev->subsystem_vendor == PCI_VENDOR_ID_APPLE) &&
+ (adev->gmc.vram_start == 0))
+ adev->mman.keep_stolen_vga_memory = true;
if (amdgpu_sriov_vf(adev) ||
!amdgpu_device_has_display_hardware(adev)) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
index f60f26470af0..db2bb5cb671d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c
@@ -51,8 +51,6 @@
#include "amdgpu_amdkfd.h"
#include "amdgpu_hmm.h"
-#define MAX_WALK_BYTE (2UL << 30)
-
/**
* amdgpu_hmm_invalidate_gfx - callback to notify about mm change
*
@@ -172,11 +170,13 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
void *owner,
struct hmm_range **phmm_range)
{
+ const u64 max_bytes = SZ_2G;
+
struct hmm_range *hmm_range;
- unsigned long end;
unsigned long timeout;
unsigned long *pfns;
- int r = 0;
+ unsigned long end;
+ int r;
hmm_range = kzalloc(sizeof(*hmm_range), GFP_KERNEL);
if (unlikely(!hmm_range))
@@ -197,8 +197,9 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
end = start + npages * PAGE_SIZE;
hmm_range->dev_private_owner = owner;
+ hmm_range->notifier_seq = mmu_interval_read_begin(notifier);
do {
- hmm_range->end = min(hmm_range->start + MAX_WALK_BYTE, end);
+ hmm_range->end = min(hmm_range->start + max_bytes, end);
pr_debug("hmm range: start = 0x%lx, end = 0x%lx",
hmm_range->start, hmm_range->end);
@@ -206,7 +207,6 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
retry:
- hmm_range->notifier_seq = mmu_interval_read_begin(notifier);
r = hmm_range_fault(hmm_range);
if (unlikely(r)) {
if (r == -EBUSY && !time_after(jiffies, timeout))
@@ -216,7 +216,7 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier,
if (hmm_range->end == end)
break;
- hmm_range->hmm_pfns += MAX_WALK_BYTE >> PAGE_SHIFT;
+ hmm_range->hmm_pfns += max_bytes >> PAGE_SHIFT;
hmm_range->start = hmm_range->end;
} while (hmm_range->end < end);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
index 915406ab9730..bc3e6ddc057c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c
@@ -30,6 +30,7 @@
#include <drm/amdgpu_drm.h>
#include <drm/drm_drv.h>
#include <drm/drm_fb_helper.h>
+#include <drm/drm_vblank.h>
#include "amdgpu_uvd.h"
#include "amdgpu_vce.h"
#include "atom.h"
@@ -1572,6 +1573,9 @@ u32 amdgpu_get_vblank_counter_kms(struct drm_crtc *crtc)
DRM_ERROR("Invalid crtc %u\n", pipe);
return -EINVAL;
}
+ if (!down_read_trylock(&adev->reset_domain->sem))
+ return drm_crtc_vblank_count(crtc);
+
/* The hw increments its frame counter at start of vsync, not at start
* of vblank, as is required by DRM core vblank counter handling.
@@ -1617,6 +1621,7 @@ u32 amdgpu_get_vblank_counter_kms(struct drm_crtc *crtc)
DRM_DEBUG_VBL("NULL mode info! Returned count may be wrong.\n");
}
+ up_read(&adev->reset_domain->sem);
return count;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index c48c266cc7a7..08c31d78b3fa 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -175,6 +175,8 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,
* @tmz: if we should setup a TMZ enabled mapping
* @size: in number of bytes to map, out number of bytes mapped
* @addr: resulting address inside the MC address space
+ * @vm_needs_flush: out, set true if a GART window was programmed (VMID 0 flush
+ * needed) or false for a direct address
*
* Setup one of the GART windows to access a specific piece of memory or return
* the physical address for local memory.
@@ -183,7 +185,8 @@ static int amdgpu_ttm_map_buffer(struct ttm_buffer_object *bo,
struct ttm_resource *mem,
struct amdgpu_res_cursor *mm_cur,
unsigned int window, struct amdgpu_ring *ring,
- bool tmz, uint64_t *size, uint64_t *addr)
+ bool tmz, uint64_t *size, uint64_t *addr,
+ bool *vm_needs_flush)
{
struct amdgpu_device *adev = ring->adev;
unsigned int offset, num_pages, num_dw, num_bytes;
@@ -204,9 +207,12 @@ static int amdgpu_ttm_map_buffer(struct ttm_buffer_object *bo,
if (!tmz && mem->start != AMDGPU_BO_INVALID_OFFSET) {
*addr = amdgpu_ttm_domain_start(adev, mem->mem_type) +
mm_cur->start;
+ *vm_needs_flush = false;
return 0;
}
+ /* A GART window is programmed below, so its VMID 0 TLB needs a flush */
+ *vm_needs_flush = true;
/*
* If start begins at an offset inside the page, then adjust the size
@@ -316,6 +322,7 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
while (src_mm.remaining) {
uint64_t from, to, cur_size, tiling_flags;
uint32_t num_type, data_format, max_com, write_compress_disable;
+ bool src_vm_flush, dst_vm_flush;
struct dma_fence *next;
/* Never copy more than 256MiB at once to avoid a timeout */
@@ -323,12 +330,12 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
/* Map src to window 0 and dst to window 1. */
r = amdgpu_ttm_map_buffer(src->bo, src->mem, &src_mm,
- 0, ring, tmz, &cur_size, &from);
+ 0, ring, tmz, &cur_size, &from, &src_vm_flush);
if (r)
goto error;
r = amdgpu_ttm_map_buffer(dst->bo, dst->mem, &dst_mm,
- 1, ring, tmz, &cur_size, &to);
+ 1, ring, tmz, &cur_size, &to, &dst_vm_flush);
if (r)
goto error;
@@ -356,7 +363,7 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
}
r = amdgpu_copy_buffer(ring, from, to, cur_size, resv,
- &next, false, true, copy_flags);
+ &next, false, src_vm_flush || dst_vm_flush, copy_flags);
if (r)
goto error;
@@ -2419,6 +2426,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
struct amdgpu_res_cursor cursor;
+ bool vm_needs_flush = false;
u64 addr;
int r = 0;
@@ -2446,12 +2454,12 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_bo *bo,
size = min(cursor.size, 256ULL << 20);
r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &cursor,
- 1, ring, false, &size, &addr);
+ 1, ring, false, &size, &addr, &vm_needs_flush);
if (r)
goto err;
r = amdgpu_ttm_fill_mem(ring, 0, addr, size, resv,
- &next, true, true,
+ &next, vm_needs_flush, true,
AMDGPU_KERNEL_JOB_ID_TTM_CLEAR_BUFFER);
if (r)
goto err;
@@ -2478,6 +2486,7 @@ int amdgpu_fill_buffer(struct amdgpu_bo *bo,
struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
struct dma_fence *fence = NULL;
struct amdgpu_res_cursor dst;
+ bool vm_needs_flush = false;
int r;
if (!adev->mman.buffer_funcs_enabled) {
@@ -2497,12 +2506,12 @@ int amdgpu_fill_buffer(struct amdgpu_bo *bo,
cur_size = min(dst.size, 256ULL << 20);
r = amdgpu_ttm_map_buffer(&bo->tbo, bo->tbo.resource, &dst,
- 1, ring, false, &cur_size, &to);
+ 1, ring, false, &cur_size, &to, &vm_needs_flush);
if (r)
goto error;
r = amdgpu_ttm_fill_mem(ring, src_data, to, cur_size, resv,
- &next, true, delayed, k_job_id);
+ &next, vm_needs_flush, delayed, k_job_id);
if (r)
goto error;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
index 0f4281c9aea2..a0bf7c82b83b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c
@@ -317,6 +317,7 @@ amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr,
uint64_t index;
struct drm_gem_object *gobj;
struct amdgpu_userq_obj *db_obj = db_info->db_obj;
+ struct amdgpu_bo *abo;
int r, db_size;
gobj = drm_gem_object_lookup(filp, db_info->doorbell_handle);
@@ -325,7 +326,17 @@ amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr,
return -EINVAL;
}
- db_obj->obj = amdgpu_bo_ref(gem_to_amdgpu_bo(gobj));
+ /*
+ * Pinning a regular BO into the doorbell domain would discard its
+ * contents, possibly those of a buffer shared by another client.
+ */
+ abo = gem_to_amdgpu_bo(gobj);
+ if (!(abo->preferred_domains & AMDGPU_GEM_DOMAIN_DOORBELL)) {
+ drm_gem_object_put(gobj);
+ return -EINVAL;
+ }
+
+ db_obj->obj = amdgpu_bo_ref(abo);
drm_gem_object_put(gobj);
r = amdgpu_bo_reserve(db_obj->obj, true);
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
index 5358ee33914e..433381f806b2 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c
@@ -3473,10 +3473,19 @@ static int gfx_v12_0_cp_resume(struct amdgpu_device *adev)
gfx_v12_0_cp_gfx_enable(adev, true);
}
- if (adev->enable_mes_kiq && adev->mes.kiq_hw_init)
+ if (adev->enable_mes_kiq && adev->mes.kiq_hw_init) {
r = amdgpu_mes_kiq_hw_init(adev);
- else
+ /*
+ * With MES, GFX KIQ ring is owned by the MES and is never
+ * initialized/used directly by the driver, so it must
+ * not be left flagged as ready. mes_v12_0_hw_init() clears
+ * but clear here if MES init fails
+ */
+ if (r)
+ adev->gfx.kiq[0].ring.sched.ready = false;
+ } else {
r = gfx_v12_0_kiq_resume(adev);
+ }
if (r)
return r;
diff --git a/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c b/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c
index f5aa83ff57f3..15bcbe9118e5 100644
--- a/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c
+++ b/drivers/gpu/drm/amd/amdgpu/mes_userqueue.c
@@ -30,71 +30,63 @@
#define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE
#define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE
-static int
-mes_userq_map_gtt_bo_to_gart(struct amdgpu_bo *bo)
-{
- int ret;
-
- ret = amdgpu_bo_reserve(bo, true);
- if (ret) {
- DRM_ERROR("Failed to reserve bo. ret %d\n", ret);
- goto err_reserve_bo_failed;
- }
-
- ret = amdgpu_ttm_alloc_gart(&bo->tbo);
- if (ret) {
- DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret);
- goto err_map_bo_gart_failed;
- }
-
- amdgpu_bo_unreserve(bo);
- bo = amdgpu_bo_ref(bo);
-
- return 0;
-
-err_map_bo_gart_failed:
- amdgpu_bo_unreserve(bo);
-err_reserve_bo_failed:
- return ret;
-}
-
static int
mes_userq_create_wptr_mapping(struct amdgpu_userq_mgr *uq_mgr,
struct amdgpu_usermode_queue *queue,
uint64_t wptr)
{
struct amdgpu_bo_va_mapping *wptr_mapping;
- struct amdgpu_vm *wptr_vm;
struct amdgpu_userq_obj *wptr_obj = &queue->wptr_obj;
+ struct amdgpu_bo *obj;
+ struct amdgpu_vm *vm = queue->vm;
+ struct drm_exec exec;
int ret;
- wptr_vm = queue->vm;
- ret = amdgpu_bo_reserve(wptr_vm->root.bo, false);
- if (ret)
- return ret;
-
wptr &= AMDGPU_GMC_HOLE_MASK;
- wptr_mapping = amdgpu_vm_bo_lookup_mapping(wptr_vm, wptr >> PAGE_SHIFT);
- amdgpu_bo_unreserve(wptr_vm->root.bo);
- if (!wptr_mapping) {
- DRM_ERROR("Failed to lookup wptr bo\n");
- return -EINVAL;
+
+ drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 2);
+ drm_exec_until_all_locked(&exec) {
+ ret = amdgpu_vm_lock_pd(vm, &exec, 1);
+ drm_exec_retry_on_contention(&exec);
+ if (unlikely(ret))
+ goto fail_lock;
+
+ wptr_mapping = amdgpu_vm_bo_lookup_mapping(vm, wptr >> PAGE_SHIFT);
+ if (!wptr_mapping) {
+ ret = -EINVAL;
+ goto fail_lock;
+ }
+
+ obj = wptr_mapping->bo_va->base.bo;
+ ret = drm_exec_lock_obj(&exec, &obj->tbo.base);
+ drm_exec_retry_on_contention(&exec);
+ if (unlikely(ret))
+ goto fail_lock;
}
- wptr_obj->obj = wptr_mapping->bo_va->base.bo;
+ wptr_obj->obj = amdgpu_bo_ref(wptr_mapping->bo_va->base.bo);
if (wptr_obj->obj->tbo.base.size > PAGE_SIZE) {
- DRM_ERROR("Requested GART mapping for wptr bo larger than one page\n");
- return -EINVAL;
+ ret = -EINVAL;
+ goto fail_map;
}
- ret = mes_userq_map_gtt_bo_to_gart(wptr_obj->obj);
+ ret = amdgpu_ttm_alloc_gart(&wptr_obj->obj->tbo);
if (ret) {
- DRM_ERROR("Failed to map wptr bo to GART\n");
- return ret;
+ DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret);
+ goto fail_map;
}
queue->wptr_obj.gpu_addr = amdgpu_bo_gpu_offset_no_check(wptr_obj->obj);
+
+ drm_exec_fini(&exec);
return 0;
+
+fail_map:
+ amdgpu_bo_unref(&wptr_obj->obj);
+fail_lock:
+ drm_exec_fini(&exec);
+ return ret;
+
}
static int convert_to_mes_priority(int priority)
@@ -290,8 +282,7 @@ static int mes_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr,
compute_mqd = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
if (IS_ERR(compute_mqd)) {
- DRM_ERROR("Failed to read user MQD\n");
- r = -ENOMEM;
+ r = PTR_ERR(compute_mqd);
goto free_mqd;
}
@@ -326,8 +317,7 @@ static int mes_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr,
mqd_gfx_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
if (IS_ERR(mqd_gfx_v11)) {
- DRM_ERROR("Failed to read user MQD\n");
- r = -ENOMEM;
+ r = PTR_ERR(mqd_gfx_v11);
goto free_mqd;
}
@@ -357,8 +347,7 @@ static int mes_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr,
mqd_sdma_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
if (IS_ERR(mqd_sdma_v11)) {
- DRM_ERROR("Failed to read sdma user MQD\n");
- r = -ENOMEM;
+ r = PTR_ERR(mqd_sdma_v11);
goto free_mqd;
}
r = amdgpu_userq_input_va_validate(queue, mqd_sdma_v11->csa_va,
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index 4a7180b46b71..b0e87bdc2911 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1404,6 +1404,23 @@ int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
while ((char *)sub_type_hdr + sizeof(struct crat_subtype_generic) <
((char *)crat_image) + image_len) {
+ if (!sub_type_hdr->length ||
+ sub_type_hdr->length < sizeof(struct crat_subtype_generic)) {
+ pr_warn("Invalid CRAT subtype length %u\n",
+ sub_type_hdr->length);
+ ret = -EINVAL;
+ break;
+ }
+
+ /* Validate subtype fits within remaining image */
+ if ((char *)sub_type_hdr + sub_type_hdr->length >
+ (char *)crat_image + image_len) {
+ pr_warn("CRAT subtype length %u exceeds image bounds\n",
+ sub_type_hdr->length);
+ ret = -EINVAL;
+ break;
+ }
+
if (sub_type_hdr->flags & CRAT_SUBTYPE_FLAGS_ENABLED) {
ret = kfd_parse_subtype(sub_type_hdr, device_list);
if (ret)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index af8e6f53249d..2818b62d1500 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -459,12 +459,32 @@ static int allocate_doorbell(struct qcm_process_device *qpd,
* we need the physical sdma engine id in order to get the
* correct doorbell offset.
*/
- uint32_t valid_id = idx_offset[qpd->dqm->dev->node_id *
- get_num_all_sdma_engines(qpd->dqm) +
- q->properties.sdma_engine_id]
- + (q->properties.sdma_queue_id & 1)
- * KFD_QUEUE_DOORBELL_MIRROR_OFFSET
- + (q->properties.sdma_queue_id >> 1);
+ u32 engine_base = idx_offset[dev->node_id *
+ get_num_all_sdma_engines(qpd->dqm) +
+ q->properties.sdma_engine_id];
+ u32 valid_id;
+
+ /*
+ * With the aqua_vanjaram doorbell layout, a shader doorbell
+ * write whose 32-byte block starts in the previous engine's
+ * range rings that engine instead. Engine ranges are 10
+ * indices apart (sdma_doorbell_range = 20 dwords, halved for
+ * 8-byte doorbells, in aqua_vanjaram_doorbell_index_init()),
+ * which puts odd engines 2 indices into a shared block.
+ * Aligning costs at most 2 indices, and 8 queues per engine use
+ * only 4 of the 10, so the aligned base stays in range. The
+ * 512-index mirror offset keeps the alignment.
+ */
+ if (KFD_GC_VERSION(dev) == IP_VERSION(9, 4, 3) ||
+ KFD_GC_VERSION(dev) == IP_VERSION(9, 4, 4) ||
+ KFD_GC_VERSION(dev) == IP_VERSION(9, 5, 0))
+ engine_base = ALIGN(engine_base,
+ KFD_SDMA_SHADER_DOORBELL_GRANULARITY);
+
+ valid_id = engine_base +
+ (q->properties.sdma_queue_id & 1) *
+ KFD_QUEUE_DOORBELL_MIRROR_OFFSET +
+ (q->properties.sdma_queue_id >> 1);
if (restore_id && *restore_id != valid_id)
return -EINVAL;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index d7e915040242..f2cda82cbed4 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -132,6 +132,13 @@
*/
#define KFD_QUEUE_DOORBELL_MIRROR_OFFSET 512
+/*
+ * On GC 9.4.3, 9.4.4 and 9.5.0, doorbell writes issued from shaders are
+ * routed to an SDMA engine at 32-byte granularity, i.e. 4 indices with the
+ * 8-byte SDMA doorbells used on SOC15.
+ */
+#define KFD_SDMA_SHADER_DOORBELL_GRANULARITY 4
+
/**
* enum kfd_ioctl_flags - KFD ioctl flags
* Various flags that can be set in &amdkfd_ioctl_desc.flags to control how
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index 19c3f56017ce..bb1fae39cd39 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -6799,10 +6799,15 @@ decide_crtc_timing_for_drm_display_mode(struct drm_display_mode *drm_mode,
const struct drm_display_mode *native_mode,
bool scale_enabled)
{
+ /* Inserted modes copy the native sync; a sink mode with equal totals may not */
if (scale_enabled || (
native_mode->clock == drm_mode->clock &&
native_mode->htotal == drm_mode->htotal &&
- native_mode->vtotal == drm_mode->vtotal)) {
+ native_mode->vtotal == drm_mode->vtotal &&
+ native_mode->hsync_start == drm_mode->hsync_start &&
+ native_mode->hsync_end == drm_mode->hsync_end &&
+ native_mode->vsync_start == drm_mode->vsync_start &&
+ native_mode->vsync_end == drm_mode->vsync_end)) {
if (native_mode->crtc_clock)
copy_crtc_timing_for_drm_display_mode(native_mode, drm_mode);
} else {
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
index a83fdd7b644a..ff334d05fd88 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
@@ -45,7 +45,7 @@
* in amdgpu_dm_kms.h file
*/
-#define AMDGPU_DM_MAX_DISPLAY_INDEX 31
+#define AMDGPU_DM_MAX_DISPLAY_COUNT 32
#define AMDGPU_DM_MAX_CRTC 6
@@ -830,6 +830,7 @@ struct amdgpu_dm_connector {
int sr_skip_count;
bool disallow_edp_enter_psr;
+ bool disallow_edp_enter_replay;
/* Record progress status of mst*/
uint8_t mst_status;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c
index 596a97092e0d..84d7a79e5267 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c
@@ -32,6 +32,7 @@
#include "dc.h"
#include "amdgpu_securedisplay.h"
#include "amdgpu_dm_psr.h"
+#include "amdgpu_dm_replay.h"
static const char *const pipe_crc_sources[] = {
"none",
@@ -500,6 +501,7 @@ int amdgpu_dm_crtc_configure_crc_source(struct drm_crtc *crtc,
{
struct amdgpu_device *adev = drm_to_adev(crtc->dev);
struct dc_stream_state *stream_state = dm_crtc_state->stream;
+ struct amdgpu_dm_connector *aconnector = NULL;
bool enable = amdgpu_dm_is_valid_crc_source(source);
int ret = 0;
@@ -507,11 +509,22 @@ int amdgpu_dm_crtc_configure_crc_source(struct drm_crtc *crtc,
if (!stream_state)
return -EINVAL;
+ /* Get connector from stream */
+ aconnector = (struct amdgpu_dm_connector *)stream_state->dm_stream_context;
+
mutex_lock(&adev->dm.dc_lock);
- /* For PSR1, check that the panel has exited PSR */
- if (stream_state->link->psr_settings.psr_version < DC_PSR_VERSION_SU_1)
- amdgpu_dm_psr_wait_disable(stream_state);
+
+ if (enable) {
+ /* For PSR1, check that the panel has exited PSR */
+ if (stream_state->link->psr_settings.psr_version < DC_PSR_VERSION_SU_1)
+ amdgpu_dm_psr_wait_disable(stream_state);
+
+ /* Set flag to disallow enter replay when CRC source is enabled */
+ if (aconnector)
+ aconnector->disallow_edp_enter_replay = true;
+ amdgpu_dm_replay_disable(stream_state);
+ }
/* Enable or disable CRTC CRC generation */
if (dm_is_crc_source_crtc(source) || source == AMDGPU_DM_PIPE_CRC_SOURCE_NONE) {
@@ -534,6 +547,12 @@ int amdgpu_dm_crtc_configure_crc_source(struct drm_crtc *crtc,
DYN_EXPANSION_AUTO);
}
+ if (!enable) {
+ /* Clear flag to allow enter replay when CRC source is disabled */
+ if (aconnector)
+ aconnector->disallow_edp_enter_replay = false;
+ }
+
unlock:
mutex_unlock(&adev->dm.dc_lock);
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
index 9082b862e7aa..db202f27ca38 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
@@ -508,13 +508,13 @@ static void amdgpu_dm_crtc_reset_state(struct drm_crtc *crtc)
{
struct dm_crtc_state *state;
- if (crtc->state)
- amdgpu_dm_crtc_destroy_state(crtc, crtc->state);
-
state = kzalloc(sizeof(*state), GFP_KERNEL);
- if (WARN_ON(!state))
+ if (!state)
return;
+ if (crtc->state)
+ amdgpu_dm_crtc_destroy_state(crtc, crtc->state);
+
__drm_atomic_helper_crtc_reset(crtc, &state->base);
}
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
index 19038f336155..1b0a242513c3 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
@@ -287,7 +287,7 @@ void hdcp_reset_display(struct hdcp_workqueue *hdcp_work, unsigned int link_inde
cancel_delayed_work(&hdcp_w->property_validate_dwork);
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) {
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) {
hdcp_w->encryption_status[conn_index] =
MOD_HDCP_ENCRYPTION_STATUS_HDCP_OFF;
if (hdcp_w->aconnector[conn_index]) {
@@ -334,7 +334,7 @@ static void event_property_update(struct work_struct *work)
struct drm_connector *connector;
struct drm_connector_state *conn_state;
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) {
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) {
aconnector = hdcp_work->aconnector[conn_index];
if (!aconnector)
@@ -403,7 +403,7 @@ static void event_property_validate(struct work_struct *work)
guard(mutex)(&hdcp_work->mutex);
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX;
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT;
conn_index++) {
aconnector = hdcp_work->aconnector[conn_index];
@@ -798,10 +798,10 @@ struct hdcp_workqueue *hdcp_create_workqueue(struct amdgpu_device *adev,
memset(hdcp_work[i].aconnector, 0,
sizeof(struct amdgpu_dm_connector *) *
- AMDGPU_DM_MAX_DISPLAY_INDEX);
+ AMDGPU_DM_MAX_DISPLAY_COUNT);
memset(hdcp_work[i].encryption_status, 0,
sizeof(enum mod_hdcp_encryption_status) *
- AMDGPU_DM_MAX_DISPLAY_INDEX);
+ AMDGPU_DM_MAX_DISPLAY_COUNT);
}
cp_psp->funcs.update_stream_config = update_config;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
index 4faa344f196e..1e31fc92ac71 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
@@ -44,7 +44,7 @@ struct hdcp_workqueue {
struct delayed_work callback_dwork;
struct delayed_work watchdog_timer_dwork;
struct delayed_work property_validate_dwork;
- struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_COUNT];
struct mutex mutex;
struct mod_hdcp hdcp;
@@ -52,7 +52,7 @@ struct hdcp_workqueue {
struct mod_hdcp_display display;
struct mod_hdcp_link link;
- enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_COUNT];
/* when display is unplugged from mst hub, connctor will be
* destroyed within dm_dp_mst_connector_destroy. connector
* hdcp perperties, like type, undesired, desired, enabled,
@@ -62,9 +62,9 @@ struct hdcp_workqueue {
* will be retrieved from hdcp_work within dm_dp_mst_get_modes
*/
/* un-desired, desired, enabled */
- unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_COUNT];
/* hdcp1.x, hdcp2.x */
- unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_COUNT];
uint8_t max_link;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c
index 59c7efc60c90..6cafbd228d87 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c
@@ -926,6 +926,8 @@ void amdgpu_dm_hpd_init(struct amdgpu_device *adev)
amdgpu_dm_connector = to_amdgpu_dm_connector(connector);
dc_link = amdgpu_dm_connector->dc_link;
+ if (!dc_link)
+ continue;
/*
* Get a base driver irq reference for hpd ints for the lifetime
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
index 2ecebf9a00fa..54e4b48bd78a 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
@@ -109,25 +109,7 @@ void amdgpu_dm_plane_fill_blending_from_plane_state(const struct drm_plane_state
if (plane_state->pixel_blend_mode == DRM_MODE_BLEND_PREMULTI ||
plane_state->pixel_blend_mode == DRM_MODE_BLEND_COVERAGE) {
- static const uint32_t alpha_formats[] = {
- DRM_FORMAT_ARGB8888,
- DRM_FORMAT_RGBA8888,
- DRM_FORMAT_ABGR8888,
- DRM_FORMAT_ARGB2101010,
- DRM_FORMAT_ABGR2101010,
- DRM_FORMAT_ARGB16161616,
- DRM_FORMAT_ABGR16161616,
- DRM_FORMAT_ARGB16161616F,
- };
- uint32_t format = plane_state->fb->format->format;
- unsigned int i;
-
- for (i = 0; i < ARRAY_SIZE(alpha_formats); ++i) {
- if (format == alpha_formats[i]) {
- *per_pixel_alpha = true;
- break;
- }
- }
+ *per_pixel_alpha = plane_state->fb->format->has_alpha;
if (*per_pixel_alpha && plane_state->pixel_blend_mode == DRM_MODE_BLEND_COVERAGE)
*pre_multiplied_alpha = false;
@@ -1464,17 +1446,15 @@ static const struct drm_plane_helper_funcs dm_primary_plane_helper_funcs = {
static void amdgpu_dm_plane_drm_plane_reset(struct drm_plane *plane)
{
- struct dm_plane_state *amdgpu_state = NULL;
-
- if (plane->state)
- plane->funcs->atomic_destroy_state(plane, plane->state);
+ struct dm_plane_state *amdgpu_state;
amdgpu_state = kzalloc(sizeof(*amdgpu_state), GFP_KERNEL);
- WARN_ON(amdgpu_state == NULL);
-
if (!amdgpu_state)
return;
+ if (plane->state)
+ plane->funcs->atomic_destroy_state(plane, plane->state);
+
__drm_atomic_helper_plane_reset(plane, &amdgpu_state->base);
amdgpu_state->degamma_tf = AMDGPU_TRANSFER_FUNCTION_DEFAULT;
amdgpu_state->hdr_mult = AMDGPU_HDR_MULT_DEFAULT;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c
index 80704d709e44..024bae662116 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c
@@ -154,10 +154,17 @@ bool amdgpu_dm_replay_enable(struct dc_stream_state *stream, bool wait)
{
bool replay_active = true;
struct dc_link *link = NULL;
+ struct amdgpu_dm_connector *aconnector = NULL;
if (stream == NULL)
return false;
+ /* Check if replay is disabled by connector flag */
+ aconnector = (struct amdgpu_dm_connector *)stream->dm_stream_context;
+ if (!aconnector || aconnector->disallow_edp_enter_replay) {
+ return false;
+ }
+
link = stream->link;
if (link) {
diff --git a/drivers/gpu/drm/amd/display/dc/core/dc.c b/drivers/gpu/drm/amd/display/dc/core/dc.c
index 31094d5c0f58..1c312f2e128f 100644
--- a/drivers/gpu/drm/amd/display/dc/core/dc.c
+++ b/drivers/gpu/drm/amd/display/dc/core/dc.c
@@ -1482,7 +1482,7 @@ static void disable_vbios_mode_if_required(
struct dc *dc_create(const struct dc_init_data *init_params)
{
- struct dc *dc = kzalloc(sizeof(*dc), GFP_KERNEL);
+ struct dc *dc = kvzalloc(sizeof(*dc), GFP_KERNEL);
unsigned int full_pipe_count;
if (!dc)
@@ -1530,7 +1530,7 @@ struct dc *dc_create(const struct dc_init_data *init_params)
destruct_dc:
dc_destruct(dc);
- kfree(dc);
+ kvfree(dc);
return NULL;
}
@@ -1580,7 +1580,7 @@ void dc_deinit_callbacks(struct dc *dc)
void dc_destroy(struct dc **dc)
{
dc_destruct(*dc);
- kfree(*dc);
+ kvfree(*dc);
*dc = NULL;
}
@@ -2158,8 +2158,8 @@ static enum dc_status dc_commit_state_no_check(struct dc *dc, struct dc_state *c
*/
if (dc->hwss.subvp_pipe_control_lock)
dc->hwss.subvp_pipe_control_lock(dc, context, true, true, NULL, subvp_prev_use);
- if (dc->hwss.fams2_global_control_lock)
- dc->hwss.fams2_global_control_lock(dc, context, true);
+ if (dc->hwss.dmub_hw_control_lock)
+ dc->hwss.dmub_hw_control_lock(dc, context, true);
if (dc->hwss.update_dsc_pg)
dc->hwss.update_dsc_pg(dc, context, false);
@@ -2229,8 +2229,8 @@ static enum dc_status dc_commit_state_no_check(struct dc *dc, struct dc_state *c
dc->hwss.commit_subvp_config(dc, context);
if (dc->hwss.subvp_pipe_control_lock)
dc->hwss.subvp_pipe_control_lock(dc, context, false, true, NULL, subvp_prev_use);
- if (dc->hwss.fams2_global_control_lock)
- dc->hwss.fams2_global_control_lock(dc, context, false);
+ if (dc->hwss.dmub_hw_control_lock)
+ dc->hwss.dmub_hw_control_lock(dc, context, false);
for (i = 0; i < context->stream_count; i++) {
const struct dc_link *link = context->streams[i]->link;
@@ -4184,16 +4184,16 @@ static void commit_planes_for_stream(struct dc *dc,
if (dc->hwss.subvp_pipe_control_lock)
dc->hwss.subvp_pipe_control_lock(dc, context, true, should_lock_all_pipes, NULL, subvp_prev_use);
- if (dc->hwss.fams2_global_control_lock)
- dc->hwss.fams2_global_control_lock(dc, context, true);
+ if (dc->hwss.dmub_hw_control_lock)
+ dc->hwss.dmub_hw_control_lock(dc, context, true);
dc->hwss.interdependent_update_lock(dc, context, true);
} else {
if (dc->hwss.subvp_pipe_control_lock)
dc->hwss.subvp_pipe_control_lock(dc, context, true, should_lock_all_pipes, top_pipe_to_program, subvp_prev_use);
- if (dc->hwss.fams2_global_control_lock)
- dc->hwss.fams2_global_control_lock(dc, context, true);
+ if (dc->hwss.dmub_hw_control_lock)
+ dc->hwss.dmub_hw_control_lock(dc, context, true);
/* Lock the top pipe while updating plane addrs, since freesync requires
* plane addr update event triggers to be synchronized.
@@ -4236,9 +4236,8 @@ static void commit_planes_for_stream(struct dc *dc,
dc->hwss.subvp_pipe_control_lock(dc, context, false, should_lock_all_pipes,
NULL, subvp_prev_use);
- if (dc->hwss.fams2_global_control_lock)
- dc->hwss.fams2_global_control_lock(dc, context, false);
-
+ if (dc->hwss.dmub_hw_control_lock)
+ dc->hwss.dmub_hw_control_lock(dc, context, false);
return;
}
@@ -4475,13 +4474,13 @@ static void commit_planes_for_stream(struct dc *dc,
if (should_lock_all_pipes && dc->hwss.interdependent_update_lock) {
if (dc->hwss.subvp_pipe_control_lock)
dc->hwss.subvp_pipe_control_lock(dc, context, false, should_lock_all_pipes, NULL, subvp_prev_use);
- if (dc->hwss.fams2_global_control_lock)
- dc->hwss.fams2_global_control_lock(dc, context, false);
+ if (dc->hwss.dmub_hw_control_lock)
+ dc->hwss.dmub_hw_control_lock(dc, context, false);
} else {
if (dc->hwss.subvp_pipe_control_lock)
dc->hwss.subvp_pipe_control_lock(dc, context, false, should_lock_all_pipes, top_pipe_to_program, subvp_prev_use);
- if (dc->hwss.fams2_global_control_lock)
- dc->hwss.fams2_global_control_lock(dc, context, false);
+ if (dc->hwss.dmub_hw_control_lock)
+ dc->hwss.dmub_hw_control_lock(dc, context, false);
}
// Fire manual trigger only when bottom plane is flipped
diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c
index f56d1ae879ee..f28f61787f16 100644
--- a/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c
+++ b/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c
@@ -764,11 +764,11 @@ void hwss_build_fast_sequence(struct dc *dc,
block_sequence[*num_steps].func = DMUB_SUBVP_PIPE_CONTROL_LOCK_FAST;
(*num_steps)++;
}
- if (dc->hwss.fams2_global_control_lock_fast) {
- block_sequence[*num_steps].params.fams2_global_control_lock_fast_params.dc = dc;
- block_sequence[*num_steps].params.fams2_global_control_lock_fast_params.lock = true;
- block_sequence[*num_steps].params.fams2_global_control_lock_fast_params.is_required = dc_state_is_fams2_in_use(dc, context);
- block_sequence[*num_steps].func = DMUB_FAMS2_GLOBAL_CONTROL_LOCK_FAST;
+ if (dc->hwss.dmub_hw_control_lock_fast) {
+ block_sequence[*num_steps].params.dmub_hw_control_lock_fast_params.dc = dc;
+ block_sequence[*num_steps].params.dmub_hw_control_lock_fast_params.lock = true;
+ block_sequence[*num_steps].params.dmub_hw_control_lock_fast_params.is_required = dc_state_is_fams2_in_use(dc, context);
+ block_sequence[*num_steps].func = DMUB_HW_CONTROL_LOCK_FAST;
(*num_steps)++;
}
if (dc->hwss.pipe_control_lock) {
@@ -903,11 +903,11 @@ void hwss_build_fast_sequence(struct dc *dc,
block_sequence[*num_steps].func = DMUB_SUBVP_PIPE_CONTROL_LOCK_FAST;
(*num_steps)++;
}
- if (dc->hwss.fams2_global_control_lock_fast) {
- block_sequence[*num_steps].params.fams2_global_control_lock_fast_params.dc = dc;
- block_sequence[*num_steps].params.fams2_global_control_lock_fast_params.lock = false;
- block_sequence[*num_steps].params.fams2_global_control_lock_fast_params.is_required = dc_state_is_fams2_in_use(dc, context);
- block_sequence[*num_steps].func = DMUB_FAMS2_GLOBAL_CONTROL_LOCK_FAST;
+ if (dc->hwss.dmub_hw_control_lock_fast) {
+ block_sequence[*num_steps].params.dmub_hw_control_lock_fast_params.dc = dc;
+ block_sequence[*num_steps].params.dmub_hw_control_lock_fast_params.lock = false;
+ block_sequence[*num_steps].params.dmub_hw_control_lock_fast_params.is_required = dc_state_is_fams2_in_use(dc, context);
+ block_sequence[*num_steps].func = DMUB_HW_CONTROL_LOCK_FAST;
(*num_steps)++;
}
@@ -1010,8 +1010,8 @@ void hwss_execute_sequence(struct dc *dc,
params->wait_for_dcc_meta_propagation_params.dc,
params->wait_for_dcc_meta_propagation_params.top_pipe_to_program);
break;
- case DMUB_FAMS2_GLOBAL_CONTROL_LOCK_FAST:
- dc->hwss.fams2_global_control_lock_fast(params);
+ case DMUB_HW_CONTROL_LOCK_FAST:
+ dc->hwss.dmub_hw_control_lock_fast(params);
break;
default:
ASSERT(false);
diff --git a/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c b/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c
index ee721606b883..971d0756c2ad 100644
--- a/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c
+++ b/drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_utils.c
@@ -357,14 +357,13 @@ void dml21_handle_phantom_streams_planes(const struct dc *dc, struct dc_state *c
main_plane = main_stream_status->plane_states[dc_plane_index];
/* create phantom planes for subvp enabled plane */
- dml21_add_phantom_plane(dml_ctx,
- dc,
- context,
- phantom_stream,
- main_plane,
- &dml_ctx->v21.mode_programming.programming->plane_programming[dml_plane_index]);
-
- phantoms_added = true;
+ if (dml21_add_phantom_plane(dml_ctx,
+ dc,
+ context,
+ phantom_stream,
+ main_plane,
+ &dml_ctx->v21.mode_programming.programming->plane_programming[dml_plane_index]))
+ phantoms_added = true;
}
}
}
diff --git a/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c b/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c
index bf62d42b3f78..da865bd466c9 100644
--- a/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c
+++ b/drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c
@@ -2412,7 +2412,7 @@ static void calculate_mcache_row_bytes(
DML_ASSERT(*p->num_mcaches > 0);
}
-static void calculate_mcache_setting(
+static bool calculate_mcache_setting(
struct dml2_core_internal_scratch *scratch,
struct dml2_core_calcs_calculate_mcache_setting_params *p)
{
@@ -2438,7 +2438,7 @@ static void calculate_mcache_setting(
*p->lc_comb_mcache = 0;
if (!p->dcc_enable)
- return;
+ return true;
l->is_dual_plane = dml_is_420(p->source_format) || p->source_format == dml2_rgbe_alpha;
@@ -2475,7 +2475,14 @@ static void calculate_mcache_setting(
l->l_p.mvmpg_per_mcache_lb = &l->mvmpg_per_mcache_lb_l;
calculate_mcache_row_bytes(scratch, &l->l_p);
- DML_ASSERT(*p->num_mcaches_l > 0);
+ if (*p->num_mcaches_l == 0 ||
+ (p->surf_vert ? l->mvmpg_height_l : l->mvmpg_width_l) == 0) {
+ DML_LOG_VERBOSE("DML::%s: degenerate luma viewport (num_mcaches_l=%u mvmpg_%s_l=%u) — mode not supported\n",
+ __func__, *p->num_mcaches_l,
+ p->surf_vert ? "height" : "width",
+ p->surf_vert ? l->mvmpg_height_l : l->mvmpg_width_l);
+ return false;
+ }
if (l->is_dual_plane) {
l->c_p.num_chans = p->num_chans;
@@ -2511,7 +2518,14 @@ static void calculate_mcache_setting(
l->c_p.mvmpg_per_mcache_lb = &l->mvmpg_per_mcache_lb_c;
calculate_mcache_row_bytes(scratch, &l->c_p);
- DML_ASSERT(*p->num_mcaches_c > 0);
+ if (*p->num_mcaches_c == 0 ||
+ (p->surf_vert ? l->mvmpg_height_c : l->mvmpg_width_c) == 0) {
+ DML_LOG_VERBOSE("DML::%s: degenerate chroma viewport (num_mcaches_c=%u mvmpg_%s_c=%u) — mode not supported\n",
+ __func__, *p->num_mcaches_c,
+ p->surf_vert ? "height" : "width",
+ p->surf_vert ? l->mvmpg_height_c : l->mvmpg_width_c);
+ return false;
+ }
}
// Sharing for iMALL access
@@ -2621,6 +2635,7 @@ static void calculate_mcache_setting(
*p->mcache_shift_granularity_l = l->mvmpg_access_width_l;
*p->mcache_shift_granularity_c = l->mvmpg_access_width_c;
+ return true;
}
static void calculate_mall_bw_overhead_factor(
@@ -9373,7 +9388,10 @@ static bool dml_core_mode_support(struct dml2_core_calcs_mode_support_ex *in_out
calculate_mcache_setting_params->mall_comb_mcache_c = &mode_lib->ms.mall_comb_mcache_c[k];
calculate_mcache_setting_params->lc_comb_mcache = &mode_lib->ms.lc_comb_mcache[k];
- calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params);
+ if (!calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params)) {
+ mode_lib->ms.support.ModeSupport = false;
+ return false;
+ }
}
calculate_mall_bw_overhead_factor(
@@ -10844,7 +10862,8 @@ static bool dml_core_mode_programming(struct dml2_core_calcs_mode_programming_ex
calculate_mcache_setting_params->mall_comb_mcache_l = &mode_lib->mp.mall_comb_mcache_l[k];
calculate_mcache_setting_params->mall_comb_mcache_c = &mode_lib->mp.mall_comb_mcache_c[k];
calculate_mcache_setting_params->lc_comb_mcache = &mode_lib->mp.lc_comb_mcache[k];
- calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params);
+ if (!calculate_mcache_setting(&mode_lib->scratch, calculate_mcache_setting_params))
+ return false;
}
calculate_mall_bw_overhead_factor(
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c
index 05011061822c..8688ed4d7280 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c
@@ -1592,3 +1592,55 @@ void dcn35_hardware_release(struct dc *dc)
if (dc->hwss.hw_block_power_up)
dc->hwss.hw_block_power_up(dc, &pg_update_state);
}
+
+static void disable_link_output_symclk_on_tx_off(struct dc_link *link, enum dp_link_encoding link_encoding)
+{
+ struct dc *dc = link->ctx->dc;
+ struct pipe_ctx *pipe_ctx = NULL;
+ uint8_t i;
+
+ for (i = 0; i < MAX_PIPES; i++) {
+ pipe_ctx = &dc->current_state->res_ctx.pipe_ctx[i];
+ if (pipe_ctx->stream && pipe_ctx->stream->link == link && pipe_ctx->top_pipe == NULL) {
+ pipe_ctx->clock_source->funcs->program_pix_clk(
+ pipe_ctx->clock_source,
+ &pipe_ctx->stream_res.pix_clk_params,
+ link_encoding,
+ &pipe_ctx->pll_settings);
+ break;
+ }
+ }
+}
+
+void dcn35_disable_link_output(struct dc_link *link,
+ const struct link_resource *link_res,
+ enum signal_type signal)
+{
+ struct dc *dc = link->ctx->dc;
+ const struct link_hwss *link_hwss = get_link_hwss(link, link_res);
+ struct dmcu *dmcu = dc->res_pool->dmcu;
+
+ if (signal == SIGNAL_TYPE_EDP &&
+ link->dc->hwss.edp_backlight_control &&
+ !link->skip_implict_edp_power_control)
+ link->dc->hwss.edp_backlight_control(link, false);
+ else if (dmcu != NULL && dmcu->funcs->lock_phy)
+ dmcu->funcs->lock_phy(dmcu);
+
+ if (dc_is_tmds_signal(signal) && link->phy_state.symclk_ref_cnts.otg > 0) {
+ disable_link_output_symclk_on_tx_off(link, DP_UNKNOWN_ENCODING);
+ link->phy_state.symclk_state = SYMCLK_ON_TX_OFF;
+ } else {
+ link_hwss->disable_link_output(link, link_res, signal);
+ link->phy_state.symclk_state = SYMCLK_OFF_TX_OFF;
+ }
+ /*
+ * Add the logic to extract BOTH power up and power down sequences
+ * from enable/disable link output and only call edp panel control
+ * in enable_link_dp and disable_link_dp once.
+ */
+ if (dmcu != NULL && dmcu->funcs->unlock_phy)
+ dmcu->funcs->unlock_phy(dmcu);
+
+ dc->link_srv->dp_trace_source_sequence(link, DPCD_SOURCE_SEQ_AFTER_DISABLE_LINK_PHY);
+}
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h
index 0b1d6f608edd..7e4f4eeb4750 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.h
@@ -101,4 +101,8 @@ bool dcn35_is_dp_dig_pixel_rate_div_policy(struct pipe_ctx *pipe_ctx);
void dcn35_hardware_release(struct dc *dc);
+void dcn35_disable_link_output(struct dc_link *link,
+ const struct link_resource *link_res,
+ enum signal_type signal);
+
#endif /* __DC_HWSS_DCN35_H__ */
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c
index f2f16a0bdb4f..a3bcfc13aa92 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c
@@ -107,7 +107,7 @@ static const struct hw_sequencer_funcs dcn35_funcs = {
.enable_lvds_link_output = dce110_enable_lvds_link_output,
.enable_tmds_link_output = dce110_enable_tmds_link_output,
.enable_dp_link_output = dce110_enable_dp_link_output,
- .disable_link_output = dcn32_disable_link_output,
+ .disable_link_output = dcn35_disable_link_output,
.z10_restore = dcn35_z10_restore,
.z10_save_init = dcn31_z10_save_init,
.set_disp_pattern_generator = dcn30_set_disp_pattern_generator,
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c
index 914c30a42ecf..074cea5aacfe 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c
@@ -1418,9 +1418,9 @@ void dcn401_prepare_bandwidth(struct dc *dc,
}
if (dc->debug.fams2_config.bits.enable) {
- dcn401_fams2_global_control_lock(dc, context, true);
+ dcn401_dmub_hw_control_lock(dc, context, true);
dcn401_fams2_update_config(dc, context, false);
- dcn401_fams2_global_control_lock(dc, context, false);
+ dcn401_dmub_hw_control_lock(dc, context, false);
}
if (p_state_change_support != context->bw_ctx.bw.dcn.clk.p_state_change_support) {
@@ -1439,9 +1439,9 @@ void dcn401_optimize_bandwidth(
/* enable fams2 if needed */
if (dc->debug.fams2_config.bits.enable) {
- dcn401_fams2_global_control_lock(dc, context, true);
+ dcn401_dmub_hw_control_lock(dc, context, true);
dcn401_fams2_update_config(dc, context, true);
- dcn401_fams2_global_control_lock(dc, context, false);
+ dcn401_dmub_hw_control_lock(dc, context, false);
}
/* program dchubbub watermarks */
@@ -1480,7 +1480,7 @@ void dcn401_optimize_bandwidth(
}
}
-void dcn401_fams2_global_control_lock(struct dc *dc,
+void dcn401_dmub_hw_control_lock(struct dc *dc,
struct dc_state *context,
bool lock)
{
@@ -1497,12 +1497,12 @@ void dcn401_fams2_global_control_lock(struct dc *dc,
dmub_hw_lock_mgr_inbox0_cmd(dc->ctx->dmub_srv, hw_lock_cmd);
}
-void dcn401_fams2_global_control_lock_fast(union block_sequence_params *params)
+void dcn401_dmub_hw_control_lock_fast(union block_sequence_params *params)
{
- struct dc *dc = params->fams2_global_control_lock_fast_params.dc;
- bool lock = params->fams2_global_control_lock_fast_params.lock;
+ struct dc *dc = params->dmub_hw_control_lock_fast_params.dc;
+ bool lock = params->dmub_hw_control_lock_fast_params.lock;
- if (params->fams2_global_control_lock_fast_params.is_required) {
+ if (params->dmub_hw_control_lock_fast_params.is_required) {
union dmub_inbox0_cmd_lock_hw hw_lock_cmd = { 0 };
hw_lock_cmd.bits.command_code = DMUB_INBOX0_CMD__HW_LOCK;
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h
index 2621b7725267..9591657d8eee 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h
@@ -73,11 +73,11 @@ void dcn401_optimize_bandwidth(
struct dc *dc,
struct dc_state *context);
-void dcn401_fams2_global_control_lock(struct dc *dc,
+void dcn401_dmub_hw_control_lock(struct dc *dc,
struct dc_state *context,
bool lock);
void dcn401_fams2_update_config(struct dc *dc, struct dc_state *context, bool enable);
-void dcn401_fams2_global_control_lock_fast(union block_sequence_params *params);
+void dcn401_dmub_hw_control_lock_fast(union block_sequence_params *params);
void dcn401_unblank_stream(struct pipe_ctx *pipe_ctx, struct dc_link_settings *link_settings);
void dcn401_hardware_release(struct dc *dc);
void dcn401_update_odm(struct dc *dc, struct dc_state *context,
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_init.c b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_init.c
index d6e11b7e4fce..a221b8cb6d4d 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_init.c
+++ b/drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_init.c
@@ -95,9 +95,9 @@ static const struct hw_sequencer_funcs dcn401_funcs = {
.apply_update_flags_for_phantom = dcn32_apply_update_flags_for_phantom,
.wait_for_dcc_meta_propagation = dcn401_wait_for_dcc_meta_propagation,
.is_pipe_topology_transition_seamless = dcn32_is_pipe_topology_transition_seamless,
- .fams2_global_control_lock = dcn401_fams2_global_control_lock,
+ .dmub_hw_control_lock = dcn401_dmub_hw_control_lock,
.fams2_update_config = dcn401_fams2_update_config,
- .fams2_global_control_lock_fast = dcn401_fams2_global_control_lock_fast,
+ .dmub_hw_control_lock_fast = dcn401_dmub_hw_control_lock_fast,
.program_outstanding_updates = dcn401_program_outstanding_updates,
.wait_for_all_pending_updates = dcn30_wait_for_all_pending_updates,
.detect_pipe_changes = dcn401_detect_pipe_changes,
diff --git a/drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h b/drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h
index 1723bbcf2c46..619ac4dfff07 100644
--- a/drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h
+++ b/drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h
@@ -148,7 +148,7 @@ struct wait_for_dcc_meta_propagation_params {
const struct pipe_ctx *top_pipe_to_program;
};
-struct fams2_global_control_lock_fast_params {
+struct dmub_hw_control_lock_fast_params {
struct dc *dc;
bool is_required;
bool lock;
@@ -173,7 +173,7 @@ union block_sequence_params {
struct set_ocsc_default_params set_ocsc_default_params;
struct subvp_save_surf_addr subvp_save_surf_addr;
struct wait_for_dcc_meta_propagation_params wait_for_dcc_meta_propagation_params;
- struct fams2_global_control_lock_fast_params fams2_global_control_lock_fast_params;
+ struct dmub_hw_control_lock_fast_params dmub_hw_control_lock_fast_params;
};
enum block_sequence_func {
@@ -195,7 +195,7 @@ enum block_sequence_func {
MPC_SET_OCSC_DEFAULT,
DMUB_SUBVP_SAVE_SURF_ADDR,
HUBP_WAIT_FOR_DCC_META_PROP,
- DMUB_FAMS2_GLOBAL_CONTROL_LOCK_FAST,
+ DMUB_HW_CONTROL_LOCK_FAST,
/* This must be the last value in this enum, add new ones above */
HWSS_BLOCK_SEQUENCE_FUNC_COUNT
};
@@ -452,13 +452,13 @@ struct hw_sequencer_funcs {
const struct dc_state *new_ctx);
void (*wait_for_dcc_meta_propagation)(const struct dc *dc,
const struct pipe_ctx *top_pipe_to_program);
- void (*fams2_global_control_lock)(struct dc *dc,
+ void (*dmub_hw_control_lock)(struct dc *dc,
struct dc_state *context,
bool lock);
void (*fams2_update_config)(struct dc *dc,
struct dc_state *context,
bool enable);
- void (*fams2_global_control_lock_fast)(union block_sequence_params *params);
+ void (*dmub_hw_control_lock_fast)(union block_sequence_params *params);
void (*set_long_vtotal)(struct pipe_ctx **pipe_ctx, int num_pipes, uint32_t v_total_min, uint32_t v_total_max);
void (*program_outstanding_updates)(struct dc *dc,
struct dc_state *context);
diff --git a/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c b/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c
index 8f054d9b5d57..e65577298685 100644
--- a/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c
@@ -91,9 +91,14 @@
#include "dml/dcn32/dcn32_fpu.h"
#include "dc_state_priv.h"
+#include "dc_fpu.h"
#include "dml2/dml2_wrapper.h"
+#if !defined(DC_RUN_WITH_PREEMPTION_ENABLED)
+#define DC_RUN_WITH_PREEMPTION_ENABLED(code) code
+#endif
+
#define DC_LOGGER_INIT(logger)
enum dcn32_clk_src_array_id {
@@ -1650,7 +1655,8 @@ static void dcn32_enable_phantom_plane(struct dc *dc,
if (curr_pipe->top_pipe && curr_pipe->top_pipe->plane_state == curr_pipe->plane_state)
phantom_plane = prev_phantom_plane;
else
- phantom_plane = dc_state_create_phantom_plane(dc, context, curr_pipe->plane_state);
+ DC_RUN_WITH_PREEMPTION_ENABLED(phantom_plane =
+ dc_state_create_phantom_plane(dc, context, curr_pipe->plane_state));
if (!phantom_plane)
continue;
diff --git a/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c b/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c
index 46094cca2974..a226eb2fbbe6 100644
--- a/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c
+++ b/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c
@@ -190,13 +190,12 @@ static int aspeed_gfx_load(struct drm_device *drm)
}
reset_control_deassert(priv->rst);
- priv->clk = devm_clk_get(drm->dev, NULL);
+ priv->clk = devm_clk_get_enabled(drm->dev, NULL);
if (IS_ERR(priv->clk)) {
dev_err(&pdev->dev,
- "missing or invalid clk device tree entry");
+ "failed to get and enable clock\n");
return PTR_ERR(priv->clk);
}
- clk_prepare_enable(priv->clk);
/* Sanitize control registers */
writel(0, priv->base + CRT_CTRL1);
diff --git a/drivers/gpu/drm/display/drm_dp_helper.c b/drivers/gpu/drm/display/drm_dp_helper.c
index 77336bc3e62c..4fc371f0fa12 100644
--- a/drivers/gpu/drm/display/drm_dp_helper.c
+++ b/drivers/gpu/drm/display/drm_dp_helper.c
@@ -4378,7 +4378,9 @@ static int dp_aux_backlight_update_status(struct backlight_device *bd)
if (!backlight_is_blank(bd)) {
if (!bl->enabled) {
- drm_edp_backlight_enable(bl->aux, &bl->info, brightness);
+ ret = drm_edp_backlight_enable(bl->aux, &bl->info, brightness);
+ if (ret)
+ return ret;
bl->enabled = true;
return 0;
}
diff --git a/drivers/gpu/drm/drm_fb_helper.c b/drivers/gpu/drm/drm_fb_helper.c
index 71889daef3fc..3599423b774b 100644
--- a/drivers/gpu/drm/drm_fb_helper.c
+++ b/drivers/gpu/drm/drm_fb_helper.c
@@ -1800,10 +1800,10 @@ __drm_fb_helper_initial_config_and_unlock(struct drm_fb_helper *fb_helper)
ret = drm_fb_helper_single_fb_probe(fb_helper);
if (ret < 0) {
- if (ret == -EAGAIN) {
- fb_helper->deferred_setup = true;
+ /* modesets are probed but fb_helper->fb isn't; defer restore too */
+ fb_helper->deferred_setup = true;
+ if (ret == -EAGAIN)
ret = 0;
- }
mutex_unlock(&fb_helper->lock);
goto err_drm_fb_helper_release_info;
diff --git a/drivers/gpu/drm/gud/gud_pipe.c b/drivers/gpu/drm/gud/gud_pipe.c
index 8892ea631388..8a02afb92efb 100644
--- a/drivers/gpu/drm/gud/gud_pipe.c
+++ b/drivers/gpu/drm/gud/gud_pipe.c
@@ -507,6 +507,7 @@ int gud_plane_atomic_check(struct drm_plane *plane,
for_each_new_connector_in_state(state, connector, connector_state, i) {
if (connector_state->crtc)
break;
+ connector_state = NULL;
}
/*
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index bcae382acdf5..61f1b8f172a7 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -1765,6 +1765,7 @@ static int eb_reinit_userptr(struct i915_execbuffer *eb)
static noinline int eb_relocate_parse_slow(struct i915_execbuffer *eb)
{
+ const unsigned int count = eb->buffer_count;
bool have_copy = false;
struct eb_vma *ev;
int err = 0;
@@ -1867,7 +1868,6 @@ static noinline int eb_relocate_parse_slow(struct i915_execbuffer *eb)
out:
if (have_copy) {
- const unsigned int count = eb->buffer_count;
unsigned int i;
for (i = 0; i < count; i++) {
diff --git a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
index 7f389cb0bde4..e2fc1b1f0979 100644
--- a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
+++ b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
@@ -3930,11 +3930,11 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
struct drm_i915_private *i915 = siblings[0]->i915;
struct virtual_engine *ve;
unsigned int n;
- int err;
+ int err = -ENOMEM;
ve = kzalloc(struct_size(ve, siblings, count), GFP_KERNEL);
if (!ve)
- return ERR_PTR(-ENOMEM);
+ goto err;
ve->base.i915 = i915;
ve->base.gt = siblings[0]->gt;
@@ -3966,10 +3966,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
intel_engine_init_execlists(&ve->base);
ve->base.sched_engine = i915_sched_engine_create(ENGINE_VIRTUAL);
- if (!ve->base.sched_engine) {
- err = -ENOMEM;
- goto err_put;
- }
+ if (!ve->base.sched_engine)
+ goto err_noput;
ve->base.sched_engine->private_data = &ve->base;
ve->base.cops = &virtual_context_ops;
@@ -3985,10 +3983,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
intel_context_init(&ve->context, &ve->base);
ve->base.breadcrumbs = intel_breadcrumbs_create(NULL);
- if (!ve->base.breadcrumbs) {
- err = -ENOMEM;
+ if (!ve->base.breadcrumbs)
goto err_put;
- }
for (n = 0; n < count; n++) {
struct intel_engine_cs *sibling = siblings[n];
@@ -4063,8 +4059,13 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
virtual_engine_initial_hint(ve);
return &ve->context;
+err_noput:
+ kfree(ve);
+ goto err;
+
err_put:
intel_context_put(&ve->context);
+err:
return ERR_PTR(err);
}
diff --git a/drivers/gpu/drm/i915/gt/intel_rps.c b/drivers/gpu/drm/i915/gt/intel_rps.c
index 4da94098bd3e..076537f9677d 100644
--- a/drivers/gpu/drm/i915/gt/intel_rps.c
+++ b/drivers/gpu/drm/i915/gt/intel_rps.c
@@ -1904,7 +1904,8 @@ static void rps_work(struct work_struct *work)
out:
spin_lock_irq(gt->irq_lock);
- gen6_gt_pm_unmask_irq(gt, rps->pm_events);
+ if (intel_rps_is_active(rps))
+ gen6_gt_pm_unmask_irq(gt, rps->pm_events);
spin_unlock_irq(gt->irq_lock);
}
diff --git a/drivers/gpu/drm/imagination/pvr_mmu.c b/drivers/gpu/drm/imagination/pvr_mmu.c
index 146eb003d0ab..88b00ed2d542 100644
--- a/drivers/gpu/drm/imagination/pvr_mmu.c
+++ b/drivers/gpu/drm/imagination/pvr_mmu.c
@@ -2339,7 +2339,7 @@ void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx)
* @ctx: MMU context associated with owning VM context.
* @sgt: Scatter gather table containing pages pinned for use by this context.
* @device_addr: Virtual device address at the start of the requested mapping.
- * @sgt_offset: Start offset of the requested device-virtual memory mapping.
+ * @sgt_offset: Offset into @sgt of the start of the requested mapping.
* @size: Size in bytes of the requested device-virtual memory mapping. For an
* unmapping, this should be zero so that no page tables are allocated.
*
@@ -2352,7 +2352,6 @@ struct pvr_mmu_op_context *
pvr_mmu_op_context_create(struct pvr_mmu_context *ctx, struct sg_table *sgt,
u64 device_addr, u64 sgt_offset, u64 size)
{
- u64 start_addr = device_addr + sgt_offset;
int err;
struct pvr_mmu_op_context *op_ctx =
@@ -2368,18 +2367,18 @@ pvr_mmu_op_context_create(struct pvr_mmu_context *ctx, struct sg_table *sgt,
if (size) {
/*
- * The number of page table objects we need to prealloc is
- * indicated by the mapping size, start address and the sizes
- * of the areas mapped per PT or PD. The range calculation is
- * identical to that for the index into a table for a device
- * address, so we reuse those functions here.
+ * The page tables needed are set by the device-virtual range
+ * being mapped: one level 1 table per 1GiB region and one
+ * level 0 table per 2MiB region the range touches. Tables that
+ * already exist are not consumed, so this is an upper bound.
*/
- const u32 l1_start_idx = pvr_page_table_l2_idx(start_addr);
- const u32 l1_end_idx = pvr_page_table_l2_idx(start_addr + size);
- const u32 l1_count = l1_end_idx - l1_start_idx + 1;
- const u32 l0_start_idx = pvr_page_table_l1_idx(start_addr);
- const u32 l0_end_idx = pvr_page_table_l1_idx(start_addr + size);
- const u32 l0_count = l0_end_idx - l0_start_idx + 1;
+ const u64 last_addr = device_addr + size - 1;
+ const u64 l1_count =
+ (last_addr >> ROGUE_MMUCTRL_VADDR_PC_INDEX_SHIFT) -
+ (device_addr >> ROGUE_MMUCTRL_VADDR_PC_INDEX_SHIFT) + 1;
+ const u64 l0_count =
+ (last_addr >> ROGUE_MMUCTRL_VADDR_PD_INDEX_SHIFT) -
+ (device_addr >> ROGUE_MMUCTRL_VADDR_PD_INDEX_SHIFT) + 1;
/*
* Alloc and push page table entries until we have enough of
diff --git a/drivers/gpu/drm/imagination/pvr_vm.c b/drivers/gpu/drm/imagination/pvr_vm.c
index e59b93c1a6e1..bc699b600de1 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.c
+++ b/drivers/gpu/drm/imagination/pvr_vm.c
@@ -417,6 +417,8 @@ pvr_vm_gpuva_unmap(struct drm_gpuva_op *op, void *op_ctx)
static int
pvr_vm_gpuva_remap(struct drm_gpuva_op *op, void *op_ctx)
{
+ /* The split parts belong to the object of the mapping being split. */
+ struct drm_gpuvm_bo *vm_bo = op->remap.unmap->va->vm_bo;
struct pvr_vm_bind_op *ctx = op_ctx;
u64 va_start = 0, va_range = 0;
int err;
@@ -432,14 +434,12 @@ pvr_vm_gpuva_remap(struct drm_gpuva_op *op, void *op_ctx)
drm_gpuva_remap(&ctx->prev_va->base, &ctx->next_va->base, &op->remap);
if (op->remap.prev) {
- pvr_gem_object_get(gem_to_pvr_gem(ctx->prev_va->base.gem.obj));
- drm_gpuva_link(&ctx->prev_va->base, ctx->gpuvm_bo);
+ drm_gpuva_link(&ctx->prev_va->base, vm_bo);
ctx->prev_va = NULL;
}
if (op->remap.next) {
- pvr_gem_object_get(gem_to_pvr_gem(ctx->next_va->base.gem.obj));
- drm_gpuva_link(&ctx->next_va->base, ctx->gpuvm_bo);
+ drm_gpuva_link(&ctx->next_va->base, vm_bo);
ctx->next_va = NULL;
}
diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
index d1eaa849b197..f56ac2e0adbe 100644
--- a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
+++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c
@@ -2633,7 +2633,6 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev)
struct msm_drm_private *priv = dev->dev_private;
struct platform_device *pdev = priv->gpu_pdev;
struct adreno_platform_config *config = pdev->dev.platform_data;
- struct device_node *node;
struct a6xx_gpu *a6xx_gpu;
struct adreno_gpu *adreno_gpu;
struct msm_gpu *gpu;
@@ -2653,7 +2652,8 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev)
adreno_gpu->registers = NULL;
/* Check if there is a GMU phandle and set it up */
- node = of_parse_phandle(pdev->dev.of_node, "qcom,gmu", 0);
+ struct device_node *node __free(device_node) =
+ of_parse_phandle(pdev->dev.of_node, "qcom,gmu", 0);
/* FIXME: How do we gracefully handle this? */
BUG_ON(!node);
@@ -2701,7 +2701,6 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev)
ret = a6xx_gmu_wrapper_init(a6xx_gpu, node);
else
ret = a6xx_gmu_init(a6xx_gpu, node);
- of_node_put(node);
if (ret) {
a6xx_destroy(&(a6xx_gpu->base.base));
return ERR_PTR(ret);
diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c b/drivers/gpu/drm/panthor/panthor_mmu.c
index 5647a2474109..d985f324772a 100644
--- a/drivers/gpu/drm/panthor/panthor_mmu.c
+++ b/drivers/gpu/drm/panthor/panthor_mmu.c
@@ -313,6 +313,9 @@ struct panthor_vm {
u64 end;
} kernel_auto_va;
+ /** @user_va_range: Upper boundary of VAs VM users can map objects against. */
+ u64 user_va_range;
+
/** @as: Address space related fields. */
struct {
/**
@@ -2385,6 +2388,8 @@ panthor_vm_create(struct panthor_device *ptdev, bool for_mcu,
va_range = full_va_range;
}
+ vm->user_va_range = kernel_va_start;
+
mutex_init(&vm->mm_lock);
drm_mm_init(&vm->mm, kernel_va_start, kernel_va_size);
vm->kernel_auto_va.start = auto_kernel_va_start;
@@ -2471,6 +2476,10 @@ panthor_vm_bind_prepare_op_ctx(struct drm_file *file,
if (!IS_ALIGNED(op->va | op->size | op->bo_offset, vm_pgsz))
return -EINVAL;
+ /* We don't allow mappings that overlap with kbo's reserved range */
+ if (range_overflows(op->va, op->size, vm->user_va_range))
+ return -EINVAL;
+
switch (op->flags & DRM_PANTHOR_VM_BIND_OP_TYPE_MASK) {
case DRM_PANTHOR_VM_BIND_OP_TYPE_MAP:
gem = drm_gem_object_lookup(file, op->bo_handle);
diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c
index 85e24ca47448..2170373ea784 100644
--- a/drivers/gpu/drm/panthor/panthor_sched.c
+++ b/drivers/gpu/drm/panthor/panthor_sched.c
@@ -1427,7 +1427,7 @@ cs_slot_process_fault_event_locked(struct panthor_device *ptdev,
u64 cs_extract = queue->iface.output->extract;
struct panthor_job *job;
- spin_lock(&queue->fence_ctx.lock);
+ guard(spinlock_irqsave)(&queue->fence_ctx.lock);
list_for_each_entry(job, &queue->fence_ctx.in_flight_jobs, node) {
if (cs_extract >= job->ringbuf.end)
continue;
@@ -1437,7 +1437,6 @@ cs_slot_process_fault_event_locked(struct panthor_device *ptdev,
dma_fence_set_error(job->done_fence, -EINVAL);
}
- spin_unlock(&queue->fence_ctx.lock);
}
if (group) {
@@ -2104,13 +2103,13 @@ group_term_post_processing(struct panthor_group *group)
if (!queue)
continue;
- spin_lock(&queue->fence_ctx.lock);
- list_for_each_entry_safe(job, tmp, &queue->fence_ctx.in_flight_jobs, node) {
- list_move_tail(&job->node, &faulty_jobs);
- dma_fence_set_error(job->done_fence, err);
- dma_fence_signal_locked(job->done_fence);
+ scoped_guard(spinlock_irqsave, &queue->fence_ctx.lock) {
+ list_for_each_entry_safe(job, tmp, &queue->fence_ctx.in_flight_jobs, node) {
+ list_move_tail(&job->node, &faulty_jobs);
+ dma_fence_set_error(job->done_fence, err);
+ dma_fence_signal_locked(job->done_fence);
+ }
}
- spin_unlock(&queue->fence_ctx.lock);
/* Manually update the syncobj seqno to unblock waiters. */
syncobj = group->syncobjs->kmap + (i * sizeof(*syncobj));
@@ -2991,15 +2990,15 @@ static void group_sync_upd_work(struct work_struct *work)
syncobj = group->syncobjs->kmap + (queue_idx * sizeof(*syncobj));
- spin_lock(&queue->fence_ctx.lock);
- list_for_each_entry_safe(job, job_tmp, &queue->fence_ctx.in_flight_jobs, node) {
- if (syncobj->seqno < job->done_fence->seqno)
- break;
+ scoped_guard(spinlock_irqsave, &queue->fence_ctx.lock) {
+ list_for_each_entry_safe(job, job_tmp, &queue->fence_ctx.in_flight_jobs, node) {
+ if (syncobj->seqno < job->done_fence->seqno)
+ break;
- list_move_tail(&job->node, &done_jobs);
- dma_fence_signal_locked(job->done_fence);
+ list_move_tail(&job->node, &done_jobs);
+ dma_fence_signal_locked(job->done_fence);
+ }
}
- spin_unlock(&queue->fence_ctx.lock);
}
dma_fence_end_signalling(cookie);
@@ -3244,9 +3243,8 @@ queue_run_job(struct drm_sched_job *sched_job)
job->ringbuf.end = job->ringbuf.start + (instrs.count * sizeof(u64));
panthor_job_get(&job->base);
- spin_lock(&queue->fence_ctx.lock);
- list_add_tail(&job->node, &queue->fence_ctx.in_flight_jobs);
- spin_unlock(&queue->fence_ctx.lock);
+ scoped_guard(spinlock_irqsave, &queue->fence_ctx.lock)
+ list_add_tail(&job->node, &queue->fence_ctx.in_flight_jobs);
/* Make sure the ring buffer is updated before the INSERT
* register.
@@ -3549,6 +3547,9 @@ int panthor_group_create(struct panthor_file *pfile,
hweight64(group_args->tiler_core_mask) < group_args->max_tiler_cores)
return -EINVAL;
+ if (group_args->queues.count > sched->cs_slot_count)
+ return -EINVAL;
+
group = kzalloc(sizeof(*group), GFP_KERNEL);
if (!group)
return -ENOMEM;
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index d7a722d875cb..493471012820 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -1175,17 +1175,13 @@ ttm_bo_swapout_cb(struct ttm_lru_walk *walk, struct ttm_buffer_object *bo)
bdev->funcs->swap_notify(bo);
if (ttm_tt_is_populated(tt)) {
- spin_lock(&bdev->lru_lock);
- ttm_resource_del_bulk_move(bo->resource, bo);
- spin_unlock(&bdev->lru_lock);
-
ret = ttm_tt_swapout(bdev, tt, swapout_walk->gfp_flags);
-
- spin_lock(&bdev->lru_lock);
- if (ret)
- ttm_resource_add_bulk_move(bo->resource, bo);
- ttm_resource_move_to_lru_tail(bo->resource);
- spin_unlock(&bdev->lru_lock);
+ if (!ret) {
+ spin_lock(&bdev->lru_lock);
+ ttm_resource_del_bulk_move_unevictable(bo->resource, bo);
+ ttm_resource_move_to_lru_tail(bo->resource);
+ spin_unlock(&bdev->lru_lock);
+ }
}
out:
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 85bb9f9f1f61..7aa35899e33c 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -1094,19 +1094,14 @@ long ttm_bo_shrink(struct ttm_operation_ctx *ctx, struct ttm_buffer_object *bo,
if (lret < 0)
return lret;
- if (bo->bulk_move) {
- spin_lock(&bdev->lru_lock);
- ttm_resource_del_bulk_move(bo->resource, bo);
- spin_unlock(&bdev->lru_lock);
- }
-
lret = ttm_tt_backup(bdev, bo->ttm, (struct ttm_backup_flags)
{.purge = flags.purge,
.writeback = flags.writeback});
- if (lret <= 0 && bo->bulk_move) {
+ if (lret > 0) {
spin_lock(&bdev->lru_lock);
- ttm_resource_add_bulk_move(bo->resource, bo);
+ ttm_resource_del_bulk_move_unevictable(bo->resource, bo);
+ ttm_resource_move_to_lru_tail(bo->resource);
spin_unlock(&bdev->lru_lock);
}
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 8396cab34cb0..a6a948fa0b83 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -291,6 +291,19 @@ void ttm_resource_del_bulk_move(struct ttm_resource *res,
ttm_lru_bulk_move_del(bo->bulk_move, res);
}
+/*
+ * Remove a resource from its bulk_move, bypassing the unevictable check.
+ * Use only when the resource is known to still be tracked in the range despite
+ * the BO having just become unevictable; asserts that this is the case.
+ */
+void ttm_resource_del_bulk_move_unevictable(struct ttm_resource *res,
+ struct ttm_buffer_object *bo)
+{
+ WARN_ON_ONCE(!ttm_resource_unevictable(res, bo));
+ if (bo->bulk_move)
+ ttm_lru_bulk_move_del(bo->bulk_move, res);
+}
+
/* Move a resource to the LRU or bulk tail */
void ttm_resource_move_to_lru_tail(struct ttm_resource *res)
{
diff --git a/drivers/gpu/drm/vc4/vc4_bo.c b/drivers/gpu/drm/vc4/vc4_bo.c
index cd7996592db3..206be4336b0b 100644
--- a/drivers/gpu/drm/vc4/vc4_bo.c
+++ b/drivers/gpu/drm/vc4/vc4_bo.c
@@ -1054,6 +1054,7 @@ static void vc4_bo_cache_destroy(struct drm_device *dev, void *unused)
cancel_work_sync(&vc4->bo_cache.time_work);
vc4_bo_cache_purge(dev);
+ kfree(vc4->bo_cache.size_list);
for (i = 0; i < vc4->num_labels; i++) {
if (vc4->bo_labels[i].num_allocated) {
diff --git a/drivers/gpu/drm/vc4/vc4_drv.c b/drivers/gpu/drm/vc4/vc4_drv.c
index c7cb1e3a6434..298e6cf3be54 100644
--- a/drivers/gpu/drm/vc4/vc4_drv.c
+++ b/drivers/gpu/drm/vc4/vc4_drv.c
@@ -361,8 +361,11 @@ static int vc4_drm_bind(struct device *dev)
}
ret = aperture_remove_all_conflicting_devices(driver->name);
- if (ret)
+ if (ret) {
+ if (firmware)
+ rpi_firmware_put(firmware);
goto err;
+ }
if (firmware) {
ret = rpi_firmware_property(firmware,
diff --git a/drivers/gpu/drm/vc4/vc4_drv.h b/drivers/gpu/drm/vc4/vc4_drv.h
index 221d8e01d539..d9ef519d6b02 100644
--- a/drivers/gpu/drm/vc4/vc4_drv.h
+++ b/drivers/gpu/drm/vc4/vc4_drv.h
@@ -1054,7 +1054,7 @@ void vc4_plane_async_set_fb(struct drm_plane *plane,
/* vc4_v3d.c */
extern struct platform_driver vc4_v3d_driver;
extern const struct of_device_id vc4_v3d_dt_match[];
-int vc4_v3d_get_bin_slot(struct vc4_dev *vc4);
+int vc4_v3d_get_bin_slot(struct vc4_dev *vc4, long timeout);
int vc4_v3d_bin_bo_get(struct vc4_dev *vc4, bool *used);
void vc4_v3d_bin_bo_put(struct vc4_dev *vc4);
int vc4_v3d_pm_get(struct vc4_dev *vc4);
diff --git a/drivers/gpu/drm/vc4/vc4_gem.c b/drivers/gpu/drm/vc4/vc4_gem.c
index 6887631f2d8b..5eadf11f289a 100644
--- a/drivers/gpu/drm/vc4/vc4_gem.c
+++ b/drivers/gpu/drm/vc4/vc4_gem.c
@@ -877,11 +877,17 @@ vc4_complete_exec(struct drm_device *dev, struct vc4_exec_info *exec)
drm_gem_object_put(&bo->base.base);
}
- /* Free up the allocation of any bin slots we used. */
+ /* Free up the allocation of any bin slots we used. Jobs that ran to
+ * completion had their slots released in vc4_irq_finish_render_job().
+ * Only jobs that never completed still have slots to be released here.
+ */
spin_lock_irqsave(&vc4->job_lock, irqflags);
vc4->bin_alloc_used &= ~exec->bin_slots;
spin_unlock_irqrestore(&vc4->job_lock, irqflags);
+ /* Let anyone waiting on the binner pool retry. */
+ wake_up_all(&vc4->job_wait_queue);
+
/* Release the reference on the binner BO if needed. */
if (exec->bin_bo_used)
vc4_v3d_bin_bo_put(vc4);
diff --git a/drivers/gpu/drm/vc4/vc4_irq.c b/drivers/gpu/drm/vc4/vc4_irq.c
index 4fdad88e53e1..7d783441ffe2 100644
--- a/drivers/gpu/drm/vc4/vc4_irq.c
+++ b/drivers/gpu/drm/vc4/vc4_irq.c
@@ -74,7 +74,7 @@ vc4_overflow_mem_work(struct work_struct *work)
bo = vc4->bin_bo;
- bin_bo_slot = vc4_v3d_get_bin_slot(vc4);
+ bin_bo_slot = vc4_v3d_get_bin_slot(vc4, msecs_to_jiffies(500));
if (bin_bo_slot < 0) {
drm_err(&vc4->base, "Couldn't allocate binner overflow mem\n");
goto complete;
@@ -165,6 +165,10 @@ vc4_irq_finish_render_job(struct drm_device *dev)
trace_vc4_rcl_end_irq(dev, exec->seqno);
vc4->finished_seqno++;
+
+ vc4->bin_alloc_used &= ~exec->bin_slots;
+ exec->bin_slots = 0;
+
list_move_tail(&exec->head, &vc4->job_done_list);
nextbin = vc4_first_bin_job(vc4);
diff --git a/drivers/gpu/drm/vc4/vc4_v3d.c b/drivers/gpu/drm/vc4/vc4_v3d.c
index 68f098e45944..03589596b7fd 100644
--- a/drivers/gpu/drm/vc4/vc4_v3d.c
+++ b/drivers/gpu/drm/vc4/vc4_v3d.c
@@ -9,6 +9,7 @@
#include <linux/component.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
+#include <linux/timer.h>
#include "vc4_drv.h"
#include "vc4_regs.h"
@@ -170,46 +171,42 @@ static void vc4_v3d_init_hw(struct drm_device *dev)
V3D_WRITE(V3D_VPMBASE, 0);
}
-int vc4_v3d_get_bin_slot(struct vc4_dev *vc4)
+static int bin_slot_try_alloc(struct vc4_dev *vc4)
{
- struct drm_device *dev = &vc4->base;
- unsigned long irqflags;
int slot;
- uint64_t seqno = 0;
- struct vc4_exec_info *exec;
- if (WARN_ON_ONCE(vc4->gen > VC4_GEN_4))
- return -ENODEV;
+ guard(spinlock_irqsave)(&vc4->job_lock);
-try_again:
- spin_lock_irqsave(&vc4->job_lock, irqflags);
slot = ffs(~vc4->bin_alloc_used);
if (slot != 0) {
/* Switch from ffs() bit index to a 0-based index. */
slot--;
vc4->bin_alloc_used |= BIT(slot);
- spin_unlock_irqrestore(&vc4->job_lock, irqflags);
- return slot;
+ } else {
+ slot = -ENOMEM;
}
- /* Couldn't find an open slot. Wait for render to complete
- * and try again.
- */
- exec = vc4_last_render_job(vc4);
- if (exec)
- seqno = exec->seqno;
- spin_unlock_irqrestore(&vc4->job_lock, irqflags);
+ return slot;
+}
- if (seqno) {
- int ret = vc4_wait_for_seqno(dev, seqno, ~0ull, true);
+int vc4_v3d_get_bin_slot(struct vc4_dev *vc4, long timeout)
+{
+ int slot;
+ long ret;
- if (ret == 0)
- goto try_again;
+ if (WARN_ON_ONCE(vc4->gen > VC4_GEN_4))
+ return -ENODEV;
+ /* If the pool is full, wait for a job to release its slots. */
+ ret = wait_event_interruptible_timeout(vc4->job_wait_queue,
+ (slot = bin_slot_try_alloc(vc4)) >= 0,
+ timeout);
+ if (ret < 0)
return ret;
- }
+ if (ret == 0)
+ return -ENOMEM;
- return -ENOMEM;
+ return slot;
}
/*
@@ -495,8 +492,15 @@ static void vc4_v3d_unbind(struct device *dev, struct device *master,
struct drm_device *drm = data;
struct vc4_dev *vc4 = to_vc4_dev(drm);
+ /* A straggler vc4_reset() re-enables the interrupt. */
+ timer_shutdown_sync(&vc4->hangcheck.timer);
+ cancel_work_sync(&vc4->hangcheck.reset_work);
+
vc4_irq_uninstall(drm);
+ /* Flush rather than cancel, so queued completions release their jobs. */
+ flush_work(&vc4->job_done_work);
+
/* Disable the binner's overflow memory address, so the next
* driver probe (if any) doesn't try to reuse our old
* allocation.
diff --git a/drivers/gpu/drm/vc4/vc4_validate.c b/drivers/gpu/drm/vc4/vc4_validate.c
index 205d360a007a..140a525b9e59 100644
--- a/drivers/gpu/drm/vc4/vc4_validate.c
+++ b/drivers/gpu/drm/vc4/vc4_validate.c
@@ -401,7 +401,7 @@ validate_tile_binning_config(VALIDATE_ARGS)
return -EINVAL;
}
- bin_slot = vc4_v3d_get_bin_slot(vc4);
+ bin_slot = vc4_v3d_get_bin_slot(vc4, MAX_SCHEDULE_TIMEOUT);
if (bin_slot < 0) {
if (bin_slot != -EINTR && bin_slot != -ERESTARTSYS) {
drm_err(dev, "Failed to allocate binner memory: %d\n",
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_cursor_plane.c b/drivers/gpu/drm/vmwgfx/vmwgfx_cursor_plane.c
index c53bb9376b36..ff54ea9cc72b 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_cursor_plane.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_cursor_plane.c
@@ -324,6 +324,7 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
unsigned long kmap_num;
SVGA3dCopyBox *box;
u32 box_count;
+ u64 src_extent;
void *virtual;
bool is_iomem;
struct vmw_dma_cmd {
@@ -335,6 +336,8 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
vmw_surface_get_desc(VMW_CURSOR_SNOOP_FORMAT);
const u32 image_pitch = VMW_CURSOR_SNOOP_WIDTH * desc->pitchBytesPerBlock;
+ BUILD_BUG_ON(sizeof(*srf->snooper.image) != desc->pitchBytesPerBlock);
+
cmd = container_of(header, struct vmw_dma_cmd, header);
/* No snooper installed, nothing to copy */
@@ -372,8 +375,19 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
return;
}
+ if (box->w == 0 || box->h == 0)
+ return;
+
+ src_extent = (u64)(box->h - 1) * cmd->dma.guest.pitch +
+ (u64)box->w * desc->pitchBytesPerBlock;
+ if (src_extent > bo->base.size) {
+ DRM_ERROR("Cursor snoop source of %llu bytes exceeds the %zu byte buffer\n",
+ src_extent, bo->base.size);
+ return;
+ }
+
kmap_offset = cmd->dma.guest.ptr.offset >> PAGE_SHIFT;
- kmap_num = (VMW_CURSOR_SNOOP_HEIGHT * image_pitch) >> PAGE_SHIFT;
+ kmap_num = PFN_UP(src_extent);
ret = ttm_bo_reserve(bo, true, false, NULL);
if (unlikely(ret != 0)) {
@@ -387,14 +401,16 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
virtual = ttm_kmap_obj_virtual(&map, &is_iomem);
- if (box->w == VMW_CURSOR_SNOOP_WIDTH && cmd->dma.guest.pitch == image_pitch) {
+ if (box->w == VMW_CURSOR_SNOOP_WIDTH &&
+ box->h == VMW_CURSOR_SNOOP_HEIGHT &&
+ cmd->dma.guest.pitch == image_pitch) {
memcpy(srf->snooper.image, virtual,
VMW_CURSOR_SNOOP_HEIGHT * image_pitch);
} else {
- /* Image is unsigned pointer. */
+ /* The destination stride is in pixels. */
for (i = 0; i < box->h; i++)
- memcpy(srf->snooper.image + i * image_pitch,
- virtual + i * cmd->dma.guest.pitch,
+ memcpy(srf->snooper.image + i * VMW_CURSOR_SNOOP_WIDTH,
+ virtual + (size_t)i * cmd->dma.guest.pitch,
box->w * desc->pitchBytesPerBlock);
}
srf->snooper.id++;
diff --git a/drivers/gpu/drm/xe/display/xe_fb_pin.c b/drivers/gpu/drm/xe/display/xe_fb_pin.c
index 5bf00505dab5..fa5c70478713 100644
--- a/drivers/gpu/drm/xe/display/xe_fb_pin.c
+++ b/drivers/gpu/drm/xe/display/xe_fb_pin.c
@@ -308,7 +308,7 @@ static struct i915_vma *__xe_pin_fb_vma(const struct intel_framebuffer *fb,
drm_exec_retry_on_contention(&exec);
xe_validation_retry_on_oom(&ctx, &ret);
if (!ret)
- ttm_bo_pin(&bo->ttm);
+ xe_bo_pin_account(bo);
}
if (ret)
goto err;
@@ -325,7 +325,7 @@ static struct i915_vma *__xe_pin_fb_vma(const struct intel_framebuffer *fb,
err_unpin:
ttm_bo_reserve(&bo->ttm, false, false, NULL);
- ttm_bo_unpin(&bo->ttm);
+ xe_bo_unpin_account(bo);
ttm_bo_unreserve(&bo->ttm);
err:
kfree(vma);
@@ -346,7 +346,7 @@ static void __xe_unpin_fb_vma(struct i915_vma *vma)
xe_ggtt_node_remove(vma->node, false);
ttm_bo_reserve(&vma->bo->ttm, false, false, NULL);
- ttm_bo_unpin(&vma->bo->ttm);
+ xe_bo_unpin_account(vma->bo);
ttm_bo_unreserve(&vma->bo->ttm);
kfree(vma);
}
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 5eec1253c9f2..6d3db1517c8a 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -426,6 +426,100 @@ static void xe_ttm_tt_account_subtract(struct xe_device *xe, struct ttm_tt *tt)
xe_shrinker_mod_pages(xe->mem.shrinker, -(long)tt->num_pages, 0);
}
+static bool xe_ttm_bo_is_imported(struct ttm_buffer_object *tbo)
+{
+ dma_resv_assert_held(tbo->base.resv);
+
+ return tbo->ttm &&
+ (tbo->ttm->page_flags & (TTM_TT_FLAG_EXTERNAL | TTM_TT_FLAG_EXTERNAL_MAPPABLE)) ==
+ TTM_TT_FLAG_EXTERNAL;
+}
+
+/*
+ * Account @bo's pages as pinned for the shrinker. Removes @bo's pages
+ * from the shrinker's shrinkable / purgeable counts on the transition
+ * from unpinned to pinned. Must be called with @bo's dma-resv held,
+ * after &ttm_buffer_object.pin_count has been incremented by
+ * ttm_bo_pin(). Safe to call unconditionally regardless of which pin
+ * path (kernel, external, framebuffer, backup object, ...) is pinning
+ * @bo, since it only acts on the true 0->1 pin_count transition.
+ *
+ * Imported bos (xe_ttm_bo_is_imported()) are excluded: for those,
+ * xe_ttm_tt_populate() never actually populates the tt or calls
+ * xe_ttm_tt_account_add(), but TTM core's ttm_tt_populate() still
+ * unconditionally marks the tt as populated on driver-hook success.
+ * Relying on ttm_tt_is_populated() for these would subtract pages that
+ * were never added, underflowing the shrinker's counts.
+ */
+static void xe_bo_account_pin(struct xe_bo *bo)
+{
+ struct xe_device *xe = xe_bo_device(bo);
+
+ if (bo->ttm.pin_count == 1 && bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm) &&
+ !xe_ttm_bo_is_imported(&bo->ttm))
+ xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
+}
+
+/*
+ * Counterpart to xe_bo_account_pin(). Must be called with @bo's dma-resv
+ * held, before &ttm_buffer_object.pin_count is decremented by
+ * ttm_bo_unpin(), so that the check against the true 1->0 transition sees
+ * the pin count that is about to be released. See xe_bo_account_pin() for
+ * why imported bos are excluded.
+ *
+ * On the true last unpin, also removes @bo from whichever pinned-bo list
+ * (external or kernel_bo_present) it may currently be linked on, since a
+ * bo's final unpin can happen through a pin path (e.g. framebuffer,
+ * backup object) that has no notion of, or ownership over, that list.
+ * This is safe and list-agnostic: list_del_init() only needs the node
+ * itself, not knowledge of which list it is threaded through, and is a
+ * no-op if @bo is not linked.
+ */
+static void xe_bo_account_unpin(struct xe_bo *bo)
+{
+ struct xe_device *xe = xe_bo_device(bo);
+ bool last_unpin = bo->ttm.pin_count == 1;
+
+ if (last_unpin && !list_empty(&bo->pinned_link)) {
+ spin_lock(&xe->pinned.lock);
+ list_del_init(&bo->pinned_link);
+ spin_unlock(&xe->pinned.lock);
+ }
+
+ if (last_unpin && bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm) &&
+ !xe_ttm_bo_is_imported(&bo->ttm))
+ xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+}
+
+/**
+ * xe_bo_pin_account() - Pin a bo and account its pages for the shrinker
+ * @bo: The buffer object to pin.
+ *
+ * Pins @bo via ttm_bo_pin() and updates the shrinker's shrinkable /
+ * purgeable page accounting to match. Must be called with @bo's dma-resv
+ * held. Safe to call regardless of which pin path (kernel, external,
+ * framebuffer, backup object, ...) is pinning @bo.
+ */
+void xe_bo_pin_account(struct xe_bo *bo)
+{
+ ttm_bo_pin(&bo->ttm);
+ xe_bo_account_pin(bo);
+}
+
+/**
+ * xe_bo_unpin_account() - Unpin a bo and account its pages for the shrinker
+ * @bo: The buffer object to unpin.
+ *
+ * Counterpart to xe_bo_pin_account(). Updates the shrinker's shrinkable /
+ * purgeable page accounting to match, then unpins @bo via ttm_bo_unpin().
+ * Must be called with @bo's dma-resv held.
+ */
+void xe_bo_unpin_account(struct xe_bo *bo)
+{
+ xe_bo_account_unpin(bo);
+ ttm_bo_unpin(&bo->ttm);
+}
+
static void update_global_total_pages(struct ttm_device *ttm_dev,
long num_pages)
{
@@ -1202,7 +1296,8 @@ int xe_bo_notifier_prepare_pinned(struct xe_bo *bo)
}
backup->parent_obj = xe_bo_get(bo); /* Released by bo_destroy */
- ttm_bo_pin(&backup->ttm);
+ /* Note: XE_BO_FLAG_SYSTEM resolves to XE_PL_TT so already populated. */
+ xe_bo_pin_account(backup);
bo->backup_obj = backup;
}
@@ -1222,7 +1317,7 @@ int xe_bo_notifier_unprepare_pinned(struct xe_bo *bo)
{
xe_bo_lock(bo, false);
if (bo->backup_obj) {
- ttm_bo_unpin(&bo->backup_obj->ttm);
+ xe_bo_unpin_account(bo->backup_obj);
xe_bo_put(bo->backup_obj);
bo->backup_obj = NULL;
}
@@ -1433,7 +1528,7 @@ int xe_bo_restore_pinned(struct xe_bo *bo)
xe_bo_vunmap(backup);
if (!bo->backup_obj) {
if (xe_bo_is_pinned(backup))
- ttm_bo_unpin(&backup->ttm);
+ xe_bo_unpin_account(backup);
xe_bo_put(backup);
}
out_unlock_bo:
@@ -1837,15 +1932,6 @@ static vm_fault_t xe_err_to_fault_t(int err)
return VM_FAULT_SIGBUS;
}
-static bool xe_ttm_bo_is_imported(struct ttm_buffer_object *tbo)
-{
- dma_resv_assert_held(tbo->base.resv);
-
- return tbo->ttm &&
- (tbo->ttm->page_flags & (TTM_TT_FLAG_EXTERNAL | TTM_TT_FLAG_EXTERNAL_MAPPABLE)) ==
- TTM_TT_FLAG_EXTERNAL;
-}
-
static vm_fault_t xe_bo_cpu_fault_fastpath(struct vm_fault *vmf, struct xe_device *xe,
struct xe_bo *bo, bool needs_rpm)
{
@@ -2766,12 +2852,13 @@ uint64_t vram_region_gpu_offset(struct ttm_resource *res)
int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
{
struct xe_device *xe = xe_bo_device(bo);
+ bool first_pin = !xe_bo_is_pinned(bo);
int err;
xe_assert(xe, !bo->vm);
xe_assert(xe, xe_bo_is_user(bo));
- if (!xe_bo_is_pinned(bo)) {
+ if (first_pin) {
if (!in_place) {
err = xe_bo_validate(bo, NULL, false, exec);
if (err)
@@ -2783,9 +2870,7 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
spin_unlock(&xe->pinned.lock);
}
- ttm_bo_pin(&bo->ttm);
- if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
- xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
+ xe_bo_pin_account(bo);
/*
* FIXME: If we always use the reserve / unreserve functions for locking
@@ -2841,9 +2926,7 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
spin_unlock(&xe->pinned.lock);
}
- ttm_bo_pin(&bo->ttm);
- if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
- xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
+ xe_bo_pin_account(bo);
/*
* FIXME: If we always use the reserve / unreserve functions for locking
@@ -2872,14 +2955,7 @@ void xe_bo_unpin_external(struct xe_bo *bo)
xe_assert(xe, xe_bo_is_pinned(bo));
xe_assert(xe, xe_bo_is_user(bo));
- spin_lock(&xe->pinned.lock);
- if (bo->ttm.pin_count == 1 && !list_empty(&bo->pinned_link))
- list_del_init(&bo->pinned_link);
- spin_unlock(&xe->pinned.lock);
-
- ttm_bo_unpin(&bo->ttm);
- if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
- xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+ xe_bo_unpin_account(bo);
/*
* FIXME: If we always use the reserve / unreserve functions for locking
@@ -2897,21 +2973,16 @@ void xe_bo_unpin(struct xe_bo *bo)
xe_assert(xe, xe_bo_is_pinned(bo));
if (mem_type_is_vram(place->mem_type) || bo->flags & XE_BO_FLAG_GGTT) {
- spin_lock(&xe->pinned.lock);
xe_assert(xe, !list_empty(&bo->pinned_link));
- list_del_init(&bo->pinned_link);
- spin_unlock(&xe->pinned.lock);
if (bo->backup_obj) {
if (xe_bo_is_pinned(bo->backup_obj))
- ttm_bo_unpin(&bo->backup_obj->ttm);
+ xe_bo_unpin_account(bo->backup_obj);
xe_bo_put(bo->backup_obj);
bo->backup_obj = NULL;
}
}
- ttm_bo_unpin(&bo->ttm);
- if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
- xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+ xe_bo_unpin_account(bo);
}
/**
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 949b248c5adf..0ed05e4c36a8 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -225,6 +225,8 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
void xe_bo_unpin_external(struct xe_bo *bo);
void xe_bo_unpin(struct xe_bo *bo);
+void xe_bo_pin_account(struct xe_bo *bo);
+void xe_bo_unpin_account(struct xe_bo *bo);
int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
struct drm_exec *exec);
diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c
index 05a84ea6bd6a..1d5deb149137 100644
--- a/drivers/gpu/drm/xe/xe_device.c
+++ b/drivers/gpu/drm/xe/xe_device.c
@@ -487,7 +487,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
WQ_MEM_RECLAIM);
xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
xe->unordered_wq = alloc_workqueue("xe-unordered-wq", 0, 0);
- xe->destroy_wq = alloc_workqueue("xe-destroy-wq", 0, 0);
+ xe->destroy_wq = alloc_workqueue("xe-destroy-wq", WQ_MEM_RECLAIM, 0);
if (!xe->ordered_wq || !xe->unordered_wq ||
!xe->preempt_fence_wq || !xe->destroy_wq) {
/*
diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
index 2ed03fd074ff..453a4eb3ffd1 100644
--- a/drivers/gpu/drm/xe/xe_device_types.h
+++ b/drivers/gpu/drm/xe/xe_device_types.h
@@ -452,7 +452,7 @@ struct xe_device {
/** @unordered_wq: used to serialize unordered work */
struct workqueue_struct *unordered_wq;
- /** @destroy_wq: used to serialize user destroy work, like queue */
+ /** @destroy_wq: used to serialize SVM pagemap destroy work */
struct workqueue_struct *destroy_wq;
/** @tiles: device tiles */
diff --git a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
index c30c0e3ccbbb..107e136a9084 100644
--- a/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
+++ b/drivers/gpu/drm/xe/xe_guc_exec_queue_types.h
@@ -10,6 +10,7 @@
#include <linux/workqueue.h>
#include "xe_gpu_scheduler_types.h"
+#include "xe_hw_fence_types.h"
struct dma_fence;
struct xe_exec_queue;
@@ -24,6 +25,10 @@ struct xe_guc_exec_queue {
struct rcu_head rcu;
/** @sched: GPU scheduler for this xe_exec_queue */
struct xe_gpu_scheduler sched;
+ /**
+ * @name: Scheduler timeline name, kept with @sched until RCU free.
+ */
+ char name[MAX_FENCE_NAME_LEN];
/** @entity: Scheduler entity for this xe_exec_queue */
struct xe_sched_entity entity;
/**
diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
index b846fe013538..cd0e160ac5b0 100644
--- a/drivers/gpu/drm/xe/xe_guc_submit.c
+++ b/drivers/gpu/drm/xe/xe_guc_submit.c
@@ -12,6 +12,7 @@
#include <linux/dma-fence-array.h>
#include <linux/math64.h>
+#include <drm/drm_drv.h>
#include <drm/drm_managed.h>
#include "abi/guc_actions_abi.h"
@@ -40,6 +41,7 @@
#include "xe_macros.h"
#include "xe_map.h"
#include "xe_mocs.h"
+#include "xe_module.h"
#include "xe_pm.h"
#include "xe_ring_ops_types.h"
#include "xe_sched_job.h"
@@ -231,17 +233,9 @@ static bool exec_queue_killed_or_banned_or_wedged(struct xe_exec_queue *q)
static void guc_submit_fini(struct drm_device *drm, void *arg)
{
struct xe_guc *guc = arg;
- struct xe_device *xe = guc_to_xe(guc);
struct xe_gt *gt = guc_to_gt(guc);
- int ret;
-
- ret = wait_event_timeout(guc->submission_state.fini_wq,
- xa_empty(&guc->submission_state.exec_queue_lookup),
- HZ * 5);
-
- drain_workqueue(xe->destroy_wq);
- xe_gt_assert(gt, ret);
+ xe_gt_assert(gt, xa_empty(&guc->submission_state.exec_queue_lookup));
xa_destroy(&guc->submission_state.exec_queue_lookup);
}
@@ -309,8 +303,6 @@ int xe_guc_submit_init(struct xe_guc *guc, unsigned int num_ids)
xa_init(&guc->submission_state.exec_queue_lookup);
- init_waitqueue_head(&guc->submission_state.fini_wq);
-
primelockdep(guc);
guc->submission_state.initialized = true;
@@ -395,8 +387,7 @@ static void __release_guc_id(struct xe_guc *guc, struct xe_exec_queue *q, u32 xa
xe_guc_id_mgr_release_locked(&guc->submission_state.idm,
q->guc->id, q->width);
- if (xa_empty(&guc->submission_state.exec_queue_lookup))
- wake_up(&guc->submission_state.fini_wq);
+ mutex_unlock(&guc->submission_state.lock);
}
static int alloc_guc_id(struct xe_guc *guc, struct xe_exec_queue *q)
@@ -1288,8 +1279,14 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
* doesn't work for SRIOV. For now assuming timeouts in wedged mode are
* genuine timeouts.
*/
+ /*
+ * Killed queues must not newly wedge the device, but preserve an
+ * already-wedged state to avoid warning on teardown timeouts.
+ */
if (!exec_queue_killed(q))
wedged = guc_submit_hint_wedged(exec_queue_to_guc(q));
+ else
+ wedged = xe_device_wedged(xe);
/* Engine state now stable, disable scheduling to check timestamp */
if (!wedged && exec_queue_registered(q)) {
@@ -1383,9 +1380,10 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
* Kernel jobs should never fail, nor should VM jobs if they do
* somethings has gone wrong and the GT needs a reset
*/
- xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_KERNEL,
+ xe_gt_WARN(q->gt, !wedged && q->flags & EXEC_QUEUE_FLAG_KERNEL,
"Kernel-submitted job timed out\n");
- xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_VM && !exec_queue_killed(q),
+ xe_gt_WARN(q->gt, !wedged && q->flags & EXEC_QUEUE_FLAG_VM &&
+ !exec_queue_killed(q),
"VM job timed out on non-killed execqueue\n");
if (!wedged && (q->flags & EXEC_QUEUE_FLAG_KERNEL ||
(q->flags & EXEC_QUEUE_FLAG_VM && !exec_queue_killed(q)))) {
@@ -1443,6 +1441,7 @@ static void guc_exec_queue_fini(struct xe_exec_queue *q)
{
struct xe_guc_exec_queue *ge = q->guc;
struct xe_guc *guc = exec_queue_to_guc(q);
+ struct drm_device *drm = &guc_to_xe(guc)->drm;
release_guc_id(guc, q);
xe_sched_entity_fini(&ge->entity);
@@ -1453,16 +1452,24 @@ static void guc_exec_queue_fini(struct xe_exec_queue *q)
* (timeline name).
*/
kfree_rcu(ge, rcu);
+
+ drm_dev_put(drm);
}
-static void __guc_exec_queue_destroy_async(struct work_struct *w)
+static void guc_exec_queue_do_destroy(struct xe_exec_queue *q)
{
- struct xe_guc_exec_queue *ge =
- container_of(w, struct xe_guc_exec_queue, destroy_async);
- struct xe_exec_queue *q = ge->q;
+ struct xe_guc_exec_queue *ge = q->guc;
struct xe_guc *guc = exec_queue_to_guc(q);
+ struct xe_device *xe = guc_to_xe(guc);
+ struct drm_device *drm = &xe->drm;
- xe_pm_runtime_get(guc_to_xe(guc));
+ /*
+ * guc_exec_queue_fini() drops the queue's drm_device ref.
+ * Keep the device alive until the PM-runtime put.
+ */
+ drm_dev_get(drm);
+
+ xe_pm_runtime_get(xe);
trace_xe_exec_queue_destroy(q);
if (xe_exec_queue_is_lr(q))
@@ -1472,21 +1479,28 @@ static void __guc_exec_queue_destroy_async(struct work_struct *w)
xe_exec_queue_fini(q);
- xe_pm_runtime_put(guc_to_xe(guc));
+ xe_pm_runtime_put(xe);
+
+ drm_dev_put(drm);
}
-static void guc_exec_queue_destroy_async(struct xe_exec_queue *q)
+static void __guc_exec_queue_destroy_async(struct work_struct *w)
{
- struct xe_guc *guc = exec_queue_to_guc(q);
- struct xe_device *xe = guc_to_xe(guc);
+ struct xe_guc_exec_queue *ge =
+ container_of(w, struct xe_guc_exec_queue, destroy_async);
+
+ guc_exec_queue_do_destroy(ge->q);
+}
+static void guc_exec_queue_destroy_async(struct xe_exec_queue *q)
+{
INIT_WORK(&q->guc->destroy_async, __guc_exec_queue_destroy_async);
/* We must block on kernel engines so slabs are empty on driver unload */
if (q->flags & EXEC_QUEUE_FLAG_PERMANENT || exec_queue_wedged(q))
- __guc_exec_queue_destroy_async(&q->guc->destroy_async);
+ guc_exec_queue_do_destroy(q);
else
- queue_work(xe->destroy_wq, &q->guc->destroy_async);
+ xe_destroy_wq_queue(&q->guc->destroy_async);
}
static void __guc_exec_queue_destroy(struct xe_guc *guc, struct xe_exec_queue *q)
@@ -1651,6 +1665,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
{
struct xe_gpu_scheduler *sched;
struct xe_guc *guc = exec_queue_to_guc(q);
+ struct drm_device *drm = &guc_to_xe(guc)->drm;
struct xe_guc_exec_queue *ge;
long timeout;
int err, i;
@@ -1661,6 +1676,8 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
if (!ge)
return -ENOMEM;
+ drm_dev_get(drm);
+
q->guc = ge;
ge->q = q;
init_rcu_head(&ge->rcu);
@@ -1674,7 +1691,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
err = xe_sched_init(&ge->sched, &drm_sched_ops, &xe_sched_ops,
NULL, q->lrc[0]->ring.size / MAX_JOB_SIZE_BYTES, 64,
timeout, guc_to_gt(guc)->ordered_wq, NULL,
- q->name, gt_to_xe(q->gt)->drm.dev);
+ ge->name, gt_to_xe(q->gt)->drm.dev);
if (err)
goto err_free;
@@ -1701,6 +1718,8 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
xe_exec_queue_assign_name(q, q->guc->id);
+ strscpy(ge->name, q->name, sizeof(ge->name));
+
trace_xe_exec_queue_create(q);
return 0;
@@ -1712,6 +1731,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
xe_sched_fini(&ge->sched);
err_free:
kfree(ge);
+ drm_dev_put(drm);
return err;
}
diff --git a/drivers/gpu/drm/xe/xe_guc_types.h b/drivers/gpu/drm/xe/xe_guc_types.h
index c7b9642b41ba..31a2acb63ac3 100644
--- a/drivers/gpu/drm/xe/xe_guc_types.h
+++ b/drivers/gpu/drm/xe/xe_guc_types.h
@@ -100,8 +100,6 @@ struct xe_guc {
* even initialized - before that not even the lock is valid
*/
bool initialized;
- /** @submission_state.fini_wq: submit fini wait queue */
- wait_queue_head_t fini_wq;
} submission_state;
/** @hwconfig: Hardware config state */
diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c
index 1e8c0b92d712..36e163222c2d 100644
--- a/drivers/gpu/drm/xe/xe_migrate.c
+++ b/drivers/gpu/drm/xe/xe_migrate.c
@@ -1171,6 +1171,7 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
* content.
* @src_bo: The buffer object @src is currently bound to.
* @read_write : Creates BB commands for CCS read/write.
+ * @bound: Device is bound
*
* Directly clearing the BB lacks atomicity and can lead to undefined
* behavior if the vCPU is halted mid-operation during the clearing
@@ -1183,7 +1184,8 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
* Returns: None.
*/
void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo,
- enum xe_sriov_vf_ccs_rw_ctxs read_write)
+ enum xe_sriov_vf_ccs_rw_ctxs read_write,
+ bool bound)
{
struct xe_bb *bb = src_bo->bb_ccs[read_write];
struct xe_device *xe = xe_bo_device(src_bo);
@@ -1197,13 +1199,16 @@ void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo,
bb_pool = ctx->mem.ccs_bb_pool;
guard(mutex) (xe_sa_bo_swap_guard(bb_pool));
- xe_sa_bo_swap_shadow(bb_pool);
- cs = xe_sa_bo_cpu_addr(bb->bo);
- memset(cs, MI_NOOP, bb->len * sizeof(u32));
- xe_sriov_vf_ccs_rw_update_bb_addr(ctx);
+ if (bound) {
+ xe_sa_bo_swap_shadow(bb_pool);
- xe_sa_bo_sync_shadow(bb->bo);
+ cs = xe_sa_bo_cpu_addr(bb->bo);
+ memset(cs, MI_NOOP, bb->len * sizeof(u32));
+ xe_sriov_vf_ccs_rw_update_bb_addr(ctx);
+
+ xe_sa_bo_sync_shadow(bb->bo);
+ }
xe_bb_free(bb, NULL);
src_bo->bb_ccs[read_write] = NULL;
diff --git a/drivers/gpu/drm/xe/xe_migrate.h b/drivers/gpu/drm/xe/xe_migrate.h
index 9665a0aff547..dd0b0f837681 100644
--- a/drivers/gpu/drm/xe/xe_migrate.h
+++ b/drivers/gpu/drm/xe/xe_migrate.h
@@ -137,7 +137,8 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
enum xe_sriov_vf_ccs_rw_ctxs read_write);
void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo,
- enum xe_sriov_vf_ccs_rw_ctxs read_write);
+ enum xe_sriov_vf_ccs_rw_ctxs read_write,
+ bool bound);
struct xe_lrc *xe_migrate_lrc(struct xe_migrate *migrate);
struct xe_exec_queue *xe_migrate_exec_queue(struct xe_migrate *migrate);
diff --git a/drivers/gpu/drm/xe/xe_mmio.c b/drivers/gpu/drm/xe/xe_mmio.c
index 6752881af093..079a707217da 100644
--- a/drivers/gpu/drm/xe/xe_mmio.c
+++ b/drivers/gpu/drm/xe/xe_mmio.c
@@ -283,6 +283,7 @@ u64 xe_mmio_read64_2x32(struct xe_mmio *mmio, struct xe_reg reg)
return (u64)udw << 32 | ldw;
}
+#define __XE_MMIO_WAIT_MAX_BACKOFF_100MS (100 * USEC_PER_MSEC)
static int __xe_mmio_wait32(struct xe_mmio *mmio, struct xe_reg reg, u32 mask, u32 val,
u32 timeout_us, u32 *out_val, bool atomic, bool expect_match)
{
@@ -313,10 +314,10 @@ static int __xe_mmio_wait32(struct xe_mmio *mmio, struct xe_reg reg, u32 mask, u
wait = ktime_us_delta(end, cur);
if (atomic)
- udelay(wait);
+ udelay(min_t(s64, wait, (MAX_UDELAY_MS * 1000)));
else
usleep_range(wait, wait << 1);
- wait <<= 1;
+ wait = min_t(s64, wait << 1, __XE_MMIO_WAIT_MAX_BACKOFF_100MS);
}
if (ret != 0) {
diff --git a/drivers/gpu/drm/xe/xe_module.c b/drivers/gpu/drm/xe/xe_module.c
index d08338fc3bc1..88d66bee85c6 100644
--- a/drivers/gpu/drm/xe/xe_module.c
+++ b/drivers/gpu/drm/xe/xe_module.c
@@ -7,6 +7,7 @@
#include <linux/init.h>
#include <linux/module.h>
+#include <linux/workqueue.h>
#include <drm/drm_module.h>
@@ -101,6 +102,50 @@ static int xe_check_nomodeset(void)
return 0;
}
+static struct workqueue_struct *xe_destroy_wq;
+
+static int __init xe_destroy_wq_module_init(void)
+{
+ xe_destroy_wq = alloc_workqueue("xe-guc-destroy-wq", WQ_UNBOUND, 0);
+ if (!xe_destroy_wq)
+ return -ENOMEM;
+ return 0;
+}
+
+static void xe_destroy_wq_module_exit(void)
+{
+ if (xe_destroy_wq)
+ destroy_workqueue(xe_destroy_wq);
+ xe_destroy_wq = NULL;
+}
+
+/**
+ * xe_destroy_wq_queue() - Queue work on the destroy workqueue
+ * @work: work item to queue
+ *
+ * The destroy workqueue has module lifetime and is used for GuC exec queue
+ * teardown that can outlive a single xe_device. SVM pagemap destroy uses the
+ * per-device xe->destroy_wq instead.
+ *
+ * Return: %true if @work was queued, %false if it was already pending.
+ */
+bool xe_destroy_wq_queue(struct work_struct *work)
+{
+ return queue_work(xe_destroy_wq, work);
+}
+
+/**
+ * xe_destroy_wq_flush() - Flush the destroy workqueue
+ *
+ * Drains all pending destroy work. Called from PCI remove to ensure
+ * teardown ordering before the device is destroyed.
+ */
+void xe_destroy_wq_flush(void)
+{
+ if (xe_destroy_wq)
+ flush_workqueue(xe_destroy_wq);
+}
+
struct init_funcs {
int (*init)(void);
void (*exit)(void);
@@ -122,6 +167,10 @@ static const struct init_funcs init_funcs[] = {
.init = xe_sched_job_module_init,
.exit = xe_sched_job_module_exit,
},
+ {
+ .init = xe_destroy_wq_module_init,
+ .exit = xe_destroy_wq_module_exit,
+ },
{
.init = xe_register_pci_driver,
.exit = xe_unregister_pci_driver,
diff --git a/drivers/gpu/drm/xe/xe_module.h b/drivers/gpu/drm/xe/xe_module.h
index b66849539270..75a2b91c5449 100644
--- a/drivers/gpu/drm/xe/xe_module.h
+++ b/drivers/gpu/drm/xe/xe_module.h
@@ -8,6 +8,8 @@
#include <linux/types.h>
+struct work_struct;
+
/* Module modprobe variables */
struct xe_modparam {
bool force_execlist;
@@ -27,5 +29,8 @@ struct xe_modparam {
extern struct xe_modparam xe_modparam;
+bool xe_destroy_wq_queue(struct work_struct *work);
+void xe_destroy_wq_flush(void);
+
#endif
diff --git a/drivers/gpu/drm/xe/xe_pci.c b/drivers/gpu/drm/xe/xe_pci.c
index b78729211790..10c5d852cf2b 100644
--- a/drivers/gpu/drm/xe/xe_pci.c
+++ b/drivers/gpu/drm/xe/xe_pci.c
@@ -798,6 +798,12 @@ static void xe_pci_remove(struct pci_dev *pdev)
return;
xe_device_remove(xe);
+
+ /*
+ * Preserve remove-time flush after moving destroy work to module
+ * lifetime.
+ */
+ xe_destroy_wq_flush();
xe_pm_fini(xe);
}
diff --git a/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c b/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c
index e03714ef7ae9..167d74ea502d 100644
--- a/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c
+++ b/drivers/gpu/drm/xe/xe_sriov_vf_ccs.c
@@ -3,6 +3,8 @@
* Copyright © 2025 Intel Corporation
*/
+#include <drm/drm_drv.h>
+
#include "instructions/xe_mi_commands.h"
#include "instructions/xe_gpu_commands.h"
#include "xe_bb.h"
@@ -372,7 +374,7 @@ int xe_sriov_vf_ccs_attach_bo(struct xe_bo *bo, struct ttm_resource *new_mem)
*/
for_each_ccs_rw_ctx(ctx_id) {
if (bo->bb_ccs[ctx_id])
- xe_migrate_ccs_rw_copy_clear(bo, ctx_id);
+ xe_migrate_ccs_rw_copy_clear(bo, ctx_id, true);
}
return err;
}
@@ -392,19 +394,27 @@ int xe_sriov_vf_ccs_detach_bo(struct xe_bo *bo)
struct xe_device *xe = xe_bo_device(bo);
enum xe_sriov_vf_ccs_rw_ctxs ctx_id;
struct xe_bb *bb;
+ bool bound;
+ int idx;
xe_assert(xe, IS_VF_CCS_READY(xe));
if (!xe_bo_has_valid_ccs_bb(bo))
return 0;
+ bound = drm_dev_enter(&xe->drm, &idx);
+
for_each_ccs_rw_ctx(ctx_id) {
bb = bo->bb_ccs[ctx_id];
if (!bb)
continue;
- xe_migrate_ccs_rw_copy_clear(bo, ctx_id);
+ xe_migrate_ccs_rw_copy_clear(bo, ctx_id, bound);
}
+
+ if (bound)
+ drm_dev_exit(idx);
+
return 0;
}
diff --git a/drivers/hid/hid-logitech-hidpp.c b/drivers/hid/hid-logitech-hidpp.c
index 94ffda22ef41..a2bb7d8b15b6 100644
--- a/drivers/hid/hid-logitech-hidpp.c
+++ b/drivers/hid/hid-logitech-hidpp.c
@@ -2854,18 +2854,19 @@ static int hidpp_ff_init(struct hidpp_device *hidpp,
* ownership to FF core
*/
data = kmemdup(data, sizeof(*data), GFP_KERNEL);
- if (!data)
- return -ENOMEM;
+ if (!data) {
+ error = -ENOMEM;
+ goto err_destroy_ff;
+ }
data->effect_ids = kcalloc(num_slots, sizeof(int), GFP_KERNEL);
if (!data->effect_ids) {
- kfree(data);
- return -ENOMEM;
+ error = -ENOMEM;
+ goto err_free_data;
}
data->wq = create_singlethread_workqueue("hidpp-ff-sendqueue");
if (!data->wq) {
- kfree(data->effect_ids);
- kfree(data);
- return -ENOMEM;
+ error = -ENOMEM;
+ goto err_free_effect_ids;
}
data->hidpp = hidpp;
@@ -2895,6 +2896,14 @@ static int hidpp_ff_init(struct hidpp_device *hidpp,
version);
return 0;
+
+err_free_effect_ids:
+ kfree(data->effect_ids);
+err_free_data:
+ kfree(data);
+err_destroy_ff:
+ input_ff_destroy(dev);
+ return error;
}
/* ************************************************************************** */
diff --git a/drivers/hid/hid-steam.c b/drivers/hid/hid-steam.c
index 80d2eabc930e..84c90091d45f 100644
--- a/drivers/hid/hid-steam.c
+++ b/drivers/hid/hid-steam.c
@@ -327,12 +327,14 @@ struct steam_device {
spinlock_t lock;
struct hid_device *hdev, *client_hdev;
struct mutex report_mutex;
+ struct mutex registration_mutex;
unsigned long client_opened;
struct input_dev __rcu *input;
struct input_dev __rcu *sensors;
unsigned long quirks;
struct work_struct work_connect;
bool connected;
+ bool registered;
char serial_no[STEAM_SERIAL_LEN + 1];
struct power_supply_desc battery_desc;
struct power_supply __rcu *battery;
@@ -346,7 +348,6 @@ struct steam_device {
u16 rumble_right;
unsigned int sensor_timestamp_us;
unsigned int sensor_update_rate_us;
- struct work_struct unregister_work;
};
static int steam_recv_report(struct steam_device *steam,
@@ -510,7 +511,6 @@ static int steam_get_serial(struct steam_device *steam)
u8 cmd[] = {ID_GET_STRING_ATTRIBUTE, sizeof(steam->serial_no), ATTRIB_STR_UNIT_SERIAL};
u8 reply[3 + STEAM_SERIAL_LEN + 1];
- guard(mutex)(&steam->report_mutex);
ret = steam_send_report(steam, cmd, sizeof(cmd));
if (ret < 0)
return ret;
@@ -537,7 +537,6 @@ static int steam_get_attributes(struct steam_device *steam)
int i;
struct steam_controller_attribute *attr;
- guard(mutex)(&steam->report_mutex);
ret = steam_send_report(steam, cmd, sizeof(cmd));
if (ret < 0)
return ret;
@@ -599,9 +598,7 @@ static inline int steam_haptic_pulse(struct steam_device *steam, u8 pad,
report[8] = count >> 8;
report[9] = gain;
- mutex_lock(&steam->report_mutex);
ret = steam_send_report(steam, report, sizeof(report));
- mutex_unlock(&steam->report_mutex);
return ret;
}
@@ -621,9 +618,7 @@ static inline int steam_haptic_rumble(struct steam_device *steam,
report[9] = left_gain;
report[10] = right_gain;
- mutex_lock(&steam->report_mutex);
ret = steam_send_report(steam, report, sizeof(report));
- mutex_unlock(&steam->report_mutex);
return ret;
}
@@ -631,8 +626,11 @@ static void steam_haptic_rumble_cb(struct work_struct *work)
{
struct steam_device *steam = container_of(work, struct steam_device,
rumble_work);
+
+ mutex_lock(&steam->report_mutex);
steam_haptic_rumble(steam, 0, steam->rumble_left,
steam->rumble_right, 2, 0);
+ mutex_unlock(&steam->report_mutex);
}
#ifdef CONFIG_STEAM_FF
@@ -653,7 +651,6 @@ static void steam_set_lizard_mode(struct steam_device *steam, bool enable)
if (steam->gamepad_mode)
enable = false;
- mutex_lock(&steam->report_mutex);
if (enable) {
/* enable esc, enter, cursors */
steam_send_report_byte(steam, ID_SET_DEFAULT_DIGITAL_MAPPINGS);
@@ -678,14 +675,13 @@ static void steam_set_lizard_mode(struct steam_device *steam, bool enable)
0);
}
}
- mutex_unlock(&steam->report_mutex);
}
static int steam_input_open(struct input_dev *dev)
{
struct steam_device *steam = input_get_drvdata(dev);
unsigned long flags;
- bool set_lizard_mode;
+ bool client_opened;
/*
* Disabling lizard mode automatically is only done on the Steam
@@ -694,9 +690,10 @@ static int steam_input_open(struct input_dev *dev)
*/
if (!(steam->quirks & STEAM_QUIRK_DECK)) {
spin_lock_irqsave(&steam->lock, flags);
- set_lizard_mode = !steam->client_opened && lizard_mode;
+ client_opened = steam->client_opened;
spin_unlock_irqrestore(&steam->lock, flags);
- if (set_lizard_mode)
+ guard(mutex)(&steam->report_mutex);
+ if (!client_opened && lizard_mode)
steam_set_lizard_mode(steam, false);
}
@@ -707,13 +704,14 @@ static void steam_input_close(struct input_dev *dev)
{
struct steam_device *steam = input_get_drvdata(dev);
unsigned long flags;
- bool set_lizard_mode;
+ bool client_opened;
if (!(steam->quirks & STEAM_QUIRK_DECK)) {
spin_lock_irqsave(&steam->lock, flags);
- set_lizard_mode = !steam->client_opened && lizard_mode;
+ client_opened = steam->client_opened;
spin_unlock_irqrestore(&steam->lock, flags);
- if (set_lizard_mode)
+ guard(mutex)(&steam->report_mutex);
+ if (!client_opened && lizard_mode)
steam_set_lizard_mode(steam, true);
}
}
@@ -721,14 +719,11 @@ static void steam_input_close(struct input_dev *dev)
static int steam_sensor_open(struct input_dev *dev)
{
struct steam_device *steam = input_get_drvdata(dev);
- unsigned long flags;
- bool client_opened;
- spin_lock_irqsave(&steam->lock, flags);
- client_opened = steam->client_opened;
- spin_unlock_irqrestore(&steam->lock, flags);
- if (client_opened)
- return 0;
+ scoped_guard(spinlock_irqsave, &steam->lock) {
+ if (steam->client_opened)
+ return 0;
+ }
guard(mutex)(&steam->report_mutex);
steam_write_settings(steam, SETTING_IMU_MODE,
@@ -741,14 +736,11 @@ static int steam_sensor_open(struct input_dev *dev)
static void steam_sensor_close(struct input_dev *dev)
{
struct steam_device *steam = input_get_drvdata(dev);
- unsigned long flags;
- bool client_opened;
- spin_lock_irqsave(&steam->lock, flags);
- client_opened = steam->client_opened;
- spin_unlock_irqrestore(&steam->lock, flags);
- if (client_opened)
- return;
+ scoped_guard(spinlock_irqsave, &steam->lock) {
+ if (steam->client_opened)
+ return;
+ }
guard(mutex)(&steam->report_mutex);
steam_write_settings(steam, SETTING_IMU_MODE, 0, 0);
@@ -823,6 +815,7 @@ static int steam_battery_register(struct steam_device *steam)
&steam->battery_desc, &battery_cfg);
if (IS_ERR(battery)) {
ret = PTR_ERR(battery);
+ devm_kfree(&steam->hdev->dev, steam->battery_desc.name);
hid_err(steam->hdev,
"%s:power_supply_register failed with error %d\n",
__func__, ret);
@@ -1082,103 +1075,117 @@ static void steam_battery_unregister(struct steam_device *steam)
RCU_INIT_POINTER(steam->battery, NULL);
synchronize_rcu();
power_supply_unregister(battery);
+ devm_kfree(&steam->hdev->dev, steam->battery_desc.name);
}
static int steam_register(struct steam_device *steam)
{
int ret;
- unsigned long client_opened;
- unsigned long flags;
+ mutex_lock(&steam->registration_mutex);
/*
* This function can be called several times in a row with the
* wireless adaptor, without steam_unregister() between them, because
* another client send a get_connection_status command, for example.
* The battery and serial number are set just once per device.
*/
- if (!steam->serial_no[0]) {
- /*
- * Unlikely, but getting the serial could fail, and it is not so
- * important, so make up a serial number and go on.
- */
- if (steam_get_serial(steam) < 0)
- strscpy(steam->serial_no, "XXXXXXXXXX",
- sizeof(steam->serial_no));
+ if (steam->registered) {
+ mutex_unlock(&steam->registration_mutex);
+ return 0;
+ }
- ret = steam_get_attributes(steam);
- if (ret < 0)
- hid_err(steam->hdev,
- "%s:steam_get_attributes failed with error %d\n",
- __func__, ret);
+ mutex_lock(&steam->report_mutex);
+ /*
+ * Unlikely, but getting the serial could fail, and it is not so
+ * important, so make up a serial number and go on.
+ */
+ if (steam_get_serial(steam) < 0 || !steam->serial_no[0])
+ strscpy(steam->serial_no, "XXXXXXXXXX",
+ sizeof(steam->serial_no));
- hid_info(steam->hdev, "Steam Controller '%s' connected",
- steam->serial_no);
+ ret = steam_get_attributes(steam);
+ if (ret < 0)
+ hid_err(steam->hdev,
+ "%s:steam_get_attributes failed with error %d\n",
+ __func__, ret);
- /* ignore battery errors, we can live without it */
- if (steam->quirks & STEAM_QUIRK_WIRELESS)
- steam_battery_register(steam);
+ hid_info(steam->hdev, "Steam Controller '%s' connected",
+ steam->serial_no);
- mutex_lock(&steam_devices_lock);
- if (list_empty(&steam->list))
- list_add(&steam->list, &steam_devices);
- mutex_unlock(&steam_devices_lock);
- }
+ steam_set_lizard_mode(steam, lizard_mode);
+ mutex_unlock(&steam->report_mutex);
- spin_lock_irqsave(&steam->lock, flags);
- client_opened = steam->client_opened;
- spin_unlock_irqrestore(&steam->lock, flags);
+ /* ignore battery errors, we can live without it */
+ if (steam->quirks & STEAM_QUIRK_WIRELESS)
+ steam_battery_register(steam);
- if (!client_opened) {
- steam_set_lizard_mode(steam, lizard_mode);
- ret = steam_input_register(steam);
- if (ret != 0)
- goto steam_register_input_fail;
- ret = steam_sensors_register(steam);
- if (ret != 0)
- goto steam_register_sensors_fail;
- }
+ ret = steam_input_register(steam);
+ if (ret != 0)
+ goto steam_register_input_fail;
+ ret = steam_sensors_register(steam);
+ if (ret != 0)
+ goto steam_register_sensors_fail;
+
+ steam->registered = true;
+ mutex_unlock(&steam->registration_mutex);
+ mutex_lock(&steam_devices_lock);
+ if (list_empty(&steam->list))
+ list_add(&steam->list, &steam_devices);
+ mutex_unlock(&steam_devices_lock);
return 0;
steam_register_sensors_fail:
steam_input_unregister(steam);
steam_register_input_fail:
+ steam_battery_unregister(steam);
+ mutex_unlock(&steam->registration_mutex);
+ cancel_work_sync(&steam->rumble_work);
+ cancel_delayed_work_sync(&steam->mode_switch);
return ret;
}
static void steam_unregister(struct steam_device *steam)
{
+ if (!steam->registered)
+ return;
+
+ hid_info(steam->hdev, "Steam Controller '%s' disconnected",
+ steam->serial_no);
+ mutex_lock(&steam->registration_mutex);
+ steam->registered = false;
steam_battery_unregister(steam);
steam_sensors_unregister(steam);
steam_input_unregister(steam);
- if (steam->serial_no[0]) {
- hid_info(steam->hdev, "Steam Controller '%s' disconnected",
- steam->serial_no);
- mutex_lock(&steam_devices_lock);
- list_del_init(&steam->list);
- mutex_unlock(&steam_devices_lock);
- steam->serial_no[0] = 0;
- }
+ mutex_unlock(&steam->registration_mutex);
+ cancel_work_sync(&steam->rumble_work);
+ cancel_delayed_work_sync(&steam->mode_switch);
+ mutex_lock(&steam_devices_lock);
+ list_del_init(&steam->list);
+ mutex_unlock(&steam_devices_lock);
+ steam->serial_no[0] = 0;
}
static void steam_work_connect_cb(struct work_struct *work)
{
struct steam_device *steam = container_of(work, struct steam_device,
work_connect);
+
unsigned long flags;
bool connected;
+ bool opened;
int ret;
spin_lock_irqsave(&steam->lock, flags);
+ opened = steam->client_opened;
connected = steam->connected;
spin_unlock_irqrestore(&steam->lock, flags);
- if (connected) {
+ if (connected && !opened) {
ret = steam_register(steam);
- if (ret) {
+ if (ret)
hid_err(steam->hdev,
"%s:steam_register failed with error %d\n",
__func__, ret);
- }
} else {
steam_unregister(steam);
}
@@ -1190,54 +1197,32 @@ static void steam_mode_switch_cb(struct work_struct *work)
struct steam_device, mode_switch);
unsigned long flags;
bool client_opened;
+ bool gamepad_mode;
+
if (!lizard_mode)
return;
+ spin_lock_irqsave(&steam->lock, flags);
steam->gamepad_mode = !steam->gamepad_mode;
- hid_dbg(steam->hdev, "%s: switching gamepad mode to %i\n", __func__, steam->gamepad_mode);
- if (steam->gamepad_mode)
+ gamepad_mode = steam->gamepad_mode;
+ client_opened = steam->client_opened;
+ spin_unlock_irqrestore(&steam->lock, flags);
+
+ guard(mutex)(&steam->report_mutex);
+ hid_dbg(steam->hdev, "%s: switching gamepad mode to %i\n", __func__, gamepad_mode);
+ if (gamepad_mode)
steam_set_lizard_mode(steam, false);
- else {
- spin_lock_irqsave(&steam->lock, flags);
- client_opened = steam->client_opened;
- spin_unlock_irqrestore(&steam->lock, flags);
- if (!client_opened)
- steam_set_lizard_mode(steam, lizard_mode);
- }
+ else if (!client_opened)
+ steam_set_lizard_mode(steam, lizard_mode);
steam_haptic_pulse(steam, STEAM_PAD_RIGHT, 0x190, 0, 1, 0);
- if (steam->gamepad_mode) {
+ if (gamepad_mode) {
steam_haptic_pulse(steam, STEAM_PAD_LEFT, 0x14D, 0x14D, 0x2D, 0);
} else {
steam_haptic_pulse(steam, STEAM_PAD_LEFT, 0x1F4, 0x1F4, 0x1E, 0);
}
}
-static void steam_work_unregister_cb(struct work_struct *work)
-{
- struct steam_device *steam = container_of(work, struct steam_device,
- unregister_work);
- unsigned long flags;
- bool connected;
- bool opened;
-
- spin_lock_irqsave(&steam->lock, flags);
- opened = steam->client_opened;
- connected = steam->connected;
- spin_unlock_irqrestore(&steam->lock, flags);
-
- if (connected) {
- if (opened) {
- steam_sensors_unregister(steam);
- steam_input_unregister(steam);
- } else {
- steam_set_lizard_mode(steam, lizard_mode);
- steam_input_register(steam);
- steam_sensors_register(steam);
- }
- }
-}
-
static bool steam_is_valve_interface(struct hid_device *hdev)
{
struct hid_report_enum *rep_enum;
@@ -1283,7 +1268,7 @@ static int steam_client_ll_open(struct hid_device *hdev)
steam->client_opened++;
spin_unlock_irqrestore(&steam->lock, flags);
- schedule_work(&steam->unregister_work);
+ schedule_work(&steam->work_connect);
return 0;
}
@@ -1298,7 +1283,7 @@ static void steam_client_ll_close(struct hid_device *hdev)
steam->client_opened--;
spin_unlock_irqrestore(&steam->lock, flags);
- schedule_work(&steam->unregister_work);
+ schedule_work(&steam->work_connect);
}
static int steam_client_ll_raw_request(struct hid_device *hdev,
@@ -1384,6 +1369,7 @@ static int steam_probe(struct hid_device *hdev,
hid_set_drvdata(hdev, steam);
spin_lock_init(&steam->lock);
mutex_init(&steam->report_mutex);
+ mutex_init(&steam->registration_mutex);
steam->quirks = id->driver_data;
INIT_WORK(&steam->work_connect, steam_work_connect_cb);
INIT_DELAYED_WORK(&steam->mode_switch, steam_mode_switch_cb);
@@ -1394,7 +1380,6 @@ static int steam_probe(struct hid_device *hdev,
steam->sensor_update_rate_us = 4000;
else
steam->sensor_update_rate_us = 9000;
- INIT_WORK(&steam->unregister_work, steam_work_unregister_cb);
/*
* With the real steam controller interface, do not connect hidraw.
@@ -1455,7 +1440,6 @@ static int steam_probe(struct hid_device *hdev,
cancel_work_sync(&steam->work_connect);
cancel_delayed_work_sync(&steam->mode_switch);
cancel_work_sync(&steam->rumble_work);
- cancel_work_sync(&steam->unregister_work);
return ret;
}
@@ -1463,36 +1447,32 @@ static int steam_probe(struct hid_device *hdev,
static void steam_remove(struct hid_device *hdev)
{
struct steam_device *steam = hid_get_drvdata(hdev);
+ unsigned long flags;
if (!steam || hdev->group == HID_GROUP_STEAM) {
hid_hw_stop(hdev);
return;
}
+ hid_hw_close(hdev);
hid_destroy_device(steam->client_hdev);
- cancel_delayed_work_sync(&steam->mode_switch);
- cancel_work_sync(&steam->work_connect);
- cancel_work_sync(&steam->rumble_work);
- cancel_work_sync(&steam->unregister_work);
- steam->client_hdev = NULL;
+ spin_lock_irqsave(&steam->lock, flags);
steam->client_opened = 0;
+ spin_unlock_irqrestore(&steam->lock, flags);
+ cancel_work_sync(&steam->work_connect);
if (steam->quirks & STEAM_QUIRK_WIRELESS) {
hid_info(hdev, "Steam wireless receiver disconnected");
}
- hid_hw_close(hdev);
- hid_hw_stop(hdev);
steam_unregister(steam);
+ hid_hw_stop(hdev);
}
static void steam_do_connect_event(struct steam_device *steam, bool connected)
{
- unsigned long flags;
bool changed;
- spin_lock_irqsave(&steam->lock, flags);
changed = steam->connected != connected;
steam->connected = connected;
- spin_unlock_irqrestore(&steam->lock, flags);
if (changed && schedule_work(&steam->work_connect) == 0)
dbg_hid("%s: connected=%d event already queued\n",
@@ -1921,8 +1901,6 @@ static void steam_do_deck_sensors_event(struct steam_device *steam,
static void steam_do_battery_event(struct steam_device *steam,
struct power_supply *battery, u8 *data)
{
- unsigned long flags;
-
s16 volts = steam_le16(data + 12);
u8 batt = data[14];
@@ -1930,10 +1908,8 @@ static void steam_do_battery_event(struct steam_device *steam,
rcu_read_lock();
battery = rcu_dereference(steam->battery);
if (likely(battery)) {
- spin_lock_irqsave(&steam->lock, flags);
steam->voltage = volts;
steam->battery_charge = batt;
- spin_unlock_irqrestore(&steam->lock, flags);
power_supply_changed(battery);
}
rcu_read_unlock();
@@ -1951,6 +1927,7 @@ static int steam_raw_event(struct hid_device *hdev,
if (!steam)
return 0;
+ guard(spinlock_irqsave)(&steam->lock);
if (steam->client_opened)
hid_input_report(steam->client_hdev, HID_FEATURE_REPORT,
data, size, 0);
@@ -2038,6 +2015,8 @@ static int steam_param_set_lizard_mode(const char *val,
{
struct steam_device *steam;
int ret;
+ bool client_opened;
+ unsigned long flags;
ret = param_set_bool(val, kp);
if (ret)
@@ -2045,8 +2024,13 @@ static int steam_param_set_lizard_mode(const char *val,
mutex_lock(&steam_devices_lock);
list_for_each_entry(steam, &steam_devices, list) {
- if (!steam->client_opened)
+ spin_lock_irqsave(&steam->lock, flags);
+ client_opened = steam->client_opened;
+ spin_unlock_irqrestore(&steam->lock, flags);
+ if (!client_opened) {
+ guard(mutex)(&steam->report_mutex);
steam_set_lizard_mode(steam, lizard_mode);
+ }
}
mutex_unlock(&steam_devices_lock);
return 0;
diff --git a/drivers/hv/Makefile b/drivers/hv/Makefile
index 1a1677bf4dac..344ddaf37710 100644
--- a/drivers/hv/Makefile
+++ b/drivers/hv/Makefile
@@ -13,7 +13,7 @@ hv_vmbus-y := vmbus_drv.o \
hv_vmbus-$(CONFIG_HYPERV_TESTING) += hv_debugfs.o
hv_utils-y := hv_util.o hv_kvp.o hv_snapshot.o hv_utils_transport.o
mshv_root-y := mshv_root_main.o mshv_synic.o mshv_eventfd.o mshv_irq.o \
- mshv_root_hv_call.o mshv_portid_table.o
+ mshv_root_hv_call.o mshv_portid_table.o mshv_regions.o
# Code that must be built-in
obj-$(CONFIG_HYPERV) += hv_common.o
diff --git a/drivers/hv/mshv_regions.c b/drivers/hv/mshv_regions.c
new file mode 100644
index 000000000000..5372b7ec1871
--- /dev/null
+++ b/drivers/hv/mshv_regions.c
@@ -0,0 +1,177 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2025, Microsoft Corporation.
+ *
+ * Memory region management for mshv_root module.
+ *
+ * Authors: Microsoft Linux virtualization team
+ */
+
+#include <linux/mm.h>
+#include <linux/vmalloc.h>
+
+#include <asm/mshyperv.h>
+
+#include "mshv_root.h"
+
+struct mshv_mem_region *mshv_region_create(u64 guest_pfn, u64 nr_pages,
+ u64 uaddr, u32 flags,
+ bool is_mmio)
+{
+ struct mshv_mem_region *region;
+
+ region = vzalloc(sizeof(*region) + sizeof(struct page *) * nr_pages);
+ if (!region)
+ return ERR_PTR(-ENOMEM);
+
+ region->nr_pages = nr_pages;
+ region->start_gfn = guest_pfn;
+ region->start_uaddr = uaddr;
+ region->hv_map_flags = HV_MAP_GPA_READABLE | HV_MAP_GPA_ADJUSTABLE;
+ if (flags & BIT(MSHV_SET_MEM_BIT_WRITABLE))
+ region->hv_map_flags |= HV_MAP_GPA_WRITABLE;
+ if (flags & BIT(MSHV_SET_MEM_BIT_EXECUTABLE))
+ region->hv_map_flags |= HV_MAP_GPA_EXECUTABLE;
+
+ /* Note: large_pages flag populated when we pin the pages */
+ if (!is_mmio)
+ region->flags.range_pinned = true;
+
+ return region;
+}
+
+int mshv_region_share(struct mshv_mem_region *region)
+{
+ u32 flags = HV_MODIFY_SPA_PAGE_HOST_ACCESS_MAKE_SHARED;
+
+ if (region->flags.large_pages)
+ flags |= HV_MODIFY_SPA_PAGE_HOST_ACCESS_LARGE_PAGE;
+
+ return hv_call_modify_spa_host_access(region->partition->pt_id,
+ region->pages, region->nr_pages,
+ HV_MAP_GPA_READABLE | HV_MAP_GPA_WRITABLE,
+ flags, true);
+}
+
+int mshv_region_unshare(struct mshv_mem_region *region)
+{
+ u32 flags = HV_MODIFY_SPA_PAGE_HOST_ACCESS_MAKE_EXCLUSIVE;
+
+ if (region->flags.large_pages)
+ flags |= HV_MODIFY_SPA_PAGE_HOST_ACCESS_LARGE_PAGE;
+
+ return hv_call_modify_spa_host_access(region->partition->pt_id,
+ region->pages, region->nr_pages,
+ 0,
+ flags, false);
+}
+
+static int mshv_region_remap_pages(struct mshv_mem_region *region,
+ u32 map_flags,
+ u64 page_offset, u64 page_count)
+{
+ if (page_offset + page_count > region->nr_pages)
+ return -EINVAL;
+
+ if (region->flags.large_pages)
+ map_flags |= HV_MAP_GPA_LARGE_PAGE;
+
+ return hv_call_map_gpa_pages(region->partition->pt_id,
+ region->start_gfn + page_offset,
+ page_count, map_flags,
+ region->pages + page_offset);
+}
+
+int mshv_region_map(struct mshv_mem_region *region)
+{
+ u32 map_flags = region->hv_map_flags;
+
+ return mshv_region_remap_pages(region, map_flags,
+ 0, region->nr_pages);
+}
+
+static void mshv_region_invalidate_pages(struct mshv_mem_region *region,
+ u64 page_offset, u64 page_count)
+{
+ if (region->flags.range_pinned)
+ unpin_user_pages(region->pages + page_offset, page_count);
+
+ memset(region->pages + page_offset, 0,
+ page_count * sizeof(struct page *));
+}
+
+void mshv_region_invalidate(struct mshv_mem_region *region)
+{
+ mshv_region_invalidate_pages(region, 0, region->nr_pages);
+}
+
+int mshv_region_pin(struct mshv_mem_region *region)
+{
+ u64 done_count, nr_pages;
+ struct page **pages;
+ __u64 userspace_addr;
+ int ret;
+
+ for (done_count = 0; done_count < region->nr_pages; done_count += ret) {
+ pages = region->pages + done_count;
+ userspace_addr = region->start_uaddr +
+ done_count * HV_HYP_PAGE_SIZE;
+ nr_pages = min(region->nr_pages - done_count,
+ MSHV_PIN_PAGES_BATCH_SIZE);
+
+ /*
+ * Pinning assuming 4k pages works for large pages too.
+ * All page structs within the large page are returned.
+ *
+ * Pin requests are batched because pin_user_pages_fast
+ * with the FOLL_LONGTERM flag does a large temporary
+ * allocation of contiguous memory.
+ */
+ ret = pin_user_pages_fast(userspace_addr, nr_pages,
+ FOLL_WRITE | FOLL_LONGTERM,
+ pages);
+ if (ret != nr_pages)
+ goto release_pages;
+ }
+
+ if (PageHuge(region->pages[0]))
+ region->flags.large_pages = true;
+
+ return 0;
+
+release_pages:
+ if (ret > 0)
+ done_count += ret;
+ mshv_region_invalidate_pages(region, 0, done_count);
+ return ret < 0 ? ret : -ENOMEM;
+}
+
+void mshv_region_destroy(struct mshv_mem_region *region)
+{
+ struct mshv_partition *partition = region->partition;
+ u32 unmap_flags = 0;
+ int ret;
+
+ hlist_del(®ion->hnode);
+
+ if (mshv_partition_encrypted(partition)) {
+ ret = mshv_region_share(region);
+ if (ret) {
+ pt_err(partition,
+ "Failed to regain access to memory, unpinning user pages will fail and crash the host error: %d\n",
+ ret);
+ return;
+ }
+ }
+
+ if (region->flags.large_pages)
+ unmap_flags |= HV_UNMAP_GPA_LARGE_PAGE;
+
+ /* ignore unmap failures and continue as process may be exiting */
+ hv_call_unmap_gpa_pages(partition->pt_id, region->start_gfn,
+ region->nr_pages, unmap_flags);
+
+ mshv_region_invalidate(region);
+
+ vfree(region);
+}
diff --git a/drivers/hv/mshv_root.h b/drivers/hv/mshv_root.h
index db6b42db2fdc..e1d38deaae5b 100644
--- a/drivers/hv/mshv_root.h
+++ b/drivers/hv/mshv_root.h
@@ -308,4 +308,14 @@ extern struct mshv_root mshv_root;
extern enum hv_scheduler_type hv_scheduler_type;
extern u8 * __percpu *hv_synic_eventring_tail;
+struct mshv_mem_region *mshv_region_create(u64 guest_pfn, u64 nr_pages,
+ u64 uaddr, u32 flags,
+ bool is_mmio);
+int mshv_region_share(struct mshv_mem_region *region);
+int mshv_region_unshare(struct mshv_mem_region *region);
+int mshv_region_map(struct mshv_mem_region *region);
+void mshv_region_invalidate(struct mshv_mem_region *region);
+int mshv_region_pin(struct mshv_mem_region *region);
+void mshv_region_destroy(struct mshv_mem_region *region);
+
#endif /* _MSHV_ROOT_H_ */
diff --git a/drivers/hv/mshv_root_main.c b/drivers/hv/mshv_root_main.c
index 3a356b34774b..6c62db222210 100644
--- a/drivers/hv/mshv_root_main.c
+++ b/drivers/hv/mshv_root_main.c
@@ -1039,133 +1039,6 @@ static void mshv_async_hvcall_handler(void *data, u64 *status)
*status = partition->async_hypercall_status;
}
-static int
-mshv_partition_region_share(struct mshv_mem_region *region)
-{
- u32 flags = HV_MODIFY_SPA_PAGE_HOST_ACCESS_MAKE_SHARED;
-
- if (region->flags.large_pages)
- flags |= HV_MODIFY_SPA_PAGE_HOST_ACCESS_LARGE_PAGE;
-
- return hv_call_modify_spa_host_access(region->partition->pt_id,
- region->pages, region->nr_pages,
- HV_MAP_GPA_READABLE | HV_MAP_GPA_WRITABLE,
- flags, true);
-}
-
-static int
-mshv_partition_region_unshare(struct mshv_mem_region *region)
-{
- u32 flags = HV_MODIFY_SPA_PAGE_HOST_ACCESS_MAKE_EXCLUSIVE;
-
- if (region->flags.large_pages)
- flags |= HV_MODIFY_SPA_PAGE_HOST_ACCESS_LARGE_PAGE;
-
- return hv_call_modify_spa_host_access(region->partition->pt_id,
- region->pages, region->nr_pages,
- 0,
- flags, false);
-}
-
-static int
-mshv_region_remap_pages(struct mshv_mem_region *region, u32 map_flags,
- u64 page_offset, u64 page_count)
-{
- if (page_offset + page_count > region->nr_pages)
- return -EINVAL;
-
- if (region->flags.large_pages)
- map_flags |= HV_MAP_GPA_LARGE_PAGE;
-
- /* ask the hypervisor to map guest ram */
- return hv_call_map_gpa_pages(region->partition->pt_id,
- region->start_gfn + page_offset,
- page_count, map_flags,
- region->pages + page_offset);
-}
-
-static int
-mshv_region_map(struct mshv_mem_region *region)
-{
- u32 map_flags = region->hv_map_flags;
-
- return mshv_region_remap_pages(region, map_flags,
- 0, region->nr_pages);
-}
-
-static void
-mshv_region_evict_pages(struct mshv_mem_region *region,
- u64 page_offset, u64 page_count)
-{
- if (region->flags.range_pinned)
- unpin_user_pages(region->pages + page_offset, page_count);
-
- memset(region->pages + page_offset, 0,
- page_count * sizeof(struct page *));
-}
-
-static void
-mshv_region_evict(struct mshv_mem_region *region)
-{
- mshv_region_evict_pages(region, 0, region->nr_pages);
-}
-
-static int
-mshv_region_populate_pages(struct mshv_mem_region *region,
- u64 page_offset, u64 page_count)
-{
- u64 done_count, nr_pages;
- struct page **pages;
- __u64 userspace_addr;
- int ret;
-
- if (page_offset + page_count > region->nr_pages)
- return -EINVAL;
-
- for (done_count = 0; done_count < page_count; done_count += ret) {
- pages = region->pages + page_offset + done_count;
- userspace_addr = region->start_uaddr +
- (page_offset + done_count) *
- HV_HYP_PAGE_SIZE;
- nr_pages = min(page_count - done_count,
- MSHV_PIN_PAGES_BATCH_SIZE);
-
- /*
- * Pinning assuming 4k pages works for large pages too.
- * All page structs within the large page are returned.
- *
- * Pin requests are batched because pin_user_pages_fast
- * with the FOLL_LONGTERM flag does a large temporary
- * allocation of contiguous memory.
- */
- if (region->flags.range_pinned)
- ret = pin_user_pages_fast(userspace_addr,
- nr_pages,
- FOLL_WRITE | FOLL_LONGTERM,
- pages);
- else
- ret = -EOPNOTSUPP;
-
- if (ret < 0)
- goto release_pages;
- }
-
- if (PageHuge(region->pages[page_offset]))
- region->flags.large_pages = true;
-
- return 0;
-
-release_pages:
- mshv_region_evict_pages(region, page_offset, done_count);
- return ret;
-}
-
-static int
-mshv_region_populate(struct mshv_mem_region *region)
-{
- return mshv_region_populate_pages(region, 0, region->nr_pages);
-}
-
static struct mshv_mem_region *
mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
{
@@ -1189,7 +1062,7 @@ static int mshv_partition_create_region(struct mshv_partition *partition,
struct mshv_mem_region **regionpp,
bool is_mmio)
{
- struct mshv_mem_region *region, *rg;
+ struct mshv_mem_region *rg;
u64 nr_pages = HVPFN_DOWN(mem->size);
/* Reject overlapping regions */
@@ -1205,43 +1078,40 @@ static int mshv_partition_create_region(struct mshv_partition *partition,
return -EEXIST;
}
- region = vzalloc(sizeof(*region) + sizeof(struct page *) * nr_pages);
- if (!region)
- return -ENOMEM;
-
- region->nr_pages = nr_pages;
- region->start_gfn = mem->guest_pfn;
- region->start_uaddr = mem->userspace_addr;
- region->hv_map_flags = HV_MAP_GPA_READABLE | HV_MAP_GPA_ADJUSTABLE;
- if (mem->flags & BIT(MSHV_SET_MEM_BIT_WRITABLE))
- region->hv_map_flags |= HV_MAP_GPA_WRITABLE;
- if (mem->flags & BIT(MSHV_SET_MEM_BIT_EXECUTABLE))
- region->hv_map_flags |= HV_MAP_GPA_EXECUTABLE;
+ rg = mshv_region_create(mem->guest_pfn, nr_pages,
+ mem->userspace_addr, mem->flags,
+ is_mmio);
+ if (IS_ERR(rg))
+ return PTR_ERR(rg);
- /* Note: large_pages flag populated when we pin the pages */
- if (!is_mmio)
- region->flags.range_pinned = true;
+ rg->partition = partition;
- region->partition = partition;
-
- *regionpp = region;
+ *regionpp = rg;
return 0;
}
-/*
- * Map guest ram. if snp, make sure to release that from the host first
- * Side Effects: In case of failure, pages are unpinned when feasible.
+/**
+ * mshv_prepare_pinned_region - Pin and map memory regions
+ * @region: Pointer to the memory region structure
+ *
+ * This function processes memory regions that are explicitly marked as pinned.
+ * Pinned regions are preallocated, mapped upfront, and do not rely on fault-based
+ * population. The function ensures the region is properly populated, handles
+ * encryption requirements for SNP partitions if applicable, maps the region,
+ * and performs necessary sharing or eviction operations based on the mapping
+ * result.
+ *
+ * Return: 0 on success, negative error code on failure.
*/
-static int
-mshv_partition_mem_region_map(struct mshv_mem_region *region)
+static int mshv_prepare_pinned_region(struct mshv_mem_region *region)
{
struct mshv_partition *partition = region->partition;
int ret;
- ret = mshv_region_populate(region);
+ ret = mshv_region_pin(region);
if (ret) {
- pt_err(partition, "Failed to populate memory region: %d\n",
+ pt_err(partition, "Failed to pin memory region: %d\n",
ret);
goto err_out;
}
@@ -1254,12 +1124,12 @@ mshv_partition_mem_region_map(struct mshv_mem_region *region)
* access to guest memory regions.
*/
if (mshv_partition_encrypted(partition)) {
- ret = mshv_partition_region_unshare(region);
+ ret = mshv_region_unshare(region);
if (ret) {
pt_err(partition,
"Failed to unshare memory region (guest_pfn: %llu): %d\n",
region->start_gfn, ret);
- goto evict_region;
+ goto invalidate_region;
}
}
@@ -1267,9 +1137,9 @@ mshv_partition_mem_region_map(struct mshv_mem_region *region)
if (ret && mshv_partition_encrypted(partition)) {
int shrc;
- shrc = mshv_partition_region_share(region);
+ shrc = mshv_region_share(region);
if (!shrc)
- goto evict_region;
+ goto invalidate_region;
pt_err(partition,
"Failed to share memory region (guest_pfn: %llu): %d\n",
@@ -1283,8 +1153,8 @@ mshv_partition_mem_region_map(struct mshv_mem_region *region)
return 0;
-evict_region:
- mshv_region_evict(region);
+invalidate_region:
+ mshv_region_invalidate(region);
err_out:
return ret;
}
@@ -1333,7 +1203,7 @@ mshv_map_user_memory(struct mshv_partition *partition,
ret = hv_call_map_mmio_pages(partition->pt_id, mem.guest_pfn,
mmio_pfn, HVPFN_DOWN(mem.size));
else
- ret = mshv_partition_mem_region_map(region);
+ ret = mshv_prepare_pinned_region(region);
if (ret)
goto errout;
@@ -1354,7 +1224,6 @@ mshv_unmap_user_memory(struct mshv_partition *partition,
struct mshv_user_mem_region mem)
{
struct mshv_mem_region *region;
- u32 unmap_flags = 0;
if (!(mem.flags & BIT(MSHV_SET_MEM_BIT_UNMAP)))
return -EINVAL;
@@ -1369,18 +1238,8 @@ mshv_unmap_user_memory(struct mshv_partition *partition,
region->nr_pages != HVPFN_DOWN(mem.size))
return -EINVAL;
- hlist_del(®ion->hnode);
-
- if (region->flags.large_pages)
- unmap_flags |= HV_UNMAP_GPA_LARGE_PAGE;
+ mshv_region_destroy(region);
- /* ignore unmap failures and continue as process may be exiting */
- hv_call_unmap_gpa_pages(partition->pt_id, region->start_gfn,
- region->nr_pages, unmap_flags);
-
- mshv_region_evict(region);
-
- vfree(region);
return 0;
}
@@ -1716,8 +1575,8 @@ static void destroy_partition(struct mshv_partition *partition)
{
struct mshv_vp *vp;
struct mshv_mem_region *region;
- int i, ret;
struct hlist_node *n;
+ int i;
if (refcount_read(&partition->pt_ref_count)) {
pt_err(partition,
@@ -1777,25 +1636,9 @@ static void destroy_partition(struct mshv_partition *partition)
remove_partition(partition);
- /* Remove regions, regain access to the memory and unpin the pages */
hlist_for_each_entry_safe(region, n, &partition->pt_mem_regions,
- hnode) {
- hlist_del(®ion->hnode);
-
- if (mshv_partition_encrypted(partition)) {
- ret = mshv_partition_region_share(region);
- if (ret) {
- pt_err(partition,
- "Failed to regain access to memory, unpinning user pages will fail and crash the host error: %d\n",
- ret);
- return;
- }
- }
-
- mshv_region_evict(region);
-
- vfree(region);
- }
+ hnode)
+ mshv_region_destroy(region);
/* Withdraw and free all pages we deposited */
hv_call_withdraw_memory(U64_MAX, NUMA_NO_NODE, partition->pt_id);
diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c
index 522e12817446..be05ef41fd73 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
@@ -752,8 +752,7 @@ static int etm4_parse_event_config(struct coresight_device *csdev,
struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
struct etmv4_config *config = &drvdata->config;
struct perf_event_attr *attr = &event->attr;
- unsigned long cfg_hash;
- int preset, cc_threshold;
+ int cc_threshold;
/* Clear configuration from previous run */
memset(config, 0, sizeof(struct etmv4_config));
@@ -834,20 +833,6 @@ static int etm4_parse_event_config(struct coresight_device *csdev,
/* bit[12], Return stack enable bit */
config->cfg |= TRCCONFIGR_RS;
- /*
- * Set any selected configuration and preset.
- *
- * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset)
- * in the perf attributes defined in coresight-etm-perf.c.
- * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config.
- * A zero configid means no configuration active, preset = 0 means no preset selected.
- */
- if (attr->config2 & GENMASK_ULL(63, 32)) {
- cfg_hash = (u32)(attr->config2 >> 32);
- preset = attr->config & 0xF;
- ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset);
- }
-
/* branch broadcast - enable if selected and supported */
if (attr->config & BIT(ETM_OPT_BRANCH_BROADCAST)) {
if (!drvdata->trcbb) {
@@ -871,7 +856,9 @@ static int etm4_enable_perf(struct coresight_device *csdev,
struct coresight_path *path)
{
struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
- int ret;
+ struct perf_event_attr *attr = &event->attr;
+ unsigned long cfg_hash;
+ int ret, preset;
if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id()))
return -EINVAL;
@@ -882,7 +869,23 @@ static int etm4_enable_perf(struct coresight_device *csdev,
/* Configure the tracer based on the session's specifics */
ret = etm4_parse_event_config(csdev, event);
if (ret)
- goto out;
+ goto err;
+
+ /*
+ * Set any selected configuration and preset.
+ *
+ * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset)
+ * in the perf attributes defined in coresight-etm-perf.c.
+ * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config.
+ * A zero configid means no configuration active, preset = 0 means no preset selected.
+ */
+ cfg_hash = (u32)(attr->config2 >> 32);
+ if (cfg_hash) {
+ preset = attr->config & 0xF;
+ ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset);
+ if (ret)
+ goto err;
+ }
drvdata->trcid = path->trace_id;
@@ -891,11 +894,17 @@ static int etm4_enable_perf(struct coresight_device *csdev,
/* And enable it */
ret = etm4_enable_hw(drvdata);
+ if (ret) {
+ if (cfg_hash)
+ cscfg_csdev_disable_active_config(csdev);
+ goto err;
+ }
-out:
+ return 0;
+
+err:
/* Failed to start tracer; roll back to DISABLED mode */
- if (ret)
- coresight_set_mode(csdev, CS_MODE_DISABLED);
+ coresight_set_mode(csdev, CS_MODE_DISABLED);
return ret;
}
diff --git a/drivers/i3c/master/mipi-i3c-hci/core.c b/drivers/i3c/master/mipi-i3c-hci/core.c
index 1eeab337e5e5..e9fd314eb17b 100644
--- a/drivers/i3c/master/mipi-i3c-hci/core.c
+++ b/drivers/i3c/master/mipi-i3c-hci/core.c
@@ -21,6 +21,7 @@
#include "ext_caps.h"
#include "cmd.h"
#include "dat.h"
+#include "ibi.h"
/*
@@ -119,6 +120,7 @@ static inline struct i3c_hci *to_i3c_hci(struct i3c_master_controller *m)
static int i3c_hci_bus_init(struct i3c_master_controller *m)
{
struct i3c_hci *hci = to_i3c_hci(m);
+ struct device *dev = hci->master.dev.parent;
struct i3c_device_info info;
int ret;
@@ -139,6 +141,10 @@ static int i3c_hci_bus_init(struct i3c_master_controller *m)
if (ret)
return ret;
+ hci->ibi_devs = devm_kcalloc(dev, hci->DAT_entries, sizeof(*hci->ibi_devs), GFP_KERNEL);
+ if (!hci->ibi_devs)
+ return -ENOMEM;
+
ret = hci->io->init(hci);
if (ret)
return ret;
@@ -477,14 +483,40 @@ static int i3c_hci_request_ibi(struct i3c_dev_desc *dev,
return hci->io->request_ibi(hci, dev, req);
}
+static void __i3c_hci_disable_ibi(struct i3c_hci *hci, struct i3c_dev_desc *dev)
+{
+ struct i3c_hci_dev_data *dev_data = i3c_dev_get_master_data(dev);
+
+ mipi_i3c_hci_dat_v1.set_flags(hci, dev_data->dat_idx, DAT_0_SIR_REJECT, 0);
+ scoped_guard(spinlock_irqsave, &hci->lock)
+ hci->ibi_devs[dev_data->dat_idx] = NULL;
+}
+
static void i3c_hci_free_ibi(struct i3c_dev_desc *dev)
{
struct i3c_master_controller *m = i3c_dev_get_master(dev);
struct i3c_hci *hci = to_i3c_hci(m);
+ /* Must ensure the IBI has been disabled */
+ __i3c_hci_disable_ibi(hci, dev);
hci->io->free_ibi(hci, dev);
}
+struct i3c_dev_desc *i3c_hci_addr_to_dev(struct i3c_hci *hci, unsigned int addr)
+{
+ int dat_idx;
+
+ lockdep_assert_held(&hci->lock);
+
+ for (dat_idx = 0; dat_idx < hci->DAT_entries; dat_idx++) {
+ struct i3c_dev_desc *dev = hci->ibi_devs[dat_idx];
+
+ if (dev && dev->info.dyn_addr == addr)
+ return dev;
+ }
+ return NULL;
+}
+
static int i3c_hci_enable_ibi(struct i3c_dev_desc *dev)
{
struct i3c_master_controller *m = i3c_dev_get_master(dev);
@@ -492,6 +524,8 @@ static int i3c_hci_enable_ibi(struct i3c_dev_desc *dev)
struct i3c_hci_dev_data *dev_data = i3c_dev_get_master_data(dev);
mipi_i3c_hci_dat_v1.clear_flags(hci, dev_data->dat_idx, DAT_0_SIR_REJECT, 0);
+ scoped_guard(spinlock_irqsave, &hci->lock)
+ hci->ibi_devs[dev_data->dat_idx] = dev;
return i3c_master_enec_locked(m, dev->info.dyn_addr, I3C_CCC_EVENT_SIR);
}
@@ -499,9 +533,8 @@ static int i3c_hci_disable_ibi(struct i3c_dev_desc *dev)
{
struct i3c_master_controller *m = i3c_dev_get_master(dev);
struct i3c_hci *hci = to_i3c_hci(m);
- struct i3c_hci_dev_data *dev_data = i3c_dev_get_master_data(dev);
- mipi_i3c_hci_dat_v1.set_flags(hci, dev_data->dat_idx, DAT_0_SIR_REJECT, 0);
+ __i3c_hci_disable_ibi(hci, dev);
return i3c_master_disec_locked(m, dev->info.dyn_addr, I3C_CCC_EVENT_SIR);
}
diff --git a/drivers/i3c/master/mipi-i3c-hci/dma.c b/drivers/i3c/master/mipi-i3c-hci/dma.c
index 34c21a56a25d..37717e530a25 100644
--- a/drivers/i3c/master/mipi-i3c-hci/dma.c
+++ b/drivers/i3c/master/mipi-i3c-hci/dma.c
@@ -710,10 +710,15 @@ static void hci_dma_process_ibi(struct i3c_hci *hci, struct hci_rh_data *rh)
}
/* determine who this is for */
+ spin_lock(&hci->lock);
dev = i3c_hci_addr_to_dev(hci, ibi_addr);
if (!dev) {
- dev_err(&hci->master.dev,
- "IBI for unknown device %#x\n", ibi_addr);
+ /*
+ * Either an IBI received just before IBI's were disabled, or
+ * the controller is broken. Assume the former.
+ */
+ dev_dbg(&hci->master.dev, "IBI when not enabled at address %#x\n", ibi_addr);
+ spin_unlock(&hci->lock);
goto done;
}
@@ -722,6 +727,7 @@ static void hci_dma_process_ibi(struct i3c_hci *hci, struct hci_rh_data *rh)
if (ibi_size > dev_ibi->max_len) {
dev_err(&hci->master.dev, "IBI payload too big (%d > %d)\n",
ibi_size, dev_ibi->max_len);
+ spin_unlock(&hci->lock);
goto done;
}
@@ -738,6 +744,7 @@ static void hci_dma_process_ibi(struct i3c_hci *hci, struct hci_rh_data *rh)
slot = i3c_generic_ibi_get_free_slot(dev_ibi->pool);
if (!slot) {
dev_err(&hci->master.dev, "no free slot for IBI\n");
+ spin_unlock(&hci->lock);
goto done;
}
@@ -768,6 +775,7 @@ static void hci_dma_process_ibi(struct i3c_hci *hci, struct hci_rh_data *rh)
slot->dev = dev;
slot->len = ibi_size;
i3c_master_queue_ibi(dev, slot);
+ spin_unlock(&hci->lock);
done:
/* take care to update the ibi dequeue pointer atomically */
diff --git a/drivers/i3c/master/mipi-i3c-hci/hci.h b/drivers/i3c/master/mipi-i3c-hci/hci.h
index 5502ab0a3535..dad6b71bac35 100644
--- a/drivers/i3c/master/mipi-i3c-hci/hci.h
+++ b/drivers/i3c/master/mipi-i3c-hci/hci.h
@@ -54,6 +54,7 @@ struct i3c_hci {
unsigned int DAT_entries;
unsigned int DAT_entry_size;
void *DAT_data;
+ struct i3c_dev_desc **ibi_devs;
unsigned int DCT_entries;
unsigned int DCT_entry_size;
u8 version_major;
diff --git a/drivers/i3c/master/mipi-i3c-hci/ibi.h b/drivers/i3c/master/mipi-i3c-hci/ibi.h
index e1f98e264da0..073ca67b7d04 100644
--- a/drivers/i3c/master/mipi-i3c-hci/ibi.h
+++ b/drivers/i3c/master/mipi-i3c-hci/ibi.h
@@ -26,17 +26,6 @@
#define IBI_DATA_LENGTH GENMASK(7, 0)
/* handy helpers */
-static inline struct i3c_dev_desc *
-i3c_hci_addr_to_dev(struct i3c_hci *hci, unsigned int addr)
-{
- struct i3c_bus *bus = i3c_master_get_bus(&hci->master);
- struct i3c_dev_desc *dev;
-
- i3c_bus_for_each_i3cdev(bus, dev) {
- if (dev->info.dyn_addr == addr)
- return dev;
- }
- return NULL;
-}
+struct i3c_dev_desc *i3c_hci_addr_to_dev(struct i3c_hci *hci, unsigned int addr);
#endif
diff --git a/drivers/i3c/master/mipi-i3c-hci/pio.c b/drivers/i3c/master/mipi-i3c-hci/pio.c
index 35e6acc958a8..3a89f1628f6d 100644
--- a/drivers/i3c/master/mipi-i3c-hci/pio.c
+++ b/drivers/i3c/master/mipi-i3c-hci/pio.c
@@ -839,8 +839,11 @@ static bool hci_pio_prep_new_ibi(struct i3c_hci *hci, struct hci_pio_data *pio)
dev = i3c_hci_addr_to_dev(hci, ibi->addr);
if (!dev) {
- dev_err(&hci->master.dev,
- "IBI for unknown device %#x\n", ibi->addr);
+ /*
+ * Either an IBI received just before IBI's were disabled, or
+ * the controller is broken. Assume the former.
+ */
+ dev_dbg(&hci->master.dev, "IBI when not enabled at address %#x\n", ibi->addr);
return true;
}
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 64b6e572e704..fcd57d34ee07 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -1473,6 +1473,9 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd,
}
qplqp->type = (u8)qptype;
qplqp->wqe_mode = bnxt_re_is_var_size_supported(rdev, uctx);
+ if (uctx && qplqp->wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE &&
+ (!ureq->sq_slots || ureq->sq_slots > BNXT_RE_MAX_SQ_SLOTS))
+ return -EINVAL;
if (init_attr->qp_type == IB_QPT_RC) {
qplqp->max_rd_atomic = dev_attr->max_qp_rd_atom;
qplqp->max_dest_rd_atomic = dev_attr->max_qp_init_rd_atom;
@@ -3102,11 +3105,11 @@ int bnxt_re_destroy_cq(struct ib_cq *ib_cq, struct ib_udata *udata)
nq = cq->qplib_cq.nq;
cctx = rdev->chip_ctx;
- if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT) {
- free_page((unsigned long)cq->uctx_cq_page);
+ if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
hash_del(&cq->hash_entry);
- }
bnxt_qplib_destroy_cq(&rdev->qplib_res, &cq->qplib_cq);
+ if (cctx->modes.toggle_bits & BNXT_QPLIB_CQ_TOGGLE_BIT)
+ free_page((unsigned long)cq->uctx_cq_page);
bnxt_re_put_nq(rdev, nq);
ib_umem_release(cq->umem);
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
index 5a45c55c6464..7e332aca5016 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
@@ -367,6 +367,7 @@ int bnxt_qplib_destroy_flow(struct bnxt_qplib_res *res);
#define BNXT_VAR_MAX_SLOT_ALIGN 256
#define BNXT_VAR_MAX_SGE 13
#define BNXT_RE_MAX_RQ_WQES 65536
+#define BNXT_RE_MAX_SQ_SLOTS 65536
#define BNXT_STATIC_MAX_SGE 6
diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c
index ac349de9d2e4..9f0b85606937 100644
--- a/drivers/infiniband/hw/cxgb4/mem.c
+++ b/drivers/infiniband/hw/cxgb4/mem.c
@@ -199,7 +199,8 @@ static int _c4iw_write_mem_dma(struct c4iw_rdev *rdev, u32 addr, u32 len,
daddr = dma_map_single(&rdev->lldi.pdev->dev, data, len, DMA_TO_DEVICE);
if (dma_mapping_error(&rdev->lldi.pdev->dev, daddr))
- return -1;
+ return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
+ wr_waitp);
save = daddr;
while (remain > inline_threshold) {
@@ -235,30 +236,12 @@ static int write_adapter_mem(struct c4iw_rdev *rdev, u32 addr, u32 len,
void *data, struct sk_buff *skb,
struct c4iw_wr_wait *wr_waitp)
{
- int ret;
-
- if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl) {
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
- wr_waitp);
- goto out;
- }
-
- if (len <= inline_threshold) {
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
+ if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl ||
+ len <= inline_threshold)
+ return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
wr_waitp);
- goto out;
- }
-
- ret = _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
- if (ret) {
- pr_warn_ratelimited("%s: dma map failure (non fatal)\n",
- pci_name(rdev->lldi.pdev));
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
- wr_waitp);
- }
-out:
- return ret;
+ return _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
}
/*
@@ -280,12 +263,16 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
bool stag_idx_allocated = false;
static atomic_t key;
- if (c4iw_fatal_error(rdev))
+ if (c4iw_fatal_error(rdev)) {
+ kfree_skb(skb);
return -EIO;
+ }
tpt = kmalloc(sizeof(*tpt), GFP_KERNEL);
- if (!tpt)
+ if (!tpt) {
+ kfree_skb(skb);
return -ENOMEM;
+ }
stag_state = stag_state > 0;
stag_idx = (*stag) >> 8;
@@ -297,6 +284,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
rdev->stats.stag.fail++;
mutex_unlock(&rdev->stats.lock);
kfree(tpt);
+ kfree_skb(skb);
return -ENOMEM;
}
mutex_lock(&rdev->stats.lock);
@@ -471,8 +459,10 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
FW_RI_STAG_NSMR, mhp->attr.perms,
mhp->attr.mw_bind_enable, 0, 0, ~0ULL, 0, 0, 0,
NULL, mhp->wr_waitp);
- if (ret)
- goto err_free_skb;
+ if (ret) {
+ kfree_skb(mhp->dereg_skb);
+ goto err_free_wr_wait;
+ }
ret = finish_mem_reg(mhp, stag);
if (ret)
@@ -481,8 +471,6 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
err_dereg_mem:
dereg_mem(&rhp->rdev, mhp->attr.stag, mhp->attr.pbl_size,
mhp->attr.pbl_addr, mhp->dereg_skb, mhp->wr_waitp);
-err_free_skb:
- kfree_skb(mhp->dereg_skb);
err_free_wr_wait:
c4iw_put_wr_wait(mhp->wr_waitp);
err_free_mhp:
diff --git a/drivers/infiniband/hw/erdma/erdma_cm.c b/drivers/infiniband/hw/erdma/erdma_cm.c
index f0c8ebf62a05..189f6c942467 100644
--- a/drivers/infiniband/hw/erdma/erdma_cm.c
+++ b/drivers/infiniband/hw/erdma/erdma_cm.c
@@ -993,7 +993,7 @@ static int kernel_bindconnect(struct socket *s, struct sockaddr *laddr,
int ret;
sock_set_reuseaddr(s->sk);
- ret = s->ops->bind(s, laddr, laddrlen);
+ ret = s->ops->bind(s, (struct sockaddr_unsized *)laddr, laddrlen);
if (ret)
return ret;
ret = s->ops->connect(s, raddr, raddrlen, flags);
@@ -1313,7 +1313,7 @@ int erdma_create_listen(struct iw_cm_id *id, int backlog)
if (ipv4_is_zeronet(laddr->sin_addr.s_addr))
s->sk->sk_bound_dev_if = dev->netdev->ifindex;
- ret = s->ops->bind(s, (struct sockaddr *)laddr,
+ ret = s->ops->bind(s, (struct sockaddr_unsized *)laddr,
sizeof(struct sockaddr_in));
if (ret)
goto error;
diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c
index d22d610c2696..4f73e643b9e5 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -54,8 +54,10 @@ struct rvt_dev_info *rvt_alloc_device(size_t size, int nports)
return rdi;
rdi->ports = kcalloc(nports, sizeof(*rdi->ports), GFP_KERNEL);
- if (!rdi->ports)
+ if (!rdi->ports) {
ib_dealloc_device(&rdi->ibdev);
+ return NULL;
+ }
return rdi;
}
diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c
index 04fabab44058..31fc9987a275 100644
--- a/drivers/infiniband/sw/siw/siw_cm.c
+++ b/drivers/infiniband/sw/siw/siw_cm.c
@@ -1342,7 +1342,7 @@ static int kernel_bindconnect(struct socket *s, struct sockaddr *laddr,
return rv;
}
- rv = s->ops->bind(s, laddr, size);
+ rv = s->ops->bind(s, (struct sockaddr_unsized *)laddr, size);
if (rv < 0)
return rv;
@@ -1793,7 +1793,7 @@ int siw_create_listen(struct iw_cm_id *id, int backlog)
goto error;
}
}
- rv = s->ops->bind(s, (struct sockaddr *)laddr,
+ rv = s->ops->bind(s, (struct sockaddr_unsized *)laddr,
sizeof(struct sockaddr_in));
} else {
struct sockaddr_in6 *laddr = &to_sockaddr_in6(id->local_addr);
@@ -1817,7 +1817,7 @@ int siw_create_listen(struct iw_cm_id *id, int backlog)
goto error;
}
}
- rv = s->ops->bind(s, (struct sockaddr *)laddr,
+ rv = s->ops->bind(s, (struct sockaddr_unsized *)laddr,
sizeof(struct sockaddr_in6));
}
if (rv) {
diff --git a/drivers/input/joystick/xpad.c b/drivers/input/joystick/xpad.c
index 3e0c52c74906..126f99cd0b29 100644
--- a/drivers/input/joystick/xpad.c
+++ b/drivers/input/joystick/xpad.c
@@ -336,6 +336,8 @@ static const struct xpad_device {
{ 0x17ef, 0x6182, "Lenovo Legion Controller for Windows", 0, XTYPE_XBOX360 },
{ 0x1949, 0x041a, "Amazon Game Controller", 0, XTYPE_XBOX360 },
{ 0x1a86, 0xe310, "Legion Go S", 0, XTYPE_XBOX360 },
+ { 0x1b1c, 0x2b1f, "Corsair Novablade Pro (Wired)", 0, XTYPE_XBOX360 },
+ { 0x1b1c, 0x2b2b, "Corsair Novablade Pro (2.4 GHz Receiver)", 0, XTYPE_XBOX360 },
{ 0x1bad, 0x0002, "Harmonix Rock Band Guitar", 0, XTYPE_XBOX360 },
{ 0x1bad, 0x0003, "Harmonix Rock Band Drumkit", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x1bad, 0x0130, "Ion Drum Rocker", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 },
@@ -413,6 +415,7 @@ static const struct xpad_device {
{ 0x2993, 0x2001, "TECNO Pocket Go", 0, XTYPE_XBOX360 },
{ 0x2dc8, 0x2000, "8BitDo Pro 2 Wired Controller fox Xbox", 0, XTYPE_XBOXONE },
{ 0x2dc8, 0x200f, "8BitDo Ultimate 3-mode Controller for Xbox", MAP_SHARE_BUTTON, XTYPE_XBOXONE },
+ { 0x2dc8, 0x201e, "8BitDo Pro 2 Wired Controller for Xbox", MAP_SHARE_BUTTON, XTYPE_XBOXONE },
{ 0x2dc8, 0x3106, "8BitDo Ultimate Wireless / Pro 2 Wired Controller", 0, XTYPE_XBOX360 },
{ 0x2dc8, 0x3109, "8BitDo Ultimate Wireless Bluetooth", 0, XTYPE_XBOX360 },
{ 0x2dc8, 0x310a, "8BitDo Ultimate 2C Wireless Controller", 0, XTYPE_XBOX360 },
@@ -579,6 +582,7 @@ static const struct usb_device_id xpad_table[] = {
XPAD_XBOX360_VENDOR(0x17ef), /* Lenovo */
XPAD_XBOX360_VENDOR(0x1949), /* Amazon controllers */
XPAD_XBOX360_VENDOR(0x1a86), /* Nanjing Qinheng Microelectronics (WCH) */
+ XPAD_XBOX360_VENDOR(0x1b1c), /* Corsair controllers */
XPAD_XBOX360_VENDOR(0x1bad), /* Harmonix Rock Band guitar and drums */
XPAD_XBOX360_VENDOR(0x1ee9), /* ZOTAC Technology Limited */
XPAD_XBOX360_VENDOR(0x20bc), /* BETOP wireless dongles */
diff --git a/drivers/input/misc/ati_remote2.c b/drivers/input/misc/ati_remote2.c
index e84649af801d..3e3db645e71f 100644
--- a/drivers/input/misc/ati_remote2.c
+++ b/drivers/input/misc/ati_remote2.c
@@ -672,8 +672,7 @@ static ssize_t ati_remote2_show_channel_mask(struct device *dev,
struct device_attribute *attr,
char *buf)
{
- struct usb_device *udev = to_usb_device(dev);
- struct usb_interface *intf = usb_ifnum_to_if(udev, 0);
+ struct usb_interface *intf = to_usb_interface(dev);
struct ati_remote2 *ar2 = usb_get_intfdata(intf);
return sprintf(buf, "0x%04x\n", ar2->channel_mask);
@@ -683,8 +682,7 @@ static ssize_t ati_remote2_store_channel_mask(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
- struct usb_device *udev = to_usb_device(dev);
- struct usb_interface *intf = usb_ifnum_to_if(udev, 0);
+ struct usb_interface *intf = to_usb_interface(dev);
struct ati_remote2 *ar2 = usb_get_intfdata(intf);
unsigned int mask;
int r;
@@ -720,8 +718,7 @@ static ssize_t ati_remote2_show_mode_mask(struct device *dev,
struct device_attribute *attr,
char *buf)
{
- struct usb_device *udev = to_usb_device(dev);
- struct usb_interface *intf = usb_ifnum_to_if(udev, 0);
+ struct usb_interface *intf = to_usb_interface(dev);
struct ati_remote2 *ar2 = usb_get_intfdata(intf);
return sprintf(buf, "0x%02x\n", ar2->mode_mask);
@@ -731,8 +728,7 @@ static ssize_t ati_remote2_store_mode_mask(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
- struct usb_device *udev = to_usb_device(dev);
- struct usb_interface *intf = usb_ifnum_to_if(udev, 0);
+ struct usb_interface *intf = to_usb_interface(dev);
struct ati_remote2 *ar2 = usb_get_intfdata(intf);
unsigned int mask;
int err;
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 12302647e267..e64e6da08a5f 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -70,8 +70,8 @@ int amd_iommu_max_glx_val = -1;
*/
DEFINE_IDA(pdom_ids);
-static int amd_iommu_attach_device(struct iommu_domain *dom,
- struct device *dev);
+static int amd_iommu_attach_device(struct iommu_domain *dom, struct device *dev,
+ struct iommu_domain *old);
static void set_dte_entry(struct amd_iommu *iommu,
struct iommu_dev_data *dev_data);
@@ -2655,7 +2655,8 @@ void amd_iommu_domain_free(struct iommu_domain *dom)
}
static int blocked_domain_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
@@ -2713,8 +2714,8 @@ static struct iommu_domain release_domain = {
}
};
-static int amd_iommu_attach_device(struct iommu_domain *dom,
- struct device *dev)
+static int amd_iommu_attach_device(struct iommu_domain *dom, struct device *dev,
+ struct iommu_domain *old)
{
struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
struct protection_domain *domain = to_pdomain(dom);
diff --git a/drivers/iommu/apple-dart.c b/drivers/iommu/apple-dart.c
index 9804022c7f59..83a5aabcd15d 100644
--- a/drivers/iommu/apple-dart.c
+++ b/drivers/iommu/apple-dart.c
@@ -672,7 +672,8 @@ static int apple_dart_domain_add_streams(struct apple_dart_domain *domain,
}
static int apple_dart_attach_dev_paging(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
int ret, i;
struct apple_dart_stream_map *stream_map;
@@ -693,7 +694,8 @@ static int apple_dart_attach_dev_paging(struct iommu_domain *domain,
}
static int apple_dart_attach_dev_identity(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev);
struct apple_dart_stream_map *stream_map;
@@ -717,7 +719,8 @@ static struct iommu_domain apple_dart_identity_domain = {
};
static int apple_dart_attach_dev_blocked(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct apple_dart_master_cfg *cfg = dev_iommu_priv_get(dev);
struct apple_dart_stream_map *stream_map;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
index a52f86454f77..7afb57414039 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-iommufd.c
@@ -138,14 +138,15 @@ void arm_smmu_master_clear_vmaster(struct arm_smmu_master *master)
}
static int arm_smmu_attach_dev_nested(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old_domain)
{
struct arm_smmu_nested_domain *nested_domain =
to_smmu_nested_domain(domain);
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_attach_state state = {
.master = master,
- .old_domain = iommu_get_domain_for_dev(dev),
+ .old_domain = old_domain,
.ssid = IOMMU_NO_PASID,
};
struct arm_smmu_ste ste;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 3d796ff6c2c6..2c237ca4a383 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3070,7 +3070,8 @@ void arm_smmu_attach_commit(struct arm_smmu_attach_state *state)
master->ats_enabled = state->ats_enabled;
}
-static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
+static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev,
+ struct iommu_domain *old_domain)
{
int ret = 0;
struct arm_smmu_ste target;
@@ -3078,7 +3079,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
struct arm_smmu_device *smmu;
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct arm_smmu_attach_state state = {
- .old_domain = iommu_get_domain_for_dev(dev),
+ .old_domain = old_domain,
.ssid = IOMMU_NO_PASID,
};
struct arm_smmu_master *master;
@@ -3254,7 +3255,7 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain,
/*
* When the last user of the CD table goes away downgrade the STE back
- * to a non-cd_table one.
+ * to a non-cd_table one, by re-attaching its sid_domain.
*/
if (!arm_smmu_ssids_in_use(&master->cd_table)) {
struct iommu_domain *sid_domain =
@@ -3262,12 +3263,14 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain,
if (sid_domain->type == IOMMU_DOMAIN_IDENTITY ||
sid_domain->type == IOMMU_DOMAIN_BLOCKED)
- sid_domain->ops->attach_dev(sid_domain, dev);
+ sid_domain->ops->attach_dev(sid_domain, dev,
+ sid_domain);
}
return 0;
}
static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
+ struct iommu_domain *old_domain,
struct device *dev,
struct arm_smmu_ste *ste,
unsigned int s1dss)
@@ -3275,7 +3278,7 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
struct arm_smmu_attach_state state = {
.master = master,
- .old_domain = iommu_get_domain_for_dev(dev),
+ .old_domain = old_domain,
.ssid = IOMMU_NO_PASID,
};
@@ -3316,14 +3319,16 @@ static void arm_smmu_attach_dev_ste(struct iommu_domain *domain,
}
static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old_domain)
{
struct arm_smmu_ste ste;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
arm_smmu_master_clear_vmaster(master);
arm_smmu_make_bypass_ste(master->smmu, &ste);
- arm_smmu_attach_dev_ste(domain, dev, &ste, STRTAB_STE_1_S1DSS_BYPASS);
+ arm_smmu_attach_dev_ste(domain, old_domain, dev, &ste,
+ STRTAB_STE_1_S1DSS_BYPASS);
return 0;
}
@@ -3337,14 +3342,15 @@ static struct iommu_domain arm_smmu_identity_domain = {
};
static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old_domain)
{
struct arm_smmu_ste ste;
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
arm_smmu_master_clear_vmaster(master);
arm_smmu_make_abort_ste(&ste);
- arm_smmu_attach_dev_ste(domain, dev, &ste,
+ arm_smmu_attach_dev_ste(domain, old_domain, dev, &ste,
STRTAB_STE_1_S1DSS_TERMINATE);
return 0;
}
@@ -3647,14 +3653,17 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev)
static void arm_smmu_release_device(struct device *dev)
{
struct arm_smmu_master *master = dev_iommu_priv_get(dev);
+ struct iommu_domain *old_domain = iommu_get_domain_for_dev(dev);
WARN_ON(master->iopf_refcount);
/* Put the STE back to what arm_smmu_init_strtab() sets */
if (dev->iommu->require_direct)
- arm_smmu_attach_dev_identity(&arm_smmu_identity_domain, dev);
+ arm_smmu_attach_dev_identity(&arm_smmu_identity_domain, dev,
+ old_domain);
else
- arm_smmu_attach_dev_blocked(&arm_smmu_blocked_domain, dev);
+ arm_smmu_attach_dev_blocked(&arm_smmu_blocked_domain, dev,
+ old_domain);
arm_smmu_disable_pasid(master);
arm_smmu_remove_master(master);
diff --git a/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c b/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c
index 2b55a221c35a..fdbd487848e6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c
+++ b/drivers/iommu/arm/arm-smmu-v3/tegra241-cmdqv.c
@@ -1015,16 +1015,22 @@ static int tegra241_cmdqv_init_structures(struct arm_smmu_device *smmu)
static struct dentry *cmdqv_debugfs_dir;
#endif
-static struct arm_smmu_device *
-__tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
- int irq)
+/*
+ * Probe the CMDQV and reallocate @smmu into the larger cmdqv->smmu.
+ *
+ * devm_krealloc() may relocate and free the original @smmu, so update *smmu to
+ * the new pointer once it succeeds. The error paths after it do the same, so a
+ * caller falling back keeps a live @smmu instead of the freed original.
+ */
+static int __tegra241_cmdqv_probe(struct arm_smmu_device **smmu,
+ struct resource *res, int irq)
{
static const struct arm_smmu_impl_ops init_ops = {
.init_structures = tegra241_cmdqv_init_structures,
.device_remove = tegra241_cmdqv_remove,
};
- struct tegra241_cmdqv *cmdqv = NULL;
- struct arm_smmu_device *new_smmu;
+ struct device *dev = (*smmu)->dev;
+ struct tegra241_cmdqv *cmdqv;
void __iomem *base;
u32 regval;
int ret;
@@ -1033,25 +1039,28 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
base = ioremap(res->start, resource_size(res));
if (!base) {
- dev_err(smmu->dev, "failed to ioremap\n");
- return NULL;
+ dev_err(dev, "failed to ioremap\n");
+ return -ENOMEM;
}
regval = readl(base + TEGRA241_CMDQV_CONFIG);
if (disable_cmdqv) {
- dev_info(smmu->dev, "Detected disable_cmdqv=true\n");
+ dev_info(dev, "Detected disable_cmdqv=true\n");
writel(regval & ~CMDQV_EN, base + TEGRA241_CMDQV_CONFIG);
+ ret = -ENODEV;
goto iounmap;
}
- cmdqv = devm_krealloc(smmu->dev, smmu, sizeof(*cmdqv), GFP_KERNEL);
- if (!cmdqv)
+ cmdqv = devm_krealloc(dev, *smmu, sizeof(*cmdqv), GFP_KERNEL);
+ if (!cmdqv) {
+ ret = -ENOMEM;
goto iounmap;
- new_smmu = &cmdqv->smmu;
+ }
+ *smmu = &cmdqv->smmu;
cmdqv->irq = irq;
cmdqv->base = base;
- cmdqv->dev = smmu->impl_dev;
+ cmdqv->dev = (*smmu)->impl_dev;
cmdqv->base_phys = res->start;
regval = readl_relaxed(REG_CMDQV(cmdqv, PARAM));
@@ -1063,8 +1072,10 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
cmdqv->vintfs =
kcalloc(cmdqv->num_vintfs, sizeof(*cmdqv->vintfs), GFP_KERNEL);
- if (!cmdqv->vintfs)
+ if (!cmdqv->vintfs) {
+ ret = -ENOMEM;
goto iounmap;
+ }
ida_init(&cmdqv->vintf_ids);
@@ -1093,40 +1104,39 @@ __tegra241_cmdqv_probe(struct arm_smmu_device *smmu, struct resource *res,
#endif
/* Provide init-level ops only, until tegra241_cmdqv_init_structures */
- new_smmu->impl_ops = &init_ops;
+ cmdqv->smmu.impl_ops = &init_ops;
- return new_smmu;
+ return 0;
free_vintfs:
ida_destroy(&cmdqv->vintf_ids);
kfree(cmdqv->vintfs);
iounmap:
iounmap(base);
- return NULL;
+ return ret;
}
struct arm_smmu_device *tegra241_cmdqv_probe(struct arm_smmu_device *smmu)
{
- struct arm_smmu_device *new_smmu;
struct resource *res = NULL;
- int irq;
+ int irq, ret;
if (!smmu->dev->of_node)
res = tegra241_cmdqv_find_acpi_resource(smmu->impl_dev, &irq);
if (!res)
goto out_fallback;
- new_smmu = __tegra241_cmdqv_probe(smmu, res, irq);
+ ret = __tegra241_cmdqv_probe(&smmu, res, irq);
kfree(res);
- if (new_smmu)
- return new_smmu;
+ if (!ret)
+ return smmu;
out_fallback:
dev_info(smmu->impl_dev, "Falling back to standard SMMU CMDQ\n");
smmu->options &= ~ARM_SMMU_OPT_TEGRA241_CMDQV;
put_device(smmu->impl_dev);
- return ERR_PTR(-ENODEV);
+ return smmu;
}
/* User space VINTF and VCMDQ Functions */
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
index 488632b8eeab..1e218fbea35a 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
@@ -1165,7 +1165,8 @@ static void arm_smmu_master_install_s2crs(struct arm_smmu_master_cfg *cfg,
}
}
-static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
+static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev,
+ struct iommu_domain *old)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
@@ -1234,7 +1235,8 @@ static int arm_smmu_attach_dev_type(struct device *dev,
}
static int arm_smmu_attach_dev_identity(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
return arm_smmu_attach_dev_type(dev, S2CR_TYPE_BYPASS);
}
@@ -1249,7 +1251,8 @@ static struct iommu_domain arm_smmu_identity_domain = {
};
static int arm_smmu_attach_dev_blocked(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
return arm_smmu_attach_dev_type(dev, S2CR_TYPE_FAULT);
}
diff --git a/drivers/iommu/arm/arm-smmu/qcom_iommu.c b/drivers/iommu/arm/arm-smmu/qcom_iommu.c
index 82c3ffe0c783..9bee85259c69 100644
--- a/drivers/iommu/arm/arm-smmu/qcom_iommu.c
+++ b/drivers/iommu/arm/arm-smmu/qcom_iommu.c
@@ -359,7 +359,8 @@ static void qcom_iommu_domain_free(struct iommu_domain *domain)
kfree(qcom_domain);
}
-static int qcom_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
+static int qcom_iommu_attach_dev(struct iommu_domain *domain,
+ struct device *dev, struct iommu_domain *old)
{
struct qcom_iommu_dev *qcom_iommu = dev_iommu_priv_get(dev);
struct qcom_iommu_domain *qcom_domain = to_qcom_iommu_domain(domain);
@@ -388,18 +389,18 @@ static int qcom_iommu_attach_dev(struct iommu_domain *domain, struct device *dev
}
static int qcom_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct qcom_iommu_domain *qcom_domain;
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct qcom_iommu_dev *qcom_iommu = dev_iommu_priv_get(dev);
unsigned int i;
- if (domain == identity_domain || !domain)
+ if (old == identity_domain || !old)
return 0;
- qcom_domain = to_qcom_iommu_domain(domain);
+ qcom_domain = to_qcom_iommu_domain(old);
if (WARN_ON(!qcom_domain->iommu))
return -EINVAL;
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 039e84a452d7..9da773ba550b 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1441,7 +1441,7 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
*/
s->dma_address = pci_p2pdma_bus_addr_map(&p2pdma_state,
sg_phys(s));
- sg_dma_len(s) = sg->length;
+ sg_dma_len(s) = s->length;
sg_dma_mark_bus_address(s);
continue;
default:
diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index ce9e935cb84c..38c435d0de39 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -984,7 +984,8 @@ static void exynos_iommu_domain_free(struct iommu_domain *iommu_domain)
}
static int exynos_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct exynos_iommu_owner *owner = dev_iommu_priv_get(dev);
struct exynos_iommu_domain *domain;
@@ -1035,7 +1036,8 @@ static struct iommu_domain exynos_identity_domain = {
};
static int exynos_iommu_attach_device(struct iommu_domain *iommu_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain);
struct exynos_iommu_owner *owner = dev_iommu_priv_get(dev);
@@ -1044,7 +1046,7 @@ static int exynos_iommu_attach_device(struct iommu_domain *iommu_domain,
unsigned long flags;
int err;
- err = exynos_iommu_identity_attach(&exynos_identity_domain, dev);
+ err = exynos_iommu_identity_attach(&exynos_identity_domain, dev, old);
if (err)
return err;
@@ -1429,7 +1431,8 @@ static void exynos_iommu_release_device(struct device *dev)
struct exynos_iommu_owner *owner = dev_iommu_priv_get(dev);
struct sysmmu_drvdata *data;
- WARN_ON(exynos_iommu_identity_attach(&exynos_identity_domain, dev));
+ WARN_ON(exynos_iommu_identity_attach(&exynos_identity_domain, dev,
+ NULL));
list_for_each_entry(data, &owner->controllers, owner_node)
device_link_del(data->link);
diff --git a/drivers/iommu/fsl_pamu_domain.c b/drivers/iommu/fsl_pamu_domain.c
index 5f08523f97cb..9664ef9840d2 100644
--- a/drivers/iommu/fsl_pamu_domain.c
+++ b/drivers/iommu/fsl_pamu_domain.c
@@ -238,7 +238,7 @@ static int update_domain_stash(struct fsl_dma_domain *dma_domain, u32 val)
}
static int fsl_pamu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct fsl_dma_domain *dma_domain = to_fsl_dma_domain(domain);
unsigned long flags;
@@ -298,9 +298,9 @@ static int fsl_pamu_attach_device(struct iommu_domain *domain,
* switches to what looks like BLOCKING.
*/
static int fsl_pamu_platform_attach(struct iommu_domain *platform_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct fsl_dma_domain *dma_domain;
const u32 *prop;
int len;
@@ -311,11 +311,11 @@ static int fsl_pamu_platform_attach(struct iommu_domain *platform_domain,
* Hack to keep things working as they always have, only leaving an
* UNMANAGED domain makes it BLOCKING.
*/
- if (domain == platform_domain || !domain ||
- domain->type != IOMMU_DOMAIN_UNMANAGED)
+ if (old == platform_domain || !old ||
+ old->type != IOMMU_DOMAIN_UNMANAGED)
return 0;
- dma_domain = to_fsl_dma_domain(domain);
+ dma_domain = to_fsl_dma_domain(old);
/*
* Use LIODN of the PCI controller while detaching a
diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 77e51712040b..94781f620680 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -3240,7 +3240,8 @@ void device_block_translation(struct device *dev)
}
static int blocking_domain_attach_dev(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct device_domain_info *info = dev_iommu_priv_get(dev);
@@ -3547,7 +3548,8 @@ int paging_domain_compatible(struct iommu_domain *domain, struct device *dev)
}
static int intel_iommu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
int ret;
@@ -3557,13 +3559,13 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,
if (ret)
return ret;
- ret = iopf_for_domain_set(domain, dev);
+ ret = iopf_for_domain_replace(domain, old, dev);
if (ret)
return ret;
ret = dmar_domain_attach_device(to_dmar_domain(domain), dev);
if (ret)
- iopf_for_domain_remove(domain, dev);
+ iopf_for_domain_replace(old, domain, dev);
return ret;
}
@@ -4423,7 +4425,9 @@ static int device_setup_pass_through(struct device *dev)
context_setup_pass_through_cb, dev);
}
-static int identity_domain_attach_dev(struct iommu_domain *domain, struct device *dev)
+static int identity_domain_attach_dev(struct iommu_domain *domain,
+ struct device *dev,
+ struct iommu_domain *old)
{
struct device_domain_info *info = dev_iommu_priv_get(dev);
struct intel_iommu *iommu = info->iommu;
@@ -4435,10 +4439,13 @@ static int identity_domain_attach_dev(struct iommu_domain *domain, struct device
return 0;
/*
- * No PRI support with the global identity domain. No need to enable or
- * disable PRI in this path as the iommu has been put in the blocking
- * state.
+ * The identity domain has no iopf_handler, so no IOPF reference is
+ * taken for it. The reference held by the old domain must still be
+ * released here; putting the device in the blocking state above does
+ * not affect the IOPF reference count.
*/
+ iopf_for_domain_remove(old, dev);
+
if (sm_supported(iommu))
ret = intel_pasid_setup_pass_through(iommu, dev, IOMMU_NO_PASID);
else
diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c
index a909ab1a665a..eb3e6d687e04 100644
--- a/drivers/iommu/intel/nested.c
+++ b/drivers/iommu/intel/nested.c
@@ -19,7 +19,7 @@
#include "pasid.h"
static int intel_nested_attach_dev(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct device_domain_info *info = dev_iommu_priv_get(dev);
struct dmar_domain *dmar_domain = to_dmar_domain(domain);
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 1547f3587a4e..34c06fb26bf5 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -100,7 +100,7 @@ static int iommu_bus_notifier(struct notifier_block *nb,
unsigned long action, void *data);
static void iommu_release_device(struct device *dev);
static int __iommu_attach_device(struct iommu_domain *domain,
- struct device *dev);
+ struct device *dev, struct iommu_domain *old);
static int __iommu_attach_group(struct iommu_domain *domain,
struct iommu_group *group);
static struct iommu_domain *__iommu_paging_domain_alloc_flags(struct device *dev,
@@ -114,6 +114,7 @@ enum {
static int __iommu_device_set_domain(struct iommu_group *group,
struct device *dev,
struct iommu_domain *new_domain,
+ struct iommu_domain *old_domain,
unsigned int flags);
static int __iommu_group_set_domain_internal(struct iommu_group *group,
struct iommu_domain *new_domain,
@@ -543,7 +544,8 @@ static void iommu_deinit_device(struct device *dev)
* should still avoid touching any hardware configuration either.
*/
if (!dev->iommu->attach_deferred && ops->release_domain)
- ops->release_domain->ops->attach_dev(ops->release_domain, dev);
+ ops->release_domain->ops->attach_dev(ops->release_domain, dev,
+ group->domain);
if (ops->release_device)
ops->release_device(dev);
@@ -628,7 +630,8 @@ static int __iommu_probe_device(struct device *dev, struct list_head *group_list
if (group->default_domain)
iommu_create_device_direct_mappings(group->default_domain, dev);
if (group->domain) {
- ret = __iommu_device_set_domain(group, dev, group->domain, 0);
+ ret = __iommu_device_set_domain(group, dev, group->domain, NULL,
+ 0);
if (ret)
goto err_remove_gdev;
} else if (!group->default_domain && !group_list) {
@@ -2115,14 +2118,14 @@ static void __iommu_group_set_core_domain(struct iommu_group *group)
}
static int __iommu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
int ret;
if (unlikely(domain->ops->attach_dev == NULL))
return -ENODEV;
- ret = domain->ops->attach_dev(domain, dev);
+ ret = domain->ops->attach_dev(domain, dev, old);
if (ret)
return ret;
dev->iommu->attach_deferred = 0;
@@ -2171,7 +2174,7 @@ EXPORT_SYMBOL_GPL(iommu_attach_device);
int iommu_deferred_attach(struct device *dev, struct iommu_domain *domain)
{
if (dev->iommu && dev->iommu->attach_deferred)
- return __iommu_attach_device(domain, dev);
+ return __iommu_attach_device(domain, dev, NULL);
return 0;
}
@@ -2284,6 +2287,7 @@ EXPORT_SYMBOL_GPL(iommu_attach_group);
static int __iommu_device_set_domain(struct iommu_group *group,
struct device *dev,
struct iommu_domain *new_domain,
+ struct iommu_domain *old_domain,
unsigned int flags)
{
int ret;
@@ -2309,7 +2313,7 @@ static int __iommu_device_set_domain(struct iommu_group *group,
dev->iommu->attach_deferred = 0;
}
- ret = __iommu_attach_device(new_domain, dev);
+ ret = __iommu_attach_device(new_domain, dev, old_domain);
if (ret) {
/*
* If we have a blocking domain then try to attach that in hopes
@@ -2319,7 +2323,8 @@ static int __iommu_device_set_domain(struct iommu_group *group,
if ((flags & IOMMU_SET_DOMAIN_MUST_SUCCEED) &&
group->blocking_domain &&
group->blocking_domain != new_domain)
- __iommu_attach_device(group->blocking_domain, dev);
+ __iommu_attach_device(group->blocking_domain, dev,
+ old_domain);
return ret;
}
return 0;
@@ -2366,7 +2371,7 @@ static int __iommu_group_set_domain_internal(struct iommu_group *group,
result = 0;
for_each_group_device(group, gdev) {
ret = __iommu_device_set_domain(group, gdev->dev, new_domain,
- flags);
+ group->domain, flags);
if (ret) {
result = ret;
/*
@@ -2398,7 +2403,7 @@ static int __iommu_group_set_domain_internal(struct iommu_group *group,
*/
if (group->domain)
WARN_ON(__iommu_device_set_domain(
- group, gdev->dev, group->domain,
+ group, gdev->dev, group->domain, new_domain,
IOMMU_SET_DOMAIN_MUST_SUCCEED));
if (gdev == last_gdev)
break;
diff --git a/drivers/iommu/iommufd/selftest.c b/drivers/iommu/iommufd/selftest.c
index cd3d1380cf6c..369311f629d1 100644
--- a/drivers/iommu/iommufd/selftest.c
+++ b/drivers/iommu/iommufd/selftest.c
@@ -218,7 +218,7 @@ static inline struct selftest_obj *to_selftest_obj(struct iommufd_object *obj)
}
static int mock_domain_nop_attach(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct mock_dev *mdev = to_mock_dev(dev);
struct mock_viommu *new_viommu = NULL;
diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c
index 02a2a55ffa0a..ca848288dbf2 100644
--- a/drivers/iommu/ipmmu-vmsa.c
+++ b/drivers/iommu/ipmmu-vmsa.c
@@ -590,7 +590,7 @@ static void ipmmu_domain_free(struct iommu_domain *io_domain)
}
static int ipmmu_attach_device(struct iommu_domain *io_domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct ipmmu_vmsa_device *mmu = to_ipmmu(dev);
@@ -637,17 +637,17 @@ static int ipmmu_attach_device(struct iommu_domain *io_domain,
}
static int ipmmu_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
- struct iommu_domain *io_domain = iommu_get_domain_for_dev(dev);
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct ipmmu_vmsa_domain *domain;
unsigned int i;
- if (io_domain == identity_domain || !io_domain)
+ if (old == identity_domain || !old)
return 0;
- domain = to_vmsa_domain(io_domain);
+ domain = to_vmsa_domain(old);
for (i = 0; i < fwspec->num_ids; ++i)
ipmmu_utlb_disable(domain, fwspec->ids[i]);
diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c
index 1dfa6f69c28f..db14cdc0a508 100644
--- a/drivers/iommu/msm_iommu.c
+++ b/drivers/iommu/msm_iommu.c
@@ -391,7 +391,8 @@ static struct iommu_device *msm_iommu_probe_device(struct device *dev)
return &iommu->iommu;
}
-static int msm_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
+static int msm_iommu_attach_dev(struct iommu_domain *domain, struct device *dev,
+ struct iommu_domain *old)
{
int ret = 0;
unsigned long flags;
@@ -441,19 +442,19 @@ static int msm_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
}
static int msm_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct msm_priv *priv;
unsigned long flags;
struct msm_iommu_dev *iommu;
struct msm_iommu_ctx_dev *master;
int ret = 0;
- if (domain == identity_domain || !domain)
+ if (old == identity_domain || !old)
return 0;
- priv = to_msm_priv(domain);
+ priv = to_msm_priv(old);
free_io_pgtable_ops(priv->iop);
spin_lock_irqsave(&msm_iommu_lock, flags);
diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
index 64ce041238fd..54d8936d9d11 100644
--- a/drivers/iommu/mtk_iommu.c
+++ b/drivers/iommu/mtk_iommu.c
@@ -705,7 +705,7 @@ static void mtk_iommu_domain_free(struct iommu_domain *domain)
}
static int mtk_iommu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct mtk_iommu_data *data = dev_iommu_priv_get(dev), *frstdata;
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
@@ -773,12 +773,12 @@ static int mtk_iommu_attach_device(struct iommu_domain *domain,
}
static int mtk_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct mtk_iommu_data *data = dev_iommu_priv_get(dev);
- if (domain == identity_domain || !domain)
+ if (old == identity_domain || !old)
return 0;
mtk_iommu_config(data, dev, false, 0);
diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c
index ad2dbd29fef8..0433906ed8f5 100644
--- a/drivers/iommu/mtk_iommu_v1.c
+++ b/drivers/iommu/mtk_iommu_v1.c
@@ -303,7 +303,9 @@ static void mtk_iommu_v1_domain_free(struct iommu_domain *domain)
kfree(to_mtk_domain(domain));
}
-static int mtk_iommu_v1_attach_device(struct iommu_domain *domain, struct device *dev)
+static int mtk_iommu_v1_attach_device(struct iommu_domain *domain,
+ struct device *dev,
+ struct iommu_domain *old)
{
struct mtk_iommu_v1_data *data = dev_iommu_priv_get(dev);
struct mtk_iommu_v1_domain *dom = to_mtk_domain(domain);
@@ -329,7 +331,8 @@ static int mtk_iommu_v1_attach_device(struct iommu_domain *domain, struct device
}
static int mtk_iommu_v1_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct mtk_iommu_v1_data *data = dev_iommu_priv_get(dev);
diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c
index c0315c86cd18..1c995463cfb0 100644
--- a/drivers/iommu/omap-iommu.c
+++ b/drivers/iommu/omap-iommu.c
@@ -1431,8 +1431,8 @@ static void omap_iommu_detach_fini(struct omap_iommu_domain *odomain)
odomain->iommus = NULL;
}
-static int
-omap_iommu_attach_dev(struct iommu_domain *domain, struct device *dev)
+static int omap_iommu_attach_dev(struct iommu_domain *domain,
+ struct device *dev, struct iommu_domain *old)
{
struct omap_iommu_arch_data *arch_data = dev_iommu_priv_get(dev);
struct omap_iommu_domain *omap_domain = to_omap_domain(domain);
@@ -1536,15 +1536,15 @@ static void _omap_iommu_detach_dev(struct omap_iommu_domain *omap_domain,
}
static int omap_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct omap_iommu_domain *omap_domain;
- if (domain == identity_domain || !domain)
+ if (old == identity_domain || !old)
return 0;
- omap_domain = to_omap_domain(domain);
+ omap_domain = to_omap_domain(old);
spin_lock(&omap_domain->lock);
_omap_iommu_detach_dev(omap_domain, dev);
spin_unlock(&omap_domain->lock);
diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
index 1f7b8e46812c..890a31262b19 100644
--- a/drivers/iommu/riscv/iommu.c
+++ b/drivers/iommu/riscv/iommu.c
@@ -381,75 +381,91 @@ static int riscv_iommu_queue_wait(struct riscv_iommu_queue *queue,
(int)(cons - index) > 0, 0, timeout_us);
}
-/* Enqueue an entry and wait to be processed if timeout_us > 0
- *
- * Error handling for IOMMU hardware not responding in reasonable time
- * will be added as separate patch series along with other RAS features.
- * For now, only report hardware failure and continue.
- */
+static int riscv_iommu_queue_wait_for_space(struct riscv_iommu_queue *queue,
+ unsigned int last)
+{
+ unsigned int head;
+ unsigned int tail;
+ unsigned int hw_head;
+ unsigned long flags;
+ int ret;
+
+ ret = riscv_iommu_readl_timeout(queue->iommu, Q_HEAD(queue), hw_head,
+ !(hw_head & ~queue->mask) && hw_head != last,
+ 0, RISCV_IOMMU_QUEUE_TIMEOUT);
+ if (ret)
+ return ret;
+
+ raw_spin_lock_irqsave(&queue->lock, flags);
+ head = atomic_read(&queue->head);
+ tail = atomic_read(&queue->tail);
+ if ((tail - head) >= queue->mask) {
+ last = Q_ITEM(queue, head);
+ /*
+ * Re-read hw_head under the lock so that it is consistent with
+ * the freshly computed 'last'. Using the pre-lock snapshot
+ * could produce a stale value that wraps around relative to the
+ * new 'last', advancing the shadow head past entries that have
+ * not yet been consumed by the hardware.
+ */
+ hw_head = riscv_iommu_readl(queue->iommu, Q_HEAD(queue));
+ if (!(hw_head & ~queue->mask) && hw_head != last)
+ atomic_add((hw_head - last) & queue->mask, &queue->head);
+ }
+ raw_spin_unlock_irqrestore(&queue->lock, flags);
+
+ return 0;
+}
+
+/* Enqueue an entry and publish it to the hardware queue. */
static unsigned int riscv_iommu_queue_send(struct riscv_iommu_queue *queue,
void *entry, size_t entry_size)
{
unsigned int prod;
unsigned int head;
- unsigned int tail;
unsigned long flags;
+ int ret;
- /* Do not preempt submission flow. */
- local_irq_save(flags);
+ /* 1. Wait for space availability and reserve the next slot. */
+ for (;;) {
+ raw_spin_lock_irqsave(&queue->lock, flags);
- /* 1. Allocate some space in the queue */
- prod = atomic_inc_return(&queue->prod) - 1;
- head = atomic_read(&queue->head);
+ prod = atomic_read(&queue->tail);
+ head = atomic_read(&queue->head);
- /* 2. Wait for space availability. */
- if ((prod - head) > queue->mask) {
- if (readx_poll_timeout(atomic_read, &queue->head,
- head, (prod - head) < queue->mask,
- 0, RISCV_IOMMU_QUEUE_TIMEOUT))
- goto err_busy;
- } else if ((prod - head) == queue->mask) {
- const unsigned int last = Q_ITEM(queue, head);
+ if ((prod - head) < queue->mask)
+ break;
+
+ head = Q_ITEM(queue, head);
+ raw_spin_unlock_irqrestore(&queue->lock, flags);
- if (riscv_iommu_readl_timeout(queue->iommu, Q_HEAD(queue), head,
- !(head & ~queue->mask) && head != last,
- 0, RISCV_IOMMU_QUEUE_TIMEOUT))
+ ret = riscv_iommu_queue_wait_for_space(queue, head);
+ if (ret)
goto err_busy;
- atomic_add((head - last) & queue->mask, &queue->head);
}
- /* 3. Store entry in the ring buffer */
+ /* 2. Store entry in the ring buffer. */
memcpy(queue->base + Q_ITEM(queue, prod) * entry_size, entry, entry_size);
- /* 4. Wait for all previous entries to be ready */
- if (readx_poll_timeout(atomic_read, &queue->tail, tail, prod == tail,
- 0, RISCV_IOMMU_QUEUE_TIMEOUT))
- goto err_busy;
-
- /*
- * 5. Make sure the ring buffer update (whether in normal or I/O memory) is
- * completed and visible before signaling the tail doorbell to fetch
- * the next command. 'fence ow, ow'
- */
+ /* 3. Make sure the entry is visible before updating the queue tail. */
dma_wmb();
riscv_iommu_writel(queue->iommu, Q_TAIL(queue), Q_ITEM(queue, prod + 1));
/*
- * 6. Make sure the doorbell write to the device has finished before updating
- * the shadow tail index in normal memory. 'fence o, w'
+ * 4. Make sure the doorbell write to the device has finished before
+ * updating the shadow tail index in normal memory. 'fence o, w'
*/
mmiowb();
- atomic_inc(&queue->tail);
+ atomic_set(&queue->tail, prod + 1);
+ atomic_set(&queue->prod, prod + 1);
- /* 7. Complete submission and restore local interrupts */
- local_irq_restore(flags);
+ raw_spin_unlock_irqrestore(&queue->lock, flags);
return prod;
err_busy:
- local_irq_restore(flags);
+ /* Report the failure and continue; full RAS recovery is not implemented. */
dev_err_once(queue->iommu->dev, "Hardware error: command enqueue failed\n");
-
return prod;
}
@@ -1392,7 +1408,8 @@ static bool riscv_iommu_pt_supported(struct riscv_iommu_device *iommu, int pgd_m
}
static int riscv_iommu_attach_paging_domain(struct iommu_domain *iommu_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
struct riscv_iommu_device *iommu = dev_to_iommu(dev);
@@ -1496,7 +1513,8 @@ static struct iommu_domain *riscv_iommu_alloc_paging_domain(struct device *dev)
}
static int riscv_iommu_attach_blocking_domain(struct iommu_domain *iommu_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct riscv_iommu_device *iommu = dev_to_iommu(dev);
struct riscv_iommu_info *info = dev_iommu_priv_get(dev);
@@ -1517,7 +1535,8 @@ static struct iommu_domain riscv_iommu_blocking_domain = {
};
static int riscv_iommu_attach_identity_domain(struct iommu_domain *iommu_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct riscv_iommu_device *iommu = dev_to_iommu(dev);
struct riscv_iommu_info *info = dev_iommu_priv_get(dev);
diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c
index 2d0dabb0d101..309ba78e27d4 100644
--- a/drivers/iommu/rockchip-iommu.c
+++ b/drivers/iommu/rockchip-iommu.c
@@ -968,7 +968,8 @@ static int rk_iommu_enable(struct rk_iommu *iommu)
}
static int rk_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct rk_iommu *iommu;
struct rk_iommu_domain *rk_domain;
@@ -1013,7 +1014,7 @@ static struct iommu_domain rk_identity_domain = {
};
static int rk_iommu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct rk_iommu *iommu;
struct rk_iommu_domain *rk_domain = to_rk_domain(domain);
@@ -1034,7 +1035,7 @@ static int rk_iommu_attach_device(struct iommu_domain *domain,
if (iommu->domain == domain)
return 0;
- ret = rk_iommu_identity_attach(&rk_identity_domain, dev);
+ ret = rk_iommu_identity_attach(&rk_identity_domain, dev, old);
if (ret)
return ret;
@@ -1049,8 +1050,17 @@ static int rk_iommu_attach_device(struct iommu_domain *domain,
return 0;
ret = rk_iommu_enable(iommu);
- if (ret)
- WARN_ON(rk_iommu_identity_attach(&rk_identity_domain, dev));
+ if (ret) {
+ /*
+ * Note rk_iommu_identity_attach() might fail before physically
+ * attaching the dev to iommu->domain, in which case the actual
+ * old domain for this revert should be rk_identity_domain v.s.
+ * iommu->domain. Since rk_iommu_identity_attach() does not care
+ * about the old domain argument for now, this is not a problem.
+ */
+ WARN_ON(rk_iommu_identity_attach(&rk_identity_domain, dev,
+ iommu->domain));
+ }
pm_runtime_put(iommu->dev);
diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
index e36d08a0fd1d..97a39b3d6ba9 100644
--- a/drivers/iommu/s390-iommu.c
+++ b/drivers/iommu/s390-iommu.c
@@ -670,7 +670,8 @@ int zpci_iommu_register_ioat(struct zpci_dev *zdev, u8 *status)
}
static int blocking_domain_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct zpci_dev *zdev = to_zpci_dev(dev);
struct s390_domain *s390_domain;
@@ -694,7 +695,8 @@ static int blocking_domain_attach_device(struct iommu_domain *domain,
}
static int s390_iommu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct s390_domain *s390_domain = to_s390_domain(domain);
struct zpci_dev *zdev = to_zpci_dev(dev);
@@ -709,7 +711,7 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
domain->geometry.aperture_end < zdev->start_dma))
return -EINVAL;
- blocking_domain_attach_device(&blocking_domain, dev);
+ blocking_domain_attach_device(&blocking_domain, dev, old);
/* If we fail now DMA remains blocked via blocking domain */
cc = s390_iommu_domain_reg_ioat(zdev, domain, &status);
@@ -1133,13 +1135,14 @@ static int __init s390_iommu_init(void)
subsys_initcall(s390_iommu_init);
static int s390_attach_dev_identity(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct zpci_dev *zdev = to_zpci_dev(dev);
u8 status;
int cc;
- blocking_domain_attach_device(&blocking_domain, dev);
+ blocking_domain_attach_device(&blocking_domain, dev, old);
/* If we fail now DMA remains blocked via blocking domain */
cc = s390_iommu_domain_reg_ioat(zdev, domain, &status);
diff --git a/drivers/iommu/sprd-iommu.c b/drivers/iommu/sprd-iommu.c
index c7ca1d8a0b15..555d4505c747 100644
--- a/drivers/iommu/sprd-iommu.c
+++ b/drivers/iommu/sprd-iommu.c
@@ -247,7 +247,8 @@ static void sprd_iommu_domain_free(struct iommu_domain *domain)
}
static int sprd_iommu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct sprd_iommu_device *sdev = dev_iommu_priv_get(dev);
struct sprd_iommu_domain *dom = to_sprd_domain(domain);
diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
index 6306570d57db..90b26fe21817 100644
--- a/drivers/iommu/sun50i-iommu.c
+++ b/drivers/iommu/sun50i-iommu.c
@@ -771,7 +771,8 @@ static void sun50i_iommu_detach_domain(struct sun50i_iommu *iommu,
}
static int sun50i_iommu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct sun50i_iommu *iommu = dev_iommu_priv_get(dev);
struct sun50i_iommu_domain *sun50i_domain;
@@ -797,7 +798,8 @@ static struct iommu_domain sun50i_iommu_identity_domain = {
};
static int sun50i_iommu_attach_device(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
struct sun50i_iommu_domain *sun50i_domain = to_sun50i_domain(domain);
struct sun50i_iommu *iommu;
@@ -813,7 +815,7 @@ static int sun50i_iommu_attach_device(struct iommu_domain *domain,
if (iommu->domain == domain)
return 0;
- sun50i_iommu_identity_attach(&sun50i_iommu_identity_domain, dev);
+ sun50i_iommu_identity_attach(&sun50i_iommu_identity_domain, dev, old);
sun50i_iommu_attach_domain(iommu, sun50i_domain);
diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c
index f6f26a072820..c391e7f2cde6 100644
--- a/drivers/iommu/tegra-smmu.c
+++ b/drivers/iommu/tegra-smmu.c
@@ -490,7 +490,7 @@ static void tegra_smmu_as_unprepare(struct tegra_smmu *smmu,
}
static int tegra_smmu_attach_dev(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev, struct iommu_domain *old)
{
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct tegra_smmu *smmu = dev_iommu_priv_get(dev);
@@ -524,9 +524,9 @@ static int tegra_smmu_attach_dev(struct iommu_domain *domain,
}
static int tegra_smmu_identity_attach(struct iommu_domain *identity_domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
- struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
struct tegra_smmu_as *as;
struct tegra_smmu *smmu;
@@ -535,10 +535,10 @@ static int tegra_smmu_identity_attach(struct iommu_domain *identity_domain,
if (!fwspec)
return -ENODEV;
- if (domain == identity_domain || !domain)
+ if (old == identity_domain || !old)
return 0;
- as = to_smmu_as(domain);
+ as = to_smmu_as(old);
smmu = as->smmu;
for (index = 0; index < fwspec->num_ids; index++) {
tegra_smmu_disable(smmu, fwspec->ids[index], as->id);
diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c
index b39d6f134ab2..d314fa5cd847 100644
--- a/drivers/iommu/virtio-iommu.c
+++ b/drivers/iommu/virtio-iommu.c
@@ -730,7 +730,8 @@ static struct iommu_domain *viommu_domain_alloc_identity(struct device *dev)
return domain;
}
-static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev)
+static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev,
+ struct iommu_domain *old)
{
int ret = 0;
struct virtio_iommu_req_attach req;
@@ -781,7 +782,8 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev)
}
static int viommu_attach_identity_domain(struct iommu_domain *domain,
- struct device *dev)
+ struct device *dev,
+ struct iommu_domain *old)
{
int ret = 0;
struct virtio_iommu_req_attach req;
diff --git a/drivers/irqchip/irq-gic-v5-irs.c b/drivers/irqchip/irq-gic-v5-irs.c
index a650eccdb37a..cf951d4071cb 100644
--- a/drivers/irqchip/irq-gic-v5-irs.c
+++ b/drivers/irqchip/irq-gic-v5-irs.c
@@ -652,6 +652,18 @@ static int __init gicv5_irs_of_init_affinity(struct device_node *node,
return ret;
}
+static void __init gicv5_irs_clear_affinity(struct gicv5_irs_chip_data *irs_data)
+{
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ if (per_cpu(per_cpu_irs_data, cpu) == irs_data) {
+ per_cpu(cpu_iaffid, cpu).valid = false;
+ per_cpu(per_cpu_irs_data, cpu) = NULL;
+ }
+ }
+}
+
static void irs_setup_pri_bits(u32 idr1)
{
switch (FIELD_GET(GICV5_IRS_IDR1_PRIORITY_BITS, idr1)) {
@@ -762,6 +774,7 @@ static int __init gicv5_irs_init(struct device_node *node)
return 0;
out_iomem:
+ gicv5_irs_clear_affinity(irs_data);
iounmap(irs_base);
out_err:
kfree(irs_data);
@@ -776,6 +789,7 @@ void __init gicv5_irs_remove(void)
gicv5_deinit_lpis();
list_for_each_entry_safe(irs_data, tmp_data, &irs_nodes, entry) {
+ gicv5_irs_clear_affinity(irs_data);
iounmap(irs_data->irs_base);
list_del(&irs_data->entry);
kfree(irs_data);
diff --git a/drivers/isdn/mISDN/l1oip_core.c b/drivers/isdn/mISDN/l1oip_core.c
index f732f6614d37..6ab036e4a35f 100644
--- a/drivers/isdn/mISDN/l1oip_core.c
+++ b/drivers/isdn/mISDN/l1oip_core.c
@@ -676,7 +676,7 @@ l1oip_socket_thread(void *data)
hc->sin_remote.sin_port = htons((unsigned short)hc->remoteport);
/* bind to incoming port */
- if (socket->ops->bind(socket, (struct sockaddr *)&hc->sin_local,
+ if (socket->ops->bind(socket, (struct sockaddr_unsized *)&hc->sin_local,
sizeof(hc->sin_local))) {
printk(KERN_ERR "%s: Failed to bind socket to port %d.\n",
__func__, hc->localport);
diff --git a/drivers/isdn/mISDN/socket.c b/drivers/isdn/mISDN/socket.c
index b215b28cad7b..77b900db1cac 100644
--- a/drivers/isdn/mISDN/socket.c
+++ b/drivers/isdn/mISDN/socket.c
@@ -462,7 +462,7 @@ static int data_sock_getsockopt(struct socket *sock, int level, int optname,
}
static int
-data_sock_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
+data_sock_bind(struct socket *sock, struct sockaddr_unsized *addr, int addr_len)
{
struct sockaddr_mISDN *maddr = (struct sockaddr_mISDN *) addr;
struct sock *sk = sock->sk;
@@ -696,7 +696,7 @@ base_sock_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg)
}
static int
-base_sock_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
+base_sock_bind(struct socket *sock, struct sockaddr_unsized *addr, int addr_len)
{
struct sockaddr_mISDN *maddr = (struct sockaddr_mISDN *) addr;
struct sock *sk = sock->sk;
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index fd735155336b..db8108acb312 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -2974,8 +2974,13 @@ static int crypt_ctr_cipher_new(struct dm_target *ti, char *cipher_in, char *key
}
}
- if (*ivmode && !strcmp(*ivmode, "lmk"))
+ if (*ivmode && !strcmp(*ivmode, "lmk")) {
+ if (crypt_integrity_aead(cc)) {
+ ti->error = "AEAD transforms not supported for LMK";
+ return -EINVAL;
+ }
cc->tfms_count = 64;
+ }
if (*ivmode && !strcmp(*ivmode, "essiv")) {
if (!*ivopts) {
diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c
index 6536143f8912..415e763041bd 100644
--- a/drivers/md/dm-integrity.c
+++ b/drivers/md/dm-integrity.c
@@ -2720,7 +2720,7 @@ static void dm_integrity_inline_recheck(struct work_struct *w)
outgoing_data = dio->integrity_payload + PAGE_SIZE;
while (dio->bio_details.bi_iter.bi_size) {
- char digest[HASH_MAX_DIGESTSIZE];
+ char digest[MAX_T(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)];
int r;
struct bio_integrity_payload *bip;
struct bio_vec bv;
@@ -2787,7 +2787,7 @@ static inline bool dm_integrity_check(struct dm_integrity_c *ic, struct dm_integ
unsigned pos = 0;
while (dio->bio_details.bi_iter.bi_size) {
- char digest[HASH_MAX_DIGESTSIZE];
+ char digest[MAX_T(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)];
struct bio_vec bv = bio_iter_iovec(bio, dio->bio_details.bi_iter);
char *mem = integrity_kmap(ic, bv.bv_page);
integrity_sector_checksum(ic, &dio->ahash_req, dio->bio_details.bi_iter.bi_sector, mem, bv.bv_offset, digest);
diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
index f8d048ece982..4495489c488d 100644
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -556,10 +556,16 @@ static void *get_result_buffer(struct dm_ioctl *param, size_t param_size,
{
param->data_start = align_ptr(param + 1) - (void *) param;
- if (param->data_start < param_size)
+ if (param->data_start < param_size) {
*len = param_size - param->data_start;
- else
+ } else {
+ /*
+ * The buffer can be as small as offsetof(*param, data), which
+ * is less than sizeof(*param), so don't run off the end of it.
+ */
*len = 0;
+ param->data_start = param_size;
+ }
return ((void *) param) + param->data_start;
}
@@ -1175,6 +1181,7 @@ static int do_resume(struct dm_ioctl *param)
/* Do we need to load a new map ? */
if (new_map) {
sector_t old_size, new_size;
+ blk_mode_t new_map_mode;
/* Suspend if it isn't already suspended */
if (param->flags & DM_SKIP_LOCKFS_FLAG)
@@ -1202,6 +1209,7 @@ static int do_resume(struct dm_ioctl *param)
}
}
+ new_map_mode = dm_table_get_mode(new_map);
old_size = dm_get_size(md);
old_map = dm_swap_table(md, new_map);
if (IS_ERR(old_map)) {
@@ -1214,7 +1222,7 @@ static int do_resume(struct dm_ioctl *param)
if (old_size && new_size && old_size != new_size)
need_resize_uevent = true;
- if (dm_table_get_mode(new_map) & BLK_OPEN_WRITE)
+ if (new_map_mode & BLK_OPEN_WRITE)
set_disk_ro(dm_disk(md), 0);
else
set_disk_ro(dm_disk(md), 1);
diff --git a/drivers/md/dm-pcache/cache.c b/drivers/md/dm-pcache/cache.c
index 8635fbf780aa..013dfb171f0f 100644
--- a/drivers/md/dm-pcache/cache.c
+++ b/drivers/md/dm-pcache/cache.c
@@ -118,6 +118,9 @@ int cache_pos_decode(struct pcache_cache *cache,
if (!latest_addr)
return -EIO;
+ if (!cache_seg_id_valid(cache, latest.cache_seg_id))
+ return -EIO;
+
pos->cache_seg = &cache->segments[latest.cache_seg_id];
if (latest.seg_off >= pos->cache_seg->segment.data_size)
@@ -158,6 +161,7 @@ static int cache_init(struct dm_pcache *pcache)
cache->cache_dev = &pcache->cache_dev;
cache->n_segs = cache_dev->seg_num;
atomic_set(&cache->gc_errors, 0);
+ atomic_set(&cache->writeback_errors, 0);
spin_lock_init(&cache->seg_map_lock);
spin_lock_init(&cache->key_head_lock);
diff --git a/drivers/md/dm-pcache/cache.h b/drivers/md/dm-pcache/cache.h
index bf61a4a487bc..8809ec5ae943 100644
--- a/drivers/md/dm-pcache/cache.h
+++ b/drivers/md/dm-pcache/cache.h
@@ -421,6 +421,20 @@ static inline bool cache_seg_is_ctrl_seg(u32 cache_seg_id)
return (cache_seg_id == 0);
}
+/**
+ * cache_seg_id_valid - Validate a cache segment id read from the cache device.
+ * @cache: Pointer to the pcache_cache structure.
+ * @cache_seg_id: Segment id decoded from on-media metadata.
+ *
+ * On-media segment ids are only protected by a CRC, which an attacker who can
+ * format the cache device computes over their chosen value. Reject any id that
+ * would index cache->segments[] out of bounds before it is dereferenced.
+ */
+static inline bool cache_seg_id_valid(struct pcache_cache *cache, u32 cache_seg_id)
+{
+ return cache_seg_id < cache->cache_info.n_segs;
+}
+
/**
* cache_key_cutfront - Cuts a specified length from the front of a cache key.
* @key: Pointer to pcache_cache_key structure.
diff --git a/drivers/md/dm-pcache/cache_gc.c b/drivers/md/dm-pcache/cache_gc.c
index 99999a92c4e6..d551848912b4 100644
--- a/drivers/md/dm-pcache/cache_gc.c
+++ b/drivers/md/dm-pcache/cache_gc.c
@@ -74,11 +74,17 @@ static bool need_gc(struct pcache_cache *cache, struct pcache_cache_pos *dirty_t
* @cache: Pointer to the pcache_cache structure.
* @kset_onmedia: Pointer to the kset_onmedia structure for the last kset.
*/
-static void last_kset_gc(struct pcache_cache *cache, struct pcache_cache_kset_onmedia *kset_onmedia)
+static int last_kset_gc(struct pcache_cache *cache, struct pcache_cache_kset_onmedia *kset_onmedia)
{
struct dm_pcache *pcache = CACHE_TO_PCACHE(cache);
struct pcache_cache_segment *cur_seg, *next_seg;
+ if (!cache_seg_id_valid(cache, kset_onmedia->next_cache_seg_id)) {
+ pcache_dev_err(pcache, "invalid next_cache_seg_id %u in gc (n_segs %u)\n",
+ kset_onmedia->next_cache_seg_id, cache->n_segs);
+ return -EIO;
+ }
+
cur_seg = cache->key_tail.cache_seg;
next_seg = &cache->segments[kset_onmedia->next_cache_seg_id];
@@ -94,6 +100,8 @@ static void last_kset_gc(struct pcache_cache *cache, struct pcache_cache_kset_on
spin_lock(&cache->seg_map_lock);
__clear_bit(cur_seg->cache_seg_id, cache->seg_map);
spin_unlock(&cache->seg_map_lock);
+
+ return 0;
}
void pcache_cache_gc_fn(struct work_struct *work)
@@ -130,7 +138,11 @@ void pcache_cache_gc_fn(struct work_struct *work)
if (dirty_tail.cache_seg == key_tail.cache_seg)
break;
- last_kset_gc(cache, kset_onmedia);
+ ret = last_kset_gc(cache, kset_onmedia);
+ if (ret) {
+ atomic_inc(&cache->gc_errors);
+ return;
+ }
continue;
}
diff --git a/drivers/md/dm-pcache/cache_key.c b/drivers/md/dm-pcache/cache_key.c
index c5a8ac02b216..1526f5aa6a8d 100644
--- a/drivers/md/dm-pcache/cache_key.c
+++ b/drivers/md/dm-pcache/cache_key.c
@@ -103,6 +103,12 @@ int cache_key_decode(struct pcache_cache *cache,
return -EIO;
}
+ if (!cache_seg_id_valid(cache, key_onmedia->cache_seg_id)) {
+ pcache_dev_err(pcache, "invalid cache_seg_id %u in cache key (n_segs %u)\n",
+ key_onmedia->cache_seg_id, cache->n_segs);
+ return -EIO;
+ }
+
key->cache_pos.cache_seg = &cache->segments[key_onmedia->cache_seg_id];
key->cache_pos.seg_off = key_onmedia->cache_seg_off;
@@ -810,6 +816,11 @@ int cache_replay(struct pcache_cache *cache)
goto out;
}
+ if (!cache_seg_id_valid(cache, kset_onmedia->next_cache_seg_id)) {
+ ret = -EIO;
+ goto out;
+ }
+
next_seg = &cache->segments[kset_onmedia->next_cache_seg_id];
pos->cache_seg = next_seg;
diff --git a/drivers/md/dm-pcache/cache_writeback.c b/drivers/md/dm-pcache/cache_writeback.c
index 656e4d048912..c8a4c8110a58 100644
--- a/drivers/md/dm-pcache/cache_writeback.c
+++ b/drivers/md/dm-pcache/cache_writeback.c
@@ -196,12 +196,18 @@ static int cache_kset_insert_tree(struct pcache_cache *cache, struct pcache_cach
return ret;
}
-static void last_kset_writeback(struct pcache_cache *cache,
+static int last_kset_writeback(struct pcache_cache *cache,
struct pcache_cache_kset_onmedia *last_kset_onmedia)
{
struct dm_pcache *pcache = CACHE_TO_PCACHE(cache);
struct pcache_cache_segment *next_seg;
+ if (!cache_seg_id_valid(cache, last_kset_onmedia->next_cache_seg_id)) {
+ pcache_dev_err(pcache, "invalid next_cache_seg_id %u in writeback (n_segs %u)\n",
+ last_kset_onmedia->next_cache_seg_id, cache->n_segs);
+ return -EIO;
+ }
+
pcache_dev_debug(pcache, "last kset, next: %u\n", last_kset_onmedia->next_cache_seg_id);
next_seg = &cache->segments[last_kset_onmedia->next_cache_seg_id];
@@ -211,6 +217,8 @@ static void last_kset_writeback(struct pcache_cache *cache,
cache->dirty_tail.seg_off = 0;
cache_encode_dirty_tail(cache);
mutex_unlock(&cache->dirty_tail_lock);
+
+ return 0;
}
void cache_writeback_fn(struct work_struct *work)
@@ -244,7 +252,11 @@ void cache_writeback_fn(struct work_struct *work)
}
if (kset_onmedia->flags & PCACHE_KSET_FLAGS_LAST) {
- last_kset_writeback(cache, kset_onmedia);
+ ret = last_kset_writeback(cache, kset_onmedia);
+ if (ret) {
+ atomic_inc(&cache->writeback_errors);
+ goto unlock;
+ }
delay = 0;
goto queue_work;
}
@@ -256,8 +268,8 @@ void cache_writeback_fn(struct work_struct *work)
ret = cache_kset_insert_tree(cache, kset_onmedia);
if (ret) {
- delay = PCACHE_CACHE_WRITEBACK_INTERVAL;
- goto queue_work;
+ atomic_inc(&cache->writeback_errors);
+ goto unlock;
}
cache_wb_tree_writeback(cache, get_kset_onmedia_size(kset_onmedia));
diff --git a/drivers/md/dm-snap.c b/drivers/md/dm-snap.c
index dbd148967de4..4a3a6cf772f7 100644
--- a/drivers/md/dm-snap.c
+++ b/drivers/md/dm-snap.c
@@ -1298,6 +1298,20 @@ static int snapshot_ctr(struct dm_target *ti, unsigned int argc, char **argv)
goto bad_store;
}
+ /*
+ * snapshot-merge requires the optional prepare_merge/commit_merge
+ * callbacks; the transient exception store does not implement them.
+ * The handover path checks this in __validate_exception_handover(),
+ * but a directly constructed snapshot-merge never goes through it.
+ */
+ if (dm_target_is_snapshot_merge(ti) &&
+ (!s->store->type->prepare_merge ||
+ !s->store->type->commit_merge)) {
+ ti->error = "Snapshot exception store does not support snapshot-merge.";
+ r = -EINVAL;
+ goto bad_hash_tables;
+ }
+
argv += args_used;
argc -= args_used;
diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
index 781cbfb2d348..cef236893db0 100644
--- a/drivers/md/md-bitmap.c
+++ b/drivers/md/md-bitmap.c
@@ -2064,23 +2064,18 @@ static void bitmap_end_behind_write(struct mddev *mddev)
bitmap->mddev->bitmap_info.max_write_behind);
}
-static bool bitmap_wait_behind_writes(struct mddev *mddev, bool nowait)
+static void bitmap_wait_behind_writes(struct mddev *mddev)
{
struct bitmap *bitmap = mddev->bitmap;
/* wait for behind writes to complete */
if (bitmap && atomic_read(&bitmap->behind_writes) > 0) {
- if (nowait)
- return false;
-
pr_debug("md:%s: behind writes in progress - waiting to stop.\n",
mdname(mddev));
/* need to kick something here to make sure I/O goes? */
wait_event(bitmap->behind_wait,
atomic_read(&bitmap->behind_writes) == 0);
}
-
- return true;
}
static void bitmap_destroy(struct mddev *mddev)
@@ -2090,7 +2085,7 @@ static void bitmap_destroy(struct mddev *mddev)
if (!bitmap) /* there was no bitmap */
return;
- bitmap_wait_behind_writes(mddev, false);
+ bitmap_wait_behind_writes(mddev);
if (!test_bit(MD_SERIALIZE_POLICY, &mddev->flags))
mddev_destroy_serial_pool(mddev, NULL);
diff --git a/drivers/md/md-bitmap.h b/drivers/md/md-bitmap.h
index 7535742982fc..890276d9c66e 100644
--- a/drivers/md/md-bitmap.h
+++ b/drivers/md/md-bitmap.h
@@ -99,7 +99,7 @@ struct bitmap_operations {
void (*start_behind_write)(struct mddev *mddev);
void (*end_behind_write)(struct mddev *mddev);
- bool (*wait_behind_writes)(struct mddev *mddev, bool nowait);
+ void (*wait_behind_writes)(struct mddev *mddev);
md_bitmap_fn *start_write;
md_bitmap_fn *end_write;
diff --git a/drivers/md/md-linear.c b/drivers/md/md-linear.c
index 7033d982d377..a07620bb6d51 100644
--- a/drivers/md/md-linear.c
+++ b/drivers/md/md-linear.c
@@ -71,6 +71,7 @@ static int linear_set_limits(struct mddev *mddev)
int err;
md_init_stacking_limits(&lim);
+ lim.features |= BLK_FEAT_NOWAIT;
lim.max_hw_sectors = mddev->chunk_sectors;
lim.max_write_zeroes_sectors = mddev->chunk_sectors;
lim.max_hw_wzeroes_unmap_sectors = mddev->chunk_sectors;
diff --git a/drivers/md/md-llbitmap.c b/drivers/md/md-llbitmap.c
index 5c60b26ad246..903ce6bb2418 100644
--- a/drivers/md/md-llbitmap.c
+++ b/drivers/md/md-llbitmap.c
@@ -1442,19 +1442,13 @@ static void llbitmap_end_behind_write(struct mddev *mddev)
wake_up(&llbitmap->behind_wait);
}
-static bool llbitmap_wait_behind_writes(struct mddev *mddev, bool nowait)
+static void llbitmap_wait_behind_writes(struct mddev *mddev)
{
struct llbitmap *llbitmap = mddev->bitmap;
- if (llbitmap && atomic_read(&llbitmap->behind_writes) > 0) {
- if (nowait)
- return false;
-
+ if (llbitmap && atomic_read(&llbitmap->behind_writes) > 0)
wait_event(llbitmap->behind_wait,
atomic_read(&llbitmap->behind_writes) == 0);
- }
-
- return true;
}
static ssize_t bits_show(struct mddev *mddev, char *page)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index d03d0fdf763d..03d328506b28 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -396,25 +396,24 @@ static bool is_suspended(struct mddev *mddev, struct bio *bio)
bool md_handle_request(struct mddev *mddev, struct bio *bio)
{
check_suspended:
- if (is_suspended(mddev, bio)) {
- DEFINE_WAIT(__wait);
- /* Bail out if REQ_NOWAIT is set for the bio */
- if (bio->bi_opf & REQ_NOWAIT) {
- bio_wouldblock_error(bio);
- return true;
- }
- for (;;) {
- prepare_to_wait(&mddev->sb_wait, &__wait,
- TASK_UNINTERRUPTIBLE);
- if (!is_suspended(mddev, bio))
- break;
- schedule();
+ if (unlikely(md_cloned_bio(mddev, bio))) {
+ /*
+ * This bio is an MD cloned bio and already holds an
+ * active_io reference, so percpu_ref_get() is safe here.
+ */
+ percpu_ref_get(&mddev->active_io);
+ } else {
+ if (is_suspended(mddev, bio)) {
+ /* Bail out if REQ_NOWAIT is set for the bio */
+ if (bio->bi_opf & REQ_NOWAIT) {
+ bio_wouldblock_error(bio);
+ return true;
+ }
+ wait_event(mddev->sb_wait, !is_suspended(mddev, bio));
}
- finish_wait(&mddev->sb_wait, &__wait);
+ if (!percpu_ref_tryget_live(&mddev->active_io))
+ goto check_suspended;
}
- if (!percpu_ref_tryget_live(&mddev->active_io))
- goto check_suspended;
-
if (!mddev->pers->make_request(mddev, bio)) {
percpu_ref_put(&mddev->active_io);
if (!mddev->gendisk && mddev->pers->prepare_suspend)
@@ -6175,7 +6174,7 @@ void md_init_stacking_limits(struct queue_limits *lim)
{
blk_set_stacking_limits(lim);
lim->features = BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA |
- BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT;
+ BLK_FEAT_IO_STAT;
}
EXPORT_SYMBOL_GPL(md_init_stacking_limits);
@@ -6523,7 +6522,6 @@ int md_run(struct mddev *mddev)
int err;
struct md_rdev *rdev;
struct md_personality *pers;
- bool nowait = true;
if (list_empty(&mddev->disks))
/* cannot run an array with no devices.. */
@@ -6594,7 +6592,6 @@ int md_run(struct mddev *mddev)
}
}
sysfs_notify_dirent_safe(rdev->sysfs_state);
- nowait = nowait && bdev_nowait(rdev->bdev);
}
pers = get_pers(mddev->level, mddev->clevel);
@@ -6941,7 +6938,7 @@ EXPORT_SYMBOL_GPL(md_stop_writes);
static void mddev_detach(struct mddev *mddev)
{
if (md_bitmap_enabled(mddev, false))
- mddev->bitmap_ops->wait_behind_writes(mddev, false);
+ mddev->bitmap_ops->wait_behind_writes(mddev);
if (mddev->pers && mddev->pers->quiesce && !is_md_suspended(mddev)) {
mddev->pers->quiesce(mddev, 1);
mddev->pers->quiesce(mddev, 0);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index bba0f6790c03..cab3d7360b09 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -1043,6 +1043,11 @@ void mddev_update_io_opt(struct mddev *mddev, unsigned int nr_stripes);
extern const struct block_device_operations md_fops;
+static inline bool md_cloned_bio(struct mddev *mddev, struct bio *bio)
+{
+ return bio->bi_pool == &mddev->io_clone_set;
+}
+
/*
* MD devices can be used undeneath by DM, in which case ->gendisk is NULL.
*/
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 0bdba5944af9..dc4be4c9d955 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -382,6 +382,7 @@ static int raid0_set_limits(struct mddev *mddev)
int err;
md_init_stacking_limits(&lim);
+ lim.features |= BLK_FEAT_NOWAIT;
lim.max_hw_sectors = mddev->chunk_sectors;
lim.max_write_zeroes_sectors = mddev->chunk_sectors;
lim.max_hw_wzeroes_unmap_sectors = mddev->chunk_sectors;
diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c
index aba8c751e192..df47bce8178b 100644
--- a/drivers/md/raid1-10.c
+++ b/drivers/md/raid1-10.c
@@ -295,9 +295,8 @@ static inline bool raid1_should_read_first(struct mddev *mddev,
}
/*
- * bio with REQ_RAHEAD or REQ_NOWAIT can fail at anytime, before such IO is
- * submitted to the underlying disks, hence don't record badblocks or retry
- * in this case.
+ * bio with REQ_RAHEAD can fail at anytime, before such IO is submitted to the
+ * underlying disks, hence don't record badblocks or retry in this case.
*
* BLK_STS_INVAL means the bio was not valid for the underlying device. This
* is a user error, not a device failure, so retrying or recording bad blocks
@@ -305,6 +304,5 @@ static inline bool raid1_should_read_first(struct mddev *mddev,
*/
static inline bool raid1_should_handle_error(struct bio *bio)
{
- return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT)) &&
- bio->bi_status != BLK_STS_INVAL;
+ return !(bio->bi_opf & REQ_RAHEAD) && bio->bi_status != BLK_STS_INVAL;
}
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 1370be1e5cc7..fa0f140c85a2 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1028,10 +1028,8 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr)
wake_up(&conf->wait_barrier);
}
-static bool _wait_barrier(struct r1conf *conf, int idx, bool nowait)
+static void _wait_barrier(struct r1conf *conf, int idx)
{
- bool ret = true;
-
/*
* We need to increase conf->nr_pending[idx] very early here,
* then raise_barrier() can be blocked when it waits for
@@ -1062,7 +1060,7 @@ static bool _wait_barrier(struct r1conf *conf, int idx, bool nowait)
*/
if (!READ_ONCE(conf->array_frozen) &&
!atomic_read(&conf->barrier[idx]))
- return ret;
+ return;
/*
* After holding conf->resync_lock, conf->nr_pending[idx]
@@ -1081,26 +1079,18 @@ static bool _wait_barrier(struct r1conf *conf, int idx, bool nowait)
wake_up_barrier(conf);
/* Wait for the barrier in same barrier unit bucket to drop. */
- /* Return false when nowait flag is set */
- if (nowait) {
- ret = false;
- } else {
- wait_event_lock_irq(conf->wait_barrier,
- !conf->array_frozen &&
- !atomic_read(&conf->barrier[idx]),
- conf->resync_lock);
- atomic_inc(&conf->nr_pending[idx]);
- }
+ wait_event_lock_irq(conf->wait_barrier, !conf->array_frozen &&
+ !atomic_read(&conf->barrier[idx]),
+ conf->resync_lock);
+ atomic_inc(&conf->nr_pending[idx]);
atomic_dec(&conf->nr_waiting[idx]);
spin_unlock_irq(&conf->resync_lock);
- return ret;
}
-static bool wait_read_barrier(struct r1conf *conf, sector_t sector_nr, bool nowait)
+static void wait_read_barrier(struct r1conf *conf, sector_t sector_nr)
{
int idx = sector_to_idx(sector_nr);
- bool ret = true;
/*
* Very similar to _wait_barrier(). The difference is, for read
@@ -1112,7 +1102,7 @@ static bool wait_read_barrier(struct r1conf *conf, sector_t sector_nr, bool nowa
atomic_inc(&conf->nr_pending[idx]);
if (!READ_ONCE(conf->array_frozen))
- return ret;
+ return;
spin_lock_irq(&conf->resync_lock);
atomic_inc(&conf->nr_waiting[idx]);
@@ -1124,27 +1114,19 @@ static bool wait_read_barrier(struct r1conf *conf, sector_t sector_nr, bool nowa
wake_up_barrier(conf);
/* Wait for array to be unfrozen */
- /* Return false when nowait flag is set */
- if (nowait) {
- /* Return false when nowait flag is set */
- ret = false;
- } else {
- wait_event_lock_irq(conf->wait_barrier,
- !conf->array_frozen,
- conf->resync_lock);
- atomic_inc(&conf->nr_pending[idx]);
- }
+ wait_event_lock_irq(conf->wait_barrier, !conf->array_frozen,
+ conf->resync_lock);
+ atomic_inc(&conf->nr_pending[idx]);
atomic_dec(&conf->nr_waiting[idx]);
spin_unlock_irq(&conf->resync_lock);
- return ret;
}
-static bool wait_barrier(struct r1conf *conf, sector_t sector_nr, bool nowait)
+static void wait_barrier(struct r1conf *conf, sector_t sector_nr)
{
int idx = sector_to_idx(sector_nr);
- return _wait_barrier(conf, idx, nowait);
+ _wait_barrier(conf, idx);
}
static void _allow_barrier(struct r1conf *conf, int idx)
@@ -1319,29 +1301,26 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
int max_sectors;
int rdisk;
bool r1bio_existed = !!r1_bio;
- bool nowait = bio->bi_opf & REQ_NOWAIT;
/*
- * If r1_bio is set, we are blocking the raid1d thread
- * so there is a tiny risk of deadlock. So ask for
+ * An md cloned bio indicates we are in the error path.
+ * This is more reliable than checking r1_bio, which might
+ * be NULL even in the error path if a failed bio was split.
+ */
+ bool err_path = md_cloned_bio(mddev, bio);
+
+ /*
+ * If we are in the error path, we are blocking the raid1d
+ * thread so there is a tiny risk of deadlock. So ask for
* emergency memory if needed.
*/
- gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
+ gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
/*
* Still need barrier for READ in case that whole
* array is frozen.
*/
- if (!wait_read_barrier(conf, bio->bi_iter.bi_sector, nowait)) {
- bio_wouldblock_error(bio);
-
- if (r1bio_existed) {
- set_bit(R1BIO_Returned, &r1_bio->state);
- raid_end_bio_io(r1_bio);
- }
-
- return;
- }
+ wait_read_barrier(conf, bio->bi_iter.bi_sector);
if (!r1_bio)
r1_bio = alloc_r1bio(mddev, bio);
@@ -1379,11 +1358,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
* over-take any writes that are 'behind'
*/
mddev_add_trace_msg(mddev, "raid1 wait behind writes");
- if (!mddev->bitmap_ops->wait_behind_writes(mddev, nowait)) {
- bio_wouldblock_error(bio);
- set_bit(R1BIO_Returned, &r1_bio->state);
- goto err_handle;
- }
+ mddev->bitmap_ops->wait_behind_writes(mddev);
}
if (max_sectors < bio_sectors(bio)) {
@@ -1399,13 +1374,12 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
}
r1_bio->read_disk = rdisk;
- if (!r1bio_existed) {
+ if (likely(!md_cloned_bio(mddev, bio))) {
md_account_bio(mddev, &bio);
r1_bio->master_bio = bio;
}
read_bio = bio_alloc_clone(mirror->rdev->bdev, bio, gfp,
&mddev->bio_set);
- read_bio->bi_opf &= ~REQ_NOWAIT;
r1_bio->bios[rdisk] = read_bio;
read_bio->bi_iter.bi_sector = r1_bio->sector +
@@ -1424,7 +1398,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
raid_end_bio_io(r1_bio);
}
-static bool wait_blocked_rdev(struct mddev *mddev, struct bio *bio)
+static void wait_blocked_rdev(struct mddev *mddev, struct bio *bio)
{
struct r1conf *conf = mddev->private;
int disks = conf->raid_disks * 2;
@@ -1444,9 +1418,6 @@ static bool wait_blocked_rdev(struct mddev *mddev, struct bio *bio)
set_bit(BlockedBadBlocks, &rdev->flags);
if (rdev_blocked(rdev)) {
- if (bio->bi_opf & REQ_NOWAIT)
- return false;
-
mddev_add_trace_msg(rdev->mddev, "raid1 wait rdev %d blocked",
rdev->raid_disk);
atomic_inc(&rdev->nr_pending);
@@ -1454,8 +1425,6 @@ static bool wait_blocked_rdev(struct mddev *mddev, struct bio *bio)
goto retry;
}
}
-
- return true;
}
static void raid1_start_write_behind(struct mddev *mddev, struct r1bio *r1_bio,
@@ -1482,7 +1451,7 @@ static void raid1_start_write_behind(struct mddev *mddev, struct r1bio *r1_bio,
}
-static void raid1_write_request(struct mddev *mddev, struct bio *bio,
+static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
int max_write_sectors)
{
struct r1conf *conf = mddev->private;
@@ -1493,16 +1462,12 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
int max_sectors;
bool write_behind = false;
bool is_discard = (bio_op(bio) == REQ_OP_DISCARD);
+ sector_t sector = bio->bi_iter.bi_sector;
if (mddev_is_clustered(mddev) &&
mddev->cluster_ops->area_resyncing(mddev, WRITE,
bio->bi_iter.bi_sector, bio_end_sector(bio))) {
-
DEFINE_WAIT(w);
- if (bio->bi_opf & REQ_NOWAIT) {
- bio_wouldblock_error(bio);
- return;
- }
for (;;) {
prepare_to_wait(&conf->wait_barrier,
&w, TASK_IDLE);
@@ -1520,16 +1485,9 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
* thread has put up a bar for new requests.
* Continue immediately if no resync is active currently.
*/
- if (!wait_barrier(conf, bio->bi_iter.bi_sector,
- bio->bi_opf & REQ_NOWAIT)) {
- bio_wouldblock_error(bio);
- return;
- }
+ wait_barrier(conf, sector);
- if (!wait_blocked_rdev(mddev, bio)) {
- bio_wouldblock_error(bio);
- return;
- }
+ wait_blocked_rdev(mddev, bio);
r1_bio = alloc_r1bio(mddev, bio);
r1_bio->sectors = max_write_sectors;
@@ -1660,7 +1618,6 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
wait_for_serialization(rdev, r1_bio);
}
- mbio->bi_opf &= ~REQ_NOWAIT;
r1_bio->bios[i] = mbio;
mbio->bi_iter.bi_sector = (r1_bio->sector + rdev->data_offset);
@@ -1687,7 +1644,8 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
/* In case raid1d snuck in to freeze_array */
wake_up_barrier(conf);
- return;
+ return true;
+
err_handle:
for (k = 0; k < i; k++) {
if (r1_bio->bios[k]) {
@@ -1697,6 +1655,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
}
raid_end_bio_io(r1_bio);
+ return false;
}
static bool raid1_make_request(struct mddev *mddev, struct bio *bio)
@@ -1720,8 +1679,9 @@ static bool raid1_make_request(struct mddev *mddev, struct bio *bio)
if (bio_data_dir(bio) == READ)
raid1_read_request(mddev, bio, sectors, NULL);
else {
- md_write_start(mddev,bio);
- raid1_write_request(mddev, bio, sectors);
+ md_write_start(mddev, bio);
+ if (!raid1_write_request(mddev, bio, sectors))
+ md_write_end(mddev);
}
return true;
}
@@ -1822,7 +1782,7 @@ static void close_sync(struct r1conf *conf)
int idx;
for (idx = 0; idx < BARRIER_BUCKETS_NR; idx++) {
- _wait_barrier(conf, idx, false);
+ _wait_barrier(conf, idx);
_allow_barrier(conf, idx);
}
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index a60e4ba919a7..60c8b3414790 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1002,32 +1002,22 @@ static bool wait_barrier_nolock(struct r10conf *conf)
return false;
}
-static bool wait_barrier(struct r10conf *conf, bool nowait)
+static void wait_barrier(struct r10conf *conf)
{
- bool ret = true;
-
if (wait_barrier_nolock(conf))
- return true;
+ return;
write_seqlock_irq(&conf->resync_lock);
if (conf->barrier) {
- /* Return false when nowait flag is set */
- if (nowait) {
- ret = false;
- } else {
- conf->nr_waiting++;
- mddev_add_trace_msg(conf->mddev, "raid10 wait barrier");
- wait_event_barrier(conf, stop_waiting_barrier(conf));
- conf->nr_waiting--;
- }
+ conf->nr_waiting++;
+ mddev_add_trace_msg(conf->mddev, "raid10 wait barrier");
+ wait_event_barrier(conf, stop_waiting_barrier(conf));
+ conf->nr_waiting--;
if (!conf->nr_waiting)
wake_up(&conf->wait_barrier);
}
- /* Only increment nr_pending when we wait */
- if (ret)
- atomic_inc(&conf->nr_pending);
+ atomic_inc(&conf->nr_pending);
write_sequnlock_irq(&conf->resync_lock);
- return ret;
}
static void allow_barrier(struct r10conf *conf)
@@ -1119,34 +1109,26 @@ static void raid10_unplug(struct blk_plug_cb *cb, bool from_schedule)
* currently.
* 2. If IO spans the reshape position. Need to wait for reshape to pass.
*/
-static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf,
+static void regular_request_wait(struct mddev *mddev, struct r10conf *conf,
struct bio *bio, sector_t sectors)
{
- /* Bail out if REQ_NOWAIT is set for the bio */
- if (!wait_barrier(conf, bio->bi_opf & REQ_NOWAIT)) {
- bio_wouldblock_error(bio);
- return false;
- }
+ wait_barrier(conf);
+
while (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
bio->bi_iter.bi_sector < conf->reshape_progress &&
bio->bi_iter.bi_sector + sectors > conf->reshape_progress) {
allow_barrier(conf);
- if (bio->bi_opf & REQ_NOWAIT) {
- bio_wouldblock_error(bio);
- return false;
- }
mddev_add_trace_msg(conf->mddev, "raid10 wait reshape");
wait_event(conf->wait_barrier,
conf->reshape_progress <= bio->bi_iter.bi_sector ||
conf->reshape_progress >= bio->bi_iter.bi_sector +
sectors);
- wait_barrier(conf, false);
+ wait_barrier(conf);
}
- return true;
}
static void raid10_read_request(struct mddev *mddev, struct bio *bio,
- struct r10bio *r10_bio, bool io_accounting)
+ struct r10bio *r10_bio)
{
struct r10conf *conf = mddev->private;
struct bio *read_bio;
@@ -1155,7 +1137,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
char b[BDEVNAME_SIZE];
int slot = r10_bio->read_slot;
struct md_rdev *err_rdev = NULL;
- gfp_t gfp = GFP_NOIO;
+
+ /*
+ * An md cloned bio indicates we are in the error path.
+ * This is more reliable than checking slot, which might
+ * be -1 even in the error path if a failed bio was split.
+ */
+ bool err_path = md_cloned_bio(mddev, bio);
+
+ /*
+ * If we are in the error path, we are blocking the raid10d
+ * thread so there is a tiny risk of deadlock. So ask for
+ * emergency memory if needed.
+ */
+ gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO;
if (slot >= 0 && r10_bio->devs[slot].rdev) {
/*
@@ -1166,11 +1161,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
* we lose the device name in error messages.
*/
int disk;
- /*
- * As we are blocking raid10, it is a little safer to
- * use __GFP_HIGH.
- */
- gfp = GFP_NOIO | __GFP_HIGH;
disk = r10_bio->devs[slot].devnum;
err_rdev = conf->mirrors[disk].rdev;
@@ -1183,10 +1173,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
}
}
- if (!regular_request_wait(mddev, conf, bio, r10_bio->sectors)) {
- free_r10bio(r10_bio);
- return;
- }
+ regular_request_wait(mddev, conf, bio, r10_bio->sectors);
rdev = read_balance(conf, r10_bio, &max_sectors);
if (!rdev) {
@@ -1207,7 +1194,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
allow_barrier(conf);
bio = bio_submit_split_bioset(bio, max_sectors,
&conf->bio_split);
- wait_barrier(conf, false);
+ wait_barrier(conf);
if (!bio) {
set_bit(R10BIO_Returned, &r10_bio->state);
goto err_handle;
@@ -1218,12 +1205,11 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
}
slot = r10_bio->read_slot;
- if (io_accounting) {
+ if (likely(!md_cloned_bio(mddev, bio))) {
md_account_bio(mddev, &bio);
r10_bio->master_bio = bio;
}
read_bio = bio_alloc_clone(rdev->bdev, bio, gfp, &mddev->bio_set);
- read_bio->bi_opf &= ~REQ_NOWAIT;
r10_bio->devs[slot].bio = read_bio;
r10_bio->devs[slot].rdev = rdev;
@@ -1257,7 +1243,6 @@ static void raid10_write_one_disk(struct mddev *mddev, struct r10bio *r10_bio,
conf->mirrors[devnum].rdev;
mbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, &mddev->bio_set);
- mbio->bi_opf &= ~REQ_NOWAIT;
if (replacement)
r10_bio->devs[n_copy].repl_bio = mbio;
else
@@ -1336,12 +1321,12 @@ static void wait_blocked_dev(struct mddev *mddev, struct r10bio *r10_bio)
"raid10 %s wait rdev %d blocked",
__func__, blocked_rdev->raid_disk);
md_wait_for_blocked_rdev(blocked_rdev, mddev);
- wait_barrier(conf, false);
+ wait_barrier(conf);
goto retry_wait;
}
}
-static void raid10_write_request(struct mddev *mddev, struct bio *bio,
+static bool raid10_write_request(struct mddev *mddev, struct bio *bio,
struct r10bio *r10_bio)
{
struct r10conf *conf = mddev->private;
@@ -1354,28 +1339,14 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
mddev->cluster_ops->area_resyncing(mddev, WRITE,
bio->bi_iter.bi_sector,
bio_end_sector(bio)))) {
- DEFINE_WAIT(w);
- /* Bail out if REQ_NOWAIT is set for the bio */
- if (bio->bi_opf & REQ_NOWAIT) {
- bio_wouldblock_error(bio);
- return;
- }
- for (;;) {
- prepare_to_wait(&conf->wait_barrier,
- &w, TASK_IDLE);
- if (!mddev->cluster_ops->area_resyncing(mddev, WRITE,
- bio->bi_iter.bi_sector, bio_end_sector(bio)))
- break;
- schedule();
- }
- finish_wait(&conf->wait_barrier, &w);
+ wait_event_idle(conf->wait_barrier,
+ !mddev->cluster_ops->area_resyncing(mddev, WRITE,
+ bio->bi_iter.bi_sector,
+ bio_end_sector(bio)));
}
sectors = r10_bio->sectors;
- if (!regular_request_wait(mddev, conf, bio, sectors)) {
- free_r10bio(r10_bio);
- return;
- }
+ regular_request_wait(mddev, conf, bio, sectors);
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
(mddev->reshape_backwards
@@ -1388,11 +1359,6 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
set_mask_bits(&mddev->sb_flags, 0,
BIT(MD_SB_CHANGE_DEVS) | BIT(MD_SB_CHANGE_PENDING));
md_wakeup_thread(mddev->thread);
- if (bio->bi_opf & REQ_NOWAIT) {
- allow_barrier(conf);
- bio_wouldblock_error(bio);
- return;
- }
mddev_add_trace_msg(conf->mddev,
"raid10 wait reshape metadata");
wait_event(mddev->sb_wait,
@@ -1480,7 +1446,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
allow_barrier(conf);
bio = bio_submit_split_bioset(bio, r10_bio->sectors,
&conf->bio_split);
- wait_barrier(conf, false);
+ wait_barrier(conf);
if (!bio) {
set_bit(R10BIO_Returned, &r10_bio->state);
goto err_handle;
@@ -1500,7 +1466,8 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
raid10_write_one_disk(mddev, r10_bio, bio, true, i);
}
one_write_done(r10_bio);
- return;
+ return true;
+
err_handle:
for (k = 0; k < i; k++) {
int d = r10_bio->devs[k].devnum;
@@ -1518,10 +1485,12 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
}
raid_end_bio_io(r10_bio);
+ return false;
}
-static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
+static bool __make_request(struct mddev *mddev, struct bio *bio, int sectors)
{
+ bool ret;
struct r10conf *conf = mddev->private;
struct r10bio *r10_bio;
@@ -1537,10 +1506,13 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) *
conf->geo.raid_disks);
+ ret = true;
if (bio_data_dir(bio) == READ)
- raid10_read_request(mddev, bio, r10_bio, true);
+ raid10_read_request(mddev, bio, r10_bio);
else
- raid10_write_request(mddev, bio, r10_bio);
+ ret = raid10_write_request(mddev, bio, r10_bio);
+
+ return ret;
}
static void raid_end_discard_bio(struct r10bio *r10bio)
@@ -1617,11 +1589,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery))
return -EAGAIN;
- if (!wait_barrier(conf, bio->bi_opf & REQ_NOWAIT)) {
- bio_wouldblock_error(bio);
- md_write_end(mddev);
- return 0;
- }
+ wait_barrier(conf);
/*
* Check reshape again to avoid reshape happens after checking
@@ -1672,7 +1640,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
allow_barrier(conf);
/* Resend the fist split part */
submit_bio_noacct(split);
- wait_barrier(conf, false);
+ wait_barrier(conf);
}
div_u64_rem(bio_end, stripe_size, &remainder);
if (remainder) {
@@ -1692,7 +1660,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
/* Resend the second split part */
submit_bio_noacct(bio);
bio = split;
- wait_barrier(conf, false);
+ wait_barrier(conf);
}
bio_start = bio->bi_iter.bi_sector;
@@ -1850,7 +1818,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
end_disk_offset += geo->stride;
atomic_inc(&first_r10bio->remaining);
raid_end_discard_bio(r10_bio);
- wait_barrier(conf, false);
+ wait_barrier(conf);
goto retry_discard;
}
@@ -1891,7 +1859,8 @@ static bool raid10_make_request(struct mddev *mddev, struct bio *bio)
sectors = chunk_sects -
(bio->bi_iter.bi_sector &
(chunk_sects - 1));
- __make_request(mddev, bio, sectors);
+ if (!__make_request(mddev, bio, sectors))
+ md_write_end(mddev);
/* In case raid10d snuck in to freeze_array */
wake_up_barrier(conf);
@@ -2048,7 +2017,7 @@ static void print_conf(struct r10conf *conf)
static void close_sync(struct r10conf *conf)
{
- wait_barrier(conf, false);
+ wait_barrier(conf);
allow_barrier(conf);
mempool_exit(&conf->r10buf_pool);
@@ -2864,7 +2833,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)
rdev_dec_pending(rdev, mddev);
r10_bio->state = 0;
- raid10_read_request(mddev, r10_bio->master_bio, r10_bio, false);
+ raid10_read_request(mddev, r10_bio->master_bio, r10_bio);
/*
* allow_barrier after re-submit to ensure no sync io
* can be issued while regular io pending.
@@ -4748,7 +4717,7 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr,
if (need_flush ||
time_after(jiffies, conf->reshape_checkpoint + 10*HZ)) {
/* Need to update reshape_position in metadata */
- wait_barrier(conf, false);
+ wait_barrier(conf);
mddev->reshape_position = conf->reshape_progress;
if (mddev->reshape_backwards)
mddev->curr_resync_completed = raid10_size(mddev, 0, 0)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 03d11f4420b0..2817cd7d197e 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5706,10 +5706,6 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
struct bio *orig_bi = bi;
int stripe_sectors;
- /* We need to handle this when io_uring supports discard/trim */
- if (WARN_ON_ONCE(bi->bi_opf & REQ_NOWAIT))
- return;
-
if (mddev->reshape_position != MaxSector)
/* Skip discard while reshape is happening */
return;
@@ -6171,14 +6167,6 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
pr_debug("raid456: %s, logical %llu to %llu\n", __func__,
bi->bi_iter.bi_sector, ctx.last_sector);
- /* Bail out if conflicts with reshape and REQ_NOWAIT is set */
- if ((bi->bi_opf & REQ_NOWAIT) &&
- get_reshape_loc(mddev, conf, logical_sector) == LOC_INSIDE_RESHAPE) {
- bio_wouldblock_error(bi);
- if (rw == WRITE)
- md_write_end(mddev);
- return true;
- }
md_account_bio(mddev, &bi);
/*
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
index 30f523a5edb1..e1ad685a42e5 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
@@ -809,7 +809,15 @@ static int wave5_vpu_dec_stop(struct vpu_instance *inst)
* calls do not block on a mutex while inside this spinlock.
*/
spin_unlock_irqrestore(&inst->state_spinlock, flags);
+ /*
+ * V4L2_DEC_CMD_STOP can arrive while the device is runtime
+ * suspended (e.g. on pipeline teardown). Setting the EOS flag
+ * accesses VPU registers via send_firmware_command(), so the
+ * device must be resumed first to avoid an asynchronous SError.
+ */
+ pm_runtime_resume_and_get(inst->dev->dev);
ret = wave5_vpu_dec_set_eos_on_firmware(inst);
+ pm_runtime_put_autosuspend(inst->dev->dev);
if (ret)
return ret;
@@ -1706,10 +1714,21 @@ static void wave5_vpu_dec_job_abort(void *priv)
if (ret)
return;
+ /*
+ * job_abort() runs from the STREAMOFF path and may be called while the
+ * device is runtime suspended. Setting the EOS flag talks to the
+ * firmware (send_firmware_command() accesses VPU registers), so the
+ * device must be resumed first; otherwise the register access faults
+ * with an asynchronous SError.
+ */
+ pm_runtime_resume_and_get(inst->dev->dev);
+
ret = wave5_vpu_dec_set_eos_on_firmware(inst);
if (ret)
dev_warn(inst->dev->dev,
"Setting EOS for the bitstream, fail: %d\n", ret);
+
+ pm_runtime_put_autosuspend(inst->dev->dev);
}
static int wave5_vpu_dec_job_ready(void *priv)
diff --git a/drivers/media/platform/qcom/iris/iris_ctrls.c b/drivers/media/platform/qcom/iris/iris_ctrls.c
index 754a5ad718bc..9da050aa1f7c 100644
--- a/drivers/media/platform/qcom/iris/iris_ctrls.c
+++ b/drivers/media/platform/qcom/iris/iris_ctrls.c
@@ -301,7 +301,7 @@ int iris_ctrls_init(struct iris_inst *inst)
void iris_session_init_caps(struct iris_core *core)
{
- struct platform_inst_fw_cap *caps;
+ const struct platform_inst_fw_cap *caps;
u32 i, num_cap, cap_id;
caps = core->iris_platform_data->inst_fw_caps_dec;
diff --git a/drivers/media/platform/qcom/iris/iris_platform_common.h b/drivers/media/platform/qcom/iris/iris_platform_common.h
index 69720d679050..5ffc1874e8c6 100644
--- a/drivers/media/platform/qcom/iris/iris_platform_common.h
+++ b/drivers/media/platform/qcom/iris/iris_platform_common.h
@@ -215,9 +215,9 @@ struct iris_platform_data {
const char *fwname;
u32 pas_id;
struct platform_inst_caps *inst_caps;
- struct platform_inst_fw_cap *inst_fw_caps_dec;
+ const struct platform_inst_fw_cap *inst_fw_caps_dec;
u32 inst_fw_caps_dec_size;
- struct platform_inst_fw_cap *inst_fw_caps_enc;
+ const struct platform_inst_fw_cap *inst_fw_caps_enc;
u32 inst_fw_caps_enc_size;
struct tz_cp_config *tz_cp_config_data;
u32 core_arch;
diff --git a/drivers/media/platform/qcom/iris/iris_platform_gen2.c b/drivers/media/platform/qcom/iris/iris_platform_gen2.c
index 768362683a74..0f130ca7cb93 100644
--- a/drivers/media/platform/qcom/iris/iris_platform_gen2.c
+++ b/drivers/media/platform/qcom/iris/iris_platform_gen2.c
@@ -19,7 +19,7 @@
#define VIDEO_ARCH_LX 1
#define BITRATE_MAX 245000000
-static struct platform_inst_fw_cap inst_fw_cap_sm8550_dec[] = {
+static const struct platform_inst_fw_cap inst_fw_cap_sm8550_dec[] = {
{
.cap_id = PROFILE_H264,
.min = V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE,
@@ -203,7 +203,7 @@ static struct platform_inst_fw_cap inst_fw_cap_sm8550_dec[] = {
},
};
-static struct platform_inst_fw_cap inst_fw_cap_sm8550_enc[] = {
+static const struct platform_inst_fw_cap inst_fw_cap_sm8550_enc[] = {
{
.cap_id = PROFILE_H264,
.min = V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE,
diff --git a/drivers/media/platform/qcom/iris/iris_platform_qcs8300.h b/drivers/media/platform/qcom/iris/iris_platform_qcs8300.h
index 35ea0efade73..87517361a1cf 100644
--- a/drivers/media/platform/qcom/iris/iris_platform_qcs8300.h
+++ b/drivers/media/platform/qcom/iris/iris_platform_qcs8300.h
@@ -5,7 +5,7 @@
#define BITRATE_MAX 245000000
-static struct platform_inst_fw_cap inst_fw_cap_qcs8300_dec[] = {
+static const struct platform_inst_fw_cap inst_fw_cap_qcs8300_dec[] = {
{
.cap_id = PROFILE_H264,
.min = V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE,
@@ -189,7 +189,7 @@ static struct platform_inst_fw_cap inst_fw_cap_qcs8300_dec[] = {
},
};
-static struct platform_inst_fw_cap inst_fw_cap_qcs8300_enc[] = {
+static const struct platform_inst_fw_cap inst_fw_cap_qcs8300_enc[] = {
{
.cap_id = PROFILE_H264,
.min = V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE,
diff --git a/drivers/media/platform/qcom/iris/iris_platform_sm8250.c b/drivers/media/platform/qcom/iris/iris_platform_sm8250.c
index d130d18c6543..e3b99c42a7b6 100644
--- a/drivers/media/platform/qcom/iris/iris_platform_sm8250.c
+++ b/drivers/media/platform/qcom/iris/iris_platform_sm8250.c
@@ -17,7 +17,7 @@
#define BITRATE_PEAK_DEFAULT (BITRATE_DEFAULT * 2)
#define BITRATE_STEP 100
-static struct platform_inst_fw_cap inst_fw_cap_sm8250_dec[] = {
+static const struct platform_inst_fw_cap inst_fw_cap_sm8250_dec[] = {
{
.cap_id = PIPE,
.min = PIPE_1,
@@ -38,7 +38,7 @@ static struct platform_inst_fw_cap inst_fw_cap_sm8250_dec[] = {
},
};
-static struct platform_inst_fw_cap inst_fw_cap_sm8250_enc[] = {
+static const struct platform_inst_fw_cap inst_fw_cap_sm8250_enc[] = {
{
.cap_id = STAGE,
.min = STAGE_1,
diff --git a/drivers/media/platform/qcom/iris/iris_vdec.c b/drivers/media/platform/qcom/iris/iris_vdec.c
index c9f9c16cf084..36bdc56c579f 100644
--- a/drivers/media/platform/qcom/iris/iris_vdec.c
+++ b/drivers/media/platform/qcom/iris/iris_vdec.c
@@ -61,12 +61,6 @@ int iris_vdec_inst_init(struct iris_inst *inst)
return iris_ctrls_init(inst);
}
-void iris_vdec_inst_deinit(struct iris_inst *inst)
-{
- kfree(inst->fmt_dst);
- kfree(inst->fmt_src);
-}
-
static const struct iris_fmt iris_vdec_formats[] = {
[IRIS_FMT_H264] = {
.pixfmt = V4L2_PIX_FMT_H264,
diff --git a/drivers/media/platform/qcom/iris/iris_vdec.h b/drivers/media/platform/qcom/iris/iris_vdec.h
index ec1ce55d1375..5123d2a340e1 100644
--- a/drivers/media/platform/qcom/iris/iris_vdec.h
+++ b/drivers/media/platform/qcom/iris/iris_vdec.h
@@ -9,7 +9,6 @@
struct iris_inst;
int iris_vdec_inst_init(struct iris_inst *inst);
-void iris_vdec_inst_deinit(struct iris_inst *inst);
int iris_vdec_enum_fmt(struct iris_inst *inst, struct v4l2_fmtdesc *f);
int iris_vdec_try_fmt(struct iris_inst *inst, struct v4l2_format *f);
int iris_vdec_s_fmt(struct iris_inst *inst, struct v4l2_format *f);
diff --git a/drivers/media/platform/qcom/iris/iris_venc.c b/drivers/media/platform/qcom/iris/iris_venc.c
index c11ed778cc9e..6f66943dc198 100644
--- a/drivers/media/platform/qcom/iris/iris_venc.c
+++ b/drivers/media/platform/qcom/iris/iris_venc.c
@@ -74,12 +74,6 @@ int iris_venc_inst_init(struct iris_inst *inst)
return iris_ctrls_init(inst);
}
-void iris_venc_inst_deinit(struct iris_inst *inst)
-{
- kfree(inst->fmt_dst);
- kfree(inst->fmt_src);
-}
-
static const struct iris_fmt iris_venc_formats[] = {
[IRIS_FMT_H264] = {
.pixfmt = V4L2_PIX_FMT_H264,
diff --git a/drivers/media/platform/qcom/iris/iris_venc.h b/drivers/media/platform/qcom/iris/iris_venc.h
index c4db7433da53..00c1716b2747 100644
--- a/drivers/media/platform/qcom/iris/iris_venc.h
+++ b/drivers/media/platform/qcom/iris/iris_venc.h
@@ -9,7 +9,6 @@
struct iris_inst;
int iris_venc_inst_init(struct iris_inst *inst);
-void iris_venc_inst_deinit(struct iris_inst *inst);
int iris_venc_enum_fmt(struct iris_inst *inst, struct v4l2_fmtdesc *f);
int iris_venc_try_fmt(struct iris_inst *inst, struct v4l2_format *f);
int iris_venc_s_fmt(struct iris_inst *inst, struct v4l2_format *f);
diff --git a/drivers/media/platform/qcom/iris/iris_vidc.c b/drivers/media/platform/qcom/iris/iris_vidc.c
index 4c43d816e830..83b2b0e59f03 100644
--- a/drivers/media/platform/qcom/iris/iris_vidc.c
+++ b/drivers/media/platform/qcom/iris/iris_vidc.c
@@ -288,10 +288,6 @@ int iris_close(struct file *filp)
v4l2_m2m_ctx_release(inst->m2m_ctx);
v4l2_m2m_release(inst->m2m_dev);
mutex_lock(&inst->lock);
- if (inst->domain == DECODER)
- iris_vdec_inst_deinit(inst);
- else if (inst->domain == ENCODER)
- iris_venc_inst_deinit(inst);
iris_session_close(inst);
iris_inst_change_state(inst, IRIS_INST_DEINIT);
iris_v4l2_fh_deinit(inst, filp);
@@ -303,6 +299,8 @@ int iris_close(struct file *filp)
mutex_unlock(&inst->lock);
mutex_destroy(&inst->ctx_q_lock);
mutex_destroy(&inst->lock);
+ kfree(inst->fmt_src);
+ kfree(inst->fmt_dst);
kfree(inst);
return 0;
diff --git a/drivers/media/platform/verisilicon/hantro.h b/drivers/media/platform/verisilicon/hantro.h
index e0fdc4535b2d..0353de154a1e 100644
--- a/drivers/media/platform/verisilicon/hantro.h
+++ b/drivers/media/platform/verisilicon/hantro.h
@@ -77,6 +77,7 @@ struct hantro_irq {
* @double_buffer: core needs double buffering
* @legacy_regs: core uses legacy register set
* @late_postproc: postproc must be set up at the end of the job
+ * @shared_devices: an array of device ids that cannot run concurrently
*/
struct hantro_variant {
unsigned int enc_offset;
@@ -101,6 +102,7 @@ struct hantro_variant {
unsigned int double_buffer : 1;
unsigned int legacy_regs : 1;
unsigned int late_postproc : 1;
+ const struct of_device_id *shared_devices;
};
/**
diff --git a/drivers/media/platform/verisilicon/hantro_drv.c b/drivers/media/platform/verisilicon/hantro_drv.c
index e0c11fe8b55c..418cfe3a1414 100644
--- a/drivers/media/platform/verisilicon/hantro_drv.c
+++ b/drivers/media/platform/verisilicon/hantro_drv.c
@@ -13,6 +13,7 @@
#include <linux/clk.h>
#include <linux/module.h>
#include <linux/of.h>
+#include <linux/of_platform.h>
#include <linux/platform_device.h>
#include <linux/pm.h>
#include <linux/pm_runtime.h>
@@ -1035,6 +1036,41 @@ static int hantro_disable_multicore(struct hantro_dev *vpu)
return 0;
}
+static struct v4l2_m2m_dev *hantro_get_v4l2_m2m_dev(struct hantro_dev *vpu)
+{
+ struct device_node *node;
+ struct hantro_dev *shared_vpu;
+
+ if (!vpu->variant || !vpu->variant->shared_devices)
+ goto init_new_m2m_dev;
+
+ for_each_matching_node(node, vpu->variant->shared_devices) {
+ struct platform_device *pdev;
+ struct v4l2_m2m_dev *m2m_dev;
+
+ pdev = of_find_device_by_node(node);
+ if (!pdev)
+ continue;
+
+ shared_vpu = platform_get_drvdata(pdev);
+ if (IS_ERR_OR_NULL(shared_vpu) || shared_vpu == vpu) {
+ platform_device_put(pdev);
+ continue;
+ }
+
+ v4l2_m2m_get(shared_vpu->m2m_dev);
+ m2m_dev = shared_vpu->m2m_dev;
+ platform_device_put(pdev);
+
+ of_node_put(node);
+
+ return m2m_dev;
+ }
+
+init_new_m2m_dev:
+ return v4l2_m2m_init(&vpu_m2m_ops);
+}
+
static int hantro_probe(struct platform_device *pdev)
{
const struct of_device_id *match;
@@ -1186,7 +1222,7 @@ static int hantro_probe(struct platform_device *pdev)
}
platform_set_drvdata(pdev, vpu);
- vpu->m2m_dev = v4l2_m2m_init(&vpu_m2m_ops);
+ vpu->m2m_dev = hantro_get_v4l2_m2m_dev(vpu);
if (IS_ERR(vpu->m2m_dev)) {
v4l2_err(&vpu->v4l2_dev, "Failed to init mem2mem device\n");
ret = PTR_ERR(vpu->m2m_dev);
@@ -1225,7 +1261,7 @@ static int hantro_probe(struct platform_device *pdev)
hantro_remove_enc_func(vpu);
err_m2m_rel:
media_device_cleanup(&vpu->mdev);
- v4l2_m2m_release(vpu->m2m_dev);
+ v4l2_m2m_put(vpu->m2m_dev);
err_v4l2_unreg:
v4l2_device_unregister(&vpu->v4l2_dev);
err_clk_unprepare:
@@ -1248,7 +1284,7 @@ static void hantro_remove(struct platform_device *pdev)
hantro_remove_dec_func(vpu);
hantro_remove_enc_func(vpu);
media_device_cleanup(&vpu->mdev);
- v4l2_m2m_release(vpu->m2m_dev);
+ v4l2_m2m_put(vpu->m2m_dev);
v4l2_device_unregister(&vpu->v4l2_dev);
clk_bulk_unprepare(vpu->variant->num_clocks, vpu->clocks);
reset_control_assert(vpu->resets);
diff --git a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
index 5be0e2e76882..fa4224de4b99 100644
--- a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
+++ b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
@@ -343,6 +343,12 @@ const struct hantro_variant imx8mq_vpu_variant = {
.num_regs = ARRAY_SIZE(imx8mq_reg_names)
};
+static const struct of_device_id imx8mq_vpu_shared_resources[] = {
+ { .compatible = "nxp,imx8mq-vpu-g1", },
+ { .compatible = "nxp,imx8mq-vpu-g2", },
+ { /* sentinel */ }
+};
+
const struct hantro_variant imx8mq_vpu_g1_variant = {
.dec_fmts = imx8m_vpu_dec_fmts,
.num_dec_fmts = ARRAY_SIZE(imx8m_vpu_dec_fmts),
@@ -356,6 +362,7 @@ const struct hantro_variant imx8mq_vpu_g1_variant = {
.num_irqs = ARRAY_SIZE(imx8mq_irqs),
.clk_names = imx8mq_g1_clk_names,
.num_clocks = ARRAY_SIZE(imx8mq_g1_clk_names),
+ .shared_devices = imx8mq_vpu_shared_resources,
};
const struct hantro_variant imx8mq_vpu_g2_variant = {
@@ -371,6 +378,7 @@ const struct hantro_variant imx8mq_vpu_g2_variant = {
.num_irqs = ARRAY_SIZE(imx8mq_g2_irqs),
.clk_names = imx8mq_g2_clk_names,
.num_clocks = ARRAY_SIZE(imx8mq_g2_clk_names),
+ .shared_devices = imx8mq_vpu_shared_resources,
};
const struct hantro_variant imx8mm_vpu_g1_variant = {
diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
index 21acd9bc8607..bc8218d1cab9 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -90,6 +90,7 @@ static const char * const m2m_entity_name[] = {
* @job_work: worker to run queued jobs.
* @job_queue_flags: flags of the queue status, %QUEUE_PAUSED.
* @m2m_ops: driver callbacks
+ * @kref: device reference count
*/
struct v4l2_m2m_dev {
struct v4l2_m2m_ctx *curr_ctx;
@@ -109,6 +110,8 @@ struct v4l2_m2m_dev {
unsigned long job_queue_flags;
const struct v4l2_m2m_ops *m2m_ops;
+
+ struct kref kref;
};
static struct v4l2_m2m_queue_ctx *get_queue_ctx(struct v4l2_m2m_ctx *m2m_ctx,
@@ -1206,6 +1209,7 @@ struct v4l2_m2m_dev *v4l2_m2m_init(const struct v4l2_m2m_ops *m2m_ops)
INIT_LIST_HEAD(&m2m_dev->job_queue);
spin_lock_init(&m2m_dev->job_spinlock);
INIT_WORK(&m2m_dev->job_work, v4l2_m2m_device_run_work);
+ kref_init(&m2m_dev->kref);
return m2m_dev;
}
@@ -1217,6 +1221,25 @@ void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev)
}
EXPORT_SYMBOL_GPL(v4l2_m2m_release);
+void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev)
+{
+ kref_get(&m2m_dev->kref);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_get);
+
+static void v4l2_m2m_release_from_kref(struct kref *kref)
+{
+ struct v4l2_m2m_dev *m2m_dev = container_of(kref, struct v4l2_m2m_dev, kref);
+
+ v4l2_m2m_release(m2m_dev);
+}
+
+void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev)
+{
+ kref_put(&m2m_dev->kref, v4l2_m2m_release_from_kref);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_put);
+
struct v4l2_m2m_ctx *v4l2_m2m_ctx_init(struct v4l2_m2m_dev *m2m_dev,
void *drv_priv,
int (*queue_init)(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq))
diff --git a/drivers/memstick/core/memstick.c b/drivers/memstick/core/memstick.c
index acafc910bbac..e4275f8ee5db 100644
--- a/drivers/memstick/core/memstick.c
+++ b/drivers/memstick/core/memstick.c
@@ -370,9 +370,7 @@ int memstick_set_rw_addr(struct memstick_dev *card)
{
card->next_request = h_memstick_set_rw_addr;
memstick_new_req(card->host);
- if (!wait_for_completion_timeout(&card->mrq_complete,
- msecs_to_jiffies(500)))
- card->current_mrq.error = -ETIMEDOUT;
+ wait_for_completion(&card->mrq_complete);
return card->current_mrq.error;
}
@@ -406,9 +404,7 @@ static struct memstick_dev *memstick_alloc_card(struct memstick_host *host)
card->next_request = h_memstick_read_dev_id;
memstick_new_req(host);
- if (!wait_for_completion_timeout(&card->mrq_complete,
- msecs_to_jiffies(500)))
- card->current_mrq.error = -ETIMEDOUT;
+ wait_for_completion(&card->mrq_complete);
if (card->current_mrq.error)
goto err_out;
diff --git a/drivers/memstick/host/rtsx_usb_ms.c b/drivers/memstick/host/rtsx_usb_ms.c
index beadc389f15f..d5b3a96fc609 100644
--- a/drivers/memstick/host/rtsx_usb_ms.c
+++ b/drivers/memstick/host/rtsx_usb_ms.c
@@ -27,7 +27,6 @@ struct rtsx_usb_ms {
struct memstick_host *msh;
struct memstick_request *req;
- struct mutex host_mutex;
struct work_struct handle_req;
struct delayed_work poll_card;
@@ -514,6 +513,13 @@ static void rtsx_usb_ms_handle_req(struct work_struct *work)
struct memstick_host *msh = host->msh;
int rc;
+ /* Fail requests after eject so their waiters are released. */
+ if (host->eject) {
+ while (!memstick_next_req(msh, &host->req))
+ host->req->error = -ENOMEDIUM;
+ return;
+ }
+
if (!host->req) {
pm_runtime_get_sync(ms_dev(host));
do {
@@ -547,8 +553,7 @@ static void rtsx_usb_ms_request(struct memstick_host *msh)
dev_dbg(ms_dev(host), "--> %s\n", __func__);
- if (!host->eject)
- schedule_work(&host->handle_req);
+ schedule_work(&host->handle_req);
}
static int rtsx_usb_ms_set_param(struct memstick_host *msh,
@@ -781,7 +786,6 @@ static int rtsx_usb_ms_drv_probe(struct platform_device *pdev)
host->power_mode = MEMSTICK_POWER_OFF;
platform_set_drvdata(pdev, host);
- mutex_init(&host->host_mutex);
INIT_WORK(&host->handle_req, rtsx_usb_ms_handle_req);
INIT_DELAYED_WORK(&host->poll_card, rtsx_usb_ms_poll_card);
@@ -812,27 +816,12 @@ static void rtsx_usb_ms_drv_remove(struct platform_device *pdev)
{
struct rtsx_usb_ms *host = platform_get_drvdata(pdev);
struct memstick_host *msh = host->msh;
- int err;
host->eject = true;
msh->removing = true;
- cancel_work_sync(&host->handle_req);
+ flush_work(&host->handle_req);
cancel_delayed_work_sync(&host->poll_card);
- mutex_lock(&host->host_mutex);
- if (host->req) {
- dev_dbg(ms_dev(host),
- "%s: Controller removed during transfer\n",
- dev_name(&msh->dev));
- host->req->error = -ENOMEDIUM;
- do {
- err = memstick_next_req(msh, &host->req);
- if (!err)
- host->req->error = -ENOMEDIUM;
- } while (!err);
- }
- mutex_unlock(&host->host_mutex);
-
/* Balance possible unbalanced usage count
* e.g. unconditional module removal
*/
@@ -841,6 +830,8 @@ static void rtsx_usb_ms_drv_remove(struct platform_device *pdev)
pm_runtime_disable(ms_dev(host));
memstick_remove_host(msh);
+ /* No card, no new requests; wait for the last failed one to finish. */
+ cancel_work_sync(&host->handle_req);
dev_dbg(ms_dev(host),
": Realtek USB Memstick controller has been removed\n");
memstick_free_host(msh);
diff --git a/drivers/mmc/host/cavium-octeon.c b/drivers/mmc/host/cavium-octeon.c
index 0592f356b1e5..b522abe4ffdb 100644
--- a/drivers/mmc/host/cavium-octeon.c
+++ b/drivers/mmc/host/cavium-octeon.c
@@ -302,9 +302,12 @@ static void octeon_mmc_remove(struct platform_device *pdev)
u64 dma_cfg;
int i;
- for (i = 0; i < CAVIUM_MAX_MMC; i++)
+ for (i = 0; i < CAVIUM_MAX_MMC; i++) {
if (host->slot[i])
cvm_mmc_of_slot_remove(host->slot[i]);
+ if (host->slot_pdev[i])
+ of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL);
+ }
dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host));
dma_cfg &= ~MIO_EMM_DMA_CFG_EN;
diff --git a/drivers/mmc/host/cavium-thunderx.c b/drivers/mmc/host/cavium-thunderx.c
index 1373deb3f531..e8a0f9976be1 100644
--- a/drivers/mmc/host/cavium-thunderx.c
+++ b/drivers/mmc/host/cavium-thunderx.c
@@ -173,9 +173,12 @@ static void thunder_mmc_remove(struct pci_dev *pdev)
u64 dma_cfg;
int i;
- for (i = 0; i < CAVIUM_MAX_MMC; i++)
+ for (i = 0; i < CAVIUM_MAX_MMC; i++) {
if (host->slot[i])
cvm_mmc_of_slot_remove(host->slot[i]);
+ if (host->slot_pdev[i])
+ of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL);
+ }
dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host));
dma_cfg &= ~MIO_EMM_DMA_CFG_EN;
diff --git a/drivers/mmc/host/mtk-sd.c b/drivers/mmc/host/mtk-sd.c
index 79074291e9d2..69717efeffaa 100644
--- a/drivers/mmc/host/mtk-sd.c
+++ b/drivers/mmc/host/mtk-sd.c
@@ -3194,6 +3194,7 @@ static void msdc_drv_remove(struct platform_device *pdev)
platform_set_drvdata(pdev, NULL);
mmc_remove_host(mmc);
+ cancel_delayed_work_sync(&host->req_timeout);
msdc_deinit_hw(host);
msdc_gate_clock(host);
diff --git a/drivers/mmc/host/sdhci-sprd.c b/drivers/mmc/host/sdhci-sprd.c
index 3584a2b314a9..0c8f389fe584 100644
--- a/drivers/mmc/host/sdhci-sprd.c
+++ b/drivers/mmc/host/sdhci-sprd.c
@@ -890,6 +890,10 @@ static void sdhci_sprd_remove(struct platform_device *pdev)
struct sdhci_host *host = platform_get_drvdata(pdev);
struct sdhci_sprd_host *sprd_host = TO_SPRD_HOST(host);
+ pm_runtime_get_sync(&pdev->dev);
+ pm_runtime_disable(&pdev->dev);
+ pm_runtime_put_noidle(&pdev->dev);
+
sdhci_remove_host(host, 0);
clk_disable_unprepare(sprd_host->clk_sdio);
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 01d2223fce09..3f4e97476292 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -80,15 +80,6 @@ static struct in6_addr mld2_all_node = MLD2_ALL_NODE_INIT;
static struct mld2_grec mldv2_zero_grec;
#endif
-static struct amt_skb_cb *amt_skb_cb(struct sk_buff *skb)
-{
- BUILD_BUG_ON(sizeof(struct amt_skb_cb) + sizeof(struct tc_skb_cb) >
- sizeof_field(struct sk_buff, cb));
-
- return (struct amt_skb_cb *)((void *)skb->cb +
- sizeof(struct tc_skb_cb));
-}
-
static void __amt_source_gc_work(void)
{
struct amt_source_node *snode;
@@ -789,6 +780,11 @@ static void amt_send_request(struct amt_dev *amt, bool v6)
rcu_read_unlock();
}
+static bool amt_send_membership_query(struct amt_dev *amt,
+ struct sk_buff *skb,
+ struct amt_tunnel_list *tunnel,
+ bool v6);
+
static void amt_send_igmp_gq(struct amt_dev *amt,
struct amt_tunnel_list *tunnel)
{
@@ -798,8 +794,11 @@ static void amt_send_igmp_gq(struct amt_dev *amt,
if (!skb)
return;
- amt_skb_cb(skb)->tunnel = tunnel;
- dev_queue_xmit(skb);
+ skb_pull(skb, sizeof(struct ethhdr));
+ if (amt_send_membership_query(amt, skb, tunnel, false)) {
+ amt->dev->stats.tx_dropped++;
+ kfree_skb(skb);
+ }
}
#if IS_ENABLED(CONFIG_IPV6)
@@ -883,8 +882,11 @@ static void amt_send_mld_gq(struct amt_dev *amt, struct amt_tunnel_list *tunnel)
if (!skb)
return;
- amt_skb_cb(skb)->tunnel = tunnel;
- dev_queue_xmit(skb);
+ skb_pull(skb, sizeof(struct ethhdr));
+ if (amt_send_membership_query(amt, skb, tunnel, true)) {
+ amt->dev->stats.tx_dropped++;
+ kfree_skb(skb);
+ }
}
#else
static void amt_send_mld_gq(struct amt_dev *amt, struct amt_tunnel_list *tunnel)
@@ -1183,7 +1185,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
#endif
bool report = false;
struct igmphdr *ih;
- bool query = false;
struct iphdr *iph;
bool data = false;
bool v6 = false;
@@ -1201,9 +1202,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
case IGMP_HOST_MEMBERSHIP_REPORT:
report = true;
break;
- case IGMP_HOST_MEMBERSHIP_QUERY:
- query = true;
- break;
default:
goto free;
}
@@ -1225,9 +1223,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
case ICMPV6_MLD2_REPORT:
report = true;
break;
- case ICMPV6_MGM_QUERY:
- query = true;
- break;
default:
goto free;
}
@@ -1258,19 +1253,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
goto free;
goto unlock;
} else if (amt->mode == AMT_MODE_RELAY) {
- if (query) {
- tunnel = amt_skb_cb(skb)->tunnel;
- if (!tunnel) {
- WARN_ON(1);
- goto free;
- }
-
- /* Do not forward unexpected query */
- if (amt_send_membership_query(amt, skb, tunnel, v6))
- goto free;
- goto unlock;
- }
-
if (!data)
goto free;
list_for_each_entry_rcu(tunnel, &amt->tunnel_list, list) {
diff --git a/drivers/net/can/dev/skb.c b/drivers/net/can/dev/skb.c
index 0da615afa04d..3ebd4f779b9b 100644
--- a/drivers/net/can/dev/skb.c
+++ b/drivers/net/can/dev/skb.c
@@ -202,6 +202,7 @@ static void init_can_skb_reserve(struct sk_buff *skb)
skb_reset_transport_header(skb);
can_skb_reserve(skb);
+ can_skb_prv(skb)->skbcnt = 0;
}
struct sk_buff *alloc_can_skb(struct net_device *dev, struct can_frame **cf)
@@ -311,6 +312,7 @@ static bool can_skb_headroom_valid(struct net_device *dev, struct sk_buff *skb)
if (skb->ip_summed == CHECKSUM_NONE) {
/* init headroom */
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb->ip_summed = CHECKSUM_UNNECESSARY;
diff --git a/drivers/net/can/usb/f81604.c b/drivers/net/can/usb/f81604.c
index 6b8b2795c018..63d365fa5cfc 100644
--- a/drivers/net/can/usb/f81604.c
+++ b/drivers/net/can/usb/f81604.c
@@ -169,7 +169,7 @@ struct f81604_int_data {
u8 rxerr;
u8 txerr;
u8 val;
-} __packed __aligned(4);
+} __packed;
struct f81604_sff {
__be16 id;
diff --git a/drivers/net/dsa/microchip/ksz_common.h b/drivers/net/dsa/microchip/ksz_common.h
index c65188cd3c0a..04ec05847395 100644
--- a/drivers/net/dsa/microchip/ksz_common.h
+++ b/drivers/net/dsa/microchip/ksz_common.h
@@ -775,7 +775,7 @@ static inline bool ksz_is_sgmii_port(struct ksz_device *dev, int port)
#define KSZ88_FAMILY_ID 0x88
#define KSZ8895_FAMILY_ID 0x95
-#define KSZ8_PORT_STATUS_0 0x08
+#define KSZ8_PORT_STATUS_0 0x18
#define KSZ8_PORT_FIBER_MODE BIT(7)
#define SW_CHIP_ID_M GENMASK(7, 4)
diff --git a/drivers/net/dsa/mt7530.c b/drivers/net/dsa/mt7530.c
index 4001779a9159..bd0a37ead0de 100644
--- a/drivers/net/dsa/mt7530.c
+++ b/drivers/net/dsa/mt7530.c
@@ -25,6 +25,9 @@
#include "mt7530.h"
+#define MT7530_STATS_POLL_INTERVAL (1 * HZ)
+#define MT7530_STATS_RATE_LIMIT (HZ / 10)
+
static struct mt753x_pcs *pcs_to_mt753x_pcs(struct phylink_pcs *pcs)
{
return container_of(pcs, struct mt753x_pcs, pcs);
@@ -888,10 +891,9 @@ static void mt7530_get_rmon_stats(struct dsa_switch *ds, int port,
*ranges = mt7530_rmon_ranges;
}
-static void mt7530_get_stats64(struct dsa_switch *ds, int port,
- struct rtnl_link_stats64 *storage)
+static void mt7530_read_port_stats64(struct mt7530_priv *priv, int port,
+ struct rtnl_link_stats64 *storage)
{
- struct mt7530_priv *priv = ds->priv;
uint64_t data;
/* MIB counter doesn't provide a FramesTransmittedOK but instead
@@ -933,6 +935,54 @@ static void mt7530_get_stats64(struct dsa_switch *ds, int port,
&storage->rx_crc_errors);
}
+static void mt7530_stats_refresh(struct mt7530_priv *priv)
+{
+ struct rtnl_link_stats64 stats = {};
+ struct dsa_port *dp;
+ int port;
+
+ dsa_switch_for_each_user_port(dp, priv->ds) {
+ port = dp->index;
+
+ mt7530_read_port_stats64(priv, port, &stats);
+
+ spin_lock_bh(&priv->stats_lock);
+ priv->ports[port].stats = stats;
+ priv->stats_last = jiffies;
+ spin_unlock_bh(&priv->stats_lock);
+ }
+}
+
+static void mt7530_stats_poll(struct work_struct *work)
+{
+ struct mt7530_priv *priv = container_of(work, struct mt7530_priv,
+ stats_work.work);
+
+ mt7530_stats_refresh(priv);
+ schedule_delayed_work(&priv->stats_work,
+ MT7530_STATS_POLL_INTERVAL);
+}
+
+static void mt7530_get_stats64(struct dsa_switch *ds, int port,
+ struct rtnl_link_stats64 *storage)
+{
+ struct mt7530_priv *priv = ds->priv;
+ bool refresh;
+
+ if (priv->bus) {
+ spin_lock_bh(&priv->stats_lock);
+ *storage = priv->ports[port].stats;
+ refresh = time_after(jiffies, priv->stats_last +
+ MT7530_STATS_RATE_LIMIT);
+ spin_unlock_bh(&priv->stats_lock);
+ if (refresh)
+ mod_delayed_work(system_percpu_wq,
+ &priv->stats_work, 0);
+ } else {
+ mt7530_read_port_stats64(priv, port, storage);
+ }
+}
+
static void mt7530_get_eth_ctrl_stats(struct dsa_switch *ds, int port,
struct ethtool_eth_ctrl_stats *ctrl_stats)
{
@@ -3169,9 +3219,24 @@ mt753x_setup(struct dsa_switch *ds)
if (ret && priv->irq_domain)
mt7530_free_mdio_irq(priv);
+ if (!ret && priv->bus) {
+ mt7530_stats_refresh(priv);
+ schedule_delayed_work(&priv->stats_work,
+ MT7530_STATS_POLL_INTERVAL);
+ }
+
return ret;
}
+static void
+mt753x_teardown(struct dsa_switch *ds)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ if (priv->bus)
+ cancel_delayed_work_sync(&priv->stats_work);
+}
+
static int mt753x_set_mac_eee(struct dsa_switch *ds, int port,
struct ethtool_keee *e)
{
@@ -3290,6 +3355,7 @@ static int mt7988_setup(struct dsa_switch *ds)
const struct dsa_switch_ops mt7530_switch_ops = {
.get_tag_protocol = mtk_get_tag_protocol,
.setup = mt753x_setup,
+ .teardown = mt753x_teardown,
.preferred_default_local_cpu_port = mt753x_preferred_default_local_cpu_port,
.get_strings = mt7530_get_strings,
.get_ethtool_stats = mt7530_get_ethtool_stats,
@@ -3437,6 +3503,9 @@ mt7530_probe_common(struct mt7530_priv *priv)
priv->ds->ops = &mt7530_switch_ops;
priv->ds->phylink_mac_ops = &mt753x_phylink_mac_ops;
mutex_init(&priv->reg_mutex);
+ spin_lock_init(&priv->stats_lock);
+ INIT_DELAYED_WORK(&priv->stats_work, mt7530_stats_poll);
+
dev_set_drvdata(dev, priv);
return 0;
diff --git a/drivers/net/dsa/mt7530.h b/drivers/net/dsa/mt7530.h
index 69be713bab80..ca665f87f1f6 100644
--- a/drivers/net/dsa/mt7530.h
+++ b/drivers/net/dsa/mt7530.h
@@ -797,6 +797,7 @@ struct mt7530_fdb {
* @pvid: The VLAN specified is to be considered a PVID at ingress. Any
* untagged frames will be assigned to the related VLAN.
* @sgmii_pcs: Pointer to PCS instance for SerDes ports
+ * @stats: Cached port statistics for MDIO-connected switches
*/
struct mt7530_port {
bool enable;
@@ -804,6 +805,7 @@ struct mt7530_port {
u32 pm;
u16 pvid;
struct phylink_pcs *sgmii_pcs;
+ struct rtnl_link_stats64 stats;
};
/* Port 5 mode definitions of the MT7530 switch */
@@ -876,6 +878,9 @@ struct mt753x_info {
* @create_sgmii: Pointer to function creating SGMII PCS instance(s)
* @active_cpu_ports: Holding the active CPU ports
* @mdiodev: The pointer to the MDIO device structure
+ * @stats_lock: Protects cached per-port stats from concurrent access
+ * @stats_work: Delayed work for polling MIB counters on MDIO switches
+ * @stats_last: Jiffies timestamp of last MIB counter poll
*/
struct mt7530_priv {
struct device *dev;
@@ -901,6 +906,9 @@ struct mt7530_priv {
int (*create_sgmii)(struct mt7530_priv *priv);
u8 active_cpu_ports;
struct mdio_device *mdiodev;
+ spinlock_t stats_lock; /* protects cached stats counters */
+ struct delayed_work stats_work;
+ unsigned long stats_last;
};
struct mt7530_hw_vlan_entry {
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 52cb2a25600b..602b97d00e9a 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -474,7 +474,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
struct netdev_queue *txq;
int i;
dma_addr_t mapping;
- unsigned int length, pad = 0;
+ unsigned int length;
u32 len, free_size, vlan_tag_flags, cfa_action, flags;
struct bnxt_ptp_cfg *ptp = bp->ptp_cfg;
struct pci_dev *pdev = bp->pdev;
@@ -503,6 +503,14 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
goto tx_free;
}
#endif
+ /* Pad before skb_shinfo(skb)->nr_frags is sampled, because
+ * skb_put_padto() might linearize the skb.
+ */
+ if (skb_put_padto(skb, BNXT_MIN_PKT_SIZE)) {
+ /* SKB already freed. */
+ goto tx_kick_pending;
+ }
+
free_size = bnxt_tx_avail(bp, txr);
if (unlikely(free_size < skb_shinfo(skb)->nr_frags + 2)) {
/* We must have raced with NAPI cleanup */
@@ -642,14 +650,6 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
}
normal_tx:
- if (length < BNXT_MIN_PKT_SIZE) {
- pad = BNXT_MIN_PKT_SIZE - length;
- if (skb_pad(skb, pad))
- /* SKB already freed. */
- goto tx_kick_pending;
- length = BNXT_MIN_PKT_SIZE;
- }
-
mapping = dma_map_single(&pdev->dev, skb->data, len, DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(&pdev->dev, mapping)))
@@ -732,10 +732,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
txbd->tx_bd_len_flags_type = cpu_to_le32(flags);
}
- flags &= ~TX_BD_LEN;
- txbd->tx_bd_len_flags_type =
- cpu_to_le32(((len + pad) << TX_BD_LEN_SHIFT) | flags |
- TX_BD_FLAGS_PACKET_END);
+ txbd->tx_bd_len_flags_type |= cpu_to_le32(TX_BD_FLAGS_PACKET_END);
netdev_tx_sent_queue(txq, skb->len);
@@ -6897,7 +6894,8 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, struct bnxt_vnic_info *vnic)
#endif
if ((bp->flags & BNXT_FLAG_STRIP_VLAN) || def_vlan)
req->flags |= cpu_to_le32(VNIC_CFG_REQ_FLAGS_VLAN_STRIP_MODE);
- if (vnic->vnic_id == BNXT_VNIC_DEFAULT && bnxt_ulp_registered(bp->edev))
+ if (vnic->vnic_id == BNXT_VNIC_DEFAULT &&
+ bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA]))
req->flags |= cpu_to_le32(bnxt_get_roce_vnic_mode(bp));
return hwrm_req_send(bp, req);
@@ -8015,6 +8013,7 @@ static int bnxt_get_avail_msix(struct bnxt *bp, int num);
static int __bnxt_reserve_rings(struct bnxt *bp)
{
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
struct bnxt_hw_rings hwr = {0};
int rx_rings, old_rx_rings, rc;
int cp = bp->cp_nr_rings;
@@ -8025,7 +8024,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
if (!bnxt_need_reserve_rings(bp))
return 0;
- if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(bp->edev)) {
+ if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(edev)) {
ulp_msix = bnxt_get_avail_msix(bp, bp->ulp_num_msix_want);
if (!ulp_msix)
bnxt_set_ulp_stat_ctxs(bp, 0);
@@ -8078,8 +8077,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
}
rx_rings = min_t(int, rx_rings, hwr.grp);
hwr.cp = min_t(int, hwr.cp, bp->cp_nr_rings);
- if (bnxt_ulp_registered(bp->edev) &&
- hwr.stat > bnxt_get_ulp_stat_ctxs(bp))
+ if (bnxt_ulp_registered(edev) && hwr.stat > bnxt_get_ulp_stat_ctxs(bp))
hwr.stat -= bnxt_get_ulp_stat_ctxs(bp);
hwr.cp = min_t(int, hwr.cp, hwr.stat);
rc = bnxt_trim_rings(bp, &rx_rings, &hwr.tx, hwr.cp, sh);
@@ -8118,7 +8116,7 @@ static int __bnxt_reserve_rings(struct bnxt *bp)
!netif_is_rxfh_configured(bp->dev))
bnxt_set_dflt_rss_indir_tbl(bp, NULL);
- if (!bnxt_ulp_registered(bp->edev) && BNXT_NEW_RM(bp)) {
+ if (!bnxt_ulp_registered(edev) && BNXT_NEW_RM(bp)) {
int resv_msix, resv_ctx, ulp_ctxs;
struct bnxt_hw_resc *hw_resc;
@@ -11517,6 +11515,7 @@ static void bnxt_clear_int_mode(struct bnxt *bp)
int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init)
{
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
bool irq_cleared = false;
bool irq_change = false;
int tcs = bp->num_tc;
@@ -11526,7 +11525,7 @@ int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init)
if (!bnxt_need_reserve_rings(bp))
return 0;
- if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(bp->edev)) {
+ if (BNXT_NEW_RM(bp) && !bnxt_ulp_registered(edev)) {
int ulp_msix = bnxt_get_avail_msix(bp, bp->ulp_num_msix_want);
if (ulp_msix > bp->ulp_num_msix_want)
@@ -11648,12 +11647,13 @@ static void bnxt_irq_affinity_notify(struct irq_affinity_notify *notify,
{
struct bnxt_irq *irq;
u16 tag;
- int err;
irq = container_of(notify, struct bnxt_irq, affinity_notify);
#ifdef CONFIG_RFS_ACCEL
if (irq->bp->dev->rx_cpu_rmap && irq->ring_nr < irq->bp->rx_nr_rings) {
+ int err;
+
err = cpu_rmap_update(irq->bp->dev->rx_cpu_rmap, irq->ring_nr,
mask);
if (err)
@@ -11671,20 +11671,11 @@ static void bnxt_irq_affinity_notify(struct irq_affinity_notify *notify,
return;
if (pcie_tph_get_cpu_st(irq->bp->pdev, TPH_MEM_TYPE_VM,
- cpumask_first(irq->cpu_mask), &tag))
+ cpumask_first(mask), &tag))
return;
- if (pcie_tph_set_st_entry(irq->bp->pdev, irq->msix_nr, tag))
- return;
-
- netdev_lock(irq->bp->dev);
- if (netif_running(irq->bp->dev)) {
- err = netdev_rx_queue_restart(irq->bp->dev, irq->ring_nr);
- if (err)
- netdev_err(irq->bp->dev,
- "RX queue restart failed: err=%d\n", err);
- }
- netdev_unlock(irq->bp->dev);
+ WRITE_ONCE(irq->new_tag, tag);
+ bnxt_queue_sp_work(irq->bp, BNXT_TPH_UPDATE_SP_EVENT);
}
static void bnxt_irq_affinity_release(struct kref *ref)
@@ -11755,6 +11746,8 @@ static void bnxt_free_irq(struct bnxt *bp)
}
irq->requested = 0;
+ irq->tag = 0;
+ irq->new_tag = 0;
}
/* Disable TPH support */
@@ -11769,6 +11762,7 @@ static void bnxt_free_irq(struct bnxt *bp)
static int bnxt_request_irq(struct bnxt *bp)
{
+ const int numa_node = dev_to_node(&bp->pdev->dev);
struct cpu_rmap *rmap = NULL;
int i, j, rc = 0;
unsigned long flags = 0;
@@ -11793,6 +11787,8 @@ static int bnxt_request_irq(struct bnxt *bp)
for (i = 0, j = 0; i < bp->cp_nr_rings; i++) {
int map_idx = bnxt_cp_num_to_irq_num(bp, i);
struct bnxt_irq *irq = &bp->irq_tbl[map_idx];
+ unsigned int cpu_num;
+ u16 tag;
if (IS_ENABLED(CONFIG_RFS_ACCEL) &&
rmap && bp->bnapi[i]->rx_ring) {
@@ -11811,32 +11807,31 @@ static int bnxt_request_irq(struct bnxt *bp)
netif_napi_set_irq_locked(&bp->bnapi[i]->napi, irq->vector);
irq->requested = 1;
- if (zalloc_cpumask_var(&irq->cpu_mask, GFP_KERNEL)) {
- int numa_node = dev_to_node(&bp->pdev->dev);
- u16 tag;
-
- irq->have_cpumask = 1;
- irq->msix_nr = map_idx;
- irq->ring_nr = i;
- cpumask_set_cpu(cpumask_local_spread(i, numa_node),
- irq->cpu_mask);
- rc = irq_update_affinity_hint(irq->vector, irq->cpu_mask);
- if (rc) {
- netdev_warn(bp->dev,
- "Update affinity hint failed, IRQ = %d\n",
- irq->vector);
- break;
- }
+ if (!zalloc_cpumask_var(&irq->cpu_mask, GFP_KERNEL))
+ continue;
- bnxt_register_irq_notifier(bp, irq);
+ irq->have_cpumask = 1;
+ irq->msix_nr = map_idx;
+ irq->ring_nr = i;
+ cpu_num = cpumask_local_spread(i, numa_node);
+ cpumask_set_cpu(cpu_num, irq->cpu_mask);
+
+ /* Init ST table entry if we can get the mapping */
+ if (!pcie_tph_get_cpu_st(bp->pdev, TPH_MEM_TYPE_VM,
+ cpu_num, &tag)) {
+ pcie_tph_set_st_entry(bp->pdev, irq->msix_nr, tag);
+ irq->tag = tag;
+ irq->new_tag = tag;
+ }
- /* Init ST table entry */
- if (pcie_tph_get_cpu_st(irq->bp->pdev, TPH_MEM_TYPE_VM,
- cpumask_first(irq->cpu_mask),
- &tag))
- continue;
+ bnxt_register_irq_notifier(bp, irq);
- pcie_tph_set_st_entry(irq->bp->pdev, irq->msix_nr, tag);
+ rc = irq_update_affinity_hint(irq->vector, irq->cpu_mask);
+ if (rc) {
+ netdev_warn(bp->dev,
+ "Update affinity hint failed, IRQ = %d\n",
+ irq->vector);
+ break;
}
}
return rc;
@@ -14302,6 +14297,43 @@ static void bnxt_rtnl_unlock_sp(struct bnxt *bp)
rtnl_unlock();
}
+static void bnxt_tph_update(struct bnxt *bp)
+{
+ struct net_device *dev = bp->dev;
+ int i;
+
+ bnxt_rtnl_lock_sp(bp);
+ if (!test_bit(BNXT_STATE_OPEN, &bp->state))
+ goto unlock;
+
+ for (i = 0; i < bp->rx_nr_rings; i++) {
+ struct bnxt_irq *irq;
+ int map_idx, err;
+ u16 tag;
+
+ map_idx = bnxt_cp_num_to_irq_num(bp, i);
+ irq = &bp->irq_tbl[map_idx];
+ tag = READ_ONCE(irq->new_tag);
+ if (irq->tag == tag)
+ continue;
+
+ if (pcie_tph_set_st_entry(bp->pdev, irq->msix_nr, tag))
+ continue;
+
+ err = netdev_rx_queue_restart(dev, irq->ring_nr);
+ if (err) {
+ netdev_err(dev, "RX queue restart failed: err=%d\n",
+ err);
+ continue;
+ }
+
+ irq->tag = tag;
+ }
+
+unlock:
+ bnxt_rtnl_unlock_sp(bp);
+}
+
/* Only called from bnxt_sp_task() */
static void bnxt_reset(struct bnxt *bp, bool silent)
{
@@ -14629,7 +14661,7 @@ static void bnxt_fw_echo_reply(struct bnxt *bp)
static void bnxt_ulp_restart(struct bnxt *bp)
{
bnxt_ulp_stop(bp);
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
}
static void bnxt_sp_task(struct work_struct *work)
@@ -14740,6 +14772,9 @@ static void bnxt_sp_task(struct work_struct *work)
bnxt_devlink_health_fw_report(bp);
}
+ if (test_and_clear_bit(BNXT_TPH_UPDATE_SP_EVENT, &bp->sp_event))
+ bnxt_tph_update(bp);
+
smp_mb__before_atomic();
clear_bit(BNXT_STATE_IN_SP_TASK, &bp->state);
}
@@ -14789,7 +14824,7 @@ int bnxt_check_rings(struct bnxt *bp, int tx, int rx, bool sh, int tcs,
hwr.cp_p5 = hwr.tx + rx;
rc = bnxt_hwrm_check_rings(bp, &hwr);
if (!rc && pci_msix_can_alloc_dyn(bp->pdev)) {
- if (!bnxt_ulp_registered(bp->edev)) {
+ if (!bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA])) {
hwr.cp += bnxt_get_ulp_msix_num(bp);
hwr.cp = min_t(int, hwr.cp, bnxt_get_max_func_irqs(bp));
}
@@ -15258,7 +15293,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
if (test_and_clear_bit(BNXT_STATE_FW_ACTIVATE_RESET, &bp->state) &&
!test_bit(BNXT_STATE_FW_ACTIVATE, &bp->state))
bnxt_dl_remote_reload(bp);
- if (pci_enable_device(bp->pdev)) {
+ if (!pci_is_enabled(bp->pdev) && pci_enable_device(bp->pdev)) {
netdev_err(bp->dev, "Cannot re-enable PCI device\n");
rc = -ENODEV;
goto fw_reset_abort;
@@ -15312,7 +15347,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
}
netdev_unlock(bp->dev);
rtnl_unlock();
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
bnxt_reenable_sriov(bp);
netdev_lock(bp->dev);
bnxt_vf_reps_alloc(bp);
@@ -15334,7 +15369,8 @@ static void bnxt_fw_reset_task(struct work_struct *work)
bnxt_fw_reset_abort(bp, rc);
netdev_unlock(bp->dev);
ulp_start:
- bnxt_ulp_start(bp, rc);
+ if (!rc)
+ bnxt_ulp_start(bp);
}
static int bnxt_init_board(struct pci_dev *pdev, struct net_device *dev)
@@ -16381,12 +16417,13 @@ static void bnxt_remove_one(struct pci_dev *pdev)
if (BNXT_PF(bp))
__bnxt_sriov_disable(bp);
- bnxt_rdma_aux_device_del(bp);
+ bnxt_aux_devices_del(bp);
unregister_netdev(dev);
bnxt_ptp_clear(bp);
- bnxt_rdma_aux_device_uninit(bp);
+ bnxt_aux_devices_uninit(bp);
+ bnxt_auxdev_id_free(bp, bp->auxdev_id);
bnxt_free_l2_filters(bp, true);
bnxt_free_ntp_fltrs(bp, true);
@@ -16795,6 +16832,51 @@ void bnxt_print_device_info(struct bnxt *bp)
pcie_print_link_status(bp->pdev);
}
+static void bnxt_clear_bars(struct pci_dev *pdev)
+{
+ int off;
+
+ for (off = PCI_BASE_ADDRESS_0; off <= PCI_BASE_ADDRESS_5; off += 4)
+ pci_write_config_dword(pdev, off, 0);
+}
+
+/* Clear any pending DMA transactions from crash kernel while loading driver in
+ * capture kernel.
+ */
+static int bnxt_kdump_reset(struct pci_dev *pdev)
+{
+ int rc, i;
+ u16 cmd;
+
+ pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+ cmd &= ~(PCI_COMMAND_MASTER | PCI_COMMAND_MEMORY);
+ pci_write_config_word(pdev, PCI_COMMAND, cmd);
+
+ if (pci_save_state(pdev))
+ dev_warn(&pdev->dev, "Failed to save PCI state, PCI restore may be incomplete\n");
+
+ rc = pcie_flr(pdev);
+ if (rc)
+ dev_warn(&pdev->dev, "pcie_flr() failed (rc: %d), trying to continue\n",
+ rc);
+
+ /* In case device is not returning CRS, wait 5 seconds longer */
+ for (i = 0; i < 50; i++) {
+ pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+ if (!PCI_POSSIBLE_ERROR(cmd))
+ break;
+ msleep(100);
+ }
+ if (PCI_POSSIBLE_ERROR(cmd)) {
+ dev_err(&pdev->dev, "PCI config space inaccessible after FLR, aborting\n");
+ return -ENODEV;
+ }
+
+ bnxt_clear_bars(pdev);
+ pci_restore_state(pdev);
+ return 0;
+}
+
static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
{
struct bnxt_hw_resc *hw_resc;
@@ -16810,12 +16892,10 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
return -ENODEV;
}
- /* Clear any pending DMA transactions from crash kernel
- * while loading driver in capture kernel.
- */
if (is_kdump_kernel()) {
- pci_clear_master(pdev);
- pcie_flr(pdev);
+ rc = bnxt_kdump_reset(pdev);
+ if (rc)
+ return rc;
}
max_irqs = bnxt_get_max_irq(pdev);
@@ -16972,7 +17052,9 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
bnxt_set_tpa_flags(bp);
bnxt_init_ring_params(bp);
bnxt_set_ring_params(bp);
- bnxt_rdma_aux_device_init(bp);
+ mutex_init(&bp->auxdev_lock);
+ if (!bnxt_auxdev_id_alloc(bp))
+ bnxt_aux_devices_init(bp);
rc = bnxt_set_dflt_rings(bp, true);
if (rc) {
if (BNXT_VF(bp) && rc == -ENODEV) {
@@ -17037,7 +17119,7 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
bnxt_dl_fw_reporters_create(bp);
- bnxt_rdma_aux_device_add(bp);
+ bnxt_aux_devices_add(bp);
bnxt_print_device_info(bp);
@@ -17045,7 +17127,8 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
return 0;
init_err_cleanup:
- bnxt_rdma_aux_device_uninit(bp);
+ bnxt_aux_devices_uninit(bp);
+ bnxt_auxdev_id_free(bp, bp->auxdev_id);
bnxt_dl_unregister(bp);
init_err_dl:
bnxt_shutdown_tc(bp);
@@ -17181,9 +17264,10 @@ static int bnxt_resume(struct device *device)
resume_exit:
netdev_unlock(bp->dev);
rtnl_unlock();
- bnxt_ulp_start(bp, rc);
- if (!rc)
+ if (!rc) {
+ bnxt_ulp_start(bp);
bnxt_reenable_sriov(bp);
+ }
return rc;
}
@@ -17232,10 +17316,8 @@ static pci_ers_result_t bnxt_io_error_detected(struct pci_dev *pdev,
* so we disable bus master to prevent any potential bad DMAs before
* freeing kernel memory.
*/
- if (state == pci_channel_io_frozen) {
- set_bit(BNXT_STATE_PCI_CHANNEL_IO_FROZEN, &bp->state);
+ if (state == pci_channel_io_frozen)
bnxt_fw_fatal_close(bp);
- }
if (netif_running(netdev))
__bnxt_close_nic(bp, true, true);
@@ -17265,65 +17347,80 @@ static pci_ers_result_t bnxt_io_slot_reset(struct pci_dev *pdev)
struct bnxt *bp = netdev_priv(netdev);
int retry = 0;
int err = 0;
- int off;
+ u16 cmd;
netdev_info(bp->dev, "PCI Slot Reset\n");
- if (!(bp->flags & BNXT_FLAG_CHIP_P5_PLUS) &&
- test_bit(BNXT_STATE_PCI_CHANNEL_IO_FROZEN, &bp->state))
- msleep(900);
+ /* After a PCIe hot reset, the chip should return CRS when the
+ * vendor ID config register is read until it is ready. On all
+ * chips, this is not happening reliably so add a 5-second delay
+ * as a workaround.
+ */
+ msleep(5000);
netdev_lock(netdev);
- if (pci_enable_device(pdev)) {
+ pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+ if (PCI_POSSIBLE_ERROR(cmd)) {
dev_err(&pdev->dev,
- "Cannot re-enable PCI device after reset.\n");
- } else {
- pci_set_master(pdev);
- /* Upon fatal error, our device internal logic that latches to
- * BAR value is getting reset and will restore only upon
- * rewriting the BARs.
- *
- * As pci_restore_state() does not re-write the BARs if the
- * value is same as saved value earlier, driver needs to
- * write the BARs to 0 to force restore, in case of fatal error.
- */
- if (test_and_clear_bit(BNXT_STATE_PCI_CHANNEL_IO_FROZEN,
- &bp->state)) {
- for (off = PCI_BASE_ADDRESS_0;
- off <= PCI_BASE_ADDRESS_5; off += 4)
- pci_write_config_dword(bp->pdev, off, 0);
- }
- pci_restore_state(pdev);
- pci_save_state(pdev);
+ "PCI config space inaccessible after reset\n");
+ goto reset_exit;
+ }
- bnxt_inv_fw_health_reg(bp);
- bnxt_try_map_fw_health_reg(bp);
+ /* Upon PCIe error, our device internal logic that latches to
+ * BAR value is getting reset and will restore only upon
+ * rewriting the BARs.
+ *
+ * As pci_restore_state() does not re-write the BARs if the
+ * value is same as saved value earlier, driver needs to
+ * write the BARs to 0 to force restore.
+ */
+ pci_clear_master(pdev);
+ pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+ cmd &= ~PCI_COMMAND_MEMORY;
+ pci_write_config_word(pdev, PCI_COMMAND, cmd);
- /* In some PCIe AER scenarios, firmware may take up to
- * 10 seconds to become ready in the worst case.
- */
- do {
- err = bnxt_try_recover_fw(bp);
- if (!err)
- break;
- retry++;
- } while (retry < BNXT_FW_SLOT_RESET_RETRY);
+ bnxt_clear_bars(pdev);
+ pci_restore_state(pdev);
- if (err) {
- dev_err(&pdev->dev, "Firmware not ready\n");
- goto reset_exit;
+ if (!pci_is_enabled(pdev) && pci_enable_device(pdev)) {
+ dev_err(&pdev->dev,
+ "Cannot re-enable PCI device after reset.\n");
+ pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+ if (!PCI_POSSIBLE_ERROR(cmd)) {
+ cmd &= ~(PCI_COMMAND_MASTER | PCI_COMMAND_MEMORY);
+ pci_write_config_word(pdev, PCI_COMMAND, cmd);
}
+ goto reset_exit;
+ }
+ pci_set_master(pdev);
+
+ bnxt_inv_fw_health_reg(bp);
+ bnxt_try_map_fw_health_reg(bp);
- err = bnxt_hwrm_func_reset(bp);
+ /* In some PCIe AER scenarios, firmware may take up to
+ * 10 seconds to become ready in the worst case.
+ */
+ do {
+ err = bnxt_try_recover_fw(bp);
if (!err)
- result = PCI_ERS_RESULT_RECOVERED;
+ break;
+ retry++;
+ } while (retry < BNXT_FW_SLOT_RESET_RETRY);
- /* IRQ will be initialized later in bnxt_io_resume */
- bnxt_ulp_irq_stop(bp);
- bnxt_clear_int_mode(bp);
+ if (err) {
+ dev_err(&pdev->dev, "Firmware not ready\n");
+ goto reset_exit;
}
+ err = bnxt_hwrm_func_reset(bp);
+ if (!err)
+ result = PCI_ERS_RESULT_RECOVERED;
+
+ /* IRQ will be initialized later in bnxt_io_resume */
+ bnxt_ulp_irq_stop(bp);
+ bnxt_clear_int_mode(bp);
+
reset_exit:
clear_bit(BNXT_STATE_IN_FW_RESET, &bp->state);
bnxt_clear_reservations(bp, true);
@@ -17365,9 +17462,10 @@ static void bnxt_io_resume(struct pci_dev *pdev)
netdev_unlock(netdev);
rtnl_unlock();
- bnxt_ulp_start(bp, err);
- if (!err)
+ if (!err) {
+ bnxt_ulp_start(bp);
bnxt_reenable_sriov(bp);
+ }
}
static const struct pci_error_handlers bnxt_err_handler = {
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index f02508600c1d..1d17ffb7415e 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -24,12 +24,12 @@
#include <linux/interrupt.h>
#include <linux/rhashtable.h>
#include <linux/crash_dump.h>
-#include <linux/auxiliary_bus.h>
#include <net/devlink.h>
#include <net/dst_metadata.h>
#include <net/xdp.h>
#include <linux/dim.h>
#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/bnxt/ulp.h>
#ifdef CONFIG_TEE_BNXT_FW
#include <linux/firmware/broadcom/tee_bnxt_fw.h>
#endif
@@ -1253,6 +1253,8 @@ struct bnxt_irq {
struct bnxt *bp;
int msix_nr;
int ring_nr;
+ u16 tag;
+ u16 new_tag;
struct irq_affinity_notify affinity_notify;
};
@@ -2084,12 +2086,6 @@ struct bnxt_fw_health {
#define BNXT_FW_IF_RETRY 10
#define BNXT_FW_SLOT_RESET_RETRY 4
-struct bnxt_aux_priv {
- struct auxiliary_device aux_dev;
- struct bnxt_en_dev *edev;
- int id;
-};
-
enum board_idx {
BCM57301,
BCM57302,
@@ -2349,8 +2345,8 @@ struct bnxt {
#define BNXT_CHIP_P5_AND_MINUS(bp) \
(BNXT_CHIP_P3(bp) || BNXT_CHIP_P4(bp) || BNXT_CHIP_P5(bp))
- struct bnxt_aux_priv *aux_priv;
- struct bnxt_en_dev *edev;
+ struct bnxt_aux_priv *aux_priv[__BNXT_AUXDEV_MAX];
+ struct bnxt_en_dev *edev[__BNXT_AUXDEV_MAX];
struct bnxt_napi **bnapi;
@@ -2453,7 +2449,6 @@ struct bnxt {
#define BNXT_STATE_ABORT_ERR 5
#define BNXT_STATE_FW_FATAL_COND 6
#define BNXT_STATE_DRV_REGISTERED 7
-#define BNXT_STATE_PCI_CHANNEL_IO_FROZEN 8
#define BNXT_STATE_NAPI_DISABLED 9
#define BNXT_STATE_L2_FILTER_RETRY 10
#define BNXT_STATE_FW_ACTIVATE 11
@@ -2628,6 +2623,7 @@ struct bnxt {
#define BNXT_RING_COAL_NOW_SP_EVENT 17
#define BNXT_FW_RESET_NOTIFY_SP_EVENT 18
#define BNXT_FW_EXCEPTION_SP_EVENT 19
+#define BNXT_TPH_UPDATE_SP_EVENT 20
#define BNXT_LINK_CFG_CHANGE_SP_EVENT 21
#define BNXT_THERMAL_THRESHOLD_SP_EVENT 22
#define BNXT_FW_ECHO_REQUEST_SP_EVENT 23
@@ -2761,6 +2757,13 @@ struct bnxt {
struct bnxt_ctx_pg_info *fw_crash_mem;
u32 fw_crash_len;
struct bnxt_bs_trace_info bs_trace[BNXT_TRACE_MAX];
+ int auxdev_id;
+ /* synchronize validity checks of available aux devices */
+ struct mutex auxdev_lock;
+ u8 auxdev_state[__BNXT_AUXDEV_MAX];
+#define BNXT_ADEV_STATE_NONE 0
+#define BNXT_ADEV_STATE_INIT 1
+#define BNXT_ADEV_STATE_ADD 2
};
#define BNXT_NUM_RX_RING_STATS 8
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c
index 0147b3a9ad86..071d07821ac1 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c
@@ -440,13 +440,13 @@ static int bnxt_dl_reload_down(struct devlink *dl, bool netns_change,
"reload is unsupported while VFs are allocated or being configured");
netdev_unlock(bp->dev);
rtnl_unlock();
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
return -EOPNOTSUPP;
}
if (bp->dev->reg_state == NETREG_UNREGISTERED) {
netdev_unlock(bp->dev);
rtnl_unlock();
- bnxt_ulp_start(bp, 0);
+ bnxt_ulp_start(bp);
return -ENODEV;
}
if (netif_running(bp->dev))
@@ -578,8 +578,8 @@ static int bnxt_dl_reload_up(struct devlink *dl, enum devlink_reload_action acti
}
netdev_unlock(bp->dev);
rtnl_unlock();
- if (action == DEVLINK_RELOAD_ACTION_DRIVER_REINIT)
- bnxt_ulp_start(bp, rc);
+ if (!rc && action == DEVLINK_RELOAD_ACTION_DRIVER_REINIT)
+ bnxt_ulp_start(bp);
return rc;
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
index 8035c32802e8..7f60d7b9f557 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
@@ -5088,7 +5088,7 @@ static void bnxt_self_test(struct net_device *dev, struct ethtool_test *etest,
memset(buf, 0, sizeof(u64) * bp->num_tests);
if (etest->flags & ETH_TEST_FL_OFFLINE &&
- bnxt_ulp_registered(bp->edev)) {
+ bnxt_ulp_registered(bp->edev[BNXT_AUXDEV_RDMA])) {
etest->flags |= ETH_TEST_FL_FAILED;
netdev_warn(dev, "Offline tests cannot be run with RoCE driver loaded\n");
return;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
index 56a35c65d193..1078b5ac39a5 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c
@@ -29,9 +29,32 @@
static DEFINE_IDA(bnxt_aux_dev_ids);
+struct bnxt_aux_device {
+ const char *name;
+};
+
+static void bnxt_auxdev_set_state(struct bnxt *bp, int idx, int state)
+{
+ bp->auxdev_state[idx] = state;
+}
+
+static bool bnxt_auxdev_is_init(struct bnxt *bp, int idx)
+{
+ return (bp->auxdev_state[idx] == BNXT_ADEV_STATE_INIT);
+}
+
+static bool bnxt_auxdev_is_active(struct bnxt *bp, int idx)
+{
+ return (bp->auxdev_state[idx] == BNXT_ADEV_STATE_ADD);
+}
+
+static struct bnxt_aux_device bnxt_aux_devices[__BNXT_AUXDEV_MAX] = {{
+ .name = "rdma",
+}};
+
static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent)
{
- struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
int num_msix, i;
if (!edev->ulp_tbl->msix_requested) {
@@ -51,61 +74,75 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent)
int bnxt_get_ulp_msix_num(struct bnxt *bp)
{
- if (bp->edev)
- return bp->edev->ulp_num_msix_vec;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ return edev->ulp_num_msix_vec;
return 0;
}
void bnxt_set_ulp_msix_num(struct bnxt *bp, int num)
{
- if (bp->edev)
- bp->edev->ulp_num_msix_vec = num;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ edev->ulp_num_msix_vec = num;
}
int bnxt_get_ulp_msix_num_in_use(struct bnxt *bp)
{
- if (bnxt_ulp_registered(bp->edev))
- return bp->edev->ulp_num_msix_vec;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (bnxt_ulp_registered(edev))
+ return edev->ulp_num_msix_vec;
return 0;
}
int bnxt_get_ulp_stat_ctxs(struct bnxt *bp)
{
- if (bp->edev)
- return bp->edev->ulp_num_ctxs;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ return edev->ulp_num_ctxs;
return 0;
}
void bnxt_set_ulp_stat_ctxs(struct bnxt *bp, int num_ulp_ctx)
{
- if (bp->edev)
- bp->edev->ulp_num_ctxs = num_ulp_ctx;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev)
+ edev->ulp_num_ctxs = num_ulp_ctx;
}
int bnxt_get_ulp_stat_ctxs_in_use(struct bnxt *bp)
{
- if (bnxt_ulp_registered(bp->edev))
- return bp->edev->ulp_num_ctxs;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (bnxt_ulp_registered(edev))
+ return edev->ulp_num_ctxs;
return 0;
}
void bnxt_set_dflt_ulp_stat_ctxs(struct bnxt *bp)
{
- if (bp->edev) {
- bp->edev->ulp_num_ctxs = BNXT_MIN_ROCE_STAT_CTXS;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
+
+ if (edev) {
+ edev->ulp_num_ctxs = BNXT_MIN_ROCE_STAT_CTXS;
/* Reserve one additional stat_ctx for PF0 (except
* on 1-port NICs) as it also creates one stat_ctx
* for PF1 in case of RoCE bonding.
*/
if (BNXT_PF(bp) && !bp->pf.port_id &&
bp->port_count > 1)
- bp->edev->ulp_num_ctxs++;
+ edev->ulp_num_ctxs++;
/* Reserve one additional stat_ctx when the device is capable
* of supporting port mirroring on RDMA device.
*/
if (BNXT_MIRROR_ON_ROCE_CAP(bp))
- bp->edev->ulp_num_ctxs++;
+ edev->ulp_num_ctxs++;
}
}
@@ -141,7 +178,7 @@ int bnxt_register_dev(struct bnxt_en_dev *edev,
edev->ulp_tbl->msix_requested = bnxt_get_ulp_msix_num(bp);
- bnxt_fill_msix_vecs(bp, bp->edev->msix_entries);
+ bnxt_fill_msix_vecs(bp, edev->msix_entries);
edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED;
exit:
mutex_unlock(&edev->en_dev_lock);
@@ -230,20 +267,27 @@ EXPORT_SYMBOL(bnxt_send_msg);
void bnxt_ulp_stop(struct bnxt *bp)
{
- struct bnxt_aux_priv *aux_priv = bp->aux_priv;
- struct bnxt_en_dev *edev = bp->edev;
+ int i;
- if (!edev)
- return;
-
- mutex_lock(&edev->en_dev_lock);
- if (!bnxt_ulp_registered(edev) ||
- (edev->flags & BNXT_EN_FLAG_ULP_STOPPED))
- goto ulp_stop_exit;
-
- edev->flags |= BNXT_EN_FLAG_ULP_STOPPED;
- if (aux_priv) {
+ mutex_lock(&bp->auxdev_lock);
+ for (i = 0; i < __BNXT_AUXDEV_MAX; i++) {
+ struct bnxt_aux_priv *aux_priv;
struct auxiliary_device *adev;
+ struct bnxt_en_dev *edev;
+
+ if (!bnxt_auxdev_is_active(bp, i))
+ continue;
+
+ aux_priv = bp->aux_priv[i];
+ edev = bp->edev[i];
+ mutex_lock(&edev->en_dev_lock);
+ if (!bnxt_ulp_registered(edev) ||
+ (edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) {
+ mutex_unlock(&edev->en_dev_lock);
+ continue;
+ }
+
+ edev->flags |= BNXT_EN_FLAG_ULP_STOPPED;
adev = &aux_priv->aux_dev;
if (adev->dev.driver) {
@@ -254,29 +298,35 @@ void bnxt_ulp_stop(struct bnxt *bp)
edev->en_state = bp->state;
adrv->suspend(adev, pm);
}
+ mutex_unlock(&edev->en_dev_lock);
}
-ulp_stop_exit:
- mutex_unlock(&edev->en_dev_lock);
+ mutex_unlock(&bp->auxdev_lock);
}
-void bnxt_ulp_start(struct bnxt *bp, int err)
+void bnxt_ulp_start(struct bnxt *bp)
{
- struct bnxt_aux_priv *aux_priv = bp->aux_priv;
- struct bnxt_en_dev *edev = bp->edev;
+ int i;
- if (!edev || err)
- return;
+ mutex_lock(&bp->auxdev_lock);
+ for (i = 0; i < __BNXT_AUXDEV_MAX; i++) {
+ struct bnxt_aux_priv *aux_priv;
+ struct auxiliary_device *adev;
+ struct bnxt_en_dev *edev;
- mutex_lock(&edev->en_dev_lock);
- if (!bnxt_ulp_registered(edev) ||
- !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED))
- goto ulp_start_exit;
+ if (!bnxt_auxdev_is_active(bp, i))
+ continue;
+
+ aux_priv = bp->aux_priv[i];
+ edev = bp->edev[i];
+ mutex_lock(&edev->en_dev_lock);
+ if (!bnxt_ulp_registered(edev) ||
+ !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) {
+ goto clear_flag_continue;
+ }
- if (edev->ulp_tbl->msix_requested)
- bnxt_fill_msix_vecs(bp, edev->msix_entries);
+ if (edev->ulp_tbl->msix_requested)
+ bnxt_fill_msix_vecs(bp, edev->msix_entries);
- if (aux_priv) {
- struct auxiliary_device *adev;
adev = &aux_priv->aux_dev;
if (adev->dev.driver) {
@@ -286,22 +336,23 @@ void bnxt_ulp_start(struct bnxt *bp, int err)
edev->en_state = bp->state;
adrv->resume(adev);
}
+clear_flag_continue:
+ edev->flags &= ~BNXT_EN_FLAG_ULP_STOPPED;
+ mutex_unlock(&edev->en_dev_lock);
}
-ulp_start_exit:
- edev->flags &= ~BNXT_EN_FLAG_ULP_STOPPED;
- mutex_unlock(&edev->en_dev_lock);
+ mutex_unlock(&bp->auxdev_lock);
}
void bnxt_ulp_irq_stop(struct bnxt *bp)
{
- struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
struct bnxt_ulp_ops *ops;
bool reset = false;
if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED))
return;
- if (bnxt_ulp_registered(bp->edev)) {
+ if (bnxt_ulp_registered(edev)) {
struct bnxt_ulp *ulp = edev->ulp_tbl;
if (!ulp->msix_requested)
@@ -318,13 +369,13 @@ void bnxt_ulp_irq_stop(struct bnxt *bp)
void bnxt_ulp_irq_restart(struct bnxt *bp, int err)
{
- struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
struct bnxt_ulp_ops *ops;
if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED))
return;
- if (bnxt_ulp_registered(bp->edev)) {
+ if (bnxt_ulp_registered(edev)) {
struct bnxt_ulp *ulp = edev->ulp_tbl;
struct bnxt_msix_entry *ent = NULL;
@@ -350,7 +401,7 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err)
void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl)
{
u16 event_id = le16_to_cpu(cmpl->event_id);
- struct bnxt_en_dev *edev = bp->edev;
+ struct bnxt_en_dev *edev = bp->edev[BNXT_AUXDEV_RDMA];
struct bnxt_ulp_ops *ops;
struct bnxt_ulp *ulp;
@@ -391,18 +442,21 @@ void bnxt_register_async_events(struct bnxt_en_dev *edev,
}
EXPORT_SYMBOL(bnxt_register_async_events);
-void bnxt_rdma_aux_device_uninit(struct bnxt *bp)
+void bnxt_aux_devices_uninit(struct bnxt *bp)
{
struct bnxt_aux_priv *aux_priv;
struct auxiliary_device *adev;
-
- /* Skip if no auxiliary device init was done. */
- if (!bp->aux_priv)
- return;
-
- aux_priv = bp->aux_priv;
- adev = &aux_priv->aux_dev;
- auxiliary_device_uninit(adev);
+ int idx;
+
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ if (bnxt_auxdev_is_init(bp, idx)) {
+ aux_priv = bp->aux_priv[idx];
+ adev = &aux_priv->aux_dev;
+ auxiliary_device_uninit(adev);
+ }
+ }
+ mutex_unlock(&bp->auxdev_lock);
}
static void bnxt_aux_dev_release(struct device *dev)
@@ -411,20 +465,25 @@ static void bnxt_aux_dev_release(struct device *dev)
container_of(dev, struct bnxt_aux_priv, aux_dev.dev);
struct bnxt *bp = netdev_priv(aux_priv->edev->net);
- ida_free(&bnxt_aux_dev_ids, aux_priv->id);
kfree(aux_priv->edev->ulp_tbl);
- bp->edev = NULL;
+ bp->edev[aux_priv->id] = NULL;
kfree(aux_priv->edev);
+ bp->aux_priv[aux_priv->id] = NULL;
kfree(aux_priv);
- bp->aux_priv = NULL;
}
-void bnxt_rdma_aux_device_del(struct bnxt *bp)
+void bnxt_aux_devices_del(struct bnxt *bp)
{
- if (!bp->edev)
- return;
+ int idx;
- auxiliary_device_delete(&bp->aux_priv->aux_dev);
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ if (bnxt_auxdev_is_active(bp, idx)) {
+ auxiliary_device_delete(&bp->aux_priv[idx]->aux_dev);
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_INIT);
+ }
+ }
+ mutex_unlock(&bp->auxdev_lock);
}
static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp)
@@ -454,83 +513,108 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp)
edev->bar0 = bp->bar0;
}
-void bnxt_rdma_aux_device_add(struct bnxt *bp)
+void bnxt_aux_devices_add(struct bnxt *bp)
{
struct auxiliary_device *aux_dev;
- int rc;
-
- if (!bp->edev)
- return;
-
- aux_dev = &bp->aux_priv->aux_dev;
- rc = auxiliary_device_add(aux_dev);
- if (rc) {
- netdev_warn(bp->dev, "Failed to add auxiliary device for ROCE\n");
- auxiliary_device_uninit(aux_dev);
- bp->flags &= ~BNXT_FLAG_ROCE_CAP;
+ int rc, idx;
+
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ if (bnxt_auxdev_is_init(bp, idx)) {
+ aux_dev = &bp->aux_priv[idx]->aux_dev;
+ rc = auxiliary_device_add(aux_dev);
+ if (rc) {
+ netdev_warn(bp->dev, "Failed to add auxiliary device for ROCE\n");
+ auxiliary_device_uninit(aux_dev);
+ bnxt_auxdev_set_state(bp, idx,
+ BNXT_ADEV_STATE_NONE);
+ if (idx == BNXT_AUXDEV_RDMA)
+ bp->flags &= ~BNXT_FLAG_ROCE_CAP;
+ continue;
+ }
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_ADD);
+ }
}
+ mutex_unlock(&bp->auxdev_lock);
}
-void bnxt_rdma_aux_device_init(struct bnxt *bp)
+void bnxt_aux_devices_init(struct bnxt *bp)
{
struct auxiliary_device *aux_dev;
struct bnxt_aux_priv *aux_priv;
struct bnxt_en_dev *edev;
struct bnxt_ulp *ulp;
- int rc;
-
- if (!(bp->flags & BNXT_FLAG_ROCE_CAP))
- return;
-
- aux_priv = kzalloc(sizeof(*bp->aux_priv), GFP_KERNEL);
- if (!aux_priv)
- goto exit;
-
- aux_priv->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL);
- if (aux_priv->id < 0) {
- netdev_warn(bp->dev,
- "ida alloc failed for ROCE auxiliary device\n");
- kfree(aux_priv);
- goto exit;
- }
-
- aux_dev = &aux_priv->aux_dev;
- aux_dev->id = aux_priv->id;
- aux_dev->name = "rdma";
- aux_dev->dev.parent = &bp->pdev->dev;
- aux_dev->dev.release = bnxt_aux_dev_release;
-
- rc = auxiliary_device_init(aux_dev);
- if (rc) {
- ida_free(&bnxt_aux_dev_ids, aux_priv->id);
+ int rc, idx;
+
+ mutex_lock(&bp->auxdev_lock);
+ for (idx = 0; idx < __BNXT_AUXDEV_MAX; idx++) {
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_NONE);
+
+ if (idx == BNXT_AUXDEV_RDMA &&
+ !(bp->flags & BNXT_FLAG_ROCE_CAP))
+ continue;
+
+ aux_priv = kzalloc_obj(*aux_priv);
+ if (!aux_priv)
+ goto next_auxdev;
+
+ edev = kzalloc_obj(*edev);
+ if (!edev)
+ goto aux_priv_free;
+ aux_priv->edev = edev;
+ bnxt_set_edev_info(edev, bp);
+
+ ulp = kzalloc_obj(*ulp);
+ if (!ulp)
+ goto edev_free;
+ edev->ulp_tbl = ulp;
+ aux_priv->id = idx;
+
+ aux_dev = &aux_priv->aux_dev;
+ aux_dev->id = bp->auxdev_id;
+ aux_dev->name = bnxt_aux_devices[idx].name;
+ aux_dev->dev.parent = &bp->pdev->dev;
+ aux_dev->dev.release = bnxt_aux_dev_release;
+
+ rc = auxiliary_device_init(aux_dev);
+ if (rc)
+ goto ulp_free;
+ bp->aux_priv[idx] = aux_priv;
+
+ /* From this point, all cleanup will happen via the .release
+ * callback & any error unwinding will need to include a call
+ * to auxiliary_device_uninit.
+ */
+ bp->edev[idx] = edev;
+ if (idx == BNXT_AUXDEV_RDMA)
+ bp->ulp_num_msix_want = bnxt_set_dflt_ulp_msix(bp);
+ bnxt_auxdev_set_state(bp, idx, BNXT_ADEV_STATE_INIT);
+
+ continue;
+ulp_free:
+ kfree(ulp);
+edev_free:
+ kfree(edev);
+aux_priv_free:
kfree(aux_priv);
- goto exit;
+next_auxdev:
+ if (idx == BNXT_AUXDEV_RDMA)
+ bp->flags &= ~BNXT_FLAG_ROCE_CAP;
}
- bp->aux_priv = aux_priv;
-
- /* From this point, all cleanup will happen via the .release callback &
- * any error unwinding will need to include a call to
- * auxiliary_device_uninit.
- */
- edev = kzalloc(sizeof(*edev), GFP_KERNEL);
- if (!edev)
- goto aux_dev_uninit;
-
- aux_priv->edev = edev;
-
- ulp = kzalloc(sizeof(*ulp), GFP_KERNEL);
- if (!ulp)
- goto aux_dev_uninit;
+ mutex_unlock(&bp->auxdev_lock);
+}
- edev->ulp_tbl = ulp;
- bp->edev = edev;
- bnxt_set_edev_info(edev, bp);
- bp->ulp_num_msix_want = bnxt_set_dflt_ulp_msix(bp);
+int bnxt_auxdev_id_alloc(struct bnxt *bp)
+{
+ bp->auxdev_id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL);
+ if (bp->auxdev_id < 0)
+ return bp->auxdev_id;
- return;
+ return 0;
+}
-aux_dev_uninit:
- auxiliary_device_uninit(aux_dev);
-exit:
- bp->flags &= ~BNXT_FLAG_ROCE_CAP;
+void bnxt_auxdev_id_free(struct bnxt *bp, int id)
+{
+ if (bp->auxdev_id >= 0)
+ ida_free(&bnxt_aux_dev_ids, id);
}
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
index 96d5d4f7f51f..984432952963 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
@@ -253,6 +253,17 @@ int bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
reg = bcmgenet_umac_readl(priv, UMAC_CMD);
reg &= ~CMD_CRC_FWD;
bcmgenet_umac_writel(priv, reg, UMAC_CMD);
+
+ /*
+ * Mirror wol_power_down_cfg(). If only UMAC_RX
+ * is enabled, then we must place the UMAC back
+ * into SW_RESET.
+ */
+ reg = bcmgenet_umac_readl(priv, UMAC_CMD);
+ if ((reg & CMD_RX_EN) && !(reg & CMD_TX_EN)) {
+ reg |= CMD_SW_RESET;
+ bcmgenet_umac_writel(priv, reg, UMAC_CMD);
+ }
spin_unlock_bh(&priv->reg_lock);
/* Resume link status tracking */
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_main.c b/drivers/net/ethernet/cavium/liquidio/lio_main.c
index eb620e8544cf..4d73db01cae2 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_main.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_main.c
@@ -3791,9 +3791,7 @@ static int setup_nic_devices(struct octeon_device *octeon_dev)
static int octeon_enable_sriov(struct octeon_device *oct)
{
unsigned int num_vfs_alloced = oct->sriov_info.num_vfs_alloced;
- struct pci_dev *vfdev;
int err;
- u32 u;
if (OCTEON_CN23XX_PF(oct) && num_vfs_alloced) {
err = pci_enable_sriov(oct->pci_dev,
@@ -3806,23 +3804,6 @@ static int octeon_enable_sriov(struct octeon_device *oct)
return err;
}
oct->sriov_info.sriov_enabled = 1;
-
- /* init lookup table that maps DPI ring number to VF pci_dev
- * struct pointer
- */
- u = 0;
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, NULL);
- while (vfdev) {
- if (vfdev->is_virtfn &&
- (vfdev->physfn == oct->pci_dev)) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] =
- vfdev;
- u += oct->sriov_info.rings_per_vf;
- }
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, vfdev);
- }
}
return num_vfs_alloced;
@@ -3830,8 +3811,6 @@ static int octeon_enable_sriov(struct octeon_device *oct)
static int lio_pci_sriov_disable(struct octeon_device *oct)
{
- int u;
-
if (pci_vfs_assigned(oct->pci_dev)) {
dev_err(&oct->pci_dev->dev, "VFs are still assigned to VMs.\n");
return -EPERM;
@@ -3839,12 +3818,6 @@ static int lio_pci_sriov_disable(struct octeon_device *oct)
pci_disable_sriov(oct->pci_dev);
- u = 0;
- while (u < MAX_POSSIBLE_VFS) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] = NULL;
- u += oct->sriov_info.rings_per_vf;
- }
-
oct->sriov_info.num_vfs_alloced = 0;
dev_info(&oct->pci_dev->dev, "oct->pf_num:%d disabled VFs\n",
oct->pf_num);
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_device.h b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
index 19344b21f8fb..858a0fff2cc0 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_device.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
@@ -390,9 +390,6 @@ struct octeon_sriov_info {
struct lio_trusted_vf trusted_vf;
- /*lookup table that maps DPI ring number to VF pci_dev struct pointer*/
- struct pci_dev *dpiring_to_vfpcidev_lut[MAX_POSSIBLE_VFS];
-
u64 vf_macaddr[MAX_POSSIBLE_VFS];
u16 vf_vlantci[MAX_POSSIBLE_VFS];
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
index ad685f5d0a13..697fcdc41e3c 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
@@ -26,6 +26,31 @@
#include "octeon_mailbox.h"
#include "cn23xx_pf_device.h"
+static struct pci_dev *lio_vf_pci_dev_by_qno(struct octeon_device *oct, u32 q_no)
+{
+ struct pci_dev *vfdev = NULL;
+ int vfidx;
+
+ if (!oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ if (q_no % oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ vfidx = q_no / oct->sriov_info.rings_per_vf;
+ if (vfidx >= oct->sriov_info.num_vfs_alloced)
+ return NULL;
+
+ while ((vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
+ OCTEON_CN23XX_VF_VID, vfdev))) {
+ if (pci_physfn(vfdev) && pci_physfn(vfdev) == oct->pci_dev &&
+ pci_iov_vf_id(vfdev) == vfidx)
+ return vfdev;
+ }
+
+ return NULL;
+}
+
/**
* octeon_mbox_read:
* @mbox: Pointer mailbox
@@ -237,6 +262,7 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox,
struct octeon_mbox_cmd *mbox_cmd)
{
struct octeon_device *oct = mbox->oct_dev;
+ struct pci_dev *vfdev;
switch (mbox_cmd->msg.s.cmd) {
case OCTEON_VF_ACTIVE:
@@ -260,7 +286,12 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox,
dev_info(&oct->pci_dev->dev,
"got a request for FLR from VF that owns DPI ring %u\n",
mbox->q_no);
- pcie_flr(oct->sriov_info.dpiring_to_vfpcidev_lut[mbox->q_no]);
+ vfdev = lio_vf_pci_dev_by_qno(oct, mbox->q_no);
+ if (!vfdev)
+ break;
+
+ pcie_flr(vfdev);
+ pci_dev_put(vfdev);
break;
case OCTEON_PF_CHANGED_VF_MACADDR:
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
index 4e2096e49684..4da7695db89b 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
@@ -389,7 +389,7 @@ static void chcr_ktls_dev_del(struct net_device *netdev,
cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan,
tx_info->tid, tx_info->ip_family);
- xa_erase(&u_ctx->tid_list, tx_info->tid);
+ xa_erase_bh(&u_ctx->tid_list, tx_info->tid);
}
port_stats = &tx_info->adap->ch_ktls_stats.ktls_port[tx_info->port_id];
@@ -577,7 +577,7 @@ static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk,
cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan,
tx_info->tid, tx_info->ip_family);
- xa_erase(&u_ctx->tid_list, tx_info->tid);
+ xa_erase_bh(&u_ctx->tid_list, tx_info->tid);
put_module:
/* release module refcount */
diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c
index 5fbb02d14dc3..9b1f8933813c 100644
--- a/drivers/net/ethernet/google/gve/gve_main.c
+++ b/drivers/net/ethernet/google/gve/gve_main.c
@@ -1663,17 +1663,17 @@ static int gve_xsk_pool_disable(struct net_device *dev,
}
napi_rx = &priv->ntfy_blocks[priv->rx[qid].ntfy_id].napi;
- napi_disable(napi_rx); /* make sure current rx poll is done */
+ napi_disable_locked(napi_rx); /* make sure current rx poll is done */
tx_qid = gve_xdp_tx_queue_id(priv, qid);
napi_tx = &priv->ntfy_blocks[priv->tx[tx_qid].ntfy_id].napi;
- napi_disable(napi_tx); /* make sure current tx poll is done */
+ napi_disable_locked(napi_tx); /* make sure current tx poll is done */
gve_unreg_xsk_pool(priv, qid);
smp_mb(); /* Make sure it is visible to the workers on datapath */
- napi_enable(napi_rx);
- napi_enable(napi_tx);
+ napi_enable_locked(napi_rx);
+ napi_enable_locked(napi_tx);
if (gve_is_gqi(priv)) {
if (gve_rx_work_pending(&priv->rx[qid]))
napi_schedule(napi_rx);
diff --git a/drivers/net/ethernet/google/gve/gve_rx_dqo.c b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
index 0c7ab6f4a540..49ae40c72013 100644
--- a/drivers/net/ethernet/google/gve/gve_rx_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_rx_dqo.c
@@ -113,6 +113,12 @@ static void gve_rx_reset_ring_dqo(struct gve_priv *priv, int idx)
gve_free_to_page_pool(rx, bs, false);
else
gve_free_qpl_page_dqo(bs);
+
+ if (gve_buf_state_is_allocated(rx, bs) &&
+ bs->xsk_buff) {
+ xsk_buff_free(bs->xsk_buff);
+ bs->xsk_buff = NULL;
+ }
}
}
@@ -812,7 +818,12 @@ static int gve_rx_dqo(struct napi_struct *napi, struct gve_rx_ring *rx,
}
if (unlikely(compl_desc->rx_error)) {
- gve_free_buffer(rx, buf_state);
+ if (buf_state->xsk_buff) {
+ xsk_buff_free(buf_state->xsk_buff);
+ gve_free_buf_state(rx, buf_state);
+ } else {
+ gve_free_buffer(rx, buf_state);
+ }
return -EINVAL;
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index 6e9552ce4c8c..c490dc078066 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -553,10 +553,15 @@ static int ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter,
static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
{
+ void *ptr = adapter->tx_ltb_ptr[idx];
+
+ if (!ptr)
+ return;
+
+ adapter->tx_ltb_ptr[idx] = NULL;
dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
adapter->tx_ltb_size, DMA_TO_DEVICE);
- kfree(adapter->tx_ltb_ptr[idx]);
- adapter->tx_ltb_ptr[idx] = NULL;
+ kfree(ptr);
}
static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
@@ -667,7 +672,7 @@ static int ibmveth_open(struct net_device *netdev)
for (i = 0; i < netdev->real_num_tx_queues; i++) {
if (ibmveth_allocate_tx_ltb(adapter, i))
- goto out_free_tx_ltb;
+ goto out_unmap_filter_list;
}
adapter->rx_queue.index = 0;
@@ -718,10 +723,6 @@ static int ibmveth_open(struct net_device *netdev)
if (rc != 0) {
netdev_err(netdev, "unable to request irq 0x%x, rc %d\n",
netdev->irq, rc);
- do {
- lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
- } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-
goto out_free_buffer_pools;
}
@@ -737,6 +738,9 @@ static int ibmveth_open(struct net_device *netdev)
return 0;
out_free_buffer_pools:
+ do {
+ lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
while (--i >= 0) {
if (adapter->rx_buff_pool[i].active)
ibmveth_free_buffer_pool(adapter,
@@ -746,10 +750,8 @@ static int ibmveth_open(struct net_device *netdev)
dma_unmap_single(dev, adapter->filter_list_dma, 4096,
DMA_BIDIRECTIONAL);
-out_free_tx_ltb:
- while (--i >= 0) {
+ for (i = netdev->real_num_tx_queues - 1; i >= 0; i--)
ibmveth_free_tx_ltb(adapter, i);
- }
out_unmap_buffer_list:
dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
diff --git a/drivers/net/ethernet/intel/ice/devlink/port.c b/drivers/net/ethernet/intel/ice/devlink/port.c
index 63fb36fc4b3d..9f1a73f3dd5a 100644
--- a/drivers/net/ethernet/intel/ice/devlink/port.c
+++ b/drivers/net/ethernet/intel/ice/devlink/port.c
@@ -590,8 +590,8 @@ static void ice_dealloc_dynamic_port(struct ice_dynamic_port *dyn_port)
xa_erase(&pf->sf_nums, devlink_port->attrs.pci_sf.sf);
ice_eswitch_detach_sf(pf, dyn_port);
- ice_vsi_free(dyn_port->vsi);
xa_erase(&pf->dyn_ports, dyn_port->vsi->idx);
+ ice_vsi_free(dyn_port->vsi);
kfree(dyn_port);
}
diff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h
index fc4eae0a00cd..2b285ffa9d4b 100644
--- a/drivers/net/ethernet/intel/ice/ice.h
+++ b/drivers/net/ethernet/intel/ice/ice.h
@@ -751,7 +751,7 @@ static inline bool ice_is_xdp_ena_vsi(struct ice_vsi *vsi)
static inline void ice_set_ring_xdp(struct ice_tx_ring *ring)
{
- ring->flags |= ICE_TX_FLAGS_RING_XDP;
+ set_bit(ICE_TX_RING_FLAGS_XDP, ring->flags);
}
/**
@@ -779,7 +779,7 @@ static inline bool ice_is_txtime_ena(const struct ice_tx_ring *ring)
*/
static inline bool ice_is_txtime_cfg(const struct ice_tx_ring *ring)
{
- return !!(ring->flags & ICE_TX_FLAGS_TXTIME);
+ return test_bit(ICE_TX_RING_FLAGS_TXTIME, ring->flags);
}
/**
diff --git a/drivers/net/ethernet/intel/ice/ice_dcb_lib.c b/drivers/net/ethernet/intel/ice/ice_dcb_lib.c
index 270f5a00ece3..3d0332026c0d 100644
--- a/drivers/net/ethernet/intel/ice/ice_dcb_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_dcb_lib.c
@@ -943,7 +943,7 @@ ice_tx_prepare_vlan_flags_dcb(struct ice_tx_ring *tx_ring,
/* if this is not already set it means a VLAN 0 + priority needs
* to be offloaded
*/
- if (tx_ring->flags & ICE_TX_FLAGS_RING_VLAN_L2TAG2)
+ if (test_bit(ICE_TX_RING_FLAGS_VLAN_L2TAG2, tx_ring->flags))
first->tx_flags |= ICE_TX_FLAGS_HW_OUTER_SINGLE_VLAN;
else
first->tx_flags |= ICE_TX_FLAGS_HW_VLAN;
diff --git a/drivers/net/ethernet/intel/ice/ice_eswitch.c b/drivers/net/ethernet/intel/ice/ice_eswitch.c
index b069e6c514fb..6e7bba473898 100644
--- a/drivers/net/ethernet/intel/ice/ice_eswitch.c
+++ b/drivers/net/ethernet/intel/ice/ice_eswitch.c
@@ -95,7 +95,7 @@ ice_eswitch_release_repr(struct ice_pf *pf, struct ice_repr *repr)
return;
ice_vsi_update_security(vsi, ice_vsi_ctx_set_antispoof);
- metadata_dst_free(repr->dst);
+ dst_release(&repr->dst->dst);
repr->dst = NULL;
ice_fltr_add_mac_and_broadcast(vsi, repr->parent_mac,
ICE_FWD_TO_VSI);
diff --git a/drivers/net/ethernet/intel/ice/ice_ethtool.c b/drivers/net/ethernet/intel/ice/ice_ethtool.c
index c10d0b92e63e..ec33586b6da5 100644
--- a/drivers/net/ethernet/intel/ice/ice_ethtool.c
+++ b/drivers/net/ethernet/intel/ice/ice_ethtool.c
@@ -3249,6 +3249,7 @@ ice_set_ringparam(struct net_device *netdev, struct ethtool_ringparam *ring,
tx_rings[i].desc = NULL;
tx_rings[i].tx_buf = NULL;
tx_rings[i].tstamp_ring = NULL;
+ clear_bit(ICE_TX_RING_FLAGS_TXTIME, tx_rings[i].flags);
tx_rings[i].tx_tstamps = &pf->ptp.port.tx;
err = ice_setup_tx_ring(&tx_rings[i]);
if (err) {
diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c
index 851fab181251..6de55d2424f4 100644
--- a/drivers/net/ethernet/intel/ice/ice_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_lib.c
@@ -1412,9 +1412,9 @@ static int ice_vsi_alloc_rings(struct ice_vsi *vsi)
ring->count = vsi->num_tx_desc;
ring->txq_teid = ICE_INVAL_TEID;
if (dvm_ena)
- ring->flags |= ICE_TX_FLAGS_RING_VLAN_L2TAG2;
+ set_bit(ICE_TX_RING_FLAGS_VLAN_L2TAG2, ring->flags);
else
- ring->flags |= ICE_TX_FLAGS_RING_VLAN_L2TAG1;
+ set_bit(ICE_TX_RING_FLAGS_VLAN_L2TAG1, ring->flags);
WRITE_ONCE(vsi->tx_rings[i], ring);
}
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c
index b843f66c4a6e..4831b9a5ac83 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ -188,9 +188,10 @@ void ice_free_tstamp_ring(struct ice_tx_ring *tx_ring)
void ice_free_tx_tstamp_ring(struct ice_tx_ring *tx_ring)
{
ice_free_tstamp_ring(tx_ring);
+ clear_bit(ICE_TX_RING_FLAGS_TXTIME, tx_ring->flags);
+ smp_wmb(); /* order flag clear before pointer NULL */
kfree_rcu(tx_ring->tstamp_ring, rcu);
- tx_ring->tstamp_ring = NULL;
- tx_ring->flags &= ~ICE_TX_FLAGS_TXTIME;
+ WRITE_ONCE(tx_ring->tstamp_ring, NULL);
}
/**
@@ -403,7 +404,7 @@ static int ice_alloc_tstamp_ring(struct ice_tx_ring *tx_ring)
tx_ring->tstamp_ring = tstamp_ring;
tstamp_ring->desc = NULL;
tstamp_ring->count = ice_calc_ts_ring_count(tx_ring);
- tx_ring->flags |= ICE_TX_FLAGS_TXTIME;
+ set_bit(ICE_TX_RING_FLAGS_TXTIME, tx_ring->flags);
return 0;
}
@@ -1957,13 +1958,20 @@ ice_tx_map(struct ice_tx_ring *tx_ring, struct ice_tx_buf *first,
return;
if (ice_is_txtime_cfg(tx_ring)) {
- struct ice_tstamp_ring *tstamp_ring = tx_ring->tstamp_ring;
- u32 tstamp_count = tstamp_ring->count;
- u32 j = tstamp_ring->next_to_use;
+ struct ice_tstamp_ring *tstamp_ring;
+ u32 tstamp_count, j;
struct ice_ts_desc *ts_desc;
struct timespec64 ts;
u32 tstamp;
+ smp_rmb(); /* order flag read before pointer read */
+ tstamp_ring = READ_ONCE(tx_ring->tstamp_ring);
+ if (unlikely(!tstamp_ring))
+ goto ring_kick;
+
+ tstamp_count = tstamp_ring->count;
+ j = tstamp_ring->next_to_use;
+
ts = ktime_to_timespec64(first->skb->tstamp);
tstamp = ts.tv_nsec >> ICE_TXTIME_CTX_RESOLUTION_128NS;
@@ -1989,9 +1997,10 @@ ice_tx_map(struct ice_tx_ring *tx_ring, struct ice_tx_buf *first,
}
}
tstamp_ring->next_to_use = j;
- writel_relaxed(j, tstamp_ring->tail);
+ writel(j, tstamp_ring->tail);
} else {
- writel_relaxed(i, tx_ring->tail);
+ring_kick:
+ writel(i, tx_ring->tail);
}
return;
@@ -2248,7 +2257,7 @@ ice_tx_prepare_vlan_flags(struct ice_tx_ring *tx_ring, struct ice_tx_buf *first)
*/
if (skb_vlan_tag_present(skb)) {
first->vid = skb_vlan_tag_get(skb);
- if (tx_ring->flags & ICE_TX_FLAGS_RING_VLAN_L2TAG2)
+ if (test_bit(ICE_TX_RING_FLAGS_VLAN_L2TAG2, tx_ring->flags))
first->tx_flags |= ICE_TX_FLAGS_HW_OUTER_SINGLE_VLAN;
else
first->tx_flags |= ICE_TX_FLAGS_HW_VLAN;
@@ -2592,6 +2601,9 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
ice_trace(xmit_frame_ring, tx_ring, skb);
+ /* record the location of the first descriptor for this packet */
+ first = &tx_ring->tx_buf[tx_ring->next_to_use];
+
if (unlikely(ipv6_hopopt_jumbo_remove(skb)))
goto out_drop;
@@ -2620,8 +2632,6 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
offload.tx_ring = tx_ring;
- /* record the location of the first descriptor for this packet */
- first = &tx_ring->tx_buf[tx_ring->next_to_use];
first->skb = skb;
first->type = ICE_TX_BUF_SKB;
first->bytecount = max_t(unsigned int, skb->len, ETH_ZLEN);
@@ -2686,6 +2696,7 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
out_drop:
ice_trace(xmit_frame_ring_drop, tx_ring, skb);
dev_kfree_skb_any(skb);
+ first->type = ICE_TX_BUF_EMPTY;
return NETDEV_TX_OK;
}
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h
index 841a07bfba54..51b23ec1953e 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.h
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.h
@@ -257,6 +257,14 @@ enum ice_rx_dtype {
ICE_RX_DTYPE_SPLIT_ALWAYS = 2,
};
+enum ice_tx_ring_flags {
+ ICE_TX_RING_FLAGS_XDP,
+ ICE_TX_RING_FLAGS_VLAN_L2TAG1,
+ ICE_TX_RING_FLAGS_VLAN_L2TAG2,
+ ICE_TX_RING_FLAGS_TXTIME,
+ ICE_TX_RING_FLAGS_NBITS,
+};
+
struct ice_pkt_ctx {
u64 cached_phctime;
__be16 vlan_proto;
@@ -383,43 +391,52 @@ struct ice_rx_ring {
} ____cacheline_internodealigned_in_smp;
struct ice_tx_ring {
- /* CL1 - 1st cacheline starts here */
- struct ice_tx_ring *next; /* pointer to next ring in q_vector */
+ __cacheline_group_begin_aligned(read_mostly);
void *desc; /* Descriptor ring memory */
struct device *dev; /* Used for DMA mapping */
u8 __iomem *tail;
struct ice_tx_buf *tx_buf;
+
struct ice_q_vector *q_vector; /* Backreference to associated vector */
struct net_device *netdev; /* netdev ring maps to */
struct ice_vsi *vsi; /* Backreference to associated VSI */
- /* CL2 - 2nd cacheline starts here */
- dma_addr_t dma; /* physical address of ring */
- struct xsk_buff_pool *xsk_pool;
- u16 next_to_use;
- u16 next_to_clean;
- u16 q_handle; /* Queue handle per TC */
- u16 reg_idx; /* HW register index of the ring */
+
u16 count; /* Number of descriptors */
u16 q_index; /* Queue number of ring */
- u16 xdp_tx_active;
+
+ DECLARE_BITMAP(flags, ICE_TX_RING_FLAGS_NBITS);
+
+ struct xsk_buff_pool *xsk_pool;
+
/* stats structs */
struct ice_ring_stats *ring_stats;
- /* CL3 - 3rd cacheline starts here */
+ struct ice_tx_ring *next; /* pointer to next ring in q_vector */
+
+ struct ice_tstamp_ring *tstamp_ring;
+ struct ice_ptp_tx *tx_tstamps;
+ __cacheline_group_end_aligned(read_mostly);
+
+ __cacheline_group_begin_aligned(read_write);
+ u16 next_to_use;
+ u16 next_to_clean;
+
+ u16 xdp_tx_active;
+ spinlock_t tx_lock;
+ __cacheline_group_end_aligned(read_write);
+
+ __cacheline_group_begin_aligned(cold);
struct rcu_head rcu; /* to avoid race on free */
DECLARE_BITMAP(xps_state, ICE_TX_NBITS); /* XPS Config State */
struct ice_channel *ch;
- struct ice_ptp_tx *tx_tstamps;
- spinlock_t tx_lock;
- u32 txq_teid; /* Added Tx queue TEID */
- /* CL4 - 4th cacheline starts here */
- struct ice_tstamp_ring *tstamp_ring;
-#define ICE_TX_FLAGS_RING_XDP BIT(0)
-#define ICE_TX_FLAGS_RING_VLAN_L2TAG1 BIT(1)
-#define ICE_TX_FLAGS_RING_VLAN_L2TAG2 BIT(2)
-#define ICE_TX_FLAGS_TXTIME BIT(3)
- u8 flags;
+
+ dma_addr_t dma; /* physical address of ring */
+ u16 q_handle; /* Queue handle per TC */
+ u16 reg_idx; /* HW register index of the ring */
u8 dcb_tc; /* Traffic class of ring */
+
u16 quanta_prof_id;
+ u32 txq_teid; /* Added Tx queue TEID */
+ __cacheline_group_end_aligned(cold);
} ____cacheline_internodealigned_in_smp;
static inline bool ice_ring_uses_build_skb(struct ice_rx_ring *ring)
@@ -444,7 +461,7 @@ static inline bool ice_ring_ch_enabled(struct ice_tx_ring *ring)
static inline bool ice_ring_is_xdp(struct ice_tx_ring *ring)
{
- return !!(ring->flags & ICE_TX_FLAGS_RING_XDP);
+ return test_bit(ICE_TX_RING_FLAGS_XDP, ring->flags);
}
enum ice_container_type {
diff --git a/drivers/net/ethernet/intel/idpf/idpf_dev.c b/drivers/net/ethernet/intel/idpf/idpf_dev.c
index ee93987f9018..1df5e3f75606 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_dev.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_dev.c
@@ -88,7 +88,7 @@ static int idpf_intr_reg_init(struct idpf_vport *vport,
if (!reg_vals)
return -ENOMEM;
- num_regs = idpf_get_reg_intr_vecs(vport, reg_vals);
+ num_regs = idpf_get_reg_intr_vecs(vport, reg_vals, total_vecs);
if (num_regs < num_vecs) {
err = -EINVAL;
goto free_reg_vals;
diff --git a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c b/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
index bc47e194fbd2..9617911a8730 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
@@ -87,7 +87,7 @@ static int idpf_vf_intr_reg_init(struct idpf_vport *vport,
if (!reg_vals)
return -ENOMEM;
- num_regs = idpf_get_reg_intr_vecs(vport, reg_vals);
+ num_regs = idpf_get_reg_intr_vecs(vport, reg_vals, total_vecs);
if (num_regs < num_vecs) {
err = -EINVAL;
goto free_reg_vals;
diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
index 963ffb1b62cc..7d9c46ab0ca5 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
@@ -1310,11 +1310,12 @@ idpf_vport_init_queue_reg_chunks(struct idpf_vport_config *vport_config,
* idpf_get_reg_intr_vecs - Get vector queue register offset
* @vport: virtual port structure
* @reg_vals: Register offsets to store in
+ * @num_vecs: number of entries the @reg_vals array can hold
*
* Returns number of registers that got populated
*/
int idpf_get_reg_intr_vecs(struct idpf_vport *vport,
- struct idpf_vec_regs *reg_vals)
+ struct idpf_vec_regs *reg_vals, int num_vecs)
{
struct virtchnl2_vector_chunks *chunks;
struct idpf_vec_regs reg_val;
@@ -1338,7 +1339,7 @@ int idpf_get_reg_intr_vecs(struct idpf_vport *vport,
dynctl_reg_spacing = le32_to_cpu(chunk->dynctl_reg_spacing);
itrn_reg_spacing = le32_to_cpu(chunk->itrn_reg_spacing);
- for (i = 0; i < num_vec; i++) {
+ for (i = 0; i < num_vec && num_regs < num_vecs; i++) {
reg_vals[num_regs].dyn_ctl_reg = reg_val.dyn_ctl_reg;
reg_vals[num_regs].itrn_reg = reg_val.itrn_reg;
reg_vals[num_regs].itrn_index_spacing =
@@ -3496,7 +3497,6 @@ int idpf_vc_core_init(struct idpf_adapter *adapter)
pci_sriov_set_totalvfs(adapter->pdev, idpf_get_max_vfs(adapter));
num_max_vports = idpf_get_max_vports(adapter);
- adapter->max_vports = num_max_vports;
adapter->vports = kcalloc(num_max_vports, sizeof(*adapter->vports),
GFP_KERNEL);
if (!adapter->vports)
@@ -3519,6 +3519,12 @@ int idpf_vc_core_init(struct idpf_adapter *adapter)
goto err_netdev_alloc;
}
+ /* Set max_vports only after vports, netdevs and vport_config buffers
+ * are allocated to make sure max_vport bound loops don't end up
+ * crashing, following allocation errors on init.
+ */
+ adapter->max_vports = num_max_vports;
+
/* Start the mailbox task before requesting vectors. This will ensure
* vector information response from mailbox is handled
*/
diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
index dc2d50cfb338..f58be7329a4f 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
+++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
@@ -104,7 +104,7 @@ int idpf_vc_core_init(struct idpf_adapter *adapter);
void idpf_vc_core_deinit(struct idpf_adapter *adapter);
int idpf_get_reg_intr_vecs(struct idpf_vport *vport,
- struct idpf_vec_regs *reg_vals);
+ struct idpf_vec_regs *reg_vals, int num_vecs);
int idpf_queue_reg_init(struct idpf_vport *vport,
struct idpf_queue_id_reg_info *chunks);
int idpf_vport_queue_ids_init(struct idpf_vport *vport,
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 240f7cc3f213..f950570909ab 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -3958,7 +3958,8 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter,
}
/* initialize XPS */
- if (!test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) {
+ if (!ring_is_xdp(ring) &&
+ !test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) {
struct ixgbe_q_vector *q_vector = ring->q_vector;
if (q_vector)
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
index c6b5f6564ed0..5233b2e18695 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
@@ -518,6 +518,18 @@ int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind)
return 0;
}
+int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind)
+{
+ struct cgx *cgx = cgxd;
+
+ if (!is_lmac_valid(cgx, lmac_id))
+ return -ENODEV;
+
+ *pkind = cgx_read(cgx, lmac_id, cgx->mac_ops->rxid_map_offset);
+ *pkind = *pkind & 0x3F;
+ return 0;
+}
+
static u8 cgx_get_lmac_type(void *cgxd, int lmac_id)
{
struct cgx *cgx = cgxd;
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
index 92ccf343dfe0..8411a75dd723 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
@@ -141,6 +141,7 @@ int cgx_get_cgxid(void *cgxd);
int cgx_get_lmac_cnt(void *cgxd);
void *cgx_get_pdata(int cgx_id);
int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind);
+int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind);
int cgx_lmac_evh_register(struct cgx_event_cb *cb, void *cgxd, int lmac_id);
int cgx_lmac_evh_unregister(void *cgxd, int lmac_id);
int cgx_get_tx_stats(void *cgxd, int lmac_id, int idx, u64 *tx_stat);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cn20k/api.h b/drivers/net/ethernet/marvell/octeontx2/af/cn20k/api.h
index 4285b5d6a6a2..f36a1d5f236f 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cn20k/api.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cn20k/api.h
@@ -21,6 +21,7 @@ int cn20k_rvu_mbox_init(struct rvu *rvu, int type, int num);
int cn20k_rvu_get_mbox_regions(struct rvu *rvu, void **mbox_addr,
int num, int type, unsigned long *pf_bmap);
void cn20k_free_mbox_memory(struct rvu *rvu);
+void cn20k_free_mbox_memory_type(struct rvu *rvu, int type);
int cn20k_register_afpf_mbox_intr(struct rvu *rvu);
int cn20k_register_afvf_mbox_intr(struct rvu *rvu, int pf_vec_start);
void cn20k_rvu_enable_mbox_intr(struct rvu *rvu);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cn20k/mbox_init.c b/drivers/net/ethernet/marvell/octeontx2/af/cn20k/mbox_init.c
index bd3aab7770dd..ded91a91bb3d 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cn20k/mbox_init.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cn20k/mbox_init.c
@@ -335,13 +335,30 @@ int cn20k_rvu_mbox_init(struct rvu *rvu, int type, int ndevs)
return rvu_alloc_mbox_memory(rvu, type, ndevs, MBOX_SIZE);
}
+void cn20k_free_mbox_memory_type(struct rvu *rvu, int type)
+{
+ if (!is_cn20k(rvu->pdev) || !rvu->ng_rvu)
+ return;
+
+ switch (type) {
+ case TYPE_AFPF:
+ qmem_free(rvu->dev, rvu->ng_rvu->pf_mbox_addr);
+ rvu->ng_rvu->pf_mbox_addr = NULL;
+ break;
+ case TYPE_AFVF:
+ qmem_free(rvu->dev, rvu->ng_rvu->vf_mbox_addr);
+ rvu->ng_rvu->vf_mbox_addr = NULL;
+ break;
+ }
+}
+
void cn20k_free_mbox_memory(struct rvu *rvu)
{
if (!is_cn20k(rvu->pdev))
return;
- qmem_free(rvu->dev, rvu->ng_rvu->pf_mbox_addr);
- qmem_free(rvu->dev, rvu->ng_rvu->vf_mbox_addr);
+ cn20k_free_mbox_memory_type(rvu, TYPE_AFPF);
+ cn20k_free_mbox_memory_type(rvu, TYPE_AFVF);
}
void cn20k_rvu_disable_afvf_intr(struct rvu *rvu, int vfs)
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu.c
index 0590313bcd81..bf6440c55ea2 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.c
@@ -2516,12 +2516,6 @@ static int rvu_mbox_init(struct rvu *rvu, struct mbox_wq_info *mw,
if (!pf_bmap)
return -ENOMEM;
- ng_rvu_mbox = kzalloc(sizeof(*ng_rvu_mbox), GFP_KERNEL);
- if (!ng_rvu_mbox) {
- err = -ENOMEM;
- goto free_bitmap;
- }
-
/* RVU VFs */
if (type == TYPE_AFVF)
bitmap_set(pf_bmap, 0, num);
@@ -2535,15 +2529,22 @@ static int rvu_mbox_init(struct rvu *rvu, struct mbox_wq_info *mw,
}
}
- rvu->ng_rvu = ng_rvu_mbox;
+ if (!rvu->ng_rvu) {
+ ng_rvu_mbox = devm_kzalloc(rvu->dev, sizeof(*ng_rvu_mbox), GFP_KERNEL);
+ if (!ng_rvu_mbox) {
+ err = -ENOMEM;
+ goto free_bitmap;
+ }
+
+ rvu->ng_rvu = ng_rvu_mbox;
- rvu->ng_rvu->rvu_mbox_ops = &rvu_mbox_ops;
+ rvu->ng_rvu->rvu_mbox_ops = &rvu_mbox_ops;
+ mutex_init(&rvu->mbox_lock);
+ }
err = cn20k_rvu_mbox_init(rvu, type, num);
if (err)
- goto free_mem;
-
- mutex_init(&rvu->mbox_lock);
+ goto free_bitmap;
mbox_regions = kcalloc(num, sizeof(void __iomem *), GFP_KERNEL);
if (!mbox_regions) {
@@ -2633,14 +2634,18 @@ static int rvu_mbox_init(struct rvu *rvu, struct mbox_wq_info *mw,
free_regions:
kfree(mbox_regions);
free_qmem:
- cn20k_free_mbox_memory(rvu);
-free_mem:
- kfree(rvu->ng_rvu);
+ cn20k_free_mbox_memory_type(rvu, type);
free_bitmap:
bitmap_free(pf_bmap);
return err;
}
+static void rvu_free_cn20k_mbox_memory(struct rvu *rvu)
+{
+ if (is_cn20k(rvu->pdev))
+ cn20k_free_mbox_memory(rvu);
+}
+
static void rvu_mbox_destroy(struct mbox_wq_info *mw)
{
struct otx2_mbox *mbox = &mw->mbox;
@@ -3438,6 +3443,7 @@ static int rvu_enable_sriov(struct rvu *rvu)
if (err) {
rvu_disable_afvf_intr(rvu);
rvu_mbox_destroy(&rvu->afvf_wq_info);
+ cn20k_free_mbox_memory_type(rvu, TYPE_AFVF);
return err;
}
@@ -3590,6 +3596,7 @@ static int rvu_probe(struct pci_dev *pdev, const struct pci_device_id *id)
err_mbox:
rvu_mbox_destroy(&rvu->afpf_wq_info);
err_hwsetup:
+ rvu_free_cn20k_mbox_memory(rvu);
rvu_cgx_exit(rvu);
rvu_fwdata_exit(rvu);
rvu_mcs_exit(rvu);
@@ -3631,9 +3638,7 @@ static void rvu_remove(struct pci_dev *pdev)
pci_set_drvdata(pdev, NULL);
devm_kfree(&pdev->dev, rvu->hw);
- if (is_cn20k(rvu->pdev))
- cn20k_free_mbox_memory(rvu);
- kfree(rvu->ng_rvu);
+ rvu_free_cn20k_mbox_memory(rvu);
devm_kfree(&pdev->dev, rvu);
}
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
index c9ed5276690d..19fc110de4f4 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
@@ -1093,6 +1093,8 @@ void npc_read_mcam_entry(struct rvu *rvu, struct npc_mcam *mcam,
u8 *intf, u8 *ena);
int npc_config_cntr_default_entries(struct rvu *rvu, bool enable);
bool is_cgx_config_permitted(struct rvu *rvu, u16 pcifunc);
+bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind);
+bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc);
bool is_mac_feature_supported(struct rvu *rvu, int pf, int feature);
u32 rvu_cgx_get_fifolen(struct rvu *rvu);
void *rvu_first_cgx_pdata(struct rvu *rvu);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
index 3d91a34f8b57..6f1d242db88b 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
@@ -1355,3 +1355,82 @@ void rvu_mac_reset(struct rvu *rvu, u16 pcifunc)
if (mac_ops->mac_reset(cgxd, lmac, !is_vf(pcifunc)))
dev_err(rvu->dev, "Failed to reset MAC\n");
}
+
+/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds
+ * (HiGig, EDSA, etc.) are in use on the shared LMAC. VFs must not program
+ * NPC_TX_DEF_PKIND on NIX_AF_LFX_TX_PARSE_CFG in that case: the PF owns
+ * parse mode and no separate NPC_TX_HIGIG_PKIND is installed on the VF LF.
+ * TX-parse callers skip the write when denied; rvu_lf_reset() clears each LF
+ * before alloc so the next permitted owner programs NPC_TX_DEF_PKIND.
+ */
+bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc)
+{
+ int pf, err, rxpkind;
+ u8 cgx_id, lmac_id;
+ void *cgxd;
+
+ pf = rvu_get_pf(rvu->pdev, pcifunc);
+
+ if (!(pcifunc & RVU_PFVF_FUNC_MASK))
+ return true;
+
+ if (!is_pf_cgxmapped(rvu, pf))
+ return true;
+
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ err = cgx_get_pkind(cgxd, lmac_id, &rxpkind);
+ if (err)
+ return false;
+
+ switch (rxpkind) {
+ case NPC_RX_HIGIG_PKIND:
+ case NPC_RX_EDSA_PKIND:
+ return false;
+ default:
+ return true;
+ }
+}
+
+/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds
+ * (HiGig, EDSA, etc.) are in use on the shared LMAC.
+ */
+bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind)
+{
+ int pf, err, rxpkind;
+ u8 cgx_id, lmac_id;
+ struct cgx *cgxd;
+
+ pf = rvu_get_pf(rvu->pdev, pcifunc);
+
+ if (!is_pf_cgxmapped(rvu, pf))
+ return false;
+
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+
+ mutex_lock(&cgxd->lock);
+ if (!is_vf(pcifunc))
+ goto set;
+
+ err = cgx_get_pkind(cgxd, lmac_id, &rxpkind);
+ if (err)
+ goto err;
+
+ switch (rxpkind) {
+ case NPC_RX_HIGIG_PKIND:
+ case NPC_RX_EDSA_PKIND:
+ goto err;
+ default:
+ break;
+ }
+
+set:
+ cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind);
+ mutex_unlock(&cgxd->lock);
+ return true;
+
+err:
+ mutex_unlock(&cgxd->lock);
+ return false;
+}
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
index adedca065210..27c8a4319266 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
@@ -362,8 +362,8 @@ static int nix_interface_init(struct rvu *rvu, u16 pcifunc, int type, int nixlf,
pfvf->tx_chan_cnt = 1;
rsp->tx_link = cgx_id * hw->lmac_per_cgx + lmac_id;
- cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind);
- rvu_npc_set_pkind(rvu, pkind, pfvf);
+ if (rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, pkind))
+ rvu_npc_set_pkind(rvu, pkind, pfvf);
break;
case NIX_INTF_TYPE_LBK:
vf = (pcifunc & RVU_PFVF_FUNC_MASK) - 1;
@@ -1498,11 +1498,13 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu,
struct nix_lf_alloc_req *req,
struct nix_lf_alloc_rsp *rsp)
{
- int nixlf, qints, hwctx_size, intf, err, rc = 0;
+ int nixlf, qints, hwctx_size, intf, err, rc = 0, pf;
struct rvu_hwinfo *hw = rvu->hw;
u16 pcifunc = req->hdr.pcifunc;
+ u8 cgx_id = 0, lmac_id = 0;
struct rvu_block *block;
struct rvu_pfvf *pfvf;
+ struct cgx *cgxd;
u64 cfg, ctx_cfg;
int blkaddr;
@@ -1670,8 +1672,20 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu,
rvu_write64(rvu, blkaddr, NIX_AF_LFX_RX_CFG(nixlf), req->rx_cfg);
/* Configure pkind for TX parse config */
+
+ pf = rvu_get_pf(rvu->pdev, pcifunc);
cfg = NPC_TX_DEF_PKIND;
- rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+
+ if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) {
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ mutex_lock(&cgxd->lock);
+ if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc))
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+ mutex_unlock(&cgxd->lock);
+ } else {
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+ }
if (is_rep_dev(rvu, pcifunc)) {
pfvf->tx_chan_base = RVU_SWITCH_LBK_CHAN;
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
index fbe7dbe2fea5..2ded4a0a0c41 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
@@ -16,6 +16,7 @@
#include "cgx.h"
#include "npc_profile.h"
#include "rvu_npc_hash.h"
+#include "lmac_common.h"
#define RSVD_MCAM_ENTRIES_PER_PF 3 /* Broadcast, Promisc and AllMulticast */
#define RSVD_MCAM_ENTRIES_PER_NIXLF 1 /* Ucast for LFs */
@@ -3428,10 +3429,11 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
{
struct rvu_pfvf *pfvf = rvu_get_pfvf(rvu, pcifunc);
- int blkaddr, nixlf, rc, intf_mode;
int pf = rvu_get_pf(rvu->pdev, pcifunc);
+ int blkaddr, nixlf, rc, intf_mode;
+ u8 cgx_id = 0, lmac_id = 0;
u64 rxpkind, txpkind;
- u8 cgx_id, lmac_id;
+ struct cgx *cgxd;
/* use default pkind to disable edsa/higig */
rxpkind = rvu_npc_get_pkind(rvu, pf);
@@ -3455,12 +3457,8 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
/* rx pkind set req valid only for cgx mapped PFs */
if (!is_cgx_config_permitted(rvu, pcifunc))
return 0;
- rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
-
- rc = cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id,
- rxpkind);
- if (rc)
- return rc;
+ if (!rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, rxpkind))
+ return -EINVAL;
}
if (dir & PKIND_TX) {
@@ -3469,8 +3467,19 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
if (rc)
return rc;
- rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
- txpkind);
+ if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) {
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id,
+ &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ mutex_lock(&cgxd->lock);
+ if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc))
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
+ txpkind);
+ mutex_unlock(&cgxd->lock);
+ } else {
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
+ txpkind);
+ }
}
pfvf->intf_mode = intf_mode;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
index 2b05536d564a..e17283ee6a05 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
@@ -3,6 +3,7 @@
#include "rx.h"
#include "en/xdp.h"
+#include <linux/bitmap.h>
#include <net/xdp_sock_drv.h>
#include <linux/filter.h>
@@ -153,6 +154,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
xsk_buff_free(xsk_buffs[batch]);
err:
+ bitmap_fill(wi->skip_release_bitmap, rq->mpwqe.pages_per_wqe);
rq->stats->buff_alloc_err++;
return -ENOMEM;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 17b74a97138b..6fced90cae60 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -383,8 +383,11 @@ static inline void mlx5e_free_rx_wqe(struct mlx5e_rq *rq,
static void mlx5e_xsk_free_rx_wqe(struct mlx5e_wqe_frag_info *wi)
{
- if (!(wi->flags & BIT(MLX5E_WQE_FRAG_SKIP_RELEASE)))
- xsk_buff_free(*wi->xskp);
+ if (wi->flags & BIT(MLX5E_WQE_FRAG_SKIP_RELEASE))
+ return;
+
+ xsk_buff_free(*wi->xskp);
+ wi->flags |= BIT(MLX5E_WQE_FRAG_SKIP_RELEASE);
}
static void mlx5e_dealloc_rx_wqe(struct mlx5e_rq *rq, u16 ix)
@@ -1076,10 +1079,12 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
} while (!last_wqe);
} while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
- sq->cc = sqcc;
-
mlx5_cqwq_update_db_record(&cq->wq);
+ /* ensure cq space is freed before enabling more cqes */
+ dma_wmb();
+
+ sq->cc = sqcc;
return i;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c
index 8eaef73f3d77..bdd07c1c6824 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/adj_vport.c
@@ -11,26 +11,6 @@ int mlx5_esw_adj_vport_modify(struct mlx5_core_dev *dev, u16 vport,
lockdep_assert_held(&dev->priv.eswitch->state_lock);
- if (MLX5_CAP_ESW(dev, esw_vport_state_max_tx_speed)) {
- u8 op_mod = MLX5_VPORT_STATE_OP_MOD_ESW_VPORT;
- struct mlx5_vport *esw_vport;
- u32 speed = 0;
- int err;
-
- err = mlx5_query_vport_max_tx_speed(dev, op_mod, vport,
- true, &speed, NULL);
- if (err) {
- esw_vport = mlx5_eswitch_get_vport(dev->priv.eswitch,
- vport);
- speed = IS_ERR(esw_vport) ? 0 :
- esw_vport->agg_max_tx_speed;
- mlx5_core_dbg(dev,
- "Failed to query vport %d max tx speed, err=%d, using cached %u\n",
- vport, err, speed);
- }
- MLX5_SET(modify_vport_state_in, in, max_tx_speed, speed);
- }
-
MLX5_SET(modify_vport_state_in, in, opcode,
MLX5_CMD_OP_MODIFY_VPORT_STATE);
MLX5_SET(modify_vport_state_in, in, op_mod,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 353101dee4dc..17b9a5078f6c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -685,7 +685,7 @@ static int mlx5_lag_create_single_fdb(struct mlx5_lag *ldev)
}
return 0;
err:
- mlx5_ldev_for_each_reverse(j, i, first_idx + 1, ldev)
+ mlx5_ldev_for_each_reverse(j, i - 1, first_idx + 1, ldev)
mlx5_eswitch_offloads_single_fdb_del_one(master_esw,
ldev->pf[j].dev->priv.eswitch);
return err;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/vport.c b/drivers/net/ethernet/mellanox/mlx5/core/vport.c
index 5d895287df54..7dfaf3cb963c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/vport.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/vport.c
@@ -93,30 +93,6 @@ int mlx5_modify_vport_admin_state(struct mlx5_core_dev *mdev, u8 opmod,
lockdep_assert_held(&mdev->priv.eswitch->state_lock);
#endif
- if (MLX5_CAP_ESW(mdev, esw_vport_state_max_tx_speed) &&
- opmod == MLX5_VPORT_STATE_OP_MOD_ESW_VPORT &&
- vport != MLX5_VPORT_UPLINK) {
- u32 speed = 0;
- int err;
-
- err = mlx5_query_vport_max_tx_speed(mdev, opmod, vport,
- other_vport, &speed, NULL);
- if (err) {
-#ifdef CONFIG_MLX5_ESWITCH
- struct mlx5_vport *esw_vport;
-
- esw_vport = mlx5_eswitch_get_vport(mdev->priv.eswitch,
- vport);
- speed = IS_ERR(esw_vport) ? 0 :
- esw_vport->agg_max_tx_speed;
-#endif
- mlx5_core_dbg(mdev,
- "Failed to query vport %d max tx speed, err=%d, using cached %u\n",
- vport, err, speed);
- }
- MLX5_SET(modify_vport_state_in, in, max_tx_speed, speed);
- }
-
MLX5_SET(modify_vport_state_in, in, opcode,
MLX5_CMD_OP_MODIFY_VPORT_STATE);
MLX5_SET(modify_vport_state_in, in, op_mod, opmod);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
index b03ff9e044f9..10af08ce72ef 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
@@ -989,6 +989,8 @@ void mlxsw_sp_acl_ruleset_prio_get(struct mlxsw_sp_acl_ruleset *ruleset,
struct mlxsw_sp_acl_rule_info *
mlxsw_sp_acl_rulei_create(struct mlxsw_sp_acl *acl,
struct mlxsw_afa_block *afa_block);
+void mlxsw_sp_acl_rulei_free(struct mlxsw_sp *mlxsw_sp,
+ const struct mlxsw_sp_acl_rule_info *rulei);
void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_acl_rule_info *rulei);
int mlxsw_sp_acl_rulei_commit(struct mlxsw_sp_acl_rule_info *rulei);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c
index 3e70cee4d2f3..cac2f15d91c2 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c
@@ -340,8 +340,8 @@ mlxsw_sp_acl_rulei_create(struct mlxsw_sp_acl *acl,
return ERR_PTR(err);
}
-void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_acl_rule_info *rulei)
+void mlxsw_sp_acl_rulei_free(struct mlxsw_sp *mlxsw_sp,
+ const struct mlxsw_sp_acl_rule_info *rulei)
{
if (rulei->action_created)
mlxsw_afa_block_destroy(rulei->act_block);
@@ -351,6 +351,12 @@ void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp,
if (rulei->dst_port_range_reg_valid)
mlxsw_sp_port_range_reg_put(mlxsw_sp,
rulei->dst_port_range_reg_index);
+}
+
+void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei)
+{
+ mlxsw_sp_acl_rulei_free(mlxsw_sp, rulei);
kfree(rulei);
}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c
index 353fd9ca89a6..5e4632cb792d 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c
@@ -862,14 +862,19 @@ int mlxsw_sp_flower_tmplt_create(struct mlxsw_sp *mlxsw_sp,
memset(&rulei, 0, sizeof(rulei));
err = mlxsw_sp_flower_parse(mlxsw_sp, block, &rulei, f);
if (err)
- return err;
+ goto out;
+
ruleset = mlxsw_sp_acl_ruleset_get(mlxsw_sp, block,
f->common.chain_index,
MLXSW_SP_ACL_PROFILE_FLOWER,
&rulei.values.elusage);
+ err = PTR_ERR_OR_ZERO(ruleset);
+
+out:
+ mlxsw_sp_acl_rulei_free(mlxsw_sp, &rulei);
/* keep the reference to the ruleset */
- return PTR_ERR_OR_ZERO(ruleset);
+ return err;
}
void mlxsw_sp_flower_tmplt_destroy(struct mlxsw_sp *mlxsw_sp,
diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_ptp.c b/drivers/net/ethernet/microchip/sparx5/sparx5_ptp.c
index 5953e04ef14e..2826e48af103 100644
--- a/drivers/net/ethernet/microchip/sparx5/sparx5_ptp.c
+++ b/drivers/net/ethernet/microchip/sparx5/sparx5_ptp.c
@@ -622,18 +622,9 @@ int sparx5_ptp_init(struct sparx5 *sparx5)
sparx5->ptp = 1;
}
- if (!sparx5->ptp)
- return 0;
-
- for (i = 0; i < SPARX5_PHC_COUNT; ++i) {
- err = sparx5_ptp_phc_init(sparx5, i, &sparx5_ptp_clock_info);
- if (err)
- return err;
- }
-
- spin_lock_init(&sparx5->ptp_clock_lock);
- spin_lock_init(&sparx5->ptp_ts_id_lock);
- mutex_init(&sparx5->ptp_lock);
+ /* The base, non-PTP-capable lan969x variants need the
+ * first TOD counter running to forward frames.
+ */
/* Disable master counters */
spx5_wr(PTP_PTP_DOM_CFG_PTP_ENA_SET(0), sparx5, PTP_PTP_DOM_CFG);
@@ -657,6 +648,19 @@ int sparx5_ptp_init(struct sparx5 *sparx5)
/* Enable master counters */
spx5_wr(PTP_PTP_DOM_CFG_PTP_ENA_SET(0x7), sparx5, PTP_PTP_DOM_CFG);
+ if (!sparx5->ptp)
+ return 0;
+
+ for (i = 0; i < SPARX5_PHC_COUNT; ++i) {
+ err = sparx5_ptp_phc_init(sparx5, i, &sparx5_ptp_clock_info);
+ if (err)
+ return err;
+ }
+
+ spin_lock_init(&sparx5->ptp_clock_lock);
+ spin_lock_init(&sparx5->ptp_ts_id_lock);
+ mutex_init(&sparx5->ptp_lock);
+
for (i = 0; i < sparx5->data->consts->n_ports; i++) {
port = sparx5->ports[i];
if (!port)
diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_tc_matchall.c b/drivers/net/ethernet/microchip/sparx5/sparx5_tc_matchall.c
index 6b4d1d7b9730..9e24053711ac 100644
--- a/drivers/net/ethernet/microchip/sparx5/sparx5_tc_matchall.c
+++ b/drivers/net/ethernet/microchip/sparx5/sparx5_tc_matchall.c
@@ -160,6 +160,7 @@ static int sparx5_tc_matchall_destroy(struct net_device *ndev,
}
list_del(&entry->list);
+ kfree(entry);
return err;
}
diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c
index 81cbc47de0c6..29bbaebf3e57 100644
--- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
+++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
@@ -107,6 +107,7 @@ static int mana_gd_query_max_resources(struct pci_dev *pdev)
struct gdma_context *gc = pci_get_drvdata(pdev);
struct gdma_query_max_resources_resp resp = {};
struct gdma_general_req req = {};
+ unsigned int msix_vec_count;
int err;
mana_gd_init_req_hdr(&req.hdr, GDMA_QUERY_MAX_RESOURCES,
@@ -131,6 +132,24 @@ static int mana_gd_query_max_resources(struct pci_dev *pdev)
gc->num_msix_usable = min(resp.max_msix, num_online_cpus() + 1);
}
+ /* MSI-X vectors are allocated by index into the device MSI-X table, so
+ * never ask for more than the table holds. It can be smaller than both
+ * resp.max_msix and the CPU count.
+ */
+ err = pci_msix_vec_count(pdev);
+ if (err <= 0) {
+ dev_err(gc->dev, "Failed to query MSI-X table size: %d\n", err);
+ return err < 0 ? err : -ENOSPC;
+ }
+ msix_vec_count = err;
+
+ if (gc->num_msix_usable > msix_vec_count) {
+ dev_info(gc->dev,
+ "Limiting MSI-X vectors from %u to table size %u\n",
+ gc->num_msix_usable, msix_vec_count);
+ gc->num_msix_usable = msix_vec_count;
+ }
+
if (gc->num_msix_usable <= 1)
return -ENOSPC;
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index c3b32c3cbcc2..5cbe24a1596c 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -2535,9 +2535,12 @@ static void mana_destroy_rxq(struct mana_port_context *apc,
napi_disable_locked(napi);
netif_napi_del_locked(napi);
}
- xdp_rxq_info_unreg(&rxq->xdp_rxq);
- mana_destroy_wq_obj(apc, GDMA_RQ, rxq->rxobj);
+ if (xdp_rxq_info_is_reg(&rxq->xdp_rxq))
+ xdp_rxq_info_unreg(&rxq->xdp_rxq);
+
+ if (rxq->rxobj != INVALID_MANA_HANDLE)
+ mana_destroy_wq_obj(apc, GDMA_RQ, rxq->rxobj);
mana_deinit_cq(apc, &rxq->rx_cq);
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 8fa378bd591d..63c045616c9b 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -4100,6 +4100,7 @@ static bool stmmac_vlan_insert(struct stmmac_priv *priv, struct sk_buff *skb,
/**
* stmmac_tso_allocator - close entry point of the driver
* @priv: driver private structure
+ * @entry: TX queue buffer index
* @des: buffer start address
* @total_len: total length to fill in descriptors
* @last_segment: condition for the last descriptor
@@ -4108,8 +4109,9 @@ static bool stmmac_vlan_insert(struct stmmac_priv *priv, struct sk_buff *skb,
* This function fills descriptor and request new descriptors according to
* buffer length to fill
*/
-static void stmmac_tso_allocator(struct stmmac_priv *priv, dma_addr_t des,
- int total_len, bool last_segment, u32 queue)
+static void stmmac_tso_allocator(struct stmmac_priv *priv, u32 *entry,
+ dma_addr_t des, int total_len,
+ bool last_segment, u32 queue)
{
struct stmmac_tx_queue *tx_q = &priv->dma_conf.tx_queue[queue];
struct dma_desc *desc;
@@ -4121,14 +4123,13 @@ static void stmmac_tso_allocator(struct stmmac_priv *priv, dma_addr_t des,
while (tmp_len > 0) {
dma_addr_t curr_addr;
- tx_q->cur_tx = STMMAC_NEXT_ENTRY(tx_q->cur_tx,
- priv->dma_conf.dma_tx_size);
- WARN_ON(tx_q->tx_skbuff[tx_q->cur_tx]);
+ *entry = STMMAC_NEXT_ENTRY(*entry, priv->dma_conf.dma_tx_size);
+ WARN_ON(tx_q->tx_skbuff[*entry]);
if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc = &tx_q->dma_entx[tx_q->cur_tx].basic;
+ desc = &tx_q->dma_entx[*entry].basic;
else
- desc = &tx_q->dma_tx[tx_q->cur_tx];
+ desc = &tx_q->dma_tx[*entry];
curr_addr = des + (total_len - tmp_len);
stmmac_set_desc_addr(priv, desc, curr_addr);
@@ -4253,7 +4254,7 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct dma_desc *desc, *first, *mss_desc = NULL;
struct stmmac_priv *priv = netdev_priv(dev);
- unsigned int first_entry, tx_packets;
+ unsigned int first_entry, entry, tx_packets;
struct stmmac_txq_stats *txq_stats;
int i, first_tx, nfrags, ndesc;
struct stmmac_tx_queue *tx_q;
@@ -4328,22 +4329,24 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
}
first_entry = tx_q->cur_tx;
- WARN_ON(tx_q->tx_skbuff[first_entry]);
+ entry = first_entry;
+
+ WARN_ON(tx_q->tx_skbuff[entry]);
if (tx_q->tbs & STMMAC_TBS_AVAIL)
- desc = &tx_q->dma_entx[first_entry].basic;
+ desc = &tx_q->dma_entx[entry].basic;
else
- desc = &tx_q->dma_tx[first_entry];
+ desc = &tx_q->dma_tx[entry];
first = desc;
/* first descriptor: fill Headers on Buf1 */
des = dma_map_single(priv->device, skb->data, skb_headlen(skb),
DMA_TO_DEVICE);
if (dma_mapping_error(priv->device, des))
- goto dma_map_err;
+ goto error;
stmmac_set_desc_addr(priv, first, des);
- stmmac_tso_allocator(priv, des + proto_hdr_len, pay_len,
+ stmmac_tso_allocator(priv, &entry, des + proto_hdr_len, pay_len,
(nfrags == 0), queue);
/* In case two or more DMA transmit descriptors are allocated for this
@@ -4358,10 +4361,10 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
* this DMA buffer right after the DMA engine completely finishes the
* full buffer transmission.
*/
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf = des;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].len = skb_headlen(skb);
- tx_q->tx_skbuff_dma[tx_q->cur_tx].map_as_page = false;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
+ tx_q->tx_skbuff_dma[entry].buf = des;
+ tx_q->tx_skbuff_dma[entry].len = skb_headlen(skb);
+ tx_q->tx_skbuff_dma[entry].map_as_page = false;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
/* Prepare fragments */
for (i = 0; i < nfrags; i++) {
@@ -4371,16 +4374,17 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
skb_frag_size(frag),
DMA_TO_DEVICE);
if (dma_mapping_error(priv->device, des))
- goto dma_map_err;
+ goto error_dma_unmap;
- stmmac_tso_allocator(priv, des, skb_frag_size(frag),
+ stmmac_tso_allocator(priv, &entry, des, skb_frag_size(frag),
(i == nfrags - 1), queue);
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf = des;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].len = skb_frag_size(frag);
- tx_q->tx_skbuff_dma[tx_q->cur_tx].map_as_page = true;
- tx_q->tx_skbuff_dma[tx_q->cur_tx].buf_type = STMMAC_TXBUF_T_SKB;
+ tx_q->tx_skbuff_dma[entry].buf = des;
+ tx_q->tx_skbuff_dma[entry].len = skb_frag_size(frag);
+ tx_q->tx_skbuff_dma[entry].map_as_page = true;
+ tx_q->tx_skbuff_dma[entry].buf_type = STMMAC_TXBUF_T_SKB;
}
+ tx_q->cur_tx = entry;
tx_q->tx_skbuff_dma[tx_q->cur_tx].last_segment = true;
@@ -4477,7 +4481,22 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
return NETDEV_TX_OK;
-dma_map_err:
+error_dma_unmap:
+ for (;;) {
+ if (tx_q->tbs & STMMAC_TBS_AVAIL)
+ desc = &tx_q->dma_entx[first_entry].basic;
+ else
+ desc = &tx_q->dma_tx[first_entry];
+ stmmac_release_tx_desc(priv, desc, priv->mode);
+ stmmac_free_tx_buffer(priv, &priv->dma_conf, queue,
+ first_entry);
+ if (first_entry == entry)
+ break;
+
+ first_entry = STMMAC_NEXT_ENTRY(first_entry,
+ priv->dma_conf.dma_tx_size);
+ }
+error:
dev_err(priv->device, "Tx dma map failed\n");
dev_kfree_skb(skb);
priv->xstats.tx_dropped++;
diff --git a/drivers/net/ethernet/ti/icssg/icssg_common.c b/drivers/net/ethernet/ti/icssg/icssg_common.c
index 93bc054e3fba..c005ff7c72a0 100644
--- a/drivers/net/ethernet/ti/icssg/icssg_common.c
+++ b/drivers/net/ethernet/ti/icssg/icssg_common.c
@@ -544,15 +544,15 @@ void emac_rx_timestamp(struct prueth_emac *emac,
* emac_xmit_xdp_frame - transmits an XDP frame
* @emac: emac device
* @xdpf: data to transmit
- * @page: page from page pool if already DMA mapped
* @q_idx: queue id
+ * @buff_type: Type of buffer to be transmitted
*
* Return: XDP state
*/
u32 emac_xmit_xdp_frame(struct prueth_emac *emac,
struct xdp_frame *xdpf,
- struct page *page,
- unsigned int q_idx)
+ unsigned int q_idx,
+ enum prueth_tx_buff_type buff_type)
{
struct cppi5_host_desc_t *first_desc;
struct net_device *ndev = emac->ndev;
@@ -560,6 +560,7 @@ u32 emac_xmit_xdp_frame(struct prueth_emac *emac,
struct prueth_tx_chn *tx_chn;
dma_addr_t desc_dma, buf_dma;
struct prueth_swdata *swdata;
+ struct page *page;
u32 *epib;
int ret;
@@ -576,7 +577,12 @@ u32 emac_xmit_xdp_frame(struct prueth_emac *emac,
return ICSSG_XDP_CONSUMED; /* drop */
}
- if (page) { /* already DMA mapped by page_pool */
+ if (buff_type == PRUETH_TX_BUFF_TYPE_XDP_TX) { /* already DMA mapped by page_pool */
+ page = virt_to_head_page(xdpf->data);
+ if (unlikely(!page)) {
+ netdev_err(ndev, "xdp tx: failed to get page from xdpf\n");
+ goto drop_free_descs;
+ }
buf_dma = page_pool_get_dma_addr(page);
buf_dma += xdpf->headroom + sizeof(struct xdp_frame);
} else { /* Map the linear buffer */
@@ -631,13 +637,11 @@ EXPORT_SYMBOL_GPL(emac_xmit_xdp_frame);
* emac_run_xdp - run an XDP program
* @emac: emac device
* @xdp: XDP buffer containing the frame
- * @page: page with RX data if already DMA mapped
* @len: Rx descriptor packet length
*
* Return: XDP state
*/
-static u32 emac_run_xdp(struct prueth_emac *emac, struct xdp_buff *xdp,
- struct page *page, u32 *len)
+static u32 emac_run_xdp(struct prueth_emac *emac, struct xdp_buff *xdp, u32 *len)
{
struct net_device *ndev = emac->ndev;
struct netdev_queue *netif_txq;
@@ -664,7 +668,8 @@ static u32 emac_run_xdp(struct prueth_emac *emac, struct xdp_buff *xdp,
q_idx = cpu % emac->tx_ch_num;
netif_txq = netdev_get_tx_queue(ndev, q_idx);
__netif_tx_lock(netif_txq, cpu);
- result = emac_xmit_xdp_frame(emac, xdpf, page, q_idx);
+ result = emac_xmit_xdp_frame(emac, xdpf, q_idx,
+ PRUETH_TX_BUFF_TYPE_XDP_TX);
__netif_tx_unlock(netif_txq);
if (result == ICSSG_XDP_CONSUMED) {
ndev->stats.tx_dropped++;
@@ -689,7 +694,8 @@ static u32 emac_run_xdp(struct prueth_emac *emac, struct xdp_buff *xdp,
fallthrough; /* handle aborts by dropping packet */
case XDP_DROP:
ndev->stats.rx_dropped++;
- page_pool_recycle_direct(emac->rx_chns.pg_pool, page);
+ page_pool_recycle_direct(emac->rx_chns.pg_pool,
+ virt_to_head_page(xdp->data));
return ICSSG_XDP_CONSUMED;
}
}
@@ -719,8 +725,10 @@ static int emac_rx_packet(struct prueth_emac *emac, u32 flow_id, u32 *xdp_state)
return ret;
}
- if (cppi5_desc_is_tdcm(desc_dma)) /* Teardown ? */
+ if (cppi5_desc_is_tdcm(desc_dma)) {
+ complete(&emac->tdown_complete);
return 0;
+ }
desc_rx = k3_cppi_desc_pool_dma2virt(rx_chn->desc_pool, desc_dma);
swdata = cppi5_hdesc_get_swdata(desc_rx);
@@ -739,8 +747,6 @@ static int emac_rx_packet(struct prueth_emac *emac, u32 flow_id, u32 *xdp_state)
pkt_len -= 4;
cppi5_desc_get_tags_ids(&desc_rx->hdr, &port_id, NULL);
- k3_cppi_desc_pool_free(rx_chn->desc_pool, desc_rx);
-
/* if allocation fails we drop the packet but push the
* descriptor back to the ring with old page to prevent a stall
*/
@@ -756,7 +762,7 @@ static int emac_rx_packet(struct prueth_emac *emac, u32 flow_id, u32 *xdp_state)
xdp_init_buff(&xdp, PAGE_SIZE, &rx_chn->xdp_rxq);
xdp_prepare_buff(&xdp, pa, PRUETH_HEADROOM, pkt_len, false);
- *xdp_state = emac_run_xdp(emac, &xdp, page, &pkt_len);
+ *xdp_state = emac_run_xdp(emac, &xdp, &pkt_len);
if (*xdp_state != ICSSG_XDP_PASS)
goto requeue;
headroom = xdp.data - xdp.data_hard_start;
@@ -792,6 +798,7 @@ static int emac_rx_packet(struct prueth_emac *emac, u32 flow_id, u32 *xdp_state)
ndev->stats.rx_packets++;
requeue:
+ k3_cppi_desc_pool_free(rx_chn->desc_pool, desc_rx);
/* queue another RX DMA */
ret = prueth_dma_rx_push_mapped(emac, &emac->rx_chns, new_page,
PRUETH_MAX_PKT_SIZE);
@@ -804,7 +811,7 @@ static int emac_rx_packet(struct prueth_emac *emac, u32 flow_id, u32 *xdp_state)
return ret;
}
-static void prueth_rx_cleanup(void *data, dma_addr_t desc_dma)
+void prueth_rx_cleanup(void *data, dma_addr_t desc_dma)
{
struct prueth_rx_chn *rx_chn = data;
struct cppi5_host_desc_t *desc_rx;
@@ -822,6 +829,7 @@ static void prueth_rx_cleanup(void *data, dma_addr_t desc_dma)
k3_cppi_desc_pool_free(rx_chn->desc_pool, desc_rx);
}
+EXPORT_SYMBOL_GPL(prueth_rx_cleanup);
static int prueth_tx_ts_cookie_get(struct prueth_emac *emac)
{
@@ -1025,7 +1033,7 @@ enum netdev_tx icssg_ndo_start_xmit(struct sk_buff *skb, struct net_device *ndev
}
EXPORT_SYMBOL_GPL(icssg_ndo_start_xmit);
-static void prueth_tx_cleanup(void *data, dma_addr_t desc_dma)
+void prueth_tx_cleanup(void *data, dma_addr_t desc_dma)
{
struct prueth_tx_chn *tx_chn = data;
struct cppi5_host_desc_t *desc_tx;
@@ -1051,6 +1059,7 @@ static void prueth_tx_cleanup(void *data, dma_addr_t desc_dma)
prueth_xmit_free(tx_chn, desc_tx);
}
+EXPORT_SYMBOL_GPL(prueth_tx_cleanup);
irqreturn_t prueth_rx_irq(int irq, void *dev_id)
{
@@ -1223,15 +1232,13 @@ void icssg_ndo_tx_timeout(struct net_device *ndev, unsigned int txqueue)
}
EXPORT_SYMBOL_GPL(icssg_ndo_tx_timeout);
-static int emac_set_ts_config(struct net_device *ndev, struct ifreq *ifr)
+int icssg_ndo_set_ts_config(struct net_device *ndev,
+ struct kernel_hwtstamp_config *config,
+ struct netlink_ext_ack *extack)
{
struct prueth_emac *emac = netdev_priv(ndev);
- struct hwtstamp_config config;
- if (copy_from_user(&config, ifr->ifr_data, sizeof(config)))
- return -EFAULT;
-
- switch (config.tx_type) {
+ switch (config->tx_type) {
case HWTSTAMP_TX_OFF:
emac->tx_ts_enabled = 0;
break;
@@ -1242,7 +1249,7 @@ static int emac_set_ts_config(struct net_device *ndev, struct ifreq *ifr)
return -ERANGE;
}
- switch (config.rx_filter) {
+ switch (config->rx_filter) {
case HWTSTAMP_FILTER_NONE:
emac->rx_ts_enabled = 0;
break;
@@ -1262,43 +1269,28 @@ static int emac_set_ts_config(struct net_device *ndev, struct ifreq *ifr)
case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
case HWTSTAMP_FILTER_NTP_ALL:
emac->rx_ts_enabled = 1;
- config.rx_filter = HWTSTAMP_FILTER_ALL;
+ config->rx_filter = HWTSTAMP_FILTER_ALL;
break;
default:
return -ERANGE;
}
- return copy_to_user(ifr->ifr_data, &config, sizeof(config)) ?
- -EFAULT : 0;
+ return 0;
}
+EXPORT_SYMBOL_GPL(icssg_ndo_set_ts_config);
-static int emac_get_ts_config(struct net_device *ndev, struct ifreq *ifr)
+int icssg_ndo_get_ts_config(struct net_device *ndev,
+ struct kernel_hwtstamp_config *config)
{
struct prueth_emac *emac = netdev_priv(ndev);
- struct hwtstamp_config config;
- config.flags = 0;
- config.tx_type = emac->tx_ts_enabled ? HWTSTAMP_TX_ON : HWTSTAMP_TX_OFF;
- config.rx_filter = emac->rx_ts_enabled ? HWTSTAMP_FILTER_ALL : HWTSTAMP_FILTER_NONE;
+ config->flags = 0;
+ config->tx_type = emac->tx_ts_enabled ? HWTSTAMP_TX_ON : HWTSTAMP_TX_OFF;
+ config->rx_filter = emac->rx_ts_enabled ? HWTSTAMP_FILTER_ALL : HWTSTAMP_FILTER_NONE;
- return copy_to_user(ifr->ifr_data, &config, sizeof(config)) ?
- -EFAULT : 0;
-}
-
-int icssg_ndo_ioctl(struct net_device *ndev, struct ifreq *ifr, int cmd)
-{
- switch (cmd) {
- case SIOCGHWTSTAMP:
- return emac_get_ts_config(ndev, ifr);
- case SIOCSHWTSTAMP:
- return emac_set_ts_config(ndev, ifr);
- default:
- break;
- }
-
- return phy_do_ioctl(ndev, ifr, cmd);
+ return 0;
}
-EXPORT_SYMBOL_GPL(icssg_ndo_ioctl);
+EXPORT_SYMBOL_GPL(icssg_ndo_get_ts_config);
void icssg_ndo_get_stats64(struct net_device *ndev,
struct rtnl_link_stats64 *stats)
diff --git a/drivers/net/ethernet/ti/icssg/icssg_prueth.c b/drivers/net/ethernet/ti/icssg/icssg_prueth.c
index 07489564270b..ad3dec91e4da 100644
--- a/drivers/net/ethernet/ti/icssg/icssg_prueth.c
+++ b/drivers/net/ethernet/ti/icssg/icssg_prueth.c
@@ -743,6 +743,114 @@ static int icssg_update_vlan_mcast(struct net_device *vdev, int vid,
return 0;
}
+static void prueth_destroy_txq(struct prueth_emac *emac)
+{
+ int ret, i;
+
+ atomic_set(&emac->tdown_cnt, emac->tx_ch_num);
+ /* ensure new tdown_cnt value is visible */
+ smp_mb__after_atomic();
+ /* tear down and disable UDMA channels */
+ reinit_completion(&emac->tdown_complete);
+ for (i = 0; i < emac->tx_ch_num; i++)
+ k3_udma_glue_tdown_tx_chn(emac->tx_chns[i].tx_chn, false);
+
+ ret = wait_for_completion_timeout(&emac->tdown_complete,
+ msecs_to_jiffies(1000));
+ if (!ret)
+ netdev_err(emac->ndev, "tx teardown timeout\n");
+
+ for (i = 0; i < emac->tx_ch_num; i++) {
+ napi_disable(&emac->tx_chns[i].napi_tx);
+ hrtimer_cancel(&emac->tx_chns[i].tx_hrtimer);
+ k3_udma_glue_reset_tx_chn(emac->tx_chns[i].tx_chn,
+ &emac->tx_chns[i],
+ prueth_tx_cleanup);
+ k3_udma_glue_disable_tx_chn(emac->tx_chns[i].tx_chn);
+ }
+}
+
+static void prueth_destroy_rxq(struct prueth_emac *emac)
+{
+ int i, ret;
+
+ /* tear down and disable UDMA channels */
+ reinit_completion(&emac->tdown_complete);
+ k3_udma_glue_tdown_rx_chn(emac->rx_chns.rx_chn, true);
+
+ /* When RX DMA Channel Teardown is initiated, it will result in an
+ * interrupt and a Teardown Completion Marker (TDCM) is queued into
+ * the RX Completion queue. Acknowledging the interrupt involves
+ * popping the TDCM descriptor from the RX Completion queue via the
+ * RX NAPI Handler. To avoid timing out when waiting for the TDCM to
+ * be popped, schedule the RX NAPI handler to run immediately.
+ */
+ if (!napi_if_scheduled_mark_missed(&emac->napi_rx)) {
+ if (napi_schedule_prep(&emac->napi_rx))
+ __napi_schedule(&emac->napi_rx);
+ }
+
+ ret = wait_for_completion_timeout(&emac->tdown_complete,
+ msecs_to_jiffies(1000));
+ if (!ret)
+ netdev_err(emac->ndev, "rx teardown timeout\n");
+
+ for (i = 0; i < PRUETH_MAX_RX_FLOWS; i++) {
+ napi_disable(&emac->napi_rx);
+ hrtimer_cancel(&emac->rx_hrtimer);
+ k3_udma_glue_reset_rx_chn(emac->rx_chns.rx_chn, i,
+ &emac->rx_chns,
+ prueth_rx_cleanup);
+ }
+
+ prueth_destroy_xdp_rxqs(emac);
+ k3_udma_glue_disable_rx_chn(emac->rx_chns.rx_chn);
+}
+
+static int prueth_create_txq(struct prueth_emac *emac)
+{
+ int ret, i;
+
+ for (i = 0; i < emac->tx_ch_num; i++) {
+ ret = k3_udma_glue_enable_tx_chn(emac->tx_chns[i].tx_chn);
+ if (ret)
+ goto reset_tx_chan;
+ napi_enable(&emac->tx_chns[i].napi_tx);
+ }
+ return 0;
+
+reset_tx_chan:
+ /* Since interface is not yet up, there is wouldn't be
+ * any SKB for completion. So set false to free_skb
+ */
+ prueth_reset_tx_chan(emac, i, false);
+ return ret;
+}
+
+static int prueth_create_rxq(struct prueth_emac *emac)
+{
+ int ret;
+
+ ret = prueth_prepare_rx_chan(emac, &emac->rx_chns, PRUETH_MAX_PKT_SIZE);
+ if (ret)
+ return ret;
+
+ ret = k3_udma_glue_enable_rx_chn(emac->rx_chns.rx_chn);
+ if (ret)
+ goto reset_rx_chn;
+
+ ret = prueth_create_xdp_rxqs(emac);
+ if (ret)
+ goto reset_rx_chn;
+
+ napi_enable(&emac->napi_rx);
+ return 0;
+
+reset_rx_chn:
+ prueth_reset_rx_chan(&emac->rx_chns, PRUETH_MAX_RX_FLOWS, false);
+ return ret;
+}
+
/**
* emac_ndo_open - EMAC device open
* @ndev: network adapter device
@@ -754,7 +862,7 @@ static int icssg_update_vlan_mcast(struct net_device *vdev, int vid,
static int emac_ndo_open(struct net_device *ndev)
{
struct prueth_emac *emac = netdev_priv(ndev);
- int ret, i, num_data_chn = emac->tx_ch_num;
+ int ret, num_data_chn = emac->tx_ch_num;
struct icssg_flow_cfg __iomem *flow_cfg;
struct prueth *prueth = emac->prueth;
int slice = prueth_emac_slice(emac);
@@ -827,28 +935,13 @@ static int emac_ndo_open(struct net_device *ndev)
goto stop;
/* Prepare RX */
- ret = prueth_prepare_rx_chan(emac, &emac->rx_chns, PRUETH_MAX_PKT_SIZE);
+ ret = prueth_create_rxq(emac);
if (ret)
goto free_tx_ts_irq;
- ret = prueth_create_xdp_rxqs(emac);
- if (ret)
- goto reset_rx_chn;
-
- ret = k3_udma_glue_enable_rx_chn(emac->rx_chns.rx_chn);
+ ret = prueth_create_txq(emac);
if (ret)
- goto destroy_xdp_rxqs;
-
- for (i = 0; i < emac->tx_ch_num; i++) {
- ret = k3_udma_glue_enable_tx_chn(emac->tx_chns[i].tx_chn);
- if (ret)
- goto reset_tx_chan;
- }
-
- /* Enable NAPI in Tx and Rx direction */
- for (i = 0; i < emac->tx_ch_num; i++)
- napi_enable(&emac->tx_chns[i].napi_tx);
- napi_enable(&emac->napi_rx);
+ goto destroy_rxq;
/* start PHY */
phy_start(ndev->phydev);
@@ -859,15 +952,8 @@ static int emac_ndo_open(struct net_device *ndev)
return 0;
-reset_tx_chan:
- /* Since interface is not yet up, there is wouldn't be
- * any SKB for completion. So set false to free_skb
- */
- prueth_reset_tx_chan(emac, i, false);
-destroy_xdp_rxqs:
- prueth_destroy_xdp_rxqs(emac);
-reset_rx_chn:
- prueth_reset_rx_chan(&emac->rx_chns, max_rx_flows, false);
+destroy_rxq:
+ prueth_destroy_rxq(emac);
free_tx_ts_irq:
free_irq(emac->tx_ts_irq, emac);
stop:
@@ -897,9 +983,6 @@ static int emac_ndo_stop(struct net_device *ndev)
{
struct prueth_emac *emac = netdev_priv(ndev);
struct prueth *prueth = emac->prueth;
- int rx_flow = PRUETH_RX_FLOW_DATA;
- int max_rx_flows;
- int ret, i;
/* inform the upper layers. */
netif_tx_stop_all_queues(ndev);
@@ -913,32 +996,8 @@ static int emac_ndo_stop(struct net_device *ndev)
else
__dev_mc_unsync(ndev, icssg_prueth_del_mcast);
- atomic_set(&emac->tdown_cnt, emac->tx_ch_num);
- /* ensure new tdown_cnt value is visible */
- smp_mb__after_atomic();
- /* tear down and disable UDMA channels */
- reinit_completion(&emac->tdown_complete);
- for (i = 0; i < emac->tx_ch_num; i++)
- k3_udma_glue_tdown_tx_chn(emac->tx_chns[i].tx_chn, false);
-
- ret = wait_for_completion_timeout(&emac->tdown_complete,
- msecs_to_jiffies(1000));
- if (!ret)
- netdev_err(ndev, "tx teardown timeout\n");
-
- prueth_reset_tx_chan(emac, emac->tx_ch_num, true);
- for (i = 0; i < emac->tx_ch_num; i++) {
- napi_disable(&emac->tx_chns[i].napi_tx);
- hrtimer_cancel(&emac->tx_chns[i].tx_hrtimer);
- }
-
- max_rx_flows = PRUETH_MAX_RX_FLOWS;
- k3_udma_glue_tdown_rx_chn(emac->rx_chns.rx_chn, true);
-
- prueth_reset_rx_chan(&emac->rx_chns, max_rx_flows, true);
- prueth_destroy_xdp_rxqs(emac);
- napi_disable(&emac->napi_rx);
- hrtimer_cancel(&emac->rx_hrtimer);
+ prueth_destroy_txq(emac);
+ prueth_destroy_rxq(emac);
cancel_work_sync(&emac->rx_mode_work);
@@ -951,10 +1010,10 @@ static int emac_ndo_stop(struct net_device *ndev)
free_irq(emac->tx_ts_irq, emac);
- free_irq(emac->rx_chns.irq[rx_flow], emac);
+ free_irq(emac->rx_chns.irq[PRUETH_RX_FLOW_DATA], emac);
prueth_ndev_del_tx_napi(emac, emac->tx_ch_num);
- prueth_cleanup_rx_chns(emac, &emac->rx_chns, max_rx_flows);
+ prueth_cleanup_rx_chns(emac, &emac->rx_chns, PRUETH_MAX_RX_FLOWS);
prueth_cleanup_tx_chns(emac);
prueth->emacs_initialized--;
@@ -1116,7 +1175,8 @@ static int emac_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frame
__netif_tx_lock(netif_txq, cpu);
for (i = 0; i < n; i++) {
xdpf = frames[i];
- err = emac_xmit_xdp_frame(emac, xdpf, NULL, q_idx);
+ err = emac_xmit_xdp_frame(emac, xdpf, q_idx,
+ PRUETH_TX_BUFF_TYPE_XDP_NDO);
if (err != ICSSG_XDP_TX) {
ndev->stats.tx_dropped++;
break;
@@ -1176,7 +1236,7 @@ static const struct net_device_ops emac_netdev_ops = {
.ndo_validate_addr = eth_validate_addr,
.ndo_tx_timeout = icssg_ndo_tx_timeout,
.ndo_set_rx_mode = emac_ndo_set_rx_mode,
- .ndo_eth_ioctl = icssg_ndo_ioctl,
+ .ndo_eth_ioctl = phy_do_ioctl,
.ndo_get_stats64 = icssg_ndo_get_stats64,
.ndo_get_phys_port_name = icssg_ndo_get_phys_port_name,
.ndo_fix_features = emac_ndo_fix_features,
@@ -1184,6 +1244,8 @@ static const struct net_device_ops emac_netdev_ops = {
.ndo_vlan_rx_kill_vid = emac_ndo_vlan_rx_del_vid,
.ndo_bpf = emac_ndo_bpf,
.ndo_xdp_xmit = emac_xdp_xmit,
+ .ndo_hwtstamp_get = icssg_ndo_get_ts_config,
+ .ndo_hwtstamp_set = icssg_ndo_set_ts_config,
};
static int prueth_netdev_init(struct prueth *prueth,
diff --git a/drivers/net/ethernet/ti/icssg/icssg_prueth.h b/drivers/net/ethernet/ti/icssg/icssg_prueth.h
index ca8a22a4a5da..2092537f546e 100644
--- a/drivers/net/ethernet/ti/icssg/icssg_prueth.h
+++ b/drivers/net/ethernet/ti/icssg/icssg_prueth.h
@@ -148,6 +148,11 @@ enum prueth_swdata_type {
PRUETH_SWDATA_XDPF,
};
+enum prueth_tx_buff_type {
+ PRUETH_TX_BUFF_TYPE_XDP_TX,
+ PRUETH_TX_BUFF_TYPE_XDP_NDO,
+};
+
struct prueth_swdata {
enum prueth_swdata_type type;
union prueth_data {
@@ -479,7 +484,11 @@ void prueth_reset_tx_chan(struct prueth_emac *emac, int ch_num,
void prueth_reset_rx_chan(struct prueth_rx_chn *chn,
int num_flows, bool disable);
void icssg_ndo_tx_timeout(struct net_device *ndev, unsigned int txqueue);
-int icssg_ndo_ioctl(struct net_device *ndev, struct ifreq *ifr, int cmd);
+int icssg_ndo_get_ts_config(struct net_device *ndev,
+ struct kernel_hwtstamp_config *config);
+int icssg_ndo_set_ts_config(struct net_device *ndev,
+ struct kernel_hwtstamp_config *config,
+ struct netlink_ext_ack *extack);
void icssg_ndo_get_stats64(struct net_device *ndev,
struct rtnl_link_stats64 *stats);
int icssg_ndo_get_phys_port_name(struct net_device *ndev, char *name,
@@ -495,7 +504,9 @@ void prueth_put_cores(struct prueth *prueth, int slice);
u64 icssg_ts_to_ns(u32 hi_sw, u32 hi, u32 lo, u32 cycle_time_ns);
u32 emac_xmit_xdp_frame(struct prueth_emac *emac,
struct xdp_frame *xdpf,
- struct page *page,
- unsigned int q_idx);
+ unsigned int q_idx,
+ enum prueth_tx_buff_type buff_type);
+void prueth_rx_cleanup(void *data, dma_addr_t desc_dma);
+void prueth_tx_cleanup(void *data, dma_addr_t desc_dma);
#endif /* __NET_TI_ICSSG_PRUETH_H */
diff --git a/drivers/net/ethernet/ti/icssg/icssg_prueth_sr1.c b/drivers/net/ethernet/ti/icssg/icssg_prueth_sr1.c
index 5e225310c9de..2a8c8847a6bd 100644
--- a/drivers/net/ethernet/ti/icssg/icssg_prueth_sr1.c
+++ b/drivers/net/ethernet/ti/icssg/icssg_prueth_sr1.c
@@ -747,9 +747,11 @@ static const struct net_device_ops emac_netdev_ops = {
.ndo_validate_addr = eth_validate_addr,
.ndo_tx_timeout = icssg_ndo_tx_timeout,
.ndo_set_rx_mode = emac_ndo_set_rx_mode_sr1,
- .ndo_eth_ioctl = icssg_ndo_ioctl,
+ .ndo_eth_ioctl = phy_do_ioctl,
.ndo_get_stats64 = icssg_ndo_get_stats64,
.ndo_get_phys_port_name = icssg_ndo_get_phys_port_name,
+ .ndo_hwtstamp_get = icssg_ndo_get_ts_config,
+ .ndo_hwtstamp_set = icssg_ndo_set_ts_config,
};
static int prueth_netdev_init(struct prueth *prueth,
diff --git a/drivers/net/ovpn/crypto.c b/drivers/net/ovpn/crypto.c
index 2e95f29514fc..7e545428900a 100644
--- a/drivers/net/ovpn/crypto.c
+++ b/drivers/net/ovpn/crypto.c
@@ -18,20 +18,12 @@
#include "crypto_aead.h"
#include "crypto.h"
-static void ovpn_ks_destroy_rcu(struct rcu_head *head)
-{
- struct ovpn_crypto_key_slot *ks;
-
- ks = container_of(head, struct ovpn_crypto_key_slot, rcu);
- ovpn_aead_crypto_key_slot_destroy(ks);
-}
-
void ovpn_crypto_key_slot_release(struct kref *kref)
{
struct ovpn_crypto_key_slot *ks;
ks = container_of(kref, struct ovpn_crypto_key_slot, refcount);
- call_rcu(&ks->rcu, ovpn_ks_destroy_rcu);
+ queue_rcu_work(ovpn_wq, &ks->free_work);
}
/* can only be invoked when all peer references have been dropped (i.e. RCU
diff --git a/drivers/net/ovpn/crypto.h b/drivers/net/ovpn/crypto.h
index 0e284fec3a75..e3feb16d5498 100644
--- a/drivers/net/ovpn/crypto.h
+++ b/drivers/net/ovpn/crypto.h
@@ -10,6 +10,8 @@
#ifndef _NET_OVPN_OVPNCRYPTO_H_
#define _NET_OVPN_OVPNCRYPTO_H_
+#include <linux/workqueue.h>
+
#include "pktid.h"
#include "proto.h"
@@ -45,8 +47,8 @@ struct ovpn_crypto_key_slot {
struct ovpn_pktid_recv pid_recv ____cacheline_aligned_in_smp;
struct ovpn_pktid_xmit pid_xmit ____cacheline_aligned_in_smp;
+ struct rcu_work free_work;
struct kref refcount;
- struct rcu_head rcu;
};
struct ovpn_crypto_state {
diff --git a/drivers/net/ovpn/crypto_aead.c b/drivers/net/ovpn/crypto_aead.c
index 2cca759feffa..aa3cd9837cc0 100644
--- a/drivers/net/ovpn/crypto_aead.c
+++ b/drivers/net/ovpn/crypto_aead.c
@@ -9,6 +9,7 @@
#include <crypto/aead.h>
#include <linux/skbuff.h>
+#include <linux/workqueue.h>
#include <net/ip.h>
#include <net/ipv6.h>
#include <net/udp.h>
@@ -294,13 +295,19 @@ static struct crypto_aead *ovpn_aead_init(const char *title,
return ERR_PTR(ret);
}
-void ovpn_aead_crypto_key_slot_destroy(struct ovpn_crypto_key_slot *ks)
+static void ovpn_aead_crypto_key_slot_free(struct ovpn_crypto_key_slot *ks)
{
- if (!ks)
- return;
-
crypto_free_aead(ks->encrypt);
crypto_free_aead(ks->decrypt);
+}
+
+static void ovpn_aead_crypto_key_slot_free_work(struct work_struct *work)
+{
+ struct ovpn_crypto_key_slot *ks;
+
+ ks = container_of(to_rcu_work(work), struct ovpn_crypto_key_slot,
+ free_work);
+ ovpn_aead_crypto_key_slot_free(ks);
kfree(ks);
}
@@ -334,6 +341,7 @@ ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc)
ks->encrypt = NULL;
ks->decrypt = NULL;
+ INIT_RCU_WORK(&ks->free_work, ovpn_aead_crypto_key_slot_free_work);
kref_init(&ks->refcount);
ks->key_id = kc->key_id;
@@ -367,7 +375,8 @@ ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc)
return ks;
destroy_ks:
- ovpn_aead_crypto_key_slot_destroy(ks);
+ ovpn_aead_crypto_key_slot_free(ks);
+ kfree(ks);
return ERR_PTR(ret);
}
diff --git a/drivers/net/ovpn/crypto_aead.h b/drivers/net/ovpn/crypto_aead.h
index 65a2ff307898..fae3b585a43b 100644
--- a/drivers/net/ovpn/crypto_aead.h
+++ b/drivers/net/ovpn/crypto_aead.h
@@ -22,7 +22,6 @@ int ovpn_aead_decrypt(struct ovpn_peer *peer, struct ovpn_crypto_key_slot *ks,
struct ovpn_crypto_key_slot *
ovpn_aead_crypto_key_slot_new(const struct ovpn_key_config *kc);
-void ovpn_aead_crypto_key_slot_destroy(struct ovpn_crypto_key_slot *ks);
enum ovpn_cipher_alg ovpn_aead_crypto_alg(struct ovpn_crypto_key_slot *ks);
diff --git a/drivers/net/pfcp.c b/drivers/net/pfcp.c
index c1c8eaea63fc..e61bf33c4ce6 100644
--- a/drivers/net/pfcp.c
+++ b/drivers/net/pfcp.c
@@ -64,6 +64,13 @@ static int pfcp_encap_recv(struct sock *sk, struct sk_buff *skb)
goto drop;
unparsed = pfcp_hdr(skb);
+ if (unparsed->flags & PFCP_SEID_FLAG) {
+ if (unlikely(!pskb_may_pull(skb, PFCP_HLEN +
+ offsetofend(struct pfcphdr_session,
+ seid))))
+ goto drop;
+ unparsed = pfcp_hdr(skb);
+ }
ip_tunnel_flags_zero(flags);
tun_dst = udp_tun_rx_dst(skb, sk->sk_family, flags, 0,
diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c
index 90737cb71892..d07e87a0974c 100644
--- a/drivers/net/ppp/pptp.c
+++ b/drivers/net/ppp/pptp.c
@@ -382,8 +382,8 @@ static int pptp_rcv(struct sk_buff *skb)
return NET_RX_DROP;
}
-static int pptp_bind(struct socket *sock, struct sockaddr *uservaddr,
- int sockaddr_len)
+static int pptp_bind(struct socket *sock, struct sockaddr_unsized *uservaddr,
+ int sockaddr_len)
{
struct sock *sk = sock->sk;
struct sockaddr_pppox *sp = (struct sockaddr_pppox *) uservaddr;
diff --git a/drivers/net/team/team_core.c b/drivers/net/team/team_core.c
index ff4dcd6035cd..489bb321b310 100644
--- a/drivers/net/team/team_core.c
+++ b/drivers/net/team/team_core.c
@@ -822,7 +822,8 @@ static struct list_head *__team_get_qom_list(struct team *team, u16 queue_id)
/*
* note: already called with rcu_read_lock
*/
-static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb)
+static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb,
+ bool *tx_success)
{
struct list_head *qom_list;
struct team_port *port;
@@ -830,11 +831,11 @@ static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb)
if (!team->queue_override_enabled || !skb->queue_mapping)
return false;
qom_list = __team_get_qom_list(team, skb->queue_mapping);
- list_for_each_entry_rcu(port, qom_list, qom_list) {
- if (!team_dev_queue_xmit(team, port, skb))
- return true;
- }
- return false;
+ port = list_first_or_null_rcu(qom_list, struct team_port, qom_list);
+ if (!port)
+ return false;
+ *tx_success = !team_dev_queue_xmit(team, port, skb);
+ return true;
}
static void __team_queue_override_port_del(struct team *team,
@@ -1700,8 +1701,7 @@ static netdev_tx_t team_xmit(struct sk_buff *skb, struct net_device *dev)
bool tx_success;
unsigned int len = skb->len;
- tx_success = team_queue_override_transmit(team, skb);
- if (!tx_success)
+ if (!team_queue_override_transmit(team, skb, &tx_success))
tx_success = READ_ONCE(team->ops.transmit)(team, skb);
if (tx_success) {
struct team_pcpu_stats *pcpu_stats;
diff --git a/drivers/net/team/team_mode_broadcast.c b/drivers/net/team/team_mode_broadcast.c
index 61d7d79f0c36..69dc25a5847a 100644
--- a/drivers/net/team/team_mode_broadcast.c
+++ b/drivers/net/team/team_mode_broadcast.c
@@ -37,6 +37,8 @@ static bool bc_transmit(struct team *team, struct sk_buff *skb)
ret = !team_dev_queue_xmit(team, last, skb);
if (!sum_ret)
sum_ret = ret;
+ } else {
+ dev_kfree_skb_any(skb);
}
return sum_ret;
}
diff --git a/drivers/net/usb/ax88179_178a.c b/drivers/net/usb/ax88179_178a.c
index cf40b604f135..9a6fd3726a6e 100644
--- a/drivers/net/usb/ax88179_178a.c
+++ b/drivers/net/usb/ax88179_178a.c
@@ -1463,11 +1463,11 @@ static int ax88179_rx_fixup(struct usbnet *dev, struct sk_buff *skb)
return 1;
}
- ax_skb = netdev_alloc_skb_ip_align(dev->net, pkt_len);
+ ax_skb = netdev_alloc_skb_ip_align(dev->net, pkt_len - 2);
if (!ax_skb)
return 0;
- skb_put(ax_skb, pkt_len);
- memcpy(ax_skb->data, skb->data + 2, pkt_len);
+ skb_put(ax_skb, pkt_len - 2);
+ memcpy(ax_skb->data, skb->data + 2, pkt_len - 2);
ax88179_rx_checksum(ax_skb, pkt_hdr);
usbnet_skb_return(dev, ax_skb);
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index fea031d491b3..148114ab85ac 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -1414,6 +1414,7 @@ static int veth_open(struct net_device *dev)
}
veth_set_xdp_features(dev);
+ veth_set_xdp_features(peer);
return 0;
}
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index f337fed30c6f..b7ab57f4e277 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -3525,17 +3525,31 @@ static void virtnet_rx_resume(struct virtnet_info *vi, struct receive_queue *rq)
static int virtnet_rx_resize(struct virtnet_info *vi,
struct receive_queue *rq, u32 ring_num)
{
+ unsigned int old_ring_num = virtqueue_get_vring_size(rq->vq);
+ struct xdp_buff **tmp_xsk_buffs = NULL;
int err, qindex;
qindex = rq - vi->rq;
+ if (rq->xsk_pool && ring_num > old_ring_num) {
+ tmp_xsk_buffs = kvzalloc_objs(*tmp_xsk_buffs, ring_num);
+ if (!tmp_xsk_buffs)
+ return -ENOMEM;
+ }
+
virtnet_rx_pause(vi, rq);
err = virtqueue_resize(rq->vq, ring_num, virtnet_rq_unmap_free_buf, NULL);
+
+ /* virtqueue_resize may have changed the size even if err != 0 */
+ if (tmp_xsk_buffs && virtqueue_get_vring_size(rq->vq) > old_ring_num)
+ swap(rq->xsk_buffs, tmp_xsk_buffs);
+
if (err)
netdev_err(vi->dev, "resize rx fail: rx queue index: %d err: %d\n", qindex, err);
virtnet_rx_resume(vi, rq);
+ kvfree(tmp_xsk_buffs);
return err;
}
diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
index 02cc79169945..580fdec50bd5 100644
--- a/drivers/net/vxlan/vxlan_core.c
+++ b/drivers/net/vxlan/vxlan_core.c
@@ -1894,6 +1894,7 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
if (n) {
struct vxlan_rdst *rdst = NULL;
+ u8 ha[ETH_ALEN] __aligned(2);
struct vxlan_fdb *f;
struct sk_buff *reply;
@@ -1902,8 +1903,10 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
+ neigh_ha_snapshot(ha, n, n->dev);
+
rcu_read_lock();
- f = vxlan_find_mac_tx(vxlan, n->ha, vni);
+ f = vxlan_find_mac_tx(vxlan, ha, vni);
if (f)
rdst = first_remote_rcu(f);
if (rdst && vxlan_addr_any(&rdst->remote_ip)) {
@@ -1915,7 +1918,7 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
rcu_read_unlock();
reply = arp_create(ARPOP_REPLY, ETH_P_ARP, sip, dev, tip, sha,
- n->ha, sha);
+ ha, sha);
neigh_release(n);
@@ -1948,7 +1951,8 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
#if IS_ENABLED(CONFIG_IPV6)
static struct sk_buff *vxlan_na_create(struct sk_buff *request,
- struct neighbour *n, bool isrouter)
+ struct neighbour *n, u8 *ha,
+ bool isrouter)
{
struct net_device *dev = request->dev;
struct sk_buff *reply;
@@ -1996,7 +2000,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
/* Ethernet header */
ether_addr_copy(eth_hdr(reply)->h_dest, daddr);
- ether_addr_copy(eth_hdr(reply)->h_source, n->ha);
+ ether_addr_copy(eth_hdr(reply)->h_source, ha);
eth_hdr(reply)->h_proto = htons(ETH_P_IPV6);
reply->protocol = htons(ETH_P_IPV6);
@@ -2025,7 +2029,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
na->icmph.icmp6_override = 1;
na->icmph.icmp6_solicited = 1;
na->target = ns->target;
- ether_addr_copy(&na->opt[2], n->ha);
+ ether_addr_copy(&na->opt[2], ha);
na->opt[0] = ND_OPT_TARGET_LL_ADDR;
na->opt[1] = na_olen >> 3;
@@ -2068,6 +2072,7 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
if (n) {
struct vxlan_rdst *rdst = NULL;
+ u8 ha[ETH_ALEN] __aligned(2);
struct vxlan_fdb *f;
struct sk_buff *reply;
@@ -2076,7 +2081,8 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
- f = vxlan_find_mac_tx(vxlan, n->ha, vni);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = vxlan_find_mac_tx(vxlan, ha, vni);
if (f)
rdst = first_remote_rcu(f);
if (rdst && vxlan_addr_any(&rdst->remote_ip)) {
@@ -2085,7 +2091,7 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
- reply = vxlan_na_create(skb, n,
+ reply = vxlan_na_create(skb, n, ha,
!!(f ? f->flags & NTF_ROUTER : 0));
neigh_release(n);
diff --git a/drivers/net/wan/hdlc_ppp.c b/drivers/net/wan/hdlc_ppp.c
index 22affdcb4cea..302ed27944e7 100644
--- a/drivers/net/wan/hdlc_ppp.c
+++ b/drivers/net/wan/hdlc_ppp.c
@@ -126,14 +126,12 @@ static inline struct proto *get_proto(struct net_device *dev, u16 pid)
static inline const char *proto_name(u16 pid)
{
switch (pid) {
- case PID_LCP:
- return "LCP";
case PID_IPCP:
return "IPCP";
case PID_IPV6CP:
return "IPV6CP";
default:
- return NULL;
+ return "LCP";
}
}
diff --git a/drivers/net/wireguard/noise.c b/drivers/net/wireguard/noise.c
index 7eb9a23a3d4d..7eca02e603fb 100644
--- a/drivers/net/wireguard/noise.c
+++ b/drivers/net/wireguard/noise.c
@@ -784,10 +784,11 @@ wg_noise_handshake_consume_response(struct message_handshake_response *src,
/* Success! Copy everything to peer */
down_write(&handshake->lock);
- /* It's important to check that the state is still the same, while we
- * have an exclusive lock.
+ /* Check that the state is the same and that this is still the
+ * initiation we started with, while we have an exclusive lock.
*/
- if (handshake->state != state) {
+ if (handshake->state != state ||
+ crypto_memneq(handshake->ephemeral_private, ephemeral_private, NOISE_PUBLIC_KEY_LEN)) {
up_write(&handshake->lock);
goto fail;
}
diff --git a/drivers/net/wireguard/queueing.h b/drivers/net/wireguard/queueing.h
index 79b6d70de236..663b19b78395 100644
--- a/drivers/net/wireguard/queueing.h
+++ b/drivers/net/wireguard/queueing.h
@@ -78,12 +78,14 @@ static inline void wg_reset_packet(struct sk_buff *skb, bool encapsulating)
u8 l4_hash = skb->l4_hash;
u8 sw_hash = skb->sw_hash;
u32 hash = skb->hash;
+ u8 tstamp_type = skb->tstamp_type;
skb_scrub_packet(skb, true);
memset(&skb->headers, 0, sizeof(skb->headers));
if (encapsulating) {
skb->l4_hash = l4_hash;
skb->sw_hash = sw_hash;
skb->hash = hash;
+ skb->tstamp_type = tstamp_type;
}
skb->queue_mapping = 0;
skb->nohdr = 0;
diff --git a/drivers/net/wireless/ath/ath12k/core.c b/drivers/net/wireless/ath/ath12k/core.c
index a02db6fd2354..dc969b2516ea 100644
--- a/drivers/net/wireless/ath/ath12k/core.c
+++ b/drivers/net/wireless/ath/ath12k/core.c
@@ -1733,29 +1733,6 @@ int ath12k_core_pre_init(struct ath12k_base *ab)
return 0;
}
-static int ath12k_core_panic_handler(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- struct ath12k_base *ab = container_of(nb, struct ath12k_base,
- panic_nb);
-
- return ath12k_hif_panic_handler(ab);
-}
-
-static int ath12k_core_panic_notifier_register(struct ath12k_base *ab)
-{
- ab->panic_nb.notifier_call = ath12k_core_panic_handler;
-
- return atomic_notifier_chain_register(&panic_notifier_list,
- &ab->panic_nb);
-}
-
-static void ath12k_core_panic_notifier_unregister(struct ath12k_base *ab)
-{
- atomic_notifier_chain_unregister(&panic_notifier_list,
- &ab->panic_nb);
-}
-
static inline
bool ath12k_core_hw_group_create_ready(struct ath12k_hw_group *ag)
{
@@ -2148,18 +2125,13 @@ int ath12k_core_init(struct ath12k_base *ab)
struct ath12k_hw_group *ag;
int ret;
- ret = ath12k_core_panic_notifier_register(ab);
- if (ret)
- ath12k_warn(ab, "failed to register panic handler: %d\n", ret);
-
mutex_lock(&ath12k_hw_group_mutex);
ag = ath12k_core_hw_group_assign(ab);
if (!ag) {
mutex_unlock(&ath12k_hw_group_mutex);
ath12k_warn(ab, "unable to get hw group\n");
- ret = -ENODEV;
- goto err_unregister_notifier;
+ return -ENODEV;
}
mutex_unlock(&ath12k_hw_group_mutex);
@@ -2184,8 +2156,6 @@ int ath12k_core_init(struct ath12k_base *ab)
err_unassign_hw_group:
ath12k_core_hw_group_unassign(ab);
-err_unregister_notifier:
- ath12k_core_panic_notifier_unregister(ab);
return ret;
}
@@ -2194,7 +2164,6 @@ void ath12k_core_deinit(struct ath12k_base *ab)
{
ath12k_core_hw_group_destroy(ab->ag);
ath12k_core_hw_group_unassign(ab);
- ath12k_core_panic_notifier_unregister(ab);
}
void ath12k_core_free(struct ath12k_base *ab)
diff --git a/drivers/net/wireless/ath/ath12k/core.h b/drivers/net/wireless/ath/ath12k/core.h
index 93cdfcaf95bd..67ddf84a8a55 100644
--- a/drivers/net/wireless/ath/ath12k/core.h
+++ b/drivers/net/wireless/ath/ath12k/core.h
@@ -15,7 +15,6 @@
#include <linux/ctype.h>
#include <linux/firmware.h>
#include <linux/of_reserved_mem.h>
-#include <linux/panic_notifier.h>
#include <linux/average.h>
#include <linux/of.h>
#include "qmi.h"
@@ -1221,8 +1220,6 @@ struct ath12k_base {
#endif /* CONFIG_ACPI */
- struct notifier_block panic_nb;
-
struct ath12k_hw_group *ag;
struct ath12k_wsi_info wsi_info;
enum ath12k_firmware_mode fw_mode;
diff --git a/drivers/net/wireless/ath/ath12k/dp_rx.c b/drivers/net/wireless/ath/ath12k/dp_rx.c
index 99d29eda26cf..79ec83256e38 100644
--- a/drivers/net/wireless/ath/ath12k/dp_rx.c
+++ b/drivers/net/wireless/ath/ath12k/dp_rx.c
@@ -3238,8 +3238,10 @@ static int ath12k_dp_rx_h_verify_tkip_mic(struct ath12k *ar, struct ath12k_peer
RX_FLAG_IV_STRIPPED | RX_FLAG_DECRYPTED;
skb_pull(msdu, hal_rx_desc_sz);
- if (unlikely(!ath12k_dp_rx_check_nwifi_hdr_len_valid(ab, rx_desc, msdu)))
+ if (unlikely(!ath12k_dp_rx_check_nwifi_hdr_len_valid(ab, rx_desc, msdu))) {
+ dev_kfree_skb_any(msdu);
return -EINVAL;
+ }
ath12k_dp_rx_h_ppdu(ar, &rx_info);
ath12k_dp_rx_h_undecap(ar, msdu, rx_desc,
diff --git a/drivers/net/wireless/ath/ath12k/hif.h b/drivers/net/wireless/ath/ath12k/hif.h
index e8840fab6061..42bb00c13ef3 100644
--- a/drivers/net/wireless/ath/ath12k/hif.h
+++ b/drivers/net/wireless/ath/ath12k/hif.h
@@ -30,7 +30,6 @@ struct ath12k_hif_ops {
void (*ce_irq_enable)(struct ath12k_base *ab);
void (*ce_irq_disable)(struct ath12k_base *ab);
void (*get_ce_msi_idx)(struct ath12k_base *ab, u32 ce_id, u32 *msi_idx);
- int (*panic_handler)(struct ath12k_base *ab);
void (*coredump_download)(struct ath12k_base *ab);
};
@@ -149,14 +148,6 @@ static inline void ath12k_hif_power_down(struct ath12k_base *ab, bool is_suspend
ab->hif.ops->power_down(ab, is_suspend);
}
-static inline int ath12k_hif_panic_handler(struct ath12k_base *ab)
-{
- if (!ab->hif.ops->panic_handler)
- return NOTIFY_DONE;
-
- return ab->hif.ops->panic_handler(ab);
-}
-
static inline void ath12k_hif_coredump_download(struct ath12k_base *ab)
{
if (ab->hif.ops->coredump_download)
diff --git a/drivers/net/wireless/ath/ath12k/mac.c b/drivers/net/wireless/ath/ath12k/mac.c
index 2f47054efa44..7f7eddfa9771 100644
--- a/drivers/net/wireless/ath/ath12k/mac.c
+++ b/drivers/net/wireless/ath/ath12k/mac.c
@@ -9745,7 +9745,7 @@ int ath12k_mac_vdev_create(struct ath12k *ar, struct ath12k_link_vif *arvif)
if (ret) {
ath12k_warn(ab, "failed to create WMI vdev %d: %d\n",
arvif->vdev_id, ret);
- return ret;
+ goto err;
}
ar->num_created_vdevs++;
@@ -9874,7 +9874,7 @@ int ath12k_mac_vdev_create(struct ath12k *ar, struct ath12k_link_vif *arvif)
if (ret) {
ath12k_warn(ar->ab, "failed to delete peer vdev_id %d addr %pM\n",
arvif->vdev_id, arvif->bssid);
- goto err;
+ goto err_vdev_del;
}
ret = ath12k_wait_for_peer_delete_done(ar, arvif->vdev_id,
@@ -9893,8 +9893,6 @@ int ath12k_mac_vdev_create(struct ath12k *ar, struct ath12k_link_vif *arvif)
ath12k_wmi_vdev_delete(ar, arvif->vdev_id);
ar->num_created_vdevs--;
- arvif->is_created = false;
- arvif->ar = NULL;
ar->allocated_vdev_map &= ~(1LL << arvif->vdev_id);
ab->free_vdev_map |= 1LL << arvif->vdev_id;
ab->free_vdev_stats_id_map &= ~(1LL << arvif->vdev_stats_id);
@@ -9903,6 +9901,7 @@ int ath12k_mac_vdev_create(struct ath12k *ar, struct ath12k_link_vif *arvif)
spin_unlock_bh(&ar->data_lock);
err:
+ arvif->is_created = false;
arvif->ar = NULL;
return ret;
}
diff --git a/drivers/net/wireless/ath/ath12k/pci.c b/drivers/net/wireless/ath/ath12k/pci.c
index 37f41635d3b0..c26c41f62feb 100644
--- a/drivers/net/wireless/ath/ath12k/pci.c
+++ b/drivers/net/wireless/ath/ath12k/pci.c
@@ -1523,13 +1523,6 @@ void ath12k_pci_power_down(struct ath12k_base *ab, bool is_suspend)
ath12k_pci_sw_reset(ab_pci->ab, false);
}
-static int ath12k_pci_panic_handler(struct ath12k_base *ab)
-{
- ath12k_pci_sw_reset(ab, false);
-
- return NOTIFY_OK;
-}
-
static const struct ath12k_hif_ops ath12k_pci_hif_ops = {
.start = ath12k_pci_start,
.stop = ath12k_pci_stop,
@@ -1547,7 +1540,6 @@ static const struct ath12k_hif_ops ath12k_pci_hif_ops = {
.ce_irq_enable = ath12k_pci_hif_ce_irq_enable,
.ce_irq_disable = ath12k_pci_hif_ce_irq_disable,
.get_ce_msi_idx = ath12k_pci_get_ce_msi_idx,
- .panic_handler = ath12k_pci_panic_handler,
#ifdef CONFIG_ATH12K_COREDUMP
.coredump_download = ath12k_pci_coredump_download,
#endif
diff --git a/drivers/net/wireless/ath/ath12k/wmi.c b/drivers/net/wireless/ath/ath12k/wmi.c
index f4392462ae3a..ade14ba6703f 100644
--- a/drivers/net/wireless/ath/ath12k/wmi.c
+++ b/drivers/net/wireless/ath/ath12k/wmi.c
@@ -190,6 +190,8 @@ static const struct ath12k_wmi_tlv_policy ath12k_wmi_tlv_policies[] = {
.min_len = sizeof(struct wmi_11d_new_cc_event) },
[WMI_TAG_PER_CHAIN_RSSI_STATS] = {
.min_len = sizeof(struct wmi_per_chain_rssi_stat_params) },
+ [WMI_TAG_PDEV_CSA_SWITCH_COUNT_STATUS_EVENT] = {
+ .min_len = sizeof(struct ath12k_wmi_pdev_csa_event) },
};
__le32 ath12k_wmi_tlv_hdr(u32 cmd, u32 len)
@@ -402,6 +404,13 @@ ath12k_wmi_tlv_parse_alloc(struct ath12k_base *ab,
return tb;
}
+static u32 ath12k_wmi_tlv_data_len(const void *data)
+{
+ const struct wmi_tlv *tlv = (const struct wmi_tlv *)data - 1;
+
+ return le32_get_bits(tlv->header, WMI_TLV_LEN);
+}
+
static int ath12k_wmi_cmd_send_nowait(struct ath12k_wmi_pdev *wmi, struct sk_buff *skb,
u32 cmd_id)
{
@@ -8529,12 +8538,19 @@ ath12k_wmi_process_csa_switch_count_event(struct ath12k_base *ab,
const u32 *vdev_ids)
{
u32 current_switch_count = le32_to_cpu(ev->current_switch_count);
+ u32 vdev_ids_len = ath12k_wmi_tlv_data_len(vdev_ids);
u32 num_vdevs = le32_to_cpu(ev->num_vdevs);
struct ieee80211_bss_conf *conf;
struct ath12k_link_vif *arvif;
struct ath12k_vif *ahvif;
int i;
+ if (num_vdevs > vdev_ids_len / sizeof(*vdev_ids)) {
+ ath12k_warn(ab, "csa switch count num_vdevs %u exceeds tlv array length %u\n",
+ num_vdevs, vdev_ids_len);
+ return;
+ }
+
rcu_read_lock();
for (i = 0; i < num_vdevs; i++) {
arvif = ath12k_mac_get_arvif_by_vdev_id(ab, vdev_ids[i]);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
index 048b5d500215..ac5643b8bbcb 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
@@ -1489,8 +1489,31 @@ void brcmf_detach(struct device *dev)
brcmf_bus_change_state(bus_if, BRCMF_BUS_DOWN);
/* make sure primary interface removed last */
for (i = BRCMF_MAX_IFS - 1; i > -1; i--) {
- if (drvr->iflist[i])
- brcmf_remove_interface(drvr->iflist[i], false);
+ struct brcmf_if *ifp = drvr->iflist[i];
+
+ if (!ifp)
+ continue;
+
+ if (ifp->ndev) {
+ brcmf_remove_interface(ifp, false);
+ continue;
+ }
+
+ /* The P2P device interface has no netdev. Its removal can
+ * race with NL80211_CMD_DEL_INTERFACE issued by a user space
+ * process that is exiting (e.g. wpa_supplicant), which ends
+ * in brcmf_p2p_del_vif() under RTNL and the wiphy mutex. If
+ * both paths remove the interface, the wdev is unregistered
+ * twice and the vif is used after being freed. Take the same
+ * locks and re-read iflist, so only one path removes it.
+ */
+ rtnl_lock();
+ wiphy_lock(drvr->wiphy);
+ ifp = drvr->iflist[i];
+ if (ifp)
+ brcmf_remove_interface(ifp, true);
+ wiphy_unlock(drvr->wiphy);
+ rtnl_unlock();
}
brcmf_bus_stop(drvr->bus_if);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
index e1752a513c73..c2511a5a11a4 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
@@ -1281,6 +1281,9 @@ static s32 brcmf_p2p_abort_action_frame(struct brcmf_cfg80211_info *cfg)
brcmf_dbg(TRACE, "Enter\n");
vif = p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif;
+ if (!vif)
+ vif = p2p->bss_idx[P2PAPI_BSSCFG_PRIMARY].vif;
+
err = brcmf_fil_bsscfg_data_set(vif->ifp, "actframe_abort", &int_val,
sizeof(s32));
if (err)
@@ -1817,6 +1820,7 @@ bool brcmf_p2p_send_action_frame(struct brcmf_if *ifp,
/* validate channel and p2p ies */
if (config_af_params.search_channel &&
IS_P2P_SOCIAL_CHANNEL(le32_to_cpu(af_params->channel)) &&
+ p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif &&
p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif->saved_ie.probe_req_ie_len) {
afx_hdl = &p2p->afx_hdl;
afx_hdl->peer_listen_chan = le32_to_cpu(af_params->channel);
diff --git a/drivers/net/wireless/marvell/libertas/tx.c b/drivers/net/wireless/marvell/libertas/tx.c
index 13d08022e414..27304a98787d 100644
--- a/drivers/net/wireless/marvell/libertas/tx.c
+++ b/drivers/net/wireless/marvell/libertas/tx.c
@@ -117,13 +117,6 @@ netdev_tx_t lbs_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
if (priv->wdev->iftype == NL80211_IFTYPE_MONITOR) {
struct tx_radiotap_hdr *rtap_hdr = (void *)skb->data;
- if (skb->len < sizeof(*rtap_hdr) + 4 + ETH_ALEN) {
- lbs_deb_tx("tx err: short monitor frame %u\n", skb->len);
- dev->stats.tx_dropped++;
- dev->stats.tx_errors++;
- goto free;
- }
-
/* set txpd fields from the radiotap header */
txpd->tx_control = cpu_to_le32(convert_radiotap_rate_to_mv(rtap_hdr->rate));
diff --git a/drivers/net/wireless/mediatek/mt76/channel.c b/drivers/net/wireless/mediatek/mt76/channel.c
index 6adaeaf63e46..065e613289e0 100644
--- a/drivers/net/wireless/mediatek/mt76/channel.c
+++ b/drivers/net/wireless/mediatek/mt76/channel.c
@@ -192,68 +192,74 @@ int mt76_switch_vif_chanctx(struct ieee80211_hw *hw,
int n_vifs,
enum ieee80211_chanctx_switch_mode mode)
{
- struct mt76_chanctx *old_ctx = (struct mt76_chanctx *)vifs->old_ctx->drv_priv;
- struct mt76_chanctx *new_ctx = (struct mt76_chanctx *)vifs->new_ctx->drv_priv;
- struct ieee80211_chanctx_conf *conf = vifs->new_ctx;
- struct mt76_phy *old_phy = old_ctx->phy;
- struct mt76_phy *phy = hw->priv;
+ struct ieee80211_vif_chanctx_switch *v;
+ struct mt76_chanctx *old_ctx, *new_ctx;
+ struct mt76_phy *old_phy, *phy = hw->priv;
struct mt76_dev *dev = phy->dev;
struct mt76_vif_link *mlink;
- bool update_chan;
+ bool need_update[__MT_MAX_BAND] = {};
int i, ret = 0;
- if (mode == CHANCTX_SWMODE_SWAP_CONTEXTS)
- phy = new_ctx->phy = dev->band_phys[conf->def.chan->band];
- else
- phy = new_ctx->phy;
- if (!phy)
- return -EINVAL;
+ for (i = 0; i < n_vifs; i++) {
+ v = &vifs[i];
+ new_ctx = (struct mt76_chanctx *)v->new_ctx->drv_priv;
+ if (mode == CHANCTX_SWMODE_SWAP_CONTEXTS)
+ phy = new_ctx->phy = dev->band_phys[v->new_ctx->def.chan->band];
+ else
+ phy = new_ctx->phy;
- update_chan = phy->chanctx != new_ctx;
- if (update_chan) {
- if (dev->scan.phy == phy)
- mt76_abort_scan(dev);
+ if (!phy)
+ return -EINVAL;
- cancel_delayed_work_sync(&phy->mac_work);
+ if (need_update[phy->band_idx])
+ continue;
+
+ if (phy->chanctx != new_ctx) {
+ if (dev->scan.phy == phy)
+ mt76_abort_scan(dev);
+
+ cancel_delayed_work_sync(&phy->mac_work);
+ need_update[phy->band_idx] = true;
+ }
}
mutex_lock(&dev->mutex);
- if (mode == CHANCTX_SWMODE_SWAP_CONTEXTS &&
- phy != old_phy && old_phy->chanctx == old_ctx)
- old_phy->chanctx = NULL;
+ for (i = 0; i < n_vifs; i++) {
+ v = &vifs[i];
+ old_ctx = (struct mt76_chanctx *)v->old_ctx->drv_priv;
+ old_phy = old_ctx->phy;
+
+ new_ctx = (struct mt76_chanctx *)v->new_ctx->drv_priv;
+ phy = new_ctx->phy;
- if (update_chan)
- ret = mt76_phy_update_channel(phy, vifs->new_ctx);
+ if (mode == CHANCTX_SWMODE_SWAP_CONTEXTS && old_phy->chanctx &&
+ old_phy->chanctx == old_ctx && phy != old_phy)
+ old_phy->chanctx = NULL;
- if (ret)
- goto out;
+ if (need_update[phy->band_idx]) {
+ ret = mt76_phy_update_channel(phy, v->new_ctx);
+ if (ret)
+ goto out;
- if (old_phy == phy)
- goto skip_link_replace;
+ need_update[phy->band_idx] = false;
+ }
- for (i = 0; i < n_vifs; i++) {
- mlink = mt76_vif_conf_link(dev, vifs[i].vif, vifs[i].link_conf);
+ mlink = mt76_vif_conf_link(dev, v->vif, v->link_conf);
if (!mlink)
continue;
- dev->drv->vif_link_remove(old_phy, vifs[i].vif,
- vifs[i].link_conf, mlink);
-
- ret = dev->drv->vif_link_add(phy, vifs[i].vif,
- vifs[i].link_conf, mlink);
- if (ret)
- goto out;
-
- }
+ if (old_phy != phy) {
+ dev->drv->vif_link_remove(old_phy, v->vif, v->link_conf,
+ mlink);
-skip_link_replace:
- for (i = 0; i < n_vifs; i++) {
- mlink = mt76_vif_conf_link(dev, vifs[i].vif, vifs[i].link_conf);
- if (!mlink)
- continue;
+ ret = dev->drv->vif_link_add(phy, v->vif, v->link_conf,
+ mlink);
+ if (ret)
+ goto out;
+ }
- mlink->ctx = vifs->new_ctx;
+ mlink->ctx = v->new_ctx;
}
out:
diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h
index 2a186b47659b..6ca5f3b2c6ba 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76.h
+++ b/drivers/net/wireless/mediatek/mt76/mt76.h
@@ -533,6 +533,7 @@ struct mt76_hw_cap {
#define MT_DRV_HW_MGMT_TXQ BIT(4)
#define MT_DRV_AMSDU_OFFLOAD BIT(5)
#define MT_DRV_IGNORE_TXS_FAILED BIT(6)
+#define MT_DRV_HW_PS_BUFFERING BIT(7)
struct mt76_driver_ops {
u32 drv_flags;
@@ -1402,6 +1403,13 @@ mtxq_to_txq(struct mt76_txq *mtxq)
return container_of(ptr, struct ieee80211_txq, drv_priv);
}
+/* peer-wide state uses the wcid of the primary link */
+static inline struct mt76_wcid *
+mt76_wcid_primary(struct mt76_wcid *wcid)
+{
+ return wcid->def_wcid ? wcid->def_wcid : wcid;
+}
+
static inline struct ieee80211_sta *
wcid_to_sta(struct mt76_wcid *wcid)
{
@@ -1518,6 +1526,8 @@ void mt76_release_buffered_frames(struct ieee80211_hw *hw,
u16 tids, int nframes,
enum ieee80211_frame_release_type reason,
bool more_data);
+void mt76_sta_ps_transition(struct mt76_dev *dev, struct mt76_wcid *wcid,
+ bool ps);
bool mt76_has_tx_pending(struct mt76_phy *phy);
int mt76_update_channel(struct mt76_phy *phy);
void mt76_update_survey(struct mt76_phy *phy);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/init.c b/drivers/net/wireless/mediatek/mt76/mt7603/init.c
index 86617a3e4328..1981201374fd 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7603/init.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7603/init.c
@@ -230,6 +230,7 @@ mt7603_mac_init(struct mt7603_dev *dev)
eth_broadcast_addr(bc_addr);
mt7603_wtbl_init(dev, MT7603_WTBL_RESERVED, -1, bc_addr);
+ mt76_wcid_init(&dev->global_sta.wcid, 0);
dev->global_sta.wcid.idx = MT7603_WTBL_RESERVED;
rcu_assign_pointer(dev->mt76.wcid[MT7603_WTBL_RESERVED],
&dev->global_sta.wcid);
diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h
index 0b02f34c40a4..5ed2fffbe107 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h
+++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.h
@@ -1057,6 +1057,7 @@ enum {
MCU_UNI_EVENT_IE_COUNTDOWN = 0x09,
MCU_UNI_EVENT_COREDUMP = 0x0a,
MCU_UNI_EVENT_BSS_BEACON_LOSS = 0x0c,
+ MCU_UNI_EVENT_PS_SYNC = 0x0d,
MCU_UNI_EVENT_SCAN_DONE = 0x0e,
MCU_UNI_EVENT_RDD_REPORT = 0x11,
MCU_UNI_EVENT_ROC = 0x27,
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/init.c b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
index f24784652013..75fc990f2dd7 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/init.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
@@ -1201,6 +1201,12 @@ static void mt7915_unregister_ext_phy(struct mt7915_dev *dev)
static void mt7915_stop_hardware(struct mt7915_dev *dev)
{
+ int i;
+
+ mt76_for_each_q_rx(&dev->mt76, i) {
+ if (!mt76_queue_is_wed_rro(&dev->mt76.q_rx[i]))
+ napi_disable(&dev->mt76.napi[i]);
+ }
mt7915_mcu_exit(dev);
mt76_connac2_tx_token_put(&dev->mt76);
mt7915_dma_cleanup(dev);
@@ -1293,6 +1299,8 @@ int mt7915_register_device(struct mt7915_dev *dev)
void mt7915_unregister_device(struct mt7915_dev *dev)
{
cancel_work_sync(&dev->dump_work);
+ cancel_work_sync(&dev->reset_work);
+ cancel_work_sync(&dev->rc_work);
mt7915_unregister_ext_phy(dev);
mt7915_coredump_unregister(dev);
mt7915_unregister_thermal(&dev->phy);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/main.c b/drivers/net/wireless/mediatek/mt76/mt7915/main.c
index 9d546eb8022a..ff14085dc331 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/main.c
@@ -802,6 +802,7 @@ __mt7915_drop_sta(void *ptr, u8 *mac, struct ieee80211_vif *vif)
msta = (struct mt7915_sta *)sta->drv_priv;
mt7915_mcu_add_sta(data->dev, vif, sta, CONN_STATE_DISCONNECT, false);
+ msta->wcid.tx_info &= ~MT_WCID_TX_INFO_SET;
msta->wcid.sta_disabled = 1;
msta->wcid.sta = 0;
}
@@ -866,6 +867,7 @@ int mt7915_mac_sta_event(struct mt76_dev *mdev, struct ieee80211_vif *vif,
mt7915_mac_twt_teardown_flow(dev, msta, i);
mt7915_mcu_add_sta(dev, vif, sta, CONN_STATE_DISCONNECT, false);
+ msta->wcid.tx_info &= ~MT_WCID_TX_INFO_SET;
msta->wcid.sta_disabled = 1;
msta->wcid.sta = 0;
return 0;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/main.c b/drivers/net/wireless/mediatek/mt76/mt7921/main.c
index ce11666a32db..35ece0f61733 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7921/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7921/main.c
@@ -1508,6 +1508,9 @@ static void mt7921_channel_switch_rx_beacon(struct ieee80211_hw *hw,
struct mt792x_vif *mvif = (struct mt792x_vif *)vif->drv_priv;
u16 beacon_interval = vif->bss_conf.beacon_int;
+ if (!dev->new_ctx)
+ return;
+
if (cfg80211_chandef_identical(&chsw->chandef,
&dev->new_ctx->def) &&
chsw->count) {
diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/mac.c b/drivers/net/wireless/mediatek/mt76/mt7925/mac.c
index a8572fdba634..ce81ac1c2c63 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7925/mac.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7925/mac.c
@@ -780,7 +780,8 @@ mt7925_mac_write_txwi(struct mt76_dev *dev, __le32 *txwi,
val = FIELD_PREP(MT_TXD1_WLAN_IDX, wcid->idx) |
FIELD_PREP(MT_TXD1_OWN_MAC, omac_idx);
- if (band_idx)
+ /* 0xff means BAND_AUTO to the firmware, and TGID cannot encode it */
+ if (band_idx && band_idx != 0xff)
val |= FIELD_PREP(MT_TXD1_TGID, band_idx);
txwi[1] = cpu_to_le32(val);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/main.c b/drivers/net/wireless/mediatek/mt76/mt7925/main.c
index 1eeb33981c74..7c63c7f5caf3 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7925/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7925/main.c
@@ -2089,6 +2089,18 @@ mt7925_change_vif_links(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
mvif->valid_links = new_links;
+ /* Restore the legacy BSS after disabling MLO */
+ if (old_links && !new_links) {
+ mt792x_mac_link_bss_remove(dev, &mvif->bss_conf,
+ &mvif->sta.deflink);
+ err = mt7925_mac_link_bss_add(dev, &vif->bss_conf,
+ &mvif->sta.deflink);
+ if (err < 0) {
+ mt792x_mutex_release(dev);
+ return err;
+ }
+ }
+
mt792x_mutex_release(dev);
return 0;
@@ -2218,6 +2230,40 @@ static void mt7925_stop(struct ieee80211_hw *hw, bool suspend)
mt792x_stop(hw, suspend);
}
+static void mt7925_sta_pre_rcu_remove(struct ieee80211_hw *hw,
+ struct ieee80211_vif *vif,
+ struct ieee80211_sta *sta)
+{
+ struct mt76_phy *phy = hw->priv;
+ struct mt76_dev *dev = phy->dev;
+ struct mt76_wcid *wcid = (struct mt76_wcid *)sta->drv_priv;
+
+ mutex_lock(&dev->mutex);
+ spin_lock_bh(&dev->status_lock);
+
+ if (ieee80211_vif_is_mld(vif)) {
+ struct mt792x_sta *msta = (struct mt792x_sta *)sta->drv_priv;
+ struct mt792x_vif *mvif = (struct mt792x_vif *)vif->drv_priv;
+ unsigned long valid = mvif->valid_links;
+ struct mt792x_link_sta *mlink;
+ unsigned int link_id;
+
+ for_each_set_bit(link_id, &valid, IEEE80211_MLD_MAX_NUM_LINKS) {
+ mlink = mt792x_sta_to_link(msta, link_id);
+ if (!mlink || !mlink->wcid.sta)
+ continue;
+ if (mlink->wcid.idx < ARRAY_SIZE(dev->wcid))
+ rcu_assign_pointer(dev->wcid[mlink->wcid.idx],
+ NULL);
+ }
+ } else {
+ rcu_assign_pointer(dev->wcid[wcid->idx], NULL);
+ }
+
+ spin_unlock_bh(&dev->status_lock);
+ mutex_unlock(&dev->mutex);
+}
+
const struct ieee80211_ops mt7925_ops = {
.tx = mt792x_tx,
.start = mt7925_start,
@@ -2230,7 +2276,7 @@ const struct ieee80211_ops mt7925_ops = {
.start_ap = mt7925_start_ap,
.stop_ap = mt7925_stop_ap,
.sta_state = mt76_sta_state,
- .sta_pre_rcu_remove = mt76_sta_pre_rcu_remove,
+ .sta_pre_rcu_remove = mt7925_sta_pre_rcu_remove,
.set_key = mt7925_set_key,
.sta_set_decap_offload = mt7925_sta_set_decap_offload,
#if IS_ENABLED(CONFIG_IPV6)
diff --git a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c
index d1aebadd50aa..042b4caf2f76 100644
--- a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c
+++ b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c
@@ -108,68 +108,98 @@ mt792x_asar_acpi_read_mtcl(struct mt792x_dev *dev, u8 **table, u8 *version)
return ret;
}
+/* A table's layout is decided by the version byte in MTCL, but some firmware
+ * labels a version 2 table as version 1. Accept whichever layout the table
+ * actually has: for every accepted entry count the two do not overlap, since
+ * one is 6 + 5n bytes and the other 7 + 7n.
+ */
+static bool
+mt792x_asar_fits(u8 *table, int len, int prelen, int sarlen, int nr_off,
+ int min, int max)
+{
+ int tblcnt;
+
+ if (len < prelen || (len - prelen) % sarlen)
+ return false;
+
+ tblcnt = (len - prelen) / sarlen;
+ if (tblcnt < min || tblcnt > max)
+ return false;
+
+ return tblcnt == table[nr_off];
+}
+
/* MTDS : Dynamic SAR Power Table */
static int
-mt792x_asar_acpi_read_mtds(struct mt792x_dev *dev, u8 **table, u8 version)
+mt792x_asar_acpi_read_mtds(struct mt792x_dev *dev, u8 **table, u8 version,
+ u8 *used)
{
- int len, ret, sarlen, prelen, tblcnt;
- bool enable;
+ int len, ret;
+ u8 *t;
ret = mt792x_acpi_read(dev, MT792x_ACPI_MTDS, table, &len);
if (ret)
return ret;
+ t = *table;
+
/* Table content validation */
- switch (version) {
- case 1:
- enable = ((struct mt792x_asar_dyn *)*table)->enable;
- sarlen = sizeof(struct mt792x_asar_dyn_limit);
- prelen = sizeof(struct mt792x_asar_dyn);
- break;
- case 2:
- enable = ((struct mt792x_asar_dyn_v2 *)*table)->enable;
- sarlen = sizeof(struct mt792x_asar_dyn_limit_v2);
- prelen = sizeof(struct mt792x_asar_dyn_v2);
- break;
- default:
+ if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_dyn),
+ sizeof(struct mt792x_asar_dyn_limit),
+ offsetof(struct mt792x_asar_dyn, nr_tbl),
+ MT792x_ASAR_MIN_DYN, MT792x_ASAR_MAX_DYN))
+ *used = 1;
+ else if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_dyn_v2),
+ sizeof(struct mt792x_asar_dyn_limit_v2),
+ offsetof(struct mt792x_asar_dyn_v2, nr_tbl),
+ MT792x_ASAR_MIN_DYN, MT792x_ASAR_MAX_DYN))
+ *used = 2;
+ else
return -EINVAL;
- }
- tblcnt = (len - prelen) / sarlen;
- if (!enable ||
- tblcnt > MT792x_ASAR_MAX_DYN || tblcnt < MT792x_ASAR_MIN_DYN)
+ if (!t[offsetof(struct mt792x_asar_dyn, enable)])
return -EINVAL;
+ if (version && *used != version)
+ dev_info(dev->mt76.dev,
+ "MTDS is v%u, MTCL says v%u; using v%u\n",
+ *used, version, *used);
+
return 0;
}
/* MTGS : Geo SAR Power Table */
static int
-mt792x_asar_acpi_read_mtgs(struct mt792x_dev *dev, u8 **table, u8 version)
+mt792x_asar_acpi_read_mtgs(struct mt792x_dev *dev, u8 **table, u8 version,
+ u8 *used)
{
- int len, ret, sarlen, prelen, tblcnt;
+ int len, ret;
+ u8 *t;
ret = mt792x_acpi_read(dev, MT792x_ACPI_MTGS, table, &len);
if (ret)
return ret;
+ t = *table;
+
/* Table content validation */
- switch (version) {
- case 1:
- sarlen = sizeof(struct mt792x_asar_geo_limit);
- prelen = sizeof(struct mt792x_asar_geo);
- break;
- case 2:
- sarlen = sizeof(struct mt792x_asar_geo_limit_v2);
- prelen = sizeof(struct mt792x_asar_geo_v2);
- break;
- default:
+ if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_geo),
+ sizeof(struct mt792x_asar_geo_limit),
+ offsetof(struct mt792x_asar_geo, nr_tbl),
+ MT792x_ASAR_MIN_GEO, MT792x_ASAR_MAX_GEO))
+ *used = 1;
+ else if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_geo_v2),
+ sizeof(struct mt792x_asar_geo_limit_v2),
+ offsetof(struct mt792x_asar_geo_v2, nr_tbl),
+ MT792x_ASAR_MIN_GEO, MT792x_ASAR_MAX_GEO))
+ *used = 2;
+ else
return -EINVAL;
- }
- tblcnt = (len - prelen) / sarlen;
- if (tblcnt > MT792x_ASAR_MAX_GEO || tblcnt < MT792x_ASAR_MIN_GEO)
- return -EINVAL;
+ if (version && *used != version)
+ dev_info(dev->mt76.dev,
+ "MTGS is v%u, MTCL says v%u; using v%u\n",
+ *used, version, *used);
return 0;
}
@@ -205,14 +235,16 @@ int mt792x_init_acpi_sar(struct mt792x_dev *dev)
asar->countrylist = NULL;
}
- ret = mt792x_asar_acpi_read_mtds(dev, (u8 **)&asar->dyn, asar->ver);
+ ret = mt792x_asar_acpi_read_mtds(dev, (u8 **)&asar->dyn, asar->ver,
+ &asar->dyn_ver);
if (ret) {
devm_kfree(dev->mt76.dev, asar->dyn);
asar->dyn = NULL;
}
/* MTGS is optional */
- ret = mt792x_asar_acpi_read_mtgs(dev, (u8 **)&asar->geo, asar->ver);
+ ret = mt792x_asar_acpi_read_mtgs(dev, (u8 **)&asar->geo, asar->ver,
+ &asar->geo_ver);
if (ret) {
devm_kfree(dev->mt76.dev, asar->geo);
asar->geo = NULL;
@@ -254,7 +286,7 @@ mt792x_asar_get_geo_pwr(struct mt792x_phy *phy,
break;
}
- if (asar->ver == 1) {
+ if (asar->geo_ver == 1) {
band_pwr = &asar->geo->tbl[idx].band[0];
max = ARRAY_SIZE(asar->geo->tbl[idx].band);
} else {
@@ -279,10 +311,18 @@ mt792x_asar_get_geo_pwr(struct mt792x_phy *phy,
if (idx >= max)
return dyn_power;
- geo_power = (band_pwr + idx)->pwr;
- dyn_power += (band_pwr + idx)->offset;
+ band_pwr += idx;
+
+ /* Some OEM tables carry 0xff here; narrowed to s8 that is -1 and
+ * wins the min() below. Take it as no limit.
+ */
+ if (band_pwr->pwr == 0xff)
+ return dyn_power;
+
+ geo_power = band_pwr->pwr;
- return min(geo_power, dyn_power);
+ return min_t(int, geo_power,
+ min_t(int, dyn_power + band_pwr->offset, 127));
}
static s8
@@ -297,7 +337,7 @@ mt792x_asar_range_pwr(struct mt792x_phy *phy,
if (!capa)
return 127;
- if (asar->ver == 1) {
+ if (asar->dyn_ver == 1) {
limit = &asar->dyn->tbl[0].frp[0];
max = ARRAY_SIZE(asar->dyn->tbl[0].frp);
} else {
@@ -315,7 +355,9 @@ mt792x_asar_range_pwr(struct mt792x_phy *phy,
else
band = NL80211_BAND_2GHZ;
- return mt792x_asar_get_geo_pwr(phy, band, limit[idx]);
+ /* 0xff: no dynamic limit, see mt792x_asar_get_geo_pwr() */
+ return mt792x_asar_get_geo_pwr(phy, band,
+ limit[idx] == 0xff ? 127 : limit[idx]);
}
int mt792x_init_acpi_sar_power(struct mt792x_phy *phy, bool set_default)
diff --git a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h
index e45dcd7fbdb1..1a33f66bfd5f 100644
--- a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h
+++ b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h
@@ -105,6 +105,8 @@ struct mt792x_asar_fg {
struct mt792x_acpi_sar {
u8 ver;
+ u8 dyn_ver;
+ u8 geo_ver;
union {
struct mt792x_asar_dyn *dyn;
struct mt792x_asar_dyn_v2 *dyn_v2;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/init.c b/drivers/net/wireless/mediatek/mt76/mt7996/init.c
index d0a4530db1cf..778a9ca8d5fa 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/init.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/init.c
@@ -1178,7 +1178,7 @@ static int mt7996_variant_fem_init(struct mt7996_dev *dev)
adie_ver = u32_get_bits(regval, MT_ADIE_VERSION_MASK);
idx = MT7976C_EFUSE_OFFSET % MT7996_EEPROM_BLOCK_SIZE;
is_7976c = adie_ver == 0x8a10 || adie_ver == 0x8b00 ||
- adie_ver == 0x8c10 || buf[idx] == 0xc;
+ adie_ver == 0x8c10 || (!ret && buf[idx] == 0xc);
adie_id = u32_get_bits(regval, MT_ADIE_CHIP_ID_MASK);
if (adie_id == 0x7975 || adie_id == 0x7979 ||
@@ -1736,6 +1736,8 @@ void mt7996_unregister_device(struct mt7996_dev *dev)
{
cancel_work_sync(&dev->dump_work);
cancel_work_sync(&dev->wed_rro.work);
+ cancel_work_sync(&dev->reset_work);
+ cancel_work_sync(&dev->rc_work);
mt7996_unregister_phy(mt7996_phy3(dev));
mt7996_unregister_phy(mt7996_phy2(dev));
mt7996_unregister_thermal(&dev->phy);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c
index dc1a3e91218a..ed3508afc084 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c
@@ -2503,6 +2503,7 @@ mt7996_mac_reset_vif_iter(void *data, u8 *mac, struct ieee80211_vif *vif)
rcu_assign_pointer(mvif->link[i], NULL);
kfree_rcu(mlink, rcu_head);
}
+ mvif->valid_links = 0;
rcu_read_unlock();
}
@@ -2537,6 +2538,8 @@ mt7996_mac_full_reset(struct mt7996_dev *dev)
mt7996_for_each_phy(dev, phy)
phy->omac_mask = 0;
+ dev->mld_idx_mask = 0;
+ dev->mld_remap_idx_mask = 0;
ieee80211_iterate_stations_atomic(hw, mt7996_mac_reset_sta_iter, dev);
mt76_reset_device(&dev->mt76);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/main.c b/drivers/net/wireless/mediatek/mt76/mt7996/main.c
index c6d7088b15ca..7812f03b3d38 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/main.c
@@ -1274,6 +1274,7 @@ mt7996_mac_sta_event(struct mt7996_dev *dev, struct ieee80211_vif *vif,
else if (sta->mlo && links == BIT(link_id)) /* last link */
mt7996_mcu_teardown_mld_sta(dev, link,
msta_link);
+ msta_link->wcid.tx_info &= ~MT_WCID_TX_INFO_SET;
msta_link->wcid.sta_disabled = 1;
msta_link->wcid.sta = 0;
links = links & ~BIT(link_id);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h
index 848c6c575115..29e8f6d64ede 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.h
@@ -249,6 +249,9 @@ struct mt7996_mcu_wed_rro_ba_delete_event {
__le16 len;
__le16 session_id;
+ __le16 wlan_id;
+ u8 tid;
+ u8 __rsv1;
u8 __rsv2[2];
} __packed;
diff --git a/drivers/net/wireless/mediatek/mt76/tx.c b/drivers/net/wireless/mediatek/mt76/tx.c
index 28630d75bdb2..731caeda72b5 100644
--- a/drivers/net/wireless/mediatek/mt76/tx.c
+++ b/drivers/net/wireless/mediatek/mt76/tx.c
@@ -227,14 +227,29 @@ mt76_tx_check_non_aql(struct mt76_dev *dev, struct mt76_wcid *wcid,
struct sk_buff *skb)
{
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+ struct ieee80211_sta *sta;
int pending;
+ int i;
if (!wcid || info->tx_time_est)
return;
+ wcid = mt76_wcid_primary(wcid);
+
pending = atomic_dec_return(&wcid->non_aql_packets);
if (pending < 0)
atomic_cmpxchg(&wcid->non_aql_packets, pending, 0);
+
+ sta = wcid_to_sta(wcid);
+ if (!sta || pending != MT_MAX_NON_AQL_PKT - 1)
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(sta->txq); i++) {
+ if (!sta->txq[i])
+ continue;
+
+ ieee80211_schedule_txq(dev->hw, sta->txq[i]);
+ }
}
void __mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid_idx, struct sk_buff *skb,
@@ -254,6 +269,21 @@ void __mt76_tx_complete_skb(struct mt76_dev *dev, u16 wcid_idx, struct sk_buff *
wcid = __mt76_wcid_ptr(dev, wcid_idx);
mt76_tx_check_non_aql(dev, wcid, skb);
+ if (wcid && (dev->drv->drv_flags & MT_DRV_HW_PS_BUFFERING) &&
+ test_bit(MT_WCID_FLAG_PS, &wcid->flags)) {
+ struct ieee80211_sta *sta = wcid_to_sta(wcid);
+
+ if (sta) {
+ struct ieee80211_hw *hw = mt76_phy_hw(dev, wcid->phy_idx);
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(sta->txq); i++)
+ if (sta->txq[i])
+ ieee80211_schedule_txq(hw, sta->txq[i]);
+ mt76_worker_schedule(&dev->tx_worker);
+ }
+ }
+
#ifdef CONFIG_NL80211_TESTMODE
if (mt76_is_testmode_skb(dev, skb, &hw)) {
struct mt76_phy *phy = hw->priv;
@@ -316,7 +346,8 @@ __mt76_tx_queue_skb(struct mt76_phy *phy, int qid, struct sk_buff *skb,
if (!non_aql)
return idx;
- pending = atomic_inc_return(&wcid->non_aql_packets);
+ /* the hardware can report the completion on a different link */
+ pending = atomic_inc_return(&mt76_wcid_primary(wcid)->non_aql_packets);
if (stop && pending >= MT_MAX_NON_AQL_PKT)
*stop = true;
@@ -463,10 +494,15 @@ mt76_txq_send_burst(struct mt76_phy *phy, struct mt76_queue *q,
bool stop = false;
int idx;
- if (test_bit(MT_WCID_FLAG_PS, &wcid->flags))
- return 0;
+ if (test_bit(MT_WCID_FLAG_PS, &wcid->flags)) {
+ if (!(dev->drv->drv_flags & MT_DRV_HW_PS_BUFFERING))
+ return 0;
+ if (ieee80211_txq_aql_pending(phy->hw, txq))
+ return 0;
+ }
- if (atomic_read(&wcid->non_aql_packets) >= MT_MAX_NON_AQL_PKT)
+ if (atomic_read(&mt76_wcid_primary(wcid)->non_aql_packets) >=
+ MT_MAX_NON_AQL_PKT)
return 0;
skb = mt76_txq_dequeue(phy, mtxq);
@@ -484,6 +520,9 @@ mt76_txq_send_burst(struct mt76_phy *phy, struct mt76_queue *q,
if (idx < 0)
return idx;
+ if (test_bit(MT_WCID_FLAG_PS, &wcid->flags))
+ goto out;
+
do {
if (test_bit(MT76_RESET, &phy->state) || phy->offchannel)
break;
@@ -509,6 +548,7 @@ mt76_txq_send_burst(struct mt76_phy *phy, struct mt76_queue *q,
n_frames++;
} while (1);
+out:
spin_lock(&q->lock);
dev->queue_ops->kick(dev, q);
spin_unlock(&q->lock);
@@ -535,7 +575,7 @@ mt76_txq_schedule_list(struct mt76_phy *phy, enum mt76_txq_id qid)
mtxq = (struct mt76_txq *)txq->drv_priv;
wcid = __mt76_wcid_ptr(dev, mtxq->wcid);
- if (!wcid || test_bit(MT_WCID_FLAG_PS, &wcid->flags))
+ if (!wcid)
continue;
phy = mt76_dev_phy(dev, wcid->phy_idx);
@@ -543,6 +583,25 @@ mt76_txq_schedule_list(struct mt76_phy *phy, enum mt76_txq_id qid)
continue;
q = phy->q_tx[qid];
+
+ if (test_bit(MT_WCID_FLAG_PS, &wcid->flags)) {
+ if (!(dev->drv->drv_flags & MT_DRV_HW_PS_BUFFERING))
+ continue;
+
+ if (!mt76_txq_stopped(q))
+ n_frames = mt76_txq_send_burst(phy, q, mtxq, wcid);
+
+ ieee80211_return_txq(phy->hw, txq, false);
+
+ if (unlikely(n_frames < 0))
+ return n_frames;
+ ret += n_frames;
+ continue;
+ }
+
+ if (atomic_read(&mt76_wcid_primary(wcid)->non_aql_packets) >=
+ MT_MAX_NON_AQL_PKT)
+ continue;
if (dev->queue_ops->tx_cleanup &&
q->queued + 2 * MT_TXQ_FREE_THR >= q->ndesc) {
dev->queue_ops->tx_cleanup(dev, q, false);
@@ -612,8 +671,9 @@ mt76_txq_schedule_pending_wcid(struct mt76_phy *phy, struct mt76_wcid *wcid,
if ((dev->drv->drv_flags & MT_DRV_HW_MGMT_TXQ) &&
!(info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP) &&
- !ieee80211_is_data(hdr->frame_control) &&
- (!ieee80211_is_bufferable_mmpdu(skb) ||
+ !ieee80211_is_data_present(hdr->frame_control) &&
+ (!(wcid->tx_info & MT_WCID_TX_INFO_SET) ||
+ !ieee80211_is_bufferable_mmpdu(skb) ||
ieee80211_is_deauth(hdr->frame_control) ||
ieee80211_is_disassoc(hdr->frame_control) ||
head == &wcid->tx_offchannel))
@@ -749,6 +809,33 @@ void mt76_stop_tx_queues(struct mt76_phy *phy, struct ieee80211_sta *sta,
}
EXPORT_SYMBOL_GPL(mt76_stop_tx_queues);
+void mt76_sta_ps_transition(struct mt76_dev *dev, struct mt76_wcid *wcid,
+ bool ps)
+{
+ struct ieee80211_sta *sta;
+ struct ieee80211_hw *hw;
+ int i;
+
+ if (ps) {
+ set_bit(MT_WCID_FLAG_PS, &wcid->flags);
+ mt76_worker_schedule(&dev->tx_worker);
+ return;
+ }
+
+ clear_bit(MT_WCID_FLAG_PS, &wcid->flags);
+
+ sta = wcid_to_sta(wcid);
+ if (!sta)
+ return;
+
+ hw = mt76_phy_hw(dev, wcid->phy_idx);
+ for (i = 0; i < ARRAY_SIZE(sta->txq); i++)
+ if (sta->txq[i])
+ ieee80211_schedule_txq(hw, sta->txq[i]);
+ mt76_worker_schedule(&dev->tx_worker);
+}
+EXPORT_SYMBOL_GPL(mt76_sta_ps_transition);
+
void mt76_wake_tx_queue(struct ieee80211_hw *hw, struct ieee80211_txq *txq)
{
struct mt76_phy *phy = hw->priv;
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index ce3b2d98f6b4..5be8d977d075 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -1346,8 +1346,10 @@ static int xennet_poll(struct napi_struct *napi, int budget)
skb->data_len = rx->status;
skb->len += rx->status;
- if (unlikely(xennet_fill_frags(queue, skb, &tmpq)))
+ if (unlikely(xennet_fill_frags(queue, skb, &tmpq))) {
+ __skb_queue_head(&tmpq, skb);
goto err;
+ }
if (rx->flags & XEN_NETRXF_csum_blank)
skb->ip_summed = CHECKSUM_PARTIAL;
diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index aea63fab2ea3..4cba659a657d 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -212,6 +212,7 @@ struct ntb_transport_mw {
void *alloc_addr;
void *virt_addr;
dma_addr_t dma_addr;
+ dma_addr_t original_dma_addr;
};
struct ntb_transport_client_dev {
@@ -797,14 +798,14 @@ static void ntb_transport_msi_desc_changed(void *data)
static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw)
{
struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
- struct pci_dev *pdev = nt->ndev->pdev;
+ struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
if (!mw->virt_addr)
return;
ntb_mw_clear_trans(nt->ndev, PIDX, num_mw);
- dma_free_coherent(&pdev->dev, mw->alloc_size,
- mw->alloc_addr, mw->dma_addr);
+ dma_free_coherent(dma_dev, mw->alloc_size,
+ mw->alloc_addr, mw->original_dma_addr);
mw->xlat_size = 0;
mw->buff_size = 0;
mw->alloc_size = 0;
@@ -840,6 +841,7 @@ static int ntb_alloc_mw_buffer(struct ntb_transport_mw *mw,
return -ENOMEM;
}
virt_addr = alloc_addr;
+ mw->original_dma_addr = dma_addr;
/*
* we must ensure that the memory address allocated is BAR size
@@ -873,7 +875,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
resource_size_t size)
{
struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
- struct pci_dev *pdev = nt->ndev->pdev;
+ struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
size_t xlat_size, buff_size;
resource_size_t xlat_align;
resource_size_t xlat_align_size;
@@ -902,12 +904,12 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
mw->buff_size = buff_size;
mw->alloc_size = buff_size;
- rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+ rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
if (rc) {
mw->alloc_size *= 2;
- rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+ rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
if (rc) {
- dev_err(&pdev->dev,
+ dev_err(dma_dev,
"Unable to alloc aligned MW buff\n");
mw->xlat_size = 0;
mw->buff_size = 0;
@@ -920,7 +922,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
rc = ntb_mw_set_trans(nt->ndev, PIDX, num_mw, mw->dma_addr,
mw->xlat_size);
if (rc) {
- dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw);
+ dev_err(dma_dev, "Unable to set mw%d translation", num_mw);
ntb_free_mw(nt, num_mw);
return -EIO;
}
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index f424cfba833e..4efb55f1e383 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -795,7 +795,8 @@ static bool nvme_pci_prp_save_mapping(struct request *req,
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
- if (dma_use_iova(&iod->dma_state) || !dma_need_unmap(dma_dev))
+ if (dma_use_iova(&iod->dma_state) || !dma_need_unmap(dma_dev) ||
+ (req->cmd_flags & REQ_P2PDMA))
return true;
if (!iod->nr_dma_vecs) {
@@ -827,6 +828,17 @@ static bool nvme_pci_prp_iter_next(struct request *req, struct device *dma_dev,
return nvme_pci_prp_save_mapping(req, dma_dev, iter);
}
+static void nvme_unmap_iter(struct request *req, struct blk_dma_iter *iter,
+ struct dma_iova_state *state)
+{
+ struct nvme_queue *nvmeq = req->mq_hctx->driver_data;
+ struct device *dev = nvmeq->dev->dev;
+
+ if (!blk_dma_unmap(req, dev, state, iter->len,
+ iter->p2pdma.map == PCI_P2PDMA_MAP_BUS_ADDR))
+ dma_unmap_page(dev, iter->addr, iter->len, rq_dma_dir(req));
+}
+
static blk_status_t nvme_pci_setup_data_prp(struct request *req,
struct blk_dma_iter *iter)
{
@@ -837,8 +849,10 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req,
unsigned int prp_len, i;
__le64 *prp_list;
- if (!nvme_pci_prp_save_mapping(req, nvmeq->dev->dev, iter))
+ if (!nvme_pci_prp_save_mapping(req, nvmeq->dev->dev, iter)) {
+ nvme_unmap_iter(req, iter, &iod->dma_state);
return iter->status;
+ }
/*
* PRP1 always points to the start of the DMA transfers.
@@ -943,6 +957,7 @@ static blk_status_t nvme_pci_setup_data_prp(struct request *req,
dev_err_once(nvmeq->dev->dev,
"Incorrectly formed request for payload:%d nents:%d\n",
blk_rq_payload_bytes(req), blk_rq_nr_phys_segments(req));
+ nvme_unmap_data(req);
return BLK_STS_IOERR;
}
@@ -986,8 +1001,11 @@ static blk_status_t nvme_pci_setup_data_sgl(struct request *req,
sg_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,
&sgl_dma);
- if (!sg_list)
+ if (!sg_list) {
+ nvme_unmap_iter(req, iter, &iod->dma_state);
return BLK_STS_RESOURCE;
+ }
+
iod->descriptors[iod->nr_descriptors++] = sg_list;
do {
@@ -1119,8 +1137,10 @@ static blk_status_t nvme_pci_setup_meta_iter(struct request *req)
sg_list = dma_pool_alloc(nvmeq->descriptor_pools.small, GFP_ATOMIC,
&sgl_dma);
- if (!sg_list)
+ if (!sg_list) {
+ nvme_unmap_iter(req, &iter, &iod->meta_dma_state);
return BLK_STS_RESOURCE;
+ }
iod->meta_descriptor = sg_list;
iod->meta_dma = sgl_dma;
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 86e064d058d3..beef2b576102 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1895,7 +1895,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
sk_set_memalloc(queue->sock->sk);
if (nctrl->opts->mask & NVMF_OPT_HOST_TRADDR) {
- ret = kernel_bind(queue->sock, (struct sockaddr *)&ctrl->src_addr,
+ ret = kernel_bind(queue->sock, (struct sockaddr_unsized *)&ctrl->src_addr,
sizeof(ctrl->src_addr));
if (ret) {
dev_err(nctrl->device,
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index fe7b6f8639f0..fa9920787a8f 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -309,8 +309,10 @@ static void nvmet_execute_get_log_page_rmi(struct nvmet_req *req)
}
log = kzalloc(sizeof(*log), GFP_KERNEL);
- if (!log)
+ if (!log) {
+ status = NVME_SC_INTERNAL;
goto out;
+ }
log->endgid = req->cmd->get_log_page.lsi;
disk = req->ns->bdev->bd_disk;
@@ -1338,7 +1340,7 @@ static u16 nvmet_set_feat_arbitration(struct nvmet_req *req)
void nvmet_execute_set_features(struct nvmet_req *req)
{
- struct nvmet_subsys *subsys = nvmet_req_subsys(req);
+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
u16 status = 0;
@@ -1360,7 +1362,7 @@ void nvmet_execute_set_features(struct nvmet_req *req)
break;
}
nvmet_set_result(req,
- (subsys->max_qid - 1) | ((subsys->max_qid - 1) << 16));
+ (ctrl->max_qid - 1) | ((ctrl->max_qid - 1) << 16));
break;
case NVME_FEAT_IRQ_COALESCE:
status = nvmet_set_feat_irq_coalesce(req);
@@ -1497,7 +1499,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req)
void nvmet_execute_get_features(struct nvmet_req *req)
{
- struct nvmet_subsys *subsys = nvmet_req_subsys(req);
+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u16 status = 0;
@@ -1537,7 +1539,7 @@ void nvmet_execute_get_features(struct nvmet_req *req)
break;
case NVME_FEAT_NUM_QUEUES:
nvmet_set_result(req,
- (subsys->max_qid-1) | ((subsys->max_qid-1) << 16));
+ (ctrl->max_qid-1) | ((ctrl->max_qid-1) << 16));
break;
case NVME_FEAT_KATO:
nvmet_get_feat_kato(req);
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 3310d0b44ccf..9b64c313c703 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -867,7 +867,7 @@ u16 nvmet_check_cqid(struct nvmet_ctrl *ctrl, u16 cqid, bool create)
if (!ctrl->cqs)
return NVME_SC_INTERNAL | NVME_STATUS_DNR;
- if (cqid > ctrl->subsys->max_qid)
+ if (cqid > ctrl->max_qid)
return NVME_SC_QID_INVALID | NVME_STATUS_DNR;
if ((create && ctrl->cqs[cqid]) || (!create && !ctrl->cqs[cqid]))
@@ -915,7 +915,7 @@ u16 nvmet_check_sqid(struct nvmet_ctrl *ctrl, u16 sqid,
if (!ctrl->sqs)
return NVME_SC_INTERNAL | NVME_STATUS_DNR;
- if (sqid > ctrl->subsys->max_qid)
+ if (sqid > ctrl->max_qid)
return NVME_SC_QID_INVALID | NVME_STATUS_DNR;
if ((create && ctrl->sqs[sqid]) ||
@@ -1301,7 +1301,7 @@ static int nvmet_req_alloc_p2pmem_sgls(struct pci_dev *p2p_dev,
return 0;
out_free_sg:
- pci_p2pmem_free_sgl(req->p2p_dev, req->sg);
+ pci_p2pmem_free_sgl(p2p_dev, req->sg);
out_err:
return -ENOMEM;
}
@@ -1647,13 +1647,31 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
if (!ctrl->changed_ns_list)
goto out_free_ctrl;
- ctrl->sqs = kcalloc(subsys->max_qid + 1,
+ /*
+ * Discovery controllers may use some arbitrary high value
+ * in order to cleanup stale discovery sessions
+ */
+ if (nvmet_is_disc_subsys(ctrl->subsys) && !kato)
+ kato = NVMET_DISC_KATO_MS;
+
+ /* keep-alive timeout in seconds */
+ ctrl->kato = DIV_ROUND_UP(kato, 1000);
+
+ ctrl->err_counter = 0;
+ spin_lock_init(&ctrl->error_lock);
+
+ down_read(&nvmet_config_sem);
+ mutex_lock(&subsys->lock);
+
+ ctrl->max_qid = subsys->max_qid;
+
+ ctrl->sqs = kcalloc(ctrl->max_qid + 1,
sizeof(struct nvmet_sq *),
GFP_KERNEL);
if (!ctrl->sqs)
goto out_free_changed_ns_list;
- ctrl->cqs = kcalloc(subsys->max_qid + 1, sizeof(struct nvmet_cq *),
+ ctrl->cqs = kcalloc(ctrl->max_qid + 1, sizeof(struct nvmet_cq *),
GFP_KERNEL);
if (!ctrl->cqs)
goto out_free_sqs;
@@ -1667,22 +1685,6 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
}
ctrl->cntlid = ret;
- /*
- * Discovery controllers may use some arbitrary high value
- * in order to cleanup stale discovery sessions
- */
- if (nvmet_is_disc_subsys(ctrl->subsys) && !kato)
- kato = NVMET_DISC_KATO_MS;
-
- /* keep-alive timeout in seconds */
- ctrl->kato = DIV_ROUND_UP(kato, 1000);
-
- ctrl->err_counter = 0;
- spin_lock_init(&ctrl->error_lock);
-
- nvmet_start_keep_alive_timer(ctrl);
-
- mutex_lock(&subsys->lock);
ret = nvmet_ctrl_init_pr(ctrl);
if (ret)
goto init_pr_fail;
@@ -1690,6 +1692,9 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
nvmet_setup_p2p_ns_map(ctrl, args->p2p_client);
nvmet_debugfs_ctrl_setup(ctrl);
mutex_unlock(&subsys->lock);
+ up_read(&nvmet_config_sem);
+
+ nvmet_start_keep_alive_timer(ctrl);
dhchap_status = nvmet_setup_auth(ctrl, args->sq, false);
if (dhchap_status) {
@@ -1716,14 +1721,14 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
return ctrl;
init_pr_fail:
- mutex_unlock(&subsys->lock);
- nvmet_stop_keep_alive_timer(ctrl);
ida_free(&cntlid_ida, ctrl->cntlid);
out_free_cqs:
kfree(ctrl->cqs);
out_free_sqs:
kfree(ctrl->sqs);
out_free_changed_ns_list:
+ mutex_unlock(&subsys->lock);
+ up_read(&nvmet_config_sem);
kfree(ctrl->changed_ns_list);
out_free_ctrl:
kfree(ctrl);
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index 7b8d8b397802..bc837fbfada1 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -370,7 +370,7 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
goto out;
}
- if (unlikely(qid > ctrl->subsys->max_qid)) {
+ if (unlikely(qid > ctrl->max_qid)) {
pr_warn("invalid queue id (%d)\n", qid);
status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
req->cqe->result.u32 = IPO_IATTR_CONNECT_SQE(qid);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 3e8a43df57a6..87e2fdbd1865 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -264,6 +264,7 @@ struct nvmet_ctrl {
uuid_t hostid;
u16 cntlid;
+ u16 max_qid;
u32 kato;
struct nvmet_port *port;
@@ -750,6 +751,11 @@ static inline struct nvmet_subsys *nvmet_req_subsys(struct nvmet_req *req)
return req->sq->ctrl->subsys;
}
+static inline struct nvmet_ctrl *nvmet_req_ctrl(struct nvmet_req *req)
+{
+ return req->sq->ctrl;
+}
+
static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys)
{
return subsys->type != NVME_NQN_NVME;
diff --git a/drivers/nvme/target/pci-epf.c b/drivers/nvme/target/pci-epf.c
index 1afda3686311..e3815be98ee6 100644
--- a/drivers/nvme/target/pci-epf.c
+++ b/drivers/nvme/target/pci-epf.c
@@ -2085,7 +2085,7 @@ static int nvmet_pci_epf_create_ctrl(struct nvmet_pci_epf *nvme_epf,
}
/* Allocate our queues, up to the maximum number. */
- ctrl->nr_queues = min(ctrl->tctrl->subsys->max_qid + 1, max_nr_queues);
+ ctrl->nr_queues = min(ctrl->tctrl->max_qid + 1, max_nr_queues);
ret = nvmet_pci_epf_alloc_queues(ctrl);
if (ret)
goto out_put_ctrl;
diff --git a/drivers/nvme/target/pr.c b/drivers/nvme/target/pr.c
index 783e6a5ec3e0..a3cad41323aa 100644
--- a/drivers/nvme/target/pr.c
+++ b/drivers/nvme/target/pr.c
@@ -51,6 +51,8 @@ u16 nvmet_set_feat_resv_notif_mask(struct nvmet_req *req, u32 mask)
if (nsid != U32_MAX) {
status = nvmet_req_find_ns(req);
+ if (status == (NVME_SC_INVALID_NS | NVME_STATUS_DNR))
+ return NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
if (status)
return status;
if (!req->ns->pr.enable)
@@ -75,6 +77,8 @@ u16 nvmet_get_feat_resv_notif_mask(struct nvmet_req *req)
u16 status;
status = nvmet_req_find_ns(req);
+ if (status == (NVME_SC_INVALID_NS | NVME_STATUS_DNR))
+ return NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
if (status)
return status;
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index b9a904f18489..b4d212bbbe12 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -2150,7 +2150,7 @@ static int nvmet_tcp_add_port(struct nvmet_port *nport)
if (so_priority > 0)
sock_set_priority(port->sock->sk, so_priority);
- ret = kernel_bind(port->sock, (struct sockaddr *)&port->addr,
+ ret = kernel_bind(port->sock, (struct sockaddr_unsized *)&port->addr,
sizeof(port->addr));
if (ret) {
pr_err("failed to bind port socket %d\n", ret);
diff --git a/drivers/pci/endpoint/functions/pci-epf-mhi.c b/drivers/pci/endpoint/functions/pci-epf-mhi.c
index 2f077d0b7957..191f4e6c30f9 100644
--- a/drivers/pci/endpoint/functions/pci-epf-mhi.c
+++ b/drivers/pci/endpoint/functions/pci-epf-mhi.c
@@ -690,7 +690,7 @@ static int pci_epf_mhi_dma_init(struct pci_epf_mhi *epf_mhi)
goto err_release_tx;
}
- epf_mhi->dma_wq = alloc_workqueue("pci_epf_mhi_dma_wq", 0, 0);
+ epf_mhi->dma_wq = alloc_workqueue("pci_epf_mhi_dma_wq", WQ_PERCPU, 0);
if (!epf_mhi->dma_wq) {
ret = -ENOMEM;
goto err_release_rx;
diff --git a/drivers/pci/endpoint/functions/pci-epf-ntb.c b/drivers/pci/endpoint/functions/pci-epf-ntb.c
index d12cf226f328..5314aca2188a 100644
--- a/drivers/pci/endpoint/functions/pci-epf-ntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-ntb.c
@@ -2073,8 +2073,13 @@ static int __init epf_ntb_init(void)
{
int ret;
- kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM |
- WQ_HIGHPRI, 0);
+ kpcintb_workqueue = alloc_workqueue("kpcintb",
+ WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_PERCPU, 0);
+ if (!kpcintb_workqueue) {
+ pr_err("Failed to allocate kpcintb workqueue\n");
+ return -ENOMEM;
+ }
+
ret = pci_epf_register_driver(&epf_ntb_driver);
if (ret) {
destroy_workqueue(kpcintb_workqueue);
diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
index 9bc6d283eae2..97e003782216 100644
--- a/drivers/pci/endpoint/functions/pci-epf-test.c
+++ b/drivers/pci/endpoint/functions/pci-epf-test.c
@@ -1191,7 +1191,7 @@ static int __init pci_epf_test_init(void)
int ret;
kpcitest_workqueue = alloc_workqueue("kpcitest",
- WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
+ WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_PERCPU, 0);
if (!kpcitest_workqueue) {
pr_err("Failed to allocate the kpcitest work queue\n");
return -ENOMEM;
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index d3d0780fb481..68110855ea1c 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -1513,8 +1513,13 @@ static int __init epf_ntb_init(void)
{
int ret;
- kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM |
- WQ_HIGHPRI, 0);
+ kpcintb_workqueue = alloc_workqueue("kpcintb",
+ WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_PERCPU, 0);
+ if (!kpcintb_workqueue) {
+ pr_err("Failed to allocate kpcintb workqueue\n");
+ return -ENOMEM;
+ }
+
ret = pci_epf_register_driver(&epf_ntb_driver);
if (ret) {
destroy_workqueue(kpcintb_workqueue);
diff --git a/drivers/pci/hotplug/pnv_php.c b/drivers/pci/hotplug/pnv_php.c
index e9c8f7fb2213..44fca9586453 100644
--- a/drivers/pci/hotplug/pnv_php.c
+++ b/drivers/pci/hotplug/pnv_php.c
@@ -802,7 +802,7 @@ static struct pnv_php_slot *pnv_php_alloc_slot(struct device_node *dn)
}
/* Allocate workqueue for this slot's interrupt handling */
- php_slot->wq = alloc_workqueue("pciehp-%s", 0, 0, php_slot->name);
+ php_slot->wq = alloc_workqueue("pciehp-%s", WQ_PERCPU, 0, php_slot->name);
if (!php_slot->wq) {
SLOT_WARN(php_slot, "Cannot alloc workqueue\n");
kfree(php_slot->name);
diff --git a/drivers/pci/hotplug/shpchp_core.c b/drivers/pci/hotplug/shpchp_core.c
index 0c341453afc6..56308515ecba 100644
--- a/drivers/pci/hotplug/shpchp_core.c
+++ b/drivers/pci/hotplug/shpchp_core.c
@@ -80,7 +80,8 @@ static int init_slots(struct controller *ctrl)
slot->device = ctrl->slot_device_offset + i;
slot->number = ctrl->first_slot + (ctrl->slot_num_inc * i);
- slot->wq = alloc_workqueue("shpchp-%d", 0, 0, slot->number);
+ slot->wq = alloc_workqueue("shpchp-%d", WQ_PERCPU, 0,
+ slot->number);
if (!slot->wq) {
retval = -ENOMEM;
goto error_slot;
diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
index 1219f39b1c69..a2121abb1abf 100644
--- a/drivers/pci/pci-driver.c
+++ b/drivers/pci/pci-driver.c
@@ -29,6 +29,47 @@ struct pci_dynid {
struct pci_device_id id;
};
+/**
+ * do_pci_add_dynid - Add a new PCI device ID to this driver and re-probe
+ * @drv: target PCI driver
+ * @id: ID to be added
+ * @check_dup: whether to check if matching ID is already present
+ *
+ * Add a new dynamic PCI device ID to this driver and causes the driver to
+ * probe for all devices again. @drv must have been registered prior to calling
+ * this function.
+ *
+ * Context: Does GFP_KERNEL allocation.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+static int do_pci_add_dynid(struct pci_driver *drv,
+ const struct pci_device_id *id,
+ bool check_dup)
+{
+ struct pci_dynid *dynid, *existing_dynid;
+
+ dynid = kzalloc(sizeof(*dynid), GFP_KERNEL);
+ if (!dynid)
+ return -ENOMEM;
+
+ dynid->id = *id;
+
+ scoped_guard(spinlock, &drv->dynids.lock) {
+ if (check_dup) {
+ list_for_each_entry(existing_dynid, &drv->dynids.list, node) {
+ if (pci_match_one_id(&existing_dynid->id, id)) {
+ kfree(dynid);
+ return -EEXIST;
+ }
+ }
+ }
+ list_add_tail(&dynid->node, &drv->dynids.list);
+ }
+
+ return driver_attach(&drv->driver);
+}
+
/**
* pci_add_dynid - add a new PCI device ID to this driver and re-probe devices
* @drv: target pci driver
@@ -56,25 +97,17 @@ int pci_add_dynid(struct pci_driver *drv,
unsigned int class, unsigned int class_mask,
unsigned long driver_data)
{
- struct pci_dynid *dynid;
-
- dynid = kzalloc(sizeof(*dynid), GFP_KERNEL);
- if (!dynid)
- return -ENOMEM;
-
- dynid->id.vendor = vendor;
- dynid->id.device = device;
- dynid->id.subvendor = subvendor;
- dynid->id.subdevice = subdevice;
- dynid->id.class = class;
- dynid->id.class_mask = class_mask;
- dynid->id.driver_data = driver_data;
-
- spin_lock(&drv->dynids.lock);
- list_add_tail(&dynid->node, &drv->dynids.list);
- spin_unlock(&drv->dynids.lock);
-
- return driver_attach(&drv->driver);
+ struct pci_device_id id = {
+ .vendor = vendor,
+ .device = device,
+ .subvendor = subvendor,
+ .subdevice = subdevice,
+ .class = class,
+ .class_mask = class_mask,
+ .driver_data = driver_data,
+ };
+
+ return do_pci_add_dynid(drv, &id, false);
}
EXPORT_SYMBOL_GPL(pci_add_dynid);
@@ -90,6 +123,31 @@ static void pci_free_dynids(struct pci_driver *drv)
spin_unlock(&drv->dynids.lock);
}
+/**
+ * do_pci_match_id - See if a PCI ID matches a given pci_id table
+ * @ids: array of PCI device ID structures to search in
+ * @dev_id: the actual PCI device ID structure to match against.
+ * @include_override_only: also match against device ID entries marked as
+ * override only.
+ *
+ * Return: the matching pci_device_id structure or %NULL if there is no match.
+ */
+static const struct pci_device_id *
+do_pci_match_id(const struct pci_device_id *ids,
+ const struct pci_device_id *dev_id,
+ bool include_override_only)
+{
+ if (ids) {
+ while (ids->vendor || ids->subvendor || ids->class_mask) {
+ if ((!ids->override_only || include_override_only) &&
+ pci_match_one_id(ids, dev_id))
+ return ids;
+ ids++;
+ }
+ }
+ return NULL;
+}
+
/**
* pci_match_id - See if a PCI device matches a given pci_id table
* @ids: array of PCI device ID structures to search in
@@ -105,14 +163,9 @@ static void pci_free_dynids(struct pci_driver *drv)
const struct pci_device_id *pci_match_id(const struct pci_device_id *ids,
struct pci_dev *dev)
{
- if (ids) {
- while (ids->vendor || ids->subvendor || ids->class_mask) {
- if (pci_match_one_device(ids, dev))
- return ids;
- ids++;
- }
- }
- return NULL;
+ struct pci_device_id dev_id = pci_id_from_device(dev);
+
+ return do_pci_match_id(ids, &dev_id, true);
}
EXPORT_SYMBOL(pci_match_id);
@@ -127,6 +180,7 @@ static const struct pci_device_id pci_device_id_any = {
* pci_match_device - See if a device matches a driver's list of IDs
* @drv: the PCI driver to match against
* @dev: the PCI device structure to match against
+ * @id_copy: place to store copy of pci_device_id for dynamic ID
*
* Used by a driver to check whether a PCI device is in its list of
* supported devices or in the dynids list, which may have been augmented
@@ -134,10 +188,11 @@ static const struct pci_device_id pci_device_id_any = {
* structure or %NULL if there is no match.
*/
static const struct pci_device_id *pci_match_device(struct pci_driver *drv,
- struct pci_dev *dev)
+ struct pci_dev *dev,
+ struct pci_device_id *id_copy)
{
- struct pci_dynid *dynid;
- const struct pci_device_id *found_id = NULL, *ids;
+ const struct pci_device_id *found_id = NULL;
+ struct pci_device_id dev_id;
int ret;
/* When driver_override is set, only bind to the matching driver */
@@ -145,45 +200,29 @@ static const struct pci_device_id *pci_match_device(struct pci_driver *drv,
if (ret == 0)
return NULL;
+ dev_id = pci_id_from_device(dev);
/* Look at the dynamic ids first, before the static ones */
- spin_lock(&drv->dynids.lock);
- list_for_each_entry(dynid, &drv->dynids.list, node) {
- if (pci_match_one_device(&dynid->id, dev)) {
- found_id = &dynid->id;
- break;
+ scoped_guard(spinlock, &drv->dynids.lock) {
+ struct pci_dynid *dynid;
+
+ list_for_each_entry(dynid, &drv->dynids.list, node) {
+ if (pci_match_one_id(&dynid->id, &dev_id)) {
+ *id_copy = dynid->id;
+ return id_copy;
+ }
}
}
- spin_unlock(&drv->dynids.lock);
+ found_id = do_pci_match_id(drv->id_table, &dev_id, ret > 0);
if (found_id)
return found_id;
- for (ids = drv->id_table; (found_id = pci_match_id(ids, dev));
- ids = found_id + 1) {
- /*
- * The match table is split based on driver_override.
- * In case override_only was set, enforce driver_override
- * matching.
- */
- if (found_id->override_only) {
- if (ret > 0)
- return found_id;
- } else {
- return found_id;
- }
- }
-
/* driver_override will always match, send a dummy id */
if (ret > 0)
return &pci_device_id_any;
return NULL;
}
-static void _pci_free_device(struct device *dev)
-{
- kfree(to_pci_dev(dev));
-}
-
/**
* new_id_store - sysfs frontend to pci_add_dynid()
* @driver: target device driver
@@ -197,38 +236,22 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
{
struct pci_driver *pdrv = to_pci_driver(driver);
const struct pci_device_id *ids = pdrv->id_table;
- u32 vendor, device, subvendor = PCI_ANY_ID,
- subdevice = PCI_ANY_ID, class = 0, class_mask = 0;
- unsigned long driver_data = 0;
+ struct pci_device_id id = {
+ .subvendor = PCI_ANY_ID,
+ .subdevice = PCI_ANY_ID
+ };
int fields;
int retval = 0;
fields = sscanf(buf, "%x %x %x %x %x %x %lx",
- &vendor, &device, &subvendor, &subdevice,
- &class, &class_mask, &driver_data);
+ &id.vendor, &id.device, &id.subvendor, &id.subdevice,
+ &id.class, &id.class_mask, &id.driver_data);
if (fields < 2)
return -EINVAL;
if (fields != 7) {
- struct pci_dev *pdev = kzalloc(sizeof(*pdev), GFP_KERNEL);
- if (!pdev)
- return -ENOMEM;
-
- pdev->vendor = vendor;
- pdev->device = device;
- pdev->subsystem_vendor = subvendor;
- pdev->subsystem_device = subdevice;
- pdev->class = class;
- pdev->dev.release = _pci_free_device;
-
- device_initialize(&pdev->dev);
- if (pci_match_device(pdrv, pdev))
- retval = -EEXIST;
-
- put_device(&pdev->dev);
-
- if (retval)
- return retval;
+ if (do_pci_match_id(pdrv->id_table, &id, false))
+ return -EEXIST;
}
/* Only accept driver_data values that match an existing id_table
@@ -236,7 +259,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
if (ids) {
retval = -EINVAL;
while (ids->vendor || ids->subvendor || ids->class_mask) {
- if (driver_data == ids->driver_data) {
+ if (id.driver_data == ids->driver_data) {
retval = 0;
break;
}
@@ -246,8 +269,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
return retval;
}
- retval = pci_add_dynid(pdrv, vendor, device, subvendor, subdevice,
- class, class_mask, driver_data);
+ retval = do_pci_add_dynid(pdrv, &id, fields != 7);
if (retval)
return retval;
return count;
@@ -416,12 +438,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
static int __pci_device_probe(struct pci_driver *drv, struct pci_dev *pci_dev)
{
const struct pci_device_id *id;
+ struct pci_device_id id_copy;
int error = 0;
if (drv->probe) {
error = -ENODEV;
- id = pci_match_device(drv, pci_dev);
+ id = pci_match_device(drv, pci_dev, &id_copy);
if (id)
error = pci_call_probe(drv, pci_dev, id);
}
@@ -1511,12 +1534,13 @@ static int pci_bus_match(struct device *dev, const struct device_driver *drv)
struct pci_dev *pci_dev = to_pci_dev(dev);
struct pci_driver *pci_drv;
const struct pci_device_id *found_id;
+ struct pci_device_id id_copy;
if (pci_dev_binding_disallowed(pci_dev))
return 0;
pci_drv = (struct pci_driver *)to_pci_driver(drv);
- found_id = pci_match_device(pci_drv, pci_dev);
+ found_id = pci_match_device(pci_drv, pci_dev, &id_copy);
if (found_id)
return 1;
diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index 76966343e018..178ee8351f66 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -950,6 +950,11 @@ static ssize_t pci_write_legacy_io(struct file *filp, struct kobject *kobj,
{
struct pci_bus *bus = to_pci_bus(kobj_to_dev(kobj));
u32 val;
+ int ret;
+
+ ret = security_locked_down(LOCKDOWN_PCI_ACCESS);
+ if (ret)
+ return ret;
/* Only support 1, 2 or 4 byte accesses. */
switch (count) {
@@ -985,6 +990,11 @@ static int pci_mmap_legacy_mem(struct file *filp, struct kobject *kobj,
struct vm_area_struct *vma)
{
struct pci_bus *bus = to_pci_bus(kobj_to_dev(kobj));
+ int ret;
+
+ ret = security_locked_down(LOCKDOWN_PCI_ACCESS);
+ if (ret)
+ return ret;
return pci_mmap_legacy_page_range(bus, vma, pci_mmap_mem);
}
@@ -1005,6 +1015,11 @@ static int pci_mmap_legacy_io(struct file *filp, struct kobject *kobj,
struct vm_area_struct *vma)
{
struct pci_bus *bus = to_pci_bus(kobj_to_dev(kobj));
+ int ret;
+
+ ret = security_locked_down(LOCKDOWN_PCI_ACCESS);
+ if (ret)
+ return ret;
return pci_mmap_legacy_page_range(bus, vma, pci_mmap_io);
}
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 8acded592150..fd302523f5a0 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -398,21 +398,40 @@ extern unsigned long pci_cardbus_io_size;
extern unsigned long pci_cardbus_mem_size;
/**
- * pci_match_one_device - Tell if a PCI device structure has a matching
- * PCI device id structure
- * @id: single PCI device id structure to match
- * @dev: the PCI device structure to match against
+ * pci_id_from_device - Obtain a pci_device_id from a PCI device
+ * @dev: the PCI device
*
- * Returns the matching pci_device_id structure or %NULL if there is no match.
+ * Return: a pci_device_id filled.
+ */
+static inline struct pci_device_id pci_id_from_device(const struct pci_dev *dev)
+{
+ return (struct pci_device_id) {
+ .vendor = dev->vendor,
+ .device = dev->device,
+ .subvendor = dev->subsystem_vendor,
+ .subdevice = dev->subsystem_device,
+ .class = dev->class,
+ };
+}
+
+/**
+ * pci_match_one_id - Tell if a PCI device ID matches a needle PCI device ID
+ * @id: single PCI device id structure to match against (needle)
+ * @dev_id: the actual ID from the PCI device
+ *
+ * ID can be retrieved from device using pci_id_from_device().
+ *
+ * Return: the matching pci_device_id structure or %NULL if there is no match.
*/
static inline const struct pci_device_id *
-pci_match_one_device(const struct pci_device_id *id, const struct pci_dev *dev)
+pci_match_one_id(const struct pci_device_id *id,
+ const struct pci_device_id *dev_id)
{
- if ((id->vendor == PCI_ANY_ID || id->vendor == dev->vendor) &&
- (id->device == PCI_ANY_ID || id->device == dev->device) &&
- (id->subvendor == PCI_ANY_ID || id->subvendor == dev->subsystem_vendor) &&
- (id->subdevice == PCI_ANY_ID || id->subdevice == dev->subsystem_device) &&
- !((id->class ^ dev->class) & id->class_mask))
+ if ((id->vendor == PCI_ANY_ID || id->vendor == dev_id->vendor) &&
+ (id->device == PCI_ANY_ID || id->device == dev_id->device) &&
+ (id->subvendor == PCI_ANY_ID || id->subvendor == dev_id->subvendor) &&
+ (id->subdevice == PCI_ANY_ID || id->subdevice == dev_id->subdevice) &&
+ !((id->class ^ dev_id->class) & id->class_mask))
return id;
return NULL;
}
diff --git a/drivers/pci/search.c b/drivers/pci/search.c
index 53840634fbfc..705ab725321a 100644
--- a/drivers/pci/search.c
+++ b/drivers/pci/search.c
@@ -243,8 +243,10 @@ static int match_pci_dev_by_id(struct device *dev, const void *data)
{
struct pci_dev *pdev = to_pci_dev(dev);
const struct pci_device_id *id = data;
+ struct pci_device_id dev_id;
- if (pci_match_one_device(id, pdev))
+ dev_id = pci_id_from_device(pdev);
+ if (pci_match_one_id(id, &dev_id))
return 1;
return 0;
}
@@ -368,9 +370,9 @@ EXPORT_SYMBOL(pci_get_class);
* @class: search for a PCI device with this base class code
* @from: Previous PCI device found in search, or %NULL for new search.
*
- * Iterates through the list of known PCI devices. If a PCI device is found
+ * Iterate through the list of known PCI devices. If a PCI device is found
* with a matching base class code, the reference count to the device is
- * incremented. See pci_match_one_device() to figure out how does this works.
+ * incremented. See pci_match_one_id() to figure out how this works.
* A new search is initiated by passing %NULL as the @from argument.
* Otherwise if @from is not %NULL, searches continue from next device on the
* global list. The reference count for @from is always decremented if it is
diff --git a/drivers/pinctrl/mediatek/pinctrl-moore.c b/drivers/pinctrl/mediatek/pinctrl-moore.c
index 70f608347a5f..2cb3b4e0cca7 100644
--- a/drivers/pinctrl/mediatek/pinctrl-moore.c
+++ b/drivers/pinctrl/mediatek/pinctrl-moore.c
@@ -575,7 +575,7 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
chip->base = -1;
chip->ngpio = hw->soc->npins;
- ret = gpiochip_add_data(chip, hw);
+ ret = devm_gpiochip_add_data(hw->dev, chip, hw);
if (ret < 0)
return ret;
@@ -589,10 +589,8 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
if (!of_property_present(hw->dev->of_node, "gpio-ranges")) {
ret = gpiochip_add_pin_range(chip, dev_name(hw->dev), 0, 0,
chip->ngpio);
- if (ret < 0) {
- gpiochip_remove(chip);
+ if (ret < 0)
return ret;
- }
}
return 0;
diff --git a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
index dd2c8aa03938..791eddd7a2c6 100644
--- a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
+++ b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
@@ -1130,30 +1130,24 @@ int mtk_pctrl_init(struct platform_device *pdev,
pctl->chip->parent = &pdev->dev;
pctl->chip->base = -1;
- ret = gpiochip_add_data(pctl->chip, pctl);
+ ret = devm_gpiochip_add_data(&pdev->dev, pctl->chip, pctl);
if (ret)
return -EINVAL;
/* Register the GPIO to pin mappings. */
ret = gpiochip_add_pin_range(pctl->chip, dev_name(&pdev->dev),
0, 0, pctl->devdata->npins);
- if (ret) {
- ret = -EINVAL;
- goto chip_error;
- }
+ if (ret)
+ return -EINVAL;
/* Only initialize EINT if we have EINT pins */
if (data->eint_hw.ap_num > 0) {
ret = mtk_eint_init(pctl, pdev);
if (ret)
- goto chip_error;
+ return ret;
}
return 0;
-
-chip_error:
- gpiochip_remove(pctl->chip);
- return ret;
}
int mtk_pctrl_common_probe(struct platform_device *pdev)
diff --git a/drivers/pinctrl/mediatek/pinctrl-paris.c b/drivers/pinctrl/mediatek/pinctrl-paris.c
index 23f04b24fd65..09098b68f725 100644
--- a/drivers/pinctrl/mediatek/pinctrl-paris.c
+++ b/drivers/pinctrl/mediatek/pinctrl-paris.c
@@ -957,7 +957,7 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
chip->base = -1;
chip->ngpio = hw->soc->npins;
- ret = gpiochip_add_data(chip, hw);
+ ret = devm_gpiochip_add_data(hw->dev, chip, hw);
if (ret < 0)
return ret;
diff --git a/drivers/pinctrl/pinctrl-zynqmp.c b/drivers/pinctrl/pinctrl-zynqmp.c
index 585fe1661e17..aba129ead04c 100644
--- a/drivers/pinctrl/pinctrl-zynqmp.c
+++ b/drivers/pinctrl/pinctrl-zynqmp.c
@@ -100,7 +100,6 @@ struct zynqmp_pctrl_group {
static struct pinctrl_desc zynqmp_desc;
static u32 family_code;
-static u32 sub_family_code;
static int zynqmp_pctrl_get_groups_count(struct pinctrl_dev *pctldev)
{
@@ -605,7 +604,7 @@ static int zynqmp_pinctrl_prepare_func_groups(struct device *dev, u32 fid,
return -ENOMEM;
for (pin = 0; pin < groups[resp[i]].npins; pin++) {
- if (family_code == ZYNQMP_FAMILY_CODE)
+ if (family_code == PM_ZYNQMP_FAMILY_CODE)
__set_bit(groups[resp[i]].pins[pin], used_pins);
else
__set_bit((u8)groups[resp[i]].pins[pin] - 1, used_pins);
@@ -958,11 +957,11 @@ static int zynqmp_pinctrl_probe(struct platform_device *pdev)
if (!pctrl)
return -ENOMEM;
- ret = zynqmp_pm_get_family_info(&family_code, &sub_family_code);
+ ret = zynqmp_pm_get_family_info(&family_code);
if (ret < 0)
return ret;
- if (family_code == ZYNQMP_FAMILY_CODE) {
+ if (family_code == PM_ZYNQMP_FAMILY_CODE) {
ret = zynqmp_pinctrl_prepare_pin_desc(&pdev->dev, &zynqmp_desc.pins,
&zynqmp_desc.npins);
} else {
diff --git a/drivers/platform/x86/asus-wmi.c b/drivers/platform/x86/asus-wmi.c
index 9026744d7226..9f0f79030b5b 100644
--- a/drivers/platform/x86/asus-wmi.c
+++ b/drivers/platform/x86/asus-wmi.c
@@ -4877,20 +4877,20 @@ static int asus_wmi_add(struct platform_device *pdev)
return 0;
fail_wmi_handler:
+ asus_screenpad_exit(asus);
+fail_screenpad:
asus_wmi_backlight_exit(asus);
fail_backlight:
asus_wmi_rfkill_exit(asus);
-fail_screenpad:
- asus_screenpad_exit(asus);
fail_rfkill:
asus_wmi_led_exit(asus);
fail_leds:
+fail_custom_fan_curve:
fail_hwmon:
asus_wmi_input_exit(asus);
fail_input:
asus_wmi_sysfs_exit(asus->platform_device);
fail_sysfs:
-fail_custom_fan_curve:
fail_platform_profile_setup:
fail_fan_boost_mode:
fail_platform:
diff --git a/drivers/platform/x86/dell/dell-wmi-base.c b/drivers/platform/x86/dell/dell-wmi-base.c
index ba6c88e2a5ec..1f922559e92f 100644
--- a/drivers/platform/x86/dell/dell-wmi-base.c
+++ b/drivers/platform/x86/dell/dell-wmi-base.c
@@ -454,7 +454,7 @@ static int dell_wmi_process_key(struct wmi_device *wdev, int type, int code, u16
key++;
used = 1;
} else if (type == 0x0012 && code == 0x000d && remaining > 0) {
- value = (buffer[2] == 2);
+ value = (buffer[0] == 2);
used = 1;
}
diff --git a/drivers/pmdomain/imx/imx8m-blk-ctrl.c b/drivers/pmdomain/imx/imx8m-blk-ctrl.c
index 99d100e1d923..62c4f829605c 100644
--- a/drivers/pmdomain/imx/imx8m-blk-ctrl.c
+++ b/drivers/pmdomain/imx/imx8m-blk-ctrl.c
@@ -15,6 +15,7 @@
#include <linux/pm_runtime.h>
#include <linux/regmap.h>
#include <linux/clk.h>
+#include <linux/mutex.h>
#include <dt-bindings/power/imx8mm-power.h>
#include <dt-bindings/power/imx8mn-power.h>
@@ -34,6 +35,12 @@ struct imx8m_blk_ctrl {
struct regmap *regmap;
struct imx8m_blk_ctrl_domain *domains;
struct genpd_onecell_data onecell_data;
+ /*
+ * Serializes the blk-ctrl reset/clock sequence across sibling domains;
+ * their transitions interact through the shared VPUMIX bus domain,
+ * VPU_NOC and the not-ack-verified ADB400 handshake (ERR050531).
+ */
+ struct mutex power_lock;
};
struct imx8m_blk_ctrl_domain_data {
@@ -98,6 +105,8 @@ static int imx8m_blk_ctrl_power_on(struct generic_pm_domain *genpd)
struct imx8m_blk_ctrl *bc = domain->bc;
int ret;
+ guard(mutex)(&bc->power_lock);
+
/* make sure bus domain is awake */
ret = pm_runtime_get_sync(bc->bus_power_dev);
if (ret < 0) {
@@ -164,6 +173,8 @@ static int imx8m_blk_ctrl_power_off(struct generic_pm_domain *genpd)
const struct imx8m_blk_ctrl_domain_data *data = domain->data;
struct imx8m_blk_ctrl *bc = domain->bc;
+ guard(mutex)(&bc->power_lock);
+
/* put devices into reset and disable clocks */
if (data->mipi_phy_rst_mask)
regmap_clear_bits(bc->regmap, BLK_MIPI_RESET_DIV, data->mipi_phy_rst_mask);
@@ -202,6 +213,10 @@ static int imx8m_blk_ctrl_probe(struct platform_device *pdev)
bc->dev = dev;
+ ret = devm_mutex_init(dev, &bc->power_lock);
+ if (ret)
+ return ret;
+
bc_data = of_device_get_match_data(dev);
base = devm_platform_ioremap_resource(pdev, 0);
diff --git a/drivers/pmdomain/rockchip/pm-domains.c b/drivers/pmdomain/rockchip/pm-domains.c
index 6e4a29ee439a..79bd0d9040fc 100644
--- a/drivers/pmdomain/rockchip/pm-domains.c
+++ b/drivers/pmdomain/rockchip/pm-domains.c
@@ -728,7 +728,19 @@ static int rockchip_pd_attach_dev(struct generic_pm_domain *genpd,
}
i = 0;
- while ((clk = of_clk_get(dev->of_node, i++)) && !IS_ERR(clk)) {
+ while (1) {
+ clk = of_clk_get(dev->of_node, i++);
+ if (IS_ERR(clk)) {
+ error = PTR_ERR(clk);
+ if (error == -ENOENT)
+ break;
+
+ dev_err(dev, "failed to get clock %d: %d\n", i - 1,
+ error);
+ pm_clk_destroy(dev);
+ return error;
+ }
+
dev_dbg(dev, "adding clock '%pC' to list of PM clocks\n", clk);
error = pm_clk_add_clk(dev, clk);
if (error) {
@@ -811,7 +823,7 @@ static int rockchip_pm_add_one_domain(struct rockchip_pmu *pmu,
dev_err(pmu->dev,
"%pOFn: failed to get clk at index %d: %d\n",
node, i, error);
- return error;
+ goto err_put_clocks;
}
}
@@ -987,7 +999,9 @@ static int rockchip_pm_add_subdomain(struct rockchip_pmu *pmu,
parent_domain->name, child_domain->name);
}
- rockchip_pm_add_subdomain(pmu, np);
+ error = rockchip_pm_add_subdomain(pmu, np);
+ if (error)
+ return error;
}
return 0;
diff --git a/drivers/power/supply/max77705_charger.c b/drivers/power/supply/max77705_charger.c
index 1044bf58aeac..07ad309f5411 100644
--- a/drivers/power/supply/max77705_charger.c
+++ b/drivers/power/supply/max77705_charger.c
@@ -613,42 +613,30 @@ static int max77705_charger_probe(struct i2c_client *i2c)
if (ret)
return dev_err_probe(dev, ret, "failed to add irq chip\n");
- chg->wqueue = create_singlethread_workqueue(dev_name(dev));
+ chg->wqueue = devm_alloc_ordered_workqueue(dev, "%s", 0, dev_name(dev));
if (!chg->wqueue)
return -ENOMEM;
ret = devm_work_autocancel(dev, &chg->chgin_work, max77705_chgin_isr_work);
- if (ret) {
- dev_err_probe(dev, ret, "failed to initialize interrupt work\n");
- goto destroy_wq;
- }
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to initialize interrupt work\n");
ret = max77705_charger_initialize(chg);
- if (ret) {
- dev_err_probe(dev, ret, "failed to initialize charger IC\n");
- goto destroy_wq;
- }
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to initialize charger IC\n");
ret = devm_request_threaded_irq(dev, regmap_irq_get_virq(irq_data, MAX77705_CHGIN_I),
NULL, max77705_chgin_irq,
IRQF_TRIGGER_NONE,
"chgin-irq", chg);
- if (ret) {
- dev_err_probe(dev, ret, "Failed to Request chgin IRQ\n");
- goto destroy_wq;
- }
+ if (ret)
+ return ret;
ret = max77705_charger_enable(chg);
- if (ret) {
- dev_err_probe(dev, ret, "failed to enable charge\n");
- goto destroy_wq;
- }
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to enable charge\n");
return devm_add_action_or_reset(dev, max77705_charger_disable, chg);
-
-destroy_wq:
- destroy_workqueue(chg->wqueue);
- return ret;
}
static const struct of_device_id max77705_charger_of_match[] = {
diff --git a/drivers/ptp/ptp_ocp.c b/drivers/ptp/ptp_ocp.c
index 249bb3877f67..1893ccee570e 100644
--- a/drivers/ptp/ptp_ocp.c
+++ b/drivers/ptp/ptp_ocp.c
@@ -1564,7 +1564,7 @@ ptp_ocp_watchdog(struct timer_list *t)
static void
ptp_ocp_estimate_pci_timing(struct ptp_ocp *bp)
{
- ktime_t start, end, delay = U64_MAX;
+ ktime_t start, end, delay = KTIME_MAX;
u32 ctrl;
int i;
@@ -2217,8 +2217,13 @@ ptp_ocp_ts_enable(void *priv, u32 req, bool enable)
iowrite32(1, ®->intr_mask);
iowrite32(1, ®->intr);
} else {
+ int irq_vec = pci_irq_vector(bp->pdev, ext->irq_vec);
+
iowrite32(0, ®->intr_mask);
iowrite32(0, ®->enable);
+ ioread32(®->intr_mask);
+ if (irq_vec > 0)
+ synchronize_irq(irq_vec);
}
return 0;
@@ -2227,6 +2232,9 @@ ptp_ocp_ts_enable(void *priv, u32 req, bool enable)
static void
ptp_ocp_unregister_ext(struct ptp_ocp_ext_src *ext)
{
+ if (!ext)
+ return;
+
ext->info->enable(ext, ~0, false);
pci_free_irq(ext->bp->pdev, ext->irq_vec, ext);
kfree(ext);
@@ -4558,21 +4566,30 @@ ptp_ocp_detach(struct ptp_ocp *bp)
ptp_ocp_detach_sysfs(bp);
ptp_ocp_attr_group_del(bp);
timer_delete_sync(&bp->watchdog);
+ /* Disable interrupts on all timestampers */
if (bp->ts0)
- ptp_ocp_unregister_ext(bp->ts0);
+ ptp_ocp_ts_enable(bp->ts0, 0, false);
if (bp->ts1)
- ptp_ocp_unregister_ext(bp->ts1);
+ ptp_ocp_ts_enable(bp->ts1, 0, false);
if (bp->ts2)
- ptp_ocp_unregister_ext(bp->ts2);
+ ptp_ocp_ts_enable(bp->ts2, 0, false);
if (bp->ts3)
- ptp_ocp_unregister_ext(bp->ts3);
+ ptp_ocp_ts_enable(bp->ts3, 0, false);
if (bp->ts4)
- ptp_ocp_unregister_ext(bp->ts4);
+ ptp_ocp_ts_enable(bp->ts4, 0, false);
if (bp->pps)
- ptp_ocp_unregister_ext(bp->pps);
+ ptp_ocp_ts_enable(bp->pps, ~0, false);
+ if (bp->ptp)
+ ptp_clock_unregister(bp->ptp);
+ kfree(bp->ptp_info.pin_config);
+ ptp_ocp_unregister_ext(bp->ts0);
+ ptp_ocp_unregister_ext(bp->ts1);
+ ptp_ocp_unregister_ext(bp->ts2);
+ ptp_ocp_unregister_ext(bp->ts3);
+ ptp_ocp_unregister_ext(bp->ts4);
+ ptp_ocp_unregister_ext(bp->pps);
for (i = 0; i < 4; i++)
- if (bp->signal_out[i])
- ptp_ocp_unregister_ext(bp->signal_out[i]);
+ ptp_ocp_unregister_ext(bp->signal_out[i]);
for (i = 0; i < __PORT_COUNT; i++)
if (bp->port[i].line != -1)
serial8250_unregister_port(bp->port[i].line);
@@ -4582,9 +4599,6 @@ ptp_ocp_detach(struct ptp_ocp *bp)
clk_hw_unregister_fixed_rate(bp->i2c_clk);
if (bp->n_irqs)
pci_free_irq_vectors(bp->pdev);
- if (bp->ptp)
- ptp_clock_unregister(bp->ptp);
- kfree(bp->ptp_info.pin_config);
device_unregister(&bp->dev);
}
diff --git a/drivers/pwm/pwm-samsung.c b/drivers/pwm/pwm-samsung.c
index 951b38ff5f8e..14fb460a4565 100644
--- a/drivers/pwm/pwm-samsung.c
+++ b/drivers/pwm/pwm-samsung.c
@@ -102,7 +102,7 @@ struct samsung_pwm_chip {
* IP. Should this change, both drivers will need to be modified to
* properly synchronize accesses to particular instances.
*/
-static DEFINE_SPINLOCK(samsung_pwm_lock);
+static DEFINE_RAW_SPINLOCK(samsung_pwm_lock);
#endif
static inline
@@ -141,14 +141,14 @@ static void pwm_samsung_set_divisor(struct samsung_pwm_chip *our_chip,
bits = (fls(divisor) - 1) - our_chip->variant.div_base;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(our_chip->base + REG_TCFG1);
reg &= ~(TCFG1_MUX_MASK << shift);
reg |= bits << shift;
writel(reg, our_chip->base + REG_TCFG1);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int pwm_samsung_is_tdiv(struct samsung_pwm_chip *our_chip, unsigned int chan)
@@ -249,7 +249,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm)
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
@@ -263,7 +263,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm)
our_chip->disabled_mask &= ~BIT(pwm->hwpwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
return 0;
}
@@ -275,7 +275,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm)
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
tcon &= ~TCON_AUTORELOAD(tcon_chan);
@@ -290,7 +290,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm)
our_chip->disabled_mask |= BIT(pwm->hwpwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void pwm_samsung_manual_update(struct samsung_pwm_chip *our_chip,
@@ -298,11 +298,11 @@ static void pwm_samsung_manual_update(struct samsung_pwm_chip *our_chip,
{
unsigned long flags;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
__pwm_samsung_manual_update(our_chip, pwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int __pwm_samsung_config(struct pwm_chip *chip, struct pwm_device *pwm,
@@ -390,7 +390,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *our_chip,
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
@@ -404,7 +404,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *our_chip,
writel(tcon, our_chip->base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int pwm_samsung_set_polarity(struct pwm_chip *chip,
diff --git a/drivers/regulator/ltc3676.c b/drivers/regulator/ltc3676.c
index 768cdce0d4ec..1ec218951829 100644
--- a/drivers/regulator/ltc3676.c
+++ b/drivers/regulator/ltc3676.c
@@ -232,7 +232,7 @@ static const struct regulator_desc ltc3676_regulators[LTC3676_NUM_REGULATORS] =
LTC3676_FIXED_REG(LDO4, ldo4, LDOB, 2),
};
-static bool ltc3676_readable_writeable_reg(struct device *dev, unsigned int reg)
+static bool ltc3676_writeable_reg(struct device *dev, unsigned int reg)
{
switch (reg) {
case LTC3676_BUCK1 ... LTC3676_IRQSTAT:
@@ -243,6 +243,15 @@ static bool ltc3676_readable_writeable_reg(struct device *dev, unsigned int reg)
return false;
}
+static bool ltc3676_readable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case LTC3676_BUCK1 ... LTC3676_IRQSTAT:
+ return true;
+ }
+ return false;
+}
+
static bool ltc3676_volatile_reg(struct device *dev, unsigned int reg)
{
switch (reg) {
@@ -255,8 +264,8 @@ static bool ltc3676_volatile_reg(struct device *dev, unsigned int reg)
static const struct regmap_config ltc3676_regmap_config = {
.reg_bits = 8,
.val_bits = 8,
- .writeable_reg = ltc3676_readable_writeable_reg,
- .readable_reg = ltc3676_readable_writeable_reg,
+ .writeable_reg = ltc3676_writeable_reg,
+ .readable_reg = ltc3676_readable_reg,
.volatile_reg = ltc3676_volatile_reg,
.max_register = LTC3676_CLIRQ,
.use_single_read = true,
diff --git a/drivers/regulator/of_regulator.c b/drivers/regulator/of_regulator.c
index 32e88cada47a..c6ed75eb1ece 100644
--- a/drivers/regulator/of_regulator.c
+++ b/drivers/regulator/of_regulator.c
@@ -935,15 +935,15 @@ static int is_supply_name(const char *name)
int of_regulator_bulk_get_all(struct device *dev, struct device_node *np,
struct regulator_bulk_data **consumers)
{
- int num_consumers = 0;
+ int num_consumers = 0, names_len = 0;
struct regulator *tmp;
struct regulator_bulk_data *_consumers = NULL;
struct property *prop;
+ char *names;
int i, n = 0, ret;
- char name[64];
/*
- * first pass: get numbers of xxx-supply
+ * first pass: get numbers of xxx-supply and the room their names take
* second pass: fill consumers
*/
restart:
@@ -953,16 +953,19 @@ int of_regulator_bulk_get_all(struct device *dev, struct device_node *np,
continue;
if (!_consumers) {
num_consumers++;
+ names_len += i + 1;
continue;
} else {
- memcpy(name, prop->name, i);
- name[i] = '\0';
- tmp = regulator_get(dev, name);
+ memcpy(names, prop->name, i);
+ names[i] = '\0';
+ tmp = regulator_get(dev, names);
if (IS_ERR(tmp)) {
ret = PTR_ERR(tmp);
goto error;
}
+ _consumers[n].supply = names;
_consumers[n].consumer = tmp;
+ names += i + 1;
n++;
continue;
}
@@ -973,11 +976,16 @@ int of_regulator_bulk_get_all(struct device *dev, struct device_node *np,
}
if (num_consumers == 0)
return 0;
- _consumers = kmalloc_array(num_consumers,
- sizeof(struct regulator_bulk_data),
- GFP_KERNEL);
+ /*
+ * The supply names are kept in the same allocation as the array, so
+ * that they share its lifetime and the caller has nothing extra to
+ * free.
+ */
+ _consumers = kzalloc(size_add(size_mul(num_consumers, sizeof(*_consumers)),
+ names_len), GFP_KERNEL);
if (!_consumers)
return -ENOMEM;
+ names = (char *)(_consumers + num_consumers);
goto restart;
error:
diff --git a/drivers/regulator/tps6594-regulator.c b/drivers/regulator/tps6594-regulator.c
index 31a5218d5510..503b89a2f5f8 100644
--- a/drivers/regulator/tps6594-regulator.c
+++ b/drivers/regulator/tps6594-regulator.c
@@ -241,21 +241,21 @@ static const struct regulator_desc tps65224_buck_regs[] = {
TPS65224_MASK_BUCKS_VSET,
TPS6594_REG_BUCKX_CTRL(1),
TPS6594_BIT_BUCK_EN, 0, 0, tps65224_bucks_2_3_4_ranges,
- 4, 0, 0, NULL, 0, 0),
+ ARRAY_SIZE(tps65224_bucks_2_3_4_ranges), 0, 0, NULL, 0, 0),
TPS6594_REGULATOR("BUCK3", "buck3", TPS6594_BUCK_3,
REGULATOR_VOLTAGE, tps6594_bucks_ops, TPS65224_MASK_BUCKS_VSET,
TPS6594_REG_BUCKX_VOUT_1(2),
TPS65224_MASK_BUCKS_VSET,
TPS6594_REG_BUCKX_CTRL(2),
TPS6594_BIT_BUCK_EN, 0, 0, tps65224_bucks_2_3_4_ranges,
- 4, 0, 0, NULL, 0, 0),
+ ARRAY_SIZE(tps65224_bucks_2_3_4_ranges), 0, 0, NULL, 0, 0),
TPS6594_REGULATOR("BUCK4", "buck4", TPS6594_BUCK_4,
REGULATOR_VOLTAGE, tps6594_bucks_ops, TPS65224_MASK_BUCKS_VSET,
TPS6594_REG_BUCKX_VOUT_1(3),
TPS65224_MASK_BUCKS_VSET,
TPS6594_REG_BUCKX_CTRL(3),
TPS6594_BIT_BUCK_EN, 0, 0, tps65224_bucks_2_3_4_ranges,
- 4, 0, 0, NULL, 0, 0),
+ ARRAY_SIZE(tps65224_bucks_2_3_4_ranges), 0, 0, NULL, 0, 0),
};
static const struct tps6594_regulator_irq_type tps6594_buck1_irq_types[] = {
diff --git a/drivers/remoteproc/qcom_wcnss.c b/drivers/remoteproc/qcom_wcnss.c
index 2c7e519a2254..0a0eb1d7da82 100644
--- a/drivers/remoteproc/qcom_wcnss.c
+++ b/drivers/remoteproc/qcom_wcnss.c
@@ -441,25 +441,31 @@ static void wcnss_release_pds(struct qcom_wcnss *wcnss)
}
static int wcnss_init_regulators(struct qcom_wcnss *wcnss,
- const struct wcnss_vreg_info *info,
- int num_vregs, int num_pd_vregs)
+ const struct wcnss_data *data)
{
+ const struct wcnss_vreg_info *info = data->vregs;
struct regulator_bulk_data *bulk;
+ size_t i, possible_pds = 0, num_vregs = data->num_vregs;
int ret;
- int i;
+
+ for (i = 0; i < WCNSS_MAX_PDS; i++)
+ if (data->pd_names[i])
+ possible_pds++;
/*
* If attaching the power domains suceeded we can skip requesting
* the regulators for the power domains. For old device trees we need to
* reserve extra space to manage them through the regulator interface.
*/
- if (wcnss->num_pds) {
+ if (possible_pds >= num_vregs) {
+ /* Do nothing if vregs do not include PD regulators (pronto-v3) */
+ } else if (wcnss->num_pds) {
info += wcnss->num_pds;
/* Handle single power domain case */
- if (wcnss->num_pds < num_pd_vregs)
- num_vregs += num_pd_vregs - wcnss->num_pds;
+ if (wcnss->num_pds < data->num_pd_vregs)
+ num_vregs += data->num_pd_vregs - wcnss->num_pds;
} else {
- num_vregs += num_pd_vregs;
+ num_vregs += data->num_pd_vregs;
}
bulk = devm_kcalloc(wcnss->dev,
@@ -612,8 +618,7 @@ static int wcnss_probe(struct platform_device *pdev)
if (ret && (ret != -ENODATA || !data->num_pd_vregs))
return ret;
- ret = wcnss_init_regulators(wcnss, data->vregs, data->num_vregs,
- data->num_pd_vregs);
+ ret = wcnss_init_regulators(wcnss, data);
if (ret)
goto detach_pds;
diff --git a/drivers/s390/crypto/zcrypt_api.c b/drivers/s390/crypto/zcrypt_api.c
index a25b58e06f92..eff9ad7aa14e 100644
--- a/drivers/s390/crypto/zcrypt_api.c
+++ b/drivers/s390/crypto/zcrypt_api.c
@@ -854,13 +854,12 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms,
struct ica_xcRB *xcrb)
{
bool userspace = xflags & ZCRYPT_XFLAG_USERSPACE;
- struct zcrypt_card *zc, *pref_zc;
- struct zcrypt_queue *zq, *pref_zq;
- struct ap_message ap_msg;
+ unsigned int domain, func_code = 0;
unsigned int wgt = 0, pref_wgt = 0;
- unsigned int func_code = 0;
- unsigned short *domain, tdom;
+ struct zcrypt_queue *zq, *pref_zq;
+ struct zcrypt_card *zc, *pref_zc;
int cpen, qpen, qid = 0, rc;
+ struct ap_message ap_msg;
struct module *mod;
trace_s390_zcrypt_req(xcrb, TB_ZSECSENDCPRB);
@@ -878,10 +877,10 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms,
print_hex_dump_debug("ccareq: ", DUMP_PREFIX_ADDRESS, 16, 1,
ap_msg.msg, ap_msg.len, false);
- tdom = *domain;
- if (perms != &ap_perms && tdom < AP_DOMAINS) {
+ if (perms != &ap_perms && domain < AP_DOMAINS) {
if (ap_msg.flags & AP_MSG_FLAG_ADMIN) {
- if (!test_bit_inv(tdom, perms->adm)) {
+ domain = array_index_nospec(domain, AP_DOMAINS);
+ if (!test_bit_inv(domain, perms->adm)) {
rc = -ENODEV;
goto out;
}
@@ -894,10 +893,10 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms,
* If a valid target domain is set and this domain is NOT a usage
* domain but a control only domain, autoselect target domain.
*/
- if (tdom < AP_DOMAINS &&
- !ap_test_config_usage_domain(tdom) &&
- ap_test_config_ctrl_domain(tdom))
- tdom = AUTOSEL_DOM;
+ if (domain < AP_DOMAINS &&
+ !ap_test_config_usage_domain(domain) &&
+ ap_test_config_ctrl_domain(domain))
+ domain = AUTOSEL_DOM;
pref_zc = NULL;
pref_zq = NULL;
@@ -929,8 +928,8 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms,
/* check for device usable and eligible */
if (!zq->online || !zq->ops->send_cprb ||
!ap_queue_usable(zq->queue) ||
- (tdom != AUTOSEL_DOM &&
- tdom != AP_QID_QUEUE(zq->queue->qid)))
+ (domain != AUTOSEL_DOM &&
+ domain != AP_QID_QUEUE(zq->queue->qid)))
continue;
/* check if device node has admission for this queue */
if (!zcrypt_check_queue(perms,
@@ -953,16 +952,11 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms,
if (!pref_zq) {
pr_debug("no match for address %02x.%04x => ENODEV\n",
- xcrb->user_defined, *domain);
+ xcrb->user_defined, domain);
rc = -ENODEV;
goto out;
}
- /* in case of auto select, provide the correct domain */
- qid = pref_zq->queue->qid;
- if (*domain == AUTOSEL_DOM)
- *domain = AP_QID_QUEUE(qid);
-
rc = pref_zq->ops->send_cprb(userspace, pref_zq, xcrb, &ap_msg);
if (!rc) {
print_hex_dump_debug("ccarpl: ", DUMP_PREFIX_ADDRESS, 16, 1,
@@ -1085,6 +1079,7 @@ static long _zcrypt_send_ep11_cprb(u32 xflags, struct ap_perms *perms,
if (perms != &ap_perms && domain < AP_DOMAINS) {
if (ap_msg.flags & AP_MSG_FLAG_ADMIN) {
+ domain = array_index_nospec(domain, AP_DOMAINS);
if (!test_bit_inv(domain, perms->adm)) {
rc = -ENODEV;
goto out;
@@ -1218,7 +1213,6 @@ static long zcrypt_rng(char *buffer)
unsigned int wgt = 0, pref_wgt = 0;
unsigned int func_code = 0;
struct ap_message ap_msg;
- unsigned int domain;
int qid = 0, rc = -ENODEV;
struct module *mod;
@@ -1227,7 +1221,7 @@ static long zcrypt_rng(char *buffer)
rc = ap_init_apmsg(&ap_msg, 0);
if (rc)
goto out;
- rc = prep_rng_ap_msg(&ap_msg, &func_code, &domain);
+ rc = prep_rng_ap_msg(&ap_msg, &func_code, NULL);
if (rc)
goto out;
diff --git a/drivers/s390/crypto/zcrypt_msgtype6.c b/drivers/s390/crypto/zcrypt_msgtype6.c
index 5b9dce59db8b..f0ad39a33fb0 100644
--- a/drivers/s390/crypto/zcrypt_msgtype6.c
+++ b/drivers/s390/crypto/zcrypt_msgtype6.c
@@ -20,6 +20,7 @@
#include <linux/slab.h>
#include <linux/atomic.h>
#include <linux/uaccess.h>
+#include <linux/unaligned.h>
#include "ap_bus.h"
#include "zcrypt_api.h"
@@ -35,6 +36,9 @@
#define CEXXC_RESPONSE_TYPE_XCRB 1
#define CEXXC_RESPONSE_TYPE_EP11 2
+/* smallest possible EP11 payload size */
+#define MIN_EP11_PAYLOAD_SIZE 5
+
MODULE_AUTHOR("IBM Corporation");
MODULE_DESCRIPTION("Cryptographic Coprocessor (message type 6), " \
"Copyright IBM Corp. 2001, 2023");
@@ -329,7 +333,7 @@ struct type86_fmt2_msg {
static int xcrb_msg_to_type6cprb_msgx(bool userspace, struct ap_message *ap_msg,
struct ica_xcRB *xcrb,
unsigned int *fcode,
- unsigned short **dom)
+ unsigned int *domain)
{
static struct type6_hdr static_type6_hdrX = {
.type = 0x06,
@@ -343,49 +347,39 @@ static int xcrb_msg_to_type6cprb_msgx(bool userspace, struct ap_message *ap_msg,
};
} __packed * msg = ap_msg->msg;
- int rcblen = CEIL4(xcrb->request_control_blk_length);
- int req_sumlen, resp_sumlen;
- char *req_data = ap_msg->msg + sizeof(struct type6_hdr) + rcblen;
- char *function_code;
-
- if (CEIL4(xcrb->request_control_blk_length) <
- xcrb->request_control_blk_length)
- return -EINVAL; /* overflow after alignment*/
+ size_t req_cblen, rep_cblen, req_sumlen, rep_sumlen;
+ char *function_code, *req_data;
- /* length checks */
- ap_msg->len = sizeof(struct type6_hdr) +
- CEIL4(xcrb->request_control_blk_length) +
- xcrb->request_data_length;
+ /* request length and overflow checks */
+ if (xcrb->request_control_blk_length < sizeof(struct CPRBX))
+ return -EINVAL;
+ req_cblen = CEIL4((size_t)xcrb->request_control_blk_length);
+ if (req_cblen > U32_MAX)
+ return -EINVAL;
+ req_sumlen = req_cblen + xcrb->request_data_length;
+ if (req_sumlen > U32_MAX)
+ return -EINVAL;
+ ap_msg->len = sizeof(struct type6_hdr) + req_sumlen;
if (ap_msg->len > ap_msg->bufsize)
return -EINVAL;
-
- /*
- * Overflow check
- * sum must be greater (or equal) than the largest operand
- */
- req_sumlen = CEIL4(xcrb->request_control_blk_length) +
- xcrb->request_data_length;
- if ((CEIL4(xcrb->request_control_blk_length) <=
- xcrb->request_data_length) ?
+ if (req_cblen <= xcrb->request_data_length ?
req_sumlen < xcrb->request_data_length :
- req_sumlen < CEIL4(xcrb->request_control_blk_length)) {
+ req_sumlen < req_cblen) {
return -EINVAL;
}
- if (CEIL4(xcrb->reply_control_blk_length) <
- xcrb->reply_control_blk_length)
- return -EINVAL; /* overflow after alignment*/
-
- /*
- * Overflow check
- * sum must be greater (or equal) than the largest operand
- */
- resp_sumlen = CEIL4(xcrb->reply_control_blk_length) +
- xcrb->reply_data_length;
- if ((CEIL4(xcrb->reply_control_blk_length) <=
- xcrb->reply_data_length) ?
- resp_sumlen < xcrb->reply_data_length :
- resp_sumlen < CEIL4(xcrb->reply_control_blk_length)) {
+ /* reply length and overflow checks */
+ if (xcrb->reply_control_blk_length < sizeof(struct CPRBX))
+ return -EINVAL;
+ rep_cblen = CEIL4((size_t)xcrb->reply_control_blk_length);
+ if (rep_cblen > U32_MAX)
+ return -EINVAL;
+ rep_sumlen = rep_cblen + xcrb->reply_data_length;
+ if (rep_sumlen > U32_MAX)
+ return -EINVAL;
+ if (rep_cblen <= xcrb->reply_data_length ?
+ rep_sumlen < xcrb->reply_data_length :
+ rep_sumlen < rep_cblen) {
return -EINVAL;
}
@@ -394,7 +388,7 @@ static int xcrb_msg_to_type6cprb_msgx(bool userspace, struct ap_message *ap_msg,
memcpy(msg->hdr.agent_id, &xcrb->agent_ID, sizeof(xcrb->agent_ID));
msg->hdr.tocardlen1 = xcrb->request_control_blk_length;
if (xcrb->request_data_length) {
- msg->hdr.offset2 = msg->hdr.offset1 + rcblen;
+ msg->hdr.offset2 = msg->hdr.offset1 + req_cblen;
msg->hdr.tocardlen2 = xcrb->request_data_length;
}
msg->hdr.fromcardlen1 = xcrb->reply_control_blk_length;
@@ -405,15 +399,20 @@ static int xcrb_msg_to_type6cprb_msgx(bool userspace, struct ap_message *ap_msg,
xcrb->request_control_blk_addr,
xcrb->request_control_blk_length))
return -EFAULT;
- if (msg->cprbx.cprb_len + sizeof(msg->hdr.function_code) >
- xcrb->request_control_blk_length)
+ /* pad tail with 0 up to req_cblen */
+ if (xcrb->request_control_blk_length < req_cblen)
+ memset(msg->userdata + xcrb->request_control_blk_length,
+ 0, req_cblen - xcrb->request_control_blk_length);
+ /* copy subfunction code into AP msg type 6 function code field */
+ if (msg->cprbx.cprb_len > req_cblen - sizeof(msg->hdr.function_code))
return -EINVAL;
function_code = ((unsigned char *)&msg->cprbx) + msg->cprbx.cprb_len;
memcpy(msg->hdr.function_code, function_code,
sizeof(msg->hdr.function_code));
*fcode = (msg->hdr.function_code[0] << 8) | msg->hdr.function_code[1];
- *dom = (unsigned short *)&msg->cprbx.domain;
+ if (domain)
+ *domain = msg->cprbx.domain;
/* check subfunction, US and AU need special flag with NQAP */
if (memcmp(function_code, "US", 2) == 0 ||
@@ -437,20 +436,70 @@ static int xcrb_msg_to_type6cprb_msgx(bool userspace, struct ap_message *ap_msg,
}
/* copy data block */
- if (xcrb->request_data_length &&
- z_copy_from_user(userspace, req_data, xcrb->request_data_address,
- xcrb->request_data_length))
- return -EFAULT;
+ if (xcrb->request_data_length) {
+ req_data = ap_msg->msg + sizeof(struct type6_hdr) + req_cblen;
+ if (z_copy_from_user(userspace, req_data,
+ xcrb->request_data_address,
+ xcrb->request_data_length))
+ return -EFAULT;
+ }
return 0;
}
+/*
+ * Simple asn1 int reader/decoder helper function
+ * Returns number of bytes processed or < 0 on failure
+ * Only accepts int length values of 1, 2 or 4.
+ */
+static inline int asn1_int_decode(const u8 *buf, size_t intlen, u32 *u)
+{
+ switch (intlen) {
+ case 1:
+ *u = (u32)(*buf);
+ return 1;
+ case 2:
+ *u = (u32)get_unaligned_be16(buf);
+ return 2;
+ case 4:
+ *u = (u32)get_unaligned_be32(buf);
+ return 4;
+ default:
+ return -EINVAL;
+ }
+}
+
+/*
+ * Simple asn1 length parse helper function
+ * Returns number of bytes processed or < 0 on failure
+ * Only accepts length encoded within the length octet
+ * or for long form 1, 2 or 4 octet length bytes.
+ */
+static inline int asn1_length_decode(const u8 *buf, size_t buflen, u32 *u)
+{
+ int i;
+
+ if (buflen < 1)
+ return -EINVAL;
+
+ if (*buf < 128) {
+ *u = (u32)(*buf & 0x7F);
+ return 1;
+ }
+
+ i = *buf & 0x7F;
+ if (--buflen < i)
+ return -EINVAL;
+ i = asn1_int_decode(++buf, i, u);
+
+ return i < 0 ? i : i + 1;
+}
+
static int xcrb_msg_to_type6_ep11cprb_msgx(bool userspace, struct ap_message *ap_msg,
struct ep11_urb *xcrb,
unsigned int *fcode,
unsigned int *domain)
{
- unsigned int lfmt;
static struct type6_hdr static_type6_ep11_hdr = {
.type = 0x06,
.rqid = {0x00, 0x01},
@@ -463,34 +512,32 @@ static int xcrb_msg_to_type6_ep11cprb_msgx(bool userspace, struct ap_message *ap
struct {
struct type6_hdr hdr;
union {
- struct {
- struct ep11_cprb cprbx;
- unsigned char pld_tag; /* fixed value 0x30 */
- unsigned char pld_lenfmt; /* length format */
- } __packed;
+ struct ep11_cprb cprbx;
DECLARE_FLEX_ARRAY(u8, userdata);
};
} __packed * msg = ap_msg->msg;
- struct pld_hdr {
- unsigned char func_tag; /* fixed value 0x4 */
- unsigned char func_len; /* fixed value 0x4 */
- unsigned int func_val; /* function ID */
- unsigned char dom_tag; /* fixed value 0x4 */
- unsigned char dom_len; /* fixed value 0x4 */
- unsigned int dom_val; /* domain id */
- } __packed * payload_hdr = NULL;
+ size_t req_len, rep_len, pld_len;
+ unsigned char *pld;
+ int offs = 0, i;
+ unsigned int u;
- if (CEIL4(xcrb->req_len) < xcrb->req_len)
- return -EINVAL; /* overflow after alignment*/
-
- /* length checks */
- ap_msg->len = sizeof(struct type6_hdr) + CEIL4(xcrb->req_len);
+ /* request length and overflow checks */
+ if (xcrb->req_len < sizeof(struct ep11_cprb) + MIN_EP11_PAYLOAD_SIZE)
+ return -EINVAL;
+ req_len = CEIL4(xcrb->req_len);
+ if (req_len < xcrb->req_len || req_len > U32_MAX)
+ return -EINVAL;
+ ap_msg->len = sizeof(struct type6_hdr) + req_len;
if (ap_msg->len > ap_msg->bufsize)
return -EINVAL;
- if (CEIL4(xcrb->resp_len) < xcrb->resp_len)
- return -EINVAL; /* overflow after alignment*/
+ /* reply length and overflow checks */
+ if (xcrb->resp_len < sizeof(struct ep11_cprb))
+ return -EINVAL;
+ rep_len = CEIL4(xcrb->resp_len);
+ if (rep_len < xcrb->resp_len || rep_len > U32_MAX)
+ return -EINVAL;
/* prepare type6 header */
msg->hdr = static_type6_ep11_hdr;
@@ -499,26 +546,55 @@ static int xcrb_msg_to_type6_ep11cprb_msgx(bool userspace, struct ap_message *ap
/* Import CPRB data from the ioctl input parameter */
if (z_copy_from_user(userspace, msg->userdata,
- (char __force __user *)xcrb->req, xcrb->req_len)) {
+ (char __force __user *)xcrb->req, xcrb->req_len))
return -EFAULT;
- }
+ /* pad tail with 0 up to req_len */
+ if (xcrb->req_len < req_len)
+ memset(msg->userdata + xcrb->req_len, 0,
+ req_len - xcrb->req_len);
+
+ pld = msg->userdata + sizeof(struct ep11_cprb);
+ pld_len = msg->cprbx.payload_len;
+ if (pld_len != xcrb->req_len - sizeof(struct ep11_cprb))
+ return -EINVAL;
+ /*
+ * At this point pld_len is always >= MIN_EP11_PAYLOAD_SIZE
+ * and the smallest supported asn1 payload is:
+ * payload tag (1 octet)
+ * payload length (1-5 octets)
+ * function tag (1 octet)
+ * function length (1-5 octets)
+ * function value (1-4 octets)
+ */
- if ((msg->pld_lenfmt & 0x80) == 0x80) { /*ext.len.fmt 2 or 3*/
- switch (msg->pld_lenfmt & 0x03) {
- case 1:
- lfmt = 2;
- break;
- case 2:
- lfmt = 3;
- break;
- default:
- return -EINVAL;
- }
- } else {
- lfmt = 1; /* length format #1 */
- }
- payload_hdr = (struct pld_hdr *)((&msg->pld_lenfmt) + lfmt);
- *fcode = payload_hdr->func_val & 0xFFFF;
+ /* payload tag */
+ if (pld[offs++] != 0x30)
+ return -EINVAL;
+ /* payload length field */
+ i = asn1_length_decode(pld + offs, pld_len - offs, &u);
+ if (i < 0)
+ return -EINVAL;
+ offs += i;
+ if (offs >= pld_len || u > pld_len - offs)
+ return -EINVAL;
+ /* function tag */
+ if (pld[offs++] != 0x04)
+ return -EINVAL;
+ /* function length */
+ if (offs >= pld_len)
+ return -EINVAL;
+ i = asn1_length_decode(pld + offs, pld_len - offs, &u);
+ if (i < 0)
+ return -EINVAL;
+ offs += i;
+ if (offs >= pld_len || u > pld_len - offs)
+ return -EINVAL;
+ /* function value */
+ i = asn1_int_decode(pld + offs, u, &u);
+ if (i < 0)
+ return -EINVAL;
+ offs += i;
+ *fcode = 0xFFFF & u;
/* enable special processing based on the cprbs flags special bit */
if (msg->cprbx.flags & 0x20)
@@ -530,7 +606,8 @@ static int xcrb_msg_to_type6_ep11cprb_msgx(bool userspace, struct ap_message *ap
else
ap_msg->flags |= AP_MSG_FLAG_USAGE;
- *domain = msg->cprbx.target_id;
+ if (domain)
+ *domain = msg->cprbx.target_id;
return 0;
}
@@ -1053,7 +1130,7 @@ static long zcrypt_msgtype6_modexpo_crt(struct zcrypt_queue *zq,
*/
int prep_cca_ap_msg(bool userspace, struct ica_xcRB *xcrb,
struct ap_message *ap_msg,
- unsigned int *func_code, unsigned short **dom)
+ unsigned int *func_code, unsigned int *domain)
{
struct ap_response_type *resp_type = &ap_msg->response;
@@ -1061,7 +1138,8 @@ int prep_cca_ap_msg(bool userspace, struct ica_xcRB *xcrb,
ap_msg->psmid = (((unsigned long)current->pid) << 32) +
atomic_inc_return(&zcrypt_step);
resp_type->type = CEXXC_RESPONSE_TYPE_XCRB;
- return xcrb_msg_to_type6cprb_msgx(userspace, ap_msg, xcrb, func_code, dom);
+ return xcrb_msg_to_type6cprb_msgx(userspace, ap_msg,
+ xcrb, func_code, domain);
}
/*
@@ -1106,6 +1184,9 @@ static long zcrypt_msgtype6_send_cprb(bool userspace, struct zcrypt_queue *zq,
msg->hdr.fromcardlen1 -= delta;
}
+ /* update domain field within the CPRB struct */
+ msg->cprbx.domain = AP_QID_QUEUE(zq->queue->qid);
+
init_completion(&resp_type->work);
rc = ap_queue_message(zq->queue, ap_msg);
if (rc)
@@ -1155,6 +1236,28 @@ int prep_ep11_ap_msg(bool userspace, struct ep11_urb *xcrb,
func_code, domain);
}
+/*
+ * Simple asn1 int writer/encoder helper function
+ * Returns number of bytes processed or < 0 on failure
+ * Only accepts int length values of 1, 2 or 4.
+ */
+static inline int asn1_int_encode(u8 *buf, size_t intlen, u32 u)
+{
+ switch (intlen) {
+ case 1:
+ *buf = (u8)u;
+ return 1;
+ case 2:
+ put_unaligned_be16((u16)u, buf);
+ return 2;
+ case 4:
+ put_unaligned_be32((u32)u, buf);
+ return 4;
+ default:
+ return -EINVAL;
+ }
+}
+
/*
* The request distributor calls this function if it picked the CEX4P
* device to handle a send_ep11_cprb request.
@@ -1167,52 +1270,94 @@ static long zcrypt_msgtype6_send_ep11_cprb(bool userspace, struct zcrypt_queue *
struct ap_message *ap_msg)
{
int rc;
- unsigned int lfmt;
struct ap_response_type *resp_type = &ap_msg->response;
struct {
struct type6_hdr hdr;
struct ep11_cprb cprbx;
- unsigned char pld_tag; /* fixed value 0x30 */
- unsigned char pld_lenfmt; /* payload length format */
} __packed * msg = ap_msg->msg;
- struct pld_hdr {
- unsigned char func_tag; /* fixed value 0x4 */
- unsigned char func_len; /* fixed value 0x4 */
- unsigned int func_val; /* function ID */
- unsigned char dom_tag; /* fixed value 0x4 */
- unsigned char dom_len; /* fixed value 0x4 */
- unsigned int dom_val; /* domain id */
- } __packed * payload_hdr = NULL;
/*
* The target domain field within the cprb body/payload block will be
* replaced by the usage domain for non-management commands only.
* Therefore we check the first bit of the 'flags' parameter for
* management command indication.
- * 0 - non management command
- * 1 - management command
*/
- if (!((msg->cprbx.flags & 0x80) == 0x80)) {
- msg->cprbx.target_id = (unsigned int)
- AP_QID_QUEUE(zq->queue->qid);
-
- if ((msg->pld_lenfmt & 0x80) == 0x80) { /*ext.len.fmt 2 or 3*/
- switch (msg->pld_lenfmt & 0x03) {
- case 1:
- lfmt = 2;
- break;
- case 2:
- lfmt = 3;
- break;
- default:
+ if (!(msg->cprbx.flags & 0x80)) {
+ int i, offs = 0;
+ size_t pld_len;
+ u8 *pld;
+ u32 u;
+
+ /* update target field in ep11_cprb */
+ msg->cprbx.target_id = (u32)AP_QID_QUEUE(zq->queue->qid);
+
+ /* ptr and length to payload */
+ pld = ap_msg->msg +
+ sizeof(struct type6_hdr) + sizeof(struct ep11_cprb);
+ pld_len = msg->cprbx.payload_len;
+ if (pld_len < MIN_EP11_PAYLOAD_SIZE)
+ return -EINVAL;
+
+ /*
+ * Parse the asn1 payload, at least we have
+ * pld tag (1 octet)
+ * payload length (1-5 octets)
+ * function tag (1 octet)
+ * function length (1-5 octets)
+ * function value (1-4 octets)
+ * ----- optional fields -----
+ * domain tag (1 octet)
+ * domain length (1-5 octets)
+ * domain value (1-4 octets)
+ * ... maybe much more data ...
+ */
+
+ /* payload tag */
+ if (pld[offs++] != 0x30)
+ return -EINVAL;
+ /* payload length field */
+ i = asn1_length_decode(pld + offs, pld_len - offs, &u);
+ if (i < 0)
+ return -EINVAL;
+ offs += i;
+ if (offs >= pld_len || u > pld_len - offs)
+ return -EINVAL;
+ /* function tag */
+ if (pld[offs++] != 0x04)
+ return -EINVAL;
+ /* function length */
+ if (offs >= pld_len)
+ return -EINVAL;
+ i = asn1_length_decode(pld + offs, pld_len - offs, &u);
+ if (i < 0)
+ return -EINVAL;
+ offs += i;
+ if (u > pld_len - offs)
+ return -EINVAL;
+ /* skip over the function value */
+ offs += u;
+ /* is there some payload left which could hold a domain value ? */
+ if (offs < pld_len && pld_len - offs >= 3) {
+ /* domain tag */
+ if (pld[offs++] != 0x04)
return -EINVAL;
- }
- } else {
- lfmt = 1; /* length format #1 */
+ /* domain length */
+ i = asn1_length_decode(pld + offs, pld_len - offs, &u);
+ if (i < 0)
+ return -EINVAL;
+ offs += i;
+ if (offs >= pld_len || u > pld_len - offs)
+ return -EINVAL;
+ /*
+ * pld[offs] is now at the start of the domain value
+ * with the value sprawled in u octets.
+ */
+ i = asn1_int_encode(pld + offs, u,
+ AP_QID_QUEUE(zq->queue->qid));
+ if (i < 0)
+ return -EINVAL;
+ offs += i;
}
- payload_hdr = (struct pld_hdr *)((&msg->pld_lenfmt) + lfmt);
- payload_hdr->dom_val = (unsigned int)
- AP_QID_QUEUE(zq->queue->qid);
}
/*
@@ -1247,6 +1392,56 @@ static long zcrypt_msgtype6_send_ep11_cprb(bool userspace, struct zcrypt_queue *
return rc;
}
+/**
+ * Prepare a type6 CPRB message for random number generation
+ *
+ * @ap_dev: AP device pointer
+ * @ap_msg: pointer to AP message
+ */
+static inline void rng_type6cprb_msgx(struct ap_message *ap_msg,
+ unsigned int random_number_length,
+ unsigned int *domain)
+{
+ struct {
+ struct type6_hdr hdr;
+ struct CPRBX cprbx;
+ char function_code[2];
+ short int rule_length;
+ char rule[8];
+ short int verb_length;
+ short int key_length;
+ } __packed * msg = ap_msg->msg;
+ static struct type6_hdr static_type6_hdrX = {
+ .type = 0x06,
+ .offset1 = 0x00000058,
+ .agent_id = {'C', 'A'},
+ .function_code = {'R', 'L'},
+ .tocardlen1 = sizeof(*msg) - sizeof(msg->hdr),
+ .fromcardlen1 = sizeof(*msg) - sizeof(msg->hdr),
+ };
+ static struct CPRBX local_cprbx = {
+ .cprb_len = 0x00dc,
+ .cprb_ver_id = 0x02,
+ .func_id = {0x54, 0x32},
+ .req_parml = sizeof(*msg) - sizeof(msg->hdr) -
+ sizeof(msg->cprbx),
+ .rpl_msgbl = sizeof(*msg) - sizeof(msg->hdr),
+ };
+
+ msg->hdr = static_type6_hdrX;
+ msg->hdr.fromcardlen2 = random_number_length;
+ msg->cprbx = local_cprbx;
+ msg->cprbx.rpl_datal = random_number_length;
+ memcpy(msg->function_code, msg->hdr.function_code, 0x02);
+ msg->rule_length = 0x0a;
+ memcpy(msg->rule, "RANDOM ", 8);
+ msg->verb_length = 0x02;
+ msg->key_length = 0x02;
+ ap_msg->len = sizeof(*msg);
+ if (domain)
+ *domain = msg->cprbx.domain;
+}
+
/*
* Prepare a CEXXC get random request ap message.
* This function assumes that ap_msg has been initialized with
diff --git a/drivers/s390/crypto/zcrypt_msgtype6.h b/drivers/s390/crypto/zcrypt_msgtype6.h
index 6f5ced8d6cda..9cbe6e2b2d33 100644
--- a/drivers/s390/crypto/zcrypt_msgtype6.h
+++ b/drivers/s390/crypto/zcrypt_msgtype6.h
@@ -96,7 +96,7 @@ struct type86_fmt2_ext {
int prep_cca_ap_msg(bool userspace, struct ica_xcRB *xcrb,
struct ap_message *ap_msg,
- unsigned int *fc, unsigned short **dom);
+ unsigned int *fc, unsigned int *dom);
int prep_ep11_ap_msg(bool userspace, struct ep11_urb *xcrb,
struct ap_message *ap_msg,
unsigned int *fc, unsigned int *dom);
@@ -110,55 +110,6 @@ int prep_rng_ap_msg(struct ap_message *ap_msg,
int speed_idx_cca(int);
int speed_idx_ep11(int);
-/**
- * Prepare a type6 CPRB message for random number generation
- *
- * @ap_dev: AP device pointer
- * @ap_msg: pointer to AP message
- */
-static inline void rng_type6cprb_msgx(struct ap_message *ap_msg,
- unsigned int random_number_length,
- unsigned int *domain)
-{
- struct {
- struct type6_hdr hdr;
- struct CPRBX cprbx;
- char function_code[2];
- short int rule_length;
- char rule[8];
- short int verb_length;
- short int key_length;
- } __packed * msg = ap_msg->msg;
- static struct type6_hdr static_type6_hdrX = {
- .type = 0x06,
- .offset1 = 0x00000058,
- .agent_id = {'C', 'A'},
- .function_code = {'R', 'L'},
- .tocardlen1 = sizeof(*msg) - sizeof(msg->hdr),
- .fromcardlen1 = sizeof(*msg) - sizeof(msg->hdr),
- };
- static struct CPRBX local_cprbx = {
- .cprb_len = 0x00dc,
- .cprb_ver_id = 0x02,
- .func_id = {0x54, 0x32},
- .req_parml = sizeof(*msg) - sizeof(msg->hdr) -
- sizeof(msg->cprbx),
- .rpl_msgbl = sizeof(*msg) - sizeof(msg->hdr),
- };
-
- msg->hdr = static_type6_hdrX;
- msg->hdr.fromcardlen2 = random_number_length;
- msg->cprbx = local_cprbx;
- msg->cprbx.rpl_datal = random_number_length;
- memcpy(msg->function_code, msg->hdr.function_code, 0x02);
- msg->rule_length = 0x0a;
- memcpy(msg->rule, "RANDOM ", 8);
- msg->verb_length = 0x02;
- msg->key_length = 0x02;
- ap_msg->len = sizeof(*msg);
- *domain = (unsigned short)msg->cprbx.domain;
-}
-
void zcrypt_msgtype6_init(void);
void zcrypt_msgtype6_exit(void);
diff --git a/drivers/scsi/qedi/qedi_main.c b/drivers/scsi/qedi/qedi_main.c
index b168bb2178e9..98669d967afa 100644
--- a/drivers/scsi/qedi/qedi_main.c
+++ b/drivers/scsi/qedi/qedi_main.c
@@ -2654,8 +2654,6 @@ static int __qedi_probe(struct pci_dev *pdev, int mode)
snprintf(host_buf, sizeof(host_buf), "host_%d", qedi->shost->host_no);
qedi_ops->common->set_name(qedi->cdev, host_buf);
- qedi_ops->register_ops(qedi->cdev, &qedi_cb_ops, qedi);
-
memset(¶ms, 0, sizeof(params));
params.mtu = DEF_PATH_MTU + IPV6_HDR_LEN + TCP_HDR_LEN;
qedi->ll2_mtu = DEF_PATH_MTU;
@@ -2672,6 +2670,7 @@ static int __qedi_probe(struct pci_dev *pdev, int mode)
spin_lock_init(&qedi->task_idx_lock);
mutex_init(&qedi->stats_lock);
}
+ qedi_ops->register_ops(qedi->cdev, &qedi_cb_ops, qedi);
qedi_ops->ll2->register_cb_ops(qedi->cdev, &qedi_ll2_cb_ops, qedi);
qedi_ops->ll2->start(qedi->cdev, ¶ms);
diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c
index 5ad867c6f26e..9a3adf40e0d7 100644
--- a/drivers/scsi/qla2xxx/qla_iocb.c
+++ b/drivers/scsi/qla2xxx/qla_iocb.c
@@ -2758,7 +2758,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode,
if (!elsio->u.els_logo.els_logo_pyld) {
/* ref: INIT */
kref_put(&sp->cmd_kref, qla2x00_sp_release);
- qla2x00_free_fcport(fcport);
return QLA_FUNCTION_FAILED;
}
@@ -2783,7 +2782,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode,
if (rval != QLA_SUCCESS) {
/* ref: INIT */
kref_put(&sp->cmd_kref, qla2x00_sp_release);
- qla2x00_free_fcport(fcport);
return QLA_FUNCTION_FAILED;
}
diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 857c2acc9964..39870ae74890 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -2358,6 +2358,7 @@ static void scsi_unjam_host(struct Scsi_Host *shost)
int scsi_error_handler(void *data)
{
struct Scsi_Host *shost = data;
+ bool eh_noresume;
/*
* We use TASK_INTERRUPTIBLE so that the thread is not
@@ -2399,7 +2400,8 @@ int scsi_error_handler(void *data)
* what we need to do to get it up and online again (if we can).
* If we fail, we end up taking the thing offline.
*/
- if (!shost->eh_noresume && scsi_autopm_get_host(shost) != 0) {
+ eh_noresume = READ_ONCE(shost->eh_noresume);
+ if (!eh_noresume && scsi_autopm_get_host(shost) != 0) {
SCSI_LOG_ERROR_RECOVERY(1,
shost_printk(KERN_ERR, shost,
"scsi_eh_%d: unable to autoresume\n",
@@ -2423,7 +2425,7 @@ int scsi_error_handler(void *data)
* which are still online.
*/
scsi_restart_operations(shost);
- if (!shost->eh_noresume)
+ if (!eh_noresume)
scsi_autopm_put_host(shost);
}
__set_current_state(TASK_RUNNING);
diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index 41ad0e9101b8..145670b4a1cc 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -102,8 +102,6 @@ MODULE_ALIAS_SCSI_DEVICE(TYPE_ZBC);
#define SD_MINORS 16
-static void sd_config_discard(struct scsi_disk *sdkp, struct queue_limits *lim,
- unsigned int mode);
static void sd_config_write_same(struct scsi_disk *sdkp,
struct queue_limits *lim);
static void sd_revalidate_disk(struct gendisk *);
@@ -121,6 +119,62 @@ static const char *sd_cache_types[] = {
"write back, no read (daft)"
};
+static void sd_disable_discard(struct scsi_disk *sdkp)
+{
+ sdkp->provisioning_mode = SD_LBP_DISABLE;
+ blk_queue_disable_discard(sdkp->disk->queue);
+}
+
+static void sd_config_discard(struct scsi_disk *sdkp, struct queue_limits *lim,
+ unsigned int mode)
+{
+ unsigned int logical_block_size = sdkp->device->sector_size;
+ unsigned int max_blocks = 0;
+
+ lim->discard_alignment = sdkp->unmap_alignment * logical_block_size;
+ lim->discard_granularity = max(sdkp->physical_block_size,
+ sdkp->unmap_granularity * logical_block_size);
+ sdkp->provisioning_mode = mode;
+
+ switch (mode) {
+
+ case SD_LBP_FULL:
+ case SD_LBP_DISABLE:
+ break;
+
+ case SD_LBP_UNMAP:
+ max_blocks = min_not_zero(sdkp->max_unmap_blocks,
+ (u32)SD_MAX_WS16_BLOCKS);
+ break;
+
+ case SD_LBP_WS16:
+ if (sdkp->device->unmap_limit_for_ws)
+ max_blocks = sdkp->max_unmap_blocks;
+ else
+ max_blocks = sdkp->max_ws_blocks;
+
+ max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS16_BLOCKS);
+ break;
+
+ case SD_LBP_WS10:
+ if (sdkp->device->unmap_limit_for_ws)
+ max_blocks = sdkp->max_unmap_blocks;
+ else
+ max_blocks = sdkp->max_ws_blocks;
+
+ max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS10_BLOCKS);
+ break;
+
+ case SD_LBP_ZERO:
+ max_blocks = min_not_zero(sdkp->max_ws_blocks,
+ (u32)SD_MAX_WS10_BLOCKS);
+ break;
+ }
+
+ lim->max_hw_discard_sectors = max_blocks *
+ (logical_block_size >> SECTOR_SHIFT);
+}
+
static void sd_set_flush_flag(struct scsi_disk *sdkp,
struct queue_limits *lim)
{
@@ -836,60 +890,15 @@ static unsigned char sd_setup_protect_cmnd(struct scsi_cmnd *scmd,
return protect;
}
-static void sd_disable_discard(struct scsi_disk *sdkp)
-{
- sdkp->provisioning_mode = SD_LBP_DISABLE;
- blk_queue_disable_discard(sdkp->disk->queue);
-}
-
-static void sd_config_discard(struct scsi_disk *sdkp, struct queue_limits *lim,
- unsigned int mode)
+static void sd_uninit_command(struct scsi_cmnd *cmd)
{
- unsigned int logical_block_size = sdkp->device->sector_size;
- unsigned int max_blocks = 0;
-
- lim->discard_alignment = sdkp->unmap_alignment * logical_block_size;
- lim->discard_granularity = max(sdkp->physical_block_size,
- sdkp->unmap_granularity * logical_block_size);
- sdkp->provisioning_mode = mode;
-
- switch (mode) {
-
- case SD_LBP_FULL:
- case SD_LBP_DISABLE:
- break;
-
- case SD_LBP_UNMAP:
- max_blocks = min_not_zero(sdkp->max_unmap_blocks,
- (u32)SD_MAX_WS16_BLOCKS);
- break;
-
- case SD_LBP_WS16:
- if (sdkp->device->unmap_limit_for_ws)
- max_blocks = sdkp->max_unmap_blocks;
- else
- max_blocks = sdkp->max_ws_blocks;
-
- max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS16_BLOCKS);
- break;
-
- case SD_LBP_WS10:
- if (sdkp->device->unmap_limit_for_ws)
- max_blocks = sdkp->max_unmap_blocks;
- else
- max_blocks = sdkp->max_ws_blocks;
-
- max_blocks = min_not_zero(max_blocks, (u32)SD_MAX_WS10_BLOCKS);
- break;
+ struct request *rq = scsi_cmd_to_rq(cmd);
- case SD_LBP_ZERO:
- max_blocks = min_not_zero(sdkp->max_ws_blocks,
- (u32)SD_MAX_WS10_BLOCKS);
- break;
- }
+ if (!(rq->rq_flags & RQF_SPECIAL_PAYLOAD))
+ return;
- lim->max_hw_discard_sectors = max_blocks *
- (logical_block_size >> SECTOR_SHIFT);
+ mempool_free(rq->special_vec.bv_page, sd_page_pool);
+ rq->rq_flags &= ~RQF_SPECIAL_PAYLOAD;
}
static void *sd_set_special_bvec(struct request *rq, unsigned int data_len)
@@ -914,6 +923,7 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd)
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
unsigned int data_len = 24;
char *buf;
+ blk_status_t ret;
buf = sd_set_special_bvec(rq, data_len);
if (!buf)
@@ -932,7 +942,10 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd)
cmd->transfersize = data_len;
rq->timeout = SD_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static void sd_config_atomic(struct scsi_disk *sdkp, struct queue_limits *lim)
@@ -1003,6 +1016,7 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd,
u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq));
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
u32 data_len = sdp->sector_size;
+ blk_status_t ret;
if (!sd_set_special_bvec(rq, data_len))
return BLK_STS_RESOURCE;
@@ -1018,7 +1032,10 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd,
cmd->transfersize = data_len;
rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
@@ -1030,6 +1047,7 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq));
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
u32 data_len = sdp->sector_size;
+ blk_status_t ret;
if (!sd_set_special_bvec(rq, data_len))
return BLK_STS_RESOURCE;
@@ -1045,7 +1063,10 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
cmd->transfersize = data_len;
rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static blk_status_t sd_setup_write_zeroes_cmnd(struct scsi_cmnd *cmd)
@@ -1474,14 +1495,6 @@ static blk_status_t sd_init_command(struct scsi_cmnd *cmd)
}
}
-static void sd_uninit_command(struct scsi_cmnd *SCpnt)
-{
- struct request *rq = scsi_cmd_to_rq(SCpnt);
-
- if (rq->rq_flags & RQF_SPECIAL_PAYLOAD)
- mempool_free(rq->special_vec.bv_page, sd_page_pool);
-}
-
static bool sd_need_revalidate(struct gendisk *disk, struct scsi_disk *sdkp)
{
if (sdkp->device->removable || sdkp->write_prot) {
diff --git a/drivers/soc/aspeed/aspeed-lpc-snoop.c b/drivers/soc/aspeed/aspeed-lpc-snoop.c
index b03310c0830d..486d43406bc6 100644
--- a/drivers/soc/aspeed/aspeed-lpc-snoop.c
+++ b/drivers/soc/aspeed/aspeed-lpc-snoop.c
@@ -11,6 +11,7 @@
*/
#include <linux/bitops.h>
+#include <linux/cleanup.h>
#include <linux/clk.h>
#include <linux/dev_printk.h>
#include <linux/interrupt.h>
@@ -74,6 +75,7 @@ struct aspeed_lpc_snoop_channel_cfg {
struct aspeed_lpc_snoop_channel {
const struct aspeed_lpc_snoop_channel_cfg *cfg;
bool enabled;
+ spinlock_t lock;
struct kfifo fifo;
wait_queue_head_t wq;
struct miscdevice miscdev;
@@ -114,6 +116,7 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer,
size_t count, loff_t *ppos)
{
struct aspeed_lpc_snoop_channel *chan = snoop_file_to_chan(file);
+ u8 *buf __free(kfree) = NULL;
unsigned int copied;
int ret = 0;
@@ -125,9 +128,16 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer,
if (ret == -ERESTARTSYS)
return -EINTR;
}
- ret = kfifo_to_user(&chan->fifo, buffer, count, &copied);
- if (ret)
- return ret;
+
+ count = min_t(size_t, count, SNOOP_FIFO_SIZE);
+
+ buf = kmalloc(count, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ copied = kfifo_out_spinlocked(&chan->fifo, buf, count, &chan->lock);
+ if (copied && copy_to_user(buffer, buf, copied))
+ return -EFAULT;
return copied;
}
@@ -151,11 +161,13 @@ static const struct file_operations snoop_fops = {
/* Save a byte to a FIFO and discard the oldest byte if FIFO is full */
static void put_fifo_with_discard(struct aspeed_lpc_snoop_channel *chan, u8 val)
{
- if (!kfifo_initialized(&chan->fifo))
- return;
- if (kfifo_is_full(&chan->fifo))
- kfifo_skip(&chan->fifo);
- kfifo_put(&chan->fifo, val);
+ scoped_guard(spinlock, &chan->lock) {
+ if (!kfifo_initialized(&chan->fifo))
+ return;
+ if (kfifo_is_full(&chan->fifo))
+ kfifo_skip(&chan->fifo);
+ kfifo_put(&chan->fifo, val);
+ }
wake_up_interruptible(&chan->wq);
}
@@ -239,6 +251,7 @@ static int aspeed_lpc_enable_snoop(struct device *dev,
if (!channel->miscdev.name)
return -ENOMEM;
+ spin_lock_init(&channel->lock);
rc = kfifo_alloc(&channel->fifo, SNOOP_FIFO_SIZE, GFP_KERNEL);
if (rc)
return rc;
diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
index 93bbebd68001..911085d8a04b 100644
--- a/drivers/soc/tegra/pmc.c
+++ b/drivers/soc/tegra/pmc.c
@@ -28,6 +28,7 @@
#include <linux/iopoll.h>
#include <linux/irqdomain.h>
#include <linux/irq.h>
+#include <linux/irq_work.h>
#include <linux/kernel.h>
#include <linux/of_address.h>
#include <linux/of_clk.h>
@@ -467,6 +468,10 @@ struct tegra_pmc {
unsigned long *wake_sw_status_map;
unsigned long *wake_cntrl_level_map;
struct syscore syscore;
+
+ /* Pending wake IRQ processing */
+ struct irq_work wake_work;
+ u32 *wake_status;
};
static struct tegra_pmc *pmc = &(struct tegra_pmc) {
@@ -1904,6 +1909,50 @@ static int tegra_pmc_parse_dt(struct tegra_pmc *pmc, struct device_node *np)
return 0;
}
+/* translate sc7 wake sources back into IRQs to catch edge triggered wakeups */
+static void tegra186_pmc_wake_handler(struct irq_work *work)
+{
+ struct tegra_pmc *pmc = container_of(work, struct tegra_pmc, wake_work);
+ unsigned int i, wake;
+
+ for (i = 0; i < pmc->soc->max_wake_vectors; i++) {
+ unsigned long status = pmc->wake_status[i];
+
+ for_each_set_bit(wake, &status, 32) {
+ irq_hw_number_t hwirq = wake + (i * 32);
+ struct irq_desc *desc;
+ unsigned int irq;
+
+ irq = irq_find_mapping(pmc->domain, hwirq);
+ if (!irq) {
+ dev_warn(pmc->dev,
+ "No IRQ found for WAKE#%lu!\n",
+ hwirq);
+ continue;
+ }
+
+ dev_dbg(pmc->dev,
+ "Resume caused by WAKE#%lu mapped to IRQ#%u\n",
+ hwirq, irq);
+
+ desc = irq_to_desc(irq);
+ if (!desc) {
+ dev_warn(pmc->dev,
+ "No descriptor found for IRQ#%u\n",
+ irq);
+ continue;
+ }
+
+ if (!desc->action || !desc->action->name)
+ continue;
+
+ generic_handle_irq(irq);
+ }
+
+ pmc->wake_status[i] = 0;
+ }
+}
+
static int tegra_pmc_init(struct tegra_pmc *pmc)
{
if (pmc->soc->max_wake_events > 0) {
@@ -1922,6 +1971,18 @@ static int tegra_pmc_init(struct tegra_pmc *pmc)
pmc->wake_cntrl_level_map = bitmap_zalloc(pmc->soc->max_wake_events, GFP_KERNEL);
if (!pmc->wake_cntrl_level_map)
return -ENOMEM;
+
+ pmc->wake_status = kcalloc(pmc->soc->max_wake_vectors, sizeof(u32), GFP_KERNEL);
+ if (!pmc->wake_status)
+ return -ENOMEM;
+
+ /*
+ * Initialize IRQ work for processing wake IRQs. Must use
+ * HARD_IRQ variant to run in hard IRQ context on PREEMPT_RT
+ * because we call generic_handle_irq() which requires hard
+ * IRQ context.
+ */
+ pmc->wake_work = IRQ_WORK_INIT_HARD(tegra186_pmc_wake_handler);
}
if (pmc->soc->init)
@@ -3121,47 +3182,30 @@ static void wke_clear_wake_status(struct tegra_pmc *pmc)
}
}
-/* translate sc7 wake sources back into IRQs to catch edge triggered wakeups */
-static void tegra186_pmc_process_wake_events(struct tegra_pmc *pmc, unsigned int index,
- unsigned long status)
-{
- unsigned int wake;
-
- dev_dbg(pmc->dev, "Wake[%d:%d] status=%#lx\n", (index * 32) + 31, index * 32, status);
-
- for_each_set_bit(wake, &status, 32) {
- irq_hw_number_t hwirq = wake + 32 * index;
- struct irq_desc *desc;
- unsigned int irq;
-
- irq = irq_find_mapping(pmc->domain, hwirq);
-
- desc = irq_to_desc(irq);
- if (!desc || !desc->action || !desc->action->name) {
- dev_dbg(pmc->dev, "Resume caused by WAKE%ld, IRQ %d\n", hwirq, irq);
- continue;
- }
-
- dev_dbg(pmc->dev, "Resume caused by WAKE%ld, %s\n", hwirq, desc->action->name);
- generic_handle_irq(irq);
- }
-}
-
static void tegra186_pmc_wake_syscore_resume(void *data)
{
- u32 status, mask;
unsigned int i;
+ u32 mask;
for (i = 0; i < pmc->soc->max_wake_vectors; i++) {
mask = readl(pmc->wake + WAKE_AOWAKE_TIER2_ROUTING(i));
- status = readl(pmc->wake + WAKE_AOWAKE_STATUS_R(i)) & mask;
-
- tegra186_pmc_process_wake_events(pmc, i, status);
+ pmc->wake_status[i] = readl(pmc->wake + WAKE_AOWAKE_STATUS_R(i)) & mask;
}
+
+ /* Schedule IRQ work to process wake IRQs (if any) */
+ irq_work_queue(&pmc->wake_work);
}
static int tegra186_pmc_wake_syscore_suspend(void *data)
{
+ unsigned int i;
+
+ /* Check if there are unhandled wake IRQs */
+ for (i = 0; i < pmc->soc->max_wake_vectors; i++)
+ if (pmc->wake_status[i])
+ dev_warn(pmc->dev,
+ "Unhandled wake IRQs pending vector[%u]: 0x%x\n",
+ i, pmc->wake_status[i]);
wke_read_sw_wake_status(pmc);
/* flip the wakeup trigger for dual-edge triggered pads
diff --git a/drivers/soc/xilinx/xlnx_event_manager.c b/drivers/soc/xilinx/xlnx_event_manager.c
index a572d15f6161..6fdf4d14b7e7 100644
--- a/drivers/soc/xilinx/xlnx_event_manager.c
+++ b/drivers/soc/xilinx/xlnx_event_manager.c
@@ -77,17 +77,17 @@ struct registered_event_data {
static bool xlnx_is_error_event(const u32 node_id)
{
- u32 pm_family_code, pm_sub_family_code;
+ u32 pm_family_code;
- zynqmp_pm_get_family_info(&pm_family_code, &pm_sub_family_code);
+ zynqmp_pm_get_family_info(&pm_family_code);
- if (pm_sub_family_code == VERSAL_SUB_FAMILY_CODE) {
+ if (pm_family_code == PM_VERSAL_FAMILY_CODE) {
if (node_id == VERSAL_EVENT_ERROR_PMC_ERR1 ||
node_id == VERSAL_EVENT_ERROR_PMC_ERR2 ||
node_id == VERSAL_EVENT_ERROR_PSM_ERR1 ||
node_id == VERSAL_EVENT_ERROR_PSM_ERR2)
return true;
- } else {
+ } else if (pm_family_code == PM_VERSAL_NET_FAMILY_CODE) {
if (node_id == VERSAL_NET_EVENT_ERROR_PMC_ERR1 ||
node_id == VERSAL_NET_EVENT_ERROR_PMC_ERR2 ||
node_id == VERSAL_NET_EVENT_ERROR_PMC_ERR3 ||
diff --git a/drivers/soc/xilinx/zynqmp_power.c b/drivers/soc/xilinx/zynqmp_power.c
index 84ef7c87dfaa..e5d5c4f8cbdb 100644
--- a/drivers/soc/xilinx/zynqmp_power.c
+++ b/drivers/soc/xilinx/zynqmp_power.c
@@ -285,7 +285,7 @@ static int register_event(struct device *dev, const enum pm_api_cb_id cb_type, c
static int zynqmp_pm_probe(struct platform_device *pdev)
{
int ret, irq;
- u32 pm_api_version, pm_family_code, pm_sub_family_code, node_id;
+ u32 pm_api_version, pm_family_code, node_id;
struct mbox_client *client;
ret = zynqmp_pm_get_api_version(&pm_api_version);
@@ -303,34 +303,28 @@ static int zynqmp_pm_probe(struct platform_device *pdev)
* is not available to use) or -ENODEV(Xilinx Event Manager not compiled),
* then use ipi-mailbox or interrupt method.
*/
+ zynqmp_pm_init_suspend_work = devm_kzalloc(&pdev->dev,
+ sizeof(struct zynqmp_pm_work_struct),
+ GFP_KERNEL);
+ if (!zynqmp_pm_init_suspend_work)
+ return -ENOMEM;
+
+ INIT_WORK(&zynqmp_pm_init_suspend_work->callback_work,
+ zynqmp_pm_init_suspend_work_fn);
+
ret = register_event(&pdev->dev, PM_INIT_SUSPEND_CB, 0, 0, false,
suspend_event_callback);
if (!ret) {
- zynqmp_pm_init_suspend_work = devm_kzalloc(&pdev->dev,
- sizeof(struct zynqmp_pm_work_struct),
- GFP_KERNEL);
- if (!zynqmp_pm_init_suspend_work)
- return -ENOMEM;
-
- INIT_WORK(&zynqmp_pm_init_suspend_work->callback_work,
- zynqmp_pm_init_suspend_work_fn);
-
- ret = zynqmp_pm_get_family_info(&pm_family_code, &pm_sub_family_code);
+ ret = zynqmp_pm_get_family_info(&pm_family_code);
if (ret < 0)
return ret;
- if (pm_sub_family_code == VERSALNET_SUB_FAMILY_CODE)
+ if (pm_family_code == PM_VERSAL_NET_FAMILY_CODE)
node_id = PM_DEV_ACPU_0_0;
- else
+ else if (pm_family_code == PM_VERSAL_FAMILY_CODE)
node_id = PM_DEV_ACPU_0;
-
- ret = register_event(&pdev->dev, PM_NOTIFY_CB, node_id, EVENT_SUBSYSTEM_RESTART,
- false, subsystem_restart_event_callback);
- if (ret) {
- dev_err(&pdev->dev, "Failed to Register with Xilinx Event manager %d\n",
- ret);
- return ret;
- }
+ else
+ return -ENODEV;
zynqmp_pm_init_restart_work = devm_kzalloc(&pdev->dev,
sizeof(struct zynqmp_pm_work_struct),
@@ -340,19 +334,18 @@ static int zynqmp_pm_probe(struct platform_device *pdev)
INIT_WORK(&zynqmp_pm_init_restart_work->callback_work,
zynqmp_pm_subsystem_restart_work_fn);
+
+ ret = register_event(&pdev->dev, PM_NOTIFY_CB, node_id, EVENT_SUBSYSTEM_RESTART,
+ false, subsystem_restart_event_callback);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to Register with Xilinx Event manager %d\n",
+ ret);
+ return ret;
+ }
} else if (ret != -EACCES && ret != -ENODEV) {
dev_err(&pdev->dev, "Failed to Register with Xilinx Event manager %d\n", ret);
return ret;
} else if (of_property_present(pdev->dev.of_node, "mboxes")) {
- zynqmp_pm_init_suspend_work =
- devm_kzalloc(&pdev->dev,
- sizeof(struct zynqmp_pm_work_struct),
- GFP_KERNEL);
- if (!zynqmp_pm_init_suspend_work)
- return -ENOMEM;
-
- INIT_WORK(&zynqmp_pm_init_suspend_work->callback_work,
- zynqmp_pm_init_suspend_work_fn);
client = devm_kzalloc(&pdev->dev, sizeof(*client), GFP_KERNEL);
if (!client)
return -ENOMEM;
diff --git a/drivers/spi/spi-sg2044-nor.c b/drivers/spi/spi-sg2044-nor.c
index 37f1cfe10be4..185ab22d0211 100644
--- a/drivers/spi/spi-sg2044-nor.c
+++ b/drivers/spi/spi-sg2044-nor.c
@@ -307,15 +307,15 @@ static ssize_t sg2044_spifmc_tran_cmd(struct sg2044_spifmc *spifmc,
return 0;
}
-static void sg2044_spifmc_trans(struct sg2044_spifmc *spifmc,
- const struct spi_mem_op *op)
+static int sg2044_spifmc_trans(struct sg2044_spifmc *spifmc,
+ const struct spi_mem_op *op)
{
if (op->data.dir == SPI_MEM_DATA_IN)
- sg2044_spifmc_read(spifmc, op);
+ return sg2044_spifmc_read(spifmc, op);
else if (op->data.dir == SPI_MEM_DATA_OUT)
- sg2044_spifmc_write(spifmc, op);
+ return sg2044_spifmc_write(spifmc, op);
else
- sg2044_spifmc_tran_cmd(spifmc, op);
+ return sg2044_spifmc_tran_cmd(spifmc, op);
}
static ssize_t sg2044_spifmc_trans_reg(struct sg2044_spifmc *spifmc,
@@ -385,29 +385,64 @@ static ssize_t sg2044_spifmc_trans_reg(struct sg2044_spifmc *spifmc,
return 0;
}
+static int sg2044_spifmc_set_speed(struct sg2044_spifmc *spifmc,
+ unsigned int speed_hz)
+{
+ unsigned long rate = clk_get_rate(spifmc->clk);
+ u64 div;
+ u32 reg;
+
+ if (!rate)
+ return -EINVAL;
+
+ /* SCK = HCLK / (2 * (SckDiv + 1)); round down the resulting SCK. */
+ div = speed_hz ? DIV_ROUND_UP_ULL(rate, 2ULL * speed_hz) : 1;
+ if (div > SPIFMC_CTRL_SCK_DIV_MASK + 1)
+ return -EINVAL;
+
+ reg = readl(spifmc->io_base + SPIFMC_CTRL);
+ reg &= ~SPIFMC_CTRL_SCK_DIV_MASK;
+ reg |= div - 1;
+ writel(reg, spifmc->io_base + SPIFMC_CTRL);
+
+ return 0;
+}
+
static int sg2044_spifmc_exec_op(struct spi_mem *mem,
const struct spi_mem_op *op)
{
struct sg2044_spifmc *spifmc;
+ int ret;
spifmc = spi_controller_get_devdata(mem->spi->controller);
mutex_lock(&spifmc->lock);
+ ret = sg2044_spifmc_set_speed(spifmc, op->max_freq ?
+ min(op->max_freq, mem->spi->max_speed_hz) :
+ mem->spi->max_speed_hz);
+ if (ret)
+ goto out_unlock;
+
if (op->addr.nbytes == 0)
- sg2044_spifmc_trans_reg(spifmc, op);
+ ret = sg2044_spifmc_trans_reg(spifmc, op);
else
- sg2044_spifmc_trans(spifmc, op);
+ ret = sg2044_spifmc_trans(spifmc, op);
+out_unlock:
mutex_unlock(&spifmc->lock);
- return 0;
+ return ret;
}
static const struct spi_controller_mem_ops sg2044_spifmc_mem_ops = {
.exec_op = sg2044_spifmc_exec_op,
};
+static const struct spi_controller_mem_caps sg2044_spifmc_mem_caps = {
+ .per_op_freq = true,
+};
+
static void sg2044_spifmc_init(struct sg2044_spifmc *spifmc)
{
u32 tran_csr;
@@ -447,6 +482,10 @@ static int sg2044_spifmc_probe(struct platform_device *pdev)
if (IS_ERR(spifmc->clk))
return dev_err_probe(dev, PTR_ERR(spifmc->clk), "Cannot get and enable AHB clock\n");
+ ctrl->max_speed_hz = clk_get_rate(spifmc->clk) / 2;
+ if (!ctrl->max_speed_hz)
+ return dev_err_probe(dev, -EINVAL, "Invalid AHB clock rate\n");
+
spifmc->dev = &pdev->dev;
spifmc->ctrl = ctrl;
@@ -459,6 +498,7 @@ static int sg2044_spifmc_probe(struct platform_device *pdev)
ctrl->bits_per_word_mask = SPI_BPW_MASK(8);
ctrl->auto_runtime_pm = false;
ctrl->mem_ops = &sg2044_spifmc_mem_ops;
+ ctrl->mem_caps = &sg2044_spifmc_mem_caps;
ctrl->mode_bits = SPI_RX_DUAL | SPI_TX_DUAL | SPI_RX_QUAD | SPI_TX_QUAD;
ret = devm_mutex_init(dev, &spifmc->lock);
diff --git a/drivers/staging/greybus/raw.c b/drivers/staging/greybus/raw.c
index 71de6776739c..e668438e1a2c 100644
--- a/drivers/staging/greybus/raw.c
+++ b/drivers/staging/greybus/raw.c
@@ -21,9 +21,8 @@ struct gb_raw {
struct list_head list;
int list_data;
struct mutex list_lock;
- dev_t dev;
struct cdev cdev;
- struct device *device;
+ struct device dev;
};
struct raw_data {
@@ -148,6 +147,15 @@ static int gb_raw_send(struct gb_raw *raw, u32 len, const char __user *data)
return retval;
}
+static void raw_dev_release(struct device *dev)
+{
+ struct gb_raw *raw = container_of(dev, struct gb_raw, dev);
+
+ ida_free(&minors, MINOR(raw->dev.devt));
+
+ kfree(raw);
+}
+
static int gb_raw_probe(struct gb_bundle *bundle,
const struct greybus_bundle_id *id)
{
@@ -164,15 +172,30 @@ static int gb_raw_probe(struct gb_bundle *bundle,
if (cport_desc->protocol_id != GREYBUS_PROTOCOL_RAW)
return -ENODEV;
+ minor = ida_alloc(&minors, GFP_KERNEL);
+ if (minor < 0)
+ return minor;
+
raw = kzalloc(sizeof(*raw), GFP_KERNEL);
- if (!raw)
+ if (!raw) {
+ ida_free(&minors, minor);
return -ENOMEM;
+ }
+
+ device_initialize(&raw->dev);
+ raw->dev.devt = MKDEV(raw_major, minor);
+ raw->dev.class = &raw_class;
+ raw->dev.parent = &bundle->dev;
+ raw->dev.release = raw_dev_release;
+ retval = dev_set_name(&raw->dev, "gb!raw%d", minor);
+ if (retval)
+ goto error_put_device;
connection = gb_connection_create(bundle, le16_to_cpu(cport_desc->id),
gb_raw_request_handler);
if (IS_ERR(connection)) {
retval = PTR_ERR(connection);
- goto error_free;
+ goto error_put_device;
}
INIT_LIST_HEAD(&raw->list);
@@ -181,46 +204,26 @@ static int gb_raw_probe(struct gb_bundle *bundle,
raw->connection = connection;
greybus_set_drvdata(bundle, raw);
- minor = ida_alloc(&minors, GFP_KERNEL);
- if (minor < 0) {
- retval = minor;
- goto error_connection_destroy;
- }
-
- raw->dev = MKDEV(raw_major, minor);
cdev_init(&raw->cdev, &raw_fops);
retval = gb_connection_enable(connection);
if (retval)
- goto error_remove_ida;
+ goto error_connection_destroy;
- retval = cdev_add(&raw->cdev, raw->dev, 1);
+ retval = cdev_device_add(&raw->cdev, &raw->dev);
if (retval)
goto error_connection_disable;
- raw->device = device_create(&raw_class, &connection->bundle->dev,
- raw->dev, raw, "gb!raw%d", minor);
- if (IS_ERR(raw->device)) {
- retval = PTR_ERR(raw->device);
- goto error_del_cdev;
- }
-
return 0;
-error_del_cdev:
- cdev_del(&raw->cdev);
-
error_connection_disable:
gb_connection_disable(connection);
-error_remove_ida:
- ida_free(&minors, minor);
-
error_connection_destroy:
gb_connection_destroy(connection);
-error_free:
- kfree(raw);
+error_put_device:
+ put_device(&raw->dev);
return retval;
}
@@ -231,11 +234,8 @@ static void gb_raw_disconnect(struct gb_bundle *bundle)
struct raw_data *raw_data;
struct raw_data *temp;
- // FIXME - handle removing a connection when the char device node is open.
- device_destroy(&raw_class, raw->dev);
- cdev_del(&raw->cdev);
+ cdev_device_del(&raw->cdev, &raw->dev);
gb_connection_disable(connection);
- ida_free(&minors, MINOR(raw->dev));
gb_connection_destroy(connection);
mutex_lock(&raw->list_lock);
@@ -244,8 +244,7 @@ static void gb_raw_disconnect(struct gb_bundle *bundle)
kfree(raw_data);
}
mutex_unlock(&raw->list_lock);
-
- kfree(raw);
+ put_device(&raw->dev);
}
/*
diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
index 6108a9f3bdd5..4e455d8a20e7 100644
--- a/drivers/target/iscsi/iscsi_target_login.c
+++ b/drivers/target/iscsi/iscsi_target_login.c
@@ -822,7 +822,7 @@ int iscsit_setup_np(
sock_set_reuseaddr(sock->sk);
ip_sock_set_freebind(sock->sk);
- ret = kernel_bind(sock, (struct sockaddr *)&np->np_sockaddr, len);
+ ret = kernel_bind(sock, (struct sockaddr_unsized *)&np->np_sockaddr, len);
if (ret < 0) {
pr_err("kernel_bind() failed: %d\n", ret);
goto fail;
diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
index bf8390789ecf..41bb103221bc 100644
--- a/drivers/tee/optee/ffa_abi.c
+++ b/drivers/tee/optee/ffa_abi.c
@@ -198,7 +198,8 @@ static int to_msg_param_ffa_mem(struct optee_msg_param *mp,
if (shm) {
u64 shm_offs = p->u.memref.shm_offs;
- mp->u.fmem.internal_offs = shm->offset;
+ mp->u.fmem.internal_offs = tee_shm_get_page_offset(shm) &
+ (FFA_PAGE_SIZE - 1);
mp->u.fmem.offs_low = shm_offs;
mp->u.fmem.offs_high = shm_offs >> 32;
@@ -284,14 +285,26 @@ static int optee_ffa_shm_register(struct tee_context *ctx, struct tee_shm *shm,
.nattrs = 1,
};
struct sg_table sgt;
+ size_t ffa_offs;
int rc;
+ if (!num_pages)
+ return -EINVAL;
+
rc = optee_check_mem_type(start, num_pages);
if (rc)
return rc;
- rc = sg_alloc_table_from_pages(&sgt, pages, num_pages, 0,
- num_pages * PAGE_SIZE, GFP_KERNEL);
+ /*
+ * Start the FF-A descriptor at the 4 KiB page containing the shared
+ * buffer, skipping unused leading 4 KiB pages when PAGE_SIZE is
+ * larger. Same approach as optee_fill_pages_list() in the SMC ABI.
+ * This leaves only the offset into that 4 KiB page for internal_offs.
+ */
+ ffa_offs = round_down(tee_shm_get_page_offset(shm), FFA_PAGE_SIZE);
+ rc = sg_alloc_table_from_pages(&sgt, pages, num_pages, ffa_offs,
+ num_pages * PAGE_SIZE - ffa_offs,
+ GFP_KERNEL);
if (rc)
return rc;
args.sg = sgt.sgl;
@@ -458,7 +471,8 @@ static void handle_ffa_rpc_func_cmd_shm_alloc(struct tee_context *ctx,
.attr = OPTEE_MSG_ATTR_TYPE_FMEM_OUTPUT,
.u.fmem.size = tee_shm_get_size(shm),
.u.fmem.global_id = shm->sec_world_id,
- .u.fmem.internal_offs = shm->offset,
+ .u.fmem.internal_offs = tee_shm_get_page_offset(shm) &
+ (FFA_PAGE_SIZE - 1),
};
arg->ret = TEEC_SUCCESS;
@@ -698,6 +712,9 @@ static int optee_ffa_lend_protmem(struct optee *optee, struct tee_shm *protmem,
int rc;
mem_attr = kcalloc(ma_count, sizeof(*mem_attr), GFP_KERNEL);
+ if (!mem_attr)
+ return -ENOMEM;
+
for (n = 0; n < ma_count; n++) {
mem_attr[n].receiver = mem_attrs[n] & U16_MAX;
mem_attr[n].attrs = mem_attrs[n] >> 16;
@@ -1094,14 +1111,6 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
optee_set_dev_group(optee);
- rc = tee_device_register(optee->teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
- rc = tee_device_register(optee->supp_teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
rc = rhashtable_init(&optee->ffa.global_ids, &shm_rhash_params);
if (rc)
goto err_unreg_supp_teedev;
@@ -1130,6 +1139,14 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
if (optee_ffa_protmem_pool_init(optee, sec_caps))
pr_info("Protected memory service not available\n");
+ rc = tee_device_register(optee->teedev);
+ if (rc)
+ goto err_unregister_devices;
+
+ rc = tee_device_register(optee->supp_teedev);
+ if (rc)
+ goto err_unregister_devices;
+
rc = optee_enumerate_devices(PTA_CMD_GET_DEVICES);
if (rc)
goto err_unregister_devices;
diff --git a/drivers/tee/optee/optee_msg.h b/drivers/tee/optee/optee_msg.h
index 838e1d4a22f0..6c3043f8da33 100644
--- a/drivers/tee/optee/optee_msg.h
+++ b/drivers/tee/optee/optee_msg.h
@@ -103,9 +103,9 @@
/**
* struct optee_msg_param_tmem - temporary memory reference parameter
- * @buf_ptr: Address of the buffer
- * @size: Size of the buffer
- * @shm_ref: Temporary shared memory reference, pointer to a struct tee_shm
+ * @buf_ptr: address of the buffer
+ * @size: size of the buffer
+ * @shm_ref: temporary shared memory reference, pointer to a struct tee_shm
*
* Secure and normal world communicates pointers as physical address
* instead of the virtual address. This is because secure and normal world
@@ -122,9 +122,9 @@ struct optee_msg_param_tmem {
/**
* struct optee_msg_param_rmem - registered memory reference parameter
- * @offs: Offset into shared memory reference
- * @size: Size of the buffer
- * @shm_ref: Shared memory reference, pointer to a struct tee_shm
+ * @offs: offset into shared memory reference
+ * @size: size of the buffer
+ * @shm_ref: shared memory reference, pointer to a struct tee_shm
*/
struct optee_msg_param_rmem {
u64 offs;
@@ -134,12 +134,12 @@ struct optee_msg_param_rmem {
/**
* struct optee_msg_param_fmem - FF-A memory reference parameter
- * @offs_lower: Lower bits of offset into shared memory reference
- * @offs_upper: Upper bits of offset into shared memory reference
- * @internal_offs: Internal offset into the first page of shared memory
- * reference
- * @size: Size of the buffer
- * @global_id: Global identifier of the shared memory
+ * @offs_low: lower bits of offset into shared memory reference
+ * @offs_high: higher bits of offset into shared memory reference
+ * @internal_offs: offset into the first 4 KiB page of the FF-A shared
+ * memory region
+ * @size: size of the buffer
+ * @global_id: global identifier of the shared memory
*/
struct optee_msg_param_fmem {
u32 offs_low;
@@ -151,6 +151,9 @@ struct optee_msg_param_fmem {
/**
* struct optee_msg_param_value - opaque value parameter
+ * @a: first opaque value
+ * @b: second opaque value
+ * @c: third opaque value
*
* Value parameters are passed unchecked between normal and secure world.
*/
@@ -168,6 +171,7 @@ struct optee_msg_param_value {
* @fmem: parameter by FF-A registered memory reference
* @value: parameter by opaque value
* @octets: parameter by octet string
+ * @u: union holding OP-TEE msg parameter
*
* @attr & OPTEE_MSG_ATTR_TYPE_MASK indicates if tmem, rmem or value is used in
* the union. OPTEE_MSG_ATTR_TYPE_VALUE_* indicates value or octets,
@@ -189,16 +193,18 @@ struct optee_msg_param {
/**
* struct optee_msg_arg - call argument
- * @cmd: Command, one of OPTEE_MSG_CMD_* or OPTEE_MSG_RPC_CMD_*
- * @func: Trusted Application function, specific to the Trusted Application,
- * used if cmd == OPTEE_MSG_CMD_INVOKE_COMMAND
- * @session: In parameter for all OPTEE_MSG_CMD_* except
- * OPTEE_MSG_CMD_OPEN_SESSION where it's an output parameter instead
- * @cancel_id: Cancellation id, a unique value to identify this request
- * @ret: return value
- * @ret_origin: origin of the return value
- * @num_params: number of parameters supplied to the OS Command
- * @params: the parameters supplied to the OS Command
+ * @cmd: command, one of OPTEE_MSG_CMD_* or OPTEE_MSG_RPC_CMD_*
+ * @func: Trusted Application function, specific to the Trusted
+ * Application, used if cmd == OPTEE_MSG_CMD_INVOKE_COMMAND
+ * @session: in parameter for all OPTEE_MSG_CMD_* except
+ * OPTEE_MSG_CMD_OPEN_SESSION where it's an output parameter
+ * instead
+ * @cancel_id: cancellation id, a unique value to identify this request
+ * @pad: padding for alignment
+ * @ret: return value
+ * @ret_origin: origin of the return value
+ * @num_params: number of parameters supplied to the OS Command
+ * @params: the parameters supplied to the OS Command
*
* All normal calls to Trusted OS uses this struct. If cmd requires further
* information than what these fields hold it can be passed as a parameter
diff --git a/drivers/tee/optee/optee_private.h b/drivers/tee/optee/optee_private.h
index db9ea673fbca..0c546f8fd0a3 100644
--- a/drivers/tee/optee/optee_private.h
+++ b/drivers/tee/optee/optee_private.h
@@ -47,11 +47,11 @@ typedef void (optee_invoke_fn)(unsigned long, unsigned long, unsigned long,
unsigned long, unsigned long,
struct arm_smccc_res *);
-/*
+/**
* struct optee_call_waiter - TEE entry may need to wait for a free TEE thread
- * @list_node Reference in waiters list
- * @c Waiting completion reference
- * @sys_thread True if waiter belongs to a system thread
+ * @list_node: reference in waiters list
+ * @c: waiting completion reference
+ * @sys_thread: true if waiter belongs to a system thread
*/
struct optee_call_waiter {
struct list_head list_node;
@@ -59,13 +59,13 @@ struct optee_call_waiter {
bool sys_thread;
};
-/*
+/**
* struct optee_call_queue - OP-TEE call queue management
- * @mutex Serializes access to this struct
- * @waiters List of threads waiting to enter OP-TEE
- * @total_thread_count Overall number of thread context in OP-TEE or 0
- * @free_thread_count Number of threads context free in OP-TEE
- * @sys_thread_req_count Number of registered system thread sessions
+ * @mutex: serializes access to this struct
+ * @waiters: list of threads waiting to enter OP-TEE
+ * @total_thread_count: overall number of thread context in OP-TEE or 0
+ * @free_thread_count: number of threads context free in OP-TEE
+ * @sys_thread_req_count: number of registered system thread sessions
*/
struct optee_call_queue {
/* Serializes access to this struct */
@@ -96,17 +96,17 @@ struct optee_shm_arg_cache {
/**
* struct optee_supp - supplicant synchronization struct
- * @ctx the context of current connected supplicant.
- * if !NULL the supplicant device is available for use,
- * else busy
- * @mutex: held while accessing content of this struct
- * @req_id: current request id if supplicant is doing synchronous
- * communication, else -1
- * @reqs: queued request not yet retrieved by supplicant
- * @idr: IDR holding all requests currently being processed
- * by supplicant
- * @reqs_c: completion used by supplicant when waiting for a
- * request to be queued.
+ * @mutex: held while accessing content of this struct
+ * @ctx: the context of current connected supplicant.
+ * if !NULL the supplicant device is available for use,
+ * else busy
+ * @req_id: current request id if supplicant is doing synchronous
+ * communication, else -1
+ * @reqs: queued request not yet retrieved by supplicant
+ * @idr: IDR holding all requests currently being processed
+ * by supplicant
+ * @reqs_c: completion used by supplicant when waiting for a
+ * request to be queued.
*/
struct optee_supp {
/* Serializes access to this struct */
@@ -119,25 +119,25 @@ struct optee_supp {
struct completion reqs_c;
};
-/*
+/**
* struct optee_pcpu - per cpu notif private struct passed to work functions
- * @optee optee device reference
+ * @optee: optee device reference
*/
struct optee_pcpu {
struct optee *optee;
};
-/*
+/**
* struct optee_smc - optee smc communication struct
- * @invoke_fn handler function to invoke secure monitor
- * @memremaped_shm virtual address of memory in shared memory pool
+ * @invoke_fn: handler function to invoke secure monitor
+ * @memremaped_shm: virtual address of memory in shared memory pool
* @sec_caps: secure world capabilities defined by
* OPTEE_SMC_SEC_CAP_* in optee_smc.h
- * @notif_irq interrupt used as async notification by OP-TEE or 0
- * @optee_pcpu per_cpu optee instance for per cpu work or NULL
- * @notif_pcpu_wq workqueue for per cpu asynchronous notification or NULL
- * @notif_pcpu_work work for per cpu asynchronous notification
- * @notif_cpuhp_state CPU hotplug state assigned for pcpu interrupt management
+ * @notif_irq: interrupt used as async notification by OP-TEE or 0
+ * @optee_pcpu: per_cpu optee instance for per cpu work or NULL
+ * @notif_pcpu_wq: workqueue for per cpu asynchronous notification or NULL
+ * @notif_pcpu_work: work for per cpu asynchronous notification
+ * @notif_cpuhp_state: CPU hotplug state assigned for pcpu interrupt management
*/
struct optee_smc {
optee_invoke_fn *invoke_fn;
@@ -151,13 +151,15 @@ struct optee_smc {
};
/**
- * struct optee_ffa_data - FFA communication struct
- * @ffa_dev FFA device, contains the destination id, the id of
+ * struct optee_ffa - FFA communication struct
+ * @ffa_dev: FFA device, contains the destination id, the id of
* OP-TEE in secure world
- * @bottom_half_value Notification ID used for bottom half signalling or
+ * @bottom_half_value: notification ID used for bottom half signalling or
* U32_MAX if unused
- * @mutex Serializes access to @global_ids
- * @global_ids FF-A shared memory global handle translation
+ * @mutex: serializes access to @global_ids
+ * @global_ids: FF-A shared memory global handle translation
+ * @notif_wq: workqueue for FF-A asynchronous notification
+ * @notif_work: work for FF-A asynchronous notification
*/
struct optee_ffa {
struct ffa_device *ffa_dev;
@@ -204,26 +206,32 @@ struct optee_ops {
/**
* struct optee - main service struct
- * @supp_teedev: supplicant device
- * @teedev: client device
- * @ops: internal callbacks for different ways to reach secure
- * world
- * @ctx: driver internal TEE context
- * @smc: specific to SMC ABI
- * @ffa: specific to FF-A ABI
- * @call_queue: queue of threads waiting to call @invoke_fn
- * @notif: notification synchronization struct
- * @supp: supplicant synchronization struct for RPC to supplicant
- * @pool: shared memory pool
- * @mutex: mutex protecting @rpmb_dev
- * @rpmb_dev: current RPMB device or NULL
- * @rpmb_scan_bus_done flag if device registation of RPMB dependent devices
- * was already done
- * @rpmb_scan_bus_work workq to for an RPMB device and to scan optee bus
- * and register RPMB dependent optee drivers
- * @rpc_param_count: If > 0 number of RPC parameters to make room for
- * @scan_bus_done flag if device registation was already done.
- * @scan_bus_work workq to scan optee bus and register optee drivers
+ * @supp_teedev: supplicant device
+ * @teedev: client device
+ * @ops: internal callbacks for different ways to reach
+ * secure world
+ * @ctx: driver internal TEE context
+ * @smc: specific to SMC ABI
+ * @ffa: specific to FF-A ABI
+ * @shm_arg_cache: shared memory cache argument
+ * @call_queue: queue of threads waiting to call @invoke_fn
+ * @notif: notification synchronization struct
+ * @supp: supplicant synchronization struct for RPC to
+ * supplicant
+ * @pool: shared memory pool
+ * @rpmb_dev_mutex: mutex protecting @rpmb_dev
+ * @rpmb_dev: current RPMB device or NULL
+ * @rpmb_intf: RPMB notifier block
+ * @rpc_param_count: if > 0 number of RPC parameters to make room for
+ * @scan_bus_done: flag if device registation was already done
+ * @rpmb_scan_bus_done: flag if device registation of RPMB dependent
+ * devices was already done
+ * @in_kernel_rpmb_routing: flag if OP-TEE supports in-kernel RPMB routing
+ * @scan_bus_work: workq to scan optee bus and register optee
+ * drivers
+ * @rpmb_scan_bus_work: workq to for an RPMB device and to scan optee
+ * bus and register RPMB dependent optee drivers
+ * @revision: OP-TEE OS revision
*/
struct optee {
struct tee_device *supp_teedev;
diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
index 0be663fcd52b..bf51019e063b 100644
--- a/drivers/tee/optee/smc_abi.c
+++ b/drivers/tee/optee/smc_abi.c
@@ -1840,14 +1840,6 @@ static int optee_probe(struct platform_device *pdev)
optee_set_dev_group(optee);
- rc = tee_device_register(optee->teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
- rc = tee_device_register(optee->supp_teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
optee_cq_init(&optee->call_queue, thread_count);
optee_supp_init(&optee->supp);
optee->smc.memremaped_shm = memremaped_shm;
@@ -1907,6 +1899,14 @@ static int optee_probe(struct platform_device *pdev)
if (optee->smc.sec_caps & OPTEE_SMC_SEC_CAP_DYNAMIC_SHM)
pr_info("dynamic shared memory is enabled\n");
+ rc = tee_device_register(optee->teedev);
+ if (rc)
+ goto err_disable_shm_cache;
+
+ rc = tee_device_register(optee->supp_teedev);
+ if (rc)
+ goto err_disable_shm_cache;
+
rc = optee_enumerate_devices(PTA_CMD_GET_DEVICES);
if (rc)
goto err_disable_shm_cache;
@@ -1933,7 +1933,6 @@ static int optee_probe(struct platform_device *pdev)
optee_shm_arg_cache_uninit(optee);
optee_supp_uninit(&optee->supp);
mutex_destroy(&optee->call_queue.mutex);
-err_unreg_supp_teedev:
tee_device_unregister(optee->supp_teedev);
err_unreg_teedev:
tee_device_unregister(optee->teedev);
diff --git a/drivers/tee/tee_shm.c b/drivers/tee/tee_shm.c
index 6d2f2c4e3471..d047438a485e 100644
--- a/drivers/tee/tee_shm.c
+++ b/drivers/tee/tee_shm.c
@@ -343,6 +343,10 @@ int tee_dyn_shm_alloc_helper(struct tee_shm *shm, size_t size, size_t align,
unsigned int i;
int rc = 0;
+ /* get_order(0) is undefined and exceeds MAX_PAGE_ORDER. */
+ if (!nr_pages)
+ return -EINVAL;
+
/*
* Ignore alignment since this is already going to be page aligned
* and there's no need for any larger alignment.
diff --git a/drivers/thermal/thermal_hwmon.c b/drivers/thermal/thermal_hwmon.c
index 342a5eb85093..a136501bca26 100644
--- a/drivers/thermal/thermal_hwmon.c
+++ b/drivers/thermal/thermal_hwmon.c
@@ -95,34 +95,12 @@ thermal_hwmon_lookup_by_type(const struct thermal_zone_device *tz)
struct thermal_hwmon_device *hwmon;
char type[THERMAL_NAME_LENGTH];
- mutex_lock(&thermal_hwmon_list_lock);
list_for_each_entry(hwmon, &thermal_hwmon_list, node) {
strscpy(type, tz->type);
strreplace(type, '-', '_');
- if (!strcmp(hwmon->type, type)) {
- mutex_unlock(&thermal_hwmon_list_lock);
+ if (!strcmp(hwmon->type, type))
return hwmon;
- }
}
- mutex_unlock(&thermal_hwmon_list_lock);
-
- return NULL;
-}
-
-/* Find the temperature input matching a given thermal zone */
-static struct thermal_hwmon_temp *
-thermal_hwmon_lookup_temp(const struct thermal_hwmon_device *hwmon,
- const struct thermal_zone_device *tz)
-{
- struct thermal_hwmon_temp *temp;
-
- mutex_lock(&thermal_hwmon_list_lock);
- list_for_each_entry(temp, &hwmon->tz_list, hwmon_node)
- if (temp->tz == tz) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return temp;
- }
- mutex_unlock(&thermal_hwmon_list_lock);
return NULL;
}
@@ -138,7 +116,9 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
struct thermal_hwmon_device *hwmon;
struct thermal_hwmon_temp *temp;
int new_hwmon_device = 1;
- int result;
+ int result = 0;
+
+ mutex_lock(&thermal_hwmon_list_lock);
hwmon = thermal_hwmon_lookup_by_type(tz);
if (hwmon) {
@@ -147,8 +127,10 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
}
hwmon = kzalloc(sizeof(*hwmon), GFP_KERNEL);
- if (!hwmon)
- return -ENOMEM;
+ if (!hwmon) {
+ result = -ENOMEM;
+ goto unlock;
+ }
INIT_LIST_HEAD(&hwmon->tz_list);
strscpy(hwmon->type, tz->type, THERMAL_NAME_LENGTH);
@@ -196,24 +178,24 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
temp->temp_crit_present = true;
}
- mutex_lock(&thermal_hwmon_list_lock);
if (new_hwmon_device)
list_add_tail(&hwmon->node, &thermal_hwmon_list);
list_add_tail(&temp->hwmon_node, &hwmon->tz_list);
- mutex_unlock(&thermal_hwmon_list_lock);
- return 0;
+ goto unlock;
- unregister_input:
+unregister_input:
device_remove_file(hwmon->device, &temp->temp_input.attr);
- free_temp_mem:
+free_temp_mem:
kfree(temp);
- unregister_name:
+unregister_name:
if (new_hwmon_device)
hwmon_device_unregister(hwmon->device);
- free_mem:
+free_mem:
if (new_hwmon_device)
kfree(hwmon);
+unlock:
+ mutex_unlock(&thermal_hwmon_list_lock);
return result;
}
@@ -221,8 +203,11 @@ EXPORT_SYMBOL_GPL(thermal_add_hwmon_sysfs);
void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz)
{
+ struct thermal_hwmon_temp *temp, *entry;
struct thermal_hwmon_device *hwmon;
- struct thermal_hwmon_temp *temp;
+ bool unregister;
+
+ guard(mutex)(&thermal_hwmon_list_lock);
hwmon = thermal_hwmon_lookup_by_type(tz);
if (unlikely(!hwmon)) {
@@ -231,29 +216,25 @@ void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz)
return;
}
- temp = thermal_hwmon_lookup_temp(hwmon, tz);
- if (unlikely(!temp)) {
- /* Should never happen... */
- dev_dbg(&tz->device, "temperature input lookup failed!\n");
- return;
- }
+ unregister = hwmon->device->parent == &tz->device;
- device_remove_file(hwmon->device, &temp->temp_input.attr);
- if (temp->temp_crit_present)
- device_remove_file(hwmon->device, &temp->temp_crit.attr);
+ list_for_each_entry_safe_reverse(temp, entry, &hwmon->tz_list, hwmon_node) {
+ if (!unregister && temp->tz != tz)
+ continue;
- mutex_lock(&thermal_hwmon_list_lock);
- list_del(&temp->hwmon_node);
- kfree(temp);
- if (!list_empty(&hwmon->tz_list)) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return;
+ device_remove_file(hwmon->device, &temp->temp_input.attr);
+ if (temp->temp_crit_present)
+ device_remove_file(hwmon->device, &temp->temp_crit.attr);
+
+ list_del(&temp->hwmon_node);
+ kfree(temp);
}
- list_del(&hwmon->node);
- mutex_unlock(&thermal_hwmon_list_lock);
- hwmon_device_unregister(hwmon->device);
- kfree(hwmon);
+ if (unregister) {
+ list_del(&hwmon->node);
+ hwmon_device_unregister(hwmon->device);
+ kfree(hwmon);
+ }
}
EXPORT_SYMBOL_GPL(thermal_remove_hwmon_sysfs);
diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
index 32cdfc59a9ad..593f15213b72 100644
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -2810,8 +2810,10 @@ int uart_register_driver(struct uart_driver *drv)
for (i = 0; i < drv->nr; i++)
tty_port_destroy(&drv->state[i].port);
tty_driver_kref_put(normal);
+ drv->tty_driver = NULL;
out_kfree:
kfree(drv->state);
+ drv->state = NULL;
out:
return retval;
}
@@ -3089,7 +3091,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
struct uart_state *state;
struct tty_port *port;
struct device *tty_dev;
- int num_groups;
if (uport->line >= drv->nr)
return -EINVAL;
@@ -3101,6 +3102,22 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
if (state->uart_port)
return -EINVAL;
+ uport->name = kasprintf(GFP_KERNEL, "%s%u", drv->dev_name,
+ drv->tty_driver->name_base + uport->line);
+ if (!uport->name)
+ return -ENOMEM;
+
+ /*
+ * uart_configure_port() may set uport->attr_group and register the
+ * console. Allocate room for both groups and a NULL terminator first.
+ */
+ uport->tty_groups = kcalloc(3, sizeof(*uport->tty_groups), GFP_KERNEL);
+ if (!uport->tty_groups) {
+ kfree(uport->name);
+ return -ENOMEM;
+ }
+ uport->tty_groups[0] = &tty_dev_attr_group;
+
/* Link the port to the driver state table and vice versa */
atomic_set(&state->refcount, 1);
init_waitqueue_head(&state->remove_wait);
@@ -3117,10 +3134,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
state->pm_state = UART_PM_STATE_UNDEFINED;
uart_port_set_cons(uport, drv->cons);
uport->minor = drv->tty_driver->minor_start + uport->line;
- uport->name = kasprintf(GFP_KERNEL, "%s%u", drv->dev_name,
- drv->tty_driver->name_base + uport->line);
- if (!uport->name)
- return -ENOMEM;
if (uport->cons && uport->dev)
of_console_check(uport->dev->of_node, uport->cons->name, uport->line);
@@ -3135,16 +3148,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u
port->console = uart_console(uport);
- num_groups = 2;
- if (uport->attr_group)
- num_groups++;
-
- uport->tty_groups = kcalloc(num_groups, sizeof(*uport->tty_groups),
- GFP_KERNEL);
- if (!uport->tty_groups)
- return -ENOMEM;
-
- uport->tty_groups[0] = &tty_dev_attr_group;
if (uport->attr_group)
uport->tty_groups[1] = uport->attr_group;
diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c
index dd7231d9cab9..07e6bf5f8c6f 100644
--- a/drivers/tty/tty_io.c
+++ b/drivers/tty/tty_io.c
@@ -3327,7 +3327,7 @@ EXPORT_SYMBOL_GPL(tty_register_device_attr);
void tty_unregister_device(struct tty_driver *driver, unsigned index)
{
device_destroy(&tty_class, MKDEV(driver->major, driver->minor_start) + index);
- if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) {
+ if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC) && driver->cdevs[index]) {
cdev_del(driver->cdevs[index]);
driver->cdevs[index] = NULL;
}
diff --git a/drivers/ufs/core/ufshcd-priv.h b/drivers/ufs/core/ufshcd-priv.h
index 1f0d38aa37f9..f3eeaf45cfdb 100644
--- a/drivers/ufs/core/ufshcd-priv.h
+++ b/drivers/ufs/core/ufshcd-priv.h
@@ -80,11 +80,17 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag);
void ufshcd_release_scsi_cmd(struct ufs_hba *hba,
struct ufshcd_lrb *lrbp);
-#define SD_ASCII_STD true
-#define SD_RAW false
-int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
- u8 **buf, bool ascii);
+/**
+ * enum ufs_descr_fmt - UFS string descriptor format
+ * @SD_RAW: Raw UTF-16 format
+ * @SD_ASCII_STD: Convert to null-terminated ASCII string
+ */
+enum ufs_descr_fmt {
+ SD_RAW = 0,
+ SD_ASCII_STD = 1,
+};
+int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, u8 **buf, enum ufs_descr_fmt fmt);
int ufshcd_send_uic_cmd(struct ufs_hba *hba, struct uic_command *uic_cmd);
int ufshcd_send_bsg_uic_cmd(struct ufs_hba *hba, struct uic_command *uic_cmd);
diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index 87578e8824d2..9d32e658ad60 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -3774,20 +3774,18 @@ static inline char ufshcd_remove_non_printable(u8 ch)
* @desc_index: descriptor index
* @buf: pointer to buffer where descriptor would be read,
* the caller should free the memory.
- * @ascii: if true convert from unicode to ascii characters
- * null terminated string.
+ * @fmt: if %SD_ASCII_STD, convert from UTF-16 to ASCII
*
* Return:
* * string size on success.
* * -ENOMEM: on allocation failure
* * -EINVAL: on a wrong parameter
*/
-int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
- u8 **buf, bool ascii)
+int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, u8 **buf, enum ufs_descr_fmt fmt)
{
struct uc_string_id *uc_str;
u8 *str;
- int ret;
+ int ret, uc_len;
if (!buf)
return -EINVAL;
@@ -3812,11 +3810,19 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
goto out;
}
- if (ascii) {
+ uc_len = uc_str->len - QUERY_DESC_HDR_SIZE;
+ if (uc_len % sizeof(*uc_str->uc)) {
+ dev_err(hba->dev, "String Desc has an odd UTF-16 payload length\n");
+ str = NULL;
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (fmt == SD_ASCII_STD) {
ssize_t ascii_len;
int i;
- /* remove header and divide by 2 to move from UTF16 to UTF8 */
- ascii_len = (uc_str->len - QUERY_DESC_HDR_SIZE) / 2 + 1;
+ /* Allow up to three UTF-8 bytes per UTF-16 code unit plus a NUL. */
+ ascii_len = uc_len / sizeof(*uc_str->uc) * 3 + 1;
str = kzalloc(ascii_len, GFP_KERNEL);
if (!str) {
ret = -ENOMEM;
@@ -3828,7 +3834,7 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
* we need to convert to utf-8 so it can be displayed
*/
ret = utf16s_to_utf8s(uc_str->uc,
- uc_str->len - QUERY_DESC_HDR_SIZE,
+ uc_len / sizeof(*uc_str->uc),
UTF16_BIG_ENDIAN, str, ascii_len - 1);
/* replace non-printable or non-ASCII characters with spaces */
@@ -3838,11 +3844,17 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
str[ret++] = '\0';
} else {
- str = kmemdup(uc_str->uc, uc_str->len, GFP_KERNEL);
+ /*
+ * Keep the bLength-sized raw output for the RPMB device ID ABI.
+ * The two bytes beyond the UTF-16 payload are explicitly zeroed
+ * instead of being read past the descriptor buffer.
+ */
+ str = kzalloc(uc_str->len, GFP_KERNEL);
if (!str) {
ret = -ENOMEM;
goto out;
}
+ memcpy(str, uc_str->uc, uc_len);
ret = uc_str->len;
}
out:
@@ -9637,7 +9649,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
* we are functional while we are here, skip host resume in error
* handling context.
*/
- hba->host->eh_noresume = 1;
+ WRITE_ONCE(hba->host->eh_noresume, 1);
/*
* Current function would be generally called from the power management
@@ -9659,7 +9671,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
}
scsi_device_put(sdp);
- hba->host->eh_noresume = 0;
+ WRITE_ONCE(hba->host->eh_noresume, 0);
return ret;
}
diff --git a/drivers/usb/gadget/function/f_uac1.c b/drivers/usb/gadget/function/f_uac1.c
index 9da9fb4e1239..92d23a294f6d 100644
--- a/drivers/usb/gadget/function/f_uac1.c
+++ b/drivers/usb/gadget/function/f_uac1.c
@@ -1594,7 +1594,8 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
const char *page, size_t len) \
{ \
struct f_uac1_opts *opts = to_f_uac1_opts(item); \
- char *split_page = NULL; \
+ char *buf = NULL; \
+ char *split_page; \
int ret = -EINVAL; \
char *token; \
u32 num; \
@@ -1608,18 +1609,22 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
\
i = 0; \
memset(opts->name##s, 0x00, sizeof(opts->name##s)); \
- split_page = kstrdup(page, GFP_KERNEL); \
+ buf = kstrdup(page, GFP_KERNEL); \
+ split_page = buf; \
while ((token = strsep(&split_page, ",")) != NULL) { \
ret = kstrtou32(token, 0, &num); \
if (ret) \
goto end; \
- \
+ if (i >= UAC_MAX_RATES) { \
+ ret = -EINVAL; \
+ goto end; \
+ } \
opts->name##s[i++] = num; \
ret = len; \
}; \
\
end: \
- kfree(split_page); \
+ kfree(buf); \
mutex_unlock(&opts->lock); \
return ret; \
} \
diff --git a/drivers/usb/gadget/function/f_uac2.c b/drivers/usb/gadget/function/f_uac2.c
index dd252ff2fb4e..cec51a47c478 100644
--- a/drivers/usb/gadget/function/f_uac2.c
+++ b/drivers/usb/gadget/function/f_uac2.c
@@ -2012,7 +2012,8 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \
const char *page, size_t len) \
{ \
struct f_uac2_opts *opts = to_f_uac2_opts(item); \
- char *split_page = NULL; \
+ char *buf = NULL; \
+ char *split_page; \
int ret = -EINVAL; \
char *token; \
u32 num; \
@@ -2026,18 +2027,22 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \
\
i = 0; \
memset(opts->name##s, 0x00, sizeof(opts->name##s)); \
- split_page = kstrdup(page, GFP_KERNEL); \
+ buf = kstrdup(page, GFP_KERNEL); \
+ split_page = buf; \
while ((token = strsep(&split_page, ",")) != NULL) { \
ret = kstrtou32(token, 0, &num); \
if (ret) \
goto end; \
- \
+ if (i >= UAC_MAX_RATES) { \
+ ret = -EINVAL; \
+ goto end; \
+ } \
opts->name##s[i++] = num; \
ret = len; \
}; \
\
end: \
- kfree(split_page); \
+ kfree(buf); \
mutex_unlock(&opts->lock); \
return ret; \
} \
diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
index 2cedf7e2dbc4..42f2f44b383c 100644
--- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
@@ -11,6 +11,8 @@
#define MLX5V_ETH_HARD_MTU (ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN)
+extern int mlx5_vdpa_max_iotlb_entries;
+
struct mlx5_vdpa_direct_mr {
u64 start;
u64 end;
diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
index 5ec01d672481..d46125309df9 100644
--- a/drivers/vdpa/mlx5/core/mr.c
+++ b/drivers/vdpa/mlx5/core/mr.c
@@ -778,6 +778,9 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev,
{
int err;
+ if (mlx5_vdpa_max_iotlb_entries < 2)
+ return -EINVAL;
+
if (iotlb)
err = create_user_mr(mvdev, mr, iotlb);
else
@@ -786,7 +789,7 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev,
if (err)
return err;
- mr->iotlb = vhost_iotlb_alloc(0, 0);
+ mr->iotlb = vhost_iotlb_alloc(mlx5_vdpa_max_iotlb_entries, 0);
if (!mr->iotlb) {
err = -ENOMEM;
goto err_mr;
diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c
index aeae31d0cefa..28a4d7a35bf4 100644
--- a/drivers/vdpa/mlx5/core/resources.c
+++ b/drivers/vdpa/mlx5/core/resources.c
@@ -3,8 +3,14 @@
#include <linux/iova.h>
#include <linux/mlx5/driver.h>
+#include <linux/moduleparam.h>
#include "mlx5_vdpa.h"
+int mlx5_vdpa_max_iotlb_entries = 2048;
+module_param_named(max_iotlb_entries, mlx5_vdpa_max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
static int alloc_pd(struct mlx5_vdpa_dev *dev, u32 *pdn, u16 uid)
{
struct mlx5_core_dev *mdev = dev->mdev;
@@ -229,7 +235,10 @@ int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey)
static int init_ctrl_vq(struct mlx5_vdpa_dev *mvdev)
{
- mvdev->cvq.iotlb = vhost_iotlb_alloc(0, 0);
+ if (mlx5_vdpa_max_iotlb_entries < 2)
+ return -EINVAL;
+
+ mvdev->cvq.iotlb = vhost_iotlb_alloc(mlx5_vdpa_max_iotlb_entries, 0);
if (!mvdev->cvq.iotlb)
return -ENOMEM;
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index 445c45080d45..9df300c46745 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -34,7 +34,7 @@ MODULE_PARM_DESC(batch_mapping, "Batched mapping 1 -Enable; 0 - Disable");
static int max_iotlb_entries = 2048;
module_param(max_iotlb_entries, int, 0444);
MODULE_PARM_DESC(max_iotlb_entries,
- "Maximum number of iotlb entries for each address space. 0 means unlimited. (default: 2048)");
+ "Maximum number of iotlb entries for each address space. (default: 2048)");
static bool use_va = true;
module_param(use_va, bool, 0444);
@@ -201,6 +201,8 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr,
if (!dev_attr->alloc_size)
return ERR_PTR(-EINVAL);
+ if (max_iotlb_entries < 2)
+ return ERR_PTR(-EINVAL);
if (config->mask & BIT_ULL(VDPA_ATTR_DEV_FEATURES)) {
if (config->device_features &
@@ -264,10 +266,14 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr,
if (!vdpasim->iommu_pt)
goto err_iommu;
- for (i = 0; i < vdpasim->dev_attr.nas; i++) {
+ for (i = 0; i < vdpasim->dev_attr.nas; i++)
vhost_iotlb_init(&vdpasim->iommu[i], max_iotlb_entries, 0);
- vhost_iotlb_add_range(&vdpasim->iommu[i], 0, ULONG_MAX, 0,
- VHOST_MAP_RW);
+
+ for (i = 0; i < vdpasim->dev_attr.nas; i++) {
+ ret = vhost_iotlb_add_range(&vdpasim->iommu[i], 0, ULONG_MAX,
+ 0, VHOST_MAP_RW);
+ if (ret)
+ goto err_iommu;
vdpasim->iommu_pt[i] = true;
}
diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c
index ec743bed361c..e8b837df328f 100644
--- a/drivers/vdpa/vdpa_user/iova_domain.c
+++ b/drivers/vdpa/vdpa_user/iova_domain.c
@@ -12,11 +12,17 @@
#include <linux/file.h>
#include <linux/anon_inodes.h>
#include <linux/highmem.h>
+#include <linux/moduleparam.h>
#include <linux/vmalloc.h>
#include <linux/vdpa.h>
#include "iova_domain.h"
+static int max_iotlb_entries = 2048;
+module_param(max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
static int vduse_iotlb_add_range(struct vduse_iova_domain *domain,
u64 start, u64 last,
u64 addr, unsigned int perm,
@@ -622,11 +628,14 @@ vduse_domain_create(unsigned long iova_limit, size_t bounce_size)
if (iova_limit <= bounce_size)
return NULL;
+ if (max_iotlb_entries <= 0)
+ return NULL;
+
domain = kzalloc(sizeof(*domain), GFP_KERNEL);
if (!domain)
return NULL;
- domain->iotlb = vhost_iotlb_alloc(0, 0);
+ domain->iotlb = vhost_iotlb_alloc(max_iotlb_entries, 0);
if (!domain->iotlb)
goto err_iotlb;
diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 1cc826742ac9..f9547ede71be 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -1837,7 +1837,9 @@ static bool vduse_validate_config(struct vduse_dev_config *config)
sizeof(config->reserved)))
return false;
- if (config->vq_align > PAGE_SIZE)
+ if (config->vq_align < VRING_USED_ALIGN_SIZE ||
+ !is_power_of_2(config->vq_align) ||
+ config->vq_align > PAGE_SIZE)
return false;
if (config->config_size > PAGE_SIZE)
diff --git a/drivers/vhost/iotlb.c b/drivers/vhost/iotlb.c
index ea61330a3431..a6228dd24931 100644
--- a/drivers/vhost/iotlb.c
+++ b/drivers/vhost/iotlb.c
@@ -20,6 +20,14 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map,
rb, __u64, __subtree_last,
START, LAST, static inline, vhost_iotlb_itree);
+static void vhost_iotlb_map_unlink(struct vhost_iotlb *iotlb,
+ struct vhost_iotlb_map *map)
+{
+ vhost_iotlb_itree_remove(map, &iotlb->root);
+ list_del(&map->link);
+ iotlb->nmaps--;
+}
+
/**
* vhost_iotlb_map_free - remove a map node and free it
* @iotlb: the IOTLB
@@ -28,10 +36,8 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map,
void vhost_iotlb_map_free(struct vhost_iotlb *iotlb,
struct vhost_iotlb_map *map)
{
- vhost_iotlb_itree_remove(map, &iotlb->root);
- list_del(&map->link);
+ vhost_iotlb_map_unlink(iotlb, map);
kfree(map);
- iotlb->nmaps--;
}
EXPORT_SYMBOL_GPL(vhost_iotlb_map_free);
@@ -57,14 +63,25 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb,
if (last < start)
return -EFAULT;
+ if (!iotlb->limit)
+ return -EINVAL;
+
/* If the range being mapped is [0, ULONG_MAX], split it into two entries
* otherwise its size would overflow u64.
*/
if (start == 0 && last == ULONG_MAX) {
u64 mid = last / 2;
- int err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr,
- perm, opaque);
+ int err;
+
+ if (iotlb->limit < 2)
+ return -ENOSPC;
+ if (!(iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) &&
+ iotlb->nmaps > iotlb->limit - 2)
+ return -ENOSPC;
+
+ err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr,
+ perm, opaque);
if (err)
return err;
@@ -72,17 +89,19 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb,
start = mid + 1;
}
- if (iotlb->limit &&
- iotlb->nmaps == iotlb->limit &&
- iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) {
- map = list_first_entry(&iotlb->list, typeof(*map), link);
- vhost_iotlb_map_free(iotlb, map);
+ if (iotlb->nmaps >= iotlb->limit) {
+ if (iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) {
+ map = list_first_entry(&iotlb->list, typeof(*map), link);
+ vhost_iotlb_map_unlink(iotlb, map);
+ } else {
+ return -ENOSPC;
+ }
+ } else {
+ map = kmalloc(sizeof(*map), GFP_ATOMIC);
+ if (!map)
+ return -ENOMEM;
}
- map = kmalloc(sizeof(*map), GFP_ATOMIC);
- if (!map)
- return -ENOMEM;
-
map->start = start;
map->size = last - start + 1;
map->last = last;
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 72bd44f2f142..19929e9b8b0a 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1698,6 +1698,8 @@ static int vhost_net_set_features(struct vhost_net *n, const u64 *features)
if (virtio_features_test_bit(features, VIRTIO_F_ACCESS_PLATFORM)) {
if (vhost_init_device_iotlb(&n->dev))
goto out_unlock;
+ } else {
+ vhost_clear_device_iotlb(&n->dev);
}
for (i = 0; i < VHOST_NET_VQ_MAX; ++i) {
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 2dd2a5ccd09e..219af4f0d06a 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -34,6 +34,11 @@ enum {
#define VHOST_VDPA_DEV_MAX (1U << MINORBITS)
+static int max_iotlb_entries = 2048;
+module_param(max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
#define VHOST_VDPA_IOTLB_BUCKETS 16
struct vhost_vdpa_as {
@@ -112,12 +117,14 @@ static struct vhost_vdpa_as *vhost_vdpa_alloc_as(struct vhost_vdpa *v, u32 asid)
if (asid >= v->vdpa->nas)
return NULL;
+ if (max_iotlb_entries <= 0)
+ return NULL;
as = kmalloc(sizeof(*as), GFP_KERNEL);
if (!as)
return NULL;
- vhost_iotlb_init(&as->iotlb, 0, 0);
+ vhost_iotlb_init(&as->iotlb, max_iotlb_entries, 0);
as->id = asid;
hlist_add_head(&as->hash_link, head);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 583fa2ec9bc3..ebdbf1cf9143 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -344,6 +344,17 @@ static void __vhost_vq_meta_reset(struct vhost_virtqueue *vq)
vq->meta_iotlb[j] = NULL;
}
+/* Caller must hold the virtqueue mutex. */
+static void vhost_vq_invalidate_access(struct vhost_virtqueue *vq)
+{
+ vq->desc = NULL;
+ vq->avail = NULL;
+ vq->used = NULL;
+ vq->log_used = false;
+ vq->log_addr = -1ull;
+ __vhost_vq_meta_reset(vq);
+}
+
static void vhost_vq_meta_reset(struct vhost_dev *d)
{
int i;
@@ -1141,6 +1152,9 @@ EXPORT_SYMBOL_GPL(vhost_dev_set_owner);
static struct vhost_iotlb *iotlb_alloc(void)
{
+ if (max_iotlb_entries <= 0)
+ return NULL;
+
return vhost_iotlb_alloc(max_iotlb_entries,
VHOST_IOTLB_FLAG_RETIRE);
}
@@ -1915,6 +1929,13 @@ int vq_meta_prefetch(struct vhost_virtqueue *vq)
{
unsigned int num = vq->num;
+ /*
+ * vhost_vq_invalidate_access() clears all three addresses together.
+ * A single zero address may be a valid GIOVA in IOTLB mode.
+ */
+ if (!vq->desc && !vq->avail && !vq->used)
+ return 0;
+
if (!vq->iotlb)
return 1;
@@ -1985,6 +2006,8 @@ static long vhost_set_memory(struct vhost_dev *d, struct vhost_memory __user *m)
return -EOPNOTSUPP;
if (mem.nregions > max_mem_regions)
return -E2BIG;
+ if (max_iotlb_entries <= 0)
+ return -EINVAL;
newmem = kvzalloc(struct_size(newmem, regions, mem.nregions),
GFP_KERNEL);
if (!newmem)
@@ -2283,11 +2306,48 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
}
EXPORT_SYMBOL_GPL(vhost_vring_ioctl);
+/* Caller must hold the device mutex. */
+void vhost_clear_device_iotlb(struct vhost_dev *d)
+{
+ struct vhost_iotlb *iotlb;
+ int i;
+
+ iotlb = d->iotlb;
+ if (!iotlb)
+ return;
+
+ vhost_dev_lock_vqs(d);
+
+ /*
+ * vhost_dev_lock_vqs() takes all VQ mutexes in index order. Drop the
+ * device-wide view while they are held, then clear each per-VQ view
+ * and its cached ring access before releasing the locks. Workers
+ * cannot observe a mixed address-space state during this handoff.
+ */
+ d->iotlb = NULL;
+
+ for (i = 0; i < d->nvqs; ++i) {
+ struct vhost_virtqueue *vq = d->vqs[i];
+
+ vq->iotlb = NULL;
+ vhost_vq_invalidate_access(vq);
+ }
+
+ vhost_dev_unlock_vqs(d);
+ vhost_clear_msg(d);
+ vhost_iotlb_free(iotlb);
+ wake_up_interruptible_poll(&d->wait, EPOLLIN | EPOLLRDNORM);
+}
+EXPORT_SYMBOL_GPL(vhost_clear_device_iotlb);
+
int vhost_init_device_iotlb(struct vhost_dev *d)
{
struct vhost_iotlb *niotlb, *oiotlb;
int i;
+ if (max_iotlb_entries <= 0)
+ return -EINVAL;
+
niotlb = iotlb_alloc();
if (!niotlb)
return -ENOMEM;
@@ -2300,7 +2360,10 @@ int vhost_init_device_iotlb(struct vhost_dev *d)
mutex_lock(&vq->mutex);
vq->iotlb = niotlb;
- __vhost_vq_meta_reset(vq);
+ if (oiotlb)
+ __vhost_vq_meta_reset(vq);
+ else
+ vhost_vq_invalidate_access(vq);
mutex_unlock(&vq->mutex);
}
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index b49f08e4a1b4..7ec10050a508 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -276,6 +276,7 @@ ssize_t vhost_chr_read_iter(struct vhost_dev *dev, struct iov_iter *to,
int noblock);
ssize_t vhost_chr_write_iter(struct vhost_dev *dev,
struct iov_iter *from);
+void vhost_clear_device_iotlb(struct vhost_dev *d);
int vhost_init_device_iotlb(struct vhost_dev *d);
void vhost_iotlb_map_free(struct vhost_iotlb *iotlb,
diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
index 78cc66fbb3dd..65297f245b27 100644
--- a/drivers/vhost/vsock.c
+++ b/drivers/vhost/vsock.c
@@ -819,6 +819,8 @@ static int vhost_vsock_set_features(struct vhost_vsock *vsock, u64 features)
if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
if (vhost_init_device_iotlb(&vsock->dev))
goto err;
+ } else {
+ vhost_clear_device_iotlb(&vsock->dev);
}
vsock->seqpacket_allow = features & (1ULL << VIRTIO_VSOCK_F_SEQPACKET);
diff --git a/drivers/video/fbdev/core/fb_chrdev.c b/drivers/video/fbdev/core/fb_chrdev.c
index 4ebd16b7e3b8..54f926fb411b 100644
--- a/drivers/video/fbdev/core/fb_chrdev.c
+++ b/drivers/video/fbdev/core/fb_chrdev.c
@@ -85,11 +85,7 @@ static long do_fb_ioctl(struct fb_info *info, unsigned int cmd,
var.activate &= ~FB_ACTIVATE_KD_TEXT;
console_lock();
lock_fb_info(info);
- ret = fbcon_modechange_possible(info, &var);
- if (!ret)
- ret = fb_set_var(info, &var);
- if (!ret)
- fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL);
+ ret = fb_set_var_from_user(info, &var);
unlock_fb_info(info);
console_unlock();
if (!ret && copy_to_user(argp, &var, sizeof(var)))
diff --git a/drivers/video/fbdev/core/fbcon.c b/drivers/video/fbdev/core/fbcon.c
index 32d394035762..532522c2aed0 100644
--- a/drivers/video/fbdev/core/fbcon.c
+++ b/drivers/video/fbdev/core/fbcon.c
@@ -2757,7 +2757,6 @@ void fbcon_update_vcs(struct fb_info *info, bool all)
else
fbcon_modechanged(info);
}
-EXPORT_SYMBOL(fbcon_update_vcs);
/* let fbcon check if it supports a new screen resolution */
int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *var)
@@ -2785,7 +2784,6 @@ int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *va
return 0;
}
-EXPORT_SYMBOL_GPL(fbcon_modechange_possible);
int fbcon_mode_deleted(struct fb_info *info,
struct fb_videomode *mode)
diff --git a/drivers/video/fbdev/core/fbmem.c b/drivers/video/fbdev/core/fbmem.c
index 30a2c0d47e5c..1533d43a0a0c 100644
--- a/drivers/video/fbdev/core/fbmem.c
+++ b/drivers/video/fbdev/core/fbmem.c
@@ -346,6 +346,19 @@ fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var)
}
EXPORT_SYMBOL(fb_set_var);
+int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var)
+{
+ int ret = fbcon_modechange_possible(info, var);
+
+ if (!ret)
+ ret = fb_set_var(info, var);
+ if (!ret)
+ fbcon_update_vcs(info, var->activate & FB_ACTIVATE_ALL);
+
+ return ret;
+}
+EXPORT_SYMBOL(fb_set_var_from_user);
+
static void fb_lcd_notify_blank(struct fb_info *info)
{
int power;
diff --git a/drivers/video/fbdev/core/fbsysfs.c b/drivers/video/fbdev/core/fbsysfs.c
index fe8bd33e64ab..d363f94207c3 100644
--- a/drivers/video/fbdev/core/fbsysfs.c
+++ b/drivers/video/fbdev/core/fbsysfs.c
@@ -20,9 +20,7 @@ static int activate(struct fb_info *fb_info, struct fb_var_screeninfo *var)
var->activate |= FB_ACTIVATE_FORCE;
console_lock();
lock_fb_info(fb_info);
- err = fb_set_var(fb_info, var);
- if (!err)
- fbcon_update_vcs(fb_info, var->activate & FB_ACTIVATE_ALL);
+ err = fb_set_var_from_user(fb_info, var);
unlock_fb_info(fb_info);
console_unlock();
if (err)
diff --git a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
index 211f23648686..2b99a58ad675 100644
--- a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
+++ b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
@@ -1099,7 +1099,11 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
rg = omapfb_get_mem_region(ofbi->region);
- start = omapfb_get_region_paddr(ofbi);
+ if (ofbi->rotation_type == OMAP_DSS_ROT_VRFB)
+ start = rg->vrfb.paddr[0];
+ else
+ start = rg->paddr;
+
len = fix->smem_len;
DBG("user mmap region start %lx, len %d, off %lx\n", start, len,
@@ -1109,6 +1113,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
vma->vm_ops = &mmap_user_ops;
vma->vm_private_data = rg;
+ atomic_inc(&rg->map_count);
+
r = vm_iomap_memory(vma, start, len);
if (r)
goto error;
@@ -1121,7 +1127,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
return 0;
error:
- omapfb_put_mem_region(ofbi->region);
+ atomic_dec(&rg->map_count);
+ omapfb_put_mem_region(rg);
return r;
}
diff --git a/drivers/video/fbdev/ps3fb.c b/drivers/video/fbdev/ps3fb.c
index dbcda307f6a6..1376d19b19ae 100644
--- a/drivers/video/fbdev/ps3fb.c
+++ b/drivers/video/fbdev/ps3fb.c
@@ -29,7 +29,6 @@
#include <linux/freezer.h>
#include <linux/uaccess.h>
#include <linux/fb.h>
-#include <linux/fbcon.h>
#include <linux/init.h>
#include <asm/cell-regs.h>
@@ -830,9 +829,7 @@ static int ps3fb_ioctl(struct fb_info *info, unsigned int cmd,
/* Force, in case only special bits changed */
var.activate |= FB_ACTIVATE_FORCE;
par->new_mode_id = val;
- retval = fb_set_var(info, &var);
- if (!retval)
- fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL);
+ retval = fb_set_var_from_user(info, &var);
console_unlock();
}
break;
diff --git a/drivers/video/fbdev/sh_mobile_lcdcfb.c b/drivers/video/fbdev/sh_mobile_lcdcfb.c
index dd950e4ab5ce..6038f217ab42 100644
--- a/drivers/video/fbdev/sh_mobile_lcdcfb.c
+++ b/drivers/video/fbdev/sh_mobile_lcdcfb.c
@@ -15,7 +15,6 @@
#include <linux/ctype.h>
#include <linux/dma-mapping.h>
#include <linux/delay.h>
-#include <linux/fbcon.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/ioctl.h>
@@ -1765,11 +1764,9 @@ static void sh_mobile_fb_reconfig(struct fb_info *info)
var.height = ch->display.height;
var.activate = FB_ACTIVATE_NOW;
- if (fb_set_var(info, &var) < 0)
+ if (fb_set_var_from_user(info, &var) < 0)
/* Couldn't reconfigure, hopefully, can continue as before */
return;
-
- fbcon_update_vcs(info, true);
}
/*
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index ddab68959671..b8199e578ea9 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -1459,7 +1459,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq,
struct scatterlist *sg;
unsigned int i, n, c, descs_used, err_idx, len;
__le16 head_flags, flags;
- u16 head, id, prev, curr, avail_used_flags;
+ u16 head, id, prev, curr, avail_used_flags, unpub_flags;
int err;
START_USE(vq);
@@ -1584,15 +1584,30 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq,
curr = vq->free_head;
vq->packed.avail_used_flags = avail_used_flags;
+ unpub_flags = avail_used_flags ^ (1 << VRING_PACKED_DESC_F_AVAIL |
+ 1 << VRING_PACKED_DESC_F_USED);
for (n = 0; n < total_sg; n++) {
if (i == err_idx)
break;
+ /*
+ * The mapping loop made every descriptor but the head
+ * available. Stamp the previous wrap counter's AVAIL and USED
+ * bits on those, so that a later and shorter chain at this head
+ * does not leave one of them available beyond its own last
+ * descriptor. Marking them used instead would hand
+ * is_used_desc_packed() a completion we never made.
+ */
+ if (i != head)
+ desc[i].flags = cpu_to_le16(unpub_flags);
vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr]);
curr = vq->packed.desc_extra[curr].next;
i++;
- if (i >= vq->packed.vring.num)
+ if (i >= vq->packed.vring.num) {
i = 0;
+ unpub_flags ^= 1 << VRING_PACKED_DESC_F_AVAIL |
+ 1 << VRING_PACKED_DESC_F_USED;
+ }
}
END_USE(vq);
diff --git a/drivers/xen/pvcalls-back.c b/drivers/xen/pvcalls-back.c
index fd7ed65e0197..da1b516b9cfd 100644
--- a/drivers/xen/pvcalls-back.c
+++ b/drivers/xen/pvcalls-back.c
@@ -650,7 +650,7 @@ static int pvcalls_back_bind(struct xenbus_device *dev,
if (ret < 0)
goto out;
- ret = inet_bind(map->sock, (struct sockaddr *)&req->u.bind.addr,
+ ret = inet_bind(map->sock, (struct sockaddr_unsized *)&req->u.bind.addr,
req->u.bind.len);
if (ret < 0)
goto out;
diff --git a/drivers/xen/pvcalls-front.c b/drivers/xen/pvcalls-front.c
index 4926d4badc57..59f591682628 100644
--- a/drivers/xen/pvcalls-front.c
+++ b/drivers/xen/pvcalls-front.c
@@ -32,6 +32,7 @@ struct pvcalls_bedata {
struct xen_pvcalls_front_ring ring;
grant_ref_t ref;
int irq;
+ bool disabled;
struct list_head socket_mappings;
spinlock_t socket_lock;
@@ -131,6 +132,20 @@ static inline int get_request(struct pvcalls_bedata *bedata, int *req_id)
return 0;
}
+/*
+ * Wait for the backend's response to req_id, or for the frontend to be
+ * disabled because the backend violated the wire protocol. Returns 0 once
+ * the response has arrived, or -EIO if the frontend was disabled.
+ */
+static int pvcalls_front_wait_rsp(struct pvcalls_bedata *bedata, u32 req_id)
+{
+ wait_event(bedata->inflight_req,
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled));
+
+ return READ_ONCE(bedata->disabled) ? -EIO : 0;
+}
+
static bool pvcalls_front_write_todo(struct sock_mapping *map)
{
struct pvcalls_data_intf *intf = map->active.ring;
@@ -168,7 +183,8 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
struct pvcalls_bedata *bedata;
struct xen_pvcalls_response *rsp;
uint8_t *src, *dst;
- int req_id = 0, more = 0, done = 0;
+ u32 req_id = 0;
+ int more = 0, done = 0;
if (dev == NULL)
return IRQ_HANDLED;
@@ -179,12 +195,31 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
pvcalls_exit();
return IRQ_HANDLED;
}
+ if (READ_ONCE(bedata->disabled)) {
+ pvcalls_exit();
+ return IRQ_HANDLED;
+ }
again:
while (RING_HAS_UNCONSUMED_RESPONSES(&bedata->ring)) {
rsp = RING_GET_RESPONSE(&bedata->ring, bedata->ring.rsp_cons);
req_id = rsp->req_id;
+ if (req_id >= PVCALLS_NR_RSP_PER_RING) {
+ /*
+ * The backend supplied a req_id that would index
+ * bedata->rsp[] out of bounds: a protocol violation
+ * from a malicious or buggy backend. Log once, stop
+ * trusting this backend and disable the frontend rather
+ * than silently dropping the response and continuing.
+ */
+ pr_err_once("pvcalls: backend sent out-of-range req_id %u, disabling frontend\n",
+ req_id);
+ WRITE_ONCE(bedata->disabled, true);
+ bedata->ring.rsp_cons++;
+ done = 1;
+ break;
+ }
if (rsp->cmd == PVCALLS_POLL) {
struct sock_mapping *map = (struct sock_mapping *)(uintptr_t)
rsp->u.poll.id;
@@ -217,7 +252,7 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
}
RING_FINAL_CHECK_FOR_RESPONSES(&bedata->ring, more);
- if (more)
+ if (more && !READ_ONCE(bedata->disabled))
goto again;
if (done)
wake_up(&bedata->inflight_req);
@@ -330,8 +365,11 @@ int pvcalls_front_socket(struct socket *sock)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit();
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -477,8 +515,11 @@ int pvcalls_front_connect(struct socket *sock, struct sockaddr *addr,
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -711,8 +752,11 @@ int pvcalls_front_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -761,8 +805,11 @@ int pvcalls_front_listen(struct socket *sock, int backlog)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -820,6 +867,14 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock,
}
}
+ if (READ_ONCE(bedata->disabled)) {
+ clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
+ (void *)&map->passive.flags);
+ wake_up(&map->passive.inflight_accept_req);
+ pvcalls_exit_sock(sock);
+ return -EIO;
+ }
+
map2 = kzalloc(sizeof(*map2), GFP_KERNEL);
if (map2 == NULL) {
clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
@@ -880,10 +935,18 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock,
}
if (wait_event_interruptible(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id)) {
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled))) {
pvcalls_exit_sock(sock);
return -EINTR;
}
+ if (READ_ONCE(bedata->disabled)) {
+ clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
+ (void *)&map->passive.flags);
+ wake_up(&map->passive.inflight_accept_req);
+ pvcalls_exit_sock(sock);
+ return -EIO;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -1054,7 +1117,8 @@ int pvcalls_front_release(struct socket *sock)
notify_remote_via_irq(bedata->irq);
wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled));
if (map->active_socket) {
/*
diff --git a/fs/afs/Makefile b/fs/afs/Makefile
index b49b8fe682f3..0d8f1982d596 100644
--- a/fs/afs/Makefile
+++ b/fs/afs/Makefile
@@ -30,6 +30,7 @@ kafs-y := \
server.o \
server_list.o \
super.o \
+ symlink.o \
validation.o \
vlclient.o \
vl_alias.o \
diff --git a/fs/afs/dir.c b/fs/afs/dir.c
index 7c3cef0835a8..a35148d14395 100644
--- a/fs/afs/dir.c
+++ b/fs/afs/dir.c
@@ -46,6 +46,8 @@ static int afs_symlink(struct mnt_idmap *idmap, struct inode *dir,
static int afs_rename(struct mnt_idmap *idmap, struct inode *old_dir,
struct dentry *old_dentry, struct inode *new_dir,
struct dentry *new_dentry, unsigned int flags);
+static int afs_dir_writepages(struct address_space *mapping,
+ struct writeback_control *wbc);
const struct file_operations afs_dir_file_operations = {
.open = afs_dir_open,
@@ -70,7 +72,7 @@ const struct inode_operations afs_dir_inode_operations = {
};
const struct address_space_operations afs_dir_aops = {
- .writepages = afs_single_writepages,
+ .writepages = afs_dir_writepages,
};
const struct dentry_operations afs_fs_dentry_operations = {
@@ -235,22 +237,13 @@ static ssize_t afs_do_read_single(struct afs_vnode *dvnode, struct file *file)
struct iov_iter iter;
ssize_t ret;
loff_t i_size;
- bool is_dir = (S_ISDIR(dvnode->netfs.inode.i_mode) &&
- !test_bit(AFS_VNODE_MOUNTPOINT, &dvnode->flags));
i_size = i_size_read(&dvnode->netfs.inode);
- if (is_dir) {
- if (i_size < AFS_DIR_BLOCK_SIZE)
- return afs_bad(dvnode, afs_file_error_dir_small);
- if (i_size > AFS_DIR_BLOCK_SIZE * 1024) {
- trace_afs_file_error(dvnode, -EFBIG, afs_file_error_dir_big);
- return -EFBIG;
- }
- } else {
- if (i_size > AFSPATHMAX) {
- trace_afs_file_error(dvnode, -EFBIG, afs_file_error_dir_big);
- return -EFBIG;
- }
+ if (i_size < AFS_DIR_BLOCK_SIZE)
+ return afs_bad(dvnode, afs_file_error_dir_small);
+ if (i_size > AFS_DIR_BLOCK_SIZE * 1024) {
+ trace_afs_file_error(dvnode, -EFBIG, afs_file_error_dir_big);
+ return -EFBIG;
}
/* Expand the storage. TODO: Shrink the storage too. */
@@ -279,24 +272,18 @@ static ssize_t afs_do_read_single(struct afs_vnode *dvnode, struct file *file)
* buffer.
*/
ret = -ESTALE;
- } else if (is_dir) {
+ } else {
int ret2 = afs_dir_check(dvnode);
if (ret2 < 0)
ret = ret2;
- } else if (i_size < folioq_folio_size(dvnode->directory, 0)) {
- /* NUL-terminate a symlink. */
- char *symlink = kmap_local_folio(folioq_folio(dvnode->directory, 0), 0);
-
- symlink[i_size] = 0;
- kunmap_local(symlink);
}
}
return ret;
}
-ssize_t afs_read_single(struct afs_vnode *dvnode, struct file *file)
+static ssize_t afs_read_single(struct afs_vnode *dvnode, struct file *file)
{
ssize_t ret;
@@ -1774,13 +1761,20 @@ static int afs_link(struct dentry *from, struct inode *dir,
return ret;
}
+static void afs_symlink_put(struct afs_operation *op)
+{
+ kfree(op->create.symlink);
+ op->create.symlink = NULL;
+ afs_create_put(op);
+}
+
static const struct afs_operation_ops afs_symlink_operation = {
.issue_afs_rpc = afs_fs_symlink,
.issue_yfs_rpc = yfs_fs_symlink,
.success = afs_create_success,
.aborted = afs_check_for_remote_deletion,
.edit_dir = afs_create_edit_dir,
- .put = afs_create_put,
+ .put = afs_symlink_put,
};
/*
@@ -1790,7 +1784,9 @@ static int afs_symlink(struct mnt_idmap *idmap, struct inode *dir,
struct dentry *dentry, const char *content)
{
struct afs_operation *op;
+ struct afs_symlink *symlink;
struct afs_vnode *dvnode = AFS_FS_I(dir);
+ size_t clen = strlen(content);
int ret;
_enter("{%llx:%llu},{%pd},%s",
@@ -1802,12 +1798,20 @@ static int afs_symlink(struct mnt_idmap *idmap, struct inode *dir,
goto error;
ret = -EINVAL;
- if (strlen(content) >= AFSPATHMAX)
+ if (clen >= AFSPATHMAX)
+ goto error;
+
+ ret = -ENOMEM;
+ symlink = kmalloc_flex(struct afs_symlink, content, clen + 1, GFP_KERNEL);
+ if (!symlink)
goto error;
+ refcount_set(&symlink->ref, 1);
+ memcpy(symlink->content, content, clen + 1);
op = afs_alloc_operation(NULL, dvnode->volume);
if (IS_ERR(op)) {
ret = PTR_ERR(op);
+ kfree(symlink);
goto error;
}
@@ -1819,7 +1823,7 @@ static int afs_symlink(struct mnt_idmap *idmap, struct inode *dir,
op->dentry = dentry;
op->ops = &afs_symlink_operation;
op->create.reason = afs_edit_dir_for_symlink;
- op->create.symlink = content;
+ op->create.symlink = symlink;
op->mtime = current_time(dir);
ret = afs_do_sync_operation(op);
afs_dir_unuse_cookie(dvnode, ret);
@@ -2203,15 +2207,13 @@ static int afs_rename(struct mnt_idmap *idmap, struct inode *old_dir,
}
/*
- * Write the file contents to the cache as a single blob.
+ * Write the directory contents to the cache as a single blob.
*/
-int afs_single_writepages(struct address_space *mapping,
- struct writeback_control *wbc)
+static int afs_dir_writepages(struct address_space *mapping,
+ struct writeback_control *wbc)
{
struct afs_vnode *dvnode = AFS_FS_I(mapping->host);
struct iov_iter iter;
- bool is_dir = (S_ISDIR(dvnode->netfs.inode.i_mode) &&
- !test_bit(AFS_VNODE_MOUNTPOINT, &dvnode->flags));
int ret = 0;
/* Need to lock to prevent the folio queue and folios from being thrown
@@ -2226,9 +2228,7 @@ int afs_single_writepages(struct address_space *mapping,
down_read(&dvnode->validate_lock);
}
- if (is_dir ?
- test_bit(AFS_VNODE_DIR_VALID, &dvnode->flags) :
- atomic64_read(&dvnode->cb_expires_at) != AFS_NO_CB_PROMISE) {
+ if (test_bit(AFS_VNODE_DIR_VALID, &dvnode->flags)) {
iov_iter_folio_queue(&iter, ITER_SOURCE, dvnode->directory, 0, 0,
i_size_read(&dvnode->netfs.inode));
ret = netfs_writeback_single(mapping, wbc, &iter);
diff --git a/fs/afs/fsclient.c b/fs/afs/fsclient.c
index 2dfcd4a20d10..38709e07b296 100644
--- a/fs/afs/fsclient.c
+++ b/fs/afs/fsclient.c
@@ -889,7 +889,7 @@ void afs_fs_symlink(struct afs_operation *op)
namesz = name->len;
padsz = (4 - (namesz & 3)) & 3;
- c_namesz = strlen(op->create.symlink);
+ c_namesz = strlen(op->create.symlink->content);
c_padsz = (4 - (c_namesz & 3)) & 3;
reqsz = (6 * 4) + namesz + padsz + c_namesz + c_padsz + (6 * 4);
@@ -913,7 +913,7 @@ void afs_fs_symlink(struct afs_operation *op)
bp = (void *) bp + padsz;
}
*bp++ = htonl(c_namesz);
- memcpy(bp, op->create.symlink, c_namesz);
+ memcpy(bp, op->create.symlink->content, c_namesz);
bp = (void *) bp + c_namesz;
if (c_padsz > 0) {
memset(bp, 0, c_padsz);
diff --git a/fs/afs/inode.c b/fs/afs/inode.c
index f8d9ceb914ce..e170315f5471 100644
--- a/fs/afs/inode.c
+++ b/fs/afs/inode.c
@@ -25,96 +25,6 @@
#include "internal.h"
#include "afs_fs.h"
-void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)
-{
- size_t size = strlen(op->create.symlink) + 1;
- size_t dsize = 0;
- char *p;
-
- if (netfs_alloc_folioq_buffer(NULL, &vnode->directory, &dsize, size,
- mapping_gfp_mask(vnode->netfs.inode.i_mapping)) < 0)
- return;
-
- vnode->directory_size = dsize;
- p = kmap_local_folio(folioq_folio(vnode->directory, 0), 0);
- memcpy(p, op->create.symlink, size);
- kunmap_local(p);
- set_bit(AFS_VNODE_DIR_READ, &vnode->flags);
- netfs_single_mark_inode_dirty(&vnode->netfs.inode);
-}
-
-static void afs_put_link(void *arg)
-{
- struct folio *folio = virt_to_folio(arg);
-
- kunmap_local(arg);
- folio_put(folio);
-}
-
-const char *afs_get_link(struct dentry *dentry, struct inode *inode,
- struct delayed_call *callback)
-{
- struct afs_vnode *vnode = AFS_FS_I(inode);
- struct folio *folio;
- char *content;
- ssize_t ret;
-
- if (!dentry) {
- /* RCU pathwalk. */
- if (!test_bit(AFS_VNODE_DIR_READ, &vnode->flags) || !afs_check_validity(vnode))
- return ERR_PTR(-ECHILD);
- goto good;
- }
-
- if (test_bit(AFS_VNODE_DIR_READ, &vnode->flags))
- goto fetch;
-
- ret = afs_validate(vnode, NULL);
- if (ret < 0)
- return ERR_PTR(ret);
-
- if (!test_and_clear_bit(AFS_VNODE_ZAP_DATA, &vnode->flags) &&
- test_bit(AFS_VNODE_DIR_READ, &vnode->flags))
- goto good;
-
-fetch:
- ret = afs_read_single(vnode, NULL);
- if (ret < 0)
- return ERR_PTR(ret);
- set_bit(AFS_VNODE_DIR_READ, &vnode->flags);
-
-good:
- folio = folioq_folio(vnode->directory, 0);
- folio_get(folio);
- content = kmap_local_folio(folio, 0);
- set_delayed_call(callback, afs_put_link, content);
- return content;
-}
-
-int afs_readlink(struct dentry *dentry, char __user *buffer, int buflen)
-{
- DEFINE_DELAYED_CALL(done);
- const char *content;
- int len;
-
- content = afs_get_link(dentry, d_inode(dentry), &done);
- if (IS_ERR(content)) {
- do_delayed_call(&done);
- return PTR_ERR(content);
- }
-
- len = umin(strlen(content), buflen);
- if (copy_to_user(buffer, content, len))
- len = -EFAULT;
- do_delayed_call(&done);
- return len;
-}
-
-static const struct inode_operations afs_symlink_inode_operations = {
- .get_link = afs_get_link,
- .readlink = afs_readlink,
-};
-
static noinline void dump_vnode(struct afs_vnode *vnode, struct afs_vnode *parent_vnode)
{
static unsigned long once_only;
@@ -214,7 +124,7 @@ static int afs_inode_init_from_status(struct afs_operation *op,
inode->i_mode = S_IFLNK | status->mode;
inode->i_op = &afs_symlink_inode_operations;
}
- inode->i_mapping->a_ops = &afs_dir_aops;
+ inode->i_mapping->a_ops = &afs_symlink_aops;
inode_nohighmem(inode);
mapping_set_release_always(inode->i_mapping);
break;
@@ -755,13 +665,15 @@ void afs_evict_inode(struct inode *inode)
.range_end = LLONG_MAX,
};
- afs_single_writepages(inode->i_mapping, &wbc);
+ inode->i_mapping->a_ops->writepages(inode->i_mapping, &wbc);
}
flush_delayed_work(&vnode->lock_work);
netfs_wait_for_outstanding_io(inode);
truncate_inode_pages_final(&inode->i_data);
netfs_free_folioq_buffer(vnode->directory);
+ if (vnode->symlink)
+ afs_evict_symlink(vnode);
afs_set_cache_aux(vnode, &aux);
netfs_clear_inode_writeback(inode, &aux);
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index 25b5fb3eda95..61cba72f0c40 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -712,6 +712,7 @@ struct afs_vnode {
#define AFS_VNODE_DIR_READ 11 /* Set if we've read a dir's contents */
struct folio_queue *directory; /* Directory contents */
+ struct afs_symlink __rcu *symlink; /* Symlink content */
struct list_head wb_keys; /* List of keys available for writeback */
struct list_head pending_locks; /* locks waiting to be granted */
struct list_head granted_locks; /* locks granted on this file */
@@ -778,6 +779,15 @@ struct afs_permits {
struct afs_permit permits[] __counted_by(nr_permits); /* List of permits sorted by key pointer */
};
+/*
+ * Copy of symlink content for normal use.
+ */
+struct afs_symlink {
+ struct rcu_head rcu;
+ refcount_t ref;
+ char content[];
+};
+
/*
* Error prioritisation and accumulation.
*/
@@ -889,7 +899,7 @@ struct afs_operation {
struct {
int reason; /* enum afs_edit_dir_reason */
mode_t mode;
- const char *symlink;
+ struct afs_symlink *symlink;
} create;
struct {
bool need_rehash;
@@ -1100,13 +1110,10 @@ extern const struct inode_operations afs_dir_inode_operations;
extern const struct address_space_operations afs_dir_aops;
extern const struct dentry_operations afs_fs_dentry_operations;
-ssize_t afs_read_single(struct afs_vnode *dvnode, struct file *file);
ssize_t afs_read_dir(struct afs_vnode *dvnode, struct file *file)
__acquires(&dvnode->validate_lock);
extern void afs_d_release(struct dentry *);
extern void afs_check_for_remote_deletion(struct afs_operation *);
-int afs_single_writepages(struct address_space *mapping,
- struct writeback_control *wbc);
/*
* dir_edit.c
@@ -1256,10 +1263,6 @@ extern void afs_fs_probe_cleanup(struct afs_net *);
*/
extern const struct afs_operation_ops afs_fetch_status_operation;
-void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op);
-const char *afs_get_link(struct dentry *dentry, struct inode *inode,
- struct delayed_call *callback);
-int afs_readlink(struct dentry *dentry, char __user *buffer, int buflen);
extern void afs_vnode_commit_status(struct afs_operation *, struct afs_vnode_param *);
extern int afs_fetch_status(struct afs_vnode *, struct key *, bool, afs_access_t *);
extern int afs_ilookup5_test_by_fid(struct inode *, void *);
@@ -1610,6 +1613,21 @@ void afs_detach_volume_from_servers(struct afs_volume *volume, struct afs_server
extern int __init afs_fs_init(void);
extern void afs_fs_exit(void);
+/*
+ * symlink.c
+ */
+extern const struct inode_operations afs_symlink_inode_operations;
+extern const struct address_space_operations afs_symlink_aops;
+
+void afs_invalidate_symlink(struct afs_vnode *vnode);
+void afs_evict_symlink(struct afs_vnode *vnode);
+void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op);
+const char *afs_get_link(struct dentry *dentry, struct inode *inode,
+ struct delayed_call *callback);
+int afs_readlink(struct dentry *dentry, char __user *buffer, int buflen);
+int afs_symlink_writepages(struct address_space *mapping,
+ struct writeback_control *wbc);
+
/*
* validation.c
*/
diff --git a/fs/afs/rxrpc.c b/fs/afs/rxrpc.c
index b3207e141343..6415f42cf7ff 100644
--- a/fs/afs/rxrpc.c
+++ b/fs/afs/rxrpc.c
@@ -82,16 +82,16 @@ int afs_open_socket(struct afs_net *net)
if (ret < 0)
pr_err("Couldn't create RxGK CM key: %d\n", ret);
- ret = kernel_bind(socket, (struct sockaddr *) &srx, sizeof(srx));
+ ret = kernel_bind(socket, (struct sockaddr_unsized *) &srx, sizeof(srx));
if (ret == -EADDRINUSE) {
srx.transport.sin6.sin6_port = 0;
- ret = kernel_bind(socket, (struct sockaddr *) &srx, sizeof(srx));
+ ret = kernel_bind(socket, (struct sockaddr_unsized *) &srx, sizeof(srx));
}
if (ret < 0)
goto error_2;
srx.srx_service = YFS_CM_SERVICE;
- ret = kernel_bind(socket, (struct sockaddr *) &srx, sizeof(srx));
+ ret = kernel_bind(socket, (struct sockaddr_unsized *) &srx, sizeof(srx));
if (ret < 0)
goto error_2;
diff --git a/fs/afs/symlink.c b/fs/afs/symlink.c
new file mode 100644
index 000000000000..ed5868369f37
--- /dev/null
+++ b/fs/afs/symlink.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/* AFS filesystem symbolic link handling
+ *
+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#include <linux/kernel.h>
+#include <linux/fs.h>
+#include <linux/namei.h>
+#include <linux/pagemap.h>
+#include <linux/iov_iter.h>
+#include "internal.h"
+
+static void afs_put_symlink(struct afs_symlink *symlink)
+{
+ if (refcount_dec_and_test(&symlink->ref))
+ kfree_rcu(symlink, rcu);
+}
+
+static void afs_replace_symlink(struct afs_vnode *vnode, struct afs_symlink *symlink)
+{
+ struct afs_symlink *old;
+
+ old = rcu_replace_pointer(vnode->symlink, symlink,
+ lockdep_is_held(&vnode->validate_lock));
+ if (old)
+ afs_put_symlink(old);
+}
+
+/*
+ * In the event that a third-party update of a symlink occurs, dispose of the
+ * copy of the old contents. Called under ->validate_lock.
+ */
+void afs_invalidate_symlink(struct afs_vnode *vnode)
+{
+ afs_replace_symlink(vnode, NULL);
+}
+
+/*
+ * Dispose of a symlink copy during inode deletion.
+ */
+void afs_evict_symlink(struct afs_vnode *vnode)
+{
+ struct afs_symlink *old;
+
+ old = rcu_replace_pointer(vnode->symlink, NULL, true);
+ if (old)
+ afs_put_symlink(old);
+
+}
+
+/*
+ * Set up a locally created symlink inode for immediate write to the cache.
+ */
+void afs_init_new_symlink(struct afs_vnode *vnode, struct afs_operation *op)
+{
+ struct afs_symlink *symlink = op->create.symlink;
+ size_t dsize = 0;
+ size_t size = strlen(symlink->content) + 1;
+ char *p;
+
+ rcu_assign_pointer(vnode->symlink, symlink);
+ op->create.symlink = NULL;
+
+ if (!fscache_cookie_enabled(netfs_i_cookie(&vnode->netfs)))
+ return;
+
+ if (netfs_alloc_folioq_buffer(NULL, &vnode->directory, &dsize, size,
+ mapping_gfp_mask(vnode->netfs.inode.i_mapping)) < 0)
+ return;
+
+ vnode->directory_size = dsize;
+ p = kmap_local_folio(folioq_folio(vnode->directory, 0), 0);
+ memcpy(p, symlink->content, size);
+ kunmap_local(p);
+ netfs_single_mark_inode_dirty(&vnode->netfs.inode);
+}
+
+/*
+ * Read a symlink in a single download.
+ */
+static ssize_t afs_do_read_symlink(struct afs_vnode *vnode)
+{
+ struct afs_symlink *symlink;
+ struct iov_iter iter;
+ ssize_t ret;
+ loff_t i_size;
+
+ i_size = i_size_read(&vnode->netfs.inode);
+ if (i_size > PAGE_SIZE - 1) {
+ trace_afs_file_error(vnode, -EFBIG, afs_file_error_dir_big);
+ return -EFBIG;
+ }
+
+ if (!vnode->directory) {
+ size_t cur_size = 0;
+
+ ret = netfs_alloc_folioq_buffer(NULL,
+ &vnode->directory, &cur_size, PAGE_SIZE,
+ mapping_gfp_mask(vnode->netfs.inode.i_mapping));
+ vnode->directory_size = PAGE_SIZE - 1;
+ if (ret < 0)
+ return ret;
+ }
+
+ iov_iter_folio_queue(&iter, ITER_DEST, vnode->directory, 0, 0, PAGE_SIZE);
+
+ /* AFS requires us to perform the read of a symlink as a single unit to
+ * avoid issues with the content being changed between reads.
+ */
+ ret = netfs_read_single(&vnode->netfs.inode, NULL, &iter);
+ if (ret >= 0) {
+ i_size = ret;
+ if (i_size > PAGE_SIZE - 1) {
+ trace_afs_file_error(vnode, -EFBIG, afs_file_error_dir_big);
+ return -EFBIG;
+ }
+ vnode->directory_size = i_size;
+
+ /* Copy the symlink. */
+ symlink = kmalloc_flex(struct afs_symlink, content, i_size + 1,
+ GFP_KERNEL);
+ if (!symlink)
+ return -ENOMEM;
+
+ refcount_set(&symlink->ref, 1);
+ symlink->content[i_size] = 0;
+
+ const char *s = kmap_local_folio(folioq_folio(vnode->directory, 0), 0);
+
+ memcpy(symlink->content, s, i_size);
+ kunmap_local(s);
+
+ afs_replace_symlink(vnode, symlink);
+ }
+
+ if (!fscache_cookie_enabled(netfs_i_cookie(&vnode->netfs))) {
+ netfs_free_folioq_buffer(vnode->directory);
+ vnode->directory = NULL;
+ vnode->directory_size = 0;
+ }
+
+ return ret;
+}
+
+static ssize_t afs_read_symlink(struct afs_vnode *vnode)
+{
+ ssize_t ret;
+
+ fscache_use_cookie(afs_vnode_cache(vnode), false);
+ ret = afs_do_read_symlink(vnode);
+ fscache_unuse_cookie(afs_vnode_cache(vnode), NULL, NULL);
+ return ret;
+}
+
+static void afs_put_link(void *arg)
+{
+ afs_put_symlink(arg);
+}
+
+const char *afs_get_link(struct dentry *dentry, struct inode *inode,
+ struct delayed_call *callback)
+{
+ struct afs_symlink *symlink;
+ struct afs_vnode *vnode = AFS_FS_I(inode);
+ ssize_t ret;
+
+ if (!dentry) {
+ /* RCU pathwalk. */
+ symlink = rcu_dereference(vnode->symlink);
+ if (!symlink || !afs_check_validity(vnode))
+ return ERR_PTR(-ECHILD);
+ set_delayed_call(callback, NULL, NULL);
+ return symlink->content;
+ }
+
+ if (vnode->symlink) {
+ ret = afs_validate(vnode, NULL);
+ if (ret < 0)
+ return ERR_PTR(ret);
+
+ down_read(&vnode->validate_lock);
+ if (vnode->symlink)
+ goto good;
+ up_read(&vnode->validate_lock);
+ }
+
+ if (down_write_killable(&vnode->validate_lock) < 0)
+ return ERR_PTR(-ERESTARTSYS);
+ if (!vnode->symlink) {
+ ret = afs_read_symlink(vnode);
+ if (ret < 0) {
+ up_write(&vnode->validate_lock);
+ return ERR_PTR(ret);
+ }
+ }
+
+ downgrade_write(&vnode->validate_lock);
+
+good:
+ symlink = rcu_dereference_protected(vnode->symlink,
+ lockdep_is_held(&vnode->validate_lock));
+ refcount_inc(&symlink->ref);
+ up_read(&vnode->validate_lock);
+
+ set_delayed_call(callback, afs_put_link, symlink);
+ return symlink->content;
+}
+
+int afs_readlink(struct dentry *dentry, char __user *buffer, int buflen)
+{
+ DEFINE_DELAYED_CALL(done);
+ const char *content;
+ int len;
+
+ content = afs_get_link(dentry, d_inode(dentry), &done);
+ if (IS_ERR(content)) {
+ do_delayed_call(&done);
+ return PTR_ERR(content);
+ }
+
+ len = umin(strlen(content), buflen);
+ if (copy_to_user(buffer, content, len))
+ len = -EFAULT;
+ do_delayed_call(&done);
+ return len;
+}
+
+/*
+ * Write the symlink contents to the cache as a single blob. We then throw
+ * away the page we used to receive it.
+ */
+int afs_symlink_writepages(struct address_space *mapping,
+ struct writeback_control *wbc)
+{
+ struct afs_vnode *vnode = AFS_FS_I(mapping->host);
+ struct iov_iter iter;
+ int ret = 0;
+
+ if (!down_read_trylock(&vnode->validate_lock)) {
+ if (wbc->sync_mode == WB_SYNC_NONE) {
+ /* The VFS will have undirtied the inode. */
+ netfs_single_mark_inode_dirty(&vnode->netfs.inode);
+ return 0;
+ }
+ down_read(&vnode->validate_lock);
+ }
+
+ if (vnode->directory &&
+ atomic64_read(&vnode->cb_expires_at) != AFS_NO_CB_PROMISE) {
+ iov_iter_folio_queue(&iter, ITER_SOURCE, vnode->directory, 0, 0,
+ i_size_read(&vnode->netfs.inode));
+ ret = netfs_writeback_single(mapping, wbc, &iter);
+ }
+
+ if (ret == 0) {
+ mutex_lock(&vnode->netfs.wb_lock);
+ netfs_free_folioq_buffer(vnode->directory);
+ vnode->directory = NULL;
+ vnode->directory_size = 0;
+ mutex_unlock(&vnode->netfs.wb_lock);
+ } else if (ret == 1) {
+ ret = 0; /* Skipped write due to lock conflict. */
+ }
+
+ up_read(&vnode->validate_lock);
+ return ret;
+}
+
+const struct inode_operations afs_symlink_inode_operations = {
+ .get_link = afs_get_link,
+ .readlink = afs_readlink,
+};
+
+const struct address_space_operations afs_symlink_aops = {
+ .writepages = afs_symlink_writepages,
+};
diff --git a/fs/afs/validation.c b/fs/afs/validation.c
index 0ba8336c9025..e997563af658 100644
--- a/fs/afs/validation.c
+++ b/fs/afs/validation.c
@@ -465,11 +465,17 @@ int afs_validate(struct afs_vnode *vnode, struct key *key)
vnode->cb_ro_snapshot = cb_ro_snapshot;
vnode->cb_scrub = cb_scrub;
- /* if the vnode's data version number changed then its contents are
- * different */
+ /* If the vnode's data version number changed then its contents are
+ * different. Note that afs_apply_status() doesn't set ZAP_DATA on
+ * directories.
+ */
zap |= test_and_clear_bit(AFS_VNODE_ZAP_DATA, &vnode->flags);
- if (zap)
- afs_zap_data(vnode);
+ if (zap) {
+ if (S_ISREG(vnode->netfs.inode.i_mode))
+ afs_zap_data(vnode);
+ else if (S_ISLNK(vnode->netfs.inode.i_mode))
+ afs_invalidate_symlink(vnode);
+ }
up_write(&vnode->validate_lock);
_leave(" = 0");
return 0;
diff --git a/fs/afs/yfsclient.c b/fs/afs/yfsclient.c
index febf13a49f0b..21fba9c309ee 100644
--- a/fs/afs/yfsclient.c
+++ b/fs/afs/yfsclient.c
@@ -960,7 +960,7 @@ void yfs_fs_symlink(struct afs_operation *op)
_enter("");
- contents_sz = strlen(op->create.symlink);
+ contents_sz = strlen(op->create.symlink->content);
call = afs_alloc_flat_call(op->net, &yfs_RXYFSSymlink,
sizeof(__be32) +
sizeof(struct yfs_xdr_RPCFlags) +
@@ -981,7 +981,7 @@ void yfs_fs_symlink(struct afs_operation *op)
bp = xdr_encode_u32(bp, 0); /* RPC flags */
bp = xdr_encode_YFSFid(bp, &dvp->fid);
bp = xdr_encode_name(bp, name);
- bp = xdr_encode_string(bp, op->create.symlink, contents_sz);
+ bp = xdr_encode_string(bp, op->create.symlink->content, contents_sz);
bp = xdr_encode_YFSStoreStatus(bp, &mode, &op->mtime);
yfs_check_req(call, bp);
diff --git a/fs/btrfs/acl.c b/fs/btrfs/acl.c
index e0ba00d64ea0..76d47e7d35ee 100644
--- a/fs/btrfs/acl.c
+++ b/fs/btrfs/acl.c
@@ -14,12 +14,14 @@
#include "ctree.h"
#include "xattr.h"
#include "acl.h"
+#include "misc.h"
+#include "btrfs_inode.h"
struct posix_acl *btrfs_get_acl(struct inode *inode, int type, bool rcu)
{
int size;
const char *name;
- char *value = NULL;
+ char AUTO_KFREE(value);
struct posix_acl *acl;
if (rcu)
@@ -49,7 +51,6 @@ struct posix_acl *btrfs_get_acl(struct inode *inode, int type, bool rcu)
acl = NULL;
else
acl = ERR_PTR(size);
- kfree(value);
return acl;
}
@@ -59,7 +60,7 @@ int __btrfs_set_acl(struct btrfs_trans_handle *trans, struct inode *inode,
{
int ret, size = 0;
const char *name;
- char *value = NULL;
+ char AUTO_KFREE(value);
switch (type) {
case ACL_TYPE_ACCESS:
@@ -85,28 +86,23 @@ int __btrfs_set_acl(struct btrfs_trans_handle *trans, struct inode *inode,
nofs_flag = memalloc_nofs_save();
value = kmalloc(size, GFP_KERNEL);
memalloc_nofs_restore(nofs_flag);
- if (!value) {
- ret = -ENOMEM;
- goto out;
- }
+ if (!value)
+ return -ENOMEM;
ret = posix_acl_to_xattr(&init_user_ns, acl, value, size);
if (ret < 0)
- goto out;
+ return ret;
}
if (trans)
ret = btrfs_setxattr(trans, inode, name, value, size, 0);
else
ret = btrfs_setxattr_trans(inode, name, value, size, 0);
+ if (ret < 0)
+ return ret;
-out:
- kfree(value);
-
- if (!ret)
- set_cached_acl(inode, type, acl);
-
- return ret;
+ set_cached_acl(inode, type, acl);
+ return 0;
}
int btrfs_set_acl(struct mnt_idmap *idmap, struct dentry *dentry,
@@ -116,6 +112,9 @@ int btrfs_set_acl(struct mnt_idmap *idmap, struct dentry *dentry,
struct inode *inode = d_inode(dentry);
umode_t old_mode = inode->i_mode;
+ if (btrfs_root_readonly(BTRFS_I(inode)->root))
+ return -EROFS;
+
if (type == ACL_TYPE_ACCESS && acl) {
ret = posix_acl_update_mode(idmap, inode,
&inode->i_mode, &acl);
diff --git a/fs/btrfs/bio.c b/fs/btrfs/bio.c
index afe022a3b8fc..d13a14314922 100644
--- a/fs/btrfs/bio.c
+++ b/fs/btrfs/bio.c
@@ -902,7 +902,6 @@ int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 fileoff,
struct bio *bio = NULL;
int ret = 0;
- ASSERT(!(fs_info->sb->s_flags & SB_RDONLY));
BUG_ON(!mirror_num);
/* Basic alignment checks. */
@@ -914,6 +913,13 @@ int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 fileoff,
ASSERT(step <= length);
ASSERT(is_power_of_2(step));
+ /*
+ * The fs either mounted RO or hit critical errors, no need
+ * to continue repairing.
+ */
+ if (unlikely(sb_rdonly(fs_info->sb)))
+ return 0;
+
if (btrfs_repair_one_zone(fs_info, logical))
return 0;
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index e9ee10bee2ec..271afe14a686 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -761,7 +761,7 @@ static int load_extent_tree_free(struct btrfs_caching_control *caching_ctl)
nritems = btrfs_header_nritems(leaf);
while (1) {
- if (btrfs_fs_closing(fs_info) > 1) {
+ if (btrfs_fs_closing_done(fs_info)) {
last = (u64)-1;
break;
}
diff --git a/fs/btrfs/block-rsv.c b/fs/btrfs/block-rsv.c
index fc378d2038a2..fbe75b7e9511 100644
--- a/fs/btrfs/block-rsv.c
+++ b/fs/btrfs/block-rsv.c
@@ -324,10 +324,25 @@ void btrfs_block_rsv_add_bytes(struct btrfs_block_rsv *block_rsv,
void btrfs_update_global_block_rsv(struct btrfs_fs_info *fs_info)
{
struct btrfs_block_rsv *block_rsv = &fs_info->global_block_rsv;
- struct btrfs_space_info *sinfo = block_rsv->space_info;
+ struct btrfs_space_info *sinfo;
struct btrfs_root *root, *tmp;
- u64 num_bytes = btrfs_root_used(&fs_info->tree_root->root_item);
unsigned int min_items = 1;
+ u64 num_bytes;
+
+ /*
+ * A full read-only mount (rescue options) cannot start transactions,
+ * so the global reserve is never consumed. Mark it as full and skip
+ * the accounting.
+ */
+ if (btrfs_is_full_ro(fs_info)) {
+ spin_lock(&block_rsv->lock);
+ block_rsv->full = true;
+ spin_unlock(&block_rsv->lock);
+ return;
+ }
+
+ sinfo = block_rsv->space_info;
+ num_bytes = btrfs_root_used(&fs_info->tree_root->root_item);
/*
* The global block rsv is based on the size of the extent tree, the
diff --git a/fs/btrfs/delayed-inode.c b/fs/btrfs/delayed-inode.c
index e1d1f71153fc..382ad94badef 100644
--- a/fs/btrfs/delayed-inode.c
+++ b/fs/btrfs/delayed-inode.c
@@ -670,7 +670,7 @@ static int btrfs_insert_delayed_item(struct btrfs_trans_handle *trans,
struct btrfs_key first_key;
const u32 first_data_size = first_item->data_len;
int total_size;
- char *ins_data = NULL;
+ char AUTO_KFREE(ins_data);
int ret;
bool continuous_keys_only = false;
@@ -742,10 +742,8 @@ static int btrfs_insert_delayed_item(struct btrfs_trans_handle *trans,
ins_data = kmalloc_array(batch.nr,
sizeof(u32) + sizeof(struct btrfs_key), GFP_NOFS);
- if (!ins_data) {
- ret = -ENOMEM;
- goto out;
- }
+ if (!ins_data)
+ return -ENOMEM;
ins_sizes = (u32 *)ins_data;
ins_keys = (struct btrfs_key *)(ins_data + batch.nr * sizeof(u32));
batch.keys = ins_keys;
@@ -761,7 +759,7 @@ static int btrfs_insert_delayed_item(struct btrfs_trans_handle *trans,
ret = btrfs_insert_empty_items(trans, root, path, &batch);
if (ret)
- goto out;
+ return ret;
list_for_each_entry(curr, &item_list, tree_list) {
char *data_ptr;
@@ -816,9 +814,8 @@ static int btrfs_insert_delayed_item(struct btrfs_trans_handle *trans,
list_del(&curr->tree_list);
btrfs_release_delayed_item(curr);
}
-out:
- kfree(ins_data);
- return ret;
+
+ return 0;
}
static int btrfs_insert_delayed_items(struct btrfs_trans_handle *trans,
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 5396f900d5f1..083be891fc99 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -3258,13 +3258,65 @@ int btrfs_check_features(struct btrfs_fs_info *fs_info, bool is_rw_mount)
return 0;
}
-static bool fs_is_full_ro(const struct btrfs_fs_info *fs_info)
+/*
+ * Try to wait for any metadata readahead, and invalidate all btree folios.
+ *
+ * If the invalidation failed, report any dirty/held extent buffers.
+ */
+static void invalidate_and_check_btree_folios(struct btrfs_fs_info *fs_info)
{
- if (!sb_rdonly(fs_info->sb))
- return false;
- if (unlikely(fs_info->mount_opt & BTRFS_MOUNT_FULL_RO_MASK))
- return true;
- return false;
+ unsigned long index = 0;
+ struct extent_buffer *eb;
+ int ret;
+
+ ret = invalidate_inode_pages2(fs_info->btree_inode->i_mapping);
+ if (likely(ret == 0))
+ return;
+
+ /*
+ * Some btree pages can not be invalidated, this happens when some tree
+ * blocks are still held (either by readahead or some task is holding a ref).
+ */
+ rcu_read_lock();
+ xa_for_each(&fs_info->buffer_tree, index, eb) {
+ unsigned int refs;
+
+ /* Increase the ref so that the eb won't disappear. */
+ if (!refcount_inc_not_zero(&eb->refs))
+ continue;
+ rcu_read_unlock();
+
+ /* Wait for any readahead first. */
+ if (test_bit(EXTENT_BUFFER_READING, &eb->bflags))
+ wait_on_bit_io(&eb->bflags, EXTENT_BUFFER_READING,
+ TASK_UNINTERRUPTIBLE);
+ /*
+ * We hold the spinlock to make sure above
+ * EXTENT_BUFFER_READING flag is cleared with the held
+ * ref dropped.
+ * Or we can hit a race window and lead to false alerts.
+ */
+ spin_lock(&eb->refs_lock);
+ refs = refcount_read(&eb->refs);
+ spin_unlock(&eb->refs_lock);
+
+ /*
+ * The refs threshold is 2, one held by us at the beginning
+ * of the loop, one for the ownership in the buffer tree.
+ */
+ if (unlikely(refs > 2 || extent_buffer_under_io(eb))) {
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_BTRFS_DEBUG));
+ btrfs_warn(fs_info,
+ "unable to release extent buffer %llu owner %llu gen %llu refs %u flags 0x%lx",
+ eb->start, btrfs_header_owner(eb),
+ btrfs_header_generation(eb),
+ refs, eb->bflags);
+ }
+ free_extent_buffer(eb);
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+ invalidate_inode_pages2(fs_info->btree_inode->i_mapping);
}
int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_devices)
@@ -3376,7 +3428,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
WRITE_ONCE(fs_info->fs_error, -EUCLEAN);
/* If the fs has any rescue options, no transaction is allowed. */
- if (fs_is_full_ro(fs_info))
+ if (btrfs_is_full_ro(fs_info))
WRITE_ONCE(fs_info->fs_error, -EROFS);
/* Set up fs_info before parsing mount options */
@@ -3682,7 +3734,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
if (fs_info->data_reloc_root)
btrfs_drop_and_free_fs_root(fs_info, fs_info->data_reloc_root);
free_root_pointers(fs_info, true);
- invalidate_inode_pages2(fs_info->btree_inode->i_mapping);
+ invalidate_and_check_btree_folios(fs_info);
fail_sb_buffer:
btrfs_stop_all_workers(fs_info);
@@ -4412,13 +4464,6 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
btrfs_put_block_group_cache(fs_info);
- /*
- * we must make sure there is not any read request to
- * submit after we stopping all workers.
- */
- invalidate_inode_pages2(fs_info->btree_inode->i_mapping);
- btrfs_stop_all_workers(fs_info);
-
/* We shouldn't have any transaction open at this point */
warn_about_uncommitted_trans(fs_info);
@@ -4426,6 +4471,13 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
free_root_pointers(fs_info, true);
btrfs_free_fs_roots(fs_info);
+ /*
+ * We must make sure there is not any read request to
+ * submit after we stop all workers.
+ */
+ invalidate_and_check_btree_folios(fs_info);
+ btrfs_stop_all_workers(fs_info);
+
/*
* We must free the block groups after dropping the fs_roots as we could
* have had an IO error and have left over tree log blocks that aren't
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 7c199d59a546..95bff1616f4a 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -1760,32 +1760,36 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans,
struct btrfs_delayed_extent_op *extent_op,
bool insert_reserved)
{
+ struct btrfs_fs_info *fs_info = trans->fs_info;
int ret = 0;
if (TRANS_ABORTED(trans)) {
if (insert_reserved) {
- btrfs_pin_extent(trans, node->bytenr, node->num_bytes, 1);
- free_head_ref_squota_rsv(trans->fs_info, href);
+ btrfs_pin_extent(trans, node->bytenr, node->num_bytes);
+ free_head_ref_squota_rsv(fs_info, href);
}
return 0;
}
if (node->type == BTRFS_TREE_BLOCK_REF_KEY ||
- node->type == BTRFS_SHARED_BLOCK_REF_KEY)
+ node->type == BTRFS_SHARED_BLOCK_REF_KEY) {
ret = run_delayed_tree_ref(trans, href, node, extent_op,
insert_reserved);
- else if (node->type == BTRFS_EXTENT_DATA_REF_KEY ||
- node->type == BTRFS_SHARED_DATA_REF_KEY)
+ } else if (node->type == BTRFS_EXTENT_DATA_REF_KEY ||
+ node->type == BTRFS_SHARED_DATA_REF_KEY) {
ret = run_delayed_data_ref(trans, href, node, extent_op,
insert_reserved);
- else if (node->type == BTRFS_EXTENT_OWNER_REF_KEY)
+ } else if (node->type == BTRFS_EXTENT_OWNER_REF_KEY) {
ret = 0;
- else
- BUG();
+ } else {
+ ret = -EUCLEAN;
+ btrfs_err(fs_info, "unexpected delayed ref node type: %u", node->type);
+ }
+
if (ret && insert_reserved)
- btrfs_pin_extent(trans, node->bytenr, node->num_bytes, 1);
+ btrfs_pin_extent(trans, node->bytenr, node->num_bytes);
if (ret < 0)
- btrfs_err(trans->fs_info,
+ btrfs_err(fs_info,
"failed to run delayed ref for logical %llu num_bytes %llu type %u action %u ref_mod %d: %d",
node->bytenr, node->num_bytes, node->type,
node->action, node->ref_mod, ret);
@@ -1890,7 +1894,7 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans,
spin_unlock(&delayed_refs->lock);
if (head->must_insert_reserved) {
- btrfs_pin_extent(trans, head->bytenr, head->num_bytes, 1);
+ btrfs_pin_extent(trans, head->bytenr, head->num_bytes);
if (head->is_data) {
struct btrfs_root *csum_root;
@@ -2607,15 +2611,14 @@ static int pin_down_extent(struct btrfs_trans_handle *trans,
return 0;
}
-int btrfs_pin_extent(struct btrfs_trans_handle *trans,
- u64 bytenr, u64 num_bytes, int reserved)
+int btrfs_pin_extent(struct btrfs_trans_handle *trans, u64 bytenr, u64 num_bytes)
{
struct btrfs_block_group *cache;
cache = btrfs_lookup_block_group(trans->fs_info, bytenr);
BUG_ON(!cache); /* Logic error */
- pin_down_extent(trans, cache, bytenr, num_bytes, reserved);
+ pin_down_extent(trans, cache, bytenr, num_bytes, 1);
btrfs_put_block_group(cache);
return 0;
@@ -3524,7 +3527,7 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans, struct btrfs_ref *ref)
* tree, just update pinning info and exit early.
*/
if (ref->ref_root == BTRFS_TREE_LOG_OBJECTID) {
- btrfs_pin_extent(trans, ref->bytenr, ref->num_bytes, 1);
+ btrfs_pin_extent(trans, ref->bytenr, ref->num_bytes);
ret = 0;
} else if (ref->type == BTRFS_REF_METADATA) {
ret = btrfs_add_delayed_tree_ref(trans, ref, NULL);
@@ -5007,7 +5010,7 @@ int btrfs_alloc_logged_file_extent(struct btrfs_trans_handle *trans,
ret = alloc_reserved_file_extent(trans, 0, root_objectid, 0, owner,
offset, ins, 1, root_objectid);
if (ret)
- btrfs_pin_extent(trans, ins->objectid, ins->offset, 1);
+ btrfs_pin_extent(trans, ins->objectid, ins->offset);
ret = btrfs_record_squota_delta(fs_info, &delta);
btrfs_put_block_group(block_group);
return ret;
@@ -6046,7 +6049,7 @@ int btrfs_drop_snapshot(struct btrfs_root *root, bool update_ref, bool for_reloc
struct btrfs_trans_handle *trans;
struct btrfs_root *tree_root = fs_info->tree_root;
struct btrfs_root_item *root_item = &root->root_item;
- struct walk_control *wc;
+ struct walk_control AUTO_KFREE(wc);
struct btrfs_key key;
const u64 rootid = btrfs_root_id(root);
int ret = 0;
@@ -6064,9 +6067,8 @@ int btrfs_drop_snapshot(struct btrfs_root *root, bool update_ref, bool for_reloc
wc = kzalloc(sizeof(*wc), GFP_NOFS);
if (!wc) {
- btrfs_free_path(path);
ret = -ENOMEM;
- goto out;
+ goto out_free;
}
/*
@@ -6276,7 +6278,6 @@ int btrfs_drop_snapshot(struct btrfs_root *root, bool update_ref, bool for_reloc
btrfs_end_transaction_throttle(trans);
out_free:
- kfree(wc);
btrfs_free_path(path);
out:
if (!ret && root_dropped) {
@@ -6319,7 +6320,7 @@ int btrfs_drop_subtree(struct btrfs_trans_handle *trans,
{
struct btrfs_fs_info *fs_info = root->fs_info;
BTRFS_PATH_AUTO_FREE(path);
- struct walk_control *wc;
+ struct walk_control AUTO_KFREE(wc);
int level;
int parent_level;
int ret = 0;
@@ -6358,18 +6359,17 @@ int btrfs_drop_subtree(struct btrfs_trans_handle *trans,
while (1) {
ret = walk_down_tree(trans, root, path, wc);
if (ret < 0)
- break;
+ return ret;
ret = walk_up_tree(trans, root, path, wc, parent_level);
if (ret) {
- if (ret > 0)
- ret = 0;
+ if (ret < 0)
+ return ret;
break;
}
}
- kfree(wc);
- return ret;
+ return 0;
}
/*
diff --git a/fs/btrfs/extent-tree.h b/fs/btrfs/extent-tree.h
index e970ac42a871..e573509c5a71 100644
--- a/fs/btrfs/extent-tree.h
+++ b/fs/btrfs/extent-tree.h
@@ -110,8 +110,7 @@ int btrfs_lookup_extent_info(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info, u64 bytenr,
u64 offset, int metadata, u64 *refs, u64 *flags,
u64 *owner_root);
-int btrfs_pin_extent(struct btrfs_trans_handle *trans, u64 bytenr, u64 num,
- int reserved);
+int btrfs_pin_extent(struct btrfs_trans_handle *trans, u64 bytenr, u64 num);
int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,
const struct extent_buffer *eb);
int btrfs_exclude_logged_extents(struct extent_buffer *eb);
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 90a6b161d7c1..f808134f60f5 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -2316,6 +2316,76 @@ void btrfs_btree_wait_writeback_range(struct btrfs_fs_info *fs_info, u64 start,
}
}
+static int write_meta_extent_buffer(struct btrfs_eb_write_context *ctx,
+ struct writeback_control *wbc)
+{
+ struct extent_buffer *eb = ctx->eb;
+ int ret;
+
+ ret = btrfs_check_meta_write_pointer(eb->fs_info, ctx);
+ if (ret)
+ return ret;
+
+ if (!lock_extent_buffer_for_io(eb, wbc))
+ return 0;
+
+ /* Implies write in zoned mode. */
+ if (ctx->zoned_bg) {
+ /* Mark the last eb in the block group. */
+ btrfs_schedule_zone_finish_bg(ctx->zoned_bg, eb);
+ ctx->zoned_bg->meta_write_pointer += eb->len;
+ }
+ write_one_eb(eb, wbc);
+ return 0;
+}
+
+/*
+ * On a zoned filesystem, write out the currently dirty metadata extent buffers
+ * of @bg. Used to flush the active metadata/system block group before the
+ * ascending-address walk in btree_writepages(), so that walk can pivot the
+ * active block group away (finishing it) instead of aborting the commit; see
+ * the caller for details.
+ */
+static void flush_active_meta_bg(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct btrfs_eb_write_context *ctx,
+ struct btrfs_block_group *bg)
+{
+ struct btrfs_fs_info *fs_info = inode_to_fs_info(mapping->host);
+ unsigned long index = bg->start >> fs_info->nodesize_bits;
+ unsigned long end = (btrfs_block_group_end(bg) - 1) >> fs_info->nodesize_bits;
+ struct eb_batch batch;
+ unsigned int nr_ebs;
+
+ ASSERT(btrfs_is_zoned(fs_info));
+ lockdep_assert_held(&fs_info->zoned_meta_io_lock);
+
+ eb_batch_init(&batch);
+ while (index <= end &&
+ (nr_ebs = buffer_tree_get_ebs_tag(fs_info, &index, end,
+ PAGECACHE_TAG_DIRTY, &batch))) {
+ struct extent_buffer *eb;
+
+ while ((eb = eb_batch_next(&batch)) != NULL) {
+ ctx->eb = eb;
+
+ /*
+ * If the eb is behind the write pointer (-EBUSY, e.g.
+ * already being written by someone else) skip it and
+ * carry on. Only a hole at the write pointer (-EAGAIN)
+ * stops the flush. The main walk in btree_writepages()
+ * then deals with it.
+ */
+ if (write_meta_extent_buffer(ctx, wbc) == -EAGAIN) {
+ eb_batch_release(&batch);
+ return;
+ }
+ }
+ eb_batch_release(&batch);
+ cond_resched();
+ }
+}
+
int btree_writepages(struct address_space *mapping, struct writeback_control *wbc)
{
struct btrfs_eb_write_context ctx = { .wbc = wbc };
@@ -2351,6 +2421,22 @@ int btree_writepages(struct address_space *mapping, struct writeback_control *wb
else
tag = PAGECACHE_TAG_DIRTY;
btrfs_zoned_meta_io_lock(fs_info);
+
+ /*
+ * On a zoned filesystem, flush the currently active metadata/system
+ * block group(s) first, under this same lock, so the ascending-address
+ * walk below can pivot the active block group instead of aborting the
+ * transaction commit with -EAGAIN.
+ */
+ if (btrfs_is_zoned(fs_info) && wbc->sync_mode == WB_SYNC_ALL &&
+ !wbc->for_sync) {
+ if (fs_info->active_meta_bg)
+ flush_active_meta_bg(mapping, wbc, &ctx,
+ fs_info->active_meta_bg);
+ if (fs_info->active_system_bg)
+ flush_active_meta_bg(mapping, wbc, &ctx,
+ fs_info->active_system_bg);
+ }
retry:
if (wbc->sync_mode == WB_SYNC_ALL)
buffer_tree_tag_for_writeback(fs_info, index, end);
@@ -2361,28 +2447,13 @@ int btree_writepages(struct address_space *mapping, struct writeback_control *wb
while ((eb = eb_batch_next(&batch)) != NULL) {
ctx.eb = eb;
- ret = btrfs_check_meta_write_pointer(eb->fs_info, &ctx);
- if (ret) {
- if (ret == -EBUSY)
- ret = 0;
-
- if (ret) {
- done = 1;
- break;
- }
- continue;
- }
-
- if (!lock_extent_buffer_for_io(eb, wbc))
- continue;
-
- /* Implies write in zoned mode. */
- if (ctx.zoned_bg) {
- /* Mark the last eb in the block group. */
- btrfs_schedule_zone_finish_bg(ctx.zoned_bg, eb);
- ctx.zoned_bg->meta_write_pointer += eb->len;
+ ret = write_meta_extent_buffer(&ctx, wbc);
+ if (ret == -EBUSY) {
+ ret = 0;
+ } else if (ret) {
+ done = 1;
+ break;
}
- write_one_eb(eb, wbc);
}
nr_to_write_done = (wbc->nr_to_write <= 0);
eb_batch_release(&batch);
@@ -2906,12 +2977,6 @@ bool try_release_extent_mapping(struct folio *folio, gfp_t mask)
return try_release_extent_state(io_tree, folio);
}
-static int extent_buffer_under_io(const struct extent_buffer *eb)
-{
- return (test_bit(EXTENT_BUFFER_WRITEBACK, &eb->bflags) ||
- test_bit(EXTENT_BUFFER_DIRTY, &eb->bflags));
-}
-
static bool folio_range_has_eb(struct folio *folio)
{
struct btrfs_folio_state *bfs;
@@ -3661,12 +3726,31 @@ static int release_extent_buffer(struct extent_buffer *eb)
return 0;
}
-void free_extent_buffer(struct extent_buffer *eb)
+static void clear_extent_buffer_reading(struct extent_buffer *eb)
+{
+ clear_and_wake_up_bit(EXTENT_BUFFER_READING, &eb->bflags);
+}
+
+static void free_extent_buffer_clear_reading(struct extent_buffer *eb,
+ bool clear_reading)
{
int refs;
+
if (!eb)
return;
+ /*
+ * We want to clear EXTENT_BUFFER_READING flag and decrease refs
+ * in the same critical section.
+ * This will make sure invalidate_and_check_btree_folios() won't
+ * see an eb with EXTENT_BUFFER_READING cleared but refs not yet
+ * decreased.
+ */
+ if (clear_reading) {
+ spin_lock(&eb->refs_lock);
+ clear_extent_buffer_reading(eb);
+ }
+
refs = refcount_read(&eb->refs);
while (1) {
if (test_bit(EXTENT_BUFFER_UNMAPPED, &eb->bflags)) {
@@ -3677,11 +3761,16 @@ void free_extent_buffer(struct extent_buffer *eb)
}
/* Optimization to avoid locking eb->refs_lock. */
- if (atomic_try_cmpxchg(&eb->refs.refs, &refs, refs - 1))
+ if (atomic_try_cmpxchg(&eb->refs.refs, &refs, refs - 1)) {
+ if (clear_reading)
+ spin_unlock(&eb->refs_lock);
return;
+ }
}
- spin_lock(&eb->refs_lock);
+ if (!clear_reading)
+ spin_lock(&eb->refs_lock);
+
if (refcount_read(&eb->refs) == 2 &&
test_bit(EXTENT_BUFFER_STALE, &eb->bflags) &&
!extent_buffer_under_io(eb) &&
@@ -3695,6 +3784,11 @@ void free_extent_buffer(struct extent_buffer *eb)
release_extent_buffer(eb);
}
+void free_extent_buffer(struct extent_buffer *eb)
+{
+ return free_extent_buffer_clear_reading(eb, false);
+}
+
void free_extent_buffer_stale(struct extent_buffer *eb)
{
if (!eb)
@@ -3831,11 +3925,6 @@ void set_extent_buffer_uptodate(struct extent_buffer *eb)
btrfs_meta_folio_set_uptodate(eb->folios[i], eb);
}
-static void clear_extent_buffer_reading(struct extent_buffer *eb)
-{
- clear_and_wake_up_bit(EXTENT_BUFFER_READING, &eb->bflags);
-}
-
static void end_bbio_meta_read(struct btrfs_bio *bbio)
{
struct extent_buffer *eb = bbio->private;
@@ -3859,8 +3948,7 @@ static void end_bbio_meta_read(struct btrfs_bio *bbio)
else
clear_extent_buffer_uptodate(eb);
- clear_extent_buffer_reading(eb);
- free_extent_buffer(eb);
+ free_extent_buffer_clear_reading(eb, true);
bio_put(&bbio->bio);
}
diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h
index 65d1ed3fcd5b..68e9fce50521 100644
--- a/fs/btrfs/extent_io.h
+++ b/fs/btrfs/extent_io.h
@@ -324,6 +324,12 @@ static inline int extent_buffer_uptodate(const struct extent_buffer *eb)
return test_bit(EXTENT_BUFFER_UPTODATE, &eb->bflags);
}
+static inline bool extent_buffer_under_io(const struct extent_buffer *eb)
+{
+ return (test_bit(EXTENT_BUFFER_WRITEBACK, &eb->bflags) ||
+ test_bit(EXTENT_BUFFER_DIRTY, &eb->bflags));
+}
+
int memcmp_extent_buffer(const struct extent_buffer *eb, const void *ptrv,
unsigned long start, unsigned long len);
void read_extent_buffer(const struct extent_buffer *eb, void *dst,
diff --git a/fs/btrfs/fs.h b/fs/btrfs/fs.h
index eccc61463947..8687cf6952d0 100644
--- a/fs/btrfs/fs.h
+++ b/fs/btrfs/fs.h
@@ -1096,15 +1096,26 @@ void __btrfs_clear_fs_compat_ro(struct btrfs_fs_info *fs_info, u64 flag,
#define btrfs_test_opt(fs_info, opt) ((fs_info)->mount_opt & \
BTRFS_MOUNT_##opt)
-static inline int btrfs_fs_closing(const struct btrfs_fs_info *fs_info)
+static inline bool btrfs_is_full_ro(const struct btrfs_fs_info *fs_info)
{
- /* Do it this way so we only ever do one test_bit in the normal case. */
- if (test_bit(BTRFS_FS_CLOSING_START, &fs_info->flags)) {
- if (test_bit(BTRFS_FS_CLOSING_DONE, &fs_info->flags))
- return 2;
- return 1;
- }
- return 0;
+ if (!sb_rdonly(fs_info->sb))
+ return false;
+ if (unlikely(fs_info->mount_opt & BTRFS_MOUNT_FULL_RO_MASK))
+ return true;
+ return false;
+}
+
+static inline bool btrfs_fs_closing(const struct btrfs_fs_info *fs_info)
+{
+ return unlikely(test_bit(BTRFS_FS_CLOSING_START, &fs_info->flags));
+}
+
+static inline bool btrfs_fs_closing_done(const struct btrfs_fs_info *fs_info)
+{
+ if (btrfs_fs_closing(fs_info) && test_bit(BTRFS_FS_CLOSING_DONE, &fs_info->flags))
+ return true;
+
+ return false;
}
/*
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index c63b2f5edae9..62858d8d66bd 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -9499,7 +9499,6 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,
struct completion sync_reads;
unsigned long i = 0;
struct btrfs_bio *bbio;
- int ret;
/*
* Fast path for synchronous reads which completes in this call, io_uring
@@ -9546,10 +9545,10 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,
if (uring_ctx) {
if (refcount_dec_and_test(&priv->pending_refs)) {
- ret = blk_status_to_errno(READ_ONCE(priv->status));
- btrfs_uring_read_extent_endio(uring_ctx, ret);
+ int error = blk_status_to_errno(READ_ONCE(priv->status));
+
+ btrfs_uring_read_extent_endio(uring_ctx, error);
kfree(priv);
- return ret;
}
return -EIOCBQUEUED;
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 45dbfa8cce96..9a8902012ae6 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -510,7 +510,7 @@ static noinline int create_subvol(struct mnt_idmap *idmap,
struct btrfs_fs_info *fs_info = inode_to_fs_info(dir);
struct btrfs_trans_handle *trans;
struct btrfs_key key;
- struct btrfs_root_item *root_item;
+ struct btrfs_root_item AUTO_KFREE(root_item);
struct btrfs_inode_item *inode_item;
struct extent_buffer *leaf;
struct btrfs_root *root = BTRFS_I(dir)->root;
@@ -534,20 +534,18 @@ static noinline int create_subvol(struct mnt_idmap *idmap,
ret = btrfs_get_free_objectid(fs_info->tree_root, &objectid);
if (ret)
- goto out_root_item;
+ return ret;
/*
* Don't create subvolume whose level is not zero. Or qgroup will be
* screwed up since it assumes subvolume qgroup's level to be 0.
*/
- if (btrfs_qgroup_level(objectid)) {
- ret = -ENOSPC;
- goto out_root_item;
- }
+ if (btrfs_qgroup_level(objectid))
+ return -ENOSPC;
ret = get_anon_bdev(&anon_dev);
if (ret < 0)
- goto out_root_item;
+ return ret;
new_inode_args.inode = btrfs_new_subvol_inode(idmap, dir);
if (!new_inode_args.inode) {
@@ -706,8 +704,7 @@ static noinline int create_subvol(struct mnt_idmap *idmap,
out_anon_dev:
if (anon_dev)
free_anon_bdev(anon_dev);
-out_root_item:
- kfree(root_item);
+
return ret;
}
@@ -2939,7 +2936,7 @@ static long btrfs_ioctl_space_info(struct btrfs_fs_info *fs_info,
struct btrfs_ioctl_space_args space_args = { 0 };
struct btrfs_ioctl_space_info space;
struct btrfs_ioctl_space_info *dest;
- struct btrfs_ioctl_space_info *dest_orig;
+ struct btrfs_ioctl_space_info AUTO_KFREE(dest_orig);
struct btrfs_ioctl_space_info __user *user_dest;
struct btrfs_space_info *info;
static const u64 types[] = {
@@ -3063,7 +3060,6 @@ static long btrfs_ioctl_space_info(struct btrfs_fs_info *fs_info,
space_args.total_spaces * sizeof(*dest_orig)))
ret = -EFAULT;
- kfree(dest_orig);
out:
if (ret == 0 && copy_to_user(arg, &space_args, sizeof(space_args)))
ret = -EFAULT;
@@ -3595,7 +3591,7 @@ static long btrfs_ioctl_balance_ctl(struct btrfs_fs_info *fs_info, int cmd)
static long btrfs_ioctl_balance_progress(struct btrfs_fs_info *fs_info,
void __user *arg)
{
- struct btrfs_ioctl_balance_args *bargs;
+ struct btrfs_ioctl_balance_args AUTO_KFREE(bargs);
int ret = 0;
if (!capable(CAP_SYS_ADMIN))
@@ -3617,8 +3613,6 @@ static long btrfs_ioctl_balance_progress(struct btrfs_fs_info *fs_info,
if (copy_to_user(arg, bargs, sizeof(*bargs)))
ret = -EFAULT;
-
- kfree(bargs);
out:
mutex_unlock(&fs_info->balance_mutex);
return ret;
@@ -4236,7 +4230,7 @@ static int check_feature_bits(const struct btrfs_fs_info *fs_info,
u64 safe_set, u64 safe_clear)
{
const char *type = btrfs_feature_set_name(set);
- char *names;
+ const char AUTO_KFREE(names);
u64 disallowed, unsupported;
u64 set_mask = flags & change_mask;
u64 clear_mask = ~flags & change_mask;
@@ -4244,12 +4238,11 @@ static int check_feature_bits(const struct btrfs_fs_info *fs_info,
unsupported = set_mask & ~supported_flags;
if (unsupported) {
names = btrfs_printable_features(set, unsupported);
- if (names) {
+ if (names)
btrfs_warn(fs_info,
"this kernel does not support the %s feature bit%s",
names, strchr(names, ',') ? "s" : "");
- kfree(names);
- } else
+ else
btrfs_warn(fs_info,
"this kernel does not support %s bits 0x%llx",
type, unsupported);
@@ -4259,12 +4252,11 @@ static int check_feature_bits(const struct btrfs_fs_info *fs_info,
disallowed = set_mask & ~safe_set;
if (disallowed) {
names = btrfs_printable_features(set, disallowed);
- if (names) {
+ if (names)
btrfs_warn(fs_info,
"can't set the %s feature bit%s while mounted",
names, strchr(names, ',') ? "s" : "");
- kfree(names);
- } else
+ else
btrfs_warn(fs_info,
"can't set %s bits 0x%llx while mounted",
type, disallowed);
@@ -4274,12 +4266,11 @@ static int check_feature_bits(const struct btrfs_fs_info *fs_info,
disallowed = clear_mask & ~safe_clear;
if (disallowed) {
names = btrfs_printable_features(set, disallowed);
- if (names) {
+ if (names)
btrfs_warn(fs_info,
"can't clear the %s feature bit%s while mounted",
names, strchr(names, ',') ? "s" : "");
- kfree(names);
- } else
+ else
btrfs_warn(fs_info,
"can't clear %s bits 0x%llx while mounted",
type, disallowed);
@@ -4668,7 +4659,7 @@ static void btrfs_uring_read_finished(struct io_uring_cmd *cmd, unsigned int iss
size_t page_offset;
ssize_t ret;
- /* The inode lock has already been acquired in btrfs_uring_read_extent. */
+ /* The inode lock has already been acquired in btrfs_encoded_read(). */
btrfs_lockdep_inode_acquire(inode, i_rwsem);
if (priv->err) {
@@ -4734,7 +4725,6 @@ static int btrfs_uring_read_extent(struct kiocb *iocb, struct iov_iter *iter,
struct iovec *iov, struct io_uring_cmd *cmd)
{
struct btrfs_inode *inode = BTRFS_I(file_inode(iocb->ki_filp));
- struct extent_io_tree *io_tree = &inode->io_tree;
struct page **pages = NULL;
struct btrfs_uring_priv *priv = NULL;
unsigned long nr_pages;
@@ -4790,8 +4780,6 @@ static int btrfs_uring_read_extent(struct kiocb *iocb, struct iov_iter *iter,
return -EIOCBQUEUED;
out_fail:
- btrfs_unlock_extent(io_tree, start, lockend, &cached_state);
- btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED);
kfree(priv);
for (int i = 0; i < nr_pages; i++) {
if (pages[i])
@@ -4937,8 +4925,10 @@ static int btrfs_uring_encoded_read(struct io_uring_cmd *cmd, unsigned int issue
cached_state, disk_bytenr, disk_io_size,
count, data->args.compression,
data->iov, cmd);
-
- goto out_acct;
+ if (ret == -EIOCBQUEUED)
+ goto out_acct;
+ btrfs_unlock_extent(io_tree, start, lockend, &cached_state);
+ btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED);
}
out_free:
diff --git a/fs/btrfs/lzo.c b/fs/btrfs/lzo.c
index 4758f66da449..2f8a73999f94 100644
--- a/fs/btrfs/lzo.c
+++ b/fs/btrfs/lzo.c
@@ -409,6 +409,17 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
return -EIO;
}
+ /* The segment must not extend beyond the compressed input. */
+ if (unlikely(cur_in + seg_len > cb->compressed_len)) {
+ struct btrfs_inode *inode = cb->bbio.inode;
+
+ btrfs_err(fs_info,
+ "lzo segment overflows compressed input, root %llu inode %llu offset %llu cur_in %u len %u compressed len %u",
+ btrfs_root_id(inode->root), btrfs_ino(inode),
+ cb->start, cur_in, seg_len, cb->compressed_len);
+ return -EUCLEAN;
+ }
+
/* Copy the compressed segment payload into workspace */
copy_compressed_segment(cb, workspace->cbuf, seg_len, &cur_in);
@@ -458,7 +469,8 @@ int lzo_decompress(struct list_head *ws, const u8 *data_in,
size_t max_segment_len = workspace_buf_length(fs_info);
int ret = 0;
- if (unlikely(srclen < LZO_LEN || srclen > max_segment_len + LZO_LEN * 2))
+ if (unlikely(srclen <= LZO_LEN * 2 ||
+ srclen > max_segment_len + LZO_LEN * 2))
return -EUCLEAN;
in_len = read_compress_length(data_in);
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index f654639334fc..87eddd5a3ea7 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -4860,7 +4860,7 @@ int btrfs_qgroup_trace_subtree_after_cow(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info = root->fs_info;
struct btrfs_tree_parent_check check = { 0 };
struct btrfs_qgroup_swapped_blocks *blocks = &root->swapped_blocks;
- struct btrfs_qgroup_swapped_block *block;
+ struct btrfs_qgroup_swapped_block AUTO_KFREE(block);
struct extent_buffer *reloc_eb = NULL;
struct rb_node *node;
bool swapped = false;
@@ -4917,7 +4917,6 @@ int btrfs_qgroup_trace_subtree_after_cow(struct btrfs_trans_handle *trans,
ret = qgroup_trace_subtree_swap(trans, reloc_eb, subvol_eb,
block->last_snapshot, block->trace_leaf);
free_out:
- kfree(block);
free_extent_buffer(reloc_eb);
out:
if (ret < 0) {
diff --git a/fs/btrfs/raid-stripe-tree.c b/fs/btrfs/raid-stripe-tree.c
index d31b29260047..707fe664b077 100644
--- a/fs/btrfs/raid-stripe-tree.c
+++ b/fs/btrfs/raid-stripe-tree.c
@@ -19,7 +19,7 @@ static int btrfs_partially_delete_raid_extent(struct btrfs_trans_handle *trans,
u64 newlen, u64 frontpad)
{
struct btrfs_root *stripe_root = trans->fs_info->stripe_root;
- struct btrfs_stripe_extent *extent, *newitem;
+ struct btrfs_stripe_extent *extent, AUTO_KFREE(newitem);
struct extent_buffer *leaf;
int slot;
size_t item_size;
@@ -53,14 +53,10 @@ static int btrfs_partially_delete_raid_extent(struct btrfs_trans_handle *trans,
ret = btrfs_del_item(trans, stripe_root, path);
if (ret)
- goto out;
+ return ret;
btrfs_release_path(path);
- ret = btrfs_insert_item(trans, stripe_root, &newkey, newitem, item_size);
-
-out:
- kfree(newitem);
- return ret;
+ return btrfs_insert_item(trans, stripe_root, &newkey, newitem, item_size);
}
int btrfs_delete_raid_extent(struct btrfs_trans_handle *trans, u64 start, u64 length)
@@ -301,7 +297,7 @@ int btrfs_insert_one_raid_extent(struct btrfs_trans_handle *trans,
struct btrfs_key stripe_key;
struct btrfs_root *stripe_root = fs_info->stripe_root;
const int num_stripes = btrfs_bg_type_to_factor(bioc->map_type);
- struct btrfs_stripe_extent *stripe_extent;
+ struct btrfs_stripe_extent AUTO_KFREE(stripe_extent);
const size_t item_size = struct_size(stripe_extent, strides, num_stripes);
int ret;
@@ -337,8 +333,6 @@ int btrfs_insert_one_raid_extent(struct btrfs_trans_handle *trans,
btrfs_abort_transaction(trans, ret);
}
- kfree(stripe_extent);
-
return ret;
}
diff --git a/fs/btrfs/reflink.c b/fs/btrfs/reflink.c
index 8356e3b74cd3..c6f8db5027a4 100644
--- a/fs/btrfs/reflink.c
+++ b/fs/btrfs/reflink.c
@@ -399,7 +399,7 @@ static int btrfs_clone(struct inode *src, struct inode *inode,
BTRFS_PATH_AUTO_FREE(path);
struct extent_buffer *leaf;
struct btrfs_trans_handle *trans;
- char *buf = NULL;
+ char AUTO_KVFREE(buf);
struct btrfs_key key;
u32 nritems;
int slot;
@@ -414,10 +414,8 @@ static int btrfs_clone(struct inode *src, struct inode *inode,
return ret;
path = btrfs_alloc_path();
- if (!path) {
- kvfree(buf);
+ if (!path)
return ret;
- }
path->reada = READA_FORWARD;
/* Clone data */
@@ -667,7 +665,6 @@ static int btrfs_clone(struct inode *src, struct inode *inode,
}
out:
- kvfree(buf);
clear_bit(BTRFS_INODE_NO_DELALLOC_FLUSH, &BTRFS_I(inode)->runtime_flags);
return ret;
diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
index 5f05eb88a4fa..eab9686b1bcd 100644
--- a/fs/btrfs/relocation.c
+++ b/fs/btrfs/relocation.c
@@ -512,7 +512,7 @@ static void __del_reloc_root(struct btrfs_root *root)
{
struct btrfs_fs_info *fs_info = root->fs_info;
struct rb_node *rb_node;
- struct mapping_node *node = NULL;
+ struct mapping_node AUTO_KFREE(node);
struct reloc_control *rc = fs_info->reloc_ctl;
bool put_ref = false;
@@ -545,7 +545,6 @@ static void __del_reloc_root(struct btrfs_root *root)
spin_unlock(&fs_info->trans_lock);
if (put_ref)
btrfs_put_root(root);
- kfree(node);
}
/*
@@ -587,10 +586,9 @@ static struct btrfs_root *create_reloc_root(struct btrfs_trans_handle *trans,
struct btrfs_fs_info *fs_info = root->fs_info;
struct btrfs_root *reloc_root;
struct extent_buffer *eb;
- struct btrfs_root_item *root_item;
+ struct btrfs_root_item AUTO_KFREE(root_item);
struct btrfs_key root_key;
int ret = 0;
- bool must_abort = false;
root_item = kmalloc(sizeof(*root_item), GFP_NOFS);
if (!root_item)
@@ -618,15 +616,14 @@ static struct btrfs_root *create_reloc_root(struct btrfs_trans_handle *trans,
btrfs_err(fs_info,
"cannot relocate partially dropped subvolume %llu, drop progress key " BTRFS_KEY_FMT,
objectid, BTRFS_KEY_FMT_VALUE(&cpu_key));
- ret = -EUCLEAN;
- goto fail;
+ return ERR_PTR(-EUCLEAN);
}
/* called by btrfs_init_reloc_root */
ret = btrfs_copy_root(trans, root, root->commit_root, &eb,
BTRFS_TREE_RELOC_OBJECTID);
if (ret)
- goto fail;
+ return ERR_PTR(ret);
/*
* Set the last_snapshot field to the generation of the commit
@@ -649,14 +646,13 @@ static struct btrfs_root *create_reloc_root(struct btrfs_trans_handle *trans,
ret = btrfs_copy_root(trans, root, root->node, &eb,
BTRFS_TREE_RELOC_OBJECTID);
if (ret)
- goto fail;
+ return ERR_PTR(ret);
}
/*
* We have changed references at this point, we must abort the
- * transaction if anything fails.
+ * transaction if anything fails (i.e. 'goto abort').
*/
- must_abort = true;
memcpy(root_item, &root->root_item, sizeof(*root_item));
btrfs_set_root_bytenr(root_item, eb->start);
@@ -676,9 +672,7 @@ static struct btrfs_root *create_reloc_root(struct btrfs_trans_handle *trans,
ret = btrfs_insert_root(trans, fs_info->tree_root,
&root_key, root_item);
if (ret)
- goto fail;
-
- kfree(root_item);
+ goto abort;
reloc_root = btrfs_read_tree_root(fs_info->tree_root, &root_key);
if (IS_ERR(reloc_root)) {
@@ -688,11 +682,9 @@ static struct btrfs_root *create_reloc_root(struct btrfs_trans_handle *trans,
set_bit(BTRFS_ROOT_SHAREABLE, &reloc_root->state);
btrfs_set_root_last_trans(reloc_root, trans->transid);
return reloc_root;
-fail:
- kfree(root_item);
+
abort:
- if (must_abort)
- btrfs_abort_transaction(trans, ret);
+ btrfs_abort_transaction(trans, ret);
return ERR_PTR(ret);
}
@@ -2972,7 +2964,7 @@ static int relocate_file_extent_cluster(struct reloc_control *rc)
const struct file_extent_cluster *cluster = &rc->cluster;
u64 offset = BTRFS_I(inode)->reloc_block_group_start;
u64 cur_file_offset = cluster->start - offset;
- struct file_ra_state *ra;
+ struct file_ra_state AUTO_KFREE(ra);
int cluster_nr = 0;
int ret = 0;
@@ -2985,13 +2977,13 @@ static int relocate_file_extent_cluster(struct reloc_control *rc)
ret = prealloc_file_extent_cluster(rc);
if (ret)
- goto out;
+ return ret;
file_ra_state_init(ra, inode->i_mapping);
ret = setup_relocation_extent_mapping(rc);
if (ret)
- goto out;
+ return ret;
while (cur_file_offset < cluster->end - offset) {
ret = relocate_one_folio(rc, ra, &cluster_nr, &cur_file_offset);
@@ -3000,8 +2992,6 @@ static int relocate_file_extent_cluster(struct reloc_control *rc)
}
if (ret == 0)
WARN_ON(cluster_nr != cluster->nr);
-out:
- kfree(ra);
return ret;
}
diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index f02188acd048..fdf37653907c 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -551,8 +551,10 @@ static int scrub_print_warning_inode(u64 inum, u64 offset, u64 num_bytes,
}
ret = paths_from_inode(inum, ipath);
- if (ret < 0)
+ if (ret < 0) {
+ btrfs_put_root(local_root);
goto err;
+ }
/*
* we deliberately ignore the bit ipath might have been too small to
diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 22190902be81..9067c3ce00cf 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -2458,7 +2458,7 @@ static int send_subvol_begin(struct send_ctx *sctx)
struct btrfs_key key;
struct btrfs_root_ref *ref;
struct extent_buffer *leaf;
- char *name = NULL;
+ char AUTO_KFREE(name);
int namelen;
path = btrfs_alloc_path();
@@ -2476,18 +2476,15 @@ static int send_subvol_begin(struct send_ctx *sctx)
ret = btrfs_search_slot_for_read(send_root->fs_info->tree_root,
&key, path, 1, 0);
if (ret < 0)
- goto out;
- if (ret) {
- ret = -ENOENT;
- goto out;
- }
+ return ret;
+ if (ret)
+ return -ENOENT;
leaf = path->nodes[0];
btrfs_item_key_to_cpu(leaf, &key, path->slots[0]);
if (key.type != BTRFS_ROOT_BACKREF_KEY ||
key.objectid != btrfs_root_id(send_root)) {
- ret = -ENOENT;
- goto out;
+ return -ENOENT;
}
ref = btrfs_item_ptr(leaf, path->slots[0], struct btrfs_root_ref);
namelen = btrfs_root_ref_name_len(leaf, ref);
@@ -2497,11 +2494,11 @@ static int send_subvol_begin(struct send_ctx *sctx)
if (parent_root) {
ret = begin_cmd(sctx, BTRFS_SEND_C_SNAPSHOT);
if (ret < 0)
- goto out;
+ return ret;
} else {
ret = begin_cmd(sctx, BTRFS_SEND_C_SUBVOL);
if (ret < 0)
- goto out;
+ return ret;
}
TLV_PUT_STRING(sctx, BTRFS_SEND_A_PATH, name, namelen);
@@ -2529,8 +2526,6 @@ static int send_subvol_begin(struct send_ctx *sctx)
ret = send_cmd(sctx);
tlv_put_failure:
-out:
- kfree(name);
return ret;
}
@@ -4077,7 +4072,7 @@ static int update_ref_path(struct send_ctx *sctx, struct recorded_ref *ref)
*/
static int refresh_ref_path(struct send_ctx *sctx, struct recorded_ref *ref)
{
- char *name;
+ char AUTO_KFREE(name);
int ret;
name = kmemdup(ref->name, ref->name_len, GFP_KERNEL);
@@ -4087,17 +4082,16 @@ static int refresh_ref_path(struct send_ctx *sctx, struct recorded_ref *ref)
fs_path_reset(ref->full_path);
ret = get_cur_path(sctx, ref->dir, ref->dir_gen, ref->full_path);
if (ret < 0)
- goto out;
+ return ret;
ret = fs_path_add(ref->full_path, name, ref->name_len);
if (ret < 0)
- goto out;
+ return ret;
/* Update the reference's base name pointer. */
set_ref_path(ref, ref->full_path);
-out:
- kfree(name);
- return ret;
+
+ return 0;
}
static int rbtree_check_dir_ref_comp(const void *k, const struct rb_node *node)
@@ -5006,8 +5000,8 @@ static int __process_changed_new_xattr(int num, struct btrfs_key *di_key,
{
int ret;
struct send_ctx *sctx = ctx;
- char *found_data = NULL;
- int found_data_len = 0;
+ char AUTO_KFREE(found_data);
+ int found_data_len = 0;
ret = find_xattr(sctx->parent_root, sctx->right_path,
sctx->cmp_key, name, name_len, &found_data,
@@ -5025,7 +5019,6 @@ static int __process_changed_new_xattr(int num, struct btrfs_key *di_key,
}
}
- kfree(found_data);
return ret;
}
@@ -5762,7 +5755,7 @@ static int send_capabilities(struct send_ctx *sctx)
struct btrfs_dir_item *di;
struct extent_buffer *leaf;
unsigned long data_ptr;
- char *buf = NULL;
+ char AUTO_KFREE(buf);
int buf_len;
int ret = 0;
@@ -5774,28 +5767,23 @@ static int send_capabilities(struct send_ctx *sctx)
XATTR_NAME_CAPS, strlen(XATTR_NAME_CAPS), 0);
if (!di) {
/* There is no xattr for this inode */
- goto out;
+ return 0;
} else if (IS_ERR(di)) {
- ret = PTR_ERR(di);
- goto out;
+ return PTR_ERR(di);
}
leaf = path->nodes[0];
buf_len = btrfs_dir_data_len(leaf, di);
buf = kmalloc(buf_len, GFP_KERNEL);
- if (!buf) {
- ret = -ENOMEM;
- goto out;
- }
+ if (!buf)
+ return -ENOMEM;
data_ptr = (unsigned long)(di + 1) + btrfs_dir_name_len(leaf, di);
read_extent_buffer(leaf, buf, data_ptr, buf_len);
ret = send_set_xattr(sctx, XATTR_NAME_CAPS,
strlen(XATTR_NAME_CAPS), buf, buf_len);
-out:
- kfree(buf);
return ret;
}
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 105f21759c99..6cc4fe2ccd8a 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -1610,7 +1610,7 @@ static inline void btrfs_descending_sort_devices(
static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info,
u64 *free_bytes)
{
- struct btrfs_device_info *devices_info;
+ struct btrfs_device_info AUTO_KFREE(devices_info);
struct btrfs_fs_devices *fs_devices = fs_info->fs_devices;
struct btrfs_device *device;
u64 type;
@@ -1708,7 +1708,6 @@ static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info,
nr_devices--;
}
- kfree(devices_info);
*free_bytes = avail_space;
return 0;
}
diff --git a/fs/btrfs/tests/extent-io-tests.c b/fs/btrfs/tests/extent-io-tests.c
index b19328d077d3..a0187d6163df 100644
--- a/fs/btrfs/tests/extent-io-tests.c
+++ b/fs/btrfs/tests/extent-io-tests.c
@@ -505,7 +505,7 @@ static int __test_eb_bitmaps(unsigned long *bitmap, struct extent_buffer *eb)
static int test_eb_bitmaps(u32 sectorsize, u32 nodesize)
{
struct btrfs_fs_info *fs_info;
- unsigned long *bitmap = NULL;
+ unsigned long AUTO_KFREE(bitmap);
struct extent_buffer *eb = NULL;
int ret;
@@ -551,7 +551,6 @@ static int test_eb_bitmaps(u32 sectorsize, u32 nodesize)
ret = __test_eb_bitmaps(bitmap, eb);
out:
free_extent_buffer(eb);
- kfree(bitmap);
btrfs_free_dummy_fs_info(fs_info);
return ret;
}
diff --git a/fs/btrfs/tests/extent-map-tests.c b/fs/btrfs/tests/extent-map-tests.c
index 42af6c737c6e..0b9f25dd1a68 100644
--- a/fs/btrfs/tests/extent-map-tests.c
+++ b/fs/btrfs/tests/extent-map-tests.c
@@ -1013,7 +1013,7 @@ static int test_rmap_block(struct btrfs_fs_info *fs_info,
struct rmap_test_vector *test)
{
struct btrfs_chunk_map *map;
- u64 *logical = NULL;
+ u64 AUTO_KFREE(logical);
int i, out_ndaddrs, out_stripe_len;
int ret;
@@ -1046,7 +1046,7 @@ static int test_rmap_block(struct btrfs_fs_info *fs_info,
if (ret) {
test_err("error adding chunk map to mapping tree");
btrfs_free_chunk_map(map);
- goto out_free;
+ return ret;
}
ret = btrfs_rmap_block(fs_info, map->start, btrfs_sb_offset(1),
@@ -1079,8 +1079,6 @@ static int test_rmap_block(struct btrfs_fs_info *fs_info,
ret = 0;
out:
btrfs_remove_chunk_map(fs_info, map);
-out_free:
- kfree(logical);
return ret;
}
diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index f84065b69464..472bae49e489 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -4014,7 +4014,7 @@ static int flush_dir_items_batch(struct btrfs_trans_handle *trans,
int count)
{
struct btrfs_root *log = inode->root->log_root;
- char *ins_data = NULL;
+ char AUTO_KFREE(ins_data);
struct btrfs_item_batch batch;
struct extent_buffer *dst;
unsigned long src_offset;
@@ -4059,7 +4059,7 @@ static int flush_dir_items_batch(struct btrfs_trans_handle *trans,
ret = btrfs_insert_empty_items(trans, log, dst_path, &batch);
if (ret)
- goto out;
+ return ret;
dst = dst_path->nodes[0];
/*
@@ -4091,8 +4091,6 @@ static int flush_dir_items_batch(struct btrfs_trans_handle *trans,
if (btrfs_get_first_dir_index_to_log(inode) == 0)
btrfs_set_first_dir_index_to_log(inode, batch.keys[0].offset);
-out:
- kfree(ins_data);
return ret;
}
@@ -4773,7 +4771,7 @@ static noinline int copy_items(struct btrfs_trans_handle *trans,
struct btrfs_key *ins_keys;
u32 *ins_sizes;
struct btrfs_item_batch batch;
- char *ins_data;
+ char AUTO_KFREE(ins_data);
int dst_index;
const bool skip_csum = (inode->flags & BTRFS_INODE_NODATASUM);
const u64 i_size = i_size_read(&inode->vfs_inode);
@@ -4901,7 +4899,7 @@ static noinline int copy_items(struct btrfs_trans_handle *trans,
disk_bytenr + extent_num_bytes - 1,
&ordered_sums, false);
if (ret < 0)
- goto out;
+ return ret;
ret = 0;
list_for_each_entry_safe(sums, sums_next, &ordered_sums, list) {
@@ -4911,7 +4909,7 @@ static noinline int copy_items(struct btrfs_trans_handle *trans,
kfree(sums);
}
if (ret)
- goto out;
+ return ret;
add_to_batch:
ins_sizes[dst_index] = btrfs_item_size(src, src_slot);
@@ -4925,11 +4923,11 @@ static noinline int copy_items(struct btrfs_trans_handle *trans,
* so we don't need to do anything.
*/
if (batch.nr == 0)
- goto out;
+ return 0;
ret = btrfs_insert_empty_items(trans, log, dst_path, &batch);
if (ret)
- goto out;
+ return ret;
dst_index = 0;
for (int i = 0; i < nr; i++) {
@@ -4982,8 +4980,6 @@ static noinline int copy_items(struct btrfs_trans_handle *trans,
}
btrfs_release_path(dst_path);
-out:
- kfree(ins_data);
return ret;
}
@@ -5723,9 +5719,8 @@ static int btrfs_check_ref_name_override(struct extent_buffer *eb,
struct btrfs_inode *inode,
u64 *other_ino, u64 *other_parent)
{
- int ret;
BTRFS_PATH_AUTO_FREE(search_path);
- char *name = NULL;
+ char AUTO_KFREE(name);
u32 name_len = 0;
u32 item_size = btrfs_item_size(eb, slot);
u32 cur_offset = 0;
@@ -5768,10 +5763,8 @@ static int btrfs_check_ref_name_override(struct extent_buffer *eb,
char *new_name;
new_name = krealloc(name, this_name_len, GFP_NOFS);
- if (!new_name) {
- ret = -ENOMEM;
- goto out;
- }
+ if (!new_name)
+ return -ENOMEM;
name_len = this_name_len;
name = new_name;
}
@@ -5789,28 +5782,24 @@ static int btrfs_check_ref_name_override(struct extent_buffer *eb,
di, &di_key);
if (di_key.type == BTRFS_INODE_ITEM_KEY) {
if (di_key.objectid != key->objectid) {
- ret = 1;
*other_ino = di_key.objectid;
*other_parent = parent;
+ return 1;
} else {
- ret = 0;
+ return 0;
}
} else {
- ret = -EAGAIN;
+ return -EAGAIN;
}
- goto out;
} else if (IS_ERR(di)) {
- ret = PTR_ERR(di);
- goto out;
+ return PTR_ERR(di);
}
btrfs_release_path(search_path);
cur_offset += this_len;
}
- ret = 0;
-out:
- kfree(name);
- return ret;
+
+ return 0;
}
/*
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index b4404d09a489..200b4e4a0799 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -739,7 +739,7 @@ static bool is_same_device(struct btrfs_device *device, const char *new_path)
{
struct path old = { .mnt = NULL, .dentry = NULL };
struct path new = { .mnt = NULL, .dentry = NULL };
- char *old_path = NULL;
+ char AUTO_KFREE(old_path);
bool is_same = false;
int ret;
@@ -765,7 +765,6 @@ static bool is_same_device(struct btrfs_device *device, const char *new_path)
if (path_equal(&old, &new))
is_same = true;
out:
- kfree(old_path);
path_put(&old);
path_put(&new);
return is_same;
@@ -4519,7 +4518,7 @@ static void describe_balance_start_or_resume(struct btrfs_fs_info *fs_info)
{
u32 size_buf = 1024;
char tmp_buf[192] = {'\0'};
- char *buf;
+ char AUTO_KFREE(buf);
char *bp;
u32 size_bp = size_buf;
int ret;
@@ -4567,8 +4566,6 @@ static void describe_balance_start_or_resume(struct btrfs_fs_info *fs_info)
btrfs_info(fs_info, "balance: %s %s",
(bctl->flags & BTRFS_BALANCE_RESUME) ?
"resume" : "start", buf);
-
- kfree(buf);
}
/*
@@ -5698,9 +5695,8 @@ struct btrfs_block_group *btrfs_create_chunk(struct btrfs_trans_handle *trans,
{
struct btrfs_fs_info *info = trans->fs_info;
struct btrfs_fs_devices *fs_devices = info->fs_devices;
- struct btrfs_device_info *devices_info = NULL;
+ struct btrfs_device_info AUTO_KFREE(devices_info);
struct alloc_chunk_ctl ctl;
- struct btrfs_block_group *block_group;
int ret;
lockdep_assert_held(&info->chunk_mutex);
@@ -5733,22 +5729,14 @@ struct btrfs_block_group *btrfs_create_chunk(struct btrfs_trans_handle *trans,
return ERR_PTR(-ENOMEM);
ret = gather_device_info(fs_devices, &ctl, devices_info);
- if (ret < 0) {
- block_group = ERR_PTR(ret);
- goto out;
- }
+ if (ret < 0)
+ return ERR_PTR(ret);
ret = decide_stripe_size(fs_devices, &ctl, devices_info);
- if (ret < 0) {
- block_group = ERR_PTR(ret);
- goto out;
- }
-
- block_group = create_chunk(trans, &ctl, devices_info);
+ if (ret < 0)
+ return ERR_PTR(ret);
-out:
- kfree(devices_info);
- return block_group;
+ return create_chunk(trans, &ctl, devices_info);
}
/*
diff --git a/fs/btrfs/xattr.c b/fs/btrfs/xattr.c
index ab55d10bd71f..cb001a211f55 100644
--- a/fs/btrfs/xattr.c
+++ b/fs/btrfs/xattr.c
@@ -161,6 +161,7 @@ int btrfs_setxattr(struct btrfs_trans_handle *trans, struct inode *inode,
const u16 old_data_len = btrfs_dir_data_len(leaf, di);
const u32 item_size = btrfs_item_size(leaf, slot);
const u32 data_size = sizeof(*di) + name_len + size;
+ unsigned long name_ptr;
unsigned long data_ptr;
char *ptr;
@@ -189,8 +190,16 @@ int btrfs_setxattr(struct btrfs_trans_handle *trans, struct inode *inode,
ptr = btrfs_item_ptr(leaf, slot, char);
ptr += btrfs_item_size(leaf, slot) - data_size;
di = (struct btrfs_dir_item *)ptr;
+ memzero_extent_buffer(leaf, (unsigned long)ptr +
+ offsetof(struct btrfs_dir_item, location),
+ sizeof(struct btrfs_disk_key));
+ btrfs_set_dir_flags(leaf, di, BTRFS_FT_XATTR);
+ btrfs_set_dir_transid(leaf, di, trans->transid);
+ btrfs_set_dir_name_len(leaf, di, name_len);
btrfs_set_dir_data_len(leaf, di, size);
+ name_ptr = (unsigned long)(di + 1);
data_ptr = ((unsigned long)(di + 1)) + name_len;
+ write_extent_buffer(leaf, name, name_ptr, name_len);
write_extent_buffer(leaf, value, data_ptr, size);
} else {
/*
@@ -320,10 +329,8 @@ ssize_t btrfs_listxattr(struct dentry *dentry, char *buffer, size_t size)
if (!size)
goto next;
- if (!buffer || (name_len + 1) > size_left) {
- iter_ret = -ERANGE;
- break;
- }
+ if (!buffer || (name_len + 1) > size_left)
+ return -ERANGE;
read_extent_buffer(leaf, buffer, name_ptr, name_len);
buffer[name_len] = '\0';
diff --git a/fs/btrfs/zoned.c b/fs/btrfs/zoned.c
index 862475ae4977..36c09492e0bd 100644
--- a/fs/btrfs/zoned.c
+++ b/fs/btrfs/zoned.c
@@ -1839,7 +1839,7 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache, bool new)
struct btrfs_chunk_map *map;
u64 logical = cache->start;
u64 length = cache->length;
- struct zone_info *zone_info = NULL;
+ struct zone_info AUTO_KFREE(zone_info);
int ret;
int i;
unsigned long *active = NULL;
@@ -1993,7 +1993,6 @@ int btrfs_load_block_group_zone_info(struct btrfs_block_group *cache, bool new)
cache->physical_map = NULL;
}
bitmap_free(active);
- kfree(zone_info);
return ret;
}
diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c
index eae5045d5008..e83056d9aee4 100644
--- a/fs/cachefiles/namei.c
+++ b/fs/cachefiles/namei.c
@@ -129,10 +129,13 @@ struct dentry *cachefiles_get_directory(struct cachefiles_cache *cache,
if (ret < 0)
goto mkdir_error;
ret = cachefiles_inject_write_error();
- if (ret == 0)
+ if (ret == 0) {
subdir = vfs_mkdir(&nop_mnt_idmap, d_inode(dir), subdir, 0700);
- else
+ if (IS_ERR(subdir))
+ ret = PTR_ERR(subdir);
+ } else {
subdir = ERR_PTR(ret);
+ }
if (IS_ERR(subdir)) {
trace_cachefiles_vfs_error(NULL, d_inode(dir), ret,
cachefiles_trace_mkdir_error);
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index faa3de95caff..cfbd88412f04 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -1338,6 +1338,7 @@ int ceph_process_folio_batch(struct address_space *mapping,
if (rc == -ENODATA) {
rc = 0;
folio_unlock(folio);
+ folio_put(folio);
ceph_wbc->fbatch.folios[i] = NULL;
continue;
} else if (rc == -E2BIG) {
@@ -1349,6 +1350,7 @@ int ceph_process_folio_batch(struct address_space *mapping,
if (!folio_clear_dirty_for_io(folio)) {
doutc(cl, "%p !folio_clear_dirty_for_io\n", folio);
folio_unlock(folio);
+ folio_put(folio);
ceph_wbc->fbatch.folios[i] = NULL;
continue;
}
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index a2a63ea1f0ac..342d8778b69c 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -5172,9 +5172,11 @@ static void check_new_map(struct ceph_mds_client *mdsc,
ceph_mdsmap_get_addr(newmap, i),
sizeof(struct ceph_entity_addr))) {
/* just close it */
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
mutex_lock(&s->s_mutex);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
ceph_con_close(&s->s_con);
mutex_unlock(&s->s_mutex);
s->s_state = CEPH_MDS_SESSION_RESTARTING;
@@ -5187,10 +5189,12 @@ static void check_new_map(struct ceph_mds_client *mdsc,
*/
if (s->s_state == CEPH_MDS_SESSION_RESTARTING &&
newstate >= CEPH_MDS_STATE_RECONNECT) {
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
clear_bit(i, targets);
send_mds_reconnect(mdsc, s);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
}
/*
@@ -5203,9 +5207,11 @@ static void check_new_map(struct ceph_mds_client *mdsc,
pr_info_client(cl, "mds%d recovery completed\n",
s->s_mds);
kick_requests(mdsc, i);
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
mutex_lock(&s->s_mutex);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
ceph_kick_flushing_caps(mdsc, s);
mutex_unlock(&s->s_mutex);
wake_up_session_caps(s, RECONNECT);
diff --git a/fs/dcache.c b/fs/dcache.c
index 8bf82b002b4d..5ee1c3d592c8 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -1431,6 +1431,8 @@ EXPORT_SYMBOL(path_has_submounts);
*
* Only one of d_invalidate() and d_set_mounted() must succeed. For
* this reason take rename_lock and d_lock on dentry and ancestors.
+ * Likewise for dont_mount() which marks a dentry that is being removed
+ * under d_lock.
*/
int d_set_mounted(struct dentry *dentry)
{
@@ -1447,7 +1449,7 @@ int d_set_mounted(struct dentry *dentry)
spin_unlock(&p->d_lock);
}
spin_lock(&dentry->d_lock);
- if (!d_unlinked(dentry)) {
+ if (!d_unlinked(dentry) && !cant_mount(dentry)) {
ret = -EBUSY;
if (!d_mountpoint(dentry)) {
dentry->d_flags |= DCACHE_MOUNTED;
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 9a0b6c2b6b01..0500421b6e3b 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -1134,7 +1134,7 @@ static int sctp_bind_addrs(struct socket *sock, __be16 port)
make_sockaddr(&localaddr, port, &addr_len);
if (!i)
- result = kernel_bind(sock, addr, addr_len);
+ result = kernel_bind(sock, (struct sockaddr_unsized *)addr, addr_len);
else
result = sock_bind_add(sock->sk, addr, addr_len);
@@ -1813,7 +1813,7 @@ static int dlm_tcp_bind(struct socket *sock)
memcpy(&src_addr, &dlm_local_addr[0], sizeof(src_addr));
make_sockaddr(&src_addr, 0, &addr_len);
- result = kernel_bind(sock, (struct sockaddr *)&src_addr,
+ result = kernel_bind(sock, (struct sockaddr_unsized *)&src_addr,
addr_len);
if (result < 0) {
/* This *may* not indicate a critical error */
@@ -1852,7 +1852,7 @@ static int dlm_tcp_listen_bind(struct socket *sock)
/* Bind to our port */
make_sockaddr(&dlm_local_addr[0], dlm_config.ci_tcp_port, &addr_len);
- return kernel_bind(sock, (struct sockaddr *)&dlm_local_addr[0],
+ return kernel_bind(sock, (struct sockaddr_unsized *)&dlm_local_addr[0],
addr_len);
}
diff --git a/fs/erofs/xattr.c b/fs/erofs/xattr.c
index 396536d9a862..59de3253e466 100644
--- a/fs/erofs/xattr.c
+++ b/fs/erofs/xattr.c
@@ -34,6 +34,8 @@ static int erofs_init_inode_xattrs(struct inode *inode)
struct super_block *sb = inode->i_sb;
int ret = 0;
+ it.buf = __EROFS_BUF_INITIALIZER;
+
/* the most case is that xattrs of this inode are initialized. */
if (test_bit(EROFS_I_EA_INITED_BIT, &vi->flags)) {
/*
@@ -76,7 +78,6 @@ static int erofs_init_inode_xattrs(struct inode *inode)
goto out_unlock;
}
- it.buf = __EROFS_BUF_INITIALIZER;
ret = erofs_init_metabuf(&it.buf, sb, erofs_inode_in_metabox(inode));
if (ret)
goto out_unlock;
@@ -95,7 +96,6 @@ static int erofs_init_inode_xattrs(struct inode *inode)
vi->xattr_shared_xattrs = kmalloc_array(vi->xattr_shared_count,
sizeof(uint), GFP_KERNEL);
if (!vi->xattr_shared_xattrs) {
- erofs_put_metabuf(&it.buf);
ret = -ENOMEM;
goto out_unlock;
}
@@ -114,13 +114,13 @@ static int erofs_init_inode_xattrs(struct inode *inode)
vi->xattr_shared_xattrs[i] = le32_to_cpu(*(__le32 *)it.kaddr);
it.pos += sizeof(__le32);
}
- erofs_put_metabuf(&it.buf);
/* paired with smp_mb() at the beginning of the function. */
smp_mb();
set_bit(EROFS_I_EA_INITED_BIT, &vi->flags);
out_unlock:
+ erofs_put_metabuf(&it.buf);
clear_and_wake_up_bit(EROFS_I_BL_XATTR_BIT, &vi->flags);
return ret;
}
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 4aac18942ecc..50d431949fd4 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -1301,11 +1301,10 @@ static int ext2_setsize(struct inode *inode, loff_t newsize)
filemap_invalidate_unlock(inode->i_mapping);
inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode));
+ mark_inode_dirty(inode);
if (inode_needs_sync(inode)) {
sync_mapping_buffers(inode->i_mapping);
sync_inode_metadata(inode, 1);
- } else {
- mark_inode_dirty(inode);
}
return 0;
diff --git a/fs/ext2/xattr.c b/fs/ext2/xattr.c
index c885dcc3bd0d..5574b601ae09 100644
--- a/fs/ext2/xattr.c
+++ b/fs/ext2/xattr.c
@@ -777,6 +777,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh,
/* Update the inode. */
EXT2_I(inode)->i_file_acl = new_bh ? new_bh->b_blocknr : 0;
inode_set_ctime_current(inode);
+ mark_inode_dirty(inode);
if (IS_SYNC(inode)) {
error = sync_inode_metadata(inode, 1);
/* In case sync failed due to ENOSPC the inode was actually
@@ -789,8 +790,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh,
}
goto cleanup;
}
- } else
- mark_inode_dirty(inode);
+ }
error = 0;
if (old_bh && old_bh != new_bh) {
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 996ffdc2dd13..da6428312c36 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -3081,7 +3081,8 @@ typedef enum {
EXT4_IGET_SPECIAL = 0x0001, /* OK to iget a system inode */
EXT4_IGET_HANDLE = 0x0002, /* Inode # is from a handle */
EXT4_IGET_BAD = 0x0004, /* Allow to iget a bad inode */
- EXT4_IGET_EA_INODE = 0x0008 /* Inode should contain an EA value */
+ EXT4_IGET_EA_INODE = 0x0008, /* Inode should contain an EA value */
+ EXT4_IGET_NOWAIT = 0x0010 /* Non-blocking lookup (skip if freeing) */
} ext4_iget_flags;
extern struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index d2977c560c96..bbc437be5e63 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -5210,6 +5210,20 @@ void ext4_set_inode_mapping_order(struct inode *inode)
EXT4_MAX_PAGECACHE_ORDER(inode));
}
+static int ext4_iget_match(struct inode *inode, unsigned long ino, void *data)
+{
+ if (inode->i_ino != ino)
+ return 0;
+ spin_lock(&inode->i_lock);
+ if (inode->i_state & (I_FREEING | I_WILL_FREE | I_CREATING)) {
+ spin_unlock(&inode->i_lock);
+ return -1;
+ }
+ __iget(inode);
+ spin_unlock(&inode->i_lock);
+ return 1;
+}
+
struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
ext4_iget_flags flags, const char *function,
unsigned int line)
@@ -5238,9 +5252,24 @@ struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
return ERR_PTR(-EFSCORRUPTED);
}
- inode = iget_locked(sb, ino);
- if (!inode)
- return ERR_PTR(-ENOMEM);
+ if (flags & EXT4_IGET_NOWAIT) {
+ inode = find_inode_nowait(sb, ino, ext4_iget_match, NULL);
+ if (!inode)
+ return ERR_PTR(-ENOENT);
+
+ if (inode->i_state & I_NEW)
+ wait_on_inode(inode);
+
+ if (unlikely(inode_unhashed(inode))) {
+ iput(inode);
+ return ERR_PTR(-ENOENT);
+ }
+ } else {
+ inode = iget_locked(sb, ino);
+ if (!inode)
+ return ERR_PTR(-ENOMEM);
+ }
+
if (!(inode->i_state & I_NEW)) {
ret = check_igot_inode(inode, flags, function, line);
if (ret) {
diff --git a/fs/ext4/ioctl.c b/fs/ext4/ioctl.c
index fbf5771eca99..9aec4ee780a7 100644
--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -1654,15 +1654,8 @@ static long __ext4_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
if (!(fd_file(donor)->f_mode & FMODE_WRITE))
return -EBADF;
- if (ext4_has_feature_bigalloc(sb)) {
- ext4_msg(sb, KERN_ERR,
- "Online defrag not supported with bigalloc");
- return -EOPNOTSUPP;
- } else if (IS_DAX(inode)) {
- ext4_msg(sb, KERN_ERR,
- "Online defrag not supported with DAX");
- return -EOPNOTSUPP;
- }
+ if (file_inode(filp)->i_sb != file_inode(fd_file(donor))->i_sb)
+ return -EXDEV;
err = mnt_want_write_file(filp);
if (err)
diff --git a/fs/ext4/move_extent.c b/fs/ext4/move_extent.c
index 0f4b7c89edd3..e2705ea5e5c6 100644
--- a/fs/ext4/move_extent.c
+++ b/fs/ext4/move_extent.c
@@ -442,6 +442,68 @@ move_extent_per_page(struct file *o_filp, struct inode *donor_inode,
goto unlock_folios;
}
+/*
+ * Check the validity of the basic filesystem environment and the
+ * inodes' support status.
+ */
+static int mext_check_validity(struct inode *orig_inode,
+ struct inode *donor_inode)
+{
+ struct super_block *sb = orig_inode->i_sb;
+
+ /* origin and donor should be different inodes */
+ if (orig_inode == donor_inode) {
+ ext4_debug("ext4 move extent: The argument files should not be same inode [ino:orig %lu, donor %lu]\n",
+ orig_inode->i_ino, donor_inode->i_ino);
+ return -EINVAL;
+ }
+
+ /* origin and donor should belone to the same filesystem */
+ if (orig_inode->i_sb != donor_inode->i_sb) {
+ ext4_debug("ext4 move extent: The argument files should be in same FS [ino:orig %lu, donor %lu]\n",
+ orig_inode->i_ino, donor_inode->i_ino);
+ return -EINVAL;
+ }
+
+ /* Regular file check */
+ if (!S_ISREG(orig_inode->i_mode) || !S_ISREG(donor_inode->i_mode)) {
+ ext4_debug("ext4 move extent: The argument files should be regular file [ino:orig %lu, donor %lu]\n",
+ orig_inode->i_ino, donor_inode->i_ino);
+ return -EINVAL;
+ }
+
+ if (ext4_has_feature_bigalloc(sb)) {
+ ext4_msg(sb, KERN_ERR,
+ "Online defrag not supported with bigalloc");
+ return -EOPNOTSUPP;
+ }
+
+ if (IS_DAX(orig_inode)) {
+ ext4_msg(sb, KERN_ERR,
+ "Online defrag not supported with DAX");
+ return -EOPNOTSUPP;
+ }
+
+ /*
+ * TODO: it's not obvious how to swap blocks for inodes with full
+ * journaling enabled.
+ */
+ if (ext4_should_journal_data(orig_inode) ||
+ ext4_should_journal_data(donor_inode)) {
+ ext4_msg(sb, KERN_ERR,
+ "Online defrag not supported with data journaling");
+ return -EOPNOTSUPP;
+ }
+
+ if (IS_ENCRYPTED(orig_inode) || IS_ENCRYPTED(donor_inode)) {
+ ext4_msg(sb, KERN_ERR,
+ "Online defrag not supported for encrypted files");
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
/**
* mext_check_arguments - Check whether move extent can be done
*
@@ -570,43 +632,9 @@ ext4_move_extents(struct file *o_filp, struct file *d_filp, __u64 orig_blk,
ext4_lblk_t d_start = donor_blk;
int ret;
- if (orig_inode->i_sb != donor_inode->i_sb) {
- ext4_debug("ext4 move extent: The argument files "
- "should be in same FS [ino:orig %lu, donor %lu]\n",
- orig_inode->i_ino, donor_inode->i_ino);
- return -EINVAL;
- }
-
- /* orig and donor should be different inodes */
- if (orig_inode == donor_inode) {
- ext4_debug("ext4 move extent: The argument files should not "
- "be same inode [ino:orig %lu, donor %lu]\n",
- orig_inode->i_ino, donor_inode->i_ino);
- return -EINVAL;
- }
-
- /* Regular file check */
- if (!S_ISREG(orig_inode->i_mode) || !S_ISREG(donor_inode->i_mode)) {
- ext4_debug("ext4 move extent: The argument files should be "
- "regular file [ino:orig %lu, donor %lu]\n",
- orig_inode->i_ino, donor_inode->i_ino);
- return -EINVAL;
- }
-
- /* TODO: it's not obvious how to swap blocks for inodes with full
- journaling enabled */
- if (ext4_should_journal_data(orig_inode) ||
- ext4_should_journal_data(donor_inode)) {
- ext4_msg(orig_inode->i_sb, KERN_ERR,
- "Online defrag not supported with data journaling");
- return -EOPNOTSUPP;
- }
-
- if (IS_ENCRYPTED(orig_inode) || IS_ENCRYPTED(donor_inode)) {
- ext4_msg(orig_inode->i_sb, KERN_ERR,
- "Online defrag not supported for encrypted files");
- return -EOPNOTSUPP;
- }
+ ret = mext_check_validity(orig_inode, donor_inode);
+ if (ret)
+ return ret;
/* Protect orig and donor inodes against a truncate */
lock_two_nondirectories(orig_inode, donor_inode);
diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index 02095ca17be0..789e7087a549 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -1550,7 +1550,7 @@ ext4_xattr_inode_cache_find(struct inode *inode, const void *value,
while (ce) {
ea_inode = ext4_iget(inode->i_sb, ce->e_value,
- EXT4_IGET_EA_INODE);
+ EXT4_IGET_EA_INODE | EXT4_IGET_NOWAIT);
if (IS_ERR(ea_inode))
goto next_entry;
ext4_xattr_inode_set_class(ea_inode);
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 05ab77446b71..3455565b8245 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -3562,7 +3562,7 @@ static int __reserve_data_block(struct inode *inode, pgoff_t index,
static int prepare_atomic_write_begin(struct f2fs_sb_info *sbi,
struct folio *folio, loff_t pos, unsigned int len,
- block_t *blk_addr, bool *node_changed, bool *use_cow)
+ block_t *blk_addr, bool *node_changed)
{
struct inode *inode = folio->mapping->host;
struct inode *cow_inode = F2FS_I(inode)->cow_inode;
@@ -3577,14 +3577,14 @@ static int prepare_atomic_write_begin(struct f2fs_sb_info *sbi,
/* Look for the block in COW inode first */
err = __find_data_block(cow_inode, index, blk_addr);
- if (err) {
+ if (err)
return err;
- } else if (__is_valid_data_blkaddr(*blk_addr)) {
- *use_cow = true;
+
+ if (__is_valid_data_blkaddr(*blk_addr))
return 0;
- } else if (*blk_addr == NEW_ADDR) {
+
+ if (*blk_addr == NEW_ADDR)
cow_has_reserved_block = true;
- }
if (is_inode_flag_set(inode, FI_ATOMIC_REPLACE))
goto reserve_block;
@@ -3619,7 +3619,6 @@ static int f2fs_write_begin(const struct kiocb *iocb,
struct folio *folio;
pgoff_t index = pos >> PAGE_SHIFT;
bool need_balance = false;
- bool use_cow = false;
block_t blkaddr = NULL_ADDR;
int err = 0;
@@ -3681,7 +3680,7 @@ static int f2fs_write_begin(const struct kiocb *iocb,
if (f2fs_is_atomic_file(inode))
err = prepare_atomic_write_begin(sbi, folio, pos, len,
- &blkaddr, &need_balance, &use_cow);
+ &blkaddr, &need_balance);
else
err = prepare_write_begin(sbi, folio, pos, len,
&blkaddr, &need_balance);
@@ -3721,9 +3720,15 @@ static int f2fs_write_begin(const struct kiocb *iocb,
err = -EFSCORRUPTED;
goto put_folio;
}
- f2fs_submit_page_read(use_cow ?
- F2FS_I(inode)->cow_inode : inode,
- folio, blkaddr, 0, true);
+ /*
+ * Although the block may be stored in the COW inode, the folio
+ * belongs to @inode and its data was encrypted (or not) using
+ * @inode's context (see f2fs_encrypt_one_page()). Read with
+ * @inode so the post-read decryption decision matches the
+ * folio's owner; otherwise an unencrypted @inode whose COW inode
+ * is encrypted hits a NULL ->i_crypt_info on decryption.
+ */
+ f2fs_submit_page_read(inode, folio, blkaddr, 0, true);
folio_lock(folio);
if (unlikely(folio->mapping != mapping)) {
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index f14c364cacc5..ec1b9831e5af 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -4189,7 +4189,7 @@ int f2fs_gc(struct f2fs_sb_info *sbi, struct f2fs_gc_control *gc_control);
void f2fs_build_gc_manager(struct f2fs_sb_info *sbi);
int f2fs_gc_range(struct f2fs_sb_info *sbi,
unsigned int start_seg, unsigned int end_seg,
- bool dry_run, unsigned int dry_run_sections);
+ bool dry_run, unsigned int dry_run_sections, bool lock);
int f2fs_resize_fs(struct file *filp, __u64 block_count);
int __init f2fs_create_garbage_collection_cache(void);
void f2fs_destroy_garbage_collection_cache(void);
diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
index ca1c87e14700..02bd008d79b5 100644
--- a/fs/f2fs/gc.c
+++ b/fs/f2fs/gc.c
@@ -2121,7 +2121,7 @@ void f2fs_build_gc_manager(struct f2fs_sb_info *sbi)
int f2fs_gc_range(struct f2fs_sb_info *sbi,
unsigned int start_seg, unsigned int end_seg,
- bool dry_run, unsigned int dry_run_sections)
+ bool dry_run, unsigned int dry_run_sections, bool lock)
{
unsigned int segno;
unsigned int gc_secs = dry_run_sections;
@@ -2135,28 +2135,46 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
.ilist = LIST_HEAD_INIT(gc_list.ilist),
.iroot = RADIX_TREE_INIT(gc_list.iroot, GFP_NOFS),
};
+ int err = 0;
+
+ if (lock)
+ f2fs_down_write(&sbi->gc_lock);
/*
* avoid migrating empty section, as it can be allocated by
* log in parallel.
*/
if (!get_valid_blocks(sbi, segno, true))
- continue;
+ goto next;
if (is_cursec(sbi, GET_SEC_FROM_SEG(sbi, segno)))
- continue;
+ goto next;
do_garbage_collect(sbi, segno, &gc_list, FG_GC, true, false);
put_gc_inode(&gc_list);
- if (!dry_run && get_valid_blocks(sbi, segno, true))
- return -EAGAIN;
+ /* reset all pinned status during fggc */
+ f2fs_unpin_all_sections(sbi, true);
+
+ if (!dry_run && get_valid_blocks(sbi, segno, true)) {
+ err = -EAGAIN;
+ goto next;
+ }
if (dry_run && dry_run_sections &&
- !get_valid_blocks(sbi, segno, true) && --gc_secs == 0)
- break;
+ !get_valid_blocks(sbi, segno, true)) {
+ --gc_secs;
+ goto next;
+ }
if (fatal_signal_pending(current))
- return -ERESTARTSYS;
+ err = -ERESTARTSYS;
+next:
+ if (lock)
+ f2fs_up_write(&sbi->gc_lock);
+ if (err)
+ return err;
+ if (dry_run && dry_run_sections && !gc_secs)
+ return 0;
}
return 0;
@@ -2204,7 +2222,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
}
/* do GC to move out valid blocks in the range */
- err = f2fs_gc_range(sbi, start, end, dry_run, 0);
+ err = f2fs_gc_range(sbi, start, end, dry_run, 0, false);
if (err || dry_run)
goto out;
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index 823b78c4aa5b..60969e5f1201 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -428,6 +428,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
struct nat_entry *e;
bool need_update = true;
+ f2fs_down_read(&sbi->node_write);
f2fs_down_read(&nm_i->nat_tree_lock);
e = __lookup_nat_cache(nm_i, ino, false);
if (e && get_nat_flag(e, HAS_LAST_FSYNC) &&
@@ -435,6 +436,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
get_nat_flag(e, HAS_FSYNCED_INODE)))
need_update = false;
f2fs_up_read(&nm_i->nat_tree_lock);
+ f2fs_up_read(&sbi->node_write);
return need_update;
}
diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index 2fa2a005c384..510d073f8ec5 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -3359,10 +3359,8 @@ int f2fs_allocate_pinning_section(struct f2fs_sb_info *sbi)
f2fs_unlock_op(sbi);
if (f2fs_sb_has_blkzoned(sbi) && err == -EAGAIN && gc_required) {
- f2fs_down_write(&sbi->gc_lock);
err = f2fs_gc_range(sbi, 0, sbi->first_seq_zone_segno - 1,
- true, ZONED_PIN_SEC_REQUIRED_COUNT);
- f2fs_up_write(&sbi->gc_lock);
+ true, ZONED_PIN_SEC_REQUIRED_COUNT, true);
if (err)
return err;
err = f2fs_sync_fs(sbi->sb, 1);
diff --git a/fs/fcntl.c b/fs/fcntl.c
index 091bb4465b71..80236993b404 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -372,6 +372,9 @@ static long fcntl_set_rw_hint(struct file *file, unsigned long arg)
u64 __user *argp = (u64 __user *)arg;
u64 hint;
+ /* nothing is ever written to it */
+ if (IS_IMMUTABLE(inode))
+ return -EPERM;
if (!inode_owner_or_capable(file_mnt_idmap(file), inode))
return -EPERM;
diff --git a/fs/fsopen.c b/fs/fsopen.c
index f645c99204eb..ce1d55e150c6 100644
--- a/fs/fsopen.c
+++ b/fs/fsopen.c
@@ -190,6 +190,9 @@ SYSCALL_DEFINE3(fspick, int, dfd, const char __user *, path, unsigned int, flags
ret = -EINVAL;
if (target.mnt->mnt_root != target.dentry)
goto err_path;
+ /* kernel-internal superblocks are nobody's to reconfigure */
+ if (target.dentry->d_sb->s_flags & SB_NOUSER)
+ goto err_path;
fc = fs_context_for_reconfigure(target.dentry, 0, 0);
if (IS_ERR(fc)) {
diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index 9b57544935b5..b52d5c2bd4e4 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -1253,23 +1253,31 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
}
fc = fud->fc;
- /* Once a connection has io-uring enabled on it, it can't be disabled */
- if (!enable_uring && !fc->io_uring) {
- pr_info_ratelimited("fuse-io-uring is disabled\n");
- return -EOPNOTSUPP;
- }
+ /*
+ * The ring is sized from values negotiated by FUSE_INIT
+ */
+ if (!fc->initialized)
+ return -EAGAIN;
+
+ /* Pairs with smp_wmb() in fuse_set_initialized() */
+ smp_rmb();
if (fc->aborted)
return -ECONNABORTED;
if (!fc->connected)
return -ENOTCONN;
- /*
- * fuse_uring_register() needs the ring to be initialized,
- * we need to know the max payload size
- */
- if (!fc->initialized)
- return -EAGAIN;
+ /* Once a connection has io-uring enabled on it, it can't be disabled */
+ if (!enable_uring && !fc->io_uring) {
+ pr_info_ratelimited("fuse-io-uring is disabled by module parameter\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (cmd_op == FUSE_IO_URING_CMD_REGISTER && !fc->io_uring) {
+ pr_info_ratelimited(
+ "fuse-io-uring not enabled on this connection\n");
+ return -EOPNOTSUPP;
+ }
switch (cmd_op) {
case FUSE_IO_URING_CMD_REGISTER:
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index a0a64dfffb75..091a375c32a8 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -1517,10 +1517,6 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm)
if (IS_ENABLED(CONFIG_FUSE_PASSTHROUGH))
flags |= FUSE_PASSTHROUGH;
- /*
- * This is just an information flag for fuse server. No need to check
- * the reply - server is either sending IORING_OP_URING_CMD or not.
- */
if (fuse_uring_enabled())
flags |= FUSE_OVER_IO_URING;
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index eafe0488b10d..2174688ff7a2 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -436,7 +436,7 @@ static int gfs2_read_folio(struct file *file, struct folio *folio)
error = mpage_read_folio(folio, gfs2_block_map);
}
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
return -EIO;
return error;
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index ef57d6aa42eb..b430bbd917f1 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -1456,7 +1456,7 @@ static int gfs2_lock(struct file *file, int cmd, struct file_lock *fl)
if (!(fl->c.flc_flags & FL_POSIX))
return -ENOLCK;
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
if (lock_is_unlock(fl))
locks_lock_file_wait(file, fl);
return -EIO;
diff --git a/fs/gfs2/glock.c b/fs/gfs2/glock.c
index 7aec6cfdfd91..281470d72a58 100644
--- a/fs/gfs2/glock.c
+++ b/fs/gfs2/glock.c
@@ -154,7 +154,7 @@ static bool glock_blocked_by_withdraw(struct gfs2_glock *gl)
{
struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
- if (!gfs2_withdrawing_or_withdrawn(sdp))
+ if (!gfs2_withdrawn(sdp))
return false;
if (gl->gl_ops->go_flags & GLOF_NONDISK)
return false;
@@ -270,7 +270,7 @@ static void __gfs2_glock_put(struct gfs2_glock *gl)
GLOCK_BUG_ON(gl, !list_empty(&gl->gl_holders));
if (mapping) {
truncate_inode_pages_final(mapping);
- if (!gfs2_withdrawing_or_withdrawn(sdp))
+ if (!gfs2_withdrawn(sdp))
GLOCK_BUG_ON(gl, !mapping_empty(mapping));
}
trace_gfs2_glock_put(gl);
@@ -755,8 +755,8 @@ __acquires(&gl->gl_lockref.lock)
* gfs2_gl_hash_clear calls clear_glock) and recovery is complete
* then it's okay to tell dlm to unlock it.
*/
- if (unlikely(sdp->sd_log_error) && !gfs2_withdrawing_or_withdrawn(sdp))
- gfs2_withdraw_delayed(sdp);
+ if (unlikely(sdp->sd_log_error) && !gfs2_withdrawn(sdp))
+ gfs2_withdraw(sdp);
if (glock_blocked_by_withdraw(gl) &&
(target != LM_ST_UNLOCKED ||
test_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags))) {
@@ -803,7 +803,7 @@ __acquires(&gl->gl_lockref.lock)
*/
} else {
fs_err(sdp, "lm_lock ret %d\n", ret);
- GLOCK_BUG_ON(gl, !gfs2_withdrawing_or_withdrawn(sdp));
+ GLOCK_BUG_ON(gl, !gfs2_withdrawn(sdp));
return;
}
}
diff --git a/fs/gfs2/glock.h b/fs/gfs2/glock.h
index 2d4fd1a2bbbb..ad0143bd30d5 100644
--- a/fs/gfs2/glock.h
+++ b/fs/gfs2/glock.h
@@ -136,7 +136,7 @@ struct lm_lockops {
void (*lm_first_done) (struct gfs2_sbd *sdp);
void (*lm_recovery_result) (struct gfs2_sbd *sdp, unsigned int jid,
unsigned int result);
- void (*lm_unmount) (struct gfs2_sbd *sdp);
+ void (*lm_unmount) (struct gfs2_sbd *sdp, bool clean);
void (*lm_withdraw) (struct gfs2_sbd *sdp);
void (*lm_put_lock) (struct gfs2_glock *gl);
int (*lm_lock) (struct gfs2_glock *gl, unsigned int req_state,
diff --git a/fs/gfs2/glops.c b/fs/gfs2/glops.c
index 0c68ab4432b0..242e47926b39 100644
--- a/fs/gfs2/glops.c
+++ b/fs/gfs2/glops.c
@@ -45,7 +45,7 @@ static void gfs2_ail_error(struct gfs2_glock *gl, const struct buffer_head *bh)
gl->gl_name.ln_type, gl->gl_name.ln_number,
gfs2_glock2aspace(gl));
gfs2_lm(sdp, "AIL error\n");
- gfs2_withdraw_delayed(sdp);
+ gfs2_withdraw(sdp);
}
/**
@@ -83,9 +83,6 @@ static void __gfs2_ail_flush(struct gfs2_glock *gl, bool fsync,
GLOCK_BUG_ON(gl, !fsync && atomic_read(&gl->gl_ail_count));
spin_unlock(&sdp->sd_ail_lock);
gfs2_log_unlock(sdp);
-
- if (gfs2_withdrawing(sdp))
- gfs2_withdraw(sdp);
}
@@ -178,7 +175,7 @@ static int gfs2_rgrp_metasync(struct gfs2_glock *gl)
filemap_fdatawrite_range(metamapping, start, end);
error = filemap_fdatawait_range(metamapping, start, end);
- WARN_ON_ONCE(error && !gfs2_withdrawing_or_withdrawn(sdp));
+ WARN_ON_ONCE(error && !gfs2_withdrawn(sdp));
mapping_set_error(metamapping, error);
if (error)
gfs2_io_error(sdp);
@@ -608,10 +605,10 @@ static int freeze_go_xmote_bh(struct gfs2_glock *gl)
j_gl->gl_ops->go_inval(j_gl, DIO_METADATA);
error = gfs2_find_jhead(sdp->sd_jdesc, &head);
- if (gfs2_assert_withdraw_delayed(sdp, !error))
+ if (gfs2_assert_withdraw(sdp, !error))
return error;
- if (gfs2_assert_withdraw_delayed(sdp, head.lh_flags &
- GFS2_LOG_HEAD_UNMOUNT))
+ if (gfs2_assert_withdraw(sdp, head.lh_flags &
+ GFS2_LOG_HEAD_UNMOUNT))
return -EIO;
gfs2_log_pointers_init(sdp, &head);
}
diff --git a/fs/gfs2/incore.h b/fs/gfs2/incore.h
index 5a0ea416cfda..5dfdd1f3c5da 100644
--- a/fs/gfs2/incore.h
+++ b/fs/gfs2/incore.h
@@ -599,7 +599,6 @@ enum {
SDF_SKIP_DLM_UNLOCK = 8,
SDF_FORCE_AIL_FLUSH = 9,
SDF_FREEZE_INITIATOR = 10,
- SDF_WITHDRAWING = 11, /* Will withdraw eventually */
SDF_WITHDRAW_IN_PROG = 12, /* Withdraw is in progress */
SDF_REMOTE_WITHDRAW = 13, /* Performing remote recovery */
SDF_WITHDRAW_RECOVERY = 14, /* Wait for journal recovery when we are
@@ -716,6 +715,7 @@ struct gfs2_sbd {
struct gfs2_glock *sd_rename_gl;
struct gfs2_glock *sd_freeze_gl;
struct work_struct sd_freeze_work;
+ struct work_struct sd_withdraw_work;
wait_queue_head_t sd_kill_wait;
wait_queue_head_t sd_async_glock_wait;
atomic_t sd_glock_disposal;
diff --git a/fs/gfs2/lock_dlm.c b/fs/gfs2/lock_dlm.c
index 4f00af7dd256..301e17345f52 100644
--- a/fs/gfs2/lock_dlm.c
+++ b/fs/gfs2/lock_dlm.c
@@ -1195,7 +1195,7 @@ static void gdlm_recover_prep(void *arg)
struct gfs2_sbd *sdp = arg;
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
fs_err(sdp, "recover_prep ignored due to withdraw.\n");
return;
}
@@ -1221,7 +1221,7 @@ static void gdlm_recover_slot(void *arg, struct dlm_slot *slot)
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
int jid = slot->slot - 1;
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
fs_err(sdp, "recover_slot jid %d ignored due to withdraw.\n",
jid);
return;
@@ -1250,7 +1250,7 @@ static void gdlm_recover_done(void *arg, struct dlm_slot *slots, int num_slots,
struct gfs2_sbd *sdp = arg;
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
fs_err(sdp, "recover_done ignored due to withdraw.\n");
return;
}
@@ -1281,7 +1281,7 @@ static void gdlm_recovery_result(struct gfs2_sbd *sdp, unsigned int jid,
{
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
fs_err(sdp, "recovery_result jid %d ignored due to withdraw.\n",
jid);
return;
@@ -1438,7 +1438,15 @@ static void gdlm_first_done(struct gfs2_sbd *sdp)
fs_err(sdp, "mount first_done error %d\n", error);
}
-static void gdlm_unmount(struct gfs2_sbd *sdp)
+/*
+ * gdlm_unmount - release our lockspace
+ * @sdp: the superblock
+ * @clean: Indicates whether or not the remaining nodes in the cluster should
+ * perform recovery. Recovery is necessary when a node withdraws and
+ * its journal remains dirty. Recovery isn't necessary when a node
+ * cleanly unmounts a filesystem.
+ */
+static void gdlm_unmount(struct gfs2_sbd *sdp, bool clean)
{
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
@@ -1456,7 +1464,9 @@ static void gdlm_unmount(struct gfs2_sbd *sdp)
release:
down_write(&ls->ls_sem);
if (ls->ls_dlm) {
- dlm_release_lockspace(ls->ls_dlm, DLM_RELEASE_NORMAL);
+ dlm_release_lockspace(ls->ls_dlm,
+ clean ? DLM_RELEASE_NORMAL :
+ DLM_RELEASE_RECOVER);
ls->ls_dlm = NULL;
}
up_write(&ls->ls_sem);
diff --git a/fs/gfs2/log.c b/fs/gfs2/log.c
index ecc5c59b8700..84cd9d6ad262 100644
--- a/fs/gfs2/log.c
+++ b/fs/gfs2/log.c
@@ -114,11 +114,11 @@ __acquires(&sdp->sd_ail_lock)
}
if (!cmpxchg(&sdp->sd_log_error, 0, -EIO)) {
gfs2_io_error_bh(sdp, bh);
- gfs2_withdraw_delayed(sdp);
+ gfs2_withdraw(sdp);
}
}
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
gfs2_remove_from_ail(bd);
continue;
}
@@ -326,7 +326,7 @@ static int gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr,
if (!buffer_uptodate(bh) &&
!cmpxchg(&sdp->sd_log_error, 0, -EIO)) {
gfs2_io_error_bh(sdp, bh);
- gfs2_withdraw_delayed(sdp);
+ gfs2_withdraw(sdp);
}
/*
* If we have space for revokes and the bd is no longer on any
@@ -808,9 +808,6 @@ void gfs2_flush_revokes(struct gfs2_sbd *sdp)
gfs2_log_lock(sdp);
gfs2_ail1_empty(sdp, max_revokes);
gfs2_log_unlock(sdp);
-
- if (gfs2_withdrawing(sdp))
- gfs2_withdraw(sdp);
}
/**
@@ -838,7 +835,7 @@ void gfs2_write_log_header(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd,
struct super_block *sb = sdp->sd_vfs;
u64 dblock;
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
return;
page = mempool_alloc(gfs2_page_pool, GFP_NOIO);
@@ -985,12 +982,9 @@ static void empty_ail1_list(struct gfs2_sbd *sdp)
gfs2_ail1_wait(sdp);
empty = gfs2_ail1_empty(sdp, 0);
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
break;
}
-
- if (gfs2_withdrawing(sdp))
- gfs2_withdraw(sdp);
}
/**
@@ -1051,7 +1045,7 @@ static void __gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl,
* Do this check while holding the log_flush_lock to prevent new
* buffers from being added to the ail via gfs2_pin()
*/
- if (gfs2_withdrawing_or_withdrawn(sdp) ||
+ if (gfs2_withdrawn(sdp) ||
!test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags))
goto out;
@@ -1072,7 +1066,7 @@ static void __gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl,
sdp->sd_log_tr = NULL;
tr->tr_first = first_log_head;
if (unlikely(frozen)) {
- if (gfs2_assert_withdraw_delayed(sdp,
+ if (gfs2_assert_withdraw(sdp,
!tr->tr_num_buf_new && !tr->tr_num_databuf_new))
goto out_withdraw;
}
@@ -1097,18 +1091,18 @@ static void __gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl,
clear_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
if (unlikely(frozen))
- if (gfs2_assert_withdraw_delayed(sdp, !reserved_revokes))
+ if (gfs2_assert_withdraw(sdp, !reserved_revokes))
goto out_withdraw;
gfs2_ordered_write(sdp);
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
goto out_withdraw;
lops_before_commit(sdp, tr);
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
goto out_withdraw;
if (sdp->sd_jdesc)
gfs2_log_submit_bio(&sdp->sd_jdesc->jd_log_bio, REQ_OP_WRITE);
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
goto out_withdraw;
if (sdp->sd_log_head != sdp->sd_log_flush_head) {
@@ -1116,7 +1110,7 @@ static void __gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl,
} else if (sdp->sd_log_tail != sdp->sd_log_flush_tail && !sdp->sd_log_idle) {
log_write_header(sdp, flags);
}
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
goto out_withdraw;
lops_after_commit(sdp, tr);
@@ -1134,7 +1128,7 @@ static void __gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl,
if (!(flags & GFS2_LOG_HEAD_FLUSH_NORMAL)) {
if (!sdp->sd_log_idle) {
empty_ail1_list(sdp);
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
goto out_withdraw;
log_write_header(sdp, flags);
}
@@ -1152,12 +1146,10 @@ static void __gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl,
reserved_blocks += (reserved_revokes - sdp->sd_ldptrs) / sdp->sd_inptrs;
out:
if (used_blocks != reserved_blocks) {
- gfs2_assert_withdraw_delayed(sdp, used_blocks < reserved_blocks);
+ gfs2_assert_withdraw(sdp, used_blocks < reserved_blocks);
gfs2_log_release(sdp, reserved_blocks - used_blocks);
}
gfs2_trans_free(sdp, tr);
- if (gfs2_withdrawing(sdp))
- gfs2_withdraw(sdp);
trace_gfs2_log_flush(sdp, 0, flags);
return;
@@ -1311,7 +1303,7 @@ int gfs2_logd(void *data)
set_freezable();
while (!kthread_should_stop()) {
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
break;
/* Check for errors writing to the journal */
@@ -1354,14 +1346,11 @@ int gfs2_logd(void *data)
gfs2_ail_flush_reqd(sdp) ||
gfs2_jrnl_flush_reqd(sdp) ||
sdp->sd_log_error ||
- gfs2_withdrawing_or_withdrawn(sdp) ||
+ gfs2_withdrawn(sdp) ||
kthread_should_stop(),
t);
}
- if (gfs2_withdrawing(sdp))
- gfs2_withdraw(sdp);
-
return 0;
}
diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c
index 9c8c305a75c4..0ed81a685fcd 100644
--- a/fs/gfs2/lops.c
+++ b/fs/gfs2/lops.c
@@ -209,7 +209,7 @@ static void gfs2_end_log_write(struct bio *bio)
if (!cmpxchg(&sdp->sd_log_error, 0, err))
fs_err(sdp, "Error %d writing to journal, jid=%u\n",
err, sdp->sd_jdesc->jd_jid);
- gfs2_withdraw_delayed(sdp);
+ gfs2_withdraw(sdp);
/* prevent more writes to the journal */
clear_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
wake_up(&sdp->sd_logd_waitq);
diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c
index 7fb11ff71b5a..3ee90a4452b9 100644
--- a/fs/gfs2/meta_io.c
+++ b/fs/gfs2/meta_io.c
@@ -263,7 +263,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
struct buffer_head *bh, *bhs[2];
int num = 0;
- if (gfs2_withdrawing_or_withdrawn(sdp) &&
+ if (gfs2_withdrawn(sdp) &&
!gfs2_withdraw_in_prog(sdp)) {
*bhp = NULL;
return -EIO;
@@ -322,7 +322,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
int gfs2_meta_wait(struct gfs2_sbd *sdp, struct buffer_head *bh)
{
- if (gfs2_withdrawing_or_withdrawn(sdp) &&
+ if (gfs2_withdrawn(sdp) &&
!gfs2_withdraw_in_prog(sdp))
return -EIO;
@@ -334,7 +334,7 @@ int gfs2_meta_wait(struct gfs2_sbd *sdp, struct buffer_head *bh)
gfs2_io_error_bh_wd(sdp, bh);
return -EIO;
}
- if (gfs2_withdrawing_or_withdrawn(sdp) &&
+ if (gfs2_withdrawn(sdp) &&
!gfs2_withdraw_in_prog(sdp))
return -EIO;
diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c
index 1a2db8053da0..4ac80436c4dc 100644
--- a/fs/gfs2/ops_fstype.c
+++ b/fs/gfs2/ops_fstype.c
@@ -1040,8 +1040,8 @@ static int gfs2_lm_mount(struct gfs2_sbd *sdp, int silent)
void gfs2_lm_unmount(struct gfs2_sbd *sdp)
{
const struct lm_lockops *lm = sdp->sd_lockstruct.ls_ops;
- if (!gfs2_withdrawing_or_withdrawn(sdp) && lm->lm_unmount)
- lm->lm_unmount(sdp);
+ if (!gfs2_withdrawn(sdp) && lm->lm_unmount)
+ lm->lm_unmount(sdp, true);
}
static int wait_on_journal(struct gfs2_sbd *sdp)
@@ -1215,6 +1215,8 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc)
if (error)
goto fail_debug;
+ INIT_WORK(&sdp->sd_withdraw_work, gfs2_withdraw_func);
+
error = init_locking(sdp, &mount_gh, DO);
if (error)
goto fail_lm;
@@ -1281,7 +1283,6 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc)
if (error) {
gfs2_freeze_unlock(sdp);
- gfs2_destroy_threads(sdp);
fs_err(sdp, "can't make FS RW: %d\n", error);
goto fail_per_node;
}
@@ -1291,6 +1292,7 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc)
fail_per_node:
init_per_node(sdp, UNDO);
+ gfs2_destroy_threads(sdp);
fail_inodes:
init_inodes(sdp, UNDO);
fail_sb:
diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c
index 4620f586411a..56dab2f0ae01 100644
--- a/fs/gfs2/quota.c
+++ b/fs/gfs2/quota.c
@@ -125,7 +125,7 @@ static void gfs2_qd_dispose(struct gfs2_quota_data *qd)
hlist_bl_del_rcu(&qd->qd_hlist);
spin_unlock_bucket(qd->qd_hash);
- if (!gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (!gfs2_withdrawn(sdp)) {
gfs2_assert_warn(sdp, !qd->qd_change);
gfs2_assert_warn(sdp, !qd->qd_slot_ref);
gfs2_assert_warn(sdp, !qd->qd_bh_count);
@@ -1564,27 +1564,13 @@ static void quotad_error(struct gfs2_sbd *sdp, const char *msg, int error)
{
if (error == 0 || error == -EROFS)
return;
- if (!gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (!gfs2_withdrawn(sdp)) {
if (!cmpxchg(&sdp->sd_log_error, 0, error))
fs_err(sdp, "gfs2_quotad: %s error %d\n", msg, error);
wake_up(&sdp->sd_logd_waitq);
}
}
-static void quotad_check_timeo(struct gfs2_sbd *sdp, const char *msg,
- int (*fxn)(struct super_block *sb, int type),
- unsigned long t, unsigned long *timeo,
- unsigned int *new_timeo)
-{
- if (t >= *timeo) {
- int error = fxn(sdp->sd_vfs, 0);
- quotad_error(sdp, msg, error);
- *timeo = gfs2_tune_get_i(&sdp->sd_tune, new_timeo) * HZ;
- } else {
- *timeo -= t;
- }
-}
-
void gfs2_wake_up_statfs(struct gfs2_sbd *sdp) {
if (!sdp->sd_statfs_force_sync) {
sdp->sd_statfs_force_sync = 1;
@@ -1602,36 +1588,46 @@ void gfs2_wake_up_statfs(struct gfs2_sbd *sdp) {
int gfs2_quotad(void *data)
{
struct gfs2_sbd *sdp = data;
- struct gfs2_tune *tune = &sdp->sd_tune;
- unsigned long statfs_timeo = 0;
- unsigned long quotad_timeo = 0;
- unsigned long t = 0;
+ unsigned long now = jiffies;
+ unsigned long statfs_deadline = now;
+ unsigned long quotad_deadline = now;
set_freezable();
while (!kthread_should_stop()) {
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ unsigned long t;
+
+ if (gfs2_withdrawn(sdp))
break;
- /* Update the master statfs file */
- if (sdp->sd_statfs_force_sync) {
- int error = gfs2_statfs_sync(sdp->sd_vfs, 0);
+ now = jiffies;
+ if (sdp->sd_statfs_force_sync ||
+ time_after(now, statfs_deadline)) {
+ unsigned int quantum;
+ int error;
+
+ /* Update the master statfs file */
+ error = gfs2_statfs_sync(sdp->sd_vfs, 0);
quotad_error(sdp, "statfs", error);
- statfs_timeo = gfs2_tune_get(sdp, gt_statfs_quantum) * HZ;
+
+ quantum = gfs2_tune_get(sdp, gt_statfs_quantum);
+ statfs_deadline = now + quantum * HZ;
}
- else
- quotad_check_timeo(sdp, "statfs", gfs2_statfs_sync, t,
- &statfs_timeo,
- &tune->gt_statfs_quantum);
+ if (time_after(now, quotad_deadline)) {
+ unsigned int quantum;
+ int error;
- /* Update quota file */
- quotad_check_timeo(sdp, "sync", gfs2_quota_sync, t,
- "ad_timeo, &tune->gt_quota_quantum);
+ /* Update the quota file */
+ error = gfs2_quota_sync(sdp->sd_vfs, 0);
+ quotad_error(sdp, "sync", error);
- t = min(quotad_timeo, statfs_timeo);
+ quantum = gfs2_tune_get(sdp, gt_quota_quantum);
+ quotad_deadline = now + quantum * HZ;
+ }
- t -= wait_event_freezable_timeout(sdp->sd_quota_wait,
+ t = min(statfs_deadline - now, quotad_deadline - now);
+ wait_event_freezable_timeout(sdp->sd_quota_wait,
sdp->sd_statfs_force_sync ||
- gfs2_withdrawing_or_withdrawn(sdp) ||
+ gfs2_withdrawn(sdp) ||
kthread_should_stop(),
t);
diff --git a/fs/gfs2/recovery.c b/fs/gfs2/recovery.c
index 24250478b085..4873b1c0e1bd 100644
--- a/fs/gfs2/recovery.c
+++ b/fs/gfs2/recovery.c
@@ -408,7 +408,7 @@ void gfs2_recover_func(struct work_struct *work)
int error = 0;
int jlocked = 0;
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
fs_err(sdp, "jid=%u: Recovery not attempted due to withdraw.\n",
jd->jd_jid);
goto fail;
diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c
index 7b5fcd78a3a1..e82a7cac00df 100644
--- a/fs/gfs2/super.c
+++ b/fs/gfs2/super.c
@@ -137,7 +137,7 @@ int gfs2_make_fs_rw(struct gfs2_sbd *sdp)
int error;
j_gl->gl_ops->go_inval(j_gl, DIO_METADATA);
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
return -EIO;
if (sdp->sd_log_sequence == 0) {
@@ -147,8 +147,10 @@ int gfs2_make_fs_rw(struct gfs2_sbd *sdp)
}
error = gfs2_quota_init(sdp);
- if (!error && gfs2_withdrawing_or_withdrawn(sdp))
+ if (!error && gfs2_withdrawn(sdp)) {
+ gfs2_quota_cleanup(sdp);
error = -EIO;
+ }
if (!error)
set_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
return error;
@@ -491,7 +493,7 @@ static void gfs2_dirty_inode(struct inode *inode, int flags)
if (unlikely(!ip->i_gl))
return;
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
return;
if (!gfs2_glock_is_locked_by_me(ip->i_gl)) {
ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_EXCLUSIVE, 0, &gh);
@@ -597,13 +599,13 @@ static void gfs2_put_super(struct super_block *sb)
if (!sb_rdonly(sb))
gfs2_make_fs_ro(sdp);
else {
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
gfs2_destroy_threads(sdp);
gfs2_quota_cleanup(sdp);
}
- WARN_ON(gfs2_withdrawing(sdp));
+ flush_work(&sdp->sd_withdraw_work);
/* At this point, we're through modifying the disk */
@@ -777,7 +779,7 @@ static int gfs2_freeze_fs(struct super_block *sb)
if (test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags)) {
gfs2_log_flush(sdp, NULL, GFS2_LOG_HEAD_FLUSH_FREEZE |
GFS2_LFC_FREEZE_GO_SYNC);
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
return -EIO;
}
return 0;
diff --git a/fs/gfs2/sys.c b/fs/gfs2/sys.c
index c3c8842920d2..5439a65549af 100644
--- a/fs/gfs2/sys.c
+++ b/fs/gfs2/sys.c
@@ -59,7 +59,7 @@ static struct kset *gfs2_kset;
static ssize_t id_show(struct gfs2_sbd *sdp, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%u:%u\n",
+ return sysfs_emit(buf, "%u:%u\n",
MAJOR(sdp->sd_vfs->s_dev), MINOR(sdp->sd_vfs->s_dev));
}
@@ -68,7 +68,7 @@ static ssize_t status_show(struct gfs2_sbd *sdp, char *buf)
unsigned long f = sdp->sd_flags;
ssize_t s;
- s = snprintf(buf, PAGE_SIZE,
+ s = sysfs_emit(buf,
"Journal Checked: %d\n"
"Journal Live: %d\n"
"Journal ID: %d\n"
@@ -84,7 +84,6 @@ static ssize_t status_show(struct gfs2_sbd *sdp, char *buf)
"Force AIL Flush: %d\n"
"FS Freeze Initiator: %d\n"
"FS Frozen: %d\n"
- "Withdrawing: %d\n"
"Withdraw In Prog: %d\n"
"Remote Withdraw: %d\n"
"Withdraw Recovery: %d\n"
@@ -117,7 +116,6 @@ static ssize_t status_show(struct gfs2_sbd *sdp, char *buf)
test_bit(SDF_FORCE_AIL_FLUSH, &f),
test_bit(SDF_FREEZE_INITIATOR, &f),
test_bit(SDF_FROZEN, &f),
- test_bit(SDF_WITHDRAWING, &f),
test_bit(SDF_WITHDRAW_IN_PROG, &f),
test_bit(SDF_REMOTE_WITHDRAW, &f),
test_bit(SDF_WITHDRAW_RECOVERY, &f),
@@ -140,7 +138,7 @@ static ssize_t status_show(struct gfs2_sbd *sdp, char *buf)
static ssize_t fsname_show(struct gfs2_sbd *sdp, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%s\n", sdp->sd_fsname);
+ return sysfs_emit(buf, "%s\n", sdp->sd_fsname);
}
static ssize_t uuid_show(struct gfs2_sbd *sdp, char *buf)
@@ -150,7 +148,7 @@ static ssize_t uuid_show(struct gfs2_sbd *sdp, char *buf)
buf[0] = '\0';
if (uuid_is_null(&s->s_uuid))
return 0;
- return snprintf(buf, PAGE_SIZE, "%pUB\n", &s->s_uuid);
+ return sysfs_emit(buf, "%pUB\n", &s->s_uuid);
}
static ssize_t freeze_show(struct gfs2_sbd *sdp, char *buf)
@@ -158,7 +156,7 @@ static ssize_t freeze_show(struct gfs2_sbd *sdp, char *buf)
struct super_block *sb = sdp->sd_vfs;
int frozen = (sb->s_writers.frozen == SB_UNFROZEN) ? 0 : 1;
- return snprintf(buf, PAGE_SIZE, "%d\n", frozen);
+ return sysfs_emit(buf, "%d\n", frozen);
}
static ssize_t freeze_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
@@ -193,8 +191,8 @@ static ssize_t freeze_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
static ssize_t withdraw_show(struct gfs2_sbd *sdp, char *buf)
{
- unsigned int b = gfs2_withdrawing_or_withdrawn(sdp);
- return snprintf(buf, PAGE_SIZE, "%u\n", b);
+ unsigned int b = gfs2_withdrawn(sdp);
+ return sysfs_emit(buf, "%u\n", b);
}
static ssize_t withdraw_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
@@ -397,7 +395,7 @@ static struct kobj_type gfs2_ktype = {
static ssize_t proto_name_show(struct gfs2_sbd *sdp, char *buf)
{
const struct lm_lockops *ops = sdp->sd_lockstruct.ls_ops;
- return sprintf(buf, "%s\n", ops->lm_proto_name);
+ return sysfs_emit(buf, "%s\n", ops->lm_proto_name);
}
static ssize_t block_show(struct gfs2_sbd *sdp, char *buf)
@@ -408,7 +406,7 @@ static ssize_t block_show(struct gfs2_sbd *sdp, char *buf)
if (test_bit(DFL_BLOCK_LOCKS, &ls->ls_recover_flags))
val = 1;
- ret = sprintf(buf, "%d\n", val);
+ ret = sysfs_emit(buf, "%d\n", val);
return ret;
}
@@ -459,7 +457,7 @@ static ssize_t wdack_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
static ssize_t lkfirst_show(struct gfs2_sbd *sdp, char *buf)
{
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
- return sprintf(buf, "%d\n", ls->ls_first);
+ return sysfs_emit(buf, "%d\n", ls->ls_first);
}
static ssize_t lkfirst_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
@@ -492,7 +490,7 @@ static ssize_t lkfirst_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
static ssize_t first_done_show(struct gfs2_sbd *sdp, char *buf)
{
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
- return sprintf(buf, "%d\n", !!test_bit(DFL_FIRST_MOUNT_DONE, &ls->ls_recover_flags));
+ return sysfs_emit(buf, "%d\n", !!test_bit(DFL_FIRST_MOUNT_DONE, &ls->ls_recover_flags));
}
int gfs2_recover_set(struct gfs2_sbd *sdp, unsigned jid)
@@ -550,18 +548,18 @@ static ssize_t recover_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
static ssize_t recover_done_show(struct gfs2_sbd *sdp, char *buf)
{
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
- return sprintf(buf, "%d\n", ls->ls_recover_jid_done);
+ return sysfs_emit(buf, "%d\n", ls->ls_recover_jid_done);
}
static ssize_t recover_status_show(struct gfs2_sbd *sdp, char *buf)
{
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
- return sprintf(buf, "%d\n", ls->ls_recover_jid_status);
+ return sysfs_emit(buf, "%d\n", ls->ls_recover_jid_status);
}
static ssize_t jid_show(struct gfs2_sbd *sdp, char *buf)
{
- return sprintf(buf, "%d\n", sdp->sd_lockstruct.ls_jid);
+ return sysfs_emit(buf, "%d\n", sdp->sd_lockstruct.ls_jid);
}
static ssize_t jid_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
@@ -626,7 +624,7 @@ static struct attribute *lock_module_attrs[] = {
static ssize_t quota_scale_show(struct gfs2_sbd *sdp, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%u %u\n",
+ return sysfs_emit(buf, "%u %u\n",
sdp->sd_tune.gt_quota_scale_num,
sdp->sd_tune.gt_quota_scale_den);
}
@@ -679,7 +677,7 @@ static struct gfs2_attr tune_attr_##name = __ATTR(name, 0644, show, store)
#define TUNE_ATTR_2(name, store) \
static ssize_t name##_show(struct gfs2_sbd *sdp, char *buf) \
{ \
- return snprintf(buf, PAGE_SIZE, "%u\n", sdp->sd_tune.gt_##name); \
+ return sysfs_emit(buf, "%u\n", sdp->sd_tune.gt_##name); \
} \
TUNE_ATTR_3(name, name##_show, store)
diff --git a/fs/gfs2/trans.c b/fs/gfs2/trans.c
index 075f7e9abe47..f0cfdb5c5f2d 100644
--- a/fs/gfs2/trans.c
+++ b/fs/gfs2/trans.c
@@ -289,7 +289,7 @@ void gfs2_trans_add_meta(struct gfs2_glock *gl, struct buffer_head *bh)
(unsigned long long)bd->bd_bh->b_blocknr);
BUG();
}
- if (gfs2_withdrawing_or_withdrawn(sdp)) {
+ if (gfs2_withdrawn(sdp)) {
fs_info(sdp, "GFS2:adding buf while withdrawn! 0x%llx\n",
(unsigned long long)bd->bd_bh->b_blocknr);
goto out_unlock;
diff --git a/fs/gfs2/util.c b/fs/gfs2/util.c
index 56412f63f3bb..4c3b44eafc9f 100644
--- a/fs/gfs2/util.c
+++ b/fs/gfs2/util.c
@@ -309,43 +309,50 @@ void gfs2_lm(struct gfs2_sbd *sdp, const char *fmt, ...)
va_end(args);
}
-void gfs2_withdraw(struct gfs2_sbd *sdp)
+void gfs2_withdraw_func(struct work_struct *work)
{
+ struct gfs2_sbd *sdp = container_of(work, struct gfs2_sbd, sd_withdraw_work);
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
const struct lm_lockops *lm = ls->ls_ops;
+ if (test_bit(SDF_KILL, &sdp->sd_flags))
+ return;
+
+ BUG_ON(sdp->sd_args.ar_debug);
+
+ signal_our_withdraw(sdp);
+
+ kobject_uevent(&sdp->sd_kobj, KOBJ_OFFLINE);
+
+ if (!strcmp(sdp->sd_lockstruct.ls_ops->lm_proto_name, "lock_dlm"))
+ wait_for_completion(&sdp->sd_wdack);
+
+ if (lm->lm_unmount)
+ lm->lm_unmount(sdp, false);
+ fs_err(sdp, "file system withdrawn\n");
+ clear_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags);
+}
+
+void gfs2_withdraw(struct gfs2_sbd *sdp)
+{
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW) {
unsigned long old = READ_ONCE(sdp->sd_flags), new;
do {
- if (old & BIT(SDF_WITHDRAWN)) {
- wait_on_bit(&sdp->sd_flags,
- SDF_WITHDRAW_IN_PROG,
- TASK_UNINTERRUPTIBLE);
+ if (old & BIT(SDF_WITHDRAWN))
return;
- }
new = old | BIT(SDF_WITHDRAWN) | BIT(SDF_WITHDRAW_IN_PROG);
} while (unlikely(!try_cmpxchg(&sdp->sd_flags, &old, new)));
- fs_err(sdp, "about to withdraw this file system\n");
- BUG_ON(sdp->sd_args.ar_debug);
-
- signal_our_withdraw(sdp);
-
- kobject_uevent(&sdp->sd_kobj, KOBJ_OFFLINE);
-
- if (!strcmp(sdp->sd_lockstruct.ls_ops->lm_proto_name, "lock_dlm"))
- wait_for_completion(&sdp->sd_wdack);
-
- if (lm->lm_unmount) {
- fs_err(sdp, "telling LM to unmount\n");
- lm->lm_unmount(sdp);
- }
- fs_err(sdp, "File system withdrawn\n");
dump_stack();
- clear_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags);
- smp_mb__after_atomic();
- wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_IN_PROG);
+ /*
+ * There is no need to withdraw when the superblock hasn't been
+ * fully initialized, yet.
+ */
+ if (!(sdp->sd_vfs->s_flags & SB_BORN))
+ return;
+ fs_err(sdp, "about to withdraw this file system\n");
+ schedule_work(&sdp->sd_withdraw_work);
}
if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
@@ -357,10 +364,9 @@ void gfs2_withdraw(struct gfs2_sbd *sdp)
*/
void gfs2_assert_withdraw_i(struct gfs2_sbd *sdp, char *assertion,
- const char *function, char *file, unsigned int line,
- bool delayed)
+ const char *function, char *file, unsigned int line)
{
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
return;
fs_err(sdp,
@@ -368,17 +374,7 @@ void gfs2_assert_withdraw_i(struct gfs2_sbd *sdp, char *assertion,
"function = %s, file = %s, line = %u\n",
assertion, function, file, line);
- /*
- * If errors=panic was specified on mount, it won't help to delay the
- * withdraw.
- */
- if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
- delayed = false;
-
- if (delayed)
- gfs2_withdraw_delayed(sdp);
- else
- gfs2_withdraw(sdp);
+ gfs2_withdraw(sdp);
dump_stack();
}
@@ -528,7 +524,7 @@ void gfs2_io_error_bh_i(struct gfs2_sbd *sdp, struct buffer_head *bh,
const char *function, char *file, unsigned int line,
bool withdraw)
{
- if (gfs2_withdrawing_or_withdrawn(sdp))
+ if (gfs2_withdrawn(sdp))
return;
fs_err(sdp, "fatal: I/O error - "
diff --git a/fs/gfs2/util.h b/fs/gfs2/util.h
index da0373b1e82b..832c6c6eef41 100644
--- a/fs/gfs2/util.h
+++ b/fs/gfs2/util.h
@@ -37,24 +37,14 @@ do { \
void gfs2_assert_withdraw_i(struct gfs2_sbd *sdp, char *assertion,
- const char *function, char *file, unsigned int line,
- bool delayed);
+ const char *function, char *file, unsigned int line);
#define gfs2_assert_withdraw(sdp, assertion) \
({ \
bool _bool = (assertion); \
if (unlikely(!_bool)) \
gfs2_assert_withdraw_i((sdp), #assertion, \
- __func__, __FILE__, __LINE__, false); \
- !_bool; \
- })
-
-#define gfs2_assert_withdraw_delayed(sdp, assertion) \
- ({ \
- bool _bool = (assertion); \
- if (unlikely(!_bool)) \
- gfs2_assert_withdraw_i((sdp), #assertion, \
- __func__, __FILE__, __LINE__, true); \
+ __func__, __FILE__, __LINE__); \
!_bool; \
})
@@ -193,33 +183,12 @@ static inline unsigned int gfs2_tune_get_i(struct gfs2_tune *gt,
}
/**
- * gfs2_withdraw_delayed - withdraw as soon as possible without deadlocks
+ * gfs2_withdrawn - test whether the file system is withdrawn
* @sdp: the superblock
*/
-static inline void gfs2_withdraw_delayed(struct gfs2_sbd *sdp)
+static inline bool gfs2_withdrawn(struct gfs2_sbd *sdp)
{
- set_bit(SDF_WITHDRAWING, &sdp->sd_flags);
-}
-
-/**
- * gfs2_withdrawing_or_withdrawn - test whether the file system is withdrawing
- * or withdrawn
- * @sdp: the superblock
- */
-static inline bool gfs2_withdrawing_or_withdrawn(struct gfs2_sbd *sdp)
-{
- return unlikely(test_bit(SDF_WITHDRAWN, &sdp->sd_flags) ||
- test_bit(SDF_WITHDRAWING, &sdp->sd_flags));
-}
-
-/**
- * gfs2_withdrawing - check if a withdraw is pending
- * @sdp: the superblock
- */
-static inline bool gfs2_withdrawing(struct gfs2_sbd *sdp)
-{
- return unlikely(test_bit(SDF_WITHDRAWING, &sdp->sd_flags) &&
- !test_bit(SDF_WITHDRAWN, &sdp->sd_flags));
+ return unlikely(test_bit(SDF_WITHDRAWN, &sdp->sd_flags));
}
static inline bool gfs2_withdraw_in_prog(struct gfs2_sbd *sdp)
@@ -232,6 +201,8 @@ gfs2_tune_get_i(&(sdp)->sd_tune, &(sdp)->sd_tune.field)
__printf(2, 3)
void gfs2_lm(struct gfs2_sbd *sdp, const char *fmt, ...);
+
+void gfs2_withdraw_func(struct work_struct *work);
void gfs2_withdraw(struct gfs2_sbd *sdp);
#endif /* __UTIL_DOT_H__ */
diff --git a/fs/hfs/dir.c b/fs/hfs/dir.c
index 130c2f3a417f..e585471823fe 100644
--- a/fs/hfs/dir.c
+++ b/fs/hfs/dir.c
@@ -190,8 +190,8 @@ static int hfs_create(struct mnt_idmap *idmap, struct inode *dir,
int res;
inode = hfs_new_inode(dir, &dentry->d_name, mode);
- if (!inode)
- return -ENOMEM;
+ if (IS_ERR(inode))
+ return PTR_ERR(inode);
res = hfs_cat_create(inode->i_ino, dir, &dentry->d_name, inode);
if (res) {
@@ -220,8 +220,8 @@ static struct dentry *hfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
int res;
inode = hfs_new_inode(dir, &dentry->d_name, S_IFDIR | mode);
- if (!inode)
- return ERR_PTR(-ENOMEM);
+ if (IS_ERR(inode))
+ return ERR_CAST(inode);
res = hfs_cat_create(inode->i_ino, dir, &dentry->d_name, inode);
if (res) {
@@ -248,11 +248,18 @@ static struct dentry *hfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
*/
static int hfs_remove(struct inode *dir, struct dentry *dentry)
{
+ struct super_block *sb = dir->i_sb;
struct inode *inode = d_inode(dentry);
int res;
if (S_ISDIR(inode->i_mode) && inode->i_size != 2)
return -ENOTEMPTY;
+
+ if (unlikely(!is_hfs_cnid_counts_valid(sb))) {
+ pr_err("cannot remove file/folder\n");
+ return -ERANGE;
+ }
+
res = hfs_cat_delete(inode->i_ino, dir, &dentry->d_name);
if (res)
return res;
diff --git a/fs/hfs/hfs_fs.h b/fs/hfs/hfs_fs.h
index d9665226f547..86b5ae62247b 100644
--- a/fs/hfs/hfs_fs.h
+++ b/fs/hfs/hfs_fs.h
@@ -198,6 +198,7 @@ extern void hfs_delete_inode(struct inode *inode);
extern const struct xattr_handler * const hfs_xattr_handlers[];
/* mdb.c */
+extern bool is_hfs_cnid_counts_valid(struct super_block *sb);
extern int hfs_mdb_get(struct super_block *sb);
extern void hfs_mdb_commit(struct super_block *sb);
extern void hfs_mdb_close(struct super_block *sb);
diff --git a/fs/hfs/inode.c b/fs/hfs/inode.c
index cd3dd5bd456e..f1796e4b07e8 100644
--- a/fs/hfs/inode.c
+++ b/fs/hfs/inode.c
@@ -187,14 +187,21 @@ struct inode *hfs_new_inode(struct inode *dir, const struct qstr *name, umode_t
s64 next_id;
s64 file_count;
s64 folder_count;
+ int err = -ENOMEM;
if (!inode)
- return NULL;
+ goto out_err;
+
+ err = -ERANGE;
mutex_init(&HFS_I(inode)->extents_lock);
hfs_cat_build_key(sb, (btree_key *)&HFS_I(inode)->cat_key, dir->i_ino, name);
next_id = atomic64_inc_return(&HFS_SB(sb)->next_id);
- BUG_ON(next_id > U32_MAX);
+ if (next_id > U32_MAX) {
+ atomic64_dec(&HFS_SB(sb)->next_id);
+ pr_err("cannot create new inode: next CNID exceeds limit\n");
+ goto out_discard;
+ }
inode->i_ino = (u32)next_id;
inode->i_mode = mode;
inode->i_uid = current_fsuid();
@@ -208,7 +215,11 @@ struct inode *hfs_new_inode(struct inode *dir, const struct qstr *name, umode_t
if (S_ISDIR(mode)) {
inode->i_size = 2;
folder_count = atomic64_inc_return(&HFS_SB(sb)->folder_count);
- BUG_ON(folder_count > U32_MAX);
+ if (folder_count> U32_MAX) {
+ atomic64_dec(&HFS_SB(sb)->folder_count);
+ pr_err("cannot create new inode: folder count exceeds limit\n");
+ goto out_discard;
+ }
if (dir->i_ino == HFS_ROOT_CNID)
HFS_SB(sb)->root_dirs++;
inode->i_op = &hfs_dir_inode_operations;
@@ -218,7 +229,11 @@ struct inode *hfs_new_inode(struct inode *dir, const struct qstr *name, umode_t
} else if (S_ISREG(mode)) {
HFS_I(inode)->clump_blocks = HFS_SB(sb)->clumpablks;
file_count = atomic64_inc_return(&HFS_SB(sb)->file_count);
- BUG_ON(file_count > U32_MAX);
+ if (file_count > U32_MAX) {
+ atomic64_dec(&HFS_SB(sb)->file_count);
+ pr_err("cannot create new inode: file count exceeds limit\n");
+ goto out_discard;
+ }
if (dir->i_ino == HFS_ROOT_CNID)
HFS_SB(sb)->root_files++;
inode->i_op = &hfs_file_inode_operations;
@@ -242,6 +257,11 @@ struct inode *hfs_new_inode(struct inode *dir, const struct qstr *name, umode_t
hfs_mark_mdb_dirty(sb);
return inode;
+
+ out_discard:
+ iput(inode);
+ out_err:
+ return ERR_PTR(err);
}
void hfs_delete_inode(struct inode *inode)
@@ -250,7 +270,6 @@ void hfs_delete_inode(struct inode *inode)
hfs_dbg("ino %lu\n", inode->i_ino);
if (S_ISDIR(inode->i_mode)) {
- BUG_ON(atomic64_read(&HFS_SB(sb)->folder_count) > U32_MAX);
atomic64_dec(&HFS_SB(sb)->folder_count);
if (HFS_I(inode)->cat_key.ParID == cpu_to_be32(HFS_ROOT_CNID))
HFS_SB(sb)->root_dirs--;
@@ -259,7 +278,6 @@ void hfs_delete_inode(struct inode *inode)
return;
}
- BUG_ON(atomic64_read(&HFS_SB(sb)->file_count) > U32_MAX);
atomic64_dec(&HFS_SB(sb)->file_count);
if (HFS_I(inode)->cat_key.ParID == cpu_to_be32(HFS_ROOT_CNID))
HFS_SB(sb)->root_files--;
diff --git a/fs/hfs/mdb.c b/fs/hfs/mdb.c
index f28cd24dee84..a97cea35ca2e 100644
--- a/fs/hfs/mdb.c
+++ b/fs/hfs/mdb.c
@@ -64,6 +64,27 @@ static int hfs_get_last_session(struct super_block *sb,
return 0;
}
+bool is_hfs_cnid_counts_valid(struct super_block *sb)
+{
+ struct hfs_sb_info *sbi = HFS_SB(sb);
+ bool corrupted = false;
+
+ if (unlikely(atomic64_read(&sbi->next_id) > U32_MAX)) {
+ pr_warn("next CNID exceeds limit\n");
+ corrupted = true;
+ }
+ if (unlikely(atomic64_read(&sbi->file_count) > U32_MAX)) {
+ pr_warn("file count exceeds limit\n");
+ corrupted = true;
+ }
+ if (unlikely(atomic64_read(&sbi->folder_count) > U32_MAX)) {
+ pr_warn("folder count exceeds limit\n");
+ corrupted = true;
+ }
+
+ return !corrupted;
+}
+
/*
* hfs_mdb_get()
*
@@ -159,6 +180,11 @@ int hfs_mdb_get(struct super_block *sb)
atomic64_set(&HFS_SB(sb)->file_count, be32_to_cpu(mdb->drFilCnt));
atomic64_set(&HFS_SB(sb)->folder_count, be32_to_cpu(mdb->drDirCnt));
+ if (!is_hfs_cnid_counts_valid(sb)) {
+ pr_warn("filesystem possibly corrupted, running fsck.hfs is recommended. Mounting read-only.\n");
+ sb->s_flags |= SB_RDONLY;
+ }
+
/* TRY to get the alternate (backup) MDB. */
sect = part_start + part_size - 2;
bh = sb_bread512(sb, sect, mdb2);
@@ -212,7 +238,7 @@ int hfs_mdb_get(struct super_block *sb)
attrib = mdb->drAtrb;
if (!(attrib & cpu_to_be16(HFS_SB_ATTRIB_UNMNT))) {
- pr_warn("filesystem was not cleanly unmounted, running fsck.hfs is recommended. mounting read-only.\n");
+ pr_warn("filesystem was not cleanly unmounted, running fsck.hfs is recommended. Mounting read-only.\n");
sb->s_flags |= SB_RDONLY;
}
if ((attrib & cpu_to_be16(HFS_SB_ATTRIB_SLOCK))) {
@@ -270,15 +296,12 @@ void hfs_mdb_commit(struct super_block *sb)
/* These parameters may have been modified, so write them back */
mdb->drLsMod = hfs_mtime();
mdb->drFreeBks = cpu_to_be16(HFS_SB(sb)->free_ablocks);
- BUG_ON(atomic64_read(&HFS_SB(sb)->next_id) > U32_MAX);
mdb->drNxtCNID =
cpu_to_be32((u32)atomic64_read(&HFS_SB(sb)->next_id));
mdb->drNmFls = cpu_to_be16(HFS_SB(sb)->root_files);
mdb->drNmRtDirs = cpu_to_be16(HFS_SB(sb)->root_dirs);
- BUG_ON(atomic64_read(&HFS_SB(sb)->file_count) > U32_MAX);
mdb->drFilCnt =
cpu_to_be32((u32)atomic64_read(&HFS_SB(sb)->file_count));
- BUG_ON(atomic64_read(&HFS_SB(sb)->folder_count) > U32_MAX);
mdb->drDirCnt =
cpu_to_be32((u32)atomic64_read(&HFS_SB(sb)->folder_count));
diff --git a/fs/hfs/super.c b/fs/hfs/super.c
index df289cbdd4e8..97546d6b41f4 100644
--- a/fs/hfs/super.c
+++ b/fs/hfs/super.c
@@ -34,6 +34,7 @@ MODULE_LICENSE("GPL");
static int hfs_sync_fs(struct super_block *sb, int wait)
{
+ is_hfs_cnid_counts_valid(sb);
hfs_mdb_commit(sb);
return 0;
}
@@ -65,6 +66,8 @@ static void flush_mdb(struct work_struct *work)
sbi->work_queued = 0;
spin_unlock(&sbi->work_lock);
+ is_hfs_cnid_counts_valid(sb);
+
hfs_mdb_commit(sb);
}
diff --git a/fs/inode.c b/fs/inode.c
index 44891fb7cd43..24762fd1f2f7 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -745,21 +745,18 @@ void dump_mapping(const struct address_space *mapping)
void clear_inode(struct inode *inode)
{
/*
- * We have to cycle the i_pages lock here because reclaim can be in the
- * process of removing the last page (in __filemap_remove_folio())
- * and we must not free the mapping under it.
+ * We have to cycle the i_pages lock here because reclaim
+ * can be in the process of removing the last page (in
+ * __filemap_remove_folio()) and we must not free the mapping
+ * under it. We also remove nodes which are empty; these
+ * can occur in two different ways. The first is that radix
+ * tree expansion can fail partway and the second is that THP
+ * collapse_file() can allocate some temporary nodes and not
+ * clean them up.
*/
- xa_lock_irq(&inode->i_data.i_pages);
+ xa_destroy(&inode->i_data.i_pages);
+
BUG_ON(inode->i_data.nrpages);
- /*
- * Almost always, mapping_empty(&inode->i_data) here; but there are
- * two known and long-standing ways in which nodes may get left behind
- * (when deep radix-tree node allocation failed partway; or when THP
- * collapse_file() failed). Until those two known cases are cleaned up,
- * or a cleanup function is called here, do not BUG_ON(!mapping_empty),
- * nor even WARN_ON(!mapping_empty).
- */
- xa_unlock_irq(&inode->i_data.i_pages);
BUG_ON(!list_empty(&inode->i_data.i_private_list));
BUG_ON(!(inode->i_state & I_FREEING));
BUG_ON(inode->i_state & I_CLEAR);
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index 92f32e02f77f..bb276321c952 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -470,6 +470,7 @@ static int iomap_dio_bio_iter(struct iomap_iter *iter, struct iomap_dio *dio)
* bio_iov_iter_get_pages() returns an error, above.
*/
ret = -EINVAL;
+ bio_release_pages(bio, false);
bio_put(bio);
goto zero_tail;
}
diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c
index d68afa196535..28b9886394c8 100644
--- a/fs/lockd/svc.c
+++ b/fs/lockd/svc.c
@@ -48,7 +48,7 @@
static struct svc_program nlmsvc_program;
-const struct nlmsvc_binding *nlmsvc_ops;
+const struct nlmsvc_binding __rcu *nlmsvc_ops;
EXPORT_SYMBOL_GPL(nlmsvc_ops);
static DEFINE_MUTEX(nlmsvc_mutex);
@@ -143,7 +143,7 @@ lockd(void *vrqstp)
nlmsvc_retry_blocked(rqstp);
svc_recv(rqstp);
}
- if (nlmsvc_ops)
+ if (rcu_access_pointer(nlmsvc_ops))
nlmsvc_invalidate_all();
nlm_shutdown_hosts();
cancel_delayed_work_sync(&ln->grace_period_end);
diff --git a/fs/lockd/svc4proc.c b/fs/lockd/svc4proc.c
index 75e020a8bfd0..a1adfd66dd23 100644
--- a/fs/lockd/svc4proc.c
+++ b/fs/lockd/svc4proc.c
@@ -31,7 +31,7 @@ nlm4svc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp,
__be32 error = 0;
/* nfsd callbacks must have been installed for this procedure */
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return nlm_lck_denied_nolocks;
if (lock->lock_start > OFFSET_MAX ||
@@ -484,7 +484,7 @@ nlm4svc_proc_granted_res(struct svc_rqst *rqstp)
{
struct nlm_res *argp = rqstp->rq_argp;
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return rpc_success;
dprintk("lockd: GRANTED_RES called\n");
diff --git a/fs/lockd/svcproc.c b/fs/lockd/svcproc.c
index d98e8d684376..49d20df3b8ae 100644
--- a/fs/lockd/svcproc.c
+++ b/fs/lockd/svcproc.c
@@ -61,7 +61,7 @@ nlmsvc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp,
__be32 error = 0;
/* nfsd callbacks must have been installed for this procedure */
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return nlm_lck_denied_nolocks;
/* Obtain host handle */
@@ -519,7 +519,7 @@ nlmsvc_proc_granted_res(struct svc_rqst *rqstp)
{
struct nlm_res *argp = rqstp->rq_argp;
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return rpc_success;
dprintk("lockd: GRANTED_RES called\n");
diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
index 4075d0eab263..405debfdf3ee 100644
--- a/fs/lockd/svcsubs.c
+++ b/fs/lockd/svcsubs.c
@@ -90,24 +90,38 @@ int lock_to_openmode(struct file_lock *lock)
static __be32 nlm_do_fopen(struct svc_rqst *rqstp,
struct nlm_file *file, int mode)
{
+ const struct nlmsvc_binding *ops;
__be32 nfserr = nlm_lck_denied_nolocks;
__be32 deferred = 0;
struct file **fp;
int m;
+ rcu_read_lock();
+ ops = rcu_dereference(nlmsvc_ops);
+ if (!ops || !try_module_get(ops->owner)) {
+ rcu_read_unlock();
+ return nlm_lck_denied_nolocks;
+ }
+ rcu_read_unlock();
+
for (m = O_RDONLY ; m <= O_WRONLY ; m++) {
if (mode != O_RDWR && mode != m)
continue;
fp = &file->f_file[m];
- if (*fp)
+ if (*fp) {
+ module_put(ops->owner);
return 0;
- nfserr = nlmsvc_ops->fopen(rqstp, &file->f_handle, fp, m);
- if (!nfserr)
+ }
+ nfserr = ops->fopen(rqstp, &file->f_handle, fp, m);
+ if (!nfserr) {
+ module_put(ops->owner);
return 0;
+ }
if (nfserr == nlm_drop_reply)
deferred = nfserr;
}
+ module_put(ops->owner);
if (deferred)
return deferred;
dprintk("lockd: open failed (error %d)\n", ntohl(nfserr));
@@ -174,6 +188,33 @@ nlm_lookup_file(struct svc_rqst *rqstp, struct nlm_file **result,
goto out_unlock;
}
+/*
+ * Release the struct file references held by a nlm_file.
+ */
+static void nlm_release_files(struct nlm_file *file)
+{
+ const struct nlmsvc_binding *ops;
+ bool have_ops;
+
+ rcu_read_lock();
+ ops = rcu_dereference(nlmsvc_ops);
+ have_ops = ops && try_module_get(ops->owner);
+ rcu_read_unlock();
+
+ if (have_ops) {
+ if (file->f_file[O_RDONLY])
+ ops->fclose(file->f_file[O_RDONLY]);
+ if (file->f_file[O_WRONLY])
+ ops->fclose(file->f_file[O_WRONLY]);
+ module_put(ops->owner);
+ } else {
+ if (file->f_file[O_RDONLY])
+ fput(file->f_file[O_RDONLY]);
+ if (file->f_file[O_WRONLY])
+ fput(file->f_file[O_WRONLY]);
+ }
+}
+
/*
* Delete a file after having released all locks, blocks and shares
*/
@@ -183,10 +224,7 @@ nlm_delete_file(struct nlm_file *file)
nlm_debug_print_file("closing file", file);
if (!hlist_unhashed(&file->f_list)) {
hlist_del(&file->f_list);
- if (file->f_file[O_RDONLY])
- nlmsvc_ops->fclose(file->f_file[O_RDONLY]);
- if (file->f_file[O_WRONLY])
- nlmsvc_ops->fclose(file->f_file[O_WRONLY]);
+ nlm_release_files(file);
kfree(file);
} else {
printk(KERN_WARNING "lockd: attempt to release unknown file!\n");
diff --git a/fs/mount.h b/fs/mount.h
index ceceac13501c..dcf908049f56 100644
--- a/fs/mount.h
+++ b/fs/mount.h
@@ -30,7 +30,8 @@ struct mnt_namespace {
} __randomize_layout;
struct mnt_pcp {
- int mnt_count;
+ unsigned int mnt_gets;
+ unsigned int mnt_puts;
int mnt_writers;
};
@@ -227,6 +228,9 @@ static inline struct mnt_namespace *to_mnt_ns(struct ns_common *ns)
#ifdef CONFIG_FSNOTIFY
static inline void mnt_notify_add(struct mount *m)
{
+ /* queued already under this namespace_sem hold */
+ if (!list_empty(&m->to_notify))
+ return;
/* Optimize the case where there are no watches */
if ((m->mnt_ns && m->mnt_ns->n_fsnotify_marks) ||
(m->prev_ns && m->prev_ns->n_fsnotify_marks))
diff --git a/fs/namei.c b/fs/namei.c
index 003c9a1b2f90..2e86733b4bac 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -1973,6 +1973,33 @@ static const char *pick_link(struct nameidata *nd, struct path *link,
return NULL;
}
+/*
+ * Be careful in case the dentry is unlinked or renamed. Any mounts
+ * stacked on top of it are going away. We need to make sure that we
+ * don't reveal the underyling dentry during refwalk. In rcuwalk we
+ * catch this via d_seq and another lookup for the name. Give the same
+ * guarantee in refwalk.
+ */
+static bool unlink_may_reveal(struct nameidata *nd, int flags,
+ struct path *path, struct dentry *dentry)
+{
+ /* ".." and LOOKUP_DOWN may land on an unhashed directory */
+ if (flags & WALK_NOFOLLOW)
+ return false;
+ if (nd->flags & LOOKUP_REVAL)
+ return false;
+ /* We crossed onto a mount and the name led us here while it still existed */
+ if (path->mnt != nd->path.mnt)
+ return false;
+ /* only a name on its way out is flagged */
+ if (likely(!cant_mount(dentry)))
+ return false;
+ if (!d_unlinked(dentry))
+ return false;
+ dput(no_free_ptr(path->dentry));
+ return true;
+}
+
/*
* Do we need to follow links? We _really_ want to be able
* to do this check without having to look at inode->i_op,
@@ -2001,6 +2028,8 @@ static const char *step_into(struct nameidata *nd, int flags,
if (unlikely(!inode))
return ERR_PTR(-ENOENT);
} else {
+ if (unlikely(unlink_may_reveal(nd, flags, &path, dentry)))
+ return ERR_PTR(-ESTALE);
dput(nd->path.dentry);
if (nd->path.mnt != path.mnt)
mntput(nd->path.mnt);
diff --git a/fs/namespace.c b/fs/namespace.c
index d886ae255e40..3b6d7271eb3d 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -136,7 +136,6 @@ static void mnt_ns_release(struct mnt_namespace *ns)
{
/* keep alive for {list,stat}mount() */
if (ns && refcount_dec_and_test(&ns->passive)) {
- fsnotify_mntns_delete(ns);
put_user_ns(ns->user_ns);
kfree(ns);
}
@@ -255,16 +254,24 @@ void mnt_release_group_id(struct mount *mnt)
mnt->mnt_group_id = 0;
}
-/*
- * vfsmount lock must be held for read
- */
-static inline void mnt_add_count(struct mount *mnt, int n)
+static inline void mnt_inc_count(struct mount *mnt)
+{
+#ifdef CONFIG_SMP
+ this_cpu_inc(mnt->mnt_pcp->mnt_gets);
+#else
+ preempt_disable();
+ mnt->mnt_count++;
+ preempt_enable();
+#endif
+}
+
+static inline void mnt_dec_count(struct mount *mnt)
{
#ifdef CONFIG_SMP
- this_cpu_add(mnt->mnt_pcp->mnt_count, n);
+ this_cpu_inc(mnt->mnt_pcp->mnt_puts);
#else
preempt_disable();
- mnt->mnt_count += n;
+ mnt->mnt_count--;
preempt_enable();
#endif
}
@@ -275,14 +282,17 @@ static inline void mnt_add_count(struct mount *mnt, int n)
int mnt_get_count(struct mount *mnt)
{
#ifdef CONFIG_SMP
- int count = 0;
+ unsigned int gets = 0, puts = 0;
int cpu;
- for_each_possible_cpu(cpu) {
- count += per_cpu_ptr(mnt->mnt_pcp, cpu)->mnt_count;
- }
+ /* puts first, so a put counted here has its get counted below */
+ for_each_possible_cpu(cpu)
+ puts += per_cpu_ptr(mnt->mnt_pcp, cpu)->mnt_puts;
+ smp_mb(); /* pairs with the smp_wmb() in mntput_no_expire() */
+ for_each_possible_cpu(cpu)
+ gets += per_cpu_ptr(mnt->mnt_pcp, cpu)->mnt_gets;
- return count;
+ return gets - puts;
#else
return mnt->mnt_count;
#endif
@@ -311,7 +321,7 @@ static struct mount *alloc_vfsmnt(const char *name)
if (!mnt->mnt_pcp)
goto out_free_devname;
- this_cpu_add(mnt->mnt_pcp->mnt_count, 1);
+ this_cpu_inc(mnt->mnt_pcp->mnt_gets);
#else
mnt->mnt_count = 1;
mnt->mnt_writers = 0;
@@ -327,6 +337,9 @@ static struct mount *alloc_vfsmnt(const char *name)
INIT_HLIST_NODE(&mnt->mnt_slave);
INIT_HLIST_NODE(&mnt->mnt_mp_list);
INIT_HLIST_HEAD(&mnt->mnt_stuck_children);
+#ifdef CONFIG_FSNOTIFY
+ INIT_LIST_HEAD(&mnt->to_notify);
+#endif
RB_CLEAR_NODE(&mnt->mnt_node);
mnt->mnt.mnt_idmap = &nop_mnt_idmap;
}
@@ -751,13 +764,13 @@ int __legitimize_mnt(struct vfsmount *bastard, unsigned seq)
if (bastard == NULL)
return 0;
mnt = real_mount(bastard);
- mnt_add_count(mnt, 1);
- smp_mb(); // see mntput_no_expire() and do_umount()
+ mnt_inc_count(mnt);
+ smp_mb(); /* see mntput_no_expire_slowpath() and do_umount() */
if (likely(!read_seqretry(&mount_lock, seq)))
return 0;
lock_mount_hash();
if (unlikely(bastard->mnt_flags & (MNT_SYNC_UMOUNT | MNT_DOOMED))) {
- mnt_add_count(mnt, -1);
+ mnt_dec_count(mnt);
unlock_mount_hash();
return 1;
}
@@ -1063,7 +1076,10 @@ static void attach_mnt(struct mount *mnt, struct mount *parent,
void mnt_change_mountpoint(struct mount *parent, struct mountpoint *mp, struct mount *mnt)
{
struct mountpoint *old_mp = mnt->mnt_mp;
+ struct mount *old_parent = mnt->mnt_parent;
+ if (old_parent->overmount == mnt)
+ old_parent->overmount = NULL;
list_del_init(&mnt->mnt_child);
hlist_del_init(&mnt->mnt_mp_list);
hlist_del_init_rcu(&mnt->mnt_hash);
@@ -1255,6 +1271,7 @@ static struct mount *clone_mnt(struct mount *old, struct dentry *root,
mnt->mnt.mnt_flags = READ_ONCE(old->mnt.mnt_flags) &
~MNT_INTERNAL_FLAGS;
+ mnt->mnt_t_flags = old->mnt_t_flags & T_UNBINDABLE;
if (flag & (CL_SLAVE | CL_PRIVATE))
mnt->mnt_group_id = 0; /* not a peer of original */
@@ -1336,33 +1353,19 @@ static void delayed_mntput(struct work_struct *unused)
}
static DECLARE_DELAYED_WORK(delayed_mntput_work, delayed_mntput);
-static void mntput_no_expire(struct mount *mnt)
+static void noinline mntput_no_expire_slowpath(struct mount *mnt)
{
LIST_HEAD(list);
int count;
- rcu_read_lock();
- if (likely(READ_ONCE(mnt->mnt_ns))) {
- /*
- * Since we don't do lock_mount_hash() here,
- * ->mnt_ns can change under us. However, if it's
- * non-NULL, then there's a reference that won't
- * be dropped until after an RCU delay done after
- * turning ->mnt_ns NULL. So if we observe it
- * non-NULL under rcu_read_lock(), the reference
- * we are dropping is not the final one.
- */
- mnt_add_count(mnt, -1);
- rcu_read_unlock();
- return;
- }
+ VFS_BUG_ON(mnt->mnt_ns);
lock_mount_hash();
/*
* make sure that if __legitimize_mnt() has not seen us grab
* mount_lock, we'll see their refcount increment here.
*/
smp_mb();
- mnt_add_count(mnt, -1);
+ mnt_dec_count(mnt);
count = mnt_get_count(mnt);
if (count != 0) {
WARN_ON(count < 0);
@@ -1406,6 +1409,27 @@ static void mntput_no_expire(struct mount *mnt)
cleanup_mnt(mnt);
}
+static void mntput_no_expire(struct mount *mnt)
+{
+ rcu_read_lock();
+ if (likely(READ_ONCE(mnt->mnt_ns))) {
+ /*
+ * Since we don't do lock_mount_hash() here,
+ * ->mnt_ns can change under us. However, if it's
+ * non-NULL, then there's a reference that won't
+ * be dropped until after an RCU delay done after
+ * turning ->mnt_ns NULL. So if we observe it
+ * non-NULL under rcu_read_lock(), the reference
+ * we are dropping is not the final one.
+ */
+ smp_wmb(); /* pairs with the smp_mb() in mnt_get_count() */
+ mnt_dec_count(mnt);
+ rcu_read_unlock();
+ return;
+ }
+ mntput_no_expire_slowpath(mnt);
+}
+
void mntput(struct vfsmount *mnt)
{
if (mnt) {
@@ -1421,7 +1445,7 @@ EXPORT_SYMBOL(mntput);
struct vfsmount *mntget(struct vfsmount *mnt)
{
if (mnt)
- mnt_add_count(real_mount(mnt), 1);
+ mnt_inc_count(real_mount(mnt));
return mnt;
}
EXPORT_SYMBOL(mntget);
@@ -1778,6 +1802,8 @@ static void umount_tree(struct mount *mnt, enum umount_tree_flags how)
/* Gather the mounts to umount */
for (p = mnt; p; p = next_mnt(p, mnt)) {
+ /* A mount is unmounted once. */
+ VFS_WARN_ON_ONCE(p->mnt.mnt_flags & MNT_UMOUNT);
p->mnt.mnt_flags |= MNT_UMOUNT;
if (mnt_ns_attached(p))
move_from_ns(p);
@@ -1971,6 +1997,9 @@ static int do_umount(struct mount *mnt, int flags)
* detach_mounts allows lazily unmounting those mounts instead of
* leaking them.
*
+ * The dentry is unhashed before the mounts go so that no lookup finds
+ * what they covered. The caller removes it for good afterwards.
+ *
* The caller may hold dentry->d_inode->i_rwsem.
*/
void __detach_mounts(struct dentry *dentry)
@@ -1984,6 +2013,8 @@ void __detach_mounts(struct dentry *dentry)
if (!lookup_mountpoint(dentry, &mp))
return;
+ /* the name goes first, what covered it goes second */
+ d_drop(dentry);
event++;
while (mp.node.next) {
mnt = hlist_entry(mp.node.next, struct mount, mnt_mp_list);
@@ -2559,11 +2590,11 @@ enum mnt_tree_flags_t {
static int attach_recursive_mnt(struct mount *source_mnt,
const struct pinned_mountpoint *dest)
{
- struct user_namespace *user_ns = current->nsproxy->mnt_ns->user_ns;
struct mount *dest_mnt = dest->parent;
struct mountpoint *dest_mp = dest->mp;
HLIST_HEAD(tree_list);
struct mnt_namespace *ns = dest_mnt->mnt_ns;
+ struct user_namespace *user_ns = ns->user_ns;
struct pinned_mountpoint root = {};
struct mountpoint *shorter = NULL;
struct mount *child, *p;
@@ -2572,13 +2603,31 @@ static int attach_recursive_mnt(struct mount *source_mnt,
int err = 0;
bool moving = mnt_has_parent(source_mnt);
+ /*
+ * A caller in an unprivileged mount namespaces may trigger an
+ * automount and propagate locked mounts into privileged mount
+ * namespaces. Take ownership from the target mount namespace.
+ * It's equivalent for everything but the automount case.
+ *
+ * Detached trees in anonymous mount namespaces by be handed
+ * over via SCM_RIGHTS or inherited in other ways on purpose
+ * the attaching task's mount namespace is authoritative, not
+ * the creator of the detached tree.
+ */
+ if (is_anon_ns(ns))
+ user_ns = current->nsproxy->mnt_ns->user_ns;
+ else
+ user_ns = ns->user_ns;
+
/*
* Preallocate a mountpoint in case the new mounts need to be
* mounted beneath mounts on the same mountpoint.
*/
- for (top = source_mnt; unlikely(top->overmount); top = top->overmount) {
+ for (top = source_mnt; ; top = top->overmount) {
if (!shorter && is_mnt_ns_file(top->mnt.mnt_root))
shorter = top->mnt_mp;
+ if (likely(!top->overmount))
+ break;
}
err = get_mountpoint(top->mnt.mnt_root, &root);
if (err)
@@ -2961,6 +3010,13 @@ static struct mount *__do_loopback(const struct path *old_path, int recurse)
if (!may_copy_tree(old_path))
return ERR_PTR(-EINVAL);
+ /* a pseudo dentry is freed without an RCU delay, no walk may find it */
+ if (old_path->dentry->d_flags & DCACHE_NORCU)
+ return ERR_PTR(-EINVAL);
+
+ if (recurse && !old->mnt_ns)
+ return ERR_PTR(-EINVAL);
+
if (!recurse && __has_locked_children(old, old_path->dentry))
return ERR_PTR(-EINVAL);
@@ -2999,7 +3055,11 @@ static int do_loopback(const struct path *path, const char *old_name,
if (IS_ERR(mnt))
return PTR_ERR(mnt);
- err = graft_tree(mnt, &mp);
+ /* the copy may carry mount namespace files from below the source */
+ if (recurse && !check_for_nsfs_mounts(mnt))
+ err = -EINVAL;
+ else
+ err = graft_tree(mnt, &mp);
if (err) {
lock_mount_hash();
umount_tree(mnt, UMOUNT_SYNC);
@@ -3337,7 +3397,7 @@ static int do_set_group(const struct path *from_path, const struct path *to_path
return -EINVAL;
/* Setting sharing groups is only allowed on private mounts */
- if (IS_MNT_SHARED(to) || IS_MNT_SLAVE(to))
+ if (IS_MNT_SHARED(to) || IS_MNT_SLAVE(to) || IS_MNT_UNBINDABLE(to))
return -EINVAL;
/* From should not be private */
@@ -3623,7 +3683,7 @@ static int do_add_mount(struct mount *newmnt, const struct pinned_mountpoint *mp
if (!(mnt_flags & MNT_SHRINKABLE))
return -EINVAL;
/* ... and for those we'd better have mountpoint still alive */
- if (!parent->mnt_ns)
+ if (!is_mounted(&parent->mnt))
return -EINVAL;
}
@@ -3823,6 +3883,8 @@ void mark_mounts_for_expiry(struct list_head *mounts)
list_for_each_entry_safe(mnt, next, mounts, mnt_expire) {
if (!is_mounted(&mnt->mnt))
continue;
+ /* lock_mnt_tree() leaves expirable mounts alone */
+ VFS_WARN_ON_ONCE(IS_MNT_LOCKED(mnt));
if (!xchg(&mnt->mnt_expiry_mark, 1) ||
propagate_mount_busy(mnt, 1))
continue;
@@ -3830,6 +3892,11 @@ void mark_mounts_for_expiry(struct list_head *mounts)
}
while (!list_empty(&graveyard)) {
mnt = list_first_entry(&graveyard, struct mount, mnt_expire);
+ /* an earlier umount_tree() may have moved a busy mount here */
+ if (propagate_mount_busy(mnt, 1)) {
+ list_move(&mnt->mnt_expire, mounts);
+ continue;
+ }
touch_mnt_namespace(mnt->mnt_ns);
umount_tree(mnt, UMOUNT_PROPAGATE|UMOUNT_SYNC);
}
@@ -3837,17 +3904,38 @@ void mark_mounts_for_expiry(struct list_head *mounts)
EXPORT_SYMBOL_GPL(mark_mounts_for_expiry);
+/*
+ * Unmount @mnt if it's a shrinkable mount without children that nobody uses.
+ *
+ * mount_lock must be held for write
+ */
+static bool shrink_submount(struct mount *mnt)
+{
+ /* not the kernel's to remove either */
+ if (IS_MNT_LOCKED(mnt) || propagate_mount_busy(mnt, 1))
+ return false;
+ touch_mnt_namespace(mnt->mnt_ns);
+ umount_tree(mnt, UMOUNT_PROPAGATE|UMOUNT_SYNC);
+ return true;
+}
+
/*
* Ripoff of 'select_parent()'
*
- * search the list of submounts for a given mountpoint, and move any
- * shrinkable submounts to the 'graveyard' list.
+ * unmount the shrinkable submounts of @parent that aren't busy, children
+ * before their parent, and say whether anything went
+ *
+ * The cursor into the children of @this_parent survives the umount of a
+ * child mount without child mounts. The mounts that get umounted together with
+ * it are located under receiving mounts of @this_parent and never under
+ * @this_parent itself. The one exception is @this_parent getting unmounted
+ * then the walk starts over.
*/
-static int select_submounts(struct mount *parent, struct list_head *graveyard)
+static bool __shrink_submounts(struct mount *parent)
{
struct mount *this_parent = parent;
struct list_head *next;
- int found = 0;
+ bool shrunk = false;
repeat:
next = this_parent->mnt_mounts.next;
@@ -3866,42 +3954,45 @@ static int select_submounts(struct mount *parent, struct list_head *graveyard)
this_parent = mnt;
goto repeat;
}
-
- if (!propagate_mount_busy(mnt, 1)) {
- list_move_tail(&mnt->mnt_expire, graveyard);
- found++;
- }
+ if (!shrink_submount(mnt))
+ continue;
+ shrunk = true;
+ if (unlikely(this_parent->mnt.mnt_flags & MNT_UMOUNT))
+ return true;
}
/*
* All done at this level ... ascend and resume the search
*/
if (this_parent != parent) {
- next = this_parent->mnt_child.next;
- this_parent = this_parent->mnt_parent;
+ struct mount *mnt = this_parent;
+
+ next = mnt->mnt_child.next;
+ this_parent = mnt->mnt_parent;
+ /* its children are gone, maybe it can go as well */
+ if (shrink_submount(mnt)) {
+ shrunk = true;
+ if (unlikely(this_parent->mnt.mnt_flags & MNT_UMOUNT))
+ return true;
+ }
goto resume;
}
- return found;
+ return shrunk;
}
/*
- * process a list of expirable mountpoints with the intent of discarding any
- * submounts of a specific parent mountpoint
+ * unmount the shrinkable submounts of @mnt that aren't busy
+ *
+ * The busy check and the umount of a mount are adjacent. An umount can
+ * still empty or move a mount in a part of the tree that was walked
+ * already, so walk again until nothing goes.
*
* mount_lock must be held for write
*/
static void shrink_submounts(struct mount *mnt)
{
- LIST_HEAD(graveyard);
- struct mount *m;
-
- /* extract submounts of 'mountpoint' from the expiration list */
- while (select_submounts(mnt, &graveyard)) {
- while (!list_empty(&graveyard)) {
- m = list_first_entry(&graveyard, struct mount,
- mnt_expire);
- touch_mnt_namespace(m->mnt_ns);
- umount_tree(m, UMOUNT_PROPAGATE|UMOUNT_SYNC);
- }
+ for (;;) {
+ if (!__shrink_submounts(mnt))
+ break;
}
}
@@ -4066,6 +4157,8 @@ static void free_mnt_ns(struct mnt_namespace *ns)
if (!is_anon_ns(ns))
ns_common_free(ns);
dec_mnt_namespaces(ns->ucounts);
+ /* the last active reference is gone, no mark can show up anymore */
+ fsnotify_mntns_delete(ns);
mnt_ns_tree_remove(ns);
}
@@ -4588,14 +4681,15 @@ SYSCALL_DEFINE2(pivot_root, const char __user *, new_root,
new_mnt = real_mount(new.mnt);
root_mnt = real_mount(root.mnt);
+ /* only a mounted mount has a parent that namespace_sem pins */
+ if (!check_mnt(root_mnt) || !check_mnt(new_mnt))
+ return -EINVAL;
ex_parent = new_mnt->mnt_parent;
root_parent = root_mnt->mnt_parent;
if (IS_MNT_SHARED(old_mnt) ||
IS_MNT_SHARED(ex_parent) ||
IS_MNT_SHARED(root_parent))
return -EINVAL;
- if (!check_mnt(root_mnt) || !check_mnt(new_mnt))
- return -EINVAL;
if (new_mnt->mnt.mnt_flags & MNT_LOCKED)
return -EINVAL;
if (d_unlinked(new.dentry))
@@ -4801,6 +4895,11 @@ static int do_mount_setattr(const struct path *path, struct mount_kattr *kattr)
* propagation.
*/
namespace_lock();
+ /* invent_group_ids() walks the tree, only walk a mounted one */
+ if (!anon_ns_root(mnt) && !check_mnt(mnt)) {
+ namespace_unlock();
+ return -EINVAL;
+ }
if (kattr->propagation == MS_SHARED) {
err = invent_group_ids(mnt, kattr->kflags & MOUNT_KATTR_RECURSE);
if (err) {
@@ -4813,6 +4912,7 @@ static int do_mount_setattr(const struct path *path, struct mount_kattr *kattr)
err = -EINVAL;
lock_mount_hash();
+ /* Checked under namespace_sem already if the propagation changes. */
if (!anon_ns_root(mnt) && !check_mnt(mnt))
goto out;
diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c
index 29868dcd1ec3..42afe367d0e0 100644
--- a/fs/netfs/buffered_read.c
+++ b/fs/netfs/buffered_read.c
@@ -170,7 +170,8 @@ void netfs_queue_read(struct netfs_io_request *rreq,
* remove entries off of the front.
*/
spin_lock(&rreq->lock);
- list_add_tail(&subreq->rreq_link, &stream->subrequests);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ list_add_tail_release(&subreq->rreq_link, &stream->subrequests);
if (list_is_first(&subreq->rreq_link, &stream->subrequests)) {
if (!stream->active) {
stream->collected_to = subreq->start;
@@ -360,7 +361,7 @@ void netfs_readahead(struct readahead_control *ractl)
netfs_rreq_expand(rreq, ractl);
rreq->submitted = rreq->start;
- if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST) < 0)
+ if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST, rreq->gfp) < 0)
goto cleanup_free;
netfs_read_to_pagecache(rreq, ractl);
@@ -379,10 +380,10 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo
{
ssize_t added;
- if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST) < 0)
+ if (rolling_buffer_init(&rreq->buffer, rreq->debug_id, ITER_DEST, rreq->gfp) < 0)
return -ENOMEM;
- added = rolling_buffer_append(&rreq->buffer, folio, rollbuf_flags);
+ added = rolling_buffer_append(&rreq->buffer, folio, rollbuf_flags, rreq->gfp);
if (added < 0)
return added;
rreq->submitted = rreq->start + added;
diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h
index a4a5cec256b0..2889df591d94 100644
--- a/fs/netfs/internal.h
+++ b/fs/netfs/internal.h
@@ -43,6 +43,7 @@ extern struct list_head netfs_io_requests;
extern spinlock_t netfs_proc_lock;
extern mempool_t netfs_request_pool;
extern mempool_t netfs_subrequest_pool;
+extern mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
static inline void netfs_proc_add_rreq(struct netfs_io_request *rreq)
diff --git a/fs/netfs/main.c b/fs/netfs/main.c
index 73da6c9f5777..927badf3989d 100644
--- a/fs/netfs/main.c
+++ b/fs/netfs/main.c
@@ -28,6 +28,7 @@ static struct kmem_cache *netfs_request_slab;
static struct kmem_cache *netfs_subrequest_slab;
mempool_t netfs_request_pool;
mempool_t netfs_subrequest_pool;
+mempool_t netfs_folioq_pool;
#ifdef CONFIG_PROC_FS
LIST_HEAD(netfs_io_requests);
@@ -108,6 +109,9 @@ static int __init netfs_init(void)
{
int ret = -ENOMEM;
+ if (mempool_init_kmalloc_pool(&netfs_folioq_pool, 100, sizeof(struct folio_queue)) < 0)
+ goto error_folioq_pool;
+
netfs_request_slab = kmem_cache_create("netfs_request",
sizeof(struct netfs_io_request), 0,
SLAB_HWCACHE_ALIGN | SLAB_ACCOUNT,
@@ -160,6 +164,8 @@ static int __init netfs_init(void)
error_reqpool:
kmem_cache_destroy(netfs_request_slab);
error_req:
+ mempool_exit(&netfs_folioq_pool);
+error_folioq_pool:
return ret;
}
fs_initcall(netfs_init);
@@ -172,5 +178,6 @@ static void __exit netfs_exit(void)
kmem_cache_destroy(netfs_subrequest_slab);
mempool_exit(&netfs_request_pool);
kmem_cache_destroy(netfs_request_slab);
+ mempool_exit(&netfs_folioq_pool);
}
module_exit(netfs_exit);
diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c
index 628c45f53d83..d8a67ea348f7 100644
--- a/fs/netfs/misc.c
+++ b/fs/netfs/misc.c
@@ -303,7 +303,7 @@ bool netfs_release_folio(struct folio *folio, gfp_t gfp)
if (folio_test_dirty(folio))
return false;
- end = umin(folio_pos(folio) + folio_size(folio), i_size_read(&ctx->inode));
+ end = umin(folio_pos(folio) + folio_size(folio), ctx->remote_i_size);
if (end > ctx->zero_point)
ctx->zero_point = end;
@@ -361,6 +361,7 @@ void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq,
DEFINE_WAIT(myself);
list_for_each_entry(subreq, &stream->subrequests, rreq_link) {
+ smp_rmb(); /* Read ->next before IN_PROGRESS. */
if (!netfs_check_subreq_in_progress(subreq))
continue;
diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c
index b8c4918d3dcd..e3e7c8ffc463 100644
--- a/fs/netfs/objects.c
+++ b/fs/netfs/objects.c
@@ -7,7 +7,6 @@
#include <linux/slab.h>
#include <linux/mempool.h>
-#include <linux/delay.h>
#include "internal.h"
static void netfs_free_request(struct work_struct *work);
@@ -26,17 +25,23 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping,
struct netfs_io_request *rreq;
mempool_t *mempool = ctx->ops->request_pool ?: &netfs_request_pool;
struct kmem_cache *cache = mempool->pool_data;
+ gfp_t gfp = GFP_KERNEL;
int ret;
- for (;;) {
- rreq = mempool_alloc(mempool, GFP_KERNEL);
- if (rreq)
- break;
- msleep(10);
+ /* Writeback is part of memory reclaim and must not fail due to ENOMEM. */
+ if (origin == NETFS_WRITEBACK || origin == NETFS_WRITEBACK_SINGLE) {
+ gfp = GFP_NOFS; /* Allows use of mempools. */
+
+ rreq = mempool_alloc(mempool, gfp);
+ } else {
+ rreq = mempool_alloc_noreserve(mempool, gfp);
+ if (!rreq)
+ return ERR_PTR(-ENOMEM);
}
memset(rreq, 0, kmem_cache_size(cache));
INIT_WORK(&rreq->cleanup_work, netfs_free_request);
+ rreq->gfp = gfp;
rreq->start = start;
rreq->len = len;
rreq->origin = origin;
@@ -200,13 +205,12 @@ struct netfs_io_subrequest *netfs_alloc_subrequest(struct netfs_io_request *rreq
mempool_t *mempool = rreq->netfs_ops->subrequest_pool ?: &netfs_subrequest_pool;
struct kmem_cache *cache = mempool->pool_data;
- for (;;) {
- subreq = mempool_alloc(rreq->netfs_ops->subrequest_pool ?: &netfs_subrequest_pool,
- GFP_KERNEL);
- if (subreq)
- break;
- msleep(10);
- }
+ if (rreq->gfp == GFP_KERNEL)
+ subreq = mempool_alloc_noreserve(mempool, rreq->gfp);
+ else
+ subreq = mempool_alloc(mempool, rreq->gfp);
+ if (!subreq)
+ return NULL;
memset(subreq, 0, kmem_cache_size(cache));
INIT_WORK(&subreq->work, NULL);
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 4c7312a4c859..a2f3f31eb979 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -34,6 +34,22 @@ static void netfs_clear_unread(struct netfs_io_subrequest *subreq)
__set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags);
}
+static void netfs_clear_unread_dio(struct netfs_io_subrequest *subreq)
+{
+ unsigned long long pos = subreq->start + subreq->transferred;
+ struct netfs_io_request *rreq = subreq->rreq;
+ size_t fill;
+
+ if (pos >= rreq->i_size)
+ return;
+
+ fill = min_t(unsigned long long, rreq->i_size - pos,
+ subreq->len - subreq->transferred);
+
+ netfs_reset_iter(subreq);
+ subreq->transferred += iov_iter_zero(fill, &subreq->io_iter);
+}
+
/*
* Flush, mark and unlock a folio that's now completely read. If we want to
* cache the folio, we set the group to NETFS_FOLIO_COPY_TO_CACHE, mark it
@@ -205,8 +221,10 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
* in progress. The issuer thread may be adding stuff to the tail
* whilst we're doing this.
*/
- front = list_first_entry_or_null(&stream->subrequests,
- struct netfs_io_subrequest, rreq_link);
+ front = list_first_entry_or_null_acquire(&stream->subrequests,
+ struct netfs_io_subrequest, rreq_link);
+ /* Read first subreq pointer before IN_PROGRESS flag. */
+
while (front) {
size_t transferred;
@@ -259,6 +277,14 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq)
test_bit(NETFS_SREQ_HIT_EOF, &front->flags))
netfs_read_unlock_folios(rreq, ¬es);
} else {
+ if (!(notes & HIT_PENDING) &&
+ front->error == 0 &&
+ transferred < front->len &&
+ test_bit(NETFS_SREQ_CLEAR_TAIL, &front->flags)) {
+ netfs_clear_unread_dio(front);
+ transferred = front->transferred;
+ trace_netfs_sreq(front, netfs_sreq_trace_clear);
+ }
stream->collected_to = front->start + transferred;
rreq->collected_to = stream->collected_to;
}
@@ -383,6 +409,11 @@ static void netfs_rreq_assess_single(struct netfs_io_request *rreq)
netfs_single_mark_inode_dirty(rreq->inode);
}
+ /* To do DIO, the cache has to round the size up, so we need to undo
+ * the rounding.
+ */
+ rreq->transferred = min(rreq->transferred, rreq->i_size);
+
if (rreq->iocb) {
rreq->iocb->ki_pos += rreq->transferred;
if (rreq->iocb->ki_complete) {
diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c
index 7eacc58abadb..c31190993b76 100644
--- a/fs/netfs/read_pgpriv2.c
+++ b/fs/netfs/read_pgpriv2.c
@@ -53,7 +53,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio
trace_netfs_folio(folio, netfs_folio_trace_store_copy);
/* Attach the folio to the rolling buffer. */
- if (rolling_buffer_append(&creq->buffer, folio, 0) < 0) {
+ if (rolling_buffer_append(&creq->buffer, folio, 0, creq->gfp) < 0) {
folio_end_private_2(folio);
clear_bit(NETFS_RREQ_FOLIO_COPY_TO_CACHE, &creq->flags);
return;
diff --git a/fs/netfs/rolling_buffer.c b/fs/netfs/rolling_buffer.c
index 207b6a326651..e1342312a2c7 100644
--- a/fs/netfs/rolling_buffer.c
+++ b/fs/netfs/rolling_buffer.c
@@ -6,6 +6,7 @@
*/
#include <linux/bitops.h>
+#include <linux/mempool.h>
#include <linux/pagemap.h>
#include <linux/rolling_buffer.h>
#include <linux/slab.h>
@@ -27,7 +28,10 @@ struct folio_queue *netfs_folioq_alloc(unsigned int rreq_id, gfp_t gfp,
{
struct folio_queue *fq;
- fq = kmalloc(sizeof(*fq), gfp);
+ if (gfp == GFP_KERNEL)
+ fq = mempool_alloc_noreserve(&netfs_folioq_pool, gfp);
+ else
+ fq = mempool_alloc(&netfs_folioq_pool, gfp);
if (fq) {
netfs_stat(&netfs_n_folioq);
folioq_init(fq, rreq_id);
@@ -50,7 +54,7 @@ void netfs_folioq_free(struct folio_queue *folioq,
{
trace_netfs_folioq(folioq, trace);
netfs_stat_d(&netfs_n_folioq);
- kfree(folioq);
+ mempool_free(folioq, &netfs_folioq_pool);
}
EXPORT_SYMBOL(netfs_folioq_free);
@@ -60,11 +64,11 @@ EXPORT_SYMBOL(netfs_folioq_free);
* consumer.
*/
int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
- unsigned int direction)
+ unsigned int direction, gfp_t gfp)
{
struct folio_queue *fq;
- fq = netfs_folioq_alloc(rreq_id, GFP_NOFS, netfs_trace_folioq_rollbuf_init);
+ fq = netfs_folioq_alloc(rreq_id, gfp, netfs_trace_folioq_rollbuf_init);
if (!fq)
return -ENOMEM;
@@ -77,14 +81,14 @@ int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
/*
* Add another folio_queue to a rolling buffer if there's no space left.
*/
-int rolling_buffer_make_space(struct rolling_buffer *roll)
+int rolling_buffer_make_space(struct rolling_buffer *roll, gfp_t gfp)
{
struct folio_queue *fq, *head = roll->head;
if (!folioq_full(head))
return 0;
- fq = netfs_folioq_alloc(head->rreq_id, GFP_NOFS, netfs_trace_folioq_make_space);
+ fq = netfs_folioq_alloc(head->rreq_id, gfp, netfs_trace_folioq_make_space);
if (!fq)
return -ENOMEM;
fq->prev = head;
@@ -122,7 +126,7 @@ ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
int nr, ix, to;
ssize_t size = 0;
- if (rolling_buffer_make_space(roll) < 0)
+ if (rolling_buffer_make_space(roll, GFP_KERNEL) < 0)
return -ENOMEM;
fq = roll->head;
@@ -153,12 +157,12 @@ ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
* Append a folio to the rolling buffer.
*/
ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
- unsigned int flags)
+ unsigned int flags, gfp_t gfp)
{
ssize_t size = folio_size(folio);
int slot;
- if (rolling_buffer_make_space(roll) < 0)
+ if (rolling_buffer_make_space(roll, gfp) < 0)
return -ENOMEM;
slot = folioq_append(roll->head, folio);
diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c
index b194447f4b11..7fbf50907a7f 100644
--- a/fs/netfs/write_collect.c
+++ b/fs/netfs/write_collect.c
@@ -228,8 +228,10 @@ static void netfs_collect_write_results(struct netfs_io_request *wreq)
if (!smp_load_acquire(&stream->active))
continue;
- front = list_first_entry_or_null(&stream->subrequests,
- struct netfs_io_subrequest, rreq_link);
+ front = list_first_entry_or_null_acquire(&stream->subrequests,
+ struct netfs_io_subrequest, rreq_link);
+ /* Read first subreq pointer before IN_PROGRESS flag. */
+
while (front) {
trace_netfs_collect_sreq(wreq, front);
//_debug("sreq [%x] %llx %zx/%zx",
diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c
index ceb1a684a758..3fd8908c251b 100644
--- a/fs/netfs/write_issue.c
+++ b/fs/netfs/write_issue.c
@@ -108,7 +108,7 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping,
ictx = netfs_inode(wreq->inode);
if (is_cacheable)
fscache_begin_write_operation(&wreq->cache_resources, netfs_i_cookie(ictx));
- if (rolling_buffer_init(&wreq->buffer, wreq->debug_id, ITER_SOURCE) < 0)
+ if (rolling_buffer_init(&wreq->buffer, wreq->debug_id, ITER_SOURCE, wreq->gfp) < 0)
goto nomem;
wreq->cleaned_to = wreq->start;
@@ -167,7 +167,7 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
*/
if (iov_iter_is_folioq(wreq_iter) &&
wreq_iter->folioq_slot >= folioq_nr_slots(wreq_iter->folioq))
- rolling_buffer_make_space(&wreq->buffer);
+ rolling_buffer_make_space(&wreq->buffer, wreq->gfp);
subreq = netfs_alloc_subrequest(wreq);
if (!subreq)
@@ -206,7 +206,8 @@ void netfs_prepare_write(struct netfs_io_request *wreq,
* remove entries off of the front.
*/
spin_lock(&wreq->lock);
- list_add_tail(&subreq->rreq_link, &stream->subrequests);
+ /* Write IN_PROGRESS before pointer to new subreq */
+ list_add_tail_release(&subreq->rreq_link, &stream->subrequests);
if (list_is_first(&subreq->rreq_link, &stream->subrequests)) {
if (!stream->active) {
stream->collected_to = subreq->start;
@@ -335,7 +336,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
_enter("");
- if (rolling_buffer_make_space(&wreq->buffer) < 0)
+ if (rolling_buffer_make_space(&wreq->buffer, wreq->gfp) < 0)
return -ENOMEM;
/* netfs_perform_write() may shift i_size around the page or from out
@@ -437,7 +438,7 @@ static int netfs_write_folio(struct netfs_io_request *wreq,
}
/* Attach the folio to the rolling buffer. */
- rolling_buffer_append(&wreq->buffer, folio, 0);
+ rolling_buffer_append(&wreq->buffer, folio, 0, wreq->gfp);
/* Move the submission point forward to allow for write-streaming data
* not starting at the front of the page. We don't do write-streaming
@@ -761,7 +762,7 @@ static int netfs_write_folio_single(struct netfs_io_request *wreq,
/* Attach the folio to the rolling buffer. */
folio_get(folio);
- ret = rolling_buffer_append(&wreq->buffer, folio, NETFS_ROLLBUF_PUT_MARK);
+ ret = rolling_buffer_append(&wreq->buffer, folio, NETFS_ROLLBUF_PUT_MARK, wreq->gfp);
if (ret < 0) {
folio_put(folio);
return ret;
diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c
index 4254ba3ee7c5..ada04a7230c2 100644
--- a/fs/nfs/callback_xdr.c
+++ b/fs/nfs/callback_xdr.c
@@ -422,9 +422,9 @@ static __be32 decode_cb_sequence_args(struct svc_rqst *rqstp,
args->csa_nrclists = ntohl(*p++);
args->csa_rclists = NULL;
if (args->csa_nrclists) {
- args->csa_rclists = kmalloc_array(args->csa_nrclists,
- sizeof(*args->csa_rclists),
- GFP_KERNEL);
+ args->csa_rclists = kcalloc(args->csa_nrclists,
+ sizeof(*args->csa_rclists),
+ GFP_KERNEL);
if (unlikely(args->csa_rclists == NULL))
return htonl(NFS4ERR_RESOURCE);
diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index f95707fc0edf..50b002377b8c 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -468,11 +468,14 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
struct nfs_inode *nfsi = NFS_I(inode);
struct nfs_delegation *delegation, *old_delegation;
struct nfs_delegation *freeme = NULL;
+ bool orphaned = false;
int status = 0;
delegation = kmalloc(sizeof(*delegation), GFP_KERNEL_ACCOUNT);
- if (delegation == NULL)
+ if (delegation == NULL) {
+ nfs4_proc_delegreturn(inode, cred, stateid, NULL, 0);
return -ENOMEM;
+ }
nfs4_stateid_copy(&delegation->stateid, stateid);
refcount_set(&delegation->refcount, 1);
delegation->type = type;
@@ -520,12 +523,16 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
goto out;
}
if (test_and_set_bit(NFS_DELEGATION_RETURNING,
- &old_delegation->flags))
+ &old_delegation->flags)) {
+ orphaned = true;
goto out;
+ }
}
freeme = nfs_detach_delegation_locked(nfsi, old_delegation, clp);
- if (freeme == NULL)
+ if (freeme == NULL) {
+ orphaned = true;
goto out;
+ }
add_new:
/*
* If we didn't revalidate the change attribute before setting
@@ -559,8 +566,11 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
nfs_update_delegated_mtime(inode);
out:
spin_unlock(&clp->cl_lock);
- if (delegation != NULL)
+ if (delegation != NULL) {
+ if (orphaned)
+ nfs_do_return_delegation(inode, delegation, 0);
__nfs_free_delegation(delegation);
+ }
if (freeme != NULL) {
nfs_do_return_delegation(inode, freeme, 0);
nfs_free_delegation(server, freeme);
diff --git a/fs/nfsd/lockd.c b/fs/nfsd/lockd.c
index c774ce9aa296..6cd028c142fe 100644
--- a/fs/nfsd/lockd.c
+++ b/fs/nfsd/lockd.c
@@ -86,6 +86,7 @@ nlm_fclose(struct file *filp)
}
static const struct nlmsvc_binding nfsd_nlm_ops = {
+ .owner = THIS_MODULE,
.fopen = nlm_fopen, /* open file for locking */
.fclose = nlm_fclose, /* close file */
};
@@ -94,11 +95,12 @@ void
nfsd_lockd_init(void)
{
dprintk("nfsd: initializing lockd\n");
- nlmsvc_ops = &nfsd_nlm_ops;
+ rcu_assign_pointer(nlmsvc_ops, &nfsd_nlm_ops);
}
void
nfsd_lockd_shutdown(void)
{
- nlmsvc_ops = NULL;
+ RCU_INIT_POINTER(nlmsvc_ops, NULL);
+ synchronize_rcu();
}
diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
index f3b440ae98fe..b9485b24df0a 100644
--- a/fs/nfsd/netns.h
+++ b/fs/nfsd/netns.h
@@ -27,6 +27,16 @@
struct cld_net;
struct nfsd4_client_tracking_ops;
+enum nfsd_net_flag {
+ NFSD_NET_GRACE_ENDED,
+ NFSD_NET_GRACE_END_FORCED,
+ NFSD_NET_IN_GRACE,
+ NFSD_NET_SOMEBODY_RECLAIMED,
+ NFSD_NET_TRACK_RECLAIM_COMPLETES,
+ NFSD_NET_UP,
+ NFSD_NET_LOCKD_UP,
+};
+
enum {
/* cache misses due only to checksum comparison failures */
NFSD_STATS_PAYLOAD_MISSES,
@@ -65,9 +75,7 @@ struct nfsd_net {
struct cache_detail *nametoid_cache;
struct lock_manager nfsd4_manager;
- bool grace_ended;
- bool grace_end_forced;
- bool client_tracking_active;
+ unsigned long flags;
time64_t boot_time;
time64_t boot_time_bt; /* same instant in CLOCK_BOOTTIME */
@@ -120,19 +128,13 @@ struct nfsd_net {
spinlock_t blocked_locks_lock;
struct file *rec_file;
- bool in_grace;
const struct nfsd4_client_tracking_ops *client_tracking_ops;
time64_t nfsd4_lease;
time64_t nfsd4_grace;
- bool somebody_reclaimed;
- bool track_reclaim_completes;
atomic_t nr_reclaim_complete;
- bool nfsd_net_up;
- bool lockd_up;
-
seqlock_t writeverf_lock;
unsigned char writeverf[8];
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index bc14c3939a5c..898aaeea2059 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -631,7 +631,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
pr_warn("nfsd4_process_open2 failed to open newly-created file: status=%u\n",
be32_to_cpu(status));
if (reclaim && !status)
- nn->somebody_reclaimed = true;
+ set_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags);
out:
if (open->op_filp) {
fput(open->op_filp);
diff --git a/fs/nfsd/nfs4recover.c b/fs/nfsd/nfs4recover.c
index 8c5b5b8a5c88..37006d0776eb 100644
--- a/fs/nfsd/nfs4recover.c
+++ b/fs/nfsd/nfs4recover.c
@@ -211,7 +211,7 @@ nfsd4_create_clid_dir(struct nfs4_client *clp)
out_unlock:
inode_unlock(d_inode(dir));
if (status == 0) {
- if (nn->in_grace)
+ if (test_bit(NFSD_NET_IN_GRACE, &nn->flags))
__nfsd4_create_reclaim_record_grace(clp, dname, nn);
vfs_fsync(nn->rec_file, 0);
} else {
@@ -371,7 +371,7 @@ nfsd4_remove_clid_dir(struct nfs4_client *clp)
nfs4_reset_creds(original_cred);
if (status == 0) {
vfs_fsync(nn->rec_file, 0);
- if (nn->in_grace)
+ if (test_bit(NFSD_NET_IN_GRACE, &nn->flags))
__nfsd4_remove_reclaim_record_grace(dname,
HEXDIR_LEN, nn);
}
@@ -429,7 +429,7 @@ nfsd4_recdir_purge_old(struct nfsd_net *nn)
{
int status;
- nn->in_grace = false;
+ clear_bit(NFSD_NET_IN_GRACE, &nn->flags);
if (!nn->rec_file)
return;
status = mnt_want_write_file(nn->rec_file);
@@ -511,7 +511,7 @@ nfsd4_init_recdir(struct net *net)
nfs4_reset_creds(original_cred);
if (!status)
- nn->in_grace = true;
+ set_bit(NFSD_NET_IN_GRACE, &nn->flags);
return status;
}
@@ -1431,7 +1431,7 @@ nfs4_cld_state_init(struct net *net)
for (i = 0; i < CLIENT_HASH_SIZE; i++)
INIT_LIST_HEAD(&nn->reclaim_str_hashtbl[i]);
nn->reclaim_str_hashtbl_size = 0;
- nn->track_reclaim_completes = true;
+ set_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags);
atomic_set(&nn->nr_reclaim_complete, 0);
return 0;
@@ -1442,7 +1442,7 @@ nfs4_cld_state_shutdown(struct net *net)
{
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- nn->track_reclaim_completes = false;
+ clear_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags);
kfree(nn->reclaim_str_hashtbl);
}
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index fbaf2f135274..59a9ce4858b0 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -2681,7 +2681,7 @@ static void inc_reclaim_complete(struct nfs4_client *clp)
{
struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id);
- if (!nn->track_reclaim_completes)
+ if (!test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags))
return;
if (!nfsd4_find_reclaim_client(clp->cl_name, nn))
return;
@@ -5066,8 +5066,6 @@ nfsd4_init_leases_net(struct nfsd_net *nn)
nn->nfsd4_lease = 90; /* default lease time */
nn->nfsd4_grace = 90;
- nn->somebody_reclaimed = false;
- nn->track_reclaim_completes = false;
nn->clverifier_counter = get_random_u32();
nn->clientid_base = get_random_u32();
nn->clientid_counter = nn->clientid_base + 1;
@@ -6759,12 +6757,21 @@ nfsd4_renew(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
static void
nfsd4_end_grace(struct nfsd_net *nn)
{
- /* do nothing if grace period already ended */
- if (nn->grace_ended)
+ /*
+ * nfsd4_end_grace() can be entered concurrently from the
+ * laundromat workqueue and from an nfsd compound thread
+ * handling RECLAIM_COMPLETE. Without serialization, both
+ * callers can observe NFSD_NET_GRACE_ENDED clear and proceed
+ * into nfsd4_record_grace_done(). For tracking ops whose
+ * grace_done drains reclaim_str_hashtbl, that results in
+ * list corruption and a double free of every
+ * nfs4_client_reclaim entry. Use an atomic test-and-set so
+ * exactly one caller proceeds.
+ */
+ if (test_and_set_bit(NFSD_NET_GRACE_ENDED, &nn->flags))
return;
trace_nfsd_grace_complete(nn);
- nn->grace_ended = true;
/*
* If the server goes down again right now, an NFSv4
* client will still be allowed to reclaim after it comes back up,
@@ -6805,14 +6812,14 @@ bool nfsd4_force_end_grace(struct nfsd_net *nn)
{
if (!nn->client_tracking_ops)
return false;
- spin_lock(&nn->client_lock);
- if (nn->grace_ended || !nn->client_tracking_active) {
- spin_unlock(&nn->client_lock);
+ if (test_bit(NFSD_NET_GRACE_ENDED, &nn->flags))
return false;
- }
- WRITE_ONCE(nn->grace_end_forced, true);
+ /* laundromat_work must be initialised now, though it might be disabled */
+ set_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags);
+ /* mod_delayed_work() doesn't queue work after
+ * nfs4_state_shutdown_net() has called disable_delayed_work_sync()
+ */
mod_delayed_work(laundry_wq, &nn->laundromat_work, 0);
- spin_unlock(&nn->client_lock);
return true;
}
@@ -6825,15 +6832,15 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)
time64_t double_grace_period_end = nn->boot_time_bt +
2 * nn->nfsd4_lease;
- if (READ_ONCE(nn->grace_end_forced))
+ if (test_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags))
return false;
- if (nn->track_reclaim_completes &&
+ if (test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags) &&
atomic_read(&nn->nr_reclaim_complete) ==
nn->reclaim_str_hashtbl_size)
return false;
- if (!nn->somebody_reclaimed)
+ if (!test_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags))
return false;
- nn->somebody_reclaimed = false;
+ clear_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags);
/*
* If we've given them *two* lease times to reclaim, and they're
* still not done, give up:
@@ -8667,7 +8674,7 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid);
status = 0;
if (lock->lk_reclaim)
- nn->somebody_reclaimed = true;
+ set_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags);
break;
case FILE_LOCK_DEFERRED:
kref_put(&nbl->nbl_kref, free_nbl);
@@ -9199,9 +9206,8 @@ static int nfs4_state_create_net(struct net *net)
nn->unconf_name_tree = RB_ROOT;
nn->boot_time = ktime_get_real_seconds();
nn->boot_time_bt = ktime_get_boottime_seconds();
- nn->grace_ended = false;
- nn->grace_end_forced = false;
- nn->client_tracking_active = false;
+ clear_bit(NFSD_NET_GRACE_ENDED, &nn->flags);
+ clear_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags);
nn->nfsd4_manager.block_opens = true;
INIT_LIST_HEAD(&nn->nfsd4_manager.list);
INIT_LIST_HEAD(&nn->client_lru);
@@ -9217,6 +9223,8 @@ static int nfs4_state_create_net(struct net *net)
INIT_LIST_HEAD(&nn->blocked_locks_lru);
INIT_DELAYED_WORK(&nn->laundromat_work, laundromat_main);
+ /* Make sure this cannot run until client tracking is initialised */
+ disable_delayed_work(&nn->laundromat_work);
INIT_WORK(&nn->nfsd_shrinker_work, nfsd4_state_shrinker_worker);
get_net(net);
@@ -9284,10 +9292,9 @@ nfs4_state_start_net(struct net *net)
locks_start_grace(net, &nn->nfsd4_manager);
nfsd4_client_tracking_init(net);
/* safe for laundromat to run now */
- spin_lock(&nn->client_lock);
- nn->client_tracking_active = true;
- spin_unlock(&nn->client_lock);
- if (nn->track_reclaim_completes && nn->reclaim_str_hashtbl_size == 0)
+ enable_delayed_work(&nn->laundromat_work);
+ if (test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags) &&
+ nn->reclaim_str_hashtbl_size == 0)
goto skip_grace;
printk(KERN_INFO "NFSD: starting %lld-second grace period (net %x)\n",
nn->nfsd4_grace, net->ns.inum);
@@ -9335,10 +9342,7 @@ nfs4_state_shutdown_net(struct net *net)
shrinker_free(nn->nfsd_client_shrinker);
cancel_work_sync(&nn->nfsd_shrinker_work);
- spin_lock(&nn->client_lock);
- nn->client_tracking_active = false;
- spin_unlock(&nn->client_lock);
- cancel_delayed_work_sync(&nn->laundromat_work);
+ disable_delayed_work_sync(&nn->laundromat_work);
locks_end_grace(&nn->nfsd4_manager);
INIT_LIST_HEAD(&reaplist);
diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index 2fcf14d530b2..888872bb7cf5 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -298,7 +298,7 @@ static ssize_t write_unlock_fs(struct file *file, char *buf, size_t size)
error = nlmsvc_unlock_all_by_sb(path.dentry->d_sb);
mutex_lock(&nfsd_mutex);
nn = net_generic(netns(file), nfsd_net_id);
- if (nn->nfsd_net_up)
+ if (test_bit(NFSD_NET_UP, &nn->flags))
nfsd4_revoke_states(nn, path.dentry->d_sb);
else
error = -EINVAL;
@@ -1109,7 +1109,7 @@ static ssize_t write_v4_end_grace(struct file *file, char *buf, size_t size)
}
return scnprintf(buf, SIMPLE_TRANSACTION_LIMIT, "%c\n",
- nn->grace_ended ? 'Y' : 'N');
+ test_bit(NFSD_NET_GRACE_ENDED, &nn->flags) ? 'Y' : 'N');
}
#endif
diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 242b8c7b6e0f..7551be99becf 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -371,7 +371,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
int ret;
- if (nn->nfsd_net_up)
+ if (test_bit(NFSD_NET_UP, &nn->flags))
return 0;
ret = nfsd_startup_generic();
@@ -381,11 +381,11 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
if (ret)
goto out_socks;
- if (nfsd_needs_lockd(nn) && !nn->lockd_up) {
+ if (nfsd_needs_lockd(nn) && !test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) {
ret = lockd_up(net, cred);
if (ret)
goto out_socks;
- nn->lockd_up = true;
+ set_bit(NFSD_NET_LOCKD_UP, &nn->flags);
}
ret = nfsd_file_cache_start_net(net);
@@ -403,7 +403,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
if (ret)
goto out_reply_cache;
- nn->nfsd_net_up = true;
+ set_bit(NFSD_NET_UP, &nn->flags);
return 0;
out_reply_cache:
@@ -411,9 +411,9 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
out_filecache:
nfsd_file_cache_shutdown_net(net);
out_lockd:
- if (nn->lockd_up) {
+ if (test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) {
lockd_down(net);
- nn->lockd_up = false;
+ clear_bit(NFSD_NET_LOCKD_UP, &nn->flags);
}
out_socks:
nfsd_shutdown_generic();
@@ -424,7 +424,7 @@ static void nfsd_shutdown_net(struct net *net)
{
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- if (nn->nfsd_net_up) {
+ if (test_bit(NFSD_NET_UP, &nn->flags)) {
percpu_ref_kill_and_confirm(&nn->nfsd_net_ref, nfsd_net_done);
wait_for_completion(&nn->nfsd_net_confirm_done);
@@ -432,18 +432,18 @@ static void nfsd_shutdown_net(struct net *net)
nfs4_state_shutdown_net(net);
nfsd_reply_cache_shutdown(nn);
nfsd_file_cache_shutdown_net(net);
- if (nn->lockd_up) {
+ if (test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) {
lockd_down(net);
- nn->lockd_up = false;
+ clear_bit(NFSD_NET_LOCKD_UP, &nn->flags);
}
wait_for_completion(&nn->nfsd_net_free_done);
}
percpu_ref_exit(&nn->nfsd_net_ref);
- if (nn->nfsd_net_up)
+ if (test_bit(NFSD_NET_UP, &nn->flags))
nfsd_shutdown_generic();
- nn->nfsd_net_up = false;
+ clear_bit(NFSD_NET_UP, &nn->flags);
}
static DEFINE_SPINLOCK(nfsd_notifier_lock);
@@ -934,6 +934,20 @@ nfsd(void *vrqstp)
return 0;
}
+/*
+ * Set rq_status_counter back to an even value, indicating that the rqstp
+ * fields are no longer meaningful to a lockless reader. This pairs with the
+ * odd-valued store made once the request has been decoded, and must run on
+ * every return path that follows it so that the seq-lock like protocol used
+ * by nfsd_nl_rpc_status_get_dumpit() is not left permanently odd. The store
+ * also advances the counter so a concurrent reader detects the transition.
+ */
+static void nfsd_status_counter_set_idle(struct svc_rqst *rqstp)
+{
+ smp_store_release(&rqstp->rq_status_counter,
+ (rqstp->rq_status_counter | 1) + 1);
+}
+
/**
* nfsd_dispatch - Process an NFS or NFSACL or LOCALIO Request
* @rqstp: incoming request
@@ -995,14 +1009,9 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
if (!proc->pc_encode(rqstp, &rqstp->rq_res_stream))
goto out_encode_err;
- /*
- * Release rq_status_counter setting it to an even value after the rpc
- * request has been properly processed.
- */
- smp_store_release(&rqstp->rq_status_counter, rqstp->rq_status_counter + 1);
-
nfsd_cache_update(rqstp, rp, rqstp->rq_cachetype, nfs_reply);
out_cached_reply:
+ nfsd_status_counter_set_idle(rqstp);
return 1;
out_decode_err:
@@ -1013,12 +1022,14 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
out_update_drop:
nfsd_cache_update(rqstp, rp, RC_NOCACHE, NULL);
out_dropit:
+ nfsd_status_counter_set_idle(rqstp);
return 0;
out_encode_err:
trace_nfsd_cant_encode_err(rqstp);
nfsd_cache_update(rqstp, rp, RC_NOCACHE, NULL);
*statp = rpc_system_err;
+ nfsd_status_counter_set_idle(rqstp);
return 1;
}
diff --git a/fs/ntfs3/dir.c b/fs/ntfs3/dir.c
index a68eff8cf1d6..6e1d2ad8c12e 100644
--- a/fs/ntfs3/dir.c
+++ b/fs/ntfs3/dir.c
@@ -334,8 +334,7 @@ static inline bool ntfs_dir_emit(struct ntfs_sb_info *sbi,
* It does additional locks/reads just to get the type of name.
* Should we use additional mount option to enable branch below?
*/
- if (fname->dup.extend_data &&
- ino != ni->mi.rno) {
+ if (fname->dup.extend_data && ino != ni->mi.rno) {
struct inode *inode = ntfs_iget5(sbi->sb, &e->ref, NULL);
if (!IS_ERR_OR_NULL(inode)) {
dt_type = fs_umode_to_dtype(inode->i_mode);
diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c
index 6a19bb042208..13d623560120 100644
--- a/fs/ntfs3/file.c
+++ b/fs/ntfs3/file.c
@@ -1114,8 +1114,8 @@ static ssize_t ntfs_compress_write(struct kiocb *iocb, struct iov_iter *from)
size_t cp, tail = PAGE_SIZE - off;
folio = page_folio(pages[ip]);
- cp = copy_folio_from_iter_atomic(folio, off,
- min(tail, bytes), from);
+ cp = copy_folio_from_iter_atomic(
+ folio, off, min(tail, bytes), from);
flush_dcache_folio(folio);
copied += cp;
@@ -1312,7 +1312,7 @@ static int ntfs_file_release(struct inode *inode, struct file *file)
if (sbi->options->prealloc &&
((file->f_mode & FMODE_WRITE) &&
atomic_read(&inode->i_writecount) == 1)
- /*
+ /*
* The only file when inode->i_fop = &ntfs_file_operations and
* init_rwsem(&ni->file.run_lock) is not called explicitly is MFT.
*
diff --git a/fs/ntfs3/frecord.c b/fs/ntfs3/frecord.c
index 7aa37cf01df5..1cc905a26555 100644
--- a/fs/ntfs3/frecord.c
+++ b/fs/ntfs3/frecord.c
@@ -3072,8 +3072,8 @@ int ni_rename(struct ntfs_inode *dir_ni, struct ntfs_inode *new_dir_ni,
err = ni_add_name(new_dir_ni, ni, new_de);
if (!err) {
err = ni_remove_name(dir_ni, ni, de, &de2, &undo);
- WARN_ON(err && ni_remove_name(new_dir_ni, ni, new_de, &de2,
- &undo));
+ if (err && ni_remove_name(new_dir_ni, ni, new_de, &de2, &undo))
+ _ntfs_bad_inode(&ni->vfs_inode);
}
/*
@@ -3224,7 +3224,8 @@ static bool ni_update_parent(struct ntfs_inode *ni, struct NTFS_DUP_INFO *dup,
if (attr) {
const struct REPARSE_POINT *rp;
- rp = resident_data_ex(attr, sizeof(struct REPARSE_POINT));
+ rp = resident_data_ex(attr,
+ sizeof(struct REPARSE_POINT));
/* If ATTR_REPARSE exists 'rp' can't be NULL. */
if (rp)
dup->extend_data = rp->ReparseTag;
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index f34e0a786bb6..120691bc71ee 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -989,9 +989,9 @@ int ntfs_write_begin(const struct kiocb *iocb, struct address_space *mapping,
/*
* ntfs_write_end - Address_space_operations::write_end.
*/
-int ntfs_write_end(const struct kiocb *iocb,
- struct address_space *mapping, loff_t pos,
- u32 len, u32 copied, struct folio *folio, void *fsdata)
+int ntfs_write_end(const struct kiocb *iocb, struct address_space *mapping,
+ loff_t pos, u32 len, u32 copied, struct folio *folio,
+ void *fsdata)
{
struct inode *inode = mapping->host;
struct ntfs_inode *ni = ntfs_i(inode);
@@ -1113,7 +1113,7 @@ ntfs_create_reparse_buffer(struct ntfs_sb_info *sbi, const char *symname,
typeof(rp->SymbolicLinkReparseBuffer) *rs;
bool is_absolute;
- is_absolute = (strlen(symname) > 1 && symname[1] == ':');
+ is_absolute = symname[0] && symname[1] == ':';
rp = kzalloc(ntfs_reparse_bytes(2 * size + 2, is_absolute), GFP_NOFS);
if (!rp)
@@ -1150,17 +1150,19 @@ ntfs_create_reparse_buffer(struct ntfs_sb_info *sbi, const char *symname,
/* PrintName + SubstituteName. */
rs->SubstituteNameOffset = cpu_to_le16(sizeof(short) * err);
- rs->SubstituteNameLength = cpu_to_le16(sizeof(short) * err + (is_absolute ? 8 : 0));
+ rs->SubstituteNameLength =
+ cpu_to_le16(sizeof(short) * err + (is_absolute ? 8 : 0));
rs->PrintNameLength = rs->SubstituteNameOffset;
/*
* TODO: Use relative path if possible to allow Windows to
* parse this path.
- * 0-absolute path 1- relative path (SYMLINK_FLAG_RELATIVE).
+ * 0-absolute path, 1- relative path (SYMLINK_FLAG_RELATIVE).
*/
rs->Flags = cpu_to_le32(is_absolute ? 0 : SYMLINK_FLAG_RELATIVE);
- memmove(rp_name + err + (is_absolute ? 4 : 0), rp_name, sizeof(short) * err);
+ memmove(rp_name + err + (is_absolute ? 4 : 0), rp_name,
+ sizeof(short) * err);
if (is_absolute) {
/* Decorate SubstituteName. */
@@ -1649,7 +1651,8 @@ int ntfs_create_inode(struct mnt_idmap *idmap, struct inode *dir,
* Use ni_find_attr cause layout of MFT record may be changed
* in ntfs_init_acl and ntfs_save_wsl_perm.
*/
- attr = ni_find_attr(ni, NULL, NULL, ATTR_NAME, NULL, 0, NULL, NULL);
+ attr = ni_find_attr(ni, NULL, NULL, ATTR_NAME, NULL, 0, NULL,
+ NULL);
if (attr) {
struct ATTR_FILE_NAME *fn;
diff --git a/fs/ntfs3/namei.c b/fs/ntfs3/namei.c
index 747a2a66c488..64cde1a856f4 100644
--- a/fs/ntfs3/namei.c
+++ b/fs/ntfs3/namei.c
@@ -207,13 +207,13 @@ static int ntfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
}
/*
- * ntfs_mkdir- inode_operations::mkdir
+ * ntfs_mkdir - inode_operations::mkdir
*/
static struct dentry *ntfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
struct dentry *dentry, umode_t mode)
{
- return ERR_PTR(ntfs_create_inode(idmap, dir, dentry, NULL, S_IFDIR | mode, 0,
- NULL, 0, NULL));
+ return ERR_PTR(ntfs_create_inode(idmap, dir, dentry, NULL,
+ S_IFDIR | mode, 0, NULL, 0, NULL));
}
/*
diff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h
index 7e3d51b8d2dd..99a78204b986 100644
--- a/fs/ntfs3/ntfs_fs.h
+++ b/fs/ntfs3/ntfs_fs.h
@@ -585,7 +585,8 @@ int ni_add_name(struct ntfs_inode *dir_ni, struct ntfs_inode *ni,
struct NTFS_DE *de);
int ni_rename(struct ntfs_inode *dir_ni, struct ntfs_inode *new_dir_ni,
- struct ntfs_inode *ni, struct NTFS_DE *de, struct NTFS_DE *new_de);
+ struct ntfs_inode *ni, struct NTFS_DE *de,
+ struct NTFS_DE *new_de);
bool ni_is_dirty(struct inode *inode);
int ni_write_parents(struct ntfs_inode *ni, int sync);
@@ -711,8 +712,7 @@ int ntfs_set_size(struct inode *inode, u64 new_size);
int ntfs_get_block(struct inode *inode, sector_t vbn,
struct buffer_head *bh_result, int create);
int ntfs_write_begin(const struct kiocb *iocb, struct address_space *mapping,
- loff_t pos, u32 len, struct folio **foliop,
- void **fsdata);
+ loff_t pos, u32 len, struct folio **foliop, void **fsdata);
int ntfs_write_end(const struct kiocb *iocb, struct address_space *mapping,
loff_t pos, u32 len, u32 copied, struct folio *folio,
void *fsdata);
diff --git a/fs/ntfs3/super.c b/fs/ntfs3/super.c
index 66ad6d82bf5c..346c52f95207 100644
--- a/fs/ntfs3/super.c
+++ b/fs/ntfs3/super.c
@@ -16,6 +16,13 @@
* mi - MFT inode - One MFT record(usually 1024 bytes or 4K), consists of attributes.
* ni - NTFS inode - Extends linux inode. consists of one or more mft inodes.
* index - unit inside directory - 2K, 4K, <=page size, does not depend on cluster size.
+ * resident attribute - Attribute with content stored directly in the MFT record
+ * non-resident attribute - Attribute with content stored in clusters
+ * data_size - Size of attribute content in bytes. Equal to inode->i_size
+ * valid_size - Number of bytes written to the non-resident attribute
+ * allocated_size - Total size of clusters allocated for non-resident content
+ * total_size - Actual size of allocated clusters for sparse or compressed attributes
+ * - Constraint: valid_size <= data_size <= allocated_size
*
* WSL - Windows Subsystem for Linux
* https://docs.microsoft.com/en-us/windows/wsl/file-permissions
@@ -289,10 +296,8 @@ static const struct fs_parameter_spec ntfs_fs_parameters[] = {
/*
* Load nls table or if @nls is utf8 then return NULL.
*
- * It is good idea to use here "const char *nls".
- * But load_nls accepts "char*".
*/
-static struct nls_table *ntfs_load_nls(char *nls)
+static struct nls_table *ntfs_load_nls(const char *nls)
{
struct nls_table *ret;
@@ -567,10 +572,8 @@ static void ntfs_create_procdir(struct super_block *sb)
if (e) {
struct ntfs_sb_info *sbi = sb->s_fs_info;
- proc_create_data("volinfo", 0444, e,
- &ntfs3_volinfo_fops, sb);
- proc_create_data("label", 0644, e,
- &ntfs3_label_fops, sb);
+ proc_create_data("volinfo", 0444, e, &ntfs3_volinfo_fops, sb);
+ proc_create_data("label", 0644, e, &ntfs3_label_fops, sb);
sbi->procdir = e;
}
}
@@ -601,10 +604,12 @@ static void ntfs_remove_proc_root(void)
}
}
#else
-static void ntfs_create_procdir(struct super_block *sb) {}
-static void ntfs_remove_procdir(struct super_block *sb) {}
-static void ntfs_create_proc_root(void) {}
-static void ntfs_remove_proc_root(void) {}
+// clang-format off
+static void ntfs_create_procdir(struct super_block *sb){}
+static void ntfs_remove_procdir(struct super_block *sb){}
+static void ntfs_create_proc_root(void){}
+static void ntfs_remove_proc_root(void){}
+// clang-format on
#endif
static struct kmem_cache *ntfs_inode_cachep;
@@ -1241,8 +1246,7 @@ static int ntfs_fill_super(struct super_block *sb, struct fs_context *fc)
sb->s_export_op = &ntfs_export_ops;
sb->s_time_gran = NTFS_TIME_GRAN; // 100 nsec
sb->s_xattr = ntfs_xattr_handlers;
- if (options->nocase)
- set_default_d_op(sb, &ntfs_dentry_ops);
+ set_default_d_op(sb, options->nocase ? &ntfs_dentry_ops : NULL);
options->nls = ntfs_load_nls(options->nls_name);
if (IS_ERR(options->nls)) {
@@ -1666,7 +1670,6 @@ static int ntfs_fill_super(struct super_block *sb, struct fs_context *fc)
out:
ntfs3_put_sbi(sbi);
kfree(boot2);
- ntfs3_put_sbi(sbi);
return err;
}
diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c
index 38bf8fadb346..2779f5fad59d 100644
--- a/fs/ocfs2/cluster/heartbeat.c
+++ b/fs/ocfs2/cluster/heartbeat.c
@@ -15,6 +15,7 @@
#include <linux/file.h>
#include <linux/kthread.h>
#include <linux/configfs.h>
+#include <linux/mutex.h>
#include <linux/random.h>
#include <linux/crc32.h>
#include <linux/time.h>
@@ -266,6 +267,9 @@ struct o2hb_region {
/* Message key for negotiate timeout message. */
unsigned int hr_key;
struct list_head hr_handler_list;
+ /* Serializes timeout arming against failed-start and teardown. */
+ struct mutex hr_arming_mutex;
+ bool hr_stopping;
/* last hb status, 0 for success, other value for error. */
int hr_last_hb_status;
@@ -330,9 +334,14 @@ static void o2hb_write_timeout(struct work_struct *work)
static void o2hb_arm_timeout(struct o2hb_region *reg)
{
+ mutex_lock(®->hr_arming_mutex);
+
+ if (reg->hr_stopping)
+ goto out_unlock;
+
/* Arm writeout only after thread reaches steady state */
if (atomic_read(®->hr_steady_iterations) != 0)
- return;
+ goto out_unlock;
mlog(ML_HEARTBEAT, "Queue write timeout for %u ms\n",
O2HB_MAX_WRITE_TIMEOUT_MS);
@@ -351,6 +360,18 @@ static void o2hb_arm_timeout(struct o2hb_region *reg)
schedule_delayed_work(®->hr_nego_timeout_work,
msecs_to_jiffies(O2HB_NEGO_TIMEOUT_MS));
bitmap_zero(reg->hr_nego_node_bitmap, O2NM_MAX_NODES);
+
+out_unlock:
+ mutex_unlock(®->hr_arming_mutex);
+}
+
+static void o2hb_queue_nego_timeout(struct o2hb_region *reg,
+ unsigned long delay)
+{
+ mutex_lock(®->hr_arming_mutex);
+ if (!reg->hr_stopping)
+ schedule_delayed_work(®->hr_nego_timeout_work, delay);
+ mutex_unlock(®->hr_arming_mutex);
}
static void o2hb_disarm_timeout(struct o2hb_region *reg)
@@ -359,6 +380,19 @@ static void o2hb_disarm_timeout(struct o2hb_region *reg)
cancel_delayed_work_sync(®->hr_nego_timeout_work);
}
+static void o2hb_set_region_stopping(struct o2hb_region *reg, bool stopping)
+{
+ mutex_lock(®->hr_arming_mutex);
+ reg->hr_stopping = stopping;
+ mutex_unlock(®->hr_arming_mutex);
+}
+
+static void o2hb_quiesce_timeout(struct o2hb_region *reg)
+{
+ o2hb_set_region_stopping(reg, true);
+ o2hb_disarm_timeout(reg);
+}
+
static int o2hb_send_nego_msg(int key, int type, u8 target, u8 node_num)
{
struct o2hb_nego_msg msg;
@@ -408,8 +442,7 @@ static void o2hb_nego_timeout(struct work_struct *work)
/* check negotiate bitmap every second to do timeout
* approve decision.
*/
- schedule_delayed_work(®->hr_nego_timeout_work,
- msecs_to_jiffies(1000));
+ o2hb_queue_nego_timeout(reg, msecs_to_jiffies(1000));
return;
}
@@ -1532,33 +1565,47 @@ static struct o2hb_region *to_o2hb_region(struct config_item *item)
return item ? container_of(item, struct o2hb_region, hr_item) : NULL;
}
-/* drop_item only drops its ref after killing the thread, nothing should
- * be using the region anymore. this has to clean up any state that
- * attributes might have built up. */
-static void o2hb_region_release(struct config_item *item)
+static void o2hb_unmap_slot_data(struct o2hb_region *reg)
{
int i;
struct page *page;
- struct o2hb_region *reg = to_o2hb_region(item);
-
- mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg_bdev(reg));
-
- kfree(reg->hr_tmp_block);
if (reg->hr_slot_data) {
for (i = 0; i < reg->hr_num_pages; i++) {
page = reg->hr_slot_data[i];
- if (page)
+ if (page) {
__free_page(page);
+ reg->hr_slot_data[i] = NULL;
+ }
}
kfree(reg->hr_slot_data);
+ reg->hr_slot_data = NULL;
}
+ kfree(reg->hr_slots);
+ reg->hr_slots = NULL;
+
+ kfree(reg->hr_tmp_block);
+ reg->hr_tmp_block = NULL;
+}
+
+/* drop_item only drops its ref after killing the thread, nothing should
+ * be using the region anymore. this has to clean up any state that
+ * attributes might have built up.
+ */
+static void o2hb_region_release(struct config_item *item)
+{
+ struct o2hb_region *reg = to_o2hb_region(item);
+
+ mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg_bdev(reg));
+
+ o2hb_quiesce_timeout(reg);
+ o2net_unregister_and_flush_handler_list(®->hr_handler_list);
+ o2hb_unmap_slot_data(reg);
+
if (reg->hr_bdev_file)
fput(reg->hr_bdev_file);
- kfree(reg->hr_slots);
-
debugfs_remove_recursive(reg->hr_debug_dir);
kfree(reg->hr_db_livenodes);
kfree(reg->hr_db_regnum);
@@ -1569,7 +1616,6 @@ static void o2hb_region_release(struct config_item *item)
list_del(®->hr_all_item);
spin_unlock(&o2hb_live_lock);
- o2net_unregister_handler_list(®->hr_handler_list);
kfree(reg);
}
@@ -1711,6 +1757,7 @@ static void o2hb_init_region_params(struct o2hb_region *reg)
static int o2hb_map_slot_data(struct o2hb_region *reg)
{
int i, j;
+ int ret = -ENOMEM;
unsigned int last_slot;
unsigned int spp = reg->hr_slots_per_page;
struct page *page;
@@ -1718,15 +1765,15 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
struct o2hb_disk_slot *slot;
reg->hr_tmp_block = kmalloc(reg->hr_block_bytes, GFP_KERNEL);
- if (reg->hr_tmp_block == NULL)
- return -ENOMEM;
+ if (!reg->hr_tmp_block)
+ goto out;
reg->hr_slots = kcalloc(reg->hr_blocks,
sizeof(struct o2hb_disk_slot), GFP_KERNEL);
- if (reg->hr_slots == NULL)
- return -ENOMEM;
+ if (!reg->hr_slots)
+ goto out;
- for(i = 0; i < reg->hr_blocks; i++) {
+ for (i = 0; i < reg->hr_blocks; i++) {
slot = ®->hr_slots[i];
slot->ds_node_num = i;
INIT_LIST_HEAD(&slot->ds_live_item);
@@ -1741,12 +1788,12 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
reg->hr_slot_data = kcalloc(reg->hr_num_pages, sizeof(struct page *),
GFP_KERNEL);
if (!reg->hr_slot_data)
- return -ENOMEM;
+ goto out;
- for(i = 0; i < reg->hr_num_pages; i++) {
+ for (i = 0; i < reg->hr_num_pages; i++) {
page = alloc_page(GFP_KERNEL);
if (!page)
- return -ENOMEM;
+ goto out;
reg->hr_slot_data[i] = page;
@@ -1766,6 +1813,10 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
}
return 0;
+
+out:
+ o2hb_unmap_slot_data(reg);
+ return ret;
}
/* Read in all the slots available and populate the tracking
@@ -1856,9 +1907,11 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
"blocksize %u incorrect for device, expected %d",
reg->hr_block_bytes, sectsize);
ret = -EINVAL;
- goto out3;
+ goto out;
}
+ reg->hr_aborted_start = 0;
+ reg->hr_node_deleted = 0;
o2hb_init_region_params(reg);
/* Generation of zero is invalid */
@@ -1870,18 +1923,15 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
ret = o2hb_map_slot_data(reg);
if (ret) {
mlog_errno(ret);
- goto out3;
+ goto out;
}
ret = o2hb_populate_slot_data(reg);
if (ret) {
mlog_errno(ret);
- goto out3;
+ goto out;
}
- INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout);
- INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout);
-
/*
* A node is considered live after it has beat LIVE_THRESHOLD
* times. We're not steady until we've given them a chance
@@ -1901,13 +1951,14 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
atomic_set(®->hr_steady_iterations, live_threshold);
/* unsteady_iterations is triple the steady_iterations */
atomic_set(®->hr_unsteady_iterations, (live_threshold * 3));
+ o2hb_set_region_stopping(reg, false);
hb_task = kthread_run(o2hb_thread, reg, "o2hb-%s",
reg->hr_item.ci_name);
if (IS_ERR(hb_task)) {
ret = PTR_ERR(hb_task);
mlog_errno(ret);
- goto out3;
+ goto out;
}
spin_lock(&o2hb_live_lock);
@@ -1924,12 +1975,12 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
if (reg->hr_aborted_start) {
ret = -EIO;
- goto out3;
+ goto out;
}
if (reg->hr_node_deleted) {
ret = -EINVAL;
- goto out3;
+ goto out;
}
/* Ok, we were woken. Make sure it wasn't by drop_item() */
@@ -1948,8 +1999,20 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
printk(KERN_NOTICE "o2hb: Heartbeat started on region %s (%pg)\n",
config_item_name(®->hr_item), reg_bdev(reg));
-out3:
+out:
if (ret < 0) {
+ o2hb_quiesce_timeout(reg);
+
+ spin_lock(&o2hb_live_lock);
+ hb_task = reg->hr_task;
+ reg->hr_task = NULL;
+ spin_unlock(&o2hb_live_lock);
+
+ if (hb_task)
+ kthread_stop(hb_task);
+
+ o2hb_unmap_slot_data(reg);
+
fput(reg->hr_bdev_file);
reg->hr_bdev_file = NULL;
}
@@ -2079,6 +2142,10 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g
*/
reg->hr_key = crc32_le(reg->hr_region_num + O2NM_MAX_REGIONS,
name, strlen(name));
+ mutex_init(®->hr_arming_mutex);
+ reg->hr_stopping = true;
+ INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout);
+ INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout);
INIT_LIST_HEAD(®->hr_handler_list);
ret = o2net_register_handler(O2HB_NEGO_TIMEOUT_MSG, reg->hr_key,
sizeof(struct o2hb_nego_msg),
@@ -2099,7 +2166,7 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g
return ®->hr_item;
unregister_handler:
- o2net_unregister_handler_list(®->hr_handler_list);
+ o2net_unregister_and_flush_handler_list(®->hr_handler_list);
remove_item:
spin_lock(&o2hb_live_lock);
list_del(®->hr_all_item);
@@ -2118,6 +2185,8 @@ static void o2hb_heartbeat_group_drop_item(struct config_group *group,
struct o2hb_region *reg = to_o2hb_region(item);
int quorum_region = 0;
+ o2hb_quiesce_timeout(reg);
+
/* stop the thread when the user removes the region dir */
spin_lock(&o2hb_live_lock);
hb_task = reg->hr_task;
diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c
index 822553530e74..d65a445535a3 100644
--- a/fs/ocfs2/cluster/tcp.c
+++ b/fs/ocfs2/cluster/tcp.c
@@ -38,6 +38,8 @@
*/
#include <linux/kernel.h>
+#include <linux/completion.h>
+#include <linux/mutex.h>
#include <linux/sched/mm.h>
#include <linux/jiffies.h>
#include <linux/slab.h>
@@ -102,9 +104,14 @@ static struct socket *o2net_listen_sock;
* quorum work is queued as sock containers are shutdown.. stop_listening
* tears down all the node's sock containers, preventing future shutdowns
* and queued quorum work, before canceling delayed quorum work and
- * destroying the work queue.
+ * destroying the work queue. Handler teardown can also race local listener
+ * shutdown, so keep a waitable destroying pointer until the old ordered
+ * queue has finished draining.
*/
static struct workqueue_struct *o2net_wq;
+static struct workqueue_struct *o2net_wq_destroying;
+static DEFINE_MUTEX(o2net_wq_mutex);
+static DECLARE_COMPLETION(o2net_wq_destroyed);
/* Heartbeat callbacks stay registered across local-node off/on. */
static bool o2net_listening;
static struct work_struct o2net_listen_work;
@@ -886,6 +893,27 @@ void o2net_unregister_handler_list(struct list_head *list)
}
EXPORT_SYMBOL_GPL(o2net_unregister_handler_list);
+static void o2net_flush_wq(void)
+{
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ wait_for_completion(&o2net_wq_destroyed);
+ return;
+ }
+
+ if (o2net_wq)
+ flush_workqueue(o2net_wq);
+ mutex_unlock(&o2net_wq_mutex);
+}
+
+void o2net_unregister_and_flush_handler_list(struct list_head *list)
+{
+ o2net_unregister_handler_list(list);
+ o2net_flush_wq();
+}
+EXPORT_SYMBOL_GPL(o2net_unregister_and_flush_handler_list);
+
static struct o2net_msg_handler *o2net_handler_get(u32 msg_type, u32 key)
{
struct o2net_msg_handler *nmh;
@@ -1617,7 +1645,7 @@ static void o2net_start_connect(struct work_struct *work)
myaddr.sin_addr.s_addr = mynode->nd_ipv4_address;
myaddr.sin_port = htons(0); /* any port */
- ret = sock->ops->bind(sock, (struct sockaddr *)&myaddr,
+ ret = sock->ops->bind(sock, (struct sockaddr_unsized *)&myaddr,
sizeof(myaddr));
if (ret) {
mlog(ML_ERROR, "bind failed with %d at address %pI4\n",
@@ -1717,12 +1745,10 @@ void o2net_disconnect_node(struct o2nm_node *node)
o2net_set_nn_state(nn, NULL, 0, -ENOTCONN);
spin_unlock(&nn->nn_lock);
- if (o2net_wq) {
- cancel_delayed_work(&nn->nn_connect_expired);
- cancel_delayed_work(&nn->nn_connect_work);
- cancel_delayed_work(&nn->nn_still_up);
- flush_workqueue(o2net_wq);
- }
+ cancel_delayed_work(&nn->nn_connect_expired);
+ cancel_delayed_work(&nn->nn_connect_work);
+ cancel_delayed_work(&nn->nn_still_up);
+ o2net_flush_wq();
}
static void o2net_hb_node_down_cb(struct o2nm_node *node, int node_num,
@@ -2046,7 +2072,7 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port)
INIT_WORK(&o2net_listen_work, o2net_accept_many);
sock->sk->sk_reuse = SK_CAN_REUSE;
- ret = sock->ops->bind(sock, (struct sockaddr *)&sin, sizeof(sin));
+ ret = sock->ops->bind(sock, (struct sockaddr_unsized *)&sin, sizeof(sin));
if (ret < 0) {
printk(KERN_ERR "o2net: Error %d while binding socket at "
"%pI4:%u\n", ret, &addr, ntohs(port));
@@ -2067,6 +2093,36 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port)
return ret;
}
+static void o2net_destroy_wq(void)
+{
+ struct workqueue_struct *wq;
+
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ wait_for_completion(&o2net_wq_destroyed);
+ return;
+ }
+
+ wq = o2net_wq;
+ if (!wq) {
+ mutex_unlock(&o2net_wq_mutex);
+ return;
+ }
+
+ reinit_completion(&o2net_wq_destroyed);
+ o2net_wq_destroying = wq;
+ mutex_unlock(&o2net_wq_mutex);
+
+ destroy_workqueue(wq);
+
+ mutex_lock(&o2net_wq_mutex);
+ o2net_wq = NULL;
+ o2net_wq_destroying = NULL;
+ complete_all(&o2net_wq_destroyed);
+ mutex_unlock(&o2net_wq_mutex);
+}
+
/*
* called from node manager when we should bring up our network listening
* socket. node manager handles all the serialization to only call this
@@ -2077,24 +2133,44 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port)
int o2net_start_listening(struct o2nm_node *node)
{
int ret = 0;
+ struct workqueue_struct *wq;
if (WARN_ON_ONCE(READ_ONCE(o2net_listening)))
return -EBUSY;
- BUG_ON(o2net_wq != NULL);
+
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ return -EBUSY;
+ }
+ if (WARN_ON_ONCE(o2net_wq)) {
+ mutex_unlock(&o2net_wq_mutex);
+ return -EBUSY;
+ }
+ mutex_unlock(&o2net_wq_mutex);
+
BUG_ON(o2net_listen_sock != NULL);
mlog(ML_KTHREAD, "starting o2net thread...\n");
- o2net_wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM);
- if (o2net_wq == NULL) {
+ wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM);
+ if (!wq) {
mlog(ML_ERROR, "unable to launch o2net thread\n");
return -ENOMEM; /* ? */
}
+ mutex_lock(&o2net_wq_mutex);
+ if (unlikely(o2net_wq_destroying || o2net_wq)) {
+ mutex_unlock(&o2net_wq_mutex);
+ destroy_workqueue(wq);
+ return -EBUSY;
+ }
+ o2net_wq = wq;
+ mutex_unlock(&o2net_wq_mutex);
+
ret = o2net_open_listening_sock(node->nd_ipv4_address,
node->nd_ipv4_port);
if (ret) {
- destroy_workqueue(o2net_wq);
- o2net_wq = NULL;
+ o2net_destroy_wq();
} else
o2quo_conn_up(node->nd_num);
@@ -2130,8 +2206,7 @@ void o2net_stop_listening(struct o2nm_node *node)
/* finish all work and tear down the work queue */
mlog(ML_KTHREAD, "waiting for o2net thread to exit....\n");
- destroy_workqueue(o2net_wq);
- o2net_wq = NULL;
+ o2net_destroy_wq();
sock_release(o2net_listen_sock);
o2net_listen_sock = NULL;
diff --git a/fs/ocfs2/cluster/tcp.h b/fs/ocfs2/cluster/tcp.h
index 2e86d42b5faf..a11bcee28947 100644
--- a/fs/ocfs2/cluster/tcp.h
+++ b/fs/ocfs2/cluster/tcp.h
@@ -89,6 +89,7 @@ int o2net_register_handler(u32 msg_type, u32 key, u32 max_len,
o2net_post_msg_handler_func *post_func,
struct list_head *unreg_list);
void o2net_unregister_handler_list(struct list_head *list);
+void o2net_unregister_and_flush_handler_list(struct list_head *list);
void o2net_fill_node_map(unsigned long *map, unsigned bytes);
diff --git a/fs/ocfs2/inode.c b/fs/ocfs2/inode.c
index 9ec3a7657b5b..835c00b90b7c 100644
--- a/fs/ocfs2/inode.c
+++ b/fs/ocfs2/inode.c
@@ -1665,6 +1665,29 @@ int ocfs2_validate_inode_block(struct super_block *sb,
}
}
+ if (S_ISLNK(le16_to_cpu(di->i_mode)) &&
+ !le32_to_cpu(di->i_clusters)) {
+ int max_inline = ocfs2_fast_symlink_chars(sb);
+ u64 i_size = le64_to_cpu(di->i_size);
+
+ if (i_size >= max_inline) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode #%llu: fast symlink i_size %llu exceeds max %d\n",
+ (unsigned long long)bh->b_blocknr,
+ (unsigned long long)i_size,
+ max_inline - 1);
+ goto bail;
+ }
+
+ if (strnlen((char *)di->id2.i_symlink, i_size + 1) != i_size) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode #%llu: fast symlink is not NUL-terminated at i_size %llu\n",
+ (unsigned long long)bh->b_blocknr,
+ (unsigned long long)i_size);
+ goto bail;
+ }
+ }
+
if (ocfs2_dinode_has_extents(di)) {
struct ocfs2_extent_list *el = &di->id2.i_list;
u16 count = le16_to_cpu(el->l_count);
diff --git a/fs/pnode.c b/fs/pnode.c
index 5d91c3e58d2a..2cd667958efe 100644
--- a/fs/pnode.c
+++ b/fs/pnode.c
@@ -410,19 +410,99 @@ bool propagation_would_overmount(const struct mount *from,
return false;
}
+/* Does @m receive propagation from @parent? */
+static bool receives_from(struct mount *m, struct mount *parent)
+{
+ if (m == parent)
+ return false;
+ for (; m; m = m->mnt_master)
+ if (m == parent || peers(m, parent))
+ return true;
+ return false;
+}
+
+/*
+ * Does @m receive propagation from the victim's parent as well? If so, then
+ * the mount at the victim's mountpoint inside of @m is a umount candidate as
+ * well. So it's the next candidate in the chain. Otherwise the chain ends at
+ * @m.
+ */
+static struct mount *next_candidate(struct mount *m, struct mount *victim)
+{
+ if (!receives_from(m, victim->mnt_parent))
+ return NULL;
+ return __lookup_mnt(&m->mnt, victim->mnt_mountpoint);
+}
+
+/*
+ * Would propagate_umount() pull out a mount of the chain of candidates that
+ * starts at @c, and does that mount have references beyond its own?
+ *
+ * This mirrors how trim_one(), trim_ancestors() and handle_locked() handle a
+ * synchronous umount:
+ *
+ * - single victim
+ * - without children
+ * - with MNT_LOCKED already cleared on every candidate by propagate_mount_unlock()
+ *
+ * A copy of the victim gets unmounted when each of its children is
+ * the next candidate in the chain or its overmount, unless the next
+ * unmount candidate is not its overmount and some unmount candidate further
+ * down has a child outside the chain. Keep this in sync with
+ * Documentation/filesystems/propagate_umount.txt.
+ */
+static bool chain_busy(struct mount *c, struct mount *victim)
+{
+ struct mount *m, *n, *next, *deepest = NULL;
+ bool above;
+
+ /* the deepest candidate with a child outside the chain */
+ for (m = c; m; m = next) {
+ next = next_candidate(m, victim);
+ list_for_each_entry(n, &m->mnt_mounts, mnt_child) {
+ if (n != next && n != victim) {
+ deepest = m;
+ break;
+ }
+ }
+ }
+
+ above = deepest != NULL; /* @deepest is at or below @m */
+ for (m = c; m; m = next) {
+ bool goes = true;
+
+ next = next_candidate(m, victim);
+ list_for_each_entry(n, &m->mnt_mounts, mnt_child) {
+ if (n != next && n != m->overmount && n != victim) {
+ goes = false;
+ break;
+ }
+ }
+ if (goes && next && next != m->overmount && above && m != deepest)
+ goes = false;
+ if (m == deepest)
+ above = false;
+ if (goes && do_refcount_check(m, 1))
+ return true;
+ }
+ return false;
+}
+
/*
* check if the mount 'mnt' can be unmounted successfully.
* @mnt: the mount to be checked for unmount
* NOTE: unmounting 'mnt' would naturally propagate to all
* other mounts its parent propagates to.
- * Check if any of these mounts that **do not have submounts**
- * have more references than 'refcnt'. If so return busy.
+ * Check if any of the mounts that propagate_umount() would pull out
+ * along with it have more references than their own. If so return busy.
*
* vfsmount lock must be held for write
*/
int propagate_mount_busy(struct mount *mnt, int refcnt)
{
struct mount *parent = mnt->mnt_parent;
+ struct dentry *mp = mnt->mnt_mountpoint;
+ struct mount *m;
/*
* quickly check if the current mount can be unmounted.
@@ -435,24 +515,16 @@ int propagate_mount_busy(struct mount *mnt, int refcnt)
if (mnt == parent)
return 0;
- for (struct mount *m = propagation_next(parent, parent); m;
- m = propagation_next(m, parent)) {
- struct list_head *head;
- struct mount *child = __lookup_mnt(&m->mnt, mnt->mnt_mountpoint);
+ /* the candidates are the mounts at @mp below the receivers */
+ for (m = propagation_next(parent, parent); m;
+ m = propagation_next(m, parent)) {
+ struct mount *c = __lookup_mnt(&m->mnt, mp);
- if (!child)
+ /* each chain once, from its top: skip receivers that are candidates */
+ if (!c || (mnt_has_parent(m) && m->mnt_mountpoint == mp &&
+ receives_from(m->mnt_parent, parent)))
continue;
-
- head = &child->mnt_mounts;
- if (!list_empty(head)) {
- /*
- * a mount that covers child completely wouldn't prevent
- * it being pulled out; any other would.
- */
- if (!list_is_singular(head) || !child->overmount)
- continue;
- }
- if (do_refcount_check(child, 1))
+ if (chain_busy(c, mnt))
return 1;
}
return 0;
diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c
index 55db47e969ce..e2529894a0df 100644
--- a/fs/resctrl/monitor.c
+++ b/fs/resctrl/monitor.c
@@ -580,14 +580,22 @@ void mon_event_count(void *info)
rr->err = 0;
}
-static struct rdt_ctrl_domain *get_ctrl_domain_from_cpu(int cpu,
- struct rdt_resource *r)
+/*
+ * Find the software controller's ctrl domain that contains @cpu on resource @r.
+ *
+ * Only called from the mbm_over worker via update_mba_bw() where the returned
+ * domain is kept alive by cancel_delayed_work_sync() in
+ * resctrl_offline_ctrl_domain(). This drains this worker and then waits on
+ * rdtgroup_mutex held here before the architecture can free the ctrl domain.
+ *
+ * Context: Call from RCU read-side critical section.
+ */
+static struct rdt_ctrl_domain *get_sc_ctrl_domain_from_cpu(int cpu,
+ struct rdt_resource *r)
{
struct rdt_ctrl_domain *d;
- lockdep_assert_cpus_held();
-
- list_for_each_entry(d, &r->ctrl_domains, hdr.list) {
+ list_for_each_entry_rcu(d, &r->ctrl_domains, hdr.list) {
/* Find the domain that contains this CPU */
if (cpumask_test_cpu(cpu, &d->hdr.cpu_mask))
return d;
@@ -648,7 +656,8 @@ static void update_mba_bw(struct rdtgroup *rgrp, struct rdt_l3_mon_domain *dom_m
if (WARN_ON_ONCE(!pmbm_data))
return;
- dom_mba = get_ctrl_domain_from_cpu(smp_processor_id(), r_mba);
+ guard(rcu)();
+ dom_mba = get_sc_ctrl_domain_from_cpu(smp_processor_id(), r_mba);
if (!dom_mba) {
pr_warn_once("Failure to get domain for MBA update\n");
return;
@@ -751,11 +760,25 @@ void cqm_handle_limbo(struct work_struct *work)
unsigned long delay = msecs_to_jiffies(CQM_LIMBOCHECK_INTERVAL);
struct rdt_l3_mon_domain *d;
- cpus_read_lock();
+ /*
+ * Safe to run without CPU hotplug lock. Work is guaranteed to be
+ * canceled before the domain structure is removed.
+ */
mutex_lock(&rdtgroup_mutex);
+ /*
+ * Ensure the worker is dedicated to a CPU as intended and not
+ * relocated by workqueue subsystem as part of CPU going offline.
+ */
+ if (!is_percpu_thread())
+ goto out_unlock;
+
d = container_of(work, struct rdt_l3_mon_domain, cqm_limbo.work);
+ /* Domain is going offline */
+ if (cpumask_empty(&d->hdr.cpu_mask))
+ goto out_unlock;
+
__check_limbo(d, false);
if (has_busy_rmid(d)) {
@@ -765,8 +788,8 @@ void cqm_handle_limbo(struct work_struct *work)
delay);
}
+out_unlock:
mutex_unlock(&rdtgroup_mutex);
- cpus_read_unlock();
}
/**
@@ -798,7 +821,10 @@ void mbm_handle_overflow(struct work_struct *work)
struct list_head *head;
struct rdt_resource *r;
- cpus_read_lock();
+ /*
+ * Safe to run without CPU hotplug lock. Work is guaranteed to be
+ * canceled before the domain structure is removed.
+ */
mutex_lock(&rdtgroup_mutex);
/*
@@ -808,9 +834,24 @@ void mbm_handle_overflow(struct work_struct *work)
if (!resctrl_mounted || !resctrl_arch_mon_capable())
goto out_unlock;
+ /*
+ * Ensure the worker is dedicated to a CPU and not relocated by
+ * workqueue subsystem as part of CPU going offline since reading
+ * events depend on smp_processor_id(). After passing this check
+ * smp_processor_id() is valid for entire duration of this worker
+ * since it runs with rdtgroup_mutex held and the offline handler needs
+ * rdtgroup_mutex to offline the CPU being run on here.
+ */
+ if (!is_percpu_thread())
+ goto out_unlock;
+
r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
d = container_of(work, struct rdt_l3_mon_domain, mbm_over.work);
+ /* Domain is going offline */
+ if (cpumask_empty(&d->hdr.cpu_mask))
+ goto out_unlock;
+
list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
mbm_update(r, d, prgrp);
@@ -832,7 +873,6 @@ void mbm_handle_overflow(struct work_struct *work)
out_unlock:
mutex_unlock(&rdtgroup_mutex);
- cpus_read_unlock();
}
/**
diff --git a/fs/resctrl/rdtgroup.c b/fs/resctrl/rdtgroup.c
index 1f81bd671155..df92305f2a19 100644
--- a/fs/resctrl/rdtgroup.c
+++ b/fs/resctrl/rdtgroup.c
@@ -4211,6 +4211,29 @@ static void domain_destroy_l3_mon_state(struct rdt_l3_mon_domain *d)
void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d)
{
+ /*
+ * mbm_handle_overflow() may dereference this ctrl domain via
+ * update_mba_bw()->get_sc_ctrl_domain_from_cpu(). The architecture has
+ * unlinked the domain from the RCU list and waited a grace period, so
+ * no new worker iteration can find it; drain any worker that already
+ * holds a pointer to it before the architecture frees the domain.
+ *
+ * Software controller is enabled/disabled on mount/unmount with
+ * cpus_read_lock() held. Running here with cpus_write_lock() so
+ * there are no concurrent changes to software controller status.
+ */
+ if (r->rid == RDT_RESOURCE_MBA && is_mba_sc(r)) {
+ struct rdt_resource *l3 = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+ struct rdt_l3_mon_domain *mon_d;
+
+ list_for_each_entry_rcu(mon_d, &l3->mon_domains, hdr.list, lockdep_is_cpus_held()) {
+ if (mon_d->hdr.id == d->hdr.id) {
+ cancel_delayed_work_sync(&mon_d->mbm_over);
+ break;
+ }
+ }
+ }
+
mutex_lock(&rdtgroup_mutex);
if (supports_mba_mbps() && r->rid == RDT_RESOURCE_MBA)
@@ -4221,6 +4244,20 @@ void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain
void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
{
+ /*
+ * Called by architecture under CPU hotplug lock as it prepares to remove
+ * the domain which is guaranteed to be accessible here.
+ * The domain has been unlinked from the RCU list and a grace period
+ * has elapsed, so no new worker can be scheduled. Drain any worker that
+ * is in flight or pending before letting architecture proceed to free
+ * the domain that has the workers' struct delayed_work embedded.
+ * Do so before taking rdtgroup_mutex since the workers also acquire it.
+ */
+ if (resctrl_is_mbm_enabled())
+ cancel_delayed_work_sync(&d->mbm_over);
+ if (resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID))
+ cancel_delayed_work_sync(&d->cqm_limbo);
+
mutex_lock(&rdtgroup_mutex);
/*
@@ -4230,8 +4267,6 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_l3_mon_domain
if (resctrl_mounted && resctrl_arch_mon_capable())
rmdir_mondata_subdir_allrdtgrp(r, d);
- if (resctrl_is_mbm_enabled())
- cancel_delayed_work(&d->mbm_over);
if (resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID) && has_busy_rmid(d)) {
/*
* When a package is going down, forcefully
@@ -4242,7 +4277,6 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_l3_mon_domain
* package never comes back.
*/
__check_limbo(d, true);
- cancel_delayed_work(&d->cqm_limbo);
}
domain_destroy_l3_mon_state(d);
@@ -4407,12 +4441,16 @@ void resctrl_offline_cpu(unsigned int cpu)
d = get_mon_domain_from_cpu(cpu, l3);
if (d) {
if (resctrl_is_mbm_enabled() && cpu == d->mbm_work_cpu) {
- cancel_delayed_work(&d->mbm_over);
+ mutex_unlock(&rdtgroup_mutex);
+ cancel_delayed_work_sync(&d->mbm_over);
+ mutex_lock(&rdtgroup_mutex);
mbm_setup_overflow_handler(d, 0, cpu);
}
if (resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID) &&
cpu == d->cqm_work_cpu && has_busy_rmid(d)) {
- cancel_delayed_work(&d->cqm_limbo);
+ mutex_unlock(&rdtgroup_mutex);
+ cancel_delayed_work_sync(&d->cqm_limbo);
+ mutex_lock(&rdtgroup_mutex);
cqm_setup_limbo_handler(d, 0, cpu);
}
}
diff --git a/fs/smb/client/cifsacl.c b/fs/smb/client/cifsacl.c
index ef72ec55c60f..b03560f5cd87 100644
--- a/fs/smb/client/cifsacl.c
+++ b/fs/smb/client/cifsacl.c
@@ -1314,6 +1314,7 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
{
int rc = 0;
struct smb_sid *owner_sid_ptr, *group_sid_ptr;
+ unsigned int sbflags = cifs_sb_flags(cifs_sb);
struct smb_acl *dacl_ptr; /* no need for SACL ptr */
char *end_of_acl = ((char *)pntsd) + acl_len;
__u32 dacloffset;
@@ -1331,16 +1332,21 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
le32_to_cpu(pntsd->gsidoffset),
le32_to_cpu(pntsd->sacloffset), dacloffset);
/* cifs_dump_mem("owner_sid: ", owner_sid_ptr, 64); */
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+
rc = parse_sid(owner_sid_ptr, end_of_acl);
if (rc) {
cifs_dbg(FYI, "%s: Error %d parsing Owner SID\n", __func__, rc);
return rc;
}
- rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER);
- if (rc) {
- cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n",
- __func__, rc);
- return rc;
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID)) {
+ rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER);
+ if (rc) {
+ cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n",
+ __func__, rc);
+ return rc;
+ }
}
rc = parse_sid(group_sid_ptr, end_of_acl);
@@ -1349,11 +1355,13 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
__func__, rc);
return rc;
}
- rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP);
- if (rc) {
- cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n",
- __func__, rc);
- return rc;
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID)) {
+ rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP);
+ if (rc) {
+ cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n",
+ __func__, rc);
+ return rc;
+ }
}
if (dacloffset) {
diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c
index 30daa09ab1b6..f63519a0b1d8 100644
--- a/fs/smb/client/cifsfs.c
+++ b/fs/smb/client/cifsfs.c
@@ -474,6 +474,7 @@ cifs_alloc_inode(struct super_block *sb)
return NULL;
cifs_inode->cifsAttrs = ATTR_ARCHIVE; /* default */
cifs_inode->time = 0;
+ cifs_inode->time_last_write = 0;
/*
* Until the file is open and we have gotten oplock info back from the
* server, can not assume caching of file data or metadata.
diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h
index 562967d44bf3..8d751026dd4a 100644
--- a/fs/smb/client/cifsglob.h
+++ b/fs/smb/client/cifsglob.h
@@ -1579,6 +1579,7 @@ struct cifsInodeInfo {
spinlock_t writers_lock;
unsigned int writers; /* Number of writers on this inode */
unsigned long time; /* jiffies of last update of inode */
+ unsigned long time_last_write; /* jiffies of last writable close or truncate */
u64 uniqueid; /* server inode number */
u64 createtime; /* creation time on server */
__u8 lease_key[SMB2_LEASE_KEY_SIZE]; /* lease key for this inode */
diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c
index f589b188ccfc..151399de7d6a 100644
--- a/fs/smb/client/connect.c
+++ b/fs/smb/client/connect.c
@@ -3121,7 +3121,7 @@ bind_socket(struct TCP_Server_Info *server)
struct socket *socket = server->ssocket;
rc = kernel_bind(socket,
- (struct sockaddr *) &server->srcaddr,
+ (struct sockaddr_unsized *) &server->srcaddr,
sizeof(server->srcaddr));
if (rc < 0) {
struct sockaddr_in *saddr4;
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index 1e557601804f..4eb1ac4f7b60 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -906,6 +906,14 @@ void _cifsFileInfo_put(struct cifsFileInfo *cifs_file,
cifs_set_oplock_level(cifsi, 0);
}
+ if (OPEN_FMODE(cifs_file->f_flags) & FMODE_WRITE) {
+ /* Stamp while open_file_lock is held; covers all close paths
+ * including background I/O. Pairs with smp_load_acquire() in
+ * is_size_safe_to_change().
+ */
+ smp_store_release(&cifsi->time_last_write, jiffies);
+ }
+
spin_unlock(&cifsi->open_file_lock);
spin_unlock(&tcon->open_file_lock);
@@ -1414,11 +1422,12 @@ void smb2_deferred_work_close(struct work_struct *work)
{
struct cifsFileInfo *cfile = container_of(work,
struct cifsFileInfo, deferred.work);
+ struct cifsInodeInfo *cinode = CIFS_I(d_inode(cfile->dentry));
- spin_lock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
+ spin_lock(&cinode->deferred_lock);
cifs_del_deferred_close(cfile);
cfile->deferred_close_scheduled = false;
- spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
+ spin_unlock(&cinode->deferred_lock);
_cifsFileInfo_put(cfile, true, false);
}
@@ -3120,15 +3129,27 @@ static int is_inode_writable(struct cifsInodeInfo *cifs_inode)
bool is_size_safe_to_change(struct cifsInodeInfo *cifsInode, __u64 end_of_file,
bool from_readdir)
{
+ struct cifs_sb_info *cifs_sb;
+ unsigned long tlw;
+
if (!cifsInode)
return true;
+ cifs_sb = CIFS_SB(cifsInode->netfs.inode.i_sb);
+
if (is_inode_writable(cifsInode) ||
((cifsInode->oplock & CIFS_CACHE_RW_FLG) != 0 && from_readdir)) {
/* This inode is open for write at least once */
- struct cifs_sb_info *cifs_sb;
- cifs_sb = CIFS_SB(cifsInode->netfs.inode.i_sb);
+ /*
+ * Readdir data is unreliable when we have writable handles or
+ * an exclusive lease -- never allow it to change i_size, even
+ * on direct-IO mounts where the server's directory metadata
+ * can still lag behind the actual file state.
+ */
+ if (from_readdir)
+ return false;
+
if (cifs_sb_flags(cifs_sb) & CIFS_MOUNT_DIRECT_IO) {
/* since no page cache to corrupt on directio
we can change size safely */
@@ -3139,8 +3160,40 @@ bool is_size_safe_to_change(struct cifsInodeInfo *cifsInode, __u64 end_of_file,
return true;
return false;
- } else
- return true;
+ }
+
+ /*
+ * No writable handles open. Check whether we are within the attribute
+ * cache validity window of a recent local modification.
+ *
+ * For the close() path: _cifsFileInfo_put() stamps time_last_write
+ * (via smp_store_release()) before releasing open_file_lock. That
+ * spin_unlock() is a store-release that pairs with the spin_lock()
+ * (load-acquire) in is_inode_writable() above, so if
+ * is_inode_writable() returned false the smp_load_acquire() below is
+ * guaranteed to observe any time_last_write update from a concurrent
+ * close(), covering all close paths including background I/O.
+ *
+ * For the setattr/truncate paths: those callers use smp_store_release()
+ * directly; the smp_load_acquire() below pairs with that store. There
+ * is no shared lock between setattr and readdir, so this relies on
+ * acquire-release semantics alone. The store propagation latency on
+ * weakly-ordered architectures (nanoseconds) is negligible relative to
+ * the acregmax window (seconds) and the readdir RPC round-trip
+ * (milliseconds), making this a sound design choice in practice.
+ *
+ * time_last_write == 0 means the inode has never been written locally;
+ * skip the window check to avoid false positives near boot time when
+ * jiffies is still close to INITIAL_JIFFIES on 32-bit systems.
+ */
+ if (from_readdir) {
+ /* Pairs with smp_store_release() in _cifsFileInfo_put() and setattr. */
+ tlw = smp_load_acquire(&cifsInode->time_last_write);
+ if (tlw && time_before(jiffies, tlw + cifs_sb->ctx->acregmax))
+ return false;
+ }
+
+ return true;
}
void cifs_oplock_break(struct work_struct *work)
diff --git a/fs/smb/client/inode.c b/fs/smb/client/inode.c
index 7f8d2edb828e..ec6d27ba4083 100644
--- a/fs/smb/client/inode.c
+++ b/fs/smb/client/inode.c
@@ -236,6 +236,8 @@ cifs_fattr_to_inode(struct inode *inode, struct cifs_fattr *fattr,
if (is_size_safe_to_change(cifs_i, fattr->cf_eof, from_readdir)) {
i_size_write(inode, fattr->cf_eof);
inode->i_blocks = CIFS_INO_BLOCKS(fattr->cf_bytes);
+ } else if (from_readdir && i_size_read(inode) != fattr->cf_eof) {
+ cifs_i->time = 0;
}
if (S_ISLNK(fattr->cf_mode) && fattr->cf_symlink_target) {
@@ -847,6 +849,7 @@ static void smb311_posix_info_to_fattr(struct cifs_fattr *fattr,
struct smb311_posix_qinfo *info = &data->posix_fi;
struct cifs_sb_info *cifs_sb = CIFS_SB(sb);
struct cifs_tcon *tcon = cifs_sb_master_tcon(cifs_sb);
+ unsigned int sbflags = cifs_sb_flags(cifs_sb);
memset(fattr, 0, sizeof(*fattr));
@@ -891,8 +894,12 @@ static void smb311_posix_info_to_fattr(struct cifs_fattr *fattr,
fattr->cf_symlink_target = data->symlink_target;
data->symlink_target = NULL;
}
- sid_to_id(cifs_sb, &data->posix_owner, fattr, SIDOWNER);
- sid_to_id(cifs_sb, &data->posix_group, fattr, SIDGROUP);
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID))
+ sid_to_id(cifs_sb, &data->posix_owner, fattr, SIDOWNER);
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID))
+ sid_to_id(cifs_sb, &data->posix_group, fattr, SIDGROUP);
cifs_dbg(FYI, "POSIX query info: mode 0x%x uniqueid 0x%llx nlink %d\n",
fattr->cf_mode, fattr->cf_uniqueid, fattr->cf_nlink);
@@ -3028,7 +3035,10 @@ int cifs_fiemap(struct inode *inode, struct fiemap_extent_info *fei, u64 start,
void cifs_setsize(struct inode *inode, loff_t offset)
{
+ loff_t old_size;
+
spin_lock(&inode->i_lock);
+ old_size = i_size_read(inode);
i_size_write(inode, offset);
/*
* Until we can query the server for actual allocation size,
@@ -3037,6 +3047,8 @@ void cifs_setsize(struct inode *inode, loff_t offset)
inode->i_blocks = CIFS_INO_BLOCKS(offset);
spin_unlock(&inode->i_lock);
inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode));
+ if (offset > old_size)
+ pagecache_isize_extended(inode, old_size, offset);
truncate_pagecache(inode, offset);
netfs_wait_for_outstanding_io(inode);
fscache_resize_cookie(cifs_inode_cookie(inode), offset);
@@ -3123,8 +3135,11 @@ int cifs_file_set_size(const unsigned int xid, struct dentry *dentry,
cifs_put_tlink(tlink);
set_size_out:
- if (rc == 0)
+ if (rc == 0) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cifsInode->time_last_write, jiffies);
cifs_resize_file_locked(inode, size);
+ }
return rc;
}
@@ -3382,7 +3397,8 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs)
gid = attrs->ia_gid;
if ((cifs_sb_flags(cifs_sb) & CIFS_MOUNT_CIFS_ACL) ||
- (cifs_sb_flags(cifs_sb) & CIFS_MOUNT_MODE_FROM_SID)) {
+ (cifs_sb_flags(cifs_sb) & CIFS_MOUNT_MODE_FROM_SID) ||
+ posix) {
if (uid_valid(uid) || gid_valid(gid)) {
mode = NO_CHANGE_64;
rc = id_mode_to_cifs_acl(inode, full_path, &mode,
diff --git a/fs/smb/client/misc.c b/fs/smb/client/misc.c
index 9c32213e9379..3bb9e60c0ea4 100644
--- a/fs/smb/client/misc.c
+++ b/fs/smb/client/misc.c
@@ -776,7 +776,7 @@ cifs_del_deferred_close(struct cifsFileInfo *cfile)
void
cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
{
- struct cifsFileInfo *cfile = NULL;
+ struct cifsFileInfo *cfile = NULL, *failed_cfile = NULL;
struct file_list *tmp_list, *tmp_next_list;
LIST_HEAD(file_head);
@@ -791,9 +791,12 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
cifs_del_deferred_close(cfile);
spin_unlock(&cifs_inode->deferred_lock);
- tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ tmp_list = kmalloc_obj(struct file_list,
+ GFP_ATOMIC);
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
@@ -801,6 +804,9 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
}
spin_unlock(&cifs_inode->open_file_lock);
+ if (failed_cfile)
+ _cifsFileInfo_put(failed_cfile, false, false);
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
_cifsFileInfo_put(tmp_list->cfile, false, false);
list_del(&tmp_list->list);
@@ -811,7 +817,7 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
void
cifs_close_all_deferred_files(struct cifs_tcon *tcon)
{
- struct cifsFileInfo *cfile;
+ struct cifsFileInfo *cfile, *failed_cfile = NULL;
struct file_list *tmp_list, *tmp_next_list;
LIST_HEAD(file_head);
@@ -823,9 +829,12 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon)
cifs_del_deferred_close(cfile);
spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
- tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ tmp_list = kmalloc_obj(struct file_list,
+ GFP_ATOMIC);
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
@@ -833,6 +842,9 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon)
}
spin_unlock(&tcon->open_file_lock);
+ if (failed_cfile)
+ _cifsFileInfo_put(failed_cfile, true, false);
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
_cifsFileInfo_put(tmp_list->cfile, true, false);
list_del(&tmp_list->list);
@@ -881,7 +893,7 @@ void cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon,
struct dentry *dentry)
{
struct file_list *tmp_list, *tmp_next_list;
- struct cifsFileInfo *cfile;
+ struct cifsFileInfo *cfile, *failed_cfile = NULL;
LIST_HEAD(file_head);
spin_lock(&tcon->open_file_lock);
@@ -893,15 +905,20 @@ void cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon,
cifs_del_deferred_close(cfile);
spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
- tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ tmp_list = kmalloc_obj(struct file_list, GFP_ATOMIC);
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
}
spin_unlock(&tcon->open_file_lock);
+ if (failed_cfile)
+ _cifsFileInfo_put(failed_cfile, true, false);
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
_cifsFileInfo_put(tmp_list->cfile, true, false);
list_del(&tmp_list->list);
diff --git a/fs/smb/client/readdir.c b/fs/smb/client/readdir.c
index 9ca859a2451e..0cc5ed56571a 100644
--- a/fs/smb/client/readdir.c
+++ b/fs/smb/client/readdir.c
@@ -242,9 +242,11 @@ static void
cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info,
struct cifs_sb_info *cifs_sb)
{
+ unsigned int sbflags = cifs_sb_flags(cifs_sb);
struct smb2_posix_info_parsed parsed;
+ int rc;
- posix_info_parse(info, NULL, &parsed);
+ rc = posix_info_parse(info, NULL, &parsed);
memset(fattr, 0, sizeof(*fattr));
fattr->cf_uniqueid = le64_to_cpu(info->Inode);
@@ -281,8 +283,17 @@ cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info,
le32_to_cpu(info->ReparseTag),
le32_to_cpu(info->Mode));
- sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER);
- sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP);
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+ if (rc < 0) {
+ cifs_dbg(VFS, "%s: failed to parse SIDs: %d\n",
+ __func__, rc);
+ } else {
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID))
+ sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER);
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID))
+ sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP);
+ }
}
static void __dir_info_to_fattr(struct cifs_fattr *fattr, const void *info)
diff --git a/fs/smb/server/connection.c b/fs/smb/server/connection.c
index ff6cdd720dd4..16fcedf45739 100644
--- a/fs/smb/server/connection.c
+++ b/fs/smb/server/connection.c
@@ -242,15 +242,37 @@ void ksmbd_conn_unlock(struct ksmbd_conn *conn)
mutex_unlock(&conn->srv_mutex);
}
-void ksmbd_all_conn_set_status(u64 sess_id, u32 status)
+static bool ksmbd_session_is_bound_to_conn(struct ksmbd_session *sess,
+ struct ksmbd_conn *conn)
+{
+ bool found;
+
+ rcu_read_lock();
+ found = xa_load(&conn->sessions, sess->id) == sess;
+ rcu_read_unlock();
+ if (found)
+ return true;
+
+ down_read(&sess->chann_lock);
+ found = xa_load(&sess->ksmbd_chann_list, (long)conn);
+ up_read(&sess->chann_lock);
+ return found;
+}
+
+void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status)
{
struct ksmbd_conn *conn;
int bkt;
down_read(&conn_list_lock);
hash_for_each(conn_list, bkt, conn, hlist) {
- if (conn->binding || xa_load(&conn->sessions, sess_id))
- WRITE_ONCE(conn->status, status);
+ if (ksmbd_session_is_bound_to_conn(sess, conn)) {
+ spin_lock(&conn->request_lock);
+ if (!ksmbd_conn_exiting(conn) &&
+ !ksmbd_conn_releasing(conn))
+ WRITE_ONCE(conn->status, status);
+ spin_unlock(&conn->request_lock);
+ }
}
up_read(&conn_list_lock);
}
@@ -276,7 +298,8 @@ void ksmbd_conn_wait_idle(struct ksmbd_conn *conn)
wait_event(conn->req_running_q, atomic_read(&conn->req_running) < 2);
}
-int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id)
+int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn,
+ struct ksmbd_session *sess)
{
struct ksmbd_conn *conn;
int rc, retry_count = 0, max_timeout = 120;
@@ -288,7 +311,7 @@ int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id)
down_read(&conn_list_lock);
hash_for_each(conn_list, bkt, conn, hlist) {
- if (conn->binding || xa_load(&conn->sessions, sess_id)) {
+ if (ksmbd_session_is_bound_to_conn(sess, conn)) {
rcount = (conn == curr_conn) ? 2 : 1;
if (atomic_read(&conn->req_running) >= rcount) {
rc = wait_event_timeout(conn->req_running_q,
@@ -610,8 +633,10 @@ static void stop_sessions(void)
* handler exited its receive loop for an unrelated
* reason).
*/
- if (READ_ONCE(conn->status) != KSMBD_SESS_RELEASING)
+ spin_lock(&conn->request_lock);
+ if (!ksmbd_conn_releasing(conn))
ksmbd_conn_set_exiting(conn);
+ spin_unlock(&conn->request_lock);
target = conn;
break;
}
diff --git a/fs/smb/server/connection.h b/fs/smb/server/connection.h
index ceb1c1aefbf6..d6c116875be9 100644
--- a/fs/smb/server/connection.h
+++ b/fs/smb/server/connection.h
@@ -21,6 +21,7 @@
#include "ksmbd_work.h"
struct smbdirect_buffer_descriptor_v1;
+struct ksmbd_session;
#define KSMBD_SOCKET_BACKLOG 16
@@ -163,7 +164,8 @@ extern struct rw_semaphore conn_list_lock;
bool ksmbd_conn_alive(struct ksmbd_conn *conn);
void ksmbd_conn_wait_idle(struct ksmbd_conn *conn);
-int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id);
+int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn,
+ struct ksmbd_session *sess);
struct ksmbd_conn *ksmbd_conn_alloc(void);
void ksmbd_conn_free(struct ksmbd_conn *conn);
struct ksmbd_conn *ksmbd_conn_get(struct ksmbd_conn *conn);
@@ -268,5 +270,5 @@ static inline void ksmbd_conn_set_releasing(struct ksmbd_conn *conn)
WRITE_ONCE(conn->status, KSMBD_SESS_RELEASING);
}
-void ksmbd_all_conn_set_status(u64 sess_id, u32 status);
+void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status);
#endif /* __CONNECTION_H__ */
diff --git a/fs/smb/server/mgmt/tree_connect.c b/fs/smb/server/mgmt/tree_connect.c
index d3483d9c757c..c7a89eddcda7 100644
--- a/fs/smb/server/mgmt/tree_connect.c
+++ b/fs/smb/server/mgmt/tree_connect.c
@@ -79,8 +79,10 @@ ksmbd_tree_conn_connect(struct ksmbd_work *work, const char *share_name)
status.tree_conn = tree_conn;
atomic_set(&tree_conn->refcount, 1);
+ down_write(&sess->tree_conns_lock);
ret = xa_err(xa_store(&sess->tree_conns, tree_conn->id, tree_conn,
KSMBD_DEFAULT_GFP));
+ up_write(&sess->tree_conns_lock);
if (ret) {
status.ret = -ENOMEM;
goto out_error;
@@ -103,15 +105,11 @@ void ksmbd_tree_connect_put(struct ksmbd_tree_connect *tcon)
kfree(tcon);
}
-int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
- struct ksmbd_tree_connect *tree_conn)
+static int __ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
+ struct ksmbd_tree_connect *tree_conn)
{
int ret;
- write_lock(&sess->tree_conns_lock);
- xa_erase(&sess->tree_conns, tree_conn->id);
- write_unlock(&sess->tree_conns_lock);
-
ret = ksmbd_ipc_tree_disconnect_request(sess->id, tree_conn->id);
ksmbd_release_tree_conn_id(sess, tree_conn->id);
ksmbd_share_config_put(tree_conn->share_conf);
@@ -120,12 +118,22 @@ int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
return ret;
}
+int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
+ struct ksmbd_tree_connect *tree_conn)
+{
+ down_write(&sess->tree_conns_lock);
+ xa_erase(&sess->tree_conns, tree_conn->id);
+ up_write(&sess->tree_conns_lock);
+
+ return __ksmbd_tree_conn_disconnect(sess, tree_conn);
+}
+
struct ksmbd_tree_connect *ksmbd_tree_conn_lookup(struct ksmbd_session *sess,
unsigned int id)
{
struct ksmbd_tree_connect *tcon;
- read_lock(&sess->tree_conns_lock);
+ down_read(&sess->tree_conns_lock);
tcon = xa_load(&sess->tree_conns, id);
if (tcon) {
if (tcon->t_state != TREE_CONNECTED)
@@ -133,7 +141,7 @@ struct ksmbd_tree_connect *ksmbd_tree_conn_lookup(struct ksmbd_session *sess,
else if (!atomic_inc_not_zero(&tcon->refcount))
tcon = NULL;
}
- read_unlock(&sess->tree_conns_lock);
+ up_read(&sess->tree_conns_lock);
return tcon;
}
@@ -147,18 +155,19 @@ int ksmbd_tree_conn_session_logoff(struct ksmbd_session *sess)
if (!sess)
return -EINVAL;
+ down_write(&sess->tree_conns_lock);
xa_for_each(&sess->tree_conns, id, tc) {
- write_lock(&sess->tree_conns_lock);
if (tc->t_state == TREE_DISCONNECTED) {
- write_unlock(&sess->tree_conns_lock);
ret = -ENOENT;
continue;
}
tc->t_state = TREE_DISCONNECTED;
- write_unlock(&sess->tree_conns_lock);
- ret |= ksmbd_tree_conn_disconnect(sess, tc);
+ xa_erase(&sess->tree_conns, tc->id);
+ ret |= __ksmbd_tree_conn_disconnect(sess, tc);
}
xa_destroy(&sess->tree_conns);
+ up_write(&sess->tree_conns_lock);
+
return ret;
}
diff --git a/fs/smb/server/mgmt/user_config.c b/fs/smb/server/mgmt/user_config.c
index 53db14e91bfd..0e5fcd6f95c1 100644
--- a/fs/smb/server/mgmt/user_config.c
+++ b/fs/smb/server/mgmt/user_config.c
@@ -37,6 +37,17 @@ struct ksmbd_user *ksmbd_alloc_user(struct ksmbd_login_response *resp,
{
struct ksmbd_user *user;
+ /*
+ * resp->hash_sz is a __u16 taken from the mountd IPC login response but
+ * resp->hash[] is only KSMBD_REQ_MAX_HASH_SZ bytes. A malformed or
+ * malicious response can set hash_sz far beyond that (up to 65535),
+ * making the memcpy() below read past the response object
+ * (slab-out-of-bounds in ksmbd_alloc_user()). Reject any oversized
+ * hash rather than trust the length.
+ */
+ if (resp->hash_sz > sizeof(resp->hash))
+ return NULL;
+
user = kmalloc(sizeof(struct ksmbd_user), KSMBD_DEFAULT_GFP);
if (!user)
return NULL;
diff --git a/fs/smb/server/mgmt/user_session.c b/fs/smb/server/mgmt/user_session.c
index 920eb3963ae9..3a2591434b5f 100644
--- a/fs/smb/server/mgmt/user_session.c
+++ b/fs/smb/server/mgmt/user_session.c
@@ -413,16 +413,16 @@ void destroy_previous_session(struct ksmbd_conn *conn,
memcmp(user->passkey, prev_user->passkey, user->passkey_sz))
goto out;
- ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_RECONNECT);
- err = ksmbd_conn_wait_idle_sess_id(conn, id);
+ ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_RECONNECT);
+ err = ksmbd_conn_wait_idle_sess(conn, prev_sess);
if (err) {
- ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP);
+ ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP);
goto out;
}
ksmbd_destroy_file_table(prev_sess);
prev_sess->state = SMB2_SESSION_EXPIRED;
- ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP);
+ ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP);
ksmbd_launch_ksmbd_durable_scavenger();
out:
up_write(&conn->session_lock);
@@ -481,8 +481,8 @@ static struct ksmbd_session *__session_create(int protocol)
xa_init(&sess->ksmbd_chann_list);
xa_init(&sess->rpc_handle_list);
sess->sequence_number = 1;
- rwlock_init(&sess->tree_conns_lock);
atomic_set(&sess->refcnt, 2);
+ init_rwsem(&sess->tree_conns_lock);
init_rwsem(&sess->rpc_lock);
init_rwsem(&sess->chann_lock);
diff --git a/fs/smb/server/mgmt/user_session.h b/fs/smb/server/mgmt/user_session.h
index 525650d6876c..10132a3f1e4a 100644
--- a/fs/smb/server/mgmt/user_session.h
+++ b/fs/smb/server/mgmt/user_session.h
@@ -61,7 +61,7 @@ struct ksmbd_session {
struct ksmbd_file_table file_table;
unsigned long last_active;
- rwlock_t tree_conns_lock;
+ struct rw_semaphore tree_conns_lock;
atomic_t refcnt;
struct rw_semaphore rpc_lock;
diff --git a/fs/smb/server/oplock.c b/fs/smb/server/oplock.c
index 6183bd52ca9d..08e3e4504b03 100644
--- a/fs/smb/server/oplock.c
+++ b/fs/smb/server/oplock.c
@@ -56,6 +56,7 @@ static struct oplock_info *alloc_opinfo(struct ksmbd_work *work,
opinfo->conn = ksmbd_conn_get(work->conn);
opinfo->level = SMB2_OPLOCK_LEVEL_NONE;
opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock_init(&opinfo->state_lock);
opinfo->pending_break = 0;
opinfo->fid = id;
opinfo->Tid = Tid;
@@ -424,14 +425,23 @@ void close_id_del_oplock(struct ksmbd_file *fp)
opinfo_del(opinfo);
rcu_assign_pointer(fp->f_opinfo, NULL);
- if (opinfo->op_state == OPLOCK_ACK_WAIT) {
- opinfo->op_state = OPLOCK_CLOSING;
- wake_up_interruptible_all(&opinfo->oplock_q);
- if (opinfo->is_lease) {
- atomic_set(&opinfo->breaking_cnt, 0);
- wake_up_interruptible_all(&opinfo->oplock_brk);
- }
- }
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state == OPLOCK_ACK_WAIT && opinfo->is_lease)
+ atomic_set(&opinfo->breaking_cnt, 0);
+ /*
+ * An opinfo that has been removed from the inode list is terminal. Keep
+ * this transition and releasing pending_break under state_lock. a breaker
+ * takes the same lock before it acquires pending_break or sets ACK_WAIT.
+ */
+ opinfo->op_state = OPLOCK_CLOSING;
+ clear_bit_unlock(0, &opinfo->pending_break);
+ spin_unlock(&opinfo->state_lock);
+ wake_up_interruptible_all(&opinfo->oplock_q);
+ if (opinfo->is_lease)
+ wake_up_interruptible_all(&opinfo->oplock_brk);
+ /* memory barrier is needed for wake_up_bit() */
+ smp_mb__after_atomic();
+ wake_up_bit(&opinfo->pending_break, 0);
opinfo_count_dec(fp);
atomic_dec(&opinfo->refcount);
@@ -610,10 +620,16 @@ static void wait_for_break_ack(struct oplock_info *opinfo)
/* is this a timeout ? */
if (!rc) {
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state == OPLOCK_CLOSING) {
+ spin_unlock(&opinfo->state_lock);
+ return;
+ }
if (opinfo->is_lease)
opinfo->o_lease->state = SMB2_LEASE_NONE_LE;
opinfo->level = SMB2_OPLOCK_LEVEL_NONE;
opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
}
}
@@ -625,17 +641,46 @@ static void wake_up_oplock_break(struct oplock_info *opinfo)
wake_up_bit(&opinfo->pending_break, 0);
}
+static bool oplock_break_set_ack_wait(struct oplock_info *opinfo)
+{
+ bool ret = false;
+
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING) {
+ opinfo->op_state = OPLOCK_ACK_WAIT;
+ ret = true;
+ }
+ spin_unlock(&opinfo->state_lock);
+
+ return ret;
+}
+
static int oplock_break_pending(struct oplock_info *opinfo, int req_op_level)
{
- while (test_and_set_bit(0, &opinfo->pending_break)) {
+ for (;;) {
+ bool closing;
+
+ spin_lock(&opinfo->state_lock);
+ closing = opinfo->op_state == OPLOCK_CLOSING;
+ if (!closing && !test_and_set_bit(0, &opinfo->pending_break)) {
+ spin_unlock(&opinfo->state_lock);
+ break;
+ }
+ spin_unlock(&opinfo->state_lock);
+ if (closing)
+ return -ENOENT;
+
wait_on_bit(&opinfo->pending_break, 0, TASK_UNINTERRUPTIBLE);
/* Not immediately break to none. */
opinfo->open_trunc = 0;
- if (opinfo->op_state == OPLOCK_CLOSING)
+ spin_lock(&opinfo->state_lock);
+ closing = opinfo->op_state == OPLOCK_CLOSING;
+ spin_unlock(&opinfo->state_lock);
+ if (closing)
return -ENOENT;
- else if (opinfo->level <= req_op_level) {
+ if (opinfo->level <= req_op_level) {
if (opinfo->is_lease == false)
return 1;
@@ -1005,7 +1050,11 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci,
release_async_work(in_work);
}
- brk_opinfo->op_state = OPLOCK_ACK_WAIT;
+ if (!oplock_break_set_ack_wait(brk_opinfo)) {
+ atomic_dec_if_positive(&brk_opinfo->breaking_cnt);
+ wake_up_oplock_break(brk_opinfo);
+ return -ENOENT;
+ }
} else
atomic_dec(&brk_opinfo->breaking_cnt);
} else {
@@ -1014,8 +1063,12 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci,
return err < 0 ? err : 0;
if (brk_opinfo->level == SMB2_OPLOCK_LEVEL_BATCH ||
- brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE)
- brk_opinfo->op_state = OPLOCK_ACK_WAIT;
+ brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) {
+ if (!oplock_break_set_ack_wait(brk_opinfo)) {
+ wake_up_oplock_break(brk_opinfo);
+ return -ENOENT;
+ }
+ }
}
if (brk_opinfo->is_lease)
diff --git a/fs/smb/server/oplock.h b/fs/smb/server/oplock.h
index 795a9119dad9..6e8680c06db9 100644
--- a/fs/smb/server/oplock.h
+++ b/fs/smb/server/oplock.h
@@ -59,6 +59,7 @@ struct oplock_info {
struct ksmbd_file *o_fp;
int level;
int op_state;
+ spinlock_t state_lock;
unsigned long pending_break;
u64 fid;
atomic_t breaking_cnt;
diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c
index 9cc6110fbcf9..e73edf006df7 100644
--- a/fs/smb/server/smb2pdu.c
+++ b/fs/smb/server/smb2pdu.c
@@ -2051,9 +2051,9 @@ int smb2_tree_connect(struct ksmbd_work *work)
if (conn->posix_ext_supported)
status.tree_conn->posix_extensions = true;
- write_lock(&sess->tree_conns_lock);
+ down_write(&sess->tree_conns_lock);
status.tree_conn->t_state = TREE_CONNECTED;
- write_unlock(&sess->tree_conns_lock);
+ up_write(&sess->tree_conns_lock);
rsp->StructureSize = cpu_to_le16(16);
out_err1:
/*
@@ -2066,8 +2066,16 @@ int smb2_tree_connect(struct ksmbd_work *work)
rsp->ShareFlags = SMB2_SHAREFLAG_MANUAL_CACHING;
rc = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_tree_connect_rsp));
- if (rc)
+ if (rc) {
+ if (status.ret == KSMBD_TREE_CONN_STATUS_OK) {
+ down_write(&sess->tree_conns_lock);
+ status.tree_conn->t_state = TREE_DISCONNECTED;
+ up_write(&sess->tree_conns_lock);
+ ksmbd_tree_conn_disconnect(sess, status.tree_conn);
+ status.tree_conn = NULL;
+ }
status.ret = KSMBD_TREE_CONN_STATUS_NOMEM;
+ }
if (!IS_ERR(treename))
kfree(treename);
@@ -2206,16 +2214,16 @@ int smb2_tree_disconnect(struct ksmbd_work *work)
ksmbd_close_tree_conn_fds(work);
- write_lock(&sess->tree_conns_lock);
+ down_write(&sess->tree_conns_lock);
if (tcon->t_state == TREE_DISCONNECTED) {
- write_unlock(&sess->tree_conns_lock);
+ up_write(&sess->tree_conns_lock);
rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED;
err = -ENOENT;
goto err_out;
}
tcon->t_state = TREE_DISCONNECTED;
- write_unlock(&sess->tree_conns_lock);
+ up_write(&sess->tree_conns_lock);
err = ksmbd_tree_conn_disconnect(sess, tcon);
if (err) {
@@ -2251,7 +2259,6 @@ int smb2_session_logoff(struct ksmbd_work *work)
struct ksmbd_session *sess = work->sess;
struct smb2_logoff_req *req;
struct smb2_logoff_rsp *rsp;
- u64 sess_id;
int err;
WORK_BUFFERS(work, req, rsp);
@@ -2265,8 +2272,7 @@ int smb2_session_logoff(struct ksmbd_work *work)
smb2_set_err_rsp(work);
return -ENOENT;
}
- sess_id = le64_to_cpu(req->hdr.SessionId);
- ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_RECONNECT);
+ ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_RECONNECT);
ksmbd_conn_unlock(conn);
ksmbd_close_session_fds(work);
@@ -2283,7 +2289,7 @@ int smb2_session_logoff(struct ksmbd_work *work)
sess->state = SMB2_SESSION_EXPIRED;
up_write(&conn->session_lock);
- ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_SETUP);
+ ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_SETUP);
rsp->StructureSize = cpu_to_le16(4);
err = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_logoff_rsp));
@@ -3261,6 +3267,7 @@ int smb2_open(struct ksmbd_work *work)
if (file_present && !(req->CreateOptions & FILE_DELETE_ON_CLOSE_LE)) {
rc = smb_check_perm_dacl(conn, &path, &daccess,
+ req->DesiredAccess,
sess->user->uid);
if (rc)
goto err_out;
@@ -3807,8 +3814,12 @@ int smb2_open(struct ksmbd_work *work)
rsp->hdr.Status = STATUS_INVALID_PARAMETER;
else if (rc == -EOPNOTSUPP)
rsp->hdr.Status = STATUS_NOT_SUPPORTED;
- else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV)
- rsp->hdr.Status = STATUS_ACCESS_DENIED;
+ else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV) {
+ if (req->DesiredAccess & FILE_ACCESS_SYSTEM_SECURITY_LE)
+ rsp->hdr.Status = STATUS_PRIVILEGE_NOT_HELD;
+ else
+ rsp->hdr.Status = STATUS_ACCESS_DENIED;
+ }
else if (rc == -ENOENT)
rsp->hdr.Status = STATUS_OBJECT_NAME_INVALID;
else if (rc == -EPERM)
@@ -7558,6 +7569,19 @@ static void smb2_remove_blocked_lock(void **argv)
locks_wake_up(flock);
}
+static void smb2_free_lock(struct file_lock *flock)
+{
+ ksmbd_vfs_posix_lock_unblock(flock);
+ locks_free_lock(flock);
+}
+
+static void smb2_free_blocked_lock(struct file_lock *flock)
+{
+ ksmbd_vfs_posix_lock_unblock(flock);
+ locks_wake_up(flock);
+ locks_free_lock(flock);
+}
+
static inline bool lock_defer_pending(struct file_lock *fl)
{
/* check pending lock waiters */
@@ -7711,14 +7735,14 @@ int smb2_lock(struct ksmbd_work *work)
cmp_lock->end == smb_lock->end &&
!lock_defer_pending(cmp_lock->fl)) {
nolock = 0;
- list_del(&cmp_lock->flist);
- list_del(&cmp_lock->clist);
+ list_del_init(&cmp_lock->flist);
+ list_del_init(&cmp_lock->clist);
cmp_lock->conn = NULL;
spin_unlock(&conn->llist_lock);
up_read(&conn_list_lock);
ksmbd_conn_put(conn);
- locks_free_lock(cmp_lock->fl);
+ smb2_free_lock(cmp_lock->fl);
kfree(cmp_lock);
goto out_check_cl;
}
@@ -7800,11 +7824,12 @@ int smb2_lock(struct ksmbd_work *work)
ksmbd_debug(SMB,
"would have to wait for getting lock\n");
- list_add(&smb_lock->llist, &rollback_list);
argv = kmalloc(sizeof(void *), KSMBD_DEFAULT_GFP);
if (!argv) {
err = -ENOMEM;
+ smb2_free_blocked_lock(flock);
+ kfree(smb_lock);
goto out;
}
argv[0] = flock;
@@ -7815,8 +7840,11 @@ int smb2_lock(struct ksmbd_work *work)
if (rc) {
kfree(argv);
err = -ENOMEM;
+ smb2_free_blocked_lock(flock);
+ kfree(smb_lock);
goto out;
}
+ list_add(&smb_lock->llist, &rollback_list);
spin_lock(&fp->f_lock);
list_add(&work->fp_entry, &fp->blocked_works);
spin_unlock(&fp->f_lock);
@@ -7849,16 +7877,10 @@ int smb2_lock(struct ksmbd_work *work)
rsp->hdr.Status =
STATUS_RANGE_NOT_LOCKED;
kfree(smb_lock);
- goto out2;
+ /* rollback_list may still hold earlier grants */
+ goto out;
} else if (!rc) {
list_add(&smb_lock->llist, &rollback_list);
- smb_lock->conn = ksmbd_conn_get(work->conn);
- spin_lock(&work->conn->llist_lock);
- list_add_tail(&smb_lock->clist,
- &work->conn->lock_list);
- list_add_tail(&smb_lock->flist,
- &fp->lock_list);
- spin_unlock(&work->conn->llist_lock);
ksmbd_debug(SMB, "successful in taking lock\n");
} else {
locks_free_lock(flock);
@@ -7880,6 +7902,19 @@ int smb2_lock(struct ksmbd_work *work)
if (err)
goto out;
+ /* publish only once the whole batch has committed */
+ if (!list_empty(&rollback_list)) {
+ spin_lock(&work->conn->llist_lock);
+ list_for_each_entry(smb_lock, &rollback_list, llist) {
+ smb_lock->conn = ksmbd_conn_get(work->conn);
+ list_add_tail(&smb_lock->clist,
+ &work->conn->lock_list);
+ list_add_tail(&smb_lock->flist,
+ &fp->lock_list);
+ }
+ spin_unlock(&work->conn->llist_lock);
+ }
+
ksmbd_fd_put(work, fp);
return 0;
@@ -7907,16 +7942,7 @@ int smb2_lock(struct ksmbd_work *work)
}
list_del(&smb_lock->llist);
- conn = smb_lock->conn;
- spin_lock(&conn->llist_lock);
- if (!list_empty(&smb_lock->flist))
- list_del(&smb_lock->flist);
- list_del(&smb_lock->clist);
- smb_lock->conn = NULL;
- spin_unlock(&conn->llist_lock);
- ksmbd_conn_put(conn);
-
- locks_free_lock(smb_lock->fl);
+ smb2_free_lock(smb_lock->fl);
if (rlock)
locks_free_lock(rlock);
kfree(smb_lock);
@@ -8855,7 +8881,10 @@ static void smb20_oplock_break_ack(struct ksmbd_work *work)
smb2_set_err_rsp(work);
out:
- opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING)
+ opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
wake_up_interruptible_all(&opinfo->oplock_q);
opinfo_put(opinfo);
ksmbd_fd_put(work, fp);
@@ -8941,9 +8970,12 @@ static void smb21_lease_break_ack(struct ksmbd_work *work)
if (ret)
goto err_out;
- opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING)
+ opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
wake_up_interruptible_all(&opinfo->oplock_q);
- atomic_dec(&opinfo->breaking_cnt);
+ atomic_dec_if_positive(&opinfo->breaking_cnt);
wake_up_interruptible_all(&opinfo->oplock_brk);
opinfo_put(opinfo);
return;
diff --git a/fs/smb/server/smbacl.c b/fs/smb/server/smbacl.c
index 2a46646b3a0e..a4447c9c660d 100644
--- a/fs/smb/server/smbacl.c
+++ b/fs/smb/server/smbacl.c
@@ -1445,7 +1445,7 @@ bool smb_inherit_flags(int flags, bool is_dir)
}
int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
- __le32 *pdaccess, int uid)
+ __le32 *pdaccess, __le32 raw_daccess, int uid)
{
struct mnt_idmap *idmap = mnt_idmap(path->mnt);
struct smb_ntsd *pntsd = NULL;
@@ -1455,10 +1455,11 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
unsigned int dacl_offset;
size_t dacl_struct_end;
struct smb_sid sid;
- int granted = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE);
+ int requested = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE);
+ int granted = requested;
struct smb_ace *ace;
int i, found = 0;
- unsigned int access_bits = 0;
+ unsigned int access_bits = 0, denied = 0;
struct smb_ace *others_ace = NULL;
struct posix_acl_entry *pa_entry;
unsigned int sid_type = SIDOWNER;
@@ -1492,10 +1493,13 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
goto err_out;
}
+ if (!uid)
+ sid_type = SIDUNIX_USER;
+ id_to_sid(uid, sid_type, &sid);
+
if (*pdaccess & FILE_MAXIMAL_ACCESS_LE) {
- granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
+ access_bits = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
DELETE;
-
ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl));
aces_size = pdacl_size - sizeof(struct smb_acl);
for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) {
@@ -1508,18 +1512,41 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
CIFS_SID_BASE_SIZE)
break;
aces_size -= ace_size;
- granted |= le32_to_cpu(ace->access_req);
+
+ if (ace->sid.num_subauth > SID_MAX_SUB_AUTHORITIES ||
+ ace_size < offsetof(struct smb_ace, sid) +
+ CIFS_SID_BASE_SIZE +
+ sizeof(__le32) * ace->sid.num_subauth)
+ break;
+
+ if (ace->flags & INHERIT_ONLY_ACE ||
+ (compare_sids(&sid, &ace->sid) &&
+ compare_sids(&sid_unix_NFS_mode, &ace->sid) &&
+ compare_sids(&sid_everyone, &ace->sid) &&
+ compare_sids(&sid_authusers, &ace->sid)))
+ goto next_ace;
+
+ switch (ace->type) {
+ case ACCESS_ALLOWED_ACE_TYPE:
+ access_bits |= le32_to_cpu(ace->access_req);
+ break;
+ case ACCESS_DENIED_ACE_TYPE:
+ case ACCESS_DENIED_CALLBACK_ACE_TYPE:
+ denied |= ~access_bits &
+ le32_to_cpu(ace->access_req);
+ break;
+ }
+next_ace:
ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size));
}
-
- if (!pdacl->num_aces)
- granted = GENERIC_ALL_FLAGS;
+ access_bits &= ~denied;
+ if ((raw_daccess & FILE_GENERIC_EXECUTE_LE) &&
+ S_ISREG(d_inode(path->dentry)->i_mode) &&
+ (access_bits & GENERIC_READ_FLAGS) == GENERIC_READ_FLAGS)
+ access_bits |= FILE_EXECUTE;
+ granted = requested | access_bits;
}
- if (!uid)
- sid_type = SIDUNIX_USER;
- id_to_sid(uid, sid_type, &sid);
-
ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl));
aces_size = pdacl_size - sizeof(struct smb_acl);
for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) {
@@ -1543,25 +1570,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
found = 1;
break;
}
- if (!compare_sids(&sid_everyone, &ace->sid))
+ if (!compare_sids(&sid_everyone, &ace->sid) ||
+ !compare_sids(&sid_authusers, &ace->sid))
others_ace = ace;
ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size));
}
- if (*pdaccess & FILE_MAXIMAL_ACCESS_LE && found) {
- granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
- DELETE;
-
- granted |= le32_to_cpu(ace->access_req);
-
- if (!pdacl->num_aces)
- granted = GENERIC_ALL_FLAGS;
- }
-
if (IS_ENABLED(CONFIG_FS_POSIX_ACL)) {
posix_acls = get_inode_acl(d_inode(path->dentry), ACL_TYPE_ACCESS);
- if (!IS_ERR_OR_NULL(posix_acls) && !found) {
+ if (!IS_ERR_OR_NULL(posix_acls) && !found && !others_ace) {
unsigned int id = -1;
pa_entry = posix_acls->a_entries;
@@ -1599,14 +1617,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
}
}
- switch (ace->type) {
- case ACCESS_ALLOWED_ACE_TYPE:
- access_bits = le32_to_cpu(ace->access_req);
- break;
- case ACCESS_DENIED_ACE_TYPE:
- case ACCESS_DENIED_CALLBACK_ACE_TYPE:
- access_bits = le32_to_cpu(~ace->access_req);
- break;
+ if (!(*pdaccess & FILE_MAXIMAL_ACCESS_LE)) {
+ switch (ace->type) {
+ case ACCESS_ALLOWED_ACE_TYPE:
+ access_bits = le32_to_cpu(ace->access_req);
+ break;
+ case ACCESS_DENIED_ACE_TYPE:
+ case ACCESS_DENIED_CALLBACK_ACE_TYPE:
+ access_bits = le32_to_cpu(~ace->access_req);
+ break;
+ }
}
check_access_bits:
diff --git a/fs/smb/server/smbacl.h b/fs/smb/server/smbacl.h
index ab21ba2cd4df..bf0312bfa040 100644
--- a/fs/smb/server/smbacl.h
+++ b/fs/smb/server/smbacl.h
@@ -95,7 +95,7 @@ bool smb_inherit_flags(int flags, bool is_dir);
int smb_inherit_dacl(struct ksmbd_conn *conn, const struct path *path,
unsigned int uid, unsigned int gid);
int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
- __le32 *pdaccess, int uid);
+ __le32 *pdaccess, __le32 raw_daccess, int uid);
int set_info_sec(struct ksmbd_conn *conn, struct ksmbd_tree_connect *tcon,
const struct path *path, struct smb_ntsd *pntsd, int ntsd_len,
bool type_check, bool get_write);
diff --git a/fs/smb/server/transport_tcp.c b/fs/smb/server/transport_tcp.c
index 78c909d66814..472d38709876 100644
--- a/fs/smb/server/transport_tcp.c
+++ b/fs/smb/server/transport_tcp.c
@@ -530,10 +530,10 @@ static int create_socket(struct interface *iface)
}
if (ipv4)
- ret = kernel_bind(ksmbd_socket, (struct sockaddr *)&sin,
+ ret = kernel_bind(ksmbd_socket, (struct sockaddr_unsized *)&sin,
sizeof(sin));
else
- ret = kernel_bind(ksmbd_socket, (struct sockaddr *)&sin6,
+ ret = kernel_bind(ksmbd_socket, (struct sockaddr_unsized *)&sin6,
sizeof(sin6));
if (ret) {
pr_err("Failed to bind socket: %d\n", ret);
diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c
index ec9d04e4de41..3ac5f3f48121 100644
--- a/fs/smb/server/vfs.c
+++ b/fs/smb/server/vfs.c
@@ -1238,7 +1238,7 @@ int __ksmbd_vfs_kern_path(struct ksmbd_work *work, char *filepath,
retry:
err = ksmbd_vfs_path_lookup(share_conf, filepath, flags, path, do_lock);
- if (!err || !caseless)
+ if (!err || err != -ENOENT || !caseless)
return err;
path_len = strlen(filepath);
diff --git a/fs/smb/server/vfs_cache.c b/fs/smb/server/vfs_cache.c
index ce6e9886df8b..970b8f185aac 100644
--- a/fs/smb/server/vfs_cache.c
+++ b/fs/smb/server/vfs_cache.c
@@ -388,8 +388,9 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp)
if (!IS_ERR_OR_NULL(filp))
fput(filp);
- /* because the reference count of fp is 0, it is guaranteed that
- * there are not accesses to fp->lock_list.
+ /*
+ * The zero fp reference count serializes access to fp->lock_list, but
+ * the VFS may still have blocked requests chained below these locks.
*/
list_for_each_entry_safe(smb_lock, tmp_lock, &fp->lock_list, flist) {
struct ksmbd_conn *conn = smb_lock->conn;
@@ -402,7 +403,8 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp)
ksmbd_conn_put(conn);
}
- list_del(&smb_lock->flist);
+ list_del_init(&smb_lock->flist);
+ ksmbd_vfs_posix_lock_unblock(smb_lock->fl);
locks_free_lock(smb_lock->fl);
kfree(smb_lock);
}
diff --git a/fs/tracefs/event_inode.c b/fs/tracefs/event_inode.c
index a7b082a665bd..abb0ea57601b 100644
--- a/fs/tracefs/event_inode.c
+++ b/fs/tracefs/event_inode.c
@@ -124,7 +124,17 @@ static inline void put_ei(struct eventfs_inode *ei)
static inline void free_ei(struct eventfs_inode *ei)
{
if (ei) {
+ /* The ei should have no children if it is being freed. */
+ WARN_ON_ONCE(!list_empty(&ei->children));
ei->is_freed = 1;
+ /*
+ * The SRCU iteration has a smp_rmb() to make sure it
+ * sees a child (that may have already been freed)
+ * before it reads is_free. If is_free is set, it must
+ * not use the child it acquired from ei->children, as
+ * the list may be used for SRCU.
+ */
+ smp_wmb();
put_ei(ei);
}
}
@@ -649,6 +659,20 @@ static int eventfs_iterate(struct file *file, struct dir_context *ctx)
list_for_each_entry_srcu(ei_child, &ei->children, list,
srcu_read_lock_held(&eventfs_srcu)) {
+ /*
+ * If the ei is being freed, then the ei->children may be
+ * being used as the rcu list, which means the next element
+ * may be garbage. The ei->is_free is set before switching
+ * the ei->children over to ei->rcu. The read memory barrier
+ * here makes sure the ei_child is read before is_free is
+ * updated.
+ *
+ * Matches the smp_wmb() in free_ei()
+ */
+ smp_rmb();
+ if (ei->is_freed)
+ return -EINVAL;
+
if (c > 0) {
c--;
continue;
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index a4a7f8431aff..c61481d04c8e 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -46,11 +46,11 @@ struct eventfs_attr {
* @ino: The saved inode number
*/
struct eventfs_inode {
+ struct list_head list;
union {
- struct list_head list;
+ struct list_head children;
struct rcu_head rcu;
};
- struct list_head children;
const struct eventfs_entry *entries;
const char *name;
struct eventfs_attr *entry_attrs;
diff --git a/fs/udf/super.c b/fs/udf/super.c
index ff170bc0962b..f91b231b7d7b 100644
--- a/fs/udf/super.c
+++ b/fs/udf/super.c
@@ -1704,8 +1704,9 @@ static struct udf_vds_record *handle_partition_descriptor(
return &(data->part_descs_loc[i].rec);
if (data->num_part_descs >= data->size_part_descs) {
struct part_desc_seq_scan_data *new_loc;
- unsigned int new_size = ALIGN(partnum, PART_DESC_ALLOC_STEP);
+ unsigned int new_size;
+ new_size = data->num_part_descs + PART_DESC_ALLOC_STEP;
new_loc = kcalloc(new_size, sizeof(*new_loc), GFP_KERNEL);
if (!new_loc)
return ERR_PTR(-ENOMEM);
@@ -1715,6 +1716,7 @@ static struct udf_vds_record *handle_partition_descriptor(
data->part_descs_loc = new_loc;
data->size_part_descs = new_size;
}
+ data->part_descs_loc[data->num_part_descs].partnum = partnum;
return &(data->part_descs_loc[data->num_part_descs++].rec);
}
diff --git a/fs/xfs/libxfs/xfs_exchmaps.c b/fs/xfs/libxfs/xfs_exchmaps.c
index 0cabbef65547..5f2cf280f9dd 100644
--- a/fs/xfs/libxfs/xfs_exchmaps.c
+++ b/fs/xfs/libxfs/xfs_exchmaps.c
@@ -49,7 +49,7 @@ struct xfs_exchmaps_adjacent {
.right2 = { .br_startblock = HOLESTARTBLOCK }, \
}
-/* Information to reset reflink flag / CoW fork state after an exchange. */
+/* Information to reset CoW fork state after an exchange. */
/*
* If the reflink flag is set on either inode, make sure it has an incore CoW
@@ -127,9 +127,7 @@ xmi_has_more_exchange_work(const struct xfs_exchmaps_intent *xmi)
static inline bool
xmi_has_postop_work(const struct xfs_exchmaps_intent *xmi)
{
- return xmi->xmi_flags & (XFS_EXCHMAPS_CLEAR_INO1_REFLINK |
- XFS_EXCHMAPS_CLEAR_INO2_REFLINK |
- __XFS_EXCHMAPS_INO2_SHORTFORM);
+ return xmi->xmi_flags & __XFS_EXCHMAPS_INO2_SHORTFORM;
}
/* Check all mappings to make sure we can actually exchange them. */
@@ -525,18 +523,6 @@ xfs_exchmaps_link_to_sf(
return error;
}
-/* Clear the reflink flag after an exchange. */
-static inline void
-xfs_exchmaps_clear_reflink(
- struct xfs_trans *tp,
- struct xfs_inode *ip)
-{
- trace_xfs_reflink_unset_inode_flag(ip);
-
- ip->i_diflags2 &= ~XFS_DIFLAG2_REFLINK;
- xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
-}
-
/* Finish whatever work might come after an exchange operation. */
static int
xfs_exchmaps_do_postop_work(
@@ -557,16 +543,6 @@ xfs_exchmaps_do_postop_work(
return error;
}
- if (xmi->xmi_flags & XFS_EXCHMAPS_CLEAR_INO1_REFLINK) {
- xfs_exchmaps_clear_reflink(tp, xmi->xmi_ip1);
- xmi->xmi_flags &= ~XFS_EXCHMAPS_CLEAR_INO1_REFLINK;
- }
-
- if (xmi->xmi_flags & XFS_EXCHMAPS_CLEAR_INO2_REFLINK) {
- xfs_exchmaps_clear_reflink(tp, xmi->xmi_ip2);
- xmi->xmi_flags &= ~XFS_EXCHMAPS_CLEAR_INO2_REFLINK;
- }
-
return 0;
}
@@ -948,56 +924,14 @@ xfs_exchmaps_intent_destroy_cache(void)
}
/*
- * Decide if we will exchange the reflink flags between the two files after the
- * exchange. The only time we want to do this is if we're exchanging all
- * mappings under EOF and the inode reflink flags have different states.
+ * Allocate and initialize a new incore intent item from a request.
*/
-static inline bool
-xmi_can_exchange_reflink_flags(
- const struct xfs_exchmaps_req *req,
- unsigned int reflink_state)
-{
- struct xfs_mount *mp = req->ip1->i_mount;
-
- /*
- * The INO1_WRITTEN optimization can skip exchanging hole and
- * unwritten mappings, which means we cannot guarantee that all
- * shared extents actually moved to the other file. Clearing the
- * reflink flag of an inode that still holds shared extents breaks
- * the CoW write path, so refuse to exchange the flags in that case.
- */
- if (req->flags & XFS_EXCHMAPS_INO1_WRITTEN)
- return false;
-
- /*
- * The INO1_WRITTEN optimization can skip exchanging hole and
- * unwritten mappings, which means we cannot guarantee that all
- * shared extents actually moved to the other file. Clearing the
- * reflink flag of an inode that still holds shared extents breaks
- * the CoW write path, so refuse to exchange the flags in that case.
- */
- if (req->flags & XFS_EXCHMAPS_INO1_WRITTEN)
- return false;
-
- if (hweight32(reflink_state) != 1)
- return false;
- if (req->startoff1 != 0 || req->startoff2 != 0)
- return false;
- if (req->blockcount != XFS_B_TO_FSB(mp, req->ip1->i_disk_size))
- return false;
- if (req->blockcount != XFS_B_TO_FSB(mp, req->ip2->i_disk_size))
- return false;
- return true;
-}
-
-
-/* Allocate and initialize a new incore intent item from a request. */
struct xfs_exchmaps_intent *
xfs_exchmaps_init_intent(
const struct xfs_exchmaps_req *req)
{
struct xfs_exchmaps_intent *xmi;
- unsigned int rs = 0;
+ struct xfs_mount *mp = req->ip1->i_mount;
xmi = kmem_cache_zalloc(xfs_exchmaps_intent_cache,
GFP_NOFS | __GFP_NOFAIL);
@@ -1015,27 +949,18 @@ xfs_exchmaps_init_intent(
return xmi;
}
- if (req->flags & XFS_EXCHMAPS_SET_SIZES) {
- xmi->xmi_flags |= XFS_EXCHMAPS_SET_SIZES;
- xmi->xmi_isize1 = req->ip2->i_disk_size;
- xmi->xmi_isize2 = req->ip1->i_disk_size;
- }
-
- /* Record the state of each inode's reflink flag before the op. */
- if (xfs_is_reflink_inode(req->ip1))
- rs |= 1;
- if (xfs_is_reflink_inode(req->ip2))
- rs |= 2;
-
/*
- * Figure out if we're clearing the reflink flags (which effectively
- * exchanges them) after the operation.
+ * If the caller wanted, set each file's size to that file's exchange
+ * offset + length exchanged from the other file because the ranges in
+ * each file might be different lengths.
*/
- if (xmi_can_exchange_reflink_flags(req, rs)) {
- if (rs & 1)
- xmi->xmi_flags |= XFS_EXCHMAPS_CLEAR_INO1_REFLINK;
- if (rs & 2)
- xmi->xmi_flags |= XFS_EXCHMAPS_CLEAR_INO2_REFLINK;
+ if (req->flags & XFS_EXCHMAPS_SET_SIZES) {
+ loff_t off1 = XFS_FSB_TO_B(mp, xmi->xmi_startoff1);
+ loff_t off2 = XFS_FSB_TO_B(mp, xmi->xmi_startoff2);
+
+ xmi->xmi_flags |= XFS_EXCHMAPS_SET_SIZES;
+ xmi->xmi_isize1 = off1 + (req->ip2->i_disk_size - off2);
+ xmi->xmi_isize2 = off2 + (req->ip1->i_disk_size - off1);
}
if (S_ISDIR(VFS_I(xmi->xmi_ip2)->i_mode) ||
diff --git a/fs/xfs/libxfs/xfs_log_format.h b/fs/xfs/libxfs/xfs_log_format.h
index 6c50cb2ece19..7cec81918b32 100644
--- a/fs/xfs/libxfs/xfs_log_format.h
+++ b/fs/xfs/libxfs/xfs_log_format.h
@@ -921,10 +921,8 @@ struct xfs_xmi_log_format {
*/
#define XFS_EXCHMAPS_INO1_WRITTEN (1ULL << 2)
-/* Clear the reflink flag from inode1 after the operation. */
+/* No longer set, but older intents may carry them; keep them valid. */
#define XFS_EXCHMAPS_CLEAR_INO1_REFLINK (1ULL << 3)
-
-/* Clear the reflink flag from inode2 after the operation. */
#define XFS_EXCHMAPS_CLEAR_INO2_REFLINK (1ULL << 4)
#define XFS_EXCHMAPS_LOGGED_FLAGS (XFS_EXCHMAPS_ATTR_FORK | \
diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c
index 06ca11731e43..225046fb2854 100644
--- a/fs/xfs/xfs_bmap_util.c
+++ b/fs/xfs/xfs_bmap_util.c
@@ -1514,7 +1514,6 @@ xfs_swap_extents(
struct xfs_bstat *sbp = &sxp->sx_stat;
int src_log_flags, target_log_flags;
int error = 0;
- uint64_t f;
int resblks = 0;
unsigned int flags = 0;
struct timespec64 ctime, mtime;
@@ -1670,14 +1669,12 @@ xfs_swap_extents(
if (error)
goto out_trans_cancel;
- /* Do we have to swap reflink flags? */
- if ((ip->i_diflags2 & XFS_DIFLAG2_REFLINK) ^
- (tip->i_diflags2 & XFS_DIFLAG2_REFLINK)) {
- f = ip->i_diflags2 & XFS_DIFLAG2_REFLINK;
- ip->i_diflags2 &= ~XFS_DIFLAG2_REFLINK;
- ip->i_diflags2 |= tip->i_diflags2 & XFS_DIFLAG2_REFLINK;
- tip->i_diflags2 &= ~XFS_DIFLAG2_REFLINK;
- tip->i_diflags2 |= f & XFS_DIFLAG2_REFLINK;
+ /* Either inode may now own shared blocks; flag both, with CoW forks. */
+ if ((ip->i_diflags2 | tip->i_diflags2) & XFS_DIFLAG2_REFLINK) {
+ ip->i_diflags2 |= XFS_DIFLAG2_REFLINK;
+ tip->i_diflags2 |= XFS_DIFLAG2_REFLINK;
+ xfs_ifork_init_cow(ip);
+ xfs_ifork_init_cow(tip);
}
/* Swap the cow forks. */
diff --git a/fs/xfs/xfs_exchrange.c b/fs/xfs/xfs_exchrange.c
index 04da020f3577..31a6847d0f86 100644
--- a/fs/xfs/xfs_exchrange.c
+++ b/fs/xfs/xfs_exchrange.c
@@ -290,17 +290,20 @@ xfs_exchrange_mappings(
goto out_unlock;
/*
- * If the caller wanted us to exchange the contents of two complete
- * files of unequal length, exchange the incore sizes now. This should
- * be safe because we flushed both files' page caches, exchanged all
- * the mappings, and updated the ondisk sizes.
+ * If the caller wanted, set each file's size to that file's exchange
+ * offset + length exchanged from the other file because the ranges in
+ * each file might be different lengths. This should be safe because
+ * we flushed both files' page caches, exchanged all the mappings, and
+ * updated the ondisk sizes.
*/
if (fxr->flags & XFS_EXCHANGE_RANGE_TO_EOF) {
- loff_t temp;
+ loff_t old_ip1_size = i_size_read(VFS_I(ip1));
+ loff_t old_ip2_size = i_size_read(VFS_I(ip2));
- temp = i_size_read(VFS_I(ip2));
- i_size_write(VFS_I(ip2), i_size_read(VFS_I(ip1)));
- i_size_write(VFS_I(ip1), temp);
+ i_size_write(VFS_I(ip2), fxr->file2_offset +
+ (old_ip1_size - fxr->file1_offset));
+ i_size_write(VFS_I(ip1), fxr->file1_offset +
+ (old_ip2_size - fxr->file2_offset));
}
out_unlock:
@@ -445,6 +448,30 @@ xfs_exchange_range_checks(
return blen == fxr->length ? 0 : -EINVAL;
}
+/* Flush all relevant dirty pagecache near the range to be exchanged. */
+static inline int
+xfs_exchange_range_flush(
+ struct xfs_inode *ip,
+ const struct xfs_exchrange *fxr,
+ loff_t offset)
+{
+ /*
+ * If TO_EOF is set, the ondisk file size update logic depends on the
+ * ondisk file size matching the incore file size. Flush everything.
+ */
+ if (fxr->flags & XFS_EXCHANGE_RANGE_TO_EOF)
+ return filemap_write_and_wait(VFS_I(ip)->i_mapping);
+
+ /*
+ * Because we're updating the ondisk mappings, flush all dirty data
+ * between the ondisk size and the exchange offset to reduce the chance
+ * of zeroed file contents in that gap after a crash.
+ */
+ return filemap_write_and_wait_range(VFS_I(ip)->i_mapping,
+ min(offset, ip->i_disk_size),
+ offset + fxr->length - 1);
+}
+
/*
* Check that the two inodes are eligible for range exchanges, the ranges make
* sense, and then flush all dirty data. Caller must ensure that the inodes
@@ -470,15 +497,11 @@ xfs_exchange_range_prep(
if (!same_inode)
inode_dio_wait(inode2);
- error = filemap_write_and_wait_range(inode1->i_mapping,
- fxr->file1_offset,
- fxr->file1_offset + fxr->length - 1);
+ error = xfs_exchange_range_flush(XFS_I(inode1), fxr, fxr->file1_offset);
if (error)
return error;
- error = filemap_write_and_wait_range(inode2->i_mapping,
- fxr->file2_offset,
- fxr->file2_offset + fxr->length - 1);
+ error = xfs_exchange_range_flush(XFS_I(inode2), fxr, fxr->file2_offset);
if (error)
return error;
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 9a8a796a870d..66fc14f869cb 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -677,6 +677,7 @@ void acpi_bus_unregister_driver(struct acpi_driver *driver);
int acpi_bus_scan(acpi_handle handle);
void acpi_bus_trim(struct acpi_device *start);
acpi_status acpi_bus_get_ejd(acpi_handle handle, acpi_handle * ejd);
+struct device *acpi_bus_get_primary_device(struct acpi_device *adev);
int acpi_match_device_ids(struct acpi_device *device,
const struct acpi_device_id *ids);
void acpi_set_modalias(struct acpi_device *adev, const char *default_id,
@@ -1000,6 +1001,11 @@ int acpi_scan_add_dep(acpi_handle handle, struct acpi_handle_list *dep_devices);
u32 arch_acpi_add_auto_dep(acpi_handle handle);
#else /* CONFIG_ACPI */
+static inline struct device *acpi_bus_get_primary_device(struct acpi_device *adev)
+{
+ return NULL;
+}
+
static inline int register_acpi_bus_type(void *bus) { return 0; }
static inline int unregister_acpi_bus_type(void *bus) { return 0; }
diff --git a/include/clocksource/samsung_pwm.h b/include/clocksource/samsung_pwm.h
index 9b435caa95fe..36f6f246e559 100644
--- a/include/clocksource/samsung_pwm.h
+++ b/include/clocksource/samsung_pwm.h
@@ -15,7 +15,7 @@
* spinlock is not shared between both drivers.
*/
#ifdef CONFIG_CLKSRC_SAMSUNG_PWM
-extern spinlock_t samsung_pwm_lock;
+extern raw_spinlock_t samsung_pwm_lock;
#endif
struct samsung_pwm_variant {
diff --git a/include/cxl/mailbox.h b/include/cxl/mailbox.h
index c4e99e2e3a9d..d008b9db07aa 100644
--- a/include/cxl/mailbox.h
+++ b/include/cxl/mailbox.h
@@ -50,6 +50,7 @@ struct cxl_mbox_cmd {
* @payload_size: Size of space for payload
* (CXL 3.1 8.2.8.4.3 Mailbox Capabilities Register)
* @mbox_mutex: mutex protects device mailbox and firmware
+ * @feat_mutex: serializes multi-part Get/Set Feature transfers
* @mbox_wait: rcuwait for mailbox
* @mbox_send: @dev specific transport for transmitting mailbox commands
* @feat_cap: Features capability
@@ -60,6 +61,7 @@ struct cxl_mailbox {
DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
size_t payload_size;
struct mutex mbox_mutex; /* lock to protect mailbox context */
+ struct mutex feat_mutex;
struct rcuwait mbox_wait;
int (*mbox_send)(struct cxl_mailbox *cxl_mbox, struct cxl_mbox_cmd *cmd);
enum cxl_features_capability feat_cap;
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index f49daa504c36..a4e3f0447d38 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -434,6 +434,8 @@ void ttm_resource_add_bulk_move(struct ttm_resource *res,
struct ttm_buffer_object *bo);
void ttm_resource_del_bulk_move(struct ttm_resource *res,
struct ttm_buffer_object *bo);
+void ttm_resource_del_bulk_move_unevictable(struct ttm_resource *res,
+ struct ttm_buffer_object *bo);
void ttm_resource_move_to_lru_tail(struct ttm_resource *res);
void ttm_resource_init(struct ttm_buffer_object *bo,
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 7a0b972eb1b1..25eb6ba9dd23 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -140,9 +140,6 @@ struct vgic_irq {
* the pending state for both level
* and edge triggered IRQs. */
bool active;
- bool pending_release; /* Used for LPIs only, unreferenced IRQ
- * pending a release */
-
bool enabled;
bool hw; /* Tied to HW IRQ */
refcount_t refcount; /* Used for LPIs */
diff --git a/include/linux/bnxt/ulp.h b/include/linux/bnxt/ulp.h
index 7b9dd8ebe4bc..c88571f12a14 100644
--- a/include/linux/bnxt/ulp.h
+++ b/include/linux/bnxt/ulp.h
@@ -10,6 +10,8 @@
#ifndef BNXT_ULP_H
#define BNXT_ULP_H
+#include <linux/auxiliary_bus.h>
+
#define BNXT_MIN_ROCE_CP_RINGS 2
#define BNXT_MIN_ROCE_STAT_CTXS 1
@@ -20,6 +22,17 @@
struct hwrm_async_event_cmpl;
struct bnxt;
+enum bnxt_auxdev_type {
+ BNXT_AUXDEV_RDMA = 0,
+ __BNXT_AUXDEV_MAX
+};
+
+struct bnxt_aux_priv {
+ struct auxiliary_device aux_dev;
+ struct bnxt_en_dev *edev;
+ int id;
+};
+
struct bnxt_msix_entry {
u32 vector;
u32 ring_idx;
@@ -111,19 +124,21 @@ void bnxt_set_ulp_stat_ctxs(struct bnxt *bp, int num_ctxs);
int bnxt_get_ulp_stat_ctxs_in_use(struct bnxt *bp);
void bnxt_set_dflt_ulp_stat_ctxs(struct bnxt *bp);
void bnxt_ulp_stop(struct bnxt *bp);
-void bnxt_ulp_start(struct bnxt *bp, int err);
+void bnxt_ulp_start(struct bnxt *bp);
void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs);
void bnxt_ulp_irq_stop(struct bnxt *bp);
void bnxt_ulp_irq_restart(struct bnxt *bp, int err);
void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl);
-void bnxt_rdma_aux_device_uninit(struct bnxt *bp);
-void bnxt_rdma_aux_device_del(struct bnxt *bp);
-void bnxt_rdma_aux_device_add(struct bnxt *bp);
-void bnxt_rdma_aux_device_init(struct bnxt *bp);
+void bnxt_aux_devices_uninit(struct bnxt *bp);
+void bnxt_aux_devices_del(struct bnxt *bp);
+void bnxt_aux_devices_add(struct bnxt *bp);
+void bnxt_aux_devices_init(struct bnxt *bp);
int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops,
void *handle);
void bnxt_unregister_dev(struct bnxt_en_dev *edev);
int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg);
void bnxt_register_async_events(struct bnxt_en_dev *edev,
unsigned long *events_bmap, u16 max_id);
+int bnxt_auxdev_id_alloc(struct bnxt *bp);
+void bnxt_auxdev_id_free(struct bnxt *bp, int id);
#endif
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 862d6e0d5a28..d018a74a882d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1399,6 +1399,8 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
+void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags);
+
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
* indirection with a direct call to the bpf program. If the architecture does
@@ -1507,6 +1509,7 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
{
return false;
}
+static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {}
#endif
struct bpf_func_info_aux {
@@ -2827,7 +2830,7 @@ int btf_struct_access(struct bpf_verifier_log *log,
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
- bool strict);
+ bool strict, bool walk_flex_arrays);
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d305d30ebe95..56a6c5b8625d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -580,6 +580,8 @@ struct bpf_insn_aux_data {
* accepts callback function as a parameter.
*/
bool calls_callback;
+ /* true if some jump or call instruction targets this instruction */
+ bool jump_target;
/*
* CFG strongly connected component this instruction belongs to,
* zero if it is a singleton SCC.
diff --git a/include/linux/can/skb.h b/include/linux/can/skb.h
index 869ea574a40a..27d0ca115c6c 100644
--- a/include/linux/can/skb.h
+++ b/include/linux/can/skb.h
@@ -49,11 +49,13 @@ bool can_dropped_invalid_skb(struct net_device *dev, struct sk_buff *skb);
/**
* struct can_skb_priv - private additional data inside CAN sk_buffs
* @ifindex: ifindex of the first interface the CAN frame appeared on
+ * @skbcnt: atomic counter to have an unique id together with skb pointer
* @frame_len: length of CAN frame in data link layer
* @cf: align to the following CAN frame at skb->data
*/
struct can_skb_priv {
int ifindex;
+ u32 skbcnt;
unsigned int frame_len;
struct can_frame cf[];
};
diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index ba49413f704b..cb007601d8c3 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -11,6 +11,10 @@
#define __has_builtin(x) (0)
#endif
+/* Indirect macros required for expanded argument pasting, eg. __LINE__. */
+#define ___PASTE(a, b) a##b
+#define __PASTE(a, b) ___PASTE(a, b)
+
#ifndef __ASSEMBLY__
/*
@@ -78,10 +82,6 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
# define __builtin_warning(x, y...) (1)
#endif /* __CHECKER__ */
-/* Indirect macros required for expanded argument pasting, eg. __LINE__. */
-#define ___PASTE(a,b) a##b
-#define __PASTE(a,b) ___PASTE(a,b)
-
#ifdef __KERNEL__
/* Attributes */
diff --git a/include/linux/fb.h b/include/linux/fb.h
index da2fdabd18cb..f622f5f520cf 100644
--- a/include/linux/fb.h
+++ b/include/linux/fb.h
@@ -541,6 +541,8 @@ extern int fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var);
extern int fb_pan_display(struct fb_info *info, struct fb_var_screeninfo *var);
extern int fb_blank(struct fb_info *info, int blank);
+int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var);
+
/*
* Helpers for framebuffers in I/O memory
*/
diff --git a/include/linux/firmware/xlnx-zynqmp.h b/include/linux/firmware/xlnx-zynqmp.h
index 2f1c71f0cea7..2f96f91be6c2 100644
--- a/include/linux/firmware/xlnx-zynqmp.h
+++ b/include/linux/firmware/xlnx-zynqmp.h
@@ -51,16 +51,10 @@
#define PM_PINCTRL_PARAM_SET_VERSION 2
-#define ZYNQMP_FAMILY_CODE 0x23
-#define VERSAL_FAMILY_CODE 0x26
-
-/* When all subfamily of platform need to support */
-#define ALL_SUB_FAMILY_CODE 0x00
-#define VERSAL_SUB_FAMILY_CODE 0x01
-#define VERSALNET_SUB_FAMILY_CODE 0x03
-
-#define FAMILY_CODE_MASK GENMASK(27, 21)
-#define SUB_FAMILY_CODE_MASK GENMASK(20, 19)
+/* Family codes */
+#define PM_ZYNQMP_FAMILY_CODE 0x1 /* ZynqMP family code */
+#define PM_VERSAL_FAMILY_CODE 0x2 /* Versal family code */
+#define PM_VERSAL_NET_FAMILY_CODE 0x3 /* Versal NET family code */
#define API_ID_MASK GENMASK(7, 0)
#define MODULE_ID_MASK GENMASK(11, 8)
@@ -577,7 +571,7 @@ int zynqmp_pm_invoke_fw_fn(u32 pm_api_id, u32 *ret_payload, u32 num_args, ...);
#if IS_REACHABLE(CONFIG_ZYNQMP_FIRMWARE)
int zynqmp_pm_get_api_version(u32 *version);
int zynqmp_pm_get_chipid(u32 *idcode, u32 *version);
-int zynqmp_pm_get_family_info(u32 *family, u32 *subfamily);
+int zynqmp_pm_get_family_info(u32 *family);
int zynqmp_pm_query_data(struct zynqmp_pm_query_data qdata, u32 *out);
int zynqmp_pm_clock_enable(u32 clock_id);
int zynqmp_pm_clock_disable(u32 clock_id);
@@ -656,7 +650,7 @@ static inline int zynqmp_pm_get_chipid(u32 *idcode, u32 *version)
return -ENODEV;
}
-static inline int zynqmp_pm_get_family_info(u32 *family, u32 *subfamily)
+static inline int zynqmp_pm_get_family_info(u32 *family)
{
return -ENODEV;
}
diff --git a/include/linux/ieee80211-s1g.h b/include/linux/ieee80211-s1g.h
new file mode 100644
index 000000000000..afc01379801f
--- /dev/null
+++ b/include/linux/ieee80211-s1g.h
@@ -0,0 +1,575 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * IEEE 802.11 S1G definitions
+ *
+ * Copyright (c) 2001-2002, SSH Communications Security Corp and Jouni Malinen
+ * <jkmaline@cc.hut.fi>
+ * Copyright (c) 2002-2003, Jouni Malinen <jkmaline@cc.hut.fi>
+ * Copyright (c) 2005, Devicescape Software, Inc.
+ * Copyright (c) 2006, Michael Wu <flamingice@sourmilk.net>
+ * Copyright (c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright (c) 2016 - 2017 Intel Deutschland GmbH
+ * Copyright (c) 2018 - 2025 Intel Corporation
+ */
+
+#ifndef LINUX_IEEE80211_S1G_H
+#define LINUX_IEEE80211_S1G_H
+
+#include <linux/types.h>
+#include <linux/if_ether.h>
+
+/* bits unique to S1G beacon frame control */
+#define IEEE80211_S1G_BCN_NEXT_TBTT 0x100
+#define IEEE80211_S1G_BCN_CSSID 0x200
+#define IEEE80211_S1G_BCN_ANO 0x400
+
+/* see 802.11ah-2016 9.9 NDP CMAC frames */
+#define IEEE80211_S1G_1MHZ_NDP_BITS 25
+#define IEEE80211_S1G_1MHZ_NDP_BYTES 4
+#define IEEE80211_S1G_2MHZ_NDP_BITS 37
+#define IEEE80211_S1G_2MHZ_NDP_BYTES 5
+
+/**
+ * ieee80211_is_s1g_beacon - check if IEEE80211_FTYPE_EXT &&
+ * IEEE80211_STYPE_S1G_BEACON
+ * @fc: frame control bytes in little-endian byteorder
+ * Return: whether or not the frame is an S1G beacon
+ */
+static inline bool ieee80211_is_s1g_beacon(__le16 fc)
+{
+ return (fc & cpu_to_le16(IEEE80211_FCTL_FTYPE |
+ IEEE80211_FCTL_STYPE)) ==
+ cpu_to_le16(IEEE80211_FTYPE_EXT | IEEE80211_STYPE_S1G_BEACON);
+}
+
+/**
+ * ieee80211_s1g_has_next_tbtt - check if IEEE80211_S1G_BCN_NEXT_TBTT
+ * @fc: frame control bytes in little-endian byteorder
+ * Return: whether or not the frame contains the variable-length
+ * next TBTT field
+ */
+static inline bool ieee80211_s1g_has_next_tbtt(__le16 fc)
+{
+ return ieee80211_is_s1g_beacon(fc) &&
+ (fc & cpu_to_le16(IEEE80211_S1G_BCN_NEXT_TBTT));
+}
+
+/**
+ * ieee80211_s1g_has_ano - check if IEEE80211_S1G_BCN_ANO
+ * @fc: frame control bytes in little-endian byteorder
+ * Return: whether or not the frame contains the variable-length
+ * ANO field
+ */
+static inline bool ieee80211_s1g_has_ano(__le16 fc)
+{
+ return ieee80211_is_s1g_beacon(fc) &&
+ (fc & cpu_to_le16(IEEE80211_S1G_BCN_ANO));
+}
+
+/**
+ * ieee80211_s1g_has_cssid - check if IEEE80211_S1G_BCN_CSSID
+ * @fc: frame control bytes in little-endian byteorder
+ * Return: whether or not the frame contains the variable-length
+ * compressed SSID field
+ */
+static inline bool ieee80211_s1g_has_cssid(__le16 fc)
+{
+ return ieee80211_is_s1g_beacon(fc) &&
+ (fc & cpu_to_le16(IEEE80211_S1G_BCN_CSSID));
+}
+
+/**
+ * enum ieee80211_s1g_chanwidth - S1G channel widths
+ * These are defined in IEEE802.11-2016ah Table 10-20
+ * as BSS Channel Width
+ *
+ * @IEEE80211_S1G_CHANWIDTH_1MHZ: 1MHz operating channel
+ * @IEEE80211_S1G_CHANWIDTH_2MHZ: 2MHz operating channel
+ * @IEEE80211_S1G_CHANWIDTH_4MHZ: 4MHz operating channel
+ * @IEEE80211_S1G_CHANWIDTH_8MHZ: 8MHz operating channel
+ * @IEEE80211_S1G_CHANWIDTH_16MHZ: 16MHz operating channel
+ */
+enum ieee80211_s1g_chanwidth {
+ IEEE80211_S1G_CHANWIDTH_1MHZ = 0,
+ IEEE80211_S1G_CHANWIDTH_2MHZ = 1,
+ IEEE80211_S1G_CHANWIDTH_4MHZ = 3,
+ IEEE80211_S1G_CHANWIDTH_8MHZ = 7,
+ IEEE80211_S1G_CHANWIDTH_16MHZ = 15,
+};
+
+/**
+ * enum ieee80211_s1g_pri_chanwidth - S1G primary channel widths
+ * described in IEEE80211-2024 Table 10-39.
+ *
+ * @IEEE80211_S1G_PRI_CHANWIDTH_2MHZ: 2MHz primary channel
+ * @IEEE80211_S1G_PRI_CHANWIDTH_1MHZ: 1MHz primary channel
+ */
+enum ieee80211_s1g_pri_chanwidth {
+ IEEE80211_S1G_PRI_CHANWIDTH_2MHZ = 0,
+ IEEE80211_S1G_PRI_CHANWIDTH_1MHZ = 1,
+};
+
+/**
+ * struct ieee80211_s1g_bcn_compat_ie - S1G Beacon Compatibility element
+ * @compat_info: Compatibility Information
+ * @beacon_int: Beacon Interval
+ * @tsf_completion: TSF Completion
+ *
+ * This structure represents the payload of the "S1G Beacon
+ * Compatibility element" as described in IEEE Std 802.11-2020 section
+ * 9.4.2.196.
+ */
+struct ieee80211_s1g_bcn_compat_ie {
+ __le16 compat_info;
+ __le16 beacon_int;
+ __le32 tsf_completion;
+} __packed;
+
+/**
+ * struct ieee80211_s1g_oper_ie - S1G Operation element
+ * @ch_width: S1G Operation Information Channel Width
+ * @oper_class: S1G Operation Information Operating Class
+ * @primary_ch: S1G Operation Information Primary Channel Number
+ * @oper_ch: S1G Operation Information Channel Center Frequency
+ * @basic_mcs_nss: Basic S1G-MCS and NSS Set
+ *
+ * This structure represents the payload of the "S1G Operation
+ * element" as described in IEEE Std 802.11-2020 section 9.4.2.212.
+ */
+struct ieee80211_s1g_oper_ie {
+ u8 ch_width;
+ u8 oper_class;
+ u8 primary_ch;
+ u8 oper_ch;
+ __le16 basic_mcs_nss;
+} __packed;
+
+/**
+ * struct ieee80211_aid_response_ie - AID Response element
+ * @aid: AID/Group AID
+ * @switch_count: AID Switch Count
+ * @response_int: AID Response Interval
+ *
+ * This structure represents the payload of the "AID Response element"
+ * as described in IEEE Std 802.11-2020 section 9.4.2.194.
+ */
+struct ieee80211_aid_response_ie {
+ __le16 aid;
+ u8 switch_count;
+ __le16 response_int;
+} __packed;
+
+struct ieee80211_s1g_cap {
+ u8 capab_info[10];
+ u8 supp_mcs_nss[5];
+} __packed;
+
+/**
+ * ieee80211_s1g_optional_len - determine length of optional S1G beacon fields
+ * @fc: frame control bytes in little-endian byteorder
+ * Return: total length in bytes of the optional fixed-length fields
+ *
+ * S1G beacons may contain up to three optional fixed-length fields that
+ * precede the variable-length elements. Whether these fields are present
+ * is indicated by flags in the frame control field.
+ *
+ * From IEEE 802.11-2024 section 9.3.4.3:
+ * - Next TBTT field may be 0 or 3 bytes
+ * - Short SSID field may be 0 or 4 bytes
+ * - Access Network Options (ANO) field may be 0 or 1 byte
+ */
+static inline size_t
+ieee80211_s1g_optional_len(__le16 fc)
+{
+ size_t len = 0;
+
+ if (ieee80211_s1g_has_next_tbtt(fc))
+ len += 3;
+
+ if (ieee80211_s1g_has_cssid(fc))
+ len += 4;
+
+ if (ieee80211_s1g_has_ano(fc))
+ len += 1;
+
+ return len;
+}
+
+/* S1G Capabilities Information field */
+#define IEEE80211_S1G_CAPABILITY_LEN 15
+
+#define S1G_CAP0_S1G_LONG BIT(0)
+#define S1G_CAP0_SGI_1MHZ BIT(1)
+#define S1G_CAP0_SGI_2MHZ BIT(2)
+#define S1G_CAP0_SGI_4MHZ BIT(3)
+#define S1G_CAP0_SGI_8MHZ BIT(4)
+#define S1G_CAP0_SGI_16MHZ BIT(5)
+#define S1G_CAP0_SUPP_CH_WIDTH GENMASK(7, 6)
+
+#define S1G_SUPP_CH_WIDTH_2 0
+#define S1G_SUPP_CH_WIDTH_4 1
+#define S1G_SUPP_CH_WIDTH_8 2
+#define S1G_SUPP_CH_WIDTH_16 3
+#define S1G_SUPP_CH_WIDTH_MAX(cap) ((1 << FIELD_GET(S1G_CAP0_SUPP_CH_WIDTH, \
+ cap[0])) << 1)
+
+#define S1G_CAP1_RX_LDPC BIT(0)
+#define S1G_CAP1_TX_STBC BIT(1)
+#define S1G_CAP1_RX_STBC BIT(2)
+#define S1G_CAP1_SU_BFER BIT(3)
+#define S1G_CAP1_SU_BFEE BIT(4)
+#define S1G_CAP1_BFEE_STS GENMASK(7, 5)
+
+#define S1G_CAP2_SOUNDING_DIMENSIONS GENMASK(2, 0)
+#define S1G_CAP2_MU_BFER BIT(3)
+#define S1G_CAP2_MU_BFEE BIT(4)
+#define S1G_CAP2_PLUS_HTC_VHT BIT(5)
+#define S1G_CAP2_TRAVELING_PILOT GENMASK(7, 6)
+
+#define S1G_CAP3_RD_RESPONDER BIT(0)
+#define S1G_CAP3_HT_DELAYED_BA BIT(1)
+#define S1G_CAP3_MAX_MPDU_LEN BIT(2)
+#define S1G_CAP3_MAX_AMPDU_LEN_EXP GENMASK(4, 3)
+#define S1G_CAP3_MIN_MPDU_START GENMASK(7, 5)
+
+#define S1G_CAP4_UPLINK_SYNC BIT(0)
+#define S1G_CAP4_DYNAMIC_AID BIT(1)
+#define S1G_CAP4_BAT BIT(2)
+#define S1G_CAP4_TIME_ADE BIT(3)
+#define S1G_CAP4_NON_TIM BIT(4)
+#define S1G_CAP4_GROUP_AID BIT(5)
+#define S1G_CAP4_STA_TYPE GENMASK(7, 6)
+
+#define S1G_CAP5_CENT_AUTH_CONTROL BIT(0)
+#define S1G_CAP5_DIST_AUTH_CONTROL BIT(1)
+#define S1G_CAP5_AMSDU BIT(2)
+#define S1G_CAP5_AMPDU BIT(3)
+#define S1G_CAP5_ASYMMETRIC_BA BIT(4)
+#define S1G_CAP5_FLOW_CONTROL BIT(5)
+#define S1G_CAP5_SECTORIZED_BEAM GENMASK(7, 6)
+
+#define S1G_CAP6_OBSS_MITIGATION BIT(0)
+#define S1G_CAP6_FRAGMENT_BA BIT(1)
+#define S1G_CAP6_NDP_PS_POLL BIT(2)
+#define S1G_CAP6_RAW_OPERATION BIT(3)
+#define S1G_CAP6_PAGE_SLICING BIT(4)
+#define S1G_CAP6_TXOP_SHARING_IMP_ACK BIT(5)
+#define S1G_CAP6_VHT_LINK_ADAPT GENMASK(7, 6)
+
+#define S1G_CAP7_TACK_AS_PS_POLL BIT(0)
+#define S1G_CAP7_DUP_1MHZ BIT(1)
+#define S1G_CAP7_MCS_NEGOTIATION BIT(2)
+#define S1G_CAP7_1MHZ_CTL_RESPONSE_PREAMBLE BIT(3)
+#define S1G_CAP7_NDP_BFING_REPORT_POLL BIT(4)
+#define S1G_CAP7_UNSOLICITED_DYN_AID BIT(5)
+#define S1G_CAP7_SECTOR_TRAINING_OPERATION BIT(6)
+#define S1G_CAP7_TEMP_PS_MODE_SWITCH BIT(7)
+
+#define S1G_CAP8_TWT_GROUPING BIT(0)
+#define S1G_CAP8_BDT BIT(1)
+#define S1G_CAP8_COLOR GENMASK(4, 2)
+#define S1G_CAP8_TWT_REQUEST BIT(5)
+#define S1G_CAP8_TWT_RESPOND BIT(6)
+#define S1G_CAP8_PV1_FRAME BIT(7)
+
+#define S1G_CAP9_LINK_ADAPT_PER_CONTROL_RESPONSE BIT(0)
+
+#define S1G_OPER_CH_WIDTH_PRIMARY BIT(0)
+#define S1G_OPER_CH_WIDTH_OPER GENMASK(4, 1)
+#define S1G_OPER_CH_PRIMARY_LOCATION BIT(5)
+
+#define S1G_2M_PRIMARY_LOCATION_LOWER 0
+#define S1G_2M_PRIMARY_LOCATION_UPPER 1
+
+#define LISTEN_INT_USF GENMASK(15, 14)
+#define LISTEN_INT_UI GENMASK(13, 0)
+
+#define IEEE80211_MAX_USF FIELD_MAX(LISTEN_INT_USF)
+#define IEEE80211_MAX_UI FIELD_MAX(LISTEN_INT_UI)
+
+/* S1G encoding types */
+#define IEEE80211_S1G_TIM_ENC_MODE_BLOCK 0
+#define IEEE80211_S1G_TIM_ENC_MODE_SINGLE 1
+#define IEEE80211_S1G_TIM_ENC_MODE_OLB 2
+
+enum ieee80211_s1g_actioncode {
+ WLAN_S1G_AID_SWITCH_REQUEST,
+ WLAN_S1G_AID_SWITCH_RESPONSE,
+ WLAN_S1G_SYNC_CONTROL,
+ WLAN_S1G_STA_INFO_ANNOUNCE,
+ WLAN_S1G_EDCA_PARAM_SET,
+ WLAN_S1G_EL_OPERATION,
+ WLAN_S1G_TWT_SETUP,
+ WLAN_S1G_TWT_TEARDOWN,
+ WLAN_S1G_SECT_GROUP_ID_LIST,
+ WLAN_S1G_SECT_ID_FEEDBACK,
+ WLAN_S1G_TWT_INFORMATION = 11,
+};
+
+/**
+ * ieee80211_is_s1g_short_beacon - check if frame is an S1G short beacon
+ * @fc: frame control bytes in little-endian byteorder
+ * @variable: pointer to the beacon frame elements
+ * @variable_len: length of the frame elements
+ * Return: whether or not the frame is an S1G short beacon. As per
+ * IEEE80211-2024 11.1.3.10.1, The S1G beacon compatibility element shall
+ * always be present as the first element in beacon frames generated at a
+ * TBTT (Target Beacon Transmission Time), so any frame not containing
+ * this element must have been generated at a TSBTT (Target Short Beacon
+ * Transmission Time) that is not a TBTT. Additionally, short beacons are
+ * prohibited from containing the S1G beacon compatibility element as per
+ * IEEE80211-2024 9.3.4.3 Table 9-76, so if we have an S1G beacon with
+ * either no elements or the first element is not the beacon compatibility
+ * element, we have a short beacon.
+ */
+static inline bool ieee80211_is_s1g_short_beacon(__le16 fc, const u8 *variable,
+ size_t variable_len)
+{
+ if (!ieee80211_is_s1g_beacon(fc))
+ return false;
+
+ /*
+ * If the frame does not contain at least 1 element (this is perfectly
+ * valid in a short beacon) and is an S1G beacon, we have a short
+ * beacon.
+ */
+ if (variable_len < 2)
+ return true;
+
+ return variable[0] != WLAN_EID_S1G_BCN_COMPAT;
+}
+
+struct s1g_tim_aid {
+ u16 aid;
+ u8 target_blk; /* Target block index */
+ u8 target_subblk; /* Target subblock index */
+ u8 target_subblk_bit; /* Target subblock bit */
+};
+
+struct s1g_tim_enc_block {
+ u8 enc_mode;
+ bool inverse;
+ const u8 *ptr;
+ u8 len;
+
+ /*
+ * For an OLB encoded block that spans multiple blocks, this
+ * is the offset into the span described by that encoded block.
+ */
+ u8 olb_blk_offset;
+};
+
+/*
+ * Helper routines to quickly extract the length of an encoded block. Validation
+ * is also performed to ensure the length extracted lies within the TIM.
+ */
+
+static inline int ieee80211_s1g_len_bitmap(const u8 *ptr, const u8 *end)
+{
+ u8 blkmap;
+ u8 n_subblks;
+
+ if (ptr >= end)
+ return -EINVAL;
+
+ blkmap = *ptr;
+ n_subblks = hweight8(blkmap);
+
+ if (ptr + 1 + n_subblks > end)
+ return -EINVAL;
+
+ return 1 + n_subblks;
+}
+
+static inline int ieee80211_s1g_len_single(const u8 *ptr, const u8 *end)
+{
+ return (ptr + 1 > end) ? -EINVAL : 1;
+}
+
+static inline int ieee80211_s1g_len_olb(const u8 *ptr, const u8 *end)
+{
+ if (ptr >= end)
+ return -EINVAL;
+
+ return (ptr + 1 + *ptr > end) ? -EINVAL : 1 + *ptr;
+}
+
+/*
+ * Enumerate all encoded blocks until we find the encoded block that describes
+ * our target AID. OLB is a special case as a single encoded block can describe
+ * multiple blocks as a single encoded block.
+ */
+static inline int ieee80211_s1g_find_target_block(struct s1g_tim_enc_block *enc,
+ const struct s1g_tim_aid *aid,
+ const u8 *ptr, const u8 *end)
+{
+ /* need at least block-control octet */
+ while (ptr + 1 <= end) {
+ u8 ctrl = *ptr++;
+ u8 mode = ctrl & 0x03;
+ bool contains, inverse = ctrl & BIT(2);
+ u8 span, blk_off = ctrl >> 3;
+ int len;
+
+ switch (mode) {
+ case IEEE80211_S1G_TIM_ENC_MODE_BLOCK:
+ len = ieee80211_s1g_len_bitmap(ptr, end);
+ contains = blk_off == aid->target_blk;
+ break;
+ case IEEE80211_S1G_TIM_ENC_MODE_SINGLE:
+ len = ieee80211_s1g_len_single(ptr, end);
+ contains = blk_off == aid->target_blk;
+ break;
+ case IEEE80211_S1G_TIM_ENC_MODE_OLB:
+ len = ieee80211_s1g_len_olb(ptr, end);
+ /*
+ * An OLB encoded block can describe more then one
+ * block, meaning an encoded OLB block can span more
+ * then a single block.
+ */
+ if (len > 0) {
+ /* Minus one for the length octet */
+ span = DIV_ROUND_UP(len - 1, 8);
+ /*
+ * Check if our target block lies within the
+ * block span described by this encoded block.
+ */
+ contains = (aid->target_blk >= blk_off) &&
+ (aid->target_blk < blk_off + span);
+ }
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ if (len < 0)
+ return len;
+
+ if (contains) {
+ enc->enc_mode = mode;
+ enc->inverse = inverse;
+ enc->ptr = ptr;
+ enc->len = (u8)len;
+ enc->olb_blk_offset = blk_off;
+ return 0;
+ }
+
+ ptr += len;
+ }
+
+ return -ENOENT;
+}
+
+static inline bool ieee80211_s1g_parse_bitmap(struct s1g_tim_enc_block *enc,
+ struct s1g_tim_aid *aid)
+{
+ const u8 *ptr = enc->ptr;
+ u8 blkmap = *ptr++;
+
+ /*
+ * If our block bitmap does not contain a set bit that corresponds
+ * to our AID, it could mean a variety of things depending on if
+ * the encoding mode is inverted or not.
+ *
+ * 1. If inverted, it means the entire subblock is present and hence
+ * our AID has been set.
+ * 2. If not inverted, it means our subblock is not present and hence
+ * it is all zero meaning our AID is not set.
+ */
+ if (!(blkmap & BIT(aid->target_subblk)))
+ return enc->inverse;
+
+ /*
+ * Increment ptr by the number of set subblocks that appear before our
+ * target subblock. If our target subblock is 0, do nothing as ptr
+ * already points to our target subblock.
+ */
+ if (aid->target_subblk)
+ ptr += hweight8(blkmap & GENMASK(aid->target_subblk - 1, 0));
+
+ return !!(*ptr & BIT(aid->target_subblk_bit)) ^ enc->inverse;
+}
+
+static inline bool ieee80211_s1g_parse_single(struct s1g_tim_enc_block *enc,
+ struct s1g_tim_aid *aid)
+{
+ /*
+ * Single AID mode describes, as the name suggests, a single AID
+ * within the block described by the encoded block. The octet
+ * contains the 6 LSBs of the AID described in the block. The other
+ * 2 bits are reserved. When inversed, every single AID described
+ * by the current block have buffered traffic except for the AID
+ * described in the single AID octet.
+ */
+ return ((*enc->ptr & 0x3f) == (aid->aid & 0x3f)) ^ enc->inverse;
+}
+
+static inline bool ieee80211_s1g_parse_olb(struct s1g_tim_enc_block *enc,
+ struct s1g_tim_aid *aid)
+{
+ const u8 *ptr = enc->ptr;
+ u8 blk_len = *ptr++;
+ /*
+ * Given an OLB encoded block that describes multiple blocks,
+ * calculate the offset into the span. Then calculate the
+ * subblock location normally.
+ */
+ u16 span_offset = aid->target_blk - enc->olb_blk_offset;
+ u16 subblk_idx = span_offset * 8 + aid->target_subblk;
+
+ if (subblk_idx >= blk_len)
+ return enc->inverse;
+
+ return !!(ptr[subblk_idx] & BIT(aid->target_subblk_bit)) ^ enc->inverse;
+}
+
+/*
+ * An S1G PVB has 3 non optional encoding types, each that can be inverted.
+ * An S1G PVB is constructed with zero or more encoded block subfields. Each
+ * encoded block represents a single "block" of AIDs (64), and each encoded
+ * block can contain one of the 3 encoding types alongside a single bit for
+ * whether the bits should be inverted.
+ *
+ * As the standard makes no guarantee about the ordering of encoded blocks,
+ * we must parse every encoded block in the worst case scenario given an
+ * AID that lies within the last block.
+ */
+static inline bool ieee80211_s1g_check_tim(const struct ieee80211_tim_ie *tim,
+ u8 tim_len, u16 aid)
+{
+ int err;
+ struct s1g_tim_aid target_aid;
+ struct s1g_tim_enc_block enc_blk;
+
+ if (tim_len < 3)
+ return false;
+
+ target_aid.aid = aid;
+ target_aid.target_blk = (aid >> 6) & 0x1f;
+ target_aid.target_subblk = (aid >> 3) & 0x7;
+ target_aid.target_subblk_bit = aid & 0x7;
+
+ /*
+ * Find our AIDs target encoded block and fill &enc_blk with the
+ * encoded blocks information. If no entry is found or an error
+ * occurs return false.
+ */
+ err = ieee80211_s1g_find_target_block(&enc_blk, &target_aid,
+ tim->virtual_map,
+ (const u8 *)tim + tim_len);
+ if (err)
+ return false;
+
+ switch (enc_blk.enc_mode) {
+ case IEEE80211_S1G_TIM_ENC_MODE_BLOCK:
+ return ieee80211_s1g_parse_bitmap(&enc_blk, &target_aid);
+ case IEEE80211_S1G_TIM_ENC_MODE_SINGLE:
+ return ieee80211_s1g_parse_single(&enc_blk, &target_aid);
+ case IEEE80211_S1G_TIM_ENC_MODE_OLB:
+ return ieee80211_s1g_parse_olb(&enc_blk, &target_aid);
+ default:
+ return false;
+ }
+}
+
+#endif /* LINUX_IEEE80211_H */
diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h
index 2cf3ae4b940a..44cca5f4c721 100644
--- a/include/linux/ieee80211.h
+++ b/include/linux/ieee80211.h
@@ -109,17 +109,6 @@
#define IEEE80211_STYPE_DMG_BEACON 0x0000
#define IEEE80211_STYPE_S1G_BEACON 0x0010
-/* bits unique to S1G beacon */
-#define IEEE80211_S1G_BCN_NEXT_TBTT 0x100
-#define IEEE80211_S1G_BCN_CSSID 0x200
-#define IEEE80211_S1G_BCN_ANO 0x400
-
-/* see 802.11ah-2016 9.9 NDP CMAC frames */
-#define IEEE80211_S1G_1MHZ_NDP_BITS 25
-#define IEEE80211_S1G_1MHZ_NDP_BYTES 4
-#define IEEE80211_S1G_2MHZ_NDP_BITS 37
-#define IEEE80211_S1G_2MHZ_NDP_BYTES 5
-
#define IEEE80211_NDP_FTYPE_CTS 0
#define IEEE80211_NDP_FTYPE_CF_END 0
#define IEEE80211_NDP_FTYPE_PS_POLL 1
@@ -221,11 +210,6 @@ static inline u16 ieee80211_sn_sub(u16 sn1, u16 sn2)
#define IEEE80211_MAX_TIM_LEN 251
#define IEEE80211_MAX_MESH_PEERINGS 63
-/* S1G encoding types */
-#define IEEE80211_S1G_TIM_ENC_MODE_BLOCK 0
-#define IEEE80211_S1G_TIM_ENC_MODE_SINGLE 1
-#define IEEE80211_S1G_TIM_ENC_MODE_OLB 2
-
/* Maximum size for the MA-UNITDATA primitive, 802.11 standard section
6.2.1.1.2.
@@ -604,55 +588,6 @@ static inline bool ieee80211_is_beacon(__le16 fc)
cpu_to_le16(IEEE80211_FTYPE_MGMT | IEEE80211_STYPE_BEACON);
}
-/**
- * ieee80211_is_s1g_beacon - check if IEEE80211_FTYPE_EXT &&
- * IEEE80211_STYPE_S1G_BEACON
- * @fc: frame control bytes in little-endian byteorder
- * Return: whether or not the frame is an S1G beacon
- */
-static inline bool ieee80211_is_s1g_beacon(__le16 fc)
-{
- return (fc & cpu_to_le16(IEEE80211_FCTL_FTYPE |
- IEEE80211_FCTL_STYPE)) ==
- cpu_to_le16(IEEE80211_FTYPE_EXT | IEEE80211_STYPE_S1G_BEACON);
-}
-
-/**
- * ieee80211_s1g_has_next_tbtt - check if IEEE80211_S1G_BCN_NEXT_TBTT
- * @fc: frame control bytes in little-endian byteorder
- * Return: whether or not the frame contains the variable-length
- * next TBTT field
- */
-static inline bool ieee80211_s1g_has_next_tbtt(__le16 fc)
-{
- return ieee80211_is_s1g_beacon(fc) &&
- (fc & cpu_to_le16(IEEE80211_S1G_BCN_NEXT_TBTT));
-}
-
-/**
- * ieee80211_s1g_has_ano - check if IEEE80211_S1G_BCN_ANO
- * @fc: frame control bytes in little-endian byteorder
- * Return: whether or not the frame contains the variable-length
- * ANO field
- */
-static inline bool ieee80211_s1g_has_ano(__le16 fc)
-{
- return ieee80211_is_s1g_beacon(fc) &&
- (fc & cpu_to_le16(IEEE80211_S1G_BCN_ANO));
-}
-
-/**
- * ieee80211_s1g_has_cssid - check if IEEE80211_S1G_BCN_CSSID
- * @fc: frame control bytes in little-endian byteorder
- * Return: whether or not the frame contains the variable-length
- * compressed SSID field
- */
-static inline bool ieee80211_s1g_has_cssid(__le16 fc)
-{
- return ieee80211_is_s1g_beacon(fc) &&
- (fc & cpu_to_le16(IEEE80211_S1G_BCN_CSSID));
-}
-
/**
* ieee80211_is_atim - check if IEEE80211_FTYPE_MGMT && IEEE80211_STYPE_ATIM
* @fc: frame control bytes in little-endian byteorder
@@ -984,37 +919,6 @@ struct ieee80211_tim_ie {
};
} __packed;
-/**
- * enum ieee80211_s1g_chanwidth - S1G channel widths
- * These are defined in IEEE802.11-2016ah Table 10-20
- * as BSS Channel Width
- *
- * @IEEE80211_S1G_CHANWIDTH_1MHZ: 1MHz operating channel
- * @IEEE80211_S1G_CHANWIDTH_2MHZ: 2MHz operating channel
- * @IEEE80211_S1G_CHANWIDTH_4MHZ: 4MHz operating channel
- * @IEEE80211_S1G_CHANWIDTH_8MHZ: 8MHz operating channel
- * @IEEE80211_S1G_CHANWIDTH_16MHZ: 16MHz operating channel
- */
-enum ieee80211_s1g_chanwidth {
- IEEE80211_S1G_CHANWIDTH_1MHZ = 0,
- IEEE80211_S1G_CHANWIDTH_2MHZ = 1,
- IEEE80211_S1G_CHANWIDTH_4MHZ = 3,
- IEEE80211_S1G_CHANWIDTH_8MHZ = 7,
- IEEE80211_S1G_CHANWIDTH_16MHZ = 15,
-};
-
-/**
- * enum ieee80211_s1g_pri_chanwidth - S1G primary channel widths
- * described in IEEE80211-2024 Table 10-39.
- *
- * @IEEE80211_S1G_PRI_CHANWIDTH_2MHZ: 2MHz primary channel
- * @IEEE80211_S1G_PRI_CHANWIDTH_1MHZ: 1MHz primary channel
- */
-enum ieee80211_s1g_pri_chanwidth {
- IEEE80211_S1G_PRI_CHANWIDTH_2MHZ = 0,
- IEEE80211_S1G_PRI_CHANWIDTH_1MHZ = 1,
-};
-
#define WLAN_SA_QUERY_TR_ID_LEN 2
#define WLAN_MEMBERSHIP_LEN 8
#define WLAN_USER_POSITION_LEN 16
@@ -1042,61 +946,6 @@ struct ieee80211_addba_ext_ie {
u8 data;
} __packed;
-/**
- * struct ieee80211_s1g_bcn_compat_ie - S1G Beacon Compatibility element
- * @compat_info: Compatibility Information
- * @beacon_int: Beacon Interval
- * @tsf_completion: TSF Completion
- *
- * This structure represents the payload of the "S1G Beacon
- * Compatibility element" as described in IEEE Std 802.11-2020 section
- * 9.4.2.196.
- */
-struct ieee80211_s1g_bcn_compat_ie {
- __le16 compat_info;
- __le16 beacon_int;
- __le32 tsf_completion;
-} __packed;
-
-/**
- * struct ieee80211_s1g_oper_ie - S1G Operation element
- * @ch_width: S1G Operation Information Channel Width
- * @oper_class: S1G Operation Information Operating Class
- * @primary_ch: S1G Operation Information Primary Channel Number
- * @oper_ch: S1G Operation Information Channel Center Frequency
- * @basic_mcs_nss: Basic S1G-MCS and NSS Set
- *
- * This structure represents the payload of the "S1G Operation
- * element" as described in IEEE Std 802.11-2020 section 9.4.2.212.
- */
-struct ieee80211_s1g_oper_ie {
- u8 ch_width;
- u8 oper_class;
- u8 primary_ch;
- u8 oper_ch;
- __le16 basic_mcs_nss;
-} __packed;
-
-/**
- * struct ieee80211_aid_response_ie - AID Response element
- * @aid: AID/Group AID
- * @switch_count: AID Switch Count
- * @response_int: AID Response Interval
- *
- * This structure represents the payload of the "AID Response element"
- * as described in IEEE Std 802.11-2020 section 9.4.2.194.
- */
-struct ieee80211_aid_response_ie {
- __le16 aid;
- u8 switch_count;
- __le16 response_int;
-} __packed;
-
-struct ieee80211_s1g_cap {
- u8 capab_info[10];
- u8 supp_mcs_nss[5];
-} __packed;
-
struct ieee80211_ext {
__le16 frame_control;
__le16 duration;
@@ -1110,37 +959,6 @@ struct ieee80211_ext {
} u;
} __packed __aligned(2);
-/**
- * ieee80211_s1g_optional_len - determine length of optional S1G beacon fields
- * @fc: frame control bytes in little-endian byteorder
- * Return: total length in bytes of the optional fixed-length fields
- *
- * S1G beacons may contain up to three optional fixed-length fields that
- * precede the variable-length elements. Whether these fields are present
- * is indicated by flags in the frame control field.
- *
- * From IEEE 802.11-2024 section 9.3.4.3:
- * - Next TBTT field may be 0 or 3 bytes
- * - Short SSID field may be 0 or 4 bytes
- * - Access Network Options (ANO) field may be 0 or 1 byte
- */
-static inline size_t
-ieee80211_s1g_optional_len(__le16 fc)
-{
- size_t len = 0;
-
- if (ieee80211_s1g_has_next_tbtt(fc))
- len += 3;
-
- if (ieee80211_s1g_has_cssid(fc))
- len += 4;
-
- if (ieee80211_s1g_has_ano(fc))
- len += 1;
-
- return len;
-}
-
/**
* struct ieee80211_bss_load_elem - BSS Load elemen
*
@@ -1568,98 +1386,6 @@ struct ieee80211_p2p_noa_attr {
#define IEEE80211_P2P_OPPPS_ENABLE_BIT BIT(7)
#define IEEE80211_P2P_OPPPS_CTWINDOW_MASK 0x7F
-/* S1G Capabilities Information field */
-#define IEEE80211_S1G_CAPABILITY_LEN 15
-
-#define S1G_CAP0_S1G_LONG BIT(0)
-#define S1G_CAP0_SGI_1MHZ BIT(1)
-#define S1G_CAP0_SGI_2MHZ BIT(2)
-#define S1G_CAP0_SGI_4MHZ BIT(3)
-#define S1G_CAP0_SGI_8MHZ BIT(4)
-#define S1G_CAP0_SGI_16MHZ BIT(5)
-#define S1G_CAP0_SUPP_CH_WIDTH GENMASK(7, 6)
-
-#define S1G_SUPP_CH_WIDTH_2 0
-#define S1G_SUPP_CH_WIDTH_4 1
-#define S1G_SUPP_CH_WIDTH_8 2
-#define S1G_SUPP_CH_WIDTH_16 3
-#define S1G_SUPP_CH_WIDTH_MAX(cap) ((1 << FIELD_GET(S1G_CAP0_SUPP_CH_WIDTH, \
- cap[0])) << 1)
-
-#define S1G_CAP1_RX_LDPC BIT(0)
-#define S1G_CAP1_TX_STBC BIT(1)
-#define S1G_CAP1_RX_STBC BIT(2)
-#define S1G_CAP1_SU_BFER BIT(3)
-#define S1G_CAP1_SU_BFEE BIT(4)
-#define S1G_CAP1_BFEE_STS GENMASK(7, 5)
-
-#define S1G_CAP2_SOUNDING_DIMENSIONS GENMASK(2, 0)
-#define S1G_CAP2_MU_BFER BIT(3)
-#define S1G_CAP2_MU_BFEE BIT(4)
-#define S1G_CAP2_PLUS_HTC_VHT BIT(5)
-#define S1G_CAP2_TRAVELING_PILOT GENMASK(7, 6)
-
-#define S1G_CAP3_RD_RESPONDER BIT(0)
-#define S1G_CAP3_HT_DELAYED_BA BIT(1)
-#define S1G_CAP3_MAX_MPDU_LEN BIT(2)
-#define S1G_CAP3_MAX_AMPDU_LEN_EXP GENMASK(4, 3)
-#define S1G_CAP3_MIN_MPDU_START GENMASK(7, 5)
-
-#define S1G_CAP4_UPLINK_SYNC BIT(0)
-#define S1G_CAP4_DYNAMIC_AID BIT(1)
-#define S1G_CAP4_BAT BIT(2)
-#define S1G_CAP4_TIME_ADE BIT(3)
-#define S1G_CAP4_NON_TIM BIT(4)
-#define S1G_CAP4_GROUP_AID BIT(5)
-#define S1G_CAP4_STA_TYPE GENMASK(7, 6)
-
-#define S1G_CAP5_CENT_AUTH_CONTROL BIT(0)
-#define S1G_CAP5_DIST_AUTH_CONTROL BIT(1)
-#define S1G_CAP5_AMSDU BIT(2)
-#define S1G_CAP5_AMPDU BIT(3)
-#define S1G_CAP5_ASYMMETRIC_BA BIT(4)
-#define S1G_CAP5_FLOW_CONTROL BIT(5)
-#define S1G_CAP5_SECTORIZED_BEAM GENMASK(7, 6)
-
-#define S1G_CAP6_OBSS_MITIGATION BIT(0)
-#define S1G_CAP6_FRAGMENT_BA BIT(1)
-#define S1G_CAP6_NDP_PS_POLL BIT(2)
-#define S1G_CAP6_RAW_OPERATION BIT(3)
-#define S1G_CAP6_PAGE_SLICING BIT(4)
-#define S1G_CAP6_TXOP_SHARING_IMP_ACK BIT(5)
-#define S1G_CAP6_VHT_LINK_ADAPT GENMASK(7, 6)
-
-#define S1G_CAP7_TACK_AS_PS_POLL BIT(0)
-#define S1G_CAP7_DUP_1MHZ BIT(1)
-#define S1G_CAP7_MCS_NEGOTIATION BIT(2)
-#define S1G_CAP7_1MHZ_CTL_RESPONSE_PREAMBLE BIT(3)
-#define S1G_CAP7_NDP_BFING_REPORT_POLL BIT(4)
-#define S1G_CAP7_UNSOLICITED_DYN_AID BIT(5)
-#define S1G_CAP7_SECTOR_TRAINING_OPERATION BIT(6)
-#define S1G_CAP7_TEMP_PS_MODE_SWITCH BIT(7)
-
-#define S1G_CAP8_TWT_GROUPING BIT(0)
-#define S1G_CAP8_BDT BIT(1)
-#define S1G_CAP8_COLOR GENMASK(4, 2)
-#define S1G_CAP8_TWT_REQUEST BIT(5)
-#define S1G_CAP8_TWT_RESPOND BIT(6)
-#define S1G_CAP8_PV1_FRAME BIT(7)
-
-#define S1G_CAP9_LINK_ADAPT_PER_CONTROL_RESPONSE BIT(0)
-
-#define S1G_OPER_CH_WIDTH_PRIMARY BIT(0)
-#define S1G_OPER_CH_WIDTH_OPER GENMASK(4, 1)
-#define S1G_OPER_CH_PRIMARY_LOCATION BIT(5)
-
-#define S1G_2M_PRIMARY_LOCATION_LOWER 0
-#define S1G_2M_PRIMARY_LOCATION_UPPER 1
-
-#define LISTEN_INT_USF GENMASK(15, 14)
-#define LISTEN_INT_UI GENMASK(13, 0)
-
-#define IEEE80211_MAX_USF FIELD_MAX(LISTEN_INT_USF)
-#define IEEE80211_MAX_UI FIELD_MAX(LISTEN_INT_UI)
-
/* Authentication algorithms */
#define WLAN_AUTH_OPEN 0
#define WLAN_AUTH_SHARED_KEY 1
@@ -2199,20 +1925,6 @@ enum ieee80211_key_len {
WLAN_KEY_LEN_BIP_GMAC_256 = 32,
};
-enum ieee80211_s1g_actioncode {
- WLAN_S1G_AID_SWITCH_REQUEST,
- WLAN_S1G_AID_SWITCH_RESPONSE,
- WLAN_S1G_SYNC_CONTROL,
- WLAN_S1G_STA_INFO_ANNOUNCE,
- WLAN_S1G_EDCA_PARAM_SET,
- WLAN_S1G_EL_OPERATION,
- WLAN_S1G_TWT_SETUP,
- WLAN_S1G_TWT_TEARDOWN,
- WLAN_S1G_SECT_GROUP_ID_LIST,
- WLAN_S1G_SECT_ID_FEEDBACK,
- WLAN_S1G_TWT_INFORMATION = 11,
-};
-
/* Radio measurement action codes as defined in IEEE 802.11-2024 - Table 9-470 */
enum ieee80211_radio_measurement_actioncode {
WLAN_RM_ACTION_RADIO_MEASUREMENT_REQUEST = 0,
@@ -2892,254 +2604,6 @@ static inline bool __ieee80211_check_tim(const struct ieee80211_tim_ie *tim,
return !!(tim->virtual_map[index] & mask);
}
-struct s1g_tim_aid {
- u16 aid;
- u8 target_blk; /* Target block index */
- u8 target_subblk; /* Target subblock index */
- u8 target_subblk_bit; /* Target subblock bit */
-};
-
-struct s1g_tim_enc_block {
- u8 enc_mode;
- bool inverse;
- const u8 *ptr;
- u8 len;
-
- /*
- * For an OLB encoded block that spans multiple blocks, this
- * is the offset into the span described by that encoded block.
- */
- u8 olb_blk_offset;
-};
-
-/*
- * Helper routines to quickly extract the length of an encoded block. Validation
- * is also performed to ensure the length extracted lies within the TIM.
- */
-
-static inline int ieee80211_s1g_len_bitmap(const u8 *ptr, const u8 *end)
-{
- u8 blkmap;
- u8 n_subblks;
-
- if (ptr >= end)
- return -EINVAL;
-
- blkmap = *ptr;
- n_subblks = hweight8(blkmap);
-
- if (ptr + 1 + n_subblks > end)
- return -EINVAL;
-
- return 1 + n_subblks;
-}
-
-static inline int ieee80211_s1g_len_single(const u8 *ptr, const u8 *end)
-{
- return (ptr + 1 > end) ? -EINVAL : 1;
-}
-
-static inline int ieee80211_s1g_len_olb(const u8 *ptr, const u8 *end)
-{
- if (ptr >= end)
- return -EINVAL;
-
- return (ptr + 1 + *ptr > end) ? -EINVAL : 1 + *ptr;
-}
-
-/*
- * Enumerate all encoded blocks until we find the encoded block that describes
- * our target AID. OLB is a special case as a single encoded block can describe
- * multiple blocks as a single encoded block.
- */
-static inline int ieee80211_s1g_find_target_block(struct s1g_tim_enc_block *enc,
- const struct s1g_tim_aid *aid,
- const u8 *ptr, const u8 *end)
-{
- /* need at least block-control octet */
- while (ptr + 1 <= end) {
- u8 ctrl = *ptr++;
- u8 mode = ctrl & 0x03;
- bool contains, inverse = ctrl & BIT(2);
- u8 span, blk_off = ctrl >> 3;
- int len;
-
- switch (mode) {
- case IEEE80211_S1G_TIM_ENC_MODE_BLOCK:
- len = ieee80211_s1g_len_bitmap(ptr, end);
- contains = blk_off == aid->target_blk;
- break;
- case IEEE80211_S1G_TIM_ENC_MODE_SINGLE:
- len = ieee80211_s1g_len_single(ptr, end);
- contains = blk_off == aid->target_blk;
- break;
- case IEEE80211_S1G_TIM_ENC_MODE_OLB:
- len = ieee80211_s1g_len_olb(ptr, end);
- /*
- * An OLB encoded block can describe more then one
- * block, meaning an encoded OLB block can span more
- * then a single block.
- */
- if (len > 0) {
- /* Minus one for the length octet */
- span = DIV_ROUND_UP(len - 1, 8);
- /*
- * Check if our target block lies within the
- * block span described by this encoded block.
- */
- contains = (aid->target_blk >= blk_off) &&
- (aid->target_blk < blk_off + span);
- }
- break;
- default:
- return -EOPNOTSUPP;
- }
-
- if (len < 0)
- return len;
-
- if (contains) {
- enc->enc_mode = mode;
- enc->inverse = inverse;
- enc->ptr = ptr;
- enc->len = (u8)len;
- enc->olb_blk_offset = blk_off;
- return 0;
- }
-
- ptr += len;
- }
-
- return -ENOENT;
-}
-
-static inline bool ieee80211_s1g_parse_bitmap(struct s1g_tim_enc_block *enc,
- struct s1g_tim_aid *aid)
-{
- const u8 *ptr = enc->ptr;
- u8 blkmap = *ptr++;
-
- /*
- * If our block bitmap does not contain a set bit that corresponds
- * to our AID, it could mean a variety of things depending on if
- * the encoding mode is inverted or not.
- *
- * 1. If inverted, it means the entire subblock is present and hence
- * our AID has been set.
- * 2. If not inverted, it means our subblock is not present and hence
- * it is all zero meaning our AID is not set.
- */
- if (!(blkmap & BIT(aid->target_subblk)))
- return enc->inverse;
-
- /*
- * Increment ptr by the number of set subblocks that appear before our
- * target subblock. If our target subblock is 0, do nothing as ptr
- * already points to our target subblock.
- */
- if (aid->target_subblk)
- ptr += hweight8(blkmap & GENMASK(aid->target_subblk - 1, 0));
-
- return !!(*ptr & BIT(aid->target_subblk_bit)) ^ enc->inverse;
-}
-
-static inline bool ieee80211_s1g_parse_single(struct s1g_tim_enc_block *enc,
- struct s1g_tim_aid *aid)
-{
- /*
- * Single AID mode describes, as the name suggests, a single AID
- * within the block described by the encoded block. The octet
- * contains the 6 LSBs of the AID described in the block. The other
- * 2 bits are reserved. When inversed, every single AID described
- * by the current block have buffered traffic except for the AID
- * described in the single AID octet.
- */
- return ((*enc->ptr & 0x3f) == (aid->aid & 0x3f)) ^ enc->inverse;
-}
-
-static inline bool ieee80211_s1g_parse_olb(struct s1g_tim_enc_block *enc,
- struct s1g_tim_aid *aid)
-{
- const u8 *ptr = enc->ptr;
- u8 blk_len = *ptr++;
- /*
- * Given an OLB encoded block that describes multiple blocks,
- * calculate the offset into the span. Then calculate the
- * subblock location normally.
- */
- u16 span_offset = aid->target_blk - enc->olb_blk_offset;
- u16 subblk_idx = span_offset * 8 + aid->target_subblk;
-
- if (subblk_idx >= blk_len)
- return enc->inverse;
-
- return !!(ptr[subblk_idx] & BIT(aid->target_subblk_bit)) ^ enc->inverse;
-}
-
-/*
- * An S1G PVB has 3 non optional encoding types, each that can be inverted.
- * An S1G PVB is constructed with zero or more encoded block subfields. Each
- * encoded block represents a single "block" of AIDs (64), and each encoded
- * block can contain one of the 3 encoding types alongside a single bit for
- * whether the bits should be inverted.
- *
- * As the standard makes no guarantee about the ordering of encoded blocks,
- * we must parse every encoded block in the worst case scenario given an
- * AID that lies within the last block.
- */
-static inline bool ieee80211_s1g_check_tim(const struct ieee80211_tim_ie *tim,
- u8 tim_len, u16 aid)
-{
- int err;
- struct s1g_tim_aid target_aid;
- struct s1g_tim_enc_block enc_blk;
-
- if (tim_len < 3)
- return false;
-
- target_aid.aid = aid;
- target_aid.target_blk = (aid >> 6) & 0x1f;
- target_aid.target_subblk = (aid >> 3) & 0x7;
- target_aid.target_subblk_bit = aid & 0x7;
-
- /*
- * Find our AIDs target encoded block and fill &enc_blk with the
- * encoded blocks information. If no entry is found or an error
- * occurs return false.
- */
- err = ieee80211_s1g_find_target_block(&enc_blk, &target_aid,
- tim->virtual_map,
- (const u8 *)tim + tim_len + 2);
- if (err)
- return false;
-
- switch (enc_blk.enc_mode) {
- case IEEE80211_S1G_TIM_ENC_MODE_BLOCK:
- return ieee80211_s1g_parse_bitmap(&enc_blk, &target_aid);
- case IEEE80211_S1G_TIM_ENC_MODE_SINGLE:
- return ieee80211_s1g_parse_single(&enc_blk, &target_aid);
- case IEEE80211_S1G_TIM_ENC_MODE_OLB:
- return ieee80211_s1g_parse_olb(&enc_blk, &target_aid);
- default:
- return false;
- }
-}
-
-/**
- * ieee80211_check_tim - check if AID bit is set in TIM
- * @tim: the TIM IE
- * @tim_len: length of the TIM IE
- * @aid: the AID to look for
- * @s1g: whether the TIM is from an S1G PPDU
- * Return: whether or not traffic is indicated in the TIM for the given AID
- */
-static inline bool ieee80211_check_tim(const struct ieee80211_tim_ie *tim,
- u8 tim_len, u16 aid, bool s1g)
-{
- return s1g ? ieee80211_s1g_check_tim(tim, tim_len, aid) :
- __ieee80211_check_tim(tim, tim_len, aid);
-}
-
/**
* ieee80211_get_tdls_action - get TDLS action code
* @skb: the skb containing the frame, length will not be checked
@@ -3273,39 +2737,6 @@ static inline bool ieee80211_is_ftm(struct sk_buff *skb)
return false;
}
-/**
- * ieee80211_is_s1g_short_beacon - check if frame is an S1G short beacon
- * @fc: frame control bytes in little-endian byteorder
- * @variable: pointer to the beacon frame elements
- * @variable_len: length of the frame elements
- * Return: whether or not the frame is an S1G short beacon. As per
- * IEEE80211-2024 11.1.3.10.1, The S1G beacon compatibility element shall
- * always be present as the first element in beacon frames generated at a
- * TBTT (Target Beacon Transmission Time), so any frame not containing
- * this element must have been generated at a TSBTT (Target Short Beacon
- * Transmission Time) that is not a TBTT. Additionally, short beacons are
- * prohibited from containing the S1G beacon compatibility element as per
- * IEEE80211-2024 9.3.4.3 Table 9-76, so if we have an S1G beacon with
- * either no elements or the first element is not the beacon compatibility
- * element, we have a short beacon.
- */
-static inline bool ieee80211_is_s1g_short_beacon(__le16 fc, const u8 *variable,
- size_t variable_len)
-{
- if (!ieee80211_is_s1g_beacon(fc))
- return false;
-
- /*
- * If the frame does not contain at least 1 element (this is perfectly
- * valid in a short beacon) and is an S1G beacon, we have a short
- * beacon.
- */
- if (variable_len < 2)
- return true;
-
- return variable[0] != WLAN_EID_S1G_BCN_COMPAT;
-}
-
struct element {
u8 id;
u8 datalen;
@@ -3480,5 +2911,21 @@ struct ieee80211_tbtt_info_ge_11 {
#include "ieee80211-eht.h"
#include "ieee80211-uhr.h"
#include "ieee80211-mesh.h"
+#include "ieee80211-s1g.h"
+
+/**
+ * ieee80211_check_tim - check if AID bit is set in TIM
+ * @tim: the TIM IE
+ * @tim_len: length of the TIM IE
+ * @aid: the AID to look for
+ * @s1g: whether the TIM is from an S1G PPDU
+ * Return: whether or not traffic is indicated in the TIM for the given AID
+ */
+static inline bool ieee80211_check_tim(const struct ieee80211_tim_ie *tim,
+ u8 tim_len, u16 aid, bool s1g)
+{
+ return s1g ? ieee80211_s1g_check_tim(tim, tim_len, aid) :
+ __ieee80211_check_tim(tim, tim_len, aid);
+}
#endif /* LINUX_IEEE80211_H */
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 6e51f5a48c74..c242712565f6 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -752,7 +752,8 @@ struct iommu_ops {
* @free: Release the domain after use.
*/
struct iommu_domain_ops {
- int (*attach_dev)(struct iommu_domain *domain, struct device *dev);
+ int (*attach_dev)(struct iommu_domain *domain, struct device *dev,
+ struct iommu_domain *old);
int (*set_dev_pasid)(struct iommu_domain *domain, struct device *dev,
ioasid_t pasid, struct iommu_domain *old);
diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
index 136f2980cba3..c5afd053ae32 100644
--- a/include/linux/irq_work.h
+++ b/include/linux/irq_work.h
@@ -2,8 +2,9 @@
#ifndef _LINUX_IRQ_WORK_H
#define _LINUX_IRQ_WORK_H
-#include <linux/smp_types.h>
+#include <linux/irq_work_types.h>
#include <linux/rcuwait.h>
+#include <linux/smp_types.h>
/*
* An entry can be in one of four states:
@@ -14,12 +15,6 @@
* busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed
*/
-struct irq_work {
- struct __call_single_node node;
- void (*func)(struct irq_work *);
- struct rcuwait irqwait;
-};
-
#define __IRQ_WORK_INIT(_func, _flags) (struct irq_work){ \
.node = { .u_flags = (_flags), }, \
.func = (_func), \
diff --git a/include/linux/irq_work_types.h b/include/linux/irq_work_types.h
new file mode 100644
index 000000000000..73abec5bb06e
--- /dev/null
+++ b/include/linux/irq_work_types.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_IRQ_WORK_TYPES_H
+#define _LINUX_IRQ_WORK_TYPES_H
+
+#include <linux/smp_types.h>
+#include <linux/types.h>
+
+struct irq_work {
+ struct __call_single_node node;
+ void (*func)(struct irq_work *);
+ struct rcuwait irqwait;
+};
+
+#endif
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 3c4e2401de07..56033126bcde 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2599,6 +2599,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t gfn, void __user *src, long npages
#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
#endif
#ifdef CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY
diff --git a/include/linux/list.h b/include/linux/list.h
index 0121e2f88f89..59f8aa0905d3 100644
--- a/include/linux/list.h
+++ b/include/linux/list.h
@@ -198,6 +198,29 @@ static inline void list_add_tail(struct list_head *new, struct list_head *head)
__list_add(new, head->prev, head);
}
+/**
+ * list_add_tail_release - add a new entry with release barrier
+ * @new: new entry to be added
+ * @head: list head to add it before
+ *
+ * Insert a new entry before the specified head, using a release barrier to set
+ * the ->next pointer that points to it. This is useful for implementing
+ * queues, in particular one that the elements will be walked through forwards
+ * locklessly.
+ */
+static inline void list_add_tail_release(struct list_head *new,
+ struct list_head *head)
+{
+ struct list_head *prev = head->prev;
+
+ if (__list_add_valid(new, prev, head)) {
+ new->next = head;
+ new->prev = prev;
+ head->prev = new;
+ smp_store_release(&prev->next, new);
+ }
+}
+
/*
* Delete a list entry by making the prev/next entries
* point to each other.
@@ -651,6 +674,20 @@ static inline void list_splice_tail_init(struct list_head *list,
pos__ != head__ ? list_entry(pos__, type, member) : NULL; \
})
+/**
+ * list_first_entry_or_null_acquire - get the first element from a list with barrier
+ * @ptr: the list head to take the element from.
+ * @type: the type of the struct this is embedded in.
+ * @member: the name of the list_head within the struct.
+ *
+ * Note that if the list is empty, it returns NULL.
+ */
+#define list_first_entry_or_null_acquire(ptr, type, member) ({ \
+ struct list_head *head__ = (ptr); \
+ struct list_head *pos__ = smp_load_acquire(&head__->next); \
+ pos__ != head__ ? list_entry(pos__, type, member) : NULL; \
+})
+
/**
* list_last_entry_or_null - get the last element from a list
* @ptr: the list head to take the element from.
diff --git a/include/linux/lockd/bind.h b/include/linux/lockd/bind.h
index c53c81242e72..e25943e485c5 100644
--- a/include/linux/lockd/bind.h
+++ b/include/linux/lockd/bind.h
@@ -21,11 +21,17 @@
struct svc_rqst;
struct rpc_task;
struct rpc_clnt;
+struct module;
-/*
- * This is the set of functions for lockd->nfsd communication
+/**
+ * struct nlmsvc_binding - lockd -> nfsd callback table
+ * @owner: module that provides this binding.
+ * @fopen: open a file by NFS file handle on behalf of an NLM request.
+ * @fclose: close a file that was previously opened via @fopen.
+ * Implementations MUST be semantically equivalent to fput().
*/
struct nlmsvc_binding {
+ struct module *owner;
__be32 (*fopen)(struct svc_rqst *,
struct nfs_fh *,
struct file **,
@@ -33,7 +39,7 @@ struct nlmsvc_binding {
void (*fclose)(struct file *);
};
-extern const struct nlmsvc_binding *nlmsvc_ops;
+extern const struct nlmsvc_binding __rcu *nlmsvc_ops;
/*
* Similar to nfs_client_initdata, but without the NFS-specific
diff --git a/include/linux/mempool.h b/include/linux/mempool.h
index 34941a4b9026..e440b5d60b8d 100644
--- a/include/linux/mempool.h
+++ b/include/linux/mempool.h
@@ -67,6 +67,13 @@ extern void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) __malloc;
#define mempool_alloc(...) \
alloc_hooks(mempool_alloc_noprof(__VA_ARGS__))
+/*
+ * Allocate a new element without dipping into the pool's reserves or
+ * waiting. Returns NULL on failure.
+ */
+#define mempool_alloc_noreserve(_pool, _gfp) \
+ alloc_hooks((_pool)->alloc(_gfp, (_pool)->pool_data))
+
extern void *mempool_alloc_preallocated(mempool_t *pool) __malloc;
extern void mempool_free(void *element, mempool_t *pool);
diff --git a/include/linux/mount.h b/include/linux/mount.h
index acfe7ef86a1b..b16ba5d45605 100644
--- a/include/linux/mount.h
+++ b/include/linux/mount.h
@@ -52,7 +52,7 @@ enum mount_flags {
MNT_ATIME_MASK = MNT_NOATIME | MNT_NODIRATIME | MNT_RELATIME,
MNT_INTERNAL_FLAGS = MNT_INTERNAL | MNT_DOOMED |
- MNT_SYNC_UMOUNT | MNT_LOCKED
+ MNT_SYNC_UMOUNT | MNT_LOCKED | MNT_UMOUNT
};
struct vfsmount {
diff --git a/include/linux/net.h b/include/linux/net.h
index ec09620f40f7..0e316f063113 100644
--- a/include/linux/net.h
+++ b/include/linux/net.h
@@ -163,7 +163,7 @@ struct proto_ops {
struct module *owner;
int (*release) (struct socket *sock);
int (*bind) (struct socket *sock,
- struct sockaddr *myaddr,
+ struct sockaddr_unsized *myaddr,
int sockaddr_len);
int (*connect) (struct socket *sock,
struct sockaddr *vaddr,
@@ -345,7 +345,7 @@ int kernel_sendmsg(struct socket *sock, struct msghdr *msg, struct kvec *vec,
int kernel_recvmsg(struct socket *sock, struct msghdr *msg, struct kvec *vec,
size_t num, size_t len, int flags);
-int kernel_bind(struct socket *sock, struct sockaddr *addr, int addrlen);
+int kernel_bind(struct socket *sock, struct sockaddr_unsized *addr, int addrlen);
int kernel_listen(struct socket *sock, int backlog);
int kernel_accept(struct socket *sock, struct socket **newsock, int flags);
int kernel_connect(struct socket *sock, struct sockaddr *addr, int addrlen,
diff --git a/include/linux/netfilter/x_tables.h b/include/linux/netfilter/x_tables.h
index 3aef60abd362..6fd365f7b35b 100644
--- a/include/linux/netfilter/x_tables.h
+++ b/include/linux/netfilter/x_tables.h
@@ -297,6 +297,7 @@ struct xt_counters *xt_counters_alloc(unsigned int counters);
struct xt_table *xt_register_table(struct net *net,
const struct xt_table *table,
+ const struct nf_hook_ops *template_ops,
struct xt_table_info *bootstrap,
struct xt_table_info *newinfo);
void xt_unregister_table_pre_exit(struct net *net, u8 af, const char *name);
diff --git a/include/linux/netfs.h b/include/linux/netfs.h
index 40eedb7bb06d..fff3d4fc69eb 100644
--- a/include/linux/netfs.h
+++ b/include/linux/netfs.h
@@ -253,6 +253,7 @@ struct netfs_io_request {
unsigned long long cleaned_to; /* Position we've cleaned folios to */
unsigned long long abandon_to; /* Position to abandon folios to */
const struct folio *no_unlock_folio; /* Don't unlock this folio after read */
+ gfp_t gfp; /* GFP flags to use */
unsigned int direct_bv_count; /* Number of elements in direct_bv[] */
unsigned int debug_id;
unsigned int rsize; /* Maximum read size (0 for none) */
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index 191b524e5c0d..5c29e904a9c5 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -256,6 +256,7 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
* @msg_clear_mask: See ntb_msg_clear_mask().
* @msg_read: See ntb_msg_read().
* @peer_msg_write: See ntb_peer_msg_write().
+ * @get_dma_dev: See ntb_get_dma_dev().
*/
struct ntb_dev_ops {
int (*port_number)(struct ntb_dev *ntb);
@@ -329,6 +330,7 @@ struct ntb_dev_ops {
int (*msg_clear_mask)(struct ntb_dev *ntb, u64 mask_bits);
u32 (*msg_read)(struct ntb_dev *ntb, int *pidx, int midx);
int (*peer_msg_write)(struct ntb_dev *ntb, int pidx, int midx, u32 msg);
+ struct device *(*get_dma_dev)(struct ntb_dev *ntb);
};
static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
@@ -391,6 +393,8 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
/* !ops->msg_clear_mask == !ops->msg_count && */
!ops->msg_read == !ops->msg_count &&
!ops->peer_msg_write == !ops->msg_count &&
+
+ /* ops->get_dma_dev is optional */
1;
}
@@ -1563,6 +1567,26 @@ static inline int ntb_peer_msg_write(struct ntb_dev *ntb, int pidx, int midx,
return ntb->ops->peer_msg_write(ntb, pidx, midx, msg);
}
+/**
+ * ntb_get_dma_dev() - get the device to use for DMA allocations/mappings
+ * @ntb: NTB device context.
+ *
+ * Return a struct device suitable for DMA API allocations and mappings.
+ * This is typically the parent of the NTB device, but may be overridden by a
+ * driver by implementing .get_dma_dev().
+ *
+ * Drivers that implement .get_dma_dev() must return a non-NULL pointer.
+ *
+ * Return: device pointer to use for DMA operations.
+ */
+static inline struct device *ntb_get_dma_dev(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->get_dma_dev)
+ return ntb->dev.parent;
+
+ return ntb->ops->get_dma_dev(ntb);
+}
+
/**
* ntb_peer_resource_idx() - get a resource index for a given peer idx
* @ntb: NTB device context.
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index b9d3dbe2564d..2a5d2d941d84 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -1255,8 +1255,8 @@ enum {
NVME_ZRASF_ZONE_STATE_IMP_OPEN = 0x02,
NVME_ZRASF_ZONE_STATE_EXP_OPEN = 0x03,
NVME_ZRASF_ZONE_STATE_CLOSED = 0x04,
- NVME_ZRASF_ZONE_STATE_READONLY = 0x05,
- NVME_ZRASF_ZONE_STATE_FULL = 0x06,
+ NVME_ZRASF_ZONE_STATE_FULL = 0x05,
+ NVME_ZRASF_ZONE_STATE_READONLY = 0x06,
NVME_ZRASF_ZONE_STATE_OFFLINE = 0x07,
NVME_REPORT_ZONE_PARTIAL = 1,
};
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 148b71af80b3..95c77e2a1d1f 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -937,6 +937,7 @@ struct module;
* function returns zero when the driver chooses to
* take "ownership" of the device or an error code
* (negative number) otherwise.
+ * The pci_device_id parameter is only valid during probe.
* The probe function always gets called from process
* context, so it can sleep.
* @remove: The remove() function gets called whenever a device
diff --git a/include/linux/ring_buffer.h b/include/linux/ring_buffer.h
index d862fa610270..be0e57b89c12 100644
--- a/include/linux/ring_buffer.h
+++ b/include/linux/ring_buffer.h
@@ -218,14 +218,15 @@ bool ring_buffer_time_stamp_abs(struct trace_buffer *buffer);
size_t ring_buffer_nr_dirty_pages(struct trace_buffer *buffer, int cpu);
struct buffer_data_read_page;
-struct buffer_data_read_page *
-ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu);
+int ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu,
+ struct buffer_data_read_page **rpage);
void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
struct buffer_data_read_page *page);
int ring_buffer_read_page(struct trace_buffer *buffer,
struct buffer_data_read_page *data_page,
size_t len, int cpu, int full);
void *ring_buffer_read_page_data(struct buffer_data_read_page *page);
+unsigned int ring_buffer_read_page_size(struct buffer_data_read_page *rpage);
struct trace_seq;
diff --git a/include/linux/rolling_buffer.h b/include/linux/rolling_buffer.h
index ac15b1ffdd83..9e5dad29669c 100644
--- a/include/linux/rolling_buffer.h
+++ b/include/linux/rolling_buffer.h
@@ -43,13 +43,13 @@ struct rolling_buffer_snapshot {
#define ROLLBUF_MARK_2 BIT(1)
int rolling_buffer_init(struct rolling_buffer *roll, unsigned int rreq_id,
- unsigned int direction);
-int rolling_buffer_make_space(struct rolling_buffer *roll);
+ unsigned int direction, gfp_t gfp);
+int rolling_buffer_make_space(struct rolling_buffer *roll, gfp_t gfp);
ssize_t rolling_buffer_load_from_ra(struct rolling_buffer *roll,
struct readahead_control *ractl,
struct folio_batch *put_batch);
ssize_t rolling_buffer_append(struct rolling_buffer *roll, struct folio *folio,
- unsigned int flags);
+ unsigned int flags, gfp_t gfp);
struct folio_queue *rolling_buffer_delete_spent(struct rolling_buffer *roll);
void rolling_buffer_clear(struct rolling_buffer *roll);
diff --git a/include/linux/sched/ext.h b/include/linux/sched/ext.h
index 9848aeab2786..7e5614e2a331 100644
--- a/include/linux/sched/ext.h
+++ b/include/linux/sched/ext.h
@@ -152,6 +152,7 @@ struct sched_ext_entity {
s32 holding_cpu;
s32 selected_cpu;
u32 kf_mask; /* see scx_kf_mask above */
+ u32 ops_qseq; /* protected by rq lock */
struct task_struct *kf_tasks[2]; /* see SCX_CALL_OP_TASK() */
atomic_long_t ops_state;
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 237e8b0cb08a..5af1127dd870 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -821,6 +821,7 @@ enum skb_tstamp_type {
* @_sk_redir: socket redirection information for skmsg
* @_nfct: Associated connection, if any (with nfctinfo bits)
* @skb_iif: ifindex of device we arrived on
+ * @tc_depth: counter for packet duplication
* @tc_index: Traffic control index
* @hash: the packet hash
* @queue_mapping: Queue mapping for multiqueue devices
@@ -1030,6 +1031,7 @@ struct sk_buff {
__u8 csum_not_inet:1;
#endif
__u8 unreadable:1;
+ __u8 tc_depth:2;
#if defined(CONFIG_NET_SCHED) || defined(CONFIG_NET_XGRESS)
__u16 tc_index; /* traffic control index */
#endif
diff --git a/include/linux/socket.h b/include/linux/socket.h
index 3b262487ec06..7b1a01be29da 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -40,6 +40,23 @@ struct sockaddr {
};
};
+/**
+ * struct sockaddr_unsized - Unspecified size sockaddr for callbacks
+ * @sa_family: Address family (AF_UNIX, AF_INET, AF_INET6, etc.)
+ * @sa_data: Flexible array for address data
+ *
+ * This structure is designed for callback interfaces where the
+ * total size is known via the sockaddr_len parameter. Unlike struct
+ * sockaddr which has a fixed 14-byte sa_data limit or struct
+ * sockaddr_storage which has a fixed 128-byte sa_data limit, this
+ * structure can accommodate addresses of any size, but must be used
+ * carefully.
+ */
+struct sockaddr_unsized {
+ __kernel_sa_family_t sa_family; /* address family, AF_xxx */
+ char sa_data[]; /* flexible address data */
+};
+
struct linger {
int l_onoff; /* Linger active */
int l_linger; /* How long to linger for */
diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h
index 51ce25f07930..1f9a226e6fd8 100644
--- a/include/linux/srcutiny.h
+++ b/include/linux/srcutiny.h
@@ -11,6 +11,7 @@
#ifndef _LINUX_SRCU_TINY_H
#define _LINUX_SRCU_TINY_H
+#include <linux/irq_work_types.h>
#include <linux/swait.h>
struct srcu_struct {
@@ -24,18 +25,21 @@ struct srcu_struct {
struct rcu_head *srcu_cb_head; /* Pending callbacks: Head. */
struct rcu_head **srcu_cb_tail; /* Pending callbacks: Tail. */
struct work_struct srcu_work; /* For driving grace periods. */
+ struct irq_work srcu_irq_work; /* Defer schedule_work() to irq work. */
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lockdep_map dep_map;
#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
};
void srcu_drive_gp(struct work_struct *wp);
+void srcu_tiny_irq_work(struct irq_work *irq_work);
#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \
{ \
.srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \
.srcu_cb_tail = &name.srcu_cb_head, \
.srcu_work = __WORK_INITIALIZER(name.srcu_work, srcu_drive_gp), \
+ .srcu_irq_work = { .func = srcu_tiny_irq_work }, \
__SRCU_DEP_MAP_INIT(name) \
}
diff --git a/include/linux/static_call_types.h b/include/linux/static_call_types.h
index 5a00b8b2cf9f..cfb6ddeb292b 100644
--- a/include/linux/static_call_types.h
+++ b/include/linux/static_call_types.h
@@ -25,6 +25,8 @@
#define STATIC_CALL_SITE_INIT 2UL /* init section */
#define STATIC_CALL_SITE_FLAGS 3UL
+#ifndef __ASSEMBLY__
+
/*
* The static call site table needs to be created by external tooling (objtool
* or a compiler plugin).
@@ -100,4 +102,6 @@ struct static_call_key {
#endif /* CONFIG_HAVE_STATIC_CALL */
+#endif /* __ASSEMBLY__ */
+
#endif /* _STATIC_CALL_TYPES_H */
diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h
index 22704c2e5b9b..6e1cdc3335e0 100644
--- a/include/linux/sunrpc/svc_rdma.h
+++ b/include/linux/sunrpc/svc_rdma.h
@@ -213,6 +213,7 @@ struct svc_rdma_recv_ctxt {
*/
struct svc_rdma_write_info {
struct svcxprt_rdma *wi_rdma;
+ struct list_head wi_list;
const struct svc_rdma_chunk *wi_chunk;
@@ -241,7 +242,10 @@ struct svc_rdma_send_ctxt {
struct ib_cqe sc_cqe;
struct xdr_buf sc_hdrbuf;
struct xdr_stream sc_stream;
+
+ struct list_head sc_write_info_list;
struct svc_rdma_write_info sc_reply_info;
+
void *sc_xprt_buf;
int sc_page_count;
int sc_cur_sge_no;
@@ -274,11 +278,14 @@ extern void svc_rdma_cc_init(struct svcxprt_rdma *rdma,
extern void svc_rdma_cc_release(struct svcxprt_rdma *rdma,
struct svc_rdma_chunk_ctxt *cc,
enum dma_data_direction dir);
+extern void svc_rdma_write_chunk_release(struct svcxprt_rdma *rdma,
+ struct svc_rdma_send_ctxt *ctxt);
extern void svc_rdma_reply_chunk_release(struct svcxprt_rdma *rdma,
struct svc_rdma_send_ctxt *ctxt);
-extern int svc_rdma_send_write_list(struct svcxprt_rdma *rdma,
- const struct svc_rdma_recv_ctxt *rctxt,
- const struct xdr_buf *xdr);
+extern int svc_rdma_prepare_write_list(struct svcxprt_rdma *rdma,
+ const struct svc_rdma_recv_ctxt *rctxt,
+ struct svc_rdma_send_ctxt *sctxt,
+ const struct xdr_buf *xdr);
extern int svc_rdma_prepare_reply_chunk(struct svcxprt_rdma *rdma,
const struct svc_rdma_pcl *write_pcl,
const struct svc_rdma_pcl *reply_pcl,
@@ -311,6 +318,7 @@ extern int svc_rdma_result_payload(struct svc_rqst *rqstp, unsigned int offset,
unsigned int length);
/* svc_rdma_transport.c */
+extern void svc_rdma_xprt_deferred_close(struct svcxprt_rdma *rdma);
extern struct svc_xprt_class svc_rdma_class;
#ifdef CONFIG_SUNRPC_BACKCHANNEL
extern struct svc_xprt_class svc_rdma_bc_class;
diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
index dabc351cc127..08dc532336fc 100644
--- a/include/linux/workqueue.h
+++ b/include/linux/workqueue.h
@@ -512,6 +512,28 @@ __printf(1, 4) struct workqueue_struct *
alloc_workqueue_noprof(const char *fmt, unsigned int flags, int max_active, ...);
#define alloc_workqueue(...) alloc_hooks(alloc_workqueue_noprof(__VA_ARGS__))
+/**
+ * devm_alloc_workqueue - Resource-managed allocate a workqueue
+ * @dev: Device to allocate workqueue for
+ * @fmt: printf format for the name of the workqueue
+ * @flags: WQ_* flags
+ * @max_active: max in-flight work items, 0 for default
+ * @...: args for @fmt
+ *
+ * Resource managed workqueue, see alloc_workqueue() for details.
+ *
+ * The workqueue will be automatically destroyed on driver detach. Typically
+ * this should be used in drivers already relying on devm interafaces.
+ *
+ * RETURNS:
+ * Pointer to the allocated workqueue on success, %NULL on failure.
+ */
+__printf(2, 5) struct workqueue_struct *
+devm_alloc_workqueue_noprof(struct device *dev, const char *fmt,
+ unsigned int flags, int max_active, ...);
+#define devm_alloc_workqueue(...) \
+ alloc_hooks(devm_alloc_workqueue_noprof(__VA_ARGS__))
+
#ifdef CONFIG_LOCKDEP
/**
* alloc_workqueue_lockdep_map - allocate a workqueue with user-defined lockdep_map
@@ -568,6 +590,8 @@ alloc_workqueue_lockdep_map(const char *fmt, unsigned int flags, int max_active,
*/
#define alloc_ordered_workqueue(fmt, flags, args...) \
alloc_workqueue(fmt, WQ_UNBOUND | __WQ_ORDERED | (flags), 1, ##args)
+#define devm_alloc_ordered_workqueue(dev, fmt, flags, args...) \
+ devm_alloc_workqueue(dev, fmt, WQ_UNBOUND | __WQ_ORDERED | (flags), 1, ##args)
#define create_workqueue(name) \
alloc_workqueue("%s", __WQ_LEGACY | WQ_MEM_RECLAIM | WQ_PERCPU, 1, (name))
diff --git a/include/media/v4l2-mem2mem.h b/include/media/v4l2-mem2mem.h
index 500f81f399df..08e3c9c4f1e9 100644
--- a/include/media/v4l2-mem2mem.h
+++ b/include/media/v4l2-mem2mem.h
@@ -544,6 +544,27 @@ v4l2_m2m_register_media_controller(struct v4l2_m2m_dev *m2m_dev,
*/
void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev);
+/**
+ * v4l2_m2m_get() - take a reference to the m2m_dev structure
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * This is used to share the M2M device across multiple devices. This
+ * can be used to avoid scheduling two hardware nodes concurrently.
+ */
+void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_put() - remove a reference to the m2m_dev structure
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Once the M2M device has no more references, v4l2_m2m_release() will be
+ * called automatically. Users of this method should never call
+ * v4l2_m2m_release() directly. See v4l2_m2m_get() for more details.
+ */
+void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev);
+
/**
* v4l2_m2m_ctx_init() - allocate and initialize a m2m context
*
diff --git a/include/net/amt.h b/include/net/amt.h
index c881bc8b673b..ad844d65a64d 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -231,10 +231,6 @@ struct amt_relay_headers {
};
} __packed;
-struct amt_skb_cb {
- struct amt_tunnel_list *tunnel;
-};
-
struct amt_tunnel_list {
struct list_head list;
/* Protect All resources under an amt_tunne_list */
diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h
index 4dc8e6c39c01..c416f7f2af6a 100644
--- a/include/net/bluetooth/hci_core.h
+++ b/include/net/bluetooth/hci_core.h
@@ -729,6 +729,8 @@ struct hci_conn {
__u16 le_per_adv_data_offset;
__u8 le_adv_phy;
__u8 le_adv_sec_phy;
+ __u8 le_tx_def_phys;
+ __u8 le_rx_def_phys;
__u8 le_tx_phy;
__u8 le_rx_phy;
__s8 rssi;
diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h
index 0b9bc2247d32..39f6d7ac77ee 100644
--- a/include/net/cfg80211.h
+++ b/include/net/cfg80211.h
@@ -3655,6 +3655,7 @@ enum wiphy_params_flags {
/* The per TXQ device queue limit in airtime */
#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_L 5000
#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_H 12000
+#define IEEE80211_DEFAULT_AQL_TXQ_LIMIT_MC 50000
/* The per interface airtime threshold to switch to lower queue limit */
#define IEEE80211_AQL_THRESHOLD 24000
@@ -6813,7 +6814,7 @@ struct wireless_dev {
enum ieee80211_bss_type conn_bss_type;
u32 conn_owner_nlportid;
- struct work_struct disconnect_wk;
+ struct wiphy_work disconnect_wk;
u8 disconnect_bssid[ETH_ALEN];
struct list_head event_list;
diff --git a/include/net/inet_common.h b/include/net/inet_common.h
index 0f4626776b29..0978b2e27d35 100644
--- a/include/net/inet_common.h
+++ b/include/net/inet_common.h
@@ -42,7 +42,7 @@ int inet_shutdown(struct socket *sock, int how);
int inet_listen(struct socket *sock, int backlog);
int __inet_listen_sk(struct sock *sk, int backlog);
void inet_sock_destruct(struct sock *sk);
-int inet_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len);
+int inet_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len);
int inet_bind_sk(struct sock *sk, struct sockaddr *uaddr, int addr_len);
/* Don't allocate port at this moment, defer to connect. */
#define BIND_FORCE_ADDRESS_NO_PORT (1 << 0)
diff --git a/include/net/ip.h b/include/net/ip.h
index 25a71c46792b..5041202ccce2 100644
--- a/include/net/ip.h
+++ b/include/net/ip.h
@@ -74,6 +74,13 @@ static inline unsigned int ip_hdrlen(const struct sk_buff *skb)
return ip_hdr(skb)->ihl * 4;
}
+static inline int ip_check_csum_start(const struct sk_buff *skb)
+{
+ if (unlikely(skb->csum_start < skb->network_header + ip_hdrlen(skb)))
+ return -EINVAL;
+ return 0;
+}
+
struct ipcm_cookie {
struct sockcm_cookie sockc;
__be32 addr;
diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h
index 94dfc3a265ee..955185eff130 100644
--- a/include/net/ip_vs.h
+++ b/include/net/ip_vs.h
@@ -1726,7 +1726,7 @@ static inline char ip_vs_fwd_tag(struct ip_vs_conn *cp)
return fwd;
}
-void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
+bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct ip_vs_conn *cp, int dir, unsigned int toff,
bool has_ports, struct ip_vs_iphdr *ciph);
diff --git a/include/net/ipv6.h b/include/net/ipv6.h
index aa8b525908ea..cd798f3569cb 100644
--- a/include/net/ipv6.h
+++ b/include/net/ipv6.h
@@ -1206,7 +1206,7 @@ void ipv6_local_rxpmtu(struct sock *sk, struct flowi6 *fl6, u32 mtu);
void inet6_cleanup_sock(struct sock *sk);
void inet6_sock_destruct(struct sock *sk);
int inet6_release(struct socket *sock);
-int inet6_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len);
+int inet6_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len);
int inet6_bind_sk(struct sock *sk, struct sockaddr *uaddr, int addr_len);
int inet6_getname(struct socket *sock, struct sockaddr *uaddr,
int peer);
diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index 2fa6fc593df6..f8fa1f55694b 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -1212,6 +1212,7 @@ ieee80211_rate_get_vht_nss(const struct ieee80211_tx_rate *rate)
* @status_data: internal data for TX status handling, assigned privately,
* see also &enum ieee80211_status_data for the internal documentation
* @status_data_idr: indicates status data is IDR allocated ID for ack frame
+ * @tx_time_mc: TX time estimate is for a multicast frame, used internally
* @tx_time_est: TX time estimate in units of 4us, used internally
* @control: union part for control data
* @control.rates: TX rates array to try
@@ -1249,8 +1250,8 @@ struct ieee80211_tx_info {
status_data_idr:1,
status_data:13,
hw_queue:4,
+ tx_time_mc:1,
tx_time_est:10;
- /* 1 free bit */
union {
struct {
@@ -6606,6 +6607,17 @@ void ieee80211_sta_register_airtime(struct ieee80211_sta *pubsta, u8 tid,
bool
ieee80211_txq_airtime_check(struct ieee80211_hw *hw, struct ieee80211_txq *txq);
+/**
+ * ieee80211_txq_aql_pending - get pending AQL airtime for a txq
+ *
+ * @hw: pointer obtained from ieee80211_alloc_hw()
+ * @txq: pointer obtained from station or virtual interface
+ *
+ * Return: pending airtime (in usec) for the given txq.
+ */
+u32 ieee80211_txq_aql_pending(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq);
+
/**
* ieee80211_iter_keys - iterate keys programmed into the device
* @hw: pointer obtained from ieee80211_alloc_hw()
diff --git a/include/net/netfilter/nf_dup_netdev.h b/include/net/netfilter/nf_dup_netdev.h
index b175d271aec9..f6b05bd80c3f 100644
--- a/include/net/netfilter/nf_dup_netdev.h
+++ b/include/net/netfilter/nf_dup_netdev.h
@@ -3,10 +3,47 @@
#define _NF_DUP_NETDEV_H_
#include <net/netfilter/nf_tables.h>
+#include <linux/netdevice.h>
+#include <linux/sched.h>
void nf_dup_netdev_egress(const struct nft_pktinfo *pkt, int oif);
void nf_fwd_netdev_egress(const struct nft_pktinfo *pkt, int oif);
+#define NF_RECURSION_LIMIT 2
+
+#ifndef CONFIG_PREEMPT_RT
+static inline bool nf_dev_xmit_recursion(void)
+{
+ return unlikely(__this_cpu_read(softnet_data.xmit.nf_dup_skb_recursion) >
+ NF_RECURSION_LIMIT);
+}
+
+static inline void nf_dev_xmit_recursion_inc(void)
+{
+ __this_cpu_inc(softnet_data.xmit.nf_dup_skb_recursion);
+}
+
+static inline void nf_dev_xmit_recursion_dec(void)
+{
+ __this_cpu_dec(softnet_data.xmit.nf_dup_skb_recursion);
+}
+#else
+static inline bool nf_dev_xmit_recursion(void)
+{
+ return unlikely(current->net_xmit.nf_dup_skb_recursion > NF_RECURSION_LIMIT);
+}
+
+static inline void nf_dev_xmit_recursion_inc(void)
+{
+ current->net_xmit.nf_dup_skb_recursion++;
+}
+
+static inline void nf_dev_xmit_recursion_dec(void)
+{
+ current->net_xmit.nf_dup_skb_recursion--;
+}
+#endif
+
struct nft_offload_ctx;
struct nft_flow_rule;
diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h
index e6b31aa9d6b7..4c691d5e4ae9 100644
--- a/include/net/netfilter/nf_tables.h
+++ b/include/net/netfilter/nf_tables.h
@@ -867,8 +867,6 @@ struct nft_elem_priv *nft_set_elem_init(const struct nft_set *set,
const u32 *key, const u32 *key_end,
const u32 *data,
u64 timeout, u64 expiration, gfp_t gfp);
-int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set,
- struct nft_expr *expr_array[]);
void nft_set_elem_expr_destroy(const struct nft_ctx *ctx,
struct nft_set_elem_expr *elem_expr);
void nft_set_elem_destroy(const struct nft_set *set,
@@ -1208,12 +1206,15 @@ struct nft_stats {
struct u64_stats_sync syncp;
};
+#define NFT_HOOK_REMOVE (1 << 0)
+
struct nft_hook {
struct list_head list;
struct list_head ops_list;
struct rcu_head rcu;
char ifname[IFNAMSIZ];
u8 ifnamelen;
+ u8 flags;
};
struct nf_hook_ops *nft_hook_find_ops(const struct nft_hook *hook,
@@ -1670,6 +1671,16 @@ struct nft_trans {
u8 put_net:1;
};
+/**
+ * struct nft_trans_hook - nf_tables hook update in transaction
+ * @list: used internally
+ * @hook: struct nft_hook with the device hook
+ */
+struct nft_trans_hook {
+ struct list_head list;
+ struct nft_hook *hook;
+};
+
/**
* struct nft_trans_binding - nf_tables object with binding support in transaction
* @nft_trans: base structure, MUST be first member
diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h
index 99ac747b7906..7f0c422307f4 100644
--- a/include/net/pkt_cls.h
+++ b/include/net/pkt_cls.h
@@ -156,8 +156,20 @@ static inline int tcf_classify(struct sk_buff *skb,
{
return TC_ACT_UNSPEC;
}
-
#endif
+static inline int tcf_classify_qdisc(struct sk_buff *skb,
+ const struct tcf_proto *tp,
+ struct tcf_result *res, bool compat_mode)
+{
+ int ret = tcf_classify(skb, NULL, tp, res, compat_mode);
+
+ /* TC_ACT_REDIRECT from qdisc filter chains is not supported.
+ * Use BPF via tcx or mirred redirect instead.
+ */
+ if (unlikely(ret == TC_ACT_REDIRECT))
+ ret = TC_ACT_SHOT;
+ return ret;
+}
static inline unsigned long
__cls_set_class(unsigned long *clp, unsigned long cl)
diff --git a/include/net/sock.h b/include/net/sock.h
index fb91d72a955d..50c11bd0fcfc 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1907,7 +1907,7 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
* Functions to fill in entries in struct proto_ops when a protocol
* does not implement a particular function.
*/
-int sock_no_bind(struct socket *, struct sockaddr *, int);
+int sock_no_bind(struct socket *sock, struct sockaddr_unsized *saddr, int len);
int sock_no_connect(struct socket *, struct sockaddr *, int, int);
int sock_no_socketpair(struct socket *, struct socket *);
int sock_no_accept(struct socket *, struct socket *, struct proto_accept_arg *);
diff --git a/include/net/strparser.h b/include/net/strparser.h
index 0ed73e364faa..881b7ae34498 100644
--- a/include/net/strparser.h
+++ b/include/net/strparser.h
@@ -91,11 +91,11 @@ static inline struct strp_msg *strp_msg(struct sk_buff *skb)
struct strparser {
struct sock *sk;
- u32 stopped : 1;
- u32 paused : 1;
- u32 aborted : 1;
- u32 interrupted : 1;
- u32 unrecov_intr : 1;
+ u8 stopped;
+ u8 paused;
+ u8 aborted;
+ u8 interrupted;
+ u8 unrecov_intr;
struct sk_buff **skb_nextp;
struct sk_buff *skb_head;
diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h
index 10e564d85a90..a461bb63761b 100644
--- a/include/net/xsk_buff_pool.h
+++ b/include/net/xsk_buff_pool.h
@@ -78,6 +78,9 @@ struct xsk_buff_pool {
u32 chunk_size;
u32 chunk_shift;
u32 frame_len;
+ u32 tx_descs_nentries;
+ u32 reclaim_descs;
+ u32 tx_zc_pending_descs;
u32 xdp_zc_max_segs;
u8 tx_metadata_len; /* inherited from umem */
u8 cached_need_wakeup;
@@ -107,12 +110,14 @@ struct xsk_buff_pool {
/* AF_XDP core. */
struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
- struct xdp_umem *umem);
+ struct xdp_umem *umem,
+ u32 max_segs);
int xp_assign_dev(struct xsk_buff_pool *pool, struct net_device *dev,
u16 queue_id, u16 flags);
int xp_assign_dev_shared(struct xsk_buff_pool *pool, struct xdp_sock *umem_xs,
struct net_device *dev, u16 queue_id);
-int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs);
+int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs,
+ u32 max_segs);
void xp_destroy(struct xsk_buff_pool *pool);
void xp_get_pool(struct xsk_buff_pool *pool);
bool xp_put_pool(struct xsk_buff_pool *pool);
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 936403c15e53..c02e2d42297c 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -631,6 +631,10 @@ struct Scsi_Host {
*/
unsigned nr_hw_queues;
unsigned nr_maps;
+
+ /* Don't resume host in EH */
+ bool eh_noresume;
+
unsigned active_mode:2;
/*
@@ -652,9 +656,6 @@ struct Scsi_Host {
/* Asynchronous scan in progress */
unsigned async_scan:1;
- /* Don't resume host in EH */
- unsigned eh_noresume:1;
-
/* The controller does not support WRITE SAME */
unsigned no_write_same:1;
diff --git a/include/ufs/ufshcd.h b/include/ufs/ufshcd.h
index 17fe07dac6a7..a829db551080 100644
--- a/include/ufs/ufshcd.h
+++ b/include/ufs/ufshcd.h
@@ -1433,10 +1433,6 @@ static inline int ufshcd_disable_host_tx_lcc(struct ufs_hba *hba)
void ufshcd_auto_hibern8_update(struct ufs_hba *hba, u32 ahit);
void ufshcd_fixup_dev_quirks(struct ufs_hba *hba,
const struct ufs_dev_quirk *fixups);
-#define SD_ASCII_STD true
-#define SD_RAW false
-int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
- u8 **buf, bool ascii);
void ufshcd_hold(struct ufs_hba *hba);
void ufshcd_release(struct ufs_hba *hba);
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index a41e6730edcf..472c93b3ff1b 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -446,6 +446,18 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
goto errout;
}
+ /*
+ * A >8 byte return value is passed back in a register pair,
+ * which the struct_ops trampoline does not preserve (only
+ * 8 bytes of the return value are saved and restored).
+ */
+ if (st_ops->func_models[i].ret_size > 8) {
+ pr_warn("func ptr %s in struct %s has a >8 byte return value, which is not supported\n",
+ mname, st_ops->name);
+ err = -EOPNOTSUPP;
+ goto errout;
+ }
+
stub_func_addr = *(void **)(st_ops->cfi_stubs + moff);
err = prepare_arg_info(btf, st_ops->name, mname,
func_proto, stub_func_addr,
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 3509bbe39efa..706b3eefd9ef 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6986,7 +6986,7 @@ enum bpf_struct_walk_result {
static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, int off, int size,
u32 *next_btf_id, enum bpf_type_flag *flag,
- const char **field_name)
+ const char **field_name, bool walk_flex_arrays)
{
u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
const struct btf_type *mtype, *elem_type = NULL;
@@ -7013,11 +7013,14 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
*flag |= PTR_UNTRUSTED;
if (off + size > t->size) {
+ struct btf_array *array_elem;
+
+ if (!walk_flex_arrays)
+ goto error;
+
/* If the last element is a variable size array, we may
* need to relax the rule.
*/
- struct btf_array *array_elem;
-
if (vlen == 0)
goto error;
@@ -7280,7 +7283,8 @@ int btf_struct_access(struct bpf_verifier_log *log,
t = btf_type_by_id(btf, id);
do {
- err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, field_name);
+ err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag,
+ field_name, !type_is_alloc(reg->type));
switch (err) {
case WALK_PTR:
@@ -7339,7 +7343,7 @@ bool btf_types_are_same(const struct btf *btf1, u32 id1,
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
- bool strict)
+ bool strict, bool walk_flex_arrays)
{
const struct btf_type *type;
enum bpf_type_flag flag = 0;
@@ -7358,7 +7362,8 @@ bool btf_struct_ids_match(struct bpf_verifier_log *log,
type = btf_type_by_id(btf, id);
if (!type)
return false;
- err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL);
+ err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL,
+ walk_flex_arrays);
if (err != WALK_STRUCT)
return false;
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
index c479f649596f..02b6b0e9cb6c 100644
--- a/kernel/bpf/stackmap.c
+++ b/kernel/bpf/stackmap.c
@@ -621,19 +621,55 @@ const struct bpf_func_proto bpf_get_stackid_proto_pe = {
.arg3_type = ARG_ANYTHING,
};
+static u32 callchain_store(struct perf_callchain_entry *trace, void *buf,
+ u32 elem_size, u64 flags)
+{
+ bool user_build_id = flags & BPF_F_USER_BUILD_ID;
+ u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
+ u32 trace_nr, copy_len;
+ u64 *ips;
+
+ trace_nr = trace->nr - skip;
+ copy_len = trace_nr * elem_size;
+
+ ips = trace->ip + skip;
+ if (user_build_id) {
+ struct bpf_stack_build_id *id_offs = buf;
+
+ for (u32 i = 0; i < trace_nr; i++)
+ id_offs[i].ip = ips[i];
+ } else {
+ memcpy(buf, ips, copy_len);
+ }
+ return trace_nr;
+}
+
+static long callchain_finalize(void *buf, u32 size, u32 trace_nr, u32 elem_size,
+ u64 flags, bool may_fault)
+{
+ bool user_build_id = flags & BPF_F_USER_BUILD_ID;
+ bool user = flags & BPF_F_USER_STACK;
+ u32 copy_len = trace_nr * elem_size;
+
+ if (user_build_id)
+ stack_map_get_build_id_offset(buf, trace_nr, user, may_fault);
+
+ if (size > copy_len)
+ memset(buf + copy_len, 0, size - copy_len);
+ return copy_len;
+}
+
static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
- struct perf_callchain_entry *trace_in,
void *buf, u32 size, u64 flags, bool may_fault)
{
- u32 trace_nr, copy_len, elem_size, max_depth;
bool user_build_id = flags & BPF_F_USER_BUILD_ID;
bool crosstask = task && task != current;
u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
bool user = flags & BPF_F_USER_STACK;
struct perf_callchain_entry *trace;
+ u32 trace_nr, elem_size, max_depth;
bool kernel = !user;
int err = -EINVAL;
- u64 *ips;
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_USER_BUILD_ID)))
@@ -663,10 +699,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
if (may_fault)
rcu_read_lock(); /* need RCU for perf's callchain below */
- if (trace_in) {
- trace = trace_in;
- trace->nr = min_t(u32, trace->nr, max_depth);
- } else if (kernel && task) {
+ if (kernel && task) {
trace = get_callchain_entry_for_task(task, max_depth);
} else {
trace = get_perf_callchain(regs, kernel, user, max_depth,
@@ -680,31 +713,14 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
goto err_fault;
}
- trace_nr = trace->nr - skip;
- copy_len = trace_nr * elem_size;
-
- ips = trace->ip + skip;
- if (user_build_id) {
- struct bpf_stack_build_id *id_offs = buf;
- u32 i;
-
- for (i = 0; i < trace_nr; i++)
- id_offs[i].ip = ips[i];
- } else {
- memcpy(buf, ips, copy_len);
- }
+ trace_nr = callchain_store(trace, buf, elem_size, flags);
- /* trace/ips should not be dereferenced after this point */
+ /* trace should not be dereferenced after this point */
if (may_fault)
rcu_read_unlock();
preempt_enable();
- if (user_build_id)
- stack_map_get_build_id_offset(buf, trace_nr, user, may_fault);
-
- if (size > copy_len)
- memset(buf + copy_len, 0, size - copy_len);
- return copy_len;
+ return callchain_finalize(buf, size, trace_nr, elem_size, flags, may_fault);
err_fault:
err = -EFAULT;
@@ -716,7 +732,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
BPF_CALL_4(bpf_get_stack, struct pt_regs *, regs, void *, buf, u32, size,
u64, flags)
{
- return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */);
+ return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */);
}
const struct bpf_func_proto bpf_get_stack_proto = {
@@ -732,7 +748,7 @@ const struct bpf_func_proto bpf_get_stack_proto = {
BPF_CALL_4(bpf_get_stack_sleepable, struct pt_regs *, regs, void *, buf, u32, size,
u64, flags)
{
- return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, true /* may_fault */);
+ return __bpf_get_stack(regs, NULL, buf, size, flags, true /* may_fault */);
}
const struct bpf_func_proto bpf_get_stack_sleepable_proto = {
@@ -752,14 +768,17 @@ static long __bpf_get_task_stack(struct task_struct *task, void *buf, u32 size,
struct pt_regs *regs;
long res = -EINVAL;
- if (!try_get_task_stack(task))
+ if (!try_get_task_stack(task)) {
+ memset(buf, 0, size);
return -EFAULT;
+ }
regs = task_pt_regs(task);
if (regs)
- res = __bpf_get_stack(regs, task, NULL, buf, size, flags, may_fault);
+ res = __bpf_get_stack(regs, task, buf, size, flags, may_fault);
+ else
+ memset(buf, 0, size);
put_task_stack(task);
-
return res;
}
@@ -798,6 +817,32 @@ const struct bpf_func_proto bpf_get_task_stack_sleepable_proto = {
.arg4_type = ARG_ANYTHING,
};
+static int __bpf_get_stack_pe(struct perf_callchain_entry *trace, void *buf, u32 size,
+ u64 flags)
+{
+ bool user_build_id = flags & BPF_F_USER_BUILD_ID;
+ u64 skip = flags & BPF_F_SKIP_FIELD_MASK;
+ bool user = flags & BPF_F_USER_STACK;
+ u32 elem_size, max_depth, nr_trace;
+ bool kernel = !user;
+
+ if (kernel && user_build_id)
+ return -EINVAL;
+
+ elem_size = user_build_id ? sizeof(struct bpf_stack_build_id) : sizeof(u64);
+ if (unlikely(size % elem_size))
+ return -EINVAL;
+
+ max_depth = stack_map_calculate_max_depth(size, elem_size, flags);
+ trace->nr = min_t(u32, trace->nr, max_depth);
+
+ if (trace->nr < skip)
+ return -EFAULT;
+
+ nr_trace = callchain_store(trace, buf, elem_size, flags);
+ return callchain_finalize(buf, size, nr_trace, elem_size, flags, false /* !may_fault */);
+}
+
BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
void *, buf, u32, size, u64, flags)
{
@@ -809,7 +854,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
__u64 nr_kernel;
if (!(event->attr.sample_type & PERF_SAMPLE_CALLCHAIN))
- return __bpf_get_stack(regs, NULL, NULL, buf, size, flags, false /* !may_fault */);
+ return __bpf_get_stack(regs, NULL, buf, size, flags, false /* !may_fault */);
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_USER_BUILD_ID)))
@@ -829,7 +874,7 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
__u64 nr = trace->nr;
trace->nr = nr_kernel;
- err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */);
+ err = __bpf_get_stack_pe(trace, buf, size, flags);
/* restore nr */
trace->nr = nr;
@@ -839,14 +884,13 @@ BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
skip += nr_kernel;
if (skip > BPF_F_SKIP_FIELD_MASK)
goto clear;
-
flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip;
- err = __bpf_get_stack(regs, NULL, trace, buf, size, flags, false /* !may_fault */);
+ err = __bpf_get_stack_pe(trace, buf, size, flags);
}
- return err;
clear:
- memset(buf, 0, size);
+ if (err < 0)
+ memset(buf, 0, size);
return err;
}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index dec993c90dc3..144d8b1183a9 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -6396,6 +6396,7 @@ EXPORT_SYMBOL_NS(kern_sys_bpf, "BPF_INTERNAL");
static const struct bpf_func_proto bpf_sys_bpf_proto = {
.func = bpf_sys_bpf,
.gpl_only = false,
+ .might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
@@ -6421,6 +6422,7 @@ BPF_CALL_1(bpf_sys_close, u32, fd)
static const struct bpf_func_proto bpf_sys_close_proto = {
.func = bpf_sys_close,
.gpl_only = false,
+ .might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 40f19147f227..ccb943b7ac2c 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -397,6 +397,13 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, int size)
return ERR_PTR(err);
}
+void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags)
+{
+ mutex_lock(&tr->mutex);
+ tr->flags |= flags;
+ mutex_unlock(&tr->mutex);
+}
+
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex)
{
struct bpf_tramp_image *im;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 9247549b5a77..b89a5d717e42 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -405,9 +405,18 @@ static bool reg_not_null(const struct bpf_reg_state *reg)
if (type_may_be_null(type))
return false;
+ /*
+ * The types below guarantee a non-NULL base, an unbounded offset can
+ * still wrap base + offset to zero.
+ */
+ if (reg->smin_value <= -BPF_MAX_VAR_OFF || reg->smax_value >= BPF_MAX_VAR_OFF)
+ return false;
+
type = base_type(type);
return type == PTR_TO_SOCKET ||
type == PTR_TO_TCP_SOCK ||
+ type == PTR_TO_XDP_SOCK ||
+ type == PTR_TO_BUF ||
type == PTR_TO_MAP_VALUE ||
type == PTR_TO_MAP_KEY ||
type == PTR_TO_SOCK_COMMON ||
@@ -597,10 +606,15 @@ static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id,
return ref_obj_uses > 1;
}
+/*
+ * Matches BPF_PROBE_ATOMIC too: bpf_convert_ctx_accesses() rewrites arena
+ * atomics before bpf_opt_subreg_zext_lo32_rnd_hi32() runs.
+ */
static bool is_cmpxchg_insn(const struct bpf_insn *insn)
{
return BPF_CLASS(insn->code) == BPF_STX &&
- BPF_MODE(insn->code) == BPF_ATOMIC &&
+ (BPF_MODE(insn->code) == BPF_ATOMIC ||
+ BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) &&
insn->imm == BPF_CMPXCHG;
}
@@ -3882,6 +3896,16 @@ static bool is_jmp_point(struct bpf_verifier_env *env, int insn_idx)
return env->insn_aux_data[insn_idx].jmp_point;
}
+static void mark_jump_target(struct bpf_verifier_env *env, int idx)
+{
+ env->insn_aux_data[idx].jump_target = true;
+}
+
+static bool bpf_is_jump_target(struct bpf_verifier_env *env, int insn_idx)
+{
+ return env->insn_aux_data[insn_idx].jump_target;
+}
+
#define LR_FRAMENO_BITS 3
#define LR_SPI_BITS 6
#define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)
@@ -4492,37 +4516,34 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -EFAULT;
}
} else if (opcode == BPF_EXIT) {
- bool r0_precise;
+ bool from_subprog_call, r0_precise;
+
+ /* BPF_EXIT in subprog or callback always returns
+ * right after the call instruction, so by checking
+ * whether the instruction at subseq_idx-1 is subprog
+ * call or not we can distinguish actual exit from
+ * *subprog* from exit from *callback*. In the former
+ * case, we need to propagate r0 precision, if
+ * necessary. In the former we never do that.
+ */
+ from_subprog_call = subseq_idx - 1 >= 0 &&
+ bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+ r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
* In case of a regular function call, instructions giving
* precision to registers R1-R5 should have been found already.
- * In case of a callback, it is ok to have R1-R5 marked for
- * backtracking, as these registers are set by the function
- * invoking callback.
+ * In case of a callback from bpf_loop(), R{1,4} in the calling
+ * frame would be set as precise and that is correct.
*/
- if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
- for (i = BPF_REG_1; i <= BPF_REG_5; i++)
- bt_clear_reg(bt, i);
- if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
+ if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
verifier_bug(env, "backtracking exit unexpected regs %x",
bt_reg_mask(bt));
return -EFAULT;
}
- /* BPF_EXIT in subprog or callback always returns
- * right after the call instruction, so by checking
- * whether the instruction at subseq_idx-1 is subprog
- * call or not we can distinguish actual exit from
- * *subprog* from exit from *callback*. In the former
- * case, we need to propagate r0 precision, if
- * necessary. In the former we never do that.
- */
- r0_precise = subseq_idx - 1 >= 0 &&
- bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
- bt_is_reg_set(bt, BPF_REG_0);
-
bt_clear_reg(bt, BPF_REG_0);
if (bt_subprog_enter(bt))
return -EFAULT;
@@ -4554,16 +4575,29 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
*/
}
} else if (class == BPF_LD) {
- if (!bt_is_reg_set(bt, dreg))
- return 0;
- bt_clear_reg(bt, dreg);
/* It's ld_imm64 or ld_abs or ld_ind.
* For ld_imm64 no further tracking of precision
* into parent is necessary
*/
- if (mode == BPF_IND || mode == BPF_ABS)
- /* to be analyzed */
- return -ENOTSUPP;
+ if (mode == BPF_IMM) {
+ bt_clear_reg(bt, dreg);
+ return 0;
+ }
+ /*
+ * BPF_{IND,ABS} are modelled as two branches:
+ * - fallthrough;
+ * - implicit subprogram exit.
+ * It is necessary to switch current frame if
+ * implicit subprogram exit branch is backtracked.
+ */
+ if (mode == BPF_IND || mode == BPF_ABS) {
+ if (bt_is_reg_set(bt, dreg))
+ return -ENOTSUPP;
+ if (subseq_idx != idx + 1)
+ if (bt_subprog_enter(bt))
+ return -EFAULT;
+ return 0;
+ }
}
/* Propagate precision marks to linked registers, to account for
* registers marked as precise in this function.
@@ -5934,7 +5968,8 @@ static int map_kptr_match_type(struct bpf_verifier_env *env,
*/
if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off,
kptr_field->kptr.btf, kptr_field->kptr.btf_id,
- kptr_field->type != BPF_KPTR_UNREF))
+ kptr_field->type != BPF_KPTR_UNREF,
+ !type_is_alloc(reg->type)))
goto bad_type;
return 0;
bad_type:
@@ -6096,8 +6131,15 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,
return ret;
} else if (class == BPF_STX) {
val_reg = reg_state(env, value_regno);
- if (!register_is_null(val_reg) &&
- map_kptr_match_type(env, kptr_field, val_reg, value_regno))
+ if (register_is_null(val_reg)) {
+ /*
+ * This store is valid only because the scalar is known to be
+ * zero. Mark it precise so another scalar cannot be pruned
+ * against this state.
+ */
+ return mark_chain_precision(env, value_regno);
+ }
+ if (map_kptr_match_type(env, kptr_field, val_reg, value_regno))
return -EACCES;
} else if (class == BPF_ST) {
if (insn->imm) {
@@ -7372,6 +7414,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
return -EACCES;
}
+ if (atype != BPF_READ && (type_flag(reg->type) & PTR_UNTRUSTED)) {
+ verbose(env, "only read is supported\n");
+ return -EACCES;
+ }
+
if (env->ops->btf_struct_access && !type_is_alloc(reg->type) && atype == BPF_WRITE) {
if (!btf_is_kernel(reg->btf)) {
verifier_bug(env, "reg->btf must be kernel btf");
@@ -7380,8 +7427,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,
ret = env->ops->btf_struct_access(&env->log, reg, off, size);
} else {
/* Writes are permitted with default btf_struct_access for
- * program allocated objects (which always have ref_obj_id > 0),
- * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC.
+ * program allocated objects (which always have ref_obj_id > 0).
*/
if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) {
verbose(env, "only read is supported\n");
@@ -8036,11 +8082,9 @@ static int check_atomic_rmw(struct bpf_verifier_env *env,
if (err)
return err;
- if (is_arena_reg(env, insn->dst_reg)) {
- err = save_aux_ptr_type(env, PTR_TO_ARENA, false);
- if (err)
- return err;
- }
+ err = save_aux_ptr_type(env, reg_state(env, insn->dst_reg)->type, false);
+ if (err)
+ return err;
/* Check whether we can write into the same memory. */
err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_WRITE, -1, true, false);
@@ -8587,6 +8631,10 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags)
return err;
}
} else {
+ u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER);
+ struct bpf_func_state *state;
+ struct bpf_reg_state *rcu_reg;
+ bool was_in_rcu_cs;
void *ptr;
int type;
@@ -8614,10 +8662,19 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags)
verbose(env, "%s_unlock cannot be out of order\n", lock_str);
return -EINVAL;
}
+ was_in_rcu_cs = in_rcu_cs(env);
if (release_lock_state(cur, type, reg->id, ptr)) {
verbose(env, "%s_unlock of different lock\n", lock_str);
return -EINVAL;
}
+ if (was_in_rcu_cs && !in_rcu_cs(env)) {
+ bpf_for_each_reg_in_vstate_mask(env->cur_state, state, rcu_reg, clear_mask, ({
+ if (rcu_reg->type & MEM_RCU) {
+ rcu_reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL);
+ rcu_reg->type |= PTR_UNTRUSTED;
+ }
+ }));
+ }
invalidate_non_owning_refs(env);
}
@@ -9514,7 +9571,8 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off,
btf_vmlinux, *arg_btf_id,
- strict_type_match)) {
+ strict_type_match,
+ !type_is_alloc(reg->type))) {
verbose(env, "R%d is of type %s but %s is expected\n",
regno, btf_type_name(reg->btf, reg->btf_id),
btf_type_name(btf_vmlinux, *arg_btf_id));
@@ -9854,11 +9912,15 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
return err;
}
- if (register_is_null(reg) && type_may_be_null(arg_type))
+ if (register_is_null(reg) && type_may_be_null(arg_type)) {
/* A NULL register has a SCALAR_VALUE type, so skip
* type checking.
*/
+ err = mark_chain_precision(env, regno);
+ if (err)
+ return err;
goto skip_type_check;
+ }
/* arg_btf_id and arg_size are in a union. */
if (base_type(arg_type) == ARG_PTR_TO_BTF_ID ||
@@ -10914,10 +10976,12 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env,
callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY;
__mark_reg_known_zero(&callee->regs[BPF_REG_2]);
callee->regs[BPF_REG_2].map_ptr = caller->regs[BPF_REG_1].map_ptr;
+ callee->regs[BPF_REG_2].map_uid = caller->regs[BPF_REG_1].map_uid;
callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE;
__mark_reg_known_zero(&callee->regs[BPF_REG_3]);
callee->regs[BPF_REG_3].map_ptr = caller->regs[BPF_REG_1].map_ptr;
+ callee->regs[BPF_REG_3].map_uid = caller->regs[BPF_REG_1].map_uid;
/* pointer to stack or null */
callee->regs[BPF_REG_4] = caller->regs[BPF_REG_3];
@@ -10995,6 +11059,7 @@ static int set_timer_callback_state(struct bpf_verifier_env *env,
int insn_idx)
{
struct bpf_map *map_ptr = caller->regs[BPF_REG_1].map_ptr;
+ u32 map_uid = caller->regs[BPF_REG_1].map_uid;
/* bpf_timer_set_callback(struct bpf_timer *timer, void *callback_fn);
* callback_fn(struct bpf_map *map, void *key, void *value);
@@ -11002,14 +11067,17 @@ static int set_timer_callback_state(struct bpf_verifier_env *env,
callee->regs[BPF_REG_1].type = CONST_PTR_TO_MAP;
__mark_reg_known_zero(&callee->regs[BPF_REG_1]);
callee->regs[BPF_REG_1].map_ptr = map_ptr;
+ callee->regs[BPF_REG_1].map_uid = map_uid;
callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY;
__mark_reg_known_zero(&callee->regs[BPF_REG_2]);
callee->regs[BPF_REG_2].map_ptr = map_ptr;
+ callee->regs[BPF_REG_2].map_uid = map_uid;
callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE;
__mark_reg_known_zero(&callee->regs[BPF_REG_3]);
callee->regs[BPF_REG_3].map_ptr = map_ptr;
+ callee->regs[BPF_REG_3].map_uid = map_uid;
/* unused */
__mark_reg_not_init(env, &callee->regs[BPF_REG_4]);
@@ -11108,6 +11176,7 @@ static int set_task_work_schedule_callback_state(struct bpf_verifier_env *env,
int insn_idx)
{
struct bpf_map *map_ptr = caller->regs[BPF_REG_3].map_ptr;
+ u32 map_uid = caller->regs[BPF_REG_3].map_uid;
/*
* callback_fn(struct bpf_map *map, void *key, void *value);
@@ -11115,14 +11184,17 @@ static int set_task_work_schedule_callback_state(struct bpf_verifier_env *env,
callee->regs[BPF_REG_1].type = CONST_PTR_TO_MAP;
__mark_reg_known_zero(&callee->regs[BPF_REG_1]);
callee->regs[BPF_REG_1].map_ptr = map_ptr;
+ callee->regs[BPF_REG_1].map_uid = map_uid;
callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY;
__mark_reg_known_zero(&callee->regs[BPF_REG_2]);
callee->regs[BPF_REG_2].map_ptr = map_ptr;
+ callee->regs[BPF_REG_2].map_uid = map_uid;
callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE;
__mark_reg_known_zero(&callee->regs[BPF_REG_3]);
callee->regs[BPF_REG_3].map_ptr = map_ptr;
+ callee->regs[BPF_REG_3].map_uid = map_uid;
/* unused */
__mark_reg_not_init(env, &callee->regs[BPF_REG_4]);
@@ -11794,6 +11866,9 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
verbose(env, "get_local_storage() doesn't support non-zero flags\n");
return -EINVAL;
}
+ err = mark_chain_precision(env, BPF_REG_2);
+ if (err)
+ return err;
break;
case BPF_FUNC_for_each_map_elem:
err = push_callback_call(env, insn, insn_idx, meta.subprogno,
@@ -12814,7 +12889,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,
reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, ®_ref_id);
reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off);
- struct_same = btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match);
+ struct_same = btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match,
+ !type_is_alloc(reg->type));
/* If kfunc is accepting a projection type (ie. __sk_buff), it cannot
* actually use it -- it must cast to the underlying type. So we allow
* caller to pass in the underlying type.
@@ -13251,7 +13327,8 @@ __process_kf_arg_ptr_to_graph_node(struct bpf_verifier_env *env,
et = btf_type_by_id(field->graph_root.btf, field->graph_root.value_btf_id);
t = btf_type_by_id(reg->btf, reg->btf_id);
if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, 0, field->graph_root.btf,
- field->graph_root.value_btf_id, true)) {
+ field->graph_root.value_btf_id, true,
+ !type_is_alloc(reg->type))) {
verbose(env, "operation on %s expects arg#1 %s at offset=%d "
"in struct %s, but arg is at offset=%d in struct %s\n",
btf_field_type_name(head_field_type),
@@ -13399,6 +13476,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
}
meta->r0_size = reg->var_off.value;
+ if (meta->r0_size > U32_MAX) {
+ verbose(env, "R%d rdonly/rdwr_buf_size exceeds u32 max\n",
+ regno);
+ return -EINVAL;
+ }
ret = mark_chain_precision(env, regno);
if (ret)
return ret;
@@ -13764,7 +13846,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
verbose(env, "arg#%d is neither owning or non-owning ref\n", i);
return -EINVAL;
}
- if (!type_is_non_owning_ref(reg->type))
+ if (!type_is_non_owning_ref(reg->type) && reg->ref_obj_id)
meta->arg_owning_ref = true;
rec = reg_btf_record(reg);
@@ -17348,7 +17430,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
*/
if (!is_jmp32 && BPF_SRC(insn->code) == BPF_X &&
__is_pointer_value(false, src_reg) && __is_pointer_value(false, dst_reg) &&
- type_may_be_null(src_reg->type) != type_may_be_null(dst_reg->type) &&
base_type(src_reg->type) != PTR_TO_BTF_ID &&
base_type(dst_reg->type) != PTR_TO_BTF_ID) {
eq_branch_regs = NULL;
@@ -17364,9 +17445,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
break;
}
if (eq_branch_regs) {
- if (type_may_be_null(src_reg->type))
+ /* src == dst && dst != NULL => src != NULL */
+ if (reg_not_null(dst_reg) && type_may_be_null(src_reg->type))
mark_ptr_not_null_reg(&eq_branch_regs[insn->src_reg]);
- else
+ /* src == dst && src != NULL => dst != NULL */
+ if (reg_not_null(src_reg) && type_may_be_null(dst_reg->type))
mark_ptr_not_null_reg(&eq_branch_regs[insn->dst_reg]);
}
}
@@ -17957,6 +18040,7 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
/* mark branch target for state pruning */
mark_prune_point(env, w);
mark_jmp_point(env, w);
+ mark_jump_target(env, w);
}
if (insn_state[w] == 0) {
@@ -18151,6 +18235,10 @@ static bool get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call
* r0 = *(u64 *)(r10 - 8); r0 += r1;
* r0 += r1; exit;
* exit;
+ *
+ * Both uses of the marks assume that a pattern is entered at its first
+ * spill and thus executes as a unit, hence a pattern is not grown past
+ * an instruction targeted by a jump.
*/
static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,
struct bpf_subprog_info *subprog,
@@ -18189,6 +18277,10 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,
for (i = 1, off = lowest_off; i <= ARRAY_SIZE(caller_saved); ++i, off += BPF_REG_SIZE) {
if (insn_idx - i < 0 || insn_idx + i >= env->prog->len)
break;
+ /* stx/ldx/call must not be a jump targets, a jump to the first stx is fine */
+ if (bpf_is_jump_target(env, insn_idx - i + 1) ||
+ bpf_is_jump_target(env, insn_idx + i))
+ break;
stx = &insns[insn_idx - i];
ldx = &insns[insn_idx + i];
/* must be a stack spill/fill pair */
@@ -18374,6 +18466,7 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
mark_prune_point(env, t + off + 1);
mark_jmp_point(env, t + off + 1);
+ mark_jump_target(env, t + off + 1);
return ret;
@@ -19490,7 +19583,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* infinite loop check triggering, see
* iter_active_depths_differ()
*/
- if (old_reg->iter.btf != cur_reg->iter.btf ||
+ if (old_reg->type != cur_reg->type ||
+ old_reg->iter.btf != cur_reg->iter.btf ||
old_reg->iter.btf_id != cur_reg->iter.btf_id ||
old_reg->iter.state != cur_reg->iter.state ||
/* ignore {old_reg,cur_reg}->iter.depth, see above */
@@ -24089,6 +24183,17 @@ static int check_non_sleepable_error_inject(u32 btf_id)
return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id);
}
+static bool attach_uses_trampoline_retval(enum bpf_attach_type type)
+{
+ switch (type) {
+ case BPF_MODIFY_RETURN:
+ case BPF_TRACE_FEXIT:
+ return true;
+ default:
+ return false;
+ }
+}
+
int bpf_check_attach_target(struct bpf_verifier_log *log,
const struct bpf_prog *prog,
const struct bpf_prog *tgt_prog,
@@ -24333,6 +24438,14 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
if (ret < 0)
return ret;
+ if (tgt_info->fmodel.ret_size > 8 &&
+ attach_uses_trampoline_retval(prog->expected_attach_type)) {
+ bpf_log(log,
+ "Attach to function %s with a >8 byte return value is not supported for this attach type\n",
+ tname);
+ return -EOPNOTSUPP;
+ }
+
if (tgt_prog) {
if (subprog == 0)
addr = (long) tgt_prog->bpf_func;
@@ -24479,8 +24592,10 @@ static bool can_be_sleepable(struct bpf_prog *prog)
return false;
}
}
- return prog->type == BPF_PROG_TYPE_LSM ||
- prog->type == BPF_PROG_TYPE_KPROBE /* only for uprobes */ ||
+ if (prog->type == BPF_PROG_TYPE_LSM)
+ return prog->expected_attach_type != BPF_LSM_CGROUP;
+
+ return prog->type == BPF_PROG_TYPE_KPROBE /* only for uprobes */ ||
prog->type == BPF_PROG_TYPE_STRUCT_OPS;
}
@@ -24568,7 +24683,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -ENOMEM;
if (tgt_prog && tgt_prog->aux->tail_call_reachable)
- tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX;
+ bpf_trampoline_set_flags(tr, BPF_TRAMP_F_TAIL_CALL_CTX);
prog->aux->dst_trampoline = tr;
return 0;
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index eebbede82070..15b6c9a6402a 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -429,10 +429,26 @@ static void guarantee_active_cpus(struct task_struct *tsk,
rcu_read_lock();
cs = task_cs(tsk);
- while (!cpumask_intersects(cs->effective_cpus, pmask))
+ while (!cpumask_intersects(cs->effective_cpus, pmask)) {
cs = parent_cs(cs);
-
+ if (unlikely(!cs)) {
+ /*
+ * The top cpuset doesn't have any active cpu as a
+ * consequence of a race between its caller and the cpu
+ * hotplug operation where cpu_active_mask is updated
+ * asynchronously before cpuset_handle_hotplug() is
+ * being called to adjust the effective_cpus of the
+ * affected cpusets. But we know the top cpuset's
+ * effective_cpus is on its way to be identical to
+ * cpu_active_mask minus the exclusive CPUs dedicated
+ * to other valid cpuset partitions. Just pass back
+ * the filtered cpu_active_mask in this case.
+ */
+ goto out_unlock;
+ }
+ }
cpumask_and(pmask, pmask, cs->effective_cpus);
+out_unlock:
rcu_read_unlock();
}
@@ -4097,15 +4113,16 @@ static void cpuset_handle_hotplug(void)
static cpumask_t new_cpus;
static nodemask_t new_mems;
bool cpus_updated, mems_updated;
- bool on_dfl = is_in_v2_mode();
+ bool on_dfl;
struct tmpmasks tmp, *ptmp = NULL;
- if (on_dfl && !alloc_tmpmasks(&tmp))
- ptmp = &tmp;
-
lockdep_assert_cpus_held();
mutex_lock(&cpuset_mutex);
+ on_dfl = is_in_v2_mode();
+ if (on_dfl && !alloc_tmpmasks(&tmp))
+ ptmp = &tmp;
+
/* fetch the available cpus/mems and find out which changed how */
cpumask_copy(&new_cpus, cpu_active_mask);
new_mems = node_states[N_MEMORY];
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 4593064c2ada..8e4732545f1c 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -1677,10 +1677,10 @@ phys_addr_t default_swiotlb_base(void)
phys_addr_t default_swiotlb_limit(void)
{
#ifdef CONFIG_SWIOTLB_DYNAMIC
- return io_tlb_default_mem.phys_limit;
-#else
- return io_tlb_default_mem.defpool.end - 1;
+ if (io_tlb_default_mem.can_grow)
+ return io_tlb_default_mem.phys_limit;
#endif
+ return io_tlb_default_mem.defpool.end - 1;
}
#ifdef CONFIG_DEBUG_FS
diff --git a/kernel/kexec_handover.c b/kernel/kexec_handover.c
index e1a06b4fc17f..ac9fa7727b5f 100644
--- a/kernel/kexec_handover.c
+++ b/kernel/kexec_handover.c
@@ -425,7 +425,7 @@ static void __init deserialize_bitmap(unsigned int order,
unsigned long bit;
for_each_set_bit(bit, bitmap->preserve, PRESERVE_BITS) {
- int sz = 1 << (order + PAGE_SHIFT);
+ u64 sz = 1UL << (order + PAGE_SHIFT);
phys_addr_t phys =
elm->phys_start + (bit << (order + PAGE_SHIFT));
struct page *page = phys_to_page(phys);
diff --git a/kernel/printk/nbcon.c b/kernel/printk/nbcon.c
index 9c1e8ec04c55..5ee30dffa810 100644
--- a/kernel/printk/nbcon.c
+++ b/kernel/printk/nbcon.c
@@ -1340,7 +1340,7 @@ bool nbcon_kthread_create(struct console *con)
return true;
kt = kthread_run(nbcon_kthread_func, con, "pr/%s%d", con->name, con->index);
- if (WARN_ON(IS_ERR(kt))) {
+ if (IS_ERR(kt)) {
con_printk(KERN_ERR, con, "failed to start printing thread\n");
return false;
}
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index cc3d57be447d..dffdbbfa4a00 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -3689,7 +3689,7 @@ static bool legacy_kthread_create(void)
lockdep_assert_console_list_lock_held();
kt = kthread_run(legacy_kthread_func, NULL, "pr/legacy");
- if (WARN_ON(IS_ERR(kt))) {
+ if (IS_ERR(kt)) {
pr_err("failed to start legacy printing thread\n");
return false;
}
diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c
index e3b64a5e0ec7..d9c11d5f0ea4 100644
--- a/kernel/rcu/srcutiny.c
+++ b/kernel/rcu/srcutiny.c
@@ -9,6 +9,7 @@
*/
#include <linux/export.h>
+#include <linux/irq_work.h>
#include <linux/mutex.h>
#include <linux/preempt.h>
#include <linux/rcupdate_wait.h>
@@ -41,6 +42,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp)
ssp->srcu_idx_max = 0;
INIT_WORK(&ssp->srcu_work, srcu_drive_gp);
INIT_LIST_HEAD(&ssp->srcu_work.entry);
+ init_irq_work(&ssp->srcu_irq_work, srcu_tiny_irq_work);
return 0;
}
@@ -84,6 +86,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct);
void cleanup_srcu_struct(struct srcu_struct *ssp)
{
WARN_ON(ssp->srcu_lock_nesting[0] || ssp->srcu_lock_nesting[1]);
+ irq_work_sync(&ssp->srcu_irq_work);
flush_work(&ssp->srcu_work);
WARN_ON(ssp->srcu_gp_running);
WARN_ON(ssp->srcu_gp_waiting);
@@ -172,6 +175,20 @@ void srcu_drive_gp(struct work_struct *wp)
}
EXPORT_SYMBOL_GPL(srcu_drive_gp);
+/*
+ * Use an irq_work to defer schedule_work() to avoid acquiring the workqueue
+ * pool->lock while the caller might hold scheduler locks, causing lockdep
+ * splats due to workqueue_init() doing a wakeup.
+ */
+void srcu_tiny_irq_work(struct irq_work *irq_work)
+{
+ struct srcu_struct *ssp;
+
+ ssp = container_of(irq_work, struct srcu_struct, srcu_irq_work);
+ schedule_work(&ssp->srcu_work);
+}
+EXPORT_SYMBOL_GPL(srcu_tiny_irq_work);
+
static void srcu_gp_start_if_needed(struct srcu_struct *ssp)
{
unsigned long cookie;
@@ -184,7 +201,7 @@ static void srcu_gp_start_if_needed(struct srcu_struct *ssp)
WRITE_ONCE(ssp->srcu_idx_max, cookie);
if (!READ_ONCE(ssp->srcu_gp_running)) {
if (likely(srcu_init_done))
- schedule_work(&ssp->srcu_work);
+ irq_work_queue(&ssp->srcu_irq_work);
else if (list_empty(&ssp->srcu_work.entry))
list_add(&ssp->srcu_work.entry, &srcu_boot_list);
}
diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 059675018c71..189a4e0658eb 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -1300,8 +1300,14 @@ static void do_enqueue_task(struct rq *rq, struct task_struct *p, u64 enq_flags,
if (unlikely(!SCX_HAS_OP(sch, enqueue)))
goto global;
- /* DSQ bypass didn't trigger, enqueue on the BPF scheduler */
- qseq = rq->scx.ops_qseq++ << SCX_OPSS_QSEQ_SHIFT;
+ /*
+ * DSQ bypass didn't trigger, enqueue on the BPF scheduler. Wrap the
+ * per-task qseq counter where the QSEQ field wraps and skip 0, which is
+ * what scx_bpf_dsq_insert() records for a task in NONE or DISPATCHING.
+ */
+ p->scx.ops_qseq = ((p->scx.ops_qseq + 1) &
+ (SCX_OPSS_QSEQ_MASK >> SCX_OPSS_QSEQ_SHIFT)) ?: 1;
+ qseq = (unsigned long)p->scx.ops_qseq << SCX_OPSS_QSEQ_SHIFT;
WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) != SCX_OPSS_NONE);
atomic_long_set(&p->scx.ops_state, SCX_OPSS_QUEUEING | qseq);
@@ -4392,10 +4398,9 @@ static void scx_dump_state(struct scx_exit_info *ei, size_t dump_len)
seq_buf_init(&ns, buf, avail);
dump_newline(&ns);
- dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d ops_qseq=%lu pnt_seq=%lu",
+ dump_line(&ns, "CPU %-4d: nr_run=%u flags=0x%x cpu_rel=%d pnt_seq=%lu",
cpu, rq->scx.nr_running, rq->scx.flags,
- rq->scx.cpu_released, rq->scx.ops_qseq,
- rq->scx.pnt_seq);
+ rq->scx.cpu_released, rq->scx.pnt_seq);
dump_line(&ns, " curr=%s[%d] class=%ps",
rq->curr->comm, rq->curr->pid,
rq->curr->sched_class);
diff --git a/kernel/sched/ext_internal.h b/kernel/sched/ext_internal.h
index 5b2dd105fa92..7cb9c8725b65 100644
--- a/kernel/sched/ext_internal.h
+++ b/kernel/sched/ext_internal.h
@@ -1057,6 +1057,11 @@ enum scx_ops_state {
* dequeue/requeue, the dispatcher can tell whether it still has a claim
* on the task being dispatched.
*
+ * QSEQ is generated from the per-task p->scx.ops_qseq counter so that
+ * it doesn't repeat across QUEUED instances of the same task even if
+ * the task moves between rqs. 0 is never used as a valid QSEQ since
+ * NONE and DISPATCHING map to this value.
+ *
* As some 32bit archs can't do 64bit store_release/load_acquire,
* p->scx.ops_state is atomic_long_t which leaves 30 bits for QSEQ on
* 32bit machines. The dispatch race window QSEQ protects is very narrow
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 97dddc148d71..4e58174d3d0b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -795,7 +795,6 @@ struct scx_rq {
struct scx_dispatch_q local_dsq;
struct list_head runnable_list; /* runnable tasks on this rq */
struct list_head ddsp_deferred_locals; /* deferred ddsps from enq */
- unsigned long ops_qseq;
u64 extra_enq_flags; /* see move_task_to_local_dsq() */
u32 nr_running;
u32 cpuperf_target; /* [0, SCHED_CAPACITY_SCALE] */
diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index 6941145b5058..da7303a5a796 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -251,8 +251,6 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
cpu = raw_smp_processor_id();
if (blk_tracer) {
- tracing_record_cmdline(current);
-
buffer = blk_tr->array_buffer.buffer;
trace_ctx = tracing_gen_ctx_flags(0);
event = trace_buffer_lock_reserve(buffer, TRACE_BLK,
@@ -260,6 +258,8 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
trace_ctx);
if (!event)
return;
+
+ tracing_record_cmdline(current);
t = ring_buffer_event_data(event);
goto record_it;
}
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 008f6c407aac..c6d3730eb6f3 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -351,6 +351,11 @@ struct buffer_data_read_page {
struct buffer_data_page *data; /* actual data, stored in this page */
};
+static __always_inline unsigned int rb_read_page_capacity(struct buffer_data_read_page *rpage)
+{
+ return (PAGE_SIZE << rpage->order) - BUF_PAGE_HDR_SIZE;
+}
+
/*
* Note, the buffer_page list must be first. The buffer pages
* are allocated in cache lines, which means that each buffer
@@ -407,6 +412,12 @@ static __always_inline unsigned int rb_page_capacity(struct buffer_page *bpage)
return (PAGE_SIZE << bpage->order) - BUF_PAGE_HDR_SIZE;
}
+/* Size is determined by what has been committed */
+static __always_inline unsigned int rb_page_size(struct buffer_page *bpage)
+{
+ return rb_page_commit(bpage) & ~RB_MISSED_MASK;
+}
+
static void free_buffer_page(struct buffer_page *bpage)
{
/* Range pages are not to be freed */
@@ -1831,7 +1842,6 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
unsigned long *subbuf_mask)
{
int subbuf_size = PAGE_SIZE;
- struct buffer_data_page *subbuf;
unsigned long buffers_start;
unsigned long buffers_end;
int i;
@@ -1839,6 +1849,16 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
if (!subbuf_mask)
return false;
+ if (meta->subbuf_size != PAGE_SIZE) {
+ pr_info("Ring buffer boot meta [%d] invalid subbuf_size\n", cpu);
+ return false;
+ }
+
+ if (meta->nr_subbufs != nr_pages + 1) {
+ pr_info("Ring buffer boot meta [%d] invalid nr_subbufs\n", cpu);
+ return false;
+ }
+
buffers_start = meta->first_buffer;
buffers_end = meta->first_buffer + (subbuf_size * meta->nr_subbufs);
@@ -1855,11 +1875,12 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
return false;
}
- subbuf = rb_subbufs_from_meta(meta);
-
bitmap_clear(subbuf_mask, 0, meta->nr_subbufs);
- /* Is the meta buffers and the subbufs themselves have correct data? */
+ /*
+ * Ensure the meta::buffers array has correct data. The data in each subbufs
+ * are checked later in rb_meta_validate_events().
+ */
for (i = 0; i < meta->nr_subbufs; i++) {
if (meta->buffers[i] < 0 ||
meta->buffers[i] >= meta->nr_subbufs) {
@@ -1867,18 +1888,12 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
return false;
}
- if ((unsigned)local_read(&subbuf->commit) > subbuf_size) {
- pr_info("Ring buffer boot meta [%d] buffer invalid commit\n", cpu);
- return false;
- }
-
if (test_bit(meta->buffers[i], subbuf_mask)) {
pr_info("Ring buffer boot meta [%d] array has duplicates\n", cpu);
return false;
}
set_bit(meta->buffers[i], subbuf_mask);
- subbuf = (void *)subbuf + subbuf_size;
}
return true;
@@ -1942,13 +1957,22 @@ static int rb_read_data_buffer(struct buffer_data_page *dpage, int tail, int cpu
return events;
}
-static int rb_validate_buffer(struct buffer_data_page *dpage, int cpu)
+static int rb_validate_buffer(struct buffer_data_page *dpage, int cpu,
+ struct ring_buffer_cpu_meta *meta)
{
unsigned long long ts;
+ unsigned long tail;
u64 delta;
- int tail;
- tail = local_read(&dpage->commit);
+ /*
+ * When a sub-buffer is recovered from a read, the commit value may
+ * have RB_MISSED_* bits set, as these bits are reset on reuse.
+ * Even after clearing these bits, a commit value greater than the
+ * subbuf_size is considered invalid.
+ */
+ tail = local_read(&dpage->commit) & ~RB_MISSED_MASK;
+ if (tail > meta->subbuf_size - BUF_PAGE_HDR_SIZE)
+ return -1;
return rb_read_data_buffer(dpage, tail, cpu, &ts, &delta);
}
@@ -1959,6 +1983,7 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
struct buffer_page *head_page, *orig_head, *orig_reader;
unsigned long entry_bytes = 0;
unsigned long entries = 0;
+ int discarded = 0;
int ret;
u64 ts;
int i;
@@ -1970,14 +1995,19 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
orig_reader = cpu_buffer->reader_page;
/* Do the reader page first */
- ret = rb_validate_buffer(orig_reader->page, cpu_buffer->cpu);
+ ret = rb_validate_buffer(orig_reader->page, cpu_buffer->cpu, meta);
if (ret < 0) {
- pr_info("Ring buffer reader page is invalid\n");
- goto invalid;
+ pr_info("Ring buffer meta [%d] invalid reader page detected\n",
+ cpu_buffer->cpu);
+ discarded++;
+ /* Instead of discard whole ring buffer, discard only this sub-buffer. */
+ local_set(&orig_reader->entries, 0);
+ local_set(&orig_reader->page->commit, 0);
+ } else {
+ entries += ret;
+ entry_bytes += rb_page_size(orig_reader);
+ local_set(&orig_reader->entries, ret);
}
- entries += ret;
- entry_bytes += local_read(&orig_reader->page->commit);
- local_set(&orig_reader->entries, ret);
ts = head_page->page->time_stamp;
@@ -2005,7 +2035,7 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
break;
/* Stop rewind if the page is invalid. */
- ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu);
+ ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu, meta);
if (ret < 0)
break;
@@ -2014,7 +2044,7 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
if (ret)
local_inc(&cpu_buffer->pages_touched);
entries += ret;
- entry_bytes += rb_page_commit(head_page);
+ entry_bytes += rb_page_size(head_page);
}
if (i)
pr_info("Ring buffer [%d] rewound %d pages\n", cpu_buffer->cpu, i);
@@ -2084,21 +2114,24 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
if (head_page == orig_reader)
continue;
- ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu);
+ ret = rb_validate_buffer(head_page->page, cpu_buffer->cpu, meta);
if (ret < 0) {
- pr_info("Ring buffer meta [%d] invalid buffer page\n",
- cpu_buffer->cpu);
- goto invalid;
- }
-
- /* If the buffer has content, update pages_touched */
- if (ret)
- local_inc(&cpu_buffer->pages_touched);
-
- entries += ret;
- entry_bytes += local_read(&head_page->page->commit);
- local_set(&head_page->entries, ret);
+ if (!discarded)
+ pr_info("Ring buffer meta [%d] invalid buffer page detected\n",
+ cpu_buffer->cpu);
+ discarded++;
+ /* Instead of discard whole ring buffer, discard only this sub-buffer. */
+ local_set(&head_page->entries, 0);
+ local_set(&head_page->page->commit, 0);
+ } else {
+ /* If the buffer has content, update pages_touched */
+ if (ret)
+ local_inc(&cpu_buffer->pages_touched);
+ entries += ret;
+ entry_bytes += rb_page_size(head_page);
+ local_set(&head_page->entries, ret);
+ }
if (head_page == cpu_buffer->commit_page)
break;
}
@@ -2112,7 +2145,10 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
local_set(&cpu_buffer->entries, entries);
local_set(&cpu_buffer->entries_bytes, entry_bytes);
- pr_info("Ring buffer meta [%d] is from previous boot!\n", cpu_buffer->cpu);
+ pr_info("Ring buffer meta [%d] is from previous boot!", cpu_buffer->cpu);
+ if (discarded)
+ pr_cont(" (%d pages discarded)", discarded);
+ pr_cont("\n");
return;
invalid:
@@ -3298,12 +3334,6 @@ rb_iter_head_event(struct ring_buffer_iter *iter)
return NULL;
}
-/* Size is determined by what has been committed */
-static __always_inline unsigned rb_page_size(struct buffer_page *bpage)
-{
- return rb_page_commit(bpage) & ~RB_MISSED_MASK;
-}
-
static __always_inline unsigned
rb_commit_index(struct ring_buffer_per_cpu *cpu_buffer)
{
@@ -5475,6 +5505,7 @@ rb_update_iter_read_stamp(struct ring_buffer_iter *iter,
static struct buffer_page *
rb_get_reader_page(struct ring_buffer_per_cpu *cpu_buffer)
{
+ int max_loops = cpu_buffer->ring_meta ? cpu_buffer->nr_pages : 3;
struct buffer_page *reader = NULL;
unsigned long overwrite;
unsigned long flags;
@@ -5487,11 +5518,14 @@ rb_get_reader_page(struct ring_buffer_per_cpu *cpu_buffer)
again:
/*
* This should normally only loop twice. But because the
- * start of the reader inserts an empty page, it causes
- * a case where we will loop three times. There should be no
- * reason to loop four times (that I know of).
+ * start of the reader inserts an empty page, it causes a
+ * case where we will loop three times. There should be no
+ * reason to loop four times unless the ring buffer is a
+ * recovered persistent ring buffer. For persistent ring buffers,
+ * invalid pages are reset during recovery, so there may be more
+ * than 3 contiguous pages can be empty, but less than nr_pages.
*/
- if (RB_WARN_ON(cpu_buffer, ++nr_loops > 3)) {
+ if (RB_WARN_ON(cpu_buffer, ++nr_loops > max_loops)) {
reader = NULL;
goto out;
}
@@ -5781,12 +5815,14 @@ rb_iter_peek(struct ring_buffer_iter *iter, u64 *ts)
struct ring_buffer_per_cpu *cpu_buffer;
struct ring_buffer_event *event;
int nr_loops = 0;
+ int max_loops;
if (ts)
*ts = 0;
cpu_buffer = iter->cpu_buffer;
buffer = cpu_buffer->buffer;
+ max_loops = cpu_buffer->ring_meta ? cpu_buffer->nr_pages : 3;
/*
* Check if someone performed a consuming read to the buffer
@@ -5809,7 +5845,7 @@ rb_iter_peek(struct ring_buffer_iter *iter, u64 *ts)
* the ring buffer with an active write as the consumer is.
* Do not warn if the three failures is reached.
*/
- if (++nr_loops > 3)
+ if (++nr_loops > max_loops)
return NULL;
if (rb_per_cpu_empty(cpu_buffer))
@@ -6536,56 +6572,78 @@ EXPORT_SYMBOL_GPL(ring_buffer_swap_cpu);
* ring_buffer_alloc_read_page - allocate a page to read from buffer
* @buffer: the buffer to allocate for.
* @cpu: the cpu buffer to allocate.
+ * @rpage: pointer to pass in an already allocated page (can be NULL)
+ * and returns the allocated page.
*
- * This function is used in conjunction with ring_buffer_read_page.
+ * This function is used in conjunction with ring_buffer_read_page().
* When reading a full page from the ring buffer, these functions
* can be used to speed up the process. The calling function should
* allocate a few pages first with this function. Then when it
* needs to get pages from the ring buffer, it passes the result
- * of this function into ring_buffer_read_page, which will swap
+ * of this function into ring_buffer_read_page(), which will swap
* the page that was allocated, with the read page of the buffer.
*
+ * If @rpage is provided, and it has a different order than the current
+ * subbuffer order, its payload will be freed and re-allocated. If it
+ * already matches the order, it is simply returned.
+ *
* Returns:
- * The page allocated, or ERR_PTR
+ * 0 on success, < 0 on error
*/
-struct buffer_data_read_page *
-ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)
+int ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu,
+ struct buffer_data_read_page **rpage)
{
struct ring_buffer_per_cpu *cpu_buffer;
- struct buffer_data_read_page *bpage = NULL;
unsigned long flags;
+ unsigned int order;
if (!cpumask_test_cpu(cpu, buffer->cpumask))
- return ERR_PTR(-ENODEV);
+ return -ENODEV;
- bpage = kzalloc(sizeof(*bpage), GFP_KERNEL);
- if (!bpage)
- return ERR_PTR(-ENOMEM);
+ if (!rpage)
+ return -EINVAL;
+
+ order = READ_ONCE(buffer->subbuf_order);
+
+ if (*rpage) {
+ if ((*rpage)->order == order)
+ return 0;
- bpage->order = buffer->subbuf_order;
+ /* We can reuse rpage, but we discard the payload */
+ free_pages((unsigned long)(*rpage)->data, (*rpage)->order);
+ (*rpage)->data = NULL;
+ } else {
+ *rpage = kzalloc_obj(**rpage);
+ if (!*rpage)
+ return -ENOMEM;
+ }
+
+ (*rpage)->order = order;
cpu_buffer = buffer->buffers[cpu];
+
local_irq_save(flags);
arch_spin_lock(&cpu_buffer->lock);
if (cpu_buffer->free_page.data) {
- *bpage = cpu_buffer->free_page;
+ **rpage = cpu_buffer->free_page;
cpu_buffer->free_page.data = NULL;
}
arch_spin_unlock(&cpu_buffer->lock);
local_irq_restore(flags);
- if (bpage->data) {
- rb_init_page(bpage->data);
+ if ((*rpage)->data) {
+ rb_init_page((*rpage)->data);
} else {
- bpage->data = alloc_cpu_data(cpu, bpage->order);
- if (!bpage->data) {
- kfree(bpage);
- return ERR_PTR(-ENOMEM);
+ (*rpage)->data = alloc_cpu_data(cpu, (*rpage)->order);
+ if (!(*rpage)->data) {
+ kfree(*rpage);
+ *rpage = NULL;
+ return -ENOMEM;
}
}
- return bpage;
+ return 0;
}
EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);
@@ -6593,21 +6651,30 @@ EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);
* ring_buffer_free_read_page - free an allocated read page
* @buffer: the buffer the page was allocate for
* @cpu: the cpu buffer the page came from
- * @data_page: the page to free
+ * @rpage: the buffer_data_read_page to free
*
* Free a page allocated from ring_buffer_alloc_read_page.
*/
void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
- struct buffer_data_read_page *data_page)
+ struct buffer_data_read_page *rpage)
{
struct ring_buffer_per_cpu *cpu_buffer;
- struct buffer_data_page *bpage = data_page->data;
- struct page *page = virt_to_page(bpage);
+ struct buffer_data_page *dpage;
unsigned long flags;
+ struct page *page;
if (!buffer || !buffer->buffers || !buffer->buffers[cpu])
return;
+ if (!rpage)
+ return;
+
+ dpage = rpage->data;
+ if (!dpage)
+ goto out;
+
+ page = virt_to_page(dpage);
+
cpu_buffer = buffer->buffers[cpu];
/*
@@ -6615,23 +6682,23 @@ void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
* is different from the subbuffer order of the buffer -
* we can't reuse it
*/
- if (page_ref_count(page) > 1 || data_page->order != buffer->subbuf_order)
+ if (page_ref_count(page) > 1 || rpage->order != READ_ONCE(buffer->subbuf_order))
goto out;
local_irq_save(flags);
arch_spin_lock(&cpu_buffer->lock);
if (!cpu_buffer->free_page.data) {
- cpu_buffer->free_page = *data_page;
- bpage = NULL;
+ cpu_buffer->free_page = *rpage;
+ dpage = NULL;
}
arch_spin_unlock(&cpu_buffer->lock);
local_irq_restore(flags);
out:
- free_pages((unsigned long)bpage, data_page->order);
- kfree(data_page);
+ free_pages((unsigned long)dpage, rpage->order);
+ kfree(rpage);
}
EXPORT_SYMBOL_GPL(ring_buffer_free_read_page);
@@ -6702,10 +6769,9 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
if (!bpage)
return -1;
- guard(raw_spinlock_irqsave)(&cpu_buffer->reader_lock);
+ len = min_t(size_t, len, rb_read_page_capacity(data_page));
- if (data_page->order != cpu_buffer->reader_page->order)
- return -1;
+ guard(raw_spinlock_irqsave)(&cpu_buffer->reader_lock);
reader = rb_get_reader_page(cpu_buffer);
if (!reader)
@@ -6720,16 +6786,18 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
/* Check if any events were dropped */
missed_events = cpu_buffer->lost_events;
- /*
- * If this page has been partially read or
- * if len is not big enough to read the rest of the page or
- * a writer is still on the page, then
- * we must copy the data from the page to the buffer.
- * Otherwise, we can simply swap the page with the one passed in.
- */
+ /*
+ * It is not possible to swap the reader page if:
+ * - It has been partially read
+ * - len is not big enough to read it entirely
+ * - A writer is still on it
+ * - The ring buffer is static
+ * - The order doesn't match
+ */
if (read || (len < (size - read)) ||
cpu_buffer->reader_page == cpu_buffer->commit_page ||
- cpu_buffer->mapped) {
+ cpu_buffer->mapped ||
+ data_page->order != reader->order) {
struct buffer_data_page *rpage = cpu_buffer->reader_page->page;
unsigned int rpos = read;
unsigned int pos = 0;
@@ -6821,7 +6889,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
/* If there is room at the end of the page to save the
* missed events, then record it there.
*/
- if (rb_page_capacity(reader) - size >= sizeof(missed_events)) {
+ if (rb_read_page_capacity(data_page) - size >= sizeof(missed_events)) {
memcpy(&bpage->data[size], &missed_events,
sizeof(missed_events));
commit |= RB_MISSED_STORED;
@@ -6833,8 +6901,8 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
/*
* This page may be off to user land. Zero it out here.
*/
- if (size < rb_page_capacity(reader))
- memset(&bpage->data[size], 0, rb_page_capacity(reader) - size);
+ if (size < rb_read_page_capacity(data_page))
+ memset(&bpage->data[size], 0, rb_read_page_capacity(data_page) - size);
return read;
}
@@ -6852,6 +6920,18 @@ void *ring_buffer_read_page_data(struct buffer_data_read_page *page)
}
EXPORT_SYMBOL_GPL(ring_buffer_read_page_data);
+/**
+ * ring_buffer_read_page_size - get size of the read page.
+ * @page: the page to get the size from
+ *
+ * Returns size of the page in bytes.
+ */
+unsigned int ring_buffer_read_page_size(struct buffer_data_read_page *rpage)
+{
+ return rpage ? PAGE_SIZE << rpage->order : 0;
+}
+EXPORT_SYMBOL_GPL(ring_buffer_read_page_size);
+
/**
* ring_buffer_subbuf_size_get - get size of the sub buffer.
* @buffer: the buffer to get the sub buffer size from
@@ -6945,7 +7025,7 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
/* Make sure all commits have finished */
synchronize_rcu();
- buffer->subbuf_order = order;
+ WRITE_ONCE(buffer->subbuf_order, order);
/* Make sure all new buffers are allocated, before deleting the old ones */
for_each_buffer_cpu(buffer, cpu) {
@@ -7049,7 +7129,7 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
return 0;
error:
- buffer->subbuf_order = old_order;
+ WRITE_ONCE(buffer->subbuf_order, old_order);
atomic_dec(&buffer->record_disabled);
diff --git a/kernel/trace/ring_buffer_benchmark.c b/kernel/trace/ring_buffer_benchmark.c
index cdc3aea12c93..9fd0dc5aff9d 100644
--- a/kernel/trace/ring_buffer_benchmark.c
+++ b/kernel/trace/ring_buffer_benchmark.c
@@ -104,7 +104,7 @@ static enum event_status read_event(int cpu)
static enum event_status read_page(int cpu)
{
- struct buffer_data_read_page *bpage;
+ struct buffer_data_read_page *bpage = NULL;
struct ring_buffer_event *event;
struct rb_page *rpage;
unsigned long commit;
@@ -114,8 +114,8 @@ static enum event_status read_page(int cpu)
int inc;
int i;
- bpage = ring_buffer_alloc_read_page(buffer, cpu);
- if (IS_ERR(bpage))
+ ret = ring_buffer_alloc_read_page(buffer, cpu, &bpage);
+ if (ret < 0)
return EVENT_DROPPED;
page_size = ring_buffer_subbuf_size_get(buffer);
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 7f25ae633094..54bdc493498b 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -7713,11 +7713,10 @@ int tracing_set_filter_buffering(struct trace_array *tr, bool set)
}
struct ftrace_buffer_info {
- struct trace_iterator iter;
- void *spare;
- unsigned int spare_cpu;
- unsigned int spare_size;
- unsigned int read;
+ struct trace_iterator iter;
+ struct buffer_data_read_page *spare;
+ unsigned int spare_cpu;
+ unsigned int read;
};
#ifdef CONFIG_TRACER_SNAPSHOT
@@ -8426,8 +8425,8 @@ tracing_buffers_read(struct file *filp, char __user *ubuf,
{
struct ftrace_buffer_info *info = filp->private_data;
struct trace_iterator *iter = &info->iter;
+ unsigned int spare_size;
void *trace_data;
- int page_size;
ssize_t ret = 0;
ssize_t size;
@@ -8439,36 +8438,22 @@ tracing_buffers_read(struct file *filp, char __user *ubuf,
return -EBUSY;
#endif
- page_size = ring_buffer_subbuf_size_get(iter->array_buffer->buffer);
+ spare_size = ring_buffer_read_page_size(info->spare);
- /* Make sure the spare matches the current sub buffer size */
- if (info->spare) {
- if (page_size != info->spare_size) {
- ring_buffer_free_read_page(iter->array_buffer->buffer,
- info->spare_cpu, info->spare);
- info->spare = NULL;
- }
- }
+again:
+ /* Do we have previous read data to read? */
+ if (info->read < spare_size)
+ goto read;
- if (!info->spare) {
- info->spare = ring_buffer_alloc_read_page(iter->array_buffer->buffer,
- iter->cpu_file);
- if (IS_ERR(info->spare)) {
- ret = PTR_ERR(info->spare);
- info->spare = NULL;
- } else {
- info->spare_cpu = iter->cpu_file;
- info->spare_size = page_size;
- }
- }
- if (!info->spare)
+ ret = ring_buffer_alloc_read_page(iter->array_buffer->buffer, iter->cpu_file,
+ &info->spare);
+ if (ret)
return ret;
- /* Do we have previous read data to read? */
- if (info->read < page_size)
- goto read;
+ spare_size = ring_buffer_read_page_size(info->spare);
+ info->read = spare_size;
+ info->spare_cpu = iter->cpu_file;
- again:
trace_access_lock(iter->cpu_file);
ret = ring_buffer_read_page(iter->array_buffer->buffer,
info->spare,
@@ -8494,8 +8479,9 @@ tracing_buffers_read(struct file *filp, char __user *ubuf,
}
info->read = 0;
+
read:
- size = page_size - info->read;
+ size = spare_size - info->read;
if (size > count)
size = count;
trace_data = ring_buffer_read_page_data(info->spare);
@@ -8536,26 +8522,24 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
__trace_array_put(iter->tr);
- if (info->spare)
- ring_buffer_free_read_page(iter->array_buffer->buffer,
- info->spare_cpu, info->spare);
+ ring_buffer_free_read_page(iter->array_buffer->buffer, info->spare_cpu, info->spare);
kvfree(info);
return 0;
}
struct buffer_ref {
- struct trace_buffer *buffer;
- void *page;
- int cpu;
- refcount_t refcount;
+ struct trace_buffer *buffer;
+ struct buffer_data_read_page *rpage;
+ int cpu;
+ refcount_t refcount;
};
static void buffer_ref_release(struct buffer_ref *ref)
{
if (!refcount_dec_and_test(&ref->refcount))
return;
- ring_buffer_free_read_page(ref->buffer, ref->cpu, ref->page);
+ ring_buffer_free_read_page(ref->buffer, ref->cpu, ref->rpage);
kfree(ref);
}
@@ -8615,9 +8599,9 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
.ops = &buffer_pipe_buf_ops,
.spd_release = buffer_spd_release,
};
+ unsigned int page_size = 0;
struct buffer_ref *ref;
bool woken = false;
- int page_size;
int entries, i;
ssize_t ret = 0;
@@ -8626,16 +8610,6 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
return -EBUSY;
#endif
- page_size = ring_buffer_subbuf_size_get(iter->array_buffer->buffer);
- if (*ppos & (page_size - 1))
- return -EINVAL;
-
- if (len & (page_size - 1)) {
- if (len < page_size)
- return -EINVAL;
- len &= (~(page_size - 1));
- }
-
if (splice_grow_spd(pipe, &spd))
return -ENOMEM;
@@ -8655,25 +8629,37 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos,
refcount_set(&ref->refcount, 1);
ref->buffer = iter->array_buffer->buffer;
- ref->page = ring_buffer_alloc_read_page(ref->buffer, iter->cpu_file);
- if (IS_ERR(ref->page)) {
- ret = PTR_ERR(ref->page);
- ref->page = NULL;
+
+ ret = ring_buffer_alloc_read_page(ref->buffer, iter->cpu_file, &ref->rpage);
+ if (ret) {
kfree(ref);
break;
}
ref->cpu = iter->cpu_file;
- r = ring_buffer_read_page(ref->buffer, ref->page,
- len, iter->cpu_file, 1);
+ page_size = ring_buffer_read_page_size(ref->rpage);
+
+ r = -EINVAL;
+ if (IS_ALIGNED(*ppos, page_size) && len >= page_size) {
+ r = ring_buffer_read_page(ref->buffer, ref->rpage, len, iter->cpu_file, 1);
+ } else if (!i) {
+ /*
+ * We failed to read because the length is too small
+ * or unaligned. If this is the first iteration, it's
+ * an invalid userspace input. Otherwise, this is due
+ * to a subbuf order change. Do not report an error
+ * and just finish the read.
+ */
+ ret = -EINVAL;
+ }
+
if (r < 0) {
- ring_buffer_free_read_page(ref->buffer, ref->cpu,
- ref->page);
+ ring_buffer_free_read_page(ref->buffer, ref->cpu, ref->rpage);
kfree(ref);
break;
}
- page = virt_to_page(ring_buffer_read_page_data(ref->page));
+ page = virt_to_page(ring_buffer_read_page_data(ref->rpage));
spd.pages[i] = page;
spd.partial[i].len = page_size;
@@ -9279,12 +9265,57 @@ trace_options_write(struct file *filp, const char __user *ubuf, size_t cnt,
return cnt;
}
+static bool tr_option_match(struct trace_array *tr, void *topt)
+{
+ for (int i = 0; i < tr->nr_topts; i++) {
+ struct trace_options *tr_topts = &tr->topts[i];
+
+ if (topt >= (void *)&tr_topts->topts[0] &&
+ topt < (void *)&tr_topts->topts[tr_topts->nr_topts])
+ return true;
+ }
+ return false;
+}
+
+/*
+ * The topt is the address of a trace_array->topts[] element that holds the
+ * the tracer options descriptor. But since the trace_array reference has not
+ * been taken yet, it cannot be dereferenced as it could have been freed by
+ * a rmdir of the instance the trace_array represents.
+ *
+ * Search the list of trace_arrays and compare the topt to the address of
+ * the entire trace_array topts array for each trace_array in the list.
+ * If one is matched, then take the reference and return it. If not, the
+ * trace_array no longer exits.
+ */
+static int trace_array_tracer_options_get(void *topt)
+{
+ struct trace_array *tr;
+ int ret;
+
+ ret = security_locked_down(LOCKDOWN_TRACEFS);
+ if (ret)
+ return ret;
+
+ if (tracing_disabled)
+ return -ENODEV;
+
+ guard(mutex)(&trace_types_lock);
+ list_for_each_entry(tr, &ftrace_trace_arrays, list) {
+ if (tr_option_match(tr, topt)) {
+ tr->ref++;
+ return 0;
+ }
+ }
+ return -ENODEV;
+}
+
static int tracing_open_options(struct inode *inode, struct file *filp)
{
struct trace_option_dentry *topt = inode->i_private;
int ret;
- ret = tracing_check_open_get_tr(topt->tr);
+ ret = trace_array_tracer_options_get(topt);
if (ret)
return ret;
@@ -9568,6 +9599,7 @@ create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
tr->topts = tr_topts;
tr->topts[tr->nr_topts].tracer = tracer;
tr->topts[tr->nr_topts].topts = topts;
+ tr->topts[tr->nr_topts].nr_topts = cnt;
tr->nr_topts++;
for (cnt = 0; opts[cnt].name; cnt++) {
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index ab88b87ac2ee..39c425a9538a 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -224,6 +224,7 @@ struct array_buffer {
struct trace_options {
struct tracer *tracer;
struct trace_option_dentry *topts;
+ int nr_topts;
};
struct trace_pid_list *trace_pid_list_alloc(void);
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 6a4406acc840..162137855058 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -41,6 +41,7 @@
#include <linux/mempolicy.h>
#include <linux/freezer.h>
#include <linux/debug_locks.h>
+#include <linux/device/devres.h>
#include <linux/lockdep.h>
#include <linux/idr.h>
#include <linux/jhash.h>
@@ -2235,7 +2236,7 @@ static bool is_chained_work(struct workqueue_struct *wq)
* Return %true iff I'm a worker executing a work item on @wq. If
* I'm @worker, it's safe to dereference it without locking.
*/
- return worker && worker->current_pwq->wq == wq;
+ return worker && worker->current_pwq && worker->current_pwq->wq == wq;
}
/*
@@ -5874,6 +5875,21 @@ static struct workqueue_struct *__alloc_workqueue(const char *fmt,
return NULL;
}
+__printf(1, 0)
+static struct workqueue_struct *alloc_workqueue_va(const char *fmt,
+ unsigned int flags,
+ int max_active,
+ va_list args)
+{
+ struct workqueue_struct *wq;
+
+ wq = __alloc_workqueue(fmt, flags, max_active, args);
+ if (wq)
+ wq_init_lockdep(wq);
+
+ return wq;
+}
+
__printf(1, 4)
struct workqueue_struct *alloc_workqueue_noprof(const char *fmt,
unsigned int flags,
@@ -5883,16 +5899,39 @@ struct workqueue_struct *alloc_workqueue_noprof(const char *fmt,
va_list args;
va_start(args, max_active);
- wq = __alloc_workqueue(fmt, flags, max_active, args);
+ wq = alloc_workqueue_va(fmt, flags, max_active, args);
+ va_end(args);
+
+ return wq;
+}
+EXPORT_SYMBOL_GPL(alloc_workqueue_noprof);
+
+static void devm_workqueue_release(void *res)
+{
+ destroy_workqueue(res);
+}
+
+__printf(2, 5) struct workqueue_struct *
+devm_alloc_workqueue_noprof(struct device *dev, const char *fmt,
+ unsigned int flags, int max_active, ...)
+{
+ struct workqueue_struct *wq;
+ va_list args;
+ int ret;
+
+ va_start(args, max_active);
+ wq = alloc_workqueue_va(fmt, flags, max_active, args);
va_end(args);
if (!wq)
return NULL;
- wq_init_lockdep(wq);
+ ret = devm_add_action_or_reset(dev, devm_workqueue_release, wq);
+ if (ret)
+ return NULL;
return wq;
}
-EXPORT_SYMBOL_GPL(alloc_workqueue_noprof);
+EXPORT_SYMBOL_GPL(devm_alloc_workqueue_noprof);
#ifdef CONFIG_LOCKDEP
__printf(1, 5)
diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c
index b7f347ed786e..5a6e537e0f48 100644
--- a/mm/pgtable-generic.c
+++ b/mm/pgtable-generic.c
@@ -436,12 +436,30 @@ static void kernel_pgtable_work_func(struct work_struct *work)
__pagetable_free(pt);
}
+static void schedule_kernel_pgtable_free(void)
+{
+ schedule_work(&kernel_pgtable_work.work);
+}
+
void pagetable_free_kernel(struct ptdesc *pt)
{
spin_lock(&kernel_pgtable_work.lock);
list_add(&pt->pt_list, &kernel_pgtable_work.list);
spin_unlock(&kernel_pgtable_work.lock);
- schedule_work(&kernel_pgtable_work.work);
+ /*
+ * The workqueue may not exist yet while the system is booting.
+ * kernel_pgtable_drain_early() schedules the work once it does.
+ */
+ if (system_state != SYSTEM_BOOTING)
+ schedule_kernel_pgtable_free();
+}
+
+static int __init kernel_pgtable_drain_early(void)
+{
+ /* Free the kernel page tables queued while booting. */
+ schedule_kernel_pgtable_free();
+ return 0;
}
+core_initcall(kernel_pgtable_drain_early);
#endif
diff --git a/net/9p/trans_fd.c b/net/9p/trans_fd.c
index a516745f732f..ef517bb307e2 100644
--- a/net/9p/trans_fd.c
+++ b/net/9p/trans_fd.c
@@ -966,7 +966,7 @@ static int p9_bind_privport(struct socket *sock)
((struct sockaddr_in *)&stor)->sin_port = htons((ushort)port);
else
((struct sockaddr_in6 *)&stor)->sin6_port = htons((ushort)port);
- err = kernel_bind(sock, (struct sockaddr *)&stor, sizeof(stor));
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&stor, sizeof(stor));
if (err != -EADDRINUSE)
break;
}
diff --git a/net/appletalk/ddp.c b/net/appletalk/ddp.c
index 30242fe10341..45db43cde67f 100644
--- a/net/appletalk/ddp.c
+++ b/net/appletalk/ddp.c
@@ -1149,7 +1149,7 @@ static int atalk_autobind(struct sock *sk)
}
/* Set the address 'our end' of the connection */
-static int atalk_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int atalk_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sockaddr_at *addr = (struct sockaddr_at *)uaddr;
struct sock *sk = sock->sk;
diff --git a/net/atm/pvc.c b/net/atm/pvc.c
index 66d9a9bd5896..62fdf07c53de 100644
--- a/net/atm/pvc.c
+++ b/net/atm/pvc.c
@@ -24,7 +24,7 @@ static int pvc_shutdown(struct socket *sock, int how)
return 0;
}
-static int pvc_bind(struct socket *sock, struct sockaddr *sockaddr,
+static int pvc_bind(struct socket *sock, struct sockaddr_unsized *sockaddr,
int sockaddr_len)
{
struct sock *sk = sock->sk;
@@ -59,7 +59,7 @@ static int pvc_bind(struct socket *sock, struct sockaddr *sockaddr,
static int pvc_connect(struct socket *sock, struct sockaddr *sockaddr,
int sockaddr_len, int flags)
{
- return pvc_bind(sock, sockaddr, sockaddr_len);
+ return pvc_bind(sock, (struct sockaddr_unsized *)sockaddr, sockaddr_len);
}
static int pvc_setsockopt(struct socket *sock, int level, int optname,
diff --git a/net/atm/svc.c b/net/atm/svc.c
index f8137ae693b0..1906a493c8aa 100644
--- a/net/atm/svc.c
+++ b/net/atm/svc.c
@@ -97,7 +97,7 @@ static int svc_release(struct socket *sock)
return 0;
}
-static int svc_bind(struct socket *sock, struct sockaddr *sockaddr,
+static int svc_bind(struct socket *sock, struct sockaddr_unsized *sockaddr,
int sockaddr_len)
{
DEFINE_WAIT(wait);
diff --git a/net/ax25/af_ax25.c b/net/ax25/af_ax25.c
index 6ef8b2a57a9b..23c558ff9682 100644
--- a/net/ax25/af_ax25.c
+++ b/net/ax25/af_ax25.c
@@ -1094,7 +1094,7 @@ static int ax25_release(struct socket *sock)
* that we've implemented support for SO_BINDTODEVICE. It is however small
* and trivially backward compatible.
*/
-static int ax25_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int ax25_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sock *sk = sock->sk;
struct full_sockaddr_ax25 *addr = (struct full_sockaddr_ax25 *)uaddr;
diff --git a/net/bluetooth/6lowpan.c b/net/bluetooth/6lowpan.c
index e5e84941c652..3733f408d2e4 100644
--- a/net/bluetooth/6lowpan.c
+++ b/net/bluetooth/6lowpan.c
@@ -1068,23 +1068,15 @@ static void disconnect_all_peers(void)
} while (nchans);
}
-struct set_enable {
- struct work_struct work;
- bool flag;
-};
-
-static void do_enable_set(struct work_struct *work)
+static void do_enable_set(bool flag)
{
- struct set_enable *set_enable = container_of(work,
- struct set_enable, work);
-
- if (!set_enable->flag || enable_6lowpan != set_enable->flag)
+ if (!flag || enable_6lowpan != flag)
/* Disconnect existing connections if 6lowpan is
* disabled
*/
disconnect_all_peers();
- enable_6lowpan = set_enable->flag;
+ enable_6lowpan = flag;
mutex_lock(&set_lock);
if (listen_chan) {
@@ -1096,22 +1088,11 @@ static void do_enable_set(struct work_struct *work)
listen_chan = bt_6lowpan_listen();
mutex_unlock(&set_lock);
-
- kfree(set_enable);
}
static int lowpan_enable_set(void *data, u64 val)
{
- struct set_enable *set_enable;
-
- set_enable = kzalloc(sizeof(*set_enable), GFP_KERNEL);
- if (!set_enable)
- return -ENOMEM;
-
- set_enable->flag = !!val;
- INIT_WORK(&set_enable->work, do_enable_set);
-
- schedule_work(&set_enable->work);
+ do_enable_set(!!val);
return 0;
}
diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c
index fab41702ea57..32cabe83351a 100644
--- a/net/bluetooth/hci_conn.c
+++ b/net/bluetooth/hci_conn.c
@@ -1017,6 +1017,11 @@ static struct hci_conn *__hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t
/* conn->src should reflect the local identity address */
hci_copy_identity_address(hdev, &conn->src, &conn->src_type);
conn->mtu = hdev->le_mtu ? hdev->le_mtu : hdev->acl_mtu;
+ /* Use the controller supported PHYS as default until the
+ * remote features are resolved.
+ */
+ conn->le_tx_def_phys = hdev->le_tx_def_phys;
+ conn->le_rx_def_phys = hdev->le_tx_def_phys;
break;
case CIS_LINK:
/* conn->src should reflect the local identity address */
@@ -2960,22 +2965,22 @@ u32 hci_conn_get_phy(struct hci_conn *conn)
break;
case LE_LINK:
- if (conn->le_tx_phy & HCI_LE_SET_PHY_1M)
+ if (conn->le_tx_def_phys & HCI_LE_SET_PHY_1M)
phys |= BT_PHY_LE_1M_TX;
- if (conn->le_rx_phy & HCI_LE_SET_PHY_1M)
+ if (conn->le_rx_def_phys & HCI_LE_SET_PHY_1M)
phys |= BT_PHY_LE_1M_RX;
- if (conn->le_tx_phy & HCI_LE_SET_PHY_2M)
+ if (conn->le_tx_def_phys & HCI_LE_SET_PHY_2M)
phys |= BT_PHY_LE_2M_TX;
- if (conn->le_rx_phy & HCI_LE_SET_PHY_2M)
+ if (conn->le_rx_def_phys & HCI_LE_SET_PHY_2M)
phys |= BT_PHY_LE_2M_RX;
- if (conn->le_tx_phy & HCI_LE_SET_PHY_CODED)
+ if (conn->le_tx_def_phys & HCI_LE_SET_PHY_CODED)
phys |= BT_PHY_LE_CODED_TX;
- if (conn->le_rx_phy & HCI_LE_SET_PHY_CODED)
+ if (conn->le_rx_def_phys & HCI_LE_SET_PHY_CODED)
phys |= BT_PHY_LE_CODED_RX;
break;
diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c
index 90287b7b62a4..e407ff39254e 100644
--- a/net/bluetooth/hci_core.c
+++ b/net/bluetooth/hci_core.c
@@ -4152,9 +4152,11 @@ static int hci_send_cmd_sync(struct hci_dev *hdev, struct sk_buff *skb)
bt_dev_dbg(hdev, "skb %p", skb);
+ hci_dev_lock(hdev);
kfree_skb(hdev->sent_cmd);
hdev->sent_cmd = skb_clone(skb, GFP_KERNEL);
+ hci_dev_unlock(hdev);
if (!hdev->sent_cmd) {
skb_queue_head(&hdev->cmd_q, skb);
queue_work(hdev->workqueue, &hdev->cmd_work);
@@ -4175,8 +4177,10 @@ static int hci_send_cmd_sync(struct hci_dev *hdev, struct sk_buff *skb)
if (READ_ONCE(hdev->req_status) == HCI_REQ_PEND &&
!hci_dev_test_and_set_flag(hdev, HCI_CMD_PENDING)) {
+ hci_dev_lock(hdev);
kfree_skb(hdev->req_skb);
hdev->req_skb = skb_get(hdev->sent_cmd);
+ hci_dev_unlock(hdev);
}
return err;
diff --git a/net/bluetooth/hci_event.c b/net/bluetooth/hci_event.c
index 031ee997bb92..23fe327a0fcd 100644
--- a/net/bluetooth/hci_event.c
+++ b/net/bluetooth/hci_event.c
@@ -80,6 +80,10 @@ static void *hci_le_ev_skb_pull(struct hci_dev *hdev, struct sk_buff *skb,
return data;
}
+static void hci_store_wake_reason(struct hci_dev *hdev,
+ const bdaddr_t *bdaddr, u8 addr_type)
+ __must_hold(&hdev->lock);
+
static u8 hci_cc_inquiry_cancel(struct hci_dev *hdev, void *data,
struct sk_buff *skb)
{
@@ -3099,6 +3103,7 @@ static void hci_conn_complete_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
/* Check for existing connection:
*
@@ -3261,6 +3266,10 @@ static void hci_conn_request_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "bdaddr %pMR type 0x%x", &ev->bdaddr, ev->link_type);
+ hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
+ hci_dev_unlock(hdev);
+
/* Reject incoming connection from device with same BD ADDR against
* CVE-2020-26555
*/
@@ -4963,6 +4972,7 @@ static void hci_sync_conn_complete_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
conn = hci_conn_hash_lookup_ba(hdev, ev->link_type, &ev->bdaddr);
if (!conn) {
@@ -5664,6 +5674,7 @@ static void le_conn_complete_evt(struct hci_dev *hdev, u8 status,
u8 addr_type;
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, bdaddr, bdaddr_type);
/* Advertising stops when a connection is created. On a failed
* connection it keeps running, so leave the state bit alone.
@@ -5976,6 +5987,7 @@ static void hci_le_past_received_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", ev->status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, ev->bdaddr_type);
hci_dev_clear_flag(hdev, HCI_PA_SYNC);
@@ -6374,6 +6386,8 @@ static void hci_le_adv_report_evt(struct hci_dev *hdev, void *data,
info->length + 1))
break;
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
if (info->length <= max_adv_len(hdev)) {
rssi = info->data[info->length];
process_adv_report(hdev, info->type, &info->bdaddr,
@@ -6462,6 +6476,8 @@ static void hci_le_ext_adv_report_evt(struct hci_dev *hdev, void *data,
info->length))
break;
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
evt_type = __le16_to_cpu(info->type) & LE_EXT_ADV_EVT_TYPE_MASK;
legacy_evt_type = ext_evt_type_to_legacy(hdev, evt_type);
@@ -6507,6 +6523,7 @@ static void hci_le_pa_sync_established_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", ev->status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, ev->bdaddr_type);
hci_dev_clear_flag(hdev, HCI_PA_SYNC);
@@ -6604,9 +6621,21 @@ static void hci_le_remote_feat_complete_evt(struct hci_dev *hdev, void *data,
conn = hci_conn_hash_lookup_handle(hdev, __le16_to_cpu(ev->handle));
if (conn) {
- if (!ev->status)
+ if (!ev->status) {
memcpy(conn->features[0], ev->features, 8);
+ /* Update supported PHYs */
+ if (!(conn->features[0][1] & HCI_LE_PHY_2M)) {
+ conn->le_tx_def_phys &= ~HCI_LE_SET_PHY_2M;
+ conn->le_rx_def_phys &= ~HCI_LE_SET_PHY_2M;
+ }
+
+ if (!(conn->features[0][1] & HCI_LE_PHY_CODED)) {
+ conn->le_tx_def_phys &= ~HCI_LE_SET_PHY_CODED;
+ conn->le_rx_def_phys &= ~HCI_LE_SET_PHY_CODED;
+ }
+ }
+
if (conn->state == BT_CONFIG) {
__u8 status;
@@ -6801,6 +6830,8 @@ static void hci_le_direct_adv_report_evt(struct hci_dev *hdev, void *data,
for (i = 0; i < ev->num; i++) {
struct hci_ev_le_direct_adv_info *info = &ev->info[i];
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
process_adv_report(hdev, info->type, &info->bdaddr,
info->bdaddr_type, &info->direct_addr,
info->direct_addr_type, HCI_ADV_PHY_1M, 0,
@@ -7430,73 +7461,29 @@ static bool hci_get_cmd_complete(struct hci_dev *hdev, u16 opcode,
return true;
}
-static void hci_store_wake_reason(struct hci_dev *hdev, u8 event,
- struct sk_buff *skb)
+static void hci_store_wake_reason(struct hci_dev *hdev,
+ const bdaddr_t *bdaddr, u8 addr_type)
+ __must_hold(&hdev->lock)
{
- struct hci_ev_le_advertising_info *adv;
- struct hci_ev_le_direct_adv_info *direct_adv;
- struct hci_ev_le_ext_adv_info *ext_adv;
- const struct hci_ev_conn_complete *conn_complete = (void *)skb->data;
- const struct hci_ev_conn_request *conn_request = (void *)skb->data;
-
- hci_dev_lock(hdev);
+ lockdep_assert_held(&hdev->lock);
/* If we are currently suspended and this is the first BT event seen,
* save the wake reason associated with the event.
*/
if (!hdev->suspended || hdev->wake_reason)
- goto unlock;
+ return;
+
+ if (!bdaddr) {
+ hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED;
+ return;
+ }
/* Default to remote wake. Values for wake_reason are documented in the
* Bluez mgmt api docs.
*/
hdev->wake_reason = MGMT_WAKE_REASON_REMOTE_WAKE;
-
- /* Once configured for remote wakeup, we should only wake up for
- * reconnections. It's useful to see which device is waking us up so
- * keep track of the bdaddr of the connection event that woke us up.
- */
- if (event == HCI_EV_CONN_REQUEST) {
- bacpy(&hdev->wake_addr, &conn_request->bdaddr);
- hdev->wake_addr_type = BDADDR_BREDR;
- } else if (event == HCI_EV_CONN_COMPLETE) {
- bacpy(&hdev->wake_addr, &conn_complete->bdaddr);
- hdev->wake_addr_type = BDADDR_BREDR;
- } else if (event == HCI_EV_LE_META) {
- struct hci_ev_le_meta *le_ev = (void *)skb->data;
- u8 subevent = le_ev->subevent;
- u8 *ptr = &skb->data[sizeof(*le_ev)];
- u8 num_reports = *ptr;
-
- if ((subevent == HCI_EV_LE_ADVERTISING_REPORT ||
- subevent == HCI_EV_LE_DIRECT_ADV_REPORT ||
- subevent == HCI_EV_LE_EXT_ADV_REPORT) &&
- num_reports) {
- adv = (void *)(ptr + 1);
- direct_adv = (void *)(ptr + 1);
- ext_adv = (void *)(ptr + 1);
-
- switch (subevent) {
- case HCI_EV_LE_ADVERTISING_REPORT:
- bacpy(&hdev->wake_addr, &adv->bdaddr);
- hdev->wake_addr_type = adv->bdaddr_type;
- break;
- case HCI_EV_LE_DIRECT_ADV_REPORT:
- bacpy(&hdev->wake_addr, &direct_adv->bdaddr);
- hdev->wake_addr_type = direct_adv->bdaddr_type;
- break;
- case HCI_EV_LE_EXT_ADV_REPORT:
- bacpy(&hdev->wake_addr, &ext_adv->bdaddr);
- hdev->wake_addr_type = ext_adv->bdaddr_type;
- break;
- }
- }
- } else {
- hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED;
- }
-
-unlock:
- hci_dev_unlock(hdev);
+ bacpy(&hdev->wake_addr, bdaddr);
+ hdev->wake_addr_type = addr_type;
}
#define HCI_EV_VL(_op, _func, _min_len, _max_len) \
@@ -7743,14 +7730,15 @@ void hci_event_packet(struct hci_dev *hdev, struct sk_buff *skb)
skb_pull(skb, HCI_EVENT_HDR_SIZE);
- /* Store wake reason if we're suspended */
- hci_store_wake_reason(hdev, event, skb);
-
bt_dev_dbg(hdev, "event 0x%2.2x", event);
hci_event_func(hdev, event, skb, &opcode, &status, &req_complete,
&req_complete_skb);
+ hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, NULL, 0);
+ hci_dev_unlock(hdev);
+
if (req_complete) {
req_complete(hdev, status, opcode);
} else if (req_complete_skb) {
diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
index 694a11fb1286..910fa5912838 100644
--- a/net/bluetooth/hci_sock.c
+++ b/net/bluetooth/hci_sock.c
@@ -1196,7 +1196,7 @@ static int hci_sock_compat_ioctl(struct socket *sock, unsigned int cmd,
}
#endif
-static int hci_sock_bind(struct socket *sock, struct sockaddr *addr,
+static int hci_sock_bind(struct socket *sock, struct sockaddr_unsized *addr,
int addr_len)
{
struct sockaddr_hci haddr;
@@ -1225,8 +1225,10 @@ static int hci_sock_bind(struct socket *sock, struct sockaddr *addr,
*/
hdev = hci_pi(sk)->hdev;
if (hdev && hci_dev_test_flag(hdev, HCI_UNREGISTER)) {
+ write_lock(&hci_sk_list.lock);
hci_pi(sk)->hdev = NULL;
sk->sk_state = BT_OPEN;
+ write_unlock(&hci_sk_list.lock);
hci_dev_put(hdev);
}
hdev = NULL;
diff --git a/net/bluetooth/hci_sync.c b/net/bluetooth/hci_sync.c
index c1db8a37726d..70fce6811d0a 100644
--- a/net/bluetooth/hci_sync.c
+++ b/net/bluetooth/hci_sync.c
@@ -32,8 +32,10 @@ static void hci_cmd_sync_complete(struct hci_dev *hdev, u8 result, u16 opcode,
WRITE_ONCE(hdev->req_status, HCI_REQ_DONE);
/* Free the request command so it is not used as response */
+ hci_dev_lock(hdev);
kfree_skb(hdev->req_skb);
hdev->req_skb = NULL;
+ hci_dev_unlock(hdev);
if (skb) {
struct sock *sk = hci_skb_sk(skb);
@@ -792,7 +794,7 @@ int hci_cmd_sync_queue_once(struct hci_dev *hdev, hci_cmd_sync_work_func_t func,
void *data, hci_cmd_sync_work_destroy_t destroy)
{
if (hci_cmd_sync_lookup_entry(hdev, func, data, destroy))
- return 0;
+ return -EEXIST;
return hci_cmd_sync_queue(hdev, func, data, destroy);
}
@@ -3055,15 +3057,21 @@ static int hci_le_set_ext_scan_param_sync(struct hci_dev *hdev, u8 type,
*/
if (hci_dev_test_flag(hdev, HCI_PA_SYNC)) {
struct hci_cp_le_add_to_accept_list *sent;
+ bdaddr_t bdaddr;
+ hci_dev_lock(hdev);
sent = hci_sent_cmd_data(hdev, HCI_OP_LE_ADD_TO_ACCEPT_LIST);
+ if (sent)
+ bacpy(&bdaddr, &sent->bdaddr);
+ hci_dev_unlock(hdev);
+
if (sent) {
struct hci_conn *conn;
rcu_read_lock();
conn = hci_conn_hash_lookup_ba(hdev, PA_LINK,
- &sent->bdaddr);
+ &bdaddr);
if (conn) {
struct bt_iso_qos *qos = &conn->iso_qos;
@@ -3382,6 +3390,8 @@ static int update_passive_scan_sync(struct hci_dev *hdev, void *data)
int hci_update_passive_scan(struct hci_dev *hdev)
{
+ int err;
+
/* Only queue if it would have any effect */
if (!test_bit(HCI_UP, &hdev->flags) ||
test_bit(HCI_INIT, &hdev->flags) ||
@@ -3391,8 +3401,9 @@ int hci_update_passive_scan(struct hci_dev *hdev)
hci_dev_test_flag(hdev, HCI_UNREGISTER))
return 0;
- return hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL,
- NULL);
+ err = hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL,
+ NULL);
+ return (err == -EEXIST) ? 0 : err;
}
int hci_write_sc_support_sync(struct hci_dev *hdev, u8 val)
@@ -7129,10 +7140,24 @@ static int hci_acl_create_conn_sync(struct hci_dev *hdev, void *data)
return err;
}
+static void hci_acl_create_conn_sync_complete(struct hci_dev *hdev, void *data,
+ int err)
+{
+ struct hci_conn *conn = data;
+
+ hci_conn_put(conn);
+}
+
int hci_connect_acl_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_acl_create_conn_sync, conn,
- NULL);
+ int err;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_acl_create_conn_sync,
+ hci_conn_get(conn),
+ hci_acl_create_conn_sync_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
static void create_le_conn_complete(struct hci_dev *hdev, void *data, int err)
@@ -7142,34 +7167,47 @@ static void create_le_conn_complete(struct hci_dev *hdev, void *data, int err)
bt_dev_dbg(hdev, "err %d", err);
if (err == -ECANCELED)
- return;
+ goto done;
hci_dev_lock(hdev);
if (!hci_conn_valid(hdev, conn))
- goto done;
+ goto unlock;
if (!err) {
hci_connect_le_scan_cleanup(conn, 0x00);
- goto done;
+ goto unlock;
}
- /* Check if connection is still pending */
- if (conn != hci_lookup_le_connect(hdev))
- goto done;
+ /* Check if this connection is still pending.
+ *
+ * hci_lookup_le_connect() returns only the first LE connection
+ * in BT_CONNECT, which is not necessarily this one when two are
+ * pending at once, so ask the connection itself.
+ */
+ if (conn->state != BT_CONNECT)
+ goto unlock;
/* Flush to make sure we send create conn cancel command if needed */
flush_delayed_work(&conn->le_conn_timeout);
hci_conn_failed(conn, bt_status(err));
-done:
+unlock:
hci_dev_unlock(hdev);
+done:
+ hci_conn_put(conn);
}
int hci_connect_le_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_le_create_conn_sync, conn,
- create_le_conn_complete);
+ int err;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_le_create_conn_sync,
+ hci_conn_get(conn),
+ create_le_conn_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
static int hci_acl_cancel_create_conn_sync(struct hci_dev *hdev,
@@ -7291,7 +7329,7 @@ static void create_pa_complete(struct hci_dev *hdev, void *data, int err)
bt_dev_dbg(hdev, "err %d", err);
if (err == -ECANCELED)
- return;
+ goto done;
hci_dev_lock(hdev);
@@ -7315,6 +7353,8 @@ static void create_pa_complete(struct hci_dev *hdev, void *data, int err)
unlock:
hci_dev_unlock(hdev);
+done:
+ hci_conn_put(conn);
}
static int hci_le_pa_create_sync(struct hci_dev *hdev, void *data)
@@ -7402,8 +7442,14 @@ static int hci_le_pa_create_sync(struct hci_dev *hdev, void *data)
int hci_connect_pa_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_le_pa_create_sync, conn,
- create_pa_complete);
+ int err;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_le_pa_create_sync,
+ hci_conn_get(conn),
+ create_pa_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
static void create_big_complete(struct hci_dev *hdev, void *data, int err)
@@ -7413,10 +7459,12 @@ static void create_big_complete(struct hci_dev *hdev, void *data, int err)
bt_dev_dbg(hdev, "err %d", err);
if (err == -ECANCELED)
- return;
+ goto done;
- if (hci_conn_valid(hdev, conn))
- clear_bit(HCI_CONN_CREATE_BIG_SYNC, &conn->flags);
+ clear_bit(HCI_CONN_CREATE_BIG_SYNC, &conn->flags);
+
+done:
+ hci_conn_put(conn);
}
static int hci_le_big_create_sync(struct hci_dev *hdev, void *data)
@@ -7466,6 +7514,15 @@ static int hci_le_big_create_sync(struct hci_dev *hdev, void *data)
int hci_connect_big_sync(struct hci_dev *hdev, struct hci_conn *conn)
{
- return hci_cmd_sync_queue_once(hdev, hci_le_big_create_sync, conn,
- create_big_complete);
+ int err;
+
+ if (!conn)
+ return 0;
+
+ err = hci_cmd_sync_queue_once(hdev, hci_le_big_create_sync,
+ hci_conn_get(conn),
+ create_big_complete);
+ if (err)
+ hci_conn_put(conn);
+ return (err == -EEXIST) ? 0 : err;
}
diff --git a/net/bluetooth/iso.c b/net/bluetooth/iso.c
index 272980ca3038..e43aaa8289a0 100644
--- a/net/bluetooth/iso.c
+++ b/net/bluetooth/iso.c
@@ -60,6 +60,7 @@ enum {
BT_SK_BIG_SYNC,
BT_SK_PA_SYNC,
BT_SK_KILLED,
+ BT_SK_CONNECTING,
};
struct iso_pinfo {
@@ -316,6 +317,9 @@ static int __iso_chan_add(struct iso_conn *conn, struct sock *sk,
return -EBUSY;
}
+ if (iso_pi(sk)->conn)
+ return -EISCONN;
+
if (!conn->hcon) {
BT_ERR("conn->hcon missing");
return -EIO;
@@ -376,6 +380,11 @@ static int iso_connect_bis(struct sock *sk)
hci_dev_lock(hdev);
lock_sock(sk);
+ if (iso_pi(sk)->conn) {
+ err = -EISCONN;
+ goto unlock;
+ }
+
if (!bis_capable(hdev)) {
err = -EOPNOTSUPP;
goto unlock;
@@ -524,6 +533,13 @@ static int iso_connect_cis(struct sock *sk)
}
}
+ /* The socket lock keeps the current attachment and its hcon stable. */
+ if (iso_pi(sk)->conn && iso_pi(sk)->conn->hcon != hcon) {
+ hci_conn_drop(hcon);
+ err = -EISCONN;
+ goto unlock;
+ }
+
conn = iso_conn_add(hcon);
if (!conn) {
hci_conn_drop(hcon);
@@ -1001,7 +1017,7 @@ static int iso_sock_create(struct net *net, struct socket *sock, int protocol,
return 0;
}
-static int iso_sock_bind_bc(struct socket *sock, struct sockaddr *addr,
+static int iso_sock_bind_bc(struct socket *sock, struct sockaddr_unsized *addr,
int addr_len)
{
struct sockaddr_iso *sa = (struct sockaddr_iso *)addr;
@@ -1079,7 +1095,7 @@ static int iso_sock_bind_pa_sk(struct sock *sk, struct sockaddr_iso *sa,
return err;
}
-static int iso_sock_bind(struct socket *sock, struct sockaddr *addr,
+static int iso_sock_bind(struct socket *sock, struct sockaddr_unsized *addr,
int addr_len)
{
struct sockaddr_iso *sa = (struct sockaddr_iso *)addr;
@@ -1150,17 +1166,26 @@ static int iso_sock_connect(struct socket *sock, struct sockaddr *addr,
addr->sa_family != AF_BLUETOOTH)
return -EINVAL;
- if (sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND)
- return -EBADFD;
+ lock_sock(sk);
- if (sk->sk_type != SOCK_SEQPACKET)
- return -EINVAL;
+ if ((sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND) ||
+ test_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags)) {
+ err = -EBADFD;
+ goto done;
+ }
+
+ if (sk->sk_type != SOCK_SEQPACKET) {
+ err = -EINVAL;
+ goto done;
+ }
/* Check if the address type is of LE type */
- if (!bdaddr_type_is_le(sa->iso_bdaddr_type))
- return -EINVAL;
+ if (!bdaddr_type_is_le(sa->iso_bdaddr_type)) {
+ err = -EINVAL;
+ goto done;
+ }
- lock_sock(sk);
+ set_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags);
bacpy(&iso_pi(sk)->dst, &sa->iso_bdaddr);
iso_pi(sk)->dst_type = sa->iso_bdaddr_type;
@@ -1172,16 +1197,18 @@ static int iso_sock_connect(struct socket *sock, struct sockaddr *addr,
else
err = iso_connect_bis(sk);
- if (err)
- return err;
-
lock_sock(sk);
+ clear_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags);
+ if (err)
+ goto done;
+
if (!test_bit(BT_SK_DEFER_SETUP, &bt_sk(sk)->flags)) {
err = bt_sock_wait_state(sk, BT_CONNECTED,
sock_sndtimeo(sk, flags & O_NONBLOCK));
}
+done:
release_sock(sk);
return err;
}
@@ -1221,6 +1248,11 @@ static int iso_listen_bis(struct sock *sk)
hci_dev_lock(hdev);
lock_sock(sk);
+ if (sk->sk_state != BT_BOUND || iso_pi(sk)->conn) {
+ err = -EBADFD;
+ goto unlock;
+ }
+
/* Fail if user set invalid QoS */
if (iso_pi(sk)->qos_user_set && !check_bcast_qos(&iso_pi(sk)->qos)) {
iso_pi(sk)->qos = default_qos;
diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c
index d332cd999d8c..9078a672e2dd 100644
--- a/net/bluetooth/l2cap_sock.c
+++ b/net/bluetooth/l2cap_sock.c
@@ -81,7 +81,7 @@ static int l2cap_validate_le_psm(u16 psm)
return 0;
}
-static int l2cap_sock_bind(struct socket *sock, struct sockaddr *addr, int alen)
+static int l2cap_sock_bind(struct socket *sock, struct sockaddr_unsized *addr, int alen)
{
struct sock *sk = sock->sk;
struct l2cap_chan *chan = l2cap_pi(sk)->chan;
@@ -438,11 +438,26 @@ static int l2cap_get_mode(struct l2cap_chan *chan)
return -EINVAL;
}
+static struct l2cap_conn *l2cap_chan_conn(struct l2cap_chan *chan)
+{
+ lockdep_assert_held(&chan->lock);
+
+ /* l2cap_conn_del() sets FLAG_DEL while holding chan->lock before
+ * conn->hcon is deleted. If not set and conn is non-NULL, conn->hcon
+ * remains alive during this chan->lock critical section.
+ */
+ if (test_bit(FLAG_DEL, &chan->flags))
+ return NULL;
+
+ return chan->conn;
+}
+
static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
char __user *optval, int __user *optlen)
{
struct sock *sk = sock->sk;
struct l2cap_chan *chan = l2cap_pi(sk)->chan;
+ struct l2cap_conn *conn;
struct l2cap_options opts;
struct l2cap_conninfo cinfo;
int err = 0;
@@ -454,6 +469,7 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
if (get_user(len, optlen))
return -EFAULT;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -539,9 +555,15 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
break;
}
+ conn = l2cap_chan_conn(chan);
+ if (!conn) {
+ err = -ENOTCONN;
+ break;
+ }
+
memset(&cinfo, 0, sizeof(cinfo));
- cinfo.hci_handle = chan->conn->hcon->handle;
- memcpy(cinfo.dev_class, chan->conn->hcon->dev_class, 3);
+ cinfo.hci_handle = conn->hcon->handle;
+ memcpy(cinfo.dev_class, conn->hcon->dev_class, 3);
len = min(len, sizeof(cinfo));
if (copy_to_user(optval, (char *) &cinfo, len))
@@ -555,6 +577,8 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
+
return err;
}
@@ -563,6 +587,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
{
struct sock *sk = sock->sk;
struct l2cap_chan *chan = l2cap_pi(sk)->chan;
+ struct l2cap_conn *conn;
struct bt_security sec;
struct bt_power pwr;
u32 phys;
@@ -579,6 +604,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
if (get_user(len, optlen))
return -EFAULT;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -590,12 +616,14 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
}
+ conn = l2cap_chan_conn(chan);
+
memset(&sec, 0, sizeof(sec));
- if (chan->conn) {
- sec.level = chan->conn->hcon->sec_level;
+ if (conn) {
+ sec.level = conn->hcon->sec_level;
if (sk->sk_state == BT_CONNECTED)
- sec.key_size = chan->conn->hcon->enc_key_size;
+ sec.key_size = conn->hcon->enc_key_size;
} else {
sec.level = chan->sec_level;
}
@@ -671,12 +699,14 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
case BT_PHY:
- if (sk->sk_state != BT_CONNECTED) {
+ conn = l2cap_chan_conn(chan);
+
+ if (sk->sk_state != BT_CONNECTED || !conn) {
err = -ENOTCONN;
break;
}
- phys = hci_conn_get_phy(chan->conn->hcon);
+ phys = hci_conn_get_phy(conn->hcon);
if (put_user(phys, (u32 __user *) optval))
err = -EFAULT;
@@ -709,6 +739,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -739,6 +770,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname,
BT_DBG("sk %p", sk);
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -840,6 +872,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -903,6 +936,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
if (level != SOL_BLUETOOTH)
return -ENOPROTOOPT;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -928,11 +962,10 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
chan->sec_level = sec.level;
- if (!chan->conn)
+ conn = l2cap_chan_conn(chan);
+ if (!conn)
break;
- conn = chan->conn;
-
/* change security for LE channels */
if (chan->scid == L2CAP_CID_ATT) {
if (smp_conn_security(conn->hcon, sec.level)) {
@@ -987,7 +1020,8 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
}
if (opt == BT_FLUSHABLE_OFF) {
- conn = chan->conn;
+ conn = l2cap_chan_conn(chan);
+
/* proceed further only when we have l2cap_conn and
No Flush support in the LM */
if (!conn || !lmp_no_flush_capable(conn->hcon->hdev)) {
@@ -1111,6 +1145,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c
index 96950bd53ab8..9b034814f29f 100644
--- a/net/bluetooth/mgmt.c
+++ b/net/bluetooth/mgmt.c
@@ -1492,7 +1492,7 @@ static void cmd_complete_rsp(struct mgmt_pending_cmd *cmd, void *data)
return;
}
- cmd_status_rsp(cmd, data);
+ cmd_status_rsp(cmd, &match->mgmt_status);
}
static int generic_cmd_complete(struct mgmt_pending_cmd *cmd, u8 status)
diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c
index 99c295b9f666..d45ea1807f8b 100644
--- a/net/bluetooth/rfcomm/core.c
+++ b/net/bluetooth/rfcomm/core.c
@@ -781,7 +781,7 @@ static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src,
addr.l2_psm = 0;
addr.l2_cid = 0;
addr.l2_bdaddr_type = BDADDR_BREDR;
- *err = kernel_bind(sock, (struct sockaddr *) &addr, sizeof(addr));
+ *err = kernel_bind(sock, (struct sockaddr_unsized *)&addr, sizeof(addr));
if (*err < 0)
goto failed;
@@ -2144,7 +2144,7 @@ static int rfcomm_add_listener(bdaddr_t *ba)
addr.l2_psm = cpu_to_le16(L2CAP_PSM_RFCOMM);
addr.l2_cid = 0;
addr.l2_bdaddr_type = BDADDR_BREDR;
- err = kernel_bind(sock, (struct sockaddr *) &addr, sizeof(addr));
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&addr, sizeof(addr));
if (err < 0) {
BT_ERR("Bind failed %d", err);
goto failed;
diff --git a/net/bluetooth/rfcomm/sock.c b/net/bluetooth/rfcomm/sock.c
index ddf2153aa832..aeb50468c439 100644
--- a/net/bluetooth/rfcomm/sock.c
+++ b/net/bluetooth/rfcomm/sock.c
@@ -50,8 +50,11 @@ static void rfcomm_sock_kill(struct sock *sk);
static void rfcomm_sk_data_ready(struct rfcomm_dlc *d, struct sk_buff *skb)
{
struct sock *sk = d->owner;
- if (!sk)
+
+ if (!sk) {
+ kfree_skb(skb);
return;
+ }
atomic_add(skb->len, &sk->sk_rmem_alloc);
skb_queue_tail(&sk->sk_receive_queue, skb);
@@ -334,7 +337,7 @@ static int rfcomm_sock_create(struct net *net, struct socket *sock,
return 0;
}
-static int rfcomm_sock_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
+static int rfcomm_sock_bind(struct socket *sock, struct sockaddr_unsized *addr, int addr_len)
{
struct sockaddr_rc sa;
struct sock *sk = sock->sk;
diff --git a/net/bluetooth/rfcomm/tty.c b/net/bluetooth/rfcomm/tty.c
index df0f62016652..d09f3afbb79f 100644
--- a/net/bluetooth/rfcomm/tty.c
+++ b/net/bluetooth/rfcomm/tty.c
@@ -131,7 +131,7 @@ static void rfcomm_dev_shutdown(struct tty_port *port)
{
struct rfcomm_dev *dev = container_of(port, struct rfcomm_dev, port);
- if (dev->tty_dev->parent)
+ if (dev->tty_dev && dev->tty_dev->parent)
device_move(dev->tty_dev, NULL, DPM_ORDER_DEV_LAST);
/* close the dlc */
diff --git a/net/bluetooth/sco.c b/net/bluetooth/sco.c
index a25b5f626358..f45bf97ad327 100644
--- a/net/bluetooth/sco.c
+++ b/net/bluetooth/sco.c
@@ -86,12 +86,14 @@ static void sco_conn_free(struct kref *ref)
if (conn->sk)
sco_pi(conn->sk)->conn = NULL;
- if (conn->hcon) {
- conn->hcon->sco_data = NULL;
- hci_conn_drop(conn->hcon);
- }
+ /* hcon->sco_data is cleared and the association's reference on the
+ * sco_conn is dropped in sco_conn_del() under hdev->lock, and the
+ * hci_conn is now owned by the socket (held in __sco_chan_add() and
+ * dropped in sco_chan_del()/sco_sock_destruct()), so there is nothing
+ * left to release towards hcon here.
+ */
- /* Ensure no more work items will run since hci_conn has been dropped */
+ /* Ensure no more work items will run before the connection is freed */
disable_delayed_work_sync(&conn->timeout_work);
kfree(conn);
@@ -190,25 +192,19 @@ static void sco_sock_clear_timer(struct sock *sk)
}
/* ---- SCO connections ---- */
-/* Consumes a reference on @hcon, which the returned sco_conn owns until it is
- * freed. On failure (NULL return) the reference is left for the caller to drop.
+/* Returns a new reference the caller must drop with sco_conn_put(). The
+ * hcon->sco_data association holds its own reference on the sco_conn for the
+ * connection's lifetime; it is dropped in sco_conn_del() under hdev->lock.
+ * @hcon is not consumed: the hci_conn reference is taken and owned by the
+ * socket in __sco_chan_add().
*/
static struct sco_conn *sco_conn_add(struct hci_conn *hcon)
{
struct sco_conn *conn = hcon->sco_data;
conn = sco_conn_hold_unless_zero(conn);
- if (conn) {
- if (!conn->hcon) {
- sco_conn_lock(conn);
- conn->hcon = hcon;
- sco_conn_unlock(conn);
- } else {
- /* conn already owns a reference on hcon */
- hci_conn_drop(hcon);
- }
+ if (conn)
return conn;
- }
conn = kzalloc(sizeof(struct sco_conn), GFP_KERNEL);
if (!conn)
@@ -229,7 +225,10 @@ static struct sco_conn *sco_conn_add(struct hci_conn *hcon)
BT_DBG("hcon %p conn %p", hcon, conn);
- return conn;
+ /* kref_init() above set the association reference owned by
+ * hcon->sco_data; hand the caller its own reference.
+ */
+ return sco_conn_hold(conn);
}
/* Delete channel.
@@ -244,6 +243,19 @@ static void sco_chan_del(struct sock *sk, int err)
BT_DBG("sk %p, conn %p, err %d", sk, conn, err);
if (conn) {
+ struct hci_conn *hcon;
+
+ sco_conn_lock(conn);
+ hcon = conn->hcon;
+ sco_conn_unlock(conn);
+
+ /* Drop the socket's hci_conn reference BEFORE clearing
+ * conn->sk, so sco_conn_del() on another CPU cannot free
+ * the hci_conn while we still hold a pointer to it.
+ */
+ if (hcon)
+ hci_conn_drop(hcon);
+
sco_conn_lock(conn);
conn->sk = NULL;
sco_conn_unlock(conn);
@@ -268,6 +280,13 @@ static void sco_conn_del(struct hci_conn *hcon, int err)
BT_DBG("hcon %p conn %p, err %d", hcon, conn, err);
+ /* Detach from the hci_conn and drop the association's reference.
+ * The caller holds hdev->lock, which serialises this against the
+ * read of hcon->sco_data in sco_recv_scodata().
+ */
+ hcon->sco_data = NULL;
+ sco_conn_put(conn);
+
sco_conn_lock(conn);
sk = sco_sock_hold(conn);
sco_conn_unlock(conn);
@@ -292,6 +311,11 @@ static void __sco_chan_add(struct sco_conn *conn, struct sock *sk,
sco_pi(sk)->conn = sco_conn_hold(conn);
conn->sk = sk;
+ /* The socket owns an hci_conn reference for as long as it stays
+ * attached; it is dropped in sco_chan_del()/sco_sock_destruct().
+ */
+ hci_conn_hold(conn->hcon);
+
if (parent)
bt_accept_enqueue(parent, sk, true);
}
@@ -373,6 +397,7 @@ static int sco_connect(struct sock *sk)
if (sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND) {
release_sock(sk);
sco_conn_put(conn);
+ hci_conn_drop(hcon);
err = -EBADFD;
goto unlock;
}
@@ -381,9 +406,13 @@ static int sco_connect(struct sock *sk)
sco_conn_put(conn);
if (err) {
release_sock(sk);
+ hci_conn_drop(hcon);
goto unlock;
}
+ /* __sco_chan_add() took its own hci_conn reference; drop ours. */
+ hci_conn_drop(hcon);
+
/* Update source addr of the socket */
bacpy(&sco_pi(sk)->src, &hcon->src);
@@ -497,9 +526,25 @@ static struct sock *sco_get_sock_listen(bdaddr_t *src)
static void sco_sock_destruct(struct sock *sk)
{
+ struct sco_conn *conn = sco_pi(sk)->conn;
+
BT_DBG("sk %p", sk);
- sco_conn_put(sco_pi(sk)->conn);
+ /* If the channel was not already torn down via sco_chan_del(), drop
+ * the socket's own references here.
+ */
+ if (conn) {
+ struct hci_conn *hcon;
+
+ sco_conn_lock(conn);
+ hcon = conn->hcon;
+ sco_conn_unlock(conn);
+
+ if (hcon)
+ hci_conn_drop(hcon);
+ sco_pi(sk)->conn = NULL;
+ sco_conn_put(conn);
+ }
skb_queue_purge(&sk->sk_receive_queue);
skb_queue_purge(&sk->sk_write_queue);
@@ -656,7 +701,7 @@ static int sco_sock_create(struct net *net, struct socket *sock, int protocol,
return 0;
}
-static int sco_sock_bind(struct socket *sock, struct sockaddr *addr,
+static int sco_sock_bind(struct socket *sock, struct sockaddr_unsized *addr,
int addr_len)
{
struct sockaddr_sco *sa = (struct sockaddr_sco *) addr;
@@ -1511,12 +1556,10 @@ static void sco_connect_cfm(struct hci_conn *hcon, __u8 status)
if (!status) {
struct sco_conn *conn;
- conn = sco_conn_add(hci_conn_hold(hcon));
+ conn = sco_conn_add(hcon);
if (conn) {
sco_conn_ready(conn);
sco_conn_put(conn);
- } else {
- hci_conn_drop(hcon);
}
} else
sco_conn_del(hcon, bt_to_errno(status));
diff --git a/net/bridge/br_arp_nd_proxy.c b/net/bridge/br_arp_nd_proxy.c
index 4cd3470e5f56..3203706d02b8 100644
--- a/net/bridge/br_arp_nd_proxy.c
+++ b/net/bridge/br_arp_nd_proxy.c
@@ -194,13 +194,15 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br,
n = neigh_lookup(&arp_tbl, &tip, vlandev);
if (n) {
struct net_bridge_fdb_entry *f;
+ u8 ha[ETH_ALEN] __aligned(2);
if (!(READ_ONCE(n->nud_state) & NUD_VALID)) {
neigh_release(n);
return;
}
- f = br_fdb_find_rcu(br, n->ha, vid);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = br_fdb_find_rcu(br, ha, vid);
if (f) {
const struct net_bridge_port *dst = READ_ONCE(f->dst);
bool replied = false;
@@ -210,10 +212,10 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br,
br_is_neigh_suppress_enabled(dst, vid)) {
if (!vid)
br_arp_send(br, p, skb->dev, sip, tip,
- sha, n->ha, sha, 0, 0);
+ sha, ha, sha, 0, 0);
else
br_arp_send(br, p, skb->dev, sip, tip,
- sha, n->ha, sha,
+ sha, ha, sha,
skb->vlan_proto,
skb_vlan_tag_get(skb));
replied = true;
@@ -251,7 +253,7 @@ struct nd_msg *br_is_nd_neigh_msg(const struct sk_buff *skb, struct nd_msg *msg)
}
static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
- struct sk_buff *request, struct neighbour *n,
+ struct sk_buff *request, struct neighbour *n, u8 *ha,
__be16 vlan_proto, u16 vlan_tci)
{
struct net_device *dev = request->dev;
@@ -309,7 +311,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
ipv6_eth_mc_map(&in6addr_linklocal_allnodes, eth_hdr(reply)->h_dest);
else
ether_addr_copy(eth_hdr(reply)->h_dest, daddr);
- ether_addr_copy(eth_hdr(reply)->h_source, n->ha);
+ ether_addr_copy(eth_hdr(reply)->h_source, ha);
eth_hdr(reply)->h_proto = htons(ETH_P_IPV6);
reply->protocol = htons(ETH_P_IPV6);
@@ -339,7 +341,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
na->icmph.icmp6_override = 1;
na->icmph.icmp6_solicited = dad ? 0 : 1;
na->target = ns->target;
- ether_addr_copy(&na->opt[2], n->ha);
+ ether_addr_copy(&na->opt[2], ha);
na->opt[0] = ND_OPT_TARGET_LL_ADDR;
na->opt[1] = na_olen >> 3;
@@ -368,7 +370,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
__vlan_hwaccel_put_tag(reply, vlan_proto, vlan_tci);
netdev_dbg(dev, "nd send dev %s dst %pI6 dst_hw %pM src %pI6 src_hw %pM\n",
- dev->name, &pip6->daddr, daddr, &pip6->saddr, n->ha);
+ dev->name, &pip6->daddr, daddr, &pip6->saddr, ha);
if (p) {
dev_queue_xmit(reply);
@@ -469,24 +471,26 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br,
n = neigh_lookup(ipv6_stub->nd_tbl, &msg->target, vlandev);
if (n) {
struct net_bridge_fdb_entry *f;
+ u8 ha[ETH_ALEN] __aligned(2);
if (!(READ_ONCE(n->nud_state) & NUD_VALID)) {
neigh_release(n);
return;
}
- f = br_fdb_find_rcu(br, n->ha, vid);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = br_fdb_find_rcu(br, ha, vid);
if (f) {
const struct net_bridge_port *dst = READ_ONCE(f->dst);
bool replied = false;
if (br_is_neigh_suppress_enabled(dst, vid)) {
if (vid != 0)
- br_nd_send(br, p, skb, n,
+ br_nd_send(br, p, skb, n, ha,
skb->vlan_proto,
skb_vlan_tag_get(skb));
else
- br_nd_send(br, p, skb, n, 0, 0);
+ br_nd_send(br, p, skb, n, ha, 0, 0);
replied = true;
}
diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c
index fda1f89ee22a..78682aace574 100644
--- a/net/bridge/br_multicast.c
+++ b/net/bridge/br_multicast.c
@@ -81,6 +81,10 @@ __br_multicast_add_group(struct net_bridge_mcast *brmctx,
bool blocked);
static void br_multicast_find_del_pg(struct net_bridge *br,
struct net_bridge_port_group *pg);
+static void __br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
+ struct net_bridge_port_group *pg,
+ struct net_bridge_port_group __rcu **pp,
+ bool sg_del_exclude_ports);
static void __br_multicast_stop(struct net_bridge_mcast *brmctx);
static int br_mc_disabled_update(struct net_device *dev, bool value,
@@ -458,7 +462,7 @@ static void br_multicast_sg_del_exclude_ports(struct net_bridge_mdb_entry *sgmp)
for (pp = &sgmp->ports;
(p = mlock_dereference(*pp, sgmp->br)) != NULL;) {
if (!(p->flags & MDB_PG_FLAGS_PERMANENT))
- br_multicast_del_pg(sgmp, p, pp);
+ __br_multicast_del_pg(sgmp, p, pp, false);
else
pp = &p->next;
}
@@ -799,9 +803,10 @@ static void br_multicast_destroy_port_group(struct net_bridge_mcast_gc *gc)
kfree_rcu(pg, rcu);
}
-void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
- struct net_bridge_port_group *pg,
- struct net_bridge_port_group __rcu **pp)
+static void __br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
+ struct net_bridge_port_group *pg,
+ struct net_bridge_port_group __rcu **pp,
+ bool sg_del_exclude_ports)
{
struct net_bridge *br = pg->key.port->br;
struct net_bridge_group_src *ent;
@@ -816,7 +821,8 @@ void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
if (!br_multicast_is_star_g(&mp->addr)) {
rhashtable_remove_fast(&br->sg_port_tbl, &pg->rhnode,
br_sg_port_rht_params);
- br_multicast_sg_del_exclude_ports(mp);
+ if (sg_del_exclude_ports)
+ br_multicast_sg_del_exclude_ports(mp);
} else {
br_multicast_star_g_handle_mode(pg, MCAST_INCLUDE);
}
@@ -828,6 +834,13 @@ void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
mod_timer(&mp->timer, jiffies);
}
+void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
+ struct net_bridge_port_group *pg,
+ struct net_bridge_port_group __rcu **pp)
+{
+ __br_multicast_del_pg(mp, pg, pp, true);
+}
+
static void br_multicast_find_del_pg(struct net_bridge *br,
struct net_bridge_port_group *pg)
{
diff --git a/net/bridge/netfilter/nf_conntrack_bridge.c b/net/bridge/netfilter/nf_conntrack_bridge.c
index e4dae5fe0b0d..cfa809110242 100644
--- a/net/bridge/netfilter/nf_conntrack_bridge.c
+++ b/net/bridge/netfilter/nf_conntrack_bridge.c
@@ -40,9 +40,13 @@ static int nf_br_ip_fragment(struct net *net, struct sock *sk,
int err = 0;
/* for offloaded checksums cleanup checksum before fragmentation */
- if (skb->ip_summed == CHECKSUM_PARTIAL &&
- (err = skb_checksum_help(skb)))
- goto blackhole;
+ if (skb->ip_summed == CHECKSUM_PARTIAL) {
+ err = ip_check_csum_start(skb);
+ if (!err)
+ err = skb_checksum_help(skb);
+ if (err)
+ goto blackhole;
+ }
iph = ip_hdr(skb);
diff --git a/net/can/af_can.c b/net/can/af_can.c
index 7a026f3a70a0..52d72d56f2dd 100644
--- a/net/can/af_can.c
+++ b/net/can/af_can.c
@@ -644,10 +644,8 @@ static int can_rcv_filter(struct can_dev_rcv_lists *dev_rcv_lists, struct sk_buf
void can_set_skb_uid(struct sk_buff *skb)
{
/* create non-zero unique skb identifier together with *skb */
- while (!(skb->hash))
- skb->hash = atomic_inc_return(&skbcounter);
-
- skb->sw_hash = 1;
+ while (!(can_skb_prv(skb)->skbcnt))
+ can_skb_prv(skb)->skbcnt = atomic_inc_return(&skbcounter);
}
EXPORT_SYMBOL(can_set_skb_uid);
diff --git a/net/can/bcm.c b/net/can/bcm.c
index 67d5e3ecc04f..6d67af0a3b14 100644
--- a/net/can/bcm.c
+++ b/net/can/bcm.c
@@ -350,6 +350,7 @@ static void bcm_can_tx(struct bcm_op *op, struct canfd_frame *cf)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb_put_data(skb, cf, op->cfsiz);
@@ -1619,6 +1620,7 @@ static int bcm_tx_send(struct msghdr *msg, int ifindex, struct sock *sk,
}
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb->dev = dev;
can_skb_set_owner(skb, sk);
err = can_send(skb, 1); /* send with loopback */
diff --git a/net/can/isotp.c b/net/can/isotp.c
index a77ac3954e41..525ed477d15c 100644
--- a/net/can/isotp.c
+++ b/net/can/isotp.c
@@ -302,6 +302,7 @@ static int isotp_send_fc(struct sock *sk, int ae, u8 flowstatus)
can_skb_reserve(nskb);
can_skb_prv(nskb)->ifindex = dev->ifindex;
+ can_skb_prv(nskb)->skbcnt = 0;
nskb->dev = dev;
can_skb_set_owner(nskb, sk);
@@ -876,6 +877,7 @@ static void isotp_send_cframe(struct isotp_sock *so)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* set uid in tx skb to identify CF echo frames */
can_set_skb_uid(skb);
@@ -904,7 +906,7 @@ static void isotp_send_cframe(struct isotp_sock *so)
pr_notice_once("can-isotp: cfecho is %08X != 0\n", old_cfecho);
/* set consecutive frame echo tag */
- WRITE_ONCE(so->cfecho, skb->hash);
+ WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt);
/* send frame with local echo enabled */
can_send_ret = can_send(skb, 1);
@@ -967,7 +969,7 @@ static void isotp_rcv_echo(struct sk_buff *skb, void *data)
spin_lock(&so->rx_lock);
/* so->cfecho may since belong to a new transfer; recheck under lock */
- if (READ_ONCE(so->cfecho) != skb->hash)
+ if (READ_ONCE(so->cfecho) != can_skb_prv(skb)->skbcnt)
goto out_unlock;
/* cancel local echo timeout */
@@ -1199,6 +1201,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* set uid in tx skb to identify CF echo frames */
can_set_skb_uid(skb);
@@ -1240,7 +1243,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
cf->data[ae] |= size;
/* set CF echo tag for isotp_rcv_echo() (SF-mode) */
- WRITE_ONCE(so->cfecho, skb->hash);
+ WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt);
} else {
/* send first frame */
@@ -1257,7 +1260,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
so->txfc.bs = 0;
/* set CF echo tag for isotp_rcv_echo() (CF-mode) */
- WRITE_ONCE(so->cfecho, skb->hash);
+ WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt);
} else {
/* standard flow control check */
new_state = ISOTP_WAIT_FIRST_FC;
@@ -1510,7 +1513,7 @@ static int isotp_release(struct socket *sock)
return 0;
}
-static int isotp_bind(struct socket *sock, struct sockaddr *uaddr, int len)
+static int isotp_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int len)
{
struct sockaddr_can *addr = (struct sockaddr_can *)uaddr;
struct sock *sk = sock->sk;
diff --git a/net/can/j1939/socket.c b/net/can/j1939/socket.c
index 70a57afdef1a..852e1b1b805f 100644
--- a/net/can/j1939/socket.c
+++ b/net/can/j1939/socket.c
@@ -441,7 +441,7 @@ static int j1939_sk_sanity_check(struct sockaddr_can *addr, int len)
return 0;
}
-static int j1939_sk_bind(struct socket *sock, struct sockaddr *uaddr, int len)
+static int j1939_sk_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int len)
{
struct sockaddr_can *addr = (struct sockaddr_can *)uaddr;
struct j1939_sock *jsk = j1939_sk(sock->sk);
@@ -898,6 +898,7 @@ static struct sk_buff *j1939_sk_alloc_skb(struct net_device *ndev,
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = ndev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb_reserve(skb, offsetof(struct can_frame, data));
ret = memcpy_from_msg(skb_put(skb, size), msg, size);
diff --git a/net/can/j1939/transport.c b/net/can/j1939/transport.c
index 9d9c342d26d5..72f6c06e1e5b 100644
--- a/net/can/j1939/transport.c
+++ b/net/can/j1939/transport.c
@@ -613,6 +613,7 @@ sk_buff *j1939_tp_tx_dat_new(struct j1939_priv *priv,
skb->dev = priv->ndev;
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = priv->ndev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* reserve CAN header */
skb_reserve(skb, offsetof(struct can_frame, data));
@@ -1548,6 +1549,7 @@ j1939_session *j1939_session_fresh_new(struct j1939_priv *priv,
skb->dev = priv->ndev;
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = priv->ndev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skcb = j1939_skb_to_cb(skb);
memcpy(skcb, rel_skcb, sizeof(*skcb));
diff --git a/net/can/raw.c b/net/can/raw.c
index dc02defa560d..def1d83c567c 100644
--- a/net/can/raw.c
+++ b/net/can/raw.c
@@ -75,8 +75,8 @@ MODULE_ALIAS("can-proto-1");
*/
struct uniqframe {
+ u32 skbcnt;
const struct sk_buff *skb;
- u32 hash;
unsigned int join_rx_count;
};
@@ -164,7 +164,7 @@ static void raw_rcv(struct sk_buff *oskb, void *data)
/* eliminate multiple filter matches for the same skb */
if (this_cpu_ptr(ro->uniq)->skb == oskb &&
- this_cpu_ptr(ro->uniq)->hash == oskb->hash) {
+ this_cpu_ptr(ro->uniq)->skbcnt == can_skb_prv(oskb)->skbcnt) {
if (!ro->join_filters)
return;
@@ -174,7 +174,7 @@ static void raw_rcv(struct sk_buff *oskb, void *data)
return;
} else {
this_cpu_ptr(ro->uniq)->skb = oskb;
- this_cpu_ptr(ro->uniq)->hash = oskb->hash;
+ this_cpu_ptr(ro->uniq)->skbcnt = can_skb_prv(oskb)->skbcnt;
this_cpu_ptr(ro->uniq)->join_rx_count = 1;
/* drop first frame to check all enabled filters? */
if (ro->join_filters && ro->count > 1)
@@ -459,7 +459,7 @@ static int raw_release(struct socket *sock)
return 0;
}
-static int raw_bind(struct socket *sock, struct sockaddr *uaddr, int len)
+static int raw_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int len)
{
struct sockaddr_can *addr = (struct sockaddr_can *)uaddr;
struct sock *sk = sock->sk;
@@ -955,6 +955,7 @@ static int raw_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* fill the skb before testing for valid CAN frames */
err = memcpy_from_msg(skb_put(skb, size), msg, size);
diff --git a/net/ceph/auth_x.c b/net/ceph/auth_x.c
index c83559f015e4..f805a813308f 100644
--- a/net/ceph/auth_x.c
+++ b/net/ceph/auth_x.c
@@ -86,6 +86,11 @@ static int __ceph_x_decrypt(struct ceph_crypto_key *secret, void *p,
if (ret)
return ret;
+ if (plaintext_len < sizeof(*hdr)) {
+ pr_err("%s plaintext too small %d\n", __func__, plaintext_len);
+ return -EINVAL;
+ }
+
if (le64_to_cpu(hdr->magic) != CEPHX_ENC_MAGIC) {
pr_err("%s bad magic\n", __func__);
return -EINVAL;
diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c
index 2e538399757f..eac5b710efcf 100644
--- a/net/core/bpf_sk_storage.c
+++ b/net/core/bpf_sk_storage.c
@@ -50,16 +50,14 @@ void bpf_sk_storage_free(struct sock *sk)
{
struct bpf_local_storage *sk_storage;
- migrate_disable();
- rcu_read_lock();
+ rcu_read_lock_dont_migrate();
sk_storage = rcu_dereference(sk->sk_bpf_storage);
if (!sk_storage)
goto out;
bpf_local_storage_destroy(sk_storage);
out:
- rcu_read_unlock();
- migrate_enable();
+ rcu_read_unlock_migrate();
}
static void bpf_sk_storage_map_free(struct bpf_map *map)
@@ -159,10 +157,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)
struct bpf_local_storage_elem *selem;
int ret = 0;
- RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
-
- migrate_disable();
- rcu_read_lock();
+ rcu_read_lock_dont_migrate();
sk_storage = rcu_dereference(sk->sk_bpf_storage);
if (!sk_storage || hlist_empty(&sk_storage->list))
@@ -213,8 +208,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)
}
out:
- rcu_read_unlock();
- migrate_enable();
+ rcu_read_unlock_migrate();
/* In case of an error, don't free anything explicitly here, the
* caller is responsible to call bpf_sk_storage_free.
diff --git a/net/core/dev.c b/net/core/dev.c
index 10e02845fea4..0bfd5e9940df 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -231,10 +231,13 @@ static bool use_backlog_threads(void)
static inline void backlog_lock_irq_save(struct softnet_data *sd,
unsigned long *flags)
{
- if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads())
+ if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
spin_lock_irqsave(&sd->input_pkt_queue.lock, *flags);
- else
+ } else {
local_irq_save(*flags);
+ if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads())
+ spin_lock(&sd->input_pkt_queue.lock);
+ }
}
static inline void backlog_lock_irq_disable(struct softnet_data *sd)
@@ -246,12 +249,15 @@ static inline void backlog_lock_irq_disable(struct softnet_data *sd)
}
static inline void backlog_unlock_irq_restore(struct softnet_data *sd,
- unsigned long *flags)
+ unsigned long flags)
{
- if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads())
- spin_unlock_irqrestore(&sd->input_pkt_queue.lock, *flags);
- else
- local_irq_restore(*flags);
+ if (IS_ENABLED(CONFIG_PREEMPT_RT)) {
+ spin_unlock_irqrestore(&sd->input_pkt_queue.lock, flags);
+ } else {
+ if (IS_ENABLED(CONFIG_RPS) || use_backlog_threads())
+ spin_unlock(&sd->input_pkt_queue.lock);
+ local_irq_restore(flags);
+ }
}
static inline void backlog_unlock_irq_enable(struct softnet_data *sd)
@@ -5271,7 +5277,7 @@ void kick_defer_list_purge(unsigned int cpu)
if (!__test_and_set_bit(NAPI_STATE_SCHED, &sd->backlog.state))
__napi_schedule_irqoff(&sd->backlog);
- backlog_unlock_irq_restore(sd, &flags);
+ backlog_unlock_irq_restore(sd, flags);
} else if (!cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) {
smp_call_function_single_async(cpu, &sd->defer_csd);
@@ -5282,14 +5288,15 @@ void kick_defer_list_purge(unsigned int cpu)
int netdev_flow_limit_table_len __read_mostly = (1 << 12);
#endif
-static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen)
+static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen,
+ int max_backlog)
{
#ifdef CONFIG_NET_FLOW_LIMIT
- struct sd_flow_limit *fl;
- struct softnet_data *sd;
unsigned int old_flow, new_flow;
+ const struct softnet_data *sd;
+ struct sd_flow_limit *fl;
- if (qlen < (READ_ONCE(net_hotdata.max_backlog) >> 1))
+ if (likely(qlen < (max_backlog >> 1)))
return false;
sd = this_cpu_ptr(&softnet_data);
@@ -5334,19 +5341,21 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
u32 tail;
reason = SKB_DROP_REASON_DEV_READY;
- if (!netif_running(skb->dev))
- goto bad_dev;
- reason = SKB_DROP_REASON_CPU_BACKLOG;
sd = &per_cpu(softnet_data, cpu);
qlen = skb_queue_len_lockless(&sd->input_pkt_queue);
max_backlog = READ_ONCE(net_hotdata.max_backlog);
- if (unlikely(qlen > max_backlog))
+ if (unlikely(qlen > max_backlog) ||
+ skb_flow_limit(skb, qlen, max_backlog))
goto cpu_backlog_drop;
backlog_lock_irq_save(sd, &flags);
qlen = skb_queue_len(&sd->input_pkt_queue);
- if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) {
+ if (likely(qlen <= max_backlog)) {
+ if (unlikely(!netif_running(skb->dev))) {
+ backlog_unlock_irq_restore(sd, flags);
+ goto bad_dev;
+ }
if (!qlen) {
/* Schedule NAPI for backlog device. We can use
* non atomic operation as we own the queue lock.
@@ -5357,16 +5366,17 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
}
__skb_queue_tail(&sd->input_pkt_queue, skb);
tail = rps_input_queue_tail_incr(sd);
- backlog_unlock_irq_restore(sd, &flags);
+ backlog_unlock_irq_restore(sd, flags);
/* save the tail outside of the critical section */
rps_input_queue_tail_save(qtail, tail);
return NET_RX_SUCCESS;
}
- backlog_unlock_irq_restore(sd, &flags);
+ backlog_unlock_irq_restore(sd, flags);
cpu_backlog_drop:
+ reason = SKB_DROP_REASON_CPU_BACKLOG;
numa_drop_add(&sd->drop_counters, 1);
bad_dev:
dev_core_stats_rx_dropped_inc(skb->dev);
@@ -10203,13 +10213,15 @@ EXPORT_SYMBOL_GPL(dev_xdp_prog_count);
u8 dev_xdp_sb_prog_count(struct net_device *dev)
{
+ struct bpf_prog *prog;
u8 count = 0;
int i;
- for (i = 0; i < __MAX_XDP_MODE; i++)
- if (dev->xdp_state[i].prog &&
- !dev->xdp_state[i].prog->aux->xdp_has_frags)
+ for (i = 0; i < __MAX_XDP_MODE; i++) {
+ prog = dev_xdp_prog(dev, i);
+ if (prog && !prog->aux->xdp_has_frags)
count++;
+ }
return count;
}
@@ -10589,11 +10601,12 @@ static int bpf_xdp_link_update(struct bpf_link *link, struct bpf_prog *new_prog,
bpf_op = dev_xdp_bpf_op(xdp_link->dev, mode);
err = dev_xdp_install(xdp_link->dev, mode, bpf_op, NULL,
xdp_link->flags, new_prog);
+ if (!err)
+ old_prog = xchg(&link->prog, new_prog);
netdev_unlock_ops(xdp_link->dev);
if (err)
goto out_unlock;
- old_prog = xchg(&link->prog, new_prog);
bpf_prog_put(old_prog);
out_unlock:
diff --git a/net/core/filter.c b/net/core/filter.c
index 308b86abe460..692690316359 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -7541,7 +7541,7 @@ BPF_CALL_5(bpf_tcp_check_syncookie, struct sock *, sk, void *, iph, u32, iph_len
return -EINVAL;
/* sk_listener() allows TCP_NEW_SYN_RECV, which makes no sense here. */
- if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN)
+ if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP)
return -EINVAL;
if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies))
@@ -7614,7 +7614,7 @@ BPF_CALL_5(bpf_tcp_gen_syncookie, struct sock *, sk, void *, iph, u32, iph_len,
if (unlikely(!sk || th_len < sizeof(*th) || th_len != th->doff * 4))
return -EINVAL;
- if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN)
+ if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP)
return -EINVAL;
if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies))
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 8aa4f9b4df81..1be6c739e3c7 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -1023,6 +1023,8 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
result = bpf_prog_run_pin_on_cpu(prog, ctx);
+ /* bpf_flow_keys offsets are u16: do not let @hlen be truncated. */
+ hlen = min_t(int, hlen, U16_MAX);
flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
flow_keys->nhoff, hlen);
@@ -1071,6 +1073,7 @@ bool __skb_flow_dissect(const struct net *net,
int mpls_lse = 0;
int num_hdrs = 0;
u8 ip_proto = 0;
+ u32 thoff;
bool ret;
if (!data) {
@@ -1692,7 +1695,13 @@ bool __skb_flow_dissect(const struct net *net,
ret = true;
out:
- key_control->thoff = min_t(u16, nhoff, skb ? skb->len : hlen);
+ thoff = min_t(u32, nhoff, skb ? skb->len : hlen);
+ if (unlikely(thoff > U16_MAX)) {
+ /* Cannot be represented in key_control->thoff. */
+ thoff = U16_MAX;
+ ret = false;
+ }
+ key_control->thoff = thoff;
key_basic->n_proto = proto;
key_basic->ip_proto = ip_proto;
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 559b027b7a2c..cdce2f72fbea 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3582,8 +3582,12 @@ __wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset,
pos = copy;
}
- if (!skb_frags_readable(skb))
+ if (!skb_frags_readable(skb)) {
+ /* Don't hand the caller a buffer with stale bytes in it. */
+ if (len > 0)
+ memset(to, 0, len);
return 0;
+ }
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
int end;
diff --git a/net/core/sock.c b/net/core/sock.c
index 9031349d08c2..4dc2b03509f8 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2468,6 +2468,9 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority)
sock_copy(newsk, sk);
newsk->sk_prot_creator = prot;
+#ifdef CONFIG_BPF_SYSCALL
+ RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
+#endif
/* SANITY */
if (likely(newsk->sk_net_refcnt)) {
@@ -3427,7 +3430,7 @@ EXPORT_SYMBOL_GPL(sk_set_peek_off);
* function, some default processing is provided.
*/
-int sock_no_bind(struct socket *sock, struct sockaddr *saddr, int len)
+int sock_no_bind(struct socket *sock, struct sockaddr_unsized *saddr, int len)
{
return -EOPNOTSUPP;
}
diff --git a/net/devlink/core.c b/net/devlink/core.c
index d8f875b22989..0a31f28c8128 100644
--- a/net/devlink/core.c
+++ b/net/devlink/core.c
@@ -96,13 +96,19 @@ static void devlink_rel_nested_in_notify_work(struct work_struct *work)
return;
reschedule_work:
- schedule_delayed_work(&rel->nested_in.notify_work, 1);
+ /* The work may have been queued again meanwhile, which took its own
+ * reference. Drop ours in that case.
+ */
+ if (!schedule_delayed_work(&rel->nested_in.notify_work, 1))
+ __devlink_rel_put(rel);
}
static void devlink_rel_nested_in_notify_work_schedule(struct devlink_rel *rel)
{
__devlink_rel_get(rel);
- schedule_delayed_work(&rel->nested_in.notify_work, 0);
+ /* The pending work holds a reference already, drop the new one. */
+ if (!schedule_delayed_work(&rel->nested_in.notify_work, 0))
+ __devlink_rel_put(rel);
}
static struct devlink_rel *devlink_rel_alloc(void)
diff --git a/net/hsr/hsr_framereg.c b/net/hsr/hsr_framereg.c
index bd2fbbc4420b..f366429aff88 100644
--- a/net/hsr/hsr_framereg.c
+++ b/net/hsr/hsr_framereg.c
@@ -129,8 +129,10 @@ void hsr_del_nodes(struct list_head *node_db)
struct hsr_node *node;
struct hsr_node *tmp;
- list_for_each_entry_safe(node, tmp, node_db, mac_list)
- kfree(node);
+ list_for_each_entry_safe(node, tmp, node_db, mac_list) {
+ list_del_rcu(&node->mac_list);
+ kfree_rcu(node, rcu_head);
+ }
}
void prp_handle_san_frame(bool san, enum hsr_port_type port,
diff --git a/net/ieee802154/socket.c b/net/ieee802154/socket.c
index dc0de5ae6afb..550485ec32da 100644
--- a/net/ieee802154/socket.c
+++ b/net/ieee802154/socket.c
@@ -96,13 +96,13 @@ static int ieee802154_sock_sendmsg(struct socket *sock, struct msghdr *msg,
return sk->sk_prot->sendmsg(sk, msg, len);
}
-static int ieee802154_sock_bind(struct socket *sock, struct sockaddr *uaddr,
+static int ieee802154_sock_bind(struct socket *sock, struct sockaddr_unsized *uaddr,
int addr_len)
{
struct sock *sk = sock->sk;
if (sk->sk_prot->bind)
- return sk->sk_prot->bind(sk, uaddr, addr_len);
+ return sk->sk_prot->bind(sk, (struct sockaddr *)uaddr, addr_len);
return sock_no_bind(sock, uaddr, addr_len);
}
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index fbde60bf52a2..1d1a03fad1f9 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -464,9 +464,9 @@ int inet_bind_sk(struct sock *sk, struct sockaddr *uaddr, int addr_len)
return __inet_bind(sk, uaddr, addr_len, flags);
}
-int inet_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+int inet_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
- return inet_bind_sk(sock->sk, uaddr, addr_len);
+ return inet_bind_sk(sock->sk, (struct sockaddr *)uaddr, addr_len);
}
EXPORT_SYMBOL(inet_bind);
diff --git a/net/ipv4/cipso_ipv4.c b/net/ipv4/cipso_ipv4.c
index 709021197e1c..e741f0a47afb 100644
--- a/net/ipv4/cipso_ipv4.c
+++ b/net/ipv4/cipso_ipv4.c
@@ -2287,6 +2287,14 @@ int cipso_v4_skbuff_delattr(struct sk_buff *skb)
new_hdr_len - new_hdr_len_actual);
opt->optlen -= hdr_len_delta;
+ if (opt->srr > opt->cipso)
+ opt->srr -= cipso_len;
+ if (opt->rr > opt->cipso)
+ opt->rr -= cipso_len;
+ if (opt->ts > opt->cipso)
+ opt->ts -= cipso_len;
+ if (opt->router_alert > opt->cipso)
+ opt->router_alert -= cipso_len;
opt->cipso = 0;
opt->is_changed = 1;
if (hdr_len_delta != 0) {
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index 346e78c23af0..53284524cf4f 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -104,6 +104,13 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
struct aead_request *req;
struct scatterlist *sg;
+ /* Managed frags are owned by the zerocopy ubuf; the skb holds no
+ * per-frag page reference, so we must not drop one here. Mirrors
+ * the SKBFL_MANAGED_FRAG_REFS handling in skb_release_data().
+ */
+ if (skb_zcopy_managed(skb))
+ return;
+
if (x->props.flags & XFRM_STATE_ESN)
extralen += sizeof(struct esp_output_extra);
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 5ab23a1fa45f..08d1bbaafc79 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -544,6 +544,9 @@ size_t fib_nlmsg_size(struct fib_info *fi)
struct fib_nh_common *nhc = fib_info_nhc(fi, i);
size_t nhsize;
+ if (!nhc)
+ break;
+
nhsize = fib_nexthop_nlmsg_size(nhc, nhs != 1);
if (nhs != 1)
diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c
index 9bf5550e2fb7..77a3598d3385 100644
--- a/net/ipv4/fib_trie.c
+++ b/net/ipv4/fib_trie.c
@@ -1079,8 +1079,6 @@ void fib_alias_hw_flags_set(struct net *net, const struct fib_rt_info *fri)
err = fib_dump_info(skb, 0, 0, RTM_NEWROUTE, fri, 0);
if (err < 0) {
- /* -EMSGSIZE implies BUG in fib_nlmsg_size() */
- WARN_ON(err == -EMSGSIZE);
kfree_skb(skb);
goto errout;
}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 4c4e9d1940f7..fedc2bf7ce7f 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -772,9 +772,13 @@ int ip_do_fragment(struct net *net, struct sock *sk, struct sk_buff *skb,
int err = 0;
/* for offloaded checksums cleanup checksum before fragmentation */
- if (skb->ip_summed == CHECKSUM_PARTIAL &&
- (err = skb_checksum_help(skb)))
- goto fail;
+ if (skb->ip_summed == CHECKSUM_PARTIAL) {
+ err = ip_check_csum_start(skb);
+ if (!err)
+ err = skb_checksum_help(skb);
+ if (err)
+ goto fail;
+ }
/*
* Point into the IP datagram header.
diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c
index 1490466b146e..0ea513bf77fb 100644
--- a/net/ipv4/netfilter/arp_tables.c
+++ b/net/ipv4/netfilter/arp_tables.c
@@ -1518,13 +1518,11 @@ int arpt_register_table(struct net *net,
const struct arpt_replace *repl,
const struct nf_hook_ops *template_ops)
{
- struct nf_hook_ops *ops;
- unsigned int num_ops;
- int ret, i;
- struct xt_table_info *newinfo;
struct xt_table_info bootstrap = {0};
- void *loc_cpu_entry;
+ struct xt_table_info *newinfo;
struct xt_table *new_table;
+ void *loc_cpu_entry;
+ int ret;
newinfo = xt_alloc_table_info(repl->size);
if (!newinfo)
@@ -1539,7 +1537,7 @@ int arpt_register_table(struct net *net,
return ret;
}
- new_table = xt_register_table(net, table, &bootstrap, newinfo);
+ new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo);
if (IS_ERR(new_table)) {
struct arpt_entry *iter;
@@ -1549,31 +1547,6 @@ int arpt_register_table(struct net *net,
return PTR_ERR(new_table);
}
- num_ops = hweight32(table->valid_hooks);
- if (num_ops == 0) {
- ret = -EINVAL;
- goto out_free;
- }
-
- ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
- if (!ops) {
- ret = -ENOMEM;
- goto out_free;
- }
-
- for (i = 0; i < num_ops; i++)
- ops[i].priv = new_table;
-
- new_table->ops = ops;
-
- ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret != 0)
- goto out_free;
-
- return ret;
-
-out_free:
- __arpt_unregister_table(net, new_table);
return ret;
}
diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index 0549fad53c03..ca8ff0ae6cdb 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -1720,13 +1720,11 @@ int ipt_register_table(struct net *net, const struct xt_table *table,
const struct ipt_replace *repl,
const struct nf_hook_ops *template_ops)
{
- struct nf_hook_ops *ops;
- unsigned int num_ops;
- int ret, i;
- struct xt_table_info *newinfo;
struct xt_table_info bootstrap = {0};
- void *loc_cpu_entry;
+ struct xt_table_info *newinfo;
struct xt_table *new_table;
+ void *loc_cpu_entry;
+ int ret;
newinfo = xt_alloc_table_info(repl->size);
if (!newinfo)
@@ -1741,7 +1739,7 @@ int ipt_register_table(struct net *net, const struct xt_table *table,
return ret;
}
- new_table = xt_register_table(net, table, &bootstrap, newinfo);
+ new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo);
if (IS_ERR(new_table)) {
struct ipt_entry *iter;
@@ -1751,37 +1749,6 @@ int ipt_register_table(struct net *net, const struct xt_table *table,
return PTR_ERR(new_table);
}
- /* No template? No need to do anything. This is used by 'nat' table, it registers
- * with the nat core instead of the netfilter core.
- */
- if (!template_ops)
- return 0;
-
- num_ops = hweight32(table->valid_hooks);
- if (num_ops == 0) {
- ret = -EINVAL;
- goto out_free;
- }
-
- ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
- if (!ops) {
- ret = -ENOMEM;
- goto out_free;
- }
-
- for (i = 0; i < num_ops; i++)
- ops[i].priv = new_table;
-
- new_table->ops = ops;
-
- ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret != 0)
- goto out_free;
-
- return ret;
-
-out_free:
- __ipt_unregister_table(net, new_table);
return ret;
}
diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c
index 66d5c996866b..21bd8b66e2fe 100644
--- a/net/ipv4/nexthop.c
+++ b/net/ipv4/nexthop.c
@@ -1600,14 +1600,21 @@ static int fib6_check_nh_list(struct nexthop *old, struct nexthop *new,
struct netlink_ext_ack *extack)
{
struct fib6_info *f6i;
+ int err = 0;
if (list_empty(&old->f6i_list))
return 0;
+ spin_lock_bh(&old->lock);
list_for_each_entry(f6i, &old->f6i_list, nh_list) {
- if (check_src_addr(&f6i->fib6_src.addr, extack) < 0)
- return -EINVAL;
+ err = check_src_addr(&f6i->fib6_src.addr, extack);
+ if (err)
+ break;
}
+ spin_unlock_bh(&old->lock);
+
+ if (err)
+ return err;
return fib6_check_nexthop(new, NULL, extack);
}
@@ -2219,15 +2226,18 @@ static void remove_nexthop(struct net *net, struct nexthop *nh,
static void nh_rt_cache_flush(struct net *net, struct nexthop *nh,
struct nexthop *replaced_nh)
{
- struct fib6_info *f6i;
struct nh_group *nhg;
+ bool have_f6i;
int i;
if (!list_empty(&nh->fi_list))
rt_cache_flush(net);
- list_for_each_entry(f6i, &nh->f6i_list, nh_list)
- ipv6_stub->fib6_update_sernum(net, f6i);
+ spin_lock_bh(&nh->lock);
+ have_f6i = !list_empty(&nh->f6i_list);
+ spin_unlock_bh(&nh->lock);
+ if (have_f6i)
+ rt_genid_bump_ipv6(net);
/* if an IPv6 group was replaced, we have to release all old
* dsts to make sure all refcounts are released
@@ -2521,8 +2531,10 @@ static void __nexthop_replace_notify(struct net *net, struct nexthop *nh,
fi->nh_updated = false;
}
+ spin_lock_bh(&nh->lock);
list_for_each_entry(f6i, &nh->f6i_list, nh_list)
ipv6_stub->fib6_rt_update(net, f6i, info);
+ spin_unlock_bh(&nh->lock);
}
/* send RTM_NEWROUTE with REPLACE flag set for all FIB entries
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index a25ab9d263ab..6e51d8f5365d 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1080,6 +1080,8 @@ static void __ip_rt_update_pmtu(struct rtable *rt, struct flowi4 *fl4, u32 mtu)
for (nhsel = 0; nhsel < fib_info_num_path(res.fi); nhsel++) {
nhc = fib_info_nhc(res.fi, nhsel);
+ if (!nhc)
+ break;
update_or_create_fnhe(nhc, fl4->daddr, 0, mtu, lock,
jiffies + net->ipv4.ip_rt_mtu_expires);
}
@@ -3166,6 +3168,9 @@ int fib_dump_info_fnhe(struct sk_buff *skb, struct netlink_callback *cb,
struct fnhe_hash_bucket *bucket;
int err;
+ if (!nhc)
+ break;
+
if (nhc->nhc_flags & RTNH_F_DEAD)
continue;
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 8c0ffe50ae5d..ec8958688c48 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1883,6 +1883,8 @@ static bool can_map_frag(const skb_frag_t *frag)
if (skb_frag_size(frag) != PAGE_SIZE || skb_frag_off(frag))
return false;
+ if (skb_frag_is_net_iov(frag))
+ return false;
page = skb_frag_page(frag);
diff --git a/net/ipv4/udp_tunnel_core.c b/net/ipv4/udp_tunnel_core.c
index 54386e06a813..11e5a88c923d 100644
--- a/net/ipv4/udp_tunnel_core.c
+++ b/net/ipv4/udp_tunnel_core.c
@@ -29,7 +29,7 @@ int udp_sock_create4(struct net *net, struct udp_port_cfg *cfg,
udp_addr.sin_family = AF_INET;
udp_addr.sin_addr = cfg->local_ip;
udp_addr.sin_port = cfg->local_udp_port;
- err = kernel_bind(sock, (struct sockaddr *)&udp_addr,
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&udp_addr,
sizeof(udp_addr));
if (err < 0)
goto error;
diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c
index 1c45e1d15506..3111db617b88 100644
--- a/net/ipv6/af_inet6.c
+++ b/net/ipv6/af_inet6.c
@@ -465,9 +465,9 @@ int inet6_bind_sk(struct sock *sk, struct sockaddr *uaddr, int addr_len)
}
/* bind for INET6 API */
-int inet6_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+int inet6_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
- return inet6_bind_sk(sock->sk, uaddr, addr_len);
+ return inet6_bind_sk(sock->sk, (struct sockaddr *)uaddr, addr_len);
}
EXPORT_SYMBOL(inet6_bind);
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index b26b85afb92e..2f19ff51ab3d 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -121,6 +121,13 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
struct aead_request *req;
struct scatterlist *sg;
+ /* Managed frags are owned by the zerocopy ubuf; the skb holds no
+ * per-frag page reference, so we must not drop one here. Mirrors
+ * the SKBFL_MANAGED_FRAG_REFS handling in skb_release_data().
+ */
+ if (skb_zcopy_managed(skb))
+ return;
+
if (x->props.flags & XFRM_STATE_ESN)
extralen += sizeof(struct esp_output_extra);
diff --git a/net/ipv6/ip6_udp_tunnel.c b/net/ipv6/ip6_udp_tunnel.c
index 0ff547a4bff7..b0d9286b33c8 100644
--- a/net/ipv6/ip6_udp_tunnel.c
+++ b/net/ipv6/ip6_udp_tunnel.c
@@ -40,7 +40,7 @@ int udp_sock_create6(struct net *net, struct udp_port_cfg *cfg,
memcpy(&udp6_addr.sin6_addr, &cfg->local_ip6,
sizeof(udp6_addr.sin6_addr));
udp6_addr.sin6_port = cfg->local_udp_port;
- err = kernel_bind(sock, (struct sockaddr *)&udp6_addr,
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&udp6_addr,
sizeof(udp6_addr));
if (err < 0)
goto error;
diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c
index d59d31ab960a..7b60e4b2a817 100644
--- a/net/ipv6/ip6mr.c
+++ b/net/ipv6/ip6mr.c
@@ -1136,10 +1136,10 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt,
msg->im6_src = ipv6_hdr(pkt)->saddr;
msg->im6_dst = ipv6_hdr(pkt)->daddr;
- skb_dst_set(skb, dst_clone(skb_dst(pkt)));
skb->ip_summed = CHECKSUM_UNNECESSARY;
}
+ skb_dst_drop(skb);
mrt6msg_netlink_event(mrt, skb);
/* Deliver to user space multicast routing algorithms */
@@ -1219,6 +1219,7 @@ static int ip6mr_cache_unresolved(struct mr_table *mrt, mifi_t mifi,
skb->dev = dev;
skb->skb_iif = dev->ifindex;
}
+ skb_dst_drop(skb);
skb_queue_tail(&c->_c.mfc_un.unres.unresolved, skb);
err = 0;
}
diff --git a/net/ipv6/mcast.c b/net/ipv6/mcast.c
index c560f3e3b11f..f37357f24a88 100644
--- a/net/ipv6/mcast.c
+++ b/net/ipv6/mcast.c
@@ -240,7 +240,8 @@ static int __ipv6_sock_mc_join(struct sock *sk, int ifindex,
return err;
}
- mc_lst->next = np->ipv6_mc_list;
+ rcu_assign_pointer(mc_lst->next,
+ sock_dereference(np->ipv6_mc_list, sk));
rcu_assign_pointer(np->ipv6_mc_list, mc_lst);
return 0;
@@ -300,7 +301,8 @@ int ipv6_sock_mc_drop(struct sock *sk, int ifindex, const struct in6_addr *addr)
lnk = &mc_lst->next) {
if ((ifindex == 0 || mc_lst->ifindex == ifindex) &&
ipv6_addr_equal(&mc_lst->addr, addr)) {
- *lnk = mc_lst->next;
+ rcu_assign_pointer(*lnk,
+ sock_dereference(mc_lst->next, sk));
__ipv6_sock_mc_drop(sk, mc_lst);
return 0;
}
@@ -333,7 +335,8 @@ void __ipv6_sock_mc_close(struct sock *sk)
struct ipv6_mc_socklist *mc_lst;
while ((mc_lst = sock_dereference(np->ipv6_mc_list, sk)) != NULL) {
- np->ipv6_mc_list = mc_lst->next;
+ rcu_assign_pointer(np->ipv6_mc_list,
+ sock_dereference(mc_lst->next, sk));
__ipv6_sock_mc_drop(sk, mc_lst);
}
}
@@ -798,9 +801,11 @@ static void mld_del_delrec(struct inet6_dev *idev, struct ifmcaddr6 *im)
if (!pmc)
return;
if (pmc_prev)
- rcu_assign_pointer(pmc_prev->next, pmc->next);
+ rcu_assign_pointer(pmc_prev->next,
+ mc_dereference(pmc->next, idev));
else
- rcu_assign_pointer(idev->mc_tomb, pmc->next);
+ rcu_assign_pointer(idev->mc_tomb,
+ mc_dereference(pmc->next, idev));
im->idev = pmc->idev;
if (im->mca_sfmode == MCAST_INCLUDE) {
@@ -890,7 +895,7 @@ static struct ifmcaddr6 *mca_alloc(struct inet6_dev *idev,
mc->mca_addr = *addr;
mc->idev = idev; /* reference taken by caller */
- mc->mca_users = 1;
+ WRITE_ONCE(mc->mca_users, 1);
/* mca_stamp should be updated upon changes */
mc->mca_cstamp = mc->mca_tstamp = jiffies;
refcount_set(&mc->mca_refcnt, 1);
@@ -964,7 +969,7 @@ static int __ipv6_dev_mc_inc(struct net_device *dev,
for_each_mc_mclock(idev, mc) {
if (ipv6_addr_equal(&mc->mca_addr, addr)) {
- mc->mca_users++;
+ WRITE_ONCE(mc->mca_users, mc->mca_users + 1);
ip6_mc_add_src(idev, &mc->mca_addr, mode, 0, NULL, 0);
mutex_unlock(&idev->mc_lock);
in6_dev_put(idev);
@@ -979,7 +984,7 @@ static int __ipv6_dev_mc_inc(struct net_device *dev,
return -ENOMEM;
}
- rcu_assign_pointer(mc->next, idev->mc_list);
+ rcu_assign_pointer(mc->next, mc_dereference(idev->mc_list, idev));
rcu_assign_pointer(idev->mc_list, mc);
mld_del_delrec(idev, mc);
@@ -1009,8 +1014,12 @@ int __ipv6_dev_mc_dec(struct inet6_dev *idev, const struct in6_addr *addr)
(ma = mc_dereference(*map, idev));
map = &ma->next) {
if (ipv6_addr_equal(&ma->mca_addr, addr)) {
- if (--ma->mca_users == 0) {
- *map = ma->next;
+ int new_users = ma->mca_users - 1;
+
+ WRITE_ONCE(ma->mca_users, new_users);
+ if (new_users == 0) {
+ rcu_assign_pointer(*map,
+ mc_dereference(ma->next, idev));
igmp6_group_dropped(ma);
inet6_ifmcaddr_notify(idev->dev, ma,
@@ -3021,7 +3030,7 @@ static int igmp6_mc_seq_show(struct seq_file *seq, void *v)
"%-4d %-15s %pi6 %5d %08X %ld\n",
state->dev->ifindex, state->dev->name,
&im->mca_addr,
- im->mca_users, im->mca_flags,
+ READ_ONCE(im->mca_users), im->mca_flags,
(im->mca_flags & MAF_TIMER_RUNNING) ?
jiffies_to_clock_t(im->mca_work.timer.expires - jiffies) : 0);
return 0;
diff --git a/net/ipv6/netfilter/ip6_tables.c b/net/ipv6/netfilter/ip6_tables.c
index 7efc2a19662f..22db263682bd 100644
--- a/net/ipv6/netfilter/ip6_tables.c
+++ b/net/ipv6/netfilter/ip6_tables.c
@@ -1734,13 +1734,11 @@ int ip6t_register_table(struct net *net, const struct xt_table *table,
const struct ip6t_replace *repl,
const struct nf_hook_ops *template_ops)
{
- struct nf_hook_ops *ops;
- unsigned int num_ops;
- int ret, i;
- struct xt_table_info *newinfo;
struct xt_table_info bootstrap = {0};
- void *loc_cpu_entry;
+ struct xt_table_info *newinfo;
struct xt_table *new_table;
+ void *loc_cpu_entry;
+ int ret;
newinfo = xt_alloc_table_info(repl->size);
if (!newinfo)
@@ -1755,7 +1753,7 @@ int ip6t_register_table(struct net *net, const struct xt_table *table,
return ret;
}
- new_table = xt_register_table(net, table, &bootstrap, newinfo);
+ new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo);
if (IS_ERR(new_table)) {
struct ip6t_entry *iter;
@@ -1765,34 +1763,6 @@ int ip6t_register_table(struct net *net, const struct xt_table *table,
return PTR_ERR(new_table);
}
- if (!template_ops)
- return 0;
-
- num_ops = hweight32(table->valid_hooks);
- if (num_ops == 0) {
- ret = -EINVAL;
- goto out_free;
- }
-
- ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
- if (!ops) {
- ret = -ENOMEM;
- goto out_free;
- }
-
- for (i = 0; i < num_ops; i++)
- ops[i].priv = new_table;
-
- new_table->ops = ops;
-
- ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret != 0)
- goto out_free;
-
- return ret;
-
-out_free:
- __ip6t_unregister_table(net, new_table);
return ret;
}
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index e99b4901091a..ce60fe53b822 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -3257,7 +3257,7 @@ void ip6_redirect_no_header(struct sk_buff *skb, struct net *net, int oif)
void ip6_sk_redirect(struct sk_buff *skb, struct sock *sk)
{
- ip6_redirect(skb, sock_net(sk), sk->sk_bound_dev_if,
+ ip6_redirect(skb, sock_net(sk), skb->dev->ifindex,
READ_ONCE(sk->sk_mark), sk_uid(sk));
}
EXPORT_SYMBOL_GPL(ip6_sk_redirect);
@@ -6468,8 +6468,6 @@ void fib6_info_hw_flags_set(struct net *net, struct fib6_info *f6i,
err = rt6_fill_node(net, skb, f6i, NULL, NULL, NULL, 0, RTM_NEWROUTE, 0,
0, 0);
if (err < 0) {
- /* -EMSGSIZE implies BUG in rt6_nlmsg_size() */
- WARN_ON(err == -EMSGSIZE);
kfree_skb(skb);
goto errout;
}
diff --git a/net/iucv/af_iucv.c b/net/iucv/af_iucv.c
index bd5838e65c31..5b48bd30efd2 100644
--- a/net/iucv/af_iucv.c
+++ b/net/iucv/af_iucv.c
@@ -576,7 +576,7 @@ static void __iucv_auto_name(struct iucv_sock *iucv)
}
/* Bind an unbound socket */
-static int iucv_sock_bind(struct socket *sock, struct sockaddr *addr,
+static int iucv_sock_bind(struct socket *sock, struct sockaddr_unsized *addr,
int addr_len)
{
DECLARE_SOCKADDR(struct sockaddr_iucv *, sa, addr);
diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
index fd78928e9be2..60bd9d5e4dea 100644
--- a/net/l2tp/l2tp_core.c
+++ b/net/l2tp/l2tp_core.c
@@ -1511,7 +1511,7 @@ static int l2tp_tunnel_sock_create(struct net *net,
memcpy(&ip6_addr.l2tp_addr, cfg->local_ip6,
sizeof(ip6_addr.l2tp_addr));
ip6_addr.l2tp_conn_id = tunnel_id;
- err = kernel_bind(sock, (struct sockaddr *)&ip6_addr,
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&ip6_addr,
sizeof(ip6_addr));
if (err < 0)
goto out;
@@ -1538,7 +1538,7 @@ static int l2tp_tunnel_sock_create(struct net *net,
ip_addr.l2tp_family = AF_INET;
ip_addr.l2tp_addr = cfg->local_ip;
ip_addr.l2tp_conn_id = tunnel_id;
- err = kernel_bind(sock, (struct sockaddr *)&ip_addr,
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&ip_addr,
sizeof(ip_addr));
if (err < 0)
goto out;
diff --git a/net/llc/af_llc.c b/net/llc/af_llc.c
index 1863cacc63a8..6f6336634ddb 100644
--- a/net/llc/af_llc.c
+++ b/net/llc/af_llc.c
@@ -338,7 +338,7 @@ static int llc_ui_autobind(struct socket *sock, struct sockaddr_llc *addr)
* otherwise all hell will break loose.
* Returns: 0 upon success, negative otherwise.
*/
-static int llc_ui_bind(struct socket *sock, struct sockaddr *uaddr, int addrlen)
+static int llc_ui_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addrlen)
{
struct sockaddr_llc *addr = (struct sockaddr_llc *)uaddr;
struct sock *sk = sock->sk;
diff --git a/net/mac80211/debugfs.c b/net/mac80211/debugfs.c
index 14ce2ea10593..347ae0f63da8 100644
--- a/net/mac80211/debugfs.c
+++ b/net/mac80211/debugfs.c
@@ -210,11 +210,13 @@ static ssize_t aql_pending_read(struct file *file,
"VI %u us\n"
"BE %u us\n"
"BK %u us\n"
+ "MC %u us\n"
"total %u us\n",
atomic_read(&local->aql_ac_pending_airtime[IEEE80211_AC_VO]),
atomic_read(&local->aql_ac_pending_airtime[IEEE80211_AC_VI]),
atomic_read(&local->aql_ac_pending_airtime[IEEE80211_AC_BE]),
atomic_read(&local->aql_ac_pending_airtime[IEEE80211_AC_BK]),
+ atomic_read(&local->aql_mc_pending_airtime),
atomic_read(&local->aql_total_pending_airtime));
return simple_read_from_buffer(user_buf, count, ppos,
buf, len);
@@ -239,7 +241,8 @@ static ssize_t aql_txq_limit_read(struct file *file,
"VO %u %u\n"
"VI %u %u\n"
"BE %u %u\n"
- "BK %u %u\n",
+ "BK %u %u\n"
+ "MC %u\n",
local->aql_txq_limit_low[IEEE80211_AC_VO],
local->aql_txq_limit_high[IEEE80211_AC_VO],
local->aql_txq_limit_low[IEEE80211_AC_VI],
@@ -247,7 +250,8 @@ static ssize_t aql_txq_limit_read(struct file *file,
local->aql_txq_limit_low[IEEE80211_AC_BE],
local->aql_txq_limit_high[IEEE80211_AC_BE],
local->aql_txq_limit_low[IEEE80211_AC_BK],
- local->aql_txq_limit_high[IEEE80211_AC_BK]);
+ local->aql_txq_limit_high[IEEE80211_AC_BK],
+ local->aql_txq_limit_mc);
return simple_read_from_buffer(user_buf, count, ppos,
buf, len);
}
@@ -273,6 +277,11 @@ static ssize_t aql_txq_limit_write(struct file *file,
else
buf[count] = '\0';
+ if (sscanf(buf, "mcast %u", &q_limit_low) == 1) {
+ local->aql_txq_limit_mc = q_limit_low;
+ return count;
+ }
+
if (sscanf(buf, "%u %u %u", &ac, &q_limit_low, &q_limit_high) != 3)
return -EINVAL;
diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h
index 01e4edf315ae..43ae7200875c 100644
--- a/net/mac80211/ieee80211_i.h
+++ b/net/mac80211/ieee80211_i.h
@@ -1424,10 +1424,12 @@ struct ieee80211_local {
spinlock_t handle_wake_tx_queue_lock;
u16 airtime_flags;
+ u32 aql_txq_limit_mc;
u32 aql_txq_limit_low[IEEE80211_NUM_ACS];
u32 aql_txq_limit_high[IEEE80211_NUM_ACS];
u32 aql_threshold;
atomic_t aql_total_pending_airtime;
+ atomic_t aql_mc_pending_airtime;
atomic_t aql_ac_pending_airtime[IEEE80211_NUM_ACS];
const struct ieee80211_ops *ops;
diff --git a/net/mac80211/main.c b/net/mac80211/main.c
index cf1d4a031af3..b62ef65db40d 100644
--- a/net/mac80211/main.c
+++ b/net/mac80211/main.c
@@ -984,6 +984,7 @@ struct ieee80211_hw *ieee80211_alloc_hw_nm(size_t priv_data_len,
spin_lock_init(&local->rx_path_lock);
spin_lock_init(&local->queue_stop_reason_lock);
+ local->aql_txq_limit_mc = IEEE80211_DEFAULT_AQL_TXQ_LIMIT_MC;
for (i = 0; i < IEEE80211_NUM_ACS; i++) {
INIT_LIST_HEAD(&local->active_txqs[i]);
spin_lock_init(&local->active_txq_lock[i]);
diff --git a/net/mac80211/rx.c b/net/mac80211/rx.c
index 3a9d7020d8d5..3567544c3404 100644
--- a/net/mac80211/rx.c
+++ b/net/mac80211/rx.c
@@ -1443,6 +1443,9 @@ ieee80211_rx_h_check_dup(struct ieee80211_rx_data *rx)
if (status->flag & RX_FLAG_DUP_VALIDATED)
return RX_CONTINUE;
+ if (ieee80211_is_ext(hdr->frame_control))
+ return RX_CONTINUE;
+
/*
* Drop duplicate 802.11 retransmissions
* (IEEE 802.11-2012: 9.3.2.10 "Duplicate detection and recovery")
@@ -4387,8 +4390,12 @@ static bool ieee80211_accept_frame(struct ieee80211_rx_data *rx)
struct ieee80211_hdr *hdr = (void *)skb->data;
struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(skb);
u8 *bssid = ieee80211_get_bssid(hdr, skb->len, sdata->vif.type);
- bool multicast = is_multicast_ether_addr(hdr->addr1) ||
- ieee80211_is_s1g_beacon(hdr->frame_control);
+ bool multicast;
+
+ if (ieee80211_is_s1g_beacon(hdr->frame_control))
+ return sdata->vif.type == NL80211_IFTYPE_STATION && bssid;
+
+ multicast = is_multicast_ether_addr(hdr->addr1);
switch (sdata->vif.type) {
case NL80211_IFTYPE_STATION:
@@ -4847,6 +4854,7 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
u8 sa[ETH_ALEN];
} addrs __aligned(2);
struct ieee80211_sta_rx_stats *stats;
+ u32 encoded_rate;
/* for parallel-rx, we need to have DUP_VALIDATED, otherwise we write
* to a common data structure; drivers can implement that per queue
@@ -4918,6 +4926,11 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
/* after this point, don't punt to the slowpath! */
+ if (fast_rx->uses_rss)
+ stats = this_cpu_ptr(rx->link_sta->pcpu_rx_stats);
+ else
+ stats = &rx->link_sta->rx_stats;
+
if (rx->key && !(status->flag & RX_FLAG_MIC_STRIPPED) &&
pskb_trim(skb, skb->len - fast_rx->icv_len))
goto drop;
@@ -4949,9 +4962,14 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
/* push the addresses in front */
memcpy(skb_push(skb, sizeof(addrs)), &addrs, sizeof(addrs));
+ /* capture before mesh forward may memset or free skb->cb */
+ encoded_rate = sta_stats_encode_rate(status);
+
res = ieee80211_rx_mesh_data(rx->sdata, rx->sta, rx->skb);
switch (res) {
case RX_QUEUED:
+ stats->last_rx = jiffies;
+ stats->last_rate = encoded_rate;
return true;
case RX_CONTINUE:
break;
@@ -4965,11 +4983,6 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
drop:
dev_kfree_skb(skb);
- if (fast_rx->uses_rss)
- stats = this_cpu_ptr(rx->link_sta->pcpu_rx_stats);
- else
- stats = &rx->link_sta->rx_stats;
-
stats->dropped++;
return true;
}
@@ -5031,6 +5044,11 @@ static bool ieee80211_prepare_and_rx_handle(struct ieee80211_rx_data *rx,
hdr = (struct ieee80211_hdr *)rx->skb->data;
}
+ if (ieee80211_is_s1g_beacon(hdr->frame_control)) {
+ ieee80211_invoke_rx_handlers(rx);
+ return true;
+ }
+
if (unlikely(rx->sta && rx->sta->sta.mlo) &&
is_unicast_ether_addr(hdr->addr1) &&
!ieee80211_is_probe_resp(hdr->frame_control) &&
@@ -5115,6 +5133,13 @@ static bool ieee80211_rx_for_interface(struct ieee80211_rx_data *rx,
struct sta_info *sta;
int link_id = -1;
+ if (ieee80211_is_s1g_beacon(hdr->frame_control)) {
+ if (!ieee80211_rx_data_set_sta(rx, NULL, -1))
+ return false;
+
+ return ieee80211_prepare_and_rx_handle(rx, skb, consume);
+ }
+
/*
* Look up link station first, in case there's a
* chance that they might have a link address that
@@ -5190,6 +5215,17 @@ static void __ieee80211_rx_handle_packet(struct ieee80211_hw *hw,
err = -ENOBUFS;
else
err = skb_linearize(skb);
+ } else if (ieee80211_is_s1g_beacon(fc)) {
+ size_t s1g_hdr_len = offsetof(struct ieee80211_ext,
+ u.s1g_beacon.variable) +
+ ieee80211_s1g_optional_len(fc);
+
+ if (skb->len < s1g_hdr_len)
+ err = -ENOBUFS;
+ else
+ err = skb_linearize(skb);
+ } else if (ieee80211_is_ext(fc)) {
+ err = -EINVAL;
} else {
err = !pskb_may_pull(skb, ieee80211_hdrlen(fc));
}
diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c
index 5d4d9e372eb7..73f92386b3c8 100644
--- a/net/mac80211/sta_info.c
+++ b/net/mac80211/sta_info.c
@@ -2440,13 +2440,28 @@ EXPORT_SYMBOL(ieee80211_sta_recalc_aggregates);
void ieee80211_sta_update_pending_airtime(struct ieee80211_local *local,
struct sta_info *sta, u8 ac,
- u16 tx_airtime, bool tx_completed)
+ u16 tx_airtime, bool tx_completed,
+ bool mcast)
{
int tx_pending;
if (!wiphy_ext_feature_isset(local->hw.wiphy, NL80211_EXT_FEATURE_AQL))
return;
+ if (mcast) {
+ if (!tx_completed) {
+ atomic_add(tx_airtime, &local->aql_mc_pending_airtime);
+ return;
+ }
+
+ tx_pending = atomic_sub_return(tx_airtime,
+ &local->aql_mc_pending_airtime);
+ if (tx_pending < 0)
+ atomic_cmpxchg(&local->aql_mc_pending_airtime,
+ tx_pending, 0);
+ return;
+ }
+
if (!tx_completed) {
if (sta)
atomic_add(tx_airtime,
diff --git a/net/mac80211/sta_info.h b/net/mac80211/sta_info.h
index 5288d5286651..bf3203e454a5 100644
--- a/net/mac80211/sta_info.h
+++ b/net/mac80211/sta_info.h
@@ -147,7 +147,8 @@ struct airtime_info {
void ieee80211_sta_update_pending_airtime(struct ieee80211_local *local,
struct sta_info *sta, u8 ac,
- u16 tx_airtime, bool tx_completed);
+ u16 tx_airtime, bool tx_completed,
+ bool mcast);
struct sta_info;
diff --git a/net/mac80211/status.c b/net/mac80211/status.c
index 8716eda8317d..f0acd8196468 100644
--- a/net/mac80211/status.c
+++ b/net/mac80211/status.c
@@ -752,7 +752,7 @@ static void ieee80211_report_used_skb(struct ieee80211_local *local,
ieee80211_sta_update_pending_airtime(local, sta,
skb_get_queue_mapping(skb),
tx_time_est,
- true);
+ true, info->tx_time_mc);
rcu_read_unlock();
}
@@ -1161,10 +1161,11 @@ void ieee80211_tx_status_ext(struct ieee80211_hw *hw,
/* Do this here to avoid the expensive lookup of the sta
* in ieee80211_report_used_skb().
*/
+ bool mcast = IEEE80211_SKB_CB(skb)->tx_time_mc;
ieee80211_sta_update_pending_airtime(local, sta,
skb_get_queue_mapping(skb),
tx_time_est,
- true);
+ true, mcast);
ieee80211_info_set_tx_time_est(IEEE80211_SKB_CB(skb), 0);
}
diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c
index d77f9f9810a0..63867702ab08 100644
--- a/net/mac80211/tx.c
+++ b/net/mac80211/tx.c
@@ -4035,20 +4035,20 @@ struct sk_buff *ieee80211_tx_dequeue(struct ieee80211_hw *hw,
encap_out:
info->control.vif = vif;
- if (tx.sta &&
- wiphy_ext_feature_isset(local->hw.wiphy, NL80211_EXT_FEATURE_AQL)) {
- bool ampdu = txq->ac != IEEE80211_AC_VO;
+ if (wiphy_ext_feature_isset(local->hw.wiphy, NL80211_EXT_FEATURE_AQL)) {
+ bool ampdu = txq->sta && txq->ac != IEEE80211_AC_VO;
u32 airtime;
airtime = ieee80211_calc_expected_tx_airtime(hw, vif, txq->sta,
skb->len, ampdu);
- if (airtime) {
- airtime = ieee80211_info_set_tx_time_est(info, airtime);
- ieee80211_sta_update_pending_airtime(local, tx.sta,
- txq->ac,
- airtime,
- false);
- }
+ if (!airtime)
+ return skb;
+
+ airtime = ieee80211_info_set_tx_time_est(info, airtime);
+ info->tx_time_mc = !tx.sta;
+ ieee80211_sta_update_pending_airtime(local, tx.sta, txq->ac,
+ airtime, false,
+ info->tx_time_mc);
}
return skb;
@@ -4100,6 +4100,7 @@ struct ieee80211_txq *ieee80211_next_txq(struct ieee80211_hw *hw, u8 ac)
struct ieee80211_txq *ret = NULL;
struct txq_info *txqi = NULL, *head = NULL;
bool found_eligible_txq = false;
+ bool aql_check;
spin_lock_bh(&local->active_txq_lock[ac]);
@@ -4123,26 +4124,28 @@ struct ieee80211_txq *ieee80211_next_txq(struct ieee80211_hw *hw, u8 ac)
if (!head)
head = txqi;
+ aql_check = ieee80211_txq_airtime_check(hw, &txqi->txq);
+ if (aql_check)
+ found_eligible_txq = true;
+
if (txqi->txq.sta) {
struct sta_info *sta = container_of(txqi->txq.sta,
struct sta_info, sta);
- bool aql_check = ieee80211_txq_airtime_check(hw, &txqi->txq);
- s32 deficit = ieee80211_sta_deficit(sta, txqi->txq.ac);
- if (aql_check)
- found_eligible_txq = true;
-
- if (deficit < 0)
+ if (ieee80211_sta_deficit(sta, txqi->txq.ac) < 0) {
sta->airtime[txqi->txq.ac].deficit +=
sta->airtime_weight;
- if (deficit < 0 || !aql_check) {
- list_move_tail(&txqi->schedule_order,
- &local->active_txqs[txqi->txq.ac]);
- goto begin;
+ aql_check = false;
}
}
+ if (!aql_check) {
+ list_move_tail(&txqi->schedule_order,
+ &local->active_txqs[txqi->txq.ac]);
+ goto begin;
+ }
+
if (txqi->schedule_round == local->schedule_round[ac])
goto out;
@@ -4209,7 +4212,8 @@ bool ieee80211_txq_airtime_check(struct ieee80211_hw *hw,
return true;
if (!txq->sta)
- return true;
+ return atomic_read(&local->aql_mc_pending_airtime) <
+ local->aql_txq_limit_mc;
if (unlikely(txq->tid == IEEE80211_NUM_TIDS))
return true;
@@ -4229,6 +4233,24 @@ bool ieee80211_txq_airtime_check(struct ieee80211_hw *hw,
}
EXPORT_SYMBOL(ieee80211_txq_airtime_check);
+u32 ieee80211_txq_aql_pending(struct ieee80211_hw *hw,
+ struct ieee80211_txq *txq)
+{
+ struct ieee80211_local *local = hw_to_local(hw);
+ struct sta_info *sta;
+
+ if (unlikely(txq->tid == IEEE80211_NUM_TIDS))
+ return 0;
+
+ if (!txq->sta)
+ return atomic_read(&local->aql_mc_pending_airtime);
+
+ sta = container_of(txq->sta, struct sta_info, sta);
+
+ return atomic_read(&sta->airtime[txq->ac].aql_tx_pending);
+}
+EXPORT_SYMBOL(ieee80211_txq_aql_pending);
+
static bool
ieee80211_txq_schedule_airtime_check(struct ieee80211_local *local, u8 ac)
{
@@ -4258,15 +4280,15 @@ bool ieee80211_txq_may_transmit(struct ieee80211_hw *hw,
spin_lock_bh(&local->active_txq_lock[ac]);
- if (!txqi->txq.sta)
- goto out;
-
if (list_empty(&txqi->schedule_order))
goto out;
if (!ieee80211_txq_schedule_airtime_check(local, ac))
goto out;
+ if (!txqi->txq.sta)
+ goto out;
+
list_for_each_entry_safe(iter, tmp, &local->active_txqs[ac],
schedule_order) {
if (iter == txqi)
diff --git a/net/mac80211/util.c b/net/mac80211/util.c
index 3ed47a8ebf5c..1b5a0ca31167 100644
--- a/net/mac80211/util.c
+++ b/net/mac80211/util.c
@@ -73,6 +73,9 @@ u8 *ieee80211_get_bssid(struct ieee80211_hdr *hdr, size_t len,
if (ieee80211_is_s1g_beacon(fc)) {
struct ieee80211_ext *ext = (void *) hdr;
+ if (len < offsetofend(struct ieee80211_ext, u.s1g_beacon.sa))
+ return NULL;
+
return ext->u.s1g_beacon.sa;
}
diff --git a/net/mctp/af_mctp.c b/net/mctp/af_mctp.c
index b99ba14f39d2..5b1ef50637b7 100644
--- a/net/mctp/af_mctp.c
+++ b/net/mctp/af_mctp.c
@@ -49,7 +49,7 @@ static bool mctp_sockaddr_ext_is_ok(const struct sockaddr_mctp_ext *addr)
!addr->__smctp_pad0[2];
}
-static int mctp_bind(struct socket *sock, struct sockaddr *addr, int addrlen)
+static int mctp_bind(struct socket *sock, struct sockaddr_unsized *addr, int addrlen)
{
struct sock *sk = sock->sk;
struct mctp_sock *msk = container_of(sk, struct mctp_sock, sk);
diff --git a/net/mctp/test/route-test.c b/net/mctp/test/route-test.c
index 69a3ccfc6310..be9149ac79dd 100644
--- a/net/mctp/test/route-test.c
+++ b/net/mctp/test/route-test.c
@@ -205,7 +205,7 @@ static void __mctp_route_test_init(struct kunit *test,
addr.smctp_network = netid;
addr.smctp_addr.s_addr = 8;
addr.smctp_type = 0;
- rc = kernel_bind(sock, (struct sockaddr *)&addr, sizeof(addr));
+ rc = kernel_bind(sock, (struct sockaddr_unsized *)&addr, sizeof(addr));
KUNIT_ASSERT_EQ(test, rc, 0);
*devp = dev;
diff --git a/net/mctp/test/utils.c b/net/mctp/test/utils.c
index 953d41902771..5a1c65fa4977 100644
--- a/net/mctp/test/utils.c
+++ b/net/mctp/test/utils.c
@@ -292,5 +292,5 @@ void mctp_test_bind_run(struct kunit *test,
addr.smctp_type = setup->bind_type;
*ret_bind_errno =
- kernel_bind(*sock, (struct sockaddr *)&addr, sizeof(addr));
+ kernel_bind(*sock, (struct sockaddr_unsized *)&addr, sizeof(addr));
}
diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index e14a49180eef..50eaa8ace5d5 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -3964,7 +3964,7 @@ static struct proto mptcp_prot = {
.no_autobind = true,
};
-static int mptcp_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int mptcp_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct mptcp_sock *msk = mptcp_sk(sock->sk);
struct sock *ssk, *sk = sock->sk;
@@ -3978,10 +3978,10 @@ static int mptcp_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
}
if (sk->sk_family == AF_INET)
- err = inet_bind_sk(ssk, uaddr, addr_len);
+ err = inet_bind_sk(ssk, (struct sockaddr *)uaddr, addr_len);
#if IS_ENABLED(CONFIG_MPTCP_IPV6)
else if (sk->sk_family == AF_INET6)
- err = inet6_bind_sk(ssk, uaddr, addr_len);
+ err = inet6_bind_sk(ssk, (struct sockaddr *)uaddr, addr_len);
#endif
if (!err)
mptcp_copy_inaddrs(sk, ssk);
diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c
index 1932fcf9f925..7e9653736524 100644
--- a/net/mptcp/subflow.c
+++ b/net/mptcp/subflow.c
@@ -1672,7 +1672,7 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_pm_local *local,
addrlen = sizeof(struct sockaddr_in6);
#endif
ssk->sk_bound_dev_if = local->ifindex;
- err = kernel_bind(sf, (struct sockaddr *)&addr, addrlen);
+ err = kernel_bind(sf, (struct sockaddr_unsized *)&addr, addrlen);
if (err) {
MPTCP_INC_STATS(sock_net(sk), MPTCP_MIB_JOINSYNTXBINDERR);
pr_debug("msk=%p local=%d remote=%d bind error: %d\n",
diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index 4e30a903f4bd..401159a66075 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -745,7 +745,7 @@ static int ip_vs_route_me_harder(struct netns_ipvs *ipvs, int af,
* Packet has been made sufficiently writable in caller
* - inout: 1=in->out, 0=out->in
*/
-void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
+bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct ip_vs_conn *cp, int inout, unsigned int toff,
bool has_ports, struct ip_vs_iphdr *ciph)
{
@@ -753,6 +753,11 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct icmphdr *icmph = (struct icmphdr *)(skb->data + toff);
struct iphdr *cih = (struct iphdr *)(icmph + 1);
+ /* Before now we may used ihl from skb frag, revalidate it after
+ * copying it into skb head to prevent out-of-bounds access
+ */
+ if (cih->ihl * 4 != ciph->len - ciph->off)
+ return false;
if (inout) {
iph->saddr = cp->vaddr.ip;
ip_send_check(iph);
@@ -786,6 +791,7 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
else
IP_VS_DBG_PKT(11, AF_INET, pp, skb, ciph->off,
"Forwarding altered incoming ICMP");
+ return true;
}
#ifdef CONFIG_IP_VS_IPV6
@@ -877,7 +883,8 @@ static int handle_response_icmp(int af, struct sk_buff *skb,
ip_vs_nat_icmp_v6(skb, pp, cp, 1, toff, has_ports, ciph);
else
#endif
- ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph);
+ if (!ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph))
+ goto out;
if (ip_vs_route_me_harder(cp->ipvs, af, skb, hooknum))
goto out;
diff --git a/net/netfilter/ipvs/ip_vs_sync.c b/net/netfilter/ipvs/ip_vs_sync.c
index b835f61886d6..4bd959a5d363 100644
--- a/net/netfilter/ipvs/ip_vs_sync.c
+++ b/net/netfilter/ipvs/ip_vs_sync.c
@@ -1459,7 +1459,7 @@ static int bind_mcastif_addr(struct socket *sock, struct net_device *dev)
sin.sin_addr.s_addr = addr;
sin.sin_port = 0;
- return kernel_bind(sock, (struct sockaddr *)&sin, sizeof(sin));
+ return kernel_bind(sock, (struct sockaddr_unsized *)&sin, sizeof(sin));
}
static void get_mcast_sockaddr(union ipvs_sockaddr *sa, int *salen,
@@ -1566,7 +1566,7 @@ static int make_receive_sock(struct netns_ipvs *ipvs, int id,
get_mcast_sockaddr(&mcast_addr, &salen, &ipvs->bcfg, id);
sock->sk->sk_bound_dev_if = dev->ifindex;
- result = kernel_bind(sock, (struct sockaddr *)&mcast_addr, salen);
+ result = kernel_bind(sock, (struct sockaddr_unsized *)&mcast_addr, salen);
if (result < 0) {
pr_err("Error binding to the multicast addr\n");
goto error;
diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index 7db74b2ea358..fa66e3168550 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -1577,7 +1577,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
if (skb_cow(skb, rt->dst.dev->hard_header_len))
goto tx_error;
- ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph);
+ if (!ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph))
+ goto tx_error;
/* Another hack: avoid icmp_send in ip_fragment */
skb->ignore_df = 1;
diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c
index 4a136fc3a9c0..aef177025512 100644
--- a/net/netfilter/nf_conntrack_bpf.c
+++ b/net/netfilter/nf_conntrack_bpf.c
@@ -64,6 +64,15 @@ enum {
NF_BPF_CT_OPTS_SZ = 16,
};
+static void *bpf_ct_opts_result(struct bpf_ct_opts *opts, u32 opts__sz, void *ret)
+{
+ if (!IS_ERR(ret))
+ return ret;
+ if (opts__sz >= offsetofend(struct bpf_ct_opts, error))
+ opts->error = PTR_ERR(ret);
+ return NULL;
+}
+
static int bpf_nf_ct_tuple_parse(struct bpf_sock_tuple *bpf_tuple,
u32 tuple_len, u8 protonum, u8 dir,
struct nf_conntrack_tuple *tuple)
@@ -112,44 +121,54 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple,
struct nf_conntrack_tuple otuple, rtuple;
struct nf_conntrack_zone ct_zone;
struct nf_conn *ct;
+ u8 ct_zone_dir = 0;
+ u16 ct_zone_id;
+ s32 netns_id;
+ u8 l4proto;
int err;
- if (!opts || !bpf_tuple)
- return ERR_PTR(-EINVAL);
if (!(opts_len == NF_BPF_CT_OPTS_SZ || opts_len == 12))
return ERR_PTR(-EINVAL);
+
+ netns_id = READ_ONCE(opts->netns_id);
+ l4proto = READ_ONCE(opts->l4proto);
+ ct_zone_id = READ_ONCE(opts->ct_zone_id);
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->reserved[0] || opts->reserved[1] || opts->reserved[2])
+ ct_zone_dir = READ_ONCE(opts->ct_zone_dir);
+ if (READ_ONCE(opts->reserved[0]) ||
+ READ_ONCE(opts->reserved[1]) ||
+ READ_ONCE(opts->reserved[2]))
return ERR_PTR(-EINVAL);
} else {
- if (opts->ct_zone_id)
+ if (ct_zone_id)
return ERR_PTR(-EINVAL);
}
- if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS))
+ if (unlikely(netns_id < BPF_F_CURRENT_NETNS))
return ERR_PTR(-EINVAL);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_ORIGINAL, &otuple);
if (err < 0)
return ERR_PTR(err);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_REPLY, &rtuple);
if (err < 0)
return ERR_PTR(err);
- if (opts->netns_id >= 0) {
- net = get_net_ns_by_id(net, opts->netns_id);
+ if (netns_id >= 0) {
+ net = get_net_ns_by_id(net, netns_id);
if (unlikely(!net))
return ERR_PTR(-ENONET);
}
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->ct_zone_dir == 0)
- opts->ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
- nf_ct_zone_init(&ct_zone,
- opts->ct_zone_id, opts->ct_zone_dir, 0);
+ if (ct_zone_dir == 0) {
+ ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
+ opts->ct_zone_dir = ct_zone_dir;
+ }
+ nf_ct_zone_init(&ct_zone, ct_zone_id, ct_zone_dir, 0);
} else {
ct_zone = nf_ct_zone_dflt;
}
@@ -163,7 +182,7 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple,
__nf_ct_set_timeout(ct, timeout * HZ);
out:
- if (opts->netns_id >= 0)
+ if (netns_id >= 0)
put_net(net);
return ct;
@@ -178,46 +197,58 @@ static struct nf_conn *__bpf_nf_ct_lookup(struct net *net,
struct nf_conntrack_tuple tuple;
struct nf_conntrack_zone ct_zone;
struct nf_conn *ct;
+ u8 ct_zone_dir = 0;
+ u16 ct_zone_id;
+ s32 netns_id;
+ u8 l4proto;
int err;
if (!opts || !bpf_tuple)
return ERR_PTR(-EINVAL);
if (!(opts_len == NF_BPF_CT_OPTS_SZ || opts_len == 12))
return ERR_PTR(-EINVAL);
+
+ netns_id = READ_ONCE(opts->netns_id);
+ l4proto = READ_ONCE(opts->l4proto);
+ ct_zone_id = READ_ONCE(opts->ct_zone_id);
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->reserved[0] || opts->reserved[1] || opts->reserved[2])
+ ct_zone_dir = READ_ONCE(opts->ct_zone_dir);
+ if (READ_ONCE(opts->reserved[0]) ||
+ READ_ONCE(opts->reserved[1]) ||
+ READ_ONCE(opts->reserved[2]))
return ERR_PTR(-EINVAL);
} else {
- if (opts->ct_zone_id)
+ if (ct_zone_id)
return ERR_PTR(-EINVAL);
}
- if (unlikely(opts->l4proto != IPPROTO_TCP && opts->l4proto != IPPROTO_UDP))
+ if (unlikely(l4proto != IPPROTO_TCP && l4proto != IPPROTO_UDP))
return ERR_PTR(-EPROTO);
- if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS))
+ if (unlikely(netns_id < BPF_F_CURRENT_NETNS))
return ERR_PTR(-EINVAL);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_ORIGINAL, &tuple);
if (err < 0)
return ERR_PTR(err);
- if (opts->netns_id >= 0) {
- net = get_net_ns_by_id(net, opts->netns_id);
+ if (netns_id >= 0) {
+ net = get_net_ns_by_id(net, netns_id);
if (unlikely(!net))
return ERR_PTR(-ENONET);
}
if (opts_len == NF_BPF_CT_OPTS_SZ) {
- if (opts->ct_zone_dir == 0)
- opts->ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
- nf_ct_zone_init(&ct_zone,
- opts->ct_zone_id, opts->ct_zone_dir, 0);
+ if (ct_zone_dir == 0) {
+ ct_zone_dir = NF_CT_DEFAULT_ZONE_DIR;
+ opts->ct_zone_dir = ct_zone_dir;
+ }
+ nf_ct_zone_init(&ct_zone, ct_zone_id, ct_zone_dir, 0);
} else {
ct_zone = nf_ct_zone_dflt;
}
hash = nf_conntrack_find_get(net, &ct_zone, &tuple);
- if (opts->netns_id >= 0)
+ if (netns_id >= 0)
put_net(net);
if (!hash)
return ERR_PTR(-ENOENT);
@@ -298,13 +329,7 @@ bpf_xdp_ct_alloc(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
nfct = __bpf_nf_ct_alloc_entry(dev_net(ctx->rxq->dev), bpf_tuple, tuple__sz,
opts, opts__sz, 10);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
-
- return (struct nf_conn___init *)nfct;
+ return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_xdp_ct_lookup - Lookup CT entry for the given tuple, and acquire a
@@ -333,12 +358,7 @@ bpf_xdp_ct_lookup(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
caller_net = dev_net(ctx->rxq->dev);
nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
- return nfct;
+ return bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_skb_ct_alloc - Allocate a new CT entry
@@ -366,13 +386,7 @@ bpf_skb_ct_alloc(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk);
nfct = __bpf_nf_ct_alloc_entry(net, bpf_tuple, tuple__sz, opts, opts__sz, 10);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
-
- return (struct nf_conn___init *)nfct;
+ return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_skb_ct_lookup - Lookup CT entry for the given tuple, and acquire a
@@ -401,12 +415,7 @@ bpf_skb_ct_lookup(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
caller_net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk);
nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
- return nfct;
+ return bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_ct_insert_entry - Add the provided entry into a CT map
diff --git a/net/netfilter/nf_dup_netdev.c b/net/netfilter/nf_dup_netdev.c
index fab8b9011098..02d1880bab6b 100644
--- a/net/netfilter/nf_dup_netdev.c
+++ b/net/netfilter/nf_dup_netdev.c
@@ -13,30 +13,9 @@
#include <net/netfilter/nf_tables_offload.h>
#include <net/netfilter/nf_dup_netdev.h>
-#define NF_RECURSION_LIMIT 2
-
-#ifndef CONFIG_PREEMPT_RT
-static u8 *nf_get_nf_dup_skb_recursion(void)
-{
- return this_cpu_ptr(&softnet_data.xmit.nf_dup_skb_recursion);
-}
-#else
-
-static u8 *nf_get_nf_dup_skb_recursion(void)
-{
- return ¤t->net_xmit.nf_dup_skb_recursion;
-}
-
-#endif
-
static void nf_do_netdev_egress(struct sk_buff *skb, struct net_device *dev,
enum nf_dev_hooks hook)
{
- u8 *nf_dup_skb_recursion = nf_get_nf_dup_skb_recursion();
-
- if (*nf_dup_skb_recursion > NF_RECURSION_LIMIT)
- goto err;
-
if (hook == NF_NETDEV_INGRESS && skb_mac_header_was_set(skb)) {
if (skb_cow_head(skb, skb->mac_len))
goto err;
@@ -46,9 +25,15 @@ static void nf_do_netdev_egress(struct sk_buff *skb, struct net_device *dev,
skb->dev = dev;
skb_clear_tstamp(skb);
- (*nf_dup_skb_recursion)++;
+ local_bh_disable();
+ if (nf_dev_xmit_recursion()) {
+ local_bh_enable();
+ goto err;
+ }
+ nf_dev_xmit_recursion_inc();
dev_queue_xmit(skb);
- (*nf_dup_skb_recursion)--;
+ nf_dev_xmit_recursion_dec();
+ local_bh_enable();
return;
err:
kfree_skb(skb);
diff --git a/net/netfilter/nf_flow_table_path.c b/net/netfilter/nf_flow_table_path.c
index c5bb142f329b..7a3b795f8627 100644
--- a/net/netfilter/nf_flow_table_path.c
+++ b/net/netfilter/nf_flow_table_path.c
@@ -147,10 +147,11 @@ static void nft_dev_path_info(const struct net_device_path_stack *stack,
info->num_encaps++;
break;
case DEV_PATH_BR_VLAN_UNTAG:
- if (WARN_ON_ONCE(info->num_encaps-- == 0)) {
+ if (info->num_encaps == 0) {
info->indev = NULL;
break;
}
+ info->num_encaps--;
break;
case DEV_PATH_BR_VLAN_KEEP:
break;
diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c
index b4c0cae18252..31c3ff79b2d5 100644
--- a/net/netfilter/nf_tables_api.c
+++ b/net/netfilter/nf_tables_api.c
@@ -377,6 +377,32 @@ static void nft_netdev_hook_unlink_free_rcu(struct nft_hook *hook)
nft_netdev_hook_free_rcu(hook);
}
+static void nft_trans_hook_destroy(struct nft_trans_hook *trans_hook)
+{
+ list_del(&trans_hook->list);
+ kfree(trans_hook);
+}
+
+static void nft_netdev_unregister_trans_hook(struct net *net,
+ const struct nft_table *table,
+ struct list_head *hook_list)
+{
+ struct nft_trans_hook *trans_hook, *next;
+ struct nf_hook_ops *ops;
+ struct nft_hook *hook;
+
+ list_for_each_entry_safe(trans_hook, next, hook_list, list) {
+ hook = trans_hook->hook;
+
+ if (!(table->flags & NFT_TABLE_F_DORMANT)) {
+ list_for_each_entry(ops, &hook->ops_list, list)
+ nf_unregister_net_hook(net, ops);
+ }
+ nft_netdev_hook_unlink_free_rcu(hook);
+ nft_trans_hook_destroy(trans_hook);
+ }
+}
+
static void nft_netdev_unregister_hooks(struct net *net,
struct list_head *hook_list,
bool release_netdev)
@@ -1950,15 +1976,69 @@ static int nft_nla_put_hook_dev(struct sk_buff *skb, struct nft_hook *hook)
return nla_put_string(skb, attr, hook->ifname);
}
+struct nft_hook_dump_ctx {
+ struct nft_hook *first;
+ int n;
+};
+
+static int nft_dump_basechain_hook_one(struct sk_buff *skb,
+ struct nft_hook *hook,
+ struct nft_hook_dump_ctx *dump_ctx)
+{
+ if (!dump_ctx->first)
+ dump_ctx->first = hook;
+
+ if (nft_nla_put_hook_dev(skb, hook))
+ return -1;
+
+ dump_ctx->n++;
+
+ return 0;
+}
+
+static int nft_dump_basechain_hook_list(struct sk_buff *skb,
+ const struct net *net,
+ const struct list_head *hook_list,
+ struct nft_hook_dump_ctx *dump_ctx)
+{
+ struct nft_hook *hook;
+ int err;
+
+ list_for_each_entry_rcu(hook, hook_list, list,
+ lockdep_commit_lock_is_held(net)) {
+ err = nft_dump_basechain_hook_one(skb, hook, dump_ctx);
+ if (err < 0)
+ return err;
+ }
+
+ return 0;
+}
+
+static int nft_dump_basechain_trans_hook_list(struct sk_buff *skb,
+ const struct list_head *trans_hook_list,
+ struct nft_hook_dump_ctx *dump_ctx)
+{
+ struct nft_trans_hook *trans_hook;
+ int err;
+
+ list_for_each_entry(trans_hook, trans_hook_list, list) {
+ err = nft_dump_basechain_hook_one(skb, trans_hook->hook, dump_ctx);
+ if (err < 0)
+ return err;
+ }
+
+ return 0;
+}
+
static int nft_dump_basechain_hook(struct sk_buff *skb,
const struct net *net, int family,
const struct nft_base_chain *basechain,
- const struct list_head *hook_list)
+ const struct list_head *hook_list,
+ const struct list_head *trans_hook_list)
{
const struct nf_hook_ops *ops = &basechain->ops;
- struct nft_hook *hook, *first = NULL;
+ struct nft_hook_dump_ctx dump_hook_ctx = {};
struct nlattr *nest, *nest_devs;
- int n = 0;
nest = nla_nest_start_noflag(skb, NFTA_CHAIN_HOOK);
if (nest == NULL)
@@ -1973,23 +2053,23 @@ static int nft_dump_basechain_hook(struct sk_buff *skb,
if (!nest_devs)
goto nla_put_failure;
- if (!hook_list)
+ if (!hook_list && !trans_hook_list)
hook_list = &basechain->hook_list;
- list_for_each_entry_rcu(hook, hook_list, list,
- lockdep_commit_lock_is_held(net)) {
- if (!first)
- first = hook;
-
- if (nft_nla_put_hook_dev(skb, hook))
- goto nla_put_failure;
- n++;
+ if (hook_list &&
+ nft_dump_basechain_hook_list(skb, net, hook_list, &dump_hook_ctx)) {
+ goto nla_put_failure;
+ } else if (trans_hook_list &&
+ nft_dump_basechain_trans_hook_list(skb, trans_hook_list,
+ &dump_hook_ctx)) {
+ goto nla_put_failure;
}
+
nla_nest_end(skb, nest_devs);
- if (n == 1 &&
- !hook_is_prefix(first) &&
- nla_put_string(skb, NFTA_HOOK_DEV, first->ifname))
+ if (dump_hook_ctx.n == 1 &&
+ !hook_is_prefix(dump_hook_ctx.first) &&
+ nla_put_string(skb, NFTA_HOOK_DEV, dump_hook_ctx.first->ifname))
goto nla_put_failure;
}
nla_nest_end(skb, nest);
@@ -2003,7 +2083,8 @@ static int nf_tables_fill_chain_info(struct sk_buff *skb, struct net *net,
u32 portid, u32 seq, int event, u32 flags,
int family, const struct nft_table *table,
const struct nft_chain *chain,
- const struct list_head *hook_list)
+ const struct list_head *hook_list,
+ const struct list_head *trans_hook_list)
{
struct nlmsghdr *nlh;
@@ -2019,7 +2100,7 @@ static int nf_tables_fill_chain_info(struct sk_buff *skb, struct net *net,
NFTA_CHAIN_PAD))
goto nla_put_failure;
- if (!hook_list &&
+ if (!hook_list && !trans_hook_list &&
(event == NFT_MSG_DELCHAIN ||
event == NFT_MSG_DESTROYCHAIN)) {
nlmsg_end(skb, nlh);
@@ -2030,7 +2111,8 @@ static int nf_tables_fill_chain_info(struct sk_buff *skb, struct net *net,
const struct nft_base_chain *basechain = nft_base_chain(chain);
struct nft_stats __percpu *stats;
- if (nft_dump_basechain_hook(skb, net, family, basechain, hook_list))
+ if (nft_dump_basechain_hook(skb, net, family, basechain,
+ hook_list, trans_hook_list))
goto nla_put_failure;
if (nla_put_be32(skb, NFTA_CHAIN_POLICY,
@@ -2066,7 +2148,8 @@ static int nf_tables_fill_chain_info(struct sk_buff *skb, struct net *net,
}
static void nf_tables_chain_notify(const struct nft_ctx *ctx, int event,
- const struct list_head *hook_list)
+ const struct list_head *hook_list,
+ const struct list_head *trans_hook_list)
{
struct nftables_pernet *nft_net;
struct sk_buff *skb;
@@ -2086,7 +2169,7 @@ static void nf_tables_chain_notify(const struct nft_ctx *ctx, int event,
err = nf_tables_fill_chain_info(skb, ctx->net, ctx->portid, ctx->seq,
event, flags, ctx->family, ctx->table,
- ctx->chain, hook_list);
+ ctx->chain, hook_list, trans_hook_list);
if (err < 0) {
kfree_skb(skb);
goto err;
@@ -2132,7 +2215,7 @@ static int nf_tables_dump_chains(struct sk_buff *skb,
NFT_MSG_NEWCHAIN,
NLM_F_MULTI,
table->family, table,
- chain, NULL) < 0)
+ chain, NULL, NULL) < 0)
goto done;
nl_dump_check_consistent(cb, nlmsg_hdr(skb));
@@ -2186,7 +2269,7 @@ static int nf_tables_getchain(struct sk_buff *skb, const struct nfnl_info *info,
err = nf_tables_fill_chain_info(skb2, net, NETLINK_CB(skb).portid,
info->nlh->nlmsg_seq, NFT_MSG_NEWCHAIN,
- 0, family, table, chain, NULL);
+ 0, family, table, chain, NULL, NULL);
if (err < 0)
goto err_fill_chain_info;
@@ -2352,8 +2435,12 @@ static struct nft_hook *nft_hook_list_find(struct list_head *hook_list,
if (strict && hook->ifnamelen != this->ifnamelen)
continue;
if (!strncmp(hook->ifname, this->ifname,
- min(hook->ifnamelen, this->ifnamelen)))
+ min(hook->ifnamelen, this->ifnamelen))) {
+ if (hook->flags & NFT_HOOK_REMOVE)
+ continue;
+
return hook;
+ }
}
return NULL;
@@ -3113,6 +3200,32 @@ static int nf_tables_newchain(struct sk_buff *skb, const struct nfnl_info *info,
return nf_tables_addchain(&ctx, family, policy, flags, extack);
}
+static int nft_trans_delhook(struct nft_hook *hook,
+ struct list_head *del_list)
+{
+ struct nft_trans_hook *trans_hook;
+
+ trans_hook = kmalloc_obj(*trans_hook, GFP_KERNEL);
+ if (!trans_hook)
+ return -ENOMEM;
+
+ trans_hook->hook = hook;
+ list_add_tail(&trans_hook->list, del_list);
+ hook->flags |= NFT_HOOK_REMOVE;
+
+ return 0;
+}
+
+static void nft_trans_delhook_abort(struct list_head *del_list)
+{
+ struct nft_trans_hook *trans_hook, *next;
+
+ list_for_each_entry_safe(trans_hook, next, del_list, list) {
+ trans_hook->hook->flags &= ~NFT_HOOK_REMOVE;
+ nft_trans_hook_destroy(trans_hook);
+ }
+}
+
static int nft_delchain_hook(struct nft_ctx *ctx,
struct nft_base_chain *basechain,
struct netlink_ext_ack *extack)
@@ -3139,7 +3252,10 @@ static int nft_delchain_hook(struct nft_ctx *ctx,
err = -ENOENT;
goto err_chain_del_hook;
}
- list_move(&hook->list, &chain_del_list);
+ if (nft_trans_delhook(hook, &chain_del_list) < 0) {
+ err = -ENOMEM;
+ goto err_chain_del_hook;
+ }
}
trans = nft_trans_alloc_chain(ctx, NFT_MSG_DELCHAIN);
@@ -3159,7 +3275,7 @@ static int nft_delchain_hook(struct nft_ctx *ctx,
return 0;
err_chain_del_hook:
- list_splice(&chain_del_list, &basechain->hook_list);
+ nft_trans_delhook_abort(&chain_del_list);
nft_chain_release_hook(&chain_hook);
return err;
@@ -6790,39 +6906,11 @@ static void nft_trans_elems_destroy(const struct nft_ctx *ctx,
nf_tables_set_elem_destroy(ctx, te->set, te->elems[i].priv);
}
-int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set,
- struct nft_expr *expr_array[])
-{
- struct nft_expr *expr;
- int err, i, k;
-
- for (i = 0; i < set->num_exprs; i++) {
- expr = kzalloc(set->exprs[i]->ops->size, GFP_KERNEL_ACCOUNT);
- if (!expr)
- goto err_expr;
-
- err = nft_expr_clone(expr, set->exprs[i], GFP_KERNEL_ACCOUNT);
- if (err < 0) {
- kfree(expr);
- goto err_expr;
- }
- expr_array[i] = expr;
- }
-
- return 0;
-
-err_expr:
- for (k = i - 1; k >= 0; k--)
- nft_expr_destroy(ctx, expr_array[k]);
-
- return -ENOMEM;
-}
-
static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
const struct nft_set_ext_tmpl *tmpl,
const struct nft_set_ext *ext,
struct nft_expr *expr_array[],
- u32 num_exprs)
+ u32 num_exprs, bool override_exprs)
{
struct nft_set_elem_expr *elem_expr = nft_set_ext_expr(ext);
u32 len = sizeof(struct nft_set_elem_expr);
@@ -6845,7 +6933,8 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
goto err_elem_expr_setup;
elem_expr->size += expr_array[i]->ops->size;
- nft_expr_destroy(ctx, expr_array[i]);
+ if (override_exprs)
+ nft_expr_destroy(ctx, expr_array[i]);
expr_array[i] = NULL;
}
@@ -6853,7 +6942,9 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
err_elem_expr_setup:
for (; i < num_exprs; i++) {
- nft_expr_destroy(ctx, expr_array[i]);
+ if (override_exprs)
+ nft_expr_destroy(ctx, expr_array[i]);
+
expr_array[i] = NULL;
}
@@ -7139,6 +7230,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
struct nft_set_binding *binding;
struct nft_elem_priv *elem_priv;
struct nft_object *obj = NULL;
+ bool override_exprs = false;
struct nft_userdata *udata;
struct nft_data_desc desc;
enum nft_registers dreg;
@@ -7244,6 +7336,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
expr_array[0] = expr;
num_exprs = 1;
+ override_exprs = true;
if (set->num_exprs && set->exprs[0]->ops != expr->ops) {
err = -EOPNOTSUPP;
@@ -7272,6 +7365,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
}
expr_array[i] = expr;
num_exprs++;
+ override_exprs = true;
if (set->num_exprs && expr->ops != set->exprs[i]->ops) {
err = -EOPNOTSUPP;
@@ -7285,9 +7379,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
}
} else if (set->num_exprs > 0 &&
!(flags & NFT_SET_ELEM_INTERVAL_END)) {
- err = nft_set_elem_expr_clone(ctx, set, expr_array);
- if (err < 0)
- goto err_set_elem_expr_clone;
+ for (i = 0; i < set->num_exprs; i++)
+ expr_array[i] = set->exprs[i];
num_exprs = set->num_exprs;
}
@@ -7426,7 +7519,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
udata->len = ulen - 1;
nla_memcpy(&udata->data, nla[NFTA_SET_ELEM_USERDATA], ulen);
}
- err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs);
+ err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs,
+ override_exprs);
if (err < 0)
goto err_elem_free;
@@ -7534,9 +7628,11 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
err_parse_key:
nft_data_release(&elem.key.val, NFT_DATA_VALUE);
err_set_elem_expr:
- for (i = 0; i < num_exprs && expr_array[i]; i++)
- nft_expr_destroy(ctx, expr_array[i]);
-err_set_elem_expr_clone:
+ if (override_exprs) {
+ for (i = 0; i < num_exprs && expr_array[i]; i++)
+ nft_expr_destroy(ctx, expr_array[i]);
+ }
+
return err;
}
@@ -8970,6 +9066,24 @@ static void nft_hooks_destroy(struct list_head *hook_list)
nft_netdev_hook_unlink_free_rcu(hook);
}
+static void nft_flowtable_unregister_trans_hook(struct net *net,
+ struct nft_flowtable *flowtable,
+ struct list_head *hook_list)
+{
+ struct nft_trans_hook *trans_hook, *next;
+ struct nf_hook_ops *ops;
+ struct nft_hook *hook;
+
+ list_for_each_entry_safe(trans_hook, next, hook_list, list) {
+ hook = trans_hook->hook;
+ list_for_each_entry(ops, &hook->ops_list, list)
+ nft_unregister_flowtable_ops(net, flowtable, ops);
+
+ nft_netdev_hook_unlink_free_rcu(hook);
+ nft_trans_hook_destroy(trans_hook);
+ }
+}
+
static int nft_flowtable_update(struct nft_ctx *ctx, const struct nlmsghdr *nlh,
struct nft_flowtable *flowtable,
struct netlink_ext_ack *extack)
@@ -9228,7 +9342,10 @@ static int nft_delflowtable_hook(struct nft_ctx *ctx,
err = -ENOENT;
goto err_flowtable_del_hook;
}
- list_move(&hook->list, &flowtable_del_list);
+ if (nft_trans_delhook(hook, &flowtable_del_list) < 0) {
+ err = -ENOMEM;
+ goto err_flowtable_del_hook;
+ }
}
trans = nft_trans_alloc(ctx, NFT_MSG_DELFLOWTABLE,
@@ -9249,7 +9366,7 @@ static int nft_delflowtable_hook(struct nft_ctx *ctx,
return 0;
err_flowtable_del_hook:
- list_splice(&flowtable_del_list, &flowtable->hook_list);
+ nft_trans_delhook_abort(&flowtable_del_list);
nft_flowtable_hook_release(&flowtable_hook);
return err;
@@ -9314,8 +9431,10 @@ static int nf_tables_fill_flowtable_info(struct sk_buff *skb, struct net *net,
u32 portid, u32 seq, int event,
u32 flags, int family,
struct nft_flowtable *flowtable,
- struct list_head *hook_list)
+ struct list_head *hook_list,
+ struct list_head *trans_hook_list)
{
+ struct nft_trans_hook *trans_hook;
struct nlattr *nest, *nest_devs;
struct nft_hook *hook;
struct nlmsghdr *nlh;
@@ -9332,7 +9451,7 @@ static int nf_tables_fill_flowtable_info(struct sk_buff *skb, struct net *net,
NFTA_FLOWTABLE_PAD))
goto nla_put_failure;
- if (!hook_list &&
+ if (!hook_list && !trans_hook_list &&
(event == NFT_MSG_DELFLOWTABLE ||
event == NFT_MSG_DESTROYFLOWTABLE)) {
nlmsg_end(skb, nlh);
@@ -9354,13 +9473,20 @@ static int nf_tables_fill_flowtable_info(struct sk_buff *skb, struct net *net,
if (!nest_devs)
goto nla_put_failure;
- if (!hook_list)
+ if (!hook_list && !trans_hook_list)
hook_list = &flowtable->hook_list;
- list_for_each_entry_rcu(hook, hook_list, list,
- lockdep_commit_lock_is_held(net)) {
- if (nft_nla_put_hook_dev(skb, hook))
- goto nla_put_failure;
+ if (hook_list) {
+ list_for_each_entry_rcu(hook, hook_list, list,
+ lockdep_commit_lock_is_held(net)) {
+ if (nft_nla_put_hook_dev(skb, hook))
+ goto nla_put_failure;
+ }
+ } else if (trans_hook_list) {
+ list_for_each_entry(trans_hook, trans_hook_list, list) {
+ if (nft_nla_put_hook_dev(skb, trans_hook->hook))
+ goto nla_put_failure;
+ }
}
nla_nest_end(skb, nest_devs);
nla_nest_end(skb, nest);
@@ -9414,7 +9540,7 @@ static int nf_tables_dump_flowtable(struct sk_buff *skb,
NFT_MSG_NEWFLOWTABLE,
NLM_F_MULTI | NLM_F_APPEND,
table->family,
- flowtable, NULL) < 0)
+ flowtable, NULL, NULL) < 0)
goto done;
nl_dump_check_consistent(cb, nlmsg_hdr(skb));
@@ -9514,7 +9640,7 @@ static int nf_tables_getflowtable(struct sk_buff *skb,
err = nf_tables_fill_flowtable_info(skb2, net, NETLINK_CB(skb).portid,
info->nlh->nlmsg_seq,
NFT_MSG_NEWFLOWTABLE, 0, family,
- flowtable, NULL);
+ flowtable, NULL, NULL);
if (err < 0)
goto err_fill_flowtable_info;
@@ -9527,7 +9653,9 @@ static int nf_tables_getflowtable(struct sk_buff *skb,
static void nf_tables_flowtable_notify(struct nft_ctx *ctx,
struct nft_flowtable *flowtable,
- struct list_head *hook_list, int event)
+ struct list_head *hook_list,
+ struct list_head *trans_hook_list,
+ int event)
{
struct nftables_pernet *nft_net = nft_pernet(ctx->net);
struct sk_buff *skb;
@@ -9547,7 +9675,8 @@ static void nf_tables_flowtable_notify(struct nft_ctx *ctx,
err = nf_tables_fill_flowtable_info(skb, ctx->net, ctx->portid,
ctx->seq, event, flags,
- ctx->family, flowtable, hook_list);
+ ctx->family, flowtable,
+ hook_list, trans_hook_list);
if (err < 0) {
kfree_skb(skb);
goto err;
@@ -10081,9 +10210,7 @@ static void nft_commit_release(struct nft_trans *trans)
break;
case NFT_MSG_DELCHAIN:
case NFT_MSG_DESTROYCHAIN:
- if (nft_trans_chain_update(trans))
- nft_hooks_destroy(&nft_trans_chain_hooks(trans));
- else
+ if (!nft_trans_chain_update(trans))
nf_tables_chain_destroy(nft_trans_chain(trans));
break;
case NFT_MSG_DELRULE:
@@ -10104,9 +10231,7 @@ static void nft_commit_release(struct nft_trans *trans)
break;
case NFT_MSG_DELFLOWTABLE:
case NFT_MSG_DESTROYFLOWTABLE:
- if (nft_trans_flowtable_update(trans))
- nft_hooks_destroy(&nft_trans_flowtable_hooks(trans));
- else
+ if (!nft_trans_flowtable_update(trans))
nf_tables_flowtable_destroy(nft_trans_flowtable(trans));
break;
}
@@ -10872,31 +10997,28 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
if (nft_trans_chain_update(trans)) {
nft_chain_commit_update(nft_trans_container_chain(trans));
nf_tables_chain_notify(&ctx, NFT_MSG_NEWCHAIN,
- &nft_trans_chain_hooks(trans));
+ &nft_trans_chain_hooks(trans), NULL);
list_splice_rcu(&nft_trans_chain_hooks(trans),
&nft_trans_basechain(trans)->hook_list);
/* trans destroyed after rcu grace period */
} else {
nft_chain_commit_drop_policy(nft_trans_container_chain(trans));
nft_clear(net, nft_trans_chain(trans));
- nf_tables_chain_notify(&ctx, NFT_MSG_NEWCHAIN, NULL);
+ nf_tables_chain_notify(&ctx, NFT_MSG_NEWCHAIN, NULL, NULL);
nft_trans_destroy(trans);
}
break;
case NFT_MSG_DELCHAIN:
case NFT_MSG_DESTROYCHAIN:
if (nft_trans_chain_update(trans)) {
- nf_tables_chain_notify(&ctx, NFT_MSG_DELCHAIN,
+ nf_tables_chain_notify(&ctx, NFT_MSG_DELCHAIN, NULL,
&nft_trans_chain_hooks(trans));
- if (!(table->flags & NFT_TABLE_F_DORMANT)) {
- nft_netdev_unregister_hooks(net,
- &nft_trans_chain_hooks(trans),
- true);
- }
+ nft_netdev_unregister_trans_hook(net, table,
+ &nft_trans_chain_hooks(trans));
} else {
nft_chain_del(nft_trans_chain(trans));
nf_tables_chain_notify(&ctx, NFT_MSG_DELCHAIN,
- NULL);
+ NULL, NULL);
nf_tables_unregister_hook(ctx.net, ctx.table,
nft_trans_chain(trans));
}
@@ -11002,6 +11124,7 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
nf_tables_flowtable_notify(&ctx,
nft_trans_flowtable(trans),
&nft_trans_flowtable_hooks(trans),
+ NULL,
NFT_MSG_NEWFLOWTABLE);
list_splice_rcu(&nft_trans_flowtable_hooks(trans),
&nft_trans_flowtable(trans)->hook_list);
@@ -11010,6 +11133,7 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
nf_tables_flowtable_notify(&ctx,
nft_trans_flowtable(trans),
NULL,
+ NULL,
NFT_MSG_NEWFLOWTABLE);
}
nft_trans_destroy(trans);
@@ -11019,16 +11143,18 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
if (nft_trans_flowtable_update(trans)) {
nf_tables_flowtable_notify(&ctx,
nft_trans_flowtable(trans),
+ NULL,
&nft_trans_flowtable_hooks(trans),
trans->msg_type);
- nft_unregister_flowtable_net_hooks(net,
- nft_trans_flowtable(trans),
- &nft_trans_flowtable_hooks(trans));
+ nft_flowtable_unregister_trans_hook(net,
+ nft_trans_flowtable(trans),
+ &nft_trans_flowtable_hooks(trans));
} else {
list_del_rcu(&nft_trans_flowtable(trans)->list);
nf_tables_flowtable_notify(&ctx,
nft_trans_flowtable(trans),
NULL,
+ NULL,
trans->msg_type);
nft_unregister_flowtable_net_hooks(net,
nft_trans_flowtable(trans),
@@ -11192,8 +11318,7 @@ static int __nf_tables_abort(struct net *net, enum nfnl_abort_action action)
case NFT_MSG_DELCHAIN:
case NFT_MSG_DESTROYCHAIN:
if (nft_trans_chain_update(trans)) {
- list_splice(&nft_trans_chain_hooks(trans),
- &nft_trans_basechain(trans)->hook_list);
+ nft_trans_delhook_abort(&nft_trans_chain_hooks(trans));
} else {
nft_use_inc_restore(&table->use);
nft_clear(trans->net, nft_trans_chain(trans));
@@ -11307,8 +11432,7 @@ static int __nf_tables_abort(struct net *net, enum nfnl_abort_action action)
case NFT_MSG_DELFLOWTABLE:
case NFT_MSG_DESTROYFLOWTABLE:
if (nft_trans_flowtable_update(trans)) {
- list_splice(&nft_trans_flowtable_hooks(trans),
- &nft_trans_flowtable(trans)->hook_list);
+ nft_trans_delhook_abort(&nft_trans_flowtable_hooks(trans));
} else {
nft_use_inc_restore(&table->use);
nft_clear(trans->net, nft_trans_flowtable(trans));
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 82187ddd26c1..cf545ed26161 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -33,6 +33,7 @@
#include <linux/rhashtable.h>
#include <linux/jhash.h>
#include <net/gso.h>
+#include <net/ip.h>
#include <net/sock.h>
#include <net/tcp_states.h>
#include <net/netfilter/nf_queue.h>
@@ -653,6 +654,12 @@ static int nfqnl_put_bridge(struct nf_queue_entry *entry, struct sk_buff *skb)
static int nf_queue_checksum_help(struct sk_buff *entskb)
{
+ if (entskb->protocol == htons(ETH_P_IP)) {
+ if (!pskb_network_may_pull(entskb, sizeof(struct iphdr)) ||
+ ip_check_csum_start(entskb))
+ return -EINVAL;
+ }
+
if (skb_csum_is_sctp(entskb))
return skb_crc32c_csum_help(entskb);
diff --git a/net/netfilter/nft_dynset.c b/net/netfilter/nft_dynset.c
index 57bf94ae8724..827bcf4dcd69 100644
--- a/net/netfilter/nft_dynset.c
+++ b/net/netfilter/nft_dynset.c
@@ -19,7 +19,8 @@ struct nft_dynset {
u8 sreg_key;
u8 sreg_data;
bool invert;
- bool expr;
+ bool expr:1,
+ override_exprs:1;
u8 num_exprs;
u64 timeout;
struct nft_expr *expr_array[NFT_SET_EXPR_MAX];
@@ -256,6 +257,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
priv->num_exprs++;
priv->expr_array[0] = dynset_expr;
+ priv->override_exprs = true;
if (set->num_exprs > 1 ||
(set->num_exprs == 1 &&
@@ -288,6 +290,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
}
priv->expr_array[i] = dynset_expr;
priv->num_exprs++;
+ priv->override_exprs = true;
if (set->num_exprs) {
if (i >= set->num_exprs) {
@@ -306,9 +309,8 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
goto err_expr_free;
}
} else if (set->num_exprs > 0) {
- err = nft_set_elem_expr_clone(ctx, set, priv->expr_array);
- if (err < 0)
- return err;
+ for (i = 0; i < set->num_exprs; i++)
+ priv->expr_array[i] = set->exprs[i];
priv->num_exprs = set->num_exprs;
}
@@ -338,8 +340,10 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
return 0;
err_expr_free:
- for (i = 0; i < priv->num_exprs; i++)
- nft_expr_destroy(ctx, priv->expr_array[i]);
+ if (priv->override_exprs) {
+ for (i = 0; i < priv->num_exprs; i++)
+ nft_expr_destroy(ctx, priv->expr_array[i]);
+ }
return err;
}
@@ -366,9 +370,10 @@ static void nft_dynset_destroy(const struct nft_ctx *ctx,
struct nft_dynset *priv = nft_expr_priv(expr);
int i;
- for (i = 0; i < priv->num_exprs; i++)
- nft_expr_destroy(ctx, priv->expr_array[i]);
-
+ if (priv->override_exprs) {
+ for (i = 0; i < priv->num_exprs; i++)
+ nft_expr_destroy(ctx, priv->expr_array[i]);
+ }
nf_tables_destroy_set(ctx, priv->set);
}
@@ -392,7 +397,7 @@ static int nft_dynset_dump(struct sk_buff *skb,
nf_jiffies64_to_msecs(priv->timeout),
NFTA_DYNSET_PAD))
goto nla_put_failure;
- if (priv->set->num_exprs == 0) {
+ if (priv->set->num_exprs == 0 || priv->override_exprs) {
if (priv->num_exprs == 1) {
if (nft_expr_dump(skb, NFTA_DYNSET_EXPR,
priv->expr_array[0], reset))
diff --git a/net/netfilter/nft_fwd_netdev.c b/net/netfilter/nft_fwd_netdev.c
index 4bce36c3a6a0..eceab479bd8c 100644
--- a/net/netfilter/nft_fwd_netdev.c
+++ b/net/netfilter/nft_fwd_netdev.c
@@ -101,6 +101,7 @@ static void nft_fwd_neigh_eval(const struct nft_expr *expr,
unsigned int verdict = NF_STOLEN;
struct sk_buff *skb = pkt->skb;
struct net_device *dev;
+ unsigned int hh_len;
int neigh_table;
switch (priv->nfproto) {
@@ -152,12 +153,34 @@ static void nft_fwd_neigh_eval(const struct nft_expr *expr,
}
dev = dev_get_by_index_rcu(nft_net(pkt), oif);
- if (dev == NULL)
- return;
+ if (!dev) {
+ verdict = NF_DROP;
+ goto out;
+ }
+
+ local_bh_disable();
+ if (nf_dev_xmit_recursion()) {
+ local_bh_enable();
+ verdict = NF_DROP;
+ goto out;
+ }
+
+ hh_len = LL_RESERVED_SPACE(dev);
+ if (unlikely(skb_headroom(skb) < hh_len && dev->header_ops)) {
+ skb = skb_expand_head(skb, hh_len);
+ if (!skb) {
+ local_bh_enable();
+ goto out;
+ }
+ }
skb->dev = dev;
skb_clear_tstamp(skb);
+
+ nf_dev_xmit_recursion_inc();
neigh_xmit(neigh_table, dev, addr, skb);
+ nf_dev_xmit_recursion_dec();
+ local_bh_enable();
out:
regs->verdict.code = verdict;
}
diff --git a/net/netfilter/x_tables.c b/net/netfilter/x_tables.c
index 76fd0999db4a..f4b32b6c70bc 100644
--- a/net/netfilter/x_tables.c
+++ b/net/netfilter/x_tables.c
@@ -1408,11 +1408,9 @@ struct xt_counters *xt_counters_alloc(unsigned int counters)
}
EXPORT_SYMBOL(xt_counters_alloc);
-struct xt_table_info *
-xt_replace_table(struct xt_table *table,
- unsigned int num_counters,
- struct xt_table_info *newinfo,
- int *error)
+static struct xt_table_info *
+do_replace_table(struct xt_table *table, unsigned int num_counters,
+ struct xt_table_info *newinfo, int *error)
{
struct xt_table_info *private;
unsigned int cpu;
@@ -1467,30 +1465,54 @@ xt_replace_table(struct xt_table *table,
}
}
- audit_log_nfcfg(table->name, table->af, private->number,
- !private->number ? AUDIT_XT_OP_REGISTER :
- AUDIT_XT_OP_REPLACE,
- GFP_KERNEL);
+ return private;
+}
+
+struct xt_table_info *
+xt_replace_table(struct xt_table *table, unsigned int num_counters,
+ struct xt_table_info *newinfo,
+ int *error)
+{
+ struct xt_table_info *private;
+
+ private = do_replace_table(table, num_counters, newinfo, error);
+ if (private)
+ audit_log_nfcfg(table->name, table->af, private->number,
+ AUDIT_XT_OP_REPLACE,
+ GFP_KERNEL);
+
return private;
}
EXPORT_SYMBOL_GPL(xt_replace_table);
struct xt_table *xt_register_table(struct net *net,
const struct xt_table *input_table,
+ const struct nf_hook_ops *template_ops,
struct xt_table_info *bootstrap,
struct xt_table_info *newinfo)
{
struct xt_pernet *xt_net = net_generic(net, xt_pernet_id);
+ struct xt_table *t, *table = NULL;
+ struct nf_hook_ops *ops = NULL;
struct xt_table_info *private;
- struct xt_table *t, *table;
- int ret;
+ unsigned int num_ops;
+ int ret = -EINVAL;
+
+ num_ops = hweight32(input_table->valid_hooks);
+ if (num_ops == 0)
+ goto out;
+
+ ret = -ENOMEM;
+ if (template_ops) {
+ ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL);
+ if (!ops)
+ goto out;
+ }
/* Don't add one object to multiple lists. */
table = kmemdup(input_table, sizeof(struct xt_table), GFP_KERNEL);
- if (!table) {
- ret = -ENOMEM;
+ if (!table)
goto out;
- }
mutex_lock(&xt[table->af].mutex);
/* Don't autoload: we'd eat our tail... */
@@ -1504,7 +1526,7 @@ struct xt_table *xt_register_table(struct net *net,
/* Simplifies replace_table code. */
table->private = bootstrap;
- if (!xt_replace_table(table, 0, newinfo, &ret))
+ if (!do_replace_table(table, 0, newinfo, &ret))
goto unlock;
private = table->private;
@@ -1513,14 +1535,37 @@ struct xt_table *xt_register_table(struct net *net,
/* save number of initial entries */
private->initial_entries = private->number;
+ if (ops) {
+ int i;
+
+ for (i = 0; i < num_ops; i++)
+ ops[i].priv = table;
+
+ ret = nf_register_net_hooks(net, ops, num_ops);
+ if (ret != 0) {
+ mutex_unlock(&xt[table->af].mutex);
+ /* nf_register_net_hooks() might have published a
+ * base chain before internal error unwind.
+ */
+ synchronize_rcu();
+ goto out;
+ }
+
+ table->ops = ops;
+ }
+
+ audit_log_nfcfg(table->name, table->af, private->number,
+ AUDIT_XT_OP_REGISTER, GFP_KERNEL);
+
list_add(&table->list, &xt_net->tables[table->af]);
mutex_unlock(&xt[table->af].mutex);
return table;
unlock:
mutex_unlock(&xt[table->af].mutex);
- kfree(table);
out:
+ kfree(table);
+ kfree(ops);
return ERR_PTR(ret);
}
EXPORT_SYMBOL_GPL(xt_register_table);
diff --git a/net/netfilter/xt_CHECKSUM.c b/net/netfilter/xt_CHECKSUM.c
index 9d99f5a3d176..1fb0f8118404 100644
--- a/net/netfilter/xt_CHECKSUM.c
+++ b/net/netfilter/xt_CHECKSUM.c
@@ -15,6 +15,7 @@
#include <linux/netfilter_ipv4/ip_tables.h>
#include <linux/netfilter_ipv6/ip6_tables.h>
+#include <net/ip.h>
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Michael S. Tsirkin <mst@redhat.com>");
@@ -25,8 +26,11 @@ MODULE_ALIAS("ip6t_CHECKSUM");
static unsigned int
checksum_tg(struct sk_buff *skb, const struct xt_action_param *par)
{
- if (skb->ip_summed == CHECKSUM_PARTIAL && !skb_is_gso(skb))
+ if (skb->ip_summed == CHECKSUM_PARTIAL && !skb_is_gso(skb)) {
+ if (xt_family(par) == NFPROTO_IPV4 && ip_check_csum_start(skb))
+ return NF_DROP;
skb_checksum_help(skb);
+ }
return XT_CONTINUE;
}
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index 1d1b3bd54a91..fd8e9cfe6171 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -968,7 +968,7 @@ static void netlink_undo_bind(int group, long unsigned int groups,
nlk->netlink_unbind(sock_net(sk), undo + 1);
}
-static int netlink_bind(struct socket *sock, struct sockaddr *addr,
+static int netlink_bind(struct socket *sock, struct sockaddr_unsized *addr,
int addr_len)
{
struct sock *sk = sock->sk;
diff --git a/net/netrom/af_netrom.c b/net/netrom/af_netrom.c
index 3331669d8e33..33468124d53d 100644
--- a/net/netrom/af_netrom.c
+++ b/net/netrom/af_netrom.c
@@ -561,7 +561,7 @@ static int nr_release(struct socket *sock)
return 0;
}
-static int nr_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int nr_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sock *sk = sock->sk;
struct nr_sock *nr = nr_sk(sk);
diff --git a/net/nfc/llcp_sock.c b/net/nfc/llcp_sock.c
index 0601ddb4304a..21cef27e2032 100644
--- a/net/nfc/llcp_sock.c
+++ b/net/nfc/llcp_sock.c
@@ -56,7 +56,7 @@ static struct proto llcp_sock_proto = {
.obj_size = sizeof(struct nfc_llcp_sock),
};
-static int llcp_sock_bind(struct socket *sock, struct sockaddr *addr, int alen)
+static int llcp_sock_bind(struct socket *sock, struct sockaddr_unsized *addr, int alen)
{
struct sock *sk = sock->sk;
struct nfc_llcp_sock *llcp_sock = nfc_llcp_sock(sk);
@@ -146,7 +146,7 @@ static int llcp_sock_bind(struct socket *sock, struct sockaddr *addr, int alen)
return ret;
}
-static int llcp_raw_sock_bind(struct socket *sock, struct sockaddr *addr,
+static int llcp_raw_sock_bind(struct socket *sock, struct sockaddr_unsized *addr,
int alen)
{
struct sock *sk = sock->sk;
diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c
index 5d6a4c6b7ea6..9d9697cf6329 100644
--- a/net/openvswitch/actions.c
+++ b/net/openvswitch/actions.c
@@ -357,7 +357,12 @@ static void set_ip_addr(struct sk_buff *skb, struct iphdr *nh,
static void update_ipv6_checksum(struct sk_buff *skb, u8 l4_proto,
__be32 addr[4], const __be32 new_addr[4])
{
- int transport_len = skb->len - skb_transport_offset(skb);
+ int transport_len;
+
+ if (l4_proto == NEXTHDR_FRAGMENT)
+ return;
+
+ transport_len = skb->len - skb_transport_offset(skb);
if (l4_proto == NEXTHDR_TCP) {
if (likely(transport_len >= sizeof(struct tcphdr)))
diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c
index c7991b9db6a7..060b163555fa 100644
--- a/net/openvswitch/datapath.c
+++ b/net/openvswitch/datapath.c
@@ -1529,6 +1529,12 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb)
return -ENODEV;
}
+ /*
+ * Not needed for safety. Stops every spin_unlock_bh() in
+ * ovs_flow_stats_get() from running the softirq backlog.
+ */
+ local_bh_disable();
+
ti = rcu_dereference(dp->table.ti);
for (;;) {
struct sw_flow *flow;
@@ -1549,6 +1555,8 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb)
cb->args[0] = bucket;
cb->args[1] = obj;
}
+
+ local_bh_enable();
rcu_read_unlock();
return skb->len;
}
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 003727af1748..fc56efe64a28 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1910,7 +1910,8 @@ static int packet_rcv_spkt(struct sk_buff *skb, struct net_device *dev,
spkt = &PACKET_SKB_CB(skb)->sa.pkt;
- skb_push(skb, skb->data - skb_mac_header(skb));
+ if (dev_has_header(dev))
+ skb_push(skb, skb->data - skb_mac_header(skb));
/*
* The SOCK_PACKET socket receives _all_ frames.
@@ -3345,11 +3346,12 @@ static int packet_do_bind(struct sock *sk, const char *name, int ifindex,
* Bind a packet socket to a device
*/
-static int packet_bind_spkt(struct socket *sock, struct sockaddr *uaddr,
+static int packet_bind_spkt(struct socket *sock, struct sockaddr_unsized *uaddr,
int addr_len)
{
struct sock *sk = sock->sk;
- char name[sizeof(uaddr->sa_data_min) + 1];
+ struct sockaddr *sa = (struct sockaddr *)uaddr;
+ char name[sizeof(sa->sa_data_min) + 1];
/*
* Check legality
@@ -3360,13 +3362,13 @@ static int packet_bind_spkt(struct socket *sock, struct sockaddr *uaddr,
/* uaddr->sa_data comes from the userspace, it's not guaranteed to be
* zero-terminated.
*/
- memcpy(name, uaddr->sa_data, sizeof(uaddr->sa_data_min));
- name[sizeof(uaddr->sa_data_min)] = 0;
+ memcpy(name, sa->sa_data, sizeof(sa->sa_data_min));
+ name[sizeof(sa->sa_data_min)] = 0;
return packet_do_bind(sk, name, 0, 0);
}
-static int packet_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int packet_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sockaddr_ll *sll = (struct sockaddr_ll *)uaddr;
struct sock *sk = sock->sk;
diff --git a/net/phonet/socket.c b/net/phonet/socket.c
index db2d552e9b32..ac21a83d9a4d 100644
--- a/net/phonet/socket.c
+++ b/net/phonet/socket.c
@@ -153,7 +153,7 @@ EXPORT_SYMBOL(pn_sock_unhash);
static DEFINE_MUTEX(port_mutex);
-static int pn_socket_bind(struct socket *sock, struct sockaddr *addr, int len)
+static int pn_socket_bind(struct socket *sock, struct sockaddr_unsized *addr, int len)
{
struct sock *sk = sock->sk;
struct pn_sock *pn = pn_sk(sk);
@@ -163,7 +163,7 @@ static int pn_socket_bind(struct socket *sock, struct sockaddr *addr, int len)
u8 saddr;
if (sk->sk_prot->bind)
- return sk->sk_prot->bind(sk, addr, len);
+ return sk->sk_prot->bind(sk, (struct sockaddr *)addr, len);
if (len < sizeof(struct sockaddr_pn))
return -EINVAL;
@@ -206,11 +206,17 @@ static int pn_socket_autobind(struct socket *sock)
memset(&sa, 0, sizeof(sa));
sa.spn_family = AF_PHONET;
- err = pn_socket_bind(sock, (struct sockaddr *)&sa,
- sizeof(struct sockaddr_pn));
- if (err != -EINVAL)
+ err = pn_socket_bind(sock, (struct sockaddr_unsized *)&sa,
+ sizeof(struct sockaddr_pn));
+ /*
+ * pn_socket_bind() also returns -EINVAL when sk_state != TCP_CLOSE
+ * without a prior bind, so -EINVAL alone is not sufficient to infer
+ * that the socket was already bound. Only treat it as "already
+ * bound" when the port is non-zero; otherwise propagate the error
+ * instead of crashing the kernel.
+ */
+ if (err != -EINVAL || unlikely(!pn_port(pn_sk(sock->sk)->sobject)))
return err;
- BUG_ON(!pn_port(pn_sk(sock->sk)->sobject));
return 0; /* socket was already bound */
}
diff --git a/net/qrtr/af_qrtr.c b/net/qrtr/af_qrtr.c
index 1f40496d84a3..0c08ac0d8881 100644
--- a/net/qrtr/af_qrtr.c
+++ b/net/qrtr/af_qrtr.c
@@ -819,7 +819,7 @@ static int qrtr_autobind(struct socket *sock)
}
/* Bind socket to specified sockaddr. */
-static int qrtr_bind(struct socket *sock, struct sockaddr *saddr, int len)
+static int qrtr_bind(struct socket *sock, struct sockaddr_unsized *saddr, int len)
{
DECLARE_SOCKADDR(struct sockaddr_qrtr *, addr, saddr);
struct qrtr_sock *ipc = qrtr_sk(sock->sk);
diff --git a/net/qrtr/ns.c b/net/qrtr/ns.c
index e8e3ba1b3b82..b418188822c1 100644
--- a/net/qrtr/ns.c
+++ b/net/qrtr/ns.c
@@ -771,7 +771,7 @@ int qrtr_ns_init(void)
sq.sq_port = QRTR_PORT_CTRL;
qrtr_ns.local_node = sq.sq_node;
- ret = kernel_bind(qrtr_ns.sock, (struct sockaddr *)&sq, sizeof(sq));
+ ret = kernel_bind(qrtr_ns.sock, (struct sockaddr_unsized *)&sq, sizeof(sq));
if (ret < 0) {
pr_err("failed to bind to socket\n");
goto err_wq;
diff --git a/net/rds/bind.c b/net/rds/bind.c
index 97a29172a8ee..f800d920d969 100644
--- a/net/rds/bind.c
+++ b/net/rds/bind.c
@@ -160,7 +160,7 @@ void rds_remove_bound(struct rds_sock *rs)
rs->rs_bound_addr = in6addr_any;
}
-int rds_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+int rds_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sock *sk = sock->sk;
struct rds_sock *rs = rds_sk_to_rs(sk);
diff --git a/net/rds/rds.h b/net/rds/rds.h
index 5b1c072e2e7f..a029e5fcdea7 100644
--- a/net/rds/rds.h
+++ b/net/rds/rds.h
@@ -735,7 +735,7 @@ extern wait_queue_head_t rds_poll_waitq;
/* bind.c */
-int rds_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len);
+int rds_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len);
void rds_remove_bound(struct rds_sock *rs);
struct rds_sock *rds_find_bound(const struct in6_addr *addr, __be16 port,
__u32 scope_id);
diff --git a/net/rds/tcp_connect.c b/net/rds/tcp_connect.c
index a0046e99d6df..1eff3b03ab77 100644
--- a/net/rds/tcp_connect.c
+++ b/net/rds/tcp_connect.c
@@ -145,7 +145,7 @@ int rds_tcp_conn_path_connect(struct rds_conn_path *cp)
addrlen = sizeof(sin);
}
- ret = kernel_bind(sock, addr, addrlen);
+ ret = kernel_bind(sock, (struct sockaddr_unsized *)addr, addrlen);
if (ret) {
rdsdebug("bind failed with %d at address %pI6c\n",
ret, &conn->c_laddr);
diff --git a/net/rds/tcp_listen.c b/net/rds/tcp_listen.c
index 93e8ee10bc8d..53e59e227521 100644
--- a/net/rds/tcp_listen.c
+++ b/net/rds/tcp_listen.c
@@ -289,7 +289,7 @@ struct socket *rds_tcp_listen_init(struct net *net, bool isv6)
addr_len = sizeof(*sin);
}
- ret = kernel_bind(sock, (struct sockaddr *)&ss, addr_len);
+ ret = kernel_bind(sock, (struct sockaddr_unsized *)&ss, addr_len);
if (ret < 0) {
rdsdebug("could not bind %s listener socket: %d\n",
isv6 ? "IPv6" : "IPv4", ret);
diff --git a/net/rose/af_rose.c b/net/rose/af_rose.c
index 38bfe60665c0..97893498e611 100644
--- a/net/rose/af_rose.c
+++ b/net/rose/af_rose.c
@@ -713,7 +713,7 @@ static int rose_release(struct socket *sock)
return 0;
}
-static int rose_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int rose_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sock *sk = sock->sk;
struct rose_sock *rose = rose_sk(sk);
diff --git a/net/rxrpc/af_rxrpc.c b/net/rxrpc/af_rxrpc.c
index 885a28e3afc5..76c6b2875d31 100644
--- a/net/rxrpc/af_rxrpc.c
+++ b/net/rxrpc/af_rxrpc.c
@@ -127,7 +127,7 @@ static int rxrpc_validate_address(struct rxrpc_sock *rx,
/*
* bind a local address to an RxRPC socket
*/
-static int rxrpc_bind(struct socket *sock, struct sockaddr *saddr, int len)
+static int rxrpc_bind(struct socket *sock, struct sockaddr_unsized *saddr, int len)
{
struct sockaddr_rxrpc *srx = (struct sockaddr_rxrpc *)saddr;
struct rxrpc_local *local;
diff --git a/net/rxrpc/rxperf.c b/net/rxrpc/rxperf.c
index 2ea71e3831f7..98ea76fae70f 100644
--- a/net/rxrpc/rxperf.c
+++ b/net/rxrpc/rxperf.c
@@ -211,7 +211,7 @@ static int rxperf_open_socket(void)
ret = rxrpc_sock_set_security_keyring(socket->sk, rxperf_sec_keyring);
- ret = kernel_bind(socket, (struct sockaddr *)&srx, sizeof(srx));
+ ret = kernel_bind(socket, (struct sockaddr_unsized *)&srx, sizeof(srx));
if (ret < 0)
goto error_2;
diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c
index 0bc0c010434c..975d22dcf6b8 100644
--- a/net/sched/cls_api.c
+++ b/net/sched/cls_api.c
@@ -3386,7 +3386,7 @@ int tcf_exts_init_ex(struct tcf_exts *exts, struct net *net, int action,
*/
exts->net = net;
exts->actions = kcalloc(TCA_ACT_MAX_PRIO, sizeof(struct tc_action *),
- GFP_KERNEL);
+ GFP_KERNEL_ACCOUNT);
if (!exts->actions)
return -ENOMEM;
#endif
@@ -4082,9 +4082,7 @@ struct sk_buff *tcf_qevent_handle(struct tcf_qevent *qe, struct Qdisc *sch, stru
fl = rcu_dereference_bh(qe->filter_chain);
- switch (tcf_classify(skb, NULL, fl, &cl_res, false)) {
- case TC_ACT_REDIRECT:
- fallthrough;
+ switch (tcf_classify_qdisc(skb, fl, &cl_res, false)) {
case TC_ACT_SHOT:
qdisc_qstats_drop(sch);
__qdisc_drop(skb, to_free);
diff --git a/net/sched/cls_basic.c b/net/sched/cls_basic.c
index ecfaa4f9a04e..d5ef0a66a73f 100644
--- a/net/sched/cls_basic.c
+++ b/net/sched/cls_basic.c
@@ -193,7 +193,7 @@ static int basic_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
@@ -212,9 +212,11 @@ static int basic_change(struct net *net, struct sk_buff *in_skb,
if (err)
goto errout;
fnew->handle = handle;
- fnew->pf = alloc_percpu(struct tc_basic_pcnt);
+ fnew->pf = alloc_percpu_gfp(struct tc_basic_pcnt, GFP_KERNEL_ACCOUNT);
if (!fnew->pf) {
err = -ENOMEM;
+ if (!fold)
+ idr_remove(&head->handle_idr, fnew->handle);
goto errout;
}
diff --git a/net/sched/cls_bpf.c b/net/sched/cls_bpf.c
index a6003fa35c7c..66b525a85287 100644
--- a/net/sched/cls_bpf.c
+++ b/net/sched/cls_bpf.c
@@ -352,7 +352,7 @@ static int cls_bpf_prog_from_ops(struct nlattr **tb, struct cls_bpf_prog *prog)
if (bpf_size != nla_len(tb[TCA_BPF_OPS]))
return -EINVAL;
- bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL);
+ bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL_ACCOUNT);
if (bpf_ops == NULL)
return -ENOMEM;
@@ -403,7 +403,7 @@ static int cls_bpf_prog_from_efd(struct nlattr **tb, struct cls_bpf_prog *prog,
}
if (tb[TCA_BPF_NAME]) {
- name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL);
+ name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL_ACCOUNT);
if (!name) {
bpf_prog_put(fp);
return -ENOMEM;
@@ -443,7 +443,7 @@ static int cls_bpf_change(struct net *net, struct sk_buff *in_skb,
if (ret < 0)
return ret;
- prog = kzalloc(sizeof(*prog), GFP_KERNEL);
+ prog = kzalloc(sizeof(*prog), GFP_KERNEL_ACCOUNT);
if (!prog)
return -ENOBUFS;
diff --git a/net/sched/cls_cgroup.c b/net/sched/cls_cgroup.c
index 424252982d6a..8f6563e50e63 100644
--- a/net/sched/cls_cgroup.c
+++ b/net/sched/cls_cgroup.c
@@ -95,7 +95,7 @@ static int cls_cgroup_change(struct net *net, struct sk_buff *in_skb,
if (head && handle != head->handle)
return -ENOENT;
- new = kzalloc(sizeof(*head), GFP_KERNEL);
+ new = kzalloc(sizeof(*head), GFP_KERNEL_ACCOUNT);
if (!new)
return -ENOBUFS;
diff --git a/net/sched/cls_flow.c b/net/sched/cls_flow.c
index 9b472578bf75..61632e64d1fb 100644
--- a/net/sched/cls_flow.c
+++ b/net/sched/cls_flow.c
@@ -438,7 +438,7 @@ static int flow_change(struct net *net, struct sk_buff *in_skb,
return -EOPNOTSUPP;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c
index 099ff6a3e1f5..5e61bf334ed8 100644
--- a/net/sched/cls_flower.c
+++ b/net/sched/cls_flower.c
@@ -2237,7 +2237,7 @@ static struct fl_flow_mask *fl_create_new_mask(struct cls_fl_head *head,
struct fl_flow_mask *newmask;
int err;
- newmask = kzalloc(sizeof(*newmask), GFP_KERNEL);
+ newmask = kzalloc(sizeof(*newmask), GFP_KERNEL_ACCOUNT);
if (!newmask)
return ERR_PTR(-ENOMEM);
@@ -2398,7 +2398,7 @@ static int fl_change(struct net *net, struct sk_buff *in_skb,
goto errout_tb;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew) {
err = -ENOBUFS;
goto errout_tb;
diff --git a/net/sched/cls_fw.c b/net/sched/cls_fw.c
index fd9c6c2815a1..e32eecdf675f 100644
--- a/net/sched/cls_fw.c
+++ b/net/sched/cls_fw.c
@@ -276,7 +276,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb,
if (f->id != handle && handle)
return -EINVAL;
- fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL);
+ fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
@@ -330,7 +330,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb,
rcu_assign_pointer(tp->root, head);
}
- f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL);
+ f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT);
if (f == NULL)
return -ENOBUFS;
diff --git a/net/sched/cls_matchall.c b/net/sched/cls_matchall.c
index f03bf5da39ee..762ba74aa7be 100644
--- a/net/sched/cls_matchall.c
+++ b/net/sched/cls_matchall.c
@@ -189,7 +189,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
}
- new = kzalloc(sizeof(*new), GFP_KERNEL);
+ new = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT);
if (!new)
return -ENOBUFS;
@@ -201,7 +201,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb,
handle = 1;
new->handle = handle;
new->flags = userflags;
- new->pf = alloc_percpu(struct tc_matchall_pcnt);
+ new->pf = alloc_percpu_gfp(struct tc_matchall_pcnt, GFP_KERNEL_ACCOUNT);
if (!new->pf) {
err = -ENOMEM;
goto err_alloc_percpu;
diff --git a/net/sched/cls_route.c b/net/sched/cls_route.c
index 5a0ddcf3e2b4..1a41237d07aa 100644
--- a/net/sched/cls_route.c
+++ b/net/sched/cls_route.c
@@ -434,6 +434,12 @@ static int route4_set_parms(struct net *net, struct tcf_proto *tp,
if (handle && (!fold || nhandle == (handle & ~0x7F00)))
nhandle |= handle & 0x7F00;
+ if (fold && !tb[TCA_ROUTE4_TO] && !tb[TCA_ROUTE4_FROM] &&
+ !tb[TCA_ROUTE4_IIF] && nhandle != fold->handle) {
+ NL_SET_ERR_MSG(extack, "Routing attributes required");
+ return -EINVAL;
+ }
+
if (handle && !fold && nhandle != handle) {
NL_SET_ERR_MSG_FMT(extack,
"Handle mismatch constructed: %x (expected: %x)",
@@ -449,7 +455,7 @@ static int route4_set_parms(struct net *net, struct tcf_proto *tp,
h1 = to_hash(nhandle);
b = rtnl_dereference(head->table[h1]);
if (!b) {
- b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL);
+ b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL_ACCOUNT);
if (b == NULL)
return -ENOBUFS;
@@ -524,7 +530,7 @@ static int route4_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
err = -ENOBUFS;
- f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL);
+ f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL_ACCOUNT);
if (!f)
goto errout;
diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c
index e5e5533f866d..ac344aabe859 100644
--- a/net/sched/cls_u32.c
+++ b/net/sched/cls_u32.c
@@ -841,7 +841,7 @@ static struct tc_u_knode *u32_init_knode(struct net *net, struct tcf_proto *tp,
struct tc_u32_sel *s = &n->sel;
struct tc_u_knode *new;
- new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL);
+ new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL_ACCOUNT);
if (!new)
return NULL;
@@ -1132,15 +1132,23 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
goto erridr;
}
- n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL);
+ if (s->offshift >= 16) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offshift must be less than 16");
+ err = -EINVAL;
+ goto erridr;
+ }
+
+ n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL_ACCOUNT);
if (n == NULL) {
err = -ENOBUFS;
goto erridr;
}
#ifdef CONFIG_CLS_U32_PERF
- n->pf = __alloc_percpu(struct_size(n->pf, kcnts, s->nkeys),
- __alignof__(struct tc_u32_pcnt));
+ n->pf = __alloc_percpu_gfp(struct_size(n->pf, kcnts, s->nkeys),
+ __alignof__(struct tc_u32_pcnt),
+ GFP_KERNEL_ACCOUNT);
if (!n->pf) {
err = -ENOBUFS;
goto errfree;
@@ -1162,7 +1170,7 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
goto errout;
#ifdef CONFIG_CLS_U32_MARK
- n->pcpu_success = alloc_percpu(u32);
+ n->pcpu_success = alloc_percpu_gfp(u32, GFP_KERNEL_ACCOUNT);
if (!n->pcpu_success) {
err = -ENOMEM;
goto errout;
diff --git a/net/sched/em_text.c b/net/sched/em_text.c
index 8052c55b2f4d..6ee8c653bb94 100644
--- a/net/sched/em_text.c
+++ b/net/sched/em_text.c
@@ -58,6 +58,9 @@ static int em_text_change(struct net *net, void *data, int len,
if (len < sizeof(*conf) || len < (sizeof(*conf) + conf->pattern_len))
return -EINVAL;
+ if (!memchr(conf->algo, '\0', sizeof(conf->algo)))
+ return -EINVAL;
+
if (conf->from_layer > conf->to_layer)
return -EINVAL;
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index 8ed8ac120693..0e7007c545e8 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -1709,7 +1709,7 @@ static u32 cake_classify(struct Qdisc *sch, struct cake_tin_data **t,
goto hash;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
diff --git a/net/sched/sch_drr.c b/net/sched/sch_drr.c
index ba802db27993..8035ffeef2e9 100644
--- a/net/sched/sch_drr.c
+++ b/net/sched/sch_drr.c
@@ -313,7 +313,7 @@ static struct drr_class *drr_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
fl = rcu_dereference_bh(q->filter_list);
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_dualpi2.c b/net/sched/sch_dualpi2.c
index 4ee841f7a94d..4642549f2f52 100644
--- a/net/sched/sch_dualpi2.c
+++ b/net/sched/sch_dualpi2.c
@@ -367,7 +367,7 @@ static int dualpi2_skb_classify(struct dualpi2_sched_data *q,
if (!fl)
return NET_XMIT_SUCCESS;
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_ets.c b/net/sched/sch_ets.c
index 27e549646dee..03f5bb812a27 100644
--- a/net/sched/sch_ets.c
+++ b/net/sched/sch_ets.c
@@ -389,7 +389,7 @@ static struct ets_class *ets_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
if (TC_H_MAJ(skb->priority) != sch->handle) {
fl = rcu_dereference_bh(q->filter_list);
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c
index c135b1dd5972..4d32fd90f590 100644
--- a/net/sched/sch_fq_codel.c
+++ b/net/sched/sch_fq_codel.c
@@ -91,7 +91,7 @@ static unsigned int fq_codel_classify(struct sk_buff *skb, struct Qdisc *sch,
return fq_codel_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_fq_pie.c b/net/sched/sch_fq_pie.c
index ff26999afe4c..0d09c7e9f754 100644
--- a/net/sched/sch_fq_pie.c
+++ b/net/sched/sch_fq_pie.c
@@ -96,7 +96,7 @@ static unsigned int fq_pie_classify(struct sk_buff *skb, struct Qdisc *sch,
return fq_pie_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c
index df3501dfe01a..b7c7121c3611 100644
--- a/net/sched/sch_hfsc.c
+++ b/net/sched/sch_hfsc.c
@@ -1154,7 +1154,7 @@ hfsc_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
head = &q->root;
drift = HFSC_CLASSIFY_MAX_DRIFT;
tcf = rcu_dereference_bh(q->root.filter_list);
- while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) {
+ while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
case TC_ACT_QUEUED:
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index fc8c54039343..ae1bb2834c74 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -243,7 +243,7 @@ static struct htb_class *htb_classify(struct sk_buff *skb, struct Qdisc *sch,
}
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) {
+ while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
case TC_ACT_QUEUED:
diff --git a/net/sched/sch_multiq.c b/net/sched/sch_multiq.c
index e50a37b7cec1..7962fc0292a5 100644
--- a/net/sched/sch_multiq.c
+++ b/net/sched/sch_multiq.c
@@ -36,7 +36,7 @@ multiq_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
int err;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_netem.c b/net/sched/sch_netem.c
index 73b3a8ce2f43..84356f302bf6 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -460,7 +460,8 @@ static int netem_enqueue(struct sk_buff *skb, struct Qdisc *sch,
skb->prev = NULL;
/* Random duplication */
- if (q->duplicate && q->duplicate >= get_crandom(&q->dup_cor, &q->prng))
+ if (q->duplicate && skb->tc_depth == 0 &&
+ q->duplicate >= get_crandom(&q->dup_cor, &q->prng))
++count;
/* Drop packet? */
@@ -539,11 +540,9 @@ static int netem_enqueue(struct sk_buff *skb, struct Qdisc *sch,
*/
if (skb2) {
struct Qdisc *rootq = qdisc_root_bh(sch);
- u32 dupsave = q->duplicate; /* prevent duplicating a dup... */
- q->duplicate = 0;
+ skb2->tc_depth++; /* prevent duplicating a dup... */
rootq->enqueue(skb2, rootq, to_free);
- q->duplicate = dupsave;
skb2 = NULL;
}
diff --git a/net/sched/sch_prio.c b/net/sched/sch_prio.c
index 9e2b9a490db2..6f4388cf1048 100644
--- a/net/sched/sch_prio.c
+++ b/net/sched/sch_prio.c
@@ -39,7 +39,7 @@ prio_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
if (TC_H_MAJ(skb->priority) != sch->handle) {
fl = rcu_dereference_bh(q->filter_list);
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c
index f94c9c9c9042..7d93690cb436 100644
--- a/net/sched/sch_qfq.c
+++ b/net/sched/sch_qfq.c
@@ -709,7 +709,7 @@ static struct qfq_class *qfq_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
fl = rcu_dereference_bh(q->filter_list);
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_sfb.c b/net/sched/sch_sfb.c
index 14ac88977847..1b326b459ce9 100644
--- a/net/sched/sch_sfb.c
+++ b/net/sched/sch_sfb.c
@@ -260,7 +260,7 @@ static bool sfb_classify(struct sk_buff *skb, struct tcf_proto *fl,
struct tcf_result res;
int result;
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
index d11b0b192e2c..123c0d199b0b 100644
--- a/net/sched/sch_sfq.c
+++ b/net/sched/sch_sfq.c
@@ -171,7 +171,7 @@ static unsigned int sfq_classify(struct sk_buff *skb, struct Qdisc *sch,
return sfq_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index b08e829cd195..0559f58c6cab 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -7516,11 +7516,7 @@ static int sctp_getsockopt_pr_streamstatus(struct sock *sk, int len,
/* Not allocated yet, means all stats are 0 */
params.sprstat_abandoned_unsent = 0;
params.sprstat_abandoned_sent = 0;
- retval = 0;
- goto out;
- }
-
- if (policy == SCTP_PR_SCTP_ALL) {
+ } else if (policy == SCTP_PR_SCTP_ALL) {
params.sprstat_abandoned_unsent = 0;
params.sprstat_abandoned_sent = 0;
for (policy = 0; policy <= SCTP_PR_INDEX(MAX); policy++) {
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index e23390d9326b..daf235b6a112 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -435,7 +435,7 @@ static struct sock *smc_sock_alloc(struct net *net, struct socket *sock,
return sk;
}
-int smc_bind(struct socket *sock, struct sockaddr *uaddr,
+int smc_bind(struct socket *sock, struct sockaddr_unsized *uaddr,
int addr_len)
{
struct sockaddr_in *addr = (struct sockaddr_in *)uaddr;
@@ -2788,6 +2788,7 @@ int smc_getname(struct socket *sock, struct sockaddr *addr,
int peer)
{
struct smc_sock *smc;
+ int rc = -EBADF;
if (peer && (sock->sk->sk_state != SMC_ACTIVE) &&
(sock->sk->sk_state != SMC_APPCLOSEWAIT1))
@@ -2795,7 +2796,11 @@ int smc_getname(struct socket *sock, struct sockaddr *addr,
smc = smc_sk(sock->sk);
- return smc->clcsock->ops->getname(smc->clcsock, addr, peer);
+ mutex_lock(&smc->clcsock_release_lock);
+ if (smc->clcsock)
+ rc = smc->clcsock->ops->getname(smc->clcsock, addr, peer);
+ mutex_unlock(&smc->clcsock_release_lock);
+ return rc;
}
int smc_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
@@ -3374,11 +3379,10 @@ int smc_create_clcsk(struct net *net, struct sock *sk, int family)
return 0;
}
-static int __smc_create(struct net *net, struct socket *sock, int protocol,
- int kern, struct socket *clcsock)
+static int smc_create(struct net *net, struct socket *sock, int protocol,
+ int kern)
{
int family = (protocol == SMCPROTO_SMC6) ? PF_INET6 : PF_INET;
- struct smc_sock *smc;
struct sock *sk;
int rc;
@@ -3397,15 +3401,7 @@ static int __smc_create(struct net *net, struct socket *sock, int protocol,
if (!sk)
goto out;
- /* create internal TCP socket for CLC handshake and fallback */
- smc = smc_sk(sk);
-
- rc = 0;
- if (clcsock)
- smc->clcsock = clcsock;
- else
- rc = smc_create_clcsk(net, sk, family);
-
+ rc = smc_create_clcsk(net, sk, family);
if (rc) {
sk_common_release(sk);
sock->sk = NULL;
@@ -3414,76 +3410,12 @@ static int __smc_create(struct net *net, struct socket *sock, int protocol,
return rc;
}
-static int smc_create(struct net *net, struct socket *sock, int protocol,
- int kern)
-{
- return __smc_create(net, sock, protocol, kern, NULL);
-}
-
static const struct net_proto_family smc_sock_family_ops = {
.family = PF_SMC,
.owner = THIS_MODULE,
.create = smc_create,
};
-static int smc_ulp_init(struct sock *sk)
-{
- struct socket *tcp = sk->sk_socket;
- struct net *net = sock_net(sk);
- struct socket *smcsock;
- int protocol, ret;
-
- /* only TCP can be replaced */
- if (tcp->type != SOCK_STREAM || sk->sk_protocol != IPPROTO_TCP ||
- (sk->sk_family != AF_INET && sk->sk_family != AF_INET6))
- return -ESOCKTNOSUPPORT;
- /* don't handle wq now */
- if (tcp->state != SS_UNCONNECTED || !tcp->file || tcp->wq.fasync_list)
- return -ENOTCONN;
-
- if (sk->sk_family == AF_INET)
- protocol = SMCPROTO_SMC;
- else
- protocol = SMCPROTO_SMC6;
-
- smcsock = sock_alloc();
- if (!smcsock)
- return -ENFILE;
-
- smcsock->type = SOCK_STREAM;
- __module_get(THIS_MODULE); /* tried in __tcp_ulp_find_autoload */
- ret = __smc_create(net, smcsock, protocol, 1, tcp);
- if (ret) {
- sock_release(smcsock); /* module_put() which ops won't be NULL */
- return ret;
- }
-
- /* replace tcp socket to smc */
- smcsock->file = tcp->file;
- smcsock->file->private_data = smcsock;
- smcsock->file->f_inode = SOCK_INODE(smcsock); /* replace inode when sock_close */
- smcsock->file->f_path.dentry->d_inode = SOCK_INODE(smcsock); /* dput() in __fput */
- tcp->file = NULL;
-
- return ret;
-}
-
-static void smc_ulp_clone(const struct request_sock *req, struct sock *newsk,
- const gfp_t priority)
-{
- struct inet_connection_sock *icsk = inet_csk(newsk);
-
- /* don't inherit ulp ops to child when listen */
- icsk->icsk_ulp_ops = NULL;
-}
-
-static struct tcp_ulp_ops smc_ulp_ops __read_mostly = {
- .name = "smc",
- .owner = THIS_MODULE,
- .init = smc_ulp_init,
- .clone = smc_ulp_clone,
-};
-
unsigned int smc_net_id;
static __net_init int smc_net_init(struct net *net)
@@ -3604,21 +3536,13 @@ static int __init smc_init(void)
pr_err("%s: ib_register fails with %d\n", __func__, rc);
goto out_sock;
}
-
- rc = tcp_register_ulp(&smc_ulp_ops);
- if (rc) {
- pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc);
- goto out_ib;
- }
rc = smc_inet_init();
if (rc) {
pr_err("%s: smc_inet_init fails with %d\n", __func__, rc);
- goto out_ulp;
+ goto out_ib;
}
static_branch_enable(&tcp_have_smc);
return 0;
-out_ulp:
- tcp_unregister_ulp(&smc_ulp_ops);
out_ib:
smc_ib_unregister_client();
out_sock:
@@ -3654,7 +3578,6 @@ static void __exit smc_exit(void)
{
static_branch_disable(&tcp_have_smc);
smc_inet_exit();
- tcp_unregister_ulp(&smc_ulp_ops);
sock_unregister(PF_SMC);
smc_core_exit();
smc_ib_unregister_client();
@@ -3679,7 +3602,6 @@ MODULE_AUTHOR("Ursula Braun <ubraun@linux.vnet.ibm.com>");
MODULE_DESCRIPTION("smc socket address family");
MODULE_LICENSE("GPL");
MODULE_ALIAS_NETPROTO(PF_SMC);
-MODULE_ALIAS_TCP_ULP("smc");
/* 256 for IPPROTO_SMC and 1 for SOCK_STREAM */
MODULE_ALIAS_NET_PF_PROTO_TYPE(PF_INET, 256, 1);
#if IS_ENABLED(CONFIG_IPV6)
diff --git a/net/smc/smc.h b/net/smc/smc.h
index 1425fd756d34..107d813fa996 100644
--- a/net/smc/smc.h
+++ b/net/smc/smc.h
@@ -42,7 +42,7 @@ void smc_unhash_sk(struct sock *sk);
void smc_release_cb(struct sock *sk);
int smc_release(struct socket *sock);
-int smc_bind(struct socket *sock, struct sockaddr *uaddr,
+int smc_bind(struct socket *sock, struct sockaddr_unsized *uaddr,
int addr_len);
int smc_connect(struct socket *sock, struct sockaddr *addr,
int alen, int flags);
diff --git a/net/socket.c b/net/socket.c
index 4ce6ddd768fb..7da40583ff29 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -1873,7 +1873,7 @@ int __sys_bind_socket(struct socket *sock, struct sockaddr_storage *address,
addrlen);
if (!err)
err = READ_ONCE(sock->ops)->bind(sock,
- (struct sockaddr *)address,
+ (struct sockaddr_unsized *)address,
addrlen);
return err;
}
@@ -3584,13 +3584,13 @@ static long compat_sock_ioctl(struct file *file, unsigned int cmd,
* Returns 0 or an error.
*/
-int kernel_bind(struct socket *sock, struct sockaddr *addr, int addrlen)
+int kernel_bind(struct socket *sock, struct sockaddr_unsized *addr, int addrlen)
{
struct sockaddr_storage address;
memcpy(&address, addr, addrlen);
- return READ_ONCE(sock->ops)->bind(sock, (struct sockaddr *)&address,
+ return READ_ONCE(sock->ops)->bind(sock, (struct sockaddr_unsized *)&address,
addrlen);
}
EXPORT_SYMBOL(kernel_bind);
diff --git a/net/strparser/strparser.c b/net/strparser/strparser.c
index e3c6abe58f44..7e9aa139cbde 100644
--- a/net/strparser/strparser.c
+++ b/net/strparser/strparser.c
@@ -505,12 +505,16 @@ void strp_unpause(struct strparser *strp)
EXPORT_SYMBOL_GPL(strp_unpause);
/* strp must already be stopped so that strp_recv will no longer be called.
- * Note that strp_done is not called with the lower socket held.
+ * Note that strp_done must not be called with the lower socket held.
*/
void strp_done(struct strparser *strp)
{
WARN_ON(!strp->stopped);
+ lock_sock(strp->sk);
+ /* sync with pending strp_recv */
+ release_sock(strp->sk);
+
cancel_delayed_work_sync(&strp->msg_timer_work);
cancel_work_sync(&strp->work);
diff --git a/net/sunrpc/Kconfig b/net/sunrpc/Kconfig
index a570e7adf270..381e76975ea9 100644
--- a/net/sunrpc/Kconfig
+++ b/net/sunrpc/Kconfig
@@ -21,6 +21,7 @@ config RPCSEC_GSS_KRB5
depends on SUNRPC && CRYPTO
default y
select SUNRPC_GSS
+ select CRYPTO_KRB5
select CRYPTO_SKCIPHER
select CRYPTO_HASH
help
diff --git a/net/sunrpc/auth_gss/gss_krb5_crypto.c b/net/sunrpc/auth_gss/gss_krb5_crypto.c
index 16dcf115de1e..3a8e6710a51b 100644
--- a/net/sunrpc/auth_gss/gss_krb5_crypto.c
+++ b/net/sunrpc/auth_gss/gss_krb5_crypto.c
@@ -953,3 +953,235 @@ krb5_etm_decrypt(struct krb5_ctx *kctx, u32 offset, u32 len,
ret = GSS_S_FAILURE;
return ret;
}
+
+/**
+ * gss_krb5_aead_encrypt - Encrypt a wrap token using crypto/krb5
+ * @kctx: Kerberos context
+ * @offset: byte offset of the GSS token header in @buf
+ * @buf: OUT: send buffer
+ * @pages: plaintext payload pages (page cache data)
+ *
+ * The xdr_buf setup mirrors the original per-enctype encrypt
+ * functions, but the CBC-CTS encryption and HMAC are replaced
+ * by a single AEAD operation through the crypto/krb5 library.
+ *
+ * Return values:
+ * %GSS_S_COMPLETE: Encryption successful
+ * %GSS_S_FAILURE: Encryption failed
+ */
+u32
+gss_krb5_aead_encrypt(struct krb5_ctx *kctx, u32 offset,
+ struct xdr_buf *buf, struct page **pages)
+{
+ const struct krb5_enctype *krb5 = kctx->krb5e;
+ struct crypto_aead *aead = kctx->initiate ?
+ kctx->initiator_enc_aead : kctx->acceptor_enc_aead;
+ unsigned int conflen = krb5->conf_len;
+ unsigned int cksum_len = krb5->cksum_len;
+ unsigned int sec_offset, sec_len, data_len;
+ struct scatterlist sg[XDR_BUF_TO_SG_NENTS];
+ struct scatterlist *sg_overflow = NULL;
+ ssize_t ret;
+ int nsg;
+
+ /* Insert space for the confounder */
+ if (xdr_extend_head(buf, offset + GSS_KRB5_TOK_HDR_LEN, conflen))
+ return GSS_S_FAILURE;
+
+ /* Ensure a tail segment exists */
+ if (buf->tail[0].iov_base == NULL) {
+ buf->tail[0].iov_base = buf->head[0].iov_base
+ + buf->head[0].iov_len;
+ buf->tail[0].iov_len = 0;
+ }
+
+ /* Append a copy of the plaintext GSS token header (RFC 4121 Sec 4.2.4) */
+ memcpy(buf->tail[0].iov_base + buf->tail[0].iov_len,
+ buf->head[0].iov_base + offset, GSS_KRB5_TOK_HDR_LEN);
+ buf->tail[0].iov_len += GSS_KRB5_TOK_HDR_LEN;
+ buf->len += GSS_KRB5_TOK_HDR_LEN;
+
+ /* Reserve space for the integrity checksum */
+ buf->tail[0].iov_len += cksum_len;
+ buf->len += cksum_len;
+
+ /*
+ * The AEAD operates in-place, but on the client send path the
+ * plaintext payload lives in page cache pages that must not be
+ * modified. Copy the payload into the scratch output pages
+ * first. On the server send path @pages and buf->pages are
+ * the same array, and no copy is needed.
+ *
+ * Both arrays share buf->page_base, so the same index and
+ * intra-page offset address corresponding data in each.
+ */
+ if (pages != buf->pages) {
+ unsigned int poff = buf->page_base;
+ unsigned int plen = buf->page_len;
+ unsigned int i = poff >> PAGE_SHIFT;
+ unsigned int off = offset_in_page(poff);
+
+ while (plen) {
+ unsigned int n = min_t(unsigned int, plen,
+ PAGE_SIZE - off);
+ memcpy_page(buf->pages[i], off, pages[i], off, n);
+ plen -= n;
+ i++;
+ off = 0;
+ }
+ }
+
+ /* Build scatterlist covering the secured region */
+ sec_offset = offset + GSS_KRB5_TOK_HDR_LEN;
+ sec_len = buf->len - sec_offset;
+ data_len = sec_len - conflen - cksum_len;
+
+ nsg = xdr_buf_to_sg_alloc(buf, sec_offset, sec_len,
+ sg, ARRAY_SIZE(sg),
+ &sg_overflow, GFP_NOFS);
+ if (nsg < 0)
+ return GSS_S_FAILURE;
+
+ ret = crypto_krb5_encrypt(krb5, aead, sg, nsg, sec_len,
+ conflen, data_len, false);
+ kfree(sg_overflow);
+ if (ret < 0)
+ return GSS_S_FAILURE;
+
+ return GSS_S_COMPLETE;
+}
+
+/**
+ * gss_krb5_aead_decrypt - Decrypt a wrap token using crypto/krb5
+ * @kctx: Kerberos context
+ * @offset: byte offset of the GSS token header in @buf
+ * @len: total length of the GSS token
+ * @buf: ciphertext buffer, decrypted in-place
+ * @headskip: OUT: confounder length, in octets
+ * @tailskip: OUT: checksum length, in octets
+ *
+ * Return values:
+ * %GSS_S_COMPLETE: Decryption and integrity verification succeeded
+ * %GSS_S_BAD_SIG: Integrity checksum did not match
+ * %GSS_S_DEFECTIVE_TOKEN: Token is malformed or truncated
+ * %GSS_S_FAILURE: Decryption failed
+ */
+u32
+gss_krb5_aead_decrypt(struct krb5_ctx *kctx, u32 offset, u32 len,
+ struct xdr_buf *buf, u32 *headskip, u32 *tailskip)
+{
+ const struct krb5_enctype *krb5 = kctx->krb5e;
+ struct crypto_aead *aead = kctx->initiate ?
+ kctx->acceptor_enc_aead : kctx->initiator_enc_aead;
+ unsigned int sec_offset, sec_len;
+ size_t data_offset, data_len;
+ struct scatterlist sg[XDR_BUF_TO_SG_NENTS];
+ struct scatterlist *sg_overflow = NULL;
+ int nsg, ret;
+
+ /* Secured region starts after the GSS token header */
+ sec_offset = offset + GSS_KRB5_TOK_HDR_LEN;
+ if (len < sec_offset)
+ return GSS_S_DEFECTIVE_TOKEN;
+ sec_len = len - sec_offset;
+
+ nsg = xdr_buf_to_sg_alloc(buf, sec_offset, sec_len,
+ sg, ARRAY_SIZE(sg),
+ &sg_overflow, GFP_NOFS);
+ if (nsg < 0)
+ return GSS_S_FAILURE;
+
+ data_offset = 0;
+ data_len = sec_len;
+ ret = crypto_krb5_decrypt(krb5, aead, sg, nsg,
+ &data_offset, &data_len);
+ kfree(sg_overflow);
+ if (ret < 0)
+ return gss_krb5_errno_to_status(ret);
+
+ *headskip = data_offset;
+ *tailskip = sec_len - data_offset - data_len;
+ return GSS_S_COMPLETE;
+}
+
+/**
+ * gss_krb5_mic_build_sg - Build scatterlist for MIC token operations
+ * @body: xdr_buf containing the message body
+ * @cksum: pointer to checksum area in the token buffer
+ * @cksum_len: length of checksum area
+ * @hdr: pointer to GSS token header
+ * @sg_head: caller-provided scatterlist array; if more than
+ * XDR_BUF_TO_SG_NENTS entries are needed, an overflow
+ * scatterlist is allocated and chained automatically
+ * @sg_overflow: OUT: overflow scatterlist, caller must kfree
+ *
+ * Per RFC 4121 Section 4.2.4, MIC token checksums cover the
+ * message body followed by the token header. The checksum
+ * output or received checksum occupies the first scatterlist
+ * entry. This layout cannot be constructed by
+ * xdr_buf_to_sg_alloc() because the checksum area and the GSS
+ * header lie outside the xdr_buf.
+ *
+ * Returns the number of scatterlist entries on success, or a
+ * negative errno on failure.
+ */
+int gss_krb5_mic_build_sg(const struct xdr_buf *body,
+ void *cksum, unsigned int cksum_len,
+ void *hdr,
+ struct scatterlist *sg_head,
+ struct scatterlist **sg_overflow)
+{
+ struct scatterlist *entry;
+ int body_max, body_nsg, nsg;
+
+ *sg_overflow = NULL;
+
+ body_max = 2;
+ if (body->page_len)
+ body_max += DIV_ROUND_UP(body->page_len +
+ offset_in_page(body->page_base),
+ PAGE_SIZE);
+ nsg = 1 + body_max + 1;
+ if (nsg <= XDR_BUF_TO_SG_NENTS) {
+ sg_init_table(sg_head, nsg);
+ } else {
+ unsigned int overflow_nents =
+ nsg - XDR_BUF_TO_SG_NENTS + 1;
+
+ *sg_overflow = kmalloc_array(overflow_nents,
+ sizeof(**sg_overflow),
+ GFP_NOFS);
+ if (!*sg_overflow)
+ return -ENOMEM;
+
+ sg_init_table(sg_head, XDR_BUF_TO_SG_NENTS);
+ sg_init_table(*sg_overflow, overflow_nents);
+ sg_chain(sg_head, XDR_BUF_TO_SG_NENTS, *sg_overflow);
+ }
+
+ sg_set_buf(&sg_head[0], cksum, cksum_len);
+ body_nsg = xdr_buf_to_sg(body, 0, body->len,
+ sg_next(&sg_head[0]), body_max);
+ if (body_nsg < 0)
+ goto out_err;
+
+ /*
+ * xdr_buf_to_sg marks the last body entry as end-of-list;
+ * clear it so the trailing header entry is reachable.
+ */
+ if (body_nsg > 0) {
+ entry = sg_last(sg_next(&sg_head[0]), body_nsg);
+ sg_unmark_end(entry);
+ entry = sg_next(entry);
+ } else {
+ entry = sg_next(&sg_head[0]);
+ }
+ sg_set_buf(entry, hdr, GSS_KRB5_TOK_HDR_LEN);
+ sg_mark_end(entry);
+ return 1 + body_nsg + 1;
+
+out_err:
+ kfree(*sg_overflow);
+ *sg_overflow = NULL;
+ return body_nsg;
+}
diff --git a/net/sunrpc/auth_gss/gss_krb5_internal.h b/net/sunrpc/auth_gss/gss_krb5_internal.h
index 8769e9e705bf..83e969494b54 100644
--- a/net/sunrpc/auth_gss/gss_krb5_internal.h
+++ b/net/sunrpc/auth_gss/gss_krb5_internal.h
@@ -8,6 +8,8 @@
#ifndef _NET_SUNRPC_AUTH_GSS_KRB5_INTERNAL_H
#define _NET_SUNRPC_AUTH_GSS_KRB5_INTERNAL_H
+#include <crypto/krb5.h>
+
/*
* The RFCs often specify payload lengths in bits. This helper
* converts a specified bit-length to the number of octets/bytes.
@@ -62,6 +64,11 @@ struct krb5_ctx {
u32 enctype;
u32 flags;
const struct gss_krb5_enctype *gk5e; /* enctype-specific info */
+ const struct krb5_enctype *krb5e; /* crypto/krb5 enctype */
+ struct crypto_aead *initiator_enc_aead;
+ struct crypto_aead *acceptor_enc_aead;
+ struct crypto_shash *initiator_sign_shash;
+ struct crypto_shash *acceptor_sign_shash;
struct crypto_sync_skcipher *enc;
struct crypto_sync_skcipher *seq;
struct crypto_sync_skcipher *acceptor_enc;
@@ -177,6 +184,19 @@ u32 krb5_etm_encrypt(struct krb5_ctx *kctx, u32 offset, struct xdr_buf *buf,
u32 krb5_etm_decrypt(struct krb5_ctx *kctx, u32 offset, u32 len,
struct xdr_buf *buf, u32 *headskip, u32 *tailskip);
+u32 gss_krb5_errno_to_status(int err);
+
+int gss_krb5_mic_build_sg(const struct xdr_buf *body,
+ void *cksum, unsigned int cksum_len,
+ void *hdr,
+ struct scatterlist *sg_head,
+ struct scatterlist **sg_overflow);
+
+u32 gss_krb5_aead_encrypt(struct krb5_ctx *kctx, u32 offset,
+ struct xdr_buf *buf, struct page **pages);
+u32 gss_krb5_aead_decrypt(struct krb5_ctx *kctx, u32 offset, u32 len,
+ struct xdr_buf *buf, u32 *headskip, u32 *tailskip);
+
#if IS_ENABLED(CONFIG_KUNIT)
void krb5_nfold(u32 inbits, const u8 *in, u32 outbits, u8 *out);
const struct gss_krb5_enctype *gss_krb5_lookup_enctype(u32 etype);
diff --git a/net/sunrpc/auth_gss/gss_krb5_mech.c b/net/sunrpc/auth_gss/gss_krb5_mech.c
index 3366505bc669..8e61be0e8ec5 100644
--- a/net/sunrpc/auth_gss/gss_krb5_mech.c
+++ b/net/sunrpc/auth_gss/gss_krb5_mech.c
@@ -43,8 +43,8 @@ static const struct gss_krb5_enctype supported_gss_krb5_enctypes[] = {
.aux_cipher = "cbc(aes)",
.cksum_name = "hmac(sha1)",
.derive_key = krb5_derive_key_v2,
- .encrypt = gss_krb5_aes_encrypt,
- .decrypt = gss_krb5_aes_decrypt,
+ .encrypt = gss_krb5_aead_encrypt,
+ .decrypt = gss_krb5_aead_decrypt,
.get_mic = gss_krb5_get_mic_v2,
.verify_mic = gss_krb5_verify_mic_v2,
@@ -72,8 +72,8 @@ static const struct gss_krb5_enctype supported_gss_krb5_enctypes[] = {
.aux_cipher = "cbc(aes)",
.cksum_name = "hmac(sha1)",
.derive_key = krb5_derive_key_v2,
- .encrypt = gss_krb5_aes_encrypt,
- .decrypt = gss_krb5_aes_decrypt,
+ .encrypt = gss_krb5_aead_encrypt,
+ .decrypt = gss_krb5_aead_decrypt,
.get_mic = gss_krb5_get_mic_v2,
.verify_mic = gss_krb5_verify_mic_v2,
@@ -111,7 +111,7 @@ static const struct gss_krb5_enctype supported_gss_krb5_enctypes[] = {
.Ki_length = BITS2OCTETS(128),
.derive_key = krb5_kdf_feedback_cmac,
- .encrypt = gss_krb5_aes_encrypt,
+ .encrypt = gss_krb5_aead_encrypt,
.decrypt = gss_krb5_aes_decrypt,
.get_mic = gss_krb5_get_mic_v2,
@@ -137,7 +137,7 @@ static const struct gss_krb5_enctype supported_gss_krb5_enctypes[] = {
.Ki_length = BITS2OCTETS(256),
.derive_key = krb5_kdf_feedback_cmac,
- .encrypt = gss_krb5_aes_encrypt,
+ .encrypt = gss_krb5_aead_encrypt,
.decrypt = gss_krb5_aes_decrypt,
.get_mic = gss_krb5_get_mic_v2,
@@ -166,8 +166,8 @@ static const struct gss_krb5_enctype supported_gss_krb5_enctypes[] = {
.Ki_length = BITS2OCTETS(128),
.derive_key = krb5_kdf_hmac_sha2,
- .encrypt = krb5_etm_encrypt,
- .decrypt = krb5_etm_decrypt,
+ .encrypt = gss_krb5_aead_encrypt,
+ .decrypt = gss_krb5_aead_decrypt,
.get_mic = gss_krb5_get_mic_v2,
.verify_mic = gss_krb5_verify_mic_v2,
@@ -192,8 +192,8 @@ static const struct gss_krb5_enctype supported_gss_krb5_enctypes[] = {
.Ki_length = BITS2OCTETS(192),
.derive_key = krb5_kdf_hmac_sha2,
- .encrypt = krb5_etm_encrypt,
- .decrypt = krb5_etm_decrypt,
+ .encrypt = gss_krb5_aead_encrypt,
+ .decrypt = gss_krb5_aead_decrypt,
.get_mic = gss_krb5_get_mic_v2,
.verify_mic = gss_krb5_verify_mic_v2,
@@ -300,6 +300,10 @@ gss_krb5_import_ctx_v2(struct krb5_ctx *ctx, gfp_t gfp_mask)
.len = ctx->gk5e->keylength,
.data = ctx->Ksess,
};
+ struct krb5_buffer TK = {
+ .len = ctx->gk5e->keylength,
+ .data = ctx->Ksess,
+ };
struct xdr_netobj keyout;
int ret = -EINVAL;
@@ -374,12 +378,49 @@ gss_krb5_import_ctx_v2(struct krb5_ctx *ctx, gfp_t gfp_mask)
if (ctx->acceptor_integ == NULL)
goto out_free;
+ ctx->initiator_enc_aead =
+ crypto_krb5_prepare_encryption(ctx->krb5e, &TK,
+ KG_USAGE_INITIATOR_SEAL,
+ gfp_mask);
+ if (IS_ERR(ctx->initiator_enc_aead)) {
+ ret = PTR_ERR(ctx->initiator_enc_aead);
+ goto out_free;
+ }
+ ctx->acceptor_enc_aead =
+ crypto_krb5_prepare_encryption(ctx->krb5e, &TK,
+ KG_USAGE_ACCEPTOR_SEAL,
+ gfp_mask);
+ if (IS_ERR(ctx->acceptor_enc_aead)) {
+ ret = PTR_ERR(ctx->acceptor_enc_aead);
+ goto out_free;
+ }
+ ctx->initiator_sign_shash =
+ crypto_krb5_prepare_checksum(ctx->krb5e, &TK,
+ KG_USAGE_INITIATOR_SIGN,
+ gfp_mask);
+ if (IS_ERR(ctx->initiator_sign_shash)) {
+ ret = PTR_ERR(ctx->initiator_sign_shash);
+ goto out_free;
+ }
+ ctx->acceptor_sign_shash =
+ crypto_krb5_prepare_checksum(ctx->krb5e, &TK,
+ KG_USAGE_ACCEPTOR_SIGN,
+ gfp_mask);
+ if (IS_ERR(ctx->acceptor_sign_shash)) {
+ ret = PTR_ERR(ctx->acceptor_sign_shash);
+ goto out_free;
+ }
+
ret = 0;
out:
kfree_sensitive(keyout.data);
return ret;
out_free:
+ crypto_free_shash(ctx->acceptor_sign_shash);
+ crypto_free_shash(ctx->initiator_sign_shash);
+ crypto_free_aead(ctx->acceptor_enc_aead);
+ crypto_free_aead(ctx->initiator_enc_aead);
crypto_free_ahash(ctx->acceptor_integ);
crypto_free_ahash(ctx->initiator_integ);
crypto_free_ahash(ctx->acceptor_sign);
@@ -432,6 +473,13 @@ gss_import_v2_context(const void *p, const void *end, struct krb5_ctx *ctx,
p = ERR_PTR(-EINVAL);
goto out_err;
}
+ ctx->krb5e = crypto_krb5_find_enctype(ctx->enctype);
+ if (!ctx->krb5e) {
+ dprintk("gss_kerberos_mech: crypto/krb5 missing enctype %u\n",
+ ctx->enctype);
+ p = ERR_PTR(-EINVAL);
+ goto out_err;
+ }
keylen = ctx->gk5e->keylength;
p = simple_get_bytes(p, end, ctx->Ksess, keylen);
@@ -495,6 +543,10 @@ gss_krb5_delete_sec_context(void *internal_ctx)
{
struct krb5_ctx *kctx = internal_ctx;
+ crypto_free_shash(kctx->acceptor_sign_shash);
+ crypto_free_shash(kctx->initiator_sign_shash);
+ crypto_free_aead(kctx->acceptor_enc_aead);
+ crypto_free_aead(kctx->initiator_enc_aead);
crypto_free_sync_skcipher(kctx->seq);
crypto_free_sync_skcipher(kctx->enc);
crypto_free_sync_skcipher(kctx->acceptor_enc);
@@ -509,6 +561,30 @@ gss_krb5_delete_sec_context(void *internal_ctx)
kfree(kctx);
}
+/**
+ * gss_krb5_errno_to_status - Map a negative errno to a GSS major status
+ * @err: negative errno value, or zero
+ *
+ * Returns:
+ * %GSS_S_COMPLETE if @err is zero
+ * %GSS_S_BAD_SIG if @err is -EBADMSG (integrity check failure)
+ * %GSS_S_DEFECTIVE_TOKEN if @err is -EPROTO (malformed token)
+ * %GSS_S_FAILURE for all other negative values
+ */
+u32 gss_krb5_errno_to_status(int err)
+{
+ switch (err) {
+ case 0:
+ return GSS_S_COMPLETE;
+ case -EBADMSG:
+ return GSS_S_BAD_SIG;
+ case -EPROTO:
+ return GSS_S_DEFECTIVE_TOKEN;
+ default:
+ return GSS_S_FAILURE;
+ }
+}
+
/**
* gss_krb5_get_mic - get_mic for the Kerberos GSS mechanism
* @gctx: GSS context
diff --git a/net/sunrpc/auth_gss/gss_krb5_seal.c b/net/sunrpc/auth_gss/gss_krb5_seal.c
index ce540df9bce4..66c179337029 100644
--- a/net/sunrpc/auth_gss/gss_krb5_seal.c
+++ b/net/sunrpc/auth_gss/gss_krb5_seal.c
@@ -64,6 +64,8 @@
#include <linux/random.h>
#include <linux/crypto.h>
#include <linux/atomic.h>
+#include <linux/scatterlist.h>
+#include <linux/slab.h>
#include "gss_krb5_internal.h"
@@ -78,10 +80,10 @@ setup_token_v2(struct krb5_ctx *ctx, struct xdr_netobj *token)
void *krb5_hdr;
u8 *p, flags = 0x00;
- if ((ctx->flags & KRB5_CTX_FLAG_INITIATOR) == 0)
- flags |= 0x01;
+ if (!ctx->initiate)
+ flags |= KG2_TOKEN_FLAG_SENTBYACCEPTOR;
if (ctx->flags & KRB5_CTX_FLAG_ACCEPTOR_SUBKEY)
- flags |= 0x04;
+ flags |= KG2_TOKEN_FLAG_ACCEPTORSUBKEY;
/* Per rfc 4121, sec 4.2.6.1, there is no header,
* just start the token.
@@ -97,7 +99,7 @@ setup_token_v2(struct krb5_ctx *ctx, struct xdr_netobj *token)
*ptr++ = 0xffff;
*ptr = 0xffff;
- token->len = GSS_KRB5_TOK_HDR_LEN + ctx->gk5e->cksumlength;
+ token->len = GSS_KRB5_TOK_HDR_LEN + ctx->krb5e->cksum_len;
return krb5_hdr;
}
@@ -105,14 +107,17 @@ u32
gss_krb5_get_mic_v2(struct krb5_ctx *ctx, struct xdr_buf *text,
struct xdr_netobj *token)
{
- struct crypto_ahash *tfm = ctx->initiate ?
- ctx->initiator_sign : ctx->acceptor_sign;
- struct xdr_netobj cksumobj = {
- .len = ctx->gk5e->cksumlength,
- };
+ const struct krb5_enctype *krb5 = ctx->krb5e;
+ struct crypto_shash *shash = ctx->initiate ?
+ ctx->initiator_sign_shash : ctx->acceptor_sign_shash;
+ unsigned int cksum_len = krb5->cksum_len;
+ struct scatterlist sg_head[XDR_BUF_TO_SG_NENTS];
+ struct scatterlist *sg_overflow;
__be64 seq_send_be64;
void *krb5_hdr;
time64_t now;
+ ssize_t ret;
+ int nsg;
dprintk("RPC: %s\n", __func__);
@@ -123,9 +128,25 @@ gss_krb5_get_mic_v2(struct krb5_ctx *ctx, struct xdr_buf *text,
seq_send_be64 = cpu_to_be64(atomic64_fetch_inc(&ctx->seq_send64));
memcpy(krb5_hdr + 8, (char *) &seq_send_be64, 8);
- cksumobj.data = krb5_hdr + GSS_KRB5_TOK_HDR_LEN;
- if (gss_krb5_checksum(tfm, krb5_hdr, GSS_KRB5_TOK_HDR_LEN,
- text, 0, &cksumobj))
+ /*
+ * The checksum is written directly into the token buffer.
+ * This is safe: crypto_krb5_get_mic uses shash (software
+ * hash), so the scatterlist is never DMA-mapped.
+ */
+ nsg = gss_krb5_mic_build_sg(text,
+ krb5_hdr + GSS_KRB5_TOK_HDR_LEN,
+ cksum_len, krb5_hdr,
+ sg_head, &sg_overflow);
+ if (nsg < 0)
+ return GSS_S_FAILURE;
+
+ ret = crypto_krb5_get_mic(krb5, shash, NULL, sg_head, nsg,
+ cksum_len + text->len +
+ GSS_KRB5_TOK_HDR_LEN,
+ cksum_len,
+ text->len + GSS_KRB5_TOK_HDR_LEN);
+ kfree(sg_overflow);
+ if (ret < 0)
return GSS_S_FAILURE;
now = ktime_get_real_seconds();
diff --git a/net/sunrpc/auth_gss/gss_krb5_unseal.c b/net/sunrpc/auth_gss/gss_krb5_unseal.c
index ef0e6af9fc95..4d12d49434c2 100644
--- a/net/sunrpc/auth_gss/gss_krb5_unseal.c
+++ b/net/sunrpc/auth_gss/gss_krb5_unseal.c
@@ -60,6 +60,8 @@
#include <linux/types.h>
#include <linux/jiffies.h>
#include <linux/sunrpc/gss_krb5.h>
+#include <linux/scatterlist.h>
+#include <linux/slab.h>
#include "gss_krb5_internal.h"
@@ -71,21 +73,25 @@ u32
gss_krb5_verify_mic_v2(struct krb5_ctx *ctx, struct xdr_buf *message_buffer,
struct xdr_netobj *read_token)
{
- struct crypto_ahash *tfm = ctx->initiate ?
- ctx->acceptor_sign : ctx->initiator_sign;
- char cksumdata[GSS_KRB5_MAX_CKSUM_LEN];
- struct xdr_netobj cksumobj = {
- .len = ctx->gk5e->cksumlength,
- .data = cksumdata,
- };
+ const struct krb5_enctype *krb5 = ctx->krb5e;
+ struct crypto_shash *shash = ctx->initiate ?
+ ctx->acceptor_sign_shash : ctx->initiator_sign_shash;
+ unsigned int cksum_len = krb5->cksum_len;
+ struct scatterlist sg_head[XDR_BUF_TO_SG_NENTS];
+ struct scatterlist *sg_overflow;
+ size_t mic_offset, mic_len;
u8 *ptr = read_token->data;
__be16 be16_ptr;
time64_t now;
u8 flags;
- int i;
+ int nsg, i;
+ int ret;
dprintk("RPC: %s\n", __func__);
+ if (read_token->len < GSS_KRB5_TOK_HDR_LEN + cksum_len)
+ return GSS_S_DEFECTIVE_TOKEN;
+
memcpy(&be16_ptr, (char *) ptr, 2);
if (be16_to_cpu(be16_ptr) != KG2_TOK_MIC)
return GSS_S_DEFECTIVE_TOKEN;
@@ -104,13 +110,20 @@ gss_krb5_verify_mic_v2(struct krb5_ctx *ctx, struct xdr_buf *message_buffer,
if (ptr[i] != 0xff)
return GSS_S_DEFECTIVE_TOKEN;
- if (gss_krb5_checksum(tfm, ptr, GSS_KRB5_TOK_HDR_LEN,
- message_buffer, 0, &cksumobj))
+ nsg = gss_krb5_mic_build_sg(message_buffer,
+ ptr + GSS_KRB5_TOK_HDR_LEN,
+ cksum_len, ptr,
+ sg_head, &sg_overflow);
+ if (nsg < 0)
return GSS_S_FAILURE;
- if (memcmp(cksumobj.data, ptr + GSS_KRB5_TOK_HDR_LEN,
- ctx->gk5e->cksumlength))
- return GSS_S_BAD_SIG;
+ mic_offset = 0;
+ mic_len = cksum_len + message_buffer->len + GSS_KRB5_TOK_HDR_LEN;
+ ret = crypto_krb5_verify_mic(krb5, shash, NULL, sg_head, nsg,
+ &mic_offset, &mic_len);
+ kfree(sg_overflow);
+ if (ret)
+ return gss_krb5_errno_to_status(ret);
/* it got through unscathed. Make sure the context is unexpired */
now = ktime_get_real_seconds();
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 07de0efaf37d..c6cfa04dc85f 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1484,12 +1484,12 @@ static int rpc_sockname(struct net *net, struct sockaddr *sap, size_t salen,
switch (sap->sa_family) {
case AF_INET:
err = kernel_bind(sock,
- (struct sockaddr *)&rpc_inaddr_loopback,
+ (struct sockaddr_unsized *)&rpc_inaddr_loopback,
sizeof(rpc_inaddr_loopback));
break;
case AF_INET6:
err = kernel_bind(sock,
- (struct sockaddr *)&rpc_in6addr_loopback,
+ (struct sockaddr_unsized *)&rpc_in6addr_loopback,
sizeof(rpc_in6addr_loopback));
break;
default:
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 0ed8d5637471..0ebd038095ed 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1617,7 +1617,7 @@ static struct svc_xprt *svc_create_socket(struct svc_serv *serv,
ip6_sock_set_v6only(sock->sk);
if (type == SOCK_STREAM)
sock->sk->sk_reuse = SK_CAN_REUSE; /* allow address reuse */
- error = kernel_bind(sock, sin, len);
+ error = kernel_bind(sock, (struct sockaddr_unsized *)sin, len);
if (error < 0)
goto bummer;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
index 1075bfd9fed6..a7c09041cc9b 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
@@ -376,7 +376,7 @@ static void svc_rdma_wc_receive(struct ib_cq *cq, struct ib_wc *wc)
trace_svcrdma_wc_recv_err(wc, &ctxt->rc_cid);
dropped:
svc_rdma_recv_ctxt_put(rdma, ctxt);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -1018,7 +1018,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)
if (ret == -EINVAL)
svc_rdma_send_error(rdma_xprt, ctxt, ret);
svc_rdma_recv_ctxt_put(rdma_xprt, ctxt);
- svc_xprt_deferred_close(xprt);
+ svc_rdma_xprt_deferred_close(rdma_xprt);
return ret;
}
return 0;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_rw.c b/net/sunrpc/xprtrdma/svc_rdma_rw.c
index 626e79a31767..371ec8278d51 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_rw.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_rw.c
@@ -230,6 +230,28 @@ static void svc_rdma_write_info_free(struct svc_rdma_write_info *info)
queue_work(svcrdma_wq, &info->wi_work);
}
+/**
+ * svc_rdma_write_chunk_release - Release Write chunk I/O resources
+ * @rdma: controlling transport
+ * @ctxt: Send context that is being released
+ *
+ * Write chunk resources remain live until Send completion because
+ * Write WRs are chained to the Send WR. This function releases all
+ * write_info structures accumulated on @ctxt->sc_write_info_list.
+ */
+void svc_rdma_write_chunk_release(struct svcxprt_rdma *rdma,
+ struct svc_rdma_send_ctxt *ctxt)
+{
+ struct svc_rdma_write_info *info;
+
+ while (!list_empty(&ctxt->sc_write_info_list)) {
+ info = list_first_entry(&ctxt->sc_write_info_list,
+ struct svc_rdma_write_info, wi_list);
+ list_del(&info->wi_list);
+ svc_rdma_write_info_free(info);
+ }
+}
+
/**
* svc_rdma_reply_chunk_release - Release Reply chunk I/O resources
* @rdma: controlling transport
@@ -270,7 +292,7 @@ static void svc_rdma_reply_done(struct ib_cq *cq, struct ib_wc *wc)
trace_svcrdma_wc_reply_err(wc, &cc->cc_cid);
}
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -286,13 +308,11 @@ static void svc_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc)
struct ib_cqe *cqe = wc->wr_cqe;
struct svc_rdma_chunk_ctxt *cc =
container_of(cqe, struct svc_rdma_chunk_ctxt, cc_cqe);
- struct svc_rdma_write_info *info =
- container_of(cc, struct svc_rdma_write_info, wi_cc);
switch (wc->status) {
case IB_WC_SUCCESS:
trace_svcrdma_wc_write(&cc->cc_cid);
- break;
+ return;
case IB_WC_WR_FLUSH_ERR:
trace_svcrdma_wc_write_flush(wc, &cc->cc_cid);
break;
@@ -300,12 +320,11 @@ static void svc_rdma_write_done(struct ib_cq *cq, struct ib_wc *wc)
trace_svcrdma_wc_write_err(wc, &cc->cc_cid);
}
- svc_rdma_wake_send_waiters(rdma, cc->cc_sqecount);
-
- if (unlikely(wc->status != IB_WC_SUCCESS))
- svc_xprt_deferred_close(&rdma->sc_xprt);
-
- svc_rdma_write_info_free(info);
+ /* The RDMA Write has flushed, so the client won't get
+ * some of the outgoing RPC message. Signal the loss
+ * to the client by closing the connection.
+ */
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -350,7 +369,7 @@ static void svc_rdma_wc_read_done(struct ib_cq *cq, struct ib_wc *wc)
*/
svc_rdma_cc_release(rdma, cc, DMA_FROM_DEVICE);
svc_rdma_recv_ctxt_put(rdma, ctxt);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/*
@@ -403,7 +422,7 @@ static int svc_rdma_post_chunk_ctxt(struct svcxprt_rdma *rdma,
} while (1);
trace_svcrdma_sq_post_err(rdma, &cc->cc_cid, ret);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
/* If even one was posted, there will be a completion. */
if (bad_wr != first_wr)
@@ -601,13 +620,27 @@ static int svc_rdma_xb_write(const struct xdr_buf *xdr, void *data)
return xdr->len;
}
-static int svc_rdma_send_write_chunk(struct svcxprt_rdma *rdma,
- const struct svc_rdma_chunk *chunk,
- const struct xdr_buf *xdr)
+/*
+ * svc_rdma_prepare_write_chunk - Link Write WRs for @chunk onto @sctxt's chain
+ *
+ * Write WRs are prepended to the Send WR chain so that a single
+ * ib_post_send() posts both RDMA Writes and the final Send. Only
+ * the first WR in each chunk gets a CQE for error detection;
+ * subsequent WRs complete without individual completion events.
+ * The Send WR's signaled completion indicates all chained
+ * operations have finished.
+ */
+static int svc_rdma_prepare_write_chunk(struct svcxprt_rdma *rdma,
+ struct svc_rdma_send_ctxt *sctxt,
+ const struct svc_rdma_chunk *chunk,
+ const struct xdr_buf *xdr)
{
struct svc_rdma_write_info *info;
struct svc_rdma_chunk_ctxt *cc;
+ struct ib_send_wr *first_wr;
struct xdr_buf payload;
+ struct list_head *pos;
+ struct ib_cqe *cqe;
int ret;
if (xdr_buf_subsegment(xdr, &payload, chunk->ch_position,
@@ -623,10 +656,25 @@ static int svc_rdma_send_write_chunk(struct svcxprt_rdma *rdma,
if (ret != payload.len)
goto out_err;
- trace_svcrdma_post_write_chunk(&cc->cc_cid, cc->cc_sqecount);
- ret = svc_rdma_post_chunk_ctxt(rdma, cc);
- if (ret < 0)
+ ret = -EINVAL;
+ if (unlikely(sctxt->sc_sqecount + cc->cc_sqecount > rdma->sc_sq_depth))
goto out_err;
+
+ first_wr = sctxt->sc_wr_chain;
+ cqe = &cc->cc_cqe;
+ list_for_each(pos, &cc->cc_rwctxts) {
+ struct svc_rdma_rw_ctxt *rwc;
+
+ rwc = list_entry(pos, struct svc_rdma_rw_ctxt, rw_list);
+ first_wr = rdma_rw_ctx_wrs(&rwc->rw_ctx, rdma->sc_qp,
+ rdma->sc_port_num, cqe, first_wr);
+ cqe = NULL;
+ }
+ sctxt->sc_wr_chain = first_wr;
+ sctxt->sc_sqecount += cc->cc_sqecount;
+ list_add(&info->wi_list, &sctxt->sc_write_info_list);
+
+ trace_svcrdma_post_write_chunk(&cc->cc_cid, cc->cc_sqecount);
return 0;
out_err:
@@ -635,17 +683,19 @@ static int svc_rdma_send_write_chunk(struct svcxprt_rdma *rdma,
}
/**
- * svc_rdma_send_write_list - Send all chunks on the Write list
+ * svc_rdma_prepare_write_list - Construct WR chain for sending Write list
* @rdma: controlling RDMA transport
* @rctxt: Write list provisioned by the client
+ * @sctxt: Send WR resources
* @xdr: xdr_buf containing an RPC Reply message
*
- * Returns zero on success, or a negative errno if one or more
- * Write chunks could not be sent.
+ * Returns zero on success, or a negative errno if WR chain
+ * construction fails for one or more Write chunks.
*/
-int svc_rdma_send_write_list(struct svcxprt_rdma *rdma,
- const struct svc_rdma_recv_ctxt *rctxt,
- const struct xdr_buf *xdr)
+int svc_rdma_prepare_write_list(struct svcxprt_rdma *rdma,
+ const struct svc_rdma_recv_ctxt *rctxt,
+ struct svc_rdma_send_ctxt *sctxt,
+ const struct xdr_buf *xdr)
{
struct svc_rdma_chunk *chunk;
int ret;
@@ -653,7 +703,7 @@ int svc_rdma_send_write_list(struct svcxprt_rdma *rdma,
pcl_for_each_chunk(chunk, &rctxt->rc_write_pcl) {
if (!chunk->ch_payload_length)
break;
- ret = svc_rdma_send_write_chunk(rdma, chunk, xdr);
+ ret = svc_rdma_prepare_write_chunk(rdma, sctxt, chunk, xdr);
if (ret < 0)
return ret;
}
diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
index 082db5cea213..4c376b0b1931 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
@@ -149,6 +149,7 @@ svc_rdma_send_ctxt_alloc(struct svcxprt_rdma *rdma)
ctxt->sc_send_wr.sg_list = ctxt->sc_sges;
ctxt->sc_send_wr.send_flags = IB_SEND_SIGNALED;
ctxt->sc_cqe.done = svc_rdma_wc_send;
+ INIT_LIST_HEAD(&ctxt->sc_write_info_list);
ctxt->sc_xprt_buf = buffer;
xdr_buf_init(&ctxt->sc_hdrbuf, ctxt->sc_xprt_buf,
rdma->sc_max_req_size);
@@ -237,6 +238,7 @@ static void svc_rdma_send_ctxt_release(struct svcxprt_rdma *rdma,
struct ib_device *device = rdma->sc_cm_id->device;
unsigned int i;
+ svc_rdma_write_chunk_release(rdma, ctxt);
svc_rdma_reply_chunk_release(rdma, ctxt);
if (ctxt->sc_page_count)
@@ -324,7 +326,7 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)
else
trace_svcrdma_wc_send_flush(wc, &ctxt->sc_cid);
svc_rdma_send_ctxt_put(rdma, ctxt);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
}
/**
@@ -385,7 +387,7 @@ int svc_rdma_post_send(struct svcxprt_rdma *rdma,
ret = ib_post_send(rdma->sc_qp, first_wr, &bad_wr);
if (ret) {
trace_svcrdma_sq_post_err(rdma, &cid, ret);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
/* If even one WR was posted, there will be a
* Send completion that bumps sc_sq_avail.
@@ -993,6 +995,12 @@ void svc_rdma_send_error_msg(struct svcxprt_rdma *rdma,
sctxt->sc_send_wr.num_sge = 1;
sctxt->sc_send_wr.opcode = IB_WR_SEND;
sctxt->sc_sges[0].length = sctxt->sc_hdrbuf.len;
+
+ /* Ensure only the error message is posted, not any previously
+ * prepared Write chunk WRs.
+ */
+ sctxt->sc_wr_chain = &sctxt->sc_send_wr;
+ sctxt->sc_sqecount = 1;
if (svc_rdma_post_send(rdma, sctxt))
goto put_ctxt;
return;
@@ -1040,7 +1048,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
if (!p)
goto put_ctxt;
- ret = svc_rdma_send_write_list(rdma, rctxt, &rqstp->rq_res);
+ ret = svc_rdma_prepare_write_list(rdma, rctxt, sctxt, &rqstp->rq_res);
if (ret < 0)
goto put_ctxt;
@@ -1089,7 +1097,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
svc_rdma_send_ctxt_put(rdma, sctxt);
drop_connection:
trace_svcrdma_send_err(rqstp, ret);
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ svc_rdma_xprt_deferred_close(rdma);
return -ENOTCONN;
}
diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
index 1f9d14e46c9d..4afa662430ca 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
@@ -99,10 +99,26 @@ struct svc_xprt_class svc_rdma_class = {
.xcl_ident = XPRT_TRANSPORT_RDMA,
};
+/**
+ * svc_rdma_xprt_deferred_close - Close an RDMA transport (deferred)
+ * @rdma: transport to close
+ */
+void svc_rdma_xprt_deferred_close(struct svcxprt_rdma *rdma)
+{
+ svc_xprt_deferred_close(&rdma->sc_xprt);
+
+ /* Release parked sc_send_wait waiters.
+ * Once XPT_CLOSE is observed each returns -ENOTCONN.
+ */
+ wake_up_all(&rdma->sc_send_wait);
+}
+
/* QP event handler */
static void qp_event_handler(struct ib_event *event, void *context)
{
struct svc_xprt *xprt = context;
+ struct svcxprt_rdma *rdma =
+ container_of(xprt, struct svcxprt_rdma, sc_xprt);
trace_svcrdma_qp_error(event, (struct sockaddr *)&xprt->xpt_remote);
switch (event->event) {
@@ -120,7 +136,7 @@ static void qp_event_handler(struct ib_event *event, void *context)
case IB_EVENT_QP_ACCESS_ERR:
case IB_EVENT_DEVICE_FATAL:
default:
- svc_xprt_deferred_close(xprt);
+ svc_rdma_xprt_deferred_close(rdma);
break;
}
}
@@ -311,6 +327,7 @@ static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id,
if (IS_ERR(listen_id)) {
pr_err("Listener dead, address change failed for device %s\n",
cma_id->device->name);
+ cma_xprt->sc_cm_id = NULL;
} else
cma_xprt->sc_cm_id = listen_id;
return 1;
@@ -344,7 +361,7 @@ static int svc_rdma_cma_handler(struct rdma_cm_id *cma_id,
svc_xprt_enqueue(xprt);
break;
case RDMA_CM_EVENT_DISCONNECTED:
- svc_xprt_deferred_close(xprt);
+ svc_rdma_xprt_deferred_close(rdma);
break;
default:
break;
@@ -616,7 +633,16 @@ static void svc_rdma_detach(struct svc_xprt *xprt)
struct svcxprt_rdma *rdma =
container_of(xprt, struct svcxprt_rdma, sc_xprt);
- rdma_disconnect(rdma->sc_cm_id);
+ if (rdma->sc_cm_id)
+ rdma_disconnect(rdma->sc_cm_id);
+
+ /*
+ * Most close paths go through svc_rdma_xprt_deferred_close(),
+ * which wakes the SQ waitqueue. svc_xprt_close() reaches
+ * detach without that helper, so wake any threads parked in
+ * the Send Queue here as well.
+ */
+ wake_up_all(&rdma->sc_send_wait);
}
/**
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 2946f6b562fb..183891b511a5 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -1847,8 +1847,8 @@ static int xs_bind(struct sock_xprt *transport, struct socket *sock)
memcpy(&myaddr, &transport->srcaddr, transport->xprt.addrlen);
do {
rpc_set_port((struct sockaddr *)&myaddr, port);
- err = kernel_bind(sock, (struct sockaddr *)&myaddr,
- transport->xprt.addrlen);
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&myaddr,
+ transport->xprt.addrlen);
if (err == 0) {
if (transport->xprt.reuseport)
transport->srcport = port;
diff --git a/net/tipc/core.c b/net/tipc/core.c
index 1ddecea1df6e..315975c3be81 100644
--- a/net/tipc/core.c
+++ b/net/tipc/core.c
@@ -45,6 +45,7 @@
#include "crypto.h"
#include <linux/module.h>
+#include <linux/wait_bit.h>
/* configurable TIPC parameters */
unsigned int tipc_net_id __read_mostly;
@@ -118,8 +119,7 @@ static void __net_exit tipc_exit_net(struct net *net)
#ifdef CONFIG_TIPC_CRYPTO
tipc_crypto_stop(&tipc_net(net)->crypto_tx);
#endif
- while (atomic_read(&tn->wq_count))
- cond_resched();
+ wait_var_event(&tn->wq_count, atomic_read(&tn->wq_count) == 0);
}
static void __net_exit tipc_pernet_pre_exit(struct net *net)
diff --git a/net/tipc/node.c b/net/tipc/node.c
index 6eeb8bef3a43..768c5501b42e 100644
--- a/net/tipc/node.c
+++ b/net/tipc/node.c
@@ -2419,8 +2419,11 @@ static struct tipc_node *tipc_node_find_by_name(struct net *net,
}
}
tipc_node_read_unlock(n);
- if (found_node)
+ if (found_node) {
+ if (!kref_get_unless_zero(&found_node->kref))
+ found_node = NULL;
break;
+ }
}
rcu_read_unlock();
@@ -2505,6 +2508,7 @@ int tipc_nl_node_set_link(struct sk_buff *skb, struct genl_info *info)
tipc_node_read_unlock(node);
tipc_bearer_xmit(net, bearer_id, &xmitq, &node->links[bearer_id].maddr,
NULL);
+ tipc_node_put(node);
return res;
}
@@ -2556,12 +2560,14 @@ int tipc_nl_node_get_link(struct sk_buff *skb, struct genl_info *info)
link = node->links[bearer_id].link;
if (!link) {
tipc_node_read_unlock(node);
+ tipc_node_put(node);
err = -EINVAL;
goto err_free;
}
err = __tipc_nl_add_link(net, &msg, link, 0);
tipc_node_read_unlock(node);
+ tipc_node_put(node);
if (err)
goto err_free;
}
@@ -2632,11 +2638,13 @@ int tipc_nl_node_reset_link_stats(struct sk_buff *skb, struct genl_info *info)
if (!link) {
spin_unlock_bh(&le->lock);
tipc_node_read_unlock(node);
+ tipc_node_put(node);
return -EINVAL;
}
tipc_link_reset_stats(link);
spin_unlock_bh(&le->lock);
tipc_node_read_unlock(node);
+ tipc_node_put(node);
return 0;
}
diff --git a/net/tipc/socket.c b/net/tipc/socket.c
index af14b50933f5..ae7d47c2dc30 100644
--- a/net/tipc/socket.c
+++ b/net/tipc/socket.c
@@ -711,7 +711,7 @@ int tipc_sk_bind(struct socket *sock, struct sockaddr *skaddr, int alen)
return res;
}
-static int tipc_bind(struct socket *sock, struct sockaddr *skaddr, int alen)
+static int tipc_bind(struct socket *sock, struct sockaddr_unsized *skaddr, int alen)
{
struct tipc_uaddr *ua = (struct tipc_uaddr *)skaddr;
u32 atype = ua->addrtype;
@@ -727,7 +727,7 @@ static int tipc_bind(struct socket *sock, struct sockaddr *skaddr, int alen)
return -EACCES;
}
}
- return tipc_sk_bind(sock, skaddr, alen);
+ return tipc_sk_bind(sock, (struct sockaddr *)skaddr, alen);
}
/**
diff --git a/net/tipc/topsrv.c b/net/tipc/topsrv.c
index aad7f96b6009..f5e71532840b 100644
--- a/net/tipc/topsrv.c
+++ b/net/tipc/topsrv.c
@@ -55,7 +55,7 @@
/**
* struct tipc_topsrv - TIPC server structure
* @conn_idr: identifier set of connection
- * @idr_lock: protect the connection identifier set
+ * @idr_lock: protect the connection identifier set and listener
* @idr_in_use: amount of allocated identifier entry
* @net: network namespace instance
* @awork: accept work item
@@ -218,6 +218,10 @@ static struct tipc_conn *tipc_conn_lookup(struct tipc_topsrv *s, int conid)
struct tipc_conn *con;
spin_lock_bh(&s->idr_lock);
+ if (!s->listener) {
+ spin_unlock_bh(&s->idr_lock);
+ return NULL;
+ }
con = idr_find(&s->conn_idr, conid);
if (!connected(con) || !kref_get_unless_zero(&con->kref))
con = NULL;
@@ -301,10 +305,20 @@ static void tipc_conn_send_to_sock(struct tipc_conn *con)
static void tipc_conn_send_work(struct work_struct *work)
{
struct tipc_conn *con = container_of(work, struct tipc_conn, swork);
+ struct tipc_topsrv *srv;
+
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (!srv->listener) {
+ spin_unlock_bh(&srv->idr_lock);
+ goto out;
+ }
+ spin_unlock_bh(&srv->idr_lock);
if (connected(con))
tipc_conn_send_to_sock(con);
+out:
conn_put(con);
}
@@ -334,8 +348,14 @@ void tipc_topsrv_queue_evt(struct net *net, int conid,
list_add_tail(&e->list, &con->outqueue);
spin_unlock_bh(&con->outqueue_lock);
- if (queue_work(srv->send_wq, &con->swork))
- return;
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener) {
+ if (queue_work(srv->send_wq, &con->swork)) {
+ spin_unlock_bh(&srv->idr_lock);
+ return;
+ }
+ }
+ spin_unlock_bh(&srv->idr_lock);
err:
conn_put(con);
}
@@ -346,14 +366,20 @@ void tipc_topsrv_queue_evt(struct net *net, int conid,
*/
static void tipc_conn_write_space(struct sock *sk)
{
+ struct tipc_topsrv *srv;
struct tipc_conn *con;
read_lock_bh(&sk->sk_callback_lock);
con = sk->sk_user_data;
if (connected(con)) {
- conn_get(con);
- if (!queue_work(con->server->send_wq, &con->swork))
- conn_put(con);
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener) {
+ conn_get(con);
+ if (!queue_work(srv->send_wq, &con->swork))
+ conn_put(con);
+ }
+ spin_unlock_bh(&srv->idr_lock);
}
read_unlock_bh(&sk->sk_callback_lock);
}
@@ -418,8 +444,17 @@ static int tipc_conn_rcv_from_sock(struct tipc_conn *con)
static void tipc_conn_recv_work(struct work_struct *work)
{
struct tipc_conn *con = container_of(work, struct tipc_conn, rwork);
+ struct tipc_topsrv *srv;
int count = 0;
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (!srv->listener) {
+ spin_unlock_bh(&srv->idr_lock);
+ goto out;
+ }
+ spin_unlock_bh(&srv->idr_lock);
+
while (connected(con)) {
if (tipc_conn_rcv_from_sock(con))
break;
@@ -430,6 +465,7 @@ static void tipc_conn_recv_work(struct work_struct *work)
count = 0;
}
}
+out:
conn_put(con);
}
@@ -438,6 +474,7 @@ static void tipc_conn_recv_work(struct work_struct *work)
*/
static void tipc_conn_data_ready(struct sock *sk)
{
+ struct tipc_topsrv *srv;
struct tipc_conn *con;
trace_sk_data_ready(sk);
@@ -445,9 +482,14 @@ static void tipc_conn_data_ready(struct sock *sk)
read_lock_bh(&sk->sk_callback_lock);
con = sk->sk_user_data;
if (connected(con)) {
- conn_get(con);
- if (!queue_work(con->server->rcv_wq, &con->rwork))
- conn_put(con);
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener) {
+ conn_get(con);
+ if (!queue_work(srv->rcv_wq, &con->rwork))
+ conn_put(con);
+ }
+ spin_unlock_bh(&srv->idr_lock);
}
read_unlock_bh(&sk->sk_callback_lock);
}
@@ -503,8 +545,12 @@ static void tipc_topsrv_listener_data_ready(struct sock *sk)
read_lock_bh(&sk->sk_callback_lock);
srv = sk->sk_user_data;
- if (srv)
- queue_work(srv->rcv_wq, &srv->awork);
+ if (srv) {
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener)
+ queue_work(srv->rcv_wq, &srv->awork);
+ spin_unlock_bh(&srv->idr_lock);
+ }
read_unlock_bh(&sk->sk_callback_lock);
}
@@ -700,23 +746,27 @@ static void tipc_topsrv_stop(struct net *net)
struct tipc_conn *con;
int id;
+ spin_lock_bh(&srv->idr_lock);
+ srv->listener = NULL;
+ spin_unlock_bh(&srv->idr_lock);
+ tipc_topsrv_work_stop(srv);
+
spin_lock_bh(&srv->idr_lock);
for (id = 0; srv->idr_in_use; id++) {
con = idr_find(&srv->conn_idr, id);
if (con) {
- conn_get(con);
spin_unlock_bh(&srv->idr_lock);
tipc_conn_close(con);
- conn_put(con);
spin_lock_bh(&srv->idr_lock);
+ continue;
}
+ spin_unlock_bh(&srv->idr_lock);
+ spin_lock_bh(&srv->idr_lock);
}
__module_get(lsock->ops->owner);
__module_get(lsock->sk->sk_prot_creator->owner);
- srv->listener = NULL;
spin_unlock_bh(&srv->idr_lock);
- tipc_topsrv_work_stop(srv);
sock_release(lsock);
idr_destroy(&srv->conn_idr);
kfree(srv);
diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c
index b85ab0fb3b8c..d732afaeab3d 100644
--- a/net/tipc/udp_media.c
+++ b/net/tipc/udp_media.c
@@ -40,6 +40,7 @@
#include <linux/igmp.h>
#include <linux/kernel.h>
#include <linux/workqueue.h>
+#include <linux/wait_bit.h>
#include <linux/list.h>
#include <net/sock.h>
#include <net/ip.h>
@@ -94,6 +95,8 @@ struct udp_replicast {
* @ifindex: local address scope
* @work: used to schedule deferred work on a bearer
* @rcast: associated udp_replicast container
+ * @rcast_lock: serialize updates to @rcast.list against concurrent updaters
+ * @disabled: bearer is being torn down; reject further @rcast.list additions
*/
struct udp_bearer {
struct tipc_bearer __rcu *bearer;
@@ -101,6 +104,8 @@ struct udp_bearer {
u32 ifindex;
struct work_struct work;
struct udp_replicast rcast;
+ spinlock_t rcast_lock;
+ bool disabled;
};
static int tipc_udp_is_mcast_addr(struct udp_media_addr *addr)
@@ -279,26 +284,6 @@ static int tipc_udp_send_msg(struct net *net, struct sk_buff *skb,
return err;
}
-static bool tipc_udp_is_known_peer(struct tipc_bearer *b,
- struct udp_media_addr *addr)
-{
- struct udp_replicast *rcast, *tmp;
- struct udp_bearer *ub;
-
- ub = rcu_dereference_rtnl(b->media_ptr);
- if (!ub) {
- pr_err_ratelimited("UDP bearer instance not found\n");
- return false;
- }
-
- list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) {
- if (!memcmp(&rcast->addr, addr, sizeof(struct udp_media_addr)))
- return true;
- }
-
- return false;
-}
-
static int tipc_udp_rcast_add(struct tipc_bearer *b,
struct udp_media_addr *addr)
{
@@ -309,16 +294,34 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b,
if (!ub)
return -ENODEV;
+ spin_lock_bh(&ub->rcast_lock);
+ if (ub->disabled) {
+ spin_unlock_bh(&ub->rcast_lock);
+ return 0;
+ }
+ list_for_each_entry(rcast, &ub->rcast.list, list) {
+ if (!memcmp(&rcast->addr, addr, sizeof(*addr))) {
+ spin_unlock_bh(&ub->rcast_lock);
+ return 0;
+ }
+ }
+
rcast = kmalloc(sizeof(*rcast), GFP_ATOMIC);
- if (!rcast)
+ if (!rcast) {
+ spin_unlock_bh(&ub->rcast_lock);
return -ENOMEM;
+ }
if (dst_cache_init(&rcast->dst_cache, GFP_ATOMIC)) {
+ spin_unlock_bh(&ub->rcast_lock);
kfree(rcast);
return -ENOMEM;
}
memcpy(&rcast->addr, addr, sizeof(struct udp_media_addr));
+ list_add_rcu(&rcast->list, &ub->rcast.list);
+ b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT;
+ spin_unlock_bh(&ub->rcast_lock);
if (ntohs(addr->proto) == ETH_P_IP)
pr_info("New replicast peer: %pI4\n", &rcast->addr.ipv4);
@@ -326,8 +329,6 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b,
else if (ntohs(addr->proto) == ETH_P_IPV6)
pr_info("New replicast peer: %pI6\n", &rcast->addr.ipv6);
#endif
- b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT;
- list_add_rcu(&rcast->list, &ub->rcast.list);
return 0;
}
@@ -362,9 +363,6 @@ static int tipc_udp_rcast_disc(struct tipc_bearer *b, struct sk_buff *skb)
return 0;
}
- if (likely(tipc_udp_is_known_peer(b, &src)))
- return 0;
-
return tipc_udp_rcast_add(b, &src);
}
@@ -646,9 +644,6 @@ int tipc_udp_nl_bearer_add(struct tipc_bearer *b, struct nlattr *attr)
return -EINVAL;
}
- if (tipc_udp_is_known_peer(b, &addr))
- return 0;
-
return tipc_udp_rcast_add(b, &addr);
}
@@ -680,6 +675,7 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b,
return -ENOMEM;
INIT_LIST_HEAD(&ub->rcast.list);
+ spin_lock_init(&ub->rcast_lock);
if (!attrs[TIPC_NLA_BEARER_UDP_OPTS])
goto err;
@@ -802,6 +798,14 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b,
return err;
}
+static void rcast_free_rcu(struct rcu_head *rcu)
+{
+ struct udp_replicast *rcast = container_of(rcu, struct udp_replicast, rcu);
+
+ dst_cache_destroy(&rcast->dst_cache);
+ kfree(rcast);
+}
+
/* cleanup_bearer - break the socket/bearer association */
static void cleanup_bearer(struct work_struct *work)
{
@@ -809,20 +813,23 @@ static void cleanup_bearer(struct work_struct *work)
struct udp_replicast *rcast, *tmp;
struct tipc_net *tn;
+ spin_lock_bh(&ub->rcast_lock);
list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) {
- dst_cache_destroy(&rcast->dst_cache);
list_del_rcu(&rcast->list);
- kfree_rcu(rcast, rcu);
+ call_rcu_hurry(&rcast->rcu, rcast_free_rcu);
}
+ ub->disabled = true;
+ spin_unlock_bh(&ub->rcast_lock);
tn = tipc_net(sock_net(ub->ubsock->sk));
- dst_cache_destroy(&ub->rcast.dst_cache);
udp_tunnel_sock_release(ub->ubsock);
- /* Note: could use a call_rcu() to avoid another synchronize_net() */
synchronize_net();
- atomic_dec(&tn->wq_count);
+
+ dst_cache_destroy(&ub->rcast.dst_cache);
+ if (atomic_dec_and_test(&tn->wq_count))
+ wake_up_var(&tn->wq_count);
kfree(ub);
}
diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index a1e832e531bf..d1615e3f44c4 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -855,7 +855,7 @@ static int unix_listen(struct socket *sock, int backlog)
}
static int unix_release(struct socket *);
-static int unix_bind(struct socket *, struct sockaddr *, int);
+static int unix_bind(struct socket *, struct sockaddr_unsized *, int);
static int unix_stream_connect(struct socket *, struct sockaddr *,
int addr_len, int flags);
static int unix_socketpair(struct socket *, struct socket *);
@@ -1478,7 +1478,7 @@ static int unix_bind_abstract(struct sock *sk, struct sockaddr_un *sunaddr,
return err;
}
-static int unix_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int unix_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sockaddr_un *sunaddr = (struct sockaddr_un *)uaddr;
struct sock *sk = sock->sk;
diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c
index 4ebf1cab2d97..a540b3d29f9b 100644
--- a/net/vmw_vsock/af_vsock.c
+++ b/net/vmw_vsock/af_vsock.c
@@ -890,6 +890,8 @@ static void vsock_sk_destruct(struct sock *sk)
{
struct vsock_sock *vsk = vsock_sk(sk);
+ /* Socket buffers may remain in the queue in VMCI datagram mode */
+ __skb_queue_purge(&sk->sk_receive_queue);
/* Flush MSG_ZEROCOPY leftovers. */
__skb_queue_purge(&sk->sk_error_queue);
@@ -987,7 +989,7 @@ static int vsock_release(struct socket *sock)
}
static int
-vsock_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
+vsock_bind(struct socket *sock, struct sockaddr_unsized *addr, int addr_len)
{
int err;
struct sock *sk;
@@ -995,7 +997,7 @@ vsock_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
sk = sock->sk;
- if (vsock_addr_cast(addr, addr_len, &vm_addr) != 0)
+ if (vsock_addr_cast((struct sockaddr *)addr, addr_len, &vm_addr) != 0)
return -EINVAL;
lock_sock(sk);
diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index 518ef9fca33d..829feae9c0e0 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c
@@ -1903,6 +1903,7 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
struct sock *sk = sk_vsock(vsk);
struct virtio_vsock_hdr *hdr;
struct sk_buff *skb;
+ u32 bytes_read;
u32 pkt_len;
int off = 0;
int err;
@@ -1922,7 +1923,8 @@ int virtio_transport_read_skb(struct vsock_sock *vsk, skb_read_actor_t recv_acto
vvs->msg_count--;
pkt_len = le32_to_cpu(hdr->len);
- virtio_transport_dec_rx_pkt(vvs, pkt_len, pkt_len);
+ bytes_read = skb->len - VIRTIO_VSOCK_SKB_CB(skb)->offset;
+ virtio_transport_dec_rx_pkt(vvs, bytes_read, pkt_len);
spin_unlock_bh(&vvs->rx_lock);
virtio_transport_send_credit_update(vsk);
diff --git a/net/wireless/core.c b/net/wireless/core.c
index 01e44ba4eddd..c7d449868254 100644
--- a/net/wireless/core.c
+++ b/net/wireless/core.c
@@ -1385,6 +1385,7 @@ static void _cfg80211_unregister_wdev(struct wireless_dev *wdev,
list_del_rcu(&wdev->list);
synchronize_net();
rdev->devlist_generation++;
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
cfg80211_mlme_purge_registrations(wdev);
@@ -1568,7 +1569,7 @@ void cfg80211_init_wdev(struct wireless_dev *wdev)
wdev->iftype == NL80211_IFTYPE_ADHOC) && !wdev->use_4addr)
wdev->netdev->priv_flags |= IFF_DONT_BRIDGE;
- INIT_WORK(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
+ wiphy_work_init(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
}
void cfg80211_register_wdev(struct cfg80211_registered_device *rdev,
@@ -1676,9 +1677,9 @@ static int cfg80211_netdev_notifier_call(struct notifier_block *nb,
scoped_guard(wiphy, &rdev->wiphy) {
cfg80211_leave(rdev, wdev);
cfg80211_remove_links(wdev);
+ /* since we just did cfg80211_leave() nothing to do there */
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
}
- /* since we just did cfg80211_leave() nothing to do there */
- cancel_work_sync(&wdev->disconnect_wk);
break;
case NETDEV_DOWN:
wiphy_lock(&rdev->wiphy);
diff --git a/net/wireless/core.h b/net/wireless/core.h
index d7ac3145fd1a..04c014d9ad07 100644
--- a/net/wireless/core.h
+++ b/net/wireless/core.h
@@ -434,7 +434,7 @@ void __cfg80211_port_authorized(struct wireless_dev *wdev, const u8 *peer_addr,
const u8 *td_bitmap, u8 td_bitmap_len);
int cfg80211_mgd_wext_connect(struct cfg80211_registered_device *rdev,
struct wireless_dev *wdev);
-void cfg80211_autodisconnect_wk(struct work_struct *work);
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work);
/* SME implementation */
void cfg80211_conn_work(struct work_struct *work);
diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c
index 356b612d935f..b851e398b1b3 100644
--- a/net/wireless/nl80211.c
+++ b/net/wireless/nl80211.c
@@ -6532,6 +6532,12 @@ static int nl80211_calculate_ap_params(struct cfg80211_ap_settings *params)
return -EINVAL;
}
+ if (!!params->he_cap != !!params->he_oper)
+ return -EINVAL;
+
+ if (!!params->eht_cap != !!params->eht_oper)
+ return -EINVAL;
+
return 0;
}
@@ -17788,15 +17794,15 @@ static int nl80211_color_change(struct sk_buff *skb, struct genl_info *info)
if (!wdev->links[params.link_id].ap.beacon_interval)
return -EINVAL;
- err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next,
- info->extack);
- if (err)
- return err;
-
tb = kcalloc(NL80211_ATTR_MAX + 1, sizeof(*tb), GFP_KERNEL);
if (!tb)
return -ENOMEM;
+ err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next,
+ info->extack);
+ if (err)
+ goto out;
+
err = nla_parse_nested(tb, NL80211_ATTR_MAX,
info->attrs[NL80211_ATTR_COLOR_CHANGE_ELEMS],
nl80211_policy, info->extack);
@@ -21950,7 +21956,8 @@ static int nl80211_netlink_notify(struct notifier_block * nb,
wdev->nl_owner_dead = true;
schedule_work(&rdev->destroy_work);
} else if (wdev->conn_owner_nlportid == notify->portid) {
- schedule_work(&wdev->disconnect_wk);
+ wiphy_work_queue(wdev->wiphy,
+ &wdev->disconnect_wk);
}
cfg80211_release_pmsr(wdev, notify->portid);
diff --git a/net/wireless/sme.c b/net/wireless/sme.c
index 4e629ca305bc..72166b70ee67 100644
--- a/net/wireless/sme.c
+++ b/net/wireless/sme.c
@@ -1577,13 +1577,11 @@ int cfg80211_disconnect(struct cfg80211_registered_device *rdev,
* Used to clean up after the connection / connection attempt owner socket
* disconnects
*/
-void cfg80211_autodisconnect_wk(struct work_struct *work)
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work)
{
struct wireless_dev *wdev =
container_of(work, struct wireless_dev, disconnect_wk);
- struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
-
- guard(wiphy)(wdev->wiphy);
+ struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
if (wdev->conn_owner_nlportid) {
switch (wdev->iftype) {
diff --git a/net/x25/af_x25.c b/net/x25/af_x25.c
index 13ba76effe5d..65d8057ae1b4 100644
--- a/net/x25/af_x25.c
+++ b/net/x25/af_x25.c
@@ -670,7 +670,7 @@ static int x25_release(struct socket *sock)
return 0;
}
-static int x25_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+static int x25_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sock *sk = sock->sk;
struct sockaddr_x25 *addr = (struct sockaddr_x25 *)uaddr;
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index 2e41770a9988..68194711dded 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -454,6 +454,23 @@ void __xsk_map_flush(struct list_head *flush_list)
void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries)
{
+ u32 reclaim_descs = READ_ONCE(pool->reclaim_descs);
+
+ if (unlikely(reclaim_descs)) {
+ u32 pending_descs = READ_ONCE(pool->tx_zc_pending_descs);
+
+ if (nb_entries < pending_descs) {
+ WRITE_ONCE(pool->tx_zc_pending_descs,
+ pending_descs - nb_entries);
+ xskq_prod_submit_n(pool->cq, nb_entries);
+ return;
+ }
+
+ WRITE_ONCE(pool->tx_zc_pending_descs, 0);
+ nb_entries += reclaim_descs;
+ WRITE_ONCE(pool->reclaim_descs, 0);
+ }
+
xskq_prod_submit_n(pool->cq, nb_entries);
}
EXPORT_SYMBOL(xsk_tx_completed);
@@ -529,24 +546,157 @@ static u32 xsk_tx_peek_release_fallback(struct xsk_buff_pool *pool, u32 max_entr
return nb_pkts;
}
+static void xsk_tx_commit_batch(struct xsk_buff_pool *pool,
+ struct xsk_tx_batch *batch)
+{
+ u32 nb_descs = xsk_tx_batch_cq_descs(batch);
+ u32 cq_cached_prod;
+
+ if (!nb_descs)
+ return;
+
+ cq_cached_prod = pool->cq->cached_prod;
+ xskq_prod_write_addr_batch(pool->cq, pool->tx_descs, nb_descs);
+
+ if (unlikely(batch->reclaim_descs)) {
+ u32 cq_pending_descs;
+
+ /* CQ is positional. Descriptors already written but not
+ * submitted must complete before any reclaim-only descriptors
+ * appended below.
+ */
+ cq_pending_descs = cq_cached_prod - xskq_get_prod(pool->cq);
+
+ WRITE_ONCE(pool->tx_zc_pending_descs,
+ batch->tx_descs + cq_pending_descs);
+ WRITE_ONCE(pool->reclaim_descs, batch->reclaim_descs);
+ if (unlikely(!pool->tx_zc_pending_descs))
+ xsk_tx_completed(pool, 0);
+ }
+}
+
+static struct xsk_tx_batch
+__xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, struct xdp_sock *xs,
+ struct xdp_desc *descs, u32 max_descs)
+{
+ struct xsk_tx_batch batch = {};
+ u32 entries;
+
+ entries = xskq_cons_nb_entries(xs->tx, max_descs);
+ if (!entries)
+ return batch;
+
+ batch = xskq_cons_read_desc_batch(xs, pool, descs, max_descs);
+ if (!xsk_tx_batch_cq_descs(&batch)) {
+ xs->tx->queue_empty_descs++;
+ } else {
+ __xskq_cons_release(xs->tx);
+ xs->sk.sk_write_space(&xs->sk);
+ }
+ return batch;
+}
+
+static struct xsk_tx_batch
+xsk_tx_peek_release_shared_desc_batch(struct xsk_buff_pool *pool, u32 max_descs)
+{
+ u32 cq_descs_before, cq_descs_after;
+ struct xsk_tx_batch sum_batch = {};
+ bool budget_exhausted;
+ u32 per_socket_budget;
+ struct xdp_sock *xs;
+
+ /* The fairness quota must allow one maximum-sized valid packet. */
+ per_socket_budget = max_t(u32, MAX_PER_SOCKET_BUDGET,
+ pool->xdp_zc_max_segs);
+
+again:
+ budget_exhausted = false;
+ cq_descs_before = xsk_tx_batch_cq_descs(&sum_batch);
+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) {
+ u32 budget, budget_left, offset, remaining, used;
+ struct xsk_tx_batch curr_batch;
+
+ /* Once reclaim-only descriptors have been appended to the CQ
+ * address area, do not append driver-visible Tx descriptors
+ * from another socket after them. xsk_tx_completed() relies on
+ * all driver-visible descriptors preceding all reclaim-only
+ * descriptors in CQ order.
+ */
+ if (sum_batch.reclaim_descs)
+ break;
+
+ /* be gentle when playing with pool->tx_descs */
+ offset = xsk_tx_batch_cq_descs(&sum_batch);
+ if (offset >= max_descs)
+ break;
+
+ if (xs->tx_budget_spent >= per_socket_budget) {
+ if (xskq_cons_nb_entries(xs->tx, 1))
+ budget_exhausted = true;
+ continue;
+ }
+
+ budget_left = per_socket_budget - xs->tx_budget_spent;
+ remaining = max_descs - offset;
+ budget = min(remaining, budget_left);
+
+ curr_batch = __xsk_tx_peek_release_desc_batch(pool, xs,
+ pool->tx_descs + offset,
+ budget);
+ used = xsk_tx_batch_cq_descs(&curr_batch);
+ if (!used) {
+ if (curr_batch.budget_limited && budget_left < remaining)
+ budget_exhausted = true;
+ continue;
+ }
+
+ xs->tx_budget_spent += used;
+ sum_batch.tx_descs += curr_batch.tx_descs;
+ sum_batch.reclaim_descs = curr_batch.reclaim_descs;
+ }
+
+ cq_descs_after = xsk_tx_batch_cq_descs(&sum_batch);
+
+ if (sum_batch.reclaim_descs || cq_descs_after >= max_descs)
+ return sum_batch;
+
+ /* Continue filling the batch while this pass made progress */
+ if (cq_descs_before != cq_descs_after)
+ goto again;
+
+ if (!budget_exhausted)
+ return sum_batch;
+
+ list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list)
+ xs->tx_budget_spent = 0;
+ goto again;
+}
+
u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 nb_pkts)
{
+ struct xsk_tx_batch batch = {};
struct xdp_sock *xs;
+ bool umem_shared;
rcu_read_lock();
- if (!list_is_singular(&pool->xsk_tx_list)) {
- /* Fallback to the non-batched version */
- rcu_read_unlock();
- return xsk_tx_peek_release_fallback(pool, nb_pkts);
- }
+ if (unlikely(READ_ONCE(pool->reclaim_descs)))
+ goto out;
- xs = list_first_or_null_rcu(&pool->xsk_tx_list, struct xdp_sock, tx_list);
- if (!xs) {
- nb_pkts = 0;
+ xs = list_first_or_null_rcu(&pool->xsk_tx_list, struct xdp_sock,
+ tx_list);
+ if (!xs)
goto out;
- }
- nb_pkts = xskq_cons_nb_entries(xs->tx, nb_pkts);
+ nb_pkts = min(nb_pkts, pool->tx_descs_nentries);
+ if (!nb_pkts)
+ goto out;
+
+ umem_shared = !list_is_singular(&pool->xsk_tx_list);
+
+ if (umem_shared && !(pool->umem->flags & XDP_UMEM_SG_FLAG)) {
+ rcu_read_unlock();
+ return xsk_tx_peek_release_fallback(pool, nb_pkts);
+ }
/* This is the backpressure mechanism for the Tx path. Try to
* reserve space in the completion queue for all packets, but
@@ -558,19 +708,16 @@ u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 nb_pkts)
if (!nb_pkts)
goto out;
- nb_pkts = xskq_cons_read_desc_batch(xs->tx, pool, nb_pkts);
- if (!nb_pkts) {
- xs->tx->queue_empty_descs++;
- goto out;
- }
-
- __xskq_cons_release(xs->tx);
- xskq_prod_write_addr_batch(pool->cq, pool->tx_descs, nb_pkts);
- xs->sk.sk_write_space(&xs->sk);
+ batch = umem_shared ?
+ xsk_tx_peek_release_shared_desc_batch(pool, nb_pkts) :
+ __xsk_tx_peek_release_desc_batch(pool, xs,
+ pool->tx_descs,
+ nb_pkts);
+ xsk_tx_commit_batch(pool, &batch);
out:
rcu_read_unlock();
- return nb_pkts;
+ return batch.tx_descs;
}
EXPORT_SYMBOL(xsk_tx_peek_release_desc_batch);
@@ -1361,7 +1508,7 @@ static bool xsk_validate_queues(struct xdp_sock *xs)
return xs->fq_tmp && xs->cq_tmp;
}
-static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
+static int xsk_bind(struct socket *sock, struct sockaddr_unsized *addr, int addr_len)
{
struct sockaddr_xdp *sxdp = (struct sockaddr_xdp *)addr;
struct sock *sk = sock->sk;
@@ -1442,7 +1589,8 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
* and/or device.
*/
xs->pool = xp_create_and_assign_umem(xs,
- umem_xs->umem);
+ umem_xs->umem,
+ dev->xdp_zc_max_segs);
if (!xs->pool) {
err = -ENOMEM;
sockfd_put(sock);
@@ -1474,7 +1622,8 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
* utilizes
*/
if (xs->tx && !xs->pool->tx_descs) {
- err = xp_alloc_tx_descs(xs->pool, xs);
+ err = xp_alloc_tx_descs(xs->pool, xs,
+ dev->xdp_zc_max_segs);
if (err) {
xp_put_pool(xs->pool);
xs->pool = NULL;
@@ -1492,7 +1641,9 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
goto out_unlock;
} else {
/* This xsk has its own umem. */
- xs->pool = xp_create_and_assign_umem(xs, xs->umem);
+ xs->pool = xp_create_and_assign_umem(xs, xs->umem,
+ dev->xdp_zc_max_segs);
+
if (!xs->pool) {
err = -ENOMEM;
goto out_unlock;
diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c
index 6e9c51b6b430..5ee0edaf075c 100644
--- a/net/xdp/xsk_buff_pool.c
+++ b/net/xdp/xsk_buff_pool.c
@@ -46,18 +46,23 @@ void xp_destroy(struct xsk_buff_pool *pool)
kvfree(pool);
}
-int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs)
+int xp_alloc_tx_descs(struct xsk_buff_pool *pool, struct xdp_sock *xs,
+ u32 max_segs)
{
- pool->tx_descs = kvcalloc(xs->tx->nentries, sizeof(*pool->tx_descs),
+ u32 nentries = max(xs->tx->nentries, max_segs);
+
+ pool->tx_descs = kvcalloc(nentries, sizeof(*pool->tx_descs),
GFP_KERNEL);
if (!pool->tx_descs)
return -ENOMEM;
+ pool->tx_descs_nentries = nentries;
return 0;
}
struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
- struct xdp_umem *umem)
+ struct xdp_umem *umem,
+ u32 max_segs)
{
bool unaligned = umem->flags & XDP_UMEM_UNALIGNED_CHUNK_FLAG;
struct xsk_buff_pool *pool;
@@ -74,7 +79,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
goto out;
if (xs->tx)
- if (xp_alloc_tx_descs(pool, xs))
+ if (xp_alloc_tx_descs(pool, xs, max_segs))
goto out;
pool->chunk_mask = ~((u64)umem->chunk_size - 1);
diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h
index 4f764b5748d2..69a48f1f26b8 100644
--- a/net/xdp/xsk_queue.h
+++ b/net/xdp/xsk_queue.h
@@ -53,6 +53,17 @@ struct parsed_desc {
u32 valid;
};
+struct xsk_tx_batch {
+ u32 tx_descs;
+ u32 reclaim_descs;
+ bool budget_limited;
+};
+
+static inline u32 xsk_tx_batch_cq_descs(const struct xsk_tx_batch *batch)
+{
+ return batch->tx_descs + batch->reclaim_descs;
+}
+
/* The structure of the shared state of the rings are a simple
* circular buffer, as outlined in
* Documentation/core-api/circular-buffers.rst. For the Rx and
@@ -258,17 +269,18 @@ static inline void parse_desc(struct xsk_queue *q, struct xsk_buff_pool *pool,
parsed->mb = xp_mb_desc(desc);
}
-static inline
-u32 xskq_cons_read_desc_batch(struct xsk_queue *q, struct xsk_buff_pool *pool,
- u32 max)
+static inline struct xsk_tx_batch
+xskq_cons_read_desc_batch(struct xdp_sock *xs, struct xsk_buff_pool *pool,
+ struct xdp_desc *descs, u32 max)
{
- u32 cached_cons = q->cached_cons, nb_entries = 0;
- struct xdp_desc *descs = pool->tx_descs;
- u32 total_descs = 0, nr_frags = 0;
+ bool drain = READ_ONCE(xs->drain_cont);
+ u32 cached_cons, nb_entries = 0;
+ struct xsk_tx_batch batch = {};
+ struct xsk_queue *q = xs->tx;
+ u32 nr_frags = 0;
+
+ cached_cons = q->cached_cons;
- /* track first entry, if stumble upon *any* invalid descriptor, rewind
- * current packet that consists of frags and stop the processing
- */
while (cached_cons != q->cached_prod && nb_entries < max) {
struct xdp_rxtx_ring *ring = (struct xdp_rxtx_ring *)q->ring;
u32 idx = cached_cons & q->ring_mask;
@@ -278,25 +290,42 @@ u32 xskq_cons_read_desc_batch(struct xsk_queue *q, struct xsk_buff_pool *pool,
cached_cons++;
parse_desc(q, pool, &descs[nb_entries], &parsed);
if (unlikely(!parsed.valid))
- break;
+ drain = true;
+
+ nr_frags++;
+ nb_entries++;
if (likely(!parsed.mb)) {
- total_descs += (nr_frags + 1);
- nr_frags = 0;
- } else {
- nr_frags++;
- if (nr_frags == pool->xdp_zc_max_segs) {
+ if (unlikely(drain)) {
+ batch.reclaim_descs = nr_frags;
+ WRITE_ONCE(xs->drain_cont, false);
nr_frags = 0;
break;
}
+
+ batch.tx_descs += nr_frags;
+ nr_frags = 0;
+ continue;
+ }
+
+ if (nr_frags == pool->xdp_zc_max_segs)
+ drain = true;
+ }
+
+ if (nr_frags) {
+ if (drain) {
+ batch.reclaim_descs = nr_frags;
+ WRITE_ONCE(xs->drain_cont, true);
+ } else {
+ if (nb_entries == max)
+ batch.budget_limited = true;
+ cached_cons -= nr_frags;
}
- nb_entries++;
}
- cached_cons -= nr_frags;
/* Release valid plus any invalid entries */
xskq_cons_release_n(q, cached_cons - q->cached_cons);
- return total_descs;
+ return batch;
}
/* Functions for consumers */
diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c
index b152edf50fec..262eea7bdea3 100644
--- a/net/xfrm/xfrm_output.c
+++ b/net/xfrm/xfrm_output.c
@@ -816,16 +816,22 @@ int xfrm_output(struct sock *sk, struct sk_buff *skb)
}
if (skb->ip_summed == CHECKSUM_PARTIAL) {
- err = skb_checksum_help(skb);
- if (err) {
- XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
- kfree_skb(skb);
- return err;
+ if (skb->protocol == htons(ETH_P_IP)) {
+ err = ip_check_csum_start(skb);
+ if (err)
+ goto error;
}
+ err = skb_checksum_help(skb);
+ if (err)
+ goto error;
}
out:
return xfrm_output2(net, sk, skb);
+error:
+ XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
+ kfree_skb(skb);
+ return err;
}
EXPORT_SYMBOL_GPL(xfrm_output);
diff --git a/security/keys/key.c b/security/keys/key.c
index 3bbdde778631..aebcf7c66436 100644
--- a/security/keys/key.c
+++ b/security/keys/key.c
@@ -840,9 +840,6 @@ static key_ref_t __key_create_or_update(key_ref_t keyring_ref,
key_check(keyring);
- if (!(flags & KEY_ALLOC_BYPASS_RESTRICTION))
- restrict_link = keyring->restrict_link;
-
key_ref = ERR_PTR(-ENOTDIR);
if (keyring->type != &key_type_keyring)
goto error_put_type;
@@ -880,6 +877,9 @@ static key_ref_t __key_create_or_update(key_ref_t keyring_ref,
goto error_link_end;
}
+ if (!(flags & KEY_ALLOC_BYPASS_RESTRICTION))
+ restrict_link = keyring->restrict_link;
+
if (restrict_link && restrict_link->check) {
ret = restrict_link->check(keyring, index_key.type,
&prep.payload, restrict_link->key);
diff --git a/security/keys/persistent.c b/security/keys/persistent.c
index 97af230aa4b2..bdab99539326 100644
--- a/security/keys/persistent.c
+++ b/security/keys/persistent.c
@@ -33,25 +33,31 @@ static int key_create_persistent_register(struct user_namespace *ns)
}
/*
- * Create the persistent keyring for the specified user.
+ * Get or create the persistent keyring for the specified user.
*
* Called with the namespace's sem locked for writing.
+ *
+ * Returns a boolean indicating whether the keyring already existed,
+ * or a negative error. On success, *persistent_ref holds a reference
+ * to the keyring.
*/
-static key_ref_t key_create_persistent(struct user_namespace *ns, kuid_t uid,
- struct keyring_index_key *index_key)
+static int key_get_or_create_persistent(struct user_namespace *ns, kuid_t uid,
+ struct keyring_index_key *index_key,
+ key_ref_t *persistent_ref)
{
struct key *persistent;
- key_ref_t reg_ref, persistent_ref;
+ key_ref_t reg_ref;
if (!ns->persistent_keyring_register) {
- long err = key_create_persistent_register(ns);
+ int err = key_create_persistent_register(ns);
+
if (err < 0)
- return ERR_PTR(err);
+ return err;
} else {
reg_ref = make_key_ref(ns->persistent_keyring_register, true);
- persistent_ref = find_key_to_update(reg_ref, index_key);
- if (persistent_ref)
- return persistent_ref;
+ *persistent_ref = find_key_to_update(reg_ref, index_key);
+ if (*persistent_ref)
+ return 1;
}
persistent = keyring_alloc(index_key->description,
@@ -61,9 +67,10 @@ static key_ref_t key_create_persistent(struct user_namespace *ns, kuid_t uid,
KEY_ALLOC_NOT_IN_QUOTA, NULL,
ns->persistent_keyring_register);
if (IS_ERR(persistent))
- return ERR_CAST(persistent);
+ return PTR_ERR(persistent);
- return make_key_ref(persistent, true);
+ *persistent_ref = make_key_ref(persistent, true);
+ return 0;
}
/*
@@ -78,6 +85,7 @@ static long key_get_persistent(struct user_namespace *ns, kuid_t uid,
key_ref_t reg_ref, persistent_ref;
char buf[32];
long ret;
+ bool created = false;
/* Look in the register if it exists */
memset(&index_key, 0, sizeof(index_key));
@@ -100,23 +108,24 @@ static long key_get_persistent(struct user_namespace *ns, kuid_t uid,
* also need to create the register.
*/
down_write(&ns->keyring_sem);
- persistent_ref = key_create_persistent(ns, uid, &index_key);
+ ret = key_get_or_create_persistent(ns, uid, &index_key,
+ &persistent_ref);
up_write(&ns->keyring_sem);
- if (!IS_ERR(persistent_ref))
- goto found;
-
- return PTR_ERR(persistent_ref);
+ if (ret < 0)
+ return ret;
+ created = ret == 0;
found:
+ persistent = key_ref_to_ptr(persistent_ref);
ret = key_task_permission(persistent_ref, current_cred(), KEY_NEED_LINK);
- if (ret == 0) {
- persistent = key_ref_to_ptr(persistent_ref);
+ if (ret == 0)
ret = key_link(key_ref_to_ptr(dest_ref), persistent);
- if (ret == 0) {
- key_set_timeout(persistent, persistent_keyring_expiry);
- ret = persistent->serial;
- }
- }
+
+ if (ret == 0 || created)
+ key_set_timeout(persistent, persistent_keyring_expiry);
+
+ if (ret == 0)
+ ret = persistent->serial;
key_ref_put(persistent_ref);
return ret;
diff --git a/security/selinux/avc.c b/security/selinux/avc.c
index 77d8828a8560..9bc1ec8b79af 100644
--- a/security/selinux/avc.c
+++ b/security/selinux/avc.c
@@ -578,7 +578,7 @@ static int avc_latest_notif_update(u32 seqno, int is_insert)
}
} else {
if (seqno > selinux_avc.avc_cache.latest_notif)
- selinux_avc.avc_cache.latest_notif = seqno;
+ WRITE_ONCE(selinux_avc.avc_cache.latest_notif, seqno);
}
spin_unlock_irqrestore(¬if_lock, flag);
@@ -1206,5 +1206,6 @@ int avc_has_perm(u32 ssid, u32 tsid, u16 tclass,
u32 avc_policy_seqno(void)
{
- return selinux_avc.avc_cache.latest_notif;
+ /* Writers hold notif_lock; readers only sample the sequence number. */
+ return READ_ONCE(selinux_avc.avc_cache.latest_notif);
}
diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c
index eef75bde66f5..86c82a6df54c 100644
--- a/security/selinux/hooks.c
+++ b/security/selinux/hooks.c
@@ -686,8 +686,12 @@ static int selinux_set_mnt_opts(struct super_block *sb,
* will be used for both mounts)
*/
if ((sbsec->flags & SE_SBINITIALIZED) && (sb->s_type->fs_flags & FS_BINARY_MOUNTDATA)
- && !opts)
+ && !opts) {
+ if ((kern_flags & SECURITY_LSM_NATIVE_LABELS) &&
+ sbsec->behavior == SECURITY_FS_USE_NATIVE)
+ *set_kern_flags |= SECURITY_LSM_NATIVE_LABELS;
goto out;
+ }
root_isec = backing_inode_security_novalidate(root);
@@ -731,6 +735,9 @@ static int selinux_set_mnt_opts(struct super_block *sb,
/* previously mounted with options, but not on this attempt? */
if ((sbsec->flags & SE_MNTMASK) && !opts)
goto out_double_mount;
+ if ((kern_flags & SECURITY_LSM_NATIVE_LABELS) &&
+ sbsec->behavior == SECURITY_FS_USE_NATIVE)
+ *set_kern_flags |= SECURITY_LSM_NATIVE_LABELS;
rc = 0;
goto out;
}
diff --git a/sound/hda/codecs/side-codecs/tas2781_hda.c b/sound/hda/codecs/side-codecs/tas2781_hda.c
index 96e6d82dc69e..00628e0b5c3e 100644
--- a/sound/hda/codecs/side-codecs/tas2781_hda.c
+++ b/sound/hda/codecs/side-codecs/tas2781_hda.c
@@ -139,9 +139,21 @@ static void tas2781_apply_calib(struct tasdevice_priv *p)
*/
crc = crc32(~0, data, 84) ^ ~0;
if (crc != tmp_val[21]) {
- cali_data->total_sz = 0;
- dev_err(p->dev, "%s: V1 CRC error\n", __func__);
- return;
+ /*
+ * Some Lenovo BIOSes (e.g. Legion Pro 7 16IRX9H,
+ * SSID 17aa:38cd) store valid V1 calibration data
+ * but leave TimeStamp and CRC zeroed.
+ */
+ if (!tmp_val[20] && !tmp_val[21] &&
+ tmp_val[0] && tmp_val[1]) {
+ dev_warn(p->dev,
+ "%s: V1 data without CRC, accepting\n",
+ __func__);
+ } else {
+ cali_data->total_sz = 0;
+ dev_err(p->dev, "%s: V1 CRC error\n", __func__);
+ return;
+ }
}
for (j = p->ndev - 1; j >= 0; j--) {
diff --git a/sound/pci/ctxfi/ctatc.c b/sound/pci/ctxfi/ctatc.c
index d65c7dd73806..7cb33d7c43d2 100644
--- a/sound/pci/ctxfi/ctatc.c
+++ b/sound/pci/ctxfi/ctatc.c
@@ -25,6 +25,9 @@
#include <sound/control.h>
#include <sound/asoundef.h>
+#define NUM_ATC_SRCS 6
+#define NUM_ATC_PCM (2 * 4)
+
#define MONO_SUM_SCALE 0x19a8 /* 2^(-0.5) in 14-bit floating format */
#define MAX_MULTI_CHN 8
@@ -1162,9 +1165,11 @@ static int atc_release_resources(struct ct_atc *atc)
if (atc->daios) {
daio_mgr = (struct daio_mgr *)atc->rsc_mgrs[DAIO];
- for (i = 0; i < atc->n_daio; i++) {
+ for (i = 0; i < NUM_DAIOTYP; i++) {
daio = atc->daios[i];
- if (daio->type < LINEIM) {
+ if (!daio)
+ continue;
+ if (daio->output) {
dao = container_of(daio, struct dao, daio);
dao->ops->clear_left_input(dao);
dao->ops->clear_right_input(dao);
@@ -1177,8 +1182,9 @@ static int atc_release_resources(struct ct_atc *atc)
if (atc->pcm) {
sum_mgr = atc->rsc_mgrs[SUM];
- for (i = 0; i < atc->n_pcm; i++)
- sum_mgr->put_sum(sum_mgr, atc->pcm[i]);
+ for (i = 0; i < NUM_ATC_PCM; i++)
+ if (atc->pcm[i])
+ sum_mgr->put_sum(sum_mgr, atc->pcm[i]);
kfree(atc->pcm);
atc->pcm = NULL;
@@ -1186,8 +1192,9 @@ static int atc_release_resources(struct ct_atc *atc)
if (atc->srcs) {
src_mgr = atc->rsc_mgrs[SRC];
- for (i = 0; i < atc->n_src; i++)
- src_mgr->put_src(src_mgr, atc->srcs[i]);
+ for (i = 0; i < NUM_ATC_SRCS; i++)
+ if (atc->srcs[i])
+ src_mgr->put_src(src_mgr, atc->srcs[i]);
kfree(atc->srcs);
atc->srcs = NULL;
@@ -1195,7 +1202,9 @@ static int atc_release_resources(struct ct_atc *atc)
if (atc->srcimps) {
srcimp_mgr = atc->rsc_mgrs[SRCIMP];
- for (i = 0; i < atc->n_srcimp; i++) {
+ for (i = 0; i < NUM_ATC_SRCS; i++) {
+ if (!atc->srcimps[i])
+ continue;
srcimp = atc->srcimps[i];
srcimp->ops->unmap(srcimp);
srcimp_mgr->put_srcimp(srcimp_mgr, atc->srcimps[i]);
@@ -1368,32 +1377,35 @@ static int atc_get_resources(struct ct_atc *atc)
struct srcimp_mgr *srcimp_mgr;
struct sum_desc sum_dsc = {0};
struct sum_mgr *sum_mgr;
- int err, i, num_srcs, num_daios;
+ struct capabilities cap;
+ int err, i;
- num_daios = ((atc->model == CTSB1270) ? 8 : 7);
- num_srcs = ((atc->model == CTSB1270) ? 6 : 4);
+ cap = atc->capabilities(atc);
- atc->daios = kcalloc(num_daios, sizeof(void *), GFP_KERNEL);
+ atc->daios = kcalloc(NUM_DAIOTYP, sizeof(void *), GFP_KERNEL);
if (!atc->daios)
return -ENOMEM;
- atc->srcs = kcalloc(num_srcs, sizeof(void *), GFP_KERNEL);
+ atc->srcs = kcalloc(NUM_ATC_SRCS, sizeof(void *), GFP_KERNEL);
if (!atc->srcs)
return -ENOMEM;
- atc->srcimps = kcalloc(num_srcs, sizeof(void *), GFP_KERNEL);
+ atc->srcimps = kcalloc(NUM_ATC_SRCS, sizeof(void *), GFP_KERNEL);
if (!atc->srcimps)
return -ENOMEM;
- atc->pcm = kcalloc(2 * 4, sizeof(void *), GFP_KERNEL);
+ atc->pcm = kcalloc(NUM_ATC_PCM, sizeof(void *), GFP_KERNEL);
if (!atc->pcm)
return -ENOMEM;
daio_mgr = (struct daio_mgr *)atc->rsc_mgrs[DAIO];
da_desc.msr = atc->msr;
- for (i = 0, atc->n_daio = 0; i < num_daios; i++) {
+ for (i = 0; i < NUM_DAIOTYP; i++) {
+ if ((i == MIC) && !cap.dedicated_mic)
+ continue;
da_desc.type = (atc->model != CTSB073X) ? i :
((i == SPDIFIO) ? SPDIFI1 : i);
+ da_desc.output = i < LINEIM;
err = daio_mgr->get_daio(daio_mgr, &da_desc,
(struct daio **)&atc->daios[i]);
if (err) {
@@ -1402,42 +1414,39 @@ static int atc_get_resources(struct ct_atc *atc)
i);
return err;
}
- atc->n_daio++;
}
src_mgr = atc->rsc_mgrs[SRC];
src_dsc.multi = 1;
src_dsc.msr = atc->msr;
src_dsc.mode = ARCRW;
- for (i = 0, atc->n_src = 0; i < num_srcs; i++) {
+ for (i = 0; i < NUM_ATC_SRCS; i++) {
+ if (((i > 3) && !cap.dedicated_mic))
+ continue;
err = src_mgr->get_src(src_mgr, &src_dsc,
(struct src **)&atc->srcs[i]);
if (err)
return err;
-
- atc->n_src++;
}
srcimp_mgr = atc->rsc_mgrs[SRCIMP];
srcimp_dsc.msr = 8;
- for (i = 0, atc->n_srcimp = 0; i < num_srcs; i++) {
+ for (i = 0; i < NUM_ATC_SRCS; i++) {
+ if (((i > 3) && !cap.dedicated_mic))
+ continue;
err = srcimp_mgr->get_srcimp(srcimp_mgr, &srcimp_dsc,
(struct srcimp **)&atc->srcimps[i]);
if (err)
return err;
-
- atc->n_srcimp++;
}
sum_mgr = atc->rsc_mgrs[SUM];
sum_dsc.msr = atc->msr;
- for (i = 0, atc->n_pcm = 0; i < (2*4); i++) {
+ for (i = 0; i < NUM_ATC_PCM; i++) {
err = sum_mgr->get_sum(sum_mgr, &sum_dsc,
(struct sum **)&atc->pcm[i]);
if (err)
return err;
-
- atc->n_pcm++;
}
return 0;
@@ -1490,9 +1499,11 @@ static void atc_connect_resources(struct ct_atc *atc)
struct sum *sum;
struct ct_mixer *mixer;
struct rsc *rscs[2] = {NULL};
+ struct capabilities cap;
int i, j;
mixer = atc->mixer;
+ cap = atc->capabilities(atc);
for (i = MIX_WAVE_FRONT, j = LINEO1; i <= MIX_SPDIF_OUT; i++, j++) {
mixer->get_output_ports(mixer, i, &rscs[0], &rscs[1]);
@@ -1510,7 +1521,7 @@ static void atc_connect_resources(struct ct_atc *atc)
src = atc->srcs[3];
mixer->set_input_right(mixer, MIX_LINE_IN, &src->rsc);
- if (atc->model == CTSB1270) {
+ if (cap.dedicated_mic) {
/* Titanium HD has a dedicated ADC for the Mic. */
dai = container_of(atc->daios[MIC], struct dai, daio);
atc_connect_dai(atc->rsc_mgrs[SRC], dai,
diff --git a/sound/pci/ctxfi/ctatc.h b/sound/pci/ctxfi/ctatc.h
index 0bc7b71d910b..671edc46bd50 100644
--- a/sound/pci/ctxfi/ctatc.h
+++ b/sound/pci/ctxfi/ctatc.h
@@ -132,10 +132,6 @@ struct ct_atc {
void **pcm; /* SUMs for collecting all pcm stream */
void **srcs; /* Sample Rate Converters for input signal */
void **srcimps; /* input mappers for SRCs */
- unsigned char n_daio;
- unsigned char n_src;
- unsigned char n_srcimp;
- unsigned char n_pcm;
struct ct_timer *timer;
diff --git a/sound/pci/ctxfi/ctdaio.c b/sound/pci/ctxfi/ctdaio.c
index 9ac9f172b383..fe40fcee3673 100644
--- a/sound/pci/ctxfi/ctdaio.c
+++ b/sound/pci/ctxfi/ctdaio.c
@@ -18,8 +18,6 @@
#include <linux/slab.h>
#include <linux/kernel.h>
-#define DAIO_OUT_MAX SPDIFOO
-
struct daio_usage {
unsigned short data;
};
@@ -100,7 +98,7 @@ static const struct rsc_ops daio_in_rsc_ops_20k2 = {
.output_slot = daio_index,
};
-static unsigned int daio_device_index(enum DAIOTYP type, struct hw *hw)
+static int daio_device_index(enum DAIOTYP type, struct hw *hw)
{
switch (hw->chip_type) {
case ATC20K1:
@@ -113,7 +111,9 @@ static unsigned int daio_device_index(enum DAIOTYP type, struct hw *hw)
case LINEO3: return 5;
case LINEO4: return 6;
case LINEIM: return 7;
- default: return -EINVAL;
+ default:
+ pr_err("ctxfi: Invalid type %d for hw20k1\n", type);
+ return -EINVAL;
}
case ATC20K2:
switch (type) {
@@ -126,9 +126,12 @@ static unsigned int daio_device_index(enum DAIOTYP type, struct hw *hw)
case LINEO4: return 6;
case LINEIM: return 4;
case MIC: return 5;
- default: return -EINVAL;
+ default:
+ pr_err("ctxfi: Invalid type %d for hw20k2\n", type);
+ return -EINVAL;
}
default:
+ pr_err("ctxfi: Invalid chip type %d\n", hw->chip_type);
return -EINVAL;
}
}
@@ -149,8 +152,11 @@ static int dao_spdif_set_spos(struct dao *dao, unsigned int spos)
static int dao_commit_write(struct dao *dao)
{
- dao->hw->dao_commit_write(dao->hw,
- daio_device_index(dao->daio.type, dao->hw), dao->ctrl_blk);
+ int idx = daio_device_index(dao->daio.type, dao->hw);
+
+ if (idx < 0)
+ return idx;
+ dao->hw->dao_commit_write(dao->hw, idx, dao->ctrl_blk);
return 0;
}
@@ -214,14 +220,18 @@ static int dao_set_right_input(struct dao *dao, struct rsc *input)
static int dao_clear_input(struct dao *dao, unsigned int start, unsigned int end)
{
+ struct imapper *entry;
unsigned int i;
- if (!dao->imappers[start])
+ entry = dao->imappers[start];
+ if (!entry)
return 0;
+
for (i = start; i < end; i++) {
dao->mgr->imap_delete(dao->mgr, dao->imappers[i]);
dao->imappers[i] = NULL;
}
+ kfree(entry);
return 0;
}
@@ -288,8 +298,11 @@ static int dai_set_enb_srt(struct dai *dai, unsigned int enb)
static int dai_commit_write(struct dai *dai)
{
- dai->hw->dai_commit_write(dai->hw,
- daio_device_index(dai->daio.type, dai->hw), dai->ctrl_blk);
+ int idx = daio_device_index(dai->daio.type, dai->hw);
+
+ if (idx < 0)
+ return idx;
+ dai->hw->dai_commit_write(dai->hw, idx, dai->ctrl_blk);
return 0;
}
@@ -330,7 +343,7 @@ static int daio_rsc_init(struct daio *daio,
goto error1;
/* Set daio->rscl/r->ops to daio specific ones */
- if (desc->type <= DAIO_OUT_MAX) {
+ if (desc->output) {
daio->rscl.ops = daio->rscr.ops = &daio_out_rsc_ops;
} else {
switch (hw->chip_type) {
@@ -345,6 +358,7 @@ static int daio_rsc_init(struct daio *daio,
}
}
daio->type = desc->type;
+ daio->output = desc->output;
return 0;
@@ -367,7 +381,7 @@ static int dao_rsc_init(struct dao *dao,
{
struct hw *hw = mgr->mgr.hw;
unsigned int conf;
- int err;
+ int idx, err;
err = daio_rsc_init(&dao->daio, desc, mgr->mgr.hw);
if (err)
@@ -386,15 +400,18 @@ static int dao_rsc_init(struct dao *dao,
if (err)
goto error2;
- hw->daio_mgr_dsb_dao(mgr->mgr.ctrl_blk,
- daio_device_index(dao->daio.type, hw));
+ idx = daio_device_index(dao->daio.type, hw);
+ if (idx < 0) {
+ err = idx;
+ goto error2;
+ }
+
+ hw->daio_mgr_dsb_dao(mgr->mgr.ctrl_blk, idx);
hw->daio_mgr_commit_write(hw, mgr->mgr.ctrl_blk);
conf = (desc->msr & 0x7) | (desc->passthru << 3);
- hw->daio_mgr_dao_init(mgr->mgr.ctrl_blk,
- daio_device_index(dao->daio.type, hw), conf);
- hw->daio_mgr_enb_dao(mgr->mgr.ctrl_blk,
- daio_device_index(dao->daio.type, hw));
+ hw->daio_mgr_dao_init(hw, mgr->mgr.ctrl_blk, idx, conf);
+ hw->daio_mgr_enb_dao(mgr->mgr.ctrl_blk, idx);
hw->daio_mgr_commit_write(hw, mgr->mgr.ctrl_blk);
return 0;
@@ -434,6 +451,7 @@ static int dao_rsc_reinit(struct dao *dao, const struct dao_desc *desc)
dsc.type = dao->daio.type;
dsc.msr = desc->msr;
dsc.passthru = desc->passthru;
+ dsc.output = dao->daio.output;
dao_rsc_uninit(dao);
return dao_rsc_init(dao, &dsc, mgr);
}
@@ -442,7 +460,7 @@ static int dai_rsc_init(struct dai *dai,
const struct daio_desc *desc,
struct daio_mgr *mgr)
{
- int err;
+ int idx, err;
struct hw *hw = mgr->mgr.hw;
unsigned int rsr, msr;
@@ -456,6 +474,12 @@ static int dai_rsc_init(struct dai *dai,
if (err)
goto error1;
+ idx = daio_device_index(dai->daio.type, dai->hw);
+ if (idx < 0) {
+ err = idx;
+ goto error1;
+ }
+
for (rsr = 0, msr = desc->msr; msr > 1; msr >>= 1)
rsr++;
@@ -464,8 +488,7 @@ static int dai_rsc_init(struct dai *dai,
/* default to disabling control of a SRC */
hw->dai_srt_set_ec(dai->ctrl_blk, 0);
hw->dai_srt_set_et(dai->ctrl_blk, 0); /* default to disabling SRT */
- hw->dai_commit_write(hw,
- daio_device_index(dai->daio.type, dai->hw), dai->ctrl_blk);
+ hw->dai_commit_write(hw, idx, dai->ctrl_blk);
return 0;
@@ -519,7 +542,7 @@ static int get_daio_rsc(struct daio_mgr *mgr,
err = -ENOMEM;
/* Allocate mem for daio resource */
- if (desc->type <= DAIO_OUT_MAX) {
+ if (desc->output) {
struct dao *dao = kzalloc(sizeof(*dao), GFP_KERNEL);
if (!dao)
goto error;
@@ -566,7 +589,7 @@ static int put_daio_rsc(struct daio_mgr *mgr, struct daio *daio)
daio_mgr_put_rsc(&mgr->mgr, daio->type);
}
- if (daio->type <= DAIO_OUT_MAX) {
+ if (daio->output) {
dao_rsc_uninit(container_of(daio, struct dao, daio));
kfree(container_of(daio, struct dao, daio));
} else {
@@ -580,28 +603,28 @@ static int put_daio_rsc(struct daio_mgr *mgr, struct daio *daio)
static int daio_mgr_enb_daio(struct daio_mgr *mgr, struct daio *daio)
{
struct hw *hw = mgr->mgr.hw;
-
- if (DAIO_OUT_MAX >= daio->type) {
- hw->daio_mgr_enb_dao(mgr->mgr.ctrl_blk,
- daio_device_index(daio->type, hw));
- } else {
- hw->daio_mgr_enb_dai(mgr->mgr.ctrl_blk,
- daio_device_index(daio->type, hw));
- }
+ int idx = daio_device_index(daio->type, hw);
+
+ if (idx < 0)
+ return idx;
+ if (daio->output)
+ hw->daio_mgr_enb_dao(mgr->mgr.ctrl_blk, idx);
+ else
+ hw->daio_mgr_enb_dai(mgr->mgr.ctrl_blk, idx);
return 0;
}
static int daio_mgr_dsb_daio(struct daio_mgr *mgr, struct daio *daio)
{
struct hw *hw = mgr->mgr.hw;
-
- if (DAIO_OUT_MAX >= daio->type) {
- hw->daio_mgr_dsb_dao(mgr->mgr.ctrl_blk,
- daio_device_index(daio->type, hw));
- } else {
- hw->daio_mgr_dsb_dai(mgr->mgr.ctrl_blk,
- daio_device_index(daio->type, hw));
- }
+ int idx = daio_device_index(daio->type, hw);
+
+ if (idx < 0)
+ return idx;
+ if (daio->output)
+ hw->daio_mgr_dsb_dao(mgr->mgr.ctrl_blk, idx);
+ else
+ hw->daio_mgr_dsb_dai(mgr->mgr.ctrl_blk, idx);
return 0;
}
diff --git a/sound/pci/ctxfi/ctdaio.h b/sound/pci/ctxfi/ctdaio.h
index 15147fe5f74a..b337da2de8b5 100644
--- a/sound/pci/ctxfi/ctdaio.h
+++ b/sound/pci/ctxfi/ctdaio.h
@@ -43,6 +43,7 @@ struct daio {
struct rsc rscl; /* Basic resource info for left TX/RX */
struct rsc rscr; /* Basic resource info for right TX/RX */
enum DAIOTYP type;
+ unsigned char output;
};
struct dao {
@@ -91,6 +92,7 @@ struct daio_desc {
unsigned int type:4;
unsigned int msr:4;
unsigned int passthru:1;
+ unsigned int output:1;
};
struct daio_mgr {
diff --git a/sound/pci/ctxfi/cthardware.h b/sound/pci/ctxfi/cthardware.h
index 2875cec83b8f..d29b4e5b3fcc 100644
--- a/sound/pci/ctxfi/cthardware.h
+++ b/sound/pci/ctxfi/cthardware.h
@@ -167,7 +167,7 @@ struct hw {
int (*daio_mgr_dsb_dai)(void *blk, unsigned int idx);
int (*daio_mgr_enb_dao)(void *blk, unsigned int idx);
int (*daio_mgr_dsb_dao)(void *blk, unsigned int idx);
- int (*daio_mgr_dao_init)(void *blk, unsigned int idx,
+ int (*daio_mgr_dao_init)(struct hw *hw, void *blk, unsigned int idx,
unsigned int conf);
int (*daio_mgr_set_imaparc)(void *blk, unsigned int slot);
int (*daio_mgr_set_imapnxt)(void *blk, unsigned int next);
diff --git a/sound/pci/ctxfi/cthw20k1.c b/sound/pci/ctxfi/cthw20k1.c
index 42b90c9b2ee9..60cc1d14453a 100644
--- a/sound/pci/ctxfi/cthw20k1.c
+++ b/sound/pci/ctxfi/cthw20k1.c
@@ -1031,7 +1031,7 @@ static int daio_mgr_dsb_dao(void *blk, unsigned int idx)
return 0;
}
-static int daio_mgr_dao_init(void *blk, unsigned int idx, unsigned int conf)
+static int daio_mgr_dao_init(struct hw *hw __maybe_unused, void *blk, unsigned int idx, unsigned int conf)
{
struct daio_mgr_ctrl_blk *ctl = blk;
diff --git a/sound/pci/ctxfi/cthw20k2.c b/sound/pci/ctxfi/cthw20k2.c
index 4d970f25a2c6..e45c655528f5 100644
--- a/sound/pci/ctxfi/cthw20k2.c
+++ b/sound/pci/ctxfi/cthw20k2.c
@@ -985,7 +985,7 @@ static int daio_mgr_dsb_dao(void *blk, unsigned int idx)
return 0;
}
-static int daio_mgr_dao_init(void *blk, unsigned int idx, unsigned int conf)
+static int daio_mgr_dao_init(struct hw *hw __maybe_unused, void *blk, unsigned int idx, unsigned int conf)
{
struct daio_mgr_ctrl_blk *ctl = blk;
diff --git a/sound/soc/amd/acp/acp-mach-common.c b/sound/soc/amd/acp/acp-mach-common.c
index a2031e7f6142..d695b9c2b913 100644
--- a/sound/soc/amd/acp/acp-mach-common.c
+++ b/sound/soc/amd/acp/acp-mach-common.c
@@ -133,8 +133,13 @@ static int acp_card_rt5682_init(struct snd_soc_pcm_runtime *rtd)
if (drvdata->hs_codec_id != RT5682)
return -EINVAL;
- drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk");
- drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk");
+ drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk");
+ if (IS_ERR(drvdata->wclk))
+ return PTR_ERR(drvdata->wclk);
+
+ drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk");
+ if (IS_ERR(drvdata->bclk))
+ return PTR_ERR(drvdata->bclk);
ret = snd_soc_dapm_new_controls(&card->dapm, rt5682_widgets,
ARRAY_SIZE(rt5682_widgets));
@@ -375,8 +380,13 @@ static int acp_card_rt5682s_init(struct snd_soc_pcm_runtime *rtd)
return -EINVAL;
if (!drvdata->soc_mclk) {
- drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk");
- drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk");
+ drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk");
+ if (IS_ERR(drvdata->wclk))
+ return PTR_ERR(drvdata->wclk);
+
+ drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk");
+ if (IS_ERR(drvdata->bclk))
+ return PTR_ERR(drvdata->bclk);
}
ret = snd_soc_dapm_new_controls(&card->dapm, rt5682s_widgets,
diff --git a/sound/soc/amd/yc/acp6x-mach.c b/sound/soc/amd/yc/acp6x-mach.c
index 8abebf17bac5..666ac5b36e7d 100644
--- a/sound/soc/amd/yc/acp6x-mach.c
+++ b/sound/soc/amd/yc/acp6x-mach.c
@@ -703,6 +703,13 @@ static const struct dmi_system_id yc_acp_quirk_table[] = {
DMI_MATCH(DMI_BOARD_NAME, "Herbag_MDU"),
}
},
+ {
+ .driver_data = &acp6x_card,
+ .matches = {
+ DMI_MATCH(DMI_BOARD_VENDOR, "MDC"),
+ DMI_MATCH(DMI_BOARD_NAME, "Herbag2_MDU"),
+ }
+ },
{
.driver_data = &acp6x_card,
.matches = {
diff --git a/sound/soc/codecs/adau1372.c b/sound/soc/codecs/adau1372.c
index 879afeb81c42..f5b135799b14 100644
--- a/sound/soc/codecs/adau1372.c
+++ b/sound/soc/codecs/adau1372.c
@@ -799,7 +799,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable)
if (ret)
return ret;
if (adau1372->pd_gpio)
- gpiod_set_value(adau1372->pd_gpio, 0);
+ gpiod_set_value_cansleep(adau1372->pd_gpio, 0);
if (adau1372->switch_mode)
adau1372->switch_mode(adau1372->dev);
@@ -819,8 +819,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable)
ADAU1372_CLK_CTRL_PLL_EN,
0);
regcache_cache_only(adau1372->regmap, true);
- if (adau1372->pd_gpio)
- gpiod_set_value(adau1372->pd_gpio, 1);
+ gpiod_set_value_cansleep(adau1372->pd_gpio, 1);
clk_disable_unprepare(adau1372->mclk);
return ret;
}
@@ -837,7 +836,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable)
* map. No need to do any register writes to manually
* turn things off.
*/
- gpiod_set_value(adau1372->pd_gpio, 1);
+ gpiod_set_value_cansleep(adau1372->pd_gpio, 1);
regcache_mark_dirty(adau1372->regmap);
} else {
regmap_update_bits(adau1372->regmap, ADAU1372_REG_CLK_CTRL,
@@ -1004,6 +1003,17 @@ int adau1372_probe(struct device *dev, struct regmap *regmap,
regmap_write(regmap, ADAU1372_REG_OP_STAGE_MUTE, 0x0);
+ /*
+ * Set sane default values for the Output ASRC and DAC input muxes,
+ * since the power-on reset defaults are invalid "Reserved" states.
+ */
+ regmap_write(regmap, ADAU1372_REG_ASRCO_SOURCE_0_1,
+ 0x54); /* Decimator0, Decimator1 */
+ regmap_write(regmap, ADAU1372_REG_ASRCO_SOURCE_2_3,
+ 0x76); /* Decimator2, Decimator3 */
+ regmap_write(regmap, ADAU1372_REG_DAC_SOURCE,
+ 0xdc); /* Input ASRC0, Input ASRC1 */
+
regmap_write(regmap, 0x7, 0x01); /* CLOCK OUT */
return devm_snd_soc_register_component(dev, &adau1372_driver, &adau1372_dai_driver, 1);
diff --git a/sound/soc/qcom/qdsp6/q6asm.c b/sound/soc/qcom/qdsp6/q6asm.c
index 67e9ca18883c..60c81edd945d 100644
--- a/sound/soc/qcom/qdsp6/q6asm.c
+++ b/sound/soc/qcom/qdsp6/q6asm.c
@@ -654,6 +654,11 @@ static int32_t q6asm_stream_callback(struct apr_device *adev,
goto done;
}
break;
+ case ASM_DATA_CMD_EOS:
+ case ASM_DATA_CMD_READ_V2:
+ case ASM_DATA_CMD_WRITE_V2:
+ /* response as result of close stream */
+ goto done;
default:
dev_err(ac->dev, "command[0x%x] not expecting rsp\n",
result->opcode);
diff --git a/sound/soc/samsung/i2s.c b/sound/soc/samsung/i2s.c
index 140907a41a70..7bb80d73bd66 100644
--- a/sound/soc/samsung/i2s.c
+++ b/sound/soc/samsung/i2s.c
@@ -1543,9 +1543,16 @@ static int samsung_i2s_probe(struct platform_device *pdev)
goto err_disable_pm;
priv->op_clk = clk_get_parent(priv->clk_table[CLK_I2S_RCLK_SRC]);
+ ret = clk_prepare_enable(priv->op_clk);
+ if (ret < 0) {
+ priv->op_clk = NULL;
+ goto err_unregister_clk;
+ }
return 0;
+err_unregister_clk:
+ i2s_unregister_clock_provider(priv);
err_disable_pm:
pm_runtime_disable(&pdev->dev);
err_del_sec:
@@ -1568,6 +1575,7 @@ static void samsung_i2s_remove(struct platform_device *pdev)
i2s_unregister_clock_provider(priv);
i2s_delete_secondary_device(priv);
+ clk_disable_unprepare(priv->op_clk);
clk_disable_unprepare(priv->clk);
pm_runtime_put_noidle(&pdev->dev);
diff --git a/tools/include/linux/static_call_types.h b/tools/include/linux/static_call_types.h
index 5a00b8b2cf9f..cfb6ddeb292b 100644
--- a/tools/include/linux/static_call_types.h
+++ b/tools/include/linux/static_call_types.h
@@ -25,6 +25,8 @@
#define STATIC_CALL_SITE_INIT 2UL /* init section */
#define STATIC_CALL_SITE_FLAGS 3UL
+#ifndef __ASSEMBLY__
+
/*
* The static call site table needs to be created by external tooling (objtool
* or a compiler plugin).
@@ -100,4 +102,6 @@ struct static_call_key {
#endif /* CONFIG_HAVE_STATIC_CALL */
+#endif /* __ASSEMBLY__ */
+
#endif /* _STATIC_CALL_TYPES_H */
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
index ac3e418c2a96..d55fd063dc59 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
@@ -287,9 +287,9 @@ __msg("17: (b7) r0 = 0")
__msg("18: (95) exit")
__msg("returning from callee:")
__msg("to caller at 9:")
-__msg("frame 0: propagating r1,r4")
+__msg("frame 0: propagating r1,r3,r4")
__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r4 stack= before 18: (95) exit")
+__msg("mark_precise: frame0: regs=r1,r3,r4 stack= before 18: (95) exit")
__msg("from 18 to 9: safe")
__naked int callback_result_precise(void)
{
@@ -419,9 +419,9 @@ __msg("to caller at 9:")
/* r1, r4 are always precise for bpf_loop(),
* r6 was marked before backtracking to callback body.
*/
-__msg("frame 0: propagating r1,r4,r6")
+__msg("frame 0: propagating r1,r3,r4,r6")
__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r4,r6 stack= before 16: (95) exit")
+__msg("mark_precise: frame0: regs=r1,r3,r4,r6 stack= before 16: (95) exit")
__msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0")
__msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop")
__msg("mark_precise: frame0: parent state regs= stack=:")
@@ -575,9 +575,9 @@ __msg("to caller at 10:")
/* r1, r4 are always precise for bpf_loop(),
* fp-8 was marked before backtracking to callback body.
*/
-__msg("frame 0: propagating r1,r4,fp-8")
+__msg("frame 0: propagating r1,r3,r4,fp-8")
__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r4 stack=-8 before 18: (95) exit")
+__msg("mark_precise: frame0: regs=r1,r3,r4 stack=-8 before 18: (95) exit")
__msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0")
__msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181")
__msg("mark_precise: frame0: parent state regs= stack=:")
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index ed0a4721d8fd..37d2723062a6 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -949,7 +949,7 @@ __bpf_kfunc int bpf_kfunc_call_kernel_bind(struct addr_args *args)
goto out;
}
- err = kernel_bind(sock, (struct sockaddr *)&args->addr, args->addrlen);
+ err = kernel_bind(sock, (struct sockaddr_unsized *)&args->addr, args->addrlen);
out:
mutex_unlock(&sock_lock);
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index ffadc5ee8e04..a59a4446b846 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -138,6 +138,10 @@ static void __kvm_gmem_invalidate_begin(struct kvm_gmem *gmem, pgoff_t start,
}
flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
+
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+ kvm_arch_gmem_invalidate_range(kvm, &gfn_range);
+#endif
}
if (flush)
@@ -560,15 +564,16 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
}
int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
- unsigned int fd, loff_t offset)
+ unsigned int fd, u64 offset)
{
- loff_t size = slot->npages << PAGE_SHIFT;
+ u64 size = slot->npages << PAGE_SHIFT;
unsigned long start, end;
struct kvm_gmem *gmem;
struct inode *inode;
struct file *file;
int r = -EINVAL;
+ BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset));
BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff));
file = fget(fd);
@@ -584,8 +589,7 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
inode = file_inode(file);
- if (offset < 0 || !PAGE_ALIGNED(offset) ||
- offset + size > i_size_read(inode))
+ if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode))
goto err;
filemap_invalidate_lock(inode->i_mapping);
diff --git a/virt/kvm/kvm_mm.h b/virt/kvm/kvm_mm.h
index 31defb08ccba..0e68675a8a63 100644
--- a/virt/kvm/kvm_mm.h
+++ b/virt/kvm/kvm_mm.h
@@ -3,6 +3,9 @@
#ifndef __KVM_MM_H__
#define __KVM_MM_H__ 1
+#include <linux/kvm.h>
+#include <linux/kvm_types.h>
+
/*
* Architectures can choose whether to use an rwlock or spinlock
* for the mmu_lock. These macros, for use in common code
@@ -71,7 +74,7 @@ static inline void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm,
void kvm_gmem_init(struct module *module);
int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args);
int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
- unsigned int fd, loff_t offset);
+ unsigned int fd, u64 offset);
void kvm_gmem_unbind(struct kvm_memory_slot *slot);
#else
static inline void kvm_gmem_init(struct module *module)
@@ -81,7 +84,7 @@ static inline void kvm_gmem_init(struct module *module)
static inline int kvm_gmem_bind(struct kvm *kvm,
struct kvm_memory_slot *slot,
- unsigned int fd, loff_t offset)
+ unsigned int fd, u64 offset)
{
WARN_ON_ONCE(1);
return -EIO;
^ permalink raw reply [flat|nested] 2+ messages in thread