* Re: Linux 6.12.112
2026-10-03 10:53 Linux 6.12.112 Greg Kroah-Hartman
@ 2026-10-03 10:53 ` Greg Kroah-Hartman
0 siblings, 0 replies; 2+ messages in thread
From: Greg Kroah-Hartman @ 2026-10-03 10:53 UTC (permalink / raw)
To: linux-kernel, akpm, torvalds, stable; +Cc: lwn, jslaby, Greg Kroah-Hartman
diff --git a/Documentation/ABI/testing/sysfs-devices-platform-trackpoint b/Documentation/ABI/testing/sysfs-devices-platform-trackpoint
index df11901a6b3d..7954434b0434 100644
--- a/Documentation/ABI/testing/sysfs-devices-platform-trackpoint
+++ b/Documentation/ABI/testing/sysfs-devices-platform-trackpoint
@@ -5,7 +5,7 @@ Contact: linux-input@vger.kernel.org
Description:
(RW) Trackpoint sensitivity.
-What: /sys/devices/platform/i8042/.../intertia
+What: /sys/devices/platform/i8042/.../inertia
Date: Aug, 2005
KernelVersion: 2.6.14
Contact: linux-input@vger.kernel.org
diff --git a/Documentation/ABI/testing/sysfs-fs-erofs b/Documentation/ABI/testing/sysfs-fs-erofs
index d76de22b6ef3..2aecd4b83335 100644
--- a/Documentation/ABI/testing/sysfs-fs-erofs
+++ b/Documentation/ABI/testing/sysfs-fs-erofs
@@ -5,7 +5,7 @@ Description: Shows all enabled kernel features.
Supported features:
zero_padding, compr_cfgs, big_pcluster, chunked_file,
device_table, compr_head2, sb_chksum, ztailpacking,
- dedupe, fragments.
+ dedupe, fragments, xattr_prefixes.
What: /sys/fs/erofs/<disk>/sync_decompress
Date: November 2021
diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst
index 15bcd1b4003a..4c43d38d5d70 100644
--- a/Documentation/arch/arm64/booting.rst
+++ b/Documentation/arch/arm64/booting.rst
@@ -400,6 +400,7 @@ Before jumping into the kernel, the following conditions must be met:
- HDFGWTR2_EL2.nPMICNTR_EL0 (bit 2) must be initialised to 0b1.
- HDFGWTR2_EL2.nPMICFILTR_EL0 (bit 3) must be initialised to 0b1.
- HDFGWTR2_EL2.nPMUACR_EL1 (bit 4) must be initialised to 0b1.
+ - HDFGWTR2_EL2.nPMZR_EL0 (bit 21) must be initialised to 0b1.
For CPUs with Memory Copy and Memory Set instructions (FEAT_MOPS):
diff --git a/Documentation/arch/s390/pci.rst b/Documentation/arch/s390/pci.rst
index d5755484d8e7..8dc09634163b 100644
--- a/Documentation/arch/s390/pci.rst
+++ b/Documentation/arch/s390/pci.rst
@@ -6,6 +6,7 @@ S/390 PCI
Authors:
- Pierre Morel
+ - Niklas Schnelle
Copyright, IBM Corp. 2020
@@ -27,14 +28,16 @@ Command line parameters
debugfs entries
---------------
-The S/390 debug feature (s390dbf) generates views to hold various debug results in sysfs directories of the form:
+The S/390 debug feature (s390dbf) generates views to hold various debug results
+in sysfs directories of the form:
* /sys/kernel/debug/s390dbf/pci_*/
For example:
- /sys/kernel/debug/s390dbf/pci_msg/sprintf
- Holds messages from the processing of PCI events, like machine check handling
+
+ holds messages from the processing of PCI events, like machine check handling
and setting of global functionality, like UID checking.
Change the level of logging to be more or less verbose by piping
@@ -47,87 +50,134 @@ Sysfs entries
Entries specific to zPCI functions and entries that hold zPCI information.
-* /sys/bus/pci/slots/XXXXXXXX
+* /sys/bus/pci/slots/XXXXXXXX:
- The slot entries are set up using the function identifier (FID) of the
- PCI function. The format depicted as XXXXXXXX above is 8 hexadecimal digits
- with 0 padding and lower case hexadecimal digits.
+ The slot entries are set up using the function identifier (FID) of the PCI
+ function as slot name. The format depicted as XXXXXXXX above is 8 hexadecimal
+ digits with 0 padding and lower case hexadecimal digits.
- /sys/bus/pci/slots/XXXXXXXX/power
A physical function that currently supports a virtual function cannot be
powered off until all virtual functions are removed with:
- echo 0 > /sys/bus/pci/devices/XXXX:XX:XX.X/sriov_numvf
+ echo 0 > /sys/bus/pci/devices/DDDD:BB:dd.f/sriov_numvfs
-* /sys/bus/pci/devices/XXXX:XX:XX.X/
+* /sys/bus/pci/devices/DDDD:BB:dd.f/:
- - function_id
- A zPCI function identifier that uniquely identifies the function in the Z server.
+ - function_id:
+ The zPCI function identifier (FID) is a 32-bit hexadecimal value that
+ uniquely identifies the PCI function. Unless the hypervisor provides
+ a virtual FID e.g. on KVM this identifier is unique across the machine even
+ between different partitions.
- - function_handle
- Low-level identifier used for a configured PCI function.
- It might be useful for debugging.
+ - function_handle:
+ This 32-bit hexadecimal value is a low-level identifier used for a PCI
+ function. Note that the function handle may be changed and become invalid
+ on PCI events and when enabling/disabling the PCI function.
- - pchid
- Model-dependent location of the I/O adapter.
+ - pchid:
+ This 16-bit hexadecimal value encodes a model-dependent location for
+ the PCI function.
- - pfgid
- PCI function group ID, functions that share identical functionality
+ - pfgid:
+ PCI function group ID; functions that share identical functionality
use a common identifier.
A PCI group defines interrupts, IOMMU, IOTLB, and DMA specifics.
- - vfn
+ - vfn:
The virtual function number, from 1 to N for virtual functions,
0 for physical functions.
- - pft
- The PCI function type
-
- - port
- The port corresponds to the physical port the function is attached to.
- It also gives an indication of the physical function a virtual function
- is attached to.
-
- - uid
- The user identifier (UID) may be defined as part of the machine
- configuration or the z/VM or KVM guest configuration. If the accompanying
- uid_is_unique attribute is 1 the platform guarantees that the UID is unique
- within that instance and no devices with the same UID can be attached
- during the lifetime of the system.
-
- - uid_is_unique
- Indicates whether the user identifier (UID) is guaranteed to be and remain
- unique within this Linux instance.
-
- - pfip/segmentX
+ - pft:
+ The PCI function type is an s390-specific type attribute. It indicates
+ a more general, usage oriented, type than PCI Specification
+ class/vendor/device identifiers. That is PCI functions with the same pft
+ value may be backed by different hardware implementations. At the same time
+ apart from unclassified functions (pft is 0x00) the same pft value
+ generally implies a similar usage model. At the same time the same
+ PCI hardware device may appear with different pft values when in a
+ different usage model. For example NETD and NETH VFs may be implemented
+ by the same PCI hardware device but in NETD the parent Physical Function
+ is user managed while with NETH it is platform managed.
+
+ Currently the following PFT values are defined:
+
+ - 0x00 (UNC): Unclassified
+ - 0x02 (ROCE): RoCE Express
+ - 0x05 (ISM): Internal Shared Memory
+ - 0x0a (ROC2): RoCE Express 2
+ - 0x0b (NVMe): NVMe
+ - 0x0c (NETH): Network Express hybrid
+ - 0x0d (CNW): Cloud Network Adapter
+ - 0x0f (NETD): Network Express direct
+
+ - port:
+ The port is a decimal value corresponding to the physical port the function
+ is attached to. Virtual Functions (VFs) share the port with their parent
+ Physical Function (PF). A value of 0 indicates that the port attribute is
+ not applicable for that PCI function type.
+
+ - uid:
+ The user-defined identifier (UID) for a PCI function is a 32-bit
+ hexadecimal value. It is defined on a per instance basis as part of the
+ partition, KVM guest, or z/VM guest configuration. If UID Checking is
+ enabled the platform ensures that the UID is unique within that instance
+ and no two PCI functions with the same UID will be visible to the instance.
+
+ Independent of this guarantee and unlike the function ID (FID) the UID may
+ be the same in different partitions within the same machine. This allows to
+ create PCI configurations in multiple partitions to be identical in the
+ UID-namespace.
+
+ - uid_is_unique:
+ A 0 or 1 flag indicating whether the user-defined identifier (UID) is
+ guaranteed to be and remain unique within this Linux instance. This
+ platform feature is called UID Checking.
+
+ - pfip/segmentX:
The segments determine the isolation of a function.
They correspond to the physical path to the function.
The more the segments are different, the more the functions are isolated.
+ - fidparm:
+ Contains an 8-bit-per-PCI function parameter field in hexadecimal provided
+ by the platform. The meaning of this field is PCI function type specific.
+ For NETH VFs a value of 0x01 indicates that the function supports
+ promiscuous mode.
+
+* /sys/firmware/clp/uid_checking:
+
+ In addition to the per-device uid_is_unique attribute this presents a
+ global indication of whether UID Checking is enabled. This allows users
+ to check for UID Checking even when no PCI functions are configured.
+
Enumeration and hotplug
=======================
The PCI address consists of four parts: domain, bus, device and function,
-and is of this form: DDDD:BB:dd.f
+and is of this form: DDDD:BB:dd.f.
-* When not using multi-functions (norid is set, or the firmware does not
- support multi-functions):
+* For a PCI function for which the platform does not expose the RID, the
+ pci=norid kernel parameter is used, or a so-called isolated Virtual Function
+ which does have RID information but is used without its parent Physical
+ Function being part of the same PCI configuration:
- There is only one function per domain.
- - The domain is set from the zPCI function's UID as defined during the
- LPAR creation.
+ - The domain is set from the zPCI function's UID if UID Checking is on;
+ otherwise the domain ID is generated dynamically and is not stable
+ across reboots or hot plug.
-* When using multi-functions (norid parameter is not set),
- zPCI functions are addressed differently:
+* For a PCI function for which the platform exposes the RID and which
+ is not an Isolated Virtual Function:
- There is still only one bus per domain.
- - There can be up to 256 functions per bus.
+ - There can be up to 256 PCI functions per bus.
- - The domain part of the address of all functions for
- a multi-Function device is set from the zPCI function's UID as defined
- in the LPAR creation for the function zero.
+ - The domain part of the address of all functions within the same topology is
+ that of the configured PCI function with the lowest devfn within that
+ topology.
- - New functions will only be ready for use after the function zero
- (the function with devfn 0) has been enumerated.
+ - Virtual Functions generated by an SR-IOV capable Physical Function only
+ become visible once SR-IOV is enabled.
diff --git a/Documentation/hwmon/emc1403.rst b/Documentation/hwmon/emc1403.rst
index ebf2435a76a6..57f833b1a800 100644
--- a/Documentation/hwmon/emc1403.rst
+++ b/Documentation/hwmon/emc1403.rst
@@ -71,10 +71,10 @@ and EMC14x8 support eight sensors (one internal, seven external).
The chips implement three limits for each sensor: low (tempX_min), high
(tempX_max) and critical (tempX_crit.) The chips also implement an
-hysteresis mechanism which applies to high and critical limits. The relative
-difference is stored in a single register on the chip, which means that the
-relative difference between the limit and its hysteresis is always the same
-for high and critical limits.
+hysteresis mechanism which applies to all limits. The relative difference
+is stored in a single register on the chip, which means that the relative
+difference between the limit and its hysteresis is always the same for
+all three limits.
This implementation detail implies the following:
diff --git a/Documentation/networking/ip-sysctl.rst b/Documentation/networking/ip-sysctl.rst
index dcbb6f6caf6d..eb2c136be63d 100644
--- a/Documentation/networking/ip-sysctl.rst
+++ b/Documentation/networking/ip-sysctl.rst
@@ -735,6 +735,8 @@ tcp_rmem - vector of 3 INTEGERs: min, default, max
case this value is ignored.
Default: between 131072 and 6MB, depending on RAM size.
+ Each of the three values cannot be set below 4096.
+
tcp_sack - BOOLEAN
Enable select acknowledgments (SACKS).
diff --git a/Documentation/networking/net_cachelines/inet_connection_sock.rst b/Documentation/networking/net_cachelines/inet_connection_sock.rst
index 7a911dc95652..ffae3c419649 100644
--- a/Documentation/networking/net_cachelines/inet_connection_sock.rst
+++ b/Documentation/networking/net_cachelines/inet_connection_sock.rst
@@ -14,6 +14,7 @@ struct_inet_bind2_bucket icsk_bind2_hash read_mostly -
unsigned_long icsk_timeout read_mostly - inet_csk_reset_xmit_timer,tcp_connect
struct_timer_list icsk_retransmit_timer read_mostly - inet_csk_reset_xmit_timer,tcp_connect
struct_timer_list icsk_delack_timer read_mostly - inet_csk_reset_xmit_timer,tcp_connect
+struct_timer_list icsk_keepalive_timer - -
u32 icsk_rto read_write - tcp_cwnd_validate,tcp_schedule_loss_probe,tcp_connect_init,tcp_connect,tcp_write_xmit,tcp_push_one
u32 icsk_rto_min - -
u32 icsk_delack_max - -
diff --git a/Documentation/networking/net_cachelines/inet_sock.rst b/Documentation/networking/net_cachelines/inet_sock.rst
index 595d7ef5fc8b..2485ef9cdd7d 100644
--- a/Documentation/networking/net_cachelines/inet_sock.rst
+++ b/Documentation/networking/net_cachelines/inet_sock.rst
@@ -9,6 +9,7 @@ Type Name fastpath_tx_access fastpath_rx_ac
..struct ..inet_sock
struct_sock sk read_mostly read_mostly tcp_init_buffer_space,tcp_init_transfer,tcp_finish_connect,tcp_connect,tcp_send_rcvq,tcp_send_syn_data
struct_ipv6_pinfo* pinet6 - -
+struct_ipv6_fl_socklist* ipv6_fl_list read_mostly - tcp_v6_connect,__ip6_datagram_connect,udpv6_sendmsg,rawv6_sendmsg
be16 inet_sport read_mostly - __tcp_transmit_skb
be32 inet_daddr read_mostly - ip_select_ident_segs
be32 inet_rcv_saddr - -
diff --git a/Documentation/trace/eprobetrace.rst b/Documentation/trace/eprobetrace.rst
new file mode 100644
index 000000000000..89b5157cfab8
--- /dev/null
+++ b/Documentation/trace/eprobetrace.rst
@@ -0,0 +1,269 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================================
+Eprobe - Event-based Probe Tracing
+==================================
+
+:Author: Steven Rostedt <rostedt@goodmis.org>
+
+- Written for v6.17
+
+Overview
+========
+
+Eprobes are dynamic events that are placed on existing events to either
+dereference a field that is a pointer, or simply to limit what fields are
+recorded in the trace event.
+
+Eprobes depend on kprobe events so to enable this feature, build your kernel
+with CONFIG_EPROBE_EVENTS=y.
+
+Eprobes are created via the /sys/kernel/tracing/dynamic_events file.
+
+Synopsis of eprobe_events
+-------------------------
+::
+
+ e[:[EGRP/][EEVENT]] GRP.EVENT [FETCHARGS] : Set a probe
+ -:[EGRP/][EEVENT] : Clear a probe
+
+ EGRP : Group name of the new event. If omitted, use "eprobes" for it.
+ EEVENT : Event name. If omitted, the event name is generated and will
+ be the same event name as the event it attached to.
+ GRP : Group name of the event to attach to.
+ EVENT : Event name of the event to attach to.
+
+ FETCHARGS : Arguments. Each probe can have up to 128 args.
+ $FIELD : Fetch the value of the event field called FIELD.
+ @ADDR : Fetch memory at ADDR (ADDR should be in kernel)
+ @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
+ $comm : Fetch current task comm.
+ +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4)
+ \IMM : Store an immediate value to the argument.
+ NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
+ FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
+ (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
+ (x8/x16/x32/x64), VFS layer common type(%pd/%pD), "char",
+ "string", "ustring", "symbol", "symstr" and "bitfield" are
+ supported.
+
+Types
+-----
+The FETCHARGS above is very similar to the kprobe events as described in
+Documentation/trace/kprobetrace.rst.
+
+The difference between eprobes and kprobes FETCHARGS is that eprobes has a
+$FIELD command that returns the content of the event field of the event
+that is attached. Eprobes do not have access to registers, stacks and function
+arguments that kprobes has.
+
+If a field argument is a pointer, it may be dereferenced just like a memory
+address using the FETCHARGS syntax.
+
+
+Attaching to dynamic events
+---------------------------
+
+Eprobes may attach to dynamic events as well as to normal events. It may
+attach to a kprobe event, a synthetic event or a fprobe event. This is useful
+if the type of a field needs to be changed. See Example 2 below.
+
+Usage examples
+==============
+
+Example 1
+---------
+
+The basic usage of eprobes is to limit the data that is being recorded into
+the tracing buffer. For example, a common event to trace is the sched_switch
+trace event. That has a format of::
+
+ field:unsigned short common_type; offset:0; size:2; signed:0;
+ field:unsigned char common_flags; offset:2; size:1; signed:0;
+ field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
+ field:int common_pid; offset:4; size:4; signed:1;
+
+ field:char prev_comm[16]; offset:8; size:16; signed:0;
+ field:pid_t prev_pid; offset:24; size:4; signed:1;
+ field:int prev_prio; offset:28; size:4; signed:1;
+ field:long prev_state; offset:32; size:8; signed:1;
+ field:char next_comm[16]; offset:40; size:16; signed:0;
+ field:pid_t next_pid; offset:56; size:4; signed:1;
+ field:int next_prio; offset:60; size:4; signed:1;
+
+The first four fields are common to all events and can not be limited. But the
+rest of the event has 60 bytes of information. It records the names of the
+previous and next tasks being scheduled out and in, as well as their pids and
+priorities. It also records the state of the previous task. If only the pids
+of the tasks are of interest, why waste the ring buffer with all the other
+fields?
+
+An eprobe can limit what gets recorded. Note, it does not help in performance,
+as all the fields are recorded in a temporary buffer to process the eprobe.
+::
+
+ # echo 'e:sched/switch sched.sched_switch prev=$prev_pid:u32 next=$next_pid:u32' >> /sys/kernel/tracing/dynamic_events
+ # echo 1 > /sys/kernel/tracing/events/sched/switch/enable
+ # cat /sys/kernel/tracing/trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 2721/2721 #P:8
+ #
+ # _-----=> irqs-off/BH-disabled
+ # / _----=> need-resched
+ # | / _---=> hardirq/softirq
+ # || / _--=> preempt-depth
+ # ||| / _-=> migrate-disable
+ # |||| / delay
+ # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
+ # | | | ||||| | |
+ sshd-session-1082 [004] d..4. 5041.239906: switch: (sched.sched_switch) prev=1082 next=0
+ bash-1085 [001] d..4. 5041.240198: switch: (sched.sched_switch) prev=1085 next=141
+ kworker/u34:5-141 [001] d..4. 5041.240259: switch: (sched.sched_switch) prev=141 next=1085
+ <idle>-0 [004] d..4. 5041.240354: switch: (sched.sched_switch) prev=0 next=1082
+ bash-1085 [001] d..4. 5041.240385: switch: (sched.sched_switch) prev=1085 next=141
+ kworker/u34:5-141 [001] d..4. 5041.240410: switch: (sched.sched_switch) prev=141 next=1085
+ bash-1085 [001] d..4. 5041.240478: switch: (sched.sched_switch) prev=1085 next=0
+ sshd-session-1082 [004] d..4. 5041.240526: switch: (sched.sched_switch) prev=1082 next=0
+ <idle>-0 [001] d..4. 5041.247524: switch: (sched.sched_switch) prev=0 next=90
+ <idle>-0 [002] d..4. 5041.247545: switch: (sched.sched_switch) prev=0 next=16
+ kworker/1:1-90 [001] d..4. 5041.247580: switch: (sched.sched_switch) prev=90 next=0
+ rcu_sched-16 [002] d..4. 5041.247591: switch: (sched.sched_switch) prev=16 next=0
+ <idle>-0 [002] d..4. 5041.257536: switch: (sched.sched_switch) prev=0 next=16
+ rcu_sched-16 [002] d..4. 5041.257573: switch: (sched.sched_switch) prev=16 next=0
+
+Note, without adding the "u32" after the prev_pid and next_pid, the values
+would default showing in hexadecimal.
+
+Example 2
+---------
+
+If a specific system call is to be recorded but the syscalls events are not
+enabled, the raw_syscalls can still be used (syscalls are system call
+events are not normal events, but are created from the raw_syscalls events
+within the kernel). In order to trace the openat system call, one can create
+an event probe on top of the raw_syscalls event:
+::
+
+ # cd /sys/kernel/tracing
+ # cat events/raw_syscalls/sys_enter/format
+ name: sys_enter
+ ID: 395
+ format:
+ field:unsigned short common_type; offset:0; size:2; signed:0;
+ field:unsigned char common_flags; offset:2; size:1; signed:0;
+ field:unsigned char common_preempt_count; offset:3; size:1; signed:0;
+ field:int common_pid; offset:4; size:4; signed:1;
+
+ field:long id; offset:8; size:8; signed:1;
+ field:unsigned long args[6]; offset:16; size:48; signed:0;
+
+ print fmt: "NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]
+
+From the source code, the sys_openat() has:
+::
+
+ int sys_openat(int dirfd, const char *path, int flags, mode_t mode)
+ {
+ return my_syscall4(__NR_openat, dirfd, path, flags, mode);
+ }
+
+The path is the second parameter, and that is what is wanted.
+::
+
+ # echo 'e:openat raw_syscalls.sys_enter nr=$id filename=+8($args):ustring' >> dynamic_events
+
+This is being run on x86_64 where the word size is 8 bytes and the openat
+system call __NR_openat is set at 257.
+::
+
+ # echo 'nr == 257' > events/eprobes/openat/filter
+
+Now enable the event and look at the trace.
+::
+
+ # echo 1 > events/eprobes/openat/enable
+ # cat trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 4/4 #P:8
+ #
+ # _-----=> irqs-off/BH-disabled
+ # / _----=> need-resched
+ # | / _---=> hardirq/softirq
+ # || / _--=> preempt-depth
+ # ||| / _-=> migrate-disable
+ # |||| / delay
+ # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
+ # | | | ||||| | |
+ cat-1298 [003] ...2. 2060.875970: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+ cat-1298 [003] ...2. 2060.876197: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+ cat-1298 [003] ...2. 2060.879126: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+ cat-1298 [003] ...2. 2060.879639: openat: (raw_syscalls.sys_enter) nr=0x101 filename=(fault)
+
+The filename shows "(fault)". This is likely because the filename has not been
+pulled into memory yet and currently trace events cannot fault in memory that
+is not present. When an eprobe tries to read memory that has not been faulted
+in yet, it will show the "(fault)" text.
+
+To get around this, as the kernel will likely pull in this filename and make
+it present, attaching it to a synthetic event that can pass the address of the
+filename from the entry of the event to the end of the event, this can be used
+to show the filename when the system call returns.
+
+Remove the old eprobe::
+
+ # echo 1 > events/eprobes/openat/enable
+ # echo '-:openat' >> dynamic_events
+
+This time make an eprobe where the address of the filename is saved::
+
+ # echo 'e:openat_start raw_syscalls.sys_enter nr=$id filename=+8($args):x64' >> dynamic_events
+
+Create a synthetic event that passes the address of the filename to the
+end of the event::
+
+ # echo 's:filename u64 file' >> dynamic_events
+ # echo 'hist:keys=common_pid:f=filename if nr == 257' > events/eprobes/openat_start/trigger
+ # echo 'hist:keys=common_pid:file=$f:onmatch(eprobes.openat_start).trace(filename,$file) if id == 257' > events/raw_syscalls/sys_exit/trigger
+
+Now that the address of the filename has been passed to the end of the
+system call, create another eprobe to attach to the exit event to show the
+string::
+
+ # echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
+ # echo 1 > events/eprobes/openat/enable
+ # cat trace
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 4/4 #P:8
+ #
+ # _-----=> irqs-off/BH-disabled
+ # / _----=> need-resched
+ # | / _---=> hardirq/softirq
+ # || / _--=> preempt-depth
+ # ||| / _-=> migrate-disable
+ # |||| / delay
+ # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
+ # | | | ||||| | |
+ cat-1331 [001] ...5. 2944.787977: openat: (synthetic.filename) filename="/etc/ld.so.cache"
+ cat-1331 [001] ...5. 2944.788480: openat: (synthetic.filename) filename="/lib/x86_64-linux-gnu/libc.so.6"
+ cat-1331 [001] ...5. 2944.793426: openat: (synthetic.filename) filename="/usr/lib/locale/locale-archive"
+ cat-1331 [001] ...5. 2944.831362: openat: (synthetic.filename) filename="trace"
+
+Example 3
+---------
+
+If syscall trace events are available, the above would not need the first
+eprobe, but it would still need the last one::
+
+ # echo 's:filename u64 file' >> dynamic_events
+ # echo 'hist:keys=common_pid:f=filename' > events/syscalls/sys_enter_openat/trigger
+ # echo 'hist:keys=common_pid:file=$f:onmatch(syscalls.sys_enter_openat).trace(filename,$file)' > events/syscalls/sys_exit_openat/trigger
+ # echo 'e:openat synthetic.filename filename=+0($file):ustring' >> dynamic_events
+ # echo 1 > events/eprobes/openat/enable
+
+And this would produce the same result as Example 2.
diff --git a/Documentation/trace/index.rst b/Documentation/trace/index.rst
index e9bcb9d9f7f3..f8a4682478a6 100644
--- a/Documentation/trace/index.rst
+++ b/Documentation/trace/index.rst
@@ -36,6 +36,7 @@ the Linux kernel.
kprobes
kprobetrace
fprobetrace
+ eprobetrace
fprobe
ring-buffer-design
diff --git a/Makefile b/Makefile
index 6f7b25c6461c..24d86c14edd8 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
VERSION = 6
PATCHLEVEL = 12
-SUBLEVEL = 111
+SUBLEVEL = 112
EXTRAVERSION =
NAME = Baby Opossum Posse
diff --git a/arch/arm/mach-socfpga/Kconfig b/arch/arm/mach-socfpga/Kconfig
index eb72c240c248..528c5c1368c3 100644
--- a/arch/arm/mach-socfpga/Kconfig
+++ b/arch/arm/mach-socfpga/Kconfig
@@ -16,7 +16,7 @@ menuconfig ARCH_INTEL_SOCFPGA
select ARM_ERRATA_775420
select PL310_ERRATA_588369
select PL310_ERRATA_727915
- select PL310_ERRATA_753970 if PL310
+ select PL310_ERRATA_753970
select PL310_ERRATA_769419
select RESET_CONTROLLER
diff --git a/arch/arm64/boot/dts/renesas/r8a779f0.dtsi b/arch/arm64/boot/dts/renesas/r8a779f0.dtsi
index 9629adb47d99..900dc183bd6d 100644
--- a/arch/arm64/boot/dts/renesas/r8a779f0.dtsi
+++ b/arch/arm64/boot/dts/renesas/r8a779f0.dtsi
@@ -876,6 +876,7 @@ ufs: ufs@e6860000 {
clocks = <&cpg CPG_MOD 1514>, <&ufs30_clk>;
clock-names = "fck", "ref_clk";
freq-table-hz = <200000000 200000000>, <38400000 38400000>;
+ lanes-per-direction = <1>;
power-domains = <&sysc R8A779F0_PD_ALWAYS_ON>;
resets = <&cpg 1514>;
status = "disabled";
diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index 9d1c88536ee4..e05218fbd58b 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -274,6 +274,7 @@
b.lt .Lskip_fgt2_\@
mov x0, xzr
+ mov x2, xzr
mrs x1, id_aa64dfr0_el1
ubfx x1, x1, #ID_AA64DFR0_EL1_PMUVer_SHIFT, #4
cmp x1, #ID_AA64DFR0_EL1_PMUVer_V3P9
@@ -282,9 +283,14 @@
orr x0, x0, #HDFGRTR2_EL2_nPMICNTR_EL0
orr x0, x0, #HDFGRTR2_EL2_nPMICFILTR_EL0
orr x0, x0, #HDFGRTR2_EL2_nPMUACR_EL1
+ orr x2, x2, #HDFGWTR2_EL2_nPMICNTR_EL0
+ orr x2, x2, #HDFGWTR2_EL2_nPMICFILTR_EL0
+ orr x2, x2, #HDFGWTR2_EL2_nPMUACR_EL1
+ /* PMZR_EL0 is write-only, so it has no read trap to disable */
+ orr x2, x2, #HDFGWTR2_EL2_nPMZR_EL0
.Lskip_pmuv3p9_\@:
msr_s SYS_HDFGRTR2_EL2, x0
- msr_s SYS_HDFGWTR2_EL2, x0
+ msr_s SYS_HDFGWTR2_EL2, x2
msr_s SYS_HFGRTR2_EL2, xzr
msr_s SYS_HFGWTR2_EL2, xzr
msr_s SYS_HFGITR2_EL2, xzr
diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
index 46bd37707e08..16a423101bdc 100644
--- a/arch/arm64/include/asm/io.h
+++ b/arch/arm64/include/asm/io.h
@@ -282,7 +282,8 @@ static inline void __iomem *ioremap_prot(phys_addr_t phys, size_t size,
pgprot_t prot;
pteval_t user_prot_val = pgprot_val(__pgprot(user_prot));
- if (WARN_ON_ONCE(!(user_prot_val & PTE_USER)))
+ /* Reject PROT_NONE and exec-only */
+ if (!(user_prot_val & PTE_USER))
return NULL;
prot = __pgprot_modify(PAGE_KERNEL, PTE_ATTRINDX_MASK,
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 9abcc8ef3087..8cf4068ce1b5 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -77,7 +77,7 @@ __percpu_##name##_case_##sz(void *ptr, unsigned long val) \
" stxr" #sfx "\t%w[loop], %" #w "[tmp], %[ptr]\n" \
" cbnz %w[loop], 1b", \
/* LSE atomics */ \
- #op_lse "\t%" #w "[val], %[ptr]\n" \
+ #op_lse #sfx "\t%" #w "[val], %" #w "[tmp], %[ptr]\n" \
__nops(3)) \
: [loop] "=&r" (loop), [tmp] "=&r" (tmp), \
[ptr] "+Q"(*(u##sz *)ptr) \
@@ -98,7 +98,7 @@ __percpu_##name##_return_case_##sz(void *ptr, unsigned long val) \
" stxr" #sfx "\t%w[loop], %" #w "[ret], %[ptr]\n" \
" cbnz %w[loop], 1b", \
/* LSE atomics */ \
- #op_lse "\t%" #w "[val], %" #w "[ret], %[ptr]\n" \
+ #op_lse #sfx "\t%" #w "[val], %" #w "[ret], %[ptr]\n" \
#op_llsc "\t%" #w "[ret], %" #w "[ret], %" #w "[val]\n" \
__nops(2)) \
: [loop] "=&r" (loop), [ret] "=&r" (ret), \
@@ -124,9 +124,16 @@ PERCPU_RW_OPS(8)
PERCPU_RW_OPS(16)
PERCPU_RW_OPS(32)
PERCPU_RW_OPS(64)
-PERCPU_OP(add, add, stadd)
-PERCPU_OP(andnot, bic, stclr)
-PERCPU_OP(or, orr, stset)
+
+/*
+ * Use value-returning atomics for CPU-local ops as they are more likely
+ * to execute "near" to the CPU (e.g. in L1$).
+ *
+ * https://lore.kernel.org/r/e7d539ed-ced0-4b96-8ecd-048a5b803b85@paulmck-laptop
+ */
+PERCPU_OP(add, add, ldadd)
+PERCPU_OP(andnot, bic, ldclr)
+PERCPU_OP(or, orr, ldset)
PERCPU_RET_OP(add, add, ldadd)
#undef PERCPU_RW_OPS
@@ -172,13 +179,13 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_protect_return(__percpu_read_64, pcp)
#define this_cpu_write_1(pcp, val) \
- _pcp_protect(__percpu_write_8, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_8, pcp, (unsigned long)(val))
#define this_cpu_write_2(pcp, val) \
- _pcp_protect(__percpu_write_16, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_16, pcp, (unsigned long)(val))
#define this_cpu_write_4(pcp, val) \
- _pcp_protect(__percpu_write_32, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_32, pcp, (unsigned long)(val))
#define this_cpu_write_8(pcp, val) \
- _pcp_protect(__percpu_write_64, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_64, pcp, (unsigned long)(val))
#define this_cpu_add_1(pcp, val) \
_pcp_protect(__percpu_add_case_8, pcp, val)
@@ -199,13 +206,13 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_protect_return(__percpu_add_return_case_64, pcp, val)
#define this_cpu_and_1(pcp, val) \
- _pcp_protect(__percpu_andnot_case_8, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_8, pcp, ~(u8)(val))
#define this_cpu_and_2(pcp, val) \
- _pcp_protect(__percpu_andnot_case_16, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_16, pcp, ~(u16)(val))
#define this_cpu_and_4(pcp, val) \
- _pcp_protect(__percpu_andnot_case_32, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_32, pcp, ~(u32)(val))
#define this_cpu_and_8(pcp, val) \
- _pcp_protect(__percpu_andnot_case_64, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val))
#define this_cpu_or_1(pcp, val) \
_pcp_protect(__percpu_or_case_8, pcp, val)
diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S
index 0e1d9c3c6a93..2baefe7a82d3 100644
--- a/arch/arm64/kernel/hibernate-asm.S
+++ b/arch/arm64/kernel/hibernate-asm.S
@@ -89,6 +89,8 @@ alternative_insn "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE
isb
cbz x24, 3f /* Do we need to re-initialise EL2? */
+ mov x1, x24
+ mov x0, #HVC_SET_VECTORS
hvc #0
3: ret
SYM_CODE_END(swsusp_arch_suspend_exit)
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 44d31b019efd..b883f6d833b6 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -417,8 +417,8 @@ int __nocfi swsusp_arch_resume(void)
* Create a second copy of just the linear map, and use this when
* restoring.
*/
- rc = trans_pgd_create_copy(&trans_info, &tmp_pg_dir, PAGE_OFFSET,
- PAGE_END);
+ rc = trans_pgd_create_copy(&trans_info, &tmp_pg_dir,
+ _PAGE_OFFSET(vabits_actual), PAGE_END);
if (rc)
return rc;
diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 05b6435d02a9..364beb6f27a1 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -1287,7 +1287,7 @@ static const struct encoding_to_trap_config encoding_to_fgt[] __initconst = {
SR_FGT(OP_AT_S1E1A, HFGITR, ATS1E1A, 1),
SR_FGT(OP_COSP_RCTX, HFGITR, COSPRCTX, 1),
SR_FGT(OP_GCSPUSHX, HFGITR, nGCSEPP, 0),
- SR_FGT(OP_GCSPOPX, HFGITR, nGCSEPP, 0),
+ SR_FGT(OP_GCSPOPCX, HFGITR, nGCSEPP, 0),
SR_FGT(OP_GCSPUSHM, HFGITR, nGCSPUSHM_EL1, 0),
SR_FGT(OP_BRB_IALL, HFGITR, nBRBIALL, 0),
SR_FGT(OP_BRB_INJ, HFGITR, nBRBINJ, 0),
diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
index ee6573befb81..82af201d6028 100644
--- a/arch/arm64/kvm/hypercalls.c
+++ b/arch/arm64/kvm/hypercalls.c
@@ -183,7 +183,8 @@ static int kvm_smccc_set_filter(struct kvm *kvm, struct kvm_smccc_filter __user
start = filter.base;
end = start + filter.nr_functions - 1;
- if (end < start || filter.action >= NR_SMCCC_FILTER_ACTIONS)
+ if (!filter.nr_functions || end < start ||
+ filter.action >= NR_SMCCC_FILTER_ACTIONS)
return -EINVAL;
mutex_lock(&kvm->arch.config_lock);
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 0f7658aefa1a..6586008dc4df 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -53,27 +53,36 @@ static phys_addr_t stage2_range_addr_end(phys_addr_t addr, phys_addr_t end)
* long will also starve other vCPUs. We have to also make sure that the page
* tables are not freed while we released the lock.
*/
-static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t addr,
+static int stage2_apply_range(struct kvm_s2_mmu *mmu, phys_addr_t start,
phys_addr_t end,
int (*fn)(struct kvm_pgtable *, u64, u64),
bool resched)
{
struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
+ bool lock_dropped = false;
+ phys_addr_t addr = start;
int ret;
u64 next;
do {
struct kvm_pgtable *pgt = mmu->pgt;
+ /*
+ * We may be raced on PGT teardown when we release the
+ * kvm->mmu_lock. That's fine as the PGT is legitimately no
+ * longer present.
+ */
if (!pgt)
- return -EINVAL;
+ return lock_dropped ? 0 : -EINVAL;
next = stage2_range_addr_end(addr, end);
ret = fn(pgt, addr, next - addr);
if (ret)
break;
- if (resched && next != end)
+ if (resched && next != end) {
cond_resched_rwlock_write(&kvm->mmu_lock);
+ lock_dropped = true;
+ }
} while (addr = next, addr != end);
return ret;
diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c
index 7282305de347..4b56de335205 100644
--- a/arch/arm64/kvm/vgic/vgic-its.c
+++ b/arch/arm64/kvm/vgic/vgic-its.c
@@ -31,6 +31,41 @@ static int vgic_its_commit_v0(struct vgic_its *its);
static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
struct kvm_vcpu *filter_vcpu, bool needs_inv);
+#define vgic_its_read_entry_lock(i, g, valp, t) \
+ ({ \
+ int __sz = vgic_its_get_abi(i)->t##_esz; \
+ struct kvm *__k = (i)->dev->kvm; \
+ int __ret; \
+ \
+ BUILD_BUG_ON(NR_ITS_ABIS == 1 && \
+ sizeof(*(valp)) != ABI_0_ESZ); \
+ if (NR_ITS_ABIS > 1 && \
+ KVM_BUG_ON(__sz != sizeof(*(valp)), __k)) \
+ __ret = -EINVAL; \
+ else \
+ __ret = kvm_read_guest_lock(__k, (g), \
+ valp, __sz); \
+ __ret; \
+ })
+
+#define vgic_its_write_entry_lock(i, g, val, t) \
+ ({ \
+ int __sz = vgic_its_get_abi(i)->t##_esz; \
+ struct kvm *__k = (i)->dev->kvm; \
+ typeof(val) __v = (val); \
+ int __ret; \
+ \
+ BUILD_BUG_ON(NR_ITS_ABIS == 1 && \
+ sizeof(__v) != ABI_0_ESZ); \
+ if (NR_ITS_ABIS > 1 && \
+ KVM_BUG_ON(__sz != sizeof(__v), __k)) \
+ __ret = -EINVAL; \
+ else \
+ __ret = vgic_write_guest_lock(__k, (g), \
+ &__v, __sz); \
+ __ret; \
+ })
+
/*
* Creates a new (reference to a) struct vgic_irq for a given LPI.
* If this LPI is already mapped on another ITS, we increase its refcount
@@ -798,7 +833,7 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
its_free_ite(kvm, ite);
- return vgic_its_write_entry_lock(its, gpa, 0, ite_esz);
+ return vgic_its_write_entry_lock(its, gpa, 0ULL, ite);
}
return E_ITS_DISCARD_UNMAPPED_INTERRUPT;
@@ -1147,7 +1182,6 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
bool valid = its_cmd_get_validbit(its_cmd);
u8 num_eventid_bits = its_cmd_get_size(its_cmd);
gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
- int dte_esz = vgic_its_get_abi(its)->dte_esz;
struct its_device *device;
gpa_t gpa;
@@ -1172,7 +1206,7 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
* is an error, so we are done in any case.
*/
if (!valid)
- return vgic_its_write_entry_lock(its, gpa, 0, dte_esz);
+ return vgic_its_write_entry_lock(its, gpa, 0ULL, dte);
device = vgic_its_alloc_device(its, device_id, itt_addr,
num_eventid_bits);
@@ -1637,7 +1671,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
unsigned long val)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
- u64 entry_size, table_type;
+ u64 old, entry_size, table_type;
u64 reg, *regptr, clearbits = 0;
/* When GITS_CTLR.Enable is 1, we ignore write accesses. */
@@ -1660,7 +1694,9 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
return;
}
- reg = update_64bit_reg(*regptr, addr & 7, len, val);
+ old = *regptr;
+
+ reg = update_64bit_reg(old, addr & 7, len, val);
reg &= ~GITS_BASER_RO_MASK;
reg &= ~clearbits;
@@ -1670,7 +1706,8 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
*regptr = reg;
- if (!(reg & GITS_BASER_VALID)) {
+ /* The ITS driver rewrites an unchanged GITS_BASER<n> on resume. */
+ if (reg != old) {
/* Take the its_lock to prevent a race with a save/restore */
mutex_lock(&its->its_lock);
switch (table_type) {
@@ -1681,6 +1718,8 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
vgic_its_free_collection_list(kvm, its);
break;
}
+ /* A concurrent injection may have cached a translation. */
+ vgic_its_invalidate_cache(its);
mutex_unlock(&its->its_lock);
}
}
@@ -1999,18 +2038,22 @@ static int vgic_its_attr_regs_access(struct kvm_device *dev,
return ret;
}
-static u32 compute_next_devid_offset(struct list_head *h,
+static u32 compute_next_devid_offset(struct vgic_its *its, u64 baser,
struct its_device *dev)
{
- struct its_device *next;
- u32 next_offset;
+ struct its_device *next = dev;
- if (list_is_last(&dev->dev_list, h))
- return 0;
- next = list_next_entry(dev, dev_list);
- next_offset = next->device_id - dev->device_id;
+ /*
+ * Point at the next device vgic_its_save_device_tables() saves. It
+ * sorts device_list first, so the subtraction cannot underflow.
+ */
+ list_for_each_entry_continue(next, &its->device_list, dev_list) {
+ if (vgic_its_check_id(its, baser, next->device_id, NULL))
+ return min_t(u32, next->device_id - dev->device_id,
+ VITS_DTE_MAX_DEVID_OFFSET);
+ }
- return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
+ return 0;
}
static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
@@ -2094,7 +2137,7 @@ static int scan_its_table(struct vgic_its *its, gpa_t base, int size, u32 esz,
* vgic_its_save_ite - Save an interrupt translation entry at @gpa
*/
static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
- struct its_ite *ite, gpa_t gpa, int ite_esz)
+ struct its_ite *ite, gpa_t gpa)
{
u32 next_offset;
u64 val;
@@ -2105,7 +2148,7 @@ static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
ite->collection->collection_id;
val = cpu_to_le64(val);
- return vgic_its_write_entry_lock(its, gpa, val, ite_esz);
+ return vgic_its_write_entry_lock(its, gpa, val, ite);
}
/**
@@ -2205,7 +2248,7 @@ static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
if (ite->irq->hw && !kvm_vgic_global_state.has_gicv4_1)
return -EACCES;
- ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
+ ret = vgic_its_save_ite(its, device, ite, gpa);
if (ret)
return ret;
}
@@ -2242,25 +2285,25 @@ static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
* vgic_its_save_dte - Save a device table entry at a given GPA
*
* @its: ITS handle
+ * @baser: GITS_BASER<dev> the caller is saving against
* @dev: ITS device
* @ptr: GPA
- * @dte_esz: device table entry size
*/
-static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
- gpa_t ptr, int dte_esz)
+static int vgic_its_save_dte(struct vgic_its *its, u64 baser,
+ struct its_device *dev, gpa_t ptr)
{
u64 val, itt_addr_field;
u32 next_offset;
itt_addr_field = dev->itt_addr >> 8;
- next_offset = compute_next_devid_offset(&its->device_list, dev);
+ next_offset = compute_next_devid_offset(its, baser, dev);
val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
(itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
(dev->num_eventid_bits - 1));
val = cpu_to_le64(val);
- return vgic_its_write_entry_lock(its, ptr, val, dte_esz);
+ return vgic_its_write_entry_lock(its, ptr, val, dte);
}
/**
@@ -2340,10 +2383,8 @@ static int vgic_its_device_cmp(void *priv, const struct list_head *a,
*/
static int vgic_its_save_device_tables(struct vgic_its *its)
{
- const struct vgic_its_abi *abi = vgic_its_get_abi(its);
u64 baser = its->baser_device_table;
struct its_device *dev;
- int dte_esz = abi->dte_esz;
if (!(baser & GITS_BASER_VALID))
return 0;
@@ -2354,15 +2395,16 @@ static int vgic_its_save_device_tables(struct vgic_its *its)
int ret;
gpa_t eaddr;
+ /* Don't fail a save that userspace must be able to issue. */
if (!vgic_its_check_id(its, baser,
dev->device_id, &eaddr))
- return -EINVAL;
+ continue;
ret = vgic_its_save_itt(its, dev);
if (ret)
return ret;
- ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
+ ret = vgic_its_save_dte(its, baser, dev, eaddr);
if (ret)
return ret;
}
@@ -2443,7 +2485,7 @@ static int vgic_its_restore_device_tables(struct vgic_its *its)
static int vgic_its_save_cte(struct vgic_its *its,
struct its_collection *collection,
- gpa_t gpa, int esz)
+ gpa_t gpa)
{
u64 val;
@@ -2452,7 +2494,7 @@ static int vgic_its_save_cte(struct vgic_its *its,
collection->collection_id);
val = cpu_to_le64(val);
- return vgic_its_write_entry_lock(its, gpa, val, esz);
+ return vgic_its_write_entry_lock(its, gpa, val, cte);
}
/*
@@ -2460,7 +2502,7 @@ static int vgic_its_save_cte(struct vgic_its *its,
* Return +1 on success, 0 if the entry was invalid (which should be
* interpreted as end-of-table), and a negative error value for generic errors.
*/
-static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa, int esz)
+static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa)
{
struct its_collection *collection;
struct kvm *kvm = its->dev->kvm;
@@ -2468,7 +2510,7 @@ static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa, int esz)
u64 val;
int ret;
- ret = vgic_its_read_entry_lock(its, gpa, &val, esz);
+ ret = vgic_its_read_entry_lock(its, gpa, &val, cte);
if (ret)
return ret;
val = le64_to_cpu(val);
@@ -2515,7 +2557,7 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
max_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
list_for_each_entry(collection, &its->collection_list, coll_list) {
- ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
+ ret = vgic_its_save_cte(its, collection, gpa);
if (ret)
return ret;
gpa += cte_esz;
@@ -2529,7 +2571,7 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
* table is not fully filled, add a last dummy element
* with valid bit unset
*/
- return vgic_its_write_entry_lock(its, gpa, 0, cte_esz);
+ return vgic_its_write_entry_lock(its, gpa, 0ULL, cte);
}
/*
@@ -2554,7 +2596,7 @@ static int vgic_its_restore_collection_table(struct vgic_its *its)
max_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
while (read < max_size) {
- ret = vgic_its_restore_cte(its, gpa, cte_esz);
+ ret = vgic_its_restore_cte(its, gpa);
if (ret <= 0)
break;
gpa += cte_esz;
diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h
index 309295f5e1b0..f2486b4d9f95 100644
--- a/arch/arm64/kvm/vgic/vgic.h
+++ b/arch/arm64/kvm/vgic/vgic.h
@@ -146,29 +146,6 @@ static inline int vgic_write_guest_lock(struct kvm *kvm, gpa_t gpa,
return ret;
}
-static inline int vgic_its_read_entry_lock(struct vgic_its *its, gpa_t eaddr,
- u64 *eval, unsigned long esize)
-{
- struct kvm *kvm = its->dev->kvm;
-
- if (KVM_BUG_ON(esize != sizeof(*eval), kvm))
- return -EINVAL;
-
- return kvm_read_guest_lock(kvm, eaddr, eval, esize);
-
-}
-
-static inline int vgic_its_write_entry_lock(struct vgic_its *its, gpa_t eaddr,
- u64 eval, unsigned long esize)
-{
- struct kvm *kvm = its->dev->kvm;
-
- if (KVM_BUG_ON(esize != sizeof(eval), kvm))
- return -EINVAL;
-
- return vgic_write_guest_lock(kvm, eaddr, &eval, esize);
-}
-
/*
* This struct provides an intermediate representation of the fields contained
* in the GICH_VMCR and ICH_VMCR registers, such that code exporting the GIC
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 27d27848db52..9c28bb7b69ae 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -537,6 +537,8 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
* 12 registers are on the stack
*/
emit(A64_SUB_I(1, A64_SP, A64_FP, 96), ctx);
+ /* The callback may use its own BPF stack, set up fp for it. */
+ ctx->fp_used = true;
}
if (ctx->fp_used)
diff --git a/arch/mips/cavium-octeon/octeon-platform.c b/arch/mips/cavium-octeon/octeon-platform.c
index 5e1dd4e6e82f..4b5c99e6f5bc 100644
--- a/arch/mips/cavium-octeon/octeon-platform.c
+++ b/arch/mips/cavium-octeon/octeon-platform.c
@@ -17,7 +17,7 @@
#include <asm/octeon/octeon.h>
#include <asm/octeon/cvmx-helper-board.h>
-#ifdef CONFIG_USB
+#if IS_ENABLED(CONFIG_USB)
#include <linux/usb/ehci_def.h>
#include <linux/usb/ehci_pdriver.h>
#include <linux/usb/ohci_pdriver.h>
@@ -1080,7 +1080,7 @@ int __init octeon_prune_device_tree(void)
;
}
-#ifdef CONFIG_USB
+#if IS_ENABLED(CONFIG_USB)
/* OHCI/UHCI USB */
alias_prop = fdt_getprop(initial_boot_params, aliases,
"uctl", NULL);
diff --git a/arch/mips/net/bpf_jit_comp32.c b/arch/mips/net/bpf_jit_comp32.c
index 40a878b672f5..15a2a153dc87 100644
--- a/arch/mips/net/bpf_jit_comp32.c
+++ b/arch/mips/net/bpf_jit_comp32.c
@@ -1111,7 +1111,7 @@ static void emit_jmp_i64(struct jit_context *ctx,
emit(ctx, xor, tmp, lo(dst), tmp);
}
if (imm < 0) { /* Compare sign extension */
- emit(ctx, addu, MIPS_R_T9, hi(dst), 1);
+ emit(ctx, addiu, MIPS_R_T9, hi(dst), 1);
emit(ctx, or, tmp, tmp, MIPS_R_T9);
} else { /* Compare zero extension */
emit(ctx, or, tmp, tmp, hi(dst));
diff --git a/arch/mips/net/bpf_jit_comp64.c b/arch/mips/net/bpf_jit_comp64.c
index fa7e9aa37f49..6681ccac9dd9 100644
--- a/arch/mips/net/bpf_jit_comp64.c
+++ b/arch/mips/net/bpf_jit_comp64.c
@@ -305,8 +305,7 @@ static void emit_bswap_r64(struct jit_context *ctx, u8 dst, u32 width)
case 16:
emit_sext(ctx, dst, dst);
emit_bswap_r(ctx, dst, width);
- if (cpu_has_mips64r2 || cpu_has_mips64r6)
- emit_zext(ctx, dst);
+ emit_zext(ctx, dst);
break;
}
clobber_reg(ctx, dst);
diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c
index 0ebae6e4c19d..50c180cd1fa3 100644
--- a/arch/powerpc/kernel/iommu.c
+++ b/arch/powerpc/kernel/iommu.c
@@ -1074,7 +1074,7 @@ int iommu_tce_check_ioba(unsigned long page_shift,
if (ioba < offset)
return -EINVAL;
- if ((ioba + 1) > (offset + size))
+ if ((ioba + npages < ioba) || (ioba - offset + npages > size))
return -EINVAL;
return 0;
diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
index 125440a606ee..a4dd8d983ec7 100644
--- a/arch/powerpc/kvm/book3s_hv_nested.c
+++ b/arch/powerpc/kvm/book3s_hv_nested.c
@@ -1202,8 +1202,10 @@ static void kvmhv_emulate_tlbie_all_lpid(struct kvm_vcpu *vcpu, int ric)
spin_lock(&kvm->mmu_lock);
idr_for_each_entry(&kvm->arch.kvm_nested_guest_idr, gp, lpid) {
+ ++gp->refcnt;
spin_unlock(&kvm->mmu_lock);
kvmhv_emulate_tlbie_lpid(vcpu, gp, ric);
+ kvmhv_put_nested(gp);
spin_lock(&kvm->mmu_lock);
}
spin_unlock(&kvm->mmu_lock);
diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c
index 92f33115144b..7ea0c5862227 100644
--- a/arch/powerpc/kvm/book3s_hv_uvmem.c
+++ b/arch/powerpc/kvm/book3s_hv_uvmem.c
@@ -779,8 +779,11 @@ static int kvmppc_svm_page_in(struct vm_area_struct *vma,
if (spage) {
ret = uv_page_in(kvm->arch.lpid, pfn << page_shift,
gpa, 0, page_shift);
- if (ret)
+ if (ret) {
+ unlock_page(dpage);
+ put_page(dpage);
goto out_finalize;
+ }
}
}
diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c
index a8085cd8215e..abc247f7c903 100644
--- a/arch/riscv/kvm/aia_imsic.c
+++ b/arch/riscv/kvm/aia_imsic.c
@@ -895,8 +895,15 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type,
if (!vcpu)
return -ENODEV;
- isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type);
+ if (mutex_lock_killable(&vcpu->mutex))
+ return -EINTR;
+
imsic = vcpu->arch.aia_context.imsic_state;
+ if (!imsic) {
+ rc = -ENODEV;
+ goto out_unlock;
+ }
+ isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type);
read_lock_irqsave(&imsic->vsfile_lock, flags);
@@ -919,6 +926,8 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type,
rc = imsic_vsfile_rw(vsfile_hgei, vsfile_cpu, imsic->nr_eix,
isel, write, val);
+out_unlock:
+ mutex_unlock(&vcpu->mutex);
return rc;
}
diff --git a/arch/riscv/kvm/vcpu_pmu.c b/arch/riscv/kvm/vcpu_pmu.c
index 9ec3280db91d..147fc0d107ca 100644
--- a/arch/riscv/kvm/vcpu_pmu.c
+++ b/arch/riscv/kvm/vcpu_pmu.c
@@ -245,12 +245,13 @@ static int pmu_ctr_read(struct kvm_vcpu *vcpu, unsigned long cidx,
if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW) {
fevent_code = get_event_code(pmc->event_idx);
pmc->counter_val = kvpmu->fw_event[fevent_code].value;
+ *out_val = pmc->counter_val;
} else if (pmc->perf_event) {
- pmc->counter_val += perf_event_read_value(pmc->perf_event, &enabled, &running);
+ *out_val = pmc->counter_val +
+ perf_event_read_value(pmc->perf_event, &enabled, &running);
} else {
return -EINVAL;
}
- *out_val = pmc->counter_val;
return 0;
}
@@ -561,7 +562,6 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
{
struct kvm_pmu *kvpmu = vcpu_to_pmu(vcpu);
int i, pmc_index, sbiret = 0;
- u64 enabled, running;
struct kvm_pmc *pmc;
int fevent_code;
bool snap_flag_set = flags & SBI_PMU_STOP_FLAG_TAKE_SNAPSHOT;
@@ -590,14 +590,19 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
goto out;
}
- if (!kvpmu->fw_event[fevent_code].started)
+ if (!kvpmu->fw_event[fevent_code].started) {
sbiret = SBI_ERR_ALREADY_STOPPED;
-
- kvpmu->fw_event[fevent_code].started = false;
+ } else {
+ kvpmu->fw_event[fevent_code].started = false;
+ pmc->counter_val = kvpmu->fw_event[fevent_code].value;
+ }
} else if (pmc->perf_event) {
if (pmc->started) {
- /* Stop counting the counter */
- perf_event_disable(pmc->perf_event);
+ /*
+ * Stop the counter and fold the live count into counter_val.
+ * Reset the event value to avoid redundant accumulation.
+ */
+ pmc->counter_val += perf_event_pause(pmc->perf_event, true);
pmc->started = false;
} else {
sbiret = SBI_ERR_ALREADY_STOPPED;
@@ -611,11 +616,6 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
}
if (snap_flag_set && !sbiret) {
- if (pmc->cinfo.type == SBI_PMU_CTR_TYPE_FW)
- pmc->counter_val = kvpmu->fw_event[fevent_code].value;
- else if (pmc->perf_event)
- pmc->counter_val += perf_event_read_value(pmc->perf_event,
- &enabled, &running);
/*
* The counter and overflow indicies in the snapshot region are w.r.to
* cbase. Modify the set bit in the counter mask instead of the pmc_index
@@ -642,9 +642,10 @@ int kvm_riscv_vcpu_pmu_ctr_stop(struct kvm_vcpu *vcpu, unsigned long ctr_base,
}
}
- if (shmem_needs_update)
- kvm_vcpu_write_guest(vcpu, kvpmu->snapshot_addr, kvpmu->sdata,
- sizeof(struct riscv_pmu_snapshot_data));
+ if (shmem_needs_update &&
+ kvm_vcpu_write_guest(vcpu, kvpmu->snapshot_addr, kvpmu->sdata,
+ sizeof(struct riscv_pmu_snapshot_data)))
+ sbiret = SBI_ERR_FAILURE;
out:
retdata->err_val = sbiret;
diff --git a/arch/riscv/kvm/vcpu_sbi_hsm.c b/arch/riscv/kvm/vcpu_sbi_hsm.c
index 3070bb31745d..c4cf6d95029e 100644
--- a/arch/riscv/kvm/vcpu_sbi_hsm.c
+++ b/arch/riscv/kvm/vcpu_sbi_hsm.c
@@ -106,9 +106,9 @@ static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
ret = kvm_sbi_hsm_vcpu_get_status(vcpu);
if (ret >= 0) {
retdata->out_val = ret;
- retdata->err_val = 0;
+ ret = 0;
}
- return 0;
+ break;
case SBI_EXT_HSM_HART_SUSPEND:
switch (lower_32_bits(cp->a0)) {
case SBI_HSM_SUSPEND_RET_DEFAULT:
diff --git a/arch/riscv/kvm/vcpu_timer.c b/arch/riscv/kvm/vcpu_timer.c
index 75486b25ac45..3a03c1fbfff0 100644
--- a/arch/riscv/kvm/vcpu_timer.c
+++ b/arch/riscv/kvm/vcpu_timer.c
@@ -60,10 +60,13 @@ static enum hrtimer_restart kvm_riscv_vcpu_hrtimer_expired(struct hrtimer *h)
static int kvm_riscv_vcpu_timer_cancel(struct kvm_vcpu_timer *t)
{
- if (!t->init_done || !t->next_set)
+ if (!t->init_done)
return -EINVAL;
hrtimer_cancel(&t->hrt);
+
+ if (!t->next_set)
+ return -EINVAL;
t->next_set = false;
return 0;
diff --git a/arch/s390/boot/ipl_parm.c b/arch/s390/boot/ipl_parm.c
index 557462e62cd7..6c77afb3b8a1 100644
--- a/arch/s390/boot/ipl_parm.c
+++ b/arch/s390/boot/ipl_parm.c
@@ -21,6 +21,7 @@ struct parmarea parmarea __section(".parmarea") = {
};
char __bootdata(early_command_line)[COMMAND_LINE_SIZE];
+static char command_line_buf[COMMAND_LINE_SIZE];
unsigned int __bootdata_preserved(zlib_dfltcc_support) = ZLIB_DFLTCC_FULL;
struct ipl_parameter_block __bootdata_preserved(ipl_block);
@@ -148,31 +149,29 @@ static size_t ipl_block_get_ascii_scpdata(char *dest, size_t size,
static void append_ipl_block_parm(void)
{
- char *parm, *delim;
- size_t len, rc = 0;
+ size_t len, extra = 0;
+ char *delim;
len = strlen(early_command_line);
-
- delim = early_command_line + len; /* '\0' character position */
- parm = early_command_line + len + 1; /* append right after '\0' */
+ delim = early_command_line + len; /* '\0' character position */
switch (ipl_block.pb0_hdr.pbt) {
case IPL_PBT_CCW:
- rc = ipl_block_get_ascii_vmparm(
- parm, COMMAND_LINE_SIZE - len - 1, &ipl_block);
+ extra = ipl_block_get_ascii_vmparm(command_line_buf, sizeof(command_line_buf), &ipl_block);
break;
case IPL_PBT_FCP:
case IPL_PBT_NVME:
case IPL_PBT_ECKD:
- rc = ipl_block_get_ascii_scpdata(
- parm, COMMAND_LINE_SIZE - len - 1, &ipl_block);
+ extra = ipl_block_get_ascii_scpdata(command_line_buf, sizeof(command_line_buf), &ipl_block);
break;
}
- if (rc) {
- if (*parm == '=')
- memmove(early_command_line, parm + 1, rc);
- else
+ if (extra) {
+ if (command_line_buf[0] == '=') {
+ memmove(early_command_line, command_line_buf + 1, extra);
+ } else if (len < COMMAND_LINE_SIZE - 2) {
*delim = ' '; /* replace '\0' with space */
+ sized_strscpy(delim + 1, command_line_buf, COMMAND_LINE_SIZE - len - 1);
+ }
}
}
@@ -258,7 +257,6 @@ static void modify_fac_list(char *str)
check_cleared_facilities();
}
-static char command_line_buf[COMMAND_LINE_SIZE];
void parse_boot_command_line(void)
{
char *param, *val;
diff --git a/arch/s390/boot/string.c b/arch/s390/boot/string.c
index f6b9b1df48a8..bd68161434a6 100644
--- a/arch/s390/boot/string.c
+++ b/arch/s390/boot/string.c
@@ -29,6 +29,18 @@ int strncmp(const char *cs, const char *ct, size_t count)
return 0;
}
+ssize_t sized_strscpy(char *dst, const char *src, size_t count)
+{
+ size_t len;
+
+ if (count == 0)
+ return -E2BIG;
+ len = strnlen(src, count - 1);
+ memcpy(dst, src, len);
+ dst[len] = '\0';
+ return src[len] ? -E2BIG : len;
+}
+
void *memset64(uint64_t *s, uint64_t v, size_t count)
{
uint64_t *xs = s;
diff --git a/arch/s390/include/asm/debug.h b/arch/s390/include/asm/debug.h
index ccd4e148b5ed..7c3b058495d9 100644
--- a/arch/s390/include/asm/debug.h
+++ b/arch/s390/include/asm/debug.h
@@ -451,7 +451,11 @@ static int VNAME(var, active_entries)[EARLY_AREAS] __initdata
#define __REGISTER_STATIC_DEBUG_INFO(var, name, pages, areas, view) \
static int __init VNAME(var, reg)(void) \
{ \
- debug_register_static(&var, (pages), (areas)); \
+ int rc; \
+ \
+ rc = debug_register_static(&var, (pages), (areas)); \
+ if (rc) \
+ return rc; \
debug_register_view(&var, (view)); \
return 0; \
} \
@@ -483,7 +487,7 @@ static debug_info_t __refdata var = \
__DEBUG_INFO_INIT(var, (name), (buf_size)); \
__REGISTER_STATIC_DEBUG_INFO(var, name, pages, nr_areas, view)
-void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas);
+int debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas);
#endif /* MODULE */
diff --git a/arch/s390/kernel/debug.c b/arch/s390/kernel/debug.c
index f275fa8b3873..3aaab76f43d9 100644
--- a/arch/s390/kernel/debug.c
+++ b/arch/s390/kernel/debug.c
@@ -344,7 +344,8 @@ static debug_info_t *debug_info_copy(debug_info_t *in, int mode)
debug_info_free(rc);
} while (1);
- if (mode == NO_AREAS)
+ /* debug_register_static() failure leaves areas NULL, bounds intact */
+ if (mode == NO_AREAS || !in->areas)
goto out;
for (i = 0; i < in->nr_areas; i++) {
@@ -702,8 +703,12 @@ EXPORT_SYMBOL(debug_register);
*
* Note: This function is called automatically via an initcall generated by
* DEFINE_STATIC_DEBUG_INFO.
+ *
+ * Return:
+ * - 0 on success
+ * - negative error code on failure
*/
-void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
+int debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
{
unsigned long flags;
debug_info_t *copy;
@@ -711,7 +716,7 @@ void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
if (!initialized) {
pr_err("Tried to register debug feature %s too early\n",
id->name);
- return;
+ return -EINVAL;
}
copy = debug_info_alloc("", pages_per_area, nr_areas, id->buf_size,
@@ -726,7 +731,7 @@ void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
id->active_entries = NULL;
spin_unlock_irqrestore(&id->lock, flags);
- return;
+ return -ENOMEM;
}
/* Replace static trace area with dynamic copy. */
@@ -744,6 +749,8 @@ void debug_register_static(debug_info_t *id, int pages_per_area, int nr_areas)
mutex_lock(&debug_mutex);
_debug_register(id);
mutex_unlock(&debug_mutex);
+
+ return 0;
}
/* Remove debugfs entries. */
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index 3fc07da7fd7a..0f3ad8c80381 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -1586,23 +1586,21 @@ static int __inject_set_prefix(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq)
}
#define KVM_S390_STOP_SUPP_FLAGS (KVM_S390_STOP_FLAG_STORE_STATUS)
-static int __inject_sigp_stop(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq)
+static int __inject_sigp_stop(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq, bool *storestatus)
{
struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
struct kvm_s390_stop_info *stop = &li->irq.stop;
- int rc = 0;
vcpu->stat.inject_stop_signal++;
trace_kvm_s390_inject_vcpu(vcpu->vcpu_id, KVM_S390_SIGP_STOP, 0, 0);
if (irq->u.stop.flags & ~KVM_S390_STOP_SUPP_FLAGS)
return -EINVAL;
-
if (is_vcpu_stopped(vcpu)) {
- if (irq->u.stop.flags & KVM_S390_STOP_FLAG_STORE_STATUS)
- rc = kvm_s390_store_status_unloaded(vcpu,
- KVM_S390_STORE_STATUS_NOADDR);
- return rc;
+ if (!(irq->u.stop.flags & KVM_S390_STOP_FLAG_STORE_STATUS))
+ return 0;
+ *storestatus = true;
+ return -EWOULDBLOCK;
}
if (test_and_set_bit(IRQ_PEND_SIGP_STOP, &li->pending_irqs))
@@ -2142,7 +2140,7 @@ void kvm_s390_clear_stop_irq(struct kvm_vcpu *vcpu)
spin_unlock(&li->lock);
}
-static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq)
+static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq, bool *storestatus)
{
int rc;
@@ -2154,7 +2152,7 @@ static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq)
rc = __inject_set_prefix(vcpu, irq);
break;
case KVM_S390_SIGP_STOP:
- rc = __inject_sigp_stop(vcpu, irq);
+ rc = __inject_sigp_stop(vcpu, irq, storestatus);
break;
case KVM_S390_RESTART:
rc = __inject_sigp_restart(vcpu);
@@ -2190,11 +2188,16 @@ static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq)
int kvm_s390_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq)
{
struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
+ bool storestatus = false;
int rc;
spin_lock(&li->lock);
- rc = do_inject_vcpu(vcpu, irq);
+ rc = do_inject_vcpu(vcpu, irq, &storestatus);
spin_unlock(&li->lock);
+
+ if (rc == -EWOULDBLOCK && storestatus)
+ rc = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR);
+
if (!rc)
kvm_s390_vcpu_wakeup(vcpu);
return rc;
@@ -2930,7 +2933,8 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e, struct kvm *kvm,
int kvm_s390_set_irq_state(struct kvm_vcpu *vcpu, void __user *irqstate, int len)
{
struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;
- struct kvm_s390_irq *buf;
+ struct kvm_s390_irq *buf __free(kvfree) = NULL;
+ bool tmp, storestatus = false;
int r = 0;
int n;
@@ -2938,31 +2942,33 @@ int kvm_s390_set_irq_state(struct kvm_vcpu *vcpu, void __user *irqstate, int len
if (!buf)
return -ENOMEM;
- if (copy_from_user((void *) buf, irqstate, len)) {
- r = -EFAULT;
- goto out_free;
- }
+ if (copy_from_user((void *)buf, irqstate, len))
+ return -EFAULT;
- /*
- * Don't allow setting the interrupt state
- * when there are already interrupts pending
- */
- spin_lock(&li->lock);
- if (li->pending_irqs) {
- r = -EBUSY;
- goto out_unlock;
- }
+ scoped_guard(spinlock, &li->lock) {
+ /*
+ * Don't allow setting the interrupt state
+ * when there are already interrupts pending
+ */
+ if (li->pending_irqs)
+ return -EBUSY;
- for (n = 0; n < len / sizeof(*buf); n++) {
- r = do_inject_vcpu(vcpu, &buf[n]);
- if (r)
- break;
+ for (n = 0; n < len / sizeof(*buf); n++) {
+ tmp = false;
+ r = do_inject_vcpu(vcpu, &buf[n], &tmp);
+ if (r == -EWOULDBLOCK && tmp) {
+ storestatus = true;
+ r = 0;
+ }
+ if (r)
+ break;
+ }
}
-out_unlock:
- spin_unlock(&li->lock);
-out_free:
- vfree(buf);
+ if (storestatus) {
+ n = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR);
+ return r ? r : n;
+ }
return r;
}
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index f2e2e9385d72..0e3b8ee0ec26 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4601,8 +4601,8 @@ long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable)
static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token,
unsigned long token)
{
- struct kvm_s390_interrupt inti;
- struct kvm_s390_irq irq;
+ struct kvm_s390_interrupt inti = {};
+ struct kvm_s390_irq irq = {};
if (start_token) {
irq.u.ext.ext_params2 = token;
diff --git a/arch/x86/entry/entry_fred.c b/arch/x86/entry/entry_fred.c
index 9f50f0c1c00f..c43c750fa26d 100644
--- a/arch/x86/entry/entry_fred.c
+++ b/arch/x86/entry/entry_fred.c
@@ -10,6 +10,7 @@
#include <asm/desc.h>
#include <asm/fred.h>
#include <asm/idtentry.h>
+#include <asm/processor-flags.h>
#include <asm/syscall.h>
#include <asm/trapnr.h>
#include <asm/traps.h>
@@ -71,7 +72,15 @@ static noinstr void fred_intx(struct pt_regs *regs)
#endif
default:
- return exc_general_protection(regs, 0);
+ /*
+ * Reconstruct the #GP fault state that IDT delivery would produce.
+ * Clear the software event flag so ERETU with TF set does not trap
+ * before the resumed instruction. See prevent_single_step_upon_eretu().
+ */
+ regs->ip -= regs->fred_ss.insnlen;
+ regs->flags |= X86_EFLAGS_RF;
+ regs->fred_ss.swevent = 0;
+ return exc_general_protection(regs, (regs->fred_ss.vector << 3) | 2);
}
}
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 27001daade10..64d52911be3d 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -411,11 +411,12 @@ static struct event_constraint intel_lnc_event_constraints[] = {
INTEL_UEVENT_CONSTRAINT(0x0ca3, 0x4),
INTEL_UEVENT_CONSTRAINT(0x04a4, 0x1),
INTEL_UEVENT_CONSTRAINT(0x08a4, 0x1),
- INTEL_UEVENT_CONSTRAINT(0x10a4, 0x1),
+ INTEL_UEVENT_CONSTRAINT(0x10a4, 0x8),
INTEL_UEVENT_CONSTRAINT(0x01b1, 0x8),
INTEL_UEVENT_CONSTRAINT(0x01cd, 0x3fc),
INTEL_UEVENT_CONSTRAINT(0x02cd, 0x3),
+ INTEL_UEVENT_CONSTRAINT(0x87d0, 0x3ff),
INTEL_EVENT_CONSTRAINT_RANGE(0xd0, 0xdf, 0xf),
INTEL_UEVENT_CONSTRAINT(0x00e0, 0xf),
@@ -4288,12 +4289,15 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
struct kvm_pmu *kvm_pmu = (struct kvm_pmu *)data;
u64 intel_ctrl = hybrid(cpuc->pmu, intel_ctrl);
u64 pebs_mask = cpuc->pebs_enabled & x86_pmu.pebs_capable;
- int global_ctrl, pebs_enable;
+ u64 guest_pebs_mask;
+ int global_ctrl;
/*
* In addition to obeying exclude_guest/exclude_host, remove bits being
* used for PEBS when running a guest, because PEBS writes to virtual
- * addresses (not physical addresses).
+ * addresses (not physical addresses). If the guest wants to utilize
+ * PEBS, and PEBS can be safely enabled in the guest, bits for the guest's
+ * PEBS-enabled counters will be OR'd back in as appropriate.
*/
*nr = 0;
global_ctrl = (*nr)++;
@@ -4323,41 +4327,68 @@ static struct perf_guest_switch_msr *intel_guest_get_msrs(int *nr, void *data)
return arr;
}
- if (!kvm_pmu || !x86_pmu.pebs_ept)
+ /*
+ * If the CPU doesn't support PEBS in the guest, then there's nothing
+ * more to do as disabling PMCs via PERF_GLOBAL_CTRL is sufficient on
+ * CPUs with guest/host isolation.
+ */
+ if (!x86_pmu.pebs_ept)
return arr;
+ /*
+ * Restrict guest PEBS events to counters that (a) perf supports, (b)
+ * the guest wants to use for PEBS, (c) are not excluded from counting
+ * in the guest, and (d) _are_ excluded from counting in the host.
+ */
+ guest_pebs_mask = pebs_mask & intel_ctrl & kvm_pmu->pebs_enable &
+ ~cpuc->intel_ctrl_host_mask &
+ cpuc->intel_ctrl_guest_mask;
+
+ /*
+ * Disable counters where the guest PMC is different than the host PMC
+ * being used on behalf of the guest, as the PEBS record includes
+ * PERF_GLOBAL_STATUS, i.e. the guest will see overflow status for the
+ * wrong counter(s).
+ */
+ guest_pebs_mask &= ~kvm_pmu->host_cross_mapped_mask;
+
+ /*
+ * FIXME: Allow guest and host usage of PEBS events to co-exist instead
+ * of disabling guest PEBS entirely if the host is using PEBS.
+ * What exactly goes wrong if guest and host are using PEBS is
+ * unknown.
+ */
+ if (pebs_mask & ~cpuc->intel_ctrl_guest_mask)
+ guest_pebs_mask = 0;
+
+ /*
+ * Context switch DS_AREA and PEBS_DATA_CFG if and only if PEBS will be
+ * active in the guest; if no records will be generated while the guest
+ * is running, then simply keep the host values resident in hardware.
+ */
arr[(*nr)++] = (struct perf_guest_switch_msr){
.msr = MSR_IA32_DS_AREA,
.host = (unsigned long)cpuc->ds,
- .guest = kvm_pmu->ds_area,
+ .guest = guest_pebs_mask ? kvm_pmu->ds_area : (unsigned long)cpuc->ds,
};
if (x86_pmu.intel_cap.pebs_baseline) {
arr[(*nr)++] = (struct perf_guest_switch_msr){
.msr = MSR_PEBS_DATA_CFG,
.host = cpuc->active_pebs_data_cfg,
- .guest = kvm_pmu->pebs_data_cfg,
+ .guest = guest_pebs_mask ? kvm_pmu->pebs_data_cfg :
+ cpuc->active_pebs_data_cfg,
};
}
- pebs_enable = (*nr)++;
- arr[pebs_enable] = (struct perf_guest_switch_msr){
- .msr = MSR_IA32_PEBS_ENABLE,
- .host = cpuc->pebs_enabled & ~cpuc->intel_ctrl_guest_mask,
- .guest = pebs_mask & ~cpuc->intel_ctrl_host_mask & kvm_pmu->pebs_enable,
- };
-
- if (arr[pebs_enable].host) {
- /* Disable guest PEBS if host PEBS is enabled. */
- arr[pebs_enable].guest = 0;
- } else {
- /* Disable guest PEBS thoroughly for cross-mapped PEBS counters. */
- arr[pebs_enable].guest &= ~kvm_pmu->host_cross_mapped_mask;
- arr[global_ctrl].guest &= ~kvm_pmu->host_cross_mapped_mask;
- /* Set hw GLOBAL_CTRL bits for PEBS counter when it runs for guest */
- arr[global_ctrl].guest |= arr[pebs_enable].guest;
- }
-
+ /*
+ * Do NOT mess with PEBS_ENABLED. As above, disabling counters via
+ * PERF_GLOBAL_CTRL is sufficient, and loading a stale PEBS_ENABLED,
+ * e.g. on VM-Exit, can put the system in a bad state. Simply enable
+ * counters in PERF_GLOBAL_CTRL, as perf load PEBS_ENABLED with the
+ * full value, i.e. perf *also* relies on PERF_GLOBAL_CTRL.
+ */
+ arr[global_ctrl].guest |= guest_pebs_mask;
return arr;
}
@@ -6538,12 +6569,6 @@ __init int intel_pmu_init(void)
x86_add_quirk(intel_arch_events_quirk); /* Install first, so it runs last */
- if (version >= 5) {
- x86_pmu.intel_cap.anythread_deprecated = edx.split.anythread_deprecated;
- if (x86_pmu.intel_cap.anythread_deprecated)
- pr_cont(" AnyThread deprecated, ");
- }
-
/*
* Install the hw-cache-events table:
*/
@@ -7271,8 +7296,10 @@ __init int intel_pmu_init(void)
&x86_pmu.intel_ctrl);
/* AnyThread may be deprecated on arch perfmon v5 or later */
- if (x86_pmu.intel_cap.anythread_deprecated)
+ if (version >= 5 && edx.split.anythread_deprecated) {
x86_pmu.format_attrs = intel_arch_formats_attr;
+ pr_cont("AnyThread deprecated, ");
+ }
intel_pmu_check_event_constraints(x86_pmu.event_constraints,
x86_pmu.cntr_mask64,
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 39f5038f5ce4..4eac31693210 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -303,6 +303,7 @@ static inline void pebs_set_tlb_lock(u64 *val, bool tlb, bool lock)
static u64 __grt_latency_data(struct perf_event *event, u64 status,
u8 dse, bool tlb, bool lock, bool blk)
{
+ union perf_mem_data_src src;
u64 val;
WARN_ON_ONCE(is_hybrid() &&
@@ -318,7 +319,16 @@ static u64 __grt_latency_data(struct perf_event *event, u64 status,
else
val |= P(BLK, NA);
- return val;
+ src.val = val;
+
+ if (event->hw.flags &
+ (PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW))
+ src.mem_op = P(OP, LOAD);
+ if (event->hw.flags &
+ (PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW))
+ src.mem_op = P(OP, STORE);
+
+ return src.val;
}
u64 grt_latency_data(struct perf_event *event, u64 status)
@@ -373,7 +383,11 @@ static u64 lnc_latency_data(struct perf_event *event, u64 status)
val |= P(BLK, NA);
src.val = val;
- if (event->hw.flags & PERF_X86_EVENT_PEBS_ST_HSW)
+ if (event->hw.flags &
+ (PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW))
+ src.mem_op = P(OP, LOAD);
+ if (event->hw.flags &
+ (PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW))
src.mem_op = P(OP, STORE);
return src.val;
@@ -981,8 +995,8 @@ struct event_constraint intel_glm_pebs_event_constraints[] = {
struct event_constraint intel_grt_pebs_event_constraints[] = {
/* Allow all events as PEBS with no flags */
- INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0x3),
- INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xf),
+ INTEL_HYBRID_LDLAT_CONSTRAINT(0x5d0, 0x3),
+ INTEL_HYBRID_STLAT_CONSTRAINT(0x6d0, 0x3f),
EVENT_CONSTRAINT_END
};
@@ -1179,19 +1193,11 @@ struct event_constraint intel_lnc_pebs_event_constraints[] = {
INTEL_FLAGS_UEVENT_CONSTRAINT(0x100, 0x100000000ULL), /* INST_RETIRED.PREC_DIST */
INTEL_FLAGS_UEVENT_CONSTRAINT(0x0400, 0x800000000ULL),
+ INTEL_FLAGS_UEVENT_CONSTRAINT(0x012a, 0x1), /* OCR.* events */
+ INTEL_FLAGS_UEVENT_CONSTRAINT(0x012b, 0x1), /* OCR.* events */
+
INTEL_HYBRID_LDLAT_CONSTRAINT(0x1cd, 0x3fc),
INTEL_HYBRID_STLAT_CONSTRAINT(0x2cd, 0x3),
- INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x11d0, 0xf), /* MEM_INST_RETIRED.STLB_MISS_LOADS */
- INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x12d0, 0xf), /* MEM_INST_RETIRED.STLB_MISS_STORES */
- INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x21d0, 0xf), /* MEM_INST_RETIRED.LOCK_LOADS */
- INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x41d0, 0xf), /* MEM_INST_RETIRED.SPLIT_LOADS */
- INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x42d0, 0xf), /* MEM_INST_RETIRED.SPLIT_STORES */
- INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_LD(0x81d0, 0xf), /* MEM_INST_RETIRED.ALL_LOADS */
- INTEL_FLAGS_UEVENT_CONSTRAINT_DATALA_ST(0x82d0, 0xf), /* MEM_INST_RETIRED.ALL_STORES */
-
- INTEL_FLAGS_EVENT_CONSTRAINT_DATALA_LD_RANGE(0xd1, 0xd4, 0xf),
-
- INTEL_FLAGS_EVENT_CONSTRAINT(0xd0, 0xf),
/*
* Everything else is handled by PMU_FL_PEBS_ALL, because we
diff --git a/arch/x86/events/intel/pt.c b/arch/x86/events/intel/pt.c
index 86f0b17446e6..1974272ab128 100644
--- a/arch/x86/events/intel/pt.c
+++ b/arch/x86/events/intel/pt.c
@@ -1896,6 +1896,8 @@ static __init int pt_init(void)
if (!intel_pt_validate_hw_cap(PT_CAP_topa_multiple_entries))
pt_pmu.pmu.capabilities = PERF_PMU_CAP_AUX_NO_SG;
+ else
+ pt_pmu.pmu.capabilities = PERF_PMU_CAP_AUX_PREFER_LARGE;
pt_pmu.pmu.capabilities |= PERF_PMU_CAP_EXCLUSIVE |
PERF_PMU_CAP_ITRACE |
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index c4c0a897fdd8..3878173933be 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -630,7 +630,7 @@ union perf_capabilities {
u64 perf_metrics:1;
u64 pebs_output_pt_available:1;
u64 pebs_timing_info:1;
- u64 anythread_deprecated:1;
+ u64 __reserved:1;
};
u64 capabilities;
};
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index bb04de781b69..565582cd9639 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1307,7 +1307,7 @@ struct kvm_arch {
bool has_private_mem;
bool has_protected_state;
bool pre_fault_allowed;
- struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
+ struct hlist_head *mmu_page_hash;
struct list_head active_mmu_pages;
struct list_head zapped_obsolete_pages;
/*
@@ -1954,7 +1954,7 @@ void kvm_mmu_vendor_module_exit(void);
void kvm_mmu_destroy(struct kvm_vcpu *vcpu);
int kvm_mmu_create(struct kvm_vcpu *vcpu);
-void kvm_mmu_init_vm(struct kvm *kvm);
+int kvm_mmu_init_vm(struct kvm *kvm);
void kvm_mmu_uninit_vm(struct kvm *kvm);
void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index 01a17e729a80..7be30b377578 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -2135,6 +2135,9 @@ bool filter_mce(struct mce *m)
static __always_inline void exc_machine_check_kernel(struct pt_regs *regs)
{
irqentry_state_t irq_state;
+ unsigned long dr7;
+
+ dr7 = local_db_save();
WARN_ON_ONCE(user_mode(regs));
@@ -2143,20 +2146,26 @@ static __always_inline void exc_machine_check_kernel(struct pt_regs *regs)
* mce_check_crashing_cpu() for details.
*/
if (mca_cfg.initialized && mce_check_crashing_cpu())
- return;
+ goto out;
irq_state = irqentry_nmi_enter(regs);
do_machine_check(regs);
irqentry_nmi_exit(regs, irq_state);
+out:
+ local_db_restore(dr7);
}
static __always_inline void exc_machine_check_user(struct pt_regs *regs)
{
+ unsigned long dr7;
+
irqentry_enter_from_user_mode(regs);
+ dr7 = local_db_save();
do_machine_check(regs);
+ local_db_restore(dr7);
irqentry_exit_to_user_mode(regs);
}
@@ -2165,21 +2174,13 @@ static __always_inline void exc_machine_check_user(struct pt_regs *regs)
/* MCE hit kernel mode */
DEFINE_IDTENTRY_MCE(exc_machine_check)
{
- unsigned long dr7;
-
- dr7 = local_db_save();
exc_machine_check_kernel(regs);
- local_db_restore(dr7);
}
/* The user mode variant. */
DEFINE_IDTENTRY_MCE_USER(exc_machine_check)
{
- unsigned long dr7;
-
- dr7 = local_db_save();
exc_machine_check_user(regs);
- local_db_restore(dr7);
}
#ifdef CONFIG_X86_FRED
@@ -2196,28 +2197,20 @@ DEFINE_IDTENTRY_MCE_USER(exc_machine_check)
*/
DEFINE_FREDENTRY_MCE(exc_machine_check)
{
- unsigned long dr7;
-
- dr7 = local_db_save();
if (user_mode(regs))
exc_machine_check_user(regs);
else
exc_machine_check_kernel(regs);
- local_db_restore(dr7);
}
#endif
#else
/* 32bit unified entry point */
DEFINE_IDTENTRY_RAW(exc_machine_check)
{
- unsigned long dr7;
-
- dr7 = local_db_save();
if (user_mode(regs))
exc_machine_check_user(regs);
else
exc_machine_check_kernel(regs);
- local_db_restore(dr7);
}
#endif
diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c
index 362cc71bbc86..0e203dff1f92 100644
--- a/arch/x86/kernel/cpu/microcode/intel.c
+++ b/arch/x86/kernel/cpu/microcode/intel.c
@@ -257,6 +257,26 @@ static void save_microcode_patch(struct microcode_intel *patch)
pr_err("Unable to allocate microcode memory size: %u\n", size);
}
+static bool revision_is_safe(struct cpu_signature *sig, u32 rev)
+{
+ u32 vfm = IFM(x86_family(sig->sig), x86_model(sig->sig));
+
+ /*
+ * Erratum GNR98 can cause #MCs if "jumping over" revision 0x1000405.
+ * Avoid the jumps.
+ */
+ if (vfm == INTEL_GRANITERAPIDS_X &&
+ x86_stepping(sig->sig) == 1 &&
+ sig->pf & 0x95 &&
+ sig->rev < 0x1000405 &&
+ rev > 0x1000405) {
+ pr_err_once("Erratum GNR98: skipping revision 0x%x.\n", rev);
+ return false;
+ }
+
+ return true;
+}
+
/* Scan blob for microcode matching the boot CPUs family, model, stepping */
static __init struct microcode_intel *scan_microcode(void *data, size_t size,
struct ucode_cpu_info *uci,
@@ -278,6 +298,9 @@ static __init struct microcode_intel *scan_microcode(void *data, size_t size,
if (!intel_find_matching_signature(data, &uci->cpu_sig))
continue;
+ if (!revision_is_safe(&uci->cpu_sig, mc_header->rev))
+ continue;
+
/*
* For saving the early microcode, find the matching revision which
* was loaded on the BSP.
@@ -540,6 +563,9 @@ static enum ucode_state parse_microcode_blobs(int cpu, struct iov_iter *iter)
if (!intel_find_matching_signature(mc, &uci->cpu_sig))
continue;
+ if (!revision_is_safe(&uci->cpu_sig, mc_header.rev))
+ continue;
+
is_safe = ucode_validate_minrev(&mc_header);
if (force_minrev && !is_safe)
continue;
diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index f9dddb8cb466..8e5b4e29d482 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -6,7 +6,7 @@ ccflags-$(CONFIG_KVM_WERROR) += -Werror
include $(srctree)/virt/kvm/Makefile.kvm
kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \
- i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \
+ i8254.o ioapic.o irq_comm.o cpuid.o pmu.o regs.o mtrr.o \
debugfs.o mmu/mmu.o mmu/page_track.o \
mmu/spte.o
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index ed0cc781f1c7..86c903f80d93 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -3772,6 +3772,18 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)
return r;
}
+static int kvm_mmu_alloc_page_hash(struct kvm *kvm)
+{
+ struct hlist_head *h;
+
+ h = kvcalloc(KVM_NUM_MMU_PAGES, sizeof(*h), GFP_KERNEL_ACCOUNT);
+ if (!h)
+ return -ENOMEM;
+
+ kvm->arch.mmu_page_hash = h;
+ return 0;
+}
+
static int mmu_first_shadow_root_alloc(struct kvm *kvm)
{
struct kvm_memslots *slots;
@@ -6477,20 +6489,11 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)
kvm_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages);
}
-/*
- * Fast invalidate all shadow pages and use lock-break technique
- * to zap obsolete pages.
- *
- * It's required when memslot is being deleted or VM is being
- * destroyed, in these cases, we should ensure that KVM MMU does
- * not use any resource of the being-deleted slot or all slots
- * after calling the function.
- */
-static void kvm_mmu_zap_all_fast(struct kvm *kvm)
+static void __kvm_mmu_zap_all_fast_front_half(struct kvm *kvm)
{
lockdep_assert_held(&kvm->slots_lock);
+ lockdep_assert_held_write(&kvm->mmu_lock);
- write_lock(&kvm->mmu_lock);
trace_kvm_mmu_zap_all_fast(kvm);
/*
@@ -6522,8 +6525,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_FREE_OBSOLETE_ROOTS);
kvm_zap_obsolete_pages(kvm);
+}
- write_unlock(&kvm->mmu_lock);
+static void __kvm_mmu_zap_all_fast_back_half(struct kvm *kvm)
+{
+ lockdep_assert_held(&kvm->slots_lock);
+ lockdep_assert_not_held(&kvm->mmu_lock);
/*
* Zap the invalidated TDP MMU roots, all SPTEs must be dropped before
@@ -6542,14 +6549,38 @@ static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm)
return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages));
}
-void kvm_mmu_init_vm(struct kvm *kvm)
+/*
+ * Fast invalidate all shadow pages and use lock-break technique
+ * to zap obsolete pages.
+ *
+ * It's required when memslot is being deleted or VM is being
+ * destroyed, in these cases, we should ensure that KVM MMU does
+ * not use any resource of the being-deleted slot or all slots
+ * after calling the function.
+ */
+static void kvm_mmu_zap_all_fast(struct kvm *kvm)
{
+ write_lock(&kvm->mmu_lock);
+ __kvm_mmu_zap_all_fast_front_half(kvm);
+ write_unlock(&kvm->mmu_lock);
+
+ __kvm_mmu_zap_all_fast_back_half(kvm);
+}
+
+int kvm_mmu_init_vm(struct kvm *kvm)
+{
+ int r;
+
kvm->arch.shadow_mmio_value = shadow_mmio_value;
INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
INIT_LIST_HEAD(&kvm->arch.zapped_obsolete_pages);
INIT_LIST_HEAD(&kvm->arch.possible_nx_huge_pages);
spin_lock_init(&kvm->arch.mmu_unsync_pages_lock);
+ r = kvm_mmu_alloc_page_hash(kvm);
+ if (r)
+ return r;
+
if (tdp_mmu_enabled)
kvm_mmu_init_tdp_mmu(kvm);
@@ -6560,6 +6591,7 @@ void kvm_mmu_init_vm(struct kvm *kvm)
kvm->arch.split_desc_cache.kmem_cache = pte_list_desc_cache;
kvm->arch.split_desc_cache.gfp_zero = __GFP_ZERO;
+ return 0;
}
static void mmu_free_vm_memory_caches(struct kvm *kvm)
@@ -6571,6 +6603,8 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
void kvm_mmu_uninit_vm(struct kvm *kvm)
{
+ kvfree(kvm->arch.mmu_page_hash);
+
if (tdp_mmu_enabled)
kvm_mmu_uninit_tdp_mmu(kvm);
diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
index a26e8f5ad790..a0efe59aa079 100644
--- a/arch/x86/kvm/pmu.c
+++ b/arch/x86/kvm/pmu.c
@@ -614,14 +614,6 @@ void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu)
bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
{
- switch (msr) {
- case MSR_CORE_PERF_GLOBAL_STATUS:
- case MSR_CORE_PERF_GLOBAL_CTRL:
- case MSR_CORE_PERF_GLOBAL_OVF_CTRL:
- return kvm_pmu_has_perf_global_ctrl(vcpu_to_pmu(vcpu));
- default:
- break;
- }
return kvm_pmu_call(msr_idx_to_pmc)(vcpu, msr) ||
kvm_pmu_call(is_valid_msr)(vcpu, msr);
}
diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c
new file mode 100644
index 000000000000..26ee933ef549
--- /dev/null
+++ b/arch/x86/kvm/regs.c
@@ -0,0 +1,872 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/kvm_host.h>
+
+#include "lapic.h"
+#include "mmu.h"
+#include "kvm_cache_regs.h"
+#include "x86.h"
+
+unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu)
+{
+ /* Can't read the RIP when guest state is protected, just return 0 */
+ if (vcpu->arch.guest_state_protected)
+ return 0;
+
+ if (is_64_bit_mode(vcpu))
+ return kvm_rip_read(vcpu);
+ return (u32)(get_segment_base(vcpu, VCPU_SREG_CS) +
+ kvm_rip_read(vcpu));
+}
+EXPORT_SYMBOL_GPL(kvm_get_linear_rip);
+
+bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip)
+{
+ return kvm_get_linear_rip(vcpu) == linear_rip;
+}
+EXPORT_SYMBOL_GPL(kvm_is_linear_rip);
+
+unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu)
+{
+ unsigned long rflags;
+
+ rflags = kvm_x86_call(get_rflags)(vcpu);
+ if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
+ rflags &= ~X86_EFLAGS_TF;
+ return rflags;
+}
+EXPORT_SYMBOL_GPL(kvm_get_rflags);
+
+void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
+{
+ if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP &&
+ kvm_is_linear_rip(vcpu, vcpu->arch.singlestep_rip))
+ rflags |= X86_EFLAGS_TF;
+ kvm_x86_call(set_rflags)(vcpu, rflags);
+}
+
+void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
+{
+ __kvm_set_rflags(vcpu, rflags);
+ kvm_make_request(KVM_REQ_EVENT, vcpu);
+}
+EXPORT_SYMBOL_GPL(kvm_set_rflags);
+
+static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
+{
+ if (vcpu->arch.emulate_regs_need_sync_to_vcpu) {
+ /*
+ * We are here if userspace calls get_regs() in the middle of
+ * instruction emulation. Registers state needs to be copied
+ * back from emulation context to vcpu. Userspace shouldn't do
+ * that usually, but some bad designed PV devices (vmware
+ * backdoor interface) need this to work
+ */
+ emulator_writeback_register_cache(vcpu->arch.emulate_ctxt);
+ vcpu->arch.emulate_regs_need_sync_to_vcpu = false;
+ }
+ regs->rax = kvm_rax_read(vcpu);
+ regs->rbx = kvm_rbx_read(vcpu);
+ regs->rcx = kvm_rcx_read(vcpu);
+ regs->rdx = kvm_rdx_read(vcpu);
+ regs->rsi = kvm_rsi_read(vcpu);
+ regs->rdi = kvm_rdi_read(vcpu);
+ regs->rsp = kvm_rsp_read(vcpu);
+ regs->rbp = kvm_rbp_read(vcpu);
+#ifdef CONFIG_X86_64
+ regs->r8 = kvm_r8_read(vcpu);
+ regs->r9 = kvm_r9_read(vcpu);
+ regs->r10 = kvm_r10_read(vcpu);
+ regs->r11 = kvm_r11_read(vcpu);
+ regs->r12 = kvm_r12_read(vcpu);
+ regs->r13 = kvm_r13_read(vcpu);
+ regs->r14 = kvm_r14_read(vcpu);
+ regs->r15 = kvm_r15_read(vcpu);
+#endif
+
+ regs->rip = kvm_rip_read(vcpu);
+ regs->rflags = kvm_get_rflags(vcpu);
+}
+
+int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
+{
+ if (vcpu->kvm->arch.has_protected_state &&
+ vcpu->arch.guest_state_protected)
+ return -EINVAL;
+
+ vcpu_load(vcpu);
+ __get_regs(vcpu, regs);
+ vcpu_put(vcpu);
+ return 0;
+}
+
+static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
+{
+ vcpu->arch.emulate_regs_need_sync_from_vcpu = true;
+ vcpu->arch.emulate_regs_need_sync_to_vcpu = false;
+
+ kvm_rax_write(vcpu, regs->rax);
+ kvm_rbx_write(vcpu, regs->rbx);
+ kvm_rcx_write(vcpu, regs->rcx);
+ kvm_rdx_write(vcpu, regs->rdx);
+ kvm_rsi_write(vcpu, regs->rsi);
+ kvm_rdi_write(vcpu, regs->rdi);
+ kvm_rsp_write(vcpu, regs->rsp);
+ kvm_rbp_write(vcpu, regs->rbp);
+#ifdef CONFIG_X86_64
+ kvm_r8_write(vcpu, regs->r8);
+ kvm_r9_write(vcpu, regs->r9);
+ kvm_r10_write(vcpu, regs->r10);
+ kvm_r11_write(vcpu, regs->r11);
+ kvm_r12_write(vcpu, regs->r12);
+ kvm_r13_write(vcpu, regs->r13);
+ kvm_r14_write(vcpu, regs->r14);
+ kvm_r15_write(vcpu, regs->r15);
+#endif
+
+ kvm_rip_write(vcpu, regs->rip);
+ kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED);
+
+ vcpu->arch.exception.pending = false;
+ vcpu->arch.exception_vmexit.pending = false;
+
+ kvm_make_request(KVM_REQ_EVENT, vcpu);
+}
+
+int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
+{
+ if (vcpu->kvm->arch.has_protected_state &&
+ vcpu->arch.guest_state_protected)
+ return -EINVAL;
+
+ vcpu_load(vcpu);
+ __set_regs(vcpu, regs);
+ vcpu_put(vcpu);
+ return 0;
+}
+
+static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu)
+{
+ return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2);
+}
+
+/*
+ * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise.
+ */
+int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
+{
+ struct kvm_mmu *mmu = vcpu->arch.walk_mmu;
+ gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT;
+ gpa_t real_gpa;
+ int i;
+ int ret;
+ u64 pdpte[ARRAY_SIZE(mmu->pdptrs)];
+
+ /*
+ * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated
+ * to an L1 GPA.
+ */
+ real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn),
+ PFERR_USER_MASK | PFERR_WRITE_MASK, NULL);
+ if (real_gpa == INVALID_GPA)
+ return 0;
+
+ /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */
+ ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte,
+ cr3 & GENMASK(11, 5), sizeof(pdpte));
+ if (ret < 0)
+ return 0;
+
+ for (i = 0; i < ARRAY_SIZE(pdpte); ++i) {
+ if ((pdpte[i] & PT_PRESENT_MASK) &&
+ (pdpte[i] & pdptr_rsvd_bits(vcpu))) {
+ return 0;
+ }
+ }
+
+ /*
+ * Marking VCPU_EXREG_PDPTR dirty doesn't work for !tdp_enabled.
+ * Shadow page roots need to be reconstructed instead.
+ */
+ if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
+ kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT);
+
+ memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
+ kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
+ kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+ vcpu->arch.pdptrs_from_userspace = false;
+
+ return 1;
+}
+EXPORT_SYMBOL_GPL(load_pdptrs);
+
+static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
+{
+#ifdef CONFIG_X86_64
+ if (cr0 & 0xffffffff00000000UL)
+ return false;
+#endif
+
+ if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD))
+ return false;
+
+ if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE))
+ return false;
+
+ return kvm_x86_call(is_valid_cr0)(vcpu, cr0);
+}
+
+void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0)
+{
+ /*
+ * CR0.WP is incorporated into the MMU role, but only for non-nested,
+ * indirect shadow MMUs. If paging is disabled, no updates are needed
+ * as there are no permission bits to emulate. If TDP is enabled, the
+ * MMU's metadata needs to be updated, e.g. so that emulating guest
+ * translations does the right thing, but there's no need to unload the
+ * root as CR0.WP doesn't affect SPTEs.
+ */
+ if ((cr0 ^ old_cr0) == X86_CR0_WP) {
+ if (!(cr0 & X86_CR0_PG))
+ return;
+
+ if (tdp_enabled) {
+ kvm_init_mmu(vcpu);
+ return;
+ }
+ }
+
+ if ((cr0 ^ old_cr0) & X86_CR0_PG) {
+ /*
+ * Clearing CR0.PG is defined to flush the TLB from the guest's
+ * perspective.
+ */
+ if (!(cr0 & X86_CR0_PG))
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+ /*
+ * Check for async #PF completion events when enabling paging,
+ * as the vCPU may have previously encountered async #PFs (it's
+ * entirely legal for the guest to toggle paging on/off without
+ * waiting for the async #PF queue to drain).
+ */
+ else if (kvm_pv_async_pf_enabled(vcpu))
+ kvm_make_request(KVM_REQ_APF_READY, vcpu);
+ }
+
+ if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS)
+ kvm_mmu_reset_context(vcpu);
+}
+EXPORT_SYMBOL_GPL(kvm_post_set_cr0);
+
+int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
+{
+ unsigned long old_cr0 = kvm_read_cr0(vcpu);
+
+ if (!kvm_is_valid_cr0(vcpu, cr0))
+ return 1;
+
+ cr0 |= X86_CR0_ET;
+
+ /* Write to CR0 reserved bits are ignored, even on Intel. */
+ cr0 &= ~CR0_RESERVED_BITS;
+
+#ifdef CONFIG_X86_64
+ if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) &&
+ (cr0 & X86_CR0_PG)) {
+ int cs_db, cs_l;
+
+ if (!is_pae(vcpu))
+ return 1;
+ kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l);
+ if (cs_l)
+ return 1;
+ }
+#endif
+ if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) &&
+ is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) &&
+ !load_pdptrs(vcpu, kvm_read_cr3(vcpu)))
+ return 1;
+
+ if (!(cr0 & X86_CR0_PG) &&
+ (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)))
+ return 1;
+
+ kvm_x86_call(set_cr0)(vcpu, cr0);
+
+ kvm_post_set_cr0(vcpu, old_cr0, cr0);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_set_cr0);
+
+void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw)
+{
+ (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f));
+}
+EXPORT_SYMBOL_GPL(kvm_lmsw);
+
+int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
+{
+ bool skip_tlb_flush = false;
+ unsigned long pcid = 0;
+#ifdef CONFIG_X86_64
+ if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) {
+ skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH;
+ cr3 &= ~X86_CR3_PCID_NOFLUSH;
+ pcid = cr3 & X86_CR3_PCID_MASK;
+ }
+#endif
+
+ /* PDPTRs are always reloaded for PAE paging. */
+ if (cr3 == kvm_read_cr3(vcpu) && !is_pae_paging(vcpu))
+ goto handle_tlb_flush;
+
+ /*
+ * Do not condition the GPA check on long mode, this helper is used to
+ * stuff CR3, e.g. for RSM emulation, and there is no guarantee that
+ * the current vCPU mode is accurate.
+ */
+ if (!kvm_vcpu_is_legal_cr3(vcpu, cr3))
+ return 1;
+
+ if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3))
+ return 1;
+
+ if (cr3 != kvm_read_cr3(vcpu))
+ kvm_mmu_new_pgd(vcpu, cr3);
+
+ vcpu->arch.cr3 = cr3;
+ kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
+ /* Do not call post_set_cr3, we do not get here for confidential guests. */
+
+handle_tlb_flush:
+ /*
+ * A load of CR3 that flushes the TLB flushes only the current PCID,
+ * even if PCID is disabled, in which case PCID=0 is flushed. It's a
+ * moot point in the end because _disabling_ PCID will flush all PCIDs,
+ * and it's impossible to use a non-zero PCID when PCID is disabled,
+ * i.e. only PCID=0 can be relevant.
+ */
+ if (!skip_tlb_flush)
+ kvm_invalidate_pcid(vcpu, pcid);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_set_cr3);
+
+static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
+{
+ return __kvm_is_valid_cr4(vcpu, cr4) &&
+ kvm_x86_call(is_valid_cr4)(vcpu, cr4);
+}
+
+void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4)
+{
+ if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS)
+ kvm_mmu_reset_context(vcpu);
+
+ /*
+ * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB
+ * according to the SDM; however, stale prev_roots could be reused
+ * incorrectly in the future after a MOV to CR3 with NOFLUSH=1, so we
+ * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST
+ * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed,
+ * so fall through.
+ */
+ if (!tdp_enabled &&
+ (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE))
+ kvm_mmu_unload(vcpu);
+
+ /*
+ * The TLB has to be flushed for all PCIDs if any of the following
+ * (architecturally required) changes happen:
+ * - CR4.PCIDE is changed from 1 to 0
+ * - CR4.PGE is toggled
+ *
+ * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT.
+ */
+ if (((cr4 ^ old_cr4) & X86_CR4_PGE) ||
+ (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE)))
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+
+ /*
+ * The TLB has to be flushed for the current PCID if any of the
+ * following (architecturally required) changes happen:
+ * - CR4.SMEP is changed from 0 to 1
+ * - CR4.PAE is toggled
+ */
+ else if (((cr4 ^ old_cr4) & X86_CR4_PAE) ||
+ ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP)))
+ kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+
+}
+EXPORT_SYMBOL_GPL(kvm_post_set_cr4);
+
+int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
+{
+ unsigned long old_cr4 = kvm_read_cr4(vcpu);
+
+ if (!kvm_is_valid_cr4(vcpu, cr4))
+ return 1;
+
+ if (is_long_mode(vcpu)) {
+ if (!(cr4 & X86_CR4_PAE))
+ return 1;
+ if ((cr4 ^ old_cr4) & X86_CR4_LA57)
+ return 1;
+ } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE)
+ && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS)
+ && !load_pdptrs(vcpu, kvm_read_cr3(vcpu)))
+ return 1;
+
+ if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) {
+ /* PCID can not be enabled when cr3[11:0]!=000H or EFER.LMA=0 */
+ if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu))
+ return 1;
+ }
+
+ kvm_x86_call(set_cr4)(vcpu, cr4);
+
+ kvm_post_set_cr4(vcpu, old_cr4, cr4);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_set_cr4);
+
+int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8)
+{
+ if (cr8 & CR8_RESERVED_BITS)
+ return 1;
+ if (lapic_in_kernel(vcpu))
+ kvm_lapic_set_tpr(vcpu, cr8);
+ else
+ vcpu->arch.cr8 = cr8;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_set_cr8);
+
+unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu)
+{
+ if (lapic_in_kernel(vcpu))
+ return kvm_lapic_get_cr8(vcpu);
+ else
+ return vcpu->arch.cr8;
+}
+EXPORT_SYMBOL_GPL(kvm_get_cr8);
+
+static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
+{
+ struct desc_ptr dt;
+
+ if (vcpu->arch.guest_state_protected)
+ goto skip_protected_regs;
+
+ kvm_handle_exception_payload_quirk(vcpu);
+
+ kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS);
+ kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS);
+ kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES);
+ kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS);
+ kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS);
+ kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS);
+
+ kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR);
+ kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR);
+
+ kvm_x86_call(get_idt)(vcpu, &dt);
+ sregs->idt.limit = dt.size;
+ sregs->idt.base = dt.address;
+ kvm_x86_call(get_gdt)(vcpu, &dt);
+ sregs->gdt.limit = dt.size;
+ sregs->gdt.base = dt.address;
+
+ sregs->cr2 = vcpu->arch.cr2;
+ sregs->cr3 = kvm_read_cr3(vcpu);
+
+skip_protected_regs:
+ sregs->cr0 = kvm_read_cr0(vcpu);
+ sregs->cr4 = kvm_read_cr4(vcpu);
+ sregs->cr8 = kvm_get_cr8(vcpu);
+ sregs->efer = vcpu->arch.efer;
+ sregs->apic_base = vcpu->arch.apic_base;
+}
+
+static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
+{
+ __get_sregs_common(vcpu, sregs);
+
+ if (vcpu->arch.guest_state_protected)
+ return;
+
+ if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft)
+ set_bit(vcpu->arch.interrupt.nr,
+ (unsigned long *)sregs->interrupt_bitmap);
+}
+
+int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
+ struct kvm_sregs *sregs)
+{
+ if (vcpu->kvm->arch.has_protected_state &&
+ vcpu->arch.guest_state_protected)
+ return -EINVAL;
+
+ vcpu_load(vcpu);
+ __get_sregs(vcpu, sregs);
+ vcpu_put(vcpu);
+ return 0;
+}
+
+void kvm_vcpu_ioctl_x86_get_sregs2(struct kvm_vcpu *vcpu,
+ struct kvm_sregs2 *sregs2)
+{
+ int i;
+
+ __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2);
+
+ if (vcpu->arch.guest_state_protected)
+ return;
+
+ if (is_pae_paging(vcpu)) {
+ kvm_vcpu_srcu_read_lock(vcpu);
+ for (i = 0 ; i < 4 ; i++)
+ sregs2->pdptrs[i] = kvm_pdptr_read(vcpu, i);
+ sregs2->flags |= KVM_SREGS2_FLAGS_PDPTRS_VALID;
+ kvm_vcpu_srcu_read_unlock(vcpu);
+ }
+}
+
+static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
+{
+ if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) {
+ /*
+ * When EFER.LME and CR0.PG are set, the processor is in
+ * 64-bit mode (though maybe in a 32-bit code segment).
+ * CR4.PAE and EFER.LMA must be set.
+ */
+ if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA))
+ return false;
+ if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3))
+ return false;
+ } else {
+ /*
+ * Not in 64-bit mode: EFER.LMA is clear and the code
+ * segment cannot be 64-bit.
+ */
+ if (sregs->efer & EFER_LMA || sregs->cs.l)
+ return false;
+ }
+
+ return kvm_is_valid_cr4(vcpu, sregs->cr4) &&
+ kvm_is_valid_cr0(vcpu, sregs->cr0) &&
+ kvm_valid_efer(vcpu, sregs->efer);
+}
+
+static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
+ int *mmu_reset_needed, bool update_pdptrs)
+{
+ struct msr_data apic_base_msr;
+ int idx;
+ struct desc_ptr dt;
+
+ if (!kvm_is_valid_sregs(vcpu, sregs))
+ return -EINVAL;
+
+ apic_base_msr.data = sregs->apic_base;
+ apic_base_msr.host_initiated = true;
+ if (kvm_set_apic_base(vcpu, &apic_base_msr))
+ return -EINVAL;
+
+ if (vcpu->arch.guest_state_protected)
+ return 0;
+
+ dt.size = sregs->idt.limit;
+ dt.address = sregs->idt.base;
+ kvm_x86_call(set_idt)(vcpu, &dt);
+ dt.size = sregs->gdt.limit;
+ dt.address = sregs->gdt.base;
+ kvm_x86_call(set_gdt)(vcpu, &dt);
+
+ vcpu->arch.cr2 = sregs->cr2;
+ *mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3;
+ vcpu->arch.cr3 = sregs->cr3;
+ kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
+ kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3);
+
+ kvm_set_cr8(vcpu, sregs->cr8);
+
+ *mmu_reset_needed |= vcpu->arch.efer != sregs->efer;
+ kvm_x86_call(set_efer)(vcpu, sregs->efer);
+
+ *mmu_reset_needed |= kvm_read_cr0(vcpu) != sregs->cr0;
+ kvm_x86_call(set_cr0)(vcpu, sregs->cr0);
+
+ *mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4;
+ kvm_x86_call(set_cr4)(vcpu, sregs->cr4);
+
+ if (update_pdptrs) {
+ idx = srcu_read_lock(&vcpu->kvm->srcu);
+ if (is_pae_paging(vcpu)) {
+ load_pdptrs(vcpu, kvm_read_cr3(vcpu));
+ *mmu_reset_needed = 1;
+ }
+ srcu_read_unlock(&vcpu->kvm->srcu, idx);
+ }
+
+ kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS);
+ kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS);
+ kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES);
+ kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS);
+ kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS);
+ kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS);
+
+ kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR);
+ kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR);
+
+ update_cr8_intercept(vcpu);
+
+ /* Older userspace won't unhalt the vcpu on reset. */
+ if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 &&
+ sregs->cs.selector == 0xf000 && sregs->cs.base == 0xffff0000 &&
+ !is_protmode(vcpu))
+ vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
+
+ return 0;
+}
+
+static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
+{
+ int pending_vec, max_bits;
+ int mmu_reset_needed = 0;
+ int ret = __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true);
+
+ if (ret)
+ return ret;
+
+ if (mmu_reset_needed) {
+ kvm_mmu_reset_context(vcpu);
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+ }
+
+ max_bits = KVM_NR_INTERRUPTS;
+ pending_vec = find_first_bit(
+ (const unsigned long *)sregs->interrupt_bitmap, max_bits);
+
+ if (pending_vec < max_bits) {
+ kvm_queue_interrupt(vcpu, pending_vec, false);
+ pr_debug("Set back pending irq %d\n", pending_vec);
+ kvm_make_request(KVM_REQ_EVENT, vcpu);
+ }
+ return 0;
+}
+
+int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
+ struct kvm_sregs *sregs)
+{
+ int ret;
+
+ if (vcpu->kvm->arch.has_protected_state &&
+ vcpu->arch.guest_state_protected)
+ return -EINVAL;
+
+ vcpu_load(vcpu);
+ ret = __set_sregs(vcpu, sregs);
+ vcpu_put(vcpu);
+ return ret;
+}
+
+int kvm_vcpu_ioctl_x86_set_sregs2(struct kvm_vcpu *vcpu,
+ struct kvm_sregs2 *sregs2)
+{
+ int mmu_reset_needed = 0;
+ bool valid_pdptrs = sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID;
+ bool pae = (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) &&
+ !(sregs2->efer & EFER_LMA);
+ int i, ret;
+
+ if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID)
+ return -EINVAL;
+
+ if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected))
+ return -EINVAL;
+
+ ret = __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2,
+ &mmu_reset_needed, !valid_pdptrs);
+ if (ret)
+ return ret;
+
+ if (valid_pdptrs) {
+ for (i = 0; i < 4 ; i++)
+ kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]);
+
+ kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
+ mmu_reset_needed = 1;
+ vcpu->arch.pdptrs_from_userspace = true;
+ }
+ if (mmu_reset_needed) {
+ kvm_mmu_reset_context(vcpu);
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+ }
+ return 0;
+}
+
+void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu)
+{
+ BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES);
+
+ if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS)
+ __get_regs(vcpu, &vcpu->run->s.regs.regs);
+
+ if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS)
+ __get_sregs(vcpu, &vcpu->run->s.regs.sregs);
+}
+
+int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu)
+{
+ if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) {
+ __set_regs(vcpu, &vcpu->run->s.regs.regs);
+ vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_REGS;
+ }
+
+ if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) {
+ struct kvm_sregs sregs = vcpu->run->s.regs.sregs;
+
+ if (__set_sregs(vcpu, &sregs))
+ return -EINVAL;
+
+ vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_SREGS;
+ }
+
+ return 0;
+}
+
+void kvm_update_dr0123(struct kvm_vcpu *vcpu)
+{
+ int i;
+
+ if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) {
+ for (i = 0; i < KVM_NR_DB_REGS; i++)
+ vcpu->arch.eff_db[i] = vcpu->arch.db[i];
+ }
+}
+
+void kvm_update_dr7(struct kvm_vcpu *vcpu)
+{
+ unsigned long dr7;
+
+ if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)
+ dr7 = vcpu->arch.guest_debug_dr7;
+ else
+ dr7 = vcpu->arch.dr7;
+ kvm_x86_call(set_dr7)(vcpu, dr7);
+ vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_BP_ENABLED;
+ if (dr7 & DR7_BP_EN_MASK)
+ vcpu->arch.switch_db_regs |= KVM_DEBUGREG_BP_ENABLED;
+}
+EXPORT_SYMBOL_GPL(kvm_update_dr7);
+
+static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu)
+{
+ u64 fixed = DR6_FIXED_1;
+
+ if (!guest_cpuid_has(vcpu, X86_FEATURE_RTM))
+ fixed |= DR6_RTM;
+
+ if (!guest_cpuid_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT))
+ fixed |= DR6_BUS_LOCK;
+ return fixed;
+}
+
+int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val)
+{
+ size_t size = ARRAY_SIZE(vcpu->arch.db);
+
+ switch (dr) {
+ case 0 ... 3:
+ vcpu->arch.db[array_index_nospec(dr, size)] = val;
+ if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP))
+ vcpu->arch.eff_db[dr] = val;
+ break;
+ case 4:
+ case 6:
+ if (!kvm_dr6_valid(val))
+ return 1; /* #GP */
+ vcpu->arch.dr6 = (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu);
+ break;
+ case 5:
+ default: /* 7 */
+ if (!kvm_dr7_valid(val))
+ return 1; /* #GP */
+ vcpu->arch.dr7 = (val & DR7_VOLATILE) | DR7_FIXED_1;
+ kvm_update_dr7(vcpu);
+ break;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvm_set_dr);
+
+unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr)
+{
+ size_t size = ARRAY_SIZE(vcpu->arch.db);
+
+ switch (dr) {
+ case 0 ... 3:
+ return vcpu->arch.db[array_index_nospec(dr, size)];
+ case 4:
+ case 6:
+ return vcpu->arch.dr6;
+ case 5:
+ default: /* 7 */
+ return vcpu->arch.dr7;
+ }
+}
+EXPORT_SYMBOL_GPL(kvm_get_dr);
+
+int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu,
+ struct kvm_debugregs *dbgregs)
+{
+ unsigned int i;
+
+ if (vcpu->kvm->arch.has_protected_state &&
+ vcpu->arch.guest_state_protected)
+ return -EINVAL;
+
+ kvm_handle_exception_payload_quirk(vcpu);
+
+ memset(dbgregs, 0, sizeof(*dbgregs));
+
+ BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) != ARRAY_SIZE(dbgregs->db));
+ for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++)
+ dbgregs->db[i] = vcpu->arch.db[i];
+
+ dbgregs->dr6 = vcpu->arch.dr6;
+ dbgregs->dr7 = vcpu->arch.dr7;
+ return 0;
+}
+
+int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu,
+ struct kvm_debugregs *dbgregs)
+{
+ unsigned int i;
+
+ if (vcpu->kvm->arch.has_protected_state &&
+ vcpu->arch.guest_state_protected)
+ return -EINVAL;
+
+ if (dbgregs->flags)
+ return -EINVAL;
+
+ if (!kvm_dr6_valid(dbgregs->dr6))
+ return -EINVAL;
+ if (!kvm_dr7_valid(dbgregs->dr7))
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++)
+ vcpu->arch.db[i] = dbgregs->db[i];
+
+ kvm_update_dr0123(vcpu);
+ vcpu->arch.dr6 = dbgregs->dr6;
+ vcpu->arch.dr7 = dbgregs->dr7;
+ kvm_update_dr7(vcpu);
+
+ return 0;
+}
diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c
index a5edc623166a..4f6fa564028f 100644
--- a/arch/x86/kvm/vmx/pmu_intel.c
+++ b/arch/x86/kvm/vmx/pmu_intel.c
@@ -154,6 +154,9 @@ static bool intel_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr)
int ret;
switch (msr) {
+ case MSR_CORE_PERF_GLOBAL_STATUS:
+ case MSR_CORE_PERF_GLOBAL_CTRL:
+ case MSR_CORE_PERF_GLOBAL_OVF_CTRL:
case MSR_CORE_PERF_FIXED_CTR_CTRL:
return kvm_pmu_has_perf_global_ctrl(pmu);
case MSR_IA32_PEBS_ENABLE:
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d0c7a7098143..99549df4c633 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -108,16 +108,8 @@ EXPORT_SYMBOL_GPL(kvm_host);
#define emul_to_vcpu(ctxt) \
((struct kvm_vcpu *)(ctxt)->vcpu)
-/* EFER defaults:
- * - enable syscall per default because its emulated by KVM
- * - enable LME and LMA per default on 64 bit KVM
- */
-#ifdef CONFIG_X86_64
-static
-u64 __read_mostly efer_reserved_bits = ~((u64)(EFER_SCE | EFER_LME | EFER_LMA));
-#else
+/* Enable syscall by default because its emulated by KVM */
static u64 __read_mostly efer_reserved_bits = ~((u64)EFER_SCE);
-#endif
static u64 __read_mostly cr4_reserved_bits = CR4_RESERVED_BITS;
@@ -128,16 +120,11 @@ static u64 __read_mostly cr4_reserved_bits = CR4_RESERVED_BITS;
#define KVM_X2APIC_API_VALID_FLAGS (KVM_X2APIC_API_USE_32BIT_IDS | \
KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK)
-static void update_cr8_intercept(struct kvm_vcpu *vcpu);
static void process_nmi(struct kvm_vcpu *vcpu);
-static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);
static void store_regs(struct kvm_vcpu *vcpu);
static int sync_regs(struct kvm_vcpu *vcpu);
static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu);
-static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
-static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
-
static DEFINE_MUTEX(vendor_module_lock);
struct kvm_x86_ops kvm_x86_ops __read_mostly;
@@ -1032,173 +1019,6 @@ bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr)
}
EXPORT_SYMBOL_GPL(kvm_require_dr);
-static bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu)
-{
- u64 mask = KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT;
-
- return (vcpu->arch.apf.msr_en_val & mask) == mask;
-}
-
-static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu)
-{
- return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2);
-}
-
-/*
- * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise.
- */
-int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
-{
- struct kvm_mmu *mmu = vcpu->arch.walk_mmu;
- gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT;
- gpa_t real_gpa;
- int i;
- int ret;
- u64 pdpte[ARRAY_SIZE(mmu->pdptrs)];
-
- /*
- * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated
- * to an L1 GPA.
- */
- real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn),
- PFERR_USER_MASK | PFERR_WRITE_MASK, NULL);
- if (real_gpa == INVALID_GPA)
- return 0;
-
- /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */
- ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte,
- cr3 & GENMASK(11, 5), sizeof(pdpte));
- if (ret < 0)
- return 0;
-
- for (i = 0; i < ARRAY_SIZE(pdpte); ++i) {
- if ((pdpte[i] & PT_PRESENT_MASK) &&
- (pdpte[i] & pdptr_rsvd_bits(vcpu))) {
- return 0;
- }
- }
-
- /*
- * Marking VCPU_EXREG_PDPTR dirty doesn't work for !tdp_enabled.
- * Shadow page roots need to be reconstructed instead.
- */
- if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)))
- kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT);
-
- memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs));
- kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
- kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
- vcpu->arch.pdptrs_from_userspace = false;
-
- return 1;
-}
-EXPORT_SYMBOL_GPL(load_pdptrs);
-
-static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
-{
-#ifdef CONFIG_X86_64
- if (cr0 & 0xffffffff00000000UL)
- return false;
-#endif
-
- if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD))
- return false;
-
- if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE))
- return false;
-
- return kvm_x86_call(is_valid_cr0)(vcpu, cr0);
-}
-
-void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0)
-{
- /*
- * CR0.WP is incorporated into the MMU role, but only for non-nested,
- * indirect shadow MMUs. If paging is disabled, no updates are needed
- * as there are no permission bits to emulate. If TDP is enabled, the
- * MMU's metadata needs to be updated, e.g. so that emulating guest
- * translations does the right thing, but there's no need to unload the
- * root as CR0.WP doesn't affect SPTEs.
- */
- if ((cr0 ^ old_cr0) == X86_CR0_WP) {
- if (!(cr0 & X86_CR0_PG))
- return;
-
- if (tdp_enabled) {
- kvm_init_mmu(vcpu);
- return;
- }
- }
-
- if ((cr0 ^ old_cr0) & X86_CR0_PG) {
- /*
- * Clearing CR0.PG is defined to flush the TLB from the guest's
- * perspective.
- */
- if (!(cr0 & X86_CR0_PG))
- kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
- /*
- * Check for async #PF completion events when enabling paging,
- * as the vCPU may have previously encountered async #PFs (it's
- * entirely legal for the guest to toggle paging on/off without
- * waiting for the async #PF queue to drain).
- */
- else if (kvm_pv_async_pf_enabled(vcpu))
- kvm_make_request(KVM_REQ_APF_READY, vcpu);
- }
-
- if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS)
- kvm_mmu_reset_context(vcpu);
-}
-EXPORT_SYMBOL_GPL(kvm_post_set_cr0);
-
-int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
-{
- unsigned long old_cr0 = kvm_read_cr0(vcpu);
-
- if (!kvm_is_valid_cr0(vcpu, cr0))
- return 1;
-
- cr0 |= X86_CR0_ET;
-
- /* Write to CR0 reserved bits are ignored, even on Intel. */
- cr0 &= ~CR0_RESERVED_BITS;
-
-#ifdef CONFIG_X86_64
- if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) &&
- (cr0 & X86_CR0_PG)) {
- int cs_db, cs_l;
-
- if (!is_pae(vcpu))
- return 1;
- kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l);
- if (cs_l)
- return 1;
- }
-#endif
- if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) &&
- is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) &&
- !load_pdptrs(vcpu, kvm_read_cr3(vcpu)))
- return 1;
-
- if (!(cr0 & X86_CR0_PG) &&
- (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)))
- return 1;
-
- kvm_x86_call(set_cr0)(vcpu, cr0);
-
- kvm_post_set_cr0(vcpu, old_cr0, cr0);
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(kvm_set_cr0);
-
-void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw)
-{
- (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f));
-}
-EXPORT_SYMBOL_GPL(kvm_lmsw);
-
void kvm_load_guest_xsave_state(struct kvm_vcpu *vcpu)
{
if (vcpu->arch.guest_state_protected)
@@ -1325,86 +1145,7 @@ bool __kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
}
EXPORT_SYMBOL_GPL(__kvm_is_valid_cr4);
-static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
-{
- return __kvm_is_valid_cr4(vcpu, cr4) &&
- kvm_x86_call(is_valid_cr4)(vcpu, cr4);
-}
-
-void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4)
-{
- if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS)
- kvm_mmu_reset_context(vcpu);
-
- /*
- * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB
- * according to the SDM; however, stale prev_roots could be reused
- * incorrectly in the future after a MOV to CR3 with NOFLUSH=1, so we
- * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST
- * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed,
- * so fall through.
- */
- if (!tdp_enabled &&
- (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE))
- kvm_mmu_unload(vcpu);
-
- /*
- * The TLB has to be flushed for all PCIDs if any of the following
- * (architecturally required) changes happen:
- * - CR4.PCIDE is changed from 1 to 0
- * - CR4.PGE is toggled
- *
- * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT.
- */
- if (((cr4 ^ old_cr4) & X86_CR4_PGE) ||
- (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE)))
- kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
-
- /*
- * The TLB has to be flushed for the current PCID if any of the
- * following (architecturally required) changes happen:
- * - CR4.SMEP is changed from 0 to 1
- * - CR4.PAE is toggled
- */
- else if (((cr4 ^ old_cr4) & X86_CR4_PAE) ||
- ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP)))
- kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
-
-}
-EXPORT_SYMBOL_GPL(kvm_post_set_cr4);
-
-int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
-{
- unsigned long old_cr4 = kvm_read_cr4(vcpu);
-
- if (!kvm_is_valid_cr4(vcpu, cr4))
- return 1;
-
- if (is_long_mode(vcpu)) {
- if (!(cr4 & X86_CR4_PAE))
- return 1;
- if ((cr4 ^ old_cr4) & X86_CR4_LA57)
- return 1;
- } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE)
- && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS)
- && !load_pdptrs(vcpu, kvm_read_cr3(vcpu)))
- return 1;
-
- if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) {
- /* PCID can not be enabled when cr3[11:0]!=000H or EFER.LMA=0 */
- if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu))
- return 1;
- }
-
- kvm_x86_call(set_cr4)(vcpu, cr4);
-
- kvm_post_set_cr4(vcpu, old_cr4, cr4);
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(kvm_set_cr4);
-
-static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid)
+void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid)
{
struct kvm_mmu *mmu = vcpu->arch.mmu;
unsigned long roots_to_free = 0;
@@ -1447,159 +1188,6 @@ static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid)
kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free);
}
-int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
-{
- bool skip_tlb_flush = false;
- unsigned long pcid = 0;
-#ifdef CONFIG_X86_64
- if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) {
- skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH;
- cr3 &= ~X86_CR3_PCID_NOFLUSH;
- pcid = cr3 & X86_CR3_PCID_MASK;
- }
-#endif
-
- /* PDPTRs are always reloaded for PAE paging. */
- if (cr3 == kvm_read_cr3(vcpu) && !is_pae_paging(vcpu))
- goto handle_tlb_flush;
-
- /*
- * Do not condition the GPA check on long mode, this helper is used to
- * stuff CR3, e.g. for RSM emulation, and there is no guarantee that
- * the current vCPU mode is accurate.
- */
- if (!kvm_vcpu_is_legal_cr3(vcpu, cr3))
- return 1;
-
- if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3))
- return 1;
-
- if (cr3 != kvm_read_cr3(vcpu))
- kvm_mmu_new_pgd(vcpu, cr3);
-
- vcpu->arch.cr3 = cr3;
- kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
- /* Do not call post_set_cr3, we do not get here for confidential guests. */
-
-handle_tlb_flush:
- /*
- * A load of CR3 that flushes the TLB flushes only the current PCID,
- * even if PCID is disabled, in which case PCID=0 is flushed. It's a
- * moot point in the end because _disabling_ PCID will flush all PCIDs,
- * and it's impossible to use a non-zero PCID when PCID is disabled,
- * i.e. only PCID=0 can be relevant.
- */
- if (!skip_tlb_flush)
- kvm_invalidate_pcid(vcpu, pcid);
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(kvm_set_cr3);
-
-int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8)
-{
- if (cr8 & CR8_RESERVED_BITS)
- return 1;
- if (lapic_in_kernel(vcpu))
- kvm_lapic_set_tpr(vcpu, cr8);
- else
- vcpu->arch.cr8 = cr8;
- return 0;
-}
-EXPORT_SYMBOL_GPL(kvm_set_cr8);
-
-unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu)
-{
- if (lapic_in_kernel(vcpu))
- return kvm_lapic_get_cr8(vcpu);
- else
- return vcpu->arch.cr8;
-}
-EXPORT_SYMBOL_GPL(kvm_get_cr8);
-
-static void kvm_update_dr0123(struct kvm_vcpu *vcpu)
-{
- int i;
-
- if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) {
- for (i = 0; i < KVM_NR_DB_REGS; i++)
- vcpu->arch.eff_db[i] = vcpu->arch.db[i];
- }
-}
-
-void kvm_update_dr7(struct kvm_vcpu *vcpu)
-{
- unsigned long dr7;
-
- if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)
- dr7 = vcpu->arch.guest_debug_dr7;
- else
- dr7 = vcpu->arch.dr7;
- kvm_x86_call(set_dr7)(vcpu, dr7);
- vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_BP_ENABLED;
- if (dr7 & DR7_BP_EN_MASK)
- vcpu->arch.switch_db_regs |= KVM_DEBUGREG_BP_ENABLED;
-}
-EXPORT_SYMBOL_GPL(kvm_update_dr7);
-
-static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu)
-{
- u64 fixed = DR6_FIXED_1;
-
- if (!guest_cpuid_has(vcpu, X86_FEATURE_RTM))
- fixed |= DR6_RTM;
-
- if (!guest_cpuid_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT))
- fixed |= DR6_BUS_LOCK;
- return fixed;
-}
-
-int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val)
-{
- size_t size = ARRAY_SIZE(vcpu->arch.db);
-
- switch (dr) {
- case 0 ... 3:
- vcpu->arch.db[array_index_nospec(dr, size)] = val;
- if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP))
- vcpu->arch.eff_db[dr] = val;
- break;
- case 4:
- case 6:
- if (!kvm_dr6_valid(val))
- return 1; /* #GP */
- vcpu->arch.dr6 = (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu);
- break;
- case 5:
- default: /* 7 */
- if (!kvm_dr7_valid(val))
- return 1; /* #GP */
- vcpu->arch.dr7 = (val & DR7_VOLATILE) | DR7_FIXED_1;
- kvm_update_dr7(vcpu);
- break;
- }
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(kvm_set_dr);
-
-unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr)
-{
- size_t size = ARRAY_SIZE(vcpu->arch.db);
-
- switch (dr) {
- case 0 ... 3:
- return vcpu->arch.db[array_index_nospec(dr, size)];
- case 4:
- case 6:
- return vcpu->arch.dr6;
- case 5:
- default: /* 7 */
- return vcpu->arch.dr7;
- }
-}
-EXPORT_SYMBOL_GPL(kvm_get_dr);
-
int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu)
{
u32 ecx = kvm_rcx_read(vcpu);
@@ -5342,7 +4930,7 @@ static struct kvm_queued_exception *kvm_get_exception_to_save(struct kvm_vcpu *v
return &vcpu->arch.exception;
}
-static void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu)
+void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu)
{
struct kvm_queued_exception *ex = kvm_get_exception_to_save(vcpu);
@@ -5552,56 +5140,6 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
return 0;
}
-static int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu,
- struct kvm_debugregs *dbgregs)
-{
- unsigned int i;
-
- if (vcpu->kvm->arch.has_protected_state &&
- vcpu->arch.guest_state_protected)
- return -EINVAL;
-
- kvm_handle_exception_payload_quirk(vcpu);
-
- memset(dbgregs, 0, sizeof(*dbgregs));
-
- BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) != ARRAY_SIZE(dbgregs->db));
- for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++)
- dbgregs->db[i] = vcpu->arch.db[i];
-
- dbgregs->dr6 = vcpu->arch.dr6;
- dbgregs->dr7 = vcpu->arch.dr7;
- return 0;
-}
-
-static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu,
- struct kvm_debugregs *dbgregs)
-{
- unsigned int i;
-
- if (vcpu->kvm->arch.has_protected_state &&
- vcpu->arch.guest_state_protected)
- return -EINVAL;
-
- if (dbgregs->flags)
- return -EINVAL;
-
- if (!kvm_dr6_valid(dbgregs->dr6))
- return -EINVAL;
- if (!kvm_dr7_valid(dbgregs->dr7))
- return -EINVAL;
-
- for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++)
- vcpu->arch.db[i] = dbgregs->db[i];
-
- kvm_update_dr0123(vcpu);
- vcpu->arch.dr6 = dbgregs->dr6;
- vcpu->arch.dr7 = dbgregs->dr7;
- kvm_update_dr7(vcpu);
-
- return 0;
-}
-
static int kvm_vcpu_ioctl_x86_get_xsave2(struct kvm_vcpu *vcpu,
u8 *state, unsigned int size)
@@ -6304,7 +5842,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
r = -ENOMEM;
if (!u.sregs2)
goto out;
- __get_sregs2(vcpu, u.sregs2);
+ kvm_vcpu_ioctl_x86_get_sregs2(vcpu, u.sregs2);
r = -EFAULT;
if (copy_to_user(argp, u.sregs2, sizeof(struct kvm_sregs2)))
goto out;
@@ -6323,7 +5861,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
u.sregs2 = NULL;
goto out;
}
- r = __set_sregs2(vcpu, u.sregs2);
+ r = kvm_vcpu_ioctl_x86_set_sregs2(vcpu, u.sregs2);
break;
}
case KVM_HAS_DEVICE_ATTR:
@@ -8272,11 +7810,6 @@ static int emulator_pio_out_emulated(struct x86_emulate_ctxt *ctxt,
return emulator_pio_out(emul_to_vcpu(ctxt), size, port, val, count);
}
-static unsigned long get_segment_base(struct kvm_vcpu *vcpu, int seg)
-{
- return kvm_x86_call(get_segment_base)(vcpu, seg);
-}
-
static void emulator_invlpg(struct x86_emulate_ctxt *ctxt, ulong address)
{
kvm_mmu_invlpg(emul_to_vcpu(ctxt), address);
@@ -9825,6 +9358,9 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops)
if (r != 0)
goto out_mmu_exit;
+ if (kvm_cpu_cap_has(X86_FEATURE_LM))
+ kvm_enable_efer_bits(EFER_LME | EFER_LMA);
+
kvm_ops_update(ops);
for_each_online_cpu(cpu) {
@@ -10247,7 +9783,7 @@ static void post_kvm_run_save(struct kvm_vcpu *vcpu)
kvm_run->flags |= KVM_RUN_X86_GUEST_MODE;
}
-static void update_cr8_intercept(struct kvm_vcpu *vcpu)
+void update_cr8_intercept(struct kvm_vcpu *vcpu)
{
int max_irr, tpr;
@@ -11703,179 +11239,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
return r;
}
-static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
-{
- if (vcpu->arch.emulate_regs_need_sync_to_vcpu) {
- /*
- * We are here if userspace calls get_regs() in the middle of
- * instruction emulation. Registers state needs to be copied
- * back from emulation context to vcpu. Userspace shouldn't do
- * that usually, but some bad designed PV devices (vmware
- * backdoor interface) need this to work
- */
- emulator_writeback_register_cache(vcpu->arch.emulate_ctxt);
- vcpu->arch.emulate_regs_need_sync_to_vcpu = false;
- }
- regs->rax = kvm_rax_read(vcpu);
- regs->rbx = kvm_rbx_read(vcpu);
- regs->rcx = kvm_rcx_read(vcpu);
- regs->rdx = kvm_rdx_read(vcpu);
- regs->rsi = kvm_rsi_read(vcpu);
- regs->rdi = kvm_rdi_read(vcpu);
- regs->rsp = kvm_rsp_read(vcpu);
- regs->rbp = kvm_rbp_read(vcpu);
-#ifdef CONFIG_X86_64
- regs->r8 = kvm_r8_read(vcpu);
- regs->r9 = kvm_r9_read(vcpu);
- regs->r10 = kvm_r10_read(vcpu);
- regs->r11 = kvm_r11_read(vcpu);
- regs->r12 = kvm_r12_read(vcpu);
- regs->r13 = kvm_r13_read(vcpu);
- regs->r14 = kvm_r14_read(vcpu);
- regs->r15 = kvm_r15_read(vcpu);
-#endif
-
- regs->rip = kvm_rip_read(vcpu);
- regs->rflags = kvm_get_rflags(vcpu);
-}
-
-int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
-{
- if (vcpu->kvm->arch.has_protected_state &&
- vcpu->arch.guest_state_protected)
- return -EINVAL;
-
- vcpu_load(vcpu);
- __get_regs(vcpu, regs);
- vcpu_put(vcpu);
- return 0;
-}
-
-static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
-{
- vcpu->arch.emulate_regs_need_sync_from_vcpu = true;
- vcpu->arch.emulate_regs_need_sync_to_vcpu = false;
-
- kvm_rax_write(vcpu, regs->rax);
- kvm_rbx_write(vcpu, regs->rbx);
- kvm_rcx_write(vcpu, regs->rcx);
- kvm_rdx_write(vcpu, regs->rdx);
- kvm_rsi_write(vcpu, regs->rsi);
- kvm_rdi_write(vcpu, regs->rdi);
- kvm_rsp_write(vcpu, regs->rsp);
- kvm_rbp_write(vcpu, regs->rbp);
-#ifdef CONFIG_X86_64
- kvm_r8_write(vcpu, regs->r8);
- kvm_r9_write(vcpu, regs->r9);
- kvm_r10_write(vcpu, regs->r10);
- kvm_r11_write(vcpu, regs->r11);
- kvm_r12_write(vcpu, regs->r12);
- kvm_r13_write(vcpu, regs->r13);
- kvm_r14_write(vcpu, regs->r14);
- kvm_r15_write(vcpu, regs->r15);
-#endif
-
- kvm_rip_write(vcpu, regs->rip);
- kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED);
-
- vcpu->arch.exception.pending = false;
- vcpu->arch.exception_vmexit.pending = false;
-
- kvm_make_request(KVM_REQ_EVENT, vcpu);
-}
-
-int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
-{
- if (vcpu->kvm->arch.has_protected_state &&
- vcpu->arch.guest_state_protected)
- return -EINVAL;
-
- vcpu_load(vcpu);
- __set_regs(vcpu, regs);
- vcpu_put(vcpu);
- return 0;
-}
-
-static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
-{
- struct desc_ptr dt;
-
- if (vcpu->arch.guest_state_protected)
- goto skip_protected_regs;
-
- kvm_handle_exception_payload_quirk(vcpu);
-
- kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS);
- kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS);
- kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES);
- kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS);
- kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS);
- kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS);
-
- kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR);
- kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR);
-
- kvm_x86_call(get_idt)(vcpu, &dt);
- sregs->idt.limit = dt.size;
- sregs->idt.base = dt.address;
- kvm_x86_call(get_gdt)(vcpu, &dt);
- sregs->gdt.limit = dt.size;
- sregs->gdt.base = dt.address;
-
- sregs->cr2 = vcpu->arch.cr2;
- sregs->cr3 = kvm_read_cr3(vcpu);
-
-skip_protected_regs:
- sregs->cr0 = kvm_read_cr0(vcpu);
- sregs->cr4 = kvm_read_cr4(vcpu);
- sregs->cr8 = kvm_get_cr8(vcpu);
- sregs->efer = vcpu->arch.efer;
- sregs->apic_base = vcpu->arch.apic_base;
-}
-
-static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
-{
- __get_sregs_common(vcpu, sregs);
-
- if (vcpu->arch.guest_state_protected)
- return;
-
- if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft)
- set_bit(vcpu->arch.interrupt.nr,
- (unsigned long *)sregs->interrupt_bitmap);
-}
-
-static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2)
-{
- int i;
-
- __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2);
-
- if (vcpu->arch.guest_state_protected)
- return;
-
- if (is_pae_paging(vcpu)) {
- kvm_vcpu_srcu_read_lock(vcpu);
- for (i = 0 ; i < 4 ; i++)
- sregs2->pdptrs[i] = kvm_pdptr_read(vcpu, i);
- sregs2->flags |= KVM_SREGS2_FLAGS_PDPTRS_VALID;
- kvm_vcpu_srcu_read_unlock(vcpu);
- }
-}
-
-int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
- struct kvm_sregs *sregs)
-{
- if (vcpu->kvm->arch.has_protected_state &&
- vcpu->arch.guest_state_protected)
- return -EINVAL;
-
- vcpu_load(vcpu);
- __get_sregs(vcpu, sregs);
- vcpu_put(vcpu);
- return 0;
-}
-
int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
struct kvm_mp_state *mp_state)
{
@@ -11985,178 +11348,6 @@ int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index,
}
EXPORT_SYMBOL_GPL(kvm_task_switch);
-static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
-{
- if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) {
- /*
- * When EFER.LME and CR0.PG are set, the processor is in
- * 64-bit mode (though maybe in a 32-bit code segment).
- * CR4.PAE and EFER.LMA must be set.
- */
- if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA))
- return false;
- if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3))
- return false;
- } else {
- /*
- * Not in 64-bit mode: EFER.LMA is clear and the code
- * segment cannot be 64-bit.
- */
- if (sregs->efer & EFER_LMA || sregs->cs.l)
- return false;
- }
-
- return kvm_is_valid_cr4(vcpu, sregs->cr4) &&
- kvm_is_valid_cr0(vcpu, sregs->cr0);
-}
-
-static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs,
- int *mmu_reset_needed, bool update_pdptrs)
-{
- struct msr_data apic_base_msr;
- int idx;
- struct desc_ptr dt;
-
- if (!kvm_is_valid_sregs(vcpu, sregs))
- return -EINVAL;
-
- apic_base_msr.data = sregs->apic_base;
- apic_base_msr.host_initiated = true;
- if (kvm_set_apic_base(vcpu, &apic_base_msr))
- return -EINVAL;
-
- if (vcpu->arch.guest_state_protected)
- return 0;
-
- dt.size = sregs->idt.limit;
- dt.address = sregs->idt.base;
- kvm_x86_call(set_idt)(vcpu, &dt);
- dt.size = sregs->gdt.limit;
- dt.address = sregs->gdt.base;
- kvm_x86_call(set_gdt)(vcpu, &dt);
-
- vcpu->arch.cr2 = sregs->cr2;
- *mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3;
- vcpu->arch.cr3 = sregs->cr3;
- kvm_register_mark_dirty(vcpu, VCPU_EXREG_CR3);
- kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3);
-
- kvm_set_cr8(vcpu, sregs->cr8);
-
- *mmu_reset_needed |= vcpu->arch.efer != sregs->efer;
- kvm_x86_call(set_efer)(vcpu, sregs->efer);
-
- *mmu_reset_needed |= kvm_read_cr0(vcpu) != sregs->cr0;
- kvm_x86_call(set_cr0)(vcpu, sregs->cr0);
-
- *mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4;
- kvm_x86_call(set_cr4)(vcpu, sregs->cr4);
-
- if (update_pdptrs) {
- idx = srcu_read_lock(&vcpu->kvm->srcu);
- if (is_pae_paging(vcpu)) {
- load_pdptrs(vcpu, kvm_read_cr3(vcpu));
- *mmu_reset_needed = 1;
- }
- srcu_read_unlock(&vcpu->kvm->srcu, idx);
- }
-
- kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS);
- kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS);
- kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES);
- kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS);
- kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS);
- kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS);
-
- kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR);
- kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR);
-
- update_cr8_intercept(vcpu);
-
- /* Older userspace won't unhalt the vcpu on reset. */
- if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 &&
- sregs->cs.selector == 0xf000 && sregs->cs.base == 0xffff0000 &&
- !is_protmode(vcpu))
- vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
-
- return 0;
-}
-
-static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)
-{
- int pending_vec, max_bits;
- int mmu_reset_needed = 0;
- int ret = __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true);
-
- if (ret)
- return ret;
-
- if (mmu_reset_needed) {
- kvm_mmu_reset_context(vcpu);
- kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
- }
-
- max_bits = KVM_NR_INTERRUPTS;
- pending_vec = find_first_bit(
- (const unsigned long *)sregs->interrupt_bitmap, max_bits);
-
- if (pending_vec < max_bits) {
- kvm_queue_interrupt(vcpu, pending_vec, false);
- pr_debug("Set back pending irq %d\n", pending_vec);
- kvm_make_request(KVM_REQ_EVENT, vcpu);
- }
- return 0;
-}
-
-static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2)
-{
- int mmu_reset_needed = 0;
- bool valid_pdptrs = sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID;
- bool pae = (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) &&
- !(sregs2->efer & EFER_LMA);
- int i, ret;
-
- if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID)
- return -EINVAL;
-
- if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected))
- return -EINVAL;
-
- ret = __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2,
- &mmu_reset_needed, !valid_pdptrs);
- if (ret)
- return ret;
-
- if (valid_pdptrs) {
- for (i = 0; i < 4 ; i++)
- kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]);
-
- kvm_register_mark_dirty(vcpu, VCPU_EXREG_PDPTR);
- mmu_reset_needed = 1;
- vcpu->arch.pdptrs_from_userspace = true;
- }
- if (mmu_reset_needed) {
- kvm_mmu_reset_context(vcpu);
- kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
- }
- return 0;
-}
-
-int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
- struct kvm_sregs *sregs)
-{
- int ret;
-
- if (vcpu->kvm->arch.has_protected_state &&
- vcpu->arch.guest_state_protected)
- return -EINVAL;
-
- vcpu_load(vcpu);
- ret = __set_sregs(vcpu, sregs);
- vcpu_put(vcpu);
- return ret;
-}
-
static void kvm_arch_vcpu_guestdbg_update_apicv_inhibit(struct kvm *kvm)
{
bool set = false;
@@ -12312,13 +11503,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
static void store_regs(struct kvm_vcpu *vcpu)
{
- BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES);
-
- if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS)
- __get_regs(vcpu, &vcpu->run->s.regs.regs);
-
- if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS)
- __get_sregs(vcpu, &vcpu->run->s.regs.sregs);
+ kvm_run_sync_regs_to_user(vcpu);
if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_EVENTS)
kvm_vcpu_ioctl_x86_get_vcpu_events(
@@ -12327,19 +11512,8 @@ static void store_regs(struct kvm_vcpu *vcpu)
static int sync_regs(struct kvm_vcpu *vcpu)
{
- if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) {
- __set_regs(vcpu, &vcpu->run->s.regs.regs);
- vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_REGS;
- }
-
- if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) {
- struct kvm_sregs sregs = vcpu->run->s.regs.sregs;
-
- if (__set_sregs(vcpu, &sregs))
- return -EINVAL;
-
- vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_SREGS;
- }
+ if (kvm_run_sync_regs_from_user(vcpu))
+ return -EINVAL;
if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_EVENTS) {
struct kvm_vcpu_events events = vcpu->run->s.regs.events;
@@ -12827,7 +12001,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
if (ret)
goto out;
- kvm_mmu_init_vm(kvm);
+ ret = kvm_mmu_init_vm(kvm);
+ if (ret)
+ goto out_cleanup_page_track;
ret = kvm_x86_call(vm_init)(kvm);
if (ret)
@@ -12872,6 +12048,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
out_uninit_mmu:
kvm_mmu_uninit_vm(kvm);
+out_cleanup_page_track:
kvm_page_track_cleanup(kvm);
out:
return ret;
@@ -13370,51 +12547,6 @@ int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu)
return kvm_x86_call(interrupt_allowed)(vcpu, false);
}
-unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu)
-{
- /* Can't read the RIP when guest state is protected, just return 0 */
- if (vcpu->arch.guest_state_protected)
- return 0;
-
- if (is_64_bit_mode(vcpu))
- return kvm_rip_read(vcpu);
- return (u32)(get_segment_base(vcpu, VCPU_SREG_CS) +
- kvm_rip_read(vcpu));
-}
-EXPORT_SYMBOL_GPL(kvm_get_linear_rip);
-
-bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip)
-{
- return kvm_get_linear_rip(vcpu) == linear_rip;
-}
-EXPORT_SYMBOL_GPL(kvm_is_linear_rip);
-
-unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu)
-{
- unsigned long rflags;
-
- rflags = kvm_x86_call(get_rflags)(vcpu);
- if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
- rflags &= ~X86_EFLAGS_TF;
- return rflags;
-}
-EXPORT_SYMBOL_GPL(kvm_get_rflags);
-
-static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
-{
- if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP &&
- kvm_is_linear_rip(vcpu, vcpu->arch.singlestep_rip))
- rflags |= X86_EFLAGS_TF;
- kvm_x86_call(set_rflags)(vcpu, rflags);
-}
-
-void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
-{
- __kvm_set_rflags(vcpu, rflags);
- kvm_make_request(KVM_REQ_EVENT, vcpu);
-}
-EXPORT_SYMBOL_GPL(kvm_set_rflags);
-
static inline u32 kvm_async_pf_hash_fn(gfn_t gfn)
{
BUILD_BUG_ON(!is_power_of_2(ASYNC_PF_PER_VCPU));
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 69145d116aa0..abab4bd96d0a 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -615,4 +615,32 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size,
unsigned int port, void *data, unsigned int count,
int in);
+static inline bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu)
+{
+ u64 mask = KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT;
+
+ return (vcpu->arch.apf.msr_en_val & mask) == mask;
+}
+
+static inline unsigned long get_segment_base(struct kvm_vcpu *vcpu, int seg)
+{
+ return kvm_x86_call(get_segment_base)(vcpu, seg);
+}
+
+void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);
+void kvm_vcpu_ioctl_x86_get_sregs2(struct kvm_vcpu *vcpu,
+ struct kvm_sregs2 *sregs2);
+int kvm_vcpu_ioctl_x86_set_sregs2(struct kvm_vcpu *vcpu,
+ struct kvm_sregs2 *sregs2);
+void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu);
+int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu);
+void kvm_update_dr0123(struct kvm_vcpu *vcpu);
+int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu,
+ struct kvm_debugregs *dbgregs);
+int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu,
+ struct kvm_debugregs *dbgregs);
+void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid);
+void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu);
+void update_cr8_intercept(struct kvm_vcpu *vcpu);
+
#endif
diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index 12d5a0f37432..a1f7f7ddaaf0 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -846,6 +846,105 @@ static void quirk_clear_strap_no_soft_reset_dev2_f0(struct pci_dev *dev)
}
}
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x15b8, quirk_clear_strap_no_soft_reset_dev2_f0);
+
+/*
+ * Enhanced atomic operations can cause corruption with 64-bit DMA
+ * on these devices.
+ */
+#define RX_ENH_ATOMIC_EN BIT(8)
+
+static const u32 nbio_7_7_pcie_smn_addrs[] = {
+ 0x111401d0,
+ 0x111411d0,
+ 0x111421d0,
+ 0x111431d0,
+ 0x111441d0,
+ 0x112401d0,
+ 0x112411d0,
+ 0x112421d0,
+ 0x112431d0,
+ 0x112441d0,
+ 0x112451d0,
+ 0x113401d0,
+ 0x114401d0,
+};
+
+static const u32 nbio_7_11_pcie_smn_addrs[] = {
+ 0x112401d0,
+ 0x112411d0,
+ 0x112421d0,
+ 0x112431d0,
+ 0x112441d0,
+ 0x112451d0,
+ 0x113401d0,
+ 0x113411d0,
+ 0x113421d0,
+ 0x113431d0,
+ 0x113441d0,
+ 0x113451d0,
+};
+
+static void quirk_amd_nbio_enhanced_atomic(struct pci_dev *host_bridge,
+ const u32 *smn_addrs,
+ size_t nr_smn_addrs)
+{
+ bool changed = false;
+ size_t i;
+ u32 data;
+ int ret;
+
+ for (i = 0; i < nr_smn_addrs; i++) {
+ ret = amd_smn_read(0, smn_addrs[i], &data);
+ if (ret)
+ continue;
+ if (!(data & RX_ENH_ATOMIC_EN))
+ continue;
+ data = data & ~RX_ENH_ATOMIC_EN;
+ ret = amd_smn_write(0, smn_addrs[i], data);
+ if (ret)
+ continue;
+ if (changed)
+ continue;
+ ret = amd_smn_read(0, smn_addrs[i], &data);
+ if (ret)
+ continue;
+ if (data & RX_ENH_ATOMIC_EN)
+ continue;
+ changed = true;
+ }
+
+ if (changed)
+ pci_info(host_bridge, "enhanced atomics disabled\n");
+}
+
+static void quirk_amd_nbio_7_7_disable_enhanced_atomic(struct pci_dev *dev)
+{
+ quirk_amd_nbio_enhanced_atomic(dev, nbio_7_7_pcie_smn_addrs,
+ ARRAY_SIZE(nbio_7_7_pcie_smn_addrs));
+}
+
+static void quirk_amd_nbio_7_11_disable_enhanced_atomic(struct pci_dev *dev)
+{
+ quirk_amd_nbio_enhanced_atomic(dev, nbio_7_11_pcie_smn_addrs,
+ ARRAY_SIZE(nbio_7_11_pcie_smn_addrs));
+}
+
+/* Phoenix, Hawk Point (NBIO 7.7) */
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x14E8,
+ quirk_amd_nbio_7_7_disable_enhanced_atomic);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x14E8,
+ quirk_amd_nbio_7_7_disable_enhanced_atomic);
+
+/* Strix, Krackan, Strix Halo (NBIO 7.11) */
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1507,
+ quirk_amd_nbio_7_11_disable_enhanced_atomic);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1507,
+ quirk_amd_nbio_7_11_disable_enhanced_atomic);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1122,
+ quirk_amd_nbio_7_11_disable_enhanced_atomic);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1122,
+ quirk_amd_nbio_7_11_disable_enhanced_atomic);
+
#endif
/*
diff --git a/drivers/ata/libahci.c b/drivers/ata/libahci.c
index 9b5b1e4c7148..0a8a48384f17 100644
--- a/drivers/ata/libahci.c
+++ b/drivers/ata/libahci.c
@@ -748,15 +748,28 @@ void ahci_start_fis_rx(struct ata_port *ap)
struct ahci_port_priv *pp = ap->private_data;
u32 tmp;
- /* set FIS registers */
+ /*
+ * On HBAs that only support 32-bit addressing PxCLBU is read only '0'.
+ * When applying the AHCI_HFLAG_32BIT_ONLY quirk, PxCLBU is RW, and the
+ * reset value is Implementation Specific, so we need to clear it to 0.
+ */
if (hpriv->cap & HOST_CAP_64)
writel((pp->cmd_slot_dma >> 16) >> 16,
port_mmio + PORT_LST_ADDR_HI);
+ else if (hpriv->flags & AHCI_HFLAG_32BIT_ONLY)
+ writel(0, port_mmio + PORT_LST_ADDR_HI);
writel(pp->cmd_slot_dma & 0xffffffff, port_mmio + PORT_LST_ADDR);
+ /*
+ * On HBAs that only support 32-bit addressing PxFBU is read only '0'.
+ * When applying the AHCI_HFLAG_32BIT_ONLY quirk, PxFBU is RW, and the
+ * reset value is Implementation Specific, so we need to clear it to 0.
+ */
if (hpriv->cap & HOST_CAP_64)
writel((pp->rx_fis_dma >> 16) >> 16,
port_mmio + PORT_FIS_ADDR_HI);
+ else if (hpriv->flags & AHCI_HFLAG_32BIT_ONLY)
+ writel(0, port_mmio + PORT_FIS_ADDR_HI);
writel(pp->rx_fis_dma & 0xffffffff, port_mmio + PORT_FIS_ADDR);
/* enable FIS reception */
diff --git a/drivers/ata/libahci_platform.c b/drivers/ata/libahci_platform.c
index 91d44302eac9..7308d48cf748 100644
--- a/drivers/ata/libahci_platform.c
+++ b/drivers/ata/libahci_platform.c
@@ -623,10 +623,10 @@ struct ahci_host_priv *ahci_platform_get_resources(struct platform_device *pdev,
of_platform_device_create(child, NULL, NULL);
port_dev = of_find_device_by_node(child);
-
if (port_dev) {
rc = ahci_platform_get_regulator(hpriv, port,
&port_dev->dev);
+ put_device(&port_dev->dev);
if (rc == -EPROBE_DEFER)
goto err_out;
}
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index f6160c1dd57f..d2777255129c 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -257,12 +257,18 @@ static void ata_scsi_set_passthru_sense_fields(struct ata_queued_cmd *qc)
/* descriptor format */
len = sb[7];
- desc = (char *)scsi_sense_desc_find(sb, len + 8, 9);
+ desc = (char *)scsi_sense_desc_find(sb, SCSI_SENSE_BUFFERSIZE, 9);
if (!desc) {
- if (SCSI_SENSE_BUFFERSIZE < len + 14)
+ /*
+ * The descriptor is written at sb[8 + len] and is 14
+ * bytes long, so it needs len + 22 bytes of buffer.
+ */
+ if (len + 22 > SCSI_SENSE_BUFFERSIZE)
return;
sb[7] = len + 14;
desc = sb + 8 + len;
+ } else if (desc - sb > SCSI_SENSE_BUFFERSIZE - 14) {
+ return;
}
desc[0] = 9;
desc[1] = 12;
diff --git a/drivers/base/core.c b/drivers/base/core.c
index 2e1cce6265e0..a6134a5e5e11 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -5112,6 +5112,49 @@ define_dev_printk_level(_dev_info, KERN_INFO);
#endif
+static void __dev_probe_failed(const struct device *dev, int err, bool fatal,
+ const char *fmt, va_list vargsp)
+{
+ struct va_format vaf;
+ va_list vargs;
+
+ /*
+ * On x86_64 and possibly on other architectures, va_list is actually a
+ * size-1 array containing a structure. As a result, function parameter
+ * vargsp decays from T[1] to T*, and &vargsp has type T** rather than
+ * T(*)[1], which is expected by its assignment to vaf.va below.
+ *
+ * One standard way to solve this mess is by creating a copy in a local
+ * variable of type va_list and then using a pointer to that local copy
+ * instead, which is the approach employed here.
+ */
+ va_copy(vargs, vargsp);
+
+ vaf.fmt = fmt;
+ vaf.va = &vargs;
+
+ switch (err) {
+ case -EPROBE_DEFER:
+ device_set_deferred_probe_reason(dev, &vaf);
+ dev_dbg(dev, "error %pe: %pV", ERR_PTR(err), &vaf);
+ break;
+
+ case -ENOMEM:
+ /* Don't print anything on -ENOMEM, there's already enough output */
+ break;
+
+ default:
+ /* Log fatal final failures as errors, otherwise produce warnings */
+ if (fatal)
+ dev_err(dev, "error %pe: %pV", ERR_PTR(err), &vaf);
+ else
+ dev_warn(dev, "error %pe: %pV", ERR_PTR(err), &vaf);
+ break;
+ }
+
+ va_end(vargs);
+}
+
/**
* dev_err_probe - probe error check and log helper
* @dev: the pointer to the struct device
@@ -5124,7 +5167,7 @@ define_dev_printk_level(_dev_info, KERN_INFO);
* -EPROBE_DEFER and propagate error upwards.
* In case of -EPROBE_DEFER it sets also defer probe reason, which can be
* checked later by reading devices_deferred debugfs attribute.
- * It replaces code sequence::
+ * It replaces the following code sequence::
*
* if (err != -EPROBE_DEFER)
* dev_err(dev, ...);
@@ -5136,47 +5179,77 @@ define_dev_printk_level(_dev_info, KERN_INFO);
*
* return dev_err_probe(dev, err, ...);
*
- * Using this helper in your probe function is totally fine even if @err is
- * known to never be -EPROBE_DEFER.
+ * Using this helper in your probe function is totally fine even if @err
+ * is known to never be -EPROBE_DEFER.
* The benefit compared to a normal dev_err() is the standardized format
- * of the error code, it being emitted symbolically (i.e. you get "EAGAIN"
- * instead of "-35") and the fact that the error code is returned which allows
- * more compact error paths.
+ * of the error code, which is emitted symbolically (i.e. you get "EAGAIN"
+ * instead of "-35"), and having the error code returned allows more
+ * compact error paths.
*
* Returns @err.
*/
int dev_err_probe(const struct device *dev, int err, const char *fmt, ...)
{
- struct va_format vaf;
- va_list args;
+ va_list vargs;
- va_start(args, fmt);
- vaf.fmt = fmt;
- vaf.va = &args;
+ va_start(vargs, fmt);
- switch (err) {
- case -EPROBE_DEFER:
- device_set_deferred_probe_reason(dev, &vaf);
- dev_dbg(dev, "error %pe: %pV", ERR_PTR(err), &vaf);
- break;
+ /* Use dev_err() for logging when err doesn't equal -EPROBE_DEFER */
+ __dev_probe_failed(dev, err, true, fmt, vargs);
- case -ENOMEM:
- /*
- * We don't print anything on -ENOMEM, there is already enough
- * output.
- */
- break;
+ va_end(vargs);
- default:
- dev_err(dev, "error %pe: %pV", ERR_PTR(err), &vaf);
- break;
- }
+ return err;
+}
+EXPORT_SYMBOL_GPL(dev_err_probe);
- va_end(args);
+/**
+ * dev_warn_probe - probe error check and log helper
+ * @dev: the pointer to the struct device
+ * @err: error value to test
+ * @fmt: printf-style format string
+ * @...: arguments as specified in the format string
+ *
+ * This helper implements common pattern present in probe functions for error
+ * checking: print debug or warning message depending if the error value is
+ * -EPROBE_DEFER and propagate error upwards.
+ * In case of -EPROBE_DEFER it sets also defer probe reason, which can be
+ * checked later by reading devices_deferred debugfs attribute.
+ * It replaces the following code sequence::
+ *
+ * if (err != -EPROBE_DEFER)
+ * dev_warn(dev, ...);
+ * else
+ * dev_dbg(dev, ...);
+ * return err;
+ *
+ * with::
+ *
+ * return dev_warn_probe(dev, err, ...);
+ *
+ * Using this helper in your probe function is totally fine even if @err
+ * is known to never be -EPROBE_DEFER.
+ * The benefit compared to a normal dev_warn() is the standardized format
+ * of the error code, which is emitted symbolically (i.e. you get "EAGAIN"
+ * instead of "-35"), and having the error code returned allows more
+ * compact error paths.
+ *
+ * Returns @err.
+ */
+int dev_warn_probe(const struct device *dev, int err, const char *fmt, ...)
+{
+ va_list vargs;
+
+ va_start(vargs, fmt);
+
+ /* Use dev_warn() for logging when err doesn't equal -EPROBE_DEFER */
+ __dev_probe_failed(dev, err, false, fmt, vargs);
+
+ va_end(vargs);
return err;
}
-EXPORT_SYMBOL_GPL(dev_err_probe);
+EXPORT_SYMBOL_GPL(dev_warn_probe);
static inline bool fwnode_is_primary(struct fwnode_handle *fwnode)
{
diff --git a/drivers/block/sunvdc.c b/drivers/block/sunvdc.c
index 97fe566465d7..16953341e501 100644
--- a/drivers/block/sunvdc.c
+++ b/drivers/block/sunvdc.c
@@ -557,6 +557,7 @@ static blk_status_t vdc_queue_rq(struct blk_mq_hw_ctx *hctx,
struct vdc_port *port = hctx->queue->queuedata;
struct vio_dring_state *dr;
unsigned long flags;
+ int ret;
dr = &port->vio.drings[VIO_DRIVER_TX_RING];
@@ -578,7 +579,13 @@ static blk_status_t vdc_queue_rq(struct blk_mq_hw_ctx *hctx,
return BLK_STS_DEV_RESOURCE;
}
- if (__send_request(bd->rq) < 0) {
+ ret = __send_request(bd->rq);
+ if (ret == -EAGAIN) {
+ spin_unlock_irqrestore(&port->vio.lock, flags);
+ /* already spun for 10msec, defer 10msec and retry */
+ blk_mq_delay_kick_requeue_list(hctx->queue, 10);
+ return BLK_STS_DEV_RESOURCE;
+ } else if (ret < 0) {
spin_unlock_irqrestore(&port->vio.lock, flags);
return BLK_STS_IOERR;
}
diff --git a/drivers/bluetooth/btintel_pcie.c b/drivers/bluetooth/btintel_pcie.c
index 8ebdfd9744ca..9facd47ff89b 100644
--- a/drivers/bluetooth/btintel_pcie.c
+++ b/drivers/bluetooth/btintel_pcie.c
@@ -210,7 +210,7 @@ static int btintel_pcie_submit_rx(struct btintel_pcie_data *data)
frbd_index = data->ia.tr_hia[BTINTEL_PCIE_RXQ_NUM];
- if (frbd_index > rxq->count)
+ if (frbd_index >= rxq->count)
return -ERANGE;
/* Prepare for RX submit. It updates the FRBD with the address of DMA
@@ -513,6 +513,11 @@ static void btintel_pcie_msix_tx_handle(struct btintel_pcie_data *data)
txq = &data->txq;
+ if (cr_hia >= txq->count) {
+ bt_dev_err(data->hdev, "TXQ: invalid cr_hia %u", cr_hia);
+ return;
+ }
+
while (cr_tia != cr_hia) {
data->tx_wait_done = true;
wake_up(&data->tx_wait_q);
@@ -786,6 +791,11 @@ static void btintel_pcie_msix_rx_handle(struct btintel_pcie_data *data)
rxq = &data->rxq;
+ if (cr_hia >= rxq->count) {
+ bt_dev_err(hdev, "RXQ: invalid cr_hia %u", cr_hia);
+ return;
+ }
+
/* The firmware sends multiple CD in a single MSI-X and it needs to
* process all received CDs in this interrupt.
*/
@@ -793,6 +803,12 @@ static void btintel_pcie_msix_rx_handle(struct btintel_pcie_data *data)
urbd1 = &rxq->urbd1s[cr_tia];
ipc_print_urbd1(data->hdev, urbd1, cr_tia);
+ if (urbd1->frbd_tag >= rxq->count) {
+ bt_dev_err(hdev, "RXQ: invalid frbd_tag %u",
+ urbd1->frbd_tag);
+ return;
+ }
+
buf = &rxq->bufs[urbd1->frbd_tag];
if (!buf) {
bt_dev_err(hdev, "RXQ: failed to get the DMA buffer for %d",
diff --git a/drivers/bluetooth/btmtk.c b/drivers/bluetooth/btmtk.c
index eb42b694da7f..33e0e3eb6515 100644
--- a/drivers/bluetooth/btmtk.c
+++ b/drivers/bluetooth/btmtk.c
@@ -709,7 +709,12 @@ static int btmtk_usb_hci_wmt_sync(struct hci_dev *hdev,
case BTMTK_WMT_FUNC_CTRL:
if (!skb_pull_data(data->evt_skb,
sizeof(wmt_evt_funcc->status))) {
- status = BTMTK_WMT_ON_UNDONE;
+ /* A plain enable/disable request is acked with just
+ * the WMT header and no trailing status word; the
+ * result is carried in the header's own flag byte.
+ */
+ status = wmt_evt->whdr.flag ? BTMTK_WMT_ON_UNDONE :
+ BTMTK_WMT_ON_DONE;
break;
}
diff --git a/drivers/bluetooth/btmtksdio.c b/drivers/bluetooth/btmtksdio.c
index 0e8fb0198143..575bac139715 100644
--- a/drivers/bluetooth/btmtksdio.c
+++ b/drivers/bluetooth/btmtksdio.c
@@ -1245,10 +1245,8 @@ static int btmtksdio_shutdown(struct hci_dev *hdev)
wmt_params.status = NULL;
err = mtk_hci_wmt_sync(hdev, &wmt_params);
- if (err < 0) {
+ if (err < 0)
bt_dev_err(hdev, "Failed to send wmt func ctrl (%d)", err);
- return err;
- }
ignore_wmt_cmd:
pm_runtime_put_noidle(bdev->dev);
diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c
index 3afb36a84c64..677919e02d29 100644
--- a/drivers/bluetooth/btusb.c
+++ b/drivers/bluetooth/btusb.c
@@ -68,6 +68,15 @@ static struct usb_driver btusb_driver;
#define BTUSB_BARROT BIT(28)
static const struct usb_device_id btusb_table[] = {
+ /*
+ * NXP IW610 (0471:0215): the composite device reports Bluetooth
+ * class at the whole-device level, so the generic entry below
+ * would also match this WiFi vendor interface. Ignore it here
+ * first so mwifiex-nxp can bind it instead.
+ */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x0471, 0x0215, 0xff, 0xff, 0xff),
+ .driver_info = BTUSB_IGNORE },
+
/* Generic Bluetooth USB device */
{ USB_DEVICE_INFO(0xe0, 0x01, 0x01) },
@@ -470,6 +479,14 @@ static const struct usb_device_id quirks_table[] = {
{ USB_DEVICE(0x1286, 0x2046), .driver_info = BTUSB_MARVELL },
{ USB_DEVICE(0x1286, 0x204e), .driver_info = BTUSB_MARVELL },
+ /*
+ * NXP IW610 BT interfaces (Marvell-lineage silicon, same quirk as
+ * the 0x1286 entries above). Scoped to the BT interface class,
+ * not just VID/PID -- see the btusb_table entry above.
+ */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x0471, 0x0215, 0xe0, 0x01, 0x01),
+ .driver_info = BTUSB_MARVELL },
+
/* Intel Bluetooth devices */
{ USB_DEVICE(0x8087, 0x0025), .driver_info = BTUSB_INTEL_COMBINED },
{ USB_DEVICE(0x8087, 0x0026), .driver_info = BTUSB_INTEL_COMBINED },
diff --git a/drivers/char/hw_random/atmel-rng.c b/drivers/char/hw_random/atmel-rng.c
index e9157255f851..143406bc6939 100644
--- a/drivers/char/hw_random/atmel-rng.c
+++ b/drivers/char/hw_random/atmel-rng.c
@@ -216,7 +216,7 @@ MODULE_DEVICE_TABLE(of, atmel_trng_dt_ids);
static struct platform_driver atmel_trng_driver = {
.probe = atmel_trng_probe,
- .remove_new = atmel_trng_remove,
+ .remove = atmel_trng_remove,
.driver = {
.name = "atmel-trng",
.pm = pm_ptr(&atmel_trng_pm_ops),
diff --git a/drivers/char/hw_random/cctrng.c b/drivers/char/hw_random/cctrng.c
index 4c50efc46483..4db198849695 100644
--- a/drivers/char/hw_random/cctrng.c
+++ b/drivers/char/hw_random/cctrng.c
@@ -653,7 +653,7 @@ static struct platform_driver cctrng_driver = {
.pm = &cctrng_pm,
},
.probe = cctrng_probe,
- .remove_new = cctrng_remove,
+ .remove = cctrng_remove,
};
module_platform_driver(cctrng_driver);
diff --git a/drivers/char/hw_random/exynos-trng.c b/drivers/char/hw_random/exynos-trng.c
index 9f039fddaee3..02e207c09e81 100644
--- a/drivers/char/hw_random/exynos-trng.c
+++ b/drivers/char/hw_random/exynos-trng.c
@@ -335,7 +335,7 @@ static struct platform_driver exynos_trng_driver = {
.of_match_table = exynos_trng_dt_match,
},
.probe = exynos_trng_probe,
- .remove_new = exynos_trng_remove,
+ .remove = exynos_trng_remove,
};
module_platform_driver(exynos_trng_driver);
diff --git a/drivers/char/hw_random/ingenic-rng.c b/drivers/char/hw_random/ingenic-rng.c
index 2f9b6483c4a1..bbfd662d25a6 100644
--- a/drivers/char/hw_random/ingenic-rng.c
+++ b/drivers/char/hw_random/ingenic-rng.c
@@ -132,7 +132,7 @@ MODULE_DEVICE_TABLE(of, ingenic_rng_of_match);
static struct platform_driver ingenic_rng_driver = {
.probe = ingenic_rng_probe,
- .remove_new = ingenic_rng_remove,
+ .remove = ingenic_rng_remove,
.driver = {
.name = "ingenic-rng",
.of_match_table = ingenic_rng_of_match,
diff --git a/drivers/char/hw_random/ks-sa-rng.c b/drivers/char/hw_random/ks-sa-rng.c
index 64369d4f1644..4494f1e4ab4d 100644
--- a/drivers/char/hw_random/ks-sa-rng.c
+++ b/drivers/char/hw_random/ks-sa-rng.c
@@ -272,7 +272,7 @@ static struct platform_driver ks_sa_rng_driver = {
.of_match_table = ks_sa_rng_dt_match,
},
.probe = ks_sa_rng_probe,
- .remove_new = ks_sa_rng_remove,
+ .remove = ks_sa_rng_remove,
};
module_platform_driver(ks_sa_rng_driver);
diff --git a/drivers/char/hw_random/mxc-rnga.c b/drivers/char/hw_random/mxc-rnga.c
index f01eb95bee31..e3fcb8bcc29b 100644
--- a/drivers/char/hw_random/mxc-rnga.c
+++ b/drivers/char/hw_random/mxc-rnga.c
@@ -188,7 +188,7 @@ static struct platform_driver mxc_rnga_driver = {
.of_match_table = mxc_rnga_of_match,
},
.probe = mxc_rnga_probe,
- .remove_new = mxc_rnga_remove,
+ .remove = mxc_rnga_remove,
};
module_platform_driver(mxc_rnga_driver);
diff --git a/drivers/char/hw_random/n2-drv.c b/drivers/char/hw_random/n2-drv.c
index 1b49e3a86d57..ea6d5599242f 100644
--- a/drivers/char/hw_random/n2-drv.c
+++ b/drivers/char/hw_random/n2-drv.c
@@ -858,7 +858,7 @@ static struct platform_driver n2rng_driver = {
.of_match_table = n2rng_match,
},
.probe = n2rng_probe,
- .remove_new = n2rng_remove,
+ .remove = n2rng_remove,
};
module_platform_driver(n2rng_driver);
diff --git a/drivers/char/hw_random/npcm-rng.c b/drivers/char/hw_random/npcm-rng.c
index bce8c4829a1f..9ff00f096f38 100644
--- a/drivers/char/hw_random/npcm-rng.c
+++ b/drivers/char/hw_random/npcm-rng.c
@@ -176,7 +176,7 @@ static struct platform_driver npcm_rng_driver = {
.of_match_table = of_match_ptr(rng_dt_id),
},
.probe = npcm_rng_probe,
- .remove_new = npcm_rng_remove,
+ .remove = npcm_rng_remove,
};
module_platform_driver(npcm_rng_driver);
diff --git a/drivers/char/hw_random/omap-rng.c b/drivers/char/hw_random/omap-rng.c
index 4914a8720e58..5e8b50f15db7 100644
--- a/drivers/char/hw_random/omap-rng.c
+++ b/drivers/char/hw_random/omap-rng.c
@@ -558,7 +558,7 @@ static struct platform_driver omap_rng_driver = {
.of_match_table = of_match_ptr(omap_rng_of_match),
},
.probe = omap_rng_probe,
- .remove_new = omap_rng_remove,
+ .remove = omap_rng_remove,
};
module_platform_driver(omap_rng_driver);
diff --git a/drivers/char/hw_random/stm32-rng.c b/drivers/char/hw_random/stm32-rng.c
index 9d041a67c295..4b3fd3bd039a 100644
--- a/drivers/char/hw_random/stm32-rng.c
+++ b/drivers/char/hw_random/stm32-rng.c
@@ -366,11 +366,6 @@ static int stm32_rng_init(struct hwrng *rng)
return 0;
}
-static void stm32_rng_remove(struct platform_device *ofdev)
-{
- pm_runtime_disable(&ofdev->dev);
-}
-
static int __maybe_unused stm32_rng_runtime_suspend(struct device *dev)
{
struct stm32_rng_private *priv = dev_get_drvdata(dev);
@@ -516,6 +511,7 @@ static int stm32_rng_probe(struct platform_device *ofdev)
struct device_node *np = ofdev->dev.of_node;
struct stm32_rng_private *priv;
struct resource *res;
+ int ret;
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
if (!priv)
@@ -553,7 +549,9 @@ static int stm32_rng_probe(struct platform_device *ofdev)
pm_runtime_set_autosuspend_delay(dev, 100);
pm_runtime_use_autosuspend(dev);
- pm_runtime_enable(dev);
+ ret = devm_pm_runtime_enable(dev);
+ if (ret)
+ return ret;
return devm_hwrng_register(dev, &priv->rng);
}
@@ -565,7 +563,6 @@ static struct platform_driver stm32_rng_driver = {
.of_match_table = stm32_rng_match,
},
.probe = stm32_rng_probe,
- .remove_new = stm32_rng_remove,
};
module_platform_driver(stm32_rng_driver);
diff --git a/drivers/char/hw_random/timeriomem-rng.c b/drivers/char/hw_random/timeriomem-rng.c
index 65b8260339f5..7174bfccc7b3 100644
--- a/drivers/char/hw_random/timeriomem-rng.c
+++ b/drivers/char/hw_random/timeriomem-rng.c
@@ -193,7 +193,7 @@ static struct platform_driver timeriomem_rng_driver = {
.of_match_table = timeriomem_rng_match,
},
.probe = timeriomem_rng_probe,
- .remove_new = timeriomem_rng_remove,
+ .remove = timeriomem_rng_remove,
};
module_platform_driver(timeriomem_rng_driver);
diff --git a/drivers/char/hw_random/xgene-rng.c b/drivers/char/hw_random/xgene-rng.c
index 642d13519464..39acaa503fec 100644
--- a/drivers/char/hw_random/xgene-rng.c
+++ b/drivers/char/hw_random/xgene-rng.c
@@ -375,7 +375,7 @@ MODULE_DEVICE_TABLE(of, xgene_rng_of_match);
static struct platform_driver xgene_rng_driver = {
.probe = xgene_rng_probe,
- .remove_new = xgene_rng_remove,
+ .remove = xgene_rng_remove,
.driver = {
.name = "xgene-rng",
.of_match_table = xgene_rng_of_match,
diff --git a/drivers/clk/clk-scpi.c b/drivers/clk/clk-scpi.c
index 50ac1cd25578..598dc1fd99ad 100644
--- a/drivers/clk/clk-scpi.c
+++ b/drivers/clk/clk-scpi.c
@@ -85,7 +85,7 @@ static unsigned long scpi_dvfs_recalc_rate(struct clk_hw *hw,
int idx = clk->scpi_ops->dvfs_get_idx(clk->id);
const struct scpi_opp *opp;
- if (idx < 0)
+ if (idx < 0 || idx >= clk->info->count)
return 0;
opp = clk->info->opps + idx;
diff --git a/drivers/clk/qcom/gcc-qcm2290.c b/drivers/clk/qcom/gcc-qcm2290.c
index bbcbe847fd95..766d0fb430f7 100644
--- a/drivers/clk/qcom/gcc-qcm2290.c
+++ b/drivers/clk/qcom/gcc-qcm2290.c
@@ -116,7 +116,7 @@ static const struct alpha_pll_config gpll10_config = {
.vco_mask = GENMASK(21, 20),
.main_output_mask = BIT(0),
.config_ctl_val = 0x4001055B,
- .test_ctl_hi1_val = 0x1,
+ .test_ctl_hi_val = 0x1,
};
static struct clk_alpha_pll gpll10 = {
@@ -148,7 +148,7 @@ static const struct alpha_pll_config gpll11_config = {
.vco_mask = GENMASK(21, 20),
.main_output_mask = BIT(0),
.config_ctl_val = 0x4001055B,
- .test_ctl_hi1_val = 0x1,
+ .test_ctl_hi_val = 0x1,
};
static struct clk_alpha_pll gpll11 = {
@@ -309,7 +309,7 @@ static const struct alpha_pll_config gpll8_config = {
.post_div_val = 0x1 << 8,
.post_div_mask = GENMASK(11, 8),
.config_ctl_val = 0x4001055B,
- .test_ctl_hi1_val = 0x1,
+ .test_ctl_hi_val = 0x1,
};
static struct clk_alpha_pll gpll8 = {
diff --git a/drivers/clk/qcom/gcc-sm6115.c b/drivers/clk/qcom/gcc-sm6115.c
index 4c3804701e24..4bd2ebc15894 100644
--- a/drivers/clk/qcom/gcc-sm6115.c
+++ b/drivers/clk/qcom/gcc-sm6115.c
@@ -120,7 +120,7 @@ static const struct alpha_pll_config gpll10_config = {
.vco_mask = GENMASK(21, 20),
.main_output_mask = BIT(0),
.config_ctl_val = 0x4001055b,
- .test_ctl_hi1_val = 0x1,
+ .test_ctl_hi_val = 0x1,
.test_ctl_hi_mask = 0x1,
};
@@ -173,7 +173,7 @@ static const struct alpha_pll_config gpll11_config = {
.vco_val = 0x2 << 20,
.vco_mask = GENMASK(21, 20),
.config_ctl_val = 0x4001055b,
- .test_ctl_hi1_val = 0x1,
+ .test_ctl_hi_val = 0x1,
.test_ctl_hi_mask = 0x1,
};
@@ -367,7 +367,7 @@ static const struct alpha_pll_config gpll8_config = {
.post_div_val = 0x1 << 8,
.post_div_mask = GENMASK(11, 8),
.config_ctl_val = 0x4001055b,
- .test_ctl_hi1_val = 0x1,
+ .test_ctl_hi_val = 0x1,
.test_ctl_hi_mask = 0x1,
};
diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
index 3cbe87d4a464..273e39290cf1 100644
--- a/drivers/dma-buf/dma-heap.c
+++ b/drivers/dma-buf/dma-heap.c
@@ -48,33 +48,6 @@ static dev_t dma_heap_devt;
static struct class *dma_heap_class;
static DEFINE_XARRAY_ALLOC(dma_heap_minors);
-static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
- u32 fd_flags,
- u64 heap_flags)
-{
- struct dma_buf *dmabuf;
- int fd;
-
- /*
- * Allocations from all heaps have to begin
- * and end on page boundaries.
- */
- len = PAGE_ALIGN(len);
- if (!len)
- return -EINVAL;
-
- dmabuf = heap->ops->allocate(heap, len, fd_flags, heap_flags);
- if (IS_ERR(dmabuf))
- return PTR_ERR(dmabuf);
-
- fd = dma_buf_fd(dmabuf, fd_flags);
- if (fd < 0) {
- dma_buf_put(dmabuf);
- /* just return, as put will call release and that will free */
- }
- return fd;
-}
-
static int dma_heap_open(struct inode *inode, struct file *file)
{
struct dma_heap *heap;
@@ -92,30 +65,42 @@ static int dma_heap_open(struct inode *inode, struct file *file)
return 0;
}
-static long dma_heap_ioctl_allocate(struct file *file, void *data)
+static struct dma_buf *dma_heap_ioctl_allocate(struct file *file, void *data)
{
struct dma_heap_allocation_data *heap_allocation = data;
struct dma_heap *heap = file->private_data;
+ struct dma_buf *dmabuf;
int fd;
+ size_t len;
if (heap_allocation->fd)
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
if (heap_allocation->fd_flags & ~DMA_HEAP_VALID_FD_FLAGS)
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
+
+ len = PAGE_ALIGN(heap_allocation->len);
+ if (!len)
+ return ERR_PTR(-EINVAL);
+
+ dmabuf = heap->ops->allocate(heap, len, heap_allocation->fd_flags,
+ heap_allocation->heap_flags);
- fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
- heap_allocation->fd_flags,
- heap_allocation->heap_flags);
- if (fd < 0)
- return fd;
+ if (IS_ERR(dmabuf))
+ return dmabuf;
+
+ fd = get_unused_fd_flags(heap_allocation->fd_flags);
+ if (fd < 0) {
+ dma_buf_put(dmabuf);
+ return ERR_PTR(fd);
+ }
heap_allocation->fd = fd;
- return 0;
+ return dmabuf;
}
static unsigned int dma_heap_ioctl_cmds[] = {
@@ -131,6 +116,8 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd,
unsigned int in_size, out_size, drv_size, ksize;
int nr = _IOC_NR(ucmd);
int ret = 0;
+ int fd;
+ struct dma_buf *dmabuf;
if (nr >= ARRAY_SIZE(dma_heap_ioctl_cmds))
return -EINVAL;
@@ -167,15 +154,28 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd,
switch (kcmd) {
case DMA_HEAP_IOCTL_ALLOC:
- ret = dma_heap_ioctl_allocate(file, kdata);
+ dmabuf = dma_heap_ioctl_allocate(file, kdata);
+
+ if (IS_ERR(dmabuf)) {
+ ret = PTR_ERR(dmabuf);
+ break;
+ }
+
+ fd = ((struct dma_heap_allocation_data *)kdata)->fd;
+ if (copy_to_user((void __user *)arg, kdata, out_size) != 0) {
+ put_unused_fd(fd);
+ dma_buf_put(dmabuf);
+ ret = -EFAULT;
+ } else {
+ fd_install(fd, dmabuf->file);
+ }
+
break;
default:
ret = -ENOTTY;
goto err;
}
- if (copy_to_user((void __user *)arg, kdata, out_size) != 0)
- ret = -EFAULT;
err:
if (kdata != stack_kdata)
kfree(kdata);
diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index e241b7b01233..045787330843 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -426,6 +426,7 @@ static void dma_device_release(struct kref *ref)
list_del_rcu(&device->global_node);
dma_channel_rebalance();
+ synchronize_rcu();
if (device->device_release)
device->device_release(device);
@@ -493,10 +494,13 @@ static int dma_chan_get(struct dma_chan *chan)
*/
static void dma_chan_put(struct dma_chan *chan)
{
+ struct module *owner;
+
/* This channel is not in use, bail out */
if (!chan->client_count)
return;
+ owner = dma_chan_to_owner(chan);
chan->client_count--;
/* This channel is not in use anymore, free it */
@@ -513,8 +517,10 @@ static void dma_chan_put(struct dma_chan *chan)
chan->route_data = NULL;
}
- dma_device_put(chan->device);
- module_put(dma_chan_to_owner(chan));
+ /* This channel is not in use anymore, drop the device ref */
+ if (!chan->client_count)
+ dma_device_put(chan->device);
+ module_put(owner);
}
enum dma_status dma_sync_wait(struct dma_chan *chan, dma_cookie_t cookie)
@@ -927,6 +933,36 @@ void dma_release_channel(struct dma_chan *chan)
}
EXPORT_SYMBOL_GPL(dma_release_channel);
+static void dmaenginem_release_channel(void *chan)
+{
+ dma_release_channel(chan);
+}
+
+/**
+ * devm_dma_request_chan - try to allocate an exclusive slave channel
+ * @dev: pointer to client device structure
+ * @name: slave channel name
+ *
+ * Returns pointer to appropriate DMA channel on success or an error pointer.
+ *
+ * The operation is managed and will be undone on driver detach.
+ */
+
+struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name)
+{
+ struct dma_chan *chan = dma_request_chan(dev, name);
+ int ret = 0;
+
+ if (!IS_ERR(chan))
+ ret = devm_add_action_or_reset(dev, dmaenginem_release_channel, chan);
+
+ if (ret)
+ return ERR_PTR(ret);
+
+ return chan;
+}
+EXPORT_SYMBOL_GPL(devm_dma_request_chan);
+
/**
* dmaengine_get - register interest in dma_channels
*/
diff --git a/drivers/dma/mmp_pdma.c b/drivers/dma/mmp_pdma.c
index 852e6714d9f2..59227fbed904 100644
--- a/drivers/dma/mmp_pdma.c
+++ b/drivers/dma/mmp_pdma.c
@@ -541,7 +541,7 @@ mmp_pdma_prep_slave_sg(struct dma_chan *dchan, struct scatterlist *sgl,
for_each_sg(sgl, sg, sg_len, i) {
addr = sg_dma_address(sg);
- avail = sg_dma_len(sgl);
+ avail = sg_dma_len(sg);
do {
len = min_t(size_t, avail, PDMA_MAX_DESC_BYTES);
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index 3f54ff37c5e0..f000d5a5add5 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -1212,7 +1212,7 @@ static int sprd_dma_probe(struct platform_device *pdev)
ret = pm_runtime_get_sync(&pdev->dev);
if (ret < 0)
- goto err_rpm;
+ goto err_register;
ret = dma_async_device_register(&sdev->dma_dev);
if (ret < 0) {
@@ -1234,7 +1234,6 @@ static int sprd_dma_probe(struct platform_device *pdev)
err_register:
pm_runtime_put_noidle(&pdev->dev);
pm_runtime_disable(&pdev->dev);
-err_rpm:
sprd_dma_disable(sdev);
return ret;
}
diff --git a/drivers/dma/sun6i-dma.c b/drivers/dma/sun6i-dma.c
index 1f1e21d48ad7..a2085bf206b7 100644
--- a/drivers/dma/sun6i-dma.c
+++ b/drivers/dma/sun6i-dma.c
@@ -353,8 +353,10 @@ static size_t sun6i_get_chan_size(struct sun6i_pchan *pchan)
size_t bytes;
dma_addr_t pos;
- pos = readl(pchan->base + DMA_CHAN_LLI_ADDR);
- bytes = readl(pchan->base + DMA_CHAN_CUR_CNT);
+ do {
+ pos = readl(pchan->base + DMA_CHAN_LLI_ADDR);
+ bytes = readl(pchan->base + DMA_CHAN_CUR_CNT);
+ } while (pos != readl(pchan->base + DMA_CHAN_LLI_ADDR));
if (pos == LLI_LAST_ITEM)
return bytes;
@@ -968,7 +970,6 @@ static enum dma_status sun6i_dma_tx_status(struct dma_chan *chan,
struct sun6i_pchan *pchan = vchan->phy;
struct sun6i_dma_lli *lli;
struct virt_dma_desc *vd;
- struct sun6i_desc *txd;
enum dma_status ret;
unsigned long flags;
size_t bytes = 0;
@@ -980,9 +981,9 @@ static enum dma_status sun6i_dma_tx_status(struct dma_chan *chan,
spin_lock_irqsave(&vchan->vc.lock, flags);
vd = vchan_find_desc(&vchan->vc, cookie);
- txd = to_sun6i_desc(&vd->tx);
if (vd) {
+ struct sun6i_desc *txd = to_sun6i_desc(&vd->tx);
for (lli = txd->v_lli; lli != NULL; lli = lli->v_lli_next)
bytes += lli->len;
} else if (!pchan || !pchan->desc) {
diff --git a/drivers/dma/ti/k3-udma-glue.c b/drivers/dma/ti/k3-udma-glue.c
index 7c224c3ab7a0..bf177879cc28 100644
--- a/drivers/dma/ti/k3-udma-glue.c
+++ b/drivers/dma/ti/k3-udma-glue.c
@@ -1236,8 +1236,9 @@ void k3_udma_glue_release_rx_chn(struct k3_udma_glue_rx_channel *rx_chn)
rx_chn->psil_paired = false;
}
- for (i = 0; i < rx_chn->flow_num; i++)
- k3_udma_glue_release_rx_flow(rx_chn, i);
+ if (rx_chn->flows)
+ for (i = 0; i < rx_chn->flow_num; i++)
+ k3_udma_glue_release_rx_flow(rx_chn, i);
if (xudma_rflow_is_gp(rx_chn->common.udmax, rx_chn->flow_id_base))
xudma_free_gp_rflow_range(rx_chn->common.udmax,
diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
index bea55dc99673..4a89ad8c90a3 100644
--- a/drivers/dma/xilinx/xilinx_dma.c
+++ b/drivers/dma/xilinx/xilinx_dma.c
@@ -753,15 +753,25 @@ xilinx_aximcdma_alloc_tx_segment(struct xilinx_dma_chan *chan)
return segment;
}
-static void xilinx_dma_clean_hw_desc(struct xilinx_axidma_desc_hw *hw)
+static void xilinx_dma_clean_hw_desc(struct xilinx_dma_chan *chan,
+ struct xilinx_axidma_tx_segment *segment)
{
- u32 next_desc = hw->next_desc;
- u32 next_desc_msb = hw->next_desc_msb;
+ dma_addr_t next;
+ u32 i;
- memset(hw, 0, sizeof(struct xilinx_axidma_desc_hw));
+ /*
+ * Restore the buffer descriptor's next descriptor pointer to the value
+ * set up in xilinx_dma_alloc_chan_resources(). Otherwise using the DMA
+ * in cyclic mode leaves the next descriptor pointer altered and
+ * prevents subsequent non-cyclic transfers.
+ */
+ i = segment - chan->seg_v;
+ next = chan->seg_p +
+ sizeof(*chan->seg_v) * ((i + 1) % XILINX_DMA_NUM_DESCS);
- hw->next_desc = next_desc;
- hw->next_desc_msb = next_desc_msb;
+ memset(&segment->hw, 0, sizeof(segment->hw));
+ segment->hw.next_desc = lower_32_bits(next);
+ segment->hw.next_desc_msb = upper_32_bits(next);
}
static void xilinx_mcdma_clean_hw_desc(struct xilinx_aximcdma_desc_hw *hw)
@@ -783,7 +793,7 @@ static void xilinx_mcdma_clean_hw_desc(struct xilinx_aximcdma_desc_hw *hw)
static void xilinx_dma_free_tx_segment(struct xilinx_dma_chan *chan,
struct xilinx_axidma_tx_segment *segment)
{
- xilinx_dma_clean_hw_desc(&segment->hw);
+ xilinx_dma_clean_hw_desc(chan, segment);
list_add_tail(&segment->node, &chan->free_seg_list);
}
@@ -917,9 +927,9 @@ static void xilinx_dma_free_descriptors(struct xilinx_dma_chan *chan)
spin_lock_irqsave(&chan->lock, flags);
- xilinx_dma_free_desc_list(chan, &chan->pending_list);
xilinx_dma_free_desc_list(chan, &chan->done_list);
xilinx_dma_free_desc_list(chan, &chan->active_list);
+ xilinx_dma_free_desc_list(chan, &chan->pending_list);
spin_unlock_irqrestore(&chan->lock, flags);
}
diff --git a/drivers/dpll/dpll_netlink.c b/drivers/dpll/dpll_netlink.c
index d6281bc46988..270ea8f7248a 100644
--- a/drivers/dpll/dpll_netlink.c
+++ b/drivers/dpll/dpll_netlink.c
@@ -956,8 +956,7 @@ dpll_pin_ref_sync_state_set(struct dpll_pin *pin,
unsigned long i;
int ret;
- ref_sync_pin = xa_find(&pin->ref_sync_pins, &ref_sync_pin_idx,
- ULONG_MAX, XA_PRESENT);
+ ref_sync_pin = xa_load(&pin->ref_sync_pins, ref_sync_pin_idx);
if (!ref_sync_pin) {
NL_SET_ERR_MSG(extack, "reference sync pin not found");
return -EINVAL;
diff --git a/drivers/edac/altera_edac.c b/drivers/edac/altera_edac.c
index 6c5fb0697641..ef99e59c7324 100644
--- a/drivers/edac/altera_edac.c
+++ b/drivers/edac/altera_edac.c
@@ -1507,6 +1507,7 @@ static int altr_portb_setup(struct altr_edac_device_dev *device)
int edac_idx, rc;
struct device_node *np;
const struct edac_device_prv_data *prv = &a10_sdmmceccb_data;
+ bool is_s10 = device->edac->is_s10;
rc = altr_check_ecc_deps(device);
if (rc)
@@ -1533,7 +1534,7 @@ static int altr_portb_setup(struct altr_edac_device_dev *device)
altdev = dci->pvt_info;
*altdev = *device;
- if (!devres_open_group(&altdev->ddev, altr_portb_setup, GFP_KERNEL))
+ if (!devres_open_group(device->edac->dev, altr_portb_setup, GFP_KERNEL))
return -ENOMEM;
/* Update PortB specific values */
@@ -1548,21 +1549,20 @@ static int altr_portb_setup(struct altr_edac_device_dev *device)
/*
* Update the PortB IRQs - A10 has 4, S10 has 2, Index accordingly
- *
- * FIXME: Instead of ifdefs with different architectures the driver
- * should properly use compatibles.
*/
-#ifdef CONFIG_64BIT
- altdev->sb_irq = irq_of_parse_and_map(np, 1);
-#else
- altdev->sb_irq = irq_of_parse_and_map(np, 2);
-#endif
+
+ /* Using compatibles to determine the IRQ Index */
+ if (is_s10)
+ altdev->sb_irq = irq_of_parse_and_map(np, 1);
+ else
+ altdev->sb_irq = irq_of_parse_and_map(np, 2);
+
if (!altdev->sb_irq) {
edac_printk(KERN_ERR, EDAC_DEVICE, "Error PortB SBIRQ alloc\n");
rc = -ENODEV;
goto err_release_group_1;
}
- rc = devm_request_irq(&altdev->ddev, altdev->sb_irq,
+ rc = devm_request_irq(device->edac->dev, altdev->sb_irq,
prv->ecc_irq_handler, IRQF_TRIGGER_HIGH,
ecc_name, altdev);
if (rc) {
@@ -1570,29 +1570,28 @@ static int altr_portb_setup(struct altr_edac_device_dev *device)
goto err_release_group_1;
}
-#ifdef CONFIG_64BIT
- /* Use IRQ to determine SError origin instead of assigning IRQ */
- rc = of_property_read_u32_index(np, "interrupts", 1, &altdev->db_irq);
- if (rc) {
- edac_printk(KERN_ERR, EDAC_DEVICE,
- "Error PortB DBIRQ alloc\n");
- goto err_release_group_1;
- }
-#else
- altdev->db_irq = irq_of_parse_and_map(np, 3);
- if (!altdev->db_irq) {
- edac_printk(KERN_ERR, EDAC_DEVICE, "Error PortB DBIRQ alloc\n");
- rc = -ENODEV;
- goto err_release_group_1;
- }
- rc = devm_request_irq(&altdev->ddev, altdev->db_irq,
- prv->ecc_irq_handler, IRQF_TRIGGER_HIGH,
- ecc_name, altdev);
- if (rc) {
- edac_printk(KERN_ERR, EDAC_DEVICE, "PortB DBERR IRQ error\n");
- goto err_release_group_1;
+ if (is_s10) {
+ /* Use IRQ to determine SError origin instead of assigning IRQ */
+ rc = of_property_read_u32_index(np, "interrupts", 1, &altdev->db_irq);
+ if (rc) {
+ edac_printk(KERN_ERR, EDAC_DEVICE, "Error PortB DBIRQ alloc\n");
+ goto err_release_group_1;
+ }
+ } else {
+ altdev->db_irq = irq_of_parse_and_map(np, 3);
+ if (!altdev->db_irq) {
+ edac_printk(KERN_ERR, EDAC_DEVICE, "Error PortB DBIRQ alloc\n");
+ rc = -ENODEV;
+ goto err_release_group_1;
+ }
+ rc = devm_request_irq(device->edac->dev, altdev->db_irq,
+ prv->ecc_irq_handler, IRQF_TRIGGER_HIGH,
+ ecc_name, altdev);
+ if (rc) {
+ edac_printk(KERN_ERR, EDAC_DEVICE, "PortB DBERR IRQ error\n");
+ goto err_release_group_1;
+ }
}
-#endif
rc = edac_device_add_device(dci);
if (rc) {
@@ -1605,13 +1604,13 @@ static int altr_portb_setup(struct altr_edac_device_dev *device)
list_add(&altdev->next, &altdev->edac->a10_ecc_devices);
- devres_remove_group(&altdev->ddev, altr_portb_setup);
+ devres_remove_group(device->edac->dev, altr_portb_setup);
return 0;
err_release_group_1:
edac_device_free_ctl_info(dci);
- devres_release_group(&altdev->ddev, altr_portb_setup);
+ devres_release_group(device->edac->dev, altr_portb_setup);
edac_printk(KERN_ERR, EDAC_DEVICE,
"%s:Error setting up EDAC device: %d\n", ecc_name, rc);
return rc;
@@ -1974,29 +1973,29 @@ static int altr_edac_a10_device_add(struct altr_arria10_edac *edac,
goto err_release_group1;
}
-#ifdef CONFIG_64BIT
- /* Use IRQ to determine SError origin instead of assigning IRQ */
- rc = of_property_read_u32_index(np, "interrupts", 0, &altdev->db_irq);
- if (rc) {
- edac_printk(KERN_ERR, EDAC_DEVICE,
- "Unable to parse DB IRQ index\n");
- goto err_release_group1;
- }
-#else
- altdev->db_irq = irq_of_parse_and_map(np, 1);
- if (!altdev->db_irq) {
- edac_printk(KERN_ERR, EDAC_DEVICE, "Error allocating DBIRQ\n");
- rc = -ENODEV;
- goto err_release_group1;
- }
- rc = devm_request_irq(edac->dev, altdev->db_irq, prv->ecc_irq_handler,
- IRQF_TRIGGER_HIGH,
- ecc_name, altdev);
- if (rc) {
- edac_printk(KERN_ERR, EDAC_DEVICE, "No DBERR IRQ resource\n");
- goto err_release_group1;
+ if (edac->is_s10) {
+ /* Use IRQ to determine SError origin instead of assigning IRQ */
+ rc = of_property_read_u32_index(np, "interrupts", 0, &altdev->db_irq);
+ if (rc) {
+ edac_printk(KERN_ERR, EDAC_DEVICE,
+ "Unable to parse DB IRQ index\n");
+ goto err_release_group1;
+ }
+ } else {
+ altdev->db_irq = irq_of_parse_and_map(np, 1);
+ if (!altdev->db_irq) {
+ edac_printk(KERN_ERR, EDAC_DEVICE, "Error allocating DBIRQ\n");
+ rc = -ENODEV;
+ goto err_release_group1;
+ }
+ rc = devm_request_irq(edac->dev, altdev->db_irq, prv->ecc_irq_handler,
+ IRQF_TRIGGER_HIGH,
+ ecc_name, altdev);
+ if (rc) {
+ edac_printk(KERN_ERR, EDAC_DEVICE, "No DBERR IRQ resource\n");
+ goto err_release_group1;
+ }
}
-#endif
rc = edac_device_add_device(dci);
if (rc) {
@@ -2122,6 +2121,8 @@ static int altr_edac_a10_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, edac);
INIT_LIST_HEAD(&edac->a10_ecc_devices);
+ edac->is_s10 = !!device_get_match_data(&pdev->dev);
+
edac->ecc_mgr_map =
altr_sysmgr_regmap_lookup_by_phandle(pdev->dev.of_node,
"altr,sysmgr-syscon");
@@ -2207,7 +2208,7 @@ static int altr_edac_a10_probe(struct platform_device *pdev)
static const struct of_device_id altr_edac_a10_of_match[] = {
{ .compatible = "altr,socfpga-a10-ecc-manager" },
- { .compatible = "altr,socfpga-s10-ecc-manager" },
+ { .compatible = "altr,socfpga-s10-ecc-manager", .data = (void *)1 },
{},
};
MODULE_DEVICE_TABLE(of, altr_edac_a10_of_match);
diff --git a/drivers/edac/altera_edac.h b/drivers/edac/altera_edac.h
index 7248d24c4908..5b26ef0cfe60 100644
--- a/drivers/edac/altera_edac.h
+++ b/drivers/edac/altera_edac.h
@@ -395,6 +395,7 @@ struct altr_arria10_edac {
struct irq_chip irq_chip;
struct list_head a10_ecc_devices;
struct notifier_block panic_notifier;
+ bool is_s10;
};
#endif /* #ifndef _ALTERA_EDAC_H */
diff --git a/drivers/firmware/arm_scpi.c b/drivers/firmware/arm_scpi.c
index 2d33771917bb..ec6f3d0cdd6a 100644
--- a/drivers/firmware/arm_scpi.c
+++ b/drivers/firmware/arm_scpi.c
@@ -631,8 +631,8 @@ static struct scpi_dvfs_info *scpi_dvfs_get_info(u8 domain)
if (ret)
return ERR_PTR(ret);
- if (!buf.opp_count)
- return ERR_PTR(-ENOENT);
+ if (!buf.opp_count || buf.opp_count > MAX_DVFS_OPPS)
+ return ERR_PTR(-EINVAL);
info = kmalloc(sizeof(*info), GFP_KERNEL);
if (!info)
diff --git a/drivers/firmware/qcom/qcom_scm.c b/drivers/firmware/qcom/qcom_scm.c
index afbf0ad80c85..f8b5650eaeb2 100644
--- a/drivers/firmware/qcom/qcom_scm.c
+++ b/drivers/firmware/qcom/qcom_scm.c
@@ -572,21 +572,21 @@ static void qcom_scm_set_download_mode(u32 dload_mode)
* qcom_scm_pas_init_image() - Initialize peripheral authentication service
* state machine for a given peripheral, using the
* metadata
- * @peripheral: peripheral id
+ * @pas_id: peripheral authentication service id
* @metadata: pointer to memory containing ELF header, program header table
* and optional blob of data used for authenticating the metadata
* and the rest of the firmware
* @size: size of the metadata
- * @ctx: optional metadata context
+ * @ctx: optional pas context
*
* Return: 0 on success.
*
* Upon successful return, the PAS metadata context (@ctx) will be used to
* track the metadata allocation, this needs to be released by invoking
- * qcom_scm_pas_metadata_release() by the caller.
+ * qcom_pas_metadata_release() by the caller.
*/
-int qcom_scm_pas_init_image(u32 peripheral, const void *metadata, size_t size,
- struct qcom_scm_pas_metadata *ctx)
+int qcom_scm_pas_init_image(u32 pas_id, const void *metadata, size_t size,
+ struct qcom_scm_pas_context *ctx)
{
dma_addr_t mdata_phys;
void *mdata_buf;
@@ -595,7 +595,7 @@ int qcom_scm_pas_init_image(u32 peripheral, const void *metadata, size_t size,
.svc = QCOM_SCM_SVC_PIL,
.cmd = QCOM_SCM_PIL_PAS_INIT_IMAGE,
.arginfo = QCOM_SCM_ARGS(2, QCOM_SCM_VAL, QCOM_SCM_RW),
- .args[0] = peripheral,
+ .args[0] = pas_id,
.owner = ARM_SMCCC_OWNER_SIP,
};
struct qcom_scm_res res;
@@ -649,10 +649,10 @@ int qcom_scm_pas_init_image(u32 peripheral, const void *metadata, size_t size,
EXPORT_SYMBOL_GPL(qcom_scm_pas_init_image);
/**
- * qcom_scm_pas_metadata_release() - release metadata context
- * @ctx: metadata context
+ * qcom_pas_metadata_release() - release metadata context
+ * @ctx: pas context
*/
-void qcom_scm_pas_metadata_release(struct qcom_scm_pas_metadata *ctx)
+void qcom_pas_metadata_release(struct qcom_scm_pas_context *ctx)
{
if (!ctx->ptr)
return;
@@ -663,25 +663,25 @@ void qcom_scm_pas_metadata_release(struct qcom_scm_pas_metadata *ctx)
ctx->phys = 0;
ctx->size = 0;
}
-EXPORT_SYMBOL_GPL(qcom_scm_pas_metadata_release);
+EXPORT_SYMBOL_GPL(qcom_pas_metadata_release);
/**
* qcom_scm_pas_mem_setup() - Prepare the memory related to a given peripheral
* for firmware loading
- * @peripheral: peripheral id
+ * @pas_id: peripheral authentication service id
* @addr: start address of memory area to prepare
* @size: size of the memory area to prepare
*
* Returns 0 on success.
*/
-int qcom_scm_pas_mem_setup(u32 peripheral, phys_addr_t addr, phys_addr_t size)
+int qcom_scm_pas_mem_setup(u32 pas_id, phys_addr_t addr, phys_addr_t size)
{
int ret;
struct qcom_scm_desc desc = {
.svc = QCOM_SCM_SVC_PIL,
.cmd = QCOM_SCM_PIL_PAS_MEM_SETUP,
.arginfo = QCOM_SCM_ARGS(3),
- .args[0] = peripheral,
+ .args[0] = pas_id,
.args[1] = addr,
.args[2] = size,
.owner = ARM_SMCCC_OWNER_SIP,
@@ -709,18 +709,18 @@ EXPORT_SYMBOL_GPL(qcom_scm_pas_mem_setup);
/**
* qcom_scm_pas_auth_and_reset() - Authenticate the given peripheral firmware
* and reset the remote processor
- * @peripheral: peripheral id
+ * @pas_id: peripheral authentication service id
*
* Return 0 on success.
*/
-int qcom_scm_pas_auth_and_reset(u32 peripheral)
+int qcom_scm_pas_auth_and_reset(u32 pas_id)
{
int ret;
struct qcom_scm_desc desc = {
.svc = QCOM_SCM_SVC_PIL,
.cmd = QCOM_SCM_PIL_PAS_AUTH_AND_RESET,
.arginfo = QCOM_SCM_ARGS(1),
- .args[0] = peripheral,
+ .args[0] = pas_id,
.owner = ARM_SMCCC_OWNER_SIP,
};
struct qcom_scm_res res;
@@ -745,18 +745,18 @@ EXPORT_SYMBOL_GPL(qcom_scm_pas_auth_and_reset);
/**
* qcom_scm_pas_shutdown() - Shut down the remote processor
- * @peripheral: peripheral id
+ * @pas_id: peripheral authentication service id
*
* Returns 0 on success.
*/
-int qcom_scm_pas_shutdown(u32 peripheral)
+int qcom_scm_pas_shutdown(u32 pas_id)
{
int ret;
struct qcom_scm_desc desc = {
.svc = QCOM_SCM_SVC_PIL,
.cmd = QCOM_SCM_PIL_PAS_SHUTDOWN,
.arginfo = QCOM_SCM_ARGS(1),
- .args[0] = peripheral,
+ .args[0] = pas_id,
.owner = ARM_SMCCC_OWNER_SIP,
};
struct qcom_scm_res res;
@@ -782,18 +782,18 @@ EXPORT_SYMBOL_GPL(qcom_scm_pas_shutdown);
/**
* qcom_scm_pas_supported() - Check if the peripheral authentication service is
* available for the given peripherial
- * @peripheral: peripheral id
+ * @pas_id: peripheral authentication service id
*
* Returns true if PAS is supported for this peripheral, otherwise false.
*/
-bool qcom_scm_pas_supported(u32 peripheral)
+bool qcom_scm_pas_supported(u32 pas_id)
{
int ret;
struct qcom_scm_desc desc = {
.svc = QCOM_SCM_SVC_PIL,
.cmd = QCOM_SCM_PIL_PAS_IS_SUPPORTED,
.arginfo = QCOM_SCM_ARGS(1),
- .args[0] = peripheral,
+ .args[0] = pas_id,
.owner = ARM_SMCCC_OWNER_SIP,
};
struct qcom_scm_res res;
diff --git a/drivers/firmware/sysfb_simplefb.c b/drivers/firmware/sysfb_simplefb.c
index 75a186bf8f8e..592d8a644619 100644
--- a/drivers/firmware/sysfb_simplefb.c
+++ b/drivers/firmware/sysfb_simplefb.c
@@ -35,36 +35,7 @@ __init bool sysfb_parse_mode(const struct screen_info *si,
if (type != VIDEO_TYPE_VLFB && type != VIDEO_TYPE_EFI)
return false;
- /*
- * The meaning of depth and bpp for direct-color formats is
- * inconsistent:
- *
- * - DRM format info specifies depth as the number of color
- * bits; including alpha, but not including filler bits.
- * - Linux' EFI platform code computes lfb_depth from the
- * individual color channels, including the reserved bits.
- * - VBE 1.1 defines lfb_depth for XRGB1555 as 16, but later
- * versions use 15.
- * - On the kernel command line, 'bpp' of 32 is usually
- * XRGB8888 including the filler bits, but 15 is XRGB1555
- * not including the filler bit.
- *
- * It's not easily possible to fix this in struct screen_info,
- * as this could break UAPI. The best solution is to compute
- * bits_per_pixel from the color bits, reserved bits and
- * reported lfb_depth, whichever is highest. In the loop below,
- * ignore simplefb formats with alpha bits, as EFI and VESA
- * don't specify alpha channels.
- */
- if (si->lfb_depth > 8) {
- bits_per_pixel = max(max3(si->red_size + si->red_pos,
- si->green_size + si->green_pos,
- si->blue_size + si->blue_pos),
- si->rsvd_size + si->rsvd_pos);
- bits_per_pixel = max_t(u32, bits_per_pixel, si->lfb_depth);
- } else {
- bits_per_pixel = si->lfb_depth;
- }
+ bits_per_pixel = __screen_info_lfb_bits_per_pixel(si);
for (i = 0; i < ARRAY_SIZE(formats); ++i) {
const struct simplefb_format *f = &formats[i];
diff --git a/drivers/gpio/gpio-arizona.c b/drivers/gpio/gpio-arizona.c
index c15fda99120a..531f85c19be8 100644
--- a/drivers/gpio/gpio-arizona.c
+++ b/drivers/gpio/gpio-arizona.c
@@ -117,8 +117,12 @@ static int arizona_gpio_direction_out(struct gpio_chip *chip,
if (value)
value = ARIZONA_GPN_LVL;
- return regmap_update_bits(arizona->regmap, ARIZONA_GPIO1_CTRL + offset,
- ARIZONA_GPN_DIR | ARIZONA_GPN_LVL, value);
+ ret = regmap_update_bits(arizona->regmap, ARIZONA_GPIO1_CTRL + offset,
+ ARIZONA_GPN_DIR | ARIZONA_GPN_LVL, value);
+ if (ret < 0 && (val & ARIZONA_GPN_DIR) && persistent)
+ pm_runtime_put_autosuspend(chip->parent);
+
+ return ret;
}
static void arizona_gpio_set(struct gpio_chip *chip, unsigned offset, int value)
diff --git a/drivers/gpio/gpio-tps65219.c b/drivers/gpio/gpio-tps65219.c
index cd1f17041f8c..a26dc1078203 100644
--- a/drivers/gpio/gpio-tps65219.c
+++ b/drivers/gpio/gpio-tps65219.c
@@ -53,7 +53,7 @@ static int tps65219_gpio_get(struct gpio_chip *gc, unsigned int offset)
if (ret)
return ret;
- ret = !!(val & BIT(TPS65219_MFP_GPIO_STATUS_MASK));
+ ret = !!(val & TPS65219_MFP_GPIO_STATUS_MASK);
dev_warn(dev, "GPIO%d = %d, MULTI_DEVICE_ENABLE, not a standard GPIO\n", offset, ret);
/*
diff --git a/drivers/gpio/gpio-virtuser.c b/drivers/gpio/gpio-virtuser.c
index ff1977b26991..5de9cee51fd0 100644
--- a/drivers/gpio/gpio-virtuser.c
+++ b/drivers/gpio/gpio-virtuser.c
@@ -698,7 +698,8 @@ static int gpio_virtuser_interrupts_set(void *data, u64 val)
atomic_set(&ld->irq, irq);
} else {
irq = atomic_xchg(&ld->irq, 0);
- free_irq(irq, ld);
+ if (irq)
+ free_irq(irq, ld);
}
return 0;
diff --git a/drivers/gpio/gpio-zynq.c b/drivers/gpio/gpio-zynq.c
index 50fa4938161d..c0a8d8055796 100644
--- a/drivers/gpio/gpio-zynq.c
+++ b/drivers/gpio/gpio-zynq.c
@@ -795,15 +795,7 @@ static int __maybe_unused zynq_gpio_runtime_resume(struct device *dev)
static int zynq_gpio_request(struct gpio_chip *chip, unsigned int offset)
{
- int ret;
-
- ret = pm_runtime_get_sync(chip->parent);
-
- /*
- * If the device is already active pm_runtime_get() will return 1 on
- * success, but gpio_request still needs to return 0.
- */
- return ret < 0 ? ret : 0;
+ return pm_runtime_resume_and_get(chip->parent);
}
static void zynq_gpio_free(struct gpio_chip *chip, unsigned int offset)
diff --git a/drivers/gpio/gpiolib-cdev.c b/drivers/gpio/gpiolib-cdev.c
index e77ec4b205f4..b85ba9989b14 100644
--- a/drivers/gpio/gpiolib-cdev.c
+++ b/drivers/gpio/gpiolib-cdev.c
@@ -2365,17 +2365,18 @@ static void gpio_v2_line_info_changed_to_v1(
#endif /* CONFIG_GPIO_CDEV_V1 */
-static void gpio_desc_to_lineinfo(struct gpio_desc *desc,
- struct gpio_v2_line_info *info)
+static int gpio_desc_to_lineinfo(struct gpio_desc *desc,
+ struct gpio_v2_line_info *info)
{
unsigned long dflags;
const char *label;
+ memset(info, 0, sizeof(*info));
+
CLASS(gpio_chip_guard, guard)(desc);
if (!guard.gc)
- return;
+ return -ENODEV;
- memset(info, 0, sizeof(*info));
info->offset = gpio_chip_hwgpio(desc);
if (desc->name)
@@ -2440,6 +2441,8 @@ static void gpio_desc_to_lineinfo(struct gpio_desc *desc,
info->flags |= GPIO_V2_LINE_FLAG_EVENT_CLOCK_REALTIME;
else if (test_bit(FLAG_EVENT_CLOCK_HTE, &dflags))
info->flags |= GPIO_V2_LINE_FLAG_EVENT_CLOCK_HTE;
+
+ return 0;
}
struct gpio_chardev_data {
@@ -2490,6 +2493,7 @@ static int lineinfo_get_v1(struct gpio_chardev_data *cdev, void __user *ip,
struct gpio_desc *desc;
struct gpioline_info lineinfo;
struct gpio_v2_line_info lineinfo_v2;
+ int ret;
if (copy_from_user(&lineinfo, ip, sizeof(lineinfo)))
return -EFAULT;
@@ -2507,7 +2511,10 @@ static int lineinfo_get_v1(struct gpio_chardev_data *cdev, void __user *ip,
return -EBUSY;
}
- gpio_desc_to_lineinfo(desc, &lineinfo_v2);
+ ret = gpio_desc_to_lineinfo(desc, &lineinfo_v2);
+ if (ret)
+ return ret;
+
gpio_v2_line_info_to_v1(&lineinfo_v2, &lineinfo);
if (copy_to_user(ip, &lineinfo, sizeof(lineinfo))) {
@@ -2525,6 +2532,7 @@ static int lineinfo_get(struct gpio_chardev_data *cdev, void __user *ip,
{
struct gpio_desc *desc;
struct gpio_v2_line_info lineinfo;
+ int ret;
if (copy_from_user(&lineinfo, ip, sizeof(lineinfo)))
return -EFAULT;
@@ -2544,7 +2552,11 @@ static int lineinfo_get(struct gpio_chardev_data *cdev, void __user *ip,
if (test_and_set_bit(lineinfo.offset, cdev->watched_lines))
return -EBUSY;
}
- gpio_desc_to_lineinfo(desc, &lineinfo);
+
+ ret = gpio_desc_to_lineinfo(desc, &lineinfo);
+ if (ret)
+ return ret;
+
supinfo_to_lineinfo(desc, &lineinfo);
if (copy_to_user(ip, &lineinfo, sizeof(lineinfo))) {
@@ -2637,7 +2649,11 @@ static int lineinfo_changed_notify(struct notifier_block *nb,
memset(&chg, 0, sizeof(chg));
chg.event_type = action;
chg.timestamp_ns = ktime_get_ns();
- gpio_desc_to_lineinfo(desc, &chg.info);
+
+ ret = gpio_desc_to_lineinfo(desc, &chg.info);
+ if (ret)
+ return NOTIFY_DONE;
+
supinfo_to_lineinfo(desc, &chg.info);
ret = kfifo_in_spinlocked(&cdev->events, &chg, 1, &cdev->wait.lock);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c
index d48480418175..442fd81f1cf6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c
@@ -1114,6 +1114,7 @@ static int amdgpu_acpi_enumerate_xcc(void)
GFP_KERNEL);
if (!xcc_info) {
DRM_ERROR("Failed to allocate memory for xcc info\n");
+ acpi_dev_put(acpi_dev);
return -ENOMEM;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index b6a928a0e6ca..9070edea1d12 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -368,7 +368,7 @@ void amdgpu_amdkfd_free_gtt_mem(struct amdgpu_device *adev, void **mem_obj)
{
struct amdgpu_bo **bo = (struct amdgpu_bo **) mem_obj;
- amdgpu_bo_reserve(*bo, true);
+ (void)amdgpu_bo_reserve(*bo, true);
amdgpu_bo_kunmap(*bo);
amdgpu_bo_unpin(*bo);
amdgpu_bo_unreserve(*bo);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 850f1b61694f..6dabdf3fd30d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -491,7 +491,7 @@ static int vm_update_pds(struct amdgpu_vm *vm, struct amdgpu_sync *sync)
if (ret)
return ret;
- return amdgpu_sync_fence(sync, vm->last_update);
+ return amdgpu_sync_fence(sync, vm->last_update, GFP_KERNEL);
}
static uint64_t get_pte_flags(struct amdgpu_device *adev, struct kgd_mem *mem)
@@ -722,7 +722,7 @@ kfd_mem_dmaunmap_userptr(struct kgd_mem *mem,
return;
amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_CPU);
- ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
+ (void)ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
dma_unmap_sgtable(adev->dev, ttm->sg, direction, 0);
sg_free_table(ttm->sg);
@@ -771,7 +771,7 @@ kfd_mem_dmaunmap_sg_bo(struct kgd_mem *mem,
}
amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_CPU);
- ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
+ (void)ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
dir = mem->alloc_flags & KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE ?
DMA_BIDIRECTIONAL : DMA_TO_DEVICE;
@@ -981,7 +981,7 @@ static int kfd_mem_attach(struct amdgpu_device *adev, struct kgd_mem *mem,
if (!attachment[i])
continue;
if (attachment[i]->bo_va) {
- amdgpu_bo_reserve(bo[i], true);
+ (void)amdgpu_bo_reserve(bo[i], true);
if (--attachment[i]->bo_va->ref_count == 0)
amdgpu_vm_bo_del(adev, attachment[i]->bo_va);
amdgpu_bo_unreserve(bo[i]);
@@ -1251,11 +1251,11 @@ static int unmap_bo_from_gpuvm(struct kgd_mem *mem,
return -EBUSY;
}
- amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
+ (void)amdgpu_vm_bo_unmap(adev, bo_va, entry->va);
- amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
+ (void)amdgpu_vm_clear_freed(adev, vm, &bo_va->last_pt_update);
- amdgpu_sync_fence(sync, bo_va->last_pt_update);
+ (void)amdgpu_sync_fence(sync, bo_va->last_pt_update, GFP_KERNEL);
return 0;
}
@@ -1279,7 +1279,7 @@ static int update_gpuvm_pte(struct kgd_mem *mem,
return ret;
}
- return amdgpu_sync_fence(sync, bo_va->last_pt_update);
+ return amdgpu_sync_fence(sync, bo_va->last_pt_update, GFP_KERNEL);
}
static int map_bo_to_gpuvm(struct kgd_mem *mem,
@@ -2358,7 +2358,7 @@ void amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(struct kgd_mem *mem)
{
struct amdgpu_bo *bo = mem->bo;
- amdgpu_bo_reserve(bo, true);
+ (void)amdgpu_bo_reserve(bo, true);
amdgpu_bo_kunmap(bo);
amdgpu_bo_unpin(bo);
amdgpu_bo_unreserve(bo);
@@ -2971,7 +2971,7 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *
}
dma_resv_for_each_fence(&cursor, bo->tbo.base.resv,
DMA_RESV_USAGE_KERNEL, fence) {
- ret = amdgpu_sync_fence(&sync_obj, fence);
+ ret = amdgpu_sync_fence(&sync_obj, fence, GFP_KERNEL);
if (ret) {
pr_debug("Memory eviction: Sync BO fence failed. Try again\n");
goto validate_map_fail;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
index 1b4e21986e5f..eb5f528459ec 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -462,7 +462,7 @@ static int amdgpu_cs_p2_dependencies(struct amdgpu_cs_parser *p,
dma_fence_put(old);
}
- r = amdgpu_sync_fence(&p->sync, fence);
+ r = amdgpu_sync_fence(&p->sync, fence, GFP_KERNEL);
dma_fence_put(fence);
if (r)
return r;
@@ -484,7 +484,7 @@ static int amdgpu_syncobj_lookup_and_add(struct amdgpu_cs_parser *p,
return r;
}
- r = amdgpu_sync_fence(&p->sync, fence);
+ r = amdgpu_sync_fence(&p->sync, fence, GFP_KERNEL);
dma_fence_put(fence);
return r;
}
@@ -1157,7 +1157,8 @@ static int amdgpu_cs_vm_handling(struct amdgpu_cs_parser *p)
if (r)
return r;
- r = amdgpu_sync_fence(&p->sync, fpriv->prt_va->last_pt_update);
+ r = amdgpu_sync_fence(&p->sync, fpriv->prt_va->last_pt_update,
+ GFP_KERNEL);
if (r)
return r;
@@ -1168,7 +1169,8 @@ static int amdgpu_cs_vm_handling(struct amdgpu_cs_parser *p)
if (r)
return r;
- r = amdgpu_sync_fence(&p->sync, bo_va->last_pt_update);
+ r = amdgpu_sync_fence(&p->sync, bo_va->last_pt_update,
+ GFP_KERNEL);
if (r)
return r;
}
@@ -1187,7 +1189,8 @@ static int amdgpu_cs_vm_handling(struct amdgpu_cs_parser *p)
if (r)
return r;
- r = amdgpu_sync_fence(&p->sync, bo_va->last_pt_update);
+ r = amdgpu_sync_fence(&p->sync, bo_va->last_pt_update,
+ GFP_KERNEL);
if (r)
return r;
}
@@ -1200,7 +1203,7 @@ static int amdgpu_cs_vm_handling(struct amdgpu_cs_parser *p)
if (r)
return r;
- r = amdgpu_sync_fence(&p->sync, vm->last_update);
+ r = amdgpu_sync_fence(&p->sync, vm->last_update, GFP_KERNEL);
if (r)
return r;
@@ -1267,7 +1270,8 @@ static int amdgpu_cs_sync_rings(struct amdgpu_cs_parser *p)
continue;
}
- r = amdgpu_sync_fence(&p->gang_leader->explicit_sync, fence);
+ r = amdgpu_sync_fence(&p->gang_leader->explicit_sync, fence,
+ GFP_KERNEL);
dma_fence_put(fence);
if (r)
return r;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 81f16e4447f8..a78fc6326f79 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -1671,8 +1671,10 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused)
/* Avoid accidently unparking the sched thread during GPU reset */
r = down_write_killable(&adev->reset_domain->sem);
- if (r)
+ if (r) {
+ pm_runtime_put_autosuspend(dev->dev);
return r;
+ }
/* hold on the scheduler */
for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
@@ -1960,7 +1962,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val)
/* swap out the old fences */
amdgpu_ib_preempt_fences_swap(ring, fences);
- amdgpu_fence_driver_force_completion(ring);
+ amdgpu_fence_driver_force_completion(ring, NULL);
/* resubmit unfinished jobs */
amdgpu_ib_preempt_job_recovery(&ring->sched);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 908f4539c779..cd05ddacd811 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4776,7 +4776,6 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev)
void amdgpu_device_fini_sw(struct amdgpu_device *adev)
{
- int idx;
bool px;
amdgpu_device_ip_fini(adev);
@@ -4814,11 +4813,9 @@ void amdgpu_device_fini_sw(struct amdgpu_device *adev)
if ((adev->pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA)
vga_client_unregister(adev->pdev);
- if (drm_dev_enter(adev_to_drm(adev), &idx)) {
-
+ if (adev->rmmio) {
iounmap(adev->rmmio);
adev->rmmio = NULL;
- drm_dev_exit(idx);
}
if (IS_ENABLED(CONFIG_PERF_EVENTS))
@@ -5470,6 +5467,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
{
int i, r = 0;
struct amdgpu_job *job = NULL;
+ struct dma_fence *fence = NULL;
struct amdgpu_device *tmp_adev = reset_context->reset_req_dev;
bool need_full_reset =
test_bit(AMDGPU_NEED_FULL_RESET, &reset_context->flags);
@@ -5482,6 +5480,9 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
amdgpu_fence_driver_isr_toggle(adev, true);
+ if (job)
+ fence = &job->hw_fence.base;
+
/* block all schedulers and reset given job's ring */
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
struct amdgpu_ring *ring = adev->rings[i];
@@ -5495,7 +5496,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
amdgpu_fence_driver_clear_job_fences(ring);
/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
- amdgpu_fence_driver_force_completion(ring);
+ amdgpu_fence_driver_force_completion(ring, fence);
}
amdgpu_fence_driver_isr_toggle(adev, false);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index 0b576481f231..e6cc6c88cb60 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -41,6 +41,7 @@
#include <linux/dma-buf.h>
#include <linux/dma-fence-array.h>
#include <linux/pci-p2pdma.h>
+#include <linux/pm_runtime.h>
static const struct dma_buf_attach_ops amdgpu_dma_buf_attach_ops;
@@ -97,9 +98,43 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf,
pci_p2pdma_distance(adev->pdev, attach->dev, false) < 0)
attach->peer2peer = false;
+ /*
+ * Only allow P2P while the exporter is active, and keep it active
+ * until detach. With runtime PM disabled take a plain reference so
+ * the put in detach stays balanced.
+ */
+ if (attach->peer2peer) {
+ struct device *dev = adev_to_drm(adev)->dev;
+ int ret = pm_runtime_get_if_active(dev);
+
+ if (!ret)
+ attach->peer2peer = false;
+ else if (ret < 0)
+ pm_runtime_get_noresume(dev);
+ }
+
return 0;
}
+/**
+ * amdgpu_dma_buf_detach - &dma_buf_ops.detach implementation
+ *
+ * @dmabuf: DMA-buf where we remove the attachment from
+ * @attach: the attachment to remove
+ *
+ * Drop the runtime PM reference taken in amdgpu_dma_buf_attach().
+ */
+static void amdgpu_dma_buf_detach(struct dma_buf *dmabuf,
+ struct dma_buf_attachment *attach)
+{
+ struct drm_gem_object *obj = dmabuf->priv;
+ struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
+ struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
+
+ if (attach->peer2peer)
+ pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
+}
+
/**
* amdgpu_dma_buf_pin - &dma_buf_ops.pin implementation
*
@@ -270,6 +305,7 @@ static int amdgpu_dma_buf_begin_cpu_access(struct dma_buf *dma_buf,
const struct dma_buf_ops amdgpu_dmabuf_ops = {
.attach = amdgpu_dma_buf_attach,
+ .detach = amdgpu_dma_buf_detach,
.pin = amdgpu_dma_buf_pin,
.unpin = amdgpu_dma_buf_unpin,
.map_dma_buf = amdgpu_dma_buf_map,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
index 569e0e537392..7c7da452dfab 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
@@ -594,7 +594,7 @@ void amdgpu_fence_driver_hw_fini(struct amdgpu_device *adev)
r = -ENODEV;
/* no need to trigger GPU reset as we are unloading */
if (r)
- amdgpu_fence_driver_force_completion(ring);
+ amdgpu_fence_driver_force_completion(ring, NULL);
if (!drm_dev_is_unplugged(adev_to_drm(adev)) &&
ring->fence_drv.irq_src &&
@@ -739,11 +739,30 @@ void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error)
* amdgpu_fence_driver_force_completion - force signal latest fence of ring
*
* @ring: fence of the ring to signal
+ * @timedout_fence: fence of the timedout job
*
*/
-void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring)
+void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring,
+ struct dma_fence *timedout_fence)
{
- amdgpu_fence_driver_set_error(ring, -ECANCELED);
+ struct amdgpu_fence_driver *drv = &ring->fence_drv;
+ unsigned long flags;
+
+ spin_lock_irqsave(&drv->lock, flags);
+ for (unsigned int i = 0; i <= drv->num_fences_mask; ++i) {
+ struct dma_fence *fence;
+
+ fence = rcu_dereference_protected(drv->fences[i],
+ lockdep_is_held(&drv->lock));
+ if (fence && !dma_fence_is_signaled_locked(fence)) {
+ if (fence == timedout_fence)
+ dma_fence_set_error(fence, -ETIME);
+ else
+ dma_fence_set_error(fence, -ECANCELED);
+ }
+ }
+ spin_unlock_irqrestore(&drv->lock, flags);
+
amdgpu_fence_write(ring, ring->fence_drv.sync_seq);
amdgpu_fence_process(ring);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
index 12e73351d6ff..6b3a44c8681f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
@@ -220,7 +220,7 @@ static int amdgpu_vmid_grab_idle(struct amdgpu_ring *ring,
return 0;
}
- fences = kmalloc_array(id_mgr->num_ids, sizeof(void *), GFP_KERNEL);
+ fences = kmalloc_array(id_mgr->num_ids, sizeof(void *), GFP_NOWAIT);
if (!fences)
return -ENOMEM;
@@ -337,7 +337,8 @@ static int amdgpu_vmid_grab_reserved(struct amdgpu_vm *vm,
/* Good we can use this VMID. Remember this submission as
* user of the VMID.
*/
- r = amdgpu_sync_fence(&(*id)->active, &job->base.s_fence->finished);
+ r = amdgpu_sync_fence(&(*id)->active, &job->base.s_fence->finished,
+ GFP_NOWAIT);
if (r)
return r;
@@ -398,7 +399,8 @@ static int amdgpu_vmid_grab_used(struct amdgpu_vm *vm,
* user of the VMID.
*/
r = amdgpu_sync_fence(&(*id)->active,
- &job->base.s_fence->finished);
+ &job->base.s_fence->finished,
+ GFP_NOWAIT);
if (r)
return r;
@@ -450,7 +452,8 @@ int amdgpu_vmid_grab(struct amdgpu_vm *vm, struct amdgpu_ring *ring,
/* Remember this submission as user of the VMID */
r = amdgpu_sync_fence(&id->active,
- &job->base.s_fence->finished);
+ &job->base.s_fence->finished,
+ GFP_NOWAIT);
if (r)
goto error;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index 99e9e96f5b7f..0abc38bb242c 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -148,7 +148,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
if (amdgpu_ring_sched_ready(ring))
drm_sched_stop(&ring->sched, s_job);
atomic_inc(&ring->adev->gpu_reset_counter);
- amdgpu_fence_driver_force_completion(ring);
+ amdgpu_fence_driver_force_completion(ring, &job->hw_fence.base);
if (amdgpu_ring_sched_ready(ring))
drm_sched_start(&ring->sched);
goto exit;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
index 41b88e0ea98b..aab9ac978aea 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
@@ -1354,14 +1354,14 @@ int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
DRM_ERROR("failed to do vm_bo_update on meta data\n");
goto error_del_bo_va;
}
- amdgpu_sync_fence(&sync, bo_va->last_pt_update);
+ amdgpu_sync_fence(&sync, bo_va->last_pt_update, GFP_KERNEL);
r = amdgpu_vm_update_pdes(adev, vm, false);
if (r) {
DRM_ERROR("failed to update pdes on meta data\n");
goto error_del_bo_va;
}
- amdgpu_sync_fence(&sync, vm->last_update);
+ amdgpu_sync_fence(&sync, vm->last_update, GFP_KERNEL);
amdgpu_sync_wait(&sync, false);
drm_exec_fini(&exec);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
index bc86a4455090..2430c16d86a4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
@@ -226,7 +226,6 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
ring->adev = adev;
ring->num_hw_submission = sched_hw_submission;
ring->sched_score = sched_score;
- ring->vmid_wait = dma_fence_get_stub();
if (!ring->is_mes_queue) {
ring->idx = adev->num_rings++;
@@ -355,6 +354,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
ring->max_dw = max_dw;
ring->hw_prio = hw_prio;
+ ring->vmid_wait = dma_fence_get_stub();
if (!ring->no_scheduler && ring->funcs->type < AMDGPU_HW_IP_NUM) {
hw_ip = ring->funcs->type;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
index 3eb74f845780..30252cd45840 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h
@@ -146,7 +146,8 @@ extern const struct drm_sched_backend_ops amdgpu_sched_ops;
void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring);
void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error);
-void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring);
+void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring,
+ struct dma_fence *timedout_fence);
int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring);
int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
@@ -431,6 +432,17 @@ static inline void amdgpu_ring_write_multiple(struct amdgpu_ring *ring,
ring->count_dw -= count_dw;
}
+static inline unsigned int amdgpu_ring_get_dw_distance(struct amdgpu_ring *ring,
+ u64 start_wptr, u64 end_wptr)
+{
+ unsigned int start = start_wptr & ring->buf_mask;
+ unsigned int end = end_wptr & ring->buf_mask;
+
+ if (end < start)
+ end += ring->ring_size >> 2;
+ return end - start;
+}
+
/**
* amdgpu_ring_patch_cond_exec - patch dw count of conditional execute
* @ring: amdgpu_ring structure
@@ -441,18 +453,13 @@ static inline void amdgpu_ring_write_multiple(struct amdgpu_ring *ring,
static inline void amdgpu_ring_patch_cond_exec(struct amdgpu_ring *ring,
unsigned int offset)
{
- unsigned cur;
-
if (!ring->funcs->init_cond_exec)
return;
WARN_ON(offset > ring->buf_mask);
WARN_ON(ring->ring[offset] != 0);
- cur = (ring->wptr - 1) & ring->buf_mask;
- if (cur < offset)
- cur += ring->ring_size >> 2;
- ring->ring[offset] = cur - offset;
+ ring->ring[offset] = amdgpu_ring_get_dw_distance(ring, offset, ring->wptr - 1);
}
#define amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset) \
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c
index c586ab4c911b..86c17a8946f5 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c
@@ -152,7 +152,8 @@ static bool amdgpu_sync_add_later(struct amdgpu_sync *sync, struct dma_fence *f)
*
* Add the fence to the sync object.
*/
-int amdgpu_sync_fence(struct amdgpu_sync *sync, struct dma_fence *f)
+int amdgpu_sync_fence(struct amdgpu_sync *sync, struct dma_fence *f,
+ gfp_t flags)
{
struct amdgpu_sync_entry *e;
@@ -162,7 +163,7 @@ int amdgpu_sync_fence(struct amdgpu_sync *sync, struct dma_fence *f)
if (amdgpu_sync_add_later(sync, f))
return 0;
- e = kmem_cache_alloc(amdgpu_sync_slab, GFP_KERNEL);
+ e = kmem_cache_alloc(amdgpu_sync_slab, flags);
if (!e)
return -ENOMEM;
@@ -249,7 +250,7 @@ int amdgpu_sync_resv(struct amdgpu_device *adev, struct amdgpu_sync *sync,
struct dma_fence *tmp = dma_fence_chain_contained(f);
if (amdgpu_sync_test_fence(adev, mode, owner, tmp)) {
- r = amdgpu_sync_fence(sync, f);
+ r = amdgpu_sync_fence(sync, f, GFP_KERNEL);
dma_fence_put(f);
if (r)
return r;
@@ -281,7 +282,7 @@ int amdgpu_sync_kfd(struct amdgpu_sync *sync, struct dma_resv *resv)
if (fence_owner != AMDGPU_FENCE_OWNER_KFD)
continue;
- r = amdgpu_sync_fence(sync, f);
+ r = amdgpu_sync_fence(sync, f, GFP_KERNEL);
if (r)
break;
}
@@ -388,7 +389,7 @@ int amdgpu_sync_clone(struct amdgpu_sync *source, struct amdgpu_sync *clone)
hash_for_each_safe(source->fences, i, tmp, e, node) {
f = e->fence;
if (!dma_fence_is_signaled(f)) {
- r = amdgpu_sync_fence(clone, f);
+ r = amdgpu_sync_fence(clone, f, GFP_KERNEL);
if (r)
return r;
} else {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.h
index e3272dce798d..1504f5e7fc46 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.h
@@ -47,7 +47,8 @@ struct amdgpu_sync {
};
void amdgpu_sync_create(struct amdgpu_sync *sync);
-int amdgpu_sync_fence(struct amdgpu_sync *sync, struct dma_fence *f);
+int amdgpu_sync_fence(struct amdgpu_sync *sync, struct dma_fence *f,
+ gfp_t flags);
int amdgpu_sync_resv(struct amdgpu_device *adev, struct amdgpu_sync *sync,
struct dma_resv *resv, enum amdgpu_sync_mode mode,
void *owner);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c
index 2b9bf1c1951c..6284fe784bc3 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_umsch_mm.c
@@ -32,463 +32,6 @@
#include "amdgpu_umsch_mm.h"
#include "umsch_mm_v4_0.h"
-struct umsch_mm_test_ctx_data {
- uint8_t process_csa[PAGE_SIZE];
- uint8_t vpe_ctx_csa[PAGE_SIZE];
- uint8_t vcn_ctx_csa[PAGE_SIZE];
-};
-
-struct umsch_mm_test_mqd_data {
- uint8_t vpe_mqd[PAGE_SIZE];
- uint8_t vcn_mqd[PAGE_SIZE];
-};
-
-struct umsch_mm_test_ring_data {
- uint8_t vpe_ring[PAGE_SIZE];
- uint8_t vpe_ib[PAGE_SIZE];
- uint8_t vcn_ring[PAGE_SIZE];
- uint8_t vcn_ib[PAGE_SIZE];
-};
-
-struct umsch_mm_test_queue_info {
- uint64_t mqd_addr;
- uint64_t csa_addr;
- uint32_t doorbell_offset_0;
- uint32_t doorbell_offset_1;
- enum UMSCH_SWIP_ENGINE_TYPE engine;
-};
-
-struct umsch_mm_test {
- struct amdgpu_bo *ctx_data_obj;
- uint64_t ctx_data_gpu_addr;
- uint32_t *ctx_data_cpu_addr;
-
- struct amdgpu_bo *mqd_data_obj;
- uint64_t mqd_data_gpu_addr;
- uint32_t *mqd_data_cpu_addr;
-
- struct amdgpu_bo *ring_data_obj;
- uint64_t ring_data_gpu_addr;
- uint32_t *ring_data_cpu_addr;
-
-
- struct amdgpu_vm *vm;
- struct amdgpu_bo_va *bo_va;
- uint32_t pasid;
- uint32_t vm_cntx_cntl;
- uint32_t num_queues;
-};
-
-static int map_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm,
- struct amdgpu_bo *bo, struct amdgpu_bo_va **bo_va,
- uint64_t addr, uint32_t size)
-{
- struct amdgpu_sync sync;
- struct drm_exec exec;
- int r;
-
- amdgpu_sync_create(&sync);
-
- drm_exec_init(&exec, 0, 0);
- drm_exec_until_all_locked(&exec) {
- r = drm_exec_lock_obj(&exec, &bo->tbo.base);
- drm_exec_retry_on_contention(&exec);
- if (unlikely(r))
- goto error_fini_exec;
-
- r = amdgpu_vm_lock_pd(vm, &exec, 0);
- drm_exec_retry_on_contention(&exec);
- if (unlikely(r))
- goto error_fini_exec;
- }
-
- *bo_va = amdgpu_vm_bo_add(adev, vm, bo);
- if (!*bo_va) {
- r = -ENOMEM;
- goto error_fini_exec;
- }
-
- r = amdgpu_vm_bo_map(adev, *bo_va, addr, 0, size,
- AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
- AMDGPU_PTE_EXECUTABLE);
-
- if (r)
- goto error_del_bo_va;
-
-
- r = amdgpu_vm_bo_update(adev, *bo_va, false);
- if (r)
- goto error_del_bo_va;
-
- amdgpu_sync_fence(&sync, (*bo_va)->last_pt_update);
-
- r = amdgpu_vm_update_pdes(adev, vm, false);
- if (r)
- goto error_del_bo_va;
-
- amdgpu_sync_fence(&sync, vm->last_update);
-
- amdgpu_sync_wait(&sync, false);
- drm_exec_fini(&exec);
-
- amdgpu_sync_free(&sync);
-
- return 0;
-
-error_del_bo_va:
- amdgpu_vm_bo_del(adev, *bo_va);
- amdgpu_sync_free(&sync);
-
-error_fini_exec:
- drm_exec_fini(&exec);
- amdgpu_sync_free(&sync);
- return r;
-}
-
-static int unmap_ring_data(struct amdgpu_device *adev, struct amdgpu_vm *vm,
- struct amdgpu_bo *bo, struct amdgpu_bo_va *bo_va,
- uint64_t addr)
-{
- struct drm_exec exec;
- long r;
-
- drm_exec_init(&exec, 0, 0);
- drm_exec_until_all_locked(&exec) {
- r = drm_exec_lock_obj(&exec, &bo->tbo.base);
- drm_exec_retry_on_contention(&exec);
- if (unlikely(r))
- goto out_unlock;
-
- r = amdgpu_vm_lock_pd(vm, &exec, 0);
- drm_exec_retry_on_contention(&exec);
- if (unlikely(r))
- goto out_unlock;
- }
-
-
- r = amdgpu_vm_bo_unmap(adev, bo_va, addr);
- if (r)
- goto out_unlock;
-
- amdgpu_vm_bo_del(adev, bo_va);
-
-out_unlock:
- drm_exec_fini(&exec);
-
- return r;
-}
-
-static void setup_vpe_queue(struct amdgpu_device *adev,
- struct umsch_mm_test *test,
- struct umsch_mm_test_queue_info *qinfo)
-{
- struct MQD_INFO *mqd = (struct MQD_INFO *)test->mqd_data_cpu_addr;
- uint64_t ring_gpu_addr = test->ring_data_gpu_addr;
-
- mqd->rb_base_lo = (ring_gpu_addr >> 8);
- mqd->rb_base_hi = (ring_gpu_addr >> 40);
- mqd->rb_size = PAGE_SIZE / 4;
- mqd->wptr_val = 0;
- mqd->rptr_val = 0;
- mqd->unmapped = 1;
-
- if (adev->vpe.collaborate_mode)
- memcpy(++mqd, test->mqd_data_cpu_addr, sizeof(struct MQD_INFO));
-
- qinfo->mqd_addr = test->mqd_data_gpu_addr;
- qinfo->csa_addr = test->ctx_data_gpu_addr +
- offsetof(struct umsch_mm_test_ctx_data, vpe_ctx_csa);
- qinfo->doorbell_offset_0 = 0;
- qinfo->doorbell_offset_1 = 0;
-}
-
-static void setup_vcn_queue(struct amdgpu_device *adev,
- struct umsch_mm_test *test,
- struct umsch_mm_test_queue_info *qinfo)
-{
-}
-
-static int add_test_queue(struct amdgpu_device *adev,
- struct umsch_mm_test *test,
- struct umsch_mm_test_queue_info *qinfo)
-{
- struct umsch_mm_add_queue_input queue_input = {};
- int r;
-
- queue_input.process_id = test->pasid;
- queue_input.page_table_base_addr = amdgpu_gmc_pd_addr(test->vm->root.bo);
-
- queue_input.process_va_start = 0;
- queue_input.process_va_end = (adev->vm_manager.max_pfn - 1) << AMDGPU_GPU_PAGE_SHIFT;
-
- queue_input.process_quantum = 100000; /* 10ms */
- queue_input.process_csa_addr = test->ctx_data_gpu_addr +
- offsetof(struct umsch_mm_test_ctx_data, process_csa);
-
- queue_input.context_quantum = 10000; /* 1ms */
- queue_input.context_csa_addr = qinfo->csa_addr;
-
- queue_input.inprocess_context_priority = CONTEXT_PRIORITY_LEVEL_NORMAL;
- queue_input.context_global_priority_level = CONTEXT_PRIORITY_LEVEL_NORMAL;
- queue_input.doorbell_offset_0 = qinfo->doorbell_offset_0;
- queue_input.doorbell_offset_1 = qinfo->doorbell_offset_1;
-
- queue_input.engine_type = qinfo->engine;
- queue_input.mqd_addr = qinfo->mqd_addr;
- queue_input.vm_context_cntl = test->vm_cntx_cntl;
-
- amdgpu_umsch_mm_lock(&adev->umsch_mm);
- r = adev->umsch_mm.funcs->add_queue(&adev->umsch_mm, &queue_input);
- amdgpu_umsch_mm_unlock(&adev->umsch_mm);
- if (r)
- return r;
-
- return 0;
-}
-
-static int remove_test_queue(struct amdgpu_device *adev,
- struct umsch_mm_test *test,
- struct umsch_mm_test_queue_info *qinfo)
-{
- struct umsch_mm_remove_queue_input queue_input = {};
- int r;
-
- queue_input.doorbell_offset_0 = qinfo->doorbell_offset_0;
- queue_input.doorbell_offset_1 = qinfo->doorbell_offset_1;
- queue_input.context_csa_addr = qinfo->csa_addr;
-
- amdgpu_umsch_mm_lock(&adev->umsch_mm);
- r = adev->umsch_mm.funcs->remove_queue(&adev->umsch_mm, &queue_input);
- amdgpu_umsch_mm_unlock(&adev->umsch_mm);
- if (r)
- return r;
-
- return 0;
-}
-
-static int submit_vpe_queue(struct amdgpu_device *adev, struct umsch_mm_test *test)
-{
- struct MQD_INFO *mqd = (struct MQD_INFO *)test->mqd_data_cpu_addr;
- uint32_t *ring = test->ring_data_cpu_addr +
- offsetof(struct umsch_mm_test_ring_data, vpe_ring) / 4;
- uint32_t *ib = test->ring_data_cpu_addr +
- offsetof(struct umsch_mm_test_ring_data, vpe_ib) / 4;
- uint64_t ib_gpu_addr = test->ring_data_gpu_addr +
- offsetof(struct umsch_mm_test_ring_data, vpe_ib);
- uint32_t *fence = ib + 2048 / 4;
- uint64_t fence_gpu_addr = ib_gpu_addr + 2048;
- const uint32_t test_pattern = 0xdeadbeef;
- int i;
-
- ib[0] = VPE_CMD_HEADER(VPE_CMD_OPCODE_FENCE, 0);
- ib[1] = lower_32_bits(fence_gpu_addr);
- ib[2] = upper_32_bits(fence_gpu_addr);
- ib[3] = test_pattern;
-
- ring[0] = VPE_CMD_HEADER(VPE_CMD_OPCODE_INDIRECT, 0);
- ring[1] = (ib_gpu_addr & 0xffffffe0);
- ring[2] = upper_32_bits(ib_gpu_addr);
- ring[3] = 4;
- ring[4] = 0;
- ring[5] = 0;
-
- mqd->wptr_val = (6 << 2);
- if (adev->vpe.collaborate_mode)
- (++mqd)->wptr_val = (6 << 2);
-
- WDOORBELL32(adev->umsch_mm.agdb_index[CONTEXT_PRIORITY_LEVEL_NORMAL], mqd->wptr_val);
-
- for (i = 0; i < adev->usec_timeout; i++) {
- if (*fence == test_pattern)
- return 0;
- udelay(1);
- }
-
- dev_err(adev->dev, "vpe queue submission timeout\n");
-
- return -ETIMEDOUT;
-}
-
-static int submit_vcn_queue(struct amdgpu_device *adev, struct umsch_mm_test *test)
-{
- return 0;
-}
-
-static int setup_umsch_mm_test(struct amdgpu_device *adev,
- struct umsch_mm_test *test)
-{
- struct amdgpu_vmhub *hub = &adev->vmhub[AMDGPU_MMHUB0(0)];
- int r;
-
- test->vm_cntx_cntl = hub->vm_cntx_cntl;
-
- test->vm = kzalloc(sizeof(*test->vm), GFP_KERNEL);
- if (!test->vm) {
- r = -ENOMEM;
- return r;
- }
-
- r = amdgpu_vm_init(adev, test->vm, -1);
- if (r)
- goto error_free_vm;
-
- r = amdgpu_pasid_alloc(16);
- if (r < 0)
- goto error_fini_vm;
- test->pasid = r;
-
- r = amdgpu_bo_create_kernel(adev, sizeof(struct umsch_mm_test_ctx_data),
- PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
- &test->ctx_data_obj,
- &test->ctx_data_gpu_addr,
- (void **)&test->ctx_data_cpu_addr);
- if (r)
- goto error_free_pasid;
-
- memset(test->ctx_data_cpu_addr, 0, sizeof(struct umsch_mm_test_ctx_data));
-
- r = amdgpu_bo_create_kernel(adev, PAGE_SIZE,
- PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
- &test->mqd_data_obj,
- &test->mqd_data_gpu_addr,
- (void **)&test->mqd_data_cpu_addr);
- if (r)
- goto error_free_ctx_data_obj;
-
- memset(test->mqd_data_cpu_addr, 0, PAGE_SIZE);
-
- r = amdgpu_bo_create_kernel(adev, sizeof(struct umsch_mm_test_ring_data),
- PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
- &test->ring_data_obj,
- NULL,
- (void **)&test->ring_data_cpu_addr);
- if (r)
- goto error_free_mqd_data_obj;
-
- memset(test->ring_data_cpu_addr, 0, sizeof(struct umsch_mm_test_ring_data));
-
- test->ring_data_gpu_addr = AMDGPU_VA_RESERVED_BOTTOM;
- r = map_ring_data(adev, test->vm, test->ring_data_obj, &test->bo_va,
- test->ring_data_gpu_addr, sizeof(struct umsch_mm_test_ring_data));
- if (r)
- goto error_free_ring_data_obj;
-
- return 0;
-
-error_free_ring_data_obj:
- amdgpu_bo_free_kernel(&test->ring_data_obj, NULL,
- (void **)&test->ring_data_cpu_addr);
-error_free_mqd_data_obj:
- amdgpu_bo_free_kernel(&test->mqd_data_obj, &test->mqd_data_gpu_addr,
- (void **)&test->mqd_data_cpu_addr);
-error_free_ctx_data_obj:
- amdgpu_bo_free_kernel(&test->ctx_data_obj, &test->ctx_data_gpu_addr,
- (void **)&test->ctx_data_cpu_addr);
-error_free_pasid:
- amdgpu_pasid_free(test->pasid);
-error_fini_vm:
- amdgpu_vm_fini(adev, test->vm);
-error_free_vm:
- kfree(test->vm);
-
- return r;
-}
-
-static void cleanup_umsch_mm_test(struct amdgpu_device *adev,
- struct umsch_mm_test *test)
-{
- unmap_ring_data(adev, test->vm, test->ring_data_obj,
- test->bo_va, test->ring_data_gpu_addr);
- amdgpu_bo_free_kernel(&test->mqd_data_obj, &test->mqd_data_gpu_addr,
- (void **)&test->mqd_data_cpu_addr);
- amdgpu_bo_free_kernel(&test->ring_data_obj, NULL,
- (void **)&test->ring_data_cpu_addr);
- amdgpu_bo_free_kernel(&test->ctx_data_obj, &test->ctx_data_gpu_addr,
- (void **)&test->ctx_data_cpu_addr);
- amdgpu_pasid_free(test->pasid);
- amdgpu_vm_fini(adev, test->vm);
- kfree(test->vm);
-}
-
-static int setup_test_queues(struct amdgpu_device *adev,
- struct umsch_mm_test *test,
- struct umsch_mm_test_queue_info *qinfo)
-{
- int i, r;
-
- for (i = 0; i < test->num_queues; i++) {
- if (qinfo[i].engine == UMSCH_SWIP_ENGINE_TYPE_VPE)
- setup_vpe_queue(adev, test, &qinfo[i]);
- else
- setup_vcn_queue(adev, test, &qinfo[i]);
-
- r = add_test_queue(adev, test, &qinfo[i]);
- if (r)
- return r;
- }
-
- return 0;
-}
-
-static int submit_test_queues(struct amdgpu_device *adev,
- struct umsch_mm_test *test,
- struct umsch_mm_test_queue_info *qinfo)
-{
- int i, r;
-
- for (i = 0; i < test->num_queues; i++) {
- if (qinfo[i].engine == UMSCH_SWIP_ENGINE_TYPE_VPE)
- r = submit_vpe_queue(adev, test);
- else
- r = submit_vcn_queue(adev, test);
- if (r)
- return r;
- }
-
- return 0;
-}
-
-static void cleanup_test_queues(struct amdgpu_device *adev,
- struct umsch_mm_test *test,
- struct umsch_mm_test_queue_info *qinfo)
-{
- int i;
-
- for (i = 0; i < test->num_queues; i++)
- remove_test_queue(adev, test, &qinfo[i]);
-}
-
-static int umsch_mm_test(struct amdgpu_device *adev)
-{
- struct umsch_mm_test_queue_info qinfo[] = {
- { .engine = UMSCH_SWIP_ENGINE_TYPE_VPE },
- };
- struct umsch_mm_test test = { .num_queues = ARRAY_SIZE(qinfo) };
- int r;
-
- r = setup_umsch_mm_test(adev, &test);
- if (r)
- return r;
-
- r = setup_test_queues(adev, &test, qinfo);
- if (r)
- goto cleanup;
-
- r = submit_test_queues(adev, &test, qinfo);
- if (r)
- goto cleanup;
-
- cleanup_test_queues(adev, &test, qinfo);
- cleanup_umsch_mm_test(adev, &test);
-
- return 0;
-
-cleanup:
- cleanup_test_queues(adev, &test, qinfo);
- cleanup_umsch_mm_test(adev, &test);
- return r;
-}
-
int amdgpu_umsch_mm_submit_pkt(struct amdgpu_umsch_mm *umsch, void *pkt, int ndws)
{
struct amdgpu_ring *ring = &umsch->ring;
@@ -791,7 +334,7 @@ static int umsch_mm_late_init(void *handle)
if (amdgpu_in_reset(adev) || adev->in_s0ix || adev->in_suspend)
return 0;
- return umsch_mm_test(adev);
+ return 0;
}
static int umsch_mm_sw_init(void *handle)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c
index fc6b0650ee7a..ea37ac3bc4ea 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c
@@ -516,7 +516,8 @@ int amdgpu_uvd_resume(struct amdgpu_device *adev)
}
memset_io(ptr, 0, size);
/* to restore uvd fence seq */
- amdgpu_fence_driver_force_completion(&adev->uvd.inst[i].ring);
+ if (adev->uvd.inst[i].ring.fence_drv.initialized)
+ amdgpu_fence_driver_force_completion(&adev->uvd.inst[i].ring, NULL);
}
}
return 0;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index a5d651b68729..c5e8bc5139d6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -2498,6 +2498,7 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm,
amdgpu_bo_unref(&root_bo);
error_free_delayed:
+ dma_fence_put(vm->last_update);
dma_fence_put(vm->last_tlb_flush);
dma_fence_put(vm->last_unlocked);
ttm_lru_bulk_move_fini(&adev->mman.bdev, &vm->lru_bulk_move);
diff --git a/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c b/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c
index 8e401f8b2a05..2b3a1b9f8efc 100644
--- a/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c
+++ b/drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c
@@ -518,7 +518,7 @@ static void nbio_v7_9_handle_ras_controller_intr_no_bifring(struct amdgpu_device
RAS_CNTLR_INTERRUPT_CLEAR, 1);
WREG32_SOC15(NBIO, 0, regBIF_BX0_BIF_DOORBELL_INT_CNTL, bif_doorbell_intr_cntl);
- if (!ras->disable_ras_err_cnt_harvest) {
+ if (ras && !ras->disable_ras_err_cnt_harvest && obj) {
/*
* clear error status after ras_controller_intr
* according to hw team and count ue number
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index 0222ede6a0bb..181606b4fd38 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -10834,8 +10834,10 @@ static int dm_update_crtc_state(struct amdgpu_display_manager *dm,
skip_modeset:
/* Release extra reference */
- if (new_stream)
+ if (new_stream) {
dc_stream_release(new_stream);
+ new_stream = NULL;
+ }
new_stream = NULL;
/*
diff --git a/drivers/gpu/drm/amd/display/dc/dml/Makefile b/drivers/gpu/drm/amd/display/dc/dml/Makefile
index 54a2af210b4c..5e1a295e56d3 100644
--- a/drivers/gpu/drm/amd/display/dc/dml/Makefile
+++ b/drivers/gpu/drm/amd/display/dc/dml/Makefile
@@ -36,7 +36,11 @@ ifneq ($(CONFIG_FRAME_WARN),0)
frame_warn_limit := 3072
endif
else
- frame_warn_limit := 2048
+ ifeq ($(CONFIG_CC_IS_CLANG),y)
+ frame_warn_limit := 3072
+ else
+ frame_warn_limit := 2048
+ endif
endif
ifeq ($(call test-lt, $(CONFIG_FRAME_WARN), $(frame_warn_limit)),y)
diff --git a/drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c b/drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c
index c768306ca69e..0118b398044d 100644
--- a/drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c
+++ b/drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c
@@ -297,6 +297,7 @@ void dml2_init_socbb_params(struct dml2_context *dml2, const struct dc *in_dc, s
out->smn_latency_us = 2;
out->dispclk_dppclk_vco_speed_mhz = 3600;
out->pct_ideal_dram_bw_after_urgent_pixel_only = 65.0;
+ out->gpuvm_min_page_size_kbytes = 4;
break;
case dml_project_dcn401:
diff --git a/drivers/gpu/drm/armada/armada_fbdev.c b/drivers/gpu/drm/armada/armada_fbdev.c
index e409029638b0..e33ccb9c7477 100644
--- a/drivers/gpu/drm/armada/armada_fbdev.c
+++ b/drivers/gpu/drm/armada/armada_fbdev.c
@@ -137,9 +137,9 @@ static void armada_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int armada_fbdev_client_restore(struct drm_client_dev *client)
+static int armada_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/bridge/samsung-dsim.c b/drivers/gpu/drm/bridge/samsung-dsim.c
index f3f836da6c2a..b13c68cf725c 100644
--- a/drivers/gpu/drm/bridge/samsung-dsim.c
+++ b/drivers/gpu/drm/bridge/samsung-dsim.c
@@ -1678,7 +1678,7 @@ static int samsung_dsim_register_te_irq(struct samsung_dsim *dsi, struct device
int te_gpio_irq;
int ret;
- dsi->te_gpio = devm_gpiod_get_optional(dev, "te", GPIOD_IN);
+ dsi->te_gpio = gpiod_get_optional(dev, "te", GPIOD_IN);
if (!dsi->te_gpio)
return 0;
else if (IS_ERR(dsi->te_gpio))
diff --git a/drivers/gpu/drm/bridge/tc358768.c b/drivers/gpu/drm/bridge/tc358768.c
index 686add7e9767..6b813f1402c7 100644
--- a/drivers/gpu/drm/bridge/tc358768.c
+++ b/drivers/gpu/drm/bridge/tc358768.c
@@ -1145,10 +1145,13 @@ tc358768_atomic_get_input_bus_fmts(struct drm_bridge *bridge,
return input_fmts;
}
-static bool tc358768_mode_fixup(struct drm_bridge *bridge,
- const struct drm_display_mode *mode,
- struct drm_display_mode *adjusted_mode)
+static int tc358768_bridge_atomic_check(struct drm_bridge *bridge,
+ struct drm_bridge_state *bridge_state,
+ struct drm_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
{
+ struct drm_display_mode *adjusted_mode = &crtc_state->adjusted_mode;
+
/* Default to positive sync */
if (!(adjusted_mode->flags &
@@ -1159,13 +1162,15 @@ static bool tc358768_mode_fixup(struct drm_bridge *bridge,
(DRM_MODE_FLAG_PVSYNC | DRM_MODE_FLAG_NVSYNC)))
adjusted_mode->flags |= DRM_MODE_FLAG_PVSYNC;
- return true;
+ bridge_state->input_bus_cfg.flags = bridge->timings->input_bus_flags;
+
+ return 0;
}
static const struct drm_bridge_funcs tc358768_bridge_funcs = {
.attach = tc358768_bridge_attach,
.mode_valid = tc358768_bridge_mode_valid,
- .mode_fixup = tc358768_mode_fixup,
+ .atomic_check = tc358768_bridge_atomic_check,
.pre_enable = tc358768_bridge_pre_enable,
.enable = tc358768_bridge_enable,
.disable = tc358768_bridge_disable,
diff --git a/drivers/gpu/drm/drm_atomic_uapi.c b/drivers/gpu/drm/drm_atomic_uapi.c
index fc20b039bed4..02255296cc57 100644
--- a/drivers/gpu/drm/drm_atomic_uapi.c
+++ b/drivers/gpu/drm/drm_atomic_uapi.c
@@ -1225,10 +1225,12 @@ static int prepare_signaling(struct drm_device *dev,
struct dma_fence *fence;
struct drm_out_fence_state *f;
+ ret = -ENOMEM;
+
f = krealloc(*fence_state, sizeof(**fence_state) *
(*num_fences + 1), GFP_KERNEL);
if (!f)
- return -ENOMEM;
+ goto err_free_event;
memset(&f[*num_fences], 0, sizeof(*f));
@@ -1237,12 +1239,12 @@ static int prepare_signaling(struct drm_device *dev,
fence = drm_crtc_create_fence(crtc);
if (!fence)
- return -ENOMEM;
+ goto err_free_event;
ret = setup_out_fence(&f[(*num_fences)++], fence);
if (ret) {
dma_fence_put(fence);
- return ret;
+ goto err_free_event;
}
crtc_state->event->base.fence = fence;
@@ -1298,6 +1300,11 @@ static int prepare_signaling(struct drm_device *dev,
}
return 0;
+
+err_free_event:
+ drm_event_cancel_free(dev, &crtc_state->event->base);
+ crtc_state->event = NULL;
+ return ret;
}
static void complete_signaling(struct drm_device *dev,
diff --git a/drivers/gpu/drm/drm_client.c b/drivers/gpu/drm/drm_client.c
index bfedcbf516db..b8577ca13d78 100644
--- a/drivers/gpu/drm/drm_client.c
+++ b/drivers/gpu/drm/drm_client.c
@@ -244,7 +244,7 @@ void drm_client_dev_hotplug(struct drm_device *dev)
}
EXPORT_SYMBOL(drm_client_dev_hotplug);
-void drm_client_dev_restore(struct drm_device *dev)
+void drm_client_dev_restore(struct drm_device *dev, bool force)
{
struct drm_client_dev *client;
int ret;
@@ -257,7 +257,7 @@ void drm_client_dev_restore(struct drm_device *dev)
if (!client->funcs || !client->funcs->restore)
continue;
- ret = client->funcs->restore(client);
+ ret = client->funcs->restore(client, force);
drm_dbg_kms(dev, "%s: ret=%d\n", client->name, ret);
if (!ret) /* The first one to return zero gets the privilege to restore */
break;
diff --git a/drivers/gpu/drm/drm_fb_helper.c b/drivers/gpu/drm/drm_fb_helper.c
index 91d76e5e2785..5118c865c470 100644
--- a/drivers/gpu/drm/drm_fb_helper.c
+++ b/drivers/gpu/drm/drm_fb_helper.c
@@ -249,6 +249,7 @@ __drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper,
/**
* drm_fb_helper_restore_fbdev_mode_unlocked - restore fbdev configuration
* @fb_helper: driver-allocated fbdev helper, can be NULL
+ * @force: ignore present DRM master
*
* This helper should be called from fbdev emulation's &drm_client_funcs.restore
* callback. It ensures that the user isn't greeted with a black screen when the
@@ -257,9 +258,9 @@ __drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper,
* Returns:
* 0 on success, or a negative errno code otherwise.
*/
-int drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper)
+int drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper, bool force)
{
- return __drm_fb_helper_restore_fbdev_mode_unlocked(fb_helper, false);
+ return __drm_fb_helper_restore_fbdev_mode_unlocked(fb_helper, force);
}
EXPORT_SYMBOL(drm_fb_helper_restore_fbdev_mode_unlocked);
@@ -1306,9 +1307,9 @@ int drm_fb_helper_set_par(struct fb_info *info)
* the KDSET IOCTL with KD_TEXT, and only after that drops the master
* status when exiting.
*
- * In the past this was caught by drm_fb_helper_lastclose(), but on
- * modern systems where logind always keeps a drm fd open to orchestrate
- * the vt switching, this doesn't work.
+ * In the past this was caught by drm_fb_helper_restore_fbdev_mode_unlocked(),
+ * but on modern systems where logind always keeps a drm fd open to
+ * orchestrate the vt switching, this doesn't work.
*
* To not break the userspace ABI we have this special case here, which
* is only used for the above case. Everything else uses the normal
@@ -1936,16 +1937,3 @@ int drm_fb_helper_hotplug_event(struct drm_fb_helper *fb_helper)
return 0;
}
EXPORT_SYMBOL(drm_fb_helper_hotplug_event);
-
-/**
- * drm_fb_helper_lastclose - DRM driver lastclose helper for fbdev emulation
- * @dev: DRM device
- *
- * This function is obsolete. Call drm_fb_helper_restore_fbdev_mode_unlocked()
- * instead.
- */
-void drm_fb_helper_lastclose(struct drm_device *dev)
-{
- drm_fb_helper_restore_fbdev_mode_unlocked(dev->fb_helper);
-}
-EXPORT_SYMBOL(drm_fb_helper_lastclose);
diff --git a/drivers/gpu/drm/drm_fbdev_client.c b/drivers/gpu/drm/drm_fbdev_client.c
index a09382afe2fb..91a7659ec1c8 100644
--- a/drivers/gpu/drm/drm_fbdev_client.c
+++ b/drivers/gpu/drm/drm_fbdev_client.c
@@ -25,9 +25,19 @@ static void drm_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int drm_fbdev_client_restore(struct drm_client_dev *client)
+static int drm_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ struct drm_fb_helper *fb_helper = drm_fb_helper_from_client(client);
+
+ /*
+ * The client is registered before the initial fbdev probe.
+ * If probing failed, the client remains registered but there
+ * is no valid fbdev framebuffer to restore.
+ */
+ if (!fb_helper->info || !fb_helper->fb)
+ return 0;
+
+ drm_fb_helper_restore_fbdev_mode_unlocked(fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/drm_fbdev_dma.c b/drivers/gpu/drm/drm_fbdev_dma.c
index b7ec75770f08..253266e78c1e 100644
--- a/drivers/gpu/drm/drm_fbdev_dma.c
+++ b/drivers/gpu/drm/drm_fbdev_dma.c
@@ -350,9 +350,9 @@ static void drm_fbdev_dma_client_unregister(struct drm_client_dev *client)
}
}
-static int drm_fbdev_dma_client_restore(struct drm_client_dev *client)
+static int drm_fbdev_dma_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/drm_fbdev_shmem.c b/drivers/gpu/drm/drm_fbdev_shmem.c
index e038377449d4..10e869e90839 100644
--- a/drivers/gpu/drm/drm_fbdev_shmem.c
+++ b/drivers/gpu/drm/drm_fbdev_shmem.c
@@ -216,9 +216,9 @@ static void drm_fbdev_shmem_client_unregister(struct drm_client_dev *client)
}
}
-static int drm_fbdev_shmem_client_restore(struct drm_client_dev *client)
+static int drm_fbdev_shmem_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/drm_fbdev_ttm.c b/drivers/gpu/drm/drm_fbdev_ttm.c
index e3f6ae3d7bad..9fb744d86f2c 100644
--- a/drivers/gpu/drm/drm_fbdev_ttm.c
+++ b/drivers/gpu/drm/drm_fbdev_ttm.c
@@ -257,9 +257,9 @@ static void drm_fbdev_ttm_client_unregister(struct drm_client_dev *client)
}
}
-static int drm_fbdev_ttm_client_restore(struct drm_client_dev *client)
+static int drm_fbdev_ttm_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/drm_file.c b/drivers/gpu/drm/drm_file.c
index e2cf118ff01d..06fca6d3a5cc 100644
--- a/drivers/gpu/drm/drm_file.c
+++ b/drivers/gpu/drm/drm_file.c
@@ -388,7 +388,7 @@ EXPORT_SYMBOL(drm_open);
static void drm_lastclose(struct drm_device *dev)
{
- drm_client_dev_restore(dev);
+ drm_client_dev_restore(dev, false);
if (dev_is_pci(dev->dev))
vga_switcheroo_process_delayed_switch();
@@ -840,6 +840,16 @@ static void print_size(struct drm_printer *p, const char *stat,
drm_printf(p, "drm-%s-%s:\t%llu%s\n", stat, region, sz, units[u]);
}
+int drm_memory_stats_is_zero(const struct drm_memory_stats *stats)
+{
+ return (stats->shared == 0 &&
+ stats->private == 0 &&
+ stats->resident == 0 &&
+ stats->purgeable == 0 &&
+ stats->active == 0);
+}
+EXPORT_SYMBOL(drm_memory_stats_is_zero);
+
/**
* drm_print_memory_stats - A helper to print memory stats
* @p: The printer to print output to
diff --git a/drivers/gpu/drm/exynos/exynos_drm_fbdev.c b/drivers/gpu/drm/exynos/exynos_drm_fbdev.c
index b24516998755..3289563783e2 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_fbdev.c
+++ b/drivers/gpu/drm/exynos/exynos_drm_fbdev.c
@@ -149,9 +149,9 @@ static void exynos_drm_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int exynos_drm_fbdev_client_restore(struct drm_client_dev *client)
+static int exynos_drm_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/gma500/fbdev.c b/drivers/gpu/drm/gma500/fbdev.c
index d80a18fde1aa..1d349d5dab20 100644
--- a/drivers/gpu/drm/gma500/fbdev.c
+++ b/drivers/gpu/drm/gma500/fbdev.c
@@ -214,9 +214,9 @@ static void psb_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int psb_fbdev_client_restore(struct drm_client_dev *client)
+static int psb_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/gud/gud_pipe.c b/drivers/gpu/drm/gud/gud_pipe.c
index e163649816d5..96528f01044d 100644
--- a/drivers/gpu/drm/gud/gud_pipe.c
+++ b/drivers/gpu/drm/gud/gud_pipe.c
@@ -544,8 +544,8 @@ int gud_pipe_check(struct drm_simple_display_pipe *pipe,
goto out;
}
- req->properties[num_properties + i].prop = cpu_to_le16(prop);
- req->properties[num_properties + i].val = cpu_to_le64(val);
+ req->properties[num_properties].prop = cpu_to_le16(prop);
+ req->properties[num_properties].val = cpu_to_le64(val);
num_properties++;
}
diff --git a/drivers/gpu/drm/i915/display/intel_connector.c b/drivers/gpu/drm/i915/display/intel_connector.c
index c65887870ddc..6143a9625e5e 100644
--- a/drivers/gpu/drm/i915/display/intel_connector.c
+++ b/drivers/gpu/drm/i915/display/intel_connector.c
@@ -103,8 +103,8 @@ void intel_connector_destroy(struct drm_connector *connector)
drm_connector_cleanup(connector);
- if (intel_connector->port)
- drm_dp_mst_put_port_malloc(intel_connector->port);
+ if (intel_connector->mst.port)
+ drm_dp_mst_put_port_malloc(intel_connector->mst.port);
kfree(connector);
}
diff --git a/drivers/gpu/drm/i915/display/intel_cursor.c b/drivers/gpu/drm/i915/display/intel_cursor.c
index aeadb834d332..2a70b2f7b39c 100644
--- a/drivers/gpu/drm/i915/display/intel_cursor.c
+++ b/drivers/gpu/drm/i915/display/intel_cursor.c
@@ -536,7 +536,8 @@ static void i9xx_cursor_disable_sel_fetch_arm(struct intel_dsb *dsb,
struct intel_display *display = to_intel_display(plane->base.dev);
enum pipe pipe = plane->pipe;
- if (!crtc_state->enable_psr2_sel_fetch)
+ if (!crtc_state->enable_psr2_sel_fetch &&
+ !crtc_state->clear_psr2_sel_fetch)
return;
intel_de_write_dsb(display, dsb, SEL_FETCH_CUR_CTL(pipe), 0);
@@ -571,8 +572,10 @@ static void i9xx_cursor_update_sel_fetch_arm(struct intel_dsb *dsb,
struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
enum pipe pipe = plane->pipe;
- if (!crtc_state->enable_psr2_sel_fetch)
+ if (!crtc_state->enable_psr2_sel_fetch) {
+ i9xx_cursor_disable_sel_fetch_arm(dsb, plane, crtc_state);
return;
+ }
if (drm_rect_height(&plane_state->psr2_sel_fetch_area) > 0) {
if (crtc_state->enable_psr2_su_region_et) {
diff --git a/drivers/gpu/drm/i915/display/intel_ddi.c b/drivers/gpu/drm/i915/display/intel_ddi.c
index 5b24460c0134..0eace4d21d5d 100644
--- a/drivers/gpu/drm/i915/display/intel_ddi.c
+++ b/drivers/gpu/drm/i915/display/intel_ddi.c
@@ -2585,7 +2585,7 @@ static void mtl_ddi_pre_enable_dp(struct intel_atomic_state *state,
* Train Display Port" step. Note that steps that are specific to
* MST will be handled by intel_mst_pre_enable_dp() before/after it
* calls into this function. Also intel_mst_pre_enable_dp() only calls
- * us when active_mst_links==0, so any steps designated for "single
+ * us when mst.active_links==0, so any steps designated for "single
* stream or multi-stream master transcoder" can just be performed
* unconditionally here.
*
@@ -2674,7 +2674,7 @@ static void tgl_ddi_pre_enable_dp(struct intel_atomic_state *state,
* Train Display Port" step. Note that steps that are specific to
* MST will be handled by intel_mst_pre_enable_dp() before/after it
* calls into this function. Also intel_mst_pre_enable_dp() only calls
- * us when active_mst_links==0, so any steps designated for "single
+ * us when mst.active_links==0, so any steps designated for "single
* stream or multi-stream master transcoder" can just be performed
* unconditionally here.
*/
diff --git a/drivers/gpu/drm/i915/display/intel_display_debugfs.c b/drivers/gpu/drm/i915/display/intel_display_debugfs.c
index f5f618199d39..c6e98bb57827 100644
--- a/drivers/gpu/drm/i915/display/intel_display_debugfs.c
+++ b/drivers/gpu/drm/i915/display/intel_display_debugfs.c
@@ -275,7 +275,7 @@ static void intel_connector_info(struct seq_file *m,
switch (connector->connector_type) {
case DRM_MODE_CONNECTOR_DisplayPort:
case DRM_MODE_CONNECTOR_eDP:
- if (intel_connector->mst_port)
+ if (intel_connector->mst.dp)
intel_dp_mst_info(m, intel_connector);
else
intel_dp_info(m, intel_connector);
@@ -288,7 +288,7 @@ static void intel_connector_info(struct seq_file *m,
}
seq_puts(m, "\tHDCP version: ");
- if (intel_connector->mst_port) {
+ if (intel_connector->mst.dp) {
intel_hdcp_info(m, intel_connector, true);
seq_puts(m, "\tMST Hub HDCP version: ");
}
@@ -766,7 +766,7 @@ static int i915_dp_mst_info(struct seq_file *m, void *unused)
seq_printf(m, "MST Source Port [ENCODER:%d:%s]\n",
dig_port->base.base.base.id,
dig_port->base.base.name);
- drm_dp_mst_dump_topology(m, &dig_port->dp.mst_mgr);
+ drm_dp_mst_dump_topology(m, &dig_port->dp.mst.mgr);
}
drm_connector_list_iter_end(&conn_iter);
@@ -1533,7 +1533,7 @@ void intel_connector_debugfs_add(struct intel_connector *connector)
}
if (DISPLAY_VER(i915) >= 11 &&
- ((connector_type == DRM_MODE_CONNECTOR_DisplayPort && !connector->mst_port) ||
+ ((connector_type == DRM_MODE_CONNECTOR_DisplayPort && !connector->mst.dp) ||
connector_type == DRM_MODE_CONNECTOR_eDP)) {
debugfs_create_file("i915_dsc_fec_support", 0644, root,
connector, &i915_dsc_fec_support_fops);
diff --git a/drivers/gpu/drm/i915/display/intel_display_types.h b/drivers/gpu/drm/i915/display/intel_display_types.h
index 992945b37190..a76d7abcd9cf 100644
--- a/drivers/gpu/drm/i915/display/intel_display_types.h
+++ b/drivers/gpu/drm/i915/display/intel_display_types.h
@@ -647,10 +647,6 @@ struct intel_connector {
state of connector->polled in case hotplug storm detection changes it */
u8 polled;
- struct drm_dp_mst_port *port;
-
- struct intel_dp *mst_port;
-
bool force_bigjoiner_enable;
struct {
@@ -662,6 +658,11 @@ struct intel_connector {
u8 dsc_decompression_enabled:1;
} dp;
+ struct {
+ struct drm_dp_mst_port *port;
+ struct intel_dp *dp;
+ } mst;
+
/* Work struct to schedule a uevent on link train failure */
struct work_struct modeset_retry_work;
@@ -1213,6 +1214,8 @@ struct intel_crtc_state {
bool has_sel_update;
bool enable_psr2_sel_fetch;
bool enable_psr2_su_region_et;
+ /* Drop the stale selective fetch enable bits as selective fetch is turned off */
+ bool clear_psr2_sel_fetch;
bool req_psr2_sdp_prior_scanline;
bool has_panel_replay;
bool wm_level_disabled;
@@ -1832,7 +1835,6 @@ struct intel_dp {
struct intel_pps pps;
bool is_mst;
- int active_mst_links;
enum drm_dp_mst_mode mst_detect;
/* connector directly attached - won't be use for modeset in mst world */
@@ -1842,9 +1844,11 @@ struct intel_dp {
struct drm_dp_tunnel *tunnel;
bool tunnel_suspended:1;
- /* mst connector list */
- struct intel_dp_mst_encoder *mst_encoders[I915_MAX_PIPES];
- struct drm_dp_mst_topology_mgr mst_mgr;
+ struct {
+ struct intel_dp_mst_encoder *stream_encoders[I915_MAX_PIPES];
+ struct drm_dp_mst_topology_mgr mgr;
+ int active_links;
+ } mst;
u32 (*get_aux_clock_divider)(struct intel_dp *dp, int index);
/*
diff --git a/drivers/gpu/drm/i915/display/intel_dp.c b/drivers/gpu/drm/i915/display/intel_dp.c
index bb5a2a40b64c..63c335d3d9ce 100644
--- a/drivers/gpu/drm/i915/display/intel_dp.c
+++ b/drivers/gpu/drm/i915/display/intel_dp.c
@@ -1290,7 +1290,7 @@ bool intel_dp_has_dsc(const struct intel_connector *connector)
if (!HAS_DSC(i915))
return false;
- if (connector->mst_port && !HAS_DSC_MST(i915))
+ if (connector->mst.dp && !HAS_DSC_MST(i915))
return false;
if (connector->base.connector_type == DRM_MODE_CONNECTOR_eDP &&
@@ -3016,7 +3016,7 @@ intel_dp_queue_modeset_retry_for_link(struct intel_atomic_state *state,
if (!conn_state->base.crtc)
continue;
- if (connector->mst_port == intel_dp)
+ if (connector->mst.dp == intel_dp)
intel_dp_queue_modeset_retry_work(connector);
}
}
@@ -3236,8 +3236,8 @@ intel_dp_sink_set_dsc_passthrough(const struct intel_connector *connector,
bool enable)
{
struct drm_i915_private *i915 = to_i915(connector->base.dev);
- struct drm_dp_aux *aux = connector->port ?
- connector->port->passthrough_aux : NULL;
+ struct drm_dp_aux *aux = connector->mst.port ?
+ connector->mst.port->passthrough_aux : NULL;
if (!aux)
return;
@@ -3264,7 +3264,7 @@ static int intel_dp_dsc_aux_ref_count(struct intel_atomic_state *state,
* On SST the decompression AUX device won't be shared, each connector
* uses for this its own AUX targeting the sink device.
*/
- if (!connector->mst_port)
+ if (!connector->mst.dp)
return connector->dp.dsc_decompression_enabled ? 1 : 0;
for_each_oldnew_connector_in_state(&state->base, _connector_iter,
@@ -3272,7 +3272,7 @@ static int intel_dp_dsc_aux_ref_count(struct intel_atomic_state *state,
const struct intel_connector *
connector_iter = to_intel_connector(_connector_iter);
- if (connector_iter->mst_port != connector->mst_port)
+ if (connector_iter->mst.dp != connector->mst.dp)
continue;
if (!connector_iter->dp.dsc_decompression_enabled)
@@ -4291,7 +4291,7 @@ intel_dp_mst_configure(struct intel_dp *intel_dp)
if (intel_dp->is_mst)
intel_dp_mst_prepare_probe(intel_dp);
- drm_dp_mst_topology_mgr_set_mst(&intel_dp->mst_mgr, intel_dp->is_mst);
+ drm_dp_mst_topology_mgr_set_mst(&intel_dp->mst.mgr, intel_dp->is_mst);
/* Avoid stale info on the next detect cycle. */
intel_dp->mst_detect = DRM_DP_SST;
@@ -4306,9 +4306,9 @@ intel_dp_mst_disconnect(struct intel_dp *intel_dp)
return;
drm_dbg_kms(&i915->drm, "MST device may have disappeared %d vs %d\n",
- intel_dp->is_mst, intel_dp->mst_mgr.mst_state);
+ intel_dp->is_mst, intel_dp->mst.mgr.mst_state);
intel_dp->is_mst = false;
- drm_dp_mst_topology_mgr_set_mst(&intel_dp->mst_mgr, intel_dp->is_mst);
+ drm_dp_mst_topology_mgr_set_mst(&intel_dp->mst.mgr, intel_dp->is_mst);
}
static bool
@@ -5154,7 +5154,7 @@ intel_dp_mst_hpd_irq(struct intel_dp *intel_dp, u8 *esi, u8 *ack)
{
bool handled = false;
- drm_dp_mst_hpd_irq_handle_event(&intel_dp->mst_mgr, esi, ack, &handled);
+ drm_dp_mst_hpd_irq_handle_event(&intel_dp->mst.mgr, esi, ack, &handled);
if (esi[1] & DP_CP_IRQ) {
intel_hdcp_handle_cp_irq(intel_dp->attached_connector);
@@ -5203,7 +5203,7 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp)
bool link_ok = true;
bool reprobe_needed = false;
- drm_WARN_ON_ONCE(&i915->drm, intel_dp->active_mst_links < 0);
+ drm_WARN_ON_ONCE(&i915->drm, intel_dp->mst.active_links < 0);
for (;;) {
u8 esi[4] = {};
@@ -5219,7 +5219,7 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp)
drm_dbg_kms(&i915->drm, "DPRX ESI: %4ph\n", esi);
- if (intel_dp->active_mst_links > 0 && link_ok &&
+ if (intel_dp->mst.active_links > 0 && link_ok &&
esi[3] & LINK_STATUS_CHANGED) {
if (!intel_dp_mst_link_status(intel_dp))
link_ok = false;
@@ -5242,7 +5242,7 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp)
drm_dbg_kms(&i915->drm, "Failed to ack ESI\n");
if (ack[1] & (DP_DOWN_REP_MSG_RDY | DP_UP_REQ_MSG_RDY))
- drm_dp_mst_hpd_irq_send_new_request(&intel_dp->mst_mgr);
+ drm_dp_mst_hpd_irq_send_new_request(&intel_dp->mst.mgr);
}
if (!link_ok || intel_dp->link.force_retrain)
@@ -5340,7 +5340,7 @@ static bool intel_dp_has_connector(struct intel_dp *intel_dp,
/* MST */
for_each_pipe(i915, pipe) {
- encoder = &intel_dp->mst_encoders[pipe]->base;
+ encoder = &intel_dp->mst.stream_encoders[pipe]->base;
if (conn_state->best_encoder == &encoder->base)
return true;
}
@@ -6421,7 +6421,7 @@ static int intel_dp_connector_atomic_check(struct drm_connector *conn,
return ret;
if (intel_dp_mst_source_support(intel_dp)) {
- ret = drm_dp_mst_root_conn_atomic_check(conn_state, &intel_dp->mst_mgr);
+ ret = drm_dp_mst_root_conn_atomic_check(conn_state, &intel_dp->mst.mgr);
if (ret)
return ret;
}
@@ -7000,7 +7000,7 @@ void intel_dp_mst_suspend(struct drm_i915_private *dev_priv)
continue;
if (intel_dp->is_mst)
- drm_dp_mst_topology_mgr_suspend(&intel_dp->mst_mgr);
+ drm_dp_mst_topology_mgr_suspend(&intel_dp->mst.mgr);
}
}
@@ -7023,11 +7023,11 @@ void intel_dp_mst_resume(struct drm_i915_private *dev_priv)
if (!intel_dp_mst_source_support(intel_dp))
continue;
- ret = drm_dp_mst_topology_mgr_resume(&intel_dp->mst_mgr,
+ ret = drm_dp_mst_topology_mgr_resume(&intel_dp->mst.mgr,
true);
if (ret) {
intel_dp->is_mst = false;
- drm_dp_mst_topology_mgr_set_mst(&intel_dp->mst_mgr,
+ drm_dp_mst_topology_mgr_set_mst(&intel_dp->mst.mgr,
false);
}
}
diff --git a/drivers/gpu/drm/i915/display/intel_dp_hdcp.c b/drivers/gpu/drm/i915/display/intel_dp_hdcp.c
index 3425b3643143..c31b78255d22 100644
--- a/drivers/gpu/drm/i915/display/intel_dp_hdcp.c
+++ b/drivers/gpu/drm/i915/display/intel_dp_hdcp.c
@@ -702,10 +702,10 @@ int intel_dp_hdcp_get_remote_capability(struct intel_connector *connector,
*hdcp_capable = false;
*hdcp2_capable = false;
- if (!connector->mst_port)
+ if (!connector->mst.dp)
return -EINVAL;
- aux = &connector->port->aux;
+ aux = &connector->mst.port->aux;
ret = _intel_dp_hdcp2_get_capability(aux, hdcp2_capable);
if (ret)
drm_dbg_kms(&i915->drm,
@@ -881,7 +881,7 @@ int intel_dp_hdcp_init(struct intel_digital_port *dig_port,
if (!is_hdcp_supported(dev_priv, port))
return 0;
- if (intel_connector->mst_port)
+ if (intel_connector->mst.dp)
return intel_hdcp_init(intel_connector, dig_port,
&intel_dp_mst_hdcp_shim);
else if (!intel_dp_is_edp(intel_dp))
diff --git a/drivers/gpu/drm/i915/display/intel_dp_link_training.c b/drivers/gpu/drm/i915/display/intel_dp_link_training.c
index 5d8f93d4cdc6..e112f9b4a89e 100644
--- a/drivers/gpu/drm/i915/display/intel_dp_link_training.c
+++ b/drivers/gpu/drm/i915/display/intel_dp_link_training.c
@@ -1692,8 +1692,8 @@ void intel_dp_128b132b_sdp_crc16(struct intel_dp *intel_dp,
static struct intel_dp *intel_connector_to_intel_dp(struct intel_connector *connector)
{
- if (connector->mst_port)
- return connector->mst_port;
+ if (connector->mst.dp)
+ return connector->mst.dp;
else
return enc_to_intel_dp(intel_attached_encoder(connector));
}
diff --git a/drivers/gpu/drm/i915/display/intel_dp_mst.c b/drivers/gpu/drm/i915/display/intel_dp_mst.c
index eeaedd979354..5d5d8adcfb6a 100644
--- a/drivers/gpu/drm/i915/display/intel_dp_mst.c
+++ b/drivers/gpu/drm/i915/display/intel_dp_mst.c
@@ -183,7 +183,7 @@ static int intel_dp_mst_find_vcpi_slots_for_bpp(struct intel_encoder *encoder,
int max_dpt_bpp;
int ret = 0;
- mst_state = drm_atomic_get_mst_topology_state(state, &intel_dp->mst_mgr);
+ mst_state = drm_atomic_get_mst_topology_state(state, &intel_dp->mst.mgr);
if (IS_ERR(mst_state))
return PTR_ERR(mst_state);
@@ -197,7 +197,7 @@ static int intel_dp_mst_find_vcpi_slots_for_bpp(struct intel_encoder *encoder,
crtc_state->fec_enable = !intel_dp_is_uhbr(crtc_state);
}
- mst_state->pbn_div = drm_dp_get_vc_payload_bw(&intel_dp->mst_mgr,
+ mst_state->pbn_div = drm_dp_get_vc_payload_bw(&intel_dp->mst.mgr,
crtc_state->port_clock,
crtc_state->lane_count);
@@ -283,8 +283,8 @@ static int intel_dp_mst_find_vcpi_slots_for_bpp(struct intel_encoder *encoder,
drm_WARN_ON(&i915->drm, remote_tu < crtc_state->dp_m_n.tu);
crtc_state->dp_m_n.tu = remote_tu;
- slots = drm_dp_atomic_find_time_slots(state, &intel_dp->mst_mgr,
- connector->port,
+ slots = drm_dp_atomic_find_time_slots(state, &intel_dp->mst.mgr,
+ connector->mst.port,
crtc_state->pbn);
if (slots == -EDEADLK)
return slots;
@@ -423,7 +423,7 @@ static int intel_dp_mst_update_slots(struct intel_encoder *encoder,
struct drm_i915_private *i915 = to_i915(encoder->base.dev);
struct intel_dp_mst_encoder *intel_mst = enc_to_mst(encoder);
struct intel_dp *intel_dp = &intel_mst->primary->dp;
- struct drm_dp_mst_topology_mgr *mgr = &intel_dp->mst_mgr;
+ struct drm_dp_mst_topology_mgr *mgr = &intel_dp->mst.mgr;
struct drm_dp_mst_topology_state *topology_state;
u8 link_coding_cap = intel_dp_is_uhbr(crtc_state) ?
DP_CAP_ANSI_128B132B : DP_CAP_ANSI_8B10B;
@@ -453,8 +453,8 @@ hblank_expansion_quirk_needs_dsc(const struct intel_connector *connector,
{
const struct drm_display_mode *adjusted_mode =
&crtc_state->hw.adjusted_mode;
- bool is_uhbr_sink = connector->mst_port &&
- drm_dp_128b132b_supported(connector->mst_port->dpcd);
+ bool is_uhbr_sink = connector->mst.dp &&
+ drm_dp_128b132b_supported(connector->mst.dp->dpcd);
int hblank_limit = is_uhbr_sink ? 500 : 300;
if (!connector->dp.dsc_hblank_expansion_quirk)
@@ -708,7 +708,7 @@ intel_dp_mst_transcoder_mask(struct intel_atomic_state *state,
const struct intel_crtc_state *crtc_state;
struct intel_crtc *crtc;
- if (connector->mst_port != mst_port || !conn_state->base.crtc)
+ if (connector->mst.dp != mst_port || !conn_state->base.crtc)
continue;
crtc = to_intel_crtc(conn_state->base.crtc);
@@ -736,12 +736,13 @@ static u8 get_pipes_downstream_of_mst_port(struct intel_atomic_state *state,
if (!conn_state->base.crtc)
continue;
- if (&connector->mst_port->mst_mgr != mst_mgr)
+ if (&connector->mst.dp->mst.mgr != mst_mgr)
continue;
- if (connector->port != parent_port &&
+ if (parent_port &&
+ connector->mst.port != parent_port &&
!drm_dp_mst_port_downstream_of_parent(mst_mgr,
- connector->port,
+ connector->mst.port,
parent_port))
continue;
@@ -893,7 +894,7 @@ intel_dp_mst_atomic_topology_check(struct intel_connector *connector,
struct intel_crtc_state *crtc_state;
struct intel_crtc *crtc;
- if (connector_iter->mst_port != connector->mst_port ||
+ if (connector_iter->mst.dp != connector->mst.dp ||
connector_iter == connector)
continue;
@@ -943,15 +944,15 @@ intel_dp_mst_atomic_check(struct drm_connector *connector,
if (intel_connector_needs_modeset(state, connector)) {
ret = intel_dp_tunnel_atomic_check_state(state,
- intel_connector->mst_port,
+ intel_connector->mst.dp,
intel_connector);
if (ret)
return ret;
}
return drm_dp_atomic_release_time_slots(&state->base,
- &intel_connector->mst_port->mst_mgr,
- intel_connector->port);
+ &intel_connector->mst.dp->mst.mgr,
+ intel_connector->mst.port);
}
static void clear_act_sent(struct intel_encoder *encoder,
@@ -974,7 +975,7 @@ static void wait_for_act_sent(struct intel_encoder *encoder,
DP_TP_STATUS_ACT_SENT, 1))
drm_err(&i915->drm, "Timed out waiting for ACT sent\n");
- drm_dp_check_act_status(&intel_dp->mst_mgr);
+ drm_dp_check_act_status(&intel_dp->mst.mgr);
}
static void intel_mst_disable_dp(struct intel_atomic_state *state,
@@ -990,9 +991,9 @@ static void intel_mst_disable_dp(struct intel_atomic_state *state,
struct drm_i915_private *i915 = to_i915(connector->base.dev);
drm_dbg_kms(&i915->drm, "active links %d\n",
- intel_dp->active_mst_links);
+ intel_dp->mst.active_links);
- if (intel_dp->active_mst_links == 1)
+ if (intel_dp->mst.active_links == 1)
intel_dp->link_trained = false;
intel_hdcp_disable(intel_mst->connector);
@@ -1011,19 +1012,19 @@ static void intel_mst_post_disable_dp(struct intel_atomic_state *state,
struct intel_connector *connector =
to_intel_connector(old_conn_state->connector);
struct drm_dp_mst_topology_state *old_mst_state =
- drm_atomic_get_old_mst_topology_state(&state->base, &intel_dp->mst_mgr);
+ drm_atomic_get_old_mst_topology_state(&state->base, &intel_dp->mst.mgr);
struct drm_dp_mst_topology_state *new_mst_state =
- drm_atomic_get_new_mst_topology_state(&state->base, &intel_dp->mst_mgr);
+ drm_atomic_get_new_mst_topology_state(&state->base, &intel_dp->mst.mgr);
const struct drm_dp_mst_atomic_payload *old_payload =
- drm_atomic_get_mst_payload_state(old_mst_state, connector->port);
+ drm_atomic_get_mst_payload_state(old_mst_state, connector->mst.port);
struct drm_dp_mst_atomic_payload *new_payload =
- drm_atomic_get_mst_payload_state(new_mst_state, connector->port);
+ drm_atomic_get_mst_payload_state(new_mst_state, connector->mst.port);
struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
struct intel_crtc *pipe_crtc;
bool last_mst_stream;
- intel_dp->active_mst_links--;
- last_mst_stream = intel_dp->active_mst_links == 0;
+ intel_dp->mst.active_links--;
+ last_mst_stream = intel_dp->mst.active_links == 0;
drm_WARN_ON(&dev_priv->drm,
DISPLAY_VER(dev_priv) >= 12 && last_mst_stream &&
!intel_dp_mst_is_master_trans(old_crtc_state));
@@ -1038,7 +1039,7 @@ static void intel_mst_post_disable_dp(struct intel_atomic_state *state,
intel_disable_transcoder(old_crtc_state);
- drm_dp_remove_payload_part1(&intel_dp->mst_mgr, new_mst_state, new_payload);
+ drm_dp_remove_payload_part1(&intel_dp->mst.mgr, new_mst_state, new_payload);
clear_act_sent(encoder, old_crtc_state);
@@ -1048,7 +1049,7 @@ static void intel_mst_post_disable_dp(struct intel_atomic_state *state,
wait_for_act_sent(encoder, old_crtc_state);
- drm_dp_remove_payload_part2(&intel_dp->mst_mgr, new_mst_state,
+ drm_dp_remove_payload_part2(&intel_dp->mst.mgr, new_mst_state,
old_payload, new_payload);
intel_ddi_disable_transcoder_func(old_crtc_state);
@@ -1070,7 +1071,7 @@ static void intel_mst_post_disable_dp(struct intel_atomic_state *state,
* Power down mst path before disabling the port, otherwise we end
* up getting interrupts from the sink upon detecting link loss.
*/
- drm_dp_send_power_updown_phy(&intel_dp->mst_mgr, connector->port,
+ drm_dp_send_power_updown_phy(&intel_dp->mst.mgr, connector->mst.port,
false);
/*
@@ -1096,7 +1097,7 @@ static void intel_mst_post_disable_dp(struct intel_atomic_state *state,
old_crtc_state, NULL);
drm_dbg_kms(&dev_priv->drm, "active links %d\n",
- intel_dp->active_mst_links);
+ intel_dp->mst.active_links);
}
static void intel_mst_post_pll_disable_dp(struct intel_atomic_state *state,
@@ -1108,7 +1109,7 @@ static void intel_mst_post_pll_disable_dp(struct intel_atomic_state *state,
struct intel_digital_port *dig_port = intel_mst->primary;
struct intel_dp *intel_dp = &dig_port->dp;
- if (intel_dp->active_mst_links == 0 &&
+ if (intel_dp->mst.active_links == 0 &&
dig_port->base.post_pll_disable)
dig_port->base.post_pll_disable(state, encoder, old_crtc_state, old_conn_state);
}
@@ -1122,7 +1123,7 @@ static void intel_mst_pre_pll_enable_dp(struct intel_atomic_state *state,
struct intel_digital_port *dig_port = intel_mst->primary;
struct intel_dp *intel_dp = &dig_port->dp;
- if (intel_dp->active_mst_links == 0)
+ if (intel_dp->mst.active_links == 0)
dig_port->base.pre_pll_enable(state, &dig_port->base,
pipe_config, NULL);
else
@@ -1155,7 +1156,7 @@ static void intel_mst_reprobe_topology(struct intel_dp *intel_dp,
crtc_state->port_clock, crtc_state->lane_count))
return;
- drm_dp_mst_topology_queue_probe(&intel_dp->mst_mgr);
+ drm_dp_mst_topology_queue_probe(&intel_dp->mst.mgr);
intel_mst_set_probed_link_params(intel_dp,
crtc_state->port_clock, crtc_state->lane_count);
@@ -1173,7 +1174,7 @@ static void intel_mst_pre_enable_dp(struct intel_atomic_state *state,
struct intel_connector *connector =
to_intel_connector(conn_state->connector);
struct drm_dp_mst_topology_state *mst_state =
- drm_atomic_get_new_mst_topology_state(&state->base, &intel_dp->mst_mgr);
+ drm_atomic_get_new_mst_topology_state(&state->base, &intel_dp->mst.mgr);
int ret;
bool first_mst_stream;
@@ -1182,18 +1183,18 @@ static void intel_mst_pre_enable_dp(struct intel_atomic_state *state,
*/
connector->encoder = encoder;
intel_mst->connector = connector;
- first_mst_stream = intel_dp->active_mst_links == 0;
+ first_mst_stream = intel_dp->mst.active_links == 0;
drm_WARN_ON(&dev_priv->drm,
DISPLAY_VER(dev_priv) >= 12 && first_mst_stream &&
!intel_dp_mst_is_master_trans(pipe_config));
drm_dbg_kms(&dev_priv->drm, "active links %d\n",
- intel_dp->active_mst_links);
+ intel_dp->mst.active_links);
if (first_mst_stream)
intel_dp_set_power(intel_dp, DP_SET_POWER_D0);
- drm_dp_send_power_updown_phy(&intel_dp->mst_mgr, connector->port, true);
+ drm_dp_send_power_updown_phy(&intel_dp->mst.mgr, connector->mst.port, true);
intel_dp_sink_enable_decompression(state, connector, pipe_config);
@@ -1204,10 +1205,10 @@ static void intel_mst_pre_enable_dp(struct intel_atomic_state *state,
intel_mst_reprobe_topology(intel_dp, pipe_config);
}
- intel_dp->active_mst_links++;
+ intel_dp->mst.active_links++;
- ret = drm_dp_add_payload_part1(&intel_dp->mst_mgr, mst_state,
- drm_atomic_get_mst_payload_state(mst_state, connector->port));
+ ret = drm_dp_add_payload_part1(&intel_dp->mst.mgr, mst_state,
+ drm_atomic_get_mst_payload_state(mst_state, connector->mst.port));
if (ret < 0)
intel_dp_queue_modeset_retry_for_link(state, &dig_port->base, pipe_config);
@@ -1269,9 +1270,9 @@ static void intel_mst_enable_dp(struct intel_atomic_state *state,
struct intel_connector *connector = to_intel_connector(conn_state->connector);
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct drm_dp_mst_topology_state *mst_state =
- drm_atomic_get_new_mst_topology_state(&state->base, &intel_dp->mst_mgr);
+ drm_atomic_get_new_mst_topology_state(&state->base, &intel_dp->mst.mgr);
enum transcoder trans = pipe_config->cpu_transcoder;
- bool first_mst_stream = intel_dp->active_mst_links == 1;
+ bool first_mst_stream = intel_dp->mst.active_links == 1;
struct intel_crtc *pipe_crtc;
int ret;
@@ -1298,16 +1299,16 @@ static void intel_mst_enable_dp(struct intel_atomic_state *state,
TRANS_DDI_DP_VC_PAYLOAD_ALLOC);
drm_dbg_kms(&dev_priv->drm, "active links %d\n",
- intel_dp->active_mst_links);
+ intel_dp->mst.active_links);
wait_for_act_sent(encoder, pipe_config);
if (first_mst_stream)
intel_ddi_wait_for_fec_status(encoder, pipe_config, true);
- ret = drm_dp_add_payload_part2(&intel_dp->mst_mgr,
+ ret = drm_dp_add_payload_part2(&intel_dp->mst.mgr,
drm_atomic_get_mst_payload_state(mst_state,
- connector->port));
+ connector->mst.port));
if (ret < 0)
intel_dp_queue_modeset_retry_for_link(state, &dig_port->base, pipe_config);
@@ -1363,7 +1364,7 @@ static int intel_dp_mst_get_ddc_modes(struct drm_connector *connector)
{
struct intel_connector *intel_connector = to_intel_connector(connector);
struct drm_i915_private *i915 = to_i915(intel_connector->base.dev);
- struct intel_dp *intel_dp = intel_connector->mst_port;
+ struct intel_dp *intel_dp = intel_connector->mst.dp;
const struct drm_edid *drm_edid;
int ret;
@@ -1373,7 +1374,7 @@ static int intel_dp_mst_get_ddc_modes(struct drm_connector *connector)
if (!intel_display_driver_check_access(i915))
return drm_edid_connector_add_modes(connector);
- drm_edid = drm_dp_mst_edid_read(connector, &intel_dp->mst_mgr, intel_connector->port);
+ drm_edid = drm_dp_mst_edid_read(connector, &intel_dp->mst.mgr, intel_connector->mst.port);
ret = intel_connector_update_modes(connector, drm_edid);
@@ -1389,14 +1390,14 @@ intel_dp_mst_connector_late_register(struct drm_connector *connector)
int ret;
ret = drm_dp_mst_connector_late_register(connector,
- intel_connector->port);
+ intel_connector->mst.port);
if (ret < 0)
return ret;
ret = intel_connector_register(connector);
if (ret < 0)
drm_dp_mst_connector_early_unregister(connector,
- intel_connector->port);
+ intel_connector->mst.port);
return ret;
}
@@ -1408,7 +1409,7 @@ intel_dp_mst_connector_early_unregister(struct drm_connector *connector)
intel_connector_unregister(connector);
drm_dp_mst_connector_early_unregister(connector,
- intel_connector->port);
+ intel_connector->mst.port);
}
static const struct drm_connector_funcs intel_dp_mst_connector_funcs = {
@@ -1435,9 +1436,9 @@ intel_dp_mst_mode_valid_ctx(struct drm_connector *connector,
{
struct drm_i915_private *dev_priv = to_i915(connector->dev);
struct intel_connector *intel_connector = to_intel_connector(connector);
- struct intel_dp *intel_dp = intel_connector->mst_port;
- struct drm_dp_mst_topology_mgr *mgr = &intel_dp->mst_mgr;
- struct drm_dp_mst_port *port = intel_connector->port;
+ struct intel_dp *intel_dp = intel_connector->mst.dp;
+ struct drm_dp_mst_topology_mgr *mgr = &intel_dp->mst.mgr;
+ struct drm_dp_mst_port *port = intel_connector->mst.port;
const int min_bpp = 18;
int max_dotclk = to_i915(connector->dev)->display.cdclk.max_dotclk_freq;
int max_rate, mode_rate, max_lanes, max_link_clock;
@@ -1548,10 +1549,10 @@ static struct drm_encoder *intel_mst_atomic_best_encoder(struct drm_connector *c
struct drm_connector_state *connector_state = drm_atomic_get_new_connector_state(state,
connector);
struct intel_connector *intel_connector = to_intel_connector(connector);
- struct intel_dp *intel_dp = intel_connector->mst_port;
+ struct intel_dp *intel_dp = intel_connector->mst.dp;
struct intel_crtc *crtc = to_intel_crtc(connector_state->crtc);
- return &intel_dp->mst_encoders[crtc->pipe]->base.base;
+ return &intel_dp->mst.stream_encoders[crtc->pipe]->base.base;
}
static int
@@ -1560,7 +1561,7 @@ intel_dp_mst_detect(struct drm_connector *connector,
{
struct drm_i915_private *i915 = to_i915(connector->dev);
struct intel_connector *intel_connector = to_intel_connector(connector);
- struct intel_dp *intel_dp = intel_connector->mst_port;
+ struct intel_dp *intel_dp = intel_connector->mst.dp;
if (!intel_display_device_enabled(i915))
return connector_status_disconnected;
@@ -1571,8 +1572,8 @@ intel_dp_mst_detect(struct drm_connector *connector,
if (!intel_display_driver_check_access(i915))
return connector->status;
- return drm_dp_mst_detect_port(connector, ctx, &intel_dp->mst_mgr,
- intel_connector->port);
+ return drm_dp_mst_detect_port(connector, ctx, &intel_dp->mst.mgr,
+ intel_connector->mst.port);
}
static const struct drm_connector_helper_funcs intel_dp_mst_connector_helper_funcs = {
@@ -1661,10 +1662,10 @@ static bool detect_dsc_hblank_expansion_quirk(const struct intel_connector *conn
* A logical port's OUI (at least for affected sinks) is all 0, so
* instead of that the parent port's OUI is used for identification.
*/
- if (drm_dp_mst_port_is_logical(connector->port)) {
- aux = drm_dp_mst_aux_for_parent(connector->port);
+ if (drm_dp_mst_port_is_logical(connector->mst.port)) {
+ aux = drm_dp_mst_aux_for_parent(connector->mst.port);
if (!aux)
- aux = &connector->mst_port->aux;
+ aux = &connector->mst.dp->aux;
}
if (drm_dp_read_dpcd_caps(aux, dpcd) < 0)
@@ -1698,7 +1699,7 @@ static struct drm_connector *intel_dp_add_mst_connector(struct drm_dp_mst_topolo
struct drm_dp_mst_port *port,
const char *pathprop)
{
- struct intel_dp *intel_dp = container_of(mgr, struct intel_dp, mst_mgr);
+ struct intel_dp *intel_dp = container_of(mgr, struct intel_dp, mst.mgr);
struct intel_digital_port *dig_port = dp_to_dig_port(intel_dp);
struct drm_device *dev = dig_port->base.base.dev;
struct drm_i915_private *dev_priv = to_i915(dev);
@@ -1713,8 +1714,8 @@ static struct drm_connector *intel_dp_add_mst_connector(struct drm_dp_mst_topolo
intel_connector->get_hw_state = intel_dp_mst_get_hw_state;
intel_connector->sync_state = intel_dp_connector_sync_state;
- intel_connector->mst_port = intel_dp;
- intel_connector->port = port;
+ intel_connector->mst.dp = intel_dp;
+ intel_connector->mst.port = port;
drm_dp_mst_get_port_malloc(port);
intel_dp_init_modeset_retry_work(intel_connector);
@@ -1737,7 +1738,7 @@ static struct drm_connector *intel_dp_add_mst_connector(struct drm_dp_mst_topolo
for_each_pipe(dev_priv, pipe) {
struct drm_encoder *enc =
- &intel_dp->mst_encoders[pipe]->base.base;
+ &intel_dp->mst.stream_encoders[pipe]->base.base;
ret = drm_connector_attach_encoder(&intel_connector->base, enc);
if (ret)
@@ -1763,7 +1764,7 @@ static struct drm_connector *intel_dp_add_mst_connector(struct drm_dp_mst_topolo
static void
intel_dp_mst_poll_hpd_irq(struct drm_dp_mst_topology_mgr *mgr)
{
- struct intel_dp *intel_dp = container_of(mgr, struct intel_dp, mst_mgr);
+ struct intel_dp *intel_dp = container_of(mgr, struct intel_dp, mst.mgr);
intel_hpd_trigger_irq(dp_to_dig_port(intel_dp));
}
@@ -1833,14 +1834,14 @@ intel_dp_create_fake_mst_encoders(struct intel_digital_port *dig_port)
enum pipe pipe;
for_each_pipe(dev_priv, pipe)
- intel_dp->mst_encoders[pipe] = intel_dp_create_fake_mst_encoder(dig_port, pipe);
+ intel_dp->mst.stream_encoders[pipe] = intel_dp_create_fake_mst_encoder(dig_port, pipe);
return true;
}
int
intel_dp_mst_encoder_active_links(struct intel_digital_port *dig_port)
{
- return dig_port->dp.active_mst_links;
+ return dig_port->dp.mst.active_links;
}
int
@@ -1860,14 +1861,14 @@ intel_dp_mst_encoder_init(struct intel_digital_port *dig_port, int conn_base_id)
if (DISPLAY_VER(i915) < 11 && port == PORT_E)
return 0;
- intel_dp->mst_mgr.cbs = &mst_cbs;
+ intel_dp->mst.mgr.cbs = &mst_cbs;
/* create encoders */
intel_dp_create_fake_mst_encoders(dig_port);
- ret = drm_dp_mst_topology_mgr_init(&intel_dp->mst_mgr, &i915->drm,
+ ret = drm_dp_mst_topology_mgr_init(&intel_dp->mst.mgr, &i915->drm,
&intel_dp->aux, 16, 3, conn_base_id);
if (ret) {
- intel_dp->mst_mgr.cbs = NULL;
+ intel_dp->mst.mgr.cbs = NULL;
return ret;
}
@@ -1876,7 +1877,7 @@ intel_dp_mst_encoder_init(struct intel_digital_port *dig_port, int conn_base_id)
bool intel_dp_mst_source_support(struct intel_dp *intel_dp)
{
- return intel_dp->mst_mgr.cbs;
+ return intel_dp->mst.mgr.cbs;
}
void
@@ -1887,10 +1888,10 @@ intel_dp_mst_encoder_cleanup(struct intel_digital_port *dig_port)
if (!intel_dp_mst_source_support(intel_dp))
return;
- drm_dp_mst_topology_mgr_destroy(&intel_dp->mst_mgr);
+ drm_dp_mst_topology_mgr_destroy(&intel_dp->mst.mgr);
/* encoders will get killed by normal cleanup */
- intel_dp->mst_mgr.cbs = NULL;
+ intel_dp->mst.mgr.cbs = NULL;
}
bool intel_dp_mst_is_master_trans(const struct intel_crtc_state *crtc_state)
@@ -1921,11 +1922,11 @@ intel_dp_mst_add_topology_state_for_connector(struct intel_atomic_state *state,
{
struct drm_dp_mst_topology_state *mst_state;
- if (!connector->mst_port)
+ if (!connector->mst.dp)
return 0;
mst_state = drm_atomic_get_mst_topology_state(&state->base,
- &connector->mst_port->mst_mgr);
+ &connector->mst.dp->mst.mgr);
if (IS_ERR(mst_state))
return PTR_ERR(mst_state);
@@ -2023,7 +2024,7 @@ bool intel_dp_mst_crtc_needs_modeset(struct intel_atomic_state *state,
const struct intel_crtc_state *old_crtc_state;
struct intel_crtc *crtc_iter;
- if (connector->mst_port != crtc_connector->mst_port ||
+ if (connector->mst.dp != crtc_connector->mst.dp ||
!conn_state->crtc)
continue;
@@ -2046,7 +2047,7 @@ bool intel_dp_mst_crtc_needs_modeset(struct intel_atomic_state *state,
* case.
*/
if (connector->dp.dsc_decompression_aux ==
- &connector->mst_port->aux)
+ &connector->mst.dp->aux)
return true;
}
@@ -2107,7 +2108,7 @@ bool intel_dp_mst_verify_dpcd_state(struct intel_dp *intel_dp)
if (!intel_dp->is_mst)
return true;
- ret = drm_dp_dpcd_readb(intel_dp->mst_mgr.aux, DP_MSTM_CTRL, &val);
+ ret = drm_dp_dpcd_readb(intel_dp->mst.mgr.aux, DP_MSTM_CTRL, &val);
/* Adjust the expected register value for SST + SideBand. */
if (ret < 0 || val != (DP_MST_EN | DP_UP_REQ_EN | DP_UPSTREAM_IS_SRC)) {
diff --git a/drivers/gpu/drm/i915/display/intel_fbdev.c b/drivers/gpu/drm/i915/display/intel_fbdev.c
index 2420be4c3127..d0074bd597bf 100644
--- a/drivers/gpu/drm/i915/display/intel_fbdev.c
+++ b/drivers/gpu/drm/i915/display/intel_fbdev.c
@@ -555,7 +555,7 @@ static int intel_fbdev_output_poll_changed(struct drm_device *dev)
return 0;
}
-static int intel_fbdev_restore_mode(struct drm_i915_private *dev_priv)
+static int intel_fbdev_restore_mode(struct drm_i915_private *dev_priv, bool force)
{
struct intel_fbdev *ifbdev = dev_priv->display.fbdev.fbdev;
int ret;
@@ -566,7 +566,7 @@ static int intel_fbdev_restore_mode(struct drm_i915_private *dev_priv)
if (!ifbdev->vma)
return -ENOMEM;
- ret = drm_fb_helper_restore_fbdev_mode_unlocked(&ifbdev->helper);
+ ret = drm_fb_helper_restore_fbdev_mode_unlocked(&ifbdev->helper, force);
if (ret)
return ret;
@@ -592,12 +592,12 @@ static void intel_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int intel_fbdev_client_restore(struct drm_client_dev *client)
+static int intel_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
struct drm_i915_private *dev_priv = to_i915(client->dev);
int ret;
- ret = intel_fbdev_restore_mode(dev_priv);
+ ret = intel_fbdev_restore_mode(dev_priv, force);
if (ret)
return ret;
diff --git a/drivers/gpu/drm/i915/display/intel_hdcp.c b/drivers/gpu/drm/i915/display/intel_hdcp.c
index ccc48dc14689..fb4aa8e05baf 100644
--- a/drivers/gpu/drm/i915/display/intel_hdcp.c
+++ b/drivers/gpu/drm/i915/display/intel_hdcp.c
@@ -62,13 +62,13 @@ static int intel_conn_to_vcpi(struct intel_atomic_state *state,
int vcpi = 0;
/* For HDMI this is forced to be 0x0. For DP SST also this is 0x0. */
- if (!connector->port)
+ if (!connector->mst.port)
return 0;
- mgr = connector->port->mgr;
+ mgr = connector->mst.port->mgr;
drm_modeset_lock(&mgr->base.lock, state->base.acquire_ctx);
mst_state = to_drm_dp_mst_topology_state(mgr->base.state);
- payload = drm_atomic_get_mst_payload_state(mst_state, connector->port);
+ payload = drm_atomic_get_mst_payload_state(mst_state, connector->mst.port);
if (drm_WARN_ON(mgr->dev, !payload))
goto out;
@@ -137,7 +137,7 @@ intel_hdcp_required_content_stream(struct intel_atomic_state *state,
data->k++;
/* if there is only one active stream */
- if (dig_port->dp.active_mst_links <= 1)
+ if (dig_port->dp.mst.active_links <= 1)
break;
}
drm_connector_list_iter_end(&conn_iter);
diff --git a/drivers/gpu/drm/i915/display/intel_psr.c b/drivers/gpu/drm/i915/display/intel_psr.c
index 5173f5759ce8..df16587ea964 100644
--- a/drivers/gpu/drm/i915/display/intel_psr.c
+++ b/drivers/gpu/drm/i915/display/intel_psr.c
@@ -2529,11 +2529,26 @@ static bool psr2_sel_fetch_pipe_state_supported(const struct intel_crtc_state *c
return true;
}
+static void
+intel_psr_apply_su_area_workarounds(struct intel_crtc_state *crtc_state)
+{
+ struct intel_display *display = to_intel_display(crtc_state);
+ struct drm_i915_private *i915 = to_i915(crtc_state->uapi.crtc->dev);
+
+ /* Wa_14014971492 */
+ if (!crtc_state->has_panel_replay &&
+ ((IS_DISPLAY_VER_STEP(display, IP_VER(14, 0), STEP_A0, STEP_B0) ||
+ IS_ALDERLAKE_P(i915) || IS_TIGERLAKE(i915))) &&
+ crtc_state->splitter.enable)
+ crtc_state->psr2_su_area.y1 = 0;
+}
+
int intel_psr2_sel_fetch_update(struct intel_atomic_state *state,
struct intel_crtc *crtc)
{
struct intel_display *display = to_intel_display(state);
- struct drm_i915_private *dev_priv = to_i915(state->base.dev);
+ const struct intel_crtc_state *old_crtc_state =
+ intel_atomic_get_old_crtc_state(state, crtc);
struct intel_crtc_state *crtc_state = intel_atomic_get_new_crtc_state(state, crtc);
struct intel_plane_state *new_plane_state, *old_plane_state;
struct intel_plane *plane;
@@ -2546,6 +2561,19 @@ int intel_psr2_sel_fetch_update(struct intel_atomic_state *state,
bool full_update = false, su_area_changed;
int i, ret;
+ /*
+ * Selective fetch is not always usable, for instance it is dropped
+ * while pipe CRC is active. The planes keep their selective fetch
+ * enable bit set in hardware over that, and a plane disabled while
+ * selective fetch is off never gets the bit cleared. Once selective
+ * fetch comes back the hardware would resume fetching for a plane that
+ * is no longer enabled and keep its DDB range reserved, so have the
+ * plane update drop the bit for every plane of the pipe as selective
+ * fetch is turned off.
+ */
+ crtc_state->clear_psr2_sel_fetch = old_crtc_state->enable_psr2_sel_fetch &&
+ !crtc_state->enable_psr2_sel_fetch;
+
if (!crtc_state->enable_psr2_sel_fetch)
return 0;
@@ -2644,12 +2672,7 @@ int intel_psr2_sel_fetch_update(struct intel_atomic_state *state,
if (full_update)
goto skip_sel_fetch_set_loop;
- /* Wa_14014971492 */
- if (!crtc_state->has_panel_replay &&
- ((IS_DISPLAY_VER_STEP(display, IP_VER(14, 0), STEP_A0, STEP_B0) ||
- IS_ALDERLAKE_P(dev_priv) || IS_TIGERLAKE(dev_priv))) &&
- crtc_state->splitter.enable)
- crtc_state->psr2_su_area.y1 = 0;
+ intel_psr_apply_su_area_workarounds(crtc_state);
ret = drm_atomic_add_affected_planes(&state->base, &crtc->base);
if (ret)
@@ -3928,7 +3951,7 @@ void intel_psr_connector_debugfs_add(struct intel_connector *connector)
/* TODO: Add support for MST connectors as well. */
if ((connector->base.connector_type != DRM_MODE_CONNECTOR_eDP &&
connector->base.connector_type != DRM_MODE_CONNECTOR_DisplayPort) ||
- connector->mst_port)
+ connector->mst.dp)
return;
debugfs_create_file("i915_psr_sink_status", 0444, root,
diff --git a/drivers/gpu/drm/i915/display/skl_universal_plane.c b/drivers/gpu/drm/i915/display/skl_universal_plane.c
index 7f77a76309bd..42b3bd848162 100644
--- a/drivers/gpu/drm/i915/display/skl_universal_plane.c
+++ b/drivers/gpu/drm/i915/display/skl_universal_plane.c
@@ -776,7 +776,8 @@ static void icl_plane_disable_sel_fetch_arm(struct intel_dsb *dsb,
struct intel_display *display = to_intel_display(plane->base.dev);
enum pipe pipe = plane->pipe;
- if (!crtc_state->enable_psr2_sel_fetch)
+ if (!crtc_state->enable_psr2_sel_fetch &&
+ !crtc_state->clear_psr2_sel_fetch)
return;
intel_de_write_dsb(display, dsb, SEL_FETCH_PLANE_CTL(pipe, plane->id), 0);
@@ -1505,10 +1506,8 @@ static void icl_plane_update_sel_fetch_arm(struct intel_dsb *dsb,
struct intel_display *display = to_intel_display(plane->base.dev);
enum pipe pipe = plane->pipe;
- if (!crtc_state->enable_psr2_sel_fetch)
- return;
-
- if (drm_rect_height(&plane_state->psr2_sel_fetch_area) > 0)
+ if (crtc_state->enable_psr2_sel_fetch &&
+ drm_rect_height(&plane_state->psr2_sel_fetch_area) > 0)
intel_de_write_dsb(display, dsb, SEL_FETCH_PLANE_CTL(pipe, plane->id),
SEL_FETCH_PLANE_CTL_ENABLE);
else
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 58e6c680fe0d..741dae40bd02 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -87,6 +87,7 @@ struct drm_i915_gem_object *i915_gem_object_alloc(void)
void i915_gem_object_free(struct drm_i915_gem_object *obj)
{
+ dma_resv_fini(&obj->base._resv);
return kmem_cache_free(slab_objects, obj);
}
@@ -142,7 +143,6 @@ void __i915_gem_object_fini(struct drm_i915_gem_object *obj)
{
mutex_destroy(&obj->mm.get_page.lock);
mutex_destroy(&obj->mm.get_dma_page.lock);
- dma_resv_fini(&obj->base._resv);
}
/**
diff --git a/drivers/gpu/drm/imagination/pvr_free_list.c b/drivers/gpu/drm/imagination/pvr_free_list.c
index 5e51bc980751..083ae6fa0976 100644
--- a/drivers/gpu/drm/imagination/pvr_free_list.c
+++ b/drivers/gpu/drm/imagination/pvr_free_list.c
@@ -8,6 +8,7 @@
#include "pvr_vm.h"
#include <drm/drm_gem.h>
+#include <drm/drm_print.h>
#include <linux/slab.h>
#include <linux/xarray.h>
#include <uapi/drm/pvr_drm.h>
@@ -613,13 +614,21 @@ pvr_free_list_process_reconstruct_req(struct pvr_device *pvr_dev,
};
struct rogue_fwif_freelists_reconstruction_data *resp =
&resp_cmd.cmd_data.free_lists_reconstruction_data;
+ u32 count = min_t(u32, req->freelist_count,
+ ARRAY_SIZE(req->freelist_ids));
- for (u32 i = 0; i < req->freelist_count; i++)
+ if (count != req->freelist_count) {
+ drm_warn_once(from_pvr_device(pvr_dev),
+ "Requested reconstruction of %u freelists, limiting to %u\n",
+ req->freelist_count, count);
+ }
+
+ for (u32 i = 0; i < count; i++)
pvr_free_list_reconstruct(pvr_dev, req->freelist_ids[i]);
- resp->freelist_count = req->freelist_count;
+ resp->freelist_count = count;
memcpy(resp->freelist_ids, req->freelist_ids,
- req->freelist_count * sizeof(resp->freelist_ids[0]));
+ count * sizeof(resp->freelist_ids[0]));
WARN_ON(pvr_kccb_send_cmd(pvr_dev, &resp_cmd, NULL));
}
diff --git a/drivers/gpu/drm/imagination/pvr_mmu.c b/drivers/gpu/drm/imagination/pvr_mmu.c
index 4fe70610ed94..67f56b47390c 100644
--- a/drivers/gpu/drm/imagination/pvr_mmu.c
+++ b/drivers/gpu/drm/imagination/pvr_mmu.c
@@ -12,6 +12,7 @@
#include "pvr_rogue_mmu_defs.h"
#include <drm/drm_drv.h>
+#include <drm/drm_print.h>
#include <linux/atomic.h>
#include <linux/bitops.h>
#include <linux/dma-mapping.h>
@@ -2331,6 +2332,7 @@ void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx)
* pvr_mmu_op_context_create() - Create an MMU op context.
* @ctx: MMU context associated with owning VM context.
* @sgt: Scatter gather table containing pages pinned for use by this context.
+ * @device_addr: Virtual device address at the start of the requested mapping.
* @sgt_offset: Start offset of the requested device-virtual memory mapping.
* @size: Size in bytes of the requested device-virtual memory mapping. For an
* unmapping, this should be zero so that no page tables are allocated.
@@ -2342,8 +2344,9 @@ void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx)
*/
struct pvr_mmu_op_context *
pvr_mmu_op_context_create(struct pvr_mmu_context *ctx, struct sg_table *sgt,
- u64 sgt_offset, u64 size)
+ u64 device_addr, u64 sgt_offset, u64 size)
{
+ u64 start_addr = device_addr + sgt_offset;
int err;
struct pvr_mmu_op_context *op_ctx =
@@ -2360,16 +2363,16 @@ pvr_mmu_op_context_create(struct pvr_mmu_context *ctx, struct sg_table *sgt,
if (size) {
/*
* The number of page table objects we need to prealloc is
- * indicated by the mapping size, start offset and the sizes
+ * indicated by the mapping size, start address and the sizes
* of the areas mapped per PT or PD. The range calculation is
* identical to that for the index into a table for a device
* address, so we reuse those functions here.
*/
- const u32 l1_start_idx = pvr_page_table_l2_idx(sgt_offset);
- const u32 l1_end_idx = pvr_page_table_l2_idx(sgt_offset + size);
+ const u32 l1_start_idx = pvr_page_table_l2_idx(start_addr);
+ const u32 l1_end_idx = pvr_page_table_l2_idx(start_addr + size);
const u32 l1_count = l1_end_idx - l1_start_idx + 1;
- const u32 l0_start_idx = pvr_page_table_l1_idx(sgt_offset);
- const u32 l0_end_idx = pvr_page_table_l1_idx(sgt_offset + size);
+ const u32 l0_start_idx = pvr_page_table_l1_idx(start_addr);
+ const u32 l0_end_idx = pvr_page_table_l1_idx(start_addr + size);
const u32 l0_count = l0_end_idx - l0_start_idx + 1;
/*
@@ -2550,7 +2553,9 @@ pvr_mmu_map_sgl(struct pvr_mmu_op_context *op_ctx, struct scatterlist *sgl,
err_destroy_pages:
memcpy(&op_ctx->curr_page, &ptr_copy, sizeof(op_ctx->curr_page));
- err = pvr_mmu_op_context_unmap_curr_page(op_ctx, page);
+ if (pvr_mmu_op_context_unmap_curr_page(op_ctx, page))
+ drm_err(from_pvr_device(op_ctx->mmu_ctx->pvr_dev),
+ "%s : Failure in unmapping pages\n", __func__);
return err;
}
diff --git a/drivers/gpu/drm/imagination/pvr_mmu.h b/drivers/gpu/drm/imagination/pvr_mmu.h
index a8ecd460168d..2c02d61ba0a2 100644
--- a/drivers/gpu/drm/imagination/pvr_mmu.h
+++ b/drivers/gpu/drm/imagination/pvr_mmu.h
@@ -99,7 +99,7 @@ dma_addr_t pvr_mmu_get_root_table_dma_addr(struct pvr_mmu_context *ctx);
void pvr_mmu_op_context_destroy(struct pvr_mmu_op_context *op_ctx);
struct pvr_mmu_op_context *
pvr_mmu_op_context_create(struct pvr_mmu_context *ctx,
- struct sg_table *sgt, u64 sgt_offset, u64 size);
+ struct sg_table *sgt, u64 device_addr, u64 sgt_offset, u64 size);
int pvr_mmu_map(struct pvr_mmu_op_context *op_ctx, u64 size, u64 flags,
u64 device_addr);
diff --git a/drivers/gpu/drm/imagination/pvr_vm.c b/drivers/gpu/drm/imagination/pvr_vm.c
index a118bf2b0ea1..5a4aca8be71f 100644
--- a/drivers/gpu/drm/imagination/pvr_vm.c
+++ b/drivers/gpu/drm/imagination/pvr_vm.c
@@ -270,7 +270,7 @@ pvr_vm_bind_op_map_init(struct pvr_vm_bind_op *bind_op,
goto err_bind_op_fini;
bind_op->mmu_op_ctx =
- pvr_mmu_op_context_create(vm_ctx->mmu_ctx, sgt, offset, size);
+ pvr_mmu_op_context_create(vm_ctx->mmu_ctx, sgt, device_addr, offset, size);
err = PTR_ERR_OR_ZERO(bind_op->mmu_op_ctx);
if (err) {
bind_op->mmu_op_ctx = NULL;
@@ -312,7 +312,7 @@ pvr_vm_bind_op_unmap_init(struct pvr_vm_bind_op *bind_op,
}
bind_op->mmu_op_ctx =
- pvr_mmu_op_context_create(vm_ctx->mmu_ctx, NULL, 0, 0);
+ pvr_mmu_op_context_create(vm_ctx->mmu_ctx, NULL, device_addr, 0, 0);
err = PTR_ERR_OR_ZERO(bind_op->mmu_op_ctx);
if (err) {
bind_op->mmu_op_ctx = NULL;
diff --git a/drivers/gpu/drm/msm/adreno/adreno_gpu.c b/drivers/gpu/drm/msm/adreno/adreno_gpu.c
index 465a4cd14a43..f4f0a8d4247c 100644
--- a/drivers/gpu/drm/msm/adreno/adreno_gpu.c
+++ b/drivers/gpu/drm/msm/adreno/adreno_gpu.c
@@ -1141,6 +1141,8 @@ void adreno_gpu_cleanup(struct adreno_gpu *adreno_gpu)
for (i = 0; i < ARRAY_SIZE(adreno_gpu->info->fw); i++)
release_firmware(adreno_gpu->fw[i]);
+ pm_runtime_dont_use_autosuspend(&gpu->pdev->dev);
+
if (priv && pm_runtime_enabled(&priv->gpu_pdev->dev))
pm_runtime_disable(&priv->gpu_pdev->dev);
diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_ctl.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_ctl.c
index 2e50049f2f85..b07d09171262 100644
--- a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_ctl.c
+++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_ctl.c
@@ -110,6 +110,7 @@ static inline void dpu_hw_ctl_clear_pending_flush(struct dpu_hw_ctl *ctx)
ctx->pending_flush_mask = 0x0;
ctx->pending_intf_flush_mask = 0;
ctx->pending_wb_flush_mask = 0;
+ ctx->pending_periph_flush_mask = 0;
ctx->pending_merge_3d_flush_mask = 0;
ctx->pending_dsc_flush_mask = 0;
ctx->pending_cdm_flush_mask = 0;
diff --git a/drivers/gpu/drm/msm/dsi/dsi_host.c b/drivers/gpu/drm/msm/dsi/dsi_host.c
index 77173183509f..cd605c75cef8 100644
--- a/drivers/gpu/drm/msm/dsi/dsi_host.c
+++ b/drivers/gpu/drm/msm/dsi/dsi_host.c
@@ -120,7 +120,7 @@ struct msm_dsi_host {
struct clk *byte_intf_clk;
unsigned long byte_clk_rate;
- unsigned long byte_intf_clk_rate;
+ bool byte_intf_clk_div_2;
unsigned long pixel_clk_rate;
unsigned long esc_clk_rate;
@@ -350,8 +350,20 @@ int msm_dsi_runtime_resume(struct device *dev)
int dsi_link_clk_set_rate_6g(struct msm_dsi_host *msm_host)
{
+ unsigned long byte_intf_clk_rate;
+ long rounded_byte_clk_rate;
int ret;
+ rounded_byte_clk_rate = clk_round_rate(msm_host->byte_clk,
+ msm_host->byte_clk_rate);
+ if (rounded_byte_clk_rate < 0) {
+ pr_err("%s: failed to round byte clock rate, %ld\n",
+ __func__, rounded_byte_clk_rate);
+ return rounded_byte_clk_rate;
+ }
+
+ msm_host->byte_clk_rate = rounded_byte_clk_rate;
+
DBG("Set clk rates: pclk=%lu, byteclk=%lu",
msm_host->pixel_clk_rate, msm_host->byte_clk_rate);
@@ -369,7 +381,11 @@ int dsi_link_clk_set_rate_6g(struct msm_dsi_host *msm_host)
}
if (msm_host->byte_intf_clk) {
- ret = clk_set_rate(msm_host->byte_intf_clk, msm_host->byte_intf_clk_rate);
+ byte_intf_clk_rate = msm_host->byte_clk_rate;
+ if (msm_host->byte_intf_clk_div_2)
+ byte_intf_clk_rate /= 2;
+
+ ret = clk_set_rate(msm_host->byte_intf_clk, byte_intf_clk_rate);
if (ret) {
pr_err("%s: Failed to set rate byte intf clk, %d\n",
__func__, ret);
@@ -619,24 +635,12 @@ static void dsi_calc_pclk(struct msm_dsi_host *msm_host, bool is_bonded_dsi)
int dsi_calc_clk_rate_6g(struct msm_dsi_host *msm_host, bool is_bonded_dsi)
{
- long rounded_byte_clk_rate;
-
if (!msm_host->mode) {
pr_err("%s: mode not set\n", __func__);
return -EINVAL;
}
dsi_calc_pclk(msm_host, is_bonded_dsi);
-
- rounded_byte_clk_rate = clk_round_rate(msm_host->byte_clk,
- msm_host->byte_clk_rate);
- if (rounded_byte_clk_rate < 0) {
- pr_err("%s: failed to round byte clock rate, %ld\n",
- __func__, rounded_byte_clk_rate);
- return rounded_byte_clk_rate;
- }
-
- msm_host->byte_clk_rate = rounded_byte_clk_rate;
msm_host->esc_clk_rate = clk_get_rate(msm_host->esc_clk);
return 0;
}
@@ -2419,9 +2423,7 @@ int msm_dsi_host_power_on(struct mipi_dsi_host *host,
goto unlock_ret;
}
- msm_host->byte_intf_clk_rate = msm_host->byte_clk_rate;
- if (phy_shared_timings->byte_intf_clk_div_2)
- msm_host->byte_intf_clk_rate /= 2;
+ msm_host->byte_intf_clk_div_2 = phy_shared_timings->byte_intf_clk_div_2;
msm_dsi_sfpb_config(msm_host, true);
diff --git a/drivers/gpu/drm/msm/hdmi/hdmi_phy.c b/drivers/gpu/drm/msm/hdmi/hdmi_phy.c
index 95b3f7535d84..9f3623cc747e 100644
--- a/drivers/gpu/drm/msm/hdmi/hdmi_phy.c
+++ b/drivers/gpu/drm/msm/hdmi/hdmi_phy.c
@@ -164,13 +164,13 @@ static int msm_hdmi_phy_probe(struct platform_device *pdev)
ret = msm_hdmi_phy_resource_enable(phy);
if (ret)
- return ret;
+ goto err_pm_disable;
ret = msm_hdmi_phy_pll_init(pdev, phy->cfg->type);
if (ret) {
DRM_DEV_ERROR(dev, "couldn't init PLL\n");
msm_hdmi_phy_resource_disable(phy);
- return ret;
+ goto err_pm_disable;
}
msm_hdmi_phy_resource_disable(phy);
@@ -178,6 +178,10 @@ static int msm_hdmi_phy_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, phy);
return 0;
+
+err_pm_disable:
+ pm_runtime_disable(dev);
+ return ret;
}
static void msm_hdmi_phy_remove(struct platform_device *pdev)
diff --git a/drivers/gpu/drm/msm/msm_fbdev.c b/drivers/gpu/drm/msm/msm_fbdev.c
index 174a116fba2b..4197d5decb42 100644
--- a/drivers/gpu/drm/msm/msm_fbdev.c
+++ b/drivers/gpu/drm/msm/msm_fbdev.c
@@ -173,9 +173,9 @@ static void msm_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int msm_fbdev_client_restore(struct drm_client_dev *client)
+static int msm_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/nouveau/nouveau_bo.c b/drivers/gpu/drm/nouveau/nouveau_bo.c
index fc433f39c127..6006c6d6a899 100644
--- a/drivers/gpu/drm/nouveau/nouveau_bo.c
+++ b/drivers/gpu/drm/nouveau/nouveau_bo.c
@@ -501,8 +501,9 @@ int nouveau_bo_pin_locked(struct nouveau_bo *nvbo, uint32_t domain, bool contig)
"0x%08x vs 0x%08x\n", bo,
bo->resource->mem_type, domain);
ret = -EBUSY;
+ } else {
+ ttm_bo_pin(&nvbo->bo);
}
- ttm_bo_pin(&nvbo->bo);
goto out;
}
diff --git a/drivers/gpu/drm/nouveau/nouveau_connector.c b/drivers/gpu/drm/nouveau/nouveau_connector.c
index 2764de3495e4..ea666d4f668d 100644
--- a/drivers/gpu/drm/nouveau/nouveau_connector.c
+++ b/drivers/gpu/drm/nouveau/nouveau_connector.c
@@ -601,8 +601,11 @@ nouveau_connector_detect(struct drm_connector *connector, bool force)
new_edid = drm_get_edid(connector, nv_encoder->i2c);
} else {
ret = nvif_outp_edid_get(&nv_encoder->outp, (u8 **)&new_edid);
- if (ret < 0)
+ if (ret < 0) {
+ pm_runtime_mark_last_busy(dev->dev);
+ pm_runtime_put_autosuspend(dev->dev);
return connector_status_disconnected;
+ }
}
nouveau_connector_set_edid(nv_connector, new_edid);
diff --git a/drivers/gpu/drm/nouveau/nouveau_drm.c b/drivers/gpu/drm/nouveau/nouveau_drm.c
index 2216db0aa9d5..904e236ed510 100644
--- a/drivers/gpu/drm/nouveau/nouveau_drm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_drm.c
@@ -563,6 +563,7 @@ nouveau_drm_device_fini(struct nouveau_drm *drm)
if (nouveau_pmops_runtime()) {
pm_runtime_get_sync(dev->dev);
pm_runtime_forbid(dev->dev);
+ pm_runtime_dont_use_autosuspend(dev->dev);
}
nouveau_led_fini(dev);
@@ -1191,10 +1192,8 @@ nouveau_drm_open(struct drm_device *dev, struct drm_file *fpriv)
mutex_unlock(&drm->clients_lock);
done:
- if (ret && cli) {
- nouveau_cli_fini(cli);
+ if (ret && cli)
kfree(cli);
- }
pm_runtime_mark_last_busy(dev->dev);
pm_runtime_put_autosuspend(dev->dev);
diff --git a/drivers/gpu/drm/nouveau/nouveau_gem.c b/drivers/gpu/drm/nouveau/nouveau_gem.c
index cf382aa195ec..49ccb6250d78 100644
--- a/drivers/gpu/drm/nouveau/nouveau_gem.c
+++ b/drivers/gpu/drm/nouveau/nouveau_gem.c
@@ -522,6 +522,7 @@ validate_init(struct nouveau_channel *chan, struct drm_file *file_priv,
if (unlikely(ret)) {
if (ret != -ERESTARTSYS)
NV_PRINTK(err, cli, "fail reserve\n");
+ drm_gem_object_put(gem);
break;
}
}
@@ -531,6 +532,7 @@ validate_init(struct nouveau_channel *chan, struct drm_file *file_priv,
struct nouveau_vma *vma = nouveau_vma_find(nvbo, vmm);
if (!vma) {
NV_PRINTK(err, cli, "vma not found!\n");
+ drm_gem_object_put(gem);
ret = -EINVAL;
break;
}
diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c
index 390c2cc21c2f..23f60b7d0e4f 100644
--- a/drivers/gpu/drm/nouveau/nouveau_sched.c
+++ b/drivers/gpu/drm/nouveau/nouveau_sched.c
@@ -512,7 +512,7 @@ nouveau_sched_destroy(struct nouveau_sched **psched)
struct nouveau_sched *sched = *psched;
nouveau_sched_fini(sched);
- kfree(sched);
+ kfree_rcu(sched, rcu);
*psched = NULL;
}
diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.h b/drivers/gpu/drm/nouveau/nouveau_sched.h
index 20cd1da8db73..51ce8dcf6285 100644
--- a/drivers/gpu/drm/nouveau/nouveau_sched.h
+++ b/drivers/gpu/drm/nouveau/nouveau_sched.h
@@ -98,6 +98,7 @@ void nouveau_job_free(struct nouveau_job *job);
struct nouveau_sched {
struct drm_gpu_scheduler base;
+ struct rcu_head rcu;
struct drm_sched_entity entity;
struct workqueue_struct *wq;
struct mutex mutex;
diff --git a/drivers/gpu/drm/nouveau/nouveau_uvmm.c b/drivers/gpu/drm/nouveau/nouveau_uvmm.c
index a68b7f0b013b..690375fbf91b 100644
--- a/drivers/gpu/drm/nouveau/nouveau_uvmm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_uvmm.c
@@ -780,6 +780,9 @@ op_map(struct nouveau_uvma *uvma)
{
struct nouveau_bo *nvbo = nouveau_gem_object(uvma->va.gem.obj);
+ if (drm_gpuva_invalidated(&uvma->va))
+ return;
+
nouveau_uvma_map(uvma, nouveau_mem(nvbo->bo.resource));
}
@@ -1164,6 +1167,7 @@ bind_lock_validate(struct nouveau_job *job, struct drm_exec *exec,
drm_gpuva_for_each_op(va_op, op->ops) {
struct drm_gem_object *obj = op_gem_obj(va_op);
+ struct nouveau_bo *nvbo;
if (unlikely(!obj))
continue;
@@ -1178,8 +1182,13 @@ bind_lock_validate(struct nouveau_job *job, struct drm_exec *exec,
if (va_op->op == DRM_GPUVA_OP_UNMAP)
continue;
- ret = nouveau_bo_validate(nouveau_gem_object(obj),
- true, false);
+ nvbo = nouveau_gem_object(obj);
+ if (!(nvbo->valid_domains &
+ (NOUVEAU_GEM_DOMAIN_VRAM | NOUVEAU_GEM_DOMAIN_GART)))
+ return -EINVAL;
+
+ nouveau_bo_placement_set(nvbo, nvbo->valid_domains, 0);
+ ret = nouveau_bo_validate(nvbo, true, false);
if (ret)
return ret;
}
diff --git a/drivers/gpu/drm/nouveau/nvif/vmm.c b/drivers/gpu/drm/nouveau/nvif/vmm.c
index 07c1ebc2a941..d9671807e722 100644
--- a/drivers/gpu/drm/nouveau/nvif/vmm.c
+++ b/drivers/gpu/drm/nouveau/nvif/vmm.c
@@ -192,6 +192,7 @@ void
nvif_vmm_dtor(struct nvif_vmm *vmm)
{
kfree(vmm->page);
+ vmm->page = NULL;
nvif_object_dtor(&vmm->object);
}
diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/disp/gv100.c b/drivers/gpu/drm/nouveau/nvkm/engine/disp/gv100.c
index cfa3698d3a2f..0aee9967d1d3 100644
--- a/drivers/gpu/drm/nouveau/nvkm/engine/disp/gv100.c
+++ b/drivers/gpu/drm/nouveau/nvkm/engine/disp/gv100.c
@@ -253,7 +253,7 @@ gv100_head_vblank_get(struct nvkm_head *head)
nvkm_mask(device, 0x611d80 + (head->id * 4), 0x00000004, 0x00000004);
}
-static void
+void
gv100_head_rgpos(struct nvkm_head *head, u16 *hline, u16 *vline)
{
struct nvkm_device *device = head->disp->engine.subdev.device;
@@ -263,7 +263,7 @@ gv100_head_rgpos(struct nvkm_head *head, u16 *hline, u16 *vline)
*hline = nvkm_rd32(device, 0x616334 + hoff) & 0x0000ffff;
}
-static void
+void
gv100_head_state(struct nvkm_head *head, struct nvkm_head_state *state)
{
struct nvkm_device *device = head->disp->engine.subdev.device;
diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/disp/head.h b/drivers/gpu/drm/nouveau/nvkm/engine/disp/head.h
index 856252bf559a..b642729c254f 100644
--- a/drivers/gpu/drm/nouveau/nvkm/engine/disp/head.h
+++ b/drivers/gpu/drm/nouveau/nvkm/engine/disp/head.h
@@ -53,6 +53,8 @@ void gf119_head_rgclk(struct nvkm_head *, int);
int gv100_head_cnt(struct nvkm_disp *, unsigned long *);
int gv100_head_new(struct nvkm_disp *, int id);
+void gv100_head_state(struct nvkm_head *head, struct nvkm_head_state *state);
+void gv100_head_rgpos(struct nvkm_head *head, u16 *hline, u16 *vline);
#define HEAD_MSG(h,l,f,a...) do { \
struct nvkm_head *_h = (h); \
diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/disp/r535.c b/drivers/gpu/drm/nouveau/nvkm/engine/disp/r535.c
index 99110ab2f44d..f80f9c0f85a7 100644
--- a/drivers/gpu/drm/nouveau/nvkm/engine/disp/r535.c
+++ b/drivers/gpu/drm/nouveau/nvkm/engine/disp/r535.c
@@ -623,14 +623,10 @@ r535_head_vblank_get(struct nvkm_head *head)
nvkm_mask(device, 0x611d80 + (head->id * 4), 0x00000002, 0x00000002);
}
-static void
-r535_head_state(struct nvkm_head *head, struct nvkm_head_state *state)
-{
-}
-
static const struct nvkm_head_func
r535_head = {
- .state = r535_head_state,
+ .state = gv100_head_state,
+ .rgpos = gv100_head_rgpos,
.vblank_get = r535_head_vblank_get,
.vblank_put = r535_head_vblank_put,
};
diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/disp/uoutp.c b/drivers/gpu/drm/nouveau/nvkm/engine/disp/uoutp.c
index 377d0e0cef84..9887b3898505 100644
--- a/drivers/gpu/drm/nouveau/nvkm/engine/disp/uoutp.c
+++ b/drivers/gpu/drm/nouveau/nvkm/engine/disp/uoutp.c
@@ -253,8 +253,7 @@ nvkm_uoutp_mthd_hdmi(struct nvkm_outp *outp, void *argv, u32 argc)
if (!ior->func->hdmi ||
args->v0.max_ac_packet > 0x1f ||
- args->v0.rekey > 0x7f ||
- (args->v0.scdc && !ior->func->hdmi->scdc))
+ args->v0.rekey > 0x7f)
return -EINVAL;
if (!args->v0.enable) {
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/clk/base.c b/drivers/gpu/drm/nouveau/nvkm/subdev/clk/base.c
index 178dc56909c2..e6539931a759 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/clk/base.c
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/clk/base.c
@@ -199,16 +199,18 @@ nvkm_cstate_prog(struct nvkm_clk *clk, struct nvkm_pstate *pstate, int cstatei)
}
if (volt) {
- ret = nvkm_volt_set_id(volt, cstate->voltage,
- pstate->base.voltage, clk->temp, -1);
- if (ret && ret != -ENODEV)
- nvkm_error(subdev, "failed to lower voltage: %d\n", ret);
+ int err = nvkm_volt_set_id(volt, cstate->voltage,
+ pstate->base.voltage, clk->temp, -1);
+
+ if (err && err != -ENODEV)
+ nvkm_error(subdev, "failed to lower voltage: %d\n", err);
}
if (therm) {
- ret = nvkm_therm_cstate(therm, pstate->fanspeed, -1);
- if (ret && ret != -ENODEV)
- nvkm_error(subdev, "failed to lower fan speed: %d\n", ret);
+ int err = nvkm_therm_cstate(therm, pstate->fanspeed, -1);
+
+ if (err && err != -ENODEV)
+ nvkm_error(subdev, "failed to lower fan speed: %d\n", err);
}
return ret;
@@ -473,6 +475,7 @@ static int
nvkm_clk_ustate_update(struct nvkm_clk *clk, int req)
{
struct nvkm_pstate *pstate;
+ bool found = false;
int i = 0;
if (!clk->allow_reclock)
@@ -480,12 +483,14 @@ nvkm_clk_ustate_update(struct nvkm_clk *clk, int req)
if (req != -1 && req != -2) {
list_for_each_entry(pstate, &clk->states, head) {
- if (pstate->pstate == req)
+ if (pstate->pstate == req) {
+ found = true;
break;
+ }
i++;
}
- if (pstate->pstate != req)
+ if (!found)
return -EINVAL;
req = i;
}
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv1a.c b/drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv1a.c
index 18241c6ba5fa..4d52a158f320 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv1a.c
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/fb/ramnv1a.c
@@ -51,6 +51,8 @@ nv1a_ram_new(struct nvkm_fb *fb, struct nvkm_ram **pram)
mib = ((mem >> 4) & 127) + 1;
}
+ pci_dev_put(bridge);
+
return nvkm_ram_new_(&nv04_ram_func, fb, NVKM_RAM_TYPE_STOLEN,
mib * 1024 * 1024, pram);
}
diff --git a/drivers/gpu/drm/omapdrm/omap_fbdev.c b/drivers/gpu/drm/omapdrm/omap_fbdev.c
index 483dc30d7840..e544c39ad996 100644
--- a/drivers/gpu/drm/omapdrm/omap_fbdev.c
+++ b/drivers/gpu/drm/omapdrm/omap_fbdev.c
@@ -289,9 +289,9 @@ static void omap_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int omap_fbdev_client_restore(struct drm_client_dev *client)
+static int omap_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/radeon/radeon_fbdev.c b/drivers/gpu/drm/radeon/radeon_fbdev.c
index 0a05b373bd9e..b118c99e78c8 100644
--- a/drivers/gpu/drm/radeon/radeon_fbdev.c
+++ b/drivers/gpu/drm/radeon/radeon_fbdev.c
@@ -300,9 +300,9 @@ static void radeon_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int radeon_fbdev_client_restore(struct drm_client_dev *client)
+static int radeon_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
vga_switcheroo_process_delayed_switch();
return 0;
diff --git a/drivers/gpu/drm/tegra/fbdev.c b/drivers/gpu/drm/tegra/fbdev.c
index 08cd471e5aa2..806166d2f885 100644
--- a/drivers/gpu/drm/tegra/fbdev.c
+++ b/drivers/gpu/drm/tegra/fbdev.c
@@ -154,9 +154,9 @@ static void tegra_fbdev_client_unregister(struct drm_client_dev *client)
}
}
-static int tegra_fbdev_client_restore(struct drm_client_dev *client)
+static int tegra_fbdev_client_restore(struct drm_client_dev *client, bool force)
{
- drm_fb_helper_lastclose(client->dev);
+ drm_fb_helper_restore_fbdev_mode_unlocked(client->dev->fb_helper, force);
return 0;
}
diff --git a/drivers/gpu/drm/tiny/ofdrm.c b/drivers/gpu/drm/tiny/ofdrm.c
index cc90e0035d03..52bb6bdb395e 100644
--- a/drivers/gpu/drm/tiny/ofdrm.c
+++ b/drivers/gpu/drm/tiny/ofdrm.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
#include <linux/of_address.h>
+#include <linux/overflow.h>
#include <linux/pci.h>
#include <linux/platform_device.h>
@@ -231,7 +232,7 @@ static bool is_avivo(u32 vendor, u32 device)
/* This will match most R5xx */
return (vendor == PCI_VENDOR_ID_ATI) &&
((device >= PCI_VENDOR_ID_ATI_R520 && device < 0x7800) ||
- (PCI_VENDOR_ID_ATI_R600 >= 0x9400));
+ (device >= PCI_VENDOR_ID_ATI_R600));
}
static enum ofdrm_model display_get_model_of(struct drm_device *dev, struct device_node *of_node)
@@ -1182,7 +1183,10 @@ static struct ofdrm_device *ofdrm_device_create(struct drm_driver *drv,
return ERR_PTR(-EINVAL);
}
- fb_size = linebytes * height;
+ if (check_mul_overflow(linebytes, height, &fb_size)) {
+ drm_err(dev, "framebuffer size exceeds maximum\n");
+ return ERR_PTR(-EINVAL);
+ }
/*
* Try to figure out the address of the framebuffer. Unfortunately, Open
diff --git a/drivers/gpu/drm/tiny/simpledrm.c b/drivers/gpu/drm/tiny/simpledrm.c
index 07abaf27315f..cb38ed2ae31d 100644
--- a/drivers/gpu/drm/tiny/simpledrm.c
+++ b/drivers/gpu/drm/tiny/simpledrm.c
@@ -54,7 +54,7 @@ static int
simplefb_get_validated_int0(struct drm_device *dev, const char *name,
uint32_t value)
{
- if (!value) {
+ if (!value || value > U16_MAX) {
drm_err(dev, "simplefb: invalid framebuffer %s of %u\n",
name, value);
return -EINVAL;
@@ -212,6 +212,39 @@ simplefb_get_memory_of(struct drm_device *dev, struct device_node *of_node)
return res;
}
+static int __simplefb_get_panel_size_mm_of(struct drm_device *dev,
+ struct device_node *of_panel_node,
+ const char *name)
+{
+ int ret;
+ u32 value;
+
+ ret = of_property_read_u32(of_panel_node, name, &value);
+ if (ret) {
+ drm_dbg(dev, "simplefb: cannot parse panel %s: error %d\n",
+ name, ret);
+ return ret;
+ } else if (value > U16_MAX) {
+ drm_dbg(dev, "simplefb: panel %s of %u exceeds maximum value\n",
+ name, value);
+ return -EINVAL;
+ }
+
+ return value;
+}
+
+static int simplefb_get_panel_width_mm_of(struct drm_device *dev,
+ struct device_node *of_panel_node)
+{
+ return __simplefb_get_panel_size_mm_of(dev, of_panel_node, "width-mm");
+}
+
+static int simplefb_get_panel_height_mm_of(struct drm_device *dev,
+ struct device_node *of_panel_node)
+{
+ return __simplefb_get_panel_size_mm_of(dev, of_panel_node, "height-mm");
+}
+
/*
* Simple Framebuffer device
*/
@@ -781,7 +814,7 @@ static struct simpledrm_device *simpledrm_device_create(struct drm_driver *drv,
struct simpledrm_device *sdev;
struct drm_device *dev;
int width, height, stride;
- int width_mm = 0, height_mm = 0;
+ u16 width_mm = 0, height_mm = 0;
struct device_node *panel_node;
const struct drm_format_info *format;
struct resource *res, *mem = NULL;
@@ -844,8 +877,18 @@ static struct simpledrm_device *simpledrm_device_create(struct drm_driver *drv,
return ERR_CAST(mem);
panel_node = of_parse_phandle(of_node, "panel", 0);
if (panel_node) {
- simplefb_read_u32_of(dev, panel_node, "width-mm", &width_mm);
- simplefb_read_u32_of(dev, panel_node, "height-mm", &height_mm);
+ /*
+ * Ignore errors from parsing the physical panel
+ * size. Using the pre-initialized sizes of 0 will
+ * make drm_sysfb_mode() calculate a default physical
+ * size based on a resolution of 96 dpi.
+ */
+ ret = simplefb_get_panel_width_mm_of(dev, panel_node);
+ if (ret > 0)
+ width_mm = ret;
+ ret = simplefb_get_panel_height_mm_of(dev, panel_node);
+ if (ret > 0)
+ height_mm = ret;
of_node_put(panel_node);
}
} else {
@@ -853,9 +896,15 @@ static struct simpledrm_device *simpledrm_device_create(struct drm_driver *drv,
return ERR_PTR(-ENODEV);
}
if (!stride) {
- stride = drm_format_info_min_pitch(format, 0, width);
- if (drm_WARN_ON(dev, !stride))
+ u64 pitch = drm_format_info_min_pitch(format, 0, width);
+
+ if (drm_WARN_ON(dev, !pitch)) {
+ return ERR_PTR(-EINVAL); /* driver bug */
+ } else if (pitch > INT_MAX) {
+ drm_warn(dev, "stride of %llu exceeds maximum\n", pitch);
return ERR_PTR(-EINVAL);
+ }
+ stride = pitch;
}
/*
diff --git a/drivers/gpu/drm/vc4/vc4_kms.c b/drivers/gpu/drm/vc4/vc4_kms.c
index bddfcad10950..c5e28ff3931a 100644
--- a/drivers/gpu/drm/vc4/vc4_kms.c
+++ b/drivers/gpu/drm/vc4/vc4_kms.c
@@ -1084,7 +1084,7 @@ int vc4_kms_load(struct drm_device *dev)
drm_mode_config_reset(dev);
- drm_kms_helper_poll_init(dev);
+ drmm_kms_helper_poll_init(dev);
return 0;
}
diff --git a/drivers/gpu/drm/virtio/virtgpu_gem.c b/drivers/gpu/drm/virtio/virtgpu_gem.c
index 2e75cdeb49b3..0233a1c3181c 100644
--- a/drivers/gpu/drm/virtio/virtgpu_gem.c
+++ b/drivers/gpu/drm/virtio/virtgpu_gem.c
@@ -45,7 +45,7 @@ static int virtio_gpu_gem_create(struct drm_file *file,
ret = drm_gem_handle_create(file, &obj->base.base, &handle);
if (ret) {
- drm_gem_object_release(&obj->base.base);
+ drm_gem_object_put(&obj->base.base);
return ret;
}
diff --git a/drivers/gpu/drm/virtio/virtgpu_ioctl.c b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
index e4f76f315550..d8aec5112b32 100644
--- a/drivers/gpu/drm/virtio/virtgpu_ioctl.c
+++ b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
@@ -180,7 +180,7 @@ static int virtio_gpu_resource_create_ioctl(struct drm_device *dev, void *data,
ret = drm_gem_handle_create(file, obj, &handle);
if (ret) {
- drm_gem_object_release(obj);
+ drm_gem_object_put(obj);
return ret;
}
@@ -546,14 +546,14 @@ static int virtio_gpu_resource_create_blob_ioctl(struct drm_device *dev,
if (params.blob_flags & VIRTGPU_BLOB_FLAG_USE_CROSS_DEVICE) {
ret = virtio_gpu_resource_assign_uuid(vgdev, bo);
if (ret) {
- drm_gem_object_release(obj);
+ drm_gem_object_put(obj);
return ret;
}
}
ret = drm_gem_handle_create(file, obj, &handle);
if (ret) {
- drm_gem_object_release(obj);
+ drm_gem_object_put(obj);
return ret;
}
diff --git a/drivers/gpu/drm/virtio/virtgpu_submit.c b/drivers/gpu/drm/virtio/virtgpu_submit.c
index 409ecdb0d680..3ee876cb004b 100644
--- a/drivers/gpu/drm/virtio/virtgpu_submit.c
+++ b/drivers/gpu/drm/virtio/virtgpu_submit.c
@@ -389,10 +389,13 @@ static int virtio_gpu_init_submit(struct virtio_gpu_submit *submit,
if ((exbuf->flags & VIRTGPU_EXECBUF_FENCE_FD_OUT) ||
exbuf->num_out_syncobjs ||
exbuf->num_bo_handles ||
- drm_fence_event)
+ drm_fence_event) {
out_fence = virtio_gpu_fence_alloc(vgdev, fence_ctx, ring_idx);
- else
+ if (!out_fence)
+ return -ENOMEM;
+ } else {
out_fence = NULL;
+ }
if (drm_fence_event) {
err = virtio_gpu_fence_event_create(dev, file, out_fence, ring_idx);
@@ -538,6 +541,10 @@ int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
virtio_gpu_process_post_deps(&submit);
virtio_gpu_complete_submit(&submit);
cleanup:
+ if (ret && submit.out_fence && submit.out_fence->e) {
+ drm_event_cancel_free(dev, &submit.out_fence->e->base);
+ submit.out_fence->e = NULL;
+ }
virtio_gpu_cleanup_submit(&submit);
return ret;
diff --git a/drivers/gpu/drm/virtio/virtgpu_vram.c b/drivers/gpu/drm/virtio/virtgpu_vram.c
index 25df81c02783..47a886824cd7 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vram.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vram.c
@@ -202,16 +202,12 @@ int virtio_gpu_vram_create(struct virtio_gpu_device *vgdev,
/* Create fake offset */
ret = drm_gem_create_mmap_offset(obj);
- if (ret) {
- kfree(vram);
- return ret;
- }
+ if (ret)
+ goto err_release_obj;
ret = virtio_gpu_resource_id_get(vgdev, &vram->base.hw_res_handle);
- if (ret) {
- kfree(vram);
- return ret;
- }
+ if (ret)
+ goto err_release_obj;
virtio_gpu_cmd_resource_create_blob(vgdev, &vram->base, params, NULL,
0);
@@ -225,4 +221,9 @@ int virtio_gpu_vram_create(struct virtio_gpu_device *vgdev,
*bo_ptr = &vram->base;
return 0;
+
+err_release_obj:
+ drm_gem_object_release(obj);
+ kfree(vram);
+ return ret;
}
diff --git a/drivers/gpu/drm/xe/instructions/xe_gpu_commands.h b/drivers/gpu/drm/xe/instructions/xe_gpu_commands.h
index a255946b6f77..b4fe9aa4e234 100644
--- a/drivers/gpu/drm/xe/instructions/xe_gpu_commands.h
+++ b/drivers/gpu/drm/xe/instructions/xe_gpu_commands.h
@@ -41,6 +41,7 @@
#define GFX_OP_PIPE_CONTROL(len) ((0x3<<29)|(0x3<<27)|(0x2<<24)|((len)-2))
+#define PIPE_CONTROL0_UNTYPED_DATAPORT_CACHE_FLUSH BIT(11) /* gen12 */
#define PIPE_CONTROL0_HDC_PIPELINE_FLUSH BIT(9) /* gen12 */
#define PIPE_CONTROL_COMMAND_CACHE_INVALIDATE (1<<29)
diff --git a/drivers/gpu/drm/xe/xe_bo.h b/drivers/gpu/drm/xe/xe_bo.h
index 556ce5390009..a11f716b264e 100644
--- a/drivers/gpu/drm/xe/xe_bo.h
+++ b/drivers/gpu/drm/xe/xe_bo.h
@@ -8,6 +8,8 @@
#include <drm/ttm/ttm_tt.h>
+#include <xen/xen.h>
+
#include "xe_bo_types.h"
#include "xe_macros.h"
#include "xe_vm_types.h"
@@ -328,6 +330,23 @@ static inline unsigned int xe_sg_segment_size(struct device *dev)
struct scatterlist __maybe_unused sg;
size_t max = BIT_ULL(sizeof(sg.length) * 8) - 1;
+ /*
+ * For Xen PV guests pages aren't contiguous in DMA (machine) address
+ * space. The DMA API takes care of that both in dma_alloc_* (by
+ * calling into the hypervisor to make the pages contiguous) and in
+ * dma_map_* (by bounce buffering). But xe (like i915, see commit
+ * 78a07fe777c4) ignores the coherency aspects of the DMA API and thus
+ * can't cope with bounce buffering actually happening, so add a hack
+ * here to force small allocations and mappings when running in PV
+ * mode on Xen.
+ *
+ * Note this will still break if bounce buffering is required for other
+ * reasons, like confidential computing hypervisors or PCIe root ports
+ * with addressing limitations.
+ */
+ if (xen_pv_domain())
+ return PAGE_SIZE;
+
max = min_t(size_t, max, dma_max_mapping_size(dev));
/*
diff --git a/drivers/gpu/drm/xe/xe_ring_ops.c b/drivers/gpu/drm/xe/xe_ring_ops.c
index c9e8969f99fc..50ab1e97ca26 100644
--- a/drivers/gpu/drm/xe/xe_ring_ops.c
+++ b/drivers/gpu/drm/xe/xe_ring_ops.c
@@ -174,9 +174,21 @@ static int emit_store_imm_ppgtt_posted(u64 addr, u64 value,
static int emit_render_cache_flush(struct xe_sched_job *job, u32 *dw, int i)
{
struct xe_gt *gt = job->q->gt;
+ struct xe_device *xe = gt_to_xe(gt);
bool lacks_render = !(gt->info.engine_mask & XE_HW_ENGINE_RCS_MASK);
- u32 flags;
+ u32 flags0, flags;
+ flags0 = PIPE_CONTROL0_HDC_PIPELINE_FLUSH;
+ /*
+ * Prior to MTL, HDC Pipeline Flush reliably also flushes the LSC
+ * untyped L1 dataport cache, provided HDC_CHICKEN0 is programmed
+ * correctly. Starting with MTL that coupling no longer holds
+ * regardless of how HDC_CHICKEN0 is programmed, but explicitly
+ * requesting the flush via PIPE_CONTROL is itself only reliable
+ * from Xe2 onward, so only gate it in on Xe2+.
+ */
+ if (GRAPHICS_VERx100(xe) >= 2000)
+ flags0 |= PIPE_CONTROL0_UNTYPED_DATAPORT_CACHE_FLUSH;
flags = (PIPE_CONTROL_CS_STALL |
PIPE_CONTROL_TILE_CACHE_FLUSH |
PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
@@ -192,7 +204,7 @@ static int emit_render_cache_flush(struct xe_sched_job *job, u32 *dw, int i)
else if (job->q->class == XE_ENGINE_CLASS_COMPUTE)
flags &= ~PIPE_CONTROL_3D_ENGINE_FLAGS;
- return emit_pipe_control(dw, i, PIPE_CONTROL0_HDC_PIPELINE_FLUSH, flags, 0, 0);
+ return emit_pipe_control(dw, i, flags0, flags, 0, 0);
}
static int emit_pipe_control_to_ring_end(struct xe_hw_engine *hwe, u32 *dw, int i)
diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c
index deb4d0314db3..fa0d7c641402 100644
--- a/drivers/gpu/drm/xe/xe_vm.c
+++ b/drivers/gpu/drm/xe/xe_vm.c
@@ -1678,13 +1678,21 @@ void xe_vm_close_and_put(struct xe_vm *vm)
vma->gpuva.flags |= XE_VMA_DESTROYED;
}
+ xe_vm_unlock(vm);
+
/*
- * All vm operations will add shared fences to resv.
- * The only exception is eviction for a shared object,
- * but even so, the unbind when evicted would still
- * install a fence to resv. Hence it's safe to
- * destroy the pagetables immediately.
+ * Unlink and destroy all contested external-BO VMAs before destroying
+ * the page tables. Otherwise, concurrent eviction holding only bo->resv
+ * can walk the BO's VMAs and attempt to invalidate/zap page tables that
+ * have already been freed.
*/
+ list_for_each_entry_safe(vma, next_vma, &contested,
+ combined_links.destroy) {
+ list_del_init(&vma->combined_links.destroy);
+ xe_vma_destroy_unlocked(vma);
+ }
+
+ xe_vm_lock(vm, false);
xe_vm_free_scratch(vm);
for_each_tile(tile, xe, id) {
@@ -1695,17 +1703,6 @@ void xe_vm_close_and_put(struct xe_vm *vm)
}
xe_vm_unlock(vm);
- /*
- * VM is now dead, cannot re-add nodes to vm->vmas if it's NULL
- * Since we hold a refcount to the bo, we can remove and free
- * the members safely without locking.
- */
- list_for_each_entry_safe(vma, next_vma, &contested,
- combined_links.destroy) {
- list_del_init(&vma->combined_links.destroy);
- xe_vma_destroy_unlocked(vma);
- }
-
up_write(&vm->lock);
down_write(&xe->usm.lock);
diff --git a/drivers/hid/amd-sfh-hid/amd_sfh_common.h b/drivers/hid/amd-sfh-hid/amd_sfh_common.h
index 00308d8998d4..f05078f93b13 100644
--- a/drivers/hid/amd-sfh-hid/amd_sfh_common.h
+++ b/drivers/hid/amd-sfh-hid/amd_sfh_common.h
@@ -12,11 +12,15 @@
#include <linux/mutex.h>
#include <linux/pci.h>
+#include <linux/sizes.h>
#include "amd_sfh_hid.h"
#define PCI_DEVICE_ID_AMD_MP2 0x15E4
#define PCI_DEVICE_ID_AMD_MP2_1_1 0x164A
+/* The BAR 2 size must cover the highest register offset (0x10958) */
+#define AMD_SFH_MIN_BAR_SIZE SZ_128K
+
#define AMD_C2P_MSG(regno) (0x10500 + ((regno) * 4))
#define AMD_P2C_MSG(regno) (0x10680 + ((regno) * 4))
diff --git a/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c b/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c
index 33ba9af1a249..b29f8ea39c4c 100644
--- a/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c
+++ b/drivers/hid/amd-sfh-hid/amd_sfh_pcie.c
@@ -390,6 +390,16 @@ static int amd_mp2_pci_probe(struct pci_dev *pdev, const struct pci_device_id *i
if (rc)
return rc;
+ if (!(pci_resource_flags(pdev, 2) & IORESOURCE_MEM)) {
+ dev_err(&pdev->dev, "BAR 2 is not IORESOURCE_MEM\n");
+ return -ENODEV;
+ }
+
+ if (pci_resource_len(pdev, 2) < AMD_SFH_MIN_BAR_SIZE) {
+ dev_err(&pdev->dev, "BAR 2 is too small\n");
+ return -EINVAL;
+ }
+
rc = pcim_iomap_regions(pdev, BIT(2), DRIVER_NAME);
if (rc)
return rc;
diff --git a/drivers/hid/bpf/hid_bpf_dispatch.c b/drivers/hid/bpf/hid_bpf_dispatch.c
index 1de6a7e0af7b..a421fecce9dd 100644
--- a/drivers/hid/bpf/hid_bpf_dispatch.c
+++ b/drivers/hid/bpf/hid_bpf_dispatch.c
@@ -359,7 +359,7 @@ hid_bpf_release_context(struct hid_bpf_ctx *ctx)
static int
__hid_bpf_hw_check_params(struct hid_bpf_ctx *ctx, __u8 *buf, size_t *buf__sz,
- enum hid_report_type rtype)
+ enum hid_report_type rtype, bool hw_request)
{
struct hid_report_enum *report_enum;
struct hid_report *report;
@@ -391,6 +391,10 @@ __hid_bpf_hw_check_params(struct hid_bpf_ctx *ctx, __u8 *buf, size_t *buf__sz,
report_len = hid_report_len(report);
+ /* unnumbered reports need to have a report ID reserved in the first byte */
+ if (hw_request && report_enum->numbered == 0)
+ report_len += 1;
+
if (*buf__sz > report_len)
*buf__sz = report_len;
@@ -424,7 +428,7 @@ hid_bpf_hw_request(struct hid_bpf_ctx *ctx, __u8 *buf, size_t buf__sz,
return -EDEADLOCK;
/* check arguments */
- ret = __hid_bpf_hw_check_params(ctx, buf, &size, rtype);
+ ret = __hid_bpf_hw_check_params(ctx, buf, &size, rtype, true);
if (ret)
return ret;
@@ -487,7 +491,7 @@ hid_bpf_hw_output_report(struct hid_bpf_ctx *ctx, __u8 *buf, size_t buf__sz)
return -EDEADLOCK;
/* check arguments */
- ret = __hid_bpf_hw_check_params(ctx, buf, &size, HID_OUTPUT_REPORT);
+ ret = __hid_bpf_hw_check_params(ctx, buf, &size, HID_OUTPUT_REPORT, true);
if (ret)
return ret;
@@ -515,7 +519,7 @@ __hid_bpf_input_report(struct hid_bpf_ctx *ctx, enum hid_report_type type, u8 *b
return -EDEADLOCK;
/* check arguments */
- ret = __hid_bpf_hw_check_params(ctx, buf, &size, type);
+ ret = __hid_bpf_hw_check_params(ctx, buf, &size, type, false);
if (ret)
return ret;
diff --git a/drivers/hid/hid-alps.c b/drivers/hid/hid-alps.c
index 8c3f83532ce9..0556cb5645eb 100644
--- a/drivers/hid/hid-alps.c
+++ b/drivers/hid/hid-alps.c
@@ -407,6 +407,8 @@ static int u1_raw_event(struct alps_dev *hdata, u8 *data, int size)
return 1;
case U1_SP_ABSOLUTE_REPORT_ID:
+ if (!hdata->input2)
+ return 0;
sp_x = get_unaligned_le16(data+2);
sp_y = get_unaligned_le16(data+4);
@@ -738,7 +740,6 @@ static int alps_input_configured(struct hid_device *hdev, struct hid_input *hi)
goto exit;
}
- data->input2 = input2;
input2->phys = input->phys;
input2->name = "DualPoint Stick";
input2->id.bustype = BUS_I2C;
@@ -762,11 +763,12 @@ static int alps_input_configured(struct hid_device *hdev, struct hid_input *hi)
__set_bit(INPUT_PROP_POINTER, input2->propbit);
__set_bit(INPUT_PROP_POINTING_STICK, input2->propbit);
- if (input_register_device(data->input2)) {
+ if (input_register_device(input2)) {
input_free_device(input2);
ret = -ENOENT;
goto exit;
}
+ data->input2 = input2;
}
exit:
@@ -823,6 +825,24 @@ static int alps_probe(struct hid_device *hdev, const struct hid_device_id *id)
return 0;
}
+static void alps_remove(struct hid_device *hdev)
+{
+ struct alps_dev *data = hid_get_drvdata(hdev);
+
+ /*
+ * input2 ("DualPoint Stick") is allocated separately and is not
+ * tracked in hdev->inputs, so the default remove path
+ * (hid_hw_stop -> hidinput_disconnect) does not unregister it.
+ *
+ * Stop the device first so that no URB callback can touch input2
+ * while it is being unregistered, then drop it explicitly.
+ */
+ hid_hw_stop(hdev);
+
+ if (data->input2)
+ input_unregister_device(data->input2);
+}
+
static const struct hid_device_id alps_id[] = {
{ HID_DEVICE(HID_BUS_ANY, HID_GROUP_ANY,
USB_VENDOR_ID_ALPS_JP, HID_DEVICE_ID_ALPS_U1_DUAL) },
@@ -843,10 +863,9 @@ static struct hid_driver alps_driver = {
.raw_event = alps_raw_event,
.input_mapping = alps_input_mapping,
.input_configured = alps_input_configured,
-#ifdef CONFIG_PM
- .resume = alps_post_resume,
- .reset_resume = alps_post_reset,
-#endif
+ .resume = pm_ptr(alps_post_resume),
+ .reset_resume = pm_ptr(alps_post_reset),
+ .remove = alps_remove,
};
module_hid_driver(alps_driver);
diff --git a/drivers/hid/hid-asus.c b/drivers/hid/hid-asus.c
index 28426337a2f1..1708396a0450 100644
--- a/drivers/hid/hid-asus.c
+++ b/drivers/hid/hid-asus.c
@@ -48,7 +48,7 @@ MODULE_DESCRIPTION("Asus HID Keyboard and TouchPad");
#define FEATURE_REPORT_ID 0x0d
#define INPUT_REPORT_ID 0x5d
#define FEATURE_KBD_REPORT_ID 0x5a
-#define FEATURE_KBD_REPORT_SIZE 16
+#define FEATURE_KBD_REPORT_SIZE 64
#define FEATURE_KBD_LED_REPORT_ID1 0x5d
#define FEATURE_KBD_LED_REPORT_ID2 0x5e
@@ -393,15 +393,41 @@ static int asus_kbd_set_report(struct hid_device *hdev, const u8 *buf, size_t bu
static int asus_kbd_init(struct hid_device *hdev, u8 report_id)
{
+ /*
+ * The handshake is first sent as a set_report, then retrieved
+ * from a get_report. They should be equal.
+ */
const u8 buf[] = { report_id, 0x41, 0x53, 0x55, 0x53, 0x20, 0x54,
0x65, 0x63, 0x68, 0x2e, 0x49, 0x6e, 0x63, 0x2e, 0x00 };
int ret;
ret = asus_kbd_set_report(hdev, buf, sizeof(buf));
- if (ret < 0)
- hid_err(hdev, "Asus failed to send init command: %d\n", ret);
+ if (ret < 0) {
+ hid_err(hdev, "Asus handshake %02x failed to send: %d\n",
+ report_id, ret);
+ return ret;
+ }
- return ret;
+ u8 *readbuf __free(kfree) = kzalloc(FEATURE_KBD_REPORT_SIZE, GFP_KERNEL);
+ if (!readbuf)
+ return -ENOMEM;
+
+ ret = hid_hw_raw_request(hdev, report_id, readbuf,
+ FEATURE_KBD_REPORT_SIZE, HID_FEATURE_REPORT,
+ HID_REQ_GET_REPORT);
+ if (ret < 0) {
+ hid_warn(hdev, "Asus handshake %02x failed to receive ack: %d\n",
+ report_id, ret);
+ } else if (memcmp(readbuf, buf, sizeof(buf)) != 0) {
+ hid_warn(hdev, "Asus handshake %02x returned invalid response: %*ph\n",
+ report_id, FEATURE_KBD_REPORT_SIZE, readbuf);
+ }
+
+ /*
+ * Do not return error if handshake is wrong until this is
+ * verified to work for all devices.
+ */
+ return 0;
}
static int asus_kbd_get_functions(struct hid_device *hdev,
diff --git a/drivers/hid/hid-ids.h b/drivers/hid/hid-ids.h
index 3453b5dbf589..643e3d7a041d 100644
--- a/drivers/hid/hid-ids.h
+++ b/drivers/hid/hid-ids.h
@@ -1218,6 +1218,9 @@
#define USB_DEVICE_ID_SAMSUNG_WIRELESS_UNIVERSAL_KBD 0xa006
#define USB_DEVICE_ID_SAMSUNG_WIRELESS_MULTI_HOGP_KBD 0xa064
+#define USB_VENDOR_ID_SDINNOVATION 0x36ae
+#define USB_DEVICE_ID_SDINNOVATION_GAMING_KBD 0xfeab
+
#define USB_VENDOR_ID_SEMICO 0x1a2c
#define USB_DEVICE_ID_SEMICO_USB_KEYKOARD 0x0023
#define USB_DEVICE_ID_SEMICO_USB_KEYKOARD2 0x0027
diff --git a/drivers/hid/hid-logitech-hidpp.c b/drivers/hid/hid-logitech-hidpp.c
index 676caeb1e21e..9926ab6e808e 100644
--- a/drivers/hid/hid-logitech-hidpp.c
+++ b/drivers/hid/hid-logitech-hidpp.c
@@ -305,21 +305,22 @@ static int __do_hidpp_send_message_sync(struct hidpp_device *hidpp,
if (ret) {
dbg_hid("__hidpp_send_report returned err: %d\n", ret);
memset(response, 0, sizeof(struct hidpp_report));
- return ret;
+ goto out;
}
if (!wait_event_timeout(hidpp->wait, hidpp->answer_available,
5*HZ)) {
dbg_hid("%s:timeout waiting for response\n", __func__);
memset(response, 0, sizeof(struct hidpp_report));
- return -ETIMEDOUT;
+ ret = -ETIMEDOUT;
+ goto out;
}
if (response->report_id == REPORT_ID_HIDPP_SHORT &&
response->rap.sub_id == HIDPP_ERROR) {
ret = response->rap.params[1];
dbg_hid("%s:got hidpp error %02X\n", __func__, ret);
- return ret;
+ goto out;
}
if ((response->report_id == REPORT_ID_HIDPP_LONG ||
@@ -327,10 +328,14 @@ static int __do_hidpp_send_message_sync(struct hidpp_device *hidpp,
response->fap.feature_index == HIDPP20_ERROR) {
ret = response->fap.params[1];
dbg_hid("%s:got hidpp 2.0 error %02X\n", __func__, ret);
- return ret;
+ goto out;
}
- return 0;
+ ret = 0;
+
+out:
+ hidpp->send_receive_buf = NULL;
+ return ret;
}
/*
@@ -3866,8 +3871,7 @@ static int hidpp_input_configured(struct hid_device *hdev,
static int hidpp_raw_hidpp_event(struct hidpp_device *hidpp, u8 *data,
int size)
{
- struct hidpp_report *question = hidpp->send_receive_buf;
- struct hidpp_report *answer = hidpp->send_receive_buf;
+ struct hidpp_report *question, *answer;
struct hidpp_report *report = (struct hidpp_report *)data;
int ret;
int last_online;
@@ -3877,6 +3881,12 @@ static int hidpp_raw_hidpp_event(struct hidpp_device *hidpp, u8 *data,
* previously sent command.
*/
if (unlikely(mutex_is_locked(&hidpp->send_mutex))) {
+ question = hidpp->send_receive_buf;
+ answer = hidpp->send_receive_buf;
+
+ if (!question)
+ return 0;
+
/*
* Check for a correct hidpp20 answer or the corresponding
* error
diff --git a/drivers/hid/hid-quirks.c b/drivers/hid/hid-quirks.c
index 39d81777cb7e..b95521c50d25 100644
--- a/drivers/hid/hid-quirks.c
+++ b/drivers/hid/hid-quirks.c
@@ -182,6 +182,7 @@ static const struct hid_device_id hid_quirks[] = {
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_X52_2), HID_QUIRK_INCREMENT_USAGE_ON_DUPLICATE },
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_X52_PRO), HID_QUIRK_INCREMENT_USAGE_ON_DUPLICATE },
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_X65), HID_QUIRK_INCREMENT_USAGE_ON_DUPLICATE },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SDINNOVATION, USB_DEVICE_ID_SDINNOVATION_GAMING_KBD), HID_QUIRK_ALWAYS_POLL },
{ HID_USB_DEVICE(USB_VENDOR_ID_SEMICO, USB_DEVICE_ID_SEMICO_USB_KEYKOARD2), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_SEMICO, USB_DEVICE_ID_SEMICO_USB_KEYKOARD), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_SENNHEISER, USB_DEVICE_ID_SENNHEISER_BTD500USB), HID_QUIRK_NOGET },
@@ -416,7 +417,7 @@ static const struct hid_device_id hid_have_special_driver[] = {
#endif
#if IS_ENABLED(CONFIG_HID_ELECOM)
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_ELECOM, USB_DEVICE_ID_ELECOM_BM084) },
- { HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_ELECOM, USB_DEVICE_ID_ELECOM_M_XGL20DLBK) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ELECOM, USB_DEVICE_ID_ELECOM_M_XGL20DLBK) },
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_ELECOM, USB_DEVICE_ID_ELECOM_M_HT1MRBK_01AC) },
{ HID_USB_DEVICE(USB_VENDOR_ID_ELECOM, USB_DEVICE_ID_ELECOM_M_XT3URBK_00FB) },
{ HID_USB_DEVICE(USB_VENDOR_ID_ELECOM, USB_DEVICE_ID_ELECOM_M_XT3URBK_018F) },
diff --git a/drivers/hid/wacom_sys.c b/drivers/hid/wacom_sys.c
index 8a7d24fb30b4..b8f2624f631c 100644
--- a/drivers/hid/wacom_sys.c
+++ b/drivers/hid/wacom_sys.c
@@ -97,8 +97,9 @@ static int wacom_wac_pen_serial_enforce(struct hid_device *hdev,
/* Queue events which have invalid tool type or serial number */
for (i = 0; i < report->maxfield; i++) {
- for (j = 0; j < report->field[i]->maxusage; j++) {
- struct hid_field *field = report->field[i];
+ struct hid_field *field = report->field[i];
+
+ for (j = 0; j < field->report_count; j++) {
struct hid_usage *usage = &field->usage[j];
unsigned int equivalent_usage = wacom_equivalent_usage(usage->hid);
unsigned int offset;
diff --git a/drivers/hwmon/emc1403.c b/drivers/hwmon/emc1403.c
index 39f69adb88c2..eca33220d34a 100644
--- a/drivers/hwmon/emc1403.c
+++ b/drivers/hwmon/emc1403.c
@@ -17,6 +17,7 @@
#include <linux/hwmon-sysfs.h>
#include <linux/err.h>
#include <linux/sysfs.h>
+#include <linux/mutex.h>
#include <linux/regmap.h>
#include <linux/util_macros.h>
@@ -29,6 +30,7 @@ enum emc1403_chip { emc1402, emc1403, emc1404, emc1428 };
struct thermal_data {
enum emc1403_chip chip;
struct regmap *regmap;
+ struct mutex mutex;
};
static ssize_t power_state_show(struct device *dev, struct device_attribute *attr, char *buf)
@@ -266,8 +268,8 @@ static s8 emc1403_temp_regs_low[][4] = {
},
};
-static int emc1403_get_temp(struct thermal_data *data, int channel,
- enum emc1403_reg_map map, long *val)
+static int __emc1403_get_temp(struct thermal_data *data, int channel,
+ enum emc1403_reg_map map, long *val)
{
unsigned int regvalh;
unsigned int regvall = 0;
@@ -293,22 +295,38 @@ static int emc1403_get_temp(struct thermal_data *data, int channel,
return 0;
}
+static int emc1403_get_temp(struct thermal_data *data, int channel,
+ enum emc1403_reg_map map, long *val)
+{
+ int ret;
+
+ mutex_lock(&data->mutex);
+ ret = __emc1403_get_temp(data, channel, map, val);
+ mutex_unlock(&data->mutex);
+
+ return ret;
+}
+
static int emc1403_get_hyst(struct thermal_data *data, int channel,
enum emc1403_reg_map map, long *val)
{
int hyst, ret;
long limit;
- ret = emc1403_get_temp(data, channel, map, &limit);
+ mutex_lock(&data->mutex);
+ ret = __emc1403_get_temp(data, channel, map, &limit);
if (ret < 0)
- return ret;
+ goto unlock;
ret = regmap_read(data->regmap, 0x21, &hyst);
if (ret < 0)
- return ret;
-
- *val = limit - hyst * 1000;
-
- return 0;
+ goto unlock;
+ if (map == temp_min)
+ *val = limit + hyst * 1000;
+ else
+ *val = limit - hyst * 1000;
+unlock:
+ mutex_unlock(&data->mutex);
+ return ret;
}
static int emc1403_temp_read(struct thermal_data *data, u32 attr, int channel, long *val)
@@ -323,6 +341,9 @@ static int emc1403_temp_read(struct thermal_data *data, u32 attr, int channel, l
case hwmon_temp_input:
ret = emc1403_get_temp(data, channel, ema1403_temp_map[attr], val);
break;
+ case hwmon_temp_min_hyst:
+ ret = emc1403_get_hyst(data, channel, temp_min, val);
+ break;
case hwmon_temp_max_hyst:
ret = emc1403_get_hyst(data, channel, temp_max, val);
break;
@@ -430,16 +451,20 @@ static int emc1403_set_hyst(struct thermal_data *data, long val)
else
val = clamp_val(val, 0, 255000);
- ret = emc1403_get_temp(data, 0, temp_crit, &limit);
+ mutex_lock(&data->mutex);
+ ret = __emc1403_get_temp(data, 0, temp_crit, &limit);
if (ret < 0)
- return ret;
+ goto unlock;
hyst = limit - val;
if (data->chip == emc1428)
hyst = clamp_val(DIV_ROUND_CLOSEST(hyst, 1000), 0, 127);
else
hyst = clamp_val(DIV_ROUND_CLOSEST(hyst, 1000), 0, 255);
- return regmap_write(data->regmap, 0x21, hyst);
+ ret = regmap_write(data->regmap, 0x21, hyst);
+unlock:
+ mutex_unlock(&data->mutex);
+ return ret;
}
static int emc1403_set_temp(struct thermal_data *data, int channel,
@@ -453,6 +478,7 @@ static int emc1403_set_temp(struct thermal_data *data, int channel,
regh = emc1403_temp_regs[channel][map];
regl = emc1403_temp_regs_low[channel][map];
+ mutex_lock(&data->mutex);
if (regl >= 0) {
if (data->chip == emc1428)
val = clamp_val(val, -128000, 127875);
@@ -461,7 +487,7 @@ static int emc1403_set_temp(struct thermal_data *data, int channel,
regval = DIV_ROUND_CLOSEST(val, 125);
ret = regmap_write(data->regmap, regh, (regval >> 3) & 0xff);
if (ret < 0)
- return ret;
+ goto unlock;
ret = regmap_write(data->regmap, regl, (regval & 0x07) << 5);
} else {
if (data->chip == emc1428)
@@ -471,6 +497,8 @@ static int emc1403_set_temp(struct thermal_data *data, int channel,
regval = DIV_ROUND_CLOSEST(val, 1000);
ret = regmap_write(data->regmap, regh, regval);
}
+unlock:
+ mutex_unlock(&data->mutex);
return ret;
}
@@ -544,6 +572,7 @@ static umode_t emc1403_temp_is_visible(const void *_data, u32 attr, int channel)
case hwmon_temp_max_alarm:
case hwmon_temp_crit_alarm:
case hwmon_temp_fault:
+ case hwmon_temp_min_hyst:
case hwmon_temp_max_hyst:
return 0444;
case hwmon_temp_min:
@@ -586,35 +615,35 @@ static const struct hwmon_channel_info * const emc1403_info[] = {
HWMON_CHANNEL_INFO(chip, HWMON_C_UPDATE_INTERVAL),
HWMON_CHANNEL_INFO(temp,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT,
HWMON_T_INPUT | HWMON_T_MIN | HWMON_T_MAX |
- HWMON_T_CRIT | HWMON_T_MAX_HYST |
+ HWMON_T_CRIT | HWMON_T_MIN_HYST | HWMON_T_MAX_HYST |
HWMON_T_CRIT_HYST | HWMON_T_MIN_ALARM |
HWMON_T_MAX_ALARM | HWMON_T_CRIT_ALARM | HWMON_T_FAULT
),
@@ -666,6 +695,8 @@ static int emc1403_probe(struct i2c_client *client)
if (IS_ERR(data->regmap))
return PTR_ERR(data->regmap);
+ mutex_init(&data->mutex);
+
hwmon_dev = devm_hwmon_device_register_with_info(&client->dev,
client->name, data,
&emc1403_chip_info,
diff --git a/drivers/hwmon/hp-wmi-sensors.c b/drivers/hwmon/hp-wmi-sensors.c
index d6bdad26feb1..82210fef828f 100644
--- a/drivers/hwmon/hp-wmi-sensors.c
+++ b/drivers/hwmon/hp-wmi-sensors.c
@@ -1247,7 +1247,9 @@ static int fungible_show(struct seq_file *seqf, enum hp_wmi_property prop)
break;
case HP_WMI_PROPERTY_CURRENT_STATE:
+ mutex_lock(&state->lock);
seq_printf(seqf, "%s\n", nsensor->current_state);
+ mutex_unlock(&state->lock);
break;
case HP_WMI_PROPERTY_UNIT_MODIFIER:
diff --git a/drivers/hwmon/pmbus/pmbus.h b/drivers/hwmon/pmbus/pmbus.h
index d605412a3173..c68b90ad8be3 100644
--- a/drivers/hwmon/pmbus/pmbus.h
+++ b/drivers/hwmon/pmbus/pmbus.h
@@ -241,6 +241,7 @@ enum pmbus_regs {
/*
* OPERATION
*/
+#define PB_OPERATION_CONTROL_V_SRC GENMASK(5, 4)
#define PB_OPERATION_CONTROL_ON BIT(7)
/*
@@ -385,7 +386,7 @@ enum pmbus_sensor_classes {
};
#define PMBUS_PAGES 32 /* Per PMBus specification */
-#define PMBUS_PHASES 10 /* Maximum number of phases per page */
+#define PMBUS_PHASES 16 /* Maximum number of phases per page */
/* Functionality bit mask */
#define PMBUS_HAVE_VIN BIT(0)
diff --git a/drivers/hwmon/pmbus/tps53679.c b/drivers/hwmon/pmbus/tps53679.c
index ecc1be33b3b1..8d1d0ead847e 100644
--- a/drivers/hwmon/pmbus/tps53679.c
+++ b/drivers/hwmon/pmbus/tps53679.c
@@ -168,7 +168,7 @@ static int tps53676_identify(struct i2c_client *client,
return -EIO;
for (i = 0; i < 2 * TPS53676_MAX_PHASES; i += 2) {
if (buf[i + 1] & 0x80) {
- if (buf[i] & 0x08)
+ if (buf[i] & BIT(4))
phases_b++;
else
phases_a++;
@@ -181,6 +181,15 @@ static int tps53676_identify(struct i2c_client *client,
if (phases_b > 0) {
info->pages = 2;
info->phases[1] = phases_b;
+ } else {
+ /*
+ * pmbus_set_page() does not update the PAGE register on
+ * single-page devices, so select page 0 explicitly in case
+ * the boot firmware left the device on another page.
+ */
+ ret = i2c_smbus_write_byte_data(client, PMBUS_PAGE, 0);
+ if (ret < 0)
+ return ret;
}
return 0;
}
diff --git a/drivers/hwmon/pwm-fan.c b/drivers/hwmon/pwm-fan.c
index c434db4656e7..47847aadc4fe 100644
--- a/drivers/hwmon/pwm-fan.c
+++ b/drivers/hwmon/pwm-fan.c
@@ -466,12 +466,18 @@ static void pwm_fan_cleanup(void *__ctx)
{
struct pwm_fan_ctx *ctx = __ctx;
- del_timer_sync(&ctx->rpm_timer);
/* Switch off everything */
ctx->enable_mode = pwm_disable_reg_disable;
pwm_fan_power_off(ctx, true);
}
+static void pwm_fan_timer_cleanup(void *__ctx)
+{
+ struct pwm_fan_ctx *ctx = __ctx;
+
+ timer_shutdown_sync(&ctx->rpm_timer);
+}
+
static int pwm_fan_probe(struct platform_device *pdev)
{
struct thermal_cooling_device *cdev;
@@ -614,6 +620,10 @@ static int pwm_fan_probe(struct platform_device *pdev)
}
if (ctx->tach_count > 0) {
+ ret = devm_add_action_or_reset(dev, pwm_fan_timer_cleanup, ctx);
+ if (ret)
+ return ret;
+
ctx->sample_start = ktime_get();
mod_timer(&ctx->rpm_timer, jiffies + HZ);
@@ -655,6 +665,7 @@ static void pwm_fan_shutdown(struct platform_device *pdev)
{
struct pwm_fan_ctx *ctx = platform_get_drvdata(pdev);
+ pwm_fan_timer_cleanup(ctx);
pwm_fan_cleanup(ctx);
}
diff --git a/drivers/hwmon/w83791d.c b/drivers/hwmon/w83791d.c
index 996e36951f9d..21f421d7379f 100644
--- a/drivers/hwmon/w83791d.c
+++ b/drivers/hwmon/w83791d.c
@@ -1415,6 +1415,7 @@ static void w83791d_remove(struct i2c_client *client)
struct w83791d_data *data = i2c_get_clientdata(client);
hwmon_device_unregister(data->hwmon_dev);
+ sysfs_remove_group(&client->dev.kobj, &w83791d_group_fanpwm45);
sysfs_remove_group(&client->dev.kobj, &w83791d_group);
}
diff --git a/drivers/hwmon/w83793.c b/drivers/hwmon/w83793.c
index ec03745328a6..109d0834834e 100644
--- a/drivers/hwmon/w83793.c
+++ b/drivers/hwmon/w83793.c
@@ -1928,7 +1928,9 @@ static int w83793_probe(struct i2c_client *client)
for (i = 0; i < ARRAY_SIZE(w83793_temp); i++)
device_remove_file(dev, &w83793_temp[i].dev_attr);
free_mem:
- kfree(data);
+ mutex_lock(&watchdog_data_mutex);
+ kref_put(&data->kref, w83793_release_resources);
+ mutex_unlock(&watchdog_data_mutex);
exit:
return err;
}
diff --git a/drivers/i2c/busses/i2c-at91-core.c b/drivers/i2c/busses/i2c-at91-core.c
index dc52b3530725..771e1fb00fec 100644
--- a/drivers/i2c/busses/i2c-at91-core.c
+++ b/drivers/i2c/busses/i2c-at91-core.c
@@ -255,6 +255,7 @@ static int at91_twi_probe(struct platform_device *pdev)
if (rc) {
pm_runtime_disable(dev->dev);
pm_runtime_set_suspended(dev->dev);
+ at91_twi_dma_release(dev);
return rc;
}
@@ -270,6 +271,8 @@ static void at91_twi_remove(struct platform_device *pdev)
i2c_del_adapter(&dev->adapter);
+ at91_twi_dma_release(dev);
+
pm_runtime_disable(dev->dev);
pm_runtime_set_suspended(dev->dev);
}
diff --git a/drivers/i2c/busses/i2c-at91-master.c b/drivers/i2c/busses/i2c-at91-master.c
index ee3b469ddfb9..f265a41cf31f 100644
--- a/drivers/i2c/busses/i2c-at91-master.c
+++ b/drivers/i2c/busses/i2c-at91-master.c
@@ -817,11 +817,21 @@ static int at91_twi_configure_dma(struct at91_twi_dev *dev, u32 phy_addr)
error:
if (ret != -EPROBE_DEFER)
dev_info(dev->dev, "can't get DMA channel, continue without DMA support\n");
+ at91_twi_dma_release(dev);
+ return ret;
+}
+
+void at91_twi_dma_release(struct at91_twi_dev *dev)
+{
+ struct at91_twi_dma *dma = &dev->dma;
+
if (dma->chan_rx)
dma_release_channel(dma->chan_rx);
if (dma->chan_tx)
dma_release_channel(dma->chan_tx);
- return ret;
+ dma->chan_rx = NULL;
+ dma->chan_tx = NULL;
+ dev->use_dma = false;
}
static int at91_init_twi_recovery_gpio(struct platform_device *pdev,
diff --git a/drivers/i2c/busses/i2c-at91.h b/drivers/i2c/busses/i2c-at91.h
index 942e9c3973bb..d68fcbbc3e0f 100644
--- a/drivers/i2c/busses/i2c-at91.h
+++ b/drivers/i2c/busses/i2c-at91.h
@@ -172,6 +172,7 @@ void at91_twi_irq_restore(struct at91_twi_dev *dev);
void at91_init_twi_bus(struct at91_twi_dev *dev);
void at91_init_twi_bus_master(struct at91_twi_dev *dev);
+void at91_twi_dma_release(struct at91_twi_dev *dev);
int at91_twi_probe_master(struct platform_device *pdev, u32 phy_addr,
struct at91_twi_dev *dev);
diff --git a/drivers/i2c/busses/i2c-imx.c b/drivers/i2c/busses/i2c-imx.c
index 2f98ac4ad214..85e17ca8b139 100644
--- a/drivers/i2c/busses/i2c-imx.c
+++ b/drivers/i2c/busses/i2c-imx.c
@@ -1557,6 +1557,8 @@ static int i2c_imx_probe(struct platform_device *pdev)
clk_notifier_unregister:
clk_notifier_unregister(i2c_imx->clk, &i2c_imx->clk_change_nb);
+ if (i2c_imx->dma)
+ i2c_imx_dma_free(i2c_imx);
free_irq(irq, i2c_imx);
rpm_disable:
pm_runtime_put_noidle(&pdev->dev);
@@ -1597,6 +1599,7 @@ static void i2c_imx_remove(struct platform_device *pdev)
pm_runtime_put_noidle(&pdev->dev);
pm_runtime_disable(&pdev->dev);
+ pm_runtime_dont_use_autosuspend(&pdev->dev);
}
static int i2c_imx_runtime_suspend(struct device *dev)
diff --git a/drivers/i2c/busses/i2c-mxs.c b/drivers/i2c/busses/i2c-mxs.c
index 36def0a9c95c..e3b1ee4c4195 100644
--- a/drivers/i2c/busses/i2c-mxs.c
+++ b/drivers/i2c/busses/i2c-mxs.c
@@ -831,7 +831,7 @@ static int mxs_i2c_probe(struct platform_device *pdev)
}
/* Setup the DMA */
- i2c->dmach = dma_request_chan(dev, "rx-tx");
+ i2c->dmach = devm_dma_request_chan(dev, "rx-tx");
if (IS_ERR(i2c->dmach)) {
return dev_err_probe(dev, PTR_ERR(i2c->dmach),
"Failed to request dma\n");
@@ -869,9 +869,6 @@ static void mxs_i2c_remove(struct platform_device *pdev)
i2c_del_adapter(&i2c->adapter);
- if (i2c->dmach)
- dma_release_channel(i2c->dmach);
-
writel(MXS_I2C_CTRL0_SFTRST, i2c->regs + MXS_I2C_CTRL0_SET);
}
diff --git a/drivers/i2c/busses/i2c-qcom-cci.c b/drivers/i2c/busses/i2c-qcom-cci.c
index 85de659bb362..c0309bf61595 100644
--- a/drivers/i2c/busses/i2c-qcom-cci.c
+++ b/drivers/i2c/busses/i2c-qcom-cci.c
@@ -520,11 +520,14 @@ static const struct dev_pm_ops qcom_cci_pm = {
SET_RUNTIME_PM_OPS(cci_suspend_runtime, cci_resume_runtime, NULL)
};
+static void cci_put_of_node(void *data)
+{
+ of_node_put(data);
+}
+
static int cci_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
- unsigned long cci_clk_rate = 0;
- struct device_node *child;
struct resource *r;
struct cci *cci;
int ret, i;
@@ -540,7 +543,7 @@ static int cci_probe(struct platform_device *pdev)
if (!cci->data)
return -ENOENT;
- for_each_available_child_of_node(dev->of_node, child) {
+ for_each_available_child_of_node_scoped(dev->of_node, child) {
struct cci_master *master;
u32 idx;
@@ -561,6 +564,9 @@ static int cci_probe(struct platform_device *pdev)
master->adap.algo = &cci_algo;
master->adap.dev.parent = dev;
master->adap.dev.of_node = of_node_get(child);
+ ret = devm_add_action_or_reset(dev, cci_put_of_node, child);
+ if (ret)
+ return ret;
master->master = idx;
master->cci = cci;
@@ -594,22 +600,6 @@ static int cci_probe(struct platform_device *pdev)
return dev_err_probe(dev, -EINVAL, "not enough clocks in DT\n");
cci->nclocks = ret;
- /* Retrieve CCI clock rate */
- for (i = 0; i < cci->nclocks; i++) {
- if (!strcmp(cci->clocks[i].id, "cci")) {
- cci_clk_rate = clk_get_rate(cci->clocks[i].clk);
- break;
- }
- }
-
- if (cci_clk_rate != cci->data->cci_clk_rate) {
- /* cci clock set by the bootloader or via assigned clock rate
- * in DT.
- */
- dev_warn(dev, "Found %lu cci clk rate while %lu was expected\n",
- cci_clk_rate, cci->data->cci_clk_rate);
- }
-
ret = cci_enable_clocks(cci);
if (ret < 0)
return ret;
@@ -648,10 +638,8 @@ static int cci_probe(struct platform_device *pdev)
continue;
ret = i2c_add_adapter(&cci->master[i].adap);
- if (ret < 0) {
- of_node_put(cci->master[i].adap.dev.of_node);
+ if (ret < 0)
goto error_i2c;
- }
}
return 0;
@@ -661,10 +649,8 @@ static int cci_probe(struct platform_device *pdev)
pm_runtime_dont_use_autosuspend(dev);
for (--i ; i >= 0; i--) {
- if (cci->master[i].cci) {
+ if (cci->master[i].cci)
i2c_del_adapter(&cci->master[i].adap);
- of_node_put(cci->master[i].adap.dev.of_node);
- }
}
error:
disable_irq(cci->irq);
@@ -682,7 +668,6 @@ static void cci_remove(struct platform_device *pdev)
for (i = 0; i < cci->data->num_masters; i++) {
if (cci->master[i].cci) {
i2c_del_adapter(&cci->master[i].adap);
- of_node_put(cci->master[i].adap.dev.of_node);
cci_halt(cci, i);
}
}
diff --git a/drivers/i2c/busses/i2c-qcom-geni.c b/drivers/i2c/busses/i2c-qcom-geni.c
index 96b024dc3f20..66156506ee1a 100644
--- a/drivers/i2c/busses/i2c-qcom-geni.c
+++ b/drivers/i2c/busses/i2c-qcom-geni.c
@@ -77,6 +77,8 @@ enum geni_i2c_err_code {
#define XFER_TIMEOUT HZ
#define RST_TIMEOUT HZ
+#define GENI_SE_CLK_19P2MHZ 19200000UL
+
struct geni_i2c_dev {
struct geni_se se;
u32 tx_wm;
@@ -91,6 +93,7 @@ struct geni_i2c_dev {
struct clk *core_clk;
u32 clk_freq_out;
const struct geni_i2c_clk_fld *clk_fld;
+ u32 clk_idx;
int suspended;
void *dma_buf;
size_t xfer_len;
@@ -157,13 +160,36 @@ static int geni_i2c_clk_map_idx(struct geni_i2c_dev *gi2c)
{
int i;
const struct geni_i2c_clk_fld *itr = geni_i2c_clk_map;
+ unsigned long res_freq;
+
+ /*
+ * Frequency counters are calibrated for a 19.2 MHz source clock
+ * and are not valid for any multiple of it (e.g. 38.4 MHz).
+ * Use exact=true and verify res_freq matches req_freq literally
+ * to reject harmonics that would produce an incorrect I2C frequency.
+ * ACPI systems have firmware-managed clocks and retain the default index.
+ */
+ if (!has_acpi_companion(gi2c->se.dev) &&
+ (geni_se_clk_freq_match(&gi2c->se, GENI_SE_CLK_19P2MHZ,
+ &gi2c->clk_idx, &res_freq, true) ||
+ res_freq != GENI_SE_CLK_19P2MHZ)) {
+ dev_err(gi2c->se.dev,
+ "Unsupported SE source clock: must be exactly 19.2 MHz\n");
+ return -EINVAL;
+ }
for (i = 0; i < ARRAY_SIZE(geni_i2c_clk_map); i++, itr++) {
if (itr->clk_freq_out == gi2c->clk_freq_out) {
gi2c->clk_fld = itr;
+ dev_dbg(gi2c->se.dev,
+ "I2C clk selected: freq: %u Hz, clk_idx: %u\n",
+ gi2c->clk_freq_out, gi2c->clk_idx);
return 0;
}
}
+
+ dev_err(gi2c->se.dev, "Unsupported I2C output frequency %u Hz\n", gi2c->clk_freq_out);
+
return -EINVAL;
}
@@ -172,7 +198,7 @@ static void qcom_geni_i2c_conf(struct geni_i2c_dev *gi2c)
const struct geni_i2c_clk_fld *itr = gi2c->clk_fld;
u32 val;
- writel_relaxed(0, gi2c->se.base + SE_GENI_CLK_SEL);
+ writel_relaxed(gi2c->clk_idx, gi2c->se.base + SE_GENI_CLK_SEL);
val = (itr->clk_div << CLK_DIV_SHFT) | SER_CLK_EN;
writel_relaxed(val, gi2c->se.base + GENI_SER_M_CLK_CFG);
@@ -809,11 +835,8 @@ static int geni_i2c_probe(struct platform_device *pdev)
return gi2c->irq;
ret = geni_i2c_clk_map_idx(gi2c);
- if (ret) {
- dev_err(dev, "Invalid clk frequency %d Hz: %d\n",
- gi2c->clk_freq_out, ret);
+ if (ret)
return ret;
- }
gi2c->adap.algo = &geni_i2c_algo;
init_completion(&gi2c->done);
diff --git a/drivers/i2c/i2c-atr.c b/drivers/i2c/i2c-atr.c
index 5342e934aa5e..9d56c105c1f0 100644
--- a/drivers/i2c/i2c-atr.c
+++ b/drivers/i2c/i2c-atr.c
@@ -632,6 +632,7 @@ int i2c_atr_add_adapter(struct i2c_atr *atr, u32 chan_id,
ret = i2c_add_adapter(&chan->adap);
if (ret) {
+ atr->adapter[chan_id] = NULL;
dev_err(dev, "failed to add atr-adapter %u (error=%d)\n",
chan_id, ret);
goto err_fwnode_put;
diff --git a/drivers/i3c/master.c b/drivers/i3c/master.c
index 48a91eac5ae5..4b23fe33329b 100644
--- a/drivers/i3c/master.c
+++ b/drivers/i3c/master.c
@@ -700,6 +700,11 @@ static struct attribute *i3c_masterdev_attrs[] = {
};
ATTRIBUTE_GROUPS(i3c_masterdev);
+static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev)
+{
+ kfree(dev);
+}
+
static void i3c_masterdev_release(struct device *dev)
{
struct i3c_master_controller *master = dev_to_i3cmaster(dev);
@@ -712,6 +717,8 @@ static void i3c_masterdev_release(struct device *dev)
i3c_bus_cleanup(bus);
of_node_put(dev->of_node);
+
+ i3c_master_free_i3c_dev(master->this);
}
static const struct device_type i3c_masterdev_type = {
@@ -899,11 +906,6 @@ static void i3c_device_release(struct device *dev)
kfree(i3cdev);
}
-static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev)
-{
- kfree(dev);
-}
-
static struct i3c_dev_desc *
i3c_master_alloc_i3c_dev(struct i3c_master_controller *master,
const struct i3c_device_info *info)
@@ -1784,6 +1786,8 @@ int i3c_master_set_info(struct i3c_master_controller *master,
return 0;
err_free_dev:
+ master->bus.cur_master = NULL;
+ master->this = NULL;
i3c_master_free_i3c_dev(i3cdev);
return ret;
@@ -1804,7 +1808,8 @@ static void i3c_master_detach_free_devs(struct i3c_master_controller *master)
i3cdev->boardinfo->init_dyn_addr,
I3C_ADDR_SLOT_FREE);
- i3c_master_free_i3c_dev(i3cdev);
+ if (i3cdev != master->this)
+ i3c_master_free_i3c_dev(i3cdev);
}
list_for_each_entry_safe(i2cdev, i2ctmp, &master->bus.devs.i2c,
@@ -2024,7 +2029,8 @@ i3c_master_search_i3c_dev_duplicate(struct i3c_dev_desc *refdev)
struct i3c_dev_desc *i3cdev;
i3c_bus_for_each_i3cdev(&master->bus, i3cdev) {
- if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid)
+ if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid &&
+ i3cdev != master->this)
return i3cdev;
}
diff --git a/drivers/infiniband/core/iwpm_util.c b/drivers/infiniband/core/iwpm_util.c
index eecb369898f5..2c30f5c43c85 100644
--- a/drivers/infiniband/core/iwpm_util.c
+++ b/drivers/infiniband/core/iwpm_util.c
@@ -314,10 +314,6 @@ struct iwpm_nlmsg_request *iwpm_get_nlmsg_request(__u32 nlmsg_seq,
if (!nlmsg_request)
return NULL;
- spin_lock_irqsave(&iwpm_nlmsg_req_lock, flags);
- list_add_tail(&nlmsg_request->inprocess_list, &iwpm_nlmsg_req_list);
- spin_unlock_irqrestore(&iwpm_nlmsg_req_lock, flags);
-
kref_init(&nlmsg_request->kref);
kref_get(&nlmsg_request->kref);
nlmsg_request->nlmsg_seq = nlmsg_seq;
@@ -326,6 +322,11 @@ struct iwpm_nlmsg_request *iwpm_get_nlmsg_request(__u32 nlmsg_seq,
nlmsg_request->err_code = 0;
sema_init(&nlmsg_request->sem, 1);
down(&nlmsg_request->sem);
+
+ spin_lock_irqsave(&iwpm_nlmsg_req_lock, flags);
+ list_add_tail(&nlmsg_request->inprocess_list, &iwpm_nlmsg_req_list);
+ spin_unlock_irqrestore(&iwpm_nlmsg_req_lock, flags);
+
return nlmsg_request;
}
diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
index 96c1fd8039fb..21f482e0243f 100644
--- a/drivers/infiniband/core/mad.c
+++ b/drivers/infiniband/core/mad.c
@@ -1805,6 +1805,8 @@ static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv,
int ret;
INIT_LIST_HEAD(&mad_recv_wc->rmpp_list);
+ list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list);
+
ret = ib_mad_enforce_security(mad_agent_priv,
mad_recv_wc->wc->pkey_index);
if (ret) {
@@ -1813,7 +1815,6 @@ static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv,
return;
}
- list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list);
if (is_kernel_rmpp_data_response(mad_agent_priv, mad_recv_wc)) {
spin_lock_irqsave(&mad_agent_priv->lock, flags);
mad_send_wr = ib_find_send_mad(mad_agent_priv, mad_recv_wc);
diff --git a/drivers/infiniband/core/ucma.c b/drivers/infiniband/core/ucma.c
index 5abab926e335..2ca5c4f1a70c 100644
--- a/drivers/infiniband/core/ucma.c
+++ b/drivers/infiniband/core/ucma.c
@@ -1487,9 +1487,10 @@ static ssize_t ucma_process_join(struct ucma_file *file,
mutex_lock(&ctx->mutex);
ret = rdma_join_multicast(ctx->cm_id, (struct sockaddr *)&mc->addr,
join_state, mc);
- mutex_unlock(&ctx->mutex);
- if (ret)
+ if (ret) {
+ mutex_unlock(&ctx->mutex);
goto err_xa_erase;
+ }
resp.id = mc->id;
if (copy_to_user(u64_to_user_ptr(cmd->response),
@@ -1497,6 +1498,7 @@ static ssize_t ucma_process_join(struct ucma_file *file,
ret = -EFAULT;
goto err_leave_multicast;
}
+ mutex_unlock(&ctx->mutex);
xa_store(&multicast_table, mc->id, mc, 0);
@@ -1504,7 +1506,6 @@ static ssize_t ucma_process_join(struct ucma_file *file,
return 0;
err_leave_multicast:
- mutex_lock(&ctx->mutex);
rdma_leave_multicast(ctx->cm_id, (struct sockaddr *) &mc->addr);
mutex_unlock(&ctx->mutex);
ucma_cleanup_mc_events(mc);
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index d0bd57ac7c6a..d6fd7db5cbbd 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -2000,11 +2000,13 @@ int ib_get_eth_speed(struct ib_device *dev, u32 port_num, u16 *speed, u8 *width)
return -ENODEV;
rtnl_lock();
- rc = __ethtool_get_link_ksettings(netdev, &lksettings);
- rtnl_unlock();
-
- dev_put(netdev);
+ if (READ_ONCE(netdev->reg_state) != NETREG_REGISTERED) {
+ dev_put(netdev);
+ rtnl_unlock();
+ return -ENODEV;
+ }
+ rc = __ethtool_get_link_ksettings(netdev, &lksettings);
if (!rc && lksettings.base.speed != (u32)SPEED_UNKNOWN) {
netdev_speed = lksettings.base.speed;
} else {
@@ -2013,6 +2015,8 @@ int ib_get_eth_speed(struct ib_device *dev, u32 port_num, u16 *speed, u8 *width)
pr_warn("%s speed is unknown, defaulting to %u\n",
netdev->name, netdev_speed);
}
+ dev_put(netdev);
+ rtnl_unlock();
ib_get_width_and_speed(netdev_speed, lksettings.lanes,
speed, width);
diff --git a/drivers/infiniband/hw/efa/efa_com.c b/drivers/infiniband/hw/efa/efa_com.c
index bafd210dd43e..00e339abc2c1 100644
--- a/drivers/infiniband/hw/efa/efa_com.c
+++ b/drivers/infiniband/hw/efa/efa_com.c
@@ -743,7 +743,7 @@ int efa_com_admin_init(struct efa_com_dev *edev,
aq->dmadev = edev->dmadev;
aq->efa_dev = edev->efa_dev;
- set_bit(EFA_AQ_STATE_POLLING_BIT, &aq->state);
+ efa_com_set_admin_polling_mode(edev, true);
sema_init(&aq->avail_cmds, aq->depth);
@@ -761,8 +761,6 @@ int efa_com_admin_init(struct efa_com_dev *edev,
if (err)
goto err_destroy_sq;
- efa_com_set_admin_polling_mode(edev, false);
-
err = efa_com_admin_init_aenq(edev, aenq_handlers);
if (err)
goto err_destroy_cq;
@@ -1151,7 +1149,7 @@ static void efa_com_destroy_eq(struct efa_com_dev *edev,
err);
}
-static void efa_com_arm_eq(struct efa_com_dev *edev, struct efa_com_eq *eeq)
+void efa_com_arm_eq(struct efa_com_dev *edev, struct efa_com_eq *eeq)
{
u32 val = 0;
@@ -1240,7 +1238,6 @@ int efa_com_eq_init(struct efa_com_dev *edev, struct efa_com_eq *eeq,
eeq->phase = 1;
eeq->depth = params.depth;
eeq->cb = cb;
- efa_com_arm_eq(edev, eeq);
return 0;
diff --git a/drivers/infiniband/hw/efa/efa_com.h b/drivers/infiniband/hw/efa/efa_com.h
index 77282234ce68..29a9d087db5d 100644
--- a/drivers/infiniband/hw/efa/efa_com.h
+++ b/drivers/infiniband/hw/efa/efa_com.h
@@ -157,6 +157,7 @@ int efa_com_admin_init(struct efa_com_dev *edev,
void efa_com_admin_destroy(struct efa_com_dev *edev);
int efa_com_eq_init(struct efa_com_dev *edev, struct efa_com_eq *eeq,
efa_eqe_handler cb, u16 depth, u8 msix_vec);
+void efa_com_arm_eq(struct efa_com_dev *edev, struct efa_com_eq *eeq);
void efa_com_eq_destroy(struct efa_com_dev *edev, struct efa_com_eq *eeq);
int efa_com_dev_reset(struct efa_com_dev *edev,
enum efa_regs_reset_reason_types reset_reason);
diff --git a/drivers/infiniband/hw/efa/efa_main.c b/drivers/infiniband/hw/efa/efa_main.c
index 45a4564c670c..30cefd0bb4f5 100644
--- a/drivers/infiniband/hw/efa/efa_main.c
+++ b/drivers/infiniband/hw/efa/efa_main.c
@@ -301,28 +301,30 @@ static void efa_set_host_info(struct efa_dev *dev)
static void efa_destroy_eq(struct efa_dev *dev, struct efa_eq *eq)
{
- efa_com_eq_destroy(&dev->edev, &eq->eeq);
efa_free_irq(dev, &eq->irq);
+ efa_com_eq_destroy(&dev->edev, &eq->eeq);
}
static int efa_create_eq(struct efa_dev *dev, struct efa_eq *eq, u8 msix_vec)
{
int err;
- efa_setup_comp_irq(dev, eq, msix_vec);
- err = efa_request_irq(dev, &eq->irq);
+ err = efa_com_eq_init(&dev->edev, &eq->eeq, efa_process_eqe,
+ dev->dev_attr.max_eq_depth, msix_vec);
if (err)
return err;
- err = efa_com_eq_init(&dev->edev, &eq->eeq, efa_process_eqe,
- dev->dev_attr.max_eq_depth, msix_vec);
+ efa_setup_comp_irq(dev, eq, msix_vec);
+ err = efa_request_irq(dev, &eq->irq);
if (err)
- goto err_free_comp_irq;
+ goto err_destroy_eq;
+
+ efa_com_arm_eq(&dev->edev, &eq->eeq);
return 0;
-err_free_comp_irq:
- efa_free_irq(dev, &eq->irq);
+err_destroy_eq:
+ efa_com_eq_destroy(&dev->edev, &eq->eeq);
return err;
}
@@ -615,18 +617,21 @@ static struct efa_dev *efa_probe_device(struct pci_dev *pdev)
edev->aq.msix_vector_idx = dev->admin_msix_vector_idx;
edev->aenq.msix_vector_idx = dev->admin_msix_vector_idx;
- err = efa_set_mgmnt_irq(dev);
+ err = efa_com_admin_init(edev, &aenq_handlers);
if (err)
goto err_disable_msix;
- err = efa_com_admin_init(edev, &aenq_handlers);
+ err = efa_set_mgmnt_irq(dev);
if (err)
- goto err_free_mgmnt_irq;
+ goto err_destroy_admin;
+
+ efa_com_set_admin_polling_mode(edev, false);
return dev;
-err_free_mgmnt_irq:
- efa_free_irq(dev, &dev->admin_irq);
+err_destroy_admin:
+ efa_com_dev_reset(edev, EFA_REGS_RESET_INIT_ERR);
+ efa_com_admin_destroy(edev);
err_disable_msix:
efa_disable_msix(dev);
err_reg_read_destroy:
@@ -650,8 +655,8 @@ static void efa_remove_device(struct pci_dev *pdev,
edev = &dev->edev;
efa_com_dev_reset(edev, reset_reason);
- efa_com_admin_destroy(edev);
efa_free_irq(dev, &dev->admin_irq);
+ efa_com_admin_destroy(edev);
efa_disable_msix(dev);
efa_com_mmio_reg_read_destroy(edev);
devm_iounmap(&pdev->dev, edev->reg_bar);
diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
index 503abec709c9..a7a8a39e69fb 100644
--- a/drivers/infiniband/hw/hfi1/file_ops.c
+++ b/drivers/infiniband/hw/hfi1/file_ops.c
@@ -326,6 +326,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma)
void *memvirt = NULL;
dma_addr_t memdma = 0;
u8 subctxt, mapio = 0, vmf = 0, type;
+ size_t memdmalen = 0;
ssize_t memlen = 0;
int ret = 0;
u16 ctxt;
@@ -371,7 +372,9 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma)
mapio = 1;
break;
case PIO_CRED: {
+ struct credit_return_base *cr = &dd->cr_base[uctxt->sc->node];
u64 cr_page_offset;
+
if (flags & VM_WRITE) {
ret = -EPERM;
goto done;
@@ -381,11 +384,18 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma)
* second or third page allocated for credit returns (if number
* of enabled contexts > 64 and 128 respectively).
*/
- cr_page_offset = ((u64)uctxt->sc->hw_free -
- (u64)dd->cr_base[uctxt->numa_id].va) &
- PAGE_MASK;
- memvirt = dd->cr_base[uctxt->numa_id].va + cr_page_offset;
- memdma = dd->cr_base[uctxt->numa_id].dma + cr_page_offset;
+ cr_page_offset = ((u64)uctxt->sc->hw_free - (u64)cr->va) &
+ PAGE_MASK;
+ /*
+ * dma_mmap_coherent() describes the whole coherent buffer and
+ * selects the page within it with vma->vm_pgoff, so pass the
+ * base of the allocation and its length and let vm_pgoff pick
+ * the page.
+ */
+ vma->vm_pgoff = cr_page_offset >> PAGE_SHIFT;
+ memvirt = cr->va;
+ memdma = cr->dma;
+ memdmalen = TXE_NUM_CONTEXTS * sizeof(struct credit_return);
memlen = PAGE_SIZE;
flags &= ~VM_MAYWRITE;
flags |= VM_DONTCOPY | VM_DONTEXPAND;
@@ -567,7 +577,8 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma)
ret = 0;
} else if (memdma) {
ret = dma_mmap_coherent(&dd->pcidev->dev, vma,
- memvirt, memdma, memlen);
+ memvirt, memdma,
+ memdmalen ? memdmalen : memlen);
} else if (mapio) {
ret = io_remap_pfn_range(vma, vma->vm_start,
PFN_DOWN(memaddr),
diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
index d8e101fc74ff..97fa34563552 100644
--- a/drivers/infiniband/hw/irdma/verbs.c
+++ b/drivers/infiniband/hw/irdma/verbs.c
@@ -3571,7 +3571,7 @@ static int irdma_post_send(struct ib_qp *ibqp,
stag_info.total_len = iwmr->ibmr.length;
stag_info.reg_addr_pa = *palloc->level1.addr;
stag_info.first_pm_pbl_index = palloc->level1.idx;
- stag_info.local_fence = ib_wr->send_flags & IB_SEND_FENCE;
+ stag_info.local_fence = true;
if (iwmr->npages > IRDMA_MIN_PAGES_PER_FMR)
stag_info.chunk_size = 1;
err = irdma_sc_mr_fast_register(&iwqp->sc_qp, &stag_info,
diff --git a/drivers/infiniband/hw/mlx4/sysfs.c b/drivers/infiniband/hw/mlx4/sysfs.c
index 1676f819c2f7..9e6a0a22663d 100644
--- a/drivers/infiniband/hw/mlx4/sysfs.c
+++ b/drivers/infiniband/hw/mlx4/sysfs.c
@@ -753,11 +753,13 @@ static int register_one_pkey_tree(struct mlx4_ib_dev *dev, int slave)
kobject_put(p);
}
kobject_put(dev->dev_ports_parent[slave]);
+ dev->dev_ports_parent[slave] = NULL;
err_ports:
kobject_put(dev->pkeys.device_parent[slave]);
/* extra put for the device_parent create_and_add */
kobject_put(dev->pkeys.device_parent[slave]);
+ dev->pkeys.device_parent[slave] = NULL;
fail_dev:
kobject_put(dev->iov_parent);
@@ -787,6 +789,8 @@ static void unregister_pkey_tree(struct mlx4_ib_dev *device)
return;
for (slave = device->dev->persist->num_vfs; slave >= 0; --slave) {
+ if (!device->pkeys.device_parent[slave])
+ continue;
list_for_each_entry_safe(p, t,
&device->pkeys.pkey_port_list[slave],
entry) {
diff --git a/drivers/infiniband/sw/rxe/rxe_mcast.c b/drivers/infiniband/sw/rxe/rxe_mcast.c
index 07ff47bae31d..c14680c9a536 100644
--- a/drivers/infiniband/sw/rxe/rxe_mcast.c
+++ b/drivers/infiniband/sw/rxe/rxe_mcast.c
@@ -175,7 +175,9 @@ struct rxe_mcg *rxe_lookup_mcg(struct rxe_dev *rxe, union ib_gid *mgid)
* @mgid: multicast address as a gid
* @mcg: new mcg object
*
- * Context: caller should hold rxe->mcg lock
+ * Initializes the mcg fields. The mcg is private and not yet visible in
+ * mcg_tree, so this may run without rxe->mcg_lock; __rxe_publish_mcg()
+ * makes it visible under the lock once it is ready.
*/
static void __rxe_init_mcg(struct rxe_dev *rxe, union ib_gid *mgid,
struct rxe_mcg *mcg)
@@ -184,13 +186,22 @@ static void __rxe_init_mcg(struct rxe_dev *rxe, union ib_gid *mgid,
memcpy(&mcg->mgid, mgid, sizeof(mcg->mgid));
INIT_LIST_HEAD(&mcg->qp_list);
mcg->rxe = rxe;
+}
- /* caller holds a ref on mcg but that will be
- * dropped when mcg goes out of scope. We need to take a ref
- * on the pointer that will be saved in the red-black tree
- * by __rxe_insert_mcg and used to lookup mcg from mgid later.
- * Inserting mcg makes it visible to outside so this should
- * be done last after the object is ready.
+/**
+ * __rxe_publish_mcg - make a fully initialized mcg visible in mcg_tree
+ * @mcg: the mcg object
+ *
+ * Context: caller must hold rxe->mcg_lock and a reference on mcg
+ */
+static void __rxe_publish_mcg(struct rxe_mcg *mcg)
+{
+ /* caller holds a ref on mcg but that will be dropped when mcg goes
+ * out of scope. We need to take a ref on the pointer that will be
+ * saved in the red-black tree by __rxe_insert_mcg and used to lookup
+ * mcg from mgid later. Inserting mcg makes it visible to outside so
+ * this is done last after the object is ready and the multicast
+ * address has been programmed.
*/
kref_get(&mcg->ref_cnt);
__rxe_insert_mcg(mcg);
@@ -228,26 +239,37 @@ static struct rxe_mcg *rxe_get_mcg(struct rxe_dev *rxe, union ib_gid *mgid)
err = -ENOMEM;
goto err_dec;
}
+ __rxe_init_mcg(rxe, mgid, mcg);
+
+ /* program the multicast address while mcg is still private, before
+ * it is inserted into mcg_tree. dev_mc_add() may sleep so this must
+ * run outside mcg_lock. On failure mcg was never published, so a
+ * plain free is correct and the tree is untouched.
+ */
+ err = rxe_mcast_add(rxe, mgid);
+ if (err) {
+ kfree(mcg);
+ goto err_dec;
+ }
spin_lock_bh(&rxe->mcg_lock);
- /* re-check to see if someone else just added it */
+ /* re-check to see if someone else just added it while we were adding
+ * the multicast address; if so use theirs and drop ours
+ */
tmp = __rxe_lookup_mcg(rxe, mgid);
if (tmp) {
spin_unlock_bh(&rxe->mcg_lock);
+ rxe_mcast_del(rxe, mgid);
atomic_dec(&rxe->mcg_num);
kfree(mcg);
return tmp;
}
- __rxe_init_mcg(rxe, mgid, mcg);
+ __rxe_publish_mcg(mcg);
spin_unlock_bh(&rxe->mcg_lock);
- /* add mcast address outside of lock */
- err = rxe_mcast_add(rxe, mgid);
- if (!err)
- return mcg;
+ return mcg;
- kfree(mcg);
err_dec:
atomic_dec(&rxe->mcg_num);
return ERR_PTR(err);
diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index da3dee520876..83e325b76f67 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -33,7 +33,8 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length)
case IB_MR_TYPE_USER:
case IB_MR_TYPE_MEM_REG:
if (iova < mr->ibmr.iova ||
- iova + length > mr->ibmr.iova + mr->ibmr.length) {
+ length > mr->ibmr.length ||
+ iova - mr->ibmr.iova > mr->ibmr.length - length) {
rxe_dbg_mr(mr, "iova/length out of range\n");
return -EINVAL;
}
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 9466fed6726b..da3bad301efa 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -1326,19 +1326,20 @@ static struct ib_mr *rxe_rereg_user_mr(struct ib_mr *ibmr, int flags,
if (err)
return ERR_PTR(err);
+ if ((flags & IB_MR_REREG_ACCESS) &&
+ (access & ~RXE_ACCESS_SUPPORTED_MR)) {
+ rxe_err_mr(mr, "access = %#x not supported\n", access);
+ return ERR_PTR(-EOPNOTSUPP);
+ }
+
if (flags & IB_MR_REREG_PD) {
rxe_put(old_pd);
rxe_get(pd);
mr->ibmr.pd = ibpd;
}
- if (flags & IB_MR_REREG_ACCESS) {
- if (access & ~RXE_ACCESS_SUPPORTED_MR) {
- rxe_err_mr(mr, "access = %#x not supported\n", access);
- return ERR_PTR(-EOPNOTSUPP);
- }
+ if (flags & IB_MR_REREG_ACCESS)
mr->access = access;
- }
return NULL;
}
diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c
index bb7d90963907..04fabab44058 100644
--- a/drivers/infiniband/sw/siw/siw_cm.c
+++ b/drivers/infiniband/sw/siw/siw_cm.c
@@ -1669,9 +1669,12 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE |
SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD |
SIW_QP_ATTR_MPA);
+ if (rv) {
+ qp->cep = NULL;
+ siw_cep_put(cep);
+ goto error_unlock;
+ }
up_write(&qp->state_lock);
- if (rv)
- goto error;
siw_dbg_cep(cep, "[QP %u]: send mpa reply, %d byte pdata\n",
qp_id(qp), params->private_data_len);
diff --git a/drivers/infiniband/sw/siw/siw_qp_rx.c b/drivers/infiniband/sw/siw/siw_qp_rx.c
index ad281da44cb4..4db245748af0 100644
--- a/drivers/infiniband/sw/siw/siw_qp_rx.c
+++ b/drivers/infiniband/sw/siw/siw_qp_rx.c
@@ -1079,7 +1079,7 @@ static int siw_get_hdr(struct siw_rx_stream *srx)
if (iwarp_pktinfo[opcode].hdr_len > sizeof(struct iwarp_ctrl_tagged)) {
int hdrlen = iwarp_pktinfo[opcode].hdr_len;
- bytes = min_t(int, hdrlen - MIN_DDP_HDR, srx->skb_new);
+ bytes = min_t(int, hdrlen - srx->fpdu_part_rcvd, srx->skb_new);
skb_copy_bits(skb, srx->skb_offset,
(char *)c_hdr + srx->fpdu_part_rcvd, bytes);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h
index abe0522b7df4..ee23b8d37b97 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib.h
+++ b/drivers/infiniband/ulp/ipoib/ipoib.h
@@ -87,6 +87,7 @@ enum {
IPOIB_FLAG_INITIALIZED = 1,
IPOIB_FLAG_ADMIN_UP = 2,
IPOIB_PKEY_ASSIGNED = 3,
+ IPOIB_FLAG_MCAST_FLUSH = 4,
IPOIB_FLAG_SUBINTERFACE = 5,
IPOIB_STOP_REAPER = 7,
IPOIB_FLAG_ADMIN_CM = 9,
@@ -419,6 +420,12 @@ struct ipoib_dev_priv {
const struct net_device_ops *rn_ops;
};
+static inline bool ipoib_mcast_allowed(struct ipoib_dev_priv *priv)
+{
+ return test_bit(IPOIB_FLAG_OPER_UP, &priv->flags) &&
+ !test_bit(IPOIB_FLAG_MCAST_FLUSH, &priv->flags);
+}
+
struct ipoib_ah {
struct net_device *dev;
struct ib_ah *ah;
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 5cde275daa94..76850a7d300b 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -1235,17 +1235,19 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv,
}
if (level == IPOIB_FLUSH_LIGHT) {
- int oper_up;
ipoib_mark_paths_invalid(dev);
- /* Set IPoIB operation as down to prevent races between:
+ /* Set MCAST_FLUSH to prevent races between:
* the flush flow which leaves MCG and on the fly joins
* which can happen during that time. mcast restart task
* should deal with join requests we missed.
+ *
+ * Do not clear OPER_UP for this; restoring it races with
+ * ipoib_ib_dev_down() and can leave OPER_UP set after the
+ * device is down.
*/
- oper_up = test_and_clear_bit(IPOIB_FLAG_OPER_UP, &priv->flags);
+ set_bit(IPOIB_FLAG_MCAST_FLUSH, &priv->flags);
ipoib_mcast_dev_flush(dev);
- if (oper_up)
- set_bit(IPOIB_FLAG_OPER_UP, &priv->flags);
+ clear_bit(IPOIB_FLAG_MCAST_FLUSH, &priv->flags);
ipoib_reap_dead_ahs(priv);
}
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
index 8a4ab9ff0a68..4f7639bbc7dc 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
@@ -74,7 +74,7 @@ static void __ipoib_mcast_schedule_join_thread(struct ipoib_dev_priv *priv,
struct ipoib_mcast *mcast,
bool delay)
{
- if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags))
+ if (!ipoib_mcast_allowed(priv))
return;
/*
@@ -469,7 +469,7 @@ static int ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast)
int ret = 0;
if (!priv->broadcast ||
- !test_bit(IPOIB_FLAG_OPER_UP, &priv->flags))
+ !ipoib_mcast_allowed(priv))
return -EINVAL;
init_completion(&mcast->done);
@@ -555,7 +555,7 @@ void ipoib_mcast_join_task(struct work_struct *work)
unsigned long delay_until = 0;
struct ipoib_mcast *mcast = NULL;
- if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags))
+ if (!ipoib_mcast_allowed(priv))
return;
if (ib_query_port(priv->ca, priv->port, &port_attr)) {
@@ -577,7 +577,7 @@ void ipoib_mcast_join_task(struct work_struct *work)
netif_addr_unlock_bh(dev);
spin_lock_irq(&priv->lock);
- if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags))
+ if (!ipoib_mcast_allowed(priv))
goto out;
if (!priv->broadcast) {
@@ -749,7 +749,7 @@ void ipoib_mcast_send(struct net_device *dev, u8 *daddr, struct sk_buff *skb)
spin_lock_irqsave(&priv->lock, flags);
- if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags) ||
+ if (!ipoib_mcast_allowed(priv) ||
!priv->broadcast ||
!test_bit(IPOIB_MCAST_FLAG_ATTACHED, &priv->broadcast->flags)) {
++dev->stats.tx_dropped;
@@ -871,7 +871,7 @@ void ipoib_mcast_restart_task(struct work_struct *work)
LIST_HEAD(remove_list);
struct ib_sa_mcmember_rec rec;
- if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags))
+ if (!ipoib_mcast_allowed(priv))
/*
* shortcut...on shutdown flush is called next, just
* let it do all the work
@@ -965,9 +965,9 @@ void ipoib_mcast_restart_task(struct work_struct *work)
ipoib_mcast_remove_list(&remove_list);
/*
- * Double check that we are still up
+ * Double check that we are still up and not flushing
*/
- if (test_bit(IPOIB_FLAG_OPER_UP, &priv->flags)) {
+ if (ipoib_mcast_allowed(priv)) {
spin_lock_irq(&priv->lock);
__ipoib_mcast_schedule_join_thread(priv, NULL, 0);
spin_unlock_irq(&priv->lock);
diff --git a/drivers/infiniband/ulp/iser/iser_initiator.c b/drivers/infiniband/ulp/iser/iser_initiator.c
index f5f090dc4f1e..cf234ddbaaad 100644
--- a/drivers/infiniband/ulp/iser/iser_initiator.c
+++ b/drivers/infiniband/ulp/iser/iser_initiator.c
@@ -599,11 +599,8 @@ static int iser_check_remote_inv(struct iser_conn *iser_conn, struct ib_wc *wc,
iser_dbg("conn %p: remote invalidation for rkey %#x\n",
iser_conn, rkey);
- if (unlikely(!iser_conn->snd_w_inv)) {
- iser_err("conn %p: unexpected remote invalidation, terminating connection\n",
- iser_conn);
- return -EPROTO;
- }
+ if (unlikely(!iser_conn->snd_w_inv))
+ goto bad_inv;
task = iscsi_itt_to_ctask(iser_conn->iscsi_conn, hdr->itt);
if (likely(task)) {
@@ -612,12 +609,16 @@ static int iser_check_remote_inv(struct iser_conn *iser_conn, struct ib_wc *wc,
if (iser_task->dir[ISER_DIR_IN]) {
desc = iser_task->rdma_reg[ISER_DIR_IN].desc;
+ if (unlikely(!desc))
+ goto bad_inv;
if (unlikely(iser_inv_desc(desc, rkey)))
return -EINVAL;
}
if (iser_task->dir[ISER_DIR_OUT]) {
desc = iser_task->rdma_reg[ISER_DIR_OUT].desc;
+ if (unlikely(!desc))
+ goto bad_inv;
if (unlikely(iser_inv_desc(desc, rkey)))
return -EINVAL;
}
@@ -628,6 +629,11 @@ static int iser_check_remote_inv(struct iser_conn *iser_conn, struct ib_wc *wc,
}
return 0;
+
+bad_inv:
+ iser_err("conn %p: unexpected remote invalidation, terminating connection\n",
+ iser_conn);
+ return -EPROTO;
}
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 6483a55170cd..b3fc753982a3 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -21,6 +21,7 @@
#include <target/target_core_fabric.h>
#include <target/iscsi/iscsi_transport.h>
#include <linux/semaphore.h>
+#include <linux/wait_bit.h>
#include "ib_isert.h"
@@ -311,6 +312,7 @@ isert_init_conn(struct isert_conn *isert_conn)
init_completion(&isert_conn->login_req_comp);
init_waitqueue_head(&isert_conn->rem_wait);
kref_init(&isert_conn->kref);
+ atomic_set(&isert_conn->ctrl_comp_cnt, 0);
mutex_init(&isert_conn->mutex);
INIT_WORK(&isert_conn->release_work, isert_release_work);
}
@@ -1697,6 +1699,8 @@ isert_do_control_comp(struct work_struct *work)
struct isert_conn *isert_conn = isert_cmd->conn;
struct ib_device *ib_dev = isert_conn->cm_id->device;
struct iscsit_cmd *cmd = isert_cmd->iscsit_cmd;
+ /* The switch below may free isert_cmd. */
+ bool counted = isert_cmd->ctrl_counted;
isert_dbg("Cmd %p i_state %d\n", isert_cmd, cmd->i_state);
@@ -1718,6 +1722,14 @@ isert_do_control_comp(struct work_struct *work)
dump_stack();
break;
}
+
+ /*
+ * The count is what keeps isert_conn alive, so drop it last. The wait
+ * queue lives in the global hash table, not in isert_conn, so this is
+ * safe even if the waiter has already freed the connection.
+ */
+ if (counted && atomic_dec_and_test(&isert_conn->ctrl_comp_cnt))
+ wake_up_var(&isert_conn->ctrl_comp_cnt);
}
static void
@@ -1761,6 +1773,12 @@ isert_send_done(struct ib_cq *cq, struct ib_wc *wc)
case ISTATE_SEND_TEXTRSP:
isert_unmap_tx_desc(tx_desc, ib_dev);
+ /* Paired with the wait in isert_wait_conn(). */
+ isert_cmd->ctrl_counted =
+ isert_cmd->iscsit_cmd->i_state != ISTATE_SEND_LOGOUTRSP;
+ if (isert_cmd->ctrl_counted)
+ atomic_inc(&isert_conn->ctrl_comp_cnt);
+
INIT_WORK(&isert_cmd->comp_work, isert_do_control_comp);
queue_work(isert_comp_wq, &isert_cmd->comp_work);
return;
@@ -2605,6 +2623,10 @@ static void isert_wait_conn(struct iscsit_conn *conn)
isert_wait4cmds(conn);
isert_wait4logout(isert_conn);
+ /* Paired with the count taken in isert_send_done(). */
+ wait_var_event(&isert_conn->ctrl_comp_cnt,
+ !atomic_read(&isert_conn->ctrl_comp_cnt));
+
queue_work(isert_release_wq, &isert_conn->release_work);
}
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 0bac5aa66c80..519b17e54bd3 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -153,6 +153,7 @@ struct isert_cmd {
struct work_struct comp_work;
struct scatterlist sg;
bool ctx_init_done;
+ bool ctrl_counted;
};
static inline struct isert_cmd *tx_desc_to_cmd(struct iser_tx_desc *desc)
@@ -187,6 +188,7 @@ struct isert_conn {
struct mutex mutex;
struct kref kref;
struct work_struct release_work;
+ atomic_t ctrl_comp_cnt;
bool logout_posted;
bool snd_w_inv;
wait_queue_head_t rem_wait;
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
index 8fa1d72bd20a..fea3b17611c5 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
@@ -1738,6 +1738,8 @@ static void destroy_con_cq_qp(struct rtrs_clt_con *con)
/*
* Be careful here: destroy_con_cq_qp() can be called even
* create_con_cq_qp() failed, see comments there.
+ * Caller must set con->destroyed under this lock first so a
+ * racing ADDR_RESOLVED cannot ib_cq_pool_get() after we PUT/SKIP.
*/
lockdep_assert_held(&con->con_mutex);
rtrs_cq_qp_destroy(&con->c);
@@ -1772,6 +1774,10 @@ static int rtrs_rdma_addr_resolved(struct rtrs_clt_con *con)
int err;
mutex_lock(&con->con_mutex);
+ if (con->destroyed) {
+ mutex_unlock(&con->con_mutex);
+ return -ECONNABORTED;
+ }
err = create_con_cq_qp(con);
mutex_unlock(&con->con_mutex);
if (err) {
@@ -2202,6 +2208,7 @@ static void rtrs_clt_stop_and_destroy_conns(struct rtrs_clt_path *clt_path)
break;
con = to_clt_con(clt_path->s.con[cid]);
mutex_lock(&con->con_mutex);
+ con->destroyed = true;
destroy_con_cq_qp(con);
mutex_unlock(&con->con_mutex);
destroy_cm(con);
@@ -2368,6 +2375,7 @@ static int init_conns(struct rtrs_clt_path *clt_path)
if (con->c.cm_id) {
stop_cm(con);
mutex_lock(&con->con_mutex);
+ con->destroyed = true;
destroy_con_cq_qp(con);
mutex_unlock(&con->con_mutex);
destroy_cm(con);
diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.h b/drivers/infiniband/ulp/rtrs/rtrs-clt.h
index 0f57759b3080..e1e7c7adc947 100644
--- a/drivers/infiniband/ulp/rtrs/rtrs-clt.h
+++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.h
@@ -75,6 +75,8 @@ struct rtrs_clt_con {
unsigned int cpu;
struct mutex con_mutex;
int cm_err;
+ /* Set under con_mutex before CQ/QP teardown. */
+ bool destroyed;
};
/**
diff --git a/drivers/input/evdev.c b/drivers/input/evdev.c
index 2fa5b8a13131..5d446c1d3cf0 100644
--- a/drivers/input/evdev.c
+++ b/drivers/input/evdev.c
@@ -1229,6 +1229,8 @@ static long evdev_do_ioctl(struct file *file, unsigned int cmd,
t = _IOC_NR(cmd) & ABS_MAX;
+ memset(&abs, 0, sizeof(abs));
+
if (copy_from_user(&abs, p, min_t(size_t,
size, sizeof(struct input_absinfo))))
return -EFAULT;
diff --git a/drivers/input/input-compat.c b/drivers/input/input-compat.c
index 2ccd3eedbd67..2deeb9babc21 100644
--- a/drivers/input/input-compat.c
+++ b/drivers/input/input-compat.c
@@ -75,6 +75,8 @@ int input_ff_effect_from_user(const char __user *buffer, size_t size,
*/
compat_effect = (struct ff_effect_compat *)effect;
+ memset(effect, 0, sizeof(*effect));
+
if (copy_from_user(compat_effect, buffer,
sizeof(struct ff_effect_compat)))
return -EFAULT;
diff --git a/drivers/input/joystick/xpad.c b/drivers/input/joystick/xpad.c
index 53ab665e0ba5..2c1b1cabb38f 100644
--- a/drivers/input/joystick/xpad.c
+++ b/drivers/input/joystick/xpad.c
@@ -244,7 +244,7 @@ static const struct xpad_device {
{ 0x0e6f, 0x0139, "Afterglow Prismatic Wired Controller", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x013a, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x0146, "Rock Candy Wired Controller for Xbox One", 0, XTYPE_XBOXONE },
- { 0x0e6f, 0x0147, "PDP Marvel Xbox One Controller", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0147, "PDP Marvel Xbox 360 Controller", 0, XTYPE_XBOX360 },
{ 0x0e6f, 0x015c, "PDP Xbox One Arcade Stick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE },
{ 0x0e6f, 0x015d, "PDP Mirror's Edge Official Wired Controller for Xbox One", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x0161, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
@@ -256,6 +256,7 @@ static const struct xpad_device {
{ 0x0e6f, 0x0213, "Afterglow Gamepad for Xbox 360", 0, XTYPE_XBOX360 },
{ 0x0e6f, 0x021f, "Rock Candy Gamepad for Xbox 360", 0, XTYPE_XBOX360 },
{ 0x0e6f, 0x0246, "Rock Candy Gamepad for Xbox One 2015", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x024c, "PDP Victrix Pro BFG Wired Controller for Xbox", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x02a0, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x02a1, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x02a2, "PDP Wired Controller for Xbox One - Crimson Red", 0, XTYPE_XBOXONE },
@@ -320,6 +321,12 @@ static const struct xpad_device {
{ 0x1689, 0xfd00, "Razer Onza Tournament Edition", 0, XTYPE_XBOX360 },
{ 0x1689, 0xfd01, "Razer Onza Classic Edition", 0, XTYPE_XBOX360 },
{ 0x1689, 0xfe00, "Razer Sabertooth", 0, XTYPE_XBOX360 },
+ { 0x16d0, 0x1103, "Azeron Cyro", 0, XTYPE_XBOX360 },
+ { 0x16d0, 0x113c, "Azeron Cyborg", 0, XTYPE_XBOX360 },
+ { 0x16d0, 0x1192, "Azeron Classic/Compact", 0, XTYPE_XBOX360 },
+ { 0x16d0, 0x1212, "Azeron Cyro Lefty", 0, XTYPE_XBOX360 },
+ { 0x16d0, 0x12f7, "Azeron Cyborg II", 0, XTYPE_XBOX360 },
+ { 0x16d0, 0x13ea, "Azeron Keyzen", 0, XTYPE_XBOX360 },
{ 0x17ef, 0x6182, "Lenovo Legion Controller for Windows", 0, XTYPE_XBOX360 },
{ 0x1949, 0x041a, "Amazon Game Controller", 0, XTYPE_XBOX360 },
{ 0x1a86, 0xe310, "Legion Go S", 0, XTYPE_XBOX360 },
@@ -554,6 +561,7 @@ static const struct usb_device_id xpad_table[] = {
XPAD_XBOX360_VENDOR(0x15e4), /* Numark Xbox 360 controllers */
XPAD_XBOX360_VENDOR(0x162e), /* Joytech Xbox 360 controllers */
XPAD_XBOX360_VENDOR(0x1689), /* Razer Onza */
+ XPAD_XBOX360_VENDOR(0x16d0), /* Azeron controllers */
XPAD_XBOX360_VENDOR(0x17ef), /* Lenovo */
XPAD_XBOX360_VENDOR(0x1949), /* Amazon controllers */
XPAD_XBOX360_VENDOR(0x1a86), /* Nanjing Qinheng Microelectronics (WCH) */
diff --git a/drivers/input/keyboard/adp5588-keys.c b/drivers/input/keyboard/adp5588-keys.c
index dc734974ce06..9a8fc61b5b89 100644
--- a/drivers/input/keyboard/adp5588-keys.c
+++ b/drivers/input/keyboard/adp5588-keys.c
@@ -446,12 +446,6 @@ static int adp5588_gpio_add(struct adp5588_kpad *kpad)
mutex_init(&kpad->gpio_lock);
- error = devm_gpiochip_add_data(dev, &kpad->gc, kpad);
- if (error) {
- dev_err(dev, "gpiochip_add failed: %d\n", error);
- return error;
- }
-
for (i = 0; i <= ADP5588_BANK(ADP5588_MAXGPIO); i++) {
kpad->dat_out[i] = adp5588_read(kpad->client,
GPIO_DAT_OUT1 + i);
@@ -459,6 +453,12 @@ static int adp5588_gpio_add(struct adp5588_kpad *kpad)
kpad->pull_dis[i] = adp5588_read(kpad->client, GPIO_PULL1 + i);
}
+ error = devm_gpiochip_add_data(dev, &kpad->gc, kpad);
+ if (error) {
+ dev_err(dev, "gpiochip_add failed: %d\n", error);
+ return error;
+ }
+
return 0;
}
diff --git a/drivers/input/keyboard/atkbd.c b/drivers/input/keyboard/atkbd.c
index 8eea27722b6e..f90061584ab9 100644
--- a/drivers/input/keyboard/atkbd.c
+++ b/drivers/input/keyboard/atkbd.c
@@ -1975,6 +1975,14 @@ static const struct dmi_system_id atkbd_dmi_quirk_table[] __initconst = {
},
.callback = atkbd_deactivate_fixup,
},
+ {
+ /* Xiaomi Redmi Book Pro 16 2026 (TM2425) */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "XIAOMI"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "REDMI Book Pro 16 2026"),
+ },
+ .callback = atkbd_deactivate_fixup,
+ },
{ }
};
diff --git a/drivers/input/misc/soc_button_array.c b/drivers/input/misc/soc_button_array.c
index 5c5d407fe965..3b8cf62b9529 100644
--- a/drivers/input/misc/soc_button_array.c
+++ b/drivers/input/misc/soc_button_array.c
@@ -16,6 +16,7 @@
#include <linux/gpio/consumer.h>
#include <linux/gpio_keys.h>
#include <linux/gpio.h>
+#include <linux/platform_data/x86/soc.h>
#include <linux/platform_device.h>
static bool use_low_level_irq;
@@ -160,7 +161,7 @@ soc_button_device_create(struct platform_device *pdev,
struct gpio_keys_platform_data *gpio_keys_pdata;
const struct dmi_system_id *dmi_id;
int invalid_acpi_index = -1;
- int error, gpio, irq;
+ int error, gpio, irq = 0;
int n_buttons = 0;
for (info = button_info; info->name; info++)
@@ -191,8 +192,9 @@ soc_button_device_create(struct platform_device *pdev,
error = soc_button_lookup_gpio(&pdev->dev, info->acpi_index, &gpio, &irq);
if (error || irq < 0) {
/*
- * Skip GPIO if not present. Note we deliberately
- * ignore -EPROBE_DEFER errors here. On some devices
+ * Propagate -EPROBE_DEFER, skip button on other errors.
+ *
+ * -EPROBE_DEFER is ignored on Bay & Cherry Trail. Here
* Intel is using so called virtual GPIOs which are not
* GPIOs at all but some way for AML code to check some
* random status bits without need a custom opregion.
@@ -201,6 +203,12 @@ soc_button_device_create(struct platform_device *pdev,
* we do not have a driver for these so they will never
* show up, therefore we ignore -EPROBE_DEFER.
*/
+ if ((error == -EPROBE_DEFER || irq == -EPROBE_DEFER) &&
+ !(soc_intel_is_byt() || soc_intel_is_cht())) {
+ error = -EPROBE_DEFER;
+ goto err_free_mem;
+ }
+
continue;
}
@@ -369,7 +377,7 @@ static struct soc_button_info *soc_button_get_button_info(struct device *dev)
}
}
- if (!btns_desc) {
+ if (!btns_desc || !btns_desc->package.count) {
dev_err(dev, "ACPI Button Descriptors not found\n");
button_info = ERR_PTR(-ENODEV);
goto out;
diff --git a/drivers/input/mouse/synaptics.c b/drivers/input/mouse/synaptics.c
index edd4bf4670ad..961e9fca5483 100644
--- a/drivers/input/mouse/synaptics.c
+++ b/drivers/input/mouse/synaptics.c
@@ -1837,6 +1837,14 @@ static int synaptics_setup_intertouch(struct psmouse *psmouse,
return -ENXIO;
}
+
+ /* Disable intertouch on known-broken board revisions */
+ if (info->board_id == 2722) {
+ psmouse_info(psmouse,
+ "Disabling intertouch for board id %u\n",
+ info->board_id);
+ return -ENXIO;
+ }
}
psmouse_info(psmouse, "Trying to set up SMBus access\n");
diff --git a/drivers/input/rmi4/rmi_driver.c b/drivers/input/rmi4/rmi_driver.c
index 2168b6cd7167..092852dadd07 100644
--- a/drivers/input/rmi4/rmi_driver.c
+++ b/drivers/input/rmi4/rmi_driver.c
@@ -946,6 +946,15 @@ int rmi_driver_suspend(struct rmi_device *rmi_dev, bool enable_wake)
{
int retval;
+ /*
+ * Transport driver will try to suspend RMI device even if physical
+ * driver did not bind to the RMI device, because transport device
+ * (I2C, SPI) is fully registered and operational. Exit early if
+ * there is no driver data attached to the RMI device.
+ */
+ if (!dev_get_drvdata(&rmi_dev->dev))
+ return 0;
+
retval = rmi_suspend_functions(rmi_dev);
if (retval)
dev_warn(&rmi_dev->dev, "Failed to suspend functions: %d\n",
@@ -960,6 +969,10 @@ int rmi_driver_resume(struct rmi_device *rmi_dev, bool clear_wake)
{
int retval;
+ /* Skip if not fully bound to RMI driver */
+ if (!dev_get_drvdata(&rmi_dev->dev))
+ return 0;
+
rmi_enable_irq(rmi_dev, clear_wake);
retval = rmi_resume_functions(rmi_dev);
diff --git a/drivers/input/rmi4/rmi_smbus.c b/drivers/input/rmi4/rmi_smbus.c
index f3d0b40721df..8118e755b24a 100644
--- a/drivers/input/rmi4/rmi_smbus.c
+++ b/drivers/input/rmi4/rmi_smbus.c
@@ -140,7 +140,7 @@ static int rmi_smb_write_block(struct rmi_transport_dev *xport, u16 rmiaddr,
u8 commandcode;
struct rmi_smb_xport *rmi_smb =
container_of(xport, struct rmi_smb_xport, xport);
- int cur_len = (int)len;
+ size_t cur_len = len;
mutex_lock(&rmi_smb->page_mutex);
@@ -148,7 +148,7 @@ static int rmi_smb_write_block(struct rmi_transport_dev *xport, u16 rmiaddr,
/*
* break into 32 bytes chunks to write get command code
*/
- int block_len = min_t(int, len, SMB_MAX_COUNT);
+ int block_len = min_t(size_t, cur_len, SMB_MAX_COUNT);
retval = rmi_smb_get_command_code(xport, rmiaddr, block_len,
false, &commandcode);
@@ -161,9 +161,9 @@ static int rmi_smb_write_block(struct rmi_transport_dev *xport, u16 rmiaddr,
goto exit;
/* prepare to write next block of bytes */
- cur_len -= SMB_MAX_COUNT;
- databuff += SMB_MAX_COUNT;
- rmiaddr += SMB_MAX_COUNT;
+ cur_len -= block_len;
+ databuff += block_len;
+ rmiaddr += block_len;
}
exit:
mutex_unlock(&rmi_smb->page_mutex);
diff --git a/drivers/input/serio/hp_sdc.c b/drivers/input/serio/hp_sdc.c
index 13eacf6ab431..ee096d7c7846 100644
--- a/drivers/input/serio/hp_sdc.c
+++ b/drivers/input/serio/hp_sdc.c
@@ -980,7 +980,7 @@ static void hp_sdc_exit(void)
free_irq(hp_sdc.irq, &hp_sdc);
write_unlock_irq(&hp_sdc.lock);
- del_timer_sync(&hp_sdc.kicker);
+ timer_shutdown_sync(&hp_sdc.kicker);
tasklet_kill(&hp_sdc.task);
diff --git a/drivers/input/serio/i8042-acpipnpio.h b/drivers/input/serio/i8042-acpipnpio.h
index 0cf55ec6d0b4..37f7fcd8d88c 100644
--- a/drivers/input/serio/i8042-acpipnpio.h
+++ b/drivers/input/serio/i8042-acpipnpio.h
@@ -258,6 +258,13 @@ static const struct dmi_system_id i8042_dmi_quirk_table[] __initconst = {
},
.driver_data = (void *)(SERIO_QUIRK_RESET_ALWAYS)
},
+ {
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Acer"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Aspire AG15-42P"),
+ },
+ .driver_data = (void *)(SERIO_QUIRK_RESET_ALWAYS)
+ },
{
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Acer"),
diff --git a/drivers/input/touchscreen/cyttsp5.c b/drivers/input/touchscreen/cyttsp5.c
index 071b7c9bf566..1b48cf2e6895 100644
--- a/drivers/input/touchscreen/cyttsp5.c
+++ b/drivers/input/touchscreen/cyttsp5.c
@@ -711,6 +711,7 @@ static irqreturn_t cyttsp5_handle_irq(int irq, void *handle)
size = 2;
} else {
report_id = ts->input_buf[2];
+ size = min(size, CY_MAX_INPUT);
}
switch (report_id) {
diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c
index 494976d13bf4..5dc9fa269e3a 100644
--- a/drivers/iommu/iommufd/device.c
+++ b/drivers/iommu/iommufd/device.c
@@ -367,7 +367,8 @@ static bool iommufd_device_is_attached(struct iommufd_device *idev)
}
static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt,
- struct iommufd_device *idev)
+ struct iommufd_device *idev,
+ ioasid_t pasid)
{
struct iommufd_attach_handle *handle;
int rc;
@@ -385,6 +386,7 @@ static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt,
}
handle->idev = idev;
+ WARN_ON(pasid != IOMMU_NO_PASID);
rc = iommu_attach_group_handle(hwpt->domain, idev->igroup->group,
&handle->handle);
if (rc)
@@ -401,25 +403,28 @@ static int iommufd_hwpt_attach_device(struct iommufd_hw_pagetable *hwpt,
}
static struct iommufd_attach_handle *
-iommufd_device_get_attach_handle(struct iommufd_device *idev)
+iommufd_device_get_attach_handle(struct iommufd_device *idev, ioasid_t pasid)
{
struct iommu_attach_handle *handle;
lockdep_assert_held(&idev->igroup->lock);
handle =
- iommu_attach_handle_get(idev->igroup->group, IOMMU_NO_PASID, 0);
+ iommu_attach_handle_get(idev->igroup->group, pasid, 0);
if (IS_ERR(handle))
return NULL;
return to_iommufd_handle(handle);
}
static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt,
- struct iommufd_device *idev)
+ struct iommufd_device *idev,
+ ioasid_t pasid)
{
struct iommufd_attach_handle *handle;
- handle = iommufd_device_get_attach_handle(idev);
+ WARN_ON(pasid != IOMMU_NO_PASID);
+
+ handle = iommufd_device_get_attach_handle(idev, pasid);
iommu_detach_group_handle(hwpt->domain, idev->igroup->group);
if (hwpt->fault) {
iommufd_auto_response_faults(hwpt, handle);
@@ -429,13 +434,17 @@ static void iommufd_hwpt_detach_device(struct iommufd_hw_pagetable *hwpt,
}
static int iommufd_hwpt_replace_device(struct iommufd_device *idev,
+ ioasid_t pasid,
struct iommufd_hw_pagetable *hwpt,
struct iommufd_hw_pagetable *old)
{
- struct iommufd_attach_handle *handle, *old_handle =
- iommufd_device_get_attach_handle(idev);
+ struct iommufd_attach_handle *handle, *old_handle;
int rc;
+ WARN_ON(pasid != IOMMU_NO_PASID);
+
+ old_handle = iommufd_device_get_attach_handle(idev, pasid);
+
handle = kzalloc(sizeof(*handle), GFP_KERNEL);
if (!handle)
return -ENOMEM;
@@ -470,7 +479,7 @@ static int iommufd_hwpt_replace_device(struct iommufd_device *idev,
}
int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
- struct iommufd_device *idev)
+ struct iommufd_device *idev, ioasid_t pasid)
{
struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
int rc;
@@ -496,7 +505,7 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
* attachment.
*/
if (list_empty(&idev->igroup->device_list)) {
- rc = iommufd_hwpt_attach_device(hwpt, idev);
+ rc = iommufd_hwpt_attach_device(hwpt, idev, pasid);
if (rc)
goto err_unresv;
idev->igroup->hwpt = hwpt;
@@ -514,7 +523,7 @@ int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
}
struct iommufd_hw_pagetable *
-iommufd_hw_pagetable_detach(struct iommufd_device *idev)
+iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid)
{
struct iommufd_hw_pagetable *hwpt = idev->igroup->hwpt;
struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
@@ -522,7 +531,7 @@ iommufd_hw_pagetable_detach(struct iommufd_device *idev)
mutex_lock(&idev->igroup->lock);
list_del(&idev->group_item);
if (list_empty(&idev->igroup->device_list)) {
- iommufd_hwpt_detach_device(hwpt, idev);
+ iommufd_hwpt_detach_device(hwpt, idev, pasid);
idev->igroup->hwpt = NULL;
}
if (hwpt_paging)
@@ -534,12 +543,12 @@ iommufd_hw_pagetable_detach(struct iommufd_device *idev)
}
static struct iommufd_hw_pagetable *
-iommufd_device_do_attach(struct iommufd_device *idev,
+iommufd_device_do_attach(struct iommufd_device *idev, ioasid_t pasid,
struct iommufd_hw_pagetable *hwpt)
{
int rc;
- rc = iommufd_hw_pagetable_attach(hwpt, idev);
+ rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid);
if (rc)
return ERR_PTR(rc);
return NULL;
@@ -588,7 +597,7 @@ iommufd_group_do_replace_reserved_iova(struct iommufd_group *igroup,
}
static struct iommufd_hw_pagetable *
-iommufd_device_do_replace(struct iommufd_device *idev,
+iommufd_device_do_replace(struct iommufd_device *idev, ioasid_t pasid,
struct iommufd_hw_pagetable *hwpt)
{
struct iommufd_hwpt_paging *hwpt_paging = find_hwpt_paging(hwpt);
@@ -622,7 +631,7 @@ iommufd_device_do_replace(struct iommufd_device *idev,
goto err_unlock;
}
- rc = iommufd_hwpt_replace_device(idev, hwpt, old_hwpt);
+ rc = iommufd_hwpt_replace_device(idev, pasid, hwpt, old_hwpt);
if (rc)
goto err_unresv;
@@ -655,7 +664,8 @@ iommufd_device_do_replace(struct iommufd_device *idev,
}
typedef struct iommufd_hw_pagetable *(*attach_fn)(
- struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt);
+ struct iommufd_device *idev, ioasid_t pasid,
+ struct iommufd_hw_pagetable *hwpt);
/*
* When automatically managing the domains we search for a compatible domain in
@@ -663,7 +673,7 @@ typedef struct iommufd_hw_pagetable *(*attach_fn)(
* Automatic domain selection will never pick a manually created domain.
*/
static struct iommufd_hw_pagetable *
-iommufd_device_auto_get_domain(struct iommufd_device *idev,
+iommufd_device_auto_get_domain(struct iommufd_device *idev, ioasid_t pasid,
struct iommufd_ioas *ioas, u32 *pt_id,
attach_fn do_attach)
{
@@ -692,7 +702,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev,
hwpt = &hwpt_paging->common;
if (!iommufd_lock_obj(&hwpt->obj))
continue;
- destroy_hwpt = (*do_attach)(idev, hwpt);
+ destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
if (IS_ERR(destroy_hwpt)) {
iommufd_put_object(idev->ictx, &hwpt->obj);
/*
@@ -710,8 +720,8 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev,
goto out_unlock;
}
- hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, 0,
- immediate_attach, NULL);
+ hwpt_paging = iommufd_hwpt_paging_alloc(idev->ictx, ioas, idev, pasid,
+ 0, immediate_attach, NULL);
if (IS_ERR(hwpt_paging)) {
destroy_hwpt = ERR_CAST(hwpt_paging);
goto out_unlock;
@@ -719,7 +729,7 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev,
hwpt = &hwpt_paging->common;
if (!immediate_attach) {
- destroy_hwpt = (*do_attach)(idev, hwpt);
+ destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
if (IS_ERR(destroy_hwpt))
goto out_abort;
} else {
@@ -740,8 +750,9 @@ iommufd_device_auto_get_domain(struct iommufd_device *idev,
return destroy_hwpt;
}
-static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,
- attach_fn do_attach)
+static int iommufd_device_change_pt(struct iommufd_device *idev,
+ ioasid_t pasid,
+ u32 *pt_id, attach_fn do_attach)
{
struct iommufd_hw_pagetable *destroy_hwpt;
struct iommufd_object *pt_obj;
@@ -756,7 +767,7 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,
struct iommufd_hw_pagetable *hwpt =
container_of(pt_obj, struct iommufd_hw_pagetable, obj);
- destroy_hwpt = (*do_attach)(idev, hwpt);
+ destroy_hwpt = (*do_attach)(idev, pasid, hwpt);
if (IS_ERR(destroy_hwpt))
goto out_put_pt_obj;
break;
@@ -765,8 +776,8 @@ static int iommufd_device_change_pt(struct iommufd_device *idev, u32 *pt_id,
struct iommufd_ioas *ioas =
container_of(pt_obj, struct iommufd_ioas, obj);
- destroy_hwpt = iommufd_device_auto_get_domain(idev, ioas, pt_id,
- do_attach);
+ destroy_hwpt = iommufd_device_auto_get_domain(idev, pasid, ioas,
+ pt_id, do_attach);
if (IS_ERR(destroy_hwpt))
goto out_put_pt_obj;
break;
@@ -803,7 +814,8 @@ int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id)
{
int rc;
- rc = iommufd_device_change_pt(idev, pt_id, &iommufd_device_do_attach);
+ rc = iommufd_device_change_pt(idev, IOMMU_NO_PASID, pt_id,
+ &iommufd_device_do_attach);
if (rc)
return rc;
@@ -833,7 +845,7 @@ EXPORT_SYMBOL_NS_GPL(iommufd_device_attach, IOMMUFD);
*/
int iommufd_device_replace(struct iommufd_device *idev, u32 *pt_id)
{
- return iommufd_device_change_pt(idev, pt_id,
+ return iommufd_device_change_pt(idev, IOMMU_NO_PASID, pt_id,
&iommufd_device_do_replace);
}
EXPORT_SYMBOL_NS_GPL(iommufd_device_replace, IOMMUFD);
@@ -849,7 +861,7 @@ void iommufd_device_detach(struct iommufd_device *idev)
{
struct iommufd_hw_pagetable *hwpt;
- hwpt = iommufd_hw_pagetable_detach(idev);
+ hwpt = iommufd_hw_pagetable_detach(idev, IOMMU_NO_PASID);
iommufd_hw_pagetable_put(idev->ictx, hwpt);
refcount_dec(&idev->obj.users);
}
diff --git a/drivers/iommu/iommufd/hw_pagetable.c b/drivers/iommu/iommufd/hw_pagetable.c
index 1d080f99d045..53e31470a138 100644
--- a/drivers/iommu/iommufd/hw_pagetable.c
+++ b/drivers/iommu/iommufd/hw_pagetable.c
@@ -87,6 +87,7 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging)
* @ictx: iommufd context
* @ioas: IOAS to associate the domain with
* @idev: Device to get an iommu_domain for
+ * @pasid: PASID to get an iommu_domain for
* @flags: Flags from userspace
* @immediate_attach: True if idev should be attached to the hwpt
* @user_data: The user provided driver specific data describing the domain to
@@ -102,8 +103,8 @@ iommufd_hwpt_paging_enforce_cc(struct iommufd_hwpt_paging *hwpt_paging)
*/
struct iommufd_hwpt_paging *
iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
- struct iommufd_device *idev, u32 flags,
- bool immediate_attach,
+ struct iommufd_device *idev, ioasid_t pasid,
+ u32 flags, bool immediate_attach,
const struct iommu_user_data *user_data)
{
const u32 valid_flags = IOMMU_HWPT_ALLOC_NEST_PARENT |
@@ -183,7 +184,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
* sequence. Once those drivers are fixed this should be removed.
*/
if (immediate_attach) {
- rc = iommufd_hw_pagetable_attach(hwpt, idev);
+ rc = iommufd_hw_pagetable_attach(hwpt, idev, pasid);
if (rc)
goto out_abort;
}
@@ -196,7 +197,7 @@ iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
out_detach:
if (immediate_attach)
- iommufd_hw_pagetable_detach(idev);
+ iommufd_hw_pagetable_detach(idev, pasid);
out_abort:
iommufd_object_abort_and_destroy(ictx, &hwpt->obj);
return ERR_PTR(rc);
@@ -299,8 +300,8 @@ int iommufd_hwpt_alloc(struct iommufd_ucmd *ucmd)
ioas = container_of(pt_obj, struct iommufd_ioas, obj);
mutex_lock(&ioas->mutex);
hwpt_paging = iommufd_hwpt_paging_alloc(
- ucmd->ictx, ioas, idev, cmd->flags, false,
- user_data.len ? &user_data : NULL);
+ ucmd->ictx, ioas, idev, IOMMU_NO_PASID, cmd->flags,
+ false, user_data.len ? &user_data : NULL);
if (IS_ERR(hwpt_paging)) {
rc = PTR_ERR(hwpt_paging);
goto out_unlock;
diff --git a/drivers/iommu/iommufd/iommufd_private.h b/drivers/iommu/iommufd/iommufd_private.h
index 18cdf1391a03..30476b7cecdc 100644
--- a/drivers/iommu/iommufd/iommufd_private.h
+++ b/drivers/iommu/iommufd/iommufd_private.h
@@ -365,13 +365,13 @@ int iommufd_hwpt_get_dirty_bitmap(struct iommufd_ucmd *ucmd);
struct iommufd_hwpt_paging *
iommufd_hwpt_paging_alloc(struct iommufd_ctx *ictx, struct iommufd_ioas *ioas,
- struct iommufd_device *idev, u32 flags,
- bool immediate_attach,
+ struct iommufd_device *idev, ioasid_t pasid,
+ u32 flags, bool immediate_attach,
const struct iommu_user_data *user_data);
int iommufd_hw_pagetable_attach(struct iommufd_hw_pagetable *hwpt,
- struct iommufd_device *idev);
+ struct iommufd_device *idev, ioasid_t pasid);
struct iommufd_hw_pagetable *
-iommufd_hw_pagetable_detach(struct iommufd_device *idev);
+iommufd_hw_pagetable_detach(struct iommufd_device *idev, ioasid_t pasid);
void iommufd_hwpt_paging_destroy(struct iommufd_object *obj);
void iommufd_hwpt_paging_abort(struct iommufd_object *obj);
void iommufd_hwpt_nested_destroy(struct iommufd_object *obj);
diff --git a/drivers/iommu/iommufd/selftest.c b/drivers/iommu/iommufd/selftest.c
index 8ec251acd923..6a039fa82f6e 100644
--- a/drivers/iommu/iommufd/selftest.c
+++ b/drivers/iommu/iommufd/selftest.c
@@ -139,6 +139,7 @@ enum selftest_obj_type {
struct mock_dev {
struct device dev;
+ struct rw_semaphore iopf_rwsem;
unsigned long flags;
int id;
};
@@ -702,6 +703,7 @@ static struct mock_dev *mock_dev_create(unsigned long dev_flags)
if (!mdev)
return ERR_PTR(-ENOMEM);
+ init_rwsem(&mdev->iopf_rwsem);
device_initialize(&mdev->dev);
mdev->flags = dev_flags;
mdev->dev.release = mock_dev_release;
@@ -716,7 +718,9 @@ static struct mock_dev *mock_dev_create(unsigned long dev_flags)
if (rc)
goto err_put;
+ down_write(&mdev->iopf_rwsem);
rc = device_add(&mdev->dev);
+ up_write(&mdev->iopf_rwsem);
if (rc)
goto err_put;
return mdev;
@@ -771,7 +775,9 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd,
}
sobj->idev.idev = idev;
+ down_write(&sobj->idev.mock_dev->iopf_rwsem);
rc = iommufd_device_attach(idev, &pt_id);
+ up_write(&sobj->idev.mock_dev->iopf_rwsem);
if (rc)
goto out_unbind;
@@ -786,7 +792,9 @@ static int iommufd_test_mock_domain(struct iommufd_ucmd *ucmd,
return 0;
out_detach:
+ down_write(&sobj->idev.mock_dev->iopf_rwsem);
iommufd_device_detach(idev);
+ up_write(&sobj->idev.mock_dev->iopf_rwsem);
out_unbind:
iommufd_device_unbind(idev);
out_mdev:
@@ -820,7 +828,9 @@ static int iommufd_test_mock_domain_replace(struct iommufd_ucmd *ucmd,
goto out_dev_obj;
}
+ down_write(&sobj->idev.mock_dev->iopf_rwsem);
rc = iommufd_device_replace(sobj->idev.idev, &pt_id);
+ up_write(&sobj->idev.mock_dev->iopf_rwsem);
if (rc)
goto out_dev_obj;
@@ -1418,10 +1428,16 @@ static int iommufd_test_trigger_iopf(struct iommufd_ucmd *ucmd,
{
struct iopf_fault event = { };
struct iommufd_device *idev;
+ struct mock_dev *mdev;
idev = iommufd_get_device(ucmd, cmd->trigger_iopf.dev_id);
if (IS_ERR(idev))
return PTR_ERR(idev);
+ if (!iommufd_selftest_is_mock_dev(idev->dev)) {
+ iommufd_put_object(ucmd->ictx, &idev->obj);
+ return -EINVAL;
+ }
+ mdev = container_of(idev->dev, struct mock_dev, dev);
event.fault.prm.flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE;
if (cmd->trigger_iopf.pasid != IOMMU_NO_PASID)
@@ -1432,7 +1448,9 @@ static int iommufd_test_trigger_iopf(struct iommufd_ucmd *ucmd,
event.fault.prm.grpid = cmd->trigger_iopf.grpid;
event.fault.prm.perm = cmd->trigger_iopf.perm;
+ down_read(&mdev->iopf_rwsem);
iommu_report_device_fault(idev->dev, &event);
+ up_read(&mdev->iopf_rwsem);
iommufd_put_object(ucmd->ictx, &idev->obj);
return 0;
@@ -1444,7 +1462,9 @@ void iommufd_selftest_destroy(struct iommufd_object *obj)
switch (sobj->type) {
case TYPE_IDEV:
+ down_write(&sobj->idev.mock_dev->iopf_rwsem);
iommufd_device_detach(sobj->idev.idev);
+ up_write(&sobj->idev.mock_dev->iopf_rwsem);
iommufd_device_unbind(sobj->idev.idev);
mock_dev_destroy(sobj->idev.mock_dev);
break;
diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c
index 036bacd162c7..739ac09f3a62 100644
--- a/drivers/iommu/msm_iommu.c
+++ b/drivers/iommu/msm_iommu.c
@@ -725,47 +725,32 @@ static int msm_iommu_probe(struct platform_device *pdev)
iommu->dev = &pdev->dev;
INIT_LIST_HEAD(&iommu->ctx_list);
- iommu->pclk = devm_clk_get(iommu->dev, "smmu_pclk");
+ iommu->pclk = devm_clk_get_prepared(iommu->dev, "smmu_pclk");
if (IS_ERR(iommu->pclk))
return dev_err_probe(iommu->dev, PTR_ERR(iommu->pclk),
"could not get smmu_pclk\n");
- ret = clk_prepare(iommu->pclk);
- if (ret)
- return dev_err_probe(iommu->dev, ret,
- "could not prepare smmu_pclk\n");
-
- iommu->clk = devm_clk_get(iommu->dev, "iommu_clk");
- if (IS_ERR(iommu->clk)) {
- clk_unprepare(iommu->pclk);
+ iommu->clk = devm_clk_get_prepared(iommu->dev, "iommu_clk");
+ if (IS_ERR(iommu->clk))
return dev_err_probe(iommu->dev, PTR_ERR(iommu->clk),
"could not get iommu_clk\n");
- }
-
- ret = clk_prepare(iommu->clk);
- if (ret) {
- clk_unprepare(iommu->pclk);
- return dev_err_probe(iommu->dev, ret, "could not prepare iommu_clk\n");
- }
r = platform_get_resource(pdev, IORESOURCE_MEM, 0);
iommu->base = devm_ioremap_resource(iommu->dev, r);
if (IS_ERR(iommu->base)) {
ret = dev_err_probe(iommu->dev, PTR_ERR(iommu->base), "could not get iommu base\n");
- goto fail;
+ return ret;
}
ioaddr = r->start;
iommu->irq = platform_get_irq(pdev, 0);
- if (iommu->irq < 0) {
- ret = -ENODEV;
- goto fail;
- }
+ if (iommu->irq < 0)
+ return -ENODEV;
ret = of_property_read_u32(iommu->dev->of_node, "qcom,ncb", &val);
if (ret) {
dev_err(iommu->dev, "could not get ncb\n");
- goto fail;
+ return ret;
}
iommu->ncb = val;
@@ -780,8 +765,7 @@ static int msm_iommu_probe(struct platform_device *pdev)
if (!par) {
pr_err("Invalid PAR value detected\n");
- ret = -ENODEV;
- goto fail;
+ return -ENODEV;
}
ret = devm_request_threaded_irq(iommu->dev, iommu->irq, NULL,
@@ -791,7 +775,7 @@ static int msm_iommu_probe(struct platform_device *pdev)
iommu);
if (ret) {
pr_err("Request IRQ %d failed with ret=%d\n", iommu->irq, ret);
- goto fail;
+ return ret;
}
list_add(&iommu->dev_node, &qcom_iommu_devices);
@@ -800,22 +784,24 @@ static int msm_iommu_probe(struct platform_device *pdev)
"msm-smmu.%pa", &ioaddr);
if (ret) {
pr_err("Could not add msm-smmu at %pa to sysfs\n", &ioaddr);
- goto fail;
+ goto err_remove_list;
}
ret = iommu_device_register(&iommu->iommu, &msm_iommu_ops, &pdev->dev);
if (ret) {
pr_err("Could not register msm-smmu at %pa\n", &ioaddr);
- goto fail;
+ goto err_remove_sysfs;
}
pr_info("device mapped at %p, irq %d with %d ctx banks\n",
iommu->base, iommu->irq, iommu->ncb);
return ret;
-fail:
- clk_unprepare(iommu->clk);
- clk_unprepare(iommu->pclk);
+
+err_remove_sysfs:
+ iommu_device_sysfs_remove(&iommu->iommu);
+err_remove_list:
+ list_del(&iommu->dev_node);
return ret;
}
@@ -824,20 +810,11 @@ static const struct of_device_id msm_iommu_dt_match[] = {
{}
};
-static void msm_iommu_remove(struct platform_device *pdev)
-{
- struct msm_iommu_dev *iommu = platform_get_drvdata(pdev);
-
- clk_unprepare(iommu->clk);
- clk_unprepare(iommu->pclk);
-}
-
static struct platform_driver msm_iommu_driver = {
.driver = {
.name = "msm_iommu",
.of_match_table = msm_iommu_dt_match,
},
.probe = msm_iommu_probe,
- .remove_new = msm_iommu_remove,
};
builtin_platform_driver(msm_iommu_driver);
diff --git a/drivers/media/i2c/imx355.c b/drivers/media/i2c/imx355.c
index b2dce67c0b6b..6cfd543817c5 100644
--- a/drivers/media/i2c/imx355.c
+++ b/drivers/media/i2c/imx355.c
@@ -1,11 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (C) 2018 Intel Corporation
-#include <linux/unaligned.h>
#include <linux/acpi.h>
#include <linux/i2c.h>
#include <linux/module.h>
#include <linux/pm_runtime.h>
+#include <linux/unaligned.h>
+
#include <media/v4l2-ctrls.h>
#include <media/v4l2-device.h>
#include <media/v4l2-event.h>
@@ -97,6 +98,8 @@ struct imx355_hwcfg {
};
struct imx355 {
+ struct device *dev;
+
struct v4l2_subdev sd;
struct media_pad pad;
@@ -1136,14 +1139,13 @@ static int imx355_write_reg(struct imx355 *imx355, u16 reg, u32 len, u32 val)
static int imx355_write_regs(struct imx355 *imx355,
const struct imx355_reg *regs, u32 len)
{
- struct i2c_client *client = v4l2_get_subdevdata(&imx355->sd);
int ret;
u32 i;
for (i = 0; i < len; i++) {
ret = imx355_write_reg(imx355, regs[i].address, 1, regs[i].val);
if (ret) {
- dev_err_ratelimited(&client->dev,
+ dev_err_ratelimited(imx355->dev,
"write reg 0x%4.4x return err %d",
regs[i].address, ret);
@@ -1178,7 +1180,6 @@ static int imx355_set_ctrl(struct v4l2_ctrl *ctrl)
{
struct imx355 *imx355 = container_of(ctrl->handler,
struct imx355, ctrl_handler);
- struct i2c_client *client = v4l2_get_subdevdata(&imx355->sd);
s64 max;
int ret;
@@ -1197,7 +1198,7 @@ static int imx355_set_ctrl(struct v4l2_ctrl *ctrl)
* Applying V4L2 control value only happens
* when power is up for streaming
*/
- if (!pm_runtime_get_if_in_use(&client->dev))
+ if (!pm_runtime_get_if_in_use(imx355->dev))
return 0;
switch (ctrl->id) {
@@ -1231,12 +1232,12 @@ static int imx355_set_ctrl(struct v4l2_ctrl *ctrl)
break;
default:
ret = -EINVAL;
- dev_info(&client->dev, "ctrl(id:0x%x,val:0x%x) is not handled",
+ dev_info(imx355->dev, "ctrl(id:0x%x,val:0x%x) is not handled",
ctrl->id, ctrl->val);
break;
}
- pm_runtime_put(&client->dev);
+ pm_runtime_put(imx355->dev);
return ret;
}
@@ -1385,7 +1386,6 @@ imx355_set_pad_format(struct v4l2_subdev *sd,
/* Start streaming */
static int imx355_start_streaming(struct imx355 *imx355)
{
- struct i2c_client *client = v4l2_get_subdevdata(&imx355->sd);
const struct imx355_reg_list *reg_list;
int ret;
@@ -1393,7 +1393,7 @@ static int imx355_start_streaming(struct imx355 *imx355)
reg_list = &imx355_global_setting;
ret = imx355_write_regs(imx355, reg_list->regs, reg_list->num_of_regs);
if (ret) {
- dev_err(&client->dev, "failed to set global settings");
+ dev_err(imx355->dev, "failed to set global settings");
return ret;
}
@@ -1401,7 +1401,7 @@ static int imx355_start_streaming(struct imx355 *imx355)
reg_list = &imx355->cur_mode->reg_list;
ret = imx355_write_regs(imx355, reg_list->regs, reg_list->num_of_regs);
if (ret) {
- dev_err(&client->dev, "failed to set mode");
+ dev_err(imx355->dev, "failed to set mode");
return ret;
}
@@ -1429,13 +1429,12 @@ static int imx355_stop_streaming(struct imx355 *imx355)
static int imx355_set_stream(struct v4l2_subdev *sd, int enable)
{
struct imx355 *imx355 = to_imx355(sd);
- struct i2c_client *client = v4l2_get_subdevdata(sd);
int ret = 0;
mutex_lock(&imx355->mutex);
if (enable) {
- ret = pm_runtime_resume_and_get(&client->dev);
+ ret = pm_runtime_resume_and_get(imx355->dev);
if (ret < 0)
goto err_unlock;
@@ -1448,7 +1447,7 @@ static int imx355_set_stream(struct v4l2_subdev *sd, int enable)
goto err_rpm_put;
} else {
imx355_stop_streaming(imx355);
- pm_runtime_put(&client->dev);
+ pm_runtime_put(imx355->dev);
}
/* vflip and hflip cannot change during streaming */
@@ -1460,7 +1459,7 @@ static int imx355_set_stream(struct v4l2_subdev *sd, int enable)
return ret;
err_rpm_put:
- pm_runtime_put(&client->dev);
+ pm_runtime_put(imx355->dev);
err_unlock:
mutex_unlock(&imx355->mutex);
@@ -1470,7 +1469,6 @@ static int imx355_set_stream(struct v4l2_subdev *sd, int enable)
/* Verify chip ID */
static int imx355_identify_module(struct imx355 *imx355)
{
- struct i2c_client *client = v4l2_get_subdevdata(&imx355->sd);
int ret;
u32 val;
@@ -1479,7 +1477,7 @@ static int imx355_identify_module(struct imx355 *imx355)
return ret;
if (val != IMX355_CHIP_ID) {
- dev_err(&client->dev, "chip id mismatch: %x!=%x",
+ dev_err(imx355->dev, "chip id mismatch: %x!=%x",
IMX355_CHIP_ID, val);
return -EIO;
}
@@ -1519,7 +1517,6 @@ static const struct v4l2_subdev_internal_ops imx355_internal_ops = {
/* Initialize control handlers */
static int imx355_init_controls(struct imx355 *imx355)
{
- struct i2c_client *client = v4l2_get_subdevdata(&imx355->sd);
struct v4l2_fwnode_device_properties props;
struct v4l2_ctrl_handler *ctrl_hdlr;
s64 exposure_max;
@@ -1600,11 +1597,11 @@ static int imx355_init_controls(struct imx355 *imx355)
0, 0, imx355_test_pattern_menu);
if (ctrl_hdlr->error) {
ret = ctrl_hdlr->error;
- dev_err(&client->dev, "control init failed: %d", ret);
+ dev_err(imx355->dev, "control init failed: %d", ret);
goto error;
}
- ret = v4l2_fwnode_device_parse(&client->dev, &props);
+ ret = v4l2_fwnode_device_parse(imx355->dev, &props);
if (ret)
goto error;
@@ -1689,6 +1686,8 @@ static int imx355_probe(struct i2c_client *client)
if (!imx355)
return -ENOMEM;
+ imx355->dev = &client->dev;
+
mutex_init(&imx355->mutex);
/* Initialize subdev */
@@ -1697,13 +1696,13 @@ static int imx355_probe(struct i2c_client *client)
/* Check module identity */
ret = imx355_identify_module(imx355);
if (ret) {
- dev_err(&client->dev, "failed to find sensor: %d", ret);
+ dev_err(imx355->dev, "failed to find sensor: %d", ret);
goto error_probe;
}
- imx355->hwcfg = imx355_get_hwcfg(&client->dev);
+ imx355->hwcfg = imx355_get_hwcfg(imx355->dev);
if (!imx355->hwcfg) {
- dev_err(&client->dev, "failed to get hwcfg");
+ dev_err(imx355->dev, "failed to get hwcfg");
ret = -ENODEV;
goto error_probe;
}
@@ -1713,7 +1712,7 @@ static int imx355_probe(struct i2c_client *client)
ret = imx355_init_controls(imx355);
if (ret) {
- dev_err(&client->dev, "failed to init controls: %d", ret);
+ dev_err(imx355->dev, "failed to init controls: %d", ret);
goto error_probe;
}
@@ -1728,7 +1727,7 @@ static int imx355_probe(struct i2c_client *client)
imx355->pad.flags = MEDIA_PAD_FL_SOURCE;
ret = media_entity_pads_init(&imx355->sd.entity, 1, &imx355->pad);
if (ret) {
- dev_err(&client->dev, "failed to init entity pads: %d", ret);
+ dev_err(imx355->dev, "failed to init entity pads: %d", ret);
goto error_handler_free;
}
@@ -1736,19 +1735,20 @@ static int imx355_probe(struct i2c_client *client)
* Device is already turned on by i2c-core with ACPI domain PM.
* Enable runtime PM and turn off the device.
*/
- pm_runtime_set_active(&client->dev);
- pm_runtime_enable(&client->dev);
- pm_runtime_idle(&client->dev);
+ pm_runtime_set_active(imx355->dev);
+ pm_runtime_enable(imx355->dev);
ret = v4l2_async_register_subdev_sensor(&imx355->sd);
if (ret < 0)
goto error_media_entity_runtime_pm;
+ pm_runtime_idle(imx355->dev);
+
return 0;
error_media_entity_runtime_pm:
- pm_runtime_disable(&client->dev);
- pm_runtime_set_suspended(&client->dev);
+ pm_runtime_disable(imx355->dev);
+ pm_runtime_set_suspended(imx355->dev);
media_entity_cleanup(&imx355->sd.entity);
error_handler_free:
@@ -1769,8 +1769,8 @@ static void imx355_remove(struct i2c_client *client)
media_entity_cleanup(&sd->entity);
v4l2_ctrl_handler_free(sd->ctrl_handler);
- pm_runtime_disable(&client->dev);
- pm_runtime_set_suspended(&client->dev);
+ pm_runtime_disable(imx355->dev);
+ pm_runtime_set_suspended(imx355->dev);
mutex_destroy(&imx355->mutex);
}
diff --git a/drivers/media/i2c/video-i2c.c b/drivers/media/i2c/video-i2c.c
index a091fd6d3e60..a4ddc10c5f22 100644
--- a/drivers/media/i2c/video-i2c.c
+++ b/drivers/media/i2c/video-i2c.c
@@ -465,8 +465,9 @@ static int video_i2c_thread_vid_cap(void *priv)
spin_lock(&data->slock);
if (!list_empty(&data->vid_cap_active)) {
- vid_cap_buf = list_last_entry(&data->vid_cap_active,
- struct video_i2c_buffer, list);
+ vid_cap_buf = list_first_entry(&data->vid_cap_active,
+ struct video_i2c_buffer,
+ list);
list_del(&vid_cap_buf->list);
}
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
index 2f506ef24e15..fba3d7cc4cca 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-dec.c
@@ -5,6 +5,7 @@
* Copyright (C) 2021-2023 CHIPS&MEDIA INC
*/
+#include <linux/delay.h>
#include "wave5-helper.h"
#define VPU_DEC_DEV_NAME "C&M Wave5 VPU decoder"
@@ -1567,24 +1568,23 @@ static void wave5_vpu_dec_stop_streaming(struct vb2_queue *q)
{
struct vpu_instance *inst = vb2_get_drv_priv(q);
struct v4l2_m2m_ctx *m2m_ctx = inst->v4l2_fh.m2m_ctx;
- bool check_cmd = TRUE;
+ unsigned long timeout;
dev_dbg(inst->dev->dev, "%s: type: %u\n", __func__, q->type);
- while (check_cmd) {
+ timeout = jiffies + msecs_to_jiffies(VPU_DEC_STOP_TIMEOUT);
+ while (true) {
struct queue_status_info q_status;
- struct dec_output_info dec_output_info;
wave5_vpu_dec_give_command(inst, DEC_GET_QUEUE_STATUS, &q_status);
if (q_status.report_queue_count == 0)
break;
- if (wave5_vpu_wait_interrupt(inst, VPU_DEC_TIMEOUT) < 0)
+ if (time_after(jiffies, timeout))
break;
- if (wave5_vpu_dec_get_output_info(inst, &dec_output_info))
- dev_dbg(inst->dev->dev, "there is no output info\n");
+ usleep_range(1000, 2000);
}
v4l2_m2m_update_stop_streaming_state(m2m_ctx, q);
diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
index e4bc2e467cb5..b63cc6bdcd88 100644
--- a/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
+++ b/drivers/media/platform/chips-media/wave5/wave5-vpuconfig.h
@@ -40,6 +40,7 @@
// application specific configuration
#define VPU_ENC_TIMEOUT 60000
#define VPU_DEC_TIMEOUT 60000
+#define VPU_DEC_STOP_TIMEOUT 300
// for WAVE encoder
#define USE_SRC_PRP_AXI 0
diff --git a/drivers/memstick/core/ms_block.c b/drivers/memstick/core/ms_block.c
index a1a841c3bbf8..84d9b7975b3b 100644
--- a/drivers/memstick/core/ms_block.c
+++ b/drivers/memstick/core/ms_block.c
@@ -2206,6 +2206,8 @@ static void msb_remove(struct memstick_dev *card)
msb_data_clear(msb);
mutex_unlock(&msb_disk_lock);
+ destroy_workqueue(msb->io_queue);
+
put_disk(msb->disk);
memstick_set_drvdata(card, NULL);
}
diff --git a/drivers/mmc/core/bus.c b/drivers/mmc/core/bus.c
index 4f3a26676ccb..db4268a013ef 100644
--- a/drivers/mmc/core/bus.c
+++ b/drivers/mmc/core/bus.c
@@ -399,8 +399,8 @@ void mmc_remove_card(struct mmc_card *card)
mmc_hostname(card->host), card->rca);
}
device_del(&card->dev);
- of_node_put(card->dev.of_node);
}
+ of_node_put(card->dev.of_node);
if (host->cqe_enabled) {
host->cqe_ops->cqe_disable(host);
diff --git a/drivers/mmc/core/host.c b/drivers/mmc/core/host.c
index 48bda70145ee..fd423ac1c6a7 100644
--- a/drivers/mmc/core/host.c
+++ b/drivers/mmc/core/host.c
@@ -690,6 +690,7 @@ EXPORT_SYMBOL(mmc_remove_host);
void mmc_free_host(struct mmc_host *host)
{
cancel_delayed_work_sync(&host->detect);
+ cancel_work_sync(&host->sdio_irq_work);
mmc_pwrseq_free(host);
put_device(&host->class_dev);
}
diff --git a/drivers/mmc/core/sdio_uart.c b/drivers/mmc/core/sdio_uart.c
index 6b7471dba3bf..d836ceafc960 100644
--- a/drivers/mmc/core/sdio_uart.c
+++ b/drivers/mmc/core/sdio_uart.c
@@ -104,6 +104,9 @@ static int sdio_uart_add_port(struct sdio_uart_port *port)
}
spin_unlock(&sdio_uart_table_lock);
+ if (ret)
+ kfifo_free(&port->xmit_fifo);
+
return ret;
}
diff --git a/drivers/mmc/host/mmc_hsq.c b/drivers/mmc/host/mmc_hsq.c
index 79836705c176..57e172bd3487 100644
--- a/drivers/mmc/host/mmc_hsq.c
+++ b/drivers/mmc/host/mmc_hsq.c
@@ -7,6 +7,7 @@
* Author: Baolin Wang <baolin.wang@linaro.org>
*/
+#include <linux/devm-helpers.h>
#include <linux/mmc/card.h>
#include <linux/mmc/host.h>
#include <linux/module.h>
@@ -345,6 +346,7 @@ static const struct mmc_cqe_ops mmc_hsq_ops = {
int mmc_hsq_init(struct mmc_hsq *hsq, struct mmc_host *mmc)
{
+ int ret;
int i;
hsq->num_slots = HSQ_NUM_SLOTS;
hsq->next_tag = HSQ_INVALID_TAG;
@@ -363,7 +365,11 @@ int mmc_hsq_init(struct mmc_hsq *hsq, struct mmc_host *mmc)
for (i = 0; i < HSQ_NUM_SLOTS; i++)
hsq->tag_slot[i] = HSQ_INVALID_TAG;
- INIT_WORK(&hsq->retry_work, mmc_hsq_retry_handler);
+ ret = devm_work_autocancel(mmc_dev(mmc), &hsq->retry_work,
+ mmc_hsq_retry_handler);
+ if (ret)
+ return ret;
+
spin_lock_init(&hsq->lock);
init_waitqueue_head(&hsq->wait_queue);
diff --git a/drivers/mmc/host/mmc_spi.c b/drivers/mmc/host/mmc_spi.c
index f42d5f9c48c1..f502adf5a839 100644
--- a/drivers/mmc/host/mmc_spi.c
+++ b/drivers/mmc/host/mmc_spi.c
@@ -952,6 +952,7 @@ static void mmc_spi_request(struct mmc_host *mmc, struct mmc_request *mrq)
status = mmc_spi_command_send(host, mrq, &stop, 0);
crc_retry--;
mrq->data->error = 0;
+ mrq->data->bytes_xfered = 0;
goto crc_recover;
}
diff --git a/drivers/mmc/host/mmci.c b/drivers/mmc/host/mmci.c
index b790c3c3c8f9..415d2a24b3fb 100644
--- a/drivers/mmc/host/mmci.c
+++ b/drivers/mmc/host/mmci.c
@@ -2520,6 +2520,9 @@ static void mmci_remove(struct amba_device *dev)
writel(0, host->base + MMCICOMMAND);
writel(0, host->base + MMCIDATACTRL);
+ if (variant->busy_detect)
+ disable_delayed_work_sync(&host->ux500_busy_timeout_work);
+
mmci_dma_release(host);
clk_disable_unprepare(host->clk);
mmc_free_host(mmc);
diff --git a/drivers/mmc/host/mxcmmc.c b/drivers/mmc/host/mxcmmc.c
index 1edf65291354..7234f5d7681b 100644
--- a/drivers/mmc/host/mxcmmc.c
+++ b/drivers/mmc/host/mxcmmc.c
@@ -1182,6 +1182,10 @@ static void mxcmci_remove(struct platform_device *pdev)
mmc_remove_host(mmc);
+ devm_free_irq(&pdev->dev, platform_get_irq(pdev, 0), host);
+ cancel_work_sync(&host->datawork);
+ timer_delete_sync(&host->watchdog);
+
if (host->pdata && host->pdata->exit)
host->pdata->exit(&pdev->dev, mmc);
diff --git a/drivers/mmc/host/rtsx_pci_sdmmc.c b/drivers/mmc/host/rtsx_pci_sdmmc.c
index 04aa47f1a24f..555adc4b294c 100644
--- a/drivers/mmc/host/rtsx_pci_sdmmc.c
+++ b/drivers/mmc/host/rtsx_pci_sdmmc.c
@@ -1503,6 +1503,11 @@ static void realtek_init_host(struct realtek_pci_sdmmc *host)
mmc->caps = mmc->caps | MMC_CAP_AGGRESSIVE_PM;
mmc->caps2 = MMC_CAP2_NO_PRESCAN_POWERUP | MMC_CAP2_FULL_PWR_CYCLE |
MMC_CAP2_NO_SDIO;
+
+ if (pcr->pci->device == 0x522a &&
+ pcr->pci->subsystem_vendor == PCI_VENDOR_ID_LENOVO &&
+ pcr->pci->subsystem_device == 0x504a)
+ mmc->caps2 |= MMC_CAP2_NO_WRITE_PROTECT;
mmc->max_current_330 = 400;
mmc->max_current_180 = 800;
mmc->ops = &realtek_pci_sdmmc_ops;
diff --git a/drivers/mmc/host/sdhci-of-aspeed.c b/drivers/mmc/host/sdhci-of-aspeed.c
index 37240895ffaa..ff824a3e0267 100644
--- a/drivers/mmc/host/sdhci-of-aspeed.c
+++ b/drivers/mmc/host/sdhci-of-aspeed.c
@@ -561,12 +561,14 @@ static int aspeed_sdc_probe(struct platform_device *pdev)
if (!cpdev) {
of_node_put(child);
ret = -ENODEV;
- goto err_clk;
+ goto err_children;
}
}
return 0;
+err_children:
+ device_for_each_child_reverse(&pdev->dev, NULL, of_platform_device_destroy);
err_clk:
clk_disable_unprepare(sdc->clk);
return ret;
@@ -576,6 +578,7 @@ static void aspeed_sdc_remove(struct platform_device *pdev)
{
struct aspeed_sdc *sdc = dev_get_drvdata(&pdev->dev);
+ device_for_each_child_reverse(&pdev->dev, NULL, of_platform_device_destroy);
clk_disable_unprepare(sdc->clk);
}
diff --git a/drivers/mmc/host/sdhci_am654.c b/drivers/mmc/host/sdhci_am654.c
index 47344e29a4c9..6716282f5eb0 100644
--- a/drivers/mmc/host/sdhci_am654.c
+++ b/drivers/mmc/host/sdhci_am654.c
@@ -127,7 +127,7 @@ static const struct timing_data td[] = {
NULL,
MMC_CAP_UHS_SDR104},
[MMC_TIMING_UHS_DDR50] = {"ti,otap-del-sel-ddr50",
- NULL,
+ "ti,itap-del-sel-ddr50",
MMC_CAP_UHS_DDR50},
[MMC_TIMING_MMC_DDR52] = {"ti,otap-del-sel-ddr52",
"ti,itap-del-sel-ddr52",
@@ -145,6 +145,8 @@ struct sdhci_am654_data {
u32 otap_del_sel[ARRAY_SIZE(td)];
u32 itap_del_sel[ARRAY_SIZE(td)];
u32 itap_del_ena[ARRAY_SIZE(td)];
+ u32 itap_del_sel_dt_ddr50;
+ u32 itap_del_ena_dt_ddr50;
int clkbuf_sel;
int trm_icp;
int drv_strength;
@@ -152,7 +154,6 @@ struct sdhci_am654_data {
u32 flags;
u32 quirks;
bool dll_enable;
- u32 tuning_loop;
#define SDHCI_AM654_QUIRK_FORCE_CDTEST BIT(0)
#define SDHCI_AM654_QUIRK_DISABLE_HS400 BIT(1)
@@ -419,15 +420,13 @@ static int sdhci_am654_execute_tuning(struct mmc_host *mmc, u32 opcode)
struct sdhci_host *host = mmc_priv(mmc);
int err = sdhci_execute_tuning(mmc, opcode);
- if (err)
- return err;
/*
* Tuning data remains in the buffer after tuning.
* Do a command and data reset to get rid of it
*/
sdhci_reset(host, SDHCI_RESET_CMD | SDHCI_RESET_DATA);
- return 0;
+ return err;
}
static u32 sdhci_am654_cqhci_irq(struct sdhci_host *host, u32 intmask)
@@ -506,7 +505,6 @@ static int sdhci_am654_do_tuning(struct sdhci_host *host,
{
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
struct sdhci_am654_data *sdhci_am654 = sdhci_pltfm_priv(pltfm_host);
- unsigned char timing = host->mmc->ios.timing;
struct window fail_window[ITAPDLY_LENGTH];
struct device *dev = mmc_dev(host->mmc);
u8 curr_pass, itap;
@@ -515,11 +513,8 @@ static int sdhci_am654_do_tuning(struct sdhci_host *host,
memset(fail_window, 0, sizeof(fail_window));
- /* Enable ITAPDLY */
- sdhci_am654->itap_del_ena[timing] = 0x1;
-
for (itap = 0; itap < ITAPDLY_LENGTH; itap++) {
- sdhci_am654_write_itapdly(sdhci_am654, itap, sdhci_am654->itap_del_ena[timing]);
+ sdhci_am654_write_itapdly(sdhci_am654, itap, 0x1);
curr_pass = !mmc_send_tuning(host->mmc, opcode, NULL);
@@ -552,20 +547,36 @@ static int sdhci_am654_platform_execute_tuning(struct sdhci_host *host,
struct sdhci_am654_data *sdhci_am654 = sdhci_pltfm_priv(pltfm_host);
unsigned char timing = host->mmc->ios.timing;
struct device *dev = mmc_dev(host->mmc);
+ unsigned int tuning_loop = 0;
int itapdly;
do {
itapdly = sdhci_am654_do_tuning(host, opcode);
if (itapdly >= 0)
break;
- } while (++sdhci_am654->tuning_loop < RETRY_TUNING_MAX);
+ } while (++tuning_loop < RETRY_TUNING_MAX);
if (itapdly < 0) {
- dev_err(dev, "Failed to find itapdly, fail tuning\n");
+ if (timing == MMC_TIMING_UHS_DDR50) {
+ dev_dbg(dev, "Failed DDR50 tuning, fallback to DT ITAP\n");
+ sdhci_am654->itap_del_sel[timing] = sdhci_am654->itap_del_sel_dt_ddr50;
+ sdhci_am654->itap_del_ena[timing] = sdhci_am654->itap_del_ena_dt_ddr50;
+ } else {
+ dev_err(dev, "Failed to find itapdly, fail tuning\n");
+ sdhci_am654->itap_del_ena[timing] = 0;
+ sdhci_am654->itap_del_sel[timing] = 0;
+ }
+
+ sdhci_am654_write_itapdly(sdhci_am654,
+ sdhci_am654->itap_del_sel[timing],
+ sdhci_am654->itap_del_ena[timing]);
return -1;
}
dev_dbg(dev, "Passed tuning, final itapdly=%d\n", itapdly);
+
+ /* Enable ITAPDLY */
+ sdhci_am654->itap_del_ena[timing] = 0x1;
sdhci_am654_write_itapdly(sdhci_am654, itapdly, sdhci_am654->itap_del_ena[timing]);
/* Save ITAPDLY */
sdhci_am654->itap_del_sel[timing] = itapdly;
@@ -728,6 +739,11 @@ static int sdhci_am654_get_otap_delay(struct sdhci_host *host,
}
}
+ sdhci_am654->itap_del_sel_dt_ddr50 =
+ sdhci_am654->itap_del_sel[MMC_TIMING_UHS_DDR50];
+ sdhci_am654->itap_del_ena_dt_ddr50 =
+ sdhci_am654->itap_del_ena[MMC_TIMING_UHS_DDR50];
+
return 0;
}
@@ -776,9 +792,6 @@ static int sdhci_am654_init(struct sdhci_host *host)
regmap_update_bits(sdhci_am654->base, CTL_CFG_3, TUNINGFORSDR50_MASK,
TUNINGFORSDR50_MASK);
- /* Use to re-execute tuning */
- sdhci_am654->tuning_loop = 0;
-
ret = sdhci_setup_host(host);
if (ret)
return ret;
diff --git a/drivers/mmc/host/sh_mmcif.c b/drivers/mmc/host/sh_mmcif.c
index 08b4312af94e..ebd2abcaf422 100644
--- a/drivers/mmc/host/sh_mmcif.c
+++ b/drivers/mmc/host/sh_mmcif.c
@@ -1461,6 +1461,7 @@ static int sh_mmcif_probe(struct platform_device *pdev)
host->pd = pdev;
spin_lock_init(&host->lock);
+ mutex_init(&host->thread_lock);
mmc->ops = &sh_mmcif_ops;
sh_mmcif_init_ocr(host);
@@ -1521,8 +1522,6 @@ static int sh_mmcif_probe(struct platform_device *pdev)
}
}
- mutex_init(&host->thread_lock);
-
ret = mmc_add_host(mmc);
if (ret < 0)
goto err_clk;
diff --git a/drivers/mmc/host/via-sdmmc.c b/drivers/mmc/host/via-sdmmc.c
index db1feeff0faf..5f7f2e087397 100644
--- a/drivers/mmc/host/via-sdmmc.c
+++ b/drivers/mmc/host/via-sdmmc.c
@@ -1208,6 +1208,10 @@ static void via_sd_remove(struct pci_dev *pcidev)
free_irq(pcidev->irq, sdhost);
+ cancel_work_sync(&sdhost->carddet_work);
+ /* carddet_work may re-enable the interrupt via via_reset_pcictrl(). */
+ writeb(0x0, sdhost->pcictrl_mmiobase + VIA_CRDR_PCIINTCTRL);
+
del_timer_sync(&sdhost->timer);
cancel_work_sync(&sdhost->finish_bh_work);
diff --git a/drivers/mtd/nand/raw/pl35x-nand-controller.c b/drivers/mtd/nand/raw/pl35x-nand-controller.c
index 5691d78cab98..05522e81a17a 100644
--- a/drivers/mtd/nand/raw/pl35x-nand-controller.c
+++ b/drivers/mtd/nand/raw/pl35x-nand-controller.c
@@ -914,7 +914,6 @@ static int pl35x_nand_init_hw_ecc_controller(struct pl35x_nandc *nfc,
chip->ecc.steps = mtd->writesize / chip->ecc.size;
chip->ecc.read_page = pl35x_nand_read_page_hwecc;
chip->ecc.write_page = pl35x_nand_write_page_hwecc;
- chip->ecc.write_page_raw = nand_monolithic_write_page_raw;
pl35x_smc_set_ecc_pg_size(nfc, chip, mtd->writesize);
nfc->ecc_buf = devm_kmalloc(nfc->dev, chip->ecc.bytes * chip->ecc.steps,
@@ -973,15 +972,19 @@ static int pl35x_nand_attach_chip(struct nand_chip *chip)
switch (chip->ecc.engine_type) {
case NAND_ECC_ENGINE_TYPE_ON_DIE:
+ dev_dbg(nfc->dev, "Using on-die hardware ECC\n");
/* Keep these legacy BBT descriptors for ON_DIE situations */
chip->bbt_td = &bbt_main_descr;
chip->bbt_md = &bbt_mirror_descr;
fallthrough;
case NAND_ECC_ENGINE_TYPE_NONE:
+ dev_dbg(nfc->dev, "Using no ECC engine\n");
+ break;
case NAND_ECC_ENGINE_TYPE_SOFT:
- chip->ecc.write_page_raw = nand_monolithic_write_page_raw;
+ dev_dbg(nfc->dev, "Using software ECC\n");
break;
case NAND_ECC_ENGINE_TYPE_ON_HOST:
+ dev_dbg(nfc->dev, "Using on-host hardware ECC\n");
ret = pl35x_nand_init_hw_ecc_controller(nfc, chip);
if (ret)
return ret;
@@ -992,6 +995,9 @@ static int pl35x_nand_attach_chip(struct nand_chip *chip)
return -EINVAL;
}
+ chip->ecc.read_page_raw = nand_monolithic_read_page_raw;
+ chip->ecc.write_page_raw = nand_monolithic_write_page_raw;
+
return 0;
}
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 5911dcaedf0f..d10bbe4c3571 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -494,7 +494,7 @@ static int bond_ipsec_add_sa(struct xfrm_state *xs,
!real_dev->xfrmdev_ops->xdo_dev_state_add ||
netif_is_bond_master(real_dev)) {
NL_SET_ERR_MSG_MOD(extack, "Slave does not support ipsec offload");
- err = -EINVAL;
+ err = -EOPNOTSUPP;
goto out;
}
diff --git a/drivers/net/dsa/mt7530-mdio.c b/drivers/net/dsa/mt7530-mdio.c
index e19b46449ffe..9bc142e55df2 100644
--- a/drivers/net/dsa/mt7530-mdio.c
+++ b/drivers/net/dsa/mt7530-mdio.c
@@ -230,15 +230,17 @@ mt7530_remove(struct mdio_device *mdiodev)
if (!priv)
return;
- ret = regulator_disable(priv->core_pwr);
- if (ret < 0)
- dev_err(priv->dev,
- "Failed to disable core power: %d\n", ret);
-
- ret = regulator_disable(priv->io_pwr);
- if (ret < 0)
- dev_err(priv->dev, "Failed to disable io pwr: %d\n",
- ret);
+ if (priv->id == ID_MT7530) {
+ ret = regulator_disable(priv->core_pwr);
+ if (ret < 0)
+ dev_err(priv->dev,
+ "Failed to disable core power: %d\n", ret);
+
+ ret = regulator_disable(priv->io_pwr);
+ if (ret < 0)
+ dev_err(priv->dev, "Failed to disable io pwr: %d\n",
+ ret);
+ }
mt7530_remove_common(priv);
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index d958cda9e58b..b23331ea2478 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -763,25 +763,16 @@ static int ena_com_wait_and_process_admin_cq_interrupts(struct ena_comp_ctx *com
if (comp_ctx->status == ENA_CMD_COMPLETED) {
netdev_err(admin_queue->ena_dev->net_device,
- "The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d), autopolling mode is %s\n",
- comp_ctx->cmd_opcode, admin_queue->auto_polling ? "ON" : "OFF");
- /* Check if fallback to polling is enabled */
- if (admin_queue->auto_polling)
- admin_queue->polling = true;
+ "The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d)\n",
+ comp_ctx->cmd_opcode);
} else {
netdev_err(admin_queue->ena_dev->net_device,
"The ena device didn't send a completion for the admin cmd %d status %d\n",
comp_ctx->cmd_opcode, comp_ctx->status);
}
- /* Check if shifted to polling mode.
- * This will happen if there is a completion without an interrupt
- * and autopolling mode is enabled. Continuing normal execution in such case
- */
- if (!admin_queue->polling) {
- admin_queue->running_state = false;
- ret = -ETIME;
- goto err;
- }
+ admin_queue->running_state = false;
+ ret = -ETIME;
+ goto err;
}
ret = ena_com_comp_status_to_errno(admin_queue, comp_ctx->comp_status);
@@ -1650,12 +1641,6 @@ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling)
ena_dev->admin_queue.polling = polling;
}
-void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
- bool polling)
-{
- ena_dev->admin_queue.auto_polling = polling;
-}
-
int ena_com_mmio_reg_read_request_init(struct ena_com_dev *ena_dev)
{
struct ena_com_mmio_read *mmio_read = &ena_dev->mmio_read;
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.h b/drivers/net/ethernet/amazon/ena/ena_com.h
index a372c5e768a7..975876cfe720 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.h
+++ b/drivers/net/ethernet/amazon/ena/ena_com.h
@@ -224,9 +224,6 @@ struct ena_com_admin_queue {
/* Indicate if the admin queue should poll for completion */
bool polling;
- /* Define if fallback to polling mode should occur */
- bool auto_polling;
-
u16 curr_cmd_id;
/* Indicate that the ena was initialized and can
@@ -493,17 +490,6 @@ bool ena_com_get_admin_running_state(struct ena_com_dev *ena_dev);
*/
void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling);
-/* ena_com_set_admin_auto_polling_mode - Enable autoswitch to polling mode
- * @ena_dev: ENA communication layer struct
- * @polling: Enable/Disable polling mode
- *
- * Set the autopolling mode.
- * If autopolling is on:
- * In case of missing interrupt when data is available switch to polling.
- */
-void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
- bool polling);
-
/* ena_com_admin_q_comp_intr_handler - admin queue interrupt handler
* @ena_dev: ENA communication layer struct
*
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 9b77dc4a146a..10e3e8e1898f 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -4088,6 +4088,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
err_device_destroy:
ena_com_delete_host_info(ena_dev);
ena_com_admin_destroy(ena_dev);
+ ena_com_mmio_reg_read_request_destroy(ena_dev);
err_metrics_destroy:
ena_com_delete_customer_metrics_buffer(ena_dev);
err_netdev_destroy:
diff --git a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
index c571614b1d50..96152e5a920a 100644
--- a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
+++ b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
@@ -1602,6 +1602,9 @@ static int atl1c_clean_tx(struct napi_struct *napi, int budget)
AT_READ_REGW(&adapter->hw, atl1c_qregs[tpd_ring->num].tpd_cons,
&hw_next_to_clean);
+ if (unlikely(hw_next_to_clean >= tpd_ring->count))
+ hw_next_to_clean = next_to_clean;
+
while (next_to_clean != hw_next_to_clean) {
buffer_info = &tpd_ring->buffer_info[next_to_clean];
if (buffer_info->skb) {
diff --git a/drivers/net/ethernet/atheros/atl1e/atl1e_main.c b/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
index 9b778b34b67e..e0c6be606dc3 100644
--- a/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
+++ b/drivers/net/ethernet/atheros/atl1e/atl1e_main.c
@@ -1234,6 +1234,9 @@ static bool atl1e_clean_tx_irq(struct atl1e_adapter *adapter)
u16 hw_next_to_clean = AT_READ_REGW(&adapter->hw, REG_TPD_CONS_IDX);
u16 next_to_clean = atomic_read(&tx_ring->next_to_clean);
+ if (unlikely(hw_next_to_clean >= tx_ring->count))
+ hw_next_to_clean = next_to_clean;
+
while (next_to_clean != hw_next_to_clean) {
tx_buffer = &tx_ring->tx_buffer[next_to_clean];
if (tx_buffer->dma) {
diff --git a/drivers/net/ethernet/atheros/atlx/atl1.c b/drivers/net/ethernet/atheros/atlx/atl1.c
index 9c5d61990904..b8803fc181bc 100644
--- a/drivers/net/ethernet/atheros/atlx/atl1.c
+++ b/drivers/net/ethernet/atheros/atlx/atl1.c
@@ -2066,6 +2066,9 @@ static int atl1_intr_tx(struct atl1_adapter *adapter)
sw_tpd_next_to_clean = atomic_read(&tpd_ring->next_to_clean);
cmb_tpd_next_to_clean = le16_to_cpu(adapter->cmb.cmb->tpd_cons_idx);
+ if (unlikely(cmb_tpd_next_to_clean >= tpd_ring->count))
+ cmb_tpd_next_to_clean = sw_tpd_next_to_clean;
+
while (cmb_tpd_next_to_clean != sw_tpd_next_to_clean) {
buffer_info = &tpd_ring->buffer_info[sw_tpd_next_to_clean];
if (buffer_info->dma) {
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 45f28445b420..9180fd48c710 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -867,9 +867,9 @@ static void bnxt_tx_int(struct bnxt *bp, struct bnxt_napi *bnapi, int budget)
bnapi->events &= ~BNXT_TX_CMP_EVENT;
}
-static bool bnxt_separate_head_pool(void)
+static bool bnxt_separate_head_pool(struct bnxt_rx_ring_info *rxr)
{
- return PAGE_SIZE > BNXT_RX_PAGE_SIZE;
+ return rxr->rx_page_size != PAGE_SIZE;
}
static struct page *__bnxt_alloc_rx_page(struct bnxt *bp, dma_addr_t *mapping,
@@ -879,9 +879,9 @@ static struct page *__bnxt_alloc_rx_page(struct bnxt *bp, dma_addr_t *mapping,
{
struct page *page;
- if (PAGE_SIZE > BNXT_RX_PAGE_SIZE) {
+ if (rxr->rx_page_size < PAGE_SIZE) {
page = page_pool_dev_alloc_frag(rxr->page_pool, offset,
- BNXT_RX_PAGE_SIZE);
+ rxr->rx_page_size);
} else {
page = page_pool_dev_alloc_pages(rxr->page_pool);
*offset = 0;
@@ -1101,9 +1101,9 @@ static struct sk_buff *bnxt_rx_multi_page_skb(struct bnxt *bp,
return NULL;
}
dma_addr -= bp->rx_dma_offset;
- dma_sync_single_for_cpu(&bp->pdev->dev, dma_addr, BNXT_RX_PAGE_SIZE,
+ dma_sync_single_for_cpu(&bp->pdev->dev, dma_addr, rxr->rx_page_size,
bp->rx_dir);
- skb = napi_build_skb(data_ptr - bp->rx_offset, BNXT_RX_PAGE_SIZE);
+ skb = napi_build_skb(data_ptr - bp->rx_offset, rxr->rx_page_size);
if (!skb) {
page_pool_recycle_direct(rxr->page_pool, page);
return NULL;
@@ -1135,7 +1135,7 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp,
return NULL;
}
dma_addr -= bp->rx_dma_offset;
- dma_sync_single_for_cpu(&bp->pdev->dev, dma_addr, BNXT_RX_PAGE_SIZE,
+ dma_sync_single_for_cpu(&bp->pdev->dev, dma_addr, rxr->rx_page_size,
bp->rx_dir);
if (unlikely(!payload))
@@ -1149,7 +1149,7 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp,
skb_mark_for_recycle(skb);
off = (void *)data_ptr - page_address(page);
- skb_add_rx_frag(skb, 0, page, off, len, BNXT_RX_PAGE_SIZE);
+ skb_add_rx_frag(skb, 0, page, off, len, rxr->rx_page_size);
memcpy(skb->data - NET_IP_ALIGN, data_ptr - NET_IP_ALIGN,
payload + NET_IP_ALIGN);
@@ -1253,7 +1253,7 @@ static u32 __bnxt_rx_agg_pages(struct bnxt *bp,
return 0;
}
- dma_sync_single_for_cpu(&pdev->dev, mapping, BNXT_RX_PAGE_SIZE,
+ dma_sync_single_for_cpu(&pdev->dev, mapping, rxr->rx_page_size,
bp->rx_dir);
total_frag_len += frag_len;
@@ -1268,6 +1268,7 @@ static struct sk_buff *bnxt_rx_agg_pages_skb(struct bnxt *bp,
struct sk_buff *skb, u16 idx,
u32 agg_bufs, bool tpa)
{
+ struct bnxt_rx_ring_info *rxr = cpr->bnapi->rx_ring;
struct skb_shared_info *shinfo = skb_shinfo(skb);
u32 total_frag_len = 0;
@@ -1281,7 +1282,7 @@ static struct sk_buff *bnxt_rx_agg_pages_skb(struct bnxt *bp,
skb->data_len += total_frag_len;
skb->len += total_frag_len;
- skb->truesize += BNXT_RX_PAGE_SIZE * agg_bufs;
+ skb->truesize += rxr->rx_page_size * agg_bufs;
return skb;
}
@@ -2222,8 +2223,7 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
if (!skb)
goto oom_next_rx;
} else {
- skb = bnxt_xdp_build_skb(bp, skb, agg_bufs,
- rxr->page_pool, &xdp);
+ skb = bnxt_xdp_build_skb(bp, skb, agg_bufs, rxr, &xdp);
if (!skb) {
/* we should be able to free the old skb here */
bnxt_xdp_buff_frags_free(rxr, &xdp);
@@ -3647,7 +3647,7 @@ static void bnxt_free_rx_rings(struct bnxt *bp)
xdp_rxq_info_unreg(&rxr->xdp_rxq);
page_pool_destroy(rxr->page_pool);
- if (bnxt_separate_head_pool())
+ if (bnxt_separate_head_pool(rxr))
page_pool_destroy(rxr->head_pool);
rxr->page_pool = rxr->head_pool = NULL;
@@ -3672,12 +3672,14 @@ static int bnxt_alloc_rx_page_pool(struct bnxt *bp,
pp.pool_size = bp->rx_agg_ring_size;
if (BNXT_RX_PAGE_MODE(bp))
pp.pool_size += bp->rx_ring_size;
+
+ pp.order = get_order(rxr->rx_page_size);
pp.nid = numa_node;
pp.napi = &rxr->bnapi->napi;
pp.netdev = bp->dev;
pp.dev = &bp->pdev->dev;
pp.dma_dir = bp->rx_dir;
- pp.max_len = PAGE_SIZE;
+ pp.max_len = PAGE_SIZE << pp.order;
pp.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
pool = page_pool_create(&pp);
@@ -3685,7 +3687,9 @@ static int bnxt_alloc_rx_page_pool(struct bnxt *bp,
return PTR_ERR(pool);
rxr->page_pool = pool;
- if (bnxt_separate_head_pool()) {
+ if (bnxt_separate_head_pool(rxr)) {
+ pp.order = 0;
+ pp.max_len = PAGE_SIZE;
pp.pool_size = max(bp->rx_ring_size, 1024);
pool = page_pool_create(&pp);
if (IS_ERR(pool))
@@ -4140,6 +4144,8 @@ static void bnxt_init_ring_struct(struct bnxt *bp)
if (!rxr)
goto skip_rx;
+ rxr->rx_page_size = BNXT_RX_PAGE_SIZE;
+
ring = &rxr->rx_ring_struct;
rmem = &ring->ring_mem;
rmem->nr_pages = bp->rx_nr_pages;
@@ -4299,8 +4305,15 @@ static void bnxt_init_one_rx_agg_ring_rxbd(struct bnxt *bp,
ring = &rxr->rx_agg_ring_struct;
ring->fw_ring_id = INVALID_HW_RING_ID;
if ((bp->flags & BNXT_FLAG_AGG_RINGS)) {
- type = ((u32)BNXT_RX_PAGE_SIZE << RX_BD_LEN_SHIFT) |
- RX_BD_TYPE_RX_AGG_BD | RX_BD_FLAGS_SOP;
+ type = ((u32)rxr->rx_page_size << RX_BD_LEN_SHIFT) |
+ RX_BD_TYPE_RX_AGG_BD;
+
+ /* On P7, setting EOP will cause the chip to disable
+ * Relaxed Ordering (RO) for TPA data. Disable EOP for
+ * potentially higher performance with RO.
+ */
+ if (BNXT_CHIP_P5_AND_MINUS(bp) || !(bp->flags & BNXT_FLAG_TPA))
+ type |= RX_BD_FLAGS_AGG_EOP;
bnxt_init_rxbd_pages(ring, type);
}
@@ -6858,6 +6871,7 @@ static void bnxt_hwrm_ring_grp_free(struct bnxt *bp)
}
static int hwrm_ring_alloc_send_msg(struct bnxt *bp,
+ struct bnxt_rx_ring_info *rxr,
struct bnxt_ring_struct *ring,
u32 ring_type, u32 map_index)
{
@@ -6930,7 +6944,7 @@ static int hwrm_ring_alloc_send_msg(struct bnxt *bp,
/* Association of agg ring with rx ring */
grp_info = &bp->grp_info[ring->grp_idx];
req->rx_ring_id = cpu_to_le16(grp_info->rx_fw_ring_id);
- req->rx_buf_size = cpu_to_le16(BNXT_RX_PAGE_SIZE);
+ req->rx_buf_size = cpu_to_le16(rxr->rx_page_size);
req->stat_ctx_id = cpu_to_le32(grp_info->fw_stats_ctx);
req->enables |= cpu_to_le32(
RING_ALLOC_REQ_ENABLES_RX_RING_ID_VALID |
@@ -7086,7 +7100,7 @@ static int bnxt_hwrm_rx_ring_alloc(struct bnxt *bp,
u32 map_idx = bnapi->index;
int rc;
- rc = hwrm_ring_alloc_send_msg(bp, ring, type, map_idx);
+ rc = hwrm_ring_alloc_send_msg(bp, rxr, ring, type, map_idx);
if (rc)
return rc;
@@ -7106,7 +7120,7 @@ static int bnxt_hwrm_rx_agg_ring_alloc(struct bnxt *bp,
int rc;
map_idx = grp_idx + bp->rx_nr_rings;
- rc = hwrm_ring_alloc_send_msg(bp, ring, type, map_idx);
+ rc = hwrm_ring_alloc_send_msg(bp, rxr, ring, type, map_idx);
if (rc)
return rc;
@@ -7138,7 +7152,7 @@ static int bnxt_hwrm_ring_alloc(struct bnxt *bp)
vector = bp->irq_tbl[map_idx].vector;
disable_irq_nosync(vector);
- rc = hwrm_ring_alloc_send_msg(bp, ring, type, map_idx);
+ rc = hwrm_ring_alloc_send_msg(bp, NULL, ring, type, map_idx);
if (rc) {
enable_irq(vector);
goto err_out;
@@ -7169,7 +7183,7 @@ static int bnxt_hwrm_ring_alloc(struct bnxt *bp)
ring = &cpr2->cp_ring_struct;
ring->handle = BNXT_SET_NQ_HDL(cpr2);
map_idx = bnapi->index;
- rc = hwrm_ring_alloc_send_msg(bp, ring, type2, map_idx);
+ rc = hwrm_ring_alloc_send_msg(bp, NULL, ring, type2, map_idx);
if (rc)
goto err_out;
bnxt_set_db(bp, &cpr2->cp_db, type2, map_idx,
@@ -7178,7 +7192,7 @@ static int bnxt_hwrm_ring_alloc(struct bnxt *bp)
}
ring = &txr->tx_ring_struct;
map_idx = i;
- rc = hwrm_ring_alloc_send_msg(bp, ring, type, map_idx);
+ rc = hwrm_ring_alloc_send_msg(bp, NULL, ring, type, map_idx);
if (rc)
goto err_out;
bnxt_set_db(bp, &txr->tx_db, type, map_idx, ring->fw_ring_id);
@@ -7202,7 +7216,7 @@ static int bnxt_hwrm_ring_alloc(struct bnxt *bp)
ring = &cpr2->cp_ring_struct;
ring->handle = BNXT_SET_NQ_HDL(cpr2);
- rc = hwrm_ring_alloc_send_msg(bp, ring, type2, map_idx);
+ rc = hwrm_ring_alloc_send_msg(bp, NULL, ring, type2, map_idx);
if (rc)
goto err_out;
bnxt_set_db(bp, &cpr2->cp_db, type2, map_idx,
@@ -15387,6 +15401,9 @@ static int bnxt_queue_mem_alloc(struct net_device *dev, void *qmem, int idx)
clone->rx_agg_prod = 0;
clone->rx_sw_agg_prod = 0;
clone->rx_next_cons = 0;
+ clone->rx_agg_bmap = NULL;
+ clone->rx_tpa = NULL;
+ clone->rx_tpa_idx_map = NULL;
rc = bnxt_alloc_rx_page_pool(bp, clone, rxr->page_pool->p.nid);
if (rc)
@@ -15430,22 +15447,29 @@ static int bnxt_queue_mem_alloc(struct net_device *dev, void *qmem, int idx)
bnxt_alloc_one_rx_ring_skb(bp, clone, idx);
if (bp->flags & BNXT_FLAG_AGG_RINGS)
bnxt_alloc_one_rx_ring_page(bp, clone, idx);
- if (bp->flags & BNXT_FLAG_TPA)
- bnxt_alloc_one_tpa_info_data(bp, clone);
+ if (bp->flags & BNXT_FLAG_TPA) {
+ rc = bnxt_alloc_one_tpa_info_data(bp, clone);
+ if (rc)
+ goto err_free_rx_ring_skbs;
+ }
return 0;
+err_free_rx_ring_skbs:
+ bnxt_free_one_rx_ring_skbs(bp, clone);
err_free_tpa_info:
bnxt_free_one_tpa_info(bp, clone);
err_free_rx_agg_ring:
bnxt_free_ring(bp, &clone->rx_agg_ring_struct.ring_mem);
+ kfree(clone->rx_agg_bmap);
+ clone->rx_agg_bmap = NULL;
err_free_rx_ring:
bnxt_free_ring(bp, &clone->rx_ring_struct.ring_mem);
err_rxq_info_unreg:
xdp_rxq_info_unreg(&clone->xdp_rxq);
err_page_pool_destroy:
page_pool_destroy(clone->page_pool);
- if (bnxt_separate_head_pool())
+ if (bnxt_separate_head_pool(clone))
page_pool_destroy(clone->head_pool);
clone->page_pool = NULL;
clone->head_pool = NULL;
@@ -15464,7 +15488,7 @@ static void bnxt_queue_mem_free(struct net_device *dev, void *qmem)
xdp_rxq_info_unreg(&rxr->xdp_rxq);
page_pool_destroy(rxr->page_pool);
- if (bnxt_separate_head_pool())
+ if (bnxt_separate_head_pool(rxr))
page_pool_destroy(rxr->head_pool);
rxr->page_pool = NULL;
rxr->head_pool = NULL;
@@ -15608,7 +15632,7 @@ static int bnxt_queue_stop(struct net_device *dev, void *qmem, int idx)
bnxt_hwrm_rx_agg_ring_free(bp, rxr, false);
rxr->rx_next_cons = 0;
page_pool_disable_direct_recycling(rxr->page_pool);
- if (bnxt_separate_head_pool())
+ if (bnxt_separate_head_pool(rxr))
page_pool_disable_direct_recycling(rxr->head_pool);
memcpy(qmem, rxr, sizeof(*rxr));
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index 67a82b3262d5..c10f8ad8bfbd 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -124,6 +124,7 @@ struct rx_bd {
#define RX_BD_TYPE_48B_BD_SIZE (2 << 4)
#define RX_BD_TYPE_64B_BD_SIZE (3 << 4)
#define RX_BD_FLAGS_SOP (1 << 6)
+ #define RX_BD_FLAGS_AGG_EOP (1 << 6)
#define RX_BD_FLAGS_EOP (1 << 7)
#define RX_BD_FLAGS_BUFFERS (3 << 8)
#define RX_BD_FLAGS_1_BUFFER_PACKET (0 << 8)
@@ -1097,6 +1098,7 @@ struct bnxt_rx_ring_info {
unsigned long *rx_agg_bmap;
u16 rx_agg_bmap_size;
+ u32 rx_page_size;
dma_addr_t rx_desc_mapping[MAX_RX_PAGES];
dma_addr_t rx_agg_desc_mapping[MAX_RX_AGG_PAGES];
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
index 850dabebdbf2..2c16348d3b8c 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
@@ -182,7 +182,7 @@ void bnxt_xdp_buff_init(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
u16 cons, u8 *data_ptr, unsigned int len,
struct xdp_buff *xdp)
{
- u32 buflen = BNXT_RX_PAGE_SIZE;
+ u32 buflen = rxr->rx_page_size;
struct bnxt_sw_rx_bd *rx_buf;
struct pci_dev *pdev;
dma_addr_t mapping;
@@ -454,7 +454,7 @@ int bnxt_xdp(struct net_device *dev, struct netdev_bpf *xdp)
struct sk_buff *
bnxt_xdp_build_skb(struct bnxt *bp, struct sk_buff *skb, u8 num_frags,
- struct page_pool *pool, struct xdp_buff *xdp)
+ struct bnxt_rx_ring_info *rxr, struct xdp_buff *xdp)
{
struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp);
@@ -463,7 +463,7 @@ bnxt_xdp_build_skb(struct bnxt *bp, struct sk_buff *skb, u8 num_frags,
xdp_update_skb_shared_info(skb, num_frags,
sinfo->xdp_frags_size,
- BNXT_RX_PAGE_SIZE * num_frags,
+ rxr->rx_page_size * num_frags,
xdp_buff_is_frag_pfmemalloc(xdp));
return skb;
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h
index 220285e190fc..8933a0dec09a 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h
@@ -32,6 +32,6 @@ void bnxt_xdp_buff_init(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
void bnxt_xdp_buff_frags_free(struct bnxt_rx_ring_info *rxr,
struct xdp_buff *xdp);
struct sk_buff *bnxt_xdp_build_skb(struct bnxt *bp, struct sk_buff *skb,
- u8 num_frags, struct page_pool *pool,
+ u8 num_frags, struct bnxt_rx_ring_info *rxr,
struct xdp_buff *xdp);
#endif
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
index eb6ef5ad8c2e..14e891db97ea 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
@@ -819,7 +819,8 @@ static int bcmgenet_get_coalesce(struct net_device *dev,
ec->rx_max_coalesced_frames =
bcmgenet_rdma_ring_readl(priv, 0, DMA_MBUF_DONE_THRESH);
ec->rx_coalesce_usecs =
- bcmgenet_rdma_readl(priv, DMA_RING0_TIMEOUT) * 8192 / 1000;
+ (bcmgenet_rdma_readl(priv, DMA_RING0_TIMEOUT) &
+ DMA_TIMEOUT_MASK) * 8192 / 1000;
for (i = 0; i <= priv->hw_params->rx_queues; i++) {
ring = &priv->rx_rings[i];
@@ -1289,9 +1290,8 @@ static void bcmgenet_get_ethtool_stats(struct net_device *dev,
p = (char *)&stats64;
p += s->stat_offset;
- if (sizeof(unsigned long) != sizeof(u32) &&
- s->stat_sizeof == sizeof(unsigned long))
- data[i] = *(unsigned long *)p;
+ if (s->stat_sizeof == sizeof(u64))
+ data[i] = *(u64 *)p;
else
data[i] = *(u32 *)p;
}
@@ -1694,18 +1694,18 @@ static int bcmgenet_power_down(struct bcmgenet_priv *priv,
return ret;
}
-static void bcmgenet_power_up(struct bcmgenet_priv *priv,
- enum bcmgenet_power_mode mode)
+static int bcmgenet_power_up(struct bcmgenet_priv *priv,
+ enum bcmgenet_power_mode mode)
{
+ int ret = 0;
u32 reg;
- if (!bcmgenet_has_ext(priv))
- return;
-
- reg = bcmgenet_ext_readl(priv, EXT_EXT_PWR_MGMT);
-
switch (mode) {
case GENET_POWER_PASSIVE:
+ if (!bcmgenet_has_ext(priv))
+ break;
+
+ reg = bcmgenet_ext_readl(priv, EXT_EXT_PWR_MGMT);
reg &= ~(EXT_PWR_DOWN_DLL | EXT_PWR_DOWN_BIAS |
EXT_ENERGY_DET_MASK);
if (GENET_IS_V5(priv) && !bcmgenet_has_ephy_16nm(priv)) {
@@ -1729,18 +1729,24 @@ static void bcmgenet_power_up(struct bcmgenet_priv *priv,
break;
case GENET_POWER_CABLE_SENSE:
+ if (!bcmgenet_has_ext(priv))
+ break;
+
/* enable APD */
if (!GENET_IS_V5(priv)) {
+ reg = bcmgenet_ext_readl(priv, EXT_EXT_PWR_MGMT);
reg |= EXT_PWR_DN_EN_LD;
bcmgenet_ext_writel(priv, reg, EXT_EXT_PWR_MGMT);
}
break;
case GENET_POWER_WOL_MAGIC:
- bcmgenet_wol_power_up_cfg(priv, mode);
- return;
+ ret = bcmgenet_wol_power_up_cfg(priv, mode);
+ break;
default:
break;
}
+
+ return ret;
}
static struct enet_cb *bcmgenet_get_txcb(struct bcmgenet_priv *priv,
@@ -3557,6 +3563,9 @@ static int bcmgenet_set_mac_addr(struct net_device *dev, void *p)
if (netif_running(dev))
return -EBUSY;
+ if (!is_valid_ether_addr(addr->sa_data))
+ return -EADDRNOTAVAIL;
+
eth_hw_addr_set(dev, addr->sa_data);
return 0;
@@ -4070,10 +4079,10 @@ static int bcmgenet_probe(struct platform_device *pdev)
priv->rx_rings[i].rx_max_coalesced_frames = 1;
/* Initialize u64 stats seq counter for 32bit machines */
- for (i = 0; i <= priv->hw_params->rx_queues; i++)
+ for (i = 0; i <= GENET_MAX_MQ_CNT; i++) {
u64_stats_init(&priv->rx_rings[i].stats64.syncp);
- for (i = 0; i <= priv->hw_params->tx_queues; i++)
u64_stats_init(&priv->tx_rings[i].stats64.syncp);
+ }
/* libphy will determine the link state */
netif_carrier_off(dev);
@@ -4137,8 +4146,20 @@ static int bcmgenet_resume_noirq(struct device *d)
reg = bcmgenet_intrl2_0_readl(priv, INTRL2_CPU_STAT);
if (reg & UMAC_IRQ_WAKE_EVENT)
pm_wakeup_event(&priv->pdev->dev, 0);
+
+ /* From WOL-enabled suspend, switch to regular clock */
+ bcmgenet_power_up(priv, GENET_POWER_WOL_MAGIC);
}
+ /* If this is an internal GPHY, power it back on now, before UniMAC is
+ * brought out of reset as absolutely no UniMAC activity is allowed
+ */
+ if (priv->internal_phy)
+ bcmgenet_power_up(priv, GENET_POWER_PASSIVE);
+
+ /* take MAC out of reset */
+ bcmgenet_umac_reset(priv);
+
bcmgenet_intrl2_0_writel(priv, UMAC_IRQ_WAKE_EVENT, INTRL2_CPU_CLEAR);
return 0;
@@ -4155,18 +4176,6 @@ static int bcmgenet_resume(struct device *d)
if (!netif_running(dev))
return 0;
- /* From WOL-enabled suspend, switch to regular clock */
- if (device_may_wakeup(d) && priv->wolopts)
- bcmgenet_power_up(priv, GENET_POWER_WOL_MAGIC);
-
- /* If this is an internal GPHY, power it back on now, before UniMAC is
- * brought out of reset as absolutely no UniMAC activity is allowed
- */
- if (priv->internal_phy)
- bcmgenet_power_up(priv, GENET_POWER_PASSIVE);
-
- bcmgenet_umac_reset(priv);
-
init_umac(priv);
phy_init_hw(dev->phydev);
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.h b/drivers/net/ethernet/broadcom/genet/bcmgenet.h
index 89b071da3114..532a4ea04c6e 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet.h
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.h
@@ -658,7 +658,6 @@ struct bcmgenet_priv {
struct clk *clk_wol;
u32 wolopts;
u8 sopass[SOPASS_MAX];
- bool wol_active;
struct bcmgenet_mib_counters mib;
};
@@ -740,8 +739,8 @@ void bcmgenet_get_wol(struct net_device *dev, struct ethtool_wolinfo *wol);
int bcmgenet_set_wol(struct net_device *dev, struct ethtool_wolinfo *wol);
int bcmgenet_wol_power_down_cfg(struct bcmgenet_priv *priv,
enum bcmgenet_power_mode mode);
-void bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
- enum bcmgenet_power_mode mode);
+int bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
+ enum bcmgenet_power_mode mode);
void bcmgenet_eee_enable_set(struct net_device *dev, bool enable);
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
index 98358b71cef5..945b32fe5697 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
@@ -199,7 +199,6 @@ int bcmgenet_wol_power_down_cfg(struct bcmgenet_priv *priv,
retries);
clk_prepare_enable(priv->clk_wol);
- priv->wol_active = 1;
if (hfb_enable) {
bcmgenet_hfb_reg_writel(priv, hfb_enable,
@@ -228,28 +227,27 @@ int bcmgenet_wol_power_down_cfg(struct bcmgenet_priv *priv,
return 0;
}
-void bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
- enum bcmgenet_power_mode mode)
+int bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
+ enum bcmgenet_power_mode mode)
{
u32 reg;
if (mode != GENET_POWER_WOL_MAGIC) {
netif_err(priv, wol, priv->dev, "invalid mode: %d\n", mode);
- return;
+ return -EINVAL;
}
- if (!priv->wol_active)
- return; /* failed to suspend so skip the rest */
-
- priv->wol_active = 0;
clk_disable_unprepare(priv->clk_wol);
priv->crc_fwd_en = 0;
+ bcmgenet_intrl2_0_writel(priv, UMAC_IRQ_WAKE_EVENT,
+ INTRL2_CPU_MASK_SET);
+
/* Disable Magic Packet Detection */
if (priv->wolopts & (WAKE_MAGIC | WAKE_MAGICSECURE)) {
reg = bcmgenet_umac_readl(priv, UMAC_MPD_CTRL);
if (!(reg & MPD_EN))
- return; /* already reset so skip the rest */
+ return -EPERM; /* already reset so skip the rest */
reg &= ~(MPD_EN | MPD_PW_EN);
bcmgenet_umac_writel(priv, reg, UMAC_MPD_CTRL);
}
@@ -258,7 +256,7 @@ void bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
if (priv->wolopts & WAKE_FILTER) {
reg = bcmgenet_hfb_reg_readl(priv, HFB_CTRL);
if (!(reg & RBUF_ACPI_EN))
- return; /* already reset so skip the rest */
+ return -EPERM; /* already reset so skip the rest */
reg &= ~(RBUF_HFB_EN | RBUF_ACPI_EN);
bcmgenet_hfb_reg_writel(priv, reg, HFB_CTRL);
}
@@ -269,4 +267,6 @@ void bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
reg &= ~CMD_CRC_FWD;
bcmgenet_umac_writel(priv, reg, UMAC_CMD);
spin_unlock_bh(&priv->reg_lock);
+
+ return 0;
}
diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
index 752f33ae9838..7f143e388d42 100644
--- a/drivers/net/ethernet/broadcom/tg3.c
+++ b/drivers/net/ethernet/broadcom/tg3.c
@@ -17899,11 +17899,14 @@ static int tg3_init_one(struct pci_dev *pdev,
err = tg3_get_device_address(tp, addr);
if (err) {
- dev_err(&pdev->dev,
- "Could not obtain valid ethernet address, aborting\n");
- goto err_out_apeunmap;
+ dev_warn_probe(&pdev->dev, err,
+ "Could not obtain a valid ethernet address\n");
+ if (err == -EPROBE_DEFER)
+ goto err_out_apeunmap;
+ eth_hw_addr_random(dev);
+ } else {
+ eth_hw_addr_set(dev, addr);
}
- eth_hw_addr_set(dev, addr);
intmbx = MAILBOX_INTERRUPT_0 + TG3_64BIT_REG_LOW;
rcvmbx = MAILBOX_RCVRET_CON_IDX_0 + TG3_64BIT_REG_LOW;
@@ -18031,6 +18034,10 @@ static int tg3_init_one(struct pci_dev *pdev,
return 0;
err_out_apeunmap:
+ if (tg3_flag(tp, USE_PHYLIB))
+ tg3_phy_fini(tp);
+ tg3_mdio_fini(tp);
+
if (tp->aperegs) {
iounmap(tp->aperegs);
tp->aperegs = NULL;
diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index ece6f3b48327..757ae52224f5 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -3725,9 +3725,11 @@ bnad_pci_probe(struct pci_dev *pdev,
bnad_res_free(bnad, &bnad->mod_res_info[0], BNA_MOD_RES_T_MAX);
disable_ioceth:
bnad_ioceth_disable(bnad);
- del_timer_sync(&bnad->bna.ioceth.ioc.ioc_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.sem_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ /* The IOC timers rearm one another. Shut down all of them. */
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.ioc_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.sem_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.iocpf_timer);
spin_lock_irqsave(&bnad->bna_lock, flags);
bna_uninit(bna);
spin_unlock_irqrestore(&bnad->bna_lock, flags);
@@ -3768,9 +3770,11 @@ bnad_pci_remove(struct pci_dev *pdev)
mutex_lock(&bnad->conf_mutex);
bnad_ioceth_disable(bnad);
- del_timer_sync(&bnad->bna.ioceth.ioc.ioc_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.sem_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ /* The IOC timers rearm one another. Shut down all of them. */
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.ioc_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.sem_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.iocpf_timer);
spin_lock_irqsave(&bnad->bna_lock, flags);
bna_uninit(bna);
spin_unlock_irqrestore(&bnad->bna_lock, flags);
diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c
index c8d098e8ca32..30d2dd6141a3 100644
--- a/drivers/net/ethernet/cadence/macb_main.c
+++ b/drivers/net/ethernet/cadence/macb_main.c
@@ -2020,10 +2020,7 @@ static irqreturn_t macb_interrupt(int irq, void *dev_id)
if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE)
queue_writel(queue, ISR, MACB_BIT(RCOMP));
- if (napi_schedule_prep(&queue->napi_rx)) {
- netdev_vdbg(bp->dev, "scheduling RX softirq\n");
- __napi_schedule(&queue->napi_rx);
- }
+ napi_schedule(&queue->napi_rx);
}
if (status & (MACB_BIT(TCOMP) |
@@ -2038,10 +2035,7 @@ static irqreturn_t macb_interrupt(int irq, void *dev_id)
wmb(); // ensure softirq can see update
}
- if (napi_schedule_prep(&queue->napi_tx)) {
- netdev_vdbg(bp->dev, "scheduling TX softirq\n");
- __napi_schedule(&queue->napi_tx);
- }
+ napi_schedule(&queue->napi_tx);
}
if (unlikely(status & (MACB_TX_ERR_FLAGS))) {
@@ -4246,7 +4240,7 @@ static int macb_init(struct platform_device *pdev)
queue->bp = bp;
spin_lock_init(&queue->tx_ptr_lock);
netif_napi_add(dev, &queue->napi_rx, macb_rx_poll);
- netif_napi_add(dev, &queue->napi_tx, macb_tx_poll);
+ netif_napi_add_tx(dev, &queue->napi_tx, macb_tx_poll);
if (hw_q) {
queue->ISR = GEM_ISR(hw_q - 1);
queue->IER = GEM_IER(hw_q - 1);
@@ -5233,6 +5227,7 @@ static int macb_probe(struct platform_device *pdev)
}
spin_lock_init(&bp->lock);
spin_lock_init(&bp->stats_lock);
+ spin_lock_init(&bp->tsu_clk_lock);
/* setup capabilities */
macb_configure_caps(bp, macb_config);
diff --git a/drivers/net/ethernet/cadence/macb_ptp.c b/drivers/net/ethernet/cadence/macb_ptp.c
index f2b09100f710..97b72f633d3b 100644
--- a/drivers/net/ethernet/cadence/macb_ptp.c
+++ b/drivers/net/ethernet/cadence/macb_ptp.c
@@ -48,7 +48,12 @@ static int gem_tsu_get_time(struct ptp_clock_info *ptp, struct timespec64 *ts,
spin_lock_irqsave(&bp->tsu_clk_lock, flags);
ptp_read_system_prets(sts);
+ /* explicit barriers are needed because gem_readl() is relaxed */
+ if (sts)
+ rmb();
first = gem_readl(bp, TN);
+ if (sts)
+ rmb();
ptp_read_system_postts(sts);
secl = gem_readl(bp, TSL);
sech = gem_readl(bp, TSH);
@@ -60,7 +65,11 @@ static int gem_tsu_get_time(struct ptp_clock_info *ptp, struct timespec64 *ts,
* (assume all done within 1s)
*/
ptp_read_system_prets(sts);
+ if (sts)
+ rmb();
ts->tv_nsec = gem_readl(bp, TN);
+ if (sts)
+ rmb();
ptp_read_system_postts(sts);
secl = gem_readl(bp, TSL);
sech = gem_readl(bp, TSH);
@@ -322,9 +331,9 @@ void gem_ptp_txstamp(struct macb *bp, struct sk_buff *skb,
skb_tstamp_tx(skb, &shhwtstamps);
}
-void gem_ptp_init(struct net_device *dev)
+void gem_ptp_init(struct net_device *netdev)
{
- struct macb *bp = netdev_priv(dev);
+ struct macb *bp = netdev_priv(netdev);
bp->ptp_clock_info = gem_ptp_caps_template;
@@ -332,7 +341,8 @@ void gem_ptp_init(struct net_device *dev)
bp->tsu_rate = bp->ptp_info->get_tsu_rate(bp);
bp->ptp_clock_info.max_adj = bp->ptp_info->get_ptp_max_adj();
gem_ptp_init_timer(bp);
- bp->ptp_clock = ptp_clock_register(&bp->ptp_clock_info, &dev->dev);
+ gem_ptp_init_tsu(bp);
+ bp->ptp_clock = ptp_clock_register(&bp->ptp_clock_info, &netdev->dev);
if (IS_ERR(bp->ptp_clock)) {
pr_err("ptp clock register failed: %ld\n",
PTR_ERR(bp->ptp_clock));
@@ -343,10 +353,6 @@ void gem_ptp_init(struct net_device *dev)
return;
}
- spin_lock_init(&bp->tsu_clk_lock);
-
- gem_ptp_init_tsu(bp);
-
dev_info(&bp->pdev->dev, "%s ptp clock registered.\n",
GEM_PTP_TIMER_NAME);
}
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c
index 6f6525983130..6d69c19ea7c3 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/chtls/chtls_cm.c
@@ -1197,12 +1197,12 @@ static struct sock *chtls_recv_sock(struct sock *lsk,
struct ipv6_pinfo *newnp = inet6_sk(newsk);
struct ipv6_pinfo *np = inet6_sk(lsk);
- inet_sk(newsk)->pinet6 = &newtcp6sk->inet6;
+ newinet->pinet6 = &newtcp6sk->inet6;
+ newinet->ipv6_fl_list = NULL;
memcpy(newnp, np, sizeof(struct ipv6_pinfo));
newsk->sk_v6_daddr = treq->ir_v6_rmt_addr;
newsk->sk_v6_rcv_saddr = treq->ir_v6_loc_addr;
inet6_sk(newsk)->saddr = treq->ir_v6_loc_addr;
- newnp->ipv6_fl_list = NULL;
newnp->pktoptions = NULL;
newsk->sk_bound_dev_if = treq->ir_iif;
newinet->inet_opt = NULL;
diff --git a/drivers/net/ethernet/cortina/gemini.c b/drivers/net/ethernet/cortina/gemini.c
index 96fd27545b29..1b81798914ac 100644
--- a/drivers/net/ethernet/cortina/gemini.c
+++ b/drivers/net/ethernet/cortina/gemini.c
@@ -1798,7 +1798,7 @@ static irqreturn_t gmac_irq(int irq, void *data)
if (val & (GMAC0_RX_OVERRUN_INT_BIT << (netdev->dev_id * 8))) {
spin_lock(&geth->irq_lock);
- writel(GMAC0_RXDERR_INT_BIT << (netdev->dev_id * 8),
+ writel(GMAC0_RX_OVERRUN_INT_BIT << (netdev->dev_id * 8),
geth->base + GLOBAL_INTERRUPT_STATUS_4_REG);
u64_stats_update_begin(&port->ir_stats_syncp);
++port->stats.rx_fifo_errors;
diff --git a/drivers/net/ethernet/freescale/fman/fman.c b/drivers/net/ethernet/freescale/fman/fman.c
index bfc784b7f130..f32b8a7159d0 100644
--- a/drivers/net/ethernet/freescale/fman/fman.c
+++ b/drivers/net/ethernet/freescale/fman/fman.c
@@ -2747,6 +2747,7 @@ static struct fman *read_dts_node(struct platform_device *of_dev)
}
clk_rate = clk_get_rate(clk);
+ clk_put(clk);
if (!clk_rate) {
err = -EINVAL;
dev_err(&of_dev->dev, "%s: Failed to determine FM%d clock rate\n",
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
index 58baac7103b3..344920fd6900 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
@@ -793,6 +793,7 @@ static int hns_mac_register_phy(struct hns_mac_cb *mac_cb)
dev_err(mac_cb->dev,
"mac%d mdio is NULL, dsaf will probe again later\n",
mac_cb->mac_id);
+ put_device(&pdev->dev);
return -EPROBE_DEFER;
}
@@ -801,6 +802,8 @@ static int hns_mac_register_phy(struct hns_mac_cb *mac_cb)
dev_dbg(mac_cb->dev, "mac%d register phy addr:%d\n",
mac_cb->mac_id, addr);
+ put_device(&pdev->dev);
+
return rc;
}
diff --git a/drivers/net/ethernet/intel/idpf/idpf.h b/drivers/net/ethernet/intel/idpf/idpf.h
index c420f6b71283..55b43a27f106 100644
--- a/drivers/net/ethernet/intel/idpf/idpf.h
+++ b/drivers/net/ethernet/intel/idpf/idpf.h
@@ -8,6 +8,7 @@
struct idpf_adapter;
struct idpf_vport;
struct idpf_vport_max_q;
+struct idpf_q_vec_rsrc;
#include <net/pkt_sched.h>
#include <linux/aer.h>
@@ -196,7 +197,8 @@ struct idpf_vport_max_q {
*/
struct idpf_reg_ops {
void (*ctlq_reg_init)(struct idpf_ctlq_create_info *cq);
- int (*intr_reg_init)(struct idpf_vport *vport);
+ int (*intr_reg_init)(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc);
void (*mb_intr_reg_init)(struct idpf_adapter *adapter);
void (*reset_reg_init)(struct idpf_adapter *adapter);
void (*trigger_reset)(struct idpf_adapter *adapter,
@@ -251,6 +253,18 @@ struct idpf_port_stats {
struct virtchnl2_vport_stats vport_stats;
};
+/**
+ * struct idpf_q_vec_rsrc - handle for queue and vector resources
+ * @q_vectors: array of queue vectors
+ * @q_vector_idxs: starting index of queue vectors
+ * @num_q_vectors: number of IRQ vectors allocated
+ */
+struct idpf_q_vec_rsrc {
+ struct idpf_q_vector *q_vectors;
+ u16 *q_vector_idxs;
+ u16 num_q_vectors;
+};
+
/**
* struct idpf_vport - Handle for netdevices and queue resources
* @num_txq: Number of allocated TX queues
@@ -284,9 +298,7 @@ struct idpf_port_stats {
* @idx: Software index in adapter vports struct
* @default_vport: Use this vport if one isn't specified
* @base_rxd: True if the driver should use base descriptors instead of flex
- * @num_q_vectors: Number of IRQ vectors allocated
- * @q_vectors: Array of queue vectors
- * @q_vector_idxs: Starting index of queue vectors
+ * @dflt_qv_rsrc: contains default queue and vector resources
* @max_mtu: device given max possible MTU
* @default_mac_addr: device will give a default MAC to use
* @rx_itr_profile: RX profiles for Dynamic Interrupt Moderation
@@ -326,9 +338,7 @@ struct idpf_vport {
bool default_vport;
bool base_rxd;
- u16 num_q_vectors;
- struct idpf_q_vector *q_vectors;
- u16 *q_vector_idxs;
+ struct idpf_q_vec_rsrc dflt_qv_rsrc;
u16 max_mtu;
u8 default_mac_addr[ETH_ALEN];
u16 rx_itr_profile[IDPF_DIM_PROFILE_SLOTS];
diff --git a/drivers/net/ethernet/intel/idpf/idpf_dev.c b/drivers/net/ethernet/intel/idpf/idpf_dev.c
index 41e4bd49402a..45272bd9e472 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_dev.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_dev.c
@@ -66,11 +66,13 @@ static void idpf_mb_intr_reg_init(struct idpf_adapter *adapter)
/**
* idpf_intr_reg_init - Initialize interrupt registers
* @vport: virtual port structure
+ * @rsrc: pointer to queue and vector resources
*/
-static int idpf_intr_reg_init(struct idpf_vport *vport)
+static int idpf_intr_reg_init(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
struct idpf_adapter *adapter = vport->adapter;
- int num_vecs = vport->num_q_vectors;
+ u16 num_vecs = rsrc->num_q_vectors;
struct idpf_vec_regs *reg_vals;
int num_regs, i, err = 0;
u32 rx_itr, tx_itr;
@@ -89,8 +91,8 @@ static int idpf_intr_reg_init(struct idpf_vport *vport)
}
for (i = 0; i < num_vecs; i++) {
- struct idpf_q_vector *q_vector = &vport->q_vectors[i];
- u16 vec_id = vport->q_vector_idxs[i] - IDPF_MBX_Q_VEC;
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[i];
+ u16 vec_id = rsrc->q_vector_idxs[i] - IDPF_MBX_Q_VEC;
struct idpf_intr_reg *intr = &q_vector->intr_reg;
u32 spacing;
diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ethernet/intel/idpf/idpf_lib.c
index c2101f7f6e1b..36777ca10f32 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_lib.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c
@@ -908,6 +908,7 @@ static void idpf_remove_features(struct idpf_vport *vport)
static void idpf_vport_stop(struct idpf_vport *vport)
{
struct idpf_netdev_priv *np = netdev_priv(vport->netdev);
+ struct idpf_q_vec_rsrc *rsrc = &vport->dflt_qv_rsrc;
if (np->state <= __IDPF_VPORT_DOWN)
return;
@@ -929,9 +930,9 @@ static void idpf_vport_stop(struct idpf_vport *vport)
idpf_remove_features(vport);
vport->link_up = false;
- idpf_vport_intr_deinit(vport);
+ idpf_vport_intr_deinit(vport, rsrc);
idpf_vport_queues_rel(vport);
- idpf_vport_intr_rel(vport);
+ idpf_vport_intr_rel(rsrc);
np->state = __IDPF_VPORT_DOWN;
}
@@ -991,6 +992,7 @@ static void idpf_decfg_netdev(struct idpf_vport *vport)
*/
static void idpf_vport_rel(struct idpf_vport *vport)
{
+ struct idpf_q_vec_rsrc *rsrc = &vport->dflt_qv_rsrc;
struct idpf_adapter *adapter = vport->adapter;
struct idpf_vport_config *vport_config;
struct idpf_vector_info vec_info;
@@ -1015,13 +1017,13 @@ static void idpf_vport_rel(struct idpf_vport *vport)
/* Release all the allocated vectors on the stack */
vec_info.num_req_vecs = 0;
- vec_info.num_curr_vecs = vport->num_q_vectors;
+ vec_info.num_curr_vecs = rsrc->num_q_vectors;
vec_info.default_vport = vport->default_vport;
- idpf_req_rel_vector_indexes(adapter, vport->q_vector_idxs, &vec_info);
+ idpf_req_rel_vector_indexes(adapter, rsrc->q_vector_idxs, &vec_info);
- kfree(vport->q_vector_idxs);
- vport->q_vector_idxs = NULL;
+ kfree(rsrc->q_vector_idxs);
+ rsrc->q_vector_idxs = NULL;
kfree(adapter->vport_params_recvd[idx]);
adapter->vport_params_recvd[idx] = NULL;
@@ -1146,6 +1148,7 @@ static struct idpf_vport *idpf_vport_alloc(struct idpf_adapter *adapter,
{
struct idpf_rss_data *rss_data;
u16 idx = adapter->next_vport;
+ struct idpf_q_vec_rsrc *rsrc;
struct idpf_vport *vport;
u16 num_max_q;
int err;
@@ -1193,8 +1196,9 @@ static struct idpf_vport *idpf_vport_alloc(struct idpf_adapter *adapter,
vport->default_vport = adapter->num_alloc_vports <
idpf_get_default_vports(adapter);
- vport->q_vector_idxs = kcalloc(num_max_q, sizeof(u16), GFP_KERNEL);
- if (!vport->q_vector_idxs)
+ rsrc = &vport->dflt_qv_rsrc;
+ rsrc->q_vector_idxs = kcalloc(num_max_q, sizeof(u16), GFP_KERNEL);
+ if (!rsrc->q_vector_idxs)
goto free_vport;
idpf_vport_init(vport, max_q);
@@ -1232,7 +1236,7 @@ static struct idpf_vport *idpf_vport_alloc(struct idpf_adapter *adapter,
kfree(rss_data->rss_key);
rss_data->rss_key = NULL;
free_vector_idxs:
- kfree(vport->q_vector_idxs);
+ kfree(rsrc->q_vector_idxs);
free_vport:
kfree(vport);
@@ -1405,6 +1409,7 @@ static void idpf_rx_init_buf_tail(struct idpf_vport *vport)
static int idpf_vport_open(struct idpf_vport *vport)
{
struct idpf_netdev_priv *np = netdev_priv(vport->netdev);
+ struct idpf_q_vec_rsrc *rsrc = &vport->dflt_qv_rsrc;
struct idpf_adapter *adapter = vport->adapter;
int err;
@@ -1414,7 +1419,7 @@ static int idpf_vport_open(struct idpf_vport *vport)
/* we do not allow interface up just yet */
netif_carrier_off(vport->netdev);
- err = idpf_vport_intr_alloc(vport);
+ err = idpf_vport_intr_alloc(vport, rsrc);
if (err) {
dev_err(&adapter->pdev->dev, "Failed to allocate interrupts for vport %u: %d\n",
vport->vport_id, err);
@@ -1432,7 +1437,7 @@ static int idpf_vport_open(struct idpf_vport *vport)
goto queues_rel;
}
- err = idpf_vport_intr_init(vport);
+ err = idpf_vport_intr_init(vport, rsrc);
if (err) {
dev_err(&adapter->pdev->dev, "Failed to initialize interrupts for vport %u: %d\n",
vport->vport_id, err);
@@ -1454,7 +1459,7 @@ static int idpf_vport_open(struct idpf_vport *vport)
}
idpf_rx_init_buf_tail(vport);
- idpf_vport_intr_ena(vport);
+ idpf_vport_intr_ena(vport, rsrc);
err = idpf_send_config_queues_msg(vport);
if (err) {
@@ -1510,11 +1515,11 @@ static int idpf_vport_open(struct idpf_vport *vport)
unmap_queue_vectors:
idpf_send_map_unmap_queue_vector_msg(vport, false);
intr_deinit:
- idpf_vport_intr_deinit(vport);
+ idpf_vport_intr_deinit(vport, rsrc);
queues_rel:
idpf_vport_queues_rel(vport);
intr_rel:
- idpf_vport_intr_rel(vport);
+ idpf_vport_intr_rel(rsrc);
return err;
}
@@ -1958,7 +1963,7 @@ int idpf_initiate_soft_reset(struct idpf_vport *vport,
memcpy(vport, new_vport, offsetof(struct idpf_vport, link_up));
if (reset_cause == IDPF_SR_Q_CHANGE)
- idpf_vport_alloc_vec_indexes(vport);
+ idpf_vport_alloc_vec_indexes(vport, &vport->dflt_qv_rsrc);
err = idpf_set_real_num_queues(vport);
if (err)
diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.c b/drivers/net/ethernet/intel/idpf/idpf_txrx.c
index 703dea3f9d8b..f085d3d9eb54 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_txrx.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.c
@@ -19,6 +19,8 @@ LIBETH_SQE_CHECK_PRIV(u32);
* Make sure we don't exceed maximum scatter gather buffers for a single
* packet.
* TSO case has been handled earlier from idpf_features_check().
+ *
+ * Return: %true if skb exceeds max descriptors per packet, %false otherwise.
*/
static bool idpf_chk_linearize(const struct sk_buff *skb,
unsigned int max_bufs,
@@ -148,7 +150,7 @@ static void idpf_tx_desc_rel_all(struct idpf_vport *vport)
* idpf_tx_buf_alloc_all - Allocate memory for all buffer resources
* @tx_q: queue for which the buffers are allocated
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_tx_buf_alloc_all(struct idpf_tx_queue *tx_q)
{
@@ -172,7 +174,7 @@ static int idpf_tx_buf_alloc_all(struct idpf_tx_queue *tx_q)
* @vport: vport to allocate resources for
* @tx_q: the tx ring to set up
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_tx_desc_alloc(const struct idpf_vport *vport,
struct idpf_tx_queue *tx_q)
@@ -264,7 +266,7 @@ static int idpf_compl_desc_alloc(const struct idpf_vport *vport,
* idpf_tx_desc_alloc_all - allocate all queues Tx resources
* @vport: virtual port private structure
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_tx_desc_alloc_all(struct idpf_vport *vport)
{
@@ -509,7 +511,7 @@ static void idpf_rx_buf_hw_update(struct idpf_buf_queue *bufq, u32 val)
* idpf_rx_hdr_buf_alloc_all - Allocate memory for header buffers
* @bufq: ring to use
*
- * Returns 0 on success, negative on failure.
+ * Return: 0 on success, negative on failure.
*/
static int idpf_rx_hdr_buf_alloc_all(struct idpf_buf_queue *bufq)
{
@@ -560,7 +562,7 @@ static void idpf_post_buf_refill(struct idpf_sw_queue *refillq, u16 buf_id)
* @bufq: buffer queue to post to
* @buf_id: buffer id to post
*
- * Returns false if buffer could not be allocated, true otherwise.
+ * Return: %false if buffer could not be allocated, %true otherwise.
*/
static bool idpf_rx_post_buf_desc(struct idpf_buf_queue *bufq, u16 buf_id)
{
@@ -609,7 +611,7 @@ static bool idpf_rx_post_buf_desc(struct idpf_buf_queue *bufq, u16 buf_id)
* @bufq: buffer queue to post working set to
* @working_set: number of buffers to put in working set
*
- * Returns true if @working_set bufs were posted successfully, false otherwise.
+ * Return: %true if @working_set bufs were posted successfully, %false otherwise.
*/
static bool idpf_rx_post_init_bufs(struct idpf_buf_queue *bufq,
u16 working_set)
@@ -678,7 +680,7 @@ static int idpf_rx_bufs_init_singleq(struct idpf_rx_queue *rxq)
* idpf_rx_buf_alloc_all - Allocate memory for all buffer resources
* @rxbufq: queue for which the buffers are allocated
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_rx_buf_alloc_all(struct idpf_buf_queue *rxbufq)
{
@@ -706,7 +708,7 @@ static int idpf_rx_buf_alloc_all(struct idpf_buf_queue *rxbufq)
* @bufq: buffer queue to create page pool for
* @type: type of Rx buffers to allocate
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_rx_bufs_init(struct idpf_buf_queue *bufq,
enum libeth_fqe_type type)
@@ -737,7 +739,7 @@ static int idpf_rx_bufs_init(struct idpf_buf_queue *bufq,
* idpf_rx_bufs_init_all - Initialize all RX bufs
* @vport: virtual port struct
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
int idpf_rx_bufs_init_all(struct idpf_vport *vport)
{
@@ -790,7 +792,7 @@ int idpf_rx_bufs_init_all(struct idpf_vport *vport)
* @vport: vport to allocate resources for
* @rxq: Rx queue for which the resources are setup
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_rx_desc_alloc(const struct idpf_vport *vport,
struct idpf_rx_queue *rxq)
@@ -849,7 +851,7 @@ static int idpf_bufq_desc_alloc(const struct idpf_vport *vport,
* idpf_rx_desc_alloc_all - allocate all RX queues resources
* @vport: virtual port structure
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_rx_desc_alloc_all(struct idpf_vport *vport)
{
@@ -1038,7 +1040,7 @@ void idpf_vport_queues_rel(struct idpf_vport *vport)
* dereference the queue from queue groups. This allows us to quickly pull a
* txq based on a queue index.
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_vport_init_fast_path_txqs(struct idpf_vport *vport)
{
@@ -1151,7 +1153,7 @@ void idpf_vport_calc_num_q_desc(struct idpf_vport *vport)
* @vport_msg: message to fill with data
* @max_q: vport max queue info
*
- * Return 0 on success, error value on failure.
+ * Return: 0 on success, error value on failure.
*/
int idpf_vport_calc_total_qs(struct idpf_adapter *adapter, u16 vport_idx,
struct virtchnl2_create_vport *vport_msg,
@@ -1274,7 +1276,7 @@ static void idpf_rxq_set_descids(const struct idpf_vport *vport,
* @vport: vport to allocate txq groups for
* @num_txq: number of txqs to allocate for each group
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_txq_group_alloc(struct idpf_vport *vport, u16 num_txq)
{
@@ -1365,7 +1367,7 @@ static int idpf_txq_group_alloc(struct idpf_vport *vport, u16 num_txq)
* @vport: vport to allocate rxq groups for
* @num_rxq: number of rxqs to allocate for each group
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_rxq_group_alloc(struct idpf_vport *vport, u16 num_rxq)
{
@@ -1495,7 +1497,7 @@ static int idpf_rxq_group_alloc(struct idpf_vport *vport, u16 num_rxq)
* idpf_vport_queue_grp_alloc_all - Allocate all queue groups/resources
* @vport: vport with qgrps to allocate
*
- * Returns 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
static int idpf_vport_queue_grp_alloc_all(struct idpf_vport *vport)
{
@@ -1524,8 +1526,9 @@ static int idpf_vport_queue_grp_alloc_all(struct idpf_vport *vport)
* idpf_vport_queues_alloc - Allocate memory for all queues
* @vport: virtual port
*
- * Allocate memory for queues associated with a vport. Returns 0 on success,
- * negative on failure.
+ * Allocate memory for queues associated with a vport.
+ *
+ * Return: 0 on success, negative on failure.
*/
int idpf_vport_queues_alloc(struct idpf_vport *vport)
{
@@ -1737,7 +1740,7 @@ static void idpf_tx_handle_rs_completion(struct idpf_tx_queue *txq,
* @budget: Used to determine if we are in netpoll
* @cleaned: returns number of packets cleaned
*
- * Returns true if there's any budget left (e.g. the clean is finished)
+ * Return: %true if there's any budget left (e.g. the clean is finished)
*/
static bool idpf_tx_clean_complq(struct idpf_compl_queue *complq, int budget,
int *cleaned)
@@ -1903,7 +1906,7 @@ void idpf_tx_splitq_build_flow_desc(union idpf_tx_flex_desc *desc,
}
/**
- * idpf_tx_splitq_has_room - check if enough Tx splitq resources are available
+ * idpf_txq_has_room - check if enough Tx splitq resources are available
* @tx_q: the queue to be checked
* @descs_needed: number of descriptors required for this packet
* @bufs_needed: number of Tx buffers required for this packet
@@ -2034,6 +2037,8 @@ unsigned int idpf_tx_res_count_required(struct idpf_tx_queue *txq,
* idpf_tx_splitq_bump_ntu - adjust NTU and generation
* @txq: the tx ring to wrap
* @ntu: ring index to bump
+ *
+ * Return: the next ring index hopping to 0 when wraps around
*/
static unsigned int idpf_tx_splitq_bump_ntu(struct idpf_tx_queue *txq, u16 ntu)
{
@@ -2302,7 +2307,7 @@ static void idpf_tx_splitq_map(struct idpf_tx_queue *tx_q,
* @skb: pointer to skb
* @off: pointer to struct that holds offload parameters
*
- * Returns error (negative) if TSO was requested but cannot be applied to the
+ * Return: error (negative) if TSO was requested but cannot be applied to the
* given skb, 0 if TSO does not apply to the given skb, or 1 otherwise.
*/
int idpf_tso(struct sk_buff *skb, struct idpf_tx_offload_params *off)
@@ -2380,6 +2385,8 @@ int idpf_tso(struct sk_buff *skb, struct idpf_tx_offload_params *off)
*
* Since the TX buffer rings mimics the descriptor ring, update the tx buffer
* ring entry to reflect that this index is a context descriptor
+ *
+ * Return: pointer to the next descriptor
*/
static struct idpf_flex_tx_ctx_desc *
idpf_tx_splitq_get_ctx_desc(struct idpf_tx_queue *txq)
@@ -2398,6 +2405,8 @@ idpf_tx_splitq_get_ctx_desc(struct idpf_tx_queue *txq)
* idpf_tx_drop_skb - free the SKB and bump tail if necessary
* @tx_q: queue to send buffer on
* @skb: pointer to skb
+ *
+ * Return: always NETDEV_TX_OK
*/
netdev_tx_t idpf_tx_drop_skb(struct idpf_tx_queue *tx_q, struct sk_buff *skb)
{
@@ -2432,7 +2441,7 @@ static bool idpf_tx_splitq_need_re(struct idpf_tx_queue *tx_q)
* @skb: send buffer
* @tx_q: queue to send buffer on
*
- * Returns NETDEV_TX_OK if sent, else an error code
+ * Return: NETDEV_TX_OK if sent, else an error code
*/
static netdev_tx_t idpf_tx_splitq_frame(struct sk_buff *skb,
struct idpf_tx_queue *tx_q)
@@ -2549,7 +2558,7 @@ static netdev_tx_t idpf_tx_splitq_frame(struct sk_buff *skb,
* @skb: send buffer
* @netdev: network interface device structure
*
- * Returns NETDEV_TX_OK if sent, else an error code
+ * Return: NETDEV_TX_OK if sent, else an error code
*/
netdev_tx_t idpf_tx_start(struct sk_buff *skb, struct net_device *netdev)
{
@@ -2698,10 +2707,10 @@ idpf_rx_splitq_extract_csum_bits(const struct virtchnl2_rx_flex_desc_adv_nic_3 *
* @rx_desc: Receive descriptor
* @decoded: Decoded Rx packet type related fields
*
- * Return 0 on success and error code on failure
- *
* Populate the skb fields with the total number of RSC segments, RSC payload
* length and packet type.
+ *
+ * Return: 0 on success and error code on failure
*/
static int idpf_rx_rsc(struct idpf_rx_queue *rxq, struct sk_buff *skb,
const struct virtchnl2_rx_flex_desc_adv_nic_3 *rx_desc,
@@ -2776,6 +2785,8 @@ static int idpf_rx_rsc(struct idpf_rx_queue *rxq, struct sk_buff *skb,
* This function checks the ring, descriptor, and packet information in
* order to populate the hash, checksum, protocol, and
* other fields within the skb.
+ *
+ * Return: 0 on success and error code on failure
*/
static int
idpf_rx_process_skb_fields(struct idpf_rx_queue *rxq, struct sk_buff *skb,
@@ -2894,6 +2905,7 @@ struct sk_buff *idpf_rx_build_skb(const struct libeth_fqe *buf, u32 size)
* @stat_err_field: field from descriptor to test bits in
* @stat_err_bits: value to mask
*
+ * Return: %true if any of given @stat_err_bits are set, %false otherwise.
*/
static bool idpf_rx_splitq_test_staterr(const u8 stat_err_field,
const u8 stat_err_bits)
@@ -2905,8 +2917,8 @@ static bool idpf_rx_splitq_test_staterr(const u8 stat_err_field,
* idpf_rx_splitq_is_eop - process handling of EOP buffers
* @rx_desc: Rx descriptor for current buffer
*
- * If the buffer is an EOP buffer, this function exits returning true,
- * otherwise return false indicating that this is in fact a non-EOP buffer.
+ * Return: %true if the buffer is an EOP buffer, %false otherwise, indicating
+ * that this is in fact a non-EOP buffer.
*/
static bool idpf_rx_splitq_is_eop(struct virtchnl2_rx_flex_desc_adv_nic_3 *rx_desc)
{
@@ -2925,7 +2937,7 @@ static bool idpf_rx_splitq_is_eop(struct virtchnl2_rx_flex_desc_adv_nic_3 *rx_de
* expensive overhead for IOMMU access this provides a means of avoiding
* it by maintaining the mapping of the page to the system.
*
- * Returns amount of work completed
+ * Return: amount of work completed
*/
static int idpf_rx_splitq_clean(struct idpf_rx_queue *rxq, int budget)
{
@@ -3087,7 +3099,7 @@ static int idpf_rx_splitq_clean(struct idpf_rx_queue *rxq, int budget)
* @buf_id: buffer ID
* @buf_desc: Buffer queue descriptor
*
- * Return 0 on success and negative on failure.
+ * Return: 0 on success and negative on failure.
*/
static int idpf_rx_update_bufq_desc(struct idpf_buf_queue *bufq, u32 buf_id,
struct virtchnl2_splitq_rx_buf_desc *buf_desc)
@@ -3214,6 +3226,7 @@ static void idpf_rx_clean_refillq_all(struct idpf_buf_queue *bufq, int nid)
* @irq: interrupt number
* @data: pointer to a q_vector
*
+ * Return: always IRQ_HANDLED
*/
static irqreturn_t idpf_vport_intr_clean_queues(int __always_unused irq,
void *data)
@@ -3228,39 +3241,34 @@ static irqreturn_t idpf_vport_intr_clean_queues(int __always_unused irq,
/**
* idpf_vport_intr_napi_del_all - Unregister napi for all q_vectors in vport
- * @vport: virtual port structure
- *
+ * @rsrc: pointer to queue and vector resources
*/
-static void idpf_vport_intr_napi_del_all(struct idpf_vport *vport)
+static void idpf_vport_intr_napi_del_all(struct idpf_q_vec_rsrc *rsrc)
{
- u16 v_idx;
-
- for (v_idx = 0; v_idx < vport->num_q_vectors; v_idx++)
- netif_napi_del(&vport->q_vectors[v_idx].napi);
+ for (u16 v_idx = 0; v_idx < rsrc->num_q_vectors; v_idx++)
+ netif_napi_del(&rsrc->q_vectors[v_idx].napi);
}
/**
* idpf_vport_intr_napi_dis_all - Disable NAPI for all q_vectors in the vport
- * @vport: main vport structure
+ * @rsrc: pointer to queue and vector resources
*/
-static void idpf_vport_intr_napi_dis_all(struct idpf_vport *vport)
+static void idpf_vport_intr_napi_dis_all(struct idpf_q_vec_rsrc *rsrc)
{
- int v_idx;
-
- for (v_idx = 0; v_idx < vport->num_q_vectors; v_idx++)
- napi_disable(&vport->q_vectors[v_idx].napi);
+ for (u16 v_idx = 0; v_idx < rsrc->num_q_vectors; v_idx++)
+ napi_disable(&rsrc->q_vectors[v_idx].napi);
}
/**
* idpf_vport_intr_rel - Free memory allocated for interrupt vectors
- * @vport: virtual port
+ * @rsrc: pointer to queue and vector resources
*
* Free the memory allocated for interrupt vectors associated to a vport
*/
-void idpf_vport_intr_rel(struct idpf_vport *vport)
+void idpf_vport_intr_rel(struct idpf_q_vec_rsrc *rsrc)
{
- for (u32 v_idx = 0; v_idx < vport->num_q_vectors; v_idx++) {
- struct idpf_q_vector *q_vector = &vport->q_vectors[v_idx];
+ for (u16 v_idx = 0; v_idx < rsrc->num_q_vectors; v_idx++) {
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[v_idx];
kfree(q_vector->complq);
q_vector->complq = NULL;
@@ -3274,28 +3282,29 @@ void idpf_vport_intr_rel(struct idpf_vport *vport)
free_cpumask_var(q_vector->affinity_mask);
}
- kfree(vport->q_vectors);
- vport->q_vectors = NULL;
+ kfree(rsrc->q_vectors);
+ rsrc->q_vectors = NULL;
}
/**
* idpf_vport_intr_rel_irq - Free the IRQ association with the OS
* @vport: main vport structure
+ * @rsrc: pointer to queue and vector resources
*/
-static void idpf_vport_intr_rel_irq(struct idpf_vport *vport)
+static void idpf_vport_intr_rel_irq(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
struct idpf_adapter *adapter = vport->adapter;
- int vector;
- for (vector = 0; vector < vport->num_q_vectors; vector++) {
- struct idpf_q_vector *q_vector = &vport->q_vectors[vector];
+ for (u16 vector = 0; vector < rsrc->num_q_vectors; vector++) {
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[vector];
int irq_num, vidx;
/* free only the irqs that were actually requested */
if (!q_vector)
continue;
- vidx = vport->q_vector_idxs[vector];
+ vidx = rsrc->q_vector_idxs[vector];
irq_num = adapter->msix_entries[vidx].vector;
/* clear the affinity_mask in the IRQ descriptor */
@@ -3306,20 +3315,21 @@ static void idpf_vport_intr_rel_irq(struct idpf_vport *vport)
/**
* idpf_vport_intr_dis_irq_all - Disable all interrupt
- * @vport: main vport structure
+ * @rsrc: pointer to queue and vector resources
*/
-static void idpf_vport_intr_dis_irq_all(struct idpf_vport *vport)
+static void idpf_vport_intr_dis_irq_all(struct idpf_q_vec_rsrc *rsrc)
{
- struct idpf_q_vector *q_vector = vport->q_vectors;
- int q_idx;
+ struct idpf_q_vector *q_vector = rsrc->q_vectors;
- for (q_idx = 0; q_idx < vport->num_q_vectors; q_idx++)
+ for (u16 q_idx = 0; q_idx < rsrc->num_q_vectors; q_idx++)
writel(0, q_vector[q_idx].intr_reg.dyn_ctl);
}
/**
* idpf_vport_intr_buildreg_itr - Enable default interrupt generation settings
* @q_vector: pointer to q_vector
+ *
+ * Return: value to be written back to HW to enable interrupt generation
*/
static u32 idpf_vport_intr_buildreg_itr(struct idpf_q_vector *q_vector)
{
@@ -3451,8 +3461,12 @@ void idpf_vport_intr_update_itr_ena_irq(struct idpf_q_vector *q_vector)
/**
* idpf_vport_intr_req_irq - get MSI-X vectors from the OS for the vport
* @vport: main vport structure
+ * @rsrc: pointer to queue and vector resources
+ *
+ * Return: 0 on success, negative on failure
*/
-static int idpf_vport_intr_req_irq(struct idpf_vport *vport)
+static int idpf_vport_intr_req_irq(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
struct idpf_adapter *adapter = vport->adapter;
const char *drv_name, *if_name, *vec_name;
@@ -3461,11 +3475,11 @@ static int idpf_vport_intr_req_irq(struct idpf_vport *vport)
drv_name = dev_driver_string(&adapter->pdev->dev);
if_name = netdev_name(vport->netdev);
- for (vector = 0; vector < vport->num_q_vectors; vector++) {
- struct idpf_q_vector *q_vector = &vport->q_vectors[vector];
+ for (vector = 0; vector < rsrc->num_q_vectors; vector++) {
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[vector];
char *name;
- vidx = vport->q_vector_idxs[vector];
+ vidx = rsrc->q_vector_idxs[vector];
irq_num = adapter->msix_entries[vidx].vector;
if (q_vector->num_rxq && q_vector->num_txq)
@@ -3495,9 +3509,9 @@ static int idpf_vport_intr_req_irq(struct idpf_vport *vport)
free_q_irqs:
while (--vector >= 0) {
- vidx = vport->q_vector_idxs[vector];
+ vidx = rsrc->q_vector_idxs[vector];
irq_num = adapter->msix_entries[vidx].vector;
- kfree(free_irq(irq_num, &vport->q_vectors[vector]));
+ kfree(free_irq(irq_num, &rsrc->q_vectors[vector]));
}
return err;
@@ -3526,15 +3540,16 @@ void idpf_vport_intr_write_itr(struct idpf_q_vector *q_vector, u16 itr, bool tx)
/**
* idpf_vport_intr_ena_irq_all - Enable IRQ for the given vport
* @vport: main vport structure
+ * @rsrc: pointer to queue and vector resources
*/
-static void idpf_vport_intr_ena_irq_all(struct idpf_vport *vport)
+static void idpf_vport_intr_ena_irq_all(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
bool dynamic;
- int q_idx;
u16 itr;
- for (q_idx = 0; q_idx < vport->num_q_vectors; q_idx++) {
- struct idpf_q_vector *qv = &vport->q_vectors[q_idx];
+ for (u16 q_idx = 0; q_idx < rsrc->num_q_vectors; q_idx++) {
+ struct idpf_q_vector *qv = &rsrc->q_vectors[q_idx];
/* Set the initial ITR values */
if (qv->num_txq) {
@@ -3558,16 +3573,39 @@ static void idpf_vport_intr_ena_irq_all(struct idpf_vport *vport)
}
}
+/**
+ * idpf_vport_intr_dis_dim_all - Disable DIM work for all q_vectors
+ * @rsrc: pointer to queue and vector resources
+ *
+ * The DIM works are embedded in the q_vector array that
+ * idpf_vport_intr_rel() frees, and the poll arms them after
+ * napi_complete_done() has already cleared NAPI_STATE_SCHED. Disable
+ * rather than just cancel, so that a poll tail still running past
+ * napi_disable() cannot queue them again behind the drain.
+ */
+static void idpf_vport_intr_dis_dim_all(struct idpf_q_vec_rsrc *rsrc)
+{
+ for (u16 v_idx = 0; v_idx < rsrc->num_q_vectors; v_idx++) {
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[v_idx];
+
+ disable_work_sync(&q_vector->tx_dim.work);
+ disable_work_sync(&q_vector->rx_dim.work);
+ }
+}
+
/**
* idpf_vport_intr_deinit - Release all vector associations for the vport
* @vport: main vport structure
+ * @rsrc: pointer to queue and vector resources
*/
-void idpf_vport_intr_deinit(struct idpf_vport *vport)
+void idpf_vport_intr_deinit(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
- idpf_vport_intr_dis_irq_all(vport);
- idpf_vport_intr_napi_dis_all(vport);
- idpf_vport_intr_napi_del_all(vport);
- idpf_vport_intr_rel_irq(vport);
+ idpf_vport_intr_dis_irq_all(rsrc);
+ idpf_vport_intr_napi_dis_all(rsrc);
+ idpf_vport_intr_dis_dim_all(rsrc);
+ idpf_vport_intr_napi_del_all(rsrc);
+ idpf_vport_intr_rel_irq(vport, rsrc);
}
/**
@@ -3639,16 +3677,13 @@ static void idpf_init_dim(struct idpf_q_vector *qv)
/**
* idpf_vport_intr_napi_ena_all - Enable NAPI for all q_vectors in the vport
- * @vport: main vport structure
+ * @rsrc: pointer to queue and vector resources
*/
-static void idpf_vport_intr_napi_ena_all(struct idpf_vport *vport)
+static void idpf_vport_intr_napi_ena_all(struct idpf_q_vec_rsrc *rsrc)
{
- int q_idx;
-
- for (q_idx = 0; q_idx < vport->num_q_vectors; q_idx++) {
- struct idpf_q_vector *q_vector = &vport->q_vectors[q_idx];
+ for (u16 q_idx = 0; q_idx < rsrc->num_q_vectors; q_idx++) {
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[q_idx];
- idpf_init_dim(q_vector);
napi_enable(&q_vector->napi);
}
}
@@ -3659,7 +3694,7 @@ static void idpf_vport_intr_napi_ena_all(struct idpf_vport *vport)
* @budget: Used to determine if we are in netpoll
* @cleaned: returns number of packets cleaned
*
- * Returns false if clean is not complete else returns true
+ * Return: %false if clean is not complete else returns %true
*/
static bool idpf_tx_splitq_clean_all(struct idpf_q_vector *q_vec,
int budget, int *cleaned)
@@ -3686,7 +3721,7 @@ static bool idpf_tx_splitq_clean_all(struct idpf_q_vector *q_vec,
* @budget: Used to determine if we are in netpoll
* @cleaned: returns number of packets cleaned
*
- * Returns false if clean is not complete else returns true
+ * Return: %false if clean is not complete else returns %true
*/
static bool idpf_rx_splitq_clean_all(struct idpf_q_vector *q_vec, int budget,
int *cleaned)
@@ -3725,6 +3760,8 @@ static bool idpf_rx_splitq_clean_all(struct idpf_q_vector *q_vec, int budget,
* idpf_vport_splitq_napi_poll - NAPI handler
* @napi: struct from which you get q_vector
* @budget: budget provided by stack
+ *
+ * Return: how many packets were cleaned
*/
static int idpf_vport_splitq_napi_poll(struct napi_struct *napi, int budget)
{
@@ -3773,10 +3810,12 @@ static int idpf_vport_splitq_napi_poll(struct napi_struct *napi, int budget)
/**
* idpf_vport_intr_map_vector_to_qs - Map vectors to queues
* @vport: virtual port
+ * @rsrc: pointer to queue and vector resources
*
* Mapping for vectors to queues
*/
-static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport)
+static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
bool split = idpf_is_queue_model_split(vport->rxq_model);
u16 num_txq_grp = vport->num_txq_grp;
@@ -3787,7 +3826,7 @@ static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport)
for (i = 0, qv_idx = 0; i < vport->num_rxq_grp; i++) {
u16 num_rxq;
- if (qv_idx >= vport->num_q_vectors)
+ if (qv_idx >= rsrc->num_q_vectors)
qv_idx = 0;
rx_qgrp = &vport->rxq_grps[i];
@@ -3803,7 +3842,7 @@ static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport)
q = &rx_qgrp->splitq.rxq_sets[j]->rxq;
else
q = rx_qgrp->singleq.rxqs[j];
- q->q_vector = &vport->q_vectors[qv_idx];
+ q->q_vector = &rsrc->q_vectors[qv_idx];
q_index = q->q_vector->num_rxq;
q->q_vector->rx[q_index] = q;
q->q_vector->num_rxq++;
@@ -3817,7 +3856,7 @@ static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport)
struct idpf_buf_queue *bufq;
bufq = &rx_qgrp->splitq.bufq_sets[j].bufq;
- bufq->q_vector = &vport->q_vectors[qv_idx];
+ bufq->q_vector = &rsrc->q_vectors[qv_idx];
q_index = bufq->q_vector->num_bufq;
bufq->q_vector->bufq[q_index] = bufq;
bufq->q_vector->num_bufq++;
@@ -3832,7 +3871,7 @@ static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport)
for (i = 0, qv_idx = 0; i < num_txq_grp; i++) {
u16 num_txq;
- if (qv_idx >= vport->num_q_vectors)
+ if (qv_idx >= rsrc->num_q_vectors)
qv_idx = 0;
tx_qgrp = &vport->txq_grps[i];
@@ -3842,14 +3881,14 @@ static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport)
struct idpf_tx_queue *q;
q = tx_qgrp->txqs[j];
- q->q_vector = &vport->q_vectors[qv_idx];
+ q->q_vector = &rsrc->q_vectors[qv_idx];
q->q_vector->tx[q->q_vector->num_txq++] = q;
}
if (split) {
struct idpf_compl_queue *q = tx_qgrp->complq;
- q->q_vector = &vport->q_vectors[qv_idx];
+ q->q_vector = &rsrc->q_vectors[qv_idx];
q->q_vector->complq[q->q_vector->num_complq++] = q;
}
@@ -3860,10 +3899,14 @@ static void idpf_vport_intr_map_vector_to_qs(struct idpf_vport *vport)
/**
* idpf_vport_intr_init_vec_idx - Initialize the vector indexes
* @vport: virtual port
+ * @rsrc: pointer to queue and vector resources
+ *
+ * Initialize vector indexes with values returned over mailbox.
*
- * Initialize vector indexes with values returened over mailbox
+ * Return: 0 on success, negative on failure
*/
-static int idpf_vport_intr_init_vec_idx(struct idpf_vport *vport)
+static int idpf_vport_intr_init_vec_idx(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
struct idpf_adapter *adapter = vport->adapter;
struct virtchnl2_alloc_vectors *ac;
@@ -3872,8 +3915,8 @@ static int idpf_vport_intr_init_vec_idx(struct idpf_vport *vport)
ac = adapter->req_vec_chunks;
if (!ac) {
- for (i = 0; i < vport->num_q_vectors; i++)
- vport->q_vectors[i].v_idx = vport->q_vector_idxs[i];
+ for (i = 0; i < rsrc->num_q_vectors; i++)
+ rsrc->q_vectors[i].v_idx = rsrc->q_vector_idxs[i];
return 0;
}
@@ -3885,8 +3928,8 @@ static int idpf_vport_intr_init_vec_idx(struct idpf_vport *vport)
idpf_get_vec_ids(adapter, vecids, total_vecs, &ac->vchunks);
- for (i = 0; i < vport->num_q_vectors; i++)
- vport->q_vectors[i].v_idx = vecids[vport->q_vector_idxs[i]];
+ for (i = 0; i < rsrc->num_q_vectors; i++)
+ rsrc->q_vectors[i].v_idx = vecids[rsrc->q_vector_idxs[i]];
kfree(vecids);
@@ -3896,19 +3939,20 @@ static int idpf_vport_intr_init_vec_idx(struct idpf_vport *vport)
/**
* idpf_vport_intr_napi_add_all- Register napi handler for all qvectors
* @vport: virtual port structure
+ * @rsrc: pointer to queue and vector resources
*/
-static void idpf_vport_intr_napi_add_all(struct idpf_vport *vport)
+static void idpf_vport_intr_napi_add_all(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
int (*napi_poll)(struct napi_struct *napi, int budget);
- u16 v_idx;
if (idpf_is_queue_model_split(vport->txq_model))
napi_poll = idpf_vport_splitq_napi_poll;
else
napi_poll = idpf_vport_singleq_napi_poll;
- for (v_idx = 0; v_idx < vport->num_q_vectors; v_idx++) {
- struct idpf_q_vector *q_vector = &vport->q_vectors[v_idx];
+ for (u16 v_idx = 0; v_idx < rsrc->num_q_vectors; v_idx++) {
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[v_idx];
netif_napi_add(vport->netdev, &q_vector->napi, napi_poll);
@@ -3921,40 +3965,46 @@ static void idpf_vport_intr_napi_add_all(struct idpf_vport *vport)
/**
* idpf_vport_intr_alloc - Allocate memory for interrupt vectors
* @vport: virtual port
+ * @rsrc: pointer to queue and vector resources
+ *
+ * Allocate one q_vector per queue interrupt.
*
- * We allocate one q_vector per queue interrupt. If allocation fails we
- * return -ENOMEM.
+ * Return: 0 on success, if allocation fails we return -ENOMEM.
*/
-int idpf_vport_intr_alloc(struct idpf_vport *vport)
+int idpf_vport_intr_alloc(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
u16 txqs_per_vector, rxqs_per_vector, bufqs_per_vector;
struct idpf_vport_user_config_data *user_config;
struct idpf_q_vector *q_vector;
struct idpf_q_coalesce *q_coal;
- u32 complqs_per_vector, v_idx;
+ u32 complqs_per_vector;
u16 idx = vport->idx;
user_config = &vport->adapter->vport_config[idx]->user_config;
- vport->q_vectors = kcalloc(vport->num_q_vectors,
- sizeof(struct idpf_q_vector), GFP_KERNEL);
- if (!vport->q_vectors)
+
+ rsrc->q_vectors = kcalloc(rsrc->num_q_vectors,
+ sizeof(struct idpf_q_vector), GFP_KERNEL);
+ if (!rsrc->q_vectors)
return -ENOMEM;
txqs_per_vector = DIV_ROUND_UP(vport->num_txq_grp,
- vport->num_q_vectors);
+ rsrc->num_q_vectors);
rxqs_per_vector = DIV_ROUND_UP(vport->num_rxq_grp,
- vport->num_q_vectors);
+ rsrc->num_q_vectors);
bufqs_per_vector = vport->num_bufqs_per_qgrp *
DIV_ROUND_UP(vport->num_rxq_grp,
- vport->num_q_vectors);
+ rsrc->num_q_vectors);
complqs_per_vector = DIV_ROUND_UP(vport->num_txq_grp,
- vport->num_q_vectors);
+ rsrc->num_q_vectors);
- for (v_idx = 0; v_idx < vport->num_q_vectors; v_idx++) {
- q_vector = &vport->q_vectors[v_idx];
+ for (u16 v_idx = 0; v_idx < rsrc->num_q_vectors; v_idx++) {
+ q_vector = &rsrc->q_vectors[v_idx];
q_coal = &user_config->q_coalesce[v_idx];
q_vector->vport = vport;
+ idpf_init_dim(q_vector);
+
q_vector->tx_itr_value = q_coal->tx_coalesce_usecs;
q_vector->tx_intr_mode = q_coal->tx_intr_mode;
q_vector->tx_itr_idx = VIRTCHNL2_ITR_IDX_1;
@@ -3995,7 +4045,7 @@ int idpf_vport_intr_alloc(struct idpf_vport *vport)
return 0;
error:
- idpf_vport_intr_rel(vport);
+ idpf_vport_intr_rel(rsrc);
return -ENOMEM;
}
@@ -4003,47 +4053,48 @@ int idpf_vport_intr_alloc(struct idpf_vport *vport)
/**
* idpf_vport_intr_init - Setup all vectors for the given vport
* @vport: virtual port
+ * @rsrc: pointer to queue and vector resources
*
- * Returns 0 on success or negative on failure
+ * Return: 0 on success or negative on failure
*/
-int idpf_vport_intr_init(struct idpf_vport *vport)
+int idpf_vport_intr_init(struct idpf_vport *vport, struct idpf_q_vec_rsrc *rsrc)
{
int err;
- err = idpf_vport_intr_init_vec_idx(vport);
+ err = idpf_vport_intr_init_vec_idx(vport, rsrc);
if (err)
return err;
- idpf_vport_intr_map_vector_to_qs(vport);
- idpf_vport_intr_napi_add_all(vport);
+ idpf_vport_intr_map_vector_to_qs(vport, rsrc);
+ idpf_vport_intr_napi_add_all(vport, rsrc);
- err = vport->adapter->dev_ops.reg_ops.intr_reg_init(vport);
+ err = vport->adapter->dev_ops.reg_ops.intr_reg_init(vport, rsrc);
if (err)
goto unroll_vectors_alloc;
- err = idpf_vport_intr_req_irq(vport);
+ err = idpf_vport_intr_req_irq(vport, rsrc);
if (err)
goto unroll_vectors_alloc;
return 0;
unroll_vectors_alloc:
- idpf_vport_intr_napi_del_all(vport);
+ idpf_vport_intr_napi_del_all(rsrc);
return err;
}
-void idpf_vport_intr_ena(struct idpf_vport *vport)
+void idpf_vport_intr_ena(struct idpf_vport *vport, struct idpf_q_vec_rsrc *rsrc)
{
- idpf_vport_intr_napi_ena_all(vport);
- idpf_vport_intr_ena_irq_all(vport);
+ idpf_vport_intr_napi_ena_all(rsrc);
+ idpf_vport_intr_ena_irq_all(vport, rsrc);
}
/**
* idpf_config_rss - Send virtchnl messages to configure RSS
* @vport: virtual port
*
- * Return 0 on success, negative on failure
+ * Return: 0 on success, negative on failure
*/
int idpf_config_rss(struct idpf_vport *vport)
{
diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethernet/intel/idpf/idpf_txrx.h
index 2d9960dd51ad..9faa19e2e174 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h
+++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h
@@ -1011,12 +1011,16 @@ int idpf_vport_calc_total_qs(struct idpf_adapter *adapter, u16 vport_index,
void idpf_vport_calc_num_q_groups(struct idpf_vport *vport);
int idpf_vport_queues_alloc(struct idpf_vport *vport);
void idpf_vport_queues_rel(struct idpf_vport *vport);
-void idpf_vport_intr_rel(struct idpf_vport *vport);
-int idpf_vport_intr_alloc(struct idpf_vport *vport);
+void idpf_vport_intr_rel(struct idpf_q_vec_rsrc *rsrc);
+int idpf_vport_intr_alloc(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc);
void idpf_vport_intr_update_itr_ena_irq(struct idpf_q_vector *q_vector);
-void idpf_vport_intr_deinit(struct idpf_vport *vport);
-int idpf_vport_intr_init(struct idpf_vport *vport);
-void idpf_vport_intr_ena(struct idpf_vport *vport);
+void idpf_vport_intr_deinit(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc);
+int idpf_vport_intr_init(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc);
+void idpf_vport_intr_ena(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc);
void idpf_fill_dflt_rss_lut(struct idpf_vport *vport);
int idpf_config_rss(struct idpf_vport *vport);
int idpf_init_rss_lut(struct idpf_vport *vport);
diff --git a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c b/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
index aba828abcb17..61d6f774e2f6 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_vf_dev.c
@@ -66,11 +66,13 @@ static void idpf_vf_mb_intr_reg_init(struct idpf_adapter *adapter)
/**
* idpf_vf_intr_reg_init - Initialize interrupt registers
* @vport: virtual port structure
+ * @rsrc: pointer to queue and vector resources
*/
-static int idpf_vf_intr_reg_init(struct idpf_vport *vport)
+static int idpf_vf_intr_reg_init(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
struct idpf_adapter *adapter = vport->adapter;
- int num_vecs = vport->num_q_vectors;
+ u16 num_vecs = rsrc->num_q_vectors;
struct idpf_vec_regs *reg_vals;
int num_regs, i, err = 0;
u32 rx_itr, tx_itr;
@@ -89,8 +91,8 @@ static int idpf_vf_intr_reg_init(struct idpf_vport *vport)
}
for (i = 0; i < num_vecs; i++) {
- struct idpf_q_vector *q_vector = &vport->q_vectors[i];
- u16 vec_id = vport->q_vector_idxs[i] - IDPF_MBX_Q_VEC;
+ struct idpf_q_vector *q_vector = &rsrc->q_vectors[i];
+ u16 vec_id = rsrc->q_vector_idxs[i] - IDPF_MBX_Q_VEC;
struct idpf_intr_reg *intr = &q_vector->intr_reg;
u32 spacing;
diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
index c4ae7d62aaf8..4e49e4673dd6 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.c
@@ -1993,6 +1993,7 @@ int idpf_send_enable_queues_msg(struct idpf_vport *vport)
*/
int idpf_send_disable_queues_msg(struct idpf_vport *vport)
{
+ struct idpf_q_vec_rsrc *rsrc = &vport->dflt_qv_rsrc;
int err, i;
err = idpf_send_ena_dis_queues_msg(vport, false);
@@ -2007,8 +2008,8 @@ int idpf_send_disable_queues_msg(struct idpf_vport *vport)
/* schedule the napi to receive all the marker packets */
local_bh_disable();
- for (i = 0; i < vport->num_q_vectors; i++)
- napi_schedule(&vport->q_vectors[i].napi);
+ for (u16 i = 0; i < rsrc->num_q_vectors; i++)
+ napi_schedule(&rsrc->q_vectors[i].napi);
local_bh_enable();
return idpf_wait_for_marker_event(vport);
@@ -3118,25 +3119,27 @@ void idpf_vc_core_deinit(struct idpf_adapter *adapter)
/**
* idpf_vport_alloc_vec_indexes - Get relative vector indexes
* @vport: virtual port data struct
+ * @rsrc: pointer to queue and vector resources
*
* This function requests the vector information required for the vport and
* stores the vector indexes received from the 'global vector distribution'
* in the vport's queue vectors array.
*
- * Return 0 on success, error on failure
+ * Return: 0 on success, error on failure
*/
-int idpf_vport_alloc_vec_indexes(struct idpf_vport *vport)
+int idpf_vport_alloc_vec_indexes(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc)
{
struct idpf_vector_info vec_info;
int num_alloc_vecs;
- vec_info.num_curr_vecs = vport->num_q_vectors;
+ vec_info.num_curr_vecs = rsrc->num_q_vectors;
vec_info.num_req_vecs = max(vport->num_txq, vport->num_rxq);
vec_info.default_vport = vport->default_vport;
vec_info.index = vport->idx;
num_alloc_vecs = idpf_req_rel_vector_indexes(vport->adapter,
- vport->q_vector_idxs,
+ rsrc->q_vector_idxs,
&vec_info);
if (num_alloc_vecs <= 0) {
dev_err(&vport->adapter->pdev->dev, "Vector distribution failed: %d\n",
@@ -3144,7 +3147,7 @@ int idpf_vport_alloc_vec_indexes(struct idpf_vport *vport)
return -EINVAL;
}
- vport->num_q_vectors = num_alloc_vecs;
+ rsrc->num_q_vectors = num_alloc_vecs;
return 0;
}
@@ -3158,6 +3161,7 @@ int idpf_vport_alloc_vec_indexes(struct idpf_vport *vport)
*/
void idpf_vport_init(struct idpf_vport *vport, struct idpf_vport_max_q *max_q)
{
+ struct idpf_q_vec_rsrc *rsrc = &vport->dflt_qv_rsrc;
struct idpf_adapter *adapter = vport->adapter;
struct virtchnl2_create_vport *vport_msg;
struct idpf_vport_config *vport_config;
@@ -3196,7 +3200,7 @@ void idpf_vport_init(struct idpf_vport *vport, struct idpf_vport_max_q *max_q)
idpf_vport_init_num_qs(vport, vport_msg);
idpf_vport_calc_num_q_desc(vport);
idpf_vport_calc_num_q_groups(vport);
- idpf_vport_alloc_vec_indexes(vport);
+ idpf_vport_alloc_vec_indexes(vport, rsrc);
vport->crc_enable = adapter->crc_enable;
}
diff --git a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
index 23271cf0a216..0bfe44fa0385 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
+++ b/drivers/net/ethernet/intel/idpf/idpf_virtchnl.h
@@ -45,7 +45,8 @@ int idpf_send_enable_queues_msg(struct idpf_vport *vport);
int idpf_send_disable_queues_msg(struct idpf_vport *vport);
int idpf_send_config_queues_msg(struct idpf_vport *vport);
-int idpf_vport_alloc_vec_indexes(struct idpf_vport *vport);
+int idpf_vport_alloc_vec_indexes(struct idpf_vport *vport,
+ struct idpf_q_vec_rsrc *rsrc);
int idpf_get_vec_ids(struct idpf_adapter *adapter,
u16 *vecids, int num_vecids,
struct virtchnl2_vector_chunks *chunks);
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index 325a3a657249..8096b46b654f 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -5099,7 +5099,8 @@ static int mvpp2_change_mtu(struct net_device *dev, int mtu)
netdev_warn(dev, "mtu %d too high, switching to shared buffers", mtu);
mvpp2_bm_switch_buffers(priv, false);
}
- } else {
+ } else if (priv->hw_version >= MVPP22 &&
+ mvpp2_get_nrxqs(priv) * 2 <= MVPP2_BM_MAX_POOLS) {
bool jumbo = false;
int i;
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/common.h b/drivers/net/ethernet/marvell/octeontx2/af/common.h
index 2436c1ff9ba4..680cbee17ca6 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/common.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/common.h
@@ -7,6 +7,10 @@
#ifndef COMMON_H
#define COMMON_H
+#include <linux/dma-mapping.h>
+#include <linux/gfp.h>
+#include <linux/mm.h>
+
#include "rvu_struct.h"
#define OTX2_ALIGN 128 /* Align to cacheline */
@@ -44,6 +48,33 @@ struct qmem {
u32 qsize;
};
+static inline void *otx2_dma_alloc_coherent(struct device *dev, size_t size,
+ dma_addr_t *dma_handle)
+{
+ dma_addr_t dma_addr;
+ void *vaddr;
+
+ vaddr = kzalloc(size, GFP_KERNEL);
+ if (!vaddr)
+ return NULL;
+
+ dma_addr = dma_map_single(dev, vaddr, size, DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(dev, dma_addr)) {
+ kfree(vaddr);
+ return NULL;
+ }
+
+ *dma_handle = dma_addr;
+ return vaddr;
+}
+
+static inline void otx2_dma_free_coherent(struct device *dev, size_t size,
+ void *vaddr, dma_addr_t dma_handle)
+{
+ dma_unmap_single(dev, dma_handle, size, DMA_BIDIRECTIONAL);
+ kfree(vaddr);
+}
+
static inline int qmem_alloc(struct device *dev, struct qmem **q,
int qsize, int entry_sz)
{
@@ -60,8 +91,11 @@ static inline int qmem_alloc(struct device *dev, struct qmem **q,
qmem->entry_sz = entry_sz;
qmem->alloc_sz = (qsize * entry_sz) + OTX2_ALIGN;
- qmem->base = dma_alloc_attrs(dev, qmem->alloc_sz, &qmem->iova,
- GFP_KERNEL, DMA_ATTR_FORCE_CONTIGUOUS);
+
+ if (get_order(PAGE_ALIGN(qmem->alloc_sz)) > MAX_PAGE_ORDER)
+ return -ENOMEM;
+
+ qmem->base = otx2_dma_alloc_coherent(dev, qmem->alloc_sz, &qmem->iova);
if (!qmem->base)
return -ENOMEM;
@@ -80,10 +114,9 @@ static inline void qmem_free(struct device *dev, struct qmem *qmem)
return;
if (qmem->base)
- dma_free_attrs(dev, qmem->alloc_sz,
- qmem->base - qmem->align,
- qmem->iova - qmem->align,
- DMA_ATTR_FORCE_CONTIGUOUS);
+ otx2_dma_free_coherent(dev, qmem->alloc_sz,
+ qmem->base - qmem->align,
+ qmem->iova - qmem->align);
devm_kfree(dev, qmem);
}
diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
index 42a4f52ecdc2..4c9ebbb1b4f1 100644
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
@@ -4347,6 +4347,10 @@ static int mtk_unreg_dev(struct mtk_eth *eth)
mac = netdev_priv(eth->netdev[i]);
if (MTK_HAS_CAPS(eth->soc->caps, MTK_QDMA))
unregister_netdevice_notifier(&mac->device_notifier);
+
+ if (eth->netdev[i]->reg_state != NETREG_REGISTERED)
+ continue;
+
unregister_netdev(eth->netdev[i]);
}
@@ -5128,7 +5132,7 @@ static int mtk_probe(struct platform_device *pdev)
err = register_netdev(eth->netdev[i]);
if (err) {
dev_err(eth->dev, "error bringing up device\n");
- goto err_deinit_ppe;
+ goto err_unreg_netdev;
} else
netif_info(eth, probe, eth->netdev[i],
"mediatek frame engine at 0x%08lx, irq %d\n",
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c b/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c
index 0f5d7ea8956f..d8658fa517e7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/rep/bridge.c
@@ -89,9 +89,16 @@ mlx5_esw_bridge_lower_rep_vport_num_vhca_id_get(struct net_device *dev, struct m
struct net_device *lower_dev;
struct list_head *iter;
- if (netif_is_lag_master(dev) || mlx5e_eswitch_rep(dev))
- return mlx5_esw_bridge_rep_vport_num_vhca_id_get(dev, esw, vport_num,
- esw_owner_vhca_id);
+ if (netif_is_lag_master(dev) || mlx5e_eswitch_rep(dev)) {
+ struct net_device *rep;
+
+ rep = mlx5_esw_bridge_rep_vport_num_vhca_id_get(dev, esw, vport_num,
+ esw_owner_vhca_id);
+ if (rep && !mlx5_esw_bridge_port_exists(*vport_num, *esw_owner_vhca_id,
+ esw->br_offloads))
+ return NULL;
+ return rep;
+ }
netdev_for_each_lower_dev(dev, lower_dev, iter) {
struct net_device *rep;
@@ -108,6 +115,28 @@ mlx5_esw_bridge_lower_rep_vport_num_vhca_id_get(struct net_device *dev, struct m
return NULL;
}
+static bool mlx5_esw_bridge_rep_port_lookup(struct net_device *dev,
+ struct mlx5_esw_bridge_offloads *br_offloads,
+ u16 *vport_num, u16 *esw_owner_vhca_id)
+{
+ if (!mlx5_esw_bridge_rep_vport_num_vhca_id_get(dev, br_offloads->esw, vport_num,
+ esw_owner_vhca_id))
+ return false;
+
+ return mlx5_esw_bridge_port_exists(*vport_num, *esw_owner_vhca_id, br_offloads);
+}
+
+static bool mlx5_esw_bridge_lower_rep_port_lookup(struct net_device *dev,
+ struct mlx5_esw_bridge_offloads *br_offloads,
+ u16 *vport_num, u16 *esw_owner_vhca_id)
+{
+ if (!mlx5_esw_bridge_lower_rep_vport_num_vhca_id_get(dev, br_offloads->esw, vport_num,
+ esw_owner_vhca_id))
+ return false;
+
+ return mlx5_esw_bridge_port_exists(*vport_num, *esw_owner_vhca_id, br_offloads);
+}
+
static bool mlx5_esw_bridge_is_local(struct net_device *dev, struct net_device *rep,
struct mlx5_eswitch *esw)
{
@@ -222,8 +251,7 @@ mlx5_esw_bridge_port_obj_add(struct net_device *dev,
u16 vport_num, esw_owner_vhca_id;
int err;
- if (!mlx5_esw_bridge_rep_vport_num_vhca_id_get(dev, br_offloads->esw, &vport_num,
- &esw_owner_vhca_id))
+ if (!mlx5_esw_bridge_rep_port_lookup(dev, br_offloads, &vport_num, &esw_owner_vhca_id))
return 0;
port_obj_info->handled = true;
@@ -255,8 +283,7 @@ mlx5_esw_bridge_port_obj_del(struct net_device *dev,
const struct switchdev_obj_port_mdb *mdb;
u16 vport_num, esw_owner_vhca_id;
- if (!mlx5_esw_bridge_rep_vport_num_vhca_id_get(dev, br_offloads->esw, &vport_num,
- &esw_owner_vhca_id))
+ if (!mlx5_esw_bridge_rep_port_lookup(dev, br_offloads, &vport_num, &esw_owner_vhca_id))
return 0;
port_obj_info->handled = true;
@@ -287,8 +314,8 @@ mlx5_esw_bridge_port_obj_attr_set(struct net_device *dev,
u16 vport_num, esw_owner_vhca_id;
int err = 0;
- if (!mlx5_esw_bridge_lower_rep_vport_num_vhca_id_get(dev, br_offloads->esw, &vport_num,
- &esw_owner_vhca_id))
+ if (!mlx5_esw_bridge_lower_rep_port_lookup(dev, br_offloads, &vport_num,
+ &esw_owner_vhca_id))
return 0;
port_attr_info->handled = true;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c
index 92d5cfec3dc0..0fe3273834c9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_ct.c
@@ -1082,6 +1082,9 @@ mlx5_tc_ct_shared_counter_get(struct mlx5_tc_ct_priv *ct_priv,
spin_unlock_bh(&ct_priv->ht_lock);
+ if (rev_entry)
+ mlx5_tc_ct_entry_put(rev_entry);
+
create_counter:
shared_counter = mlx5_tc_ct_counter_create(ct_priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c
index 70d8cce7c173..bd66a5bf4acf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/macsec.c
@@ -1727,6 +1727,8 @@ void mlx5e_macsec_build_netdev(struct mlx5e_priv *priv)
mlx5_core_dbg(priv->mdev, "mlx5e: MACsec acceleration enabled\n");
netdev->macsec_ops = &macsec_offload_ops;
netdev->features |= NETIF_F_HW_MACSEC;
+ netdev->hw_features |= NETIF_F_HW_MACSEC;
+ netdev->vlan_features |= NETIF_F_HW_MACSEC;
netif_keep_dst(netdev);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index c7eb58fd6e6a..c5a2fa415227 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -5433,7 +5433,6 @@ static void mlx5e_build_nic_netdev(struct net_device *netdev)
netdev->vlan_features |= NETIF_F_SG;
netdev->vlan_features |= NETIF_F_HW_CSUM;
- netdev->vlan_features |= NETIF_F_HW_MACSEC;
netdev->vlan_features |= NETIF_F_GRO;
netdev->vlan_features |= NETIF_F_TSO;
netdev->vlan_features |= NETIF_F_TSO6;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 4eb1150e7154..eaa5d6c83a82 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -1983,11 +1983,14 @@ mlx5e_shampo_fill_skb_data(struct sk_buff *skb, struct mlx5e_rq *rq,
struct mlx5e_frag_page *frag_page,
u32 data_bcnt, u32 data_offset)
{
+ u32 page_size = BIT(rq->mpwqe.page_shift);
+
net_prefetchw(skb->data);
do {
/* Non-linear mode, hence non-XSK, which always uses PAGE_SIZE. */
- u32 pg_consumed_bytes = min_t(u32, PAGE_SIZE - data_offset, data_bcnt);
+ u32 pg_consumed_bytes = min_t(u32, page_size - data_offset,
+ data_bcnt);
unsigned int truesize = pg_consumed_bytes;
mlx5e_add_skb_frag(rq, skb, frag_page, data_offset,
@@ -2008,6 +2011,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
u16 headlen = min_t(u16, MLX5E_RX_MAX_HEAD, cqe_bcnt);
struct mlx5e_frag_page *head_page = frag_page;
struct mlx5e_xdp_buff *mxbuf = &rq->mxbuf;
+ u32 page_size = BIT(rq->mpwqe.page_shift);
u32 frag_offset = head_offset;
u32 byte_cnt = cqe_bcnt;
struct skb_shared_info *sinfo;
@@ -2051,9 +2055,9 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
linear_hr = skb_headroom(skb);
linear_data_len = headlen;
linear_frame_sz = MLX5_SKB_FRAG_SZ(skb_end_offset(skb));
- if (unlikely(frag_offset >= PAGE_SIZE)) {
+ if (unlikely(frag_offset >= page_size)) {
frag_page++;
- frag_offset -= PAGE_SIZE;
+ frag_offset -= page_size;
}
}
@@ -2065,7 +2069,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
while (byte_cnt) {
/* Non-linear mode, hence non-XSK, which always uses PAGE_SIZE. */
pg_consumed_bytes =
- min_t(u32, PAGE_SIZE - frag_offset, byte_cnt);
+ min_t(u32, page_size - frag_offset, byte_cnt);
if (test_bit(MLX5E_RQ_STATE_SHAMPO, &rq->state))
truesize += pg_consumed_bytes;
@@ -2102,7 +2106,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
nr_frags_free = old_nr_frags - sinfo->nr_frags;
if (unlikely(nr_frags_free)) {
frag_page -= nr_frags_free;
- truesize -= (nr_frags_free - 1) * PAGE_SIZE +
+ truesize -= (nr_frags_free - 1) * page_size +
ALIGN(pg_consumed_bytes,
BIT(rq->mpwqe.log_stride_sz));
}
@@ -2311,15 +2315,16 @@ mlx5e_shampo_flush_skb(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe, bool match)
rq->hw_gro_data->skb = NULL;
}
-static bool
-mlx5e_hw_gro_skb_has_enough_space(struct sk_buff *skb, u16 data_bcnt)
+static bool mlx5e_hw_gro_skb_has_enough_space(struct sk_buff *skb,
+ u16 data_bcnt,
+ u32 page_size)
{
int nr_frags = skb_shinfo(skb)->nr_frags;
- if (PAGE_SIZE >= GRO_LEGACY_MAX_SIZE)
+ if (page_size >= GRO_LEGACY_MAX_SIZE)
return skb->len + data_bcnt <= GRO_LEGACY_MAX_SIZE;
else
- return PAGE_SIZE * nr_frags + data_bcnt <= GRO_LEGACY_MAX_SIZE;
+ return page_size * nr_frags + data_bcnt <= GRO_LEGACY_MAX_SIZE;
}
static void mlx5e_handle_rx_cqe_mpwrq_shampo(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
@@ -2328,18 +2333,19 @@ static void mlx5e_handle_rx_cqe_mpwrq_shampo(struct mlx5e_rq *rq, struct mlx5_cq
u16 header_index = mlx5e_shampo_get_cqe_header_index(rq, cqe);
u32 wqe_offset = be32_to_cpu(cqe->shampo.data_offset);
u16 cstrides = mpwrq_get_cqe_consumed_strides(cqe);
- u32 data_offset = wqe_offset & (PAGE_SIZE - 1);
u32 cqe_bcnt = mpwrq_get_cqe_byte_cnt(cqe);
u16 wqe_id = be16_to_cpu(cqe->wqe_id);
- u32 page_idx = wqe_offset >> PAGE_SHIFT;
u16 head_size = cqe->shampo.header_size;
struct sk_buff **skb = &rq->hw_gro_data->skb;
bool flush = cqe->shampo.flush;
bool match = cqe->shampo.match;
+ u32 page_size = BIT(rq->mpwqe.page_shift);
struct mlx5e_rq_stats *stats = rq->stats;
struct mlx5e_rx_wqe_ll *wqe;
struct mlx5e_mpw_info *wi;
struct mlx5_wq_ll *wq;
+ u32 data_offset;
+ u32 page_idx;
wi = mlx5e_get_mpw_info(rq, wqe_id);
wi->consumed_strides += cstrides;
@@ -2355,7 +2361,16 @@ static void mlx5e_handle_rx_cqe_mpwrq_shampo(struct mlx5e_rq *rq, struct mlx5_cq
goto mpwrq_cqe_out;
}
- if (*skb && (!match || !(mlx5e_hw_gro_skb_has_enough_space(*skb, data_bcnt)))) {
+ data_offset = wqe_offset & (page_size - 1);
+ page_idx = wqe_offset >> rq->mpwqe.page_shift;
+ if (unlikely(cqe_bcnt <= ETH_ZLEN + 2 * VLAN_HLEN)) {
+ match = false;
+ flush = true;
+ }
+
+ if (*skb &&
+ !(match && mlx5e_hw_gro_skb_has_enough_space(*skb, data_bcnt,
+ page_size))) {
match = false;
mlx5e_shampo_flush_skb(rq, cqe, match);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c
index 282cef59e617..684154914cf8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.c
@@ -1646,10 +1646,8 @@ int mlx5_esw_bridge_vport_unlink(struct net_device *br_netdev, u16 vport_num,
int err;
port = mlx5_esw_bridge_port_lookup(vport_num, esw_owner_vhca_id, br_offloads);
- if (!port) {
- NL_SET_ERR_MSG_MOD(extack, "Port is not attached to any bridge");
- return -EINVAL;
- }
+ if (!port)
+ return 0;
if (port->bridge->ifindex != br_netdev->ifindex) {
NL_SET_ERR_MSG_MOD(extack, "Port is attached to another bridge");
return -EINVAL;
@@ -1679,10 +1677,19 @@ int mlx5_esw_bridge_vport_peer_unlink(struct net_device *br_netdev, u16 vport_nu
struct mlx5_esw_bridge_offloads *br_offloads,
struct netlink_ext_ack *extack)
{
+ if (!MLX5_CAP_ESW(br_offloads->esw->dev, merged_eswitch))
+ return 0;
+
return mlx5_esw_bridge_vport_unlink(br_netdev, vport_num, esw_owner_vhca_id, br_offloads,
extack);
}
+bool mlx5_esw_bridge_port_exists(u16 vport_num, u16 esw_owner_vhca_id,
+ struct mlx5_esw_bridge_offloads *br_offloads)
+{
+ return mlx5_esw_bridge_port_lookup(vport_num, esw_owner_vhca_id, br_offloads);
+}
+
int mlx5_esw_bridge_port_vlan_add(u16 vport_num, u16 esw_owner_vhca_id, u16 vid, u16 flags,
struct mlx5_esw_bridge_offloads *br_offloads,
struct netlink_ext_ack *extack)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h
index d6f539161993..a4e59cc21089 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/bridge.h
@@ -80,6 +80,8 @@ int mlx5_esw_bridge_vlan_proto_set(u16 vport_num, u16 esw_owner_vhca_id, u16 pro
struct mlx5_esw_bridge_offloads *br_offloads);
int mlx5_esw_bridge_mcast_set(u16 vport_num, u16 esw_owner_vhca_id, bool enable,
struct mlx5_esw_bridge_offloads *br_offloads);
+bool mlx5_esw_bridge_port_exists(u16 vport_num, u16 esw_owner_vhca_id,
+ struct mlx5_esw_bridge_offloads *br_offloads);
int mlx5_esw_bridge_port_vlan_add(u16 vport_num, u16 esw_owner_vhca_id, u16 vid, u16 flags,
struct mlx5_esw_bridge_offloads *br_offloads,
struct netlink_ext_ack *extack);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c
index 7b0766c89f4c..cd8ce2abcf18 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.c
@@ -29,6 +29,7 @@ struct mlx5_devcom_comp {
struct list_head comp_dev_list_head;
mlx5_devcom_event_handler_t handler;
struct kref ref;
+ int nr_devs;
bool ready;
struct rw_semaphore sem;
struct lock_class_key lock_key;
@@ -158,6 +159,7 @@ devcom_alloc_comp_dev(struct mlx5_devcom_dev *devc,
down_write(&comp->sem);
list_add_tail(&devcom->list, &comp->comp_dev_list_head);
+ WRITE_ONCE(comp->nr_devs, comp->nr_devs + 1);
up_write(&comp->sem);
return devcom;
@@ -170,6 +172,7 @@ devcom_free_comp_dev(struct mlx5_devcom_comp_dev *devcom)
down_write(&comp->sem);
list_del(&devcom->list);
+ WRITE_ONCE(comp->nr_devs, comp->nr_devs - 1);
up_write(&comp->sem);
kref_put(&devcom->devc->ref, mlx5_devcom_dev_release);
@@ -260,7 +263,7 @@ int mlx5_devcom_comp_get_size(struct mlx5_devcom_comp_dev *devcom)
{
struct mlx5_devcom_comp *comp = devcom->comp;
- return kref_read(&comp->ref);
+ return READ_ONCE(comp->nr_devs);
}
int mlx5_devcom_send_event(struct mlx5_devcom_comp_dev *devcom,
diff --git a/drivers/net/ethernet/meta/fbnic/fbnic_csr.h b/drivers/net/ethernet/meta/fbnic/fbnic_csr.h
index e91b4432fddd..2da0f5f8882b 100644
--- a/drivers/net/ethernet/meta/fbnic/fbnic_csr.h
+++ b/drivers/net/ethernet/meta/fbnic/fbnic_csr.h
@@ -700,6 +700,7 @@ enum {
/* PUL User Registers */
#define FBNIC_CSR_START_PUL_USER 0x31000 /* CSR section delimiter */
#define FBNIC_PUL_OB_TLP_HDR_AW_CFG 0x3103d /* 0xc40f4 */
+#define FBNIC_PUL_OB_TLP_HDR_AW_CFG_FLUSH_MODE CSR_BIT(20)
#define FBNIC_PUL_OB_TLP_HDR_AW_CFG_FLUSH CSR_BIT(19)
#define FBNIC_PUL_OB_TLP_HDR_AW_CFG_BME CSR_BIT(18)
#define FBNIC_PUL_OB_TLP_HDR_AR_CFG 0x3103e /* 0xc40f8 */
diff --git a/drivers/net/ethernet/meta/fbnic/fbnic_fw.c b/drivers/net/ethernet/meta/fbnic/fbnic_fw.c
index 6f606bdfd229..175ad17bd47b 100644
--- a/drivers/net/ethernet/meta/fbnic/fbnic_fw.c
+++ b/drivers/net/ethernet/meta/fbnic/fbnic_fw.c
@@ -60,8 +60,15 @@ static void fbnic_mbx_reset_desc_ring(struct fbnic_dev *fbd, int mbx_idx)
*/
switch (mbx_idx) {
case FBNIC_IPC_MBX_RX_IDX:
+ /* Clearing BME blocks the device from writing to the host
+ * but leaves the requests parked in the write pipeline. The
+ * write path only clears outstanding requests when both FLUSH
+ * and FLUSH_MODE are set; FLUSH_MODE lets them drain without
+ * landing on the host.
+ */
wr32(fbd, FBNIC_PUL_OB_TLP_HDR_AW_CFG,
- FBNIC_PUL_OB_TLP_HDR_AW_CFG_FLUSH);
+ FBNIC_PUL_OB_TLP_HDR_AW_CFG_FLUSH |
+ FBNIC_PUL_OB_TLP_HDR_AW_CFG_FLUSH_MODE);
break;
case FBNIC_IPC_MBX_TX_IDX:
wr32(fbd, FBNIC_PUL_OB_TLP_HDR_AR_CFG,
diff --git a/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c b/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
index 16a0e30f8aaa..38e96b003ea0 100644
--- a/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
+++ b/drivers/net/ethernet/meta/fbnic/fbnic_txrx.c
@@ -1162,7 +1162,7 @@ int fbnic_alloc_napi_vectors(struct fbnic_net *fbn)
int err;
/* Allocate 1 Tx queue per napi vector */
- if (num_napi < FBNIC_MAX_TXQS && num_napi == num_tx + num_rx) {
+ if (num_napi <= FBNIC_MAX_TXQS && num_napi == num_tx + num_rx) {
while (num_tx) {
err = fbnic_alloc_napi_vector(fbd, fbn,
num_napi, v_idx,
diff --git a/drivers/net/ethernet/microchip/lan743x_main.c b/drivers/net/ethernet/microchip/lan743x_main.c
index d4622efc5b2b..0d848c8e7d57 100644
--- a/drivers/net/ethernet/microchip/lan743x_main.c
+++ b/drivers/net/ethernet/microchip/lan743x_main.c
@@ -2587,7 +2587,7 @@ static int lan743x_rx_process_buffer(struct lan743x_rx *rx)
rx->adapter->netdev);
if (rx->adapter->netdev->features & NETIF_F_RXCSUM) {
if (!is_ice && !is_tce && !is_icsm)
- skb->ip_summed = CHECKSUM_UNNECESSARY;
+ rx->skb_head->ip_summed = CHECKSUM_UNNECESSARY;
}
netdev_dbg(netdev, "sending %d byte frame to OS",
rx->skb_head->len);
diff --git a/drivers/net/ethernet/netronome/nfp/crypto/ipsec.c b/drivers/net/ethernet/netronome/nfp/crypto/ipsec.c
index 515069d5637b..3925a00aeaf7 100644
--- a/drivers/net/ethernet/netronome/nfp/crypto/ipsec.c
+++ b/drivers/net/ethernet/netronome/nfp/crypto/ipsec.c
@@ -637,11 +637,12 @@ int nfp_net_ipsec_rx(struct nfp_meta_parsed *meta, struct sk_buff *skb)
xa_lock(&nn->xa_ipsec);
x = xa_load(&nn->xa_ipsec, saidx);
+ if (x)
+ xfrm_state_hold(x);
xa_unlock(&nn->xa_ipsec);
if (!x)
return -EINVAL;
- xfrm_state_hold(x);
sp->xvec[sp->len++] = x;
sp->olen++;
xo = xfrm_offload(skb);
diff --git a/drivers/net/ethernet/socionext/netsec.c b/drivers/net/ethernet/socionext/netsec.c
index 5ab8b81b84e6..e96e22dd3080 100644
--- a/drivers/net/ethernet/socionext/netsec.c
+++ b/drivers/net/ethernet/socionext/netsec.c
@@ -2146,6 +2146,7 @@ static int netsec_probe(struct platform_device *pdev)
pm_runtime_put_sync(&pdev->dev);
pm_runtime_disable(&pdev->dev);
free_ndev:
+ of_node_put(priv->phy_np);
free_netdev(ndev);
dev_err(&pdev->dev, "init failed\n");
@@ -2163,6 +2164,7 @@ static void netsec_remove(struct platform_device *pdev)
netif_napi_del(&priv->napi);
pm_runtime_disable(&pdev->dev);
+ of_node_put(priv->phy_np);
free_netdev(priv->ndev);
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c
index d152afa48d5c..e093bf4cd56a 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c
@@ -1709,8 +1709,11 @@ static int gmac_clk_enable(struct rk_priv_data *bsp_priv, bool enable)
return ret;
ret = clk_prepare_enable(bsp_priv->clk_phy);
- if (ret)
+ if (ret) {
+ clk_bulk_disable_unprepare(bsp_priv->num_clks,
+ bsp_priv->clks);
return ret;
+ }
if (bsp_priv->ops && bsp_priv->ops->set_clock_selection)
bsp_priv->ops->set_clock_selection(bsp_priv,
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c b/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
index e99401bcc1f8..67c886316cbd 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
@@ -490,11 +490,11 @@ static void dwmac4_set_sarc(struct dma_desc *p, u32 sarc_type)
p->des3 |= cpu_to_le32(sarc_type & TDES3_SA_INSERT_CTRL_MASK);
}
-static int set_16kib_bfsize(int mtu)
+static int set_16kib_bfsize(int len)
{
int ret = 0;
- if (unlikely(mtu >= BUF_SIZE_8KiB))
+ if (unlikely(len > BUF_SIZE_8KiB))
ret = BUF_SIZE_16KiB;
return ret;
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/hwif.h b/drivers/net/ethernet/stmicro/stmmac/hwif.h
index d5a9f01ecac5..4376d61db969 100644
--- a/drivers/net/ethernet/stmicro/stmmac/hwif.h
+++ b/drivers/net/ethernet/stmicro/stmmac/hwif.h
@@ -541,7 +541,7 @@ struct stmmac_ops {
#define stmmac_fpe_set_add_frag_size(__priv, __args...) \
stmmac_do_void_callback(__priv, mac, fpe_set_add_frag_size, __args)
#define stmmac_fpe_map_preemption_class(__priv, __args...) \
- stmmac_do_void_callback(__priv, mac, fpe_map_preemption_class, __args)
+ stmmac_do_callback(__priv, mac, fpe_map_preemption_class, __args)
/* PTP and HW Timer helpers */
struct stmmac_hwtimestamp {
@@ -587,7 +587,7 @@ struct stmmac_mode_ops {
unsigned int (*is_jumbo_frm) (int len, int ehn_desc);
int (*jumbo_frm)(struct stmmac_tx_queue *tx_q, struct sk_buff *skb,
int csum);
- int (*set_16kib_bfsize)(int mtu);
+ int (*set_16kib_bfsize)(int len);
void (*init_desc3)(struct dma_desc *p);
void (*refill_desc3)(struct stmmac_rx_queue *rx_q, struct dma_desc *p);
void (*clean_desc3)(struct stmmac_tx_queue *tx_q, struct dma_desc *p);
diff --git a/drivers/net/ethernet/stmicro/stmmac/ring_mode.c b/drivers/net/ethernet/stmicro/stmmac/ring_mode.c
index 45c14c1bb0ea..88a8d22812f5 100644
--- a/drivers/net/ethernet/stmicro/stmmac/ring_mode.c
+++ b/drivers/net/ethernet/stmicro/stmmac/ring_mode.c
@@ -128,10 +128,10 @@ static void clean_desc3(struct stmmac_tx_queue *tx_q, struct dma_desc *p)
p->des3 = 0;
}
-static int set_16kib_bfsize(int mtu)
+static int set_16kib_bfsize(int len)
{
int ret = 0;
- if (unlikely(mtu > BUF_SIZE_8KiB))
+ if (unlikely(len > BUF_SIZE_8KiB))
ret = BUF_SIZE_16KiB;
return ret;
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
index 2a37592a6281..ca8712a573ea 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
@@ -1206,8 +1206,6 @@ static int stmmac_get_ts_info(struct net_device *dev,
if (priv->ptp_clock)
info->phc_index = ptp_clock_index(priv->ptp_clock);
- else
- info->phc_index = 0;
info->tx_types = (1 << HWTSTAMP_TX_OFF) | (1 << HWTSTAMP_TX_ON);
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c
index 266211a40139..e39ee3ea0eff 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_selftests.c
@@ -13,6 +13,7 @@
#include <linux/ip.h>
#include <linux/phy.h>
#include <linux/udp.h>
+#include <net/dsa.h>
#include <net/pkt_cls.h>
#include <net/pkt_sched.h>
#include <net/tcp.h>
@@ -238,6 +239,10 @@ struct stmmac_test_priv {
struct stmmac_packet_attrs *packet;
struct packet_type pt;
struct completion comp;
+ __be16 packet_type;
+ int (*func)(struct sk_buff *skb, struct net_device *ndev,
+ struct packet_type *pt, struct net_device *orig_ndev);
+ bool capture_all;
int double_vlan;
int vlan_id;
int ok;
@@ -317,6 +322,52 @@ static int stmmac_test_loopback_validate(struct sk_buff *skb,
return 0;
}
+static int stmmac_sft_filter(struct sk_buff *skb, struct net_device *ndev,
+ struct packet_type *pt,
+ struct net_device *orig_ndev)
+{
+ struct stmmac_test_priv *tpriv = pt->af_packet_priv;
+ struct ethhdr *hdr = eth_hdr(skb);
+ int ret = 0;
+
+ if (hdr->h_proto == tpriv->packet_type) {
+ struct sk_buff *nskb = skb_clone(skb, GFP_ATOMIC);
+
+ if (nskb)
+ ret = tpriv->func(nskb, ndev, pt, orig_ndev);
+ }
+
+ kfree_skb(skb);
+ return ret;
+}
+
+static void stmmac_sft_add_pack(struct packet_type *pt)
+{
+ struct stmmac_test_priv *tpriv = pt->af_packet_priv;
+
+ if (netdev_uses_dsa(tpriv->pt.dev) || tpriv->capture_all) {
+ tpriv->packet_type = tpriv->pt.type;
+ tpriv->func = tpriv->pt.func;
+
+ /* DSA conduit will report ETH_P_XDSA, so our packet handler
+ * won't match. Let's register a ETH_P_ALL match and filter
+ * manually in stmmac_sft_filter. This is also useful for
+ * VLAN tests, to capture packets otherwise marked as
+ * OTHERHOST.
+ */
+ tpriv->pt.type = htons(ETH_P_ALL);
+ tpriv->pt.func = stmmac_sft_filter;
+ tpriv->pt.ignore_outgoing = true;
+ }
+
+ dev_add_pack(pt);
+}
+
+static void stmmac_sft_remove_pack(struct packet_type *pt)
+{
+ dev_remove_pack(pt);
+}
+
static int __stmmac_test_loopback(struct stmmac_priv *priv,
struct stmmac_packet_attrs *attr)
{
@@ -338,7 +389,7 @@ static int __stmmac_test_loopback(struct stmmac_priv *priv,
tpriv->packet = attr;
if (!attr->dont_wait)
- dev_add_pack(&tpriv->pt);
+ stmmac_sft_add_pack(&tpriv->pt);
skb = stmmac_test_get_udp_skb(priv, attr);
if (!skb) {
@@ -361,7 +412,7 @@ static int __stmmac_test_loopback(struct stmmac_priv *priv,
cleanup:
if (!attr->dont_wait)
- dev_remove_pack(&tpriv->pt);
+ stmmac_sft_remove_pack(&tpriv->pt);
kfree(tpriv);
return ret;
}
@@ -775,7 +826,7 @@ static int stmmac_test_flowctrl(struct stmmac_priv *priv)
tpriv->pt.func = stmmac_test_flowctrl_validate;
tpriv->pt.dev = priv->dev;
tpriv->pt.af_packet_priv = tpriv;
- dev_add_pack(&tpriv->pt);
+ stmmac_sft_add_pack(&tpriv->pt);
/* Compute minimum number of packets to make FIFO full */
pkt_count = priv->plat->rx_fifo_size;
@@ -833,7 +884,7 @@ static int stmmac_test_flowctrl(struct stmmac_priv *priv)
cleanup:
dev_mc_del(priv->dev, paddr);
dev_set_promiscuity(priv->dev, -1);
- dev_remove_pack(&tpriv->pt);
+ stmmac_sft_remove_pack(&tpriv->pt);
kfree(tpriv);
return ret;
}
@@ -875,6 +926,11 @@ static int stmmac_test_vlan_validate(struct sk_buff *skb,
goto out;
if (skb_headlen(skb) < (STMMAC_TEST_PKT_SIZE - ETH_HLEN))
goto out;
+
+ ehdr = (struct ethhdr *)skb_mac_header(skb);
+ if (!ether_addr_equal_unaligned(ehdr->h_dest, tpriv->packet->dst))
+ goto out;
+
if (tpriv->vlan_id) {
if (skb->vlan_proto != htons(proto))
goto out;
@@ -886,10 +942,6 @@ static int stmmac_test_vlan_validate(struct sk_buff *skb,
}
}
- ehdr = (struct ethhdr *)skb_mac_header(skb);
- if (!ether_addr_equal_unaligned(ehdr->h_dest, tpriv->packet->dst))
- goto out;
-
ihdr = ip_hdr(skb);
if (tpriv->double_vlan)
ihdr = (struct iphdr *)(skb_network_header(skb) + 4);
@@ -931,6 +983,7 @@ static int __stmmac_test_vlanfilt(struct stmmac_priv *priv)
tpriv->pt.dev = priv->dev;
tpriv->pt.af_packet_priv = tpriv;
tpriv->packet = &attr;
+ tpriv->capture_all = true;
/*
* As we use HASH filtering, false positives may appear. This is a
@@ -938,18 +991,20 @@ static int __stmmac_test_vlanfilt(struct stmmac_priv *priv)
* HASH values.
*/
tpriv->vlan_id = 0x123;
- dev_add_pack(&tpriv->pt);
ret = vlan_vid_add(priv->dev, htons(ETH_P_8021Q), tpriv->vlan_id);
if (ret)
goto cleanup;
+ attr.vlan = 1;
+ attr.dst = priv->dev->dev_addr;
+ attr.sport = 9;
+ attr.dport = 9;
+
+ stmmac_sft_add_pack(&tpriv->pt);
+
for (i = 0; i < 4; i++) {
- attr.vlan = 1;
attr.vlan_id_out = tpriv->vlan_id + i;
- attr.dst = priv->dev->dev_addr;
- attr.sport = 9;
- attr.dport = 9;
skb = stmmac_test_get_udp_skb(priv, &attr);
if (!skb) {
@@ -976,9 +1031,9 @@ static int __stmmac_test_vlanfilt(struct stmmac_priv *priv)
}
vlan_del:
+ stmmac_sft_remove_pack(&tpriv->pt);
vlan_vid_del(priv->dev, htons(ETH_P_8021Q), tpriv->vlan_id);
cleanup:
- dev_remove_pack(&tpriv->pt);
kfree(tpriv);
return ret;
}
@@ -1025,6 +1080,7 @@ static int __stmmac_test_dvlanfilt(struct stmmac_priv *priv)
tpriv->pt.dev = priv->dev;
tpriv->pt.af_packet_priv = tpriv;
tpriv->packet = &attr;
+ tpriv->capture_all = true;
/*
* As we use HASH filtering, false positives may appear. This is a
@@ -1032,18 +1088,20 @@ static int __stmmac_test_dvlanfilt(struct stmmac_priv *priv)
* HASH values.
*/
tpriv->vlan_id = 0x123;
- dev_add_pack(&tpriv->pt);
ret = vlan_vid_add(priv->dev, htons(ETH_P_8021AD), tpriv->vlan_id);
if (ret)
goto cleanup;
+ attr.vlan = 2;
+ attr.dst = priv->dev->dev_addr;
+ attr.sport = 9;
+ attr.dport = 9;
+
+ stmmac_sft_add_pack(&tpriv->pt);
+
for (i = 0; i < 4; i++) {
- attr.vlan = 2;
attr.vlan_id_out = tpriv->vlan_id + i;
- attr.dst = priv->dev->dev_addr;
- attr.sport = 9;
- attr.dport = 9;
skb = stmmac_test_get_udp_skb(priv, &attr);
if (!skb) {
@@ -1070,9 +1128,9 @@ static int __stmmac_test_dvlanfilt(struct stmmac_priv *priv)
}
vlan_del:
+ stmmac_sft_remove_pack(&tpriv->pt);
vlan_vid_del(priv->dev, htons(ETH_P_8021AD), tpriv->vlan_id);
cleanup:
- dev_remove_pack(&tpriv->pt);
kfree(tpriv);
return ret;
}
@@ -1303,7 +1361,7 @@ static int stmmac_test_vlanoff_common(struct stmmac_priv *priv, bool svlan)
tpriv->pt.af_packet_priv = tpriv;
tpriv->packet = &attr;
tpriv->vlan_id = 0x123;
- dev_add_pack(&tpriv->pt);
+ tpriv->capture_all = true;
ret = vlan_vid_add(priv->dev, htons(proto), tpriv->vlan_id);
if (ret)
@@ -1311,6 +1369,8 @@ static int stmmac_test_vlanoff_common(struct stmmac_priv *priv, bool svlan)
attr.dst = priv->dev->dev_addr;
+ stmmac_sft_add_pack(&tpriv->pt);
+
skb = stmmac_test_get_udp_skb(priv, &attr);
if (!skb) {
ret = -ENOMEM;
@@ -1328,9 +1388,9 @@ static int stmmac_test_vlanoff_common(struct stmmac_priv *priv, bool svlan)
ret = tpriv->ok ? 0 : -ETIMEDOUT;
vlan_del:
+ stmmac_sft_remove_pack(&tpriv->pt);
vlan_vid_del(priv->dev, htons(proto), tpriv->vlan_id);
cleanup:
- dev_remove_pack(&tpriv->pt);
kfree(tpriv);
return ret;
}
@@ -1342,7 +1402,7 @@ static int stmmac_test_vlanoff(struct stmmac_priv *priv)
static int stmmac_test_svlanoff(struct stmmac_priv *priv)
{
- if (!priv->dma_cap.dvlan)
+ if (!(priv->dev->features & NETIF_F_HW_VLAN_STAG_TX))
return -EOPNOTSUPP;
return stmmac_test_vlanoff_common(priv, true);
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
index 49f133dec810..65085f9c9290 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
@@ -970,7 +970,7 @@ static int tc_taprio_configure(struct stmmac_priv *priv,
struct netlink_ext_ack *extack = qopt->mqprio.extack;
struct timespec64 time, current_time, qopt_time;
ktime_t current_time_ns;
- int i, ret = 0;
+ int err, i, ret = 0;
u64 ctr;
if (qopt->base_time < 0)
@@ -1119,9 +1119,9 @@ static int tc_taprio_configure(struct stmmac_priv *priv,
mutex_unlock(&priv->est_lock);
}
- stmmac_fpe_map_preemption_class(priv, priv->dev, extack, 0);
+ err = stmmac_fpe_map_preemption_class(priv, priv->dev, extack, 0);
- return ret;
+ return qopt->cmd == TAPRIO_CMD_DESTROY ? err : ret;
}
static void tc_taprio_stats(struct stmmac_priv *priv,
@@ -1234,14 +1234,15 @@ static int tc_query_caps(struct stmmac_priv *priv,
}
}
-static void stmmac_reset_tc_mqprio(struct net_device *ndev,
- struct netlink_ext_ack *extack)
+static int stmmac_reset_tc_mqprio(struct net_device *ndev,
+ struct netlink_ext_ack *extack)
{
struct stmmac_priv *priv = netdev_priv(ndev);
netdev_reset_tc(ndev);
netif_set_real_num_tx_queues(ndev, priv->plat->tx_queues_to_use);
- stmmac_fpe_map_preemption_class(priv, ndev, extack, 0);
+
+ return stmmac_fpe_map_preemption_class(priv, ndev, extack, 0);
}
static int tc_setup_dwmac510_mqprio(struct stmmac_priv *priv,
@@ -1254,10 +1255,8 @@ static int tc_setup_dwmac510_mqprio(struct stmmac_priv *priv,
u32 num_tc = qopt->num_tc;
int err;
- if (!num_tc) {
- stmmac_reset_tc_mqprio(ndev, extack);
- return 0;
- }
+ if (!num_tc)
+ return stmmac_reset_tc_mqprio(ndev, extack);
err = netdev_set_num_tc(ndev, num_tc);
if (err)
diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index c0a5abd8d9a8..2968b5fbe428 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -305,12 +305,19 @@ static int cpsw_purge_all_mc(struct net_device *ndev, const u8 *addr, int num)
return 0;
}
-static void cpsw_ndo_set_rx_mode(struct net_device *ndev)
+static void cpsw_ndo_set_rx_mode_work(struct work_struct *work)
{
- struct cpsw_priv *priv = netdev_priv(ndev);
+ struct cpsw_priv *priv = container_of(work, struct cpsw_priv, rx_mode_work);
struct cpsw_common *cpsw = priv->cpsw;
+ struct net_device *ndev = priv->ndev;
int slave_port = -1;
+ rtnl_lock();
+ if (!netif_running(ndev))
+ goto unlock_rtnl;
+
+ netif_addr_lock_bh(ndev);
+
if (cpsw->data.dual_emac)
slave_port = priv->emac_port + 1;
@@ -318,7 +325,7 @@ static void cpsw_ndo_set_rx_mode(struct net_device *ndev)
/* Enable promiscuous mode */
cpsw_set_promiscious(ndev, true);
cpsw_ale_set_allmulti(cpsw->ale, IFF_ALLMULTI, slave_port);
- return;
+ goto unlock_addr;
} else {
/* Disable promiscuous mode */
cpsw_set_promiscious(ndev, false);
@@ -331,6 +338,18 @@ static void cpsw_ndo_set_rx_mode(struct net_device *ndev)
/* add/remove mcast address either for real netdev or for vlan */
__hw_addr_ref_sync_dev(&ndev->mc, ndev, cpsw_add_mc_addr,
cpsw_del_mc_addr);
+
+unlock_addr:
+ netif_addr_unlock_bh(ndev);
+unlock_rtnl:
+ rtnl_unlock();
+}
+
+static void cpsw_ndo_set_rx_mode(struct net_device *ndev)
+{
+ struct cpsw_priv *priv = netdev_priv(ndev);
+
+ schedule_work(&priv->rx_mode_work);
}
static unsigned int cpsw_rxbuf_total_len(unsigned int len)
@@ -1444,6 +1463,7 @@ static int cpsw_probe_dual_emac(struct cpsw_priv *priv)
priv_sl2->ndev = ndev;
priv_sl2->dev = &ndev->dev;
priv_sl2->msg_enable = netif_msg_init(debug_level, CPSW_DEBUG);
+ INIT_WORK(&priv_sl2->rx_mode_work, cpsw_ndo_set_rx_mode_work);
if (is_valid_ether_addr(data->slave_data[1].mac_addr)) {
memcpy(priv_sl2->mac_addr, data->slave_data[1].mac_addr,
@@ -1625,6 +1645,7 @@ static int cpsw_probe(struct platform_device *pdev)
priv->dev = dev;
priv->msg_enable = netif_msg_init(debug_level, CPSW_DEBUG);
priv->emac_port = 0;
+ INIT_WORK(&priv->rx_mode_work, cpsw_ndo_set_rx_mode_work);
if (is_valid_ether_addr(data->slave_data[0].mac_addr)) {
memcpy(priv->mac_addr, data->slave_data[0].mac_addr, ETH_ALEN);
@@ -1727,6 +1748,8 @@ static int cpsw_probe(struct platform_device *pdev)
static void cpsw_remove(struct platform_device *pdev)
{
struct cpsw_common *cpsw = platform_get_drvdata(pdev);
+ struct net_device *ndev;
+ struct cpsw_priv *priv;
int i, ret;
ret = pm_runtime_resume_and_get(&pdev->dev);
@@ -1739,9 +1762,15 @@ static void cpsw_remove(struct platform_device *pdev)
return;
}
- for (i = 0; i < cpsw->data.slaves; i++)
- if (cpsw->slaves[i].ndev)
- unregister_netdev(cpsw->slaves[i].ndev);
+ for (i = 0; i < cpsw->data.slaves; i++) {
+ ndev = cpsw->slaves[i].ndev;
+ if (!ndev)
+ continue;
+
+ priv = netdev_priv(ndev);
+ unregister_netdev(ndev);
+ disable_work_sync(&priv->rx_mode_work);
+ }
cpts_release(cpsw->cpts);
cpdma_ctlr_destroy(cpsw->dma);
diff --git a/drivers/net/ethernet/ti/cpsw_new.c b/drivers/net/ethernet/ti/cpsw_new.c
index 468eaa6f785e..c2041e46b83d 100644
--- a/drivers/net/ethernet/ti/cpsw_new.c
+++ b/drivers/net/ethernet/ti/cpsw_new.c
@@ -248,16 +248,22 @@ static int cpsw_purge_all_mc(struct net_device *ndev, const u8 *addr, int num)
return 0;
}
-static void cpsw_ndo_set_rx_mode(struct net_device *ndev)
+static void cpsw_ndo_set_rx_mode_work(struct work_struct *work)
{
- struct cpsw_priv *priv = netdev_priv(ndev);
+ struct cpsw_priv *priv = container_of(work, struct cpsw_priv, rx_mode_work);
struct cpsw_common *cpsw = priv->cpsw;
+ struct net_device *ndev = priv->ndev;
+
+ rtnl_lock();
+ if (!netif_running(ndev))
+ goto unlock_rtnl;
+ netif_addr_lock_bh(ndev);
if (ndev->flags & IFF_PROMISC) {
/* Enable promiscuous mode */
cpsw_set_promiscious(ndev, true);
cpsw_ale_set_allmulti(cpsw->ale, IFF_ALLMULTI, priv->emac_port);
- return;
+ goto unlock_addr;
}
/* Disable promiscuous mode */
@@ -270,6 +276,18 @@ static void cpsw_ndo_set_rx_mode(struct net_device *ndev)
/* add/remove mcast address either for real netdev or for vlan */
__hw_addr_ref_sync_dev(&ndev->mc, ndev, cpsw_add_mc_addr,
cpsw_del_mc_addr);
+
+unlock_addr:
+ netif_addr_unlock_bh(ndev);
+unlock_rtnl:
+ rtnl_unlock();
+}
+
+static void cpsw_ndo_set_rx_mode(struct net_device *ndev)
+{
+ struct cpsw_priv *priv = netdev_priv(ndev);
+
+ schedule_work(&priv->rx_mode_work);
}
static unsigned int cpsw_rxbuf_total_len(unsigned int len)
@@ -1391,6 +1409,7 @@ static int cpsw_create_ports(struct cpsw_common *cpsw)
priv->msg_enable = netif_msg_init(debug_level, CPSW_DEBUG);
priv->emac_port = i + 1;
priv->tx_packet_min = CPSW_MIN_PACKET_SIZE;
+ INIT_WORK(&priv->rx_mode_work, cpsw_ndo_set_rx_mode_work);
if (is_valid_ether_addr(slave_data->mac_addr)) {
ether_addr_copy(priv->mac_addr, slave_data->mac_addr);
@@ -1440,14 +1459,18 @@ static int cpsw_create_ports(struct cpsw_common *cpsw)
static void cpsw_unregister_ports(struct cpsw_common *cpsw)
{
+ struct net_device *ndev;
+ struct cpsw_priv *priv;
int i = 0;
for (i = 0; i < cpsw->data.slaves; i++) {
- if (!cpsw->slaves[i].ndev ||
- cpsw->slaves[i].ndev->reg_state != NETREG_REGISTERED)
+ ndev = cpsw->slaves[i].ndev;
+ if (!ndev || ndev->reg_state != NETREG_REGISTERED)
continue;
- unregister_netdev(cpsw->slaves[i].ndev);
+ priv = netdev_priv(ndev);
+ unregister_netdev(ndev);
+ disable_work_sync(&priv->rx_mode_work);
}
}
diff --git a/drivers/net/ethernet/ti/cpsw_priv.h b/drivers/net/ethernet/ti/cpsw_priv.h
index 1f448290b9f4..bacaa855e614 100644
--- a/drivers/net/ethernet/ti/cpsw_priv.h
+++ b/drivers/net/ethernet/ti/cpsw_priv.h
@@ -391,6 +391,7 @@ struct cpsw_priv {
u32 tx_packet_min;
struct cpsw_ale_ratelimit ale_bc_ratelimit;
struct cpsw_ale_ratelimit ale_mc_ratelimit;
+ struct work_struct rx_mode_work;
};
#define ndev_to_cpsw(ndev) (((struct cpsw_priv *)netdev_priv(ndev))->cpsw)
diff --git a/drivers/net/ethernet/ti/netcp_core.c b/drivers/net/ethernet/ti/netcp_core.c
index d07dcffc2517..15c2a6dffb8c 100644
--- a/drivers/net/ethernet/ti/netcp_core.c
+++ b/drivers/net/ethernet/ti/netcp_core.c
@@ -2167,7 +2167,7 @@ static int netcp_probe(struct platform_device *pdev)
return -ENOMEM;
pm_runtime_enable(&pdev->dev);
- ret = pm_runtime_get_sync(&pdev->dev);
+ ret = pm_runtime_resume_and_get(&pdev->dev);
if (ret < 0) {
dev_err(dev, "Failed to enable NETCP power-domain\n");
pm_runtime_disable(&pdev->dev);
diff --git a/drivers/net/fddi/skfp/skfddi.c b/drivers/net/fddi/skfp/skfddi.c
index a273362c9e70..feea7baa4816 100644
--- a/drivers/net/fddi/skfp/skfddi.c
+++ b/drivers/net/fddi/skfp/skfddi.c
@@ -928,7 +928,8 @@ static int skfp_ctl_set_mac_address(struct net_device *dev, void *addr)
dev_addr_set(dev, p_sockaddr->sa_data);
spin_lock_irqsave(&bp->DriverLock, Flags);
- ResetAdapter(smc);
+ if (netif_running(dev))
+ ResetAdapter(smc);
spin_unlock_irqrestore(&bp->DriverLock, Flags);
return 0; /* always return zero */
diff --git a/drivers/net/pcs/pcs-xpcs.c b/drivers/net/pcs/pcs-xpcs.c
index 0152f820c1bb..1a6576a11667 100644
--- a/drivers/net/pcs/pcs-xpcs.c
+++ b/drivers/net/pcs/pcs-xpcs.c
@@ -1393,8 +1393,10 @@ static int xpcs_init_clks(struct dw_xpcs *xpcs)
return dev_err_probe(dev, ret, "Failed to get clocks\n");
ret = clk_bulk_prepare_enable(DW_XPCS_NUM_CLKS, xpcs->clks);
- if (ret)
+ if (ret) {
+ clk_bulk_put(DW_XPCS_NUM_CLKS, xpcs->clks);
return dev_err_probe(dev, ret, "Failed to enable clocks\n");
+ }
return 0;
}
diff --git a/drivers/net/phy/mediatek/mtk-ge-soc.c b/drivers/net/phy/mediatek/mtk-ge-soc.c
index 4b2a9a5444c5..ac356937a501 100644
--- a/drivers/net/phy/mediatek/mtk-ge-soc.c
+++ b/drivers/net/phy/mediatek/mtk-ge-soc.c
@@ -1276,17 +1276,24 @@ static int mt798x_phy_led_hw_control_get(struct phy_device *phydev, u8 index,
if (!rules)
return 0;
- if (on & MTK_PHY_LED_ON_LINK)
+ /* TRIGGER_NETDEV_LINK must not be reported together with any of the
+ * per-speed rules, the netdev trigger rejects that combination.
+ * MTK_PHY_LED_ON_LINK holds every speed this LED can indicate and is what
+ * mt798x_phy_led_hw_control_set() programs for TRIGGER_NETDEV_LINK, so
+ * report the speed independent rule only when they are all on.
+ */
+ if ((on & MTK_PHY_LED_ON_LINK) == MTK_PHY_LED_ON_LINK) {
*rules |= BIT(TRIGGER_NETDEV_LINK);
+ } else {
+ if (on & MTK_PHY_LED_ON_LINK10)
+ *rules |= BIT(TRIGGER_NETDEV_LINK_10);
- if (on & MTK_PHY_LED_ON_LINK10)
- *rules |= BIT(TRIGGER_NETDEV_LINK_10);
-
- if (on & MTK_PHY_LED_ON_LINK100)
- *rules |= BIT(TRIGGER_NETDEV_LINK_100);
+ if (on & MTK_PHY_LED_ON_LINK100)
+ *rules |= BIT(TRIGGER_NETDEV_LINK_100);
- if (on & MTK_PHY_LED_ON_LINK1000)
- *rules |= BIT(TRIGGER_NETDEV_LINK_1000);
+ if (on & MTK_PHY_LED_ON_LINK1000)
+ *rules |= BIT(TRIGGER_NETDEV_LINK_1000);
+ }
if (on & MTK_PHY_LED_ON_FDX)
*rules |= BIT(TRIGGER_NETDEV_FULL_DUPLEX);
diff --git a/drivers/net/usb/catc.c b/drivers/net/usb/catc.c
index 98346cb4ece0..dd1e29add060 100644
--- a/drivers/net/usb/catc.c
+++ b/drivers/net/usb/catc.c
@@ -239,17 +239,26 @@ static void catc_rx_done(struct urb *urb)
}
do {
- if(!catc->is_f5u011) {
- pkt_len = le16_to_cpup((__le16*)pkt_start);
- if (pkt_len > urb->actual_length) {
+ int remaining = urb->actual_length -
+ (pkt_start - (u8 *)urb->transfer_buffer);
+
+ if (!catc->is_f5u011) {
+ if (remaining < pkt_offset) {
catc->netdev->stats.rx_length_errors++;
catc->netdev->stats.rx_errors++;
break;
}
+ pkt_len = le16_to_cpup((__le16 *)pkt_start);
} else {
pkt_len = urb->actual_length;
}
+ if (pkt_len < ETH_HLEN || pkt_len + pkt_offset > remaining) {
+ catc->netdev->stats.rx_length_errors++;
+ catc->netdev->stats.rx_errors++;
+ break;
+ }
+
if (!(skb = dev_alloc_skb(pkt_len)))
return;
diff --git a/drivers/net/usb/cdc_mbim.c b/drivers/net/usb/cdc_mbim.c
index e13e4920ee9b..917c8637533b 100644
--- a/drivers/net/usb/cdc_mbim.c
+++ b/drivers/net/usb/cdc_mbim.c
@@ -634,6 +634,11 @@ static const struct usb_device_id mbim_devs[] = {
.driver_info = (unsigned long)&cdc_mbim_info,
},
+ /* MeiG Smart SRM821 ZLP conformance */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x2dee, 0x4d53, USB_CLASS_COMM, USB_CDC_SUBCLASS_MBIM, USB_CDC_PROTO_NONE),
+ .driver_info = (unsigned long)&cdc_mbim_info,
+ },
+
/* Some Huawei devices, ME906s-158 (12d1:15c1) and E3372
* (12d1:157d), are known to fail unless the NDP is placed
* after the IP packets. Applying the quirk to all Huawei
diff --git a/drivers/net/usb/lan78xx.c b/drivers/net/usb/lan78xx.c
index f6373d93e042..69960d564a97 100644
--- a/drivers/net/usb/lan78xx.c
+++ b/drivers/net/usb/lan78xx.c
@@ -4932,10 +4932,12 @@ static bool lan78xx_submit_deferred_urbs(struct lan78xx_net *dev)
!netif_carrier_ok(dev->net) ||
pipe_halted) {
lan78xx_release_tx_buf(dev, skb);
+ usb_put_urb(urb);
continue;
}
ret = usb_submit_urb(urb, GFP_ATOMIC);
+ usb_put_urb(urb);
if (ret == 0) {
netif_trans_update(dev->net);
diff --git a/drivers/net/usb/sr9700.c b/drivers/net/usb/sr9700.c
index e4d7bcd0d99c..202bb55e811f 100644
--- a/drivers/net/usb/sr9700.c
+++ b/drivers/net/usb/sr9700.c
@@ -403,7 +403,8 @@ static int sr9700_rx_fixup(struct usbnet *dev, struct sk_buff *skb)
/* ignore the CRC length */
len = (skb->data[1] | (skb->data[2] << 8)) - 4;
- if (len > ETH_FRAME_LEN || len > skb->len || len < 0)
+ if (len > ETH_FRAME_LEN || len < 0 ||
+ len > skb->len - SR_RX_OVERHEAD)
return 0;
/* the last packet of current skb */
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index 03341951a212..f3e5f8ce9ac2 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -1053,6 +1053,7 @@ static int __veth_napi_enable_range(struct net_device *dev, int start, int end)
for (i = start; i < end; i++) {
struct veth_rq *rq = &priv->rq[i];
+ rcu_assign_pointer(rq->xdp_prog, priv->_xdp_prog);
napi_enable(&rq->xdp_napi);
rcu_assign_pointer(priv->rq[i].napi, &priv->rq[i].xdp_napi);
}
@@ -1087,6 +1088,7 @@ static void veth_napi_del_range(struct net_device *dev, int start, int end)
rcu_assign_pointer(priv->rq[i].napi, NULL);
napi_disable(&rq->xdp_napi);
+ rcu_assign_pointer(rq->xdp_prog, NULL);
__netif_napi_del(&rq->xdp_napi);
}
synchronize_net();
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 0e8fa0f81e84..cb640d29d791 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -3141,6 +3141,14 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
} while (use_napi && !xmit_more &&
unlikely(!virtqueue_enable_cb_delayed(sq->vq)));
+ if (!use_napi &&
+ unlikely(skb_orphan_frags(skb, GFP_ATOMIC))) {
+ DEV_STATS_INC(dev, tx_dropped);
+ dev_kfree_skb_any(skb);
+ kick = !xmit_more || netif_xmit_stopped(txq);
+ goto kick_vq;
+ }
+
/* timestamp packet in software */
skb_tx_timestamp(skb);
@@ -3170,6 +3178,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
kick = use_napi ? __netdev_tx_sent_queue(txq, skb->len, xmit_more) :
!xmit_more || netif_xmit_stopped(txq);
+kick_vq:
if (kick) {
if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) {
u64_stats_update_begin(&sq->stats.syncp);
diff --git a/drivers/net/vrf.c b/drivers/net/vrf.c
index e684d59291ef..c071b3ccd8cb 100644
--- a/drivers/net/vrf.c
+++ b/drivers/net/vrf.c
@@ -1224,8 +1224,6 @@ static int vrf_prepare_mac_header(struct sk_buff *skb,
skb->protocol = eth->h_proto;
skb->pkt_type = PACKET_HOST;
- skb_postpush_rcsum(skb, skb->data, ETH_HLEN);
-
skb_pull_inline(skb, ETH_HLEN);
return 0;
diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
index b14dbc689207..f612def46c3c 100644
--- a/drivers/net/vxlan/vxlan_core.c
+++ b/drivers/net/vxlan/vxlan_core.c
@@ -1979,13 +1979,15 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
struct ipv6hdr *pip6;
u8 *daddr;
int na_olen = 8; /* opt hdr + ETH_ALEN for target */
+ int headroom;
int ns_olen;
int i, len;
if (dev == NULL || !pskb_may_pull(request, request->len))
return NULL;
- len = LL_RESERVED_SPACE(dev) + sizeof(struct ipv6hdr) +
+ headroom = LL_RESERVED_SPACE(dev);
+ len = headroom + sizeof(struct ipv6hdr) +
sizeof(*na) + na_olen + dev->needed_tailroom;
reply = alloc_skb(len, GFP_ATOMIC);
if (reply == NULL)
@@ -1993,7 +1995,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
reply->protocol = htons(ETH_P_IPV6);
reply->dev = dev;
- skb_reserve(reply, LL_RESERVED_SPACE(request->dev));
+ skb_reserve(reply, headroom);
skb_push(reply, sizeof(struct ethhdr));
skb_reset_mac_header(reply);
diff --git a/drivers/net/wireless/ath/ath11k/mac.c b/drivers/net/wireless/ath/ath11k/mac.c
index c4856480fffe..53bc1b90bab3 100644
--- a/drivers/net/wireless/ath/ath11k/mac.c
+++ b/drivers/net/wireless/ath/ath11k/mac.c
@@ -873,6 +873,22 @@ static int ath11k_mac_set_kickout(struct ath11k_vif *arvif)
return 0;
}
+static void ath11k_mac_station_cleanup(struct ieee80211_sta *sta)
+{
+ struct ath11k_sta *arsta;
+
+ if (!sta)
+ return;
+
+ arsta = ath11k_sta_to_arsta(sta);
+
+ kfree(arsta->tx_stats);
+ arsta->tx_stats = NULL;
+
+ kfree(arsta->rx_stats);
+ arsta->rx_stats = NULL;
+}
+
void ath11k_mac_peer_cleanup_all(struct ath11k *ar)
{
struct ath11k_peer *peer, *tmp;
@@ -885,6 +901,7 @@ void ath11k_mac_peer_cleanup_all(struct ath11k *ar)
list_for_each_entry_safe(peer, tmp, &ab->peers, list) {
ath11k_peer_rx_tid_cleanup(ar, peer);
ath11k_peer_rhash_delete(ab, peer);
+ ath11k_mac_station_cleanup(peer->sta);
list_del(&peer->list);
kfree(peer);
}
@@ -9759,7 +9776,6 @@ static int ath11k_mac_station_remove(struct ath11k *ar,
{
struct ath11k_base *ab = ar->ab;
struct ath11k_vif *arvif = ath11k_vif_to_arvif(vif);
- struct ath11k_sta *arsta = ath11k_sta_to_arsta(sta);
int ret;
if (ab->hw_params.vdev_start_delay &&
@@ -9783,12 +9799,7 @@ static int ath11k_mac_station_remove(struct ath11k *ar,
sta->addr, arvif->vdev_id);
ath11k_mac_dec_num_stations(arvif, sta);
-
- kfree(arsta->tx_stats);
- arsta->tx_stats = NULL;
-
- kfree(arsta->rx_stats);
- arsta->rx_stats = NULL;
+ ath11k_mac_station_cleanup(sta);
return ret;
}
diff --git a/drivers/net/wireless/ath/wcn36xx/dxe.c b/drivers/net/wireless/ath/wcn36xx/dxe.c
index d405a4c34059..079ca09c7517 100644
--- a/drivers/net/wireless/ath/wcn36xx/dxe.c
+++ b/drivers/net/wireless/ath/wcn36xx/dxe.c
@@ -1055,7 +1055,7 @@ void wcn36xx_dxe_deinit(struct wcn36xx *wcn)
free_irq(wcn->tx_irq, wcn);
free_irq(wcn->rx_irq, wcn);
- del_timer(&wcn->tx_ack_timer);
+ timer_shutdown_sync(&wcn->tx_ack_timer);
if (wcn->tx_ack_skb) {
ieee80211_tx_status_irqsafe(wcn->hw, wcn->tx_ack_skb);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
index 58eaf08a147b..f99accc34a36 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
@@ -550,6 +550,8 @@ void brcmf_txfinalize(struct brcmf_if *ifp, struct sk_buff *txp, bool success)
if (type == ETH_P_PAE) {
atomic_dec(&ifp->pend_8021x_cnt);
+ /* Order the decrement before waitqueue_active() */
+ smp_mb__after_atomic();
if (waitqueue_active(&ifp->pend_8021x_wait))
wake_up(&ifp->pend_8021x_wait);
}
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c b/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c
index 5b1d35601bbd..ac111bccb736 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c
@@ -1569,6 +1569,10 @@ void brcms_free_timer(struct brcms_timer *t)
/* delete the timer in case it is active */
brcms_del_timer(t);
+ /* Ensure the callback has finished before freeing the timer
+ * structure, since brcms_del_timer() uses non-synchronous cancel.
+ */
+ cancel_delayed_work_sync(&t->dly_wrk);
if (wl->timers == t) {
wl->timers = wl->timers->next;
diff --git a/drivers/net/wireless/intel/ipw2x00/libipw_rx.c b/drivers/net/wireless/intel/ipw2x00/libipw_rx.c
index 2220a9814c8a..762ed2704bf6 100644
--- a/drivers/net/wireless/intel/ipw2x00/libipw_rx.c
+++ b/drivers/net/wireless/intel/ipw2x00/libipw_rx.c
@@ -1302,6 +1302,9 @@ static int libipw_handle_assoc_resp(struct libipw_device *ieee, struct libipw_as
struct libipw_network *network = &network_resp;
struct net_device *dev = ieee->dev;
+ if (stats->len < sizeof(*frame))
+ return 1;
+
network->flags = 0;
network->qos_data.active = 0;
network->qos_data.supported = 0;
@@ -1514,6 +1517,9 @@ static void libipw_process_probe_response(struct libipw_device
#endif
unsigned long flags;
+ if (stats->len < sizeof(*beacon))
+ return;
+
LIBIPW_DEBUG_SCAN("'%*pE' (%pM): %c%c%c%c %c%c%c%c-%c%c%c%c %c%c%c%c\n",
info_element->len, info_element->data,
beacon->header.addr3,
diff --git a/drivers/net/wireless/intel/iwlegacy/common.c b/drivers/net/wireless/intel/iwlegacy/common.c
index 958dd4f9bc69..563fbac5889d 100644
--- a/drivers/net/wireless/intel/iwlegacy/common.c
+++ b/drivers/net/wireless/intel/iwlegacy/common.c
@@ -2327,7 +2327,7 @@ il_dealloc_bcast_stations(struct il_priv *il)
if (!(il->stations[i].used & IL_STA_BCAST))
continue;
- il->stations[i].used &= ~IL_STA_UCODE_ACTIVE;
+ il->stations[i].used = 0;
il->num_stations--;
BUG_ON(il->num_stations < 0);
kfree(il->stations[i].lq);
diff --git a/drivers/net/wireless/intersil/p54/eeprom.c b/drivers/net/wireless/intersil/p54/eeprom.c
index 5bd35c147e19..14646c878322 100644
--- a/drivers/net/wireless/intersil/p54/eeprom.c
+++ b/drivers/net/wireless/intersil/p54/eeprom.c
@@ -418,17 +418,22 @@ static int p54_generate_channel_lists(struct ieee80211_hw *dev)
}
static int p54_convert_rev0(struct ieee80211_hw *dev,
- struct pda_pa_curve_data *curve_data)
+ struct pda_pa_curve_data *curve_data, size_t len)
{
struct p54_common *priv = dev->priv;
struct p54_pa_curve_data_sample *dst;
struct pda_pa_curve_data_sample_rev0 *src;
+ size_t needed = curve_data->channels *
+ (sizeof(*src) * curve_data->points_per_channel + 2);
size_t cd_len = sizeof(*curve_data) +
(curve_data->points_per_channel*sizeof(*dst) + 2) *
curve_data->channels;
unsigned int i, j;
void *source, *target;
+ if (len < sizeof(*curve_data) + needed)
+ return -EINVAL;
+
priv->curve_data = kmalloc(sizeof(*priv->curve_data) + cd_len,
GFP_KERNEL);
if (!priv->curve_data)
@@ -470,17 +475,22 @@ static int p54_convert_rev0(struct ieee80211_hw *dev,
}
static int p54_convert_rev1(struct ieee80211_hw *dev,
- struct pda_pa_curve_data *curve_data)
+ struct pda_pa_curve_data *curve_data, size_t len)
{
struct p54_common *priv = dev->priv;
struct p54_pa_curve_data_sample *dst;
struct pda_pa_curve_data_sample_rev1 *src;
+ size_t needed = curve_data->channels *
+ (sizeof(*src) * curve_data->points_per_channel + 3);
size_t cd_len = sizeof(*curve_data) +
(curve_data->points_per_channel*sizeof(*dst) + 2) *
curve_data->channels;
unsigned int i, j;
void *source, *target;
+ if (len < sizeof(*curve_data) + needed)
+ return -EINVAL;
+
priv->curve_data = kzalloc(cd_len + sizeof(*priv->curve_data),
GFP_KERNEL);
if (!priv->curve_data)
@@ -767,6 +777,7 @@ int p54_parse_eeprom(struct ieee80211_hw *dev, void *eeprom, int len)
case PDR_PRISM_PA_CAL_CURVE_DATA: {
struct pda_pa_curve_data *curve_data =
(struct pda_pa_curve_data *)entry->data;
+
if (data_len < sizeof(*curve_data)) {
err = -EINVAL;
goto err;
@@ -774,10 +785,10 @@ int p54_parse_eeprom(struct ieee80211_hw *dev, void *eeprom, int len)
switch (curve_data->cal_method_rev) {
case 0:
- err = p54_convert_rev0(dev, curve_data);
+ err = p54_convert_rev0(dev, curve_data, data_len);
break;
case 1:
- err = p54_convert_rev1(dev, curve_data);
+ err = p54_convert_rev1(dev, curve_data, data_len);
break;
default:
wiphy_err(dev->wiphy,
@@ -805,7 +816,8 @@ int p54_parse_eeprom(struct ieee80211_hw *dev, void *eeprom, int len)
break;
case PDR_INTERFACE_LIST:
tmp = entry->data;
- while ((u8 *)tmp < entry->data + data_len) {
+ while ((u8 *)tmp + sizeof(struct exp_if) <=
+ entry->data + data_len) {
struct exp_if *exp_if = tmp;
if (exp_if->if_id == cpu_to_le16(IF_ID_ISL39000))
synth = le16_to_cpu(exp_if->variant);
diff --git a/drivers/net/wireless/marvell/libertas_tf/main.c b/drivers/net/wireless/marvell/libertas_tf/main.c
index 25509a873c44..b61492d1a6e8 100644
--- a/drivers/net/wireless/marvell/libertas_tf/main.c
+++ b/drivers/net/wireless/marvell/libertas_tf/main.c
@@ -173,8 +173,8 @@ static int lbtf_init_adapter(struct lbtf_private *priv)
static void lbtf_free_adapter(struct lbtf_private *priv)
{
lbtf_deb_enter(LBTF_DEB_MAIN);
- lbtf_free_cmd_buffer(priv);
timer_delete_sync(&priv->command_timer);
+ lbtf_free_cmd_buffer(priv);
lbtf_deb_leave(LBTF_DEB_MAIN);
}
diff --git a/drivers/net/wireless/marvell/mwifiex/cfg80211.c b/drivers/net/wireless/marvell/mwifiex/cfg80211.c
index dceee429bf71..100d688ff1fb 100644
--- a/drivers/net/wireless/marvell/mwifiex/cfg80211.c
+++ b/drivers/net/wireless/marvell/mwifiex/cfg80211.c
@@ -4278,6 +4278,7 @@ mwifiex_cfg80211_authenticate(struct wiphy *wiphy,
struct mwifiex_adapter *adapter = priv->adapter;
struct sk_buff *skb;
u16 pkt_len, auth_alg;
+ size_t frame_len;
int ret;
struct mwifiex_ieee80211_mgmt *mgmt;
struct mwifiex_txinfo *tx_info;
@@ -4350,10 +4351,17 @@ mwifiex_cfg80211_authenticate(struct wiphy *wiphy,
mwifiex_cancel_scan(adapter);
- pkt_len = (u16)req->ie_len + req->auth_data_len +
+ frame_len = req->ie_len + req->auth_data_len +
MWIFIEX_MGMT_HEADER_LEN + MWIFIEX_AUTH_BODY_LEN;
if (req->auth_data_len >= 4)
- pkt_len -= 4;
+ frame_len -= 4;
+
+ if (frame_len > U16_MAX) {
+ mwifiex_dbg(priv->adapter, ERROR,
+ "auth frame too long: %zu bytes\n", frame_len);
+ return -EINVAL;
+ }
+ pkt_len = frame_len;
skb = dev_alloc_skb(MWIFIEX_MIN_DATA_HEADER_LEN +
MWIFIEX_MGMT_FRAME_HEADER_SIZE +
diff --git a/drivers/net/wireless/marvell/mwifiex/pcie.c b/drivers/net/wireless/marvell/mwifiex/pcie.c
index 5f997becdbaa..d36d431f5367 100644
--- a/drivers/net/wireless/marvell/mwifiex/pcie.c
+++ b/drivers/net/wireless/marvell/mwifiex/pcie.c
@@ -3068,7 +3068,7 @@ static int mwifiex_pcie_request_irq(struct mwifiex_adapter *adapter)
ret);
for (j = 0; j < i; j++)
free_irq(card->msix_entries[j].vector,
- &card->msix_ctx[i]);
+ &card->msix_ctx[j]);
pci_disable_msix(pdev);
} else {
mwifiex_dbg(adapter, MSG, "MSIx enabled!");
diff --git a/drivers/net/wireless/marvell/mwifiex/scan.c b/drivers/net/wireless/marvell/mwifiex/scan.c
index cab889af4c4a..552e66dd0186 100644
--- a/drivers/net/wireless/marvell/mwifiex/scan.c
+++ b/drivers/net/wireless/marvell/mwifiex/scan.c
@@ -104,12 +104,24 @@ has_vendor_hdr(struct ieee_types_vendor_specific *ie, u8 key)
* a given oui in PTK.
*/
static u8
-mwifiex_search_oui_in_ie(struct ie_body *iebody, u8 *oui)
+mwifiex_search_oui_in_ie(struct ie_body *iebody, u8 *oui, int ie_len)
{
+ const size_t ptk_body_offset = offsetof(struct ie_body, ptk_body);
u8 count;
+ /* ie_len is the number of bytes available at iebody. Keep it signed
+ * and reject a negative (underflowed) length before the unsigned
+ * comparisons below, so a small or zero IE length cannot wrap.
+ */
+ if (ie_len < 0 || (size_t)ie_len < ptk_body_offset)
+ return MWIFIEX_OUI_NOT_PRESENT;
+
count = iebody->ptk_cnt[0];
+ /* Reject an OUI count whose list would run past the element. */
+ if (ptk_body_offset + count * sizeof(iebody->ptk_body) > (size_t)ie_len)
+ return MWIFIEX_OUI_NOT_PRESENT;
+
/* There could be multiple OUIs for PTK hence
1) Take the length.
2) Check all the OUIs for AES.
@@ -143,11 +155,14 @@ mwifiex_is_rsn_oui_present(struct mwifiex_bssdescriptor *bss_desc, u32 cipher)
u8 ret = MWIFIEX_OUI_NOT_PRESENT;
if (has_ieee_hdr(bss_desc->bcn_rsn_ie, WLAN_EID_RSN)) {
+ int ie_len = (int)bss_desc->bcn_rsn_ie->ieee_hdr.len -
+ RSN_GTK_OUI_OFFSET;
+
iebody = (struct ie_body *)
(((u8 *) bss_desc->bcn_rsn_ie->data) +
RSN_GTK_OUI_OFFSET);
oui = &mwifiex_rsn_oui[cipher][0];
- ret = mwifiex_search_oui_in_ie(iebody, oui);
+ ret = mwifiex_search_oui_in_ie(iebody, oui, ie_len);
if (ret)
return ret;
}
@@ -169,10 +184,14 @@ mwifiex_is_wpa_oui_present(struct mwifiex_bssdescriptor *bss_desc, u32 cipher)
u8 ret = MWIFIEX_OUI_NOT_PRESENT;
if (has_vendor_hdr(bss_desc->bcn_wpa_ie, WLAN_EID_VENDOR_SPECIFIC)) {
+ int ie_len = (int)bss_desc->bcn_wpa_ie->vend_hdr.len -
+ (int)sizeof(bss_desc->bcn_wpa_ie->vend_hdr.oui) -
+ WPA_GTK_OUI_OFFSET;
+
iebody = (struct ie_body *)((u8 *)bss_desc->bcn_wpa_ie->data +
WPA_GTK_OUI_OFFSET);
oui = &mwifiex_wpa_oui[cipher][0];
- ret = mwifiex_search_oui_in_ie(iebody, oui);
+ ret = mwifiex_search_oui_in_ie(iebody, oui, ie_len);
if (ret)
return ret;
}
@@ -2098,6 +2117,7 @@ int mwifiex_ret_802_11_scan(struct mwifiex_private *priv,
u32 bytes_left;
u32 idx;
u32 tlv_buf_size;
+ size_t fixed_size;
struct mwifiex_ie_types_chan_band_list_param_set *chan_band_tlv;
struct chan_band_param_set *chan_band;
u8 is_bgscan_resp;
@@ -2113,6 +2133,14 @@ int mwifiex_ret_802_11_scan(struct mwifiex_private *priv,
else
scan_rsp = &resp->params.scan_resp;
+ scan_resp_size = le16_to_cpu(resp->size);
+ fixed_size = scan_rsp->bss_desc_and_tlv_buffer - (u8 *)resp;
+ if (scan_resp_size < fixed_size) {
+ mwifiex_dbg(adapter, ERROR,
+ "SCAN_RESP: response is too short\n");
+ ret = -1;
+ goto check_next_scan;
+ }
if (scan_rsp->number_of_sets > MWIFIEX_MAX_AP) {
mwifiex_dbg(adapter, ERROR,
@@ -2130,8 +2158,6 @@ int mwifiex_ret_802_11_scan(struct mwifiex_private *priv,
"info: SCAN_RESP: bss_descript_size %d\n",
bytes_left);
- scan_resp_size = le16_to_cpu(resp->size);
-
mwifiex_dbg(adapter, INFO,
"info: SCAN_RESP: returned %d APs before parsing\n",
scan_rsp->number_of_sets);
@@ -2139,15 +2165,17 @@ int mwifiex_ret_802_11_scan(struct mwifiex_private *priv,
bss_info = scan_rsp->bss_desc_and_tlv_buffer;
/*
- * The size of the TLV buffer is equal to the entire command response
- * size (scan_resp_size) minus the fixed fields (sizeof()'s), the
- * BSS Descriptions (bss_descript_size as bytesLef) and the command
- * response header (S_DS_GEN)
+ * The TLV buffer follows the command-specific fixed fields and the BSS
+ * descriptions. Background-scan responses have an additional fixed
+ * field before scan_rsp, which is included in fixed_size.
*/
- tlv_buf_size = scan_resp_size - (bytes_left
- + sizeof(scan_rsp->bss_descript_size)
- + sizeof(scan_rsp->number_of_sets)
- + S_DS_GEN);
+ if (bytes_left > scan_resp_size - fixed_size) {
+ mwifiex_dbg(adapter, ERROR,
+ "SCAN_RESP: BSS data exceeds response\n");
+ ret = -1;
+ goto check_next_scan;
+ }
+ tlv_buf_size = scan_resp_size - fixed_size - bytes_left;
tlv_data = (struct mwifiex_ie_types_data *) (scan_rsp->
bss_desc_and_tlv_buffer +
diff --git a/drivers/net/wireless/marvell/mwifiex/util.c b/drivers/net/wireless/marvell/mwifiex/util.c
index 86e20edb593b..6704085908ef 100644
--- a/drivers/net/wireless/marvell/mwifiex/util.c
+++ b/drivers/net/wireless/marvell/mwifiex/util.c
@@ -335,10 +335,16 @@ mwifiex_parse_mgmt_packet(struct mwifiex_private *priv, u8 *payload, u16 len,
switch (stype) {
case IEEE80211_STYPE_ACTION:
- category = *(payload + sizeof(struct ieee80211_hdr));
+ if (len < sizeof(*ieee_hdr) + 1)
+ return -1;
+
+ category = *(payload + sizeof(*ieee_hdr));
switch (category) {
case WLAN_CATEGORY_PUBLIC:
- action_code = *(payload + sizeof(struct ieee80211_hdr)
+ if (len < sizeof(*ieee_hdr) + 2)
+ return -1;
+
+ action_code = *(payload + sizeof(*ieee_hdr)
+ 1);
if (action_code == WLAN_PUB_ACTION_TDLS_DISCOVER_RES) {
addr2 = ieee_hdr->addr2;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c
index 578013884e43..c170ea129983 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c
@@ -444,6 +444,11 @@ mt7915_rdd_monitor(struct seq_file *s, void *data)
mutex_lock(&dev->mt76.mutex);
+ if (!mt7915_eeprom_has_background_radar(dev)) {
+ seq_puts(s, "no background radar capability\n");
+ goto out;
+ }
+
if (!cfg80211_chandef_valid(chandef)) {
ret = -EINVAL;
goto out;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c
index 93956d402041..14c0770229bc 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c
@@ -147,7 +147,7 @@ static int mt7915_eeprom_load(struct mt7915_dev *dev)
/* read eeprom data from efuse */
block_num = DIV_ROUND_UP(eeprom_size, eeprom_blk_size);
for (i = 0; i < block_num; i++) {
- ret = mt7915_mcu_get_eeprom(dev, i * eeprom_blk_size);
+ ret = mt7915_mcu_get_eeprom(dev, i * eeprom_blk_size, NULL);
if (ret < 0)
return ret;
}
@@ -359,6 +359,37 @@ s8 mt7915_eeprom_get_power_delta(struct mt7915_dev *dev, int band)
return val & MT_EE_RATE_DELTA_SIGN ? delta : -delta;
}
+bool
+mt7915_eeprom_has_background_radar(struct mt7915_dev *dev)
+{
+ u8 val, buf[MT7915_EEPROM_BLOCK_SIZE];
+ u8 band_sel, tx_path, rx_path;
+ int offs = MT_EE_WIFI_CONF + 1;
+
+ switch (mt76_chip(&dev->mt76)) {
+ case 0x7915:
+ return true;
+ case 0x7906:
+ /* read efuse to check background radar capability */
+ if (mt7915_mcu_get_eeprom(dev, offs, buf))
+ break;
+
+ val = buf[offs % MT7915_EEPROM_BLOCK_SIZE];
+ band_sel = u8_get_bits(val, MT_EE_WIFI_CONF0_BAND_SEL);
+ tx_path = u8_get_bits(val, MT_EE_WIFI_CONF0_TX_PATH);
+ rx_path = u8_get_bits(val, MT_EE_WIFI_CONF0_RX_PATH);
+
+ return (band_sel == MT_EE_V2_BAND_SEL_5GHZ &&
+ tx_path == rx_path && rx_path == 2);
+ case 0x7981:
+ case 0x7986:
+ default:
+ break;
+ }
+
+ return false;
+}
+
const u8 mt7915_sku_group_len[] = {
[SKU_CCK] = 4,
[SKU_OFDM] = 8,
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h
index 2908e8113e48..73611c9d26e1 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h
@@ -55,6 +55,7 @@ enum mt7915_eeprom_field {
#define MT_EE_CAL_DPD_SIZE_V2 (300 * MT_EE_CAL_UNIT)
#define MT_EE_WIFI_CONF0_TX_PATH GENMASK(2, 0)
+#define MT_EE_WIFI_CONF0_RX_PATH GENMASK(5, 3)
#define MT_EE_WIFI_CONF0_BAND_SEL GENMASK(7, 6)
#define MT_EE_WIFI_CONF1_BAND_SEL GENMASK(7, 6)
#define MT_EE_WIFI_CONF_STREAM_NUM GENMASK(7, 5)
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/init.c b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
index 96871012bed3..985a1643f0f1 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/init.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
@@ -389,9 +389,10 @@ mt7915_init_wiphy(struct mt7915_phy *phy)
if (!is_mt7915(&dev->mt76))
wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_STA_TX_PWR);
- if (!mdev->dev->of_node ||
- !of_property_read_bool(mdev->dev->of_node,
- "mediatek,disable-radar-background"))
+ if (mt7915_eeprom_has_background_radar(phy->dev) &&
+ (!mdev->dev->of_node ||
+ !of_property_read_bool(mdev->dev->of_node,
+ "mediatek,disable-radar-background")))
wiphy_ext_feature_set(wiphy,
NL80211_EXT_FEATURE_RADAR_BACKGROUND);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c
index 343866dfc112..e2c714fb515f 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c
@@ -2853,7 +2853,7 @@ int mt7915_mcu_set_eeprom(struct mt7915_dev *dev)
&req, sizeof(req), true);
}
-int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset)
+int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset, u8 *read_buf)
{
struct mt7915_mcu_eeprom_info req = {
.addr = cpu_to_le32(round_down(offset,
@@ -2861,8 +2861,8 @@ int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset)
};
struct mt7915_mcu_eeprom_info *res;
struct sk_buff *skb;
+ u8 *buf = read_buf;
int ret;
- u8 *buf;
ret = mt76_mcu_send_and_get_msg(&dev->mt76,
MCU_EXT_QUERY(EFUSE_ACCESS),
@@ -2871,8 +2871,17 @@ int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset)
return ret;
res = (struct mt7915_mcu_eeprom_info *)skb->data;
- buf = dev->mt76.eeprom.data + le32_to_cpu(res->addr);
+ if (!buf) {
+ u32 addr = le32_to_cpu(res->addr);
+
+ if (addr > dev->mt76.eeprom.size - MT7915_EEPROM_BLOCK_SIZE) {
+ dev_kfree_skb(skb);
+ return -EINVAL;
+ }
+ buf = dev->mt76.eeprom.data + addr;
+ }
memcpy(buf, res->data, MT7915_EEPROM_BLOCK_SIZE);
+
dev_kfree_skb(skb);
return 0;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h
index 623b3077ed6f..f85a1c3bfa23 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h
@@ -445,6 +445,7 @@ int mt7915_eeprom_get_target_power(struct mt7915_dev *dev,
struct ieee80211_channel *chan,
u8 chain_idx);
s8 mt7915_eeprom_get_power_delta(struct mt7915_dev *dev, int band);
+bool mt7915_eeprom_has_background_radar(struct mt7915_dev *dev);
int mt7915_dma_init(struct mt7915_dev *dev, struct mt7915_phy *phy2);
void mt7915_dma_prefetch(struct mt7915_dev *dev);
void mt7915_dma_cleanup(struct mt7915_dev *dev);
@@ -495,7 +496,7 @@ int mt7915_mcu_set_fixed_rate_ctrl(struct mt7915_dev *dev,
struct ieee80211_sta *sta,
void *data, u32 field);
int mt7915_mcu_set_eeprom(struct mt7915_dev *dev);
-int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset);
+int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset, u8 *read_buf);
int mt7915_mcu_get_eeprom_free_block(struct mt7915_dev *dev, u8 *block_num);
int mt7915_mcu_set_mac(struct mt7915_dev *dev, int band, bool enable,
bool hdr_trans);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c
index e3f5b2b2e367..991efc9977be 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c
@@ -3586,7 +3586,13 @@ int mt7996_mcu_get_eeprom(struct mt7996_dev *dev, u32 offset)
valid = le32_to_cpu(*(__le32 *)(skb->data + 16));
if (valid) {
u32 addr = le32_to_cpu(*(__le32 *)(skb->data + 12));
- u8 *buf = (u8 *)dev->mt76.eeprom.data + addr;
+ u8 *buf;
+
+ if (addr > dev->mt76.eeprom.size - MT7996_EEPROM_BLOCK_SIZE) {
+ dev_kfree_skb(skb);
+ return -EINVAL;
+ }
+ buf = (u8 *)dev->mt76.eeprom.data + addr;
skb_pull(skb, 48);
memcpy(buf, skb->data, MT7996_EEPROM_BLOCK_SIZE);
diff --git a/drivers/net/wireless/microchip/wilc1000/cfg80211.c b/drivers/net/wireless/microchip/wilc1000/cfg80211.c
index 9977be0020fe..51ad54e88d59 100644
--- a/drivers/net/wireless/microchip/wilc1000/cfg80211.c
+++ b/drivers/net/wireless/microchip/wilc1000/cfg80211.c
@@ -1066,6 +1066,13 @@ void wilc_wfi_p2p_rx(struct wilc_vif *vif, u8 *buff, u32 size)
if (!ieee80211_is_public_action((struct ieee80211_hdr *)buff, size))
goto out_rx_mgmt;
+ /* ieee80211_is_public_action() only validates up to the category
+ * byte, so reject frames too short for the P2P public action header
+ * before dereferencing it or computing size - ie_offset.
+ */
+ if (size < ie_offset)
+ goto out_rx_mgmt;
+
d = (struct wilc_p2p_pub_act_frame *)(&mgmt->u.action);
if (d->oui_subtype != GO_NEG_REQ && d->oui_subtype != GO_NEG_RSP &&
d->oui_subtype != P2P_INV_REQ && d->oui_subtype != P2P_INV_RSP)
@@ -1214,6 +1221,13 @@ static int mgmt_tx(struct wiphy *wiphy,
goto out_set_timeout;
}
+ /* ieee80211_is_public_action() only validates up to the category
+ * byte, so reject frames too short for the P2P public action header
+ * before dereferencing it or computing len - ie_offset.
+ */
+ if (len < ie_offset)
+ goto out_set_timeout;
+
d = (struct wilc_p2p_pub_act_frame *)(&mgmt->u.action);
if (d->oui_type != WLAN_OUI_TYPE_WFA_P2P ||
d->oui_subtype != GO_NEG_CONF) {
diff --git a/drivers/net/wireless/microchip/wilc1000/wlan.c b/drivers/net/wireless/microchip/wilc1000/wlan.c
index 533939e71534..03adecf614bc 100644
--- a/drivers/net/wireless/microchip/wilc1000/wlan.c
+++ b/drivers/net/wireless/microchip/wilc1000/wlan.c
@@ -1035,6 +1035,15 @@ static void wilc_wlan_handle_isr_ext(struct wilc *wilc, u32 int_status)
if (size <= 0)
return;
+ /* A size exceeding the RX buffer is bogus; drop the transfer
+ * instead of overflowing the buffer.
+ */
+ if (size > WILC_RX_BUFF_SIZE) {
+ wilc->hif_func->hif_clear_int_ext(wilc,
+ DATA_INT_CLR | ENABLE_RX_VMM);
+ return;
+ }
+
if (WILC_RX_BUFF_SIZE - offset < size)
offset = 0;
diff --git a/drivers/net/wireless/realtek/rtw88/fw.c b/drivers/net/wireless/realtek/rtw88/fw.c
index 43e3df6a4836..fd3bcb446620 100644
--- a/drivers/net/wireless/realtek/rtw88/fw.c
+++ b/drivers/net/wireless/realtek/rtw88/fw.c
@@ -1446,7 +1446,7 @@ void rtw_add_rsvd_page_sta(struct rtw_dev *rtwdev,
int rtw_fw_write_data_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr,
u8 *buf, u32 size)
{
- u8 bckp[2];
+ u8 bckp[3];
u8 val;
u16 rsvd_pg_head;
u32 bcn_valid_addr;
@@ -1458,6 +1458,8 @@ int rtw_fw_write_data_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr,
if (!size)
return -EINVAL;
+ bckp[2] = rtw_read8(rtwdev, REG_BCN_CTRL);
+
if (rtw_chip_wcpu_11n(rtwdev)) {
rtw_write32_set(rtwdev, REG_DWBCN0_CTRL, BIT_BCN_VALID);
} else {
@@ -1471,6 +1473,9 @@ int rtw_fw_write_data_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr,
val |= BIT_ENSWBCN >> 8;
rtw_write8(rtwdev, REG_CR + 1, val);
+ rtw_write8(rtwdev, REG_BCN_CTRL,
+ (bckp[2] & ~BIT_EN_BCN_FUNCTION) | BIT_DIS_TSF_UDT);
+
if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_PCIE) {
val = rtw_read8(rtwdev, REG_FWHW_TXQ_CTRL + 2);
bckp[1] = val;
@@ -1501,6 +1506,7 @@ int rtw_fw_write_data_rsvd_page(struct rtw_dev *rtwdev, u16 pg_addr,
rsvd_pg_head = rtwdev->fifo.rsvd_boundary;
rtw_write16(rtwdev, REG_FIFOPAGE_CTRL_2,
rsvd_pg_head | BIT_BCN_VALID_V1);
+ rtw_write8(rtwdev, REG_BCN_CTRL, bckp[2]);
if (rtw_hci_type(rtwdev) == RTW_HCI_TYPE_PCIE)
rtw_write8(rtwdev, REG_FWHW_TXQ_CTRL + 2, bckp[1]);
rtw_write8(rtwdev, REG_CR + 1, bckp[0]);
diff --git a/drivers/net/wireless/realtek/rtw88/tx.c b/drivers/net/wireless/realtek/rtw88/tx.c
index 662fb27224f3..0c8817bf46c7 100644
--- a/drivers/net/wireless/realtek/rtw88/tx.c
+++ b/drivers/net/wireless/realtek/rtw88/tx.c
@@ -421,7 +421,7 @@ void rtw_tx_pkt_info_update(struct rtw_dev *rtwdev,
pkt_info->mac_id = rtwvif->mac_id;
}
- if (ieee80211_is_mgmt(fc) || ieee80211_is_nullfunc(fc))
+ if (ieee80211_is_mgmt(fc) || ieee80211_is_any_nullfunc(fc))
rtw_tx_mgmt_pkt_info_update(rtwdev, pkt_info, sta, skb);
else if (ieee80211_is_data(fc))
rtw_tx_data_pkt_info_update(rtwdev, pkt_info, sta, skb);
diff --git a/drivers/net/wireless/realtek/rtw89/core.h b/drivers/net/wireless/realtek/rtw89/core.h
index 36ba2ca255bb..36ae35afda16 100644
--- a/drivers/net/wireless/realtek/rtw89/core.h
+++ b/drivers/net/wireless/realtek/rtw89/core.h
@@ -4724,6 +4724,8 @@ enum rtw89_flags {
RTW89_FLAG_FORBIDDEN_TRACK_WROK,
RTW89_FLAG_CHANGING_INTERFACE,
RTW89_FLAG_HW_RFKILL_STATE,
+ RTW89_FLAG_UNPLUGGED,
+ RTW89_FLAG_SHUTDOWN,
NUM_OF_RTW89_FLAGS,
};
@@ -5604,7 +5606,6 @@ struct rtw89_dev {
struct delayed_work coex_rfk_chk_work;
struct delayed_work cfo_track_work;
struct delayed_work forbid_ba_work;
- struct delayed_work roc_work;
struct delayed_work antdiv_work;
struct rtw89_ppdu_sts_info ppdu_sts;
u8 total_sta_assoc;
diff --git a/drivers/net/wireless/realtek/rtw89/mac.c b/drivers/net/wireless/realtek/rtw89/mac.c
index f323417a88d2..b0bbeea84989 100644
--- a/drivers/net/wireless/realtek/rtw89/mac.c
+++ b/drivers/net/wireless/realtek/rtw89/mac.c
@@ -88,7 +88,7 @@ int rtw89_mac_write_lte(struct rtw89_dev *rtwdev, const u32 offset, u32 val)
ret = read_poll_timeout(rtw89_read8, lte_ctrl, (lte_ctrl & BIT(5)) != 0,
50, 50000, false, rtwdev, R_AX_LTE_CTRL + 3);
- if (ret)
+ if (ret && !test_bit(RTW89_FLAG_UNPLUGGED, rtwdev->flags))
rtw89_err(rtwdev, "[ERR]lte not ready(W)\n");
rtw89_write32(rtwdev, R_AX_LTE_WDATA, val);
@@ -104,7 +104,7 @@ int rtw89_mac_read_lte(struct rtw89_dev *rtwdev, const u32 offset, u32 *val)
ret = read_poll_timeout(rtw89_read8, lte_ctrl, (lte_ctrl & BIT(5)) != 0,
50, 50000, false, rtwdev, R_AX_LTE_CTRL + 3);
- if (ret)
+ if (ret && !test_bit(RTW89_FLAG_UNPLUGGED, rtwdev->flags))
rtw89_err(rtwdev, "[ERR]lte not ready(W)\n");
rtw89_write32(rtwdev, R_AX_LTE_CTRL, 0x800F0000 | offset);
@@ -5666,13 +5666,15 @@ int rtw89_mac_coex_init(struct rtw89_dev *rtwdev, const struct rtw89_mac_ax_coex
ret = rtw89_mac_read_lte(rtwdev, R_AX_LTE_SW_CFG_2, &val32);
if (ret) {
- rtw89_err(rtwdev, "Read R_AX_LTE_SW_CFG_2 fail!\n");
+ if (!test_bit(RTW89_FLAG_UNPLUGGED, rtwdev->flags))
+ rtw89_err(rtwdev, "Read R_AX_LTE_SW_CFG_2 fail!\n");
return ret;
}
val32 = val32 & B_AX_WL_RX_CTRL;
ret = rtw89_mac_write_lte(rtwdev, R_AX_LTE_SW_CFG_2, val32);
if (ret) {
- rtw89_err(rtwdev, "Write R_AX_LTE_SW_CFG_2 fail!\n");
+ if (!test_bit(RTW89_FLAG_UNPLUGGED, rtwdev->flags))
+ rtw89_err(rtwdev, "Write R_AX_LTE_SW_CFG_2 fail!\n");
return ret;
}
@@ -5796,7 +5798,8 @@ int rtw89_mac_cfg_gnt(struct rtw89_dev *rtwdev,
ret = rtw89_mac_write_lte(rtwdev, R_AX_LTE_SW_CFG_1, val);
if (ret) {
- rtw89_err(rtwdev, "Write LTE fail!\n");
+ if (!test_bit(RTW89_FLAG_UNPLUGGED, rtwdev->flags))
+ rtw89_err(rtwdev, "Write LTE fail!\n");
return ret;
}
diff --git a/drivers/net/wireless/realtek/rtw89/mac80211.c b/drivers/net/wireless/realtek/rtw89/mac80211.c
index d1f9bd41f8b5..cf766849086f 100644
--- a/drivers/net/wireless/realtek/rtw89/mac80211.c
+++ b/drivers/net/wireless/realtek/rtw89/mac80211.c
@@ -1550,7 +1550,8 @@ static void rtw89_ops_rfkill_poll(struct ieee80211_hw *hw)
mutex_lock(&rtwdev->mutex);
/* wl_disable GPIO get floating when entering LPS */
- if (test_bit(RTW89_FLAG_RUNNING, rtwdev->flags))
+ if (test_bit(RTW89_FLAG_RUNNING, rtwdev->flags) ||
+ test_bit(RTW89_FLAG_SHUTDOWN, rtwdev->flags))
goto out;
rtw89_core_rfkill_poll(rtwdev, false);
diff --git a/drivers/net/wireless/realtek/rtw89/pci.c b/drivers/net/wireless/realtek/rtw89/pci.c
index cd094b32884a..48415d108045 100644
--- a/drivers/net/wireless/realtek/rtw89/pci.c
+++ b/drivers/net/wireless/realtek/rtw89/pci.c
@@ -4465,6 +4465,19 @@ void rtw89_pci_remove(struct pci_dev *pdev)
}
EXPORT_SYMBOL(rtw89_pci_remove);
+void rtw89_pci_shutdown(struct pci_dev *pdev)
+{
+ struct ieee80211_hw *hw = pci_get_drvdata(pdev);
+ struct rtw89_dev *rtwdev;
+
+ if (!hw)
+ return;
+
+ rtwdev = hw->priv;
+ set_bit(RTW89_FLAG_SHUTDOWN, rtwdev->flags);
+}
+EXPORT_SYMBOL(rtw89_pci_shutdown);
+
MODULE_AUTHOR("Realtek Corporation");
MODULE_DESCRIPTION("Realtek PCI 802.11ax wireless driver");
MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/net/wireless/realtek/rtw89/pci.h b/drivers/net/wireless/realtek/rtw89/pci.h
index 0ea4dcb84dd8..7b438ffa41d0 100644
--- a/drivers/net/wireless/realtek/rtw89/pci.h
+++ b/drivers/net/wireless/realtek/rtw89/pci.h
@@ -1602,6 +1602,7 @@ struct pci_device_id;
int rtw89_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id);
void rtw89_pci_remove(struct pci_dev *pdev);
+void rtw89_pci_shutdown(struct pci_dev *pdev);
void rtw89_pci_ops_reset(struct rtw89_dev *rtwdev);
int rtw89_pci_ltr_set(struct rtw89_dev *rtwdev, bool en);
int rtw89_pci_ltr_set_v1(struct rtw89_dev *rtwdev, bool en);
diff --git a/drivers/net/wireless/realtek/rtw89/phy.c b/drivers/net/wireless/realtek/rtw89/phy.c
index d544077ec298..c957f88faeee 100644
--- a/drivers/net/wireless/realtek/rtw89/phy.c
+++ b/drivers/net/wireless/realtek/rtw89/phy.c
@@ -881,7 +881,8 @@ static u32 rtw89_phy_read_rf_a(struct rtw89_dev *rtwdev,
30, false, rtwdev, R_SWSI_V1,
B_SWSI_R_DATA_DONE_V1);
if (ret) {
- rtw89_err(rtwdev, "read swsi busy\n");
+ if (!test_bit(RTW89_FLAG_UNPLUGGED, rtwdev->flags))
+ rtw89_err(rtwdev, "read swsi busy\n");
return INV_RF_DATA;
}
diff --git a/drivers/net/wireless/realtek/rtw89/rtw8851be.c b/drivers/net/wireless/realtek/rtw89/rtw8851be.c
index d334924faec8..a036e2a7a2e0 100644
--- a/drivers/net/wireless/realtek/rtw89/rtw8851be.c
+++ b/drivers/net/wireless/realtek/rtw89/rtw8851be.c
@@ -84,6 +84,7 @@ static struct pci_driver rtw89_8851be_driver = {
.id_table = rtw89_8851be_id_table,
.probe = rtw89_pci_probe,
.remove = rtw89_pci_remove,
+ .shutdown = rtw89_pci_shutdown,
.driver.pm = &rtw89_pm_ops,
};
module_pci_driver(rtw89_8851be_driver);
diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852ae.c b/drivers/net/wireless/realtek/rtw89/rtw8852ae.c
index 9a675e2193bc..ab70abc80011 100644
--- a/drivers/net/wireless/realtek/rtw89/rtw8852ae.c
+++ b/drivers/net/wireless/realtek/rtw89/rtw8852ae.c
@@ -86,6 +86,7 @@ static struct pci_driver rtw89_8852ae_driver = {
.id_table = rtw89_8852ae_id_table,
.probe = rtw89_pci_probe,
.remove = rtw89_pci_remove,
+ .shutdown = rtw89_pci_shutdown,
.driver.pm = &rtw89_pm_ops,
};
module_pci_driver(rtw89_8852ae_driver);
diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852be.c b/drivers/net/wireless/realtek/rtw89/rtw8852be.c
index d8f9d92ca0fb..f787da3c65d4 100644
--- a/drivers/net/wireless/realtek/rtw89/rtw8852be.c
+++ b/drivers/net/wireless/realtek/rtw89/rtw8852be.c
@@ -88,6 +88,7 @@ static struct pci_driver rtw89_8852be_driver = {
.id_table = rtw89_8852be_id_table,
.probe = rtw89_pci_probe,
.remove = rtw89_pci_remove,
+ .shutdown = rtw89_pci_shutdown,
.driver.pm = &rtw89_pm_ops,
};
module_pci_driver(rtw89_8852be_driver);
diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852bte.c b/drivers/net/wireless/realtek/rtw89/rtw8852bte.c
index 702948119646..635edbb28a06 100644
--- a/drivers/net/wireless/realtek/rtw89/rtw8852bte.c
+++ b/drivers/net/wireless/realtek/rtw89/rtw8852bte.c
@@ -84,6 +84,7 @@ static struct pci_driver rtw89_8852bte_driver = {
.id_table = rtw89_8852bte_id_table,
.probe = rtw89_pci_probe,
.remove = rtw89_pci_remove,
+ .shutdown = rtw89_pci_shutdown,
.driver.pm = &rtw89_pm_ops,
};
module_pci_driver(rtw89_8852bte_driver);
diff --git a/drivers/net/wireless/realtek/rtw89/rtw8852ce.c b/drivers/net/wireless/realtek/rtw89/rtw8852ce.c
index 8aaad7d58c0d..1d9096854b69 100644
--- a/drivers/net/wireless/realtek/rtw89/rtw8852ce.c
+++ b/drivers/net/wireless/realtek/rtw89/rtw8852ce.c
@@ -113,6 +113,7 @@ static struct pci_driver rtw89_8852ce_driver = {
.id_table = rtw89_8852ce_id_table,
.probe = rtw89_pci_probe,
.remove = rtw89_pci_remove,
+ .shutdown = rtw89_pci_shutdown,
.driver.pm = &rtw89_pm_ops,
};
module_pci_driver(rtw89_8852ce_driver);
diff --git a/drivers/net/wireless/realtek/rtw89/rtw8922ae.c b/drivers/net/wireless/realtek/rtw89/rtw8922ae.c
index 47f855a7a268..c71de2da3cfa 100644
--- a/drivers/net/wireless/realtek/rtw89/rtw8922ae.c
+++ b/drivers/net/wireless/realtek/rtw89/rtw8922ae.c
@@ -82,6 +82,7 @@ static struct pci_driver rtw89_8922ae_driver = {
.id_table = rtw89_8922ae_id_table,
.probe = rtw89_pci_probe,
.remove = rtw89_pci_remove,
+ .shutdown = rtw89_pci_shutdown,
.driver.pm = &rtw89_pm_ops_be,
};
module_pci_driver(rtw89_8922ae_driver);
diff --git a/drivers/net/wireless/rsi/rsi_91x_mgmt.c b/drivers/net/wireless/rsi/rsi_91x_mgmt.c
index 2ddf4d158bfe..c1fc39fafeb7 100644
--- a/drivers/net/wireless/rsi/rsi_91x_mgmt.c
+++ b/drivers/net/wireless/rsi/rsi_91x_mgmt.c
@@ -852,8 +852,6 @@ int rsi_hal_load_key(struct rsi_common *common,
memcpy(set_key->tx_mic_key, &data[16], 8);
memcpy(set_key->rx_mic_key, &data[24], 8);
}
- } else {
- memset(&set_key[FRAME_DESC_SZ], 0, frame_len - FRAME_DESC_SZ);
}
skb_put(skb, frame_len);
diff --git a/drivers/net/wireless/ti/wlcore/main.c b/drivers/net/wireless/ti/wlcore/main.c
index da6db99b0d57..754c1f778553 100644
--- a/drivers/net/wireless/ti/wlcore/main.c
+++ b/drivers/net/wireless/ti/wlcore/main.c
@@ -3718,10 +3718,8 @@ void wlcore_regdomain_config(struct wl1271 *wl)
goto out;
ret = wlcore_cmd_regdomain_config_locked(wl);
- if (ret < 0) {
+ if (ret < 0)
wl12xx_queue_recovery_work(wl);
- goto out;
- }
pm_runtime_mark_last_busy(wl->dev);
pm_runtime_put_autosuspend(wl->dev);
diff --git a/drivers/net/wireless/virtual/mac80211_hwsim.c b/drivers/net/wireless/virtual/mac80211_hwsim.c
index 9410059e97f0..f1b96cd1198c 100644
--- a/drivers/net/wireless/virtual/mac80211_hwsim.c
+++ b/drivers/net/wireless/virtual/mac80211_hwsim.c
@@ -2120,7 +2120,12 @@ static void mac80211_hwsim_stop(struct ieee80211_hw *hw, bool suspend)
struct sk_buff *skb;
int i;
- data->started = false;
+ /*
+ * Serialise against wmediumd userspace, so no more frames
+ * can be handed to mac80211 after this returns.
+ */
+ scoped_guard(mutex, &data->mutex)
+ data->started = false;
for (i = 0; i < ARRAY_SIZE(data->link_data); i++)
hrtimer_cancel(&data->link_data[i].beacon_timer);
@@ -5851,12 +5856,12 @@ static int hwsim_cloned_frame_received_nl(struct sk_buff *skb_2,
if (frame_data_len < sizeof(struct ieee80211_hdr_3addr) ||
frame_data_len > IEEE80211_MAX_DATA_LEN)
- goto err;
+ goto out;
/* Allocate new skb here */
skb = alloc_skb(frame_data_len, GFP_KERNEL);
if (skb == NULL)
- goto err;
+ goto out;
/* Copy the data */
skb_put_data(skb, frame_data, frame_data_len);
@@ -5881,10 +5886,17 @@ static int hwsim_cloned_frame_received_nl(struct sk_buff *skb_2,
goto out;
}
+ /*
+ * Serialise against mac80211_hwsim_stop() - mac80211 doesn't allow
+ * frames reported while the HW is down, hence the ->started check
+ * must be under mutex.
+ */
+ mutex_lock(&data2->mutex);
+
/* check if radio is configured properly */
if ((data2->idle && !data2->tmp_chan) || !data2->started)
- goto out;
+ goto out_unlock;
/* A frame is received from user space */
memset(&rx_status, 0, sizeof(rx_status));
@@ -5900,22 +5912,18 @@ static int hwsim_cloned_frame_received_nl(struct sk_buff *skb_2,
iter_data.channel = ieee80211_get_channel(data2->hw->wiphy,
rx_status.freq);
if (!iter_data.channel)
- goto out;
+ goto out_unlock;
rx_status.band = iter_data.channel->band;
- mutex_lock(&data2->mutex);
if (!hwsim_chans_compat(iter_data.channel, channel)) {
ieee80211_iterate_active_interfaces_atomic(
data2->hw, IEEE80211_IFACE_ITER_NORMAL,
mac80211_hwsim_tx_iter, &iter_data);
- if (!iter_data.receive) {
- mutex_unlock(&data2->mutex);
- goto out;
- }
+ if (!iter_data.receive)
+ goto out_unlock;
}
- mutex_unlock(&data2->mutex);
} else if (!channel) {
- goto out;
+ goto out_unlock;
} else {
rx_status.freq = channel->center_freq;
rx_status.band = channel->band;
@@ -5923,7 +5931,7 @@ static int hwsim_cloned_frame_received_nl(struct sk_buff *skb_2,
rx_status.rate_idx = nla_get_u32(info->attrs[HWSIM_ATTR_RX_RATE]);
if (rx_status.rate_idx >= data2->hw->wiphy->bands[rx_status.band]->n_bitrates)
- goto out;
+ goto out_unlock;
rx_status.signal = nla_get_u32(info->attrs[HWSIM_ATTR_SIGNAL]);
hdr = (void *)skb->data;
@@ -5933,10 +5941,11 @@ static int hwsim_cloned_frame_received_nl(struct sk_buff *skb_2,
rx_status.boottime_ns = ktime_get_boottime_ns();
mac80211_hwsim_rx(data2, &rx_status, skb);
+ mutex_unlock(&data2->mutex);
return 0;
-err:
- pr_debug("mac80211_hwsim: error occurred in %s\n", __func__);
+out_unlock:
+ mutex_unlock(&data2->mutex);
out:
dev_kfree_skb(skb);
return -EINVAL;
diff --git a/drivers/net/wireless/virtual/virt_wifi.c b/drivers/net/wireless/virtual/virt_wifi.c
index 976edacb689d..4d36c15c1118 100644
--- a/drivers/net/wireless/virtual/virt_wifi.c
+++ b/drivers/net/wireless/virtual/virt_wifi.c
@@ -432,6 +432,7 @@ static netdev_tx_t virt_wifi_start_xmit(struct sk_buff *skb,
priv->tx_packets++;
if (!priv->is_connected) {
priv->tx_failed++;
+ dev_kfree_skb_any(skb);
return NET_XMIT_DROP;
}
@@ -553,7 +554,6 @@ static int virt_wifi_newlink(struct net *src_net, struct net_device *dev,
}
eth_hw_addr_inherit(dev, priv->lowerdev);
- netif_stacked_transfer_operstate(priv->lowerdev, dev);
dev->ieee80211_ptr = kzalloc(sizeof(*dev->ieee80211_ptr), GFP_KERNEL);
@@ -579,6 +579,8 @@ static int virt_wifi_newlink(struct net *src_net, struct net_device *dev,
goto unregister_netdev;
}
+ netif_stacked_transfer_operstate(priv->lowerdev, dev);
+
dev->priv_destructor = virt_wifi_net_device_destructor;
priv->being_deleted = false;
priv->is_connected = false;
diff --git a/drivers/net/wwan/mhi_wwan_mbim.c b/drivers/net/wwan/mhi_wwan_mbim.c
index 1d7e3ad900c1..4995e9836f76 100644
--- a/drivers/net/wwan/mhi_wwan_mbim.c
+++ b/drivers/net/wwan/mhi_wwan_mbim.c
@@ -252,6 +252,14 @@ static int mbim_rx_verify_ndp16(struct sk_buff *skb, struct usb_cdc_ncm_ndp16 *n
return ret;
}
+static void mhi_mbim_rx_drop(struct mhi_mbim_link *link, struct sk_buff *skb)
+{
+ dev_kfree_skb_any(skb);
+ u64_stats_update_begin(&link->rx_syncp);
+ u64_stats_inc(&link->rx_errors);
+ u64_stats_update_end(&link->rx_syncp);
+}
+
static void mhi_mbim_rx(struct mhi_mbim_context *mbim, struct sk_buff *skb)
{
int ndpoffset;
@@ -321,7 +329,10 @@ static void mhi_mbim_rx(struct mhi_mbim_context *mbim, struct sk_buff *skb)
continue;
skb_put(skbn, dgram_len);
- skb_copy_bits(skb, dgram_offset, skbn->data, dgram_len);
+ if (skb_copy_bits(skb, dgram_offset, skbn->data, dgram_len)) {
+ mhi_mbim_rx_drop(link, skbn);
+ continue;
+ }
switch (skbn->data[0] & 0xf0) {
case 0x40:
@@ -333,10 +344,7 @@ static void mhi_mbim_rx(struct mhi_mbim_context *mbim, struct sk_buff *skb)
default:
net_err_ratelimited("%s: unknown protocol\n",
link->ndev->name);
- dev_kfree_skb_any(skbn);
- u64_stats_update_begin(&link->rx_syncp);
- u64_stats_inc(&link->rx_errors);
- u64_stats_update_end(&link->rx_syncp);
+ mhi_mbim_rx_drop(link, skbn);
continue;
}
@@ -350,9 +358,13 @@ static void mhi_mbim_rx(struct mhi_mbim_context *mbim, struct sk_buff *skb)
unlock:
rcu_read_unlock();
next_ndp:
- /* Other NDP to process? */
- ndpoffset = (int)le16_to_cpu(ndp16.wNextNdpIndex);
- if (!ndpoffset)
+ /* Other NDP to process? The offsets must advance, or a
+ * self-referencing NDP keeps the loop spinning forever.
+ */
+ n = (int)le16_to_cpu(ndp16.wNextNdpIndex);
+ if (n > ndpoffset)
+ ndpoffset = n;
+ else
break;
}
diff --git a/drivers/net/wwan/t7xx/t7xx_netdev.c b/drivers/net/wwan/t7xx/t7xx_netdev.c
index fc0a7cb181df..8f32c2d26931 100644
--- a/drivers/net/wwan/t7xx/t7xx_netdev.c
+++ b/drivers/net/wwan/t7xx/t7xx_netdev.c
@@ -420,6 +420,10 @@ static void t7xx_ccmni_recv_skb(struct t7xx_ccmni_ctrl *ccmni_ctlb, struct sk_bu
skb_cb = T7XX_SKB_CB(skb);
netif_id = skb_cb->netif_idx;
+ if (netif_id >= NIC_DEV_MAX) {
+ dev_kfree_skb(skb);
+ return;
+ }
ccmni = READ_ONCE(ccmni_ctlb->ccmni_inst[netif_id]);
if (!ccmni) {
dev_kfree_skb(skb);
diff --git a/drivers/nfc/microread/microread.c b/drivers/nfc/microread/microread.c
index b0e483ee7eec..8746ba29d7ce 100644
--- a/drivers/nfc/microread/microread.c
+++ b/drivers/nfc/microread/microread.c
@@ -251,9 +251,9 @@ static int microread_start_poll(struct nfc_hci_dev *hdev,
param[1] |= (1 << 1);
if ((im_protocols | tm_protocols) & NFC_PROTO_NFC_DEP_MASK) {
- hdev->gb = nfc_get_local_general_bytes(hdev->ndev,
- &hdev->gb_len);
- if (hdev->gb == NULL || hdev->gb_len == 0) {
+ nfc_get_local_general_bytes(hdev->ndev, hdev->gb,
+ sizeof(hdev->gb), &hdev->gb_len);
+ if (hdev->gb_len == 0) {
im_protocols &= ~NFC_PROTO_NFC_DEP_MASK;
tm_protocols &= ~NFC_PROTO_NFC_DEP_MASK;
}
diff --git a/drivers/nfc/nfcmrvl/fw_dnld.c b/drivers/nfc/nfcmrvl/fw_dnld.c
index 93094418fd24..9716caddb88e 100644
--- a/drivers/nfc/nfcmrvl/fw_dnld.c
+++ b/drivers/nfc/nfcmrvl/fw_dnld.c
@@ -262,9 +262,14 @@ static int process_state_fw_dnld(struct nfcmrvl_private *priv,
* B8..N: payload
*/
- /* Remove NCI HDR */
- skb_pull(skb, 3);
- if (skb->data[0] != HELPER_CMD_PACKET_FORMAT || skb->len != 5) {
+ if (skb->len != NCI_DATA_HDR_SIZE + 5) {
+ nfc_err(priv->dev, "bad command");
+ return -EINVAL;
+ }
+
+ /* Remove NCI header */
+ skb_pull(skb, NCI_DATA_HDR_SIZE);
+ if (skb->data[0] != HELPER_CMD_PACKET_FORMAT) {
nfc_err(priv->dev, "bad command");
return -EINVAL;
}
diff --git a/drivers/nfc/pn533/pn533.c b/drivers/nfc/pn533/pn533.c
index c804c16f4b3b..fd380a853eb9 100644
--- a/drivers/nfc/pn533/pn533.c
+++ b/drivers/nfc/pn533/pn533.c
@@ -1355,10 +1355,11 @@ static int pn533_poll_dep(struct nfc_dev *nfc_dev)
u8 *next, nfcid3[NFC_NFCID3_MAXSIZE];
u8 passive_data[PASSIVE_DATA_LEN] = {0x00, 0xff, 0xff, 0x00, 0x3};
- if (!dev->gb) {
- dev->gb = nfc_get_local_general_bytes(nfc_dev, &dev->gb_len);
-
- if (!dev->gb || !dev->gb_len) {
+ if (!dev->gb_len) {
+ nfc_get_local_general_bytes(nfc_dev, dev->gb,
+ sizeof(dev->gb),
+ &dev->gb_len);
+ if (!dev->gb_len) {
dev->poll_dep = 0;
queue_work(dev->wq, &dev->rf_work);
}
@@ -1660,8 +1661,9 @@ static int pn533_start_poll(struct nfc_dev *nfc_dev,
}
if (tm_protocols) {
- dev->gb = nfc_get_local_general_bytes(nfc_dev, &dev->gb_len);
- if (dev->gb == NULL)
+ nfc_get_local_general_bytes(nfc_dev, dev->gb,
+ sizeof(dev->gb), &dev->gb_len);
+ if (dev->gb_len == 0)
tm_protocols = 0;
}
diff --git a/drivers/nfc/pn533/pn533.h b/drivers/nfc/pn533/pn533.h
index 09e35b8693f5..5ab668e05121 100644
--- a/drivers/nfc/pn533/pn533.h
+++ b/drivers/nfc/pn533/pn533.h
@@ -6,6 +6,8 @@
* Copyright (C) 2012-2013 Tieto Poland
*/
+#include <net/nfc/nfc.h>
+
#define PN533_DEVICE_STD 0x1
#define PN533_DEVICE_PASORI 0x2
#define PN533_DEVICE_ACR122U 0x3
@@ -166,7 +168,7 @@ struct pn533 {
struct timer_list listen_timer;
int cancel_listen;
- u8 *gb;
+ u8 gb[NFC_MAX_GT_LEN];
size_t gb_len;
u8 tgt_available_prots;
diff --git a/drivers/nfc/pn533/usb.c b/drivers/nfc/pn533/usb.c
index 0f12f86ebb02..35f3350cad32 100644
--- a/drivers/nfc/pn533/usb.c
+++ b/drivers/nfc/pn533/usb.c
@@ -319,7 +319,9 @@ static bool pn533_acr122_is_rx_frame_valid(void *_frame, struct pn533 *dev)
if (frame->ccid.type != 0x83)
return false;
- if (!frame->ccid.datalen)
+ if (frame->ccid.datalen < 2 ||
+ frame->ccid.datalen > PN533_ACR122_FRAME_MAX_PAYLOAD_LEN +
+ PN533_ACR122_RX_FRAME_TAIL_LEN)
return false;
if (frame->data[frame->ccid.datalen - 2] == 0x63)
diff --git a/drivers/nfc/pn544/pn544.c b/drivers/nfc/pn544/pn544.c
index 32a61a185142..6da666169ad2 100644
--- a/drivers/nfc/pn544/pn544.c
+++ b/drivers/nfc/pn544/pn544.c
@@ -377,10 +377,9 @@ static int pn544_hci_start_poll(struct nfc_hci_dev *hdev,
return r;
if ((im_protocols | tm_protocols) & NFC_PROTO_NFC_DEP_MASK) {
- hdev->gb = nfc_get_local_general_bytes(hdev->ndev,
- &hdev->gb_len);
- pr_debug("generate local bytes %p\n", hdev->gb);
- if (hdev->gb == NULL || hdev->gb_len == 0) {
+ nfc_get_local_general_bytes(hdev->ndev, hdev->gb,
+ sizeof(hdev->gb), &hdev->gb_len);
+ if (hdev->gb_len == 0) {
im_protocols &= ~NFC_PROTO_NFC_DEP_MASK;
tm_protocols &= ~NFC_PROTO_NFC_DEP_MASK;
}
diff --git a/drivers/nfc/port100.c b/drivers/nfc/port100.c
index 00d8ea6dcb5d..d5544965481b 100644
--- a/drivers/nfc/port100.c
+++ b/drivers/nfc/port100.c
@@ -636,6 +636,13 @@ static void port100_recv_response(struct urb *urb)
in_frame = dev->in_urb->transfer_buffer;
+ if (urb->actual_length < PORT100_FRAME_HEADER_LEN ||
+ urb->actual_length < port100_rx_frame_size(in_frame)) {
+ nfc_err(&dev->interface->dev, "Received a truncated frame\n");
+ cmd->status = -EIO;
+ goto sched_wq;
+ }
+
if (!port100_rx_frame_is_valid(in_frame)) {
nfc_err(&dev->interface->dev, "Received an invalid frame\n");
cmd->status = -EIO;
diff --git a/drivers/nfc/st21nfca/core.c b/drivers/nfc/st21nfca/core.c
index 161caf2675cf..88c4be9ec89d 100644
--- a/drivers/nfc/st21nfca/core.c
+++ b/drivers/nfc/st21nfca/core.c
@@ -351,10 +351,10 @@ static int st21nfca_hci_start_poll(struct nfc_hci_dev *hdev,
if (r < 0)
return r;
} else {
- hdev->gb = nfc_get_local_general_bytes(hdev->ndev,
- &hdev->gb_len);
-
- if (hdev->gb == NULL || hdev->gb_len == 0) {
+ nfc_get_local_general_bytes(hdev->ndev, hdev->gb,
+ sizeof(hdev->gb),
+ &hdev->gb_len);
+ if (hdev->gb_len == 0) {
im_protocols &= ~NFC_PROTO_NFC_DEP_MASK;
tm_protocols &= ~NFC_PROTO_NFC_DEP_MASK;
}
@@ -577,9 +577,7 @@ static int st21nfca_get_iso15693_inventory(struct nfc_hci_dev *hdev,
if (r < 0)
goto exit;
- skb_pull(inventory_skb, 2);
-
- if (inventory_skb->len == 0 ||
+ if (!skb_pull(inventory_skb, 2) || inventory_skb->len < 2 ||
inventory_skb->len > NFC_ISO15693_UID_MAXSIZE) {
r = -EPROTO;
goto exit;
diff --git a/drivers/nfc/st21nfca/i2c.c b/drivers/nfc/st21nfca/i2c.c
index 02c3d11a19c4..4d95e25d3724 100644
--- a/drivers/nfc/st21nfca/i2c.c
+++ b/drivers/nfc/st21nfca/i2c.c
@@ -290,27 +290,36 @@ static int check_crc(u8 *buf, int buflen)
*/
static int st21nfca_hci_i2c_repack(struct sk_buff *skb)
{
- int i, j, r, size;
+ int read, write, r, size;
- if (skb->len < 1 || (skb->len > 1 && skb->data[1] != 0))
+ if (skb->len < ST21NFCA_FRAME_HEADROOM ||
+ !IS_START_OF_FRAME(skb->data))
return -EBADMSG;
size = get_frame_size(skb->data, skb->len);
if (size > 0) {
+ if (size < ST21NFCA_FRAME_HEADROOM + 2)
+ return -EBADMSG;
+
skb_trim(skb, size);
/* remove ST21NFCA byte stuffing for upper layer */
- for (i = 1, j = 0; i < skb->len; i++) {
- if (skb->data[i + j] ==
+ for (read = 1, write = 1; read < skb->len;) {
+ if (skb->data[read] ==
(u8) ST21NFCA_ESCAPE_BYTE_STUFFING) {
- skb->data[i] = skb->data[i + j + 1]
- | ST21NFCA_BYTE_STUFFING_MASK;
- i++;
- j++;
+ if (read + 1 == skb->len)
+ return -EBADMSG;
+
+ skb->data[write++] = skb->data[read + 1]
+ | ST21NFCA_BYTE_STUFFING_MASK;
+ read += 2;
+ } else {
+ skb->data[write++] = skb->data[read++];
}
- skb->data[i] = skb->data[i + j];
}
/* remove byte stuffing useless byte */
- skb_trim(skb, i - j);
+ skb_trim(skb, write);
+ if (skb->len < ST21NFCA_FRAME_HEADROOM + 2)
+ return -EBADMSG;
/* remove ST21NFCA_SOF_EOF from head */
skb_pull(skb, 1);
diff --git a/drivers/nfc/virtual_ncidev.c b/drivers/nfc/virtual_ncidev.c
index 590b038e449e..db15d2669266 100644
--- a/drivers/nfc/virtual_ncidev.c
+++ b/drivers/nfc/virtual_ncidev.c
@@ -195,7 +195,8 @@ static const struct file_operations virtual_ncidev_fops = {
.write = virtual_ncidev_write,
.open = virtual_ncidev_open,
.release = virtual_ncidev_close,
- .unlocked_ioctl = virtual_ncidev_ioctl
+ .unlocked_ioctl = virtual_ncidev_ioctl,
+ .compat_ioctl = compat_ptr_ioctl,
};
static struct miscdevice miscdev = {
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index e8ac7425c97c..cbc0f53398f0 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -9,26 +9,48 @@
#include "virtio_pmem.h"
#include "nd.h"
+struct virtio_pmem_flush_work {
+ struct work_struct work;
+ struct nd_region *nd_region;
+ struct bio *bio;
+};
+
+static void virtio_pmem_req_release(struct kref *kref)
+{
+ struct virtio_pmem_request *req;
+
+ req = container_of(kref, struct virtio_pmem_request, kref);
+ kfree(req);
+}
+
+static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
+{
+ struct virtio_pmem_request *req_buf;
+
+ if (list_empty(&vpmem->req_list))
+ return;
+
+ req_buf = list_first_entry(&vpmem->req_list,
+ struct virtio_pmem_request, list);
+ list_del_init(&req_buf->list);
+ WRITE_ONCE(req_buf->wq_buf_avail, true);
+ wake_up(&req_buf->wq_buf);
+}
+
/* The interrupt handler */
void virtio_pmem_host_ack(struct virtqueue *vq)
{
struct virtio_pmem *vpmem = vq->vdev->priv;
- struct virtio_pmem_request *req_data, *req_buf;
+ struct virtio_pmem_request *req_data;
unsigned long flags;
unsigned int len;
spin_lock_irqsave(&vpmem->pmem_lock, flags);
while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) {
- req_data->done = true;
+ virtio_pmem_wake_one_waiter(vpmem);
+ WRITE_ONCE(req_data->done, true);
wake_up(&req_data->host_acked);
-
- if (!list_empty(&vpmem->req_list)) {
- req_buf = list_first_entry(&vpmem->req_list,
- struct virtio_pmem_request, list);
- req_buf->wq_buf_avail = true;
- wake_up(&req_buf->wq_buf);
- list_del(&req_buf->list);
- }
+ kref_put(&req_data->kref, virtio_pmem_req_release);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
}
@@ -55,11 +77,12 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
return -EIO;
}
- req_data = kmalloc(sizeof(*req_data), GFP_KERNEL);
+ req_data = kmalloc_obj(*req_data);
if (!req_data)
return -ENOMEM;
- req_data->done = false;
+ kref_init(&req_data->kref);
+ WRITE_ONCE(req_data->done, false);
init_waitqueue_head(&req_data->host_acked);
init_waitqueue_head(&req_data->wq_buf);
INIT_LIST_HEAD(&req_data->list);
@@ -76,18 +99,32 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
* to req_list and wait for host_ack to wake us up when free
* slots are available.
*/
- while ((err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
- GFP_ATOMIC)) == -ENOSPC) {
+ for (;;) {
+ err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
+ GFP_ATOMIC);
+ if (!err) {
+ /*
+ * Take the virtqueue reference while @pmem_lock is
+ * held so completion cannot run concurrently.
+ */
+ kref_get(&req_data->kref);
+ break;
+ }
+
+ if (err != -ENOSPC)
+ break;
- dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
- req_data->wq_buf_avail = false;
+ dev_info_ratelimited(&vdev->dev,
+ "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
+ WRITE_ONCE(req_data->wq_buf_avail, false);
list_add_tail(&req_data->list, &vpmem->req_list);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/* A host response results in "host_ack" getting called */
- wait_event(req_data->wq_buf, req_data->wq_buf_avail);
+ wait_event(req_data->wq_buf, READ_ONCE(req_data->wq_buf_avail));
spin_lock_irqsave(&vpmem->pmem_lock, flags);
}
+
err1 = virtqueue_kick(vpmem->req_vq);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/*
@@ -98,39 +135,55 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
dev_info(&vdev->dev, "failed to send command to virtio pmem device\n");
err = -EIO;
} else {
- /* A host repsonse results in "host_ack" getting called */
- wait_event(req_data->host_acked, req_data->done);
+ /* A host response results in "host_ack" getting called */
+ wait_event(req_data->host_acked, READ_ONCE(req_data->done));
err = le32_to_cpu(req_data->resp.ret);
}
- kfree(req_data);
+ kref_put(&req_data->kref, virtio_pmem_req_release);
return err;
};
+static void virtio_pmem_flush_work(struct work_struct *work)
+{
+ struct virtio_pmem_flush_work *flush;
+ int err;
+
+ flush = container_of(work, struct virtio_pmem_flush_work, work);
+ err = virtio_pmem_flush(flush->nd_region);
+ if (err > 0)
+ err = -EIO;
+ if (err)
+ flush->bio->bi_status = errno_to_blk_status(err);
+ bio_endio(flush->bio);
+ kfree(flush);
+}
+
/* The asynchronous flush callback function */
int async_pmem_flush(struct nd_region *nd_region, struct bio *bio)
{
- /*
- * Create child bio for asynchronous flush and chain with
- * parent bio. Otherwise directly call nd_region flush.
- */
- if (bio && bio->bi_iter.bi_sector != -1) {
- struct bio *child = bio_alloc(bio->bi_bdev, 0,
- REQ_OP_WRITE | REQ_PREFLUSH,
- GFP_ATOMIC);
+ struct virtio_device *vdev = nd_region->provider_data;
+ struct virtio_pmem *vpmem = vdev->priv;
+ struct virtio_pmem_flush_work *flush;
+ int err;
- if (!child)
+ if (bio && bio->bi_iter.bi_sector != -1) {
+ flush = kmalloc_obj(*flush, GFP_NOIO);
+ if (!flush)
return -ENOMEM;
- bio_clone_blkg_association(child, bio);
- child->bi_iter.bi_sector = -1;
- bio_chain(child, bio);
- submit_bio(child);
- return 0;
+
+ INIT_WORK(&flush->work, virtio_pmem_flush_work);
+ flush->nd_region = nd_region;
+ flush->bio = bio;
+ queue_work(vpmem->flush_wq, &flush->work);
+ return NVDIMM_FLUSH_ASYNC;
}
- if (virtio_pmem_flush(nd_region))
+
+ err = virtio_pmem_flush(nd_region);
+ if (err > 0)
return -EIO;
- return 0;
+ return err;
};
EXPORT_SYMBOL_GPL(async_pmem_flush);
MODULE_DESCRIPTION("Virtio Persistent Memory Driver");
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 210fb77f51ba..53f6587674d3 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -209,8 +209,14 @@ static void pmem_submit_bio(struct bio *bio)
struct pmem_device *pmem = bio->bi_bdev->bd_disk->private_data;
struct nd_region *nd_region = to_region(pmem);
- if (bio->bi_opf & REQ_PREFLUSH)
- ret = nvdimm_flush(nd_region, bio);
+ if (bio->bi_opf & REQ_PREFLUSH) {
+ ret = nvdimm_flush(nd_region, NULL);
+ if (ret) {
+ bio->bi_status = errno_to_blk_status(ret);
+ bio_endio(bio);
+ return;
+ }
+ }
do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue);
if (do_acct)
@@ -230,8 +236,11 @@ static void pmem_submit_bio(struct bio *bio)
if (do_acct)
bio_end_io_acct(bio, start);
- if (bio->bi_opf & REQ_FUA)
+ if ((bio->bi_opf & REQ_FUA) && !bio->bi_status) {
ret = nvdimm_flush(nd_region, bio);
+ if (ret == NVDIMM_FLUSH_ASYNC)
+ return;
+ }
if (ret)
bio->bi_status = errno_to_blk_status(ret);
diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c
index 849c131f39c7..97ad92fbc8e4 100644
--- a/drivers/nvdimm/region_devs.c
+++ b/drivers/nvdimm/region_devs.c
@@ -1112,7 +1112,10 @@ int nvdimm_flush(struct nd_region *nd_region, struct bio *bio)
if (!nd_region->flush)
rc = generic_nvdimm_flush(nd_region);
else {
- if (nd_region->flush(nd_region, bio))
+ rc = nd_region->flush(nd_region, bio);
+ if (rc > 0)
+ return rc;
+ if (rc && rc != -ENOMEM)
rc = -EIO;
}
diff --git a/drivers/nvdimm/virtio_pmem.c b/drivers/nvdimm/virtio_pmem.c
index 89d3d4fc3ce8..94f324ffdc80 100644
--- a/drivers/nvdimm/virtio_pmem.c
+++ b/drivers/nvdimm/virtio_pmem.c
@@ -67,10 +67,17 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
mutex_init(&vpmem->flush_lock);
vpmem->vdev = vdev;
vdev->priv = vpmem;
+ vpmem->flush_wq = alloc_ordered_workqueue("virtio-pmem-flush",
+ WQ_MEM_RECLAIM);
+ if (!vpmem->flush_wq) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+
err = init_vq(vpmem);
if (err) {
dev_err(&vdev->dev, "failed to initialize virtio pmem vq's\n");
- goto out_err;
+ goto out_wq;
}
if (virtio_has_feature(vdev, VIRTIO_PMEM_F_SHMEM_REGION)) {
@@ -131,6 +138,8 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
nvdimm_bus_unregister(vpmem->nvdimm_bus);
out_vq:
vdev->config->del_vqs(vdev);
+out_wq:
+ destroy_workqueue(vpmem->flush_wq);
out_err:
return err;
}
@@ -138,10 +147,13 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
static void virtio_pmem_remove(struct virtio_device *vdev)
{
struct nvdimm_bus *nvdimm_bus = dev_get_drvdata(&vdev->dev);
+ struct virtio_pmem *vpmem = vdev->priv;
nvdimm_bus_unregister(nvdimm_bus);
+ drain_workqueue(vpmem->flush_wq);
vdev->config->del_vqs(vdev);
virtio_reset_device(vdev);
+ destroy_workqueue(vpmem->flush_wq);
}
static unsigned int features[] = {
diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h
index f72cf17f9518..3af92588bd9d 100644
--- a/drivers/nvdimm/virtio_pmem.h
+++ b/drivers/nvdimm/virtio_pmem.h
@@ -12,11 +12,14 @@
#include <linux/module.h>
#include <uapi/linux/virtio_pmem.h>
+#include <linux/kref.h>
#include <linux/libnvdimm.h>
#include <linux/mutex.h>
#include <linux/spinlock.h>
+#include <linux/workqueue.h>
struct virtio_pmem_request {
+ struct kref kref;
struct virtio_pmem_req req;
struct virtio_pmem_resp resp;
@@ -39,6 +42,9 @@ struct virtio_pmem {
/* Serialize flush requests to the device. */
struct mutex flush_lock;
+ /* Complete asynchronous FUA flushes outside the submit path. */
+ struct workqueue_struct *flush_wq;
+
/* nvdimm bus registers virtio pmem device */
struct nvdimm_bus *nvdimm_bus;
struct nvdimm_bus_descriptor nd_desc;
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 3b391b632558..bb274fecbec4 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1993,20 +1993,40 @@ static void nvme_configure_metadata(struct nvme_ctrl *ctrl,
}
-static void nvme_update_atomic_write_disk_info(struct nvme_ns *ns,
- struct nvme_id_ns *id, struct queue_limits *lim,
- u32 bs, u32 atomic_bs)
+static u32 nvme_configure_atomic_write(struct nvme_ns *ns,
+ struct nvme_id_ns *id, struct queue_limits *lim, u32 bs)
{
- unsigned int boundary = 0;
+ u32 atomic_bs, boundary = 0;
- if (id->nsfeat & NVME_NS_FEAT_ATOMICS && id->nawupf) {
- if (le16_to_cpu(id->nabspf))
+ /*
+ * We do not support an offset for the atomic boundaries.
+ */
+ if (id->nabo)
+ return bs;
+
+ if ((id->nsfeat & NVME_NS_FEAT_ATOMICS) && id->nawupf) {
+ /*
+ * Use the per-namespace atomic write unit when available.
+ */
+ atomic_bs = (1 + le16_to_cpu(id->nawupf)) * bs;
+ if (id->nabspf)
boundary = (le16_to_cpu(id->nabspf) + 1) * bs;
+ } else {
+ /*
+ * Use the controller wide atomic write unit. This sucks
+ * because the limit is defined in terms of logical blocks while
+ * namespaces can have different formats, and because there is
+ * no clear language in the specification prohibiting different
+ * values for different controllers in the subsystem.
+ */
+ atomic_bs = (1 + ns->ctrl->subsys->awupf) * bs;
}
+
lim->atomic_write_hw_max = atomic_bs;
lim->atomic_write_hw_boundary = boundary;
lim->atomic_write_hw_unit_min = bs;
lim->atomic_write_hw_unit_max = rounddown_pow_of_two(atomic_bs);
+ return atomic_bs;
}
static u32 nvme_max_drv_segments(struct nvme_ctrl *ctrl)
@@ -2044,20 +2064,8 @@ static bool nvme_update_disk_info(struct nvme_ns *ns, struct nvme_id_ns *id,
valid = false;
}
- atomic_bs = phys_bs = bs;
- if (id->nabo == 0) {
- /*
- * Bit 1 indicates whether NAWUPF is defined for this namespace
- * and whether it should be used instead of AWUPF. If NAWUPF ==
- * 0 then AWUPF must be used instead.
- */
- if (id->nsfeat & NVME_NS_FEAT_ATOMICS && id->nawupf)
- atomic_bs = (1 + le16_to_cpu(id->nawupf)) * bs;
- else
- atomic_bs = (1 + ns->ctrl->subsys->awupf) * bs;
-
- nvme_update_atomic_write_disk_info(ns, id, lim, bs, atomic_bs);
- }
+ phys_bs = bs;
+ atomic_bs = nvme_configure_atomic_write(ns, id, lim, bs);
if (id->nsfeat & NVME_NS_FEAT_IO_OPT) {
/* NPWG = Namespace Preferred Write Granularity */
@@ -2081,6 +2089,7 @@ static bool nvme_update_disk_info(struct nvme_ns *ns, struct nvme_id_ns *id,
lim->max_write_zeroes_sectors = UINT_MAX;
else
lim->max_write_zeroes_sectors = ns->ctrl->max_zeroes_sectors;
+ nvme_config_discard(ns, lim);
return valid;
}
@@ -2155,6 +2164,7 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
struct nvme_id_ns *id;
sector_t capacity;
unsigned lbaf;
+ bool valid;
int ret;
ret = nvme_identify_ns(ns->ctrl, info->nsid, &id);
@@ -2191,12 +2201,31 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
nvme_set_ctrl_limits(ns->ctrl, &lim);
nvme_configure_metadata(ns->ctrl, ns->head, id, nvm, info);
nvme_set_chunk_sectors(ns, id, &lim);
- if (!nvme_update_disk_info(ns, id, &lim))
+ valid = nvme_update_disk_info(ns, id, &lim);
+
+ if (!valid)
capacity = 0;
- nvme_config_discard(ns, &lim);
+
+ /*
+ * A failed zone info query leaves zi zero-initialized, so skip the
+ * zoned limits update instead of configuring the queue from it.
+ * During a revalidation that keeps the zone geometry the queue was
+ * last validated with; on a first scan the namespace is registered
+ * without zoned limits, so that it is still available as a handle
+ * for admin commands.
+ */
if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) &&
- ns->head->ids.csi == NVME_CSI_ZNS)
- nvme_update_zone_info(ns, &lim, &zi);
+ ns->head->ids.csi == NVME_CSI_ZNS) {
+ if (zi.zone_size)
+ nvme_update_zone_info(ns, &lim, &zi);
+ else
+ dev_warn(ns->ctrl->device,
+ "zone info query failed for nsid %u, %s\n",
+ ns->head->ns_id,
+ blk_queue_is_zoned(ns->disk->queue) ?
+ "keeping the previous zone limits" :
+ "not enabling zoned mode");
+ }
if (ns->ctrl->vwc & NVME_CTRL_VWC_PRESENT)
lim.features |= BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA;
@@ -3014,6 +3043,7 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
memcpy(subsys->model, id->mn, sizeof(subsys->model));
subsys->vendor_id = le16_to_cpu(id->vid);
subsys->cmic = id->cmic;
+ subsys->awupf = le16_to_cpu(id->awupf);
/* Versions prior to 1.4 don't necessarily report a valid type */
if (id->cntrltype == NVME_CTRL_DISC ||
@@ -3029,7 +3059,6 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
kfree(subsys);
return -EINVAL;
}
- subsys->awupf = le16_to_cpu(id->awupf);
nvme_mpath_default_iopolicy(subsys);
subsys->dev.class = &nvme_subsys_class;
@@ -3439,7 +3468,6 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl)
dev_pm_qos_expose_latency_tolerance(ctrl->device);
else if (!ctrl->apst_enabled && prev_apst_enabled)
dev_pm_qos_hide_latency_tolerance(ctrl->device);
-
out_free:
kfree(id);
return ret;
@@ -3974,6 +4002,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
if (list_empty(&ns->head->list))
list_del_init(&ns->head->entry);
mutex_unlock(&ctrl->subsys->lock);
+
+ /* guarantee not available in head->list */
+ synchronize_srcu(&ns->head->srcu);
nvme_put_ns_head(ns->head);
out_cleanup_disk:
put_disk(disk);
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index dcdce7d12e44..5985cdd4c02d 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -442,7 +442,7 @@ struct nvme_subsystem {
u8 cmic;
enum nvme_subsys_type subtype;
u16 vendor_id;
- u16 awupf; /* 0's based awupf value. */
+ u16 awupf; /* 0's based value. */
struct ida ns_ida;
#ifdef CONFIG_NVME_MULTIPATH
enum nvme_iopolicy iopolicy;
diff --git a/drivers/pci/controller/plda/pcie-starfive.c b/drivers/pci/controller/plda/pcie-starfive.c
index aeaebe9cd360..7ea241ac7dee 100644
--- a/drivers/pci/controller/plda/pcie-starfive.c
+++ b/drivers/pci/controller/plda/pcie-starfive.c
@@ -55,7 +55,7 @@ struct starfive_jh7110_pcie {
struct reset_control *resets;
struct clk_bulk_data *clks;
struct regmap *reg_syscon;
- struct gpio_desc *power_gpio;
+ struct regulator *vpcie3v3;
struct gpio_desc *reset_gpio;
struct phy *phy;
@@ -153,11 +153,13 @@ static int starfive_pcie_parse_dt(struct starfive_jh7110_pcie *pcie,
return dev_err_probe(dev, PTR_ERR(pcie->reset_gpio),
"failed to get perst-gpio\n");
- pcie->power_gpio = devm_gpiod_get_optional(dev, "enable",
- GPIOD_OUT_LOW);
- if (IS_ERR(pcie->power_gpio))
- return dev_err_probe(dev, PTR_ERR(pcie->power_gpio),
- "failed to get power-gpio\n");
+ pcie->vpcie3v3 = devm_regulator_get_optional(dev, "vpcie3v3");
+ if (IS_ERR(pcie->vpcie3v3)) {
+ if (PTR_ERR(pcie->vpcie3v3) != -ENODEV)
+ return dev_err_probe(dev, PTR_ERR(pcie->vpcie3v3),
+ "failed to get vpcie3v3 regulator\n");
+ pcie->vpcie3v3 = NULL;
+ }
return 0;
}
@@ -270,8 +272,8 @@ static void starfive_pcie_host_deinit(struct plda_pcie_rp *plda)
container_of(plda, struct starfive_jh7110_pcie, plda);
starfive_pcie_clk_rst_deinit(pcie);
- if (pcie->power_gpio)
- gpiod_set_value_cansleep(pcie->power_gpio, 0);
+ if (pcie->vpcie3v3)
+ regulator_disable(pcie->vpcie3v3);
starfive_pcie_disable_phy(pcie);
}
@@ -302,10 +304,15 @@ static int starfive_pcie_host_init(struct plda_pcie_rp *plda)
ret = starfive_pcie_clk_rst_init(pcie);
if (ret)
- return ret;
+ goto err_disable_phy;
- if (pcie->power_gpio)
- gpiod_set_value_cansleep(pcie->power_gpio, 1);
+ if (pcie->vpcie3v3) {
+ ret = regulator_enable(pcie->vpcie3v3);
+ if (ret) {
+ dev_err_probe(dev, ret, "failed to enable vpcie3v3 regulator\n");
+ goto err_clk_rst;
+ }
+ }
if (pcie->reset_gpio)
gpiod_set_value_cansleep(pcie->reset_gpio, 1);
@@ -374,6 +381,13 @@ static int starfive_pcie_host_init(struct plda_pcie_rp *plda)
dev_info(dev, "port link down\n");
return 0;
+
+err_clk_rst:
+ starfive_pcie_clk_rst_deinit(pcie);
+err_disable_phy:
+ starfive_pcie_disable_phy(pcie);
+
+ return ret;
}
static const struct plda_pcie_host_ops sf_host_ops = {
diff --git a/drivers/pci/hotplug/pnv_php.c b/drivers/pci/hotplug/pnv_php.c
index 4f85e7fe29ec..d573df111a3c 100644
--- a/drivers/pci/hotplug/pnv_php.c
+++ b/drivers/pci/hotplug/pnv_php.c
@@ -800,7 +800,7 @@ static struct pnv_php_slot *pnv_php_alloc_slot(struct device_node *dn)
if (dn->child && PCI_DN(dn->child))
php_slot->slot_no = PCI_SLOT(PCI_DN(dn->child)->devfn);
else
- php_slot->slot_no = -1; /* Placeholder slot */
+ php_slot->slot_no = PCI_SLOT_PLACEHOLDER; /* Placeholder slot */
kref_init(&php_slot->kref);
php_slot->state = PNV_PHP_STATE_INITIALIZED;
diff --git a/drivers/pci/hotplug/rpaphp_slot.c b/drivers/pci/hotplug/rpaphp_slot.c
index 779eab12e981..ac8c512190df 100644
--- a/drivers/pci/hotplug/rpaphp_slot.c
+++ b/drivers/pci/hotplug/rpaphp_slot.c
@@ -85,7 +85,7 @@ int rpaphp_register_slot(struct slot *slot)
struct device_node *child;
u32 my_index;
int retval;
- int slotno = -1;
+ int slotno = PCI_SLOT_PLACEHOLDER;
dbg("%s registering slot:path[%pOF] index[%x], name[%s] pdomain[%x] type[%d]\n",
__func__, slot->dn, slot->index, slot->name,
diff --git a/drivers/pci/of_property.c b/drivers/pci/of_property.c
index 886c236e5de6..71929f73d1b4 100644
--- a/drivers/pci/of_property.c
+++ b/drivers/pci/of_property.c
@@ -91,9 +91,13 @@ static int of_pci_prop_bus_range(struct pci_dev *pdev,
struct of_changeset *ocs,
struct device_node *np)
{
- u32 bus_range[] = { pdev->subordinate->busn_res.start,
- pdev->subordinate->busn_res.end };
+ u32 bus_range[2];
+ if (!pdev->subordinate)
+ return 0;
+
+ bus_range[0] = pdev->subordinate->busn_res.start;
+ bus_range[1] = pdev->subordinate->busn_res.end;
return of_changeset_add_prop_u32_array(ocs, np, "bus-range", bus_range,
ARRAY_SIZE(bus_range));
}
@@ -216,6 +220,9 @@ static int of_pci_prop_intr_map(struct pci_dev *pdev, struct of_changeset *ocs,
int ret;
u8 pin;
+ if (!pdev->subordinate)
+ return 0;
+
pnode = pci_device_to_OF_node(pdev->bus->self);
if (!pnode)
pnode = pci_bus_to_OF_node(pdev->bus);
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 326540ed02c4..4364ed7cdc27 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4919,8 +4919,9 @@ static int pci_reset_hotplug_slot(struct hotplug_slot *hotplug, bool probe)
static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe)
{
- if (dev->multifunction || dev->subordinate || !dev->slot ||
- dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET)
+ if (dev->subordinate || !dev->slot ||
+ dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET ||
+ (dev->multifunction && !dev->slot->per_func_slot))
return -ENOTTY;
return pci_reset_hotplug_slot(dev->slot->hotplug, probe);
diff --git a/drivers/pci/rebar.c b/drivers/pci/rebar.c
index 548894a3fad5..98b49a48bcc6 100644
--- a/drivers/pci/rebar.c
+++ b/drivers/pci/rebar.c
@@ -156,7 +156,6 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size,
int exclude_bars)
{
struct pci_host_bridge *host;
- int old, ret;
u32 sizes;
u16 cmd;
@@ -176,22 +175,6 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size,
if (!(sizes & BIT(size)))
return -EINVAL;
- old = pci_rebar_get_current_size(dev, resno);
- if (old < 0)
- return old;
-
- ret = pci_rebar_set_size(dev, resno, size);
- if (ret)
- return ret;
-
- ret = pci_do_resource_release_and_resize(dev, resno, size, exclude_bars);
- if (ret)
- goto error_resize;
-
- return 0;
-
-error_resize:
- pci_rebar_set_size(dev, resno, old);
- return ret;
+ return pci_do_resource_release_and_resize(dev, resno, size, exclude_bars);
}
EXPORT_SYMBOL(pci_resize_resource);
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index 576c6112790c..34392c35b652 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -2368,6 +2368,11 @@ static int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long t
return 0;
}
+/* Keep the resize follow-up compatible with the 6.12 resource helpers. */
+#define pbus_reassign_bridge_resources(bus, res, saved) \
+ pci_reassign_bridge_resources((bus)->self, (res)->flags, (saved))
+#define resource_assigned(res) ((res)->parent != NULL)
+
int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size,
int exclude_bars)
{
@@ -2375,10 +2380,21 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
unsigned long flags = res->flags;
struct pci_dev_resource *dev_res;
struct pci_bus *bus = pdev->bus;
- struct resource *r;
+ struct pci_dev *bridge = pci_upstream_bridge(pdev);
+ struct resource *b_win, *r;
LIST_HEAD(saved);
unsigned int i;
- int ret = 0;
+ int old, ret;
+
+ b_win = res->parent;
+
+ old = pci_rebar_get_current_size(pdev, resno);
+ if (old < 0)
+ return old;
+
+ ret = pci_rebar_set_size(pdev, resno, size);
+ if (ret)
+ return ret;
down_read(&pci_bus_sem);
@@ -2389,7 +2405,10 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
if (exclude_bars & BIT(i))
continue;
- if (!pci_resource_len(pdev, i) || r->flags != flags)
+ if (!pci_resource_len(pdev, i))
+ continue;
+
+ if (b_win ? r->parent != b_win : r->flags != flags)
continue;
ret = add_to_list(&saved, pdev, r, 0, 0);
@@ -2400,12 +2419,21 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
res->end = res->start + pci_rebar_size_to_bytes(size) - 1;
- if (!bus->self)
- goto out;
+ if (bridge) {
+ ret = pbus_reassign_bridge_resources(bus, res, &saved);
+ if (ret)
+ goto restore;
+ } else {
+ /* No bridge window to adjust; let the core reassign the bus. */
+ pci_bus_assign_resources(bus);
- ret = pci_reassign_bridge_resources(bus->self, res->flags, &saved);
- if (ret)
- goto restore;
+ list_for_each_entry(dev_res, &saved, list) {
+ if (!resource_assigned(dev_res->res)) {
+ ret = -ENOSPC;
+ goto restore;
+ }
+ }
+ }
out:
up_read(&pci_bus_sem);
@@ -2413,14 +2441,22 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
return ret;
restore:
- /* Revert to the old configuration */
+ /*
+ * Revert to the old configuration.
+ *
+ * BAR Size must be restored first because it affects the read-only
+ * bits in BAR (the old address might not be restorable otherwise
+ * due to low address bits).
+ */
+ pci_rebar_set_size(pdev, resno, old);
+
list_for_each_entry(dev_res, &saved, list) {
struct resource *res = dev_res->res;
struct pci_dev *dev = dev_res->dev;
i = res - dev->resource;
- if (res->parent) {
+ if (resource_assigned(res)) {
release_child_resources(res);
pci_release_resource(dev, i);
}
diff --git a/drivers/pci/slot.c b/drivers/pci/slot.c
index ed645c7a4e4b..bd536f30c528 100644
--- a/drivers/pci/slot.c
+++ b/drivers/pci/slot.c
@@ -38,11 +38,20 @@ static const struct sysfs_ops pci_slot_sysfs_ops = {
static ssize_t address_read_file(struct pci_slot *slot, char *buf)
{
- if (slot->number == 0xff)
+ if (slot->number == PCI_SLOT_PLACEHOLDER)
return sysfs_emit(buf, "%04x:%02x\n",
pci_domain_nr(slot->bus),
slot->bus->number);
+ /*
+ * Preserve legacy ABI expectations that hotplug drivers that manage
+ * multiple devices per slot emit 0 for the device number.
+ */
+ if (slot->number == PCI_SLOT_ALL_DEVICES)
+ return sysfs_emit(buf, "%04x:%02x:00\n",
+ pci_domain_nr(slot->bus),
+ slot->bus->number);
+
return sysfs_emit(buf, "%04x:%02x:%02x\n",
pci_domain_nr(slot->bus),
slot->bus->number,
@@ -64,6 +73,23 @@ static ssize_t cur_speed_read_file(struct pci_slot *slot, char *buf)
return bus_speed_read(slot->bus->cur_bus_speed, buf);
}
+static bool pci_dev_matches_slot(struct pci_dev *dev, struct pci_slot *slot)
+{
+ if (slot->per_func_slot)
+ return dev->devfn == slot->number;
+
+ return slot->number == PCI_SLOT_ALL_DEVICES ||
+ PCI_SLOT(dev->devfn) == slot->number;
+}
+
+static bool pci_slot_enabled_per_func(void)
+{
+ if (IS_ENABLED(CONFIG_S390))
+ return true;
+
+ return false;
+}
+
static void pci_slot_release(struct kobject *kobj)
{
struct pci_dev *dev;
@@ -74,7 +100,7 @@ static void pci_slot_release(struct kobject *kobj)
down_read(&pci_bus_sem);
list_for_each_entry(dev, &slot->bus->devices, bus_list)
- if (PCI_SLOT(dev->devfn) == slot->number)
+ if (pci_dev_matches_slot(dev, slot))
dev->slot = NULL;
up_read(&pci_bus_sem);
@@ -167,7 +193,7 @@ void pci_dev_assign_slot(struct pci_dev *dev)
mutex_lock(&pci_slot_mutex);
list_for_each_entry(slot, &dev->bus->slots, list)
- if (PCI_SLOT(dev->devfn) == slot->number)
+ if (pci_dev_matches_slot(dev, slot))
dev->slot = slot;
mutex_unlock(&pci_slot_mutex);
}
@@ -189,7 +215,8 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr)
/**
* pci_create_slot - create or increment refcount for physical PCI slot
* @parent: struct pci_bus of parent bridge
- * @slot_nr: PCI_SLOT(pci_dev->devfn) or -1 for placeholder
+ * @slot_nr: PCI_SLOT(pci_dev->devfn), PCI_SLOT_PLACEHOLDER for placeholder, or
+ * PCI_SLOT_ALL_DEVICES
* @name: user visible string presented in /sys/bus/pci/slots/<name>
* @hotplug: set if caller is hotplug driver, NULL otherwise
*
@@ -214,15 +241,26 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr)
* In most cases, @pci_bus, @slot_nr will be sufficient to uniquely identify
* a slot. There is one notable exception - pSeries (rpaphp), where the
* @slot_nr cannot be determined until a device is actually inserted into
- * the slot. In this scenario, the caller may pass -1 for @slot_nr.
+ * the slot. In this scenario, the caller may pass PCI_SLOT_PLACEHOLDER for @slot_nr.
*
* The following semantics are imposed when the caller passes @slot_nr ==
- * -1. First, we no longer check for an existing %struct pci_slot, as there
- * may be many slots with @slot_nr of -1. The other change in semantics is
- * user-visible, which is the 'address' parameter presented in sysfs will
- * consist solely of a dddd:bb tuple, where dddd is the PCI domain of the
- * %struct pci_bus and bb is the bus number. In other words, the devfn of
- * the 'placeholder' slot will not be displayed.
+ * PCI_SLOT_PLACEHOLDER. First, we no longer check for an existing %struct
+ * pci_slot, as there may be many slots with @slot_nr of
+ * PCI_SLOT_PLACEHOLDER. The other change in semantics is user-visible,
+ * which is the 'address' parameter presented in sysfs will consist solely
+ * of a dddd:bb tuple, where dddd is the PCI domain of the %struct pci_bus
+ * and bb is the bus number. In other words, the devfn of the 'placeholder'
+ * slot will not be displayed.
+ *
+ * Bus-wide slots:
+ * For PCIe hotplug, the physical slot encompasses the entire secondary
+ * bus, not just a single device number. If the device supports ARI and ARI
+ * Forwarding is enabled in the upstream bridge, a multi-function device
+ * may include functions that appear to have several different device
+ * numbers, i.e., PCI_SLOT() values. Pass @slot_nr == PCI_SLOT_ALL_DEVICES
+ * to create a slot that matches all devices on the bus. Unlike placeholder
+ * slots, bus-wide slots go through normal slot lookup and reuse existing
+ * slots if present.
*/
struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
const char *name,
@@ -235,7 +273,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
mutex_lock(&pci_slot_mutex);
- if (slot_nr == -1)
+ if (slot_nr == PCI_SLOT_PLACEHOLDER)
goto placeholder;
/*
@@ -265,6 +303,9 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
slot->bus = pci_bus_get(parent);
slot->number = slot_nr;
+ if (pci_slot_enabled_per_func())
+ slot->per_func_slot = 1;
+
slot->kobj.kset = pci_slots_kset;
slot_name = make_slot_name(name);
@@ -287,7 +328,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
down_read(&pci_bus_sem);
list_for_each_entry(dev, &parent->devices, bus_list)
- if (PCI_SLOT(dev->devfn) == slot_nr)
+ if (pci_dev_matches_slot(dev, slot))
dev->slot = slot;
up_read(&pci_bus_sem);
diff --git a/drivers/perf/arm-cmn.c b/drivers/perf/arm-cmn.c
index 892bc6b5875a..3a4541c51863 100644
--- a/drivers/perf/arm-cmn.c
+++ b/drivers/perf/arm-cmn.c
@@ -1395,13 +1395,14 @@ static void arm_cmn_claim_wp_idx(struct arm_cmn_dtm *dtm,
static u32 arm_cmn_wp_config(struct perf_event *event, int wp_idx)
{
+ struct arm_cmn *cmn = to_cmn(event->pmu);
u32 config;
u32 dev = CMN_EVENT_WP_DEV_SEL(event);
u32 chn = CMN_EVENT_WP_CHN_SEL(event);
u32 grp = CMN_EVENT_WP_GRP(event);
u32 exc = CMN_EVENT_WP_EXCLUSIVE(event);
u32 combine = CMN_EVENT_WP_COMBINE(event);
- bool is_cmn600 = to_cmn(event->pmu)->part == PART_CMN600;
+ bool is_cmn600 = cmn->part == PART_CMN600;
/* CMN-600 supports only primary and secondary matching groups */
if (is_cmn600)
@@ -1409,8 +1410,11 @@ static u32 arm_cmn_wp_config(struct perf_event *event, int wp_idx)
config = FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_DEV_SEL, dev) |
FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_CHN_SEL, chn) |
- FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_GRP, grp) |
- FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_DEV_SEL2, dev >> 1);
+ FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_GRP, grp);
+
+ if (!cmn->multi_dtm)
+ config |= FIELD_PREP(CMN_DTM_WPn_CONFIG_WP_DEV_SEL2, dev >> 1);
+
if (exc)
config |= is_cmn600 ? CMN600_WPn_CONFIG_WP_EXCLUSIVE :
CMN_DTM_WPn_CONFIG_WP_EXCLUSIVE;
diff --git a/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.c b/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.c
index bbfe11d6a69d..9f2c4db84347 100644
--- a/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.c
+++ b/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.c
@@ -34,7 +34,7 @@ mtk_phy_tmds_clk_ratio(struct mtk_hdmi_phy *hdmi_phy, bool enable)
* clock bit ratio 1:40, under 3.4Gbps, clock bit ratio 1:10
*/
if (enable)
- mtk_phy_update_field(regs + HDMI20_CLK_CFG, REG_TXC_DIV, 3);
+ mtk_phy_update_field(regs + HDMI20_CLK_CFG, REG_TXC_DIV, VAL_TXC_DIV4);
else
mtk_phy_clear_bits(regs + HDMI20_CLK_CFG, REG_TXC_DIV);
}
@@ -288,7 +288,7 @@ static int mtk_hdmi_pll_calc(struct mtk_hdmi_phy *hdmi_phy, struct clk_hw *hw,
posdiv2 = 1;
/* Digital clk divider, max /32 */
- digital_div = div_u64(ns_hdmipll_ck, posdiv1 * posdiv2 * pixel_clk);
+ digital_div = div64_u64(ns_hdmipll_ck, posdiv1 * posdiv2 * pixel_clk);
if (!(digital_div <= 32 && digital_div >= 1))
return -EINVAL;
diff --git a/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.h b/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.h
index 22a68dc9550c..8e118a10ffbf 100644
--- a/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.h
+++ b/drivers/phy/mediatek/phy-mtk-hdmi-mt8195.h
@@ -17,6 +17,9 @@
#define HDMI20_CLK_CFG 0x70
#define REG_TXC_DIV GENMASK(31, 30)
+#define VAL_TXC_DIV2 1
+#define VAL_TXC_DIV4 2
+#define VAL_TXC_DIV8 3
#define HDMI_1_CFG_0 0x00
#define RG_HDMITX21_DRV_IBIAS_CLK GENMASK(10, 5)
diff --git a/drivers/phy/renesas/phy-rcar-gen3-usb2.c b/drivers/phy/renesas/phy-rcar-gen3-usb2.c
index 256c807e7066..8e5c3f2a4005 100644
--- a/drivers/phy/renesas/phy-rcar-gen3-usb2.c
+++ b/drivers/phy/renesas/phy-rcar-gen3-usb2.c
@@ -23,6 +23,7 @@
#include <linux/reset.h>
#include <linux/string.h>
#include <linux/usb/of.h>
+#include <linux/wait.h>
#include <linux/workqueue.h>
/******* USB2.0 Host registers (original offset is +0x200) *******/
@@ -88,6 +89,13 @@
#define USB2_OBINT_IDCHG_EN BIT(0)
#define USB2_LINECTRL1_USB2_IDMON BIT(0)
+/*
+ * The OTG initialization is expected to finish in 20ms. Choose a large enough
+ * timeout to avoid waiters exit prematurely the waiting section under heavy
+ * CPU load.
+ */
+#define USB2_OTG_INIT_TIMEOUT msecs_to_jiffies(120)
+
#define NUM_OF_PHYS 4
enum rcar_gen3_phy_index {
PHY_INDEX_BOTH_HC,
@@ -118,6 +126,7 @@ struct rcar_gen3_chan {
struct rcar_gen3_phy rphys[NUM_OF_PHYS];
struct regulator *vbus;
struct work_struct work;
+ wait_queue_head_t otg_init_done;
spinlock_t lock; /* protects access to hardware and driver data structure. */
enum usb_dr_mode dr_mode;
u32 obint_enable_bits;
@@ -125,6 +134,13 @@ struct rcar_gen3_chan {
bool is_otg_channel;
bool uses_otg_pins;
bool soc_no_adp_ctrl;
+ /*
+ * The OTG can be initialized only once and needs to release the spinlock
+ * and wait for 20 ms due to hardware constraints. If a thread executes
+ * PHY configuration code while the OTG PHY is waiting for the 20 ms, the
+ * thread will have to wait for the OTG PHY initialization to complete.
+ */
+ bool otg_initializing;
};
struct rcar_gen3_phy_drv_data {
@@ -347,26 +363,58 @@ static ssize_t role_store(struct device *dev, struct device_attribute *attr,
struct rcar_gen3_chan *ch = dev_get_drvdata(dev);
bool is_b_device;
enum phy_mode cur_mode, new_mode;
+ int retries = NUM_OF_PHYS;
+ unsigned long flags;
+ int ret = -EIO;
- guard(spinlock_irqsave)(&ch->lock);
+ spin_lock_irqsave(&ch->lock, flags);
- if (!ch->is_otg_channel || !rcar_gen3_is_any_otg_rphy_initialized(ch))
- return -EIO;
+ if (!ch->is_otg_channel)
+ goto unlock;
+
+ while (retries-- && ch->otg_initializing) {
+ spin_unlock_irqrestore(&ch->lock, flags);
+
+ ret = wait_event_timeout(ch->otg_init_done, !ch->otg_initializing,
+ USB2_OTG_INIT_TIMEOUT);
+ ret = ret ? 0 : -ETIMEDOUT;
+ if (ret && !retries)
+ goto exit;
+
+ spin_lock_irqsave(&ch->lock, flags);
+ }
+
+ /* If another thread started a new initialization just return -EBUSY. */
+ if (ch->otg_initializing) {
+ ret = -EBUSY;
+ goto unlock;
+ } else {
+ ret = 0;
+ }
+
+ if (!rcar_gen3_is_any_otg_rphy_initialized(ch)) {
+ ret = -EIO;
+ goto unlock;
+ }
- if (sysfs_streq(buf, "host"))
+ if (sysfs_streq(buf, "host")) {
new_mode = PHY_MODE_USB_HOST;
- else if (sysfs_streq(buf, "peripheral"))
+ } else if (sysfs_streq(buf, "peripheral")) {
new_mode = PHY_MODE_USB_DEVICE;
- else
- return -EINVAL;
+ } else {
+ ret = -EINVAL;
+ goto unlock;
+ }
/* is_b_device: true is B-Device. false is A-Device. */
is_b_device = rcar_gen3_check_id(ch);
cur_mode = rcar_gen3_get_phy_mode(ch);
/* If current and new mode is the same, this returns the error */
- if (cur_mode == new_mode)
- return -EINVAL;
+ if (cur_mode == new_mode) {
+ ret = -EINVAL;
+ goto unlock;
+ }
if (new_mode == PHY_MODE_USB_HOST) { /* And is_host must be false */
if (!is_b_device) /* A-Peripheral */
@@ -380,7 +428,10 @@ static ssize_t role_store(struct device *dev, struct device_attribute *attr,
rcar_gen3_init_for_peri(ch);
}
- return count;
+unlock:
+ spin_unlock_irqrestore(&ch->lock, flags);
+exit:
+ return ret ?: count;
}
static ssize_t role_show(struct device *dev, struct device_attribute *attr,
@@ -396,14 +447,11 @@ static ssize_t role_show(struct device *dev, struct device_attribute *attr,
}
static DEVICE_ATTR_RW(role);
-static void rcar_gen3_init_otg(struct rcar_gen3_chan *ch)
+static void rcar_gen3_init_otg_phase0(struct rcar_gen3_chan *ch)
{
void __iomem *usb2_base = ch->base;
u32 val;
- if (!ch->is_otg_channel || rcar_gen3_is_any_otg_rphy_initialized(ch))
- return;
-
/* Should not use functions of read-modify-write a register */
val = readl(usb2_base + USB2_LINECTRL1);
val = (val & ~USB2_LINECTRL1_DP_RPD) | USB2_LINECTRL1_DPRPD_EN |
@@ -417,7 +465,11 @@ static void rcar_gen3_init_otg(struct rcar_gen3_chan *ch)
val = readl(usb2_base + USB2_ADPCTRL);
writel(val | USB2_ADPCTRL_IDPULLUP, usb2_base + USB2_ADPCTRL);
}
- mdelay(20);
+}
+
+static void rcar_gen3_init_otg_phase1(struct rcar_gen3_chan *ch)
+{
+ void __iomem *usb2_base = ch->base;
writel(0xffffffff, usb2_base + USB2_OBINTSTA);
writel(ch->obint_enable_bits, usb2_base + USB2_OBINTEN);
@@ -431,6 +483,7 @@ static irqreturn_t rcar_gen3_phy_usb2_irq(int irq, void *_ch)
void __iomem *usb2_base = ch->base;
struct device *dev = ch->dev;
irqreturn_t ret = IRQ_NONE;
+ unsigned long flags;
u32 status;
pm_runtime_get_noresume(dev);
@@ -438,29 +491,98 @@ static irqreturn_t rcar_gen3_phy_usb2_irq(int irq, void *_ch)
if (pm_runtime_suspended(dev))
goto rpm_put;
- scoped_guard(spinlock, &ch->lock) {
- status = readl(usb2_base + USB2_OBINTSTA);
- if (status & ch->obint_enable_bits) {
- dev_vdbg(dev, "%s: %08x\n", __func__, status);
- writel(ch->obint_enable_bits, usb2_base + USB2_OBINTSTA);
- rcar_gen3_device_recognition(ch);
- ret = IRQ_HANDLED;
- }
+ spin_lock_irqsave(&ch->lock, flags);
+
+ status = readl(usb2_base + USB2_OBINTSTA);
+ if (status & ch->obint_enable_bits) {
+ dev_vdbg(dev, "%s: %08x\n", __func__, status);
+ writel(ch->obint_enable_bits, usb2_base + USB2_OBINTSTA);
+
+ ret = IRQ_HANDLED;
+
+ /* This should not happen! */
+ if (ch->otg_initializing)
+ goto unlock;
+
+ rcar_gen3_device_recognition(ch);
}
+unlock:
+ spin_unlock_irqrestore(&ch->lock, flags);
rpm_put:
pm_runtime_put_noidle(dev);
return ret;
}
+static void rcar_gen3_phy_usb2_irqs_mask_all(struct rcar_gen3_chan *channel,
+ u32 *masked_irqs_bits)
+{
+ u32 val, bitmask = USB2_INT_ENABLE_UCOM_INTEN;
+ void __iomem *usb2_base = channel->base;
+
+ for (unsigned int i = 0; i < NUM_OF_PHYS; i++)
+ bitmask |= channel->rphys[i].int_enable_bits;
+
+ val = readl(usb2_base + USB2_INT_ENABLE);
+ *masked_irqs_bits = val & bitmask;
+ val &= ~bitmask;
+ writel(val, usb2_base + USB2_INT_ENABLE);
+
+ /*
+ * Don't report channel->obint_enable_bits IRQs. These are
+ * unmasked anyway in rcar_gen3_init_otg_phase1().
+ */
+ val = readl(usb2_base + USB2_OBINTEN);
+ val &= ~channel->obint_enable_bits;
+ writel(val, usb2_base + USB2_OBINTEN);
+}
+
+static void rcar_gen3_phy_usb2_irqs_unmask(struct rcar_gen3_chan *channel,
+ u32 irqs_bits)
+{
+ u32 val, bitmask = USB2_INT_ENABLE_UCOM_INTEN;
+ void __iomem *usb2_base = channel->base;
+
+ for (unsigned int i = 0; i < NUM_OF_PHYS; i++)
+ bitmask |= channel->rphys[i].int_enable_bits;
+
+ val = readl(usb2_base + USB2_INT_ENABLE);
+ val &= ~bitmask;
+ val |= irqs_bits;
+ writel(val, usb2_base + USB2_INT_ENABLE);
+}
+
static int rcar_gen3_phy_usb2_init(struct phy *p)
{
struct rcar_gen3_phy *rphy = phy_get_drvdata(p);
struct rcar_gen3_chan *channel = rphy->ch;
void __iomem *usb2_base = channel->base;
+ int retries = NUM_OF_PHYS;
+ unsigned long flags;
u32 val;
+ int ret;
- guard(spinlock_irqsave)(&channel->lock);
+ spin_lock_irqsave(&channel->lock, flags);
+
+ while (retries-- && channel->otg_initializing) {
+ spin_unlock_irqrestore(&channel->lock, flags);
+
+ ret = wait_event_timeout(channel->otg_init_done, !channel->otg_initializing,
+ USB2_OTG_INIT_TIMEOUT);
+ ret = ret ? 0 : -ETIMEDOUT;
+ if (ret && !retries)
+ return ret;
+
+ spin_lock_irqsave(&channel->lock, flags);
+ }
+
+ /* If another thread started a new initialization just return -EBUSY. */
+ if (channel->otg_initializing) {
+ ret = -EBUSY;
+ goto unlock;
+ } else {
+ ret = 0;
+ }
/* Initialize USB2 part */
val = readl(usb2_base + USB2_INT_ENABLE);
@@ -473,12 +595,30 @@ static int rcar_gen3_phy_usb2_init(struct phy *p)
}
/* Initialize otg part (only if we initialize a PHY with IRQs). */
- if (rphy->int_enable_bits)
- rcar_gen3_init_otg(channel);
+ if (rphy->int_enable_bits && channel->is_otg_channel &&
+ !rcar_gen3_is_any_otg_rphy_initialized(channel)) {
+ u32 masked_irq_bits = 0;
+
+ rcar_gen3_init_otg_phase0(channel);
+ rcar_gen3_phy_usb2_irqs_mask_all(channel, &masked_irq_bits);
+ channel->otg_initializing = true;
+ spin_unlock_irqrestore(&channel->lock, flags);
+
+ fsleep(20000);
+
+ spin_lock_irqsave(&channel->lock, flags);
+ rcar_gen3_phy_usb2_irqs_unmask(channel, masked_irq_bits);
+ rcar_gen3_init_otg_phase1(channel);
+ channel->otg_initializing = false;
+ wake_up_all(&channel->otg_init_done);
+ }
rphy->initialized = true;
- return 0;
+unlock:
+ spin_unlock_irqrestore(&channel->lock, flags);
+
+ return ret;
}
static int rcar_gen3_phy_usb2_exit(struct phy *p)
@@ -486,9 +626,32 @@ static int rcar_gen3_phy_usb2_exit(struct phy *p)
struct rcar_gen3_phy *rphy = phy_get_drvdata(p);
struct rcar_gen3_chan *channel = rphy->ch;
void __iomem *usb2_base = channel->base;
+ int retries = NUM_OF_PHYS;
+ unsigned long flags;
u32 val;
+ int ret;
- guard(spinlock_irqsave)(&channel->lock);
+ spin_lock_irqsave(&channel->lock, flags);
+
+ while (retries-- && channel->otg_initializing) {
+ spin_unlock_irqrestore(&channel->lock, flags);
+
+ ret = wait_event_timeout(channel->otg_init_done, !channel->otg_initializing,
+ USB2_OTG_INIT_TIMEOUT);
+ ret = ret ? 0 : -ETIMEDOUT;
+ if (ret && !retries)
+ return ret;
+
+ spin_lock_irqsave(&channel->lock, flags);
+ }
+
+ /* If another thread started a new initialization just return -EBUSY. */
+ if (channel->otg_initializing) {
+ ret = -EBUSY;
+ goto unlock;
+ } else {
+ ret = 0;
+ }
rphy->initialized = false;
@@ -498,7 +661,9 @@ static int rcar_gen3_phy_usb2_exit(struct phy *p)
val &= ~USB2_INT_ENABLE_UCOM_INTEN;
writel(val, usb2_base + USB2_INT_ENABLE);
- return 0;
+unlock:
+ spin_unlock_irqrestore(&channel->lock, flags);
+ return ret;
}
static int rcar_gen3_phy_usb2_power_on(struct phy *p)
@@ -506,8 +671,10 @@ static int rcar_gen3_phy_usb2_power_on(struct phy *p)
struct rcar_gen3_phy *rphy = phy_get_drvdata(p);
struct rcar_gen3_chan *channel = rphy->ch;
void __iomem *usb2_base = channel->base;
+ int retries = NUM_OF_PHYS;
+ unsigned long flags;
u32 val;
- int ret = 0;
+ int ret;
if (channel->vbus) {
ret = regulator_enable(channel->vbus);
@@ -515,7 +682,27 @@ static int rcar_gen3_phy_usb2_power_on(struct phy *p)
return ret;
}
- guard(spinlock_irqsave)(&channel->lock);
+ spin_lock_irqsave(&channel->lock, flags);
+
+ while (retries-- && channel->otg_initializing) {
+ spin_unlock_irqrestore(&channel->lock, flags);
+
+ ret = wait_event_timeout(channel->otg_init_done, !channel->otg_initializing,
+ USB2_OTG_INIT_TIMEOUT);
+ ret = ret ? 0 : -ETIMEDOUT;
+ if (ret && !retries)
+ goto disable_regulator;
+
+ spin_lock_irqsave(&channel->lock, flags);
+ }
+
+ /* If another thread started a new initialization just return -EBUSY. */
+ if (channel->otg_initializing) {
+ ret = -EBUSY;
+ goto unlock;
+ } else {
+ ret = 0;
+ }
if (!rcar_gen3_are_all_rphys_power_off(channel))
goto out;
@@ -530,27 +717,59 @@ static int rcar_gen3_phy_usb2_power_on(struct phy *p)
/* The powered flag should be set for any other phys anyway */
rphy->powered = true;
- return 0;
+unlock:
+ spin_unlock_irqrestore(&channel->lock, flags);
+
+disable_regulator:
+ if (ret && channel->vbus)
+ regulator_disable(channel->vbus);
+
+ return ret;
}
static int rcar_gen3_phy_usb2_power_off(struct phy *p)
{
struct rcar_gen3_phy *rphy = phy_get_drvdata(p);
struct rcar_gen3_chan *channel = rphy->ch;
- int ret = 0;
+ int retries = NUM_OF_PHYS;
+ unsigned long flags;
+ int ret;
- scoped_guard(spinlock_irqsave, &channel->lock) {
- rphy->powered = false;
+ spin_lock_irqsave(&channel->lock, flags);
- if (rcar_gen3_are_all_rphys_power_off(channel)) {
- u32 val = readl(channel->base + USB2_USBCTR);
+ while (retries-- && channel->otg_initializing) {
+ spin_unlock_irqrestore(&channel->lock, flags);
- val |= USB2_USBCTR_PLL_RST;
- writel(val, channel->base + USB2_USBCTR);
- }
+ ret = wait_event_timeout(channel->otg_init_done, !channel->otg_initializing,
+ USB2_OTG_INIT_TIMEOUT);
+ ret = ret ? 0 : -ETIMEDOUT;
+ if (ret && !retries)
+ return ret;
+
+ spin_lock_irqsave(&channel->lock, flags);
+ }
+
+ /* If another thread started a new initialization just return -EBUSY. */
+ if (channel->otg_initializing) {
+ ret = -EBUSY;
+ goto unlock;
+ } else {
+ ret = 0;
+ }
+
+ rphy->powered = false;
+
+ if (rcar_gen3_are_all_rphys_power_off(channel)) {
+ u32 val = readl(channel->base + USB2_USBCTR);
+
+ val |= USB2_USBCTR_PLL_RST;
+ writel(val, channel->base + USB2_USBCTR);
}
- if (channel->vbus)
+unlock:
+ spin_unlock_irqrestore(&channel->lock, flags);
+
+ if (!ret && channel->vbus)
ret = regulator_disable(channel->vbus);
return ret;
@@ -774,6 +993,7 @@ static int rcar_gen3_phy_usb2_probe(struct platform_device *pdev)
channel->obint_enable_bits = USB2_OBINT_IDCHG_EN;
spin_lock_init(&channel->lock);
+ init_waitqueue_head(&channel->otg_init_done);
for (i = 0; i < NUM_OF_PHYS; i++) {
channel->rphys[i].phy = devm_phy_create(dev, NULL,
phy_data->phy_usb2_ops);
diff --git a/drivers/pinctrl/meson/pinctrl-meson-s4.c b/drivers/pinctrl/meson/pinctrl-meson-s4.c
index 872948699e9f..365dafe457a9 100644
--- a/drivers/pinctrl/meson/pinctrl-meson-s4.c
+++ b/drivers/pinctrl/meson/pinctrl-meson-s4.c
@@ -854,7 +854,7 @@ static const char * const i2c1_groups[] = {
static const char * const i2c2_groups[] = {
"i2c2_sda_d", "i2c2_scl_d",
"i2c2_sda_h8", "i2c2_scl_h9",
- "i2c2_sda_h0", "i2c2_scl_h1l,"
+ "i2c2_sda_h0", "i2c2_scl_h1",
};
static const char * const i2c3_groups[] = {
diff --git a/drivers/pinctrl/pinctrl-single.c b/drivers/pinctrl/pinctrl-single.c
index a2fb549307ad..35c7c0f1c843 100644
--- a/drivers/pinctrl/pinctrl-single.c
+++ b/drivers/pinctrl/pinctrl-single.c
@@ -1626,7 +1626,8 @@ static int pcs_irq_init_chained_handler(struct pcs_device *pcs,
&pcs_irqdomain_ops,
pcs_soc);
if (!pcs->domain) {
- irq_set_chained_handler(pcs_soc->irq, NULL);
+ pcs_irq_free(pcs);
+ pcs_soc->irq = -1;
return -EINVAL;
}
diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.c b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
index ce811cb43651..e182812c66fa 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sunxi.c
+++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
@@ -792,6 +792,21 @@ static void sunxi_pmx_set(struct pinctrl_dev *pctldev,
writel((readl(pctl->membase + reg) & ~mask) | config << shift,
pctl->membase + reg);
+ /*
+ * A pin muxed to gpio_out directly through a pinmux node bypasses
+ * sunxi_pinctrl_gpio_set() and drives whatever its output latch
+ * holds. Now that the pin is in output mode the data register
+ * reads back the latch, so refresh the shadow to keep such pins
+ * driving their pre-existing level.
+ */
+ if (config == SUN4I_FUNC_OUTPUT) {
+ u32 *shadow = &pctl->dat_shadow[pin / PINS_PER_BANK];
+
+ sunxi_data_reg(pctl, pin, ®, &shift, &mask);
+ *shadow = (*shadow & ~mask) |
+ (readl(pctl->membase + reg) & mask);
+ }
+
raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
@@ -948,21 +963,29 @@ static void sunxi_pinctrl_gpio_set(struct gpio_chip *chip,
unsigned offset, int value)
{
struct sunxi_pinctrl *pctl = gpiochip_get_data(chip);
- u32 reg, shift, mask, val;
+ u32 *shadow = &pctl->dat_shadow[offset / PINS_PER_BANK];
+ u32 reg, shift, mask;
unsigned long flags;
sunxi_data_reg(pctl, offset, ®, &shift, &mask);
raw_spin_lock_irqsave(&pctl->lock, flags);
- val = readl(pctl->membase + reg);
-
+ /*
+ * Reading the data register returns the pin level, not the output
+ * latch, for pins muxed as inputs. A read-modify-write based on
+ * the register would therefore corrupt the latches of input-muxed
+ * pins in the same bank (e.g. an emulated open-drain I2C line
+ * released high), making them drive the wrong level once switched
+ * to output. Base the read-modify-write on a shadow copy of the
+ * latches instead.
+ */
if (value)
- val |= mask;
+ *shadow |= mask;
else
- val &= ~mask;
+ *shadow &= ~mask;
- writel(val, pctl->membase + reg);
+ writel(*shadow, pctl->membase + reg);
raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
@@ -1486,7 +1509,7 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
struct pinctrl_pin_desc *pins;
struct sunxi_pinctrl *pctl;
struct pinmux_ops *pmxops;
- int i, ret, last_pin, pin_idx;
+ int i, ret, last_pin, pin_idx, nbanks;
struct clk *clk;
pctl = devm_kzalloc(&pdev->dev, sizeof(*pctl), GFP_KERNEL);
@@ -1520,6 +1543,37 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
if (!pctl->irq_array)
return -ENOMEM;
+ /*
+ * The bus clock has to be enabled before the pinctrl device
+ * registers, as the pin hogs claimed from there access registers.
+ */
+ ret = of_clk_get_parent_count(node);
+ clk = devm_clk_get_enabled(&pdev->dev, ret == 1 ? NULL : "apb");
+ if (IS_ERR(clk))
+ return PTR_ERR(clk);
+
+ /*
+ * Seed the output latch shadow from the hardware so pins the
+ * bootloader left in output mode keep their state; see
+ * sunxi_pinctrl_gpio_set() for why a shadow is needed. This must
+ * happen before the pinctrl device registers, as pin hogs can mux
+ * pins to gpio_out and thereby update the shadow.
+ */
+ last_pin = pctl->desc->pins[pctl->desc->npins - 1].pin.number;
+ nbanks = DIV_ROUND_UP(last_pin + 1 - pctl->desc->pin_base,
+ PINS_PER_BANK);
+ pctl->dat_shadow = devm_kcalloc(&pdev->dev, nbanks,
+ sizeof(*pctl->dat_shadow), GFP_KERNEL);
+ if (!pctl->dat_shadow)
+ return -ENOMEM;
+
+ for (i = 0; i < nbanks; i++) {
+ u32 reg, shift, mask;
+
+ sunxi_data_reg(pctl, i * PINS_PER_BANK, ®, &shift, &mask);
+ pctl->dat_shadow[i] = readl(pctl->membase + reg);
+ }
+
ret = sunxi_pinctrl_build_state(pdev);
if (ret) {
dev_err(&pdev->dev, "dt probe failed: %d\n", ret);
@@ -1574,7 +1628,6 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
if (!pctl->chip)
return -ENOMEM;
- last_pin = pctl->desc->pins[pctl->desc->npins - 1].pin.number;
pctl->chip->owner = THIS_MODULE;
pctl->chip->request = gpiochip_generic_request;
pctl->chip->free = gpiochip_generic_free;
@@ -1607,13 +1660,6 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
goto gpiochip_error;
}
- ret = of_clk_get_parent_count(node);
- clk = devm_clk_get_enabled(&pdev->dev, ret == 1 ? NULL : "apb");
- if (IS_ERR(clk)) {
- ret = PTR_ERR(clk);
- goto gpiochip_error;
- }
-
pctl->irq = devm_kcalloc(&pdev->dev,
pctl->desc->irq_banks,
sizeof(*pctl->irq),
diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.h b/drivers/pinctrl/sunxi/pinctrl-sunxi.h
index a87a2f944d60..244b0039480a 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sunxi.h
+++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.h
@@ -80,6 +80,7 @@
#define IO_BIAS_MASK GENMASK(3, 0)
#define SUN4I_FUNC_INPUT 0
+#define SUN4I_FUNC_OUTPUT 1
#define SUN4I_FUNC_IRQ 6
#define PINCTRL_SUN5I_A10S BIT(1)
@@ -173,6 +174,12 @@ struct sunxi_pinctrl {
int *irq;
unsigned *irq_array;
raw_spinlock_t lock;
+ /*
+ * Output latch shadow, one word per bank. Seeded lockless at
+ * probe before the pinctrl device registers, protected by @lock
+ * afterwards.
+ */
+ u32 *dat_shadow;
struct pinctrl_dev *pctl_dev;
unsigned long variant;
u32 bank_mem_size;
diff --git a/drivers/platform/x86/amd/pmc/Makefile b/drivers/platform/x86/amd/pmc/Makefile
index f1d9ab19d24c..255d94ddf999 100644
--- a/drivers/platform/x86/amd/pmc/Makefile
+++ b/drivers/platform/x86/amd/pmc/Makefile
@@ -4,6 +4,6 @@
# AMD Power Management Controller Driver
#
-amd-pmc-objs := pmc.o pmc-quirks.o
+amd-pmc-objs := pmc.o pmc-quirks.o mp1_stb.o
obj-$(CONFIG_AMD_PMC) += amd-pmc.o
amd-pmc-$(CONFIG_AMD_MP2_STB) += mp2_stb.o
diff --git a/drivers/platform/x86/amd/pmc/mp1_stb.c b/drivers/platform/x86/amd/pmc/mp1_stb.c
new file mode 100644
index 000000000000..d662b9b91da7
--- /dev/null
+++ b/drivers/platform/x86/amd/pmc/mp1_stb.c
@@ -0,0 +1,316 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * AMD MP1 Smart Trace Buffer (STB) Layer
+ *
+ * Copyright (c) 2024, Advanced Micro Devices, Inc.
+ * All Rights Reserved.
+ *
+ * Authors: Shyam Sundar S K <Shyam-sundar.S-k@amd.com>
+ * Sanket Goswami <Sanket.Goswami@amd.com>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <asm/amd_nb.h>
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+#include <linux/uaccess.h>
+
+#include "pmc.h"
+
+/* STB Spill to DRAM Parameters */
+#define S2D_TELEMETRY_DRAMBYTES_MAX 0x1000000
+#define S2D_TELEMETRY_BYTES_MAX 0x100000U
+#define S2D_RSVD_RAM_SPACE 0x100000
+
+/* STB Registers */
+#define AMD_PMC_STB_PMI_0 0x03E30600
+#define AMD_PMC_STB_DUMMY_PC 0xC6000007
+
+/* STB Spill to DRAM Message Definition */
+#define STB_FORCE_FLUSH_DATA 0xCF
+#define FIFO_SIZE 4096
+
+static bool enable_stb;
+module_param(enable_stb, bool, 0644);
+MODULE_PARM_DESC(enable_stb, "Enable the STB debug mechanism");
+
+static bool dump_custom_stb;
+module_param(dump_custom_stb, bool, 0644);
+MODULE_PARM_DESC(dump_custom_stb, "Enable to dump full STB buffer");
+
+enum s2d_arg {
+ S2D_TELEMETRY_SIZE = 0x01,
+ S2D_PHYS_ADDR_LOW,
+ S2D_PHYS_ADDR_HIGH,
+ S2D_NUM_SAMPLES,
+ S2D_DRAM_SIZE,
+};
+
+struct amd_pmc_stb_v2_data {
+ size_t size;
+ u8 data[] __counted_by(size);
+};
+
+int amd_pmc_write_stb(struct amd_pmc_dev *dev, u32 data)
+{
+ int err;
+
+ err = amd_smn_write(0, AMD_PMC_STB_PMI_0, data);
+ if (err) {
+ dev_err(dev->dev, "failed to write data in stb: 0x%X\n", AMD_PMC_STB_PMI_0);
+ return pcibios_err_to_errno(err);
+ }
+
+ return 0;
+}
+
+int amd_pmc_read_stb(struct amd_pmc_dev *dev, u32 *buf)
+{
+ int i, err;
+
+ for (i = 0; i < FIFO_SIZE; i++) {
+ err = amd_smn_read(0, AMD_PMC_STB_PMI_0, buf++);
+ if (err) {
+ dev_err(dev->dev, "error reading data from stb: 0x%X\n", AMD_PMC_STB_PMI_0);
+ return pcibios_err_to_errno(err);
+ }
+ }
+
+ return 0;
+}
+
+static int amd_pmc_stb_debugfs_open(struct inode *inode, struct file *filp)
+{
+ struct amd_pmc_dev *dev = filp->f_inode->i_private;
+ u32 size = FIFO_SIZE * sizeof(u32);
+ u32 *buf;
+ int rc;
+
+ buf = kzalloc(size, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ rc = amd_pmc_read_stb(dev, buf);
+ if (rc) {
+ kfree(buf);
+ return rc;
+ }
+
+ filp->private_data = buf;
+ return rc;
+}
+
+static ssize_t amd_pmc_stb_debugfs_read(struct file *filp, char __user *buf, size_t size,
+ loff_t *pos)
+{
+ if (!filp->private_data)
+ return -EINVAL;
+
+ return simple_read_from_buffer(buf, size, pos, filp->private_data,
+ FIFO_SIZE * sizeof(u32));
+}
+
+static int amd_pmc_stb_debugfs_release(struct inode *inode, struct file *filp)
+{
+ kfree(filp->private_data);
+ return 0;
+}
+
+static const struct file_operations amd_pmc_stb_debugfs_fops = {
+ .owner = THIS_MODULE,
+ .open = amd_pmc_stb_debugfs_open,
+ .read = amd_pmc_stb_debugfs_read,
+ .release = amd_pmc_stb_debugfs_release,
+};
+
+/* Enhanced STB Firmware Reporting Mechanism */
+static int amd_pmc_stb_handle_efr(struct file *filp)
+{
+ struct amd_pmc_dev *dev = filp->f_inode->i_private;
+ struct amd_pmc_stb_v2_data *stb_data_arr;
+ u32 fsize;
+
+ fsize = dev->dram_size - S2D_RSVD_RAM_SPACE;
+ stb_data_arr = kmalloc(struct_size(stb_data_arr, data, fsize), GFP_KERNEL);
+ if (!stb_data_arr)
+ return -ENOMEM;
+
+ stb_data_arr->size = fsize;
+ memcpy_fromio(stb_data_arr->data, dev->stb_virt_addr, fsize);
+ filp->private_data = stb_data_arr;
+
+ return 0;
+}
+
+static int amd_pmc_stb_debugfs_open_v2(struct inode *inode, struct file *filp)
+{
+ struct amd_pmc_dev *dev = filp->f_inode->i_private;
+ u32 fsize, num_samples, val, stb_rdptr_offset = 0;
+ struct amd_pmc_stb_v2_data *stb_data_arr;
+ int ret;
+
+ /* Write dummy postcode while reading the STB buffer */
+ ret = amd_pmc_write_stb(dev, AMD_PMC_STB_DUMMY_PC);
+ if (ret)
+ dev_err(dev->dev, "error writing to STB: %d\n", ret);
+
+ /* Spill to DRAM num_samples uses separate SMU message port */
+ dev->msg_port = MSG_PORT_S2D;
+
+ ret = amd_pmc_send_cmd(dev, 0, &val, STB_FORCE_FLUSH_DATA, 1);
+ if (ret)
+ dev_dbg_once(dev->dev, "S2D force flush not supported: %d\n", ret);
+
+ /*
+ * We have a custom stb size and the PMFW is supposed to give
+ * the enhanced dram size. Note that we land here only for the
+ * platforms that support enhanced dram size reporting.
+ */
+ if (dump_custom_stb)
+ return amd_pmc_stb_handle_efr(filp);
+
+ /* Get the num_samples to calculate the last push location */
+ ret = amd_pmc_send_cmd(dev, S2D_NUM_SAMPLES, &num_samples, dev->stb_arg.s2d_msg_id, true);
+ /* Clear msg_port for other SMU operation */
+ dev->msg_port = MSG_PORT_PMC;
+ if (ret) {
+ dev_err(dev->dev, "error: S2D_NUM_SAMPLES not supported : %d\n", ret);
+ return ret;
+ }
+
+ fsize = min(num_samples, S2D_TELEMETRY_BYTES_MAX);
+ stb_data_arr = kmalloc(struct_size(stb_data_arr, data, fsize), GFP_KERNEL);
+ if (!stb_data_arr)
+ return -ENOMEM;
+
+ stb_data_arr->size = fsize;
+
+ /*
+ * Start capturing data from the last push location.
+ * This is for general cases, where the stb limits
+ * are meant for standard usage.
+ */
+ if (num_samples > S2D_TELEMETRY_BYTES_MAX) {
+ /* First read oldest data starting 1 behind last write till end of ringbuffer */
+ stb_rdptr_offset = num_samples % S2D_TELEMETRY_BYTES_MAX;
+ fsize = S2D_TELEMETRY_BYTES_MAX - stb_rdptr_offset;
+
+ memcpy_fromio(stb_data_arr->data, dev->stb_virt_addr + stb_rdptr_offset, fsize);
+ /* Second copy the newer samples from offset 0 - last write */
+ memcpy_fromio(stb_data_arr->data + fsize, dev->stb_virt_addr, stb_rdptr_offset);
+ } else {
+ memcpy_fromio(stb_data_arr->data, dev->stb_virt_addr, fsize);
+ }
+
+ filp->private_data = stb_data_arr;
+
+ return 0;
+}
+
+static ssize_t amd_pmc_stb_debugfs_read_v2(struct file *filp, char __user *buf, size_t size,
+ loff_t *pos)
+{
+ struct amd_pmc_stb_v2_data *data = filp->private_data;
+
+ return simple_read_from_buffer(buf, size, pos, data->data, data->size);
+}
+
+static int amd_pmc_stb_debugfs_release_v2(struct inode *inode, struct file *filp)
+{
+ kfree(filp->private_data);
+ return 0;
+}
+
+static const struct file_operations amd_pmc_stb_debugfs_fops_v2 = {
+ .owner = THIS_MODULE,
+ .open = amd_pmc_stb_debugfs_open_v2,
+ .read = amd_pmc_stb_debugfs_read_v2,
+ .release = amd_pmc_stb_debugfs_release_v2,
+};
+
+static bool amd_pmc_is_stb_supported(struct amd_pmc_dev *dev)
+{
+ switch (dev->cpu_id) {
+ case AMD_CPU_ID_YC:
+ case AMD_CPU_ID_CB:
+ dev->stb_arg.s2d_msg_id = 0xBE;
+ return true;
+ case AMD_CPU_ID_PS:
+ dev->stb_arg.s2d_msg_id = 0x85;
+ return true;
+ case PCI_DEVICE_ID_AMD_1AH_M20H_ROOT:
+ case PCI_DEVICE_ID_AMD_1AH_M60H_ROOT:
+ dev->stb_arg.s2d_msg_id = 0xDE;
+ return true;
+ default:
+ return false;
+ }
+}
+
+int amd_stb_s2d_init(struct amd_pmc_dev *dev)
+{
+ u32 phys_addr_low, phys_addr_hi;
+ u64 stb_phys_addr;
+ u32 size = 0;
+ int ret = 0;
+
+ if (!enable_stb)
+ return 0;
+
+ if (amd_pmc_is_stb_supported(dev)) {
+ debugfs_create_file("stb_read", 0644, dev->dbgfs_dir, dev,
+ &amd_pmc_stb_debugfs_fops_v2);
+ } else {
+ debugfs_create_file("stb_read", 0644, dev->dbgfs_dir, dev,
+ &amd_pmc_stb_debugfs_fops);
+ return 0;
+ }
+
+ /* Spill to DRAM feature uses separate SMU message port */
+ dev->msg_port = MSG_PORT_S2D;
+
+ ret = amd_pmc_send_cmd(dev, S2D_TELEMETRY_SIZE, &size, dev->stb_arg.s2d_msg_id, true);
+ if (ret)
+ goto out;
+ if (size != S2D_TELEMETRY_BYTES_MAX) {
+ ret = -EIO;
+ goto out;
+ }
+
+ /* Get DRAM size */
+ ret = amd_pmc_send_cmd(dev, S2D_DRAM_SIZE, &dev->dram_size, dev->stb_arg.s2d_msg_id, true);
+ if (ret || !dev->dram_size)
+ dev->dram_size = S2D_TELEMETRY_DRAMBYTES_MAX;
+
+ /* Get STB DRAM address */
+ ret = amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_LOW, &phys_addr_low,
+ dev->stb_arg.s2d_msg_id, true);
+ if (ret)
+ goto out;
+ ret = amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_HIGH, &phys_addr_hi,
+ dev->stb_arg.s2d_msg_id, true);
+ if (ret)
+ goto out;
+
+ if (!phys_addr_hi && !phys_addr_low) {
+ dev_err(dev->dev, "STB is not enabled on the system; disable enable_stb or contact system vendor\n");
+ dev->msg_port = MSG_PORT_PMC;
+ return -EINVAL;
+ }
+
+ stb_phys_addr = ((u64)phys_addr_hi << 32 | phys_addr_low);
+
+ dev->stb_virt_addr = devm_ioremap(dev->dev, stb_phys_addr, dev->dram_size);
+ if (!dev->stb_virt_addr) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ret = 0;
+
+out:
+ /* Restore the default message port for subsequent SMU operations */
+ dev->msg_port = MSG_PORT_PMC;
+ return ret;
+}
diff --git a/drivers/platform/x86/amd/pmc/pmc.c b/drivers/platform/x86/amd/pmc/pmc.c
index 343f75eeb7f2..44e6bd91af4a 100644
--- a/drivers/platform/x86/amd/pmc/pmc.c
+++ b/drivers/platform/x86/amd/pmc/pmc.c
@@ -41,25 +41,15 @@
#define AMD_PMC_SCRATCH_REG_1AH 0xF14
/* STB Registers */
-#define AMD_PMC_STB_PMI_0 0x03E30600
#define AMD_PMC_STB_S2IDLE_PREPARE 0xC6000001
#define AMD_PMC_STB_S2IDLE_RESTORE 0xC6000002
#define AMD_PMC_STB_S2IDLE_CHECK 0xC6000003
-#define AMD_PMC_STB_DUMMY_PC 0xC6000007
/* STB S2D(Spill to DRAM) has different message port offset */
#define AMD_S2D_REGISTER_MESSAGE 0xA20
#define AMD_S2D_REGISTER_RESPONSE 0xA80
#define AMD_S2D_REGISTER_ARGUMENT 0xA88
-/* STB Spill to DRAM Parameters */
-#define S2D_TELEMETRY_BYTES_MAX 0x100000U
-#define S2D_RSVD_RAM_SPACE 0x100000
-#define S2D_TELEMETRY_DRAMBYTES_MAX 0x1000000
-
-/* STB Spill to DRAM Message Definition */
-#define STB_FORCE_FLUSH_DATA 0xCF
-
/* Base address of SMU for mapping physical address to virtual address */
#define AMD_PMC_MAPPING_SIZE 0x01000
#define AMD_PMC_BASE_ADDR_OFFSET 0x10000
@@ -98,7 +88,6 @@
#define DELAY_MIN_US 2000
#define DELAY_MAX_US 3000
-#define FIFO_SIZE 4096
enum amd_pmc_def {
MSG_TEST = 0x01,
@@ -106,19 +95,6 @@ enum amd_pmc_def {
MSG_OS_HINT_RN,
};
-enum s2d_arg {
- S2D_TELEMETRY_SIZE = 0x01,
- S2D_PHYS_ADDR_LOW,
- S2D_PHYS_ADDR_HIGH,
- S2D_NUM_SAMPLES,
- S2D_DRAM_SIZE,
-};
-
-struct amd_pmc_stb_v2_data {
- size_t size;
- u8 data[] __counted_by(size);
-};
-
struct amd_pmc_bit_map {
const char *name;
u32 bit_mask;
@@ -150,27 +126,16 @@ static const struct amd_pmc_bit_map soc15_ip_blk[] = {
{}
};
-static bool enable_stb;
-module_param(enable_stb, bool, 0644);
-MODULE_PARM_DESC(enable_stb, "Enable the STB debug mechanism");
-
static bool disable_workarounds;
module_param(disable_workarounds, bool, 0644);
MODULE_PARM_DESC(disable_workarounds, "Disable workarounds for platform bugs");
-static bool dump_custom_stb;
-module_param(dump_custom_stb, bool, 0644);
-MODULE_PARM_DESC(dump_custom_stb, "Enable to dump full STB buffer");
-
static int delay_suspend = -1;
module_param(delay_suspend, int, 0644);
MODULE_PARM_DESC(delay_suspend,
"Delays s2idle by 2.5 seconds to work around buggy ECs, often causing keyboard issues after suspend. 0: don't delay, 1: do delay, -1 (default): let amd_pmc decide. If you need this please report this to: platform-driver-x86@vger.kernel.org");
static struct amd_pmc_dev pmc;
-static int amd_pmc_send_cmd(struct amd_pmc_dev *dev, u32 arg, u32 *data, u8 msg, bool ret);
-static int amd_pmc_read_stb(struct amd_pmc_dev *dev, u32 *buf);
-static int amd_pmc_write_stb(struct amd_pmc_dev *dev, u32 data);
static inline u32 amd_pmc_reg_read(struct amd_pmc_dev *dev, int reg_offset)
{
@@ -199,155 +164,6 @@ struct smu_metrics {
u64 timecondition_notmet_totaltime[32];
} __packed;
-static int amd_pmc_stb_debugfs_open(struct inode *inode, struct file *filp)
-{
- struct amd_pmc_dev *dev = filp->f_inode->i_private;
- u32 size = FIFO_SIZE * sizeof(u32);
- u32 *buf;
- int rc;
-
- buf = kzalloc(size, GFP_KERNEL);
- if (!buf)
- return -ENOMEM;
-
- rc = amd_pmc_read_stb(dev, buf);
- if (rc) {
- kfree(buf);
- return rc;
- }
-
- filp->private_data = buf;
- return rc;
-}
-
-static ssize_t amd_pmc_stb_debugfs_read(struct file *filp, char __user *buf, size_t size,
- loff_t *pos)
-{
- if (!filp->private_data)
- return -EINVAL;
-
- return simple_read_from_buffer(buf, size, pos, filp->private_data,
- FIFO_SIZE * sizeof(u32));
-}
-
-static int amd_pmc_stb_debugfs_release(struct inode *inode, struct file *filp)
-{
- kfree(filp->private_data);
- return 0;
-}
-
-static const struct file_operations amd_pmc_stb_debugfs_fops = {
- .owner = THIS_MODULE,
- .open = amd_pmc_stb_debugfs_open,
- .read = amd_pmc_stb_debugfs_read,
- .release = amd_pmc_stb_debugfs_release,
-};
-
-/* Enhanced STB Firmware Reporting Mechanism */
-static int amd_pmc_stb_handle_efr(struct file *filp)
-{
- struct amd_pmc_dev *dev = filp->f_inode->i_private;
- struct amd_pmc_stb_v2_data *stb_data_arr;
- u32 fsize;
-
- fsize = dev->dram_size - S2D_RSVD_RAM_SPACE;
- stb_data_arr = kmalloc(struct_size(stb_data_arr, data, fsize), GFP_KERNEL);
- if (!stb_data_arr)
- return -ENOMEM;
-
- stb_data_arr->size = fsize;
- memcpy_fromio(stb_data_arr->data, dev->stb_virt_addr, fsize);
- filp->private_data = stb_data_arr;
-
- return 0;
-}
-
-static int amd_pmc_stb_debugfs_open_v2(struct inode *inode, struct file *filp)
-{
- struct amd_pmc_dev *dev = filp->f_inode->i_private;
- u32 fsize, num_samples, val, stb_rdptr_offset = 0;
- struct amd_pmc_stb_v2_data *stb_data_arr;
- int ret;
-
- /* Write dummy postcode while reading the STB buffer */
- ret = amd_pmc_write_stb(dev, AMD_PMC_STB_DUMMY_PC);
- if (ret)
- dev_err(dev->dev, "error writing to STB: %d\n", ret);
-
- /* Spill to DRAM num_samples uses separate SMU message port */
- dev->msg_port = 1;
-
- ret = amd_pmc_send_cmd(dev, 0, &val, STB_FORCE_FLUSH_DATA, 1);
- if (ret)
- dev_dbg_once(dev->dev, "S2D force flush not supported: %d\n", ret);
-
- /*
- * We have a custom stb size and the PMFW is supposed to give
- * the enhanced dram size. Note that we land here only for the
- * platforms that support enhanced dram size reporting.
- */
- if (dump_custom_stb)
- return amd_pmc_stb_handle_efr(filp);
-
- /* Get the num_samples to calculate the last push location */
- ret = amd_pmc_send_cmd(dev, S2D_NUM_SAMPLES, &num_samples, dev->s2d_msg_id, true);
- /* Clear msg_port for other SMU operation */
- dev->msg_port = 0;
- if (ret) {
- dev_err(dev->dev, "error: S2D_NUM_SAMPLES not supported : %d\n", ret);
- return ret;
- }
-
- fsize = min(num_samples, S2D_TELEMETRY_BYTES_MAX);
- stb_data_arr = kmalloc(struct_size(stb_data_arr, data, fsize), GFP_KERNEL);
- if (!stb_data_arr)
- return -ENOMEM;
-
- stb_data_arr->size = fsize;
-
- /*
- * Start capturing data from the last push location.
- * This is for general cases, where the stb limits
- * are meant for standard usage.
- */
- if (num_samples > S2D_TELEMETRY_BYTES_MAX) {
- /* First read oldest data starting 1 behind last write till end of ringbuffer */
- stb_rdptr_offset = num_samples % S2D_TELEMETRY_BYTES_MAX;
- fsize = S2D_TELEMETRY_BYTES_MAX - stb_rdptr_offset;
-
- memcpy_fromio(stb_data_arr->data, dev->stb_virt_addr + stb_rdptr_offset, fsize);
- /* Second copy the newer samples from offset 0 - last write */
- memcpy_fromio(stb_data_arr->data + fsize, dev->stb_virt_addr, stb_rdptr_offset);
- } else {
- memcpy_fromio(stb_data_arr->data, dev->stb_virt_addr, fsize);
- }
-
- filp->private_data = stb_data_arr;
-
- return 0;
-}
-
-static ssize_t amd_pmc_stb_debugfs_read_v2(struct file *filp, char __user *buf, size_t size,
- loff_t *pos)
-{
- struct amd_pmc_stb_v2_data *data = filp->private_data;
-
- return simple_read_from_buffer(buf, size, pos, data->data, data->size);
-}
-
-static int amd_pmc_stb_debugfs_release_v2(struct inode *inode, struct file *filp)
-{
- kfree(filp->private_data);
- return 0;
-}
-
-static const struct file_operations amd_pmc_stb_debugfs_fops_v2 = {
- .owner = THIS_MODULE,
- .open = amd_pmc_stb_debugfs_open_v2,
- .read = amd_pmc_stb_debugfs_read_v2,
- .release = amd_pmc_stb_debugfs_release_v2,
-};
-
static void amd_pmc_get_ip_info(struct amd_pmc_dev *dev)
{
switch (dev->cpu_id) {
@@ -357,18 +173,15 @@ static void amd_pmc_get_ip_info(struct amd_pmc_dev *dev)
case AMD_CPU_ID_YC:
case AMD_CPU_ID_CB:
dev->num_ips = 12;
- dev->s2d_msg_id = 0xBE;
dev->smu_msg = 0x538;
break;
case AMD_CPU_ID_PS:
dev->num_ips = 21;
- dev->s2d_msg_id = 0x85;
dev->smu_msg = 0x538;
break;
case PCI_DEVICE_ID_AMD_1AH_M20H_ROOT:
case PCI_DEVICE_ID_AMD_1AH_M60H_ROOT:
dev->num_ips = 22;
- dev->s2d_msg_id = 0xDE;
dev->smu_msg = 0x938;
break;
}
@@ -634,20 +447,6 @@ static void amd_pmc_dbgfs_unregister(struct amd_pmc_dev *dev)
debugfs_remove_recursive(dev->dbgfs_dir);
}
-static bool amd_pmc_is_stb_supported(struct amd_pmc_dev *dev)
-{
- switch (dev->cpu_id) {
- case AMD_CPU_ID_YC:
- case AMD_CPU_ID_CB:
- case AMD_CPU_ID_PS:
- case PCI_DEVICE_ID_AMD_1AH_M20H_ROOT:
- case PCI_DEVICE_ID_AMD_1AH_M60H_ROOT:
- return true;
- default:
- return false;
- }
-}
-
static void amd_pmc_dbgfs_register(struct amd_pmc_dev *dev)
{
dev->dbgfs_dir = debugfs_create_dir("amd_pmc", NULL);
@@ -657,14 +456,17 @@ static void amd_pmc_dbgfs_register(struct amd_pmc_dev *dev)
&s0ix_stats_fops);
debugfs_create_file("amd_pmc_idlemask", 0644, dev->dbgfs_dir, dev,
&amd_pmc_idlemask_fops);
- /* Enable STB only when the module_param is set */
- if (enable_stb) {
- if (amd_pmc_is_stb_supported(dev))
- debugfs_create_file("stb_read", 0644, dev->dbgfs_dir, dev,
- &amd_pmc_stb_debugfs_fops_v2);
- else
- debugfs_create_file("stb_read", 0644, dev->dbgfs_dir, dev,
- &amd_pmc_stb_debugfs_fops);
+}
+
+static char *amd_pmc_get_msg_port(struct amd_pmc_dev *dev)
+{
+ switch (dev->msg_port) {
+ case MSG_PORT_PMC:
+ return "PMC";
+ case MSG_PORT_S2D:
+ return "S2D";
+ default:
+ return "Invalid message port";
}
}
@@ -672,7 +474,7 @@ static void amd_pmc_dump_registers(struct amd_pmc_dev *dev)
{
u32 value, message, argument, response;
- if (dev->msg_port) {
+ if (dev->msg_port == MSG_PORT_S2D) {
message = AMD_S2D_REGISTER_MESSAGE;
argument = AMD_S2D_REGISTER_ARGUMENT;
response = AMD_S2D_REGISTER_RESPONSE;
@@ -683,23 +485,23 @@ static void amd_pmc_dump_registers(struct amd_pmc_dev *dev)
}
value = amd_pmc_reg_read(dev, response);
- dev_dbg(dev->dev, "AMD_%s_REGISTER_RESPONSE:%x\n", dev->msg_port ? "S2D" : "PMC", value);
+ dev_dbg(dev->dev, "AMD_%s_REGISTER_RESPONSE:%x\n", amd_pmc_get_msg_port(dev), value);
value = amd_pmc_reg_read(dev, argument);
- dev_dbg(dev->dev, "AMD_%s_REGISTER_ARGUMENT:%x\n", dev->msg_port ? "S2D" : "PMC", value);
+ dev_dbg(dev->dev, "AMD_%s_REGISTER_ARGUMENT:%x\n", amd_pmc_get_msg_port(dev), value);
value = amd_pmc_reg_read(dev, message);
- dev_dbg(dev->dev, "AMD_%s_REGISTER_MESSAGE:%x\n", dev->msg_port ? "S2D" : "PMC", value);
+ dev_dbg(dev->dev, "AMD_%s_REGISTER_MESSAGE:%x\n", amd_pmc_get_msg_port(dev), value);
}
-static int amd_pmc_send_cmd(struct amd_pmc_dev *dev, u32 arg, u32 *data, u8 msg, bool ret)
+int amd_pmc_send_cmd(struct amd_pmc_dev *dev, u32 arg, u32 *data, u8 msg, bool ret)
{
int rc;
u32 val, message, argument, response;
mutex_lock(&dev->lock);
- if (dev->msg_port) {
+ if (dev->msg_port == MSG_PORT_S2D) {
message = AMD_S2D_REGISTER_MESSAGE;
argument = AMD_S2D_REGISTER_ARGUMENT;
response = AMD_S2D_REGISTER_RESPONSE;
@@ -1062,74 +864,6 @@ static const struct pci_device_id pmc_pci_ids[] = {
{ }
};
-static int amd_pmc_s2d_init(struct amd_pmc_dev *dev)
-{
- u32 phys_addr_low, phys_addr_hi;
- u64 stb_phys_addr;
- u32 size = 0;
- int ret;
-
- /* Spill to DRAM feature uses separate SMU message port */
- dev->msg_port = 1;
-
- amd_pmc_send_cmd(dev, S2D_TELEMETRY_SIZE, &size, dev->s2d_msg_id, true);
- if (size != S2D_TELEMETRY_BYTES_MAX)
- return -EIO;
-
- /* Get DRAM size */
- ret = amd_pmc_send_cmd(dev, S2D_DRAM_SIZE, &dev->dram_size, dev->s2d_msg_id, true);
- if (ret || !dev->dram_size)
- dev->dram_size = S2D_TELEMETRY_DRAMBYTES_MAX;
-
- /* Get STB DRAM address */
- amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_LOW, &phys_addr_low, dev->s2d_msg_id, true);
- amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_HIGH, &phys_addr_hi, dev->s2d_msg_id, true);
-
- if (!phys_addr_hi && !phys_addr_low) {
- dev_err(dev->dev, "STB is not enabled on the system; disable enable_stb or contact system vendor\n");
- return -EINVAL;
- }
-
- stb_phys_addr = ((u64)phys_addr_hi << 32 | phys_addr_low);
-
- /* Clear msg_port for other SMU operation */
- dev->msg_port = 0;
-
- dev->stb_virt_addr = devm_ioremap(dev->dev, stb_phys_addr, dev->dram_size);
- if (!dev->stb_virt_addr)
- return -ENOMEM;
-
- return 0;
-}
-
-static int amd_pmc_write_stb(struct amd_pmc_dev *dev, u32 data)
-{
- int err;
-
- err = amd_smn_write(0, AMD_PMC_STB_PMI_0, data);
- if (err) {
- dev_err(dev->dev, "failed to write data in stb: 0x%X\n", AMD_PMC_STB_PMI_0);
- return pcibios_err_to_errno(err);
- }
-
- return 0;
-}
-
-static int amd_pmc_read_stb(struct amd_pmc_dev *dev, u32 *buf)
-{
- int i, err;
-
- for (i = 0; i < FIFO_SIZE; i++) {
- err = amd_smn_read(0, AMD_PMC_STB_PMI_0, buf++);
- if (err) {
- dev_err(dev->dev, "error reading data from stb: 0x%X\n", AMD_PMC_STB_PMI_0);
- return pcibios_err_to_errno(err);
- }
- }
-
- return 0;
-}
-
static int amd_pmc_probe(struct platform_device *pdev)
{
struct amd_pmc_dev *dev = &pmc;
@@ -1187,12 +921,6 @@ static int amd_pmc_probe(struct platform_device *pdev)
/* Get num of IP blocks within the SoC */
amd_pmc_get_ip_info(dev);
- if (enable_stb && amd_pmc_is_stb_supported(dev)) {
- err = amd_pmc_s2d_init(dev);
- if (err)
- goto err_pci_dev_put;
- }
-
platform_set_drvdata(pdev, dev);
if (IS_ENABLED(CONFIG_SUSPEND)) {
err = acpi_register_lps0_dev(&amd_pmc_s2idle_dev_ops);
@@ -1203,11 +931,19 @@ static int amd_pmc_probe(struct platform_device *pdev)
}
amd_pmc_dbgfs_register(dev);
+ err = amd_stb_s2d_init(dev);
+ if (err)
+ goto err_dbgfs_unregister;
+
if (IS_ENABLED(CONFIG_AMD_MP2_STB))
amd_mp2_stb_init(dev);
pm_report_max_hw_sleep(U64_MAX);
return 0;
+err_dbgfs_unregister:
+ amd_pmc_dbgfs_unregister(dev);
+ if (IS_ENABLED(CONFIG_SUSPEND))
+ acpi_unregister_lps0_dev(&amd_pmc_s2idle_dev_ops);
err_pci_dev_put:
pci_dev_put(rdev);
return err;
diff --git a/drivers/platform/x86/amd/pmc/pmc.h b/drivers/platform/x86/amd/pmc/pmc.h
index 89f8891d3e38..b18bc46e36e6 100644
--- a/drivers/platform/x86/amd/pmc/pmc.h
+++ b/drivers/platform/x86/amd/pmc/pmc.h
@@ -14,6 +14,11 @@
#include <linux/types.h>
#include <linux/mutex.h>
+enum s2d_msg_port {
+ MSG_PORT_PMC,
+ MSG_PORT_S2D,
+};
+
struct amd_mp2_dev {
void __iomem *mmio;
void __iomem *vslbase;
@@ -25,24 +30,27 @@ struct amd_mp2_dev {
bool is_stb_data;
};
+struct stb_arg {
+ u32 s2d_msg_id;
+};
+
struct amd_pmc_dev {
void __iomem *regbase;
void __iomem *smu_virt_addr;
void __iomem *stb_virt_addr;
void __iomem *fch_virt_addr;
- bool msg_port;
u32 base_addr;
u32 cpu_id;
u32 active_ips;
u32 dram_size;
u32 num_ips;
- u32 s2d_msg_id;
u32 smu_msg;
/* SMU version information */
u8 smu_program;
u8 major;
u8 minor;
u8 rev;
+ u8 msg_port;
struct device *dev;
struct pci_dev *rdev;
struct mutex lock; /* generic mutex lock */
@@ -51,6 +59,7 @@ struct amd_pmc_dev {
bool disable_8042_wakeup;
bool is_first_check_after_suspend;
struct amd_mp2_dev *mp2;
+ struct stb_arg stb_arg;
};
void amd_pmc_process_restore_quirks(struct amd_pmc_dev *dev);
@@ -73,4 +82,9 @@ void amd_mp2_stb_deinit(struct amd_pmc_dev *dev);
#define PCI_DEVICE_ID_AMD_1AH_M60H_ROOT 0x1122
#define PCI_DEVICE_ID_AMD_MP2_STB 0x172c
+int amd_stb_s2d_init(struct amd_pmc_dev *dev);
+int amd_pmc_read_stb(struct amd_pmc_dev *dev, u32 *buf);
+int amd_pmc_write_stb(struct amd_pmc_dev *dev, u32 data);
+int amd_pmc_send_cmd(struct amd_pmc_dev *dev, u32 arg, u32 *data, u8 msg, bool ret);
+
#endif /* PMC_H */
diff --git a/drivers/platform/x86/intel/int1092/intel_sar.c b/drivers/platform/x86/intel/int1092/intel_sar.c
index 6246c066ade2..1cfb268f6b71 100644
--- a/drivers/platform/x86/intel/int1092/intel_sar.c
+++ b/drivers/platform/x86/intel/int1092/intel_sar.c
@@ -91,8 +91,10 @@ static acpi_status parse_package(struct wwan_sar_context *context, union acpi_ob
item->package.count <= data->total_dev_mode)
return AE_ERROR;
- data->device_mode_info = kmalloc_array(data->total_dev_mode,
- sizeof(struct wwan_device_mode_info), GFP_KERNEL);
+ data->device_mode_info = devm_kmalloc_array(&context->sar_device->dev,
+ data->total_dev_mode,
+ sizeof(*data->device_mode_info),
+ GFP_KERNEL);
if (!data->device_mode_info)
return AE_ERROR;
@@ -245,21 +247,26 @@ static void sar_get_data(int reg, struct wwan_sar_context *context)
static int sar_probe(struct platform_device *device)
{
struct wwan_sar_context *context;
+ acpi_handle handle;
int reg;
int result;
- context = kzalloc(sizeof(*context), GFP_KERNEL);
+ handle = ACPI_HANDLE(&device->dev);
+ if (!handle)
+ return -ENODEV;
+
+ context = devm_kzalloc(&device->dev, sizeof(*context), GFP_KERNEL);
if (!context)
return -ENOMEM;
context->sar_device = device;
- context->handle = ACPI_HANDLE(&device->dev);
+ context->handle = handle;
dev_set_drvdata(&device->dev, context);
result = guid_parse(SAR_DSM_UUID, &context->guid);
if (result) {
dev_err(&device->dev, "SAR UUID parse error: %d\n", result);
- goto r_free;
+ return result;
}
for (reg = 0; reg < MAX_REGULATORY; reg++)
@@ -267,43 +274,29 @@ static int sar_probe(struct platform_device *device)
if (sar_get_device_mode(device) != AE_OK) {
dev_err(&device->dev, "Failed to get device mode\n");
- result = -EIO;
- goto r_free;
+ return -EIO;
}
result = sysfs_create_group(&device->dev.kobj, &intcsar_group);
if (result) {
dev_err(&device->dev, "sysfs creation failed\n");
- goto r_free;
+ return result;
}
if (acpi_install_notify_handler(ACPI_HANDLE(&device->dev), ACPI_DEVICE_NOTIFY,
sar_notify, (void *)device) != AE_OK) {
dev_err(&device->dev, "Failed acpi_install_notify_handler\n");
- result = -EIO;
- goto r_sys;
+ sysfs_remove_group(&device->dev.kobj, &intcsar_group);
+ return -EIO;
}
return 0;
-
-r_sys:
- sysfs_remove_group(&device->dev.kobj, &intcsar_group);
-r_free:
- kfree(context);
- return result;
}
static void sar_remove(struct platform_device *device)
{
- struct wwan_sar_context *context = dev_get_drvdata(&device->dev);
- int reg;
-
acpi_remove_notify_handler(ACPI_HANDLE(&device->dev),
ACPI_DEVICE_NOTIFY, sar_notify);
sysfs_remove_group(&device->dev.kobj, &intcsar_group);
- for (reg = 0; reg < MAX_REGULATORY; reg++)
- kfree(context->config_data[reg].device_mode_info);
-
- kfree(context);
}
static struct platform_driver sar_driver = {
diff --git a/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c b/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c
index 9d376306e1e2..7f1885fb4a34 100644
--- a/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c
+++ b/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c
@@ -620,7 +620,7 @@ static long isst_if_core_power_state(void __user *argp)
return -EINVAL;
if (core_power.get_set) {
- if (power_domain_info->write_blocked)
+ if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN))
return -EPERM;
if (core_power.enable > SST_CP_MAX_ENABLE ||
@@ -679,7 +679,7 @@ static long isst_if_clos_param(void __user *argp)
return -EINVAL;
if (clos_param.get_set) {
- if (power_domain_info->write_blocked)
+ if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN))
return -EPERM;
if (!in_range(clos_param.min_freq_mhz / SST_MUL_FACTOR_FREQ, 0, SST_MAX_FREQ + 1))
@@ -790,7 +790,8 @@ static long isst_if_clos_assoc(void __user *argp)
power_domain_info = &sst_inst->power_domain_info[part][punit_id];
- if (assoc_cmds.get_set && power_domain_info->write_blocked)
+ if (assoc_cmds.get_set && (power_domain_info->write_blocked ||
+ !capable(CAP_SYS_ADMIN)))
return -EPERM;
offset = SST_CLOS_ASSOC_0_OFFSET +
@@ -968,7 +969,10 @@ static int isst_if_set_perf_level(void __user *argp)
if (!power_domain_info)
return -EINVAL;
- if (power_domain_info->write_blocked)
+ if (perf_level.level > power_domain_info->max_level)
+ return -EINVAL;
+
+ if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN))
return -EPERM;
if (!(power_domain_info->pp_header.allowed_level_mask & BIT(perf_level.level)))
@@ -1028,7 +1032,7 @@ static int isst_if_set_perf_feature(void __user *argp)
if (!power_domain_info)
return -EINVAL;
- if (power_domain_info->write_blocked)
+ if (power_domain_info->write_blocked || !capable(CAP_SYS_ADMIN))
return -EPERM;
if (perf_feature.feature & ~SST_PP_FEATURE_STATE_VALID_MASK)
diff --git a/drivers/platform/x86/think-lmi.c b/drivers/platform/x86/think-lmi.c
index 6ad2af46248b..b10d641b2567 100644
--- a/drivers/platform/x86/think-lmi.c
+++ b/drivers/platform/x86/think-lmi.c
@@ -390,7 +390,7 @@ static ssize_t is_enabled_show(struct kobject *kobj, struct kobj_attribute *attr
{
struct tlmi_pwd_setting *setting = to_tlmi_pwd_setting(kobj);
- return sysfs_emit(buf, "%d\n", setting->valid);
+ return sysfs_emit(buf, "%d\n", setting->pwd_enabled || setting->cert_installed);
}
static struct kobj_attribute auth_is_pass_set = __ATTR_RO(is_enabled);
@@ -402,14 +402,13 @@ static ssize_t current_password_store(struct kobject *kobj,
struct tlmi_pwd_setting *setting = to_tlmi_pwd_setting(kobj);
size_t pwdlen;
- pwdlen = strlen(buf);
+ /* Strip newline; setting password won't work if one is present. */
+ pwdlen = strchrnul(buf, '\n') - buf;
/* pwdlen == 0 is allowed to clear the password */
if (pwdlen && ((pwdlen < setting->minlen) || (pwdlen > setting->maxlen)))
return -EINVAL;
- strscpy(setting->password, buf, setting->maxlen);
- /* Strip out CR if one is present, setting password won't work if it is present */
- strreplace(setting->password, '\n', '\0');
+ strscpy(setting->password, buf, pwdlen + 1);
return count;
}
@@ -468,7 +467,7 @@ static ssize_t new_password_store(struct kobject *kobj,
if (ret)
goto out;
- if (tlmi_priv.pwd_admin->valid) {
+ if (tlmi_priv.pwd_admin->pwd_enabled) {
ret = tlmi_opcode_setting("WmiOpcodePasswordAdmin",
tlmi_priv.pwd_admin->password);
if (ret)
@@ -672,14 +671,18 @@ static ssize_t certificate_thumbprint_show(struct kobject *kobj, struct kobj_att
char *buf)
{
struct tlmi_pwd_setting *setting = to_tlmi_pwd_setting(kobj);
- int count = 0;
+ ssize_t count = 0;
if (!tlmi_priv.certificate_support || !setting->cert_installed)
return -EOPNOTSUPP;
- count += cert_thumbprint(buf, "Md5", count);
- count += cert_thumbprint(buf, "Sha1", count);
- count += cert_thumbprint(buf, "Sha256", count);
+ count = cert_thumbprint(buf, "Md5", count);
+ if (count < 0)
+ return count;
+ count = cert_thumbprint(buf, "Sha1", count);
+ if (count < 0)
+ return count;
+ count = cert_thumbprint(buf, "Sha256", count);
return count;
}
@@ -776,7 +779,7 @@ static ssize_t certificate_store(struct kobject *kobj,
new_cert, setting->signature);
} else {
/* This is a fresh install */
- if (!setting->valid || !setting->password[0]) {
+ if (!setting->pwd_enabled || !setting->password[0]) {
kfree(new_cert);
return -EACCES;
}
@@ -1019,7 +1022,7 @@ static ssize_t current_value_store(struct kobject *kobj,
* Workstation's require the opcode to be set before changing the
* attribute.
*/
- if (tlmi_priv.pwd_admin->valid && tlmi_priv.pwd_admin->password[0]) {
+ if (tlmi_priv.pwd_admin->pwd_enabled && tlmi_priv.pwd_admin->password[0]) {
ret = tlmi_opcode_setting("WmiOpcodePasswordAdmin",
tlmi_priv.pwd_admin->password);
if (ret)
@@ -1042,7 +1045,7 @@ static ssize_t current_value_store(struct kobject *kobj,
else
ret = tlmi_save_bios_settings("");
} else { /* old non-opcode based authentication method (deprecated) */
- if (tlmi_priv.pwd_admin->valid && tlmi_priv.pwd_admin->password[0]) {
+ if (tlmi_priv.pwd_admin->pwd_enabled && tlmi_priv.pwd_admin->password[0]) {
auth_str = kasprintf(GFP_KERNEL, "%s,%s,%s;",
tlmi_priv.pwd_admin->password,
encoding_options[tlmi_priv.pwd_admin->encoding],
@@ -1218,7 +1221,7 @@ static ssize_t save_settings_store(struct kobject *kobj, struct kobj_attribute *
if (ret)
goto out;
} else if (tlmi_priv.opcode_support) {
- if (tlmi_priv.pwd_admin->valid && tlmi_priv.pwd_admin->password[0]) {
+ if (tlmi_priv.pwd_admin->pwd_enabled && tlmi_priv.pwd_admin->password[0]) {
ret = tlmi_opcode_setting("WmiOpcodePasswordAdmin",
tlmi_priv.pwd_admin->password);
if (ret)
@@ -1226,7 +1229,7 @@ static ssize_t save_settings_store(struct kobject *kobj, struct kobj_attribute *
}
ret = tlmi_save_bios_settings("");
} else { /* old non-opcode based authentication method (deprecated) */
- if (tlmi_priv.pwd_admin->valid && tlmi_priv.pwd_admin->password[0]) {
+ if (tlmi_priv.pwd_admin->pwd_enabled && tlmi_priv.pwd_admin->password[0]) {
auth_str = kasprintf(GFP_KERNEL, "%s,%s,%s;",
tlmi_priv.pwd_admin->password,
encoding_options[tlmi_priv.pwd_admin->encoding],
@@ -1276,7 +1279,7 @@ static ssize_t debug_cmd_store(struct kobject *kobj, struct kobj_attribute *attr
if (!new_setting)
return -ENOMEM;
- if (tlmi_priv.pwd_admin->valid && tlmi_priv.pwd_admin->password[0]) {
+ if (tlmi_priv.pwd_admin->pwd_enabled && tlmi_priv.pwd_admin->password[0]) {
auth_str = kasprintf(GFP_KERNEL, "%s,%s,%s;",
tlmi_priv.pwd_admin->password,
encoding_options[tlmi_priv.pwd_admin->encoding],
@@ -1603,14 +1606,14 @@ static int tlmi_analyze(void)
goto fail_clear_attr;
if (tlmi_priv.pwdcfg.core.password_state & TLMI_PAP_PWD)
- tlmi_priv.pwd_admin->valid = true;
+ tlmi_priv.pwd_admin->pwd_enabled = true;
tlmi_priv.pwd_power = tlmi_create_auth("pop", "power-on");
if (!tlmi_priv.pwd_power)
goto fail_clear_attr;
if (tlmi_priv.pwdcfg.core.password_state & TLMI_POP_PWD)
- tlmi_priv.pwd_power->valid = true;
+ tlmi_priv.pwd_power->pwd_enabled = true;
if (tlmi_priv.opcode_support) {
tlmi_priv.pwd_system = tlmi_create_auth("smp", "system");
@@ -1618,7 +1621,7 @@ static int tlmi_analyze(void)
goto fail_clear_attr;
if (tlmi_priv.pwdcfg.core.password_state & TLMI_SMP_PWD)
- tlmi_priv.pwd_system->valid = true;
+ tlmi_priv.pwd_system->pwd_enabled = true;
tlmi_priv.pwd_hdd = tlmi_create_auth("hdd", "hdd");
if (!tlmi_priv.pwd_hdd)
@@ -1636,7 +1639,7 @@ static int tlmi_analyze(void)
/* Check if PWD is configured and set index to first drive found */
if (tlmi_priv.pwdcfg.ext.hdd_user_password ||
tlmi_priv.pwdcfg.ext.hdd_master_password) {
- tlmi_priv.pwd_hdd->valid = true;
+ tlmi_priv.pwd_hdd->pwd_enabled = true;
if (tlmi_priv.pwdcfg.ext.hdd_master_password)
tlmi_priv.pwd_hdd->index =
ffs(tlmi_priv.pwdcfg.ext.hdd_master_password) - 1;
@@ -1646,7 +1649,7 @@ static int tlmi_analyze(void)
}
if (tlmi_priv.pwdcfg.ext.nvme_user_password ||
tlmi_priv.pwdcfg.ext.nvme_master_password) {
- tlmi_priv.pwd_nvme->valid = true;
+ tlmi_priv.pwd_nvme->pwd_enabled = true;
if (tlmi_priv.pwdcfg.ext.nvme_master_password)
tlmi_priv.pwd_nvme->index =
ffs(tlmi_priv.pwdcfg.ext.nvme_master_password) - 1;
diff --git a/drivers/platform/x86/think-lmi.h b/drivers/platform/x86/think-lmi.h
index 7f9632a53736..e46938c4d3d6 100644
--- a/drivers/platform/x86/think-lmi.h
+++ b/drivers/platform/x86/think-lmi.h
@@ -65,7 +65,7 @@ struct tlmi_pwdcfg {
/* password setting details */
struct tlmi_pwd_setting {
struct kobject kobj;
- bool valid;
+ bool pwd_enabled;
char password[TLMI_PWD_BUFSIZE];
const char *pwd_type;
const char *role;
diff --git a/drivers/power/sequencing/core.c b/drivers/power/sequencing/core.c
index 7e94abd913a0..fe82c0d43bc3 100644
--- a/drivers/power/sequencing/core.c
+++ b/drivers/power/sequencing/core.c
@@ -101,6 +101,7 @@ static struct pwrseq_unit *pwrseq_unit_new(const struct pwrseq_unit_data *data)
}
kref_init(&unit->ref);
+ INIT_LIST_HEAD(&unit->list);
INIT_LIST_HEAD(&unit->deps);
unit->enable = data->enable;
unit->disable = data->disable;
@@ -504,10 +505,6 @@ pwrseq_device_register(const struct pwrseq_config *config)
*/
device_initialize(&pwrseq->dev);
- ret = dev_set_name(&pwrseq->dev, "pwrseq.%d", pwrseq->id);
- if (ret)
- goto err_put_pwrseq;
-
pwrseq->owner = config->owner ?: THIS_MODULE;
pwrseq->match = config->match;
@@ -516,6 +513,10 @@ pwrseq_device_register(const struct pwrseq_config *config)
INIT_LIST_HEAD(&pwrseq->targets);
INIT_LIST_HEAD(&pwrseq->units);
+ ret = dev_set_name(&pwrseq->dev, "pwrseq.%d", pwrseq->id);
+ if (ret)
+ goto err_put_pwrseq;
+
ret = pwrseq_setup_targets(config->targets, pwrseq);
if (ret)
goto err_put_pwrseq;
@@ -913,6 +914,8 @@ int pwrseq_power_on(struct pwrseq_desc *desc)
if (!ret)
desc->powered_on = true;
}
+ if (ret)
+ return ret;
if (target->post_enable) {
ret = target->post_enable(pwrseq);
diff --git a/drivers/power/supply/ab8500_fg.c b/drivers/power/supply/ab8500_fg.c
index a71903b1bf78..89b6e599087a 100644
--- a/drivers/power/supply/ab8500_fg.c
+++ b/drivers/power/supply/ab8500_fg.c
@@ -3055,6 +3055,20 @@ static void ab8500_fg_unbind(struct device *dev, struct device *master,
flush_workqueue(di->fg_wq);
}
+/* Disable, not cancel: works stay disabled so nothing can re-arm them. */
+static void ab8500_fg_destroy_workqueue(void *data)
+{
+ struct ab8500_fg *di = data;
+
+ disable_work_sync(&di->fg_acc_cur_work);
+ disable_work_sync(&di->fg_work);
+ disable_delayed_work_sync(&di->fg_reinit_work);
+ disable_delayed_work_sync(&di->fg_low_bat_work);
+ disable_delayed_work_sync(&di->fg_check_hw_failure_work);
+ disable_delayed_work_sync(&di->fg_periodic_work);
+ destroy_workqueue(di->fg_wq);
+}
+
static const struct component_ops ab8500_fg_component_ops = {
.bind = ab8500_fg_bind,
.unbind = ab8500_fg_unbind,
@@ -3156,6 +3170,11 @@ static int ab8500_fg_probe(struct platform_device *pdev)
return PTR_ERR(di->fg_psy);
}
+ /* Registered after fg_psy, before the IRQs: devm frees IRQ -> workqueue -> fg_psy. */
+ ret = devm_add_action_or_reset(dev, ab8500_fg_destroy_workqueue, di);
+ if (ret)
+ return ret;
+
di->fg_samples = SEC_TO_SAMPLE(di->bm->fg_params->init_timer);
/*
@@ -3168,22 +3187,16 @@ static int ab8500_fg_probe(struct platform_device *pdev)
/* Register primary interrupt handlers */
for (i = 0; i < ARRAY_SIZE(ab8500_fg_irq); i++) {
irq = platform_get_irq_byname(pdev, ab8500_fg_irq[i].name);
- if (irq < 0) {
- destroy_workqueue(di->fg_wq);
+ if (irq < 0)
return irq;
- }
ret = devm_request_threaded_irq(dev, irq, NULL,
ab8500_fg_irq[i].isr,
IRQF_SHARED | IRQF_NO_SUSPEND | IRQF_ONESHOT,
ab8500_fg_irq[i].name, di);
- if (ret != 0) {
- dev_err(dev, "failed to request %s IRQ %d: %d\n",
- ab8500_fg_irq[i].name, irq, ret);
- destroy_workqueue(di->fg_wq);
+ if (ret != 0)
return ret;
- }
dev_dbg(dev, "Requested %s IRQ %d: %d\n",
ab8500_fg_irq[i].name, irq, ret);
}
@@ -3197,7 +3210,6 @@ static int ab8500_fg_probe(struct platform_device *pdev)
ret = ab8500_fg_sysfs_init(di);
if (ret) {
dev_err(dev, "failed to create sysfs entry\n");
- destroy_workqueue(di->fg_wq);
return ret;
}
@@ -3205,7 +3217,6 @@ static int ab8500_fg_probe(struct platform_device *pdev)
if (ret) {
dev_err(dev, "failed to create FG psy\n");
ab8500_fg_sysfs_exit(di);
- destroy_workqueue(di->fg_wq);
return ret;
}
@@ -3225,7 +3236,6 @@ static void ab8500_fg_remove(struct platform_device *pdev)
{
struct ab8500_fg *di = platform_get_drvdata(pdev);
- destroy_workqueue(di->fg_wq);
component_del(&pdev->dev, &ab8500_fg_component_ops);
list_del(&di->node);
ab8500_fg_sysfs_exit(di);
diff --git a/drivers/power/supply/qcom_battmgr.c b/drivers/power/supply/qcom_battmgr.c
index 59a19c6d11ec..d55982553863 100644
--- a/drivers/power/supply/qcom_battmgr.c
+++ b/drivers/power/supply/qcom_battmgr.c
@@ -4,6 +4,7 @@
* Copyright (c) 2022, Linaro Ltd
*/
#include <linux/auxiliary_bus.h>
+#include <linux/devm-helpers.h>
#include <linux/module.h>
#include <linux/mutex.h>
#include <linux/of_device.h>
@@ -1335,6 +1336,7 @@ static int qcom_battmgr_probe(struct auxiliary_device *adev,
const struct of_device_id *match;
struct qcom_battmgr *battmgr;
struct device *dev = &adev->dev;
+ int ret;
battmgr = devm_kzalloc(dev, sizeof(*battmgr), GFP_KERNEL);
if (!battmgr)
@@ -1350,7 +1352,6 @@ static int qcom_battmgr_probe(struct auxiliary_device *adev,
psy_cfg_supply.supplied_to = qcom_battmgr_battery;
psy_cfg_supply.num_supplicants = 1;
- INIT_WORK(&battmgr->enable_work, qcom_battmgr_enable_worker);
mutex_init(&battmgr->lock);
init_completion(&battmgr->ack);
@@ -1397,6 +1398,11 @@ static int qcom_battmgr_probe(struct auxiliary_device *adev,
"failed to register wireless charing power supply\n");
}
+ ret = devm_work_autocancel(dev, &battmgr->enable_work,
+ qcom_battmgr_enable_worker);
+ if (ret)
+ return ret;
+
battmgr->client = devm_pmic_glink_client_alloc(dev, PMIC_GLINK_OWNER_BATTMGR,
qcom_battmgr_callback,
qcom_battmgr_pdr_notify,
diff --git a/drivers/remoteproc/qcom_q6v5_adsp.c b/drivers/remoteproc/qcom_q6v5_adsp.c
index 80ac739621be..034594026812 100644
--- a/drivers/remoteproc/qcom_q6v5_adsp.c
+++ b/drivers/remoteproc/qcom_q6v5_adsp.c
@@ -104,6 +104,7 @@ struct qcom_adsp {
struct completion stop_done;
phys_addr_t mem_phys;
+ unsigned long iova;
phys_addr_t mem_reloc;
void *mem_region;
size_t mem_size;
@@ -333,7 +334,7 @@ static void adsp_unmap_carveout(struct rproc *rproc)
struct qcom_adsp *adsp = rproc->priv;
if (adsp->has_iommu)
- iommu_unmap(rproc->domain, adsp->mem_phys, adsp->mem_size);
+ iommu_unmap(rproc->domain, adsp->iova, adsp->mem_size);
}
static int adsp_map_carveout(struct rproc *rproc)
@@ -341,7 +342,6 @@ static int adsp_map_carveout(struct rproc *rproc)
struct qcom_adsp *adsp = rproc->priv;
struct of_phandle_args args;
long long sid;
- unsigned long iova;
int ret;
if (!adsp->has_iommu)
@@ -358,9 +358,9 @@ static int adsp_map_carveout(struct rproc *rproc)
of_node_put(args.np);
/* Add SID configuration for ADSP Firmware to SMMU */
- iova = adsp->mem_phys | (sid << 32);
+ adsp->iova = adsp->mem_phys | (sid << 32);
- ret = iommu_map(rproc->domain, iova, adsp->mem_phys,
+ ret = iommu_map(rproc->domain, adsp->iova, adsp->mem_phys,
adsp->mem_size, IOMMU_READ | IOMMU_WRITE,
GFP_KERNEL);
if (ret) {
diff --git a/drivers/remoteproc/qcom_q6v5_pas.c b/drivers/remoteproc/qcom_q6v5_pas.c
index ea4a91f37b50..23f928323bcf 100644
--- a/drivers/remoteproc/qcom_q6v5_pas.c
+++ b/drivers/remoteproc/qcom_q6v5_pas.c
@@ -115,8 +115,8 @@ struct qcom_adsp {
struct qcom_rproc_ssr ssr_subdev;
struct qcom_sysmon *sysmon;
- struct qcom_scm_pas_metadata pas_metadata;
- struct qcom_scm_pas_metadata dtb_pas_metadata;
+ struct qcom_scm_pas_context *pas_ctx;
+ struct qcom_scm_pas_context *dtb_pas_ctx;
};
static void adsp_segment_dump(struct rproc *rproc, struct rproc_dump_segment *segment,
@@ -203,60 +203,58 @@ static int adsp_unprepare(struct rproc *rproc)
struct qcom_adsp *adsp = rproc->priv;
/*
- * adsp_load() did pass pas_metadata to the SCM driver for storing
+ * adsp_load() did pass the PAS context to the SCM driver for storing
* metadata context. It might have been released already if
* auth_and_reset() was successful, but in other cases clean it up
* here.
*/
- qcom_scm_pas_metadata_release(&adsp->pas_metadata);
+ qcom_pas_metadata_release(adsp->pas_ctx);
if (adsp->dtb_pas_id)
- qcom_scm_pas_metadata_release(&adsp->dtb_pas_metadata);
+ qcom_pas_metadata_release(adsp->dtb_pas_ctx);
return 0;
}
static int adsp_load(struct rproc *rproc, const struct firmware *fw)
{
- struct qcom_adsp *adsp = rproc->priv;
+ struct qcom_adsp *pas = rproc->priv;
int ret;
/* Store firmware handle to be used in adsp_start() */
- adsp->firmware = fw;
+ pas->firmware = fw;
- if (adsp->lite_pas_id)
- ret = qcom_scm_pas_shutdown(adsp->lite_pas_id);
+ if (pas->lite_pas_id)
+ ret = qcom_scm_pas_shutdown(pas->lite_pas_id);
- if (adsp->dtb_pas_id) {
- ret = request_firmware(&adsp->dtb_firmware, adsp->dtb_firmware_name, adsp->dev);
+ if (pas->dtb_pas_id) {
+ ret = request_firmware(&pas->dtb_firmware, pas->dtb_firmware_name, pas->dev);
if (ret) {
- dev_err(adsp->dev, "request_firmware failed for %s: %d\n",
- adsp->dtb_firmware_name, ret);
+ dev_err(pas->dev, "request_firmware failed for %s: %d\n",
+ pas->dtb_firmware_name, ret);
return ret;
}
- ret = qcom_mdt_pas_init(adsp->dev, adsp->dtb_firmware, adsp->dtb_firmware_name,
- adsp->dtb_pas_id, adsp->dtb_mem_phys,
- &adsp->dtb_pas_metadata);
- if (ret)
- goto release_dtb_firmware;
+ ret = qcom_mdt_pas_init(pas->dev, pas->dtb_firmware, pas->dtb_firmware_name,
+ pas->dtb_pas_id, pas->dtb_mem_phys,
+ pas->dtb_pas_ctx);
+ if (ret) {
+ release_firmware(pas->dtb_firmware);
+ return ret;
+ }
- ret = qcom_mdt_load_no_init(adsp->dev, adsp->dtb_firmware, adsp->dtb_firmware_name,
- adsp->dtb_pas_id, adsp->dtb_mem_region,
- adsp->dtb_mem_phys, adsp->dtb_mem_size,
- &adsp->dtb_mem_reloc);
- if (ret)
- goto release_dtb_metadata;
+ ret = qcom_mdt_load_no_init(
+ pas->dev, pas->dtb_firmware, pas->dtb_firmware_name,
+ pas->dtb_pas_id, pas->dtb_mem_region,
+ pas->dtb_mem_phys, pas->dtb_mem_size,
+ &pas->dtb_mem_reloc);
+ if (ret) {
+ qcom_pas_metadata_release(pas->dtb_pas_ctx);
+ release_firmware(pas->dtb_firmware);
+ return ret;
+ }
}
return 0;
-
-release_dtb_metadata:
- qcom_scm_pas_metadata_release(&adsp->dtb_pas_metadata);
-
-release_dtb_firmware:
- release_firmware(adsp->dtb_firmware);
-
- return ret;
}
static int adsp_start(struct rproc *rproc)
@@ -302,7 +300,7 @@ static int adsp_start(struct rproc *rproc)
}
ret = qcom_mdt_pas_init(adsp->dev, adsp->firmware, rproc->firmware, adsp->pas_id,
- adsp->mem_phys, &adsp->pas_metadata);
+ adsp->mem_phys, adsp->pas_ctx);
if (ret)
goto disable_px_supply;
@@ -328,9 +326,9 @@ static int adsp_start(struct rproc *rproc)
goto release_pas_metadata;
}
- qcom_scm_pas_metadata_release(&adsp->pas_metadata);
+ qcom_pas_metadata_release(adsp->pas_ctx);
if (adsp->dtb_pas_id)
- qcom_scm_pas_metadata_release(&adsp->dtb_pas_metadata);
+ qcom_pas_metadata_release(adsp->dtb_pas_ctx);
/* Remove pointer to the loaded firmware, only valid in adsp_load() & adsp_start() */
adsp->firmware = NULL;
@@ -338,9 +336,9 @@ static int adsp_start(struct rproc *rproc)
return 0;
release_pas_metadata:
- qcom_scm_pas_metadata_release(&adsp->pas_metadata);
+ qcom_pas_metadata_release(adsp->pas_ctx);
if (adsp->dtb_pas_id)
- qcom_scm_pas_metadata_release(&adsp->dtb_pas_metadata);
+ qcom_pas_metadata_release(adsp->dtb_pas_ctx);
disable_px_supply:
if (adsp->px_supply)
regulator_disable(adsp->px_supply);
@@ -788,6 +786,27 @@ static int adsp_probe(struct platform_device *pdev)
}
qcom_add_ssr_subdev(rproc, &adsp->ssr_subdev, desc->ssr_name);
+
+ adsp->pas_ctx = devm_kzalloc(adsp->dev, sizeof(*adsp->pas_ctx), GFP_KERNEL);
+ if (!adsp->pas_ctx) {
+ ret = -ENOMEM;
+ goto remove_ssr_sysmon;
+ }
+ adsp->pas_ctx->dev = adsp->dev;
+ adsp->pas_ctx->pas_id = adsp->pas_id;
+ adsp->pas_ctx->mem_phys = adsp->mem_phys;
+ adsp->pas_ctx->mem_size = adsp->mem_size;
+
+ adsp->dtb_pas_ctx = devm_kzalloc(adsp->dev, sizeof(*adsp->dtb_pas_ctx), GFP_KERNEL);
+ if (!adsp->dtb_pas_ctx) {
+ ret = -ENOMEM;
+ goto remove_ssr_sysmon;
+ }
+ adsp->dtb_pas_ctx->dev = adsp->dev;
+ adsp->dtb_pas_ctx->pas_id = adsp->dtb_pas_id;
+ adsp->dtb_pas_ctx->mem_phys = adsp->dtb_mem_phys;
+ adsp->dtb_pas_ctx->mem_size = adsp->dtb_mem_size;
+
ret = rproc_add(rproc);
if (ret)
goto remove_ssr_sysmon;
diff --git a/drivers/rtc/rtc-rzn1.c b/drivers/rtc/rtc-rzn1.c
index c64eaf50bb39..668935dbcefe 100644
--- a/drivers/rtc/rtc-rzn1.c
+++ b/drivers/rtc/rtc-rzn1.c
@@ -193,13 +193,24 @@ static int rzn1_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
if (ret)
return ret;
+ ctl1 = readl(rtc->base + RZN1_RTC_CTL1);
+ alrm->enabled = !!(ctl1 & RZN1_RTC_CTL1_ALME);
+
min = readl(rtc->base + RZN1_RTC_ALM);
hour = readl(rtc->base + RZN1_RTC_ALH);
- wday = readl(rtc->base + RZN1_RTC_ALW);
tm->tm_sec = 0;
tm->tm_min = bcd2bin(min);
tm->tm_hour = bcd2bin(hour);
+
+ /*
+ * If wday is zero, no bit is set in RZN1_RTC_ALW. This is the
+ * register's power-on reset value.
+ */
+ wday = readl(rtc->base + RZN1_RTC_ALW);
+ if (!wday)
+ return 0;
+
delta_days = ((fls(wday) - 1) - tm->tm_wday + 7) % 7;
tm->tm_wday = fls(wday) - 1;
@@ -208,9 +219,6 @@ static int rzn1_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
rtc_time64_to_tm(alarm, tm);
}
- ctl1 = readl(rtc->base + RZN1_RTC_CTL1);
- alrm->enabled = !!(ctl1 & RZN1_RTC_CTL1_ALME);
-
return 0;
}
diff --git a/drivers/s390/cio/chp.c b/drivers/s390/cio/chp.c
index 0c5bda060249..b6ef882f9860 100644
--- a/drivers/s390/cio/chp.c
+++ b/drivers/s390/cio/chp.c
@@ -78,6 +78,9 @@ u8 chp_get_sch_opm(struct subchannel *sch)
int opm;
int i;
+ if (!sch->schib.pmcw.dnv)
+ return 0;
+
opm = 0;
chp_id_init(&chpid);
for (i = 0; i < 8; i++) {
diff --git a/drivers/s390/cio/cio.c b/drivers/s390/cio/cio.c
index ad17ab0a9314..700d8d4ba9da 100644
--- a/drivers/s390/cio/cio.c
+++ b/drivers/s390/cio/cio.c
@@ -453,7 +453,8 @@ EXPORT_SYMBOL_GPL(cio_commit_config);
/**
* cio_update_schib - Perform stsch and update schib if subchannel is valid.
* @sch: subchannel on which to perform stsch
- * Return zero on success, -ENODEV otherwise.
+ * Return zero on success, -ENODEV if the subchannel is not operational,
+ * -EACCES if the subchannel has no valid device.
*/
int cio_update_schib(struct subchannel *sch)
{
@@ -462,10 +463,12 @@ int cio_update_schib(struct subchannel *sch)
if (stsch(sch->schid, &schib))
return -ENODEV;
- memcpy(&sch->schib, &schib, sizeof(schib));
-
- if (!css_sch_is_valid(&schib))
+ if (!css_sch_is_valid(&schib)) {
+ memset(&sch->schib, 0, sizeof(sch->schib));
return -EACCES;
+ }
+
+ memcpy(&sch->schib, &schib, sizeof(schib));
return 0;
}
diff --git a/drivers/s390/cio/cio.h b/drivers/s390/cio/cio.h
index 23374604c6b0..de3658e476b7 100644
--- a/drivers/s390/cio/cio.h
+++ b/drivers/s390/cio/cio.h
@@ -7,6 +7,7 @@
#include <linux/mod_devicetable.h>
#include <asm/chpid.h>
#include <asm/cio.h>
+#include <asm/dma-types.h>
#include <asm/fcx.h>
#include <asm/schid.h>
#include <asm/tpi.h>
@@ -49,7 +50,7 @@ struct pmcw {
/* Target SCHIB configuration. */
struct schib_config {
- u64 mba;
+ dma64_t mba;
u32 intparm;
u16 mbi;
u32 isc:3;
@@ -66,7 +67,7 @@ struct schib_config {
struct schib {
struct pmcw pmcw; /* path management control word */
union scsw scsw; /* subchannel status word */
- __u64 mba; /* measurement block address */
+ dma64_t mba; /* measurement block address */
__u8 mda[4]; /* model dependent area */
} __attribute__ ((packed,aligned(4)));
diff --git a/drivers/s390/cio/cmf.c b/drivers/s390/cio/cmf.c
index f80dc18e2a76..0cd4b26e4d92 100644
--- a/drivers/s390/cio/cmf.c
+++ b/drivers/s390/cio/cmf.c
@@ -184,7 +184,7 @@ static int set_schib(struct ccw_device *cdev, u32 mme, int mbfc,
sch->config.mbfc = mbfc;
/* address can be either a block address or a block index */
if (mbfc)
- sch->config.mba = address;
+ sch->config.mba = address ? virt_to_dma64((void *)address) : 0;
else
sch->config.mbi = address;
diff --git a/drivers/s390/cio/device.c b/drivers/s390/cio/device.c
index 7e309ea2e177..b6e2839dfeb3 100644
--- a/drivers/s390/cio/device.c
+++ b/drivers/s390/cio/device.c
@@ -925,7 +925,7 @@ static int ccw_device_move_to_sch(struct ccw_device *cdev,
if (!sch_is_pseudo_sch(old_sch)) {
spin_lock_irq(&old_sch->lock);
- old_enabled = old_sch->schib.pmcw.ena;
+ old_enabled = old_sch->schib.pmcw.dnv && old_sch->schib.pmcw.ena;
rc = 0;
if (old_enabled)
rc = cio_disable_subchannel(old_sch);
@@ -944,7 +944,7 @@ static int ccw_device_move_to_sch(struct ccw_device *cdev,
CIO_MSG_EVENT(0, "device_move(0.%x.%04x,0.%x.%04x)=%d\n",
cdev->private->dev_id.ssid,
cdev->private->dev_id.devno, sch->schid.ssid,
- sch->schib.pmcw.dev, rc);
+ sch->schid.sch_no, rc);
if (old_enabled) {
/* Try to re-enable the old subchannel. */
spin_lock_irq(&old_sch->lock);
@@ -1210,7 +1210,7 @@ static void io_subchannel_quiesce(struct subchannel *sch)
cdev = sch_get_cdev(sch);
if (cio_is_console(sch->schid))
goto out_unlock;
- if (!sch->schib.pmcw.ena)
+ if (!sch->schib.pmcw.dnv || !sch->schib.pmcw.ena)
goto out_unlock;
ret = cio_disable_subchannel(sch);
if (ret != -EBUSY)
@@ -1257,7 +1257,8 @@ static int recovery_check(struct device *dev, void *data)
switch (cdev->private->state) {
case DEV_STATE_ONLINE:
sch = to_subchannel(cdev->dev.parent);
- if ((sch->schib.pmcw.pam & sch->opm) == sch->vpm)
+ if (sch->schib.pmcw.dnv &&
+ (sch->schib.pmcw.pam & sch->opm) == sch->vpm)
break;
fallthrough;
case DEV_STATE_DISCONNECTED:
diff --git a/drivers/s390/cio/device_fsm.c b/drivers/s390/cio/device_fsm.c
index 42791fa0b80e..96bab4cde37a 100644
--- a/drivers/s390/cio/device_fsm.c
+++ b/drivers/s390/cio/device_fsm.c
@@ -169,6 +169,9 @@ __recover_lost_chpids(struct subchannel *sch, int old_lpm)
int mask, i;
struct chp_id chpid;
+ if (!sch->schib.pmcw.dnv)
+ return;
+
chp_id_init(&chpid);
for (i = 0; i<8; i++) {
mask = 0x80 >> i;
diff --git a/drivers/s390/cio/device_ops.c b/drivers/s390/cio/device_ops.c
index acd6790dba4d..c0785220a01e 100644
--- a/drivers/s390/cio/device_ops.c
+++ b/drivers/s390/cio/device_ops.c
@@ -142,6 +142,8 @@ int ccw_device_clear(struct ccw_device *cdev, unsigned long intparm)
if (!cdev || !cdev->dev.parent)
return -ENODEV;
sch = to_subchannel(cdev->dev.parent);
+ if (!sch->schib.pmcw.dnv)
+ return -ENODEV;
if (!sch->schib.pmcw.ena)
return -EINVAL;
if (cdev->private->state == DEV_STATE_NOT_OPER)
@@ -198,6 +200,8 @@ int ccw_device_start_timeout_key(struct ccw_device *cdev, struct ccw1 *cpa,
if (!cdev || !cdev->dev.parent)
return -ENODEV;
sch = to_subchannel(cdev->dev.parent);
+ if (!sch->schib.pmcw.dnv)
+ return -ENODEV;
if (!sch->schib.pmcw.ena)
return -EINVAL;
if (cdev->private->state == DEV_STATE_NOT_OPER)
@@ -379,6 +383,8 @@ int ccw_device_halt(struct ccw_device *cdev, unsigned long intparm)
if (!cdev || !cdev->dev.parent)
return -ENODEV;
sch = to_subchannel(cdev->dev.parent);
+ if (!sch->schib.pmcw.dnv)
+ return -ENODEV;
if (!sch->schib.pmcw.ena)
return -EINVAL;
if (cdev->private->state == DEV_STATE_NOT_OPER)
@@ -413,6 +419,8 @@ int ccw_device_resume(struct ccw_device *cdev)
if (!cdev || !cdev->dev.parent)
return -ENODEV;
sch = to_subchannel(cdev->dev.parent);
+ if (!sch->schib.pmcw.dnv)
+ return -ENODEV;
if (!sch->schib.pmcw.ena)
return -EINVAL;
if (cdev->private->state == DEV_STATE_NOT_OPER)
@@ -482,6 +490,8 @@ struct channel_path_desc_fmt0 *ccw_device_get_chp_desc(struct ccw_device *cdev,
struct chp_id chpid;
sch = to_subchannel(cdev->dev.parent);
+ if (!sch->schib.pmcw.dnv)
+ return NULL;
chp_id_init(&chpid);
chpid.id = sch->schib.pmcw.chpid[chp_idx];
return chp_get_chp_desc(chpid);
@@ -502,9 +512,13 @@ u8 *ccw_device_get_util_str(struct ccw_device *cdev, int chp_idx)
struct chp_id chpid;
u8 *util_str;
+ if (!sch->schib.pmcw.dnv)
+ return NULL;
chp_id_init(&chpid);
chpid.id = sch->schib.pmcw.chpid[chp_idx];
chp = chpid_to_chp(chpid);
+ if (!chp)
+ return NULL;
util_str = kmalloc(sizeof(chp->desc_fmt3.util_str), GFP_KERNEL);
if (!util_str)
@@ -548,6 +562,8 @@ int ccw_device_tm_start_timeout_key(struct ccw_device *cdev, struct tcw *tcw,
int rc;
sch = to_subchannel(cdev->dev.parent);
+ if (!sch->schib.pmcw.dnv)
+ return -ENODEV;
if (!sch->schib.pmcw.ena)
return -EINVAL;
if (cdev->private->state == DEV_STATE_VERIFY) {
@@ -652,6 +668,9 @@ int ccw_device_get_mdc(struct ccw_device *cdev, u8 mask)
struct chp_id chpid;
int mdc = 0, i;
+ if (!sch->schib.pmcw.dnv)
+ return 0;
+
/* Adjust requested path mask to excluded varied off paths. */
if (mask)
mask &= sch->lpm;
@@ -694,6 +713,8 @@ int ccw_device_tm_intrg(struct ccw_device *cdev)
{
struct subchannel *sch = to_subchannel(cdev->dev.parent);
+ if (!sch->schib.pmcw.dnv)
+ return -ENODEV;
if (!sch->schib.pmcw.ena)
return -EINVAL;
if (cdev->private->state != DEV_STATE_ONLINE)
@@ -786,6 +807,8 @@ int ccw_device_get_chpid(struct ccw_device *cdev, int chp_idx, u8 *chpid)
if ((chp_idx < 0) || (chp_idx > 7))
return -EINVAL;
+ if (!sch->schib.pmcw.dnv)
+ return -ENODEV;
mask = 0x80 >> chp_idx;
if (!(sch->schib.pmcw.pim & mask))
return -ENODEV;
diff --git a/drivers/s390/cio/vfio_ccw_fsm.c b/drivers/s390/cio/vfio_ccw_fsm.c
index 5fd94e9d5c61..9a000b0231d6 100644
--- a/drivers/s390/cio/vfio_ccw_fsm.c
+++ b/drivers/s390/cio/vfio_ccw_fsm.c
@@ -399,7 +399,7 @@ static void fsm_close(struct vfio_ccw_private *private,
spin_lock_irq(&sch->lock);
- if (!sch->schib.pmcw.ena)
+ if (!sch->schib.pmcw.dnv || !sch->schib.pmcw.ena)
goto err_unlock;
ret = cio_disable_subchannel(sch);
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index ab29febebbfd..6157d420916a 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -277,9 +277,9 @@ static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q)
{
if (!q)
return;
- if (q->saved_isc != VFIO_AP_ISC_INVALID &&
- !WARN_ON(!(q->matrix_mdev && q->matrix_mdev->kvm))) {
- kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
+ if (q->saved_isc != VFIO_AP_ISC_INVALID) {
+ if (!WARN_ON(!q->matrix_mdev) && q->matrix_mdev->kvm)
+ kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc);
q->saved_isc = VFIO_AP_ISC_INVALID;
}
if (q->saved_iova && !WARN_ON(!q->matrix_mdev)) {
@@ -786,11 +786,17 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)
ret = vfio_register_emulated_iommu_dev(&matrix_mdev->vdev);
if (ret)
goto err_put_vdev;
- matrix_mdev->req_trigger = NULL;
+
+ /*
+ * Take the matrix_dev->guests_lock mutex before adding the matrix_mdev
+ * to the mdev_list. All functions that traverse the list must also hold
+ * this lock to guard against additions to or removals from the list
+ * while it is being traversed.
+ */
+ mutex_lock(&matrix_dev->guests_lock);
dev_set_drvdata(&mdev->dev, matrix_mdev);
- mutex_lock(&matrix_dev->mdevs_lock);
list_add(&matrix_mdev->node, &matrix_dev->mdev_list);
- mutex_unlock(&matrix_dev->mdevs_lock);
+ mutex_unlock(&matrix_dev->guests_lock);
return 0;
err_put_vdev:
@@ -1926,6 +1932,8 @@ static int apq_status_check(int apqn, struct ap_queue_status *status)
* a value indicating a reset needs to be performed again.
*/
return -EAGAIN;
+ case AP_RESPONSE_Q_NOT_AVAIL:
+ return -ENODEV;
default:
WARN(true,
"failed to verify reset of queue %02x.%04x: TAPQ rc=%u\n",
@@ -1952,6 +1960,10 @@ static void apq_reset_check(struct work_struct *reset_work)
ret = apq_status_check(q->apqn, &status);
if (ret == -EIO)
return;
+ if (ret == -ENODEV) {
+ vfio_ap_free_aqic_resources(q);
+ return;
+ }
if (ret == -EBUSY) {
pr_notice_ratelimited(WAIT_MSG, elapsed,
AP_QID_CARD(q->apqn),
@@ -1995,6 +2007,7 @@ static void vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q)
break;
case AP_RESPONSE_DECONFIGURED:
case AP_RESPONSE_CHECKSTOPPED:
+ case AP_RESPONSE_Q_NOT_AVAIL:
vfio_ap_free_aqic_resources(q);
break;
default:
@@ -2240,6 +2253,8 @@ static struct ap_matrix_mdev *vfio_ap_mdev_for_queue(struct vfio_ap_queue *q)
unsigned long apid = AP_QID_CARD(q->apqn);
unsigned long apqi = AP_QID_QUEUE(q->apqn);
+ lockdep_assert_held(&matrix_dev->guests_lock);
+
list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
if (test_bit_inv(apid, matrix_mdev->matrix.apm) &&
test_bit_inv(apqi, matrix_mdev->matrix.aqm))
@@ -2259,6 +2274,7 @@ static ssize_t status_show(struct device *dev,
struct ap_matrix_mdev *matrix_mdev;
struct ap_device *apdev = to_ap_dev(dev);
+ mutex_lock(&matrix_dev->guests_lock);
mutex_lock(&matrix_dev->mdevs_lock);
q = dev_get_drvdata(&apdev->device);
@@ -2307,6 +2323,7 @@ static ssize_t status_show(struct device *dev,
done:
mutex_unlock(&matrix_dev->mdevs_lock);
+ mutex_unlock(&matrix_dev->guests_lock);
return nchars;
}
@@ -2470,12 +2487,15 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
/*
* If the queue is not in the host's AP configuration, then resetting
* it will fail with response code 01, (APQN not valid); so, let's make
- * sure it is in the host's config.
+ * sure it is in the host's config. If it is not, free the KVM GISC
+ * resources.
*/
if (test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm) &&
test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm)) {
vfio_ap_mdev_reset_queue(q);
flush_work(&q->reset_work);
+ } else {
+ vfio_ap_free_aqic_resources(q);
}
done:
@@ -2736,6 +2756,12 @@ static void vfio_ap_mdev_cfg_add(unsigned long *apm_add, unsigned long *aqm_add,
vfio_ap_filter_apid_by_qtype(apm_add, aqm_add);
+ /*
+ * It is safe to traverse this list here because the
+ * required guard - matrix_dev->guests_lock - is taken in the
+ * vfio_ap_on_cfg_changed function prior to this function getting
+ * called.
+ */
list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
/*
* The mdevs_lock must be held in order to access fields
@@ -2803,6 +2829,10 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *cur_cfg_info,
if (!cur_cfg_info || !prev_cfg_info)
return;
+ /*
+ * Take the guests_lock mutex here to guard access to the
+ * matrix_dev->mdev_list in the two functions called below.
+ */
mutex_lock(&matrix_dev->guests_lock);
vfio_ap_mdev_on_cfg_remove(cur_cfg_info, prev_cfg_info);
diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h
index ce73f08ee889..3ae611625c7f 100644
--- a/drivers/scsi/fnic/fnic.h
+++ b/drivers/scsi/fnic/fnic.h
@@ -27,7 +27,7 @@
#define DRV_NAME "fnic"
#define DRV_DESCRIPTION "Cisco FCoE HBA Driver"
-#define DRV_VERSION "1.7.0.0"
+#define DRV_VERSION "1.7.0.1"
#define PFX DRV_NAME ": "
#define DFX DRV_NAME "%d: "
diff --git a/drivers/scsi/fnic/fnic_isr.c b/drivers/scsi/fnic/fnic_isr.c
index ff85441c6cea..44c58c4aa1ca 100644
--- a/drivers/scsi/fnic/fnic_isr.c
+++ b/drivers/scsi/fnic/fnic_isr.c
@@ -245,7 +245,14 @@ int fnic_set_intr_mode_msix(struct fnic *fnic)
unsigned int m = ARRAY_SIZE(fnic->wq);
unsigned int o = ARRAY_SIZE(fnic->hw_copy_wq);
unsigned int min_irqs = n + m + 1 + 1; /*rq, raw wq, wq, err*/
-
+ /*
+ * Make driver critical vectors unmanaged, or else it can get tied
+ * to an offline CPU. This can happen when hyper-threading is off.
+ */
+ struct irq_affinity affd = {
+ .pre_vectors = n + m + 1, /* rq, raw wq, 1 ioq */
+ .post_vectors = 1, /* err */
+ };
/*
* We need n RQs, m WQs, o Copy WQs, n+m+o CQs, and n+m+o+1 INTRs
* (last INTR is used for WQ/RQ errors and notification area)
@@ -263,8 +270,8 @@ int fnic_set_intr_mode_msix(struct fnic *fnic)
int vec_count = 0;
int vecs = fnic->rq_count + fnic->raw_wq_count + fnic->wq_copy_count + 1;
- vec_count = pci_alloc_irq_vectors(fnic->pdev, min_irqs, vecs,
- PCI_IRQ_MSIX | PCI_IRQ_AFFINITY);
+ vec_count = pci_alloc_irq_vectors_affinity(fnic->pdev, min_irqs,
+ vecs, PCI_IRQ_MSIX|PCI_IRQ_AFFINITY, &affd);
FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num,
"allocated %d MSI-X vectors\n",
vec_count);
diff --git a/drivers/scsi/fnic/fnic_main.c b/drivers/scsi/fnic/fnic_main.c
index 1cb517f731f4..02a5c6f4580e 100644
--- a/drivers/scsi/fnic/fnic_main.c
+++ b/drivers/scsi/fnic/fnic_main.c
@@ -579,6 +579,8 @@ static int fnic_scsi_drv_init(struct fnic *fnic)
void fnic_mq_map_queues_cpus(struct Scsi_Host *host)
{
+ const struct cpumask *mask;
+ unsigned int queue, cpu;
struct fc_lport *lp = shost_priv(host);
struct fnic *fnic = lport_priv(lp);
struct pci_dev *l_pdev = fnic->pdev;
@@ -600,7 +602,36 @@ void fnic_mq_map_queues_cpus(struct Scsi_Host *host)
return;
}
- blk_mq_map_hw_queues(qmap, &l_pdev->dev, FNIC_PCI_OFFSET);
+ for_each_possible_cpu(cpu)
+ qmap->mq_map[cpu] = 0;
+
+ /*
+ * Setup CPU to Queue mapping for all managed MSI-X IRQs.
+ * Q0 is driver critical and non-managed, hence start from Q1.
+ */
+ for (queue = 1; queue < qmap->nr_queues; queue++) {
+ int irq_num = pci_irq_vector(fnic->pdev,
+ queue + FNIC_PCI_OFFSET);
+
+ if (irq_num < 0)
+ continue;
+
+ mask = pci_irq_get_affinity(fnic->pdev,
+ queue + FNIC_PCI_OFFSET);
+ if (!mask) {
+ shost_printk(KERN_ERR, host,
+ "failed to get irq_affinity map for queue:%d\n", irq_num);
+ continue;
+ }
+ FNIC_MAIN_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num,
+ "got irq_affinity map for %d:\n", irq_num);
+ for_each_cpu(cpu, mask) {
+ qmap->mq_map[cpu] = qmap->queue_offset + queue;
+ FNIC_MAIN_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num,
+ "[Q%d] cpu:%d <=> irq:%d\n",
+ queue, cpu, irq_num);
+ }
+ }
}
static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
diff --git a/drivers/scsi/libiscsi_tcp.c b/drivers/scsi/libiscsi_tcp.c
index 4d2320512943..646a2142ed66 100644
--- a/drivers/scsi/libiscsi_tcp.c
+++ b/drivers/scsi/libiscsi_tcp.c
@@ -491,6 +491,9 @@ static int iscsi_tcp_data_in(struct iscsi_conn *conn, struct iscsi_task *task)
int datasn = be32_to_cpu(rhdr->datasn);
unsigned total_in_length = task->sc->sdb.length;
+ if (task->sc->sc_data_direction != DMA_FROM_DEVICE)
+ return ISCSI_ERR_PROTO;
+
/*
* lib iscsi will update this in the completion handling if there
* is status.
diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c b/drivers/scsi/megaraid/megaraid_sas_base.c
index 8a44e01ebf9b..04f892832c1d 100644
--- a/drivers/scsi/megaraid/megaraid_sas_base.c
+++ b/drivers/scsi/megaraid/megaraid_sas_base.c
@@ -7873,7 +7873,9 @@ megasas_resume(struct device *dev)
goto fail_init_mfi;
}
- if (megasas_get_ctrl_info(instance) != DCMD_SUCCESS)
+ scoped_guard(mutex, &instance->reset_mutex)
+ rval = megasas_get_ctrl_info(instance);
+ if (rval != DCMD_SUCCESS)
goto fail_init_mfi;
tasklet_init(&instance->isr_tasklet, instance->instancet->tasklet,
diff --git a/drivers/scsi/qla2xxx/qla_bsg.c b/drivers/scsi/qla2xxx/qla_bsg.c
index 3982f7d576a8..0fbf6dba5a36 100644
--- a/drivers/scsi/qla2xxx/qla_bsg.c
+++ b/drivers/scsi/qla2xxx/qla_bsg.c
@@ -158,6 +158,12 @@ qla24xx_proc_fcp_prio_cfg_cmd(struct bsg_job *bsg_job)
goto exit_fcp_prio_cfg;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t)) {
+ ret = -EINVAL;
+ goto exit_fcp_prio_cfg;
+ }
+
/* Get the sub command */
oper = bsg_request->rqst_data.h_vendor.vendor_cmd[1];
@@ -756,6 +762,10 @@ qla2x00_process_loopback(struct bsg_job *bsg_job)
return -EIO;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 3 * sizeof(uint32_t))
+ return -EINVAL;
+
memset(&elreq, 0, sizeof(elreq));
elreq.req_sg_cnt = dma_map_sg(&ha->pdev->dev,
@@ -988,6 +998,10 @@ qla84xx_reset(struct bsg_job *bsg_job)
return -EINVAL;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
flag = bsg_request->rqst_data.h_vendor.vendor_cmd[1];
rval = qla84xx_reset_chip(vha, flag == A84_ISSUE_RESET_DIAG_FW);
@@ -1032,6 +1046,10 @@ qla84xx_updatefw(struct bsg_job *bsg_job)
return -EINVAL;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
sg_cnt = dma_map_sg(&ha->pdev->dev, bsg_job->request_payload.sg_list,
bsg_job->request_payload.sg_cnt, DMA_TO_DEVICE);
if (!sg_cnt) {
@@ -1482,6 +1500,10 @@ qla2x00_read_optrom(struct bsg_job *bsg_job)
struct qla_hw_data *ha = vha->hw;
int rval = 0;
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
if (ha->flags.nic_core_reset_hdlr_active)
return -EBUSY;
@@ -1519,6 +1541,10 @@ qla2x00_update_optrom(struct bsg_job *bsg_job)
struct qla_hw_data *ha = vha->hw;
int rval = 0;
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
mutex_lock(&ha->optrom_mutex);
rval = qla2x00_optrom_setup(bsg_job, vha, 1);
if (rval) {
@@ -2010,6 +2036,11 @@ qlafx00_mgmt_cmd(struct bsg_job *bsg_job)
struct fc_port *fcport;
char *type = "FC_BSG_HST_FX_MGMT";
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + sizeof(uint32_t) +
+ sizeof(struct qla_mt_iocb_rqst_fx00))
+ return -EINVAL;
+
/* Copy the IOCB specific information */
piocb_rqst = (struct qla_mt_iocb_rqst_fx00 *)
&bsg_request->rqst_data.h_vendor.vendor_cmd[1];
@@ -2412,7 +2443,7 @@ qla2x00_do_dport_diagnostics(struct bsg_job *bsg_job)
!IS_QLA28XX(vha->hw))
return -EPERM;
- dd = kmalloc(sizeof(*dd), GFP_KERNEL);
+ dd = kzalloc(sizeof(*dd), GFP_KERNEL);
if (!dd) {
ql_log(ql_log_warn, vha, 0x70db,
"Failed to allocate memory for dport.\n");
@@ -2899,6 +2930,13 @@ qla2x00_process_vendor_specific(struct scsi_qla_host *vha, struct bsg_job *bsg_j
{
struct fc_bsg_request *bsg_request = bsg_job->request;
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + sizeof(uint32_t)) {
+ ql_log(ql_log_warn, vha, 0x7000,
+ "BSG request too small for vendor cmd.\n");
+ return -EINVAL;
+ }
+
ql_dbg(ql_dbg_edif, vha, 0x911b, "%s FC_BSG_HST_VENDOR cmd[0]=0x%x\n",
__func__, bsg_request->rqst_data.h_vendor.vendor_cmd[0]);
@@ -3024,8 +3062,11 @@ qla24xx_bsg_request(struct bsg_job *bsg_job)
}
/* Disable port will bring down the chip, allow enable command */
- if (bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_MANAGE_HOST_PORT ||
- bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_GET_HOST_STATS)
+ if (bsg_request->msgcode == FC_BSG_HST_VENDOR &&
+ bsg_job->request_len >=
+ sizeof(struct fc_bsg_request) + sizeof(uint32_t) &&
+ (bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_MANAGE_HOST_PORT ||
+ bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_GET_HOST_STATS))
goto skip_chip_chk;
if (vha->hw->flags.port_isolated) {
diff --git a/drivers/scsi/qla2xxx/qla_def.h b/drivers/scsi/qla2xxx/qla_def.h
index b3265952c4be..783b6c429224 100644
--- a/drivers/scsi/qla2xxx/qla_def.h
+++ b/drivers/scsi/qla2xxx/qla_def.h
@@ -2570,6 +2570,8 @@ typedef struct fc_port {
struct list_head list;
struct scsi_qla_host *vha;
struct list_head unsol_ctx_head;
+ /* Serializes unsol_ctx_head against ISR, DPC and NVMe transport. */
+ spinlock_t unsol_ctx_lock;
unsigned int conf_compl_supported:1;
unsigned int deleted:2;
diff --git a/drivers/scsi/qla2xxx/qla_fw.h b/drivers/scsi/qla2xxx/qla_fw.h
index f307beed9d29..e7ded95afb9a 100644
--- a/drivers/scsi/qla2xxx/qla_fw.h
+++ b/drivers/scsi/qla2xxx/qla_fw.h
@@ -1442,6 +1442,10 @@ struct vp_ctrl_entry_24xx {
uint8_t reserved_5[24];
};
+/* vp_idx_map is a 128-bit (16-byte) bitmap selecting target VPs. */
+#define VP_CTRL_IDX_MAP_BITS \
+ (sizeof_field(struct vp_ctrl_entry_24xx, vp_idx_map) * 8)
+
/*
* Modify Virtual Port Configuration IOCB
*/
diff --git a/drivers/scsi/qla2xxx/qla_init.c b/drivers/scsi/qla2xxx/qla_init.c
index f2ed5c559089..dc69e9b3cf8f 100644
--- a/drivers/scsi/qla2xxx/qla_init.c
+++ b/drivers/scsi/qla2xxx/qla_init.c
@@ -4408,6 +4408,19 @@ qla2x00_setup_chip(scsi_qla_host_t *vha)
MIN_MULTI_ID_FABRIC))
ha->max_npiv_vports =
MIN_MULTI_ID_FABRIC - 1;
+
+ /*
+ * The VP_CTRL IOCB selects target VPs
+ * through the fixed vp_idx_map bitmap,
+ * so a vp_index beyond it can be enabled
+ * via VP_CONFIG but never disabled via
+ * VP_CTRL, leaking the VP. Cap the count
+ * to the bitmap capacity.
+ */
+ if (ha->max_npiv_vports >=
+ VP_CTRL_IDX_MAP_BITS)
+ ha->max_npiv_vports =
+ VP_CTRL_IDX_MAP_BITS - 1;
}
qla2x00_get_resource_cnts(vha);
qla_init_iocb_limit(vha);
@@ -5636,6 +5649,7 @@ qla2x00_alloc_fcport(scsi_qla_host_t *vha, gfp_t flags)
INIT_LIST_HEAD(&fcport->gnl_entry);
INIT_LIST_HEAD(&fcport->list);
INIT_LIST_HEAD(&fcport->unsol_ctx_head);
+ spin_lock_init(&fcport->unsol_ctx_lock);
INIT_LIST_HEAD(&fcport->sess_cmd_list);
spin_lock_init(&fcport->sess_cmd_lock);
diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c
index 0b41e8a06602..17231a8dab98 100644
--- a/drivers/scsi/qla2xxx/qla_iocb.c
+++ b/drivers/scsi/qla2xxx/qla_iocb.c
@@ -11,6 +11,10 @@
#include <scsi/scsi_tcq.h>
+/* All adapters supported by this tree use the 64-byte request ring. */
+#define qla_req_ring_slot(ha, req) ((req)->ring_ptr)
+#define qla_req_entry_size(ha) REQUEST_ENTRY_SIZE
+
static int qla_start_scsi_type6(srb_t *sp);
/**
* qla2x00_get_cmd_direction() - Determine control_flag data direction.
@@ -2286,10 +2290,9 @@ __qla2x00_alloc_iocbs(struct qla_qpair *qpair, srb_t *sp)
struct req_que *req = qpair->req;
device_reg_t *reg = ISP_QUE_REG(ha, req->id);
uint32_t handle;
- request_t *pkt;
uint16_t cnt, req_cnt;
+ request_t *pkt = NULL;
- pkt = NULL;
req_cnt = 1;
handle = 0;
@@ -2343,14 +2346,18 @@ __qla2x00_alloc_iocbs(struct qla_qpair *qpair, srb_t *sp)
sp->handle = handle;
}
- /* Prep packet */
+ /*
+ * Prep the current request-ring slot.
+ */
req->cnt -= req_cnt;
- pkt = req->ring_ptr;
- memset(pkt, 0, REQUEST_ENTRY_SIZE);
+ pkt = qla_req_ring_slot(ha, req);
if (IS_QLAFX00(ha)) {
+ memset_io((void __iomem __force *)pkt, 0,
+ qla_req_entry_size(ha));
wrt_reg_byte((u8 __force __iomem *)&pkt->entry_count, req_cnt);
wrt_reg_dword((__le32 __force __iomem *)&pkt->handle, handle);
} else {
+ memset(pkt, 0, qla_req_entry_size(ha));
pkt->entry_count = req_cnt;
pkt->handle = handle;
}
@@ -3828,6 +3835,12 @@ qla25xx_ctrlvp_iocb(srb_t *sp, struct vp_ctrl_entry_24xx *vce)
*/
map = (sp->u.iocb_cmd.u.ctrlvp.vp_index - 1) / 8;
pos = (sp->u.iocb_cmd.u.ctrlvp.vp_index - 1) & 7;
+ if (map >= ARRAY_SIZE(vce->vp_idx_map)) {
+ ql_log(ql_log_warn, sp->vha, 0x307c,
+ "ctrlvp: vp_index %u exceeds vp_idx_map capacity\n",
+ sp->u.iocb_cmd.u.ctrlvp.vp_index);
+ return;
+ }
vce->vp_idx_map[map] |= 1 << pos;
}
diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c
index 9c26e4a0afc1..b4ee1f4a0d43 100644
--- a/drivers/scsi/qla2xxx/qla_isr.c
+++ b/drivers/scsi/qla2xxx/qla_isr.c
@@ -921,14 +921,6 @@ qla27xx_copy_multiple_pkt(struct scsi_qla_host *vha, void **pkt,
do {
while ((total_bytes > 0) && (entry_count_remaining > 0)) {
- if (rsp_q->ring_ptr->signature == RESPONSE_PROCESSED) {
- ql_dbg(ql_dbg_async, vha, 0x5084,
- "Ran out of IOCBs, partial data 0x%x\n",
- buffer_copy_offset);
- cpu_relax();
- continue;
- }
-
new_pkt = (sts_cont_entry_t *)rsp_q->ring_ptr;
*pkt = new_pkt;
@@ -1205,14 +1197,6 @@ qla27xx_copy_fpin_pkt(struct scsi_qla_host *vha, void **pkt,
do {
while ((total_bytes > 0) && (entry_count_remaining > 0)) {
- if (rsp_q->ring_ptr->signature == RESPONSE_PROCESSED) {
- ql_dbg(ql_dbg_async, vha, 0x5084,
- "Ran out of IOCBs, partial data 0x%x\n",
- buffer_copy_offset);
- cpu_relax();
- continue;
- }
-
new_pkt = (sts_cont_entry_t *)rsp_q->ring_ptr;
*pkt = new_pkt;
@@ -4142,9 +4126,24 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha,
"SCM not active for this port\n");
break;
}
+ if (qla_chk_cont_iocb_avail(vha, rsp,
+ (response_t *)pkt, rsp_in)) {
+ /*
+ * ring_ptr and ring_index were
+ * pre-incremented above. Reset them
+ * back to current. Wait for next
+ * interrupt with all IOCBs to arrive
+ * and re-process.
+ */
+ rsp->ring_ptr = (response_t *)pkt;
+ rsp->ring_index = cur_ring_index;
+
+ ql_dbg(ql_dbg_init, vha, 0x5095,
+ "Defer processing FPIN...\n");
+ return;
+ }
pure_item = qla27xx_copy_fpin_pkt(vha,
(void **)&pkt, &rsp);
- __update_rsp_in(is_shadow_hba, rsp, rsp_in);
if (!pure_item)
break;
qla24xx_queue_purex_item(vha, pure_item,
diff --git a/drivers/scsi/qla2xxx/qla_mbx.c b/drivers/scsi/qla2xxx/qla_mbx.c
index 5bb9241ebebd..391a69878705 100644
--- a/drivers/scsi/qla2xxx/qla_mbx.c
+++ b/drivers/scsi/qla2xxx/qla_mbx.c
@@ -4147,6 +4147,8 @@ qla24xx_report_id_acquisition(scsi_qla_host_t *vha,
spin_lock_irqsave(&ha->vport_slock, flags);
list_for_each_entry(vp, &ha->vp_list, list) {
if (rptid_entry->vp_idx == vp->vp_idx) {
+ if (test_bit(VPORT_DELETE, &vp->dpc_flags))
+ break;
found = 1;
atomic_inc(&vp->vref_count);
break;
@@ -5658,7 +5660,7 @@ qla2x00_get_data_rate(scsi_qla_host_t *vha)
ql_dbg(ql_dbg_mbx, vha, 0x1107,
"Failed=%x mb[0]=%x.\n", rval, mcp->mb[0]);
} else {
- if (mcp->mb[1] != 0x7)
+ if (mcp->mb[1] != 0x7 || IS_QLA28XX(ha))
ha->link_data_rate = mcp->mb[1];
if (IS_QLA83XX(ha) || IS_QLA27XX(ha) || IS_QLA28XX(ha)) {
@@ -5669,8 +5671,6 @@ qla2x00_get_data_rate(scsi_qla_host_t *vha)
ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x1108,
"Done %s.\n", __func__);
- if (mcp->mb[1] != 0x7)
- ha->link_data_rate = mcp->mb[1];
}
return rval;
diff --git a/drivers/scsi/qla2xxx/qla_mid.c b/drivers/scsi/qla2xxx/qla_mid.c
index 9946899dd83b..4caad0ce9e06 100644
--- a/drivers/scsi/qla2xxx/qla_mid.c
+++ b/drivers/scsi/qla2xxx/qla_mid.c
@@ -574,16 +574,19 @@ qla25xx_free_req_que(struct scsi_qla_host *vha, struct req_que *req)
{
struct qla_hw_data *ha = vha->hw;
uint16_t que_id = req->id;
+ size_t req_entry_size = sizeof(request_t);
- dma_free_coherent(&ha->pdev->dev, (req->length + 1) *
- sizeof(request_t), req->ring, req->dma);
+ if (req->ring)
+ dma_free_coherent(&ha->pdev->dev,
+ (req->length + 1) * req_entry_size,
+ req->ring, req->dma);
req->ring = NULL;
req->dma = 0;
if (que_id) {
+ mutex_lock(&ha->mq_lock);
ha->req_q_map[que_id] = NULL;
- mutex_lock(&ha->vport_lock);
clear_bit(que_id, ha->req_qid_map);
- mutex_unlock(&ha->vport_lock);
+ mutex_unlock(&ha->mq_lock);
}
kfree(req->outstanding_cmds);
kfree(req);
@@ -594,6 +597,7 @@ qla25xx_free_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp)
{
struct qla_hw_data *ha = vha->hw;
uint16_t que_id = rsp->id;
+ size_t rsp_entry_size = sizeof(response_t);
if (rsp->msix && rsp->msix->have_irq) {
free_irq(rsp->msix->vector, rsp->msix->handle);
@@ -601,15 +605,22 @@ qla25xx_free_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp)
rsp->msix->in_use = 0;
rsp->msix->handle = NULL;
}
- dma_free_coherent(&ha->pdev->dev, (rsp->length + 1) *
- sizeof(response_t), rsp->ring, rsp->dma);
+
+ /* Flush any queued response work before freeing the queue/qpair. */
+ if (rsp->qpair && ha->wq)
+ cancel_work_sync(&rsp->qpair->q_work);
+
+ if (rsp->ring)
+ dma_free_coherent(&ha->pdev->dev,
+ (rsp->length + 1) * rsp_entry_size,
+ rsp->ring, rsp->dma);
rsp->ring = NULL;
rsp->dma = 0;
if (que_id) {
+ mutex_lock(&ha->mq_lock);
ha->rsp_q_map[que_id] = NULL;
- mutex_lock(&ha->vport_lock);
clear_bit(que_id, ha->rsp_qid_map);
- mutex_unlock(&ha->vport_lock);
+ mutex_unlock(&ha->mq_lock);
}
kfree(rsp);
}
@@ -794,9 +805,6 @@ qla25xx_create_req_que(struct qla_hw_data *ha, uint16_t options,
if (ret != QLA_SUCCESS) {
ql_log(ql_log_fatal, base_vha, 0x00df,
"%s failed.\n", __func__);
- mutex_lock(&ha->mq_lock);
- clear_bit(que_id, ha->req_qid_map);
- mutex_unlock(&ha->mq_lock);
goto que_failed;
}
vha->flags.qpairs_req_created = 1;
@@ -908,9 +916,6 @@ qla25xx_create_rsp_que(struct qla_hw_data *ha, uint16_t options,
if (ret != QLA_SUCCESS) {
ql_log(ql_log_fatal, base_vha, 0x00e7,
"%s failed.\n", __func__);
- mutex_lock(&ha->mq_lock);
- clear_bit(que_id, ha->rsp_qid_map);
- mutex_unlock(&ha->mq_lock);
goto que_failed;
}
vha->flags.qpairs_rsp_created = 1;
@@ -957,6 +962,22 @@ int qla24xx_control_vp(scsi_qla_host_t *vha, int cmd)
if (vp_index == 0 || vp_index >= ha->max_npiv_vports)
return QLA_PARAMETER_ERROR;
+ /*
+ * The VP_CTRL IOCB selects the target VP through a fixed 128-bit
+ * (16-byte) vp_idx_map bitmap, so vp_index must fit within it even
+ * if firmware advertises more NPIV vports.
+ */
+ if (vp_index > sizeof_field(struct vp_ctrl_entry_24xx, vp_idx_map) * 8)
+ return QLA_PARAMETER_ERROR;
+
+ /*
+ * The VP_CTRL IOCB selects the target VP through a fixed 128-bit
+ * (16-byte) vp_idx_map bitmap, so vp_index must fit within it even
+ * if firmware advertises more NPIV vports.
+ */
+ if (vp_index > VP_CTRL_IDX_MAP_BITS)
+ return QLA_PARAMETER_ERROR;
+
/* ref: INIT */
sp = qla2x00_get_sp(base_vha, NULL, GFP_KERNEL);
if (!sp)
diff --git a/drivers/scsi/qla2xxx/qla_nvme.c b/drivers/scsi/qla2xxx/qla_nvme.c
index 3160e212fab1..650d6be3af97 100644
--- a/drivers/scsi/qla2xxx/qla_nvme.c
+++ b/drivers/scsi/qla2xxx/qla_nvme.c
@@ -257,7 +257,9 @@ static void qla_nvme_release_lsrsp_cmd_kref(struct kref *kref)
fd_rsp = uctx->fd_rsp;
+ spin_lock_irqsave(&uctx->fcport->unsol_ctx_lock, flags);
list_del(&uctx->elem);
+ spin_unlock_irqrestore(&uctx->fcport->unsol_ctx_lock, flags);
fd_rsp->done(fd_rsp);
kfree(uctx);
@@ -446,6 +448,9 @@ static int qla_nvme_xmt_ls_rsp(struct nvme_fc_local_port *lport,
qla_nvme_ls_reject_iocb(vha, ha->base_qpair, &a, true);
spin_unlock_irqrestore(ha->base_qpair->qp_lock_ptr, flags);
}
+ spin_lock_irqsave(&uctx->fcport->unsol_ctx_lock, flags);
+ list_del(&uctx->elem);
+ spin_unlock_irqrestore(&uctx->fcport->unsol_ctx_lock, flags);
kfree(uctx);
return rval;
}
@@ -1215,7 +1220,9 @@ qla2xxx_process_purls_pkt(struct scsi_qla_host *vha, struct purex_item *item)
spin_unlock_irqrestore(vha->hw->base_qpair->qp_lock_ptr,
flags);
}
+ spin_lock_irqsave(&uctx->fcport->unsol_ctx_lock, flags);
list_del(&uctx->elem);
+ spin_unlock_irqrestore(&uctx->fcport->unsol_ctx_lock, flags);
kfree(uctx);
}
}
@@ -1257,6 +1264,7 @@ void qla2xxx_process_purls_iocb(void **pkt, struct rsp_que **rsp)
struct purex_item *item;
port_id_t d_id = {0};
port_id_t id = {0};
+ unsigned long flags;
u8 *opcode;
bool xmt_reject = false;
@@ -1322,7 +1330,9 @@ void qla2xxx_process_purls_iocb(void **pkt, struct rsp_que **rsp)
uctx->ox_id = p->ox_id;
qla_rport->uctx = uctx;
INIT_LIST_HEAD(&uctx->elem);
+ spin_lock_irqsave(&fcport->unsol_ctx_lock, flags);
list_add_tail(&uctx->elem, &fcport->unsol_ctx_head);
+ spin_unlock_irqrestore(&fcport->unsol_ctx_lock, flags);
item->purls_context = (void *)uctx;
ql_dbg(ql_dbg_unsol, vha, 0x2121,
diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index 7a78cff128cf..457fb3458ca7 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -363,7 +363,7 @@ MODULE_PARM_DESC(ql2xnvme_queues,
int ql2xfc2target = 1;
module_param(ql2xfc2target, int, 0444);
-MODULE_PARM_DESC(qla2xfc2target,
+MODULE_PARM_DESC(ql2xfc2target,
"Enables FC2 Target support. "
"0 - FC2 Target support is disabled. "
"1 - FC2 Target support is enabled (default).");
@@ -4482,25 +4482,40 @@ qla2x00_mem_alloc(struct qla_hw_data *ha, uint16_t req_len, uint16_t rsp_len,
fail_lsrjt:
dma_free_coherent(&ha->pdev->dev, ha->elsrej.size,
ha->elsrej.c, ha->elsrej.cdma);
+ ha->elsrej.c = NULL;
+ ha->elsrej.cdma = 0;
fail_elsrej:
dma_pool_destroy(ha->purex_dma_pool);
+ ha->purex_dma_pool = NULL;
fail_flt:
dma_free_coherent(&ha->pdev->dev, sizeof(struct qla_flt_header) + FLT_REGIONS_SIZE,
ha->flt, ha->flt_dma);
+ ha->flt = NULL;
+ ha->flt_dma = 0;
fail_flt_buffer:
dma_free_coherent(&ha->pdev->dev, SFP_DEV_SIZE,
ha->sfp_data, ha->sfp_data_dma);
+ ha->sfp_data = NULL;
+ ha->sfp_data_dma = 0;
fail_sfp_data:
kfree(ha->loop_id_map);
+ ha->loop_id_map = NULL;
fail_loop_id_map:
dma_pool_free(ha->s_dma_pool, ha->async_pd, ha->async_pd_dma);
+ ha->async_pd = NULL;
+ ha->async_pd_dma = 0;
fail_async_pd:
dma_pool_free(ha->s_dma_pool, ha->sf_init_cb, ha->sf_init_cb_dma);
+ ha->sf_init_cb = NULL;
+ ha->sf_init_cb_dma = 0;
fail_sf_init_cb:
dma_pool_free(ha->s_dma_pool, ha->ex_init_cb, ha->ex_init_cb_dma);
+ ha->ex_init_cb = NULL;
+ ha->ex_init_cb_dma = 0;
fail_ex_init_cb:
kfree(ha->npiv_info);
+ ha->npiv_info = NULL;
fail_npiv_info:
dma_free_coherent(&ha->pdev->dev, ((*rsp)->length + 1) *
sizeof(response_t), (*rsp)->ring, (*rsp)->dma);
diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index 9d2db5bc8ee7..dfd6a406ecda 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -713,6 +713,7 @@ int scsi_cdl_enable(struct scsi_device *sdev, bool enable)
struct scsi_mode_data data;
struct scsi_sense_hdr sshdr;
char *buf_data;
+ size_t avail, offset;
int len;
ret = scsi_mode_sense(sdev, 0x08, 0x0a, 0xf2, buf, sizeof(buf),
@@ -721,11 +722,24 @@ int scsi_cdl_enable(struct scsi_device *sdev, bool enable)
return -EINVAL;
/* Enable or disable CDL using the ATA feature page */
- len = min_t(size_t, sizeof(buf),
- data.length - data.header_length -
- data.block_descriptor_length);
- buf_data = buf + data.header_length +
- data.block_descriptor_length;
+ avail = min_t(size_t, data.length, sizeof(buf));
+ if (data.header_length > avail)
+ return -EINVAL;
+
+ offset = data.header_length;
+ avail -= data.header_length;
+
+ if (data.block_descriptor_length > avail)
+ return -EINVAL;
+
+ offset += data.block_descriptor_length;
+ avail -= data.block_descriptor_length;
+
+ if (avail < 5)
+ return -EINVAL;
+
+ buf_data = buf + offset;
+ len = avail;
/*
* If we want to enable CDL and CDL is already enabled on the
diff --git a/drivers/scsi/sd_zbc.c b/drivers/scsi/sd_zbc.c
index b8d42098f0b6..b96b49763322 100644
--- a/drivers/scsi/sd_zbc.c
+++ b/drivers/scsi/sd_zbc.c
@@ -595,7 +595,7 @@ int sd_zbc_revalidate_zones(struct scsi_disk *sdkp)
int sd_zbc_read_zones(struct scsi_disk *sdkp, struct queue_limits *lim,
u8 buf[SD_BUF_SIZE])
{
- unsigned int nr_zones;
+ u64 nr_zones;
u32 zone_blocks = 0;
int ret;
@@ -627,6 +627,12 @@ int sd_zbc_read_zones(struct scsi_disk *sdkp, struct queue_limits *lim,
goto err;
nr_zones = round_up(sdkp->capacity, zone_blocks) >> ilog2(zone_blocks);
+ if (nr_zones > INT_MAX) {
+ sd_printk(KERN_ERR, sdkp, "Too many zones (%llu)\n",
+ nr_zones);
+ ret = -EINVAL;
+ goto err;
+ }
sdkp->early_zone_info.nr_zones = nr_zones;
sdkp->early_zone_info.zone_blocks = zone_blocks;
diff --git a/drivers/soc/qcom/mdt_loader.c b/drivers/soc/qcom/mdt_loader.c
index 29124aa6d03b..f38aada2cabf 100644
--- a/drivers/soc/qcom/mdt_loader.c
+++ b/drivers/soc/qcom/mdt_loader.c
@@ -234,13 +234,13 @@ EXPORT_SYMBOL_GPL(qcom_mdt_read_metadata);
* @fw_name: name of the firmware, for construction of segment file names
* @pas_id: PAS identifier
* @mem_phys: physical address of allocated memory region
- * @ctx: PAS metadata context, to be released by caller
+ * @ctx: PAS context, metadata to be released by caller
*
* Returns 0 on success, negative errno otherwise.
*/
int qcom_mdt_pas_init(struct device *dev, const struct firmware *fw,
const char *fw_name, int pas_id, phys_addr_t mem_phys,
- struct qcom_scm_pas_metadata *ctx)
+ struct qcom_scm_pas_context *ctx)
{
const struct elf32_phdr *phdrs;
const struct elf32_phdr *phdr;
diff --git a/drivers/soundwire/cadence_master.c b/drivers/soundwire/cadence_master.c
index a503ef606a62..1557d6b04449 100644
--- a/drivers/soundwire/cadence_master.c
+++ b/drivers/soundwire/cadence_master.c
@@ -1670,6 +1670,13 @@ int sdw_cdns_clock_stop(struct sdw_cdns *cdns, bool block_wake)
return 0;
}
+ /*
+ * wait for any in-flight peripheral event handling to complete before stopping the clock.
+ * No need to disable peripheral interrupts before canceling the work, as the peripheral
+ * interrupts are already masked before the work is scheduled.
+ */
+ cancel_work_sync(&cdns->work);
+
/*
* Before entering clock stop we mask the Slave
* interrupts. This helps avoid having to deal with e.g. a
diff --git a/drivers/spi/spi-zynqmp-gqspi.c b/drivers/spi/spi-zynqmp-gqspi.c
index 143c572b263e..4fa68b11709b 100644
--- a/drivers/spi/spi-zynqmp-gqspi.c
+++ b/drivers/spi/spi-zynqmp-gqspi.c
@@ -1385,11 +1385,45 @@ static void zynqmp_qspi_remove(struct platform_device *pdev)
clk_disable_unprepare(xqspi->pclk);
}
+static void zynqmp_qspi_shutdown(struct platform_device *pdev)
+{
+ struct zynqmp_qspi *xqspi = platform_get_drvdata(pdev);
+ int ret;
+
+ /*
+ * Stop the queue and reject any later transfer first, so the write
+ * below cannot cut into a message that is still being executed.
+ * Unlike ->suspend this cannot abort on error: a controller left
+ * mastering the bus is worse than a truncated transfer.
+ */
+ ret = spi_controller_suspend(xqspi->ctlr);
+ if (ret)
+ dev_warn(&pdev->dev, "could not stop the queue: %d\n", ret);
+
+ /*
+ * Only a runtime suspended controller can be left alone: its clocks
+ * are gated, so it cannot be mastering the bus, and its registers
+ * must not be accessed either. Any other answer means it may be
+ * running and has to be stopped. In particular, on a kernel built
+ * without runtime PM this returns -EINVAL, and there the clocks
+ * enabled in probe() are never gated at all.
+ */
+ ret = pm_runtime_get_if_in_use(&pdev->dev);
+ if (!ret)
+ return;
+
+ zynqmp_gqspi_write(xqspi, GQSPI_EN_OFST, 0x0);
+
+ if (ret > 0)
+ pm_runtime_put_noidle(&pdev->dev);
+}
+
MODULE_DEVICE_TABLE(of, zynqmp_qspi_of_match);
static struct platform_driver zynqmp_qspi_driver = {
.probe = zynqmp_qspi_probe,
.remove_new = zynqmp_qspi_remove,
+ .shutdown = zynqmp_qspi_shutdown,
.driver = {
.name = "zynqmp-qspi",
.of_match_table = zynqmp_qspi_of_match,
diff --git a/drivers/staging/media/rkvdec/rkvdec.c b/drivers/staging/media/rkvdec/rkvdec.c
index a1d941b0be00..85dcabe11bf9 100644
--- a/drivers/staging/media/rkvdec/rkvdec.c
+++ b/drivers/staging/media/rkvdec/rkvdec.c
@@ -1044,8 +1044,8 @@ static int rkvdec_probe(struct platform_device *pdev)
vb2_dma_contig_set_max_seg_size(&pdev->dev, DMA_BIT_MASK(32));
irq = platform_get_irq(pdev, 0);
- if (irq <= 0)
- return -ENXIO;
+ if (irq < 0)
+ return irq;
ret = devm_request_threaded_irq(&pdev->dev, irq, NULL,
rkvdec_irq_handler, IRQF_ONESHOT,
diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme.c b/drivers/staging/rtl8723bs/core/rtw_mlme.c
index d4b6e748f11f..506ba55f15b7 100644
--- a/drivers/staging/rtl8723bs/core/rtw_mlme.c
+++ b/drivers/staging/rtl8723bs/core/rtw_mlme.c
@@ -1950,7 +1950,10 @@ int rtw_restruct_wmm_ie(struct adapter *adapter, u8 *in_ie, u8 *out_ie, uint in_
break;
}
- i += (in_ie[i+1]+2); /* to the next IE element */
+ if (i + 1 >= in_len)
+ break;
+
+ i += (in_ie[i + 1] + 2); /* to the next IE element */
}
return ielength;
diff --git a/drivers/staging/sm750fb/sm750.c b/drivers/staging/sm750fb/sm750.c
index 1452edab96cc..2d9eab9ad117 100644
--- a/drivers/staging/sm750fb/sm750.c
+++ b/drivers/staging/sm750fb/sm750.c
@@ -275,7 +275,7 @@ static void lynxfb_ops_imageblit(struct fb_info *info,
spin_lock(&sm750_dev->slock);
sm750_dev->accel.de_imageblit(&sm750_dev->accel,
- image->data, image->width >> 3, 0,
+ image->data, 0,
base, pitch, Bpp,
image->dx, image->dy,
image->width, image->height,
diff --git a/drivers/staging/sm750fb/sm750.h b/drivers/staging/sm750fb/sm750.h
index aff69661c8e6..a693d9f1d0d5 100644
--- a/drivers/staging/sm750fb/sm750.h
+++ b/drivers/staging/sm750fb/sm750.h
@@ -73,7 +73,7 @@ struct lynx_accel {
u32 rop2);
int (*de_imageblit)(struct lynx_accel *accel, const char *p_srcbuf,
- u32 src_delta, u32 start_bit, u32 d_base, u32 d_pitch,
+ u32 start_bit, u32 d_base, u32 d_pitch,
u32 byte_per_pixel, u32 dx, u32 dy, u32 width,
u32 height, u32 f_color, u32 b_color, u32 rop2);
diff --git a/drivers/staging/sm750fb/sm750_accel.c b/drivers/staging/sm750fb/sm750_accel.c
index 44b9e3fe3a41..37217a496ba7 100644
--- a/drivers/staging/sm750fb/sm750_accel.c
+++ b/drivers/staging/sm750fb/sm750_accel.c
@@ -300,8 +300,6 @@ static unsigned int deGetTransparency(struct lynx_accel *accel)
* sm750_hw_imageblit
* @accel: Acceleration device data
* @pSrcbuf: pointer to start of source buffer in system memory
- * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down
- * and -ive mean button up
* @startBit: Mono data can start at any bit in a byte, this value should be
* 0 to 7
* @dBase: Address of destination: offset in frame buffer
@@ -316,7 +314,7 @@ static unsigned int deGetTransparency(struct lynx_accel *accel)
* @rop2: ROP value
*/
int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf,
- u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch,
+ u32 startBit, u32 dBase, u32 dPitch,
u32 bytePerPixel, u32 dx, u32 dy, u32 width,
u32 height, u32 fColor, u32 bColor, u32 rop2)
{
@@ -405,7 +403,7 @@ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf,
write_dpPort(accel, *(unsigned int *)ajRemain);
}
- pSrcbuf += srcDelta;
+ pSrcbuf += ulBytesPerScan;
}
return 0;
diff --git a/drivers/staging/sm750fb/sm750_accel.h b/drivers/staging/sm750fb/sm750_accel.h
index 2c79cb730a0a..df48ed57a054 100644
--- a/drivers/staging/sm750fb/sm750_accel.h
+++ b/drivers/staging/sm750fb/sm750_accel.h
@@ -220,8 +220,6 @@ int sm750_hw_copyarea(struct lynx_accel *accel,
/**
* sm750_hw_imageblit
* @pSrcbuf: pointer to start of source buffer in system memory
- * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down
- *>----- and -ive mean button up
* @startBit: Mono data can start at any bit in a byte, this value should be
*>----- 0 to 7
* @dBase: Address of destination: offset in frame buffer
@@ -236,7 +234,7 @@ int sm750_hw_copyarea(struct lynx_accel *accel,
* @rop2: ROP value
*/
int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf,
- u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch,
+ u32 startBit, u32 dBase, u32 dPitch,
u32 bytePerPixel, u32 dx, u32 dy, u32 width,
u32 height, u32 fColor, u32 bColor, u32 rop2);
diff --git a/drivers/thermal/gov_step_wise.c b/drivers/thermal/gov_step_wise.c
index b038f042ed74..b191e11e553b 100644
--- a/drivers/thermal/gov_step_wise.c
+++ b/drivers/thermal/gov_step_wise.c
@@ -65,14 +65,12 @@ static unsigned long get_target_state(struct thermal_instance *instance,
min(instance->lower + 1, instance->upper),
instance->upper);
} else if (trend == THERMAL_TREND_DROPPING) {
- if (cur_state <= instance->lower)
- return THERMAL_NO_TARGET;
-
/*
- * If 'throttle' is false, no mitigation is necessary, so
- * request the lower state for this instance.
+ * If 'throttle' is false, no mitigation is necessary and
+ * passive polling is already deactivated, so clear this
+ * instance state by returning THERMAL_NO_TARGET.
*/
- return instance->lower;
+ return THERMAL_NO_TARGET;
}
return instance->target;
diff --git a/drivers/usb/cdns3/cdns3-gadget.c b/drivers/usb/cdns3/cdns3-gadget.c
index ac0aeb135d6f..bc1157a4df00 100644
--- a/drivers/usb/cdns3/cdns3-gadget.c
+++ b/drivers/usb/cdns3/cdns3-gadget.c
@@ -3484,7 +3484,7 @@ __must_hold(&cdns->lock)
return 0;
}
-static int cdns3_gadget_resume(struct cdns *cdns, bool hibernated)
+static int cdns3_gadget_resume(struct cdns *cdns, bool lost_power)
{
struct cdns3_device *priv_dev = cdns->gadget_dev;
@@ -3492,7 +3492,7 @@ static int cdns3_gadget_resume(struct cdns *cdns, bool hibernated)
return 0;
cdns3_gadget_config(priv_dev);
- if (hibernated)
+ if (lost_power)
writel(USB_CONF_DEVEN, &priv_dev->regs->usb_conf);
return 0;
diff --git a/drivers/usb/cdns3/cdnsp-gadget.c b/drivers/usb/cdns3/cdnsp-gadget.c
index 89d3d8167e59..0fe1f9cc04db 100644
--- a/drivers/usb/cdns3/cdnsp-gadget.c
+++ b/drivers/usb/cdns3/cdnsp-gadget.c
@@ -1819,6 +1819,82 @@ static void cdnsp_get_rev_cap(struct cdnsp_device *pdev)
readl(&pdev->rev_cap->tx_buff_size));
}
+static void cdnsp_set_event_deq(struct cdnsp_device *pdev)
+{
+ dma_addr_t deq;
+ u64 temp;
+
+ deq = cdnsp_trb_virt_to_dma(pdev->event_ring->deq_seg,
+ pdev->event_ring->dequeue);
+
+ /* Update controller event ring dequeue pointer */
+ temp = cdnsp_read_64(&pdev->ir_set->erst_dequeue);
+ temp &= ERST_PTR_MASK;
+
+ /*
+ * Don't clear the EHB bit (which is RW1C) because
+ * there might be more events to service.
+ */
+ temp &= ~ERST_EHB;
+
+ cdnsp_write_64(((u64)deq & (u64)~ERST_PTR_MASK) | temp,
+ &pdev->ir_set->erst_dequeue);
+}
+
+static void cdnsp_add_interrupter(struct cdnsp_device *pdev)
+{
+ u64 erst_base;
+ u32 erst_size;
+
+ /* Set ERST count with the number of entries in the segment table. */
+ erst_size = readl(&pdev->ir_set->erst_size);
+ erst_size &= ERST_SIZE_MASK;
+ erst_size |= ERST_NUM_SEGS;
+ writel(erst_size, &pdev->ir_set->erst_size);
+
+ /* Set the segment table base address. */
+ erst_base = cdnsp_read_64(&pdev->ir_set->erst_base);
+ erst_base &= ERST_PTR_MASK;
+ erst_base |= (pdev->erst.erst_dma_addr & (u64)~ERST_PTR_MASK);
+ cdnsp_write_64(erst_base, &pdev->ir_set->erst_base);
+
+ /* Set the event ring dequeue address. */
+ cdnsp_set_event_deq(pdev);
+}
+
+/* Set up basic CDNSP registers */
+static void cdnsp_init(struct cdnsp_device *pdev)
+{
+ unsigned int val;
+ u64 val_64;
+
+ val = readl(&pdev->op_regs->config_reg);
+ val |= ((val & ~MAX_DEVS) | CDNSP_DEV_MAX_SLOTS) | CONFIG_U3E;
+ writel(val, &pdev->op_regs->config_reg);
+
+ /* Initialize the Command ring */
+ cdnsp_ring_init(pdev, pdev->cmd_ring);
+
+ /* Set the address in the Command Ring Control register */
+ val_64 = cdnsp_read_64(&pdev->op_regs->cmd_ring);
+ val_64 = (val_64 & (u64)CMD_RING_RSVD_BITS) |
+ (pdev->cmd_ring->first_seg->dma & (u64)~CMD_RING_RSVD_BITS) |
+ pdev->cmd_ring->cycle_state;
+ cdnsp_write_64(val_64, &pdev->op_regs->cmd_ring);
+
+ /* Set Device Context Base Address Array pointer */
+ cdnsp_write_64(pdev->dcbaa->dma, &pdev->op_regs->dcbaa_ptr);
+
+ /* Set Doorbell array pointer */
+ val = readl(&pdev->cap_regs->db_off);
+ val &= DBOFF_MASK;
+ pdev->dba = (void __iomem *)pdev->cap_regs + val;
+
+ /* Initialize the Primary interrupter */
+ cdnsp_ring_init(pdev, pdev->event_ring);
+ cdnsp_add_interrupter(pdev);
+}
+
static int cdnsp_gen_setup(struct cdnsp_device *pdev)
{
int ret;
@@ -1884,6 +1960,8 @@ static int cdnsp_gen_setup(struct cdnsp_device *pdev)
if (ret)
return ret;
+ cdnsp_init(pdev);
+
/*
* Software workaround for U1: after transition
* to U1 the controller starts gating clock, and in some cases,
@@ -2014,9 +2092,6 @@ static int cdnsp_gadget_suspend(struct cdns *cdns, bool do_wakeup)
struct cdnsp_device *pdev = cdns->gadget_dev;
unsigned long flags;
- if (pdev->link_state == XDEV_U3)
- return 0;
-
spin_lock_irqsave(&pdev->lock, flags);
cdnsp_disconnect_gadget(pdev);
cdnsp_stop(pdev);
@@ -2025,17 +2100,43 @@ static int cdnsp_gadget_suspend(struct cdns *cdns, bool do_wakeup)
return 0;
}
-static int cdnsp_gadget_resume(struct cdns *cdns, bool hibernated)
+static int cdnsp_gadget_resume(struct cdns *cdns, bool lost_power)
{
struct cdnsp_device *pdev = cdns->gadget_dev;
enum usb_device_speed max_speed;
unsigned long flags;
+ bool context_lost;
+ u32 val;
int ret;
if (!pdev->gadget_driver)
return 0;
spin_lock_irqsave(&pdev->lock, flags);
+ val = readl(&pdev->port3x_regs->mode_2);
+ context_lost = !!(val & CFG_3XPORT_U1_PIPE_CLK_GATE_EN) || lost_power;
+
+ if (context_lost) {
+ cdnsp_halt(pdev);
+ cdnsp_set_apb_timeout_value(pdev);
+
+ /* Reset the internal controller memory state and registers. */
+ ret = cdnsp_reset(pdev);
+ if (ret)
+ goto unlock;
+
+ val = readl(&pdev->port3x_regs->mode_2);
+ val &= ~CFG_3XPORT_U1_PIPE_CLK_GATE_EN;
+ writel(val, &pdev->port3x_regs->mode_2);
+
+ cdnsp_clear_cmd_ring(pdev);
+
+ memset(pdev->event_ring->first_seg->trbs, 0,
+ sizeof(union cdnsp_trb) * (TRBS_PER_SEGMENT));
+
+ cdnsp_init(pdev);
+ }
+
max_speed = pdev->gadget_driver->max_speed;
/* Limit speed if necessary. */
@@ -2043,9 +2144,10 @@ static int cdnsp_gadget_resume(struct cdns *cdns, bool hibernated)
ret = cdnsp_run(pdev, max_speed);
- if (pdev->link_state == XDEV_U3)
+ if (!context_lost && pdev->link_state == XDEV_U3)
__cdnsp_gadget_wakeup(pdev);
+unlock:
spin_unlock_irqrestore(&pdev->lock, flags);
return ret;
diff --git a/drivers/usb/cdns3/cdnsp-gadget.h b/drivers/usb/cdns3/cdnsp-gadget.h
index a91cca509db0..c157023c9376 100644
--- a/drivers/usb/cdns3/cdnsp-gadget.h
+++ b/drivers/usb/cdns3/cdnsp-gadget.h
@@ -1509,6 +1509,7 @@ int cdnsp_endpoint_init(struct cdnsp_device *pdev,
int cdnsp_ring_expansion(struct cdnsp_device *pdev,
struct cdnsp_ring *ring,
unsigned int num_trbs, gfp_t flags);
+void cdnsp_ring_init(struct cdnsp_device *pdev, struct cdnsp_ring *ring);
struct cdnsp_ring *cdnsp_dma_to_transfer_ring(struct cdnsp_ep *ep, u64 address);
int cdnsp_alloc_stream_info(struct cdnsp_device *pdev,
struct cdnsp_ep *pep,
diff --git a/drivers/usb/cdns3/cdnsp-mem.c b/drivers/usb/cdns3/cdnsp-mem.c
index 3f35c8af271e..4b66307baaeb 100644
--- a/drivers/usb/cdns3/cdnsp-mem.c
+++ b/drivers/usb/cdns3/cdnsp-mem.c
@@ -394,13 +394,6 @@ static struct cdnsp_ring *cdnsp_ring_alloc(struct cdnsp_device *pdev,
if (ret)
goto fail;
- /* Only event ring does not use link TRB. */
- if (type != TYPE_EVENT)
- ring->last_seg->trbs[TRBS_PER_SEGMENT - 1].link.control |=
- cpu_to_le32(LINK_TOGGLE);
-
- cdnsp_initialize_ring_info(ring);
- trace_cdnsp_ring_alloc(ring);
return ring;
fail:
kfree(ring);
@@ -604,6 +597,7 @@ int cdnsp_alloc_stream_info(struct cdnsp_device *pdev,
if (!cur_ring)
goto cleanup_rings;
+ cdnsp_ring_init(pdev, cur_ring);
cur_ring->stream_id = cur_stream;
cur_ring->trb_address_map = &stream_info->trb_address_map;
@@ -699,6 +693,8 @@ static int cdnsp_alloc_priv_device(struct cdnsp_device *pdev)
if (!pdev->eps[0].ring)
goto fail;
+ cdnsp_ring_init(pdev, pdev->eps[0].ring);
+
/* Point to output device context in dcbaa. */
pdev->dcbaa->dev_context_ptrs[1] = cpu_to_le64(pdev->out_ctx.dma);
pdev->cmd.in_ctx = &pdev->in_ctx;
@@ -992,6 +988,8 @@ int cdnsp_endpoint_init(struct cdnsp_device *pdev,
if (!pep->ring)
return -ENOMEM;
+ cdnsp_ring_init(pdev, pep->ring);
+
pep->skip = false;
/* Fill the endpoint context */
@@ -1099,28 +1097,6 @@ void cdnsp_mem_cleanup(struct cdnsp_device *pdev)
pdev->active_port = NULL;
}
-static void cdnsp_set_event_deq(struct cdnsp_device *pdev)
-{
- dma_addr_t deq;
- u64 temp;
-
- deq = cdnsp_trb_virt_to_dma(pdev->event_ring->deq_seg,
- pdev->event_ring->dequeue);
-
- /* Update controller event ring dequeue pointer */
- temp = cdnsp_read_64(&pdev->ir_set->erst_dequeue);
- temp &= ERST_PTR_MASK;
-
- /*
- * Don't clear the EHB bit (which is RW1C) because
- * there might be more events to service.
- */
- temp &= ~ERST_EHB;
-
- cdnsp_write_64(((u64)deq & (u64)~ERST_PTR_MASK) | temp,
- &pdev->ir_set->erst_dequeue);
-}
-
static void cdnsp_add_in_port(struct cdnsp_device *pdev,
struct cdnsp_port *port,
__le32 __iomem *addr)
@@ -1200,6 +1176,36 @@ static int cdnsp_setup_port_arrays(struct cdnsp_device *pdev)
return 0;
}
+static void cdnsp_initialize_ring_segments(struct cdnsp_device *pdev, struct cdnsp_ring *ring)
+{
+ struct cdnsp_segment *seg;
+
+ /* Only event ring does not use link TRB. */
+ if (ring->type == TYPE_EVENT)
+ return;
+
+ seg = ring->first_seg;
+
+ while (seg) {
+ struct cdnsp_segment *next = seg->next;
+
+ cdnsp_link_segments(pdev, seg, next, ring->type);
+ if (next == ring->first_seg)
+ break;
+
+ seg = next;
+ }
+
+ ring->last_seg->trbs[TRBS_PER_SEGMENT - 1].link.control |= cpu_to_le32(LINK_TOGGLE);
+}
+
+void cdnsp_ring_init(struct cdnsp_device *pdev, struct cdnsp_ring *ring)
+{
+ cdnsp_initialize_ring_segments(pdev, ring);
+ cdnsp_initialize_ring_info(ring);
+ trace_cdnsp_ring_alloc(ring);
+}
+
/*
* Initialize memory for CDNSP (one-time init).
*
@@ -1211,10 +1217,8 @@ int cdnsp_mem_init(struct cdnsp_device *pdev)
{
struct device *dev = pdev->dev;
int ret = -ENOMEM;
- unsigned int val;
dma_addr_t dma;
u32 page_size;
- u64 val_64;
/*
* Use 4K pages, since that's common and the minimum the
@@ -1222,10 +1226,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev)
*/
page_size = 1 << 12;
- val = readl(&pdev->op_regs->config_reg);
- val |= ((val & ~MAX_DEVS) | CDNSP_DEV_MAX_SLOTS) | CONFIG_U3E;
- writel(val, &pdev->op_regs->config_reg);
-
/*
* Doorbell array must be physically contiguous
* and 64-byte (cache line) aligned.
@@ -1237,8 +1237,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev)
pdev->dcbaa->dma = dma;
- cdnsp_write_64(dma, &pdev->op_regs->dcbaa_ptr);
-
/*
* Initialize the ring segment pool. The ring must be a contiguous
* structure comprised of TRBs. The TRBs must be 16 byte aligned,
@@ -1264,17 +1262,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev)
if (!pdev->cmd_ring)
goto destroy_device_pool;
- /* Set the address in the Command Ring Control register */
- val_64 = cdnsp_read_64(&pdev->op_regs->cmd_ring);
- val_64 = (val_64 & (u64)CMD_RING_RSVD_BITS) |
- (pdev->cmd_ring->first_seg->dma & (u64)~CMD_RING_RSVD_BITS) |
- pdev->cmd_ring->cycle_state;
- cdnsp_write_64(val_64, &pdev->op_regs->cmd_ring);
-
- val = readl(&pdev->cap_regs->db_off);
- val &= DBOFF_MASK;
- pdev->dba = (void __iomem *)pdev->cap_regs + val;
-
/* Set ir_set to interrupt register set 0 */
pdev->ir_set = &pdev->run_regs->ir_set[0];
@@ -1291,21 +1278,6 @@ int cdnsp_mem_init(struct cdnsp_device *pdev)
if (ret)
goto free_event_ring;
- /* Set ERST count with the number of entries in the segment table. */
- val = readl(&pdev->ir_set->erst_size);
- val &= ERST_SIZE_MASK;
- val |= ERST_NUM_SEGS;
- writel(val, &pdev->ir_set->erst_size);
-
- /* Set the segment table base address. */
- val_64 = cdnsp_read_64(&pdev->ir_set->erst_base);
- val_64 &= ERST_PTR_MASK;
- val_64 |= (pdev->erst.erst_dma_addr & (u64)~ERST_PTR_MASK);
- cdnsp_write_64(val_64, &pdev->ir_set->erst_base);
-
- /* Set the event ring dequeue address. */
- cdnsp_set_event_deq(pdev);
-
ret = cdnsp_setup_port_arrays(pdev);
if (ret)
goto free_erst;
diff --git a/drivers/usb/cdns3/core.h b/drivers/usb/cdns3/core.h
index ac30ee21309d..801be9e61340 100644
--- a/drivers/usb/cdns3/core.h
+++ b/drivers/usb/cdns3/core.h
@@ -30,7 +30,7 @@ struct cdns_role_driver {
int (*start)(struct cdns *cdns);
void (*stop)(struct cdns *cdns);
int (*suspend)(struct cdns *cdns, bool do_wakeup);
- int (*resume)(struct cdns *cdns, bool hibernated);
+ int (*resume)(struct cdns *cdns, bool lost_power);
const char *name;
#define CDNS_ROLE_STATE_INACTIVE 0
#define CDNS_ROLE_STATE_ACTIVE 1
diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index eb8b8f3e5216..7ee7228b31eb 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -742,6 +742,7 @@ struct dwc3_event_buffer {
*/
struct dwc3_ep {
struct usb_ep endpoint;
+ struct delayed_work nostream_work;
struct list_head cancelled_list;
struct list_head pending_list;
struct list_head started_list;
@@ -764,7 +765,7 @@ struct dwc3_ep {
#define DWC3_EP_WAIT_TRANSFER_COMPLETE BIT(7)
#define DWC3_EP_IGNORE_NEXT_NOSTREAM BIT(8)
#define DWC3_EP_FORCE_RESTART_STREAM BIT(9)
-#define DWC3_EP_FIRST_STREAM_PRIMED BIT(10)
+#define DWC3_EP_STREAM_PRIMED BIT(10)
#define DWC3_EP_PENDING_CLEAR_STALL BIT(11)
#define DWC3_EP_TXFIFO_RESIZED BIT(12)
#define DWC3_EP_DELAY_STOP BIT(13)
diff --git a/drivers/usb/dwc3/ep0.c b/drivers/usb/dwc3/ep0.c
index bcc9cc398068..5e4f35e57265 100644
--- a/drivers/usb/dwc3/ep0.c
+++ b/drivers/usb/dwc3/ep0.c
@@ -304,7 +304,7 @@ void dwc3_ep0_out_start(struct dwc3 *dwc)
dwc3_ep->flags &= ~DWC3_EP_DELAY_STOP;
if (dwc->connected)
- dwc3_stop_active_transfer(dwc3_ep, true, true);
+ dwc3_stop_active_transfer(dwc3_ep, false, true);
else
dwc3_remove_requests(dwc, dwc3_ep, -ESHUTDOWN);
}
diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index d2b07d767877..077e3ac2de46 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -991,12 +991,11 @@ static int __dwc3_gadget_ep_enable(struct dwc3_ep *dep, unsigned int action)
* controller to generate an ERDY to initiate the
* stream.
*/
- dwc3_stop_active_transfer(dep, true, true);
+ dwc3_stop_active_transfer(dep, false, true);
/*
* All stream eps will reinitiate stream on NoStream
- * rejection until we can determine that the host can
- * prime after the first transfer.
+ * rejection.
*
* However, if the controller is capable of
* TXF_FLUSH_BYPASS, then IN direction endpoints will
@@ -1020,7 +1019,7 @@ void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep, int status)
{
struct dwc3_request *req;
- dwc3_stop_active_transfer(dep, true, false);
+ dwc3_stop_active_transfer(dep, false, false);
/* If endxfer is delayed, avoid unmapping requests */
if (dep->flags & DWC3_EP_DELAY_STOP)
@@ -1710,7 +1709,7 @@ static int __dwc3_gadget_kick_transfer(struct dwc3_ep *dep)
if (ret == -EAGAIN)
return ret;
- dwc3_stop_active_transfer(dep, true, true);
+ dwc3_stop_active_transfer(dep, false, true);
list_for_each_entry_safe(req, tmp, &dep->started_list, list)
dwc3_gadget_move_cancelled_request(req, DWC3_REQUEST_STATUS_DEQUEUED);
@@ -1747,6 +1746,11 @@ static int __dwc3_gadget_get_frame(struct dwc3 *dwc)
* the controller won't update the TRB progress on command
* completion. It also won't clear the HWO bit in the TRB.
* The command will also not complete immediately in that case.
+ *
+ * Older programming guide revisions recommended setting ForceRM to 1
+ * when ending a transfer. Newer programming guide revisions now
+ * recommend keeping ForceRM cleared, and TRBs are properly updated
+ * on command completion.
*/
static int __dwc3_stop_active_transfer(struct dwc3_ep *dep, bool force, bool interrupt)
{
@@ -1872,7 +1876,7 @@ static int dwc3_gadget_start_isoc_quirk(struct dwc3_ep *dep)
* to wait for the next XferNotReady to test the command again
*/
if (cmd_status == 0) {
- dwc3_stop_active_transfer(dep, true, true);
+ dwc3_stop_active_transfer(dep, false, true);
return 0;
}
}
@@ -2155,7 +2159,7 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep,
struct dwc3_request *t;
/* wait until it is processed */
- dwc3_stop_active_transfer(dep, true, true);
+ dwc3_stop_active_transfer(dep, false, true);
/*
* Remove any started request if the transfer is
@@ -2232,7 +2236,7 @@ int __dwc3_gadget_ep_set_halt(struct dwc3_ep *dep, int value, int protocol)
return 0;
}
- dwc3_stop_active_transfer(dep, true, true);
+ dwc3_stop_active_transfer(dep, false, true);
list_for_each_entry_safe(req, tmp, &dep->started_list, list)
dwc3_gadget_move_cancelled_request(req, DWC3_REQUEST_STATUS_STALLED);
@@ -3309,6 +3313,50 @@ static int dwc3_gadget_init_out_endpoint(struct dwc3_ep *dep)
return dwc3_alloc_trb_pool(dep);
}
+#define nostream_work_to_dep(w) (container_of(to_delayed_work(w), struct dwc3_ep, nostream_work))
+static void dwc3_nostream_work(struct work_struct *work)
+{
+ struct dwc3_ep *dep = nostream_work_to_dep(work);
+ struct dwc3 *dwc = dep->dwc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&dwc->lock, flags);
+ if (dep->flags & DWC3_EP_STREAM_PRIMED)
+ goto out;
+
+ if ((dep->flags & DWC3_EP_IGNORE_NEXT_NOSTREAM) ||
+ (!DWC3_MST_CAPABLE(&dwc->hwparams) &&
+ !(dep->flags & DWC3_EP_WAIT_TRANSFER_COMPLETE)))
+ goto out;
+ /*
+ * If the host rejects a stream due to no active stream, by the
+ * USB and xHCI spec, the endpoint will be put back to idle
+ * state. When the host is ready (buffer added/updated), it will
+ * prime the endpoint to inform the usb device controller. This
+ * triggers the device controller to issue ERDY to restart the
+ * stream. However, some hosts don't follow this and keep the
+ * endpoint in the idle state. No prime will come despite host
+ * streams are updated, and the device controller will not be
+ * triggered to generate ERDY to move the next stream data. To
+ * workaround this and maintain compatibility with various
+ * hosts, force to reinitiate the stream until the host is ready
+ * instead of waiting for the host to prime the endpoint.
+ */
+ if (DWC3_VER_IS_WITHIN(DWC32, 100A, ANY)) {
+ unsigned int cmd = DWC3_DGCMD_SET_ENDPOINT_PRIME;
+
+ dwc3_send_gadget_generic_command(dwc, cmd, dep->number);
+ } else {
+ dep->flags |= DWC3_EP_DELAY_START;
+ dwc3_stop_active_transfer(dep, false, true);
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return;
+ }
+out:
+ dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM;
+ spin_unlock_irqrestore(&dwc->lock, flags);
+}
+
static int dwc3_gadget_init_endpoint(struct dwc3 *dwc, u8 epnum)
{
struct dwc3_ep *dep;
@@ -3354,6 +3402,7 @@ static int dwc3_gadget_init_endpoint(struct dwc3 *dwc, u8 epnum)
INIT_LIST_HEAD(&dep->pending_list);
INIT_LIST_HEAD(&dep->started_list);
INIT_LIST_HEAD(&dep->cancelled_list);
+ INIT_DELAYED_WORK(&dep->nostream_work, dwc3_nostream_work);
dwc3_debugfs_create_endpoint_dir(dep);
@@ -3401,6 +3450,7 @@ static void dwc3_gadget_free_endpoints(struct dwc3 *dwc)
}
dwc3_debugfs_remove_endpoint_dir(dep);
+ cancel_delayed_work_sync(&dep->nostream_work);
kfree(dep);
}
}
@@ -3643,7 +3693,7 @@ static bool dwc3_gadget_endpoint_trbs_complete(struct dwc3_ep *dep,
if (usb_endpoint_xfer_isoc(dep->endpoint.desc) &&
list_empty(&dep->started_list) &&
(list_empty(&dep->pending_list) || status == -EXDEV))
- dwc3_stop_active_transfer(dep, true, true);
+ dwc3_stop_active_transfer(dep, false, true);
else if (dwc3_gadget_ep_should_continue(dep))
if (__dwc3_gadget_kick_transfer(dep) == 0)
no_started_trb = false;
@@ -3789,66 +3839,27 @@ static void dwc3_gadget_endpoint_command_complete(struct dwc3_ep *dep,
static void dwc3_gadget_endpoint_stream_event(struct dwc3_ep *dep,
const struct dwc3_event_depevt *event)
{
- struct dwc3 *dwc = dep->dwc;
-
if (event->status == DEPEVT_STREAMEVT_FOUND) {
- dep->flags |= DWC3_EP_FIRST_STREAM_PRIMED;
- goto out;
+ cancel_delayed_work(&dep->nostream_work);
+ dep->flags |= DWC3_EP_STREAM_PRIMED;
+ dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM;
+ return;
}
/* Note: NoStream rejection event param value is 0 and not 0xFFFF */
switch (event->parameters) {
case DEPEVT_STREAM_PRIME:
- /*
- * If the host can properly transition the endpoint state from
- * idle to prime after a NoStream rejection, there's no need to
- * force restarting the endpoint to reinitiate the stream. To
- * simplify the check, assume the host follows the USB spec if
- * it primed the endpoint more than once.
- */
- if (dep->flags & DWC3_EP_FORCE_RESTART_STREAM) {
- if (dep->flags & DWC3_EP_FIRST_STREAM_PRIMED)
- dep->flags &= ~DWC3_EP_FORCE_RESTART_STREAM;
- else
- dep->flags |= DWC3_EP_FIRST_STREAM_PRIMED;
- }
-
+ cancel_delayed_work(&dep->nostream_work);
+ dep->flags |= DWC3_EP_STREAM_PRIMED;
+ dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM;
break;
case DEPEVT_STREAM_NOSTREAM:
- if ((dep->flags & DWC3_EP_IGNORE_NEXT_NOSTREAM) ||
- !(dep->flags & DWC3_EP_FORCE_RESTART_STREAM) ||
- (!DWC3_MST_CAPABLE(&dwc->hwparams) &&
- !(dep->flags & DWC3_EP_WAIT_TRANSFER_COMPLETE)))
- break;
-
- /*
- * If the host rejects a stream due to no active stream, by the
- * USB and xHCI spec, the endpoint will be put back to idle
- * state. When the host is ready (buffer added/updated), it will
- * prime the endpoint to inform the usb device controller. This
- * triggers the device controller to issue ERDY to restart the
- * stream. However, some hosts don't follow this and keep the
- * endpoint in the idle state. No prime will come despite host
- * streams are updated, and the device controller will not be
- * triggered to generate ERDY to move the next stream data. To
- * workaround this and maintain compatibility with various
- * hosts, force to reinitiate the stream until the host is ready
- * instead of waiting for the host to prime the endpoint.
- */
- if (DWC3_VER_IS_WITHIN(DWC32, 100A, ANY)) {
- unsigned int cmd = DWC3_DGCMD_SET_ENDPOINT_PRIME;
-
- dwc3_send_gadget_generic_command(dwc, cmd, dep->number);
- } else {
- dep->flags |= DWC3_EP_DELAY_START;
- dwc3_stop_active_transfer(dep, true, true);
- return;
- }
+ dep->flags &= ~DWC3_EP_STREAM_PRIMED;
+ if (dep->flags & DWC3_EP_FORCE_RESTART_STREAM)
+ queue_delayed_work(system_wq, &dep->nostream_work,
+ msecs_to_jiffies(100));
break;
}
-
-out:
- dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM;
}
static void dwc3_endpoint_interrupt(struct dwc3 *dwc,
diff --git a/drivers/usb/gadget/function/f_mass_storage.c b/drivers/usb/gadget/function/f_mass_storage.c
index 24908779bd0b..8b3deed44736 100644
--- a/drivers/usb/gadget/function/f_mass_storage.c
+++ b/drivers/usb/gadget/function/f_mass_storage.c
@@ -2748,6 +2748,9 @@ int fsg_common_set_num_buffers(struct fsg_common *common, unsigned int n)
struct fsg_buffhd *bh, *buffhds;
int i;
+ if (n < 2)
+ return -EINVAL;
+
buffhds = kcalloc(n, sizeof(*buffhds), GFP_KERNEL);
if (!buffhds)
return -ENOMEM;
diff --git a/drivers/usb/host/xhci-pci.c b/drivers/usb/host/xhci-pci.c
index 2c4dc54b0fca..2ef8ea1bc8d9 100644
--- a/drivers/usb/host/xhci-pci.c
+++ b/drivers/usb/host/xhci-pci.c
@@ -94,8 +94,7 @@
#define PCI_DEVICE_ID_ASMEDIA_3042_XHCI 0x3042
#define PCI_DEVICE_ID_ASMEDIA_3242_XHCI 0x3242
-#define PCI_DEVICE_ID_CADENCE 0x17CD
-#define PCI_DEVICE_ID_CADENCE_SSP 0x0200
+#define PCI_DEVICE_ID_CDNS_SSP 0x0200
static const char hcd_name[] = "xhci_hcd";
@@ -508,9 +507,8 @@ static void xhci_pci_quirks(struct device *dev, struct xhci_hcd *xhci)
if (pdev->device == 0x9203)
xhci->quirks |= XHCI_TRB_OVERFETCH;
}
-
- if (pdev->vendor == PCI_DEVICE_ID_CADENCE &&
- pdev->device == PCI_DEVICE_ID_CADENCE_SSP)
+ if (pdev->vendor == PCI_VENDOR_ID_CDNS &&
+ pdev->device == PCI_DEVICE_ID_CDNS_SSP)
xhci->quirks |= XHCI_CDNS_SCTX_QUIRK;
/* xHC spec requires PCI devices to support D3hot and D3cold */
diff --git a/drivers/usb/storage/realtek_cr.c b/drivers/usb/storage/realtek_cr.c
index a026c6cb6e68..de82d8260d72 100644
--- a/drivers/usb/storage/realtek_cr.c
+++ b/drivers/usb/storage/realtek_cr.c
@@ -915,7 +915,6 @@ static int realtek_cr_autosuspend_setup(struct us_data *us)
us->proto_handler = rts51x_invoke_transport;
chip->timer_expires = 0;
- timer_setup(&chip->rts51x_suspend_timer, rts51x_suspend_timer_fn, 0);
fw5895_init(us);
/* enable autosuspend function of the usb device */
@@ -933,10 +932,7 @@ static void realtek_cr_destructor(void *extra)
return;
#ifdef CONFIG_REALTEK_AUTOPM
- if (ss_en) {
- del_timer(&chip->rts51x_suspend_timer);
- chip->timer_expires = 0;
- }
+ timer_shutdown_sync(&chip->rts51x_suspend_timer);
#endif
kfree(chip->status);
}
@@ -981,6 +977,9 @@ static int init_realtek_cr(struct us_data *us)
us->extra = chip;
us->extra_destructor = realtek_cr_destructor;
+#ifdef CONFIG_REALTEK_AUTOPM
+ timer_setup(&chip->rts51x_suspend_timer, rts51x_suspend_timer_fn, 0);
+#endif
us->max_lun = chip->max_lun = rts51x_get_max_lun(us);
chip->us = us;
diff --git a/drivers/usb/typec/tcpm/tcpm.c b/drivers/usb/typec/tcpm/tcpm.c
index db0755350d06..db604b04e263 100644
--- a/drivers/usb/typec/tcpm/tcpm.c
+++ b/drivers/usb/typec/tcpm/tcpm.c
@@ -602,9 +602,12 @@ static const char * const pd_rev[] = {
(tcpm_cc_is_source((port)->cc2) && \
!tcpm_cc_is_source((port)->cc1)))
-#define tcpm_port_is_debug(port) \
+#define tcpm_port_is_debug_source(port) \
(tcpm_cc_is_source((port)->cc1) && tcpm_cc_is_source((port)->cc2))
+#define tcpm_port_is_debug(port) \
+ tcpm_port_is_debug_source(port)
+
#define tcpm_port_is_audio(port) \
(tcpm_cc_is_audio((port)->cc1) && tcpm_cc_is_audio((port)->cc2))
@@ -4742,7 +4745,7 @@ static void run_state_machine(struct tcpm_port *port)
tcpm_set_state(port, SNK_UNATTACHED, PD_T_DRP_SNK);
break;
case SRC_ATTACH_WAIT:
- if (tcpm_port_is_debug(port))
+ if (tcpm_port_is_debug_source(port))
tcpm_set_state(port, DEBUG_ACC_ATTACHED,
PD_T_CC_DEBOUNCE);
else if (tcpm_port_is_audio(port))
@@ -5868,7 +5871,7 @@ static void _tcpm_cc_change(struct tcpm_port *port, enum typec_cc_status cc1,
switch (port->state) {
case TOGGLING:
- if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) ||
+ if (tcpm_port_is_debug_source(port) || tcpm_port_is_audio(port) ||
tcpm_port_is_source(port))
tcpm_set_state(port, SRC_ATTACH_WAIT, 0);
else if (tcpm_port_is_sink(port))
@@ -5879,7 +5882,7 @@ static void _tcpm_cc_change(struct tcpm_port *port, enum typec_cc_status cc1,
break;
case SRC_UNATTACHED:
case ACC_UNATTACHED:
- if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) ||
+ if (tcpm_port_is_debug_source(port) || tcpm_port_is_audio(port) ||
tcpm_port_is_source(port))
tcpm_set_state(port, SRC_ATTACH_WAIT, 0);
break;
@@ -6382,16 +6385,32 @@ static void tcpm_pd_event_handler(struct kthread_work *work)
}
}
if (events & TCPM_SOURCING_VBUS) {
- tcpm_log(port, "sourcing vbus");
/*
* In fast role swap case TCPC autonomously sources vbus. Set vbus_source
- * true as TCPM wouldn't have called tcpm_set_vbus.
+ * true conditionally as TCPM wouldn't have called tcpm_set_vbus.
+ * If TCPM calls tcpm_set_vbus to source vbus, vbus_source would already
+ * be true.
*
- * When vbus is sourced on the command on TCPM i.e. TCPM called
- * tcpm_set_vbus to source vbus, vbus_source would already be true.
+ * When TCPM_FRS_EVENT and TCPM_SOURCING_VBUS arrive simultaneously,
+ * handling TCPM_FRS_EVENT above transitions the state to AMS_START
+ * with upcoming_state FR_SWAP_SEND.
*/
- port->vbus_source = true;
- _tcpm_pd_vbus_on(port);
+
+ if (tcpm_port_is_source(port) ||
+ tcpm_port_is_debug_source(port) ||
+ (port->state == AMS_START && port->upcoming_state == FR_SWAP_SEND) ||
+ port->state == FR_SWAP_SEND ||
+ port->state == FR_SWAP_SEND_TIMEOUT ||
+ port->state == FR_SWAP_SNK_SRC_TRANSITION_TO_OFF ||
+ port->state == FR_SWAP_SNK_SRC_NEW_SINK_READY ||
+ port->state == FR_SWAP_SNK_SRC_SOURCE_VBUS_APPLIED) {
+ tcpm_log(port, "sourcing vbus");
+ port->vbus_source = true;
+ _tcpm_pd_vbus_on(port);
+ } else {
+ tcpm_log(port, "Discarding sourcing vbus! Invalid state %s",
+ tcpm_states[port->state]);
+ }
}
if (events & TCPM_PORT_CLEAN) {
tcpm_log(port, "port clean");
diff --git a/drivers/video/screen_info_generic.c b/drivers/video/screen_info_generic.c
index 64117c6367ab..900e9386eceb 100644
--- a/drivers/video/screen_info_generic.c
+++ b/drivers/video/screen_info_generic.c
@@ -144,3 +144,39 @@ ssize_t screen_info_resources(const struct screen_info *si, struct resource *r,
return pos - r;
}
EXPORT_SYMBOL(screen_info_resources);
+
+/*
+ * The meaning of depth and bpp for direct-color formats is
+ * inconsistent:
+ *
+ * - DRM format info specifies depth as the number of color
+ * bits; including alpha, but not including filler bits.
+ * - Linux' EFI platform code computes lfb_depth from the
+ * individual color channels, including the reserved bits.
+ * - VBE 1.1 defines lfb_depth for XRGB1555 as 16, but later
+ * versions use 15.
+ * - On the kernel command line, 'bpp' of 32 is usually
+ * XRGB8888 including the filler bits, but 15 is XRGB1555
+ * not including the filler bit.
+ *
+ * It is not easily possible to fix this in struct screen_info,
+ * as this could break UAPI. The best solution is to compute
+ * bits_per_pixel from the color bits, reserved bits and
+ * reported lfb_depth, whichever is highest.
+ */
+
+u32 __screen_info_lfb_bits_per_pixel(const struct screen_info *si)
+{
+ u32 bits_per_pixel = si->lfb_depth;
+
+ if (bits_per_pixel > 8) {
+ bits_per_pixel = max(max3(si->red_size + si->red_pos,
+ si->green_size + si->green_pos,
+ si->blue_size + si->blue_pos),
+ si->rsvd_size + si->rsvd_pos);
+ bits_per_pixel = max_t(u32, bits_per_pixel, si->lfb_depth);
+ }
+
+ return bits_per_pixel;
+}
+EXPORT_SYMBOL(__screen_info_lfb_bits_per_pixel);
diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index b4422e8c2e93..f420fb5810de 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -65,7 +65,6 @@
#include <linux/platform_device.h>
#include <linux/pm.h>
#include <linux/slab.h>
-#include <linux/spinlock.h>
#include <linux/virtio.h>
#include <linux/virtio_config.h>
#include <uapi/linux/virtio_mmio.h>
@@ -89,21 +88,10 @@ struct virtio_mmio_device {
void __iomem *base;
unsigned long version;
- /* a list of queues so we can dispatch IRQs */
- spinlock_t lock;
- struct list_head virtqueues;
+ /* True if enable_irq_wake() succeeded for the shared IRQ. */
+ bool wake_irq_enabled;
};
-struct virtio_mmio_vq_info {
- /* the actual virtqueue */
- struct virtqueue *vq;
-
- /* the list node for the virtqueues list */
- struct list_head node;
-};
-
-
-
/* Configuration interface */
static u64 vm_get_features(struct virtio_device *vdev)
@@ -300,9 +288,8 @@ static bool vm_notify_with_data(struct virtqueue *vq)
static irqreturn_t vm_interrupt(int irq, void *opaque)
{
struct virtio_mmio_device *vm_dev = opaque;
- struct virtio_mmio_vq_info *info;
+ struct virtqueue *vq;
unsigned long status;
- unsigned long flags;
irqreturn_t ret = IRQ_NONE;
/* Read and acknowledge interrupts */
@@ -315,10 +302,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque)
}
if (likely(status & VIRTIO_MMIO_INT_VRING)) {
- spin_lock_irqsave(&vm_dev->lock, flags);
- list_for_each_entry(info, &vm_dev->virtqueues, node)
- ret |= vring_interrupt(irq, info->vq);
- spin_unlock_irqrestore(&vm_dev->lock, flags);
+ virtio_device_for_each_vq(&vm_dev->vdev, vq)
+ ret |= vring_interrupt(irq, vq);
}
return ret;
@@ -329,14 +314,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque)
static void vm_del_vq(struct virtqueue *vq)
{
struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vq->vdev);
- struct virtio_mmio_vq_info *info = vq->priv;
- unsigned long flags;
unsigned int index = vq->index;
- spin_lock_irqsave(&vm_dev->lock, flags);
- list_del(&info->node);
- spin_unlock_irqrestore(&vm_dev->lock, flags);
-
/* Select and deactivate the queue */
writel(index, vm_dev->base + VIRTIO_MMIO_QUEUE_SEL);
if (vm_dev->version == 1) {
@@ -347,19 +326,23 @@ static void vm_del_vq(struct virtqueue *vq)
}
vring_del_virtqueue(vq);
-
- kfree(info);
}
static void vm_del_vqs(struct virtio_device *vdev)
{
struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
struct virtqueue *vq, *n;
+ int irq = platform_get_irq(vm_dev->pdev, 0);
list_for_each_entry_safe(vq, n, &vdev->vqs, list)
vm_del_vq(vq);
- free_irq(platform_get_irq(vm_dev->pdev, 0), vm_dev);
+ if (vm_dev->wake_irq_enabled) {
+ disable_irq_wake(irq);
+ vm_dev->wake_irq_enabled = false;
+ }
+
+ free_irq(irq, vm_dev);
}
static void vm_synchronize_cbs(struct virtio_device *vdev)
@@ -375,9 +358,7 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
{
struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
bool (*notify)(struct virtqueue *vq);
- struct virtio_mmio_vq_info *info;
struct virtqueue *vq;
- unsigned long flags;
unsigned int num;
int err;
@@ -399,13 +380,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
goto error_available;
}
- /* Allocate and fill out our active queue description */
- info = kmalloc(sizeof(*info), GFP_KERNEL);
- if (!info) {
- err = -ENOMEM;
- goto error_kmalloc;
- }
-
num = readl(vm_dev->base + VIRTIO_MMIO_QUEUE_NUM_MAX);
if (num == 0) {
err = -ENOENT;
@@ -463,13 +437,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
writel(1, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);
}
- vq->priv = info;
- info->vq = vq;
-
- spin_lock_irqsave(&vm_dev->lock, flags);
- list_add(&info->node, &vm_dev->virtqueues);
- spin_unlock_irqrestore(&vm_dev->lock, flags);
-
return vq;
error_bad_pfn:
@@ -481,8 +448,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);
WARN_ON(readl(vm_dev->base + VIRTIO_MMIO_QUEUE_READY));
}
- kfree(info);
-error_kmalloc:
error_available:
return ERR_PTR(err);
}
@@ -504,8 +469,9 @@ static int vm_find_vqs(struct virtio_device *vdev, unsigned int nvqs,
if (err)
return err;
- if (of_property_read_bool(vm_dev->pdev->dev.of_node, "wakeup-source"))
- enable_irq_wake(irq);
+ if (of_property_read_bool(vm_dev->pdev->dev.of_node, "wakeup-source") &&
+ !enable_irq_wake(irq))
+ vm_dev->wake_irq_enabled = true;
for (i = 0; i < nvqs; ++i) {
struct virtqueue_info *vqi = &vqs_info[i];
@@ -627,8 +593,6 @@ static int virtio_mmio_probe(struct platform_device *pdev)
vm_dev->vdev.dev.release = virtio_mmio_release_dev;
vm_dev->vdev.config = &virtio_mmio_config_ops;
vm_dev->pdev = pdev;
- INIT_LIST_HEAD(&vm_dev->virtqueues);
- spin_lock_init(&vm_dev->lock);
vm_dev->base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(vm_dev->base)) {
diff --git a/drivers/watchdog/da9063_wdt.c b/drivers/watchdog/da9063_wdt.c
index 684667469b10..15417d582f3b 100644
--- a/drivers/watchdog/da9063_wdt.c
+++ b/drivers/watchdog/da9063_wdt.c
@@ -271,7 +271,7 @@ static int __maybe_unused da9063_wdt_suspend(struct device *dev)
if (!use_sw_pm)
return 0;
- if (watchdog_active(wdd))
+ if (watchdog_active(wdd) || watchdog_hw_running(wdd))
return da9063_wdt_stop(wdd);
return 0;
@@ -284,7 +284,7 @@ static int __maybe_unused da9063_wdt_resume(struct device *dev)
if (!use_sw_pm)
return 0;
- if (watchdog_active(wdd))
+ if (watchdog_active(wdd) || watchdog_hw_running(wdd))
return da9063_wdt_start(wdd);
return 0;
diff --git a/drivers/watchdog/digicolor_wdt.c b/drivers/watchdog/digicolor_wdt.c
index 073d37867f47..de1a3267a972 100644
--- a/drivers/watchdog/digicolor_wdt.c
+++ b/drivers/watchdog/digicolor_wdt.c
@@ -25,6 +25,7 @@ struct dc_wdt {
void __iomem *base;
struct clk *clk;
spinlock_t lock;
+ unsigned long rate;
};
static unsigned timeout;
@@ -61,7 +62,7 @@ static int dc_wdt_start(struct watchdog_device *wdog)
{
struct dc_wdt *wdt = watchdog_get_drvdata(wdog);
- dc_wdt_set(wdt, wdog->timeout * clk_get_rate(wdt->clk));
+ dc_wdt_set(wdt, wdog->timeout * wdt->rate);
return 0;
}
@@ -79,7 +80,7 @@ static int dc_wdt_set_timeout(struct watchdog_device *wdog, unsigned int t)
{
struct dc_wdt *wdt = watchdog_get_drvdata(wdog);
- dc_wdt_set(wdt, t * clk_get_rate(wdt->clk));
+ dc_wdt_set(wdt, t * wdt->rate);
wdog->timeout = t;
return 0;
@@ -90,7 +91,7 @@ static unsigned int dc_wdt_get_timeleft(struct watchdog_device *wdog)
struct dc_wdt *wdt = watchdog_get_drvdata(wdog);
uint32_t count = readl_relaxed(wdt->base + TIMER_A_COUNT);
- return count / clk_get_rate(wdt->clk);
+ return count / wdt->rate;
}
static const struct watchdog_ops dc_wdt_ops = {
@@ -130,7 +131,11 @@ static int dc_wdt_probe(struct platform_device *pdev)
wdt->clk = devm_clk_get(dev, NULL);
if (IS_ERR(wdt->clk))
return PTR_ERR(wdt->clk);
- dc_wdt_wdd.max_timeout = U32_MAX / clk_get_rate(wdt->clk);
+
+ wdt->rate = clk_get_rate(wdt->clk);
+ if (!wdt->rate)
+ return -EINVAL;
+ dc_wdt_wdd.max_timeout = U32_MAX / wdt->rate;
dc_wdt_wdd.timeout = dc_wdt_wdd.max_timeout;
dc_wdt_wdd.parent = dev;
diff --git a/drivers/watchdog/msc313e_wdt.c b/drivers/watchdog/msc313e_wdt.c
index eea26c6c95d8..8aba7de823b5 100644
--- a/drivers/watchdog/msc313e_wdt.c
+++ b/drivers/watchdog/msc313e_wdt.c
@@ -47,6 +47,9 @@ static void msc313e_wdt_set_hw_timeout(struct msc313e_wdt_priv *priv,
{
u32 t = timeout * clk_get_rate(priv->clk);
+ /* Clear before to prevent premature reset during non-atomic updates. */
+ writew(1, priv->base + REG_WDT_CLR);
+
writew(t & 0xffff, priv->base + REG_WDT_MAX_PRD_L);
writew((t >> 16) & 0xffff, priv->base + REG_WDT_MAX_PRD_H);
writew(1, priv->base + REG_WDT_CLR);
@@ -77,6 +80,9 @@ static int msc313e_wdt_stop(struct watchdog_device *wdev)
{
struct msc313e_wdt_priv *priv = watchdog_get_drvdata(wdev);
+ /* Clear before to prevent premature reset during non-atomic updates. */
+ writew(1, priv->base + REG_WDT_CLR);
+
writew(0, priv->base + REG_WDT_MAX_PRD_L);
writew(0, priv->base + REG_WDT_MAX_PRD_H);
writew(0, priv->base + REG_WDT_CLR);
@@ -192,11 +198,15 @@ static int __maybe_unused msc313e_wdt_suspend(struct device *dev)
static int __maybe_unused msc313e_wdt_resume(struct device *dev)
{
struct msc313e_wdt_priv *priv = dev_get_drvdata(dev);
+ int ret = 0;
- if (watchdog_active(&priv->wdev) || watchdog_hw_running(&priv->wdev))
- msc313e_wdt_start(&priv->wdev);
+ if (watchdog_active(&priv->wdev) || watchdog_hw_running(&priv->wdev)) {
+ ret = msc313e_wdt_start(&priv->wdev);
+ if (ret)
+ dev_err(dev, "Failed to restart watchdog (err=%d)\n", ret);
+ }
- return 0;
+ return ret;
}
static SIMPLE_DEV_PM_OPS(msc313e_wdt_pm_ops, msc313e_wdt_suspend, msc313e_wdt_resume);
diff --git a/drivers/watchdog/rtd119x_wdt.c b/drivers/watchdog/rtd119x_wdt.c
index 984905695dde..0bfadb58917b 100644
--- a/drivers/watchdog/rtd119x_wdt.c
+++ b/drivers/watchdog/rtd119x_wdt.c
@@ -98,6 +98,7 @@ static int rtd119x_wdt_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct rtd119x_watchdog_device *data;
+ unsigned long rate;
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
if (!data)
@@ -111,10 +112,14 @@ static int rtd119x_wdt_probe(struct platform_device *pdev)
if (IS_ERR(data->clk))
return PTR_ERR(data->clk);
+ rate = clk_get_rate(data->clk);
+ if (!rate)
+ return -EINVAL;
+
data->wdt_dev.info = &rtd119x_wdt_info;
data->wdt_dev.ops = &rtd119x_wdt_ops;
data->wdt_dev.timeout = 120;
- data->wdt_dev.max_timeout = 0xffffffff / clk_get_rate(data->clk);
+ data->wdt_dev.max_timeout = 0xffffffff / rate;
data->wdt_dev.min_timeout = 1;
data->wdt_dev.parent = dev;
diff --git a/drivers/watchdog/sp5100_tco.c b/drivers/watchdog/sp5100_tco.c
index 7e99c3b1f367..72ad59649ccc 100644
--- a/drivers/watchdog/sp5100_tco.c
+++ b/drivers/watchdog/sp5100_tco.c
@@ -605,8 +605,10 @@ static int __init sp5100_tco_init(void)
pr_info("SP5100/SB800 TCO WatchDog Timer Driver\n");
err = platform_driver_register(&sp5100_tco_driver);
- if (err)
+ if (err) {
+ pci_dev_put(sp5100_tco_pci);
return err;
+ }
sp5100_tco_platform_device =
platform_device_register_simple(TCO_DRIVER_NAME, -1, NULL, 0);
@@ -619,6 +621,7 @@ static int __init sp5100_tco_init(void)
unreg_platform_driver:
platform_driver_unregister(&sp5100_tco_driver);
+ pci_dev_put(sp5100_tco_pci);
return err;
}
@@ -626,6 +629,7 @@ static void __exit sp5100_tco_exit(void)
{
platform_device_unregister(sp5100_tco_platform_device);
platform_driver_unregister(&sp5100_tco_driver);
+ pci_dev_put(sp5100_tco_pci);
}
module_init(sp5100_tco_init);
diff --git a/drivers/watchdog/starfive-wdt.c b/drivers/watchdog/starfive-wdt.c
index 8244f282bee8..2f7a9b7f2ef4 100644
--- a/drivers/watchdog/starfive-wdt.c
+++ b/drivers/watchdog/starfive-wdt.c
@@ -371,7 +371,7 @@ static void starfive_wdt_stop(struct starfive_wdt *wdt)
static int starfive_wdt_pm_start(struct watchdog_device *wdd)
{
struct starfive_wdt *wdt = watchdog_get_drvdata(wdd);
- int ret = pm_runtime_get_sync(wdd->parent);
+ int ret = pm_runtime_resume_and_get(wdd->parent);
if (ret < 0)
return ret;
diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c
index d6107ac50e88..5bf1e5caaa62 100644
--- a/fs/9p/vfs_addr.c
+++ b/fs/9p/vfs_addr.c
@@ -54,9 +54,21 @@ static void v9fs_begin_writeback(struct netfs_io_request *wreq)
static void v9fs_issue_write(struct netfs_io_subrequest *subreq)
{
struct p9_fid *fid = subreq->rreq->netfs_priv;
+ struct inode *inode = subreq->rreq->inode;
+ struct netfs_inode *ictx = netfs_inode(inode);
int err, len;
len = p9_client_write(fid, subreq->start, &subreq->io_iter, &err);
+ if (len > 0) {
+ unsigned long long end = subreq->start + len;
+
+ spin_lock(&inode->i_lock);
+ if (end > i_size_read(inode))
+ i_size_write(inode, end);
+ if (end > ictx->remote_i_size)
+ ictx->remote_i_size = end;
+ spin_unlock(&inode->i_lock);
+ }
netfs_write_subrequest_terminated(subreq, len ?: err, false);
}
diff --git a/fs/autofs/inode.c b/fs/autofs/inode.c
index ec3c1ac1c1f9..7203dba0ebce 100644
--- a/fs/autofs/inode.c
+++ b/fs/autofs/inode.c
@@ -51,6 +51,10 @@ void autofs_kill_sb(struct super_block *sb)
if (sbi) {
/* Free wait queues, close pipe */
autofs_catatonic_mode(sbi);
+ if (sbi->pipe) {
+ fput(sbi->pipe);
+ sbi->pipe = NULL;
+ }
put_pid(sbi->oz_pgrp);
}
diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c
index 604399e59a3d..bc05a5d23daf 100644
--- a/fs/btrfs/dev-replace.c
+++ b/fs/btrfs/dev-replace.c
@@ -505,6 +505,7 @@ static int mark_block_group_to_copy(struct btrfs_fs_info *fs_info,
path->reada = READA_FORWARD;
path->search_commit_root = 1;
path->skip_locking = 1;
+ path->need_commit_sem = 1;
key.objectid = src_dev->devid;
key.type = BTRFS_DEV_EXTENT_KEY;
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 9c3b5ecb0b01..5627e2f7dd64 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -2545,8 +2545,10 @@ int btrfs_replace_file_extents(struct btrfs_inode *inode,
inode_set_ctime_current(&inode->vfs_inode));
ret = btrfs_update_inode(trans, inode);
- if (ret)
+ if (unlikely(ret)) {
+ btrfs_abort_transaction(trans, ret);
break;
+ }
btrfs_end_transaction(trans);
btrfs_btree_balance_dirty(fs_info);
diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs/tree-checker.c
index 92a2df6dd3a4..6b68d87cbf44 100644
--- a/fs/btrfs/tree-checker.c
+++ b/fs/btrfs/tree-checker.c
@@ -1914,7 +1914,7 @@ static int check_dev_extent_item(const struct extent_buffer *leaf,
sectorsize))) {
generic_err(leaf, slot,
"invalid dev extent chunk offset, has %llu not aligned to %u",
- btrfs_dev_extent_chunk_objectid(leaf, de),
+ btrfs_dev_extent_chunk_offset(leaf, de),
sectorsize);
return -EUCLEAN;
}
diff --git a/fs/configfs/dir.c b/fs/configfs/dir.c
index e7475ba0f8f1..e13947348906 100644
--- a/fs/configfs/dir.c
+++ b/fs/configfs/dir.c
@@ -395,8 +395,17 @@ static void remove_dir(struct dentry * d)
configfs_remove_dirent(d);
- if (d_really_is_positive(d))
- simple_rmdir(d_inode(parent),d);
+ if (d_really_is_positive(d)) {
+ if (!simple_rmdir(d_inode(parent), d))
+ /*
+ * configfs_get_config_item() takes a hashed dentry as
+ * proof that ->s_element is still alive. Our caller
+ * is about to drop the last reference to the item and
+ * the VFS will not unhash until after we return, so
+ * unhash it here.
+ */
+ d_drop(d);
+ }
pr_debug(" o %pd removing done (%d)\n", d, d_count(d));
diff --git a/fs/configfs/symlink.c b/fs/configfs/symlink.c
index 69133ec1fac2..b42f07e2ea7f 100644
--- a/fs/configfs/symlink.c
+++ b/fs/configfs/symlink.c
@@ -76,9 +76,9 @@ static int configfs_get_target_path(struct config_item *item,
static int create_link(struct config_item *parent_item,
struct config_item *item,
+ struct configfs_dirent *target_sd,
struct dentry *dentry)
{
- struct configfs_dirent *target_sd = item->ci_dentry->d_fsdata;
char *body;
int ret;
@@ -114,26 +114,35 @@ static int create_link(struct config_item *parent_item,
}
-static int get_target(const char *symname, struct path *path,
- struct config_item **target, struct super_block *sb)
+static int get_target(const char *symname, struct config_item **target,
+ struct configfs_dirent **target_sd,
+ struct super_block *sb)
{
+ struct path path __free(path_put) = {};
int ret;
- ret = kern_path(symname, LOOKUP_FOLLOW|LOOKUP_DIRECTORY, path);
- if (!ret) {
- if (path->dentry->d_sb == sb) {
- *target = configfs_get_config_item(path->dentry);
- if (!*target) {
- ret = -ENOENT;
- path_put(path);
- }
- } else {
- ret = -EPERM;
- path_put(path);
- }
- }
+ ret = kern_path(symname, LOOKUP_FOLLOW|LOOKUP_DIRECTORY, &path);
+ if (ret)
+ return ret;
+ if (path.dentry->d_sb != sb)
+ return -EPERM;
+ /*
+ * A hashed dentry guarantees that neither the item nor the dirent
+ * have been released yet, as removals unhash before dropping.
+ * Grab both references here. An item reference alone would not keep
+ * ->ci_dentry alive.
+ */
+ spin_lock(&path.dentry->d_lock);
+ if (!d_unhashed(path.dentry)) {
+ struct configfs_dirent *sd = path.dentry->d_fsdata;
- return ret;
+ *target = config_item_get(sd->s_element);
+ *target_sd = configfs_get(sd);
+ }
+ spin_unlock(&path.dentry->d_lock);
+ if (!*target)
+ return -ENOENT;
+ return 0;
}
@@ -141,10 +150,10 @@ int configfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
struct dentry *dentry, const char *symname)
{
int ret;
- struct path path;
struct configfs_dirent *sd;
struct config_item *parent_item;
struct config_item *target_item = NULL;
+ struct configfs_dirent *target_sd = NULL;
const struct config_item_type *type;
sd = dentry->d_parent->d_fsdata;
@@ -188,7 +197,7 @@ int configfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
* AV, a thoroughly annoyed bastard.
*/
inode_unlock(dir);
- ret = get_target(symname, &path, &target_item, dentry->d_sb);
+ ret = get_target(symname, &target_item, &target_sd, dentry->d_sb);
inode_lock(dir);
if (ret)
goto out_put;
@@ -202,15 +211,15 @@ int configfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
ret = type->ct_item_ops->allow_link(parent_item, target_item);
if (!ret) {
mutex_lock(&configfs_symlink_mutex);
- ret = create_link(parent_item, target_item, dentry);
+ ret = create_link(parent_item, target_item, target_sd, dentry);
mutex_unlock(&configfs_symlink_mutex);
if (ret && type->ct_item_ops->drop_link)
type->ct_item_ops->drop_link(parent_item,
target_item);
}
+ configfs_put(target_sd);
config_item_put(target_item);
- path_put(&path);
out_put:
config_item_put(parent_item);
diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c
index 20e2cb18ed1d..06987f7f6a19 100644
--- a/fs/erofs/fscache.c
+++ b/fs/erofs/fscache.c
@@ -67,6 +67,8 @@ static void erofs_fscache_req_complete(struct erofs_fscache_rq *req)
continue;
if (!failed)
folio_mark_uptodate(folio);
+ /* Skip the entire folio before unlocking allows it to be split. */
+ xas_advance(&xas, folio_next_index(folio) - 1);
folio_unlock(folio);
}
rcu_read_unlock();
diff --git a/fs/erofs/sysfs.c b/fs/erofs/sysfs.c
index 3fbce0864a66..77ec8790927b 100644
--- a/fs/erofs/sysfs.c
+++ b/fs/erofs/sysfs.c
@@ -81,6 +81,7 @@ EROFS_ATTR_FEATURE(sb_chksum);
EROFS_ATTR_FEATURE(ztailpacking);
EROFS_ATTR_FEATURE(fragments);
EROFS_ATTR_FEATURE(dedupe);
+EROFS_ATTR_FEATURE(xattr_prefixes);
static struct attribute *erofs_feat_attrs[] = {
ATTR_LIST(zero_padding),
@@ -93,6 +94,7 @@ static struct attribute *erofs_feat_attrs[] = {
ATTR_LIST(ztailpacking),
ATTR_LIST(fragments),
ATTR_LIST(dedupe),
+ ATTR_LIST(xattr_prefixes),
NULL,
};
ATTRIBUTE_GROUPS(erofs_feat);
diff --git a/fs/exec.c b/fs/exec.c
index e702cd638b49..829c74c7690d 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1217,6 +1217,17 @@ void __set_task_comm(struct task_struct *tsk, const char *buf, bool exec)
perf_event_comm(tsk, exec);
}
+static void posixtimer_exec(struct task_struct *me)
+{
+#ifdef CONFIG_POSIX_TIMERS
+ spin_lock_irq(&me->sighand->siglock);
+ posix_cpu_timers_exit(me);
+ spin_unlock_irq(&me->sighand->siglock);
+ exit_itimers(me);
+ flush_itimer_signals();
+#endif
+}
+
/*
* Calling this is the point of no return. None of the failures will be
* seen by userspace since either the process is already taking a fatal
@@ -1250,6 +1261,16 @@ int begin_new_exec(struct linux_binprm * bprm)
retval = de_thread(me);
if (retval)
goto out;
+
+ /*
+ * This must be done here to ensure that POSIX CPU timers which were
+ * armed on the current task are dequeued from me::posix_cputimers.
+ * Otherwise in case of a TID switch the deletion of the related POSIX
+ * timer would not remove an enqueued timer because the TID lookup
+ * of the old TID fails.
+ */
+ posixtimer_exec(me);
+
/* see the comment in check_unsafe_exec() */
current->fs->in_exec = 0;
/*
@@ -1304,14 +1325,6 @@ int begin_new_exec(struct linux_binprm * bprm)
if (retval)
goto out_unlock;
-#ifdef CONFIG_POSIX_TIMERS
- spin_lock_irq(&me->sighand->siglock);
- posix_cpu_timers_exit(me);
- spin_unlock_irq(&me->sighand->siglock);
- exit_itimers(me);
- flush_itimer_signals();
-#endif
-
/*
* Make the signal table private.
*/
diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c
index a3f807c3b72c..abc9bb6e4eb5 100644
--- a/fs/f2fs/debug.c
+++ b/fs/f2fs/debug.c
@@ -285,10 +285,6 @@ static void update_mem_info(struct f2fs_sb_info *sbi)
get_cache:
si->cache_mem = 0;
- /* build gc */
- if (sbi->gc_thread)
- si->cache_mem += sizeof(struct f2fs_gc_kthread);
-
/* build merge flush thread */
if (SM_I(sbi)->fcc_info)
si->cache_mem += sizeof(struct flush_cmd_control);
diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c
index 54dd52de7269..e20e62cae083 100644
--- a/fs/f2fs/dir.c
+++ b/fs/f2fs/dir.c
@@ -298,6 +298,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
de = find_in_block(dir, dentry_page, fname, &max_slots, use_hash);
if (IS_ERR(de)) {
+ f2fs_put_page(dentry_page, 0);
*res_page = ERR_CAST(de);
de = NULL;
break;
@@ -437,7 +438,7 @@ void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de,
set_page_dirty(page);
inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
- f2fs_mark_inode_dirty_sync(dir, false);
+ f2fs_mark_inode_dirty_sync(dir, true);
f2fs_put_page(page, 1);
}
@@ -591,7 +592,7 @@ void f2fs_update_parent_metadata(struct inode *dir, struct inode *inode,
clear_inode_flag(inode, FI_NEW_INODE);
}
inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
- f2fs_mark_inode_dirty_sync(dir, false);
+ f2fs_mark_inode_dirty_sync(dir, true);
if (F2FS_I(dir)->i_current_depth != current_depth)
f2fs_i_depth_write(dir, current_depth);
@@ -903,7 +904,7 @@ void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct page *page,
f2fs_put_page(page, 1);
inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
- f2fs_mark_inode_dirty_sync(dir, false);
+ f2fs_mark_inode_dirty_sync(dir, true);
if (inode)
f2fs_drop_nlink(dir, inode);
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 095f2c2953f1..053c8630d6ef 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -1578,6 +1578,33 @@ struct decompress_io_ctx {
#define MAX_COMPRESS_LOG_SIZE 8
#define MAX_COMPRESS_WINDOW_SIZE(log_size) ((PAGE_SIZE) << (log_size))
+struct f2fs_gc_kthread {
+ struct task_struct *f2fs_gc_task;
+ wait_queue_head_t gc_wait_queue_head;
+
+ /* for gc sleep time */
+ unsigned int urgent_sleep_time;
+ unsigned int min_sleep_time;
+ unsigned int max_sleep_time;
+ unsigned int no_gc_sleep_time;
+
+ /* for changing gc mode */
+ bool gc_wake;
+
+ /* for GC_MERGE mount option */
+ wait_queue_head_t fggc_wq; /*
+ * caller of f2fs_balance_fs()
+ * will wait on this wait queue.
+ */
+
+ /* for gc control for zoned devices */
+ unsigned int no_zoned_gc_percent;
+ unsigned int boost_zoned_gc_percent;
+ unsigned int valid_thresh_ratio;
+ unsigned int boost_gc_multiple;
+ unsigned int boost_gc_greedy;
+};
+
struct f2fs_sb_info {
struct super_block *sb; /* pointer to VFS super block */
struct proc_dir_entry *s_proc; /* proc entry */
@@ -1702,7 +1729,7 @@ struct f2fs_sb_info {
* semaphore for GC, avoid
* race between GC and GC or CP
*/
- struct f2fs_gc_kthread *gc_thread; /* GC thread */
+ struct f2fs_gc_kthread gc_thread; /* GC thread */
struct atgc_management am; /* atgc management */
unsigned int cur_victim_sec; /* current victim section num */
unsigned int gc_mode; /* current GC state */
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 129cdee4149a..a7ad5e02433c 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -1064,9 +1064,10 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
{
struct inode *inode = d_inode(dentry);
struct f2fs_inode_info *fi = F2FS_I(inode);
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
int err;
- if (unlikely(f2fs_cp_error(F2FS_I_SB(inode))))
+ if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
err = setattr_prepare(idmap, dentry, attr);
@@ -1096,6 +1097,23 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
!IS_ALIGNED(attr->ia_size,
F2FS_BLK_TO_BYTES(fi->i_cluster_size)))
return -EINVAL;
+
+ if (f2fs_is_pinned_file(inode)) {
+ /*
+ * It may break section-aligned fallocate recovery
+ * mechanism, so do not allow larger size truncation.
+ */
+ if (attr->ia_size > i_size_read(inode))
+ return -EINVAL;
+ /*
+ * To prevent scattered pin block generation, we don't
+ * allow smaller/equal size unaligned truncation for
+ * pinned file.
+ */
+ else if (!IS_ALIGNED(attr->ia_size,
+ F2FS_BLK_TO_BYTES(CAP_BLKS_PER_SEC(sbi))))
+ return -EINVAL;
+ }
}
if (is_quota_modification(idmap, inode, attr)) {
@@ -1105,12 +1123,11 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
}
if (i_uid_needs_update(idmap, attr, inode) ||
i_gid_needs_update(idmap, attr, inode)) {
- f2fs_lock_op(F2FS_I_SB(inode));
+ f2fs_lock_op(sbi);
err = dquot_transfer(idmap, inode, attr);
if (err) {
- set_sbi_flag(F2FS_I_SB(inode),
- SBI_QUOTA_NEED_REPAIR);
- f2fs_unlock_op(F2FS_I_SB(inode));
+ set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
+ f2fs_unlock_op(sbi);
return err;
}
/*
@@ -1120,7 +1137,7 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
i_uid_update(idmap, attr, inode);
i_gid_update(idmap, attr, inode);
f2fs_mark_inode_dirty_sync(inode, true);
- f2fs_unlock_op(F2FS_I_SB(inode));
+ f2fs_unlock_op(sbi);
}
if (attr->ia_valid & ATTR_SIZE) {
@@ -1188,7 +1205,7 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
f2fs_mark_inode_dirty_sync(inode, true);
/* inode change will produce dirty node pages flushed by checkpoint */
- f2fs_balance_fs(F2FS_I_SB(inode), true);
+ f2fs_balance_fs(sbi, true);
return err;
}
@@ -3100,8 +3117,9 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
struct inode *src = file_inode(file_in);
struct inode *dst = file_inode(file_out);
struct f2fs_sb_info *sbi = F2FS_I_SB(src);
- size_t olen = len, dst_max_i_size = 0;
- size_t dst_osize;
+ size_t olen = len;
+ loff_t dst_max_i_size = 0;
+ loff_t dst_osize, dst_end;
int ret;
if (file_in->f_path.mnt != file_out->f_path.mnt ||
@@ -3158,8 +3176,15 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
}
dst_osize = dst->i_size;
- if (pos_out + olen > dst->i_size)
- dst_max_i_size = pos_out + olen;
+ if (olen > LLONG_MAX - pos_out)
+ goto out_unlock;
+ dst_end = pos_out + olen;
+ if (dst_end > dst->i_size) {
+ ret = inode_newsize_ok(dst, dst_end);
+ if (ret)
+ goto out_unlock;
+ dst_max_i_size = dst_end;
+ }
/* verify the end result is block aligned */
if (!IS_ALIGNED(pos_in, F2FS_BLKSIZE) ||
diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
index 49d1acd26a4c..4f9fa45ebbac 100644
--- a/fs/f2fs/gc.c
+++ b/fs/f2fs/gc.c
@@ -31,9 +31,9 @@ static unsigned int count_bits(const unsigned long *addr,
static int gc_thread_func(void *data)
{
struct f2fs_sb_info *sbi = data;
- struct f2fs_gc_kthread *gc_th = sbi->gc_thread;
- wait_queue_head_t *wq = &sbi->gc_thread->gc_wait_queue_head;
- wait_queue_head_t *fggc_wq = &sbi->gc_thread->fggc_wq;
+ struct f2fs_gc_kthread *gc_th = &sbi->gc_thread;
+ wait_queue_head_t *wq = &sbi->gc_thread.gc_wait_queue_head;
+ wait_queue_head_t *fggc_wq = &sbi->gc_thread.fggc_wq;
unsigned int wait_ms;
struct f2fs_gc_control gc_control = {
.victim_segno = NULL_SEGNO,
@@ -191,13 +191,9 @@ static int gc_thread_func(void *data)
int f2fs_start_gc_thread(struct f2fs_sb_info *sbi)
{
- struct f2fs_gc_kthread *gc_th;
+ struct f2fs_gc_kthread *gc_th = &sbi->gc_thread;
dev_t dev = sbi->sb->s_bdev->bd_dev;
- gc_th = f2fs_kmalloc(sbi, sizeof(struct f2fs_gc_kthread), GFP_KERNEL);
- if (!gc_th)
- return -ENOMEM;
-
gc_th->urgent_sleep_time = DEF_GC_THREAD_URGENT_SLEEP_TIME;
gc_th->valid_thresh_ratio = DEF_GC_THREAD_VALID_THRESH_RATIO;
gc_th->boost_gc_multiple = BOOST_GC_MULTIPLE;
@@ -219,16 +215,14 @@ int f2fs_start_gc_thread(struct f2fs_sb_info *sbi)
gc_th->gc_wake = false;
- sbi->gc_thread = gc_th;
- init_waitqueue_head(&sbi->gc_thread->gc_wait_queue_head);
- init_waitqueue_head(&sbi->gc_thread->fggc_wq);
- sbi->gc_thread->f2fs_gc_task = kthread_run(gc_thread_func, sbi,
+ init_waitqueue_head(&gc_th->gc_wait_queue_head);
+ init_waitqueue_head(&gc_th->fggc_wq);
+ gc_th->f2fs_gc_task = kthread_run(gc_thread_func, sbi,
"f2fs_gc-%u:%u", MAJOR(dev), MINOR(dev));
if (IS_ERR(gc_th->f2fs_gc_task)) {
int err = PTR_ERR(gc_th->f2fs_gc_task);
- kfree(gc_th);
- sbi->gc_thread = NULL;
+ gc_th->f2fs_gc_task = NULL;
return err;
}
@@ -237,14 +231,14 @@ int f2fs_start_gc_thread(struct f2fs_sb_info *sbi)
void f2fs_stop_gc_thread(struct f2fs_sb_info *sbi)
{
- struct f2fs_gc_kthread *gc_th = sbi->gc_thread;
+ struct f2fs_gc_kthread *gc_th = &sbi->gc_thread;
- if (!gc_th)
+ if (!gc_th->f2fs_gc_task)
return;
+
kthread_stop(gc_th->f2fs_gc_task);
+ gc_th->f2fs_gc_task = NULL;
wake_up_all(&gc_th->fggc_wq);
- kfree(gc_th);
- sbi->gc_thread = NULL;
}
static int select_gc_type(struct f2fs_sb_info *sbi, int gc_type)
@@ -406,7 +400,7 @@ static inline unsigned int get_gc_cost(struct f2fs_sb_info *sbi,
return get_seg_entry(sbi, segno)->ckpt_valid_blocks;
if (p->one_time_gc && (get_valid_blocks(sbi, segno, true) >=
- CAP_BLKS_PER_SEC(sbi) * sbi->gc_thread->valid_thresh_ratio /
+ CAP_BLKS_PER_SEC(sbi) * sbi->gc_thread.valid_thresh_ratio /
100))
return UINT_MAX;
@@ -1795,9 +1789,9 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
if (f2fs_sb_has_blkzoned(sbi) &&
!has_enough_free_blocks(sbi,
- sbi->gc_thread->boost_zoned_gc_percent))
+ sbi->gc_thread.boost_zoned_gc_percent))
window_granularity *=
- sbi->gc_thread->boost_gc_multiple;
+ sbi->gc_thread.boost_gc_multiple;
end_segno = start_segno + window_granularity;
}
@@ -2150,8 +2144,9 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi,
static int free_segment_range(struct f2fs_sb_info *sbi,
unsigned int secs, bool dry_run)
{
- unsigned int next_inuse, start, end;
+ unsigned int secno, next_inuse, start, end, end_secno;
struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+ unsigned int freed_secs = 0;
int gc_mode, gc_type;
int err = 0;
int type;
@@ -2160,6 +2155,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
MAIN_SECS(sbi) -= secs;
start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi);
end = MAIN_SEGS(sbi) - 1;
+ end_secno = GET_SEC_FROM_SEG(sbi, end);
mutex_lock(&DIRTY_I(sbi)->seglist_lock);
for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2171,6 +2167,14 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
sbi->next_victim_seg[gc_type] = NULL_SEGNO;
mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
+ spin_lock(&FREE_I(sbi)->segmap_lock);
+ for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+ if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+ freed_secs++;
+ }
+ FREE_I(sbi)->free_sections -= freed_secs;
+ spin_unlock(&FREE_I(sbi)->segmap_lock);
+
/* Move out cursegs from the target range */
for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) {
err = f2fs_allocate_segment_for_resize(sbi, type, start, end);
@@ -2195,6 +2199,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
f2fs_bug_on(sbi, 1);
}
out:
+ spin_lock(&FREE_I(sbi)->segmap_lock);
+ FREE_I(sbi)->free_sections += freed_secs;
+ spin_unlock(&FREE_I(sbi)->segmap_lock);
MAIN_SECS(sbi) += secs;
return err;
}
diff --git a/fs/f2fs/gc.h b/fs/f2fs/gc.h
index 1a2e7a84b59f..5116e6366dae 100644
--- a/fs/f2fs/gc.h
+++ b/fs/f2fs/gc.h
@@ -45,32 +45,7 @@
#define NR_GC_CHECKPOINT_SECS (3) /* data/node/dentry sections */
-struct f2fs_gc_kthread {
- struct task_struct *f2fs_gc_task;
- wait_queue_head_t gc_wait_queue_head;
-
- /* for gc sleep time */
- unsigned int urgent_sleep_time;
- unsigned int min_sleep_time;
- unsigned int max_sleep_time;
- unsigned int no_gc_sleep_time;
-
- /* for changing gc mode */
- bool gc_wake;
-
- /* for GC_MERGE mount option */
- wait_queue_head_t fggc_wq; /*
- * caller of f2fs_balance_fs()
- * will wait on this wait queue.
- */
- /* for gc control for zoned devices */
- unsigned int no_zoned_gc_percent;
- unsigned int boost_zoned_gc_percent;
- unsigned int valid_thresh_ratio;
- unsigned int boost_gc_multiple;
- unsigned int boost_gc_greedy;
-};
struct gc_inode_list {
struct list_head ilist;
diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c
index b87a88c2e3c8..3ff8152965e2 100644
--- a/fs/f2fs/inline.c
+++ b/fs/f2fs/inline.c
@@ -727,7 +727,7 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, struct page *page,
f2fs_put_page(page, 1);
inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
- f2fs_mark_inode_dirty_sync(dir, false);
+ f2fs_mark_inode_dirty_sync(dir, true);
if (inode)
f2fs_drop_nlink(dir, inode);
diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c
index 1d204294abf9..5a2f99ee955b 100644
--- a/fs/f2fs/namei.c
+++ b/fs/f2fs/namei.c
@@ -1038,7 +1038,7 @@ static int f2fs_rename(struct mnt_idmap *idmap, struct inode *old_dir,
f2fs_up_write(&F2FS_I(old_inode)->i_sem);
inode_set_ctime_current(old_inode);
- f2fs_mark_inode_dirty_sync(old_inode, false);
+ f2fs_mark_inode_dirty_sync(old_inode, true);
f2fs_delete_entry(old_entry, old_page, old_dir, NULL);
old_page = NULL;
@@ -1207,7 +1207,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
f2fs_i_links_write(old_dir, old_nlink > 0);
f2fs_up_write(&F2FS_I(old_dir)->i_sem);
}
- f2fs_mark_inode_dirty_sync(old_dir, false);
+ f2fs_mark_inode_dirty_sync(old_dir, true);
/* update directory entry info of new dir inode */
f2fs_set_link(new_dir, new_entry, new_page, old_inode);
@@ -1226,7 +1226,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
f2fs_i_links_write(new_dir, new_nlink > 0);
f2fs_up_write(&F2FS_I(new_dir)->i_sem);
}
- f2fs_mark_inode_dirty_sync(new_dir, false);
+ f2fs_mark_inode_dirty_sync(new_dir, true);
if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT) {
f2fs_add_ino_entry(sbi, old_dir->i_ino, TRANS_DIR_INO);
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index 1f4f68e56d43..0898ce801f28 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -125,14 +125,14 @@ bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type)
return res;
}
-static void clear_node_page_dirty(struct page *page)
+static void clear_node_folio_dirty(struct folio *folio)
{
- if (PageDirty(page)) {
- f2fs_clear_page_cache_dirty_tag(page_folio(page));
- clear_page_dirty_for_io(page);
- dec_page_count(F2FS_P_SB(page), F2FS_DIRTY_NODES);
+ if (folio_test_dirty(folio)) {
+ f2fs_clear_page_cache_dirty_tag(folio);
+ folio_clear_dirty_for_io(folio);
+ dec_page_count(F2FS_M_SB(folio->mapping), F2FS_DIRTY_NODES);
}
- ClearPageUptodate(page);
+ folio_clear_uptodate(folio);
}
static struct page *get_current_nat_page(struct f2fs_sb_info *sbi, nid_t nid)
@@ -941,7 +941,7 @@ static int truncate_node(struct dnode_of_data *dn)
f2fs_inode_synced(dn->inode);
}
- clear_node_page_dirty(dn->node_page);
+ clear_node_folio_dirty(page_folio(dn->node_page));
set_sbi_flag(sbi, SBI_IS_DIRTY);
index = page_folio(dn->node_page)->index;
@@ -1394,7 +1394,7 @@ struct page *f2fs_new_node_page(struct dnode_of_data *dn, unsigned int ofs)
inc_valid_inode_count(sbi);
return page;
fail:
- clear_node_page_dirty(page);
+ clear_node_folio_dirty(page_folio(page));
f2fs_put_page(page, 1);
return ERR_PTR(err);
}
@@ -1476,6 +1476,7 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid,
struct page *parent, int start)
{
struct page *page;
+ struct folio *folio;
int err;
if (!nid)
@@ -1487,6 +1488,7 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid,
if (!page)
return ERR_PTR(-ENOMEM);
+ folio = page_folio(page);
err = read_node_page(page, 0);
if (err < 0) {
goto out_put_err;
@@ -1526,11 +1528,11 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid,
f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER);
err = -EFSCORRUPTED;
out_err:
- ClearPageUptodate(page);
+ clear_node_folio_dirty(folio);
out_put_err:
/* ENOENT comes from read_node_page which is not an error. */
if (err != -ENOENT)
- f2fs_handle_page_eio(sbi, page_folio(page), NODE);
+ f2fs_handle_page_eio(sbi, folio, NODE);
f2fs_put_page(page, 1);
return ERR_PTR(err);
}
diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c
index bc105da3e746..f05206e32ff5 100644
--- a/fs/f2fs/recovery.c
+++ b/fs/f2fs/recovery.c
@@ -157,6 +157,22 @@ static int init_recovered_filename(const struct inode *dir,
return 0;
}
+static const char *recover_printable_name(struct inode *inode,
+ struct f2fs_inode *raw,
+ int *name_len)
+{
+ static const char encrypted_name[] = "<encrypted>";
+
+ if (file_enc_name(inode)) {
+ *name_len = sizeof(encrypted_name) - 1;
+ return encrypted_name;
+ }
+
+ *name_len = min_t(unsigned int, le32_to_cpu(raw->i_namelen),
+ F2FS_NAME_LEN);
+ return raw->i_name;
+}
+
static int recover_dentry(struct inode *inode, struct page *ipage,
struct list_head *dir_list)
{
@@ -169,7 +185,8 @@ static int recover_dentry(struct inode *inode, struct page *ipage,
struct inode *dir, *einode;
struct fsync_inode_entry *entry;
int err = 0;
- char *name;
+ const char *name;
+ int name_len;
entry = get_fsync_inode(dir_list, pino);
if (!entry) {
@@ -228,12 +245,9 @@ static int recover_dentry(struct inode *inode, struct page *ipage,
out_put:
f2fs_put_page(page, 0);
out:
- if (file_enc_name(inode))
- name = "<encrypted>";
- else
- name = raw_inode->i_name;
- f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %s, dir = %lx, err = %d",
- __func__, ino_of_node(ipage), name,
+ name = recover_printable_name(inode, raw_inode, &name_len);
+ f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %lx, err = %d",
+ __func__, ino_of_node(ipage), name_len, name,
IS_ERR(dir) ? 0 : dir->i_ino, err);
return err;
}
@@ -281,7 +295,8 @@ static int recover_inode(struct inode *inode, struct page *page)
{
struct f2fs_inode *raw = F2FS_INODE(page);
struct f2fs_inode_info *fi = F2FS_I(inode);
- char *name;
+ const char *name;
+ int name_len;
int err;
inode->i_mode = le16_to_cpu(raw->i_mode);
@@ -330,13 +345,11 @@ static int recover_inode(struct inode *inode, struct page *page)
f2fs_mark_inode_dirty_sync(inode, true);
- if (file_enc_name(inode))
- name = "<encrypted>";
- else
- name = F2FS_INODE(page)->i_name;
+ name = recover_printable_name(inode, raw, &name_len);
- f2fs_notice(F2FS_I_SB(inode), "recover_inode: ino = %x, name = %s, inline = %x",
- ino_of_node(page), name, raw->i_inline);
+ f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x",
+ __func__, ino_of_node(page), name_len, name,
+ raw->i_inline);
return 0;
}
diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index e3797e99271e..30ea0b7fb1fb 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -441,15 +441,14 @@ void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need)
f2fs_submit_merged_write(sbi, DATA);
f2fs_submit_all_merged_ipu_writes(sbi);
- if (test_opt(sbi, GC_MERGE) && sbi->gc_thread &&
- sbi->gc_thread->f2fs_gc_task) {
+ if (test_opt(sbi, GC_MERGE) && sbi->gc_thread.f2fs_gc_task) {
DEFINE_WAIT(wait);
- prepare_to_wait(&sbi->gc_thread->fggc_wq, &wait,
+ prepare_to_wait(&sbi->gc_thread.fggc_wq, &wait,
TASK_UNINTERRUPTIBLE);
- wake_up(&sbi->gc_thread->gc_wait_queue_head);
+ wake_up(&sbi->gc_thread.gc_wait_queue_head);
io_schedule();
- finish_wait(&sbi->gc_thread->fggc_wq, &wait);
+ finish_wait(&sbi->gc_thread.fggc_wq, &wait);
} else {
struct f2fs_gc_control gc_control = {
.victim_segno = NULL_SEGNO,
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 6bf2e0c7854c..f7acb1998b2c 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -2492,11 +2492,11 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
if ((*flags & SB_RDONLY) ||
(F2FS_OPTION(sbi).bggc_mode == BGGC_MODE_OFF &&
!test_opt(sbi, GC_MERGE))) {
- if (sbi->gc_thread) {
+ if (sbi->gc_thread.f2fs_gc_task) {
f2fs_stop_gc_thread(sbi);
need_restart_gc = true;
}
- } else if (!sbi->gc_thread) {
+ } else if (!sbi->gc_thread.f2fs_gc_task) {
err = f2fs_start_gc_thread(sbi);
if (err)
goto restore_opts;
diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c
index 56c50faa3ca5..8206510020fe 100644
--- a/fs/f2fs/sysfs.c
+++ b/fs/f2fs/sysfs.c
@@ -74,7 +74,7 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a,
static unsigned char *__struct_ptr(struct f2fs_sb_info *sbi, int struct_type)
{
if (struct_type == GC_THREAD)
- return (unsigned char *)sbi->gc_thread;
+ return (unsigned char *)&sbi->gc_thread;
else if (struct_type == SM_INFO)
return (unsigned char *)SM_I(sbi);
else if (struct_type == DCC_INFO)
@@ -631,20 +631,20 @@ static ssize_t __sbi_store(struct f2fs_attr *a,
sbi->gc_mode = GC_NORMAL;
} else if (t == 1) {
sbi->gc_mode = GC_URGENT_HIGH;
- if (sbi->gc_thread) {
- sbi->gc_thread->gc_wake = true;
+ if (sbi->gc_thread.f2fs_gc_task) {
+ sbi->gc_thread.gc_wake = true;
wake_up_interruptible_all(
- &sbi->gc_thread->gc_wait_queue_head);
+ &sbi->gc_thread.gc_wait_queue_head);
wake_up_discard_thread(sbi, true);
}
} else if (t == 2) {
sbi->gc_mode = GC_URGENT_LOW;
} else if (t == 3) {
sbi->gc_mode = GC_URGENT_MID;
- if (sbi->gc_thread) {
- sbi->gc_thread->gc_wake = true;
+ if (sbi->gc_thread.f2fs_gc_task) {
+ sbi->gc_thread.gc_wake = true;
wake_up_interruptible_all(
- &sbi->gc_thread->gc_wait_queue_head);
+ &sbi->gc_thread.gc_wait_queue_head);
}
} else {
return -EINVAL;
@@ -901,14 +901,14 @@ static ssize_t __sbi_store(struct f2fs_attr *a,
if (!strcmp(a->attr.name, "gc_boost_gc_multiple")) {
if (t < 1 || t > SEGS_PER_SEC(sbi))
return -EINVAL;
- sbi->gc_thread->boost_gc_multiple = (unsigned int)t;
+ sbi->gc_thread.boost_gc_multiple = (unsigned int)t;
return count;
}
if (!strcmp(a->attr.name, "gc_boost_gc_greedy")) {
if (t > GC_GREEDY)
return -EINVAL;
- sbi->gc_thread->boost_gc_greedy = (unsigned int)t;
+ sbi->gc_thread.boost_gc_greedy = (unsigned int)t;
return count;
}
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 4e1e0dd16428..48ac4f58605e 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -693,19 +693,34 @@ static bool isw_prepare_wbs_switch(struct bdi_writeback *new_wb,
struct inode_switch_wbs_context *isw,
struct list_head *list, int *nr)
{
- struct inode *inode;
+ struct inode *inode, *tmp;
+ LIST_HEAD(scanned);
+ bool full = false;
+
+ /*
+ * Walk from the oldest end and move scanned inodes to the newest
+ * end, so the next scan resumes at unscanned inodes instead of
+ * re-walking an ever-growing run of prepared and skipped ones.
+ * For b_dirty_time this keeps the oldest unscanned inode at the
+ * end move_expired_inodes() picks from; b_attached is unordered.
+ */
+ list_for_each_entry_safe_reverse(inode, tmp, list, i_io_list) {
+ list_move(&inode->i_io_list, &scanned);
- list_for_each_entry(inode, list, i_io_list) {
if (!inode_prepare_wbs_switch(inode, new_wb))
continue;
isw->inodes[*nr] = inode;
(*nr)++;
- if (*nr >= WB_MAX_INODES_PER_ISW - 1)
- return true;
+ if (*nr >= WB_MAX_INODES_PER_ISW - 1) {
+ full = true;
+ break;
+ }
}
- return false;
+ list_splice(&scanned, list);
+
+ return full;
}
/**
diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c
index 1358c21837f1..27f88e86a53f 100644
--- a/fs/kernfs/mount.c
+++ b/fs/kernfs/mount.c
@@ -396,8 +396,8 @@ void kernfs_kill_sb(struct super_block *sb)
up_write(&root->kernfs_supers_rwsem);
/*
- * Remove the superblock from fs_supers/s_instances
- * so we can't find it, before freeing kernfs_super_info.
+ * Mark the superblock dead so sget_fc() can't find it,
+ * before freeing kernfs_super_info.
*/
kill_anon_super(sb);
kfree(info);
diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
index 616ed213adaa..90c6527ad712 100644
--- a/fs/lockd/svcsubs.c
+++ b/fs/lockd/svcsubs.c
@@ -493,7 +493,7 @@ EXPORT_SYMBOL_GPL(nlmsvc_unlock_all_by_sb);
static int
nlmsvc_match_ip(void *datap, struct nlm_host *host)
{
- return rpc_cmp_addr(nlm_srcaddr(host), datap);
+ return rpc_cmp_addr(nlm_srcaddr(datap), (struct sockaddr *)host);
}
/**
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index 93cf65db77a0..e7339f2bc5fd 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -1677,10 +1677,10 @@ int ntfs_create_inode(struct mnt_idmap *idmap, struct inode *dir,
goto out6;
/*
- * Call 'd_instantiate' after inode->i_op is set
+ * Call 'd_instantiate_new' after inode->i_op is set
* but before finish_open.
*/
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
/* Set original time. inode times (i_ctime) may be changed in ntfs_init_acl. */
inode_set_atime_to_ts(inode, ni->i_crtime);
@@ -1728,9 +1728,6 @@ int ntfs_create_inode(struct mnt_idmap *idmap, struct inode *dir,
if (!fnd)
ni_unlock(dir_ni);
- if (!err)
- unlock_new_inode(inode);
-
return err;
}
diff --git a/fs/ocfs2/namei.c b/fs/ocfs2/namei.c
index 1dcd420878dc..5ce71207bc9d 100644
--- a/fs/ocfs2/namei.c
+++ b/fs/ocfs2/namei.c
@@ -332,13 +332,8 @@ static int ocfs2_mknod(struct mnt_idmap *idmap,
goto leave;
/* calculate meta data/clusters for setting security and acl xattr */
- status = ocfs2_calc_xattr_init(dir, mode, &si, &want_clusters,
- &xattr_credits, &want_meta,
- &acl_state);
- if (status < 0) {
- mlog_errno(status);
- goto leave;
- }
+ ocfs2_calc_xattr_init(dir, mode, &si, &want_clusters, &xattr_credits,
+ &want_meta, &acl_state);
/* Reserve a cluster if creating an extent based directory. */
if (S_ISDIR(mode) && !ocfs2_supports_inline_data(osb)) {
diff --git a/fs/ocfs2/xattr.c b/fs/ocfs2/xattr.c
index 52156e849543..3cc679e5f1a1 100644
--- a/fs/ocfs2/xattr.c
+++ b/fs/ocfs2/xattr.c
@@ -607,12 +607,11 @@ int ocfs2_calc_security_init(struct inode *dir,
return ret;
}
-int ocfs2_calc_xattr_init(struct inode *dir, umode_t mode,
- struct ocfs2_security_xattr_info *si,
- int *want_clusters, int *xattr_credits,
- int *want_meta, struct ocfs2_acl_state *acl_state)
+void ocfs2_calc_xattr_init(struct inode *dir, umode_t mode,
+ struct ocfs2_security_xattr_info *si,
+ int *want_clusters, int *xattr_credits,
+ int *want_meta, struct ocfs2_acl_state *acl_state)
{
- int ret = 0;
struct ocfs2_super *osb = OCFS2_SB(dir->i_sb);
int s_size = 0, a_size = 0, acl_len = 0, new_clusters;
@@ -634,7 +633,7 @@ int ocfs2_calc_xattr_init(struct inode *dir, umode_t mode,
}
if (!(s_size + a_size))
- return ret;
+ return;
/*
* The max space of security xattr taken inline is
@@ -700,8 +699,6 @@ int ocfs2_calc_xattr_init(struct inode *dir, umode_t mode,
}
}
}
-
- return ret;
}
static int ocfs2_xattr_extend_allocation(struct inode *inode,
diff --git a/fs/ocfs2/xattr.h b/fs/ocfs2/xattr.h
index 5cdd6c6b4006..aeba323e14b3 100644
--- a/fs/ocfs2/xattr.h
+++ b/fs/ocfs2/xattr.h
@@ -57,10 +57,10 @@ int ocfs2_calc_security_init(struct inode *,
int *, int *, struct ocfs2_alloc_context **);
struct ocfs2_acl_state;
-int ocfs2_calc_xattr_init(struct inode *dir, umode_t mode,
- struct ocfs2_security_xattr_info *si,
- int *want_clusters, int *xattr_credits,
- int *want_meta, struct ocfs2_acl_state *acl_state);
+void ocfs2_calc_xattr_init(struct inode *dir, umode_t mode,
+ struct ocfs2_security_xattr_info *si,
+ int *want_clusters, int *xattr_credits,
+ int *want_meta, struct ocfs2_acl_state *acl_state);
/*
* xattrs can live inside an inode, as part of an external xattr block,
diff --git a/fs/pidfs.c b/fs/pidfs.c
index 5a8d8eb8df23..b2dc613a64f6 100644
--- a/fs/pidfs.c
+++ b/fs/pidfs.c
@@ -122,6 +122,7 @@ static long pidfd_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
struct pid *pid = pidfd_pid(file);
struct ns_common *ns_common = NULL;
struct pid_namespace *pid_ns;
+ int error;
if (arg)
return -EINVAL;
@@ -130,20 +131,33 @@ static long pidfd_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
if (!task)
return -ESRCH;
+ /*
+ * We're trying to open a file descriptor to the namespace so perform a
+ * filesystem cred ptrace check. Hold @task's exec_update_lock for the
+ * duration of the ptrace check and the namespace lookup so that the
+ * credentials used for the access decision match those of @task at the
+ * time its namespace is read, preventing a concurrent execve() from
+ * swapping the task's credentials in between the check and the use. We
+ * mirror nsfs behavior.
+ */
+ error = down_read_killable(&task->signal->exec_update_lock);
+ if (error)
+ return error;
+
+ if (!ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS)) {
+ error = -EACCES;
+ goto out_unlock;
+ }
+
scoped_guard(task_lock, task) {
nsp = task->nsproxy;
if (nsp)
get_nsproxy(nsp);
}
- if (!nsp)
- return -ESRCH; /* just pretend it didn't exist */
-
- /*
- * We're trying to open a file descriptor to the namespace so perform a
- * filesystem cred ptrace check. Also, we mirror nsfs behavior.
- */
- if (!ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS))
- return -EACCES;
+ if (!nsp) {
+ error = -ESRCH; /* just pretend it didn't exist */
+ goto out_unlock;
+ }
switch (cmd) {
/* Namespaces that hang of nsproxy. */
@@ -211,11 +225,16 @@ static long pidfd_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
}
break;
default:
- return -ENOIOCTLCMD;
+ error = -ENOIOCTLCMD;
}
- if (!ns_common)
- return -EOPNOTSUPP;
+ if (!error && !ns_common)
+ error = -EOPNOTSUPP;
+
+out_unlock:
+ up_read(&task->signal->exec_update_lock);
+ if (error)
+ return error;
/* open_namespace() unconditionally consumes the reference */
return open_namespace(ns_common);
diff --git a/fs/smb/client/cached_dir.c b/fs/smb/client/cached_dir.c
index fbc746813a6f..bb1358d64949 100644
--- a/fs/smb/client/cached_dir.c
+++ b/fs/smb/client/cached_dir.c
@@ -8,6 +8,7 @@
#include <linux/namei.h>
#include "cifsglob.h"
#include "cifsproto.h"
+#include "../common/smb2status.h"
#include "cifs_debug.h"
#include "smb2proto.h"
#include "cached_dir.h"
@@ -294,25 +295,37 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon,
rc = compound_send_recv(xid, ses, server,
flags, 2, rqst,
resp_buftype, rsp_iov);
- if (rc) {
- if (rc == -EREMCHG) {
- tcon->need_reconnect = true;
- pr_warn_once("server share %s deleted\n",
- tcon->tree_name);
- }
- goto oshr_free;
+ if (rc == -EREMCHG) {
+ tcon->need_reconnect = true;
+ pr_warn_once("server share %s deleted\n",
+ tcon->tree_name);
}
- cfid->is_open = true;
- spin_lock(&cfids->cfid_list_lock);
+ if (!rsp_iov[0].iov_base || rsp_iov[0].iov_len < sizeof(*o_rsp)) {
+ if (!rc)
+ rc = -EIO;
+ goto oshr_free;
+ }
o_rsp = (struct smb2_create_rsp *)rsp_iov[0].iov_base;
+ if (o_rsp->hdr.Status != STATUS_SUCCESS) {
+ if (!rc)
+ rc = -EIO;
+ goto oshr_free;
+ }
+
oparms.fid->persistent_fid = o_rsp->PersistentFileId;
oparms.fid->volatile_fid = o_rsp->VolatileFileId;
#ifdef CONFIG_CIFS_DEBUG2
oparms.fid->mid = le64_to_cpu(o_rsp->hdr.MessageId);
#endif /* CIFS_DEBUG2 */
+ cfid->is_open = true;
+ atomic_inc(&tcon->num_remote_opens);
+ if (rc)
+ goto oshr_free;
+
+ spin_lock(&cfids->cfid_list_lock);
if (o_rsp->OplockLevel != SMB2_OPLOCK_LEVEL_LEASE) {
spin_unlock(&cfids->cfid_list_lock);
@@ -378,7 +391,6 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon,
close_cached_dir(cfid);
} else {
*ret_cfid = cfid;
- atomic_inc(&tcon->num_remote_opens);
}
kfree(utf16_path);
diff --git a/fs/smb/client/cifs_fs_sb.h b/fs/smb/client/cifs_fs_sb.h
index 651759192280..2b4850566f15 100644
--- a/fs/smb/client/cifs_fs_sb.h
+++ b/fs/smb/client/cifs_fs_sb.h
@@ -55,6 +55,7 @@ struct cifs_sb_info {
struct smb3_fs_context *ctx;
atomic_t active;
unsigned int mnt_cifs_flags;
+ atomic_t outstanding_rreq; /* nr of rreqs not yet fully deinitialized */
struct delayed_work prune_tlinks;
struct rcu_head rcu;
diff --git a/fs/smb/client/cifsacl.c b/fs/smb/client/cifsacl.c
index 0a7c98e76836..102a7abd7274 100644
--- a/fs/smb/client/cifsacl.c
+++ b/fs/smb/client/cifsacl.c
@@ -1024,13 +1024,13 @@ unsigned int setup_special_user_owner_ACE(struct smb_ace *pntace)
static void populate_new_aces(char *nacl_base,
struct smb_sid *pownersid,
struct smb_sid *pgrpsid,
- __u64 *pnmode, u16 *pnum_aces, u16 *pnsize,
+ __u64 *pnmode, u16 *pnum_aces, u32 *pnsize,
bool modefromsid,
bool posix)
{
__u64 nmode;
u16 num_aces = 0;
- u16 nsize = 0;
+ u32 nsize = 0;
__u64 user_mode;
__u64 group_mode;
__u64 other_mode;
@@ -1129,16 +1129,17 @@ static void populate_new_aces(char *nacl_base,
*pnsize = nsize;
}
-static __u16 replace_sids_and_copy_aces(struct smb_acl *pdacl, struct smb_acl *pndacl,
- struct smb_sid *pownersid, struct smb_sid *pgrpsid,
- struct smb_sid *pnownersid, struct smb_sid *pngrpsid)
+static int replace_sids_and_copy_aces(struct smb_acl *pdacl, struct smb_acl *pndacl,
+ struct smb_sid *pownersid, struct smb_sid *pgrpsid,
+ struct smb_sid *pnownersid, struct smb_sid *pngrpsid,
+ int *aclflag, u16 *pnsize)
{
int i;
u16 size = 0;
struct smb_ace *pntace = NULL;
char *acl_base = NULL;
u16 src_num_aces = 0;
- u16 nsize = 0;
+ u32 nsize = 0;
struct smb_ace *pnntace = NULL;
char *nacl_base = NULL;
u16 ace_size = 0;
@@ -1155,18 +1156,24 @@ static __u16 replace_sids_and_copy_aces(struct smb_acl *pdacl, struct smb_acl *p
pntace = (struct smb_ace *) (acl_base + size);
pnntace = (struct smb_ace *) (nacl_base + nsize);
- if (pnownersid && compare_sids(&pntace->sid, pownersid) == 0)
+ if (pnownersid && compare_sids(&pntace->sid, pownersid) == 0) {
ace_size = cifs_copy_ace(pnntace, pntace, pnownersid);
- else if (pngrpsid && compare_sids(&pntace->sid, pgrpsid) == 0)
+ *aclflag |= CIFS_ACL_DACL;
+ } else if (pngrpsid && compare_sids(&pntace->sid, pgrpsid) == 0) {
ace_size = cifs_copy_ace(pnntace, pntace, pngrpsid);
- else
+ *aclflag |= CIFS_ACL_DACL;
+ } else {
ace_size = cifs_copy_ace(pnntace, pntace, NULL);
+ }
size += le16_to_cpu(pntace->size);
nsize += ace_size;
+ if (nsize > U16_MAX)
+ return -EOVERFLOW;
}
- return nsize;
+ *pnsize = nsize;
+ return 0;
}
static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl,
@@ -1178,7 +1185,7 @@ static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl,
struct smb_ace *pntace = NULL;
char *acl_base = NULL;
u16 src_num_aces = 0;
- u16 nsize = 0;
+ u32 nsize = 0;
struct smb_ace *pnntace = NULL;
char *nacl_base = NULL;
u16 num_aces = 0;
@@ -1229,6 +1236,8 @@ static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl,
nsize += cifs_copy_ace(pnntace, pntace, NULL);
num_aces++;
+ if (nsize > U16_MAX)
+ return -EOVERFLOW;
next_ace:
size += le16_to_cpu(pntace->size);
@@ -1245,6 +1254,10 @@ static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl,
}
finalize_dacl:
+ /* The DACL size field is 16-bit on the wire, see MS-DTYP 2.4.5 */
+ if (nsize > U16_MAX)
+ return -EOVERFLOW;
+
pndacl->num_aces = cpu_to_le16(num_aces);
pndacl->size = cpu_to_le16(nsize);
@@ -1405,6 +1418,8 @@ static int build_sec_desc(struct smb_ntsd *pntsd, struct smb_ntsd *pnntsd,
rc = set_chmod_dacl(dacl_ptr, ndacl_ptr, owner_sid_ptr, group_sid_ptr,
pnmode, mode_from_sid, posix);
+ if (rc)
+ return rc;
sidsoffset = ndacloffset + le16_to_cpu(ndacl_ptr->size);
/* copy the non-dacl portion of secdesc */
@@ -1480,9 +1495,12 @@ static int build_sec_desc(struct smb_ntsd *pntsd, struct smb_ntsd *pnntsd,
if (dacloffset) {
/* Replace ACEs for old owner with new one */
- size = replace_sids_and_copy_aces(dacl_ptr, ndacl_ptr,
- owner_sid_ptr, group_sid_ptr,
- nowner_sid_ptr, ngroup_sid_ptr);
+ rc = replace_sids_and_copy_aces(dacl_ptr, ndacl_ptr,
+ owner_sid_ptr, group_sid_ptr,
+ nowner_sid_ptr, ngroup_sid_ptr,
+ aclflag, &size);
+ if (rc)
+ goto chown_chgrp_exit;
ndacl_ptr->size = cpu_to_le16(size);
}
@@ -1697,7 +1715,7 @@ id_mode_to_cifs_acl(struct inode *inode, const char *path, __u64 *pnmode,
kuid_t uid, kgid_t gid)
{
int rc = 0;
- int aclflag = CIFS_ACL_DACL; /* default flag to set */
+ int aclflag = 0;
__u32 secdesclen = 0;
__u32 nsecdesclen = 0;
__u32 dacloffset = 0;
@@ -1798,14 +1816,23 @@ id_mode_to_cifs_acl(struct inode *inode, const char *path, __u64 *pnmode,
cifs_dbg(NOISY, "build_sec_desc rc: %d\n", rc);
- if (ops->set_acl == NULL)
- rc = -EOPNOTSUPP;
+ if (rc != 0)
+ goto id_mode_to_cifs_acl_exit;
- if (!rc) {
- /* Set the security descriptor */
- rc = ops->set_acl(pnntsd, nsecdesclen, inode, path, aclflag);
- cifs_dbg(NOISY, "set_cifs_acl rc: %d\n", rc);
+ if (aclflag == 0) {
+ cifs_dbg(FYI, "set_cifs_acl aclflag=0, no change mapped\n");
+ goto id_mode_to_cifs_acl_exit;
+ }
+
+ if (ops->set_acl == NULL) {
+ rc = -EOPNOTSUPP;
+ goto id_mode_to_cifs_acl_exit;
}
+
+ /* Set the security descriptor */
+ rc = ops->set_acl(pnntsd, nsecdesclen, inode, path, aclflag);
+ cifs_dbg(NOISY, "set_cifs_acl rc: %d\n", rc);
+
id_mode_to_cifs_acl_exit:
cifs_put_tlink(tlink);
diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c
index a14f24f8d4a5..f22c79179a23 100644
--- a/fs/smb/client/cifsfs.c
+++ b/fs/smb/client/cifsfs.c
@@ -302,6 +302,18 @@ static void cifs_kill_sb(struct super_block *sb)
/* Wait for all opened files to release */
flush_workqueue(deferredclose_wq);
+ /*
+ * Wait for all in-flight netfs I/O requests to finish their
+ * cleanup_work so that any cifsFileInfo final puts they queue
+ * to fileinfo_put_wq/serverclose_wq have been queued, then
+ * drain the workqueue so the cfile dentry refs are dropped to
+ * avoid the busy dentry warning.
+ */
+ wait_var_event(&cifs_sb->outstanding_rreq,
+ !atomic_read(&cifs_sb->outstanding_rreq));
+ flush_workqueue(serverclose_wq);
+ flush_workqueue(fileinfo_put_wq);
+
/* finally release root dentry */
dput(cifs_sb->root);
cifs_sb->root = NULL;
diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h
index 3f9c03828cd4..1734ae9bf318 100644
--- a/fs/smb/client/cifsglob.h
+++ b/fs/smb/client/cifsglob.h
@@ -1619,7 +1619,7 @@ static inline void cifs_stats_bytes_read(struct cifs_tcon *tcon,
* Returns zero on a successful receive, or an error. The receive state in
* the TCP_Server_Info will also be updated.
*/
-typedef int (mid_receive_t)(struct TCP_Server_Info *server,
+typedef int (*mid_receive_t)(struct TCP_Server_Info *server,
struct mid_q_entry *mid);
/*
@@ -1630,13 +1630,13 @@ typedef int (mid_receive_t)(struct TCP_Server_Info *server,
* - it will be called by cifsd, with no locks held
* - the mid will be removed from any lists
*/
-typedef void (mid_callback_t)(struct mid_q_entry *mid);
+typedef void (*mid_callback_t)(struct mid_q_entry *mid);
/*
* This is the protopyte for mid handle function. This is called once the mid
* has been recognized after decryption of the message.
*/
-typedef int (mid_handle_t)(struct TCP_Server_Info *server,
+typedef int (*mid_handle_t)(struct TCP_Server_Info *server,
struct mid_q_entry *mid);
/* one of these for every pending CIFS request to the server */
@@ -1654,9 +1654,9 @@ struct mid_q_entry {
unsigned long when_sent; /* time when smb send finished */
unsigned long when_received; /* when demux complete (taken off wire) */
#endif
- mid_receive_t *receive; /* call receive callback */
- mid_callback_t *callback; /* call completion callback */
- mid_handle_t *handle; /* call handle mid callback */
+ mid_receive_t receive; /* call receive callback */
+ mid_callback_t callback; /* call completion callback */
+ mid_handle_t handle; /* call handle mid callback */
void *callback_data; /* general purpose pointer for callback */
struct task_struct *creator;
void *resp_buf; /* pointer to received SMB header */
diff --git a/fs/smb/client/cifsproto.h b/fs/smb/client/cifsproto.h
index d2d004764a70..e2a7d55a2502 100644
--- a/fs/smb/client/cifsproto.h
+++ b/fs/smb/client/cifsproto.h
@@ -96,10 +96,10 @@ extern int cifs_ipaddr_cmp(struct sockaddr *srcaddr, struct sockaddr *rhs);
extern bool cifs_match_ipaddr(struct sockaddr *srcaddr, struct sockaddr *rhs);
extern int cifs_discard_remaining_data(struct TCP_Server_Info *server);
extern int cifs_call_async(struct TCP_Server_Info *server,
- struct smb_rqst *rqst,
- mid_receive_t *receive, mid_callback_t *callback,
- mid_handle_t *handle, void *cbdata, const int flags,
- const struct cifs_credits *exist_credits);
+ struct smb_rqst *rqst,
+ mid_receive_t receive, mid_callback_t callback,
+ mid_handle_t handle, void *cbdata, const int flags,
+ const struct cifs_credits *exist_credits);
extern struct TCP_Server_Info *cifs_pick_channel(struct cifs_ses *ses);
extern int cifs_send_recv(const unsigned int xid, struct cifs_ses *ses,
struct TCP_Server_Info *server,
diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c
index 01c78d9e7ec0..df02e2d9204a 100644
--- a/fs/smb/client/cifssmb.c
+++ b/fs/smb/client/cifssmb.c
@@ -2829,7 +2829,7 @@ int cifs_query_reparse_point(const unsigned int xid,
end = 2 + get_bcc(&io_rsp->hdr) + (__u8 *)&io_rsp->ByteCount;
start = (__u8 *)&io_rsp->hdr.Protocol + data_offset;
- if (start >= end) {
+ if (start >= end || (size_t)(end - start) < sizeof(*buf)) {
rc = -EIO;
goto error;
}
diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c
index 5e17462c8e37..a7261fb38379 100644
--- a/fs/smb/client/connect.c
+++ b/fs/smb/client/connect.c
@@ -187,6 +187,8 @@ cifs_signal_cifsd_for_reconnect(struct TCP_Server_Info *server,
nserver = ses->chans[i].server;
if (!nserver)
continue;
+ if (!list_empty(&nserver->rlist))
+ continue;
nserver->srv_count++;
list_add(&nserver->rlist, &reco);
}
@@ -195,11 +197,15 @@ cifs_signal_cifsd_for_reconnect(struct TCP_Server_Info *server,
}
}
+ spin_lock(&cifs_tcp_ses_lock);
list_for_each_entry_safe(server, nserver, &reco, rlist) {
list_del_init(&server->rlist);
set_need_reco(server);
+ spin_unlock(&cifs_tcp_ses_lock);
cifs_put_tcp_session(server, 0);
+ spin_lock(&cifs_tcp_ses_lock);
}
+ spin_unlock(&cifs_tcp_ses_lock);
}
/*
@@ -982,6 +988,7 @@ clean_demultiplex_info(struct TCP_Server_Info *server)
spin_unlock(&server->srv_lock);
cancel_delayed_work_sync(&server->echo);
+ cancel_delayed_work_sync(&server->reconnect);
spin_lock(&server->srv_lock);
server->tcpStatus = CifsExiting;
@@ -1767,6 +1774,7 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
spin_lock_init(&tcp_ses->mid_lock);
INIT_LIST_HEAD(&tcp_ses->tcp_ses_list);
INIT_LIST_HEAD(&tcp_ses->smb_ses_list);
+ INIT_LIST_HEAD(&tcp_ses->rlist);
INIT_DELAYED_WORK(&tcp_ses->echo, cifs_echo_request);
INIT_DELAYED_WORK(&tcp_ses->reconnect, smb2_reconnect_server);
mutex_init(&tcp_ses->reconnect_mutex);
@@ -1877,6 +1885,7 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
kfree(tcp_ses->leaf_fullpath);
if (tcp_ses->ssocket)
sock_release(tcp_ses->ssocket);
+ smbd_destroy(tcp_ses);
kfree(tcp_ses);
}
return ERR_PTR(rc);
@@ -3390,6 +3399,7 @@ int cifs_setup_cifs_sb(struct cifs_sb_info *cifs_sb)
spin_lock_init(&cifs_sb->tlink_tree_lock);
cifs_sb->tlink_tree = RB_ROOT;
+ atomic_set(&cifs_sb->outstanding_rreq, 0);
cifs_dbg(FYI, "file mode: %04ho dir mode: %04ho\n",
ctx->file_mode, ctx->dir_mode);
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index 339346a82ecf..afe00c0cc696 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -276,6 +276,7 @@ static int cifs_init_request(struct netfs_io_request *rreq, struct file *file)
return -EIO;
}
+ atomic_inc(&cifs_sb->outstanding_rreq);
return 0;
}
@@ -297,9 +298,13 @@ static void cifs_rreq_done(struct netfs_io_request *rreq)
static void cifs_free_request(struct netfs_io_request *rreq)
{
struct cifs_io_request *req = container_of(rreq, struct cifs_io_request, rreq);
+ struct cifs_sb_info *cifs_sb = CIFS_SB(rreq->inode->i_sb);
if (req->cfile)
cifsFileInfo_put(req->cfile);
+
+ if (atomic_dec_and_test(&cifs_sb->outstanding_rreq))
+ wake_up_var(&cifs_sb->outstanding_rreq);
}
static void cifs_free_subrequest(struct netfs_io_subrequest *subreq)
@@ -3062,8 +3067,6 @@ void cifs_oplock_break(struct work_struct *work)
struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo,
oplock_break);
struct inode *inode = d_inode(cfile->dentry);
- struct super_block *sb = inode->i_sb;
- struct cifs_sb_info *cifs_sb = CIFS_SB(sb);
struct cifsInodeInfo *cinode = CIFS_I(inode);
struct cifs_tcon *tcon;
struct TCP_Server_Info *server;
@@ -3076,8 +3079,8 @@ void cifs_oplock_break(struct work_struct *work)
wait_on_bit(&cinode->flags, CIFS_INODE_PENDING_WRITERS,
TASK_UNINTERRUPTIBLE);
- tlink = cifs_sb_tlink(cifs_sb);
- if (IS_ERR(tlink)) {
+ tlink = cifs_get_tlink(cfile->tlink);
+ if (IS_ERR_OR_NULL(tlink)) {
/* drop the reference taken when the break was queued */
_cifsFileInfo_put(cfile, false /* do not wait for ourself */, false);
goto out;
diff --git a/fs/smb/client/misc.c b/fs/smb/client/misc.c
index fb1eb38bce50..5ac8faeda4f7 100644
--- a/fs/smb/client/misc.c
+++ b/fs/smb/client/misc.c
@@ -1046,7 +1046,11 @@ parse_dfs_referrals(struct get_dfs_referral_rsp *rsp, u32 rsp_size,
node->ref_flag = le16_to_cpu(ref->ReferralEntryFlags);
/* copy DfsPath */
- if (le16_to_cpu(ref->DfsPathOffset) > data_end - (char *)ref) {
+ if (le16_to_cpu(ref->DfsPathOffset) < sizeof(*ref) ||
+ le16_to_cpu(ref->DfsPathOffset) > data_end - (char *)ref) {
+ cifs_dbg(VFS, "%s: DfsPathOffset %u out of range [%zu, %td]\n",
+ __func__, le16_to_cpu(ref->DfsPathOffset),
+ sizeof(*ref), data_end - (char *)ref);
rc = -EINVAL;
goto parse_DFS_referrals_exit;
}
@@ -1060,7 +1064,11 @@ parse_dfs_referrals(struct get_dfs_referral_rsp *rsp, u32 rsp_size,
}
/* copy link target UNC */
- if (le16_to_cpu(ref->NetworkAddressOffset) > data_end - (char *)ref) {
+ if (le16_to_cpu(ref->NetworkAddressOffset) < sizeof(*ref) ||
+ le16_to_cpu(ref->NetworkAddressOffset) > data_end - (char *)ref) {
+ cifs_dbg(VFS, "%s: NetworkAddressOffset %u out of range [%zu, %td]\n",
+ __func__, le16_to_cpu(ref->NetworkAddressOffset),
+ sizeof(*ref), data_end - (char *)ref);
rc = -EINVAL;
goto parse_DFS_referrals_exit;
}
diff --git a/fs/smb/client/reparse.c b/fs/smb/client/reparse.c
index fe1fa561e495..ee630eb575d1 100644
--- a/fs/smb/client/reparse.c
+++ b/fs/smb/client/reparse.c
@@ -726,9 +726,9 @@ static bool wsl_to_fattr(struct cifs_open_info_data *data,
fattr->cf_gid = wsl_make_kgid(cifs_sb, v);
else if (!strncmp(name, SMB2_WSL_XATTR_MODE, nlen)) {
/* File type in reparse point tag and in xattr mode must match. */
- if (S_DT(fattr->cf_mode) != S_DT(le32_to_cpu(*(__le32 *)v)))
+ if (S_DT(fattr->cf_mode) != S_DT(get_unaligned_le32(v)))
return false;
- fattr->cf_mode = (umode_t)le32_to_cpu(*(__le32 *)v);
+ fattr->cf_mode = (umode_t)get_unaligned_le32(v);
} else if (!strncmp(name, SMB2_WSL_XATTR_DEV, nlen)) {
fattr->cf_rdev = reparse_mkdev(v);
have_xattr_dev = true;
diff --git a/fs/smb/client/reparse.h b/fs/smb/client/reparse.h
index f080f92cb1e7..d14f23bd8396 100644
--- a/fs/smb/client/reparse.h
+++ b/fs/smb/client/reparse.h
@@ -9,6 +9,7 @@
#include <linux/fs.h>
#include <linux/stat.h>
#include <linux/uidgid.h>
+#include <linux/unaligned.h>
#include "fs_context.h"
#include "cifsglob.h"
@@ -22,7 +23,7 @@
static inline dev_t reparse_mkdev(void *ptr)
{
- u64 v = le64_to_cpu(*(__le64 *)ptr);
+ u64 v = get_unaligned_le64(ptr);
return MKDEV(v & 0xffffffff, v >> 32);
}
@@ -30,7 +31,7 @@ static inline dev_t reparse_mkdev(void *ptr)
static inline kuid_t wsl_make_kuid(struct cifs_sb_info *cifs_sb,
void *ptr)
{
- u32 uid = le32_to_cpu(*(__le32 *)ptr);
+ u32 uid = get_unaligned_le32(ptr);
if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_OVERR_UID)
return cifs_sb->ctx->linux_uid;
@@ -40,7 +41,7 @@ static inline kuid_t wsl_make_kuid(struct cifs_sb_info *cifs_sb,
static inline kgid_t wsl_make_kgid(struct cifs_sb_info *cifs_sb,
void *ptr)
{
- u32 gid = le32_to_cpu(*(__le32 *)ptr);
+ u32 gid = get_unaligned_le32(ptr);
if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_OVERR_GID)
return cifs_sb->ctx->linux_gid;
diff --git a/fs/smb/client/sess.c b/fs/smb/client/sess.c
index 458d6922826f..cc211f5b6d1b 100644
--- a/fs/smb/client/sess.c
+++ b/fs/smb/client/sess.c
@@ -175,9 +175,9 @@ int cifs_try_adding_channels(struct cifs_ses *ses)
int old_chan_count, new_chan_count;
int left;
int rc = 0;
- int tries = 0;
+ int tries = 0, attempts;
size_t iface_weight = 0, iface_min_speed = 0;
- struct cifs_server_iface *iface = NULL, *niface = NULL;
+ struct cifs_server_iface *iface = NULL, *candidate = NULL;
struct cifs_server_iface *last_iface = NULL;
spin_lock(&ses->chan_lock);
@@ -223,67 +223,89 @@ int cifs_try_adding_channels(struct cifs_ses *ses)
break;
}
- if (!iface)
- iface = list_first_entry(&ses->iface_list, struct cifs_server_iface,
- iface_head);
last_iface = list_last_entry(&ses->iface_list, struct cifs_server_iface,
iface_head);
iface_min_speed = last_iface->speed;
+ spin_unlock(&ses->iface_lock);
- list_for_each_entry_safe_from(iface, niface, &ses->iface_list,
- iface_head) {
- /* do not mix rdma and non-rdma interfaces */
- if (iface->rdma_capable != ses->server->rdma)
- continue;
-
- /* skip ifaces that are unusable */
- if (!iface->is_active ||
- (is_ses_using_iface(ses, iface) &&
- !iface->rss_capable))
- continue;
+ attempts = 0;
+ while (left > 0) {
+ spin_lock(&ses->iface_lock);
- /* check if we already allocated enough channels */
- iface_weight = iface->speed / iface_min_speed;
+ /*
+ * iface_lock must be dropped while opening a channel,
+ * and a concurrent interface refresh may remove and
+ * free entries during that window, so no list entry
+ * may be kept across it without a reference. Scan
+ * the list from the beginning each time and only pass
+ * a referenced candidate to cifs_ses_add_channel();
+ * weight_fulfilled tracks the progress so that no
+ * iface is selected beyond its weight.
+ */
+ candidate = NULL;
+ list_for_each_entry(iface, &ses->iface_list, iface_head) {
+ /* do not mix rdma and non-rdma interfaces */
+ if (iface->rdma_capable != ses->server->rdma)
+ continue;
+
+ /* skip ifaces that are unusable */
+ if (!iface->is_active ||
+ (is_ses_using_iface(ses, iface) &&
+ !iface->rss_capable))
+ continue;
+
+ /* check if we already allocated enough channels */
+ iface_weight = iface->speed / iface_min_speed;
+
+ if (iface->weight_fulfilled >= iface_weight)
+ continue;
+
+ /* take ref before unlock */
+ kref_get(&iface->refcount);
+ candidate = iface;
+ break;
+ }
- if (iface->weight_fulfilled >= iface_weight)
- continue;
+ if (!candidate) {
+ /* no usable iface. reset weight_fulfilled and start over */
+ list_for_each_entry(iface, &ses->iface_list, iface_head)
+ iface->weight_fulfilled = 0;
+ spin_unlock(&ses->iface_lock);
+ break;
+ }
- /* take ref before unlock */
- kref_get(&iface->refcount);
+ attempts++;
+ if (attempts > 3 * ses->chan_max) {
+ kref_put(&candidate->refcount, release_iface);
+ spin_unlock(&ses->iface_lock);
+ break;
+ }
spin_unlock(&ses->iface_lock);
- rc = cifs_ses_add_channel(ses, iface);
+ rc = cifs_ses_add_channel(ses, candidate);
spin_lock(&ses->iface_lock);
if (rc) {
cifs_dbg(VFS, "failed to open extra channel on iface:%pIS rc=%d\n",
- &iface->sockaddr,
+ &candidate->sockaddr,
rc);
/* failure to add chan should increase weight */
- iface->weight_fulfilled++;
- kref_put(&iface->refcount, release_iface);
+ candidate->weight_fulfilled++;
+ kref_put(&candidate->refcount, release_iface);
+ spin_unlock(&ses->iface_lock);
continue;
}
- iface->num_channels++;
- iface->weight_fulfilled++;
- cifs_dbg(VFS, "successfully opened new channel on iface:%pIS\n",
- &iface->sockaddr);
- break;
- }
-
- /* reached end of list. reset weight_fulfilled and start over */
- if (list_entry_is_head(iface, &ses->iface_list, iface_head)) {
- list_for_each_entry(iface, &ses->iface_list, iface_head)
- iface->weight_fulfilled = 0;
+ candidate->num_channels++;
+ candidate->weight_fulfilled++;
+ cifs_info("successfully opened new channel on iface:%pIS\n",
+ &candidate->sockaddr);
spin_unlock(&ses->iface_lock);
- iface = NULL;
- continue;
- }
- spin_unlock(&ses->iface_lock);
- left--;
- new_chan_count++;
+ left--;
+ new_chan_count++;
+ break;
+ }
}
return new_chan_count - old_chan_count;
diff --git a/fs/smb/client/smb2inode.c b/fs/smb/client/smb2inode.c
index 4f08c808e4a3..f0fcecfaa2b1 100644
--- a/fs/smb/client/smb2inode.c
+++ b/fs/smb/client/smb2inode.c
@@ -71,6 +71,17 @@ static int parse_posix_sids(struct cifs_open_info_data *data,
sidsbuf = (u8 *)qi + le16_to_cpu(qi->OutputBufferOffset) + qi_len;
sidsbuf_end = sidsbuf + out_len - qi_len;
+ if (sidsbuf_end < sidsbuf) {
+ cifs_dbg(VFS, "%s: server-supplied out_len %u caused pointer wraparound\n",
+ __func__, out_len);
+ return -EINVAL;
+ }
+ if (sidsbuf_end > (u8 *)rsp_iov->iov_base + rsp_iov->iov_len) {
+ cifs_dbg(VFS, "%s: server-supplied out_len %u overruns iov by %td bytes\n",
+ __func__, out_len,
+ sidsbuf_end - ((u8 *)rsp_iov->iov_base + rsp_iov->iov_len));
+ return -EINVAL;
+ }
owner_len = posix_info_sid_size(sidsbuf, sidsbuf_end);
if (owner_len == -1)
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 3f6b8745667d..528cd1a6268c 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -786,9 +786,9 @@ parse_server_interfaces(struct network_interface_info_ioctl_rsp *buf,
break;
}
/* Validate that Next doesn't point beyond the buffer */
- if (next > bytes_left) {
- cifs_dbg(VFS, "%s: invalid Next pointer %zu > %zd\n",
- __func__, next, bytes_left);
+ if (next < sizeof(*p) || next > bytes_left) {
+ cifs_dbg(VFS, "%s: invalid Next pointer %zu out of range [%zu, %zd]\n",
+ __func__, next, sizeof(*p), bytes_left);
rc = -EINVAL;
goto out;
}
@@ -1057,8 +1057,9 @@ move_smb2_ea_to_cifs(char *dst, size_t dst_size,
char *name, *value;
size_t buf_size = dst_size;
size_t name_len, value_len, user_name_len;
+ u32 next_off;
- while (src_size > 0) {
+ while (src_size >= sizeof(*src)) {
name_len = (size_t)src->ea_name_length;
value_len = (size_t)le16_to_cpu(src->ea_value_length);
@@ -1113,14 +1114,21 @@ move_smb2_ea_to_cifs(char *dst, size_t dst_size,
if (!src->next_entry_offset)
break;
- if (src_size < le32_to_cpu(src->next_entry_offset)) {
- /* stop before overrun buffer */
- rc = -ERANGE;
- break;
+ next_off = le32_to_cpu(src->next_entry_offset);
+ if (next_off < sizeof(*src) || src_size < next_off) {
+ cifs_dbg(FYI, "EA next_entry_offset %u out of range [%zu, %zu]\n",
+ next_off, sizeof(*src), src_size);
+ rc = -EIO;
+ goto out;
+ }
+ src_size -= next_off;
+ src = (void *)((char *)src + next_off);
+ if (src_size > 0 && src_size < sizeof(*src)) {
+ cifs_dbg(FYI, "EA next_entry_offset %u left truncated entry (%zu bytes)\n",
+ next_off, src_size);
+ rc = -EIO;
+ goto out;
}
- src_size -= le32_to_cpu(src->next_entry_offset);
- src = (void *)((char *)src +
- le32_to_cpu(src->next_entry_offset));
}
/* didn't find the named attribute */
@@ -2422,8 +2430,14 @@ smb3_enum_snapshots(const unsigned int xid, struct cifs_tcon *tcon,
* and retry the ioctl again with larger array size sufficient
* to hold all of the snapshot GMT tokens on the second try.
*/
- if (snapshot_in.snapshot_array_size < GMT_TOKEN_SIZE)
+ if (snapshot_in.snapshot_array_size < GMT_TOKEN_SIZE) {
+ if (ret_data_len < sizeof(struct smb_snapshot_array)) {
+ rc = -EIO;
+ kfree(retbuf);
+ return rc;
+ }
ret_data_len = sizeof(struct smb_snapshot_array);
+ }
/*
* We return struct SRV_SNAPSHOT_ARRAY, followed by
@@ -4399,25 +4413,37 @@ smb3_create_lease_buf(u8 *lease_key, u8 oplock)
static __u8
smb2_parse_lease_buf(void *buf, __u16 *epoch, char *lease_key)
{
- struct create_lease *lc = (struct create_lease *)buf;
+ struct create_context *cc = buf;
+ struct lease_context lc;
*epoch = 0; /* not used */
- if (lc->lcontext.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
+ if (le32_to_cpu(cc->DataLength) != sizeof(lc))
+ return 0;
+
+ memcpy(&lc, (u8 *)cc + le16_to_cpu(cc->DataOffset), sizeof(lc));
+ if (lc.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
return SMB2_OPLOCK_LEVEL_NOCHANGE;
- return le32_to_cpu(lc->lcontext.LeaseState);
+ return le32_to_cpu(lc.LeaseState);
}
static __u8
smb3_parse_lease_buf(void *buf, __u16 *epoch, char *lease_key)
{
- struct create_lease_v2 *lc = (struct create_lease_v2 *)buf;
+ struct create_context *cc = buf;
+ struct lease_context_v2 lc;
- *epoch = le16_to_cpu(lc->lcontext.Epoch);
- if (lc->lcontext.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
+ if (le32_to_cpu(cc->DataLength) != sizeof(lc)) {
+ *epoch = 0;
+ return 0;
+ }
+
+ memcpy(&lc, (u8 *)cc + le16_to_cpu(cc->DataOffset), sizeof(lc));
+ *epoch = le16_to_cpu(lc.Epoch);
+ if (lc.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
return SMB2_OPLOCK_LEVEL_NOCHANGE;
if (lease_key)
- memcpy(lease_key, &lc->lcontext.LeaseKey, SMB2_LEASE_KEY_SIZE);
- return le32_to_cpu(lc->lcontext.LeaseState);
+ memcpy(lease_key, lc.LeaseKey, SMB2_LEASE_KEY_SIZE);
+ return le32_to_cpu(lc.LeaseState);
}
static unsigned int
@@ -5237,11 +5263,13 @@ receive_encrypted_standard(struct TCP_Server_Info *server,
length = decrypt_raw_data(server, buf, buf_size, NULL, false);
if (length)
return length;
+ pdu_length = buf_size;
next_is_large = server->large_buf;
one_more:
shdr = (struct smb2_hdr *)buf;
next_cmd = le32_to_cpu(shdr->NextCommand);
+ server->total_read = next_cmd ? next_cmd : pdu_length;
if (*num_mids >= MAX_COMPOUND) {
cifs_server_dbg(VFS, "too many PDUs in compound\n");
@@ -5249,8 +5277,15 @@ receive_encrypted_standard(struct TCP_Server_Info *server,
}
if (next_cmd) {
- if (WARN_ON_ONCE(next_cmd > pdu_length))
+ if (next_cmd < MID_HEADER_SIZE(server) ||
+ next_cmd > pdu_length ||
+ pdu_length - next_cmd < MID_HEADER_SIZE(server)) {
+ unsigned int max_next = pdu_length > (unsigned int)MID_HEADER_SIZE(server) ?
+ pdu_length - (unsigned int)MID_HEADER_SIZE(server) : 0;
+ cifs_server_dbg(VFS, "invalid NextCommand offset %u out of range [%zu, %u]\n",
+ next_cmd, MID_HEADER_SIZE(server), max_next);
return -1;
+ }
if (next_is_large)
next_buffer = (char *)cifs_buf_get();
else
@@ -5286,6 +5321,7 @@ receive_encrypted_standard(struct TCP_Server_Info *server,
server->bigbuf = buf = next_buffer;
else
server->smallbuf = buf = next_buffer;
+ next_buffer = NULL;
goto one_more;
} else if (ret != 0) {
/*
diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c
index bfb0298f6b10..3c170d7242ee 100644
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@ -188,18 +188,19 @@ cifs_chan_skip_or_disable(struct cifs_ses *ses,
spin_unlock(&ses->chan_lock);
/*
- * the above reference of server by channel
- * needs to be dropped without holding chan_lock
- * as cifs_put_tcp_session takes a higher lock
- * i.e. cifs_tcp_ses_lock
+ * signal the channel and its primary server to
+ * reconnect before dropping the above reference of
+ * server by channel, which is done without holding
+ * chan_lock as cifs_put_tcp_session takes a higher
+ * lock i.e. cifs_tcp_ses_lock
*/
- cifs_put_tcp_session(server, from_reconnect);
-
cifs_signal_cifsd_for_reconnect(server, false);
/* mark primary server as needing reconnect */
pserver = server->primary_server;
cifs_signal_cifsd_for_reconnect(pserver, false);
+
+ cifs_put_tcp_session(server, from_reconnect);
skip_terminate:
return -EHOSTDOWN;
}
@@ -2295,23 +2296,32 @@ create_reconnect_durable_buf(struct cifs_fid *fid)
static void
parse_query_id_ctxt(struct create_context *cc, struct smb2_file_all_info *buf)
{
- struct create_disk_id_rsp *pdisk_id = (struct create_disk_id_rsp *)cc;
+ u16 doff = le16_to_cpu(cc->DataOffset);
+ u32 dlen = le32_to_cpu(cc->DataLength);
+ u8 *beg;
- cifs_dbg(FYI, "parse query id context 0x%llx 0x%llx\n",
- pdisk_id->DiskFileId, pdisk_id->VolumeId);
- buf->IndexNumber = pdisk_id->DiskFileId;
+ if (dlen < sizeof(__le64))
+ return;
+
+ beg = (u8 *)cc + doff;
+ memcpy(&buf->IndexNumber, beg, sizeof(__le64));
+ cifs_dbg(FYI, "parse query id context 0x%llx\n",
+ le64_to_cpu(buf->IndexNumber));
}
static void
parse_posix_ctxt(struct create_context *cc, struct smb2_file_all_info *info,
struct create_posix_rsp *posix)
{
- int sid_len;
u8 *beg = (u8 *)cc + le16_to_cpu(cc->DataOffset);
- u8 *end = beg + le32_to_cpu(cc->DataLength);
+ u32 dlen = le32_to_cpu(cc->DataLength);
+ u8 *end = beg + dlen;
+ int sid_len;
u8 *sid;
memset(posix, 0, sizeof(*posix));
+ if (dlen < 3 * sizeof(__le32))
+ return;
posix->nlink = get_unaligned_le32(beg);
posix->reparse_tag = get_unaligned_le32(beg + 4);
@@ -2347,6 +2357,7 @@ int smb2_parse_contexts(struct TCP_Server_Info *server,
struct smb2_create_rsp *rsp = rsp_iov->iov_base;
struct create_context *cc;
size_t rem, off, len;
+ size_t cc_len;
size_t doff, dlen;
size_t noff, nlen;
char *name;
@@ -2369,29 +2380,41 @@ int smb2_parse_contexts(struct TCP_Server_Info *server,
buf->IndexNumber = 0;
while (rem >= sizeof(*cc)) {
+ off = le32_to_cpu(cc->Next);
+ if (off) {
+ if ((off & 0x7) || off >= rem || off < sizeof(*cc))
+ return -EINVAL;
+ cc_len = off;
+ } else {
+ cc_len = rem;
+ }
+
doff = le16_to_cpu(cc->DataOffset);
dlen = le32_to_cpu(cc->DataLength);
- if (check_add_overflow(doff, dlen, &len) || len > rem)
+ if (doff < sizeof(*cc) ||
+ check_add_overflow(doff, dlen, &len) || len > cc_len)
return -EINVAL;
noff = le16_to_cpu(cc->NameOffset);
nlen = le16_to_cpu(cc->NameLength);
- if (noff + nlen > doff)
+ if (noff < sizeof(*cc) ||
+ check_add_overflow(noff, nlen, &len) || len > cc_len ||
+ (dlen && len > doff))
return -EINVAL;
name = (char *)cc + noff;
switch (nlen) {
case 4:
- if (!strncmp(name, SMB2_CREATE_REQUEST_LEASE, 4)) {
+ if (dlen && !strncmp(name, SMB2_CREATE_REQUEST_LEASE, 4)) {
*oplock = server->ops->parse_lease_buf(cc, epoch,
lease_key);
- } else if (buf &&
+ } else if (dlen && buf &&
!strncmp(name, SMB2_CREATE_QUERY_ON_DISK_ID, 4)) {
parse_query_id_ctxt(cc, buf);
}
break;
case 16:
- if (posix && !memcmp(name, smb3_create_tag_posix, 16))
+ if (dlen && posix && !memcmp(name, smb3_create_tag_posix, 16))
parse_posix_ctxt(cc, buf, posix);
break;
default:
@@ -2403,13 +2426,18 @@ int smb2_parse_contexts(struct TCP_Server_Info *server,
}
off = le32_to_cpu(cc->Next);
- if (!off)
+ if (!off) {
+ rem = 0;
break;
+ }
if (check_sub_overflow(rem, off, &rem))
return -EINVAL;
cc = (struct create_context *)((u8 *)cc + off);
}
+ if (rem)
+ return -EINVAL;
+
if (rsp->OplockLevel != SMB2_OPLOCK_LEVEL_LEASE)
*oplock = rsp->OplockLevel;
diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c
index c3940d3ecd5f..14dacf296714 100644
--- a/fs/smb/client/transport.c
+++ b/fs/smb/client/transport.c
@@ -792,8 +792,8 @@ cifs_setup_async_request(struct TCP_Server_Info *server, struct smb_rqst *rqst)
*/
int
cifs_call_async(struct TCP_Server_Info *server, struct smb_rqst *rqst,
- mid_receive_t *receive, mid_callback_t *callback,
- mid_handle_t *handle, void *cbdata, const int flags,
+ mid_receive_t receive, mid_callback_t callback,
+ mid_handle_t handle, void *cbdata, const int flags,
const struct cifs_credits *exist_credits)
{
int rc;
diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c
index 37d8db4bbbae..5795a19bb4cb 100644
--- a/fs/smb/server/smb2pdu.c
+++ b/fs/smb/server/smb2pdu.c
@@ -4648,26 +4648,31 @@ int smb2_query_dir(struct ksmbd_work *work)
/**
* buffer_check_err() - helper function to check buffer errors
* @reqOutputBufferLength: max buffer length expected in command response
+ * @fixed_len: minimum fixed response length
* @rsp: query info response buffer contains output buffer length
- * @rsp_org: base response buffer pointer in case of chained response
*
* Return: 0 on success, otherwise error
*/
static int buffer_check_err(int reqOutputBufferLength,
- struct smb2_query_info_rsp *rsp,
- void *rsp_org)
+ unsigned int fixed_len,
+ struct smb2_query_info_rsp *rsp)
{
- if (reqOutputBufferLength < le32_to_cpu(rsp->OutputBufferLength)) {
+ unsigned int output_len = le32_to_cpu(rsp->OutputBufferLength);
+
+ if (reqOutputBufferLength < fixed_len) {
pr_err("Invalid Buffer Size Requested\n");
rsp->hdr.Status = STATUS_INFO_LENGTH_MISMATCH;
- *(__be32 *)rsp_org = cpu_to_be32(sizeof(struct smb2_hdr));
return -EINVAL;
}
+
+ if (reqOutputBufferLength < output_len) {
+ rsp->hdr.Status = STATUS_BUFFER_OVERFLOW;
+ rsp->OutputBufferLength = cpu_to_le32(reqOutputBufferLength);
+ }
return 0;
}
-static void get_standard_info_pipe(struct smb2_query_info_rsp *rsp,
- void *rsp_org)
+static void get_standard_info_pipe(struct smb2_query_info_rsp *rsp)
{
struct smb2_file_standard_info *sinfo;
@@ -4682,8 +4687,7 @@ static void get_standard_info_pipe(struct smb2_query_info_rsp *rsp,
cpu_to_le32(sizeof(struct smb2_file_standard_info));
}
-static void get_internal_info_pipe(struct smb2_query_info_rsp *rsp, u64 num,
- void *rsp_org)
+static void get_internal_info_pipe(struct smb2_query_info_rsp *rsp, u64 num)
{
struct smb2_file_internal_info *file_info;
@@ -4697,8 +4701,7 @@ static void get_internal_info_pipe(struct smb2_query_info_rsp *rsp, u64 num,
static int smb2_get_info_file_pipe(struct ksmbd_session *sess,
struct smb2_query_info_req *req,
- struct smb2_query_info_rsp *rsp,
- void *rsp_org)
+ struct smb2_query_info_rsp *rsp)
{
u64 id;
int rc;
@@ -4723,14 +4726,16 @@ static int smb2_get_info_file_pipe(struct ksmbd_session *sess,
switch (req->FileInfoClass) {
case FILE_STANDARD_INFORMATION:
- get_standard_info_pipe(rsp, rsp_org);
+ get_standard_info_pipe(rsp);
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp, rsp_org);
+ le32_to_cpu(rsp->OutputBufferLength),
+ rsp);
break;
case FILE_INTERNAL_INFORMATION:
- get_internal_info_pipe(rsp, id, rsp_org);
+ get_internal_info_pipe(rsp, id);
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp, rsp_org);
+ le32_to_cpu(rsp->OutputBufferLength),
+ rsp);
break;
default:
ksmbd_debug(SMB, "smb2_info_file_pipe for %u not supported\n",
@@ -5008,7 +5013,6 @@ static int get_file_all_info(struct ksmbd_work *work,
char *filename;
u64 time;
int ret, buf_free_len, filename_len;
- struct smb2_query_info_req *req = ksmbd_req_buf_next(work);
if (!(fp->daccess & FILE_READ_ATTRIBUTES_LE)) {
ksmbd_debug(SMB, "no right to read the attributes : 0x%x\n",
@@ -5021,10 +5025,9 @@ static int get_file_all_info(struct ksmbd_work *work,
return PTR_ERR(filename);
filename_len = strlen(filename);
- buf_free_len = smb2_calc_max_out_buf_len(work,
+ buf_free_len = smb2_resp_buf_len(work,
offsetof(struct smb2_query_info_rsp, Buffer) +
- offsetof(struct smb2_file_all_info, FileName),
- le32_to_cpu(req->OutputBufferLength));
+ offsetof(struct smb2_file_all_info, FileName));
if (buf_free_len < (filename_len + 1) * 2) {
kfree(filename);
return -EINVAL;
@@ -5109,7 +5112,6 @@ static int get_file_stream_info(struct ksmbd_work *work,
ssize_t xattr_list_len;
int nbytes = 0, streamlen, stream_name_len, next, idx = 0;
int buf_free_len;
- struct smb2_query_info_req *req = ksmbd_req_buf_next(work);
int ret;
ret = vfs_getattr(&fp->filp->f_path, &stat, STATX_BASIC_STATS,
@@ -5119,10 +5121,8 @@ static int get_file_stream_info(struct ksmbd_work *work,
file_info = (struct smb2_file_stream_info *)rsp->Buffer;
- buf_free_len =
- smb2_calc_max_out_buf_len(work,
- offsetof(struct smb2_query_info_rsp, Buffer),
- le32_to_cpu(req->OutputBufferLength));
+ buf_free_len = smb2_resp_buf_len(work,
+ offsetof(struct smb2_query_info_rsp, Buffer));
if (buf_free_len < 0)
goto out;
@@ -5418,13 +5418,13 @@ static int smb2_get_info_file(struct ksmbd_work *work,
struct ksmbd_file *fp;
int fileinfoclass = 0;
int rc = 0;
+ unsigned int fixed_len;
unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID;
if (test_share_config_flag(work->tcon->share_conf,
KSMBD_SHARE_FLAG_PIPE)) {
/* smb2 info file called for pipe */
- rc = smb2_get_info_file_pipe(work->sess, req, rsp,
- work->response_buf);
+ rc = smb2_get_info_file_pipe(work->sess, req, rsp);
goto iov_pin_out;
}
@@ -5521,9 +5521,26 @@ static int smb2_get_info_file(struct ksmbd_work *work,
fileinfoclass);
rc = -EOPNOTSUPP;
}
- if (!rc)
+ if (!rc) {
+ fixed_len = le32_to_cpu(rsp->OutputBufferLength);
+ switch (fileinfoclass) {
+ case FILE_ALL_INFORMATION:
+ fixed_len = FILE_ALL_INFORMATION_SIZE;
+ break;
+ case FILE_ALTERNATE_NAME_INFORMATION:
+ fixed_len = FILE_ALTERNATE_NAME_INFORMATION_SIZE;
+ break;
+ case FILE_NORMALIZED_NAME_INFORMATION:
+ fixed_len = FILE_NORMALIZED_NAME_INFORMATION_SIZE;
+ break;
+ case FILE_STREAM_INFORMATION:
+ fixed_len = FILE_STREAM_INFORMATION_SIZE;
+ break;
+ }
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp, work->response_buf);
+ fixed_len,
+ rsp);
+ }
ksmbd_fd_put(work, fp);
iov_pin_out:
@@ -5544,6 +5561,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
struct kstatfs stfs;
struct path path;
int rc = 0, len;
+ unsigned int fixed_len = 0;
if (!share->path)
return -EIO;
@@ -5578,6 +5596,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->DeviceCharacteristics |=
cpu_to_le32(FILE_READ_ONLY_DEVICE);
rsp->OutputBufferLength = cpu_to_le32(8);
+ fixed_len = 8;
break;
}
case FS_ATTRIBUTE_INFORMATION:
@@ -5606,6 +5625,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->FileSystemNameLen = cpu_to_le32(len);
sz = sizeof(struct filesystem_attribute_info) + len;
rsp->OutputBufferLength = cpu_to_le32(sz);
+ fixed_len = 16;
break;
}
case FS_VOLUME_INFORMATION:
@@ -5632,6 +5652,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->Reserved = 0;
sz = sizeof(struct filesystem_vol_info) + len;
rsp->OutputBufferLength = cpu_to_le32(sz);
+ fixed_len = 24;
break;
}
case FS_SIZE_INFORMATION:
@@ -5644,6 +5665,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->SectorsPerAllocationUnit = cpu_to_le32(1);
info->BytesPerSector = cpu_to_le32(stfs.f_bsize);
rsp->OutputBufferLength = cpu_to_le32(24);
+ fixed_len = 24;
break;
}
case FS_FULL_SIZE_INFORMATION:
@@ -5659,6 +5681,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->SectorsPerAllocationUnit = cpu_to_le32(1);
info->BytesPerSector = cpu_to_le32(stfs.f_bsize);
rsp->OutputBufferLength = cpu_to_le32(32);
+ fixed_len = 32;
break;
}
case FS_OBJECT_ID_INFORMATION:
@@ -5679,6 +5702,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->extended_info.rel_date = 0;
memcpy(info->extended_info.version_string, "1.1.0", strlen("1.1.0"));
rsp->OutputBufferLength = cpu_to_le32(64);
+ fixed_len = 64;
break;
}
case FS_SECTOR_SIZE_INFORMATION:
@@ -5700,6 +5724,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->ByteOffsetForSectorAlignment = 0;
info->ByteOffsetForPartitionAlignment = 0;
rsp->OutputBufferLength = cpu_to_le32(28);
+ fixed_len = 28;
break;
}
case FS_CONTROL_INFORMATION:
@@ -5720,6 +5745,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->DefaultQuotaLimit = cpu_to_le64(SMB2_NO_FID);
info->Padding = 0;
rsp->OutputBufferLength = cpu_to_le32(48);
+ fixed_len = 48;
break;
}
case FS_POSIX_INFORMATION:
@@ -5740,6 +5766,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->TotalFileNodes = cpu_to_le64(stfs.f_files);
info->FreeFileNodes = cpu_to_le64(stfs.f_ffree);
rsp->OutputBufferLength = cpu_to_le32(56);
+ fixed_len = 56;
}
break;
}
@@ -5748,7 +5775,8 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
return -EOPNOTSUPP;
}
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp, work->response_buf);
+ fixed_len,
+ rsp);
path_put(&path);
if (!rc)
@@ -5862,7 +5890,8 @@ static int smb2_get_info_sec(struct ksmbd_work *work,
iov_pin:
rsp->OutputBufferLength = cpu_to_le32(secdesclen);
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp, work->response_buf);
+ le32_to_cpu(rsp->OutputBufferLength),
+ rsp);
if (rc)
goto err_out;
diff --git a/fs/smb/server/smb2pdu.h b/fs/smb/server/smb2pdu.h
index 2ae49c711ce8..f0332eab2fe0 100644
--- a/fs/smb/server/smb2pdu.h
+++ b/fs/smb/server/smb2pdu.h
@@ -219,6 +219,7 @@ struct file_sparse {
#define FILE_ALLOCATION_INFORMATION_SIZE 19
#define FILE_END_OF_FILE_INFORMATION_SIZE 20
#define FILE_ALTERNATE_NAME_INFORMATION_SIZE 8
+#define FILE_NORMALIZED_NAME_INFORMATION_SIZE 8
#define FILE_STREAM_INFORMATION_SIZE 32
#define FILE_PIPE_INFORMATION_SIZE 23
#define FILE_PIPE_LOCAL_INFORMATION_SIZE 24
diff --git a/fs/squashfs/xz_wrapper.c b/fs/squashfs/xz_wrapper.c
index 6c49481a2f8c..7d54cd524d6d 100644
--- a/fs/squashfs/xz_wrapper.c
+++ b/fs/squashfs/xz_wrapper.c
@@ -57,10 +57,10 @@ static void *squashfs_xz_comp_opts(struct squashfs_sb_info *msblk,
opts->dict_size = le32_to_cpu(comp_opts->dictionary_size);
- /* the dictionary size should be 2^n or 2^n+2^(n+1) */
+ /* the dictionary size should be positive and 2^n or 2^n+2^(n+1) */
n = ffs(opts->dict_size) - 1;
- if (opts->dict_size != (1 << n) && opts->dict_size != (1 << n) +
- (1 << (n + 1))) {
+ if (opts->dict_size <= 0 || (opts->dict_size != (1 << n) &&
+ opts->dict_size != (1 << n) + (1 << (n + 1)))) {
err = -EIO;
goto out;
}
diff --git a/fs/super.c b/fs/super.c
index 4e868bbd9b48..9d81956b9588 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -404,11 +404,16 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,
static void __put_super(struct super_block *s)
{
if (!--s->s_count) {
+ struct file_system_type *type = s->s_type;
+
list_del_init(&s->s_list);
+ hlist_del_init(&s->s_instances);
WARN_ON(s->s_dentry_lru.node);
WARN_ON(s->s_inode_lru.node);
WARN_ON(!list_empty(&s->s_mounts));
call_rcu(&s->rcu, destroy_super_rcu);
+ /* The unlink above may touch type->fs_supers, so drop it last. */
+ put_filesystem(type);
}
}
@@ -434,24 +439,17 @@ static void kill_super_notify(struct super_block *sb)
if (sb->s_flags & SB_DEAD)
return;
- /*
- * Remove it from @fs_supers so it isn't found by new
- * sget{_fc}() walkers anymore. Any concurrent mounter still
- * managing to grab a temporary reference is guaranteed to
- * already see SB_DYING and will wait until we notify them about
- * SB_DEAD.
- */
- spin_lock(&sb_lock);
- hlist_del_init(&sb->s_instances);
- spin_unlock(&sb_lock);
-
/*
* Let concurrent mounts know that this thing is really dead.
- * We don't need @sb->s_umount here as every concurrent caller
- * will see SB_DYING and either discard the superblock or wait
- * for SB_DEAD.
+ * sget{_fc}() skips SB_DEAD superblocks and calls test() under
+ * sb_lock, so set it under sb_lock: once we return no test()
+ * runs on this superblock anymore and none will start. Everyone
+ * else already saw SB_DYING and either discarded the superblock
+ * or waits for SB_DEAD.
*/
+ spin_lock(&sb_lock);
super_wake(sb, SB_DEAD);
+ spin_unlock(&sb_lock);
}
/**
@@ -482,7 +480,6 @@ void deactivate_locked_super(struct super_block *s)
list_lru_destroy(&s->s_dentry_lru);
list_lru_destroy(&s->s_inode_lru);
- put_filesystem(fs);
put_super(s);
} else {
super_unlock_excl(s);
@@ -668,12 +665,12 @@ void generic_shutdown_super(struct super_block *sb)
}
/*
* Broadcast to everyone that grabbed a temporary reference to this
- * superblock before we removed it from @fs_supers that the superblock
- * is dying. Every walker of @fs_supers outside of sget{_fc}() will now
- * discard this superblock and treat it as dead.
+ * superblock that it is dying. Every walker of @fs_supers outside
+ * of sget{_fc}() will now discard this superblock and treat it as
+ * dead.
*
- * We leave the superblock on @fs_supers so it can be found by
- * sget{_fc}() until we passed sb->kill_sb().
+ * sget{_fc}() keeps finding the superblock until SB_DEAD is set, so
+ * a concurrent mounter waits until we passed sb->kill_sb().
*/
super_wake(sb, SB_DYING);
super_unlock_excl(sb);
@@ -751,6 +748,9 @@ struct super_block *sget_fc(struct fs_context *fc,
spin_lock(&sb_lock);
if (test) {
hlist_for_each_entry(old, &fc->fs_type->fs_supers, s_instances) {
+ /* Only unlinked at the last passive reference. */
+ if (super_flags(old, SB_DEAD))
+ continue;
if (test(old, fc))
goto share_extant_sb;
}
@@ -834,6 +834,9 @@ struct super_block *sget(struct file_system_type *type,
spin_lock(&sb_lock);
if (test) {
hlist_for_each_entry(old, &type->fs_supers, s_instances) {
+ /* Only unlinked at the last passive reference. */
+ if (super_flags(old, SB_DEAD))
+ continue;
if (!test(old, data))
continue;
if (user_ns != old->s_user_ns) {
diff --git a/fs/udf/inode.c b/fs/udf/inode.c
index 2513816c48f5..402508dd839d 100644
--- a/fs/udf/inode.c
+++ b/fs/udf/inode.c
@@ -330,65 +330,6 @@ const struct address_space_operations udf_aops = {
.migrate_folio = buffer_migrate_folio,
};
-/*
- * Expand file stored in ICB to a normal one-block-file
- *
- * This function requires i_mutex held
- */
-int udf_expand_file_adinicb(struct inode *inode)
-{
- struct folio *folio;
- struct udf_inode_info *iinfo = UDF_I(inode);
- int err;
-
- WARN_ON_ONCE(!inode_is_locked(inode));
- if (!iinfo->i_lenAlloc) {
- down_write(&iinfo->i_data_sem);
- if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD))
- iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT;
- else
- iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG;
- up_write(&iinfo->i_data_sem);
- mark_inode_dirty(inode);
- return 0;
- }
-
- folio = __filemap_get_folio(inode->i_mapping, 0,
- FGP_LOCK | FGP_ACCESSED | FGP_CREAT, GFP_KERNEL);
- if (IS_ERR(folio))
- return PTR_ERR(folio);
-
- if (!folio_test_uptodate(folio))
- udf_adinicb_read_folio(folio);
- down_write(&iinfo->i_data_sem);
- memset(iinfo->i_data + iinfo->i_lenEAttr, 0x00,
- iinfo->i_lenAlloc);
- iinfo->i_lenAlloc = 0;
- if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD))
- iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT;
- else
- iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG;
- folio_mark_dirty(folio);
- folio_unlock(folio);
- up_write(&iinfo->i_data_sem);
- err = filemap_fdatawrite(inode->i_mapping);
- if (err) {
- /* Restore everything back so that we don't lose data... */
- folio_lock(folio);
- down_write(&iinfo->i_data_sem);
- memcpy_from_folio(iinfo->i_data + iinfo->i_lenEAttr,
- folio, 0, inode->i_size);
- folio_unlock(folio);
- iinfo->i_alloc_type = ICBTAG_FLAG_AD_IN_ICB;
- iinfo->i_lenAlloc = inode->i_size;
- up_write(&iinfo->i_data_sem);
- }
- folio_put(folio);
- mark_inode_dirty(inode);
-
- return err;
-}
-
#define UDF_MAP_CREATE 0x01 /* Mapping can allocate new blocks */
#define UDF_MAP_NOPREALLOC 0x02 /* Do not preallocate blocks */
@@ -449,6 +390,76 @@ static int udf_map_block(struct inode *inode, struct udf_map_rq *map)
return ret;
}
+/*
+ * Expand file stored in ICB to a normal one-block-file
+ *
+ * This function requires i_mutex held
+ */
+int udf_expand_file_adinicb(struct inode *inode)
+{
+ struct folio *folio;
+ struct udf_inode_info *iinfo = UDF_I(inode);
+ struct udf_map_rq map = {
+ .lblk = 0,
+ .iflags = UDF_MAP_CREATE,
+ };
+ int err;
+
+ WARN_ON_ONCE(!inode_is_locked(inode));
+ if (!iinfo->i_lenAlloc) {
+ down_write(&iinfo->i_data_sem);
+ if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD))
+ iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT;
+ else
+ iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG;
+ up_write(&iinfo->i_data_sem);
+ mark_inode_dirty(inode);
+ return 0;
+ }
+
+ folio = __filemap_get_folio(inode->i_mapping, 0,
+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT, GFP_KERNEL);
+ if (IS_ERR(folio))
+ return PTR_ERR(folio);
+
+ if (!folio_test_uptodate(folio))
+ udf_adinicb_read_folio(folio);
+ down_write(&iinfo->i_data_sem);
+ memset(iinfo->i_data + iinfo->i_lenEAttr, 0x00,
+ iinfo->i_lenAlloc);
+ iinfo->i_lenAlloc = 0;
+ if (UDF_QUERY_FLAG(inode->i_sb, UDF_FLAG_USE_SHORT_AD))
+ iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT;
+ else
+ iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG;
+ up_write(&iinfo->i_data_sem);
+
+ /* Allocate the block underlying the data */
+ err = udf_map_block(inode, &map);
+ if (err < 0)
+ goto restore;
+
+ folio_mark_dirty(folio);
+ folio_unlock(folio);
+ err = filemap_fdatawrite(inode->i_mapping);
+ if (err) {
+ /* Restore everything back so that we don't lose data... */
+ folio_lock(folio);
+restore:
+ down_write(&iinfo->i_data_sem);
+ memcpy_from_folio(iinfo->i_data + iinfo->i_lenEAttr,
+ folio, 0, inode->i_size);
+ iinfo->i_alloc_type = ICBTAG_FLAG_AD_IN_ICB;
+ iinfo->i_lenAlloc = inode->i_size;
+ up_write(&iinfo->i_data_sem);
+ folio_unlock(folio);
+ }
+ folio_put(folio);
+ mark_inode_dirty(inode);
+
+ return err;
+}
+
static int __udf_get_block(struct inode *inode, sector_t block,
struct buffer_head *bh_result, int flags)
{
diff --git a/fs/xfs/libxfs/xfs_da_btree.c b/fs/xfs/libxfs/xfs_da_btree.c
index 90f7fc219fcc..90fbdef23def 100644
--- a/fs/xfs/libxfs/xfs_da_btree.c
+++ b/fs/xfs/libxfs/xfs_da_btree.c
@@ -130,7 +130,7 @@ xfs_da_state_reset(
state->mp = state->args->dp->i_mount;
}
-static inline int xfs_dabuf_nfsb(struct xfs_mount *mp, int whichfork)
+inline int xfs_dabuf_nfsb(struct xfs_mount *mp, int whichfork)
{
if (whichfork == XFS_DATA_FORK)
return mp->m_dir_geo->fsbcount;
diff --git a/fs/xfs/libxfs/xfs_da_btree.h b/fs/xfs/libxfs/xfs_da_btree.h
index 354d5d65043e..cd2e213f5e74 100644
--- a/fs/xfs/libxfs/xfs_da_btree.h
+++ b/fs/xfs/libxfs/xfs_da_btree.h
@@ -242,4 +242,6 @@ xfs_failaddr_t xfs_da3_node_header_check(struct xfs_buf *bp, xfs_ino_t owner);
extern struct kmem_cache *xfs_da_state_cache;
+int xfs_dabuf_nfsb(struct xfs_mount *mp, int whichfork);
+
#endif /* __XFS_DA_BTREE_H__ */
diff --git a/fs/xfs/scrub/attr_repair.c b/fs/xfs/scrub/attr_repair.c
index fa68ecabd983..6f668ddea819 100644
--- a/fs/xfs/scrub/attr_repair.c
+++ b/fs/xfs/scrub/attr_repair.c
@@ -1294,8 +1294,8 @@ xrep_xattr_swap_prep(
.geo = sc->mp->m_attr_geo,
.whichfork = XFS_ATTR_FORK,
.trans = sc->tp,
- .total = 1,
- .owner = sc->ip->i_ino,
+ .total = xfs_dabuf_nfsb(sc->mp, XFS_ATTR_FORK),
+ .owner = I_INO(sc->ip),
};
error = xfs_attr_shortform_to_leaf(&args);
diff --git a/fs/xfs/scrub/dir_repair.c b/fs/xfs/scrub/dir_repair.c
index 53272c254020..2bc2354cd85d 100644
--- a/fs/xfs/scrub/dir_repair.c
+++ b/fs/xfs/scrub/dir_repair.c
@@ -1379,9 +1379,24 @@ xrep_dir_live_update(
if (p->delta > 0)
error = xrep_dir_stash_createname(rd, p->name,
p->ip->i_ino);
- else
- error = xrep_dir_stash_removename(rd, p->name,
+ else {
+ /*
+ * xfs_dentry_to_name in unlink or rename-exchange can
+ * pass us names with ftype FT_UNKNOWN, but we really
+ * must know the ftype of the child that is being
+ * removed so that we can do nlink updates correctly
+ * without holding inode references.
+ */
+ struct xfs_name name = {
+ .name = p->name->name,
+ .len = p->name->len,
+ .type = xfs_mode_to_ftype(
+ VFS_IC(p->ip)->i_mode),
+ };
+
+ error = xrep_dir_stash_removename(rd, &name,
p->ip->i_ino);
+ }
mutex_unlock(&rd->pscan.lock);
if (error)
goto out_abort;
@@ -1471,8 +1486,8 @@ xrep_dir_swap_prep(
.geo = sc->mp->m_dir_geo,
.whichfork = XFS_DATA_FORK,
.trans = sc->tp,
- .total = 1,
- .owner = sc->ip->i_ino,
+ .total = xfs_dabuf_nfsb(sc->mp, XFS_DATA_FORK),
+ .owner = I_INO(sc->ip),
};
error = xfs_dir2_sf_to_block(&args);
diff --git a/fs/xfs/scrub/health.c b/fs/xfs/scrub/health.c
index b712a8bd34f5..d00bdfeb0322 100644
--- a/fs/xfs/scrub/health.c
+++ b/fs/xfs/scrub/health.c
@@ -192,9 +192,9 @@ xchk_update_health(
* there's no sick flag defined for it, so we branch here ahead of the
* mask check.
*/
- if (sc->sm->sm_type == XFS_SCRUB_TYPE_HEALTHY &&
- !(sc->sm->sm_flags & XFS_SCRUB_OFLAG_CORRUPT)) {
- xchk_mark_all_healthy(sc->mp);
+ if (sc->sm->sm_type == XFS_SCRUB_TYPE_HEALTHY) {
+ if (!(sc->sm->sm_flags & XFS_SCRUB_OFLAG_CORRUPT))
+ xchk_mark_all_healthy(sc->mp);
return;
}
diff --git a/fs/xfs/scrub/inode.c b/fs/xfs/scrub/inode.c
index d32716fb2fec..a57a65fb080a 100644
--- a/fs/xfs/scrub/inode.c
+++ b/fs/xfs/scrub/inode.c
@@ -560,7 +560,7 @@ xchk_dinode(
}
/* di_forkoff */
- if (XFS_DFORK_BOFF(dip) >= mp->m_sb.sb_inodesize)
+ if (dip->di_forkoff >= (XFS_LITINO(mp) >> 3))
xchk_ino_set_corrupt(sc, ino);
if (naextents != 0 && dip->di_forkoff == 0)
xchk_ino_set_corrupt(sc, ino);
diff --git a/fs/xfs/scrub/inode_repair.c b/fs/xfs/scrub/inode_repair.c
index 7e480472e074..dee2dc648f91 100644
--- a/fs/xfs/scrub/inode_repair.c
+++ b/fs/xfs/scrub/inode_repair.c
@@ -1513,7 +1513,7 @@ xrep_inode_blockcounts(
&nextents, &acount);
if (error)
return error;
- if (count >= sc->mp->m_sb.sb_dblocks)
+ if (acount >= sc->mp->m_sb.sb_dblocks)
return -EFSCORRUPTED;
error = xrep_ino_ensure_extent_count(sc, XFS_ATTR_FORK,
nextents);
diff --git a/fs/xfs/scrub/orphanage.c b/fs/xfs/scrub/orphanage.c
index 46171f61eda4..c302fb8f4de4 100644
--- a/fs/xfs/scrub/orphanage.c
+++ b/fs/xfs/scrub/orphanage.c
@@ -193,12 +193,16 @@ xrep_orphanage_create(
/* Make sure the orphanage is owned by root. */
error = xrep_chown_orphanage(sc, XFS_I(orphanage_inode));
if (error)
- goto out_dput_orphanage;
+ goto out_rele_orphanage;
/* Stash the reference for later and bail out. */
sc->orphanage = XFS_I(orphanage_inode);
sc->orphanage_ilock_flags = 0;
+ orphanage_inode = NULL;
+out_rele_orphanage:
+ if (orphanage_inode)
+ xchk_irele(sc, XFS_I(orphanage_inode));
out_dput_orphanage:
dput(orphanage_dentry);
out_unlock_root:
diff --git a/fs/xfs/scrub/reap.c b/fs/xfs/scrub/reap.c
index 8688edec5875..adce759f1bff 100644
--- a/fs/xfs/scrub/reap.c
+++ b/fs/xfs/scrub/reap.c
@@ -861,7 +861,7 @@ xreap_bmapi_binval(
* much of the mapping we've seen so far.
*/
if (invalidated > XREAP_MAX_BINVAL) {
- imap->br_blockcount = agbno_next - bno;
+ imap->br_blockcount = bno - agbno;
goto out;
}
}
diff --git a/fs/xfs/scrub/refcount.c b/fs/xfs/scrub/refcount.c
index dc3a75ac26ce..920e72a1f50d 100644
--- a/fs/xfs/scrub/refcount.c
+++ b/fs/xfs/scrub/refcount.c
@@ -411,7 +411,7 @@ xchk_refcount_mergeable(
const struct xfs_refcount_irec *r1 = &rrc->prev_rec;
/* Ignore if prev_rec is not yet initialized. */
- if (r1->rc_blockcount > 0)
+ if (r1->rc_blockcount == 0)
return false;
if (r1->rc_domain != r2->rc_domain)
@@ -581,8 +581,12 @@ xchk_xref_is_cow_staging(
if (rc.rc_domain != XFS_REFC_DOMAIN_COW)
xchk_btree_xref_set_corrupt(sc, sc->sa.refc_cur, 0);
+ /* Can't start after bno */
+ if (rc.rc_startblock > agbno)
+ xchk_btree_xref_set_corrupt(sc, sc->sa.refc_cur, 0);
+
/* Must be at least as long as what was passed in */
- if (rc.rc_blockcount < len)
+ if (rc.rc_startblock + rc.rc_blockcount < agbno + len)
xchk_btree_xref_set_corrupt(sc, sc->sa.refc_cur, 0);
}
diff --git a/fs/xfs/scrub/scrub.h b/fs/xfs/scrub/scrub.h
index d68ac1c6533c..457499daeaef 100644
--- a/fs/xfs/scrub/scrub.h
+++ b/fs/xfs/scrub/scrub.h
@@ -40,7 +40,7 @@ static inline int xchk_maybe_relax(struct xchk_relax *widget)
return 0;
widget->resched_nr = 0;
- if (unlikely(widget->next_resched <= jiffies)) {
+ if (unlikely(time_after_eq(jiffies, widget->next_resched))) {
cond_resched();
widget->next_resched = XCHK_RELAX_NEXT;
}
diff --git a/fs/xfs/scrub/stats.c b/fs/xfs/scrub/stats.c
index 0558d587be4d..00bd9246c55b 100644
--- a/fs/xfs/scrub/stats.c
+++ b/fs/xfs/scrub/stats.c
@@ -80,6 +80,7 @@ static const char *name_map[XFS_SCRUB_TYPE_NR] = {
[XFS_SCRUB_TYPE_QUOTACHECK] = "quotacheck",
[XFS_SCRUB_TYPE_NLINKS] = "nlinks",
[XFS_SCRUB_TYPE_DIRTREE] = "dirtree",
+ [XFS_SCRUB_TYPE_HEALTHY] = "healthy",
};
/* Format the scrub stats into a text buffer, similar to pcp style. */
diff --git a/fs/xfs/scrub/tempfile.c b/fs/xfs/scrub/tempfile.c
index 56fef13bc773..8fa8b6d7ca27 100644
--- a/fs/xfs/scrub/tempfile.c
+++ b/fs/xfs/scrub/tempfile.c
@@ -536,6 +536,19 @@ xrep_tempexch_prep_request(
return 0;
}
+static inline unsigned int
+xrep_tempexch_estimate_sf_resblks(
+ struct xfs_scrub *sc,
+ int whichfork)
+{
+ /* repairing a symlink target */
+ if (S_ISLNK(VFS_I(sc->ip)->i_mode) && whichfork == XFS_DATA_FORK)
+ return 1;
+
+ /* everything else is a directory or an xattr structure */
+ return xfs_dabuf_nfsb(sc->mp, whichfork);
+}
+
/*
* Fill out the mapping exchange resource estimation structures in preparation
* for exchanging the contents of a metadata file that we've rebuilt in the
@@ -550,6 +563,8 @@ xrep_tempexch_estimate(
struct xfs_ifork *ifp;
struct xfs_ifork *tifp;
int whichfork = xfs_exchmaps_reqfork(req);
+ unsigned int sf_resblks =
+ xrep_tempexch_estimate_sf_resblks(sc, whichfork);
int state = 0;
/*
@@ -580,9 +595,9 @@ xrep_tempexch_estimate(
* plus the block we converted.
*/
req->ip1_bcount = sc->tempip->i_nblocks;
- req->ip2_bcount = 1;
+ req->ip2_bcount = sf_resblks;
req->nr_exchanges = 1 + tifp->if_nextents;
- req->resblks = 1;
+ req->resblks = sf_resblks;
break;
case 2:
/*
@@ -594,10 +609,10 @@ xrep_tempexch_estimate(
* is (worst case) the extent count of the file being repaired
* plus the block we converted.
*/
- req->ip1_bcount = 1;
+ req->ip1_bcount = sf_resblks;
req->ip2_bcount = sc->ip->i_nblocks;
req->nr_exchanges = 1 + ifp->if_nextents;
- req->resblks = 1;
+ req->resblks = sf_resblks;
break;
case 3:
/*
@@ -609,10 +624,10 @@ xrep_tempexch_estimate(
* fileoff 0. Presumably, the caller could not exchange the
* two inode fork areas directly.
*/
- req->ip1_bcount = 1;
- req->ip2_bcount = 1;
+ req->ip1_bcount = sf_resblks;
+ req->ip2_bcount = sf_resblks;
req->nr_exchanges = 1;
- req->resblks = 2;
+ req->resblks = 2 * sf_resblks;
break;
}
diff --git a/fs/xfs/xfs_exchrange.c b/fs/xfs/xfs_exchrange.c
index a37d31347b87..f54ed30b76e4 100644
--- a/fs/xfs/xfs_exchrange.c
+++ b/fs/xfs/xfs_exchrange.c
@@ -630,6 +630,9 @@ xfs_exchrange_prep(
if (error)
return error;
+ if (fxr->flags & XFS_EXCHANGE_RANGE_DRY_RUN)
+ return 0;
+
trace_xfs_exchrange_flush(fxr, ip1, ip2);
/* Flush the relevant ranges of both files. */
@@ -706,9 +709,11 @@ xfs_exchrange_contents(
* other file write would do. This may involve turning on support for
* logged xattrs if either file has security capabilities.
*/
- error = xfs_exchange_range_finish(fxr);
- if (error)
- goto out_unlock;
+ if (!(fxr->flags & XFS_EXCHANGE_RANGE_DRY_RUN)) {
+ error = xfs_exchange_range_finish(fxr);
+ if (error)
+ goto out_unlock;
+ }
out_unlock:
xfs_iunlock2_io_mmap(ip1, ip2);
@@ -905,7 +910,7 @@ xfs_ioc_commit_range(
if (copy_from_user(&args, argp, sizeof(args)))
return -EFAULT;
- if (args.flags & ~XFS_EXCHANGE_RANGE_ALL_FLAGS)
+ if (args.pad || (args.flags & ~XFS_EXCHANGE_RANGE_ALL_FLAGS))
return -EINVAL;
if (kern_f->magic != XCR_FRESH_MAGIC)
return -EBUSY;
diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
index 6b119a7a324f..08689839dd77 100644
--- a/fs/xfs/xfs_icache.c
+++ b/fs/xfs/xfs_icache.c
@@ -1593,7 +1593,7 @@ xfs_blockgc_free_dquots(
do_work = true;
}
- if (XFS_IS_UQUOTA_ENFORCED(mp) && gdqp && xfs_dquot_lowsp(gdqp)) {
+ if (XFS_IS_GQUOTA_ENFORCED(mp) && gdqp && xfs_dquot_lowsp(gdqp)) {
icw.icw_gid = make_kgid(mp->m_super->s_user_ns, gdqp->q_id);
icw.icw_flags |= XFS_ICWALK_FLAG_GID;
do_work = true;
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index 03944b6c5fba..b8b795aa6f02 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -178,6 +178,9 @@ static inline const struct inode *VFS_IC(const struct xfs_inode *ip)
return &ip->i_vnode;
}
+/* Keep the full inode number on systems with a 32-bit VFS i_ino. */
+#define I_INO(ip) ((ip)->i_ino)
+
/*
* For regular files we only update the on-disk filesize when actually
* writing data back to disk. Until then only the copy in the VFS inode
diff --git a/include/drm/drm_client.h b/include/drm/drm_client.h
index bc0e66f9c425..a31d65d869b3 100644
--- a/include/drm/drm_client.h
+++ b/include/drm/drm_client.h
@@ -47,12 +47,14 @@ struct drm_client_funcs {
*
* Note that the core does not guarantee exclusion against concurrent
* drm_open(). Clients need to ensure this themselves, for example by
- * using drm_master_internal_acquire() and
- * drm_master_internal_release().
+ * using drm_master_internal_acquire() and drm_master_internal_release().
+ *
+ * If the caller passes force, the client should ignore any present DRM
+ * master and restore the display anyway.
*
* This callback is optional.
*/
- int (*restore)(struct drm_client_dev *client);
+ int (*restore)(struct drm_client_dev *client, bool force);
/**
* @hotplug:
@@ -123,7 +125,7 @@ void drm_client_register(struct drm_client_dev *client);
void drm_client_dev_unregister(struct drm_device *dev);
void drm_client_dev_hotplug(struct drm_device *dev);
-void drm_client_dev_restore(struct drm_device *dev);
+void drm_client_dev_restore(struct drm_device *dev, bool force);
/**
* struct drm_client_buffer - DRM client buffer
diff --git a/include/drm/drm_fb_helper.h b/include/drm/drm_fb_helper.h
index eb825acedf74..f80abb89bad1 100644
--- a/include/drm/drm_fb_helper.h
+++ b/include/drm/drm_fb_helper.h
@@ -244,7 +244,8 @@ int drm_fb_helper_set_par(struct fb_info *info);
int drm_fb_helper_check_var(struct fb_var_screeninfo *var,
struct fb_info *info);
-int drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper);
+int drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper,
+ bool force);
void drm_fb_helper_unregister_info(struct drm_fb_helper *fb_helper);
void drm_fb_helper_fill_info(struct fb_info *info,
@@ -269,7 +270,6 @@ int drm_fb_helper_hotplug_event(struct drm_fb_helper *fb_helper);
int drm_fb_helper_initial_config(struct drm_fb_helper *fb_helper);
int drm_fb_helper_debug_enter(struct fb_info *info);
int drm_fb_helper_debug_leave(struct fb_info *info);
-void drm_fb_helper_lastclose(struct drm_device *dev);
#else
static inline void drm_fb_helper_prepare(struct drm_device *dev,
struct drm_fb_helper *helper,
@@ -321,7 +321,8 @@ static inline int drm_fb_helper_check_var(struct fb_var_screeninfo *var,
}
static inline int
-drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper)
+drm_fb_helper_restore_fbdev_mode_unlocked(struct drm_fb_helper *fb_helper,
+ bool force)
{
return 0;
}
@@ -383,10 +384,6 @@ static inline int drm_fb_helper_debug_leave(struct fb_info *info)
{
return 0;
}
-
-static inline void drm_fb_helper_lastclose(struct drm_device *dev)
-{
-}
#endif
#endif
diff --git a/include/drm/drm_file.h b/include/drm/drm_file.h
index 2bc88d2d4a84..92d0f9c5d788 100644
--- a/include/drm/drm_file.h
+++ b/include/drm/drm_file.h
@@ -485,6 +485,7 @@ struct drm_memory_stats {
enum drm_gem_object_status;
+int drm_memory_stats_is_zero(const struct drm_memory_stats *stats);
void drm_print_memory_stats(struct drm_printer *p,
const struct drm_memory_stats *stats,
enum drm_gem_object_status supported_status,
diff --git a/include/keys/request_key_auth-type.h b/include/keys/request_key_auth-type.h
index 01e42ee5f409..464636278c4f 100644
--- a/include/keys/request_key_auth-type.h
+++ b/include/keys/request_key_auth-type.h
@@ -22,7 +22,7 @@ struct request_key_auth {
const struct cred *cred;
void *callout_info;
size_t callout_len;
- pid_t pid;
+ struct pid *pid;
char op[8];
} __randomize_layout;
diff --git a/include/linux/bpf_mem_alloc.h b/include/linux/bpf_mem_alloc.h
index e45162ef59bb..4ce0d27f8ea2 100644
--- a/include/linux/bpf_mem_alloc.h
+++ b/include/linux/bpf_mem_alloc.h
@@ -14,6 +14,8 @@ struct bpf_mem_alloc {
struct obj_cgroup *objcg;
bool percpu;
struct work_struct work;
+ void (*dtor_ctx_free)(void *ctx);
+ void *dtor_ctx;
};
/* 'size != 0' is for bpf_mem_alloc which manages fixed-size objects.
@@ -32,6 +34,10 @@ int bpf_mem_alloc_percpu_init(struct bpf_mem_alloc *ma, struct obj_cgroup *objcg
/* The percpu allocation with a specific unit size. */
int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, int size);
void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma);
+void bpf_mem_alloc_set_dtor(struct bpf_mem_alloc *ma,
+ void (*dtor)(void *obj, void *ctx),
+ void (*dtor_ctx_free)(void *ctx),
+ void *ctx);
/* Check the allocation size for kmalloc equivalent allocator */
int bpf_mem_alloc_check_size(bool percpu, size_t size);
diff --git a/include/linux/compiler-context-analysis.h b/include/linux/compiler-context-analysis.h
new file mode 100644
index 000000000000..f8af63045281
--- /dev/null
+++ b/include/linux/compiler-context-analysis.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Macros and attributes for compiler-based static context analysis.
+ */
+
+#ifndef _LINUX_COMPILER_CONTEXT_ANALYSIS_H
+#define _LINUX_COMPILER_CONTEXT_ANALYSIS_H
+
+#ifdef __CHECKER__
+
+/* Sparse context/lock checking support. */
+# define __must_hold(x) __attribute__((context(x,1,1)))
+# define __acquires(x) __attribute__((context(x,0,1)))
+# define __cond_acquires(x) __attribute__((context(x,0,-1)))
+# define __releases(x) __attribute__((context(x,1,0)))
+# define __acquire(x) __context__(x,1)
+# define __release(x) __context__(x,-1)
+# define __cond_lock(x, c) ((c) ? ({ __acquire(x); 1; }) : 0)
+
+#else /* !__CHECKER__ */
+
+# define __must_hold(x)
+# define __acquires(x)
+# define __cond_acquires(x)
+# define __releases(x)
+# define __acquire(x) (void)0
+# define __release(x) (void)0
+# define __cond_lock(x, c) (c)
+
+#endif /* __CHECKER__ */
+
+#endif /* _LINUX_COMPILER_CONTEXT_ANALYSIS_H */
diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index b624f3e8716b..286822a3c913 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -37,6 +37,8 @@
# define BTF_TYPE_TAG(value) /* nothing */
#endif
+#include <linux/compiler-context-analysis.h>
+
/* sparse defines __CHECKER__; see Documentation/dev-tools/sparse.rst */
#ifdef __CHECKER__
/* address spaces */
@@ -47,14 +49,6 @@
# define __rcu __attribute__((noderef, address_space(__rcu)))
static inline void __chk_user_ptr(const volatile void __user *ptr) { }
static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
-/* context/locking */
-# define __must_hold(x) __attribute__((context(x,1,1)))
-# define __acquires(x) __attribute__((context(x,0,1)))
-# define __cond_acquires(x) __attribute__((context(x,0,-1)))
-# define __releases(x) __attribute__((context(x,1,0)))
-# define __acquire(x) __context__(x,1)
-# define __release(x) __context__(x,-1)
-# define __cond_lock(x,c) ((c) ? ({ __acquire(x); 1; }) : 0)
/* other */
# define __force __attribute__((force))
# define __nocast __attribute__((nocast))
@@ -75,14 +69,6 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
# define __chk_user_ptr(x) (void)0
# define __chk_io_ptr(x) (void)0
-/* context/locking */
-# define __must_hold(x)
-# define __acquires(x)
-# define __cond_acquires(x)
-# define __releases(x)
-# define __acquire(x) (void)0
-# define __release(x) (void)0
-# define __cond_lock(x,c) (c)
/* other */
# define __force
# define __nocast
diff --git a/include/linux/dev_printk.h b/include/linux/dev_printk.h
index ca32b5bb28eb..eb2094e43050 100644
--- a/include/linux/dev_printk.h
+++ b/include/linux/dev_printk.h
@@ -276,6 +276,7 @@ do { \
dev_driver_string(dev), dev_name(dev), ## arg)
__printf(3, 4) int dev_err_probe(const struct device *dev, int err, const char *fmt, ...);
+__printf(3, 4) int dev_warn_probe(const struct device *dev, int err, const char *fmt, ...);
/* Simple helper for dev_err_probe() when ERR_PTR() is to be returned. */
#define dev_err_ptr_probe(dev, ___err, fmt, ...) \
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index b137fdb56093..631519552c5a 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -1521,6 +1521,7 @@ struct dma_chan *__dma_request_channel(const dma_cap_mask_t *mask,
struct dma_chan *dma_request_chan(struct device *dev, const char *name);
struct dma_chan *dma_request_chan_by_mask(const dma_cap_mask_t *mask);
+struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name);
void dma_release_channel(struct dma_chan *chan);
int dma_get_slave_caps(struct dma_chan *chan, struct dma_slave_caps *caps);
@@ -1557,6 +1558,12 @@ static inline struct dma_chan *dma_request_chan_by_mask(
{
return ERR_PTR(-ENODEV);
}
+
+static inline struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name)
+{
+ return ERR_PTR(-ENODEV);
+}
+
static inline void dma_release_channel(struct dma_chan *chan)
{
}
diff --git a/include/linux/firmware/qcom/qcom_scm.h b/include/linux/firmware/qcom/qcom_scm.h
index 9f14976399ab..6f9862033fc3 100644
--- a/include/linux/firmware/qcom/qcom_scm.h
+++ b/include/linux/firmware/qcom/qcom_scm.h
@@ -66,19 +66,23 @@ int qcom_scm_set_warm_boot_addr(void *entry);
void qcom_scm_cpu_power_down(u32 flags);
int qcom_scm_set_remote_state(u32 state, u32 id);
-struct qcom_scm_pas_metadata {
+struct qcom_scm_pas_context {
+ struct device *dev;
+ u32 pas_id;
+ phys_addr_t mem_phys;
+ size_t mem_size;
void *ptr;
dma_addr_t phys;
ssize_t size;
};
-int qcom_scm_pas_init_image(u32 peripheral, const void *metadata, size_t size,
- struct qcom_scm_pas_metadata *ctx);
-void qcom_scm_pas_metadata_release(struct qcom_scm_pas_metadata *ctx);
-int qcom_scm_pas_mem_setup(u32 peripheral, phys_addr_t addr, phys_addr_t size);
-int qcom_scm_pas_auth_and_reset(u32 peripheral);
-int qcom_scm_pas_shutdown(u32 peripheral);
-bool qcom_scm_pas_supported(u32 peripheral);
+int qcom_scm_pas_init_image(u32 pas_id, const void *metadata, size_t size,
+ struct qcom_scm_pas_context *ctx);
+void qcom_pas_metadata_release(struct qcom_scm_pas_context *ctx);
+int qcom_scm_pas_mem_setup(u32 pas_id, phys_addr_t addr, phys_addr_t size);
+int qcom_scm_pas_auth_and_reset(u32 pas_id);
+int qcom_scm_pas_shutdown(u32 pas_id);
+bool qcom_scm_pas_supported(u32 pas_id);
int qcom_scm_io_readl(phys_addr_t addr, unsigned int *val);
int qcom_scm_io_writel(phys_addr_t addr, unsigned int val);
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 1ddc2a3ee463..97e5f692a887 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -759,8 +759,9 @@ unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec);
unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec);
extern ftrace_func_t ftrace_trace_function;
+struct trace_array;
-int ftrace_regex_open(struct ftrace_ops *ops, int flag,
+int ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag,
struct inode *inode, struct file *file);
ssize_t ftrace_filter_write(struct file *file, const char __user *ubuf,
size_t cnt, loff_t *ppos);
@@ -970,7 +971,7 @@ static inline unsigned long ftrace_location(unsigned long ip)
* have them defined when ftrace is not enabled, but these
* functions may still be called. Use a macro instead of inline.
*/
-#define ftrace_regex_open(ops, flag, inod, file) ({ -ENODEV; })
+#define ftrace_regex_open(tr, ops, flag, inode, file) ({ -ENODEV; })
#define ftrace_set_early_filter(ops, buf, enable) do { } while (0)
#define ftrace_set_filter_ip(ops, ip, remove, reset) ({ -ENODEV; })
#define ftrace_set_filter_ips(ops, ips, cnt, remove, reset) ({ -ENODEV; })
diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h
index 9551dba15cc2..d403ce386c70 100644
--- a/include/linux/if_vlan.h
+++ b/include/linux/if_vlan.h
@@ -356,6 +356,9 @@ static inline int __vlan_insert_inner_tag(struct sk_buff *skb,
{
struct vlan_ethhdr *veth;
+ if (unlikely(!pskb_may_pull(skb, mac_len)))
+ return -EINVAL;
+
if (skb_cow_head(skb, VLAN_HLEN) < 0)
return -ENOMEM;
diff --git a/include/linux/ipv6.h b/include/linux/ipv6.h
index 5aeeed22f35b..ad5f09765e0c 100644
--- a/include/linux/ipv6.h
+++ b/include/linux/ipv6.h
@@ -212,18 +212,21 @@ struct inet6_cork {
/* struct ipv6_pinfo - ipv6 private area */
struct ipv6_pinfo {
+ /* Used in tx path (inet6_csk_route_socket(), ip6_xmit()) */
struct in6_addr saddr;
- struct in6_pktinfo sticky_pktinfo;
- const struct in6_addr *daddr_cache;
+ __be32 flow_label;
+ u32 dst_cookie;
+ struct ipv6_txoptions __rcu *opt;
+ s16 hop_limit;
+ u8 pmtudisc;
+ u8 tclass;
#ifdef CONFIG_IPV6_SUBTREES
- const struct in6_addr *saddr_cache;
+ bool saddr_cache;
#endif
+ const struct in6_addr *daddr_cache;
- __be32 flow_label;
- __u32 frag_size;
-
- s16 hop_limit;
u8 mcast_hops;
+ u32 frag_size;
int ucast_oif;
int mcast_oif;
@@ -231,7 +234,7 @@ struct ipv6_pinfo {
/* pktoption flags */
union {
struct {
- __u16 srcrt:1,
+ u16 srcrt:1,
osrcrt:1,
rxinfo:1,
rxoinfo:1,
@@ -248,29 +251,24 @@ struct ipv6_pinfo {
recvfragsize:1;
/* 1 bits hole */
} bits;
- __u16 all;
+ u16 all;
} rxopt;
/* sockopt flags */
- __u8 srcprefs; /* 001: prefer temporary address
+ u8 srcprefs; /* 001: prefer temporary address
* 010: prefer public address
* 100: prefer care-of address
*/
- __u8 pmtudisc;
- __u8 min_hopcount;
- __u8 tclass;
+ u8 min_hopcount;
__be32 rcv_flowinfo;
+ struct in6_pktinfo sticky_pktinfo;
- __u32 dst_cookie;
-
- struct ipv6_mc_socklist __rcu *ipv6_mc_list;
- struct ipv6_ac_socklist *ipv6_ac_list;
- struct ipv6_fl_socklist __rcu *ipv6_fl_list;
-
- struct ipv6_txoptions __rcu *opt;
struct sk_buff *pktoptions;
struct sk_buff *rxpmtu;
struct inet6_cork cork;
+
+ struct ipv6_mc_socklist __rcu *ipv6_mc_list;
+ struct ipv6_ac_socklist *ipv6_ac_list;
};
/* We currently use available bits from inet_sk(sk)->inet_flags,
diff --git a/include/linux/libnvdimm.h b/include/linux/libnvdimm.h
index e772aae71843..04043b7d5712 100644
--- a/include/linux/libnvdimm.h
+++ b/include/linux/libnvdimm.h
@@ -122,6 +122,15 @@ struct nd_mapping_desc {
};
struct nd_region;
+
+/*
+ * Provider flush callback return values:
+ * 0: flush completed synchronously
+ * <0: flush failed
+ * >0: flush completion was queued and @bio will be completed later
+ */
+#define NVDIMM_FLUSH_ASYNC 1
+
struct nd_region_desc {
struct resource *res;
struct nd_mapping_desc *mapping;
diff --git a/include/linux/lockd/lockd.h b/include/linux/lockd/lockd.h
index 61c4b9c41904..4044502355ae 100644
--- a/include/linux/lockd/lockd.h
+++ b/include/linux/lockd/lockd.h
@@ -268,7 +268,7 @@ void nsm_release(struct nsm_handle *nsm);
* This is used in garbage collection and resource reclaim
* A return value != 0 means destroy the lock/block/share
*/
-typedef int (*nlm_host_match_fn_t)(void *cur, struct nlm_host *ref);
+typedef int (*nlm_host_match_fn_t)(void *owner, struct nlm_host *ref);
/*
* Server-side lock handling
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 2e47edd0f515..ee27f1ba01b8 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -72,12 +72,24 @@
/* return bus from PCI devid = ((u16)bus_number) << 8) | devfn */
#define PCI_BUS_NUM(x) (((x) >> 8) & 0xff)
+/*
+ * PCI_SLOT_ALL_DEVICES indicates a slot that covers all devices on the bus.
+ * Used for PCIe hotplug where the physical slot is the entire secondary bus,
+ * and, if ARI Forwarding is enabled, functions may appear to be on multiple
+ * devices.
+ */
+#define PCI_SLOT_ALL_DEVICES 0xfeff
+
+/* Used to identify a slot as a placeholder */
+#define PCI_SLOT_PLACEHOLDER 0xffff
+
/* pci_slot represents a physical slot */
struct pci_slot {
struct pci_bus *bus; /* Bus this slot is on */
struct list_head list; /* Node in list of slots */
struct hotplug_slot *hotplug; /* Hotplug info (move here) */
- unsigned char number; /* PCI_SLOT(pci_dev->devfn) */
+ u16 number; /* Device nr, or PCI_SLOT_ALL_DEVICES */
+ unsigned int per_func_slot:1; /* Allow per function slot */
struct kobject kobj;
};
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index e0af9788e656..3eaf73af41cd 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -301,6 +301,7 @@ struct perf_event_pmu_context;
#define PERF_PMU_CAP_AUX_OUTPUT 0x0080
#define PERF_PMU_CAP_EXTENDED_HW_TYPE 0x0100
#define PERF_PMU_CAP_AUX_PAUSE 0x0200
+#define PERF_PMU_CAP_AUX_PREFER_LARGE 0x0400
/**
* pmu::scope
diff --git a/include/linux/rculist.h b/include/linux/rculist.h
index 14dfa6008467..2e4d990f07ad 100644
--- a/include/linux/rculist.h
+++ b/include/linux/rculist.h
@@ -207,6 +207,35 @@ static inline void list_replace_rcu(struct list_head *old,
old->prev = LIST_POISON2;
}
+static inline void __list_splice_rcu(struct list_head *list,
+ struct list_head *prev,
+ struct list_head *next)
+{
+ struct list_head *first = list->next;
+ struct list_head *last = list->prev;
+
+ last->next = next;
+ first->prev = prev;
+ next->prev = last;
+ rcu_assign_pointer(list_next_rcu(prev), first);
+}
+
+/**
+ * list_splice_rcu - splice a non-RCU list into an RCU-protected list,
+ * designed for stacks.
+ * @list: the non RCU-protected list to splice
+ * @head: the place in the existing RCU-protected list to splice
+ *
+ * The list pointed to by @head can be RCU-read traversed concurrently with
+ * this function.
+ */
+static inline void list_splice_rcu(struct list_head *list,
+ struct list_head *head)
+{
+ if (!list_empty(list))
+ __list_splice_rcu(list, head, head->next);
+}
+
/**
* __list_splice_init_rcu - join an RCU-protected list into an existing list.
* @list: the RCU-protected list to splice
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 7b97f1ec1bc0..0fb5bb6c30d3 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1675,7 +1675,15 @@ static inline char task_state_to_char(struct task_struct *tsk)
return task_index_to_char(task_state_index(tsk));
}
-extern struct pid *cad_pid;
+#ifdef CONFIG_SMP
+extern void kick_process(struct task_struct *tsk);
+#else
+static inline void kick_process(struct task_struct *tsk)
+{
+}
+#endif
+
+extern struct pid __rcu *cad_pid;
/*
* Per process flags
@@ -1953,12 +1961,6 @@ extern int wake_up_state(struct task_struct *tsk, unsigned int state);
extern int wake_up_process(struct task_struct *tsk);
extern void wake_up_new_task(struct task_struct *tsk);
-#ifdef CONFIG_SMP
-extern void kick_process(struct task_struct *tsk);
-#else
-static inline void kick_process(struct task_struct *tsk) { }
-#endif
-
extern void __set_task_comm(struct task_struct *tsk, const char *from, bool exec);
static inline void set_task_comm(struct task_struct *tsk, const char *from)
diff --git a/include/linux/sched/rt.h b/include/linux/sched/rt.h
index 4e3338103654..922935cc3383 100644
--- a/include/linux/sched/rt.h
+++ b/include/linux/sched/rt.h
@@ -52,8 +52,10 @@ static inline bool rt_or_dl_task_policy(struct task_struct *tsk)
#ifdef CONFIG_RT_MUTEXES
extern void rt_mutex_pre_schedule(void);
+extern void rt_mutex_futex_pre_schedule(void);
extern void rt_mutex_schedule(void);
extern void rt_mutex_post_schedule(void);
+extern void rt_mutex_futex_post_schedule(void);
/*
* Must hold either p->pi_lock or task_rq(p)->lock.
diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h
index c8ed09ac29ac..21e496bb1c1b 100644
--- a/include/linux/sched/signal.h
+++ b/include/linux/sched/signal.h
@@ -558,10 +558,7 @@ static inline sigset_t *sigmask_to_save(void)
return res;
}
-static inline int kill_cad_pid(int sig, int priv)
-{
- return kill_pid(cad_pid, sig, priv);
-}
+int kill_cad_pid(int sig, int priv);
/* These can be the second arg to send_sig_info/send_group_sig_info. */
#define SEND_SIG_NOINFO ((struct kernel_siginfo *) 0)
diff --git a/include/linux/screen_info.h b/include/linux/screen_info.h
index 6a4a3cec4638..ab3cffbb58b7 100644
--- a/include/linux/screen_info.h
+++ b/include/linux/screen_info.h
@@ -128,6 +128,8 @@ static inline unsigned int screen_info_video_type(const struct screen_info *si)
ssize_t screen_info_resources(const struct screen_info *si, struct resource *r, size_t num);
+u32 __screen_info_lfb_bits_per_pixel(const struct screen_info *si);
+
#if defined(CONFIG_PCI)
void screen_info_apply_fixups(void);
struct pci_dev *screen_info_pci_dev(const struct screen_info *si);
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 1bfdc03e0d94..84f2f6f11279 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1807,22 +1807,6 @@ static inline void skb_zcopy_set(struct sk_buff *skb, struct ubuf_info *uarg,
}
}
-static inline void skb_zcopy_set_nouarg(struct sk_buff *skb, void *val)
-{
- skb_shinfo(skb)->destructor_arg = (void *)((uintptr_t) val | 0x1UL);
- skb_shinfo(skb)->flags |= SKBFL_ZEROCOPY_FRAG;
-}
-
-static inline bool skb_zcopy_is_nouarg(struct sk_buff *skb)
-{
- return (uintptr_t) skb_shinfo(skb)->destructor_arg & 0x1UL;
-}
-
-static inline void *skb_zcopy_get_nouarg(struct sk_buff *skb)
-{
- return (void *)((uintptr_t) skb_shinfo(skb)->destructor_arg & ~0x1UL);
-}
-
static inline void net_zcopy_put(struct ubuf_info *uarg)
{
if (uarg)
@@ -1845,8 +1829,7 @@ static inline void skb_zcopy_clear(struct sk_buff *skb, bool zerocopy_success)
struct ubuf_info *uarg = skb_zcopy(skb);
if (uarg) {
- if (!skb_zcopy_is_nouarg(skb))
- uarg->ops->complete(skb, uarg, zerocopy_success);
+ uarg->ops->complete(skb, uarg, zerocopy_success);
skb_shinfo(skb)->flags &= ~SKBFL_ALL_ZEROCOPY;
}
@@ -4280,7 +4263,10 @@ skb_header_pointer_careful(const struct sk_buff *skb, int offset,
static inline void * __must_check
skb_pointer_if_linear(const struct sk_buff *skb, int offset, int len)
{
- if (likely(skb_headlen(skb) - offset >= len))
+ unsigned int uoffset = (unsigned int)offset;
+
+ if (likely(uoffset <= skb_headlen(skb) &&
+ (unsigned int)len <= skb_headlen(skb) - uoffset))
return skb->data + offset;
return NULL;
}
diff --git a/include/linux/soc/qcom/mdt_loader.h b/include/linux/soc/qcom/mdt_loader.h
index 9e8e60421192..f32e2ab72754 100644
--- a/include/linux/soc/qcom/mdt_loader.h
+++ b/include/linux/soc/qcom/mdt_loader.h
@@ -10,14 +10,14 @@
struct device;
struct firmware;
-struct qcom_scm_pas_metadata;
+struct qcom_scm_pas_context;
#if IS_ENABLED(CONFIG_QCOM_MDT_LOADER)
ssize_t qcom_mdt_get_size(const struct firmware *fw);
int qcom_mdt_pas_init(struct device *dev, const struct firmware *fw,
const char *fw_name, int pas_id, phys_addr_t mem_phys,
- struct qcom_scm_pas_metadata *pas_metadata_ctx);
+ struct qcom_scm_pas_context *pas_ctx);
int qcom_mdt_load(struct device *dev, const struct firmware *fw,
const char *fw_name, int pas_id, void *mem_region,
phys_addr_t mem_phys, size_t mem_size,
@@ -39,7 +39,7 @@ static inline ssize_t qcom_mdt_get_size(const struct firmware *fw)
static inline int qcom_mdt_pas_init(struct device *dev, const struct firmware *fw,
const char *fw_name, int pas_id, phys_addr_t mem_phys,
- struct qcom_scm_pas_metadata *pas_metadata_ctx)
+ struct qcom_scm_pas_context *pas_ctx)
{
return -ENODEV;
}
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index d791d47eb00e..3ad94d939d28 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -184,7 +184,7 @@ void virtio_reset_device(struct virtio_device *dev);
size_t virtio_max_dma_size(const struct virtio_device *vdev);
#define virtio_device_for_each_vq(vdev, vq) \
- list_for_each_entry(vq, &vdev->vqs, list)
+ list_for_each_entry(vq, &(vdev)->vqs, list)
/**
* struct virtio_driver - operations for a virtio I/O driver
diff --git a/include/net/bluetooth/coredump.h b/include/net/bluetooth/coredump.h
index 72f51b587a04..00c12c7ac042 100644
--- a/include/net/bluetooth/coredump.h
+++ b/include/net/bluetooth/coredump.h
@@ -61,6 +61,7 @@ struct hci_devcoredump {
#ifdef CONFIG_DEV_COREDUMP
void hci_devcd_reset(struct hci_dev *hdev);
+void hci_devcd_shutdown(struct hci_dev *hdev);
void hci_devcd_rx(struct work_struct *work);
void hci_devcd_timeout(struct work_struct *work);
@@ -75,6 +76,7 @@ int hci_devcd_abort(struct hci_dev *hdev);
#else
static inline void hci_devcd_reset(struct hci_dev *hdev) {}
+static inline void hci_devcd_shutdown(struct hci_dev *hdev) {}
static inline void hci_devcd_rx(struct work_struct *work) {}
static inline void hci_devcd_timeout(struct work_struct *work) {}
diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h
index fcc5934a20c0..ede91e4709ee 100644
--- a/include/net/cfg80211.h
+++ b/include/net/cfg80211.h
@@ -1008,6 +1008,17 @@ cfg80211_chandef_compatible(const struct cfg80211_chan_def *chandef1,
*/
int nl80211_chan_width_to_mhz(enum nl80211_chan_width chan_width);
+/**
+ * cfg80211_chandef_get_width - return chandef width in MHz
+ * @c: chandef to return bandwidth for
+ * Return: channel width in MHz for the given chandef; note that it returns
+ * 80 for 80+80 configurations
+ */
+static inline int cfg80211_chandef_get_width(const struct cfg80211_chan_def *c)
+{
+ return nl80211_chan_width_to_mhz(c->width);
+}
+
/**
* cfg80211_chandef_valid - check if a channel definition is valid
* @chandef: the channel definition to check
@@ -6353,6 +6364,7 @@ enum ieee80211_ap_reg_power {
* entered.
* @links.cac_time_ms: CAC time in ms
* @valid_links: bitmap describing what elements of @links are valid
+ * @radio_mask: Bitmask of radios that this interface is allowed to operate on.
*/
struct wireless_dev {
struct wiphy *wiphy;
@@ -6465,6 +6477,8 @@ struct wireless_dev {
unsigned int cac_time_ms;
} links[IEEE80211_MLD_MAX_NUM_LINKS];
u16 valid_links;
+
+ u32 radio_mask;
};
static inline const u8 *wdev_address(struct wireless_dev *wdev)
@@ -6650,6 +6664,17 @@ static inline bool cfg80211_channel_is_psc(struct ieee80211_channel *chan)
bool cfg80211_radio_chandef_valid(const struct wiphy_radio *radio,
const struct cfg80211_chan_def *chandef);
+/**
+ * cfg80211_wdev_channel_allowed - Check if the wdev may use the channel
+ *
+ * @wdev: the wireless device
+ * @chan: channel to check
+ *
+ * Return: whether or not the wdev may use the channel
+ */
+bool cfg80211_wdev_channel_allowed(struct wireless_dev *wdev,
+ struct ieee80211_channel *chan);
+
/**
* ieee80211_get_response_rate - get basic rate for a given rate
*
diff --git a/include/net/gue.h b/include/net/gue.h
index caefd6da8693..d377155fd0b3 100644
--- a/include/net/gue.h
+++ b/include/net/gue.h
@@ -84,8 +84,9 @@ static inline size_t guehdr_priv_flags_len(__be32 flags)
}
/* Validate standard and private flags. Returns non-zero (meaning invalid)
- * if there is an unknown standard or private flags, or the options length for
- * the flags exceeds the options length specific in hlen of the GUE header.
+ * if there is an unknown standard or private flags, if the options length for
+ * the flags exceeds the options length specified in hlen of the GUE header, or
+ * if a private option contains invalid data.
*/
static inline int validate_gue_flags(struct guehdr *guehdr, size_t optlen)
{
@@ -103,8 +104,8 @@ static inline int validate_gue_flags(struct guehdr *guehdr, size_t optlen)
/* Private flags are last four bytes accounted in
* guehdr_flags_len
*/
- __be32 pflags = *(__be32 *)((void *)&guehdr[1] +
- len - GUE_LEN_PRIV);
+ void *data = (void *)&guehdr[1] + len;
+ __be32 pflags = *(__be32 *)(data - GUE_LEN_PRIV);
if (pflags & ~GUE_PFLAGS_ALL)
return 1;
@@ -112,6 +113,16 @@ static inline int validate_gue_flags(struct guehdr *guehdr, size_t optlen)
len += guehdr_priv_flags_len(pflags);
if (len > optlen)
return 1;
+
+ if (pflags & GUE_PFLAG_REMCSUM) {
+ __be16 *pd = data;
+
+ /* The field offset pd[1] must not be less
+ * than the start pd[0].
+ */
+ if (ntohs(pd[1]) < ntohs(pd[0]))
+ return 1;
+ }
}
return 0;
diff --git a/include/net/if_inet6.h b/include/net/if_inet6.h
index 238ad3349456..795fb41b45f5 100644
--- a/include/net/if_inet6.h
+++ b/include/net/if_inet6.h
@@ -88,8 +88,6 @@ struct ip6_sf_socklist {
struct in6_addr sl_addr[] __counted_by(sl_max);
};
-#define IP6_SFBLOCK 10 /* allocate this many at once */
-
struct ipv6_mc_socklist {
struct in6_addr addr;
int ifindex;
diff --git a/include/net/inet_connection_sock.h b/include/net/inet_connection_sock.h
index 41354036b0c8..e8ee9164ec5a 100644
--- a/include/net/inet_connection_sock.h
+++ b/include/net/inet_connection_sock.h
@@ -62,6 +62,9 @@ struct inet_connection_sock_af_ops {
* @icsk_bind2_hash: Bind node in the bhash2 table
* @icsk_timeout: Timeout
* @icsk_retransmit_timer: Resend (no ack)
+ * @icsk_delack_timer: Delayed ACK timer
+ * @icsk_keepalive_timer: Keepalive timer
+ * @mptcp_tout_timer: MPTCP timeout timer
* @icsk_rto: Retransmit timeout
* @icsk_pmtu_cookie Last pmtu seen by socket
* @icsk_ca_ops Pluggable congestion control hook
@@ -88,8 +91,12 @@ struct inet_connection_sock {
struct inet_bind_bucket *icsk_bind_hash;
struct inet_bind2_bucket *icsk_bind2_hash;
unsigned long icsk_timeout;
- struct timer_list icsk_retransmit_timer;
- struct timer_list icsk_delack_timer;
+ struct timer_list icsk_retransmit_timer;
+ struct timer_list icsk_delack_timer;
+ union {
+ struct timer_list icsk_keepalive_timer;
+ struct timer_list mptcp_tout_timer;
+ };
__u32 icsk_rto;
__u32 icsk_rto_min;
__u32 icsk_delack_max;
diff --git a/include/net/inet_sock.h b/include/net/inet_sock.h
index 394c3b66065e..ad7e740c6441 100644
--- a/include/net/inet_sock.h
+++ b/include/net/inet_sock.h
@@ -212,6 +212,7 @@ struct inet_sock {
struct sock sk;
#if IS_ENABLED(CONFIG_IPV6)
struct ipv6_pinfo *pinet6;
+ struct ipv6_fl_socklist __rcu *ipv6_fl_list;
#endif
/* Socket demultiplex comparisons on incoming packets. */
#define inet_daddr sk.__sk_common.skc_daddr
diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index eaa2f6770ce7..8be44dc5b5b7 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -94,12 +94,12 @@ static inline struct dst_entry *ip6_route_output(struct net *net,
}
/* Only conditionally release dst if flags indicates
- * !RT6_LOOKUP_F_DST_NOREF or dst is in uncached_list.
+ * !RT6_LOOKUP_F_DST_NOREF or dst is uncached.
*/
static inline void ip6_rt_put_flags(struct rt6_info *rt, int flags)
{
if (!(flags & RT6_LOOKUP_F_DST_NOREF) ||
- !list_empty(&rt->dst.rt_uncached))
+ rt->dst.rt_uncached_list)
ip6_rt_put(rt);
}
@@ -230,7 +230,7 @@ static inline const struct rt6_info *skb_rt6_info(const struct sk_buff *skb)
*/
static inline void ip6_dst_store(struct sock *sk, struct dst_entry *dst,
const struct in6_addr *daddr,
- const struct in6_addr *saddr)
+ bool saddr_set)
{
struct ipv6_pinfo *np = inet6_sk(sk);
@@ -238,7 +238,7 @@ static inline void ip6_dst_store(struct sock *sk, struct dst_entry *dst,
sk_setup_caps(sk, dst);
np->daddr_cache = daddr;
#ifdef CONFIG_IPV6_SUBTREES
- np->saddr_cache = saddr;
+ np->saddr_cache = saddr_set;
#endif
}
diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index 7d71a4149cdf..41ae682015fc 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -7228,11 +7228,14 @@ bool ieee80211_tx_prepare_skb(struct ieee80211_hw *hw,
*
* @skb: packet injected by userspace
* @dev: the &struct device of this 802.11 device
+ * @chandef: the channel definition the frame will be transmitted on, or
+ * %NULL to skip the bandwidth checks
*
* Return: %true if the radiotap header was parsed, %false otherwise
*/
bool ieee80211_parse_tx_radiotap(struct sk_buff *skb,
- struct net_device *dev);
+ struct net_device *dev,
+ const struct cfg80211_chan_def *chandef);
/**
* struct ieee80211_noa_data - holds temporary data for tracking P2P NoA state
diff --git a/include/net/netfilter/nf_conntrack.h b/include/net/netfilter/nf_conntrack.h
index 8cb70e7485e2..3d938a19d332 100644
--- a/include/net/netfilter/nf_conntrack.h
+++ b/include/net/netfilter/nf_conntrack.h
@@ -186,6 +186,11 @@ static inline void nf_ct_put(struct nf_conn *ct)
nf_ct_destroy(&ct->ct_general);
}
+static inline bool nf_ct_shared(const struct nf_conn *ct)
+{
+ return refcount_read(&ct->ct_general.use) > 1;
+}
+
/* load module; enable/disable conntrack in this namespace */
int nf_ct_netns_get(struct net *net, u8 nfproto);
void nf_ct_netns_put(struct net *net, u8 nfproto);
diff --git a/include/net/netfilter/nf_conntrack_timeout.h b/include/net/netfilter/nf_conntrack_timeout.h
index 3a66d4abb6d6..d60aa86be019 100644
--- a/include/net/netfilter/nf_conntrack_timeout.h
+++ b/include/net/netfilter/nf_conntrack_timeout.h
@@ -12,6 +12,7 @@
#define CTNL_TIMEOUT_NAME_MAX 32
struct nf_ct_timeout {
+ refcount_t refcnt;
__u16 l3num;
const struct nf_conntrack_l4proto *l4proto;
struct rcu_head rcu;
@@ -22,6 +23,22 @@ struct nf_conn_timeout {
struct nf_ct_timeout __rcu *timeout;
};
+static inline void nf_ct_timeout_put(const struct nf_conn *ct)
+{
+#ifdef CONFIG_NF_CONNTRACK_TIMEOUT
+ struct nf_conn_timeout *timeout_ext;
+ struct nf_ct_timeout *timeout;
+
+ timeout_ext = nf_ct_ext_find(ct, NF_CT_EXT_TIMEOUT);
+ if (!timeout_ext)
+ return;
+
+ timeout = rcu_dereference(timeout_ext->timeout);
+ if (timeout && refcount_dec_and_test(&timeout->refcnt))
+ kfree_rcu(timeout, rcu);
+#endif
+}
+
static inline unsigned int *
nf_ct_timeout_data(const struct nf_conn_timeout *t)
{
@@ -56,8 +73,14 @@ struct nf_conn_timeout *nf_ct_timeout_ext_add(struct nf_conn *ct,
#ifdef CONFIG_NF_CONNTRACK_TIMEOUT
struct nf_conn_timeout *timeout_ext;
+ if (!timeout)
+ return NULL;
+
timeout_ext = nf_ct_ext_add(ct, NF_CT_EXT_TIMEOUT, gfp);
- if (timeout_ext == NULL)
+ if (!timeout_ext || timeout_ext->timeout)
+ return NULL;
+
+ if (!refcount_inc_not_zero(&timeout->refcnt))
return NULL;
rcu_assign_pointer(timeout_ext->timeout, timeout);
@@ -75,7 +98,7 @@ static inline unsigned int *nf_ct_timeout_lookup(const struct nf_conn *ct)
struct nf_conn_timeout *timeout_ext;
timeout_ext = nf_ct_timeout_find(ct);
- if (timeout_ext)
+ if (timeout_ext && rcu_access_pointer(timeout_ext->timeout))
timeouts = nf_ct_timeout_data(timeout_ext);
#endif
return timeouts;
diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h
index f4b59915a49c..923b2c0d6e84 100644
--- a/include/net/netfilter/nf_tables.h
+++ b/include/net/netfilter/nf_tables.h
@@ -1238,8 +1238,6 @@ static inline bool nft_is_base_chain(const struct nft_chain *chain)
return chain->flags & NFT_CHAIN_BASE;
}
-int __nft_release_basechain(struct nft_ctx *ctx);
-
unsigned int nft_do_chain(struct nft_pktinfo *pkt, void *priv);
static inline bool nft_use_inc(u32 *use)
diff --git a/include/net/netmem.h b/include/net/netmem.h
index 8a6e20be4b9d..1b58faa4f20f 100644
--- a/include/net/netmem.h
+++ b/include/net/netmem.h
@@ -72,6 +72,22 @@ static inline bool netmem_is_net_iov(const netmem_ref netmem)
return (__force unsigned long)netmem & NET_IOV;
}
+/**
+ * __netmem_to_page - unsafely get pointer to the &page backing @netmem
+ * @netmem: netmem reference to convert
+ *
+ * Unsafe version of netmem_to_page(). When @netmem is always page-backed,
+ * e.g. when it's a header buffer, performs faster and generates smaller
+ * object code (no check for the LSB, no WARN). When @netmem points to IOV,
+ * provokes undefined behaviour.
+ *
+ * Return: pointer to the &page (garbage if @netmem is not page-backed).
+ */
+static inline struct page *__netmem_to_page(netmem_ref netmem)
+{
+ return (__force struct page *)netmem;
+}
+
/* This conversion fails (returns NULL) if the netmem_ref is not struct page
* backed.
*/
@@ -80,7 +96,7 @@ static inline struct page *netmem_to_page(netmem_ref netmem)
if (WARN_ON_ONCE(netmem_is_net_iov(netmem)))
return NULL;
- return (__force struct page *)netmem;
+ return __netmem_to_page(netmem);
}
static inline struct net_iov *netmem_to_net_iov(netmem_ref netmem)
@@ -103,6 +119,17 @@ static inline netmem_ref page_to_netmem(struct page *page)
return (__force netmem_ref)page;
}
+/**
+ * virt_to_netmem - convert virtual memory pointer to a netmem reference
+ * @data: host memory pointer to convert
+ *
+ * Return: netmem reference to the &page backing this virtual address.
+ */
+static inline netmem_ref virt_to_netmem(const void *data)
+{
+ return page_to_netmem(virt_to_page(data));
+}
+
static inline int netmem_ref_count(netmem_ref netmem)
{
/* The non-pp refcount of net_iov is always 1. On net_iov, we only
@@ -127,6 +154,22 @@ static inline struct net_iov *__netmem_clear_lsb(netmem_ref netmem)
return (struct net_iov *)((__force unsigned long)netmem & ~NET_IOV);
}
+/**
+ * __netmem_get_pp - unsafely get pointer to the &page_pool backing @netmem
+ * @netmem: netmem reference to get the pointer from
+ *
+ * Unsafe version of netmem_get_pp(). When @netmem is always page-backed,
+ * e.g. when it's a header buffer, performs faster and generates smaller
+ * object code (avoids clearing the LSB). When @netmem points to IOV,
+ * provokes invalid memory access.
+ *
+ * Return: pointer to the &page_pool (garbage if @netmem is not page-backed).
+ */
+static inline struct page_pool *__netmem_get_pp(netmem_ref netmem)
+{
+ return __netmem_to_page(netmem)->pp;
+}
+
static inline struct page_pool *netmem_get_pp(netmem_ref netmem)
{
return __netmem_clear_lsb(netmem)->pp;
@@ -158,12 +201,43 @@ static inline netmem_ref netmem_compound_head(netmem_ref netmem)
return page_to_netmem(compound_head(netmem_to_page(netmem)));
}
+/**
+ * __netmem_address - unsafely get pointer to the memory backing @netmem
+ * @netmem: netmem reference to get the pointer for
+ *
+ * Unsafe version of netmem_address(). When @netmem is always page-backed,
+ * e.g. when it's a header buffer, performs faster and generates smaller
+ * object code (no check for the LSB). When @netmem points to IOV, provokes
+ * undefined behaviour.
+ *
+ * Return: pointer to the memory (garbage if @netmem is not page-backed).
+ */
+static inline void *__netmem_address(netmem_ref netmem)
+{
+ return page_address(__netmem_to_page(netmem));
+}
+
static inline void *netmem_address(netmem_ref netmem)
{
if (netmem_is_net_iov(netmem))
return NULL;
- return page_address(netmem_to_page(netmem));
+ return __netmem_address(netmem);
+}
+
+/**
+ * netmem_is_pfmemalloc - check if @netmem was allocated under memory pressure
+ * @netmem: netmem reference to check
+ *
+ * Return: true if @netmem is page-backed and the page was allocated under
+ * memory pressure, false otherwise.
+ */
+static inline bool netmem_is_pfmemalloc(netmem_ref netmem)
+{
+ if (netmem_is_net_iov(netmem))
+ return false;
+
+ return page_is_pfmemalloc(netmem_to_page(netmem));
}
static inline unsigned long netmem_get_dma_addr(netmem_ref netmem)
diff --git a/include/net/netns/xfrm.h b/include/net/netns/xfrm.h
index 23dd647fe024..b73983a17e08 100644
--- a/include/net/netns/xfrm.h
+++ b/include/net/netns/xfrm.h
@@ -59,7 +59,7 @@ struct netns_xfrm {
struct list_head inexact_bins;
- struct sock *nlsk;
+ struct sock __rcu *nlsk;
struct sock *nlsk_stash;
u32 sysctl_aevent_etime;
diff --git a/include/net/nfc/hci.h b/include/net/nfc/hci.h
index 756c11084f65..86ed63e5d533 100644
--- a/include/net/nfc/hci.h
+++ b/include/net/nfc/hci.h
@@ -144,7 +144,7 @@ struct nfc_hci_dev {
data_exchange_cb_t async_cb;
void *async_cb_context;
- u8 *gb;
+ u8 gb[NFC_MAX_GT_LEN];
size_t gb_len;
unsigned long quirks;
diff --git a/include/net/nfc/nfc.h b/include/net/nfc/nfc.h
index 473f58e646cc..71126f1aeba3 100644
--- a/include/net/nfc/nfc.h
+++ b/include/net/nfc/nfc.h
@@ -269,7 +269,8 @@ struct sk_buff *nfc_alloc_recv_skb(unsigned int size, gfp_t gfp);
int nfc_set_remote_general_bytes(struct nfc_dev *dev,
const u8 *gt, u8 gt_len);
-u8 *nfc_get_local_general_bytes(struct nfc_dev *dev, size_t *gb_len);
+u8 *nfc_get_local_general_bytes(struct nfc_dev *dev, u8 *out_gb,
+ size_t gb_max_len, size_t *gb_len);
int nfc_fw_download_done(struct nfc_dev *dev, const char *firmware_name,
u32 result);
diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h
index 60a5347922be..036a5cca8379 100644
--- a/include/net/page_pool/helpers.h
+++ b/include/net/page_pool/helpers.h
@@ -418,7 +418,21 @@ static inline dma_addr_t page_pool_get_dma_addr_netmem(netmem_ref netmem)
*/
static inline dma_addr_t page_pool_get_dma_addr(const struct page *page)
{
- return page_pool_get_dma_addr_netmem(page_to_netmem((struct page *)page));
+ dma_addr_t ret = page->dma_addr;
+
+ if (PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA)
+ ret <<= PAGE_SHIFT;
+
+ return ret;
+}
+
+static inline void __page_pool_dma_sync_for_cpu(const struct page_pool *pool,
+ const dma_addr_t dma_addr,
+ u32 offset, u32 dma_sync_size)
+{
+ dma_sync_single_range_for_cpu(pool->p.dev, dma_addr,
+ offset + pool->p.offset, dma_sync_size,
+ page_pool_get_dma_dir(pool));
}
/**
@@ -437,10 +451,21 @@ static inline void page_pool_dma_sync_for_cpu(const struct page_pool *pool,
const struct page *page,
u32 offset, u32 dma_sync_size)
{
- dma_sync_single_range_for_cpu(pool->p.dev,
- page_pool_get_dma_addr(page),
- offset + pool->p.offset, dma_sync_size,
- page_pool_get_dma_dir(pool));
+ __page_pool_dma_sync_for_cpu(pool, page_pool_get_dma_addr(page), offset,
+ dma_sync_size);
+}
+
+static inline void
+page_pool_dma_sync_netmem_for_cpu(const struct page_pool *pool,
+ const netmem_ref netmem, u32 offset,
+ u32 dma_sync_size)
+{
+ if (!pool->dma_sync_for_cpu)
+ return;
+
+ __page_pool_dma_sync_for_cpu(pool,
+ page_pool_get_dma_addr_netmem(netmem),
+ offset, dma_sync_size);
}
static inline bool page_pool_put(struct page_pool *pool)
diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h
index 7fae0d4eef5e..2b6fd84fac6f 100644
--- a/include/net/page_pool/types.h
+++ b/include/net/page_pool/types.h
@@ -171,7 +171,8 @@ struct page_pool {
bool has_init_callback:1; /* slow::init_callback is set */
bool dma_map:1; /* Perform DMA mapping */
- bool dma_sync:1; /* Perform DMA sync */
+ bool dma_sync:1; /* Perform DMA sync for device */
+ bool dma_sync_for_cpu:1; /* Perform DMA sync for cpu */
#ifdef CONFIG_PAGE_POOL_STATS
bool system:1; /* This is a global percpu pool */
#endif
@@ -252,7 +253,7 @@ struct page_pool {
};
struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp);
-netmem_ref page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp);
+netmem_ref page_pool_alloc_netmems(struct page_pool *pool, gfp_t gfp);
struct page *page_pool_alloc_frag(struct page_pool *pool, unsigned int *offset,
unsigned int size, gfp_t gfp);
netmem_ref page_pool_alloc_frag_netmem(struct page_pool *pool,
diff --git a/include/net/rose.h b/include/net/rose.h
index 2b5491bbf39a..95d1f9c582dc 100644
--- a/include/net/rose.h
+++ b/include/net/rose.h
@@ -160,6 +160,18 @@ static inline void rose_neigh_hold(struct rose_neigh *rose_neigh)
static inline void rose_neigh_put(struct rose_neigh *rose_neigh)
{
if (refcount_dec_and_test(&rose_neigh->use)) {
+ /* We are dropping the last reference, so we are about to free the
+ * neighbour. Its timers may still be armed -- t0timer in particular
+ * re-arms itself in rose_t0timer_expiry(). rose_remove_neigh()
+ * cancels them before its own put, but callers that drop the final
+ * reference without first calling rose_remove_neigh() (the socket
+ * heartbeat reaping path) would otherwise kfree() a neighbour with a
+ * live timer -> use-after-free. timer_delete_sync() (not the async
+ * variant) is required: it waits out a concurrently running handler
+ * and loops until the self-rearming timer stays stopped.
+ */
+ timer_delete_sync(&rose_neigh->ftimer);
+ timer_delete_sync(&rose_neigh->t0timer);
if (rose_neigh->ax25)
ax25_cb_put(rose_neigh->ax25);
kfree(rose_neigh->digipeat);
diff --git a/include/net/sock.h b/include/net/sock.h
index dffbaaa7fe49..73353ee313ad 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -300,6 +300,7 @@ struct sk_filter;
* @sk_txrehash: enable TX hash rethink
* @sk_filter: socket filtering instructions
* @sk_timer: sock cleanup timer
+ * @mptcp_retransmit_timer: MPTCP retransmit timer
* @sk_stamp: time stamp of last packet received
* @sk_stamp_seq: lock for accessing sk_stamp on 32 bit architectures only
* @sk_tsflags: SO_TIMESTAMPING flags
@@ -465,7 +466,10 @@ struct sock {
u32 sk_dst_pending_confirm;
u32 sk_pacing_status; /* see enum sk_pacing */
struct page_frag sk_frag;
- struct timer_list sk_timer;
+ union {
+ struct timer_list sk_timer;
+ struct timer_list mptcp_retransmit_timer;
+ };
unsigned long sk_pacing_rate; /* bytes per second */
atomic_t sk_zckey;
@@ -1840,6 +1844,8 @@ static inline struct sk_buff *sock_alloc_send_skb(struct sock *sk,
}
void *sock_kmalloc(struct sock *sk, int size, gfp_t priority);
+void *sock_kmemdup(struct sock *sk, const void *src,
+ int size, gfp_t priority);
void sock_kfree_s(struct sock *sk, void *mem, int size);
void sock_kzfree_s(struct sock *sk, void *mem, int size);
void sk_send_sigurg(struct sock *sk);
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 87eb065e05c1..85f8102aa9da 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -1058,9 +1058,9 @@ static inline bool tcp_skb_can_collapse_to(const struct sk_buff *skb)
static inline bool tcp_skb_can_collapse(const struct sk_buff *to,
const struct sk_buff *from)
{
- /* skb_cmp_decrypted() not needed, use tcp_write_collapse_fence() */
return likely(tcp_skb_can_collapse_to(to) &&
mptcp_skb_can_collapse(to, from) &&
+ !skb_cmp_decrypted(to, from) &&
skb_pure_zcopy_same(to, from) &&
skb_frags_readable(to) == skb_frags_readable(from));
}
@@ -2119,7 +2119,7 @@ static inline void tcp_rtx_queue_unlink_and_free(struct sk_buff *skb, struct soc
static inline void tcp_write_collapse_fence(struct sock *sk)
{
- struct sk_buff *skb = tcp_write_queue_tail(sk);
+ struct sk_buff *skb = tcp_write_queue_tail(sk) ?: tcp_rtx_queue_tail(sk);
if (skb)
TCP_SKB_CB(skb)->eor = 1;
diff --git a/include/sound/soc.h b/include/sound/soc.h
index cd467f8babdb..a5f8e83e6ced 100644
--- a/include/sound/soc.h
+++ b/include/sound/soc.h
@@ -717,7 +717,8 @@ struct snd_soc_dai_link_component {
struct snd_soc_dai_link_ch_map {
unsigned int cpu;
unsigned int codec;
- unsigned int ch_mask;
+ unsigned int cpu_ch_mask;
+ unsigned int codec_ch_mask;
};
struct snd_soc_dai_link {
diff --git a/include/uapi/linux/landlock.h b/include/uapi/linux/landlock.h
index 94cfad88082f..7d9158a3e792 100644
--- a/include/uapi/linux/landlock.h
+++ b/include/uapi/linux/landlock.h
@@ -156,6 +156,23 @@ struct landlock_net_port_attr {
* :manpage:`ftruncate(2)`, :manpage:`creat(2)`, or :manpage:`open(2)` with
* ``O_TRUNC``. This access right is available since the third version of the
* Landlock ABI.
+ * - %LANDLOCK_ACCESS_FS_IOCTL_DEV: Invoke :manpage:`ioctl(2)` commands on an opened
+ * character or block device.
+ *
+ * This access right applies to all `ioctl(2)` commands implemented by device
+ * drivers. However, the following common IOCTL commands continue to be
+ * invokable independent of the %LANDLOCK_ACCESS_FS_IOCTL_DEV right:
+ *
+ * * IOCTL commands targeting file descriptors (``FIOCLEX``, ``FIONCLEX``),
+ * * IOCTL commands targeting file descriptions (``FIONBIO``, ``FIOASYNC``),
+ * * IOCTL commands targeting file systems (``FIFREEZE``, ``FITHAW``,
+ * ``FIGETBSZ``, ``FS_IOC_GETFSUUID``, ``FS_IOC_GETFSSYSFSPATH``)
+ * * Some IOCTL commands which do not make sense when used with devices, but
+ * whose implementations are safe and return the right error codes
+ * (``FS_IOC_FIEMAP``, ``FICLONE``, ``FICLONERANGE``, ``FIDEDUPERANGE``)
+ *
+ * This access right is available since the fifth version of the Landlock
+ * ABI.
*
* Whether an opened file can be truncated with :manpage:`ftruncate(2)` or used
* with `ioctl(2)` is determined during :manpage:`open(2)`, in the same way as
@@ -216,26 +233,6 @@ struct landlock_net_port_attr {
* If multiple requirements are not met, the ``EACCES`` error code takes
* precedence over ``EXDEV``.
*
- * The following access right applies both to files and directories:
- *
- * - %LANDLOCK_ACCESS_FS_IOCTL_DEV: Invoke :manpage:`ioctl(2)` commands on an opened
- * character or block device.
- *
- * This access right applies to all `ioctl(2)` commands implemented by device
- * drivers. However, the following common IOCTL commands continue to be
- * invokable independent of the %LANDLOCK_ACCESS_FS_IOCTL_DEV right:
- *
- * * IOCTL commands targeting file descriptors (``FIOCLEX``, ``FIONCLEX``),
- * * IOCTL commands targeting file descriptions (``FIONBIO``, ``FIOASYNC``),
- * * IOCTL commands targeting file systems (``FIFREEZE``, ``FITHAW``,
- * ``FIGETBSZ``, ``FS_IOC_GETFSUUID``, ``FS_IOC_GETFSSYSFSPATH``)
- * * Some IOCTL commands which do not make sense when used with devices, but
- * whose implementations are safe and return the right error codes
- * (``FS_IOC_FIEMAP``, ``FICLONE``, ``FICLONERANGE``, ``FIDEDUPERANGE``)
- *
- * This access right is available since the fifth version of the Landlock
- * ABI.
- *
* .. warning::
*
* It is currently not possible to restrict some file-related actions
diff --git a/include/uapi/linux/nl80211.h b/include/uapi/linux/nl80211.h
index c2d7faf8d87f..8ce0c143e9fd 100644
--- a/include/uapi/linux/nl80211.h
+++ b/include/uapi/linux/nl80211.h
@@ -2868,6 +2868,9 @@ enum nl80211_commands {
* nested item, it contains attributes defined in
* &enum nl80211_if_combination_attrs.
*
+ * @NL80211_ATTR_VIF_RADIO_MASK: Bitmask of allowed radios (u32).
+ * A value of 0 means all radios.
+ *
* @NUM_NL80211_ATTR: total number of nl80211_attrs available
* @NL80211_ATTR_MAX: highest attribute number currently defined
* @__NL80211_ATTR_AFTER_LAST: internal use
@@ -3416,6 +3419,8 @@ enum nl80211_attrs {
NL80211_ATTR_WIPHY_RADIOS,
NL80211_ATTR_WIPHY_INTERFACE_COMBINATIONS,
+ NL80211_ATTR_VIF_RADIO_MASK,
+
/* add attributes here, update the policy in nl80211.c */
__NL80211_ATTR_AFTER_LAST,
diff --git a/init/main.c b/init/main.c
index df9db4d47868..34bb284f6419 100644
--- a/init/main.c
+++ b/init/main.c
@@ -270,7 +270,8 @@ static void * __init get_boot_config_from_initrd(size_t *_size)
u32 *hdr;
int i;
- if (!initrd_end)
+ if (!initrd_end || initrd_end < initrd_start ||
+ initrd_end - initrd_start < BOOTCONFIG_MAGIC_LEN + 8)
return NULL;
data = (char *)initrd_end - BOOTCONFIG_MAGIC_LEN;
@@ -287,16 +288,26 @@ static void * __init get_boot_config_from_initrd(size_t *_size)
found:
hdr = (u32 *)(data - 8);
+ if ((unsigned long)hdr < initrd_start)
+ return NULL;
+
size = le32_to_cpu(hdr[0]);
csum = le32_to_cpu(hdr[1]);
- data = ((void *)hdr) - size;
- if ((unsigned long)data < initrd_start) {
- pr_err("bootconfig size %d is greater than initrd size %ld\n",
+ if (size > XBC_DATA_MAX) {
+ pr_err("bootconfig size %u is greater than max size %d\n",
+ size, XBC_DATA_MAX);
+ return NULL;
+ }
+
+ if (size > ((unsigned long)hdr - initrd_start)) {
+ pr_err("bootconfig size %u is greater than initrd size %lu\n",
size, initrd_end - initrd_start);
return NULL;
}
+ data = ((void *)hdr) - size;
+
if (xbc_calc_checksum(data, size) != csum) {
pr_err("bootconfig checksum failed\n");
return NULL;
@@ -399,12 +410,6 @@ static void __init setup_boot_config(void)
return;
}
- if (size >= XBC_DATA_MAX) {
- pr_err("bootconfig size %ld greater than max size %d\n",
- (long)size, XBC_DATA_MAX);
- return;
- }
-
ret = xbc_init(data, size, &msg, &pos);
if (ret < 0) {
if (pos < 0)
@@ -1523,7 +1528,7 @@ static noinline void __init kernel_init_freeable(void)
*/
set_mems_allowed(node_states[N_MEMORY]);
- cad_pid = get_pid(task_pid(current));
+ rcu_assign_pointer(cad_pid, get_pid(task_pid(current)));
smp_prepare_cpus(setup_max_cpus);
diff --git a/io_uring/net.c b/io_uring/net.c
index b70741b2e2bc..f738f0da87a5 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -858,7 +858,7 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
static inline bool io_recv_finish(struct io_kiocb *req,
struct io_async_msghdr *kmsg,
struct io_br_sel *sel, bool mshot_finished,
- unsigned issue_flags)
+ unsigned issue_flags, int consumed)
{
struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg);
unsigned int cflags = 0;
@@ -869,7 +869,7 @@ static inline bool io_recv_finish(struct io_kiocb *req,
if (sr->flags & IORING_RECVSEND_BUNDLE) {
size_t this_ret = sel->val - sr->done_io;
- cflags |= io_put_kbufs(req, this_ret, sel->buf_list, io_bundle_nbufs(kmsg, this_ret));
+ cflags |= io_put_kbufs(req, consumed, sel->buf_list, io_bundle_nbufs(kmsg, consumed));
if (sr->retry_flags & IO_SR_MSG_RETRY)
cflags = req->cqe.flags | (cflags & CQE_F_MASK);
/* bundle with no more immediate buffers, we're done */
@@ -888,7 +888,7 @@ static inline bool io_recv_finish(struct io_kiocb *req,
return false;
}
} else {
- cflags |= io_put_kbuf(req, sel->val, sel->buf_list);
+ cflags |= io_put_kbuf(req, consumed, sel->buf_list);
}
/*
@@ -1021,6 +1021,8 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
int ret, min_ret = 0;
bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK;
bool mshot_finished = true;
+ int consumed = 0;
+ size_t len;
sock = sock_from_file(req->file);
if (unlikely(!sock))
@@ -1036,9 +1038,8 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
retry_multishot:
sel.buf_list = NULL;
+ len = sr->len;
if (io_do_buffer_select(req)) {
- size_t len = sr->len;
-
sel = io_buffer_select(req, &len, issue_flags);
if (!sel.addr)
return -ENOBUFS;
@@ -1059,6 +1060,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
if (req->flags & REQ_F_APOLL_MULTISHOT) {
ret = io_recvmsg_multishot(sock, sr, kmsg, flags,
&mshot_finished);
+ consumed = ret;
} else {
/* disable partial retry for recvmsg with cmsg attached */
if (flags & MSG_WAITALL && !kmsg->msg.msg_controllen)
@@ -1066,6 +1068,15 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
ret = __sys_recvmsg_sock(sock, &kmsg->msg, sr->umsg,
kmsg->uaddr, flags);
+ /*
+ * With MSG_TRUNC, the net layer will return the full size of
+ * the packet, even if we only filled part of it in the buffers.
+ * Adjust the returned size to consume only the real part of the
+ * buffer.
+ */
+ consumed = ret;
+ if (ret > 0)
+ consumed = min_t(size_t, ret, len);
}
if (ret < min_ret) {
@@ -1094,7 +1105,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags)
io_kbuf_recycle(req, sel.buf_list, issue_flags);
sel.val = ret;
- if (!io_recv_finish(req, kmsg, &sel, mshot_finished, issue_flags))
+ if (!io_recv_finish(req, kmsg, &sel, mshot_finished, issue_flags, consumed))
goto retry_multishot;
return sel.val;
@@ -1177,9 +1188,10 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
struct io_br_sel sel;
struct socket *sock;
unsigned flags;
- int ret, min_ret = 0;
+ int ret, min_ret = 0, consumed = 0;
bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK;
bool mshot_finished;
+ size_t len = 0;
if (!(req->flags & REQ_F_POLLED) &&
(sr->flags & IORING_RECVSEND_POLL_FIRST))
@@ -1195,6 +1207,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
retry_multishot:
sel.buf_list = NULL;
+ len = sr->len;
if (io_do_buffer_select(req)) {
sel.val = sr->len;
ret = io_recv_buf_select(req, kmsg, &sel, issue_flags);
@@ -1202,6 +1215,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
kmsg->msg.msg_inq = -1;
goto out_free;
}
+ len = iov_iter_count(&kmsg->msg.msg_iter);
sr->buf = NULL;
}
@@ -1234,6 +1248,17 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
}
mshot_finished = ret <= 0;
+
+ /*
+ * With MSG_TRUNC, the net layer will return the full size of
+ * the packet, even if we only filled part of it in the buffers.
+ * Adjust the returned size to consume only the real part of the
+ * buffer.
+ */
+ consumed = ret;
+ if (ret > 0)
+ consumed = min_t(size_t, ret, len);
+
if (ret > 0)
ret += sr->done_io;
else if (sr->done_io)
@@ -1242,7 +1267,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags)
io_kbuf_recycle(req, sel.buf_list, issue_flags);
sel.val = ret;
- if (!io_recv_finish(req, kmsg, &sel, mshot_finished, issue_flags))
+ if (!io_recv_finish(req, kmsg, &sel, mshot_finished, issue_flags, consumed))
goto retry_multishot;
return sel.val;
diff --git a/io_uring/rw.c b/io_uring/rw.c
index a74664c4a05e..d615c61bbbd2 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -460,31 +460,23 @@ static void io_req_end_write(struct io_kiocb *req)
}
}
-/*
- * Trigger the notifications after having done some IO, and finish the write
- * accounting, if any.
- */
-static void io_req_io_end(struct io_kiocb *req)
+/* Trigger the notifications after having done some IO. */
+static void io_req_io_notify(struct io_kiocb *req)
{
struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw);
- if (rw->kiocb.ki_flags & IOCB_WRITE) {
- io_req_end_write(req);
+ if (rw->kiocb.ki_flags & IOCB_WRITE)
fsnotify_modify(req->file);
- } else {
+ else
fsnotify_access(req->file);
- }
}
static bool __io_complete_rw_common(struct io_kiocb *req, long res)
{
if (unlikely(res != req->cqe.res)) {
if ((res == -EOPNOTSUPP || res == -EAGAIN) && io_rw_should_reissue(req)) {
- /*
- * Reissue will start accounting again, finish the
- * current cycle.
- */
- io_req_io_end(req);
+ /* Reissue bypasses task_work, so notify here. */
+ io_req_io_notify(req);
req->flags |= REQ_F_REISSUE | REQ_F_BL_NO_RECYCLE;
return true;
}
@@ -519,7 +511,7 @@ void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts)
io_req_set_res(req, io_fixup_rw_res(req, res), 0);
}
- io_req_io_end(req);
+ io_req_io_notify(req);
if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))
req->cqe.flags |= io_put_kbuf(req, req->cqe.res, NULL);
@@ -533,6 +525,10 @@ static void io_complete_rw(struct kiocb *kiocb, long res)
struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb);
struct io_kiocb *req = cmd_to_io_kiocb(rw);
+ /* ring owner may block in freeze_super() before task_work runs */
+ if (kiocb->ki_flags & IOCB_WRITE)
+ io_req_end_write(req);
+
if (!kiocb->dio_complete || !(kiocb->ki_flags & IOCB_DIO_CALLER_COMP)) {
if (__io_complete_rw_common(req, res))
return;
@@ -604,11 +600,13 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret,
if (!__io_complete_rw_common(req, ret)) {
u32 cflags = 0;
+ if (rw->kiocb.ki_flags & IOCB_WRITE)
+ io_req_end_write(req);
/*
- * Safe to call io_end from here as we're inline
+ * Safe to notify from here as we're inline
* from the submission path.
*/
- io_req_io_end(req);
+ io_req_io_notify(req);
if (sel)
cflags = io_put_kbuf(req, ret, sel->buf_list);
io_req_set_res(req, final_ret, cflags);
diff --git a/io_uring/waitid.c b/io_uring/waitid.c
index ef2283728c2b..e0f049ce5280 100644
--- a/io_uring/waitid.c
+++ b/io_uring/waitid.c
@@ -115,7 +115,23 @@ static int io_waitid_finish(struct io_kiocb *req, int ret)
return ret;
}
-static void io_waitid_complete(struct io_kiocb *req, int ret)
+static void io_waitid_remove_wq(struct io_kiocb *req)
+{
+ struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid);
+ struct wait_queue_head *head;
+
+ head = READ_ONCE(iw->head);
+ if (head) {
+ struct io_waitid_async *iwa = req->async_data;
+
+ iw->head = NULL;
+ spin_lock_irq(&head->lock);
+ list_del_init(&iwa->wo.child_wait.entry);
+ spin_unlock_irq(&head->lock);
+ }
+}
+
+static void io_waitid_complete(struct io_kiocb *req, int ret, bool copy_si)
{
struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid);
@@ -125,17 +141,23 @@ static void io_waitid_complete(struct io_kiocb *req, int ret)
lockdep_assert_held(&req->ctx->uring_lock);
hlist_del_init(&req->hash_node);
+ io_waitid_remove_wq(req);
- ret = io_waitid_finish(req, ret);
+ if (copy_si)
+ ret = io_waitid_finish(req, ret);
+ else
+ io_waitid_free(req);
if (ret < 0)
req_set_fail(req);
io_req_set_res(req, ret, 0);
}
-static bool __io_waitid_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req)
+static bool __io_waitid_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req,
+ bool copy_si)
{
struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid);
- struct io_waitid_async *iwa = req->async_data;
+
+ lockdep_assert_held(&req->ctx->uring_lock);
/*
* Mark us canceled regardless of ownership. This will prevent a
@@ -147,10 +169,7 @@ static bool __io_waitid_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req)
if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK)
return false;
- spin_lock_irq(&iw->head->lock);
- list_del_init(&iwa->wo.child_wait.entry);
- spin_unlock_irq(&iw->head->lock);
- io_waitid_complete(req, -ECANCELED);
+ io_waitid_complete(req, -ECANCELED, copy_si);
io_req_queue_tw_complete(req, -ECANCELED);
return true;
}
@@ -170,7 +189,7 @@ int io_waitid_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd,
if (req->cqe.user_data != cd->data &&
!(cd->flags & IORING_ASYNC_CANCEL_ANY))
continue;
- if (__io_waitid_cancel(ctx, req))
+ if (__io_waitid_cancel(ctx, req, true))
nr++;
if (!(cd->flags & IORING_ASYNC_CANCEL_ALL))
break;
@@ -196,7 +215,7 @@ bool io_waitid_remove_all(struct io_ring_ctx *ctx, struct task_struct *task,
if (!io_match_task_safe(req, task, cancel_all))
continue;
hlist_del_init(&req->hash_node);
- __io_waitid_cancel(ctx, req);
+ __io_waitid_cancel(ctx, req, task != NULL);
found = true;
}
@@ -251,12 +270,11 @@ static void io_waitid_cb(struct io_kiocb *req, struct io_tw_state *ts)
io_waitid_drop_issue_ref(req);
return;
}
-
- remove_wait_queue(iw->head, &iwa->wo.child_wait);
+ /* fall through to complete, will kill waitqueue */
}
}
- io_waitid_complete(req, ret);
+ io_waitid_complete(req, ret, true);
io_req_task_complete(req, ts);
}
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 7ec69545fe05..fcc0ca7ee883 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -735,13 +735,13 @@ static long bpf_for_each_array_elem(struct bpf_map *map, bpf_callback_t callback
u64 ret = 0;
void *val;
+ cant_migrate();
+
if (flags != 0)
return -EINVAL;
is_percpu = map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY;
array = container_of(map, struct bpf_array, map);
- if (is_percpu)
- migrate_disable();
for (i = 0; i < map->max_entries; i++) {
if (is_percpu)
val = this_cpu_ptr(array->pptrs[i]);
@@ -756,8 +756,6 @@ static long bpf_for_each_array_elem(struct bpf_map *map, bpf_callback_t callback
break;
}
- if (is_percpu)
- migrate_enable();
return num_elems;
}
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index f219a03ceeff..5f3b1162208b 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -4062,13 +4062,10 @@ int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
{
int i;
- /* There are three types that signify ownership of some other type:
- * kptr_ref, bpf_list_head, bpf_rb_root.
- * kptr_ref only supports storing kernel types, which can't store
- * references to program allocated local types.
- *
- * Hence we only need to ensure that bpf_{list_head,rb_root} ownership
- * does not form cycles.
+ /*
+ * Check fields which require the complete BTF and initialize runtime
+ * metadata. Ownership relationships are validated after every record has
+ * been fixed up.
*/
if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & BPF_GRAPH_ROOT))
return 0;
@@ -4083,51 +4080,88 @@ int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
if (!meta)
return -EFAULT;
rec->fields[i].graph_root.value_rec = meta->record;
+ }
+ return 0;
+}
- /* We need to set value_rec for all root types, but no need
- * to check ownership cycle for a type unless it's also a
- * node type.
- */
- if (!(rec->field_mask & BPF_GRAPH_NODE))
+static int btf_owned_type_idx(const struct btf *btf, struct btf_struct_metas *tab,
+ const struct btf_field *field)
+{
+ struct btf_struct_meta *meta;
+ u32 btf_id;
+
+ if (field->type & BPF_GRAPH_ROOT) {
+ btf_id = field->graph_root.value_btf_id;
+ } else if (field->type == BPF_KPTR_REF || field->type == BPF_KPTR_PERCPU) {
+ if (btf_is_kernel(field->kptr.btf))
+ return -ENOENT;
+ btf_id = field->kptr.btf_id;
+ } else {
+ return -ENOENT;
+ }
+
+ meta = btf_find_struct_meta(btf, btf_id);
+ if (!meta)
+ return field->type & BPF_GRAPH_ROOT ? -EFAULT : -ENOENT;
+ return meta - tab->types;
+}
+
+/*
+ * Each ownership edge adds kernel frames through bpf_obj_free_fields() and
+ * __bpf_obj_drop_impl(). Keep the bound deliberately small because object
+ * destruction can itself run below a BPF call chain. A final pointee without
+ * special fields is not present in the struct metadata table and adds only a
+ * non-recursing drop.
+ */
+#define BTF_MAX_OWNERSHIP_DEPTH 8
+
+static int btf_ownership_depth(const struct btf *btf,
+ struct btf_struct_metas *tab, u8 *depth,
+ int idx, int depth_left)
+{
+ const struct btf_record *rec = tab->types[idx].record;
+ int i, ret, max_depth = 0;
+
+ if (!depth_left)
+ return -ELOOP;
+ if (depth[idx])
+ goto done;
+
+ for (i = 0; i < rec->cnt; i++) {
+ ret = btf_owned_type_idx(btf, tab, &rec->fields[i]);
+ if (ret == -ENOENT)
continue;
+ if (ret < 0)
+ return ret;
+ ret = btf_ownership_depth(btf, tab, depth, ret, depth_left - 1);
+ if (ret < 0)
+ return ret;
+ max_depth = max(max_depth, ret);
+ }
+ depth[idx] = max_depth + 1;
+done:
+ return depth[idx] > depth_left ? -ELOOP : depth[idx];
+}
- /* We need to ensure ownership acyclicity among all types. The
- * proper way to do it would be to topologically sort all BTF
- * IDs based on the ownership edges, since there can be multiple
- * bpf_{list_head,rb_node} in a type. Instead, we use the
- * following resaoning:
- *
- * - A type can only be owned by another type in user BTF if it
- * has a bpf_{list,rb}_node. Let's call these node types.
- * - A type can only _own_ another type in user BTF if it has a
- * bpf_{list_head,rb_root}. Let's call these root types.
- *
- * We ensure that if a type is both a root and node, its
- * element types cannot be root types.
- *
- * To ensure acyclicity:
- *
- * When A is an root type but not a node, its ownership
- * chain can be:
- * A -> B -> C
- * Where:
- * - A is an root, e.g. has bpf_rb_root.
- * - B is both a root and node, e.g. has bpf_rb_node and
- * bpf_list_head.
- * - C is only an root, e.g. has bpf_list_node
- *
- * When A is both a root and node, some other type already
- * owns it in the BTF domain, hence it can not own
- * another root type through any of the ownership edges.
- * A -> B
- * Where:
- * - A is both an root and node.
- * - B is only an node.
- */
- if (meta->record->field_mask & BPF_GRAPH_ROOT)
- return -ELOOP;
+static int btf_check_ownership_depth(const struct btf *btf,
+ struct btf_struct_metas *tab)
+{
+ u8 *depth;
+ int i, ret = 0;
+
+ depth = kvcalloc(tab->cnt, sizeof(*depth), GFP_KERNEL | __GFP_NOWARN);
+ if (!depth)
+ return -ENOMEM;
+
+ for (i = 0; i < tab->cnt; i++) {
+ ret = btf_ownership_depth(btf, tab, depth, i,
+ BTF_MAX_OWNERSHIP_DEPTH);
+ if (ret < 0)
+ break;
+ ret = 0;
}
- return 0;
+ kvfree(depth);
+ return ret;
}
static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
@@ -5811,6 +5845,10 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat
if (err < 0)
goto errout_meta;
}
+
+ err = btf_check_ownership_depth(btf, struct_meta_tab);
+ if (err < 0)
+ goto errout_meta;
}
err = finalize_log(&env->log, uattr, uattr_size);
@@ -6851,7 +6889,7 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
if (btf_type_is_int(t))
return WALK_SCALAR;
- if (!btf_type_is_struct(t))
+ if (!btf_type_is_struct(t) || !t->size)
goto error;
off = (off - moff) % t->size;
diff --git a/kernel/bpf/crypto.c b/kernel/bpf/crypto.c
index 1d024fe7248a..3837d4dcf48d 100644
--- a/kernel/bpf/crypto.c
+++ b/kernel/bpf/crypto.c
@@ -149,8 +149,9 @@ bpf_crypto_ctx_create(const struct bpf_crypto_params *params, u32 params__sz,
const struct bpf_crypto_type *type;
struct bpf_crypto_ctx *ctx;
- if (!params || params->reserved[0] || params->reserved[1] ||
- params__sz != sizeof(struct bpf_crypto_params)) {
+ if (!params ||
+ params__sz != sizeof(struct bpf_crypto_params) ||
+ params->reserved[0] || params->reserved[1]) {
*err = -EINVAL;
return NULL;
}
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 66eaf95f9dbe..98308f3a7f6e 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -129,6 +129,11 @@ struct htab_elem {
char key[] __aligned(8);
};
+struct htab_btf_record {
+ struct btf_record *record;
+ u32 key_size;
+};
+
static inline bool htab_is_prealloc(const struct bpf_htab *htab)
{
return !(htab->map.map_flags & BPF_F_NO_PREALLOC);
@@ -195,20 +200,25 @@ static bool htab_is_percpu(const struct bpf_htab *htab)
htab->map.map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH;
}
+static inline void *htab_elem_value(struct htab_elem *l, u32 key_size)
+{
+ return l->key + round_up(key_size, 8);
+}
+
static inline void htab_elem_set_ptr(struct htab_elem *l, u32 key_size,
void __percpu *pptr)
{
- *(void __percpu **)(l->key + roundup(key_size, 8)) = pptr;
+ *(void __percpu **)htab_elem_value(l, key_size) = pptr;
}
static inline void __percpu *htab_elem_get_ptr(struct htab_elem *l, u32 key_size)
{
- return *(void __percpu **)(l->key + roundup(key_size, 8));
+ return *(void __percpu **)htab_elem_value(l, key_size);
}
static void *fd_htab_map_get_ptr(const struct bpf_map *map, struct htab_elem *l)
{
- return *(void **)(l->key + roundup(map->key_size, 8));
+ return *(void **)htab_elem_value(l, map->key_size);
}
static struct htab_elem *get_htab_elem(struct bpf_htab *htab, int i)
@@ -235,10 +245,10 @@ static void htab_free_prealloced_timers_and_wq(struct bpf_htab *htab)
elem = get_htab_elem(htab, i);
if (btf_record_has_field(htab->map.record, BPF_TIMER))
bpf_obj_free_timer(htab->map.record,
- elem->key + round_up(htab->map.key_size, 8));
+ htab_elem_value(elem, htab->map.key_size));
if (btf_record_has_field(htab->map.record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(htab->map.record,
- elem->key + round_up(htab->map.key_size, 8));
+ htab_elem_value(elem, htab->map.key_size));
cond_resched();
}
}
@@ -265,7 +275,8 @@ static void htab_free_prealloced_fields(struct bpf_htab *htab)
cond_resched();
}
} else {
- bpf_obj_free_fields(htab->map.record, elem->key + round_up(htab->map.key_size, 8));
+ bpf_obj_free_fields(htab->map.record,
+ htab_elem_value(elem, htab->map.key_size));
cond_resched();
}
cond_resched();
@@ -469,6 +480,87 @@ static int htab_map_alloc_check(union bpf_attr *attr)
return 0;
}
+static void htab_mem_dtor(void *obj, void *ctx)
+{
+ struct htab_btf_record *hrec = ctx;
+ struct htab_elem *elem = obj;
+ void *map_value;
+
+ if (IS_ERR_OR_NULL(hrec->record))
+ return;
+
+ map_value = htab_elem_value(elem, hrec->key_size);
+ bpf_obj_free_fields(hrec->record, map_value);
+}
+
+static void htab_pcpu_mem_dtor(void *obj, void *ctx)
+{
+ void __percpu *pptr = *(void __percpu **)obj;
+ struct htab_btf_record *hrec = ctx;
+ int cpu;
+
+ if (IS_ERR_OR_NULL(hrec->record))
+ return;
+
+ for_each_possible_cpu(cpu)
+ bpf_obj_free_fields(hrec->record, per_cpu_ptr(pptr, cpu));
+}
+
+static void htab_dtor_ctx_free(void *ctx)
+{
+ struct htab_btf_record *hrec = ctx;
+
+ btf_record_free(hrec->record);
+ kfree(ctx);
+}
+
+static int htab_set_dtor(const struct bpf_htab *htab, void (*dtor)(void *, void *))
+{
+ u32 key_size = htab->map.key_size;
+ const struct bpf_mem_alloc *ma;
+ struct htab_btf_record *hrec;
+ int err;
+
+ /* No need for dtors. */
+ if (IS_ERR_OR_NULL(htab->map.record))
+ return 0;
+
+ hrec = kzalloc(sizeof(*hrec), GFP_KERNEL);
+ if (!hrec)
+ return -ENOMEM;
+ hrec->key_size = key_size;
+ hrec->record = btf_record_dup(htab->map.record);
+ if (IS_ERR(hrec->record)) {
+ err = PTR_ERR(hrec->record);
+ kfree(hrec);
+ return err;
+ }
+ ma = htab_is_percpu(htab) ? &htab->pcpu_ma : &htab->ma;
+ /* Kinda sad, but cast away const-ness since we change ma->dtor. */
+ bpf_mem_alloc_set_dtor((struct bpf_mem_alloc *)ma, dtor, htab_dtor_ctx_free, hrec);
+ return 0;
+}
+
+static int htab_map_check_btf(const struct bpf_map *map, const struct btf *btf,
+ const struct btf_type *key_type, const struct btf_type *value_type)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+
+ if (btf_type_is_void(key_type))
+ return -EINVAL;
+
+ if (htab_is_prealloc(htab))
+ return 0;
+ /*
+ * We must set the dtor using this callback, as map's BTF record is not
+ * populated in htab_map_alloc(), so it will always appear as NULL.
+ */
+ if (htab_is_percpu(htab))
+ return htab_set_dtor(htab, htab_pcpu_mem_dtor);
+ else
+ return htab_set_dtor(htab, htab_mem_dtor);
+}
+
static struct bpf_map *htab_map_alloc(union bpf_attr *attr)
{
bool percpu = (attr->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
@@ -703,7 +795,7 @@ static void *htab_map_lookup_elem(struct bpf_map *map, void *key)
struct htab_elem *l = __htab_map_lookup_elem(map, key);
if (l)
- return l->key + round_up(map->key_size, 8);
+ return htab_elem_value(l, map->key_size);
return NULL;
}
@@ -742,7 +834,7 @@ static __always_inline void *__htab_lru_map_lookup_elem(struct bpf_map *map,
if (l) {
if (mark)
bpf_lru_node_set_ref(&l->lru_node);
- return l->key + round_up(map->key_size, 8);
+ return htab_elem_value(l, map->key_size);
}
return NULL;
@@ -793,7 +885,7 @@ static void check_and_free_fields(struct bpf_htab *htab,
for_each_possible_cpu(cpu)
bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu));
} else {
- void *map_value = elem->key + round_up(htab->map.key_size, 8);
+ void *map_value = htab_elem_value(elem, htab->map.key_size);
bpf_obj_free_fields(htab->map.record, map_value);
}
@@ -1076,11 +1168,9 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
htab_elem_set_ptr(l_new, key_size, pptr);
} else if (fd_htab_map_needs_adjust(htab)) {
size = round_up(size, 8);
- memcpy(l_new->key + round_up(key_size, 8), value, size);
+ memcpy(htab_elem_value(l_new, key_size), value, size);
} else {
- copy_map_value(&htab->map,
- l_new->key + round_up(key_size, 8),
- value);
+ copy_map_value(&htab->map, htab_elem_value(l_new, key_size), value);
}
l_new->hash = hash;
@@ -1142,7 +1232,7 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value,
if (l_old) {
/* grab the element lock and update value in place */
copy_map_value_locked(map,
- l_old->key + round_up(key_size, 8),
+ htab_elem_value(l_old, key_size),
value, false);
return 0;
}
@@ -1170,7 +1260,7 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value,
* and update element in place
*/
copy_map_value_locked(map,
- l_old->key + round_up(key_size, 8),
+ htab_elem_value(l_old, key_size),
value, false);
ret = 0;
goto err;
@@ -1255,8 +1345,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
l_new = prealloc_lru_pop(htab, key, hash);
if (!l_new)
return -ENOMEM;
- copy_map_value(&htab->map,
- l_new->key + round_up(map->key_size, 8), value);
+ copy_map_value(&htab->map, htab_elem_value(l_new, map->key_size), value);
ret = htab_lock_bucket(htab, b, hash, &flags);
if (ret)
@@ -1533,10 +1622,10 @@ static void htab_free_malloced_timers_and_wq(struct bpf_htab *htab)
/* We only free timer on uref dropping to zero */
if (btf_record_has_field(htab->map.record, BPF_TIMER))
bpf_obj_free_timer(htab->map.record,
- l->key + round_up(htab->map.key_size, 8));
+ htab_elem_value(l, htab->map.key_size));
if (btf_record_has_field(htab->map.record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(htab->map.record,
- l->key + round_up(htab->map.key_size, 8));
+ htab_elem_value(l, htab->map.key_size));
}
cond_resched_rcu();
}
@@ -1637,37 +1726,35 @@ static int __htab_map_lookup_and_delete_elem(struct bpf_map *map, void *key,
l = lookup_elem_raw(head, hash, key, key_size);
if (!l) {
ret = -ENOENT;
- } else {
- if (is_percpu) {
- u32 roundup_value_size = round_up(map->value_size, 8);
- void __percpu *pptr;
- int off = 0, cpu;
+ goto out_unlock;
+ }
- pptr = htab_elem_get_ptr(l, key_size);
- for_each_possible_cpu(cpu) {
- copy_map_value_long(&htab->map, value + off, per_cpu_ptr(pptr, cpu));
- check_and_init_map_value(&htab->map, value + off);
- off += roundup_value_size;
- }
- } else {
- u32 roundup_key_size = round_up(map->key_size, 8);
+ if (is_percpu) {
+ u32 roundup_value_size = round_up(map->value_size, 8);
+ void __percpu *pptr;
+ int off = 0, cpu;
- if (flags & BPF_F_LOCK)
- copy_map_value_locked(map, value, l->key +
- roundup_key_size,
- true);
- else
- copy_map_value(map, value, l->key +
- roundup_key_size);
- /* Zeroing special fields in the temp buffer */
- check_and_init_map_value(map, value);
+ pptr = htab_elem_get_ptr(l, key_size);
+ for_each_possible_cpu(cpu) {
+ copy_map_value_long(&htab->map, value + off, per_cpu_ptr(pptr, cpu));
+ check_and_init_map_value(&htab->map, value + off);
+ off += roundup_value_size;
}
+ } else {
+ void *src = htab_elem_value(l, map->key_size);
- hlist_nulls_del_rcu(&l->hash_node);
- if (!is_lru_map)
- free_htab_elem(htab, l);
+ if (flags & BPF_F_LOCK)
+ copy_map_value_locked(map, value, src, true);
+ else
+ copy_map_value(map, value, src);
+ /* Zeroing special fields in the temp buffer */
+ check_and_init_map_value(map, value);
}
+ hlist_nulls_del_rcu(&l->hash_node);
+ if (!is_lru_map)
+ free_htab_elem(htab, l);
+out_unlock:
htab_unlock_bucket(htab, b, hash, bflags);
if (is_lru_map && l)
@@ -1706,6 +1793,12 @@ static int htab_lru_percpu_map_lookup_and_delete_elem(struct bpf_map *map,
flags);
}
+/*
+ * Max consecutive empty buckets to walk in one RCU +
+ * instrumentation-disabled section before rescheduling.
+ */
+#define HTAB_BATCH_EMPTY_RESCHED 64
+
static int
__htab_map_lookup_and_delete_batch(struct bpf_map *map,
const union bpf_attr *attr,
@@ -1714,12 +1807,12 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
bool is_percpu)
{
struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
- u32 bucket_cnt, total, key_size, value_size, roundup_key_size;
void *keys = NULL, *values = NULL, *value, *dst_key, *dst_val;
void __user *uvalues = u64_to_user_ptr(attr->batch.values);
void __user *ukeys = u64_to_user_ptr(attr->batch.keys);
void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch);
u32 batch, max_count, size, bucket_size, map_id;
+ u32 bucket_cnt, total, key_size, value_size;
struct htab_elem *node_to_free = NULL;
u64 elem_map_flags, map_flags;
struct hlist_nulls_head *head;
@@ -1727,6 +1820,7 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
unsigned long flags = 0;
bool locked = false;
struct htab_elem *l;
+ u32 empty_cnt = 0;
struct bucket *b;
int ret = 0;
@@ -1754,7 +1848,6 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
return -ENOENT;
key_size = htab->map.key_size;
- roundup_key_size = round_up(htab->map.key_size, 8);
value_size = htab->map.value_size;
size = round_up(value_size, 8);
if (is_percpu)
@@ -1846,7 +1939,7 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
off += size;
}
} else {
- value = l->key + roundup_key_size;
+ value = htab_elem_value(l, key_size);
if (map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) {
struct bpf_map **inner_map = value;
@@ -1896,12 +1989,21 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
}
next_batch:
- /* If we are not copying data, we can go to next bucket and avoid
- * unlocking the rcu.
+ /*
+ * If we are not copying data, we can go to next bucket and avoid
+ * unlocking the rcu. Bound the walk though: after
+ * HTAB_BATCH_EMPTY_RESCHED consecutive empty buckets, fully exit
+ * the critical section (no locks are held here) and reschedule.
*/
if (!bucket_cnt && (batch + 1 < htab->n_buckets)) {
batch++;
- goto again_nocopy;
+ if (++empty_cnt < HTAB_BATCH_EMPTY_RESCHED)
+ goto again_nocopy;
+ empty_cnt = 0;
+ rcu_read_unlock();
+ bpf_enable_instrumentation();
+ cond_resched_tasks_rcu_qs();
+ goto again;
}
rcu_read_unlock();
@@ -1915,11 +2017,13 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
}
total += bucket_cnt;
+ empty_cnt = 0;
batch++;
if (batch >= htab->n_buckets) {
ret = -ENOENT;
goto after_loop;
}
+ cond_resched_tasks_rcu_qs();
goto again;
after_loop:
@@ -2097,11 +2201,11 @@ static void *bpf_hash_map_seq_next(struct seq_file *seq, void *v, loff_t *pos)
static int __bpf_hash_map_seq_show(struct seq_file *seq, struct htab_elem *elem)
{
struct bpf_iter_seq_hash_map_info *info = seq->private;
- u32 roundup_key_size, roundup_value_size;
struct bpf_iter__bpf_map_elem ctx = {};
struct bpf_map *map = info->map;
struct bpf_iter_meta meta;
int ret = 0, off = 0, cpu;
+ u32 roundup_value_size;
struct bpf_prog *prog;
void __percpu *pptr;
@@ -2111,10 +2215,9 @@ static int __bpf_hash_map_seq_show(struct seq_file *seq, struct htab_elem *elem)
ctx.meta = &meta;
ctx.map = info->map;
if (elem) {
- roundup_key_size = round_up(map->key_size, 8);
ctx.key = elem->key;
if (!info->percpu_value_buf) {
- ctx.value = elem->key + roundup_key_size;
+ ctx.value = htab_elem_value(elem, map->key_size);
} else {
roundup_value_size = round_up(map->value_size, 8);
pptr = htab_elem_get_ptr(elem, map->key_size);
@@ -2199,7 +2302,6 @@ static long bpf_for_each_hash_elem(struct bpf_map *map, bpf_callback_t callback_
struct hlist_nulls_head *head;
struct hlist_nulls_node *n;
struct htab_elem *elem;
- u32 roundup_key_size;
int i, num_elems = 0;
void __percpu *pptr;
struct bucket *b;
@@ -2207,17 +2309,17 @@ static long bpf_for_each_hash_elem(struct bpf_map *map, bpf_callback_t callback_
bool is_percpu;
u64 ret = 0;
+ cant_migrate();
+
if (flags != 0)
return -EINVAL;
is_percpu = htab_is_percpu(htab);
- roundup_key_size = round_up(map->key_size, 8);
- /* disable migration so percpu value prepared here will be the
- * same as the one seen by the bpf program with bpf_map_lookup_elem().
+ /* migration has been disabled, so percpu value prepared here will be
+ * the same as the one seen by the bpf program with
+ * bpf_map_lookup_elem().
*/
- if (is_percpu)
- migrate_disable();
for (i = 0; i < htab->n_buckets; i++) {
b = &htab->buckets[i];
rcu_read_lock();
@@ -2229,7 +2331,7 @@ static long bpf_for_each_hash_elem(struct bpf_map *map, bpf_callback_t callback_
pptr = htab_elem_get_ptr(elem, map->key_size);
val = this_cpu_ptr(pptr);
} else {
- val = elem->key + roundup_key_size;
+ val = htab_elem_value(elem, map->key_size);
}
num_elems++;
ret = callback_fn((u64)(long)map, (u64)(long)key,
@@ -2243,8 +2345,6 @@ static long bpf_for_each_hash_elem(struct bpf_map *map, bpf_callback_t callback_
rcu_read_unlock();
}
out:
- if (is_percpu)
- migrate_enable();
return num_elems;
}
@@ -2302,6 +2402,7 @@ const struct bpf_map_ops htab_map_ops = {
.map_seq_show_elem = htab_map_seq_show_elem,
.map_set_for_each_callback_args = map_set_for_each_callback_args,
.map_for_each_callback = bpf_for_each_hash_elem,
+ .map_check_btf = htab_map_check_btf,
.map_mem_usage = htab_map_mem_usage,
BATCH_OPS(htab),
.map_btf_id = &htab_map_btf_ids[0],
@@ -2324,6 +2425,7 @@ const struct bpf_map_ops htab_lru_map_ops = {
.map_seq_show_elem = htab_map_seq_show_elem,
.map_set_for_each_callback_args = map_set_for_each_callback_args,
.map_for_each_callback = bpf_for_each_hash_elem,
+ .map_check_btf = htab_map_check_btf,
.map_mem_usage = htab_map_mem_usage,
BATCH_OPS(htab_lru),
.map_btf_id = &htab_map_btf_ids[0],
@@ -2497,6 +2599,7 @@ const struct bpf_map_ops htab_percpu_map_ops = {
.map_seq_show_elem = htab_percpu_map_seq_show_elem,
.map_set_for_each_callback_args = map_set_for_each_callback_args,
.map_for_each_callback = bpf_for_each_hash_elem,
+ .map_check_btf = htab_map_check_btf,
.map_mem_usage = htab_map_mem_usage,
BATCH_OPS(htab_percpu),
.map_btf_id = &htab_map_btf_ids[0],
@@ -2517,6 +2620,7 @@ const struct bpf_map_ops htab_lru_percpu_map_ops = {
.map_seq_show_elem = htab_percpu_map_seq_show_elem,
.map_set_for_each_callback_args = map_set_for_each_callback_args,
.map_for_each_callback = bpf_for_each_hash_elem,
+ .map_check_btf = htab_map_check_btf,
.map_mem_usage = htab_map_mem_usage,
BATCH_OPS(htab_lru_percpu),
.map_btf_id = &htab_map_btf_ids[0],
diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c
index 146f5b57cfb1..bbc08b869aae 100644
--- a/kernel/bpf/memalloc.c
+++ b/kernel/bpf/memalloc.c
@@ -102,6 +102,8 @@ struct bpf_mem_cache {
int percpu_size;
bool draining;
struct bpf_mem_cache *tgt;
+ void (*dtor)(void *obj, void *ctx);
+ void *dtor_ctx;
/* list of objects to be freed after RCU GP */
struct llist_head free_by_rcu;
@@ -117,6 +119,7 @@ struct bpf_mem_cache {
struct llist_head waiting_for_gp_ttrace;
struct rcu_head rcu_ttrace;
atomic_t call_rcu_ttrace_in_progress;
+ raw_spinlock_t lock;
};
struct bpf_mem_caches {
@@ -212,25 +215,24 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node, bool atomic)
gfp = __GFP_NOWARN | __GFP_ACCOUNT;
gfp |= atomic ? GFP_NOWAIT : GFP_KERNEL;
- for (i = 0; i < cnt; i++) {
- /*
- * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
- * done only by one CPU == current CPU. Other CPUs might
- * llist_add() and llist_del_all() in parallel.
- */
- obj = llist_del_first(&c->free_by_rcu_ttrace);
- if (!obj)
- break;
- add_obj_to_free_list(c, obj);
- }
- if (i >= cnt)
- return;
+ /*
+ * c->lock serializes concurrent llist_del_first() against
+ * llist_del_all() in __free_rcu() and do_call_rcu_ttrace().
+ */
+ scoped_guard(raw_spinlock_irqsave, &c->lock) {
+ for (i = 0; i < cnt; i++) {
+ obj = llist_del_first(&c->free_by_rcu_ttrace);
+ if (!obj)
+ break;
+ add_obj_to_free_list(c, obj);
+ }
- for (; i < cnt; i++) {
- obj = llist_del_first(&c->waiting_for_gp_ttrace);
- if (!obj)
- break;
- add_obj_to_free_list(c, obj);
+ for (; i < cnt; i++) {
+ obj = llist_del_first(&c->waiting_for_gp_ttrace);
+ if (!obj)
+ break;
+ add_obj_to_free_list(c, obj);
+ }
}
if (i >= cnt)
return;
@@ -263,12 +265,14 @@ static void free_one(void *obj, bool percpu)
kfree(obj);
}
-static int free_all(struct llist_node *llnode, bool percpu)
+static int free_all(struct bpf_mem_cache *c, struct llist_node *llnode, bool percpu)
{
struct llist_node *pos, *t;
int cnt = 0;
llist_for_each_safe(pos, t, llnode) {
+ if (c->dtor)
+ c->dtor((void *)pos + LLIST_NODE_SZ, c->dtor_ctx);
free_one(pos, percpu);
cnt++;
}
@@ -278,8 +282,12 @@ static int free_all(struct llist_node *llnode, bool percpu)
static void __free_rcu(struct rcu_head *head)
{
struct bpf_mem_cache *c = container_of(head, struct bpf_mem_cache, rcu_ttrace);
+ struct llist_node *llnode;
+
+ scoped_guard(raw_spinlock_irqsave, &c->lock)
+ llnode = llist_del_all(&c->waiting_for_gp_ttrace);
- free_all(llist_del_all(&c->waiting_for_gp_ttrace), !!c->percpu_size);
+ free_all(c, llnode, !!c->percpu_size);
atomic_set(&c->call_rcu_ttrace_in_progress, 0);
}
@@ -310,8 +318,9 @@ static void do_call_rcu_ttrace(struct bpf_mem_cache *c)
if (atomic_xchg(&c->call_rcu_ttrace_in_progress, 1)) {
if (unlikely(READ_ONCE(c->draining))) {
- llnode = llist_del_all(&c->free_by_rcu_ttrace);
- free_all(llnode, !!c->percpu_size);
+ scoped_guard(raw_spinlock_irqsave, &c->lock)
+ llnode = llist_del_all(&c->free_by_rcu_ttrace);
+ free_all(c, llnode, !!c->percpu_size);
}
return;
}
@@ -420,7 +429,7 @@ static void check_free_by_rcu(struct bpf_mem_cache *c)
dec_active(c, &flags);
if (unlikely(READ_ONCE(c->draining))) {
- free_all(llist_del_all(&c->waiting_for_gp), !!c->percpu_size);
+ free_all(c, llist_del_all(&c->waiting_for_gp), !!c->percpu_size);
atomic_set(&c->call_rcu_in_progress, 0);
} else {
call_rcu_hurry(&c->rcu, __free_by_rcu);
@@ -545,6 +554,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu)
c->objcg = objcg;
c->percpu_size = percpu_size;
c->tgt = c;
+ raw_spin_lock_init(&c->lock);
init_refill_work(c);
prefill_mem_cache(c, cpu);
}
@@ -567,7 +577,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu)
c->objcg = objcg;
c->percpu_size = percpu_size;
c->tgt = c;
-
+ raw_spin_lock_init(&c->lock);
init_refill_work(c);
prefill_mem_cache(c, cpu);
}
@@ -619,7 +629,7 @@ int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, int size)
c->objcg = objcg;
c->percpu_size = percpu_size;
c->tgt = c;
-
+ raw_spin_lock_init(&c->lock);
init_refill_work(c);
prefill_mem_cache(c, cpu);
}
@@ -638,13 +648,13 @@ static void drain_mem_cache(struct bpf_mem_cache *c)
* Except for waiting_for_gp_ttrace list, there are no concurrent operations
* on these lists, so it is safe to use __llist_del_all().
*/
- free_all(llist_del_all(&c->free_by_rcu_ttrace), percpu);
- free_all(llist_del_all(&c->waiting_for_gp_ttrace), percpu);
- free_all(__llist_del_all(&c->free_llist), percpu);
- free_all(__llist_del_all(&c->free_llist_extra), percpu);
- free_all(__llist_del_all(&c->free_by_rcu), percpu);
- free_all(__llist_del_all(&c->free_llist_extra_rcu), percpu);
- free_all(llist_del_all(&c->waiting_for_gp), percpu);
+ free_all(c, llist_del_all(&c->free_by_rcu_ttrace), percpu);
+ free_all(c, llist_del_all(&c->waiting_for_gp_ttrace), percpu);
+ free_all(c, __llist_del_all(&c->free_llist), percpu);
+ free_all(c, __llist_del_all(&c->free_llist_extra), percpu);
+ free_all(c, __llist_del_all(&c->free_by_rcu), percpu);
+ free_all(c, __llist_del_all(&c->free_llist_extra_rcu), percpu);
+ free_all(c, llist_del_all(&c->waiting_for_gp), percpu);
}
static void check_mem_cache(struct bpf_mem_cache *c)
@@ -683,6 +693,9 @@ static void check_leaked_objs(struct bpf_mem_alloc *ma)
static void free_mem_alloc_no_barrier(struct bpf_mem_alloc *ma)
{
+ /* We can free dtor ctx only once all callbacks are done using it. */
+ if (ma->dtor_ctx_free)
+ ma->dtor_ctx_free(ma->dtor_ctx);
check_leaked_objs(ma);
free_percpu(ma->cache);
free_percpu(ma->caches);
@@ -1017,3 +1030,32 @@ int bpf_mem_alloc_check_size(bool percpu, size_t size)
return 0;
}
+
+void bpf_mem_alloc_set_dtor(struct bpf_mem_alloc *ma, void (*dtor)(void *obj, void *ctx),
+ void (*dtor_ctx_free)(void *ctx), void *ctx)
+{
+ struct bpf_mem_caches *cc;
+ struct bpf_mem_cache *c;
+ int cpu, i;
+
+ ma->dtor_ctx_free = dtor_ctx_free;
+ ma->dtor_ctx = ctx;
+
+ if (ma->cache) {
+ for_each_possible_cpu(cpu) {
+ c = per_cpu_ptr(ma->cache, cpu);
+ c->dtor = dtor;
+ c->dtor_ctx = ctx;
+ }
+ }
+ if (ma->caches) {
+ for_each_possible_cpu(cpu) {
+ cc = per_cpu_ptr(ma->caches, cpu);
+ for (i = 0; i < NUM_CACHES; i++) {
+ c = &cc->cache[i];
+ c->dtor = dtor;
+ c->dtor_ctx = ctx;
+ }
+ }
+ }
+}
diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c
index 56115739fcfd..1b6f9e717566 100644
--- a/kernel/bpf/offload.c
+++ b/kernel/bpf/offload.c
@@ -703,6 +703,8 @@ static bool __bpf_offload_dev_match(struct bpf_prog *prog,
return false;
if (offload->netdev == netdev)
return true;
+ if (!bpf_prog_is_offloaded(prog->aux))
+ return false;
ondev1 = bpf_offload_find_netdev(offload->netdev);
ondev2 = bpf_offload_find_netdev(netdev);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 954b299f241b..857285a48d43 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -5472,7 +5472,10 @@ struct bpf_link *bpf_link_get_curr_or_next(u32 *id)
again:
link = idr_get_next(&link_idr, id);
if (link) {
- link = bpf_link_inc_not_zero(link);
+ if (link->id)
+ link = bpf_link_inc_not_zero(link);
+ else
+ link = ERR_PTR(-EAGAIN);
if (IS_ERR(link)) {
(*id)++;
goto again;
diff --git a/kernel/cgroup/pids.c b/kernel/cgroup/pids.c
index 8f61114c36dd..5b64c8dde5eb 100644
--- a/kernel/cgroup/pids.c
+++ b/kernel/cgroup/pids.c
@@ -253,6 +253,11 @@ static void pids_event(struct pids_cgroup *pids_forking,
}
if (!cgroup_subsys_on_dfl(pids_cgrp_subsys) ||
cgrp_dfl_root.flags & CGRP_ROOT_PIDS_LOCAL_EVENTS) {
+ /*
+ * pids.events reports the local counter on legacy hierarchies
+ * and when pids_localevents is enabled.
+ */
+ cgroup_file_notify(&p->events_file);
cgroup_file_notify(&p->events_local_file);
return;
}
diff --git a/kernel/dma/coherent.c b/kernel/dma/coherent.c
index 3b2bdca9f1d4..87f2f02e921a 100644
--- a/kernel/dma/coherent.c
+++ b/kernel/dma/coherent.c
@@ -336,17 +336,22 @@ static phys_addr_t dma_reserved_default_memory_size __initdata;
static int rmem_dma_device_init(struct reserved_mem *rmem, struct device *dev)
{
- if (!rmem->priv) {
- struct dma_coherent_mem *mem;
+ struct dma_coherent_mem *mem = rmem->priv;
+ if (!mem) {
mem = dma_init_coherent_memory(rmem->base, rmem->base,
rmem->size, true);
if (IS_ERR(mem))
return PTR_ERR(mem);
rmem->priv = mem;
}
- dma_assign_coherent_memory(dev, rmem->priv);
- return 0;
+
+ /* Warn if the device potentially can't use the reserved memory */
+ if (mem->device_base + rmem->size - 1 >
+ min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit))
+ dev_warn(dev, "reserved memory is beyond device's set DMA address range\n");
+
+ return dma_assign_coherent_memory(dev, mem);
}
static void rmem_dma_device_release(struct reserved_mem *rmem,
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index ffb7b021e2fb..90651d535a92 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -869,7 +869,6 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size
int index = (tlb_addr - mem->start) >> IO_TLB_SHIFT;
phys_addr_t orig_addr = mem->slots[index].orig_addr;
size_t alloc_size = mem->slots[index].alloc_size;
- unsigned long pfn = PFN_DOWN(orig_addr);
unsigned char *vaddr = mem->vaddr + tlb_addr - mem->start;
int tlb_offset;
@@ -899,7 +898,8 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size
size = alloc_size;
}
- if (PageHighMem(pfn_to_page(pfn))) {
+ if (PageHighMem(pfn_to_page(PFN_DOWN(orig_addr)))) {
+ unsigned long pfn = PFN_DOWN(orig_addr);
unsigned int offset = orig_addr & ~PAGE_MASK;
struct page *page;
unsigned int sz = 0;
diff --git a/kernel/events/core.c b/kernel/events/core.c
index bdc90afc0e37..dfbd217fd20c 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -3603,6 +3603,9 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx,
list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) {
cpc = this_cpu_ptr(pmu_ctx->pmu->cpu_pmu_context);
+ if (cpc->task_epc != pmu_ctx)
+ continue;
+
if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task)
pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in);
}
@@ -3754,7 +3757,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc,
perf_ctx_lock(cpuctx, cpuctx->task_ctx);
perf_pmu_disable(pmu);
- pmu->sched_task(cpc->task_epc, task, sched_in);
+ pmu->sched_task(&cpc->epc, task, sched_in);
perf_pmu_enable(pmu);
perf_ctx_unlock(cpuctx, cpuctx->task_ctx);
@@ -3764,15 +3767,17 @@ static void perf_pmu_sched_task(struct task_struct *prev,
struct task_struct *next,
bool sched_in)
{
- struct perf_cpu_context *cpuctx = this_cpu_ptr(&perf_cpu_context);
struct perf_cpu_pmu_context *cpc, *cpc2;
- /* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */
- if (prev == next || cpuctx->task_ctx)
+ if (prev == next)
return;
- list_for_each_entry_safe(cpc, cpc2, this_cpu_ptr(&sched_cb_list), sched_cb_entry)
+ list_for_each_entry_safe(cpc, cpc2, this_cpu_ptr(&sched_cb_list), sched_cb_entry) {
+ if (cpc->task_epc)
+ continue;
+
__perf_pmu_sched_task(cpc, sched_in ? next : prev, sched_in);
+ }
}
static void perf_event_switch(struct task_struct *task,
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 2f25a8a351e9..03cbc11a887f 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -685,7 +685,15 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
{
bool overwrite = !(flags & RING_BUFFER_WRITABLE);
int node = (event->cpu == -1) ? -1 : cpu_to_node(event->cpu);
- int ret = -ENOMEM, max_order;
+ bool use_contiguous_pages = event->pmu->capabilities & (
+ PERF_PMU_CAP_AUX_NO_SG | PERF_PMU_CAP_AUX_PREFER_LARGE);
+ /*
+ * Initialize max_order to 0 for page allocation. This allocates single
+ * pages to minimize memory fragmentation. This is overridden if the
+ * PMU needs or prefers contiguous pages (use_contiguous_pages = true).
+ */
+ int max_order = 0;
+ int ret = -ENOMEM;
if (!has_aux(event))
return -EOPNOTSUPP;
@@ -695,8 +703,8 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
if (!overwrite) {
/*
- * Watermark defaults to half the buffer, and so does the
- * max_order, to aid PMU drivers in double buffering.
+ * Watermark defaults to half the buffer, to aid PMU drivers
+ * in double buffering.
*/
if (!watermark)
watermark = min_t(unsigned long,
@@ -704,16 +712,19 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
(unsigned long)nr_pages << (PAGE_SHIFT - 1));
/*
- * Use aux_watermark as the basis for chunking to
- * help PMU drivers honor the watermark.
+ * If using contiguous pages, use aux_watermark as the basis
+ * for chunking to help PMU drivers honor the watermark.
*/
- max_order = get_order(watermark);
+ if (use_contiguous_pages)
+ max_order = get_order(watermark);
} else {
/*
- * We need to start with the max_order that fits in nr_pages,
- * not the other way around, hence ilog2() and not get_order.
+ * If using contiguous pages, we need to start with the
+ * max_order that fits in nr_pages, not the other way around,
+ * hence ilog2() and not get_order.
*/
- max_order = ilog2(nr_pages);
+ if (use_contiguous_pages)
+ max_order = ilog2(nr_pages);
watermark = 0;
}
diff --git a/kernel/futex/pi.c b/kernel/futex/pi.c
index ccd5d19dea68..258faa26e6f9 100644
--- a/kernel/futex/pi.c
+++ b/kernel/futex/pi.c
@@ -992,12 +992,6 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl
goto no_block;
}
- /*
- * Must be done before we enqueue the waiter, here is unfortunately
- * under the hb lock, but that *should* work because it does nothing.
- */
- rt_mutex_pre_schedule();
-
rt_mutex_init_waiter(&rt_waiter);
/*
@@ -1065,10 +1059,6 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl
* the
*/
spin_lock(q.lock_ptr);
- /*
- * Waiter is unqueued.
- */
- rt_mutex_post_schedule();
no_block:
/*
* Fixup the pi_state owner and possibly acquire the lock if we
diff --git a/kernel/locking/rtmutex_api.c b/kernel/locking/rtmutex_api.c
index 5b438aa71f87..51622ce539e4 100644
--- a/kernel/locking/rtmutex_api.c
+++ b/kernel/locking/rtmutex_api.c
@@ -380,6 +380,7 @@ int __sched rt_mutex_wait_proxy_lock(struct rt_mutex_base *lock,
{
int ret;
+ rt_mutex_futex_pre_schedule();
raw_spin_lock_irq(&lock->wait_lock);
/* sleep on the mutex */
set_current_state(TASK_INTERRUPTIBLE);
@@ -390,6 +391,7 @@ int __sched rt_mutex_wait_proxy_lock(struct rt_mutex_base *lock,
*/
fixup_rt_mutex_waiters(lock, true);
raw_spin_unlock_irq(&lock->wait_lock);
+ rt_mutex_futex_post_schedule();
return ret;
}
diff --git a/kernel/reboot.c b/kernel/reboot.c
index d6ee090eda94..24c5bc26385a 100644
--- a/kernel/reboot.c
+++ b/kernel/reboot.c
@@ -13,7 +13,9 @@
#include <linux/kexec.h>
#include <linux/kmod.h>
#include <linux/kmsg_dump.h>
+#include <linux/rcupdate.h>
#include <linux/reboot.h>
+#include <linux/sched/signal.h>
#include <linux/suspend.h>
#include <linux/syscalls.h>
#include <linux/syscore_ops.h>
@@ -24,8 +26,7 @@
*/
static int C_A_D = 1;
-struct pid *cad_pid;
-EXPORT_SYMBOL(cad_pid);
+struct pid __rcu *cad_pid;
#if defined(CONFIG_ARM)
#define DEFAULT_REBOOT_MODE = REBOOT_HARD
@@ -1280,7 +1281,42 @@ static struct attribute *reboot_attrs[] = {
NULL,
};
+#endif /* CONFIG_SYSFS */
+
#ifdef CONFIG_SYSCTL
+static int proc_do_cad_pid(const struct ctl_table *table, int write, void *buffer,
+ size_t *lenp, loff_t *ppos)
+{
+ struct ctl_table tmp_table = *table;
+ struct pid *new_pid;
+ struct pid *old_pid;
+ pid_t tmp_pid;
+ int r;
+
+ rcu_read_lock();
+ tmp_pid = pid_vnr(rcu_dereference(cad_pid));
+ rcu_read_unlock();
+
+ tmp_table.data = &tmp_pid;
+
+ r = proc_dointvec(&tmp_table, write, buffer, lenp, ppos);
+ if (r || !write)
+ return r;
+
+ new_pid = find_get_pid(tmp_pid);
+ if (!new_pid)
+ return -ESRCH;
+
+ old_pid = unrcu_pointer(xchg(&cad_pid, RCU_INITIALIZER(new_pid)));
+ /*
+ * Wait for cad_pid readers before put_pid(). We cannot use
+ * call_rcu() here because free_pid() already owns pid->rcu.
+ */
+ synchronize_rcu();
+ put_pid(old_pid);
+ return 0;
+}
+
static struct ctl_table kern_reboot_table[] = {
{
.procname = "poweroff_cmd",
@@ -1296,16 +1332,23 @@ static struct ctl_table kern_reboot_table[] = {
.mode = 0644,
.proc_handler = proc_dointvec,
},
+ {
+ .procname = "cad_pid",
+ .maxlen = sizeof(int),
+ .mode = 0600,
+ .proc_handler = proc_do_cad_pid,
+ },
};
-static void __init kernel_reboot_sysctls_init(void)
+static int __init kernel_reboot_sysctls_init(void)
{
register_sysctl_init("kernel", kern_reboot_table);
+ return 0;
}
-#else
-#define kernel_reboot_sysctls_init() do { } while (0)
+late_initcall(kernel_reboot_sysctls_init);
#endif /* CONFIG_SYSCTL */
+#ifdef CONFIG_SYSFS
static const struct attribute_group reboot_attr_group = {
.attrs = reboot_attrs,
};
@@ -1325,8 +1368,6 @@ static int __init reboot_ksysfs_init(void)
return ret;
}
- kernel_reboot_sysctls_init();
-
return 0;
}
late_initcall(reboot_ksysfs_init);
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ea106403abd8..719186283bf9 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -401,6 +401,17 @@ static void __sched_core_flip(bool enabled)
sched_core_lock(cpu, &flags);
+ /*
+ * A core-wide selection may have the shared rq lock temporarily
+ * released by a lock-dropping ->pick_task(). Flipping would
+ * rebind rq_lockp() under it. Wait it out.
+ */
+ while (cpu_rq(cpu)->core->core_pick_in_flight) {
+ sched_core_unlock(cpu, &flags);
+ cpu_relax();
+ sched_core_lock(cpu, &flags);
+ }
+
for_each_cpu(t, smt_mask)
cpu_rq(t)->core_enabled = enabled;
@@ -5936,10 +5947,9 @@ static inline void schedule_debug(struct task_struct *prev, bool preempt)
schedstat_inc(this_rq()->sched_count);
}
-static void prev_balance(struct rq *rq, struct task_struct *prev,
- struct rq_flags *rf)
+static void prev_balance(struct rq *rq, struct rq_flags *rf)
{
- const struct sched_class *start_class = prev->sched_class;
+ const struct sched_class *start_class = rq->donor->sched_class;
const struct sched_class *class;
#ifdef CONFIG_SCHED_CLASS_EXT
@@ -5964,7 +5974,7 @@ static void prev_balance(struct rq *rq, struct task_struct *prev,
* a runnable task of @class priority or higher.
*/
for_active_class_range(class, start_class, &idle_sched_class) {
- if (class->balance && class->balance(rq, prev, rf))
+ if (class->balance && class->balance(rq, rq->donor, rf))
break;
}
}
@@ -5973,7 +5983,7 @@ static void prev_balance(struct rq *rq, struct task_struct *prev,
* Pick up the highest-prio task:
*/
static inline struct task_struct *
-__pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
+__pick_next_task(struct rq *rq, struct rq_flags *rf)
{
const struct sched_class *class;
struct task_struct *p;
@@ -5985,38 +5995,38 @@ __pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
/*
* Optimization: we know that if all tasks are in the fair class we can
- * call that function directly, but only if the @prev task wasn't of a
+ * call that function directly, but only if the current task wasn't of a
* higher scheduling class, because otherwise those lose the
* opportunity to pull in more work from other CPUs.
*/
- if (likely(!sched_class_above(prev->sched_class, &fair_sched_class) &&
+ if (likely(!sched_class_above(rq->donor->sched_class, &fair_sched_class) &&
rq->nr_running == rq->cfs.h_nr_queued)) {
- p = pick_next_task_fair(rq, prev, rf);
+ p = pick_next_task_fair(rq, rq->donor, rf);
if (unlikely(p == RETRY_TASK))
goto restart;
/* Assume the next prioritized class is idle_sched_class */
if (!p) {
p = pick_task_idle(rq);
- put_prev_set_next_task(rq, prev, p);
+ put_prev_set_next_task(rq, rq->donor, p);
}
return p;
}
restart:
- prev_balance(rq, prev, rf);
+ prev_balance(rq, rf);
for_each_active_class(class) {
if (class->pick_next_task) {
- p = class->pick_next_task(rq, prev);
+ p = class->pick_next_task(rq, rq->donor);
if (p)
return p;
} else {
p = class->pick_task(rq);
if (p) {
- put_prev_set_next_task(rq, prev, p);
+ put_prev_set_next_task(rq, rq->donor, p);
return p;
}
}
@@ -6065,7 +6075,7 @@ extern void task_vruntime_update(struct rq *rq, struct task_struct *p, bool in_f
static void queue_core_balance(struct rq *rq);
static struct task_struct *
-pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
+pick_next_task(struct rq *rq, struct rq_flags *rf)
{
struct task_struct *next, *p, *max = NULL;
const struct cpumask *smt_mask;
@@ -6077,7 +6087,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
bool need_sync;
if (!sched_core_enabled(rq))
- return __pick_next_task(rq, prev, rf);
+ return __pick_next_task(rq, rf);
cpu = cpu_of(rq);
@@ -6090,9 +6100,11 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
*/
rq->core_pick = NULL;
rq->core_dl_server = NULL;
- return __pick_next_task(rq, prev, rf);
+ return __pick_next_task(rq, rf);
}
+ rq->core->core_pick_in_flight++;
+
/*
* If there were no {en,de}queues since we picked (IOW, the task
* pointers are all still valid), and we haven't scheduled the last
@@ -6114,7 +6126,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
goto out_set_next;
}
- prev_balance(rq, prev, rf);
+ prev_balance(rq, rf);
smt_mask = cpu_smt_mask(cpu);
need_sync = !!rq->core->core_cookie;
@@ -6287,7 +6299,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
}
out_set_next:
- put_prev_set_next_task(rq, prev, next);
+ rq->core->core_pick_in_flight--;
+ put_prev_set_next_task(rq, rq->donor, next);
if (rq->core->core_forceidle_count && next == rq->idle)
queue_core_balance(rq);
@@ -6483,6 +6496,13 @@ static void sched_core_cpu_deactivate(unsigned int cpu)
core_rq->core_forceidle_seq = rq->core_forceidle_seq;
core_rq->core_forceidle_occupation = rq->core_forceidle_occupation;
+ /*
+ * A stale leftover would bias the count forever if this CPU later
+ * returns as its own leader. Move, don't copy.
+ */
+ core_rq->core_pick_in_flight = rq->core_pick_in_flight;
+ rq->core_pick_in_flight = 0;
+
/*
* Accounting edge for forced idle is handled in pick_next_task().
* Don't need another one here, since the hotplug thread shouldn't
@@ -6512,9 +6532,9 @@ static inline void sched_core_cpu_deactivate(unsigned int cpu) {}
static inline void sched_core_cpu_dying(unsigned int cpu) {}
static struct task_struct *
-pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
+pick_next_task(struct rq *rq, struct rq_flags *rf)
{
- return __pick_next_task(rq, prev, rf);
+ return __pick_next_task(rq, rf);
}
#endif /* CONFIG_SCHED_CORE */
@@ -6682,7 +6702,7 @@ static void __sched notrace __schedule(int sched_mode)
switch_count = &prev->nvcsw;
}
- next = pick_next_task(rq, prev, &rf);
+ next = pick_next_task(rq, &rf);
picked:
clear_tsk_need_resched(prev);
clear_preempt_need_resched();
@@ -7097,6 +7117,17 @@ void rt_mutex_pre_schedule(void)
sched_submit_work(current);
}
+/*
+ * Used within the futex syscall context, skips sched_submit_work() because none
+ * its work will be done. Asserts ensure that it is indeed the case.
+ */
+void rt_mutex_futex_pre_schedule(void)
+{
+ lockdep_assert(!(current->flags & (PF_WQ_WORKER | PF_IO_WORKER)));
+ lockdep_assert(!current->plug);
+ lockdep_assert(!fetch_and_set(current->sched_rt_mutex, 1));
+}
+
void rt_mutex_schedule(void)
{
lockdep_assert(current->sched_rt_mutex);
@@ -7109,6 +7140,11 @@ void rt_mutex_post_schedule(void)
lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0));
}
+void rt_mutex_futex_post_schedule(void)
+{
+ lockdep_assert(fetch_and_set(current->sched_rt_mutex, 0));
+}
+
/*
* rt_mutex_setprio - set the current priority of a task
* @p: task to boost
@@ -8554,6 +8590,7 @@ void __init sched_init(void)
rq->core_forceidle_count = 0;
rq->core_forceidle_occupation = 0;
rq->core_forceidle_start = 0;
+ rq->core_pick_in_flight = 0;
rq->core_cookie = 0UL;
#endif
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 7956abeb9154..0073c5b7d07a 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1184,7 +1184,11 @@ struct rq {
*/
unsigned long nr_uninterruptible;
- struct task_struct __rcu *curr;
+ /* Scheduling and execution contexts coincide without proxy execution. */
+ union {
+ struct task_struct __rcu *donor;
+ struct task_struct __rcu *curr;
+ };
struct sched_dl_entity *dl_server;
struct task_struct *idle;
struct task_struct *stop;
@@ -1326,7 +1330,8 @@ struct rq {
unsigned int core_forceidle_seq;
unsigned int core_forceidle_occupation;
u64 core_forceidle_start;
-#endif
+ unsigned int core_pick_in_flight;
+#endif /* CONFIG_SCHED_CORE */
/* Scratch cpumask to be temporarily used under rq_lock */
cpumask_var_t scratch_mask;
diff --git a/kernel/signal.c b/kernel/signal.c
index a11c3c9afd2b..1a6cd0503beb 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -1936,14 +1936,22 @@ int kill_pid(struct pid *pid, int sig, int priv)
}
EXPORT_SYMBOL(kill_pid);
+int kill_cad_pid(int sig, int priv)
+{
+ int ret;
+
+ rcu_read_lock();
+ ret = kill_pid(rcu_dereference(cad_pid), sig, priv);
+ rcu_read_unlock();
+
+ return ret;
+}
+EXPORT_SYMBOL(kill_cad_pid);
+
+#ifdef CONFIG_POSIX_TIMERS
/*
- * These functions support sending signals using preallocated sigqueue
- * structures. This is needed "because realtime applications cannot
- * afford to lose notifications of asynchronous events, like timer
- * expirations or I/O completions". In the case of POSIX Timers
- * we allocate the sigqueue structure from the timer_create. If this
- * allocation fails we are able to report the failure to the application
- * with an EAGAIN error.
+ * These functions handle POSIX timer signals. POSIX timers use
+ * preallocated sigqueue structs for sending signals.
*/
struct sigqueue *sigqueue_alloc(void)
{
@@ -2043,6 +2051,8 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
return ret;
}
+#endif /* CONFIG_POSIX_TIMERS */
+
void do_notify_pidfd(struct task_struct *task)
{
struct pid *pid = task_pid(task);
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index de58128fa521..6a9040fdefe5 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1322,28 +1322,6 @@ int proc_dointvec_ms_jiffies(const struct ctl_table *table, int write, void *buf
do_proc_dointvec_ms_jiffies_conv, NULL);
}
-static int proc_do_cad_pid(const struct ctl_table *table, int write, void *buffer,
- size_t *lenp, loff_t *ppos)
-{
- struct pid *new_pid;
- pid_t tmp;
- int r;
-
- tmp = pid_vnr(cad_pid);
-
- r = __do_proc_dointvec(&tmp, table, write, buffer,
- lenp, ppos, NULL, NULL);
- if (r || !write)
- return r;
-
- new_pid = find_get_pid(tmp);
- if (!new_pid)
- return -ESRCH;
-
- put_pid(xchg(&cad_pid, new_pid));
- return 0;
-}
-
/**
* proc_do_large_bitmap - read/write from/to a large bitmap
* @table: the sysctl table
@@ -1760,15 +1738,6 @@ static struct ctl_table kern_table[] = {
.mode = 0644,
.proc_handler = sysrq_sysctl_handler,
},
-#endif
-#ifdef CONFIG_PROC_SYSCTL
- {
- .procname = "cad_pid",
- .data = NULL,
- .maxlen = sizeof (int),
- .mode = 0600,
- .proc_handler = proc_do_cad_pid,
- },
#endif
{
.procname = "threads-max",
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 7347a4c27fab..79dfbdd218ac 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -4535,7 +4535,8 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file)
/**
* ftrace_regex_open - initialize function tracer filter files
- * @ops: The ftrace_ops that hold the hash filters
+ * @tr: The trace_array that holds the ftrace_ops [optional]
+ * @ops: The ftrace_ops that hold the hash filters [optional]
* @flag: The type of filter to process
* @inode: The inode, usually passed in to your open routine
* @file: The file, usually passed in to your open routine
@@ -4549,26 +4550,45 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file)
* tracing_lseek() should be used as the lseek routine, and
* release must call ftrace_regex_release().
*
+ * Note, If @tr is not NULL, its reference has to be taken before
+ * @ops may be referenced.
+ * If @ops is NULL and @tr is not, then @tr->ops is used.
+ * If @tr is NULL and @ops is not then @ops->private is uesd for @tr.
+ * If both @tr and @ops are NULL, then the &global_ops is
+ * to be used, and @tr will be the global_ops.private pointer.
+ *
* Returns: 0 on success or a negative errno value on failure
*/
int
-ftrace_regex_open(struct ftrace_ops *ops, int flag,
+ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag,
struct inode *inode, struct file *file)
{
- struct ftrace_iterator *iter;
+ struct ftrace_iterator *iter = NULL;
struct ftrace_hash *hash;
struct list_head *mod_head;
- struct trace_array *tr = ops->private;
- int ret = -ENOMEM;
-
- ftrace_ops_init(ops);
+ int ret = -ENODEV;
if (unlikely(ftrace_disabled))
return -ENODEV;
+ if (!tr) {
+ if (!ops)
+ ops = &global_ops;
+ tr = ops->private;
+ }
+
if (tracing_check_open_get_tr(tr))
return -ENODEV;
+ if (!ops)
+ ops = tr->ops;
+
+ if (WARN_ON_ONCE(!ops))
+ goto out;
+
+ ftrace_ops_init(ops);
+
+ ret = -ENOMEM;
iter = kzalloc(sizeof(*iter), GFP_KERNEL);
if (!iter)
goto out;
@@ -4646,21 +4666,19 @@ ftrace_regex_open(struct ftrace_ops *ops, int flag,
static int
ftrace_filter_open(struct inode *inode, struct file *file)
{
- struct ftrace_ops *ops = inode->i_private;
+ struct trace_array *tr = inode->i_private;
- /* Checks for tracefs lockdown */
- return ftrace_regex_open(ops,
- FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES,
- inode, file);
+ return ftrace_regex_open(tr, NULL,
+ FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES,
+ inode, file);
}
static int
ftrace_notrace_open(struct inode *inode, struct file *file)
{
- struct ftrace_ops *ops = inode->i_private;
+ struct trace_array *tr = inode->i_private;
- /* Checks for tracefs lockdown */
- return ftrace_regex_open(ops, FTRACE_ITER_NOTRACE,
+ return ftrace_regex_open(tr, NULL, FTRACE_ITER_NOTRACE,
inode, file);
}
@@ -6969,15 +6987,15 @@ static const struct file_operations ftrace_graph_notrace_fops = {
};
#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
-void ftrace_create_filter_files(struct ftrace_ops *ops,
+void ftrace_create_filter_files(struct trace_array *tr,
struct dentry *parent)
{
trace_create_file("set_ftrace_filter", TRACE_MODE_WRITE, parent,
- ops, &ftrace_filter_fops);
+ tr, &ftrace_filter_fops);
trace_create_file("set_ftrace_notrace", TRACE_MODE_WRITE, parent,
- ops, &ftrace_notrace_fops);
+ tr, &ftrace_notrace_fops);
}
/*
@@ -7002,7 +7020,6 @@ void ftrace_destroy_filter_files(struct ftrace_ops *ops)
static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)
{
-
trace_create_file("available_filter_functions", TRACE_MODE_READ,
d_tracer, NULL, &ftrace_avail_fops);
@@ -7015,7 +7032,7 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)
trace_create_file("touched_functions", TRACE_MODE_READ,
d_tracer, NULL, &ftrace_touched_fops);
- ftrace_create_filter_files(&global_ops, d_tracer);
+ ftrace_create_filter_files(NULL, d_tracer);
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
trace_create_file("set_graph_function", TRACE_MODE_WRITE, d_tracer,
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 6bbcf3348e49..c54e7e910b43 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -382,7 +382,7 @@ struct buffer_page {
#define RB_WRITE_MASK 0xfffff
#define RB_WRITE_INTCNT (1 << 20)
-static void rb_init_page(struct buffer_data_page *bpage)
+static void rb_init_data_page(struct buffer_data_page *bpage)
{
local_set(&bpage->commit, 0);
}
@@ -411,6 +411,21 @@ static void free_buffer_page(struct buffer_page *bpage)
kfree(bpage);
}
+/* Try hard to allocate data pages without invoking the OOM killer. */
+#define alloc_cpu_data(cpu, order) ({ \
+ struct buffer_data_page *__rb_data = NULL; \
+ struct page *__rb_page; \
+ \
+ __rb_page = alloc_pages_node(cpu_to_node(cpu), \
+ GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_COMP | __GFP_ZERO, \
+ (order)); \
+ if (__rb_page) { \
+ __rb_data = page_address(__rb_page); \
+ rb_init_data_page(__rb_data); \
+ } \
+ __rb_data; \
+})
+
/*
* We need to fit the time_stamp delta into 27 bits.
*/
@@ -492,7 +507,7 @@ struct ring_buffer_per_cpu {
raw_spinlock_t reader_lock; /* serialize readers */
arch_spinlock_t lock;
struct lock_class_key lock_key;
- struct buffer_data_page *free_page;
+ struct buffer_data_read_page free_page;
unsigned long nr_pages;
unsigned int current_context;
struct list_head *pages;
@@ -2016,7 +2031,7 @@ static void rb_range_meta_init(struct trace_buffer *buffer, int nr_pages)
*/
for (i = 0; i < meta->nr_subbufs; i++) {
meta->buffers[i] = i;
- rb_init_page(subbuf);
+ rb_init_data_page(subbuf);
subbuf += meta->subbuf_size;
}
}
@@ -2187,7 +2202,7 @@ static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
if (!page)
goto free_pages;
bpage->page = page_address(page);
- rb_init_page(bpage->page);
+ rb_init_data_page(bpage->page);
}
bpage->order = cpu_buffer->buffer->subbuf_order;
@@ -2293,7 +2308,7 @@ rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)
if (!page)
goto fail_free_reader;
bpage->page = page_address(page);
- rb_init_page(bpage->page);
+ rb_init_data_page(bpage->page);
}
INIT_LIST_HEAD(&cpu_buffer->reader_page->list);
@@ -2361,7 +2376,7 @@ static void rb_free_cpu_buffer(struct ring_buffer_per_cpu *cpu_buffer)
free_buffer_page(bpage);
}
- free_pages((unsigned long)cpu_buffer->free_page, cpu_buffer->buffer->subbuf_order);
+ free_pages((unsigned long)cpu_buffer->free_page.data, cpu_buffer->free_page.order);
kfree(cpu_buffer);
}
@@ -6008,7 +6023,7 @@ static void rb_clear_buffer_page(struct buffer_page *page)
{
local_set(&page->write, 0);
local_set(&page->entries, 0);
- rb_init_page(page->page);
+ rb_init_data_page(page->page);
page->read = 0;
}
@@ -6394,7 +6409,6 @@ ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)
struct ring_buffer_per_cpu *cpu_buffer;
struct buffer_data_read_page *bpage = NULL;
unsigned long flags;
- struct page *page;
if (!cpumask_test_cpu(cpu, buffer->cpumask))
return ERR_PTR(-ENODEV);
@@ -6408,30 +6422,24 @@ ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)
local_irq_save(flags);
arch_spin_lock(&cpu_buffer->lock);
- if (cpu_buffer->free_page) {
- bpage->data = cpu_buffer->free_page;
- cpu_buffer->free_page = NULL;
+ if (cpu_buffer->free_page.data) {
+ *bpage = cpu_buffer->free_page;
+ cpu_buffer->free_page.data = NULL;
}
arch_spin_unlock(&cpu_buffer->lock);
local_irq_restore(flags);
- if (bpage->data)
- goto out;
-
- page = alloc_pages_node(cpu_to_node(cpu),
- GFP_KERNEL | __GFP_NORETRY | __GFP_COMP | __GFP_ZERO,
- cpu_buffer->buffer->subbuf_order);
- if (!page) {
- kfree(bpage);
- return ERR_PTR(-ENOMEM);
+ if (bpage->data) {
+ rb_init_data_page(bpage->data);
+ } else {
+ bpage->data = alloc_cpu_data(cpu, bpage->order);
+ if (!bpage->data) {
+ kfree(bpage);
+ return ERR_PTR(-ENOMEM);
+ }
}
- bpage->data = page_address(page);
-
- out:
- rb_init_page(bpage->data);
-
return bpage;
}
EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);
@@ -6468,8 +6476,8 @@ void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
local_irq_save(flags);
arch_spin_lock(&cpu_buffer->lock);
- if (!cpu_buffer->free_page) {
- cpu_buffer->free_page = bpage;
+ if (!cpu_buffer->free_page.data) {
+ cpu_buffer->free_page = *data_page;
bpage = NULL;
}
@@ -6526,6 +6534,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
struct buffer_page *reader;
unsigned long missed_events;
unsigned int commit;
+ unsigned int size;
unsigned int read;
u64 save_timestamp;
@@ -6560,7 +6569,8 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
event = rb_reader_event(cpu_buffer);
read = reader->read;
- commit = rb_page_size(reader);
+ commit = rb_page_commit(reader);
+ size = rb_page_size(reader);
/* Check if any events were dropped */
missed_events = cpu_buffer->lost_events;
@@ -6572,34 +6582,31 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
* we must copy the data from the page to the buffer.
* Otherwise, we can simply swap the page with the one passed in.
*/
- if (read || (len < (commit - read)) ||
+ if (read || (len < (size - read)) ||
cpu_buffer->reader_page == cpu_buffer->commit_page ||
cpu_buffer->mapped) {
struct buffer_data_page *rpage = cpu_buffer->reader_page->page;
unsigned int rpos = read;
unsigned int pos = 0;
- unsigned int size;
+ unsigned int event_size;
+ unsigned int flags = 0;
- /*
- * If a full page is expected, this can still be returned
- * if there's been a previous partial read and the
- * rest of the page can be read and the commit page is off
- * the reader page.
- */
- if (full &&
- (!read || (len < (commit - read)) ||
- cpu_buffer->reader_page == cpu_buffer->commit_page))
+ /* If a full page is requested, it cannot be the commit page */
+ if (full && cpu_buffer->reader_page == cpu_buffer->commit_page)
return -1;
- if (len > (commit - read))
- len = (commit - read);
+ if (len > (size - read))
+ len = (size - read);
/* Always keep the time extend and data together */
- size = rb_event_ts_length(event);
+ event_size = rb_event_ts_length(event);
- if (len < size)
+ if (len < event_size)
return -1;
+ if (commit & RB_MISSED_EVENTS)
+ flags = RB_MISSED_EVENTS;
+
/* save the current timestamp, since the user will need it */
save_timestamp = cpu_buffer->read_stamp;
@@ -6611,25 +6618,25 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
* one or two events.
* We have already ensured there's enough space if this
* is a time extend. */
- size = rb_event_length(event);
- memcpy(bpage->data + pos, rpage->data + rpos, size);
+ event_size = rb_event_length(event);
+ memcpy(bpage->data + pos, rpage->data + rpos, event_size);
- len -= size;
+ len -= event_size;
rb_advance_reader(cpu_buffer);
rpos = reader->read;
- pos += size;
+ pos += event_size;
- if (rpos >= commit)
+ if (rpos >= size)
break;
event = rb_reader_event(cpu_buffer);
/* Always keep the time extend and data together */
- size = rb_event_ts_length(event);
- } while (len >= size);
+ event_size = rb_event_ts_length(event);
+ } while (len >= event_size);
/* update bpage */
- local_set(&bpage->commit, pos);
+ local_set(&bpage->commit, pos | flags);
bpage->time_stamp = save_timestamp;
/* we copied everything to the beginning */
@@ -6640,7 +6647,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
cpu_buffer->read_bytes += rb_page_size(reader);
/* swap the pages */
- rb_init_page(bpage);
+ rb_init_data_page(bpage);
bpage = reader->page;
reader->page = data_page->data;
local_set(&reader->write, 0);
@@ -6654,12 +6661,14 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
* on the page.
*/
if (reader->real_end)
- local_set(&bpage->commit, reader->real_end);
+ local_set(&bpage->commit, reader->real_end |
+ (commit & RB_MISSED_EVENTS));
}
cpu_buffer->lost_events = 0;
commit = local_read(&bpage->commit);
+ size = commit & ~RB_MISSED_MASK;
/*
* Set a flag in the commit field if we lost events
*/
@@ -6667,20 +6676,20 @@ int ring_buffer_read_page(struct trace_buffer *buffer,
/* If there is room at the end of the page to save the
* missed events, then record it there.
*/
- if (rb_page_capacity(reader) - commit >= sizeof(missed_events)) {
- memcpy(&bpage->data[commit], &missed_events,
+ if (rb_page_capacity(reader) - size >= sizeof(missed_events)) {
+ memcpy(&bpage->data[size], &missed_events,
sizeof(missed_events));
- local_add(RB_MISSED_STORED, &bpage->commit);
- commit += sizeof(missed_events);
+ commit |= RB_MISSED_STORED;
+ size += sizeof(missed_events);
}
- local_add(RB_MISSED_EVENTS, &bpage->commit);
+ local_set(&bpage->commit, commit | RB_MISSED_EVENTS);
}
/*
* This page may be off to user land. Zero it out here.
*/
- if (commit < rb_page_capacity(reader))
- memset(&bpage->data[commit], 0, rb_page_capacity(reader) - commit);
+ if (size < rb_page_capacity(reader))
+ memset(&bpage->data[size], 0, rb_page_capacity(reader) - size);
return read;
}
@@ -6830,7 +6839,7 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
}
for_each_buffer_cpu(buffer, cpu) {
- struct buffer_data_page *old_free_data_page;
+ struct buffer_data_read_page old_free_data_page;
struct list_head old_pages;
unsigned long flags;
@@ -6873,7 +6882,7 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
arch_spin_lock(&cpu_buffer->lock);
old_free_data_page = cpu_buffer->free_page;
- cpu_buffer->free_page = NULL;
+ cpu_buffer->free_page.data = NULL;
arch_spin_unlock(&cpu_buffer->lock);
rb_head_page_activate(cpu_buffer);
@@ -6885,7 +6894,7 @@ int ring_buffer_subbuf_order_set(struct trace_buffer *buffer, int order)
list_del_init(&bpage->list);
free_buffer_page(bpage);
}
- free_pages((unsigned long)old_free_data_page, old_order);
+ free_pages((unsigned long)old_free_data_page.data, old_free_data_page.order);
rb_check_pages(cpu_buffer);
}
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 522b9e11d0ba..ed1014292c08 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -1830,10 +1830,7 @@ static ssize_t trace_seq_to_buffer(struct trace_seq *s, void *buf, size_t cnt)
unsigned long __read_mostly tracing_thresh;
#ifdef CONFIG_TRACER_MAX_TRACE
-static const struct file_operations tracing_max_lat_fops;
-
#ifdef LATENCY_FS_NOTIFY
-
static struct workqueue_struct *fsnotify_wq;
static void latency_fsnotify_workfn(struct work_struct *work)
@@ -1850,17 +1847,6 @@ static void latency_fsnotify_workfn_irq(struct irq_work *iwork)
queue_work(fsnotify_wq, &tr->fsnotify_work);
}
-static void trace_create_maxlat_file(struct trace_array *tr,
- struct dentry *d_tracer)
-{
- INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn);
- init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq);
- tr->d_max_latency = trace_create_file("tracing_max_latency",
- TRACE_MODE_WRITE,
- d_tracer, tr,
- &tracing_max_lat_fops);
-}
-
__init static int latency_fsnotify_init(void)
{
fsnotify_wq = alloc_workqueue("tr_max_lat_wq",
@@ -1885,14 +1871,22 @@ void latency_fsnotify(struct trace_array *tr)
*/
irq_work_queue(&tr->fsnotify_irqwork);
}
+#endif /* !LATENCY_FS_NOTIFY */
-#else /* !LATENCY_FS_NOTIFY */
-
-#define trace_create_maxlat_file(tr, d_tracer) \
- trace_create_file("tracing_max_latency", TRACE_MODE_WRITE, \
- d_tracer, tr, &tracing_max_lat_fops)
+static const struct file_operations tracing_max_lat_fops;
+static void trace_create_maxlat_file(struct trace_array *tr,
+ struct dentry *d_tracer)
+{
+#ifdef LATENCY_FS_NOTIFY
+ INIT_WORK(&tr->fsnotify_work, latency_fsnotify_workfn);
+ init_irq_work(&tr->fsnotify_irqwork, latency_fsnotify_workfn_irq);
#endif
+ tr->d_max_latency = trace_create_file("tracing_max_latency",
+ TRACE_MODE_WRITE,
+ d_tracer, tr,
+ &tracing_max_lat_fops);
+}
/*
* Copy the new maximum trace into the separate maximum-trace
@@ -2027,7 +2021,9 @@ update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu)
__update_max_tr(tr, tsk, cpu);
arch_spin_unlock(&tr->max_lock);
}
-
+#else /* !CONFIG_TRACER_MAX_TRACE */
+static inline void trace_create_maxlat_file(struct trace_array *tr,
+ struct dentry *d_tracer) { }
#endif /* CONFIG_TRACER_MAX_TRACE */
struct pipe_wait {
@@ -8741,12 +8737,57 @@ trace_options_write(struct file *filp, const char __user *ubuf, size_t cnt,
return cnt;
}
+static bool tr_option_match(struct trace_array *tr, void *topt)
+{
+ for (int i = 0; i < tr->nr_topts; i++) {
+ struct trace_options *tr_topts = &tr->topts[i];
+
+ if (topt >= (void *)&tr_topts->topts[0] &&
+ topt < (void *)&tr_topts->topts[tr_topts->nr_topts])
+ return true;
+ }
+ return false;
+}
+
+/*
+ * The topt is the address of a trace_array->topts[] element that holds the
+ * the tracer options descriptor. But since the trace_array reference has not
+ * been taken yet, it cannot be dereferenced as it could have been freed by
+ * a rmdir of the instance the trace_array represents.
+ *
+ * Search the list of trace_arrays and compare the topt to the address of
+ * the entire trace_array topts array for each trace_array in the list.
+ * If one is matched, then take the reference and return it. If not, the
+ * trace_array no longer exits.
+ */
+static int trace_array_tracer_options_get(void *topt)
+{
+ struct trace_array *tr;
+ int ret;
+
+ ret = security_locked_down(LOCKDOWN_TRACEFS);
+ if (ret)
+ return ret;
+
+ if (tracing_disabled)
+ return -ENODEV;
+
+ guard(mutex)(&trace_types_lock);
+ list_for_each_entry(tr, &ftrace_trace_arrays, list) {
+ if (tr_option_match(tr, topt)) {
+ tr->ref++;
+ return 0;
+ }
+ }
+ return -ENODEV;
+}
+
static int tracing_open_options(struct inode *inode, struct file *filp)
{
struct trace_option_dentry *topt = inode->i_private;
int ret;
- ret = tracing_check_open_get_tr(topt->tr);
+ ret = trace_array_tracer_options_get(topt);
if (ret)
return ret;
@@ -8855,11 +8896,72 @@ trace_options_core_write(struct file *filp, const char __user *ubuf, size_t cnt,
return cnt;
}
+/*
+ * The tr_index is the address of a trace_array->trace_flags_index[]
+ * element that holds the index of the trace flag. But since the
+ * trace_array reference has not been taken yet, it cannot be referenced
+ * as it could have been freed by a rmdir of the instance the trace_array
+ * represents.
+ *
+ * Search the list of trace_arrays and compare the tr_index to the
+ * address of the entire trace_array trace_flags_index array for each
+ * trace_array in the list. If one is matched, then take the reference
+ * and return it. If not, the trace_array no longer exits.
+ */
+static int trace_array_options_get(void *tr_index)
+{
+ struct trace_array *tr;
+ int ret;
+
+ ret = security_locked_down(LOCKDOWN_TRACEFS);
+ if (ret)
+ return ret;
+
+ if (tracing_disabled)
+ return -ENODEV;
+
+ guard(mutex)(&trace_types_lock);
+ list_for_each_entry(tr, &ftrace_trace_arrays, list) {
+ if (tr_index >= (void *)&tr->trace_flags_index[0] &&
+ tr_index < (void *)&tr->trace_flags_index[TRACE_FLAGS_MAX_SIZE]) {
+ tr->ref++;
+ return 0;
+ }
+ }
+ return -ENODEV;
+}
+
+static int trace_options_open(struct inode *inode, struct file *filp)
+{
+ void *tr_index = inode->i_private;
+
+ if (trace_array_options_get(tr_index) < 0)
+ return -ENODEV;
+
+ filp->private_data = tr_index;
+
+ return 0;
+}
+
+static int trace_options_release(struct inode *inode, struct file *filp)
+{
+ void *tr_index = filp->private_data;
+ struct trace_array *tr;
+ unsigned int index;
+
+ get_tr_index(tr_index, &tr, &index);
+
+ trace_array_put(tr);
+
+ return 0;
+}
+
static const struct file_operations trace_options_core_fops = {
- .open = tracing_open_generic,
- .read = trace_options_core_read,
- .write = trace_options_core_write,
- .llseek = generic_file_llseek,
+ .open = trace_options_open,
+ .read = trace_options_core_read,
+ .write = trace_options_core_write,
+ .llseek = generic_file_llseek,
+ .release = trace_options_release,
};
struct dentry *trace_create_file(const char *name,
@@ -8969,6 +9071,7 @@ create_trace_option_files(struct trace_array *tr, struct tracer *tracer)
tr->topts = tr_topts;
tr->topts[tr->nr_topts].tracer = tracer;
tr->topts[tr->nr_topts].topts = topts;
+ tr->topts[tr->nr_topts].nr_topts = cnt;
tr->nr_topts++;
for (cnt = 0; opts[cnt].name; cnt++) {
@@ -9720,9 +9823,7 @@ init_tracer_tracefs(struct trace_array *tr, struct dentry *d_tracer)
create_trace_options_dir(tr);
-#ifdef CONFIG_TRACER_MAX_TRACE
trace_create_maxlat_file(tr, d_tracer);
-#endif
if (ftrace_create_function_files(tr, d_tracer))
MEM_FAIL(1, "Could not allocate function filter files");
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 176e493f79db..1f59dd8940e9 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -223,6 +223,7 @@ struct array_buffer {
struct trace_options {
struct tracer *tracer;
struct trace_option_dentry *topts;
+ int nr_topts;
};
struct trace_pid_list *trace_pid_list_alloc(void);
@@ -340,8 +341,8 @@ struct trace_array {
spinlock_t snapshot_trigger_lock;
unsigned int snapshot;
unsigned long max_latency;
-#ifdef CONFIG_FSNOTIFY
struct dentry *d_max_latency;
+#ifdef CONFIG_FSNOTIFY
struct work_struct fsnotify_work;
struct irq_work fsnotify_irqwork;
#endif
@@ -1191,7 +1192,7 @@ extern void clear_ftrace_function_probes(struct trace_array *tr);
int register_ftrace_command(struct ftrace_func_command *cmd);
int unregister_ftrace_command(struct ftrace_func_command *cmd);
-void ftrace_create_filter_files(struct ftrace_ops *ops,
+void ftrace_create_filter_files(struct trace_array *tr,
struct dentry *parent);
void ftrace_destroy_filter_files(struct ftrace_ops *ops);
@@ -1214,11 +1215,11 @@ static inline void clear_ftrace_function_probes(struct trace_array *tr)
{
}
+#define ftrace_create_filter_files(tr, parent) do { } while (0)
/*
* The ops parameter passed in is usually undefined.
* This must be a macro.
*/
-#define ftrace_create_filter_files(ops, parent) do { } while (0)
#define ftrace_destroy_filter_files(ops) do { } while (0)
#endif /* CONFIG_FUNCTION_TRACER && CONFIG_DYNAMIC_FTRACE */
@@ -1692,13 +1693,13 @@ extern void clear_event_triggers(struct trace_array *tr);
enum {
EVENT_TRIGGER_FL_PROBE = BIT(0),
+ EVENT_TRIGGER_FL_COUNT = BIT(1),
};
struct event_trigger_data {
unsigned long count;
int ref;
int flags;
- struct event_trigger_ops *ops;
struct event_command *cmd_ops;
struct event_filter __rcu *filter;
char *filter_str;
@@ -1723,6 +1724,10 @@ struct enable_trigger_data {
bool hist;
};
+bool event_trigger_count(struct event_trigger_data *data,
+ struct trace_buffer *buffer, void *rec,
+ struct ring_buffer_event *event);
+
extern int event_enable_trigger_print(struct seq_file *m,
struct event_trigger_data *data);
extern void event_enable_trigger_free(struct event_trigger_data *data);
@@ -1785,64 +1790,6 @@ extern void event_trigger_unregister(struct event_command *cmd_ops,
extern void event_file_get(struct trace_event_file *file);
extern void event_file_put(struct trace_event_file *file);
-/**
- * struct event_trigger_ops - callbacks for trace event triggers
- *
- * The methods in this structure provide per-event trigger hooks for
- * various trigger operations.
- *
- * The @init and @free methods are used during trigger setup and
- * teardown, typically called from an event_command's @parse()
- * function implementation.
- *
- * The @print method is used to print the trigger spec.
- *
- * The @trigger method is the function that actually implements the
- * trigger and is called in the context of the triggering event
- * whenever that event occurs.
- *
- * All the methods below, except for @init() and @free(), must be
- * implemented.
- *
- * @trigger: The trigger 'probe' function called when the triggering
- * event occurs. The data passed into this callback is the data
- * that was supplied to the event_command @reg() function that
- * registered the trigger (see struct event_command) along with
- * the trace record, rec.
- *
- * @init: An optional initialization function called for the trigger
- * when the trigger is registered (via the event_command reg()
- * function). This can be used to perform per-trigger
- * initialization such as incrementing a per-trigger reference
- * count, for instance. This is usually implemented by the
- * generic utility function @event_trigger_init() (see
- * trace_event_triggers.c).
- *
- * @free: An optional de-initialization function called for the
- * trigger when the trigger is unregistered (via the
- * event_command @reg() function). This can be used to perform
- * per-trigger de-initialization such as decrementing a
- * per-trigger reference count and freeing corresponding trigger
- * data, for instance. This is usually implemented by the
- * generic utility function @event_trigger_free() (see
- * trace_event_triggers.c).
- *
- * @print: The callback function invoked to have the trigger print
- * itself. This is usually implemented by a wrapper function
- * that calls the generic utility function @event_trigger_print()
- * (see trace_event_triggers.c).
- */
-struct event_trigger_ops {
- void (*trigger)(struct event_trigger_data *data,
- struct trace_buffer *buffer,
- void *rec,
- struct ring_buffer_event *rbe);
- int (*init)(struct event_trigger_data *data);
- void (*free)(struct event_trigger_data *data);
- int (*print)(struct seq_file *m,
- struct event_trigger_data *data);
-};
-
/**
* struct event_command - callbacks and data members for event commands
*
@@ -1892,7 +1839,7 @@ struct event_trigger_ops {
*
* @reg: Adds the trigger to the list of triggers associated with the
* event, and enables the event trigger itself, after
- * initializing it (via the event_trigger_ops @init() function).
+ * initializing it (via the event_command @init() function).
* This is also where commands can use the @trigger_type value to
* make the decision as to whether or not multiple instances of
* the trigger should be allowed. This is usually implemented by
@@ -1901,7 +1848,7 @@ struct event_trigger_ops {
*
* @unreg: Removes the trigger from the list of triggers associated
* with the event, and disables the event trigger itself, after
- * initializing it (via the event_trigger_ops @free() function).
+ * initializing it (via the event_command @free() function).
* This is usually implemented by the generic utility function
* @unregister_trigger() (see trace_event_triggers.c).
*
@@ -1915,12 +1862,41 @@ struct event_trigger_ops {
* ignored. This is usually implemented by the generic utility
* function @set_trigger_filter() (see trace_event_triggers.c).
*
- * @get_trigger_ops: The callback function invoked to retrieve the
- * event_trigger_ops implementation associated with the command.
- * This callback function allows a single event_command to
- * support multiple trigger implementations via different sets of
- * event_trigger_ops, depending on the value of the @param
- * string.
+ * All the methods below, except for @init() and @free(), must be
+ * implemented.
+ *
+ * @trigger: The trigger 'probe' function called when the triggering
+ * event occurs. The data passed into this callback is the data
+ * that was supplied to the event_command @reg() function that
+ * registered the trigger (see struct event_command) along with
+ * the trace record, rec.
+ *
+ * @count_func: If defined and a numeric parameter is passed to the
+ * trigger, then this function will be called before @trigger
+ * is called. If this function returns false, then @trigger is not
+ * executed.
+ *
+ * @init: An optional initialization function called for the trigger
+ * when the trigger is registered (via the event_command reg()
+ * function). This can be used to perform per-trigger
+ * initialization such as incrementing a per-trigger reference
+ * count, for instance. This is usually implemented by the
+ * generic utility function @event_trigger_init() (see
+ * trace_event_triggers.c).
+ *
+ * @free: An optional de-initialization function called for the
+ * trigger when the trigger is unregistered (via the
+ * event_command @reg() function). This can be used to perform
+ * per-trigger de-initialization such as decrementing a
+ * per-trigger reference count and freeing corresponding trigger
+ * data, for instance. This is usually implemented by the
+ * generic utility function @event_trigger_free() (see
+ * trace_event_triggers.c).
+ *
+ * @print: The callback function invoked to have the trigger print
+ * itself. This is usually implemented by a wrapper function
+ * that calls the generic utility function @event_trigger_print()
+ * (see trace_event_triggers.c).
*/
struct event_command {
struct list_head list;
@@ -1941,7 +1917,18 @@ struct event_command {
int (*set_filter)(char *filter_str,
struct event_trigger_data *data,
struct trace_event_file *file);
- struct event_trigger_ops *(*get_trigger_ops)(char *cmd, char *param);
+ void (*trigger)(struct event_trigger_data *data,
+ struct trace_buffer *buffer,
+ void *rec,
+ struct ring_buffer_event *rbe);
+ bool (*count_func)(struct event_trigger_data *data,
+ struct trace_buffer *buffer,
+ void *rec,
+ struct ring_buffer_event *rbe);
+ int (*init)(struct event_trigger_data *data);
+ void (*free)(struct event_trigger_data *data);
+ int (*print)(struct seq_file *m,
+ struct event_trigger_data *data);
};
/**
@@ -1962,7 +1949,7 @@ struct event_command {
* either committed or discarded. At that point, if any commands
* have deferred their triggers, those commands are finally
* invoked following the close of the current event. In other
- * words, if the event_trigger_ops @func() probe implementation
+ * words, if the event_command @func() probe implementation
* itself logs to the trace buffer, this flag should be set,
* otherwise it can be left unspecified.
*
diff --git a/kernel/trace/trace_btf.c b/kernel/trace/trace_btf.c
index 5bbdbcbbde3c..43e379801177 100644
--- a/kernel/trace/trace_btf.c
+++ b/kernel/trace/trace_btf.c
@@ -61,47 +61,50 @@ struct btf_anon_stack {
/*
* Find a member of data structure/union by name and return it.
- * Return NULL if not found, or -EINVAL if parameter is invalid.
- * If the member is an member of anonymous union/structure, the offset
- * of that anonymous union/structure is stored into @anon_offset. Caller
- * can calculate the correct offset from the root data structure by
- * adding anon_offset to the member's offset.
+ * Return NULL if not found, or ERR_PTR(-EINVAL) if parameter is invalid.
+ * If the member is a member of an anonymous union/structure, the bit offset
+ * of that anonymous union/structure is stored into @anon_offset.
+ * If @member_type is non-NULL, the actual containing structure/union type
+ * of the found member is stored into @member_type.
*/
const struct btf_member *btf_find_struct_member(struct btf *btf,
const struct btf_type *type,
const char *member_name,
- u32 *anon_offset)
+ u32 *anon_offset,
+ const struct btf_type **member_type)
{
struct btf_anon_stack *anon_stack;
const struct btf_member *member;
+ const struct btf_type *mtype;
u32 tid, cur_offset = 0;
const char *name;
int i, top = 0;
+ if (!btf_type_is_struct(type))
+ return ERR_PTR(-EINVAL);
+
anon_stack = kcalloc(BTF_ANON_STACK_MAX, sizeof(*anon_stack), GFP_KERNEL);
if (!anon_stack)
return ERR_PTR(-ENOMEM);
retry:
- if (!btf_type_is_struct(type)) {
- member = ERR_PTR(-EINVAL);
- goto out;
- }
-
for_each_member(i, type, member) {
if (!member->name_off) {
/* Anonymous union/struct: push it for later use */
- if (btf_type_skip_modifiers(btf, member->type, &tid) &&
+ mtype = btf_type_skip_modifiers(btf, member->type, &tid);
+ if (mtype && btf_type_is_struct(mtype) &&
top < BTF_ANON_STACK_MAX) {
anon_stack[top].tid = tid;
- anon_stack[top++].offset =
- cur_offset + member->offset;
+ anon_stack[top++].offset = cur_offset +
+ __btf_member_bit_offset(type, member);
}
} else {
name = btf_name_by_offset(btf, member->name_off);
if (name && !strcmp(member_name, name)) {
if (anon_offset)
*anon_offset = cur_offset;
+ if (member_type)
+ *member_type = type;
goto out;
}
}
diff --git a/kernel/trace/trace_btf.h b/kernel/trace/trace_btf.h
index 4bc44bc261e6..4bd26bceae23 100644
--- a/kernel/trace/trace_btf.h
+++ b/kernel/trace/trace_btf.h
@@ -8,4 +8,5 @@ const struct btf_param *btf_get_func_param(const struct btf_type *func_proto,
const struct btf_member *btf_find_struct_member(struct btf *btf,
const struct btf_type *type,
const char *member_name,
- u32 *anon_offset);
+ u32 *anon_offset,
+ const struct btf_type **member_type);
diff --git a/kernel/trace/trace_eprobe.c b/kernel/trace/trace_eprobe.c
index e3dd6551c468..88bf2ec8b0c7 100644
--- a/kernel/trace/trace_eprobe.c
+++ b/kernel/trace/trace_eprobe.c
@@ -479,13 +479,6 @@ static void eprobe_trigger_func(struct event_trigger_data *data,
__eprobe_trace_func(edata, rec);
}
-static struct event_trigger_ops eprobe_trigger_ops = {
- .trigger = eprobe_trigger_func,
- .print = eprobe_trigger_print,
- .init = eprobe_trigger_init,
- .free = eprobe_trigger_free,
-};
-
static int eprobe_trigger_cmd_parse(struct event_command *cmd_ops,
struct trace_event_file *file,
char *glob, char *cmd,
@@ -508,12 +501,6 @@ static void eprobe_trigger_unreg_func(char *glob,
}
-static struct event_trigger_ops *eprobe_trigger_get_ops(char *cmd,
- char *param)
-{
- return &eprobe_trigger_ops;
-}
-
static struct event_command event_trigger_cmd = {
.name = "eprobe",
.trigger_type = ETT_EVENT_EPROBE,
@@ -522,8 +509,11 @@ static struct event_command event_trigger_cmd = {
.reg = eprobe_trigger_reg_func,
.unreg = eprobe_trigger_unreg_func,
.unreg_all = NULL,
- .get_trigger_ops = eprobe_trigger_get_ops,
.set_filter = NULL,
+ .trigger = eprobe_trigger_func,
+ .print = eprobe_trigger_print,
+ .init = eprobe_trigger_init,
+ .free = eprobe_trigger_free,
};
static struct event_trigger_data *
@@ -543,7 +533,6 @@ new_eprobe_trigger(struct trace_eprobe *ep, struct trace_event_file *file)
trigger->flags = EVENT_TRIGGER_FL_PROBE;
trigger->count = -1;
- trigger->ops = &eprobe_trigger_ops;
/*
* EVENT PROBE triggers are not registered as commands with
diff --git a/kernel/trace/trace_events_hist.c b/kernel/trace/trace_events_hist.c
index 75490d088467..59f9b54a6d0a 100644
--- a/kernel/trace/trace_events_hist.c
+++ b/kernel/trace/trace_events_hist.c
@@ -2379,6 +2379,7 @@ parse_field(struct hist_trigger_data *hist_data, struct trace_event_file *file,
*flags |= HIST_FIELD_FL_CPU;
} else if (field && field->filter_type == FILTER_STACKTRACE) {
*flags |= HIST_FIELD_FL_STACKTRACE;
+ field = NULL;
} else {
hist_err(tr, HIST_ERR_FIELD_NOT_FOUND,
errpos(field_name));
@@ -5692,7 +5693,7 @@ static void hist_trigger_show(struct seq_file *m,
seq_puts(m, "\n\n");
seq_puts(m, "# event histogram\n#\n# trigger info: ");
- data->ops->print(m, data);
+ data->cmd_ops->print(m, data);
seq_puts(m, "#\n\n");
hist_data = data->private_data;
@@ -6014,7 +6015,7 @@ static void hist_trigger_debug_show(struct seq_file *m,
seq_puts(m, "\n\n");
seq_puts(m, "# event histogram\n#\n# trigger info: ");
- data->ops->print(m, data);
+ data->cmd_ops->print(m, data);
seq_puts(m, "#\n\n");
hist_data = data->private_data;
@@ -6322,20 +6323,22 @@ static void event_hist_trigger_free(struct event_trigger_data *data)
free_hist_pad();
}
-static struct event_trigger_ops event_hist_trigger_ops = {
- .trigger = event_hist_trigger,
- .print = event_hist_trigger_print,
- .init = event_hist_trigger_init,
- .free = event_hist_trigger_free,
-};
-
static int event_hist_trigger_named_init(struct event_trigger_data *data)
{
+ int ret;
+
+ ret = event_hist_trigger_init(data->named_data);
+ if (ret < 0) {
+ kfree(data->cmd_ops);
+ data->cmd_ops = &trigger_hist_cmd;
+ return ret;
+ }
+
data->ref++;
save_named_trigger(data->named_data->name, data);
- return event_hist_trigger_init(data->named_data);
+ return 0;
}
static void event_hist_trigger_named_free(struct event_trigger_data *data)
@@ -6347,24 +6350,14 @@ static void event_hist_trigger_named_free(struct event_trigger_data *data)
data->ref--;
if (!data->ref) {
+ struct event_command *cmd_ops = data->cmd_ops;
+
del_named_trigger(data);
trigger_data_free(data);
+ kfree(cmd_ops);
}
}
-static struct event_trigger_ops event_hist_trigger_named_ops = {
- .trigger = event_hist_trigger,
- .print = event_hist_trigger_print,
- .init = event_hist_trigger_named_init,
- .free = event_hist_trigger_named_free,
-};
-
-static struct event_trigger_ops *event_hist_get_trigger_ops(char *cmd,
- char *param)
-{
- return &event_hist_trigger_ops;
-}
-
static void hist_clear(struct event_trigger_data *data)
{
struct hist_trigger_data *hist_data = data->private_data;
@@ -6558,15 +6551,20 @@ static int hist_register_trigger(char *glob,
data->paused = true;
if (named_data) {
+ struct event_command *cmd_ops;
+
data->private_data = named_data->private_data;
set_named_trigger_data(data, named_data);
- data->ops = &event_hist_trigger_named_ops;
- }
-
- if (data->ops->init) {
- ret = data->ops->init(data);
- if (ret < 0)
+ /* Copy the command ops and update some of the functions */
+ cmd_ops = kmalloc(sizeof(*cmd_ops), GFP_KERNEL);
+ if (!cmd_ops) {
+ ret = -ENOMEM;
goto out;
+ }
+ *cmd_ops = *data->cmd_ops;
+ cmd_ops->init = event_hist_trigger_named_init;
+ cmd_ops->free = event_hist_trigger_named_free;
+ data->cmd_ops = cmd_ops;
}
if (hist_data->enable_timestamps) {
@@ -6581,6 +6579,15 @@ static int hist_register_trigger(char *glob,
tracing_set_filter_buffering(file->tr, true);
}
+ if (data->cmd_ops->init) {
+ ret = data->cmd_ops->init(data);
+ if (ret < 0) {
+ if (hist_data->enable_timestamps)
+ tracing_set_filter_buffering(file->tr, false);
+ goto out;
+ }
+ }
+
if (named_data) {
remove_hist_vars(hist_data);
destroy_hist_data(hist_data);
@@ -6598,11 +6605,12 @@ static int hist_trigger_enable(struct event_trigger_data *data,
update_cond_flag(file);
- if (trace_event_trigger_enable_disable(file, 1) < 0) {
- list_del_rcu(&data->list);
- update_cond_flag(file);
+ /*
+ * On failure the caller undoes the registration, and
+ * hist_unregister_trigger() can only find the trigger here.
+ */
+ if (trace_event_trigger_enable_disable(file, 1) < 0)
ret--;
- }
return ret;
}
@@ -6680,13 +6688,13 @@ static void hist_unregister_trigger(char *glob,
}
}
- if (test && test->ops->free)
- test->ops->free(test);
-
if (hist_data->enable_timestamps) {
if (!hist_data->remove || test)
tracing_set_filter_buffering(file->tr, false);
}
+
+ if (test && test->cmd_ops->free)
+ test->cmd_ops->free(test);
}
static bool hist_file_check_refs(struct trace_event_file *file)
@@ -6733,8 +6741,8 @@ static void hist_unreg_all(struct trace_event_file *file)
update_cond_flag(file);
if (hist_data->enable_timestamps)
tracing_set_filter_buffering(file->tr, false);
- if (test->ops->free)
- test->ops->free(test);
+ if (test->cmd_ops->free)
+ test->cmd_ops->free(test);
}
}
}
@@ -6891,6 +6899,8 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops,
return ret;
out_unreg:
event_trigger_unregister(cmd_ops, file, glob+1, trigger_data);
+ /* The unregister frees trigger_data, skip out_free */
+ goto out;
out_free:
event_trigger_reset_filter(cmd_ops, trigger_data);
@@ -6910,8 +6920,11 @@ static struct event_command trigger_hist_cmd = {
.reg = hist_register_trigger,
.unreg = hist_unregister_trigger,
.unreg_all = hist_unreg_all,
- .get_trigger_ops = event_hist_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = event_hist_trigger,
+ .print = event_hist_trigger_print,
+ .init = event_hist_trigger_init,
+ .free = event_hist_trigger_free,
};
__init int register_trigger_hist_cmd(void)
@@ -6943,66 +6956,6 @@ hist_enable_trigger(struct event_trigger_data *data,
}
}
-static void
-hist_enable_count_trigger(struct event_trigger_data *data,
- struct trace_buffer *buffer, void *rec,
- struct ring_buffer_event *event)
-{
- if (!data->count)
- return;
-
- if (data->count != -1)
- (data->count)--;
-
- hist_enable_trigger(data, buffer, rec, event);
-}
-
-static struct event_trigger_ops hist_enable_trigger_ops = {
- .trigger = hist_enable_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
-static struct event_trigger_ops hist_enable_count_trigger_ops = {
- .trigger = hist_enable_count_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
-static struct event_trigger_ops hist_disable_trigger_ops = {
- .trigger = hist_enable_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
-static struct event_trigger_ops hist_disable_count_trigger_ops = {
- .trigger = hist_enable_count_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
-static struct event_trigger_ops *
-hist_enable_get_trigger_ops(char *cmd, char *param)
-{
- struct event_trigger_ops *ops;
- bool enable;
-
- enable = (strcmp(cmd, ENABLE_HIST_STR) == 0);
-
- if (enable)
- ops = param ? &hist_enable_count_trigger_ops :
- &hist_enable_trigger_ops;
- else
- ops = param ? &hist_disable_count_trigger_ops :
- &hist_disable_trigger_ops;
-
- return ops;
-}
-
static void hist_enable_unreg_all(struct trace_event_file *file)
{
struct event_trigger_data *test, *n;
@@ -7012,8 +6965,8 @@ static void hist_enable_unreg_all(struct trace_event_file *file)
list_del_rcu(&test->list);
update_cond_flag(file);
trace_event_trigger_enable_disable(file, 0);
- if (test->ops->free)
- test->ops->free(test);
+ if (test->cmd_ops->free)
+ test->cmd_ops->free(test);
}
}
}
@@ -7025,8 +6978,12 @@ static struct event_command trigger_hist_enable_cmd = {
.reg = event_enable_register_trigger,
.unreg = event_enable_unregister_trigger,
.unreg_all = hist_enable_unreg_all,
- .get_trigger_ops = hist_enable_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = hist_enable_trigger,
+ .count_func = event_trigger_count,
+ .print = event_enable_trigger_print,
+ .init = event_trigger_init,
+ .free = event_enable_trigger_free,
};
static struct event_command trigger_hist_disable_cmd = {
@@ -7036,8 +6993,12 @@ static struct event_command trigger_hist_disable_cmd = {
.reg = event_enable_register_trigger,
.unreg = event_enable_unregister_trigger,
.unreg_all = hist_enable_unreg_all,
- .get_trigger_ops = hist_enable_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = hist_enable_trigger,
+ .count_func = event_trigger_count,
+ .print = event_enable_trigger_print,
+ .init = event_trigger_init,
+ .free = event_enable_trigger_free,
};
static __init void unregister_trigger_hist_enable_disable_cmds(void)
diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
index 1e4e699c2547..4ee3d602a22e 100644
--- a/kernel/trace/trace_events_trigger.c
+++ b/kernel/trace/trace_events_trigger.c
@@ -31,6 +31,20 @@ void trigger_data_free(struct event_trigger_data *data)
kfree(data);
}
+static inline void data_ops_trigger(struct event_trigger_data *data,
+ struct trace_buffer *buffer, void *rec,
+ struct ring_buffer_event *event)
+{
+ const struct event_command *cmd_ops = data->cmd_ops;
+
+ if (data->flags & EVENT_TRIGGER_FL_COUNT) {
+ if (!cmd_ops->count_func(data, buffer, rec, event))
+ return;
+ }
+
+ cmd_ops->trigger(data, buffer, rec, event);
+}
+
/**
* event_triggers_call - Call triggers associated with a trace event
* @file: The trace_event_file associated with the event
@@ -73,7 +87,7 @@ event_triggers_call(struct trace_event_file *file,
if (data->paused)
continue;
if (!rec) {
- data->ops->trigger(data, buffer, rec, event);
+ data_ops_trigger(data, buffer, rec, event);
continue;
}
filter = rcu_dereference_sched(data->filter);
@@ -83,7 +97,7 @@ event_triggers_call(struct trace_event_file *file,
tt |= data->cmd_ops->trigger_type;
continue;
}
- data->ops->trigger(data, buffer, rec, event);
+ data_ops_trigger(data, buffer, rec, event);
}
return tt;
}
@@ -125,7 +139,7 @@ event_triggers_post_call(struct trace_event_file *file,
if (data->paused)
continue;
if (data->cmd_ops->trigger_type & tt)
- data->ops->trigger(data, NULL, NULL, NULL);
+ data_ops_trigger(data, NULL, NULL, NULL);
}
}
EXPORT_SYMBOL_GPL(event_triggers_post_call);
@@ -194,7 +208,7 @@ static int trigger_show(struct seq_file *m, void *v)
}
data = list_entry(v, struct event_trigger_data, list);
- data->ops->print(m, data);
+ data->cmd_ops->print(m, data);
return 0;
}
@@ -402,7 +416,37 @@ __init int unregister_event_command(struct event_command *cmd)
}
/**
- * event_trigger_print - Generic event_trigger_ops @print implementation
+ * event_trigger_count - Optional count function for event triggers
+ * @data: Trigger-specific data
+ * @buffer: The ring buffer that the event is being written to
+ * @rec: The trace entry for the event, NULL for unconditional invocation
+ * @event: The event meta data in the ring buffer
+ *
+ * For triggers that can take a count parameter that doesn't do anything
+ * special, they can use this function to assign to their .count_func
+ * field.
+ *
+ * This simply does a count down of the @data->count field.
+ *
+ * If the @data->count is greater than zero, it will decrement it.
+ *
+ * Returns false if @data->count is zero, otherwise true.
+ */
+bool event_trigger_count(struct event_trigger_data *data,
+ struct trace_buffer *buffer, void *rec,
+ struct ring_buffer_event *event)
+{
+ if (!data->count)
+ return false;
+
+ if (data->count != -1)
+ (data->count)--;
+
+ return true;
+}
+
+/**
+ * event_trigger_print - Generic event_command @print implementation
* @name: The name of the event trigger
* @m: The seq_file being printed to
* @data: Trigger-specific data
@@ -437,7 +481,7 @@ event_trigger_print(const char *name, struct seq_file *m,
}
/**
- * event_trigger_init - Generic event_trigger_ops @init implementation
+ * event_trigger_init - Generic event_command @init implementation
* @data: Trigger-specific data
*
* Common implementation of event trigger initialization.
@@ -454,7 +498,7 @@ int event_trigger_init(struct event_trigger_data *data)
}
/**
- * event_trigger_free - Generic event_trigger_ops @free implementation
+ * event_trigger_free - Generic event_command @free implementation
* @data: Trigger-specific data
*
* Common implementation of event trigger de-initialization.
@@ -516,8 +560,8 @@ clear_event_triggers(struct trace_array *tr)
list_for_each_entry_safe(data, n, &file->triggers, list) {
trace_event_trigger_enable_disable(file, 0);
list_del_rcu(&data->list);
- if (data->ops->free)
- data->ops->free(data);
+ if (data->cmd_ops->free)
+ data->cmd_ops->free(data);
}
}
}
@@ -582,8 +626,8 @@ static int register_trigger(char *glob,
}
}
- if (data->ops->init) {
- ret = data->ops->init(data);
+ if (data->cmd_ops->init) {
+ ret = data->cmd_ops->init(data);
if (ret < 0)
goto out;
}
@@ -622,8 +666,8 @@ static bool try_unregister_trigger(char *glob,
}
if (data) {
- if (data->ops->free)
- data->ops->free(data);
+ if (data->cmd_ops->free)
+ data->cmd_ops->free(data);
return true;
}
@@ -835,9 +879,13 @@ int event_trigger_separate_filter(char *param_and_filter, char **param,
* @private_data: User data to associate with the event trigger
*
* Allocate an event_trigger_data instance and initialize it. The
- * @cmd_ops are used along with the @cmd and @param to get the
- * trigger_ops to assign to the event_trigger_data. @private_data can
- * also be passed in and associated with the event_trigger_data.
+ * @cmd_ops defines how the trigger will operate. If @param is set,
+ * and @cmd_ops->trigger_ops->count_func is non NULL, then the
+ * data->count is set to @param and before the trigger is executed, the
+ * @cmd_ops->trigger_ops->count_func() is called. If that function returns
+ * false, the @cmd_ops->trigger_ops->trigger() function will not be called.
+ * @private_data can also be passed in and associated with the
+ * event_trigger_data.
*
* Use trigger_data_free() to free an event_trigger_data object.
*
@@ -849,18 +897,16 @@ struct event_trigger_data *trigger_data_alloc(struct event_command *cmd_ops,
void *private_data)
{
struct event_trigger_data *trigger_data;
- struct event_trigger_ops *trigger_ops;
-
- trigger_ops = cmd_ops->get_trigger_ops(cmd, param);
trigger_data = kzalloc(sizeof(*trigger_data), GFP_KERNEL);
if (!trigger_data)
return NULL;
trigger_data->count = -1;
- trigger_data->ops = trigger_ops;
trigger_data->cmd_ops = cmd_ops;
trigger_data->private_data = private_data;
+ if (param && cmd_ops->count_func)
+ trigger_data->flags |= EVENT_TRIGGER_FL_COUNT;
INIT_LIST_HEAD(&trigger_data->list);
INIT_LIST_HEAD(&trigger_data->named_list);
@@ -1302,31 +1348,28 @@ traceon_trigger(struct event_trigger_data *data,
tracing_on();
}
-static void
-traceon_count_trigger(struct event_trigger_data *data,
- struct trace_buffer *buffer, void *rec,
- struct ring_buffer_event *event)
+static bool
+traceon_count_func(struct event_trigger_data *data,
+ struct trace_buffer *buffer, void *rec,
+ struct ring_buffer_event *event)
{
struct trace_event_file *file = data->private_data;
if (file) {
if (tracer_tracing_is_on(file->tr))
- return;
+ return false;
} else {
if (tracing_is_on())
- return;
+ return false;
}
if (!data->count)
- return;
+ return false;
if (data->count != -1)
(data->count)--;
- if (file)
- tracer_tracing_on(file->tr);
- else
- tracing_on();
+ return true;
}
static void
@@ -1350,31 +1393,28 @@ traceoff_trigger(struct event_trigger_data *data,
tracing_off();
}
-static void
-traceoff_count_trigger(struct event_trigger_data *data,
- struct trace_buffer *buffer, void *rec,
- struct ring_buffer_event *event)
+static bool
+traceoff_count_func(struct event_trigger_data *data,
+ struct trace_buffer *buffer, void *rec,
+ struct ring_buffer_event *event)
{
struct trace_event_file *file = data->private_data;
if (file) {
if (!tracer_tracing_is_on(file->tr))
- return;
+ return false;
} else {
if (!tracing_is_on())
- return;
+ return false;
}
if (!data->count)
- return;
+ return false;
if (data->count != -1)
(data->count)--;
- if (file)
- tracer_tracing_off(file->tr);
- else
- tracing_off();
+ return true;
}
static int
@@ -1391,58 +1431,18 @@ traceoff_trigger_print(struct seq_file *m, struct event_trigger_data *data)
data->filter_str);
}
-static struct event_trigger_ops traceon_trigger_ops = {
- .trigger = traceon_trigger,
- .print = traceon_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops traceon_count_trigger_ops = {
- .trigger = traceon_count_trigger,
- .print = traceon_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops traceoff_trigger_ops = {
- .trigger = traceoff_trigger,
- .print = traceoff_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops traceoff_count_trigger_ops = {
- .trigger = traceoff_count_trigger,
- .print = traceoff_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops *
-onoff_get_trigger_ops(char *cmd, char *param)
-{
- struct event_trigger_ops *ops;
-
- /* we register both traceon and traceoff to this callback */
- if (strcmp(cmd, "traceon") == 0)
- ops = param ? &traceon_count_trigger_ops :
- &traceon_trigger_ops;
- else
- ops = param ? &traceoff_count_trigger_ops :
- &traceoff_trigger_ops;
-
- return ops;
-}
-
static struct event_command trigger_traceon_cmd = {
.name = "traceon",
.trigger_type = ETT_TRACE_ONOFF,
.parse = event_trigger_parse,
.reg = register_trigger,
.unreg = unregister_trigger,
- .get_trigger_ops = onoff_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = traceon_trigger,
+ .count_func = traceon_count_func,
+ .print = traceon_trigger_print,
+ .init = event_trigger_init,
+ .free = event_trigger_free,
};
static struct event_command trigger_traceoff_cmd = {
@@ -1452,8 +1452,12 @@ static struct event_command trigger_traceoff_cmd = {
.parse = event_trigger_parse,
.reg = register_trigger,
.unreg = unregister_trigger,
- .get_trigger_ops = onoff_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = traceoff_trigger,
+ .count_func = traceoff_count_func,
+ .print = traceoff_trigger_print,
+ .init = event_trigger_init,
+ .free = event_trigger_free,
};
#ifdef CONFIG_TRACER_SNAPSHOT
@@ -1470,20 +1474,6 @@ snapshot_trigger(struct event_trigger_data *data,
tracing_snapshot();
}
-static void
-snapshot_count_trigger(struct event_trigger_data *data,
- struct trace_buffer *buffer, void *rec,
- struct ring_buffer_event *event)
-{
- if (!data->count)
- return;
-
- if (data->count != -1)
- (data->count)--;
-
- snapshot_trigger(data, buffer, rec, event);
-}
-
static int
register_snapshot_trigger(char *glob,
struct event_trigger_data *data,
@@ -1515,34 +1505,18 @@ snapshot_trigger_print(struct seq_file *m, struct event_trigger_data *data)
data->filter_str);
}
-static struct event_trigger_ops snapshot_trigger_ops = {
- .trigger = snapshot_trigger,
- .print = snapshot_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops snapshot_count_trigger_ops = {
- .trigger = snapshot_count_trigger,
- .print = snapshot_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops *
-snapshot_get_trigger_ops(char *cmd, char *param)
-{
- return param ? &snapshot_count_trigger_ops : &snapshot_trigger_ops;
-}
-
static struct event_command trigger_snapshot_cmd = {
.name = "snapshot",
.trigger_type = ETT_SNAPSHOT,
.parse = event_trigger_parse,
.reg = register_snapshot_trigger,
.unreg = unregister_snapshot_trigger,
- .get_trigger_ops = snapshot_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = snapshot_trigger,
+ .count_func = event_trigger_count,
+ .print = snapshot_trigger_print,
+ .init = event_trigger_init,
+ .free = event_trigger_free,
};
static __init int register_trigger_snapshot_cmd(void)
@@ -1589,20 +1563,6 @@ stacktrace_trigger(struct event_trigger_data *data,
trace_dump_stack(STACK_SKIP);
}
-static void
-stacktrace_count_trigger(struct event_trigger_data *data,
- struct trace_buffer *buffer, void *rec,
- struct ring_buffer_event *event)
-{
- if (!data->count)
- return;
-
- if (data->count != -1)
- (data->count)--;
-
- stacktrace_trigger(data, buffer, rec, event);
-}
-
static int
stacktrace_trigger_print(struct seq_file *m, struct event_trigger_data *data)
{
@@ -1610,26 +1570,6 @@ stacktrace_trigger_print(struct seq_file *m, struct event_trigger_data *data)
data->filter_str);
}
-static struct event_trigger_ops stacktrace_trigger_ops = {
- .trigger = stacktrace_trigger,
- .print = stacktrace_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops stacktrace_count_trigger_ops = {
- .trigger = stacktrace_count_trigger,
- .print = stacktrace_trigger_print,
- .init = event_trigger_init,
- .free = event_trigger_free,
-};
-
-static struct event_trigger_ops *
-stacktrace_get_trigger_ops(char *cmd, char *param)
-{
- return param ? &stacktrace_count_trigger_ops : &stacktrace_trigger_ops;
-}
-
static struct event_command trigger_stacktrace_cmd = {
.name = "stacktrace",
.trigger_type = ETT_STACKTRACE,
@@ -1637,8 +1577,12 @@ static struct event_command trigger_stacktrace_cmd = {
.parse = event_trigger_parse,
.reg = register_trigger,
.unreg = unregister_trigger,
- .get_trigger_ops = stacktrace_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = stacktrace_trigger,
+ .count_func = event_trigger_count,
+ .print = stacktrace_trigger_print,
+ .init = event_trigger_init,
+ .free = event_trigger_free,
};
static __init int register_trigger_stacktrace_cmd(void)
@@ -1673,24 +1617,24 @@ event_enable_trigger(struct event_trigger_data *data,
set_bit(EVENT_FILE_FL_SOFT_DISABLED_BIT, &enable_data->file->flags);
}
-static void
-event_enable_count_trigger(struct event_trigger_data *data,
- struct trace_buffer *buffer, void *rec,
- struct ring_buffer_event *event)
+static bool
+event_enable_count_func(struct event_trigger_data *data,
+ struct trace_buffer *buffer, void *rec,
+ struct ring_buffer_event *event)
{
struct enable_trigger_data *enable_data = data->private_data;
if (!data->count)
- return;
+ return false;
/* Skip if the event is in a state we want to switch to */
if (enable_data->enable == !(enable_data->file->flags & EVENT_FILE_FL_SOFT_DISABLED))
- return;
+ return false;
if (data->count != -1)
(data->count)--;
- event_enable_trigger(data, buffer, rec, event);
+ return true;
}
int event_enable_trigger_print(struct seq_file *m,
@@ -1735,34 +1679,6 @@ void event_enable_trigger_free(struct event_trigger_data *data)
}
}
-static struct event_trigger_ops event_enable_trigger_ops = {
- .trigger = event_enable_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
-static struct event_trigger_ops event_enable_count_trigger_ops = {
- .trigger = event_enable_count_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
-static struct event_trigger_ops event_disable_trigger_ops = {
- .trigger = event_enable_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
-static struct event_trigger_ops event_disable_count_trigger_ops = {
- .trigger = event_enable_count_trigger,
- .print = event_enable_trigger_print,
- .init = event_trigger_init,
- .free = event_enable_trigger_free,
-};
-
int event_enable_trigger_parse(struct event_command *cmd_ops,
struct trace_event_file *file,
char *glob, char *cmd, char *param_and_filter)
@@ -1894,8 +1810,8 @@ int event_enable_register_trigger(char *glob,
}
}
- if (data->ops->init) {
- ret = data->ops->init(data);
+ if (data->cmd_ops->init) {
+ ret = data->cmd_ops->init(data);
if (ret < 0)
goto out;
}
@@ -1936,30 +1852,8 @@ void event_enable_unregister_trigger(char *glob,
}
}
- if (data && data->ops->free)
- data->ops->free(data);
-}
-
-static struct event_trigger_ops *
-event_enable_get_trigger_ops(char *cmd, char *param)
-{
- struct event_trigger_ops *ops;
- bool enable;
-
-#ifdef CONFIG_HIST_TRIGGERS
- enable = ((strcmp(cmd, ENABLE_EVENT_STR) == 0) ||
- (strcmp(cmd, ENABLE_HIST_STR) == 0));
-#else
- enable = strcmp(cmd, ENABLE_EVENT_STR) == 0;
-#endif
- if (enable)
- ops = param ? &event_enable_count_trigger_ops :
- &event_enable_trigger_ops;
- else
- ops = param ? &event_disable_count_trigger_ops :
- &event_disable_trigger_ops;
-
- return ops;
+ if (data && data->cmd_ops->free)
+ data->cmd_ops->free(data);
}
static struct event_command trigger_enable_cmd = {
@@ -1968,8 +1862,12 @@ static struct event_command trigger_enable_cmd = {
.parse = event_enable_trigger_parse,
.reg = event_enable_register_trigger,
.unreg = event_enable_unregister_trigger,
- .get_trigger_ops = event_enable_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = event_enable_trigger,
+ .count_func = event_enable_count_func,
+ .print = event_enable_trigger_print,
+ .init = event_trigger_init,
+ .free = event_enable_trigger_free,
};
static struct event_command trigger_disable_cmd = {
@@ -1978,8 +1876,12 @@ static struct event_command trigger_disable_cmd = {
.parse = event_enable_trigger_parse,
.reg = event_enable_register_trigger,
.unreg = event_enable_unregister_trigger,
- .get_trigger_ops = event_enable_get_trigger_ops,
.set_filter = set_trigger_filter,
+ .trigger = event_enable_trigger,
+ .count_func = event_enable_count_func,
+ .print = event_enable_trigger_print,
+ .init = event_trigger_init,
+ .free = event_enable_trigger_free,
};
static __init void unregister_trigger_enable_disable_cmds(void)
diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c
index fbb6cf8dc047..e668c2697dcc 100644
--- a/kernel/trace/trace_functions.c
+++ b/kernel/trace/trace_functions.c
@@ -97,7 +97,7 @@ int ftrace_create_function_files(struct trace_array *tr,
return ret;
}
- ftrace_create_filter_files(tr->ops, parent);
+ ftrace_create_filter_files(tr, parent);
return 0;
}
diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c
index 3ea47feadc25..9c6ca26ffb92 100644
--- a/kernel/trace/trace_probe.c
+++ b/kernel/trace/trace_probe.c
@@ -534,6 +534,7 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type,
{
struct fetch_insn *code = *pcode;
const struct btf_member *field;
+ const struct btf_type *mtype;
u32 bitoffs, anon_offs;
char *next;
int is_ptr;
@@ -562,7 +563,7 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type,
anon_offs = 0;
field = btf_find_struct_member(ctx->btf, type, fieldname,
- &anon_offs);
+ &anon_offs, &mtype);
if (IS_ERR(field)) {
trace_probe_log_err(ctx->offset, BAD_BTF_TID);
return PTR_ERR(field);
@@ -575,7 +576,7 @@ static int parse_btf_field(char *fieldname, const struct btf_type *type,
bitoffs += anon_offs;
/* Accumulate the bit-offsets of the dot-connected fields */
- if (btf_type_kflag(type)) {
+ if (btf_type_kflag(mtype)) {
bitoffs += BTF_MEMBER_BIT_OFFSET(field->offset);
ctx->last_bitsize = BTF_MEMBER_BITFIELD_SIZE(field->offset);
} else {
diff --git a/kernel/trace/trace_stack.c b/kernel/trace/trace_stack.c
index 7f9572a37333..d602c86de18f 100644
--- a/kernel/trace/trace_stack.c
+++ b/kernel/trace/trace_stack.c
@@ -499,7 +499,7 @@ stack_trace_filter_open(struct inode *inode, struct file *file)
struct ftrace_ops *ops = inode->i_private;
/* Checks for tracefs lockdown */
- return ftrace_regex_open(ops, FTRACE_ITER_FILTER,
+ return ftrace_regex_open(NULL, ops, FTRACE_ITER_FILTER,
inode, file);
}
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 376a988658b3..b030303fd1e4 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -3762,7 +3762,7 @@ static void check_flush_dependency(struct workqueue_struct *target_wq,
WARN_ONCE(current->flags & PF_MEMALLOC,
"workqueue: PF_MEMALLOC task %d(%s) is flushing !WQ_MEM_RECLAIM %s:%ps",
current->pid, current->comm, target_wq->name, target_func);
- WARN_ONCE(worker && ((worker->current_pwq->wq->flags &
+ WARN_ONCE(worker && worker->current_pwq && ((worker->current_pwq->wq->flags &
(WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM),
"workqueue: WQ_MEM_RECLAIM %s:%ps is flushing !WQ_MEM_RECLAIM %s:%ps",
worker->current_pwq->wq->name, worker->current_func,
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 956a7e23b5d6..2cf1f283b84d 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -910,8 +910,9 @@ static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
continue;
spin_unlock_irq(&cgwb_lock);
- while (cleanup_offline_cgwb(wb))
- cond_resched();
+ do {
+ cond_resched_tasks_rcu_qs();
+ } while (cleanup_offline_cgwb(wb));
spin_lock_irq(&cgwb_lock);
wb_put(wb);
diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c
index 089ee23e07de..963a098af9ec 100644
--- a/mm/damon/vaddr.c
+++ b/mm/damon/vaddr.c
@@ -347,6 +347,7 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,
if (pte_young(entry)) {
referenced = true;
+ entry = huge_ptep_get_and_clear(mm, addr, pte, psize);
entry = pte_mkold(entry);
set_huge_pte_at(mm, addr, pte, entry, psize);
}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 6fc4e1fb88ae..1a76a21724d4 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3420,6 +3420,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
/* reset xarray order to new order after split */
XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order);
bool is_anon = folio_test_anon(folio);
+ struct mem_cgroup *memcg, *old_memcg;
struct address_space *mapping = NULL;
struct anon_vma *anon_vma = NULL;
int order = folio_order(folio);
@@ -3483,6 +3484,13 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
if (folio_test_writeback(folio))
return -EBUSY;
+ /*
+ * switch to folio's memcg as xarray node allocation can happen and
+ * needs to charge to it.
+ */
+ memcg = folio_memcg(folio);
+ old_memcg = set_active_memcg(memcg);
+
if (is_anon) {
/*
* The caller does not necessarily hold an mmap_lock that would
@@ -3629,6 +3637,8 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list,
if (mapping)
i_mmap_unlock_read(mapping);
out:
+ /* restore to caller's old_memcg */
+ set_active_memcg(old_memcg);
xas_destroy(&xas);
if (order == HPAGE_PMD_ORDER)
count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 0d98b591bcd1..7fccb866dab9 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -104,6 +104,17 @@ static void hugetlb_free_folio(struct folio *folio)
folio_put(folio);
}
+/*
+ * Check if the hstate represents gigantic pages but gigantic page
+ * runtime support is not available. This is a common condition used to
+ * skip operations that cannot be performed on gigantic pages when runtime
+ * support is disabled.
+ */
+static inline bool hstate_is_gigantic_no_runtime(struct hstate *h)
+{
+ return hstate_is_gigantic(h) && !gigantic_page_runtime_supported();
+}
+
static inline bool subpool_is_free(struct hugepage_subpool *spool)
{
if (spool->count)
@@ -1594,7 +1605,7 @@ static void remove_hugetlb_folio(struct hstate *h, struct folio *folio,
VM_BUG_ON_FOLIO(hugetlb_cgroup_from_folio_rsvd(folio), folio);
lockdep_assert_held(&hugetlb_lock);
- if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported())
+ if (hstate_is_gigantic_no_runtime(h))
return;
list_del(&folio->lru);
@@ -1656,7 +1667,7 @@ static void __update_and_free_hugetlb_folio(struct hstate *h,
{
bool clear_flag = folio_test_hugetlb_vmemmap_optimized(folio);
- if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported())
+ if (hstate_is_gigantic_no_runtime(h))
return;
/*
@@ -2201,6 +2212,17 @@ int dissolve_free_hugetlb_folio(struct folio *folio)
if (!folio_ref_count(folio)) {
struct hstate *h = folio_hstate(folio);
+ bool adjust_surplus = false;
+
+ /*
+ * remove_hugetlb_folio()/update_and_free_hugetlb_folio() bail
+ * for gigantic hstates without runtime support, so dissolving one
+ * here would leave it on the free list and, on vmemmap restore
+ * failure, the add_hugetlb_folio() rollback corrupts that list.
+ */
+ if (hstate_is_gigantic_no_runtime(h))
+ goto out;
+
if (!available_huge_pages(h))
goto out;
@@ -2223,8 +2245,11 @@ int dissolve_free_hugetlb_folio(struct folio *folio)
goto retry;
}
- remove_hugetlb_folio(h, folio, false);
- h->max_huge_pages--;
+ if (h->surplus_huge_pages_node[folio_nid(folio)])
+ adjust_surplus = true;
+ remove_hugetlb_folio(h, folio, adjust_surplus);
+ if (!adjust_surplus)
+ h->max_huge_pages--;
spin_unlock_irq(&hugetlb_lock);
/*
@@ -2243,8 +2268,9 @@ int dissolve_free_hugetlb_folio(struct folio *folio)
rc = hugetlb_vmemmap_restore_folio(h, folio);
if (rc) {
spin_lock_irq(&hugetlb_lock);
- add_hugetlb_folio(h, folio, false);
- h->max_huge_pages++;
+ add_hugetlb_folio(h, folio, adjust_surplus);
+ if (!adjust_surplus)
+ h->max_huge_pages++;
goto out;
}
} else
@@ -2560,7 +2586,7 @@ static void return_unused_surplus_pages(struct hstate *h,
/* Uncommit the reservation */
h->resv_huge_pages -= unused_resv_pages;
- if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported())
+ if (hstate_is_gigantic_no_runtime(h))
goto out;
/*
@@ -3489,7 +3515,7 @@ static void __init hugetlb_init_hstates(void)
* - If CMA allocation is possible, we can not demote
* HUGETLB_PAGE_ORDER or smaller size pages.
*/
- if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported())
+ if (hstate_is_gigantic_no_runtime(h))
continue;
if (hugetlb_cma_size && h->order <= HUGETLB_PAGE_ORDER)
continue;
@@ -3934,7 +3960,7 @@ static ssize_t __nr_hugepages_store_common(bool obey_mempolicy,
int err;
nodemask_t nodes_allowed, *n_mask;
- if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported())
+ if (hstate_is_gigantic_no_runtime(h))
return -EINVAL;
if (nid == NUMA_NO_NODE) {
@@ -5473,18 +5499,21 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
hugetlb_vma_lock_write(vma);
i_mmap_lock_write(mapping);
for (; old_addr < old_end; old_addr += sz, new_addr += sz) {
+ const unsigned long offset_to_last_entry =
+ (old_addr | last_addr_mask) - old_addr;
+
src_pte = hugetlb_walk(vma, old_addr, sz);
if (!src_pte) {
- old_addr |= last_addr_mask;
- new_addr |= last_addr_mask;
+ old_addr += offset_to_last_entry;
+ new_addr += offset_to_last_entry;
continue;
}
if (huge_pte_none(huge_ptep_get(mm, old_addr, src_pte)))
continue;
if (huge_pmd_unshare(&tlb, vma, old_addr, src_pte)) {
- old_addr |= last_addr_mask;
- new_addr |= last_addr_mask;
+ old_addr += offset_to_last_entry;
+ new_addr += offset_to_last_entry;
continue;
}
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index a6b2c7d7d73b..84bacd7e1962 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -369,12 +369,6 @@ int hugepage_madvise(struct vm_area_struct *vma,
#endif
*vm_flags &= ~VM_NOHUGEPAGE;
*vm_flags |= VM_HUGEPAGE;
- /*
- * If the vma become good for khugepaged to scan,
- * register it here without waiting a page fault that
- * may not happen any time soon.
- */
- khugepaged_enter_vma(vma, *vm_flags);
break;
case MADV_NOHUGEPAGE:
*vm_flags &= ~VM_HUGEPAGE;
diff --git a/mm/madvise.c b/mm/madvise.c
index d2b315993a1c..4abc466d3750 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -158,6 +158,14 @@ static int madvise_update_vma(struct vm_area_struct *vma,
/* vm_flags is protected by the mmap_lock held in write mode. */
vma_start_write(vma);
vm_flags_reset(vma, new_flags);
+ /*
+ * If the vma become good for khugepaged to scan,
+ * register it here without waiting a page fault that
+ * may not happen any time soon.
+ */
+ if (new_flags & VM_HUGEPAGE)
+ khugepaged_enter_vma(vma, new_flags);
+
if (!vma->vm_file || vma_is_anon_shmem(vma)) {
error = replace_anon_vma_name(vma, anon_name);
if (error)
diff --git a/mm/memblock.c b/mm/memblock.c
index 3d7b0114442c..00ef6bea6fe8 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -2416,14 +2416,18 @@ static int memblock_debug_show(struct seq_file *m, void *private)
else
seq_printf(m, "%4c ", 'x');
if (reg->flags) {
- for (j = 0; j < count; j++) {
- if (reg->flags & (1U << j)) {
- seq_printf(m, "%s\n", flagname[j]);
- break;
- }
+ unsigned int flags = reg->flags;
+ bool first = true;
+
+ for (j = 0; flags; j++, flags >>= 1) {
+ if (!(flags & 1))
+ continue;
+ if (!first)
+ seq_putc(m, '|');
+ seq_puts(m, j < count ? flagname[j] : "UNKNOWN");
+ first = false;
}
- if (j == count)
- seq_printf(m, "%s\n", "UNKNOWN");
+ seq_putc(m, '\n');
} else {
seq_printf(m, "%s\n", "NONE");
}
diff --git a/mm/mlock.c b/mm/mlock.c
index d16bf4dbd06d..8293d2d03ab0 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -141,7 +141,7 @@ static struct lruvec *__munlock_folio(struct folio *folio, struct lruvec *lruvec
munlock:
if (folio_test_clear_mlocked(folio)) {
- __zone_stat_mod_folio(folio, NR_MLOCK, -nr_pages);
+ zone_stat_mod_folio(folio, NR_MLOCK, -nr_pages);
if (isolated || !folio_test_unevictable(folio))
__count_vm_events(UNEVICTABLE_PGMUNLOCKED, nr_pages);
else
diff --git a/mm/mremap.c b/mm/mremap.c
index 12af89b4342a..9685ea816fde 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -811,8 +811,18 @@ static unsigned long move_vma(struct vm_area_struct *vma,
* table has been moved.
*/
if (new_vma != vma && vma->vm_start == old_addr &&
- vma->vm_end == (old_addr + old_len))
+ vma->vm_end == (old_addr + old_len)) {
+ const pgoff_t pgoff_unfaulted = vma->vm_start >> PAGE_SHIFT;
+
unlink_anon_vmas(vma);
+ /*
+ * The VMA is now unfaulted and it is an invariant that
+ * unfaulted anonymous VMAs have page offset equal to
+ * vma->vm_start >> PAGE_SHIFT.
+ */
+ if (vma_is_anonymous(vma) && !vma->vm_file)
+ vma->vm_pgoff = pgoff_unfaulted;
+ }
/* Because we won't unmap we don't need to touch locked_vm */
return new_addr;
diff --git a/mm/rmap.c b/mm/rmap.c
index c051651de6f9..bc9c66cc950b 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -148,14 +148,13 @@ static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain)
kmem_cache_free(anon_vma_chain_cachep, anon_vma_chain);
}
-static void anon_vma_chain_link(struct vm_area_struct *vma,
- struct anon_vma_chain *avc,
- struct anon_vma *anon_vma)
+static void anon_vma_chain_assign(struct vm_area_struct *vma,
+ struct anon_vma_chain *avc,
+ struct anon_vma *anon_vma)
{
avc->vma = vma;
avc->anon_vma = anon_vma;
list_add(&avc->same_vma, &vma->anon_vma_chain);
- anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
}
/**
@@ -211,8 +210,13 @@ int __anon_vma_prepare(struct vm_area_struct *vma)
/* page_table_lock to protect against threads */
spin_lock(&mm->page_table_lock);
if (likely(!vma->anon_vma)) {
- vma->anon_vma = anon_vma;
- anon_vma_chain_link(vma, avc, anon_vma);
+ /*
+ * Make anon_vma fields visible before anon_vma is published.
+ * Paired with an address dependency in reusable_anon_vma().
+ */
+ smp_store_release(&vma->anon_vma, anon_vma);
+ anon_vma_chain_assign(vma, avc, anon_vma);
+ anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
anon_vma->num_active_vmas++;
allocated = NULL;
avc = NULL;
@@ -296,7 +300,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
}
anon_vma = pavc->anon_vma;
root = lock_anon_vma_root(root, anon_vma);
- anon_vma_chain_link(dst, avc, anon_vma);
+ anon_vma_chain_assign(dst, avc, anon_vma);
+ anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
/*
* Reuse existing anon_vma if it has no vma and only one
@@ -380,8 +385,10 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)
get_anon_vma(anon_vma->root);
/* Mark this anon_vma as the one where our new (COWed) pages go. */
vma->anon_vma = anon_vma;
+ anon_vma_chain_assign(vma, avc, anon_vma);
+ /* Now let rmap see it. */
anon_vma_lock_write(anon_vma);
- anon_vma_chain_link(vma, avc, anon_vma);
+ anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
anon_vma->parent->num_children++;
anon_vma_unlock_write(anon_vma);
diff --git a/mm/vma.c b/mm/vma.c
index 140f7017bb63..3df4cf796c7f 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -1760,6 +1760,13 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct *
* acceptable for merging, so we can do all of this optimistically. But
* we do that READ_ONCE() to make sure that we never re-load the pointer.
*
+ * The READ_ONCE() establishes an address dependency between anon_vma and
+ * any access to its fields, which pairs with the assignment to
+ * vma->anon_vma performed with release semantics in __anon_vma_prepare().
+ *
+ * This is especially important as anon_vma's are SLAB_TYPESAFE_BY_RCU so
+ * accessing an uninitialised anon_vma's fields may result in a UAF.
+ *
* IOW: that the "list_is_singular()" test on the anon_vma_chain only
* matters for the 'stable anon_vma' case (ie the thing we want to avoid
* is to return an anon_vma that is "complex" due to having gone through
@@ -1774,6 +1781,7 @@ static struct anon_vma *reusable_anon_vma(struct vm_area_struct *old,
struct vm_area_struct *b)
{
if (anon_vma_compatible(a, b)) {
+ /* Paired with a memory barrier in __anon_vma_prepare(). */
struct anon_vma *anon_vma = READ_ONCE(old->anon_vma);
if (anon_vma && list_is_singular(&old->anon_vma_chain))
diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c
index e7e5b903fa8c..61fb68fadea3 100644
--- a/net/8021q/vlan_dev.c
+++ b/net/8021q/vlan_dev.c
@@ -54,6 +54,11 @@ static int vlan_dev_hard_header(struct sk_buff *skb, struct net_device *dev,
int rc;
if (!(vlan->flags & VLAN_FLAG_REORDER_HDR)) {
+ unsigned int hlen = READ_ONCE(dev->hard_header_len) +
+ READ_ONCE(dev->needed_headroom);
+
+ if (skb_cow_head(skb, hlen) < 0)
+ return -ENOMEM;
vhdr = skb_push(skb, VLAN_HLEN);
vlan_tci = vlan->vlan_id;
diff --git a/net/atm/pppoatm.c b/net/atm/pppoatm.c
index 3e4f17d335fe..b668a30b67a8 100644
--- a/net/atm/pppoatm.c
+++ b/net/atm/pppoatm.c
@@ -292,10 +292,13 @@ static int pppoatm_send(struct ppp_channel *chan, struct sk_buff *skb)
struct atm_vcc *vcc;
int ret;
+ if (!pskb_may_pull(skb, 1)) {
+ kfree_skb(skb);
+ return DROP_PACKET;
+ }
+
ATM_SKB(skb)->vcc = pvcc->atmvcc;
pr_debug("(skb=0x%p, vcc=0x%p)\n", skb, pvcc->atmvcc);
- if (skb->data[0] == '\0' && (pvcc->flags & SC_COMP_PROT))
- (void) skb_pull(skb, 1);
vcc = ATM_SKB(skb)->vcc;
bh_lock_sock(sk_atm(vcc));
@@ -318,23 +321,13 @@ static int pppoatm_send(struct ppp_channel *chan, struct sk_buff *skb)
switch (pvcc->encaps) { /* LLC encapsulation needed */
case e_llc:
- if (skb_headroom(skb) < LLC_LEN) {
- struct sk_buff *n;
- n = skb_realloc_headroom(skb, LLC_LEN);
- if (n != NULL &&
- !pppoatm_may_send(pvcc, n->truesize)) {
- kfree_skb(n);
- goto nospace;
- }
- consume_skb(skb);
- skb = n;
- if (skb == NULL) {
- bh_unlock_sock(sk_atm(vcc));
- return DROP_PACKET;
- }
- } else if (!pppoatm_may_send(pvcc, skb->truesize))
+ if (skb_cow_head(skb, LLC_LEN)) {
+ bh_unlock_sock(sk_atm(vcc));
+ kfree_skb(skb);
+ return DROP_PACKET;
+ }
+ if (!pppoatm_may_send(pvcc, skb->truesize))
goto nospace;
- memcpy(skb_push(skb, LLC_LEN), pppllc, LLC_LEN);
break;
case e_vc:
if (!pppoatm_may_send(pvcc, skb->truesize))
@@ -347,6 +340,12 @@ static int pppoatm_send(struct ppp_channel *chan, struct sk_buff *skb)
return 1;
}
+ if (skb->data[0] == '\0' && (pvcc->flags & SC_COMP_PROT))
+ skb_pull(skb, 1);
+
+ if (pvcc->encaps == e_llc)
+ memcpy(skb_push(skb, LLC_LEN), pppllc, LLC_LEN);
+
atm_account_tx(vcc, skb);
pr_debug("atm_skb(%p)->vcc(%p)->dev(%p)\n",
skb, ATM_SKB(skb)->vcc, ATM_SKB(skb)->vcc->dev);
@@ -356,13 +355,6 @@ static int pppoatm_send(struct ppp_channel *chan, struct sk_buff *skb)
return ret;
nospace:
bh_unlock_sock(sk_atm(vcc));
- /*
- * We don't have space to send this SKB now, but we might have
- * already applied SC_COMP_PROT compression, so may need to undo
- */
- if ((pvcc->flags & SC_COMP_PROT) && skb_headroom(skb) > 0 &&
- skb->data[-1] == '\0')
- (void) skb_push(skb, 1);
return 0;
}
diff --git a/net/bluetooth/bnep/core.c b/net/bluetooth/bnep/core.c
index ed3ca38d8a6e..e93c79982abc 100644
--- a/net/bluetooth/bnep/core.c
+++ b/net/bluetooth/bnep/core.c
@@ -273,9 +273,14 @@ static int bnep_rx_extension(struct bnep_session *s, struct sk_buff *skb)
BT_DBG("type 0x%x len %u", h->type, h->len);
+ if (skb->len < h->len) {
+ err = -EILSEQ;
+ break;
+ }
+
switch (h->type & BNEP_TYPE_MASK) {
case BNEP_EXT_CONTROL:
- bnep_rx_control(s, skb->data, skb->len);
+ bnep_rx_control(s, skb->data, h->len);
break;
default:
@@ -376,6 +381,11 @@ static int bnep_rx_frame(struct bnep_session *s, struct sk_buff *skb)
goto badframe;
}
+ if ((type & BNEP_TYPE_MASK) == BNEP_CONTROL) {
+ kfree_skb(skb);
+ return 0;
+ }
+
/* Strip 802.1p header */
if (ntohs(s->eh.h_proto) == ETH_P_8021Q) {
if (!skb_pull(skb, 4))
@@ -454,6 +464,11 @@ static int bnep_tx_frame(struct bnep_session *s, struct sk_buff *skb)
goto send;
}
+ if (skb->len < ETH_HLEN) {
+ kfree_skb(skb);
+ return 0;
+ }
+
iv[il++] = (struct kvec) { &type, 1 };
len++;
diff --git a/net/bluetooth/bnep/netdev.c b/net/bluetooth/bnep/netdev.c
index cc1cff63194f..5d9d851db926 100644
--- a/net/bluetooth/bnep/netdev.c
+++ b/net/bluetooth/bnep/netdev.c
@@ -169,6 +169,12 @@ static netdev_tx_t bnep_net_xmit(struct sk_buff *skb,
BT_DBG("skb %p, dev %p", skb, dev);
+ if (!pskb_may_pull(skb, ETH_HLEN)) {
+ dev->stats.tx_dropped++;
+ kfree_skb(skb);
+ return NETDEV_TX_OK;
+ }
+
#ifdef CONFIG_BT_BNEP_MC_FILTER
if (bnep_net_mc_filter(skb, s)) {
kfree_skb(skb);
@@ -221,7 +227,7 @@ void bnep_net_setup(struct net_device *dev)
dev->addr_len = ETH_ALEN;
ether_setup(dev);
- dev->min_mtu = 0;
+ dev->min_mtu = ETH_MIN_MTU;
dev->max_mtu = ETH_MAX_MTU;
dev->priv_flags &= ~IFF_TX_SKB_SHARING;
dev->netdev_ops = &bnep_netdev_ops;
diff --git a/net/bluetooth/coredump.c b/net/bluetooth/coredump.c
index c18df3a08607..517a61234ef4 100644
--- a/net/bluetooth/coredump.c
+++ b/net/bluetooth/coredump.c
@@ -105,6 +105,22 @@ static void hci_devcd_free(struct hci_dev *hdev)
hci_devcd_reset(hdev);
}
+void hci_devcd_shutdown(struct hci_dev *hdev)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&hdev->dump.dump_q.lock, flags);
+ hdev->dump.supported = false;
+ spin_unlock_irqrestore(&hdev->dump.dump_q.lock, flags);
+
+ disable_work_sync(&hdev->dump.dump_rx);
+ disable_delayed_work_sync(&hdev->dump.dump_timeout);
+
+ hci_dev_lock(hdev);
+ hci_devcd_free(hdev);
+ hci_dev_unlock(hdev);
+}
+
/* Call with hci_dev_lock only. */
static int hci_devcd_alloc(struct hci_dev *hdev, u32 size)
{
@@ -426,7 +442,29 @@ EXPORT_SYMBOL(hci_devcd_register);
static inline bool hci_devcd_enabled(struct hci_dev *hdev)
{
- return hdev->dump.supported;
+ return READ_ONCE(hdev->dump.supported);
+}
+
+static int hci_devcd_queue(struct hci_dev *hdev, struct sk_buff *skb)
+{
+ unsigned long flags;
+ int err = 0;
+
+ spin_lock_irqsave(&hdev->dump.dump_q.lock, flags);
+ if (!hdev->dump.supported)
+ err = -EOPNOTSUPP;
+ else
+ __skb_queue_tail(&hdev->dump.dump_q, skb);
+ spin_unlock_irqrestore(&hdev->dump.dump_q.lock, flags);
+
+ if (err) {
+ kfree_skb(skb);
+ return err;
+ }
+
+ queue_work(hdev->workqueue, &hdev->dump.dump_rx);
+
+ return 0;
}
int hci_devcd_init(struct hci_dev *hdev, u32 dump_size)
@@ -443,10 +481,7 @@ int hci_devcd_init(struct hci_dev *hdev, u32 dump_size)
hci_dmp_cb(skb)->pkt_type = HCI_DEVCOREDUMP_PKT_INIT;
put_unaligned_le32(dump_size, skb_put(skb, 4));
- skb_queue_tail(&hdev->dump.dump_q, skb);
- queue_work(hdev->workqueue, &hdev->dump.dump_rx);
-
- return 0;
+ return hci_devcd_queue(hdev, skb);
}
EXPORT_SYMBOL(hci_devcd_init);
@@ -462,10 +497,7 @@ int hci_devcd_append(struct hci_dev *hdev, struct sk_buff *skb)
hci_dmp_cb(skb)->pkt_type = HCI_DEVCOREDUMP_PKT_SKB;
- skb_queue_tail(&hdev->dump.dump_q, skb);
- queue_work(hdev->workqueue, &hdev->dump.dump_rx);
-
- return 0;
+ return hci_devcd_queue(hdev, skb);
}
EXPORT_SYMBOL(hci_devcd_append);
@@ -487,10 +519,7 @@ int hci_devcd_append_pattern(struct hci_dev *hdev, u8 pattern, u32 len)
hci_dmp_cb(skb)->pkt_type = HCI_DEVCOREDUMP_PKT_PATTERN;
skb_put_data(skb, &p, sizeof(p));
- skb_queue_tail(&hdev->dump.dump_q, skb);
- queue_work(hdev->workqueue, &hdev->dump.dump_rx);
-
- return 0;
+ return hci_devcd_queue(hdev, skb);
}
EXPORT_SYMBOL(hci_devcd_append_pattern);
@@ -507,10 +536,7 @@ int hci_devcd_complete(struct hci_dev *hdev)
hci_dmp_cb(skb)->pkt_type = HCI_DEVCOREDUMP_PKT_COMPLETE;
- skb_queue_tail(&hdev->dump.dump_q, skb);
- queue_work(hdev->workqueue, &hdev->dump.dump_rx);
-
- return 0;
+ return hci_devcd_queue(hdev, skb);
}
EXPORT_SYMBOL(hci_devcd_complete);
@@ -527,9 +553,6 @@ int hci_devcd_abort(struct hci_dev *hdev)
hci_dmp_cb(skb)->pkt_type = HCI_DEVCOREDUMP_PKT_ABORT;
- skb_queue_tail(&hdev->dump.dump_q, skb);
- queue_work(hdev->workqueue, &hdev->dump.dump_rx);
-
- return 0;
+ return hci_devcd_queue(hdev, skb);
}
EXPORT_SYMBOL(hci_devcd_abort);
diff --git a/net/bluetooth/eir.c b/net/bluetooth/eir.c
index a55696820b22..ee0136bfae40 100644
--- a/net/bluetooth/eir.c
+++ b/net/bluetooth/eir.c
@@ -373,7 +373,15 @@ void *eir_get_service_data(u8 *eir, size_t eir_len, u16 uuid, size_t *len)
size_t dlen;
while ((eir = eir_get_data(eir, eir_len, EIR_SERVICE_DATA, &dlen))) {
- u16 value = get_unaligned_le16(eir);
+ u16 value;
+
+ if (dlen < sizeof(value)) {
+ eir += dlen;
+ eir_len = eir_end - eir;
+ continue;
+ }
+
+ value = get_unaligned_le16(eir);
if (uuid == value) {
if (len)
diff --git a/net/bluetooth/hci_codec.c b/net/bluetooth/hci_codec.c
index 5bc5003c387c..7a7e813dcdda 100644
--- a/net/bluetooth/hci_codec.c
+++ b/net/bluetooth/hci_codec.c
@@ -145,11 +145,12 @@ void hci_read_supported_codecs(struct hci_dev *hdev)
skb_pull(skb, sizeof(rp->status));
- std_codecs = (void *)skb->data;
+ std_codecs = skb_pull_data(skb, sizeof(*std_codecs));
+ if (!std_codecs)
+ goto error;
/* validate codecs length before accessing */
- if (skb->len < flex_array_size(std_codecs, codec, std_codecs->num)
- + sizeof(std_codecs->num))
+ if (skb->len < flex_array_size(std_codecs, codec, std_codecs->num))
goto error;
/* enumerate codec capabilities of standard codecs */
@@ -161,15 +162,14 @@ void hci_read_supported_codecs(struct hci_dev *hdev)
LOCAL_CODEC_ACL_MASK | LOCAL_CODEC_SCO_MASK, &caps);
}
- skb_pull(skb, flex_array_size(std_codecs, codec, std_codecs->num)
- + sizeof(std_codecs->num));
+ skb_pull(skb, flex_array_size(std_codecs, codec, std_codecs->num));
- vnd_codecs = (void *)skb->data;
+ vnd_codecs = skb_pull_data(skb, sizeof(*vnd_codecs));
+ if (!vnd_codecs)
+ goto error;
/* validate vendor codecs length before accessing */
- if (skb->len <
- flex_array_size(vnd_codecs, codec, vnd_codecs->num)
- + sizeof(vnd_codecs->num))
+ if (skb->len < flex_array_size(vnd_codecs, codec, vnd_codecs->num))
goto error;
/* enumerate vendor codec capabilities */
@@ -214,11 +214,12 @@ void hci_read_supported_codecs_v2(struct hci_dev *hdev)
skb_pull(skb, sizeof(rp->status));
- std_codecs = (void *)skb->data;
+ std_codecs = skb_pull_data(skb, sizeof(*std_codecs));
+ if (!std_codecs)
+ goto error;
/* check for payload data length before accessing */
- if (skb->len < flex_array_size(std_codecs, codec, std_codecs->num)
- + sizeof(std_codecs->num))
+ if (skb->len < flex_array_size(std_codecs, codec, std_codecs->num))
goto error;
memset(&caps, 0, sizeof(caps));
@@ -229,15 +230,14 @@ void hci_read_supported_codecs_v2(struct hci_dev *hdev)
&caps);
}
- skb_pull(skb, flex_array_size(std_codecs, codec, std_codecs->num)
- + sizeof(std_codecs->num));
+ skb_pull(skb, flex_array_size(std_codecs, codec, std_codecs->num));
- vnd_codecs = (void *)skb->data;
+ vnd_codecs = skb_pull_data(skb, sizeof(*vnd_codecs));
+ if (!vnd_codecs)
+ goto error;
/* check for payload data length before accessing */
- if (skb->len <
- flex_array_size(vnd_codecs, codec, vnd_codecs->num)
- + sizeof(vnd_codecs->num))
+ if (skb->len < flex_array_size(vnd_codecs, codec, vnd_codecs->num))
goto error;
for (i = 0; i < vnd_codecs->num; i++) {
diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c
index d2ad9a5dd1ac..43be347ea91c 100644
--- a/net/bluetooth/hci_conn.c
+++ b/net/bluetooth/hci_conn.c
@@ -1937,6 +1937,8 @@ struct hci_conn *hci_bind_cis(struct hci_dev *hdev, bdaddr_t *dst,
cis->iso_qos.ucast.cis = BT_ISO_QOS_CIS_UNSET;
}
+ hci_conn_hold(cis);
+
if (cis->state == BT_CONNECTED)
return cis;
@@ -1978,7 +1980,6 @@ struct hci_conn *hci_bind_cis(struct hci_dev *hdev, bdaddr_t *dst,
return ERR_PTR(-EINVAL);
}
- hci_conn_hold(cis);
cis->state = BT_BOUND;
return cis;
@@ -2233,10 +2234,13 @@ struct hci_conn *hci_bind_bis(struct hci_dev *hdev, bdaddr_t *dst, __u8 sid,
parent = hci_conn_hash_lookup_big(hdev,
conn->iso_qos.bcast.big);
if (parent && parent != conn) {
+ hci_conn_hold(parent);
link = hci_conn_link(parent, conn);
hci_conn_drop(conn);
- if (!link)
+ if (!link) {
+ hci_conn_drop(parent);
return ERR_PTR(-ENOLINK);
+ }
}
return conn;
@@ -2341,6 +2345,12 @@ struct hci_conn *hci_connect_cis(struct hci_dev *hdev, bdaddr_t *dst,
return cis;
}
+ /* The existing link already owns the hold on its parent. */
+ if (cis->link) {
+ hci_conn_drop(le);
+ return cis;
+ }
+
link = hci_conn_link(le, cis);
hci_conn_drop(cis);
if (!link) {
diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c
index f68238406ad2..a85c77ed4a13 100644
--- a/net/bluetooth/hci_core.c
+++ b/net/bluetooth/hci_core.c
@@ -2731,6 +2731,7 @@ void hci_unregister_dev(struct hci_dev *hdev)
disable_work_sync(&hdev->error_reset);
disable_delayed_work_sync(&hdev->cmd_timer);
disable_delayed_work_sync(&hdev->ncmd_timer);
+ hci_devcd_shutdown(hdev);
hci_cmd_sync_clear(hdev);
@@ -3273,6 +3274,19 @@ static void hci_queue_acl(struct hci_chan *chan, struct sk_buff_head *queue,
spin_unlock_bh(&queue->lock);
}
+
+ bt_dev_dbg(hdev, "chan %p queued %d", chan, skb_queue_len(queue));
+}
+
+/* Queue hdev->tx_work, unless hdev->workqueue is being drained by
+ * hci_dev_close_sync(), which would otherwise WARN and drop the work.
+ */
+static void hci_sched_tx(struct hci_dev *hdev)
+{
+ rcu_read_lock();
+ if (!hci_dev_test_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE))
+ queue_work(hdev->workqueue, &hdev->tx_work);
+ rcu_read_unlock();
}
void hci_send_acl(struct hci_chan *chan, struct sk_buff *skb, __u16 flags)
@@ -3283,7 +3297,7 @@ void hci_send_acl(struct hci_chan *chan, struct sk_buff *skb, __u16 flags)
hci_queue_acl(chan, &chan->data_q, skb, flags);
- queue_work(hdev->workqueue, &hdev->tx_work);
+ hci_sched_tx(hdev);
}
/* Send SCO data */
@@ -3304,7 +3318,11 @@ void hci_send_sco(struct hci_conn *conn, struct sk_buff *skb)
hci_skb_pkt_type(skb) = HCI_SCODATA_PKT;
skb_queue_tail(&conn->data_q, skb);
- queue_work(hdev->workqueue, &hdev->tx_work);
+
+ bt_dev_dbg(hdev, "hcon %p queued %d", conn,
+ skb_queue_len(&conn->data_q));
+
+ hci_sched_tx(hdev);
}
/* Send ISO data */
@@ -3363,6 +3381,8 @@ static void hci_queue_iso(struct hci_conn *conn, struct sk_buff_head *queue,
__skb_queue_tail(queue, skb);
} while (list);
}
+
+ bt_dev_dbg(hdev, "hcon %p queued %d", conn, skb_queue_len(queue));
}
void hci_send_iso(struct hci_conn *conn, struct sk_buff *skb)
@@ -3373,7 +3393,7 @@ void hci_send_iso(struct hci_conn *conn, struct sk_buff *skb)
hci_queue_iso(conn, &conn->data_q, skb);
- queue_work(hdev->workqueue, &hdev->tx_work);
+ hci_sched_tx(hdev);
}
/* ---- HCI TX task (outgoing data) ---- */
diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
index fbfc962374a8..ae2e79b2eb81 100644
--- a/net/bluetooth/hci_sock.c
+++ b/net/bluetooth/hci_sock.c
@@ -166,6 +166,7 @@ static bool is_filtered_packet(struct sock *sk, struct sk_buff *skb)
{
struct hci_filter *flt;
int flt_type, flt_event;
+ u8 event;
/* Apply filter */
flt = &hci_pi(sk)->filter;
@@ -179,7 +180,11 @@ static bool is_filtered_packet(struct sock *sk, struct sk_buff *skb)
if (hci_skb_pkt_type(skb) != HCI_EVENT_PKT)
return false;
- flt_event = (*(__u8 *)skb->data & HCI_FLT_EVENT_BITS);
+ if (skb->len < 1)
+ return true;
+
+ event = *(__u8 *)skb->data;
+ flt_event = event & HCI_FLT_EVENT_BITS;
if (!hci_test_bit(flt_event, &flt->event_mask))
return true;
@@ -188,11 +193,17 @@ static bool is_filtered_packet(struct sock *sk, struct sk_buff *skb)
if (!flt->opcode)
return false;
- if (flt_event == HCI_EV_CMD_COMPLETE &&
+ if (event == HCI_EV_CMD_COMPLETE && skb->len < 5)
+ return true;
+
+ if (event == HCI_EV_CMD_COMPLETE &&
flt->opcode != get_unaligned((__le16 *)(skb->data + 3)))
return true;
- if (flt_event == HCI_EV_CMD_STATUS &&
+ if (event == HCI_EV_CMD_STATUS && skb->len < 6)
+ return true;
+
+ if (event == HCI_EV_CMD_STATUS &&
flt->opcode != get_unaligned((__le16 *)(skb->data + 4)))
return true;
@@ -1875,7 +1886,8 @@ static int hci_sock_sendmsg(struct socket *sock, struct msghdr *msg,
u16 ocf = hci_opcode_ocf(opcode);
if (((ogf > HCI_SFLT_MAX_OGF) ||
- !hci_test_bit(ocf & HCI_FLT_OCF_BITS,
+ (ocf > HCI_FLT_OCF_BITS) ||
+ !hci_test_bit(ocf,
&hci_sec_filter.ocf_mask[ogf])) &&
!capable(CAP_NET_RAW)) {
err = -EPERM;
diff --git a/net/bluetooth/hci_sync.c b/net/bluetooth/hci_sync.c
index cc2b0b85426f..78d9623f5fae 100644
--- a/net/bluetooth/hci_sync.c
+++ b/net/bluetooth/hci_sync.c
@@ -5480,7 +5480,9 @@ int hci_dev_close_sync(struct hci_dev *hdev)
memset(hdev->eir, 0, sizeof(hdev->eir));
memset(hdev->dev_class, 0, sizeof(hdev->dev_class));
bacpy(&hdev->random_addr, BDADDR_ANY);
+ hci_dev_lock(hdev);
hci_codec_list_clear(&hdev->local_codecs);
+ hci_dev_unlock(hdev);
hci_dev_put(hdev);
return err;
diff --git a/net/bluetooth/iso.c b/net/bluetooth/iso.c
index 6e1fac4b1cf6..3cf4ef291bf5 100644
--- a/net/bluetooth/iso.c
+++ b/net/bluetooth/iso.c
@@ -738,19 +738,24 @@ static void iso_sock_destruct(struct sock *sk)
skb_queue_purge(&sk->sk_write_queue);
}
-static void iso_sock_cleanup_listen(struct sock *parent)
+/* Close not yet accepted channels */
+static void iso_sock_flush_accept_q(struct sock *parent)
{
struct sock *sk;
- BT_DBG("parent %p", parent);
-
- /* Close not yet accepted channels */
while ((sk = bt_accept_dequeue(parent, NULL))) {
iso_sock_close(sk);
iso_sock_kill(sk);
/* Drop the reference handed back by bt_accept_dequeue(). */
sock_put(sk);
}
+}
+
+static void iso_sock_cleanup_listen(struct sock *parent)
+{
+ BT_DBG("parent %p", parent);
+
+ iso_sock_flush_accept_q(parent);
/* If listening socket has a hcon, properly disconnect it */
if (iso_pi(parent)->conn && iso_pi(parent)->conn->hcon) {
@@ -1524,6 +1529,13 @@ static int iso_sock_recvmsg(struct socket *sock, struct msghdr *msg,
switch (sk->sk_state) {
case BT_CONNECT2:
if (test_bit(BT_SK_PA_SYNC, &pi->flags)) {
+ /* Move to BT_LISTEN before requesting the BIG
+ * sync: the BIS connections are matched to a
+ * parent socket in BT_LISTEN state, and they
+ * may be notified before the request returns.
+ */
+ sk->sk_state = BT_LISTEN;
+
release_sock(sk);
err = iso_conn_big_sync(sk);
lock_sock(sk);
@@ -1532,12 +1544,20 @@ static int iso_sock_recvmsg(struct socket *sock, struct msghdr *msg,
* connection may have been torn down
* meanwhile and iso_chan_del() may have
* already moved the socket to BT_CLOSED.
- * Only move on to BT_LISTEN if the BIG sync
- * was actually started and nothing else has
- * changed the state.
+ * Only move back if the BIG sync could not be
+ * started and nothing else has changed the
+ * state.
*/
- if (!err && sk->sk_state == BT_CONNECT2)
- sk->sk_state = BT_LISTEN;
+ if (err && sk->sk_state == BT_LISTEN) {
+ /* Discard any child socket that may
+ * have been queued while the socket
+ * was in BT_LISTEN, as the cleanup of
+ * BT_CONNECT2 doesn't drain the
+ * accept queue.
+ */
+ iso_sock_flush_accept_q(sk);
+ sk->sk_state = BT_CONNECT2;
+ }
} else {
iso_conn_defer_accept(pi->conn->hcon);
sk->sk_state = BT_CONFIG;
@@ -1547,12 +1567,22 @@ static int iso_sock_recvmsg(struct socket *sock, struct msghdr *msg,
break;
case BT_CONNECTED:
if (test_bit(BT_SK_PA_SYNC, &iso_pi(sk)->flags)) {
+ /* As above, the BIS connections may be
+ * notified before the request returns.
+ */
+ sk->sk_state = BT_LISTEN;
+
release_sock(sk);
err = iso_conn_big_sync(sk);
lock_sock(sk);
- if (!err && sk->sk_state == BT_CONNECTED)
- sk->sk_state = BT_LISTEN;
+ if (err && sk->sk_state == BT_LISTEN) {
+ /* As above, don't leave any child
+ * socket behind in the accept queue.
+ */
+ iso_sock_flush_accept_q(sk);
+ sk->sk_state = BT_CONNECTED;
+ }
early_ret = true;
}
@@ -2013,6 +2043,7 @@ static void iso_conn_ready(struct iso_conn *conn)
BTPROTO_ISO, GFP_ATOMIC, 0);
if (!sk) {
release_sock(parent);
+ sock_put(parent);
return;
}
diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c
index 158719cc9e94..12d39f1e822b 100644
--- a/net/bluetooth/l2cap_core.c
+++ b/net/bluetooth/l2cap_core.c
@@ -6681,9 +6681,17 @@ static int l2cap_stream_rx(struct l2cap_chan *chan, struct l2cap_ctrl *control,
static int l2cap_data_rcv(struct l2cap_chan *chan, struct sk_buff *skb)
{
struct l2cap_ctrl *control = &bt_cb(skb)->l2cap;
- u16 len;
+ u16 len, min_len;
u8 event;
+ min_len = test_bit(FLAG_EXT_CTRL, &chan->flags) ?
+ L2CAP_EXT_CTRL_SIZE : L2CAP_ENH_CTRL_SIZE;
+ if (chan->fcs == L2CAP_FCS_CRC16)
+ min_len += L2CAP_FCS_SIZE;
+
+ if (skb->len < min_len)
+ goto drop;
+
__unpack_control(chan, skb);
len = skb->len;
diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c
index cdb2425a16a2..d55bfd102eef 100644
--- a/net/bluetooth/mgmt.c
+++ b/net/bluetooth/mgmt.c
@@ -498,13 +498,9 @@ static int read_unconf_index_list(struct sock *sk, struct hci_dev *hdev,
read_lock(&hci_dev_list_lock);
- count = 0;
- list_for_each_entry(d, &hci_dev_list, list) {
- if (hci_dev_test_flag(d, HCI_UNCONFIGURED))
- count++;
- }
+ count = list_count_nodes(&hci_dev_list);
- rp_len = sizeof(*rp) + (2 * count);
+ rp_len = sizeof(*rp) + (sizeof(__le16) * count);
rp = kmalloc(rp_len, GFP_ATOMIC);
if (!rp) {
read_unlock(&hci_dev_list_lock);
@@ -2290,6 +2286,8 @@ static void mesh_send_start_complete(struct hci_dev *hdev, void *data, int err)
hci_dev_clear_flag(hdev, HCI_MESH_SENDING);
/* Send Complete Error Code for handle */
mesh_send_complete(hdev, mesh_tx, false);
+ if (err != -ECANCELED)
+ mesh_next(hdev, NULL, 0);
return;
}
@@ -2399,19 +2397,28 @@ static int send_cancel(struct hci_dev *hdev, void *data)
do {
mesh_tx = mgmt_mesh_next(hdev, cmd->sk);
- if (mesh_tx)
- mesh_send_complete(hdev, mesh_tx, false);
+ if (mesh_tx) {
+ if (!hci_cmd_sync_dequeue(hdev, mesh_send_sync,
+ mesh_tx, NULL))
+ mesh_send_complete(hdev, mesh_tx, false);
+ }
} while (mesh_tx);
} else {
mesh_tx = mgmt_mesh_find(hdev, cancel->handle);
- if (mesh_tx && mesh_tx->sk == cmd->sk)
- mesh_send_complete(hdev, mesh_tx, false);
+ if (mesh_tx && mesh_tx->sk == cmd->sk) {
+ if (!hci_cmd_sync_dequeue(hdev, mesh_send_sync,
+ mesh_tx, NULL))
+ mesh_send_complete(hdev, mesh_tx, false);
+ }
}
mgmt_cmd_complete(cmd->sk, hdev->id, MGMT_OP_MESH_SEND_CANCEL,
0, NULL, 0);
+ if (!hci_dev_test_flag(hdev, HCI_MESH_SENDING))
+ mesh_next(hdev, NULL, 0);
+
return 0;
}
diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c
index 5dc575833371..6c57f5a766c1 100644
--- a/net/bluetooth/rfcomm/core.c
+++ b/net/bluetooth/rfcomm/core.c
@@ -1817,7 +1817,8 @@ static struct rfcomm_session *rfcomm_recv_frame(struct rfcomm_session *s,
return s;
}
- if (skb->len < sizeof(*hdr) + 1) {
+ if (skb->len < sizeof(*hdr) + 1 ||
+ (!__test_ea(hdr->len) && skb->len < sizeof(*hdr) + 2)) {
kfree_skb(skb);
return s;
}
diff --git a/net/bluetooth/rfcomm/sock.c b/net/bluetooth/rfcomm/sock.c
index 2286efef62f5..ddf2153aa832 100644
--- a/net/bluetooth/rfcomm/sock.c
+++ b/net/bluetooth/rfcomm/sock.c
@@ -243,9 +243,7 @@ static void __rfcomm_sock_close(struct sock *sk)
*/
static void rfcomm_sock_close(struct sock *sk)
{
- lock_sock(sk);
__rfcomm_sock_close(sk);
- release_sock(sk);
}
static void rfcomm_sock_init(struct sock *sk, struct sock *parent)
@@ -787,8 +785,10 @@ static int rfcomm_sock_getsockopt_old(struct socket *sock, int optname, char __u
break;
case RFCOMM_CONNINFO:
- if (sk->sk_state != BT_CONNECTED &&
- !rfcomm_pi(sk)->dlc->defer_setup) {
+ if ((sk->sk_state != BT_CONNECTED &&
+ !(sk->sk_state == BT_CONNECT2 &&
+ rfcomm_pi(sk)->dlc->defer_setup)) ||
+ !rfcomm_pi(sk)->dlc->session) {
err = -ENOTCONN;
break;
}
@@ -902,6 +902,7 @@ static int rfcomm_sock_compat_ioctl(struct socket *sock, unsigned int cmd, unsig
static int rfcomm_sock_shutdown(struct socket *sock, int how)
{
struct sock *sk = sock->sk;
+ bool cleanup_listen = false;
int err = 0;
BT_DBG("sock %p, sk %p", sock, sk);
@@ -912,9 +913,17 @@ static int rfcomm_sock_shutdown(struct socket *sock, int how)
lock_sock(sk);
if (!sk->sk_shutdown) {
sk->sk_shutdown = SHUTDOWN_MASK;
+ if (sk->sk_state == BT_LISTEN) {
+ /* Block new children before cleaning up without sk lock. */
+ sk->sk_state = BT_CLOSED;
+ cleanup_listen = true;
+ }
release_sock(sk);
- __rfcomm_sock_close(sk);
+ if (cleanup_listen)
+ rfcomm_sock_cleanup_listen(sk);
+ else
+ __rfcomm_sock_close(sk);
lock_sock(sk);
if (sock_flag(sk, SOCK_LINGER) && sk->sk_lingertime &&
diff --git a/net/bluetooth/smp.c b/net/bluetooth/smp.c
index 9e9e9e293309..078d5a19a846 100644
--- a/net/bluetooth/smp.c
+++ b/net/bluetooth/smp.c
@@ -2301,6 +2301,23 @@ static u8 smp_cmd_security_req(struct l2cap_conn *conn, struct sk_buff *skb)
bt_dev_dbg(hdev, "conn %p", conn);
+ /* SMP over BR/EDR only covers cross-transport key derivation; the
+ * Security Request procedure has no BR/EDR counterpart. Reject it
+ * here, otherwise smp_ltk_encrypt() finds the peer's LE LTK
+ * (ADDR_LE_DEV_PUBLIC and BDADDR_BREDR are both 0) and issues
+ * HCI_OP_LE_START_ENC on the ACL handle, which the controller
+ * rejects and hci_cs_le_start_enc() turns into a disconnect. Reply
+ * without smp_failure(): this is not an authentication failure, and
+ * MGMT_EV_AUTH_FAILED would make bluetoothd drop the device.
+ */
+ if (hcon->type != LE_LINK) {
+ u8 reason = SMP_CMD_NOTSUPP;
+
+ smp_send_cmd(conn, SMP_CMD_PAIRING_FAIL, sizeof(reason),
+ &reason);
+ return 0;
+ }
+
if (skb->len < sizeof(*rp))
return SMP_INVALID_PARAMS;
diff --git a/net/bridge/br_cfm.c b/net/bridge/br_cfm.c
index d76a0440e2b8..534b033a7f72 100644
--- a/net/bridge/br_cfm.c
+++ b/net/bridge/br_cfm.c
@@ -367,7 +367,7 @@ static u32 ccm_tlv_extract(struct sk_buff *skb, u32 index,
}
/* note: already called with rcu_read_lock */
-static int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb)
+int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb)
{
u32 mdlevel, interval, size, index, max;
const struct br_cfm_common_hdr *hdr;
@@ -489,11 +489,6 @@ static int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb)
return 1;
}
-static struct br_frame_type cfm_frame_type __read_mostly = {
- .type = cpu_to_be16(ETH_P_CFM),
- .frame_handler = br_cfm_frame_rx,
-};
-
int br_cfm_mep_create(struct net_bridge *br,
const u32 instance,
struct br_cfm_mep_create *const create,
@@ -559,7 +554,7 @@ int br_cfm_mep_create(struct net_bridge *br,
INIT_DELAYED_WORK(&mep->ccm_tx_dwork, ccm_tx_work_expired);
if (hlist_empty(&br->mep_list))
- br_add_frame(br, &cfm_frame_type);
+ br_opt_toggle(br, BROPT_CFM_ENABLED, true);
hlist_add_tail_rcu(&mep->head, &br->mep_list);
@@ -588,7 +583,7 @@ static void mep_delete_implementation(struct net_bridge *br,
kfree_rcu(mep, rcu);
if (hlist_empty(&br->mep_list))
- br_del_frame(br, &cfm_frame_type);
+ br_opt_toggle(br, BROPT_CFM_ENABLED, false);
}
int br_cfm_mep_delete(struct net_bridge *br,
diff --git a/net/bridge/br_device.c b/net/bridge/br_device.c
index 6b7021cebb52..446761da4e54 100644
--- a/net/bridge/br_device.c
+++ b/net/bridge/br_device.c
@@ -501,7 +501,6 @@ void br_dev_setup(struct net_device *dev)
spin_lock_init(&br->lock);
INIT_LIST_HEAD(&br->port_list);
INIT_HLIST_HEAD(&br->fdb_list);
- INIT_HLIST_HEAD(&br->frame_type_list);
#if IS_ENABLED(CONFIG_BRIDGE_MRP)
INIT_HLIST_HEAD(&br->mrp_list);
#endif
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index ab4758ec3e54..17a596098a06 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -303,17 +303,25 @@ static int nf_hook_bridge_pre(struct sk_buff *skb, struct sk_buff **pskb)
return RX_HANDLER_CONSUMED;
}
+#define BR_CFM_MRP_OPTS \
+ ((IS_ENABLED(CONFIG_BRIDGE_CFM) ? BIT(BROPT_CFM_ENABLED) : 0UL) | \
+ (IS_ENABLED(CONFIG_BRIDGE_MRP) ? BIT(BROPT_MRP_ENABLED) : 0UL))
+
/* Return 0 if the frame was not processed otherwise 1
* note: already called with rcu_read_lock
*/
static int br_process_frame_type(struct net_bridge_port *p,
struct sk_buff *skb)
{
- struct br_frame_type *tmp;
+ struct net_bridge *br = p->br;
+
+ if (skb->protocol == htons(ETH_P_CFM) &&
+ br_opt_get(br, BROPT_CFM_ENABLED))
+ return br_cfm_frame_rx(p, skb);
- hlist_for_each_entry_rcu(tmp, &p->br->frame_type_list, list)
- if (unlikely(tmp->type == skb->protocol))
- return tmp->frame_handler(p, skb);
+ if (skb->protocol == htons(ETH_P_MRP) &&
+ br_opt_get(br, BROPT_MRP_ENABLED))
+ return br_mrp_process(p, skb);
return 0;
}
@@ -407,7 +415,8 @@ static rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
}
}
- if (unlikely(br_process_frame_type(p, skb)))
+ if (unlikely((READ_ONCE(p->br->options) & BR_CFM_MRP_OPTS) &&
+ br_process_frame_type(p, skb)))
return RX_HANDLER_PASS;
forward:
@@ -448,19 +457,3 @@ rx_handler_func_t *br_get_rx_handler(const struct net_device *dev)
return br_handle_frame;
}
-
-void br_add_frame(struct net_bridge *br, struct br_frame_type *ft)
-{
- hlist_add_head_rcu(&ft->list, &br->frame_type_list);
-}
-
-void br_del_frame(struct net_bridge *br, struct br_frame_type *ft)
-{
- struct br_frame_type *tmp;
-
- hlist_for_each_entry(tmp, &br->frame_type_list, list)
- if (ft == tmp) {
- hlist_del_rcu(&ft->list);
- return;
- }
-}
diff --git a/net/bridge/br_mdb.c b/net/bridge/br_mdb.c
index 7e1ad229e133..856248b2a7e2 100644
--- a/net/bridge/br_mdb.c
+++ b/net/bridge/br_mdb.c
@@ -1505,6 +1505,8 @@ static void br_mdb_flush_pgs(struct net_bridge *br,
}
br_multicast_del_pg(mp, p, pp);
+ /* br_multicast_del_pg() can remove other groups from this list. */
+ pp = &mp->ports;
}
}
diff --git a/net/bridge/br_mrp.c b/net/bridge/br_mrp.c
index a837ff7d6f36..79d6bc62179e 100644
--- a/net/bridge/br_mrp.c
+++ b/net/bridge/br_mrp.c
@@ -6,13 +6,6 @@
static const u8 mrp_test_dmac[ETH_ALEN] = { 0x1, 0x15, 0x4e, 0x0, 0x0, 0x1 };
static const u8 mrp_in_test_dmac[ETH_ALEN] = { 0x1, 0x15, 0x4e, 0x0, 0x0, 0x3 };
-static int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb);
-
-static struct br_frame_type mrp_frame_type __read_mostly = {
- .type = cpu_to_be16(ETH_P_MRP),
- .frame_handler = br_mrp_process,
-};
-
static bool br_mrp_is_ring_port(struct net_bridge_port *p_port,
struct net_bridge_port *s_port,
struct net_bridge_port *port)
@@ -486,7 +479,7 @@ static void br_mrp_del_impl(struct net_bridge *br, struct br_mrp *mrp)
kfree_rcu(mrp, rcu);
if (hlist_empty(&br->mrp_list))
- br_del_frame(br, &mrp_frame_type);
+ br_opt_toggle(br, BROPT_MRP_ENABLED, false);
}
/* Adds a new MRP instance.
@@ -536,7 +529,7 @@ int br_mrp_add(struct net_bridge *br, struct br_mrp_instance *instance)
rcu_assign_pointer(mrp->s_port, p);
if (hlist_empty(&br->mrp_list))
- br_add_frame(br, &mrp_frame_type);
+ br_opt_toggle(br, BROPT_MRP_ENABLED, true);
INIT_DELAYED_WORK(&mrp->test_work, br_mrp_test_work_expired);
INIT_DELAYED_WORK(&mrp->in_test_work, br_mrp_in_test_work_expired);
@@ -1241,7 +1234,7 @@ static int br_mrp_rcv(struct net_bridge_port *p,
* normal forwarding.
* note: already called with rcu_read_lock
*/
-static int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb)
+int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb)
{
/* If there is no MRP instance do normal forwarding */
if (likely(!(p->flags & BR_MRP_AWARE)))
diff --git a/net/bridge/br_mst.c b/net/bridge/br_mst.c
index 43a300ae6bfa..1654efd3045b 100644
--- a/net/bridge/br_mst.c
+++ b/net/bridge/br_mst.c
@@ -107,21 +107,24 @@ int br_mst_set_state(struct net_bridge_port *p, u16 msti, u8 state,
struct net_bridge_vlan *v;
int err = 0;
- rcu_read_lock();
- vg = nbp_vlan_group_rcu(p);
- if (!vg)
- goto out;
-
/* MSTI 0 (CST) state changes are notified via the regular
- * SWITCHDEV_ATTR_ID_PORT_STP_STATE.
+ * SWITCHDEV_ATTR_ID_PORT_STP_STATE. All other MSTIs are handled via
+ * netlink with RTNL held
*/
if (msti) {
+ ASSERT_RTNL();
+
err = switchdev_port_attr_set(p->dev, &attr, extack);
if (err && err != -EOPNOTSUPP)
goto out;
+ err = 0;
}
- err = 0;
+ rcu_read_lock();
+ vg = nbp_vlan_group_rcu(p);
+ if (!vg)
+ goto out_rcu_unlock;
+
list_for_each_entry_rcu(v, &vg->vlan_list, vlist) {
if (v->brvlan->msti != msti)
continue;
@@ -129,8 +132,9 @@ int br_mst_set_state(struct net_bridge_port *p, u16 msti, u8 state,
br_mst_vlan_set_state(vg, v, state);
}
-out:
+out_rcu_unlock:
rcu_read_unlock();
+out:
return err;
}
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index 3854ea1a5154..c2d4e127c799 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -486,12 +486,13 @@ enum net_bridge_opts {
BROPT_VLAN_BRIDGE_BINDING,
BROPT_MCAST_VLAN_SNOOPING_ENABLED,
BROPT_MST_ENABLED,
+ BROPT_CFM_ENABLED,
+ BROPT_MRP_ENABLED,
};
struct net_bridge {
spinlock_t lock;
spinlock_t hash_lock;
- struct hlist_head frame_type_list;
struct net_device *dev;
unsigned long options;
/* These fields are accessed on each packet */
@@ -919,16 +920,6 @@ int nbp_backup_change(struct net_bridge_port *p, struct net_device *backup_dev);
int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb);
rx_handler_func_t *br_get_rx_handler(const struct net_device *dev);
-struct br_frame_type {
- __be16 type;
- int (*frame_handler)(struct net_bridge_port *port,
- struct sk_buff *skb);
- struct hlist_node list;
-};
-
-void br_add_frame(struct net_bridge *br, struct br_frame_type *ft);
-void br_del_frame(struct net_bridge *br, struct br_frame_type *ft);
-
static inline bool br_rx_handler_check_rcu(const struct net_device *dev)
{
return rcu_dereference(dev->rx_handler) == br_get_rx_handler(dev);
@@ -2033,6 +2024,7 @@ int br_mrp_parse(struct net_bridge *br, struct net_bridge_port *p,
bool br_mrp_enabled(struct net_bridge *br);
void br_mrp_port_del(struct net_bridge *br, struct net_bridge_port *p);
int br_mrp_fill_info(struct sk_buff *skb, struct net_bridge *br);
+int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb);
#else
static inline int br_mrp_parse(struct net_bridge *br, struct net_bridge_port *p,
struct nlattr *attr, int cmd,
@@ -2056,6 +2048,11 @@ static inline int br_mrp_fill_info(struct sk_buff *skb, struct net_bridge *br)
return 0;
}
+static inline int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb)
+{
+ return 0;
+}
+
#endif
/* br_cfm.c */
@@ -2064,6 +2061,7 @@ int br_cfm_parse(struct net_bridge *br, struct net_bridge_port *p,
struct nlattr *attr, int cmd, struct netlink_ext_ack *extack);
bool br_cfm_created(struct net_bridge *br);
void br_cfm_port_del(struct net_bridge *br, struct net_bridge_port *p);
+int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb);
int br_cfm_config_fill_info(struct sk_buff *skb, struct net_bridge *br);
int br_cfm_status_fill_info(struct sk_buff *skb,
struct net_bridge *br,
@@ -2088,6 +2086,12 @@ static inline void br_cfm_port_del(struct net_bridge *br,
{
}
+static inline int br_cfm_frame_rx(struct net_bridge_port *port,
+ struct sk_buff *skb)
+{
+ return 0;
+}
+
static inline int br_cfm_config_fill_info(struct sk_buff *skb, struct net_bridge *br)
{
return -EOPNOTSUPP;
diff --git a/net/bridge/br_stp_bpdu.c b/net/bridge/br_stp_bpdu.c
index 7895489ac6fe..e44e448b1157 100644
--- a/net/bridge/br_stp_bpdu.c
+++ b/net/bridge/br_stp_bpdu.c
@@ -52,7 +52,10 @@ static void br_send_bpdu(struct net_bridge_port *p,
LLC_SAP_BSPAN, LLC_PDU_CMD);
llc_pdu_init_as_ui_cmd(skb);
- llc_mac_hdr_init(skb, p->dev->dev_addr, p->br->group_addr);
+ if (llc_mac_hdr_init(skb, p->dev->dev_addr, p->br->group_addr)) {
+ kfree_skb(skb);
+ return;
+ }
skb_reset_mac_header(skb);
diff --git a/net/core/dev.c b/net/core/dev.c
index d4f0db320848..6c63867ff76d 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2686,7 +2686,7 @@ int __netif_set_xps_queue(struct net_device *dev, const unsigned long *mask,
dev = netdev_get_tx_queue(dev, index)->sb_dev ? : dev;
tc = netdev_txq_to_tc(dev, index);
- if (tc < 0)
+ if (tc < 0 || tc >= num_tc)
return -EINVAL;
}
diff --git a/net/core/devmem.c b/net/core/devmem.c
index 55f27a9ed121..1763af1901cb 100644
--- a/net/core/devmem.c
+++ b/net/core/devmem.c
@@ -343,6 +343,7 @@ int mp_dmabuf_devmem_init(struct page_pool *pool)
* dma_sync_for_cpu/device. Force disable dma_sync.
*/
pool->dma_sync = false;
+ pool->dma_sync_for_cpu = false;
if (pool->p.order != 0)
return -E2BIG;
diff --git a/net/core/drop_monitor.c b/net/core/drop_monitor.c
index 308e0fa8f723..1ba281bef21e 100644
--- a/net/core/drop_monitor.c
+++ b/net/core/drop_monitor.c
@@ -141,9 +141,8 @@ static struct sk_buff *reset_per_cpu_data(struct per_cpu_dm_data *data)
al = sizeof(struct net_dm_alert_msg);
al += dm_hit_limit * sizeof(struct net_dm_drop_point);
- al += sizeof(struct nlattr);
- skb = genlmsg_new(al, GFP_KERNEL);
+ skb = genlmsg_new(nla_total_size(al), GFP_KERNEL);
if (!skb)
goto err;
@@ -1174,6 +1173,7 @@ static int net_dm_trace_on_set(struct netlink_ext_ack *extack)
err_unregister_trace:
unregister_trace_kfree_skb(ops->kfree_skb_probe, NULL);
+ tracepoint_synchronize_unregister();
err_module_put:
for_each_possible_cpu(cpu) {
struct per_cpu_dm_data *data = &per_cpu(dm_cpu_data, cpu);
diff --git a/net/core/filter.c b/net/core/filter.c
index e0f757ffcfc5..a680cf27ad82 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -3833,12 +3833,6 @@ static u32 __bpf_skb_min_len(const struct sk_buff *skb)
if (offset > 0)
min_len = offset;
}
- if (skb->ip_summed == CHECKSUM_PARTIAL) {
- offset = skb_checksum_start_offset(skb) +
- skb->csum_offset + sizeof(__sum16);
- if (offset > 0)
- min_len = offset;
- }
return min_len;
}
@@ -3855,6 +3849,11 @@ static int bpf_skb_grow_rcsum(struct sk_buff *skb, unsigned int new_len)
static int bpf_skb_trim_rcsum(struct sk_buff *skb, unsigned int new_len)
{
+ if (skb->ip_summed == CHECKSUM_PARTIAL &&
+ new_len < skb_checksum_start_offset(skb) + skb->csum_offset +
+ sizeof(__sum16))
+ skb->ip_summed = CHECKSUM_NONE;
+
return __skb_trim_rcsum(skb, new_len);
}
@@ -8667,6 +8666,8 @@ static const struct bpf_func_proto *
lwt_seg6local_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
+ case BPF_FUNC_skb_pull_data:
+ return NULL;
#if IS_ENABLED(CONFIG_IPV6_SEG6_BPF)
case BPF_FUNC_lwt_seg6_store_bytes:
return &bpf_lwt_seg6_store_bytes_proto;
@@ -10186,11 +10187,12 @@ u32 bpf_sock_convert_ctx_access(enum bpf_access_type type,
target_size));
*insn++ = BPF_JMP_IMM(BPF_JNE, si->dst_reg, NO_QUEUE_MAPPING,
1);
- *insn++ = BPF_MOV64_IMM(si->dst_reg, -1);
+ *insn++ = BPF_MOV32_IMM(si->dst_reg, -1);
#else
- *insn++ = BPF_MOV64_IMM(si->dst_reg, -1);
- *target_size = 2;
+ *insn++ = BPF_MOV32_IMM(si->dst_reg, -1);
#endif
+ *target_size = sizeof_field(struct bpf_sock, rx_queue_mapping);
+
break;
}
@@ -10730,18 +10732,7 @@ static u32 sock_ops_convert_ctx_access(enum bpf_access_type type,
break;
case offsetof(struct bpf_sock_ops, rtt_min):
- BUILD_BUG_ON(sizeof_field(struct tcp_sock, rtt_min) !=
- sizeof(struct minmax));
- BUILD_BUG_ON(sizeof(struct minmax) <
- sizeof(struct minmax_sample));
-
- *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(
- struct bpf_sock_ops_kern, sk),
- si->dst_reg, si->src_reg,
- offsetof(struct bpf_sock_ops_kern, sk));
- *insn++ = BPF_LDX_MEM(BPF_W, si->dst_reg, si->dst_reg,
- offsetof(struct tcp_sock, rtt_min) +
- sizeof_field(struct minmax_sample, t));
+ SOCK_OPS_GET_FIELD(rtt_min, rtt_min.s[0].v, struct tcp_sock);
break;
case offsetof(struct bpf_sock_ops, bpf_sock_ops_cb_flags):
@@ -12261,8 +12252,9 @@ __bpf_kfunc_start_defs();
* @sock: Pointer to socket to be destroyed
*
* Return:
- * On error, may return EPROTONOSUPPORT, EINVAL.
- * EPROTONOSUPPORT if protocol specific destroy handler is not supported.
+ * On error, may return EOPNOTSUPP, or whatever the protocol specific
+ * destroy handler returns.
+ * EOPNOTSUPP if protocol specific destroy handler is not supported.
* 0 otherwise
*/
__bpf_kfunc int bpf_sock_destroy(struct sock_common *sock)
@@ -12274,8 +12266,12 @@ __bpf_kfunc int bpf_sock_destroy(struct sock_common *sock)
* Supporting protocols will need to acquire sock lock in the BPF context
* prior to invoking this kfunc.
*/
- if (!sk->sk_prot->diag_destroy || (sk->sk_protocol != IPPROTO_TCP &&
- sk->sk_protocol != IPPROTO_UDP))
+ if (!sk->sk_prot->diag_destroy)
+ return -EOPNOTSUPP;
+
+ if (sk_fullsock(sk) &&
+ sk->sk_protocol != IPPROTO_TCP &&
+ sk->sk_protocol != IPPROTO_UDP)
return -EOPNOTSUPP;
return sk->sk_prot->diag_destroy(sk, ECONNABORTED);
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index af4c7e62fe8d..ce4569a56887 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -203,6 +203,7 @@ static int page_pool_init(struct page_pool *pool,
memcpy(&pool->slow, ¶ms->slow, sizeof(pool->slow));
pool->cpuid = cpuid;
+ pool->dma_sync_for_cpu = true;
/* Validate only known flags were used */
if (pool->slow.flags & ~PP_FLAG_ALL)
@@ -673,7 +674,7 @@ static noinline netmem_ref __page_pool_alloc_pages_slow(struct page_pool *pool,
/* For using page_pool replace: alloc_pages() API calls, but provide
* synchronization guarantee for allocation side.
*/
-netmem_ref page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp)
+netmem_ref page_pool_alloc_netmems(struct page_pool *pool, gfp_t gfp)
{
netmem_ref netmem;
@@ -689,11 +690,11 @@ netmem_ref page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp)
netmem = __page_pool_alloc_pages_slow(pool, gfp);
return netmem;
}
-EXPORT_SYMBOL(page_pool_alloc_netmem);
+EXPORT_SYMBOL(page_pool_alloc_netmems);
struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp)
{
- return netmem_to_page(page_pool_alloc_netmem(pool, gfp));
+ return netmem_to_page(page_pool_alloc_netmems(pool, gfp));
}
EXPORT_SYMBOL(page_pool_alloc_pages);
ALLOW_ERROR_INJECTION(page_pool_alloc_pages, NULL);
@@ -1051,7 +1052,7 @@ netmem_ref page_pool_alloc_frag_netmem(struct page_pool *pool,
}
if (!netmem) {
- netmem = page_pool_alloc_netmem(pool, gfp);
+ netmem = page_pool_alloc_netmems(pool, gfp);
if (unlikely(!netmem)) {
pool->frag_page = 0;
return 0;
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 449c84fa7353..623d9d10a208 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -5841,7 +5841,8 @@ static int skb_checksum_setup_ipv6(struct sk_buff *skb, bool recalculate)
err = skb_maybe_pull_tail(skb,
off +
sizeof(struct ipv6_opt_hdr),
- MAX_IPV6_HDR_LEN);
+ off +
+ sizeof(struct ipv6_opt_hdr));
if (err < 0)
goto out;
@@ -5856,7 +5857,8 @@ static int skb_checksum_setup_ipv6(struct sk_buff *skb, bool recalculate)
err = skb_maybe_pull_tail(skb,
off +
sizeof(struct ip_auth_hdr),
- MAX_IPV6_HDR_LEN);
+ off +
+ sizeof(struct ip_auth_hdr));
if (err < 0)
goto out;
@@ -5871,7 +5873,8 @@ static int skb_checksum_setup_ipv6(struct sk_buff *skb, bool recalculate)
err = skb_maybe_pull_tail(skb,
off +
sizeof(struct frag_hdr),
- MAX_IPV6_HDR_LEN);
+ off +
+ sizeof(struct frag_hdr));
if (err < 0)
goto out;
@@ -6696,6 +6699,34 @@ struct sk_buff *alloc_skb_with_frags(unsigned long header_len,
}
EXPORT_SYMBOL(alloc_skb_with_frags);
+/* pskb_carve_inside_header() and pskb_carve_inside_nonlinear()
+ * remove the first bytes of a packet and reallocate skb->head.
+ *
+ * Whatever headers were present before the operation are gone,
+ * we must not leave stale offsets, otherwise users of this skb
+ * (skb_dump(), drop_monitor, taps, ...) would read or pull garbage.
+ */
+static void skb_carve_reset_headers(struct sk_buff *skb)
+{
+ skb_unset_mac_header(skb);
+ skb_unset_transport_header(skb);
+ skb_reset_network_header(skb);
+ skb->mac_len = 0;
+
+ /* Inner offsets have no "unset" marker, zero them so that
+ * skb_inner_network_header_was_set() becomes false and no
+ * consumer mistakes them for a real (and long gone) header.
+ */
+ skb->inner_mac_header = 0;
+ skb->inner_network_header = 0;
+ skb->inner_transport_header = 0;
+ skb->inner_protocol = 0;
+ skb->encapsulation = 0;
+
+ if (skb->ip_summed == CHECKSUM_PARTIAL)
+ skb->ip_summed = CHECKSUM_NONE;
+}
+
/* carve out the first off bytes from skb when off < headlen */
static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off,
const int headlen, gfp_t gfp_mask)
@@ -6751,7 +6782,7 @@ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off,
skb->head_frag = 0;
skb_set_end_offset(skb, size);
skb_set_tail_pointer(skb, skb_headlen(skb));
- skb_headers_offset_update(skb, 0);
+ skb_carve_reset_headers(skb);
skb->cloned = 0;
skb->hdr_len = 0;
skb->nohdr = 0;
@@ -6892,7 +6923,7 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off,
skb->data = data;
skb_set_end_offset(skb, size);
skb_reset_tail_pointer(skb);
- skb_headers_offset_update(skb, 0);
+ skb_carve_reset_headers(skb);
skb->cloned = 0;
skb->hdr_len = 0;
skb->nohdr = 0;
diff --git a/net/core/skmsg.c b/net/core/skmsg.c
index 7e68b4fdbfce..6eca57da3cff 100644
--- a/net/core/skmsg.c
+++ b/net/core/skmsg.c
@@ -1035,6 +1035,10 @@ static int sk_psock_verdict_apply(struct sk_psock *psock, struct sk_buff *skb,
int err = 0;
u32 len, off;
+ if (verdict == __SK_REDIRECT && skb_bpf_ingress(skb) &&
+ skb_bpf_redirect_fetch(skb) == psock->sk)
+ verdict = __SK_PASS;
+
switch (verdict) {
case __SK_PASS:
err = -EIO;
diff --git a/net/core/sock.c b/net/core/sock.c
index e8b03cf3a428..ef8faa6bddf9 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2837,6 +2837,22 @@ void *sock_kmalloc(struct sock *sk, int size, gfp_t priority)
}
EXPORT_SYMBOL(sock_kmalloc);
+/*
+ * Duplicate the input "src" memory block using the socket's
+ * option memory buffer.
+ */
+void *sock_kmemdup(struct sock *sk, const void *src,
+ int size, gfp_t priority)
+{
+ void *mem;
+
+ mem = sock_kmalloc(sk, size, priority);
+ if (mem)
+ memcpy(mem, src, size);
+ return mem;
+}
+EXPORT_SYMBOL(sock_kmemdup);
+
/* Free an option memory block. Note, we actually want the inline
* here as this allows gcc to detect the nullify and fold away the
* condition entirely.
@@ -3724,7 +3740,14 @@ int sock_gettstamp(struct socket *sock, void __user *userstamp,
struct sock *sk = sock->sk;
struct timespec64 ts;
- sock_enable_timestamp(sk, SOCK_TIMESTAMP);
+ /* sk->sk_flags must only be changed under the socket lock,
+ * because sock_set_flag() uses non atomic operations.
+ */
+ if (!sock_flag(sk, SOCK_TIMESTAMP)) {
+ lock_sock(sk);
+ sock_enable_timestamp(sk, SOCK_TIMESTAMP);
+ release_sock(sk);
+ }
ts = ktime_to_timespec64(sock_read_timestamp(sk));
if (ts.tv_sec == -1)
return -ENOENT;
diff --git a/net/core/sock_map.c b/net/core/sock_map.c
index 6440ce16fc62..be4a4f8f346c 100644
--- a/net/core/sock_map.c
+++ b/net/core/sock_map.c
@@ -41,6 +41,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr)
struct bpf_stab *stab;
if (attr->max_entries == 0 ||
+ attr->max_entries > INT_MAX ||
attr->key_size != 4 ||
(attr->value_size != sizeof(u32) &&
attr->value_size != sizeof(u64)) ||
diff --git a/net/dccp/ipv6.c b/net/dccp/ipv6.c
index c1db85366747..3c43dd8561f8 100644
--- a/net/dccp/ipv6.c
+++ b/net/dccp/ipv6.c
@@ -456,7 +456,7 @@ static struct sock *dccp_v6_request_recv_sock(const struct sock *sk,
newnp->opt = NULL;
newnp->ipv6_mc_list = NULL;
newnp->ipv6_ac_list = NULL;
- newnp->ipv6_fl_list = NULL;
+ newinet->ipv6_fl_list = NULL;
newnp->mcast_oif = inet_iif(skb);
newnp->mcast_hops = ip_hdr(skb)->ttl;
@@ -523,7 +523,7 @@ static struct sock *dccp_v6_request_recv_sock(const struct sock *sk,
newnp->ipv6_mc_list = NULL;
newnp->ipv6_ac_list = NULL;
- newnp->ipv6_fl_list = NULL;
+ newinet->ipv6_fl_list = NULL;
newnp->pktoptions = NULL;
newnp->opt = NULL;
newnp->mcast_oif = inet6_iif(skb);
diff --git a/net/dccp/timer.c b/net/dccp/timer.c
index a4cfb47b60e5..fc1d88d6bc3f 100644
--- a/net/dccp/timer.c
+++ b/net/dccp/timer.c
@@ -160,7 +160,9 @@ static void dccp_write_timer(struct timer_list *t)
static void dccp_keepalive_timer(struct timer_list *t)
{
- struct sock *sk = from_timer(sk, t, sk_timer);
+ struct inet_connection_sock *icsk =
+ from_timer(icsk, t, icsk_keepalive_timer);
+ struct sock *sk = &icsk->icsk_inet.sk;
pr_err("dccp should not use a keepalive timer !\n");
sock_put(sk);
diff --git a/net/ipv4/arp.c b/net/ipv4/arp.c
index 7822b2144514..13c6c9a3a9c2 100644
--- a/net/ipv4/arp.c
+++ b/net/ipv4/arp.c
@@ -1276,6 +1276,7 @@ int arp_ioctl(struct net *net, unsigned int cmd, void __user *arg)
err = copy_from_user(&r, arg, sizeof(struct arpreq));
if (err)
return -EFAULT;
+ r.arp_dev[IFNAMSIZ - 1] = '\0';
break;
default:
return -EINVAL;
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index 6c8c789ded0e..fb78dc6b7e14 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -438,6 +438,12 @@ int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
esp->inplace = false;
+ /* Take real page refs and clear SKBFL_MANAGED_FRAG_REFS before
+ * we mutate the frag array, so the per-frag unref stays balanced
+ * for zerocopy managed frags (see __ip_append_data()).
+ */
+ skb_zcopy_downgrade_managed(skb);
+
allocsize = ALIGN(tailen, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
diff --git a/net/ipv4/fou_core.c b/net/ipv4/fou_core.c
index 790791d08a1c..f1cfd5381e45 100644
--- a/net/ipv4/fou_core.c
+++ b/net/ipv4/fou_core.c
@@ -607,6 +607,10 @@ static int fou_create(struct net *net, struct fou_cfg *cfg,
/* Initial for fou type */
switch (cfg->type) {
case FOU_ENCAP_DIRECT:
+ if (!cfg->protocol) {
+ err = -EINVAL;
+ goto error;
+ }
tunnel_cfg.encap_rcv = fou_udp_recv;
tunnel_cfg.gro_receive = fou_gro_receive;
tunnel_cfg.gro_complete = fou_gro_complete;
diff --git a/net/ipv4/icmp.c b/net/ipv4/icmp.c
index 7e391c2bc5bc..3b0475f65810 100644
--- a/net/ipv4/icmp.c
+++ b/net/ipv4/icmp.c
@@ -573,16 +573,19 @@ static struct rtable *icmp_route_lookup(struct net *net, struct flowi4 *fl4,
skb_dstref_restore(skb_in, orefdst);
/*
- * At this point, fl4_dec.daddr should NOT be local (we
- * checked fl4_dec.saddr above). However, a race condition
- * may occur if the address is added to the interface
- * concurrently. In that case, ip_route_input() returns a
- * LOCAL route with dst.output=ip_rt_bug, which must not
- * be used for output.
+ * fl4_dec.daddr is not expected to be local here, but it can be
+ * added to an interface concurrently, in which case
+ * ip_route_input() returns a LOCAL route. It can also fail to
+ * build a forwarding route towards fl4_dec.daddr, for example,
+ * when forwarding is disabled, and return an UNREACHABLE route.
+ * Both cases will result in a route with dst.output=ip_rt_bug,
+ * which must not be used for output.
*/
- if (!err && rt2 && rt2->rt_type == RTN_LOCAL) {
+ if (!err && rt2 && rt2->rt_type == RTN_LOCAL)
net_warn_ratelimited("detected local route for %pI4 during ICMP sending, src %pI4\n",
&fl4_dec.daddr, &fl4_dec.saddr);
+ if (!err && rt2 &&
+ (rt2->rt_type == RTN_LOCAL || rt2->rt_type == RTN_UNREACHABLE)) {
dst_release(&rt2->dst);
err = -EINVAL;
}
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 117a424d1fee..bd671653273e 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -750,7 +750,7 @@ void inet_csk_init_xmit_timers(struct sock *sk,
timer_setup(&icsk->icsk_retransmit_timer, retransmit_handler, 0);
timer_setup(&icsk->icsk_delack_timer, delack_handler, 0);
- timer_setup(&sk->sk_timer, keepalive_handler, 0);
+ timer_setup(&icsk->icsk_keepalive_timer, keepalive_handler, 0);
icsk->icsk_pending = icsk->icsk_ack.pending = 0;
}
EXPORT_SYMBOL(inet_csk_init_xmit_timers);
@@ -763,7 +763,7 @@ void inet_csk_clear_xmit_timers(struct sock *sk)
sk_stop_timer(sk, &icsk->icsk_retransmit_timer);
sk_stop_timer(sk, &icsk->icsk_delack_timer);
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &icsk->icsk_keepalive_timer);
}
EXPORT_SYMBOL(inet_csk_clear_xmit_timers);
@@ -778,18 +778,18 @@ void inet_csk_clear_xmit_timers_sync(struct sock *sk)
sk_stop_timer_sync(sk, &icsk->icsk_retransmit_timer);
sk_stop_timer_sync(sk, &icsk->icsk_delack_timer);
- sk_stop_timer_sync(sk, &sk->sk_timer);
+ sk_stop_timer_sync(sk, &icsk->icsk_keepalive_timer);
}
void inet_csk_delete_keepalive_timer(struct sock *sk)
{
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &inet_csk(sk)->icsk_keepalive_timer);
}
EXPORT_SYMBOL(inet_csk_delete_keepalive_timer);
void inet_csk_reset_keepalive_timer(struct sock *sk, unsigned long len)
{
- sk_reset_timer(sk, &sk->sk_timer, jiffies + len);
+ sk_reset_timer(sk, &inet_csk(sk)->icsk_keepalive_timer, jiffies + len);
}
EXPORT_SYMBOL(inet_csk_reset_keepalive_timer);
@@ -1541,7 +1541,8 @@ void inet_csk_listen_stop(struct sock *sk)
local_bh_enable();
sock_put(child);
- cond_resched();
+ if (!has_current_bpf_ctx())
+ cond_resched();
}
if (queue->fastopenq.rskq_rst_head) {
/* Free all the reqs queued in rskq_rst_head. */
diff --git a/net/ipv4/inet_diag.c b/net/ipv4/inet_diag.c
index f2de2f296259..01b6b50ae11d 100644
--- a/net/ipv4/inet_diag.c
+++ b/net/ipv4/inet_diag.c
@@ -319,11 +319,11 @@ int inet_sk_diag_fill(struct sock *sk, struct inet_connection_sock *icsk,
r->idiag_retrans = icsk->icsk_probes_out;
r->idiag_expires =
jiffies_delta_to_msecs(icsk->icsk_timeout - jiffies);
- } else if (timer_pending(&sk->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
r->idiag_timer = 2;
r->idiag_retrans = icsk->icsk_probes_out;
r->idiag_expires =
- jiffies_delta_to_msecs(sk->sk_timer.expires - jiffies);
+ jiffies_delta_to_msecs(icsk->icsk_keepalive_timer.expires - jiffies);
}
if ((ext & (1 << (INET_DIAG_INFO - 1))) && handler->idiag_info_size) {
diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
index f344e9bb31fb..5c5ea36584bf 100644
--- a/net/ipv4/ip_gre.c
+++ b/net/ipv4/ip_gre.c
@@ -1449,6 +1449,12 @@ static int ipgre_changelink(struct net_device *dev, struct nlattr *tb[],
if (!rtnl_dev_link_net_capable(dev, t->net))
return -EPERM;
+ if (data && data[IFLA_GRE_COLLECT_METADATA] && !t->collect_md) {
+ NL_SET_ERR_MSG(extack,
+ "Enabling collect_md on an existing device is not supported");
+ return -EOPNOTSUPP;
+ }
+
err = ipgre_newlink_encap_setup(dev, data);
if (err)
return err;
@@ -1481,6 +1487,12 @@ static int erspan_changelink(struct net_device *dev, struct nlattr *tb[],
if (!rtnl_dev_link_net_capable(dev, t->net))
return -EPERM;
+ if (data && data[IFLA_GRE_COLLECT_METADATA] && !t->collect_md) {
+ NL_SET_ERR_MSG(extack,
+ "Enabling collect_md on an existing device is not supported");
+ return -EOPNOTSUPP;
+ }
+
err = ipgre_newlink_encap_setup(dev, data);
if (err)
return err;
diff --git a/net/ipv4/ipconfig.c b/net/ipv4/ipconfig.c
index c56b6fe6f0d7..74a770ea2288 100644
--- a/net/ipv4/ipconfig.c
+++ b/net/ipv4/ipconfig.c
@@ -676,11 +676,40 @@ static const u8 ic_bootp_cookie[4] = { 99, 130, 83, 99 };
#ifdef IPCONFIG_DHCP
+static bool __init
+ic_dhcp_add_option(u8 **options, const u8 *end, u8 type, const void *value,
+ int len)
+{
+ u8 *e = *options;
+
+ /* leave room for the option header and the END marker */
+ if (len > U8_MAX || end - e < len + 3)
+ return false;
+
+ *e++ = type;
+ *e++ = len;
+ memcpy(e, value, len);
+ *options = e + len;
+
+ return true;
+}
+
static void __init
ic_dhcp_init_options(u8 *options, struct ic_device *d)
{
- u8 mt = ((ic_servaddr == NONE)
- ? DHCPDISCOVER : DHCPREQUEST);
+ static const u8 ic_req_params[] = {
+ 1, /* Subnet mask */
+ 3, /* Default gateway */
+ 6, /* DNS server */
+ 12, /* Host name */
+ 15, /* Domain name */
+ 17, /* Boot path */
+ 26, /* MTU */
+ 40, /* NIS domain name */
+ 42, /* NTP servers */
+ };
+ u8 mt = (ic_servaddr == NONE) ? DHCPDISCOVER : DHCPREQUEST;
+ u8 *end = options + sizeof(((struct bootp_pkt *)0)->exten);
u8 *e = options;
int len;
@@ -705,52 +734,25 @@ ic_dhcp_init_options(u8 *options, struct ic_device *d)
e += 4;
}
- /* always? */
- {
- static const u8 ic_req_params[] = {
- 1, /* Subnet mask */
- 3, /* Default gateway */
- 6, /* DNS server */
- 12, /* Host name */
- 15, /* Domain name */
- 17, /* Boot path */
- 26, /* MTU */
- 40, /* NIS domain name */
- 42, /* NTP servers */
- };
-
- *e++ = 55; /* Parameter request list */
- *e++ = sizeof(ic_req_params);
- memcpy(e, ic_req_params, sizeof(ic_req_params));
- e += sizeof(ic_req_params);
-
- if (ic_host_name_set) {
- *e++ = 12; /* host-name */
- len = strlen(utsname()->nodename);
- *e++ = len;
- memcpy(e, utsname()->nodename, len);
- e += len;
- }
- if (*vendor_class_identifier) {
+ *e++ = 55; /* Parameter request list */
+ *e++ = sizeof(ic_req_params);
+ memcpy(e, ic_req_params, sizeof(ic_req_params));
+ e += sizeof(ic_req_params);
+
+ if (ic_host_name_set) {
+ len = strlen(utsname()->nodename);
+ ic_dhcp_add_option(&e, end, 12, utsname()->nodename, len);
+ }
+ if (*vendor_class_identifier) {
+ len = strlen(vendor_class_identifier);
+ if (ic_dhcp_add_option(&e, end, 60, vendor_class_identifier, len))
pr_info("DHCP: sending class identifier \"%s\"\n",
vendor_class_identifier);
- *e++ = 60; /* Class-identifier */
- len = strlen(vendor_class_identifier);
- *e++ = len;
- memcpy(e, vendor_class_identifier, len);
- e += len;
- }
- len = strlen(dhcp_client_identifier + 1);
- /* the minimum length of identifier is 2, include 1 byte type,
- * and can not be larger than the length of options
- */
- if (len >= 1 && len < 312 - (e - options) - 1) {
- *e++ = 61;
- *e++ = len + 1;
- memcpy(e, dhcp_client_identifier, len + 1);
- e += len + 1;
- }
}
+ len = strlen(dhcp_client_identifier + 1);
+ /* the minimum length of identifier is 2, include 1 byte type */
+ if (len >= 1)
+ ic_dhcp_add_option(&e, end, 61, dhcp_client_identifier, len + 1);
*e++ = 255; /* End of the list */
}
diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c
index 0c6a9dedde49..e37853d70db6 100644
--- a/net/ipv4/ipmr.c
+++ b/net/ipv4/ipmr.c
@@ -3162,7 +3162,8 @@ int __init ip_mr_init(void)
{
int err;
- mrt_cachep = KMEM_CACHE(mfc_cache, SLAB_HWCACHE_ALIGN | SLAB_PANIC);
+ mrt_cachep = KMEM_CACHE(mfc_cache,
+ SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT);
err = register_pernet_subsys(&ipmr_net_ops);
if (err)
diff --git a/net/ipv4/ipmr_base.c b/net/ipv4/ipmr_base.c
index 28d77d454d44..66fe1a253b9f 100644
--- a/net/ipv4/ipmr_base.c
+++ b/net/ipv4/ipmr_base.c
@@ -38,7 +38,7 @@ mr_table_alloc(struct net *net, u32 id,
struct mr_table *mrt;
int err;
- mrt = kzalloc(sizeof(*mrt), GFP_KERNEL);
+ mrt = kzalloc(sizeof(*mrt), GFP_KERNEL_ACCOUNT);
if (!mrt)
return ERR_PTR(-ENOMEM);
mrt->id = id;
diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
index 63625a5038af..00d3be968a53 100644
--- a/net/ipv4/sysctl_net_ipv4.c
+++ b/net/ipv4/sysctl_net_ipv4.c
@@ -46,6 +46,8 @@ static unsigned int udp_child_hash_entries_max = UDP_HTABLE_SIZE_MAX;
static int tcp_plb_max_rounds = 31;
static int tcp_plb_max_cong_thresh = 256;
+static int tcp_min_rcvbuf = 4096;
+
/* obsolete */
static int sysctl_tcp_low_latency __read_mostly;
@@ -1405,7 +1407,7 @@ static struct ctl_table ipv4_net_table[] = {
.maxlen = sizeof(init_net.ipv4.sysctl_tcp_rmem),
.mode = 0644,
.proc_handler = proc_dointvec_minmax,
- .extra1 = SYSCTL_ONE,
+ .extra1 = &tcp_min_rcvbuf,
},
{
.procname = "tcp_comp_sack_delay_ns",
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 55b34cf309be..c3aeb985f819 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -6138,6 +6138,7 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
* or pure receivers (this means either the sequence number or the ack
* value must stay constant)
* - Unexpected TCP option.
+ * - ACK sequence number is outside [SND.UNA, SND.NXT].
*
* When these conditions are not satisfied it drops into a standard
* receive procedure patterned after RFC793 to handle all cases.
@@ -6186,7 +6187,7 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
if ((tcp_flag_word(th) & TCP_HP_BITS) == tp->pred_flags &&
TCP_SKB_CB(skb)->seq == tp->rcv_nxt &&
- !after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt)) {
+ between(TCP_SKB_CB(skb)->ack_seq, tp->snd_una, tp->snd_nxt)) {
int tcp_header_len = tp->tcp_header_len;
/* Timestamp header prediction: tcp_header_len
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index beca398b838f..6beb3f0922ac 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -2918,9 +2918,9 @@ static void get_tcp4_sock(struct sock *sk, struct seq_file *f, int i)
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sk->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sk->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = jiffies;
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index a8c480122867..aa00a9fb2506 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -3679,6 +3679,7 @@ void tcp_send_active_reset(struct sock *sk, enum sk_rst_reason reason)
*/
int tcp_send_synack(struct sock *sk)
{
+ struct tcp_sock *tp = tcp_sk(sk);
struct sk_buff *skb;
skb = tcp_rtx_queue_head(sk);
@@ -3696,6 +3697,8 @@ int tcp_send_synack(struct sock *sk)
if (!nskb)
return -ENOMEM;
INIT_LIST_HEAD(&nskb->tcp_tsorted_anchor);
+ if (skb == tp->retransmit_skb_hint)
+ tp->retransmit_skb_hint = nskb;
tcp_highest_sack_replace(sk, skb, nskb);
tcp_rtx_queue_unlink_and_free(skb, sk);
__skb_header_release(nskb);
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 1af9235fe5ae..375b9b980739 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -755,8 +755,9 @@ EXPORT_IPV6_MOD_GPL(tcp_set_keepalive);
static void tcp_keepalive_timer (struct timer_list *t)
{
- struct sock *sk = from_timer(sk, t, sk_timer);
- struct inet_connection_sock *icsk = inet_csk(sk);
+ struct inet_connection_sock *icsk =
+ from_timer(icsk, t, icsk_keepalive_timer);
+ struct sock *sk = &icsk->icsk_inet.sk;
struct tcp_sock *tp = tcp_sk(sk);
u32 elapsed;
diff --git a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c
index adf21d6b6076..9deee1558e3d 100644
--- a/net/ipv4/xfrm4_input.c
+++ b/net/ipv4/xfrm4_input.c
@@ -76,8 +76,6 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async)
NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING,
dev_net(dev), NULL, skb, dev, NULL,
xfrm4_rcv_encap_finish);
- if (async)
- dev_put(dev);
return 0;
}
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index ef938fb1d754..fb239e8f8359 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -2255,12 +2255,11 @@ void addrconf_join_solict(struct net_device *dev, const struct in6_addr *addr)
ipv6_dev_mc_inc(dev, &maddr);
}
-/* caller must hold RTNL */
void addrconf_leave_solict(struct inet6_dev *idev, const struct in6_addr *addr)
{
struct in6_addr maddr;
- if (idev->dev->flags&(IFF_LOOPBACK|IFF_NOARP))
+ if (READ_ONCE(idev->dev->flags) & (IFF_LOOPBACK | IFF_NOARP))
return;
addrconf_addr_solict_mult(addr, &maddr);
diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c
index 0fb93bb4c0ad..5cbb20e3788c 100644
--- a/net/ipv6/af_inet6.c
+++ b/net/ipv6/af_inet6.c
@@ -856,7 +856,7 @@ int inet6_sk_rebuild_header(struct sock *sk)
return PTR_ERR(dst);
}
- ip6_dst_store(sk, dst, NULL, NULL);
+ ip6_dst_store(sk, dst, NULL, false);
}
return 0;
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index 80981596236a..7411aac0707b 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -467,6 +467,12 @@ int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
esp->inplace = false;
+ /* Take real page refs and clear SKBFL_MANAGED_FRAG_REFS before
+ * we mutate the frag array, so the per-frag unref stays balanced
+ * for zerocopy managed frags (see __ip_append_data()).
+ */
+ skb_zcopy_downgrade_managed(skb);
+
allocsize = ALIGN(tailen, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
diff --git a/net/ipv6/exthdrs_core.c b/net/ipv6/exthdrs_core.c
index 49e31e4ae7b7..ce3724f49e33 100644
--- a/net/ipv6/exthdrs_core.c
+++ b/net/ipv6/exthdrs_core.c
@@ -271,6 +271,9 @@ int ipv6_find_hdr(const struct sk_buff *skb, unsigned int *offset,
hdrlen = ipv6_optlen(hp);
if (!found) {
+ if (skb->len - start < hdrlen)
+ return -EBADMSG;
+
nexthdr = hp->nexthdr;
start += hdrlen;
}
diff --git a/net/ipv6/inet6_connection_sock.c b/net/ipv6/inet6_connection_sock.c
index 80043e46117c..1bc4510b4561 100644
--- a/net/ipv6/inet6_connection_sock.c
+++ b/net/ipv6/inet6_connection_sock.c
@@ -106,7 +106,7 @@ static struct dst_entry *inet6_csk_route_socket(struct sock *sk,
dst = ip6_dst_lookup_flow(sock_net(sk), sk, fl6, final_p);
if (!IS_ERR(dst))
- ip6_dst_store(sk, dst, NULL, NULL);
+ ip6_dst_store(sk, dst, NULL, false);
}
return dst;
}
diff --git a/net/ipv6/ip6_flowlabel.c b/net/ipv6/ip6_flowlabel.c
index 9d06d6a50c6d..ce6c2b551ef5 100644
--- a/net/ipv6/ip6_flowlabel.c
+++ b/net/ipv6/ip6_flowlabel.c
@@ -66,8 +66,8 @@ EXPORT_SYMBOL(ipv6_flowlabel_exclusive);
fl != NULL; \
fl = rcu_dereference(fl->next))
-#define for_each_sk_fl_rcu(np, sfl) \
- for (sfl = rcu_dereference(np->ipv6_fl_list); \
+#define for_each_sk_fl_rcu(sk, sfl) \
+ for (sfl = rcu_dereference(inet_sk(sk)->ipv6_fl_list); \
sfl != NULL; \
sfl = rcu_dereference(sfl->next))
@@ -257,12 +257,11 @@ static struct ip6_flowlabel *fl_intern(struct net *net,
struct ip6_flowlabel *__fl6_sock_lookup(struct sock *sk, __be32 label)
{
struct ipv6_fl_socklist *sfl;
- struct ipv6_pinfo *np = inet6_sk(sk);
label &= IPV6_FLOWLABEL_MASK;
rcu_read_lock();
- for_each_sk_fl_rcu(np, sfl) {
+ for_each_sk_fl_rcu(sk, sfl) {
struct ip6_flowlabel *fl = sfl->fl;
if (fl->label == label && atomic_inc_not_zero(&fl->users)) {
@@ -278,16 +277,16 @@ EXPORT_SYMBOL_GPL(__fl6_sock_lookup);
void fl6_free_socklist(struct sock *sk)
{
- struct ipv6_pinfo *np = inet6_sk(sk);
+ struct inet_sock *inet = inet_sk(sk);
struct ipv6_fl_socklist *sfl;
- if (!rcu_access_pointer(np->ipv6_fl_list))
+ if (!rcu_access_pointer(inet->ipv6_fl_list))
return;
spin_lock_bh(&ip6_sk_fl_lock);
- while ((sfl = rcu_dereference_protected(np->ipv6_fl_list,
+ while ((sfl = rcu_dereference_protected(inet->ipv6_fl_list,
lockdep_is_held(&ip6_sk_fl_lock))) != NULL) {
- np->ipv6_fl_list = sfl->next;
+ inet->ipv6_fl_list = sfl->next;
spin_unlock_bh(&ip6_sk_fl_lock);
fl_release(sfl->fl);
@@ -463,37 +462,55 @@ fl_create(struct net *net, struct sock *sk, struct in6_flowlabel_req *freq,
return NULL;
}
+static bool fl_sock_at_lease_limit(const struct sock *sk)
+{
+ const struct ipv6_fl_socklist *sfl;
+ int count = 0;
+
+ rcu_read_lock();
+ for_each_sk_fl_rcu(sk, sfl) {
+ if (++count >= FL_MAX_PER_SOCK)
+ break;
+ }
+ rcu_read_unlock();
+
+ return count >= FL_MAX_PER_SOCK;
+}
+
static int mem_check(struct sock *sk)
{
- struct ipv6_pinfo *np = inet6_sk(sk);
- struct ipv6_fl_socklist *sfl;
+ const int unpriv_total_limit = FL_MAX_SIZE - (FL_MAX_SIZE / 4);
int room = FL_MAX_SIZE - atomic_read(&fl_size);
+ struct ipv6_fl_socklist *sfl;
int count = 0;
if (room > FL_MAX_SIZE - FL_MAX_PER_SOCK)
return 0;
rcu_read_lock();
- for_each_sk_fl_rcu(np, sfl)
+ for_each_sk_fl_rcu(sk, sfl)
count++;
rcu_read_unlock();
if (room <= 0 ||
((count >= FL_MAX_PER_SOCK ||
- (count > 0 && room < FL_MAX_SIZE/2) || room < FL_MAX_SIZE/4) &&
+ (count > 0 && room < FL_MAX_SIZE / 2) ||
+ room < FL_MAX_SIZE - unpriv_total_limit) &&
!capable(CAP_NET_ADMIN)))
return -ENOBUFS;
return 0;
}
-static inline void fl_link(struct ipv6_pinfo *np, struct ipv6_fl_socklist *sfl,
- struct ip6_flowlabel *fl)
+static inline void fl_link(struct sock *sk, struct ipv6_fl_socklist *sfl,
+ struct ip6_flowlabel *fl)
{
+ struct inet_sock *inet = inet_sk(sk);
+
spin_lock_bh(&ip6_sk_fl_lock);
sfl->fl = fl;
- sfl->next = np->ipv6_fl_list;
- rcu_assign_pointer(np->ipv6_fl_list, sfl);
+ sfl->next = inet->ipv6_fl_list;
+ rcu_assign_pointer(inet->ipv6_fl_list, sfl);
spin_unlock_bh(&ip6_sk_fl_lock);
}
@@ -515,7 +532,7 @@ int ipv6_flowlabel_opt_get(struct sock *sk, struct in6_flowlabel_req *freq,
rcu_read_lock();
- for_each_sk_fl_rcu(np, sfl) {
+ for_each_sk_fl_rcu(sk, sfl) {
if (sfl->fl->label == (np->flow_label & IPV6_FLOWLABEL_MASK)) {
spin_lock_bh(&ip6_fl_lock);
freq->flr_label = sfl->fl->label;
@@ -554,7 +571,7 @@ static int ipv6_flowlabel_put(struct sock *sk, struct in6_flowlabel_req *freq)
}
spin_lock_bh(&ip6_sk_fl_lock);
- for (sflp = &np->ipv6_fl_list;
+ for (sflp = &inet_sk(sk)->ipv6_fl_list;
(sfl = socklist_dereference(*sflp)) != NULL;
sflp = &sfl->next) {
if (sfl->fl->label == freq->flr_label)
@@ -574,13 +591,12 @@ static int ipv6_flowlabel_put(struct sock *sk, struct in6_flowlabel_req *freq)
static int ipv6_flowlabel_renew(struct sock *sk, struct in6_flowlabel_req *freq)
{
- struct ipv6_pinfo *np = inet6_sk(sk);
struct net *net = sock_net(sk);
struct ipv6_fl_socklist *sfl;
int err;
rcu_read_lock();
- for_each_sk_fl_rcu(np, sfl) {
+ for_each_sk_fl_rcu(sk, sfl) {
if (sfl->fl->label == freq->flr_label) {
err = fl6_renew(sfl->fl, freq->flr_linger,
freq->flr_expires);
@@ -609,7 +625,6 @@ static int ipv6_flowlabel_get(struct sock *sk, struct in6_flowlabel_req *freq,
{
struct ipv6_fl_socklist *sfl, *sfl1 = NULL;
struct ip6_flowlabel *fl, *fl1 = NULL;
- struct ipv6_pinfo *np = inet6_sk(sk);
struct net *net = sock_net(sk);
int err;
@@ -640,7 +655,7 @@ static int ipv6_flowlabel_get(struct sock *sk, struct in6_flowlabel_req *freq,
if (freq->flr_label) {
err = -EEXIST;
rcu_read_lock();
- for_each_sk_fl_rcu(np, sfl) {
+ for_each_sk_fl_rcu(sk, sfl) {
if (sfl->fl->label == freq->flr_label) {
if (freq->flr_flags & IPV6_FL_F_EXCL) {
rcu_read_unlock();
@@ -673,11 +688,15 @@ static int ipv6_flowlabel_get(struct sock *sk, struct in6_flowlabel_req *freq,
err = -ENOMEM;
if (!sfl1)
goto release;
+ err = -ENOBUFS;
+ if (fl_sock_at_lease_limit(sk) &&
+ !capable(CAP_NET_ADMIN))
+ goto release;
if (fl->linger > fl1->linger)
fl1->linger = fl->linger;
if ((long)(fl->expires - fl1->expires) > 0)
fl1->expires = fl->expires;
- fl_link(np, sfl1, fl1);
+ fl_link(sk, sfl1, fl1);
fl_free(fl);
return 0;
@@ -711,7 +730,7 @@ static int ipv6_flowlabel_get(struct sock *sk, struct in6_flowlabel_req *freq,
}
}
- fl_link(np, sfl1, fl);
+ fl_link(sk, sfl1, fl);
return 0;
done:
fl_free(fl);
diff --git a/net/ipv6/ip6_gre.c b/net/ipv6/ip6_gre.c
index 4b0c166fe7e2..ce1ba9d2af40 100644
--- a/net/ipv6/ip6_gre.c
+++ b/net/ipv6/ip6_gre.c
@@ -2314,7 +2314,7 @@ static int ip6erspan_changelink(struct net_device *dev, struct nlattr *tb[],
return PTR_ERR(t);
ip6erspan_set_version(data, &p);
- ip6gre_tunnel_unlink_md(ign, t);
+ ip6erspan_tunnel_unlink_md(ign, t);
ip6gre_tunnel_unlink(ign, t);
ip6erspan_tnl_change(t, &p, !tb[IFLA_MTU]);
ip6erspan_tunnel_link_md(ign, t);
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index a4786ef73da5..e53dbf14873a 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1103,7 +1103,8 @@ static struct dst_entry *ip6_sk_dst_check(struct sock *sk,
*/
if (ip6_rt_check(&rt->rt6i_dst, &fl6->daddr, np->daddr_cache) ||
#ifdef CONFIG_IPV6_SUBTREES
- ip6_rt_check(&rt->rt6i_src, &fl6->saddr, np->saddr_cache) ||
+ ip6_rt_check(&rt->rt6i_src, &fl6->saddr,
+ np->saddr_cache ? &np->saddr : NULL) ||
#endif
(fl6->flowi6_oif && fl6->flowi6_oif != dst_dev(dst)->ifindex)) {
dst_release(dst);
diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c
index 1f9653be58d4..ac0120b75ff2 100644
--- a/net/ipv6/ip6mr.c
+++ b/net/ipv6/ip6mr.c
@@ -1387,7 +1387,7 @@ int __init ip6_mr_init(void)
{
int err;
- mrt_cachep = KMEM_CACHE(mfc6_cache, SLAB_HWCACHE_ALIGN);
+ mrt_cachep = KMEM_CACHE(mfc6_cache, SLAB_HWCACHE_ALIGN | SLAB_ACCOUNT);
if (!mrt_cachep)
return -ENOMEM;
diff --git a/net/ipv6/ipv6_sockglue.c b/net/ipv6/ipv6_sockglue.c
index c2ea92c0f916..0f1242138d17 100644
--- a/net/ipv6/ipv6_sockglue.c
+++ b/net/ipv6/ipv6_sockglue.c
@@ -121,17 +121,8 @@ static bool setsockopt_needs_rtnl(int optname)
{
switch (optname) {
case IPV6_ADDRFORM:
- case IPV6_ADD_MEMBERSHIP:
- case IPV6_DROP_MEMBERSHIP:
case IPV6_JOIN_ANYCAST:
case IPV6_LEAVE_ANYCAST:
- case MCAST_JOIN_GROUP:
- case MCAST_LEAVE_GROUP:
- case MCAST_JOIN_SOURCE_GROUP:
- case MCAST_LEAVE_SOURCE_GROUP:
- case MCAST_BLOCK_SOURCE:
- case MCAST_UNBLOCK_SOURCE:
- case MCAST_MSFILTER:
return true;
}
return false;
diff --git a/net/ipv6/mcast.c b/net/ipv6/mcast.c
index c060954c5275..d0eefadea5c3 100644
--- a/net/ipv6/mcast.c
+++ b/net/ipv6/mcast.c
@@ -172,14 +172,12 @@ static int unsolicited_report_interval(struct inet6_dev *idev)
static int __ipv6_sock_mc_join(struct sock *sk, int ifindex,
const struct in6_addr *addr, unsigned int mode)
{
- struct net_device *dev = NULL;
- struct ipv6_mc_socklist *mc_lst;
struct ipv6_pinfo *np = inet6_sk(sk);
+ struct ipv6_mc_socklist *mc_lst;
struct net *net = sock_net(sk);
+ struct net_device *dev = NULL;
int err;
- ASSERT_RTNL();
-
if (!ipv6_addr_is_multicast(addr))
return -EINVAL;
@@ -199,13 +197,18 @@ static int __ipv6_sock_mc_join(struct sock *sk, int ifindex,
if (ifindex == 0) {
struct rt6_info *rt;
+
+ rcu_read_lock();
rt = rt6_lookup(net, addr, NULL, 0, NULL, 0);
if (rt) {
- dev = rt->dst.dev;
+ dev = dst_dev(&rt->dst);
+ dev_hold(dev);
ip6_rt_put(rt);
}
- } else
- dev = __dev_get_by_index(net, ifindex);
+ rcu_read_unlock();
+ } else {
+ dev = dev_get_by_index(net, ifindex);
+ }
if (!dev) {
sock_kfree_s(sk, mc_lst, sizeof(*mc_lst));
@@ -216,12 +219,11 @@ static int __ipv6_sock_mc_join(struct sock *sk, int ifindex,
mc_lst->sfmode = mode;
RCU_INIT_POINTER(mc_lst->sflist, NULL);
- /*
- * now add/increase the group membership on the device
- */
-
+ /* now add/increase the group membership on the device */
err = __ipv6_dev_mc_inc(dev, addr, mode);
+ dev_put(dev);
+
if (err) {
sock_kfree_s(sk, mc_lst, sizeof(*mc_lst));
return err;
@@ -248,14 +250,36 @@ int ipv6_sock_mc_join_ssm(struct sock *sk, int ifindex,
/*
* socket leave on multicast group
*/
+static void __ipv6_sock_mc_drop(struct sock *sk, struct ipv6_mc_socklist *mc_lst)
+{
+ struct net *net = sock_net(sk);
+ struct net_device *dev;
+
+ dev = dev_get_by_index(net, mc_lst->ifindex);
+ if (dev) {
+ struct inet6_dev *idev = in6_dev_get(dev);
+
+ ip6_mc_leave_src(sk, mc_lst, idev);
+
+ if (idev) {
+ __ipv6_dev_mc_dec(idev, &mc_lst->addr);
+ in6_dev_put(idev);
+ }
+
+ dev_put(dev);
+ } else {
+ ip6_mc_leave_src(sk, mc_lst, NULL);
+ }
+
+ atomic_sub(sizeof(*mc_lst), &sk->sk_omem_alloc);
+ kfree_rcu(mc_lst, rcu);
+}
+
int ipv6_sock_mc_drop(struct sock *sk, int ifindex, const struct in6_addr *addr)
{
struct ipv6_pinfo *np = inet6_sk(sk);
- struct ipv6_mc_socklist *mc_lst;
struct ipv6_mc_socklist __rcu **lnk;
- struct net *net = sock_net(sk);
-
- ASSERT_RTNL();
+ struct ipv6_mc_socklist *mc_lst;
if (!ipv6_addr_is_multicast(addr))
return -EINVAL;
@@ -265,23 +289,8 @@ int ipv6_sock_mc_drop(struct sock *sk, int ifindex, const struct in6_addr *addr)
lnk = &mc_lst->next) {
if ((ifindex == 0 || mc_lst->ifindex == ifindex) &&
ipv6_addr_equal(&mc_lst->addr, addr)) {
- struct net_device *dev;
-
*lnk = mc_lst->next;
-
- dev = __dev_get_by_index(net, mc_lst->ifindex);
- if (dev) {
- struct inet6_dev *idev = __in6_dev_get(dev);
-
- ip6_mc_leave_src(sk, mc_lst, idev);
- if (idev)
- __ipv6_dev_mc_dec(idev, &mc_lst->addr);
- } else {
- ip6_mc_leave_src(sk, mc_lst, NULL);
- }
-
- atomic_sub(sizeof(*mc_lst), &sk->sk_omem_alloc);
- kfree_rcu(mc_lst, rcu);
+ __ipv6_sock_mc_drop(sk, mc_lst);
return 0;
}
}
@@ -290,31 +299,36 @@ int ipv6_sock_mc_drop(struct sock *sk, int ifindex, const struct in6_addr *addr)
}
EXPORT_SYMBOL(ipv6_sock_mc_drop);
-static struct inet6_dev *ip6_mc_find_dev_rtnl(struct net *net,
- const struct in6_addr *group,
- int ifindex)
+static struct inet6_dev *ip6_mc_find_dev(struct net *net,
+ const struct in6_addr *group,
+ int ifindex)
{
struct net_device *dev = NULL;
- struct inet6_dev *idev = NULL;
+ struct inet6_dev *idev;
if (ifindex == 0) {
- struct rt6_info *rt = rt6_lookup(net, group, NULL, 0, NULL, 0);
+ struct rt6_info *rt;
+ rcu_read_lock();
+ rt = rt6_lookup(net, group, NULL, 0, NULL, 0);
if (rt) {
- dev = rt->dst.dev;
+ dev = dst_dev(&rt->dst);
+ dev_hold(dev);
ip6_rt_put(rt);
}
+ rcu_read_unlock();
} else {
- dev = __dev_get_by_index(net, ifindex);
+ dev = dev_get_by_index(net, ifindex);
}
-
if (!dev)
return NULL;
- idev = __in6_dev_get(dev);
+
+ idev = in6_dev_get(dev);
+ dev_put(dev);
+
if (!idev)
return NULL;
- if (idev->dead)
- return NULL;
+
return idev;
}
@@ -362,16 +376,16 @@ void ipv6_sock_mc_close(struct sock *sk)
}
int ip6_mc_source(int add, int omode, struct sock *sk,
- struct group_source_req *pgsr)
+ struct group_source_req *pgsr)
{
+ struct ipv6_pinfo *inet6 = inet6_sk(sk);
struct in6_addr *source, *group;
+ struct ip6_sf_socklist *newpsl, *psl;
+ struct net *net = sock_net(sk);
struct ipv6_mc_socklist *pmc;
struct inet6_dev *idev;
- struct ipv6_pinfo *inet6 = inet6_sk(sk);
- struct ip6_sf_socklist *psl;
- struct net *net = sock_net(sk);
- int i, j, rv;
int leavegroup = 0;
+ int i, j;
int err;
source = &((struct sockaddr_in6 *)&pgsr->gsr_source)->sin6_addr;
@@ -380,13 +394,19 @@ int ip6_mc_source(int add, int omode, struct sock *sk,
if (!ipv6_addr_is_multicast(group))
return -EINVAL;
- idev = ip6_mc_find_dev_rtnl(net, group, pgsr->gsr_interface);
+ idev = ip6_mc_find_dev(net, group, pgsr->gsr_interface);
if (!idev)
return -ENODEV;
+ mutex_lock(&idev->mc_lock);
+
+ if (idev->dead) {
+ err = -ENODEV;
+ goto done;
+ }
+
err = -EADDRNOTAVAIL;
- mutex_lock(&idev->mc_lock);
for_each_pmc_socklock(inet6, sk, pmc) {
if (pgsr->gsr_interface && pmc->ifindex != pgsr->gsr_interface)
continue;
@@ -414,13 +434,11 @@ int ip6_mc_source(int add, int omode, struct sock *sk,
if (!add) {
if (!psl)
goto done; /* err = -EADDRNOTAVAIL */
- rv = !0;
for (i = 0; i < psl->sl_count; i++) {
- rv = !ipv6_addr_equal(&psl->sl_addr[i], source);
- if (rv == 0)
+ if (ipv6_addr_equal(&psl->sl_addr[i], source))
break;
}
- if (rv) /* source not found */
+ if (i == psl->sl_count) /* source not found */
goto done; /* err = -EADDRNOTAVAIL */
/* special case - (INCLUDE, empty) == LEAVE_GROUP */
@@ -429,60 +447,77 @@ int ip6_mc_source(int add, int omode, struct sock *sk,
goto done;
}
+ atomic_sub(struct_size(psl, sl_addr, psl->sl_max),
+ &sk->sk_omem_alloc);
+
+ if (psl->sl_count == 1) {
+ newpsl = NULL;
+ } else {
+ newpsl = sock_kmalloc(sk, struct_size(newpsl, sl_addr,
+ psl->sl_count - 1),
+ GFP_KERNEL);
+ if (!newpsl) {
+ atomic_add(struct_size(psl, sl_addr, psl->sl_max),
+ &sk->sk_omem_alloc);
+ err = -ENOBUFS;
+ goto done;
+ }
+ newpsl->sl_max = psl->sl_count - 1;
+ newpsl->sl_count = psl->sl_count - 1;
+ for (j = 0; j < i; j++)
+ newpsl->sl_addr[j] = psl->sl_addr[j];
+ for (j = i + 1; j < psl->sl_count; j++)
+ newpsl->sl_addr[j - 1] = psl->sl_addr[j];
+ }
+
/* update the interface filter */
ip6_mc_del_src(idev, group, omode, 1, source, 1);
- for (j = i+1; j < psl->sl_count; j++)
- psl->sl_addr[j-1] = psl->sl_addr[j];
- psl->sl_count--;
+ rcu_assign_pointer(pmc->sflist, newpsl);
+ kfree_rcu(psl, rcu);
err = 0;
goto done;
}
/* else, add a new source to the filter */
- if (psl && psl->sl_count >= sysctl_mld_max_msf) {
+ if (psl && psl->sl_count >= READ_ONCE(sysctl_mld_max_msf)) {
err = -ENOBUFS;
goto done;
}
- if (!psl || psl->sl_count == psl->sl_max) {
- struct ip6_sf_socklist *newpsl;
- int count = IP6_SFBLOCK;
-
- if (psl)
- count += psl->sl_max;
- newpsl = sock_kmalloc(sk, struct_size(newpsl, sl_addr, count),
- GFP_KERNEL);
- if (!newpsl) {
- err = -ENOBUFS;
- goto done;
- }
- newpsl->sl_max = count;
- newpsl->sl_count = count - IP6_SFBLOCK;
- if (psl) {
- for (i = 0; i < psl->sl_count; i++)
- newpsl->sl_addr[i] = psl->sl_addr[i];
- atomic_sub(struct_size(psl, sl_addr, psl->sl_max),
- &sk->sk_omem_alloc);
+ if (psl) {
+ for (i = 0; i < psl->sl_count; i++) {
+ if (ipv6_addr_equal(&psl->sl_addr[i], source))
+ goto done; /* err = -EADDRNOTAVAIL */
}
- rcu_assign_pointer(pmc->sflist, newpsl);
- kfree_rcu(psl, rcu);
- psl = newpsl;
}
- rv = 1; /* > 0 for insert logic below if sl_count is 0 */
- for (i = 0; i < psl->sl_count; i++) {
- rv = !ipv6_addr_equal(&psl->sl_addr[i], source);
- if (rv == 0) /* There is an error in the address. */
- goto done;
+
+ i = psl ? psl->sl_count + 1 : 1;
+ newpsl = sock_kmalloc(sk, struct_size(newpsl, sl_addr, i),
+ GFP_KERNEL);
+ if (!newpsl) {
+ err = -ENOBUFS;
+ goto done;
}
- for (j = psl->sl_count-1; j >= i; j--)
- psl->sl_addr[j+1] = psl->sl_addr[j];
- psl->sl_addr[i] = *source;
- psl->sl_count++;
- err = 0;
+ newpsl->sl_max = i;
+ newpsl->sl_count = i;
+ if (psl) {
+ for (j = 0; j < psl->sl_count; j++)
+ newpsl->sl_addr[j] = psl->sl_addr[j];
+ }
+ newpsl->sl_addr[i - 1] = *source;
+
/* update the interface list */
ip6_mc_add_src(idev, group, omode, 1, source, 1);
+
+ if (psl)
+ atomic_sub(struct_size(psl, sl_addr, psl->sl_max),
+ &sk->sk_omem_alloc);
+ rcu_assign_pointer(pmc->sflist, newpsl);
+ kfree_rcu(psl, rcu);
+ err = 0;
done:
mutex_unlock(&idev->mc_lock);
+ in6_dev_put(idev);
if (leavegroup)
err = ipv6_sock_mc_drop(sk, pgsr->gsr_interface, group);
return err;
@@ -491,12 +526,12 @@ int ip6_mc_source(int add, int omode, struct sock *sk,
int ip6_mc_msfilter(struct sock *sk, struct group_filter *gsf,
struct sockaddr_storage *list)
{
- const struct in6_addr *group;
- struct ipv6_mc_socklist *pmc;
- struct inet6_dev *idev;
struct ipv6_pinfo *inet6 = inet6_sk(sk);
struct ip6_sf_socklist *newpsl, *psl;
struct net *net = sock_net(sk);
+ const struct in6_addr *group;
+ struct ipv6_mc_socklist *pmc;
+ struct inet6_dev *idev;
int leavegroup = 0;
int i, err;
@@ -508,10 +543,17 @@ int ip6_mc_msfilter(struct sock *sk, struct group_filter *gsf,
gsf->gf_fmode != MCAST_EXCLUDE)
return -EINVAL;
- idev = ip6_mc_find_dev_rtnl(net, group, gsf->gf_interface);
+ idev = ip6_mc_find_dev(net, group, gsf->gf_interface);
if (!idev)
return -ENODEV;
+ mutex_lock(&idev->mc_lock);
+
+ if (idev->dead) {
+ err = -ENODEV;
+ goto done;
+ }
+
err = 0;
if (gsf->gf_fmode == MCAST_INCLUDE && gsf->gf_numsrc == 0) {
@@ -544,24 +586,19 @@ int ip6_mc_msfilter(struct sock *sk, struct group_filter *gsf,
psin6 = (struct sockaddr_in6 *)list;
newpsl->sl_addr[i] = psin6->sin6_addr;
}
- mutex_lock(&idev->mc_lock);
+
err = ip6_mc_add_src(idev, group, gsf->gf_fmode,
newpsl->sl_count, newpsl->sl_addr, 0);
if (err) {
- mutex_unlock(&idev->mc_lock);
sock_kfree_s(sk, newpsl, struct_size(newpsl, sl_addr,
newpsl->sl_max));
goto done;
}
- mutex_unlock(&idev->mc_lock);
} else {
newpsl = NULL;
- mutex_lock(&idev->mc_lock);
ip6_mc_add_src(idev, group, gsf->gf_fmode, 0, NULL, 0);
- mutex_unlock(&idev->mc_lock);
}
- mutex_lock(&idev->mc_lock);
psl = sock_dereference(pmc->sflist, sk);
if (psl) {
ip6_mc_del_src(idev, group, pmc->sfmode,
@@ -571,12 +608,14 @@ int ip6_mc_msfilter(struct sock *sk, struct group_filter *gsf,
} else {
ip6_mc_del_src(idev, group, pmc->sfmode, 0, NULL, 0);
}
+
rcu_assign_pointer(pmc->sflist, newpsl);
- mutex_unlock(&idev->mc_lock);
kfree_rcu(psl, rcu);
pmc->sfmode = gsf->gf_fmode;
err = 0;
done:
+ mutex_unlock(&idev->mc_lock);
+ in6_dev_put(idev);
if (leavegroup)
err = ipv6_sock_mc_drop(sk, gsf->gf_interface, group);
return err;
@@ -972,9 +1011,8 @@ int __ipv6_dev_mc_dec(struct inet6_dev *idev, const struct in6_addr *addr)
{
struct ifmcaddr6 *ma, __rcu **map;
- ASSERT_RTNL();
-
mutex_lock(&idev->mc_lock);
+
for (map = &idev->mc_list;
(ma = mc_dereference(*map, idev));
map = &ma->next) {
@@ -1003,13 +1041,12 @@ int ipv6_dev_mc_dec(struct net_device *dev, const struct in6_addr *addr)
struct inet6_dev *idev;
int err;
- ASSERT_RTNL();
-
- idev = __in6_dev_get(dev);
+ idev = in6_dev_get(dev);
if (!idev)
- err = -ENODEV;
- else
- err = __ipv6_dev_mc_dec(idev, addr);
+ return -ENODEV;
+
+ err = __ipv6_dev_mc_dec(idev, addr);
+ in6_dev_put(idev);
return err;
}
diff --git a/net/ipv6/netfilter/ip6t_rpfilter.c b/net/ipv6/netfilter/ip6t_rpfilter.c
index 67c87a88cde4..b5def30c3127 100644
--- a/net/ipv6/netfilter/ip6t_rpfilter.c
+++ b/net/ipv6/netfilter/ip6t_rpfilter.c
@@ -61,7 +61,7 @@ static bool rpfilter_lookup_reverse6(struct net *net, const struct sk_buff *skb,
fl6.flowi6_oif = dev->ifindex;
rt = (void *)ip6_route_lookup(net, &fl6, skb, lookup_flags);
- if (rt->dst.error)
+ if (rt->dst.error || !rt->rt6i_idev)
goto out;
if (rt->rt6i_flags & (RTF_REJECT|RTF_ANYCAST))
diff --git a/net/ipv6/netfilter/ip6t_rt.c b/net/ipv6/netfilter/ip6t_rt.c
index 278b52752f36..2f6261d06a07 100644
--- a/net/ipv6/netfilter/ip6t_rt.c
+++ b/net/ipv6/netfilter/ip6t_rt.c
@@ -96,7 +96,8 @@ static bool rt_mt6(const struct sk_buff *skb, struct xt_action_param *par)
unsigned int i = 0;
for (temp = 0;
- temp < (unsigned int)((hdrlen - 8) / 16);
+ temp < (unsigned int)((hdrlen - 8) / 16) &&
+ i < rtinfo->addrnr;
temp++) {
ap = skb_header_pointer(skb,
ptr
@@ -112,8 +113,6 @@ static bool rt_mt6(const struct sk_buff *skb, struct xt_action_param *par)
if (ipv6_addr_equal(ap, &rtinfo->addrs[i]))
i++;
- if (i == rtinfo->addrnr)
- break;
}
if (i == rtinfo->addrnr)
return ret;
@@ -162,6 +161,12 @@ static int rt_mt6_check(const struct xt_mtchk_param *par)
pr_debug("too many addresses specified\n");
return -EINVAL;
}
+
+ if ((rtinfo->flags & IP6T_RT_FST_MASK) && !rtinfo->addrnr) {
+ pr_info_ratelimited("address list match requested but addrnr is 0\n");
+ return -EINVAL;
+ }
+
if ((rtinfo->flags & (IP6T_RT_RES | IP6T_RT_FST_MASK)) &&
(!(rtinfo->flags & IP6T_RT_TYP) ||
(rtinfo->rt_type != 0) ||
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 19c970978e86..e2064ee45845 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -139,6 +139,7 @@ void rt6_uncached_list_add(struct rt6_info *rt)
{
struct uncached_list *ul = raw_cpu_ptr(&rt6_uncached_list);
+ /* Set once and never cleared: non-NULL marks an uncached route. */
rt->dst.rt_uncached_list = ul;
spin_lock_bh(&ul->lock);
@@ -820,9 +821,11 @@ static int rt6_nh_find_match(struct fib6_nh *nh, void *_arg)
{
struct fib6_nh_frl_arg *arg = _arg;
- arg->nh = nh;
- return find_match(nh, arg->flags, arg->oif, arg->strict,
- arg->mpri, arg->do_rr);
+ if (find_match(nh, arg->flags, arg->oif, arg->strict, arg->mpri,
+ arg->do_rr))
+ arg->nh = nh;
+
+ return 0;
}
static void __find_rr_leaf(struct fib6_info *f6i_start,
@@ -862,11 +865,10 @@ static void __find_rr_leaf(struct fib6_info *f6i_start,
res->nh = nexthop_fib6_nh(f6i->nh);
return;
}
- if (nexthop_for_each_fib6_nh(f6i->nh, rt6_nh_find_match,
- &arg)) {
- matched = true;
- nh = arg.nh;
- }
+ nexthop_for_each_fib6_nh(f6i->nh, rt6_nh_find_match,
+ &arg);
+ matched = !!arg.nh;
+ nh = arg.nh;
} else {
nh = f6i->fib6_nh;
if (find_match(nh, f6i->fib6_flags, oif, strict,
@@ -2268,6 +2270,7 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table,
{
struct fib6_result res = {};
struct rt6_info *rt = NULL;
+ bool have_oif_match;
int strict = 0;
WARN_ON_ONCE((flags & RT6_LOOKUP_F_DST_NOREF) &&
@@ -2284,7 +2287,9 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table,
if (res.f6i == net->ipv6.fib6_null_entry)
goto out;
- fib6_select_path(net, &res, fl6, oif, false, skb, strict);
+ have_oif_match = fl6->flowi6_iif == LOOPBACK_IFINDEX &&
+ oif == res.nh->fib_nh_dev->ifindex;
+ fib6_select_path(net, &res, fl6, oif, have_oif_match, skb, strict);
/*Search through exception table */
rt = rt6_find_cached_rt(&res, &fl6->daddr, &fl6->saddr);
@@ -2702,8 +2707,8 @@ struct dst_entry *ip6_route_output_flags(struct net *net,
rcu_read_lock();
dst = ip6_route_output_flags_noref(net, sk, fl6, flags);
rt6 = dst_rt6_info(dst);
- /* For dst cached in uncached_list, refcnt is already taken. */
- if (list_empty(&rt6->dst.rt_uncached) && !dst_hold_safe(dst)) {
+ /* For an uncached dst, refcnt is already taken. */
+ if (!rt6->dst.rt_uncached_list && !dst_hold_safe(dst)) {
dst = &net->ipv6.ip6_null_entry->dst;
dst_hold(dst);
}
@@ -2813,7 +2818,7 @@ INDIRECT_CALLABLE_SCOPE struct dst_entry *ip6_dst_check(struct dst_entry *dst,
from = rcu_dereference(rt->from);
if (from && (rt->rt6i_flags & RTF_PCPU ||
- unlikely(!list_empty(&rt->dst.rt_uncached))))
+ unlikely(rt->dst.rt_uncached_list)))
dst_ret = rt6_dst_from_check(rt, from, cookie);
else
dst_ret = rt6_check(rt, from, cookie);
@@ -3044,9 +3049,9 @@ void ip6_sk_dst_store_flow(struct sock *sk, struct dst_entry *dst,
&sk->sk_v6_daddr : NULL,
#ifdef CONFIG_IPV6_SUBTREES
ipv6_addr_equal(&fl6->saddr, &np->saddr) ?
- &np->saddr :
+ true :
#endif
- NULL);
+ false);
}
static bool ip6_redirect_nh_match(const struct fib6_result *res,
diff --git a/net/ipv6/seg6.c b/net/ipv6/seg6.c
index 5b57ea4fdda9..c175d6cb5789 100644
--- a/net/ipv6/seg6.c
+++ b/net/ipv6/seg6.c
@@ -138,8 +138,8 @@ void seg6_icmp_srh(struct sk_buff *skb, struct inet6_skb_parm *opt)
static struct genl_family seg6_genl_family;
static const struct nla_policy seg6_genl_policy[SEG6_ATTR_MAX + 1] = {
- [SEG6_ATTR_DST] = { .type = NLA_BINARY,
- .len = sizeof(struct in6_addr) },
+ [SEG6_ATTR_DST] =
+ NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)),
[SEG6_ATTR_DSTLEN] = { .type = NLA_S32, },
[SEG6_ATTR_HMACKEYID] = { .type = NLA_U32, },
[SEG6_ATTR_SECRET] = { .type = NLA_BINARY, },
diff --git a/net/ipv6/seg6_local.c b/net/ipv6/seg6_local.c
index 10f29bb8e608..191b99b073b0 100644
--- a/net/ipv6/seg6_local.c
+++ b/net/ipv6/seg6_local.c
@@ -257,10 +257,13 @@ static bool decap_and_validate(struct sk_buff *skb, int proto)
return false;
if (proto == IPPROTO_IPIP) {
+ bool l3slave = ipv6_l3mdev_skb(IP6CB(skb)->flags);
int iif = IP6CB(skb)->iif;
memset(IPCB(skb), 0, sizeof(*IPCB(skb)));
IPCB(skb)->iif = iif;
+ if (l3slave)
+ IPCB(skb)->flags |= IPSKB_L3SLAVE;
} else if (proto == IPPROTO_IPV6) {
bool l3slave = ipv6_l3mdev_skb(IP6CB(skb)->flags);
int iif = IP6CB(skb)->iif;
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 533943a7127d..69e37c3d4b1c 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -293,7 +293,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr,
inet->inet_rcv_saddr = LOOPBACK4_IPV6;
sk->sk_gso_type = SKB_GSO_TCPV6;
- ip6_dst_store(sk, dst, NULL, NULL);
+ ip6_dst_store(sk, dst, NULL, false);
icsk->icsk_ext_hdr_len = 0;
if (opt)
@@ -1349,6 +1349,7 @@ static void tcp_v6_mapped_child_init(struct sock *newsk, const struct sock *sk)
struct ipv6_pinfo *newnp;
newinet->pinet6 = newnp = tcp_inet6_sk(newsk);
+ newinet->ipv6_fl_list = NULL;
memcpy(newnp, tcp_inet6_sk(sk), sizeof(struct ipv6_pinfo));
@@ -1364,7 +1365,6 @@ static void tcp_v6_mapped_child_init(struct sock *newsk, const struct sock *sk)
newnp->ipv6_mc_list = NULL;
newnp->ipv6_ac_list = NULL;
- newnp->ipv6_fl_list = NULL;
newnp->pktoptions = NULL;
newnp->opt = NULL;
@@ -1427,15 +1427,17 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
newsk->sk_gso_type = SKB_GSO_TCPV6;
inet6_sk_rx_dst_set(newsk, skb);
- inet_sk(newsk)->pinet6 = tcp_inet6_sk(newsk);
+ newinet = inet_sk(newsk);
+ newinet->pinet6 = tcp_inet6_sk(newsk);
+ newinet->ipv6_fl_list = NULL;
+ newinet->inet_opt = NULL;
newtp = tcp_sk(newsk);
- newinet = inet_sk(newsk);
newnp = tcp_inet6_sk(newsk);
memcpy(newnp, np, sizeof(struct ipv6_pinfo));
- ip6_dst_store(newsk, dst, NULL, NULL);
+ ip6_dst_store(newsk, dst, NULL, false);
newsk->sk_v6_daddr = ireq->ir_v6_rmt_addr;
newnp->saddr = ireq->ir_v6_loc_addr;
@@ -1446,10 +1448,8 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
First: no IPv4 options.
*/
- newinet->inet_opt = NULL;
newnp->ipv6_mc_list = NULL;
newnp->ipv6_ac_list = NULL;
- newnp->ipv6_fl_list = NULL;
/* Clone RX bits */
newnp->rxopt.all = np->rxopt.all;
@@ -1605,7 +1605,8 @@ int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
by tcp. Feel free to propose better solution.
--ANK (980728)
*/
- if (np->rxopt.all && sk->sk_state != TCP_LISTEN)
+ if (np->rxopt.all &&
+ !((1 << sk->sk_state) & (TCPF_LISTEN | TCPF_CLOSE)))
opt_skb = skb_clone_and_charge_r(skb, sk);
if (sk->sk_state == TCP_ESTABLISHED) { /* Fast path */
@@ -2178,9 +2179,9 @@ static void get_tcp6_sock(struct seq_file *seq, struct sock *sp, int i)
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sp->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sp->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = jiffies;
diff --git a/net/ipv6/xfrm6_input.c b/net/ipv6/xfrm6_input.c
index c9e208c57a6b..07edef258984 100644
--- a/net/ipv6/xfrm6_input.c
+++ b/net/ipv6/xfrm6_input.c
@@ -71,8 +71,6 @@ int xfrm6_transport_finish(struct sk_buff *skb, int async)
NF_HOOK(NFPROTO_IPV6, NF_INET_PRE_ROUTING,
dev_net(dev), NULL, skb, dev, NULL,
xfrm6_transport_finish2);
- if (async)
- dev_put(dev);
return 0;
}
diff --git a/net/ipv6/xfrm6_output.c b/net/ipv6/xfrm6_output.c
index 512bdaf13699..44b221a09a0c 100644
--- a/net/ipv6/xfrm6_output.c
+++ b/net/ipv6/xfrm6_output.c
@@ -19,7 +19,10 @@
void xfrm6_local_rxpmtu(struct sk_buff *skb, u32 mtu)
{
struct flowi6 fl6;
- struct sock *sk = skb->sk;
+ struct sock *sk = skb_to_full_sk(skb);
+
+ if (!sk)
+ return;
fl6.flowi6_oif = sk->sk_bound_dev_if;
fl6.daddr = ipv6_hdr(skb)->daddr;
@@ -31,7 +34,10 @@ void xfrm6_local_error(struct sk_buff *skb, u32 mtu)
{
struct flowi6 fl6;
const struct ipv6hdr *hdr;
- struct sock *sk = skb->sk;
+ struct sock *sk = skb_to_full_sk(skb);
+
+ if (!sk)
+ return;
hdr = skb->encapsulation ? inner_ipv6_hdr(skb) : ipv6_hdr(skb);
fl6.fl6_dport = inet_sk(sk)->inet_dport;
diff --git a/net/llc/llc_c_ac.c b/net/llc/llc_c_ac.c
index 40ca3c1e42a2..5c8483f63ebb 100644
--- a/net/llc/llc_c_ac.c
+++ b/net/llc/llc_c_ac.c
@@ -443,7 +443,7 @@ int llc_conn_ac_resend_i_xxx_x_set_0_or_send_rr(struct sock *sk,
if (likely(!rc))
llc_conn_send_pdu(sk, nskb);
else
- kfree_skb(skb);
+ kfree_skb(nskb);
}
if (rc) {
nr = LLC_I_GET_NR(pdu);
diff --git a/net/llc/llc_s_ac.c b/net/llc/llc_s_ac.c
index 7a0cae9a8111..cc6fb29a9510 100644
--- a/net/llc/llc_s_ac.c
+++ b/net/llc/llc_s_ac.c
@@ -127,6 +127,8 @@ int llc_sap_action_send_xid_r(struct llc_sap *sap, struct sk_buff *skb)
rc = llc_mac_hdr_init(nskb, mac_sa, mac_da);
if (likely(!rc))
rc = dev_queue_xmit(nskb);
+ else
+ kfree_skb(nskb);
out:
return rc;
}
@@ -176,6 +178,8 @@ int llc_sap_action_send_test_r(struct llc_sap *sap, struct sk_buff *skb)
rc = llc_mac_hdr_init(nskb, mac_sa, mac_da);
if (likely(!rc))
rc = dev_queue_xmit(nskb);
+ else
+ kfree_skb(nskb);
out:
return rc;
}
diff --git a/net/llc/llc_sap.c b/net/llc/llc_sap.c
index 6cd03c2ae7d5..5d51ead8a2b3 100644
--- a/net/llc/llc_sap.c
+++ b/net/llc/llc_sap.c
@@ -25,12 +25,12 @@
#include <linux/llc.h>
#include <linux/slab.h>
-static int llc_mac_header_len(unsigned short devtype)
+static int llc_mac_header_len(struct net_device *dev)
{
- switch (devtype) {
+ switch (dev->type) {
case ARPHRD_ETHER:
case ARPHRD_LOOPBACK:
- return sizeof(struct ethhdr);
+ return LL_RESERVED_SPACE(dev);
}
return 0;
}
@@ -51,7 +51,7 @@ struct sk_buff *llc_alloc_frame(struct sock *sk, struct net_device *dev,
int hlen = type == LLC_PDU_TYPE_U ? 3 : 4;
struct sk_buff *skb;
- hlen += llc_mac_header_len(dev->type);
+ hlen += llc_mac_header_len(dev);
skb = alloc_skb(hlen + data_size, GFP_ATOMIC);
if (skb) {
diff --git a/net/mac80211/cfg.c b/net/mac80211/cfg.c
index f6c5de994898..5b65788f2f82 100644
--- a/net/mac80211/cfg.c
+++ b/net/mac80211/cfg.c
@@ -63,12 +63,14 @@ static void ieee80211_set_mu_mimo_follow(struct ieee80211_sub_if_data *sdata,
memcpy(sdata->vif.bss_conf.mu_group.position,
params->vht_mumimo_groups + WLAN_MEMBERSHIP_LEN,
WLAN_USER_POSITION_LEN);
- ieee80211_link_info_change_notify(sdata, &sdata->deflink,
- BSS_CHANGED_MU_GROUPS);
+
/* don't care about endianness - just check for 0 */
memcpy(&membership, params->vht_mumimo_groups,
WLAN_MEMBERSHIP_LEN);
mu_mimo_groups = membership != 0;
+
+ /* Unset following if configured explicitly */
+ eth_broadcast_addr(sdata->u.mntr.mu_follow_addr);
}
if (params->vht_mumimo_follow_addr) {
@@ -76,16 +78,26 @@ static void ieee80211_set_mu_mimo_follow(struct ieee80211_sub_if_data *sdata,
is_valid_ether_addr(params->vht_mumimo_follow_addr);
ether_addr_copy(sdata->u.mntr.mu_follow_addr,
params->vht_mumimo_follow_addr);
+
+ /* Unset current membership until a management frame is RXed */
+ memset(sdata->vif.bss_conf.mu_group.membership, 0,
+ WLAN_MEMBERSHIP_LEN);
}
sdata->vif.bss_conf.mu_mimo_owner = mu_mimo_groups || mu_mimo_follow;
+
+ /* Notify only after setting mu_mimo_owner */
+ if (sdata->vif.bss_conf.mu_mimo_owner &&
+ sdata->flags & IEEE80211_SDATA_IN_DRIVER)
+ ieee80211_link_info_change_notify(sdata, &sdata->deflink,
+ BSS_CHANGED_MU_GROUPS);
}
static int ieee80211_set_mon_options(struct ieee80211_sub_if_data *sdata,
struct vif_params *params)
{
struct ieee80211_local *local = sdata->local;
- struct ieee80211_sub_if_data *monitor_sdata;
+ struct ieee80211_sub_if_data *monitor_sdata = NULL;
/* check flags first */
if (params->flags && ieee80211_sdata_running(sdata)) {
@@ -104,18 +116,29 @@ static int ieee80211_set_mon_options(struct ieee80211_sub_if_data *sdata,
return -EBUSY;
}
- /* also validate MU-MIMO change */
- monitor_sdata = wiphy_dereference(local->hw.wiphy,
- local->monitor_sdata);
+ /* TXQs are reserved in ieee80211_if_add() and cannot be added later */
+ if ((params->flags & MONITOR_FLAG_ACTIVE) && !sdata->vif.txq)
+ return -EOPNOTSUPP;
- if (!monitor_sdata &&
+ /* validate whether MU-MIMO can be configured */
+ if (!ieee80211_hw_check(&local->hw, WANT_MONITOR_VIF) &&
(params->vht_mumimo_groups || params->vht_mumimo_follow_addr))
return -EOPNOTSUPP;
+ /* Also update dependent monitor_sdata if required */
+ if (test_bit(SDATA_STATE_RUNNING, &sdata->state))
+ monitor_sdata = wiphy_dereference(local->hw.wiphy,
+ local->monitor_sdata);
+
/* apply all changes now - no failures allowed */
- if (monitor_sdata && ieee80211_hw_check(&local->hw, WANT_MONITOR_VIF))
- ieee80211_set_mu_mimo_follow(monitor_sdata, params);
+ if (ieee80211_hw_check(&local->hw, WANT_MONITOR_VIF)) {
+ /* This is copied in when the VIF is activated */
+ ieee80211_set_mu_mimo_follow(sdata, params);
+
+ if (monitor_sdata)
+ ieee80211_set_mu_mimo_follow(monitor_sdata, params);
+ }
if (params->flags) {
if (ieee80211_sdata_running(sdata)) {
@@ -1514,6 +1537,9 @@ static int ieee80211_start_ap(struct wiphy *wiphy, struct net_device *dev,
return 0;
error:
+ link_conf->enable_beacon = false;
+ link_conf->beacon_int = prev_beacon_int;
+ sdata->vif.cfg.ssid_len = 0;
ieee80211_link_release_channel(link);
return err;
@@ -2688,7 +2714,11 @@ static int ieee80211_join_mesh(struct wiphy *wiphy, struct net_device *dev,
if (err)
return err;
- return ieee80211_start_mesh(sdata);
+ err = ieee80211_start_mesh(sdata);
+ if (err)
+ ieee80211_link_release_channel(&sdata->deflink);
+
+ return err;
}
static int ieee80211_leave_mesh(struct wiphy *wiphy, struct net_device *dev)
@@ -2844,7 +2874,7 @@ static int ieee80211_set_txq_params(struct wiphy *wiphy,
static int ieee80211_suspend(struct wiphy *wiphy,
struct cfg80211_wowlan *wowlan)
{
- return __ieee80211_suspend(wiphy_priv(wiphy), wowlan);
+ return __ieee80211_suspend(wiphy_priv(wiphy), wowlan, false);
}
static int ieee80211_resume(struct wiphy *wiphy)
@@ -3419,6 +3449,9 @@ static int ieee80211_set_bitrate_mask(struct wiphy *wiphy,
if (!ieee80211_sdata_running(sdata))
return -ENETDOWN;
+ if (!(sdata->flags & IEEE80211_SDATA_IN_DRIVER))
+ return -ENETDOWN;
+
/*
* If active validate the setting and reject it if it doesn't leave
* at least one basic rate usable, since we really have to be able
diff --git a/net/mac80211/debugfs.c b/net/mac80211/debugfs.c
index e9b3b2c7b6fa..28b1355cc062 100644
--- a/net/mac80211/debugfs.c
+++ b/net/mac80211/debugfs.c
@@ -418,7 +418,7 @@ static ssize_t reset_write(struct file *file, const char __user *user_buf,
rtnl_lock();
wiphy_lock(local->hw.wiphy);
- __ieee80211_suspend(&local->hw, NULL);
+ __ieee80211_suspend(&local->hw, NULL, true);
ret = __ieee80211_resume(&local->hw);
wiphy_unlock(local->hw.wiphy);
diff --git a/net/mac80211/debugfs_netdev.c b/net/mac80211/debugfs_netdev.c
index d0b145888e13..d3519bbc4285 100644
--- a/net/mac80211/debugfs_netdev.c
+++ b/net/mac80211/debugfs_netdev.c
@@ -659,6 +659,9 @@ static ssize_t ieee80211_if_fmt_tsf(
struct ieee80211_local *local = sdata->local;
u64 tsf;
+ if (!ieee80211_sdata_running((struct ieee80211_sub_if_data *)sdata))
+ return -ENETDOWN;
+
tsf = drv_get_tsf(local, (struct ieee80211_sub_if_data *)sdata);
return scnprintf(buf, buflen, "0x%016llx\n", (unsigned long long) tsf);
@@ -672,6 +675,9 @@ static ssize_t ieee80211_if_parse_tsf(
int ret;
int tsf_is_delta = 0;
+ if (!ieee80211_sdata_running(sdata))
+ return -ENETDOWN;
+
if (strncmp(buf, "reset", 5) == 0) {
if (local->ops->reset_tsf) {
drv_reset_tsf(local, sdata);
@@ -731,6 +737,9 @@ static ssize_t ieee80211_if_parse_active_links(struct ieee80211_sub_if_data *sda
if (kstrtou16(buf, 0, &active_links) || !active_links)
return -EINVAL;
+ if (!ieee80211_sdata_running(sdata))
+ return -ENETDOWN;
+
return ieee80211_set_active_links(&sdata->vif, active_links) ?: buflen;
}
IEEE80211_IF_FILE_RW(active_links);
diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h
index 3b00b3f9f17d..3486f3d55c42 100644
--- a/net/mac80211/ieee80211_i.h
+++ b/net/mac80211/ieee80211_i.h
@@ -2045,7 +2045,8 @@ void ieee80211_adjust_monitor_flags(struct ieee80211_sub_if_data *sdata,
const int offset);
int ieee80211_do_open(struct wireless_dev *wdev, bool coming_up);
void ieee80211_sdata_stop(struct ieee80211_sub_if_data *sdata);
-int ieee80211_add_virtual_monitor(struct ieee80211_local *local);
+int ieee80211_add_virtual_monitor(struct ieee80211_local *local,
+ struct ieee80211_sub_if_data *creator_sdata);
void ieee80211_del_virtual_monitor(struct ieee80211_local *local);
bool __ieee80211_recalc_txpower(struct ieee80211_sub_if_data *sdata);
@@ -2287,7 +2288,7 @@ int ieee80211_reconfig(struct ieee80211_local *local);
void ieee80211_stop_device(struct ieee80211_local *local, bool suspend);
int __ieee80211_suspend(struct ieee80211_hw *hw,
- struct cfg80211_wowlan *wowlan);
+ struct cfg80211_wowlan *wowlan, bool reset);
static inline int __ieee80211_resume(struct ieee80211_hw *hw)
{
diff --git a/net/mac80211/iface.c b/net/mac80211/iface.c
index 7a79b40d23b3..155c6269119c 100644
--- a/net/mac80211/iface.c
+++ b/net/mac80211/iface.c
@@ -618,6 +618,8 @@ static void ieee80211_do_stop(struct ieee80211_sub_if_data *sdata, bool going_do
RCU_INIT_POINTER(sdata->vif.bss_conf.chanctx_conf, NULL);
/* see comment in the default case below */
ieee80211_free_keys(sdata, true);
+ /* increased by AP value on ifup, so reset on ifdown */
+ sdata->crypto_tx_tailroom_needed_cnt = 0;
/* no need to tell driver */
break;
case NL80211_IFTYPE_MONITOR:
@@ -761,8 +763,9 @@ static void ieee80211_do_stop(struct ieee80211_sub_if_data *sdata, bool going_do
ieee80211_configure_filter(local);
ieee80211_hw_config(local, hw_reconf_flags);
+ /* Passing NULL means an interface is picked for configuration */
if (local->monitors == local->open_count)
- ieee80211_add_virtual_monitor(local);
+ ieee80211_add_virtual_monitor(local, NULL);
}
static void ieee80211_stop_mbssid(struct ieee80211_sub_if_data *sdata)
@@ -862,9 +865,33 @@ static void ieee80211_teardown_sdata(struct ieee80211_sub_if_data *sdata)
ieee80211_link_stop(&sdata->deflink);
}
+/*
+ * The netdev can be unregistered without mac80211 doing it, e.g. by the netdev
+ * core when cfg80211 couldn't move it out of a network namespace that's being
+ * destroyed. Drop it from the interface list either way.
+ */
+static void ieee80211_unlist_sdata(struct ieee80211_sub_if_data *sdata)
+{
+ struct ieee80211_local *local = sdata->local;
+ struct ieee80211_sub_if_data *iter;
+
+ ASSERT_RTNL();
+
+ list_for_each_entry(iter, &local->interfaces, list) {
+ if (iter != sdata)
+ continue;
+ guard(mutex)(&local->iflist_mtx);
+ list_del_rcu(&sdata->list);
+ return;
+ }
+}
+
static void ieee80211_uninit(struct net_device *dev)
{
- ieee80211_teardown_sdata(IEEE80211_DEV_TO_SUB_IF(dev));
+ struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev);
+
+ ieee80211_unlist_sdata(sdata);
+ ieee80211_teardown_sdata(sdata);
}
static int ieee80211_netdev_setup_tc(struct net_device *dev,
@@ -873,6 +900,9 @@ static int ieee80211_netdev_setup_tc(struct net_device *dev,
struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev);
struct ieee80211_local *local = sdata->local;
+ if (sdata->vif.type == NL80211_IFTYPE_AP_VLAN)
+ return -EOPNOTSUPP;
+
return drv_net_setup_tc(local, sdata, dev, type, type_data);
}
@@ -902,7 +932,7 @@ static u16 ieee80211_monitor_select_queue(struct net_device *dev,
/* reset flags and info before parsing radiotap header */
memset(info, 0, sizeof(*info));
- if (!ieee80211_parse_tx_radiotap(skb, dev))
+ if (!ieee80211_parse_tx_radiotap(skb, dev, NULL))
return 0; /* doesn't matter, frame will be dropped */
len_rthdr = ieee80211_get_radiotap_len(skb->data);
@@ -1159,7 +1189,8 @@ static void ieee80211_sdata_init(struct ieee80211_local *local,
ieee80211_link_init(sdata, -1, &sdata->deflink, &sdata->vif.bss_conf);
}
-int ieee80211_add_virtual_monitor(struct ieee80211_local *local)
+int ieee80211_add_virtual_monitor(struct ieee80211_local *local,
+ struct ieee80211_sub_if_data *creator_sdata)
{
struct ieee80211_sub_if_data *sdata;
int ret;
@@ -1167,8 +1198,10 @@ int ieee80211_add_virtual_monitor(struct ieee80211_local *local)
ASSERT_RTNL();
lockdep_assert_wiphy(local->hw.wiphy);
- if (local->monitor_sdata)
- return 0;
+ /* Already have a monitor set up, configure it */
+ sdata = wiphy_dereference(local->hw.wiphy, local->monitor_sdata);
+ if (sdata)
+ goto configure_monitor;
sdata = kzalloc(sizeof(*sdata) + local->hw.vif_data_size, GFP_KERNEL);
if (!sdata)
@@ -1222,6 +1255,32 @@ int ieee80211_add_virtual_monitor(struct ieee80211_local *local)
skb_queue_head_init(&sdata->status_queue);
wiphy_work_init(&sdata->work, ieee80211_iface_work);
+configure_monitor:
+ /* Copy in the MU-MIMO configuration if set */
+ if (!creator_sdata) {
+ struct ieee80211_sub_if_data *other;
+
+ list_for_each_entry(other, &local->mon_list, u.mntr.list) {
+ if (!other->vif.bss_conf.mu_mimo_owner)
+ continue;
+
+ creator_sdata = other;
+ break;
+ }
+ }
+
+ if (creator_sdata && creator_sdata->vif.bss_conf.mu_mimo_owner) {
+ sdata->vif.bss_conf.mu_mimo_owner = true;
+ memcpy(&sdata->vif.bss_conf.mu_group,
+ &creator_sdata->vif.bss_conf.mu_group,
+ sizeof(sdata->vif.bss_conf.mu_group));
+ memcpy(&sdata->u.mntr.mu_follow_addr,
+ creator_sdata->u.mntr.mu_follow_addr, ETH_ALEN);
+
+ ieee80211_link_info_change_notify(sdata, &sdata->deflink,
+ BSS_CHANGED_MU_GROUPS);
+ }
+
return 0;
}
@@ -1376,8 +1435,9 @@ int ieee80211_do_open(struct wireless_dev *wdev, bool coming_up)
res = drv_add_interface(local, sdata);
if (res)
goto err_stop;
- } else if (local->monitors == 0 && local->open_count == 0) {
- res = ieee80211_add_virtual_monitor(local);
+ } else if (local->monitors == local->open_count) {
+ /* add/configure if there is no non-monitor interface */
+ res = ieee80211_add_virtual_monitor(local, sdata);
if (res)
goto err_stop;
}
diff --git a/net/mac80211/main.c b/net/mac80211/main.c
index 84cc2a21a9a5..a65d49e4402a 100644
--- a/net/mac80211/main.c
+++ b/net/mac80211/main.c
@@ -1424,6 +1424,10 @@ int ieee80211_register_hw(struct ieee80211_hw *hw)
sizeof(struct ieee80211_he_mcs_nss_supp) +
IEEE80211_HE_PPE_THRES_MAX_LEN;
+ if (local->hw.wiphy->bands[NL80211_BAND_6GHZ])
+ local->scan_ies_len +=
+ 3 + sizeof(struct ieee80211_he_6ghz_capa);
+
if (supp_eht)
local->scan_ies_len +=
3 + sizeof(struct ieee80211_eht_cap_elem) +
diff --git a/net/mac80211/mesh.c b/net/mac80211/mesh.c
index 253f4b064284..258c86592178 100644
--- a/net/mac80211/mesh.c
+++ b/net/mac80211/mesh.c
@@ -1217,6 +1217,21 @@ int ieee80211_start_mesh(struct ieee80211_sub_if_data *sdata)
return 0;
}
+static void ieee80211_mesh_reset_csa(struct ieee80211_sub_if_data *sdata)
+{
+ struct ieee80211_if_mesh *ifmsh = &sdata->u.mesh;
+ struct mesh_csa_settings *csa;
+
+ /* Reset the TTL value and Initiator flag */
+ ifmsh->csa_role = IEEE80211_MESH_CSA_ROLE_NONE;
+ ifmsh->chsw_ttl = 0;
+
+ /* Remove the CSA and MCSP elements from the beacon */
+ csa = sdata_dereference(ifmsh->csa, sdata);
+ RCU_INIT_POINTER(ifmsh->csa, NULL);
+ kfree_rcu(csa, rcu_head);
+}
+
void ieee80211_stop_mesh(struct ieee80211_sub_if_data *sdata)
{
struct ieee80211_local *local = sdata->local;
@@ -1225,6 +1240,11 @@ void ieee80211_stop_mesh(struct ieee80211_sub_if_data *sdata)
netif_carrier_off(sdata->dev);
+ /* abort any running channel switch */
+ sdata->vif.bss_conf.csa_active = false;
+ ieee80211_mesh_reset_csa(sdata);
+ ieee80211_vif_unblock_queues_csa(sdata);
+
/* flush STAs and mpaths on this iface */
sta_info_flush(sdata, -1);
ieee80211_free_keys(sdata, true);
@@ -1527,19 +1547,10 @@ static void ieee80211_mesh_rx_bcn_presp(struct ieee80211_sub_if_data *sdata,
int ieee80211_mesh_finish_csa(struct ieee80211_sub_if_data *sdata, u64 *changed)
{
- struct ieee80211_if_mesh *ifmsh = &sdata->u.mesh;
- struct mesh_csa_settings *tmp_csa_settings;
- int ret = 0;
+ int ret;
- /* Reset the TTL value and Initiator flag */
- ifmsh->csa_role = IEEE80211_MESH_CSA_ROLE_NONE;
- ifmsh->chsw_ttl = 0;
+ ieee80211_mesh_reset_csa(sdata);
- /* Remove the CSA and MCSP elements from the beacon */
- tmp_csa_settings = sdata_dereference(ifmsh->csa, sdata);
- RCU_INIT_POINTER(ifmsh->csa, NULL);
- if (tmp_csa_settings)
- kfree_rcu(tmp_csa_settings, rcu_head);
ret = ieee80211_mesh_rebuild_beacon(sdata);
if (ret)
return -EINVAL;
@@ -1573,7 +1584,6 @@ int ieee80211_mesh_csa_beacon(struct ieee80211_sub_if_data *sdata,
ret = ieee80211_mesh_rebuild_beacon(sdata);
if (ret) {
- tmp_csa_settings = rcu_dereference(ifmsh->csa);
RCU_INIT_POINTER(ifmsh->csa, NULL);
kfree_rcu(tmp_csa_settings, rcu_head);
return ret;
diff --git a/net/mac80211/offchannel.c b/net/mac80211/offchannel.c
index 29fab7ae47b4..0d9807526ecb 100644
--- a/net/mac80211/offchannel.c
+++ b/net/mac80211/offchannel.c
@@ -462,6 +462,13 @@ static void __ieee80211_roc_work(struct ieee80211_local *local)
return;
if (!roc->started) {
+ /*
+ * The work can be started by a previous ROC work, but a scan
+ * can get between things; scan finish will retrigger us.
+ */
+ if (local->scanning)
+ return;
+
WARN_ON(!local->emulate_chanctx);
_ieee80211_start_next_roc(local);
} else {
diff --git a/net/mac80211/pm.c b/net/mac80211/pm.c
index 7be52345f218..56c222bdd490 100644
--- a/net/mac80211/pm.c
+++ b/net/mac80211/pm.c
@@ -18,7 +18,8 @@ static void ieee80211_sched_scan_cancel(struct ieee80211_local *local)
cfg80211_sched_scan_stopped_locked(local->hw.wiphy, 0);
}
-int __ieee80211_suspend(struct ieee80211_hw *hw, struct cfg80211_wowlan *wowlan)
+int __ieee80211_suspend(struct ieee80211_hw *hw, struct cfg80211_wowlan *wowlan,
+ bool reset)
{
struct ieee80211_local *local = hw_to_local(hw);
struct ieee80211_sub_if_data *sdata;
@@ -166,9 +167,10 @@ int __ieee80211_suspend(struct ieee80211_hw *hw, struct cfg80211_wowlan *wowlan)
/*
* We disconnected on all interfaces before suspend, all channel
- * contexts should be released.
+ * contexts should be released, but on 'reset' debugfs that's
+ * not true so don't check there.
*/
- WARN_ON(!list_empty(&local->chanctx_list));
+ WARN_ON(!reset && !list_empty(&local->chanctx_list));
/* stop hardware - this must stop RX */
ieee80211_stop_device(local, true);
diff --git a/net/mac80211/scan.c b/net/mac80211/scan.c
index 8675d2e99c56..19e374fa0079 100644
--- a/net/mac80211/scan.c
+++ b/net/mac80211/scan.c
@@ -1180,14 +1180,14 @@ int ieee80211_request_ibss_scan(struct ieee80211_sub_if_data *sdata,
unsigned int n_channels)
{
struct ieee80211_local *local = sdata->local;
- int ret = -EBUSY, i, n_ch = 0;
+ int i, n_ch = 0;
enum nl80211_band band;
lockdep_assert_wiphy(local->hw.wiphy);
/* busy scanning */
if (local->scan_req)
- goto unlock;
+ return -EBUSY;
/* fill internal scan request */
if (!channels) {
@@ -1204,7 +1204,9 @@ int ieee80211_request_ibss_scan(struct ieee80211_sub_if_data *sdata,
&local->hw.wiphy->bands[band]->channels[i];
if (tmp_ch->flags & (IEEE80211_CHAN_NO_IR |
- IEEE80211_CHAN_DISABLED))
+ IEEE80211_CHAN_DISABLED) ||
+ !cfg80211_wdev_channel_allowed(&sdata->wdev,
+ tmp_ch))
continue;
local->int_scan_req->channels[n_ch] = tmp_ch;
@@ -1212,22 +1214,24 @@ int ieee80211_request_ibss_scan(struct ieee80211_sub_if_data *sdata,
}
}
- if (WARN_ON_ONCE(n_ch == 0))
- goto unlock;
+ if (n_ch == 0)
+ return -EINVAL;
local->int_scan_req->n_channels = n_ch;
} else {
for (i = 0; i < n_channels; i++) {
if (channels[i]->flags & (IEEE80211_CHAN_NO_IR |
- IEEE80211_CHAN_DISABLED))
+ IEEE80211_CHAN_DISABLED) ||
+ !cfg80211_wdev_channel_allowed(&sdata->wdev,
+ channels[i]))
continue;
local->int_scan_req->channels[n_ch] = channels[i];
n_ch++;
}
- if (WARN_ON_ONCE(n_ch == 0))
- goto unlock;
+ if (n_ch == 0)
+ return -EINVAL;
local->int_scan_req->n_channels = n_ch;
}
@@ -1237,9 +1241,7 @@ int ieee80211_request_ibss_scan(struct ieee80211_sub_if_data *sdata,
memcpy(local->int_scan_req->ssids[0].ssid, ssid, IEEE80211_MAX_SSID_LEN);
local->int_scan_req->ssids[0].ssid_len = ssid_len;
- ret = __ieee80211_start_scan(sdata, sdata->local->int_scan_req);
- unlock:
- return ret;
+ return __ieee80211_start_scan(sdata, sdata->local->int_scan_req);
}
void ieee80211_scan_cancel(struct ieee80211_local *local)
diff --git a/net/mac80211/tdls.c b/net/mac80211/tdls.c
index 92ab7be3d482..b4bedf0b2b55 100644
--- a/net/mac80211/tdls.c
+++ b/net/mac80211/tdls.c
@@ -1285,6 +1285,24 @@ int ieee80211_tdls_mgmt(struct wiphy *wiphy, struct net_device *dev,
peer_capability, initiator,
extra_ies, extra_ies_len);
break;
+ case WLAN_TDLS_SETUP_CONFIRM: {
+ struct sta_info *sta;
+
+ sta = sta_info_get(sdata, peer);
+ if (!sta || !sta->sta.tdls) {
+ ret = -ENOLINK;
+ break;
+ }
+
+ ret = ieee80211_tdls_prep_mgmt_packet(wiphy, dev, peer,
+ link_id, action_code,
+ dialog_token,
+ status_code,
+ peer_capability,
+ initiator, extra_ies,
+ extra_ies_len, 0, NULL);
+ break;
+ }
case WLAN_TDLS_DISCOVERY_REQUEST:
/*
* Protect the discovery so we can hear the TDLS discovery
@@ -1293,7 +1311,6 @@ int ieee80211_tdls_mgmt(struct wiphy *wiphy, struct net_device *dev,
*/
drv_mgd_protect_tdls_discover(sdata->local, sdata, link_id);
fallthrough;
- case WLAN_TDLS_SETUP_CONFIRM:
case WLAN_PUB_ACTION_TDLS_DISCOVER_RES:
/* no special handling */
ret = ieee80211_tdls_prep_mgmt_packet(wiphy, dev, peer,
diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c
index 9bc1a80e8057..228c717ea314 100644
--- a/net/mac80211/tx.c
+++ b/net/mac80211/tx.c
@@ -2084,8 +2084,29 @@ static bool ieee80211_validate_radiotap_len(struct sk_buff *skb)
return true;
}
+static bool ieee80211_rate_bw_usable(u16 rate_flags,
+ const struct cfg80211_chan_def *chandef)
+{
+ int width;
+
+ if (!chandef)
+ return true;
+
+ if (rate_flags & IEEE80211_TX_RC_160_MHZ_WIDTH)
+ width = 160;
+ else if (rate_flags & IEEE80211_TX_RC_80_MHZ_WIDTH)
+ width = 80;
+ else if (rate_flags & IEEE80211_TX_RC_40_MHZ_WIDTH)
+ width = 40;
+ else
+ return true;
+
+ return width <= cfg80211_chandef_get_width(chandef);
+}
+
bool ieee80211_parse_tx_radiotap(struct sk_buff *skb,
- struct net_device *dev)
+ struct net_device *dev,
+ const struct cfg80211_chan_def *chandef)
{
struct ieee80211_local *local = wdev_priv(dev->ieee80211_ptr);
struct ieee80211_radiotap_iterator iterator;
@@ -2259,6 +2280,9 @@ bool ieee80211_parse_tx_radiotap(struct sk_buff *skb,
struct ieee80211_supported_band *sband =
local->hw.wiphy->bands[info->band];
+ if (!ieee80211_rate_bw_usable(rate_flags, chandef))
+ return false;
+
info->control.flags |= IEEE80211_TX_CTRL_RATE_INJECT;
for (i = 0; i < IEEE80211_TX_MAX_RATES; i++) {
@@ -2448,7 +2472,7 @@ netdev_tx_t ieee80211_monitor_start_xmit(struct sk_buff *skb,
* selected chandef above to accurately set injection rates and
* retransmissions.
*/
- if (!ieee80211_parse_tx_radiotap(skb, dev))
+ if (!ieee80211_parse_tx_radiotap(skb, dev, chandef))
goto fail_rcu;
/* remove the injection radiotap header */
@@ -2903,10 +2927,23 @@ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata,
*/
skb = skb_share_check(skb, GFP_ATOMIC);
if (unlikely(!skb)) {
- ret = -ENOMEM;
- goto free;
+ /* skb_share_check() already freed the skb */
+ if (info_id)
+ ieee80211_remove_ack_skb(local, info_id);
+ return ERR_PTR(-ENOMEM);
}
+ /* set this up so failure paths can clean up ack skb */
+ info = IEEE80211_SKB_CB(skb);
+ memset(info, 0, sizeof(*info));
+
+ info->flags = info_flags;
+ if (info_id) {
+ info->status_data = info_id;
+ info->status_data_idr = 1;
+ }
+ info->band = band;
+
hdr.frame_control = fc;
hdr.duration_id = 0;
hdr.seq_ctrl = 0;
@@ -2945,10 +2982,8 @@ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata,
head_need += local->tx_headroom;
head_need = max_t(int, 0, head_need);
if (ieee80211_skb_resize(sdata, skb, head_need, ENCRYPT_DATA)) {
- ieee80211_free_txskb(&local->hw, skb);
- skb = NULL;
ret = -ENOMEM;
- goto free;
+ goto free_txskb;
}
}
@@ -2975,16 +3010,6 @@ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata,
skb_reset_mac_header(skb);
- info = IEEE80211_SKB_CB(skb);
- memset(info, 0, sizeof(*info));
-
- info->flags = info_flags;
- if (info_id) {
- info->status_data = info_id;
- info->status_data_idr = 1;
- }
- info->band = band;
-
if (likely(!cookie)) {
ctrl_flags |= u32_encode_bits(link_id,
IEEE80211_TX_CTRL_MLO_LINK);
@@ -3008,16 +3033,17 @@ static struct sk_buff *ieee80211_build_hdr(struct ieee80211_sub_if_data *sdata,
pre_conf_link_id, link_id);
#endif
ret = -EINVAL;
- goto free;
+ goto free_txskb;
}
}
info->control.flags = ctrl_flags;
return skb;
+ free_txskb:
+ ieee80211_free_txskb(&local->hw, skb);
+ return ERR_PTR(ret);
free:
- if (info_id)
- ieee80211_remove_ack_skb(local, info_id);
kfree_skb(skb);
return ERR_PTR(ret);
}
diff --git a/net/mac80211/util.c b/net/mac80211/util.c
index 8ffd7e064375..2dfb9c089b47 100644
--- a/net/mac80211/util.c
+++ b/net/mac80211/util.c
@@ -2144,8 +2144,9 @@ int ieee80211_reconfig(struct ieee80211_local *local)
wake_up:
+ /* Passing NULL means an interface is picked for configuration */
if (local->monitors == local->open_count && local->monitors > 0)
- ieee80211_add_virtual_monitor(local);
+ ieee80211_add_virtual_monitor(local, NULL);
/*
* Clear the WLAN_STA_BLOCK_BA flag so new aggregation
diff --git a/net/mctp/route.c b/net/mctp/route.c
index 35a0681123a3..eb7c1166d11b 100644
--- a/net/mctp/route.c
+++ b/net/mctp/route.c
@@ -775,7 +775,7 @@ static struct mctp_sk_key *mctp_lookup_prealloc_tag(struct mctp_sock *msk,
spin_lock_irqsave(&mns->keys_lock, flags);
- hlist_for_each_entry(tmp, &mns->keys, hlist) {
+ hlist_for_each_entry(tmp, &msk->keys, sklist) {
if (tmp->net != netid)
continue;
diff --git a/net/mptcp/Makefile b/net/mptcp/Makefile
index bcf1dbf3a432..8d5efaed6f98 100644
--- a/net/mptcp/Makefile
+++ b/net/mptcp/Makefile
@@ -2,7 +2,7 @@
obj-$(CONFIG_MPTCP) += mptcp.o
mptcp-y := protocol.o subflow.o options.o token.o crypto.o ctrl.o pm.o diag.o \
- mib.o pm_netlink.o sockopt.o pm_userspace.o fastopen.o sched.o \
+ mib.o pm_kernel.o sockopt.o pm_userspace.o fastopen.o sched.o \
mptcp_pm_gen.o
obj-$(CONFIG_SYN_COOKIES) += syncookies.o
diff --git a/net/mptcp/ctrl.c b/net/mptcp/ctrl.c
index 0d60556cfefa..34169b1cb924 100644
--- a/net/mptcp/ctrl.c
+++ b/net/mptcp/ctrl.c
@@ -407,9 +407,9 @@ void mptcp_active_detect_blackhole(struct sock *ssk, bool expired)
if (subflow->request_mptcp && ssk->sk_state == TCP_SYN_SENT) {
if (timeouts == 2 || (timeouts < 2 && expired)) {
- MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_MPCAPABLEACTIVEDROP);
subflow->mpc_drop = 1;
- mptcp_subflow_early_fallback(mptcp_sk(subflow->conn), subflow);
+ mptcp_subflow_early_fallback(mptcp_sk(subflow->conn), subflow,
+ MPTCP_MIB_MPCAPABLEACTIVEDROP);
}
} else if (ssk->sk_state == TCP_SYN_SENT) {
subflow->mpc_drop = 0;
diff --git a/net/mptcp/mib.c b/net/mptcp/mib.c
index 19eb9292bd60..7d65c507427d 100644
--- a/net/mptcp/mib.c
+++ b/net/mptcp/mib.c
@@ -79,6 +79,11 @@ static const struct snmp_mib mptcp_snmp_list[] = {
SNMP_MIB_ITEM("RcvWndConflict", MPTCP_MIB_RCVWNDCONFLICT),
SNMP_MIB_ITEM("MPCurrEstab", MPTCP_MIB_CURRESTAB),
SNMP_MIB_ITEM("Blackhole", MPTCP_MIB_BLACKHOLE),
+ SNMP_MIB_ITEM("MPCapableDataFallback", MPTCP_MIB_MPCAPABLEDATAFALLBACK),
+ SNMP_MIB_ITEM("MD5SigFallback", MPTCP_MIB_MD5SIGFALLBACK),
+ SNMP_MIB_ITEM("DssFallback", MPTCP_MIB_DSSFALLBACK),
+ SNMP_MIB_ITEM("SimultConnectFallback", MPTCP_MIB_SIMULTCONNFALLBACK),
+ SNMP_MIB_ITEM("FallbackFailed", MPTCP_MIB_FALLBACKFAILED),
SNMP_MIB_SENTINEL
};
diff --git a/net/mptcp/mib.h b/net/mptcp/mib.h
index 128282982843..8776cfc77b97 100644
--- a/net/mptcp/mib.h
+++ b/net/mptcp/mib.h
@@ -80,6 +80,11 @@ enum linux_mptcp_mib_field {
MPTCP_MIB_RCVWNDCONFLICT, /* Conflict with while updating msk rcv wnd */
MPTCP_MIB_CURRESTAB, /* Current established MPTCP connections */
MPTCP_MIB_BLACKHOLE, /* A blackhole has been detected */
+ MPTCP_MIB_MPCAPABLEDATAFALLBACK, /* No DSS/MPC+data on first established packet */
+ MPTCP_MIB_MD5SIGFALLBACK, /* Conflicting TCP option enabled */
+ MPTCP_MIB_DSSFALLBACK, /* Bad or missing DSS */
+ MPTCP_MIB_SIMULTCONNFALLBACK, /* Simultaneous connect */
+ MPTCP_MIB_FALLBACKFAILED, /* Can't fallback due to msk status */
__MPTCP_MIB_MAX
};
diff --git a/net/mptcp/options.c b/net/mptcp/options.c
index d238dd374ce9..b920181a6a13 100644
--- a/net/mptcp/options.c
+++ b/net/mptcp/options.c
@@ -473,7 +473,7 @@ bool mptcp_syn_options(struct sock *sk, const struct sk_buff *skb,
subflow->snd_isn = TCP_SKB_CB(skb)->end_seq;
if (subflow->request_mptcp) {
if (unlikely(subflow_simultaneous_connect(sk))) {
- WARN_ON_ONCE(!mptcp_try_fallback(sk));
+ WARN_ON_ONCE(!mptcp_try_fallback(sk, MPTCP_MIB_SIMULTCONNFALLBACK));
/* Ensure mptcp_finish_connect() will not process the
* MPC handshake.
@@ -1043,8 +1043,10 @@ static bool check_fully_established(struct mptcp_sock *msk, struct sock *ssk,
if (subflow->mp_join)
goto reset;
subflow->mp_capable = 0;
- if (!mptcp_try_fallback(ssk))
+ if (!mptcp_try_fallback(ssk, MPTCP_MIB_MPCAPABLEDATAFALLBACK)) {
+ MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_FALLBACKFAILED);
goto reset;
+ }
pr_fallback(msk);
return false;
}
diff --git a/net/mptcp/pm_kernel.c b/net/mptcp/pm_kernel.c
new file mode 100644
index 000000000000..4de1375bf2a5
--- /dev/null
+++ b/net/mptcp/pm_kernel.c
@@ -0,0 +1,2610 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Multipath TCP
+ *
+ * Copyright (c) 2020, Red Hat, Inc.
+ */
+
+#define pr_fmt(fmt) "MPTCP: " fmt
+
+#include <linux/inet.h>
+#include <linux/kernel.h>
+#include <net/inet_common.h>
+#include <net/netns/generic.h>
+#include <net/mptcp.h>
+
+#include "protocol.h"
+#include "mib.h"
+#include "mptcp_pm_gen.h"
+
+static int pm_nl_pernet_id;
+
+struct mptcp_pm_add_addr {
+ struct list_head list;
+ struct mptcp_addr_info addr;
+ u8 retrans_times;
+ bool timer_done;
+ struct timer_list add_timer;
+ struct mptcp_sock *sock;
+ struct rcu_head rcu;
+};
+
+struct pm_nl_pernet {
+ /* protects pernet updates */
+ spinlock_t lock;
+ struct list_head local_addr_list;
+ unsigned int addrs;
+ unsigned int stale_loss_cnt;
+ unsigned int add_addr_signal_max;
+ unsigned int add_addr_accept_max;
+ unsigned int local_addr_max;
+ unsigned int subflows_max;
+ unsigned int next_id;
+ DECLARE_BITMAP(id_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
+};
+
+#define MPTCP_PM_ADDR_MAX 8
+#define ADD_ADDR_RETRANS_MAX 3
+
+static struct pm_nl_pernet *pm_nl_get_pernet(const struct net *net)
+{
+ return net_generic(net, pm_nl_pernet_id);
+}
+
+static struct pm_nl_pernet *
+pm_nl_get_pernet_from_msk(const struct mptcp_sock *msk)
+{
+ return pm_nl_get_pernet(sock_net((struct sock *)msk));
+}
+
+bool mptcp_addresses_equal(const struct mptcp_addr_info *a,
+ const struct mptcp_addr_info *b, bool use_port)
+{
+ bool addr_equals = false;
+
+ if (a->family == b->family) {
+ if (a->family == AF_INET)
+ addr_equals = a->addr.s_addr == b->addr.s_addr;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ else
+ addr_equals = !ipv6_addr_cmp(&a->addr6, &b->addr6);
+ } else if (a->family == AF_INET) {
+ if (ipv6_addr_v4mapped(&b->addr6))
+ addr_equals = a->addr.s_addr == b->addr6.s6_addr32[3];
+ } else if (b->family == AF_INET) {
+ if (ipv6_addr_v4mapped(&a->addr6))
+ addr_equals = a->addr6.s6_addr32[3] == b->addr.s_addr;
+#endif
+ }
+
+ if (!addr_equals)
+ return false;
+ if (!use_port)
+ return true;
+
+ return a->port == b->port;
+}
+
+void mptcp_local_address(const struct sock_common *skc, struct mptcp_addr_info *addr)
+{
+ addr->family = skc->skc_family;
+ addr->port = htons(skc->skc_num);
+ if (addr->family == AF_INET)
+ addr->addr.s_addr = skc->skc_rcv_saddr;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ else if (addr->family == AF_INET6)
+ addr->addr6 = skc->skc_v6_rcv_saddr;
+#endif
+}
+
+static void remote_address(const struct sock_common *skc,
+ struct mptcp_addr_info *addr)
+{
+ addr->family = skc->skc_family;
+ addr->port = skc->skc_dport;
+ if (addr->family == AF_INET)
+ addr->addr.s_addr = skc->skc_daddr;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ else if (addr->family == AF_INET6)
+ addr->addr6 = skc->skc_v6_daddr;
+#endif
+}
+
+static bool has_subflow_saddr(const struct mptcp_sock *msk,
+ const struct mptcp_addr_info *saddr)
+{
+ struct mptcp_subflow_context *subflow;
+ struct mptcp_addr_info cur;
+ struct sock_common *skc;
+
+ mptcp_for_each_subflow(msk, subflow) {
+ skc = (struct sock_common *)mptcp_subflow_tcp_sock(subflow);
+
+ mptcp_local_address(skc, &cur);
+ if (mptcp_addresses_equal(&cur, saddr, saddr->port))
+ return true;
+ }
+
+ return false;
+}
+
+static bool has_subflow_daddr(const struct mptcp_sock *msk,
+ const struct mptcp_addr_info *daddr)
+{
+ struct mptcp_subflow_context *subflow;
+ struct mptcp_addr_info cur;
+
+ mptcp_for_each_subflow(msk, subflow) {
+ struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+
+ if (!((1 << inet_sk_state_load(ssk)) &
+ (TCPF_ESTABLISHED | TCPF_SYN_SENT | TCPF_SYN_RECV)))
+ continue;
+
+ remote_address((struct sock_common *)ssk, &cur);
+ if (mptcp_addresses_equal(&cur, daddr, daddr->port))
+ return true;
+ }
+
+ return false;
+}
+
+static bool
+select_local_address(const struct pm_nl_pernet *pernet,
+ const struct mptcp_sock *msk,
+ struct mptcp_pm_local *new_local)
+{
+ struct mptcp_pm_addr_entry *entry;
+ bool found = false;
+
+ msk_owned_by_me(msk);
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(entry, &pernet->local_addr_list, list) {
+ if (!(entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW))
+ continue;
+
+ if (!test_bit(entry->addr.id, msk->pm.id_avail_bitmap))
+ continue;
+
+ new_local->addr = entry->addr;
+ new_local->flags = entry->flags;
+ new_local->ifindex = entry->ifindex;
+ found = true;
+ break;
+ }
+ rcu_read_unlock();
+
+ return found;
+}
+
+static bool
+select_signal_address(struct pm_nl_pernet *pernet, const struct mptcp_sock *msk,
+ struct mptcp_pm_local *new_local)
+{
+ struct mptcp_pm_addr_entry *entry;
+ bool found = false;
+
+ rcu_read_lock();
+ /* do not keep any additional per socket state, just signal
+ * the address list in order.
+ * Note: removal from the local address list during the msk life-cycle
+ * can lead to additional addresses not being announced.
+ */
+ list_for_each_entry_rcu(entry, &pernet->local_addr_list, list) {
+ if (!test_bit(entry->addr.id, msk->pm.id_avail_bitmap))
+ continue;
+
+ if (!(entry->flags & MPTCP_PM_ADDR_FLAG_SIGNAL))
+ continue;
+
+ new_local->addr = entry->addr;
+ new_local->flags = entry->flags;
+ new_local->ifindex = entry->ifindex;
+ found = true;
+ break;
+ }
+ rcu_read_unlock();
+
+ return found;
+}
+
+unsigned int mptcp_pm_get_add_addr_signal_max(const struct mptcp_sock *msk)
+{
+ const struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
+
+ return READ_ONCE(pernet->add_addr_signal_max);
+}
+EXPORT_SYMBOL_GPL(mptcp_pm_get_add_addr_signal_max);
+
+unsigned int mptcp_pm_get_add_addr_accept_max(const struct mptcp_sock *msk)
+{
+ struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
+
+ return READ_ONCE(pernet->add_addr_accept_max);
+}
+EXPORT_SYMBOL_GPL(mptcp_pm_get_add_addr_accept_max);
+
+unsigned int mptcp_pm_get_subflows_max(const struct mptcp_sock *msk)
+{
+ struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
+
+ return READ_ONCE(pernet->subflows_max);
+}
+EXPORT_SYMBOL_GPL(mptcp_pm_get_subflows_max);
+
+unsigned int mptcp_pm_get_local_addr_max(const struct mptcp_sock *msk)
+{
+ struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
+
+ return READ_ONCE(pernet->local_addr_max);
+}
+EXPORT_SYMBOL_GPL(mptcp_pm_get_local_addr_max);
+
+bool mptcp_pm_nl_check_work_pending(struct mptcp_sock *msk)
+{
+ struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
+
+ if (msk->pm.subflows == mptcp_pm_get_subflows_max(msk) ||
+ (find_next_and_bit(pernet->id_bitmap, msk->pm.id_avail_bitmap,
+ MPTCP_PM_MAX_ADDR_ID + 1, 0) == MPTCP_PM_MAX_ADDR_ID + 1)) {
+ WRITE_ONCE(msk->pm.work_pending, false);
+ return false;
+ }
+ return true;
+}
+
+struct mptcp_pm_add_addr *
+mptcp_lookup_anno_list_by_saddr(const struct mptcp_sock *msk,
+ const struct mptcp_addr_info *addr)
+{
+ struct mptcp_pm_add_addr *entry;
+
+ lockdep_assert_held(&msk->pm.lock);
+
+ list_for_each_entry(entry, &msk->pm.anno_list, list) {
+ if (mptcp_addresses_equal(&entry->addr, addr, true))
+ return entry;
+ }
+
+ return NULL;
+}
+
+bool mptcp_pm_sport_in_anno_list(struct mptcp_sock *msk, const struct sock *sk)
+{
+ struct mptcp_pm_add_addr *entry;
+ struct mptcp_addr_info saddr;
+ bool ret = false;
+
+ mptcp_local_address((struct sock_common *)sk, &saddr);
+
+ spin_lock_bh(&msk->pm.lock);
+ list_for_each_entry(entry, &msk->pm.anno_list, list) {
+ if (mptcp_addresses_equal(&entry->addr, &saddr, true)) {
+ ret = true;
+ goto out;
+ }
+ }
+
+out:
+ spin_unlock_bh(&msk->pm.lock);
+ return ret;
+}
+
+static void mptcp_pm_add_timer(struct timer_list *timer)
+{
+ struct mptcp_pm_add_addr *entry = from_timer(entry, timer, add_timer);
+ struct mptcp_sock *msk = entry->sock;
+ struct sock *sk = (struct sock *)msk;
+ unsigned int timeout = 0;
+ bool retransmit;
+
+ pr_debug("msk=%p\n", msk);
+
+ bh_lock_sock(sk);
+ if (unlikely(inet_sk_state_load(sk) == TCP_CLOSE))
+ goto out;
+
+ if (sock_owned_by_user(sk)) {
+ /* Try again later. */
+ timeout = HZ / 20;
+ goto out;
+ }
+
+ if (mptcp_pm_should_add_signal_addr(msk)) {
+ timeout = HZ;
+ goto out;
+ }
+
+ timeout = mptcp_get_add_addr_timeout(sock_net(sk));
+ if (!timeout)
+ goto out;
+
+ spin_lock_bh(&msk->pm.lock);
+
+ /* The cancel path (mptcp_pm_del_add_timer()) can race with this
+ * callback. Once cancel updates retrans_times to MAX, suppress further
+ * retransmissions here. If this callback acquires pm.lock first, one
+ * final transmit attempt is still possible.
+ */
+ if (entry->retrans_times < ADD_ADDR_RETRANS_MAX &&
+ !mptcp_pm_should_add_signal_addr(msk)) {
+ pr_debug("retransmit ADD_ADDR id=%d\n", entry->addr.id);
+ mptcp_pm_announce_addr(msk, &entry->addr, false);
+ mptcp_pm_add_addr_send_ack(msk);
+ entry->retrans_times++;
+ }
+
+ retransmit = entry->retrans_times < ADD_ADDR_RETRANS_MAX;
+ if (!retransmit)
+ timeout = 0;
+
+ spin_unlock_bh(&msk->pm.lock);
+
+ if (!retransmit)
+ mptcp_pm_subflow_established(msk);
+
+out:
+ if (timeout)
+ sk_reset_timer(sk, timer, jiffies + timeout);
+ else
+ /* if sock_put calls sk_free: avoid waiting for this timer */
+ entry->timer_done = true;
+ bh_unlock_sock(sk);
+ sock_put(sk);
+}
+
+struct mptcp_pm_add_addr *
+mptcp_pm_del_add_timer(struct mptcp_sock *msk,
+ const struct mptcp_addr_info *addr, bool check_id)
+{
+ struct mptcp_pm_add_addr *entry;
+ struct sock *sk = (struct sock *)msk;
+ bool stop_timer = false;
+
+ rcu_read_lock();
+
+ spin_lock_bh(&msk->pm.lock);
+ entry = mptcp_lookup_anno_list_by_saddr(msk, addr);
+ if (entry && (!check_id || entry->addr.id == addr->id)) {
+ entry->retrans_times = ADD_ADDR_RETRANS_MAX;
+ stop_timer = true;
+ }
+ if (!check_id && entry)
+ list_del(&entry->list);
+ spin_unlock_bh(&msk->pm.lock);
+
+ /* Note: entry might have been removed by another thread.
+ * We hold rcu_read_lock() to ensure it is not freed under us.
+ */
+ if (stop_timer) {
+ if (check_id)
+ sk_stop_timer(sk, &entry->add_timer);
+ else
+ sk_stop_timer_sync(sk, &entry->add_timer);
+ }
+
+ rcu_read_unlock();
+ return entry;
+}
+
+bool mptcp_pm_alloc_anno_list(struct mptcp_sock *msk,
+ const struct mptcp_addr_info *addr)
+{
+ struct mptcp_pm_add_addr *add_entry = NULL;
+ struct sock *sk = (struct sock *)msk;
+ struct net *net = sock_net(sk);
+ unsigned int timeout;
+
+ lockdep_assert_held(&msk->pm.lock);
+
+ if (msk->pm.status & BIT(MPTCP_PM_DESTROYING))
+ return false;
+
+ add_entry = mptcp_lookup_anno_list_by_saddr(msk, addr);
+
+ if (add_entry) {
+ if (WARN_ON_ONCE(mptcp_pm_is_kernel(msk)))
+ return false;
+
+ goto reset_timer;
+ }
+
+ add_entry = kmalloc(sizeof(*add_entry), GFP_ATOMIC);
+ if (!add_entry)
+ return false;
+
+ list_add(&add_entry->list, &msk->pm.anno_list);
+
+ add_entry->addr = *addr;
+ add_entry->sock = msk;
+
+ timer_setup(&add_entry->add_timer, mptcp_pm_add_timer, 0);
+reset_timer:
+ add_entry->retrans_times = 0;
+ add_entry->timer_done = false;
+ timeout = mptcp_get_add_addr_timeout(net);
+ if (timeout)
+ sk_reset_timer(sk, &add_entry->add_timer, jiffies + timeout);
+
+ return true;
+}
+
+void mptcp_pm_free_anno_list(struct mptcp_sock *msk)
+{
+ struct mptcp_pm_add_addr *entry, *tmp;
+ struct sock *sk = (struct sock *)msk;
+ LIST_HEAD(free_list);
+
+ pr_debug("msk=%p\n", msk);
+
+ spin_lock_bh(&msk->pm.lock);
+ list_splice_init(&msk->pm.anno_list, &free_list);
+ spin_unlock_bh(&msk->pm.lock);
+
+ list_for_each_entry_safe(entry, tmp, &free_list, list) {
+ if (!entry->timer_done)
+ sk_stop_timer_sync(sk, &entry->add_timer);
+ kfree_rcu(entry, rcu);
+ }
+}
+
+/* Fill all the remote addresses into the array addrs[],
+ * and return the array size.
+ */
+static unsigned int fill_remote_addresses_vec(struct mptcp_sock *msk,
+ struct mptcp_addr_info *local,
+ bool fullmesh,
+ struct mptcp_addr_info *addrs)
+{
+ bool deny_id0 = READ_ONCE(msk->pm.remote_deny_join_id0);
+ struct sock *sk = (struct sock *)msk, *ssk;
+ struct mptcp_subflow_context *subflow;
+ struct mptcp_addr_info remote = { 0 };
+ unsigned int subflows_max;
+ int i = 0;
+
+ subflows_max = mptcp_pm_get_subflows_max(msk);
+ remote_address((struct sock_common *)sk, &remote);
+
+ /* Non-fullmesh endpoint, fill in the single entry
+ * corresponding to the primary MPC subflow remote address
+ */
+ if (!fullmesh) {
+ if (deny_id0)
+ return 0;
+
+ if (!mptcp_pm_addr_families_match(sk, local, &remote))
+ return 0;
+
+ msk->pm.subflows++;
+ addrs[i++] = remote;
+ } else {
+ DECLARE_BITMAP(unavail_id, MPTCP_PM_MAX_ADDR_ID + 1);
+
+ /* Forbid creation of new subflows matching existing
+ * ones, possibly already created by incoming ADD_ADDR
+ */
+ bitmap_zero(unavail_id, MPTCP_PM_MAX_ADDR_ID + 1);
+ mptcp_for_each_subflow(msk, subflow)
+ if (READ_ONCE(subflow->local_id) == local->id)
+ __set_bit(subflow->remote_id, unavail_id);
+
+ mptcp_for_each_subflow(msk, subflow) {
+ ssk = mptcp_subflow_tcp_sock(subflow);
+ remote_address((struct sock_common *)ssk, &addrs[i]);
+ addrs[i].id = READ_ONCE(subflow->remote_id);
+ if (deny_id0 && !addrs[i].id)
+ continue;
+
+ if (test_bit(addrs[i].id, unavail_id))
+ continue;
+
+ if (!mptcp_pm_addr_families_match(sk, local, &addrs[i]))
+ continue;
+
+ if (msk->pm.subflows < subflows_max) {
+ /* forbid creating multiple address towards
+ * this id
+ */
+ __set_bit(addrs[i].id, unavail_id);
+ msk->pm.subflows++;
+ i++;
+ }
+ }
+ }
+
+ return i;
+}
+
+static void __mptcp_pm_send_ack(struct mptcp_sock *msk, struct mptcp_subflow_context *subflow,
+ bool prio, bool backup)
+{
+ struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+ bool slow;
+
+ pr_debug("send ack for %s\n",
+ prio ? "mp_prio" : (mptcp_pm_should_add_signal(msk) ? "add_addr" : "rm_addr"));
+
+ slow = lock_sock_fast(ssk);
+ if (prio) {
+ subflow->send_mp_prio = 1;
+ subflow->request_bkup = backup;
+ }
+
+ __mptcp_subflow_send_ack(ssk);
+ unlock_sock_fast(ssk, slow);
+}
+
+static void mptcp_pm_send_ack(struct mptcp_sock *msk, struct mptcp_subflow_context *subflow,
+ bool prio, bool backup)
+{
+ spin_unlock_bh(&msk->pm.lock);
+ __mptcp_pm_send_ack(msk, subflow, prio, backup);
+ spin_lock_bh(&msk->pm.lock);
+}
+
+static struct mptcp_pm_addr_entry *
+__lookup_addr_by_id(struct pm_nl_pernet *pernet, unsigned int id)
+{
+ struct mptcp_pm_addr_entry *entry;
+
+ list_for_each_entry(entry, &pernet->local_addr_list, list) {
+ if (entry->addr.id == id)
+ return entry;
+ }
+ return NULL;
+}
+
+static struct mptcp_pm_addr_entry *
+__lookup_addr(struct pm_nl_pernet *pernet, const struct mptcp_addr_info *info)
+{
+ struct mptcp_pm_addr_entry *entry;
+
+ list_for_each_entry_rcu(entry, &pernet->local_addr_list, list,
+ lockdep_is_held(&pernet->lock)) {
+ if (mptcp_addresses_equal(&entry->addr, info, entry->addr.port))
+ return entry;
+ }
+ return NULL;
+}
+
+static void mptcp_pm_create_subflow_or_signal_addr(struct mptcp_sock *msk)
+{
+ struct sock *sk = (struct sock *)msk;
+ unsigned int add_addr_signal_max;
+ bool signal_and_subflow = false;
+ unsigned int local_addr_max;
+ struct pm_nl_pernet *pernet;
+ struct mptcp_pm_local local;
+ unsigned int subflows_max;
+
+ pernet = pm_nl_get_pernet(sock_net(sk));
+
+ add_addr_signal_max = mptcp_pm_get_add_addr_signal_max(msk);
+ local_addr_max = mptcp_pm_get_local_addr_max(msk);
+ subflows_max = mptcp_pm_get_subflows_max(msk);
+
+ /* do lazy endpoint usage accounting for the MPC subflows */
+ if (unlikely(!(msk->pm.status & BIT(MPTCP_PM_MPC_ENDPOINT_ACCOUNTED))) && msk->first) {
+ struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(msk->first);
+ struct mptcp_pm_addr_entry *entry;
+ struct mptcp_addr_info mpc_addr;
+ bool backup = false;
+
+ mptcp_local_address((struct sock_common *)msk->first, &mpc_addr);
+ rcu_read_lock();
+ entry = __lookup_addr(pernet, &mpc_addr);
+ if (entry) {
+ __clear_bit(entry->addr.id, msk->pm.id_avail_bitmap);
+ msk->mpc_endpoint_id = entry->addr.id;
+ backup = !!(entry->flags & MPTCP_PM_ADDR_FLAG_BACKUP);
+ }
+ rcu_read_unlock();
+
+ if (backup)
+ mptcp_pm_send_ack(msk, subflow, true, backup);
+
+ msk->pm.status |= BIT(MPTCP_PM_MPC_ENDPOINT_ACCOUNTED);
+ }
+
+ pr_debug("local %d:%d signal %d:%d subflows %d:%d\n",
+ msk->pm.local_addr_used, local_addr_max,
+ msk->pm.add_addr_signaled, add_addr_signal_max,
+ msk->pm.subflows, subflows_max);
+
+ /* check first for announce */
+ if (msk->pm.add_addr_signaled < add_addr_signal_max) {
+ u8 endp_id;
+
+ /* due to racing events on both ends we can reach here while
+ * previous add address is still running: if we invoke now
+ * mptcp_pm_announce_addr(), that will fail and the
+ * corresponding id will be marked as used.
+ * Instead let the PM machinery reschedule us when the
+ * current address announce will be completed.
+ */
+ if (msk->pm.addr_signal & BIT(MPTCP_ADD_ADDR_SIGNAL))
+ return;
+
+ if (!select_signal_address(pernet, msk, &local))
+ goto subflow;
+
+ /* Special case for ID0: set the correct ID */
+ endp_id = local.addr.id;
+ if (endp_id == msk->mpc_endpoint_id)
+ local.addr.id = 0;
+
+ /* If the alloc fails, we are on memory pressure, not worth
+ * continuing, and trying to create subflows.
+ */
+ if (!mptcp_pm_alloc_anno_list(msk, &local.addr))
+ return;
+
+ __clear_bit(endp_id, msk->pm.id_avail_bitmap);
+ msk->pm.add_addr_signaled++;
+
+ mptcp_pm_announce_addr(msk, &local.addr, false);
+ mptcp_pm_nl_addr_send_ack(msk);
+
+ if (local.flags & MPTCP_PM_ADDR_FLAG_SUBFLOW)
+ signal_and_subflow = true;
+ }
+
+subflow:
+ /* No need to try establishing subflows to remote id0 if not allowed */
+ if (mptcp_pm_add_addr_c_flag_case(msk))
+ goto exit;
+
+ /* check if should create a new subflow */
+ while (msk->pm.local_addr_used < local_addr_max &&
+ msk->pm.subflows < subflows_max) {
+ struct mptcp_addr_info addrs[MPTCP_PM_ADDR_MAX];
+ bool fullmesh;
+ int i, nr;
+
+ if (signal_and_subflow)
+ signal_and_subflow = false;
+ else if (!select_local_address(pernet, msk, &local))
+ break;
+
+ fullmesh = !!(local.flags & MPTCP_PM_ADDR_FLAG_FULLMESH);
+
+ __clear_bit(local.addr.id, msk->pm.id_avail_bitmap);
+
+ /* Special case for ID0: set the correct ID */
+ if (local.addr.id == msk->mpc_endpoint_id)
+ local.addr.id = 0;
+ else /* local_addr_used is not decr for ID 0 */
+ msk->pm.local_addr_used++;
+
+ nr = fill_remote_addresses_vec(msk, &local.addr, fullmesh, addrs);
+ if (nr == 0)
+ continue;
+
+ spin_unlock_bh(&msk->pm.lock);
+ for (i = 0; i < nr; i++)
+ __mptcp_subflow_connect(sk, &local, &addrs[i]);
+ spin_lock_bh(&msk->pm.lock);
+ }
+
+exit:
+ /* If an endpoint has both the signal and subflow flags, but it is not
+ * possible to create subflows -- the 'while' loop body above never
+ * executed -- then still mark the endp as used, which is somehow the
+ * case. This avoids issues later when removing the endpoint and calling
+ * __mark_subflow_endp_available(), which expects the increment here.
+ */
+ if (signal_and_subflow && local.addr.id != msk->mpc_endpoint_id)
+ msk->pm.local_addr_used++;
+
+ mptcp_pm_nl_check_work_pending(msk);
+}
+
+static void mptcp_pm_nl_fully_established(struct mptcp_sock *msk)
+{
+ mptcp_pm_create_subflow_or_signal_addr(msk);
+}
+
+static void mptcp_pm_nl_subflow_established(struct mptcp_sock *msk)
+{
+ mptcp_pm_create_subflow_or_signal_addr(msk);
+}
+
+/* Fill all the local addresses into the array addrs[],
+ * and return the array size.
+ */
+static unsigned int fill_local_addresses_vec(struct mptcp_sock *msk,
+ struct mptcp_addr_info *remote,
+ struct mptcp_pm_local *locals)
+{
+ struct sock *sk = (struct sock *)msk;
+ struct mptcp_pm_addr_entry *entry;
+ struct mptcp_addr_info mpc_addr;
+ struct pm_nl_pernet *pernet;
+ unsigned int subflows_max;
+ bool c_flag_case;
+ int i = 0;
+
+ pernet = pm_nl_get_pernet_from_msk(msk);
+ subflows_max = mptcp_pm_get_subflows_max(msk);
+ c_flag_case = remote->id && mptcp_pm_add_addr_c_flag_case(msk);
+
+ mptcp_local_address((struct sock_common *)msk, &mpc_addr);
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(entry, &pernet->local_addr_list, list) {
+ if (!(entry->flags & MPTCP_PM_ADDR_FLAG_FULLMESH))
+ continue;
+
+ if (!mptcp_pm_addr_families_match(sk, &entry->addr, remote))
+ continue;
+
+ if (msk->pm.subflows < subflows_max) {
+ bool is_id0;
+
+ locals[i].addr = entry->addr;
+ locals[i].flags = entry->flags;
+ locals[i].ifindex = entry->ifindex;
+
+ is_id0 = mptcp_addresses_equal(&locals[i].addr,
+ &mpc_addr,
+ locals[i].addr.port);
+
+ if (c_flag_case &&
+ (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW)) {
+ __clear_bit(locals[i].addr.id,
+ msk->pm.id_avail_bitmap);
+
+ if (!is_id0)
+ msk->pm.local_addr_used++;
+ }
+
+ /* Special case for ID0: set the correct ID */
+ if (is_id0)
+ locals[i].addr.id = 0;
+
+ msk->pm.subflows++;
+ i++;
+ }
+ }
+ rcu_read_unlock();
+
+ /* Special case: peer sets the C flag, accept one ADD_ADDR if default
+ * limits are used -- accepting no ADD_ADDR -- and use subflow endpoints
+ */
+ if (!i && c_flag_case) {
+ unsigned int local_addr_max = mptcp_pm_get_local_addr_max(msk);
+
+ while (msk->pm.local_addr_used < local_addr_max &&
+ msk->pm.subflows < subflows_max) {
+ struct mptcp_pm_local *local = &locals[i];
+
+ if (!select_local_address(pernet, msk, local))
+ break;
+
+ __clear_bit(local->addr.id, msk->pm.id_avail_bitmap);
+
+ if (!mptcp_pm_addr_families_match(sk, &local->addr,
+ remote))
+ continue;
+
+ if (mptcp_addresses_equal(&local->addr, &mpc_addr,
+ local->addr.port))
+ continue;
+
+ msk->pm.local_addr_used++;
+ msk->pm.subflows++;
+ i++;
+ }
+
+ return i;
+ }
+
+ /* If the array is empty, fill in the single
+ * 'IPADDRANY' local address
+ */
+ if (!i) {
+ memset(&locals[i], 0, sizeof(locals[i]));
+ locals[i].addr.family =
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ remote->family == AF_INET6 &&
+ ipv6_addr_v4mapped(&remote->addr6) ? AF_INET :
+#endif
+ remote->family;
+
+ if (!mptcp_pm_addr_families_match(sk, &locals[i].addr, remote))
+ return 0;
+
+ msk->pm.subflows++;
+ i++;
+ }
+
+ return i;
+}
+
+static void mptcp_pm_nl_add_addr_received(struct mptcp_sock *msk)
+{
+ struct mptcp_pm_local locals[MPTCP_PM_ADDR_MAX];
+ struct sock *sk = (struct sock *)msk;
+ unsigned int add_addr_accept_max;
+ struct mptcp_addr_info remote;
+ unsigned int subflows_max;
+ bool sf_created = false;
+ int i, nr;
+
+ add_addr_accept_max = mptcp_pm_get_add_addr_accept_max(msk);
+ subflows_max = mptcp_pm_get_subflows_max(msk);
+
+ pr_debug("accepted %d:%d remote family %d\n",
+ msk->pm.add_addr_accepted, add_addr_accept_max,
+ msk->pm.remote.family);
+
+ remote = msk->pm.remote;
+ mptcp_pm_announce_addr(msk, &remote, true);
+ mptcp_pm_nl_addr_send_ack(msk);
+
+ if (has_subflow_daddr(msk, &remote))
+ return;
+
+ /* pick id 0 port, if none is provided the remote address */
+ if (!remote.port)
+ remote.port = sk->sk_dport;
+
+ /* connect to the specified remote address, using whatever
+ * local address the routing configuration will pick.
+ */
+ nr = fill_local_addresses_vec(msk, &remote, locals);
+ if (nr == 0)
+ return;
+
+ spin_unlock_bh(&msk->pm.lock);
+ for (i = 0; i < nr; i++)
+ if (__mptcp_subflow_connect(sk, &locals[i], &remote) == 0)
+ sf_created = true;
+ spin_lock_bh(&msk->pm.lock);
+
+ if (sf_created) {
+ /* add_addr_accepted is not decr for ID 0 */
+ if (remote.id)
+ msk->pm.add_addr_accepted++;
+ if (msk->pm.add_addr_accepted >= add_addr_accept_max ||
+ msk->pm.subflows >= subflows_max)
+ WRITE_ONCE(msk->pm.accept_addr, false);
+ }
+}
+
+bool mptcp_pm_nl_is_init_remote_addr(struct mptcp_sock *msk,
+ const struct mptcp_addr_info *remote)
+{
+ struct mptcp_addr_info mpc_remote;
+
+ remote_address((struct sock_common *)msk, &mpc_remote);
+ return mptcp_addresses_equal(&mpc_remote, remote, remote->port);
+}
+
+static bool subflow_in_rm_list(const struct mptcp_subflow_context *subflow,
+ const struct mptcp_rm_list *rm_list)
+{
+ u8 i, id = subflow_get_local_id(subflow);
+
+ for (i = 0; i < rm_list->nr; i++) {
+ if (rm_list->ids[i] == id)
+ return true;
+ }
+
+ return false;
+}
+
+void mptcp_pm_nl_addr_send_ack_avoid_list(struct mptcp_sock *msk,
+ const struct mptcp_rm_list *rm_list)
+{
+ struct mptcp_subflow_context *subflow, *same_id = NULL;
+
+ msk_owned_by_me(msk);
+ lockdep_assert_held(&msk->pm.lock);
+
+ if (!mptcp_pm_should_add_signal(msk) &&
+ !mptcp_pm_should_rm_signal(msk))
+ return;
+
+ mptcp_for_each_subflow(msk, subflow) {
+ if (!__mptcp_subflow_active(subflow))
+ continue;
+
+ if (unlikely(rm_list &&
+ subflow_in_rm_list(subflow, rm_list))) {
+ if (!same_id)
+ same_id = subflow;
+ } else {
+ goto send_ack;
+ }
+ }
+
+ if (same_id)
+ subflow = same_id;
+ else
+ return;
+
+send_ack:
+ mptcp_pm_send_ack(msk, subflow, false, false);
+}
+
+void mptcp_pm_nl_addr_send_ack(struct mptcp_sock *msk)
+{
+ mptcp_pm_nl_addr_send_ack_avoid_list(msk, NULL);
+}
+
+int mptcp_pm_nl_mp_prio_send_ack(struct mptcp_sock *msk,
+ struct mptcp_addr_info *addr,
+ struct mptcp_addr_info *rem,
+ u8 bkup)
+{
+ struct mptcp_subflow_context *subflow;
+
+ pr_debug("bkup=%d\n", bkup);
+
+ mptcp_for_each_subflow(msk, subflow) {
+ struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+ struct mptcp_addr_info local, remote;
+
+ if (!__mptcp_subflow_active(subflow))
+ continue;
+
+ mptcp_local_address((struct sock_common *)ssk, &local);
+ if (!mptcp_addresses_equal(&local, addr, addr->port))
+ continue;
+
+ if (rem && rem->family != AF_UNSPEC) {
+ remote_address((struct sock_common *)ssk, &remote);
+ if (!mptcp_addresses_equal(&remote, rem, rem->port))
+ continue;
+ }
+
+ __mptcp_pm_send_ack(msk, subflow, true, bkup);
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static void mptcp_pm_nl_rm_addr_or_subflow(struct mptcp_sock *msk,
+ const struct mptcp_rm_list *rm_list,
+ enum linux_mptcp_mib_field rm_type)
+{
+ struct mptcp_subflow_context *subflow, *tmp;
+ struct sock *sk = (struct sock *)msk;
+ u8 i;
+
+ pr_debug("%s rm_list_nr %d\n",
+ rm_type == MPTCP_MIB_RMADDR ? "address" : "subflow", rm_list->nr);
+
+ msk_owned_by_me(msk);
+
+ if (sk->sk_state == TCP_LISTEN)
+ return;
+
+ if (!rm_list->nr)
+ return;
+
+ if (list_empty(&msk->conn_list))
+ return;
+
+ for (i = 0; i < rm_list->nr; i++) {
+ u8 rm_id = rm_list->ids[i];
+ bool removed = false;
+
+ mptcp_for_each_subflow_safe(msk, subflow, tmp) {
+ struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
+ u8 remote_id = READ_ONCE(subflow->remote_id);
+ int how = RCV_SHUTDOWN | SEND_SHUTDOWN;
+ u8 id = subflow_get_local_id(subflow);
+
+ if ((1 << inet_sk_state_load(ssk)) &
+ (TCPF_FIN_WAIT1 | TCPF_FIN_WAIT2 | TCPF_CLOSING | TCPF_CLOSE))
+ continue;
+ if (rm_type == MPTCP_MIB_RMADDR && remote_id != rm_id)
+ continue;
+ if (rm_type == MPTCP_MIB_RMSUBFLOW && id != rm_id)
+ continue;
+
+ pr_debug(" -> %s rm_list_ids[%d]=%u local_id=%u remote_id=%u mpc_id=%u\n",
+ rm_type == MPTCP_MIB_RMADDR ? "address" : "subflow",
+ i, rm_id, id, remote_id, msk->mpc_endpoint_id);
+ spin_unlock_bh(&msk->pm.lock);
+ mptcp_subflow_shutdown(sk, ssk, how);
+ removed |= subflow->request_join;
+
+ /* the following takes care of updating the subflows counter */
+ mptcp_close_ssk(sk, ssk, subflow);
+ spin_lock_bh(&msk->pm.lock);
+
+ if (rm_type == MPTCP_MIB_RMSUBFLOW)
+ __MPTCP_INC_STATS(sock_net(sk), rm_type);
+ }
+
+ if (rm_type == MPTCP_MIB_RMADDR)
+ __MPTCP_INC_STATS(sock_net(sk), rm_type);
+
+ if (!removed)
+ continue;
+
+ if (!mptcp_pm_is_kernel(msk))
+ continue;
+
+ if (rm_type == MPTCP_MIB_RMADDR && rm_id &&
+ !WARN_ON_ONCE(msk->pm.add_addr_accepted == 0)) {
+ /* Note: if the subflow has been closed before, this
+ * add_addr_accepted counter will not be decremented.
+ */
+ if (--msk->pm.add_addr_accepted < mptcp_pm_get_add_addr_accept_max(msk))
+ WRITE_ONCE(msk->pm.accept_addr, true);
+ }
+ }
+}
+
+static void mptcp_pm_nl_rm_addr_received(struct mptcp_sock *msk)
+{
+ mptcp_pm_nl_rm_addr_or_subflow(msk, &msk->pm.rm_list_rx, MPTCP_MIB_RMADDR);
+}
+
+static void mptcp_pm_rm_subflow(struct mptcp_sock *msk,
+ const struct mptcp_rm_list *rm_list)
+{
+ mptcp_pm_nl_rm_addr_or_subflow(msk, rm_list, MPTCP_MIB_RMSUBFLOW);
+}
+
+void mptcp_pm_nl_work(struct mptcp_sock *msk)
+{
+ struct mptcp_pm_data *pm = &msk->pm;
+
+ msk_owned_by_me(msk);
+
+ if (!(pm->status & MPTCP_PM_WORK_MASK))
+ return;
+
+ spin_lock_bh(&msk->pm.lock);
+
+ pr_debug("msk=%p status=%x\n", msk, pm->status);
+ if (pm->status & BIT(MPTCP_PM_ADD_ADDR_RECEIVED)) {
+ pm->status &= ~BIT(MPTCP_PM_ADD_ADDR_RECEIVED);
+ mptcp_pm_nl_add_addr_received(msk);
+ }
+ if (pm->status & BIT(MPTCP_PM_ADD_ADDR_SEND_ACK)) {
+ pm->status &= ~BIT(MPTCP_PM_ADD_ADDR_SEND_ACK);
+ mptcp_pm_nl_addr_send_ack(msk);
+ }
+ if (pm->status & BIT(MPTCP_PM_RM_ADDR_RECEIVED)) {
+ pm->status &= ~BIT(MPTCP_PM_RM_ADDR_RECEIVED);
+ mptcp_pm_nl_rm_addr_received(msk);
+ }
+ if (pm->status & BIT(MPTCP_PM_ESTABLISHED)) {
+ pm->status &= ~BIT(MPTCP_PM_ESTABLISHED);
+ mptcp_pm_nl_fully_established(msk);
+ }
+ if (pm->status & BIT(MPTCP_PM_SUBFLOW_ESTABLISHED)) {
+ pm->status &= ~BIT(MPTCP_PM_SUBFLOW_ESTABLISHED);
+ mptcp_pm_nl_subflow_established(msk);
+ }
+
+ spin_unlock_bh(&msk->pm.lock);
+}
+
+static bool address_use_port(struct mptcp_pm_addr_entry *entry)
+{
+ return (entry->flags &
+ (MPTCP_PM_ADDR_FLAG_SIGNAL | MPTCP_PM_ADDR_FLAG_SUBFLOW)) ==
+ MPTCP_PM_ADDR_FLAG_SIGNAL;
+}
+
+/* caller must ensure the RCU grace period is already elapsed */
+static void __mptcp_pm_release_addr_entry(struct mptcp_pm_addr_entry *entry)
+{
+ if (entry->lsk)
+ sock_release(entry->lsk);
+ kfree(entry);
+}
+
+static int mptcp_pm_nl_append_new_local_addr(struct pm_nl_pernet *pernet,
+ struct mptcp_pm_addr_entry *entry,
+ bool replace)
+{
+ struct mptcp_pm_addr_entry *cur, *del_entry = NULL;
+ unsigned int addr_max;
+ int ret = -EINVAL;
+
+ spin_lock_bh(&pernet->lock);
+ /* to keep the code simple, don't do IDR-like allocation for address ID,
+ * just bail when we exceed limits
+ */
+ if (pernet->next_id == MPTCP_PM_MAX_ADDR_ID)
+ pernet->next_id = 1;
+ if (pernet->addrs >= MPTCP_PM_ADDR_MAX) {
+ ret = -ERANGE;
+ goto out;
+ }
+ if (test_bit(entry->addr.id, pernet->id_bitmap)) {
+ ret = -EBUSY;
+ goto out;
+ }
+
+ /* do not insert duplicate address, differentiate on port only
+ * singled addresses
+ */
+ if (!address_use_port(entry))
+ entry->addr.port = 0;
+ list_for_each_entry(cur, &pernet->local_addr_list, list) {
+ if (mptcp_addresses_equal(&cur->addr, &entry->addr,
+ cur->addr.port || entry->addr.port)) {
+ /* allow replacing the exiting endpoint only if such
+ * endpoint is an implicit one and the user-space
+ * did not provide an endpoint id
+ */
+ if (!(cur->flags & MPTCP_PM_ADDR_FLAG_IMPLICIT)) {
+ ret = -EEXIST;
+ goto out;
+ }
+ if (entry->addr.id)
+ goto out;
+
+ /* allow callers that only need to look up the local
+ * addr's id to skip replacement. This allows them to
+ * avoid calling synchronize_rcu in the packet recv
+ * path.
+ */
+ if (!replace) {
+ kfree(entry);
+ ret = cur->addr.id;
+ goto out;
+ }
+
+ pernet->addrs--;
+ entry->addr.id = cur->addr.id;
+ list_del_rcu(&cur->list);
+ del_entry = cur;
+ break;
+ }
+ }
+
+ if (!entry->addr.id) {
+find_next:
+ entry->addr.id = find_next_zero_bit(pernet->id_bitmap,
+ MPTCP_PM_MAX_ADDR_ID + 1,
+ pernet->next_id);
+ if (!entry->addr.id && pernet->next_id != 1) {
+ pernet->next_id = 1;
+ goto find_next;
+ }
+ }
+
+ if (!entry->addr.id)
+ goto out;
+
+ __set_bit(entry->addr.id, pernet->id_bitmap);
+ if (entry->addr.id > pernet->next_id)
+ pernet->next_id = entry->addr.id;
+
+ if (entry->flags & MPTCP_PM_ADDR_FLAG_SIGNAL) {
+ addr_max = pernet->add_addr_signal_max;
+ WRITE_ONCE(pernet->add_addr_signal_max, addr_max + 1);
+ }
+ if (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW) {
+ addr_max = pernet->local_addr_max;
+ WRITE_ONCE(pernet->local_addr_max, addr_max + 1);
+ }
+
+ pernet->addrs++;
+ if (!entry->addr.port)
+ list_add_tail_rcu(&entry->list, &pernet->local_addr_list);
+ else
+ list_add_rcu(&entry->list, &pernet->local_addr_list);
+ ret = entry->addr.id;
+
+out:
+ spin_unlock_bh(&pernet->lock);
+
+ /* just replaced an existing entry, free it */
+ if (del_entry) {
+ synchronize_rcu();
+ __mptcp_pm_release_addr_entry(del_entry);
+ }
+ return ret;
+}
+
+static struct lock_class_key mptcp_slock_keys[2];
+static struct lock_class_key mptcp_keys[2];
+
+static int mptcp_pm_nl_create_listen_socket(struct sock *sk,
+ struct mptcp_pm_addr_entry *entry)
+{
+ bool is_ipv6 = entry->addr.family == AF_INET6;
+ int addrlen = sizeof(struct sockaddr_in);
+ struct sockaddr_storage addr;
+ struct sock *newsk, *ssk;
+ int backlog = 1024;
+ int err;
+
+ err = sock_create_kern(sock_net(sk), entry->addr.family,
+ SOCK_STREAM, IPPROTO_MPTCP, &entry->lsk);
+ if (err)
+ return err;
+
+ newsk = entry->lsk->sk;
+ if (!newsk)
+ return -EINVAL;
+
+ /* The subflow socket lock is acquired in a nested to the msk one
+ * in several places, even by the TCP stack, and this msk is a kernel
+ * socket: lockdep complains. Instead of propagating the _nested
+ * modifiers in several places, re-init the lock class for the msk
+ * socket to an mptcp specific one.
+ */
+ sock_lock_init_class_and_name(newsk,
+ is_ipv6 ? "mlock-AF_INET6" : "mlock-AF_INET",
+ &mptcp_slock_keys[is_ipv6],
+ is_ipv6 ? "msk_lock-AF_INET6" : "msk_lock-AF_INET",
+ &mptcp_keys[is_ipv6]);
+
+ lock_sock(newsk);
+ ssk = __mptcp_nmpc_sk(mptcp_sk(newsk));
+ release_sock(newsk);
+ if (IS_ERR(ssk))
+ return PTR_ERR(ssk);
+
+ mptcp_info2sockaddr(&entry->addr, &addr, entry->addr.family);
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ if (entry->addr.family == AF_INET6)
+ addrlen = sizeof(struct sockaddr_in6);
+#endif
+ if (ssk->sk_family == AF_INET)
+ err = inet_bind_sk(ssk, (struct sockaddr *)&addr, addrlen);
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ else if (ssk->sk_family == AF_INET6)
+ err = inet6_bind_sk(ssk, (struct sockaddr *)&addr, addrlen);
+#endif
+ if (err)
+ return err;
+
+ /* We don't use mptcp_set_state() here because it needs to be called
+ * under the msk socket lock. For the moment, that will not bring
+ * anything more than only calling inet_sk_state_store(), because the
+ * old status is known (TCP_CLOSE).
+ */
+ inet_sk_state_store(newsk, TCP_LISTEN);
+ lock_sock(ssk);
+ WRITE_ONCE(mptcp_subflow_ctx(ssk)->pm_listener, true);
+ err = __inet_listen_sk(ssk, backlog);
+ if (!err)
+ mptcp_event_pm_listener(ssk, MPTCP_EVENT_LISTENER_CREATED);
+ release_sock(ssk);
+ return err;
+}
+
+int mptcp_pm_nl_get_local_id(struct mptcp_sock *msk,
+ struct mptcp_pm_addr_entry *skc)
+{
+ struct mptcp_pm_addr_entry *entry;
+ struct pm_nl_pernet *pernet;
+ int ret;
+
+ pernet = pm_nl_get_pernet_from_msk(msk);
+
+ rcu_read_lock();
+ entry = __lookup_addr(pernet, &skc->addr);
+ ret = entry ? entry->addr.id : -1;
+ rcu_read_unlock();
+ if (ret >= 0)
+ return ret;
+
+ /* address not found, add to local list */
+ entry = kmalloc(sizeof(*entry), GFP_ATOMIC);
+ if (!entry)
+ return -ENOMEM;
+
+ *entry = *skc;
+ entry->addr.port = 0;
+ ret = mptcp_pm_nl_append_new_local_addr(pernet, entry, false);
+ if (ret < 0)
+ kfree(entry);
+
+ return ret;
+}
+
+bool mptcp_pm_nl_is_backup(struct mptcp_sock *msk, struct mptcp_addr_info *skc)
+{
+ struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
+ struct mptcp_pm_addr_entry *entry;
+ bool backup;
+
+ rcu_read_lock();
+ entry = __lookup_addr(pernet, skc);
+ backup = entry && !!(entry->flags & MPTCP_PM_ADDR_FLAG_BACKUP);
+ rcu_read_unlock();
+
+ return backup;
+}
+
+#define MPTCP_PM_CMD_GRP_OFFSET 0
+#define MPTCP_PM_EV_GRP_OFFSET 1
+
+static const struct genl_multicast_group mptcp_pm_mcgrps[] = {
+ [MPTCP_PM_CMD_GRP_OFFSET] = { .name = MPTCP_PM_CMD_GRP_NAME, },
+ [MPTCP_PM_EV_GRP_OFFSET] = { .name = MPTCP_PM_EV_GRP_NAME,
+ .flags = GENL_MCAST_CAP_NET_ADMIN,
+ },
+};
+
+void mptcp_pm_nl_subflow_chk_stale(const struct mptcp_sock *msk, struct sock *ssk)
+{
+ struct mptcp_subflow_context *iter, *subflow = mptcp_subflow_ctx(ssk);
+ struct sock *sk = (struct sock *)msk;
+ unsigned int active_max_loss_cnt;
+ struct net *net = sock_net(sk);
+ unsigned int stale_loss_cnt;
+ bool slow;
+
+ stale_loss_cnt = mptcp_stale_loss_cnt(net);
+ if (subflow->stale || !stale_loss_cnt || subflow->stale_count <= stale_loss_cnt)
+ return;
+
+ /* look for another available subflow not in loss state */
+ active_max_loss_cnt = max_t(int, stale_loss_cnt - 1, 1);
+ mptcp_for_each_subflow(msk, iter) {
+ if (iter != subflow && mptcp_subflow_active(iter) &&
+ iter->stale_count < active_max_loss_cnt) {
+ /* we have some alternatives, try to mark this subflow as idle ...*/
+ slow = lock_sock_fast(ssk);
+ if (!tcp_rtx_and_write_queues_empty(ssk)) {
+ subflow->stale = 1;
+ __mptcp_retransmit_pending_data(sk);
+ MPTCP_INC_STATS(net, MPTCP_MIB_SUBFLOWSTALE);
+ }
+ unlock_sock_fast(ssk, slow);
+
+ /* always try to push the pending data regardless of re-injections:
+ * we can possibly use backup subflows now, and subflow selection
+ * is cheap under the msk socket lock
+ */
+ __mptcp_push_pending(sk, 0);
+ return;
+ }
+ }
+}
+
+static int mptcp_pm_family_to_addr(int family)
+{
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ if (family == AF_INET6)
+ return MPTCP_PM_ADDR_ATTR_ADDR6;
+#endif
+ return MPTCP_PM_ADDR_ATTR_ADDR4;
+}
+
+static int mptcp_pm_parse_pm_addr_attr(struct nlattr *tb[],
+ const struct nlattr *attr,
+ struct genl_info *info,
+ struct mptcp_addr_info *addr,
+ bool require_family)
+{
+ int err, addr_addr;
+
+ if (!attr) {
+ GENL_SET_ERR_MSG(info, "missing address info");
+ return -EINVAL;
+ }
+
+ /* no validation needed - was already done via nested policy */
+ err = nla_parse_nested_deprecated(tb, MPTCP_PM_ADDR_ATTR_MAX, attr,
+ mptcp_pm_address_nl_policy, info->extack);
+ if (err)
+ return err;
+
+ if (tb[MPTCP_PM_ADDR_ATTR_ID])
+ addr->id = nla_get_u8(tb[MPTCP_PM_ADDR_ATTR_ID]);
+
+ if (!tb[MPTCP_PM_ADDR_ATTR_FAMILY]) {
+ if (!require_family)
+ return 0;
+
+ NL_SET_ERR_MSG_ATTR(info->extack, attr,
+ "missing family");
+ return -EINVAL;
+ }
+
+ addr->family = nla_get_u16(tb[MPTCP_PM_ADDR_ATTR_FAMILY]);
+ if (addr->family != AF_INET
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ && addr->family != AF_INET6
+#endif
+ ) {
+ NL_SET_ERR_MSG_ATTR(info->extack, attr,
+ "unknown address family");
+ return -EINVAL;
+ }
+ addr_addr = mptcp_pm_family_to_addr(addr->family);
+ if (!tb[addr_addr]) {
+ NL_SET_ERR_MSG_ATTR(info->extack, attr,
+ "missing address data");
+ return -EINVAL;
+ }
+
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ if (addr->family == AF_INET6)
+ addr->addr6 = nla_get_in6_addr(tb[addr_addr]);
+ else
+#endif
+ addr->addr.s_addr = nla_get_in_addr(tb[addr_addr]);
+
+ if (tb[MPTCP_PM_ADDR_ATTR_PORT])
+ addr->port = htons(nla_get_u16(tb[MPTCP_PM_ADDR_ATTR_PORT]));
+
+ return 0;
+}
+
+int mptcp_pm_parse_addr(struct nlattr *attr, struct genl_info *info,
+ struct mptcp_addr_info *addr)
+{
+ struct nlattr *tb[MPTCP_PM_ADDR_ATTR_MAX + 1];
+
+ memset(addr, 0, sizeof(*addr));
+
+ return mptcp_pm_parse_pm_addr_attr(tb, attr, info, addr, true);
+}
+
+int mptcp_pm_parse_entry(struct nlattr *attr, struct genl_info *info,
+ bool require_family,
+ struct mptcp_pm_addr_entry *entry)
+{
+ struct nlattr *tb[MPTCP_PM_ADDR_ATTR_MAX + 1];
+ int err;
+
+ memset(entry, 0, sizeof(*entry));
+
+ err = mptcp_pm_parse_pm_addr_attr(tb, attr, info, &entry->addr, require_family);
+ if (err)
+ return err;
+
+ if (tb[MPTCP_PM_ADDR_ATTR_IF_IDX]) {
+ u32 val = nla_get_s32(tb[MPTCP_PM_ADDR_ATTR_IF_IDX]);
+
+ entry->ifindex = val;
+ }
+
+ if (tb[MPTCP_PM_ADDR_ATTR_FLAGS])
+ entry->flags = nla_get_u32(tb[MPTCP_PM_ADDR_ATTR_FLAGS]) &
+ MPTCP_PM_ADDR_FLAGS_MASK;
+
+ if (tb[MPTCP_PM_ADDR_ATTR_PORT])
+ entry->addr.port = htons(nla_get_u16(tb[MPTCP_PM_ADDR_ATTR_PORT]));
+
+ return 0;
+}
+
+static struct pm_nl_pernet *genl_info_pm_nl(struct genl_info *info)
+{
+ return pm_nl_get_pernet(genl_info_net(info));
+}
+
+static int mptcp_nl_add_subflow_or_signal_addr(struct net *net,
+ struct mptcp_addr_info *addr)
+{
+ struct mptcp_sock *msk;
+ long s_slot = 0, s_num = 0;
+
+ while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
+ struct sock *sk = (struct sock *)msk;
+ struct mptcp_addr_info mpc_addr;
+
+ if (!READ_ONCE(msk->fully_established) ||
+ mptcp_pm_is_userspace(msk))
+ goto next;
+
+ /* if the endp linked to the init sf is re-added with a != ID */
+ mptcp_local_address((struct sock_common *)msk, &mpc_addr);
+
+ lock_sock(sk);
+ spin_lock_bh(&msk->pm.lock);
+ if (mptcp_addresses_equal(addr, &mpc_addr, addr->port))
+ msk->mpc_endpoint_id = addr->id;
+ mptcp_pm_create_subflow_or_signal_addr(msk);
+ spin_unlock_bh(&msk->pm.lock);
+ release_sock(sk);
+
+next:
+ sock_put(sk);
+ cond_resched();
+ }
+
+ return 0;
+}
+
+int mptcp_pm_nl_add_addr_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct nlattr *attr = info->attrs[MPTCP_PM_ENDPOINT_ADDR];
+ struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
+ struct mptcp_pm_addr_entry addr, *entry;
+ int ret;
+
+ ret = mptcp_pm_parse_entry(attr, info, true, &addr);
+ if (ret < 0)
+ return ret;
+
+ if (addr.addr.port && !address_use_port(&addr)) {
+ GENL_SET_ERR_MSG(info, "flags must have signal and not subflow when using port");
+ return -EINVAL;
+ }
+
+ if (addr.flags & MPTCP_PM_ADDR_FLAG_SIGNAL &&
+ addr.flags & MPTCP_PM_ADDR_FLAG_FULLMESH) {
+ GENL_SET_ERR_MSG(info, "flags mustn't have both signal and fullmesh");
+ return -EINVAL;
+ }
+
+ if (addr.flags & MPTCP_PM_ADDR_FLAG_IMPLICIT) {
+ GENL_SET_ERR_MSG(info, "can't create IMPLICIT endpoint");
+ return -EINVAL;
+ }
+
+ entry = kzalloc(sizeof(*entry), GFP_KERNEL_ACCOUNT);
+ if (!entry) {
+ GENL_SET_ERR_MSG(info, "can't allocate addr");
+ return -ENOMEM;
+ }
+
+ *entry = addr;
+ if (entry->addr.port) {
+ ret = mptcp_pm_nl_create_listen_socket(skb->sk, entry);
+ if (ret) {
+ GENL_SET_ERR_MSG_FMT(info, "create listen socket error: %d", ret);
+ goto out_free;
+ }
+ }
+ ret = mptcp_pm_nl_append_new_local_addr(pernet, entry, true);
+ if (ret < 0) {
+ GENL_SET_ERR_MSG_FMT(info, "too many addresses or duplicate one: %d", ret);
+ goto out_free;
+ }
+
+ mptcp_nl_add_subflow_or_signal_addr(sock_net(skb->sk), &entry->addr);
+ return 0;
+
+out_free:
+ __mptcp_pm_release_addr_entry(entry);
+ return ret;
+}
+
+static bool mptcp_pm_announced_remove(struct mptcp_sock *msk,
+ const struct mptcp_addr_info *addr)
+{
+ struct mptcp_pm_add_addr *entry;
+
+ entry = mptcp_pm_del_add_timer(msk, addr, false);
+ if (entry) {
+ kfree_rcu(entry, rcu);
+ return true;
+ }
+
+ return false;
+}
+
+static u8 mptcp_endp_get_local_id(struct mptcp_sock *msk,
+ const struct mptcp_addr_info *addr)
+{
+ return msk->mpc_endpoint_id == addr->id ? 0 : addr->id;
+}
+
+static bool mptcp_pm_remove_anno_addr(struct mptcp_sock *msk,
+ const struct mptcp_addr_info *addr,
+ bool force)
+{
+ struct mptcp_rm_list list = { .nr = 0 };
+ bool ret;
+
+ list.ids[list.nr++] = mptcp_endp_get_local_id(msk, addr);
+
+ ret = mptcp_pm_announced_remove(msk, addr);
+ if (ret || force) {
+ spin_lock_bh(&msk->pm.lock);
+ if (ret)
+ msk->pm.add_addr_signaled--;
+ mptcp_pm_remove_addr(msk, &list);
+ spin_unlock_bh(&msk->pm.lock);
+ }
+ return ret;
+}
+
+static void __mark_subflow_endp_available(struct mptcp_sock *msk, u8 id)
+{
+ /* If it was marked as used, and not ID 0, decrement local_addr_used */
+ if (!__test_and_set_bit(id ? : msk->mpc_endpoint_id, msk->pm.id_avail_bitmap) &&
+ id && !WARN_ON_ONCE(msk->pm.local_addr_used == 0))
+ msk->pm.local_addr_used--;
+}
+
+static int mptcp_nl_remove_subflow_and_signal_addr(struct net *net,
+ const struct mptcp_pm_addr_entry *entry)
+{
+ const struct mptcp_addr_info *addr = &entry->addr;
+ struct mptcp_rm_list list = { .nr = 1 };
+ long s_slot = 0, s_num = 0;
+ struct mptcp_sock *msk;
+
+ pr_debug("remove_id=%d\n", addr->id);
+
+ while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
+ struct sock *sk = (struct sock *)msk;
+ bool remove_subflow;
+
+ if (mptcp_pm_is_userspace(msk))
+ goto next;
+
+ lock_sock(sk);
+ remove_subflow = has_subflow_saddr(msk, addr);
+ mptcp_pm_remove_anno_addr(msk, addr, remove_subflow &&
+ !(entry->flags & MPTCP_PM_ADDR_FLAG_IMPLICIT));
+
+ list.ids[0] = mptcp_endp_get_local_id(msk, addr);
+
+ spin_lock_bh(&msk->pm.lock);
+ if (remove_subflow)
+ mptcp_pm_rm_subflow(msk, &list);
+ if (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW)
+ __mark_subflow_endp_available(msk, list.ids[0]);
+ else /* mark endp ID as available, e.g. Signal or MPC endp */
+ __set_bit(addr->id, msk->pm.id_avail_bitmap);
+ spin_unlock_bh(&msk->pm.lock);
+
+ if (msk->mpc_endpoint_id == entry->addr.id)
+ msk->mpc_endpoint_id = 0;
+ release_sock(sk);
+
+next:
+ sock_put(sk);
+ cond_resched();
+ }
+
+ return 0;
+}
+
+static int mptcp_nl_remove_id_zero_address(struct net *net,
+ struct mptcp_addr_info *addr)
+{
+ struct mptcp_rm_list list = { .nr = 0 };
+ long s_slot = 0, s_num = 0;
+ struct mptcp_sock *msk;
+
+ list.ids[list.nr++] = 0;
+
+ while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
+ struct sock *sk = (struct sock *)msk;
+ struct mptcp_addr_info msk_local;
+ struct mptcp_addr_info anno_addr;
+ bool announced;
+
+ if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
+ goto next;
+
+ mptcp_local_address((struct sock_common *)msk, &msk_local);
+ if (!mptcp_addresses_equal(&msk_local, addr, addr->port))
+ goto next;
+
+ lock_sock(sk);
+ /* Drop a possibly pending ADD_ADDR for this address. */
+ anno_addr = msk_local;
+ anno_addr.port = 0;
+ announced = mptcp_pm_announced_remove(msk, &anno_addr);
+ spin_lock_bh(&msk->pm.lock);
+ if (announced)
+ msk->pm.add_addr_signaled--;
+ mptcp_pm_remove_addr(msk, &list);
+ mptcp_pm_rm_subflow(msk, &list);
+ __mark_subflow_endp_available(msk, 0);
+ spin_unlock_bh(&msk->pm.lock);
+ release_sock(sk);
+
+next:
+ sock_put(sk);
+ cond_resched();
+ }
+
+ return 0;
+}
+
+int mptcp_pm_nl_del_addr_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct nlattr *attr = info->attrs[MPTCP_PM_ENDPOINT_ADDR];
+ struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
+ struct mptcp_pm_addr_entry addr, *entry;
+ unsigned int addr_max;
+ int ret;
+
+ ret = mptcp_pm_parse_entry(attr, info, false, &addr);
+ if (ret < 0)
+ return ret;
+
+ /* the zero id address is special: the first address used by the msk
+ * always gets such an id, so different subflows can have different zero
+ * id addresses. Additionally zero id is not accounted for in id_bitmap.
+ * Let's use an 'mptcp_rm_list' instead of the common remove code.
+ */
+ if (addr.addr.id == 0)
+ return mptcp_nl_remove_id_zero_address(sock_net(skb->sk), &addr.addr);
+
+ spin_lock_bh(&pernet->lock);
+ entry = __lookup_addr_by_id(pernet, addr.addr.id);
+ if (!entry) {
+ GENL_SET_ERR_MSG(info, "address not found");
+ spin_unlock_bh(&pernet->lock);
+ return -EINVAL;
+ }
+ if (entry->flags & MPTCP_PM_ADDR_FLAG_SIGNAL) {
+ addr_max = pernet->add_addr_signal_max;
+ WRITE_ONCE(pernet->add_addr_signal_max, addr_max - 1);
+ }
+ if (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW) {
+ addr_max = pernet->local_addr_max;
+ WRITE_ONCE(pernet->local_addr_max, addr_max - 1);
+ }
+
+ pernet->addrs--;
+ list_del_rcu(&entry->list);
+ __clear_bit(entry->addr.id, pernet->id_bitmap);
+ spin_unlock_bh(&pernet->lock);
+
+ mptcp_nl_remove_subflow_and_signal_addr(sock_net(skb->sk), entry);
+ synchronize_rcu();
+ __mptcp_pm_release_addr_entry(entry);
+
+ return ret;
+}
+
+/* Called from the userspace PM only */
+void mptcp_pm_remove_addrs(struct mptcp_sock *msk, struct list_head *rm_list)
+{
+ struct mptcp_rm_list alist = { .nr = 0 };
+ struct mptcp_pm_addr_entry *entry;
+ int anno_nr = 0;
+
+ list_for_each_entry(entry, rm_list, list) {
+ if (alist.nr >= MPTCP_RM_IDS_MAX)
+ break;
+
+ /* only delete if either announced or matching a subflow */
+ if (mptcp_pm_announced_remove(msk, &entry->addr))
+ anno_nr++;
+ else if (!has_subflow_saddr(msk, &entry->addr))
+ continue;
+
+ alist.ids[alist.nr++] = entry->addr.id;
+ }
+
+ if (alist.nr) {
+ spin_lock_bh(&msk->pm.lock);
+ msk->pm.add_addr_signaled -= anno_nr;
+ mptcp_pm_remove_addr(msk, &alist);
+ spin_unlock_bh(&msk->pm.lock);
+ }
+}
+
+/* Called from the in-kernel PM only */
+static void mptcp_pm_flush_addrs_and_subflows(struct mptcp_sock *msk,
+ struct list_head *rm_list)
+{
+ struct mptcp_rm_list alist = { .nr = 0 }, slist = { .nr = 0 };
+ struct mptcp_pm_addr_entry *entry;
+
+ list_for_each_entry(entry, rm_list, list) {
+ if (slist.nr < MPTCP_RM_IDS_MAX &&
+ has_subflow_saddr(msk, &entry->addr))
+ slist.ids[slist.nr++] = mptcp_endp_get_local_id(msk, &entry->addr);
+
+ if (alist.nr < MPTCP_RM_IDS_MAX &&
+ mptcp_pm_announced_remove(msk, &entry->addr))
+ alist.ids[alist.nr++] = mptcp_endp_get_local_id(msk, &entry->addr);
+ }
+
+ spin_lock_bh(&msk->pm.lock);
+ if (alist.nr) {
+ msk->pm.add_addr_signaled -= alist.nr;
+ mptcp_pm_remove_addr(msk, &alist);
+ }
+ if (slist.nr)
+ mptcp_pm_rm_subflow(msk, &slist);
+ /* Reset counters: maybe some subflows have been removed before */
+ bitmap_fill(msk->pm.id_avail_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
+ msk->pm.local_addr_used = 0;
+ spin_unlock_bh(&msk->pm.lock);
+}
+
+static void mptcp_nl_flush_addrs_list(struct net *net,
+ struct list_head *rm_list)
+{
+ long s_slot = 0, s_num = 0;
+ struct mptcp_sock *msk;
+
+ if (list_empty(rm_list))
+ return;
+
+ while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
+ struct sock *sk = (struct sock *)msk;
+
+ if (!mptcp_pm_is_userspace(msk)) {
+ lock_sock(sk);
+ mptcp_pm_flush_addrs_and_subflows(msk, rm_list);
+ release_sock(sk);
+ }
+
+ sock_put(sk);
+ cond_resched();
+ }
+}
+
+/* caller must ensure the RCU grace period is already elapsed */
+static void __flush_addrs(struct list_head *list)
+{
+ while (!list_empty(list)) {
+ struct mptcp_pm_addr_entry *cur;
+
+ cur = list_entry(list->next,
+ struct mptcp_pm_addr_entry, list);
+ list_del_rcu(&cur->list);
+ __mptcp_pm_release_addr_entry(cur);
+ }
+}
+
+static void __reset_counters(struct pm_nl_pernet *pernet)
+{
+ WRITE_ONCE(pernet->add_addr_signal_max, 0);
+ WRITE_ONCE(pernet->local_addr_max, 0);
+ pernet->addrs = 0;
+}
+
+int mptcp_pm_nl_flush_addrs_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
+ struct list_head free_list;
+
+ spin_lock_bh(&pernet->lock);
+ free_list = pernet->local_addr_list;
+ INIT_LIST_HEAD_RCU(&pernet->local_addr_list);
+ __reset_counters(pernet);
+ pernet->next_id = 1;
+ bitmap_zero(pernet->id_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
+ spin_unlock_bh(&pernet->lock);
+
+ if (free_list.next == &pernet->local_addr_list)
+ return 0;
+
+ synchronize_rcu();
+
+ /* Adjust the pointers to free_list instead of pernet->local_addr_list */
+ free_list.prev->next = &free_list;
+ free_list.next->prev = &free_list;
+
+ mptcp_nl_flush_addrs_list(sock_net(skb->sk), &free_list);
+ __flush_addrs(&free_list);
+ return 0;
+}
+
+int mptcp_nl_fill_addr(struct sk_buff *skb,
+ struct mptcp_pm_addr_entry *entry)
+{
+ struct mptcp_addr_info *addr = &entry->addr;
+ struct nlattr *attr;
+
+ attr = nla_nest_start(skb, MPTCP_PM_ATTR_ADDR);
+ if (!attr)
+ return -EMSGSIZE;
+
+ if (nla_put_u16(skb, MPTCP_PM_ADDR_ATTR_FAMILY, addr->family))
+ goto nla_put_failure;
+ if (nla_put_u16(skb, MPTCP_PM_ADDR_ATTR_PORT, ntohs(addr->port)))
+ goto nla_put_failure;
+ if (nla_put_u8(skb, MPTCP_PM_ADDR_ATTR_ID, addr->id))
+ goto nla_put_failure;
+ if (nla_put_u32(skb, MPTCP_PM_ADDR_ATTR_FLAGS, entry->flags))
+ goto nla_put_failure;
+ if (entry->ifindex &&
+ nla_put_s32(skb, MPTCP_PM_ADDR_ATTR_IF_IDX, entry->ifindex))
+ goto nla_put_failure;
+
+ if (addr->family == AF_INET &&
+ nla_put_in_addr(skb, MPTCP_PM_ADDR_ATTR_ADDR4,
+ addr->addr.s_addr))
+ goto nla_put_failure;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ else if (addr->family == AF_INET6 &&
+ nla_put_in6_addr(skb, MPTCP_PM_ADDR_ATTR_ADDR6, &addr->addr6))
+ goto nla_put_failure;
+#endif
+ nla_nest_end(skb, attr);
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, attr);
+ return -EMSGSIZE;
+}
+
+int mptcp_pm_nl_get_addr(struct sk_buff *skb, struct genl_info *info)
+{
+ struct nlattr *attr = info->attrs[MPTCP_PM_ENDPOINT_ADDR];
+ struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
+ struct mptcp_pm_addr_entry addr, *entry;
+ struct sk_buff *msg;
+ void *reply;
+ int ret;
+
+ ret = mptcp_pm_parse_entry(attr, info, false, &addr);
+ if (ret < 0)
+ return ret;
+
+ msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+ if (!msg)
+ return -ENOMEM;
+
+ reply = genlmsg_put_reply(msg, info, &mptcp_genl_family, 0,
+ info->genlhdr->cmd);
+ if (!reply) {
+ GENL_SET_ERR_MSG(info, "not enough space in Netlink message");
+ ret = -EMSGSIZE;
+ goto fail;
+ }
+
+ spin_lock_bh(&pernet->lock);
+ entry = __lookup_addr_by_id(pernet, addr.addr.id);
+ if (!entry) {
+ GENL_SET_ERR_MSG(info, "address not found");
+ ret = -EINVAL;
+ goto unlock_fail;
+ }
+
+ ret = mptcp_nl_fill_addr(msg, entry);
+ if (ret)
+ goto unlock_fail;
+
+ genlmsg_end(msg, reply);
+ ret = genlmsg_reply(msg, info);
+ spin_unlock_bh(&pernet->lock);
+ return ret;
+
+unlock_fail:
+ spin_unlock_bh(&pernet->lock);
+
+fail:
+ nlmsg_free(msg);
+ return ret;
+}
+
+int mptcp_pm_nl_get_addr_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ return mptcp_pm_get_addr(skb, info);
+}
+
+int mptcp_pm_nl_dump_addr(struct sk_buff *msg,
+ struct netlink_callback *cb)
+{
+ struct net *net = sock_net(msg->sk);
+ struct mptcp_pm_addr_entry *entry;
+ struct pm_nl_pernet *pernet;
+ int id = cb->args[0];
+ void *hdr;
+ int i;
+
+ pernet = pm_nl_get_pernet(net);
+
+ spin_lock_bh(&pernet->lock);
+ for (i = id; i < MPTCP_PM_MAX_ADDR_ID + 1; i++) {
+ if (test_bit(i, pernet->id_bitmap)) {
+ entry = __lookup_addr_by_id(pernet, i);
+ if (!entry)
+ break;
+
+ if (entry->addr.id <= id)
+ continue;
+
+ hdr = genlmsg_put(msg, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, &mptcp_genl_family,
+ NLM_F_MULTI, MPTCP_PM_CMD_GET_ADDR);
+ if (!hdr)
+ break;
+
+ if (mptcp_nl_fill_addr(msg, entry) < 0) {
+ genlmsg_cancel(msg, hdr);
+ break;
+ }
+
+ id = entry->addr.id;
+ genlmsg_end(msg, hdr);
+ }
+ }
+ spin_unlock_bh(&pernet->lock);
+
+ cb->args[0] = id;
+ return msg->len;
+}
+
+int mptcp_pm_nl_get_addr_dumpit(struct sk_buff *msg,
+ struct netlink_callback *cb)
+{
+ return mptcp_pm_dump_addr(msg, cb);
+}
+
+static int parse_limit(struct genl_info *info, int id, unsigned int *limit)
+{
+ struct nlattr *attr = info->attrs[id];
+
+ if (!attr)
+ return 0;
+
+ *limit = nla_get_u32(attr);
+ if (*limit > MPTCP_PM_ADDR_MAX) {
+ GENL_SET_ERR_MSG(info, "limit greater than maximum");
+ return -EINVAL;
+ }
+ return 0;
+}
+
+int mptcp_pm_nl_set_limits_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
+ unsigned int rcv_addrs, subflows;
+ int ret;
+
+ spin_lock_bh(&pernet->lock);
+ rcv_addrs = pernet->add_addr_accept_max;
+ ret = parse_limit(info, MPTCP_PM_ATTR_RCV_ADD_ADDRS, &rcv_addrs);
+ if (ret)
+ goto unlock;
+
+ subflows = pernet->subflows_max;
+ ret = parse_limit(info, MPTCP_PM_ATTR_SUBFLOWS, &subflows);
+ if (ret)
+ goto unlock;
+
+ WRITE_ONCE(pernet->add_addr_accept_max, rcv_addrs);
+ WRITE_ONCE(pernet->subflows_max, subflows);
+
+unlock:
+ spin_unlock_bh(&pernet->lock);
+ return ret;
+}
+
+int mptcp_pm_nl_get_limits_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
+ struct sk_buff *msg;
+ void *reply;
+
+ msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+ if (!msg)
+ return -ENOMEM;
+
+ reply = genlmsg_put_reply(msg, info, &mptcp_genl_family, 0,
+ MPTCP_PM_CMD_GET_LIMITS);
+ if (!reply)
+ goto fail;
+
+ if (nla_put_u32(msg, MPTCP_PM_ATTR_RCV_ADD_ADDRS,
+ READ_ONCE(pernet->add_addr_accept_max)))
+ goto fail;
+
+ if (nla_put_u32(msg, MPTCP_PM_ATTR_SUBFLOWS,
+ READ_ONCE(pernet->subflows_max)))
+ goto fail;
+
+ genlmsg_end(msg, reply);
+ return genlmsg_reply(msg, info);
+
+fail:
+ GENL_SET_ERR_MSG(info, "not enough space in Netlink message");
+ nlmsg_free(msg);
+ return -EMSGSIZE;
+}
+
+static void mptcp_pm_nl_fullmesh(struct mptcp_sock *msk,
+ struct mptcp_addr_info *addr)
+{
+ struct mptcp_rm_list list = { .nr = 0 };
+
+ list.ids[list.nr++] = mptcp_endp_get_local_id(msk, addr);
+
+ spin_lock_bh(&msk->pm.lock);
+ mptcp_pm_rm_subflow(msk, &list);
+ __mark_subflow_endp_available(msk, list.ids[0]);
+ mptcp_pm_create_subflow_or_signal_addr(msk);
+ spin_unlock_bh(&msk->pm.lock);
+}
+
+static int mptcp_nl_set_flags(struct net *net,
+ struct mptcp_addr_info *addr,
+ u8 bkup, u8 changed)
+{
+ long s_slot = 0, s_num = 0;
+ struct mptcp_sock *msk;
+ int ret = -EINVAL;
+
+ while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
+ struct sock *sk = (struct sock *)msk;
+
+ if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
+ goto next;
+
+ lock_sock(sk);
+ if (changed & MPTCP_PM_ADDR_FLAG_BACKUP)
+ ret = mptcp_pm_nl_mp_prio_send_ack(msk, addr, NULL, bkup);
+ if (changed & MPTCP_PM_ADDR_FLAG_FULLMESH)
+ mptcp_pm_nl_fullmesh(msk, addr);
+ release_sock(sk);
+
+next:
+ sock_put(sk);
+ cond_resched();
+ }
+
+ return ret;
+}
+
+int mptcp_pm_nl_set_flags(struct sk_buff *skb, struct genl_info *info)
+{
+ struct mptcp_pm_addr_entry addr = { .addr = { .family = AF_UNSPEC }, };
+ struct nlattr *attr = info->attrs[MPTCP_PM_ATTR_ADDR];
+ u8 changed, mask = MPTCP_PM_ADDR_FLAG_BACKUP |
+ MPTCP_PM_ADDR_FLAG_FULLMESH;
+ struct net *net = sock_net(skb->sk);
+ struct mptcp_pm_addr_entry *entry;
+ struct pm_nl_pernet *pernet;
+ u8 lookup_by_id = 0;
+ u8 bkup = 0;
+ int ret;
+
+ pernet = pm_nl_get_pernet(net);
+
+ ret = mptcp_pm_parse_entry(attr, info, false, &addr);
+ if (ret < 0)
+ return ret;
+
+ if (addr.addr.family == AF_UNSPEC) {
+ lookup_by_id = 1;
+ if (!addr.addr.id) {
+ GENL_SET_ERR_MSG(info, "missing required inputs");
+ return -EOPNOTSUPP;
+ }
+ }
+
+ if (addr.flags & MPTCP_PM_ADDR_FLAG_BACKUP)
+ bkup = 1;
+
+ spin_lock_bh(&pernet->lock);
+ entry = lookup_by_id ? __lookup_addr_by_id(pernet, addr.addr.id) :
+ __lookup_addr(pernet, &addr.addr);
+ if (!entry) {
+ spin_unlock_bh(&pernet->lock);
+ GENL_SET_ERR_MSG(info, "address not found");
+ return -EINVAL;
+ }
+ if ((addr.flags & MPTCP_PM_ADDR_FLAG_FULLMESH) &&
+ (entry->flags & (MPTCP_PM_ADDR_FLAG_SIGNAL |
+ MPTCP_PM_ADDR_FLAG_IMPLICIT))) {
+ spin_unlock_bh(&pernet->lock);
+ GENL_SET_ERR_MSG(info, "invalid addr flags");
+ return -EINVAL;
+ }
+
+ changed = (addr.flags ^ entry->flags) & mask;
+ entry->flags = (entry->flags & ~mask) | (addr.flags & mask);
+ addr = *entry;
+ spin_unlock_bh(&pernet->lock);
+
+ mptcp_nl_set_flags(net, &addr.addr, bkup, changed);
+ return 0;
+}
+
+int mptcp_pm_nl_set_flags_doit(struct sk_buff *skb, struct genl_info *info)
+{
+ return mptcp_pm_set_flags(skb, info);
+}
+
+static void mptcp_nl_mcast_send(struct net *net, struct sk_buff *nlskb, gfp_t gfp)
+{
+ genlmsg_multicast_netns(&mptcp_genl_family, net,
+ nlskb, 0, MPTCP_PM_EV_GRP_OFFSET, gfp);
+}
+
+bool mptcp_userspace_pm_active(const struct mptcp_sock *msk)
+{
+ return genl_has_listeners(&mptcp_genl_family,
+ sock_net((const struct sock *)msk),
+ MPTCP_PM_EV_GRP_OFFSET);
+}
+
+static int mptcp_event_add_subflow(struct sk_buff *skb, const struct sock *ssk)
+{
+ const struct inet_sock *issk = inet_sk(ssk);
+ const struct mptcp_subflow_context *sf;
+
+ if (nla_put_u16(skb, MPTCP_ATTR_FAMILY, ssk->sk_family))
+ return -EMSGSIZE;
+
+ switch (ssk->sk_family) {
+ case AF_INET:
+ if (nla_put_in_addr(skb, MPTCP_ATTR_SADDR4, issk->inet_saddr))
+ return -EMSGSIZE;
+ if (nla_put_in_addr(skb, MPTCP_ATTR_DADDR4, issk->inet_daddr))
+ return -EMSGSIZE;
+ break;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ case AF_INET6: {
+ const struct ipv6_pinfo *np = inet6_sk(ssk);
+
+ if (nla_put_in6_addr(skb, MPTCP_ATTR_SADDR6, &np->saddr))
+ return -EMSGSIZE;
+ if (nla_put_in6_addr(skb, MPTCP_ATTR_DADDR6, &ssk->sk_v6_daddr))
+ return -EMSGSIZE;
+ break;
+ }
+#endif
+ default:
+ WARN_ON_ONCE(1);
+ return -EMSGSIZE;
+ }
+
+ if (nla_put_be16(skb, MPTCP_ATTR_SPORT, issk->inet_sport))
+ return -EMSGSIZE;
+ if (nla_put_be16(skb, MPTCP_ATTR_DPORT, issk->inet_dport))
+ return -EMSGSIZE;
+
+ sf = mptcp_subflow_ctx(ssk);
+ if (WARN_ON_ONCE(!sf))
+ return -EINVAL;
+
+ if (nla_put_u8(skb, MPTCP_ATTR_LOC_ID, subflow_get_local_id(sf)))
+ return -EMSGSIZE;
+
+ if (nla_put_u8(skb, MPTCP_ATTR_REM_ID, sf->remote_id))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int mptcp_event_put_token_and_ssk(struct sk_buff *skb,
+ const struct mptcp_sock *msk,
+ const struct sock *ssk)
+{
+ const struct sock *sk = (const struct sock *)msk;
+ const struct mptcp_subflow_context *sf;
+ u8 sk_err;
+
+ if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)))
+ return -EMSGSIZE;
+
+ if (mptcp_event_add_subflow(skb, ssk))
+ return -EMSGSIZE;
+
+ sf = mptcp_subflow_ctx(ssk);
+ if (WARN_ON_ONCE(!sf))
+ return -EINVAL;
+
+ if (nla_put_u8(skb, MPTCP_ATTR_BACKUP, sf->backup))
+ return -EMSGSIZE;
+
+ if (ssk->sk_bound_dev_if &&
+ nla_put_s32(skb, MPTCP_ATTR_IF_IDX, ssk->sk_bound_dev_if))
+ return -EMSGSIZE;
+
+ sk_err = READ_ONCE(ssk->sk_err);
+ if (sk_err && sk->sk_state == TCP_ESTABLISHED &&
+ nla_put_u8(skb, MPTCP_ATTR_ERROR, sk_err))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int mptcp_event_sub_established(struct sk_buff *skb,
+ const struct mptcp_sock *msk,
+ const struct sock *ssk)
+{
+ return mptcp_event_put_token_and_ssk(skb, msk, ssk);
+}
+
+static int mptcp_event_sub_closed(struct sk_buff *skb,
+ const struct mptcp_sock *msk,
+ const struct sock *ssk)
+{
+ const struct mptcp_subflow_context *sf;
+
+ if (mptcp_event_put_token_and_ssk(skb, msk, ssk))
+ return -EMSGSIZE;
+
+ sf = mptcp_subflow_ctx(ssk);
+ if (!sf->reset_seen)
+ return 0;
+
+ if (nla_put_u32(skb, MPTCP_ATTR_RESET_REASON, sf->reset_reason))
+ return -EMSGSIZE;
+
+ if (nla_put_u32(skb, MPTCP_ATTR_RESET_FLAGS, sf->reset_transient))
+ return -EMSGSIZE;
+
+ return 0;
+}
+
+static int mptcp_event_created(struct sk_buff *skb,
+ const struct mptcp_sock *msk,
+ const struct sock *ssk)
+{
+ int err = nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token));
+ u16 flags = 0;
+
+ if (err)
+ return err;
+
+ if (nla_put_u8(skb, MPTCP_ATTR_SERVER_SIDE, READ_ONCE(msk->pm.server_side)))
+ return -EMSGSIZE;
+
+ if (READ_ONCE(msk->pm.remote_deny_join_id0))
+ flags |= MPTCP_PM_EV_FLAG_DENY_JOIN_ID0;
+
+ if (flags && nla_put_u16(skb, MPTCP_ATTR_FLAGS, flags))
+ return -EMSGSIZE;
+
+ return mptcp_event_add_subflow(skb, ssk);
+}
+
+void mptcp_event_addr_removed(const struct mptcp_sock *msk, uint8_t id)
+{
+ struct net *net = sock_net((const struct sock *)msk);
+ struct nlmsghdr *nlh;
+ struct sk_buff *skb;
+
+ if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
+ return;
+
+ skb = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_ATOMIC);
+ if (!skb)
+ return;
+
+ nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0, MPTCP_EVENT_REMOVED);
+ if (!nlh)
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)))
+ goto nla_put_failure;
+
+ if (nla_put_u8(skb, MPTCP_ATTR_REM_ID, id))
+ goto nla_put_failure;
+
+ genlmsg_end(skb, nlh);
+ mptcp_nl_mcast_send(net, skb, GFP_ATOMIC);
+ return;
+
+nla_put_failure:
+ nlmsg_free(skb);
+}
+
+void mptcp_event_addr_announced(const struct sock *ssk,
+ const struct mptcp_addr_info *info)
+{
+ struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
+ struct mptcp_sock *msk = mptcp_sk(subflow->conn);
+ struct net *net = sock_net(ssk);
+ struct nlmsghdr *nlh;
+ struct sk_buff *skb;
+
+ if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
+ return;
+
+ skb = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_ATOMIC);
+ if (!skb)
+ return;
+
+ nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0,
+ MPTCP_EVENT_ANNOUNCED);
+ if (!nlh)
+ goto nla_put_failure;
+
+ if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)))
+ goto nla_put_failure;
+
+ if (nla_put_u8(skb, MPTCP_ATTR_REM_ID, info->id))
+ goto nla_put_failure;
+
+ if (nla_put_be16(skb, MPTCP_ATTR_DPORT,
+ info->port == 0 ?
+ inet_sk(ssk)->inet_dport :
+ info->port))
+ goto nla_put_failure;
+
+ switch (info->family) {
+ case AF_INET:
+ if (nla_put_in_addr(skb, MPTCP_ATTR_DADDR4, info->addr.s_addr))
+ goto nla_put_failure;
+ break;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ case AF_INET6:
+ if (nla_put_in6_addr(skb, MPTCP_ATTR_DADDR6, &info->addr6))
+ goto nla_put_failure;
+ break;
+#endif
+ default:
+ WARN_ON_ONCE(1);
+ goto nla_put_failure;
+ }
+
+ genlmsg_end(skb, nlh);
+ mptcp_nl_mcast_send(net, skb, GFP_ATOMIC);
+ return;
+
+nla_put_failure:
+ nlmsg_free(skb);
+}
+
+void mptcp_event_pm_listener(const struct sock *ssk,
+ enum mptcp_event_type event)
+{
+ const struct inet_sock *issk = inet_sk(ssk);
+ struct net *net = sock_net(ssk);
+ struct nlmsghdr *nlh;
+ struct sk_buff *skb;
+
+ if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
+ return;
+
+ skb = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+ if (!skb)
+ return;
+
+ nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0, event);
+ if (!nlh)
+ goto nla_put_failure;
+
+ if (nla_put_u16(skb, MPTCP_ATTR_FAMILY, ssk->sk_family))
+ goto nla_put_failure;
+
+ if (nla_put_be16(skb, MPTCP_ATTR_SPORT, issk->inet_sport))
+ goto nla_put_failure;
+
+ switch (ssk->sk_family) {
+ case AF_INET:
+ if (nla_put_in_addr(skb, MPTCP_ATTR_SADDR4, issk->inet_saddr))
+ goto nla_put_failure;
+ break;
+#if IS_ENABLED(CONFIG_MPTCP_IPV6)
+ case AF_INET6: {
+ const struct ipv6_pinfo *np = inet6_sk(ssk);
+
+ if (nla_put_in6_addr(skb, MPTCP_ATTR_SADDR6, &np->saddr))
+ goto nla_put_failure;
+ break;
+ }
+#endif
+ default:
+ WARN_ON_ONCE(1);
+ goto nla_put_failure;
+ }
+
+ genlmsg_end(skb, nlh);
+ mptcp_nl_mcast_send(net, skb, GFP_KERNEL);
+ return;
+
+nla_put_failure:
+ nlmsg_free(skb);
+}
+
+void mptcp_event(enum mptcp_event_type type, const struct mptcp_sock *msk,
+ const struct sock *ssk, gfp_t gfp)
+{
+ struct net *net = sock_net((const struct sock *)msk);
+ struct nlmsghdr *nlh;
+ struct sk_buff *skb;
+
+ if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
+ return;
+
+ skb = nlmsg_new(NLMSG_DEFAULT_SIZE, gfp);
+ if (!skb)
+ return;
+
+ nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0, type);
+ if (!nlh)
+ goto nla_put_failure;
+
+ switch (type) {
+ case MPTCP_EVENT_UNSPEC:
+ WARN_ON_ONCE(1);
+ break;
+ case MPTCP_EVENT_CREATED:
+ case MPTCP_EVENT_ESTABLISHED:
+ if (mptcp_event_created(skb, msk, ssk) < 0)
+ goto nla_put_failure;
+ break;
+ case MPTCP_EVENT_CLOSED:
+ if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)) < 0)
+ goto nla_put_failure;
+ break;
+ case MPTCP_EVENT_ANNOUNCED:
+ case MPTCP_EVENT_REMOVED:
+ /* call mptcp_event_addr_announced()/removed instead */
+ WARN_ON_ONCE(1);
+ break;
+ case MPTCP_EVENT_SUB_ESTABLISHED:
+ case MPTCP_EVENT_SUB_PRIORITY:
+ if (mptcp_event_sub_established(skb, msk, ssk) < 0)
+ goto nla_put_failure;
+ break;
+ case MPTCP_EVENT_SUB_CLOSED:
+ if (mptcp_event_sub_closed(skb, msk, ssk) < 0)
+ goto nla_put_failure;
+ break;
+ case MPTCP_EVENT_LISTENER_CREATED:
+ case MPTCP_EVENT_LISTENER_CLOSED:
+ break;
+ }
+
+ genlmsg_end(skb, nlh);
+ mptcp_nl_mcast_send(net, skb, gfp);
+ return;
+
+nla_put_failure:
+ nlmsg_free(skb);
+}
+
+struct genl_family mptcp_genl_family __ro_after_init = {
+ .name = MPTCP_PM_NAME,
+ .version = MPTCP_PM_VER,
+ .netnsok = true,
+ .module = THIS_MODULE,
+ .ops = mptcp_pm_nl_ops,
+ .n_ops = ARRAY_SIZE(mptcp_pm_nl_ops),
+ .resv_start_op = MPTCP_PM_CMD_SUBFLOW_DESTROY + 1,
+ .mcgrps = mptcp_pm_mcgrps,
+ .n_mcgrps = ARRAY_SIZE(mptcp_pm_mcgrps),
+};
+
+static int __net_init pm_nl_init_net(struct net *net)
+{
+ struct pm_nl_pernet *pernet = pm_nl_get_pernet(net);
+
+ INIT_LIST_HEAD_RCU(&pernet->local_addr_list);
+
+ /* Cit. 2 subflows ought to be enough for anybody. */
+ pernet->subflows_max = 2;
+ pernet->next_id = 1;
+ pernet->stale_loss_cnt = 4;
+ spin_lock_init(&pernet->lock);
+
+ /* No need to initialize other pernet fields, the struct is zeroed at
+ * allocation time.
+ */
+
+ return 0;
+}
+
+static void __net_exit pm_nl_exit_net(struct list_head *net_list)
+{
+ struct net *net;
+
+ list_for_each_entry(net, net_list, exit_list) {
+ struct pm_nl_pernet *pernet = pm_nl_get_pernet(net);
+
+ /* net is removed from namespace list, can't race with
+ * other modifiers, also netns core already waited for a
+ * RCU grace period.
+ */
+ __flush_addrs(&pernet->local_addr_list);
+ }
+}
+
+static struct pernet_operations mptcp_pm_pernet_ops = {
+ .init = pm_nl_init_net,
+ .exit_batch = pm_nl_exit_net,
+ .id = &pm_nl_pernet_id,
+ .size = sizeof(struct pm_nl_pernet),
+};
+
+void __init mptcp_pm_nl_init(void)
+{
+ if (register_pernet_subsys(&mptcp_pm_pernet_ops) < 0)
+ panic("Failed to register MPTCP PM pernet subsystem.\n");
+
+ if (genl_register_family(&mptcp_genl_family))
+ panic("Failed to register MPTCP PM netlink family\n");
+}
diff --git a/net/mptcp/pm_netlink.c b/net/mptcp/pm_netlink.c
deleted file mode 100644
index 7ea8ccd16df2..000000000000
--- a/net/mptcp/pm_netlink.c
+++ /dev/null
@@ -1,2603 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/* Multipath TCP
- *
- * Copyright (c) 2020, Red Hat, Inc.
- */
-
-#define pr_fmt(fmt) "MPTCP: " fmt
-
-#include <linux/inet.h>
-#include <linux/kernel.h>
-#include <net/inet_common.h>
-#include <net/netns/generic.h>
-#include <net/mptcp.h>
-
-#include "protocol.h"
-#include "mib.h"
-#include "mptcp_pm_gen.h"
-
-static int pm_nl_pernet_id;
-
-struct mptcp_pm_add_entry {
- struct list_head list;
- struct mptcp_addr_info addr;
- u8 retrans_times;
- bool timer_done;
- struct timer_list add_timer;
- struct mptcp_sock *sock;
- struct rcu_head rcu;
-};
-
-struct pm_nl_pernet {
- /* protects pernet updates */
- spinlock_t lock;
- struct list_head local_addr_list;
- unsigned int addrs;
- unsigned int stale_loss_cnt;
- unsigned int add_addr_signal_max;
- unsigned int add_addr_accept_max;
- unsigned int local_addr_max;
- unsigned int subflows_max;
- unsigned int next_id;
- DECLARE_BITMAP(id_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
-};
-
-#define MPTCP_PM_ADDR_MAX 8
-#define ADD_ADDR_RETRANS_MAX 3
-
-static struct pm_nl_pernet *pm_nl_get_pernet(const struct net *net)
-{
- return net_generic(net, pm_nl_pernet_id);
-}
-
-static struct pm_nl_pernet *
-pm_nl_get_pernet_from_msk(const struct mptcp_sock *msk)
-{
- return pm_nl_get_pernet(sock_net((struct sock *)msk));
-}
-
-bool mptcp_addresses_equal(const struct mptcp_addr_info *a,
- const struct mptcp_addr_info *b, bool use_port)
-{
- bool addr_equals = false;
-
- if (a->family == b->family) {
- if (a->family == AF_INET)
- addr_equals = a->addr.s_addr == b->addr.s_addr;
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- else
- addr_equals = !ipv6_addr_cmp(&a->addr6, &b->addr6);
- } else if (a->family == AF_INET) {
- if (ipv6_addr_v4mapped(&b->addr6))
- addr_equals = a->addr.s_addr == b->addr6.s6_addr32[3];
- } else if (b->family == AF_INET) {
- if (ipv6_addr_v4mapped(&a->addr6))
- addr_equals = a->addr6.s6_addr32[3] == b->addr.s_addr;
-#endif
- }
-
- if (!addr_equals)
- return false;
- if (!use_port)
- return true;
-
- return a->port == b->port;
-}
-
-void mptcp_local_address(const struct sock_common *skc, struct mptcp_addr_info *addr)
-{
- addr->family = skc->skc_family;
- addr->port = htons(skc->skc_num);
- if (addr->family == AF_INET)
- addr->addr.s_addr = skc->skc_rcv_saddr;
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- else if (addr->family == AF_INET6)
- addr->addr6 = skc->skc_v6_rcv_saddr;
-#endif
-}
-
-static void remote_address(const struct sock_common *skc,
- struct mptcp_addr_info *addr)
-{
- addr->family = skc->skc_family;
- addr->port = skc->skc_dport;
- if (addr->family == AF_INET)
- addr->addr.s_addr = skc->skc_daddr;
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- else if (addr->family == AF_INET6)
- addr->addr6 = skc->skc_v6_daddr;
-#endif
-}
-
-static bool lookup_subflow_by_saddr(const struct list_head *list,
- const struct mptcp_addr_info *saddr)
-{
- struct mptcp_subflow_context *subflow;
- struct mptcp_addr_info cur;
- struct sock_common *skc;
-
- list_for_each_entry(subflow, list, node) {
- skc = (struct sock_common *)mptcp_subflow_tcp_sock(subflow);
-
- mptcp_local_address(skc, &cur);
- if (mptcp_addresses_equal(&cur, saddr, saddr->port))
- return true;
- }
-
- return false;
-}
-
-static bool lookup_subflow_by_daddr(const struct list_head *list,
- const struct mptcp_addr_info *daddr)
-{
- struct mptcp_subflow_context *subflow;
- struct mptcp_addr_info cur;
-
- list_for_each_entry(subflow, list, node) {
- struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
-
- if (!((1 << inet_sk_state_load(ssk)) &
- (TCPF_ESTABLISHED | TCPF_SYN_SENT | TCPF_SYN_RECV)))
- continue;
-
- remote_address((struct sock_common *)ssk, &cur);
- if (mptcp_addresses_equal(&cur, daddr, daddr->port))
- return true;
- }
-
- return false;
-}
-
-static bool
-select_local_address(const struct pm_nl_pernet *pernet,
- const struct mptcp_sock *msk,
- struct mptcp_pm_local *new_local)
-{
- struct mptcp_pm_addr_entry *entry;
- bool found = false;
-
- msk_owned_by_me(msk);
-
- rcu_read_lock();
- list_for_each_entry_rcu(entry, &pernet->local_addr_list, list) {
- if (!(entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW))
- continue;
-
- if (!test_bit(entry->addr.id, msk->pm.id_avail_bitmap))
- continue;
-
- new_local->addr = entry->addr;
- new_local->flags = entry->flags;
- new_local->ifindex = entry->ifindex;
- found = true;
- break;
- }
- rcu_read_unlock();
-
- return found;
-}
-
-static bool
-select_signal_address(struct pm_nl_pernet *pernet, const struct mptcp_sock *msk,
- struct mptcp_pm_local *new_local)
-{
- struct mptcp_pm_addr_entry *entry;
- bool found = false;
-
- rcu_read_lock();
- /* do not keep any additional per socket state, just signal
- * the address list in order.
- * Note: removal from the local address list during the msk life-cycle
- * can lead to additional addresses not being announced.
- */
- list_for_each_entry_rcu(entry, &pernet->local_addr_list, list) {
- if (!test_bit(entry->addr.id, msk->pm.id_avail_bitmap))
- continue;
-
- if (!(entry->flags & MPTCP_PM_ADDR_FLAG_SIGNAL))
- continue;
-
- new_local->addr = entry->addr;
- new_local->flags = entry->flags;
- new_local->ifindex = entry->ifindex;
- found = true;
- break;
- }
- rcu_read_unlock();
-
- return found;
-}
-
-unsigned int mptcp_pm_get_add_addr_signal_max(const struct mptcp_sock *msk)
-{
- const struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
-
- return READ_ONCE(pernet->add_addr_signal_max);
-}
-EXPORT_SYMBOL_GPL(mptcp_pm_get_add_addr_signal_max);
-
-unsigned int mptcp_pm_get_add_addr_accept_max(const struct mptcp_sock *msk)
-{
- struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
-
- return READ_ONCE(pernet->add_addr_accept_max);
-}
-EXPORT_SYMBOL_GPL(mptcp_pm_get_add_addr_accept_max);
-
-unsigned int mptcp_pm_get_subflows_max(const struct mptcp_sock *msk)
-{
- struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
-
- return READ_ONCE(pernet->subflows_max);
-}
-EXPORT_SYMBOL_GPL(mptcp_pm_get_subflows_max);
-
-unsigned int mptcp_pm_get_local_addr_max(const struct mptcp_sock *msk)
-{
- struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
-
- return READ_ONCE(pernet->local_addr_max);
-}
-EXPORT_SYMBOL_GPL(mptcp_pm_get_local_addr_max);
-
-bool mptcp_pm_nl_check_work_pending(struct mptcp_sock *msk)
-{
- struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
-
- if (msk->pm.subflows == mptcp_pm_get_subflows_max(msk) ||
- (find_next_and_bit(pernet->id_bitmap, msk->pm.id_avail_bitmap,
- MPTCP_PM_MAX_ADDR_ID + 1, 0) == MPTCP_PM_MAX_ADDR_ID + 1)) {
- WRITE_ONCE(msk->pm.work_pending, false);
- return false;
- }
- return true;
-}
-
-struct mptcp_pm_add_entry *
-mptcp_lookup_anno_list_by_saddr(const struct mptcp_sock *msk,
- const struct mptcp_addr_info *addr)
-{
- struct mptcp_pm_add_entry *entry;
-
- lockdep_assert_held(&msk->pm.lock);
-
- list_for_each_entry(entry, &msk->pm.anno_list, list) {
- if (mptcp_addresses_equal(&entry->addr, addr, true))
- return entry;
- }
-
- return NULL;
-}
-
-bool mptcp_pm_sport_in_anno_list(struct mptcp_sock *msk, const struct sock *sk)
-{
- struct mptcp_pm_add_entry *entry;
- struct mptcp_addr_info saddr;
- bool ret = false;
-
- mptcp_local_address((struct sock_common *)sk, &saddr);
-
- spin_lock_bh(&msk->pm.lock);
- list_for_each_entry(entry, &msk->pm.anno_list, list) {
- if (mptcp_addresses_equal(&entry->addr, &saddr, true)) {
- ret = true;
- goto out;
- }
- }
-
-out:
- spin_unlock_bh(&msk->pm.lock);
- return ret;
-}
-
-static void mptcp_pm_add_timer(struct timer_list *timer)
-{
- struct mptcp_pm_add_entry *entry = from_timer(entry, timer, add_timer);
- struct mptcp_sock *msk = entry->sock;
- struct sock *sk = (struct sock *)msk;
- unsigned int timeout = 0;
- bool retransmit;
-
- pr_debug("msk=%p\n", msk);
-
- bh_lock_sock(sk);
- if (unlikely(inet_sk_state_load(sk) == TCP_CLOSE))
- goto out;
-
- if (sock_owned_by_user(sk)) {
- /* Try again later. */
- timeout = HZ / 20;
- goto out;
- }
-
- if (mptcp_pm_should_add_signal_addr(msk)) {
- timeout = HZ;
- goto out;
- }
-
- timeout = mptcp_get_add_addr_timeout(sock_net(sk));
- if (!timeout)
- goto out;
-
- spin_lock_bh(&msk->pm.lock);
-
- /* The cancel path (mptcp_pm_del_add_timer()) can race with this
- * callback. Once cancel updates retrans_times to MAX, suppress further
- * retransmissions here. If this callback acquires pm.lock first, one
- * final transmit attempt is still possible.
- */
- if (entry->retrans_times < ADD_ADDR_RETRANS_MAX &&
- !mptcp_pm_should_add_signal_addr(msk)) {
- pr_debug("retransmit ADD_ADDR id=%d\n", entry->addr.id);
- mptcp_pm_announce_addr(msk, &entry->addr, false);
- mptcp_pm_add_addr_send_ack(msk);
- entry->retrans_times++;
- }
-
- retransmit = entry->retrans_times < ADD_ADDR_RETRANS_MAX;
- if (!retransmit)
- timeout = 0;
-
- spin_unlock_bh(&msk->pm.lock);
-
- if (!retransmit)
- mptcp_pm_subflow_established(msk);
-
-out:
- if (timeout)
- sk_reset_timer(sk, timer, jiffies + timeout);
- else
- /* if sock_put calls sk_free: avoid waiting for this timer */
- entry->timer_done = true;
- bh_unlock_sock(sk);
- sock_put(sk);
-}
-
-struct mptcp_pm_add_entry *
-mptcp_pm_del_add_timer(struct mptcp_sock *msk,
- const struct mptcp_addr_info *addr, bool check_id)
-{
- struct mptcp_pm_add_entry *entry;
- struct sock *sk = (struct sock *)msk;
- bool stop_timer = false;
-
- rcu_read_lock();
-
- spin_lock_bh(&msk->pm.lock);
- entry = mptcp_lookup_anno_list_by_saddr(msk, addr);
- if (entry && (!check_id || entry->addr.id == addr->id)) {
- entry->retrans_times = ADD_ADDR_RETRANS_MAX;
- stop_timer = true;
- }
- if (!check_id && entry)
- list_del(&entry->list);
- spin_unlock_bh(&msk->pm.lock);
-
- /* Note: entry might have been removed by another thread.
- * We hold rcu_read_lock() to ensure it is not freed under us.
- */
- if (stop_timer) {
- if (check_id)
- sk_stop_timer(sk, &entry->add_timer);
- else
- sk_stop_timer_sync(sk, &entry->add_timer);
- }
-
- rcu_read_unlock();
- return entry;
-}
-
-bool mptcp_pm_alloc_anno_list(struct mptcp_sock *msk,
- const struct mptcp_addr_info *addr)
-{
- struct mptcp_pm_add_entry *add_entry = NULL;
- struct sock *sk = (struct sock *)msk;
- struct net *net = sock_net(sk);
- unsigned int timeout;
-
- lockdep_assert_held(&msk->pm.lock);
-
- if (msk->pm.status & BIT(MPTCP_PM_DESTROYING))
- return false;
-
- add_entry = mptcp_lookup_anno_list_by_saddr(msk, addr);
-
- if (add_entry) {
- if (WARN_ON_ONCE(mptcp_pm_is_kernel(msk)))
- return false;
-
- goto reset_timer;
- }
-
- add_entry = kmalloc(sizeof(*add_entry), GFP_ATOMIC);
- if (!add_entry)
- return false;
-
- list_add(&add_entry->list, &msk->pm.anno_list);
-
- add_entry->addr = *addr;
- add_entry->sock = msk;
- add_entry->retrans_times = 0;
-
- timer_setup(&add_entry->add_timer, mptcp_pm_add_timer, 0);
-reset_timer:
- add_entry->timer_done = false;
- timeout = mptcp_get_add_addr_timeout(net);
- if (timeout)
- sk_reset_timer(sk, &add_entry->add_timer, jiffies + timeout);
-
- return true;
-}
-
-void mptcp_pm_free_anno_list(struct mptcp_sock *msk)
-{
- struct mptcp_pm_add_entry *entry, *tmp;
- struct sock *sk = (struct sock *)msk;
- LIST_HEAD(free_list);
-
- pr_debug("msk=%p\n", msk);
-
- spin_lock_bh(&msk->pm.lock);
- list_splice_init(&msk->pm.anno_list, &free_list);
- spin_unlock_bh(&msk->pm.lock);
-
- list_for_each_entry_safe(entry, tmp, &free_list, list) {
- if (!entry->timer_done)
- sk_stop_timer_sync(sk, &entry->add_timer);
- kfree_rcu(entry, rcu);
- }
-}
-
-/* Fill all the remote addresses into the array addrs[],
- * and return the array size.
- */
-static unsigned int fill_remote_addresses_vec(struct mptcp_sock *msk,
- struct mptcp_addr_info *local,
- bool fullmesh,
- struct mptcp_addr_info *addrs)
-{
- bool deny_id0 = READ_ONCE(msk->pm.remote_deny_join_id0);
- struct sock *sk = (struct sock *)msk, *ssk;
- struct mptcp_subflow_context *subflow;
- struct mptcp_addr_info remote = { 0 };
- unsigned int subflows_max;
- int i = 0;
-
- subflows_max = mptcp_pm_get_subflows_max(msk);
- remote_address((struct sock_common *)sk, &remote);
-
- /* Non-fullmesh endpoint, fill in the single entry
- * corresponding to the primary MPC subflow remote address
- */
- if (!fullmesh) {
- if (deny_id0)
- return 0;
-
- if (!mptcp_pm_addr_families_match(sk, local, &remote))
- return 0;
-
- msk->pm.subflows++;
- addrs[i++] = remote;
- } else {
- DECLARE_BITMAP(unavail_id, MPTCP_PM_MAX_ADDR_ID + 1);
-
- /* Forbid creation of new subflows matching existing
- * ones, possibly already created by incoming ADD_ADDR
- */
- bitmap_zero(unavail_id, MPTCP_PM_MAX_ADDR_ID + 1);
- mptcp_for_each_subflow(msk, subflow)
- if (READ_ONCE(subflow->local_id) == local->id)
- __set_bit(subflow->remote_id, unavail_id);
-
- mptcp_for_each_subflow(msk, subflow) {
- ssk = mptcp_subflow_tcp_sock(subflow);
- remote_address((struct sock_common *)ssk, &addrs[i]);
- addrs[i].id = READ_ONCE(subflow->remote_id);
- if (deny_id0 && !addrs[i].id)
- continue;
-
- if (test_bit(addrs[i].id, unavail_id))
- continue;
-
- if (!mptcp_pm_addr_families_match(sk, local, &addrs[i]))
- continue;
-
- if (msk->pm.subflows < subflows_max) {
- /* forbid creating multiple address towards
- * this id
- */
- __set_bit(addrs[i].id, unavail_id);
- msk->pm.subflows++;
- i++;
- }
- }
- }
-
- return i;
-}
-
-static void __mptcp_pm_send_ack(struct mptcp_sock *msk, struct mptcp_subflow_context *subflow,
- bool prio, bool backup)
-{
- struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
- bool slow;
-
- pr_debug("send ack for %s\n",
- prio ? "mp_prio" : (mptcp_pm_should_add_signal(msk) ? "add_addr" : "rm_addr"));
-
- slow = lock_sock_fast(ssk);
- if (prio) {
- subflow->send_mp_prio = 1;
- subflow->request_bkup = backup;
- }
-
- __mptcp_subflow_send_ack(ssk);
- unlock_sock_fast(ssk, slow);
-}
-
-static void mptcp_pm_send_ack(struct mptcp_sock *msk, struct mptcp_subflow_context *subflow,
- bool prio, bool backup)
-{
- spin_unlock_bh(&msk->pm.lock);
- __mptcp_pm_send_ack(msk, subflow, prio, backup);
- spin_lock_bh(&msk->pm.lock);
-}
-
-static struct mptcp_pm_addr_entry *
-__lookup_addr_by_id(struct pm_nl_pernet *pernet, unsigned int id)
-{
- struct mptcp_pm_addr_entry *entry;
-
- list_for_each_entry(entry, &pernet->local_addr_list, list) {
- if (entry->addr.id == id)
- return entry;
- }
- return NULL;
-}
-
-static struct mptcp_pm_addr_entry *
-__lookup_addr(struct pm_nl_pernet *pernet, const struct mptcp_addr_info *info)
-{
- struct mptcp_pm_addr_entry *entry;
-
- list_for_each_entry_rcu(entry, &pernet->local_addr_list, list,
- lockdep_is_held(&pernet->lock)) {
- if (mptcp_addresses_equal(&entry->addr, info, entry->addr.port))
- return entry;
- }
- return NULL;
-}
-
-static void mptcp_pm_create_subflow_or_signal_addr(struct mptcp_sock *msk)
-{
- struct sock *sk = (struct sock *)msk;
- unsigned int add_addr_signal_max;
- bool signal_and_subflow = false;
- unsigned int local_addr_max;
- struct pm_nl_pernet *pernet;
- struct mptcp_pm_local local;
- unsigned int subflows_max;
-
- pernet = pm_nl_get_pernet(sock_net(sk));
-
- add_addr_signal_max = mptcp_pm_get_add_addr_signal_max(msk);
- local_addr_max = mptcp_pm_get_local_addr_max(msk);
- subflows_max = mptcp_pm_get_subflows_max(msk);
-
- /* do lazy endpoint usage accounting for the MPC subflows */
- if (unlikely(!(msk->pm.status & BIT(MPTCP_PM_MPC_ENDPOINT_ACCOUNTED))) && msk->first) {
- struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(msk->first);
- struct mptcp_pm_addr_entry *entry;
- struct mptcp_addr_info mpc_addr;
- bool backup = false;
-
- mptcp_local_address((struct sock_common *)msk->first, &mpc_addr);
- rcu_read_lock();
- entry = __lookup_addr(pernet, &mpc_addr);
- if (entry) {
- __clear_bit(entry->addr.id, msk->pm.id_avail_bitmap);
- msk->mpc_endpoint_id = entry->addr.id;
- backup = !!(entry->flags & MPTCP_PM_ADDR_FLAG_BACKUP);
- }
- rcu_read_unlock();
-
- if (backup)
- mptcp_pm_send_ack(msk, subflow, true, backup);
-
- msk->pm.status |= BIT(MPTCP_PM_MPC_ENDPOINT_ACCOUNTED);
- }
-
- pr_debug("local %d:%d signal %d:%d subflows %d:%d\n",
- msk->pm.local_addr_used, local_addr_max,
- msk->pm.add_addr_signaled, add_addr_signal_max,
- msk->pm.subflows, subflows_max);
-
- /* check first for announce */
- if (msk->pm.add_addr_signaled < add_addr_signal_max) {
- u8 endp_id;
-
- /* due to racing events on both ends we can reach here while
- * previous add address is still running: if we invoke now
- * mptcp_pm_announce_addr(), that will fail and the
- * corresponding id will be marked as used.
- * Instead let the PM machinery reschedule us when the
- * current address announce will be completed.
- */
- if (msk->pm.addr_signal & BIT(MPTCP_ADD_ADDR_SIGNAL))
- return;
-
- if (!select_signal_address(pernet, msk, &local))
- goto subflow;
-
- /* Special case for ID0: set the correct ID */
- endp_id = local.addr.id;
- if (endp_id == msk->mpc_endpoint_id)
- local.addr.id = 0;
-
- /* If the alloc fails, we are on memory pressure, not worth
- * continuing, and trying to create subflows.
- */
- if (!mptcp_pm_alloc_anno_list(msk, &local.addr))
- return;
-
- __clear_bit(endp_id, msk->pm.id_avail_bitmap);
- msk->pm.add_addr_signaled++;
-
- mptcp_pm_announce_addr(msk, &local.addr, false);
- mptcp_pm_nl_addr_send_ack(msk);
-
- if (local.flags & MPTCP_PM_ADDR_FLAG_SUBFLOW)
- signal_and_subflow = true;
- }
-
-subflow:
- /* No need to try establishing subflows to remote id0 if not allowed */
- if (mptcp_pm_add_addr_c_flag_case(msk))
- goto exit;
-
- /* check if should create a new subflow */
- while (msk->pm.local_addr_used < local_addr_max &&
- msk->pm.subflows < subflows_max) {
- struct mptcp_addr_info addrs[MPTCP_PM_ADDR_MAX];
- bool fullmesh;
- int i, nr;
-
- if (signal_and_subflow)
- signal_and_subflow = false;
- else if (!select_local_address(pernet, msk, &local))
- break;
-
- fullmesh = !!(local.flags & MPTCP_PM_ADDR_FLAG_FULLMESH);
-
- __clear_bit(local.addr.id, msk->pm.id_avail_bitmap);
-
- /* Special case for ID0: set the correct ID */
- if (local.addr.id == msk->mpc_endpoint_id)
- local.addr.id = 0;
- else /* local_addr_used is not decr for ID 0 */
- msk->pm.local_addr_used++;
-
- nr = fill_remote_addresses_vec(msk, &local.addr, fullmesh, addrs);
- if (nr == 0)
- continue;
-
- spin_unlock_bh(&msk->pm.lock);
- for (i = 0; i < nr; i++)
- __mptcp_subflow_connect(sk, &local, &addrs[i]);
- spin_lock_bh(&msk->pm.lock);
- }
-
-exit:
- /* If an endpoint has both the signal and subflow flags, but it is not
- * possible to create subflows -- the 'while' loop body above never
- * executed -- then still mark the endp as used, which is somehow the
- * case. This avoids issues later when removing the endpoint and calling
- * __mark_subflow_endp_available(), which expects the increment here.
- */
- if (signal_and_subflow && local.addr.id != msk->mpc_endpoint_id)
- msk->pm.local_addr_used++;
-
- mptcp_pm_nl_check_work_pending(msk);
-}
-
-static void mptcp_pm_nl_fully_established(struct mptcp_sock *msk)
-{
- mptcp_pm_create_subflow_or_signal_addr(msk);
-}
-
-static void mptcp_pm_nl_subflow_established(struct mptcp_sock *msk)
-{
- mptcp_pm_create_subflow_or_signal_addr(msk);
-}
-
-/* Fill all the local addresses into the array addrs[],
- * and return the array size.
- */
-static unsigned int fill_local_addresses_vec(struct mptcp_sock *msk,
- struct mptcp_addr_info *remote,
- struct mptcp_pm_local *locals)
-{
- struct sock *sk = (struct sock *)msk;
- struct mptcp_pm_addr_entry *entry;
- struct mptcp_addr_info mpc_addr;
- struct pm_nl_pernet *pernet;
- unsigned int subflows_max;
- bool c_flag_case;
- int i = 0;
-
- pernet = pm_nl_get_pernet_from_msk(msk);
- subflows_max = mptcp_pm_get_subflows_max(msk);
- c_flag_case = remote->id && mptcp_pm_add_addr_c_flag_case(msk);
-
- mptcp_local_address((struct sock_common *)msk, &mpc_addr);
-
- rcu_read_lock();
- list_for_each_entry_rcu(entry, &pernet->local_addr_list, list) {
- if (!(entry->flags & MPTCP_PM_ADDR_FLAG_FULLMESH))
- continue;
-
- if (!mptcp_pm_addr_families_match(sk, &entry->addr, remote))
- continue;
-
- if (msk->pm.subflows < subflows_max) {
- bool is_id0;
-
- locals[i].addr = entry->addr;
- locals[i].flags = entry->flags;
- locals[i].ifindex = entry->ifindex;
-
- is_id0 = mptcp_addresses_equal(&locals[i].addr,
- &mpc_addr,
- locals[i].addr.port);
-
- if (c_flag_case &&
- (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW)) {
- __clear_bit(locals[i].addr.id,
- msk->pm.id_avail_bitmap);
-
- if (!is_id0)
- msk->pm.local_addr_used++;
- }
-
- /* Special case for ID0: set the correct ID */
- if (is_id0)
- locals[i].addr.id = 0;
-
- msk->pm.subflows++;
- i++;
- }
- }
- rcu_read_unlock();
-
- /* Special case: peer sets the C flag, accept one ADD_ADDR if default
- * limits are used -- accepting no ADD_ADDR -- and use subflow endpoints
- */
- if (!i && c_flag_case) {
- unsigned int local_addr_max = mptcp_pm_get_local_addr_max(msk);
-
- while (msk->pm.local_addr_used < local_addr_max &&
- msk->pm.subflows < subflows_max) {
- struct mptcp_pm_local *local = &locals[i];
-
- if (!select_local_address(pernet, msk, local))
- break;
-
- __clear_bit(local->addr.id, msk->pm.id_avail_bitmap);
-
- if (!mptcp_pm_addr_families_match(sk, &local->addr,
- remote))
- continue;
-
- if (mptcp_addresses_equal(&local->addr, &mpc_addr,
- local->addr.port))
- continue;
-
- msk->pm.local_addr_used++;
- msk->pm.subflows++;
- i++;
- }
-
- return i;
- }
-
- /* If the array is empty, fill in the single
- * 'IPADDRANY' local address
- */
- if (!i) {
- memset(&locals[i], 0, sizeof(locals[i]));
- locals[i].addr.family =
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- remote->family == AF_INET6 &&
- ipv6_addr_v4mapped(&remote->addr6) ? AF_INET :
-#endif
- remote->family;
-
- if (!mptcp_pm_addr_families_match(sk, &locals[i].addr, remote))
- return 0;
-
- msk->pm.subflows++;
- i++;
- }
-
- return i;
-}
-
-static void mptcp_pm_nl_add_addr_received(struct mptcp_sock *msk)
-{
- struct mptcp_pm_local locals[MPTCP_PM_ADDR_MAX];
- struct sock *sk = (struct sock *)msk;
- unsigned int add_addr_accept_max;
- struct mptcp_addr_info remote;
- unsigned int subflows_max;
- bool sf_created = false;
- int i, nr;
-
- add_addr_accept_max = mptcp_pm_get_add_addr_accept_max(msk);
- subflows_max = mptcp_pm_get_subflows_max(msk);
-
- pr_debug("accepted %d:%d remote family %d\n",
- msk->pm.add_addr_accepted, add_addr_accept_max,
- msk->pm.remote.family);
-
- remote = msk->pm.remote;
- mptcp_pm_announce_addr(msk, &remote, true);
- mptcp_pm_nl_addr_send_ack(msk);
-
- if (lookup_subflow_by_daddr(&msk->conn_list, &remote))
- return;
-
- /* pick id 0 port, if none is provided the remote address */
- if (!remote.port)
- remote.port = sk->sk_dport;
-
- /* connect to the specified remote address, using whatever
- * local address the routing configuration will pick.
- */
- nr = fill_local_addresses_vec(msk, &remote, locals);
- if (nr == 0)
- return;
-
- spin_unlock_bh(&msk->pm.lock);
- for (i = 0; i < nr; i++)
- if (__mptcp_subflow_connect(sk, &locals[i], &remote) == 0)
- sf_created = true;
- spin_lock_bh(&msk->pm.lock);
-
- if (sf_created) {
- /* add_addr_accepted is not decr for ID 0 */
- if (remote.id)
- msk->pm.add_addr_accepted++;
- if (msk->pm.add_addr_accepted >= add_addr_accept_max ||
- msk->pm.subflows >= subflows_max)
- WRITE_ONCE(msk->pm.accept_addr, false);
- }
-}
-
-bool mptcp_pm_nl_is_init_remote_addr(struct mptcp_sock *msk,
- const struct mptcp_addr_info *remote)
-{
- struct mptcp_addr_info mpc_remote;
-
- remote_address((struct sock_common *)msk, &mpc_remote);
- return mptcp_addresses_equal(&mpc_remote, remote, remote->port);
-}
-
-static bool subflow_in_rm_list(const struct mptcp_subflow_context *subflow,
- const struct mptcp_rm_list *rm_list)
-{
- u8 i, id = subflow_get_local_id(subflow);
-
- for (i = 0; i < rm_list->nr; i++) {
- if (rm_list->ids[i] == id)
- return true;
- }
-
- return false;
-}
-
-void mptcp_pm_nl_addr_send_ack_avoid_list(struct mptcp_sock *msk,
- const struct mptcp_rm_list *rm_list)
-{
- struct mptcp_subflow_context *subflow, *same_id = NULL;
-
- msk_owned_by_me(msk);
- lockdep_assert_held(&msk->pm.lock);
-
- if (!mptcp_pm_should_add_signal(msk) &&
- !mptcp_pm_should_rm_signal(msk))
- return;
-
- mptcp_for_each_subflow(msk, subflow) {
- if (!__mptcp_subflow_active(subflow))
- continue;
-
- if (unlikely(rm_list &&
- subflow_in_rm_list(subflow, rm_list))) {
- if (!same_id)
- same_id = subflow;
- } else {
- goto send_ack;
- }
- }
-
- if (same_id)
- subflow = same_id;
- else
- return;
-
-send_ack:
- mptcp_pm_send_ack(msk, subflow, false, false);
-}
-
-void mptcp_pm_nl_addr_send_ack(struct mptcp_sock *msk)
-{
- mptcp_pm_nl_addr_send_ack_avoid_list(msk, NULL);
-}
-
-int mptcp_pm_nl_mp_prio_send_ack(struct mptcp_sock *msk,
- struct mptcp_addr_info *addr,
- struct mptcp_addr_info *rem,
- u8 bkup)
-{
- struct mptcp_subflow_context *subflow;
-
- pr_debug("bkup=%d\n", bkup);
-
- mptcp_for_each_subflow(msk, subflow) {
- struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
- struct mptcp_addr_info local, remote;
-
- if (!__mptcp_subflow_active(subflow))
- continue;
-
- mptcp_local_address((struct sock_common *)ssk, &local);
- if (!mptcp_addresses_equal(&local, addr, addr->port))
- continue;
-
- if (rem && rem->family != AF_UNSPEC) {
- remote_address((struct sock_common *)ssk, &remote);
- if (!mptcp_addresses_equal(&remote, rem, rem->port))
- continue;
- }
-
- __mptcp_pm_send_ack(msk, subflow, true, bkup);
- return 0;
- }
-
- return -EINVAL;
-}
-
-static void mptcp_pm_nl_rm_addr_or_subflow(struct mptcp_sock *msk,
- const struct mptcp_rm_list *rm_list,
- enum linux_mptcp_mib_field rm_type)
-{
- struct mptcp_subflow_context *subflow, *tmp;
- struct sock *sk = (struct sock *)msk;
- u8 i;
-
- pr_debug("%s rm_list_nr %d\n",
- rm_type == MPTCP_MIB_RMADDR ? "address" : "subflow", rm_list->nr);
-
- msk_owned_by_me(msk);
-
- if (sk->sk_state == TCP_LISTEN)
- return;
-
- if (!rm_list->nr)
- return;
-
- if (list_empty(&msk->conn_list))
- return;
-
- for (i = 0; i < rm_list->nr; i++) {
- u8 rm_id = rm_list->ids[i];
- bool removed = false;
-
- mptcp_for_each_subflow_safe(msk, subflow, tmp) {
- struct sock *ssk = mptcp_subflow_tcp_sock(subflow);
- u8 remote_id = READ_ONCE(subflow->remote_id);
- int how = RCV_SHUTDOWN | SEND_SHUTDOWN;
- u8 id = subflow_get_local_id(subflow);
-
- if ((1 << inet_sk_state_load(ssk)) &
- (TCPF_FIN_WAIT1 | TCPF_FIN_WAIT2 | TCPF_CLOSING | TCPF_CLOSE))
- continue;
- if (rm_type == MPTCP_MIB_RMADDR && remote_id != rm_id)
- continue;
- if (rm_type == MPTCP_MIB_RMSUBFLOW && id != rm_id)
- continue;
-
- pr_debug(" -> %s rm_list_ids[%d]=%u local_id=%u remote_id=%u mpc_id=%u\n",
- rm_type == MPTCP_MIB_RMADDR ? "address" : "subflow",
- i, rm_id, id, remote_id, msk->mpc_endpoint_id);
- spin_unlock_bh(&msk->pm.lock);
- mptcp_subflow_shutdown(sk, ssk, how);
- removed |= subflow->request_join;
-
- /* the following takes care of updating the subflows counter */
- mptcp_close_ssk(sk, ssk, subflow);
- spin_lock_bh(&msk->pm.lock);
-
- if (rm_type == MPTCP_MIB_RMSUBFLOW)
- __MPTCP_INC_STATS(sock_net(sk), rm_type);
- }
-
- if (rm_type == MPTCP_MIB_RMADDR)
- __MPTCP_INC_STATS(sock_net(sk), rm_type);
-
- if (!removed)
- continue;
-
- if (!mptcp_pm_is_kernel(msk))
- continue;
-
- if (rm_type == MPTCP_MIB_RMADDR && rm_id &&
- !WARN_ON_ONCE(msk->pm.add_addr_accepted == 0)) {
- /* Note: if the subflow has been closed before, this
- * add_addr_accepted counter will not be decremented.
- */
- if (--msk->pm.add_addr_accepted < mptcp_pm_get_add_addr_accept_max(msk))
- WRITE_ONCE(msk->pm.accept_addr, true);
- }
- }
-}
-
-static void mptcp_pm_nl_rm_addr_received(struct mptcp_sock *msk)
-{
- mptcp_pm_nl_rm_addr_or_subflow(msk, &msk->pm.rm_list_rx, MPTCP_MIB_RMADDR);
-}
-
-static void mptcp_pm_nl_rm_subflow_received(struct mptcp_sock *msk,
- const struct mptcp_rm_list *rm_list)
-{
- mptcp_pm_nl_rm_addr_or_subflow(msk, rm_list, MPTCP_MIB_RMSUBFLOW);
-}
-
-void mptcp_pm_nl_work(struct mptcp_sock *msk)
-{
- struct mptcp_pm_data *pm = &msk->pm;
-
- msk_owned_by_me(msk);
-
- if (!(pm->status & MPTCP_PM_WORK_MASK))
- return;
-
- spin_lock_bh(&msk->pm.lock);
-
- pr_debug("msk=%p status=%x\n", msk, pm->status);
- if (pm->status & BIT(MPTCP_PM_ADD_ADDR_RECEIVED)) {
- pm->status &= ~BIT(MPTCP_PM_ADD_ADDR_RECEIVED);
- mptcp_pm_nl_add_addr_received(msk);
- }
- if (pm->status & BIT(MPTCP_PM_ADD_ADDR_SEND_ACK)) {
- pm->status &= ~BIT(MPTCP_PM_ADD_ADDR_SEND_ACK);
- mptcp_pm_nl_addr_send_ack(msk);
- }
- if (pm->status & BIT(MPTCP_PM_RM_ADDR_RECEIVED)) {
- pm->status &= ~BIT(MPTCP_PM_RM_ADDR_RECEIVED);
- mptcp_pm_nl_rm_addr_received(msk);
- }
- if (pm->status & BIT(MPTCP_PM_ESTABLISHED)) {
- pm->status &= ~BIT(MPTCP_PM_ESTABLISHED);
- mptcp_pm_nl_fully_established(msk);
- }
- if (pm->status & BIT(MPTCP_PM_SUBFLOW_ESTABLISHED)) {
- pm->status &= ~BIT(MPTCP_PM_SUBFLOW_ESTABLISHED);
- mptcp_pm_nl_subflow_established(msk);
- }
-
- spin_unlock_bh(&msk->pm.lock);
-}
-
-static bool address_use_port(struct mptcp_pm_addr_entry *entry)
-{
- return (entry->flags &
- (MPTCP_PM_ADDR_FLAG_SIGNAL | MPTCP_PM_ADDR_FLAG_SUBFLOW)) ==
- MPTCP_PM_ADDR_FLAG_SIGNAL;
-}
-
-/* caller must ensure the RCU grace period is already elapsed */
-static void __mptcp_pm_release_addr_entry(struct mptcp_pm_addr_entry *entry)
-{
- if (entry->lsk)
- sock_release(entry->lsk);
- kfree(entry);
-}
-
-static int mptcp_pm_nl_append_new_local_addr(struct pm_nl_pernet *pernet,
- struct mptcp_pm_addr_entry *entry,
- bool replace)
-{
- struct mptcp_pm_addr_entry *cur, *del_entry = NULL;
- unsigned int addr_max;
- int ret = -EINVAL;
-
- spin_lock_bh(&pernet->lock);
- /* to keep the code simple, don't do IDR-like allocation for address ID,
- * just bail when we exceed limits
- */
- if (pernet->next_id == MPTCP_PM_MAX_ADDR_ID)
- pernet->next_id = 1;
- if (pernet->addrs >= MPTCP_PM_ADDR_MAX) {
- ret = -ERANGE;
- goto out;
- }
- if (test_bit(entry->addr.id, pernet->id_bitmap)) {
- ret = -EBUSY;
- goto out;
- }
-
- /* do not insert duplicate address, differentiate on port only
- * singled addresses
- */
- if (!address_use_port(entry))
- entry->addr.port = 0;
- list_for_each_entry(cur, &pernet->local_addr_list, list) {
- if (mptcp_addresses_equal(&cur->addr, &entry->addr,
- cur->addr.port || entry->addr.port)) {
- /* allow replacing the exiting endpoint only if such
- * endpoint is an implicit one and the user-space
- * did not provide an endpoint id
- */
- if (!(cur->flags & MPTCP_PM_ADDR_FLAG_IMPLICIT)) {
- ret = -EEXIST;
- goto out;
- }
- if (entry->addr.id)
- goto out;
-
- /* allow callers that only need to look up the local
- * addr's id to skip replacement. This allows them to
- * avoid calling synchronize_rcu in the packet recv
- * path.
- */
- if (!replace) {
- kfree(entry);
- ret = cur->addr.id;
- goto out;
- }
-
- pernet->addrs--;
- entry->addr.id = cur->addr.id;
- list_del_rcu(&cur->list);
- del_entry = cur;
- break;
- }
- }
-
- if (!entry->addr.id) {
-find_next:
- entry->addr.id = find_next_zero_bit(pernet->id_bitmap,
- MPTCP_PM_MAX_ADDR_ID + 1,
- pernet->next_id);
- if (!entry->addr.id && pernet->next_id != 1) {
- pernet->next_id = 1;
- goto find_next;
- }
- }
-
- if (!entry->addr.id)
- goto out;
-
- __set_bit(entry->addr.id, pernet->id_bitmap);
- if (entry->addr.id > pernet->next_id)
- pernet->next_id = entry->addr.id;
-
- if (entry->flags & MPTCP_PM_ADDR_FLAG_SIGNAL) {
- addr_max = pernet->add_addr_signal_max;
- WRITE_ONCE(pernet->add_addr_signal_max, addr_max + 1);
- }
- if (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW) {
- addr_max = pernet->local_addr_max;
- WRITE_ONCE(pernet->local_addr_max, addr_max + 1);
- }
-
- pernet->addrs++;
- if (!entry->addr.port)
- list_add_tail_rcu(&entry->list, &pernet->local_addr_list);
- else
- list_add_rcu(&entry->list, &pernet->local_addr_list);
- ret = entry->addr.id;
-
-out:
- spin_unlock_bh(&pernet->lock);
-
- /* just replaced an existing entry, free it */
- if (del_entry) {
- synchronize_rcu();
- __mptcp_pm_release_addr_entry(del_entry);
- }
- return ret;
-}
-
-static struct lock_class_key mptcp_slock_keys[2];
-static struct lock_class_key mptcp_keys[2];
-
-static int mptcp_pm_nl_create_listen_socket(struct sock *sk,
- struct mptcp_pm_addr_entry *entry)
-{
- bool is_ipv6 = entry->addr.family == AF_INET6;
- int addrlen = sizeof(struct sockaddr_in);
- struct sockaddr_storage addr;
- struct sock *newsk, *ssk;
- int backlog = 1024;
- int err;
-
- err = sock_create_kern(sock_net(sk), entry->addr.family,
- SOCK_STREAM, IPPROTO_MPTCP, &entry->lsk);
- if (err)
- return err;
-
- newsk = entry->lsk->sk;
- if (!newsk)
- return -EINVAL;
-
- /* The subflow socket lock is acquired in a nested to the msk one
- * in several places, even by the TCP stack, and this msk is a kernel
- * socket: lockdep complains. Instead of propagating the _nested
- * modifiers in several places, re-init the lock class for the msk
- * socket to an mptcp specific one.
- */
- sock_lock_init_class_and_name(newsk,
- is_ipv6 ? "mlock-AF_INET6" : "mlock-AF_INET",
- &mptcp_slock_keys[is_ipv6],
- is_ipv6 ? "msk_lock-AF_INET6" : "msk_lock-AF_INET",
- &mptcp_keys[is_ipv6]);
-
- lock_sock(newsk);
- ssk = __mptcp_nmpc_sk(mptcp_sk(newsk));
- release_sock(newsk);
- if (IS_ERR(ssk))
- return PTR_ERR(ssk);
-
- mptcp_info2sockaddr(&entry->addr, &addr, entry->addr.family);
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- if (entry->addr.family == AF_INET6)
- addrlen = sizeof(struct sockaddr_in6);
-#endif
- if (ssk->sk_family == AF_INET)
- err = inet_bind_sk(ssk, (struct sockaddr *)&addr, addrlen);
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- else if (ssk->sk_family == AF_INET6)
- err = inet6_bind_sk(ssk, (struct sockaddr *)&addr, addrlen);
-#endif
- if (err)
- return err;
-
- /* We don't use mptcp_set_state() here because it needs to be called
- * under the msk socket lock. For the moment, that will not bring
- * anything more than only calling inet_sk_state_store(), because the
- * old status is known (TCP_CLOSE).
- */
- inet_sk_state_store(newsk, TCP_LISTEN);
- lock_sock(ssk);
- WRITE_ONCE(mptcp_subflow_ctx(ssk)->pm_listener, true);
- err = __inet_listen_sk(ssk, backlog);
- if (!err)
- mptcp_event_pm_listener(ssk, MPTCP_EVENT_LISTENER_CREATED);
- release_sock(ssk);
- return err;
-}
-
-int mptcp_pm_nl_get_local_id(struct mptcp_sock *msk,
- struct mptcp_pm_addr_entry *skc)
-{
- struct mptcp_pm_addr_entry *entry;
- struct pm_nl_pernet *pernet;
- int ret;
-
- pernet = pm_nl_get_pernet_from_msk(msk);
-
- rcu_read_lock();
- entry = __lookup_addr(pernet, &skc->addr);
- ret = entry ? entry->addr.id : -1;
- rcu_read_unlock();
- if (ret >= 0)
- return ret;
-
- /* address not found, add to local list */
- entry = kmalloc(sizeof(*entry), GFP_ATOMIC);
- if (!entry)
- return -ENOMEM;
-
- *entry = *skc;
- entry->addr.port = 0;
- ret = mptcp_pm_nl_append_new_local_addr(pernet, entry, false);
- if (ret < 0)
- kfree(entry);
-
- return ret;
-}
-
-bool mptcp_pm_nl_is_backup(struct mptcp_sock *msk, struct mptcp_addr_info *skc)
-{
- struct pm_nl_pernet *pernet = pm_nl_get_pernet_from_msk(msk);
- struct mptcp_pm_addr_entry *entry;
- bool backup;
-
- rcu_read_lock();
- entry = __lookup_addr(pernet, skc);
- backup = entry && !!(entry->flags & MPTCP_PM_ADDR_FLAG_BACKUP);
- rcu_read_unlock();
-
- return backup;
-}
-
-#define MPTCP_PM_CMD_GRP_OFFSET 0
-#define MPTCP_PM_EV_GRP_OFFSET 1
-
-static const struct genl_multicast_group mptcp_pm_mcgrps[] = {
- [MPTCP_PM_CMD_GRP_OFFSET] = { .name = MPTCP_PM_CMD_GRP_NAME, },
- [MPTCP_PM_EV_GRP_OFFSET] = { .name = MPTCP_PM_EV_GRP_NAME,
- .flags = GENL_MCAST_CAP_NET_ADMIN,
- },
-};
-
-void mptcp_pm_nl_subflow_chk_stale(const struct mptcp_sock *msk, struct sock *ssk)
-{
- struct mptcp_subflow_context *iter, *subflow = mptcp_subflow_ctx(ssk);
- struct sock *sk = (struct sock *)msk;
- unsigned int active_max_loss_cnt;
- struct net *net = sock_net(sk);
- unsigned int stale_loss_cnt;
- bool slow;
-
- stale_loss_cnt = mptcp_stale_loss_cnt(net);
- if (subflow->stale || !stale_loss_cnt || subflow->stale_count <= stale_loss_cnt)
- return;
-
- /* look for another available subflow not in loss state */
- active_max_loss_cnt = max_t(int, stale_loss_cnt - 1, 1);
- mptcp_for_each_subflow(msk, iter) {
- if (iter != subflow && mptcp_subflow_active(iter) &&
- iter->stale_count < active_max_loss_cnt) {
- /* we have some alternatives, try to mark this subflow as idle ...*/
- slow = lock_sock_fast(ssk);
- if (!tcp_rtx_and_write_queues_empty(ssk)) {
- subflow->stale = 1;
- __mptcp_retransmit_pending_data(sk);
- MPTCP_INC_STATS(net, MPTCP_MIB_SUBFLOWSTALE);
- }
- unlock_sock_fast(ssk, slow);
-
- /* always try to push the pending data regardless of re-injections:
- * we can possibly use backup subflows now, and subflow selection
- * is cheap under the msk socket lock
- */
- __mptcp_push_pending(sk, 0);
- return;
- }
- }
-}
-
-static int mptcp_pm_family_to_addr(int family)
-{
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- if (family == AF_INET6)
- return MPTCP_PM_ADDR_ATTR_ADDR6;
-#endif
- return MPTCP_PM_ADDR_ATTR_ADDR4;
-}
-
-static int mptcp_pm_parse_pm_addr_attr(struct nlattr *tb[],
- const struct nlattr *attr,
- struct genl_info *info,
- struct mptcp_addr_info *addr,
- bool require_family)
-{
- int err, addr_addr;
-
- if (!attr) {
- GENL_SET_ERR_MSG(info, "missing address info");
- return -EINVAL;
- }
-
- /* no validation needed - was already done via nested policy */
- err = nla_parse_nested_deprecated(tb, MPTCP_PM_ADDR_ATTR_MAX, attr,
- mptcp_pm_address_nl_policy, info->extack);
- if (err)
- return err;
-
- if (tb[MPTCP_PM_ADDR_ATTR_ID])
- addr->id = nla_get_u8(tb[MPTCP_PM_ADDR_ATTR_ID]);
-
- if (!tb[MPTCP_PM_ADDR_ATTR_FAMILY]) {
- if (!require_family)
- return 0;
-
- NL_SET_ERR_MSG_ATTR(info->extack, attr,
- "missing family");
- return -EINVAL;
- }
-
- addr->family = nla_get_u16(tb[MPTCP_PM_ADDR_ATTR_FAMILY]);
- if (addr->family != AF_INET
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- && addr->family != AF_INET6
-#endif
- ) {
- NL_SET_ERR_MSG_ATTR(info->extack, attr,
- "unknown address family");
- return -EINVAL;
- }
- addr_addr = mptcp_pm_family_to_addr(addr->family);
- if (!tb[addr_addr]) {
- NL_SET_ERR_MSG_ATTR(info->extack, attr,
- "missing address data");
- return -EINVAL;
- }
-
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- if (addr->family == AF_INET6)
- addr->addr6 = nla_get_in6_addr(tb[addr_addr]);
- else
-#endif
- addr->addr.s_addr = nla_get_in_addr(tb[addr_addr]);
-
- if (tb[MPTCP_PM_ADDR_ATTR_PORT])
- addr->port = htons(nla_get_u16(tb[MPTCP_PM_ADDR_ATTR_PORT]));
-
- return 0;
-}
-
-int mptcp_pm_parse_addr(struct nlattr *attr, struct genl_info *info,
- struct mptcp_addr_info *addr)
-{
- struct nlattr *tb[MPTCP_PM_ADDR_ATTR_MAX + 1];
-
- memset(addr, 0, sizeof(*addr));
-
- return mptcp_pm_parse_pm_addr_attr(tb, attr, info, addr, true);
-}
-
-int mptcp_pm_parse_entry(struct nlattr *attr, struct genl_info *info,
- bool require_family,
- struct mptcp_pm_addr_entry *entry)
-{
- struct nlattr *tb[MPTCP_PM_ADDR_ATTR_MAX + 1];
- int err;
-
- memset(entry, 0, sizeof(*entry));
-
- err = mptcp_pm_parse_pm_addr_attr(tb, attr, info, &entry->addr, require_family);
- if (err)
- return err;
-
- if (tb[MPTCP_PM_ADDR_ATTR_IF_IDX]) {
- u32 val = nla_get_s32(tb[MPTCP_PM_ADDR_ATTR_IF_IDX]);
-
- entry->ifindex = val;
- }
-
- if (tb[MPTCP_PM_ADDR_ATTR_FLAGS])
- entry->flags = nla_get_u32(tb[MPTCP_PM_ADDR_ATTR_FLAGS]) &
- MPTCP_PM_ADDR_FLAGS_MASK;
-
- if (tb[MPTCP_PM_ADDR_ATTR_PORT])
- entry->addr.port = htons(nla_get_u16(tb[MPTCP_PM_ADDR_ATTR_PORT]));
-
- return 0;
-}
-
-static struct pm_nl_pernet *genl_info_pm_nl(struct genl_info *info)
-{
- return pm_nl_get_pernet(genl_info_net(info));
-}
-
-static int mptcp_nl_add_subflow_or_signal_addr(struct net *net,
- struct mptcp_addr_info *addr)
-{
- struct mptcp_sock *msk;
- long s_slot = 0, s_num = 0;
-
- while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
- struct sock *sk = (struct sock *)msk;
- struct mptcp_addr_info mpc_addr;
-
- if (!READ_ONCE(msk->fully_established) ||
- mptcp_pm_is_userspace(msk))
- goto next;
-
- /* if the endp linked to the init sf is re-added with a != ID */
- mptcp_local_address((struct sock_common *)msk, &mpc_addr);
-
- lock_sock(sk);
- spin_lock_bh(&msk->pm.lock);
- if (mptcp_addresses_equal(addr, &mpc_addr, addr->port))
- msk->mpc_endpoint_id = addr->id;
- mptcp_pm_create_subflow_or_signal_addr(msk);
- spin_unlock_bh(&msk->pm.lock);
- release_sock(sk);
-
-next:
- sock_put(sk);
- cond_resched();
- }
-
- return 0;
-}
-
-int mptcp_pm_nl_add_addr_doit(struct sk_buff *skb, struct genl_info *info)
-{
- struct nlattr *attr = info->attrs[MPTCP_PM_ENDPOINT_ADDR];
- struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
- struct mptcp_pm_addr_entry addr, *entry;
- int ret;
-
- ret = mptcp_pm_parse_entry(attr, info, true, &addr);
- if (ret < 0)
- return ret;
-
- if (addr.addr.port && !address_use_port(&addr)) {
- GENL_SET_ERR_MSG(info, "flags must have signal and not subflow when using port");
- return -EINVAL;
- }
-
- if (addr.flags & MPTCP_PM_ADDR_FLAG_SIGNAL &&
- addr.flags & MPTCP_PM_ADDR_FLAG_FULLMESH) {
- GENL_SET_ERR_MSG(info, "flags mustn't have both signal and fullmesh");
- return -EINVAL;
- }
-
- if (addr.flags & MPTCP_PM_ADDR_FLAG_IMPLICIT) {
- GENL_SET_ERR_MSG(info, "can't create IMPLICIT endpoint");
- return -EINVAL;
- }
-
- entry = kzalloc(sizeof(*entry), GFP_KERNEL_ACCOUNT);
- if (!entry) {
- GENL_SET_ERR_MSG(info, "can't allocate addr");
- return -ENOMEM;
- }
-
- *entry = addr;
- if (entry->addr.port) {
- ret = mptcp_pm_nl_create_listen_socket(skb->sk, entry);
- if (ret) {
- GENL_SET_ERR_MSG_FMT(info, "create listen socket error: %d", ret);
- goto out_free;
- }
- }
- ret = mptcp_pm_nl_append_new_local_addr(pernet, entry, true);
- if (ret < 0) {
- GENL_SET_ERR_MSG_FMT(info, "too many addresses or duplicate one: %d", ret);
- goto out_free;
- }
-
- mptcp_nl_add_subflow_or_signal_addr(sock_net(skb->sk), &entry->addr);
- return 0;
-
-out_free:
- __mptcp_pm_release_addr_entry(entry);
- return ret;
-}
-
-static bool remove_anno_list_by_saddr(struct mptcp_sock *msk,
- const struct mptcp_addr_info *addr)
-{
- struct mptcp_pm_add_entry *entry;
-
- entry = mptcp_pm_del_add_timer(msk, addr, false);
- if (entry) {
- kfree_rcu(entry, rcu);
- return true;
- }
-
- return false;
-}
-
-static u8 mptcp_endp_get_local_id(struct mptcp_sock *msk,
- const struct mptcp_addr_info *addr)
-{
- return msk->mpc_endpoint_id == addr->id ? 0 : addr->id;
-}
-
-static bool mptcp_pm_remove_anno_addr(struct mptcp_sock *msk,
- const struct mptcp_addr_info *addr,
- bool force)
-{
- struct mptcp_rm_list list = { .nr = 0 };
- bool ret;
-
- list.ids[list.nr++] = mptcp_endp_get_local_id(msk, addr);
-
- ret = remove_anno_list_by_saddr(msk, addr);
- if (ret || force) {
- spin_lock_bh(&msk->pm.lock);
- if (ret)
- msk->pm.add_addr_signaled--;
- mptcp_pm_remove_addr(msk, &list);
- spin_unlock_bh(&msk->pm.lock);
- }
- return ret;
-}
-
-static void __mark_subflow_endp_available(struct mptcp_sock *msk, u8 id)
-{
- /* If it was marked as used, and not ID 0, decrement local_addr_used */
- if (!__test_and_set_bit(id ? : msk->mpc_endpoint_id, msk->pm.id_avail_bitmap) &&
- id && !WARN_ON_ONCE(msk->pm.local_addr_used == 0))
- msk->pm.local_addr_used--;
-}
-
-static int mptcp_nl_remove_subflow_and_signal_addr(struct net *net,
- const struct mptcp_pm_addr_entry *entry)
-{
- const struct mptcp_addr_info *addr = &entry->addr;
- struct mptcp_rm_list list = { .nr = 1 };
- long s_slot = 0, s_num = 0;
- struct mptcp_sock *msk;
-
- pr_debug("remove_id=%d\n", addr->id);
-
- while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
- struct sock *sk = (struct sock *)msk;
- bool remove_subflow;
-
- if (mptcp_pm_is_userspace(msk))
- goto next;
-
- lock_sock(sk);
- remove_subflow = lookup_subflow_by_saddr(&msk->conn_list, addr);
- mptcp_pm_remove_anno_addr(msk, addr, remove_subflow &&
- !(entry->flags & MPTCP_PM_ADDR_FLAG_IMPLICIT));
-
- list.ids[0] = mptcp_endp_get_local_id(msk, addr);
-
- spin_lock_bh(&msk->pm.lock);
- if (remove_subflow)
- mptcp_pm_nl_rm_subflow_received(msk, &list);
- if (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW)
- __mark_subflow_endp_available(msk, list.ids[0]);
- else /* mark endp ID as available, e.g. Signal or MPC endp */
- __set_bit(addr->id, msk->pm.id_avail_bitmap);
- spin_unlock_bh(&msk->pm.lock);
-
- if (msk->mpc_endpoint_id == entry->addr.id)
- msk->mpc_endpoint_id = 0;
- release_sock(sk);
-
-next:
- sock_put(sk);
- cond_resched();
- }
-
- return 0;
-}
-
-static int mptcp_nl_remove_id_zero_address(struct net *net,
- struct mptcp_addr_info *addr)
-{
- struct mptcp_rm_list list = { .nr = 0 };
- long s_slot = 0, s_num = 0;
- struct mptcp_sock *msk;
-
- list.ids[list.nr++] = 0;
-
- while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
- struct sock *sk = (struct sock *)msk;
- struct mptcp_addr_info msk_local;
-
- if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
- goto next;
-
- mptcp_local_address((struct sock_common *)msk, &msk_local);
- if (!mptcp_addresses_equal(&msk_local, addr, addr->port))
- goto next;
-
- lock_sock(sk);
- spin_lock_bh(&msk->pm.lock);
- mptcp_pm_remove_addr(msk, &list);
- mptcp_pm_nl_rm_subflow_received(msk, &list);
- __mark_subflow_endp_available(msk, 0);
- spin_unlock_bh(&msk->pm.lock);
- release_sock(sk);
-
-next:
- sock_put(sk);
- cond_resched();
- }
-
- return 0;
-}
-
-int mptcp_pm_nl_del_addr_doit(struct sk_buff *skb, struct genl_info *info)
-{
- struct nlattr *attr = info->attrs[MPTCP_PM_ENDPOINT_ADDR];
- struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
- struct mptcp_pm_addr_entry addr, *entry;
- unsigned int addr_max;
- int ret;
-
- ret = mptcp_pm_parse_entry(attr, info, false, &addr);
- if (ret < 0)
- return ret;
-
- /* the zero id address is special: the first address used by the msk
- * always gets such an id, so different subflows can have different zero
- * id addresses. Additionally zero id is not accounted for in id_bitmap.
- * Let's use an 'mptcp_rm_list' instead of the common remove code.
- */
- if (addr.addr.id == 0)
- return mptcp_nl_remove_id_zero_address(sock_net(skb->sk), &addr.addr);
-
- spin_lock_bh(&pernet->lock);
- entry = __lookup_addr_by_id(pernet, addr.addr.id);
- if (!entry) {
- GENL_SET_ERR_MSG(info, "address not found");
- spin_unlock_bh(&pernet->lock);
- return -EINVAL;
- }
- if (entry->flags & MPTCP_PM_ADDR_FLAG_SIGNAL) {
- addr_max = pernet->add_addr_signal_max;
- WRITE_ONCE(pernet->add_addr_signal_max, addr_max - 1);
- }
- if (entry->flags & MPTCP_PM_ADDR_FLAG_SUBFLOW) {
- addr_max = pernet->local_addr_max;
- WRITE_ONCE(pernet->local_addr_max, addr_max - 1);
- }
-
- pernet->addrs--;
- list_del_rcu(&entry->list);
- __clear_bit(entry->addr.id, pernet->id_bitmap);
- spin_unlock_bh(&pernet->lock);
-
- mptcp_nl_remove_subflow_and_signal_addr(sock_net(skb->sk), entry);
- synchronize_rcu();
- __mptcp_pm_release_addr_entry(entry);
-
- return ret;
-}
-
-/* Called from the userspace PM only */
-void mptcp_pm_remove_addrs(struct mptcp_sock *msk, struct list_head *rm_list)
-{
- struct mptcp_rm_list alist = { .nr = 0 };
- struct mptcp_pm_addr_entry *entry;
- int anno_nr = 0;
-
- list_for_each_entry(entry, rm_list, list) {
- if (alist.nr >= MPTCP_RM_IDS_MAX)
- break;
-
- /* only delete if either announced or matching a subflow */
- if (remove_anno_list_by_saddr(msk, &entry->addr))
- anno_nr++;
- else if (!lookup_subflow_by_saddr(&msk->conn_list,
- &entry->addr))
- continue;
-
- alist.ids[alist.nr++] = entry->addr.id;
- }
-
- if (alist.nr) {
- spin_lock_bh(&msk->pm.lock);
- msk->pm.add_addr_signaled -= anno_nr;
- mptcp_pm_remove_addr(msk, &alist);
- spin_unlock_bh(&msk->pm.lock);
- }
-}
-
-/* Called from the in-kernel PM only */
-static void mptcp_pm_flush_addrs_and_subflows(struct mptcp_sock *msk,
- struct list_head *rm_list)
-{
- struct mptcp_rm_list alist = { .nr = 0 }, slist = { .nr = 0 };
- struct mptcp_pm_addr_entry *entry;
-
- list_for_each_entry(entry, rm_list, list) {
- if (slist.nr < MPTCP_RM_IDS_MAX &&
- lookup_subflow_by_saddr(&msk->conn_list, &entry->addr))
- slist.ids[slist.nr++] = mptcp_endp_get_local_id(msk, &entry->addr);
-
- if (alist.nr < MPTCP_RM_IDS_MAX &&
- remove_anno_list_by_saddr(msk, &entry->addr))
- alist.ids[alist.nr++] = mptcp_endp_get_local_id(msk, &entry->addr);
- }
-
- spin_lock_bh(&msk->pm.lock);
- if (alist.nr) {
- msk->pm.add_addr_signaled -= alist.nr;
- mptcp_pm_remove_addr(msk, &alist);
- }
- if (slist.nr)
- mptcp_pm_nl_rm_subflow_received(msk, &slist);
- /* Reset counters: maybe some subflows have been removed before */
- bitmap_fill(msk->pm.id_avail_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
- msk->pm.local_addr_used = 0;
- spin_unlock_bh(&msk->pm.lock);
-}
-
-static void mptcp_nl_flush_addrs_list(struct net *net,
- struct list_head *rm_list)
-{
- long s_slot = 0, s_num = 0;
- struct mptcp_sock *msk;
-
- if (list_empty(rm_list))
- return;
-
- while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
- struct sock *sk = (struct sock *)msk;
-
- if (!mptcp_pm_is_userspace(msk)) {
- lock_sock(sk);
- mptcp_pm_flush_addrs_and_subflows(msk, rm_list);
- release_sock(sk);
- }
-
- sock_put(sk);
- cond_resched();
- }
-}
-
-/* caller must ensure the RCU grace period is already elapsed */
-static void __flush_addrs(struct list_head *list)
-{
- while (!list_empty(list)) {
- struct mptcp_pm_addr_entry *cur;
-
- cur = list_entry(list->next,
- struct mptcp_pm_addr_entry, list);
- list_del_rcu(&cur->list);
- __mptcp_pm_release_addr_entry(cur);
- }
-}
-
-static void __reset_counters(struct pm_nl_pernet *pernet)
-{
- WRITE_ONCE(pernet->add_addr_signal_max, 0);
- WRITE_ONCE(pernet->local_addr_max, 0);
- pernet->addrs = 0;
-}
-
-int mptcp_pm_nl_flush_addrs_doit(struct sk_buff *skb, struct genl_info *info)
-{
- struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
- struct list_head free_list;
-
- spin_lock_bh(&pernet->lock);
- free_list = pernet->local_addr_list;
- INIT_LIST_HEAD_RCU(&pernet->local_addr_list);
- __reset_counters(pernet);
- pernet->next_id = 1;
- bitmap_zero(pernet->id_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
- spin_unlock_bh(&pernet->lock);
-
- if (free_list.next == &pernet->local_addr_list)
- return 0;
-
- synchronize_rcu();
-
- /* Adjust the pointers to free_list instead of pernet->local_addr_list */
- free_list.prev->next = &free_list;
- free_list.next->prev = &free_list;
-
- mptcp_nl_flush_addrs_list(sock_net(skb->sk), &free_list);
- __flush_addrs(&free_list);
- return 0;
-}
-
-int mptcp_nl_fill_addr(struct sk_buff *skb,
- struct mptcp_pm_addr_entry *entry)
-{
- struct mptcp_addr_info *addr = &entry->addr;
- struct nlattr *attr;
-
- attr = nla_nest_start(skb, MPTCP_PM_ATTR_ADDR);
- if (!attr)
- return -EMSGSIZE;
-
- if (nla_put_u16(skb, MPTCP_PM_ADDR_ATTR_FAMILY, addr->family))
- goto nla_put_failure;
- if (nla_put_u16(skb, MPTCP_PM_ADDR_ATTR_PORT, ntohs(addr->port)))
- goto nla_put_failure;
- if (nla_put_u8(skb, MPTCP_PM_ADDR_ATTR_ID, addr->id))
- goto nla_put_failure;
- if (nla_put_u32(skb, MPTCP_PM_ADDR_ATTR_FLAGS, entry->flags))
- goto nla_put_failure;
- if (entry->ifindex &&
- nla_put_s32(skb, MPTCP_PM_ADDR_ATTR_IF_IDX, entry->ifindex))
- goto nla_put_failure;
-
- if (addr->family == AF_INET &&
- nla_put_in_addr(skb, MPTCP_PM_ADDR_ATTR_ADDR4,
- addr->addr.s_addr))
- goto nla_put_failure;
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- else if (addr->family == AF_INET6 &&
- nla_put_in6_addr(skb, MPTCP_PM_ADDR_ATTR_ADDR6, &addr->addr6))
- goto nla_put_failure;
-#endif
- nla_nest_end(skb, attr);
- return 0;
-
-nla_put_failure:
- nla_nest_cancel(skb, attr);
- return -EMSGSIZE;
-}
-
-int mptcp_pm_nl_get_addr(struct sk_buff *skb, struct genl_info *info)
-{
- struct nlattr *attr = info->attrs[MPTCP_PM_ENDPOINT_ADDR];
- struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
- struct mptcp_pm_addr_entry addr, *entry;
- struct sk_buff *msg;
- void *reply;
- int ret;
-
- ret = mptcp_pm_parse_entry(attr, info, false, &addr);
- if (ret < 0)
- return ret;
-
- msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
- if (!msg)
- return -ENOMEM;
-
- reply = genlmsg_put_reply(msg, info, &mptcp_genl_family, 0,
- info->genlhdr->cmd);
- if (!reply) {
- GENL_SET_ERR_MSG(info, "not enough space in Netlink message");
- ret = -EMSGSIZE;
- goto fail;
- }
-
- spin_lock_bh(&pernet->lock);
- entry = __lookup_addr_by_id(pernet, addr.addr.id);
- if (!entry) {
- GENL_SET_ERR_MSG(info, "address not found");
- ret = -EINVAL;
- goto unlock_fail;
- }
-
- ret = mptcp_nl_fill_addr(msg, entry);
- if (ret)
- goto unlock_fail;
-
- genlmsg_end(msg, reply);
- ret = genlmsg_reply(msg, info);
- spin_unlock_bh(&pernet->lock);
- return ret;
-
-unlock_fail:
- spin_unlock_bh(&pernet->lock);
-
-fail:
- nlmsg_free(msg);
- return ret;
-}
-
-int mptcp_pm_nl_get_addr_doit(struct sk_buff *skb, struct genl_info *info)
-{
- return mptcp_pm_get_addr(skb, info);
-}
-
-int mptcp_pm_nl_dump_addr(struct sk_buff *msg,
- struct netlink_callback *cb)
-{
- struct net *net = sock_net(msg->sk);
- struct mptcp_pm_addr_entry *entry;
- struct pm_nl_pernet *pernet;
- int id = cb->args[0];
- void *hdr;
- int i;
-
- pernet = pm_nl_get_pernet(net);
-
- spin_lock_bh(&pernet->lock);
- for (i = id; i < MPTCP_PM_MAX_ADDR_ID + 1; i++) {
- if (test_bit(i, pernet->id_bitmap)) {
- entry = __lookup_addr_by_id(pernet, i);
- if (!entry)
- break;
-
- if (entry->addr.id <= id)
- continue;
-
- hdr = genlmsg_put(msg, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq, &mptcp_genl_family,
- NLM_F_MULTI, MPTCP_PM_CMD_GET_ADDR);
- if (!hdr)
- break;
-
- if (mptcp_nl_fill_addr(msg, entry) < 0) {
- genlmsg_cancel(msg, hdr);
- break;
- }
-
- id = entry->addr.id;
- genlmsg_end(msg, hdr);
- }
- }
- spin_unlock_bh(&pernet->lock);
-
- cb->args[0] = id;
- return msg->len;
-}
-
-int mptcp_pm_nl_get_addr_dumpit(struct sk_buff *msg,
- struct netlink_callback *cb)
-{
- return mptcp_pm_dump_addr(msg, cb);
-}
-
-static int parse_limit(struct genl_info *info, int id, unsigned int *limit)
-{
- struct nlattr *attr = info->attrs[id];
-
- if (!attr)
- return 0;
-
- *limit = nla_get_u32(attr);
- if (*limit > MPTCP_PM_ADDR_MAX) {
- GENL_SET_ERR_MSG(info, "limit greater than maximum");
- return -EINVAL;
- }
- return 0;
-}
-
-int mptcp_pm_nl_set_limits_doit(struct sk_buff *skb, struct genl_info *info)
-{
- struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
- unsigned int rcv_addrs, subflows;
- int ret;
-
- spin_lock_bh(&pernet->lock);
- rcv_addrs = pernet->add_addr_accept_max;
- ret = parse_limit(info, MPTCP_PM_ATTR_RCV_ADD_ADDRS, &rcv_addrs);
- if (ret)
- goto unlock;
-
- subflows = pernet->subflows_max;
- ret = parse_limit(info, MPTCP_PM_ATTR_SUBFLOWS, &subflows);
- if (ret)
- goto unlock;
-
- WRITE_ONCE(pernet->add_addr_accept_max, rcv_addrs);
- WRITE_ONCE(pernet->subflows_max, subflows);
-
-unlock:
- spin_unlock_bh(&pernet->lock);
- return ret;
-}
-
-int mptcp_pm_nl_get_limits_doit(struct sk_buff *skb, struct genl_info *info)
-{
- struct pm_nl_pernet *pernet = genl_info_pm_nl(info);
- struct sk_buff *msg;
- void *reply;
-
- msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
- if (!msg)
- return -ENOMEM;
-
- reply = genlmsg_put_reply(msg, info, &mptcp_genl_family, 0,
- MPTCP_PM_CMD_GET_LIMITS);
- if (!reply)
- goto fail;
-
- if (nla_put_u32(msg, MPTCP_PM_ATTR_RCV_ADD_ADDRS,
- READ_ONCE(pernet->add_addr_accept_max)))
- goto fail;
-
- if (nla_put_u32(msg, MPTCP_PM_ATTR_SUBFLOWS,
- READ_ONCE(pernet->subflows_max)))
- goto fail;
-
- genlmsg_end(msg, reply);
- return genlmsg_reply(msg, info);
-
-fail:
- GENL_SET_ERR_MSG(info, "not enough space in Netlink message");
- nlmsg_free(msg);
- return -EMSGSIZE;
-}
-
-static void mptcp_pm_nl_fullmesh(struct mptcp_sock *msk,
- struct mptcp_addr_info *addr)
-{
- struct mptcp_rm_list list = { .nr = 0 };
-
- list.ids[list.nr++] = mptcp_endp_get_local_id(msk, addr);
-
- spin_lock_bh(&msk->pm.lock);
- mptcp_pm_nl_rm_subflow_received(msk, &list);
- __mark_subflow_endp_available(msk, list.ids[0]);
- mptcp_pm_create_subflow_or_signal_addr(msk);
- spin_unlock_bh(&msk->pm.lock);
-}
-
-static int mptcp_nl_set_flags(struct net *net,
- struct mptcp_addr_info *addr,
- u8 bkup, u8 changed)
-{
- long s_slot = 0, s_num = 0;
- struct mptcp_sock *msk;
- int ret = -EINVAL;
-
- while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
- struct sock *sk = (struct sock *)msk;
-
- if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
- goto next;
-
- lock_sock(sk);
- if (changed & MPTCP_PM_ADDR_FLAG_BACKUP)
- ret = mptcp_pm_nl_mp_prio_send_ack(msk, addr, NULL, bkup);
- if (changed & MPTCP_PM_ADDR_FLAG_FULLMESH)
- mptcp_pm_nl_fullmesh(msk, addr);
- release_sock(sk);
-
-next:
- sock_put(sk);
- cond_resched();
- }
-
- return ret;
-}
-
-int mptcp_pm_nl_set_flags(struct sk_buff *skb, struct genl_info *info)
-{
- struct mptcp_pm_addr_entry addr = { .addr = { .family = AF_UNSPEC }, };
- struct nlattr *attr = info->attrs[MPTCP_PM_ATTR_ADDR];
- u8 changed, mask = MPTCP_PM_ADDR_FLAG_BACKUP |
- MPTCP_PM_ADDR_FLAG_FULLMESH;
- struct net *net = sock_net(skb->sk);
- struct mptcp_pm_addr_entry *entry;
- struct pm_nl_pernet *pernet;
- u8 lookup_by_id = 0;
- u8 bkup = 0;
- int ret;
-
- pernet = pm_nl_get_pernet(net);
-
- ret = mptcp_pm_parse_entry(attr, info, false, &addr);
- if (ret < 0)
- return ret;
-
- if (addr.addr.family == AF_UNSPEC) {
- lookup_by_id = 1;
- if (!addr.addr.id) {
- GENL_SET_ERR_MSG(info, "missing required inputs");
- return -EOPNOTSUPP;
- }
- }
-
- if (addr.flags & MPTCP_PM_ADDR_FLAG_BACKUP)
- bkup = 1;
-
- spin_lock_bh(&pernet->lock);
- entry = lookup_by_id ? __lookup_addr_by_id(pernet, addr.addr.id) :
- __lookup_addr(pernet, &addr.addr);
- if (!entry) {
- spin_unlock_bh(&pernet->lock);
- GENL_SET_ERR_MSG(info, "address not found");
- return -EINVAL;
- }
- if ((addr.flags & MPTCP_PM_ADDR_FLAG_FULLMESH) &&
- (entry->flags & (MPTCP_PM_ADDR_FLAG_SIGNAL |
- MPTCP_PM_ADDR_FLAG_IMPLICIT))) {
- spin_unlock_bh(&pernet->lock);
- GENL_SET_ERR_MSG(info, "invalid addr flags");
- return -EINVAL;
- }
-
- changed = (addr.flags ^ entry->flags) & mask;
- entry->flags = (entry->flags & ~mask) | (addr.flags & mask);
- addr = *entry;
- spin_unlock_bh(&pernet->lock);
-
- mptcp_nl_set_flags(net, &addr.addr, bkup, changed);
- return 0;
-}
-
-int mptcp_pm_nl_set_flags_doit(struct sk_buff *skb, struct genl_info *info)
-{
- return mptcp_pm_set_flags(skb, info);
-}
-
-static void mptcp_nl_mcast_send(struct net *net, struct sk_buff *nlskb, gfp_t gfp)
-{
- genlmsg_multicast_netns(&mptcp_genl_family, net,
- nlskb, 0, MPTCP_PM_EV_GRP_OFFSET, gfp);
-}
-
-bool mptcp_userspace_pm_active(const struct mptcp_sock *msk)
-{
- return genl_has_listeners(&mptcp_genl_family,
- sock_net((const struct sock *)msk),
- MPTCP_PM_EV_GRP_OFFSET);
-}
-
-static int mptcp_event_add_subflow(struct sk_buff *skb, const struct sock *ssk)
-{
- const struct inet_sock *issk = inet_sk(ssk);
- const struct mptcp_subflow_context *sf;
-
- if (nla_put_u16(skb, MPTCP_ATTR_FAMILY, ssk->sk_family))
- return -EMSGSIZE;
-
- switch (ssk->sk_family) {
- case AF_INET:
- if (nla_put_in_addr(skb, MPTCP_ATTR_SADDR4, issk->inet_saddr))
- return -EMSGSIZE;
- if (nla_put_in_addr(skb, MPTCP_ATTR_DADDR4, issk->inet_daddr))
- return -EMSGSIZE;
- break;
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- case AF_INET6: {
- const struct ipv6_pinfo *np = inet6_sk(ssk);
-
- if (nla_put_in6_addr(skb, MPTCP_ATTR_SADDR6, &np->saddr))
- return -EMSGSIZE;
- if (nla_put_in6_addr(skb, MPTCP_ATTR_DADDR6, &ssk->sk_v6_daddr))
- return -EMSGSIZE;
- break;
- }
-#endif
- default:
- WARN_ON_ONCE(1);
- return -EMSGSIZE;
- }
-
- if (nla_put_be16(skb, MPTCP_ATTR_SPORT, issk->inet_sport))
- return -EMSGSIZE;
- if (nla_put_be16(skb, MPTCP_ATTR_DPORT, issk->inet_dport))
- return -EMSGSIZE;
-
- sf = mptcp_subflow_ctx(ssk);
- if (WARN_ON_ONCE(!sf))
- return -EINVAL;
-
- if (nla_put_u8(skb, MPTCP_ATTR_LOC_ID, subflow_get_local_id(sf)))
- return -EMSGSIZE;
-
- if (nla_put_u8(skb, MPTCP_ATTR_REM_ID, sf->remote_id))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int mptcp_event_put_token_and_ssk(struct sk_buff *skb,
- const struct mptcp_sock *msk,
- const struct sock *ssk)
-{
- const struct sock *sk = (const struct sock *)msk;
- const struct mptcp_subflow_context *sf;
- u8 sk_err;
-
- if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)))
- return -EMSGSIZE;
-
- if (mptcp_event_add_subflow(skb, ssk))
- return -EMSGSIZE;
-
- sf = mptcp_subflow_ctx(ssk);
- if (WARN_ON_ONCE(!sf))
- return -EINVAL;
-
- if (nla_put_u8(skb, MPTCP_ATTR_BACKUP, sf->backup))
- return -EMSGSIZE;
-
- if (ssk->sk_bound_dev_if &&
- nla_put_s32(skb, MPTCP_ATTR_IF_IDX, ssk->sk_bound_dev_if))
- return -EMSGSIZE;
-
- sk_err = READ_ONCE(ssk->sk_err);
- if (sk_err && sk->sk_state == TCP_ESTABLISHED &&
- nla_put_u8(skb, MPTCP_ATTR_ERROR, sk_err))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int mptcp_event_sub_established(struct sk_buff *skb,
- const struct mptcp_sock *msk,
- const struct sock *ssk)
-{
- return mptcp_event_put_token_and_ssk(skb, msk, ssk);
-}
-
-static int mptcp_event_sub_closed(struct sk_buff *skb,
- const struct mptcp_sock *msk,
- const struct sock *ssk)
-{
- const struct mptcp_subflow_context *sf;
-
- if (mptcp_event_put_token_and_ssk(skb, msk, ssk))
- return -EMSGSIZE;
-
- sf = mptcp_subflow_ctx(ssk);
- if (!sf->reset_seen)
- return 0;
-
- if (nla_put_u32(skb, MPTCP_ATTR_RESET_REASON, sf->reset_reason))
- return -EMSGSIZE;
-
- if (nla_put_u32(skb, MPTCP_ATTR_RESET_FLAGS, sf->reset_transient))
- return -EMSGSIZE;
-
- return 0;
-}
-
-static int mptcp_event_created(struct sk_buff *skb,
- const struct mptcp_sock *msk,
- const struct sock *ssk)
-{
- int err = nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token));
- u16 flags = 0;
-
- if (err)
- return err;
-
- if (nla_put_u8(skb, MPTCP_ATTR_SERVER_SIDE, READ_ONCE(msk->pm.server_side)))
- return -EMSGSIZE;
-
- if (READ_ONCE(msk->pm.remote_deny_join_id0))
- flags |= MPTCP_PM_EV_FLAG_DENY_JOIN_ID0;
-
- if (flags && nla_put_u16(skb, MPTCP_ATTR_FLAGS, flags))
- return -EMSGSIZE;
-
- return mptcp_event_add_subflow(skb, ssk);
-}
-
-void mptcp_event_addr_removed(const struct mptcp_sock *msk, uint8_t id)
-{
- struct net *net = sock_net((const struct sock *)msk);
- struct nlmsghdr *nlh;
- struct sk_buff *skb;
-
- if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
- return;
-
- skb = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_ATOMIC);
- if (!skb)
- return;
-
- nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0, MPTCP_EVENT_REMOVED);
- if (!nlh)
- goto nla_put_failure;
-
- if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)))
- goto nla_put_failure;
-
- if (nla_put_u8(skb, MPTCP_ATTR_REM_ID, id))
- goto nla_put_failure;
-
- genlmsg_end(skb, nlh);
- mptcp_nl_mcast_send(net, skb, GFP_ATOMIC);
- return;
-
-nla_put_failure:
- nlmsg_free(skb);
-}
-
-void mptcp_event_addr_announced(const struct sock *ssk,
- const struct mptcp_addr_info *info)
-{
- struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
- struct mptcp_sock *msk = mptcp_sk(subflow->conn);
- struct net *net = sock_net(ssk);
- struct nlmsghdr *nlh;
- struct sk_buff *skb;
-
- if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
- return;
-
- skb = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_ATOMIC);
- if (!skb)
- return;
-
- nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0,
- MPTCP_EVENT_ANNOUNCED);
- if (!nlh)
- goto nla_put_failure;
-
- if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)))
- goto nla_put_failure;
-
- if (nla_put_u8(skb, MPTCP_ATTR_REM_ID, info->id))
- goto nla_put_failure;
-
- if (nla_put_be16(skb, MPTCP_ATTR_DPORT,
- info->port == 0 ?
- inet_sk(ssk)->inet_dport :
- info->port))
- goto nla_put_failure;
-
- switch (info->family) {
- case AF_INET:
- if (nla_put_in_addr(skb, MPTCP_ATTR_DADDR4, info->addr.s_addr))
- goto nla_put_failure;
- break;
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- case AF_INET6:
- if (nla_put_in6_addr(skb, MPTCP_ATTR_DADDR6, &info->addr6))
- goto nla_put_failure;
- break;
-#endif
- default:
- WARN_ON_ONCE(1);
- goto nla_put_failure;
- }
-
- genlmsg_end(skb, nlh);
- mptcp_nl_mcast_send(net, skb, GFP_ATOMIC);
- return;
-
-nla_put_failure:
- nlmsg_free(skb);
-}
-
-void mptcp_event_pm_listener(const struct sock *ssk,
- enum mptcp_event_type event)
-{
- const struct inet_sock *issk = inet_sk(ssk);
- struct net *net = sock_net(ssk);
- struct nlmsghdr *nlh;
- struct sk_buff *skb;
-
- if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
- return;
-
- skb = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
- if (!skb)
- return;
-
- nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0, event);
- if (!nlh)
- goto nla_put_failure;
-
- if (nla_put_u16(skb, MPTCP_ATTR_FAMILY, ssk->sk_family))
- goto nla_put_failure;
-
- if (nla_put_be16(skb, MPTCP_ATTR_SPORT, issk->inet_sport))
- goto nla_put_failure;
-
- switch (ssk->sk_family) {
- case AF_INET:
- if (nla_put_in_addr(skb, MPTCP_ATTR_SADDR4, issk->inet_saddr))
- goto nla_put_failure;
- break;
-#if IS_ENABLED(CONFIG_MPTCP_IPV6)
- case AF_INET6: {
- const struct ipv6_pinfo *np = inet6_sk(ssk);
-
- if (nla_put_in6_addr(skb, MPTCP_ATTR_SADDR6, &np->saddr))
- goto nla_put_failure;
- break;
- }
-#endif
- default:
- WARN_ON_ONCE(1);
- goto nla_put_failure;
- }
-
- genlmsg_end(skb, nlh);
- mptcp_nl_mcast_send(net, skb, GFP_KERNEL);
- return;
-
-nla_put_failure:
- nlmsg_free(skb);
-}
-
-void mptcp_event(enum mptcp_event_type type, const struct mptcp_sock *msk,
- const struct sock *ssk, gfp_t gfp)
-{
- struct net *net = sock_net((const struct sock *)msk);
- struct nlmsghdr *nlh;
- struct sk_buff *skb;
-
- if (!genl_has_listeners(&mptcp_genl_family, net, MPTCP_PM_EV_GRP_OFFSET))
- return;
-
- skb = nlmsg_new(NLMSG_DEFAULT_SIZE, gfp);
- if (!skb)
- return;
-
- nlh = genlmsg_put(skb, 0, 0, &mptcp_genl_family, 0, type);
- if (!nlh)
- goto nla_put_failure;
-
- switch (type) {
- case MPTCP_EVENT_UNSPEC:
- WARN_ON_ONCE(1);
- break;
- case MPTCP_EVENT_CREATED:
- case MPTCP_EVENT_ESTABLISHED:
- if (mptcp_event_created(skb, msk, ssk) < 0)
- goto nla_put_failure;
- break;
- case MPTCP_EVENT_CLOSED:
- if (nla_put_u32(skb, MPTCP_ATTR_TOKEN, READ_ONCE(msk->token)) < 0)
- goto nla_put_failure;
- break;
- case MPTCP_EVENT_ANNOUNCED:
- case MPTCP_EVENT_REMOVED:
- /* call mptcp_event_addr_announced()/removed instead */
- WARN_ON_ONCE(1);
- break;
- case MPTCP_EVENT_SUB_ESTABLISHED:
- case MPTCP_EVENT_SUB_PRIORITY:
- if (mptcp_event_sub_established(skb, msk, ssk) < 0)
- goto nla_put_failure;
- break;
- case MPTCP_EVENT_SUB_CLOSED:
- if (mptcp_event_sub_closed(skb, msk, ssk) < 0)
- goto nla_put_failure;
- break;
- case MPTCP_EVENT_LISTENER_CREATED:
- case MPTCP_EVENT_LISTENER_CLOSED:
- break;
- }
-
- genlmsg_end(skb, nlh);
- mptcp_nl_mcast_send(net, skb, gfp);
- return;
-
-nla_put_failure:
- nlmsg_free(skb);
-}
-
-struct genl_family mptcp_genl_family __ro_after_init = {
- .name = MPTCP_PM_NAME,
- .version = MPTCP_PM_VER,
- .netnsok = true,
- .module = THIS_MODULE,
- .ops = mptcp_pm_nl_ops,
- .n_ops = ARRAY_SIZE(mptcp_pm_nl_ops),
- .resv_start_op = MPTCP_PM_CMD_SUBFLOW_DESTROY + 1,
- .mcgrps = mptcp_pm_mcgrps,
- .n_mcgrps = ARRAY_SIZE(mptcp_pm_mcgrps),
-};
-
-static int __net_init pm_nl_init_net(struct net *net)
-{
- struct pm_nl_pernet *pernet = pm_nl_get_pernet(net);
-
- INIT_LIST_HEAD_RCU(&pernet->local_addr_list);
-
- /* Cit. 2 subflows ought to be enough for anybody. */
- pernet->subflows_max = 2;
- pernet->next_id = 1;
- pernet->stale_loss_cnt = 4;
- spin_lock_init(&pernet->lock);
-
- /* No need to initialize other pernet fields, the struct is zeroed at
- * allocation time.
- */
-
- return 0;
-}
-
-static void __net_exit pm_nl_exit_net(struct list_head *net_list)
-{
- struct net *net;
-
- list_for_each_entry(net, net_list, exit_list) {
- struct pm_nl_pernet *pernet = pm_nl_get_pernet(net);
-
- /* net is removed from namespace list, can't race with
- * other modifiers, also netns core already waited for a
- * RCU grace period.
- */
- __flush_addrs(&pernet->local_addr_list);
- }
-}
-
-static struct pernet_operations mptcp_pm_pernet_ops = {
- .init = pm_nl_init_net,
- .exit_batch = pm_nl_exit_net,
- .id = &pm_nl_pernet_id,
- .size = sizeof(struct pm_nl_pernet),
-};
-
-void __init mptcp_pm_nl_init(void)
-{
- if (register_pernet_subsys(&mptcp_pm_pernet_ops) < 0)
- panic("Failed to register MPTCP PM pernet subsystem.\n");
-
- if (genl_register_family(&mptcp_genl_family))
- panic("Failed to register MPTCP PM netlink family\n");
-}
diff --git a/net/mptcp/pm_userspace.c b/net/mptcp/pm_userspace.c
index 33494c4a8fb2..e0a858df932a 100644
--- a/net/mptcp/pm_userspace.c
+++ b/net/mptcp/pm_userspace.c
@@ -41,7 +41,6 @@ static int mptcp_userspace_pm_append_new_local_addr(struct mptcp_sock *msk,
bool needs_id)
{
DECLARE_BITMAP(id_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
- struct mptcp_pm_addr_entry *match = NULL;
struct sock *sk = (struct sock *)msk;
struct mptcp_pm_addr_entry *e;
bool addr_match = false;
@@ -60,34 +59,39 @@ static int mptcp_userspace_pm_append_new_local_addr(struct mptcp_sock *msk,
if (addr_match && entry->addr.id == 0 && needs_id)
entry->addr.id = e->addr.id;
id_match = (e->addr.id == entry->addr.id);
- if (addr_match && id_match) {
- match = e;
+ if (addr_match || id_match)
break;
- } else if (addr_match || id_match) {
- break;
- }
__set_bit(e->addr.id, id_bitmap);
}
- if (!match && !addr_match && !id_match) {
+ if (!addr_match && !id_match) {
+ unsigned int id;
+
+ if (!entry->addr.id && needs_id) {
+ id = find_next_zero_bit(id_bitmap,
+ MPTCP_PM_MAX_ADDR_ID + 1, 1);
+ if (id > MPTCP_PM_MAX_ADDR_ID) {
+ ret = -ENOSPC;
+ goto append_err;
+ }
+ } else {
+ id = entry->addr.id;
+ }
+
/* Memory for the entry is allocated from the
* sock option buffer.
*/
- e = sock_kmalloc(sk, sizeof(*e), GFP_ATOMIC);
+ e = sock_kmemdup(sk, entry, sizeof(*entry), GFP_ATOMIC);
if (!e) {
ret = -ENOMEM;
goto append_err;
}
- *e = *entry;
- if (!e->addr.id && needs_id)
- e->addr.id = find_next_zero_bit(id_bitmap,
- MPTCP_PM_MAX_ADDR_ID + 1,
- 1);
+ e->addr.id = id;
list_add_tail_rcu(&e->list, &msk->pm.userspace_pm_local_addr_list);
msk->pm.local_addr_used++;
ret = e->addr.id;
- } else if (match) {
+ } else if (addr_match && id_match) {
ret = entry->addr.id;
}
diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index ee160c783ff5..7a64223c3884 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -67,6 +67,27 @@ static const struct proto_ops *mptcp_fallback_tcp_ops(const struct sock *sk)
return &inet_stream_ops;
}
+bool __mptcp_try_fallback(struct mptcp_sock *msk, int fb_mib)
+{
+ struct net *net = sock_net((struct sock *)msk);
+
+ if (__mptcp_check_fallback(msk))
+ return true;
+
+ spin_lock_bh(&msk->fallback_lock);
+ if (!msk->allow_infinite_fallback) {
+ spin_unlock_bh(&msk->fallback_lock);
+ return false;
+ }
+
+ msk->allow_subflows = false;
+ set_bit(MPTCP_FALLBACK_DONE, &msk->flags);
+ clear_bit(MPTCP_RTX_ENABLED, &msk->flags);
+ __MPTCP_INC_STATS(net, fb_mib);
+ spin_unlock_bh(&msk->fallback_lock);
+ return true;
+}
+
static int __mptcp_socket_create(struct mptcp_sock *msk)
{
struct mptcp_subflow_context *subflow;
@@ -414,9 +435,7 @@ static bool __mptcp_move_skb(struct sock *sk, struct sk_buff *skb)
static void mptcp_stop_rtx_timer(struct sock *sk)
{
- struct inet_connection_sock *icsk = inet_csk(sk);
-
- sk_stop_timer(sk, &icsk->icsk_retransmit_timer);
+ sk_stop_timer(sk, &sk->mptcp_retransmit_timer);
mptcp_sk(sk)->timer_ival = 0;
}
@@ -659,10 +678,7 @@ static bool mptcp_check_data_fin(struct sock *sk)
static void mptcp_dss_corruption(struct mptcp_sock *msk, struct sock *ssk)
{
- if (mptcp_try_fallback(ssk)) {
- MPTCP_INC_STATS(sock_net(ssk),
- MPTCP_MIB_DSSCORRUPTIONFALLBACK);
- } else {
+ if (!mptcp_try_fallback(ssk, MPTCP_MIB_DSSCORRUPTIONFALLBACK)) {
MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_DSSCORRUPTIONRESET);
mptcp_subflow_reset(ssk);
}
@@ -973,20 +989,20 @@ static void __mptcp_flush_join_list(struct sock *sk, struct list_head *join_list
static bool mptcp_rtx_timer_pending(struct sock *sk)
{
- return timer_pending(&inet_csk(sk)->icsk_retransmit_timer);
+ return timer_pending(&sk->mptcp_retransmit_timer);
}
static void mptcp_reset_rtx_timer(struct sock *sk)
{
- struct inet_connection_sock *icsk = inet_csk(sk);
+ struct mptcp_sock *msk = mptcp_sk(sk);
unsigned long tout;
- /* prevent rescheduling on close */
- if (unlikely(inet_sk_state_load(sk) == TCP_CLOSE))
+ /* Prevent rescheduling on close and in case of fallback. */
+ if (!test_bit(MPTCP_RTX_ENABLED, &msk->flags))
return;
- tout = mptcp_sk(sk)->timer_ival;
- sk_reset_timer(sk, &icsk->icsk_retransmit_timer, jiffies + tout);
+ tout = msk->timer_ival;
+ sk_reset_timer(sk, &sk->mptcp_retransmit_timer, jiffies + tout);
}
bool mptcp_schedule_work(struct sock *sk)
@@ -1293,12 +1309,12 @@ static void mptcp_update_infinite_map(struct mptcp_sock *msk,
mpext->infinite_map = 1;
mpext->data_len = 0;
- if (!mptcp_try_fallback(ssk)) {
+ if (!mptcp_try_fallback(ssk, MPTCP_MIB_INFINITEMAPTX)) {
+ MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_FALLBACKFAILED);
mptcp_subflow_reset(ssk);
return;
}
- MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_INFINITEMAPTX);
mptcp_subflow_ctx(ssk)->send_infinite_map = 0;
pr_fallback(msk);
}
@@ -2364,7 +2380,7 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
mptcp_cleanup_rbuf(msk, copied);
err = sk_wait_data(sk, &timeo, NULL);
if (err < 0) {
- err = copied ? : err;
+ copied = copied ? : err;
goto out_err;
}
}
@@ -2393,9 +2409,7 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
static void mptcp_retransmit_timer(struct timer_list *t)
{
- struct inet_connection_sock *icsk = from_timer(icsk, t,
- icsk_retransmit_timer);
- struct sock *sk = &icsk->icsk_inet.sk;
+ struct sock *sk = from_timer(sk, t, mptcp_retransmit_timer);
struct mptcp_sock *msk = mptcp_sk(sk);
bh_lock_sock(sk);
@@ -2413,7 +2427,9 @@ static void mptcp_retransmit_timer(struct timer_list *t)
static void mptcp_tout_timer(struct timer_list *t)
{
- struct sock *sk = from_timer(sk, t, sk_timer);
+ struct inet_connection_sock *icsk =
+ from_timer(icsk, t, mptcp_tout_timer);
+ struct sock *sk = &icsk->icsk_inet.sk;
mptcp_schedule_work(sk);
sock_put(sk);
@@ -2856,7 +2872,7 @@ void mptcp_reset_tout_timer(struct mptcp_sock *msk, unsigned long fail_tout)
*/
timeout = inet_csk(sk)->icsk_mtup.probe_timestamp ? close_timeout : fail_tout;
- sk_reset_timer(sk, &sk->sk_timer, timeout);
+ sk_reset_timer(sk, &inet_csk(sk)->mptcp_tout_timer, timeout);
}
static void mptcp_mp_fail_no_response(struct mptcp_sock *msk)
@@ -2988,9 +3004,8 @@ static void __mptcp_init_sock(struct sock *sk)
mptcp_pm_data_init(msk);
spin_lock_init(&msk->fallback_lock);
- /* re-use the csk retrans timer for MPTCP-level retrans */
- timer_setup(&msk->sk.icsk_retransmit_timer, mptcp_retransmit_timer, 0);
- timer_setup(&sk->sk_timer, mptcp_tout_timer, 0);
+ timer_setup(&sk->mptcp_retransmit_timer, mptcp_retransmit_timer, 0);
+ timer_setup(&msk->sk.mptcp_tout_timer, mptcp_tout_timer, 0);
}
static void mptcp_ca_reset(struct sock *sk)
@@ -3108,6 +3123,9 @@ void mptcp_set_state(struct sock *sk, int state)
* transition from TCP_SYN_RECV to TCP_CLOSE_WAIT.
*/
break;
+ case TCP_CLOSE:
+ clear_bit(MPTCP_RTX_ENABLED, &mptcp_sk(sk)->flags);
+ fallthrough;
default:
if (oldstate == TCP_ESTABLISHED || oldstate == TCP_CLOSE_WAIT)
MPTCP_DEC_STATS(sock_net(sk), MPTCP_MIB_CURRESTAB);
@@ -3192,7 +3210,7 @@ static void __mptcp_destroy_sock(struct sock *sk)
might_sleep();
mptcp_stop_rtx_timer(sk);
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &inet_csk(sk)->mptcp_tout_timer);
msk->pm.status = 0;
mptcp_release_sched(msk);
@@ -3931,6 +3949,7 @@ static int mptcp_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
if (IS_ERR(ssk))
return PTR_ERR(ssk);
+ set_bit(MPTCP_RTX_ENABLED, &msk->flags);
mptcp_set_state(sk, TCP_SYN_SENT);
subflow = mptcp_subflow_ctx(ssk);
#ifdef CONFIG_TCP_MD5SIG
@@ -3938,16 +3957,15 @@ static int mptcp_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
* TCP option space.
*/
if (rcu_access_pointer(tcp_sk(ssk)->md5sig_info))
- mptcp_subflow_early_fallback(msk, subflow);
+ mptcp_subflow_early_fallback(msk, subflow, MPTCP_MIB_MD5SIGFALLBACK);
#endif
if (subflow->request_mptcp) {
- if (mptcp_active_should_disable(sk)) {
- MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_MPCAPABLEACTIVEDISABLED);
- mptcp_subflow_early_fallback(msk, subflow);
- } else if (mptcp_token_new_connect(ssk) < 0) {
- MPTCP_INC_STATS(sock_net(ssk), MPTCP_MIB_TOKENFALLBACKINIT);
- mptcp_subflow_early_fallback(msk, subflow);
- }
+ if (mptcp_active_should_disable(sk))
+ mptcp_subflow_early_fallback(msk, subflow,
+ MPTCP_MIB_MPCAPABLEACTIVEDISABLED);
+ else if (mptcp_token_new_connect(ssk) < 0)
+ mptcp_subflow_early_fallback(msk, subflow,
+ MPTCP_MIB_TOKENFALLBACKINIT);
}
WRITE_ONCE(msk->write_seq, subflow->idsn);
@@ -4080,6 +4098,7 @@ static int mptcp_listen(struct socket *sock, int backlog)
goto unlock;
}
+ set_bit(MPTCP_RTX_ENABLED, &msk->flags);
mptcp_set_state(sk, TCP_LISTEN);
sock_set_flag(sk, SOCK_RCU_FREE);
diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h
index 5fdf2b55b3cc..dbfe959d3a18 100644
--- a/net/mptcp/protocol.h
+++ b/net/mptcp/protocol.h
@@ -116,6 +116,7 @@
#define MPTCP_WORK_RTX 1
#define MPTCP_FALLBACK_DONE 2
#define MPTCP_WORK_CLOSE_SUBFLOW 3
+#define MPTCP_RTX_ENABLED 4
/* MPTCP socket release cb flags */
#define MPTCP_PUSH_PENDING 1
@@ -845,7 +846,7 @@ static inline void mptcp_stop_tout_timer(struct sock *sk)
if (!inet_csk(sk)->icsk_mtup.probe_timestamp)
return;
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &inet_csk(sk)->mptcp_tout_timer);
inet_csk(sk)->icsk_mtup.probe_timestamp = 0;
}
@@ -1049,10 +1050,10 @@ bool mptcp_pm_alloc_anno_list(struct mptcp_sock *msk,
const struct mptcp_addr_info *addr);
void mptcp_pm_free_anno_list(struct mptcp_sock *msk);
bool mptcp_pm_sport_in_anno_list(struct mptcp_sock *msk, const struct sock *sk);
-struct mptcp_pm_add_entry *
+struct mptcp_pm_add_addr *
mptcp_pm_del_add_timer(struct mptcp_sock *msk,
const struct mptcp_addr_info *addr, bool check_id);
-struct mptcp_pm_add_entry *
+struct mptcp_pm_add_addr *
mptcp_lookup_anno_list_by_saddr(const struct mptcp_sock *msk,
const struct mptcp_addr_info *addr);
int mptcp_pm_set_flags(struct sk_buff *skb, struct genl_info *info);
@@ -1216,24 +1217,6 @@ static inline bool mptcp_check_fallback(const struct sock *sk)
return __mptcp_check_fallback(msk);
}
-static inline bool __mptcp_try_fallback(struct mptcp_sock *msk)
-{
- if (__mptcp_check_fallback(msk)) {
- pr_debug("TCP fallback already done (msk=%p)\n", msk);
- return true;
- }
- spin_lock_bh(&msk->fallback_lock);
- if (!msk->allow_infinite_fallback) {
- spin_unlock_bh(&msk->fallback_lock);
- return false;
- }
-
- msk->allow_subflows = false;
- set_bit(MPTCP_FALLBACK_DONE, &msk->flags);
- spin_unlock_bh(&msk->fallback_lock);
- return true;
-}
-
static inline bool __mptcp_has_initial_subflow(const struct mptcp_sock *msk)
{
struct sock *ssk = READ_ONCE(msk->first);
@@ -1243,14 +1226,16 @@ static inline bool __mptcp_has_initial_subflow(const struct mptcp_sock *msk)
TCPF_SYN_RECV | TCPF_LISTEN));
}
-static inline bool mptcp_try_fallback(struct sock *ssk)
+bool __mptcp_try_fallback(struct mptcp_sock *msk, int fb_mib);
+
+static inline bool mptcp_try_fallback(struct sock *ssk, int fb_mib)
{
struct mptcp_subflow_context *subflow = mptcp_subflow_ctx(ssk);
struct sock *sk = subflow->conn;
struct mptcp_sock *msk;
msk = mptcp_sk(sk);
- if (!__mptcp_try_fallback(msk))
+ if (!__mptcp_try_fallback(msk, fb_mib))
return false;
if (READ_ONCE(msk->snd_data_fin_enable) && !(ssk->sk_shutdown & SEND_SHUTDOWN)) {
gfp_t saved_allocation = ssk->sk_allocation;
@@ -1269,11 +1254,12 @@ static inline bool mptcp_try_fallback(struct sock *ssk)
#define pr_fallback(a) pr_debug("%s:fallback to TCP (msk=%p)\n", __func__, a)
static inline void mptcp_subflow_early_fallback(struct mptcp_sock *msk,
- struct mptcp_subflow_context *subflow)
+ struct mptcp_subflow_context *subflow,
+ int fb_mib)
{
pr_fallback(msk);
subflow->request_mptcp = 0;
- WARN_ON_ONCE(!__mptcp_try_fallback(msk));
+ WARN_ON_ONCE(!__mptcp_try_fallback(msk, fb_mib));
}
static inline bool mptcp_check_infinite_map(struct sk_buff *skb)
diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c
index cfa029b4d7f3..2b2d0369009b 100644
--- a/net/mptcp/subflow.c
+++ b/net/mptcp/subflow.c
@@ -550,11 +550,13 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
mptcp_get_options(skb, &mp_opt);
if (subflow->request_mptcp) {
if (!(mp_opt.suboptions & OPTION_MPTCP_MPC_SYNACK)) {
- if (!mptcp_try_fallback(sk))
+ if (!mptcp_try_fallback(sk,
+ MPTCP_MIB_MPCAPABLEACTIVEFALLBACK)) {
+ MPTCP_INC_STATS(sock_net(sk),
+ MPTCP_MIB_FALLBACKFAILED);
goto do_reset;
+ }
- MPTCP_INC_STATS(sock_net(sk),
- MPTCP_MIB_MPCAPABLEACTIVEFALLBACK);
pr_fallback(msk);
goto fallback;
}
@@ -1404,7 +1406,7 @@ static bool subflow_check_data_avail(struct sock *ssk)
return true;
}
- if (!mptcp_try_fallback(ssk)) {
+ if (!mptcp_try_fallback(ssk, MPTCP_MIB_DSSFALLBACK)) {
/* fatal protocol error, close the socket.
* subflow_error_report() will introduce the appropriate barriers
*/
diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index 1f62ca2f7330..4e30a903f4bd 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -1776,6 +1776,12 @@ ip_vs_in_icmp(struct netns_ipvs *ipvs, struct sk_buff *skb, int *related,
/* Ensure the IP header is present in headroom */
if (!pskb_may_pull(skb, hlen_orig))
goto ignore_tunnel;
+ skb_set_transport_header(skb, hlen_orig);
+ /* Before now we may used ihl from skb frag, revalidate it after
+ * copying it into skb head to prevent out-of-bounds access
+ */
+ if (ip_hdr(skb)->ihl * 4 != hlen_orig)
+ goto ignore_tunnel;
IP_VS_DBG(12, "Sending ICMP for %pI4->%pI4: t=%u, c=%u, i=%u\n",
&ip_hdr(skb)->saddr, &ip_hdr(skb)->daddr,
type, code, ntohl(info));
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index 0c457e159727..77a445a59af3 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -1692,16 +1692,18 @@ void nf_conntrack_free(struct nf_conn *ct)
*/
WARN_ON(refcount_read(&ct->ct_general.use) != 0);
+ rcu_read_lock();
if (ct->status & IPS_SRC_NAT_DONE) {
const struct nf_nat_hook *nat_hook;
- rcu_read_lock();
nat_hook = rcu_dereference(nf_nat_hook);
if (nat_hook)
nat_hook->remove_nat_bysrc(ct);
- rcu_read_unlock();
}
+ nf_ct_timeout_put(ct);
+ rcu_read_unlock();
+
kfree(ct->ext);
kmem_cache_free(nf_conntrack_cachep, ct);
cnet = nf_ct_pernet(net);
diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c
index a081d38702f5..8bb32965edb8 100644
--- a/net/netfilter/nf_conntrack_netlink.c
+++ b/net/netfilter/nf_conntrack_netlink.c
@@ -3390,7 +3390,8 @@ static bool expect_iter_name(struct nf_conntrack_expect *exp, void *data)
struct nf_conntrack_helper *helper;
const char *name = data;
- helper = rcu_dereference(exp->helper);
+ helper = rcu_dereference_protected(exp->helper,
+ lockdep_is_held(&nf_conntrack_expect_lock));
if (!helper)
return false;
diff --git a/net/netfilter/nf_conntrack_timeout.c b/net/netfilter/nf_conntrack_timeout.c
index 0cc584d3dbb1..c81becde2afa 100644
--- a/net/netfilter/nf_conntrack_timeout.c
+++ b/net/netfilter/nf_conntrack_timeout.c
@@ -25,17 +25,32 @@
const struct nf_ct_timeout_hooks __rcu *nf_ct_timeout_hook __read_mostly;
EXPORT_SYMBOL_GPL(nf_ct_timeout_hook);
+/* nf_ct_iterate_cleanup() holds the conntrack lock. */
static int untimeout(struct nf_conn *ct, void *timeout)
{
struct nf_conn_timeout *timeout_ext = nf_ct_timeout_find(ct);
if (timeout_ext) {
- const struct nf_ct_timeout *t;
+ struct nf_ct_timeout *t;
- t = rcu_access_pointer(timeout_ext->timeout);
+ rcu_read_lock();
+ t = rcu_dereference(timeout_ext->timeout);
+ if (!t) {
+ rcu_read_unlock();
+ return 0;
+ }
- if (!timeout || t == timeout)
+ if (!timeout || t == timeout) {
RCU_INIT_POINTER(timeout_ext->timeout, NULL);
+
+ /* No race with nf_conntrack_free() which is called
+ * only after the conntrack has been removed from
+ * the hashes.
+ */
+ if (refcount_dec_and_test(&t->refcnt))
+ kfree_rcu(t, rcu);
+ }
+ rcu_read_unlock();
}
/* We are not intended to delete this conntrack. */
@@ -70,6 +85,8 @@ int nf_ct_set_timeout(struct net *net, struct nf_conn *ct,
const char *errmsg = NULL;
int ret = 0;
+ WARN_ON_ONCE(!nf_ct_is_template(ct));
+
rcu_read_lock();
h = rcu_dereference(nf_ct_timeout_hook);
if (!h) {
@@ -127,6 +144,8 @@ void nf_ct_destroy_timeout(struct nf_conn *ct)
struct nf_conn_timeout *timeout_ext;
const struct nf_ct_timeout_hooks *h;
+ WARN_ON_ONCE(!nf_ct_is_template(ct));
+
rcu_read_lock();
h = rcu_dereference(nf_ct_timeout_hook);
@@ -139,6 +158,8 @@ void nf_ct_destroy_timeout(struct nf_conn *ct)
if (t)
h->timeout_put(t);
RCU_INIT_POINTER(timeout_ext->timeout, NULL);
+ if (t && refcount_dec_and_test(&t->refcnt))
+ kfree_rcu(t, rcu);
}
}
rcu_read_unlock();
diff --git a/net/netfilter/nf_flow_table_core.c b/net/netfilter/nf_flow_table_core.c
index b9e3ac950894..b00f1c68a8e7 100644
--- a/net/netfilter/nf_flow_table_core.c
+++ b/net/netfilter/nf_flow_table_core.c
@@ -205,6 +205,14 @@ static void flow_offload_route_release(struct flow_offload *flow)
nft_flow_dst_release(flow, FLOW_OFFLOAD_DIR_REPLY);
}
+static void flow_offload_free_rcu(struct rcu_head *rcu_head)
+{
+ struct flow_offload *flow = container_of(rcu_head, struct flow_offload, rcu_head);
+
+ nf_ct_put(flow->ct);
+ kfree(flow);
+}
+
void flow_offload_free(struct flow_offload *flow)
{
switch (flow->type) {
@@ -214,8 +222,7 @@ void flow_offload_free(struct flow_offload *flow)
default:
break;
}
- nf_ct_put(flow->ct);
- kfree_rcu(flow, rcu_head);
+ call_rcu(&flow->rcu_head, flow_offload_free_rcu);
}
EXPORT_SYMBOL_GPL(flow_offload_free);
@@ -686,6 +693,7 @@ static int __init nf_flow_table_module_init(void)
static void __exit nf_flow_table_module_exit(void)
{
+ rcu_barrier();
nf_flow_table_offload_exit();
unregister_pernet_subsys(&nf_flow_table_net_ops);
}
diff --git a/net/netfilter/nf_flow_table_offload.c b/net/netfilter/nf_flow_table_offload.c
index 4f346f51d7d7..8e3edc7c37b3 100644
--- a/net/netfilter/nf_flow_table_offload.c
+++ b/net/netfilter/nf_flow_table_offload.c
@@ -993,7 +993,6 @@ static void flow_offload_work_del(struct flow_offload_work *offload)
flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_ORIGINAL);
if (test_bit(NF_FLOW_HW_BIDIRECTIONAL, &offload->flow->flags))
flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_REPLY);
- set_bit(NF_FLOW_HW_DEAD, &offload->flow->flags);
}
static void flow_offload_tuple_stats(struct flow_offload_work *offload,
@@ -1057,6 +1056,12 @@ static void flow_offload_work_handler(struct work_struct *work)
}
clear_bit(NF_FLOW_HW_PENDING, &offload->flow->flags);
+ if (offload->cmd == FLOW_CLS_DESTROY) {
+ /* Publish after the worker's last flow access. */
+ smp_mb__before_atomic();
+ set_bit(NF_FLOW_HW_DEAD, &offload->flow->flags);
+ }
+
kfree(offload);
}
diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c
index 2613ebfdc854..e4325686953e 100644
--- a/net/netfilter/nf_tables_api.c
+++ b/net/netfilter/nf_tables_api.c
@@ -6617,11 +6617,14 @@ static int nft_setelem_catchall_insert(const struct net *net,
{
struct nft_set_elem_catchall *catchall;
u8 genmask = nft_genmask_next(net);
+ u64 tstamp = nft_net_tstamp(net);
struct nft_set_ext *ext;
list_for_each_entry(catchall, &set->catchall_list, list) {
ext = nft_set_elem_ext(set, catchall->elem);
- if (nft_set_elem_active(ext, genmask)) {
+ if (nft_set_elem_active(ext, genmask) &&
+ !__nft_set_elem_expired(ext, tstamp) &&
+ !nft_set_elem_is_dead(ext)) {
*priv = catchall->elem;
return -EEXIST;
}
@@ -6682,11 +6685,14 @@ static int nft_setelem_catchall_deactivate(const struct net *net,
struct nft_set_elem *elem)
{
struct nft_set_elem_catchall *catchall;
+ u64 tstamp = nft_net_tstamp(net);
struct nft_set_ext *ext;
list_for_each_entry(catchall, &set->catchall_list, list) {
ext = nft_set_elem_ext(set, catchall->elem);
- if (!nft_is_active_next(net, ext))
+ if (!nft_is_active_next(net, ext) ||
+ __nft_set_elem_expired(ext, tstamp) ||
+ nft_set_elem_is_dead(ext))
continue;
kfree(elem->priv);
@@ -10360,8 +10366,8 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
nft_chain_commit_update(nft_trans_container_chain(trans));
nf_tables_chain_notify(&ctx, NFT_MSG_NEWCHAIN,
&nft_trans_chain_hooks(trans));
- list_splice(&nft_trans_chain_hooks(trans),
- &nft_trans_basechain(trans)->hook_list);
+ list_splice_rcu(&nft_trans_chain_hooks(trans),
+ &nft_trans_basechain(trans)->hook_list);
/* trans destroyed after rcu grace period */
} else {
nft_chain_commit_drop_policy(nft_trans_container_chain(trans));
@@ -10513,8 +10519,8 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb)
nft_trans_flowtable(trans),
&nft_trans_flowtable_hooks(trans),
NFT_MSG_NEWFLOWTABLE);
- list_splice(&nft_trans_flowtable_hooks(trans),
- &nft_trans_flowtable(trans)->hook_list);
+ list_splice_rcu(&nft_trans_flowtable_hooks(trans),
+ &nft_trans_flowtable(trans)->hook_list);
} else {
nft_clear(net, nft_trans_flowtable(trans));
nf_tables_flowtable_notify(&ctx,
@@ -11362,47 +11368,6 @@ int nft_data_dump(struct sk_buff *skb, int attr, const struct nft_data *data,
}
EXPORT_SYMBOL_GPL(nft_data_dump);
-static void __nft_release_basechain_now(struct nft_ctx *ctx)
-{
- struct nft_rule *rule, *nr;
-
- list_for_each_entry_safe(rule, nr, &ctx->chain->rules, list) {
- list_del(&rule->list);
- nf_tables_rule_release(ctx, rule);
- }
- nf_tables_chain_destroy(ctx->chain);
-}
-
-int __nft_release_basechain(struct nft_ctx *ctx)
-{
- struct nft_rule *rule;
-
- if (WARN_ON_ONCE(!nft_is_base_chain(ctx->chain)))
- return 0;
-
- nf_tables_unregister_hook(ctx->net, ctx->chain->table, ctx->chain);
- list_for_each_entry(rule, &ctx->chain->rules, list)
- nft_use_dec(&ctx->chain->use);
-
- nft_chain_del(ctx->chain);
- nft_use_dec(&ctx->table->use);
-
- if (!maybe_get_net(ctx->net)) {
- __nft_release_basechain_now(ctx);
- return 0;
- }
-
- /* wait for ruleset dumps to complete. Owning chain is no longer in
- * lists, so new dumps can't find any of these rules anymore.
- */
- synchronize_rcu();
-
- __nft_release_basechain_now(ctx);
- put_net(ctx->net);
- return 0;
-}
-EXPORT_SYMBOL_GPL(__nft_release_basechain);
-
static void __nft_release_hook(struct net *net, struct nft_table *table)
{
struct nft_flowtable *flowtable;
diff --git a/net/netfilter/nfnetlink_cttimeout.c b/net/netfilter/nfnetlink_cttimeout.c
index eab4f476b47f..639b1d4efd7a 100644
--- a/net/netfilter/nfnetlink_cttimeout.c
+++ b/net/netfilter/nfnetlink_cttimeout.c
@@ -37,11 +37,8 @@ struct ctnl_timeout {
struct list_head head;
struct list_head free_head;
struct rcu_head rcu_head;
- refcount_t refcnt;
char name[CTNL_TIMEOUT_NAME_MAX];
-
- /* must be at the end */
- struct nf_ct_timeout timeout;
+ struct nf_ct_timeout *timeout;
};
struct nfct_timeout_pernet {
@@ -133,12 +130,12 @@ static int cttimeout_new_timeout(struct sk_buff *skb,
/* You cannot replace one timeout policy by another of
* different kind, sorry.
*/
- if (matching->timeout.l3num != l3num ||
- matching->timeout.l4proto->l4proto != l4num)
+ if (matching->timeout->l3num != l3num ||
+ matching->timeout->l4proto->l4proto != l4num)
return -EINVAL;
- return ctnl_timeout_parse_policy(&matching->timeout.data,
- matching->timeout.l4proto,
+ return ctnl_timeout_parse_policy(&matching->timeout->data,
+ matching->timeout->l4proto,
info->net,
cda[CTA_TIMEOUT_DATA]);
}
@@ -154,26 +151,35 @@ static int cttimeout_new_timeout(struct sk_buff *skb,
goto err_proto_put;
}
- timeout = kzalloc(sizeof(struct ctnl_timeout) +
- l4proto->ctnl_timeout.obj_size, GFP_KERNEL);
+ timeout = kzalloc(sizeof(*timeout), GFP_KERNEL);
if (timeout == NULL) {
ret = -ENOMEM;
goto err_proto_put;
}
- ret = ctnl_timeout_parse_policy(&timeout->timeout.data, l4proto,
+ timeout->timeout = kzalloc(sizeof(*timeout->timeout) +
+ l4proto->ctnl_timeout.obj_size, GFP_KERNEL);
+ if (!timeout->timeout) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ ret = ctnl_timeout_parse_policy(&timeout->timeout->data, l4proto,
info->net, cda[CTA_TIMEOUT_DATA]);
if (ret < 0)
- goto err;
+ goto err_free_timeout_policy;
strcpy(timeout->name, nla_data(cda[CTA_TIMEOUT_NAME]));
- timeout->timeout.l3num = l3num;
- timeout->timeout.l4proto = l4proto;
- refcount_set(&timeout->refcnt, 1);
+ timeout->timeout->l3num = l3num;
+ timeout->timeout->l4proto = l4proto;
+ refcount_set(&timeout->timeout->refcnt, 1);
__module_get(THIS_MODULE);
list_add_tail_rcu(&timeout->head, &pernet->nfct_timeout_list);
return 0;
+
+err_free_timeout_policy:
+ kfree(timeout->timeout);
err:
kfree(timeout);
err_proto_put:
@@ -186,7 +192,7 @@ ctnl_timeout_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type,
{
struct nlmsghdr *nlh;
unsigned int flags = portid ? NLM_F_MULTI : 0;
- const struct nf_conntrack_l4proto *l4proto = timeout->timeout.l4proto;
+ const struct nf_conntrack_l4proto *l4proto = timeout->timeout->l4proto;
struct nlattr *nest_parms;
int ret;
@@ -198,17 +204,17 @@ ctnl_timeout_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type,
if (nla_put_string(skb, CTA_TIMEOUT_NAME, timeout->name) ||
nla_put_be16(skb, CTA_TIMEOUT_L3PROTO,
- htons(timeout->timeout.l3num)) ||
+ htons(timeout->timeout->l3num)) ||
nla_put_u8(skb, CTA_TIMEOUT_L4PROTO, l4proto->l4proto) ||
nla_put_be32(skb, CTA_TIMEOUT_USE,
- htonl(refcount_read(&timeout->refcnt))))
+ htonl(refcount_read(&timeout->timeout->refcnt))))
goto nla_put_failure;
nest_parms = nla_nest_start(skb, CTA_TIMEOUT_DATA);
if (!nest_parms)
goto nla_put_failure;
- ret = l4proto->ctnl_timeout.obj_to_nlattr(skb, &timeout->timeout.data);
+ ret = l4proto->ctnl_timeout.obj_to_nlattr(skb, &timeout->timeout->data);
if (ret < 0)
goto nla_put_failure;
@@ -308,23 +314,17 @@ static int cttimeout_get_timeout(struct sk_buff *skb,
return ret;
}
-/* try to delete object, fail if it is still in use. */
-static int ctnl_timeout_try_del(struct net *net, struct ctnl_timeout *timeout)
+static void ctnl_timeout_del(struct net *net, struct ctnl_timeout *timeout)
{
- int ret = 0;
+ /* We are protected by nfnl mutex. */
+ list_del_rcu(&timeout->head);
+ nf_ct_untimeout(net, timeout->timeout);
- /* We want to avoid races with ctnl_timeout_put. So only when the
- * current refcnt is 1, we decrease it to 0.
- */
- if (refcount_dec_if_one(&timeout->refcnt)) {
- /* We are protected by nfnl mutex. */
- list_del_rcu(&timeout->head);
- nf_ct_untimeout(net, &timeout->timeout);
- kfree_rcu(timeout, rcu_head);
- } else {
- ret = -EBUSY;
- }
- return ret;
+ if (refcount_dec_and_test(&timeout->timeout->refcnt))
+ kfree_rcu(timeout->timeout, rcu);
+
+ kfree_rcu(timeout, rcu_head);
+ module_put(THIS_MODULE);
}
static int cttimeout_del_timeout(struct sk_buff *skb,
@@ -339,7 +339,7 @@ static int cttimeout_del_timeout(struct sk_buff *skb,
if (!cda[CTA_TIMEOUT_NAME]) {
list_for_each_entry_safe(cur, tmp, &pernet->nfct_timeout_list,
head)
- ctnl_timeout_try_del(info->net, cur);
+ ctnl_timeout_del(info->net, cur);
return 0;
}
@@ -349,10 +349,8 @@ static int cttimeout_del_timeout(struct sk_buff *skb,
if (strncmp(cur->name, name, CTNL_TIMEOUT_NAME_MAX) != 0)
continue;
- ret = ctnl_timeout_try_del(info->net, cur);
- if (ret < 0)
- return ret;
-
+ ctnl_timeout_del(info->net, cur);
+ ret = 0;
break;
}
return ret;
@@ -518,24 +516,22 @@ static struct nf_ct_timeout *ctnl_timeout_find_get(struct net *net,
if (strncmp(timeout->name, name, CTNL_TIMEOUT_NAME_MAX) != 0)
continue;
- if (!refcount_inc_not_zero(&timeout->refcnt))
+ if (!refcount_inc_not_zero(&timeout->timeout->refcnt))
goto err;
matching = timeout;
+ __module_get(THIS_MODULE);
break;
}
err:
- return matching ? &matching->timeout : NULL;
+ return matching ? matching->timeout : NULL;
}
-static void ctnl_timeout_put(struct nf_ct_timeout *t)
+static void ctnl_timeout_put(struct nf_ct_timeout *timeout)
{
- struct ctnl_timeout *timeout =
- container_of(t, struct ctnl_timeout, timeout);
+ if (refcount_dec_and_test(&timeout->refcnt))
+ kfree_rcu(timeout, rcu);
- if (refcount_dec_and_test(&timeout->refcnt)) {
- kfree_rcu(timeout, rcu_head);
- module_put(THIS_MODULE);
- }
+ module_put(THIS_MODULE);
}
static const struct nfnl_callback cttimeout_cb[IPCTNL_MSG_TIMEOUT_MAX] = {
@@ -616,8 +612,11 @@ static void __net_exit cttimeout_net_exit(struct net *net)
list_for_each_entry_safe(cur, tmp, &pernet->nfct_timeout_freelist, free_head) {
list_del(&cur->free_head);
- if (refcount_dec_and_test(&cur->refcnt))
- kfree_rcu(cur, rcu_head);
+ if (refcount_dec_and_test(&cur->timeout->refcnt))
+ kfree_rcu(cur->timeout, rcu);
+
+ kfree_rcu(cur, rcu_head);
+ module_put(THIS_MODULE);
}
}
@@ -656,24 +655,13 @@ static int __init cttimeout_init(void)
return ret;
}
-static int untimeout(struct nf_conn *ct, void *timeout)
-{
- struct nf_conn_timeout *timeout_ext = nf_ct_timeout_find(ct);
-
- if (timeout_ext)
- RCU_INIT_POINTER(timeout_ext->timeout, NULL);
-
- return 0;
-}
-
static void __exit cttimeout_exit(void)
{
nfnetlink_subsys_unregister(&cttimeout_subsys);
- unregister_pernet_subsys(&cttimeout_ops);
RCU_INIT_POINTER(nf_ct_timeout_hook, NULL);
-
- nf_ct_iterate_destroy(untimeout, NULL);
+ synchronize_net();
+ unregister_pernet_subsys(&cttimeout_ops);
}
module_init(cttimeout_init);
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 2e00a07eb545..797c99c626d2 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -1338,6 +1338,7 @@ nfqnl_rcv_nl_event(struct notifier_block *this,
if (event == NETLINK_URELEASE && n->protocol == NETLINK_NETFILTER) {
int i;
+ nfnl_lock(NFNL_SUBSYS_QUEUE);
/* destroy all instances for this portid */
spin_lock(&q->instances_lock);
for (i = 0; i < INSTANCE_BUCKETS; i++) {
@@ -1351,6 +1352,7 @@ nfqnl_rcv_nl_event(struct notifier_block *this,
}
}
spin_unlock(&q->instances_lock);
+ nfnl_unlock(NFNL_SUBSYS_QUEUE);
}
return NOTIFY_DONE;
}
@@ -1672,9 +1674,9 @@ static int nfqnl_recv_config(struct sk_buff *skb, const struct nfnl_info *info,
/* Lookup queue under RCU. After peer_portid check (or for new queue
* in BIND case), the queue is owned by the socket sending this message.
- * A socket cannot simultaneously send a message and close, so while
- * processing this CONFIG message, nfqnl_rcv_nl_event() (triggered by
- * socket close) cannot destroy this queue. Safe to use without RCU.
+ * nfqnl_rcv_nl_event() will block on the nfnl subsys mutex that is
+ * held by the caller, so the queue cannot be destroyed in parallel,
+ * even after we drop the RCU read lock.
*/
rcu_read_lock();
queue = instance_lookup(q, queue_num);
diff --git a/net/netfilter/nft_chain_filter.c b/net/netfilter/nft_chain_filter.c
index 7010541fcca6..19a553550c76 100644
--- a/net/netfilter/nft_chain_filter.c
+++ b/net/netfilter/nft_chain_filter.c
@@ -319,37 +319,21 @@ static const struct nft_chain_type nft_chain_filter_netdev = {
};
static void nft_netdev_event(unsigned long event, struct net_device *dev,
- struct nft_ctx *ctx)
+ struct nft_base_chain *basechain)
{
- struct nft_base_chain *basechain = nft_base_chain(ctx->chain);
- struct nft_hook *hook, *found = NULL;
- int n = 0;
+ struct nft_hook *hook;
list_for_each_entry(hook, &basechain->hook_list, list) {
- if (hook->ops.dev == dev)
- found = hook;
-
- n++;
- }
- if (!found)
- return;
+ if (hook->ops.dev != dev)
+ continue;
- if (n > 1) {
- if (!(ctx->chain->table->flags & NFT_TABLE_F_DORMANT))
- nf_unregister_net_hook(ctx->net, &found->ops);
+ if (!(basechain->chain.table->flags & NFT_TABLE_F_DORMANT))
+ nf_unregister_net_hook(dev_net(dev), &hook->ops);
- list_del_rcu(&found->list);
- kfree_rcu(found, rcu);
- return;
+ list_del_rcu(&hook->list);
+ kfree_rcu(hook, rcu);
+ break;
}
-
- /* UNREGISTER events are also happening on netns exit.
- *
- * Although nf_tables core releases all tables/chains, only this event
- * handler provides guarantee that hook->ops.dev is still accessible,
- * so we cannot skip exiting net namespaces.
- */
- __nft_release_basechain(ctx);
}
static int nf_tables_netdev_event(struct notifier_block *this,
@@ -358,25 +342,20 @@ static int nf_tables_netdev_event(struct notifier_block *this,
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
struct nft_base_chain *basechain;
struct nftables_pernet *nft_net;
- struct nft_chain *chain, *nr;
+ struct nft_chain *chain;
struct nft_table *table;
- struct nft_ctx ctx = {
- .net = dev_net(dev),
- };
if (event != NETDEV_UNREGISTER)
return NOTIFY_DONE;
- nft_net = nft_pernet(ctx.net);
+ nft_net = nft_pernet(dev_net(dev));
mutex_lock(&nft_net->commit_mutex);
list_for_each_entry(table, &nft_net->tables, list) {
if (table->family != NFPROTO_NETDEV &&
table->family != NFPROTO_INET)
continue;
- ctx.family = table->family;
- ctx.table = table;
- list_for_each_entry_safe(chain, nr, &table->chains, list) {
+ list_for_each_entry(chain, &table->chains, list) {
if (!nft_is_base_chain(chain))
continue;
@@ -385,8 +364,7 @@ static int nf_tables_netdev_event(struct notifier_block *this,
basechain->ops.hooknum != NF_INET_INGRESS)
continue;
- ctx.chain = chain;
- nft_netdev_event(event, dev, &ctx);
+ nft_netdev_event(event, dev, basechain);
}
}
mutex_unlock(&nft_net->commit_mutex);
diff --git a/net/netfilter/nft_ct.c b/net/netfilter/nft_ct.c
index 277fc60c3372..8feea8b2422b 100644
--- a/net/netfilter/nft_ct.c
+++ b/net/netfilter/nft_ct.c
@@ -879,8 +879,6 @@ static void nft_ct_timeout_obj_eval(struct nft_object *obj,
}
}
- rcu_assign_pointer(timeout->timeout, priv->timeout);
-
/* adjust the timeout as per 'new' state. ct is unconfirmed,
* so the current timestamp must not be added.
*/
@@ -931,6 +929,7 @@ static int nft_ct_timeout_obj_init(const struct nft_ctx *ctx,
timeout->l3num = l3num;
timeout->l4proto = l4proto;
+ refcount_set(&timeout->refcnt, 1);
ret = nf_ct_netns_get(ctx->net, ctx->family);
if (ret < 0)
@@ -951,10 +950,10 @@ static void nft_ct_timeout_obj_destroy(const struct nft_ctx *ctx,
struct nft_ct_timeout_obj *priv = nft_obj_data(obj);
struct nf_ct_timeout *timeout = priv->timeout;
- nf_queue_nf_hook_drop(ctx->net);
nf_ct_untimeout(ctx->net, timeout);
nf_ct_netns_put(ctx->net, ctx->family);
- kfree_rcu(priv->timeout, rcu);
+ if (refcount_dec_and_test(&timeout->refcnt))
+ kfree_rcu(priv->timeout, rcu);
}
static int nft_ct_timeout_obj_dump(struct sk_buff *skb,
diff --git a/net/netfilter/nft_nat.c b/net/netfilter/nft_nat.c
index e32cd9fbc7c2..cdbd800cac96 100644
--- a/net/netfilter/nft_nat.c
+++ b/net/netfilter/nft_nat.c
@@ -64,8 +64,8 @@ static void nft_nat_setup_netmap(struct nf_nat_range2 *range,
const struct nft_pktinfo *pkt,
const struct nft_nat *priv)
{
+ union nf_inet_addr new_addr = {};
struct sk_buff *skb = pkt->skb;
- union nf_inet_addr new_addr;
__be32 netmask;
int i, len = 0;
diff --git a/net/netfilter/nft_synproxy.c b/net/netfilter/nft_synproxy.c
index a80bdce38546..a6253595a167 100644
--- a/net/netfilter/nft_synproxy.c
+++ b/net/netfilter/nft_synproxy.c
@@ -117,7 +117,8 @@ static void nft_synproxy_do_eval(const struct nft_synproxy *priv,
return;
}
- if (nf_ip_checksum(skb, nft_hook(pkt), thoff, IPPROTO_TCP)) {
+ if (nf_checksum(skb, nft_hook(pkt), thoff, IPPROTO_TCP,
+ nft_pf(pkt))) {
regs->verdict.code = NF_DROP;
return;
}
diff --git a/net/netfilter/xt_CT.c b/net/netfilter/xt_CT.c
index 498f5871c84a..90a295cf5ec6 100644
--- a/net/netfilter/xt_CT.c
+++ b/net/netfilter/xt_CT.c
@@ -284,7 +284,7 @@ static void xt_ct_tg_destroy(const struct xt_tgdtor_param *par,
struct nf_conn_help *help;
if (ct) {
- if (info->helper[0] || info->timeout[0])
+ if (info->helper[0])
nf_queue_nf_hook_drop(par->net);
help = nfct_help(ct);
diff --git a/net/netlink/genetlink.c b/net/netlink/genetlink.c
index 3327d8451814..338e02f5a96a 100644
--- a/net/netlink/genetlink.c
+++ b/net/netlink/genetlink.c
@@ -1514,6 +1514,7 @@ struct ctrl_dump_policy_ctx {
struct netlink_policy_dump_state *state;
const struct genl_family *rt;
struct genl_op_iter *op_iter;
+ struct module *owner;
u32 op;
u16 fam_id;
u8 dump_map:1,
@@ -1556,6 +1557,9 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
return -ENOENT;
ctx->rt = rt;
+ ctx->owner = rt->module;
+ if (!try_module_get(ctx->owner))
+ return -ENOENT;
if (tb[CTRL_ATTR_OP]) {
struct genl_split_ops doit, dump;
@@ -1566,7 +1570,7 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
err = genl_get_cmd_both(ctx->op, rt, &doit, &dump);
if (err) {
NL_SET_BAD_ATTR(cb->extack, tb[CTRL_ATTR_OP]);
- return err;
+ goto err_put_owner;
}
if (doit.policy) {
@@ -1584,16 +1588,20 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
goto err_free_state;
}
- if (!ctx->state)
- return -ENODATA;
+ if (!ctx->state) {
+ err = -ENODATA;
+ goto err_put_owner;
+ }
ctx->dump_map = 1;
return 0;
}
ctx->op_iter = kmalloc(sizeof(*ctx->op_iter), GFP_KERNEL);
- if (!ctx->op_iter)
- return -ENOMEM;
+ if (!ctx->op_iter) {
+ err = -ENOMEM;
+ goto err_put_owner;
+ }
genl_op_iter_init(rt, ctx->op_iter);
ctx->dump_map = genl_op_iter_next(ctx->op_iter);
@@ -1625,6 +1633,8 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
netlink_policy_dump_free(ctx->state);
err_free_op_iter:
kfree(ctx->op_iter);
+err_put_owner:
+ module_put(ctx->owner);
return err;
}
@@ -1647,7 +1657,7 @@ static void *ctrl_dumppolicy_prep(struct sk_buff *skb,
}
static int ctrl_dumppolicy_put_op(struct sk_buff *skb,
- struct netlink_callback *cb,
+ struct netlink_callback *cb, u32 cmd,
struct genl_split_ops *doit,
struct genl_split_ops *dumpit)
{
@@ -1668,7 +1678,7 @@ static int ctrl_dumppolicy_put_op(struct sk_buff *skb,
if (!nest_pol)
goto err;
- nest_op = nla_nest_start(skb, doit->cmd);
+ nest_op = nla_nest_start(skb, cmd);
if (!nest_op)
goto err;
@@ -1712,7 +1722,8 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)
&doit, &dumpit)))
return -ENOENT;
- if (ctrl_dumppolicy_put_op(skb, cb, &doit, &dumpit))
+ if (ctrl_dumppolicy_put_op(skb, cb, ctx->op,
+ &doit, &dumpit))
return skb->len;
/* done with the per-op policy index list */
@@ -1721,6 +1732,7 @@ static int ctrl_dumppolicy(struct sk_buff *skb, struct netlink_callback *cb)
while (ctx->dump_map) {
if (ctrl_dumppolicy_put_op(skb, cb,
+ ctx->op_iter->cmd,
&ctx->op_iter->doit,
&ctx->op_iter->dumpit))
return skb->len;
@@ -1761,6 +1773,7 @@ static int ctrl_dumppolicy_done(struct netlink_callback *cb)
kfree(ctx->op_iter);
netlink_policy_dump_free(ctx->state);
+ module_put(ctx->owner);
return 0;
}
diff --git a/net/nfc/core.c b/net/nfc/core.c
index 96dc0e678601..7e2e2cf0c3d1 100644
--- a/net/nfc/core.c
+++ b/net/nfc/core.c
@@ -279,10 +279,10 @@ static struct nfc_target *nfc_find_target(struct nfc_dev *dev, u32 target_idx)
int nfc_dep_link_up(struct nfc_dev *dev, int target_index, u8 comm_mode)
{
- int rc = 0;
- u8 *gb;
- size_t gb_len;
struct nfc_target *target;
+ u8 gb[NFC_MAX_GT_LEN];
+ size_t gb_len = 0;
+ int rc = 0;
pr_debug("dev_name=%s comm %d\n", dev_name(&dev->dev), comm_mode);
@@ -301,7 +301,7 @@ int nfc_dep_link_up(struct nfc_dev *dev, int target_index, u8 comm_mode)
goto error;
}
- gb = nfc_llcp_general_bytes(dev, &gb_len);
+ nfc_get_local_general_bytes(dev, gb, sizeof(gb), &gb_len);
if (gb_len > NFC_MAX_GT_LEN) {
rc = -EINVAL;
goto error;
@@ -644,11 +644,10 @@ int nfc_set_remote_general_bytes(struct nfc_dev *dev, const u8 *gb, u8 gb_len)
}
EXPORT_SYMBOL(nfc_set_remote_general_bytes);
-u8 *nfc_get_local_general_bytes(struct nfc_dev *dev, size_t *gb_len)
+u8 *nfc_get_local_general_bytes(struct nfc_dev *dev, u8 *out_gb,
+ size_t gb_max_len, size_t *gb_len)
{
- pr_debug("dev_name=%s\n", dev_name(&dev->dev));
-
- return nfc_llcp_general_bytes(dev, gb_len);
+ return nfc_llcp_general_bytes(dev, out_gb, gb_max_len, gb_len);
}
EXPORT_SYMBOL(nfc_get_local_general_bytes);
diff --git a/net/nfc/digital_dep.c b/net/nfc/digital_dep.c
index 3982fa084737..968547c306a5 100644
--- a/net/nfc/digital_dep.c
+++ b/net/nfc/digital_dep.c
@@ -1490,14 +1490,14 @@ static int digital_tg_send_atr_res(struct nfc_digital_dev *ddev,
struct digital_atr_req *atr_req)
{
struct digital_atr_res *atr_res;
+ u8 gb[NFC_MAX_GT_LEN];
struct sk_buff *skb;
- u8 *gb, payload_bits;
+ u8 payload_bits;
size_t gb_len;
int rc;
- gb = nfc_get_local_general_bytes(ddev->nfc_dev, &gb_len);
- if (!gb)
- gb_len = 0;
+ nfc_get_local_general_bytes(ddev->nfc_dev, gb, sizeof(gb),
+ &gb_len);
skb = digital_skb_alloc(ddev, sizeof(struct digital_atr_res) + gb_len);
if (!skb)
diff --git a/net/nfc/llcp.h b/net/nfc/llcp.h
index d8345ed57c95..23ae7a0112d3 100644
--- a/net/nfc/llcp.h
+++ b/net/nfc/llcp.h
@@ -91,6 +91,7 @@ struct nfc_llcp_local {
struct hlist_head pending_sdreqs;
struct timer_list sdreq_timer;
struct work_struct sdreq_timeout_work;
+ struct work_struct release_work;
u8 sdreq_next_tid;
/* sockets array */
diff --git a/net/nfc/llcp_commands.c b/net/nfc/llcp_commands.c
index a93bf0b43d50..2d9e0925b1cc 100644
--- a/net/nfc/llcp_commands.c
+++ b/net/nfc/llcp_commands.c
@@ -135,7 +135,7 @@ struct nfc_llcp_sdp_tlv *nfc_llcp_build_sdreq_tlv(u8 tid, const char *uri,
{
struct nfc_llcp_sdp_tlv *sdreq;
- pr_debug("uri: %s, len: %zu\n", uri, uri_len);
+ pr_debug("uri: %.*s, len: %zu\n", (int)uri_len, uri, uri_len);
/* sdreq->tlv_len is u8, takes uri_len, + 3 for header, + 1 for NULL */
if (WARN_ON_ONCE(uri_len > U8_MAX - 4))
diff --git a/net/nfc/llcp_core.c b/net/nfc/llcp_core.c
index 7b86faf3f4ce..15b9b736319e 100644
--- a/net/nfc/llcp_core.c
+++ b/net/nfc/llcp_core.c
@@ -20,6 +20,8 @@ static LIST_HEAD(llcp_devices);
/* Protects llcp_devices list */
static DEFINE_SPINLOCK(llcp_devices_lock);
+static struct workqueue_struct *llcp_wq;
+
static void nfc_llcp_rx_skb(struct nfc_llcp_local *local, struct sk_buff *skb);
void nfc_llcp_sock_link(struct llcp_sock_list *l, struct sock *sk)
@@ -63,21 +65,33 @@ static void nfc_llcp_socket_purge(struct nfc_llcp_sock *sock)
}
}
+static struct sock *nfc_llcp_sock_list_pop(struct llcp_sock_list *l)
+{
+ struct sock *sk;
+
+ write_lock(&l->lock);
+ sk = sk_head(&l->head);
+ if (sk) {
+ sock_hold(sk);
+ sk_del_node_init(sk);
+ }
+ write_unlock(&l->lock);
+
+ return sk;
+}
+
static void nfc_llcp_socket_release(struct nfc_llcp_local *local, bool device,
int err)
{
struct sock *sk;
- struct hlist_node *tmp;
struct nfc_llcp_sock *llcp_sock;
skb_queue_purge(&local->tx_queue);
- write_lock(&local->sockets.lock);
-
- sk_for_each_safe(sk, tmp, &local->sockets.head) {
+ while ((sk = nfc_llcp_sock_list_pop(&local->sockets))) {
llcp_sock = nfc_llcp_sock(sk);
- bh_lock_sock(sk);
+ lock_sock(sk);
nfc_llcp_socket_purge(llcp_sock);
@@ -91,17 +105,27 @@ static void nfc_llcp_socket_release(struct nfc_llcp_local *local, bool device,
list_for_each_entry_safe(lsk, n,
&llcp_sock->accept_queue,
accept_queue) {
- accept_sk = &lsk->sk;
- bh_lock_sock(accept_sk);
+ bool put_creation = false;
- nfc_llcp_accept_unlink(accept_sk);
-
- if (err)
- accept_sk->sk_err = err;
- accept_sk->sk_state = LLCP_CLOSED;
- accept_sk->sk_state_change(sk);
+ accept_sk = &lsk->sk;
+ lock_sock_nested(accept_sk,
+ SINGLE_DEPTH_NESTING);
+
+ if (nfc_llcp_sock(accept_sk)->parent == sk) {
+ nfc_llcp_accept_unlink(accept_sk);
+ nfc_llcp_sock_unlink(&local->sockets, accept_sk);
+
+ if (err)
+ accept_sk->sk_err = err;
+ accept_sk->sk_state = LLCP_CLOSED;
+ accept_sk->sk_state_change(accept_sk);
+ sock_orphan(accept_sk);
+ put_creation = true;
+ }
- bh_unlock_sock(accept_sk);
+ release_sock(accept_sk);
+ if (put_creation)
+ sock_put(accept_sk); /* creation ref */
}
}
@@ -110,23 +134,18 @@ static void nfc_llcp_socket_release(struct nfc_llcp_local *local, bool device,
sk->sk_state = LLCP_CLOSED;
sk->sk_state_change(sk);
- bh_unlock_sock(sk);
-
- sk_del_node_init(sk);
+ release_sock(sk);
+ sock_put(sk);
}
- write_unlock(&local->sockets.lock);
-
/* If we still have a device, we keep the RAW sockets alive */
if (device == true)
return;
- write_lock(&local->raw_sockets.lock);
-
- sk_for_each_safe(sk, tmp, &local->raw_sockets.head) {
+ while ((sk = nfc_llcp_sock_list_pop(&local->raw_sockets))) {
llcp_sock = nfc_llcp_sock(sk);
- bh_lock_sock(sk);
+ lock_sock(sk);
nfc_llcp_socket_purge(llcp_sock);
@@ -135,26 +154,20 @@ static void nfc_llcp_socket_release(struct nfc_llcp_local *local, bool device,
sk->sk_state = LLCP_CLOSED;
sk->sk_state_change(sk);
- bh_unlock_sock(sk);
-
- sk_del_node_init(sk);
+ release_sock(sk);
+ sock_put(sk);
}
-
- write_unlock(&local->raw_sockets.lock);
}
static struct nfc_llcp_local *nfc_llcp_local_get(struct nfc_llcp_local *local)
{
- /* Since using nfc_llcp_local may result in usage of nfc_dev, whenever
- * we hold a reference to local, we also need to hold a reference to
- * the device to avoid UAF.
- */
- if (!nfc_get_device(local->dev->idx))
+ if (!local)
return NULL;
- kref_get(&local->ref);
+ if (kref_get_unless_zero(&local->ref))
+ return local;
- return local;
+ return NULL;
}
static void local_cleanup(struct nfc_llcp_local *local)
@@ -172,30 +185,34 @@ static void local_cleanup(struct nfc_llcp_local *local)
nfc_llcp_free_sdp_tlv_list(&local->pending_sdreqs);
}
-static void local_release(struct kref *ref)
+static void local_release_work(struct work_struct *work)
{
struct nfc_llcp_local *local;
+ struct nfc_dev *dev;
- local = container_of(ref, struct nfc_llcp_local, ref);
+ local = container_of(work, struct nfc_llcp_local, release_work);
+ dev = local->dev;
local_cleanup(local);
kfree(local);
+ nfc_put_device(dev);
}
-int nfc_llcp_local_put(struct nfc_llcp_local *local)
+static void local_release(struct kref *ref)
{
- struct nfc_dev *dev;
- int ret;
+ struct nfc_llcp_local *local;
- if (local == NULL)
- return 0;
+ local = container_of(ref, struct nfc_llcp_local, ref);
- dev = local->dev;
+ queue_work(llcp_wq, &local->release_work);
+}
- ret = kref_put(&local->ref, local_release);
- nfc_put_device(dev);
+int nfc_llcp_local_put(struct nfc_llcp_local *local)
+{
+ if (!local)
+ return 0;
- return ret;
+ return kref_put(&local->ref, local_release);
}
static struct nfc_llcp_sock *nfc_llcp_sock_get(struct nfc_llcp_local *local,
@@ -339,7 +356,7 @@ static int nfc_llcp_wks_sap(const char *service_name, size_t service_name_len)
{
int sap, num_wks;
- pr_debug("%s\n", service_name);
+ pr_debug("%.*s\n", (int)service_name_len, service_name);
if (service_name == NULL)
return -EINVAL;
@@ -350,7 +367,8 @@ static int nfc_llcp_wks_sap(const char *service_name, size_t service_name_len)
if (wks[sap] == NULL)
continue;
- if (strncmp(wks[sap], service_name, service_name_len) == 0)
+ if (strlen(wks[sap]) == service_name_len &&
+ !strncmp(wks[sap], service_name, service_name_len))
return sap;
}
@@ -633,23 +651,32 @@ static int nfc_llcp_build_gb(struct nfc_llcp_local *local)
return ret;
}
-u8 *nfc_llcp_general_bytes(struct nfc_dev *dev, size_t *general_bytes_len)
+u8 *nfc_llcp_general_bytes(struct nfc_dev *dev, u8 *out_gb, size_t gb_max_len,
+ size_t *general_bytes_len)
{
struct nfc_llcp_local *local;
+ if (!out_gb || !general_bytes_len)
+ return NULL;
+
local = nfc_llcp_find_local(dev);
- if (local == NULL) {
+ if (!local) {
*general_bytes_len = 0;
return NULL;
}
nfc_llcp_build_gb(local);
- *general_bytes_len = local->gb_len;
+ if (local->gb_len) {
+ *general_bytes_len = min_t(size_t, local->gb_len, gb_max_len);
+ memcpy(out_gb, local->gb, *general_bytes_len);
+ } else {
+ *general_bytes_len = 0;
+ }
nfc_llcp_local_put(local);
- return local->gb;
+ return out_gb;
}
int nfc_llcp_set_remote_gb(struct nfc_dev *dev, const u8 *gb, u8 gb_len)
@@ -1072,6 +1099,9 @@ static void nfc_llcp_recv_hdlc(struct nfc_llcp_local *local,
struct sock *sk;
u8 dsap, ssap, ptype, ns, nr;
+ if (!pskb_may_pull(skb, LLCP_HEADER_SIZE + LLCP_SEQUENCE_SIZE))
+ return;
+
ptype = nfc_llcp_ptype(skb);
dsap = nfc_llcp_dsap(skb);
ssap = nfc_llcp_ssap(skb);
@@ -1249,6 +1279,7 @@ static void nfc_llcp_recv_dm(struct nfc_llcp_local *local,
struct nfc_llcp_sock *llcp_sock;
struct sock *sk;
u8 dsap, ssap, reason;
+ bool connecting = false;
dsap = nfc_llcp_dsap(skb);
ssap = nfc_llcp_ssap(skb);
@@ -1260,6 +1291,7 @@ static void nfc_llcp_recv_dm(struct nfc_llcp_local *local,
case LLCP_DM_NOBOUND:
case LLCP_DM_REJ:
llcp_sock = nfc_llcp_connecting_sock_get(local, dsap);
+ connecting = true;
break;
default:
@@ -1274,10 +1306,33 @@ static void nfc_llcp_recv_dm(struct nfc_llcp_local *local,
sk = &llcp_sock->sk;
+ lock_sock(sk);
+
+ /* Check if socket was destroyed whilst waiting for the lock */
+ if (!sk_hashed(sk)) {
+ release_sock(sk);
+ nfc_llcp_sock_put(llcp_sock);
+ return;
+ }
+
+ /*
+ * For DM(NOBOUND)/DM(REJ) the socket is still linked on the
+ * connecting_sockets list. Unlink it here, under the socket lock,
+ * before moving it to LLCP_CLOSED: llcp_sock_release() selects the
+ * list to unlink from by sk_state, so leaving a connecting socket
+ * in the CLOSED state would make it unlink from the wrong list and
+ * corrupt the connecting_sockets list / desync the socket refcount.
+ * This mirrors nfc_llcp_recv_cc().
+ */
+ if (connecting)
+ nfc_llcp_sock_unlink(&local->connecting_sockets, sk);
+
sk->sk_err = ENXIO;
sk->sk_state = LLCP_CLOSED;
sk->sk_state_change(sk);
+ release_sock(sk);
+
nfc_llcp_sock_put(llcp_sock);
}
@@ -1678,6 +1733,7 @@ int nfc_llcp_register_device(struct nfc_dev *ndev)
INIT_WORK(&local->rx_work, nfc_llcp_rx_work);
INIT_WORK(&local->timeout_work, nfc_llcp_timeout_work);
+ INIT_WORK(&local->release_work, local_release_work);
rwlock_init(&local->sockets.lock);
rwlock_init(&local->connecting_sockets.lock);
@@ -1721,10 +1777,23 @@ void nfc_llcp_unregister_device(struct nfc_dev *dev)
int __init nfc_llcp_init(void)
{
- return nfc_llcp_sock_init();
+ int ret;
+
+ llcp_wq = alloc_workqueue("nfc_llcp_wq", WQ_UNBOUND, 0);
+ if (!llcp_wq)
+ return -ENOMEM;
+
+ ret = nfc_llcp_sock_init();
+ if (ret) {
+ destroy_workqueue(llcp_wq);
+ return ret;
+ }
+
+ return 0;
}
void nfc_llcp_exit(void)
{
nfc_llcp_sock_exit();
+ destroy_workqueue(llcp_wq);
}
diff --git a/net/nfc/llcp_sock.c b/net/nfc/llcp_sock.c
index 507447bbceea..0601ddb4304a 100644
--- a/net/nfc/llcp_sock.c
+++ b/net/nfc/llcp_sock.c
@@ -392,11 +392,12 @@ void nfc_llcp_accept_unlink(struct sock *sk)
pr_debug("state %d\n", sk->sk_state);
- list_del_init(&llcp_sock->accept_queue);
- sk_acceptq_removed(llcp_sock->parent);
- llcp_sock->parent = NULL;
-
- sock_put(sk);
+ if (llcp_sock->parent) {
+ list_del_init(&llcp_sock->accept_queue);
+ sk_acceptq_removed(llcp_sock->parent);
+ llcp_sock->parent = NULL;
+ sock_put(sk);
+ }
}
void nfc_llcp_accept_enqueue(struct sock *parent, struct sock *sk)
@@ -423,12 +424,20 @@ struct sock *nfc_llcp_accept_dequeue(struct sock *parent,
list_for_each_entry_safe(lsk, n, &llcp_parent->accept_queue,
accept_queue) {
+ struct nfc_llcp_local *local;
+
sk = &lsk->sk;
- lock_sock(sk);
+ lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
if (sk->sk_state == LLCP_CLOSED) {
- release_sock(sk);
+ local = nfc_llcp_sock(sk)->local;
+
nfc_llcp_accept_unlink(sk);
+ if (local)
+ nfc_llcp_sock_unlink(&local->sockets, sk);
+ sock_orphan(sk);
+ release_sock(sk);
+ sock_put(sk);
continue;
}
@@ -464,7 +473,7 @@ static int llcp_sock_accept(struct socket *sock, struct socket *newsock,
pr_debug("parent %p\n", sk);
- lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+ lock_sock(sk);
if (sk->sk_state != LLCP_LISTEN) {
ret = -EBADFD;
@@ -490,7 +499,12 @@ static int llcp_sock_accept(struct socket *sock, struct socket *newsock,
release_sock(sk);
timeo = schedule_timeout(timeo);
- lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
+ lock_sock(sk);
+
+ if (sk->sk_state != LLCP_LISTEN) {
+ ret = -EBADFD;
+ break;
+ }
}
__set_current_state(TASK_RUNNING);
remove_wait_queue(sk_sleep(sk), &wait);
@@ -629,13 +643,24 @@ static int llcp_sock_release(struct socket *sock)
list_for_each_entry_safe(lsk, n, &llcp_sock->accept_queue,
accept_queue) {
+ bool put_creation = false;
+
accept_sk = &lsk->sk;
- lock_sock(accept_sk);
+ lock_sock_nested(accept_sk, SINGLE_DEPTH_NESTING);
+
+ if (nfc_llcp_sock(accept_sk)->parent == sk) {
+ nfc_llcp_send_disconnect(lsk);
+ nfc_llcp_accept_unlink(accept_sk);
+ nfc_llcp_sock_unlink(&local->sockets, accept_sk);
- nfc_llcp_send_disconnect(lsk);
- nfc_llcp_accept_unlink(accept_sk);
+ accept_sk->sk_state = LLCP_CLOSED;
+ sock_orphan(accept_sk);
+ put_creation = true;
+ }
release_sock(accept_sk);
+ if (put_creation)
+ sock_put(accept_sk); /* creation ref */
}
}
@@ -734,12 +759,16 @@ static int llcp_sock_connect(struct socket *sock, struct sockaddr *_addr,
llcp_sock->service_name_len = min_t(unsigned int,
addr->service_name_len,
NFC_LLCP_MAX_SERVICE_NAME);
- llcp_sock->service_name = kmemdup(addr->service_name,
- llcp_sock->service_name_len,
- GFP_KERNEL);
- if (!llcp_sock->service_name) {
- ret = -ENOMEM;
- goto sock_llcp_release;
+ if (llcp_sock->service_name_len == 0) {
+ llcp_sock->service_name = NULL;
+ } else {
+ llcp_sock->service_name = kmemdup(addr->service_name,
+ llcp_sock->service_name_len,
+ GFP_KERNEL);
+ if (!llcp_sock->service_name) {
+ ret = -ENOMEM;
+ goto sock_llcp_release;
+ }
}
nfc_llcp_sock_link(&local->connecting_sockets, sk);
diff --git a/net/nfc/nci/core.c b/net/nfc/nci/core.c
index 058d4eb530fb..be081bc048e2 100644
--- a/net/nfc/nci/core.c
+++ b/net/nfc/nci/core.c
@@ -772,15 +772,15 @@ static int nci_set_local_general_bytes(struct nfc_dev *nfc_dev)
{
struct nci_dev *ndev = nfc_get_drvdata(nfc_dev);
struct nci_set_config_param param;
+ u8 gb[NFC_MAX_GT_LEN];
int rc;
- param.val = nfc_get_local_general_bytes(nfc_dev, ¶m.len);
- if ((param.val == NULL) || (param.len == 0))
+ nfc_get_local_general_bytes(nfc_dev, gb, sizeof(gb),
+ ¶m.len);
+ if (param.len == 0)
return 0;
- if (param.len > NFC_MAX_GT_LEN)
- return -EINVAL;
-
+ param.val = gb;
param.id = NCI_PN_ATR_REQ_GEN_BYTES;
rc = nci_request(ndev, nci_set_config_req, ¶m,
diff --git a/net/nfc/netlink.c b/net/nfc/netlink.c
index dd2ce73a24fb..ce18dda58a83 100644
--- a/net/nfc/netlink.c
+++ b/net/nfc/netlink.c
@@ -1176,7 +1176,7 @@ static int nfc_genl_llc_sdreq(struct sk_buff *skb, struct genl_info *info)
if (rc != 0) {
rc = -EINVAL;
- goto put_local;
+ goto free_list;
}
if (!sdp_attrs[NFC_SDP_ATTR_URI])
@@ -1195,7 +1195,7 @@ static int nfc_genl_llc_sdreq(struct sk_buff *skb, struct genl_info *info)
sdreq = nfc_llcp_build_sdreq_tlv(tid, uri, uri_len);
if (sdreq == NULL) {
rc = -ENOMEM;
- goto put_local;
+ goto free_list;
}
tlvs_len += sdreq->tlv_len;
@@ -1210,6 +1210,9 @@ static int nfc_genl_llc_sdreq(struct sk_buff *skb, struct genl_info *info)
rc = nfc_llcp_send_snl_sdreq(local, &sdreq_list, tlvs_len);
+free_list:
+ nfc_llcp_free_sdp_tlv_list(&sdreq_list);
+
put_local:
nfc_llcp_local_put(local);
diff --git a/net/nfc/nfc.h b/net/nfc/nfc.h
index 0b1e6466f4fb..82c5dfdad10e 100644
--- a/net/nfc/nfc.h
+++ b/net/nfc/nfc.h
@@ -49,7 +49,8 @@ void nfc_llcp_mac_is_up(struct nfc_dev *dev, u32 target_idx,
int nfc_llcp_register_device(struct nfc_dev *dev);
void nfc_llcp_unregister_device(struct nfc_dev *dev);
int nfc_llcp_set_remote_gb(struct nfc_dev *dev, const u8 *gb, u8 gb_len);
-u8 *nfc_llcp_general_bytes(struct nfc_dev *dev, size_t *general_bytes_len);
+u8 *nfc_llcp_general_bytes(struct nfc_dev *dev, u8 *out_gb, size_t gb_max_len,
+ size_t *general_bytes_len);
int nfc_llcp_data_received(struct nfc_dev *dev, struct sk_buff *skb);
struct nfc_llcp_local *nfc_llcp_find_local(struct nfc_dev *dev);
int nfc_llcp_local_put(struct nfc_llcp_local *local);
diff --git a/net/openvswitch/conntrack.c b/net/openvswitch/conntrack.c
index 3f51f1968654..04fa06c6f222 100644
--- a/net/openvswitch/conntrack.c
+++ b/net/openvswitch/conntrack.c
@@ -366,7 +366,7 @@ static struct nf_conn_labels *ovs_ct_get_conn_labels(struct nf_conn *ct)
struct nf_conn_labels *cl;
cl = nf_ct_labels_find(ct);
- if (!cl) {
+ if (!cl && !nf_ct_is_confirmed(ct)) {
nf_ct_labels_ext_add(ct);
cl = nf_ct_labels_find(ct);
}
@@ -734,6 +734,18 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
enum ip_conntrack_info ctinfo;
struct nf_conn *ct;
+ /* If the ct entry is not confirmed and shared with some other skb,
+ * e.g., a cloned one, we can't just modify it with the commit as we
+ * must not modify the extension set. Reset.
+ */
+ if (cached && info->commit) {
+ ct = nf_ct_get(skb, &ctinfo);
+ if (ct && !nf_ct_is_confirmed(ct) && nf_ct_shared(ct)) {
+ nf_reset_ct(skb);
+ cached = false;
+ }
+ }
+
if (!cached) {
struct nf_hook_state state = {
.hook = NF_INET_PRE_ROUTING,
@@ -767,8 +779,6 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
ct = nf_ct_get(skb, &ctinfo);
if (ct) {
- bool add_helper = false;
-
/* Packets starting a new connection must be NATted before the
* helper, so that the helper knows about the NAT. We enforce
* this by delaying both NAT and helper calls for unconfirmed
@@ -800,7 +810,6 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
GFP_ATOMIC);
if (err)
return err;
- add_helper = true;
/* helper installed, add seqadj if NAT is required */
if (info->nat && !nfct_seqadj(ct)) {
@@ -809,14 +818,14 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
}
}
- /* Call the helper only if:
- * - nf_conntrack_in() was executed above ("!cached") or a
- * helper was just attached ("add_helper") for a confirmed
- * connection, or
- * - When committing an unconfirmed connection.
+ /* Call the helper only if nf_conntrack_in() was executed
+ * above ("!cached").
+ *
+ * For unconfirmed connections it will be called later during
+ * commit as we need to have all the other extensions allocated
+ * before the call.
*/
- if ((nf_ct_is_confirmed(ct) ? !cached || add_helper :
- info->commit)) {
+ if (nf_ct_is_confirmed(ct) && !cached) {
int err = nf_ct_helper(skb, ct, ctinfo, info->family);
err = verdict_to_errno(err);
@@ -1019,6 +1028,14 @@ static int ovs_ct_commit(struct net *net, struct sw_flow_key *key,
return err;
nf_conn_act_ct_ext_add(skb, ct, ctinfo);
+
+ /* Call the helpers now. We couldn't do this before as
+ * all the extensions must be allocated before the call.
+ */
+ err = nf_ct_helper(skb, ct, ctinfo, info->family);
+ err = verdict_to_errno(err);
+ if (err)
+ return err;
} else if (IS_ENABLED(CONFIG_NF_CONNTRACK_LABELS) &&
labels_nonzero(&info->labels.mask)) {
err = ovs_ct_set_labels(ct, key, &info->labels.value,
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index b0021b41b228..45915412fd96 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -636,7 +636,7 @@ static int prb_calc_retire_blk_tmo(struct packet_sock *po,
return DEFAULT_PRB_RETIRE_TOV;
div = ecmd.base.speed / 1000;
- mbits = (blk_size_in_bytes * 8) / (1024 * 1024);
+ mbits = (u64)blk_size_in_bytes * 8 / (1024 * 1024);
if (div)
mbits /= div;
@@ -2436,7 +2436,9 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
virtio_net_hdr_from_skb(skb, h.raw + macoff -
sizeof(struct virtio_net_hdr),
vio_le(), true, 0)) {
- if (po->tp_version == TPACKET_V3)
+ if (po->tp_version <= TPACKET_V2)
+ __clear_bit(slot_id, po->rx_ring.rx_owner_map);
+ else
prb_clear_blk_fill_status(&po->rx_ring);
goto drop_n_account;
}
@@ -2580,26 +2582,6 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
goto drop_n_restore;
}
-static void tpacket_destruct_skb(struct sk_buff *skb)
-{
- struct packet_sock *po = pkt_sk(skb->sk);
-
- if (likely(po->tx_ring.pg_vec)) {
- void *ph;
- __u32 ts;
-
- ph = skb_zcopy_get_nouarg(skb);
-
- ts = __packet_set_timestamp(po, ph, skb);
- __packet_set_status(po, ph, TP_STATUS_AVAILABLE | ts);
-
- packet_dec_pending(&po->tx_ring);
- complete(&po->skb_completion);
- }
-
- sock_wfree(skb);
-}
-
static int __packet_snd_vnet_parse(struct virtio_net_hdr *vnet_hdr, size_t len)
{
if ((vnet_hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) &&
@@ -2639,27 +2621,56 @@ static int packet_snd_vnet_parse(struct msghdr *msg, size_t *len,
return 0;
}
+struct tpacket_uarg {
+ struct ubuf_info ubuf;
+ struct packet_sock *po;
+ void *ph;
+};
+
+static void tpacket_ubuf_complete(struct sk_buff *skb, struct ubuf_info *uarg,
+ bool success)
+{
+ struct tpacket_uarg *tu = container_of(uarg, struct tpacket_uarg, ubuf);
+ struct packet_sock *po = tu->po;
+ void *ph = tu->ph;
+ __u32 ts;
+
+ DEBUG_NET_WARN_ON_ONCE(!skb);
+
+ if (!refcount_dec_and_test(&uarg->refcnt))
+ return;
+
+ ts = __packet_set_timestamp(po, ph, skb);
+ __packet_set_status(po, ph, TP_STATUS_AVAILABLE | ts);
+
+ packet_dec_pending(&po->tx_ring);
+ complete(&po->skb_completion);
+
+ kfree(tu);
+ sk_free(&po->sk);
+}
+
+static const struct ubuf_info_ops tpacket_ubuf_ops = {
+ .complete = tpacket_ubuf_complete,
+};
+
static int tpacket_fill_skb(struct packet_sock *po, struct sk_buff *skb,
- void *frame, struct net_device *dev, void *data, int tp_len,
+ struct net_device *dev, void *data, int tp_len,
__be16 proto, unsigned char *addr, int hlen, int copylen,
int hard_header_len,
const struct sockcm_cookie *sockc)
{
- union tpacket_uhdr ph;
int to_write, offset, len, nr_frags, len_max;
struct socket *sock = po->sk.sk_socket;
struct page *page;
int err;
- ph.raw = frame;
-
skb->protocol = proto;
skb->dev = dev;
skb->priority = READ_ONCE(po->sk.sk_priority);
skb->mark = READ_ONCE(po->sk.sk_mark);
skb_set_delivery_type_by_clockid(skb, sockc->transmit_time, po->sk.sk_clockid);
skb_setup_tx_timestamp(skb, sockc->tsflags);
- skb_zcopy_set_nouarg(skb, ph.raw);
skb_reserve(skb, hlen);
skb_reset_network_header(skb);
@@ -2799,6 +2810,7 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg)
struct virtio_net_hdr vnet_hdr;
bool has_vnet_hdr = false;
struct sockcm_cookie sockc;
+ struct tpacket_uarg *uarg;
__be16 proto;
int err, reserve = 0;
void *ph;
@@ -2926,7 +2938,7 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg)
err = len_sum;
goto out_status;
}
- tp_len = tpacket_fill_skb(po, skb, ph, dev, data, tp_len, proto,
+ tp_len = tpacket_fill_skb(po, skb, dev, data, tp_len, proto,
addr, hlen, copylen, hard_header_len,
&sockc);
if (likely(tp_len >= 0) &&
@@ -2958,7 +2970,24 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg)
virtio_net_hdr_set_proto(skb, &vnet_hdr);
}
- skb->destructor = tpacket_destruct_skb;
+ uarg = kmalloc(sizeof(*uarg), GFP_KERNEL);
+ if (unlikely(!uarg)) {
+ if (likely(len_sum > 0))
+ err = len_sum;
+ else
+ err = -ENOMEM;
+ goto out_status;
+ }
+ uarg->po = po;
+ uarg->ph = ph;
+ uarg->ubuf.ops = &tpacket_ubuf_ops;
+ uarg->ubuf.flags = SKBFL_ZEROCOPY_FRAG;
+ refcount_set(&uarg->ubuf.refcnt, 1);
+
+ /* Hold a sk_wmem_alloc reference until completion */
+ refcount_inc(&po->sk.sk_wmem_alloc);
+ skb_zcopy_init(skb, &uarg->ubuf);
+
__packet_set_status(po, ph, TP_STATUS_SENDING);
packet_inc_pending(&po->tx_ring);
@@ -4527,21 +4556,20 @@ static struct pgv *alloc_pg_vec(struct tpacket_req *req, int order, bool tx_ring
vec->len = block_nr;
pg_vec = vec->pg_vec;
+ if (tx_ring) {
+ vec->deferred = kzalloc_obj(*vec->deferred,
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!vec->deferred)
+ goto out_free_pgvec;
+ vec->deferred->vec = vec;
+ INIT_DELAYED_WORK(&vec->deferred->work,
+ packet_free_pg_vec_work);
+ }
+
for (i = 0; i < block_nr; i++) {
pg_vec[i].buffer = alloc_one_pg_vec_page(order);
if (unlikely(!pg_vec[i].buffer))
goto out_free_pgvec;
-
- if (tx_ring && !vec->deferred &&
- is_vmalloc_addr(pg_vec[i].buffer)) {
- vec->deferred = kzalloc_obj(*vec->deferred,
- GFP_KERNEL | __GFP_NOWARN);
- if (!vec->deferred)
- goto out_free_pgvec;
- vec->deferred->vec = vec;
- INIT_DELAYED_WORK(&vec->deferred->work,
- packet_free_pg_vec_work);
- }
}
out:
diff --git a/net/packet/internal.h b/net/packet/internal.h
index d5d70712007a..f56a31ec20c8 100644
--- a/net/packet/internal.h
+++ b/net/packet/internal.h
@@ -21,7 +21,7 @@ struct tpacket_kbdq_core {
unsigned char reset_pending_on_curr_blk;
unsigned char delete_blk_timer;
unsigned short kactive_blk_num;
- unsigned short blk_sizeof_priv;
+ unsigned int blk_sizeof_priv;
/* last_kactive_blk_num:
* trick to see if user-space has caught up
diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c
index d9b6c9d2f679..1c6e3b3f39d0 100644
--- a/net/rds/ib_cm.c
+++ b/net/rds/ib_cm.c
@@ -115,7 +115,7 @@ void rds_ib_cm_connect_complete(struct rds_connection *conn, struct rdma_cm_even
&conn->c_laddr, &conn->c_faddr,
RDS_PROTOCOL_MAJOR(conn->c_version),
RDS_PROTOCOL_MINOR(conn->c_version));
- rds_conn_destroy(conn);
+ rds_conn_drop(conn);
return;
}
}
diff --git a/net/rds/ib_frmr.c b/net/rds/ib_frmr.c
index bd861191157b..8397aa4a17ac 100644
--- a/net/rds/ib_frmr.c
+++ b/net/rds/ib_frmr.c
@@ -204,19 +204,16 @@ static int rds_ib_map_frmr(struct rds_ib_device *rds_ibdev,
*/
rds_ib_teardown_mr(ibmr);
- ibmr->sg = sg;
- ibmr->sg_len = sg_len;
- ibmr->sg_dma_len = 0;
frmr->sg_byte_len = 0;
- WARN_ON(ibmr->sg_dma_len);
- ibmr->sg_dma_len = ib_dma_map_sg(dev, ibmr->sg, ibmr->sg_len,
+ ibmr->sg_dma_len = ib_dma_map_sg(dev, sg, sg_len,
DMA_BIDIRECTIONAL);
if (unlikely(!ibmr->sg_dma_len)) {
pr_warn("RDS/IB: %s failed!\n", __func__);
return -EBUSY;
}
- frmr->sg_byte_len = 0;
+ ibmr->sg = sg;
+ ibmr->sg_len = sg_len;
frmr->dma_npages = 0;
len = 0;
@@ -264,6 +261,8 @@ static int rds_ib_map_frmr(struct rds_ib_device *rds_ibdev,
ib_dma_unmap_sg(rds_ibdev->dev, ibmr->sg, ibmr->sg_len,
DMA_BIDIRECTIONAL);
ibmr->sg_dma_len = 0;
+ ibmr->sg = NULL;
+ ibmr->sg_len = 0;
return ret;
}
diff --git a/net/rose/af_rose.c b/net/rose/af_rose.c
index 0223d6c34f0b..03a298ecb832 100644
--- a/net/rose/af_rose.c
+++ b/net/rose/af_rose.c
@@ -626,9 +626,7 @@ static struct sock *rose_make_new(struct sock *osk)
rose->hb = orose->hb;
rose->idle = orose->idle;
rose->defer = orose->defer;
- rose->device = orose->device;
- if (rose->device)
- netdev_hold(rose->device, &rose->dev_tracker, GFP_ATOMIC);
+ rose->device = NULL; /* rose_rx_call_request() sets this */
rose->qbitincl = orose->qbitincl;
return sk;
@@ -1078,9 +1076,11 @@ int rose_rx_call_request(struct sk_buff *skb, struct net_device *dev, struct ros
make_rose->source_digis[n] = facilities.source_digis[n];
make_rose->neighbour = neigh;
make_rose->device = dev;
- /* Caller got a reference for us. */
- netdev_tracker_alloc(make_rose->device, &make_rose->dev_tracker,
- GFP_ATOMIC);
+ /* Take an independent reference for this socket; callers keep their
+ * own reference (from rose_dev_get / dev_hold) and will release it
+ * themselves via dev_put().
+ */
+ netdev_hold(make_rose->device, &make_rose->dev_tracker, GFP_ATOMIC);
make_rose->facilities = facilities;
rose_neigh_hold(make_rose->neighbour);
@@ -1667,19 +1667,28 @@ static void __exit rose_exit(void)
#ifdef CONFIG_SYSCTL
rose_unregister_sysctl();
#endif
- unregister_netdevice_notifier(&rose_dev_notifier);
-
sock_unregister(PF_ROSE);
for (i = 0; i < rose_ndevs; i++) {
struct net_device *dev = dev_rose[i];
if (dev) {
+ /* unregister_netdev() fires NETDEV_DOWN, which -- while the
+ * notifier is still registered below -- invokes
+ * rose_kill_by_device(dev). That releases every socket's
+ * netdev reference and disconnects all active circuits.
+ * Unregistering the notifier before this loop was the
+ * original bug: NETDEV_DOWN was never delivered, leaving
+ * 165 netdev_tracker entries leaked and stale timers live.
+ */
unregister_netdev(dev);
free_netdev(dev);
}
}
+ /* Now safe to remove the notifier -- all ROSE devices are gone. */
+ unregister_netdevice_notifier(&rose_dev_notifier);
+
kfree(dev_rose);
proto_unregister(&rose_proto);
}
diff --git a/net/rose/rose_in.c b/net/rose/rose_in.c
index 7caae93937ee..666153a3dd8f 100644
--- a/net/rose/rose_in.c
+++ b/net/rose/rose_in.c
@@ -57,6 +57,7 @@ static int rose_state1_machine(struct sock *sk, struct sk_buff *skb, int framety
rose_write_internal(sk, ROSE_CLEAR_CONFIRMATION);
rose_disconnect(sk, ECONNREFUSED, skb->data[3], skb->data[4]);
rose_neigh_put(rose->neighbour);
+ rose->neighbour = NULL;
break;
default:
@@ -80,11 +81,13 @@ static int rose_state2_machine(struct sock *sk, struct sk_buff *skb, int framety
rose_write_internal(sk, ROSE_CLEAR_CONFIRMATION);
rose_disconnect(sk, 0, skb->data[3], skb->data[4]);
rose_neigh_put(rose->neighbour);
+ rose->neighbour = NULL;
break;
case ROSE_CLEAR_CONFIRMATION:
rose_disconnect(sk, 0, -1, -1);
rose_neigh_put(rose->neighbour);
+ rose->neighbour = NULL;
break;
default:
@@ -121,6 +124,7 @@ static int rose_state3_machine(struct sock *sk, struct sk_buff *skb, int framety
rose_write_internal(sk, ROSE_CLEAR_CONFIRMATION);
rose_disconnect(sk, 0, skb->data[3], skb->data[4]);
rose_neigh_put(rose->neighbour);
+ rose->neighbour = NULL;
break;
case ROSE_RR:
@@ -234,6 +238,7 @@ static int rose_state4_machine(struct sock *sk, struct sk_buff *skb, int framety
rose_write_internal(sk, ROSE_CLEAR_CONFIRMATION);
rose_disconnect(sk, 0, skb->data[3], skb->data[4]);
rose_neigh_put(rose->neighbour);
+ rose->neighbour = NULL;
break;
default:
@@ -254,6 +259,7 @@ static int rose_state5_machine(struct sock *sk, struct sk_buff *skb, int framety
rose_write_internal(sk, ROSE_CLEAR_CONFIRMATION);
rose_disconnect(sk, 0, skb->data[3], skb->data[4]);
rose_neigh_put(rose_sk(sk)->neighbour);
+ rose_sk(sk)->neighbour = NULL;
}
return 0;
diff --git a/net/rose/rose_loopback.c b/net/rose/rose_loopback.c
index 036d92c0ad79..fd4e4795776a 100644
--- a/net/rose/rose_loopback.c
+++ b/net/rose/rose_loopback.c
@@ -12,13 +12,15 @@
#include <net/rose.h>
#include <linux/init.h>
-static struct sk_buff_head loopback_queue;
#define ROSE_LOOPBACK_LIMIT 1000
-static struct timer_list loopback_timer;
+static struct timer_list loopback_timer;
+static struct sk_buff_head loopback_queue;
static void rose_set_loopback_timer(void);
static void rose_loopback_timer(struct timer_list *unused);
+static atomic_t loopback_stopping = ATOMIC_INIT(0);
+
void rose_loopback_init(void)
{
skb_queue_head_init(&loopback_queue);
@@ -66,10 +68,25 @@ static void rose_loopback_timer(struct timer_list *unused)
unsigned int lci_i, lci_o;
int count;
+ if (atomic_read(&loopback_stopping))
+ return;
+
+ if (rose_loopback_neigh)
+ rose_neigh_hold(rose_loopback_neigh);
+ else
+ return;
+
for (count = 0; count < ROSE_LOOPBACK_LIMIT; count++) {
skb = skb_dequeue(&loopback_queue);
if (!skb)
- return;
+ goto out;
+
+ if (atomic_read(&loopback_stopping)) {
+ kfree_skb(skb);
+ skb_queue_purge(&loopback_queue);
+ goto out;
+ }
+
if (skb->len < ROSE_MIN_LEN) {
kfree_skb(skb);
continue;
@@ -96,27 +113,34 @@ static void rose_loopback_timer(struct timer_list *unused)
}
if (frametype == ROSE_CALL_REQUEST) {
- if (!rose_loopback_neigh->dev &&
- !rose_loopback_neigh->loopback) {
- kfree_skb(skb);
- continue;
- }
-
dev = rose_dev_get(dest);
if (!dev) {
kfree_skb(skb);
continue;
}
-
- if (rose_rx_call_request(skb, dev, rose_loopback_neigh, lci_o) == 0) {
+ /* rose_kill_by_device() runs on NETDEV_DOWN (IFF_UP cleared)
+ * before the device is unregistered. If we create a new
+ * socket here after that cleanup, the ref never gets released
+ * because NETDEV_DOWN fires only once. Drop the call instead.
+ */
+ if (!netif_running(dev)) {
dev_put(dev);
kfree_skb(skb);
+ continue;
}
+
+ if (rose_rx_call_request(skb, dev, rose_loopback_neigh, lci_o) == 0)
+ kfree_skb(skb);
+ dev_put(dev);
} else {
kfree_skb(skb);
}
}
- if (!skb_queue_empty(&loopback_queue))
+
+out:
+ rose_neigh_put(rose_loopback_neigh);
+
+ if (!atomic_read(&loopback_stopping) && !skb_queue_empty(&loopback_queue))
mod_timer(&loopback_timer, jiffies + 1);
}
@@ -124,10 +148,15 @@ void __exit rose_loopback_clear(void)
{
struct sk_buff *skb;
- del_timer(&loopback_timer);
+ atomic_set(&loopback_stopping, 1);
+ /* Pairs with atomic_read() in rose_loopback_timer(): ensure the
+ * stopping flag is visible before we cancel, so a concurrent
+ * callback aborts its loop early rather than re-arming the timer.
+ */
+ smp_mb();
- while ((skb = skb_dequeue(&loopback_queue)) != NULL) {
- skb->sk = NULL;
+ timer_delete_sync(&loopback_timer);
+
+ while ((skb = skb_dequeue(&loopback_queue)) != NULL)
kfree_skb(skb);
- }
}
diff --git a/net/rose/rose_timer.c b/net/rose/rose_timer.c
index c52d7d20c519..53d198ac96e8 100644
--- a/net/rose/rose_timer.c
+++ b/net/rose/rose_timer.c
@@ -180,7 +180,10 @@ static void rose_timer_expiry(struct timer_list *t)
break;
case ROSE_STATE_2: /* T3 */
- rose_neigh_put(rose->neighbour);
+ if (rose->neighbour) {
+ rose_neigh_put(rose->neighbour);
+ rose->neighbour = NULL;
+ }
rose_disconnect(sk, ETIMEDOUT, -1, -1);
break;
diff --git a/net/sched/act_api.c b/net/sched/act_api.c
index da66b1933813..ca851146f695 100644
--- a/net/sched/act_api.c
+++ b/net/sched/act_api.c
@@ -714,7 +714,7 @@ static int tcf_idr_delete_index(struct tcf_idrinfo *idrinfo, u32 index)
mutex_lock(&idrinfo->lock);
p = idr_find(&idrinfo->action_idr, index);
- if (!p) {
+ if (IS_ERR_OR_NULL(p)) {
mutex_unlock(&idrinfo->lock);
return -ENOENT;
}
@@ -1165,11 +1165,16 @@ static int tcf_action_put(struct tc_action *p)
static void tcf_action_put_many(struct tc_action *actions[])
{
- struct tc_action *a;
int i;
- tcf_act_for_each_action(i, a, actions) {
- const struct tc_action_ops *ops = a->ops;
+ /* Deletion may have cleared entries before failing. */
+ for (i = 0; i < TCA_ACT_MAX_PRIO; i++) {
+ struct tc_action *a = actions[i];
+ const struct tc_action_ops *ops;
+
+ if (!a)
+ continue;
+ ops = a->ops;
if (tcf_action_put(a))
module_put(ops->owner);
}
diff --git a/net/sched/act_ct.c b/net/sched/act_ct.c
index 8aed25cad9ee..154be631b038 100644
--- a/net/sched/act_ct.c
+++ b/net/sched/act_ct.c
@@ -430,11 +430,10 @@ static void tcf_ct_flow_table_add(struct tcf_ct_flow_table *ct_ft,
if (test_and_set_bit(IPS_OFFLOAD_BIT, &ct->status))
return;
+ /* NULL if ct is dying (raced flush) or the atomic alloc failed. */
entry = flow_offload_alloc(ct);
- if (!entry) {
- WARN_ON_ONCE(1);
+ if (!entry)
goto err_alloc;
- }
if (tcp) {
ct->proto.tcp.seen[0].flags |= IP_CT_TCP_FLAG_BE_LIBERAL;
@@ -978,11 +977,11 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a,
struct tcf_result *res)
{
struct net *net = dev_net(skb->dev);
+ bool cached, commit, clear, nat;
enum ip_conntrack_info ctinfo;
struct tcf_ct *c = to_ct(a);
struct nf_conn *tmpl = NULL;
struct nf_hook_state state;
- bool cached, commit, clear;
int nh_ofs, err, retval;
struct tcf_ct_params *p;
bool add_helper = false;
@@ -997,6 +996,7 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a,
retval = p->action;
commit = p->ct_action & TCA_CT_ACT_COMMIT;
clear = p->ct_action & TCA_CT_ACT_CLEAR;
+ nat = p->ct_action & TCA_CT_ACT_NAT;
tmpl = p->tmpl;
tcf_lastuse_update(&c->tcf_tm);
@@ -1045,6 +1045,19 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a,
* different zone.
*/
cached = tcf_ct_skb_nfct_cached(net, skb, p);
+
+ /* If the ct entry is not confirmed and shared with some other skb,
+ * e.g., a cloned one, we can't just modify it with a commit or nat
+ * as we must not modify the extension set. Reset.
+ */
+ if (cached && (commit || nat)) {
+ ct = nf_ct_get(skb, &ctinfo);
+ if (ct && !nf_ct_is_confirmed(ct) && nf_ct_shared(ct)) {
+ nf_reset_ct(skb);
+ cached = false;
+ }
+ }
+
if (!cached) {
if (tcf_ct_flow_table_lookup(p, skb, family)) {
skip_add = true;
@@ -1082,25 +1095,31 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a,
if (err)
goto drop;
add_helper = true;
- if (p->ct_action & TCA_CT_ACT_NAT && !nfct_seqadj(ct)) {
+ if (nat && !nfct_seqadj(ct)) {
if (!nfct_seqadj_ext_add(ct))
goto drop;
}
}
- if (nf_ct_is_confirmed(ct) ? ((!cached && !skip_add) || add_helper) : commit) {
- err = nf_ct_helper(skb, ct, ctinfo, family);
- if (err != NF_ACCEPT)
- goto nf_error;
- }
-
if (commit) {
tcf_ct_act_set_mark(ct, p->mark, p->mark_mask);
tcf_ct_act_set_labels(ct, p->labels, p->labels_mask);
if (!nf_ct_is_confirmed(ct))
nf_conn_act_ct_ext_add(skb, ct, ctinfo);
+ }
+ /* Run helpers for the connection if nf_conntrack_in() was executed
+ * or if we're about to commit. This has to be done after all the
+ * extensions are already added.
+ */
+ if (nf_ct_is_confirmed(ct) ? ((!cached && !skip_add) || add_helper) : commit) {
+ err = nf_ct_helper(skb, ct, ctinfo, family);
+ if (err != NF_ACCEPT)
+ goto nf_error;
+ }
+
+ if (commit) {
/* This will take care of sending queued events
* even if the connection is already confirmed.
*/
diff --git a/net/sched/act_gate.c b/net/sched/act_gate.c
index 58cae012a439..d66c07aef95a 100644
--- a/net/sched/act_gate.c
+++ b/net/sched/act_gate.c
@@ -686,7 +686,35 @@ static void tcf_gate_stats_update(struct tc_action *a, u64 bytes, u64 packets,
static size_t tcf_gate_get_fill_size(const struct tc_action *act)
{
- return nla_total_size(sizeof(struct tc_gate));
+ struct tcf_gate *gact = to_gate(act);
+ const struct tcf_gate_params *p;
+ struct tcfg_gate_entry *entry;
+ size_t size = nla_total_size(sizeof(struct tc_gate)) /* TCA_GATE_PARMS */
+ + 3 * nla_total_size_64bit(sizeof(u64)) /* TCA_GATE_BASE_TIME
+ * TCA_GATE_CYCLE_TIME
+ * TCA_GATE_CYCLE_TIME_EXT
+ */
+ + nla_total_size(sizeof(s32)) /* TCA_GATE_CLOCKID */
+ + nla_total_size(sizeof(u32)) /* TCA_GATE_FLAGS */
+ + nla_total_size(sizeof(s32)) /* TCA_GATE_PRIORITY */
+ + nla_total_size(0); /* TCA_GATE_ENTRY_LIST */
+ /* TCA_GATE_TM is budgeted by tcf_action_shared_attrs_size() */
+
+ rcu_read_lock();
+ p = rcu_dereference(gact->param);
+ if (p) {
+ list_for_each_entry_rcu(entry, &p->entries, list)
+ /* TCA_GATE_ONE_ENTRY nest and its attributes */
+ size += nla_total_size(0)
+ + nla_total_size(sizeof(u32)) /* TCA_GATE_ENTRY_INDEX */
+ + nla_total_size(0) /* TCA_GATE_ENTRY_GATE */
+ + nla_total_size(sizeof(u32)) /* TCA_GATE_ENTRY_INTERVAL */
+ + nla_total_size(sizeof(s32)) /* TCA_GATE_ENTRY_MAX_OCTETS */
+ + nla_total_size(sizeof(s32)); /* TCA_GATE_ENTRY_IPV */
+ }
+ rcu_read_unlock();
+
+ return size;
}
static void tcf_gate_entry_destructor(void *priv)
diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c
index 83a60698cc00..e5e5533f866d 100644
--- a/net/sched/cls_u32.c
+++ b/net/sched/cls_u32.c
@@ -1000,8 +1000,16 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
return -ENOMEM;
}
} else {
- err = idr_alloc_u32(&tp_c->handle_idr, ht, &handle,
- handle, GFP_KERNEL);
+ /* The IDR is keyed on the mapped id, and that is
+ * what the destroy paths remove. Ask for it here,
+ * so a manual handle colliding with the
+ * auto-allocated id space is rejected (-ENOSPC)
+ * instead of aliasing a future auto id.
+ */
+ u32 id = handle2id(handle);
+
+ err = idr_alloc_u32(&tp_c->handle_idr, ht, &id, id,
+ GFP_KERNEL);
if (err) {
kfree(ht);
return err;
diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c
index 601f99dd989a..df3501dfe01a 100644
--- a/net/sched/sch_hfsc.c
+++ b/net/sched/sch_hfsc.c
@@ -386,6 +386,15 @@ cftree_update(struct hfsc_class *cl)
#define SM_MASK ((1ULL << SM_SHIFT) - 1)
#define ISM_MASK ((1ULL << ISM_SHIFT) - 1)
+/*
+ * Cap on the non-descending hops a classify walk may take before its
+ * filter chain is treated as misconfigured. A flowid binding that was
+ * legal at bind time can become lateral once hfsc_adjust_levels()
+ * raises a class level; a few such hops are legitimate, an unbounded
+ * run means the chain cycles.
+ */
+#define HFSC_CLASSIFY_MAX_DRIFT 8
+
static inline u64
seg_x2y(u64 x, u64 sm)
{
@@ -1133,6 +1142,7 @@ hfsc_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
struct hfsc_class *head, *cl;
struct tcf_result res;
struct tcf_proto *tcf;
+ unsigned int drift;
int result;
if (TC_H_MAJ(skb->priority ^ sch->handle) == 0 &&
@@ -1142,6 +1152,7 @@ hfsc_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
head = &q->root;
+ drift = HFSC_CLASSIFY_MAX_DRIFT;
tcf = rcu_dereference_bh(q->root.filter_list);
while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) {
#ifdef CONFIG_NET_CLS_ACT
@@ -1167,6 +1178,17 @@ hfsc_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
if (cl->level == 0)
return cl; /* hit leaf class */
+ /*
+ * flowid binds skip the level check above (res.class is set
+ * at bind time and levels drift after), so a walk can follow
+ * lateral hops without descending; a bounded number of them
+ * is legal, more means the chain cycles.
+ */
+ if (cl->level >= head->level && drift-- == 0) {
+ pr_warn_ratelimited("hfsc: classify hop budget exhausted, dropping packet\n");
+ return NULL;
+ }
+
/* apply inner filter chain */
tcf = rcu_dereference_bh(cl->filter_list);
head = cl;
diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c
index e7df723c9f62..7d7bc1d7e049 100644
--- a/net/sched/sch_hhf.c
+++ b/net/sched/sch_hhf.c
@@ -526,7 +526,7 @@ static void hhf_destroy(struct Qdisc *sch)
static const struct nla_policy hhf_policy[TCA_HHF_MAX + 1] = {
[TCA_HHF_BACKLOG_LIMIT] = { .type = NLA_U32 },
[TCA_HHF_QUANTUM] = { .type = NLA_U32 },
- [TCA_HHF_HH_FLOWS_LIMIT] = { .type = NLA_U32 },
+ [TCA_HHF_HH_FLOWS_LIMIT] = NLA_POLICY_MAX(NLA_U32, 2 * HH_FLOWS_CNT),
[TCA_HHF_RESET_TIMEOUT] = { .type = NLA_U32 },
[TCA_HHF_ADMIT_BYTES] = { .type = NLA_U32 },
[TCA_HHF_EVICT_TIMEOUT] = { .type = NLA_U32 },
@@ -545,7 +545,7 @@ static int hhf_change(struct Qdisc *sch, struct nlattr *opt,
u32 new_hhf_non_hh_weight = q->hhf_non_hh_weight;
err = nla_parse_nested_deprecated(tb, TCA_HHF_MAX, opt, hhf_policy,
- NULL);
+ extack);
if (err < 0)
return err;
@@ -621,6 +621,9 @@ static int hhf_init(struct Qdisc *sch, struct nlattr *opt,
q->hhf_evict_timeout = HZ; /* 1 sec */
q->hhf_non_hh_weight = 2;
+ /* Cap max active HHs at twice len of hh_flows table. */
+ q->hh_flows_limit = 2 * HH_FLOWS_CNT;
+
if (opt) {
int err = hhf_change(sch, opt, extack);
@@ -637,8 +640,6 @@ static int hhf_init(struct Qdisc *sch, struct nlattr *opt,
for (i = 0; i < HH_FLOWS_CNT; i++)
INIT_LIST_HEAD(&q->hh_flows[i]);
- /* Cap max active HHs at twice len of hh_flows table. */
- q->hh_flows_limit = 2 * HH_FLOWS_CNT;
q->hh_flows_overlimit = 0;
q->hh_flows_total_cnt = 0;
q->hh_flows_current_cnt = 0;
diff --git a/net/sched/sch_teql.c b/net/sched/sch_teql.c
index eb424b20027e..d3baf17ebe6c 100644
--- a/net/sched/sch_teql.c
+++ b/net/sched/sch_teql.c
@@ -265,14 +265,11 @@ __teql_resolve(struct sk_buff *skb, struct sk_buff *skb_res,
}
if (neigh_event_send(n, skb_res) == 0) {
- int err;
char haddr[MAX_ADDR_LEN];
neigh_ha_snapshot(haddr, n, dev);
- err = dev_hard_header(skb, dev, ntohs(skb_protocol(skb, false)),
- haddr, NULL, skb->len);
-
- if (err < 0)
+ if (dev_hard_header(skb, dev, ntohs(skb_protocol(skb, false)),
+ haddr, NULL, skb->len) < 0)
err = -EINVAL;
} else {
err = (skb_res == NULL) ? -EAGAIN : 1;
diff --git a/net/sctp/associola.c b/net/sctp/associola.c
index f409b9f6b228..0a19aea3b524 100644
--- a/net/sctp/associola.c
+++ b/net/sctp/associola.c
@@ -1303,18 +1303,19 @@ void sctp_assoc_update_retran_path(struct sctp_association *asoc)
/* Manually skip the head element. */
if (&trans->transports == &asoc->peer.transport_addr_list)
continue;
- if (trans->state == SCTP_UNCONFIRMED)
- continue;
- trans_next = sctp_trans_elect_best(trans, trans_next);
- /* Active is good enough for immediate return. */
- if (trans_next->state == SCTP_ACTIVE)
- break;
+ if (trans->state != SCTP_UNCONFIRMED) {
+ trans_next = sctp_trans_elect_best(trans, trans_next);
+ /* Active is good enough for immediate return. */
+ if (trans_next->state == SCTP_ACTIVE)
+ break;
+ }
/* We've reached the end, time to update path. */
if (trans == asoc->peer.retran_path)
break;
}
- asoc->peer.retran_path = trans_next;
+ if (trans_next)
+ asoc->peer.retran_path = trans_next;
pr_debug("%s: association:%p updated new path to addr:%pISpc\n",
__func__, asoc, &asoc->peer.retran_path->ipaddr.sa);
diff --git a/net/sctp/input.c b/net/sctp/input.c
index df5b2187b8fa..e946c1b4abfe 100644
--- a/net/sctp/input.c
+++ b/net/sctp/input.c
@@ -436,9 +436,10 @@ void sctp_icmp_proto_unreachable(struct sock *sk,
if (timer_pending(&t->proto_unreach_timer))
return;
else {
- if (!mod_timer(&t->proto_unreach_timer,
- jiffies + (HZ/20)))
- sctp_transport_hold(t);
+ sctp_transport_hold(t);
+ if (mod_timer(&t->proto_unreach_timer,
+ jiffies + (HZ / 20)))
+ sctp_transport_put(t);
}
} else {
struct net *net = sock_net(sk);
diff --git a/net/sctp/ipv6.c b/net/sctp/ipv6.c
index bf54fb1ee654..28ea3a851862 100644
--- a/net/sctp/ipv6.c
+++ b/net/sctp/ipv6.c
@@ -784,9 +784,10 @@ static struct sock *sctp_v6_create_accept_sk(struct sock *sk,
struct sctp_association *asoc,
bool kern)
{
- struct sock *newsk;
struct ipv6_pinfo *newnp, *np = inet6_sk(sk);
struct sctp6_sock *newsctp6sk;
+ struct inet_sock *newinet;
+ struct sock *newsk;
newsk = sk_alloc(sock_net(sk), PF_INET6, GFP_KERNEL, sk->sk_prot, kern);
if (!newsk)
@@ -798,7 +799,9 @@ static struct sock *sctp_v6_create_accept_sk(struct sock *sk,
sock_reset_flag(sk, SOCK_ZAPPED);
newsctp6sk = (struct sctp6_sock *)newsk;
- inet_sk(newsk)->pinet6 = &newsctp6sk->inet6;
+ newinet = inet_sk(newsk);
+ newinet->pinet6 = &newsctp6sk->inet6;
+ newinet->ipv6_fl_list = NULL;
sctp_sk(newsk)->v4mapped = sctp_sk(sk)->v4mapped;
@@ -807,7 +810,6 @@ static struct sock *sctp_v6_create_accept_sk(struct sock *sk,
memcpy(newnp, np, sizeof(struct ipv6_pinfo));
newnp->ipv6_mc_list = NULL;
newnp->ipv6_ac_list = NULL;
- newnp->ipv6_fl_list = NULL;
sctp_v6_copy_ip_options(sk, newsk);
diff --git a/net/sctp/sm_sideeffect.c b/net/sctp/sm_sideeffect.c
index c05f1b073696..7d451e6c5d27 100644
--- a/net/sctp/sm_sideeffect.c
+++ b/net/sctp/sm_sideeffect.c
@@ -244,8 +244,9 @@ void sctp_generate_t3_rtx_event(struct timer_list *t)
pr_debug("%s: sock is busy\n", __func__);
/* Try again later. */
- if (!mod_timer(&transport->T3_rtx_timer, jiffies + (HZ/20)))
- sctp_transport_hold(transport);
+ sctp_transport_hold(transport);
+ if (mod_timer(&transport->T3_rtx_timer, jiffies + (HZ / 20)))
+ sctp_transport_put(transport);
goto out_unlock;
}
@@ -280,8 +281,9 @@ static void sctp_generate_timeout_event(struct sctp_association *asoc,
timeout_type);
/* Try again later. */
- if (!mod_timer(&asoc->timers[timeout_type], jiffies + (HZ/20)))
- sctp_association_hold(asoc);
+ sctp_association_hold(asoc);
+ if (mod_timer(&asoc->timers[timeout_type], jiffies + (HZ / 20)))
+ sctp_association_put(asoc);
goto out_unlock;
}
@@ -373,8 +375,9 @@ void sctp_generate_heartbeat_event(struct timer_list *t)
pr_debug("%s: sock is busy\n", __func__);
/* Try again later. */
- if (!mod_timer(&transport->hb_timer, jiffies + (HZ/20)))
- sctp_transport_hold(transport);
+ sctp_transport_hold(transport);
+ if (mod_timer(&transport->hb_timer, jiffies + (HZ / 20)))
+ sctp_transport_put(transport);
goto out_unlock;
}
@@ -383,8 +386,9 @@ void sctp_generate_heartbeat_event(struct timer_list *t)
timeout = sctp_transport_timeout(transport);
if (elapsed < timeout) {
elapsed = timeout - elapsed;
- if (!mod_timer(&transport->hb_timer, jiffies + elapsed))
- sctp_transport_hold(transport);
+ sctp_transport_hold(transport);
+ if (mod_timer(&transport->hb_timer, jiffies + elapsed))
+ sctp_transport_put(transport);
goto out_unlock;
}
@@ -417,9 +421,10 @@ void sctp_generate_proto_unreach_event(struct timer_list *t)
pr_debug("%s: sock is busy\n", __func__);
/* Try again later. */
- if (!mod_timer(&transport->proto_unreach_timer,
- jiffies + (HZ/20)))
- sctp_transport_hold(transport);
+ sctp_transport_hold(transport);
+ if (mod_timer(&transport->proto_unreach_timer,
+ jiffies + (HZ / 20)))
+ sctp_transport_put(transport);
goto out_unlock;
}
@@ -453,8 +458,9 @@ void sctp_generate_reconf_event(struct timer_list *t)
pr_debug("%s: sock is busy\n", __func__);
/* Try again later. */
- if (!mod_timer(&transport->reconf_timer, jiffies + (HZ / 20)))
- sctp_transport_hold(transport);
+ sctp_transport_hold(transport);
+ if (mod_timer(&transport->reconf_timer, jiffies + (HZ / 20)))
+ sctp_transport_put(transport);
goto out_unlock;
}
@@ -489,8 +495,9 @@ void sctp_generate_probe_event(struct timer_list *t)
pr_debug("%s: sock is busy\n", __func__);
/* Try again later. */
- if (!mod_timer(&transport->probe_timer, jiffies + (HZ / 20)))
- sctp_transport_hold(transport);
+ sctp_transport_hold(transport);
+ if (mod_timer(&transport->probe_timer, jiffies + (HZ / 20)))
+ sctp_transport_put(transport);
goto out_unlock;
}
diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index 5be54b119c39..ee4269b75e36 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -2624,6 +2624,8 @@ static enum sctp_disposition sctp_sf_do_5_2_6_stale(
sctp_add_cmd_sf(commands, SCTP_CMD_REPLY, SCTP_CHUNK(reply));
+ sctp_add_cmd_sf(commands, SCTP_CMD_DISCARD_PACKET, SCTP_NULL());
+
return SCTP_DISPOSITION_CONSUME;
nomem:
diff --git a/net/smc/smc.h b/net/smc/smc.h
index f9d364a2167a..1425fd756d34 100644
--- a/net/smc/smc.h
+++ b/net/smc/smc.h
@@ -277,9 +277,9 @@ struct smc_connection {
* 0 for SMC-R, 32 for SMC-D
*/
u64 peer_token; /* SMC-D token of peer */
- u8 killed : 1; /* abnormal termination */
- u8 freed : 1; /* normal termiation */
- u8 out_of_sync : 1; /* out of sync with peer */
+ u8 killed; /* abnormal termination */
+ u8 freed; /* normal termination */
+ u8 out_of_sync; /* out of sync with peer */
};
struct smc_sock { /* smc sock container */
diff --git a/net/smc/smc_clc.h b/net/smc/smc_clc.h
index 1a7676227f16..767289925410 100644
--- a/net/smc/smc_clc.h
+++ b/net/smc/smc_clc.h
@@ -156,7 +156,7 @@ struct smc_clc_msg_proposal_prefix { /* prefix part of clc proposal message*/
} __aligned(4);
struct smc_clc_msg_smcd { /* SMC-D GID information */
- struct smc_clc_smcd_gid_chid ism; /* ISM native GID+CHID of requestor */
+ struct smc_clc_smcd_gid_chid ism; /* ISM native GID+CHID of requester */
__be16 v2_ext_offset; /* SMC Version 2 Extension Offset */
u8 vendor_oui[3]; /* vendor organizationally unique identifier */
u8 vendor_exp_options[5];
diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
index ed0027ba66c7..a71bb1803c40 100644
--- a/net/smc/smc_core.c
+++ b/net/smc/smc_core.c
@@ -1830,6 +1830,7 @@ void smcr_port_err(struct smc_ib_device *smcibdev, u8 ibport)
struct smc_link_group *lgr, *n;
int i;
+ spin_lock_bh(&smc_lgr_list.lock);
list_for_each_entry_safe(lgr, n, &smc_lgr_list.list, list) {
if (strncmp(smcibdev->pnetid[ibport - 1], lgr->pnet_id,
SMC_MAX_PNETID_LEN))
@@ -1844,6 +1845,7 @@ void smcr_port_err(struct smc_ib_device *smcibdev, u8 ibport)
smcr_link_down_cond_sched(lnk);
}
}
+ spin_unlock_bh(&smc_lgr_list.lock);
}
static void smc_link_down_work(struct work_struct *work)
@@ -2324,7 +2326,7 @@ static struct smc_buf_desc *smcr_new_buf_create(struct smc_link_group *lgr,
}
if (lgr->buf_type == SMCR_PHYS_CONT_BUFS)
goto out;
- fallthrough; // try virtually continguous buf
+ fallthrough; // try virtually contiguous buf
case SMCR_VIRT_CONT_BUFS:
buf_desc->order = get_order(bufsize);
buf_desc->cpu_addr = vzalloc(PAGE_SIZE << buf_desc->order);
diff --git a/net/smc/smc_core.h b/net/smc/smc_core.h
index 0db4e5f79ac4..69b54ecd6503 100644
--- a/net/smc/smc_core.h
+++ b/net/smc/smc_core.h
@@ -30,7 +30,7 @@
*/
#define SMC_CONN_PER_LGR_PREFER 255 /* Preferred connections per link group used for
* SMC-R v2.1 and later negotiation, vendors or
- * distrubutions may modify it to a value between
+ * distributions may modify it to a value between
* 16-255 as needed.
*/
@@ -181,7 +181,7 @@ struct smc_link {
*/
#define SMC_LINKS_PER_LGR_MAX_PREFER 2 /* Preferred max links per link group used for
* SMC-R v2.1 and later negotiation, vendors or
- * distrubutions may modify it to a value between
+ * distributions may modify it to a value between
* 1-2 as needed.
*/
diff --git a/net/smc/smc_ib.c b/net/smc/smc_ib.c
index fc07fc4ed998..0cca7b9ca044 100644
--- a/net/smc/smc_ib.c
+++ b/net/smc/smc_ib.c
@@ -333,6 +333,7 @@ static bool smc_ib_check_link_gid(u8 gid[SMC_GID_SIZE], bool smcrv2,
static void smc_ib_gid_check(struct smc_ib_device *smcibdev, u8 ibport)
{
struct smc_link_group *lgr;
+ bool stale_gid = false;
int i;
spin_lock_bh(&smc_lgr_list.lock);
@@ -348,11 +349,16 @@ static void smc_ib_gid_check(struct smc_ib_device *smcibdev, u8 ibport)
continue;
if (!smc_ib_check_link_gid(lgr->lnk[i].gid,
lgr->smc_version == SMC_V2,
- smcibdev, ibport))
- smcr_port_err(smcibdev, ibport);
+ smcibdev, ibport)) {
+ stale_gid = true;
+ goto out;
+ }
}
}
+out:
spin_unlock_bh(&smc_lgr_list.lock);
+ if (stale_gid)
+ smcr_port_err(smcibdev, ibport);
}
static int smc_ib_remember_port_attr(struct smc_ib_device *smcibdev, u8 ibport)
diff --git a/net/tipc/group.c b/net/tipc/group.c
index 215f2a7d8458..0da7b2713394 100644
--- a/net/tipc/group.c
+++ b/net/tipc/group.c
@@ -797,10 +797,10 @@ void tipc_group_proto_rcv(struct tipc_group *grp, bool *usr_wakeup,
tipc_group_open(m, usr_wakeup);
return;
case GRP_ACK_MSG:
- if (!m)
+ if (!m || !grp->bc_ackers)
return;
acked = msg_grp_bc_acked(hdr);
- if (less_eq(acked, m->bc_acked))
+ if (acked != grp->bc_snd_nxt || m->bc_acked == acked)
return;
m->bc_acked = acked;
if (--grp->bc_ackers)
diff --git a/net/tipc/monitor.c b/net/tipc/monitor.c
index b45c5b91bc7a..5e4585b03497 100644
--- a/net/tipc/monitor.c
+++ b/net/tipc/monitor.c
@@ -632,9 +632,10 @@ static void mon_timeout(struct timer_list *t)
{
struct tipc_monitor *mon = from_timer(mon, t, timer);
struct tipc_peer *self;
- int best_member_cnt = dom_size(mon->peer_cnt) - 1;
+ int best_member_cnt;
write_lock_bh(&mon->lock);
+ best_member_cnt = dom_size(mon->peer_cnt) - 1;
self = mon->self;
if (self && (best_member_cnt != self->applied)) {
mon_update_local_domain(mon);
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index a41fd85279bc..323ac8b1ad0c 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -2337,9 +2337,9 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,
if (copied < 0)
goto splice_requeue;
- if (chunk < rxm->full_len) {
- rxm->offset += len;
- rxm->full_len -= len;
+ if (copied < rxm->full_len) {
+ rxm->offset += copied;
+ rxm->full_len -= copied;
goto splice_requeue;
}
diff --git a/net/wireless/chan.c b/net/wireless/chan.c
index 515d40c09e78..6f4b4770bf50 100644
--- a/net/wireless/chan.c
+++ b/net/wireless/chan.c
@@ -55,6 +55,51 @@ void cfg80211_chandef_create(struct cfg80211_chan_def *chandef,
}
EXPORT_SYMBOL(cfg80211_chandef_create);
+static u32 cfg80211_get_start_freq(const struct cfg80211_chan_def *chandef,
+ u32 cf)
+{
+ u32 start_freq, center_freq, bandwidth;
+
+ center_freq = MHZ_TO_KHZ((cf == 1) ?
+ chandef->center_freq1 : chandef->center_freq2);
+ bandwidth = MHZ_TO_KHZ(cfg80211_chandef_get_width(chandef));
+
+ if (bandwidth <= MHZ_TO_KHZ(20))
+ start_freq = center_freq;
+ else
+ start_freq = center_freq - bandwidth / 2 + MHZ_TO_KHZ(10);
+
+ return start_freq;
+}
+
+static u32 cfg80211_get_end_freq(const struct cfg80211_chan_def *chandef,
+ u32 cf)
+{
+ u32 end_freq, center_freq, bandwidth;
+
+ center_freq = MHZ_TO_KHZ((cf == 1) ?
+ chandef->center_freq1 : chandef->center_freq2);
+ bandwidth = MHZ_TO_KHZ(cfg80211_chandef_get_width(chandef));
+
+ if (bandwidth <= MHZ_TO_KHZ(20))
+ end_freq = center_freq;
+ else
+ end_freq = center_freq + bandwidth / 2 - MHZ_TO_KHZ(10);
+
+ return end_freq;
+}
+
+#define for_each_subchan(chandef, freq, cf) \
+ for (u32 punctured = chandef->punctured, \
+ cf = 1, freq = cfg80211_get_start_freq(chandef, cf); \
+ freq <= cfg80211_get_end_freq(chandef, cf); \
+ freq += MHZ_TO_KHZ(20), \
+ ((cf == 1 && chandef->center_freq2 != 0 && \
+ freq > cfg80211_get_end_freq(chandef, cf)) ? \
+ (cf++, freq = cfg80211_get_start_freq(chandef, cf), \
+ punctured = 0) : (punctured >>= 1))) \
+ if (!(punctured & 1))
+
struct cfg80211_per_bw_puncturing_values {
u8 len;
const u16 *valid_values;
@@ -258,11 +303,6 @@ int nl80211_chan_width_to_mhz(enum nl80211_chan_width chan_width)
}
EXPORT_SYMBOL(nl80211_chan_width_to_mhz);
-static int cfg80211_chandef_get_width(const struct cfg80211_chan_def *c)
-{
- return nl80211_chan_width_to_mhz(c->width);
-}
-
static bool cfg80211_valid_center_freq(u32 center,
enum nl80211_chan_width width)
{
@@ -582,29 +622,11 @@ cfg80211_chandef_compatible(const struct cfg80211_chan_def *c1,
}
EXPORT_SYMBOL(cfg80211_chandef_compatible);
-static void cfg80211_set_chans_dfs_state(struct wiphy *wiphy, u32 center_freq,
- u32 bandwidth,
- enum nl80211_dfs_state dfs_state)
-{
- struct ieee80211_channel *c;
- u32 freq;
-
- for (freq = center_freq - bandwidth/2 + 10;
- freq <= center_freq + bandwidth/2 - 10;
- freq += 20) {
- c = ieee80211_get_channel(wiphy, freq);
- if (!c || !(c->flags & IEEE80211_CHAN_RADAR))
- continue;
-
- c->dfs_state = dfs_state;
- c->dfs_state_entered = jiffies;
- }
-}
-
void cfg80211_set_dfs_state(struct wiphy *wiphy,
const struct cfg80211_chan_def *chandef,
enum nl80211_dfs_state dfs_state)
{
+ struct ieee80211_channel *c;
int width;
if (WARN_ON(!cfg80211_chandef_valid(chandef)))
@@ -614,41 +636,14 @@ void cfg80211_set_dfs_state(struct wiphy *wiphy,
if (width < 0)
return;
- cfg80211_set_chans_dfs_state(wiphy, chandef->center_freq1,
- width, dfs_state);
-
- if (!chandef->center_freq2)
- return;
- cfg80211_set_chans_dfs_state(wiphy, chandef->center_freq2,
- width, dfs_state);
-}
-
-static u32 cfg80211_get_start_freq(u32 center_freq,
- u32 bandwidth)
-{
- u32 start_freq;
-
- bandwidth = MHZ_TO_KHZ(bandwidth);
- if (bandwidth <= MHZ_TO_KHZ(20))
- start_freq = center_freq;
- else
- start_freq = center_freq - bandwidth / 2 + MHZ_TO_KHZ(10);
-
- return start_freq;
-}
-
-static u32 cfg80211_get_end_freq(u32 center_freq,
- u32 bandwidth)
-{
- u32 end_freq;
-
- bandwidth = MHZ_TO_KHZ(bandwidth);
- if (bandwidth <= MHZ_TO_KHZ(20))
- end_freq = center_freq;
- else
- end_freq = center_freq + bandwidth / 2 - MHZ_TO_KHZ(10);
+ for_each_subchan(chandef, freq, cf) {
+ c = ieee80211_get_channel_khz(wiphy, freq);
+ if (!c || !(c->flags & IEEE80211_CHAN_RADAR))
+ continue;
- return end_freq;
+ c->dfs_state = dfs_state;
+ c->dfs_state_entered = jiffies;
+ }
}
static bool
@@ -725,17 +720,12 @@ static bool cfg80211_dfs_permissive_chan(struct wiphy *wiphy,
}
static int cfg80211_get_chans_dfs_required(struct wiphy *wiphy,
- u32 center_freq,
- u32 bandwidth,
- enum nl80211_iftype iftype)
+ const struct cfg80211_chan_def *chandef,
+ enum nl80211_iftype iftype)
{
struct ieee80211_channel *c;
- u32 freq, start_freq, end_freq;
-
- start_freq = cfg80211_get_start_freq(center_freq, bandwidth);
- end_freq = cfg80211_get_end_freq(center_freq, bandwidth);
- for (freq = start_freq; freq <= end_freq; freq += MHZ_TO_KHZ(20)) {
+ for_each_subchan(chandef, freq, cf) {
c = ieee80211_get_channel_khz(wiphy, freq);
if (!c)
return -EINVAL;
@@ -768,25 +758,9 @@ int cfg80211_chandef_dfs_required(struct wiphy *wiphy,
if (width < 0)
return -EINVAL;
- ret = cfg80211_get_chans_dfs_required(wiphy,
- ieee80211_chandef_to_khz(chandef),
- width, iftype);
- if (ret < 0)
- return ret;
- else if (ret > 0)
- return BIT(chandef->width);
-
- if (!chandef->center_freq2)
- return 0;
-
- ret = cfg80211_get_chans_dfs_required(wiphy,
- MHZ_TO_KHZ(chandef->center_freq2),
- width, iftype);
- if (ret < 0)
- return ret;
- else if (ret > 0)
- return BIT(chandef->width);
+ ret = cfg80211_get_chans_dfs_required(wiphy, chandef, iftype);
+ return (ret > 0) ? BIT(chandef->width) : ret;
break;
case NL80211_IFTYPE_STATION:
case NL80211_IFTYPE_OCB:
@@ -806,16 +780,18 @@ int cfg80211_chandef_dfs_required(struct wiphy *wiphy,
}
EXPORT_SYMBOL(cfg80211_chandef_dfs_required);
-static int cfg80211_get_chans_dfs_usable(struct wiphy *wiphy,
- u32 center_freq,
- u32 bandwidth)
+bool cfg80211_chandef_dfs_usable(struct wiphy *wiphy,
+ const struct cfg80211_chan_def *chandef)
{
struct ieee80211_channel *c;
- u32 freq, start_freq, end_freq;
- int count = 0;
+ int width, count = 0;
- start_freq = cfg80211_get_start_freq(center_freq, bandwidth);
- end_freq = cfg80211_get_end_freq(center_freq, bandwidth);
+ if (WARN_ON(!cfg80211_chandef_valid(chandef)))
+ return false;
+
+ width = cfg80211_chandef_get_width(chandef);
+ if (width < 0)
+ return false;
/*
* Check entire range of channels for the bandwidth.
@@ -823,61 +799,24 @@ static int cfg80211_get_chans_dfs_usable(struct wiphy *wiphy,
* DFS_AVAILABLE). Return number of usable channels
* (require CAC). Allow DFS and non-DFS channel mix.
*/
- for (freq = start_freq; freq <= end_freq; freq += MHZ_TO_KHZ(20)) {
+ for_each_subchan(chandef, freq, cf) {
c = ieee80211_get_channel_khz(wiphy, freq);
if (!c)
- return -EINVAL;
+ return false;
if (c->flags & IEEE80211_CHAN_DISABLED)
- return -EINVAL;
+ return false;
if (c->flags & IEEE80211_CHAN_RADAR) {
if (c->dfs_state == NL80211_DFS_UNAVAILABLE)
- return -EINVAL;
+ return false;
if (c->dfs_state == NL80211_DFS_USABLE)
count++;
}
}
- return count;
-}
-
-bool cfg80211_chandef_dfs_usable(struct wiphy *wiphy,
- const struct cfg80211_chan_def *chandef)
-{
- int width;
- int r1, r2 = 0;
-
- if (WARN_ON(!cfg80211_chandef_valid(chandef)))
- return false;
-
- width = cfg80211_chandef_get_width(chandef);
- if (width < 0)
- return false;
-
- r1 = cfg80211_get_chans_dfs_usable(wiphy,
- MHZ_TO_KHZ(chandef->center_freq1),
- width);
-
- if (r1 < 0)
- return false;
-
- switch (chandef->width) {
- case NL80211_CHAN_WIDTH_80P80:
- WARN_ON(!chandef->center_freq2);
- r2 = cfg80211_get_chans_dfs_usable(wiphy,
- MHZ_TO_KHZ(chandef->center_freq2),
- width);
- if (r2 < 0)
- return false;
- break;
- default:
- WARN_ON(chandef->center_freq2);
- break;
- }
-
- return (r1 + r2 > 0);
+ return count > 0;
}
EXPORT_SYMBOL(cfg80211_chandef_dfs_usable);
@@ -1051,26 +990,29 @@ bool cfg80211_any_wiphy_oper_chan(struct wiphy *wiphy,
return false;
}
-static bool cfg80211_get_chans_dfs_available(struct wiphy *wiphy,
- u32 center_freq,
- u32 bandwidth)
+static bool cfg80211_chandef_dfs_available(struct wiphy *wiphy,
+ const struct cfg80211_chan_def *chandef)
{
struct ieee80211_channel *c;
- u32 freq, start_freq, end_freq;
+ int width;
bool dfs_offload;
+ if (WARN_ON(!cfg80211_chandef_valid(chandef)))
+ return false;
+
+ width = cfg80211_chandef_get_width(chandef);
+ if (width < 0)
+ return false;
+
dfs_offload = wiphy_ext_feature_isset(wiphy,
NL80211_EXT_FEATURE_DFS_OFFLOAD);
- start_freq = cfg80211_get_start_freq(center_freq, bandwidth);
- end_freq = cfg80211_get_end_freq(center_freq, bandwidth);
-
/*
* Check entire range of channels for the bandwidth.
* If any channel in between is disabled or has not
* had gone through CAC return false
*/
- for (freq = start_freq; freq <= end_freq; freq += MHZ_TO_KHZ(20)) {
+ for_each_subchan(chandef, freq, cf) {
c = ieee80211_get_channel_khz(wiphy, freq);
if (!c)
return false;
@@ -1087,124 +1029,54 @@ static bool cfg80211_get_chans_dfs_available(struct wiphy *wiphy,
return true;
}
-static bool cfg80211_chandef_dfs_available(struct wiphy *wiphy,
- const struct cfg80211_chan_def *chandef)
+unsigned int
+cfg80211_chandef_dfs_cac_time(struct wiphy *wiphy,
+ const struct cfg80211_chan_def *chandef)
{
+ struct ieee80211_channel *c;
int width;
- int r;
+ unsigned int t1 = 0, t2 = 0;
if (WARN_ON(!cfg80211_chandef_valid(chandef)))
- return false;
+ return 0;
width = cfg80211_chandef_get_width(chandef);
if (width < 0)
- return false;
-
- r = cfg80211_get_chans_dfs_available(wiphy,
- MHZ_TO_KHZ(chandef->center_freq1),
- width);
-
- /* If any of channels unavailable for cf1 just return */
- if (!r)
- return r;
-
- switch (chandef->width) {
- case NL80211_CHAN_WIDTH_80P80:
- WARN_ON(!chandef->center_freq2);
- r = cfg80211_get_chans_dfs_available(wiphy,
- MHZ_TO_KHZ(chandef->center_freq2),
- width);
- break;
- default:
- WARN_ON(chandef->center_freq2);
- break;
- }
-
- return r;
-}
-
-static unsigned int cfg80211_get_chans_dfs_cac_time(struct wiphy *wiphy,
- u32 center_freq,
- u32 bandwidth)
-{
- struct ieee80211_channel *c;
- u32 start_freq, end_freq, freq;
- unsigned int dfs_cac_ms = 0;
-
- start_freq = cfg80211_get_start_freq(center_freq, bandwidth);
- end_freq = cfg80211_get_end_freq(center_freq, bandwidth);
+ return 0;
- for (freq = start_freq; freq <= end_freq; freq += MHZ_TO_KHZ(20)) {
+ for_each_subchan(chandef, freq, cf) {
c = ieee80211_get_channel_khz(wiphy, freq);
- if (!c)
- return 0;
-
- if (c->flags & IEEE80211_CHAN_DISABLED)
- return 0;
+ if (!c || (c->flags & IEEE80211_CHAN_DISABLED)) {
+ if (cf == 1)
+ t1 = INT_MAX;
+ else
+ t2 = INT_MAX;
+ continue;
+ }
if (!(c->flags & IEEE80211_CHAN_RADAR))
continue;
- if (c->dfs_cac_ms > dfs_cac_ms)
- dfs_cac_ms = c->dfs_cac_ms;
- }
+ if (cf == 1 && c->dfs_cac_ms > t1)
+ t1 = c->dfs_cac_ms;
- return dfs_cac_ms;
-}
-
-unsigned int
-cfg80211_chandef_dfs_cac_time(struct wiphy *wiphy,
- const struct cfg80211_chan_def *chandef)
-{
- int width;
- unsigned int t1 = 0, t2 = 0;
-
- if (WARN_ON(!cfg80211_chandef_valid(chandef)))
- return 0;
+ if (cf == 2 && c->dfs_cac_ms > t2)
+ t2 = c->dfs_cac_ms;
+ }
- width = cfg80211_chandef_get_width(chandef);
- if (width < 0)
+ if (t1 == INT_MAX && t2 == INT_MAX)
return 0;
- t1 = cfg80211_get_chans_dfs_cac_time(wiphy,
- MHZ_TO_KHZ(chandef->center_freq1),
- width);
+ if (t1 == INT_MAX)
+ return t2;
- if (!chandef->center_freq2)
+ if (t2 == INT_MAX)
return t1;
- t2 = cfg80211_get_chans_dfs_cac_time(wiphy,
- MHZ_TO_KHZ(chandef->center_freq2),
- width);
-
return max(t1, t2);
}
EXPORT_SYMBOL(cfg80211_chandef_dfs_cac_time);
-static bool cfg80211_secondary_chans_ok(struct wiphy *wiphy,
- u32 center_freq, u32 bandwidth,
- u32 prohibited_flags,
- u32 permitting_flags)
-{
- struct ieee80211_channel *c;
- u32 freq, start_freq, end_freq;
-
- start_freq = cfg80211_get_start_freq(center_freq, bandwidth);
- end_freq = cfg80211_get_end_freq(center_freq, bandwidth);
-
- for (freq = start_freq; freq <= end_freq; freq += MHZ_TO_KHZ(20)) {
- c = ieee80211_get_channel_khz(wiphy, freq);
- if (!c)
- return false;
- if (c->flags & permitting_flags)
- continue;
- if (c->flags & prohibited_flags)
- return false;
- }
-
- return true;
-}
-
/* check if the operating channels are valid and supported */
static bool cfg80211_edmg_usable(struct wiphy *wiphy, u8 edmg_channels,
enum ieee80211_edmg_bw_config edmg_bw_config,
@@ -1270,6 +1142,7 @@ bool _cfg80211_chandef_usable(struct wiphy *wiphy,
bool ext_nss_cap, support_80_80 = false, support_320 = false;
const struct ieee80211_sband_iftype_data *iftd;
struct ieee80211_supported_band *sband;
+ struct ieee80211_channel *c;
int i;
if (WARN_ON(!cfg80211_chandef_valid(chandef)))
@@ -1420,19 +1293,17 @@ bool _cfg80211_chandef_usable(struct wiphy *wiphy,
if (width < 20)
prohibited_flags |= IEEE80211_CHAN_NO_OFDM;
+ for_each_subchan(chandef, freq, cf) {
+ c = ieee80211_get_channel_khz(wiphy, freq);
+ if (!c)
+ return false;
+ if (c->flags & permitting_flags)
+ continue;
+ if (c->flags & prohibited_flags)
+ return false;
+ }
- if (!cfg80211_secondary_chans_ok(wiphy,
- ieee80211_chandef_to_khz(chandef),
- width, prohibited_flags,
- permitting_flags))
- return false;
-
- if (!chandef->center_freq2)
- return true;
- return cfg80211_secondary_chans_ok(wiphy,
- MHZ_TO_KHZ(chandef->center_freq2),
- width, prohibited_flags,
- permitting_flags);
+ return true;
}
bool cfg80211_chandef_usable(struct wiphy *wiphy,
diff --git a/net/wireless/core.c b/net/wireless/core.c
index 8f7f84c5f440..7c278b869439 100644
--- a/net/wireless/core.c
+++ b/net/wireless/core.c
@@ -1456,6 +1456,8 @@ void cfg80211_init_wdev(struct wireless_dev *wdev)
/* allow mac80211 to determine the timeout */
wdev->ps_timeout = -1;
+ wdev->radio_mask = BIT(wdev->wiphy->n_radio) - 1;
+
if ((wdev->iftype == NL80211_IFTYPE_STATION ||
wdev->iftype == NL80211_IFTYPE_P2P_CLIENT ||
wdev->iftype == NL80211_IFTYPE_ADHOC) && !wdev->use_4addr)
diff --git a/net/wireless/lib80211_crypt_tkip.c b/net/wireless/lib80211_crypt_tkip.c
index 63e68e5e121e..8c82b3a787eb 100644
--- a/net/wireless/lib80211_crypt_tkip.c
+++ b/net/wireless/lib80211_crypt_tkip.c
@@ -38,6 +38,7 @@ MODULE_DESCRIPTION("lib80211 crypt: TKIP");
MODULE_LICENSE("GPL");
#define TKIP_HDR_LEN 8
+#define MICHAEL_MIC_LEN 8
struct lib80211_tkip_data {
#define TKIP_KEY_LEN 32
@@ -585,16 +586,18 @@ static int lib80211_michael_mic_verify(struct sk_buff *skb, int keyidx,
int hdr_len, void *priv)
{
struct lib80211_tkip_data *tkey = priv;
- u8 mic[8];
+ u8 mic[MICHAEL_MIC_LEN];
- if (!tkey->key_set)
+ if (!tkey->key_set || skb->len < hdr_len + MICHAEL_MIC_LEN)
return -1;
michael_mic_hdr(skb, tkey->rx_hdr);
if (michael_mic(tkey->rx_tfm_michael, &tkey->key[24], tkey->rx_hdr,
- skb->data + hdr_len, skb->len - 8 - hdr_len, mic))
+ skb->data + hdr_len,
+ skb->len - MICHAEL_MIC_LEN - hdr_len, mic))
return -1;
- if (memcmp(mic, skb->data + skb->len - 8, 8) != 0) {
+ if (memcmp(mic, skb->data + skb->len - MICHAEL_MIC_LEN,
+ MICHAEL_MIC_LEN) != 0) {
struct ieee80211_hdr *hdr;
hdr = (struct ieee80211_hdr *)skb->data;
printk(KERN_DEBUG "%s: Michael MIC verification failed for "
@@ -612,7 +615,7 @@ static int lib80211_michael_mic_verify(struct sk_buff *skb, int keyidx,
tkey->rx_iv32 = tkey->rx_iv32_new;
tkey->rx_iv16 = tkey->rx_iv16_new;
- skb_trim(skb, skb->len - 8);
+ skb_trim(skb, skb->len - MICHAEL_MIC_LEN);
return 0;
}
diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c
index b302caafd4d3..6261befa1e94 100644
--- a/net/wireless/nl80211.c
+++ b/net/wireless/nl80211.c
@@ -834,6 +834,7 @@ static const struct nla_policy nl80211_policy[NUM_NL80211_ATTR] = {
[NL80211_ATTR_MLO_TTLM_DLINK] = NLA_POLICY_EXACT_LEN(sizeof(u16) * 8),
[NL80211_ATTR_MLO_TTLM_ULINK] = NLA_POLICY_EXACT_LEN(sizeof(u16) * 8),
[NL80211_ATTR_ASSOC_SPP_AMSDU] = { .type = NLA_FLAG },
+ [NL80211_ATTR_VIF_RADIO_MASK] = { .type = NLA_U32 },
};
/* policy for the key attributes */
@@ -3975,7 +3976,8 @@ static int nl80211_send_iface(struct sk_buff *msg, u32 portid, u32 seq, int flag
nla_put_u32(msg, NL80211_ATTR_GENERATION,
rdev->devlist_generation ^
(cfg80211_rdev_list_generation << 2)) ||
- nla_put_u8(msg, NL80211_ATTR_4ADDR, wdev->use_4addr))
+ nla_put_u8(msg, NL80211_ATTR_4ADDR, wdev->use_4addr) ||
+ nla_put_u32(msg, NL80211_ATTR_VIF_RADIO_MASK, wdev->radio_mask))
goto nla_put_failure;
if (rdev->ops->get_channel && !wdev->valid_links) {
@@ -4296,6 +4298,29 @@ static int nl80211_valid_4addr(struct cfg80211_registered_device *rdev,
return -EOPNOTSUPP;
}
+static int nl80211_parse_vif_radio_mask(struct genl_info *info,
+ u32 *radio_mask)
+{
+ struct cfg80211_registered_device *rdev = info->user_ptr[0];
+ struct nlattr *attr = info->attrs[NL80211_ATTR_VIF_RADIO_MASK];
+ u32 mask, allowed;
+
+ if (!attr) {
+ *radio_mask = 0;
+ return 0;
+ }
+
+ allowed = BIT(rdev->wiphy.n_radio) - 1;
+ mask = nla_get_u32(attr);
+ if (mask & ~allowed)
+ return -EINVAL;
+ if (!mask)
+ mask = allowed;
+ *radio_mask = mask;
+
+ return 1;
+}
+
static int nl80211_set_interface(struct sk_buff *skb, struct genl_info *info)
{
struct cfg80211_registered_device *rdev = info->user_ptr[0];
@@ -4303,6 +4328,8 @@ static int nl80211_set_interface(struct sk_buff *skb, struct genl_info *info)
int err;
enum nl80211_iftype otype, ntype;
struct net_device *dev = info->user_ptr[1];
+ struct wireless_dev *wdev = dev->ieee80211_ptr;
+ u32 radio_mask = 0;
bool change = false;
memset(¶ms, 0, sizeof(params));
@@ -4316,8 +4343,6 @@ static int nl80211_set_interface(struct sk_buff *skb, struct genl_info *info)
}
if (info->attrs[NL80211_ATTR_MESH_ID]) {
- struct wireless_dev *wdev = dev->ieee80211_ptr;
-
if (ntype != NL80211_IFTYPE_MESH_POINT)
return -EINVAL;
if (otype != NL80211_IFTYPE_MESH_POINT)
@@ -4348,6 +4373,12 @@ static int nl80211_set_interface(struct sk_buff *skb, struct genl_info *info)
if (err > 0)
change = true;
+ err = nl80211_parse_vif_radio_mask(info, &radio_mask);
+ if (err < 0)
+ return err;
+ if (err && netif_running(dev))
+ return -EBUSY;
+
if (change)
err = cfg80211_change_iface(rdev, dev, ntype, ¶ms);
else
@@ -4356,11 +4387,11 @@ static int nl80211_set_interface(struct sk_buff *skb, struct genl_info *info)
if (!err && params.use_4addr != -1)
dev->ieee80211_ptr->use_4addr = params.use_4addr;
- if (change && !err) {
- struct wireless_dev *wdev = dev->ieee80211_ptr;
+ if (radio_mask)
+ wdev->radio_mask = radio_mask;
+ if (change && !err)
nl80211_notify_iface(rdev, wdev, NL80211_CMD_SET_INTERFACE);
- }
return err;
}
@@ -4371,6 +4402,7 @@ static int _nl80211_new_interface(struct sk_buff *skb, struct genl_info *info)
struct vif_params params;
struct wireless_dev *wdev;
struct sk_buff *msg;
+ u32 radio_mask;
int err;
enum nl80211_iftype type = NL80211_IFTYPE_UNSPECIFIED;
@@ -4408,6 +4440,10 @@ static int _nl80211_new_interface(struct sk_buff *skb, struct genl_info *info)
if (err < 0)
return err;
+ err = nl80211_parse_vif_radio_mask(info, &radio_mask);
+ if (err < 0)
+ return err;
+
msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
if (!msg)
return -ENOMEM;
@@ -4449,6 +4485,9 @@ static int _nl80211_new_interface(struct sk_buff *skb, struct genl_info *info)
break;
}
+ if (radio_mask)
+ wdev->radio_mask = radio_mask;
+
if (nl80211_send_iface(msg, info->snd_portid, info->snd_seq, 0,
rdev, wdev, NL80211_CMD_NEW_INTERFACE) < 0) {
nlmsg_free(msg);
@@ -9178,6 +9217,9 @@ static bool cfg80211_off_channel_oper_allowed(struct wireless_dev *wdev,
lockdep_assert_wiphy(wdev->wiphy);
+ if (!cfg80211_wdev_channel_allowed(wdev, chan))
+ return false;
+
if (!cfg80211_beaconing_iface_active(wdev))
return true;
@@ -9390,7 +9432,8 @@ static int nl80211_trigger_scan(struct sk_buff *skb, struct genl_info *info)
}
/* ignore disabled channels */
- if (chan->flags & IEEE80211_CHAN_DISABLED)
+ if (chan->flags & IEEE80211_CHAN_DISABLED ||
+ !cfg80211_wdev_channel_allowed(wdev, chan))
continue;
request->channels[i] = chan;
@@ -9410,7 +9453,8 @@ static int nl80211_trigger_scan(struct sk_buff *skb, struct genl_info *info)
chan = &wiphy->bands[band]->channels[j];
- if (chan->flags & IEEE80211_CHAN_DISABLED)
+ if (chan->flags & IEEE80211_CHAN_DISABLED ||
+ !cfg80211_wdev_channel_allowed(wdev, chan))
continue;
request->channels[i] = chan;
diff --git a/net/wireless/scan.c b/net/wireless/scan.c
index 3c439841578c..a7d704d0a282 100644
--- a/net/wireless/scan.c
+++ b/net/wireless/scan.c
@@ -953,7 +953,8 @@ static int cfg80211_scan_6ghz(struct cfg80211_registered_device *rdev)
struct ieee80211_channel *chan =
ieee80211_get_channel(&rdev->wiphy, ap->center_freq);
- if (!chan || chan->flags & IEEE80211_CHAN_DISABLED)
+ if (!chan || chan->flags & IEEE80211_CHAN_DISABLED ||
+ !cfg80211_wdev_channel_allowed(rdev_req->wdev, chan))
continue;
for (i = 0; i < rdev_req->n_channels; i++) {
@@ -1960,6 +1961,13 @@ __cfg80211_bss_update(struct cfg80211_registered_device *rdev,
if (!hidden)
hidden = rb_find_bss(rdev, tmp,
BSS_CMP_HIDE_NUL);
+ /*
+ * Only group with an entry with beacon data, otherwise
+ * beacon data can never be filled/updated.
+ */
+ if (hidden &&
+ !rcu_access_pointer(hidden->pub.beacon_ies))
+ hidden = NULL;
if (hidden) {
new->pub.hidden_beacon_bss = &hidden->pub;
list_add(&new->hidden_list,
@@ -3391,11 +3399,6 @@ void cfg80211_update_assoc_bss_entry(struct wireless_dev *wdev,
cbss->pub.channel = chan;
list_for_each_entry(bss, &rdev->bss_list, list) {
- if (!cfg80211_bss_type_match(bss->pub.capability,
- bss->pub.channel->band,
- wdev->conn_bss_type))
- continue;
-
if (bss == cbss)
continue;
@@ -3517,9 +3520,12 @@ int cfg80211_wext_siwscan(struct net_device *dev,
continue;
for (j = 0; j < wiphy->bands[band]->n_channels; j++) {
+ struct ieee80211_channel *chan;
+
/* ignore disabled channels */
- if (wiphy->bands[band]->channels[j].flags &
- IEEE80211_CHAN_DISABLED)
+ chan = &wiphy->bands[band]->channels[j];
+ if (chan->flags & IEEE80211_CHAN_DISABLED ||
+ !cfg80211_wdev_channel_allowed(creq->wdev, chan))
continue;
/* If we have a wireless request structure and the
diff --git a/net/wireless/util.c b/net/wireless/util.c
index 8478f3a0e940..66f95044adb2 100644
--- a/net/wireless/util.c
+++ b/net/wireless/util.c
@@ -5,7 +5,7 @@
* Copyright 2007-2009 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
* Copyright 2017 Intel Deutschland GmbH
- * Copyright (C) 2018-2023 Intel Corporation
+ * Copyright (C) 2018-2023, 2025 Intel Corporation
*/
#include <linux/export.h>
#include <linux/bitops.h>
@@ -989,12 +989,30 @@ unsigned int cfg80211_classify8021d(struct sk_buff *skb,
}
switch (skb->protocol) {
- case htons(ETH_P_IP):
- dscp = ipv4_get_dsfield(ip_hdr(skb)) & 0xfc;
+ case htons(ETH_P_IP): {
+ const struct iphdr *iph;
+ struct iphdr _iph;
+
+ iph = skb_header_pointer(skb, sizeof(struct ethhdr),
+ sizeof(*iph), &_iph);
+ if (!iph)
+ return 0;
+
+ dscp = ipv4_get_dsfield(iph) & 0xfc;
break;
- case htons(ETH_P_IPV6):
- dscp = ipv6_get_dsfield(ipv6_hdr(skb)) & 0xfc;
+ }
+ case htons(ETH_P_IPV6): {
+ const struct ipv6hdr *ip6h;
+ struct ipv6hdr _ip6h;
+
+ ip6h = skb_header_pointer(skb, sizeof(struct ethhdr),
+ sizeof(*ip6h), &_ip6h);
+ if (!ip6h)
+ return 0;
+
+ dscp = ipv6_get_dsfield(ip6h) & 0xfc;
break;
+ }
case htons(ETH_P_MPLS_UC):
case htons(ETH_P_MPLS_MC): {
struct mpls_label mpls_tmp, *mpls;
@@ -2355,16 +2373,15 @@ static void cfg80211_calculate_bi_data(struct wiphy *wiphy, u32 new_beacon_int,
if (wdev->valid_links)
continue;
+ wdev_bi = cfg80211_wdev_bi(wdev);
+ if (!wdev_bi)
+ continue;
+
/* skip wdevs not active on the given wiphy radio */
if (radio_idx >= 0 &&
!(rdev_get_radio_mask(rdev, wdev->netdev) & BIT(radio_idx)))
continue;
- wdev_bi = cfg80211_wdev_bi(wdev);
-
- if (!wdev_bi)
- continue;
-
if (!*beacon_int_gcd) {
*beacon_int_gcd = wdev_bi;
continue;
@@ -2937,7 +2954,7 @@ bool cfg80211_radio_chandef_valid(const struct wiphy_radio *radio,
u32 freq, width;
freq = ieee80211_chandef_to_khz(chandef);
- width = nl80211_chan_width_to_mhz(chandef->width);
+ width = cfg80211_chandef_get_width(chandef);
if (!ieee80211_radio_freq_range_valid(radio, freq, width))
return false;
@@ -2948,3 +2965,32 @@ bool cfg80211_radio_chandef_valid(const struct wiphy_radio *radio,
return true;
}
EXPORT_SYMBOL(cfg80211_radio_chandef_valid);
+
+bool cfg80211_wdev_channel_allowed(struct wireless_dev *wdev,
+ struct ieee80211_channel *chan)
+{
+ struct wiphy *wiphy = wdev->wiphy;
+ const struct wiphy_radio *radio;
+ struct cfg80211_chan_def chandef;
+ u32 radio_mask;
+ int i;
+
+ radio_mask = wdev->radio_mask;
+ if (!wiphy->n_radio || radio_mask == BIT(wiphy->n_radio) - 1)
+ return true;
+
+ cfg80211_chandef_create(&chandef, chan, NL80211_CHAN_HT20);
+ for (i = 0; i < wiphy->n_radio; i++) {
+ if (!(radio_mask & BIT(i)))
+ continue;
+
+ radio = &wiphy->radio[i];
+ if (!cfg80211_radio_chandef_valid(radio, &chandef))
+ continue;
+
+ return true;
+ }
+
+ return false;
+}
+EXPORT_SYMBOL(cfg80211_wdev_channel_allowed);
diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c
index afa457506274..607f8b7e16d9 100644
--- a/net/xdp/xskmap.c
+++ b/net/xdp/xskmap.c
@@ -124,7 +124,7 @@ static int xsk_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
struct bpf_insn *insn = insn_buf;
*insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
- *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 5);
+ *insn++ = BPF_JMP32_IMM(BPF_JGE, ret, map->max_entries, 5);
*insn++ = BPF_ALU64_IMM(BPF_LSH, ret, ilog2(sizeof(struct xsk_sock *)));
*insn++ = BPF_ALU64_IMM(BPF_ADD, mp, offsetof(struct xsk_map, xsk_map));
*insn++ = BPF_ALU64_REG(BPF_ADD, ret, mp);
diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c
index f8190b4c5b8d..c4291ea86d1d 100644
--- a/net/xfrm/espintcp.c
+++ b/net/xfrm/espintcp.c
@@ -33,7 +33,11 @@ static void handle_esp(struct sk_buff *skb, struct sock *sk)
{
struct tcp_skb_cb *tcp_cb = (struct tcp_skb_cb *)skb->cb;
- skb_reset_transport_header(skb);
+ if (!skb_reset_transport_header_careful(skb)) {
+ XFRM_INC_STATS(sock_net(sk), LINUX_MIB_XFRMINERROR);
+ kfree_skb(skb);
+ return;
+ }
/* restore IP CB, we need at least IP6CB->nhoff */
memmove(skb->cb, &tcp_cb->header, sizeof(tcp_cb->header));
diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c
index 5d3633ce6ba3..63d1504b50a9 100644
--- a/net/xfrm/xfrm_input.c
+++ b/net/xfrm/xfrm_input.c
@@ -457,12 +457,14 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
{
const struct xfrm_state_afinfo *afinfo;
struct net *net = dev_net(skb->dev);
+ struct net_device *dev = skb->dev;
int err;
__be32 seq;
__be32 seq_hi;
struct xfrm_state *x = NULL;
xfrm_address_t *daddr;
u32 mark = skb->mark;
+ u8 xfrm_proto = nexthdr;
unsigned int family = AF_UNSPEC;
int decaps = 0;
int async = 0;
@@ -474,6 +476,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
if (encap_type < 0 || (xo && (xo->flags & XFRM_GRO || encap_type == 0 ||
encap_type == UDP_ENCAP_ESPINUDP))) {
x = xfrm_input_state(skb);
+ xfrm_proto = x->type ? x->type->proto : nexthdr;
if (unlikely(x->km.state != XFRM_STATE_VALID)) {
if (x->km.state == XFRM_STATE_ACQ)
@@ -483,7 +486,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
LINUX_MIB_XFRMINSTATEINVALID);
if (encap_type == -1)
- dev_put(skb->dev);
+ dev_put(dev);
goto drop;
}
@@ -493,6 +496,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
if (encap_type == -1) {
async = 1;
seq = XFRM_SKB_CB(skb)->seq.input.low;
+ spin_lock(&x->lock);
goto resume;
}
/* GRO call */
@@ -529,6 +533,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
XFRM_INC_STATS(net, LINUX_MIB_XFRMINHDRERROR);
goto drop;
}
+
+ nexthdr = x->type_offload->input_tail(x, skb);
}
goto lock;
@@ -574,11 +580,13 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
x = xfrm_input_state_lookup(net, mark, daddr, spi, nexthdr, family);
if (x == NULL) {
+ xfrm_proto = nexthdr;
secpath_reset(skb);
XFRM_INC_STATS(net, LINUX_MIB_XFRMINNOSTATES);
xfrm_audit_state_notfound(skb, family, spi, seq);
goto drop;
}
+ xfrm_proto = x->type ? x->type->proto : nexthdr;
if (unlikely(x->dir && x->dir != XFRM_SA_DIR_IN)) {
secpath_reset(skb);
@@ -586,6 +594,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
xfrm_audit_state_notfound(skb, family, spi, seq);
xfrm_state_put(x);
x = NULL;
+ xfrm_proto = nexthdr;
goto drop;
}
@@ -626,11 +635,9 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
goto drop_unlock;
}
- spin_unlock(&x->lock);
-
if (xfrm_tunnel_check(skb, x, family)) {
XFRM_INC_STATS(net, LINUX_MIB_XFRMINSTATEMODEERROR);
- goto drop;
+ goto drop_unlock;
}
seq_hi = htonl(xfrm_replay_seqhi(x, seq));
@@ -638,22 +645,21 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
XFRM_SKB_CB(skb)->seq.input.low = seq;
XFRM_SKB_CB(skb)->seq.input.hi = seq_hi;
- if (crypto_done) {
- nexthdr = x->type_offload->input_tail(x, skb);
- } else {
- dev_hold(skb->dev);
+ if (!crypto_done) {
+ spin_unlock(&x->lock);
+ dev_hold(dev);
nexthdr = x->type->input(x, skb);
if (nexthdr == -EINPROGRESS) {
if (async)
- dev_put(skb->dev);
+ dev_put(dev);
return 0;
}
- dev_put(skb->dev);
+ dev_put(dev);
+ spin_lock(&x->lock);
}
resume:
- spin_lock(&x->lock);
if (nexthdr < 0) {
if (nexthdr == -EBADMSG) {
xfrm_audit_state_icvfail(x, skb,
@@ -707,9 +713,12 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
crypto_done = false;
} while (!err);
- err = xfrm_rcv_cb(skb, family, x->type->proto, 0);
- if (err)
+ rcu_read_lock();
+ err = xfrm_rcv_cb(skb, family, xfrm_proto, 0);
+ if (err) {
+ rcu_read_unlock();
goto drop;
+ }
nf_reset_ct(skb);
@@ -720,8 +729,9 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
if (skb_valid_dst(skb))
skb_dst_drop(skb);
if (async)
- dev_put(skb->dev);
+ dev_put(dev);
gro_cells_receive(&gro_cells, skb);
+ rcu_read_unlock();
return 0;
} else {
xo = xfrm_offload(skb);
@@ -729,23 +739,21 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
xfrm_gro = xo->flags & XFRM_GRO;
err = -EAFNOSUPPORT;
- rcu_read_lock();
- afinfo = xfrm_state_afinfo_get_rcu(x->props.family);
+ afinfo = xfrm_state_afinfo_get_rcu(family);
if (likely(afinfo))
err = afinfo->transport_finish(skb, xfrm_gro || async);
- rcu_read_unlock();
if (xfrm_gro) {
sp = skb_sec_path(skb);
if (sp)
sp->olen = 0;
if (skb_valid_dst(skb))
skb_dst_drop(skb);
- if (async)
- dev_put(skb->dev);
gro_cells_receive(&gro_cells, skb);
- return err;
}
+ if (async)
+ dev_put(dev);
+ rcu_read_unlock();
return err;
}
@@ -753,8 +761,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
spin_unlock(&x->lock);
drop:
if (async)
- dev_put(skb->dev);
- xfrm_rcv_cb(skb, family, x && x->type ? x->type->proto : nexthdr, -1);
+ dev_put(dev);
+ xfrm_rcv_cb(skb, family, xfrm_proto, -1);
kfree_skb(skb);
return 0;
}
@@ -778,12 +786,17 @@ static void xfrm_trans_reinject(struct work_struct *work)
spin_unlock_bh(&trans->queue_lock);
local_bh_disable();
+ rcu_read_lock();
while ((skb = __skb_dequeue(&queue))) {
struct net *net = XFRM_TRANS_SKB_CB(skb)->net;
+ struct net_device *dev = skb->dev;
XFRM_TRANS_SKB_CB(skb)->finish(net, NULL, skb);
+ if (dev)
+ dev_put(dev);
put_net(net);
}
+ rcu_read_unlock();
local_bh_enable();
}
@@ -799,12 +812,18 @@ int xfrm_trans_queue_net(struct net *net, struct sk_buff *skb,
if (skb_queue_len(&trans->queue) >= READ_ONCE(net_hotdata.max_backlog))
return -ENOBUFS;
+ if (skb_dst(skb) && !skb_dst_force(skb))
+ return -EHOSTUNREACH;
+
BUILD_BUG_ON(sizeof(struct xfrm_trans_cb) > sizeof(skb->cb));
hold_net = maybe_get_net(net);
if (!hold_net)
return -ENODEV;
+ if (skb->dev)
+ dev_hold(skb->dev);
+
XFRM_TRANS_SKB_CB(skb)->finish = finish;
XFRM_TRANS_SKB_CB(skb)->net = hold_net;
spin_lock_bh(&trans->queue_lock);
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index e399cd5696cf..701b5163b6b1 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -226,6 +226,7 @@ static struct xfrm_state_afinfo __rcu *xfrm_state_afinfo[NPROTO];
static DEFINE_SPINLOCK(xfrm_state_gc_lock);
static DEFINE_SPINLOCK(xfrm_state_dev_gc_lock);
+static DEFINE_MUTEX(xfrm_state_gc_mutex);
int __xfrm_state_delete(struct xfrm_state *x);
@@ -570,8 +571,10 @@ static void xfrm_state_gc_task(struct work_struct *work)
synchronize_rcu();
+ mutex_lock(&xfrm_state_gc_mutex);
hlist_for_each_entry_safe(x, tmp, &gc_list, gclist)
xfrm_state_gc_destroy(x);
+ mutex_unlock(&xfrm_state_gc_mutex);
}
static enum hrtimer_restart xfrm_timer_handler(struct hrtimer *me)
@@ -760,9 +763,9 @@ int __xfrm_state_delete(struct xfrm_state *x)
if (!hlist_unhashed(&x->byseq))
hlist_del_init_rcu(&x->byseq);
if (!hlist_unhashed(&x->state_cache))
- hlist_del_rcu(&x->state_cache);
+ hlist_del_init_rcu(&x->state_cache);
if (!hlist_unhashed(&x->state_cache_input))
- hlist_del_rcu(&x->state_cache_input);
+ hlist_del_init_rcu(&x->state_cache_input);
if (!hlist_unhashed(&x->byspi))
hlist_del_init_rcu(&x->byspi);
@@ -937,6 +940,7 @@ int xfrm_dev_state_flush(struct net *net, struct net_device *dev, bool task_vali
out:
spin_unlock_bh(&net->xfrm.xfrm_state_lock);
+ mutex_lock(&xfrm_state_gc_mutex);
spin_lock_bh(&xfrm_state_dev_gc_lock);
restart_gc:
hlist_for_each_entry_safe(x, tmp, &xfrm_state_dev_gc_list, dev_gclist) {
@@ -951,6 +955,7 @@ int xfrm_dev_state_flush(struct net *net, struct net_device *dev, bool task_vali
}
spin_unlock_bh(&xfrm_state_dev_gc_lock);
+ mutex_unlock(&xfrm_state_gc_mutex);
xfrm_flush_gc();
diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c
index 904c9328852f..c1e753d554e7 100644
--- a/net/xfrm/xfrm_user.c
+++ b/net/xfrm/xfrm_user.c
@@ -35,6 +35,15 @@
#endif
#include <linux/unaligned.h>
+static struct sock *xfrm_net_nlsk(const struct net *net, const struct sk_buff *skb)
+{
+ /* get the source of this request, see netlink_unicast_kernel */
+ const struct sock *sk = NETLINK_CB(skb).sk;
+
+ /* sk is refcounted, the netns stays alive and nlsk with it */
+ return rcu_dereference_protected(net->xfrm.nlsk, sk->sk_net_refcnt);
+}
+
static int verify_one_alg(struct nlattr **attrs, enum xfrm_attr_type_t type,
struct netlink_ext_ack *extack)
{
@@ -1662,7 +1671,7 @@ static int xfrm_get_spdinfo(struct sk_buff *skb, struct nlmsghdr *nlh,
err = build_spdinfo(r_skb, net, sportid, seq, *flags);
BUG_ON(err < 0);
- return nlmsg_unicast(net->xfrm.nlsk, r_skb, sportid);
+ return nlmsg_unicast(xfrm_net_nlsk(net, skb), r_skb, sportid);
}
static inline unsigned int xfrm_sadinfo_msgsize(void)
@@ -1722,7 +1731,7 @@ static int xfrm_get_sadinfo(struct sk_buff *skb, struct nlmsghdr *nlh,
err = build_sadinfo(r_skb, net, sportid, seq, *flags);
BUG_ON(err < 0);
- return nlmsg_unicast(net->xfrm.nlsk, r_skb, sportid);
+ return nlmsg_unicast(xfrm_net_nlsk(net, skb), r_skb, sportid);
}
static int xfrm_get_sa(struct sk_buff *skb, struct nlmsghdr *nlh,
@@ -1742,7 +1751,7 @@ static int xfrm_get_sa(struct sk_buff *skb, struct nlmsghdr *nlh,
if (IS_ERR(resp_skb)) {
err = PTR_ERR(resp_skb);
} else {
- err = nlmsg_unicast(net->xfrm.nlsk, resp_skb, NETLINK_CB(skb).portid);
+ err = nlmsg_unicast(xfrm_net_nlsk(net, skb), resp_skb, NETLINK_CB(skb).portid);
}
xfrm_state_put(x);
out_noput:
@@ -1756,7 +1765,6 @@ static int xfrm_alloc_userspi(struct sk_buff *skb, struct nlmsghdr *nlh,
struct net *net = sock_net(skb->sk);
struct xfrm_state *x;
struct xfrm_userspi_info *p;
- struct xfrm_translator *xtr;
struct sk_buff *resp_skb;
xfrm_address_t *daddr;
int family;
@@ -1822,18 +1830,7 @@ static int xfrm_alloc_userspi(struct sk_buff *skb, struct nlmsghdr *nlh,
goto out;
}
- xtr = xfrm_get_translator();
- if (xtr) {
- err = xtr->alloc_compat(skb, nlmsg_hdr(skb));
-
- xfrm_put_translator(xtr);
- if (err) {
- kfree_skb(resp_skb);
- goto out;
- }
- }
-
- err = nlmsg_unicast(net->xfrm.nlsk, resp_skb, NETLINK_CB(skb).portid);
+ err = nlmsg_unicast(xfrm_net_nlsk(net, skb), resp_skb, NETLINK_CB(skb).portid);
out:
xfrm_state_put(x);
@@ -2476,7 +2473,7 @@ static int xfrm_get_default(struct sk_buff *skb, struct nlmsghdr *nlh,
r_up->out = READ_ONCE(net->xfrm.policy_default[XFRM_POLICY_OUT]);
nlmsg_end(r_skb, r_nlh);
- return nlmsg_unicast(net->xfrm.nlsk, r_skb, portid);
+ return nlmsg_unicast(xfrm_net_nlsk(net, skb), r_skb, portid);
}
static int xfrm_get_policy(struct sk_buff *skb, struct nlmsghdr *nlh,
@@ -2542,7 +2539,7 @@ static int xfrm_get_policy(struct sk_buff *skb, struct nlmsghdr *nlh,
if (IS_ERR(resp_skb)) {
err = PTR_ERR(resp_skb);
} else {
- err = nlmsg_unicast(net->xfrm.nlsk, resp_skb,
+ err = nlmsg_unicast(xfrm_net_nlsk(net, skb), resp_skb,
NETLINK_CB(skb).portid);
}
} else {
@@ -2721,7 +2718,7 @@ static int xfrm_get_ae(struct sk_buff *skb, struct nlmsghdr *nlh,
return err;
}
- err = nlmsg_unicast(net->xfrm.nlsk, r_skb, NETLINK_CB(skb).portid);
+ err = nlmsg_unicast(xfrm_net_nlsk(net, skb), r_skb, NETLINK_CB(skb).portid);
spin_unlock_bh(&x->lock);
xfrm_state_put(x);
return err;
@@ -3393,7 +3390,7 @@ static int xfrm_user_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
goto err;
}
- err = netlink_dump_start(net->xfrm.nlsk, skb, nlh, &c);
+ err = netlink_dump_start(xfrm_net_nlsk(net, skb), skb, nlh, &c);
goto err;
}
diff --git a/security/ipe/eval.c b/security/ipe/eval.c
index 21439c5be336..4a7be96c84f0 100644
--- a/security/ipe/eval.c
+++ b/security/ipe/eval.c
@@ -134,10 +134,14 @@ static bool evaluate_boot_verified(const struct ipe_eval_ctx *const ctx)
static bool evaluate_dmv_roothash(const struct ipe_eval_ctx *const ctx,
struct ipe_prop *p)
{
- return !!ctx->ipe_bdev &&
- !!ctx->ipe_bdev->root_hash &&
- ipe_digest_eval(p->value,
- ctx->ipe_bdev->root_hash);
+ const struct digest_info *root_hash;
+
+ if (!ctx->ipe_bdev)
+ return false;
+
+ root_hash = rcu_dereference(ctx->ipe_bdev->root_hash);
+
+ return root_hash && ipe_digest_eval(p->value, root_hash);
}
#else
static bool evaluate_dmv_roothash(const struct ipe_eval_ctx *const ctx,
diff --git a/security/ipe/eval.h b/security/ipe/eval.h
index fef65a36468c..b26d1147d070 100644
--- a/security/ipe/eval.h
+++ b/security/ipe/eval.h
@@ -27,7 +27,7 @@ struct ipe_bdev {
#ifdef CONFIG_IPE_PROP_DM_VERITY_SIGNATURE
bool dm_verity_signed;
#endif /* CONFIG_IPE_PROP_DM_VERITY_SIGNATURE */
- struct digest_info *root_hash;
+ struct digest_info __rcu *root_hash;
};
#endif /* CONFIG_IPE_PROP_DM_VERITY */
diff --git a/security/ipe/fs.c b/security/ipe/fs.c
index 5b6d19fb844a..d6f90235d614 100644
--- a/security/ipe/fs.c
+++ b/security/ipe/fs.c
@@ -155,10 +155,6 @@ static ssize_t new_policy(struct file *f, const char __user *data,
}
rc = ipe_new_policyfs_node(p);
- if (rc)
- goto out;
-
- ipe_audit_policy_load(p);
out:
if (rc < 0)
diff --git a/security/ipe/hooks.c b/security/ipe/hooks.c
index d0323b81cd8f..d3092933abe2 100644
--- a/security/ipe/hooks.c
+++ b/security/ipe/hooks.c
@@ -9,6 +9,7 @@
#include <linux/binfmts.h>
#include <linux/mman.h>
#include <linux/blk_types.h>
+#include <linux/rcupdate.h>
#include "ipe.h"
#include "hooks.h"
@@ -204,7 +205,20 @@ void ipe_bdev_free_security(struct block_device *bdev)
{
struct ipe_bdev *blob = ipe_bdev(bdev);
- ipe_digest_free(blob->root_hash);
+ ipe_digest_free(rcu_access_pointer(blob->root_hash));
+}
+
+static void ipe_set_dmverity_roothash(struct ipe_bdev *blob,
+ struct digest_info *info)
+{
+ struct digest_info *old;
+
+ /* Protected by device-mapper's md->suspend_lock */
+ old = rcu_replace_pointer(blob->root_hash, info, true);
+ if (old) {
+ synchronize_rcu();
+ ipe_digest_free(old);
+ }
}
#ifdef CONFIG_IPE_PROP_DM_VERITY_SIGNATURE
@@ -252,8 +266,7 @@ int ipe_bdev_setintegrity(struct block_device *bdev, enum lsm_integrity_type typ
return -EINVAL;
if (!value) {
- ipe_digest_free(blob->root_hash);
- blob->root_hash = NULL;
+ ipe_set_dmverity_roothash(blob, NULL);
return 0;
}
@@ -273,8 +286,7 @@ int ipe_bdev_setintegrity(struct block_device *bdev, enum lsm_integrity_type typ
info->digest_len = digest->digest_len;
- ipe_digest_free(blob->root_hash);
- blob->root_hash = info;
+ ipe_set_dmverity_roothash(blob, info);
return 0;
err:
diff --git a/security/ipe/policy_fs.c b/security/ipe/policy_fs.c
index 3bcd8cbd09df..5eba2bb93276 100644
--- a/security/ipe/policy_fs.c
+++ b/security/ipe/policy_fs.c
@@ -12,6 +12,7 @@
#include "policy.h"
#include "eval.h"
#include "fs.h"
+#include "audit.h"
#define MAX_VERSION_SIZE ARRAY_SIZE("65535.65535.65535")
@@ -463,6 +464,9 @@ int ipe_new_policyfs_node(struct ipe_policy *p)
inode_lock(root);
p->policyfs = policyfs;
root->i_private = p;
+ /* Only audit signed policies from userspace */
+ if (p->pkcs7)
+ ipe_audit_policy_load(p);
inode_unlock(root);
return 0;
diff --git a/security/keys/encrypted-keys/encrypted.c b/security/keys/encrypted-keys/encrypted.c
index 831cb84fd75a..daf8c769e3be 100644
--- a/security/keys/encrypted-keys/encrypted.c
+++ b/security/keys/encrypted-keys/encrypted.c
@@ -18,6 +18,7 @@
#include <linux/parser.h>
#include <linux/string.h>
#include <linux/err.h>
+#include <linux/overflow.h>
#include <keys/user-type.h>
#include <keys/trusted-type.h>
#include <keys/encrypted-type.h>
@@ -606,6 +607,7 @@ static struct encrypted_key_payload *encrypted_key_alloc(struct key *key,
{
struct encrypted_key_payload *epayload = NULL;
unsigned short datablob_len;
+ unsigned short payload_totallen;
unsigned short decrypted_datalen;
unsigned short payload_datalen;
unsigned int encrypted_datalen;
@@ -659,16 +661,22 @@ static struct encrypted_key_payload *encrypted_key_alloc(struct key *key,
encrypted_datalen = roundup(decrypted_datalen, blksize);
- datablob_len = format_len + 1 + strlen(master_desc) + 1
- + strlen(datalen) + 1 + ivsize + 1 + encrypted_datalen;
+ if (check_add_overflow(format_len + 1 + strlen(master_desc) + 1
+ + strlen(datalen) + 1 + ivsize + 1,
+ encrypted_datalen, &datablob_len))
+ return ERR_PTR(-EINVAL);
+
+ if (check_add_overflow(datablob_len,
+ payload_datalen + HASH_SIZE + 1,
+ &payload_totallen))
+ return ERR_PTR(-EINVAL);
- ret = key_payload_reserve(key, payload_datalen + datablob_len
- + HASH_SIZE + 1);
+ ret = key_payload_reserve(key, payload_totallen);
if (ret < 0)
return ERR_PTR(ret);
- epayload = kzalloc(sizeof(*epayload) + payload_datalen +
- datablob_len + HASH_SIZE + 1, GFP_KERNEL);
+ epayload = kzalloc_flex(*epayload, payload_data, payload_totallen,
+ GFP_KERNEL);
if (!epayload)
return ERR_PTR(-ENOMEM);
diff --git a/security/keys/gc.c b/security/keys/gc.c
index f27223ea4578..cf71b26c0c00 100644
--- a/security/keys/gc.c
+++ b/security/keys/gc.c
@@ -318,9 +318,7 @@ static void key_garbage_collector(struct work_struct *work)
if (unlikely(gc_state & KEY_GC_REAPING_DEAD_3)) {
kdebug("dead wake");
- smp_mb();
- clear_bit(KEY_GC_REAPING_KEYTYPE, &key_gc_flags);
- wake_up_bit(&key_gc_flags, KEY_GC_REAPING_KEYTYPE);
+ clear_and_wake_up_bit(KEY_GC_REAPING_KEYTYPE, &key_gc_flags);
}
if (gc_state & KEY_GC_REAP_AGAIN)
diff --git a/security/keys/request_key_auth.c b/security/keys/request_key_auth.c
index bd34317b58cc..0f385422ddf9 100644
--- a/security/keys/request_key_auth.c
+++ b/security/keys/request_key_auth.c
@@ -9,6 +9,8 @@
#include <linux/sched.h>
#include <linux/err.h>
+#include <linux/pid.h>
+#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
@@ -73,7 +75,10 @@ static void request_key_auth_describe(const struct key *key,
seq_puts(m, "key:");
seq_puts(m, key->description);
if (key_is_positive(key))
- seq_printf(m, " pid:%d ci:%zu", rka->pid, rka->callout_len);
+ seq_printf(m, " pid:%d ci:%zu",
+ pid_nr_ns(rka->pid,
+ proc_pid_ns(file_inode(m->file)->i_sb)),
+ rka->callout_len);
}
/*
@@ -113,6 +118,7 @@ static void free_request_key_auth(struct request_key_auth *rka)
if (rka->cred)
put_cred(rka->cred);
kfree(rka->callout_info);
+ put_pid(rka->pid);
kfree(rka);
}
@@ -226,14 +232,14 @@ struct key *request_key_auth_new(struct key *target, const char *op,
irka = cred->request_key_auth->payload.data[0];
rka->cred = get_cred(irka->cred);
- rka->pid = irka->pid;
+ rka->pid = get_pid(irka->pid);
up_read(&cred->request_key_auth->sem);
}
else {
/* it isn't - use this process as the context */
rka->cred = get_cred(cred);
- rka->pid = current->pid;
+ rka->pid = get_pid(task_pid(current));
}
rka->target_key = key_get(target);
diff --git a/security/keys/trusted-keys/trusted_tpm2.c b/security/keys/trusted-keys/trusted_tpm2.c
index 74cea80ed9be..aae81332e33d 100644
--- a/security/keys/trusted-keys/trusted_tpm2.c
+++ b/security/keys/trusted-keys/trusted_tpm2.c
@@ -108,7 +108,7 @@ struct tpm2_key_context {
static int tpm2_key_decode(struct trusted_key_payload *payload,
struct trusted_key_options *options,
- u8 **buf)
+ u8 **buf, unsigned int *blob_len)
{
int ret;
struct tpm2_key_context ctx;
@@ -129,6 +129,7 @@ static int tpm2_key_decode(struct trusted_key_payload *payload,
return -ENOMEM;
*buf = blob;
+ *blob_len = ctx.priv_len + ctx.pub_len;
options->keyhandle = ctx.parent;
memcpy(blob, ctx.priv, ctx.priv_len);
@@ -402,10 +403,11 @@ static int tpm2_load_cmd(struct tpm_chip *chip,
int rc;
u32 attrs;
- rc = tpm2_key_decode(payload, options, &blob);
+ rc = tpm2_key_decode(payload, options, &blob, &blob_len);
if (rc) {
/* old form */
blob = payload->blob;
+ blob_len = payload->blob_len;
payload->old_format = 1;
} else {
/* Bind for cleanup: */
@@ -417,17 +419,17 @@ static int tpm2_load_cmd(struct tpm_chip *chip,
return -EINVAL;
/* must be big enough for at least the two be16 size counts */
- if (payload->blob_len < 4)
+ if (blob_len < 4)
return -EINVAL;
private_len = get_unaligned_be16(blob);
/* must be big enough for following public_len */
- if (private_len + 2 + 2 > (payload->blob_len))
+ if (private_len + 2 + 2 > blob_len)
return -E2BIG;
public_len = get_unaligned_be16(blob + 2 + private_len);
- if (private_len + 2 + public_len + 2 > payload->blob_len)
+ if (private_len + 2 + public_len + 2 > blob_len)
return -E2BIG;
pub = blob + 2 + private_len + 2;
diff --git a/security/landlock/fs.c b/security/landlock/fs.c
index b96a4daa5b2a..5b852c2965cf 100644
--- a/security/landlock/fs.c
+++ b/security/landlock/fs.c
@@ -1192,11 +1192,12 @@ static int current_check_refer_path(struct dentry *const old_dentry,
/*
* old_dentry may be the root of the common mount point and
* !IS_ROOT(old_dentry) at the same time (e.g. with open_tree() and
- * OPEN_TREE_CLONE). We do not need to call dget(old_parent) because
- * we keep a reference to old_dentry.
+ * OPEN_TREE_CLONE). Pin the dentry used as old_parent in either case.
+ * Otherwise, dget_parent() safely fetches and pins the current parent
+ * against a concurrent rename(2).
*/
- old_parent = (old_dentry == mnt_dir.dentry) ? old_dentry :
- old_dentry->d_parent;
+ old_parent = (old_dentry == mnt_dir.dentry) ? dget(old_dentry) :
+ dget_parent(old_dentry);
/* new_dir->dentry is equal to new_dentry->d_parent */
allow_parent1 = collect_domain_accesses(dom, mnt_dir.dentry, old_parent,
@@ -1204,8 +1205,10 @@ static int current_check_refer_path(struct dentry *const old_dentry,
allow_parent2 = collect_domain_accesses(
dom, mnt_dir.dentry, new_dir->dentry, &layer_masks_parent2);
- if (allow_parent1 && allow_parent2)
+ if (allow_parent1 && allow_parent2) {
+ dput(old_parent);
return 0;
+ }
/*
* To be able to compare source and destination domain access rights,
@@ -1216,8 +1219,11 @@ static int current_check_refer_path(struct dentry *const old_dentry,
if (is_access_to_paths_allowed(
dom, &mnt_dir, access_request_parent1, &layer_masks_parent1,
old_dentry, access_request_parent2, &layer_masks_parent2,
- exchange ? new_dentry : NULL))
+ exchange ? new_dentry : NULL)) {
+ dput(old_parent);
return 0;
+ }
+ dput(old_parent);
/*
* This prioritizes EACCES over EXDEV for all actions, including
diff --git a/security/landlock/net.c b/security/landlock/net.c
index 9c9608924fbd..c5c26e902992 100644
--- a/security/landlock/net.c
+++ b/security/landlock/net.c
@@ -193,9 +193,23 @@ static int hook_socket_connect(struct socket *const sock,
LANDLOCK_ACCESS_NET_CONNECT_TCP);
}
+static int hook_socket_sendmsg(struct socket *const sock,
+ struct msghdr *const msg, const int size)
+{
+ struct sockaddr *const address = msg->msg_name;
+
+ if ((msg->msg_flags & MSG_FASTOPEN) && address)
+ return current_check_access_socket(
+ sock, address, msg->msg_namelen,
+ LANDLOCK_ACCESS_NET_CONNECT_TCP);
+
+ return 0;
+}
+
static struct security_hook_list landlock_hooks[] __ro_after_init = {
LSM_HOOK_INIT(socket_bind, hook_socket_bind),
LSM_HOOK_INIT(socket_connect, hook_socket_connect),
+ LSM_HOOK_INIT(socket_sendmsg, hook_socket_sendmsg),
};
__init void landlock_add_net_hooks(void)
diff --git a/security/selinux/avc.c b/security/selinux/avc.c
index cc0b0af20296..5f8b383f4bfb 100644
--- a/security/selinux/avc.c
+++ b/security/selinux/avc.c
@@ -1143,8 +1143,11 @@ inline int avc_has_perm_noaudit(u32 ssid, u32 tsid,
u32 denied;
struct avc_node *node;
- if (WARN_ON(!requested))
+ if (WARN_ON(!requested)) {
+ /* Provide a deny-all, audit-all decision to the caller. */
+ *avd = (struct av_decision){ .auditdeny = 0xffffffff };
return -EACCES;
+ }
rcu_read_lock();
node = avc_lookup(ssid, tsid, tclass);
diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c
index cd60ece2fd58..3543c4fddb05 100644
--- a/security/selinux/hooks.c
+++ b/security/selinux/hooks.c
@@ -1656,26 +1656,32 @@ static int cred_has_capability(const struct cred *cred,
return rc;
}
-/* Check whether a task has a particular permission to an inode.
- The 'adp' parameter is optional and allows other audit
- data to be passed (e.g. the dentry). */
-static int inode_has_perm(const struct cred *cred,
- struct inode *inode,
- u32 perms,
- struct common_audit_data *adp)
+/*
+ * Check whether a SID has a particular permission to an inode. The 'adp'
+ * parameter is optional and allows other audit data to be passed (e.g. the
+ * dentry).
+ */
+static int inode_sid_has_perm(u32 sid, struct inode *inode, u32 perms,
+ struct common_audit_data *adp)
{
struct inode_security_struct *isec;
- u32 sid;
if (unlikely(IS_PRIVATE(inode)))
return 0;
- sid = cred_sid(cred);
isec = selinux_inode(inode);
return avc_has_perm(sid, isec->sid, isec->sclass, perms, adp);
}
+static int inode_has_perm(const struct cred *cred,
+ struct inode *inode,
+ u32 perms,
+ struct common_audit_data *adp)
+{
+ return inode_sid_has_perm(cred_sid(cred), inode, perms, adp);
+}
+
/* Same as inode_has_perm, but pass explicit audit data containing
the dentry to help the auditing code to more easily generate the
pathname if needed. */
@@ -3676,17 +3682,74 @@ static int selinux_file_alloc_security(struct file *file)
return 0;
}
+static inline u32 selinux_file_user_sid(const struct file *file)
+{
+ if (unlikely(file->f_mode & FMODE_BACKING))
+ return selinux_backing_file(file)->uf_sid;
+ return selinux_file(file)->sid;
+}
+
static int selinux_backing_file_alloc(struct file *backing_file,
const struct file *user_file)
{
struct backing_file_security_struct *bfsec;
+ const struct backing_file_security_struct *ubfsec;
+ struct backing_file_security_layer *layer;
+ u32 i;
bfsec = selinux_backing_file(backing_file);
- bfsec->uf_sid = selinux_file(user_file)->sid;
+ bfsec->uf_sid = selinux_file_user_sid(user_file);
+ if (!(user_file->f_mode & FMODE_BACKING))
+ return 0;
+
+ ubfsec = selinux_backing_file(user_file);
+ /* a wrapped count would make kmalloc_array() return ZERO_SIZE_PTR */
+ if (unlikely(ubfsec->layer_count == U32_MAX))
+ return -EOVERFLOW;
+
+ /*
+ * The final VMA only retains the lowest backing file, so record the
+ * whole chain here rather than in the mmap hook, where concurrent
+ * mappings would have to be serialized. Size it dynamically: erofs
+ * inode sharing adds a backing file without bumping s_stack_depth.
+ */
+ bfsec->layers = kmalloc_array(ubfsec->layer_count + 1,
+ sizeof(*bfsec->layers), GFP_KERNEL);
+ if (!bfsec->layers)
+ return -ENOMEM;
+
+ for (i = 0; i < ubfsec->layer_count; i++) {
+ layer = &bfsec->layers[i];
+ *layer = ubfsec->layers[i];
+ path_get(&layer->path);
+ }
+
+ /* f_path, not file_user_path(): this layer, not the top-level file */
+ layer = &bfsec->layers[i];
+ layer->path = user_file->f_path;
+ layer->mounter_sid = cred_sid(user_file->f_cred);
+ layer->fd_sid = selinux_file(user_file)->sid;
+ path_get(&layer->path);
+ bfsec->layer_count = ubfsec->layer_count + 1;
return 0;
}
+static void selinux_backing_file_free(struct file *backing_file)
+{
+ struct backing_file_security_struct *bfsec;
+
+ /* security_backing_file_free() may be called twice after an error */
+ if (!backing_file_security(backing_file))
+ return;
+
+ bfsec = selinux_backing_file(backing_file);
+ while (bfsec->layer_count)
+ path_put(&bfsec->layers[--bfsec->layer_count].path);
+ kfree(bfsec->layers);
+ bfsec->layers = NULL;
+}
+
/*
* Check whether a task has the ioctl permission and cmd
* operation to an inode.
@@ -3804,6 +3867,53 @@ static int selinux_file_ioctl_compat(struct file *file, unsigned int cmd,
static int default_noexec __ro_after_init;
+static u32 file_map_prot_to_av(unsigned long prot, bool shared)
+{
+ u32 av = FILE__READ;
+
+ if (shared && (prot & PROT_WRITE))
+ av |= FILE__WRITE;
+ if (prot & PROT_EXEC)
+ av |= FILE__EXECUTE;
+
+ return av;
+}
+
+static int backing_mounters_has_perm(const struct file *file, u32 av)
+{
+ const struct backing_file_security_struct *bfsec;
+ const struct backing_file_security_layer *layer;
+ struct common_audit_data ad;
+ struct inode *inode;
+ u32 i;
+ int rc;
+
+ if (WARN_ON_ONCE(!(file->f_mode & FMODE_BACKING)))
+ return -EIO;
+
+ bfsec = selinux_backing_file(file);
+ for (i = 0; i < bfsec->layer_count; i++) {
+ layer = &bfsec->layers[i];
+ inode = d_inode(layer->path.dentry);
+
+ ad.type = LSM_AUDIT_DATA_PATH;
+ ad.u.path = layer->path;
+
+ if (layer->mounter_sid != layer->fd_sid) {
+ rc = avc_has_perm(layer->mounter_sid, layer->fd_sid,
+ SECCLASS_FD, FD__USE, &ad);
+ if (rc)
+ return rc;
+ }
+
+ rc = inode_sid_has_perm(layer->mounter_sid, inode, av, &ad);
+ if (rc)
+ return rc;
+ }
+
+ return 0;
+}
+
static int __file_map_prot_check(const struct file *file, unsigned long prot,
bool shared, bool mounter_check,
bool bf_user_file)
@@ -3837,14 +3947,10 @@ static int __file_map_prot_check(const struct file *file, unsigned long prot,
if (file) {
const struct cred *cred = mounter_check ?
file->f_cred : current_cred();
- /* "read" always possible, "write" only if shared */
- u32 av = FILE__READ;
- if (shared && prot_write)
- av |= FILE__WRITE;
- if (prot_exec)
- av |= FILE__EXECUTE;
- return __file_has_perm(cred, file, av, bf_user_file);
+ return __file_has_perm(cred, file,
+ file_map_prot_to_av(prot, shared),
+ bf_user_file);
}
return 0;
@@ -3939,6 +4045,7 @@ static int selinux_file_mprotect(struct vm_area_struct *vma,
int rc;
const struct cred *cred = current_cred();
u32 sid = cred_sid(cred);
+ u32 av;
const struct file *file = vma->vm_file;
bool backing_file;
bool shared = vma->vm_flags & VM_SHARED;
@@ -3982,6 +4089,10 @@ static int selinux_file_mprotect(struct vm_area_struct *vma,
if (rc)
return rc;
if (backing_file) {
+ rc = backing_mounters_has_perm(file,
+ FILE__EXECMOD);
+ if (rc)
+ return rc;
rc = file_has_perm(file->f_cred, file,
FILE__EXECMOD);
if (rc)
@@ -3994,6 +4105,10 @@ static int selinux_file_mprotect(struct vm_area_struct *vma,
if (rc)
return rc;
if (backing_file) {
+ av = file_map_prot_to_av(prot, shared);
+ rc = backing_mounters_has_perm(file, av);
+ if (rc)
+ return rc;
rc = file_map_prot_check(file, prot, shared, true);
if (rc)
return rc;
@@ -7302,6 +7417,7 @@ static struct security_hook_list selinux_hooks[] __ro_after_init = {
LSM_HOOK_INIT(file_permission, selinux_file_permission),
LSM_HOOK_INIT(file_alloc_security, selinux_file_alloc_security),
LSM_HOOK_INIT(backing_file_alloc, selinux_backing_file_alloc),
+ LSM_HOOK_INIT(backing_file_free, selinux_backing_file_free),
LSM_HOOK_INIT(file_ioctl, selinux_file_ioctl),
LSM_HOOK_INIT(file_ioctl_compat, selinux_file_ioctl_compat),
LSM_HOOK_INIT(mmap_file, selinux_mmap_file),
diff --git a/security/selinux/include/objsec.h b/security/selinux/include/objsec.h
index dc42282a2c05..581ab3dd8fe1 100644
--- a/security/selinux/include/objsec.h
+++ b/security/selinux/include/objsec.h
@@ -61,8 +61,16 @@ struct file_security_struct {
u32 pseqno; /* Policy seqno at the time of file open */
};
+struct backing_file_security_layer {
+ struct path path; /* this layer's real path */
+ u32 mounter_sid; /* SID of the mounter that opened it */
+ u32 fd_sid; /* SID of its open file description */
+};
+
struct backing_file_security_struct {
- u32 uf_sid; /* associated user file fsec->sid */
+ u32 uf_sid; /* top-level user file fsec->sid */
+ u32 layer_count; /* number of intermediate backing files */
+ struct backing_file_security_layer *layers;
};
struct superblock_security_struct {
diff --git a/sound/core/init.c b/sound/core/init.c
index c35e105ef132..87c10bd900a7 100644
--- a/sound/core/init.c
+++ b/sound/core/init.c
@@ -309,7 +309,7 @@ static int snd_card_init(struct snd_card *card, struct device *parent,
kfree(card); /* manually free here, as no destructor called */
return err;
}
- card->dev = parent;
+ card->dev = get_device(parent);
card->number = idx;
WARN_ON(IS_MODULE(CONFIG_SND) && !module);
card->module = module;
@@ -593,6 +593,7 @@ static int snd_card_do_free(struct snd_card *card)
dev_warn(card->dev, "unable to free card info\n");
/* Not fatal error */
}
+ put_device(card->dev);
if (card->release_completion)
complete(card->release_completion);
if (!managed)
diff --git a/sound/core/pcm_timer.c b/sound/core/pcm_timer.c
index ab0e5bd70f8f..18bedd66435d 100644
--- a/sound/core/pcm_timer.c
+++ b/sound/core/pcm_timer.c
@@ -111,12 +111,15 @@ void snd_pcm_timer_init(struct snd_pcm_substream *substream)
snd_pcm_direction_name(substream->stream),
tid.card, tid.device, tid.subdevice);
timer->hw = snd_pcm_timer;
+ /* Set before registering: a concurrent reader can invoke our hw
+ * callbacks as soon as the timer is on the global list.
+ */
+ timer->private_data = substream;
+ timer->private_free = snd_pcm_timer_free;
if (snd_device_register(timer->card, timer) < 0) {
snd_device_free(timer->card, timer);
return;
}
- timer->private_data = substream;
- timer->private_free = snd_pcm_timer_free;
substream->timer = timer;
}
diff --git a/sound/hda/ext/hdac_ext_stream.c b/sound/hda/ext/hdac_ext_stream.c
index a3ac738f1130..f19cdc1194f1 100644
--- a/sound/hda/ext/hdac_ext_stream.c
+++ b/sound/hda/ext/hdac_ext_stream.c
@@ -212,8 +212,8 @@ void snd_hdac_ext_stream_reset(struct hdac_ext_stream *hext_stream)
break;
udelay(3);
} while (--timeout);
- val &= ~AZX_PPLCCTL_STRST;
- writel(val, hext_stream->pplc_addr + AZX_REG_PPLCCTL);
+ snd_hdac_updatel(hext_stream->pplc_addr, AZX_REG_PPLCCTL,
+ AZX_PPLCCTL_STRST, 0);
udelay(3);
timeout = 50;
diff --git a/sound/pci/hda/hda_controller.c b/sound/pci/hda/hda_controller.c
index bd13ea0c9352..7c0574eb54c6 100644
--- a/sound/pci/hda/hda_controller.c
+++ b/sound/pci/hda/hda_controller.c
@@ -593,11 +593,11 @@ static int azx_pcm_open(struct snd_pcm_substream *substream)
snd_hda_codec_pcm_get(apcm->info);
mutex_lock(&chip->open_mutex);
azx_dev = azx_assign_device(chip, substream);
- trace_azx_pcm_open(chip, azx_dev);
if (azx_dev == NULL) {
err = -EBUSY;
goto unlock;
}
+ trace_azx_pcm_open(chip, azx_dev);
runtime->private_data = azx_dev;
runtime->hw = azx_pcm_hw;
diff --git a/sound/pci/hda/patch_realtek.c b/sound/pci/hda/patch_realtek.c
index 4b0ccb3b48b2..72d540182f8d 100644
--- a/sound/pci/hda/patch_realtek.c
+++ b/sound/pci/hda/patch_realtek.c
@@ -8243,6 +8243,7 @@ enum {
ALC245_FIXUP_CLEVO_NOISY_MIC,
ALC269_FIXUP_VAIO_VJFH52_MIC_NO_PRESENCE,
ALC233_FIXUP_MEDION_MTL_SPK,
+ ALC269_FIXUP_STARLABS_LIMIT_INT_MIC_BOOST,
ALC233_FIXUP_STARLABS_STARFIGHTER,
ALC294_FIXUP_BASS_SPEAKER_15,
ALC283_FIXUP_DELL_HP_RESUME,
@@ -10652,9 +10653,15 @@ static const struct hda_fixup alc269_fixups[] = {
{ }
},
},
+ [ALC269_FIXUP_STARLABS_LIMIT_INT_MIC_BOOST] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = alc269_fixup_limit_int_mic_boost,
+ },
[ALC233_FIXUP_STARLABS_STARFIGHTER] = {
.type = HDA_FIXUP_FUNC,
.v.func = alc233_fixup_starlabs_starfighter,
+ .chained = true,
+ .chain_id = ALC269_FIXUP_STARLABS_LIMIT_INT_MIC_BOOST,
},
[ALC294_FIXUP_BASS_SPEAKER_15] = {
.type = HDA_FIXUP_FUNC,
@@ -11693,6 +11700,7 @@ static const struct hda_quirk alc269_fixup_tbl[] = {
SND_PCI_QUIRK(0x1f66, 0x0105, "Ayaneo Portable Game Player", ALC287_FIXUP_CS35L41_I2C_2),
SND_PCI_QUIRK(0x2014, 0x800a, "Positivo ARN50", ALC269_FIXUP_LIMIT_INT_MIC_BOOST),
SND_PCI_QUIRK(0x2039, 0x0001, "Inspur S14-G1", ALC295_FIXUP_CHROME_BOOK),
+ SND_PCI_QUIRK(0x2145, 0x0001, "Star Labs StarFighter", ALC233_FIXUP_STARLABS_STARFIGHTER),
SND_PCI_QUIRK(0x2782, 0x0214, "VAIO VJFE-CL", ALC269_FIXUP_LIMIT_INT_MIC_BOOST),
SND_PCI_QUIRK(0x2782, 0x0228, "Infinix ZERO BOOK 13", ALC269VB_FIXUP_INFINIX_ZERO_BOOK_13),
SND_PCI_QUIRK(0x2782, 0x0232, "CHUWI CoreBook XPro", ALC269VB_FIXUP_CHUWI_COREBOOK_XPRO),
@@ -11770,6 +11778,7 @@ static const struct hda_quirk alc269_fixup_vendor_tbl[] = {
SND_PCI_QUIRK_VENDOR(0x104d, "Sony VAIO", ALC269_FIXUP_SONY_VAIO),
SND_PCI_QUIRK_VENDOR(0x17aa, "Thinkpad", ALC269_FIXUP_THINKPAD_ACPI),
SND_PCI_QUIRK_VENDOR(0x19e5, "Huawei Matebook", ALC255_FIXUP_MIC_MUTE_LED),
+ SND_PCI_QUIRK_VENDOR(0x2145, "Star Labs", ALC269_FIXUP_STARLABS_LIMIT_INT_MIC_BOOST),
{}
};
diff --git a/sound/soc/codecs/aw88261.c b/sound/soc/codecs/aw88261.c
index 999461885a62..71ce374fd1df 100644
--- a/sound/soc/codecs/aw88261.c
+++ b/sound/soc/codecs/aw88261.c
@@ -150,7 +150,7 @@ static int aw88261_dev_get_iis_status(struct aw_device *aw_dev)
if (ret)
return ret;
if ((reg_val & AW88261_BIT_PLL_CHECK) != AW88261_BIT_PLL_CHECK) {
- dev_err(aw_dev->dev, "check pll lock fail,reg_val:0x%04x", reg_val);
+ dev_dbg(aw_dev->dev, "check pll lock fail,reg_val:0x%04x", reg_val);
return -EINVAL;
}
@@ -164,7 +164,7 @@ static int aw88261_dev_check_mode1_pll(struct aw_device *aw_dev)
for (i = 0; i < AW88261_DEV_SYSST_CHECK_MAX; i++) {
ret = aw88261_dev_get_iis_status(aw_dev);
if (ret) {
- dev_err(aw_dev->dev, "mode1 iis signal check error");
+ dev_dbg(aw_dev->dev, "mode1 iis signal check error");
usleep_range(AW88261_2000_US, AW88261_2000_US + 10);
} else {
return ret;
@@ -253,10 +253,10 @@ static int aw88261_dev_check_sysst(struct aw_device *aw_dev)
return ret;
check_val = reg_val & (~AW88261_BIT_SYSST_CHECK_MASK)
- & AW88261_BIT_SYSST_CHECK;
- if (check_val != AW88261_BIT_SYSST_CHECK) {
- dev_err(aw_dev->dev, "check sysst fail, reg_val=0x%04x, check:0x%x",
- reg_val, AW88261_BIT_SYSST_CHECK);
+ & AW88261_BIT_PLL_CHECK;
+ if (check_val != AW88261_BIT_PLL_CHECK) {
+ dev_dbg(aw_dev->dev, "check sysst fail, reg_val=0x%04x, check:0x%x",
+ reg_val, AW88261_BIT_PLL_CHECK);
usleep_range(AW88261_2000_US, AW88261_2000_US + 10);
} else {
return 0;
@@ -549,7 +549,7 @@ static int aw88261_dev_start(struct aw88261 *aw88261)
int ret;
if (aw_dev->status == AW88261_DEV_PW_ON) {
- dev_info(aw_dev->dev, "already power on");
+ dev_dbg(aw_dev->dev, "already power on");
return 0;
}
@@ -559,7 +559,7 @@ static int aw88261_dev_start(struct aw88261 *aw88261)
ret = aw88261_dev_check_syspll(aw_dev);
if (ret) {
- dev_err(aw_dev->dev, "pll check failed cannot start");
+ dev_dbg(aw_dev->dev, "pll check failed");
goto pll_check_fail;
}
@@ -570,7 +570,7 @@ static int aw88261_dev_start(struct aw88261 *aw88261)
/* check i2s status */
ret = aw88261_dev_check_sysst(aw_dev);
if (ret) {
- dev_err(aw_dev->dev, "sysst check failed");
+ dev_dbg(aw_dev->dev, "sysst check failed");
goto sysst_check_fail;
}
@@ -671,18 +671,22 @@ static void aw88261_start_pa(struct aw88261 *aw88261)
for (i = 0; i < AW88261_START_RETRIES; i++) {
ret = aw88261_reg_update(aw88261, aw88261->phase_sync);
if (ret) {
- dev_err(aw88261->aw_pa->dev, "fw update failed, cnt:%d\n", i);
+ dev_dbg(aw88261->aw_pa->dev,
+ "aw88261_reg_update failed, cnt:%d, ret:%d\n", i, ret);
continue;
}
ret = aw88261_dev_start(aw88261);
if (ret) {
- dev_err(aw88261->aw_pa->dev, "aw88261 device start failed. retry = %d", i);
+ dev_dbg(aw88261->aw_pa->dev,
+ "aw88261_dev_start failed, cnt:%d, ret:%d\n", i, ret);
continue;
} else {
- dev_info(aw88261->aw_pa->dev, "start success\n");
+ dev_dbg(aw88261->aw_pa->dev, "start success\n");
break;
}
}
+ if (ret != 0)
+ dev_err(aw88261->aw_pa->dev, "start failure (%d)\n", ret);
}
static void aw88261_startup_work(struct work_struct *work)
@@ -1200,7 +1204,7 @@ static int aw88261_init(struct aw88261 **aw88261, struct i2c_client *i2c, struct
return ret;
}
if (chip_id != AW88261_CHIP_ID) {
- dev_err(&i2c->dev, "unsupported device");
+ dev_err(&i2c->dev, "unsupported device id = %x", chip_id);
return -ENXIO;
}
diff --git a/sound/soc/codecs/aw88261.h b/sound/soc/codecs/aw88261.h
index 734d0f93ced9..786350108674 100644
--- a/sound/soc/codecs/aw88261.h
+++ b/sound/soc/codecs/aw88261.h
@@ -181,12 +181,6 @@
AW88261_OTHS_OT_VALUE | \
AW88261_PLLS_LOCKED_VALUE))
-#define AW88261_BIT_SYSST_CHECK \
- (AW88261_BSTS_FINISHED_VALUE | \
- AW88261_SWS_SWITCHING_VALUE | \
- AW88261_CLKS_STABLE_VALUE | \
- AW88261_PLLS_LOCKED_VALUE)
-
#define AW88261_ULS_HMUTE_START_BIT (14)
#define AW88261_ULS_HMUTE_BITS_LEN (1)
#define AW88261_ULS_HMUTE_MASK \
diff --git a/sound/soc/codecs/hdmi-codec.c b/sound/soc/codecs/hdmi-codec.c
index d9df29a26f4f..0ffc649c6ad2 100644
--- a/sound/soc/codecs/hdmi-codec.c
+++ b/sound/soc/codecs/hdmi-codec.c
@@ -423,10 +423,14 @@ static int hdmi_codec_iec958_default_put(struct snd_kcontrol *kcontrol,
struct snd_soc_component *component = snd_kcontrol_chip(kcontrol);
struct hdmi_codec_priv *hcp = snd_soc_component_get_drvdata(component);
+ if (!memcmp(hcp->iec_status, ucontrol->value.iec958.status,
+ sizeof(hcp->iec_status)))
+ return 0;
+
memcpy(hcp->iec_status, ucontrol->value.iec958.status,
sizeof(hcp->iec_status));
- return 0;
+ return 1;
}
static int hdmi_codec_iec958_mask_get(struct snd_kcontrol *kcontrol,
diff --git a/sound/soc/codecs/rt712-sdca-dmic.c b/sound/soc/codecs/rt712-sdca-dmic.c
index ee5435f3a80a..74c67990b5fb 100644
--- a/sound/soc/codecs/rt712-sdca-dmic.c
+++ b/sound/soc/codecs/rt712-sdca-dmic.c
@@ -14,6 +14,7 @@
#include <sound/core.h>
#include <sound/pcm.h>
#include <sound/pcm_params.h>
+#include <sound/sdw.h>
#include <sound/tlv.h>
#include "rt712-sdca.h"
#include "rt712-sdca-dmic.h"
@@ -641,10 +642,10 @@ static int rt712_sdca_dmic_hw_params(struct snd_pcm_substream *substream,
{
struct snd_soc_component *component = dai->component;
struct rt712_sdca_dmic_priv *rt712 = snd_soc_component_get_drvdata(component);
- struct sdw_stream_config stream_config;
+ struct sdw_stream_config stream_config = {0};
struct sdw_port_config port_config;
struct sdw_stream_runtime *sdw_stream;
- int retval, num_channels;
+ int retval;
unsigned int sampling_rate;
dev_dbg(dai->dev, "%s %s", __func__, dai->name);
@@ -656,13 +657,8 @@ static int rt712_sdca_dmic_hw_params(struct snd_pcm_substream *substream,
if (!rt712->slave)
return -EINVAL;
- stream_config.frame_rate = params_rate(params);
- stream_config.ch_count = params_channels(params);
- stream_config.bps = snd_pcm_format_width(params_format(params));
- stream_config.direction = SDW_DATA_DIR_TX;
-
- num_channels = params_channels(params);
- port_config.ch_mask = GENMASK(num_channels - 1, 0);
+ /* SoundWire specific configuration */
+ snd_sdw_params_to_config(substream, params, &stream_config, &port_config);
port_config.num = 2;
retval = sdw_stream_add_slave(rt712->slave, &stream_config,
diff --git a/sound/soc/sdw_utils/soc_sdw_utils.c b/sound/soc/sdw_utils/soc_sdw_utils.c
index e6ac5c0fd3be..70a7abede618 100644
--- a/sound/soc/sdw_utils/soc_sdw_utils.c
+++ b/sound/soc/sdw_utils/soc_sdw_utils.c
@@ -795,7 +795,7 @@ int asoc_sdw_hw_params(struct snd_pcm_substream *substream,
* ASoC will set the corresponding channel numbers for each cpu dai.
*/
for_each_link_ch_maps(rtd->dai_link, i, ch_maps)
- ch_maps->ch_mask = ch_mask << (i * step);
+ ch_maps->cpu_ch_mask = ch_mask << (i * step);
return 0;
}
diff --git a/sound/soc/soc-pcm.c b/sound/soc/soc-pcm.c
index 628322790c87..d52771a4a723 100644
--- a/sound/soc/soc-pcm.c
+++ b/sound/soc/soc-pcm.c
@@ -1122,7 +1122,9 @@ static int __soc_pcm_hw_params(struct snd_soc_pcm_runtime *rtd,
goto out;
for_each_rtd_codec_dais(rtd, i, codec_dai) {
- unsigned int tdm_mask = snd_soc_dai_tdm_mask_get(codec_dai, substream->stream);
+ unsigned int ch_mask = snd_soc_dai_tdm_mask_get(codec_dai, substream->stream);
+ struct snd_soc_dai_link_ch_map *ch_maps;
+ int j;
/*
* Skip CODECs which don't support the current stream type,
@@ -1144,9 +1146,15 @@ static int __soc_pcm_hw_params(struct snd_soc_pcm_runtime *rtd,
/* copy params for each codec */
tmp_params = *params;
- /* fixup params based on TDM slot masks */
- if (tdm_mask)
- soc_pcm_codec_params_fixup(&tmp_params, tdm_mask);
+ /* fixup params based on TDM or ch_map masks */
+ if (!ch_mask) {
+ for_each_rtd_ch_maps(rtd, j, ch_maps)
+ if (ch_maps->codec == i)
+ ch_mask |= ch_maps->codec_ch_mask;
+ }
+
+ if (ch_mask)
+ soc_pcm_codec_params_fixup(&tmp_params, ch_mask);
ret = snd_soc_dai_hw_params(codec_dai, substream,
&tmp_params);
@@ -1180,7 +1188,7 @@ static int __soc_pcm_hw_params(struct snd_soc_pcm_runtime *rtd,
*/
for_each_rtd_ch_maps(rtd, j, ch_maps)
if (ch_maps->cpu == i)
- ch_mask |= ch_maps->ch_mask;
+ ch_mask |= ch_maps->cpu_ch_mask;
/* fixup cpu channel number */
if (ch_mask)
diff --git a/sound/soc/tegra/tegra210_mixer.c b/sound/soc/tegra/tegra210_mixer.c
index e07e2f1d2f70..f368729066ca 100644
--- a/sound/soc/tegra/tegra210_mixer.c
+++ b/sound/soc/tegra/tegra210_mixer.c
@@ -57,10 +57,10 @@ static const struct reg_default tegra210_mixer_reg_defaults[] = {
MIXER_TX_REG_DEFAULTS(3),
MIXER_TX_REG_DEFAULTS(4),
+ { TEGRA210_MIXER_ENABLE, 0x0 },
{ TEGRA210_MIXER_CG, 0x00000001},
{ TEGRA210_MIXER_GAIN_CFG_RAM_CTRL, 0x00004000},
{ TEGRA210_MIXER_PEAKM_RAM_CTRL, 0x00004000},
- { TEGRA210_MIXER_ENABLE, 0x1 },
};
/* Default gain parameters */
@@ -86,11 +86,15 @@ static int __maybe_unused tegra210_mixer_runtime_suspend(struct device *dev)
static int __maybe_unused tegra210_mixer_runtime_resume(struct device *dev)
{
struct tegra210_mixer *mixer = dev_get_drvdata(dev);
+ int err;
regcache_cache_only(mixer->regmap, false);
- regcache_sync(mixer->regmap);
+ err = regcache_sync(mixer->regmap);
+ if (err)
+ return err;
- return 0;
+ return regmap_write(mixer->regmap, TEGRA210_MIXER_ENABLE,
+ TEGRA210_MIXER_EN);
}
static int tegra210_mixer_write_ram(struct tegra210_mixer *mixer,
diff --git a/sound/soc/ux500/ux500_msp_i2s.h b/sound/soc/ux500/ux500_msp_i2s.h
index 2bf2699bdc49..c66ef455e138 100644
--- a/sound/soc/ux500/ux500_msp_i2s.h
+++ b/sound/soc/ux500/ux500_msp_i2s.h
@@ -147,8 +147,8 @@ enum msp_direction {
#define RCKPOL_MASK BIT(0)
#define TCKPOL_MASK BIT(0)
#define SPICKM_MASK (BIT(1) | BIT(0))
-#define MSP_RX_CLKPOL_BIT(n) ((n & RCKPOL_MASK) << RCKPOL_SHIFT)
-#define MSP_TX_CLKPOL_BIT(n) ((n & TCKPOL_MASK) << TCKPOL_SHIFT)
+#define MSP_RX_CLKPOL_BIT(n) (((n) & RCKPOL_MASK) << RCKPOL_SHIFT)
+#define MSP_TX_CLKPOL_BIT(n) (((n) & TCKPOL_MASK) << TCKPOL_SHIFT)
#define P1ELEN_SHIFT 0
#define P1FLEN_SHIFT 3
diff --git a/sound/usb/6fire/chip.c b/sound/usb/6fire/chip.c
index e5916c6b75ae..30b240e06a8c 100644
--- a/sound/usb/6fire/chip.c
+++ b/sound/usb/6fire/chip.c
@@ -83,24 +83,22 @@ static int usb6fire_chip_probe(struct usb_interface *intf,
struct snd_card *card = NULL;
/* look if we already serve this card and return if so */
- mutex_lock(®ister_mutex);
- for (i = 0; i < SNDRV_CARDS; i++) {
- if (devices[i] == device) {
- if (chips[i])
- chips[i]->intf_count++;
- usb_set_intfdata(intf, chips[i]);
- mutex_unlock(®ister_mutex);
- return 0;
- } else if (!devices[i] && regidx < 0)
- regidx = i;
- }
- if (regidx < 0) {
- mutex_unlock(®ister_mutex);
- dev_err(&intf->dev, "too many cards registered.\n");
- return -ENODEV;
+ scoped_guard(mutex, ®ister_mutex) {
+ for (i = 0; i < SNDRV_CARDS; i++) {
+ if (devices[i] == device) {
+ if (chips[i])
+ chips[i]->intf_count++;
+ usb_set_intfdata(intf, chips[i]);
+ return 0;
+ } else if (!devices[i] && regidx < 0)
+ regidx = i;
+ }
+ if (regidx < 0) {
+ dev_err(&intf->dev, "too many cards registered.\n");
+ return -ENODEV;
+ }
+ devices[regidx] = device;
}
- devices[regidx] = device;
- mutex_unlock(®ister_mutex);
/* check, if firmware is present on device, upload it if not */
ret = usb6fire_fw_init(intf);
@@ -175,10 +173,10 @@ static void usb6fire_chip_disconnect(struct usb_interface *intf)
if (chip) { /* if !chip, fw upload has been performed */
chip->intf_count--;
if (!chip->intf_count) {
- mutex_lock(®ister_mutex);
- devices[chip->regidx] = NULL;
- chips[chip->regidx] = NULL;
- mutex_unlock(®ister_mutex);
+ scoped_guard(mutex, ®ister_mutex) {
+ devices[chip->regidx] = NULL;
+ chips[chip->regidx] = NULL;
+ }
/*
* Save card pointer before teardown.
diff --git a/sound/usb/6fire/comm.c b/sound/usb/6fire/comm.c
index bcfb34db37d9..cfaaad9d2402 100644
--- a/sound/usb/6fire/comm.c
+++ b/sound/usb/6fire/comm.c
@@ -21,7 +21,6 @@ enum {
static void usb6fire_comm_init_urb(struct comm_runtime *rt, struct urb *urb,
u8 *buffer, void *context, void(*handler)(struct urb *urb))
{
- usb_init_urb(urb);
urb->transfer_buffer = buffer;
urb->pipe = usb_sndintpipe(rt->chip->dev, COMM_EP);
urb->complete = handler;
@@ -142,6 +141,19 @@ static int usb6fire_comm_write16(struct comm_runtime *rt, u8 request,
return ret;
}
+static void usb6fire_comm_free(struct comm_runtime *rt)
+{
+ if (!rt)
+ return;
+
+ if (rt->chip)
+ rt->chip->comm = NULL;
+
+ usb_free_urb(rt->receiver);
+ kfree(rt->receiver_buffer);
+ kfree(rt);
+}
+
int usb6fire_comm_init(struct sfire_chip *chip)
{
struct comm_runtime *rt = kzalloc(sizeof(struct comm_runtime),
@@ -154,14 +166,18 @@ int usb6fire_comm_init(struct sfire_chip *chip)
rt->receiver_buffer = kzalloc(COMM_RECEIVER_BUFSIZE, GFP_KERNEL);
if (!rt->receiver_buffer) {
- kfree(rt);
- return -ENOMEM;
+ ret = -ENOMEM;
+ goto error;
}
- urb = &rt->receiver;
+ urb = usb_alloc_urb(0, GFP_KERNEL);
+ if (!urb) {
+ ret = -ENOMEM;
+ goto error;
+ }
+ rt->receiver = urb;
rt->serial = 1;
rt->chip = chip;
- usb_init_urb(urb);
rt->init_urb = usb6fire_comm_init_urb;
rt->write8 = usb6fire_comm_write8;
rt->write16 = usb6fire_comm_write16;
@@ -176,13 +192,15 @@ int usb6fire_comm_init(struct sfire_chip *chip)
urb->interval = 1;
ret = usb_submit_urb(urb, GFP_KERNEL);
if (ret < 0) {
- kfree(rt->receiver_buffer);
- kfree(rt);
dev_err(&chip->dev->dev, "cannot create comm data receiver.");
- return ret;
+ goto error;
}
chip->comm = rt;
return 0;
+
+ error:
+ usb6fire_comm_free(rt);
+ return ret;
}
void usb6fire_comm_abort(struct sfire_chip *chip)
@@ -190,14 +208,10 @@ void usb6fire_comm_abort(struct sfire_chip *chip)
struct comm_runtime *rt = chip->comm;
if (rt)
- usb_poison_urb(&rt->receiver);
+ usb_poison_urb(rt->receiver);
}
void usb6fire_comm_destroy(struct sfire_chip *chip)
{
- struct comm_runtime *rt = chip->comm;
-
- kfree(rt->receiver_buffer);
- kfree(rt);
- chip->comm = NULL;
+ usb6fire_comm_free(chip->comm);
}
diff --git a/sound/usb/6fire/comm.h b/sound/usb/6fire/comm.h
index 2447d7ecf179..89976f510f6c 100644
--- a/sound/usb/6fire/comm.h
+++ b/sound/usb/6fire/comm.h
@@ -19,7 +19,7 @@ enum /* settings for comm */
struct comm_runtime {
struct sfire_chip *chip;
- struct urb receiver;
+ struct urb *receiver;
u8 *receiver_buffer;
u8 serial; /* urb serial */
diff --git a/sound/usb/6fire/midi.c b/sound/usb/6fire/midi.c
index de2691d58de6..9a1dd5b6557c 100644
--- a/sound/usb/6fire/midi.c
+++ b/sound/usb/6fire/midi.c
@@ -23,9 +23,8 @@ static void usb6fire_midi_out_handler(struct urb *urb)
{
struct midi_runtime *rt = urb->context;
int ret;
- unsigned long flags;
- spin_lock_irqsave(&rt->out_lock, flags);
+ guard(spinlock_irqsave)(&rt->out_lock);
if (rt->out) {
ret = snd_rawmidi_transmit(rt->out, rt->out_buffer + 4,
@@ -43,18 +42,14 @@ static void usb6fire_midi_out_handler(struct urb *urb)
} else /* no more data to transmit */
rt->out = NULL;
}
- spin_unlock_irqrestore(&rt->out_lock, flags);
}
static void usb6fire_midi_in_received(
struct midi_runtime *rt, u8 *data, int length)
{
- unsigned long flags;
-
- spin_lock_irqsave(&rt->in_lock, flags);
+ guard(spinlock_irqsave)(&rt->in_lock);
if (rt->in)
snd_rawmidi_receive(rt->in, data, length);
- spin_unlock_irqrestore(&rt->in_lock, flags);
}
static int usb6fire_midi_out_open(struct snd_rawmidi_substream *alsa_sub)
@@ -71,16 +66,13 @@ static void usb6fire_midi_out_trigger(
struct snd_rawmidi_substream *alsa_sub, int up)
{
struct midi_runtime *rt = alsa_sub->rmidi->private_data;
- struct urb *urb = &rt->out_urb;
+ struct urb *urb = rt->out_urb;
__s8 ret;
- unsigned long flags;
- spin_lock_irqsave(&rt->out_lock, flags);
+ guard(spinlock_irqsave)(&rt->out_lock);
if (up) { /* start transfer */
- if (rt->out) { /* we are already transmitting so just return */
- spin_unlock_irqrestore(&rt->out_lock, flags);
+ if (rt->out) /* we are already transmitting so just return */
return;
- }
ret = snd_rawmidi_transmit(alsa_sub, rt->out_buffer + 4,
MIDI_BUFSIZE - 4);
@@ -99,7 +91,6 @@ static void usb6fire_midi_out_trigger(
}
} else if (rt->out == alsa_sub)
rt->out = NULL;
- spin_unlock_irqrestore(&rt->out_lock, flags);
}
static void usb6fire_midi_out_drain(struct snd_rawmidi_substream *alsa_sub)
@@ -125,14 +116,12 @@ static void usb6fire_midi_in_trigger(
struct snd_rawmidi_substream *alsa_sub, int up)
{
struct midi_runtime *rt = alsa_sub->rmidi->private_data;
- unsigned long flags;
- spin_lock_irqsave(&rt->in_lock, flags);
+ guard(spinlock_irqsave)(&rt->in_lock);
if (up)
rt->in = alsa_sub;
else
rt->in = NULL;
- spin_unlock_irqrestore(&rt->in_lock, flags);
}
static const struct snd_rawmidi_ops out_ops = {
@@ -148,6 +137,19 @@ static const struct snd_rawmidi_ops in_ops = {
.trigger = usb6fire_midi_in_trigger
};
+static void usb6fire_midi_free(struct midi_runtime *rt)
+{
+ if (!rt)
+ return;
+
+ if (rt->chip)
+ rt->chip->midi = NULL;
+
+ usb_free_urb(rt->out_urb);
+ kfree(rt->out_buffer);
+ kfree(rt);
+}
+
int usb6fire_midi_init(struct sfire_chip *chip)
{
int ret;
@@ -160,8 +162,14 @@ int usb6fire_midi_init(struct sfire_chip *chip)
rt->out_buffer = kzalloc(MIDI_BUFSIZE, GFP_KERNEL);
if (!rt->out_buffer) {
- kfree(rt);
- return -ENOMEM;
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ rt->out_urb = usb_alloc_urb(0, GFP_KERNEL);
+ if (!rt->out_urb) {
+ ret = -ENOMEM;
+ goto error;
}
rt->chip = chip;
@@ -172,15 +180,13 @@ int usb6fire_midi_init(struct sfire_chip *chip)
spin_lock_init(&rt->in_lock);
spin_lock_init(&rt->out_lock);
- comm_rt->init_urb(comm_rt, &rt->out_urb, rt->out_buffer, rt,
+ comm_rt->init_urb(comm_rt, rt->out_urb, rt->out_buffer, rt,
usb6fire_midi_out_handler);
ret = snd_rawmidi_new(chip->card, "6FireUSB", 0, 1, 1, &rt->instance);
if (ret < 0) {
- kfree(rt->out_buffer);
- kfree(rt);
dev_err(&chip->dev->dev, "unable to create midi.\n");
- return ret;
+ goto error;
}
rt->instance->private_data = rt;
strcpy(rt->instance->name, "DMX6FireUSB MIDI");
@@ -194,6 +200,10 @@ int usb6fire_midi_init(struct sfire_chip *chip)
chip->midi = rt;
return 0;
+
+ error:
+ usb6fire_midi_free(rt);
+ return ret;
}
void usb6fire_midi_abort(struct sfire_chip *chip)
@@ -201,14 +211,10 @@ void usb6fire_midi_abort(struct sfire_chip *chip)
struct midi_runtime *rt = chip->midi;
if (rt)
- usb_poison_urb(&rt->out_urb);
+ usb_poison_urb(rt->out_urb);
}
void usb6fire_midi_destroy(struct sfire_chip *chip)
{
- struct midi_runtime *rt = chip->midi;
-
- kfree(rt->out_buffer);
- kfree(rt);
- chip->midi = NULL;
+ usb6fire_midi_free(chip->midi);
}
diff --git a/sound/usb/6fire/midi.h b/sound/usb/6fire/midi.h
index 47640c845903..8716ab8a863a 100644
--- a/sound/usb/6fire/midi.h
+++ b/sound/usb/6fire/midi.h
@@ -22,7 +22,7 @@ struct midi_runtime {
spinlock_t in_lock;
spinlock_t out_lock;
struct snd_rawmidi_substream *out;
- struct urb out_urb;
+ struct urb *out_urb;
u8 out_serial; /* serial number of out packet */
u8 *out_buffer;
int buffer_offset;
diff --git a/sound/usb/6fire/pcm.c b/sound/usb/6fire/pcm.c
index 32c39d8bd2e5..5f6a63f79990 100644
--- a/sound/usb/6fire/pcm.c
+++ b/sound/usb/6fire/pcm.c
@@ -138,8 +138,8 @@ static void usb6fire_pcm_stream_stop(struct pcm_runtime *rt)
rt->stream_state = STREAM_STOPPING;
for (i = 0; i < PCM_N_URBS; i++) {
- usb_kill_urb(&rt->in_urbs[i].instance);
- usb_kill_urb(&rt->out_urbs[i].instance);
+ usb_kill_urb(rt->in_urbs[i].instance);
+ usb_kill_urb(rt->out_urbs[i].instance);
}
ctrl_rt->usb_streaming = false;
ctrl_rt->update_streaming(ctrl_rt);
@@ -161,13 +161,13 @@ static int usb6fire_pcm_stream_start(struct pcm_runtime *rt)
rt->stream_state = STREAM_STARTING;
for (i = 0; i < PCM_N_URBS; i++) {
for (k = 0; k < PCM_N_PACKETS_PER_URB; k++) {
- packet = &rt->in_urbs[i].packets[k];
+ packet = &rt->in_urbs[i].instance->iso_frame_desc[k];
packet->offset = k * rt->in_packet_size;
packet->length = rt->in_packet_size;
packet->actual_length = 0;
packet->status = 0;
}
- ret = usb_submit_urb(&rt->in_urbs[i].instance,
+ ret = usb_submit_urb(rt->in_urbs[i].instance,
GFP_ATOMIC);
if (ret) {
usb6fire_pcm_stream_stop(rt);
@@ -197,6 +197,7 @@ static void usb6fire_pcm_capture(struct pcm_substream *sub, struct pcm_urb *urb)
unsigned int total_length = 0;
struct pcm_runtime *rt = snd_pcm_substream_chip(sub->instance);
struct snd_pcm_runtime *alsa_rt = sub->instance->runtime;
+ struct usb_iso_packet_descriptor *isoc;
u32 *src = NULL;
u32 *dest = (u32 *) (alsa_rt->dma_area + sub->dma_off
* (alsa_rt->frame_bits >> 3));
@@ -207,8 +208,9 @@ static void usb6fire_pcm_capture(struct pcm_substream *sub, struct pcm_urb *urb)
for (i = 0; i < PCM_N_PACKETS_PER_URB; i++) {
/* at least 4 header bytes for valid packet.
* after that: 32 bits per sample for analog channels */
- if (urb->packets[i].actual_length > 4)
- frame_count = (urb->packets[i].actual_length - 4)
+ isoc = &urb->instance->iso_frame_desc[i];
+ if (isoc->actual_length > 4)
+ frame_count = (isoc->actual_length - 4)
/ (rt->in_n_analog << 2);
else
frame_count = 0;
@@ -220,7 +222,7 @@ static void usb6fire_pcm_capture(struct pcm_substream *sub, struct pcm_urb *urb)
else
return;
src++; /* skip leading 4 bytes of every packet */
- total_length += urb->packets[i].length;
+ total_length += isoc->length;
for (frame = 0; frame < frame_count; frame++) {
memcpy(dest, src, bytes_per_frame);
dest += alsa_rt->channels;
@@ -244,6 +246,7 @@ static void usb6fire_pcm_playback(struct pcm_substream *sub,
int frame_count;
struct pcm_runtime *rt = snd_pcm_substream_chip(sub->instance);
struct snd_pcm_runtime *alsa_rt = sub->instance->runtime;
+ struct usb_iso_packet_descriptor *isoc;
u32 *src = (u32 *) (alsa_rt->dma_area + sub->dma_off
* (alsa_rt->frame_bits >> 3));
u32 *src_end = (u32 *) (alsa_rt->dma_area + alsa_rt->buffer_size
@@ -263,8 +266,9 @@ static void usb6fire_pcm_playback(struct pcm_substream *sub,
for (i = 0; i < PCM_N_PACKETS_PER_URB; i++) {
/* at least 4 header bytes for valid packet.
* after that: 32 bits per sample for analog channels */
- if (urb->packets[i].length > 4)
- frame_count = (urb->packets[i].length - 4)
+ isoc = &urb->instance->iso_frame_desc[i];
+ if (isoc->length > 4)
+ frame_count = (isoc->length - 4)
/ (rt->out_n_analog << 2);
else
frame_count = 0;
@@ -289,7 +293,8 @@ static void usb6fire_pcm_in_urb_handler(struct urb *usb_urb)
struct pcm_urb *out_urb = in_urb->peer;
struct pcm_runtime *rt = in_urb->chip->pcm;
struct pcm_substream *sub;
- unsigned long flags;
+ struct usb_iso_packet_descriptor *isoc_out, *isoc_in;
+ bool period_elapsed;
int total_length = 0;
int frame_count;
int frame;
@@ -299,11 +304,13 @@ static void usb6fire_pcm_in_urb_handler(struct urb *usb_urb)
if (usb_urb->status || rt->panic || rt->stream_state == STREAM_STOPPING)
return;
- for (i = 0; i < PCM_N_PACKETS_PER_URB; i++)
- if (in_urb->packets[i].status) {
+ for (i = 0; i < PCM_N_PACKETS_PER_URB; i++) {
+ isoc_in = &in_urb->instance->iso_frame_desc[i];
+ if (isoc_in->status) {
rt->panic = true;
return;
}
+ }
if (rt->stream_state == STREAM_DISABLED) {
dev_err(&rt->chip->dev->dev,
@@ -313,48 +320,60 @@ static void usb6fire_pcm_in_urb_handler(struct urb *usb_urb)
/* receive our capture data */
sub = &rt->capture;
- spin_lock_irqsave(&sub->lock, flags);
- if (sub->active) {
- usb6fire_pcm_capture(sub, in_urb);
- if (sub->period_off >= sub->instance->runtime->period_size) {
- sub->period_off %= sub->instance->runtime->period_size;
- spin_unlock_irqrestore(&sub->lock, flags);
- snd_pcm_period_elapsed(sub->instance);
- } else
- spin_unlock_irqrestore(&sub->lock, flags);
- } else
- spin_unlock_irqrestore(&sub->lock, flags);
+ period_elapsed = false;
+ scoped_guard(spinlock_irqsave, &sub->lock) {
+ if (sub->active) {
+ usb6fire_pcm_capture(sub, in_urb);
+ if (sub->period_off >= sub->instance->runtime->period_size) {
+ sub->period_off %= sub->instance->runtime->period_size;
+ period_elapsed = true;
+ }
+ }
+ }
+ if (period_elapsed)
+ snd_pcm_period_elapsed(sub->instance);
/* setup out urb structure */
for (i = 0; i < PCM_N_PACKETS_PER_URB; i++) {
- out_urb->packets[i].offset = total_length;
- out_urb->packets[i].length = (in_urb->packets[i].actual_length
- - 4) / (rt->in_n_analog << 2)
- * (rt->out_n_analog << 2) + 4;
- out_urb->packets[i].status = 0;
- total_length += out_urb->packets[i].length;
+ unsigned int frames = 0;
+
+ isoc_out = &out_urb->instance->iso_frame_desc[i];
+ isoc_in = &in_urb->instance->iso_frame_desc[i];
+ if (isoc_in->actual_length > 4)
+ frames = (isoc_in->actual_length - 4)
+ / (rt->in_n_analog << 2);
+ frames = min_t(unsigned int, frames,
+ (rt->out_packet_size - 4)
+ / (rt->out_n_analog << 2));
+
+ isoc_out->offset = total_length;
+ isoc_out->length = frames * (rt->out_n_analog << 2) + 4;
+ isoc_out->status = 0;
+ total_length += isoc_out->length;
}
memset(out_urb->buffer, 0, total_length);
/* now send our playback data (if a free out urb was found) */
sub = &rt->playback;
- spin_lock_irqsave(&sub->lock, flags);
- if (sub->active) {
- usb6fire_pcm_playback(sub, out_urb);
- if (sub->period_off >= sub->instance->runtime->period_size) {
- sub->period_off %= sub->instance->runtime->period_size;
- spin_unlock_irqrestore(&sub->lock, flags);
- snd_pcm_period_elapsed(sub->instance);
- } else
- spin_unlock_irqrestore(&sub->lock, flags);
- } else
- spin_unlock_irqrestore(&sub->lock, flags);
+ period_elapsed = false;
+ scoped_guard(spinlock_irqsave, &sub->lock) {
+ if (sub->active) {
+ usb6fire_pcm_playback(sub, out_urb);
+ if (sub->period_off >= sub->instance->runtime->period_size) {
+ sub->period_off %= sub->instance->runtime->period_size;
+ period_elapsed = true;
+ }
+ }
+ }
+ if (period_elapsed)
+ snd_pcm_period_elapsed(sub->instance);
/* setup the 4th byte of each sample (0x40 for analog channels) */
dest = out_urb->buffer;
- for (i = 0; i < PCM_N_PACKETS_PER_URB; i++)
- if (out_urb->packets[i].length >= 4) {
- frame_count = (out_urb->packets[i].length - 4)
+ for (i = 0; i < PCM_N_PACKETS_PER_URB; i++) {
+ isoc_out = &out_urb->instance->iso_frame_desc[i];
+ if (isoc_out->length >= 4) {
+ frame_count = (isoc_out->length - 4)
/ (rt->out_n_analog << 2);
*(dest++) = 0xaa;
*(dest++) = 0xaa;
@@ -368,8 +387,10 @@ static void usb6fire_pcm_in_urb_handler(struct urb *usb_urb)
*(dest++) = 0x40;
}
}
- usb_submit_urb(&out_urb->instance, GFP_ATOMIC);
- usb_submit_urb(&in_urb->instance, GFP_ATOMIC);
+ }
+
+ usb_submit_urb(out_urb->instance, GFP_ATOMIC);
+ usb_submit_urb(in_urb->instance, GFP_ATOMIC);
}
static void usb6fire_pcm_out_urb_handler(struct urb *usb_urb)
@@ -392,7 +413,7 @@ static int usb6fire_pcm_open(struct snd_pcm_substream *alsa_sub)
if (rt->panic)
return -EPIPE;
- mutex_lock(&rt->stream_mutex);
+ guard(mutex)(&rt->stream_mutex);
alsa_rt->hw = pcm_hw;
if (alsa_sub->stream == SNDRV_PCM_STREAM_PLAYBACK) {
@@ -408,14 +429,12 @@ static int usb6fire_pcm_open(struct snd_pcm_substream *alsa_sub)
}
if (!sub) {
- mutex_unlock(&rt->stream_mutex);
dev_err(&rt->chip->dev->dev, "invalid stream type.\n");
return -EINVAL;
}
sub->instance = alsa_sub;
sub->active = false;
- mutex_unlock(&rt->stream_mutex);
return 0;
}
@@ -423,18 +442,17 @@ static int usb6fire_pcm_close(struct snd_pcm_substream *alsa_sub)
{
struct pcm_runtime *rt = snd_pcm_substream_chip(alsa_sub);
struct pcm_substream *sub = usb6fire_pcm_get_substream(alsa_sub);
- unsigned long flags;
if (rt->panic)
return 0;
- mutex_lock(&rt->stream_mutex);
+ guard(mutex)(&rt->stream_mutex);
if (sub) {
/* deactivate substream */
- spin_lock_irqsave(&sub->lock, flags);
- sub->instance = NULL;
- sub->active = false;
- spin_unlock_irqrestore(&sub->lock, flags);
+ scoped_guard(spinlock_irqsave, &sub->lock) {
+ sub->instance = NULL;
+ sub->active = false;
+ }
/* all substreams closed? if so, stop streaming */
if (!rt->playback.instance && !rt->capture.instance) {
@@ -442,7 +460,6 @@ static int usb6fire_pcm_close(struct snd_pcm_substream *alsa_sub)
rt->rate = ARRAY_SIZE(rates);
}
}
- mutex_unlock(&rt->stream_mutex);
return 0;
}
@@ -458,7 +475,7 @@ static int usb6fire_pcm_prepare(struct snd_pcm_substream *alsa_sub)
if (!sub)
return -ENODEV;
- mutex_lock(&rt->stream_mutex);
+ guard(mutex)(&rt->stream_mutex);
sub->dma_off = 0;
sub->period_off = 0;
@@ -467,7 +484,6 @@ static int usb6fire_pcm_prepare(struct snd_pcm_substream *alsa_sub)
if (alsa_rt->rate == rates[rt->rate])
break;
if (rt->rate == ARRAY_SIZE(rates)) {
- mutex_unlock(&rt->stream_mutex);
dev_err(&rt->chip->dev->dev,
"invalid rate %d in prepare.\n",
alsa_rt->rate);
@@ -475,19 +491,15 @@ static int usb6fire_pcm_prepare(struct snd_pcm_substream *alsa_sub)
}
ret = usb6fire_pcm_set_rate(rt);
- if (ret) {
- mutex_unlock(&rt->stream_mutex);
+ if (ret)
return ret;
- }
ret = usb6fire_pcm_stream_start(rt);
if (ret) {
- mutex_unlock(&rt->stream_mutex);
dev_err(&rt->chip->dev->dev,
"could not start pcm stream.\n");
return ret;
}
}
- mutex_unlock(&rt->stream_mutex);
return 0;
}
@@ -495,26 +507,22 @@ static int usb6fire_pcm_trigger(struct snd_pcm_substream *alsa_sub, int cmd)
{
struct pcm_substream *sub = usb6fire_pcm_get_substream(alsa_sub);
struct pcm_runtime *rt = snd_pcm_substream_chip(alsa_sub);
- unsigned long flags;
if (rt->panic)
return -EPIPE;
if (!sub)
return -ENODEV;
+ guard(spinlock_irqsave)(&sub->lock);
switch (cmd) {
case SNDRV_PCM_TRIGGER_START:
case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
- spin_lock_irqsave(&sub->lock, flags);
sub->active = true;
- spin_unlock_irqrestore(&sub->lock, flags);
return 0;
case SNDRV_PCM_TRIGGER_STOP:
case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
- spin_lock_irqsave(&sub->lock, flags);
sub->active = false;
- spin_unlock_irqrestore(&sub->lock, flags);
return 0;
default:
@@ -527,15 +535,13 @@ static snd_pcm_uframes_t usb6fire_pcm_pointer(
{
struct pcm_substream *sub = usb6fire_pcm_get_substream(alsa_sub);
struct pcm_runtime *rt = snd_pcm_substream_chip(alsa_sub);
- unsigned long flags;
snd_pcm_uframes_t ret;
if (rt->panic || !sub)
return SNDRV_PCM_POS_XRUN;
- spin_lock_irqsave(&sub->lock, flags);
+ guard(spinlock_irqsave)(&sub->lock);
ret = sub->dma_off;
- spin_unlock_irqrestore(&sub->lock, flags);
return ret;
}
@@ -547,22 +553,25 @@ static const struct snd_pcm_ops pcm_ops = {
.pointer = usb6fire_pcm_pointer,
};
-static void usb6fire_pcm_init_urb(struct pcm_urb *urb,
- struct sfire_chip *chip, bool in, int ep,
- void (*handler)(struct urb *))
+static int usb6fire_pcm_init_urb(struct pcm_urb *urb,
+ struct sfire_chip *chip, bool in, int ep,
+ void (*handler)(struct urb *))
{
urb->chip = chip;
- usb_init_urb(&urb->instance);
- urb->instance.transfer_buffer = urb->buffer;
- urb->instance.transfer_buffer_length =
+ urb->instance = usb_alloc_urb(PCM_N_PACKETS_PER_URB, GFP_KERNEL);
+ if (!urb->instance)
+ return -ENOMEM;
+ urb->instance->transfer_buffer = urb->buffer;
+ urb->instance->transfer_buffer_length =
PCM_N_PACKETS_PER_URB * PCM_MAX_PACKET_SIZE;
- urb->instance.dev = chip->dev;
- urb->instance.pipe = in ? usb_rcvisocpipe(chip->dev, ep)
+ urb->instance->dev = chip->dev;
+ urb->instance->pipe = in ? usb_rcvisocpipe(chip->dev, ep)
: usb_sndisocpipe(chip->dev, ep);
- urb->instance.interval = 1;
- urb->instance.complete = handler;
- urb->instance.context = urb;
- urb->instance.number_of_packets = PCM_N_PACKETS_PER_URB;
+ urb->instance->interval = 1;
+ urb->instance->complete = handler;
+ urb->instance->context = urb;
+ urb->instance->number_of_packets = PCM_N_PACKETS_PER_URB;
+ return 0;
}
static int usb6fire_pcm_buffers_init(struct pcm_runtime *rt)
@@ -584,14 +593,23 @@ static int usb6fire_pcm_buffers_init(struct pcm_runtime *rt)
return 0;
}
-static void usb6fire_pcm_buffers_destroy(struct pcm_runtime *rt)
+static void usb6fire_pcm_free(struct pcm_runtime *rt)
{
int i;
+ if (!rt)
+ return;
+
+ if (rt->chip)
+ rt->chip->pcm = NULL;
+
for (i = 0; i < PCM_N_URBS; i++) {
+ usb_free_urb(rt->out_urbs[i].instance);
kfree(rt->out_urbs[i].buffer);
+ usb_free_urb(rt->in_urbs[i].instance);
kfree(rt->in_urbs[i].buffer);
}
+ kfree(rt);
}
int usb6fire_pcm_init(struct sfire_chip *chip)
@@ -606,11 +624,8 @@ int usb6fire_pcm_init(struct sfire_chip *chip)
return -ENOMEM;
ret = usb6fire_pcm_buffers_init(rt);
- if (ret) {
- usb6fire_pcm_buffers_destroy(rt);
- kfree(rt);
- return ret;
- }
+ if (ret)
+ goto error;
rt->chip = chip;
rt->stream_state = STREAM_DISABLED;
@@ -622,10 +637,14 @@ int usb6fire_pcm_init(struct sfire_chip *chip)
spin_lock_init(&rt->capture.lock);
for (i = 0; i < PCM_N_URBS; i++) {
- usb6fire_pcm_init_urb(&rt->in_urbs[i], chip, true, IN_EP,
- usb6fire_pcm_in_urb_handler);
- usb6fire_pcm_init_urb(&rt->out_urbs[i], chip, false, OUT_EP,
- usb6fire_pcm_out_urb_handler);
+ ret = usb6fire_pcm_init_urb(&rt->in_urbs[i], chip, true, IN_EP,
+ usb6fire_pcm_in_urb_handler);
+ if (ret < 0)
+ goto error;
+ ret = usb6fire_pcm_init_urb(&rt->out_urbs[i], chip, false, OUT_EP,
+ usb6fire_pcm_out_urb_handler);
+ if (ret < 0)
+ goto error;
rt->in_urbs[i].peer = &rt->out_urbs[i];
rt->out_urbs[i].peer = &rt->in_urbs[i];
@@ -633,10 +652,8 @@ int usb6fire_pcm_init(struct sfire_chip *chip)
ret = snd_pcm_new(chip->card, "DMX6FireUSB", 0, 1, 1, &pcm);
if (ret < 0) {
- usb6fire_pcm_buffers_destroy(rt);
- kfree(rt);
dev_err(&chip->dev->dev, "cannot create pcm instance.\n");
- return ret;
+ goto error;
}
pcm->private_data = rt;
@@ -649,6 +666,10 @@ int usb6fire_pcm_init(struct sfire_chip *chip)
chip->pcm = rt;
return 0;
+
+ error:
+ usb6fire_pcm_free(rt);
+ return ret;
}
void usb6fire_pcm_abort(struct sfire_chip *chip)
@@ -666,8 +687,8 @@ void usb6fire_pcm_abort(struct sfire_chip *chip)
snd_pcm_stop_xrun(rt->capture.instance);
for (i = 0; i < PCM_N_URBS; i++) {
- usb_poison_urb(&rt->in_urbs[i].instance);
- usb_poison_urb(&rt->out_urbs[i].instance);
+ usb_poison_urb(rt->in_urbs[i].instance);
+ usb_poison_urb(rt->out_urbs[i].instance);
}
}
@@ -675,9 +696,5 @@ void usb6fire_pcm_abort(struct sfire_chip *chip)
void usb6fire_pcm_destroy(struct sfire_chip *chip)
{
- struct pcm_runtime *rt = chip->pcm;
-
- usb6fire_pcm_buffers_destroy(rt);
- kfree(rt);
- chip->pcm = NULL;
+ usb6fire_pcm_free(chip->pcm);
}
diff --git a/sound/usb/6fire/pcm.h b/sound/usb/6fire/pcm.h
index 5a092dfd69f5..b586fe220fd1 100644
--- a/sound/usb/6fire/pcm.h
+++ b/sound/usb/6fire/pcm.h
@@ -24,10 +24,7 @@ enum /* settings for pcm */
struct pcm_urb {
struct sfire_chip *chip;
- /* BEGIN DO NOT SEPARATE */
- struct urb instance;
- struct usb_iso_packet_descriptor packets[PCM_N_PACKETS_PER_URB];
- /* END DO NOT SEPARATE */
+ struct urb *instance;
u8 *buffer;
struct pcm_urb *peer;
diff --git a/sound/usb/bcd2000/bcd2000.c b/sound/usb/bcd2000/bcd2000.c
index c7e7149c6dab..dce6d1f611f9 100644
--- a/sound/usb/bcd2000/bcd2000.c
+++ b/sound/usb/bcd2000/bcd2000.c
@@ -43,6 +43,7 @@ struct bcd2000 {
struct usb_interface *intf;
int card_index;
+ spinlock_t midi_lock;
int midi_out_active;
struct snd_rawmidi *rmidi;
struct snd_rawmidi_substream *midi_receive_substream;
@@ -90,6 +91,8 @@ static void bcd2000_midi_input_trigger(struct snd_rawmidi_substream *substream,
int up)
{
struct bcd2000 *bcd2k = substream->rmidi->private_data;
+
+ guard(spinlock_irqsave)(&bcd2k->midi_lock);
bcd2k->midi_receive_substream = up ? substream : NULL;
}
@@ -195,6 +198,8 @@ static void bcd2000_midi_output_trigger(struct snd_rawmidi_substream *substream,
{
struct bcd2000 *bcd2k = substream->rmidi->private_data;
+ guard(spinlock_irqsave)(&bcd2k->midi_lock);
+
if (up) {
bcd2k->midi_out_substream = substream;
/* check if there is data userspace wants to send */
@@ -219,6 +224,7 @@ static void bcd2000_output_complete(struct urb *urb)
return;
/* check if there is more data userspace wants to send */
+ guard(spinlock_irqsave)(&bcd2k->midi_lock);
bcd2000_midi_send(bcd2k);
}
@@ -234,6 +240,8 @@ static void bcd2000_input_complete(struct urb *urb)
if (!bcd2k || urb->status == -ESHUTDOWN)
return;
+ guard(spinlock_irqsave)(&bcd2k->midi_lock);
+
if (urb->actual_length > 0)
bcd2000_midi_handle_input(bcd2k, urb->transfer_buffer,
urb->actual_length);
@@ -348,16 +356,26 @@ static int bcd2000_init_midi(struct bcd2000 *bcd2k)
return 0;
}
+static void bcd2000_midi_free(struct bcd2000 *bcd2k,
+ struct urb **urb_p)
+{
+ struct urb *urb = *urb_p;
+
+ if (!urb)
+ return;
+
+ usb_poison_urb(urb);
+ scoped_guard(spinlock_irq, &bcd2k->midi_lock)
+ *urb_p = NULL;
+
+ usb_free_urb(urb);
+}
+
static void bcd2000_free_usb_related_resources(struct bcd2000 *bcd2k,
struct usb_interface *interface)
{
- usb_poison_urb(bcd2k->midi_out_urb);
- usb_poison_urb(bcd2k->midi_in_urb);
-
- usb_free_urb(bcd2k->midi_out_urb);
- usb_free_urb(bcd2k->midi_in_urb);
- bcd2k->midi_out_urb = NULL;
- bcd2k->midi_in_urb = NULL;
+ bcd2000_midi_free(bcd2k, &bcd2k->midi_out_urb);
+ bcd2000_midi_free(bcd2k, &bcd2k->midi_in_urb);
if (bcd2k->intf) {
usb_set_intfdata(bcd2k->intf, NULL);
@@ -397,6 +415,7 @@ static int bcd2000_probe(struct usb_interface *interface,
bcd2k->card = card;
bcd2k->card_index = card_index;
bcd2k->intf = interface;
+ spin_lock_init(&bcd2k->midi_lock);
snd_card_set_dev(card, &interface->dev);
diff --git a/sound/usb/card.h b/sound/usb/card.h
index 6ec95b2edf86..0d860c57d9d9 100644
--- a/sound/usb/card.h
+++ b/sound/usb/card.h
@@ -89,7 +89,7 @@ struct snd_usb_endpoint {
struct snd_urb_ctx urb[MAX_URBS];
struct snd_usb_packet_info {
- uint32_t packet_size[MAX_PACKS_HS];
+ int packet_size[MAX_PACKS_HS];
int packets;
} next_packet[MAX_URBS];
unsigned int next_packet_head; /* ring buffer offset to read */
@@ -115,6 +115,7 @@ struct snd_usb_endpoint {
unsigned int phase; /* phase accumulator */
unsigned int maxpacksize; /* max packet size in bytes */
unsigned int maxframesize; /* max packet size in frames */
+ unsigned int max_urb_packs; /* packets allocated per data URB */
unsigned int max_urb_frames; /* max URB size in frames */
unsigned int curpacksize; /* current packet size in bytes (for capture) */
unsigned int curframesize; /* current packet size in frames (for capture) */
diff --git a/sound/usb/endpoint.c b/sound/usb/endpoint.c
index c573d74641d4..a7ec1e31da38 100644
--- a/sound/usb/endpoint.c
+++ b/sound/usb/endpoint.c
@@ -478,7 +478,7 @@ int snd_usb_queue_pending_output_urbs(struct snd_usb_endpoint *ep,
unsigned long flags;
struct snd_usb_packet_info *packet;
struct snd_urb_ctx *ctx = NULL;
- int err, i;
+ int err;
spin_lock_irqsave(&ep->lock, flags);
if ((!implicit_fb || ep->next_packet_queued > 0) &&
@@ -497,9 +497,10 @@ int snd_usb_queue_pending_output_urbs(struct snd_usb_endpoint *ep,
/* copy over the length information */
if (implicit_fb) {
- ctx->packets = packet->packets;
- for (i = 0; i < packet->packets; i++)
- ctx->packet_size[i] = packet->packet_size[i];
+ ctx->packets = min_t(int, packet->packets,
+ ep->max_urb_packs);
+ memcpy(ctx->packet_size, packet->packet_size,
+ ctx->packets * sizeof(packet->packet_size[0]));
}
/* call the data handler to fill in playback data */
@@ -1259,15 +1260,16 @@ static int data_ep_set_params(struct snd_usb_endpoint *ep)
ep->nurbs = min(max_urbs, urbs_per_period * ep->cur_buffer_periods);
}
+ if (fmt->fmt_type == UAC_FORMAT_TYPE_II)
+ urb_packs++; /* for transfer delimiter */
+ ep->max_urb_packs = urb_packs;
+
/* allocate and initialize data urbs */
for (i = 0; i < ep->nurbs; i++) {
struct snd_urb_ctx *u = &ep->urb[i];
u->index = i;
u->ep = ep;
u->packets = urb_packs;
-
- if (fmt->fmt_type == UAC_FORMAT_TYPE_II)
- u->packets++; /* for transfer delimiter */
u->buffer_size = maxsize * u->packets;
u->urb = usb_alloc_urb(u->packets, GFP_KERNEL);
if (!u->urb)
diff --git a/sound/virtio/virtio_card.c b/sound/virtio/virtio_card.c
index 965209e1d872..b16db3838e89 100644
--- a/sound/virtio/virtio_card.c
+++ b/sound/virtio/virtio_card.c
@@ -359,8 +359,8 @@ static void virtsnd_remove(struct virtio_device *vdev)
if (snd->card)
snd_card_free(snd->card);
- vdev->config->del_vqs(vdev);
virtio_reset_device(vdev);
+ vdev->config->del_vqs(vdev);
for (i = 0; snd->substreams && i < snd->nsubstreams; ++i) {
struct virtio_pcm_substream *vss = &snd->substreams[i];
@@ -388,8 +388,8 @@ static int virtsnd_freeze(struct virtio_device *vdev)
virtsnd_disable_event_vq(snd);
virtsnd_ctl_msg_cancel_all(snd);
- vdev->config->del_vqs(vdev);
virtio_reset_device(vdev);
+ vdev->config->del_vqs(vdev);
for (i = 0; i < snd->nsubstreams; ++i)
cancel_work_sync(&snd->substreams[i].elapsed_period);
diff --git a/tools/bootconfig/main.c b/tools/bootconfig/main.c
index 32cf48f2da9a..5ccdb9881b5a 100644
--- a/tools/bootconfig/main.c
+++ b/tools/bootconfig/main.c
@@ -16,6 +16,10 @@
#define pr_err(fmt, ...) fprintf(stderr, fmt, ##__VA_ARGS__)
+/* Bootconfig footer is [size][csum][BOOTCONFIG_MAGIC]. */
+#define BOOTCONFIG_FOOTER_SIZE \
+ (sizeof(uint32_t) * 2 + BOOTCONFIG_MAGIC_LEN)
+
static int xbc_show_value(struct xbc_node *node, bool semicolon)
{
const char *val, *eol;
@@ -135,6 +139,9 @@ static int load_xbc_fd(int fd, char **buf, int size)
{
int ret;
+ if (size < 0 || size > XBC_DATA_MAX)
+ return -EINVAL;
+
*buf = malloc(size + 1);
if (!*buf)
return -ENOMEM;
@@ -163,6 +170,13 @@ static int load_xbc_file(const char *path, char **buf)
return ret;
}
+ if (stat.st_size > XBC_DATA_MAX) {
+ pr_err("%s size is too big\n", path);
+ ret = -E2BIG;
+ close(fd);
+ return ret;
+ }
+
ret = load_xbc_fd(fd, buf, stat.st_size);
close(fd);
@@ -188,10 +202,10 @@ static int load_xbc_from_initrd(int fd, char **buf)
if (ret < 0)
return -errno;
- if (stat.st_size < 8 + BOOTCONFIG_MAGIC_LEN)
+ if (stat.st_size < BOOTCONFIG_FOOTER_SIZE)
return 0;
- if (lseek(fd, -BOOTCONFIG_MAGIC_LEN, SEEK_END) < 0)
+ if (lseek(fd, -(off_t)BOOTCONFIG_MAGIC_LEN, SEEK_END) < 0)
return pr_errno("Failed to lseek for magic", -errno);
if (read(fd, magic, BOOTCONFIG_MAGIC_LEN) < 0)
@@ -201,7 +215,7 @@ static int load_xbc_from_initrd(int fd, char **buf)
if (memcmp(magic, BOOTCONFIG_MAGIC, BOOTCONFIG_MAGIC_LEN) != 0)
return 0;
- if (lseek(fd, -(8 + BOOTCONFIG_MAGIC_LEN), SEEK_END) < 0)
+ if (lseek(fd, -(off_t)BOOTCONFIG_FOOTER_SIZE, SEEK_END) < 0)
return pr_errno("Failed to lseek for size", -errno);
if (read(fd, &size, sizeof(uint32_t)) < 0)
@@ -213,12 +227,13 @@ static int load_xbc_from_initrd(int fd, char **buf)
csum = le32toh(csum);
/* Wrong size error */
- if (stat.st_size < size + 8 + BOOTCONFIG_MAGIC_LEN) {
+ if (size > XBC_DATA_MAX ||
+ size > stat.st_size - BOOTCONFIG_FOOTER_SIZE) {
pr_err("bootconfig size is too big\n");
return -E2BIG;
}
- if (lseek(fd, stat.st_size - (size + 8 + BOOTCONFIG_MAGIC_LEN),
+ if (lseek(fd, stat.st_size - (size + BOOTCONFIG_FOOTER_SIZE),
SEEK_SET) < 0)
return pr_errno("Failed to lseek", -errno);
@@ -349,7 +364,7 @@ static int delete_xbc(const char *path)
ret = fstat(fd, &stat);
if (!ret)
ret = ftruncate(fd, stat.st_size
- - size - 8 - BOOTCONFIG_MAGIC_LEN);
+ - size - BOOTCONFIG_FOOTER_SIZE);
if (ret)
ret = -errno;
} /* Ignore if there is no boot config in initrd */
@@ -379,8 +394,7 @@ static int apply_xbc(const char *path, const char *xbc_path)
csum = xbc_calc_checksum(buf, size);
/* Backup the bootconfig data */
- data = calloc(size + BOOTCONFIG_ALIGN +
- sizeof(uint32_t) + sizeof(uint32_t) + BOOTCONFIG_MAGIC_LEN, 1);
+ data = calloc(size + BOOTCONFIG_ALIGN + BOOTCONFIG_FOOTER_SIZE, 1);
if (!data)
return -ENOMEM;
memcpy(data, buf, size);
@@ -428,7 +442,7 @@ static int apply_xbc(const char *path, const char *xbc_path)
}
/* To align up the total size to BOOTCONFIG_ALIGN, get padding size */
- total_size = stat.st_size + size + sizeof(uint32_t) * 2 + BOOTCONFIG_MAGIC_LEN;
+ total_size = stat.st_size + size + BOOTCONFIG_FOOTER_SIZE;
pad = ((total_size + BOOTCONFIG_ALIGN - 1) & (~BOOTCONFIG_ALIGN_MASK)) - total_size;
size += pad;
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index ffb0d1f7e9a0..214f8b48c1ea 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -5983,6 +5983,13 @@ bpf_object__relocate_core(struct bpf_object *obj, const char *targ_btf_path)
return -EINVAL;
insn = &prog->insns[insn_idx];
+ if (is_ldimm64_insn(insn) && (size_t)insn_idx + 1 >= prog->insns_cnt) {
+ pr_warn("prog '%s': relo #%d: insn #%d (LDIMM64) is truncated\n",
+ prog->name, i, insn_idx);
+ err = -EINVAL;
+ goto out;
+ }
+
err = record_relo_core(prog, rec, insn_idx);
if (err) {
pr_warn("prog '%s': relo #%d: failed to record relocation: %d\n",
diff --git a/tools/lib/bpf/relo_core.c b/tools/lib/bpf/relo_core.c
index 04c8febfc0aa..4367378d272b 100644
--- a/tools/lib/bpf/relo_core.c
+++ b/tools/lib/bpf/relo_core.c
@@ -965,23 +965,30 @@ static int bpf_core_calc_relo(const char *prog_name,
}
/*
- * Turn instruction for which CO_RE relocation failed into invalid one with
+ * Turn instruction for which CO-RE relocation failed into invalid one with
* distinct signature.
*/
-static void bpf_core_poison_insn(const char *prog_name, int relo_idx,
- int insn_idx, struct bpf_insn *insn)
+static int bpf_core_poison_insn(const char *prog_name, int relo_idx,
+ struct bpf_insn *insn, int insn_idx)
{
- pr_debug("prog '%s': relo #%d: substituting insn #%d w/ invalid insn\n",
- prog_name, relo_idx, insn_idx);
- insn->code = BPF_JMP | BPF_CALL;
- insn->dst_reg = 0;
- insn->src_reg = 0;
- insn->off = 0;
- /* if this instruction is reachable (not a dead code),
- * verifier will complain with the following message:
- * invalid func unknown#195896080
- */
- insn->imm = 195896080; /* => 0xbad2310 => "bad relo" */
+ int insn_cnt = is_ldimm64_insn(insn) ? 2 : 1;
+ int i;
+
+ for (i = 0; i < insn_cnt; i++) {
+ pr_debug("prog '%s': relo #%d: substituting insn #%d w/ invalid insn\n",
+ prog_name, relo_idx, insn_idx + i);
+ insn[i].code = BPF_JMP | BPF_CALL;
+ insn[i].dst_reg = 0;
+ insn[i].src_reg = 0;
+ insn[i].off = 0;
+ /*
+ * If this instruction is reachable (not dead code), the verifier
+ * will complain with "invalid func unknown#195896080".
+ */
+ insn[i].imm = 195896080; /* => 0xbad2310 => "bad relo" */
+ }
+
+ return 0;
}
static int insn_bpf_size_to_bytes(struct bpf_insn *insn)
@@ -1032,17 +1039,6 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
class = BPF_CLASS(insn->code);
- if (res->poison) {
-poison:
- /* poison second part of ldimm64 to avoid confusing error from
- * verifier about "unknown opcode 00"
- */
- if (is_ldimm64_insn(insn))
- bpf_core_poison_insn(prog_name, relo_idx, insn_idx + 1, insn + 1);
- bpf_core_poison_insn(prog_name, relo_idx, insn_idx, insn);
- return 0;
- }
-
orig_val = res->orig_val;
new_val = res->new_val;
@@ -1050,7 +1046,9 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
case BPF_ALU:
case BPF_ALU64:
if (BPF_SRC(insn->code) != BPF_K)
- return -EINVAL;
+ goto bad_insn;
+ if (res->poison)
+ return bpf_core_poison_insn(prog_name, relo_idx, insn, insn_idx);
if (res->validate && insn->imm != orig_val) {
pr_warn("prog '%s': relo #%d: unexpected insn #%d (ALU/ALU64) value: got %u, exp %llu -> %llu\n",
prog_name, relo_idx,
@@ -1067,6 +1065,8 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
case BPF_LDX:
case BPF_ST:
case BPF_STX:
+ if (res->poison)
+ return bpf_core_poison_insn(prog_name, relo_idx, insn, insn_idx);
if (res->validate && insn->off != orig_val) {
pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDX/ST/STX) value: got %u, exp %llu -> %llu\n",
prog_name, relo_idx, insn_idx, insn->off, (unsigned long long)orig_val,
@@ -1082,7 +1082,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
pr_warn("prog '%s': relo #%d: insn #%d (LDX/ST/STX) accesses field incorrectly. "
"Make sure you are accessing pointers, unsigned integers, or fields of matching type and size.\n",
prog_name, relo_idx, insn_idx);
- goto poison;
+ return bpf_core_poison_insn(prog_name, relo_idx, insn, insn_idx);
}
orig_val = insn->off;
@@ -1125,6 +1125,9 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
return -EINVAL;
}
+ if (res->poison)
+ return bpf_core_poison_insn(prog_name, relo_idx, insn, insn_idx);
+
imm = (__u32)insn[0].imm | ((__u64)insn[1].imm << 32);
if (res->validate && imm != orig_val) {
pr_warn("prog '%s': relo #%d: unexpected insn #%d (LDIMM64) value: got %llu, exp %llu -> %llu\n",
@@ -1142,6 +1145,7 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn,
break;
}
default:
+bad_insn:
pr_warn("prog '%s': relo #%d: trying to relocate unrecognized insn #%d, code:0x%x, src:0x%x, dst:0x%x, off:0x%x, imm:0x%x\n",
prog_name, relo_idx, insn_idx, insn->code,
insn->src_reg, insn->dst_reg, insn->off, insn->imm);
diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
index b776ee2e445a..a3069d2b334e 100755
--- a/tools/perf/tests/shell/stat_bpf_counters.sh
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
# perf stat --bpf-counters test
# SPDX-License-Identifier: GPL-2.0
diff --git a/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c b/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c
index d23f154d3288..5d9dc8bcfbf5 100644
--- a/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c
+++ b/tools/testing/selftests/arm64/mte/check_gcr_el1_cswitch.c
@@ -69,7 +69,7 @@ void *execute_thread(void *x)
int execute_test(pid_t pid)
{
pthread_t thread_id[MAX_THREADS];
- int thread_data[MAX_THREADS];
+ intptr_t thread_data[MAX_THREADS];
for (int i = 0; i < MAX_THREADS; i++)
pthread_create(&thread_id[i], NULL,
diff --git a/tools/testing/selftests/bpf/prog_tests/linked_list.c b/tools/testing/selftests/bpf/prog_tests/linked_list.c
index 77d07e0a4a55..5744909a561b 100644
--- a/tools/testing/selftests/bpf/prog_tests/linked_list.c
+++ b/tools/testing/selftests/bpf/prog_tests/linked_list.c
@@ -711,7 +711,7 @@ static void test_btf(void)
break;
err = btf__load_into_kernel(btf);
- ASSERT_EQ(err, -ELOOP, "check btf");
+ ASSERT_EQ(err, 0, "check btf");
btf__free(btf);
break;
}
@@ -770,7 +770,7 @@ static void test_btf(void)
break;
err = btf__load_into_kernel(btf);
- ASSERT_EQ(err, -ELOOP, "check btf");
+ ASSERT_EQ(err, 0, "check btf");
btf__free(btf);
break;
}
diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c b/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c
index 92267abb462f..be540970fcc1 100644
--- a/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c
+++ b/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c
@@ -103,9 +103,9 @@ static int dump_tcp_sock(struct seq_file *seq, struct tcp_sock *tp,
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sp->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sp->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = bpf_jiffies64();
diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c b/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c
index 943f7bba180e..9f72398fc3cc 100644
--- a/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c
+++ b/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c
@@ -103,9 +103,9 @@ static int dump_tcp6_sock(struct seq_file *seq, struct tcp6_sock *tp,
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sp->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sp->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = bpf_jiffies64();
diff --git a/tools/testing/selftests/cgroup/cgroup_util.c b/tools/testing/selftests/cgroup/cgroup_util.c
index 1e2d46636a0c..0ef3b8b8d7f7 100644
--- a/tools/testing/selftests/cgroup/cgroup_util.c
+++ b/tools/testing/selftests/cgroup/cgroup_util.c
@@ -20,7 +20,7 @@
#include "../clone3/clone3_selftests.h"
/* Returns read len on success, or -errno on failure. */
-static ssize_t read_text(const char *path, char *buf, size_t max_len)
+ssize_t read_text(const char *path, char *buf, size_t max_len)
{
ssize_t len;
int fd;
@@ -39,7 +39,7 @@ static ssize_t read_text(const char *path, char *buf, size_t max_len)
}
/* Returns written len on success, or -errno on failure. */
-static ssize_t write_text(const char *path, char *buf, ssize_t len)
+ssize_t write_text(const char *path, char *buf, ssize_t len)
{
int fd;
@@ -488,84 +488,6 @@ int cg_run_nowait(const char *cgroup,
return pid;
}
-int get_temp_fd(void)
-{
- return open(".", O_TMPFILE | O_RDWR | O_EXCL);
-}
-
-int alloc_pagecache(int fd, size_t size)
-{
- char buf[PAGE_SIZE];
- struct stat st;
- int i;
-
- if (fstat(fd, &st))
- goto cleanup;
-
- size += st.st_size;
-
- if (ftruncate(fd, size))
- goto cleanup;
-
- for (i = 0; i < size; i += sizeof(buf))
- read(fd, buf, sizeof(buf));
-
- return 0;
-
-cleanup:
- return -1;
-}
-
-int alloc_anon(const char *cgroup, void *arg)
-{
- size_t size = (unsigned long)arg;
- char *buf, *ptr;
-
- buf = malloc(size);
- for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
- *ptr = 0;
-
- free(buf);
- return 0;
-}
-
-int is_swap_enabled(void)
-{
- char buf[PAGE_SIZE];
- const char delim[] = "\n";
- int cnt = 0;
- char *line;
-
- if (read_text("/proc/swaps", buf, sizeof(buf)) <= 0)
- return -1;
-
- for (line = strtok(buf, delim); line; line = strtok(NULL, delim))
- cnt++;
-
- return cnt > 1;
-}
-
-int set_oom_adj_score(int pid, int score)
-{
- char path[PATH_MAX];
- int fd, len;
-
- sprintf(path, "/proc/%d/oom_score_adj", pid);
-
- fd = open(path, O_WRONLY | O_APPEND);
- if (fd < 0)
- return fd;
-
- len = dprintf(fd, "%d", score);
- if (len < 0) {
- close(fd);
- return len;
- }
-
- close(fd);
- return 0;
-}
-
int proc_mount_contains(const char *option)
{
char buf[4 * PAGE_SIZE];
diff --git a/tools/testing/selftests/cgroup/cgroup_util.h b/tools/testing/selftests/cgroup/cgroup_util.h
index 19b131ee7707..139c870ecc28 100644
--- a/tools/testing/selftests/cgroup/cgroup_util.h
+++ b/tools/testing/selftests/cgroup/cgroup_util.h
@@ -21,6 +21,9 @@ static inline int values_close(long a, long b, int err)
return labs(a - b) <= (a + b) / 100 * err;
}
+extern ssize_t read_text(const char *path, char *buf, size_t max_len);
+extern ssize_t write_text(const char *path, char *buf, ssize_t len);
+
extern int cg_find_unified_root(char *root, size_t len, bool *nsdelegate);
extern char *cg_name(const char *root, const char *name);
extern char *cg_name_indexed(const char *root, const char *name, int index);
@@ -49,11 +52,6 @@ extern int cg_enter_current_thread(const char *cgroup);
extern int cg_run_nowait(const char *cgroup,
int (*fn)(const char *cgroup, void *arg),
void *arg);
-extern int get_temp_fd(void);
-extern int alloc_pagecache(int fd, size_t size);
-extern int alloc_anon(const char *cgroup, void *arg);
-extern int is_swap_enabled(void);
-extern int set_oom_adj_score(int pid, int score);
extern int cg_wait_for_proc_count(const char *cgroup, int count);
extern int cg_killall(const char *cgroup);
int proc_mount_contains(const char *option);
diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c
index 7a44d221b8c4..d817aec949fe 100644
--- a/tools/testing/selftests/cgroup/test_memcontrol.c
+++ b/tools/testing/selftests/cgroup/test_memcontrol.c
@@ -24,6 +24,84 @@
static bool has_localevents;
static bool has_recursiveprot;
+int get_temp_fd(void)
+{
+ return open(".", O_TMPFILE | O_RDWR | O_EXCL, 0600);
+}
+
+int alloc_pagecache(int fd, size_t size)
+{
+ char buf[PAGE_SIZE];
+ struct stat st;
+ int i;
+
+ if (fstat(fd, &st))
+ goto cleanup;
+
+ size += st.st_size;
+
+ if (ftruncate(fd, size))
+ goto cleanup;
+
+ for (i = 0; i < size; i += sizeof(buf))
+ read(fd, buf, sizeof(buf));
+
+ return 0;
+
+cleanup:
+ return -1;
+}
+
+int alloc_anon(const char *cgroup, void *arg)
+{
+ size_t size = (unsigned long)arg;
+ char *buf, *ptr;
+
+ buf = malloc(size);
+ for (ptr = buf; ptr < buf + size; ptr += PAGE_SIZE)
+ *ptr = 0;
+
+ free(buf);
+ return 0;
+}
+
+int is_swap_enabled(void)
+{
+ char buf[PAGE_SIZE];
+ const char delim[] = "\n";
+ int cnt = 0;
+ char *line;
+
+ if (read_text("/proc/swaps", buf, sizeof(buf)) <= 0)
+ return -1;
+
+ for (line = strtok(buf, delim); line; line = strtok(NULL, delim))
+ cnt++;
+
+ return cnt > 1;
+}
+
+int set_oom_adj_score(int pid, int score)
+{
+ char path[PATH_MAX];
+ int fd, len;
+
+ sprintf(path, "/proc/%d/oom_score_adj", pid);
+
+ fd = open(path, O_WRONLY | O_APPEND);
+ if (fd < 0)
+ return fd;
+
+ len = dprintf(fd, "%d", score);
+ if (len < 0) {
+ close(fd);
+ return len;
+ }
+
+ close(fd);
+ return 0;
+}
+
/*
* This test creates two nested cgroups with and without enabling
* the memory controller.
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc
index bc9514428dba..07b1177c1634 100644
--- a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_non_uniq_symbol.tc
@@ -6,7 +6,7 @@
SYMBOL='name_show'
# We skip this test on kernel where SYMBOL is unique or does not exist.
-if [ "$(grep -c -E "[[:alnum:]]+ t ${SYMBOL}" /proc/kallsyms)" -le '1' ]; then
+if [ "$(grep -c -E "[[:alnum:]]+ t ${SYMBOL}$" /proc/kallsyms)" -le '1' ]; then
exit_unsupported
fi
diff --git a/tools/testing/selftests/landlock/fs_test.c b/tools/testing/selftests/landlock/fs_test.c
index c781014e6a5c..ae96637ad40e 100644
--- a/tools/testing/selftests/landlock/fs_test.c
+++ b/tools/testing/selftests/landlock/fs_test.c
@@ -85,6 +85,8 @@ static const char file1_s3d1[] = TMP_DIR "/s3d1/f1";
/* dir_s3d2 is a mount point. */
static const char dir_s3d2[] = TMP_DIR "/s3d1/s3d2";
static const char dir_s3d3[] = TMP_DIR "/s3d1/s3d2/s3d3";
+static const char file1_s3d3[] = TMP_DIR "/s3d1/s3d2/s3d3/f1";
+static const char file1_s3d4[] = TMP_DIR "/s3d1/s3d2/s3d4/f1";
/*
* layout1 hierarchy:
@@ -358,7 +360,8 @@ static void create_layout1(struct __test_metadata *const _metadata)
ASSERT_EQ(0, mount_opt(&mnt_tmp, dir_s3d2));
clear_cap(_metadata, CAP_SYS_ADMIN);
- ASSERT_EQ(0, mkdir(dir_s3d3, 0700));
+ create_file(_metadata, file1_s3d3);
+ create_file(_metadata, file1_s3d4);
}
static void remove_layout1(struct __test_metadata *const _metadata)
@@ -378,7 +381,8 @@ static void remove_layout1(struct __test_metadata *const _metadata)
EXPECT_EQ(0, remove_path(dir_s2d2));
EXPECT_EQ(0, remove_path(file1_s3d1));
- EXPECT_EQ(0, remove_path(dir_s3d3));
+ EXPECT_EQ(0, remove_path(file1_s3d3));
+ EXPECT_EQ(0, remove_path(file1_s3d4));
set_cap(_metadata, CAP_SYS_ADMIN);
umount(dir_s3d2);
clear_cap(_metadata, CAP_SYS_ADMIN);
@@ -772,6 +776,27 @@ static int create_ruleset(struct __test_metadata *const _metadata,
return ruleset_fd;
}
+static void enforce_fs(struct __test_metadata *const _metadata,
+ const __u64 access_fs, const struct rule rules[])
+{
+ int ruleset_fd;
+
+ if (rules) {
+ ruleset_fd = create_ruleset(_metadata, access_fs, rules);
+ } else {
+ const struct landlock_ruleset_attr ruleset_attr = {
+ .handled_access_fs = access_fs,
+ };
+
+ ruleset_fd = landlock_create_ruleset(&ruleset_attr,
+ sizeof(ruleset_attr), 0);
+ ASSERT_LE(0, ruleset_fd);
+ }
+
+ enforce_ruleset(_metadata, ruleset_fd);
+ EXPECT_EQ(0, close(ruleset_fd));
+}
+
TEST_F_FORK(layout0, proc_nsfs)
{
const struct rule rules[] = {
@@ -2109,6 +2134,22 @@ static int test_exchange(const char *const oldpath, const char *const newpath)
return 0;
}
+static int test_renameat(int olddirfd, const char *oldpath, int newdirfd,
+ const char *newpath)
+{
+ if (renameat2(olddirfd, oldpath, newdirfd, newpath, 0))
+ return errno;
+ return 0;
+}
+
+static int test_exchangeat(int olddirfd, const char *oldpath, int newdirfd,
+ const char *newpath)
+{
+ if (renameat2(olddirfd, oldpath, newdirfd, newpath, RENAME_EXCHANGE))
+ return errno;
+ return 0;
+}
+
TEST_F_FORK(layout1, rename_file)
{
const struct rule rules[] = {
@@ -2191,6 +2232,170 @@ TEST_F_FORK(layout1, rename_file)
RENAME_EXCHANGE));
}
+TEST_F_FORK(layout1, rename_whiteout_denied)
+{
+ /* The affected file is a FIFO. */
+ ASSERT_EQ(0, unlink(file1_s3d3));
+ ASSERT_EQ(0, mknod(file1_s3d3, S_IFIFO | 0600, 0));
+
+ /* Deny MAKE_REG, but allow MAKE_FIFO. */
+ enforce_fs(_metadata, LANDLOCK_ACCESS_FS_MAKE_REG, NULL);
+
+ /*
+ * Try to rename a file with RENAME_WHITEOUT.
+ * file1_s3d3 is in dir_s3d2 (tmpfs), so it supports RENAME_WHITEOUT.
+ * Denied, because whiteout creation is guarded with MAKE_REG.
+ */
+ EXPECT_EQ(-1, renameat2(AT_FDCWD, file1_s3d3, AT_FDCWD,
+ TMP_DIR "/s3d1/s3d2/s3d3/f2", RENAME_WHITEOUT));
+ EXPECT_EQ(EACCES, errno);
+}
+
+static bool is_whiteout(const char *const path)
+{
+ struct stat st;
+
+ if (stat(path, &st) == -1)
+ return false;
+
+ return S_ISCHR(st.st_mode) && st.st_rdev == makedev(0, 0);
+}
+
+static bool is_fifo(const char *const path)
+{
+ struct stat st;
+
+ return stat(path, &st) == 0 && S_ISFIFO(st.st_mode);
+}
+
+TEST_F_FORK(layout1, rename_whiteout_allowed)
+{
+ const struct rule rules[] = {
+ {
+ .path = dir_s3d3,
+ .access = LANDLOCK_ACCESS_FS_MAKE_REG,
+ },
+ {},
+ };
+
+ /* The affected file is a FIFO. */
+ ASSERT_EQ(0, unlink(file1_s3d3));
+ ASSERT_EQ(0, mknod(file1_s3d3, S_IFIFO | 0600, 0));
+
+ /* Allow MAKE_REG below dir_s3d3. */
+ enforce_fs(_metadata, LANDLOCK_ACCESS_FS_MAKE_REG, rules);
+
+ /*
+ * Rename a file with RENAME_WHITEOUT within the same directory.
+ * Allowed, because MAKE_REG is granted for the whiteout object which
+ * gets created in the source location.
+ */
+ EXPECT_EQ(0, renameat2(AT_FDCWD, file1_s3d3, AT_FDCWD,
+ TMP_DIR "/s3d1/s3d2/s3d3/f2", RENAME_WHITEOUT));
+
+ /* A whiteout object took the place of the moved FIFO. */
+ EXPECT_TRUE(is_whiteout(file1_s3d3));
+ EXPECT_TRUE(is_fifo(TMP_DIR "/s3d1/s3d2/s3d3/f2"));
+}
+
+TEST_F_FORK(layout1, rename_whiteout_reparenting)
+{
+ const struct rule rules[] = {
+ {
+ .path = dir_s3d2,
+ .access = LANDLOCK_ACCESS_FS_REFER,
+ },
+ {
+ .path = dir_s3d3,
+ .access = LANDLOCK_ACCESS_FS_MAKE_REG,
+ },
+ {},
+ };
+
+ /* The moved files are FIFOs. */
+ ASSERT_EQ(0, unlink(file1_s3d3));
+ ASSERT_EQ(0, mknod(file1_s3d3, S_IFIFO | 0600, 0));
+ ASSERT_EQ(0, unlink(file1_s3d4));
+ ASSERT_EQ(0, mknod(file1_s3d4, S_IFIFO | 0600, 0));
+
+ /* Allow REFER below dir_s3d2, but MAKE_REG only below dir_s3d3. */
+ enforce_fs(_metadata,
+ LANDLOCK_ACCESS_FS_MAKE_REG | LANDLOCK_ACCESS_FS_REFER,
+ rules);
+
+ /*
+ * The whiteout object is created in the source directory: Moving the
+ * FIFO out of dir_s3d4 is denied because MAKE_REG is not granted
+ * there, even though it is granted in the destination directory
+ * dir_s3d3.
+ */
+ EXPECT_EQ(-1, renameat2(AT_FDCWD, file1_s3d4, AT_FDCWD,
+ TMP_DIR "/s3d1/s3d2/s3d3/f2", RENAME_WHITEOUT));
+ EXPECT_EQ(EACCES, errno);
+
+ /*
+ * Moving the FIFO out of dir_s3d3 is allowed, because MAKE_REG is
+ * granted there for the created whiteout object.
+ */
+ EXPECT_EQ(0, renameat2(AT_FDCWD, file1_s3d3, AT_FDCWD,
+ TMP_DIR "/s3d1/s3d2/s3d4/f2", RENAME_WHITEOUT));
+
+ /* A whiteout object took the place of the moved FIFO. */
+ EXPECT_TRUE(is_whiteout(file1_s3d3));
+ EXPECT_TRUE(is_fifo(TMP_DIR "/s3d1/s3d2/s3d4/f2"));
+}
+
+TEST_F_FORK(layout1, rename_whiteout_exchange)
+{
+ const char *const whiteout_s3d3 = TMP_DIR "/s3d1/s3d2/s3d3/f2";
+ const struct rule rules[] = {
+ {
+ .path = dir_s3d2,
+ .access = LANDLOCK_ACCESS_FS_REFER,
+ },
+ {
+ .path = dir_s3d3,
+ .access = LANDLOCK_ACCESS_FS_MAKE_REG,
+ },
+ {},
+ };
+
+ /* The exchanged files are FIFOs and an existing whiteout object. */
+ ASSERT_EQ(0, unlink(file1_s3d3));
+ ASSERT_EQ(0, mknod(file1_s3d3, S_IFIFO | 0600, 0));
+ ASSERT_EQ(0, mknod(whiteout_s3d3, S_IFCHR | 0600, makedev(0, 0)));
+ ASSERT_EQ(0, unlink(file1_s3d4));
+ ASSERT_EQ(0, mknod(file1_s3d4, S_IFIFO | 0600, 0));
+
+ /* Allow REFER below dir_s3d2, but MAKE_REG only below dir_s3d3. */
+ enforce_fs(_metadata,
+ LANDLOCK_ACCESS_FS_MAKE_REG | LANDLOCK_ACCESS_FS_REFER,
+ rules);
+
+ /*
+ * With RENAME_EXCHANGE, the whiteout object moves into the source
+ * directory of the rename: Exchanging the FIFO in dir_s3d4 with the
+ * whiteout object is denied because MAKE_REG is not granted in
+ * dir_s3d4, even though it is granted in the whiteout object's own
+ * directory dir_s3d3.
+ */
+ EXPECT_EQ(-1, renameat2(AT_FDCWD, file1_s3d4, AT_FDCWD, whiteout_s3d3,
+ RENAME_EXCHANGE));
+ EXPECT_EQ(EACCES, errno);
+
+ /*
+ * Exchanging the FIFO in dir_s3d3 with the whiteout object is
+ * allowed, because MAKE_REG is granted in the directory into which
+ * the whiteout object moves.
+ */
+ EXPECT_EQ(0, renameat2(AT_FDCWD, file1_s3d3, AT_FDCWD, whiteout_s3d3,
+ RENAME_EXCHANGE));
+
+ /* The FIFO and the whiteout object swapped places. */
+ EXPECT_TRUE(is_whiteout(file1_s3d3));
+ EXPECT_TRUE(is_fifo(whiteout_s3d3));
+}
+
TEST_F_FORK(layout1, rename_dir)
{
const struct rule rules[] = {
@@ -3244,6 +3449,18 @@ TEST_F_FORK(layout1, make_char)
makedev(1, 3));
}
+TEST_F_FORK(layout1, make_whiteout)
+{
+ /*
+ * Creates a whiteout object (creation guarded by MAKE_REG).
+ *
+ * Contrary to the other character devices, this does not require
+ * CAP_MKNOD, cf. vfs_mknod().
+ */
+ test_make_file(_metadata, LANDLOCK_ACCESS_FS_MAKE_REG, S_IFCHR,
+ makedev(0, 0));
+}
+
TEST_F_FORK(layout1, make_block)
{
/* Creates a /dev/loop0 device. */
@@ -4363,6 +4580,18 @@ TEST_F_FORK(ioctl, handle_file_access_file)
FIXTURE(layout1_bind) {};
/* clang-format on */
+static const char bind_dir_s1d3[] = TMP_DIR "/s2d1/s2d2/s1d3";
+static const char bind_file1_s1d3[] = TMP_DIR "/s2d1/s2d2/s1d3/f1";
+
+/* Move targets for disconnected path tests. */
+static const char dir_s4d1[] = TMP_DIR "/s4d1";
+static const char file1_s4d1[] = TMP_DIR "/s4d1/f1";
+static const char file2_s4d1[] = TMP_DIR "/s4d1/f2";
+static const char dir_s4d2[] = TMP_DIR "/s4d1/s4d2";
+static const char file1_s4d2[] = TMP_DIR "/s4d1/s4d2/f1";
+static const char file1_name[] = "f1";
+static const char file2_name[] = "f2";
+
FIXTURE_SETUP(layout1_bind)
{
prepare_layout(_metadata);
@@ -4378,14 +4607,14 @@ FIXTURE_TEARDOWN_PARENT(layout1_bind)
{
/* umount(dir_s2d2)) is handled by namespace lifetime. */
+ remove_path(file1_s4d1);
+ remove_path(file2_s4d1);
+
remove_layout1(_metadata);
cleanup_layout(_metadata);
}
-static const char bind_dir_s1d3[] = TMP_DIR "/s2d1/s2d2/s1d3";
-static const char bind_file1_s1d3[] = TMP_DIR "/s2d1/s2d2/s1d3/f1";
-
/*
* layout1_bind hierarchy:
*
@@ -4396,20 +4625,25 @@ static const char bind_file1_s1d3[] = TMP_DIR "/s2d1/s2d2/s1d3/f1";
* │ └── s1d2
* │ ├── f1
* │ ├── f2
- * │ └── s1d3
+ * │ └── s1d3 [disconnected by path_disconnected]
* │ ├── f1
* │ └── f2
* ├── s2d1
* │ ├── f1
- * │ └── s2d2
+ * │ └── s2d2 [bind mount from s1d2]
* │ ├── f1
* │ ├── f2
* │ └── s1d3
* │ ├── f1
* │ └── f2
- * └── s3d1
- * └── s3d2
- * └── s3d3
+ * ├── s3d1
+ * │ └── s3d2
+ * │ └── s3d3
+ * └── s4d1 [renamed from s1d3 by path_disconnected]
+ * ├── f1
+ * ├── f2
+ * └── s4d2
+ * └── f1
*/
TEST_F_FORK(layout1_bind, no_restriction)
@@ -4608,6 +4842,1431 @@ TEST_F_FORK(layout1_bind, reparent_cross_mount)
ASSERT_EQ(0, rename(bind_file1_s1d3, file1_s2d2));
}
+/*
+ * Make sure access to file through a disconnected path works as expected.
+ * This test moves s1d3 to s4d1.
+ */
+TEST_F_FORK(layout1_bind, path_disconnected)
+{
+ const struct rule layer1_allow_all[] = {
+ {
+ .path = TMP_DIR,
+ .access = ACCESS_ALL,
+ },
+ {},
+ };
+ const struct rule layer2_allow_just_f1[] = {
+ {
+ .path = file1_s1d3,
+ .access = LANDLOCK_ACCESS_FS_READ_FILE,
+ },
+ {},
+ };
+ const struct rule layer3_only_s1d2[] = {
+ {
+ .path = dir_s1d2,
+ .access = LANDLOCK_ACCESS_FS_READ_FILE,
+ },
+ {},
+ };
+
+ /* Landlock should not deny access just because it is disconnected. */
+ int ruleset_fd_l1 =
+ create_ruleset(_metadata, ACCESS_ALL, layer1_allow_all);
+
+ /* Creates the new ruleset now before we move the dir containing the file. */
+ int ruleset_fd_l2 =
+ create_ruleset(_metadata, ACCESS_RW, layer2_allow_just_f1);
+ int ruleset_fd_l3 =
+ create_ruleset(_metadata, ACCESS_RW, layer3_only_s1d2);
+ int bind_s1d3_fd;
+
+ ASSERT_LE(0, ruleset_fd_l1);
+ ASSERT_LE(0, ruleset_fd_l2);
+ ASSERT_LE(0, ruleset_fd_l3);
+
+ enforce_ruleset(_metadata, ruleset_fd_l1);
+ EXPECT_EQ(0, close(ruleset_fd_l1));
+
+ bind_s1d3_fd = open(bind_dir_s1d3, O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, bind_s1d3_fd);
+
+ /* Tests access is possible before we move. */
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file2_name, O_RDONLY));
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, "..", O_RDONLY | O_DIRECTORY));
+
+ /* Makes it disconnected. */
+ ASSERT_EQ(0, rename(dir_s1d3, dir_s4d1))
+ {
+ TH_LOG("Failed to rename %s to %s: %s", dir_s1d3, dir_s4d1,
+ strerror(errno));
+ }
+
+ /* Tests that access is still possible. */
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file2_name, O_RDONLY));
+
+ /*
+ * Tests that ".." is not possible (not because of Landlock, but just
+ * because it's disconnected).
+ */
+ EXPECT_EQ(ENOENT,
+ test_open_rel(bind_s1d3_fd, "..", O_RDONLY | O_DIRECTORY));
+
+ /* This should still work with a narrower rule. */
+ enforce_ruleset(_metadata, ruleset_fd_l2);
+ EXPECT_EQ(0, close(ruleset_fd_l2));
+
+ EXPECT_EQ(0, test_open(file1_s4d1, O_RDONLY));
+ /*
+ * Accessing a file through a disconnected file descriptor can still be
+ * allowed by a rule tied to this file, even if it is no longer visible in
+ * its mount point.
+ */
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+ EXPECT_EQ(EACCES, test_open_rel(bind_s1d3_fd, file2_name, O_RDONLY));
+
+ enforce_ruleset(_metadata, ruleset_fd_l3);
+ EXPECT_EQ(0, close(ruleset_fd_l3));
+
+ EXPECT_EQ(EACCES, test_open(file1_s4d1, O_RDONLY));
+ /*
+ * Accessing a file through a disconnected file descriptor can still be
+ * allowed by a rule tied to the original mount point, even if it is no
+ * longer visible in its mount point.
+ */
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+ EXPECT_EQ(EACCES, test_open_rel(bind_s1d3_fd, file2_name, O_RDONLY));
+}
+
+/*
+ * Test that renameat with disconnected paths works under Landlock. This test
+ * moves s1d3 to s4d2, so that we can have a rule allowing refers on the move
+ * target's immediate parent.
+ */
+TEST_F_FORK(layout1_bind, path_disconnected_rename)
+{
+ const struct rule layer1[] = {
+ {
+ .path = dir_s1d2,
+ .access = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_MAKE_DIR |
+ LANDLOCK_ACCESS_FS_REMOVE_DIR |
+ LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_REMOVE_FILE |
+ LANDLOCK_ACCESS_FS_READ_FILE,
+ },
+ {
+ .path = dir_s4d1,
+ .access = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_MAKE_DIR |
+ LANDLOCK_ACCESS_FS_REMOVE_DIR |
+ LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_REMOVE_FILE |
+ LANDLOCK_ACCESS_FS_READ_FILE,
+ },
+ {}
+ };
+
+ /* This layer only handles LANDLOCK_ACCESS_FS_READ_FILE. */
+ const struct rule layer2_only_s1d2[] = {
+ {
+ .path = dir_s1d2,
+ .access = LANDLOCK_ACCESS_FS_READ_FILE,
+ },
+ {},
+ };
+ int ruleset_fd_l1, ruleset_fd_l2;
+ pid_t child_pid;
+ int bind_s1d3_fd, status;
+
+ ASSERT_EQ(0, mkdir(dir_s4d1, 0755))
+ {
+ TH_LOG("Failed to create %s: %s", dir_s4d1, strerror(errno));
+ }
+ ruleset_fd_l1 = create_ruleset(_metadata, ACCESS_ALL, layer1);
+ ruleset_fd_l2 = create_ruleset(_metadata, LANDLOCK_ACCESS_FS_READ_FILE,
+ layer2_only_s1d2);
+ ASSERT_LE(0, ruleset_fd_l1);
+ ASSERT_LE(0, ruleset_fd_l2);
+
+ enforce_ruleset(_metadata, ruleset_fd_l1);
+ EXPECT_EQ(0, close(ruleset_fd_l1));
+
+ bind_s1d3_fd = open(bind_dir_s1d3, O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, bind_s1d3_fd);
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+
+ /* Tests ENOENT priority over EACCES for disconnected directory. */
+ EXPECT_EQ(EACCES, test_open_rel(bind_s1d3_fd, "..", O_DIRECTORY));
+ ASSERT_EQ(0, rename(dir_s1d3, dir_s4d2))
+ {
+ TH_LOG("Failed to rename %s to %s: %s", dir_s1d3, dir_s4d2,
+ strerror(errno));
+ }
+ EXPECT_EQ(ENOENT, test_open_rel(bind_s1d3_fd, "..", O_DIRECTORY));
+
+ /*
+ * The file is no longer under s1d2 but we should still be able to access it
+ * with layer 2 because its mount point is evaluated as the first valid
+ * directory because it was initially a parent. Do a fork to test this so
+ * we don't prevent ourselves from renaming it back later.
+ */
+ child_pid = fork();
+ ASSERT_LE(0, child_pid);
+ if (child_pid == 0) {
+ enforce_ruleset(_metadata, ruleset_fd_l2);
+ EXPECT_EQ(0, close(ruleset_fd_l2));
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+ EXPECT_EQ(EACCES, test_open(file1_s4d2, O_RDONLY));
+
+ /*
+ * Tests that access widening checks indeed prevents us from renaming it
+ * back.
+ */
+ EXPECT_EQ(-1, rename(dir_s4d2, dir_s1d3));
+ EXPECT_EQ(EXDEV, errno);
+
+ /*
+ * Including through the now disconnected fd (but it should return
+ * EXDEV).
+ */
+ EXPECT_EQ(-1, renameat(bind_s1d3_fd, file1_name, AT_FDCWD,
+ file1_s2d2));
+ EXPECT_EQ(EXDEV, errno);
+ _exit(_metadata->exit_code);
+ return;
+ }
+
+ EXPECT_EQ(child_pid, waitpid(child_pid, &status, 0));
+ EXPECT_EQ(1, WIFEXITED(status));
+ EXPECT_EQ(EXIT_SUCCESS, WEXITSTATUS(status));
+
+ ASSERT_EQ(0, rename(dir_s4d2, dir_s1d3))
+ {
+ TH_LOG("Failed to rename %s back to %s: %s", dir_s4d1, dir_s1d3,
+ strerror(errno));
+ }
+
+ /* Now checks that we can access it under l2. */
+ child_pid = fork();
+ ASSERT_LE(0, child_pid);
+ if (child_pid == 0) {
+ enforce_ruleset(_metadata, ruleset_fd_l2);
+ EXPECT_EQ(0, close(ruleset_fd_l2));
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+ EXPECT_EQ(0, test_open(file1_s1d3, O_RDONLY));
+ _exit(_metadata->exit_code);
+ return;
+ }
+
+ EXPECT_EQ(child_pid, waitpid(child_pid, &status, 0));
+ EXPECT_EQ(1, WIFEXITED(status));
+ EXPECT_EQ(EXIT_SUCCESS, WEXITSTATUS(status));
+
+ /*
+ * Also test that we can rename via a disconnected path. We move the
+ * dir back to the disconnected place first, then we rename file1 to
+ * file2 through our dir fd.
+ */
+ ASSERT_EQ(0, rename(dir_s1d3, dir_s4d2))
+ {
+ TH_LOG("Failed to rename %s to %s: %s", dir_s1d3, dir_s4d2,
+ strerror(errno));
+ }
+ ASSERT_EQ(0,
+ renameat(bind_s1d3_fd, file1_name, bind_s1d3_fd, file2_name))
+ {
+ TH_LOG("Failed to rename %s to %s within disconnected %s: %s",
+ file1_name, file2_name, bind_dir_s1d3, strerror(errno));
+ }
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file2_name, O_RDONLY));
+ ASSERT_EQ(0, renameat(bind_s1d3_fd, file2_name, AT_FDCWD, file1_s2d2))
+ {
+ TH_LOG("Failed to rename %s to %s through disconnected %s: %s",
+ file2_name, file1_s2d2, bind_dir_s1d3, strerror(errno));
+ }
+ EXPECT_EQ(0, test_open(file1_s2d2, O_RDONLY));
+ EXPECT_EQ(0, test_open(file1_s1d2, O_RDONLY));
+
+ /* Move it back using the disconnected path as the target. */
+ ASSERT_EQ(0, renameat(AT_FDCWD, file1_s2d2, bind_s1d3_fd, file1_name))
+ {
+ TH_LOG("Failed to rename %s to %s through disconnected %s: %s",
+ file1_s1d2, file1_name, bind_dir_s1d3, strerror(errno));
+ }
+
+ /* Now make it connected again. */
+ ASSERT_EQ(0, rename(dir_s4d2, dir_s1d3))
+ {
+ TH_LOG("Failed to rename %s back to %s: %s", dir_s4d2, dir_s1d3,
+ strerror(errno));
+ }
+
+ /* Checks again that we can access it under l2. */
+ enforce_ruleset(_metadata, ruleset_fd_l2);
+ EXPECT_EQ(0, close(ruleset_fd_l2));
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+ EXPECT_EQ(0, test_open(file1_s1d3, O_RDONLY));
+}
+
+/*
+ * Test that linkat(2) with disconnected paths works under Landlock. This
+ * test moves s1d3 to s4d1.
+ */
+TEST_F_FORK(layout1_bind, path_disconnected_link)
+{
+ /* Ruleset to be applied after renaming s1d3 to s4d1. */
+ const struct rule layer1[] = {
+ {
+ .path = dir_s4d1,
+ .access = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_REMOVE_FILE,
+ },
+ {
+ .path = dir_s2d2,
+ .access = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_REMOVE_FILE,
+ },
+ {}
+ };
+ int ruleset_fd, bind_s1d3_fd;
+
+ /* Removes unneeded files created by layout1, otherwise it will EEXIST. */
+ ASSERT_EQ(0, unlink(file1_s1d2));
+ ASSERT_EQ(0, unlink(file2_s1d3));
+
+ bind_s1d3_fd = open(bind_dir_s1d3, O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, bind_s1d3_fd);
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY));
+
+ /* Disconnects bind_s1d3_fd. */
+ ASSERT_EQ(0, rename(dir_s1d3, dir_s4d1))
+ {
+ TH_LOG("Failed to rename %s to %s: %s", dir_s1d3, dir_s4d1,
+ strerror(errno));
+ }
+
+ /* Need this later to test different parent link. */
+ ASSERT_EQ(0, mkdir(dir_s4d2, 0755))
+ {
+ TH_LOG("Failed to create %s: %s", dir_s4d2, strerror(errno));
+ }
+
+ ruleset_fd = create_ruleset(_metadata, ACCESS_ALL, layer1);
+ ASSERT_LE(0, ruleset_fd);
+ enforce_ruleset(_metadata, ruleset_fd);
+ EXPECT_EQ(0, close(ruleset_fd));
+
+ /* From disconnected to connected. */
+ ASSERT_EQ(0, linkat(bind_s1d3_fd, file1_name, AT_FDCWD, file1_s2d2, 0))
+ {
+ TH_LOG("Failed to link %s to %s via disconnected %s: %s",
+ file1_name, file1_s2d2, bind_dir_s1d3, strerror(errno));
+ }
+
+ /* Tests that we can access via the new link... */
+ EXPECT_EQ(0, test_open(file1_s2d2, O_RDONLY))
+ {
+ TH_LOG("Failed to open newly linked %s: %s", file1_s2d2,
+ strerror(errno));
+ }
+
+ /* ...as well as the old one. */
+ EXPECT_EQ(0, test_open(file1_s4d1, O_RDONLY))
+ {
+ TH_LOG("Failed to open original %s: %s", file1_s4d1,
+ strerror(errno));
+ }
+
+ /* From connected to disconnected. */
+ ASSERT_EQ(0, unlink(file1_s4d1));
+ ASSERT_EQ(0, linkat(AT_FDCWD, file1_s2d2, bind_s1d3_fd, file2_name, 0))
+ {
+ TH_LOG("Failed to link %s to %s via disconnected %s: %s",
+ file1_s2d2, file2_name, bind_dir_s1d3, strerror(errno));
+ }
+ EXPECT_EQ(0, test_open(file2_s4d1, O_RDONLY));
+ ASSERT_EQ(0, unlink(file1_s2d2));
+
+ /* From disconnected to disconnected (same parent). */
+ ASSERT_EQ(0,
+ linkat(bind_s1d3_fd, file2_name, bind_s1d3_fd, file1_name, 0))
+ {
+ TH_LOG("Failed to link %s to %s within disconnected %s: %s",
+ file2_name, file1_name, bind_dir_s1d3, strerror(errno));
+ }
+ EXPECT_EQ(0, test_open(file1_s4d1, O_RDONLY))
+ {
+ TH_LOG("Failed to open newly linked %s: %s", file1_s4d1,
+ strerror(errno));
+ }
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, file1_name, O_RDONLY))
+ {
+ TH_LOG("Failed to open %s through newly created link under disconnected path: %s",
+ file1_name, strerror(errno));
+ }
+ ASSERT_EQ(0, unlink(file2_s4d1));
+
+ /* From disconnected to disconnected (different parent). */
+ ASSERT_EQ(0,
+ linkat(bind_s1d3_fd, file1_name, bind_s1d3_fd, "s4d2/f1", 0))
+ {
+ TH_LOG("Failed to link %s to %s within disconnected %s: %s",
+ file1_name, "s4d2/f1", bind_dir_s1d3, strerror(errno));
+ }
+ EXPECT_EQ(0, test_open(file1_s4d2, O_RDONLY))
+ {
+ TH_LOG("Failed to open %s after link: %s", file1_s4d2,
+ strerror(errno));
+ }
+ EXPECT_EQ(0, test_open_rel(bind_s1d3_fd, "s4d2/f1", O_RDONLY))
+ {
+ TH_LOG("Failed to open %s through disconnected path after link: %s",
+ "s4d2/f1", strerror(errno));
+ }
+}
+
+/*
+ * layout4_disconnected_leafs with bind mount and renames:
+ *
+ * tmp
+ * ├── s1d1
+ * │ └── s1d2 [source of the bind mount]
+ * │ ├── s1d31
+ * │ │ └── s1d41 [now renamed beneath s3d1]
+ * │ │ ├── f1
+ * │ │ └── f2
+ * │ └── s1d32
+ * │ └── s1d42 [now renamed beneath s4d1]
+ * │ ├── f3
+ * │ └── f4
+ * ├── s2d1
+ * │ └── s2d2 [bind mount of s1d2]
+ * │ ├── s1d31
+ * │ │ └── s1d41 [opened FD, now renamed beneath s3d1]
+ * │ │ ├── f1
+ * │ │ └── f2
+ * │ └── s1d32
+ * │ └── s1d42 [opened FD, now renamed beneath s4d1]
+ * │ ├── f3
+ * │ └── f4
+ * ├── s3d1
+ * │ └── s1d41 [renamed here]
+ * │ ├── f1
+ * │ └── f2
+ * └── s4d1
+ * └── s1d42 [renamed here]
+ * ├── f3
+ * └── f4
+ */
+/* clang-format off */
+FIXTURE(layout4_disconnected_leafs) {
+ int s2d2_fd;
+};
+/* clang-format on */
+
+FIXTURE_SETUP(layout4_disconnected_leafs)
+{
+ prepare_layout(_metadata);
+
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d31/s1d41/f1");
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d31/s1d41/f2");
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d32/s1d42/f3");
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d32/s1d42/f4");
+ create_directory(_metadata, TMP_DIR "/s2d1/s2d2");
+ create_directory(_metadata, TMP_DIR "/s3d1");
+ create_directory(_metadata, TMP_DIR "/s4d1");
+
+ self->s2d2_fd =
+ open(TMP_DIR "/s2d1/s2d2", O_DIRECTORY | O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, self->s2d2_fd);
+
+ set_cap(_metadata, CAP_SYS_ADMIN);
+ ASSERT_EQ(0, mount(TMP_DIR "/s1d1/s1d2", TMP_DIR "/s2d1/s2d2", NULL,
+ MS_BIND, NULL));
+ clear_cap(_metadata, CAP_SYS_ADMIN);
+}
+
+FIXTURE_TEARDOWN_PARENT(layout4_disconnected_leafs)
+{
+ /* umount(TMP_DIR "/s2d1") is handled by namespace lifetime. */
+
+ /* Removes files after renames. */
+ remove_path(TMP_DIR "/s3d1/s1d41/f1");
+ remove_path(TMP_DIR "/s3d1/s1d41/f2");
+ remove_path(TMP_DIR "/s4d1/s1d42/f1");
+ remove_path(TMP_DIR "/s4d1/s1d42/f3");
+ remove_path(TMP_DIR "/s4d1/s1d42/f4");
+ remove_path(TMP_DIR "/s4d1/s1d42/f5");
+
+ cleanup_layout(_metadata);
+}
+
+FIXTURE_VARIANT(layout4_disconnected_leafs)
+{
+ /*
+ * Parent of the bind mount source. It should always be ignored when
+ * testing against files under the s1d41 or s1d42 disconnected directories.
+ */
+ const __u64 allowed_s1d1;
+ /*
+ * Source of bind mount (to s2d2). It should always be enforced when
+ * testing against files under the s1d41 or s1d42 disconnected directories.
+ */
+ const __u64 allowed_s1d2;
+ /*
+ * Original parent of s1d41. It should always be ignored when testing
+ * against files under the s1d41 disconnected directory.
+ */
+ const __u64 allowed_s1d31;
+ /*
+ * Original parent of s1d42. It should always be ignored when testing
+ * against files under the s1d42 disconnected directory.
+ */
+ const __u64 allowed_s1d32;
+ /*
+ * Opened and disconnected source directory. It should always be enforced
+ * when testing against files under the s1d41 disconnected directory.
+ */
+ const __u64 allowed_s1d41;
+ /*
+ * Opened and disconnected source directory. It should always be enforced
+ * when testing against files under the s1d42 disconnected directory.
+ */
+ const __u64 allowed_s1d42;
+ /*
+ * File in the s1d41 disconnected directory. It should always be enforced
+ * when testing against itself under the s1d41 disconnected directory.
+ */
+ const __u64 allowed_f1;
+ /*
+ * File in the s1d41 disconnected directory. It should always be enforced
+ * when testing against itself under the s1d41 disconnected directory.
+ */
+ const __u64 allowed_f2;
+ /*
+ * File in the s1d42 disconnected directory. It should always be enforced
+ * when testing against itself under the s1d42 disconnected directory.
+ */
+ const __u64 allowed_f3;
+ /*
+ * Parent of the bind mount destination. It should always be enforced when
+ * testing against files under the s1d41 or s1d42 disconnected directories.
+ */
+ const __u64 allowed_s2d1;
+ /*
+ * Directory covered by the bind mount. It should always be ignored when
+ * testing against files under the s1d41 or s1d42 disconnected directories.
+ */
+ const __u64 allowed_s2d2;
+ /*
+ * New parent of the renamed s1d41. It should always be ignored when
+ * testing against files under the s1d41 disconnected directory.
+ */
+ const __u64 allowed_s3d1;
+ /*
+ * New parent of the renamed s1d42. It should always be ignored when
+ * testing against files under the s1d42 disconnected directory.
+ */
+ const __u64 allowed_s4d1;
+
+ /* Expected result of the call to open([fd:s1d41]/f1, O_RDONLY). */
+ const int expected_read_result;
+ /* Expected result of the call to renameat([fd:s1d41]/f1, [fd:s1d42]/f1). */
+ const int expected_rename_result;
+ /*
+ * Expected result of the call to renameat([fd:s1d41]/f2, [fd:s1d42]/f3,
+ * RENAME_EXCHANGE).
+ */
+ const int expected_exchange_result;
+ /* Expected result of the call to renameat([fd:s1d42]/f4, [fd:s1d42]/f5). */
+ const int expected_same_dir_rename_result;
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d1_mount_src_parent) {
+ /* clang-format on */
+ .allowed_s1d1 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d2_mount_src_refer) {
+ /* clang-format on */
+ .allowed_s1d2 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d2_mount_src_create) {
+ /* clang-format on */
+ .allowed_s1d2 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d2_mount_src_rename) {
+ /* clang-format on */
+ .allowed_s1d2 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d31_s1d32_old_parent) {
+ /* clang-format on */
+ .allowed_s1d31 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .allowed_s1d32 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d41_s1d42_disconnected_refer) {
+ /* clang-format on */
+ .allowed_s1d41 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE,
+ .allowed_s1d42 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d41_s1d42_disconnected_create) {
+ /* clang-format on */
+ .allowed_s1d41 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .allowed_s1d42 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d41_s1d42_disconnected_rename_even) {
+ /* clang-format on */
+ .allowed_s1d41 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .allowed_s1d42 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* The destination directory has more access right. */
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d41_s1d42_disconnected_rename_more) {
+ /* clang-format on */
+ .allowed_s1d41 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .allowed_s1d42 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_EXECUTE,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ /* Access denied. */
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* The destination directory has less access right. */
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s1d41_s1d42_disconnected_rename_less) {
+ /* clang-format on */
+ .allowed_s1d41 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_EXECUTE,
+ .allowed_s1d42 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ /* Access allowed. */
+ .expected_rename_result = 0,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s2d1_mount_dst_parent_create) {
+ /* clang-format on */
+ .allowed_s2d1 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s2d1_mount_dst_parent_refer) {
+ /* clang-format on */
+ .allowed_s2d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s2d1_mount_dst_parent_mini) {
+ /* clang-format on */
+ .allowed_s2d1 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s2d2_covered_by_mount) {
+ /* clang-format on */
+ .allowed_s2d2 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* Tests collect_domain_accesses(). */
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s3d1_s4d1_new_parent_refer) {
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s3d1_s4d1_new_parent_create) {
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs,
+ s3d1_s4d1_disconnected_rename_even){
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* The destination directory has more access right. */
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s3d1_s4d1_disconnected_rename_more) {
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_EXECUTE,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ /* Access denied. */
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* The destination directory has less access right. */
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, s3d1_s4d1_disconnected_rename_less) {
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG |
+ LANDLOCK_ACCESS_FS_EXECUTE,
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ /* Access allowed. */
+ .expected_rename_result = 0,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout4_disconnected_leafs, f1_f2_f3) {
+ /* clang-format on */
+ .allowed_f1 = LANDLOCK_ACCESS_FS_READ_FILE,
+ .allowed_f2 = LANDLOCK_ACCESS_FS_READ_FILE,
+ .allowed_f3 = LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+TEST_F_FORK(layout4_disconnected_leafs, read_rename_exchange)
+{
+ const __u64 handled_access =
+ LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE | LANDLOCK_ACCESS_FS_MAKE_REG;
+ const struct rule rules[] = {
+ {
+ .path = TMP_DIR "/s1d1",
+ .access = variant->allowed_s1d1,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2",
+ .access = variant->allowed_s1d2,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d31",
+ .access = variant->allowed_s1d31,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d32",
+ .access = variant->allowed_s1d32,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d31/s1d41",
+ .access = variant->allowed_s1d41,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d32/s1d42",
+ .access = variant->allowed_s1d42,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d31/s1d41/f1",
+ .access = variant->allowed_f1,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d31/s1d41/f2",
+ .access = variant->allowed_f2,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d32/s1d42/f3",
+ .access = variant->allowed_f3,
+ },
+ {
+ .path = TMP_DIR "/s2d1",
+ .access = variant->allowed_s2d1,
+ },
+ /* s2d2_fd */
+ {
+ .path = TMP_DIR "/s3d1",
+ .access = variant->allowed_s3d1,
+ },
+ {
+ .path = TMP_DIR "/s4d1",
+ .access = variant->allowed_s4d1,
+ },
+ {},
+ };
+ int ruleset_fd, s1d41_bind_fd, s1d42_bind_fd;
+
+ ruleset_fd = create_ruleset(_metadata, handled_access, rules);
+ ASSERT_LE(0, ruleset_fd);
+
+ /* Adds rule for the covered directory. */
+ if (variant->allowed_s2d2) {
+ ASSERT_EQ(0, landlock_add_rule(
+ ruleset_fd, LANDLOCK_RULE_PATH_BENEATH,
+ &(struct landlock_path_beneath_attr){
+ .parent_fd = self->s2d2_fd,
+ .allowed_access =
+ variant->allowed_s2d2,
+ },
+ 0));
+ }
+ EXPECT_EQ(0, close(self->s2d2_fd));
+
+ s1d41_bind_fd = open(TMP_DIR "/s2d1/s2d2/s1d31/s1d41",
+ O_DIRECTORY | O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, s1d41_bind_fd);
+ s1d42_bind_fd = open(TMP_DIR "/s2d1/s2d2/s1d32/s1d42",
+ O_DIRECTORY | O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, s1d42_bind_fd);
+
+ /* Disconnects and checks source and destination directories. */
+ EXPECT_EQ(0, test_open_rel(s1d41_bind_fd, "..", O_DIRECTORY));
+ EXPECT_EQ(0, test_open_rel(s1d42_bind_fd, "..", O_DIRECTORY));
+ /* Renames to make it accessible through s3d1/s1d41 */
+ ASSERT_EQ(0, test_renameat(AT_FDCWD, TMP_DIR "/s1d1/s1d2/s1d31/s1d41",
+ AT_FDCWD, TMP_DIR "/s3d1/s1d41"));
+ /* Renames to make it accessible through s4d1/s1d42 */
+ ASSERT_EQ(0, test_renameat(AT_FDCWD, TMP_DIR "/s1d1/s1d2/s1d32/s1d42",
+ AT_FDCWD, TMP_DIR "/s4d1/s1d42"));
+ EXPECT_EQ(ENOENT, test_open_rel(s1d41_bind_fd, "..", O_DIRECTORY));
+ EXPECT_EQ(ENOENT, test_open_rel(s1d42_bind_fd, "..", O_DIRECTORY));
+
+ enforce_ruleset(_metadata, ruleset_fd);
+ EXPECT_EQ(0, close(ruleset_fd));
+
+ EXPECT_EQ(variant->expected_read_result,
+ test_open_rel(s1d41_bind_fd, "f1", O_RDONLY));
+
+ EXPECT_EQ(variant->expected_rename_result,
+ test_renameat(s1d41_bind_fd, "f1", s1d42_bind_fd, "f1"));
+ EXPECT_EQ(variant->expected_exchange_result,
+ test_exchangeat(s1d41_bind_fd, "f2", s1d42_bind_fd, "f3"));
+
+ EXPECT_EQ(variant->expected_same_dir_rename_result,
+ test_renameat(s1d42_bind_fd, "f4", s1d42_bind_fd, "f5"));
+}
+
+/*
+ * layout5_disconnected_branch before rename:
+ *
+ * tmp
+ * ├── s1d1
+ * │ └── s1d2 [source of the first bind mount]
+ * │ └── s1d3
+ * │ ├── s1d41
+ * │ │ ├── f1
+ * │ │ └── f2
+ * │ └── s1d42
+ * │ ├── f3
+ * │ └── f4
+ * ├── s2d1
+ * │ └── s2d2 [source of the second bind mount]
+ * │ └── s2d3
+ * │ └── s2d4 [first s1d2 bind mount]
+ * │ └── s1d3
+ * │ ├── s1d41
+ * │ │ ├── f1
+ * │ │ └── f2
+ * │ └── s1d42
+ * │ ├── f3
+ * │ └── f4
+ * ├── s3d1
+ * │ └── s3d2 [second s2d2 bind mount]
+ * │ └── s2d3
+ * │ └── s2d4 [first s1d2 bind mount]
+ * │ └── s1d3
+ * │ ├── s1d41
+ * │ │ ├── f1
+ * │ │ └── f2
+ * │ └── s1d42
+ * │ ├── f3
+ * │ └── f4
+ * └── s4d1
+ *
+ * After rename:
+ *
+ * tmp
+ * ├── s1d1
+ * │ └── s1d2 [source of the first bind mount]
+ * │ └── s1d3
+ * │ ├── s1d41
+ * │ │ ├── f1
+ * │ │ └── f2
+ * │ └── s1d42
+ * │ ├── f3
+ * │ └── f4
+ * ├── s2d1
+ * │ └── s2d2 [source of the second bind mount]
+ * ├── s3d1
+ * │ └── s3d2 [second s2d2 bind mount]
+ * └── s4d1
+ * └── s2d3 [renamed here]
+ * └── s2d4 [first s1d2 bind mount]
+ * └── s1d3
+ * ├── s1d41
+ * │ ├── f1
+ * │ └── f2
+ * └── s1d42
+ * ├── f3
+ * └── f4
+ *
+ * Decision path for access from the s3d1/s3d2/s2d3/s2d4/s1d3 file descriptor:
+ * 1. first bind mount: s1d3 -> s1d2
+ * 2. second bind mount: s2d3
+ * 3. tmp mount: s4d1 -> tmp [disconnected branch]
+ * 4. second bind mount: s2d2
+ * 5. tmp mount: s3d1 -> tmp
+ * 6. parent mounts: [...] -> /
+ *
+ * The s4d1 directory is evaluated even if it is not in the s2d2 mount.
+ */
+
+/* clang-format off */
+FIXTURE(layout5_disconnected_branch) {
+ int s2d4_fd, s3d2_fd;
+};
+/* clang-format on */
+
+FIXTURE_SETUP(layout5_disconnected_branch)
+{
+ prepare_layout(_metadata);
+
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d3/s1d41/f1");
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d3/s1d41/f2");
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d3/s1d42/f3");
+ create_file(_metadata, TMP_DIR "/s1d1/s1d2/s1d3/s1d42/f4");
+ create_directory(_metadata, TMP_DIR "/s2d1/s2d2/s2d3/s2d4");
+ create_directory(_metadata, TMP_DIR "/s3d1/s3d2");
+ create_directory(_metadata, TMP_DIR "/s4d1");
+
+ self->s2d4_fd = open(TMP_DIR "/s2d1/s2d2/s2d3/s2d4",
+ O_DIRECTORY | O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, self->s2d4_fd);
+
+ self->s3d2_fd =
+ open(TMP_DIR "/s3d1/s3d2", O_DIRECTORY | O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, self->s3d2_fd);
+
+ set_cap(_metadata, CAP_SYS_ADMIN);
+ ASSERT_EQ(0, mount(TMP_DIR "/s1d1/s1d2", TMP_DIR "/s2d1/s2d2/s2d3/s2d4",
+ NULL, MS_BIND, NULL));
+ ASSERT_EQ(0, mount(TMP_DIR "/s2d1/s2d2", TMP_DIR "/s3d1/s3d2", NULL,
+ MS_BIND | MS_REC, NULL));
+ clear_cap(_metadata, CAP_SYS_ADMIN);
+}
+
+FIXTURE_TEARDOWN_PARENT(layout5_disconnected_branch)
+{
+ /* Bind mounts are handled by namespace lifetime. */
+
+ /* Removes files after renames. */
+ remove_path(TMP_DIR "/s1d1/s1d2/s1d3/s1d41/f1");
+ remove_path(TMP_DIR "/s1d1/s1d2/s1d3/s1d41/f2");
+ remove_path(TMP_DIR "/s1d1/s1d2/s1d3/s1d42/f1");
+ remove_path(TMP_DIR "/s1d1/s1d2/s1d3/s1d42/f3");
+ remove_path(TMP_DIR "/s1d1/s1d2/s1d3/s1d42/f4");
+ remove_path(TMP_DIR "/s1d1/s1d2/s1d3/s1d42/f5");
+
+ cleanup_layout(_metadata);
+}
+
+FIXTURE_VARIANT(layout5_disconnected_branch)
+{
+ /*
+ * Parent of all files. It should always be enforced when testing against
+ * files under the s1d41 or s1d42 disconnected directories.
+ */
+ const __u64 allowed_base;
+ /*
+ * Parent of the first bind mount source. It should always be ignored when
+ * testing against files under the s1d41 or s1d42 disconnected directories.
+ */
+ const __u64 allowed_s1d1;
+ const __u64 allowed_s1d2;
+ const __u64 allowed_s1d3;
+ const __u64 allowed_s2d1;
+ const __u64 allowed_s2d2;
+ const __u64 allowed_s2d3;
+ const __u64 allowed_s2d4;
+ const __u64 allowed_s3d1;
+ const __u64 allowed_s3d2;
+ const __u64 allowed_s4d1;
+
+ /* Expected result of the call to open([fd:s1d3]/s1d41/f1, O_RDONLY). */
+ const int expected_read_result;
+ /*
+ * Expected result of the call to renameat([fd:s1d3]/s1d41/f1,
+ * [fd:s1d3]/s1d42/f1).
+ */
+ const int expected_rename_result;
+ /*
+ * Expected result of the call to renameat([fd:s1d3]/s1d41/f2,
+ * [fd:s1d3]/s1d42/f3, RENAME_EXCHANGE).
+ */
+ const int expected_exchange_result;
+ /*
+ * Expected result of the call to renameat([fd:s1d3]/s1d42/f4,
+ * [fd:s1d3]/s1d42/f5).
+ */
+ const int expected_same_dir_rename_result;
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d1_mount1_src_parent) {
+ /* clang-format on */
+ .allowed_s1d1 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d2_mount1_src_refer) {
+ /* clang-format on */
+ .allowed_s1d2 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d2_mount1_src_create) {
+ /* clang-format on */
+ .allowed_s1d2 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d2_mount1_src_rename) {
+ /* clang-format on */
+ .allowed_s1d2 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d3_fd_refer) {
+ /* clang-format on */
+ .allowed_s1d3 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d3_fd_create) {
+ /* clang-format on */
+ .allowed_s1d3 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d3_fd_rename) {
+ /* clang-format on */
+ .allowed_s1d3 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s1d3_fd_full) {
+ /* clang-format on */
+ .allowed_s1d3 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d1_mount2_src_parent) {
+ /* clang-format on */
+ .allowed_s2d1 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d2_mount2_src_refer) {
+ /* clang-format on */
+ .allowed_s2d2 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d2_mount2_src_create) {
+ /* clang-format on */
+ .allowed_s2d2 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d2_mount2_src_rename) {
+ /* clang-format on */
+ .allowed_s2d2 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d3_mount1_dst_parent_refer) {
+ /* clang-format on */
+ .allowed_s2d3 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d3_mount1_dst_parent_create) {
+ /* clang-format on */
+ .allowed_s2d3 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d3_mount1_dst_parent_rename) {
+ /* clang-format on */
+ .allowed_s2d3 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s2d4_mount1_dst) {
+ /* clang-format on */
+ .allowed_s2d4 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s3d1_mount2_dst_parent_refer) {
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s3d1_mount2_dst_parent_create) {
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s3d1_mount2_dst_parent_rename) {
+ /* clang-format on */
+ .allowed_s3d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s3d2_mount1_dst) {
+ /* clang-format on */
+ .allowed_s3d2 = LANDLOCK_ACCESS_FS_REFER |
+ LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s4d1_rename_parent_refer) {
+ /* clang-format on */
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = EACCES,
+ .expected_rename_result = EACCES,
+ .expected_exchange_result = EACCES,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s4d1_rename_parent_create) {
+ /* clang-format on */
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = 0,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = EXDEV,
+ .expected_exchange_result = EXDEV,
+};
+
+/* clang-format off */
+FIXTURE_VARIANT_ADD(layout5_disconnected_branch, s4d1_rename_parent_rename) {
+ /* clang-format on */
+ .allowed_s4d1 = LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_MAKE_REG,
+ .expected_read_result = EACCES,
+ .expected_same_dir_rename_result = 0,
+ .expected_rename_result = 0,
+ .expected_exchange_result = 0,
+};
+
+TEST_F_FORK(layout5_disconnected_branch, read_rename_exchange)
+{
+ const __u64 handled_access =
+ LANDLOCK_ACCESS_FS_REFER | LANDLOCK_ACCESS_FS_READ_FILE |
+ LANDLOCK_ACCESS_FS_EXECUTE | LANDLOCK_ACCESS_FS_MAKE_REG;
+ const struct rule rules[] = {
+ {
+ .path = TMP_DIR "/s1d1",
+ .access = variant->allowed_s1d1,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2",
+ .access = variant->allowed_s1d2,
+ },
+ {
+ .path = TMP_DIR "/s1d1/s1d2/s1d3",
+ .access = variant->allowed_s1d3,
+ },
+ {
+ .path = TMP_DIR "/s2d1",
+ .access = variant->allowed_s2d1,
+ },
+ {
+ .path = TMP_DIR "/s2d1/s2d2",
+ .access = variant->allowed_s2d2,
+ },
+ {
+ .path = TMP_DIR "/s2d1/s2d2/s2d3",
+ .access = variant->allowed_s2d3,
+ },
+ /* s2d4_fd */
+ {
+ .path = TMP_DIR "/s3d1",
+ .access = variant->allowed_s3d1,
+ },
+ /* s3d2_fd */
+ {
+ .path = TMP_DIR "/s4d1",
+ .access = variant->allowed_s4d1,
+ },
+ {},
+ };
+ int ruleset_fd, s1d3_bind_fd;
+
+ ruleset_fd = create_ruleset(_metadata, handled_access, rules);
+ ASSERT_LE(0, ruleset_fd);
+
+ /* Adds rules for the covered directories. */
+ if (variant->allowed_s2d4) {
+ ASSERT_EQ(0, landlock_add_rule(
+ ruleset_fd, LANDLOCK_RULE_PATH_BENEATH,
+ &(struct landlock_path_beneath_attr){
+ .parent_fd = self->s2d4_fd,
+ .allowed_access =
+ variant->allowed_s2d4,
+ },
+ 0));
+ }
+ EXPECT_EQ(0, close(self->s2d4_fd));
+
+ if (variant->allowed_s3d2) {
+ ASSERT_EQ(0, landlock_add_rule(
+ ruleset_fd, LANDLOCK_RULE_PATH_BENEATH,
+ &(struct landlock_path_beneath_attr){
+ .parent_fd = self->s3d2_fd,
+ .allowed_access =
+ variant->allowed_s3d2,
+ },
+ 0));
+ }
+ EXPECT_EQ(0, close(self->s3d2_fd));
+
+ s1d3_bind_fd = open(TMP_DIR "/s3d1/s3d2/s2d3/s2d4/s1d3",
+ O_DIRECTORY | O_PATH | O_CLOEXEC);
+ ASSERT_LE(0, s1d3_bind_fd);
+
+ /* Disconnects and checks source and destination directories. */
+ EXPECT_EQ(0, test_open_rel(s1d3_bind_fd, "..", O_DIRECTORY));
+ EXPECT_EQ(0, test_open_rel(s1d3_bind_fd, "../..", O_DIRECTORY));
+ /* Renames to make it accessible through s3d1/s1d41 */
+ ASSERT_EQ(0, test_renameat(AT_FDCWD, TMP_DIR "/s2d1/s2d2/s2d3",
+ AT_FDCWD, TMP_DIR "/s4d1/s2d3"));
+ EXPECT_EQ(0, test_open_rel(s1d3_bind_fd, "..", O_DIRECTORY));
+ EXPECT_EQ(ENOENT, test_open_rel(s1d3_bind_fd, "../..", O_DIRECTORY));
+
+ enforce_ruleset(_metadata, ruleset_fd);
+ EXPECT_EQ(0, close(ruleset_fd));
+
+ EXPECT_EQ(variant->expected_read_result,
+ test_open_rel(s1d3_bind_fd, "s1d41/f1", O_RDONLY));
+
+ EXPECT_EQ(variant->expected_rename_result,
+ test_renameat(s1d3_bind_fd, "s1d41/f1", s1d3_bind_fd,
+ "s1d42/f1"));
+ EXPECT_EQ(variant->expected_exchange_result,
+ test_exchangeat(s1d3_bind_fd, "s1d41/f2", s1d3_bind_fd,
+ "s1d42/f3"));
+
+ EXPECT_EQ(variant->expected_same_dir_rename_result,
+ test_renameat(s1d3_bind_fd, "s1d42/f4", s1d3_bind_fd,
+ "s1d42/f5"));
+}
+
#define LOWER_BASE TMP_DIR "/lower"
#define LOWER_DATA LOWER_BASE "/data"
static const char lower_fl1[] = LOWER_DATA "/fl1";
diff --git a/tools/testing/selftests/landlock/net_test.c b/tools/testing/selftests/landlock/net_test.c
index 897131bc8a13..0be69fcc4efb 100644
--- a/tools/testing/selftests/landlock/net_test.c
+++ b/tools/testing/selftests/landlock/net_test.c
@@ -257,6 +257,64 @@ static int connect_variant(const int sock_fd,
return connect_variant_addrlen(sock_fd, srv, get_addrlen(srv, false));
}
+static int sendto_variant_addrlen(const int sock_fd,
+ const struct service_fixture *const srv,
+ const socklen_t addrlen, void *buf,
+ size_t len, size_t flags)
+{
+ const struct sockaddr *dst = NULL;
+ ssize_t ret;
+
+ /*
+ * We never want our processes to be killed by SIGPIPE: we check return
+ * codes and errno, so that we have actual error messages.
+ */
+ flags |= MSG_NOSIGNAL;
+
+ if (srv != NULL) {
+ switch (srv->protocol.domain) {
+ case AF_UNSPEC:
+ case AF_INET:
+ dst = (const struct sockaddr *)&srv->ipv4_addr;
+ break;
+
+ case AF_INET6:
+ dst = (const struct sockaddr *)&srv->ipv6_addr;
+ break;
+
+ case AF_UNIX:
+ dst = (const struct sockaddr *)&srv->unix_addr;
+ break;
+
+ default:
+ errno = EAFNOSUPPORT;
+ return -errno;
+ }
+ }
+
+ ret = sendto(sock_fd, buf, len, flags, dst, addrlen);
+ if (ret < 0)
+ return -errno;
+
+ /* errno is not set in cases of partial writes. */
+ if (ret != len)
+ return -EINTR;
+
+ return 0;
+}
+
+static int sendto_variant(const int sock_fd,
+ const struct service_fixture *const srv, void *buf,
+ size_t len, size_t flags)
+{
+ socklen_t addrlen = 0;
+
+ if (srv != NULL)
+ addrlen = get_addrlen(srv, false);
+
+ return sendto_variant_addrlen(sock_fd, srv, addrlen, buf, len, flags);
+}
+
FIXTURE(protocol)
{
struct service_fixture srv0, srv1, srv2, unspec_any0, unspec_srv0;
@@ -905,7 +963,19 @@ TEST_F(protocol, connect_unspec)
EXPECT_EQ(0, close(ruleset_fd));
}
- ret = connect_variant(connect_fd, &self->unspec_any0);
+ /* Try to re-disconnect with a truncated address struct. */
+ EXPECT_EQ(-EINVAL,
+ connect_variant_addrlen(
+ connect_fd, &self->unspec_any0,
+ get_addrlen(&self->unspec_any0, true) - 1));
+
+ /*
+ * Re-disconnect, with a minimal sockaddr struct (just a
+ * bare af_family=AF_UNSPEC field).
+ */
+ ret = connect_variant_addrlen(connect_fd, &self->unspec_any0,
+ get_addrlen(&self->unspec_any0,
+ true));
if (self->srv0.protocol.domain == AF_UNIX &&
self->srv0.protocol.type == SOCK_STREAM) {
EXPECT_EQ(-EINVAL, ret);
@@ -937,6 +1007,103 @@ TEST_F(protocol, connect_unspec)
EXPECT_EQ(0, close(bind_fd));
}
+TEST_F(protocol, tcp_fastopen)
+{
+ const bool restricted = variant->sandbox == TCP_SANDBOX &&
+ variant->prot.type == SOCK_STREAM &&
+ (variant->prot.protocol == IPPROTO_TCP ||
+ variant->prot.protocol == IPPROTO_IP) &&
+ (variant->prot.domain == AF_INET ||
+ variant->prot.domain == AF_INET6);
+ const struct landlock_ruleset_attr ruleset_attr = {
+ .handled_access_net = LANDLOCK_ACCESS_NET_CONNECT_TCP,
+ };
+ int bind_fd, client_fd, status;
+ char buf;
+ pid_t child;
+
+ bind_fd = socket_variant(&self->srv0);
+ ASSERT_LE(0, bind_fd);
+ EXPECT_EQ(0, bind_variant(bind_fd, &self->srv0));
+ if (self->srv0.protocol.type == SOCK_STREAM)
+ EXPECT_EQ(0, listen(bind_fd, backlog));
+
+ child = fork();
+ ASSERT_LE(0, child);
+ if (child == 0) {
+ int connect_fd, ret;
+
+ /* Closes listening socket for the child. */
+ EXPECT_EQ(0, close(bind_fd));
+
+ connect_fd = socket_variant(&self->srv0);
+ ASSERT_LE(0, connect_fd);
+
+ if (variant->sandbox == TCP_SANDBOX) {
+ const int ruleset_fd = landlock_create_ruleset(
+ &ruleset_attr, sizeof(ruleset_attr), 0);
+ ASSERT_LE(0, ruleset_fd);
+
+ enforce_ruleset(_metadata, ruleset_fd);
+ EXPECT_EQ(0, close(ruleset_fd));
+ }
+
+ /* Fast Open with no address. */
+ ret = sendto_variant(connect_fd, NULL, NULL, 0, MSG_FASTOPEN);
+ if (self->srv0.protocol.domain == AF_UNIX) {
+ EXPECT_EQ(-ENOTCONN, ret);
+ } else if (self->srv0.protocol.type == SOCK_DGRAM) {
+ EXPECT_EQ(-EDESTADDRREQ, ret);
+ } else {
+ EXPECT_EQ(-EINVAL, ret);
+ }
+
+ /* Fast Open to a denied address. */
+ ret = sendto_variant(connect_fd, &self->srv0, "A", 1,
+ MSG_FASTOPEN);
+ if (restricted) {
+ EXPECT_EQ(-EACCES, ret);
+ } else if (self->srv0.protocol.domain == AF_UNIX &&
+ self->srv0.protocol.type == SOCK_STREAM) {
+ EXPECT_EQ(-EOPNOTSUPP, ret);
+ } else {
+ EXPECT_EQ(0, ret);
+ }
+
+ EXPECT_EQ(0, close(connect_fd));
+ _exit(_metadata->exit_code);
+ return;
+ }
+
+ client_fd = bind_fd;
+ if (!restricted && self->srv0.protocol.type == SOCK_STREAM &&
+ self->srv0.protocol.domain != AF_UNIX) {
+ client_fd = accept(bind_fd, NULL, 0);
+ ASSERT_LE(0, client_fd);
+ }
+
+ if (restricted) {
+ EXPECT_EQ(-1, read(client_fd, &buf, 1));
+ EXPECT_EQ(ENOTCONN, errno);
+ } else if (self->srv0.protocol.domain == AF_UNIX &&
+ self->srv0.protocol.type == SOCK_STREAM) {
+ EXPECT_EQ(-1, read(client_fd, &buf, 1));
+ EXPECT_EQ(EINVAL, errno);
+ } else {
+ EXPECT_EQ(1, read(client_fd, &buf, 1));
+ EXPECT_EQ('A', buf);
+ }
+
+ EXPECT_EQ(child, waitpid(child, &status, 0));
+ EXPECT_EQ(1, WIFEXITED(status));
+ EXPECT_EQ(EXIT_SUCCESS, WEXITSTATUS(status));
+
+ if (client_fd != bind_fd)
+ EXPECT_LE(0, close(client_fd));
+
+ EXPECT_EQ(0, close(bind_fd));
+}
+
FIXTURE(ipv4)
{
struct service_fixture srv0, srv1;
diff --git a/tools/testing/selftests/nci/nci_dev.c b/tools/testing/selftests/nci/nci_dev.c
index 1562aa7d60b0..ab34e08c64d6 100644
--- a/tools/testing/selftests/nci/nci_dev.c
+++ b/tools/testing/selftests/nci/nci_dev.c
@@ -8,6 +8,7 @@
#include <stdlib.h>
#include <errno.h>
+#include <stdint.h>
#include <string.h>
#include <sys/ioctl.h>
#include <fcntl.h>
@@ -87,6 +88,16 @@ struct msgtemplate {
char buf[MAX_MSG_SIZE];
};
+static int join_thread_status(pthread_t thread)
+{
+ void *thread_ret = NULL;
+
+ if (pthread_join(thread, &thread_ret))
+ return -1;
+
+ return (int)(intptr_t)thread_ret;
+}
+
static int create_nl_socket(void)
{
int fd;
@@ -182,7 +193,7 @@ static int get_family_id(int sd, __u32 pid, __u32 *event_group)
} ans;
struct nlattr *na;
int resp_len;
- __u16 id;
+ __u16 id = 0;
int len;
int rc;
@@ -438,13 +449,13 @@ FIXTURE_SETUP(NCI)
else
rc = pthread_create(&thread_t, NULL, virtual_dev_open,
(void *)&self->virtual_nci_fd);
- ASSERT_GT(rc, -1);
+ ASSERT_EQ(rc, 0);
rc = send_cmd_with_idx(self->sd, self->fid, self->pid,
NFC_CMD_DEV_UP, self->dev_idex);
EXPECT_EQ(rc, 0);
- pthread_join(thread_t, (void **)&status);
+ status = join_thread_status(thread_t);
ASSERT_EQ(status, 0);
self->open_state = true;
}
@@ -509,12 +520,12 @@ FIXTURE_TEARDOWN(NCI)
rc = pthread_create(&thread_t, NULL, virtual_deinit,
(void *)&self->virtual_nci_fd);
- ASSERT_GT(rc, -1);
+ ASSERT_EQ(rc, 0);
rc = send_cmd_with_idx(self->sd, self->fid, self->pid,
NFC_CMD_DEV_DOWN, self->dev_idex);
EXPECT_EQ(rc, 0);
- pthread_join(thread_t, (void **)&status);
+ status = join_thread_status(thread_t);
ASSERT_EQ(status, 0);
}
@@ -585,12 +596,11 @@ int start_polling(int dev_idx, int proto, int virtual_fd, int sd, int fid, int p
void *nla_start_poll_data[2] = {&dev_idx, &proto};
int nla_start_poll_len[2] = {4, 4};
pthread_t thread_t;
- int status;
int rc;
rc = pthread_create(&thread_t, NULL, virtual_poll_start,
(void *)&virtual_fd);
- if (rc < 0)
+ if (rc)
return rc;
rc = send_cmd_mt_nla(sd, fid, pid, NFC_CMD_START_POLL, 2, nla_start_poll_type,
@@ -598,19 +608,17 @@ int start_polling(int dev_idx, int proto, int virtual_fd, int sd, int fid, int p
if (rc != 0)
return rc;
- pthread_join(thread_t, (void **)&status);
- return status;
+ return join_thread_status(thread_t);
}
int stop_polling(int dev_idx, int virtual_fd, int sd, int fid, int pid)
{
pthread_t thread_t;
- int status;
int rc;
rc = pthread_create(&thread_t, NULL, virtual_poll_stop,
(void *)&virtual_fd);
- if (rc < 0)
+ if (rc)
return rc;
rc = send_cmd_with_idx(sd, fid, pid,
@@ -618,8 +626,7 @@ int stop_polling(int dev_idx, int virtual_fd, int sd, int fid, int pid)
if (rc != 0)
return rc;
- pthread_join(thread_t, (void **)&status);
- return status;
+ return join_thread_status(thread_t);
}
TEST_F(NCI, start_poll)
@@ -830,10 +837,14 @@ int disconnect_tag(int nfc_sock, int virtual_fd)
status = pthread_create(&thread_t, NULL, virtual_deactivate_proc,
(void *)&virtual_fd);
+ if (status)
+ return status;
close(nfc_sock);
- pthread_join(thread_t, (void **)&status);
- return status;
+ if (status)
+ return -1;
+
+ return join_thread_status(thread_t);
}
TEST_F(NCI, t4t_tag_read)
@@ -874,13 +885,13 @@ TEST_F(NCI, deinit)
else
rc = pthread_create(&thread_t, NULL, virtual_deinit,
(void *)&self->virtual_nci_fd);
- ASSERT_GT(rc, -1);
+ ASSERT_EQ(rc, 0);
rc = send_cmd_with_idx(self->sd, self->fid, self->pid,
NFC_CMD_DEV_DOWN, self->dev_idex);
EXPECT_EQ(rc, 0);
- pthread_join(thread_t, (void **)&status);
+ status = join_thread_status(thread_t);
self->open_state = 0;
ASSERT_EQ(status, 0);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f1351d0ceb30..15ad929ec0d0 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1164,7 +1164,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
rcuwait_init(&kvm->mn_memslots_update_rcuwait);
xa_init(&kvm->vcpu_array);
#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
- xa_init(&kvm->mem_attr_array);
+ xa_init_flags(&kvm->mem_attr_array, XA_FLAGS_ACCOUNT);
#endif
INIT_LIST_HEAD(&kvm->gpc_list);
@@ -2429,14 +2429,36 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
return (kvm_get_memory_attributes(kvm, start) & mask) == attrs;
guard(rcu)();
- if (!attrs)
- return !xas_find(&xas, end - 1);
+ /*
+ * Lookup the entry for each index instead of iterating over the xarray
+ * as KVM deletes/nullifies entries to represent "no attributes", and
+ * the xas index is effectively invalid when no entry is found. I.e.
+ * matching non-zero attributes for *every* entry effectively requires
+ * a manually lookup for each index.
+ *
+ * Skip pre-allocated, reserved entries, or restart the lookup if the
+ * xarray was concurrently modified, via xas_retry() ("retry" means the
+ * entry holds an internal xarray value, i.e. is either invalid or NULL
+ * from the caller's perspective).
+ *
+ * Use xas_next() when looking for non-zero attributes to optimize for
+ * the case where the start of the range (or the entire range) doesn't
+ * have any attributes, as xas_next() returns literally the next entry,
+ * whereas xas_next_entry() returns the next non-NULL entry (bounded by
+ * a maximum index).
+ */
for (index = start; index < end; index++) {
do {
- entry = xas_next(&xas);
+ entry = attrs ? xas_next(&xas) :
+ xas_next_entry(&xas, end - 1);
} while (xas_retry(&xas, entry));
+ if (!entry)
+ return !attrs;
+
+ WARN_ON_ONCE(!xa_to_value(entry));
+
if (xas.xa_index != index ||
(xa_to_value(entry) & mask) != attrs)
return false;
^ permalink raw reply [flat|nested] 2+ messages in thread